=== andreas__ is now known as ahasenack [01:06] thumper-gym: re Preflight, do you have a preference for splitting it into two methods, one for creating an instance, one for a container on an instance? Then the instance.Instance parameter is never optional. === thumper-gym is now known as thumper [01:29] axw: sounds very reasonable [01:29] thumper: so i missed you before, i was stuffing my face. i got that big branch landed :-) [01:29] axw: for example, the null provider can't create machines, but could perhaps create containers [01:29] axw: what did you think of my method name suggestions? [01:29] wallyworld: good to get it landed [01:29] thumper: I quite like Vet, and Precheck isn't bad [01:30] not so keen on Probe [01:30] or Review [01:30] PrecheckCreateMachine, PrecheckCreateContainer [01:30] ? [01:30] yeah I think that sounds decent [01:31] thumper: do you think different interfaces are necessary, or is that overkill? [01:31] MachinePrechecker, ContainerPrechecker... *shrug* [01:31] I don't mind either way [01:31] chances are [01:32] that every provider will end up having it anyway [01:32] for some precheck stuff [01:32] yeah true [01:32] I'll just keep it in one then [01:32] ok [01:32] cool [01:32] thanks! [01:37] thumper: if you are tired of coding, i have a small piece of work which uses the new storage stuff to set tools retries when needed. but it can wait if you have other things to do https://codereview.appspot.com/13577045/ [01:37] tired? I haven't even started [01:38] oops, sorry [01:38] * thumper looks [01:38] you don't have to [01:53] did [01:57] thumper: with bool params, i hear you but they are used everywhere. and there are others in the tools area and felt consistency was better here [01:58] I know... but I still feel the need to push back [01:58] what about type AllowRetry bool [01:58] alright [01:58] and then two const values [01:59] it doesn't have to be exactly that [01:59] but play and see what feels good [01:59] yeah, i know the pattern to use. just felt wrong ony changing one instance of it [02:00] i'll finish my current branch and go back to it soon [02:11] anyone know where we are filing doc bugs? [02:11] hazmat: juju-core docs series isn't a bad start [02:11] they all kind of go to the same place/people anyway [02:11] yeah.. [02:12] davecheney, thanks [02:32] % juju scp gccgo/0:/home/ubuntu/gccgo.tar.bz2 . [02:32] ^ :heart juju [02:56] fwereade, curious what you think of this https://bugs.launchpad.net/juju-core/+bug/1227450 [02:56] <_mup_> Bug #1227450: juju does not retry provisioning against transient provider errors [02:57] fwereade, it seems with 1.14 things are a bit better for recovering without waiting (ie i can kill machines and services/units without the unit agent coming online)... but ideally the transient would just be auto'd [03:30] wallyworld: coming? [03:30] no 1:30 yet [03:30] still 10s to go [03:31] thumper: wallyworld can you hear/see me in the hangout? [03:31] no [03:32] try this one axw wallyworld https://plus.google.com/hangouts/_/0d2e7f7c6d9229ef8ed15d6c8f7ff08b0de146cc?hl=en [03:37] davecheney: any luck on those charm reviews? :) [03:51] bradm: i am really sorry [03:51] i know they are reviewed but have not made it into the charm store [03:51] i will try to find out (again) what is going on [03:53] davecheney: no worries, I'll just prod you every now and then about it :) [03:56] bradm: i will follow up with marcoceppi [04:11] * thumper twiddles thumbs waiting for lbox before going to make coffee [04:12] * wallyworld looks at thumper's review [04:12] * thumper noticed something wrong [04:12] poo [04:12] * thumper leaves it for now as an exercise for wallyworld to find [04:13] * thumper -> coffee [04:17] * thumper needs to help with early dinner [04:17] back for meetings tonight === thumper is now known as thumper-afk === tasdomas_afk is now known as tasdomas [05:01] mornin' all [05:04] o/ [05:13] davecheney: hiya [05:41] just ran into my first real world user who has a default vpc [05:51] zomg [05:51] how did it go [05:51] ? [05:54] davecheney, it didn't work with juju .. throws an error [05:54] thumper-afk, how do you remove a container on a machine [05:55] hazmat: poop [06:30] is there some best practice about how to ship updated code to juju instances via a charm? right now I've got to update the stuff from python-moinmoin to do openid properly, and I'm shipping files, which just feels like a terrible unmaintainable hack. [06:30] I could do patches, but thats only slightly less of an unmaintainable hack [06:31] I could try to get the code upstream, but that won't always happen [06:31] bradm: if you're talking about including code inside the charm [06:32] that is one way [06:32] otherwise you could have the charm bzr branch something [06:32] but putting the code inside the chamr will at least ensure that all units of that service get the same copy [06:32] davecheney: yeah, basically I need to patch the python code that the python-moinmoin deb puts on disk for openid [06:33] sounds nasty any way you cut it [06:33] lp:~brad-marshall/charms/precise/python-moinmoin/python-rewrite [06:34] quite a few charms have a configuration settting to control adding a ppa [06:34] so if you have that config set [06:34] thats what I've gotten so far, it does seem to work ok, I've got an instance up on canonistack [06:34] the install hook will use the ppa version [06:34] ooh, that could work [06:34] it feels like a less terrible hack [06:35] bradm: the wordpress charm is a good plcae to look [06:36] davecheney: ta, I'll have a look at that, it feels like at least then I'll know it'll all work together [06:37] it does mean I'd have to keep on top of new versions of moin that come out, and have a way of getting people to update [06:38] fork's suck [06:39] ah, a downside of that is you have to assume egress access too, which wouldn't work in our environment [06:40] maybe I should just try and get the code upstream, thats the least sucky option [06:41] anyone for a review? https://codereview.appspot.com/13778043 [07:10] hi, any update on the juju failure from yesterday? [07:32] yolanda: hi [07:33] hi rogpeppe [07:33] yolanda: i got a question that you may be able to answer [07:33] tell me [07:33] yolanda: do you still have access to an instance that this happened on? [07:33] rogpeppe, i destroyed it [07:33] but i can create a new one with the same problem [07:34] yolanda: that would be great. i'd like to find out the contents of /proc/sys/vm/overcommit_memory [07:34] ok, just a moment [07:34] yolanda: thanks [07:34] i need to have a working juju for the work i'm doing now so i'm glad to help [07:36] yolanda: for the record, here's the question i asked, and its reply from one of the Go core team: http://paste.ubuntu.com/6127199/ [07:37] shall i try with a bigger instance? [07:40] yolanda: if you can, that would be an excellent thing to try, yes [07:40] yolanda: but i'd like to see the overcommit_memory thing on an instance where the problem has happened [07:40] yolanda: so that we can try to pin down the issue [07:40] ok, still bootstrapping, it takes time to sync tools [07:44] rogpeppe, /proc/sys/vm/overcommit_memory = 0 [07:46] yolanda: ah ok, i think that probably settles it [07:46] yolanda: thanks [07:46] yolanda: i guess that too much is going on at bootstrap time [07:47] yolanda: perhaps we should run juju as root [07:49] rogpeppe, shall i try with a bigger instance to discard memory? [07:49] yolanda: yes please - i predict it'll work fine on a bigger instance [07:50] let me try [07:50] by default it uses 512mb right? [07:53] yolanda: i don't think there's any default - it probably just chooses the smallest available [07:53] trying with 1024 [07:54] mgz: ping [08:07] rogpeppe, with 1gb of memory works [08:07] so yes, it's a ENOMEM issue [08:07] yolanda: ok, thanks, at least we know where we are now [08:08] but this was working until yesterday afternoon, can it be related with some update done? [08:08] yolanda: perhaps something different is being done at bootstrap [08:08] yolanda: which uses more memory [08:09] i'll deploy a service with 1gb of memory now to test [08:10] yolanda: i'm afraid i don't know much about the details of how the openstack instances are set up [08:11] at least the defaults now should be set to 1GB to work [08:22] yolanda: yes. or we could set overcommit_memory to 1, i guess [08:23] yolanda: or, even better, work out what's taking all that memory at bootstrap time :-) [08:23] yolanda: go does grab a lot of VM at init time, although it doesn't touch it until it actually needs it [08:25] rogpeppe, but that's not directly related with juju, but with cloud-init, right? [08:42] yolanda: well, juju is written in Go, so the VM issue is partially Go-related [08:43] yolanda: but what's running at bootstrap time is indeed cloud-init and image-related [08:44] rogpeppe, i was asking that to try to file a bug, but i'm not sure on how to place it [08:49] yolanda: file it against juju, because there are several possible ways to solve the problem, not all of them in cloud-init. [08:50] rogpeppe, ok [09:05] yolanda, rogpeppe: If you are not overcommitting memory, and go is asking for a bunch that it will use later (but doesn't touch yet), doesn't that cause this issue? [09:05] jam: indeed it does [09:05] or is the go request "give me address space" different from "allocate some memory to me" [09:05] rogpeppe: I know internally we have a default "minimum memory" of 1GB for VMs. [09:06] though I think you can pass your own constraints to override it. [09:06] jam: i'm surprised that when yolanda used a 1GB constraint, that it caused it to work [09:06] rogpeppe: it might depend on which provider [09:06] works like a charm now [09:06] jam: if we do have that minimum memory default constraint [09:07] jam: yeah. are we getting this stuff from simplestreams now? i wonder if something changed there. [09:07] yolanda: which charm would that be? :-) [09:09] rogpeppe: environs/instances/instancetype.go: "minMemoryHeuristic = 1024" [09:09] which I'm pretty sure is in MB [09:10] jam: if the instance types were somehow reporting the wrong value, then it might be choosing the wrong instances, i suppose [09:11] jam: hmm, but then the change in constraints value wouldn't affect it. [09:11] rogpeppe: theoretically. It is possible that we are getting our units wrong. Do you know what provider this is? [09:11] jam: this is on canonistack [09:12] jam: and the issue only just started happening [09:12] jam: so *something* has changed recently [09:13] rogpeppe: do you know if there was any change like lcy01 => lcy02 ? [09:14] 01 runs Openstack F while 02 runs Grizzly, IIRC [09:14] jam: the same issue occurs in both, afair [09:15] rogpeppe, https://bugs.launchpad.net/juju-core/+bug/1227533 [09:15] <_mup_> Bug #1227533: Juju fails to bootstrap if memory is lower than 1GB [09:16] yolanda: thanks [09:20] yolanda: are you passing a constraint? or just no constraints? [09:20] jam, now i pass a constraint for 1024mb, without constraints it fails [09:21] i tried with canonistack only, but i think it was failing under other environments [09:21] yolanda: strange, because I bootstrapped to canonistack 2 days ago, and things are working. I'll see if I can reproduces [09:21] jam, yes, it started to fail yesterday afternoon [09:21] i've been playing with juju and canonistack for weeks [09:22] yolanda: I wonder if a new instance type showed up in the catalogue or something. [09:22] it's grabbing from simple-streams, so maybe, yes [09:22] yolanda: so instance types are in the openstack Flavor catalogue. Vs *image* which is from simplestreams [09:22] Image == Ubuntu 12.04 [09:22] Instance == m1.small [09:23] yolanda: it is possible that we got a new image, too, which might have changed overcommit [09:23] jam, do you have some booted machine previous to the failure? then you can check for the overcommit [09:24] yolanda: I do, give me a sec [09:25] my instance has overcommit_memory = 0 as well [09:25] free says I have 512MB, though [09:25] same as mine that were failing [09:25] top agrees [09:26] so it is always possible that canonistack itself is under mem pressure because of a lot of people starting instances, etc. But if it is reliable and not intermittent than I doubt that is the issue [09:27] jam, it's not intermittent, happens all time since yesterday, in the 2 zones, at different hours [09:27] yolanda: so I'll try quickly with 1.14 that is in the stable ppa, I don't know if I can get 1.12 [09:28] jam, i tested with 1.14 today [09:28] same problem, now i'm using that [09:28] rogpeppe: so at the least, we have a bug that we think we use a min-memory heuristic of 1GB but don't [09:28] yesterday i tried with 1.12 [09:28] jam: well, i think that's true - but it would be good to check the actual memory of the provided instance [09:29] rogpeppe: I did, the instance I have running for a couple of days is 512MB [09:29] jam: cool [09:30] rogpeppe: And I just bootstrapped now and got the same 512MB [09:30] jam: i've got a proposal for review, if you fancy taking a look: https://codereview.appspot.com/13778043 [09:31] rogpeppe, are you expecting https://codereview.appspot.com/13493044/ to go anywhere, or are you dropping it? I've kinda forgotten context from before this week [09:31] fwereade: ^ [09:31] rogpeppe, I'm just going through from the top today, trying to approach the bottom [09:31] rogpeppe: I do see the failure like yolanda mentioned, Cloud init reports that something failed, and I have "fatal error: runtime: ..." [09:31] rogpeppe, but it's also my meeting day, so... [09:32] jam: ah, so juju failed to start [09:32] rogpeppe: right [09:32] rogpeppe, also https://codereview.appspot.com/13512051/ [09:32] fwereade: thansk for https://codereview.appspot.com/13635044/ I'm just waiting for sshstorage to land first. [09:32] fwereade: ah, i'll submit the latter; the former can wait, i think [09:32] thanks* === thumper-afk is now known as thumper [09:33] rogpeppe: hey [09:33] thumper: yo [09:34] rogpeppe: https://codereview.appspot.com/13512051/ I think Tim has a branch that changes some names, so it will conflict (eventually), otherwise seems ok [09:38] fwereade: https://bugs.launchpad.net/juju-core/+bug/1227450 Are we intending that if the first request to Provision fails, we will ever try again? [09:38] <_mup_> Bug #1227450: juju does not retry provisioning against transient provider errors [09:38] I thought we intentionally weren't restarting things people manually stopped [09:38] which sort of falls into a similar category [09:39] jam, the original intent was that we hook up juju resolved [09:39] fwereade: so you could run "juju resolved" and it would try to provision again? [09:39] jam, yeah, that's the idea [09:40] fwereade, hey, I see you didn't have time for this - can you take a look now please? https://codereview.appspot.com/13501051/ [09:41] dimitern__, I'm ...2 away from it in the list [09:42] fwereade, ok [09:42] dimitern__, thanks for your patience [09:45] fwereade, no rush, just reminding :) [10:01] fwereade: coming? [10:06] is there a tag for 1.14? [10:43] mgz: can you link me the document? [10:59] thumper, fwereade: 1 min to meeting [11:06] mgz: fwereade, rogpeppe: https://codereview.appspot.com/13562045/ is the critical fix for Openstack security groups [11:06] jam: sorry, was paged up [11:11] jam: i don't think it would be too hard to add a live test that checked exposing [11:11] jam: but it would be slow to run [11:12] jam: unless we jammed it in the kitchen sink of BootstrapAndDeploy [11:12] rogpeppe: we don't have any tests today that run a custom program on the remote machine, so I didn't have much to go off of. It would be possible, but I'd rather land the fix and have done manual testing. [11:12] jam: agreed, good to land the fix, but please file a bug [11:15] rogpeppe: https://bugs.launchpad.net/juju-core/+bug/1227586 [11:15] <_mup_> Bug #1227586: cross-provider test that we don't expose non-juju service ports

[11:15] jam: thanks [11:17] rogpeppe: note also that we have bug #1217595 which means that "juju upgrade" won't fix the security groups that exist, and "juju destroy-environment" doesn't delete security groups so "juju destroy-environment && juju bootstrap" will leave you vulnerable, but I'm thinking to fix #1217595 [11:17] <_mup_> Bug #1217595: security groups reused without ensuring validity

[11:17] <_mup_> Bug #1217595: security groups reused without ensuring validity

[11:19] bug #1227588 [11:19] <_mup_> Bug #1227588: destroy-enviroment does not delete security groups

[11:24] jam: i don't think that the old security groups pose a security hazard, on ec2 anyway [11:24] jam: just a resource problem [11:24] jam: because the ensures that if old security groups exist, they have the correct permissions [11:25] jam: s/because the/because the code/ [11:25] jam: ah, i hadn't seen 1217595 [11:26] jam: do you know what provider that's using? [11:26] rogpeppe: openstack at least [11:26] jam: i remember writing logic in the ec2 provider that specifically tried to cope with that case [11:26] rogpeppe: note that I think we copied that behavior from ec2 [11:27] it detects a duplicate, though I'm trying to see if it updates it [11:27] "if err == nil { g = resp.SecurityGroup" [11:27] jam: see ec2.environ.ensureGroup [11:27] I do see it doing a set stuff [11:28] to find what to revoke [11:28] jam: it looks like the openstack provider doesn't do the right thing [11:29] * rogpeppe gets some lunch [11:31] rogpeppe: that was changed in codereview.appspotcom/11655043 [11:32] the alternative is using the same fiddly stuff that ec2 does with security groups [11:32] really the code should be unified regardless [11:36] mgz: I think we should move the permSet into a shared module and both use it [11:38] mgz: right, so I'm fine with doing a Get before we do Create to avoid the duplicate-with-quantum bug [11:38] but we can still take the thing we have and do a set diff to figure out what to put on it [11:41] * TheMue => AFK [11:44] * thumper -> bed [11:49] fwereade, review poke :) [11:50] mgz: it doesn't look like the openstack provider ever revoked security group rules, even before that change [11:50] dimitern__: i'll do yours if you do mine: https://codereview.appspot.com/13778043 [11:51] rogpeppe, looking [11:59] rogpeppe, reviewed [11:59] dimitern__: ta! [12:14] rogpeppe, fwereade: I have 2 CLs that need reviews please https://codereview.appspot.com/13501051/ and https://codereview.appspot.com/13627051 [12:15] dimitern__: i'm already on the first one [12:15] rogpeppe, great [12:20] dimitern__ how many underscores do you need ? :) Is it possible to share the Machine object somewhere, it sure feels repeated. [12:21] jam, not sure what do you mean? [12:22] about the underscores or the Machine? :) [12:23] it's how we agreed to do the api [12:23] for underscores, there appear to be 3 of you [12:24] my irc client surely acts funny today [12:24] ah, so that's his trick [12:24] for Machine, it looks like a tiny object with 2 bits of state and a callable that has a few apis on it. If it is actually different than the other ones thats fine [12:24] jam, they're all different [12:25] jam, and expose only a subset of the state.Machine methods, as needed by each worker [12:26] hi, using config_changed hook, what should be the best way if a config var has changed since the previous invocation? we receive the hook on each config, but i'd like to take some action if a given var has a different value. Maybe store old value in some place? [12:27] yolanda, you could save the old value at each config_changed call, and the use it in the next [12:28] yolanda, that will of course trigger a config_changed on the remote unit as well [12:28] dimitern__: Id() Life() and Tag() sure seem common, but we can deal with that some other time, I guess [12:28] jam, we can yes [12:29] dimitern__, you mean, saving on a file? not sure if i follow you [12:29] yolanda, I mean doing relation-set oldValue=X, but you can use a file as well [12:30] yolanda, that way the remote config_changed hook won't be triggered [12:30] dimitern__, maybe a file is simpler, then i avoid double relation hook call [12:31] yolanda, yes [12:31] just i was wondering if juju methods had something to retrieve previous val, may be useful [12:32] yolanda: i sometimes wonder if we should provide a standard easy way to store persistent state across hook invocations [12:32] yolanda: because everyone reinvents their own wheel there [12:32] yolanda, juju doesn't provide that at the moment [12:33] rogpeppe, dimitern__, sounds like an useful feature [12:33] rogpeppe, something like relation-save key=value [12:34] dimitern__: there's not really anything relation-oriented about it [12:34] rogpeppe, config settings are stored per relation [12:35] rogpeppe, so it makes sense to have a way to store them locally with a hook command, but the name could be anything, yeah [12:35] dimitern__: i'm not necessarily talking about config settings - just persistent state that one hook invocation can set to let another one see [12:36] dimitern__: tbh it's probably just a matter for better standard bash tooling - not something that juju-core needs to be involved in [12:36] rogpeppe, should these things be stored in state as well? [12:36] dimitern__: definitely not [12:36] rogpeppe, i agree [12:48] dimitern__: https://codereview.appspot.com/13501051/ reviewed [12:48] rogpeppe, thanks! [12:50] mgz: did you get a chance to look at: https://codereview.appspot.com/13562045/ ? [12:51] or rogpeppe. You made some comments on IRC, but didn't comment on the CL from what I can see [12:51] jam: oh, sorry, i got distracted. [12:52] rogpeppe: I'm looking at dimitern's second CL [12:53] jam: cool, thanks - i was some way through it, but happy for you to do it [12:53] rogpeppe: well certainly submit what you've gotten through [12:53] jam: i had no comments yet [12:54] jam: looked at, not gone through all the test stuff yet [12:54] will do that now [13:00] jam: reviewed [13:00] dimitern: https://codereview.appspot.com/13627051/ reviewed [13:08] natefinch: you should be doing some sort of "ssh shared@maas.mallards" [13:09] eg Username == shared [13:09] jam: oh.. duh [13:10] jam: I guess I expected the config "user" setting to do that for me, but then failed to realize it obviously wasn't [13:10] jam: using shared@ works perfectly [13:10] natefinch: it should [13:10] were you accidentally doing nate@ ? [13:10] jam: nope [13:11] jam: maybe it just wasn't picking up the config [13:11] natefinch: did you put it into config.personal? [13:11] I don't *think* that file is read by default, (It has to be imported somehow) [13:11] jam: yep [13:12] I reran sshebang afterward... [13:12] natefinch: I think it is ok to use garage mass without calendar as long as you aren't trying to allocate all 16 machines, and cleanup after yourself [13:12] actually I see User nate being specified for *.mallards in config.personal above there.... depending on which one wins [13:12] natefinch: your test cases are just provisioning 1-2 machines so it should be reasonably well behaved. [13:13] natefinch: I *think* first-entry wins [13:13] that would do it. easy enough to try [13:15] jam: yep, first one in wins [13:44] writing wiki docs makes me happy [14:02] rogpeppe, fwereade, updated https://codereview.appspot.com/13501051/ [14:05] TheMue, I can'tfind where your auto-sync-tools code is, did something happen to it? [14:06] fwereade: have to look myself [14:08] fwereade: should be this one https://code.launchpad.net/~themue/juju-core/035-bootstrap-autosync merged on Aug, 2nd [14:12] fwereade: and found it in trunk [14:13] TheMue, yep, just did likewise, thanks [14:13] fwereade: yw [14:13] TheMue, was looking right past it, sorry [14:13] fwereade: np, as long as my answer is positive (and i haven't missid to merge it *phew*) [14:51] niemeyer: ping, any luck with more details on the issue? [14:52] rick_h_: Hey [14:52] rick_h_: Not really, unfortunately [14:52] rick_h_: Now that I'm consciously trying to replicate it I can't either [14:52] thanks rick_h_ . yeah niemeyer, we want to stomp this but can't dupe :-( [14:52] yes very frustrating [14:52] niemeyer: ok, we've had 4 people try to replicate without success so far. If we can find something let me know and we'll jump right on it. Where were you linked from when it died? [14:53] rick_h_: I've accessed it directly [14:53] rick_h_: It actually stopped right on entrance the first time [14:53] niemeyer: right, your second email said you were linked there? [14:53] rick_h_: I was trying to guess the URL of a charm [14:53] rick_h_: So both times were hand-written [14:54] rick_h_: Both times CTRL-R solved the issue [14:54] niemeyer: ok, hmm. So maybe some race in a deeper url. Ok, well that gives more info to go on trying to replicate [14:54] niemeyer: on entrance - do you mean, 'loading juju-gui' ? [14:54] hatch: No, just typing jujucharms.com [14:55] :-/ [14:55] rick_h_: I wouldn't be surprised if it's something timing out [14:55] niemeyer: sorry - I mean, there are two loading messages, one is for the assets and one is for the actual juju connection [14:55] hatch: Hmm, ok? [14:56] sounds like you're getting hung up on the connection [14:56] hatch: I'm not using any juju environments [14:57] hatch: Well, that I know of [14:57] yeah it still connects on sandbox [14:57] (to an in-browser-memory fake juju) [14:57] I see, ok [14:57] yeah that :D [14:58] So, a timeout might explain why it failed on the first try, when all the caches on my path were cold [14:58] yeah odly enough that should never happen [14:59] I have some ideas to track this down though [14:59] right, we need console messages [14:59] Okay, I don't have them now, but if it happens again I'll try them [15:00] thank you very much niemeyer. ok hatch, cool, glad you have some ideas. I was going to suggest that we reply to Gustavo's message with a request for anyone who encounters this issue to please get in touch with us *before* reloading. [15:00] I'll send that out quickly [15:01] gary_poster: Sorry about that.. it was kind of stupid.. I should know to have observed the console for any hnts [15:01] np thanks for raising it niemeyer. at least we know there's a likely issue somewhere [15:05] damn schrodinbugs [15:06] fwereade: hmm, i just noticed that checkers.Set was renamed to testing.PatchValue. i'm not that keen, as the reason for it being in jc was because it has minimal dependencies so that it can be used from internal tests without fear of import loops. [15:09] fwereade: i thought it's worth discussing before i propose a move back though. [15:10] is the lean kit board reasonably up to date for me to look through it for things that have already landed but are not yet in a release? [15:11] fwereade: rogpeppe: dimitern__: TheMue: ^^^^ [15:11]