[00:29] anyone else seeing new unit test failures on master? [00:29] something changed for me over the past hour or so [00:51] fuck... [00:52] cherylj: I'm going to beg forgiveness [00:52] I'm doing a drive by [00:52] during a bug fix [00:53] because what I found is unneeded code [00:58] wallyworld: ping [00:58] wot [00:58] it wasn't me [00:58] wallyworld: do you recall how to have a JujuConnSuite test use a different user? [00:58] it was someone else [00:58] it was [00:58] I've not even bothered to look who [00:58] hmmm [00:58] I don't care [00:58] in particular [00:58] was just messing with you [00:58] I'm wanting to add a test for bug 1570594 [00:58] Bug #1570594: read access to admin model allows grant [00:58] i don't recall offhand, but i can look [00:58] there are feature tests that the command is hooked up [00:59] but I want to say "run this command as this user" [00:59] seems like something we would want to do [00:59] and I thought you may have done something like this (or axw) [00:59] yeah. you can log in from the CLI. but for a test, i'd have to look [01:01] wallyworld: in particular, I was thinking around the new local user / cred storage story [01:02] thumper: i see another feature test that invokes "logout" and then "login", which will work,but there's a way to manipulate the suite directly i'm sure [01:02] if I wanted to add somthing that would say "use this user / pwd combo" for the CLI command, how do we do it [01:02] there's a current user in the yaml [01:02] ew [01:02] current-account [01:02] we want a repo suite method that says "set this user as current" [01:03] yes, i'm looking for that [01:03] the other ways invoke setting up yaml that the cli would use [01:04] that's gross [01:04] we can do better [01:04] eg create > 1 account on a controller with different access and set up one as current [01:04] yes [01:04] how about... [01:04] ... [01:04] * thumper thinks [01:04] * thumper looks at the code some more [01:05] thumper: i can't see any feature tests for grant/revoke even. so it may be something we've not added to the suite before [01:05] it appears that the JujuConnSuite uses the default config storage (ie disk) [01:05] thumper: what about OpenAPIAs() [01:06] you get a client connection as the user you want [01:06] featuretests/cmd_juju_model_test.go [01:06] that is where the grant and revoke feature tests are [01:06] it calls the cli [01:06] from the outside [01:06] so I need to set up the "current user" [01:06] thumper: see TestCanPostWithLocalLogin [01:07] wallyworld: wrong level of interaction [01:08] so you want to set up the current user for a CLI call? [01:08] yep [01:08] juju switch-user i think [01:08] ah that's gone [01:08] nah... again wrong level [01:09] I just want to set using the config manager bits [01:09] as in: start of test create read only user, set that user as the current one [01:09] you can set up an in memory config [01:09] then invoke the cli [01:09] one sec [01:10] I'm sure we do this shit elsewhere [01:10] just with all the changes around user cred storage, I'm not sure where to look any more [01:10] it isn't where I last looked [01:11] thumper: see createmodel_test [01:11] in cmd/juju/controller [01:11] the setuptest sets up an in memory config [01:12] I don't want a memory config [01:12] but at least I can see part of what I want there [01:12] you want an on disk one in a fake home? [01:13] where is the disk config? [01:13] that is what the test currently uses [01:13] jujuclient package [01:14] because we don't have enough things with the name juju in it? [01:14] geez [01:14] NewFileClientStore() [01:14] well that's what it is [01:14] functionality for the Juju Client [01:14] we could have called the package George [01:15] heh [01:16] it seeme the JujuConnSute has one already in ControllerStore [01:17] so... what are the minimal steps I need to add a user/pwd pair to the current model for the store? [01:19] thumper: controllerStore.UpdateAccount() [01:19] that will create / update a named account (user) [01:19] for a given controller [01:20] k [01:20] the AccountDetails struct has user/password plus a macaroon which is optional [01:21] well, i think you can leave it off when testing [01:22] thumper: the controllerStore on JujuConnSuite is right now just used to hold controller info when testing CLI command structs directly. if you want to add users etc it may be best to rename it [01:23] thumper: although TestAddUserAndRegister uses the account bits [01:24] in cmd_juju_register_test [01:24] I think I have what I need... [01:25] thumper: ok, np. if you get stuck, the above test shows how to setup NewAPIConnectionParams to create an api connection [01:25] * thumper nods [01:25] using the account details you added to the store [01:33] and so it was foretold [01:33] that tests would fail [01:33] on this, the 21st day of April [01:33] when thine distro-info revealed [01:33] that the lts [01:33] is [01:33] xenial [01:33] http://juju-ci.vapour.ws:8080/job/github-merge-juju/7506/console [01:34] and no merges shall merge [01:35] Bug #1572798 opened: list-clouds or show-cloud does not show default region [01:37] * thumper headdesks [01:38] mgz, sinzui ping? [01:38] hi cherylj [01:38] hey hey sinzui [01:38] happy release day [01:38] nothing can merge now [01:38] :D [01:39] what can we do to get things to merge so we can fix the failing tests incrementally? [01:40] cherylj: 1. I open the lst cherry soda. [01:40] can we hack the distro-info on whatever instance we provision? [01:40] cherylj: 2. Just after the test script injects angsty Antelope, We change Xenial to supported or devel [01:41] sinzui: 3 - we assign these bugs POST HASTE! [01:42] alright wallyworld and thumper, time to pony up some bodies to fix unit tests :) [01:42] wallyworld: how do I set the newly added user as the default user? [01:42] cherylj: I thnk merges/*ix unit tests are fixable in 30 minutes. I hope the window's unit tests don't need sucha a hack. I cannot think how they would think xenial is lts [01:42] cherylj: redir is already doing a whole bunch :-) [01:42] cherylj: and perrito666 fixes the windows ones [01:43] sinzui: that would probably be the fallback default series, which has not been updated to be xenial yet [01:43] cherylj: , No, first open soda and order child to bring ice cream, then fix test [01:43] check for shellfish [01:43] thumper: SetCurrentAccount() [01:43] since your kids seem bent on killing you [01:43] wallyworld: yeah... that didn't work [01:43] wallyworld: there are 7 bugs for different packages with failing tests [01:43] and that's just in juju/juju [01:43] I didn't test the other repos [01:45] wallyworld: perhaps a quick hangout for a screenshare and you can tell me what I'm doing wrong [01:45] cherylj: the guys have been told to pick stuff from the board and/or ask you, i'll ask them to prioritise unit test fixes [01:45] thumper: ok [01:45] cherylj: it seems like the no matching tools one should be a quick fix [01:46] TBH I'm surprised there weren't xenial tools added to the tests before [01:46] thumper: some might be solved by changing testing/environ.go:const FakeDefaultSeries = "trusty" to "xenial" [01:46] * thumper nods [01:46] but not all [01:46] wallyworld: I'm in the hangout [01:47] cherylj: can we tag the cards with "unit test" so they are easily identified? [01:47] there's a lot of cards on that board [01:47] wallyworld: sure. These are the only critical cards in tech-debt [01:47] Bug #1572798 changed: list-clouds or show-cloud does not show default region [01:48] cherylj: they are the ones i've already asked redir to look at [01:48] the LTS ones [01:48] cherylj: I think changing the born date to 2016-05-21 will work. DO you agree [01:48] * cherylj checks locally [01:49] sinzui: yes that works [01:49] :) [01:49] if by "born date" you mean "release" [01:50] same thing [01:59] Bug #1572798 opened: list-clouds or show-cloud does not show default region [02:00] cherylj: I will send an update to all the slaves. It will take about 20 minutes. We might se everything start to work then [02:00] tyvm, sinzui [02:01] * redir is eod [02:01] * thumper read that initially as redir is dead [02:01] hehe [02:02] redir: i was just letting cherylj know we did assign resources to fixing the LTS test issues and weren't ignoring her :-) [02:03] redir: didn't expect you to do any more work past EOD [02:03] cherylj: wallyworld I'll have a qq about those tomorrow. I modified stuff to work with xenial and got all but 2 passing in cmd/jj/service [02:03] wallyworld: I'm not [02:03] :) [02:03] redir: I'll be here, just ping me [02:03] cool [02:04] redir: we should make sure it works with trusty too (as we're updating the merge bot to forget about xenial for now) [02:04] and hopefully we'll save ourselves pain for the next lts [02:04] cherylj: it can't really work with both at once [02:04] we need to choose [02:05] cherylj: I modified environs/config/config.go to say xenial [02:05] wallyworld: I think it would be a case of checking not against a hard coded series, but rather against the default series [02:05] Bug #1572798 changed: list-clouds or show-cloud does not show default region [02:05] and updated tests. THey worked (execpt 2) then reverted environs... and they still passed. [02:05] cherylj: that's where is disagree - tests should use hard coded values [02:05] cherylj: so that when stuff changes, we are forced to fix the tests [02:06] and we do this every two years [02:06] ok [02:06] that's just IMHO [02:06] others may disagree [02:06] but I think the multi series test charm code is looking for precise or trusty. [02:07] redir: that's because the test charm data contains precise and trustyu [02:07] that data doesn't come from lts [02:07] I think it should use the value of the system the tests are running on, wallyworld, cherylj $.02:) [02:07] from from the metadata.yaml for the test charm [02:07] wallyworld: I figured that but hadn't found where yet. [02:08] wallyworld: of course:| [02:08] redir: testcharms/charm-repo [02:08] wallyworld: thanks [02:09] redir: that charm metadata doesn't necessarily need to include xenial - it could, but not strictly necessary to get the current tests passing as that's not a root cause issue IIANM [02:11] wallyworld: this is so terrible [02:11] wallyworld: dumped the yaml, and there is "kontroller" and "local.kontroller" [02:13] wallyworld: if you have a minute i wouldn't mind you vetting what I am doing, before I do it all over the place... [02:13] redir: sure, jump in to https://plus.google.com/hangouts/_/canonical.com/juju-core-team [02:18] axw: it is your fault [02:19] jujuclient/file.go line 541 [02:19] apparently we shouldn't have multiple users using one controller [02:20] thumper: you can have multiple users, just not logged in at once [02:20] thumper: from the same client [02:20] cherylj: The update is in place and I have requeued alexis's and mgz's merges [02:20] thanks, sinzui! [02:20] axw: the code says no [02:20] see comment on line 533 [02:20] thumper: this is client-side. you can logout and login [02:21] thumper: the reason for the change was to avoid accidentally leaving your client logged into a controller [02:22] all our juju conn suite tests use the admin model... [02:23] thumper: can you just remove the admin user and then add your new user? [02:23] how? [02:23] how do I programatically logout? [02:23] axw: you have accidentally voluneered to prove we can delay xenial's release by one month in the tests. I will replay your lp1571832-tools-layered merge if needed [02:23] thumper: RemoveAccount(controller, "admin@local") [02:24] sinzui: wat? [02:24] how do you guys stop goimports deleting "gopkg.in/mgo.v2" all the time? [02:24] axw: nothing is merging because unit tests are failing now that xenial is the lts [02:25] axw: so we're hacking the merge bot to think that it's not release day for xenial [02:25] axw: the juju test suite fails if xenial is released, and it is in some timezones. I have a hack in place to prevent the tests from seeing xenial's release. Your branch got in before mgz's [02:26] cherylj sinzui: I'm confused, did I do something wrong? cancel my job if you need to [02:26] axw: no :) [02:26] okey dokey [02:26] axw: you're just our first victim to see if sinzui's hack is working [02:26] axw, you did nothing wrong, you just got into the queue first [02:26] ah, heh :) [02:26] * axw squeaks like a guinea pig [02:27] sinzui: so are the i386 tests gone now? [02:27] cherylj: yes they are :) [02:27] yay! [02:27] because I'm fixing the arm64 tests at the expense of i386 [02:27] :) [02:27] mwhudson: I've been meaning to hack goimports to be aware of context. there's a TODO in the code to prefer import paths that are used in other files in the same package IIRC [02:28] mwhudson: it's a PITA [02:28] axw: grumble [02:36] mwhudson: is there a different mgo package that it's choosing? What I do is just remove the incorrect package from my gopath [02:37] natefinch: i don't think so [02:38] mwhudson: when is it deleting the import? I sometimes get the problem where I manually add the import before I add the code, and then save, and goimports runs and removes the unused import. :/ [02:38] natefinch: on any save [02:39] natefinch: i think it's because the package name is different from the last component of the import path [02:39] but it's also possible i'm holding it wrong [02:39] mwhudson: hmm.. weird. I've done a lot of work using gopkg.in import paths, and it seems to work ok [02:39] hmm [02:41] mwhudson: just double checked and it adds the right import for mgo.v2 if it's missing, and won't remove it if it's there (and used). [02:41] natefinch: huh, ok, good to know [02:41] now why is it not working for me ? :( [02:42] is it an old version of goimports? maybe there was a bug fix at some point [02:42] i just rebuilt it [02:42] although let's rebuild again just to be sure [02:43] ok works now, my rebuild must not have actually rebuilt it ?! [02:43] anyway [02:43] sorry for the noise [02:43] glad it's working :) [02:43] no wait, no it's not :( [02:43] aww crap [02:45] natefinch: aah works outside of emacs [02:47] * mwhudson updates things [02:50] bah doesn't help [02:54] fuck fuck fuckity fuck [02:54] * thumper bitches some more [02:58] can I get a review? http://reviews.vapour.ws/r/4669/ [03:00] cherylj: can you slap a comment about the i386 about the fact that we don't support it anymore? I wouldn't know that by looking at that code. [03:00] above the i386 that is [03:01] cherylj: otherwise, ship it :) [03:06] sorry axw my fix worked, but your merge had one failure in cloudinitSuite.TestWindowsCloudInit [03:06] sinzui: yeah, I forgot to update a test, just rescheduled it [03:07] sinzui: sorry for holding up your fix [03:07] hmm... [03:08] cherylj, natefinch: powerpc seems a pretty safe "juju will never ever support" arch [03:10] oh fark... [03:10] * thumper headdesks again [03:12] * thumper headdesks again [03:12] * thumper headdesks again [03:16] mwhudson: it's really just a string. If we want, we could use arch=abcdef ... but I'm guessing that might fail elsewhere.... i386 is something we can parse as a valid arch that we don't support. [03:20] natefinch: yeah, that's the thing. It has to be a valid arch to juju (as defined in juju/utils/arch), but not one we support [03:21] and it was decreed yesterday that we are not building for i386 anymore [03:21] yay [03:21] cherylj: it's honestly kind of funny that we even have arches defined that we don't support. You'd think we could just remove those and then... problem solved. [03:30] weird, somehow half the tests in this suite aren't getting run [03:37] * thumper headdesks [03:37] * thumper headdesks [03:37] * thumper headdesks [03:37] * thumper headdesks [03:37] * thumper headdesks [03:43] thumper: having fun? [03:44] mwhudson: no, why? [03:44] thumper: just a suspicion i had [03:44] mwhudson: you must be very astute [03:45] lol [03:45] thumper: well i'm running juju tests so i have lots of time to be observant [03:46] oh um, this doesn't seem like good news [03:46] with juju-mongodb (i.e. 2.4): [03:47] PASS: allwatcher_internal_test.go:421: allWatcherStateSuite.TestChangeAnnotations 0.309s [03:47] with 3.2: [03:47] PASS: allwatcher_internal_test.go:421: allWatcherStateSuite.TestChangeAnnotations 24.110s [03:47] FARK!!! [03:47] wow [03:47] shit [03:47] * thumper passes that to wallyworld [03:47] told you [03:48] yep, horatio has highlighted that [03:48] ah ok cool [03:48] it sucks, we need to figure out why exactly [03:49] wallyworld: http://reviews.vapour.ws/r/4670/ [03:49] ok [03:50] wallyworld: it has a drive by cleanup I couldn't let lie [03:51] wallyworld: the important bit was that sharing models currently only done by controller admins [03:51] which is crazy [03:51] but that's what we have [03:52] but read only access to the controller model made you a controller admin [03:52] which I've fixed [03:52] in this branch [03:52] good :-) [03:55] thumper: so what happened to all that angst that the grant/revoke methods were on the wrong facade? [03:55] that is a different issue [03:55] that's still fucked [03:55] but we are no more fucked than before [03:55] but more unfucking needed [03:55] * thumper is emailing... [03:55] so the root issue here was a Find() was missing a condition [03:57] wallyworld: for admins, yes [03:57] we didn't have readonly before [04:08] oh FFS === hazmat_ is now known as hazmat [06:53] wallyworld: https://github.com/juju/juju/pull/5244 -- fixes critical issue [06:53] sure [06:54] axw: oh, i thought we had that filtering where needed [06:54] wallyworld: so did I ... [06:54] is this a regression? [06:54] wallyworld: apparently only for LXC [06:54] I don't know [06:54] right ok, lxc in 1.25/2.0 worked [06:54] but not lxd [06:54] wallyworld: not sure if we were ever doing the right thing for KVM, seems unlikely that we'd have changed it [06:55] kvm never got much love [06:55] on non amd64 [07:04] axw: i just asked for an extra test [07:15] wallyworld: ok, thanks === urulama|____ is now known as urulama [08:15] mgz: ping [08:24] frobware: ping [08:41] morning [08:42] TheMue: o/ [08:52] voidspace: twittering about management experiences ;) [08:54] TheMue: :-) [09:01] dimitern, frobware, voidspace: standup? [09:01] dooferlad: omw [09:31] dimitern: so on a maas controller with extra interfaces juju bootstrap fails due to a schema error in gomaasapi [09:32] dimitern: the way networks are returned fails thumper's json validation code :-) [09:32] dimitern: so a fix in gomaasapi needed [09:32] I'm looking :-) [09:32] good to find these things out I guess [09:32] voidspace: oh, what's the error? [09:33] dimitern: ERROR failed to bootstrap model: cannot start bootstrap instance: cannot run instances: cannot run instance: space 0: subnet 3: subnet 2.0 schema check failed: dns_servers: expected list, got nothing [09:34] dimitern: it looks like it can't get all the subnet info - which is possibly a problem on the configuration [09:34] dimitern: but the controller is running fine, so gomaasapi shouldn't refuse to talk to it [09:35] voidspace: ah! it already is so much better at figuring out how exactly maas api changed [09:35] voidspace: yeah, the dns_servers should be optional [09:37] dimitern: yeah, but the old approach was tolerant against schema changes like that ;-) [09:37] oh the good old days [09:38] :D [09:41] dimitern, voidspace, dooferlad: morning - running late today :/ [09:42] frobware: morning o/ [09:42] fwereade: are you around? [09:42] dimitern, o/ [09:42] dimitern: have you commissioned a node in MAAS with xenial recently? [09:43] frobware: not since BOW IIRC [09:44] fwereade: do you have 10m or so for a quick HO? [09:44] dimitern: I have no working nodes this morning... :( [09:45] fwereade: context: I'm almost done with refactoring state ports to associate them with subnets, rather than networks [09:45] frobware: I'll try recommissioning one of the nucs now, as I've upgraded to 2.0 beta3 this morning [09:46] dimitern, sure, joining juju-sapphire [09:46] fwereade: omw === sinzui_ is now known as sinzui [10:20] frobware: thanks for the review. I don't think there's much point in tacking on the type to the name, since you'll never see the name out of the context of the type anyway [10:20] frobware: unless space awareness would change that ... [10:20] frobware: would private networks like these be exposed to the user? [10:21] axw: my comment was more from a debugging standpoint should you want to identify these "things" separately. [10:22] frobware: ok then, I guess I can buy that. I'll change it tomorrow [10:22] thanks [10:25] voidspace: after all the twitter harrassment, you got the day wrong again ;) [10:26] voidspace: anything I can do to help with the bootstrap issue? [10:26] babbageclunk: I just haven't corrected it yet :-p [10:26] babbageclunk: I'm on it, thanks [10:27] voidspace: ok, I'll keep going through the tests [10:27] babbageclunk: hah [10:27] babbageclunk: you could test my branch deploying a container in the standard case [10:27] babbageclunk: I can't test due to fixing gomaasapi [10:27] ok [10:27] voidspace: what's the standard case/ [10:28] ? [10:28] babbageclunk: add a machine (used to be juju add-machine) [10:28] babbageclunk: then juju deploy wordpress --to lxc:1 [10:28] or whatever the number of the new machine is [10:28] might be 0 if you've not switched to admin, not sure [10:28] voidspace: ok cool - trying that now [10:28] babbageclunk: standard case is without extra subnets / nics which is what I was attemptin [10:28] *attemptin [10:30] what the hell [10:30] voidspace: nice correction [10:30] *attempting [10:30] a failed attemp [10:37] frobware: that nuc I tried to recommission failed\ [10:38] frobware: telltale sign from the rackd.log: http://paste.ubuntu.com/15962619/ [10:39] dimitern: I'm totally broken too... [10:41] frobware: it's even reported: https://bugs.launchpad.net/maas/+bug/1389811 [10:41] Bug #1389811: tgtadm: out of memory crash [10:41] Man that is really annoying - sometimes my machine just freezes for about 2 minutes. [10:42] I think it's journald freezing. [10:43] babbageclunk: that used to happen quite often with all of my vmaas setups in lxc, all of which also bind-mount my $HOME inside the container [10:43] dimitern: so I see another error to: 'sh initctl - not found' [10:44] dimitern: encrypted home? [10:44] dimitern: I cannot persuade the UI to use trusty for commissioning [10:44] babbageclunk: what was observable is ssh starting to fail to connect; sudo systemctl restart logind resolves this for a while; haven't see in happen recently though [10:47] dimitern: does commissioning in 1.9 work? [10:47] dimitern: should you have the inclination and bandwidth... [10:48] frobware: have you tried changing the release used for commissioning in the UI Settings tab? [10:49] dimitern: yep, it won't let me. :/ [10:49] dimitern: I tried fiddling with maas $PROFILE boot-sources ... [10:50] frobware: perhaps it's by design - i.e. not allowing commissioning with earlier releases than the controller [10:51] dimitern: and I removed 16.04 from my images, leaving 14.04 but I still cannot select 14.04 [10:53] frobware, voidspace, dimitern: sometimes when I start my maas controller KVM the maas admin isn't accessible - apache's running but going to /MAAS just spins [10:56] frobware, voidspace, dimitern: any suggestions? [10:56] babbageclunk: how much ram did you allocate for the maas server? [10:56] frobware: 1G [10:56] babbageclunk: also, cd /var/log/maas and take a look [10:57] babbageclunk: I did that for a while, but I think it needs a little more, particularly when re-importing the images [10:57] frobware: no idea why it doesn't allow you :/ - anything in /var/log/maas/ ? [10:58] frobware: 2GB seems to be recommended [10:58] frobware, dimitern: "Can't update service statuses, no RPC connection to region." [10:59] I can ping google, for example, so there's network connection out. === mhilton-lunch is now known as mhilton [11:01] babbageclunk: check the log? [11:01] dimitern, frobware: ok, can I just bump it up in VMM, or do I need to recreate. [11:02] ? [11:02] babbageclunk: 1G may not be enough [11:02] babbageclunk: bumping it in the VM should be fine [11:02] babbageclunk: power it down, bump the value via the virt-manager interface for the node [11:02] Found a promising sounding bug from the log messages. [11:03] babbageclunk: try sudo dpkg-reconfigure maas-region-controller, then maas-region-api, then maas-rack-controller (in that order) [11:04] ok, I can see messages in the rack controller log complaining the region isn't responding. [11:04] dimitern: ok, will do [11:06] dimitern: so I reinstalled maas (gah!) and just added 14.04 images but that is not enough. the "deploy" dropdown is populated with 14.04 but the commissioning dropdown is empty. looks like 16.04 or bust. [11:09] frobware: I suspect you need to wipe /var/lib/maas/boot-resources/* and re-import the images to be sure it's clean [11:09] dimitern: I did a clean VM install. :) [11:10] frobware: oh :/ I see [11:11] * frobware productively heads for lunch :/ [11:11] Ok, poking those helped - not sure which bit, presumably the reconfigures restarted things - thanks guys! [11:16] dimitern: yep, import 16.04 gives me the only boot image. [11:18] frobware: fwiw I can see both trusty and xenial in the dropdown to choose release for commissioning [11:18] dimitern: 1.9.1? [11:18] maybe upgrading from 1.9 works better than fresh installation [11:19] frobware: no, 2.0.0 (beta3+bzr4941) [11:25] * frobware really heads for lunch [11:27] voidspace: still bootstrapping, haven't had a chance to add a container yet. [11:28] babbageclunk: heh, ok [11:29] babbageclunk: good news about your oyster card (etc) [11:29] babbageclunk: gives you hope for humanity [11:35] voidspace: yeah! And such a nice postcard as well" [11:35] ! [11:36] voidspace: please no monitoring of typing speed [11:36] babbageclunk: ah, I missed that. Nice. [11:36] babbageclunk: I'll email you the keylogger [11:36] ;-) [11:37] voidspace: bootstrap seemed to have hung after "installing package: tmux" [11:37] voidspace: killed it and kicked it off again [11:37] voidspace: any way to get debug logging before the controller is bootstrapped? [11:37] babbageclunk: anything in the logs? [11:38] babbageclunk: installing to container or machine? [11:38] voidspace: locally or on the node? [11:38] babbageclunk: on the node [11:38] babbageclunk: installing directly to the node, or to a container on the node? [11:38] voidspace: I couldn't ssh in - where's the key? [11:39] voidspace: still bootstrapping, haven't gotten to the add-machine part. [11:39] babbageclunk: oh I see - as part of the bootstrap, I'd forgotten it installed tmux [11:40] babbageclunk: uhm, not sure - dimitern ^^^ (where's the key to ssh in) [11:40] voidspace: yeah, what's that about - just so it's there? [11:40] voidspace, babbageclunk: juju ssh keys are in ~/.local/share/juju/ssh/ [11:41] dimitern: ah, ok - will try using those to snoop around if it gets "stuck" again. [11:41] you could try ssh -i ubuntu@ [11:41] while it's still bootstrapping [11:42] and if it appears hung, might be just slow to upd / inst the packages; i.e. tail -f /var/log/cloud-init-output.log once in [11:42] babbageclunk: ^^ [11:43] dimitern: awse, thanks [11:45] ok, looks like it's actually doing things with tools. [11:46] voidspace, dimitern: was there some breakage around upload-tools recently? [11:46] babbageclunk: there [11:47] babbageclunk: oh yeah - but it's fixed now (2 days ago sometime) [11:47] babbageclunk: oops, there was - but it should be fixed [11:49] babbageclunk: bootstrap --upload-tools failed almost right away for me before the fix [11:50] babbageclunk: or you mean the tools info mismatch (new thing since yesterday IIRC) ? [11:53] that happens for lxd containers due to some arch mismatch between host and container [11:53] dimitern: no, I was thinking of the one earlier this week. [11:54] dimitern, voidspace: nothing happening in the cloud-init output log after this: [11:54] 454a97ab1b6bc43e28bb149813edfeec31eceda09b86b5474cca31712528f163 /var/lib/juju/tools/2.0-rc1.1-xenial-amd64/tools.tar.gz [11:55] babbageclunk: anything in /v/l/syslog? [11:55] bootstrap still seems stalled after installing tmux (log says that was successful/already installed) === icey_ is now known as icey [11:55] I'll try [11:56] babbageclunk: is this master or my branch [11:56] dimitern: nothing suspicious looking. last message is: Apr 21 11:54:06 undisabled-miya systemd[1]: Started Cleanup of Temporary Directories. [11:57] dimitern: (I presume that's in UTC) [11:57] voidspace: yours [11:57] babbageclunk: any zombie processes? [11:57] dammit, I can't bootstrap because of my maas configuration issue [11:57] will leave it for the moment, sorry [11:59] dimitern: not that I can see [12:00] dimitern: any other logs I should look for? [12:01] babbageclunk: can you paste the current /v/l/c-i-output.log ? [12:01] can anybody help? i'm getting: WARNING cannot delete default security group: cannot retrieve default security group: "juju-3884d5ab-1317-420a-8acc-fb8bceb7d70a": The security group 'juju-3884d5ab-1317-420a-8acc-fb8bceb7d70a' does not exist in default VPC 'vpc-f34a3496' (InvalidGroup.NotFound) [12:01] ERROR failed to bootstrap model: cannot start bootstrap instance: no "xenial" images in us-east-1 with arches [amd64] [12:04] dimitern: http://pastebin.ubuntu.com/15964229/ [12:04] dimitern: hey, it just finished! [12:07] dimitern: babbageclunk: confirmed to work https://github.com/juju/gomaasapi/pull/44 === urulama is now known as urulama|afk [12:08] babbageclunk: yeah, it looks like it's just slow - a lot of published upgrades to apply [12:08] voidspace: looking [12:11] dimitern: the error was "got nothing" [12:11] voidspace: on my maas 2 I can see `"dns_servers": []` for some of the subnets [12:11] dimitern: that was the case that was already tested and worked [12:13] dimitern: ah, indeed - it is null [12:13] Bug #1567296 changed: Plugin API fails with multiple juju binaries [12:13] dimitern: I will update and change to null instead [12:13] dimitern: test still passes [12:13] voidspace: thanks! [12:14] dimitern: pushed [12:14] voidspace: it passes because go's json package allows that :) [12:14] dimitern: the test uses the schema - so we're testing the schema not the json package [12:15] dimitern: unless you meant something else [12:15] the problem was the over strict schema [12:15] without my change to the schema the test fails [12:16] dimitern: thaanks [12:16] voidspace: yeah, layers upon layers.. now LGTM though :) [12:16] lunch === urulama|afk is now known as urulama [13:16] Bug #1573020 opened: upgrade-charm --path does not recognize dot [13:36] frobware: I managed to fix my nucs and now commissioning passes ok [13:37] dimitern: see my conversation on #maas [13:43] frobware: fwiw I struggled about with this bug 1389811 until now, added comment how managed to resolve it [13:43] Bug #1389811: tgtadm: out of memory crash [13:44] frobware: (reading #maas) - nice! good catch === dooferlad_ is now known as dooferlad [14:17] voidspace, dimitern - trying to deploy to a container I get this error; [14:18] ERROR storing charm for URL "cs:trusty/ubuntu-7": cannot retrieve charm "cs:trusty/ubuntu-7": cannot get archive: Get https://api.jujucharms.com/charmstore/v5/trusty/ubuntu-7/archive?channel=stable: dial tcp: lookup api.jujucharms.com on 192.168.150.2:53: read udp 192.168.150.4:37104->192.168.150.2:53: i/o timeout [14:18] babbageclunk: yeah, no xenial versions of the charm yet [14:19] babbageclunk: but never worry :) try e.g. juju deploy ubuntu --series xenial --force --to lxd:0 [14:19] Ok - trying that, thanks [14:20] dimitern: nope, same error. [14:20] babbageclunk: try deploying wordpress instead [14:20] maybe the same problem [14:20] yeah [14:20] dimitern: hmm, maybe it can't get to t'internet? [14:21] dimitern, voidspace: ok, trying that [14:21] babbageclunk: try with --debug? [14:21] all these fun new options I'm learning" [14:21] ! [14:22] babbageclunk: well, I'd first try just add-machine lxd:0 to make sure that works, comes up with expected NICs, dns works, egress to internet works [14:22] Bug #1573055 opened: Latest modification to https://streams.canonical.com/juju/tools/streams/v1/com.ubuntu.juju-released-tools.json makes juju bootstrap fail [14:22] babbageclunk: once all the above works (at least a couple of times :), you're almost there [14:22] dimitern: looks like DNS from that machine isn't working. [14:23] This VLAN is the default VLAN in the fabric, can't delete. Can't delete this fabric it has VLANs attached. [14:23] babbageclunk: ah, well - what's in /etc/resolv.conf on the host and the container? [14:23] voidspace: first delete the vlans [14:24] dimitern: resolv.conf looks right, points to the maas controller [14:24] babbageclunk: can you ping the dns IP from the container? [14:26] Hang on, I'm not in the container yet - only have a machine. [14:26] dimitern: can't delete the VLAN it's the default in the fabric. [14:28] voidspace: what are you trying to do? get rid of extraneous fabric? [14:28] babbageclunk: what does sudo lxc list show? [14:28] Hmm, didn't have DNS forwarding turned on in my MAAS 2 controller. Added that and now pinging things on the internet works. [14:28] dimitern: to see if I can bootstrap if I go back to the previous configuration :-/ [14:28] because right now it won't bootstrap [14:28] Don't quite understand how the bootstrap worked given that. [14:29] deleting the NICs from the VM worked [14:29] well, got rid of the fabrics anyway [14:29] good :) [14:29] no, they're still there - but not listed on the controller any more - which may be enough [14:31] Bug #1573055 changed: Latest modification to https://streams.canonical.com/juju/tools/streams/v1/com.ubuntu.juju-released-tools.json makes juju bootstrap fail [14:34] voidspace, dimitern: Ok, from juju debug-log, I'm getting this error: [14:35] voidspace, dimitern: actually, these errors: http://pastebin.ubuntu.com/15966208/ [14:35] babbageclunk: try lxc instead of lxd - I don't trust lxd :-) [14:35] (yet) [14:35] voidspace: ok [14:35] babbageclunk: ah, there's an interesting error though [14:36] schema check failed [14:36] voidspace: also, this bit seems cromulent: device 2.0 schema check failed: interface_set [14:36] looks like interface_set needs to be optional too [14:36] voidspace: snap [14:36] ok, trying lxc instead [14:36] babbageclunk: see this PR and do the same for interface set in gomaasapi https://github.com/juju/gomaasapi/pull/44 [14:37] babbageclunk: it can be useful to try what juju does from the maas cli? [14:37] to look at interface_set, "machines read" should be enough [14:37] and find the place with null :-/ [14:37] dimitern: yeah, that sounds good [14:38] Bug #1573055 opened: Latest modification to https://streams.canonical.com/juju/tools/streams/v1/com.ubuntu.juju-released-tools.json makes juju bootstrap fail [14:38] babbageclunk: see here (if you haven't) http://paste.ubuntu.com/15966260/ [14:38] *phew* bootstrap is slow but it worked this time [14:39] well, is getting further anyway [14:39] dimitern: I haven't - what is that? [14:40] babbageclunk: that's my test-bed script which I used as PoC how to do the multi-nic devices [14:41] it's fairly messy, opinionated, and not portable I'm afraid [14:41] but it implements the necessary API calls in the right order [14:42] dimitern: useful as a starting point, thanks! [14:42] voidspace: deploying to lxc is working well. [14:43] babbageclunk: cool, that means AllocateContainerAddresses maybe basically works... [14:43] although I still see the same schema error in the logs. [14:43] kind of neat that it keeps going in spite of that. [14:44] Ok, I'll fix that in the gomaasapi schema [14:44] babbageclunk, voidspace: the schema error like any other issues during PrepareContainerInterfaceInfo() are simply ignored [14:45] dimitern: nice [14:45] so to truly verify whether it worked, you need at minimum: 1) be able to ssh to the host, 2) sudo lxc exec juju-machine-0-lxd-0 bash, 3) ping bbc.co.uk, 4) ip route show, 5) cat /e/n/i.d/00-juju.cfg [14:46] also helps to check sudo lxc list - it should show multiple NICs there [14:47] Bug #1573055 changed: Latest modification to https://streams.canonical.com/juju/tools/streams/v1/com.ubuntu.juju-released-tools.json makes juju bootstrap fail [14:47] dimitern: is that to me? [14:48] babbageclunk: yeah - just sharing how I verify usually [14:48] I couldn't deploy to an lxd - it failed with the simplestreams error. [14:48] What would the equivalent be for lxc? [14:49] babbageclunk: ah, sorry - well, lxc should work just the same (only slower to come up due to the initial cloning, etc.) [14:49] babbageclunk: so juju add-machine lxc:0 [14:50] dimitern: How do I list LXC containers? The container doesn't show up on the host node under lxc list. [14:50] babbageclunk: sudo lxc-ls -f [14:50] dimitern: thanks [14:51] deploying to lxc I get this in the logs (i.e. it doesn't work): [14:51] machine-0: 2016-04-21 14:50:24 WARNING juju.provisioner lxc-broker.go:115 failed to prepare container "0/lxc/0" network config: unexpected: ServerError: 500 INTERNAL SERVER ERROR (Subnet matching query does not exist.) [14:54] voidspace: it might help setting the logging-config to '=TRACE' to get (a lot more) context [14:54] yeah [14:55] dimitern: ok, and using lxc-attach bash I can get into the container. I can ping bbc, and there are routes (not exactly sure what they should be). There's no 00-juju.cfg in /e/n/i.d (which will henceforth be known as enid). [14:55] enid :-) [14:55] But this isn't a multi-nic situation, so maybe that's to be expected. [14:56] babbageclunk: lxc-attach might be misleading btw [14:57] babbageclunk: I'd use 'sudo lxc-attach -n juju-machine-0-lxc-0 -e bash' to be sure you're in the container ns and not the host [14:58] babbageclunk: check /var/log/cloud-init-output.log while inside the container for any issues [15:01] ericsnow: standup time [15:01] dimitern: all looks pretty good - at least, there's a nice looking finished message [15:07] babbageclunk: from cloud-init? [15:08] dimitern: yup [15:08] Cloud-init v. 0.7.5 finished at Thu, 21 Apr 2016 14:40:19 +0000. Datasource DataSourceNoCloudNet [seed=/var/lib/cloud/seed/nocloud-net][dsmode=net]. Up 12.0 seconds [15:15] babbageclunk: so it seems to be working inside the lxc? [15:16] babbageclunk, voidspace, fwereade: updated http://reviews.vapour.ws/r/4656/ and I'd appreciate one last look [15:19] I suppose it's to be expected to start seeing "expected series 'trusty', obtained 'xenial'" test failures on master now [15:21] more of those: adding new machine to host unit "django/0": cannot assign unit "django/0" to machine 0: series does not match [15:32] Bug #1573099 opened: Juju Register not test / script friendly; requires console interaction [15:35] babbageclunk, voidspace, fwereade: any objections to land http://reviews.vapour.ws/r/4656/ ? [15:35] dimitern, I'm afraid I can't devote proper attention to it today :( [15:36] dimitern: I've already said I love it! Unfortunately I'm not valid yet. :( [15:36] fwereade: np [15:37] babbageclunk: cheers - so we're waiting on voidspace and/or frobware :) [15:37] dimitern: looking now [15:37] frobware: thanks! [15:38] dimitern: if you are using spaces don't charms implicitly express which subnet they refer to? (I was reading the commit message) [15:40] frobware: yeah, but charms see addresses as well [15:40] frobware: and they're more important for opening ports than spaces (if the charm even cares) [15:42] frobware: did that make sense ? :) [15:43] dimitern: I got sidetracked with the diff [15:43] frobware: ah, yeah it's a bit big, but at least makes things more consistent around migrations [15:44] dimitern: why did we go from s/0:juju-public/0:/ ? [15:45] frobware: juju-public was the hardcoded name of the supposedly default public network, now it's an empty subnet CIDR instead [15:46] dimitern: so 0: represents the CIDR? [15:46] frobware: it can be also non-empty (tests verify that), but no user(charmer)-facing way to do it [15:46] voidspace, frobware, dimitern: review please? https://github.com/juju/gomaasapi/pull/45 [15:46] frobware: the colon separates machine ids and subnet cidrs, so the 0 is the machine id [15:49] babbageclunk: looking [15:49] dimitern: ok... wasn't obvious to me [15:50] frobware: it's explained in the doc comment of state.WatchOpenedPorts [15:50] oops, now I see that comment needs updating as well [15:51] babbageclunk: actually, when can we have a device without interface_set ? [15:51] dimitern: That's what I see in my MAAS now [15:52] babbageclunk: I can't see that happening using the maas cli manually: http://paste.ubuntu.com/15967504/ [15:53] babbageclunk: can you try that on your maas (the parent needs to match one of your nodes ofc) [15:53] http://pastebin.ubuntu.com/15967528/ [15:54] oh, sorry - I didn't do the create [15:54] hang on [15:54] babbageclunk: ah! the warning might be causing that [15:54] babbageclunk: try 'maas refresh' to update the api spec cached by the maas cli client [15:55] dimitern: ok, did that - still no interface_set [15:55] babbageclunk: try 'devices create' instead of 'devices read' [15:55] might be different [15:56] dimitern: just tried that, still the same. [15:56] man, I love the name generator: frondescent-rosalie [15:56] :) [15:57] babbageclunk: hmm - what does 'maas maas2 version read -d' return? [15:57] although, I'm still getting the warning [15:59] http://paste.ubuntu.com/15967646/ [15:59] babbageclunk: 'maas refresh' won't work (at least used to) if any of the profiles' urls you have ('maas list') are unaccessible [15:59] dimitern: only the one in there [15:59] babbageclunk: uuh - rather old [16:00] babbageclunk: 2.0.0 alpha3+bzr4810 [16:00] dimitern: true - shall I upgrade and try again? [16:00] babbageclunk: please do, at least we've confirmed the response format between alpha3 and beta3 is different [16:01] dimitern: okeydoke, running now [16:01] babbageclunk: are you using the packages in the experimental3 ppa? [16:01] dimitern: I don't remember adding that - how'd I check - look in sources? [16:02] babbageclunk: apt-cache madison maas [16:02] it what I use [16:03] dimitern: madison is a weird subcommand name [16:03] babbageclunk: it had some good story behind it as well, but can't remember now :) [16:04] babbageclunk: you can: sudo add-apt-repository ppa:maas-maintainers/experimental3, then update and dist-upgrade [16:10] frobware: review poke (sorry to be a pest) [16:10] dimitern: still looking at it [16:10] frobware: ok, take your time [16:11] frobware: I'll rebase the follow-up onto it and keep going for now [16:12] dimitern: I'm almost done [16:12] frobware: sweet! [16:14] dimitern - my maas version still reports the old version - is there a command to restart the various daemons at one fell swoop? [16:14] babbageclunk: reboot [16:14] babbageclunk: it's all virtual, it's all quick [16:14] frobware: yeah - I guess that would work just as well [16:14] frobware: just feels a bit windowsy [16:14] * frobware blushes [16:15] :( [16:15] Oops :) [16:15] babbageclunk: yeah, I usually reboot to ensure kernel updates etc. take effect [16:15] (with dist-upgrade esp.) [16:18] dimitern: ah, always with dist-upgrade. always. [16:18] dimitern: reviewed [16:19] frobware: thanks! [16:24] dimitern: now maas-cli is back to hanging. Do I need to do the dpkg-reconfigure every time? [16:26] Or is that because of the upgrade? [16:28] dimitern, voidspace, babbageclunk: hopefully an easy one: http://reviews.vapour.ws/r/4673/ [16:28] babbageclunk: what do you mean? it doesn't terminate? [16:28] dimitern: yeah - I run maas maas2 version read -d and have to ^C it [16:29] service maas-rackd status complains about the region controller being missing. [16:29] babbageclunk: that's not a good sign :/ how about the web ui? [16:29] babbageclunk: re 'madison' - http://unix.stackexchange.com/questions/276037/why-apt-madison [16:30] service maas-regiond status says active (exited) with nothing in the log. [16:30] babbageclunk: hmm yeah - it seems dpkg-reconf might be in order [16:30] dimitern: Already tried, no dice [16:31] dimitern: I'mma reboot again [16:31] frobware: LGTM [16:32] babbageclunk: which package did you reconf? [16:35] dimitern: maas-region-controller and maas-rack-controller [16:35] dimitern, babbageclunk, voidspace: need to drop early today... [16:36] Is there a way to make review board ignore any whitespace changes? gofmt changing the gaps for blocks of assignments makes it really hard to see what changed. [16:36] ericsnow: ^^^ [16:37] Ooh, release drinks [16:38] babbageclunk: there's a "Hide whitespace changes" toggle in the diff view, on the right below the list of files at the top [16:38] babbageclunk: is that what you're talking about? [16:42] weeelll... I now have a new lxd container failure [16:42] working through them [16:43] babbageclunk: maas-region-api IIRC is the new name of the maas-region-controller? anyway - try reconf'g that [16:44] babbageclunk: :/ release issues I guess - sorry for suggesting the upgrade :( [16:46] babbageclunk: in anycase at least I verified on a working beta3 MAAS we have interface_set, and on your alpha3 we don't, so it's good to be flexible about that in gomaasapi for the time being; so lgtm on your https://github.com/juju/gomaasapi/pull/45 [16:47] do we have the custom signed image stream thing merged? [16:53] Bug #1565044 changed: s390x unit tests fail because not tools for arch [16:53] Bug #1573136 opened: kill-controller is stuck [16:55] alexisb: ping [16:56] dimitern, pong [16:56] whats up? [17:23] Bug #1573148 opened: Juju terms language confusing for locally-deployed charms [17:23] Bug #1573149 opened: 'failed to ensure LXD image' creating LXD container [17:33] Going afk for a moment === JoseeAntonioR is now known as jose [18:24] how do I set the log level in juju 2, now there's no set-env [18:25] voidspace: set-model-config [18:25] voidspace: took me a while to figure that out too [18:26] natefinch: ok, thanks. set-model-config *what*? [18:27] ah, no [18:27] natefinch: never mind [18:27] thanks :-) [18:27] welcome :) [18:42] natefinch: PTAL: http://reviews.vapour.ws/r/4675/ [18:45] ericsnow: will do [18:45] natefinch: ta [18:50] ericsnow: http://reviews.vapour.ws/r/4676/ [18:50] natefinch: k [19:02] * perrito666 tries to discover why tomb is sudenly not working in go1.6 in windows [19:20] ericsnow: btw, an example of using the functionality added in that PR is here: http://reviews.vapour.ws/r/4677/ [19:23] Bug #1557052 changed: Can't bootstrap a trusty controller: no registered provider for "lxd" <2.0-count> [19:33] Bug #1571131 changed: juju add-ssh-key $(cat ~/.ssh/a-key.pub) needs quoting in helptext [20:16] natefinch: FYI, I left a review of your core patch [20:17] natefinch: I don't think we need that testing patch if we change the approach in core [20:17] natefinch: (though it probably stands on its own merits) [20:22] ericsnow: cool, thanks [20:24] natefinch: np === urulama is now known as urulama|____ [21:23] rogpeppe1: have a moment? [21:42] Bug #1573259 opened: Non-admin users unable to share models [21:42] * thumper sighs [21:42] I'm going to have to work out how to get the lxd provider working again... [21:44] abentley: do you know of the git command that relates to the bzr one of merge-preview? [21:46] thumper: No, sorry. Still a noob with git. [21:46] hmm.. [21:46] I want to see the diff of what my merge would be against master [21:46] so not what is on master that isn't on mine [21:46] just what is unmerged [21:46] should be easy right? [21:52] thumper: make sure you apt-get lxd to get the latest and then run "sudo dpkg-reconfigure -p medium lxd" to set up the LXD bridge [21:54] thumper: see https://github.com/juju/docs/pull/998/files [21:54] ericsnow: ta [21:54] thumper: np :) [21:55] * thumper is looking at code and getting a sinking feeling [21:55] thumper: had to do it today so it was fresh in my mind [21:56] * thumper breathes a sigh of relief [21:56] seems I don't need to worry about that... [21:56] * thumper goes back to details [21:57] * thumper chuckles [21:57] "lxc finger" [21:57] I'll give it the finger [21:59] axw_: ping me when you are awake and around? [22:02] thumper: So merge --preview actually does the merge, so it'll show conflicts and stuff. The equivalent git command, if it exists, would also need to do the merge. "diff -r submit:" just performs a diff against well-chosen revisions and is more likely to have a git analogue. [22:04] thumper: I think you can do the equivalent of "diff -r submit:" with "git diff $(git merge-base)" [22:05] hmm... [22:05] ta [22:13] ericsnow: got a minute? [22:13] redir: sure [22:13] HO? [22:38] * perrito666 no longer has a webcam but still waves when finishing a conversation...... [22:38] redir: i was just going to ping you about eric's PR? can i join the hangout too? [22:39] perrito666: I'm awake and around [22:44] ericsnow: redir: are you guys hanging out somewhere? [22:45] wallyworld: moonstone [22:46] BOOM!!! [22:46] that was my head-cannon [22:47] * thumper cries in the corner [22:48] Bug #1573286 opened: juju/mongo: oplog tests fail with mongod 3.2 [22:51] axw_: nevermind, I sorted it out [22:51] tx a lot [22:52] perrito666: okey dokey [22:52] bugger.... [22:52] I really should do this in two branches [22:53] but I won't [22:53] poo [23:02] is there some funky way i can jack up the mgo logging when running juju tests? menn0? [23:04] mwhudson: I don't know off the top of my head but I know approximately where the change would need to be made [23:04] * menn0 looks [23:04] close enough [23:05] mwhudson: the base suite which manages the mongo instances used in tests is in here: github.com/juju/testing/mgo.go [23:05] i have found some time.Sleep(100 * time.Millisecond)s in mgo and i am suspicious [23:06] mwhudson: you'll probably want to mess with the args passed to mongo in MgoInstance.run() [23:06] menn0: ah no i mean mgo's own logging, mgo.SetOutput and stuff [23:06] ahhh [23:06] * menn0 confused mgo and mongo [23:06] i guess i can jam something in there as well [23:06] right [23:07] mwhudson: all the tests that use mongodb (and therefore mgo) are based on that suite [23:07] mwhudson: so that suite is probably still the place to mess with mgo's logging [23:07] * menn0 looks at those sleeps [23:09] Bug #1573294 opened: state tests run 100x slower with mongodb3.2 [23:11] mwhudson: wow there are a bunch of sleeps throughout mgo [23:12] menn0: enterprissssssssssssssssssssssssssse [23:27] hm no doesn't seem to be a sleep in mgo [23:27] menn0: so now i guess i am interested in jacking up mongo's logging :-) [23:27] menn0: and actually seeing it, i guess [23:28] mwhudson: pass --logpath /some/where and --logappend I guess? [23:29] in MgoInstance.run() [23:34] menn0: ah haha MgoInstance.run() waits for a log message though :( [23:34] mwhudson: ah crap... [23:34] mwhudson: then you might have to monkey with the log capture stuff in the suite [23:34] yeah [23:36] oh but it looks like the mongo log output is Tracef-ed [23:36] menn0: so can i set the log level to trace when running the tests somehow? [23:37] mwhudson: which logs? juju's? [23:37] yea [23:39] i guess i can just jam a logger.SetLevel(loggo.TRACE) in