[00:25] Bug #1569097 opened: jujud fails to start with "could not find a suitable binary for "0.0/mmapv1""

[00:30] wallyworld: could you or someone from your team take bug 1569097? [00:30] Bug #1569097: jujud fails to start with "could not find a suitable binary for "0.0/mmapv1""

[00:30] ok [00:30] thanks! [00:31] cherylj: part of the issue is the mongo stuff got erged too soon, so we'll need to look into how to deal with that. i'm still ramping up on the issues [00:31] cherylj: i also added a bug to the board - HA doesn't use bootstrap constraints [00:31] wallyworld_: I guess that was a miscommunication - we thought it was safe to merge because it had a fallback :/ [00:32] cherylj: my PR didn't have a fallback - it expected mongo 3.2 to be in xenial [00:33] the above bug happened on trusty, if it makes a difference [00:33] anyways, all good, we'll fix [00:33] on trusty it was supposed to use mongo 2.4 stull, hmmm, i'll need to check [00:34] i bootstrapped yesterday without issues, but that may have been on xenial, i'll need to check [00:34] i wonder if wily is also broken [00:35] cherylj: good news though - it's in the queue, so progress :-) https://launchpad.net/ubuntu/xenial/+queue [00:35] yay! [00:35] brb [00:44] cherylj: i reckon bug 1534627 should be high rather than medium, since it quite adversely affects stakeholder deployments [00:44] Bug #1534627: Destroyed models still show up in list-models <2.0-count>

[00:46] wallyworld_: +1 and the change on it is backward incompatible [00:46] yep, that too [00:46] Bug #1569106 opened: juju deploy --to lxd:0 does not work

[01:07] wallyworld_: hey, ruthere? [01:08] maybe [01:08] depends who's asking [01:08] I would make a taxes joke, but I have no idea how is the Ausie irs called [01:08] ATO [01:09] australian tax office [01:09] will it kill you? like everything in australia? [01:09] it can [01:09] feeding it money helps [01:10] so lemme know when you can ho [01:11] anytime [01:11] k standup room? [01:11] ok [01:25] Bug #1569109 opened: Juju makes wrong network configuration when adding physical machine [01:36] evening folks [01:43] good evening all, see you in the morning [01:47] bugger... [01:48] * thumper sighs [01:48] shelving all current work to pop the stack and fix other bits. [01:55] cherylj: is there something I should be working on to help unblock master? [01:56] natefinch: want to take a look at https://bugs.launchpad.net/juju-core/+bug/1564791 ? [01:56] Bug #1564791: 2.0-beta3: LXD provider, jujud architecture mismatch

[01:56] looks like an interesting one [01:56] not really ;) But I will :) [01:57] cherylj: actually, it gets less bad toward the end of the bug :) [01:59] Bug #1569120 opened: wrong lxc bridge still used in juju beta4

[02:07] axw: got a sec? [02:07] cherylj: yup? [02:07] axw: I'm looking at bug 1569024 [02:07] Bug #1569024: Region names for rackspace should accept caps and lowercase

[02:07] and was thinking that for public clouds, we could strings.ToLower the region names [02:07] that way we don't mess with any user defined cloud regions [02:08] and maintain compatibility for rax [02:08] cherylj: gah, yeah, we should and I meant to do that [02:08] cherylj: on input, lower case [02:08] but just for public clouds, yes? [02:08] or for all? [02:09] cherylj: hrm. well, maybe not lowercase when we pass through, just compare case insensitive [02:09] * axw looks at the code [02:10] ah, that works too [02:10] strings.EqualFold() [02:10] neato [02:11] cherylj: I *think* it's just a matter of changing "getRegion" in cmd/juju/commands/bootstrap.go [02:11] where we check region.Name == [02:11] cherylj: also the set-default-region command [02:12] axw: yeah, I had some changes in there already, just wanted to verify what we should do [02:12] axw: so don't change the region, just do a case insensitive comparison? [02:12] cherylj: I think that's safest, yeah [02:12] axw: sounds good, thanks [02:24] natefinch: I have access to the arm hardware for that lxd bug. Need me to forward it your way? [02:26] cherylj: yes please [02:27] cherylj: though it probably will be a matter of looking at the code and then thinking real hard. [02:27] break out the hamster [02:27] hey rcj, slumming it with the juju devs? [02:37] cherylj: my brain refuses to read arm64 ... every time it translates it into amd64, and I have to do a double take to make sure it says the right thing [02:38] natefinch: oh me too [03:19] menn0: when you're importing a model, will it be visible during import? will it be mutable while importing? [03:19] (import as in migration) [03:20] axw: there's a migration-mode flag which will be set to "importing" [03:20] axw: that blocks critical txns as well as preventing API logins for it [03:20] axw: the former has been done but not the latter [03:20] menn0: ok, cool. but you'll still be able to see it in list-models? [03:21] axw: I guess so, but we could make it so they didn't show up [03:21] menn0: I'm thinking it might make sense to have a status entry for models [03:21] axw: that could be done [03:21] available, importing, destroying, archived [03:21] something like that [03:21] axw: sounds useful [03:22] menn0: we need to be able to filter out Dead models in list-models, but I think we should show status of Alive vs. Dying [03:22] but a more descriptive status would be better [03:22] I'll look at adding that [03:24] menn0: speaking of migrations... I added a field to charmDoc and tried to figure out if there was anything I needed to do for migration, but couldn't find code migrating charm stuff. What's up with charms and migration? [03:26] natefinch: migration of charms and tools is still in-progress... there is code but it needs reworking and isn't plugged in to the process yet [03:27] natefinch: for most collections there's tests that fail if fields are added [03:27] natefinch: but probably not for charms yet [03:27] natefinch: so just add your field for now and email thumper and me about it just to make sure [03:27] menn0: ok, cool, will do [03:28] axw: you thinking this status would replace the migration-mode field? [03:28] axw: or is the status a virtual concept only for the status API? [03:28] menn0: probably not, it's just for human consumption [03:28] axw: kk [03:29] axw: you know that there's already a environment-status (hopefully model-status) section which can optionally appear in the status output [03:29] axw: perrito666 added it to support reporting that there's a tool upgrade available [03:30] axw: model migration status will appear there too [03:30] menn0: ah ok, I'll check that out - thanks [03:37] cherylj, what did I do? [03:38] cherylj, I'm just here to remind everyone to use the 'daily' stream when running with the xenial series until it ships, otherwise you have a very stale experience. [03:39] heh [03:39] I mean, that's not why I'm here, but I'll make that public service address whenever the opportunity presents itself. [03:39] this has been a CPC public service announcement [03:39] cloud images, best consumed fresh [03:40] also, I'm not in charge of any actual branding efforts [03:45] axw: Can you do a quick review? http://reviews.vapour.ws/r/4528/ [03:45] cherylj: looking [03:46] natefinch: how's that arm bug coming? (I'm curious because it's such a weird bug) [03:46] it's not a nag :) [03:47] cherylj: looks good, but can you please do set-default-region while you're there? [03:47] gah, I forgot you said that [03:47] yes [03:47] cherylj: thanks :) [04:19] axw: can you take another look? http://reviews.vapour.ws/r/4528/ [04:19] I had to do it a bit differently for set-default-region [04:20] so that we wrote out what was in the cloud region list, not what the user specified [04:30] cherylj, thumper: what was the decision on where to land stuff while master is blocked? [04:30] bleh, I haven't done that. [04:30] * thumper waits... [04:30] cherylj: what was the decision? [04:31] it was a back and forth for a while, but the general consensus was "yeah, sure" [04:32] cherylj: sorry was afk, looking [04:33] menn0, thumper, since it's already tomorrow, I can go either way on a bug branch. [04:33] if either one of you wants to create one, go for it [04:33] I'm just waiting to land this rackspace fix so I can go to bed [04:33] cherylj: LGTM, thank you [04:33] sorry for keeping you from bed :( [04:33] it happens :) [04:33] thanks for the review! [04:33] cherylj: was it acceptable to have a release branch? [04:34] thumper: I'd rather not do that at this point because I do'nt know if CI would run on it tonight (until the QA team wakes up) [04:34] ah... good point [04:40] thumper, cherylj: let's make a "next" branch [04:41] ack [04:41] next branch created [04:41] the compression ration acheived by lrzip is amazing but geez it's slow [04:42] * menn0 has been waiting for almost 2 hours for a file to decompress [04:43] menn0: two *hours*? [04:43] menn0: seems unlikely the extra compression saved you two hours of download time... [04:46] mwhudson: I agree but that's how the file came [04:46] it's a 365MB file that currently up to 11GB and climbing [04:46] lrzip is even using every core and it's still taking this long [04:57] wow [05:05] mwhudson, thumper: just finished... a little over 2 hours. 365 MB to 14GB [05:08] what was in that giant file ? [05:09] menn0: that is quite a ratio [05:11] davecheney: DB dump from a broken system [05:14] axw: if you get a chance, here's a small mongo ha fix for beta4 http://reviews.vapour.ws/r/4529/ [05:14] wallyworld_: ok, a little later, trying not to context switch right now [05:15] (unless it's urgent) [05:15] tis fine, whenever suits [05:15] nah, can wait [05:15] so long as it lands sometime today so Ci can run [05:16] i could bug menn0 :-) if he is waiting for lrzip [05:17] m.Server = httptest.NewServer(nil) [05:17] c.Assert(m.Server, gc.NotNil) [05:17] m.oldHandler = m.Server.Config.Handler [05:17] create a new server, then save the value of it's handler ... [05:17] then restore the handler in the tear down [05:17] then the new test overwrites the value we just restored ... [05:17] wat [05:24] wallyworld_: i'm waiting for a long mgopurge run [05:24] wallyworld_: i'll take a look [05:24] ty [05:25] is there a custimer issue? [05:38] wallyworld_: ship it [05:38] menn0: yay, tyvm [05:39] wallyworld_: even though you're deleting a lot of my turd polishing :) [05:39] menn0: sorry :-) [05:39] less turds left now [05:41] wallyworld_: actually hang on [05:41] * wallyworld_ hangs [05:41] wallyworld_: can't you be a bit more aggressive about test removal [05:41] possibly [05:41] i thought about removing the whole fakeensure stuff [05:41] wallyworld_: some of those asserts you've removed were the point of those tests so I suspect the whole test can go [05:41] that's what I was thinking too [05:42] yeah, had the same thought [05:42] if it's not being used [05:42] i was trying to be a bit conservative [05:42] i'll take another look [05:42] wallyworld_: it's really just TestMachineAgentUpgradeMongo [05:42] and perhaps the fakeensuremongo [05:42] yep, i convinced myself that test remained useful [05:43] but seems not [05:50] menn0: yeah, a lot of extra code can just be deleted [05:50] wallyworld_: excellent [05:51] peergroup is having a big haircut [05:51] peergrouper [06:15] wallyworld_: before you delete all that... [06:15] already gone :-) [06:15] is it still possible to promote machines to controllers with your changes? [06:16] axw: you mean ones which are not yet has-vote [06:16] wallyworld_: I mean "enable-ha --to 0,1,2" [06:16] where we transform a non-state-server into a state-server [06:16] i'll double check, i didn't test that explicitly [07:08] whew it finally worked ! [07:31] Bug #1569196 opened: enable-ha with placement fails due to invalid JobManageNetworking [08:04] morning everyone [08:05] back to the routine of the school-run this morning [08:05] *sigh* [08:13] morning voidspace [08:23] dimitern: so thumper broke my code *again* overnight :-) [08:23] voidspace: oh yeah? :) [08:24] dimitern: see here: https://docs.google.com/document/d/1YmbdGpP7Oy5uglOwqbRXf1k_7siaxfEpoWkshk5_oPo/edit?ts=56fb30ca# [08:24] dimitern: basically you were right about not_networks so he changed the allocate machine args again [08:24] dimitern: and I was just updating the code to work with master as it was yesterday :-) [08:24] it's not a big change - so not difficult [08:25] voidspace: cool :) [08:36] voidspace: it is my mission in life to make your mornings miserable [08:37] however, dimitern will like to hear that he was right [08:37] thumper: ah, that explains why you joined our standups! [08:37] thumper: :-) [08:37] thumper: hey, so gomaasapi now has its own dependencies.tsv [08:38] voidspace: if you want to jump in the hangout now, we can chat that way I can not work later [08:38] voidspace: yeah, needed for the merge bot [08:38] thumper: sure [08:38] thumper: right, but the versions of its dependencies are different than the juju ones [08:38] thumper: I'll join the hangout [08:38] babbageclunk: you too? [08:38] voidspace: shouldn't be off by much [08:38] babbageclunk: early hangout [08:38] voidspace: probably just testing [08:38] all of them are now different I think [08:38] voidspace: sure [08:39] thumper: but everything still works [08:39] we just need to be careful [08:39] * thumper nods [08:49] thumper: I told you ;) [09:10] morning [09:10] TheMue: \o [09:15] dimitern: heavy on fire for Juju 2 and also 16.04? [09:17] TheMue: oh yeah :) [09:18] Hi, I wonder if somebody could please point to the place for a quick question on BigData charms? [09:18] dimitern: how is J2 different from the J1.*? so many incompatible changes to change the major release number? [09:20] TheMue: a lot has changed, and some things in an incompatible way, check the release notes :) [09:23] dimitern: will do. still very interested in juju and always trying to place it in projects or give interested people a hint. many don't know it. [09:35] fwereade_: the branch I put up is for 2.0, in which compatibility breaks are many and varied [09:44] Anyone know why building the next branch is failing? [09:59] axw, oops, fair enough, I do default to unthinkingly-maintain-compat [09:59] frobware: managed to figure it out - erc-email-userid needs to match my nick for i.canonical.c to accept it along with the server password [09:59] fwereade_: and I thank you for it :) [09:59] dimitern: no turning back now :) [10:00] frobware: indeed :) [10:00] babbageclunk: guessing... did you run godeps -u ... [10:01] frobware: not locally - in the github-merge-juju Jenkins job. [10:01] frobware: http://juju-ci.vapour.ws:8080/job/github-merge-juju/7313/console [10:02] frobware: looks like lots of provider/lxd tests [10:02] babbageclunk: fwiw I see the same errors even after upgrading to xenial when running make check on master tip [10:02] fwereade_: responded to your other questions on RB, will look again tomorrow. thanks for the review [10:03] if anything it got worse - I only saw a couple of failures yesterday on wily [10:03] frobware: some of the failing builds under that are against master, some against next. [10:04] frobware, dimitern: I tried running provider/lxd tests for next locally and I don't see the failures (although I didn't run the full test suite). [10:05] dimitern: I'll try running make check [10:05] babbageclunk: I'll try next now so see if it's any better [10:06] but first I need to reboot.. [10:18] hi all [10:23] menn0: hi! [10:23] babbageclunk: how's it? [10:23] Hey, I saw a build of yours failed with lots of lxd provider failures. [10:24] Did you work out why? A branch of mine had that just now too. [10:24] babbageclunk: everyone's seem to be failing like that. I wonder if there's a problem with the test runner hosts. [10:24] any QA people about? [10:24] frobware: launchpad seems to have gone read only, so I can't put this in the bridged bond bug right now. The pre-up/post-down thing is a red herring. Even if you include them cloudinit hangs. Rebooting always works and cloudinit seems to finish happily. [10:25] frobware: and I really need to get the proxy bug fix landed, so pausing on this for now. [10:25] dooferlad: ack [10:26] frobware: ah, bug just updated. Yay web services. [10:26] dooferlad: really need to conclude on an investigation of replace ENI and reboot... [10:28] menn0: Running the full test suite locally (on juju/next) I get the same failures [10:30] babbageclunk: interesting... so not the build hosts then [10:31] babbageclunk: I'm just finishing something else up and then I'll try on my machine. [10:31] menn0: takes ages though so I haven't run the tests against master as well yet - I saw that cherylj has some failing runs against master with the same errors. [10:34] dimitern: whoa! that's subtle... [10:35] dimitern: we currently have 00-juju.cfg and eth0.cfg [10:35] dimitern: which would/could/should give us 2 addresses on eth0 [10:36] dimitern: but because we specify a mac addr, the ifup via DHCP on eth0.cfg gives us the same IP addr [10:37] dimitern: ok, that explains it (for me at least) :) [10:37] babbageclunk: if you run just one of the tests that's failing in CI does it fail then? (that shouldn't take too long) [10:38] menn0: Yeah, it turns out running just ./provider/lxd that fails. [10:38] frobware: interesting [10:39] frobware: and lucky I guess :) [10:39] menn0: But now I can't find a version where it doesn't fail. [10:39] * menn0 runs those tests [10:39] babbageclunk: they pass for me [10:39] dimitern: I was trying to understand the behaviour. If I try this outside of juju the ifup on another foo.cfg (which also specifies eth0) will just add another IP addr to eth0. [10:39] menn0: is it safe to just do a checkout, godeps, then go test? [10:40] babbageclunk: yep that should do it (as long as you have mongodb installed) [10:40] babbageclunk: and I guess you probably need to have lxd installed for some tests too [10:40] menn0, babbageclunk: isn't the underlying problem related to the configuration of lxdbr0? [10:40] menn0: ok, so it'll rebuild everything. [10:40] or lack of [10:41] frobware: sure... but why is it suddenly happening in CI and on babbageclunk's machine? [10:41] menn0, frobware: ok - I installed lxd last friday. [10:41] babbageclunk: what does "lxc version" show? [10:41] 0.20 [10:41] menn0: I'm on wily [10:42] babbageclunk: I'm on vivid but I'm running 2.0.0.rc1 [10:42] so I see exactly the same test failures on next as on master [10:42] Maybe I should upgrade to that. [10:42] dimitern: yes, all recent merge attempts have had the same lxd/lxcbr0 problems [10:43] menn0: potentially that's also the problem on the build machine(s) [10:43] babbageclunk: there's a PPA for the current lxd from the lxd/lxc team [10:43] menn0, babbageclunk: to repro this just 'cd provider/lxd; go test'? [10:44] frobware: I believe so [10:44] frobware: yup - might need a godeps in there too [10:44] * menn0 prefers or "go test ./provider/lxd" but whatever [10:45] frobware: same thing with running only provider/lxd tests [10:45] menn0, babbageclunk: ok && not terribly helpful but OK: 77 passed, 1 skipped [10:45] menn0, babbageclunk: however, I am _only_ at dd9828ec7003d1a6ec1fc4dbcb7e6d17467a21f0 [10:46] menn0, frobware - ok, I'm going to add the ppa and upgrade. [10:46] babbageclunk, menn0: or go backto dd9828ec and try there... it may be something more recent in master [10:46] menn0, frobware: then I guess if that fixes it then it's an indication that someone should do the same on build hosts. [10:47] frobware: I suspect you did run `sudo dpkg-reconfigure -p medium lxd` as suggested by the tests? [10:47] otherwise how are you not seeing the failures.. [10:47] dimitern: nope, not medium. but I did reconfigure some time last week [10:47] dimitern: I can't repro the problem, and I haven't run dpkg-reconfigure in a long time [10:47] frobware: I tried going back to find a bisect start point, but got back to last Monday and the tests were still failing. [10:48] menn0: it was probably last tue/wed when I did the dpkg-reconfigure [10:48] I haven't since I installed lxd (about 2 months ago?) [10:48] * dimitern *facepalm* [10:48] babbageclunk: my /etc/default/lxd-bridge config: http://pastebin.ubuntu.com/15784102/ [10:49] anyone have the ppa handy? [10:49] I remember what I did - changed /e/default/lxd-bridge to not have IPv4 addresses as it was messing up my lxd multi-nic testing [10:50] * frobware would like to kickstart/jumpstart all his machines every morning to avoid state... [10:50] dimitern: but why is this also happening on the build hosts? [10:51] menn0: which is why I was suggesting first go back to my current rev ^^ to see if it's just recent churn in master. [10:53] frobware: I'm on that rev - it's upstream/next and upstream/master (since no one's been able to land anything) [10:53] oooohhh. I am at that rev. apologies... [10:54] babbageclunk: my lxd package is: [10:54] menn0: not sure - perhaps when /e/d/lxd-bridge was introduced it did not have IPv4 config and CI machines haven't been updated since? [10:54] $ apt-cache madison lxd [10:54] lxd | 2.0.0-0ubuntu2 | http://gb.archive.ubuntu.com/ubuntu xenial/main amd64 Packages [10:54] now all provider/lxd tests pass [10:54] dimitern: to confirm, you're running xenial? [10:54] frobware:yep [10:56] however, p/lxd tests should NOT fail if anything like that on the machine happens - they should be properly isolated [10:58] Huh. lxc version still says 0.20, but the tests pass for me now. [10:58] babbageclunk: you're on wily? [10:59] dimitern: yup [11:00] babbageclunk: so I needed to add `deb http://ppa.launchpad.net/ubuntu-lxc/lxd-stable/ubuntu wily main` to /e/a/srcs.list to get lxd to work on wily [11:01] dimitern: yeah - I did the same, but via add-apt-repository for ppa:ubuntu-lxc/lxd-stable [11:01] babbageclunk: yeah - same thing, and then a-g update && a-g dist-upgrade [11:02] dist-upgrade if you already installed lxd I mean [11:03] ok, so how do we get the tests passing on the build machines? [11:10] ooh, down to two failures [11:13] mgz: ^^ [11:14] mgz: istm the dpkg configure for lxd might have been skipped with the noninteractive frontend [11:14] mgz: it will be useful to keep the machine around when the merge job fails to see what's going on [11:15] dimitern: I'm running apt-get under ansible, so maybe it's being run noninteractively as well? [11:16] babbageclunk: well, what's in /etc/default/lxd-bridge ? [11:17] dimitern: http://pastebin.ubuntu.com/15784982/ [11:17] dimitern: But the tests are passing now that I've upgraded. [11:18] dimitern: so maybe it was just that I added the ppa. === babbageclunk is now known as babbageclunch [11:18] babbageclunk: yeah, it looks like the tests should still fail (?!) mine was very similar before I fixed it [11:21] babbageclunch: looks like most of the config is empty === babbageclunch is now known as babbageclunk [11:35] frobware: dimitern: babbageclunk http://reviews.vapour.ws/r/4535/ [11:45] frobware: dimitern: babbageclunk http://reviews.vapour.ws/r/4535/ [11:45] * voidspace lurches to lunch [11:48] voidspace: looking [11:51] dimitern: thanks [11:54] frobware, dimitern, voidspace: http://reviews.vapour.ws/r/4536/ [11:54] voidspace: looking at yours now. [11:54] babbageclunk: dimitern just reviewed it, but thanks [11:54] voidspace: yeah, I just saw that it's merging [11:55] it will fail [11:55] dammit [11:55] I missed off some test fixes - didn't push them [11:55] voidspace: probably should have a look anyway - I'm OCR tomorrow. [11:55] ooh [11:55] voidspace: Well, it was going to fail due to the lxd thing anyway, right? ;) [11:55] hah [11:56]