[00:19] menn0: ping [00:20] tanzanite was planning to do the renaming of all env to model once controller-rename branch gets into master [00:20] anastasiamac: pong [00:20] anastasiamac: how were you going to handle things like DB collections? [00:21] anastasiamac: and what about API names? [00:21] menn0: we have not gotten to the point of actually discussing it yet as controller-rename branch is still pending in limbo somewhere... [00:21] anastasiamac: ok np [00:21] menn0: i suspect that we will have a brief next week where we'll discuss the approach [00:22] menn0: i'd love to get anyone who is interested involved [00:22] (u sound like one) [00:22] involved in discussion that is :D [00:23] anastasiamac: would you mind replying to my email then... if there's going to be a major rename effort so that "environment" goes away completely then I'm fine with using "model" everywhere [00:23] anastasiamac: I think the rename is going to be hard to get right [00:24] menn0: k. and I agree.... hence i think we should brain-storm before we dive into actual rename \o/ [00:24] anastasiamac: but if anyone can do it it's your squad :) [00:24] wot?! [00:24] anastasiamac: I'm happy to participate in the discussion and planning [00:24] i think with onyx's experience and the lessons learnt from controller-rename, ur squad is very well positioned to do it ;D [00:24] awesome! [00:25] anastasiamac: onyx has been banned from working on anything but model migrations [00:25] anastasiamac: but since this affects model migrations I think we can at least be involved in the discussion [00:26] menn0: k. xross-squad pairing sounds awesome :D [00:26] sure :) [00:26] probably not going to happen though [00:26] :( i can dream [00:26] i have a dream?... [00:28] :) [00:53] cherylj: chasing this https://bugs.launchpad.net/juju-core/+bug/1534394 [00:53] Bug #1534394: trusty/i386 test failure [01:00] davecheney: I think that waigani already fixed that bug. [01:01] bug 1532777 [01:01] Bug #1532777: undertakerSuite.TestEnvironConfig fails [01:01] davecheney: the current things we're blocked on may be CI test issues. So at this point, I don't have specific bugs for you. [01:01] we will need some help with wily / xenial unit tests on 1.25 as we're extending its support for a while [01:02] but I've been focused on 2.0-alpha1 right now and don't have a list of those bugs yet [01:03] davecheney: so I don't need you right now. [01:03] was that rambling enough? [01:09] Bug #1534394 opened: trusty/i386 test failure [01:18] cherylj: noted [01:18] i'll move on to something else [01:51] katco: crickets [05:07] Bug #1534394 changed: trusty/i386 test failure [07:52] Bug #1534480 opened: LXD: bootstrap error: no registered provider [09:19] Bug #1534511 opened: Support for multiple apt sources and apt keys in isolated environments [09:46] dimitern, voidspace, dooferlad: will be 10-15 mins late for standup. [09:46] frobware: I am happy to start late [09:46] frobware: ping us when you are ready? [09:54] sure [10:13] dimitern, voidspace, dooferlad: ready [10:14] * dooferlad joins hangout [10:18] omw [10:19] dimitern: frobware: dooferlad: still need a review by the way: http://reviews.vapour.ws/r/3534/ [10:31] Bug #1534480 changed: LXD: bootstrap error: no registered provider [11:07] frobware: dimitern: dooferlad: so in the merge provider/maas/environ.go is fairly screwed [11:08] this is becauase a bunch of code was broken out into separate files on maas-spaces [11:08] but fixes applied to master (for default gateway stuff) [11:17] bless on controller-rename btw [11:51] frobware: dimitern: dooferlad: coffee [12:50] dimitern: frobware: dooferlad: still need a review by the way: http://reviews.vapour.ws/r/3534/ [12:50] voidspace, ack [12:50] dimitern: frobware: dooferlad: my merge branch now compiles, except for the tests [12:50] which are still pretty screwed [12:51] voidspace, nice [12:51] as soon as they compile I can see what's broken [12:51] I'm worried that fixes that landed on master in code that was extracted into separate files on maas-spaces might get lost [12:51] that's the biggest danger [12:51] *hopefully* the tests will pick that up [12:51] but tests got moved too [12:51] so it'll need a good review to make sure we haven't lost fixes in the merge [13:31] can I get a review? http://reviews.vapour.ws/r/3546/ [13:31] It's a quick one [13:38] Bug #1521777 changed: Allow for upgrades to 2.0 [13:55] voidspace, you've got a review btw [14:17] lazyPower: ping [14:17] dimitern: cool - thanks [14:18] dimitern: ah, I didn't know about the "if !c.Check(...) {continue}" trick [14:19] dimitern: that's why I wasn't using check in that loop [14:19] thanks [14:20] voidspace, :) yeah - it's useful [14:30] dimitern, voidspace, dooferlad: let's use https://plus.google.com/hangouts/_/canonical.com/juju-sapphire [14:30] frobware: really? [14:30] I'm already in the one you linked to [14:30] ah well. [14:31] omw [14:31] voidspace, ah, sorry. I tried to change it to that but the juju-sapphire URL but it obviously didn't work. [14:32] frobware, omw, 2m [14:40] perrito666 pong [14:45] cherylj: lgtm [14:45] thanks, mgz [14:54] chery, sorry I missed your emssage last night. Yes, that's what I'm doing now. Trying to repro with the steps in the bug. I reviewed fwereade's patch, which looks pretty sane... but I haven't been able to reproduce the bug yet, so it's hard for me to tell if the fix will fix it. [14:54] cherylj: ^ [14:55] natefinch: thanks for the update! [14:57] oh, I thought we were all calling her 'chery' now... [15:23] Bug # opened: 1534610, 1534619, 1534620, 1534623 [15:29] Bug #1534626 opened: provisionerSuite.SetUpTest fails because connection reset [15:29] Bug #1534627 opened: Destroyed models still show up in list-models [15:29] lazyPower: sorry I solved itt myself, thank you anyway [15:30] ah bueno perrito666 :) [15:31] that is pretty much like throwing a wrench into my brain gears [15:31] I was considering for a moment what channel was this [15:32] Bug #1534626 changed: provisionerSuite.SetUpTest fails because connection reset [15:32] Bug #1534627 changed: Destroyed models still show up in list-models [15:38] Bug #1534626 opened: provisionerSuite.SetUpTest fails because connection reset [15:38] Bug #1534627 opened: Destroyed models still show up in list-models [15:41] Bug #1534626 changed: provisionerSuite.SetUpTest fails because connection reset [15:41] Bug #1534627 changed: Destroyed models still show up in list-models [15:44] Bug #1534626 opened: provisionerSuite.SetUpTest fails because connection reset [15:44] Bug #1534627 opened: Destroyed models still show up in list-models [15:47] Bug #1534626 changed: provisionerSuite.SetUpTest fails because connection reset [15:47] Bug #1534627 changed: Destroyed models still show up in list-models [15:50] Bug #1534626 opened: provisionerSuite.SetUpTest fails because connection reset [15:50] Bug #1534627 opened: Destroyed models still show up in list-models [15:53] Bug # opened: 1534632, 1534636, 1534637, 1534643 [15:56] Bug # changed: 1534632, 1534636, 1534637, 1534643 [15:59] Bug # opened: 1534632, 1534636, 1534637, 1534643 [16:21] can I get another easy review? http://reviews.vapour.ws/r/3549/ [16:30] cherylj: ship it [16:30] thanks, natefinch! [17:05] katco, ericsnow: someone needs to make a "let me man pages that for you" [17:07] fwereade: I don't suppose you have a test charm that exercises leadership? [17:50] natefinch: i have one [17:57] Bug #1534757 opened: Attempting to run charm before unit provisioned, 1.26 [18:08] lazyPower: I actually figured out that I don't need a special charm... every charm gets a leader, most charms just don't care :) [18:08] lazyPower: so I can just use a dummy charm and juju run is-leader on it, and that's sufficient for my purposes :) [18:21] mgz, cherylj - still re-confirming, but two bootstrap attempts with 1.25.2 + 1.9 give me a bootstrap timeout. the machine comes online, maas state is deployed. it's reachable for a bit, then no-talky from then on. http://pastebin.ubuntu.com/14506872/ [18:31] beisner: thanks [18:32] beisner: what does maas say about it? [18:32] mgz from maas perspective, it is "deployed" [18:33] i've backed out to 1.25.0, same procedure, same maas, bootstraps ok [18:33] i suspect juju bridge foo is the moment we lose network [18:34] unfortunately, i can't inspect the aftermath b/c i can't reach it. i've got SOL / ipmi, and can see that it is up at the login screen. but of course no user/pwd login enabled. [18:34] can you ssh in parallel when it's installing and stuff and hold that connection? [18:35] I'd be interested to see logs/networking config changes on the box [18:35] me too. i shall keep trying. do you have 1.9 maas that you can try a bootstrap against? [18:37] beiser: see http://reports.vapour.ws/releases/3511/job/maas-1_9-OS-deployer/attempt/181 [18:37] log in with sso and go to that page [18:38] that's our result for 1.25.2 on maas 1.9 deploying your trusty-liberty bundle [18:40] mgz, that's using 1.25.3 [18:40] i'm using 1.25.2 from ppa:juju/proposed, plus agent-stream: proposed [18:41] ah, whoops, wrong rev, go back to /releases/3506 and attempt/163 [18:41] which is the same code, though not literally the same binaries [18:42] our maas is certainly not your maas though [18:42] beisner: I'm presuming you don't have any of the feature flags on as well? [18:42] mgz right, just as a vanilla new user [18:43] ok, i'll fire off a fresh bootstrap with 1.25.2 and see if i can sneak onto that unit before it goes lalalala [18:44] as soon as it starts the apt installing stuff you should be good, note you'll want the same ssh key/juju env key available [18:56] ha. i got this seconds before the first reboot. http://pastebin.ubuntu.com/14507351/ /me watches closely... [19:01] ok mgz. i had a total of ~10s of reachability, then unreachable. i did establish ssh in that window, did another `ip a` and got as far as the b in `brctl show` when it cut me off: [19:01] http://pastebin.ubuntu.com/14507430/ [19:01] so before maas' first reboot, sane. [19:01] after maas' first reboot, sane. [19:02] after juju init stuff, no connectivity. [19:02] oh great someone did the same I was doing [19:03] so, session did get cut off, that pretty much does mean this is a interface reconfiguration [19:05] natefinch: planning [19:06] katco: oops, coming [19:07] beisner: I think we need a bug to track. probably want maas network output included. [19:08] mgz, yep i'm raising one now. [19:11] beisner, can I give you a binary that would allow console login [19:11] frobware, yes please & thx [19:12] beisner, you would obviously have to bootstrap with --upload-tools [19:12] fyi: bug 1534795 [19:12] Bug #1534795: unit loses network connectivity during bootstrap: juju 1.25.2 + maas 1.9 [19:12] frobware, ack [19:13] beisner, ok give me 10-15mins - will verify locally [19:13] thx frobware [19:15] mgz, frobware - good news: in our openstack-provider automation, things are looking good so far :) [19:15] beisner, so any clue as to why the other setup is borked? different MAAS setup? [19:15] frobware, two different providers. [19:15] frobware: using the maas provider rather than openstack provider [19:16] beisner, on the MAAS side what's the network config look like? VLANs? Bonds? straight-forward eth0? [19:17] beisner, and version of MAAS? [19:18] maas: 1.9.0+bzr4533-0ubuntu1~trusty1 [19:18] juju: 1.25.2-0ubuntu1~14.04.1~juju1 [19:18] beisner, ack [19:19] frobware, eth0, untagged [19:19] beisner, so as vanilla as it gets.... sigh. [19:19] beisner, amd64? [19:19] yep amd64 [19:19] beisner, just checking. ;-) [19:20] frobware, yep, np [19:21] beisner, what's the easiest way to get binaries to you? [19:22] i've used our private chinstrap in the past for that. i've not used the new private-fileshare bits yet, though we can try that. [19:22] frobware ^ [19:26] beisner, any other interfaces or just eth0? [19:27] Bug #1534795 opened: unit loses network connectivity during bootstrap: juju 1.25.2 + maas 1.9 [19:27] frobware, machine has 4 nics, only eth0 and eth1 are cabled up. we've only ever used eth0 on these particular machines. [19:29] frobware, mgz: oo interesting ... note that on my 2nd paste in the bug, eth1 is up for some reason. [19:29] that is unexpected [19:35] beisner, http://178.62.20.154/~aim/juju-bin.tar.bz2 [19:36] beisner, that's 1.25.2 but before juju has a little play with /e/n/i is does `passwd -d ubuntu' - so you should be able to login on the console. [19:37] mgz, ^^ [19:37] frobware: thanks for helping out [19:38] thx, grabbing the bin now frobware [19:43] mgz, beisner: I'm very tempted to make that a capability behind a flag... as we run into it too often and it's difficult to diagnose once it goes wrong. [19:43] mgz, beisner: the issue of course is security [19:43] that would be a good t-shooting tool indeed. but yep, that. [19:47] beisner, I just tried with two NICs configured and it bridged eth0 OK [19:48] frobware, so i have two nics CONNECTED, but not two nics configured. [19:48] frobware, ie. http://pastebin.ubuntu.com/14508207/ [19:48] beisner, so when I say configure, in MAAS 1.9 parlance, they're configured and both have static ip assignment when looking in the MAAS UI [19:48] yet eth1 winds up UP [19:52] frobware, https://launchpadlibrarian.net/234276622/network.png [19:52] beisner, let's see what happened to e/n/i once you bootstrap and can login [19:53] beisner, but I see no smoking gun there [19:53] frobware, \o/ i'm in via ipmi [19:53] sol that is [19:53] with that bin [19:55] beisner, the binary I sent you didn't work? [19:55] frobware, yes. it let me log in with ubuntu. now poking around in it. [19:59] frobware, eth0 is down; eth1 is up; no bridges. no ipv4 addresses. [19:59] grabbing /etc, /var/log, anything else? [19:59] beisner, can you send me /etc/network/interfaces* [20:00] Bug #1534804 opened: debug-log is unstable with LXD local provider [20:00] beisner, and capture ip route [20:01] beisner, is is possible to HO? [20:06] frobware, https://plus.google.com/hangouts/_/canonical.com/andrew-mcdermot?authuser=1 [20:07] wow `ip route` blank [20:07] that's a new one [20:07] beisner, in there [20:19] beisner, mgz: http://pastebin.ubuntu.com/14509474/ [20:35] frobware, added tarball to bug. woot. thx again for your help. & mgz [20:39] beisner, mgz: I know exactly what the bug is. [20:40] beisner, mgz: fixed in maas-spaces, doesn't help... however, we ran into the same issue a week ago on maas-spaces - and we had not seen it there for > 1 month. Something changed upstream so that we get dns-nameserver entries partwya through /e/n/i and we don't parse that correctly. [20:41] :| ubuntu does not define xdg_config_home ? [20:42] frobware, excellent. and yep, i'll do a maas non-juju deploy of that machine and collect the same in a bit. [20:43] beisner, pretty sure you won't need to do that. uploading a new binary for you to try. [20:45] beisner, I'll leave the `passwd -d ubuntu' in just to help out. [20:46] beisner, I still don't understand why my MAAS setup doesn't give me the same dns-nameserver entries other people get. [20:49] frobware: I'm really happy you're still up to look at this :) I saw the bug report while at the doctor's office [20:52] beisner, http://178.62.20.154/~aim/juju-bin.tar.bz2 [20:52] beisner, md5sum a2d16e6f3993cc56e7c998aadfe48b74 [20:54] frobware, thx. will grab and run it shortly [21:36] cherylj, https://bugs.launchpad.net/juju-core/+bug/1534795/comments/7 [21:36] Bug #1534795: unit loses network connectivity during bootstrap: juju 1.25.2 + maas 1.9 [21:36] cherylj, https://bugs.launchpad.net/juju-core/+bug/1534795/comments/8 [21:37] beisner, see my updates to the bug ^^ [21:37] frobware: thanks so much! [21:38] frobware, cherylj - cycling now. it's progressed to pkg installation and maintained connectivity thus far \o/ [21:38] beisner, great. I would really appreciate if you could try it again with http://178.62.20.154/~aim/juju-bin-lp1534795.tar.bz2 as this is a build from the proposed PR. [21:39] beisner, I'll hang around for 10-15 mins to hear your "BOOTSTRAPPED!" message. :) [21:42] cherylj, beisner: the only thing I would point out about my binary build is it is based on the tip of 1.25... not the 1.25.2 tag. Caveat Emptor! [21:45] ericsnow: natefinch: what was your alternative proposal for "juju charm --resources"? [21:48] katco, ericsnow: anything else [21:48] katco, ericsnow: uh... maybe just charm-resources? [21:49] natefinch: I actually think it fits well, even if it requires a new juju sub-command [21:49] ericsnow: I can't abide adding a new subcommand that doesn't do anything, just so it can have a flag that does something. That's terrible. [21:49] katco, natefinch: yeah, ideally charm-resources [21:52] ericsnow: natefinch: ok, i'll float the suggestion. it's entirely possible that we intend to expand on the charm command [21:53] katco: I'd be fine with it if juju charm was a command that was goign to do other things.... though I'm generally against using flags to do completely different things... that's what subcommands are for [21:54] katco: so like, if juju charm cs:foo gave you generic info about the charm, and juju charm --resources gave you info about the charm's resources, that's cool [21:54] please dont add an empty command, that is counter intuitive [21:54] perrito666: we're fighting it [21:55] charm-resources sounds a bit less painful [21:56] my kingdom for an unblocked master [21:56] ditto [21:57] frobware, 0 started 1.25.2.1 fat-machine.dellstack [21:57] perrito666: natefinch: it's not going to unblock until cherylj decides we're good for stabilizing 2.0-alpha1: http://pad.lv/1533750 [21:58] looks like just this left: https://bugs.launchpad.net/juju-core/+bug/1534201 [21:58] Bug #1534201: TestHelpAddImage fails [21:59] yeah, I need to add back in the feature flag without breaking windows [21:59] cherylj: it's staurday here but.. i can look at it at my monday [21:59] cherylj: did u get a chance to progress this bug? [22:00] beisner, that's booted OK then? [22:00] later all, dinner time [22:00] anastasiamac: I haven't yet. going to this evening [22:00] anastasiamac: did you have anything to hand off? [22:00] Bug #1532777 changed: undertakerSuite.TestEnvironConfig fails [22:00] cherylj: k.. keep me posted [22:00] cherylj: jsut an idea on how to fix it from tim... [22:01] cherylj: the fundamental problem is tests should not touch external singleton resources, such as the windows registry [22:01] yeah... [22:01] cherylj: this is why all the tests use the environment for feature flags [22:02] cherylj: i suspect that the base test suite for these commands needs to be re-written [22:03] cherylj: but i was going to see if i can re-run juju tests on my wndows [22:03] cherylj: and had trouble getting it going [22:03] Bug #1532777 opened: undertakerSuite.TestEnvironConfig fails [22:03] anastasiamac: yeah, thanks to sinzui, I have a windows machine I can use for unit tests [22:04] cherylj: \o/ [22:04] oh, blessed master [22:05] cherylj: neeed to go to have a weekend :/ will check back in on monday and pick it up if needed [22:05] cherylj: master is reightfully bless. I parused CI because I am looking to move some jobs to other public clouds since Joyent hasn't fixed the firewall yet [22:05] frobware, yessir [22:05] thanks anastasiamac! have a good weekend! [22:05] cherylj: thanks \o/ you as well! [22:06] beisner, excellent. [22:06] * frobware is gone gone gone! [22:07] thanks, frobware! [22:09] Bug #1532777 changed: undertakerSuite.TestEnvironConfig fails [22:09] oh great, juju home is not defined by a single source of truth [22:20] cherylj, frobware, mgz - ok i'm eod + eow. off monday, US holiday. i can commit to cycling the 3rd bin (http://178.62.20.154/~aim/juju-bin-lp1534795.tar.bz2) at some point this weekend and chiming in on the bug with how that went. [22:21] thanks, beisner! [22:23] ok, this got into my nerves, EOW [22:23] yw, cherylj o/ have a nice weekend, all === ericsnow is now known as ericsnow-afk