[00:01] I am saying yay a lot on meetings its a sign of either madness or that I am becoming a teenager schoolgirl [00:02] redir: chat noe if you want [00:02] k [00:02] standup ho [00:02] same bat place? [00:02] k [00:07] * thumper wades through more migration tests [00:16] wallyworld, axw: With mongo3.2 I can't use the mongo shell to connect to Juju's DB any more [00:16] wallyworld, axw: I suspect it's that the SCRAM-SHA-1 auth mechanism is used by default now [00:17] but the 2.6 client doesn't support it [00:17] do you guys know a way around it [00:17] menn0: right, i have organised for mongo client to be packaged with our juju mongo db [00:17] this is not done yet :-( [00:17] wallyworld: ok, so at least it's a known issue [00:17] balloons: ^^^^^ how far away is our mongo packaging fix? [00:18] wallyworld: will it be in the juju-mongo3.2-tools package? [00:18] yes [00:18] so it is available on any controller [00:18] menn0: install mongodb-org ppa and then use mongodb-org-clients [00:19] perrito666: awesome - thanks [00:19] menn0: I might have misplaced some of those dashes [00:19] perrito666: i'll figure it out? [00:19] but the names of the packages/ppas are those === natefinch-afk is now known as natefinch [00:36] thumper: this time I think the race build did complete toally [00:36] and I think I can fix the build hang in the termination worker [00:40] davechen1y: while fixinf races, did u come around any in uniter and hooks? [00:40] davechen1y: i have bugs against 1.25 and m wondering if i need to start from scratch or backport.. [00:40] anastasiamac: hard to say, do you ahve a bug reference [00:40] at this point, i'd think backporting from 2.0 is near impossible [00:41] the codebases have diverged so much [00:42] davechen1y: the one i was going to start with is bug 1486712 [00:42] Bug #1486712: Race on uniter-hook-execution, prevents to resolve unit.

[00:44] anastasiamac: this is a logical race, not a data race [00:44] i guess [00:44] (as there is no output from the race detector0 [00:45] in fact, i've no idea why the OP said race in the title [00:46] Bug #1585015 changed: worker/terminationworker: test timeout during CI

[00:46] Bug #1585846 changed: worker/terminationworker: test timeout after 20 minutes

[00:52] Bug #1588572 opened: cmd supercommand.go:448 cannot read tools metadata in tools directory: open /var/lib/juju/tools/2.0-beta6-xenial-amd64/downloaded-tools.txt: no such file or directory [00:52] anastasiamac: this doesn't look like a data race, just a straight out bug from not handling an error path and the uniter getting out of sync with what the state thinks it is [00:53] ie, dying when the action will not be retried [00:53] wallyworld: is it safe to land my instancecfg change yet, or is the beta8 cutover still happening? [00:53] or maybe it's retrying in a loop [00:53] but not making any progress [00:55] davechen1y: k [01:02] axw: safe to land, they are taking a previous revision [01:02] goodo [01:02] axw: haven't looked yet, kust finished talking to people, but here's the acl wip https://github.com/juju/juju/pull/5526/files [01:03] wallyworld: yeah I just added a comment [01:22] menn0: thumper https://github.com/juju/juju/pull/5528 [01:22] fix for terminationworker hang [01:22] * thumper looks [01:29] davechen1y: you already have 2 ship its and thumper hasn't even finished yet :) [01:30] ta [01:30] landing it [01:34] Bug #1585424 changed: all: data races in tests are surpressed

[01:34] Bug #1588574 opened: Session already closed in state/presence

[01:34] Bug #1588575 opened: allwatcher_internal_test has intermittent failure

[01:40] wallyworld: would you be open to the charm upload HTTP endpoint taking an optional "schema" argument [01:40] wallyworld: for migrations I need to preserve the original schema [01:40] menn0: sure, i think [01:40] when charmstore charms are migrated they end up with the wrong charm URL [01:41] (local: instead of cs:) [01:42] wallyworld: the only concern I can think of is some sort of nefarious behaviour were someone installs a dodgy charm as if it were a charmstore charm [01:42] s/installed/uploads [01:42] gah [01:43] s/installs/uploads/ [01:43] menn0: can't think at the moment, otp with tim [01:43] wallyworld: ok np [01:53] sinzui: what happened to the DNS records for vapour.ws? [01:55] hey prople who think we're using mongodb 3.2 [01:55] we're not using mongodb 3.2 [01:55] or even mongodb 2.6 [01:55] we're using Get:20 http://archive.ubuntu.com/ubuntu/ trusty/universe juju-mongodb amd64 2.4.9-0ubuntu3 [6,936 kB] [01:55] does this come as a surprise to anyone ? [01:55] davechen1y: I think we only use 3.2 on xenial? not sure tho, need to double check [01:57] anyone got a cached IP for juju-ci.vapour.ws? [01:58] axw: this is the landing bot [01:58] davechen1y: ah right, in the tests. [01:59] % host juju-ci.vapour.ws [01:59] juju-ci.vapour.ws has address 54.86.116.234 [01:59] davechen1y: thanks [01:59] .ws does appear to be busted at the moment [02:00] wallyworld, on it [02:00] menn0: a very dull PR https://github.com/juju/juju/pull/5529 [02:00] ty [02:01] thumper: looking [02:03] * thumper heads out to get the dog some exercise [02:03] and to leave the house for the day [02:03] bbl [02:05] thumper: ship it [02:05] wallyworld: ok, time to actually figure out this bug, now that I have some data that's not just "something went wrong" [02:06] natefinch: awesome, otp will catch up soon [02:07] chrome/linux/somebody is getting mad that I'm copying & pasting 3.8megs to pastebin [02:08] * natefinch remembers that pastebinit exists [02:38] nuts, my dns entry for .ws aged out [02:40] thumper: https://bugs.launchpad.net/juju-core/+bug/1588574 [02:40] Bug #1588574: Session already closed in state/presence

[02:40] i cannot reproduce this locally [02:40] stress tested for two hours [02:40] can you ? [02:41] ironically it just hit me in CI http://juju-ci.vapour.ws:8080/job/github-merge-juju/7972/console [02:53] wallyworld: am I reviewing https://github.com/juju/juju/pull/5519 or https://github.com/juju/juju/pull/5525? [02:53] axw: the one merging into the feature branch [02:54] cmars: i don't see how the currebt PR fixes the deploy_test compile error? [02:54] wallyworld, did i forget to push? checking.. [02:56] wallyworld, it fixes it because I fucked up the field name in changeset number 2. see http://reviews.vapour.ws/r/4971/diff/2-4/ [02:56] so from 0-4 you'll see no net difference but the landing bot failed on #2 i think [02:57] cmars: ah right, ok. so we retain the field name but serialise as "service-name" [02:57] wallyworld, yeah, wireformat needs more time to change [02:58] cmars: np, lgtm [02:58] ty [02:58] wallyworld, thanks! [03:00] davechen1y: I don't think I have hit it locally for quite some time [03:00] not sure where the race [03:00] is [03:01] i'm pushing some debug into mgo.v2 to record the stack of _who_ closed the session that we got handed [03:01] it's going to be hard to test this if I cannot repro it locally [03:01] fwwi, the presence tests are unstable AF, but just not this failure ... [03:05] davechen1y: yeah, presence is kinda crap [03:05] menn0: speaking of mgo, what's the status of the idea of having mgo tell us which assert failed ? [03:06] wallyworld: https://bugs.launchpad.net/juju-core/+bug/1586089 -- I've thought about the azure instance IDs a bit, I think we should do the same thing that thumper has done for LXD/MAAS [03:06] Bug #1586089: azure arm instance-ids are not ids, cannot find machine in azure

[03:07] wallyworld: atm they're just "machine-0" (e.g.) because instance IDs never clash across models, there's the invisible resource group qualifying them [03:07] that could be a good solution [03:07] wallyworld: but then you need to know which resource group to look in, so having the suffix of the model UUID would make it easier to identify [03:07] yep, and it's no worse than other providers [03:08] wallyworld: another thing that needs to be done before beta9 I'm afraid [03:08] indeed [03:08] 1588574 [03:09] logger.Infof("watcher loop failed: %v", err) [03:09] seriously ... [03:09] davechen1y: getting notified of which assert failed would probably have saved me days worth of work on my current bug [03:09] errors logged at info [03:10] thumper: I haven't raised the idea with niemeyer [03:10] thumper: it would be a big change b/c the current API returns a static ErrAborted [03:10] thumper: all users of mgo would have to change [03:10] ugh [03:11] :( [03:14] natefinch: how is the bug going? [03:17] wallyworld: examining the transaction asserts to try to figure out why they're failing. Luckily have some success cases and some failure cases to compare [03:17] ok [03:21] wallyworld: btw, the problem was that I was replacing the machine agent's binary, but it was actually logging an error received from an API call to the controller.. so it was the controller that I really needed to replace the binary on. It just wasn't clear from the error message in the machine agent log that it was an API call. But once I thought about it, I realized it had to be the controller, since it was code in state. I was too hung up on [03:21] the assumption that I needed to patch the code where the error was being logged. [03:21] * thumper afk to collect kids from school [03:22] natefinch: makes sense. easy in hindsight [03:23] defer func() { [03:23] // If the session is killed from underneath us, it panics when we [03:23] natefinch: also, you'll see you got a bug to fix some fallout from the admin to controller name change [03:23] // try to copy it, so deal with that here. [03:23] if v := recover(); v != nil { [03:23] err = fmt.Errorf("%v", v) [03:23] } [03:23] }() [03:23] what the F [03:24] wallyworld: good times :/ [03:25] davechen1y: where's that code? [03:25] state/presence [03:29] axw: should we change the model global key to "m" (currently "e"). this will be our last chance [03:31] davechen1y: if only someone had reviewed that code: https://github.com/juju/juju/pull/361#discussion-diff-15266925 [03:32] touche [03:32] wallyworld: I think m is taken for machines? [03:32] I think it's natural to hate past you. Past me screws me all the time. [03:32] I think this is before michael foord discovered just how bad an idea this was [03:32] natefinch: https://github.com/juju/juju/pull/5530 [03:33] give that we don't apply the same logic everywhere else we call Copy [03:33] axw: yeah, m#123 is, but not "m" on its own [03:33] wallyworld: I think having m being wildly different m#123 is a recipe for confusion [03:34] ok [03:34] my OCD kicks in when I see "e" [03:36] wallyworld: "e" is for "em" ? :) [03:36] lol [03:38] haha [03:38] perhaps we should rename machine to instance [03:38] then we wouldn't have two m's [03:40] thumper: i do wish we could solve this. but too hard i think [03:41] "n" for node maybe [03:41] model for model [03:41] global keys can only be one letter i think [03:41] I was gonna say, is there a byte shortage? [03:41] bollocks [03:41] in the past i've run into issues [03:41] global keys are made up from multiple ids [03:41] the prefix bit [03:41] r#foo#bar#baz [03:41] uses split on # [03:42] right, the r# bit [03:42] the r is only one letter [03:42] they just have to be unique in the places it is used [03:42] no, can be anything [03:42] there's code that assumes it's one letter [03:42] it is short by convention [03:42] where? [03:42] i'll try and find it [03:42] i've tried to change it before and it's caused a bug [03:43] rule #1 of code club is: don't encode data in your ids [03:43] thumper: backingEntityIdForGlobalKey [03:43] that would apply to model thugh in this case [03:44] wouldn't [03:44] so we could make it "model" [03:44] there might be other places too [03:44] JFDI and see what breaks [03:44] but the above is one i can find quickly [03:44] yeah, might do just that [04:25] menn0: thumper http://reviews.vapour.ws/r/4977/ [04:25] davechen1y: lookin [04:27] ta [04:34] why are there three watcher implementations in state [04:34] we have state/ [04:34] state/multiwatcher [04:34] and state/presence [04:34] each with their own watcher model [04:34] thinggy [04:44] davechen1y: review done... not LGTM I'm afraid [04:45] davechen1y: the Watcher in state/presence is a completely different thing to the watchers elsewhere [04:45] it's unfortunate that the same name was used [04:46] wallyworld: oh, I wanted to talk about the lxc to lxd stuff [04:46] ok [04:47] davechen1y: state/watcher implements the low level infrastructure that accepts requests for things to watch, tracks the txn log, and reports when things that are being watched have changed [04:47] wallyworld: if this is the right doc: https://docs.google.com/document/d/1SXBJJ_HHDX4_WvNGVGtLhNajXaioOF_Pn8-EvP1WvaU/edit [04:47] wallyworld: then I'm pretty unclear on which things are tasks Im supposed to do [04:48] --to lxd [04:48] lxd in bundles [04:48] are the first 2 main things [04:48] davechen1y: the stuff in state/watcher.go and state/allwatcher.go and state/multiwatcher all rely on what's in state/watcher [04:48] wallyworld: this says to remove --to lxc.. but I think you want it to stick around, warn, but then convert to a --to lxd, correct? [04:49] well, we could remove that one. the thing we need to warn about is in bundles [04:49] menn0: what a facepalm of tighlyt coupled code [04:49] wallyworld: ok [04:49] since we don't want to brek existing bundles [04:49] wallyworld: right [04:50] menn0: thanks for the review [04:50] natefinch: also stuff like lxd-defaultg-mtu name change [04:50] instead of lxc-default-mtu [04:51] Adding "lxd-default-mtu" to mirror "lxc-default-mtu" and pass it through the API stack [04:51] menn0: i don't agree with your feedback [04:51] wallyworld: I have no idea what those words mean ^ [04:51] the session.Copy pattern is everywhere in the code [04:52] including multiple times in the presence package [04:52] natefinch: the default mtu is a network setting - "maximum transmission unit"; it defines packet size [04:52] wallyworld: ok, I guess I can grep for it easily enough [04:52] i think this code was an impleplete solution to that michael foord fixed later [04:53] natefinch: the doc says to mirror the setting but i don't see why we can't replace it [04:54] i'll check with john [04:54] wallyworld: yeah, just renaming it seems fine [04:54] natefinch: does this mean the other bug is fixed? [04:54] wallyworld: no, but I wanted to talk to you before tomorrow, so when I get it fixed, I know what to do afterward. [04:54] yay, ty [04:55] bet you'll be happy to see the back end of thst bug [04:55] wallyworld: yeah, I looked at the lxd stuff today and realized I was not clear about what I should be doing. Now I am. Thank you. [04:55] np [04:56] wallyworld: oh man... this thing was such a cluster just getting it going... like 3 different dumb mistakes stacked on top of one another. And added on top of needing to deploy openstack each time through another person... not good times. We lost like an hour or so today when their maas somehow imploded and lost all networking [04:56] (which is one of the times when I had a chance to go looking through the code and the lxd stuff) [04:58] if anyone wants to stare at some mgo transactions and try to figure out why they're failing, there's a nice fat 4meg log here: http://paste.ubuntu.com/16938088/ [04:58] joy [04:58] some of the logging is unfortunately duplicated as I was adding in fmt.printlns to try to ensure I wasn't getting screwed by f'd up logging. [04:59] so far I'm not seeing a pattern... the assert that fails is asserting that the current address either doesn't exist or is empty (which probably means it doesn't exist, and the precondition check is returning a zero value for the struct). Which sounds like it could be bad, except that the exact same kind of assert passes elsewhere [05:11] wallyworld: whelp, I gotta sleep. Will pick it up in the morning, now that we've figured out all the problems with getting debug builds up, I can sprinkle in smoe more logging, see if I can figure out what's goijng wrong. [05:11] ok, ty [05:23] davechen1y: I've replied [05:27] menn0: thanks, it's also frustrating that I cannot reproduce this on my machine [05:27] protip -- killall mongod while the tests are running; not a big deal [05:27] nothing panics [05:27] nothing stops [05:30] menn0: session.Copy is nuts [05:30] all it does is return a copy of the session, pointing to the same backing socket and stuff [05:31] but with it's _own_ mutex [05:37] menn0: axw thumper https://bugs.launchpad.net/juju-core/+bug/1588614 [05:37] Bug #1588614: mongodb dying does not cause tests to fail [05:37] can you please try to confirm this for me [05:37] it's a 10 second test [05:40] davechen1y: dies straight away for me [05:41] davechen1y: the tests exit straight away I mean [05:44] WOW, this is absolutely the words [05:44] https://github.com/go-mgo/mgo/blob/v2/log.go#L76 [05:44] worst [05:44] if the race detector is running, add some extra locking to make it happey [05:44] otherwise -- FUCKING! [05:44] FUCKIT [05:44] axw: can you paste the log [05:45] seriously, it's been running happily for a few minutes with no bo [05:45] no db [05:45] an, no, my mistake [05:45] davechen1y: http://paste.ubuntu.com/16940975/ [05:46] i don't get any of that output [05:47] my behaviour is entirely different, the tests sit trying to find a working cluster menber for minutes [05:47] then give up [05:47] davechen1y: you're on master right? [05:48] yes [06:30] wallyworld: we may need to remove the ability to bootstrap without first defining a cloud (e.g. bootstrap manual/), if we're requiring each model to have a cloud name [06:30] or otherwise come up with a way of generating cloud names [06:30] it's probably less confusing just to require that they be added first tho [06:32] Bug #1588636 opened: mgo: Panic: Test left sockets in a dirty state (PC=0x46257C) [06:33] hmmm, that was a nice short cut for people, but we could remove it === mup_ is now known as mup === stokachu_ is now known as stokachu [08:43] frobware: joining now the HO for top of the hour [08:45] frobware: i'm in [08:46] jamespage: hey there [08:46] dimitern: otrp [08:47] jamespage: is the ntp charm maintained by the os-charmers? [08:47] frobware: ok, np [08:50] jamespage: it's the only one not reporting 'Unit is ready' status; see - just 4 lines short of a perfect output: http://paste.ubuntu.com/16942268/ :) [08:52] frobware, dimitern: could you take a look at http://reviews.vapour.ws/r/4967 [08:52] I was hoping to get Menno to look at it but I think he's been too busy. [08:53] babbageclunk: sure, will take a look [08:54] dimitern: thanks! [09:06] wallyworld: charlotte just reminded me that it's a public holiday here on monday [09:06] wallyworld: nearly got the guts of my branch done, still tests to write though [09:35] axw: still here? [09:37] axw: I left you a priv message [09:48] babbageclunk: reviewed === \b is now known as benonsoftware [11:17] dimitern, not at the moment [11:17] dimitern, we do have todo's to ensure that all charms that are part of openstack do status [11:31] jamespage: ok [11:32] jamespage: side-note - any plans for neutron to support bindings for the external network as well as the internal (data)? [11:34] jamespage: it looks like the tenant net (and its underlying vlan in maas) also wants to use dhcp and insists on claiming a full /20 (i.e. router's int port uses the .1 address, which is also used by maas..) [11:46] dooferlad: could you try my patch as-is right now with a bonded interface (LACP)? [11:46] dooferlad: http://reviews.vapour.ws/r/4969/ [11:50] dooferlad, dimitern: we're screwed [11:50] dooferlad, dimitern: I just got caught out again by /etc/network/interfaces.d/eth0.cfg [11:51] dooferlad, dimitern: guess where my route over eth0 came from... http://pastebin.ubuntu.com/16944207/ [11:52] * frobware sulks and finds lunch... [11:57] Bug #1588784 opened: juju/state: intermittent test failure in SubnetSuite with mongod 3.2

[12:05] fwereade: Here are all the details for that bug, with the standalone reproduction attached. [12:05] babbageclunk, cool [12:17] babbageclunk, nice bug report <3 [12:18] dooferlad, dimitern: the only way I can get stuff to work atm is... https://github.com/frobware/juju/tree/master-lp1588446-hack-hack-hackity-hack [12:18] frobware: looking [12:20] dimitern: I'll save you the effort: https://github.com/frobware/juju/blob/master-lp1588446-hack-hack-hackity-hack/provider/maas/add-juju-bridge.py#L454 [12:20] frobware: aw, that.. [12:20] fwereade: why also remove /e/n/i.d/* ? [12:21] oops [12:21] frobware: ^^ [12:21] dimitern: because eth0 has a running dhclient [12:21] dimitern: from eth0.cfg [12:21] dimitern: so we end up with a route out via eth0 and the nascent br-bond0 [12:21] frobware: but, if we replace /e/n/i completely like in my patch? [12:21] dimitern: show me your wares.... [12:22] frobware: https://github.com/juju/juju/pull/5512/files#diff-59aa120b6b815c64439ef2971190a313R61 [12:22] dimitern: this is a joke. [12:22] dimitern: we need to stop SPINNING endlessly on this. [12:22] frobware: nope, that's for containers, sorry.. [12:23] dimitern: I was confused... [12:25] frobware: well, my patch doesn't change where we generate /e/n/i with bridges [12:25] frobware: but looking at one of the nodes, I can see http://paste.ubuntu.com/16944715/ [12:26] dimitern: the problem is we have a dhclient (unexpected item in bagging area) running against eth0 [12:27] frobware: and my patch does rm 50-cloud-init link from /e/systemd/network/ [12:27] dimitern: does it kill the dhclient for that iface? [12:28] dimitern: after briding bond0 (parent eth0, eth1) I have http://pastebin.ubuntu.com/16944207/ [12:28] frobware: well, it's not running now on that same node [12:28] dimitern: everything is fine apart from lne 2 [12:28] frobware: but haven't checked right after boot [12:28] frobware: I suspect that link causes sysd to spawn dhclient, not the one in /e/n/i.d/ [12:28] dimitern: if your 50-cloud-init.cfg has 'eth0: dhcp' you're screwed too [12:29] frobware: I need to try with a bond to see how is it [12:29] frobware: paste the -before-add-juju-bridge /e/n/i ? [12:30] dimitern: http://pastebin.ubuntu.com/16944804/ [12:30] dimitern: line 45 is the killer [12:31] fwereade: thx :) [12:31] frobware: yeah.. [12:31]