[00:37] ping alexisb [00:38] heya davechen1y [00:38] whats up? [00:38] alexisb: i just sent you an email that requires acknowledgement [00:38] it's very short, could you please read and respond [00:38] thanks [00:38] davechen1y, will do [00:39] thanks [00:41] davechen1y, responded [00:41] alexisb: ta [00:45] ok, next question [00:45] JFDI or not JFDI [00:45] i have a PR that is at the bottom of a lot of blocked bugs for beta8 [00:45] can I JFDI ? [00:47] davechen1y: is it blocker? u can't just use $$fixes-blah$$? [00:49] anastasiamac_: i don't know [00:49] the issue i am assigned is not the one that is blocking landing [00:49] ic. is it the race fix? [00:50] yes [00:50] well [00:50] maybe [00:50] this is not the issue that CI is blocked o [00:50] on [00:51] so u r right, plz do not JFDI unless u have an explicit "please land" from alexisb or cherylj or sinzui [00:51] ok, i've marked my PR as blcoked on the landing bot and i'll work on something else [00:51] \o/ [00:52] how can I find out when this build was run http://reports.vapour.ws/releases/3993/job/run-unit-tests-race/attempt/1495 [00:52] this is the last race build that was run [00:52] i require the CI rae infrastructure to validate the fixes I am landing [00:52] is the ci race build running currently ? [00:54] davechen1y: follow the link to the build number in the top nav. http://reports.vapour.ws/releases/3993 Wed May 25 08:59:46 2016 [00:54] i get a 404 [00:54] when I follow that link [00:55] sorry, i get a 404 when I follow the link to the jenkins job [00:55] davechen1y: are you no longer a member oc canonical? [00:55] sinzui: one can only hope [00:55] davechen1y: jenkins jobs only last a few days. All the data is on the reports site [00:56] sinzui: so does that mean the race job has not run for at least a few days ? [00:56] run-unit-tests-race1495Bug #1579051 [00:56] Bug #1579051: Race in juju/controller/destroy and TestDestroyCommandConfirmation [00:56] ^ this bug that the race build is complaining about is marked fixed release [00:56] davechen1y: no. the job is run with every revision [00:57] sinzui: that doesn't make sense [00:57] if run-unit-tests-race 1495 was run recently, as in since last night then the jenkins job would be 404 [00:57] wouldn't be 404 [00:57] davechen1y: if you cannot seen any the job at all, then you need to login. 90% of the site is not public because juju likes to send my credentials to logs. [00:58] sinzui: i dont' have a login for that site [00:58] or chrome has forgotten them :( [01:10] Bug #1585836 opened: Race in github.com/juju/juju/provider/azure [01:22] Bug #1585836 changed: Race in github.com/juju/juju/provider/azure === Makyo is now known as Makyo|away [01:28] Bug #1582408 changed: System id is listed during bootstrap and deploy of charms instead of system name [01:28] Bug #1585836 opened: Race in github.com/juju/juju/provider/azure [01:29] axw: a small one when you are free http://reviews.vapour.ws/r/4905/ === natefinch-afk is now known as natefinch [01:31] Bug #1582408 opened: System id is listed during bootstrap and deploy of charms instead of system name [01:31] wallyworld: what changed the error message change? [01:32] axw: new dep i suspect [01:32] httprequest maybe [01:32] wallyworld: that function just uses net/http AFAICS [01:34] axw: yeah, i have no idea why that message changed then, i can look into it i guess, i hust assumed it was due to one of the new deps [01:34] wallyworld: please do, doesn't look related to me [01:34] ok, otp will do soon [01:35] * redir goes eod [01:40] Bug #1582408 changed: System id is listed during bootstrap and deploy of charms instead of system name [02:37] wallyworld: I see you reverted, did you get to the bottom of it? [02:38] axw: nope, master failed in the same way, nfi [02:40] Bug #1585846 opened: worker/terminationworker: test timeout after 20 minutes [02:40] Bug #1585847 opened: LXD creates all the interfaces as the physical machine has when using MAAS. [02:48] axw: you happy to +1 with that error message reverted? [02:48] wallyworld: yep, will do [02:48] ta [02:49] wallyworld: done [02:49] ty [03:10] Bug #1585849 opened: Azure native bundle 404 creating resources [03:10] Bug #1585851 opened: Cannot restore-backup of controller model [03:31] Bug #1585856 opened: worker/txnpruner: test failure during ci [03:52] Bug #1585857 opened: Unable to bootstrap to maas 2.0 cloud with juju 2.0 beta 7 [04:00] axw: as per roger's suggestion, i added extra tests and improved the marshalling https://github.com/juju/charm/pull/210 [04:03] wallyworld: LGTM [04:03] ty [04:20] axw: small one, https://github.com/juju/idmclient/pull/17 [04:21] there's sooooo many :-( === redir is now known as redir_afk [04:56] axw: idmclient just uses UserTag so v1 and v2 are the same in that respect [04:57] wallyworld: I was thinking more that names types might exposed in the idmclient API, but it seems not [04:57] wallyworld: shipit [04:57] ta === urulama|swap is now known as urulama [05:53] Bug #1585878 opened: It does not remove the DHCP and DNS configurations on MAAS after juju removes LXD container(machine). [06:29] Bug #1585878 changed: It does not remove the DHCP and DNS configurations on MAAS after juju removes LXD container(machine). [06:38] Bug #1585878 opened: It does not remove the DHCP and DNS configurations on MAAS after juju removes LXD container(machine). [06:44] axw: do you have time to talk? [06:44] wallyworld: yes, but gotta go in ~15m to pick up charlotte [06:44] ok, 1:1 [06:45] axw: google hates me, be there in a sec [06:53] Bug #1585878 changed: It does not remove the DHCP and DNS configurations on MAAS after juju removes LXD container(machine). [07:17] Bug #1585878 opened: Removing a container does not remove the underlying MAAS device representing the container unless the host is also removed. === frankban|afk is now known as frankban === frankban is now known as frankban|afk === frankban|afk is now known as frankban [08:31] frobware: 1:1? [08:32] babbageclunk: yep, sent link via PM [09:02] dimitern, dooferlad: standup [09:02] frobware: just booking some passport stuff. There soon. [09:02] dooferlad: k === akhavr1 is now known as akhavr [10:19] http://reviews.vapour.ws/r/4900/ can anyone have another look on this PR? it fixes this https://bugs.launchpad.net/juju-core/+bug/1585430 [10:19] Bug #1585430: Cloud-init failed on windows [10:23] fwereade: ping [11:44] voidspace, pong [12:06] Bug #1585857 changed: Unable to add credential to maas cloud with juju 2.0 beta 7 [12:06] Bug #1586007 opened: cannot ssh to LXD container on non MAAS systems using juju ssh --proxy 0/lxd/0 [12:06] fwereade: philosophy question about model migration [12:07] fwereade: will we assume data integrity - i.e. referred to entities exist - or should we import entities in the right order (any circular references will be a problem) and verify during import [12:07] fwereade: i.e. addresses reference devices (amongst other things) - must we import the devices first and check when we import the addresses [12:07] fwereade: or is it ok to assume data integrity (the device must have existed for the address to be created) [12:08] fwereade: without checks the new model will only be as screwed as the old model [12:08] fwereade: with checks we may refuse to import at all [12:14] voidspace, my understanding is that we validate the model both source-side and dest-side [12:14] fwereade: so we need to ensure we do the imports in the right order [12:14] voidspace, if either side fails we can't migrate but we can at least unfreeze the model in the source controller and keep it going [12:15] fwereade: and to do the checks the imports need to be implemented in the right order [12:15] voidspace, I'm not sure we do? the interface seems to me like it's tuned for just setting lots of things and then validating the whole lot at the end [12:16] voidspace, so source validates before sending; target validates before applying; failure at any stage, including application, should roll back the whole thing [12:16] voidspace, we only drop the model from the source when everything's confirmed it's happily talking to the target controller [12:17] fwereade: certainly a validation step could be added [12:17] fwereade: I don't *see* it here though: https://docs.google.com/document/d/1HVWgLxV3CJWI-IAzb2NvFOxhkx4Q3qsePxQ9-F2ZYIE/edit# [12:19] voidspace, core/description.Model has a Validate method [12:20] fwereade: ah, indeed it does [12:20] voidspace, I don't think of validation as a separate step so much as something you Just Do when data crosses a boundary [12:20] voidspace, definitely on the way in, plausibly on the way out as well at times [12:21] fwereade: the examples I was following for implementation didn't need any validation so didn't include anything here [12:21] fwereade: ok, so some additional work needed here [12:21] voidspace, I'm not really familiar with the nuts and bolts of the validation, I'm afraid [12:22] fwereade: cool, you answered my question anyway [12:22] fwereade: appreciated [13:03] cherylj: ok to merge http://reviews.vapour.ws/r/4903/ ? [13:28] dooferlad: ping [13:30] frobware: pong [13:30] dooferlad: having trouble with your bridge script changes - can we HO? [13:32] frobware: sure [13:32] frobware: juju-sapphire? [13:32] yep [13:48] voidspace: ping? [13:51] Bug #1518806 changed: apiserver: tests to not pass with -race under Go 1.2 <2.0-count> [13:51] Bug #1519133 changed: cmd/jujud/agent: data race <2.0-count> [13:51] Bug #1519191 changed: worker/addresser: FAIL: worker_test.go:260: workerEnabledSuite.TestWorkerAcceptsBrokenRelease <2.0-count> [13:51] frobware: yes, go ahead and merge [14:08] dimitern: ping? [14:08] babbageclunk: pong [14:09] dimitern: I'm trying to debug a flaky test I'm hitting. [14:10] It's this: https://github.com/juju/juju/blob/master/state/subnets_test.go#L424 [14:10] dimitern: (asking you because voidspace didn't respond) [14:10] babbageclunk: let me have a look.. [14:11] babbageclunk: I don't think it's flaky [14:11] dimitern: It looks like the patch there isn't working - the code doesn't use the patched name, so it's calling the original function and just happening to pass. [14:11] dimitern: It is on 3.2 with my changes ;) [14:12] babbageclunk: yeah, the way PatchValue is used might cause it to be though.. [14:12] dimitern: which might be my fault, but I'm trying to understand what's happening. [14:12] babbageclunk: have a look at the internals of PatchValue - there are 2 slices of patched stuff - suite-level and test-level [14:13] dimitern: But in the code https://github.com/juju/juju/blob/master/state/subnets.go#L330 it uses the lowercase name, so it looks like it always uses the unpatched function [14:13] (Also I put in some logging and verified that.) [14:14] babbageclunk: oh .. [14:14] babbageclunk: the real pickAddress is non-deterministic [14:14] babbageclunk: as it picks random addresses from the subnet range [14:15] dimitern: So I think the test has been passing by accident. [14:15] babbageclunk: I suspect the way it's patched is wrong [14:16] babbageclunk: e.g. I'd expect to see something like s.PatchValue(&state.Foo, func(matching-args-decls) { ... }) [14:16] dimitern: Well, the test is in a different package, so I don't think it can patch the lowercase name. [14:16] babbageclunk: i.e. &mockPickAddress looks wrong [14:17] babbageclunk: depends how state.PickAddress is exported [14:17] Ok - so you think it's the & that's wrong, rather than the naming? [14:18] babbageclunk: for funcs, it's quite common to use var Foo = &foo in export_test, then s.PatchValue(pkg.Foo, func (...) { .. }) [14:18] babbageclunk: naming is fine - it looks weird, but because of export_test.go (btw such things cannot be followed with godef as well) [14:20] babbageclunk: OTOH, you have var Foo = foo in export_test of package pkg, &pkg.Foo in pkg_test points to the pkg.Foo var in export_test, NOT to the pkg.foo [14:21] ping natefinch [14:21] dimitern: Ok, I think I follow - I'll try changing export_test to make it work. [14:23] babbageclunk: that is important for patching functions, not so much for simple vars (e.g. exposing struct foo{unexp. fields..}) [14:24] babbageclunk: +1 [14:24] Bug #1580186 changed: Race in github.com/juju/juju/worker/instancepoller/aggregate_test.go [14:24] Bug #1585430 changed: Cloud-init failed on windows [14:24] Bug #1585856 changed: worker/txnpruner: test failure during ci [14:24] Bug #1586057 opened: Race in github.com/juju/juju/container/lxc [14:25] dimitern, jam: were we going to talk about fan/networking today? [14:29] frobware: we still can :) [14:44] babbageclunk: sorry, was on lunch [14:45] voidspace: No worries. Picked dimitern's brains instead, although I'm still a bit stuck [14:46] babbageclunk: yeah, read the backscroll [14:47] voidspace: can you see what's wrong with the patching? Is it possible to do that from outside the package if the code's calling the lowercase name? [14:47] babbageclunk: no, the code should call the name that's being patched [14:49] So maybe I should change subnets.go to assign PickAddress (and call it) and remove the alias from export_test? [14:50] babbageclunk: the alias is needed to allow patching [14:51] babbageclunk: without an alias it can't be patched [14:51] babbageclunk: are you suggesting removing the patching (which doesn't do anything) as well? [14:51] voidspace: I thought the alias was to make it visible from outside the package. [14:52] babbageclunk: you can't patch a function directly - however if you create a pointer alias you can patch *that* [14:52] voidspace: no - the test needs the patch, otherwise it randomly fails. [14:52] babbageclunk: so just change the code to call the upper case version (the patched one) [14:53] voidspace: pickAddress is defined as var pickAddress = func in subnets.go. [14:53] voidspace: But the uppercase one only exists in exports_test.go [14:53] babbageclunk: hmm... [14:53] voidspace: (sorry, export_test.go) [14:54] babbageclunk: so an uppercase alias in export_test is usually for testing an unexported function [14:54] babbageclunk: and a function created as var *can* be patched, but if it's lower case then only from within the package - not from a black box test [14:54] Bug #1585430 opened: Cloud-init failed on windows [14:56] voidspace: So I couldn't rebind a function defined as an uppercase var from outside the package? (As I type that it sounds like a bad idea even if possible.) [14:58] voidspace: turns out that *does* work - the patched function now gets called. But it still sometimes fail for reasons I've not yet worked out. [14:58] babbageclunk: ok, that sounds like the right thing to do for this test [14:58] babbageclunk: now work out why it fails and fix it :-) [14:59] voidspace: Maybe I should have done that in the first place. :) [14:59] babbageclunk: hah [14:59] voidspace: The first thing I did was get confused about why my added logging wasn't showing up. [15:04] babbageclunk: why wasn't it? [15:05] voidspace: because I added it to mockPickAddress, which wasn't being called. [15:06] ah! [15:10] hey, is there a mongodb 3.2 in the ubuntu archives? i noticed that we're using it for the controller now.. is there a ppa where i can apt install it from too? [15:11] cherylj: fyi - https://bugs.launchpad.net/maas/+bug/1586075 [15:11] Bug #1586075: Deploying a trusty node on MAAS 1.9.3 gives us 2 entries for eth0 in /e/n/i [15:43] * katco needs to reboot [15:44] frobware, dimitern, just sent a note, take a look and let me know [15:45] alexisb: looking [15:46] alexisb: sure [15:55] alexisb: replied [15:55] dimitern, thanks [16:16] fwereade: you around? [16:18] Bug #1586089 opened: azure arm instance-ids are not ids, cannot find machine in azure === frankban is now known as frankban|afk [16:47] perrito666: ping? [16:47] babbageclunk: pong [16:48] perrito666: I'm just trying to finish up these mongo3.2 changes... [16:49] perrito666: but I've hit a couple of places where 3.2 behaves a bit oddly [16:49] babbageclunk: ok, tell me, how can I help you? [16:49] perrito666: (Happy birthday for yesterday by the way!) [16:49] babbageclunk: tx :) [16:50] perrito666: one of them is in OplogTailer [16:51] mh, I am not familiar with it [16:51] perrito666: Doh, I thought I saw your name on it, sorry! [16:52] babbageclunk: I might have passed by changing something [16:52] but I am pretty sure its one of menn0's [16:53] perrito666: yup, you're right [16:53] sorry that was not helpful :( [16:54] perrito666: Well, I'll bug him about this instead! Sorry! [16:54] perrito666: :) [17:01] voidspace, dimitern, babbageclunk, dooferlad: anybody done $(juju add-machine lxd:0) on trusty recently, bootstrapping on MAAS? [17:01] frobware: not in a few days [17:01] frobware: nope, just mongo bashing [17:01] dimitern: I don't seem to have backports enabled in trusty any more [17:02] frobware: did you update the images? [17:02] https://bugs.launchpad.net/juju-core/+bug/1568895 [17:02] Bug #1568895: Cannot add MAAS-based LXD containers in 2.0beta4 on trusty [17:03] dimitern: my maas setup says "last update: Thu, 26 May. 2016 17:45:03" [17:03] dimitern: are you in a position to run a lxd container at all? [17:04] babbageclunk, dooferlad, voidspace ^^ [17:05] frobware: let me do a quick test locally.. [17:16] frobware: uh oh.. it failed to deploy at all.. I was messing with LVM groups and volumes to try to simulate having 2 disks to deploy OS [17:16] dimitern: so unrelated failures? [17:17] frobware: I'm trying again to verify.. [17:17] dimitern: ok, also trying on my laptop which still has maas-1.9.2 [17:18] dimitern: though I don't think that's the issue at all [17:18] frobware: ok, now it got deployed, adding lxd:0 [17:20] dimitern: in theory you shouldn't need juju [17:20] dimitern: just deploy a node [17:20] dimitern: you only need to verify the content of /e/apt/sources.list [17:21] frobware: no backports here [17:21] dimitern: I think this is bust again. Let me update the bug. Could you do so too. [17:21] E: The value 'trusty-backports' is invalid for APT::Default-Release as such a release is not available in the sources [17:23] dimitern: sigh. that was a waste of a few hours. [17:23] dimitern: sometimes it feels hard to make progress. :( [17:24] dimitern: I'm surprisd this is not showing up in CI tests [17:24] sinzui: ^^ [17:26] frobware: updated the bug (I assume you meant 1569985?) [17:26] oops not that - 1568895 [17:26] dimitern: yep [17:26] frobware. I think we have seen it., but is is not a current issue [17:27] frobware: bug 1556137 mentions it [17:27] Bug #1556137: Juju 2.0: cannot connect to LXC container when deploying bundle with LXC container placement with maas 1.9.1 as provider <2.0-count> [17:28] sinzui: ooh, I now why that one is broken [17:28] Bug #1585430 changed: Cloud-init failed on windows [17:28] Bug #1586116 opened: GCE bootstraps but fails to provision [17:30] sinzui: ah, sorry, nope. I was actually thinking of: https://bugs.launchpad.net/juju-core/+bug/1576674/ Sorry for the noise [17:30] Bug #1576674: 2.0 beta6: only able to access LXD containers (on maas deployed host) from the maas network [17:31] frobware: if you're still about for a short while, I'm finally proposing a fix for the IPv6 bug [17:31] dimitern: I can be [17:31] (take 3, totally rethought and as simple as possible) [17:31] dimitern: let's stop at 11. :) [17:34] frobware: good plan :) [17:35] frobware:http://reviews.vapour.ws/r/4915/ [17:35] dimitern: looking but will mostly try first. [17:35] frobware: +1 [17:36] dimitern: for a repro, just deploy 20 units? [17:36] frobware: yeah - less than 20 should still work [17:37] frobware: ah! important: make sure your lxdbr0 is configured to give *both* ipv4 and ipv6 addresses [17:38] sinzui: what's the correct status for reopening a bug? mark as "confirmed" again? Just want to ensure bug #1568895 get some other eyes verifying current status [17:38] Bug #1568895: Cannot add MAAS-based LXD containers in 2.0beta4 on trusty [17:38] dimitern: does it matter which series I should use? [17:38] frobware: if you have the power, mark it triaged, otherwise confirmed [17:39] frobware: it does matter if the issue is not in master [17:39] sinzui: seems more likely to be outside of juju [17:40] frobware: I used xenial, but shouldn't matter really [17:40] frobware: actually a couple of days I also tested with trusty (the earlier fix) [17:41] dimitern: but not with containers [17:41] frobware: about trusty-backports. Old images do not have them enabled. backports were enabled by default in xenial images. trusty imagages were updated and released to the clouds a few months ago [17:42] sinzui: but I did go through a stage where a trusty MAAS node had backports enabled. [17:45] frobware: I expect newish trusty images to have trusty-backports enabled. The maas images are made by the team that make cloud-images [17:46] sinzui: but wasn't this previously failing in CI, then confirmed as working once the images had been updated. [17:48] bbl [18:06] * natefinch is a team of one, evidently. [18:06] go team nate [18:06] go team natefinch even [18:06] :) [18:07] rah rah ree kick em in the knee [18:07] Bug #1568895 opened: Cannot add MAAS-based LXD containers in 2.0beta4 on trusty [18:07] I guess we're not having the team meeting today? Now that we only have managers that are asleep at this time, having a team meeting seems silly [18:53] bbiab [19:04] alexisb: [19:34] Bug #1586150 opened: destroy-model does not remove the model [19:46] Bug #1586150 changed: destroy-model does not remove the model [19:55] Bug #1586150 opened: destroy-model does not remove the model [20:49] Bug #1586150 changed: destroy-model does not remove the model [20:56] natefinch: katco: jsut as a heads up, if you have some time could you take a look at https://github.com/juju/docs/pull/1122 [20:57] we need to get terms docs to users to so they can start prepping their charms for GA [20:57] 2.0 ga, that is [20:57] mbruzek: ^ [21:09] arosales: ty, wil try to hook up cmars and his team with the docs folks as they did terms [21:11] rick_h_, arosales https://github.com/juju/docs/blob/master/src/en/developer-terms.md [21:11] cmars: <3 [21:13] cmars: :-) so those land and I already referenced them on the list :-) [21:14] s/so/saw/ [21:15] sorry earlier I meant to say resources we need a review on [21:15] per https://github.com/juju/docs/pull/1122 [21:15] I think mbruzek was looking for a final ack from katco and natefinch [21:15] for resources on terms [21:33] alexisb: release standup? === meetingology` is now known as meetingology [21:56] tych0: ping? [22:11] Bug #1585851 changed: Cannot restore-backup of controller model [22:14] hey menn0! [22:15] babbageclunk: hey hey! [22:19] menn0: happy birthday for yesterday! [22:19] menn0: Have you got a moment for a quick q? [22:19] babbageclunk: cheers [22:19] babbageclunk: sure [22:20] menn0: I'm trying to get the full test suite passing with Mongo 3.2... [22:21] menn0: But there are a couple of places where we get different errors now than we used to. [22:21] menn0: *sometimes* [22:21] menn0: So they make the tests flaky. [22:22] babbageclunk: that sounds fun :-/ [22:22] menn0: Well, it's a bit suboptimal. [22:22] babbageclunk: do you mean that for the same root cause mongodb returns different errors at times? [22:22] menn0: Yeah - for example: [22:23] https://github.com/babbageclunk/juju/commit/61da79e198d15263579f717da21e0e5925da0f3b [22:24] menn0: Instead of a nice mgo.ErrCursor, we sometimes get a generic mgo.QueryError [22:24] babbageclunk: ah right... I even know about that one. https://bugs.launchpad.net/juju-core/+bug/1573286 [22:24] Bug #1573286: juju/mongo: oplog tests fail with mongod 3.2 [22:25] menn0: Ha! [22:25] menn0: Ok, hadn't seen that. [22:25] babbageclunk: mwhudson created that ticket when he first ran into this [22:25] and told me about it [22:25] wallyworld, menn0, cherylj, sinzui, rick_h_, I need some of your time i fyou are available [22:26] babbageclunk: I haven't put any thought into it yet [22:26] alexisb: i'm around [22:26] alexisb: ok [22:26] https://plus.google.com/hangouts/_/canonical.com/juju-release [22:26] babbageclunk: wow, that general task is one hell of a welcome to the job [22:26] mwhudson, yes yes it is [22:27] babbageclunk: let me have a think about that particular issue ... have to jump on this call [22:27] alexisb, mwhudson: luckily babbageclunk is a smart cookie :) [22:27] menn0: Yeah - I'm not sure whether my fix is good - it's essentially only happpening during the tests, so it seems odd to handle it in the prod code. [22:27] menn0: Ok, talk later. [22:27] menn0, alexisb: that's ok then :-) [22:27] * babbageclunk blushes. [22:29] mwhudson: It seemed ok at first! [22:30] babbageclunk: that's a good sign, i guess :) [22:31] mwhudson: I feel like it's *just this last bit*, but I've thought that for about 4 days now. :) [22:31] babbageclunk: heh [22:32] babbageclunk: did you beat the state tests into submission? [22:32] that seemed like the worst bit [22:32] mwhudson: Yeah, that 100x slowdown is crazy! [22:33] mwhudson: I mean, I've hacked around it, mostly (although they still run slower than with 2.4). [22:34] babbageclunk: oh yes, i saw something about that, emptying the db rather than recreating it? [22:34] mwhudson: yuo [22:34] uh, yup [22:34] babbageclunk: i also saw something running out of file descriptors sometimes, you've not hit that? [22:34] mwhudson: No... [22:35] well good, i guess :) [22:35] mwhudson: I saw your mention of it, but it's not happened to me. [22:35] mwhudson: good unless it starts happening in builds I guess. [22:35] heh yes [22:37] mwhudson: Right, I have to go sleep. Nice to meet you! [22:37] babbageclunk: good night! [22:44] Bug #1585851 opened: Cannot restore-backup of controller model [22:46] ci builds are failing because gopkg.in is unwell [22:48] davechen1y: see #juju [22:49] davechen1y: sinzui is onto it \o/ but landing is paused for now... [22:55] babbageclunk: i'm off the call now... what was your fix for the oplog test problem? [22:56] menn0: I just in some code to check for the specific QueryError I was seeing, which is hacky but ok (I think). [22:57] menn0: It's really the other one that's giving me pause - sending you an email 'cause I really need to go to bed. [22:58] menn0: It turns out toddlers don't stay in bed later just because you got another baby! [22:58] babbageclunk: ok no problems. sleep is good :) i'll take a look at this today [22:58] menn0: Who knew? [23:01] babbageclunk: yep :) Sander gets up and into our bed about 5:30am every day. Amelia sleeps in most days at least [23:02] menn0: Say hi to the family! Night. [23:02] babbageclunk: goodnight! [23:23] Bug #1586197 opened: Upgrade to 1.25.5 not possible, due to cannot store tools metadata: invalid binary version \"1.25.5--amd64\"