[03:25] <natefinch> waigani: I keep seeing issues about "lingo ls" and misread it as "lingo is" ... and expect it to have the same output as "juju is" :)
[03:32] <waigani> natefinch: hehe
[05:41] <mup> Bug #1531064 opened: lxd: cannot create multiple environments <juju-core:In Progress by axwalk> <https://launchpad.net/bugs/1531064>
[05:44] <mup> Bug #1531064 changed: lxd: cannot create multiple environments <juju-core:In Progress by axwalk> <https://launchpad.net/bugs/1531064>
[05:50] <mup> Bug #1531064 opened: lxd: cannot create multiple environments <juju-core:In Progress by axwalk> <https://launchpad.net/bugs/1531064>
[08:57] <frobware> cherylj, I tried adding a container with my patch on 1.8.2 and it worked - worked in that /etc/resolv.conf was populated. Do you not see this on on 1.8.3? (Will install 1.8.3 in some spare time)
[10:23] <voidspace> dimitern: thanks!
[10:59] <dooferlad> dimitern: which branches are you wanting reviews on?
[11:00] <dooferlad> dimitern: the ones that are over two weeks old?
[11:00] <dimitern> dooferlad, nope, just the last one
[11:00] <dooferlad> dimitern: on it
[11:00] <dimitern> dooferlad, the other two will stay like that while I tease bits of them and propose them separately
[11:01] <dimitern> dooferlad, ta!
[11:09] <dooferlad> dimitern: you have a review
[11:16] <dimitern> dooferlad, awesome! thanks
[12:04] <frobware> dimitern, take a look at https://bugs.launchpad.net/juju-core/+bug/1528217/comments/12
[12:04] <mup> Bug #1528217: 1.25.2 doesn't set up DNS information with MAAS <blocker> <ci> <maas> <network> <regression> <juju-core:Incomplete> <juju-core 1.25:Triaged> <https://launchpad.net/bugs/1528217>
[12:04] <frobware> cherylj, mgz: ^^
[12:06] <mgz>  frobware: looking
[12:08] <mgz> frobware: yeah, I get the same on 1.8.3, double space where ip should be and all
[12:09] <frobware> mgz, so might just be a regression in 1.8.3
[12:09] <frobware> mgz, debugging some more.
[12:09] <mgz> I hae a maas checkout somewhere...
[12:15] <mgz> there's not much in 1.8.3
[12:27] <frobware> mgz, as in not much different?
[12:27] <mgz> indeed
[12:30] <mgz> http://paste.ubuntu.com/14410247
[12:52] <dimitern> frobware, looking
[12:53] <dimitern> frobware, hmm.. that means the ConfigType: "dhcp" wasn't taking effect?
[12:58] <dimitern> frobware, is that the result of changing static to dhcp in the InterfaceInfo ?
[13:14] <frobware> dimitern, no. That's my same change tested against .2 and .3.
[13:15] <frobware> dimitern, so 1.8.3, as reported, is behaving differently.
[13:16] <dimitern> frobware, hmm - yeah, but the lack of address in /e/n/i is troubling - got the logs to compare what happened?
[13:17] <frobware> dimitern, host and container is still up. Which would help?
[13:18] <dimitern> frobware, the machine log of the host - looking for things like "created device ..."
[13:18] <dimitern> frobware, both from 1.8.2 and 1.8.3
[13:19] <dimitern> frobware, logged the result of claim-sticky-ip-address should follow shortly after creating the device
[13:19] <dimitern> s/logged the/M-t/
[13:23] <frobware> dimitern, attached to https://bugs.launchpad.net/juju-core/+bug/1528217
[13:23] <mup> Bug #1528217: 1.25.2 doesn't set up DNS information with MAAS <blocker> <ci> <maas> <network> <regression> <juju-core:Incomplete> <juju-core 1.25:Triaged> <https://launchpad.net/bugs/1528217>
[13:23] <dimitern> frobware, looking
[13:24] <frobware> dimitern, not sure the 1.8.2 log has sticky in it. hmm.
[13:25] <dimitern> frobware, ah, sorry - those logs are not enough - machine-0 logs for both are actually better
[13:25] <dimitern> frobware, as the device registration etc. is handled by the environ provisioner (running on the api server(s) only)
[13:25] <frobware> dimitern, let's HO
[13:25] <frobware> dimitern, we can do stuff live
[13:26] <dimitern> frobware, ok, omw to standup HO
[14:04] <mattyw> fwereade, ping?
[14:07] <fwereade> mattyw, pong
[14:35] <frobware> mgz, cherylj: can you take a look on your 1.8.3 installations to see if the discovered interfaces (maas-eth0) has a default gateway. I'm guessing not. Explanation in: https://bugs.launchpad.net/juju-core/+bug/1528217/comments/15
[14:35] <mup> Bug #1528217: 1.25.2 doesn't set up DNS information with MAAS <blocker> <ci> <maas> <network> <regression> <juju-core:Incomplete> <juju-core 1.25:Triaged> <https://launchpad.net/bugs/1528217>
[14:40] <mgz> frobware: was reading the maas provide code yesterday, there's a branch that checks for default gateway and, ah I see doesn't actually do anything if it's empty
[14:52] <mgz> difference between our 18 and 19 is somewhat interesting:
[14:53] <mgz> http://paste.ubuntu.com/14411019
[14:53] <mgz> http://paste.ubuntu.com/14411028
[15:03] <frobware> mgz, and the last pastebin was 1.9?
[15:04] <mgz> frobware: yup
[15:04] <frobware> mgz, as you say, interesting!
[15:15] <frobware> dooferlad, voidspace, dimitern: http://reviews.vapour.ws/r/3451/ - PTAL now that we know why it fails on 1.8.3.
[15:16] <voidspace> frobware: will look shortly, just picking up girl from school
[15:16] <voidspace> brb
[15:17] <cherylj> frobware: yes, I see that there is no default gateway for maas-eth0
[15:20] <frobware> dimitern, can we really just s/network.ConfigStatic/network.ConfigDHCP/ in the provisioner? Apart from fixing up the failing tests is there anything else to be wary of with this change?
[15:22] <dimitern> frobware, looking at that PR
[15:22] <dimitern> frobware, I believe it's safe to do that, but only when AddressAllocationEnabled() is false
[15:22] <frobware> dimitern, right
[15:22] <dimitern> frobware, well.. and pending a subsequent live tests :)
[15:23] <frobware> dimitern, makes the tests interesting then
[15:23] <frobware> dimitern, My sample size of 1 indicates that 1 container comes with as expected.
[15:23] <frobware> s/with/up
[15:26] <dimitern> frobware, you've got a review btw
[15:26] <dimitern> frobware, that's good :) - was that on 1.8.2 without any manual setting of gateways ?
[15:27] <frobware> dimitern, it had a default gateway. I can try without now. Thanks for the review.
[15:27] <dimitern> frobware, np - well, without GW we've seen it breaks, right?
[15:28] <frobware> dimitern, fatally.
[15:29] <BrunoR> marcoceppi: is there something new regarding http://review.juju.solutions/review/2398 ?
[15:30] <dimitern> frobware, yeah.. I think at this point we only need to confirm if 1.8.3 vs 1.8.2 is broken "by default" when installed from scratch, and if it is, that settles it (and we have a simple workaround when it breaks - add GW)
[15:30] <dooferlad> dimitern/voidspace: having checked out github.com/juju/charm and switched to the unsable-v6 branch I tried to run the tests and get a bunch of errors. I have installed the dependencies. Thoughts?
[15:30] <dimitern> dooferlad, tests in juju/charm or in juju/juju ?
[15:31] <dooferlad> in juju/charm
[15:31] <dimitern> dooferlad, paste the errors?
[15:32] <dooferlad> dimitern: http://pastebin.ubuntu.com/
[15:32] <mgz> dooferlad: slight problem...
[15:33] <dooferlad> mgz: ?
[15:33] <dimitern> no ID :)
[15:33] <dooferlad> http://pastebin.ubuntu.com/14411294/
[15:33] <dooferlad> doh!
[15:34] <dooferlad> I think it may be that charm_test.go helpfully imports gopkg.in/juju/charm.v6-unstable
[15:34] <dooferlad> instead of, well, "charm"
[15:35] <dimitern> dooferlad, double-check the branch juju/charm is on is v6-unstable (commit ID 0229ee1
[15:36] <dooferlad> dimitern: I think it is all about the imports. I checked it out into github.com/juju/charm, not gopkg.in/juju/charm.v6-unstable
[15:37] <dimitern> dooferlad, hmm.. well gopkg.in/juju/charm.v6-unstable is backed by github.com/juju/charm @v6-unstable
[15:37] <dooferlad> dimitern: yep, I just find it strange to hack in a gopkg.in directory.
[15:40] <dimitern> dooferlad, yeah, quirks of gopath
[15:41] <dooferlad> dimitern: also a quirk of using package thing_test instead of package thing -- you have to import the thing you are testing instead of just being in it.
[15:41] <cherylj> frobware: could this lack of GW somehow also lead to the leaking of IPs?
[15:42] <cherylj> or maybe other things are weird with 1.8.3
[15:43] <dimitern> dooferlad, right; I've confirmed locally that even though I have e.g. github.com/juju/charm and gopkg.in/juju/charm.v6-unstable, the former's tip is an year older than the latter's tip
[15:44] <dimitern> I guess that's when we switched to gopkg.in import paths
[15:45] <voidspace> frobware: I can't look at the diff in reviewboard as it seems to conflict
[15:45] <voidspace> will try github
[16:09] <cherylj> frobware: can you ping me when you get a minute?
[16:09] <frobware> cherylj, ping
[16:09] <frobware> cherylj, have 20 mins before the OS call
[16:10] <natefinch> cherylj: you can ignore my request to edit that bug squad list page, I just had wanted to mark my bug as fixed, but I should have known you'd be on top of it. :)
[16:10] <cherylj> frobware: https://plus.google.com/hangouts/_/canonical.com/frobware-cherylj?authuser=0
[16:31] <dimitern> frobware, dooferlad, voidspace, OS call?
[16:33] <voidspace> dimitern: omw
[16:34] <voidspace> dimitern: frobware: dooferlad: if you get a chance please http://reviews.vapour.ws/r/3457/
[16:44] <katco> natefinch: ericsnow: forgot to mention, i have a doc. appt tomorrow morning overlapping with the standup. bumped it up a bit
[16:44] <ericsnow> katco: oh yeah, and I have a dentist appt. today :)
[16:45] <katco> ericsnow: calendar please :)
[16:45] <ericsnow> katco: added it yesterday :)
[16:45] <katco> ericsnow: cool ty
[16:47] <natefinch> katco: I got nothing all week :)
[16:47] <katco> natefinch: yay :D
[16:48] <marcoceppi> BrunoR: not at the moment, there are some oddities I'm looking into
[16:49] <marcoceppi> BrunoR: you may want to join #juju to continue discussion
[16:56] <dimitern> voidspace, looking
[16:57] <voidspace> dimitern: thanks
[16:58] <voidspace> dimitern: they're intended to be complete tests, so it might be worth reviewing apiserver/common/networkingcommon/spaces.go along with it
[16:58] <voidspace> it wasn't really reviewed the first time round
[16:59] <voidspace> that's *mostly* moved code though rather than new code
[17:01] <dimitern> voidspace, reviewed
[17:02] <voidspace> dimitern: thanks
[17:03] <voidspace> dimitern: cool
[17:03] <voidspace> good idea on the line wrapping, they're annoying
[17:03] <dimitern> voidspace, yeah, cheers
[17:31] <cherylj> hey ericsnow, I updated bug 1521777 with the behavior needed, and a link to the changes I had done so far
[17:31] <mup> Bug #1521777: Allow for upgrades to 2.0 <juju-core:In Progress by cherylj> <juju-core 1.25:Fix Committed by cherylj> <https://launchpad.net/bugs/1521777>
[17:31] <ericsnow> cherylj: thanks
[17:31] <cherylj> couldn't update the card, I got a permission error
[17:44] <katco> natefinch: reviewed your pr
[17:59] <frobware> cherylj, dimitern: I did a clean install of MAAS 1.8.2 and I get not default gateway for the discovered maas-eth0 interface, even when updating the Cluster controllers managed network.
[18:00] <frobware> s/and I get/and I do NOT get/
[18:05] <dimitern> frobware, hmm this means both 1.8.2 and 1.8.3 are consistently broken then
[18:08] <frobware> dimitern, I would have to try 1.9 to see if it is the same.
[18:08] <dimitern> .. unless the default GW is manually set
[18:08] <dimitern> frobware, +1
[18:08] <frobware> dimitern, we have not noticed because containers were using dhcp.
[18:08] <mgz> prettty sure 1.9 is set up with the bits as before
[18:09] <frobware> mgz, question is was the default gateway set manually?
[18:09] <mgz> perrito666: pr #3408 does not fill me with joy
[18:10] <mgz> both requiring provider api calls in an upgrade step and putting uuids in a string field seem horrible
[18:10] <perrito666> that is the number of github or rb?
[18:11] <mgz> perrito666: my bad, review ^, pr #3986
[18:11] <perrito666> oh, good because 3408 is quite nice and you where hurting my feelings
[18:11] <perrito666> mgz: well 2 things
[18:12] <perrito666> 1) google does not provide any other way, because of reasones
[18:12] <perrito666> 2) api calls in upgrade steps are about to become a thing, since we suddenly need to change stuff upstream to support big changes like multienv same name
[18:14] <mgz> perrito666: we don't really expect upgrade steps to fail
[18:14] <mgz> this absolutely can
[18:14] <perrito666> that said, I accept any form of criticism and will try to deal with it :)
[18:17] <perrito666> mm, we already have steps like these for storage, definitely something we will have to deal with
[18:17] <perrito666> ftr I think that not expecting anything to fail is wishful thinking on our part
[18:18] <mgz> meh, and I haven't got good answers on the naming that aren't major infra changes
[18:19] <mgz> perrito666: but line 65 in provider/openstack/upgrade.go is certainly a problem
[18:20] <mgz> you're not even logging/propogating the nova error
[18:21] <perrito666> mgz: what pr is this now?
[18:22] <mgz> the openstack sec group change
[18:24] <perrito666> mgz: line 65 is return errors.Annotatef(err, "upgrading security group name from %q to %q", group.Name, newName)
[18:25] <mgz> right
[18:25] <perrito666> aaand that err is the one that comes from calling nova?
[18:26] <mgz> provider returns an error, the step fails part way through upgrade and we lose the provider error
[18:27] <perrito666> that error bubles up all the way to the upgrade step, I feel like I am missing something here
[18:28] <mgz> sorry, we don't lose the error, but we don't complete the step
[18:31] <perrito666> well, I assumed that if the step failed and given that the identity of the upgradestep function allowed for an error that was the way to proceed: stop and err out
[18:33] <mgz> probably, but we don't retry steps till a new version so life seems complicated
[18:35] <perrito666> mmm, perhaps I should add some redundancy to my own step in that case
[18:35] <mgz> well, I'm not sure, I don't think individual steps should
[18:35] <mgz> but the framework above and below probably should
[18:36] <perrito666> you seem to be luring me into adding redundancy to upgrade steps
[18:36] <mgz> no, but I'm pointing out an issue with it given how things work at present
[18:37] <perrito666> sad, you could easily play the part of a british elf luring inocent devs into complex features :p
[18:54] <frobware> cherylj, switch the provisioner to use DHCP again on 1.25 works based on some very limited testing. and it works without having to worry about having a default gateway. We need to decide how we should proceed from here.
[18:55] <frobware> cherylj, (and some of that was almost a real sentence.) LOL
[18:56] <frobware> cherylj, I can continue with the move back to DHCP and fix the unit test failures but as we've seen most of the failures have come from live usage.
[19:00] <frobware> cherylj, I'm trying to avoid fixing the fix ... which is fixing another fix, and so on.
[19:15] <cherylj> frobware: I'm chatting with the maas guys now about the IPs leaking for devices... I can ask them about the default behavior with 1.8.2 and the default gateway
[19:16] <cherylj> frobware: If it's the case that 1.8.2 has the same behavior that by default, there's no default gateway, then I think we need to use the DHCP again.
[19:16] <cherylj> We can't make users specify a gateway to be able to use juju after an upgrade.
[19:16] <frobware> cherylj, playing devils ... OK
[19:17] <cherylj> does that sound reasonable?
[19:17] <frobware> cherylj, as an end-user, sure. :)
[19:17] <cherylj> frobware: haha, yeah :)
[19:18] <frobware> cherylj, I think we should consider landing my change anyway.
[19:18] <frobware> cherylj, and on top of that we switch back to DHCP. Thoughts?
[19:18] <cherylj> frobware: yeah, that was my assumption
[19:19] <perrito666> anyone very savvy on hooks willing to spare a few moments with me?
[19:34] <frobware> cherylj, I do get a default gw with 1.9
[19:35] <frobware> cherylj, that was noted on the back of a clean fresh 1.9 install
[19:35] <cherylj> frobware: ok, thanks for checking
[19:35] <cherylj> frobware: have you had the chance to try 1.8.2?
[19:36] <frobware> cherylj, yes. 1.8.2 no default GW. ditto for 1.8.3.
[19:36] <frobware> cherylj, my 1.9 was for completeness.
[19:36] <cherylj> ah, ok
[19:36] <frobware> cherylj, I vote for 1.9. :)
[19:36] <cherylj> has 1.9 even been released yet?
[19:36] <frobware> cherylj, I *think* so. I now have: 1.9.0+bzr4533-0ubuntu1
[19:37] <cherylj> ah, ok
[19:37] <frobware> cherylj, I previously had MAAS Version 1.9.0 (rc4+bzr4533)
[19:38] <frobware> cherylj, I need to EOD. Tomorrow I'll switch to DHCP, address the unit tests and perhaps we create another feature branch for testing.
[19:38] <cherylj> frobware: ok.  And just fyi - regarding the IP addresses leaking:
[19:38] <cherylj> mpontillo> cherylj: ah, in 1.8.3 it might be different - I was thinking 1.9. in 1.9 we changed the internals of how we associate IPs with devices, so it may work better
[19:39] <frobware> cherylj, do we know if it leaks with 1.8.2 as well?
[19:39] <cherylj> not yet
[19:39] <cherylj> the qa team hasn't had the chance to downgrade the MAAS
[19:40] <cherylj> frobware: if you switch to DHCP, would that eliminate using devices to register container IPs?
[19:40] <frobware> cherylj, need to double-check with dimiter
[19:42] <cherylj> frobware: ok, because if the device IPs are not released when the devices are deleted, we may need to back out that change.
[19:42] <frobware> cherylj, so the change works for 1.9, doesn't this then become a 1.8 issue?
[19:46] <frobware> cherylj, I'm now confused. Back out which change?  Because we want IPs to get cleaned up.
[19:46] <cherylj> frobware: technically, yes,  but we can't ship something that breaks people running on 1.8
[19:46] <cherylj> frobware: the change that registers containers as devices.
[19:46] <cherylj> frobware: if you need to EOD, we can continue this conversation tomorrow
[19:48]  * frobware mañana 
[19:50] <cherylj> jog, mgz, do you know if the 1.8.3 maas you guys are using has a static IP range defined?
[19:50] <jog> cherylj, see #maas... yes
[19:52] <katco> natefinch: did you see my review?
[19:53] <natefinch> katco: yes, was just about to ask you about it
[19:54] <natefinch> katco: are side comments a problem?  I actually find them to be easier to read because they don't get in the way of the code
[19:54] <katco> natefinch: not a problem, i just don't think they conform to go's standard practices
[19:54] <katco> natefinch: i don't think the tooling picks them up, only top comments
[19:55] <katco> natefinch: also the comments don't conform to the standard anyway (lead with variable name)
[19:55] <natefinch> katco: when you have multiple comments in a block like that, they get printed out enmass: https://godoc.org/github.com/natefinch/godocgo/sub#pkg-constants
[19:55] <katco> natefinch: if this weren't go, i'd say side-comments were the way to go here
[19:56] <natefinch> katco: also, pretty sure I've seen side comments on std lib constants in similar situations.... couldn't point you at it currently.
[19:56] <natefinch> katco: finally, the point of starting with the name of the thing is so grep will pick up the comment with what it's commenting, which will still work here, since it's on the same line
[20:00] <katco> natefinch: the two official sources i can find say to do doc comments: https://golang.org/doc/effective_go.html#commentary http://blog.golang.org/godoc-documenting-go-code
[20:00] <natefinch> katco: hmm, can't find an example of constants with side comments in the stdlib, only struct fields.    meh.  I can do top comments, I'm not married to it
[20:01] <katco> natefinch: fwiw i agree with what you're saying; but consistency > preference
[20:01] <natefinch> katco: yep, that's fair.
[20:03] <katco> natefinch: haven't looked; have you reviewed all of ericsnow's patches? i'm moving onto his now
[20:04] <ericsnow> katco: sorry (forgot), I've landed the first one and the second is merging
[20:04] <ericsnow> katco: working on the third now
[20:04] <katco> ericsnow: oh, did they already have ship-its?
[20:04] <ericsnow> katco: yep
[20:04] <katco> ericsnow: oh cool, so no review needed?
[20:04] <ericsnow> katco: perhaps on the remaining one of mine?
[20:04] <katco> ericsnow: sure thing
[20:05] <katco> ericsnow: listresources endpoint?
[20:08] <natefinch> katco: yes, I reviewed all his and gave them shipits pending addressing of review comments
[20:08] <katco> natefinch: cool, gj, ty
[20:10] <ericsnow> katco: yep
[20:12] <ericsnow> katco, natefinch: I also found an intermittent bug in the feature branch and now have a patch up:  http://reviews.vapour.ws/r/3458/
[20:12] <katco> ericsnow: awesome! good find :)
[20:13] <ericsnow> katco: it was annoying me :)
[20:17] <katco> ericsnow: natefinch: do you have time to hop into moonstone? i have a resources question
[20:17] <ericsnow> katco: sure
[20:19] <natefinch> katco: yep, one sec
[20:31] <natefinch> ericsnow, katco:  Dang, should have looked a little further down.  I still don't like using stub's CheckCallNames because it tests the order of the calls, which is an implementation detail that could change, causing the test to fail erroneously.  But that's more of a comment about how the stub itself works, not this change.
[20:32] <ericsnow> natefinch: checking the order of calls is exactly the point though
[20:33] <ericsnow> natefinch: we want to ensure that the internal boundary is used in the right order
[20:33] <natefinch> ericsnow: but it's not. it's testing like a dozen things that aren't actually required for the function to work correctly.
[20:34] <natefinch> ericsnow: I've had to change tests that shouldn't have failed because the order of function calls changed without the actual logic changing.
[20:35] <ericsnow> natefinch: I'm not clear on how the order of calls could change without the logic changing
[20:36] <natefinch> ericsnow: I forget the exact example, but it was basically two function calls that could have been done in any order, and switching them caused a test to fail.
[20:37] <natefinch> ericsnow: but in this case, we don't have a requirement that the files be uploaded in the same order as they are given on the command line.... except now there's a requirement enshrined in the test that says it must be so... so if we decide we want to upload the smallest one first, or upload them alphabetically or concurrently.. this test has to change.
[20:37] <ericsnow> natefinch: correct
[20:38] <katco> ericsnow: natefinch: the function calls would have to be associative (e.g. multiplication or something)
[20:38] <ericsnow> natefinch: it will reflect the interaction with the dependencies
[20:40] <natefinch> ericsnow: but it's not a requirement, and you've made it one. The only requirement is that tthe results of openresource get passed to upload
[20:42] <ericsnow> natefinch: the point of the patch is that the order *does* matter
[20:42] <natefinch> ericsnow: also, it's now a requirement that we create the client before opening the first resource... why?
[20:42] <natefinch> ericsnow: it makes the test care too much about very specific details of the order of calls in the implementation that it shouldn't care about.  It makes the tests fragile.
[20:44] <ericsnow> natefinch: testing the behavior of a function relative to its internal interface (dependencies) is just as important as relative to its external interface
[20:45] <ericsnow> natefinch: is your concern strictly with testing the order of interaction with dependencies?
[20:46] <ericsnow_afk> dentist
[20:48] <natefinch> ericsnow_afk: well, yes.  We should be checking that what is returned from OpenResource is sent to upload for the respective resource... and if somehow the code can go back in time and do the upload first, well, good for it ;)
[21:17] <natefinch> I wish I could go back and make my reviews into a "fix it, then ship it!"
[21:18] <natefinch> katco: btw, thanks for the review on my stuff
[21:25] <wallyworld> menn0: hi, welcome back. quick one - this bug has been fixed in recent jujus i think, right? ie we clean up orphaned txns bug 1528261
[21:25] <mup> Bug #1528261: jujud won't start, log shows a lot of "cannot set addresses of machine XXX: cannot find transaction ObjectIdHex("xxxxxx"); <juju-core:Incomplete> <juju-core 1.20:Triaged> <https://launchpad.net/bugs/1528261>
[21:26] <menn0> wallyworld: hi! looking
[21:27] <menn0> wallyworld: there's a few things with that
[21:27] <menn0> wallyworld: all the unnecessary addressupdater txns have been fixed
[21:28] <menn0> wallyworld: and we do clean up the txn collection
[21:28] <wallyworld> menn0: the issue arrose when mongo got its replicaset all screwed up
[21:28] <menn0> wallyworld: but the pancis in the ticket aren't fixed automatically in newer juju verisons
[21:28] <wallyworld> ok, at least we're part way there
[21:29] <menn0> wallyworld: someone will need to shut down the state servers, get the replicaset working again and then run mgopurge to fix the txns
[21:29] <wallyworld> ok, ty
[21:30] <wallyworld> niedbalski: ^^^^^^^
[21:30] <wallyworld> i'm not sure of mario's nic to tell him also
[21:30] <niedbalski> Mmike, ^^
[21:31] <Mmike> thxn!
[21:31] <wallyworld> Mmike: niedbalski: so an upgrade to 1.24 or 1.25 is required plus the mgopurge
[21:32] <Mmike> how do I upgrade when I can't get jujud started?
[21:33] <wallyworld> i think running mgopurge first will help, well hopefully?
[21:33] <wallyworld> so shut down state servers, fix replicaset, run mgopurge
[21:33] <menn0> Mmike, niedbalski, wallyworld: the upgrade to 1.24 isn't a strict requirement to get this env working again. It'll just help reduce the chance of it happening again.
[21:33] <wallyworld> yes
[21:34] <Mmike> ack
[21:34] <Mmike> where do I find mgopurge?
[21:34] <menn0> Mmike: I'll compile it for you... it's not easily available
[21:35] <waigani> axw or menn0: this needs a review when you have a moment please: http://reviews.vapour.ws/r/3422
[21:35] <wallyworld> menn0: we should consider packaging that binary with juju, like we do for plugins
[21:36] <Mmike> menn0, thnx!
[21:37] <Mmike> menn0, no rush, it's almost 11PM here and I'll doze off shortly
[21:37] <menn0> Mmike: ok. how shall I get it to you?
[21:38] <menn0> wallyworld: that could work.
[21:39] <Mmike> menn0, how large is it? Email, if it's not largish, or put it to mombin somewhere and allow me to access it? (mario is the username)
[21:39] <wallyworld> menn0: yeah, would be handy as i suspect we'll need it again
[21:39] <menn0> Mmike: it's not too big
[21:39] <Mmike> then email could do it
[21:41] <Mmike> wallyworld, menn0, thnx, lads!
[21:41] <Mmike> I'm dozing off now.
[21:42] <wallyworld> Mmike: np, menn0 did all the work :-)
[21:42] <Mmike> wallyworld, you did the connecting :) If you were in sales... :)
[21:42] <wallyworld> lol
[22:31] <perrito666> wallyworld: I might be a couple of mins late for the standup, grocery shopping
[22:31] <wallyworld> sure, np
[23:20] <perrito666> wallyworld: axw alexisb ready
[23:20] <alexisb> wallyworld, I am running late
[23:21] <wallyworld> np
[23:21] <wallyworld> perrito666: we are here already