[00:00] <menn0> jw4: i will keep digging now
[00:00] <wallyworld> ericsnow: why the dial changes?
[00:00] <jw4> menn0: I'll be in and out for the next few hours... making supper for kids, and then school open house - I'll try to get on longer later
[00:00] <menn0> jw4: no worries
[00:04] <ericsnow> wallyworld: in 1.2 http.Transport doesn't have the KeepAlive field, so I had to manually create the same effect
[00:05] <wallyworld> ok
[00:09] <ericsnow> wallyworld: thanks for the review :)
[00:09] <wallyworld> np
[00:19] <mup> Bug #1447853 was opened: Local charms are not added to storage on upgrade to 1.22.x <charms> <regression> <storage> <upgrade-juju> <juju-core:Triaged> <https://launchpad.net/bugs/1447853>
[00:23] <wallyworld> menn0: fixes pushed when you get a chance
[00:23] <menn0> wallyworld: kk
[00:26] <menn0> wallyworld: ship it
[00:26] <wallyworld> \o/
[00:26] <wallyworld> ty
[00:31] <mup> Bug #1447853 changed: Local charms are not added to storage on upgrade to 1.22.x <charms> <regression> <storage> <upgrade-juju> <juju-core:Triaged> <https://launchpad.net/bugs/1447853>
[00:40] <mup> Bug #1447853 was opened: Local charms are not added to storage on upgrade to 1.22.x <charms> <regression> <storage> <upgrade-juju> <juju-core:Triaged> <https://launchpad.net/bugs/1447853>
[00:52] <sinzui> wallyworld, I am exhausted. I haven't found much. I split bug 1447846 from the other upgrade issue and captured some unit log
[00:52] <mup> Bug #1447846: Hooks don't fire after upgrade 1.23.0 <hooks> <regression> <upgrade-juju> <juju-core:Triaged by menno.smits> <juju-core 1.23:In Progress by menno.smits> <https://launchpad.net/bugs/1447846>
[00:53] <wallyworld> sinzui: ty, menn0 is working that bug at the moment
[00:53] <sinzui> wallyworld, mgz and I also discovered bug 1447853, but I know have a cheep work around
[00:53] <mup> Bug #1447853: Local charms are not added to storage on upgrade to 1.22.x <charms> <regression> <storage> <upgrade-juju> <juju-core:Triaged> <https://launchpad.net/bugs/1447853>
[00:53] <wallyworld> ah bollocks, saw that float past
[00:54] <wallyworld> will have to fix for 1.24
[00:54] <sinzui> wallyworld, the work around is to upgrade-charm --force. it works a minute later :)
[00:54] <wallyworld> at least there is that
[02:01] <ericsnow> anyone have a minute to review a patch for a very mechanical change: https://github.com/juju/govmomi/pull/2
[02:30] <menn0> fwereade: ping?
[02:34] <menn0> jam: ping?
[03:00] <menn0> wallyworld: i have a learned a lot more about the hook not firing bug
[03:00] <menn0> wallyworld: not sure I know how to fix it though
[03:00] <wallyworld> oh?
[03:01] <wallyworld> did you want to talk?
[03:01] <menn0> wallyworld: having jam or fwereade around would be handy. it's leader related.
[03:01] <wallyworld> oh joy
[03:01] <wallyworld> jam will be online soonish
[03:01] <menn0> wallyworld: quick chat would be good
[03:01] <wallyworld> ok, same hangout
[03:02] <menn0> wallyworld: i'm there now
[03:25] <wallyworld> menn0: if you get a chance, but doesn't matter if not http://reviews.vapour.ws/r/1481/
[03:26] <menn0> wallyworld: got it
[03:40] <jw4> menn0: does that mean it's not related to my changes?
[03:41] <menn0> jw4: at this stage it's looking like the hooks not firing issue is related to the leadership work
[03:41] <menn0> jw4: i'm not any wiser about that mode error in the logs though
[03:42] <jw4> menn0: well I know about the error - that's not critical
[03:42] <jw4> menn0: cool, I'll try and dig into it too
[03:42] <jw4> for my edification
[03:43] <menn0> we just had an earthquake, 6.4 but very deep
[03:44] <jw4> whoa
[03:44] <menn0> because of the depth it wasn't that severe
[03:44] <jw4> menn0: any damage
[03:44] <menn0> but my chair was rolling around by itself
[03:45] <menn0> no damage i think but it was a strange rolling feeling and went on for a while
[03:45] <menn0> i've only experienced short "bangs" up until now
[03:45] <anastasiamac> menn0: is south island sinking?
[03:45] <jw4> that's freaky... are earthquakes common in that area? Pacific Rim and all
[03:45] <menn0> yep, quite common in NZ
[03:48] <jw4> anastasiamac: he's ignoring the question... I think he's rescuing pets or something
[03:49] <menn0> anastasiamac: probably :)
[03:50] <jw4> quite a bit of activity recently: http://www.geonet.org.nz/quakes/felt
[03:51] <anastasiamac> jw4: :D
[03:51] <jw4> :)
[03:51] <anastasiamac> menn0: u should have apost-earthquake party, considering it's friday
[03:56] <menn0> anastasiamac: they're a little too common to be having parties for each one :)
[03:57] <anastasiamac> menn0: on a serious note, it's gr8 that it was deep. 6.4 sounds really scary... :))
[03:59] <menn0> anastasiamac: yes. this one was the same magnitude as the one that did this a few years ago: http://en.wikipedia.org/wiki/2011_Christchurch_earthquake
[04:00] <menn0> the city is still a long way from recovering from that one
[04:09] <wallyworld> menn0: the earth moved for you in cape town as well didn't it?
[04:10] <menn0> wallyworld: ssh not in public :)
[04:11] <mup> Bug #1447895 was opened: Panic if jujud restarts while action is running <actions> <juju-core:New> <https://launchpad.net/bugs/1447895>
[04:11] <mup> Bug #1447899 was opened: upgrade fails if no explicit version is specified <upgrade-juju> <juju-core:Triaged> <https://launchpad.net/bugs/1447899>
[04:28] <jw4> nice sleuthing menn0
[04:28] <menn0> jw4: with the hook not firing bug?
[04:28] <jw4> yeah
[04:29] <jw4> finding the WatchLeadershipSettings missing document
[04:50] <menn0> wallyworld, jw4: bingo!
[04:50] <menn0> wallyworld, jw4: I have a fix for the hook not firing issue
[04:51] <menn0> wallyworld, jw4: adding an upgrade step that adds the leadership documents in the settings collection fixes the issue
[04:51] <menn0> wallyworld, jw4: I hacked it in without tests so let me sort that out before proposing
[04:52] <jw4> menn0: suh-weet!
[04:59] <menn0> wallyworld, jw4: initial, rough fix for hooks not firing issue
[04:59] <menn0> https://github.com/mjs/juju/tree/1447846-hooks-dont-fire-after-upgrade-1.23
[04:59] <menn0> writing the tests now
[05:04] <jw4> other than missing docstring on https://github.com/juju/juju/compare/1.23...mjs:1447846-hooks-dont-fire-after-upgrade-1.23#diff-d7b2b2c8e8ce6dfc1b7f09c3cf9744d1R1203
[05:04] <jw4> and forthcoming tests...
[05:04] <jw4> looks great menn0
[05:13] <wallyworld> menn0: branch looks good, still want to understand why listener hangs if there's no record. i think that's poor behaviour
[05:14] <jw4> wallyworld: I bet it's because the watcher isn't returning an initial empty event, like it's supposed to
[05:15] <jw4> wallyworld, menn0 it was a little tricky getting that initial guaranteed event in the Action Watchers, so I wouldn't be surprised if the Leadership Watchers have a problem there too
[05:15] <menn0> jw4: I think the problem here is that the code assumes the leadership doc will be there b/c it is for services created under 1.23
[05:16] <menn0> jw4: but someone forgot to write this upgrade step
[05:16] <jw4> the Watchers (afair) are supposed to guarantee an initial event, possibly empty, even if there are no records...
[05:16] <menn0> jw4: and without the doc there when the watcher is created the initial event isn't fired
[05:16] <menn0> jw4: are you sure about that?
[05:16] <jw4> menn0: yeah, I think you're right, but I remember fwereade stressing to me the importance of that... (but I could be remembering wrong)
[05:17] <jw4> (the importance of the initial, possibly empty, guaranteed event)
[05:17] <menn0> jw4: yeah, I know there's been talk about this but I'm not exactly sure what should happen
[05:18] <jw4> hopefully fwereade will be on soon and will chime in... I guess it's EOD for you though now (and EOW)
[05:19] <menn0> jw4: i've got a bit more time left
[05:19] <menn0> jw4: i'm going to try and get this fix done as much as I can
[05:20] <menn0> jw4: regardless of what the watchers are supposed to do I'm pretty sure we want this upgrade step
[05:20] <jw4> menn0: regardless, I think your fix is appropriate in this case
[05:20] <wallyworld> jw4: exactly, but to me that's not how i thought the watchers worked
[05:21] <wallyworld> +1
[05:21] <jw4> wallyworld: yeah, I'm fairly confident that's how I was instructed to implement the actions related watchers, but we should verify
[05:32]  * jw4 off to bed
[05:51] <menn0> wallyworld: ping
[05:52] <wallyworld> yo
[05:52] <menn0> wallyworld: i have to EOD very soon
[05:52] <wallyworld> one sec
[05:54] <wallyworld> back
[05:54] <menn0> wallyworld: just need to hand over the fix for this
[05:54] <menn0> wallyworld: pushing now...
[05:54] <menn0> wallyworld: give me a sec
[05:54] <wallyworld> sure, is it ready to land after review?
[05:54] <wallyworld> or tests needed?
[05:55] <menn0> wallyworld: the hard tests are done
[05:55] <menn0> wallyworld: just needs a test for upgrade step idempotency
[05:56] <wallyworld> ok
[05:58] <menn0> it's here: https://github.com/mjs/juju/tree/1447846-hooks-dont-fire-after-upgrade-1.23
[05:58] <wallyworld> menn0: i'll either do it or hand over; it's my son's birthday today and i have to go to dinner soonish. either way will be done. thanks for fixing
[05:58] <menn0> it's ready to go apart from that one test
[05:59] <menn0> if no-one gets it done today it'll take me almost no time on Monday
[05:59] <menn0> wallyworld: and we need to forward port to master
[05:59] <wallyworld> ok, np. tbh, i don't think we are cutting a release on friday anyway
[06:00] <wallyworld> but it's now done so the risk is removed
[06:02] <menn0> wallyworld: ok good
[06:02] <menn0> wallyworld: well have a good night and a weekend
[06:02] <menn0> speak to you next week
[06:54] <wallyworld> fwereade: you online?
[10:17] <wallyworld> fwereade: hiya
[10:28] <wallyworld> TheMue: hi there, you ocr? would love a review of this so as it needs to land for 1.24 http://reviews.vapour.ws/r/1481/ it removes a feature flag
[11:13] <wwitzel3> jam: ping
[11:27] <mattyw> wwitzel3, ping?
[11:30] <wwitzel3> mattyw: pong
[11:40] <fwereade> wallyworld_, so do we still serve client facade v0? I'm not seeing where we do
[11:40] <fwereade> wallyworld_, and that would STM to be a problem...
[11:41] <wallyworld_> fwereade: ah, right, i wasn't sure about that. i wasn't sure what the exact mechanism was there
[11:42] <fwereade> wallyworld_, the ideal, I think, is to *never touch* the client API, and carve off changing functionality into more service-specific facades
[11:42] <fwereade> wallyworld_, otherwise you have to have an almost-perfectly-cloned Client facade
[11:43] <fwereade> wallyworld_, then when (say) storage evolves, it shouldn't need to hit more than one or two small facades
[11:43] <fwereade> wallyworld_, rather than dragging the Client mess into the equation every time
[11:44] <fwereade> wallyworld_, what actually changed on Client?
[11:44] <wallyworld_> fwereade: yeah sadly i think you are right, i'll have to do that, but will miss the branching of 1.24. bollocks. will have to forward port
[11:44] <fwereade> wallyworld_, bad luck :(
[11:44] <wallyworld_> fwereade: new params to 2 apis were added
[11:45] <wallyworld_> fwereade: if a newer client adds additonal params to the call, they will be ignored, but we need a way to tell the user
[11:45] <wallyworld_> hence the api bump
[11:45] <fwereade> wallyworld_, yeah, indeed, it really is a different API
[11:46] <fwereade> wallyworld_, so I *think* that the right thing to do
[11:46] <fwereade> wallyworld_, is to pull out a Service service facade from client
[11:46] <wallyworld_> yeah, i suspect so too
[11:46] <wallyworld_> i'll revisit over the weekend
[11:47] <wallyworld_> i really want to remove the flag for 1.24
[12:35] <perrito666> morning
[12:44] <mgz> perrito666: wala
[12:45] <redelmann> perrito666, ¿good? morning
[14:02] <katco> wwitzel3: ericsnow: stand up
[15:56] <alexisb> all, looking at this: https://bugs.launchpad.net/juju-core/+bug/1447846
[15:56] <mup> Bug #1447846: Hooks don't fire after upgrade 1.23.0 <hooks> <regression> <upgrade-juju> <juju-core:Triaged by menno.smits> <juju-core 1.23:In Progress by menno.smits> <https://launchpad.net/bugs/1447846>
[15:57] <alexisb> ^^^ based on menno's last comment is this something someone can pick up and complete today?
[15:57] <jw4> alexisb: menn0 said it's almost done.. just one test for idempotency left
[15:58] <katco> alexisb: yeah looks doable for someone to pick up
[15:58] <alexisb> katco, can you please delegate
[15:58] <katco> alexisb: sure
[15:58] <alexisb> thanks
[15:59] <katco> moonstone are all currently working on bugs. any volunteers?
[16:03] <katco> cherylj: perrito666: do you two have capacity to land a simple bug?
[16:07] <perrito666> katco: sorry I dont feel really reliable right now.
[17:04] <katco> perrito666: no worries. feel better soon
[17:04] <katco> cherylj: sorry i lost connection. can you take that bug?
[17:23] <natefinch> man I really hate that we require go 1.2 for juju, it totally screws up my development environment for everything else... and many projects already require 1.3+
[17:26] <katco> natefinch: here's my setup
[17:26] <katco> natefinch: ~/.local/go-1.{2,3,4}
[17:26] <katco> natefinch: ~/.local/go -> go-1.2 during day
[17:26] <katco> natefinch: export GOPATH=~/.local/go
[17:26] <katco> natefinch: there is some churn with having to rebuild utils
[17:27] <ericsnow> hmmm...virtualenv for go...
[17:27] <katco> ericsnow: or there's that :)
[17:29] <natefinch> ericsnow: there's a tool for that... forget which of the myriad tools out there it is.  of  course, as katco said, you can always just run them side by side.
[17:32] <katco> natefinch: wwitzel3: planning time!
[17:32] <wwitzel3> I'm there?
[17:33] <wwitzel3> is it not in moonstone? .. (checks calendar)
[17:34] <natefinch> wwitzel3: not moonstone
[17:50] <wwitzel3> cmars: http://reviews.vapour.ws/r/1484/
[17:55] <cmars> wwitzel3, so i guess CI is going to set that feature flag to get that test to pass? what's the difference between not registering the provider vs. blocking bootstrap?
[17:57] <wwitzel3> cmars: the code on the unit doesn't see the provider as registered since the flag is local
[17:57] <wwitzel3> cmars: jam pointed this out no the ML and the QA ran in to it with CI tests
[17:58] <aznashwan> hey; could I please get a quick review on http://reviews.vapour.ws/r/1486/
[17:58] <cmars> wwitzel3, ok, got it. so this moves the featureflag check to the client? in what context does environ.Bootstrap run?
[17:58] <cmars> (environ's not an area of the codebase I'm familiar with)
[17:58] <aznashwan> it's a simple module; ~100 lines of code that actually does something
[18:03] <wwitzel3> cmars: just during juju bootstrap
[18:34] <natefinch> cmars: quick review?  http://reviews.vapour.ws/r/1487/diff/#
[18:40] <wwitzel3> cmars: thanks for the review
[19:53] <perrito666>  /query alexisb
[19:53] <perrito666> well, another bug in my irc client
[19:53] <perrito666> I should stop using daily snapshots
[20:24] <natefinch> sometimes I think Juju is the biggest example of what not to do with interfaces :/
[20:25] <perrito666> you exagerate
[20:25] <natefinch> perhaps slightly
[20:25] <natefinch> there might be a few worse examples in the world ;)
[20:27] <natefinch> I like the way everything is a nice generic interface and then in the comments we just say "this is for precise0
[20:27] <natefinch> s/0/"
[20:34] <perrito666> natefinch: I think that is an example on how not to used a typed language
[21:10] <mgz> anyone still around for trivial reviews?
[21:16] <mup> Bug #1448308 was opened: Skipped TestUniterUpgradeConflicts on ppc64 <skipped-test> <test-failure> <juju-core:Triaged> <https://launchpad.net/bugs/1448308>
[21:21] <ericsnow> mgz: sorry I missed that format stuff :/
[21:22] <ericsnow> anyone around that feels comfortable reviewing http://reviews.vapour.ws/r/1490/
[21:23] <ericsnow> it fixes the "hooks not firing" bug
[21:23] <ericsnow> #1447846
[21:23] <mup> Bug #1447846: Hooks don't fire after upgrade 1.23.0 <hooks> <regression> <upgrade-juju> <juju-core:Triaged by menno.smits> <juju-core 1.23:In Progress by ericsnowcurrently> <https://launchpad.net/bugs/1447846>
[21:23] <mgz> ericsnow: I can look
[21:23] <ericsnow> mgz: oh, cool
[21:24] <mgz> ericsnow: I also have a trivial ppc64 test skip branch
[21:24] <ericsnow> mgz: note that I only added the idempotency test, menn0 did the rest
[21:24] <mgz> /1488
[21:24] <ericsnow> mgz: I'll take a look
[21:24] <mgz> ericsnow: yeah, I already looked over menno's branch
[21:24] <ericsnow> mgz: nice
[21:25] <ericsnow> mgz: I'm pretty sure I got the idempotency check right
[21:27] <ericsnow> mgz: in the bug you make it sound like skipping the test might be masking a real bug
[21:28] <mgz> ericsnow: it might be, but we hav ehte bug, and know we need to do functional testing of the leadership stuff
[21:28] <ericsnow> mgz: k
[21:29] <ericsnow> mgz: so 1448308 will be closed only as soon as we resolve the issue (and not when the skip-the-test patch merges), right?
[21:29] <mgz> ericsnow: how-upgrades-work question
[21:29] <ericsnow> mgz: sure
[21:30] <mgz> ericsnow: indeed, I'm keeping the skipped-tests bugs open, because we keep filing bugs and linking them in the codebase, but counting them as fixed when the test is skipped
[21:30] <mgz> which is wrong...
[21:31] <ericsnow> mgz: cool, then LGTM
[21:31] <mgz> I'm on 1.23.0 which did not do this step, my state server notices there's a 1.23.2 with this fix and downloads it... how does it know to run this step?
[21:32] <mgz> we just always do steps on new versions and the steps must apply?
[21:32] <mgz> even minor ones, right?
[21:33] <ericsnow> mgz: I'm not sure, but...the idempotency invariant of upgrade steps ensures it's a noop if you're already okay
[21:33] <mgz> right
[21:34] <ericsnow> I expect it simply runs all the steps
[21:37] <mgz> ericsnow: test changes look good
[21:37] <ericsnow> mgz: k, cool
[21:38] <ericsnow> mgz: it made sense to factor out that stuff into helpers :)
[21:38] <mgz> I'd be somewhat interested in a test that ran the step on a as-if just bootstrapped system, no services
[21:39] <mgz> we don't try and do all error paths in these things, but that one does seem relevent
[21:41] <ericsnow> mgz: so basically the same thing as the existing test but without adding any services first?
[21:42] <mgz> (it should just return success without running any ops basically)
[21:42] <mgz> ericsnow: yup
[21:42] <ericsnow> mgz: I'll add such a test!
[21:42] <mgz> ericsnow: lgtm
[21:43] <mgz> hm, can I shipit and leave a comment in the same thing? not sure on the magic
[21:43] <mgz> I guess I jsut include that string in my text huh.
[21:58] <ericsnow> mgz: so is it correct that CI covers the upgrade scenarios now (i.e. the bug trigger will get exercised)?
[21:59] <mgz> ericsnow: see all the failures on upgrade jobs on 1.23 and trunk :)
[22:00] <ericsnow> mgz: that's what I figured :)
[22:00] <ericsnow> mgz: thanks for the review
[22:00] <mgz> http://reports.vapour.ws/releases/2557
[22:30] <mgz> "panic: rescanned document misses transaction in queue" ...that one's new to me
[22:43] <ericsnow> mgz: yeah, me too :\
[22:47] <ericsnow> mgz: hmmm...see #1318366
[22:47] <mup> Bug #1318366: jujud on state server panic misses transaction in queue <cloud-installer> <landscape> <orange-box> <panic> <performance> <sm15k> <juju-core:Fix Released by menno.smits> <juju-core 1.20:Fix Released by menno.smits> <juju-core (Ubuntu):Fix Released> <https://launchpad.net/bugs/1318366>
[22:48] <fwereade> ericsnow, menn0, dammit, sorry, thanks
[23:00] <ericsnow> mgz: I'm going to ask menn0 to double check the upgrade step and tests when he gets a chance; that failure is a little suspicious.
[23:06] <mgz> ericsnow: seems wise, we're not in a rush now
[23:07] <ericsnow> mgz: yep :)