[00:03] <thumper> menn0: are you getting time to finish off https://github.com/juju/juju/pull/2724?
[00:03] <menn0> thumper: working on that right now
[00:03] <menn0> thumper: not far
[00:03] <thumper> cool
[00:11] <menn0> thumper: done, about to merge
[01:26] <perrito666> mehh I was reviewing a pr and when I got to the end it got discarded
[01:27] <anastasiamac> perrito666: :(
[01:27] <anastasiamac> perrito666: what time is it for u?
[01:30] <thumper> wallyworld: I don't suppose someone on your team can fix the intermittent failure in uniter worker?
[01:30] <thumper> FAIL: uniter_test.go:892: UniterSuite.TestUniterUpgradeConflicts
[01:30] <thumper> happens relatively regularly
[01:31] <perrito666> anastasiamac: 22:30
[02:13] <wwitzel3> axw: ping
[02:15] <wallyworld> thumper: sorry, missed ping about failing test. will take a look. currently full with WIP fixes for 1.25 release and soon a critical customer issue which is more a feature thana bug
[02:15] <wallyworld> i suspect will be end of week or early next
[02:15] <wallyworld> also working on arm issue for 1.24.3
[03:07] <menn0> wallyworld, thumper: the mongodb timeout PR has landed in 1.24
[03:07] <wallyworld> great
[03:08] <wallyworld> do we know who to prod to see if it helps with anything?
[03:09] <menn0> wallyworld: ahasenack reported the bug that lead to me doing this work (i'm not sure if it will help or not)}
[03:10] <wallyworld> ok
[03:10] <menn0> thumper also thinks it might help with a problem env he was looking at
[03:10] <wallyworld> time will tell i guess
[03:27] <menn0> wallyworld: so what is the github.com/juju/juju/juju package all about?
[03:30] <menn0> seems like stuff that I would have expected to be in the state package
[03:41] <wallyworld> menn0: that package is sort of an attempt to get stuff out of state as far as i understand
[03:42] <wallyworld> it is horribly named
[03:44] <menn0> wallyworld: so state helpers then?
[03:45] <cherylj_> wwitzel3: ping?
[03:45] <wallyworld> menn0: more like juju core business logic
[03:45] <wallyworld> that is not persistence related
[03:45] <menn0> hmm ok
[03:45] <wallyworld> i didn't add it :-)
[03:46] <thumper> menn0: cool
[03:48] <menn0> wallyworld: I didn't think you did
[03:48] <wallyworld> i can't defend it too hard :-)
[03:53] <menn0> wallyworld: review done
[03:54] <wallyworld> ty menn0
[04:02] <wallyworld> thumper: have you picked up much about juju and arm via osmosis?
[04:02] <thumper> nope
[04:02] <wallyworld> and dave is away this week :-(
[04:08] <thumper> aye
[05:25] <anastasiamac> thumper: u got a fish trophy!
[05:56] <wallyworld> jam: hi
[05:57] <jam> hey wallyworld
[05:57] <wallyworld> jam: network related question if you have a moment
[05:57] <wallyworld> bug 1472014
[05:57] <mup> Bug #1472014: juju 1.24.0: wget cert issues causing failure to create containers on 14.04.2 with lxc 1.07 <openstack-installer> <juju-core:Triaged> <juju-core 1.24:Triaged> <https://launchpad.net/bugs/1472014>
[05:57] <wallyworld> see the last couple of comments
[05:58] <wallyworld> it seems we don't store / report all the cloud local addresses for a machine
[05:58] <wallyworld> so a machine's AddressWatcher doesn't get told about all possible addresses an https request can arrive on
[05:59] <wallyworld> do you know why we throw away some cloud local addresses?
[05:59] <wallyworld> i'm not looked at the code in detail, just going by jame's comment
[05:59] <wallyworld> but i'm wary about changing network related code
[06:00] <wallyworld> as it has the habit of breaking things
[06:02] <jam> wallyworld: so offhand I'd say we don't actually cope with having multiple addresses where things could arrive
[06:02] <jam> wallyworld: consider charms, they can only really report 1 private address to eachother
[06:02] <wallyworld> hmmm, so which one to pick then
[06:02] <jam> wallyworld: so the issue is probably that we are thinking 10.0.6.* is the right address, when really the correct cloud-local address is the 10.0.3* one
[06:02] <wallyworld> as we are picking the wrong one
[06:03] <wallyworld> hmmm, so how to pick the right one
[06:04] <jam> wallyworld: so that machine has 3 addresses that I would consider "cloud-local" sort of addresess, a 10.0.3 a 10.0.6 and a 192.168
[06:04] <wallyworld> for this purpose we should just stick everything in the SAN
[06:04] <jam> wallyworld: Surprisingly (for me) 10.0.3 is usually the LXC bridge (I thought)
[06:05] <wallyworld> jam: what about this line : setting API hostPorts
[06:05] <jam> wallyworld: I think in the  idealized model Juju would be aware of all the subnets and have labled names for them (spaces), in which case it would know that machine X is supposed to talk to machine Y on a given address.
[06:05] <wallyworld> it seems there we pass in everything
[06:05] <jam> wallyworld: internally it feels like we should be aware and save all the addresses
[06:05] <wallyworld> yes, save them all internally sounds good to me too, but if we do that now stuff would break i would tink
[06:06] <jam> wallyworld: short term, I think just adding all addresses to the SAN is fine.
[06:06] <wallyworld> but
[06:06] <wallyworld> that relies on AddressWatcher :(
[06:06] <wallyworld> so i'll need to change how it all works
[06:06] <wallyworld> bollocks
[06:06] <wallyworld> i'll go read the code and see what can be done
[06:06] <jam> wallyworld: well, I think you can certainly get help from Sapphire on this one.
[06:07] <wallyworld> jam: i asked but none were 100% sure about why only one address was saved etc
[06:07] <wallyworld> so maybe there's not the level of knowledge there to dive right in
[06:07] <jam> wallyworld: well, dimitern is away and the others probably not quite as familiar
[06:08] <jam> gophercon being this week.
[06:08] <wallyworld> yeah, that's what i figured, hence asking you :-)
[06:08] <wallyworld> i'll see if it's possible to tinker with the cert updater
[06:08] <wallyworld> i could re-read all machine addresses
[06:08] <wallyworld> but may not be triggered
[08:09] <dooferlad> wallyworld, jam: just read the backlog. Yell if you want a networking hand.
[08:10] <dooferlad> wallyworld: also with ARM, I am an ex-ARM employee so if I can help with that please call on me
[08:41] <wallyworld> dooferlad: oh, i might take you up on that arm offer. i might ping you after dinner
[09:02] <dooferlad> jam, fwereade: hangout?
[09:30] <voidspace> fwereade: git blame -L302,302 provider/ec2/config_test.go
[09:30] <voidspace> fwereade: that's the test that fails for me on master
[09:31] <voidspace> fwereade: git blame may be deceived of course...
[11:15] <mattyw> fwereade, quick ping?
[11:15] <fwereade> mattyw, heyhey
[11:16] <mattyw> fwereade, is there any doc or something about the uniter operation/ callbacks arch? I'm finding myself getting in to it and was hoping I could make some decisions about my stuff without having to hassle you
[11:16] <fwereade> mattyw, only what's inline, I'm afraid
[11:17] <mattyw> fwereade, I probably only want to call a certain function when a certain hook has finished
[11:17] <fwereade> mattyw, that sounds like the responsibility of the CommitHook bit to me?
[11:18] <fwereade> mattyw, but the callbacks themselves are basically evil
[11:18] <mattyw> fwereade, time for a 5 minute hangout?
[11:18] <fwereade> mattyw, it's basically just a cut-down uniter facade/adapter for the use of the ops
[11:18] <mattyw> fwereade, I'll try to timebox it at that
[11:18] <fwereade> mattyw, sure, start one please?
[12:36] <wwitzel3> ericsnow: ping
[12:52] <mup> Bug #1472596 opened: bootstrap failed yet retry says it succeeded <juju-core:New> <https://launchpad.net/bugs/1472596>
[13:42] <mgz> bogdanteleaga: is rr 2107 live or not atm?
[13:49] <bogdanteleaga> mgz: no it's not
[13:49] <bogdanteleaga> mgz: it's more of a weird interaction, 2109 is the same but should show a better diff
[13:49] <bogdanteleaga> mgz: any ideas if I can delete that one?
[13:52] <mgz> bogdanteleaga: it is marked as discarded, so that's probably fine, is just getting updated still I guess as it's the same github branch
[13:53] <bogdanteleaga> mgz: yeah, that was the one submitted to github, but the diff would be a bit funky since it contains another branch
[13:56] <mgz> bogdanteleaga: do you need anything else on those branches, or are you good to go?
[13:56] <mgz> bogdanteleaga: I think we'll want to backport to 1.24 after master has blessed the change
[13:59] <bogdanteleaga> mgz: no, I was doing some final tests a couple of hours ago, but everything seems fine
[13:59] <bogdanteleaga> mgz: got caught up with something else
[14:00] <mgz> bogdanteleaga: no problem
[14:01] <bogdanteleaga> mgz: squashing now and I'll start merging
[14:01]  * bogdanteleaga grabs popcorn
[14:01] <mgz> :)
[14:19] <wwitzel3> ericsnow: ping
[14:20] <ericsnow> wwitzel3: hey hey hey
[14:24] <wwitzel3> ericsnow: trying to work through an issue and I've run in to some code that I could use some help deubgging
[14:24] <ericsnow> wwitzel3: sure
[14:25] <ericsnow> wwitzel3: moonstone?
[14:25] <wwitzel3> ericsnow: we can go to a query, conference wifi probably won't work so well with a hangout
[14:25] <ericsnow> wwitzel3: right :)
[14:35] <bogdanteleaga> mgz: you said they ought to be backported?
[14:35] <mgz> bogdanteleaga: I think we do need it on 1.24, yeah
[14:35] <mgz> want to see it work on master first of course
[14:39] <bogdanteleaga> mgz: Build failed: Does not match ['fixes-1472632']
[14:39] <bogdanteleaga> can't find a bug with that number
[14:40] <mgz> abentley: ^did you mean to make that bug a blocker and private?
[14:40] <abentley> mgz: Yes.
[14:40] <mgz> it does not really strike me as either
[14:40] <abentley> mgz: Regressions are blockers.
[14:40] <abentley> mgz: It has debug logs from SSH.
[14:42] <mgz> I am reading the debug ssh log and apart from containing your name and some of juju-ci's ips addresses it seems to have nothing personal
[14:42] <mgz> and I don't see how this bug prevents us releasing, which is the point of blockers
[14:43] <mgz> we've been releasing fine with this for three 1.24-s
[14:43] <abentley> mgz: Well, I was erring on the side of caution with the SSH.  If you're willing to take responsibility for making it public, I'm fine with that.
[14:44] <abentley> mgz: I don't know why we've continued to make releases after it was discovered.  I assumed sinzui had filed a regression bug, since he knew about it.
[14:44] <sinzui> abentley: I didn't know about it
[14:44] <mgz> abentley: okay. I can confirm this does not contain your private ssh keys. :)
[14:45] <abentley> sinzui: Oh? Didn't you say you'd had to rename your ssh key to id_rsa to deal with this issue?
[14:45] <mgz> abentley: the issue with the ssh stuff is it depends somewhat on your personal setup, so I knew that the combo of my local ssh config + juju ci scripts borked ssh for juju
[14:46] <sinzui> abentley: yes, for just bootstrapping ec2 and openstack providers. I haven't seen any issue with other providers or ssh in general
[14:46] <mgz> I agree this is a regression, but given it has an annoying but somewhat trivial workaround (don't use your personal ssh config) I don't see how it's critical
[14:47] <sinzui> abentley: nd this behaviour matches the windows setup from 18 months ago
[14:50] <abentley> mgz: I don't think the existence or lack of workarounds is a factor in whether an issue should block.  We don't want to break users' existing workflows, and this does break users' existing workflows.
[14:51] <sinzui> abentley: I don't disagree
[15:11] <mup> Bug #1472632 opened: regression: juju ssh dies with (publickey) <blocker> <regression> <ssh> <juju-core:Triaged> <juju-core 1.24:Triaged> <https://launchpad.net/bugs/1472632>
[15:17] <alexisb> abentley, do we know what commit caused the regression?
[15:17] <alexisb> I would like to see bogdanteleaga be able to land his fix, it is a critical fix for 1.24.3
[15:17] <abentley> alexisb: No, we don't know which commit.  We found it by hand, not with automated tests, so we don't have logs that would show it.
[15:18] <alexisb> abentley, ok
[15:21] <alexisb> sinzui, abentley, mgz there will not be anyone from core looking at that bug until NZ/AUS comes online
[15:22] <mgz> I don't see how blocking is productive for this issue, nor how it's justified by our procedure
[15:22] <sinzui> alexisb: it doesn't blocl
[15:22] <sinzui> not tag blocker
[15:22] <mgz> sinzui: it has that tag currently, I was planning on raising in standup in 5 mins
[15:23] <alexisb> sinzui, ok, I must have miss read the back scroll
[15:23] <alexisb> I thought bogdanteleaga was blocked
[15:24] <mgz> sinzui: `./check_blockers.py check master`
[15:24] <sinzui> alexisb: if he is, he can add __JFDI__  to $$merge$$ to make it mege and test <- bogdanteleaga
[15:24] <mgz> noo..
[15:24] <mgz> either the bug blocks or it doesn't, we shouldn't be bypassing
[15:25] <mgz> I don't think it should block.
[15:25] <bogdanteleaga> it is blocked currently, but this is just the fix for master, the one for 1.24 is coming up
[15:26] <bogdanteleaga> not sure how long it takes for the upgrade ci job to test the fix though
[15:26] <sinzui> mgz: as we don't have a test for it and the regression is in the wild, we don't need to block. I think this is like the expressions closing the stable door after the horse has bolted
[15:29] <abentley> sinzui: That assumes that the number of people who have not upgraded to 1.24 is not significant.  I think that it is significant.  I think every time we put out a release, especially if we release 1.25, we encourage people who are using 1.23 and earlier to upgrade.
[15:53] <sinzui> bogdanteleaga: use "$$merge$$ fixes-1471332" to the pull requet comment to ensure CI will test and merge
[15:55] <bogdanteleaga> sinzui: I did try it with __JFDI__ but I fluked a test
[15:55] <bogdanteleaga> sinzui: should be fine on the next try; should I use jfdi or fixes?
[15:56] <sinzui> either bogdanteleaga
[15:56] <bogdanteleaga> cool
[16:10] <voidspace> dooferlad: TheMue: if you have a chance I'd appreciate a review http://reviews.vapour.ws/r/2116/
[16:11] <TheMue> voidspace: one moment, hunting a test failure, but will start in a few seconds
[16:11] <voidspace> cool, thanks
[16:11] <voidspace> good luck with the hunt :-)
[16:15] <TheMue> ah, kewl, panic is gone. now I can jump into your review
[16:18] <TheMue> voidspace: seeing your new ReleaseAddress() signature. could it be that the passed address doesn't match to the passed MAC address? IMHO they always should be a kind of pair with 1:N (one MAC, multiple IP)
[16:18] <voidspace> TheMue: no they're always 1:1
[16:18] <voidspace> TheMue: the MAC comes from state.IPAddress
[16:19] <voidspace> instance id is stored there too
[16:19] <TheMue> voidspace: ok, but technologically one MAC could have o´multiple IP, we only don't model it so right now
[16:19] <voidspace> ah
[16:19] <voidspace> our model does allow that, sorry
[16:19] <voidspace> although we don't use that capability
[16:20] <TheMue> ok
[16:20] <voidspace> so yes, one mac address could have multiple ip addresses - and we handle that
[16:20] <voidspace> we only delete the device if the IP address is the last one
[16:20] <voidspace> otherwise we just release the address normally
[16:20] <voidspace> biab, grabbing coffee whilst you read
[16:31] <TheMue> voidspace: you've got a review
[16:34]  * TheMue is afk for a moment, continue later
[17:15] <voidspace> TheMue: thanks
[17:32] <mup> Bug #1472711 opened: MAAS node has "failed deployment", juju just says "pending" <juju-core:Triaged> <juju-core 1.24:Triaged> <https://launchpad.net/bugs/1472711>
[18:20] <mup> Bug #1472729 opened: juju stuck in "upgrade in progress " for 20min <juju-core:New> <https://launchpad.net/bugs/1472729>
[18:23] <mup> Bug #1472729 changed: juju stuck in "upgrade in progress " for 20min <juju-core:New> <https://launchpad.net/bugs/1472729>
[18:32] <mup> Bug #1472729 opened: juju stuck in "upgrade in progress " for 20min <juju-core:New> <https://launchpad.net/bugs/1472729>
[19:47] <mup> Bug #1472749 opened: github.com/juju/utils has contradictory licences <juju-core:Triaged> <juju-core 1.24:Triaged> <https://launchpad.net/bugs/1472749>
[21:49] <marcoceppi> hello world
[21:49] <marcoceppi> maas 1.8.0 juju 1.24.2 deploying to LXC containers seems stuck, it's trying to wget the image from the bootstrap node/api server and it's been doing that for like 15 mins
[21:53] <marcoceppi> jk, it's moving on now
[22:00] <alexisb> hey marcoceppi welcome back from vacation
[22:01] <marcoceppi> alexisb: hey, thanks!
[22:37] <thumper> holy shit balls
[22:37] <thumper> these worker tests are taking ages to run...
[22:39] <thumper> FAIL	github.com/juju/juju/state/leadership	1200.021s
[22:39] <thumper> hmm
[22:39] <thumper> timeout kill
[22:40] <marcoceppi> nbd, 20 min tests
[22:41] <thumper> nbd?
[22:42] <marcoceppi> no big deal
[22:44] <thumper> nah man, it is a big deal
[22:44] <thumper> this is broke
[22:54] <menn0> thumper: shall I pick up the juju ssh blocker?
[22:54] <thumper> bug?
[22:55] <menn0> thumper: bug 1472632
[22:55] <mup> Bug #1472632: regression: juju ssh dies with (publickey) <blocker> <regression> <ssh> <juju-core:Triaged> <juju-core 1.24:Triaged> <https://launchpad.net/bugs/1472632>
[22:55] <menn0> it's blocking master and 1.24
[22:58] <alexisb> sinzui, ^^ I thought we had decieded that bug was not a blocker
[22:58] <alexisb> menn0, we discussed that one earlier today
[23:00] <thumper> menn0: I think abentley's analysis is wrong
[23:00] <thumper> menn0: if you look at the logs, it was his personal id_rsa that worked
[23:00] <thumper> but 1.24 and master did not appear to be trying
[23:02] <menn0> thumper: i haven't looked at it in any detail
[23:03] <alexisb> so menn0, thumper, my understanding from this morning is that bug should not be a blocker and the tag was going to be removed
[23:04]  * thumper removes blocker tag
[23:04] <alexisb> unfortunately I don't see any of the release dudes online atm
[23:04] <alexisb> sweet
[23:04] <menn0> thumper, alexisb: cool. that unblocks master
[23:04] <menn0> 1.24 is still blocked due to the window upgrade issue
[23:04] <menn0> i've just been talking to bogdanteleaga
[23:05] <menn0> he's waiting to see the problem is fixed on master before pushing the fixes to 1.24
[23:05] <thumper> that shouldn't block 1.24 then
[23:05] <menn0> the PR to fix 1.24 is ready to go though
[23:06] <menn0> thumper: no?
[23:06]  * thumper thinks
[23:06] <thumper> it will block us doing a release
[23:06] <thumper> but I don't think it should block us landing other fixes on 1.24
[23:07] <menn0> thumper: remove the blocker tag then?
[23:08] <thumper> menn0: do we know if it fixed the issue on master?
[23:08] <menn0> thumper: no we don't. CI hasn't gotten to running it yet
[23:09] <menn0> thumper: bogdanteleaga just noticed that his fix broke the unit tests under windows so he's going to do a fix for that now
[23:11] <wallyworld> waigani_: can you take a look at 1.24 bug 1472711? it claims bug 1376246 may not quite be fixed
[23:11] <mup> Bug #1472711: MAAS node has "failed deployment", juju just says "pending" <maas-provider> <juju-core:Triaged> <juju-core 1.24:Triaged> <https://launchpad.net/bugs/1472711>
[23:11] <mup> Bug #1376246: MAAS provider doesn't know about "Failed deployment" instance status <landscape> <maas-provider> <juju-core:Fix Released by waigani> <juju-core 1.24:Fix Released by waigani> <https://launchpad.net/bugs/1376246>
[23:37] <thumper> rick_h_: hey there
[23:59] <menn0> thumper: so the windows upgrade still isn't working in CI