[00:00] Bug #1494947 opened: Panic keyManagerSuite.TestImportKeys on wily

[00:00] Bug #1494948 opened: Panic charmVersionSuite.TearDownTest on wily

[00:00] Bug #1494949 opened: Panic charmVersionSuite.TearDownTest on wily

[00:03] Bug #1494947 changed: Panic keyManagerSuite.TestImportKeys on wily

[00:03] Bug #1494948 changed: Panic charmVersionSuite.TearDownTest on wily

[00:03] Bug #1494949 changed: Panic charmVersionSuite.TearDownTest on wily

[00:06] Bug #1494947 opened: Panic keyManagerSuite.TestImportKeys on wily

[00:06] Bug #1494948 opened: Panic charmVersionSuite.TearDownTest on wily

[00:06] Bug #1494949 opened: Panic charmVersionSuite.TearDownTest on wily

[00:06] hello [00:06] both the blocking bugs have been marked fixed committed [00:06] how long until CI is unblocked [00:06] thanks [00:09] Bug #1494947 changed: Panic keyManagerSuite.TestImportKeys on wily

[00:09] Bug #1494948 changed: Panic charmVersionSuite.TearDownTest on wily

[00:09] Bug #1494949 changed: Panic charmVersionSuite.TearDownTest on wily

=== wwitzel3_ is now known as wwitzel3 === sinzui_ is now known as sinzui [00:27] Bug #1496184 opened: juju bootstrap on armhf/keystone hangs juju version 1.24.5 [00:30] Bug #1496184 changed: juju bootstrap on armhf/keystone hangs juju version 1.24.5 [00:33] Bug #1496184 opened: juju bootstrap on armhf/keystone hangs juju version 1.24.5 [00:39] Bug #1496184 changed: juju bootstrap on armhf/keystone hangs juju version 1.24.5 [00:42] Bug #1496184 opened: juju bootstrap on armhf/keystone hangs juju version 1.24.5 [00:54] Bug #1496188 opened: panic in juju worker [01:07] anastasiamac_: around? [01:12] rick_h_: Hi :D [01:13] anastasiamac_: howdy, so the meeting time is a bit off for my tastes. So I wanted to check in and ask what I could provide to help in my absence? [01:13] rick_h_: apologies for funny timing for the meeting - it's just aplaceholder... plz let me know what works for u :) [01:13] anastasiamac_: aside from the info on the bundle break down lib and such, is there anything else you needed on my end? [01:13] rick_h_: ur presence? :P [01:14] anastasiamac_: :) [01:14] rick_h_: what time would suit u better? [01:14] anastasiamac_: the other one worked better, that's just getting up early [01:15] anastasiamac_: but if you know what you're looking for input-wise maybe I can do it async? [01:15] I don't want to hold things up on me [01:15] as I think we had most things in hand back in the spec aside from the new bundle stuff and I've given you all the details on that [01:17] rick_h_: from my perspective, I think m k at the moment with info [01:17] rick_h_: would it suit u better if I'll try to move the meeting by 5hrs? [01:17] anastasiamac_: ok, nvm. I'll be there [01:17] anastasiamac_: don't worry about it [01:17] anastasiamac_: forget I was here../me disapears into the night [01:17] rick_h_: i would like u to sleep :D [01:18] but I'll move the meeting to later as I think it may suit u and John better :D [01:18] anastasiamac_: naw, I like to party :P [01:18] anastasiamac_: ask my team, it's not that bad for me anyway and I can sleep in thus :) [01:19] rick_h_: if u reconsider, m happy to move to more suitable time ;D [01:19] anastasiamac_: all good, go back to happy work thoughts for your day [01:20] rick_h_: thnx :D get rest! === akhavr1 is now known as akhavr [02:49] menn0: hey, got time to talk about the unit agent upgrades? [02:49] davecheney: care to look at this bug? http://reports.vapour.ws/releases/3059/job/run-unit-tests-trusty-ppc64el/attempt/3804 [02:50] davecheney: weird that it happens on power but not amd64 [02:53] thumper: yep [02:54] davecheney: although, that code looks so fucked up [02:54] * thumper is ask menno too [02:54] menn0: standup hangout? [02:54] thumper: see you there [02:55] * thumper wonders how this code works [03:07] thumper: honestly no idea [03:07] davechen1y: anyway, I've got this one [03:07] we've had continual problems getting the right versino of gccgo and libgo5 on ppc64 machines [03:31] menn0: this is the uniter panic on power http://reviews.vapour.ws/r/2671/diff/# [03:31] davecheney: I have a golang question... [03:31] if I have a function, and that function defines a pointer variable [03:32] and in that same function I create some other closures (functions) that refer to that variable [03:32] are the closures defined to refer to that variable by reference? [03:33] thumper: yes [03:33] good [03:34] thumper: uniter fix LGTM [03:35] davecheney: ta [03:46] Bug #1496217 opened: panic in cmd/jujud on power

[03:46] Bug #1496221 opened: Whitespace or new lines in "juju status --format=tabular" output [03:49] Bug #1496217 changed: panic in cmd/jujud on power

[03:49] Bug #1496221 changed: Whitespace or new lines in "juju status --format=tabular" output [03:50] thumper: when do "fix committed" blockers roll off the dashboard ? [03:50] davecheney: when they have passed CI [03:51] wallyworld: http://reports.vapour.ws/releases/3059/job/aws-upgrade-20-trusty-amd64/attempt/29 fix didn't fix [03:52] thumper: do we have to wait til tomorrow for that ? [03:52] davecheney: I just removed one my marking it fix released [03:52] davecheney: we are down to two failures [03:52] I have submitted a fix for one of them [03:52] which in honesty I think will change a panic into a different failure [03:52] the other was the upgrade one... [03:54] thumper: true, if that valye was nil [03:54] something else failed to initalise [03:55] Bug #1496217 opened: panic in cmd/jujud on power

[03:55] Bug #1496221 opened: Whitespace or new lines in "juju status --format=tabular" output [03:55] holy shit mup is somewhat behind [03:57] wallyworld: is the uniter failure due to the missing upgrade step? [04:04] Bug #1495591 changed: TestRunCommand fails on windows

[04:05] thumper: which uniter failure r u referring to? [04:05] anastasiamac_: the most recent failure from ci - http://reports.vapour.ws/releases/3059/job/aws-upgrade-20-trusty-amd64/attempt/29 [04:06] menn0: I have a hacky fix for the uniter upgrade step in a 1.25 branch, running tests now [04:07] thumper: cool [04:10] Bug #1495591 opened: TestRunCommand fails on windows

[04:13] Bug #1495591 changed: TestRunCommand fails on windows

[04:14] wallyworld: after chatting with anastasiamac_ the answer I think is "yes, the lack of uniter upgrade steps is causing the upgrade failure" [04:14] * thumper has a hacky fix [04:15] * thumper shakes his fist at the sky [04:15] * thumper is grumpy [04:15] in our current tests... even 1.25, I've come across three different intermittent failing tests [04:19] thumper: -race [04:20] bah humbug [04:20] no presents for anyone this year [04:22] github.com/juju/juju/worker/peergrouper fails every time for me... [04:22] * thumper sighs [04:22] yeah, that one has failed for months for me [04:22] especially on non intel platforms [04:23] for bug 1495542, i think that the reported bug is no longer a blocker as the test exercising unused code which caused the trace has been skipped... [04:23] This bug still needs to be addressed but it should not be blocking master. [04:23] The failure observed now seem to be different - the trace looks different. What we currently observe MAY warant another bug ;D [04:23] Bug #1495542: 1.20.x cannot upgrade to 1.26-alpha1

[04:24] thumper gives presents? each year? [04:24] anastasiamac_: not any more... [04:24] thumper: no ned to stop on my account :) [04:25] need* [04:25] davecheney: I'm running go 1.5 now, wondering if that is related... [04:25] * thumper hits the -race fails with the go package... [04:25] no races for me [04:26] thumper: almost certainly [04:26] see long discussion about getting 1.5 rolled out everywhere before comitting to 1.6 for 16.04 [04:32] * thumper nods [04:32] wallyworld: actually back to the upgrade failure, it seems that yes, while there is an error with a bad kind, things progress anyway [04:32] thumper: "ModeAbide: cannot set invalid status "started"" - that looks like it's caused by new juju status enums not being processed correctly, but that code has been in place for months. not ure why that is shwing up now. [04:32] thumper: AFAIK juju passes on intel with 1.5 [04:32] maybe it was always there and we didn;t notice [04:33] it almost passes on non intel with go 1.5 [04:33] wallyworld: the fundamental error is that the source sets relation config, the sink never runs relation-changed hook [04:33] but slowness makes it hard to be absolute here [04:33] see previous plee for faster ppc64 hardware [04:33] davecheney: amd64 and go 1.5 fails peergrouper almost every time for me [04:33] thumper: ok, will start at that point [04:34] thumper: peergrouper tests kinda suck (from memory). they need to be transformed into proper unit tests [04:34] wallyworld: ack [04:35] thumper: try GOMAXPROCS=1 go test .../peergrouper [04:35] that will make the scheduler look more the like 1.2.1 scheduler [04:35] and may improve reliabilty [04:35] menn0: is this what you were thinking for the uniter upgrades? http://reviews.vapour.ws/r/2672/diff/# [04:35] * menn0 looks [04:35] if so, then those tests need to be fixed to not expect a certain order of operations [04:35] * thumper pokes the peergrouper [04:37] anastasiamac_: As long as master is not blessed, master is blocked. if you close bug 1495542, you need to replace it with a critical ci regression bug because stakeholder will not allow is to release if the test shows they canot upgrade from 1.20 to 1.24. [04:37] Bug #1495542: 1.20.x cannot upgrade to 1.26-alpha1

[04:38] anastasiamac_: CI Cares about tests passing, fixing bugs without getting the tests passing does not mean we can release [04:39] thumper: that's pretty much what I was thinking although in my mind those upgrade steps would be run from the unit agent Run() [04:39] sinzui: ah... my bad - the bug is not about the trace but upgrade difficulties :D of course, it should stay :) [04:40] thumper: calling them from Run is probably slightly preferrable in case there are some others we need to add that aren't uniter related [04:40] thumper: also, I thought you said there were 2 upgrade steps like this? [04:42] menn0: the other one is introduced in master, this is a 1.25 branch [04:43] where is the unit agent Run func? [04:43] cmd/jujud/agent/unit.go [04:44] thumper: ^ [04:44] ta [04:45] FFS! [04:45] I'm getting TLS handshake errors with a freshly bootstrapped env using master [04:46] actually... ignore me [04:47] davecheney: FWIW setting GOMAXPROCS=1 makes the peergrouper tests pass [04:47] I think there's agents from a hosted env trying to contact the previous state server [04:47] heh [04:56] menn0: I'll move the run upgrades func to the unit agent code [04:57] thumper: they're probably also fine where they are (in the uniter package) [04:57] menn0: nah... [04:57] moving already [05:07] Bug #1496237 opened: peergrouper tests very unstable with Go 1.5

[05:10] Bug #1496237 changed: peergrouper tests very unstable with Go 1.5

[05:10] menn0: this better? http://reviews.vapour.ws/r/2672/diff/# [05:13] Bug #1496237 opened: peergrouper tests very unstable with Go 1.5

[05:14] * menn0 looks [05:15] thumper: LGTM [05:16] menn0: I'll await fwereade's comment [05:16] cool [05:29] ok... I'm pretty much done for today [05:29] see all y'all tomorrow [06:30] wallyworld: ping [06:31] jam1: hi === jam1 is now known as jam [06:32] wallyworld: so of Series in Charm Metadata and Resources, I think we'd rather focus on the former [06:33] I know you poked me about the latter, and I can try to bring it up in 30 min, though I have other higher priority right now. [06:33] jam: yeah, that work is underway. i'd also like to get resources spec at least approved so we can start on it this cycle. eco really wants it. maybe you can do it next week, or just ask the spec to be reviewed over the next few days [06:34] jam: series in metadata won't take 100% so there will be a little spare capacity [07:39] fwereade: I'm debugging #1495542, getting a lot of "2015-09-16 07:24:41 DEBUG juju.worker.dependency engine.go:438 "uniter" manifold worker stopped: getting resource leadership-tracker: dependency not available" [07:39] Bug #1495542: 1.20.x cannot upgrade to 1.26-alpha1

[07:40] fwereade: restarting the API server did not appear to help, but restarting the unit agent did. makes me suspect the dependency engine isn't delivering a new API connection to the leadership tracker... kind of clutching at straws though [07:41] fwereade: other part to the error: "leadership-tracker" manifold worker returned unexpected error: leadership failure: leadership manager stopped [07:42] fwereade: and after I restarted the API server, I was getting: engine.go:438 "leadership-tracker" manifold worker stopped: leadership failure: error making a leadership claim: connection is shut down [07:44] cmars: ^^ on upgrade from 1.20 to 1.26, "metricspooldir" doesn't exist. is there meant to be an upgrade step for this? [07:44] axw, o/ [07:44] dimitern: heya [07:45] axw, just a heads up - we decided to do a simpler implementation (not as effective as I'd like) for provisioning instances into subnets of a space, only in EC2 and not integrating that with AZ distribution [07:46] axw, then, we should have a discussion how to properly do it with you and/or fwereade [07:47] dimitern: okey dokey, so does it disable distribution when a space is specified? [07:48] axw, it tries to accommodate both - i.e. starting from all AZs and restricting them to only those relevant to a given space [07:49] any reviewers? this is a forward port to master of the 1.25 fix for bug 1492066: http://reviews.vapour.ws/r/2673/ - please take a look [07:49] Bug #1492066: cloud-init fails when deploying CentOS with Juju.

[07:49] dimitern: hm ok, I don't really get how you implement DistributeInstances correctly then. I might need to see code to understand [07:51] axw, I'll send you a link to the PR by mail when I'm done [07:51] dimitern: thanks [07:51] axw, sorry, processing [08:01] axw, ok, I'm not really managing to form a coherent theory at the moment -- that latest log does rather look like the various workers are coming up? but we've got weird settings revision numbers? [08:02] axw, but then the leadership errors you posted above make it look like the problem is in [08:02] fwereade: it did in CI, but when I repro'd it just continuously restarted manifold workers [08:02] axw, fuck, I think I might know what's happening [08:03] axw, we've never had a mechanism for usefully bouncing the workers that live under state [08:04] fwereade: I saw the "HackLeadership" thing, which is called when the API server stops handling a client connection [08:05] fwereade: isn't that the only time we need to bounce it...? [08:05] axw, if it encounters some unknown error, it'll stop on its own [08:06] axw, but then I don't quite get what happens when the apiserver restarts [08:06] fwereade: I didn't see any "stopping leadership manager with error", but I'm about to repro again so I'll double check [08:09] axw, error logging around state/leadership/manager.go:38 would detect that, I think [08:09] axw, I don't think there's anything in state that'd otherwise notice the error until state itself is closed [08:10] axw, hopefully we'd log that one, but it could be a long time after it actually happened [08:10] fwereade: lol. third time's the charm, didn't repro this time :/ [08:10] goddammit [08:11] fwereade: settings version is still screwed though [08:12] axw, huh [08:12] axw, so the settings version was right before the upgrade and something overwrote it? [08:12] fwereade: not sure yet. I upgraded, ran a "config-set" which triggers "relation-set", remote state sees it... but it's the same as what's in local state already [08:13] axw, is the relation-set to a unique value? [08:13] axw, we elide null changes [08:13] fwereade: yes [08:14] axw, just checking :( [08:15] axw, I guess check the rel-state/mgo-state versions before the upgrade and see which get touched by the upgrade? [08:15] axw, either something's breaking the versions [08:15] fwereade: yeah. gotta got for a bit, I'll try that after [08:15] axw, or the versions are already broken [08:15] axw, cheers [08:18] * fwereade has to depend on wifi through walls for a few hours, please forgive any spottiness [08:43] menn0 or katco: could you please take a look at http://reviews.vapour.ws/r/2633/ when you have time? proposed against a feature branch [09:01] dimitern, voidspace, jam: standup time if you're coming... [09:02] frobware: I'm here [09:02] frobware: omw [09:07] fwereade: welp... before upgrade, api server sent a relation units change with version=3. after upgrade, the initial change event has version=2 [09:13] fwereade: I think this might be because of the env-uuid migration... txn-revno would reset, because the doc is removed and re-added. but not sure why it hasn't been an issue before. [09:14] axw, holy hell, that is horrifying [09:15] axw, ok, if we're going to address this -- as we must -- let's do it right and maintain our own revno [09:15] fwereade: yep, SGTM [09:15] fwereade: any thoughts on why this wouldn't have been an issue until now? [09:16] axw, no idea at all [09:17] axw, ah-ha [09:17] axw, ...or not [09:17] heh :) [09:18] axw, hmm. did we change from remove/add in one txn to remove-a-bunch then add-a-bunch? [09:18]