[00:29] <mup> Bug #1450573 changed: HA and backup recovery tests failed <backup-restore> <blocker> <ci> <ec2-provider> <ensure-availability> <regression> <juju-core:Triaged> <https://launchpad.net/bugs/1450573>
[00:37] <wallyworld> thumper: any bribe to get you to look at http://reviews.vapour.ws/r/1556/ so that i can get a 1.24 fix running through CI?
[00:59] <mup> Bug #1451283 was opened: depployer sometimes fails with a unit status not found error <blocker> <ci> <regression> <juju-core:In Progress by wallyworld> <juju-core 1.24:In Progress by wallyworld> <https://launchpad.net/bugs/1451283>
[01:23] <thumper> wallyworld: yeah... on it now
[01:25] <thumper> wallyworld: actually had that review open already before lunch
[01:25]  * thumper looks in more detail
[01:35] <thumper> wallyworld: review done, a few questions
[01:50] <wallyworld> thumper: thank you, i've pushed changes, review board UI messed up my responses and turned them into issues
[01:50] <menn0> wallyworld, thumper: I just discovered another critical problem
[01:50] <thumper> hazaah
[01:50] <wallyworld> \o/
[01:50] <menn0> wallyworld, thumper: an upgrade from 1.18 to 1.23 blows up
[01:51] <menn0> wallyworld, thumper: 1.18 to 1.22 works
[01:51] <wallyworld> oh joy
[01:51] <menn0> wallyworld, thumper: looks like it's JES related
[01:51] <wallyworld> menn0: there was that JES upgrade issue horatio fixed last week
[01:52] <wallyworld> did you pull the latest code?
[01:52] <menn0> wallyworld, thumper: the API server can't start because it can't find the machine record
[01:52] <wallyworld> ah, ok, different, but could be same root cause
[01:52] <menn0> wallyworld: I'll make sure I'm using the same code
[01:52] <wallyworld> the issue last week was that an upgrade step was missed
[01:52] <menn0> wallyworld: I'm used to fixing these kinds of issues.
[01:52] <wallyworld> so the charm collection did nt have the UUID added
[01:52] <thumper> menn0: what machine record?
[01:53] <menn0> thumper: I meant document
[01:53] <wallyworld> well, the upgrade step was there but in the wrong place
[01:53] <menn0> wallyworld: this is before the upgrades run
[01:53] <wallyworld> ah, ok
[01:53] <thumper> ah... why is it different in 1.23 from1.22?
[01:54] <menn0> wallyworld: we have a few hacks in the code to deal with code that needs to function before upgrades have run. this looks like another one of those
[01:54] <menn0> thumper: not sure yet. will do some more digging now.
[01:54] <wallyworld> waigani_: do you know where i can find the bundles.yaml file used with the aws-deployer-bundle CI tests?
[01:54] <waigani_> there was a bug last week where an upgrade step adding UUIDs to charms docs targeted 1.22 instead of 1.21
[01:54] <menn0> wallyworld: the good news is that the new approach to dealing with replicaset init looks like it'll work. (that's how I found this bug)
[01:55] <wallyworld> great :-)
[01:55] <menn0> waigani_: yep. wallyworld mentioned that.
[01:56] <waigani_> wallyworld: sorry I haven't worked with bundles.yaml or those CI tests yet
[01:58] <davechen1y> thumper: ready when you are
[02:01] <menn0> grrr... now 1.18.4 won't bootstrap all of a sudden: start: Job is already running: juju-agent-menno-local
[02:01] <menn0> there is nothing juju related already running...
[02:01] <menn0> time to kick upstart
[02:01] <waigani_> wallyworld: from looking at the script - it's passed into the command (http://bazaar.launchpad.net/~juju-qa/juju-ci-tools/trunk/view/head:/deploy_stack.py:613)
[02:04] <wallyworld> waigani_: yes exactly, so the bundle.yaml file has to be stored somewhere, i was wondering if maybe you had come across where in your recent dealings with QA
[02:05] <waigani_> wallyworld: no sorry. I haven't needed to run the deployer test yet so I don't know.  best to ask CI.
[02:05] <wallyworld> will do, ta, was hping to avoid waiting 8hrs :-)
[02:06] <waigani_> :/
[02:08] <rick_h_> wallyworld_: looking at the name 'landscape-scalable' there's this one in the store https://jujucharms.com/u/landscape/landscape-scalable/8
[02:08] <rick_h_> wallyworld_: I can't confirm it's 100% the match but seems likely.
[02:10] <rick_h_> wallyworld_: so you'd want the bundles.yaml.orig file https://api.jujucharms.com/charmstore/v4/~landscape/bundle/landscape-scalable-8/archive/bundles.yaml.orig
[02:10]  * rick_h_ bets it's the orig yaml at least. 
[02:10] <wallyworld_> rick_h_: ah, yes, that looks right, awesome, tyvm
[02:13] <wallyworld_> thumper: you happy with the revised code?
[02:13] <thumper> wallyworld_: on a call, haven't looked
[02:15] <wallyworld_> ok, np
[02:27] <wallyworld_> rick_h_: ah, turns out that wasnt the right bundle. the one i want has django in it
[02:28] <wallyworld_> might be a custom CI one /var/lib/jenkins/repository/bundles.yaml
[02:31] <menn0> wallyworld_: that 1.18 to 1.23 upgrade problem is definitely occurring with the current 1.23 branch. Bug 1451297 created.
[02:31] <mup> Bug #1451297: Upgrade from 1.18 to 1.23 fails: password for machine agent can't be set <regression> <juju-core:In Progress by menno.smits> <juju-core 1.23:In Progress by menno.smits> <juju-core 1.24:In Progress by menno.smits> <https://launchpad.net/bugs/1451297>
[02:32] <wallyworld_> ok
[02:57] <mup> Bug #1451297 was opened: Upgrade from 1.18 to 1.23 fails: password for machine agent can't be set <regression> <juju-core:In Progress by menno.smits> <juju-core 1.23:In Progress by menno.smits> <juju-core 1.24:In Progress by menno.smits> <https://launchpad.net/bugs/1451297>
[03:45] <wallyworld> thumper: thank you
[04:47] <menn0> wallyworld or thumper: here's the fix for that SetPassword upgrade issue. http://reviews.vapour.ws/r/1558/
[04:49]  * thumper looks
[04:50] <thumper> menn0: done
[04:50] <menn0> thumper: thanks
[04:55] <menn0> axw: do you have a rough idea of when 1.23 will get unblocked? I have another critical bug fix.
[04:55] <axw> menn0: didn't know it was blocked, so nope, sorry
[04:55] <wallyworld> menn0: thanks for fixing
[04:56] <menn0> axw: you fixed the blocker!
[04:56] <axw> oh that
[04:56] <menn0> axw: bug 1449436
[04:56] <mup> Bug #1449436: Environment variables are not propagated to jujud on vivid <blocker> <juju-core:Fix Committed by axwalk> <juju-core 1.23:Fix Committed by axwalk> <juju-core 1.24:Fix Committed by axwalk> <https://launchpad.net/bugs/1449436>
[04:56] <axw> menn0: I'll see if CI has picked it up
[04:56] <axw> sorry
[04:56] <menn0> axw: np
[04:59] <axw> menn0 wallyworld: do I mark it as Fix Released if it was in a blessed CI version?
[04:59] <axw> or lower the importance
[05:00] <axw> I think Fix Released is what CI do, but feels odd to say that
[05:02] <axw> menn0: I've marked Fix Released, so should be unblocked now
[05:08] <menn0> wallyworld, axw: I thought marking it was Fix Released was only really the done thing for test only fixes?
[05:09] <axw> menn0: hmm yeah, that would make sense. I'll revert that and lower to High, so it's unblocked.
[05:09] <menn0> axw: thanks
[09:57] <wallyworld> axw: reviewed, but with a question. maybe it's nothing to worry about
[10:05] <perrito666> wallyworld: seriously mate, I think you should install quassel's server portion in another place :p
[10:05] <perrito666> morning all
[10:07] <TheMue> perrito666: morning
[13:36] <katco> sinzui: re: bug 1450919: mgz's comment: a fix was committed to master, but not 1.24 yet. are we sure that PR is contributing?
[13:36] <mup> Bug #1450919: many window unit tests failures <blocker> <ci> <regression> <windows> <juju-core:Triaged> <juju-core 1.24:Triaged> <https://launchpad.net/bugs/1450919>
[13:37] <sinzui> katco, I cannot be sure. I just read that. I am sure the test failed for 1.24
[13:39] <perrito666> wow, this was actually never even remotely implemented https://bugs.launchpad.net/juju-core/+bug/1415176
[13:39] <katco> sinzui: unless i'm missing something, logically, it cannot be that PR. perhaps it's even because that PR has not been landed into 1.24...
[13:39] <mup> Bug #1415176: debug-hooks exit 1 , doesn't mark hook as failed <cts> <debug-hooks> <juju-core:Confirmed> <juju-core 1.24:In Progress by hduran-8> <https://launchpad.net/bugs/1415176>
[13:39] <katco> perrito666: if it's not implemented, it's a feature request, yeah?
[13:40] <perrito666> I am not sure, apparently our docs says it exists
[13:40] <katco> perrito666: if you feel it is... tag it with "feature" and sinzui, alexisb, and i can decide whether or not it should be in 1.24
[13:42] <sinzui> perrito666, Then maybe the feature was lost?
[13:42] <perrito666> sinzui: m?
[13:43] <sinzui> perrito666, "debug-hooks exit 1" was either not implemented or it was and was lost in a version
[13:45] <perrito666> reading at debug hooks it would seem as if it never existed, let me peek a bit more
[13:46] <sinzui> perrito666, since the version in the bug is 1.20, we could try 1.18. if 1.18 doesn't have it, then never implemented by be true.
[13:46] <perrito666> ill go read 1.18
[13:47]  * perrito666 's life would be easier if 1.18 was in git
[13:47] <katco> perrito666: lp supports git now :D
[13:48] <perrito666> katco: does it have the rest of the repo?
[13:48] <katco> perrito666: i doubt it lol
[13:48] <katco> perrito666: but i thought it was pretty neat that lp supports git now
[13:54] <perrito666> so, without even going to 1.18
[13:54] <perrito666>  Mon Jul 29 17:55:23 2013 <-- this is the first appearance of the command
[13:54] <perrito666> and the only change since then is meaningless
[14:05]  * perrito666 is freezing, its 19C
[14:09] <hazmat> sinzui: there is trick to it i found
[14:10] <hazmat> sinzui: re debug-hooks exit and leave unperturbed, its still an issue that needs resolving.. but the trick is to exit the main window, so the hook window is the only that remains then exit 1 from the hook window.
[14:11] <perrito666> hazmat: tell me more
[14:16] <hazmat> perrito666: that's all there is to it.. original bug was filed in 2013 to which i added the work around, for some reason it was marked as a dup of one filed in 2015. just added the details to it.
[14:17] <perrito666> hazmat: tx, ill keep looking where is this being handled
[14:18] <hazmat> perrito666: i suspect its implicit behavior wrt to the workaround, and tmux exit status mirroring last window used. explicit would need a cli/hook cli command to perform the same (actually probably a shell alias).
[15:22] <katco> dooferlad: hey, were you looking into landing this into master? https://bugs.launchpad.net/juju-core/+bug/1442801
[15:22] <mup> Bug #1442801: aws containers are broken in 1.23 <blocker> <ci> <deployer> <ec2-provider> <lxc> <regression> <juju-core:Triaged by dooferlad> <juju-core 1.23:Fix Released by dooferlad> <juju-core 1.24:Fix Released by dimitern> <https://launchpad.net/bugs/1442801>
[15:26] <TheMue> katco: UK has a public holiday today, so dooferlad isn't here
[15:26] <katco> TheMue: ah ok. thx
[15:26] <katco> TheMue: anyone else on your team that can take his work and land it? it's blocking
[15:27] <TheMue> katco: 1.23 and 1.24 are released, so I think he's currently porting it
[15:27] <katco> TheMue: yeah, but while he's away no one can land anything =/
[15:27] <katco> TheMue: how can we get it landed earlier?
[15:28] <TheMue> katco: I've got to take a look at it, don't know exactly how big it is
[15:32] <TheMue> katco: will do it
[15:33] <katco> TheMue: you rock, sir!
[15:33] <katco> TheMue: the throngs of juju core developers applaud you :D
[15:36] <TheMue> hehe
[16:26] <TheMue> katco: quick review of http://reviews.vapour.ws/r/1564/?
[16:26] <katco> TheMue: tal
[16:27] <katco> TheMue: is this just a port of work that dooferlad has already done? (i.e. no significant deviations from that branch)?
[16:27] <TheMue> katco: exactly
[16:28] <katco> TheMue: do you have a link handy to the original work so i can compare against it?
[16:29] <TheMue> katco: the original one has been https://github.com/juju/juju/pull/2071
[16:29] <katco> TheMue: ty
[16:29] <TheMue> katco: np
[16:31] <katco> TheMue: Ship It
[16:31] <TheMue> katco: thx
[16:31] <katco> TheMue: with my thanks :)
[16:36] <katco> sinzui: can you respond to my latest comment on bug 1450919?
[16:36] <mup> Bug #1450919: many window unit tests failures <blocker> <ci> <regression> <windows> <juju-core:Triaged> <juju-core 1.24:Triaged> <https://launchpad.net/bugs/1450919>
[16:42] <katco> sinzui: do we have a tag for customer-reported bugs?
[16:42] <sinzui> katco, yes, many
[16:43] <katco> sinzui: which one should i use?
[16:43] <sinzui> katco, cts is the common tag
[16:43] <sinzui> katco, per the bug, I added "gotesttarfile.py is forcing "PATH" and "Path" into the env. We did this to see passed before Core merged their fixes."
[16:57] <TheMue> katco: the fix is merged
[16:57] <katco> TheMue: awesome, ty so much!
[16:58] <TheMue> katco: always a pleasure ;)
[17:08] <mup> Bug #1451487 was opened: bootstrap is not downloading the disc image, but machine 0 comes up. <bootstrap> <cts> <local-provider> <lxc> <juju-core:Incomplete> <https://launchpad.net/bugs/1451487>
[17:08] <mup> Bug #1451488 was opened: When deploying a service, the charm is added, but the new instance is remains pending. <community> <juju-core:Incomplete> <https://launchpad.net/bugs/1451488>
[17:26] <TheMue> eod o/
[17:26] <katco> TheMue: tc o/
[17:32] <bdx> How do you guys feel about adding a config-flags param for ceph and ceph-osd?
[17:38] <mup> Bug #1450912 changed: quickstart is universally broken in 1.24 <blocker> <ci> <quickstart> <regression> <juju-core:Fix Released by wallyworld> <juju-core 1.24:Fix Released by wallyworld> <https://launchpad.net/bugs/1450912>
[17:38] <bdx> gnuoy, jamespage: How do you guys feel about adding a config-flags param for ceph and ceph-osd?
[18:32] <perrito666> bbl, bike pause
[19:16] <sinzui> katco, do you have a moment to review http://reviews.vapour.ws/r/1565/
[19:16] <katco> sinzui: tal
[19:17] <katco> sinzui: ship it
[19:17] <sinzui> thank you katco
[19:17] <katco> sinzui: ty!
[20:03] <alexisb> so folks, do we have a plan of attack on this critical bug: https://bugs.launchpad.net/juju-core/+bug/1450919 ?
[20:03] <mup> Bug #1450919: many window unit tests failures <blocker> <ci> <regression> <windows> <juju-core:Triaged> <juju-core 1.24:Triaged> <https://launchpad.net/bugs/1450919>
[20:04] <alexisb> if not what do we need to move forward
[20:04] <alexisb> katco, ^^
[20:04] <katco> alexisb: hey
[20:05] <alexisb> heya katco :)
[20:05] <katco> alexisb: i was going to discuss in the release meeting. i need to understand better how it's not a dupe of bug 1446871
[20:05] <mup> Bug #1446871: Unit hooks fail on windows if PATH is uppercase <ci> <hooks> <windows> <juju-core:Fix Committed by natefinch> <juju-core 1.24:In Progress by natefinch> <https://launchpad.net/bugs/1446871>
[20:05] <alexisb> katco, ack
[20:05] <katco> alexisb: sinzui responded, but i don't understand how his comments are related
[20:06] <katco> alexisb: regardless, nate will probably be fixing that tomorrow. he was already working in that area
[20:06] <alexisb> katco, ok
[20:06] <sinzui> katco, We hacked the test *before* the PATH fixes were merged to help you get a pass.
[20:07] <sinzui> katco, we can remove the PATH fix from CI so that core tests need to do all the setup
[20:08] <katco> sinzui: a pass on what? master?
[20:08] <sinzui> katco, everything
[20:09] <katco> sinzui: i need more context... i'm not connecting what you're saying at all
[20:09] <katco> sinzui: so here's what i know: this critical bug looks a lot like bug 1446871
[20:09] <mup> Bug #1446871: Unit hooks fail on windows if PATH is uppercase <ci> <hooks> <windows> <juju-core:Fix Committed by natefinch> <juju-core 1.24:In Progress by natefinch> <https://launchpad.net/bugs/1446871>
[20:09] <katco> sinzui: bug 1450919 calls out 1.24, but nate's fix isn't in 1.24 yet
[20:10] <mup> Bug #1450919: many window unit tests failures <blocker> <ci> <regression> <windows> <juju-core:Triaged> <juju-core 1.24:Triaged> <https://launchpad.net/bugs/1450919>
[20:10] <sinzui> katco, CI is globally adding PATH and Path to to all windows tests to help them pass. I add the fix in Jauary, mgz, removed it in April, and then put it back when we find Core was still broken
[20:10] <sinzui> katco, Sorry, I cannot speak of that. I didn't do any analysis
[20:11] <katco> sinzui: ok so you contend that 1450919 is not about the casing of paths, but about the paths not being there on windows, period?
[20:11] <sinzui> katco, I only contend that PATH and Path are identical. We guarantee that
[20:12] <katco> sinzui:  are you ysubg PATH and Path as placeholders for an actual environmental variable?
[20:12] <katco> *using
[20:13] <sinzui> katco, we export PATH and Path to PowerShell when we run the tests. IN the past, somethings in Juju tests (not Juju) were only looking at PATH.
[20:13] <katco> sinzui: ok, i'm back to asserting that this new critical bug is just a dupe of an existing, non-critical bug which is not yet in 1.24, but in master
[20:14] <katco> sinzui: is there a way to remove CI's PATH Path patch for just master and see if it passes?
[20:14] <sinzui> katco, I honestly don't care how it is classified. CI will tell us when the tests passes
[20:14] <sinzui> katco, I can make a code change to remove it
[20:15] <katco> sinzui: and see if master passes?
[20:15] <sinzui> katco, already failed :)
[20:15] <sinzui> But I am retesting any way because most of the failures are common flakey tests
[20:16] <katco> lol
[20:16] <sinzui> katco, http://juju-ci.vapour.ws:8080/view/Juju%20Revisions/job/run-unit-tests-win2012-amd64/345/console does look better ")
[20:16] <sinzui> :)
[20:16] <katco> that's a 404 for me
[20:17] <sinzui> login
[20:17] <sinzui> katco, my current grief is quickstart and deployer are broken in joyent. I cannot think of how I can help those tests pass
[20:17] <katco> ah doh
[20:17] <katco> sinzui: one thing at a time :)
[20:18] <katco> sinzui: so if this passes, we know the new critical bug is a dupe?
[20:19] <sinzui> katco, IF it passes. I close all bugs about master :). Actually CI will do that for us I think
[20:29] <katco> sinzui: so it didn't pass... log looks like the same error. but it also still looks like a dupe bug to me
[20:33] <sinzui> katco, okay. Make it a dupe.
[20:33] <katco> sinzui: do you agree?
[20:33] <sinzui> katco, I don't have an opinion.
[20:33] <sinzui> katco, I have had other pressing matters to deal with today
[20:34] <katco> sinzui: ok. i'll wait until the release call and get ian's opinion
[20:38] <sinzui> muwa ha ha. A container was left behind from a previous test. Deployer is fine, now I retest quickstart in the same pristine conditions
[20:39] <katco> :)
[20:50] <mup> Bug #1442801 changed: aws containers are broken in 1.23 <blocker> <ci> <deployer> <ec2-provider> <lxc> <regression> <juju-core:Fix Released by themue> <juju-core 1.23:Fix Released by dooferlad> <juju-core 1.24:Fix Released by dimitern> <https://launchpad.net/bugs/1442801>
[20:50] <mup> Bug #1450917 changed: afterHook logic skipped if hook is missing <blocker> <ci> <deployer> <maas-provider> <regression> <juju-core:Fix Released by wallyworld> <juju-core 1.24:Fix Released by wallyworld> <https://launchpad.net/bugs/1450917>
[21:05] <wallyworld> sinzui: a couple of times, aws-deployer-bundle fails with an "impossible" error. but there's not artifacts like all-machines.log etc that i can find and i can't really debug without these. i'd also like a copy of the bundle.yaml file used in that test. can things be changed so that if it happens again, the logs etc are available? and can I get a copy of the bundle.yaml file?
[21:05]  * perrito666 thinks wallyworld has a very interestin definition of impossible
[21:06] <sinzui> wallyworld, lp:/uju-ci-tools/repository  or just http://bazaar.launchpad.net/~juju-qa/juju-ci-tools/repository/view/head:/bundles.yaml
[21:07] <sinzui> wallyworld, the deployer and quickstart tests don't use the infrastructure to get get logs from an env :(
[21:40]  * thumper afk for a bit
[22:03] <sinzui> http://bazaar.launchpad.net/~juju-qa/juju-ci-tools/repository/view/head:/bundles.yaml
[22:03] <sinzui> wallyworld, ^
[23:18] <wallyworld> axw: perrito666: standup?
[23:18] <perrito666> going
[23:32] <mup> Bug #1451616 was opened: Automate backups of initial environment <juju-core:Triaged> <https://launchpad.net/bugs/1451616>
[23:41] <perrito666> swweeeeeeeeeeeeet, I fixed it
[23:41] <perrito666> :p
[23:47] <perrito666> wallyworld: I have the fix for 1415176 shall I propose that to 1.24? or into master already?