[00:00] wallyworld: can you make this PR merge please? https://github.com/juju/juju/pull/2239 [00:01] wallyworld: I think I aborted a previous build attempt in Jenkins on Friday b/c I realised it wasn't going to work [00:01] wallyworld: so now the landing bot is confused. [00:12] sure [00:16] menn0: i have resubmitted it to the bot - you just have to record a build failed message in the PR for the bot to realise it needs to process it again [01:01] wallyworld: ok thanks [01:01] menn0: except it failed again [01:01] i didn't look at the reason [01:02] wallyworld: just looking now [01:14] wallyworld: teensy review please: http://reviews.vapour.ws/r/1384/ [01:14] sure [01:15] axw: i'd just self review one as small as that [01:15] wallyworld: ok [01:25] wallyworld: did you have any time in the next hour for a catch up? since I missed Friday [01:26] menn0: has wallyworld suitably kicked the bot? [01:26] thumper: sure, i was waiting for you and you ditched me [01:26] sorry, was out Friday morning [01:27] was just pressing your buttons [01:27] and I have so many to press [01:27] indeed [01:29] wallyworld: I'm in our 1:1 hangout [01:29] ok, one sec [01:32] thumper: yep he did [02:23] Bug #1453634 was opened: juju upgrage-juju --upload-tools hangs

[02:29] Bug #1453634 changed: juju upgrage-juju --upload-tools hangs

[02:35] Bug #1453634 was opened: juju upgrage-juju --upload-tools hangs

[03:32] axw: does upgrade also work for 1.25 for you? [03:32] waigani: haven't tried [03:33] axw: I'm debugging with 1.25 now. Any chance you could try on your box? [03:33] waigani: ok [03:34] axw: thanks. I've tried about 4 times in a row now. same issue each time. I can ssh to machine 0, debug-log works, but it always hangs on upgrade-juju --upload-tools [03:35] waigani: it did take a long time to upload the tools, but it always does for me [03:35] anyway, trying with master now [03:36] waigani: have you tried with 1.22 or 1.23? that should tell you if it's a regression, or if it's just your network [03:37] axw: I'll try 1.23 now. [03:42] axw: 1.23 works fine for me [03:42] hmk, strange [03:44] waigani: open it back up with that then I guess. master is still bootstrapping for me .. [03:45] axw: though if it is my network, maybe it's better now. I'll test 1.23 one more time and wait to see what 1.25 does for you. [03:53] waigani: master works for me too [03:53] axw: really? then it's me :/ [04:00] axw: looks like 1.23 is hanging. It hangs after tools are uploaded (i.e. after it reports "available tools:..."). [04:00] waigani: you said it worked fine before? [04:05] Bug #1453644 was opened: One unit remains pending with local provider [04:06] axw: I'll dig a little more, but as it's only on my box I'll knock down the importance. Thanks for testing 1.25. [04:06] sure [04:11] ha [04:12] waigani: did u have a "eureka" moment? [04:12] waigani: or is it" high availability" moment? [04:12] hehe [04:13] so I had export JUJU_DEV_FEATURE_FLAGS=jes in my .bashrc [04:14] when I tried 1.23 the first time, it was in an old terminal window that hadn't sourced .bashrc [04:14] when I commented out the export and tried the upgrade, it worked [04:15] long story short, I think something hiding behind the jes flag is breaking upgrade-juju [04:16] axw: ^^ [04:17] waigani: ok. please add that to the bug, change the title, drop the severity and leave it open only on whatever branch JES is being released in [04:17] axw: okay [04:34] wallyworld: I updated TestCreateVolume to wait for status != "" [04:34] looking [04:35] wallyworld: createVolume returns with Status=="" [04:35] axw: i already reviewed just before [04:35] i think a test is missing [04:36] ah hang on [04:36] i see what you're saying [04:38] wallyworld: ok to land? [04:39] axw: i guess what i was looking for was that the status would be "" until the test triggered it to be not "". otherwise the old code would also have passed the test if you know what i mean. but i guess it doesn't really matter [04:39] wallyworld: I could change it to only return "available" after a couple of iterations [04:39] I'll do that [04:40] axw: and record that at least one failed attempt occurred [04:40] yep [04:40] that would be neat, thanks [04:40] wallyworld: although that means I need to patch the attempt strategy.. bleh [04:41] if it's a hassle, ship it :-) [04:41] strictly speaking, we should test for it i guess [04:41] maybe a todo [04:49] wallyworld: done, landing [04:49] ty [04:49] wallyworld: I think I might have a lie down, feeling a bit average [04:50] :-( ok, take it easy [04:50] * thumper chuckles [04:51] wallyworld: and average is so far down from normal... [04:51] damn it [04:51] depends who you are [04:51] If I stopped when I felt average, I'd never get anything done... [04:51] so you feel average a lot then [04:51] heh [05:24] ciao [05:24] * thumper is off [07:39] morning [09:02] dooferlad: stdup? [09:05] dooferlad: TheMue: I got booted out [09:05] dooferlad: TheMue: probably have to log back in, sorry [10:12] Bug #1453744 was opened: juju version 1.23.2-trusty-amd64 console need agent-metadata-url flag? [11:12] Bug #1453760 was opened: "u'Error': u'status for key "" not found" before adding relations when using juju 1.24 [11:54] Bug #1453760 changed: "u'Error': u'status for key "" not found" before adding relations when using juju 1.24

[12:00] Bug #1453760 was opened: "u'Error': u'status for key "" not found" before adding relations when using juju 1.24

[12:12] Bug #1453760 changed: "u'Error': u'status for key "" not found" before adding relations when using juju 1.24

[12:54] Bug #1453785 was opened: transaction collection (txns) grows without bound

[13:24] Bug #1453801 was opened: /var/spool/rsyslog grows without bound

[13:24] Bug #1453805 was opened: Juju takes more than 20 minutes to enable voting

[13:36] Bug #1453744 changed: juju version 1.23.2-trusty-amd64 console need agent-metadata-url flag? [13:36] Bug #1453814 was opened: rsyslog configuration forwards log messages that were already forwarded

[13:48] Bug #1453744 was opened: juju version 1.23.2-trusty-amd64 console need agent-metadata-url flag? [13:48] Bug #1453814 changed: rsyslog configuration forwards log messages that were already forwarded

[13:54] Bug #1453744 changed: juju version 1.23.2-trusty-amd64 console need agent-metadata-url flag? [14:00] Bug #1453814 was opened: rsyslog configuration forwards log messages that were already forwarded

[14:03] katco`, : I need help triaging bug 1453644. I think an engineer needs to determine if we need to/can fix this bug for 1.24 or 1.25 [14:03] Bug #1453644: One unit remains pending with local provider

[14:06] Bug #1453814 changed: rsyslog configuration forwards log messages that were already forwarded

[14:08] sinzui: just in a meeting, one sec === katco` is now known as katco [14:17] alexisb: ping [14:24] Bug #1434741 changed: PruneLogs suffers from an indeterminate map iteration order bug [14:36] mgz, have you tried juju with dreamhost yet? [14:36] jcastro: no, not recently [14:37] do you know anyone who has tried it? I'm trying to document it for the docs but I get auth errors [14:37] I've not seen anything on the list, I'll see if I have creds around somewhere still [14:37] ideally using keypairs would be lovely [14:38] yeah, I need to re-add that code to juju-core [14:38] ok so it only does userpass? [14:38] they do give me a .pem file as well [14:39] cherylj: perrito666: wwitzel3: is anyone free to triage 1453644? lxc + CA issue of some sort. possibly a race? [14:40] katco: sure [14:40] well, keypair should actually work... it's just not actually been exercised for ever now [14:40] they don't give me a keypair though [14:41] just a pem file with a private key in it [14:41] wwitzel3: ty. i don't think we have to actually fix it just yet, so just prioritization with some idea of what might be happening [14:41] is the fingerprint in horizon considered the access-key? [14:42] I'll need to look at the dreamhost docs, I'm not sure exactly what they expect [14:42] katco: k, if it is more than a 30 minute fix after I find the problem, I'll just triage and make notes [14:42] the only cloud keypair was ever used against was hpv1 [14:42] wwitzel3: sounds good [14:42] sinzui: we're on 1453644. ty [14:43] thank you katco [14:43] http://wiki.dreamhost.com/DreamCompute_Overview this seems to be the closest I can find wrt. docs === ChanServ changed the topic of #juju-dev to: https://juju.ubuntu.com | On-call reviewer: see calendar | Open critical bugs: 1453805 [16:37] dammit, can't set instance ID on ip address when creating container because we don't have the instance id set until *after* the container is created [16:37] it needs to be set when the new machine is provisioned in state [16:37] slightly more fiddly [16:38] katco, merges are blocked by bug 1453805. I recommend reverting the bad commit, then resubmitting that commit with a proven fix [16:38] Bug #1453805: Juju takes more than 20 minutes to enable voting

[16:42] sinzui: ty. it looks like at least on master, there's a lot of PRs on top of that commit. i don't know how easy it would be to back out [16:43] katco, :/ well starting with 1.24 might be most practical. If I need to release 1.23, I can use the previous commit, and 1.23.3 is intended to be our final release for 1.23 [16:44] sinzui: even on 1.24, there's a page of prs after that pr =/ [16:44] sinzui: wondering why we landed a refactoring pr in something we're trying to release [16:44] katco, great point. [16:45] sinzui: it looks like it did fix a bug [16:45] sinzui: hmmm... it may be more practical to try and fix the issue at this point [16:45] cherylj: ping [16:45] katco, I think we were clobbered by optimism. looking at the times of the commits. I think 1.24 and master were committed to before CI to demonstrate the fix was good [16:46] sinzui: ah i see [16:46] katco: what up [16:46] cherylj: we have a blocking bug that looks like it involves some of menn0's code [16:46] cherylj: bug 1453805 [16:46] Bug #1453805: Juju takes more than 20 minutes to enable voting

[16:47] cherylj: sinzui has suggested we back it out, but from the # of prs that came after, it looks like that may be difficult [16:47] cherylj: in any branch [16:47] cherylj: wondering how hard it would be to find the issue and fix it? [16:48] katco: guess we won't know 'til we try :) I can take a look [16:48] cherylj: ty [16:54] katco: you could try to apply a reverse patch and see if everything works afterwards, it is the easiest path [16:55] perrito666: feel free to ping cherylj and see if that works [16:55] ah seems like cherylj is taking a look [16:55] :p I should refrain from commenting until I read the whole backlog [16:56] perrito666: nah, valuable info! :) [16:56] perrito666: how do you apply a reverse patch? [16:57] first you get the patch the easy way [16:57] https://github.com/juju/juju/commit/c8c41b048df0e038b75c353856d021080713408f.diff [16:57] adding .diff to the commit in gh :p [16:57] bc I am lazy to do it otterwise [16:58] and then patch -R -p1 < thatdiff.diff [16:58] and that un applies the changes [16:58] in an easy to test way that doesn t mess that much with git change tree [16:59] that .diff thing in github is so useful, its a shame its not documented more clearly [17:00] perrito666: okay, let me give that a try. Thanks! [17:31] Bug #1453890 was opened: Bootstrap fails if aws feigns ignorance of an instance

[17:39] katco, perrito666: http://reviews.vapour.ws/r/1645/ [17:39] I used patch -R to back out menn0's changes for 1.23 [17:40] local testing passed fine. [17:46] perrito666: hey, just fyi, you're lending code in charm.v6-unstable. if that's your intention all is fine, just wanted to point out that you're not adding it to charm.v5. [17:47] urulama_: I did two prs, one for 5 and one for 6 [17:50] perrito666: ok. in the end, we'll have to merge new stuff from v5 to v6 anyway [17:51] urulama_: just trying to be consistent :) [17:51] I want to avoid breakage when we go from 5 to 6 in master [17:52] perrito666: ;) [17:53] urulama_: I assume merge is by hand right? [17:54] perrito666: for charm, yes [18:31] Bug #1452785 changed: os-enable-refresh-update is false and some security packages are not available === urulama_ is now known as urulama__ [19:47] natefinch, katco: do either of you have a moment to review http://reviews.vapour.ws/dashboard/?view=outgoing [19:47] sorry http://reviews.vapour.ws/r/1648/ [19:47] sinzui: natefinch is occupied, i can tal in a bit [19:47] I looked because I know his changes are usually 2 characters :) [19:47] ship it! [19:47] ah lol [19:48] thank you. [19:52] Bug #1452796 was opened: template download failed [21:00] mramm: our call now clashes with the onyx standup [21:00] mramm: are you ok to push it out half an hour? [21:00] thumper, mramm is out this afternoon [21:15] thumper: % tail -n2 /etc/hosts [21:15] # 127.0.0.100 cloud-images.ubuntu.com [21:15] ta === mwhudson_ is now known as mwhudson [21:30] menn0: you are that fix for bug 1453805 has been backed out? [21:30] Bug #1453805: Juju takes more than 20 minutes to enable voting

[21:30] wallyworld: menn0 and I are looking at it now [21:31] thatnks cherylj [21:57] wallyworld: chatted with menn0 and he found the problem. He's going to work on a fix rather than back out his changes. [21:57] \o/ [21:57] sinzui: if you comment on the mongo log bug, i think you could move it off to 1.25 milestone at the same time? [21:58] wallyworld, yep. debian doesn't have 2.8. So if we need it or newer, we need to add a task asking for it to be packages into wily and then discuss backporting schemes to trusty as jujudb [21:59] sounds like a lot of "fun" [21:59] sinzui: if we start the process now, it might be early enough in W to get traction on it? [22:00] agreed [22:00] xwwt, ^^^ we should follow-up with foundations on this [22:01] sinzui: so i reckon if we move to 1.25, then maybe we can have a chance of getting something done by the nominal 1.25 release date [22:01] okay wallyworld [22:01] well that's my hope :-) [23:17] wallyworld: axw anastasiamac_ brt, melting some chocolate [23:17] perrito666: where's ours [23:26] menn0: what's the story with the replicaset problem? [23:28] thumper: it's my fault. the replicaset init changes don't work for new state servers added for ensure-availability [23:28] thumper: i'm just trying out a likely fix now on EC2 [23:28] menn0: ok