menn0wallyworld: can you make this PR merge please? https://github.com/juju/juju/pull/223900:00
menn0wallyworld: I think I aborted a previous build attempt in Jenkins on Friday b/c I realised it wasn't going to work00:01
menn0wallyworld: so now the landing bot is confused.00:01
wallyworldmenn0: i have resubmitted it to the bot - you just have to record a build failed message in the PR for the bot to realise it needs to process it again00:16
menn0wallyworld: ok thanks01:01
wallyworldmenn0: except it failed again01:01
wallyworldi didn't look at the reason01:01
menn0wallyworld: just looking now01:02
axwwallyworld: teensy review please: http://reviews.vapour.ws/r/1384/01:14
wallyworldaxw: i'd just self review one as small as that01:15
axwwallyworld: ok01:15
thumperwallyworld: did you have any time in the next hour for a catch up? since I missed Friday01:25
thumpermenn0: has wallyworld suitably kicked the bot?01:26
wallyworldthumper: sure, i was waiting for you and you ditched me01:26
thumpersorry, was out Friday morning01:26
wallyworldwas just pressing your buttons01:27
thumperand I have so many to press01:27
thumperwallyworld: I'm in our 1:1 hangout01:29
wallyworldok, one sec01:29
menn0thumper: yep he did01:32
mupBug #1453634 was opened: juju upgrage-juju --upload-tools hangs <ec2-provider> <upload-tools> <juju-core:Triaged> <https://launchpad.net/bugs/1453634>02:23
mupBug #1453634 changed: juju upgrage-juju --upload-tools hangs <ec2-provider> <upload-tools> <juju-core:Triaged> <https://launchpad.net/bugs/1453634>02:29
mupBug #1453634 was opened: juju upgrage-juju --upload-tools hangs <ec2-provider> <upload-tools> <juju-core:Triaged> <https://launchpad.net/bugs/1453634>02:35
waiganiaxw: does upgrade also work for 1.25 for you?03:32
axwwaigani: haven't tried03:32
waiganiaxw: I'm debugging with 1.25 now. Any chance you could try on your box?03:33
axwwaigani: ok03:33
waiganiaxw: thanks. I've tried about 4 times in a row now. same issue each time. I can ssh to machine 0, debug-log works, but it always hangs on upgrade-juju --upload-tools03:34
axwwaigani: it did take a long time to upload the tools, but it always does for me03:35
axwanyway, trying with master now03:35
axwwaigani: have you tried with 1.22 or 1.23? that should tell you if it's a regression, or if it's just your network03:36
waiganiaxw: I'll try 1.23 now.03:37
waiganiaxw: 1.23 works fine for me03:42
axwhmk, strange03:42
axwwaigani: open it back up with that then I guess. master is still bootstrapping for me ..03:44
waiganiaxw: though if it is my network, maybe it's better now. I'll test 1.23 one more time and wait to see what 1.25 does for you.03:45
axwwaigani: master works for me too03:53
waiganiaxw: really? then it's me :/03:53
waiganiaxw: looks like 1.23 is hanging. It hangs after tools are uploaded (i.e. after it reports "available tools:...").04:00
axwwaigani: you said it worked fine before?04:00
mupBug #1453644 was opened: One unit remains pending with local provider <juju-core:New> <https://launchpad.net/bugs/1453644>04:05
waiganiaxw: I'll dig a little more, but as it's only on my box I'll knock down the importance. Thanks for testing 1.25.04:06
anastasiamacwaigani: did u have a "eureka" moment?04:12
anastasiamacwaigani: or is it" high availability" moment?04:12
waiganiso I had export JUJU_DEV_FEATURE_FLAGS=jes in my .bashrc04:13
waiganiwhen I tried 1.23 the first time, it was in an old terminal window that hadn't sourced .bashrc04:14
waiganiwhen I commented out the export and tried the upgrade, it worked04:14
waiganilong story short, I think something hiding behind the jes flag is breaking upgrade-juju04:15
waiganiaxw: ^^04:16
axwwaigani: ok. please add that to the bug, change the title, drop the severity and leave it open only on whatever branch JES is being released in04:17
waiganiaxw: okay04:17
axwwallyworld: I updated TestCreateVolume to wait for status != ""04:34
axwwallyworld: createVolume returns with Status==""04:35
wallyworldaxw: i already reviewed just before04:35
wallyworldi think a test is missing04:35
wallyworldah hang on04:36
wallyworldi see what you're saying04:36
axwwallyworld: ok to land?04:38
wallyworldaxw: i guess what i was looking for was that the status would be "" until the test triggered it to be not "". otherwise the old code would also have passed the test if you know what i mean. but i guess it doesn't really matter04:39
axwwallyworld: I could change it to only return "available" after a couple of iterations04:39
axwI'll do that04:39
wallyworldaxw: and record that at least one failed attempt occurred04:40
wallyworldthat would be neat, thanks04:40
axwwallyworld: although that means I need to patch the attempt strategy.. bleh04:40
wallyworldif it's a hassle, ship it :-)04:41
wallyworldstrictly speaking, we should test for it i guess04:41
wallyworldmaybe a todo04:41
axwwallyworld: done, landing04:49
axwwallyworld: I think I might have a lie down, feeling a bit average04:49
wallyworld:-( ok, take it easy04:50
* thumper chuckles04:50
thumperwallyworld: and average is so far down from normal...04:51
thumperdamn it04:51
wallyworlddepends who you are04:51
thumperIf I stopped when I felt average, I'd never get anything done...04:51
wallyworldso you feel average a lot then04:51
* thumper is off05:24
voidspacedooferlad: stdup?09:02
voidspacedooferlad: TheMue: I got booted out09:05
voidspacedooferlad: TheMue: probably have to log back in, sorry09:05
mupBug #1453744 was opened: juju version 1.23.2-trusty-amd64  console need agent-metadata-url flag? <juju-core:New> <https://launchpad.net/bugs/1453744>10:12
mupBug #1453760 was opened: "u'Error': u'status for key "<unit>" not found" before adding relations when using juju 1.24 <juju-deployer:New> <https://launchpad.net/bugs/1453760>11:12
mupBug #1453760 changed: "u'Error': u'status for key "<unit>" not found" before adding relations when using juju 1.24 <juju-core:Triaged> <juju-core 1.24:Triaged> <juju-deployer:New> <https://launchpad.net/bugs/1453760>11:54
mupBug #1453760 was opened: "u'Error': u'status for key "<unit>" not found" before adding relations when using juju 1.24 <juju-core:Triaged> <juju-core 1.24:Triaged> <juju-deployer:New> <https://launchpad.net/bugs/1453760>12:00
mupBug #1453760 changed: "u'Error': u'status for key "<unit>" not found" before adding relations when using juju 1.24 <juju-core:Triaged> <juju-core 1.24:Triaged> <juju-deployer:New> <https://launchpad.net/bugs/1453760>12:12
mupBug #1453785 was opened: transaction collection (txns) grows without bound <stakeholder> <juju-core:Triaged> <https://launchpad.net/bugs/1453785>12:54
mupBug #1453801 was opened: /var/spool/rsyslog grows without bound <stakeholder> <juju-core:Triaged> <https://launchpad.net/bugs/1453801>13:24
mupBug #1453805 was opened: Juju takes more than 20 minutes to enable voting <ensure-availability> <intermittent-failure> <juju-core:Incomplete> <juju-core 1.23:Triaged> <juju-core 1.24:Triaged> <https://launchpad.net/bugs/1453805>13:24
mupBug #1453744 changed: juju version 1.23.2-trusty-amd64  console need agent-metadata-url flag? <juju-core:Invalid> <https://launchpad.net/bugs/1453744>13:36
mupBug #1453814 was opened: rsyslog configuration forwards log messages that were already forwarded <logging> <rsyslog> <stakeholder> <juju-core:Triaged> <https://launchpad.net/bugs/1453814>13:36
mupBug #1453744 was opened: juju version 1.23.2-trusty-amd64  console need agent-metadata-url flag? <juju-core:Invalid> <https://launchpad.net/bugs/1453744>13:48
mupBug #1453814 changed: rsyslog configuration forwards log messages that were already forwarded <logging> <rsyslog> <stakeholder> <juju-core:Triaged> <https://launchpad.net/bugs/1453814>13:48
mupBug #1453744 changed: juju version 1.23.2-trusty-amd64  console need agent-metadata-url flag? <juju-core:Invalid> <https://launchpad.net/bugs/1453744>13:54
mupBug #1453814 was opened: rsyslog configuration forwards log messages that were already forwarded <logging> <rsyslog> <stakeholder> <juju-core:Triaged> <https://launchpad.net/bugs/1453814>14:00
sinzuikatco`, : I need help triaging bug 1453644. I think an engineer needs to determine if we need to/can fix this bug for 1.24 or 1.2514:03
mupBug #1453644: One unit remains pending with local provider <deploy> <lxc> <reliability> <repeatability> <juju-core:New> <https://launchpad.net/bugs/1453644>14:03
mupBug #1453814 changed: rsyslog configuration forwards log messages that were already forwarded <logging> <rsyslog> <stakeholder> <juju-core:Triaged> <https://launchpad.net/bugs/1453814>14:06
katco`sinzui: just in a meeting, one sec14:08
=== katco` is now known as katco
voidspacealexisb: ping14:17
mupBug #1434741 changed: PruneLogs suffers from an indeterminate map iteration order bug <juju-core:Fix Released by johnweldon4> <https://launchpad.net/bugs/1434741>14:24
jcastromgz, have you tried juju with dreamhost yet?14:36
mgzjcastro: no, not recently14:36
jcastrodo you know anyone who has tried it? I'm trying to document it for the docs but I get auth errors14:37
mgzI've not seen anything on the list, I'll see if I have creds around somewhere still14:37
jcastroideally using keypairs would be lovely14:37
mgzyeah, I need to re-add that code to juju-core14:38
jcastrook so it only does userpass?14:38
jcastrothey do give me a .pem file as well14:38
katcocherylj: perrito666: wwitzel3: is anyone free to triage 1453644? lxc + CA issue of some sort. possibly a race?14:39
wwitzel3katco: sure14:40
mgzwell, keypair should actually work... it's just not actually been exercised for ever now14:40
jcastrothey don't give me a keypair though14:40
jcastrojust a pem file with a private key in it14:41
katcowwitzel3: ty. i don't think we have to actually fix it just yet, so just prioritization with some idea of what might be happening14:41
jcastrois the fingerprint in horizon considered the access-key?14:41
mgzI'll need to look at the dreamhost docs, I'm not sure exactly what they expect14:42
wwitzel3katco: k, if it is more than a 30 minute fix after I find the problem, I'll just triage and make notes14:42
mgzthe only cloud keypair was ever used against was hpv114:42
katcowwitzel3: sounds good14:42
katcosinzui: we're on 1453644. ty14:42
sinzuithank you katco14:43
jcastrohttp://wiki.dreamhost.com/DreamCompute_Overview this seems to be the closest I can find wrt. docs14:43
=== ChanServ changed the topic of #juju-dev to: https://juju.ubuntu.com | On-call reviewer: see calendar | Open critical bugs: 1453805
voidspacedammit, can't set instance ID on ip address when creating container because we don't have the instance id set until *after* the container is created16:37
voidspaceit needs to be set when the new machine is provisioned in state16:37
voidspaceslightly more fiddly16:37
sinzuikatco, merges are blocked by bug 1453805. I recommend reverting the bad commit, then resubmitting  that commit with a proven fix16:38
mupBug #1453805: Juju takes more than 20 minutes to enable voting <blocker> <ci> <ensure-availability> <intermittent-failure> <regression> <juju-core:Triaged> <juju-core 1.23:Triaged> <juju-core 1.24:Triaged> <https://launchpad.net/bugs/1453805>16:38
katcosinzui: ty. it looks like at least on master, there's a lot of PRs on top of that commit. i don't know how easy it would be to back out16:42
sinzuikatco, :/ well starting with 1.24 might be most practical. If I need to release 1.23, I can use the previous commit, and 1.23.3 is intended to be our final release for 1.2316:43
katcosinzui: even on 1.24, there's a page of prs after that pr =/16:44
katcosinzui: wondering why we landed a refactoring pr in something we're trying to release16:44
sinzuikatco, great point.16:44
katcosinzui: it looks like it did fix a bug16:45
katcosinzui: hmmm... it may be more practical to try and fix the issue at this point16:45
katcocherylj: ping16:45
sinzuikatco, I think we were clobbered by optimism. looking at the times of the commits. I think 1.24 and master were committed to before CI to demonstrate the fix was good16:45
katcosinzui: ah i see16:46
cheryljkatco: what up16:46
katcocherylj: we have a blocking bug that looks like it involves some of menn0's code16:46
katcocherylj: bug 145380516:46
mupBug #1453805: Juju takes more than 20 minutes to enable voting <blocker> <ci> <ensure-availability> <intermittent-failure> <regression> <juju-core:Triaged> <juju-core 1.23:Triaged> <juju-core 1.24:Triaged> <https://launchpad.net/bugs/1453805>16:46
katcocherylj: sinzui has suggested we back it out, but from the # of prs that came after, it looks like that may be difficult16:47
katcocherylj: in any branch16:47
katcocherylj: wondering how hard it would be to find the issue and fix it?16:47
cheryljkatco: guess we won't know 'til we try :)  I can take a look16:48
katcocherylj: ty16:48
perrito666katco: you could try to apply a reverse patch and see if everything works afterwards, it is the easiest path16:54
katcoperrito666: feel free to ping cherylj and see if that works16:55
perrito666ah seems like cherylj is taking a look16:55
perrito666:p I should refrain from commenting until I read the whole backlog16:55
katcoperrito666: nah, valuable info! :)16:56
cheryljperrito666: how do you apply a reverse patch?16:56
perrito666first you get the patch the easy way16:57
perrito666adding .diff to the commit in gh :p16:57
perrito666bc I am lazy to do it otterwise16:57
perrito666and then patch -R -p1 < thatdiff.diff16:58
perrito666and that un applies the changes16:58
perrito666in an easy to test way that doesn t mess that much with git change tree16:58
perrito666that .diff thing in github is so useful, its a shame its not documented more clearly16:59
cheryljperrito666: okay, let me give that a try.  Thanks!17:00
mupBug #1453890 was opened: Bootstrap fails if aws feigns ignorance of an instance <bootstrap> <ec2-provider> <juju-core:Triaged> <https://launchpad.net/bugs/1453890>17:31
cheryljkatco, perrito666: http://reviews.vapour.ws/r/1645/17:39
cheryljI used patch -R to back out menn0's changes for 1.2317:39
cheryljlocal testing passed fine.17:40
urulama_perrito666: hey, just fyi, you're lending code in charm.v6-unstable. if that's your intention all is fine, just wanted to point out that you're not adding it to charm.v5.17:46
perrito666urulama_: I did two prs, one for 5 and one for 617:47
urulama_perrito666: ok. in the end, we'll have to merge new stuff from v5 to v6 anyway17:50
perrito666urulama_: just trying to be consistent :)17:51
perrito666I want to avoid breakage when we go from 5 to 6 in master17:51
urulama_perrito666: ;)17:52
perrito666urulama_: I assume merge is by hand right?17:53
urulama_perrito666: for charm, yes17:54
mupBug #1452785 changed: os-enable-refresh-update is false and some security packages are not available <juju-core:Invalid> <https://launchpad.net/bugs/1452785>18:31
=== urulama_ is now known as urulama__
sinzuinatefinch, katco: do either of you have a moment to review http://reviews.vapour.ws/dashboard/?view=outgoing19:47
sinzuisorry http://reviews.vapour.ws/r/1648/19:47
katcosinzui: natefinch is occupied, i can tal in a bit19:47
natefinchI looked because I know his changes are usually 2 characters :)19:47
natefinchship it!19:47
katcoah lol19:47
sinzuithank you.19:48
mupBug #1452796 was opened: template download failed <juju-core:New> <https://launchpad.net/bugs/1452796>19:52
thumpermramm: our call now clashes with the onyx standup21:00
thumpermramm: are you ok to push it out half an hour?21:00
alexisbthumper, mramm is out this afternoon21:00
davecheneythumper: % tail -n2 /etc/hosts21:15
davecheney#   cloud-images.ubuntu.com21:15
=== mwhudson_ is now known as mwhudson
wallyworldmenn0: you are that fix for bug 1453805 has been backed out?21:30
mupBug #1453805: Juju takes more than 20 minutes to enable voting <blocker> <ci> <ensure-availability> <intermittent-failure> <regression> <juju-core:Triaged by cherylj> <juju-core 1.23:Triaged by cherylj> <juju-core 1.24:Triaged by cherylj> <https://launchpad.net/bugs/1453805>21:30
cheryljwallyworld: menn0 and I are looking at it now21:30
wallyworldthatnks cherylj21:31
cheryljwallyworld: chatted with menn0 and he found the problem.  He's going to work on a fix rather than back out his changes.21:57
wallyworldsinzui: if you comment on the mongo log bug, i think you could move it off to 1.25 milestone at the same time?21:57
sinzuiwallyworld, yep. debian doesn't have 2.8. So if we need it or newer, we need to add  a task asking for it to be packages into wily and then discuss backporting schemes to trusty as jujudb21:58
wallyworldsounds like a lot of "fun"21:59
wallyworldsinzui: if we start the process now, it might be early enough in W to get traction on it?21:59
alexisbxwwt, ^^^ we should follow-up with foundations on this22:00
wallyworldsinzui: so i reckon if we move to 1.25, then maybe we can have a chance of getting something done by the nominal 1.25 release date22:01
sinzuiokay wallyworld22:01
wallyworldwell that's my hope :-)22:01
perrito666wallyworld: axw anastasiamac_ brt, melting some chocolate23:17
wallyworldperrito666: where's ours23:17
thumpermenn0: what's the story with the replicaset problem?23:26
menn0thumper: it's my fault. the replicaset init changes don't work for new state servers added for ensure-availability23:28
menn0thumper: i'm just trying out a likely fix now on EC223:28
thumpermenn0: ok23:28

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!