/srv/irclogs.ubuntu.com/2014/06/27/#juju-dev.txt

=== Ursinha is now known as Ursinha-afk
thumpermenn0: found anything yet?01:05
menn0thumper: still try to replicate it locally01:05
thumperok.01:06
menn0did you notice that the all-machines.log build artifact appears to be imcomplete?01:06
menn0incomplete even01:06
* thumper needs to do the self-review hr thingy01:06
thumpermenn0: no... didn't look in too much detail01:06
menn0it doesn't even get to the upgrade to 1.1901:06
menn0thumper: so things don't look awesome01:17
thumperin what way?01:17
menn0thumper: I've just done an upgrade from 1.18.4 to 1.19.4 and the env is pretty screwed01:17
menn0no idea if it's the same problem as what showed up in CI01:18
menn0the main thing seems to be that the addresses for machine-1 and machine-2 are now 127.0.0.1 instead of what they should be.01:18
=== Ursinha-afk is now known as Ursinha
menn0does that ring any bells01:19
thumperno, but we do have an address updater worker thingy01:19
thumperI would have just started with no other machines and just done the bootstrap node01:19
thumpersee if that works01:20
thumperthen add machines01:20
menn0also, during the upgrade the db-relation-changed hook for mysql failed (I was testing with the standard wp / mysql setup)01:20
menn0thumper: the problem is my machine is trusty and the problem is supposedly precise specific01:20
thumperis it?01:21
thumperI have a precise machine here01:21
menn0according to the bug01:21
menn0I was about to test with canonistack because I have that set up and it's all precise there01:21
thumperok, try that too01:21
menn0at any rate, what I did should have been fine and it definitely wasn't01:21
menn0even if it's not the same bug as what showed up in CI it's pretty bad01:22
menn0the CI test uses some simple dummy charms. I'll test with them on canonistack, replicating what CI does as closely as I can01:23
thumperok01:26
wallyworldmenn0: i think this also 1.20 bug could be related also https://bugs.launchpad.net/bugs/133477301:32
_mup_Bug #1334773: Upgrade from 1.19.3 to 1.19.4 cannot set machineaddress <lxc> <maas-provider> <precise> <upgrade-juju> <juju-core:Triaged> <https://launchpad.net/bugs/1334773>01:32
menn0wallyworld: could be01:36
wallyworldmenn0: can you keep me in the loop as to where you get to?01:36
menn0wallyworld: will do01:37
wallyworldta, i don't think that first bug is related necessarily, but i suspect the other could be01:37
thumpermenn0: do you want some help diagnosing this issue?01:48
* thumper just typed 'juju fetch upstream'01:49
thumperunsurprisingly it didn't work01:49
* thumper considers writing a juju-fetch plugin01:49
thumperhow do I tell git to have my 1.20 branch track upstream 1.20?01:51
thumpero/ mramm01:51
* thumper branches 1.2001:56
menn0thumper: just got back from lunch02:05
menn0thumper: yes please02:05
thumpermenn0: well there is certainly something wrong02:05
thumperi'm looking at my current setup02:06
thumpero..m..g...02:07
thumperhmm...02:07
menn0thumper: hangout?02:07
thumperyeah...02:07
thumpermenn0: https://plus.google.com/hangouts/_/canonical.com/local-debugging02:08
axwwallyworld: I'm back, ready for 1:1 whenever you are02:18
wallyworldok02:18
=== Ursinha is now known as Ursinha-afk
bodie_I'm digging in the worker/uniter/uniter_test a bit02:23
bodie_this is very unusual02:23
bodie_does anyone know how to pass values around between steps?02:23
bodie_I guess you'd have to set a value on the ctx struct?02:24
=== Ursinha-afk is now known as Ursinha
menn0axw: are you around? at least one of the upgrade issues appears to be mongo replicaset related...03:03
axwmenn0: I am03:03
axwoh? :(03:03
menn0axw: https://plus.google.com/hangouts/_/canonical.com/local-debugging03:03
axwbrt03:03
waiganiaxw: thanks03:06
waiganiaxw: setting locale to default of UTF8 for charm hooks. Is that just a matter of setting an environment variable  LANG=en_US.UTF8 ?03:13
axwwaigani: yup03:13
axwwaigani: sorry gonna need a fix to the known_hosts path quoting03:19
axwwaigani: left a comment on the PR03:20
waiganiaxw: ah shit, sorry I hit merge03:20
axwnps, it's not an immediate problem03:20
waiganiaxw: I ran the test and it did not quote03:21
axwwaigani: otp, I'll take a look in a bit03:22
waiganiaxw: I think I've got it, using utils.CommandString in ssh_openssh.go03:28
waiganiaxw: okay fixed, I pushed up changes. What happens, considering I already started the merge?03:36
axwwaigani: kinda dodgy now that I think of it, but if your original change passes it'll get merged :)03:38
axwwaigani: sorry, I don't think you need to quote the string at all in utils/ssh03:44
axwwaigani: it gets passed to os/exec.Command, not to a shell03:45
waiganiaxw: oh right. I'll take out the utils.CommandString then03:46
waiganiaxw: and not test for quotations? have a file name with no spaces?03:47
axwwaigani: sec03:47
axwwaigani: the way to test the quoting would be to update fakecommand in ssh_test.go03:48
axwwaigani: surrounding $@ with double quotes should make it quote the args03:48
waiganiaxw: $@ already has double quotes03:51
axwwaigani: not in my branch. I have: echo $@ | tee $0.args03:52
waiganiaxw: hehe, I was looking in utils03:53
waiganiaxw: okay I'll have a play03:54
waiganiaxw: just spotted scp tests failing04:11
waiganiaxw: I'll fix those and push up again04:11
thumpermenn0,axw: did you find it?04:14
menn0axw: not yet but axw is pretty sure those asserts are it04:15
menn0axw wanted to get some food so we broke the call04:15
menn0I'm currently trying to understand why the problem has stopped happening for me04:16
menn0I have a theory which I'm testing now04:16
axwthumper: just making some lunch, will bbs04:16
thumperaxw: ack04:16
thumpermenn0: because the asserts aren't failing?04:17
thumpernow04:17
menn0but they should be if I'm upgrading from 1.18 to 1.19 right?04:17
menn0thumper: where are those asserts?04:19
thumperstate/machine.go:97504:20
waiganiaxw: okay, should be good to go now.04:27
menn0thumper: I can't make the problem happen again. Trying something else...04:35
thumpermenn0, axw: so looking at the difference for me, the doc doesn't have Scope values in the db, and we are trying to set with scope values04:40
thumperand the new list has one more ipv6 address04:40
thumperhowever that doesn't explain why the transaction is aborted04:40
axwthumper: shouldn't matter what we're setting, only what we're comparing between the in-memory and in-db04:40
thumperhmm...04:41
thumpermay have it...04:41
thumpermaybe04:41
thumperwe look at the actual serialized data we have...04:41
thumperbut set using a string value in bson.D04:42
thumperwhat is the structure serialized as?04:42
thumpercould that bit it?04:42
axw"address"04:42
axwI compared the structs between 1.18 and 1.19 and didn't see a difference04:42
axwthumper: only difference is change in location of the structs... don't *think* that matters though...04:43
thumperwell, it shouldn't, but it might04:43
thumperaxw: is there any way to dump at the raw bson structure ?04:44
thumperor, where are the bson serialisation commands?04:44
axwthumper: yeah I was about to figure out how :)  there's an mgo.Raw type like in encoding/json04:44
thumperhere is a simpler case:04:45
thumper2014-06-27 04:32:09 DEBUG juju.state machine.go:931 addresses currently: []state.address{state.address{Value:"localhost", AddressType:"hostname", NetworkName:"", Scope:""}, state.address{Value:"10.0.3.1", AddressType:"ipv4", NetworkName:"", Scope:""}}04:45
thumper2014-06-27 04:32:09 DEBUG juju.state machine.go:978 updating addresses to: []state.address{state.address{Value:"localhost", AddressType:"hostname", NetworkName:"", Scope:"public"}, state.address{Value:"10.0.3.1", AddressType:"ipv4", NetworkName:"", Scope:"local-cloud"}}04:45
thumper2014-06-27 04:32:09 DEBUG juju.state.txn txn.go:91 0: err: &errors.errorString{s:"transaction aborted"}04:45
thumperupdating to add scope04:46
thumperand that is all04:46
thumperorder is less likely to be an issue here with just two values04:46
* thumper has to go and run to get sushi04:47
thumperbefore they run out04:47
thumperI'll check in when I get back04:47
=== thumper is now known as thumper-afk
menn0axw, thumper: I need to do a takeaway run myself but will join in again04:59
axwmenn0 thumper-afk: found it05:09
axwa field name in state.address was changed05:09
axwNetworkScope -> Scope05:10
axwdimitern: I'm going to change the field name of state.address.Scope back to NetworkScope as it was before. let me know if you can think of any problem with that05:10
dimiternaxw, why?05:14
axwdimitern: because it used to be called networkscope in state05:14
axwdimitern: the change breaks upgrade05:14
dimiternaxw, ah.. dreaded schema changes05:16
dimiternaxw, ok, can you make it Scope string `bson:"networkscope"` ?05:17
axwdimitern: yup, that's what I've done05:17
axwlooks like there's no queries on that field05:17
axwso should be fine05:17
dimiterngreat05:18
axwdimitern: https://github.com/juju/juju/pull/183 please05:20
dimiternaxw, looking05:21
dimiternaxw, done05:22
axwthanks05:24
=== thumper-afk is now known as thumper
thumperaxw: ah, so we were read the data, scope now blank, assert a dict which now didn't match, right?05:30
axwthumper: something like that. we expected "scope", but state had "networkscope"05:32
axwsame value, different field name05:32
thumperso our assertion failed05:32
axwyup05:32
axwuh oh, the merge bot picked up my 1.20 PR, is going to test it on trunk, and then land it in 1.2005:40
axwoh well, it's trivial05:40
thumperheh05:40
axwwaigani: sorry, looking at your changes now05:41
thumperdavecheney: making progress on bug 1334493 ?05:47
_mup_Bug #1334493: Cannot compile/exec win client <regression> <windows> <juju-core:Triaged> <https://launchpad.net/bugs/1334493>05:47
davecheneythumper: yup05:53
davecheneyhttps://github.com/juju/juju/pull/18606:22
wallyworldaxw: my irc sucks today. seems like you're making progess on bug 1334773?06:22
davecheneyshould fix the faulty tools06:22
_mup_Bug #1334773: Upgrade from 1.19.3 to 1.19.4 cannot set machineaddress <lxc> <maas-provider> <precise> <upgrade-juju> <juju-core:In Progress by axwalk> <https://launchpad.net/bugs/1334773>06:22
axwwallyworld: yeah, I have a fix in the works06:22
axwbeing merged06:22
wallyworldgreat06:23
axwmerged in fact06:23
wallyworld\o/06:23
menn0thumper, axw: do you think it's possible that bug 1334273 may also be caused by the Scope problem?06:36
_mup_Bug #1334273: Upgrades of precise localhost fail <local-provider> <precise> <regression> <upgrade-juju> <juju-core:Triaged> <https://launchpad.net/bugs/1334273>06:36
axwmenn0: possibly related, but it's not local-specific, and definitely not precise-specific06:37
menn0axw: yeah it's hard to see how it the networkscope issue could be that specific.06:39
menn0I still don't see why it magically stopped happening on my machine though. It doesn't seem like the problem should have been intermittent.06:40
menn0axw: the symptoms of the precise local upgrade problem look a lot like what Tim saw when upgrading from 1.19.1 to 1.19.4 and that was the peer grouper setup not happening.06:41
axwyeah...06:41
axwhmm06:41
menn0axw: but the CI test was going from 1.18.4 to 1.19.4 so it's not quite the same.06:41
wallyworldmenn0: when i looked at the logs on CI, there were no obvious errors logged by the peer grouper when it failed06:42
wallyworldit seems it could related to going from non-ha to ha set up, and on precise we use an older mongo06:42
menn0wallyworld: sorry, it's not the peergrouper itself but the replicaset configuration that's done at startup for the peergrouper (it uses code in the peergrouper's module)06:43
wallyworldwhich has previously caused issues with local provider, hence it was disabled for a while06:43
wallyworldyup06:43
wallyworldmy handwavy view is that mongo+precise+replicaset = shit06:43
wallyworldbut i have little evidence06:44
axwwallyworld: was there ever a known problem with the older version of mongo? the only thing I absolutely know for sure that caused a problem was the oplog size06:44
wallyworldnot sure06:44
wallyworldbut the only thing that i can see as different on precise is the version of mongo06:45
menn0wallyworld: we found today that upgrading from 1.19 before the replicaset work was done in this series to 1.19.4 causes broken upgrades because the replicaset setup is only done if you're upgrading from pre-1.19.0.06:45
wallyworldjoy06:45
menn0wallyworld: the thing is the 1.20 blocker for precise upgrades looks awfully similar.06:46
wallyworldbut that's "ok"06:46
wallyworldwe don't support upgrades for dev versions06:46
menn0very similar symptoms in the logs06:46
wallyworldmenn0: so my view is replicaset is broken somehow, seems like you agree?06:46
menn0so I wonder if replicaset setup failed for some other reason06:46
menn0wallyworld: I'm saying there could be a problem initialising the replicaset in some cases.06:47
wallyworldif it works for straight uo deploy from scratch, i wonder what's different when upgrading06:47
menn0I need to go again for a bit but I'll dig through the CI failure again when I'm back06:47
=== vladk|offline is now known as vladk
menn0something in the logs might jump out now that there's theory in to what might be happening.06:48
wallyworldyeah06:48
menn0back in 30 mins ish06:48
wallyworldi'll try and look after soccer06:48
wallyworldbut i want to talk to william also06:48
wallyworldso may not get time06:49
axwwallyworld: there's a bug in MachineAgent.ensureMongoServer, but I don't know if it's related or not. If we get as far as ensureMongoServer, but then maybeInitiateMongoServer fails (or something in between), then the process would exit and restart06:51
axwwallyworld: we'd then try the isPreHAVersion block again and fail to connect to state06:51
axwwallyworld: the reason being that we haven't yet initiated the replicaset, and haven't told mgo to make a Direct connection06:52
menn0axw: I'm hunting through the logs from the upgrade failure in CI again. Nothing yet.07:23
axwmenn0: yeah nothing jumped out at me. I've put up https://github.com/juju/juju/pull/187 - it's a long shot, but possibly related07:23
menn0axw: just had a look at that PR. Seems reasonable. That applies even with a single state server right?07:25
axwmenn0: yes, that's the only time it actually will be uesd (pre-HA)07:26
menn0axw: duh. of course :)07:29
menn0axw: have you tried a 1.18 to 1.19 upgrade with the changes in that PR?07:29
axwmenn0: yes, just tried and it works fine07:30
menn0axw: cool, I'll LGTM it.07:31
axwcheer07:31
axws07:31
menn0axw: done07:32
menn0axw: one thing that jumps out at me about the failure in CI is that it takes 8 minutes from the time jujud restarts to the new version before MaybeInitiateMongoServer gets called. That seems awfully long.07:33
axwhmm07:34
axwit does seem like a long time...07:35
menn0another thing: would we expect the mongo admin user to get set up once jujud restart into the new software version?07:41
menn0mongo was started with --noauth and all that07:41
menn0axw: ^^^07:41
menn0in the CI test failure07:41
axwmenn0:  adding the admin user is part of the expected upgrade procedure07:42
menn0axw: ok... well in the 1.18 to 1.19 upgrades on my machine which went without a hitch I see no evidence of that happening. But it did happen in the failed CI upgrade run.07:44
menn0Could be coincidence but maybe not07:44
dimiternaxw, still around?07:46
axwdimitern: yes07:47
dimiternaxw, I was wondering about the progress on relation addresses wrt charms07:47
dimiternaxw, there were some unresolved comments on the doc - did you reach agreement?07:47
dimiternaxw, about the new hooks and stuff07:48
axwdimitern: not 100%. fwereade had a chat with hazmat and came to a vague agreement. I've got a PR up atm that triggers config-changed on units whenever the machine addresses change - will do relation addresses later07:48
dimiternaxw, i have to prepare a doc to sync up on how to expose IPv6 addresses to charms, as this is the most important take on ipv6 support in core from charmers perspective07:48
dimiternaxw, right, i'll ping fwereade and hazmat, cheers07:49
axwah, that'll be interesting...07:49
axwmenn0: did you get the full log for CI?07:50
axwmenn0: I upgraded 1.18.1 to 1.19.4 on my machine and I got "starting mongo with --noauth" and "setting admin password" on upgrade07:51
menn0axw: no. all-machines.log at least is going to be incomplete because of the rsyslogd config changes during the upgrade. the separate machine logs are fine though.07:52
menn0axw: I think I see a race07:52
axwah I see07:52
axwrace? where?07:52
menn0axw: if the upgrade-steps worker finishes before ensureMongoServer is called by the state worker then the isPreHAVersion check will be false and we won't do the HA setup work.07:53
menn0Does that sound plausable?07:53
menn0upgrade-steps updates UpgradedToVersion once it's done07:53
menn0and that is what ensureMongoServer is checking against07:53
menn0might explain why I'm only able to see the issue sometimes.07:54
axwmenn0: well shit, I think you're right07:54
axwwe shouldn't do anything until we've upgraded mongo07:55
menn0by ensureMongoServer I mean the the method not the function with the same name it calls07:55
axwyep07:55
axwmenn0: that would better explain this bug07:56
menn0axw: and possibly some of the other replicaset weirdness we've seen?07:56
axwmenn0: hmm actually...07:56
axwah never mind07:57
axwyep, still looks like the culprit07:57
axwmenn0: dunno about other weirdness, this is upgrade specific07:57
menn0ah ok. I thought the other issues were upgrade specific too.07:57
menn0axw: so what's the fix? Do the mongo upgrade work before starting any of the workers?07:58
* menn0 suspects he needs to go so lest he upsets his wife07:59
menn0s/so/soon/07:59
axwmenn0: I can handle it, it's pretty late there07:59
axwbut yes I think that's the solution07:59
menn0axw: sweet.08:00
menn0have a good weekend.08:00
axwcheers, you too08:01
* menn0 is relieved that the day ended with something productive08:01
rogpeppe2if anyone has some time, i'd very much like a review of this pull request by someone on core, please. It's currently only been reviewed by gui people. https://github.com/juju/charm/pull/908:12
=== vladk is now known as vladk|offline
fwereadeaxw, ping08:22
axwfwereade: pong08:22
fwereadeaxw, any insight into https://bugs.launchpad.net/juju-core/+bug/1334683 ?08:22
_mup_Bug #1334683: juju machine numbers being incorrectly assigned <azure-provider> <juju-core:Triaged> <https://launchpad.net/bugs/1334683>08:22
axwfwereade: fraid I've been bogged down with critical 1.20 things to even look into it08:22
axwno ideas off the top of my head08:23
fwereadeaxw, no worries, it's only 18h old :)08:23
fwereadeaxw, any suggestions for someone awake now/soon who'd know azure?08:23
axwfwereade: I don't know if anyone other than me does08:24
axwfwereade: I can take a look if someone wants to take over this jujud upgrade race08:24
axwfwereade: menn0 noticed that the upgrade steps could theoretically complete before the state worker starts, causing isPreHA to return false and things to go a bit pear shaped08:25
axwfwereade: eh I think I see the problem in azure08:27
axwfwereade: in the 1.18 code it's not returning the instances in the same order as the input ids08:27
axwfwereade: good news is, it's fixed in 1.19.408:28
axwor 1.19.0 or whenever I made all the changes08:28
fwereadeaxw, you're too awesome, I'd resigned myself to hassling nate about it :)08:28
fwereadeaxw, would yu add a really quick note to the bug to that effect please?08:28
axwwill do08:28
bacfwereade: i've been trying to get an azure instance to bootstrap with trusty with no success for a day. i've a partially booted node up. who might know a thing or two about azure and have a look?09:31
fwereadebac, if axw is still up he's your best bet by a long way09:32
axwbac: are you using 1.19.x?09:33
bacaxw: 1.19.409:33
axwbac: ok, that's a good start. you've "partially booted a node up"?09:33
bacaxw: i bootstrap with --debug and 'Running apt-get upgrade' is the last thing in the console09:33
axwhow? where did it break?09:33
axwok09:33
* axw checks he can bootstrap09:34
bacaxw: i can give you access to the instance09:34
axwbac: sure, just import lp:~axwalk please09:34
bacaxw: juju-azure-ci3-7ul3u8075q.cloudapp.net09:36
axwta09:36
axwbac: the agent is running... what happens when you run juju status?09:38
bacaxw: instance-state remains ReadyRole09:39
bacaxw: let me paste the whole thing.  until then, here is my yaml config block https://pastebin.canonical.com/112600/09:39
axwthanks09:39
bacaxw: https://pastebin.canonical.com/112601/09:39
axwbac: looks fine - what's the issue with it?09:40
bacaxw: well 'juju bootstrap' never terminates.  if i ctl-c out of it, the instance is torn down09:41
axwah right. hmm weird09:41
bacaxw: i assumed since it didn't complete it was still doing stuff.  and i don't know what ReadyRole is09:41
bacso it was less than comforting09:41
axwbac: that's just azure's term for "machine is started/running"09:42
bacok09:42
baci guess if you had to jam two random words together those are as good as any09:43
bacaxw: so is my use of --debug possibly the culprit in the non-termination of the bootstrap?09:43
axwbac: I don't think so, I do that all the time...09:44
axwI'm just bootstrapping my own instance now09:44
axwbac: when you say you've been trying to get an azure instance to bootstrap for a day... do you mean that you left that one command running for a day, or you've tried it a bunch of times?09:45
bacaxw: i tried it a bunch of times.  most was on US East and i got different errors.  last night i switched to US West and launched the bootstrap at my eod.  this one has been active overnight09:46
axwmk09:47
bacaxw: upgraded from 1.19.3 in the middle of my attempts09:47
bacaxw: if i cannot get it resolved this morning, we're going to have to switch our CI setup to another provider.09:48
axwbac: are you doing this on Linux?09:49
axwyour client09:49
bactrusty vm09:49
axwand does it have ssh installed?09:50
bacyes09:50
axwbac: uh oh, I have reproduced this issue09:55
bacyay, boo.09:55
axwbac: I just realised I had "image-stream: daily" set in my environments.yaml; I took it out and bootstrap is just hanging there09:55
axwfwereade: ^^ critical bug for 1.20 I think09:56
* fwereade reads back with an unhealthy sense of impending freakout09:57
bacaxw: cool, so i can add image-stream to my config and perhaps get past, for testing purpose only, of course.09:57
fwereadeaxw, bac: so, wait, is this juju or the image that's messed up?09:58
bacfwereade: i can't say.09:59
axwfwereade: has to be juju. the machine agent is coming up, but the bootstrap client is just hanging there09:59
axwlooks like someone already reported this: https://bugs.launchpad.net/juju-core/+bug/131618509:59
_mup_Bug #1316185: juju bootstrap hangs on Azure <juju-core:Triaged> <https://launchpad.net/bugs/1316185>09:59
axwgonna try again with image-stream: daily back in10:00
bacyep, that's the same issue10:01
bacaxw: i'll tear down that instance and try again with image-stream: daily to confirm from the other direction10:03
bacaxw: unless you want the instance to remain up10:03
axwbac: nope, go for it10:04
axwthanks10:04
bacaxw: fwiw, ctl-c yields10:04
bac^C2014-06-27 10:04:21 INFO juju.cmd cmd.go:113 Interrupt signalled: waiting for bootstrap to exit10:04
bac2014-06-27 10:04:21 ERROR juju.provider.common bootstrap.go:119 bootstrap failed: subprocess encountered error code 13010:04
bacStopping instance...10:04
axwbac: thanks, 130 just means it was killed by Ctrl-C10:05
axwbac fwereade: yeah, putting "image-stream: daily" back in fixed it for me10:11
axwbugger.10:11
fwereadeaxw, I has a very confused -- so juju is depending on something in the daily image that's not in the released one?10:13
axwfwereade: I dunno what the difference is. The thing is, juju actually bootstraps and works with the released images. It's just that the ssh client doesn't see any output from the script past "apt-get upgrade"10:14
axwand doesn't get EOF10:14
axwfwereade: I guess one of the packages that gets upgraded is buggering up the communication10:14
axwabout to have a look at what the differences are10:15
bacaxw: confirmed that i can boot cleanly with image-stream: daily10:15
axwthanks bac10:15
* bac breakfast10:16
=== vladk|offline is now known as vladk
axwfwereade: is there some reason other than "we'd like people to have up-to-date machines" for running "apt-get upgrade" at bootstrap?10:29
bacaxw: this is almost certainly unrelated, but when i was trying to boot yesterday on US East i was getting repeated errors like http://paste.ubuntu.com/7707189/ -- it certainly muddied the waters.10:30
axwbac: fixed on trunk today10:30
bacty10:30
fwereadeaxw, apart from the fact that we install stuff and it's generally preferred that we update before doing so, I can't think of one10:34
axwfwereade: bootstrapping with the daily images, at least on azure, is considerably faster without it10:36
axwwhether it's considerably more broken, I don't know :)10:36
fwereadeaxw, so does it work if you drop the apt-get update?10:39
fwereadeaxw, /upgrade10:40
axwtrying now10:40
axwalso going to try holding back bash and apt10:40
fwereadecheers10:41
dimiternjam1 (if you're here), vladk, standup?10:48
vladkdimitern: yep10:48
axwfwereade: works without apt-get update10:50
perrito666morning11:05
natefinchmorn11:06
natefinching11:06
natefinchtab completion should work on the word I'm thinking of11:06
TheMuehel11:12
TheMuelo11:12
TheMuena11:12
TheMuetefi11:12
TheMuench11:12
TheMue;)11:12
wwitzel3I do that in the shell sometimes, trying to a tab complete a URL I'm calling curl with.11:32
perrito666yeah, I do it with passwords11:33
bachi mgz12:03
rogpeppe3dimitern: any chance you could review this for me? it's blocked until i can get a review from someone from juju-core: https://github.com/juju/charm/pull/912:07
=== rogpeppe3 is now known as rogpeppe
bacfwereade: now that axw has finished up, is anyone working now who i can ask about azure issues?12:12
axwbac: I'm still around for the moment...12:17
rick_h_axw: so in azure mode there's no manual placement. Is that the default in azure then? This means no machine view/colocating?12:18
bacoh, hi axw.  had a question about azure availability sets.  just found a link to the doc12:18
axwrick_h_: correct12:18
rick_h_axw: can you do any form of colocation at all?12:18
axwnot in the current implementation12:18
axw(unless you disable availability sets)12:19
fwereadeaxw, rick_h_: yeah, I thought that if you used rubbish-mode you could still do manual placement12:19
rick_h_axw: and is this somehting you can change after bootstrap? I see it's a 'bootstrap attribute' so is there a flag on juju bootstrap? Any other way to change afterwards?12:19
fwereadeaxw, rick_h_: forgive the looseness of my terminology12:19
axwrick_h_: no, it's immutable12:19
rick_h_wow, ok will process and ponder. This will definitely cause some fun with our current machine view work and GUI along with other projects.12:20
axwrick_h_: in case I wasn't clear, it's configurable at bootstrap time only, and immutable thereafter12:24
rick_h_axw: gotcha.12:24
rick_h_axw: just :(12:24
rick_h_axw: but will spend some time on it thinking through it and it's chain of effects.12:25
axwit's a bit of a PITA, I know, but azure's model dictates this12:25
bacaxw: is there a switch to juju bootstrap to turn it off?12:25
axwbac: yeah you can set availability-sets-enabled=false in environments.yaml12:25
bacaxw: ah, ok12:25
perrito666wwitzel3: ericsnow is any of you taskless?12:44
wwitzel3perrito666: nope, I'm working on tests for the env client api stuff and then moving on to the legacy setmongopassword cleanup.12:45
wallyworld__fwereade: did you want a quick chat about charm storage?12:46
fwereadewallyworld__, sure, 5 mins?12:46
wallyworld__fwereade: ok, join me in https://plus.google.com/hangouts/_/canonical.com/tanzanite-daily12:47
wallyworld__rick_h_: did you want to join too?12:47
rick_h_wallyworld__: joining12:47
bodie_morning all12:49
perrito666well ericsnow if you are interested we need to implement this suggestion by rogpeppe "you could add a jujud subcommand which updates the addresses in the agent.conf file"12:49
fwereadeperrito666, ericsnow: doesn't `juju endpoints` have that effect anyway?12:50
fwereadeperrito666, ericsnow: if not, it should, because *every* command that hits the api ought to update the .jenv addresses if they've changed12:50
perrito666fwereade: tell me/us more, the idea is to be able to let the agents know that the state server has changed12:51
fwereadeperrito666, oh! wait, agent.conf?12:51
perrito666:)12:52
fwereadeperrito666, that should not be a command12:52
fwereadeperrito666, we update them on initial connect already12:52
fwereadeperrito666, what we should have is a watcher that updates them when they change12:52
perrito666fwereade: this is something we need to do after a restore12:52
fwereadeperrito666, ah, ok I see, sorry, I misread two separate things12:53
perrito666so the idea is, state server hung and was killed and we restored, agents have no clue on what the state server is12:53
fwereadeperrito666, just ignore me, if you haven't learned that lesson already12:53
perrito666lol12:53
fwereadeperrito666, jujud command makes sense12:53
perrito666is at least an improvement from current method, which is ssh+sed12:54
bacmgz you around?13:02
=== alexisb_afk is now known as alexisb
alexisbnatefinch, fwereade either you available to join the cloudbase call?13:07
fwereadealexisb, just finishing up another call, in there in a sec13:08
rogpeppeplease, would someone be able to review this code?! https://github.com/juju/charm/pull/913:09
alexisbawesome thankyou13:09
rogpeppenatefinch, wwitzel3, jam1, wallyworld__, axw, dimitern, mgz: ^13:14
axwsorry I'm logging off - will take a look on monday if it's still unreviewed13:15
fwereadewallyworld__, are you still around?13:15
wallyworld__yeah13:15
axwfwereade: I give up on the azure problem for tonight. I ended up modifying bootstrap to upgrade the packages individually, and it worked :/13:15
* axw logs off13:16
wallyworld__fwereade: but i'm too tired to really review anything13:16
fwereadewallyworld__, np, you're not needed :)13:16
wallyworld__so situation normal then13:16
natefinchalexisb: coming13:17
sinzuiCan someone explain or fix  me so that we can merge the juju version changes for master and 1.20. Juju CI will not test something that it knows is released13:22
sinzuihttps://github.com/juju/juju/pull/181 and https://github.com/juju/juju/pull/18013:22
rogpeppe*still* looking for a review of this, please: https://github.com/juju/charm/pull/913:44
=== vladk is now known as vladk|offline
bacsinzui: turns out this was the bug that i was hitting yesterday: bug 131618514:05
_mup_Bug #1316185: juju bootstrap hangs on Azure <juju-core:In Progress by axwalk> <juju-core 1.20:In Progress by axwalk> <https://launchpad.net/bugs/1316185>14:05
sinzuiDon't use daily14:06
=== vladk|offline is now known as vladk
natefinchperrito666: how's your windows knowledge?14:08
* perrito666 sees an avalanche comming14:09
perrito666natefinch: I know my way around windows 7 I rule at windows xp :p14:09
perrito666I might work well enough with windows server if it looks anything like windows nt14:09
bacsinzui: if daily is not a proper work around should it go back to critical?14:10
sinzuiyes. daily was required last year because saucy was the only series that had azure support14:10
natefinchperrito666: we have work on getting charms deployable to windows14:11
sinzuibac also, daily is now focuses on utopic. I think you want to use an LTS14:11
natefinchI'm working with the cloudbase guys getting their code into Juju, but I'm on vacation next week and need someone else to help them out14:12
natefinchperrito666: it's honestly less windows stuff and more just helping them get their code well integrated into Juju14:13
perrito666natefinch: I guess I could help, although I wold love a bit more info :)14:13
natefinchperrito666: hop on here https://plus.google.com/hangouts/_/canonical.com/cloudbase-juju?authuser=114:16
sinzuiIf the bot is going to ignore me or the 1.20 branch, I can merge the 1.20.0 version change myself to unblock the release14:16
perrito666natefinch: hold I sec I stop the radio, I was waiting for news on the country entering into economic default or not14:17
natefinchheh14:17
natefinchperrito666: no rush14:17
natefinchperrito666: we'll be on there for a long time14:17
perrito666also apparently our vice president might get arrested for defraudation :p14:17
perrito666natefinch: says google that the party is over14:20
sinzuiI manually merged 1.20.0 version change into 1.20 branch. CI test it in about 2 hours14:20
natefinchperrito666: invited via the UI, that should work14:21
sinzuioh, it will test in in a hour because master has an invalid version, the test suite will exit early14:21
perrito666natefinch: yup, scared me to death14:21
rogpeppeanother review if anyone cares to, much simpler one this time: https://github.com/juju/charmstore/pull/1114:22
perrito666when did fwereade pop into that call? I was about to say that natefinch sounded a lot like fwereade today :p14:29
wwitzel3England, New England .. same thing14:30
natefinchhaha14:30
TheMuerogpeppe: 11 is reviewed14:30
perrito666well, clearly NewEngland is a factory for England14:30
rogpeppeTheMue: thanks14:30
natefinchlol14:31
TheMueperrito666: which package? and a reference of a copy?14:31
TheMueperrito666: usa := europe.NewEngland() ?14:31
perrito666TheMue: NewEngland() (*europe.England, error) {}14:31
perrito666I understand England used that a lot around 500 ys ago14:32
TheMueperrito666: oh, shit, compiler error, didn’t asked for the error14:32
perrito666:p14:32
perrito666more in the spirit of NewEngland(unconquered world.Country) (*europe.England, error) {}14:33
perrito666although that only makes sense in spanish where uk == england in daily use14:34
natefinchuk == england in most of the united states.  Took me forever to understand the political structure of that little batch of islands14:34
perrito666I was provided an educational video by an uk guy14:35
perrito666which explained all of that14:35
TheMuefunnily many Country instances reacted with a panic() but England defered a recover()14:35
perrito666we are a bunch of nerds14:35
wwitzel3http://xkcd.com/850/14:36
TheMueMeeeeee? Nooooo! *blush*14:36
TheMuewwitzel3: perrito666: regarding england: http://twistedsifter.files.wordpress.com/2013/08/the-only-countries-britain-has-not-invaded.jpg14:38
wwitzel3lol14:38
ericsnownatefinch, fwereade: are we having that meeting?14:40
* perrito666 notices that he forgot to cook lunch14:54
jcw4perrito666: eating it raw?14:55
jcw4perrito666: that corruption scandal must have really got your attention14:55
perrito666jcw4: no, actually I was self documenting code14:55
perrito666:p14:55
jcw4:)14:56
perrito666alexisb: natefinch something seems wrong with the hangout15:21
=== vladk|offline is now known as vladk
=== vladk is now known as vladk|offline
perrito666ericsnow: ping me when you are available16:10
ericsnowperrito666: ping16:10
perrito666ericsnow: priv16:11
=== robbiew is now known as robbiew-afk
rogpeppemgz: ping16:32
mgzrogpeppe: hey16:33
rogpeppemgz: would you be able to review a change to godeps?16:33
mgzsure thing16:33
rogpeppemgz: ta! https://codereview.appspot.com/106250043/16:33
mgzhmm16:34
rogpeppemgz: do you think that fetching by default is a bad thing?16:35
mgzjust thinking if it's going to bork anything16:35
mgzwell, it borks the wait, I'm actually on a different branch case16:35
mgzbut I guess it's not to bad to learn to use -F if needed16:36
rogpeppemgz: i can't think of a case where i'd ever actually want to use -F16:37
rogpeppemgz: after all, the repo may be updated anyway, regardless of -F16:37
rogpeppemgz: i guess the only time i might want to use it is if my network connection is poor16:37
mgzrogpeppe: wehn you want to see deps that need updating, but not screw with trees because you're not sure of their current state16:38
mgzusing godeps -u in that case is a little dodgy anyway16:38
rogpeppemgz: there's always the -n flag for that16:38
rogpeppeha, i've just spotted a bug16:39
mgz-P of 10 by default is also maybe a question16:39
mgzthat's enough to make rural broadband pretty sad16:39
rogpeppemgz: i bet your web browser fetches more than 10 things at once...16:42
mgzrogpeppe: really need some actual tests for the changes16:42
rogpeppemgz: yeah, i thought you might say that. the tests have been broken for a while :-(16:42
rogpeppei should really fix the tests16:43
mgzrogpeppe: sure, but running ten git processes in parallel is more than just an http get16:43
mgzrogpeppe: change seems fine in general though16:43
rogpeppemgz: you're worried about cpu resources?16:43
rogpeppemgz: or does git make lots of connections in fact?16:43
rogpeppemgz: thanks16:44
mgzrogpeppe: more network/memory, but yeah, depending on the url of the repo, it's not just one connection16:46
rogpeppemgz: got a suggestion for a better default?16:46
mgz4?16:47
rogpeppemgz: ok, 4 it is16:48
alexisbericsnow, ping17:04
ericsnowalexisb: coming :)17:04
alexisb:)17:04
rogpeppemgz: well, i've got the original tests passing now at any rate, but no time left to add -u tests.17:16
rogpeppemgz: i'll wait until Mon before pushing the code, as i think it's worth having the changes even without tests17:17
rogpeppemgz: and at least then i'll be around when the 'bot breaks :-)17:17
jcw4I've updated https://github.com/juju/juju/pull/164 to merge in bodie_'s work and address some of thumpers comments17:22
jcw4PTAL ^^17:22
* jcw4 is eow17:23
perrito666sinzui: I never realised you could menace the bot into merging stuff18:01
perrito666wwitzel3: ericsnow standup?18:02
ericsnowwwitzel3: yep18:02
sinzuiperrito666, I have a latent ability to do neigh impossible thing. Intimidate bots, run 386 instances in Hp cloud, create trans-cloud juju envs.18:03
wwitzel3perrito666: yep omw18:03
sinzuiI attribute this to my new way of looking at issues, and I will cheat if necessary18:03
sinzuiperrito666, wwitzel3 any insight into this critical bug https://bugs.launchpad.net/juju-core/+bug/133524318:17
_mup_Bug #1335243: No tools available TestValidateConstraintsCalledWithMetadatasource <regression> <test-failure> <juju-core:Triaged> <juju-core 1.20:Triaged> <https://launchpad.net/bugs/1335243>18:17
=== vladk|offline is now known as vladk
=== vladk is now known as vladk|offline
wallyworld__mgz: around?22:45

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!