/srv/irclogs.ubuntu.com/2014/12/12/#juju-dev.txt

=== kadams54 is now known as kadams54-away
davecheneyso I used git pull -r and now my working copy is fucked00:27
davecheneythanks00:27
thumpermenn0: that is indeed a shitty problem00:31
perrito666davecheney: rebase?00:31
perrito666man a year in go and my python is all kinds of rusty00:33
thumperha00:33
mwhudsondavecheney: o00:52
mwhudsondavecheney: i've been reduced to hacking locally and doing edits through the github web ui...00:53
menn0axw: ping02:18
axwmenn0: pong02:34
=== kadams54 is now known as kadams54-away
menn0axw: do you know anything about provisioners and cloud-init?02:35
axwmenn0: a reasonable amount, yes - what do you need to know?02:36
menn0i'm dealing with this Joyent networking issue02:36
menn0axw: hangout might be more efficient02:36
axwmenn0: sure02:36
menn0axw: calling you know02:37
menn0axw: now even02:37
=== kadams54-away is now known as kadams54
axwmenn0: I don't know what the deal is exactly, but your machine has networks that don't show up in sdc-listnetworks03:39
menn0what does sdc-listnetworks show?03:40
menn0axw: ^03:40
axwmenn0: sec, pastebinning03:41
axwmenn0: http://paste.ubuntu.com/9484090/03:44
axwmenn0: http://paste.ubuntu.com/9484113/03:45
* menn0 looks03:51
menn0axw:  that matches03:52
menn0those networks match what's on the interfaces03:52
menn0axw: well all but one03:54
axwmenn0: can you try hacking up the environ_instance.go code to add those network IDs I listed in the CloudMachineOpts.Networks field?03:55
axwI think what's happening is that in lieu of us specifying the network, Joyent automatically creates an internal and external subnet for each instance03:55
menn0axw: actually, no they all match up03:56
axwwhich may or may not be routable03:56
axwbetween each instance I mean03:56
menn0but I don't think adding those network ids will help03:56
menn0because those private networks can't route to each other03:56
axwmenn0: I mean the IDs from sdc-listnetworks03:56
menn0and that matches what the machines have anyway03:56
axwmenn0: the ones from sdc-listnetworks are different to what's allocated to the machines03:57
menn0no they're not :)03:57
menn0axw: ^03:57
menn0axw: they're just network addresses03:58
axwmenn0: ?? in my first pastebin, "id" for each network returned by sdc-listnetworks is different to what's in sdc-listmachines03:58
* menn0 looks again03:59
axwI expect the subnet in the names of the networks in the second pastebin to match what's on the machine, because those are for the networks assigned to the machines03:59
menn0axw: but they do!04:00
menn0axw: it's a little tricky to work out because they're /2104:00
menn0axw: but when you do, you'll see they match04:00
menn0axw: the networks in the output from sdc-getnetwork are network addresses04:00
menn0axw: if you take the ip and netmask of the interfaces on the hosts04:01
menn0axw: and calculate the network address04:01
menn0axw: you end up with those networks04:01
axwmenn0: right, I understand those are the same as what's on the machine04:01
axwthose are the networks assigned to the machines04:01
axwwhat I'm saying is that each machine has separate networks04:01
axwand those networks are not what's in sdc-listnetworks04:01
menn0axw: right04:02
menn0axw: now I've got you04:02
menn0axw: so I should try using the networks returned by sdx-listnetworks?04:02
axwso, try modifying CreateMachineOpts to include the IDs from the top of the first pastebin04:02
axwright04:02
axw5983940e-58a5-4543-b732-c689b1fe4c08 and 9ec60129-9034-47b4-b111-3026f9b1a10f04:02
menn0axw: yep. i'll try that04:02
menn0ok change made04:05
menn0so juju upgrade-juju --upload-tools and then add a new machine?04:05
menn0axw: what are those networks anyway? can you use sdc-getnetwork to see?04:06
axwmenn0: you'll need to rebootstrap or machine 0's network won't be changed04:07
axwwill see if I can get more detail about the networks04:07
menn0axw: yeah, but what if joyent expects the new envrionment to use a different network?04:08
axwmenn0: https://apidocs.joyent.com/cloudapi/#CreateMachine04:09
axwnetworksArrayDesired networks ids, obtained from ListNetworks04:09
menn0ok04:09
axwgiven that, I'm assuming it's valid...04:09
menn0axw: i'll give it a try04:09
axwit's not particularly clear04:09
axwsdc-getnetwork doesn't give much information, just whether it's public or private04:10
menn0axw: I've been reading the API docs as this env bootstraps and the networks stuff is very vague isn't it?04:13
axwmenn0: yes, it is a bit04:13
menn0axw: ok machine-0 is up04:18
menn0interestingly, eth0 is now the internal interface and eth1 is the external so I think this change has done something04:18
menn0axw: seems functionally fine though04:19
menn0axw: adding more machines04:19
axwmenn0: it's still created separate networks *shrug*04:20
menn0menn0: where can you see that? those machine just coming up now04:22
axwmenn0: sdc-listmachines shows it04:22
menn0axw: right04:23
menn0axw: i'm just checking now what they actually get04:23
menn0axw: yeah, different networks04:23
menn0axw: these are all going to fail to come up04:23
axwmenn0: I'm out of ideas04:27
menn0axw: so am i04:27
menn0axw: i've just been looking at the cloud-init userData for clues (I modified the tools to log that too)04:27
menn0axw: nothing stands out04:28
menn0axw: might have to wait until we hear back from Joyent04:28
axwmenn0: it has to be a Joyent thing, the API is telling me they've been assigned different subnets04:28
axwcloud-init can't affect that04:28
menn0axw: the only workaround I can think of in the mean time is to have cloud-init install that static route04:29
menn0axw: I'll email the list and ask if we can have this no longer be CI blocking04:30
menn0axw: thanks for all the help anyway04:32
axwmenn0: no worries04:36
axwmenn0: if it's not too late, and you've got some time, I'd appreciate a glance over http://reviews.vapour.ws/r/604/04:36
axwif you're about to EOD then just leave it for next week04:37
menn0axw: I was about to EOD but I might be able to squeeze it after this email...04:39
menn0axw: reviewing now05:02
axwmenn0: thanks05:02
=== kadams54 is now known as kadams54-away
menn0axw: can you remember which file in the PR had the conflict?05:12
menn0axw: never mind. all done. Ship It!05:15
=== liam_ is now known as Guest77493
=== ashipika1 is now known as ashipika
* fwereade has actually succumbed to the fluey thing that's been hovering over him all week :-/09:16
fwereadeI think it would be a very good idea if someone were to fix https://bugs.launchpad.net/bugs/1297559 for 1.18, there's a branch already, haven't looked at it but description is sane09:17
mupBug #1297559: upgrade-charm fails because of git  <canonical-is> <git> <upgrade-charm> <juju-core:Fix Released by fwereade> <juju-core 1.18:Triaged> <https://launchpad.net/bugs/1297559>09:17
dimiternfwereade, hey10:17
dimiternfwereade, I guess you'll be off mostly today then - get well soon!10:17
dimiternTheMue, still otp?10:20
jamespagedimitern, morning - can you point me in the direction of where the lxc config files are generate for Juju with MAAS provider?10:20
dimiternjamespage, morning10:21
dimiternjamespage, just a sec10:21
dimiternjamespage, is that the mtu bug you've just reported?10:22
jamespagedimitern, well its related to the mtu setting problem I've had for a while10:23
jamespage(that bug is old-ish)10:23
jamespagehttps://bugs.launchpad.net/ubuntu/+source/juju-core/+bug/135581310:23
mupBug #1355813: Interface MTU management across MAAS/juju <add-unit> <amd64> <apport-bug> <cts> <cts-cloud-review> <landscape> <network> <placement> <smoosh> <trusty> <uec-images> <juju-core:Triaged> <MAAS:New> <juju-core (Ubuntu):Triaged> <lxc (Ubuntu):Invalid> <https://launchpad.net/bugs/1355813>10:23
jamespagedimitern, configuring maas to set the physical host interfaces to mtu 9000 is quite easy using dhcp10:23
jamespagedimitern, more tricky with the LXC containers which then foobar that configuration10:24
dimiternjamespage, so does it seem it's related to juju-br0 config that gets generated at cloudinit time?10:24
dimiternjamespage, I doubt it's the lxc config for individual containers, as that's pretty basic10:25
jamespagedimitern, juju-br0 gets the correct mtu, but as soon as the veth's for LXC containers get plugged into it, it drops down to the lowest - 1500 be detaul10:25
dimiternjamespage, ah, so it's the veth then10:26
dimiternjamespage, if you tweak networkTemplate and writeLxcConfig in container/lxc.go you can add the necessary lxc.network.xyz settings there10:27
jamespagedimitern, ack10:32
jamespagedimitern, I was trying to think how this could be generally improved10:33
jamespagedimitern, detecting the physical interface mtu and using that would make sense10:33
TheMuedimitern: voidspace: so, phone call done. has been an important one by the mobile phone company, saving money now ;)10:37
dimiternTheMue, nice!10:38
TheMuedimitern: voidspace: my news for today is the ending of the yaml.v1 test fixes. done the major changes yesterday but found two new/unknowns test cases. one is done, one is now in progress, so I hope I can propose it in a few minutes10:38
dimiternTheMue, sounds good10:38
dimiternjamespage, that's definitely a possibility10:39
voidspaceTheMue: cool10:39
dimiternjamespage, especially if there's a feature bug filed so we won't forget it :)10:39
perrito666good morning13:14
perrito666natefinch: apparently chocolate and butter cookies are not something to be eaten in the amounts I did13:14
perrito666did you get my email?13:14
TheMuedimitern: ping13:36
dimiternTheMue, pong13:37
TheMuedimitern: could you do me a favor and run go test in juju/juju/worker?13:37
TheMuedimitern: I've got errors there even in trunk13:38
TheMuedimitern: notifyworker and stringsworker fail13:38
dimiternTheMue, sure, will do now - what errors?13:38
TheMuedimitern: we expect []string{"setup"} but get []string(nil)13:39
dimiternTheMue, could it be an issue that could be resolved by running godeps -u dependencies.tsv?13:41
TheMuedimitern: I've done that before13:41
dimiternTheMue, well, so far they pass - on the worker/uniter now13:42
TheMuedimitern: astonishing13:43
TheMuedimitern: so it's a local problem. have to take a deeper look at the dependencies.tsv. I've changed it, but switching the branch should had changed it back.13:44
dimiternTheMue, all passed :/13:46
TheMuedimitern: ok, thanks, helpful13:47
perrito666are we still blocked on joyent?13:57
natefinchfwereade: -14:23
natefinch3..bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbh-14:23
natefinch             14:23
natefinchdamg kids14:23
dimiternTheMue, voidspace, I'd really appreciate if you can have a look at http://reviews.vapour.ws/r/611/ - I know it's big, but 75% of the added lines are tests14:30
dimiternmaybe looking at the diff on GH will be better14:30
sinzuinatefinch, I once set IRC to only send with control+enter. Then one night a cat managed to type and send a message. People thought I fell a sleep at the keyboard.14:32
dimiternsinzui, I think bug 1401130 should be taken off the ci blockers14:32
mupBug #1401130: Joyent instances sometimes can't communicate via the internal network <ci> <joyent-provider> <regression> <juju-core:In Progress by menno.smits> <juju-core 1.21:In Progress by menno.smits> <https://launchpad.net/bugs/1401130>14:32
sinzuidimitern, I am thinking about. I agree the issue is everywhere. I am looking for evidence in CI that master had some bad luck, that the situation is not worse14:33
dimiternsinzui, or at least taken down in priority - it seems like an inane joyent issue which was there for some time now14:34
sinzuidimitern, I don't think we can lower the priority given the systemic unreliability of menno's report. I am using joeynt with 1.20x a lot and it is quite reliable. we also completed a round of reliability tests with on joyent using 1.20.13, 1.20.14, 1.21.beta4, and and older version of 1.22-alpha1, and joyent was very reliable14:36
dimiternsinzui, well, if the failure happens "sometimes" after adding more than 2 machines how can you be sure the tests were simply not hitting the issue often enough?14:38
dimiternmgz_, are you around?14:38
sinzuidimitern, I am proposing to my team that we retest master now. we can see Joyent is healthy with 1.20.x14:41
dimiternsinzui,  right, can you test both with default-series: precise and trusty ? I suspect trusty has it, while precise doesn't (different networking setup)14:42
sinzuidimitern, ci tests wil precise most of the time for joyent.14:43
sinzuiwell.14:43
sinzuithe deploy test was precise. The health check is trusty14:43
sinzuiokay, I am adding a test to cover both right now14:44
dimiternsinzui, but the health check just does bootstrap + destroy, right?14:44
dimiternsinzui, it my tests yesterday I've seen quite frequently destroy-environment hanging for a long time, even after the instances were stopped14:44
dimiternsinzui, also there's a z-joyent-something job to clean up old running machines, which is not working due to some --old-age arg not being recognized; and I could see quite a few running yesterday14:45
sinzuidimitern, the health check deploy one service14:47
dimiternsinzui, going back a few lines - the health check job is not enough to trigger the issue, menno said he had to spin up 10 machines sometimes to reproduce it, but still wasn't reliable14:47
sinzuidimitern, yes I saw the hang often this week. was deploying bundles on Monday and Tuesday. Deployments were a success, but destroy-env hung twice. I confirmed that machines were shutdown and deleted.14:48
sinzuidimitern, I agree about the machine count. We will retest master. The situation is that in 7 days master have never passed, yet 1.21-beta4 did pass when it was tested14:50
dimiternsinzui, right, the workaround menno discovered - adding a static route for 10.0.0.0 in cloudinit seems to fix the connectivity issues when they happen, so this is probably the correct fix14:51
dimiternsinzui, but without a definite way of reproducing it..14:52
sinzuidimitern, isn't that too wide?14:52
dimiternsinzui, it's both wide and also nasty, but if that's how it has to be..14:53
dimiternsinzui, I hope it won't introduce other regressions - like the ability to see and talk to instances foreign to the environment14:53
sinzuidimitern, CI noticed I added a new test and immediately started testing with 1.21-beta4. I will let it complete so that we have a comparison of the two branches14:56
dimiternsinzui, cool14:56
sinzuidimitern, if we get passes for 1.21 and master, I will change the precise version to non-voting, which will prevent the test from curing the revision. That will justify the the removal of regression14:58
sinzuidimitern, beta4+trusty+joyent just passed. I am starting the retest of master15:00
dimiternsinzui, \o/15:00
wwitzel3ericsnow: ping15:01
ericsnowwwitzel3: hey15:01
ericsnowwwitzel3: coming15:02
perrito666btw voidspace fwereade you are OCR I have a couple of things in queue if you want to take a look15:03
voidspaceperrito666: I can try shortly15:04
perrito666tx they are very small15:04
perrito666because I didnt want to spoil your friday15:04
voidspaceperrito666: :-)15:05
perrito666are the critica bugs really still open?15:06
perrito666I saw an email from menn0 with a complelling argument against having the joyent one a blocker15:06
voidspaceperrito666: joyent was "more info needed" and the other was "fix committed"15:06
voidspaceright15:06
coreycbaisrael, marcoceppi_, I'm trying to use https://github.com/juju-solutions/juju-action and when I run a command the "if service in services" check in action.py is never true.  have any tips?  I'm probably doing something dumb.15:07
marcoceppi_coreycb: we've abandonded that15:08
coreycbmarcoceppi_, got any alternatives?15:09
marcoceppi_coreycb: yeah, juju-actions fork of the juju core15:09
coreycbmarcoceppi_, k thanks15:09
marcoceppi_coreycb: https://github.com/juju-actions/juju/tree/actions15:10
jw4coreycb, marcoceppi_ hopefully we'll be landing these into juju core in the next few days - using a feature flag15:11
coreycbjw4, marcoceppi_, awesome, thanks15:11
bacmarcoceppi_, tvansteenburgh: amulet PR when you have a chance: https://github.com/juju/amulet/pull/5916:04
tvansteenburghbac: ack16:07
bacmramm: we've asked for lp:~yellow to be added to lp:~rhinos so we can work on lp:convoy.  could you approve the invite?16:11
voidspaceperrito666: two LGTMs16:14
perrito666voidspace: tx16:31
perrito666fwereade: ping16:50
* dimitern sings and dances \o/17:26
dimiternwe have lp:goamz migration to github ! https://github.com/go-amz/amz17:27
natefinchdimitern: awesome!  Man, about time :)17:27
natefinchdimitern: btw, check out https://github.com/stripe/aws-go17:28
dimiternnatefinch, cheers :)17:28
dimiternnatefinch, oh nice - autogenerated ?17:28
natefinchyep17:29
natefinchdimitern: I haven't had a chance to check it out, but if it's based on schema provided by AWS, that's pretty nice17:30
katcocan anyone tell me why this is giving a go vet warning? logger.Logf(loggo.INFO, fmt.Sprintf("%s: %s", user.Tag(), format), args...)17:38
katco"constant 3 is not a string in call to Logf"17:38
katcoi'm stumped17:39
katcobrb tea17:39
jw4katco: you switched to go 1.4 didn't you17:39
jw4katco: davecheney filed a bug in go vet for that issue17:40
jw4katco: I proposed a fix a few weeks ago that he nixed b'cause it was ugly17:40
katcojw4: ah yes i did17:42
katcoso back to 1.3 i guess?17:42
jw4katco: that's what I ended up doing17:42
katcobleh17:42
katcojw4: do you happen to have the tracker for the bug?17:43
katcojw4: the url i mean17:43
jw4katco: already looking for it17:43
jw4:)17:43
katcojw4: you are kind! :)17:43
* katco starts switching back to 1.317:43
jw4katco: https://code.google.com/p/go/issues/detail?id=908017:45
katcojw4: ty!17:45
jw4katco: my proposed (ugly) fix - https://github.com/juju/juju/pull/108717:45
jw4katco: comments on the migrated issue (https://github.com/golang/go/issues/9080) might be helpful17:48
* jw4 investigating17:48
katcojw4: don't worry about it, i'll just use 1.3 for Juju until it's fixed17:49
jw4katco: kk (but I've got my own inquisitiveness up again)17:50
katcojw4: an easier fix might just be to combine those args in audit.go into a single string17:50
katcojw4: assign it to a var and pass it in that way17:50
jw4katco: yeah - I need to look again, but I thought there was deferred execution involved - but I may be wrong17:51
jw4anyway, looking at our verify.bash script this might actually be a trivial fix17:52
voidspaceEOW17:54
voidspacehave a good weekend all17:54
katcotc voi17:55
jw4katco: tut tut - autocomplete doesn't work when they've disconnected17:55
katcohehe17:56
jw4katco: http://reviews.vapour.ws/r/630/18:53
jw4OCR PTAL ^^18:53
jw4(although I think both voidspace and fwereade are gone now)18:53
ericsnowjw4: I'll take a look18:54
jw4ericsnow: ta18:55
ericsnowjw4: Could you explain more about where go vet is finding a problem?19:03
jw4ericsnow: yes sure - otp, but I'll connect in a minute19:04
ericsnowjw4: I'm not seeing why it wants a string where loggo.INFO is being used.19:04
ericsnowjw4: no worries19:04
jw4ericsnow: I haven't actually delved into go vet - I'm just basing my fix on the new error message that go 1.4 gives with go vet19:13
ericsnowjw4: does go vet recurse? (i.e. does the loggo package pass?)19:14
jw4ericsnow: go vet actually allows you to specify -printfuncs=Logf:1 to tell go vet to ignore the first param, which avoids this error message, but introduces 30 new ones19:14
jw4ericsnow: no, I don't think go vet recurses19:14
ericsnowjw4: yeah, I'd doubt it19:15
jw4ericsnow: I think this fix actually works mostly because the name of the function doesn't match Logf anymore19:15
jw4ericsnow: my previous suggestion a month or so ago that davecheney rightfully shot down was to introduce a local func var with a different name :)19:16
ericsnowjw4: why would Logf be special?19:16
jw4ericsnow: it matches a pattern that go vet is designed to look for19:17
ericsnowjw4: the thing that makes me confused is that loggo.INFO is a loggo.Level (a uint32) and logger.Logf takes a Level in the spot where INFO is passed)19:17
jw4ericsnow: I can tell you're gonna force me to actually look at go vet source19:18
ericsnowjw4: nah19:18
jw4ericsnow: yeah - there is absolutely nothing wrong with the code19:18
jw4ericsnow: we're just trying to pacify go vet19:18
ericsnowjw4: but it may be worth noting in a comment that we're working around a go vet bug19:18
ericsnowjw4: it's a bug, right?19:19
jw4ericsnow: forest / trees19:19
jw4ericsnow: yep19:19
jw4ericsnow: I will update the comment19:19
ericsnowjw4: has a bug report been opened? If so, a link to it could go into the comment.19:19
jw4ericsnow: +119:19
jw4ericsnow: for the record: https://github.com/golang/go/issues/908019:20
ericsnowjw4: cool19:20
ericsnowjw4: I opened an issue in my review for that (and LGTM)19:23
jw4ericsnow: ta - my update is pushed now too19:23
ericsnowjw4: looks good19:24
jw4kk - no point in rushing for a senior sign off yet - I doubt CI will open before Monday :-/19:25
jw4on the bright side - I, and katco, can now begin using go 1.4 for juju dev :)19:25
katco:)19:25
jw4katco: after you've done all the work of reverting19:26
katcojw4: it's not work at all... i have them installed side-by-side ~/.local/go-1.{3.4|4.0}19:26
katcosymlink19:26
jw4katco: I actually have a fairly useful scheme involving symlinks that allows me to easily switch between go versions while maintaining separate .... well I'll send this comment anyway :)19:27
katcohaha19:27
jw4katco: you beat me to it - brevity wins every time19:27
katcoos x does much the same thing i think19:27
jw4brevity or symlinks?19:27
jw4;)19:27
katcolol19:28
jw4FWIW, ntfs has a pretty decent symlink lookalike too : junctions19:28
katcoi have no reason to use that anymore19:29
jw4"she said with a faint curl to her lip"19:29
katcohaha19:29
katcothis branch is going to be the death of me20:13
perrito666katco: welcome to restore20:14
perrito666:p20:14
katcoperrito666: lol20:15
katcoperrito666: i'm refactoring cmd/jujud =|20:15
perrito666is anyone doing anything with the joyent issue?20:15
=== kadams54 is now known as kadams54-away
=== kadams54-away is now known as kadams54
wwitzel3ericsnow: ping21:39
ericsnowwwitzel3: pong21:39
wwitzel3ericsnow: I'm in moonstone21:40
ericsnowbrt21:40
=== kadams54 is now known as kadams54-away
jw4any chartered reviewers willing to LGTM http://reviews.vapour.ws/r/630/ and maybe even authori(s|z)e JFDI? :)21:57
* jw4 likes working in go 1.421:58

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!