=== kadams54 is now known as kadams54-away [00:27] so I used git pull -r and now my working copy is fucked [00:27] thanks [00:31] menn0: that is indeed a shitty problem [00:31] davecheney: rebase? [00:33] man a year in go and my python is all kinds of rusty [00:33] ha [00:52] davecheney: o [00:53] davecheney: i've been reduced to hacking locally and doing edits through the github web ui... [02:18] axw: ping [02:34] menn0: pong === kadams54 is now known as kadams54-away [02:35] axw: do you know anything about provisioners and cloud-init? [02:36] menn0: a reasonable amount, yes - what do you need to know? [02:36] i'm dealing with this Joyent networking issue [02:36] axw: hangout might be more efficient [02:36] menn0: sure [02:37] axw: calling you know [02:37] axw: now even === kadams54-away is now known as kadams54 [03:39] menn0: I don't know what the deal is exactly, but your machine has networks that don't show up in sdc-listnetworks [03:40] what does sdc-listnetworks show? [03:40] axw: ^ [03:41] menn0: sec, pastebinning [03:44] menn0: http://paste.ubuntu.com/9484090/ [03:45] menn0: http://paste.ubuntu.com/9484113/ [03:51] * menn0 looks [03:52] axw: that matches [03:52] those networks match what's on the interfaces [03:54] axw: well all but one [03:55] menn0: can you try hacking up the environ_instance.go code to add those network IDs I listed in the CloudMachineOpts.Networks field? [03:55] I think what's happening is that in lieu of us specifying the network, Joyent automatically creates an internal and external subnet for each instance [03:56] axw: actually, no they all match up [03:56] which may or may not be routable [03:56] between each instance I mean [03:56] but I don't think adding those network ids will help [03:56] because those private networks can't route to each other [03:56] menn0: I mean the IDs from sdc-listnetworks [03:56] and that matches what the machines have anyway [03:57] menn0: the ones from sdc-listnetworks are different to what's allocated to the machines [03:57] no they're not :) [03:57] axw: ^ [03:58] axw: they're just network addresses [03:58] menn0: ?? in my first pastebin, "id" for each network returned by sdc-listnetworks is different to what's in sdc-listmachines [03:59] * menn0 looks again [03:59] I expect the subnet in the names of the networks in the second pastebin to match what's on the machine, because those are for the networks assigned to the machines [04:00] axw: but they do! [04:00] axw: it's a little tricky to work out because they're /21 [04:00] axw: but when you do, you'll see they match [04:00] axw: the networks in the output from sdc-getnetwork are network addresses [04:01] axw: if you take the ip and netmask of the interfaces on the hosts [04:01] axw: and calculate the network address [04:01] axw: you end up with those networks [04:01] menn0: right, I understand those are the same as what's on the machine [04:01] those are the networks assigned to the machines [04:01] what I'm saying is that each machine has separate networks [04:01] and those networks are not what's in sdc-listnetworks [04:02] axw: right [04:02] axw: now I've got you [04:02] axw: so I should try using the networks returned by sdx-listnetworks? [04:02] so, try modifying CreateMachineOpts to include the IDs from the top of the first pastebin [04:02] right [04:02] 5983940e-58a5-4543-b732-c689b1fe4c08 and 9ec60129-9034-47b4-b111-3026f9b1a10f [04:02] axw: yep. i'll try that [04:05] ok change made [04:05] so juju upgrade-juju --upload-tools and then add a new machine? [04:06] axw: what are those networks anyway? can you use sdc-getnetwork to see? [04:07] menn0: you'll need to rebootstrap or machine 0's network won't be changed [04:07] will see if I can get more detail about the networks [04:08] axw: yeah, but what if joyent expects the new envrionment to use a different network? [04:09] menn0: https://apidocs.joyent.com/cloudapi/#CreateMachine [04:09] networks Array Desired networks ids, obtained from ListNetworks [04:09] ok [04:09] given that, I'm assuming it's valid... [04:09] axw: i'll give it a try [04:09] it's not particularly clear [04:10] sdc-getnetwork doesn't give much information, just whether it's public or private [04:13] axw: I've been reading the API docs as this env bootstraps and the networks stuff is very vague isn't it? [04:13] menn0: yes, it is a bit [04:18] axw: ok machine-0 is up [04:18] interestingly, eth0 is now the internal interface and eth1 is the external so I think this change has done something [04:19] axw: seems functionally fine though [04:19] axw: adding more machines [04:20] menn0: it's still created separate networks *shrug* [04:22] menn0: where can you see that? those machine just coming up now [04:22] menn0: sdc-listmachines shows it [04:23] axw: right [04:23] axw: i'm just checking now what they actually get [04:23] axw: yeah, different networks [04:23] axw: these are all going to fail to come up [04:27] menn0: I'm out of ideas [04:27] axw: so am i [04:27] axw: i've just been looking at the cloud-init userData for clues (I modified the tools to log that too) [04:28] axw: nothing stands out [04:28] axw: might have to wait until we hear back from Joyent [04:28] menn0: it has to be a Joyent thing, the API is telling me they've been assigned different subnets [04:28] cloud-init can't affect that [04:29] axw: the only workaround I can think of in the mean time is to have cloud-init install that static route [04:30] axw: I'll email the list and ask if we can have this no longer be CI blocking [04:32] axw: thanks for all the help anyway [04:36] menn0: no worries [04:36] menn0: if it's not too late, and you've got some time, I'd appreciate a glance over http://reviews.vapour.ws/r/604/ [04:37] if you're about to EOD then just leave it for next week [04:39] axw: I was about to EOD but I might be able to squeeze it after this email... [05:02] axw: reviewing now [05:02] menn0: thanks === kadams54 is now known as kadams54-away [05:12] axw: can you remember which file in the PR had the conflict? [05:15] axw: never mind. all done. Ship It! === liam_ is now known as Guest77493 === ashipika1 is now known as ashipika [09:16] * fwereade has actually succumbed to the fluey thing that's been hovering over him all week :-/ [09:17] I think it would be a very good idea if someone were to fix https://bugs.launchpad.net/bugs/1297559 for 1.18, there's a branch already, haven't looked at it but description is sane [09:17] Bug #1297559: upgrade-charm fails because of git

[10:17] fwereade, hey [10:17] fwereade, I guess you'll be off mostly today then - get well soon! [10:20] TheMue, still otp? [10:20] dimitern, morning - can you point me in the direction of where the lxc config files are generate for Juju with MAAS provider? [10:21] jamespage, morning [10:21] jamespage, just a sec [10:22] jamespage, is that the mtu bug you've just reported? [10:23] dimitern, well its related to the mtu setting problem I've had for a while [10:23] (that bug is old-ish) [10:23] https://bugs.launchpad.net/ubuntu/+source/juju-core/+bug/1355813 [10:23] Bug #1355813: Interface MTU management across MAAS/juju

[10:23] dimitern, configuring maas to set the physical host interfaces to mtu 9000 is quite easy using dhcp [10:24] dimitern, more tricky with the LXC containers which then foobar that configuration [10:24] jamespage, so does it seem it's related to juju-br0 config that gets generated at cloudinit time? [10:25] jamespage, I doubt it's the lxc config for individual containers, as that's pretty basic [10:25] dimitern, juju-br0 gets the correct mtu, but as soon as the veth's for LXC containers get plugged into it, it drops down to the lowest - 1500 be detaul [10:26] jamespage, ah, so it's the veth then [10:27] jamespage, if you tweak networkTemplate and writeLxcConfig in container/lxc.go you can add the necessary lxc.network.xyz settings there [10:32] dimitern, ack [10:33] dimitern, I was trying to think how this could be generally improved [10:33] dimitern, detecting the physical interface mtu and using that would make sense [10:37] dimitern: voidspace: so, phone call done. has been an important one by the mobile phone company, saving money now ;) [10:38] TheMue, nice! [10:38] dimitern: voidspace: my news for today is the ending of the yaml.v1 test fixes. done the major changes yesterday but found two new/unknowns test cases. one is done, one is now in progress, so I hope I can propose it in a few minutes [10:38] TheMue, sounds good [10:39] jamespage, that's definitely a possibility [10:39] TheMue: cool [10:39] jamespage, especially if there's a feature bug filed so we won't forget it :) [13:14] good morning [13:14] natefinch: apparently chocolate and butter cookies are not something to be eaten in the amounts I did [13:14] did you get my email? [13:36] dimitern: ping [13:37] TheMue, pong [13:37] dimitern: could you do me a favor and run go test in juju/juju/worker? [13:38] dimitern: I've got errors there even in trunk [13:38] dimitern: notifyworker and stringsworker fail [13:38] TheMue, sure, will do now - what errors? [13:39] dimitern: we expect []string{"setup"} but get []string(nil) [13:41] TheMue, could it be an issue that could be resolved by running godeps -u dependencies.tsv? [13:41] dimitern: I've done that before [13:42] TheMue, well, so far they pass - on the worker/uniter now [13:43] dimitern: astonishing [13:44] dimitern: so it's a local problem. have to take a deeper look at the dependencies.tsv. I've changed it, but switching the branch should had changed it back. [13:46] TheMue, all passed :/ [13:47] dimitern: ok, thanks, helpful [13:57] are we still blocked on joyent? [14:23] fwereade: - [14:23] 3..bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbh- [14:23] [14:23] damg kids [14:30] TheMue, voidspace, I'd really appreciate if you can have a look at http://reviews.vapour.ws/r/611/ - I know it's big, but 75% of the added lines are tests [14:30] maybe looking at the diff on GH will be better [14:32] natefinch, I once set IRC to only send with control+enter. Then one night a cat managed to type and send a message. People thought I fell a sleep at the keyboard. [14:32] sinzui, I think bug 1401130 should be taken off the ci blockers [14:32] Bug #1401130: Joyent instances sometimes can't communicate via the internal network

[14:33] dimitern, I am thinking about. I agree the issue is everywhere. I am looking for evidence in CI that master had some bad luck, that the situation is not worse [14:34] sinzui, or at least taken down in priority - it seems like an inane joyent issue which was there for some time now [14:36] dimitern, I don't think we can lower the priority given the systemic unreliability of menno's report. I am using joeynt with 1.20x a lot and it is quite reliable. we also completed a round of reliability tests with on joyent using 1.20.13, 1.20.14, 1.21.beta4, and and older version of 1.22-alpha1, and joyent was very reliable [14:38] sinzui, well, if the failure happens "sometimes" after adding more than 2 machines how can you be sure the tests were simply not hitting the issue often enough? [14:38] mgz_, are you around? [14:41] dimitern, I am proposing to my team that we retest master now. we can see Joyent is healthy with 1.20.x [14:42] sinzui, right, can you test both with default-series: precise and trusty ? I suspect trusty has it, while precise doesn't (different networking setup) [14:43] dimitern, ci tests wil precise most of the time for joyent. [14:43] well. [14:43] the deploy test was precise. The health check is trusty [14:44] okay, I am adding a test to cover both right now [14:44] sinzui, but the health check just does bootstrap + destroy, right? [14:44] sinzui, it my tests yesterday I've seen quite frequently destroy-environment hanging for a long time, even after the instances were stopped [14:45] sinzui, also there's a z-joyent-something job to clean up old running machines, which is not working due to some --old-age arg not being recognized; and I could see quite a few running yesterday [14:47] dimitern, the health check deploy one service [14:47] sinzui, going back a few lines - the health check job is not enough to trigger the issue, menno said he had to spin up 10 machines sometimes to reproduce it, but still wasn't reliable [14:48] dimitern, yes I saw the hang often this week. was deploying bundles on Monday and Tuesday. Deployments were a success, but destroy-env hung twice. I confirmed that machines were shutdown and deleted. [14:50] dimitern, I agree about the machine count. We will retest master. The situation is that in 7 days master have never passed, yet 1.21-beta4 did pass when it was tested [14:51] sinzui, right, the workaround menno discovered - adding a static route for 10.0.0.0 in cloudinit seems to fix the connectivity issues when they happen, so this is probably the correct fix [14:52] sinzui, but without a definite way of reproducing it.. [14:52] dimitern, isn't that too wide? [14:53] sinzui, it's both wide and also nasty, but if that's how it has to be.. [14:53] sinzui, I hope it won't introduce other regressions - like the ability to see and talk to instances foreign to the environment [14:56] dimitern, CI noticed I added a new test and immediately started testing with 1.21-beta4. I will let it complete so that we have a comparison of the two branches [14:56] sinzui, cool [14:58] dimitern, if we get passes for 1.21 and master, I will change the precise version to non-voting, which will prevent the test from curing the revision. That will justify the the removal of regression [15:00] dimitern, beta4+trusty+joyent just passed. I am starting the retest of master [15:00] sinzui, \o/ [15:01] ericsnow: ping [15:01] wwitzel3: hey [15:02] wwitzel3: coming [15:03] btw voidspace fwereade you are OCR I have a couple of things in queue if you want to take a look [15:04] perrito666: I can try shortly [15:04] tx they are very small [15:04] because I didnt want to spoil your friday [15:05] perrito666: :-) [15:06] are the critica bugs really still open? [15:06] I saw an email from menn0 with a complelling argument against having the joyent one a blocker [15:06] perrito666: joyent was "more info needed" and the other was "fix committed" [15:06] right [15:07] aisrael, marcoceppi_, I'm trying to use https://github.com/juju-solutions/juju-action and when I run a command the "if service in services" check in action.py is never true. have any tips? I'm probably doing something dumb. [15:08] coreycb: we've abandonded that [15:09] marcoceppi_, got any alternatives? [15:09] coreycb: yeah, juju-actions fork of the juju core [15:09] marcoceppi_, k thanks [15:10] coreycb: https://github.com/juju-actions/juju/tree/actions [15:11] coreycb, marcoceppi_ hopefully we'll be landing these into juju core in the next few days - using a feature flag [15:11] jw4, marcoceppi_, awesome, thanks [16:04] marcoceppi_, tvansteenburgh: amulet PR when you have a chance: https://github.com/juju/amulet/pull/59 [16:07] bac: ack [16:11] mramm: we've asked for lp:~yellow to be added to lp:~rhinos so we can work on lp:convoy. could you approve the invite? [16:14] perrito666: two LGTMs [16:31] voidspace: tx [16:50] fwereade: ping [17:26] * dimitern sings and dances \o/ [17:27] we have lp:goamz migration to github ! https://github.com/go-amz/amz [17:27] dimitern: awesome! Man, about time :) [17:28] dimitern: btw, check out https://github.com/stripe/aws-go [17:28] natefinch, cheers :) [17:28] natefinch, oh nice - autogenerated ? [17:29] yep [17:30] dimitern: I haven't had a chance to check it out, but if it's based on schema provided by AWS, that's pretty nice [17:38] can anyone tell me why this is giving a go vet warning? logger.Logf(loggo.INFO, fmt.Sprintf("%s: %s", user.Tag(), format), args...) [17:38] "constant 3 is not a string in call to Logf" [17:39] i'm stumped [17:39] brb tea [17:39] katco: you switched to go 1.4 didn't you [17:40] katco: davecheney filed a bug in go vet for that issue [17:40] katco: I proposed a fix a few weeks ago that he nixed b'cause it was ugly [17:42] jw4: ah yes i did [17:42] so back to 1.3 i guess? [17:42] katco: that's what I ended up doing [17:42] bleh [17:43] jw4: do you happen to have the tracker for the bug? [17:43] jw4: the url i mean [17:43] katco: already looking for it [17:43] :) [17:43] jw4: you are kind! :) [17:43] * katco starts switching back to 1.3 [17:45] katco: https://code.google.com/p/go/issues/detail?id=9080 [17:45] jw4: ty! [17:45] katco: my proposed (ugly) fix - https://github.com/juju/juju/pull/1087 [17:48] katco: comments on the migrated issue (https://github.com/golang/go/issues/9080) might be helpful [17:48] * jw4 investigating [17:49] jw4: don't worry about it, i'll just use 1.3 for Juju until it's fixed [17:50] katco: kk (but I've got my own inquisitiveness up again) [17:50] jw4: an easier fix might just be to combine those args in audit.go into a single string [17:50] jw4: assign it to a var and pass it in that way [17:51] katco: yeah - I need to look again, but I thought there was deferred execution involved - but I may be wrong [17:52] anyway, looking at our verify.bash script this might actually be a trivial fix [17:54] EOW [17:54] have a good weekend all [17:55] tc voi [17:55] katco: tut tut - autocomplete doesn't work when they've disconnected [17:56] hehe [18:53] katco: http://reviews.vapour.ws/r/630/ [18:53] OCR PTAL ^^ [18:53] (although I think both voidspace and fwereade are gone now) [18:54] jw4: I'll take a look [18:55] ericsnow: ta [19:03] jw4: Could you explain more about where go vet is finding a problem? [19:04] ericsnow: yes sure - otp, but I'll connect in a minute [19:04] jw4: I'm not seeing why it wants a string where loggo.INFO is being used. [19:04] jw4: no worries [19:13] ericsnow: I haven't actually delved into go vet - I'm just basing my fix on the new error message that go 1.4 gives with go vet [19:14] jw4: does go vet recurse? (i.e. does the loggo package pass?) [19:14] ericsnow: go vet actually allows you to specify -printfuncs=Logf:1 to tell go vet to ignore the first param, which avoids this error message, but introduces 30 new ones [19:14] ericsnow: no, I don't think go vet recurses [19:15] jw4: yeah, I'd doubt it [19:15] ericsnow: I think this fix actually works mostly because the name of the function doesn't match Logf anymore [19:16] ericsnow: my previous suggestion a month or so ago that davecheney rightfully shot down was to introduce a local func var with a different name :) [19:16] jw4: why would Logf be special? [19:17] ericsnow: it matches a pattern that go vet is designed to look for [19:17] jw4: the thing that makes me confused is that loggo.INFO is a loggo.Level (a uint32) and logger.Logf takes a Level in the spot where INFO is passed) [19:18] ericsnow: I can tell you're gonna force me to actually look at go vet source [19:18] jw4: nah [19:18] ericsnow: yeah - there is absolutely nothing wrong with the code [19:18] ericsnow: we're just trying to pacify go vet [19:18] jw4: but it may be worth noting in a comment that we're working around a go vet bug [19:19] jw4: it's a bug, right? [19:19] ericsnow: forest / trees [19:19] ericsnow: yep [19:19] ericsnow: I will update the comment [19:19] jw4: has a bug report been opened? If so, a link to it could go into the comment. [19:19] ericsnow: +1 [19:20] ericsnow: for the record: https://github.com/golang/go/issues/9080 [19:20] jw4: cool [19:23] jw4: I opened an issue in my review for that (and LGTM) [19:23] ericsnow: ta - my update is pushed now too [19:24] jw4: looks good [19:25] kk - no point in rushing for a senior sign off yet - I doubt CI will open before Monday :-/ [19:25] on the bright side - I, and katco, can now begin using go 1.4 for juju dev :) [19:25] :) [19:26] katco: after you've done all the work of reverting [19:26] jw4: it's not work at all... i have them installed side-by-side ~/.local/go-1.{3.4|4.0} [19:26] symlink [19:27] katco: I actually have a fairly useful scheme involving symlinks that allows me to easily switch between go versions while maintaining separate .... well I'll send this comment anyway :) [19:27] haha [19:27] katco: you beat me to it - brevity wins every time [19:27] os x does much the same thing i think [19:27] brevity or symlinks? [19:27] ;) [19:28] lol [19:28] FWIW, ntfs has a pretty decent symlink lookalike too : junctions [19:29] i have no reason to use that anymore [19:29] "she said with a faint curl to her lip" [19:29] haha [20:13] this branch is going to be the death of me [20:14] katco: welcome to restore [20:14] :p [20:15] perrito666: lol [20:15] perrito666: i'm refactoring cmd/jujud =| [20:15] is anyone doing anything with the joyent issue? === kadams54 is now known as kadams54-away === kadams54-away is now known as kadams54 [21:39] ericsnow: ping [21:39] wwitzel3: pong [21:40] ericsnow: I'm in moonstone [21:40] brt === kadams54 is now known as kadams54-away [21:57] any chartered reviewers willing to LGTM http://reviews.vapour.ws/r/630/ and maybe even authori(s|z)e JFDI? :) [21:58] * jw4 likes working in go 1.4