[00:17] davecheney: seems like the azure provider is now basically blocked on a cloudinit sru to precise... [00:18] bigjools: ok, i'm making a 1.14 branch [00:18] we can backport fixes in there to get it working once cloudinit is done [01:04] davecheney: https://bugs.launchpad.net/juju-core/+bug/1216770 is merged, I just forgot to update the bug [01:04] it is in 1.13.3 [01:15] axw: no worries [01:15] https://docs.google.com/a/canonical.com/document/d/1aEvcmxSJaj1i9zNjGy48yKF-SPlTFwW-NiKfoO_Ygo4/edit#heading=h.h7wry0fbg197 [01:15] ^ hint hint [01:17] davecheney: updated [01:17] feel free to edit [01:19] axw: hi, i had a test failure locally with the manual provisioning stuff - detectSeriesAndHardwareCharacteristics() because for some reason the bash that runs doesn't like my .shinit. i think we should ensure the bash that runs ignores all such files [01:19] wallyworld: doh, thanks. I'll get onto it [01:19] np :-) [01:20] axw: cool [01:20] the builders are backed up [01:20] i won't get the build til after lunch [01:20] but we've tagged 1.13.3 now [01:20] so feel free to wreck the build [01:20] axw: i had a quick look - using --noprofile or --norc didn't work; there must be another option to ensure .shinit is ignored. .shinit is more a sh thing i think [01:21] wallyworld: k, thanks [01:21] wallyworld: do you mean the unit tests were failing? or you were playing with it? [01:22] ah the sshscript thing I guess... [01:22] axw: test failure. i ran the tests cause a merged trunk and there were api changes i needed to port into the new manual provisioning code. [01:22] i have a .shinit, the bit that failed was a mkdir, go figure [01:24] davecheney, re slowness on cli.. how long does bootstrap take for you ? i'm consistently getting 2m plus [01:24] lucky(~) % juju bootstrap -e us-west-1 -v [01:24] 2013-09-02 01:24:23 INFO juju.provider.ec2 ec2.go:209 preparing environment "us-west-1" [01:24] 2013-09-02 01:24:23 INFO juju.provider.ec2 ec2.go:187 opening environment "us-west-1" [01:25] 2013-09-02 01:24:23 INFO juju.environs.tools tools.go:81 filtering tools by released version [01:25] 2013-09-02 01:24:23 INFO juju.environs.tools tools.go:28 reading tools with major version 1 [01:25] 2013-09-02 01:24:23 INFO juju.environs.tools tools.go:36 filtering tools by series: precise [01:25] 2013-09-02 01:24:28 INFO juju.environs.tools tools.go:43 falling back to public bucket [01:25] 2013-09-02 01:24:29 INFO juju.environs.tools tools.go:92 picked newest version: 1.12.0 [01:25] 2013-09-02 01:24:38 INFO juju.environs.boostrap bootstrap.go:56 bootstrapping environment "us-west-1" [01:25] 2013-09-02 01:24:38 INFO juju.environs.tools tools.go:28 reading tools with major version 1 [01:25] 2013-09-02 01:24:38 INFO juju.environs.tools tools.go:33 filtering tools by version: 1.12.0 [01:25] 2013-09-02 01:24:38 INFO juju.environs.tools tools.go:36 filtering tools by series: precise [01:25] 2013-09-02 01:24:39 INFO juju.environs.tools tools.go:43 falling back to public bucket [01:25] 2013-09-02 01:24:48 INFO juju.provider.ec2 ec2.go:426 started instance "i-24a95f7e" [01:25] 2013-09-02 01:24:50 INFO juju supercommand.go:284 command finished [01:25] that is from australia [01:25] i suspect the growing number of tools we have in the public bucket is not helping [01:25] wow [01:26] thats a huge delta from what i see [01:26] 2013-09-02 01:14:29 DEBUG juju state.go:159 waiting for DNS name(s) of state server instances [i-9b6cfffc] [01:26] davecheney, yeah.. it would be nice to just have a latest pointer [01:26] 2013-09-02 01:14:40 INFO juju.state open.go:68 opening state; mongo addresses: ["ec2-54-226-75-153.compute-1.amazonaws.com:37017"]; entity "" [01:26] i have no idea what is happening in those 10 seconds [01:26] oh, sorry [01:26] so that's roughly 16s vs 25s [01:26] that is getting the dns name from the provider [01:26] yeah.. instance id -> ip addr [01:27] it's ec2 being slow [01:27] what can I say [01:27] its still 75% of the runtime cost that can be saved with some trivial caching [01:27] per the bug title [01:28] lucky(~) % juju status -e us-west-1 -v [01:28] 2013-09-02 01:28:26 INFO juju.provider.ec2 ec2.go:187 opening environment "us-west-1" [01:28] 2013-09-02 01:28:29 INFO juju.state open.go:68 opening state; mongo addresses: ["ec2-54-219-20-209.us-west-1.compute.amazonaws.com:37017"]; entity "" [01:28] 2013-09-02 01:28:30 INFO juju.state open.go:106 connection established [01:28] environment: us-west-1 [01:28] machines: [01:28] "0": [01:28] agent-state: started [01:29] pastebin pls [01:29] agent-version: 1.12.0 [01:29] dns-name: ec2-54-219-20-209.us-west-1.compute.amazonaws.com [01:29] instance-id: i-24a95f7e [01:29] instance-state: running [01:29] series: precise [01:29] hardware: arch=amd64 cpu-cores=1 cpu-power=100 mem=1740M [01:29] services: {} [01:29] 2013-09-02 01:28:33 INFO juju supercommand.go:284 command finished [01:29] you'l live :) [01:29] davecheney, i'm using us-west-2 incidentally [01:29] us-west-1 has higher pricing [01:29] * davecheney tried us-west-2 [01:30] davecheney, http://pastebin.ubuntu.com/6053452/ [01:30] 36s compared to 17s [01:30] hazmat: it's slow in us-west-2 'cos there are no tools [01:30] davecheney, there's tools in us-west-1? [01:31] i thought only in us-east [01:31] i have no idea why [01:31] tools are no region specific in ec2 [01:31] the automatic fallback to sync-tool is very confusing if you don't pass -v [01:31] hazmat: http://paste.ubuntu.com/6053456/ us-west-1 [01:32] us-west-2 is taking much longer [01:32] i can only assuming it's doing sync tools [01:32] generally with fios/fiber to the home, i have pretty good bandwidth & latency.. in terms of ruling out client connectivity. [01:33] davecheney, there's some sleep and retry logic in the ec2/s3 code for eventual consistency as well [01:33] hazmat: if I can bootstrap in 25 seconds from australia [01:34] isn't not a latency problem :) [01:34] wow [01:34] 2m9 vs 25s [01:34] it's sync tools [01:35] hazmat: still going ... [01:35] us-west-2 [01:35] davecheney, plus it looks up tools multiple times.. [01:35] in bootstrap [01:35] one to find/upload them, one to launch an instance [01:36] which gets duplicated by the provisioning worker for every launch [01:40] hazmat: http://paste.ubuntu.com/6053480/ [01:40] ends with a Fffffuuuuu [01:40] 2m4s [01:40] 'cos of sync tools [01:40] workaround: don't use us-east-2 [01:41] west-2 [01:41] that's interesting i don't see the tool sync [01:41] gotta use -v [01:41] always -v [01:41] use -v for everything [01:41] got a question [01:41] -v [01:41] need answers [01:41] -v [01:41] i've been using --debug [01:41] which seems like it should imply -v [01:41] sure [01:41] i use-v [01:41] did i mention i use -v [01:41] -v is great [01:42] http://paste.ubuntu.com/6053487/ [01:42] wut [01:42] it either is, or is not bootstrapped [01:42] davecheney, i know that bug [01:43] davecheney, its when you have a bucket but no environment, ie. shutdown from the console [01:43] davecheney, the only resolution was to destroy the environment again [01:43] whoo [01:44] or manually empty the bucket.. [01:44] hazmat: http://paste.ubuntu.com/6053492/ [01:44] 25seconds [01:46] davecheney, for comparison http://paste.ubuntu.com/6053494/ [01:46] axw: i looked at the simplestreams branch - i marked it as lgtm so long as a few issues are fixed. let me know if you have any questions. as soon as it lands, i need to munge simplestreams format a bit [01:46] wallyworld: okey dokey, thanks [01:47] 2013-09-01 14:47:35 INFO juju.environs.tools tools.go:93 picked newest version: 1.13.2 [01:47] 2013-09-01 14:48:04 DEBUG juju.provider.ec2 storage.go:52 Creating bucket [01:47] what the heck is it doing for 30 seconds there [01:48] 2013-09-01 14:48:35 DEBUG juju.provider.ec2 ec2.go:394 ec2 user data; 12541 bytes [01:48] 2013-09-01 14:48:56 DEBUG juju.provider.ec2 ec2.go:397 ec2 groups setup [01:48] hazmat, whatever ec2 enpoint you are talking to [01:48] it's screwed [01:48] davecheney, that's us-east [01:48] davecheney, its the same time for us-west [01:48] sure, but bgp says we don't see the same thing [01:49] yeah [01:49] wallyworld: where did you try adding --noprofile/--norc? [01:49] axw: in the call which invoked bash [01:49] md := exec.Command("ssh", sshHost, "bash") [01:50] but i may have not done it right, i was in a hurry [01:50] ok, that wouldn't work - bash doesn't actually get executed there [01:50] axw, maybe need -t [01:51] davecheney: ? [01:51] ssh -t? [01:51] otherwise bash won't have a pty [01:52] davecheney: yeah.. don't need it here. not actually running a real ssh in the tests [01:52] kk [01:52] * davecheney stops 'helping' [01:52] ;) [02:00] wallyworld: I'm not sure how to reproduce the problem. Can you please try "var sshscript = `#!/bin/bash --noprofile" in detection_test.go, when you have a minute [02:00] axw: sure, i'll just park some stuff and try it [02:03] axw: no good :-(. you could maybe reproduce it by having a ~/.shinit with a non zero exit code? [02:03] wallyworld: tried that.. [02:03] hrm [02:03] hmmm. was you .shinit run? [02:04] probably not :) [02:04] it is if I run "sh" [02:04] so i'm confused [02:04] I guess your bash profile is running something through sh? [02:04] ah, i think i might invoke it [02:04] elsewhere [02:05] I'm sure we have other tests which do this though [02:05] so I don't know why this one in particular would break [02:05] i have it in my .bashrc [02:05] BASH_ENV=$HOME/.shinit [02:06] but .bashrc should not be run, right? [02:06] wallyworld: what are you doing that is so special? [02:06] thumper: the line that fails is a mkdir -p foobar [02:06] axw: I suggest we just blame wallyworld and make him fix it [02:06] hehe [02:07] wallyworld: what is in your .shinit? [02:07] except we shouldn't be running profile scripts when doing the ssh :-) [02:07] wallyworld: so you could change --noprofile to --norc, but it's still a bit weird [02:07] * thumper reads that quickly as "shit init" [02:07] * axw snorts [02:07] I did the same :) [02:07] heh [02:08] thumper: i have a bunch of exports, and a line to make a tmp dir cause i have /tmp mapped to ram. the line which errors in the tests is mkdir -p /var/tmp/mailman/logs [02:08] the test error is [02:08] wallyworld: why does it fail? [02:08] ... "error detecting hardware characteristics: exit status 126 (/home/ian/.shinit: line 49: /bin/mkdir: Argument list too long\n" + [02:09] ... "/tmp/gocheck-8674665223082153551/3/ssh: line 14: /tmp/gocheck-8674665223082153551/3/ssh: Argument list too long\n" + [02:09] ... "/tmp/gocheck-8674665223082153551/3/ssh: line 14: /tmp/gocheck-8674665223082153551/3/ssh: Success)" [02:09] wallyworld: why have it in .shinit instead of .bashrc? [02:09] thumper: i call .shinit from my bashrc [02:09] why? [02:09] not sure now. there was a reason at some point [02:09] if you renamed it to something else, what fails? [02:10] if i rename it, the tests pass [02:10] but then it is not run [02:10] \o/ [02:10] well, there is still the roblem that my .bashrc is being run [02:10] in order to call .shinit [02:11] and we shouldn't be doing that [02:11] well, no [02:11] /bin/sh loads .shinit [02:11] wallyworld: can you try replacing --noprofile with --norc? [02:11] sure [02:12] damn. no good [02:13] wtf [02:13] yeah, makes no sense to me [02:15] i'll just rename the shinit [02:39] wallyworld: re "Please make this a big number so that we can ensure the fix actually works" [02:39] that test is not testing the fix at all [02:40] it's testing that standard JSON unmarshalling behaviour takes palce [02:40] place* [02:40] ah, ok. i thought that the new unmarshall/marshall stuff was being invoked all the time [02:40] if you unmarshal a number into an interface{}, you get a float64 [02:40] no, only if you go through ParseCloudMetadata [02:40] because only then do you have a template value [02:40] ok, no problem. thanks for clarifying [02:41] nps, I'll add a comment in the review too in case anyone else cares [02:41] did i make sense when i said i just wanted the marshall/unmarshall code moved out? [02:41] yes [02:41] I've kept "itemCollection" (the clone of ItemCollection) and the ItemCollection methods in json.go [02:41] everything else has gone back [02:41] is that right? [02:42] ItemCollection methods = UnmarshalJSON & construct [02:42] yeah, i think that's good. i wanted the business logic / data model as it was, and the on the wire stuff separate to it's easier to grok etc [02:42] cool [02:42] cause the user reading the simplestreams logic doesn't care about the json crap [02:43] dog walk time [02:43] and visa versa [02:43] ha, that last comment could also apply to thumper [02:43] even though it wasn't meant to [02:48] davecheney: the azure bugs can be fixed pretty quickly it seems [02:48] davecheney: any chance of slipping anything else in? [02:49] wallyworld: have you seen this? https://bugs.launchpad.net/juju-core/+bug/1219123 [02:49] axw: too late [02:49] release it tagged [02:49] we can always back port [02:49] okey dokey [02:50] axw: no, haven't seen that before [03:28] arse biscuits [03:28] some of our code is truly horrible [03:28] * thumper thinks [03:45] * thumper decides not to fix everything at once [04:03] axw: are you landing your simplestreams branch now? [04:04] wallyworld: yes. do you want to make another pass, or shall I go ahead? [04:04] axw: nah, Just Do It. we can iterate as needed [04:04] cool [04:04] i need to merge it into my work after it lands, and then change stuff to do with the data format [04:05] okey dokey. let me know if there's something you'd like me to do [04:06] also, sorry it took so long [04:07] no problem, it didn;t take long, it was fairly involved [04:18] wallyworld: just merging with trunk, I'll let you know when the bot's done [04:18] yay, thanks [04:24] or not, one of the tests is playing up [04:41] * thumper wonders how to withdraw a rietveld review === tasdomas_afk is now known as tasdomas [05:01] axw: i have school pickup, i check when i get back to see how the merge is going [05:01] wallyworld: ok, later. fixing tests atm [05:11] * thumper uses some mocks for agent.Config [05:11] it feels GOOD [05:14] hmm... [05:14] WTF went wrong there... [05:15] copy and paste went wrong there [06:02] axw: I bring up the SHA256 thing because it failed in the bot [06:03] jam: ah right. yeah there was a bug in my test code, I was accidentally writing dynamic content to the fake tools files [06:03] * thumper is almost done [06:03] must've been lucky with the gocheck temporary dir being the same on my machine [06:06] argh!!! [06:06] * thumper takes a deep breath [06:06] thumper is never almost done [06:06] :) [06:06] :P [06:06] well, never *actually* done [06:07] * thumper leaves it for now [06:10] jam: expect MS to call you if you've signed up for Azure [06:10] they woke me up the other day.. [06:11] axw: yeah. [06:12] axw: so when I "LGTM" with some comments, if you address and agree with the comments, you can feel free to just land it [06:13] jam: ok, will do [06:13] I'm just updating the default-precise comment now [06:13] and testing [06:34] 1.13.3 is released [06:34] the lp builders are lagged [06:34] i'll uploda the tarball and publish the release notes once the tools are uploaded [06:34] thanks davecheney [06:36] will look at the 1.14 branch next [06:36] /s/branch/series [06:41] i release 1.13.3 so y'all would stop putting bugs back in there :) [06:49] https://launchpad.net/juju-core/+milestone/1.14.0 [07:08] axw: just saw your test error - that's intermittent, i raised a bug 1219602 [07:08] i resubmitted and it worked [07:08] wallyworld: thanks, I was about to look into it... [07:08] cool [07:08] np, that's why i pinged you to save you the trouble :-) [07:09] wallyworld: it says "needs review" still? [07:09] axw: if a test run failes, the bot puts it back to needs review [07:10] you need to flip it to approved again [07:10] i resubmitted and it worked <- what worked? [07:10] axw: the landing bot ran the tests successfully and merged the branch [07:11] when i say resubmitted, i mean i marked the mp as approved [07:11] oh ok [07:11] yep.. [07:11] and so the bot noticed and re did its thing [07:12] I'm confused because it's not saying merged, but it doesn't matter. I'll do it again [07:16] axw: it's not saying merged because the merge did not happen because the tests failed [07:16] so the bot emailed the test failures and flipped the status back to needs approved [07:16] wallyworld: yeah I get that, but I thought you said it passed [07:17] sorry, I'm being a bit thick [07:17] it passed for *me* for my branch when i tried a second time [07:17] ahh right, on a different MP [07:17] yeah, sorry [07:17] morning! [07:17] morning dimitern [07:17] * wallyworld goes to soccer [07:21] wallyworld: it has finally merged [07:21] enjoy soccer [08:03] davecheney, I'm assuming that my next upload to saucy should be 1.14.0 right? [08:03] and thats really just 'stable release of 1.13.x'? [08:03] (need to phrase my changelog message right as I don't think its a huge jump from 1.13.2) [08:36] jamespage: that sounds right to me [08:36] * jam heads out to run a couple of errands. [08:41] jamespage: correct [08:41] will be ready by weeks end if not sooner [08:41] i wouldn't worry about 1.13.3 [09:14] so, it's a no US day today [09:20] jam: hey, seen mramm? [09:24] thumper: I assume he's out as it's a US holiday today [09:25] mgz: he's in the UK :) [09:26] oh, then I guess not :) [10:02] mgz: /wave [10:03] I haven't seen mramm all day anyway [10:05] hey jam [10:17] https://codereview.appspot.com/13272045 - mgz, jam, please take a look (uniter api's remaining unit ops) [10:24] fwereade: ^^? [10:25] dimitern, sure [10:26] fwereade: ty [10:27] dimitern: sure [10:31] mramm: hey, good to see you around [10:32] jam: in the london office for the gui sprint [10:33] mramm: hey, so we can book now for the sf sprint? [10:34] dimitern: yep [10:34] mramm: cool! [10:34] mramm: arriving 21, leaving 25 oct, right? [10:34] mramm: or thereabouts [10:42] dimitern: I think you need the final signoff from mramm before you can actually get the agency to book the travel [10:42] but you could probably start the conversation. [10:43] jam: well, following the steps in the mail, I'm filling in the form first, then will contact bts travel [10:44] dimitern: right, so after filling in the form mramm gives a final signoff on it, and you get a confirmation ID that you need to share with BTS before they'll book anything [10:44] I think you actually share the whole email with them. [10:44] jam: ok [10:45] will miss the standup, will catch up when I'm back shortly after [10:49] dimitern, couple of questions, not *quite* an LGTM, the Destroy test is the only one that gave me serious pause [10:50] fwereade: ok [10:52] fwereade: well, the uniter does call unit.Destroy [10:52] dimitern, but not on itself [10:52] fwereade: filter.go:453, uniter.go:498 [10:53] dimitern, bah, filter, I stand corrected [10:53] fwereade: :) so it's ok then? [10:55] dimitern, yeah, I think so [10:55] dimitern, it might be neater to implement the test for a unit destroying its own subordinate, though, because that's the one that's least sensitive to future change [10:55] dimitern, and you've already written that support code, so it should be easy ;p [10:56] fwereade: ok, will do [10:56] dimitern, thanks [11:07] fwereade: hmm [11:08] dimitern, eh, problems? [11:08] fwereade: I'm not sure it's possible to do an api call for a subordinate if you've logged in as the principal [11:09] dimitern, doesn't the principal destroy its own subordinates? maybe I misremembered [11:09] fwereade: i mean, it'll return ErrPerm because the auth'ed entity tag won't match [11:09] dimitern, modes.go:298 [11:09] fwereade: so the when logged in as a principal, you should be able to call Destroy() (and only that method?) on its subordinates as well [11:10] dimitern, that's the only one I can think of offhand [11:10] fwereade: ok, that'll require a few changes at server-side [11:10] dimitern, hopefully minor? [11:10] fwereade: they should be, yeah [11:11] dimitern, cool, thanks [11:11] dimitern, if it needs to be a separate CL so be it [11:12] fwereade: yeah it seems more like that [11:12] fwereade: more changes are needed actually - I need to be able to fetch a subordinate from a logged in principal [11:13] fwereade: so what do you say to leaving it as is, adding a TODO to fix it and doing the fix in a follow-up? [11:15] dimitern, LGTMed [11:15] cheers [11:15] dimitern, well, you don't need to get the subordinate for that piece of code [11:15] dimitern, that's just a Destroy loop over SubordinateNames [11:16] fwereade: not really [11:16] dimitern, hell, the first 10 lines of that mode should really just be a "destroy all subordinates plskthx" api call [11:17] fwereade: SubordinateNames returns names, not tags, and I need a uniter.Unit proxy object to call the server-side [11:17] fwereade: so I have to construct it in place, it'll be a bit ugly [11:17] dimitern, all the more reason to replace it all then -- but that's not the approach we agreed, I know [11:18] dimitern, that said [11:18] dimitern, why would we use names over the API? *shouldn't* they be subordinate tags? ;p [11:19] fwereade: we return names, but take tags as args [11:19] dimitern, that's incoherent [11:20] dimitern, how widespread is it? [11:20] fwereade: in most cases we use only tags, except for a few cases like SubordinateNames and the strings watchers [11:20] fwereade: not much, but I have to check [11:22] dimitern, so long as there's general agreement that the appropriate language is tags, not names, I'm reasonably happy [11:22] dimitern, bugger, we have deployed lifecycle watchers that use names, haven't we? [11:24] fwereade: so these return or use names: GetPrincipal, SubordinateNames,Relation (inside the RelationResult struct there's a ServiceName field), ReadRemoteSettings (only in the error message "cannot read...") [11:25] fwereade: lifecycle watchers are notifywatchers I think [11:26] fwereade: but the deployer uses a stringswatcher that returns names [11:26] dimitern, ideally I think all those would be fixed before we deployed something that expected names [11:26] dimitern, but the StringsWatcher thing has rather screwed us [11:28] fwereade: yep, so I looked [11:29] fwereade: stringswatchers, error messages, a few params struct fields, and these 2 methods of UniterAPI use names [11:31] fwereade: standup [11:33] TheMue: standup? https://plus.google.com/hangouts/_/f497381ca4d154890227b3b35a85a985b894b471 [12:54] wow, so with some of the cli now using the api, cmd/juju tests running time increased like 5 fold [13:05] * dimitern lunch [13:32] fwereade: hey [13:32] dimitern, heyhey [13:33] fwereade: the way uniter api works now, it calls Life when you call uniter.Unit(tag) [13:33]