[00:38] ericsnow: the auto thingisnot workingright? [01:00] sinzui: hey, remind me, did we agree that the simplestream content id for tools should be "com.ubuntu.juju:released:tools" always? even for devel/proposed? the devel/proposed tools like at a different url of course, but are we going to lie in the content ids inside the json? [01:01] the current code in master generates ids like com.ubuntu.juju:proposed:tools or com.ubuntu.juju:devel:tools which don't match what's in the json [01:04] wallyworld_, I don't like the lie, but we never agreed to remove it [01:04] sinzui: ok. in that case, i'm not sure how 1.21-alpha1 is finding tools [01:05] because the code is generating the wrong ids [01:05] wallyworld_, right. I think a real change requires a longer transition. [01:06] sinzui: we can make a fix for alpha2 which generates the expected content ids [01:06] sinzui: just to confirm, proposed will live at http://streams.canonical.com/juju/proposed/tools/ [01:06] yes [01:06] devel will live at http://streams.canonical.com/juju/devel/tools/ [01:07] yep [01:07] ok, that's what the code assumes, bt with the wronf content id [01:07] so we'll do a fix [01:07] wallyworld_, I think we need to teach 1.21 that it will change, but probably not change until 1.22 [01:08] sinzui: yeah, i guess it could look for the correct id and fall back to the incorrect one [01:08] wallyworld_, We never say you can upgrade from devel to devel though. alpha1 might be able to upgrade to alpha2, but we never say it is so [01:08] messy but doable [01:08] i would prefer just to fix the ids for alpha2 [01:09] ie have the json generated with com.ubuntu.juju:proposed:tools etc [01:09] but if that's not doable, we will have to code the fallback option [01:10] which will be very messy [01:13] sinzui: let me know your thoughts and we can code the solution to work with what's possible for you [01:15] wallyworld_, I think we can fix the values in the json, and announce that upgrades from a1 to a3 are not supported...unless sync-tool from alpha2 clients will work [01:17] sinzui: sync-tool should write the correct json, so it *should* work. but +1 to making it all correct now during the alpha release stage before it's too late to fix [01:18] * thumper takes a deep breath, and continues with deep refactoring dive [01:18] axw: standup? [01:20] oops [01:20] brt [01:35] mercurial hates me [01:35] $ hg pull -u [01:35] abort: error: _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed [01:35] i know _why_ this stopped working [01:35] but I don't know _how_ to fix it [01:39] davecheney: doesn't hg allow you to ignore the cert somehow? [01:43] perrito666: maybe [02:38] make: Warning: File `../../Make.dist' has modification time 1.4e+09 s in the future [02:38] dfc@bauer:~/go/src/cmd/7a$ date [02:39] Thu Jan 1 01:29:47 EST 1970 [02:39] oh wow [02:39] i guess that is why https wasn't working [03:55] axw: hi, review baord and 1.20 aren't playing nicely, could you take a look at this for me? https://github.com/juju/juju/pull/929 [04:05] wallyworld_: will do [04:05] ty [04:16] wallyworld_: what part bind-mounts the storage dir? [04:16] axw: container/lxc/lxc.go [04:17] mountHostToolsDir() [04:17] why is it doing that? :/ [04:18] so the container doesn't have to go to http storage to get tools cause there's a race condition on startup [04:18] containers would sometimes fail to download tools in cloud init [04:18] because the http storage wasn't available [04:18] yeah, so we put in retries right? [04:19] i think that was after this fix [04:19] the agent will rm -fr /var/lib/juju on destruction, so mounting anything in there is going to be bad [04:19] yes [04:19] I guess it's too late for 1.20 [04:19] hence this fix [04:20] what I mean is, this is working around it - it would be better not to mount it [04:20] the fix mounts it to /tmp instead but i guess now that retries are there the mount is no longer needed [04:20] at lease that assumes the retires will fix the original issue [04:21] i'dhave to re-read the orignal bug [04:22] wallyworld_: reviewed [04:22] ty [04:24] axw: ah, i think i remember - some hosts had port 8040 closed [04:24] hence the curl to the http provider failed [04:25] so retries won't help [04:28] wallyworld_: ah yeah. anyway, shouldn't be a problem in trunk at least [04:28] yup [04:54] axw: i'm wondering if we should randomise the choice of availability zone when none is explicitly specified and we just iterate over the available zones till we find one where startinstance works [04:55] for openstack? [04:55] yeah, but also for ec2 [04:55] i'm implementing the choice for openstack [04:56] wallyworld_: we could randomise at each level of population [04:56] since it currently always uses the first zone [04:56] otherwise it would defeat the algorithm [04:56] wallyworld_: it uses the first of the ones with the least population from the group [04:56] under the assumption that it wouldn't fail, which obviously was wrong [04:57] ah, right, in that case i might not do any extra [04:57] I suggest we just iterate through in ascending order of population [04:57] like in ec2 [04:57] yup, ok [04:57] we could always change the underlying code in provider/common later to randomise if it's valuable [04:58] axw: so in openstack, we do this: zoneInstances, err := availabilityZoneAllocations(e, group) and then zone-to-use = zoneInstances[0].ZoneName [04:59] i'm just going to iterate over the slice and try start instance on each zone till it succeeds [04:59] like in ec2 [04:59] the code in there now matches the bug symptom [04:59] where it always tries just the same zone each tome [05:12] wallyworld_: right, I agree with that approach [05:13] coolio [05:13] wallyworld_: in ec2 we check for a specific error code, and bail on others [05:13] not sure if there's an equivalent on openstack [05:13] axw: yep, openstack returns just a 500 sadly [05:13] so i gotta check the message :-( [05:13] wallyworld_: I meant error-code-string in the body [05:13] same as aws then [05:14] our goose library will return an UnspecifiedErrorCode I think. the underlying openstack message is "No valid host was found" [05:15] goose gets a http 500 response code [05:54] jam: bug 1383260 is related to scalability - do you have any comments based on your previous work in this area. i fear it will not be something we can easily fix [05:54] Bug #1383260: 1.20.10/MAAS 1.5 relations not being 'started' in a large environment

[06:12] wallyworld_: well he did say "it did eventually complete" but was slow and high CPU/memory. I don't have any specific notes on what we could do to fix that, we just need testing and profiling. [06:12] wallyworld_: He had ~300+ items and then added a subordinate, which creates 300 new units right away [06:13] jam: yep, i wasn't sure if you had any previous profiling which may have showed that issue [06:13] wallyworld_: I was aware that adding a subordinate is painful, but when I did it with 1k nodes, it pretty much just died. And I haven't done it for a long time. I had actually forgotten about it. [06:13] ok, np [06:14] i think the bug shoud be moved off 1.20 [06:14] as we won't get to fix it for that release [06:48] * fwereade will be off in a few mins, back when I'm in a hotel on the next island, not far but might be a bit of a wait for the boat [07:48] morning all [08:01] axw: thanks for review. i did [08:01] . [08:01] add a test for that eror path i think [08:01] TestStartInstanceWithUnknownAZError [08:02] wallyworld_: yeah I saw it later... sort of tests it indirectly, but that's good enough [08:02] yeah, i thought it would be sufficient [08:02] what I meant specifically was in that test, do we get az3 after failing to get az2, or did we go straight to az3 [08:02] but anyway, don't worry about it [08:04] ah, i see your point [08:05] the other tests sort of prove that we would hit az2 first [08:05] but yeah, not 100% sufficient [08:30] wallyworld_: axw: can one of you run this code and tell me what it outputs please [08:30] http://pastebin.ubuntu.com/8612039/ [08:30] I'm seeing "none", which surprises me [08:31] voidspace: hey. sure, one moment [08:31] it should print the id of the default-vpc for the account [08:31] axw: thanks [08:31] no hurry, I'm grabbing coffee [08:31] voidspace: I also get "none" [08:32] voidspace: none for me too [08:32] so either we have no default-vpc for that region, or goamz is buggy, or the amazon api is buggy [08:32] I tried a couple of different regions and got none for the ones I tried [08:32] wallyworld_: thanks [08:32] not just my credentials then [08:34] voidspace: "ec2-describe-account-attributes default-vpc" gives me something other than none [08:34] so it's not the first case [08:35] ah, cool [08:35] well, distinctly uncool [08:35] but thanks [08:36] and dimiter is off today of course :-) [08:45] voidspace: sorry [08:45] I was wrong [08:45] I forgot, I changed my default region in my dot file [08:45] so ec2-describe-account-attributes was describing ap-southeast-2 [08:46] voidspace: us-east-1 does return "none" when using that too [08:46] axw: ah [08:47] axw: I get "Client.Unauthorized: Not Authorized" [08:47] using ec2-describe-account-attributes? [08:47] yep [08:47] and the ec2-describe-regions [08:47] you sourced your creds? [08:47] yep :-) [08:48] if you don't do that then you get a traceback instead [08:48] and naturally it uses different environment variables to the ones we (and goamz) uses [08:48] dunno then - just works for me [08:48] axw: I'm reading the docs [08:49] but I'll try ap-southeast from goamz [08:50] ah yeah, I should've tried that. yep, works fine [08:50] axw: for me too [08:50] code is probably correct [08:51] I wonder why my command line tools don't work [09:39] Hmmm... my reviewboard review didn't have a reviewer set [09:39] Nor does it appear to have a diff [09:39] http://reviews.vapour.ws/r/211/ [09:39] I'll have to harass ericsnow [09:40] Ah, known issue but thought fixed... [10:39] wwitzel3, ping [10:47] jam: are you around? [10:47] voidspace: clicking on the link now [11:04] jam: http://reviews.vapour.ws/r/211/diff/1/#1 [12:05] if anyone's familiar with the details of what the uniter's trying to do with proxy settings, please let me know [12:19] fwereade: i know a little, what did you want to know? [12:20] wallyworld_, so, we rewrite the environment with new proxy settings [12:20] it basically allows the user to update the http/ftp/apt proxy settings via an environment config change, and have the hook context reflect those settings when a hook is run [12:20] wallyworld_, what about the impact on the http package? [12:21] wallyworld_, is there any? if so, do we depend on it? if so, how do we deal with the implicit races? [12:21] fwereade: well, it sets the HTTP_PROXY env var, so that will affect the http package too [12:21] wallyworld_, quite so [12:21] if hook tools invoke http calls [12:21] wallyworld_, that's the source of my concern [12:21] wallyworld_, it's not just the hooks that invoke http calls [12:21] wallyworld_, the uniter downloads charms itself [12:22] wallyworld_, and... hmm, the upgrader doesn't any more, does it? [12:22] sure, but the uniter downloading charms would need to use any http proxy setting the user deemed necessary [12:23] wallyworld_, that is, it doesn't because the machine agent's upgrader is guaranteed to have coincidentally downloaded everything already [12:23] yes [12:23] wallyworld_, ok, but what happens when the env setting change races with the charm download? [12:23] how do you mean? [12:24] wallyworld_, well, we rewrite the env -- and the http package reads the env -- what happens when they collide? can they collide? I may be worrying about nothing [12:24] wallyworld_, can we get nonsense in the proxy settings if we kick off a charm download at just the wrong time? [12:25] wallyworld_, are we maybe safe explicitly downloading charms with no proxy? given that they should always come from the local state server [12:25] wallyworld_, wait scratch that we're not [12:25] wallyworld_, manual env :/ [12:25] i must admit i don't see an issue. when an env i bootstrapped, the user chooses if they want to set the http proxy in their env.yaml. then later, they can change it using a set-env call, which sends the change to the uniters [12:26] wallyworld_, it's purely about the details of what's happening in-memory [12:26] wallyworld_, is a change of env vars atomic? [12:27] wallyworld_, even if it is, does the http package read them atomically, or could it get non-matching HTTP_PROXY and NO_PROXY values? [12:27] not sure. but the seq from a system perspective is: cange hits db -> watcher invoked -> uniter notices -> uniter calls os.SetEnv() [12:27] oh i see what you are asking [12:27] wallyworld_, yeah, I'm fine with what's going on at a high level [12:28] if the user changes 2 env settings via single set-env call, will that translate into a single EnvironConfig changed event [12:28] i would hope so [12:28] wallyworld_, even lower level [12:28] wallyworld_, how does a call to os.SetEnv affect a concurrent http.Get [12:29] no idea [12:29] i would expect that the tcp connection already made will stick [12:29] wallyworld_, I guess we have the same problem, if it is one, in the machine agent anyway [12:29] wallyworld_, that's doing the same stuff essentially [12:29] yeah [12:30] wallyworld_, ok, I will pretend it's not an issue for now [12:30] i can't see a very common use case of http proxy changing after bootstrap [12:30] but i guess it's something to be considered [12:30] wallyworld_, yeah, my suspicion is that it's a bug but not one anyone will actually hit in practice [12:30] i agree [12:31] but, knowing our luck, someone important will be unlucky [12:31] i hadn't thought about it too much, didn't write that code [12:31] wallyworld_, so what I should really do is scrap the yucky uniter code, parameterise machineenvironmentworker, and run one of those in the unit agent (that doesn't write to the global location and just hits the os.Env) [12:31] wallyworld_, no worries [12:32] hmm [12:32] I wonder, what's the worst that could happen if I just scrap the code in the uniter, and grab proxy settings for each hook context individually? [12:34] charm downloads won't use the proxy... but they're (1) almost certainly local and (2) ha, it looks like they're not guaranteed to anyway [12:34] ffs [12:35] fwereade: charm dowsnloads will soon be via the charn store [12:35] the local charm store [12:35] so it won't matter anyway [12:35] wallyworld_, yeah [12:37] * fwereade just saw a check on provider type inside a worker and experienced a brief moment of apoplectic rage [12:37] * fwereade better now [12:37] oh no, which worker? [12:49] fwereade: pong [12:50] wwitzel3, what's the status of the juju-run-in-relation-context work? [12:51] dammit I have to break for a moment [12:51] wwitzel3, would you let me know when I'm back on? hopefully just a few mins [12:52] fwereade: it just needs to be reviewed [12:55] * perrito666 just learned what apopletic rage is [13:04] wwitzel3, back, sorry [13:04] fwereade: np [13:44] w7z: did you get a chance to review http://reviews.vapour.ws/r/197/ ? [13:53] natefinch, do you expect golang 1.2 win-amd64 to "just work" when I build 386 juju.exe with goarch=386? [13:54] jcw4: yes, I think we should land, but we also need to coordinate when it happens, as I need to change the landing script at the same time [13:55] jcw4: lets do that [13:59] jcw4: oh yeah, my one question was have you checked calling to the other script still works when the pre-push one is symlinked into .git/blah ? [14:02] sinzui: yes [14:02] thank you natefinch [14:15] TheMue: I updated by PR by the way [14:15] TheMue: added a const for "none" [14:18] voidspace: hehe, thanks [14:19] voidspace: just found in my PR that we always use strings for the provider types, which is convenient but error prone [14:21] natefinch: we basically had a 1-on-1 yesterday, but we can go today if you want to [14:21] TheMue: :-) [14:22] natefinch, sorry, I am confused by go building. Do I need to run make.bat each time I switch the target architecture? [14:27] sinzui: just once per architecture... it'll put the binaries in a separate directory, and it'll use the right ones when crosscompiling [14:27] sinzui: er once per arch/OS [14:28] natefinch, that is what I hoped. this make.bat seems to think I want gcc :( [14:33] sinzui: to be honest, I just use Dave Cheney's scripts, it makes it trivial: http://dave.cheney.net/2013/07/09/an-introduction-to-cross-compilation-with-go-1-1 [14:33] ericsnow: let's have a quick 1 on 1 anyway [14:33] natefinch: k [14:34] natefinch, Me too, but in the case of windows, I cannot verify immediately what was built passes a smoke test. We build on windows just like we build our ubuntu packages on the real arch [14:37] w7z: yeah, sorry did test that - yep. [14:37] w7z: will wait to push til you're ready [14:38] jcw4: will give you the okay in a sec [14:38] ta [14:39] natefinch: sorry, hangouts died [14:39] ericsnow: haha ok [14:39] ericsnow: I was talking and talking and then noticed you weren't moving anymore [14:51] jcw4: okay, go [14:52] kk [14:58] sorry.. added x bit [15:03] I sort of wish the local provider didn't exist.... it confuses people into thinking they should use juju the way they use other deployment systems - i.e. SSH into the machine where you want to deploy stuff and start running juju there. [15:04] natefinch, me too :( [15:17] arosales, can you try a bootstrap in azure West US? I can only bootstrap in East US. [15:18] sinzui: sure. I am in a few meetings atm, but should be able to try at the top of the hour. [17:21] ok all, g'night [17:32] I'm out for a bit, ping me if the landing bot is unhappy should anyone try it later and have issues [17:35] ericsnow: btw, I made a review and it didn't give me a diff, so I closed the PR, made a new PR and now reviewboard isn't picking up the PR at all. This is probably an edge case, but wanted you to be aware. [17:37] natefinch: this is actually an issue of RB having a unique constraint on the repo revision on each review request and how discarded review requests hold on to that [17:37] ericsnow: figured it was something like that. What's the workaround? poke my PR with a new commit? [17:38] natefinch: that should do it or you could use rbt -u [17:38] natefinch: and make sure that old review request is open [17:39] gah, stupid "File was not found in the repository" [17:39] probably why there was no diff in the first place [17:39] this is from a clean master and only fixing a typo (no new files or anything) [17:40] it's amazing what happens to your terminal when you run "cat /dev/urandom" :( [17:41] natefinch: yeah, I'm looking into several of those right now [18:10] aghghh import cycle with dummy provider [18:10] the dummy provider imports stuff? That seems bad [18:11] natefinch: import apiserver [18:13] perrito666: so much for being a dummy :/ [18:13] lol [18:14] uff, more than 30C outside and I live in a country where drinking individually is seen with bad eyes [18:14] what a torture [18:19] perrito666: you need to get a cutout of a drinking buddy [18:21] jcw4: on the good side, beer is way easier to get in 1l bottles than in individuals [18:21] perrito666: lol [18:22] I think they call that a tallboy here [18:53] anyone knows exactly what is the purpose of the apiServer member of environState struct? [18:53] seems to be barely used [18:56] nope [18:58] perrito666: seems to only get stopped when environState is closed. that's the only use I see [18:59] perrito666: it boggles my mind that the dummy provider starts a real apiserver [19:00] natefinch: I am not sure if its there for some obscure use we dont know of [19:00] natefinch: yes, beats a bit the dumminess [19:01] perrito666: so, apiserver.NewServer() has a side effect of running a tls listener [19:01] perrito666: likely some tests are expecting to be able to actually connect to an API [19:02] perrito666: but it's probably worth seeing what happens if you just comment it out [19:28] well the test is killed by sigquit [19:29] morning folks [19:30] morning thumper [19:30] thumper: morning [19:39] * perrito666 ponders how to fix this without writing a dummy apiserver [19:42] perrito666: what's the import cycle? [19:42] * natefinch thinks that sounds like a bad song from the 90's [19:43] http://pastebin.ubuntu.com/8618809/ [19:43] natefinch: sounds something you would ask in a star trek episode during an engineering crisis [19:44] ug [19:45] hmm... [19:45] ericsnow: getting errors with rbt [19:45] howbazaar = howbazaar on IRC, right? [19:45] error uploading diff [19:46] bodie_: wat? [19:46] bodie_: howbazaar = thumper you mean [19:46] oooh [19:46] what's the error? [19:46] okay :) [19:46] bodie_: and yes [19:46] twitter? [19:46] bodie_: unfortunately thumper was taken [19:46] thumper, just looking for a ping back on reviewboard 178 when you have a spare minute [19:46] probably anyone named howbazaar anywhere is thumper :) [19:46] probably [19:47] sorry -- I'm not too savvy with IRL handles [19:47] surprisingly I'm neither howbazaar nor thumper IRL [19:47] lol [19:48] that's Thumper H. Bazaar to you [19:49] great psuedonym [19:49] thumper: what error are you getting? [19:49] is there an example of how to stub a machine with a given id for a test anywhere? [19:50] ericsnow: you took out that "screw you, thumper" easter egg, right? [19:50] natefinch: oh, shoot! [19:50] ericsnow: error uploading diff [19:51] thumper: no other info? [19:51] perrito666: so, you're changing peergrouper tests to do something that requires the testing package? [19:52] ericsnow: The file was not found in the repository. (HTTP 400, API Error 207) [19:52] Your review request still exists, but the diff is not attached. [19:52] thumper: yeah, a bunch of us are getting that [19:52] hmm... [19:52] natefinch: any suggestions? [19:53] thumper: ericsnow's been looking at it today, but I don't think he's figured it out. Manual upload of the diff is probably the only workaround right now. [19:54] oh? how do we do that? [19:54] * thumper found a link [19:56] got an error uploading diff: The file "api/client.go" (revision b373c16) was not found in the repository [19:57] which is bollocks [19:58] yeah, I got the same thing for my PR, which was just a 2 character change [19:59] so... anyone up for a 4k review? [20:00] Showing 36 changed files with 1,924 additions and 1,237 deletions. [20:00] * thumper feels hopeful [20:00] zoikes [20:00] * natefinch is OCR [20:00] natefinch: hey there buddy [20:00] :D [20:01] natefinch: was reviewed by menno, katco, axw and william already [20:01] I was just following up, tweaking [20:01] lots of new docstrings [20:01] well shit, I can just rubberstamp, right? [20:01] but now rb is all in a fuss [20:01] thumper: sorry about the dupes. menno and i overlapped a bit [20:01] katco: that's fine [20:01] made them easy to address [20:01] doh, guess it'll have to wait until tomorrow when someone else is OCR... [20:01] thumper: we didn't know we were reviewing the same thing. on the plus side, you got a new high score :) [20:02] * thumper is winning! [20:02] haha [20:02] thumper: so is there a diff? [20:03] sinzui: https://bugs.launchpad.net/juju-core/+bug/1370149 is not reproducible for me on trunk. [20:03] Bug #1370149: No defined 'state-servers' on environment file after bootstrap, works after run 'juju status'

[20:04] natefinch: https://github.com/juju/juju/pull/905/files, or http://paste.ubuntu.com/8618992/ [20:04] natefinch: http://reviews.vapour.ws/r/169 [20:05] thumper: hang on i have to reboot before the hangout [20:05] kk [20:06] natefinch, Shall we mark it fix committed in 1.21-alpha2? [20:07] sinzui: seems to be true [20:08] okay, thank you very much natefinch [20:09] natefinch: I am not, I actually added restore, which is part of backups which is registered as a facade by apiserver and since dummy env imports apiserver that creates the cycle [20:10] dang [20:10] and well dummyserver is used by testing which is used by the peergrouper tests [20:11] I wonder if we can break the cycle at testing -> dummy [20:11] unless the peergrouper tests need dummy server, which is possible I guess [20:11] er dummy provider [20:13] they dont seem to directly but I have not checked if seccond hand they do, I am guessing they do since when I tweak a bit dummy server to remove apiserver they end up being killed for too much wait [20:16] * perrito666 drinks a mate and says ugly things in spanish [20:28] ericsnow: it looks like the diff upload problem is trickling into existing reviews: http://reviews.vapour.ws/r/212/diff/# [20:29] natefinch: k [20:30] * perrito666 sees ericsnow twitching [21:00] ericsnow: there also seems to be a big difference in my branch where I've moved some files around [21:00] ericsnow: on github they show as moves with minor edits [21:00] ericsnow: oh rb, it shows as deletes and adds [21:01] thumper: ouch sorry me commenting on something you where reviewing might have caused you an issue [21:01] thumper: yeah, something is out of sync with the base revisions [21:01] thumper: I'm looking into the caching the reviewboard does [21:35] sinzui: question - given that 1.21 looks for streams content ids like "com.ubuntu.juju::tools", we will either need to teach 1.20 to look for such content ids, or publish metadata using both "release" and in the proposed and devel streams [21:37] we current use tools-metadata-url to tell 1.20 how to find devel and proposed tools, but once/if we change the content ids, 1.20 won't be able to find any metadata [21:40] and right now, 1.21 can't find proposed/devel metadata due to the different content ids [21:43]