[00:26] perhaps I should stop sparring the week before a gathering [00:26] I wonder if this facial bruising will be gone by Sunday [00:27] I should learn not to block punches with my face [00:50] bigjools: is it ok for me to use your maas? are you still getting that eof issue? [00:55] wallyworld_: I am, but hang on I am just completing a test [00:55] 10 mins [00:55] ok [00:56] rightio [01:12] wallyworld_: have you used the garage maas before? [01:12] nope [01:12] i've not used *ant* maas [01:12] any [01:12] how's the face? [01:15] wallyworld_: server is all yours [01:16] bigjools: ta. how do i get an oauth key for the env.yaml? [01:16] wallyworld_: it's all set up [01:16] just log in and bootstrap [01:17] ok [01:18] bigjools: i'm scared to look inside the folder called "backdoor-image". shudder [01:19] wallyworld_: left just for you [01:19] \o/ [01:46] thumper: do you have/use any bzr plugins to get diff summaries? (file names with +/- lines) [01:46] axw: bzr diff | diffstat [01:47] that's all I use [01:47] ah, didn't know diffstat [01:47] thanks [01:47] doesn't give full summary per file [01:47] although it may have options [01:47] thumper: that'll do nicely for me, thanks [01:50] wallyworld_: face is a little bruised, that's all [01:50] walked into a door [01:51] thumper: nothing to do with the missus? [01:51] boxing? [01:52] sparring at boxing, yes [01:52] the guy today is very fast [01:52] top 5 in the country and under 20 [01:52] so much younger and faster [01:52] but you don't learn unless you fight those better than you [01:54] your nick gets better and better [01:54] or possibly more ironic [01:56] heh [01:57] I have an interview with Rachel, Caitlin and her new principal later this afternoon [01:57] slight shiner to go in with [01:58] I should just ack all meek [01:58] and flinch when Rachel looks at me :) [01:58] school pickup time [01:58] * thumper walks a block [02:09] I want to test out my maas change to make sure that the allocated change is right [02:09] hence the email about the garage maas [02:57] bigjools: this maas eof thing is giving me the shits. it's failing doing a bog standard httpClient.Do(request) inside the gomaasapi client.go, complaining that "can't write HTTP request on broken connection" [02:57] so something is closing the maas client's connection [02:57] I know what it is [02:57] but i don't know what [02:57] pick me [02:58] pick me [02:58] pick me [02:58] pick me [02:58] ok! [02:58] wallyworld_: quick hangout? [02:58] sure [02:58] https://plus.google.com/hangouts/_/ca28925e2921123581a410b465ff00dda5e3c11c?hl=en [03:07] wallyworld_: I blame Go [03:07] bigjools: it's all fooked [03:07] \o/ [03:07] i just can't see what's wrong [03:08] wallyworld_: try emulating with a simple curl request [03:08] * bigjools looks in maas log as well [03:08] bigjools: i think i did that and still got an error. oh, i did it using wget [03:08] wallyworld_: what time did you try last? [03:08] you need auth header remember [03:09] bigjools: looks like remote end is closing the connection [03:09] and as it happens, I don't know [03:09] different problem [03:09] bigjools: ah yes. can you remind me of the syntax [03:09] wallyworld_: I can't :) [03:10] fat lot of good you are then [03:13] wallyworld_: what request is it making when it gets the EOF? [03:13] FWIW I get the EOF sometimes when not using --upload-tools [03:14] bigjools: http://10.0.0.9:80/MAAS/api/1.0/files/?op=list&prefix=tools [03:14] or there abouts [03:14] i had commented out the prefix to try something [03:14] bigjools: we blame maas [03:15] 10.0.0.9 - - [15/Oct/2013:11:15:17 +1000] "GET /MAAS/api/1.0/files/?op=list&prefix= HTTP/1.1" 200 807 "-" "Go 1.1 package http" [03:15] maas is doing just fine it seems --^ [03:15] bigjools: why does this command return an error then: response, err := httpClient.Do(request) [03:15] it's effectively a straight http get call [03:15] which fails [03:16] it did not do this until recently and maas has not changed around this call [03:16] the code has not changed [03:16] can you dump the request and response? I'll add the same to the maas log [03:16] it's a simple list request [03:17] bigjools: there is no response cause the request call errors [03:17] &http.Request{Method:"GET", URL:(*url.URL)(0xc200230850), Proto:"HTTP/1.1", ProtoMajor:1, ProtoMinor:1, Header:http.Header{"Authorization":[]string{"OAuth oauth_signature_method=\"PLAINTEXT\", oauth_version=\"1.0\", realm=\"MAAS+API\", oauth_consumer_key=\"M2X2ZeCSNVWer6AEHc\", oauth_token=\"v3jKFjma2gZkhasdQR\", oauth_signature=\"%26yVgdrUAVWKCRsxNXGUEzyTrTaHYebAmH\", oauth_timestamp=\"1381806551\", oauth_nonce=\"38010820\""}}, [03:17] Body:io.ReadCloser(nil), ContentLength:0, TransferEncoding:[]string(nil), Close:false, Host:"10.0.0.9:80", Form:url.Values(nil), PostForm:url.Values(nil), MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:"", RequestURI:"", TLS:(*tls.ConnectionState)(nil)} [03:17] is the request [03:18] I'll try again and dump the maas log, hang on [03:19] wallyworld_: ok that request isn't even hitting maas [03:19] \o/ [03:19] Go is getting the EOF alllll on its own [03:19] huzar! [03:20] thumper: http://10.0.0.9:80/MAAS/api/1.0/files/?op=list&prefix= [03:21] wallyworld_: can you paste a curl -v of that url ?> [03:21] wallyworld_: the last request that maas sees is /MAAS/api/1.0/files/provider-state/ [03:21] davecheney: i'll have to look up the syntax - i can't recall how to construct a curl request with the headers etc [03:23] curl -H "Authorization: OAuth " http://www.example.com [03:25] ta [03:28] bigjools: so i pasted in the entire oauth access token from the env.yaml file as the ACCESS_TOKEN, right? [03:29] no [03:29] something like: [03:29] Authorization: OAuth oauth_signature_method="PLAINTEXT", oauth_version="1.0", realm="MAAS+API", oauth_consumer_key="M2X2ZeCSNVWer6AEHc", oauth_token="v3jKFjma2gZkhasdQR", oauth_signature="%26yVgdrUAVWKCRsxNXGUEzyTrTaHYebAmH", oauth_timestamp="1381807262", oauth_nonce="2814807 [03:30] ok [03:30] it mght get a bit tricky actually [03:31] you can always re-create the request with maas-cli [03:31] which works fine, BTW [03:31] does the maas-cli log the request? [03:31] "maas-cli maas files list prefix=tools-" [03:32] yeah I can see it in the mas log [03:32] maas-cli maas files list prefix=releases/tools/juju- also works [03:32] right [03:33] so why is Go's http call failing [03:33] there are no polite answers to that [03:33] as a data point I can re-create this another way [03:34] bigjools: do you have the output from mass-cli that you can give to davecheney who wanted to see a curl -v output? [03:34] if I bootstrap without --upload-tools, destroy and then bootstrap without --upload-tools again, you get the EOF [03:34] yes [03:35] http://paste.ubuntu.com/6238905/ [03:35] that's the host side [03:35] but amounts to the same thing [03:36] davecheney: is that ^^^^ sufficient? it sure seems like a http get request via other means works, but the same request sent from Go's http.Client fails [03:37] wallyworld_: nah, i was hoping to see the curl output [03:37] i know how to interpret that [03:38] ok, i'll see if i can get it [03:38] given that the request from Go isn't hitting maas, I suspect you need to look at why the Go internals are misbehaving [03:38] no amount of curl log is going to help with that [03:39] yeah [03:44] davecheney: that's the pastebin https://pastebin.canonical.com/99027/ [03:44] but [03:44] that worked [03:44] the Go code doesn't [03:44] the values in the pastebin were harvested by printing the http.Request struct from Go [03:45] so the request is formed with the right header values etc [03:45] but it doesn't reach the server side, and instead the Do() method returns an EOF [03:45] or a "can't write HTTP request on broken connection" error [03:46] wtf [03:49] bigjools: how recently did you upgrade to Go 1.1.2? could that be an issue? [03:50] wallyworld_: how can that affect anything? I am using a packaged juju binary on this box [03:51] i only run Go 1.1.1 on mine. if it's a Go internal issue, the version may be relevant [03:51] bigjools: you could have 1.0.x [03:51] that is known to be broke [03:51] that was all the difficulty we had bad in July [03:51] I still don't know how that can affect a binary built by someone else [03:51] davecheney: go version says 1.1.2 on his maas server [03:52] also, does maas use your forked http code ? [03:52] my maas server Go version is irrelevant [03:53] bigjools: i compiled juju from source on your box [03:53] to add in debugging [03:53] sure, but given this is also a problem with the packaged juju ... [03:53] bigjools: ooooooooooh dear [03:53] i wonder if the host that build juju was using 1.1.2 [03:54] if it's on saucy, quite likely [03:54] ok [03:54] that is known to work [03:54] except for this http request :-( [03:55] is it re-using objects or otherwise similarly being stupid? [03:55] a new http request is created each time, as is a new http.Client object [03:55] oh google you chunk of crap [03:56] silently rendering the word EOF as just "of" in my seach is stupid [03:56] http://stackoverflow.com/questions/17714494/golang-http-request-results-in-eof-errors-when-making-multiple-requests-successi [03:57] "You need to set Req.Close to true" [03:57] just saw that [03:57] is it doing that? [03:57] nope, this is maas code [03:57] is gomaasapi doing that? [03:58] i don't think so, but i wonder if we do it *anywhere* [03:58] * bigjools knows not of golang esoterics [03:58] a quick code search seems to indicate we don't [03:59] only in the forked gwacl code [03:59] so why the fuck is this not failing elsewhere if it is a problem [04:00] maas is the only provider that uses OAUTH ? [04:00] ^ guess [04:01] I think so [04:01] but why is it failing like this now as well? [04:01] 1.1.2? [04:02] wallyworld_: this is easy to test out [04:02] yeah, doing it now [04:03] ok - I was also but noticed the file changed under my feet :) [04:08] bigjools: davecheney: adding in req.Close = true after creating the req seems to have worked. but wtf. we don't do that anywhere else in juju-core [04:08] \o/ [04:08] and this is in the gomaasapi library [04:08] so why just maas and not goose or goaws wtc [04:09] mystery [04:09] * bigjools 's work here is done [04:09] thanks for helping out wallyworld_ [04:09] np. but i'm pissed at it all. not sure where to direct my rage [04:09] Rob Pike? :) [04:09] i blame Go [04:10] i mean if req.Close is required, why not just default to that? [04:10] indeed [04:10] and why is it not consistent? [04:10] do we just change maasapi? [04:10] or do we try and change *all* the other places we create requests [04:11] something to do with HTTP1.1 I guess [04:11] the document is singularly unhelpful [04:11] // Close indicates whether to close the connection after [04:11] // replying to this request. [04:12] no hint of why you'd want to do that [04:12] yeah, and the error seems like the opposite of that [04:12] bigjools: how long should juju status take to come back on your maas setup? [04:12] about 10 minutes [04:13] me taps fingers on desk waiting, waiting [04:13] my theory is that maas already closed the connection and it didn't detect that until trying to send on the same one again [04:13] think yourself lucky, it used to be 20 minutes, the fast installer cut the time in half [04:14] wonder why it didn't fail before now though [04:14] and 8 of the 10 minutes is cloud-init and a reboot [04:14] did maas change? [04:14] I reckon davecheney may be right when he said it could be go 1.1.2 [04:15] maas hasn't changed here - it's using the exact same stuff [04:15] Apache frontend [04:15] and 1.1.4 shipped on Go 1.1.1? [04:15] 1.14 i mean [04:15] no idea [04:15] when did 1.1.2 hit the archive? [04:15] no idea here either [04:16] could build 1.1.6 on go 1.1.1 and see [04:16] 1.16 even [04:16] go on then, i dare you [04:17] and juju stat came back ok [04:17] i'll revert my debugging [04:18] so stackoverflow, I can't upvote an solution because I don't have enough reputation, but I *can* edit the answer to oblivion. *HEADDESK* [04:18] funny, i searched for and found that page also, but my google didn;t "fix" the EOF spelling [04:19] I used +"EOF" [04:19] i didn't [04:19] sigh [04:20] Go seems to be a minefield where there's an actual mine wherever you step [04:20] yep :-( [04:20] AND no version control :0( [04:21] what could possibly go wrong [04:21] bigjools: you should have a working binary in your ~ubuntu/go/bin directory [04:21] hurrah! [04:22] update PATH quick! [04:22] no need I can just run that binary [04:22] there's also a bootstrapped env now [04:22] huzzah [04:22] not if you want to upload tools [04:22] oh [04:22] i think it needs to be in your path, or is that for local provider [04:22] one last bitchslap [04:23] default gopath is that env anyway [04:23] i can't recall, it may only be for local provider [04:23] ok, see how it goes [05:17] hello jtv [05:17] Hi bigjools [05:17] you're not on the internal irc [05:20] Reconnecting [05:29] davecheney: you guys planning another release to go out with 13.10? [05:30] bigjools: nope [05:30] 1.1.2 has been in saucy since september [05:31] davecheney: juju release I mean [05:31] bigjools: i think so [05:31] i don't know the detalis [05:31] thumper and sinzui probably know [05:31] thanks [05:37] wallyworld_: if you have some time this week, would you mind having a look over https://codereview.appspot.com/14527043/ [05:37] there's some changes to simplestreams metadata merging in there [06:30] axw: sure, looking now. was out buying dinner after school pickup [06:32] wallyworld_: thanks [06:33] axw: just had a quick read of some of the comments. i agree in theory the resolve can go away eventually. but my view is we need to now while we transition and have to cope with older metadata etc [06:33] lenient on what's read, strict on what's written and all that [06:34] wallyworld_: yep, sounds fair enough [06:34] i reckon we can get rid of it for 1.18 [06:35] so maybe if this is going into trunk we don't need it [06:35] or the 1.16 backport has it, land in trunk, and then follow up with a trunk branch to remove it [06:36] davecheney: from your recollection, was juju 1.14 done with go 1.1.1? [06:36] wallyworld_: yeah this isn't going into 1.16 [06:37] hmmm. maybe we can/should get rid of it then [06:37] reduce complexity [06:37] all metadata should be good for 1.16 [06:37] and if it's not, we don't want to propagate the issue [06:39] bigjools: i think there's going to be a 1.18 for saucy [06:40] 1.18? not 1.16.1? [06:40] do we not do stable updates like that? [06:40] axw: i *thought* it was going to be 1.18, based off 1.17 trunk [06:40] mmkay [06:40] too many bugs etc we are fixing in trunk [06:41] i could be wrong [06:41] but that's what i heard [06:41] okey dokey [06:41] ot thought i heard [06:41] wallyworld_: in that case, I'd rather hold off on changing the resolve logic for this CL [06:42] ok [06:42] oh but... we don't support going multiple versions do we [06:42] upgrading [06:42] axw: with WriteMetadata stuff, i have done something similar now for images metadata. but unlike for tools, i pulled the WriteXXX methods out into generate.go instead if stuffing into simplestreams.go [06:42] ok [06:43] i think this is the link, i have 3 of the fuckers https://codereview.appspot.com/14663043/ [06:43] generate.go is a new file [06:44] axw: we support going from 1.14->1,16 and 1.16->1.18 [06:44] but not 1.14->1.18 [06:44] right, so doesn't matter either way then [06:47] axw: it's extra work for you, but if the resolve bit were to be removed prior to committing to trunk, the code comes out cleaner [06:47] i looked at the merge proposal, it look ok [06:48] wallyworld_: yeah that's cool, it'll definitely clean it up - I'll do that [06:48] thanks for checking over it [06:48] I'm reviewing your CL now, looking good so far [06:48] np, thanks for fixing it :-) [06:48] axw: thanks :-) if you are a masocist, i have 3 related ones. the one you are looking at is the last of 3 [06:48] are they prereqs? [06:49] yeah [06:49] https://codereview.appspot.com/14502059/ is first, and https://codereview.appspot.com/14540055/ is second [06:49] not a masochist, but I'll do what I can ;) [06:49] the end result is that creating image metadata for private clouds is *much* easier [06:49] thanks :-) [06:49] cool [06:50] with this work, the user can run the image metadata gtenerate plugin over and over to buiild up their metadata [06:50] for different series, arches etc [06:50] before, the tool was just a prototype and just did one image and overwrote each tome [06:51] wallyworld_: ok ta [06:52] rogpeppe: can you recall - did we release juju 1.14 using Go 1.1.1? [06:55] wallyworld_: hmm, not sure [06:56] rogpeppe: ok. there's a bug i'm fixing in gomaasapi which i want to blame on the go 1.1.2 upgrade [06:56] cause nothing else makes sense [06:56] wallyworld_: is that the EOF problem? [06:56] yeah [06:57] reg from http.NewRequest needs to have req.Close = true all of a sudden [06:57] but only in gomaasapi [06:57] nowhere else [06:57] wallyworld_: FWIW if Close didn't default to false, you'd almost never get any connection reuse [06:57] and no gomaasapi code has changed [06:57] wallyworld_: i wonder how long connections are kept around before they're dumped [06:58] not sure [06:58] wallyworld_: it might be the maas server dumping old http connections [06:58] but this happened during bootstrap, so < 1 second [06:58] wallyworld_: ah, that seems wrong [06:58] why did it on;y show up now? [06:58] and not before [06:58] wallyworld_: good question; does it happen reliably? [06:59] yep [06:59] only in 1.16 juju-core [06:59] not 1.14 on same maas box [06:59] wallyworld_: have you tried 1.14 compiled with go 1.1.2 ? [07:00] rogpeppe: the front end is Apache, so whatever Apache is doing... [07:00] bigjools: hmm, seems unlikely then [07:00] rogpeppe: no [07:00] wallyworld_: that would be a good way to confirm or deny your suspicions [07:00] yeah [07:01] rogpeppe: so will this have adverse performance impace for gomaasapi wrt connection resuse? [07:01] if Close is always set to True [07:01] which it needs to be to make it work [07:01] wallyworld_: possibly [07:02] wallyworld_: these are https connections, right? [07:02] i think so [07:02] ah no [07:02] http [07:02] eg http://10.0.0.9:80/MAAS/api/1.0/files/?op=list&prefix= [07:03] bbiab [07:07] I'd take bad performance over no performance :) [07:08] bigjools: it would be nice to know what's going on though [07:08] indeed [07:08] could be a Go difference from 1.1.1 to 1.1.2 [07:09] bigjools: it is possible [07:09] bigjools: that should be easy enough to confirm [07:09] aye [07:14] * rogpeppe is reading through the Go http.Transport logic [07:30] have you guys done the environment uuid yet? [07:31] rvba: I reckon that is still the best solution BTW, as you said the other one is seriouslu abusing tags [07:43] If the environment uuid is not available yet, would it make sense to use a hash of the hash of the admin-secret as an identifier? [07:47] rvba, bigjools: what's the issue here? [07:48] rogpeppe: we want to fix https://bugs.launchpad.net/maas/+bug/1239488 [07:48] <_mup_> Bug #1239488: Juju api client cannot distinguish between environments [07:48] rogpeppe: so we need a way to flag nodes to account for the fact that they belong to a juju environment. [07:49] rogpeppe: we thought about two solutions… and we think solution 2 is best: [07:49] = 3. Fix by storing the API key used to acquire nodes = [07:49] ??? [07:50] Cons: harder to pull a key if compromised [07:50] rogpeppe: arg, wrong paste, sorry. [07:50] rogpeppe: http://pad.ubuntu.com/DnNONX6kFB [07:50] rvba: i don't quite see how the UUID is an *alternative* to using tags [07:50] rvba: won't you need to tag with the UUID? [07:50] rogpeppe: no, we need the UUID in both cases. [07:51] * rogpeppe goes to get his SSO key [07:54] rvba: so are you thinking of solution 2 here? [07:54] rogpeppe: yeah [07:55] rvba: i've always wanted the env UUID to be generated when the environment is first created [07:56] rvba: it should actually be quite a simple change now [07:56] rvba: it can be done at Prepare time [07:56] rvba: and would be passed through in the environ config [07:57] rogpeppe: that would be great. [07:58] rogpeppe: we need to fix this problem today/tomorrow morning… is that something (the UUID thingy) that could be done… like… now? ;) [07:59] rvba: probably not that quickly - it involves changes in a few places. however... [07:59] rvba: we could make a change in the maas provider only [08:00] rvba: to make it generate its own uuid, to be replaced with the environment uuid at some point in the future [08:01] rogpeppe: changing the uuid will imply dealing with already deployed environments. [08:01] rvba: there's actually no particular need for it to be the *actual* environment UUID, is there? [08:02] rogpeppe: no, we just need an identifier, specific to each environment. [08:02] rvba: ok, so here's a possible way forward: [08:02] bigjools: so we don't overlap, were you going to compile juju 1.14 with go 1.1.2 on your maas box? [08:02] rvba: change maas's EnvironProvider.Prepare so that it generates its own UUID and stores it in the environ's configuration [08:03] rvba: then change StartInstance and Bootstrap to tag the instance with that tag [08:04] (plus change Instances() to pass that tag to MAAS when listing instances) [08:04] rvba: yeah [08:04] rvba: oh yeah, it's agent_name, not tag, also :-) [08:04] Right, that's a detail :) [08:05] rogpeppe: isn't EnvironProvider.Prepare called every time juju is run? [08:05] rvba: nop [08:05]