| thumper | perhaps I should stop sparring the week before a gathering | 00:26 |
|---|---|---|
| thumper | I wonder if this facial bruising will be gone by Sunday | 00:26 |
| thumper | I should learn not to block punches with my face | 00:27 |
| wallyworld_ | bigjools: is it ok for me to use your maas? are you still getting that eof issue? | 00:50 |
| bigjools | wallyworld_: I am, but hang on I am just completing a test | 00:55 |
| bigjools | 10 mins | 00:55 |
| wallyworld_ | ok | 00:55 |
| wallyworld_ | rightio | 00:56 |
| thumper | wallyworld_: have you used the garage maas before? | 01:12 |
| wallyworld_ | nope | 01:12 |
| wallyworld_ | i've not used *ant* maas | 01:12 |
| wallyworld_ | any | 01:12 |
| wallyworld_ | how's the face? | 01:12 |
| bigjools | wallyworld_: server is all yours | 01:15 |
| wallyworld_ | bigjools: ta. how do i get an oauth key for the env.yaml? | 01:16 |
| bigjools | wallyworld_: it's all set up | 01:16 |
| bigjools | just log in and bootstrap | 01:16 |
| wallyworld_ | ok | 01:17 |
| wallyworld_ | bigjools: i'm scared to look inside the folder called "backdoor-image". shudder | 01:18 |
| bigjools | wallyworld_: left just for you | 01:19 |
| wallyworld_ | \o/ | 01:19 |
| axw | thumper: do you have/use any bzr plugins to get diff summaries? (file names with +/- lines) | 01:46 |
| thumper | axw: bzr diff | diffstat | 01:46 |
| thumper | that's all I use | 01:47 |
| axw | ah, didn't know diffstat | 01:47 |
| axw | thanks | 01:47 |
| thumper | doesn't give full summary per file | 01:47 |
| thumper | although it may have options | 01:47 |
| axw | thumper: that'll do nicely for me, thanks | 01:47 |
| thumper | wallyworld_: face is a little bruised, that's all | 01:50 |
| wallyworld_ | walked into a door | 01:50 |
| bigjools | thumper: nothing to do with the missus? | 01:51 |
| axw | boxing? | 01:51 |
| thumper | sparring at boxing, yes | 01:52 |
| thumper | the guy today is very fast | 01:52 |
| thumper | top 5 in the country and under 20 | 01:52 |
| thumper | so much younger and faster | 01:52 |
| thumper | but you don't learn unless you fight those better than you | 01:52 |
| bigjools | your nick gets better and better | 01:54 |
| bigjools | or possibly more ironic | 01:54 |
| thumper | heh | 01:56 |
| thumper | I have an interview with Rachel, Caitlin and her new principal later this afternoon | 01:57 |
| thumper | slight shiner to go in with | 01:57 |
| thumper | I should just ack all meek | 01:58 |
| thumper | and flinch when Rachel looks at me :) | 01:58 |
| thumper | school pickup time | 01:58 |
| * thumper walks a block | 01:58 | |
| thumper | I want to test out my maas change to make sure that the allocated change is right | 02:09 |
| thumper | hence the email about the garage maas | 02:09 |
| wallyworld_ | bigjools: this maas eof thing is giving me the shits. it's failing doing a bog standard httpClient.Do(request) inside the gomaasapi client.go, complaining that "can't write HTTP request on broken connection" | 02:57 |
| wallyworld_ | so something is closing the maas client's connection | 02:57 |
| thumper | I know what it is | 02:57 |
| wallyworld_ | but i don't know what | 02:57 |
| thumper | pick me | 02:57 |
| thumper | pick me | 02:58 |
| thumper | pick me | 02:58 |
| thumper | pick me | 02:58 |
| wallyworld_ | ok! | 02:58 |
| thumper | wallyworld_: quick hangout? | 02:58 |
| wallyworld_ | sure | 02:58 |
| thumper | https://plus.google.com/hangouts/_/ca28925e2921123581a410b465ff00dda5e3c11c?hl=en | 02:58 |
| bigjools | wallyworld_: I blame Go | 03:07 |
| wallyworld_ | bigjools: it's all fooked | 03:07 |
| bigjools | \o/ | 03:07 |
| wallyworld_ | i just can't see what's wrong | 03:07 |
| bigjools | wallyworld_: try emulating with a simple curl request | 03:08 |
| * bigjools looks in maas log as well | 03:08 | |
| wallyworld_ | bigjools: i think i did that and still got an error. oh, i did it using wget | 03:08 |
| bigjools | wallyworld_: what time did you try last? | 03:08 |
| bigjools | you need auth header remember | 03:08 |
| wallyworld_ | bigjools: looks like remote end is closing the connection | 03:09 |
| thumper | and as it happens, I don't know | 03:09 |
| thumper | different problem | 03:09 |
| wallyworld_ | bigjools: ah yes. can you remind me of the syntax | 03:09 |
| bigjools | wallyworld_: I can't :) | 03:09 |
| wallyworld_ | fat lot of good you are then | 03:10 |
| bigjools | wallyworld_: what request is it making when it gets the EOF? | 03:13 |
| bigjools | FWIW I get the EOF sometimes when not using --upload-tools | 03:13 |
| wallyworld_ | bigjools: http://10.0.0.9:80/MAAS/api/1.0/files/?op=list&prefix=tools | 03:14 |
| wallyworld_ | or there abouts | 03:14 |
| wallyworld_ | i had commented out the prefix to try something | 03:14 |
| thumper | bigjools: we blame maas | 03:14 |
| bigjools | 10.0.0.9 - - [15/Oct/2013:11:15:17 +1000] "GET /MAAS/api/1.0/files/?op=list&prefix= HTTP/1.1" 200 807 "-" "Go 1.1 package http" | 03:15 |
| bigjools | maas is doing just fine it seems --^ | 03:15 |
| wallyworld_ | bigjools: why does this command return an error then: response, err := httpClient.Do(request) | 03:15 |
| wallyworld_ | it's effectively a straight http get call | 03:15 |
| wallyworld_ | which fails | 03:15 |
| bigjools | it did not do this until recently and maas has not changed around this call | 03:16 |
| wallyworld_ | the code has not changed | 03:16 |
| bigjools | can you dump the request and response? I'll add the same to the maas log | 03:16 |
| wallyworld_ | it's a simple list request | 03:16 |
| wallyworld_ | bigjools: there is no response cause the request call errors | 03:17 |
| wallyworld_ | &http.Request{Method:"GET", URL:(*url.URL)(0xc200230850), Proto:"HTTP/1.1", ProtoMajor:1, ProtoMinor:1, Header:http.Header{"Authorization":[]string{"OAuth oauth_signature_method=\"PLAINTEXT\", oauth_version=\"1.0\", realm=\"MAAS+API\", oauth_consumer_key=\"M2X2ZeCSNVWer6AEHc\", oauth_token=\"v3jKFjma2gZkhasdQR\", oauth_signature=\"%26yVgdrUAVWKCRsxNXGUEzyTrTaHYebAmH\", oauth_timestamp=\"1381806551\", oauth_nonce=\"38010820\""}}, | 03:17 |
| wallyworld_ | Body:io.ReadCloser(nil), ContentLength:0, TransferEncoding:[]string(nil), Close:false, Host:"10.0.0.9:80", Form:url.Values(nil), PostForm:url.Values(nil), MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:"", RequestURI:"", TLS:(*tls.ConnectionState)(nil)} | 03:17 |
| wallyworld_ | is the request | 03:17 |
| bigjools | I'll try again and dump the maas log, hang on | 03:18 |
| bigjools | wallyworld_: ok that request isn't even hitting maas | 03:19 |
| wallyworld_ | \o/ | 03:19 |
| bigjools | Go is getting the EOF alllll on its own | 03:19 |
| wallyworld_ | huzar! | 03:19 |
| wallyworld_ | thumper: http://10.0.0.9:80/MAAS/api/1.0/files/?op=list&prefix= | 03:20 |
| davecheney | wallyworld_: can you paste a curl -v of that url ?> | 03:21 |
| bigjools | wallyworld_: the last request that maas sees is /MAAS/api/1.0/files/provider-state/ | 03:21 |
| wallyworld_ | davecheney: i'll have to look up the syntax - i can't recall how to construct a curl request with the headers etc | 03:21 |
| bigjools | curl -H "Authorization: OAuth <ACCESS_TOKEN>" http://www.example.com | 03:23 |
| wallyworld_ | ta | 03:25 |
| wallyworld_ | bigjools: so i pasted in the entire oauth access token from the env.yaml file as the ACCESS_TOKEN, right? | 03:28 |
| bigjools | no | 03:29 |
| bigjools | something like: | 03:29 |
| bigjools | Authorization: OAuth oauth_signature_method="PLAINTEXT", oauth_version="1.0", realm="MAAS+API", oauth_consumer_key="M2X2ZeCSNVWer6AEHc", oauth_token="v3jKFjma2gZkhasdQR", oauth_signature="%26yVgdrUAVWKCRsxNXGUEzyTrTaHYebAmH", oauth_timestamp="1381807262", oauth_nonce="2814807 | 03:29 |
| wallyworld_ | ok | 03:30 |
| bigjools | it mght get a bit tricky actually | 03:30 |
| bigjools | you can always re-create the request with maas-cli | 03:31 |
| bigjools | which works fine, BTW | 03:31 |
| wallyworld_ | does the maas-cli log the request? | 03:31 |
| bigjools | "maas-cli maas files list prefix=tools-" | 03:31 |
| bigjools | yeah I can see it in the mas log | 03:32 |
| wallyworld_ | maas-cli maas files list prefix=releases/tools/juju- also works | 03:32 |
| bigjools | right | 03:32 |
| wallyworld_ | so why is Go's http call failing | 03:33 |
| bigjools | there are no polite answers to that | 03:33 |
| bigjools | as a data point I can re-create this another way | 03:33 |
| wallyworld_ | bigjools: do you have the output from mass-cli that you can give to davecheney who wanted to see a curl -v output? | 03:34 |
| bigjools | if I bootstrap without --upload-tools, destroy and then bootstrap without --upload-tools again, you get the EOF | 03:34 |
| bigjools | yes | 03:34 |
| bigjools | http://paste.ubuntu.com/6238905/ | 03:35 |
| bigjools | that's the host side | 03:35 |
| bigjools | but amounts to the same thing | 03:35 |
| wallyworld_ | davecheney: is that ^^^^ sufficient? it sure seems like a http get request via other means works, but the same request sent from Go's http.Client fails | 03:36 |
| davecheney | wallyworld_: nah, i was hoping to see the curl output | 03:37 |
| davecheney | i know how to interpret that | 03:37 |
| wallyworld_ | ok, i'll see if i can get it | 03:38 |
| bigjools | given that the request from Go isn't hitting maas, I suspect you need to look at why the Go internals are misbehaving | 03:38 |
| bigjools | no amount of curl log is going to help with that | 03:38 |
| wallyworld_ | yeah | 03:39 |
| wallyworld_ | davecheney: that's the pastebin https://pastebin.canonical.com/99027/ | 03:44 |
| wallyworld_ | but | 03:44 |
| wallyworld_ | that worked | 03:44 |
| wallyworld_ | the Go code doesn't | 03:44 |
| wallyworld_ | the values in the pastebin were harvested by printing the http.Request struct from Go | 03:44 |
| wallyworld_ | so the request is formed with the right header values etc | 03:45 |
| wallyworld_ | but it doesn't reach the server side, and instead the Do() method returns an EOF | 03:45 |
| wallyworld_ | or a "can't write HTTP request on broken connection" error | 03:45 |
| wallyworld_ | wtf | 03:46 |
| wallyworld_ | bigjools: how recently did you upgrade to Go 1.1.2? could that be an issue? | 03:49 |
| bigjools | wallyworld_: how can that affect anything? I am using a packaged juju binary on this box | 03:50 |
| wallyworld_ | i only run Go 1.1.1 on mine. if it's a Go internal issue, the version may be relevant | 03:51 |
| davecheney | bigjools: you could have 1.0.x | 03:51 |
| davecheney | that is known to be broke | 03:51 |
| davecheney | that was all the difficulty we had bad in July | 03:51 |
| bigjools | I still don't know how that can affect a binary built by someone else | 03:51 |
| wallyworld_ | davecheney: go version says 1.1.2 on his maas server | 03:51 |
| davecheney | also, does maas use your forked http code ? | 03:52 |
| bigjools | my maas server Go version is irrelevant | 03:52 |
| wallyworld_ | bigjools: i compiled juju from source on your box | 03:53 |
| wallyworld_ | to add in debugging | 03:53 |
| bigjools | sure, but given this is also a problem with the packaged juju ... | 03:53 |
| davecheney | bigjools: ooooooooooh dear | 03:53 |
| davecheney | i wonder if the host that build juju was using 1.1.2 | 03:53 |
| bigjools | if it's on saucy, quite likely | 03:54 |
| davecheney | ok | 03:54 |
| davecheney | that is known to work | 03:54 |
| wallyworld_ | except for this http request :-( | 03:54 |
| bigjools | is it re-using objects or otherwise similarly being stupid? | 03:55 |
| wallyworld_ | a new http request is created each time, as is a new http.Client object | 03:55 |
| bigjools | oh google you chunk of crap | 03:55 |
| bigjools | silently rendering the word EOF as just "of" in my seach is stupid | 03:56 |
| bigjools | http://stackoverflow.com/questions/17714494/golang-http-request-results-in-eof-errors-when-making-multiple-requests-successi | 03:56 |
| bigjools | "You need to set Req.Close to true" | 03:57 |
| wallyworld_ | just saw that | 03:57 |
| bigjools | is it doing that? | 03:57 |
| wallyworld_ | nope, this is maas code | 03:57 |
| bigjools | is gomaasapi doing that? | 03:57 |
| wallyworld_ | i don't think so, but i wonder if we do it *anywhere* | 03:58 |
| * bigjools knows not of golang esoterics | 03:58 | |
| wallyworld_ | a quick code search seems to indicate we don't | 03:58 |
| wallyworld_ | only in the forked gwacl code | 03:59 |
| wallyworld_ | so why the fuck is this not failing elsewhere if it is a problem | 03:59 |
| davecheney | maas is the only provider that uses OAUTH ? | 04:00 |
| davecheney | ^ guess | 04:00 |
| bigjools | I think so | 04:01 |
| bigjools | but why is it failing like this now as well? | 04:01 |
| bigjools | 1.1.2? | 04:01 |
| bigjools | wallyworld_: this is easy to test out | 04:02 |
| wallyworld_ | yeah, doing it now | 04:02 |
| bigjools | ok - I was also but noticed the file changed under my feet :) | 04:03 |
| wallyworld_ | bigjools: davecheney: adding in req.Close = true after creating the req seems to have worked. but wtf. we don't do that anywhere else in juju-core | 04:08 |
| bigjools | \o/ | 04:08 |
| wallyworld_ | and this is in the gomaasapi library | 04:08 |
| wallyworld_ | so why just maas and not goose or goaws wtc | 04:08 |
| bigjools | mystery | 04:09 |
| * bigjools 's work here is done | 04:09 | |
| bigjools | thanks for helping out wallyworld_ | 04:09 |
| wallyworld_ | np. but i'm pissed at it all. not sure where to direct my rage | 04:09 |
| bigjools | Rob Pike? :) | 04:09 |
| wallyworld_ | i blame Go | 04:09 |
| wallyworld_ | i mean if req.Close is required, why not just default to that? | 04:10 |
| bigjools | indeed | 04:10 |
| wallyworld_ | and why is it not consistent? | 04:10 |
| wallyworld_ | do we just change maasapi? | 04:10 |
| wallyworld_ | or do we try and change *all* the other places we create requests | 04:10 |
| bigjools | something to do with HTTP1.1 I guess | 04:11 |
| bigjools | the document is singularly unhelpful | 04:11 |
| bigjools | // Close indicates whether to close the connection after | 04:11 |
| bigjools | // replying to this request. | 04:11 |
| bigjools | no hint of why you'd want to do that | 04:12 |
| wallyworld_ | yeah, and the error seems like the opposite of that | 04:12 |
| wallyworld_ | bigjools: how long should juju status take to come back on your maas setup? | 04:12 |
| bigjools | about 10 minutes | 04:12 |
| wallyworld_ | me taps fingers on desk waiting, waiting | 04:13 |
| bigjools | my theory is that maas already closed the connection and it didn't detect that until trying to send on the same one again | 04:13 |
| bigjools | think yourself lucky, it used to be 20 minutes, the fast installer cut the time in half | 04:13 |
| wallyworld_ | wonder why it didn't fail before now though | 04:14 |
| bigjools | and 8 of the 10 minutes is cloud-init and a reboot | 04:14 |
| wallyworld_ | did maas change? | 04:14 |
| bigjools | I reckon davecheney may be right when he said it could be go 1.1.2 | 04:14 |
| bigjools | maas hasn't changed here - it's using the exact same stuff | 04:15 |
| bigjools | Apache frontend | 04:15 |
| wallyworld_ | and 1.1.4 shipped on Go 1.1.1? | 04:15 |
| wallyworld_ | 1.14 i mean | 04:15 |
| bigjools | no idea | 04:15 |
| bigjools | when did 1.1.2 hit the archive? | 04:15 |
| wallyworld_ | no idea here either | 04:15 |
| bigjools | could build 1.1.6 on go 1.1.1 and see | 04:16 |
| bigjools | 1.16 even | 04:16 |
| wallyworld_ | go on then, i dare you | 04:16 |
| wallyworld_ | and juju stat came back ok | 04:17 |
| wallyworld_ | i'll revert my debugging | 04:17 |
| bigjools | so stackoverflow, I can't upvote an solution because I don't have enough reputation, but I *can* edit the answer to oblivion. *HEADDESK* | 04:18 |
| wallyworld_ | funny, i searched for and found that page also, but my google didn;t "fix" the EOF spelling | 04:18 |
| bigjools | I used +"EOF" | 04:19 |
| wallyworld_ | i didn't | 04:19 |
| bigjools | sigh | 04:19 |
| bigjools | Go seems to be a minefield where there's an actual mine wherever you step | 04:20 |
| wallyworld_ | yep :-( | 04:20 |
| wallyworld_ | AND no version control :0( | 04:20 |
| wallyworld_ | what could possibly go wrong | 04:21 |
| wallyworld_ | bigjools: you should have a working binary in your ~ubuntu/go/bin directory | 04:21 |
| bigjools | hurrah! | 04:21 |
| wallyworld_ | update PATH quick! | 04:22 |
| bigjools | no need I can just run that binary | 04:22 |
| wallyworld_ | there's also a bootstrapped env now | 04:22 |
| bigjools | huzzah | 04:22 |
| wallyworld_ | not if you want to upload tools | 04:22 |
| bigjools | oh | 04:22 |
| wallyworld_ | i think it needs to be in your path, or is that for local provider | 04:22 |
| bigjools | one last bitchslap | 04:22 |
| bigjools | default gopath is that env anyway | 04:23 |
| wallyworld_ | i can't recall, it may only be for local provider | 04:23 |
| wallyworld_ | ok, see how it goes | 04:23 |
| bigjools | hello jtv | 05:17 |
| jtv | Hi bigjools | 05:17 |
| bigjools | you're not on the internal irc | 05:17 |
| jtv | Reconnecting | 05:20 |
| bigjools | davecheney: you guys planning another release to go out with 13.10? | 05:29 |
| davecheney | bigjools: nope | 05:30 |
| davecheney | 1.1.2 has been in saucy since september | 05:30 |
| bigjools | davecheney: juju release I mean | 05:31 |
| davecheney | bigjools: i think so | 05:31 |
| davecheney | i don't know the detalis | 05:31 |
| davecheney | thumper and sinzui probably know | 05:31 |
| bigjools | thanks | 05:31 |
| axw | wallyworld_: if you have some time this week, would you mind having a look over https://codereview.appspot.com/14527043/ | 05:37 |
| axw | there's some changes to simplestreams metadata merging in there | 05:37 |
| wallyworld_ | axw: sure, looking now. was out buying dinner after school pickup | 06:30 |
| axw | wallyworld_: thanks | 06:32 |
| wallyworld_ | axw: just had a quick read of some of the comments. i agree in theory the resolve can go away eventually. but my view is we need to now while we transition and have to cope with older metadata etc | 06:33 |
| wallyworld_ | lenient on what's read, strict on what's written and all that | 06:33 |
| axw | wallyworld_: yep, sounds fair enough | 06:34 |
| wallyworld_ | i reckon we can get rid of it for 1.18 | 06:34 |
| wallyworld_ | so maybe if this is going into trunk we don't need it | 06:35 |
| wallyworld_ | or the 1.16 backport has it, land in trunk, and then follow up with a trunk branch to remove it | 06:35 |
| wallyworld_ | davecheney: from your recollection, was juju 1.14 done with go 1.1.1? | 06:36 |
| axw | wallyworld_: yeah this isn't going into 1.16 | 06:36 |
| wallyworld_ | hmmm. maybe we can/should get rid of it then | 06:37 |
| wallyworld_ | reduce complexity | 06:37 |
| wallyworld_ | all metadata should be good for 1.16 | 06:37 |
| wallyworld_ | and if it's not, we don't want to propagate the issue | 06:37 |
| wallyworld_ | bigjools: i think there's going to be a 1.18 for saucy | 06:39 |
| axw | 1.18? not 1.16.1? | 06:40 |
| axw | do we not do stable updates like that? | 06:40 |
| wallyworld_ | axw: i *thought* it was going to be 1.18, based off 1.17 trunk | 06:40 |
| axw | mmkay | 06:40 |
| wallyworld_ | too many bugs etc we are fixing in trunk | 06:40 |
| wallyworld_ | i could be wrong | 06:41 |
| wallyworld_ | but that's what i heard | 06:41 |
| axw | okey dokey | 06:41 |
| wallyworld_ | ot thought i heard | 06:41 |
| axw | wallyworld_: in that case, I'd rather hold off on changing the resolve logic for this CL | 06:41 |
| wallyworld_ | ok | 06:42 |
| axw | oh but... we don't support going multiple versions do we | 06:42 |
| axw | upgrading | 06:42 |
| wallyworld_ | axw: with WriteMetadata stuff, i have done something similar now for images metadata. but unlike for tools, i pulled the WriteXXX methods out into generate.go instead if stuffing into simplestreams.go | 06:42 |
| axw | ok | 06:42 |
| wallyworld_ | i think this is the link, i have 3 of the fuckers https://codereview.appspot.com/14663043/ | 06:43 |
| wallyworld_ | generate.go is a new file | 06:43 |
| wallyworld_ | axw: we support going from 1.14->1,16 and 1.16->1.18 | 06:44 |
| wallyworld_ | but not 1.14->1.18 | 06:44 |
| axw | right, so doesn't matter either way then | 06:44 |
| wallyworld_ | axw: it's extra work for you, but if the resolve bit were to be removed prior to committing to trunk, the code comes out cleaner | 06:47 |
| wallyworld_ | i looked at the merge proposal, it look ok | 06:47 |
| axw | wallyworld_: yeah that's cool, it'll definitely clean it up - I'll do that | 06:48 |
| axw | thanks for checking over it | 06:48 |
| axw | I'm reviewing your CL now, looking good so far | 06:48 |
| wallyworld_ | np, thanks for fixing it :-) | 06:48 |
| wallyworld_ | axw: thanks :-) if you are a masocist, i have 3 related ones. the one you are looking at is the last of 3 | 06:48 |
| axw | are they prereqs? | 06:48 |
| wallyworld_ | yeah | 06:49 |
| wallyworld_ | https://codereview.appspot.com/14502059/ is first, and https://codereview.appspot.com/14540055/ is second | 06:49 |
| axw | not a masochist, but I'll do what I can ;) | 06:49 |
| wallyworld_ | the end result is that creating image metadata for private clouds is *much* easier | 06:49 |
| wallyworld_ | thanks :-) | 06:49 |
| axw | cool | 06:49 |
| wallyworld_ | with this work, the user can run the image metadata gtenerate plugin over and over to buiild up their metadata | 06:50 |
| wallyworld_ | for different series, arches etc | 06:50 |
| wallyworld_ | before, the tool was just a prototype and just did one image and overwrote each tome | 06:50 |
| bigjools | wallyworld_: ok ta | 06:51 |
| wallyworld_ | rogpeppe: can you recall - did we release juju 1.14 using Go 1.1.1? | 06:52 |
| rogpeppe | wallyworld_: hmm, not sure | 06:55 |
| wallyworld_ | rogpeppe: ok. there's a bug i'm fixing in gomaasapi which i want to blame on the go 1.1.2 upgrade | 06:56 |
| wallyworld_ | cause nothing else makes sense | 06:56 |
| rogpeppe | wallyworld_: is that the EOF problem? | 06:56 |
| wallyworld_ | yeah | 06:56 |
| wallyworld_ | reg from http.NewRequest needs to have req.Close = true all of a sudden | 06:57 |
| wallyworld_ | but only in gomaasapi | 06:57 |
| wallyworld_ | nowhere else | 06:57 |
| rogpeppe | wallyworld_: FWIW if Close didn't default to false, you'd almost never get any connection reuse | 06:57 |
| wallyworld_ | and no gomaasapi code has changed | 06:57 |
| rogpeppe | wallyworld_: i wonder how long connections are kept around before they're dumped | 06:57 |
| wallyworld_ | not sure | 06:58 |
| rogpeppe | wallyworld_: it might be the maas server dumping old http connections | 06:58 |
| wallyworld_ | but this happened during bootstrap, so < 1 second | 06:58 |
| rogpeppe | wallyworld_: ah, that seems wrong | 06:58 |
| wallyworld_ | why did it on;y show up now? | 06:58 |
| wallyworld_ | and not before | 06:58 |
| rogpeppe | wallyworld_: good question; does it happen reliably? | 06:58 |
| wallyworld_ | yep | 06:59 |
| wallyworld_ | only in 1.16 juju-core | 06:59 |
| wallyworld_ | not 1.14 on same maas box | 06:59 |
| rogpeppe | wallyworld_: have you tried 1.14 compiled with go 1.1.2 ? | 06:59 |
| bigjools | rogpeppe: the front end is Apache, so whatever Apache is doing... | 07:00 |
| rogpeppe | bigjools: hmm, seems unlikely then | 07:00 |
| wallyworld_ | rogpeppe: no | 07:00 |
| rogpeppe | wallyworld_: that would be a good way to confirm or deny your suspicions | 07:00 |
| wallyworld_ | yeah | 07:00 |
| wallyworld_ | rogpeppe: so will this have adverse performance impace for gomaasapi wrt connection resuse? | 07:01 |
| wallyworld_ | if Close is always set to True | 07:01 |
| wallyworld_ | which it needs to be to make it work | 07:01 |
| rogpeppe | wallyworld_: possibly | 07:01 |
| rogpeppe | wallyworld_: these are https connections, right? | 07:02 |
| wallyworld_ | i think so | 07:02 |
| wallyworld_ | ah no | 07:02 |
| wallyworld_ | http | 07:02 |
| wallyworld_ | eg http://10.0.0.9:80/MAAS/api/1.0/files/?op=list&prefix= | 07:02 |
| wallyworld_ | bbiab | 07:03 |
| bigjools | I'd take bad performance over no performance :) | 07:07 |
| rogpeppe | bigjools: it would be nice to know what's going on though | 07:08 |
| bigjools | indeed | 07:08 |
| bigjools | could be a Go difference from 1.1.1 to 1.1.2 | 07:08 |
| rogpeppe | bigjools: it is possible | 07:09 |
| rogpeppe | bigjools: that should be easy enough to confirm | 07:09 |
| bigjools | aye | 07:09 |
| * rogpeppe is reading through the Go http.Transport logic | 07:14 | |
| bigjools | have you guys done the environment uuid yet? | 07:30 |
| bigjools | rvba: I reckon that is still the best solution BTW, as you said the other one is seriouslu abusing tags | 07:31 |
| rvba | If the environment uuid is not available yet, would it make sense to use a hash of the hash of the admin-secret as an identifier? | 07:43 |
| rogpeppe | rvba, bigjools: what's the issue here? | 07:47 |
| rvba | rogpeppe: we want to fix https://bugs.launchpad.net/maas/+bug/1239488 | 07:48 |
| _mup_ | Bug #1239488: Juju api client cannot distinguish between environments <MAAS:Triaged> <https://launchpad.net/bugs/1239488> | 07:48 |
| rvba | rogpeppe: so we need a way to flag nodes to account for the fact that they belong to a juju environment. | 07:48 |
| rvba | rogpeppe: we thought about two solutions… and we think solution 2 is best: | 07:49 |
| rvba | = 3. Fix by storing the API key used to acquire nodes = | 07:49 |
| rvba | ??? | 07:49 |
| rvba | Cons: harder to pull a key if compromised | 07:50 |
| rvba | rogpeppe: arg, wrong paste, sorry. | 07:50 |
| rvba | rogpeppe: http://pad.ubuntu.com/DnNONX6kFB | 07:50 |
| rogpeppe | rvba: i don't quite see how the UUID is an *alternative* to using tags | 07:50 |
| rogpeppe | rvba: won't you need to tag with the UUID? | 07:50 |
| rvba | rogpeppe: no, we need the UUID in both cases. | 07:50 |
| * rogpeppe goes to get his SSO key | 07:51 | |
| rogpeppe | rvba: so are you thinking of solution 2 here? | 07:54 |
| rvba | rogpeppe: yeah | 07:54 |
| rogpeppe | rvba: i've always wanted the env UUID to be generated when the environment is first created | 07:55 |
| rogpeppe | rvba: it should actually be quite a simple change now | 07:56 |
| rogpeppe | rvba: it can be done at Prepare time | 07:56 |
| rogpeppe | rvba: and would be passed through in the environ config | 07:56 |
| rvba | rogpeppe: that would be great. | 07:57 |
| rvba | rogpeppe: we need to fix this problem today/tomorrow morning… is that something (the UUID thingy) that could be done… like… now? ;) | 07:58 |
| rogpeppe | rvba: probably not that quickly - it involves changes in a few places. however... | 07:59 |
| rogpeppe | rvba: we could make a change in the maas provider only | 07:59 |
| rogpeppe | rvba: to make it generate its own uuid, to be replaced with the environment uuid at some point in the future | 08:00 |
| rvba | rogpeppe: changing the uuid will imply dealing with already deployed environments. | 08:01 |
| rogpeppe | rvba: there's actually no particular need for it to be the *actual* environment UUID, is there? | 08:01 |
| rvba | rogpeppe: no, we just need an identifier, specific to each environment. | 08:02 |
| rogpeppe | rvba: ok, so here's a possible way forward: | 08:02 |
| wallyworld_ | bigjools: so we don't overlap, were you going to compile juju 1.14 with go 1.1.2 on your maas box? | 08:02 |
| rogpeppe | rvba: change maas's EnvironProvider.Prepare so that it generates its own UUID and stores it in the environ's configuration | 08:02 |
| rogpeppe | rvba: then change StartInstance and Bootstrap to tag the instance with that tag | 08:03 |
| rvba | (plus change Instances() to pass that tag to MAAS when listing instances) | 08:04 |
| rogpeppe | rvba: yeah | 08:04 |
| rogpeppe | rvba: oh yeah, it's agent_name, not tag, also :-) | 08:04 |
| rvba | Right, that's a detail :) | 08:04 |
| rvba | rogpeppe: isn't EnvironProvider.Prepare called every time juju is run? | 08:05 |
| rogpeppe | rvba: nop | 08:05 |
| rogpeppe | e | 08:05 |
| rogpeppe | rvba: it's called just once for a given environment | 08:05 |
| rvba | I mean, I'm not sure I see how the uuid would be created once and persisted. | 08:05 |
| rogpeppe | rvba: after Prepare is called, all the config attributes for that environment are stored in the .jenv file (as BootstrapAttrs) | 08:05 |
| rvba | rogpeppe: I see… so if you manually get rid of the .jenv you'll be in trouble then. | 08:06 |
| rvba | rogpeppe: what about using hash(admin-secret) ? | 08:06 |
| rogpeppe | rvba: i don't like that idea | 08:07 |
| rogpeppe | rvba: there's nothing guaranteeing that admin-secret is unique | 08:07 |
| rogpeppe | rvba: you'll be in trouble if you manually get rid of the .jenv file anyway | 08:08 |
| rvba | rogpeppe: all right :). Then this seems like the best option. | 08:08 |
| TheMue | morning | 08:11 |
| rvba | bigjools: I updated the plan with rogpeppe's idea. | 08:11 |
| rogpeppe | rvba: for an example of a Prepare method that adds an attribute, take a look at openstack's environProvider.Prepare method | 08:12 |
| rvba | rogpeppe: okay, thanks. | 08:13 |
| rogpeppe | rvba: although that method allows the control-bucket to be overridden in environments.yaml, and i'm not sure you'd want to allow that for the uuid | 08:13 |
| rvba | Probably not. | 08:14 |
| rogpeppe | rvba: you'd probably want to make the method return an error if the uuid is already specified | 08:14 |
| bigjools | wallyworld_: no, go for it | 08:18 |
| rogpeppe | afk | 08:18 |
| bigjools | rvba: ok. Existing deployments will be a problem IMO | 08:20 |
| rvba | bigjools: true. That's the only remaining problem. | 08:21 |
| bigjools | rvba: and it's a hard one | 08:21 |
| bigjools | rvba: although we could get it to work if juju only generates its uuid on bootstrapping | 08:21 |
| rvba | bigjools: one things we could do is detect that we don't have generated a UUID and that the env is already bootstraped, and in this case use '' as the agent name, | 08:21 |
| bigjools | and existing deployments can stay without the uuid | 08:21 |
| rvba | thing* | 08:22 |
| bigjools | brb | 08:23 |
| rogpeppe | rvba: if you've upgraded a legacy environment, the uuid in the config will be unset | 08:34 |
| rogpeppe | rvba: so that case should be easy to cater for | 08:35 |
| rogpeppe | rvba: although it's perhaps a problem that upgrading to the new juju won't fix the current problem for existing envs | 08:35 |
| rvba | rogpeppe: I don't really think we have a choice here. | 08:36 |
| rogpeppe | rvba: is the plan to make the agent_name dynamically changeable, or will it only be specifiable when an instance is started? | 08:36 |
| rvba | rogpeppe: I don't see why we should make it changeable. | 08:37 |
| rogpeppe | rvba: if it was, it might be possible to fix existing deployments | 08:37 |
| rogpeppe | axw_: you've got a review https://codereview.appspot.com/14430064/ | 08:54 |
| wallyworld_ | rogpeppe: sadly, juju-core does not appear to have been tagged for the 1.14 release. neither has goose. and other dependencies like gomaasapi have no tags at all :-( i can get the juju-core source for 1.14 because there's a series branch but for the dependencies i can't :-( | 08:56 |
| wallyworld_ | so a bit ahrd to recompile 1.14 :-( | 08:56 |
| rogpeppe | wallyworld_: i think all the deps are in https://launchpad.net/juju-core/1.14/1.14.1/+download/juju-core_1.14.1.tar.gz | 09:00 |
| wallyworld_ | rogpeppe: ah ok. we have a src tarball | 09:01 |
| wallyworld_ | still not really best practice :-( | 09:01 |
| rvba | Thanks a lot for your help rogpeppe, we are starting to work on a fix for our bug, we will probably come to you during the day for advice/reviews :). | 09:04 |
| rogpeppe | rvba: np | 09:05 |
| wallyworld_ | rogpeppe: juju 1.14 and go 1.1.2 seems to work. i'm not sure why. juju 1.14 uses the old tools lookup code, but still does similar things eg storage.List() etc | 09:18 |
| rogpeppe | wallyworld_: ok, so i guess we can't blame the upgrade to go 1.1.2 | 09:19 |
| wallyworld_ | i am reticent just to set Close = true without knowing why | 09:19 |
| wallyworld_ | guess not | 09:19 |
| rogpeppe | wallyworld_: i agree | 09:19 |
| rogpeppe | wallyworld_: what request is it getting the EOF on | 09:19 |
| rogpeppe | ? | 09:19 |
| wallyworld_ | a stor List() | 09:20 |
| wallyworld_ | http://10.0.0.9:80/MAAS/api/1.0/files/?op=list&prefix=tools/releases/juju- | 09:20 |
| wallyworld_ | well, with the prefix quoted | 09:20 |
| wallyworld_ | i unquoted it to paste | 09:20 |
| wallyworld_ | the gomaasapi logic creates a new request, creates a new http client and then does a client.Do(req) | 09:21 |
| wallyworld_ | not much to fail | 09:21 |
| wallyworld_ | a curl done manually of the same thing worls | 09:21 |
| rogpeppe | wallyworld_: it might be worth changing the code to log all requests and responses, to see how they've changed between the releases | 09:21 |
| wallyworld_ | yeah | 09:22 |
| wallyworld_ | i just can't see why a simple http req fails | 09:22 |
| wallyworld_ | rogpeppe: here's a pastebin of a manual curl to invoke the req that fails using httpClient.Do(req) | 09:23 |
| wallyworld_ | https://pastebin.canonical.com/99027/ | 09:23 |
| rogpeppe | wallyworld_: well, it'll be reusing a socket from a previous request | 09:23 |
| wallyworld_ | sure | 09:23 |
| rogpeppe | wallyworld_: it is possible that the maas server doesn't like that | 09:23 |
| wallyworld_ | but why now? | 09:24 |
| rogpeppe | wallyworld_: and i don't know if you can make curl do that | 09:24 |
| wallyworld_ | wouldn't sockets always be reused | 09:24 |
| rogpeppe | wallyworld_: yeah, but perhaps we're making more requests now | 09:24 |
| wallyworld_ | we would be | 09:24 |
| wallyworld_ | since tools look up check for metadata etc | 09:24 |
| wallyworld_ | and before it didn't | 09:24 |
| rogpeppe | wallyworld_: exactly - so our access pattern has changed, and perhaps that's triggering some bug/feature on the server | 09:25 |
| wallyworld_ | hard to believe the server could be that fragile | 09:25 |
| wallyworld_ | it's just an apache http server | 09:25 |
| * rogpeppe doesn't find it that hard to believe... | 09:26 | |
| wallyworld_ | rogpeppe: so, worst case, we may have to set Close = true for mass | 09:26 |
| wallyworld_ | just to get something working for release | 09:26 |
| wallyworld_ | juju doesn't really hammer the connection anyway, right? | 09:26 |
| rogpeppe | wallyworld_: i'm not sure we know the worst case now - it may be that setting Close is just papering over a bug which will re-emerge later in some form | 09:27 |
| wallyworld_ | sure, depends if we can find the root cause | 09:27 |
| wallyworld_ | i really don't know where to start. the maas logs show the req isn't even getting through | 09:28 |
| rogpeppe | wallyworld_: that's interesting in itself | 09:29 |
| wallyworld_ | so if it never arrives, it mught be getting lost inside the Go http client lib, or maybe apache is discarding it | 09:29 |
| wallyworld_ | i can ask bigjools to check the apache logs | 09:29 |
| rogpeppe | wallyworld_: that would be good, to start with | 09:30 |
| wallyworld_ | or i can check | 09:30 |
| wallyworld_ | ok, so i can see the legacy tools request | 09:30 |
| wallyworld_ | i'll fire up 1.16 | 09:30 |
| rogpeppe | wallyworld_: how long into the bootstrap do we see the EOF response? | 09:31 |
| wallyworld_ | rogpeppe: very near the start - when it is syncing tools | 09:31 |
| rogpeppe | wallyworld_: i'm still wondering if it might be a stale-connection timeout issue | 09:31 |
| rogpeppe | wallyworld_: how near? (in seconds) | 09:31 |
| wallyworld_ | um. 2? | 09:32 |
| wallyworld_ | 5? | 09:32 |
| wallyworld_ | not sure | 09:32 |
| wallyworld_ | i'll fire up 1.16 and see | 09:32 |
| wallyworld_ | rogpeppe: ffs . it worked that time | 09:34 |
| rogpeppe | wallyworld_: ok, so that's interesting too | 09:34 |
| wallyworld_ | let me check something | 09:34 |
| rogpeppe | wallyworld_: i suppose that means that it's possible that it wasn't the Close=true addition that caused it to succeed last time | 09:35 |
| wallyworld_ | rogpeppe: oh wait, i'm an idiot | 09:36 |
| wallyworld_ | i compiled a juju version with close=true for jools to use | 09:36 |
| wallyworld_ | and it's still in the path | 09:36 |
| wallyworld_ | i'll revert and try again | 09:36 |
| rogpeppe | wallyworld_: ha | 09:36 |
| rogpeppe | wallyworld_: assuming you manage to reproduce the problem, i'd like to check one thing - in gomaasapi/client.go, i'd like to change the "return nil, err" after httpClient.Do to return a more distinctive error, so we can be sure that the EOF is coming from that | 09:41 |
| wallyworld_ | rogpeppe: i've already logged that | 09:42 |
| wallyworld_ | and it is coming from the Do() | 09:42 |
| wallyworld_ | rogpeppe: error happens after about 7 seconds | 09:44 |
| rogpeppe | wallyworld_: have there been several successful requests before the one that failed? | 09:45 |
| wallyworld_ | rogpeppe: log is full. let me clear it and i'll retry. too hard to tell | 09:45 |
| rogpeppe | wallyworld_: i'd be interested to see the log actually | 09:45 |
| wallyworld_ | thought so :-) | 09:45 |
| wallyworld_ | will pastebin | 09:46 |
| rogpeppe | ta | 09:46 |
| wallyworld_ | rogpeppe: actually, i deleted to log and there is no new one. it seems like the first http get fails | 09:46 |
| wallyworld_ | which is what i think the tools look up is | 09:46 |
| wallyworld_ | ie build tools locally, then look to see what's in target bucket, boom | 09:47 |
| rogpeppe | wallyworld_: hmm, that is interesting. i'm surprised that the Close=true change makes a difference then | 09:50 |
| wallyworld_ | yeah | 09:50 |
| wallyworld_ | so the log is the apache log | 09:50 |
| wallyworld_ | so the request never leaves the client | 09:50 |
| wallyworld_ | or apache eats it | 09:50 |
| wallyworld_ | i can't see apache doing that | 09:51 |
| wallyworld_ | no new errors logged | 09:51 |
| rogpeppe | wallyworld_: can you change gomaas API dispatchRequest to log every time it makes a request? | 09:52 |
| wallyworld_ | i can | 09:52 |
| wallyworld_ | i gotta attend to a couple of things. i'll do it between now and standup | 09:53 |
| rogpeppe | wallyworld_: thanks | 09:53 |
| wallyworld_ | rogpeppe: there is one other request | 10:04 |
| wallyworld_ | ------ GET http://10.0.0.9:80/MAAS/api/1.0/files/provider-state/ | 10:04 |
| wallyworld_ | ------ GET http://10.0.0.9:80/MAAS/api/1.0/files/?op=list&prefix=tools%2Freleases%2Fjuju- | 10:04 |
| wallyworld_ | ERROR Get http://10.0.0.9:80/MAAS/api/1.0/files/?op=list&prefix=tools%2Freleases%2Fjuju-: EOF | 10:04 |
| wallyworld_ | the provider-state lookup | 10:05 |
| wallyworld_ | which is not a List() but a storage Get() I think | 10:05 |
| rogpeppe | wallyworld_: and you're saying the first request isn't in the apache log? | 10:06 |
| wallyworld_ | rogpeppe: seems not. i deleted the access.log file from /var/log/apache2 and it hasn't created a new one | 10:07 |
| wallyworld_ | that doesn't seem right | 10:08 |
| rogpeppe | wallyworld_: when you say "deleted", did you just truncate the file, or remove it? perhaps apache is just trying to append but not create? | 10:08 |
| wallyworld_ | yeah, let me touch it and try again | 10:08 |
| wallyworld_ | nope | 10:09 |
| wallyworld_ | empty | 10:09 |
| wallyworld_ | wtf | 10:09 |
| rogpeppe | wallyworld_: maybe logging just isn't configured for that apache? | 10:10 |
| rogpeppe | wallyworld_: or... are you sure it's actually going through that apache? | 10:10 |
| rogpeppe | wallyworld_: if you do a curl request, does it show up in the log? | 10:11 |
| wallyworld_ | trying a few things | 10:11 |
| wallyworld_ | i ran juju 1.14 and lo log either | 10:12 |
| wallyworld_ | perhaps apache didn't like its log file going away | 10:12 |
| wallyworld_ | i'll retstart it | 10:12 |
| rogpeppe | wallyworld_: it's possible, yeah | 10:12 |
| rogpeppe | wallyworld_: perhaps it was still trying to write to the old (removed, but still there in the fs) log file | 10:13 |
| wallyworld_ | yeah | 10:13 |
| wallyworld_ | rogpeppe: i had started a juju env using 1.14. i destroyed using 1.16. it logged about 10 requests and did it no problem | 10:15 |
| wallyworld_ | so it's only bootstrap | 10:15 |
| rogpeppe | wallyworld_: hmm | 10:16 |
| wallyworld_ | rogpeppe: and yes, the provider state lookup is logged | 10:16 |
| wallyworld_ | 10.0.0.9 - - [15/Oct/2013:20:14:58 +1000] "GET /MAAS/api/1.0/files/provider-state/ HTTP/1.1" 404 233 "-" "Go 1.1 package http" | 10:16 |
| wallyworld_ | but NOT the tools list | 10:16 |
| wallyworld_ | which is the next http get | 10:17 |
| rogpeppe | wallyworld_: could you paste the log of the requests it made when destroying the 1.16 env using 1.14? | 10:18 |
| wallyworld_ | o | 10:18 |
| wallyworld_ | k | 10:18 |
| rogpeppe | sorry, i mean the the other way around | 10:18 |
| rogpeppe | wallyworld_: destroying the 1.14 env using 1.16 | 10:18 |
| wallyworld_ | yep | 10:18 |
| wallyworld_ | rogpeppe: https://pastebin.canonical.com/99039/ that's the whole log. it has the destroy followed by bootstrap | 10:19 |
| wallyworld_ | bootstrap at 14:58 | 10:19 |
| wallyworld_ | destroy ends at 14:48 | 10:19 |
| rogpeppe | wallyworld_: hmm, and that contains a successful list request too - exactly the same request that fails in Bootstrap | 10:21 |
| rogpeppe | wallyworld_: well, *almost* | 10:21 |
| wallyworld_ | yep | 10:21 |
| wallyworld_ | go fogure | 10:21 |
| wallyworld_ | figure even | 10:21 |
| rogpeppe | wallyworld_: i wonder if we can try to repro this with a very simple example, rather than using bootstrap | 10:22 |
| rogpeppe | wallyworld_: we know that there are only two requests and the second one fails | 10:23 |
| wallyworld_ | that's pretty simple | 10:23 |
| wallyworld_ | in itself | 10:23 |
| rogpeppe | wallyworld_: so we could simply create a maas Environ and try to read its provider-state file and then list the tools | 10:23 |
| rogpeppe | wallyworld_: with a 7 second gap between them | 10:23 |
| wallyworld_ | yes. but what would that tell us | 10:24 |
| wallyworld_ | that we don't already know | 10:24 |
| rogpeppe | wallyworld_: if that failed, that would be very useful | 10:24 |
| rogpeppe | wallyworld_: because we then have a very simple example that we can reduce further to see what's actually going on | 10:24 |
| rogpeppe | wallyworld_: if that succeeds then we know something else is up | 10:24 |
| wallyworld_ | true | 10:25 |
| rogpeppe | wallyworld_: it should only take 5 minutes to write | 10:25 |
| wallyworld_ | yes | 10:25 |
| wallyworld_ | i'll try and do it after standup if i am still awake, othwrwise tomorrow | 10:26 |
| rogpeppe | wallyworld_: one mo, i'll do it if you like | 10:28 |
| rogpeppe | wallyworld_: try this (substitute "my-maas-environ-name" with your env name: http://paste.ubuntu.com/6239995/ | 10:41 |
| rogpeppe | rvba: i'm a bit confused by the maas storage handling - how are the storages from different juju environments in the same MAAS kept separate from one another? | 10:42 |
| wallyworld_ | rogpeppe: sorry, was afk. looking | 10:44 |
| wallyworld_ | ubuntu@maas:~/go/src/launchpad.net/juju-core$ go run masstest.go | 10:44 |
| wallyworld_ | ------ GET http://10.0.0.9:80/MAAS/api/1.0/files/provider-state/ | 10:44 |
| wallyworld_ | 2013/10/15 20:44:31 get provider-state: file 'provider-state' not found not found | 10:44 |
| wallyworld_ | ------ GET http://10.0.0.9:80/MAAS/api/1.0/files/?op=list&prefix=tools%2Freleases%2Fjuju- | 10:44 |
| wallyworld_ | 2013/10/15 20:44:38 list tools ok | 10:44 |
| TheMue | rogpeppe: standup? | 10:47 |
| rogpeppe | TheMue: "the video call is full" | 10:48 |
| rogpeppe | mgz: standup? | 10:49 |
| mgz | on my way | 10:49 |
| wallyworld_ | fwereade: ?? | 11:00 |
| rogpeppe | wallyworld_: http://paste.ubuntu.com/6240109/ | 11:24 |
| wallyworld_ | rogpeppe: new version ran fine | 11:27 |
| * TheMue => lunch | 11:28 | |
| rogpeppe | wallyworld_: could you try the original version with a 15 second delay? | 11:28 |
| wallyworld_ | ok | 11:28 |
| wallyworld_ | rogpeppe: except bigjools just shot it down | 11:28 |
| wallyworld_ | shut | 11:29 |
| rogpeppe | wallyworld_: oh | 11:29 |
| wallyworld_ | yeah :-( | 11:29 |
| rvba | rogpeppe: care to have a look at https://codereview.appspot.com/14696043/ ? | 11:36 |
| rogpeppe | rvba: looking | 11:36 |
| wallyworld_ | rogpeppe: i got the server powerd up again. i added the 15s delay. all passed | 11:39 |
| rogpeppe | wallyworld_: hmm | 11:39 |
| wallyworld_ | rogpeppe: i can add your ssh key | 11:39 |
| wallyworld_ | rogpeppe: done, you should be able to ssh in now | 11:40 |
| rogpeppe | wallyworld_: great, thanks | 11:41 |
| wallyworld_ | the ~ubuntu/go dir is where the 1.16 source lives | 11:41 |
| wallyworld_ | i put the test go src you gave me in the juju-core root dir | 11:41 |
| rogpeppe | wallyworld_: ok, i'm investigating now | 11:45 |
| wallyworld_ | rogpeppe: awesome. i'll try and hang around a bit. if you are finished and i'm not around, can you sudo poweroff? | 11:45 |
| fwereade | rogpeppe, wallyworld_: thanks for looking into this -- I will try to pop back on and off, and I should be able to be around properly this evening if I can help you at all | 11:49 |
| rogpeppe | fwereade: thanks | 11:49 |
| wallyworld_ | fwereade: np. we've spent a long time on it and at every turn it gets more confusing. we will kick ourselves when the issue is found i'm sure | 11:50 |
| allenap | rogpeppe: When does EnvironProvider.Prepare() get called? I only want to set the environment UUID when bootstrapping an environment; existing environments should not have a UUID. | 12:37 |
| allenap | rogpeppe: Hello too :) | 12:37 |
| rogpeppe | allenap: hi :) | 12:38 |
| rogpeppe | allenap: it gets called only if there's no .jenv file in ~/.juju/environments for that environment | 12:38 |
| rogpeppe | allenap: and only if you call juju bootstrap or juju sync-tools | 12:38 |
| rogpeppe | allenap: so that should be exactly what you need, i hope | 12:38 |
| allenap | rogpeppe: That sounds about right, thanks :) | 12:39 |
| allenap | rvba: Do you happen to know why maasEnvironConfig.attrs exists? environs.config.Config has two maps, one for known and one for unknown config. Does that not suffice, or was that not around at the time? | 13:05 |
| rogpeppe | allenap: perhaps it's to save the map being copied every time an attr is used | 13:06 |
| allenap | rogpeppe: I'm a little rusty :) Can you explain why that would happen? | 13:07 |
| rogpeppe | allenap: environ.Config makes a copy of the map that it returns, to prevent mutation of the immutable Config value. | 13:08 |
| rvba | allenap: no idea… by the looks of it, it just copies the pattern used in the openstack provider… maybe jtv (if he's around) will knwo. | 13:08 |
| rvba | know* | 13:08 |
| allenap | rogpeppe: Ah yes, I see, ta. | 13:08 |
| rogpeppe | wallyworld_: FWIW, these are the (only) requests i see. the final request is the one that fails (it fails because it sees EOF on the persistent connection) http://paste.ubuntu.com/6240523/ | 13:25 |
| wallyworld_ | rogpeppe: that looks about right. the simplestreams code uses a different http client. maybe that is related. the http client used by juju-core code has the ssl support. the http client in gomaas doesn't | 13:30 |
| wallyworld_ | i'm not sure whar is done with the http transport in juju-core though | 13:31 |
| rogpeppe | wallyworld_: they're both using the same transport | 13:31 |
| wallyworld_ | rogpeppe: so at bootstrap is when the different hhtp clients are used, whereas destoy just uses the gomaasapi one | 13:32 |
| wallyworld_ | surely not a coincidence? | 13:32 |
| wallyworld_ | rogpeppe: although, did you run bootstrap with --upload-tools? | 13:33 |
| rogpeppe | wallyworld_: no | 13:33 |
| wallyworld_ | cause using --upload-tools doesn't first do a simplestreams search | 13:34 |
| wallyworld_ | and that still fails | 13:34 |
| rogpeppe | wallyworld_: ah, i'll try that | 13:34 |
| rogpeppe | wallyworld_: http.Client isn't stateful unless it's given a custom RoundTripper | 13:37 |
| rogpeppe | wallyworld_: so it shouldn't make any difference if plain http.Get is used vs creating a new client with http.Client{} | 13:38 |
| wallyworld_ | i'm just clutching at straws, trying to think of twhat's different | 13:38 |
| wallyworld_ | sinzui: hi, did you see the streams repository is sortof set up? | 13:39 |
| sinzui | wallyworld_, I did not | 13:40 |
| wallyworld_ | sinzui: ticket 63925 | 13:41 |
| wallyworld_ | it talks about the server sawo which i'm not familiar with | 13:41 |
| wallyworld_ | sinzui: i need to change the url embedded in juju-core, and the generated simplestreams metadata and tools need to be uploaded | 13:42 |
| sinzui | I saw that and update a bug over my weekend | 13:42 |
| wallyworld_ | or the release scripts tweaked accordingly | 13:42 |
| wallyworld_ | i can easily do the juju-core side ahead of time | 13:43 |
| wallyworld_ | although if the tools were uploaded i could drop the legacy aws fallback at the same time | 13:43 |
| wallyworld_ | rogpeppe: i fell asleep on the couch before so i had better get myself off to bed for real. maybe you can drop me a quick note with any progress and i'll pick up tomorrow? also don't forget to poweroff the server | 13:53 |
| rogpeppe | wallyworld_: how should i do that? | 13:53 |
| rogpeppe | wallyworld_: just sudo shutdown? | 13:53 |
| wallyworld_ | sudo shutdown now | 13:53 |
| wallyworld_ | or does sudo poweroff work? not sure | 13:53 |
| rogpeppe | wallyworld_: i'm making progress BTW | 13:53 |
| wallyworld_ | oh? | 13:53 |
| rogpeppe | wallyworld_: i have a suspicion of what might be happening | 13:53 |
| wallyworld_ | oh great, nowyou've got me curious | 13:54 |
| rogpeppe | wallyworld_: it looks like MAAS *is* closing the connection after 5s | 13:54 |
| rogpeppe | wallyworld_: but somehow we're seeing that in the wrong place; not sure quite why yet | 13:54 |
| wallyworld_ | hmmm. ok. would be great to know the sequencing of things so explain why now and not before | 13:54 |
| rogpeppe | wallyworld_: it *may* be to do with someone not closing an http request body. i need more investigation | 13:55 |
| wallyworld_ | if maas is going to do that, maybe we do need the close=true on the juju side? | 13:55 |
| rogpeppe | wallyworld_: the http client *should* work ok even with that | 13:56 |
| wallyworld_ | ok. good luck and thanks. talk tomorrow | 13:56 |
| rogpeppe | wallyworld_: but i guess there's always going to be a race. | 13:56 |
| rogpeppe | wallyworld_: okeydokey. sweet dreams :-) | 13:56 |
| wallyworld_ | i'll ask jools tomorrow about the maas aspect of it | 13:56 |
| jamespage | fwereade, sinzui: any critical bugs for 1.16.0 that I need to push in pre-release of saucy? | 14:25 |
| * sinzui looks | 14:25 | |
| sinzui | jamespage, I don't see any criticals that can be pushed | 14:26 |
| jamespage | sinzui, coolio | 14:29 |
| * jamespage puts his feet up for a bit then | 14:29 | |
| mgz | jamespage: I have one funny report on a bug fixed for 1.16 | 14:31 |
| mgz | that I want to (un)confirm | 14:31 |
| * jamespage takes his feet off the desk and listens | 14:31 | |
| mgz | jamespage: see last two comments on bug 1236734 | 14:31 |
| _mup_ | Bug #1236734: juju 1.15.1 polls maas API continually <juju-core:Fix Released by gz> <https://launchpad.net/bugs/1236734> | 14:31 |
| mgz | I'm pretty sure he just has the old version still | 14:31 |
| jamespage | mgz, oh - I can confirm that has good | 14:32 |
| jamespage | gone | 14:32 |
| mgz | excellent. | 14:32 |
| jamespage | the load on the serverstack maas server went from 2.0 to 0.04 post upgrade | 14:32 |
| rogpeppe | jamespage: excellent news - i was a bit concerned my fix hadn't | 14:32 |
| mgz | I'll comment of the bug and see if I can help the guy. | 14:32 |
| jamespage | which is worrying - as it would indicate that the performance of the storage on MAAS is not great - I only have 6 physical and 8 lxc service units in the deployment | 14:33 |
| jamespage | mgz, maybe he forgot todo juju upgrade-juju | 14:33 |
| jamespage | fwiw that took a very long time to complete | 14:33 |
| jamespage | <jamespage> which is worrying - as it would indicate that the performance of the storage on MAAS is not great - I only have 6 physical and 8 lxc service units in the deployment | 14:33 |
| mgz | hm, storage performace, or just general api being really slow? | 14:37 |
| mgz | django is a fair bit of overhead on top of postgres | 14:38 |
| * sinzui thinks there is a space in dependencies.tsv that breaks goamx | 14:41 | |
| mgz | gah, and it's me... | 14:41 |
| rogpeppe | sinzui: ah, i was wondering what the problem there might be | 14:41 |
| mgz | sinzui: I'll fix | 14:42 |
| mgz | should have thought of that when it was falling over yesterday rog... | 14:42 |
| mgz | my bad. | 14:42 |
| rogpeppe | mgz: i should've thought of it too | 14:43 |
| rogpeppe | sinzui: nice catch | 14:43 |
| mgz | fix proposed for rubber stamping. now, did I backport that too... | 14:46 |
| natefinch | howdy all. Sorry to miss the standup this morning. | 14:46 |
| mgz | natefinch: blame columbus! | 14:46 |
| natefinch | mgz: that bastard | 14:46 |
| mattyw | fwereade, you about? | 14:48 |
| mgz | ah, actually, it wasn't my one that borked, so is just trunk | 14:48 |
| mgz | blame say nate :) | 14:48 |
| mgz | natefinch: https://codereview.appspot.com/14701043 | 14:49 |
| TheMue | natefinch: heya | 14:50 |
| TheMue | rogpeppe: could you help me with the provisioner? | 14:51 |
| natefinch | mgz: dang, sorry | 14:51 |
| rogpeppe | TheMue: sure; gimme a minute | 14:51 |
| TheMue | rogpeppe: or even a level higher, I don't know if it is the provisioner | 14:51 |
| TheMue | rogpeppe: thx | 14:51 |
| natefinch | mgz: who uses tab separated values, anyway? Why not csv like normal people? :) | 14:52 |
| * natefinch grumbles about invisible, variable width characters | 14:52 | |
| rogpeppe | TheMue: what's the issue? | 14:53 |
| rogpeppe | natefinch: because commas are more common in normal values | 14:54 |
| TheMue | rogpeppe: my question is that in case of a dying machine (e.g. by shutdown -fh now) is the a mechanism that updates the state on it? | 14:54 |
| rogpeppe | TheMue: i'm not sure | 14:54 |
| rogpeppe | /me goes to look | 14:55 |
| * rogpeppe goes to look | 14:55 | |
| rogpeppe | TheMue: i can't see anything that polls instances for their status, no | 14:59 |
| TheMue | rogpeppe: yes, that's my impression too | 14:59 |
| TheMue | rogpeppe: thx | 14:59 |
| rogpeppe | TheMue: np | 15:00 |
| TheMue | rogpeppe: cts reports of a customer which turn down machines (wonder why they do it instead of removing units) and then in status they see the machine as down but the unit as alive | 15:01 |
| TheMue | rogpeppe: that makes me wonder | 15:01 |
| rogpeppe | TheMue: i have a feeling that the address updater worker might be a good place to put this functionality (i guess it might be renamed to "machineupdater" if it that happened) | 15:02 |
| rogpeppe | mgz: does that make sense to you? | 15:02 |
| mgz | seems reasonable, though I'm not sure it's the right fix for the problem of machines just going away | 15:03 |
| TheMue | mgz, rogpeppe: I'm trying to get more details from CTS | 15:04 |
| rogpeppe | mgz: yeah, that would probably need another fix too, in the provisioner. although... | 15:04 |
| rogpeppe | mgz: we'd need to decide what we want to do about that | 15:04 |
| rogpeppe | mgz: do we want to automatically start another unit? | 15:05 |
| mgz | that was the pyjuju behaviour, roughly | 15:05 |
| mgz | not sure if it's what we want though | 15:05 |
| rogpeppe | mgz: indeed | 15:05 |
| rogpeppe | mgz: we probably do if minunits is set | 15:05 |
| TheMue | mgz: it get's more clear now, they use juju to deploy openstack on maas and there the hacluster charm. then they stop nodes for failover tests | 15:18 |
| fwereade | TheMue, rogpeppe, mgz: we dropped that behaviour, the only time anyone noticed it was when they didn't want it | 15:24 |
| fwereade | TheMue, it is surprising that the unit is not reported as down if it's not running | 15:25 |
| rogpeppe | fwereade: yeah, i thought so, although with SetMinUnits, we should perhaps rethink that | 15:25 |
| rogpeppe | fwereade: because that expresses a clear intent, i think | 15:26 |
| fwereade | TheMue, look at the api server, maybe it's not noticing that the unit's connection is gone and is hence not killing the presence bit | 15:27 |
| rogpeppe | fwereade: ah of course, i'd forgotten about the presence thing | 15:27 |
| fwereade | rogpeppe, wrt units uncautious agreement; wrt machines uncautious disagreement ;p | 15:27 |
| fwereade | er machines *cautious* disagreement | 15:27 |
| rogpeppe | fwereade: agreed | 15:28 |
| fwereade | rogpeppe, essentially I need to write some access-revocation stuff for agents we force destroy | 15:28 |
| sinzui | How permanent is the theme-oil bug tag? Is there a better tag to represent the bugs? server?, hyperscale? cloud-server? | 15:29 |
| fwereade | rogpeppe, if I have that I can be sure that removed units aren't coming back and are safe to replace without compounding confusion | 15:29 |
| rogpeppe | fwereade: i guess so, although i suppose it's not a huge problem | 15:29 |
| rogpeppe | fwereade: ah yes, of course | 15:29 |
| fwereade | rogpeppe, I am still twitchy about presence nodes as an arbiter of true existence | 15:29 |
| rogpeppe | fwereade: although... | 15:29 |
| rogpeppe | fwereade: we won't be recreating the same units, will we? | 15:29 |
| rogpeppe | fwereade: or the same machines to run them, come to that | 15:30 |
| fwereade | rogpeppe, I've seen a couple of `down (alive)` reports that have always cleared themselves up before I've been able to figure them out | 15:30 |
| fwereade | rogpeppe, so basically I'm just nervous about our ability to tell for sure what's in the system | 15:30 |
| rogpeppe | fwereade: yeah | 15:31 |
| fwereade | rogpeppe, and I don't want things coming back to life once we think they're dead | 15:31 |
| rogpeppe | fwereade: if we've destroyed their state entities, how can they? | 15:31 |
| rogpeppe | fwereade: or perhaps that's what you're thinking of when you say "access revocation" | 15:32 |
| fwereade | rogpeppe, I'm just worried about the potential for races, with things coming back and managing to do something while we think they're gone | 15:32 |
| fwereade | rogpeppe, access revocation == telling things they're dead immediately, even if they're not quite dead in state yet | 15:33 |
| rogpeppe | fwereade: we can't just kill 'em dead in state immediately? | 15:34 |
| * rogpeppe is still going down the maas EOF bug rabbit hole | 15:35 | |
| fwereade | rogpeppe, not sure we can while keeping sane guarantees | 15:35 |
| fwereade | rogpeppe, (tyvm for keeping on at that) | 15:36 |
| fwereade | rogpeppe, think of the relations a subordinate has joined, when that subordinate is in a container in a machine that itself has a unit running directly | 15:36 |
| fwereade | rogpeppe, even calculating the right transaction for smoothly cleaning up the top-level machine kinda makes me cry | 15:37 |
| rogpeppe | fwereade: there's something really weird happening, which is causing the remote http connection to be dropped *just* before we make a GET request | 15:37 |
| fwereade | rogpeppe, I would like a quick "cut all these off" to be followed by a more measured cleanup of all the various agents potentially in play | 15:37 |
| rogpeppe | fwereade: yeah, seems reasonable on second thoughts | 15:38 |
| natefinch | fwereade: do you know how far Tim got in looking at MaaS? I was looking into a couple of the bugs, but was having trouble with my maas environment (as usual). If he's progressing, it might be better for me to work on something else | 15:45 |
| kurt_ | Hi all - I asked about bug 1236734 yesterday and didn't get a response. This is listed as fixed in 1.16.0, but it's not. | 15:45 |
| _mup_ | Bug #1236734: juju 1.15.1 polls maas API continually <juju-core:Fix Released by gz> <https://launchpad.net/bugs/1236734> | 15:45 |
| kurt_ | Any chance we could get some attention on this? | 15:45 |
| fwereade | natefinch, I saw rvba and bigjools talking with rogpeppe this morning | 15:46 |
| rogpeppe | natefinch: which MaaS problem? | 15:46 |
| fwereade | kurt_, mgz was going to follow up with that -- last I heard it was confirmed fixed by jamespage | 15:47 |
| rvba | fwereade: we talked about something different, we are in the process of fixing bug 1239488. | 15:47 |
| _mup_ | Bug #1239488: Juju api client cannot distinguish between environments <MAAS:Triaged> <https://launchpad.net/bugs/1239488> | 15:47 |
| natefinch | rogpeppe: the couple I was looking at was destroying non-allocated machines and destroying machines from outside the juju environment | 15:47 |
| fwereade | kurt_, are all your agents running 1.16? | 15:47 |
| kurt_ | fwereade: its not. I have 1.16 installed and a still having issue | 15:47 |
| fwereade | rvba, I *think* that's what natefinch was thinking of | 15:47 |
| rvba | natefinch: that's the bug I just mentioned. | 15:47 |
| rogpeppe | natefinch: ah, yeah, there's some stuff being done for that | 15:47 |
| natefinch | rogpeppe, rvba, fwereade: that sounds like a great fix for the bug | 15:48 |
| kurt_ | fwereade: how can I bring up the agent inof? | 15:48 |
| kurt_ | info rather | 15:48 |
| fwereade | kurt_, juju status | 15:48 |
| fwereade | kurt_, included agent-version | 15:48 |
| natefinch | sorry, brb, diaper duty | 15:48 |
| rogpeppe | natefinch: there's a CL here: https://codereview.appspot.com/14696043/ | 15:48 |
| fwereade | kurt_, you may need to sync-tools before an upgrade-juju sees the latest tools to fix the problem | 15:49 |
| kurt_ | Oh - good point - how can I update the agent? just sync-tools? | 15:49 |
| jamespage | fwereade, mramm2: any call outs you would like to add for juju-core in https://wiki.ubuntu.com/SaucySalamander/ReleaseNotes#Ubuntu_Server ? | 15:49 |
| fwereade | kurt_, sync-tools should get them in place for you; upgrade-juju will actually upgrade the agents | 15:50 |
| fwereade | kurt_, we didn't quite get the global tools source in place in time for it to be 1-step on maas :( | 15:50 |
| kurt_ | oh - so even if I followed the correct upgrade process, I still would have ran in to this? | 15:50 |
| kurt_ | sorry, I know this is the dev list and not real appropriate for this discussion | 15:51 |
| fwereade | kurt_, no worries :) and... I guess it depends on definitions, because upgrade-juju can't upgrade further than it knows; but it's not necessarily entirely obvious that you need to sync tools before a maas environment will see them | 15:53 |
| kurt_ | interesting, right. I'm trying to blog some stuff, so I'd like to capture that | 15:54 |
| fwereade | kurt_, the bug report is appreciated all the same -- it helps make it clear to us that we still need to smooth the process | 15:54 |
| fwereade | kurt_, *hopefully* you just hit a bad window, because we've got some infrastructure just coming online now to make it more transparent (unless you're actually cut off from the internet entirely -- if you're isolated, syncing will always be required) | 15:55 |
| kurt_ | I'm not isolate | 15:55 |
| kurt_ | isolated - full connectivity to internet on MaaS | 15:55 |
| kurt_ | and juju | 15:55 |
| fwereade | kurt_, but, indeed, I don't think you'll see the benefits until 1.18 -- sorry for the infelicity | 15:55 |
| mgz | kurt_: I responded on the bug today after checking with others, you might not have seen the comment unless you subscribed to the bug | 15:56 |
| kurt_ | so when upgrading juju - can you outline what the fool proof process should be? | 15:56 |
| kurt_ | (as to avoid this situation in the future) :D | 15:56 |
| kurt_ | mgz: thanks | 15:57 |
| kurt_ | I didn't see that yet | 15:57 |
| fwereade | kurt_, for most users: `juju upgrade-juju`; for those using maas, or those who have manually synced tools in the past, you'll need a `juju sync-tools` first | 15:57 |
| kurt_ | fwereade: I believe that I did the juju upgrade via apt-get upgrade after adding the ppa | 15:58 |
| fwereade | kurt_, we will hopefully be able to cut out that step for those not in isolated environments, but it's not there today | 15:58 |
| fwereade | kurt_, ha, sorry, I was focused on the agent side of things: yes, you should also apt-get upgrade your client; and you should probably prefer to upgrade the client first, because that way round has been exercised more, but either way round should be fine in practice | 16:00 |
| kurt_ | so the upgrade of juju can be thought of at 2 levels, right? 1. the juju binary 2. the agents - is that a reasonable assumption? | 16:01 |
| kurt_ | oh there are the tools too | 16:01 |
| kurt_ | ! | 16:01 |
| fwereade | kurt_, yeah, both need to be done -- but 1.14/1.16 binaries and agents should interoperate happily | 16:02 |
| fwereade | kurt_, the tools are the agent binaries | 16:02 |
| kurt_ | i was previously on .15 :D | 16:02 |
| fwereade | kurt_, just a quirk of terminology | 16:02 |
| kurt_ | 1.15.1 that is | 16:02 |
| kurt_ | I C | 16:02 |
| fwereade | kurt_, then that *should* actually work too, but we don't make guarantees for odd-numbered minor versions | 16:02 |
| fwereade | kurt_, we have no intention of breaking upgrades ofc... but 1.15 *did* have upgrade troubles | 16:03 |
| kurt_ | 1.15.1 had some fixes I needed previously - but yeah I get that - but that's why I was asking about best way to fip bits between dev and stable stuff. | 16:03 |
| fwereade | kurt_, think of 1.odd versions as canaries to make sure we don't mess up the 1.even path | 16:03 |
| kurt_ | ok | 16:04 |
| * rogpeppe gets some lunch | 16:12 | |
| TheMue | fwereade: hmm, thx for the hint, found the machinePinger in apiserver/admin.go, but funnily it's used nowhere | 16:18 |
| natefinch | rogpeppe, fwereade: ec2 instance type constraints.... would love to have some input: https://codereview.appspot.com/14523052/ | 16:22 |
| mgz | natefinch: I was leaving that one for next week | 16:22 |
| mgz | if you read through the mail archives, you'll find various arguments, and just having a value that's ignored depending on provider and generally has uncertain meaning isn't really an improvement on not supporting it at all | 16:24 |
| allenap | fwereade: Hi William, please can you add me to ~juju? Or please mark https://code.launchpad.net/~allenap/juju-core/maas-environment-uuid/+merge/191146 as Approved :) | 16:40 |
| mgz | allenap: did you do the rename rogpeppe mentioned? | 16:47 |
| allenap | mgz: In a subsequent branch I changed it to maas-environment-uuid then back again when I read his comment. | 16:48 |
| mgz | so, we want another branch generating it in Prepare for all providers? | 16:49 |
| allenap | mgz: I guess so. I have added the code to return an error if it's set in environments.yaml, but it's still in the MAAS provider for now. | 16:50 |
| allenap | mgz: I have a very specific problem to solve :) | 16:50 |
| rogpeppe | wow, this is incredibly weird | 16:51 |
| mgz | allenap: submitted | 16:51 |
| allenap | mgz: Ta. | 16:51 |
| sinzui | fwereade, hazmat, thumper, You will be getting email about this bug https://bugs.launchpad.net/juju-core/+bug/1232304 | 16:53 |
| _mup_ | Bug #1232304: consider tuning git setup for juju-core, and document caveats <canonical-webops> <doc> <feature> <pes> <juju-core:Triaged> <https://launchpad.net/bugs/1232304> | 16:53 |
| * TheMue is stepping out for dinner, cu | 16:57 | |
| mthaddon | sinzui: hi, can you give me any context on that bug (1232304)? it's a real problem for us at the moment - means some environments can't run upgrade-charm without significant effort | 16:58 |
| === natefinch is now known as natefinch-afk | ||
| allenap | mgz: If you're itching to do a review, https://codereview.appspot.com/14644045 is scratching at the door waiting to be let into your heart. | 17:05 |
| * rogpeppe has gone up a blind alley and needs to stop and make curry. | 17:09 | |
| rogpeppe | g'night all | 17:09 |
| sinzui | mthaddon, I have asked for input from the engineers. | 17:09 |
| allenap | nn rogpeppe | 17:10 |
| rogpeppe | allenap: cheerio | 17:10 |
| mgz | allenap: may have a chance to go through that briefly | 17:21 |
| === natefinch-afk is now known as natefinch | ||
| natefinch | morning thumper. How goes? | 20:09 |
| thumper | natefinch: morning, good | 20:09 |
| thumper | can tell that today is going to get a little broken up | 20:09 |
| thumper | taking the car for a service in just under an hour | 20:10 |
| thumper | and will work from a cafe across the road while that is happening | 20:10 |
| natefinch | Wish I could do that, my mechanic is sorta in the middle of nowhere | 20:10 |
| * natefinch doesn't really want to choose mechanics based on amenities within walking distance, however.... | 20:11 | |
| thumper | :) | 20:13 |
| wallyworld_ | fwereade: you around? i was hoping for a hand off email about the maas http connection issue. do you know the status? | 20:33 |
| fwereade | wallyworld_, back soon -- rogpeppe was thinking for a while that it *was* an unclosed request body, but then I saw him saying blind alley :( | 20:39 |
| wallyworld_ | ok, thanks | 20:39 |
| thumper | is the go bot set up to land on 1.16? | 21:14 |
| wallyworld_ | thumper: yes | 21:14 |
| thumper | wallyworld_: so why isn't it | 21:15 |
| thumper | ? | 21:15 |
| wallyworld_ | um. | 21:15 |
| wallyworld_ | it did for me last week, no problems | 21:15 |
| thumper | https://code.launchpad.net/~thumper/juju-core/bootstrap-state-no-constraints/+merge/190852 | 21:15 |
| wallyworld_ | i'll log in and check | 21:15 |
| thumper | approved, commit message | 21:15 |
| thumper | not getting merged | 21:15 |
| wallyworld_ | thumper: https://pastebin.canonical.com/99089/ | 21:16 |
| thumper | wtf? | 21:17 |
| wallyworld_ | thumper: wtf is a ghost revision? | 21:17 |
| thumper | a revision with referenced as a parent that isn't in the repository | 21:17 |
| thumper | most likely, a stacking issue | 21:18 |
| thumper | but launchpad can see it | 21:18 |
| thumper | so NFI | 21:18 |
| wallyworld_ | maybe do a new branch, cherry pick changes, re-propose? | 21:18 |
| thumper | ugh | 21:22 |
| thumper | wallyworld_: I know what it is | 21:43 |
| wallyworld_ | do tell | 21:43 |
| thumper | wallyworld_: no I don't | 21:43 |
| * thumper thought he did | 21:43 | |
| wallyworld_ | nothing to see here, move on | 21:43 |
| thumper | yes I do | 21:43 |
| thumper | maybe | 21:43 |
| thumper | argh | 21:43 |
| thumper | it does have to do with stacking | 21:43 |
| thumper | what I did was start with a 1.16 branch | 21:43 |
| thumper | and then moved back 3 revisions to be what was currently merged with trunk | 21:44 |
| thumper | that was the base revision for that branch | 21:44 |
| thumper | when pushed to launchpad, it gets stacked on trunk | 21:44 |
| thumper | trunk should have those revisions from 1.16 | 21:44 |
| thumper | so now I'm confused again | 21:44 |
| thumper | bzr is doing something weird | 21:44 |
| * thumper wonders if he can unstack it | 21:45 | |
| * thumper pokes | 21:45 | |
| wallyworld_ | nothing seems to be going right this week | 21:46 |
| thumper | :) | 21:47 |
| thumper | looking forward to next week? | 21:47 |
| thumper | because that'll be awesome | 21:47 |
| thumper | :-) | 21:47 |
| thumper | well, that branch says it is reconfiguring as unstacked, we'll see if it actually works | 21:47 |
| thumper | bzr reconfigure --unstacked lp:~thumper/juju-core/bootstrap-state-no-constraints | 21:48 |
| thumper | in case you cared | 21:48 |
| wallyworld_ | cool, i have not done that before | 21:49 |
| wallyworld_ | yes, next week wil be great | 21:49 |
| thumper | wallyworld_: do you need any reviews? | 21:53 |
| wallyworld_ | thumper: i have a couple actually | 21:54 |
| wallyworld_ | there's a pipeline of 3 | 21:54 |
| wallyworld_ | andrew has done the last | 21:54 |
| wallyworld_ | https://codereview.appspot.com/14663043/ and https://codereview.appspot.com/14540055/ are the others | 21:55 |
| thumper | paste the links here and I can go through them | 21:55 |
| wallyworld_ | thanks :-) | 21:55 |
| wallyworld_ | the last is https://codereview.appspot.com/14502059/ fwiw | 21:55 |
| wallyworld_ | i've been looking at the http maas issue so haven't yet really looked at the review i do have, will do that today | 21:56 |
| * thumper nods | 22:03 | |
| * thumper starts on the first one | 22:04 | |
| thumper | looks big | 22:04 |
| thumper | btw, unstacking that branch worked | 22:05 |
| thumper | should get merge in now | 22:05 |
| thumper | hopefully | 22:05 |
| fwereade | wallyworld_, do we have docs for sync-tools --source somewhere | 22:12 |
| fwereade | ? | 22:12 |
| fwereade | thumper, hi, long time | 22:12 |
| wallyworld_ | fwereade: not sure. i haven't really read any of our docs | 22:12 |
| thumper | fwereade: hey | 22:12 |
| thumper | just noticed the provisioner tests timed out on go-bot for the 1.16 branch | 22:12 |
| thumper | anyone seen this before? | 22:13 |
| wallyworld_ | fwereade: do we have any docs for sync-tools at all? | 22:13 |
| wallyworld_ | thumper: i get random timeouts from the bot semi-regularly | 22:13 |
| fwereade | wallyworld_, hell, possibly not :/ | 22:13 |
| bigjools | wallyworld_: how'd you get on with the http problem? | 22:35 |
| wallyworld_ | bigjools: hi, long story, lots of dead ends. but it looks like maas is closing the http connection and then juju complains about it | 22:36 |
| bigjools | wallyworld_: and juju just needs to deal with that I guess | 22:36 |
| wallyworld_ | still need to figure out root cause | 22:37 |
| bigjools | it might be getting closed because it's not using http/1.1? | 22:37 |
| bigjools | or the keep-alive header is not getting set? | 22:37 |
| * bigjools stabs in the dark | 22:37 | |
| wallyworld_ | not sure, will check. but none of that has supposedly changed | 22:37 |
| wallyworld_ | bigjools: so all of the http requests to maas go via the apache server? | 22:38 |
| bigjools | did you work out if Go 1.1.1 made a difference? | 22:38 |
| wallyworld_ | bigjools: no difference | 22:38 |
| bigjools | wallyworld_: yes, maas runs as a wsgi container in Apache | 22:38 |
| wallyworld_ | compiled 1.14 with go 1.1.2 and it worked | 22:38 |
| bigjools | ! | 22:38 |
| bigjools | so something changed inbetween 1.14 and 1.16.... fun | 22:38 |
| wallyworld_ | nothing obvious though | 22:38 |
| wallyworld_ | and the EOF at the client is really due to the server closing the connection | 22:39 |
| wallyworld_ | but what triggers the closure | 22:39 |
| wallyworld_ | that is the question | 22:39 |
| bigjools | you could do something silly like compile 1.16 with the version of gomaasapi from 1.14 | 22:39 |
| bigjools | just to eliminate that | 22:39 |
| bigjools | or confirm it | 22:39 |
| wallyworld_ | could do. a very quick look at gomaasapi showed no significant changes, or nothing obvious in the http request dispatch side of thnugs | 22:40 |
| wallyworld_ | i think though the 1.14 gomaasapi won't worj with 1.16? will need to checki guess | 22:40 |
| wallyworld_ | bigjools: you using the maas server this morning? | 22:42 |
| bigjools | wallyworld_: no | 22:42 |
| bigjools | I am weeping into my coffee | 22:43 |
| wallyworld_ | i'll poke around some more then if that's ok | 22:43 |
| bigjools | because of this multi-environ crap | 22:43 |
| wallyworld_ | weeping? | 22:43 |
| bigjools | np | 22:43 |
| wallyworld_ | ah, the users getting mixed up issue? | 22:43 |
| bigjools | yeah | 22:45 |
| bigjools | I maintain it's a juju self-inflicted problem but thumper disagrees with me | 22:45 |
| wallyworld_ | what would he know, right? | 22:45 |
| thumper | not a lot | 22:46 |
| bigjools | at least we can disagree in a more violent manner in person next week :) | 22:46 |
| wallyworld_ | i'll remember to pack the popcorn | 22:46 |
| wallyworld_ | or maybe jelly | 22:46 |
| * bigjools will secretly put one fewer shots in thumper's coffee | 22:46 | |
| * thumper packs the boxing gloves | 22:47 | |
| bigjools | heh | 22:47 |
| bigjools | you still 120kg thumper? | 22:47 |
| wallyworld_ | of pure muscle | 22:47 |
| rvba | Dibs on the front row seats! | 22:47 |
| thumper | just under 97kg now | 22:48 |
| bigjools | gosh | 22:48 |
| thumper | never wes 120 | 22:48 |
| thumper | 111 maybe | 22:48 |
| bigjools | :) | 22:48 |
| wallyworld_ | wasting away to nothing | 22:48 |
| bigjools | so rvba has come up with the great idea of using juju's uuids in the filenames we send to maas | 22:50 |
| bigjools | rvba: of course we have to make sure the tools don't get those uuids in the names I think | 22:51 |
| wallyworld_ | yep, that would be bad | 22:51 |
| rvba | bigjools: well, we just need to handle that transparently in the provider's code. | 22:51 |
| wallyworld_ | not as simple as that | 22:52 |
| wallyworld_ | you mean send the tools with the uuids? | 22:52 |
| wallyworld_ | and translate inthe provider? | 22:52 |
| wallyworld_ | or strip out the uuid and send the tools uncganged? | 22:52 |
| wallyworld_ | unchanged | 22:52 |
| rvba | juju itself will ask to a file with name="zzz" but the provider will just fetch the file named uuid+"zzz" | 22:53 |
| rvba | Same when storing files. | 22:53 |
| wallyworld_ | hmm. but then we will have multiple copies of the same tools tarballs | 22:53 |
| rvba | The only trick is the anonymously accessible files… but that will also work transparently I think, because in this case we simply use generated ids. | 22:53 |
| wallyworld_ | canit be path based? ie don't use uuids for certain paths? | 22:54 |
| wallyworld_ | so stuff under tools/releases is left alone | 22:54 |
| rvba | wallyworld_: well, yes, that could be an improvement. | 22:54 |
| wallyworld_ | and tools/streams | 22:55 |
| wallyworld_ | or just /tools in general | 22:55 |
| wallyworld_ | that would save duplicating the same tools tarballs and metadata over and over again for each user | 22:55 |
| bigjools | so leave tools/ alone but add uuid/ to everything else | 22:55 |
| bigjools | WFM | 22:55 |
| wallyworld_ | yeah, think that will work | 22:55 |
| rvba | Yeah, totally possible. But again, I think this should be done as a second step. | 22:55 |
| wallyworld_ | why 2nd? i think it is flawed to needlessly duplicate lots of data | 22:56 |
| bigjools | rvba: so I don't want to change the filestorage object willy nilly because of this | 22:56 |
| bigjools | it needs to be done at the provider level | 22:56 |
| rvba | Absolutely. | 22:57 |
| rvba | There is actually very little to be done. | 22:57 |
| rvba | But that's the beauty of it. It's just a fix to the provider. | 22:57 |
| rvba | All in provider/maas/storage.go | 22:57 |
| rvba | Basically, we just need to translate filename <--> uuid+filename everywhere. | 22:58 |
| wallyworld_ | and also the output from list | 22:58 |
| rvba | Yep | 22:58 |
| rvba | storage.List(stor, "") | 22:58 |
| rvba | Becomes storage.List(stor, uuid) | 22:58 |
| wallyworld_ | i will be trivial to add the /tools exclusion | 22:58 |
| rvba | bigjools: this is really too good to be true! | 22:58 |
| bigjools | rvba: I disagree that it can only go there because it will affect the tools | 22:59 |
| bigjools | I think it needs to go in some of its callsites | 22:59 |
| wallyworld_ | tools are access via storage | 22:59 |
| rvba | Exactly. | 22:59 |
| wallyworld_ | so it should work, so long as tools are excluded from uuid mangling | 22:59 |
| bigjools | via that storage object no? | 22:59 |
| wallyworld_ | yes | 22:59 |
| bigjools | special casing tools in there is crazy | 22:59 |
| wallyworld_ | agreed, i don't care where it's done | 23:00 |
| wallyworld_ | but we should duplicate tools | 23:00 |
| wallyworld_ | shouldn't | 23:00 |
| bigjools | I'll sort the call sites out later | 23:00 |
| bigjools | right now I need breakfast and an inbox cleanout | 23:00 |
| rvba | Again, I suggest dealing with the tools second. | 23:01 |
| bigjools | or is that an outbox cleanout | 23:01 |
| rvba | Because we need to implement this and test it. | 23:01 |
| wallyworld_ | but not in a flawed way | 23:01 |
| rvba | bigjools: so, I'll let the lab running so that you can test this | 23:01 |
| wallyworld_ | tools duplication not good | 23:01 |
| bigjools | rvba: I can test it locally | 23:01 |
| rvba | not good indeed. But not fatal. | 23:01 |
| bigjools | my maas setup is working fine here | 23:01 |
| wallyworld_ | too much scope for hidden gotchas | 23:01 |
| rvba | bigjools: all right. | 23:02 |
| bigjools | I'd rather do it properly from the start, it's no big deal. | 23:02 |
| wallyworld_ | bigjools: except i'm playing with it | 23:02 |
| bigjools | rvba: thanks | 23:02 |
| bigjools | wallyworld_: easily fixed :) | 23:02 |
| wallyworld_ | bastard | 23:02 |
| bigjools | mwahahaha | 23:02 |
| bigjools | you have a while anyway | 23:02 |
| rvba | bigjools: don't forget to base your work on Gavin's branch: ~allenap/juju-core/maas-environment-uuid-use | 23:02 |
| bigjools | doubt I'll get to the QA stage until this afternoon | 23:03 |
| bigjools | rvba: affirmative, thanks | 23:03 |
| bigjools | is he finishe dcoding? | 23:03 |
| bigjools | is he finished coding? | 23:03 |
| rvba | Yeah, it's up for review. | 23:03 |
| * bigjools looks at wallyworld_ | 23:03 | |
| wallyworld_ | yeeees? | 23:04 |
| rvba | bigjools: btw, thumper reviewed it. | 23:04 |
| bigjools | \o/ | 23:04 |
| rvba | He has an interesting point about backward compatibility. It should be fine (I think extra arguments like agent_name will be ignored by earlier version of MAAS) but it's worth a test. | 23:05 |
| bigjools | rvba: it will be fine and a desirable outcome. It leaves the functionality as-is | 23:06 |
| bigjools | I'll try to land it later | 23:07 |
| rvba | All I'm saying is that it's worth a test. | 23:07 |
| bigjools | absolutely | 23:07 |
| rvba | And on that note, I think I'll call it a night. | 23:08 |
| rvba | I'll be available to do some QA tomorrow morning if we still have time. | 23:08 |
| wallyworld_ | davecheney: ping | 23:25 |
| davecheney | wallyworld_: ack | 23:31 |
| wallyworld_ | i have tweaked a go file from the std lib (net/transport.go). how do i force that to compile so that a go build -a ./... from juju-core uses my mods? | 23:31 |
| wallyworld_ | i tried go build in the /usr/lib/go/pkg/net/http dir | 23:31 |
| davecheney | wallyworld_: you should, 1 remove the packaged go | 23:32 |
| davecheney | 2. download go from source | 23:32 |
| davecheney | the packaged go cannot be modified | 23:32 |
| davecheney | it's too mutated | 23:32 |
| wallyworld_ | davecheney: this is on jool's maas server which roger played with last night and he made changes | 23:32 |
| davecheney | http://golang.org/doc/install/source | 23:32 |
| wallyworld_ | i'm trying to revert them | 23:32 |
| davecheney | if they are to the packaged version | 23:33 |
| davecheney | remove the package | 23:33 |
| davecheney | make suer /var/lib/go is empty | 23:33 |
| davecheney | reinstlal the package ? | 23:33 |
| davecheney | would that work ? | 23:33 |
| wallyworld_ | i think so. i assume go 1.1.2 is in the repos? | 23:33 |
| wallyworld_ | although if roger made his changes stick, i shoild be able to as well? | 23:33 |
| wallyworld_ | i wonder what command he ran? | 23:34 |
| davecheney | wallyworld_: i don't think I really understand the problem | 23:34 |
| davecheney | or the solutoin you are persuing | 23:34 |
| wallyworld_ | ok, so roger modified transport.go | 23:34 |
| wallyworld_ | to add extra logging | 23:34 |
| davecheney | wallyworld_: you coudl try, as root | 23:34 |
| davecheney | cd /usr/lib/go/pkg/net/http | 23:34 |
| davecheney | go install -x | 23:35 |
| wallyworld_ | and he did so by editing /usr/lib/go/pkg/net/http/transport.go directly | 23:35 |
| davecheney | i don't know if that will work | 23:35 |
| davecheney | never tried it | 23:35 |
| wallyworld_ | will try | 23:35 |
| wallyworld_ | davecheney: that looks like it worked. the timestamp on the .a file has now chaged | 23:35 |
| wallyworld_ | thanks :-) | 23:36 |
| davecheney | when screwing with this | 23:36 |
| davecheney | i recommend -x and -v flags | 23:36 |
| davecheney | so you can see when things are chaning | 23:36 |
| davecheney | or more important, when they are not 'cos the tool thinks everything is up to date | 23:36 |
| wallyworld_ | ok. i've not used -x before | 23:36 |
| wallyworld_ | yeah. i thought go build -a would have done the job | 23:37 |
| wallyworld_ | but it seems that's not for the std libs, only deps of current project | 23:37 |
| davecheney | -a has problems | 23:39 |
| wallyworld_ | don't we all | 23:39 |
| davecheney | https://groups.google.com/d/msgid/golang-nuts/9120d622-e8a8-451f-941e-34899ae0a457%40googlegroups.com | 23:39 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!