thumperperhaps I should stop sparring the week before a gathering00:26
thumperI wonder if this facial bruising will be gone by Sunday00:26
thumperI should learn not to block punches with my face00:27
wallyworld_bigjools: is it ok for me to use your maas? are you still getting that eof issue?00:50
bigjoolswallyworld_: I am, but hang on I am just completing a test00:55
bigjools10 mins00:55
thumperwallyworld_: have you used the garage maas before?01:12
wallyworld_i've not used *ant* maas01:12
wallyworld_how's the face?01:12
bigjoolswallyworld_: server is all yours01:15
wallyworld_bigjools: ta. how do i get an oauth key for the env.yaml?01:16
bigjoolswallyworld_: it's all set up01:16
bigjoolsjust log in and bootstrap01:16
wallyworld_bigjools: i'm scared to look inside the folder called "backdoor-image". shudder01:18
bigjoolswallyworld_: left just for you01:19
axwthumper: do you have/use any bzr plugins to get diff summaries? (file names with +/- lines)01:46
thumperaxw: bzr diff | diffstat01:46
thumperthat's all I use01:47
axwah, didn't know diffstat01:47
thumperdoesn't give full summary per file01:47
thumperalthough it may have options01:47
axwthumper: that'll do nicely for me, thanks01:47
thumperwallyworld_: face is a little bruised, that's all01:50
wallyworld_walked into a door01:50
bigjoolsthumper: nothing to do with the missus?01:51
thumpersparring at boxing, yes01:52
thumperthe guy today is very fast01:52
thumpertop 5 in the country and under 2001:52
thumperso much younger and faster01:52
thumperbut you don't learn unless you fight those better than you01:52
bigjoolsyour nick gets better and better01:54
bigjoolsor possibly more ironic01:54
thumperI have an interview with Rachel, Caitlin and her new principal later this afternoon01:57
thumperslight shiner to go in with01:57
thumperI should just ack all meek01:58
thumperand flinch when Rachel looks at me :)01:58
thumperschool pickup time01:58
* thumper walks a block01:58
thumperI want to test out my maas change to make sure that the allocated change is right02:09
thumperhence the email about the garage maas02:09
wallyworld_bigjools: this maas eof thing is giving me the shits. it's failing doing a bog standard httpClient.Do(request) inside the gomaasapi client.go, complaining that "can't write HTTP request on broken connection"02:57
wallyworld_so something is closing the maas client's connection02:57
thumperI know what it is02:57
wallyworld_but i don't know what02:57
thumperpick me02:57
thumperpick me02:58
thumperpick me02:58
thumperpick me02:58
thumperwallyworld_: quick hangout?02:58
bigjoolswallyworld_: I blame Go03:07
wallyworld_bigjools: it's all fooked03:07
wallyworld_i just can't see what's wrong03:07
bigjoolswallyworld_: try emulating with a simple curl request03:08
* bigjools looks in maas log as well03:08
wallyworld_bigjools: i think i did that and still got an error. oh, i did it using wget03:08
bigjoolswallyworld_: what time did you try last?03:08
bigjoolsyou need auth header remember03:08
wallyworld_bigjools: looks like remote end is closing the connection03:09
thumperand as it happens, I don't know03:09
thumperdifferent problem03:09
wallyworld_bigjools: ah yes. can you remind me of the syntax03:09
bigjoolswallyworld_: I can't :)03:09
wallyworld_fat lot of good you are then03:10
bigjoolswallyworld_: what request is it making when it gets the EOF?03:13
bigjoolsFWIW I get the EOF sometimes when not using --upload-tools03:13
wallyworld_or there abouts03:14
wallyworld_i had commented out the prefix to try something03:14
thumperbigjools: we blame maas03:14
bigjools10.0.0.9 - - [15/Oct/2013:11:15:17 +1000] "GET /MAAS/api/1.0/files/?op=list&prefix= HTTP/1.1" 200 807 "-" "Go 1.1 package http"03:15
bigjoolsmaas is doing just fine it seems --^03:15
wallyworld_bigjools: why does this command return an error then: response, err := httpClient.Do(request)03:15
wallyworld_it's effectively a straight http get call03:15
wallyworld_which fails03:15
bigjoolsit did not do this until recently and maas has not changed around this call03:16
wallyworld_the code has not changed03:16
bigjoolscan you dump the request and response?  I'll add the same to the maas log03:16
wallyworld_it's a simple list request03:16
wallyworld_bigjools: there is no response cause the request call errors03:17
wallyworld_&http.Request{Method:"GET", URL:(*url.URL)(0xc200230850), Proto:"HTTP/1.1", ProtoMajor:1, ProtoMinor:1, Header:http.Header{"Authorization":[]string{"OAuth oauth_signature_method=\"PLAINTEXT\", oauth_version=\"1.0\", realm=\"MAAS+API\", oauth_consumer_key=\"M2X2ZeCSNVWer6AEHc\", oauth_token=\"v3jKFjma2gZkhasdQR\", oauth_signature=\"%26yVgdrUAVWKCRsxNXGUEzyTrTaHYebAmH\", oauth_timestamp=\"1381806551\", oauth_nonce=\"38010820\""}},03:17
wallyworld_Body:io.ReadCloser(nil), ContentLength:0, TransferEncoding:[]string(nil), Close:false, Host:"", Form:url.Values(nil), PostForm:url.Values(nil), MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:"", RequestURI:"", TLS:(*tls.ConnectionState)(nil)}03:17
wallyworld_is the request03:17
bigjoolsI'll try again and dump the maas log, hang on03:18
bigjoolswallyworld_: ok that request isn't even hitting maas03:19
bigjoolsGo is getting the EOF alllll on its own03:19
davecheneywallyworld_: can you paste a curl -v of that url ?>03:21
bigjoolswallyworld_: the last request that maas sees is /MAAS/api/1.0/files/provider-state/03:21
wallyworld_davecheney: i'll have to look up the syntax - i can't recall how to construct a curl request with the headers etc03:21
bigjoolscurl -H "Authorization: OAuth <ACCESS_TOKEN>" http://www.example.com03:23
wallyworld_bigjools: so i pasted in the entire oauth access token from the env.yaml file as the ACCESS_TOKEN, right?03:28
bigjoolssomething like:03:29
bigjoolsAuthorization: OAuth oauth_signature_method="PLAINTEXT", oauth_version="1.0", realm="MAAS+API", oauth_consumer_key="M2X2ZeCSNVWer6AEHc", oauth_token="v3jKFjma2gZkhasdQR", oauth_signature="%26yVgdrUAVWKCRsxNXGUEzyTrTaHYebAmH", oauth_timestamp="1381807262", oauth_nonce="281480703:29
bigjoolsit mght get a bit tricky actually03:30
bigjoolsyou can always re-create the request with maas-cli03:31
bigjoolswhich works fine, BTW03:31
wallyworld_does the maas-cli log the request?03:31
bigjools"maas-cli maas files list prefix=tools-"03:31
bigjoolsyeah I can see it in the mas log03:32
wallyworld_maas-cli maas files list prefix=releases/tools/juju- also works03:32
wallyworld_so why is Go's http call failing03:33
bigjoolsthere are no polite answers to that03:33
bigjoolsas a data point I can re-create this another way03:33
wallyworld_bigjools: do you have the output from mass-cli that you can give to davecheney who wanted to see a curl -v output?03:34
bigjoolsif I bootstrap without --upload-tools, destroy and then bootstrap without --upload-tools again, you get the EOF03:34
bigjoolsthat's the host side03:35
bigjoolsbut amounts to the same thing03:35
wallyworld_davecheney: is that ^^^^ sufficient? it sure seems like a http get request via other means works, but the same request sent from Go's http.Client fails03:36
davecheneywallyworld_: nah, i was hoping to see the curl output03:37
davecheneyi know how to interpret that03:37
wallyworld_ok, i'll see if i can get it03:38
bigjoolsgiven that the request from Go isn't hitting maas, I suspect you need to look at why the Go internals are misbehaving03:38
bigjoolsno amount of curl log is going to help with that03:38
wallyworld_davecheney: that's the pastebin https://pastebin.canonical.com/99027/03:44
wallyworld_that worked03:44
wallyworld_the Go code doesn't03:44
wallyworld_the values in the pastebin were harvested by printing the http.Request struct from Go03:44
wallyworld_so the request is formed with the right header values etc03:45
wallyworld_but it doesn't reach the server side, and instead the Do() method returns an EOF03:45
wallyworld_or a "can't write HTTP request on broken connection" error03:45
wallyworld_bigjools: how recently did you upgrade to Go 1.1.2? could that be an issue?03:49
bigjoolswallyworld_: how can that affect anything?  I am using a packaged juju binary on this box03:50
wallyworld_i only run Go 1.1.1 on mine. if it's a Go internal issue, the version may be relevant03:51
davecheneybigjools: you could have 1.0.x03:51
davecheneythat is known to be broke03:51
davecheneythat was all the difficulty we had bad in July03:51
bigjoolsI still don't know how that can affect a binary built by someone else03:51
wallyworld_davecheney: go version says 1.1.2 on his maas server03:51
davecheneyalso, does maas use your forked http code ?03:52
bigjoolsmy maas server Go version is irrelevant03:52
wallyworld_bigjools: i compiled juju from source on your box03:53
wallyworld_to add in debugging03:53
bigjoolssure, but given this is also a problem with the packaged juju ...03:53
davecheneybigjools: ooooooooooh dear03:53
davecheneyi wonder if the host that build juju was using 1.1.203:53
bigjoolsif it's on saucy, quite likely03:54
davecheneythat is known to work03:54
wallyworld_except for this http request :-(03:54
bigjoolsis it re-using objects or otherwise similarly being stupid?03:55
wallyworld_a new http request is created each time, as is a new http.Client object03:55
bigjoolsoh google you chunk of crap03:55
bigjoolssilently rendering the word EOF as just "of" in my seach is stupid03:56
bigjools"You need to set Req.Close to true"03:57
wallyworld_just saw that03:57
bigjoolsis it doing that?03:57
wallyworld_nope, this is maas code03:57
bigjoolsis gomaasapi doing that?03:57
wallyworld_i don't think so, but i wonder if we do it *anywhere*03:58
* bigjools knows not of golang esoterics03:58
wallyworld_a quick code search seems to indicate we don't03:58
wallyworld_only in the forked gwacl code03:59
wallyworld_so why the fuck is this not failing elsewhere if it is a problem03:59
davecheneymaas is the only provider that uses OAUTH ?04:00
davecheney^ guess04:00
bigjoolsI think so04:01
bigjoolsbut why is it failing like this now as well?04:01
bigjoolswallyworld_: this is easy to test out04:02
wallyworld_yeah, doing it now04:02
bigjoolsok - I was also but noticed the file changed under my feet :)04:03
wallyworld_bigjools: davecheney: adding in req.Close = true after creating the req seems to have worked. but wtf. we don't do that anywhere else in juju-core04:08
wallyworld_and this is in the gomaasapi library04:08
wallyworld_so why just maas and not goose or goaws wtc04:08
* bigjools 's work here is done04:09
bigjoolsthanks for helping out wallyworld_04:09
wallyworld_np. but i'm pissed at it all. not sure where to direct my rage04:09
bigjoolsRob Pike? :)04:09
wallyworld_i blame Go04:09
wallyworld_i mean if req.Close is required, why not just default to that?04:10
wallyworld_and why is it not consistent?04:10
wallyworld_do we just change maasapi?04:10
wallyworld_or do we try and change *all* the other places we create requests04:10
bigjoolssomething to do with HTTP1.1 I guess04:11
bigjoolsthe document is singularly unhelpful04:11
bigjools        // Close indicates whether to close the connection after04:11
bigjools        // replying to this request.04:11
bigjoolsno hint of why you'd want to do that04:12
wallyworld_yeah, and the error seems like the opposite of that04:12
wallyworld_bigjools: how long should juju status take to come back on your maas setup?04:12
bigjoolsabout 10 minutes04:12
wallyworld_me taps fingers on desk waiting, waiting04:13
bigjoolsmy theory is that maas already closed the connection and it didn't detect that until trying to send on the same one again04:13
bigjoolsthink yourself lucky, it used to be 20 minutes, the fast installer cut the time in half04:13
wallyworld_wonder why it didn't fail before now though04:14
bigjoolsand 8 of the 10 minutes is cloud-init and a reboot04:14
wallyworld_did maas change?04:14
bigjoolsI reckon davecheney may be right when he said it could be go 1.1.204:14
bigjoolsmaas hasn't changed here - it's using the exact same stuff04:15
bigjoolsApache frontend04:15
wallyworld_and 1.1.4 shipped on Go 1.1.1?04:15
wallyworld_1.14 i mean04:15
bigjoolsno idea04:15
bigjoolswhen did 1.1.2 hit the archive?04:15
wallyworld_no idea here either04:15
bigjoolscould build 1.1.6 on go 1.1.1 and see04:16
bigjools1.16 even04:16
wallyworld_go on then, i dare you04:16
wallyworld_and juju stat came back ok04:17
wallyworld_i'll revert my debugging04:17
bigjoolsso stackoverflow, I can't upvote an solution because I don't have enough reputation,  but I *can* edit the answer to oblivion.  *HEADDESK*04:18
wallyworld_funny, i searched for and found that page also, but my google didn;t "fix" the EOF spelling04:18
bigjoolsI used +"EOF"04:19
wallyworld_i didn't04:19
bigjoolsGo seems to be a minefield where there's an actual mine wherever you step04:20
wallyworld_yep :-(04:20
wallyworld_AND no version control :0(04:20
wallyworld_what could possibly go wrong04:21
wallyworld_bigjools: you should have a working binary in your ~ubuntu/go/bin directory04:21
wallyworld_update PATH quick!04:22
bigjoolsno need I can just run that binary04:22
wallyworld_there's also a bootstrapped env now04:22
wallyworld_not if you want to upload tools04:22
wallyworld_i think it needs to be in your path, or is that for local provider04:22
bigjoolsone last bitchslap04:22
bigjoolsdefault gopath is that env anyway04:23
wallyworld_i can't recall, it may only be for local provider04:23
wallyworld_ok, see how it goes04:23
bigjoolshello jtv05:17
jtvHi bigjools05:17
bigjoolsyou're not on the internal irc05:17
bigjoolsdavecheney: you guys planning another release to go out with 13.10?05:29
davecheneybigjools: nope05:30
davecheney1.1.2 has been in saucy since september05:30
bigjoolsdavecheney: juju release I mean05:31
davecheneybigjools: i think so05:31
davecheneyi don't know the detalis05:31
davecheneythumper and sinzui probably know05:31
axwwallyworld_: if you have some time this week, would you mind having a look over https://codereview.appspot.com/14527043/05:37
axwthere's some changes to simplestreams metadata merging in there05:37
wallyworld_axw: sure, looking now. was out buying dinner after school pickup06:30
axwwallyworld_: thanks06:32
wallyworld_axw: just had a quick read of some of the comments. i agree in theory the resolve can go away eventually. but my view is we need to now while we transition and have to cope with older metadata etc06:33
wallyworld_lenient on what's read, strict on what's written and all that06:33
axwwallyworld_: yep, sounds fair enough06:34
wallyworld_i reckon we can get rid of it for 1.1806:34
wallyworld_so maybe if this is going into trunk we don't need it06:35
wallyworld_or the 1.16 backport has it, land in trunk, and then follow up with a trunk branch to remove it06:35
wallyworld_davecheney: from your recollection, was juju 1.14 done with go 1.1.1?06:36
axwwallyworld_: yeah this isn't going into 1.1606:36
wallyworld_hmmm. maybe we can/should get rid of it then06:37
wallyworld_reduce complexity06:37
wallyworld_all metadata should be good for 1.1606:37
wallyworld_and if it's not, we don't want to propagate the issue06:37
wallyworld_bigjools: i think there's going to be a 1.18 for saucy06:39
axw1.18? not 1.16.1?06:40
axwdo we not do stable updates like that?06:40
wallyworld_axw: i *thought* it was going to be 1.18, based off 1.17 trunk06:40
wallyworld_too many bugs etc we are fixing in trunk06:40
wallyworld_i could be wrong06:41
wallyworld_but that's what i heard06:41
axwokey dokey06:41
wallyworld_ot thought i heard06:41
axwwallyworld_: in that case, I'd rather hold off on changing the resolve logic for this CL06:41
axwoh but... we don't support going multiple versions do we06:42
wallyworld_axw: with WriteMetadata stuff, i have done something similar now for images metadata. but unlike for tools, i pulled the WriteXXX methods out into generate.go instead if stuffing into simplestreams.go06:42
wallyworld_i think this is the link, i have 3 of the fuckers https://codereview.appspot.com/14663043/06:43
wallyworld_generate.go is a new file06:43
wallyworld_axw: we support going from 1.14->1,16 and 1.16->1.1806:44
wallyworld_but not 1.14->1.1806:44
axwright, so doesn't matter either way then06:44
wallyworld_axw: it's extra work for you, but if the resolve bit were to be removed prior to committing to trunk, the code comes out cleaner06:47
wallyworld_i looked at the merge proposal, it look ok06:47
axwwallyworld_: yeah that's cool, it'll definitely clean it up - I'll do that06:48
axwthanks for checking over it06:48
axwI'm reviewing your CL now, looking good so far06:48
wallyworld_np, thanks for fixing it :-)06:48
wallyworld_axw: thanks :-) if you are a masocist, i have 3 related ones. the one you are looking at is the last of 306:48
axware they prereqs?06:48
wallyworld_https://codereview.appspot.com/14502059/ is first, and https://codereview.appspot.com/14540055/ is second06:49
axwnot a masochist, but I'll do what I can ;)06:49
wallyworld_the end result is that creating image metadata for private clouds is *much* easier06:49
wallyworld_thanks :-)06:49
wallyworld_with this work, the user can run the image metadata gtenerate plugin over and over to buiild up their metadata06:50
wallyworld_for different series, arches etc06:50
wallyworld_before, the tool was just a prototype and just did one image and overwrote each tome06:50
bigjoolswallyworld_: ok ta06:51
wallyworld_rogpeppe: can you recall - did we release juju 1.14 using Go 1.1.1?06:52
rogpeppewallyworld_: hmm, not sure06:55
wallyworld_rogpeppe: ok. there's a bug i'm fixing in gomaasapi which i want to blame on the go 1.1.2 upgrade06:56
wallyworld_cause nothing else makes sense06:56
rogpeppewallyworld_: is that the EOF problem?06:56
wallyworld_reg from http.NewRequest needs to have req.Close = true all of a sudden06:57
wallyworld_but only in gomaasapi06:57
wallyworld_nowhere else06:57
rogpeppewallyworld_: FWIW if Close didn't default to false, you'd almost never get any connection reuse06:57
wallyworld_and no gomaasapi code has changed06:57
rogpeppewallyworld_: i wonder how long connections are kept around before they're dumped06:57
wallyworld_not sure06:58
rogpeppewallyworld_: it might be the maas server dumping old http connections06:58
wallyworld_but this happened during bootstrap, so < 1 second06:58
rogpeppewallyworld_: ah, that seems wrong06:58
wallyworld_why did it on;y show up now?06:58
wallyworld_and not before06:58
rogpeppewallyworld_: good question; does it happen reliably?06:58
wallyworld_only in 1.16 juju-core06:59
wallyworld_not 1.14 on same maas box06:59
rogpeppewallyworld_: have you tried 1.14 compiled with go 1.1.2 ?06:59
bigjoolsrogpeppe: the front end is Apache, so whatever Apache is doing...07:00
rogpeppebigjools: hmm, seems unlikely then07:00
wallyworld_rogpeppe: no07:00
rogpeppewallyworld_: that would be a good way to confirm or deny your suspicions07:00
wallyworld_rogpeppe: so will this have adverse performance impace for gomaasapi wrt connection resuse?07:01
wallyworld_if Close is always set to True07:01
wallyworld_which it needs to be to make it work07:01
rogpeppewallyworld_: possibly07:01
rogpeppewallyworld_: these are https connections, right?07:02
wallyworld_i think so07:02
wallyworld_ah no07:02
bigjoolsI'd take bad performance over no performance :)07:07
rogpeppebigjools: it would be nice to know what's going on though07:08
bigjoolscould be a Go difference from 1.1.1 to 1.1.207:08
rogpeppebigjools: it is possible07:09
rogpeppebigjools: that should be easy enough to confirm07:09
* rogpeppe is reading through the Go http.Transport logic07:14
bigjoolshave you guys done the environment uuid yet?07:30
bigjoolsrvba: I reckon that is still the best solution BTW, as you said the other one is seriouslu abusing tags07:31
rvbaIf the environment uuid is not available yet, would it make sense to use a hash of the hash of the admin-secret as an identifier?07:43
rogpeppervba, bigjools: what's the issue here?07:47
rvbarogpeppe: we want to fix https://bugs.launchpad.net/maas/+bug/123948807:48
_mup_Bug #1239488: Juju api client cannot distinguish between environments <MAAS:Triaged> <https://launchpad.net/bugs/1239488>07:48
rvbarogpeppe: so we need a way to flag nodes to account for the fact that they belong to a juju environment.07:48
rvbarogpeppe: we thought about two solutions… and we think solution 2 is best:07:49
rvba= 3. Fix by storing the API key used to acquire nodes =07:49
rvbaCons: harder to pull a key if compromised07:50
rvbarogpeppe: arg, wrong paste, sorry.07:50
rvbarogpeppe: http://pad.ubuntu.com/DnNONX6kFB07:50
rogpeppervba: i don't quite see how the UUID is an *alternative* to using tags07:50
rogpeppervba: won't you need to tag with the UUID?07:50
rvbarogpeppe: no, we need the UUID in both cases.07:50
* rogpeppe goes to get his SSO key07:51
rogpeppervba: so are you thinking of solution 2 here?07:54
rvbarogpeppe: yeah07:54
rogpeppervba: i've always wanted the env UUID to be generated when the environment is first created07:55
rogpeppervba: it should actually be quite a simple change now07:56
rogpeppervba: it can be done at Prepare time07:56
rogpeppervba: and would be passed through in the environ config07:56
rvbarogpeppe: that would be great.07:57
rvbarogpeppe: we need to fix this problem today/tomorrow morning… is that something (the UUID thingy) that could be done… like… now? ;)07:58
rogpeppervba: probably not that quickly - it involves changes in a few places. however...07:59
rogpeppervba: we could make a change in the maas provider only07:59
rogpeppervba: to make it generate its own uuid, to be replaced with the environment uuid at some point in the future08:00
rvbarogpeppe: changing the uuid will imply dealing with already deployed environments.08:01
rogpeppervba: there's actually no particular need for it to be the *actual* environment UUID, is there?08:01
rvbarogpeppe: no, we just need an identifier, specific to each environment.08:02
rogpeppervba: ok, so here's a possible way forward:08:02
wallyworld_bigjools: so we don't overlap, were you going to compile juju 1.14 with go 1.1.2 on your maas box?08:02
rogpeppervba: change maas's EnvironProvider.Prepare so that it generates its own UUID and stores it in the environ's configuration08:02
rogpeppervba: then change StartInstance and Bootstrap to tag the instance with that tag08:03
rvba(plus change Instances() to pass that tag to MAAS when listing instances)08:04
rogpeppervba: yeah08:04
rogpeppervba: oh yeah, it's agent_name, not tag, also :-)08:04
rvbaRight, that's a detail :)08:04
rvbarogpeppe: isn't EnvironProvider.Prepare called every time juju is run?08:05
rogpeppervba: nop08:05
rogpeppervba: it's called just once for a given environment08:05
rvbaI mean, I'm not sure I see how the uuid would be created once and persisted.08:05
rogpeppervba: after Prepare is called, all the config attributes for that environment are stored in the .jenv file (as BootstrapAttrs)08:05
rvbarogpeppe: I see… so if you manually get rid of the .jenv you'll be in trouble then.08:06
rvbarogpeppe: what about using hash(admin-secret) ?08:06
rogpeppervba: i don't like that idea08:07
rogpeppervba: there's nothing guaranteeing that admin-secret is unique08:07
rogpeppervba: you'll be in trouble if you manually get rid of the .jenv file anyway08:08
rvbarogpeppe: all right :).  Then this seems like the best option.08:08
rvbabigjools: I updated the plan with rogpeppe's idea.08:11
rogpeppervba: for an example of a Prepare method that adds an attribute, take a look at openstack's environProvider.Prepare method08:12
rvbarogpeppe: okay, thanks.08:13
rogpeppervba: although that method allows the control-bucket to be overridden in environments.yaml, and i'm not sure you'd want to allow that for the uuid08:13
rvbaProbably not.08:14
rogpeppervba: you'd probably want to make the method return an error if the uuid is already specified08:14
bigjoolswallyworld_: no, go for it08:18
bigjoolsrvba: ok.  Existing deployments will be a problem IMO08:20
rvbabigjools: true.  That's the only remaining problem.08:21
bigjoolsrvba: and it's a hard one08:21
bigjoolsrvba: although we could get it to work if juju only generates its uuid on bootstrapping08:21
rvbabigjools: one things we could do is detect that we don't have generated a UUID and that the env is already bootstraped, and in this case use '' as the agent name,08:21
bigjoolsand existing deployments can stay without the uuid08:21
rogpeppervba: if you've upgraded a legacy environment, the uuid in the config will be unset08:34
rogpeppervba: so that case should be easy to cater for08:35
rogpeppervba: although it's perhaps a problem that upgrading to the new juju won't fix the current problem for existing envs08:35
rvbarogpeppe: I don't really think we have a choice here.08:36
rogpeppervba: is the plan to make the agent_name dynamically changeable, or will it only be specifiable when an instance is started?08:36
rvbarogpeppe: I don't see why we should make it changeable.08:37
rogpeppervba: if it was, it might be possible to fix existing deployments08:37
rogpeppeaxw_: you've got a review  https://codereview.appspot.com/14430064/08:54
wallyworld_rogpeppe: sadly, juju-core does not appear to have been tagged for the 1.14 release. neither has goose. and other dependencies like gomaasapi have no tags at all :-( i can get the juju-core source for 1.14 because there's a series branch but for the dependencies i can't :-(08:56
wallyworld_so a bit ahrd to recompile 1.14 :-(08:56
rogpeppewallyworld_: i think all the deps are in https://launchpad.net/juju-core/1.14/1.14.1/+download/juju-core_1.14.1.tar.gz09:00
wallyworld_rogpeppe: ah ok. we have a src tarball09:01
wallyworld_still not really best practice :-(09:01
rvbaThanks a lot for your help rogpeppe, we are starting to work on a fix for our bug, we will probably come to you during the day for advice/reviews :).09:04
rogpeppervba: np09:05
wallyworld_rogpeppe: juju 1.14 and go 1.1.2 seems to work. i'm not sure why. juju 1.14 uses the old tools lookup code, but still does similar things eg storage.List() etc09:18
rogpeppewallyworld_: ok, so i guess we can't blame the upgrade to go 1.1.209:19
wallyworld_i am reticent just to set Close = true without knowing why09:19
wallyworld_guess not09:19
rogpeppewallyworld_: i agree09:19
rogpeppewallyworld_: what request is it getting the EOF on09:19
wallyworld_a stor List()09:20
wallyworld_well, with the prefix quoted09:20
wallyworld_i unquoted it to paste09:20
wallyworld_the gomaasapi logic creates a new request, creates a new http client and then does a client.Do(req)09:21
wallyworld_not much to fail09:21
wallyworld_a curl done manually of the same thing worls09:21
rogpeppewallyworld_: it might be worth changing the code to log all requests and responses, to see how they've changed between the releases09:21
wallyworld_i just can't see why a simple http req fails09:22
wallyworld_rogpeppe: here's a pastebin of a manual curl to invoke the req that fails using httpClient.Do(req)09:23
rogpeppewallyworld_: well, it'll be reusing a socket from a previous request09:23
rogpeppewallyworld_: it is possible that the maas server doesn't like that09:23
wallyworld_but why now?09:24
rogpeppewallyworld_: and i don't know if you can make curl do that09:24
wallyworld_wouldn't sockets always be reused09:24
rogpeppewallyworld_: yeah, but perhaps we're making more requests now09:24
wallyworld_we would be09:24
wallyworld_since tools look up check for metadata etc09:24
wallyworld_and before it didn't09:24
rogpeppewallyworld_: exactly - so our access pattern has changed, and perhaps that's triggering some bug/feature on the server09:25
wallyworld_hard to believe the server could be that fragile09:25
wallyworld_it's just an apache http server09:25
* rogpeppe doesn't find it that hard to believe...09:26
wallyworld_rogpeppe: so, worst case, we may have to set Close = true for mass09:26
wallyworld_just to get something working for release09:26
wallyworld_juju doesn't really hammer the connection anyway, right?09:26
rogpeppewallyworld_: i'm not sure we know the worst case now - it may be that setting Close is just papering over a bug which will re-emerge later in some form09:27
wallyworld_sure, depends if we can find the root cause09:27
wallyworld_i really don't know where to start. the maas logs show the req isn't even getting through09:28
rogpeppewallyworld_: that's interesting in itself09:29
wallyworld_so if it never arrives, it mught be getting lost inside the Go http client lib, or maybe apache is discarding it09:29
wallyworld_i can ask bigjools to check the apache logs09:29
rogpeppewallyworld_: that would be good, to start with09:30
wallyworld_or i can check09:30
wallyworld_ok, so i can see the legacy tools request09:30
wallyworld_i'll fire up 1.1609:30
rogpeppewallyworld_: how long into the bootstrap do we see the EOF response?09:31
wallyworld_rogpeppe: very near the start - when it is syncing tools09:31
rogpeppewallyworld_: i'm still wondering if it might be a stale-connection timeout issue09:31
rogpeppewallyworld_: how near? (in seconds)09:31
wallyworld_um. 2?09:32
wallyworld_not sure09:32
wallyworld_i'll fire up 1.16 and see09:32
wallyworld_rogpeppe: ffs . it worked that time09:34
rogpeppewallyworld_: ok, so that's interesting too09:34
wallyworld_let me check something09:34
rogpeppewallyworld_: i suppose that means that it's possible that it wasn't the Close=true addition that caused it to succeed last time09:35
wallyworld_rogpeppe: oh wait, i'm an idiot09:36
wallyworld_i compiled a juju version with close=true for jools to use09:36
wallyworld_and it's still in the path09:36
wallyworld_i'll revert and try again09:36
rogpeppewallyworld_: ha09:36
rogpeppewallyworld_: assuming you manage to reproduce the problem, i'd like to check one thing - in gomaasapi/client.go, i'd like to change the "return nil, err" after httpClient.Do to return a more distinctive error, so we can be sure that the EOF is coming from that09:41
wallyworld_rogpeppe: i've already logged that09:42
wallyworld_and it is coming from the Do()09:42
wallyworld_rogpeppe: error happens after about 7 seconds09:44
rogpeppewallyworld_: have there been several successful requests before the one that failed?09:45
wallyworld_rogpeppe: log is full. let me clear it and i'll retry. too hard to tell09:45
rogpeppewallyworld_: i'd be interested to see the log actually09:45
wallyworld_thought so :-)09:45
wallyworld_will pastebin09:46
wallyworld_rogpeppe: actually, i deleted to log and there is no new one. it seems like the first http get fails09:46
wallyworld_which is what i think the tools look up is09:46
wallyworld_ie build tools locally, then look to see what's in target bucket, boom09:47
rogpeppewallyworld_: hmm, that is interesting. i'm surprised that the Close=true change makes a difference then09:50
wallyworld_so the log is the apache log09:50
wallyworld_so the request never leaves the client09:50
wallyworld_or apache eats it09:50
wallyworld_i can't see apache doing that09:51
wallyworld_no new errors logged09:51
rogpeppewallyworld_: can you change gomaas API dispatchRequest to log every time it makes a request?09:52
wallyworld_i can09:52
wallyworld_i gotta attend to a couple of things. i'll do it between now and standup09:53
rogpeppewallyworld_: thanks09:53
wallyworld_rogpeppe: there is one other request10:04
wallyworld_------ GET
wallyworld_------ GET
wallyworld_ERROR Get EOF10:04
wallyworld_the provider-state lookup10:05
wallyworld_which is not a List() but a storage Get() I think10:05
rogpeppewallyworld_: and you're saying the first request isn't in the apache log?10:06
wallyworld_rogpeppe: seems not. i deleted the access.log file from /var/log/apache2 and it hasn't created a new one10:07
wallyworld_that doesn't seem right10:08
rogpeppewallyworld_: when you say "deleted", did you just truncate the file, or remove it? perhaps apache is just trying to append but not create?10:08
wallyworld_yeah, let me touch it and try again10:08
rogpeppewallyworld_: maybe logging just isn't configured for that apache?10:10
rogpeppewallyworld_: or... are you sure it's actually going through that apache?10:10
rogpeppewallyworld_: if you do a curl request, does it show up in the log?10:11
wallyworld_trying a few things10:11
wallyworld_i ran juju 1.14 and lo log either10:12
wallyworld_perhaps apache didn't like its log file going away10:12
wallyworld_i'll retstart it10:12
rogpeppewallyworld_: it's possible, yeah10:12
rogpeppewallyworld_: perhaps it was still trying to write to the old (removed, but still there in the fs) log file10:13
wallyworld_rogpeppe: i had started a juju env using 1.14. i destroyed using 1.16. it logged about 10 requests and did it no problem10:15
wallyworld_so it's only bootstrap10:15
rogpeppewallyworld_: hmm10:16
wallyworld_rogpeppe:  and yes, the provider state lookup is logged10:16
wallyworld_10.0.0.9 - - [15/Oct/2013:20:14:58 +1000] "GET /MAAS/api/1.0/files/provider-state/ HTTP/1.1" 404 233 "-" "Go 1.1 package http"10:16
wallyworld_but NOT the tools list10:16
wallyworld_which is the next http get10:17
rogpeppewallyworld_: could you paste the log of the requests it made when destroying the 1.16 env using 1.14?10:18
rogpeppesorry, i mean the the other way around10:18
rogpeppewallyworld_: destroying the 1.14 env using 1.1610:18
wallyworld_rogpeppe: https://pastebin.canonical.com/99039/ that's the whole log. it has the destroy followed by bootstrap10:19
wallyworld_bootstrap at 14:5810:19
wallyworld_destroy ends at 14:4810:19
rogpeppewallyworld_: hmm, and that contains a successful list request too - exactly the same request that fails in Bootstrap10:21
rogpeppewallyworld_: well, *almost*10:21
wallyworld_go fogure10:21
wallyworld_figure even10:21
rogpeppewallyworld_: i wonder if we can try to repro this with a very simple example, rather than using bootstrap10:22
rogpeppewallyworld_: we know that there are only two requests and the second one fails10:23
wallyworld_that's pretty simple10:23
wallyworld_in itself10:23
rogpeppewallyworld_: so we could simply create a maas Environ and try to read its provider-state file and then list the tools10:23
rogpeppewallyworld_: with a 7 second gap between them10:23
wallyworld_yes. but what would that tell us10:24
wallyworld_that we don't already know10:24
rogpeppewallyworld_: if that failed, that would be very useful10:24
rogpeppewallyworld_: because we then have a very simple example that we can reduce further to see what's actually going on10:24
rogpeppewallyworld_: if that succeeds then we know something else is up10:24
rogpeppewallyworld_: it should only take 5 minutes to write10:25
wallyworld_i'll try and do it after standup if i am still awake, othwrwise tomorrow10:26
rogpeppewallyworld_: one mo, i'll do it if you like10:28
rogpeppewallyworld_: try this (substitute "my-maas-environ-name" with your env name: http://paste.ubuntu.com/6239995/10:41
rogpeppervba: i'm a bit confused by the maas storage handling - how are the storages from different juju environments in the same MAAS kept separate from one another?10:42
wallyworld_rogpeppe: sorry, was afk. looking10:44
wallyworld_ubuntu@maas:~/go/src/launchpad.net/juju-core$ go run masstest.go10:44
wallyworld_------ GET
wallyworld_2013/10/15 20:44:31 get provider-state: file 'provider-state' not found not found10:44
wallyworld_------ GET
wallyworld_2013/10/15 20:44:38 list tools ok10:44
TheMuerogpeppe: standup?10:47
rogpeppeTheMue: "the video call is full"10:48
rogpeppemgz: standup?10:49
mgzon my way10:49
wallyworld_fwereade: ??11:00
rogpeppewallyworld_: http://paste.ubuntu.com/6240109/11:24
wallyworld_rogpeppe: new version ran fine11:27
* TheMue => lunch11:28
rogpeppewallyworld_: could you try the original version with a 15 second delay?11:28
wallyworld_rogpeppe: except bigjools just shot it down11:28
rogpeppewallyworld_: oh11:29
wallyworld_yeah :-(11:29
rvbarogpeppe: care to have a look at https://codereview.appspot.com/14696043/ ?11:36
rogpeppervba: looking11:36
wallyworld_rogpeppe: i got the server powerd up again. i added the 15s delay. all passed11:39
rogpeppewallyworld_: hmm11:39
wallyworld_rogpeppe: i can add your ssh key11:39
wallyworld_rogpeppe: done, you should be able to ssh in now11:40
rogpeppewallyworld_: great, thanks11:41
wallyworld_the ~ubuntu/go dir is where the 1.16 source lives11:41
wallyworld_i put the test go src you gave me in the juju-core root dir11:41
rogpeppewallyworld_: ok, i'm investigating now11:45
wallyworld_rogpeppe: awesome. i'll try and hang around a bit. if you are finished and i'm not around, can you sudo poweroff?11:45
fwereaderogpeppe, wallyworld_: thanks for looking into this -- I will try to pop back on and off, and I should be able to be around properly this evening if I can help you at all11:49
rogpeppefwereade: thanks11:49
wallyworld_fwereade: np. we've spent a long time on it and at every turn it gets more confusing. we will kick ourselves when the issue is found i'm sure11:50
allenaprogpeppe: When does EnvironProvider.Prepare() get called? I only want to set the environment UUID when bootstrapping an environment; existing environments should not have a UUID.12:37
allenaprogpeppe: Hello too :)12:37
rogpeppeallenap: hi :)12:38
rogpeppeallenap: it gets called only if there's no .jenv file in ~/.juju/environments for that environment12:38
rogpeppeallenap: and only if you call juju bootstrap or juju sync-tools12:38
rogpeppeallenap: so that should be exactly what you need, i hope12:38
allenaprogpeppe: That sounds about right, thanks :)12:39
allenaprvba: Do you happen to know why maasEnvironConfig.attrs exists? environs.config.Config has two maps, one for known and one for unknown config. Does that not suffice, or was that not around at the time?13:05
rogpeppeallenap: perhaps it's to save the map being copied every time an attr is used13:06
allenaprogpeppe: I'm a little rusty :) Can you explain why that would happen?13:07
rogpeppeallenap: environ.Config makes a copy of the map that it returns, to prevent mutation of the immutable Config value.13:08
rvbaallenap: no idea… by the looks of it, it just copies the pattern used in the openstack provider… maybe jtv (if he's around) will knwo.13:08
allenaprogpeppe: Ah yes, I see, ta.13:08
rogpeppewallyworld_: FWIW, these are the (only) requests i see. the final request is the one that fails (it fails because it sees EOF on the persistent connection) http://paste.ubuntu.com/6240523/13:25
wallyworld_rogpeppe: that looks about right. the simplestreams code uses a different http client. maybe that is related. the http client used by juju-core code has the ssl support. the http client in gomaas doesn't13:30
wallyworld_i'm not sure whar is done with the http transport in juju-core though13:31
rogpeppewallyworld_: they're both using the same transport13:31
wallyworld_rogpeppe: so at bootstrap is when the different hhtp clients are used, whereas destoy just uses the gomaasapi one13:32
wallyworld_surely not a coincidence?13:32
wallyworld_rogpeppe: although, did you run bootstrap with --upload-tools?13:33
rogpeppewallyworld_: no13:33
wallyworld_cause using --upload-tools doesn't first do a simplestreams search13:34
wallyworld_and that still fails13:34
rogpeppewallyworld_: ah, i'll try that13:34
rogpeppewallyworld_: http.Client isn't stateful unless it's given a custom RoundTripper13:37
rogpeppewallyworld_: so it shouldn't make any difference if plain http.Get is used vs creating a new client with http.Client{}13:38
wallyworld_i'm just clutching at straws, trying to think of twhat's different13:38
wallyworld_sinzui: hi, did you see the streams repository is sortof set up?13:39
sinzuiwallyworld_, I did not13:40
wallyworld_sinzui: ticket 6392513:41
wallyworld_it talks about the server sawo which i'm not familiar with13:41
wallyworld_sinzui: i need to change the url embedded in juju-core, and the generated simplestreams metadata and tools need to be uploaded13:42
sinzuiI saw that and update a bug over my weekend13:42
wallyworld_or the release scripts tweaked accordingly13:42
wallyworld_i can easily do the juju-core side ahead of time13:43
wallyworld_although if the tools were uploaded i could drop the legacy aws fallback at the same time13:43
wallyworld_rogpeppe: i fell asleep on the couch before so i had better get myself off to bed for real. maybe you can drop me a quick note with any progress and i'll pick up tomorrow? also don't forget to poweroff the server13:53
rogpeppewallyworld_: how should i do that?13:53
rogpeppewallyworld_: just sudo shutdown?13:53
wallyworld_sudo shutdown now13:53
wallyworld_or does sudo poweroff work? not sure13:53
rogpeppewallyworld_: i'm making progress BTW13:53
rogpeppewallyworld_: i have a suspicion of what might be happening13:53
wallyworld_oh great, nowyou've got me curious13:54
rogpeppewallyworld_: it looks like MAAS *is* closing the connection after 5s13:54
rogpeppewallyworld_: but somehow we're seeing that in the wrong place; not sure quite why yet13:54
wallyworld_hmmm. ok. would be great to know the sequencing of things so explain why now and not before13:54
rogpeppewallyworld_: it *may* be to do with someone not closing an http request body. i need more investigation13:55
wallyworld_if maas is going to do that, maybe we do need the close=true on the juju side?13:55
rogpeppewallyworld_: the http client *should* work ok even with that13:56
wallyworld_ok. good luck and thanks. talk tomorrow13:56
rogpeppewallyworld_: but i guess there's always going to be a race.13:56
rogpeppewallyworld_: okeydokey. sweet dreams :-)13:56
wallyworld_i'll ask jools tomorrow about the maas aspect of it13:56
jamespagefwereade, sinzui: any critical bugs for 1.16.0 that I need to push in pre-release of saucy?14:25
* sinzui looks14:25
sinzuijamespage, I don't see any criticals that can be pushed14:26
jamespagesinzui, coolio14:29
* jamespage puts his feet up for a bit then14:29
mgzjamespage: I have one funny report on a bug fixed for 1.1614:31
mgzthat I want to (un)confirm14:31
* jamespage takes his feet off the desk and listens14:31
mgzjamespage: see last two comments on bug 123673414:31
_mup_Bug #1236734: juju 1.15.1 polls maas API continually <juju-core:Fix Released by gz> <https://launchpad.net/bugs/1236734>14:31
mgzI'm pretty sure he just has the old version still14:31
jamespagemgz, oh - I can confirm that has good14:32
jamespagethe load on the serverstack maas server went from 2.0 to 0.04 post upgrade14:32
rogpeppejamespage: excellent news - i was a bit concerned my fix hadn't14:32
mgzI'll comment of the bug and see if I can help the guy.14:32
jamespagewhich is worrying - as it would indicate that the performance of the storage on MAAS is not great - I only have 6 physical and 8 lxc service units in the deployment14:33
jamespagemgz, maybe he forgot todo juju upgrade-juju14:33
jamespagefwiw that took a very long time to complete14:33
jamespage<jamespage> which is worrying - as it would indicate that the performance of the storage on MAAS is not great - I only have 6 physical and 8 lxc service units in the deployment14:33
mgzhm, storage performace, or just general api being really slow?14:37
mgzdjango is a fair bit of overhead on top of postgres14:38
* sinzui thinks there is a space in dependencies.tsv that breaks goamx14:41
mgzgah, and it's me...14:41
rogpeppesinzui: ah, i was wondering what the problem there might be14:41
mgzsinzui: I'll fix14:42
mgzshould have thought of that when it was falling over yesterday rog...14:42
mgzmy bad.14:42
rogpeppemgz: i should've thought of it too14:43
rogpeppesinzui: nice catch14:43
mgzfix proposed for rubber stamping. now, did I backport that too...14:46
natefinchhowdy all.  Sorry to miss the standup this morning.14:46
mgznatefinch: blame columbus!14:46
natefinchmgz: that bastard14:46
mattywfwereade, you about?14:48
mgzah, actually, it wasn't my one that borked, so is just trunk14:48
mgzblame say nate :)14:48
mgznatefinch: https://codereview.appspot.com/1470104314:49
TheMuenatefinch: heya14:50
TheMuerogpeppe: could you help me with the provisioner?14:51
natefinchmgz: dang, sorry14:51
rogpeppeTheMue: sure; gimme a minute14:51
TheMuerogpeppe: or even a level higher, I don't know if it is the provisioner14:51
TheMuerogpeppe: thx14:51
natefinchmgz: who uses tab separated values, anyway?  Why not csv like normal people? :)14:52
* natefinch grumbles about invisible, variable width characters14:52
rogpeppeTheMue: what's the issue?14:53
rogpeppenatefinch: because commas are more common in normal values14:54
TheMuerogpeppe: my question is that in case of a dying machine (e.g. by shutdown -fh now) is the a mechanism that updates the state on it?14:54
rogpeppeTheMue: i'm not sure14:54
rogpeppe /me goes to look14:55
* rogpeppe goes to look14:55
rogpeppeTheMue: i can't see anything that polls instances for their status, no14:59
TheMuerogpeppe: yes, that's my impression too14:59
TheMuerogpeppe: thx14:59
rogpeppeTheMue: np15:00
TheMuerogpeppe: cts reports of a customer which turn down machines (wonder why they do it instead of removing units) and then in status they see the machine as down but the unit as alive15:01
TheMuerogpeppe: that makes me wonder15:01
rogpeppeTheMue: i have a feeling that the address updater worker might be a good place to put this functionality (i guess it might be renamed to "machineupdater" if it that happened)15:02
rogpeppemgz: does that make sense to you?15:02
mgzseems reasonable, though I'm not sure it's the right fix for the problem of machines just going away15:03
TheMuemgz, rogpeppe: I'm trying to get more details from CTS15:04
rogpeppemgz: yeah, that would probably need another fix too, in the provisioner. although...15:04
rogpeppemgz: we'd need to decide what we want to do about that15:04
rogpeppemgz: do we want to automatically start another unit?15:05
mgzthat was the pyjuju behaviour, roughly15:05
mgznot sure if it's what we want though15:05
rogpeppemgz: indeed15:05
rogpeppemgz: we probably do if minunits is set15:05
TheMuemgz: it get's more clear now, they use juju to deploy openstack on maas and there the hacluster charm. then they stop nodes for failover tests15:18
fwereadeTheMue, rogpeppe, mgz: we dropped that behaviour, the only time anyone noticed it was when they didn't want it15:24
fwereadeTheMue, it is surprising that the unit is not reported as down if it's not running15:25
rogpeppefwereade: yeah, i thought so, although with SetMinUnits, we should perhaps rethink that15:25
rogpeppefwereade: because that expresses a clear intent, i think15:26
fwereadeTheMue, look at the api server, maybe it's not noticing that the unit's connection is gone and is hence not killing the presence bit15:27
rogpeppefwereade: ah of course, i'd forgotten about the presence thing15:27
fwereaderogpeppe, wrt units uncautious agreement; wrt machines uncautious disagreement ;p15:27
fwereadeer machines *cautious* disagreement15:27
rogpeppefwereade: agreed15:28
fwereaderogpeppe, essentially I need to write some access-revocation stuff for agents we force destroy15:28
sinzuiHow permanent is the theme-oil bug tag? Is there a better tag to represent the bugs? server?, hyperscale? cloud-server?15:29
fwereaderogpeppe, if I have that I can be sure that removed units aren't coming back and are safe to replace without compounding confusion15:29
rogpeppefwereade: i guess so, although i suppose it's not a huge problem15:29
rogpeppefwereade: ah yes, of course15:29
fwereaderogpeppe, I am still twitchy about presence nodes as an arbiter of true existence15:29
rogpeppefwereade: although...15:29
rogpeppefwereade: we won't be recreating the same units, will we?15:29
rogpeppefwereade: or the same machines to run them, come to that15:30
fwereaderogpeppe, I've seen a couple of `down (alive)` reports that have always cleared themselves up before I've been able to figure them out15:30
fwereaderogpeppe, so basically I'm just nervous about our ability to tell for sure what's in the system15:30
rogpeppefwereade: yeah15:31
fwereaderogpeppe, and I don't want things coming back to life once we think they're dead15:31
rogpeppefwereade: if we've destroyed their state entities, how can they?15:31
rogpeppefwereade: or perhaps that's what you're thinking of when you say "access revocation"15:32
fwereaderogpeppe, I'm just worried about the potential for races, with things coming back and managing to do something while we think they're gone15:32
fwereaderogpeppe, access revocation == telling things they're dead immediately, even if they're not quite dead in state yet15:33
rogpeppefwereade: we can't just kill 'em dead in state immediately?15:34
* rogpeppe is still going down the maas EOF bug rabbit hole15:35
fwereaderogpeppe, not sure we can while keeping sane guarantees15:35
fwereaderogpeppe, (tyvm for keeping on at that)15:36
fwereaderogpeppe, think of the relations a subordinate has joined, when that subordinate is in a container in a machine that itself has a unit running directly15:36
fwereaderogpeppe, even calculating the right transaction for smoothly cleaning up the top-level machine kinda makes me cry15:37
rogpeppefwereade: there's something really weird happening, which is causing the remote http connection to be dropped *just* before we make a GET request15:37
fwereaderogpeppe, I would like a quick "cut all these off" to be followed by a more measured cleanup of all the various agents potentially in play15:37
rogpeppefwereade: yeah, seems reasonable on second thoughts15:38
natefinchfwereade: do you know how far Tim got in looking at MaaS?  I was looking into a couple of the bugs, but was having trouble with my maas environment (as usual).   If he's progressing, it might be better for me to work on something else15:45
kurt_Hi all - I asked about bug 1236734 yesterday and didn't get a response.  This is listed as fixed in 1.16.0, but it's not.15:45
_mup_Bug #1236734: juju 1.15.1 polls maas API continually <juju-core:Fix Released by gz> <https://launchpad.net/bugs/1236734>15:45
kurt_Any chance we could get some attention on this?15:45
fwereadenatefinch, I saw rvba and bigjools talking with rogpeppe this morning15:46
rogpeppenatefinch: which MaaS problem?15:46
fwereadekurt_, mgz was going to follow up with that -- last I heard it was confirmed fixed by jamespage15:47
rvbafwereade: we talked about something different, we are in the process of fixing bug 1239488.15:47
_mup_Bug #1239488: Juju api client cannot distinguish between environments <MAAS:Triaged> <https://launchpad.net/bugs/1239488>15:47
natefinchrogpeppe: the couple I was looking at was destroying non-allocated machines and destroying machines from outside the juju environment15:47
fwereadekurt_, are all your agents running 1.16?15:47
kurt_fwereade: its not.  I have 1.16 installed and a still having issue15:47
fwereadervba, I *think* that's what natefinch was thinking of15:47
rvbanatefinch: that's the bug I just mentioned.15:47
rogpeppenatefinch: ah, yeah, there's some stuff being done for that15:47
natefinchrogpeppe, rvba, fwereade:  that sounds like a great fix for the bug15:48
kurt_fwereade: how can I bring up the agent inof?15:48
kurt_info rather15:48
fwereadekurt_, juju status15:48
fwereadekurt_, included agent-version15:48
natefinchsorry, brb, diaper duty15:48
rogpeppenatefinch: there's a CL here: https://codereview.appspot.com/14696043/15:48
fwereadekurt_, you may need to sync-tools before an upgrade-juju sees the latest tools to fix the problem15:49
kurt_Oh - good point - how can I update the agent? just sync-tools?15:49
jamespagefwereade, mramm2: any call outs you would like to add for juju-core in https://wiki.ubuntu.com/SaucySalamander/ReleaseNotes#Ubuntu_Server ?15:49
fwereadekurt_, sync-tools should get them in place for you; upgrade-juju will actually upgrade the agents15:50
fwereadekurt_, we didn't quite get the global tools source in place in time for it to be 1-step on maas :(15:50
kurt_oh - so even if I followed the correct upgrade process, I still would have ran in to this?15:50
kurt_sorry, I know this is the dev list and not real appropriate for this discussion15:51
fwereadekurt_, no worries :) and... I guess it depends on definitions, because upgrade-juju can't upgrade further than it knows; but it's not necessarily entirely obvious that you need to sync tools before a maas environment will see them15:53
kurt_interesting, right.  I'm trying to blog some stuff, so I'd like to capture that15:54
fwereadekurt_, the bug report is appreciated all the same -- it helps make it clear to us that we still need to smooth the process15:54
fwereadekurt_, *hopefully* you just hit a bad window, because we've got some infrastructure just coming online now to make it more transparent (unless you're actually cut off from the internet entirely -- if you're isolated, syncing will always be required)15:55
kurt_I'm not isolate15:55
kurt_isolated - full connectivity to internet on MaaS15:55
kurt_and juju15:55
fwereadekurt_, but, indeed, I don't think you'll see the benefits until 1.18 -- sorry for the infelicity15:55
mgzkurt_: I responded on the bug today after checking with others, you might not have seen the comment unless you subscribed to the bug15:56
kurt_so when upgrading juju - can you outline what the fool proof process should be?15:56
kurt_(as to avoid this situation in the future) :D15:56
kurt_mgz: thanks15:57
kurt_I didn't see that yet15:57
fwereadekurt_, for most users: `juju upgrade-juju`; for those using maas, or those who have manually synced tools in the past, you'll need a `juju sync-tools` first15:57
kurt_fwereade: I believe that I did the juju upgrade via apt-get upgrade after adding the ppa15:58
fwereadekurt_, we will hopefully be able to cut out that step for those not in isolated environments, but it's not there today15:58
fwereadekurt_, ha, sorry, I was focused on the agent side of things: yes, you should also apt-get upgrade your client; and you should probably prefer to upgrade the client first, because that way round has been exercised more, but either way round should be fine in practice16:00
kurt_so the upgrade of juju can be thought of at 2 levels, right? 1. the juju binary 2. the agents - is that a reasonable assumption?16:01
kurt_oh there are the tools too16:01
fwereadekurt_, yeah, both need to be done -- but 1.14/1.16 binaries and agents should interoperate happily16:02
fwereadekurt_, the tools are the agent binaries16:02
kurt_i was previously on .15 :D16:02
fwereadekurt_, just a quirk of terminology16:02
kurt_1.15.1 that is16:02
kurt_I C16:02
fwereadekurt_, then that *should* actually work too, but we don't make guarantees for odd-numbered minor versions16:02
fwereadekurt_, we have no intention of breaking upgrades ofc... but 1.15 *did* have upgrade troubles16:03
kurt_1.15.1 had some fixes I needed previously - but yeah I get that - but that's why I was asking about best way to fip bits between dev and stable stuff.16:03
fwereadekurt_, think of 1.odd versions as canaries to make sure we don't mess up the 1.even path16:03
* rogpeppe gets some lunch16:12
TheMuefwereade: hmm, thx for the hint, found the machinePinger in apiserver/admin.go, but funnily it's used nowhere16:18
natefinchrogpeppe, fwereade:  ec2 instance type constraints.... would love to have some input: https://codereview.appspot.com/14523052/16:22
mgznatefinch: I was leaving that one for next week16:22
mgzif you read through the mail archives, you'll find various arguments, and just having a value that's ignored depending on provider and generally has uncertain meaning isn't really an improvement on not supporting it at all16:24
allenapfwereade: Hi William, please can you add me to ~juju? Or please mark https://code.launchpad.net/~allenap/juju-core/maas-environment-uuid/+merge/191146 as Approved :)16:40
mgzallenap: did you do the rename rogpeppe mentioned?16:47
allenapmgz: In a subsequent branch I changed it to maas-environment-uuid then back again when I read his comment.16:48
mgzso, we want another branch generating it in Prepare for all providers?16:49
allenapmgz: I guess so. I have added the code to return an error if it's set in environments.yaml, but it's still in the MAAS provider for now.16:50
allenapmgz: I have a very specific problem to solve :)16:50
rogpeppewow, this is incredibly weird16:51
mgzallenap: submitted16:51
allenapmgz: Ta.16:51
sinzuifwereade, hazmat, thumper, You will be getting email about this bug https://bugs.launchpad.net/juju-core/+bug/123230416:53
_mup_Bug #1232304: consider tuning git setup for juju-core, and document caveats <canonical-webops> <doc> <feature> <pes> <juju-core:Triaged> <https://launchpad.net/bugs/1232304>16:53
* TheMue is stepping out for dinner, cu16:57
mthaddonsinzui: hi, can you give me any context on that bug (1232304)? it's a real problem for us at the moment - means some environments can't run upgrade-charm without significant effort16:58
=== natefinch is now known as natefinch-afk
allenapmgz: If you're itching to do a review, https://codereview.appspot.com/14644045 is scratching at the door waiting to be let into your heart.17:05
* rogpeppe has gone up a blind alley and needs to stop and make curry.17:09
rogpeppeg'night all17:09
sinzuimthaddon, I have asked for input from the engineers.17:09
allenapnn rogpeppe17:10
rogpeppeallenap: cheerio17:10
mgzallenap: may have a chance to go through that briefly17:21
=== natefinch-afk is now known as natefinch
natefinchmorning thumper.  How goes?20:09
thumpernatefinch: morning, good20:09
thumpercan tell that today is going to get a little broken up20:09
thumpertaking the car for a service in just under an hour20:10
thumperand will work from a cafe across the road while that is happening20:10
natefinchWish I could do that, my mechanic is sorta in the middle of nowhere20:10
* natefinch doesn't really want to choose mechanics based on amenities within walking distance, however....20:11
wallyworld_fwereade: you around? i was hoping for a hand off email about the maas http connection issue. do you know the status?20:33
fwereadewallyworld_, back soon -- rogpeppe was thinking for a while that it *was* an unclosed request body, but then I saw him saying blind alley :(20:39
wallyworld_ok, thanks20:39
thumperis the go bot set up to land on 1.16?21:14
wallyworld_thumper: yes21:14
thumperwallyworld_: so why isn't it21:15
wallyworld_it did for me last week, no problems21:15
wallyworld_i'll log in and check21:15
thumperapproved, commit message21:15
thumpernot getting merged21:15
wallyworld_thumper: https://pastebin.canonical.com/99089/21:16
wallyworld_thumper: wtf is a ghost revision?21:17
thumpera revision with referenced as a parent that isn't in the repository21:17
thumpermost likely, a stacking issue21:18
thumperbut launchpad can see it21:18
thumperso NFI21:18
wallyworld_maybe do a new branch, cherry pick changes, re-propose?21:18
thumperwallyworld_: I know what it is21:43
wallyworld_do tell21:43
thumperwallyworld_: no I don't21:43
* thumper thought he did21:43
wallyworld_nothing to see here, move on21:43
thumperyes I do21:43
thumperit does have to do with stacking21:43
thumperwhat I did was start with a 1.16 branch21:43
thumperand then moved back 3 revisions to be what was currently merged with trunk21:44
thumperthat was the base revision for that branch21:44
thumperwhen pushed to launchpad, it gets stacked on trunk21:44
thumpertrunk should have those revisions from 1.1621:44
thumperso now I'm confused again21:44
thumperbzr is doing something weird21:44
* thumper wonders if he can unstack it21:45
* thumper pokes21:45
wallyworld_nothing seems to be going right this week21:46
thumperlooking forward to next week?21:47
thumperbecause that'll be awesome21:47
thumperwell, that branch says it is reconfiguring as unstacked, we'll see if it actually works21:47
thumperbzr reconfigure --unstacked lp:~thumper/juju-core/bootstrap-state-no-constraints21:48
thumperin case you cared21:48
wallyworld_cool, i have not done that before21:49
wallyworld_yes, next week wil be great21:49
thumperwallyworld_: do you need any reviews?21:53
wallyworld_thumper: i have a couple actually21:54
wallyworld_there's a pipeline of 321:54
wallyworld_andrew has done the last21:54
wallyworld_https://codereview.appspot.com/14663043/ and https://codereview.appspot.com/14540055/ are the others21:55
thumperpaste the links here and I can go through them21:55
wallyworld_thanks :-)21:55
wallyworld_the last is https://codereview.appspot.com/14502059/ fwiw21:55
wallyworld_i've been looking at the http maas issue so haven't yet really looked at the review i do have, will do that today21:56
* thumper nods22:03
* thumper starts on the first one22:04
thumperlooks big22:04
thumperbtw, unstacking that branch worked22:05
thumpershould get merge in now22:05
fwereadewallyworld_, do we have docs for sync-tools --source somewhere22:12
fwereadethumper, hi, long time22:12
wallyworld_fwereade: not sure. i haven't really read any of our docs22:12
thumperfwereade: hey22:12
thumperjust noticed the provisioner tests timed out on go-bot for the 1.16 branch22:12
thumperanyone seen this before?22:13
wallyworld_fwereade: do we have any docs for sync-tools at all?22:13
wallyworld_thumper: i get random timeouts from the bot semi-regularly22:13
fwereadewallyworld_, hell, possibly not :/22:13
bigjoolswallyworld_: how'd you get on with the http problem?22:35
wallyworld_bigjools: hi, long story, lots of dead ends. but it looks like maas is closing the http connection and then juju complains about it22:36
bigjoolswallyworld_: and juju just needs to deal with that I guess22:36
wallyworld_still need to figure out root cause22:37
bigjoolsit might be getting closed because it's not using http/1.1?22:37
bigjoolsor the keep-alive header is not getting set?22:37
* bigjools stabs in the dark22:37
wallyworld_not sure, will check. but none of that has supposedly changed22:37
wallyworld_bigjools: so all of the http requests to maas go via the apache server?22:38
bigjoolsdid you work out if Go 1.1.1 made a difference?22:38
wallyworld_bigjools: no difference22:38
bigjoolswallyworld_: yes, maas runs as a wsgi container in Apache22:38
wallyworld_compiled 1.14 with go 1.1.2 and it worked22:38
bigjoolsso something changed inbetween 1.14 and 1.16.... fun22:38
wallyworld_nothing obvious though22:38
wallyworld_and the EOF at the client is really due to the server closing the connection22:39
wallyworld_but what triggers the closure22:39
wallyworld_that is the question22:39
bigjoolsyou could do something silly like compile 1.16 with the version of gomaasapi from 1.1422:39
bigjoolsjust to eliminate that22:39
bigjoolsor confirm it22:39
wallyworld_could do. a very quick look at gomaasapi showed no significant changes, or nothing obvious in the http request dispatch side of thnugs22:40
wallyworld_i think though the 1.14 gomaasapi won't worj with 1.16? will need to checki guess22:40
wallyworld_bigjools: you using the maas server this morning?22:42
bigjoolswallyworld_:  no22:42
bigjoolsI am weeping into my coffee22:43
wallyworld_i'll poke around some more then if that's ok22:43
bigjoolsbecause of this multi-environ crap22:43
wallyworld_ah, the users getting mixed up issue?22:43
bigjoolsI maintain it's a juju self-inflicted problem but thumper disagrees with me22:45
wallyworld_what would he know, right?22:45
thumpernot a lot22:46
bigjoolsat least we can disagree in a more violent manner in person next week :)22:46
wallyworld_i'll remember to pack the popcorn22:46
wallyworld_or maybe jelly22:46
* bigjools will secretly put one fewer shots in thumper's coffee22:46
* thumper packs the boxing gloves22:47
bigjoolsyou still 120kg thumper?22:47
wallyworld_of pure muscle22:47
rvbaDibs on the front row seats!22:47
thumperjust under 97kg now22:48
thumpernever wes 12022:48
thumper111 maybe22:48
wallyworld_wasting away to nothing22:48
bigjoolsso rvba has come up with the great idea of using juju's uuids in the filenames we send to maas22:50
bigjoolsrvba: of course we have to make sure the tools don't get those uuids in the names I think22:51
wallyworld_yep, that would be bad22:51
rvbabigjools: well, we just need to handle that transparently in the provider's code.22:51
wallyworld_not as simple as that22:52
wallyworld_you mean send the tools with the uuids?22:52
wallyworld_and translate inthe provider?22:52
wallyworld_or strip out the uuid and send the tools uncganged?22:52
rvbajuju itself will ask to a file with name="zzz" but the provider will just fetch the file named uuid+"zzz"22:53
rvbaSame when storing files.22:53
wallyworld_hmm. but then we will have multiple copies of the same tools tarballs22:53
rvbaThe only trick is the anonymously accessible files… but that will also work transparently I think, because in this case we simply use generated ids.22:53
wallyworld_canit be path based? ie don't use uuids for certain paths?22:54
wallyworld_so stuff under tools/releases is left alone22:54
rvbawallyworld_: well, yes, that could be an improvement.22:54
wallyworld_and tools/streams22:55
wallyworld_or just /tools in general22:55
wallyworld_that would save duplicating the same tools tarballs and metadata over and over again for each user22:55
bigjoolsso leave tools/ alone but add uuid/ to everything else22:55
wallyworld_yeah, think that will work22:55
rvbaYeah, totally possible.  But again, I think this should be done as a second step.22:55
wallyworld_why 2nd? i think it is flawed to needlessly duplicate lots of data22:56
bigjoolsrvba: so I don't want to change the filestorage object willy nilly  because of this22:56
bigjoolsit needs to be done at the provider level22:56
rvbaThere is actually very little to be done.22:57
rvbaBut that's the beauty of it.  It's just a fix to the provider.22:57
rvbaAll in provider/maas/storage.go22:57
rvbaBasically, we just need to translate filename <--> uuid+filename everywhere.22:58
wallyworld_and also the output from list22:58
rvbastorage.List(stor, "")22:58
rvbaBecomes storage.List(stor, uuid)22:58
wallyworld_i will be trivial to add the /tools exclusion22:58
rvbabigjools: this is really too good to be true!22:58
bigjoolsrvba: I disagree that it can only go there because it will affect the tools22:59
bigjoolsI think it needs to go in some of its callsites22:59
wallyworld_tools are access via storage22:59
wallyworld_so it should work, so long as tools are excluded from uuid mangling22:59
bigjoolsvia that storage object no?22:59
bigjoolsspecial casing tools in there is crazy22:59
wallyworld_agreed, i don't care where it's done23:00
wallyworld_but we should duplicate tools23:00
bigjoolsI'll sort the call sites out later23:00
bigjoolsright now I need breakfast and an inbox cleanout23:00
rvbaAgain, I suggest dealing with the tools second.23:01
bigjoolsor is that an outbox cleanout23:01
rvbaBecause we need to implement this and test it.23:01
wallyworld_but not in a flawed way23:01
rvbabigjools: so, I'll let the lab running so that you can test this23:01
wallyworld_tools duplication not good23:01
bigjoolsrvba: I can test it locally23:01
rvbanot good indeed.  But not fatal.23:01
bigjoolsmy maas setup is working fine here23:01
wallyworld_too much scope for hidden gotchas23:01
rvbabigjools: all right.23:02
bigjoolsI'd rather do it properly from the start, it's no big deal.23:02
wallyworld_bigjools: except i'm playing with it23:02
bigjoolsrvba: thanks23:02
bigjoolswallyworld_: easily fixed :)23:02
bigjoolsyou have a while anyway23:02
rvbabigjools: don't forget to base your work on Gavin's branch: ~allenap/juju-core/maas-environment-uuid-use23:02
bigjoolsdoubt I'll get to the QA stage until this afternoon23:03
bigjoolsrvba: affirmative, thanks23:03
bigjoolsis he finishe dcoding?23:03
bigjoolsis he finished coding?23:03
rvbaYeah, it's up for review.23:03
* bigjools looks at wallyworld_23:03
rvbabigjools: btw, thumper reviewed it.23:04
rvbaHe has an interesting point about backward compatibility.  It should be fine (I think extra arguments like agent_name will be ignored by earlier version of MAAS) but it's worth a test.23:05
bigjoolsrvba: it will be fine and a desirable outcome.  It leaves the functionality as-is23:06
bigjoolsI'll try to land it later23:07
rvbaAll I'm saying is that it's worth a test.23:07
rvbaAnd on that note, I think I'll call it a night.23:08
rvbaI'll be available to do some QA tomorrow morning if we still have time.23:08
wallyworld_davecheney: ping23:25
davecheneywallyworld_: ack23:31
wallyworld_i have tweaked a go file from the std lib (net/transport.go). how do i force that to compile so that a go build -a ./... from juju-core uses my mods?23:31
wallyworld_i tried go build in the /usr/lib/go/pkg/net/http dir23:31
davecheneywallyworld_: you should, 1 remove the packaged go23:32
davecheney2. download go from source23:32
davecheneythe packaged go cannot be modified23:32
davecheneyit's too mutated23:32
wallyworld_davecheney: this is on jool's maas server which roger played with last night and he made changes23:32
wallyworld_i'm trying to revert them23:32
davecheneyif they are to the packaged version23:33
davecheneyremove the package23:33
davecheneymake suer /var/lib/go is empty23:33
davecheneyreinstlal the package ?23:33
davecheneywould that work ?23:33
wallyworld_i think so. i assume go 1.1.2 is in the repos?23:33
wallyworld_although if roger made his changes stick, i shoild be able to as well?23:33
wallyworld_i wonder what command he ran?23:34
davecheneywallyworld_: i don't think I really understand the problem23:34
davecheneyor the solutoin you are persuing23:34
wallyworld_ok, so roger modified transport.go23:34
wallyworld_to add extra logging23:34
davecheneywallyworld_: you coudl try, as root23:34
davecheneycd /usr/lib/go/pkg/net/http23:34
davecheneygo install -x23:35
wallyworld_and he did so by editing /usr/lib/go/pkg/net/http/transport.go directly23:35
davecheneyi don't know if that will work23:35
davecheneynever tried it23:35
wallyworld_will try23:35
wallyworld_davecheney: that looks like it worked. the timestamp on the .a file has now chaged23:35
wallyworld_thanks :-)23:36
davecheneywhen screwing with this23:36
davecheneyi recommend -x and -v flags23:36
davecheneyso you can see when things are chaning23:36
davecheneyor more important, when they are not 'cos the tool thinks everything is up to date23:36
wallyworld_ok. i've not used -x before23:36
wallyworld_yeah. i thought go build -a would have done the job23:37
wallyworld_but it seems that's not for the std libs, only deps of current project23:37
davecheney-a has problems23:39
wallyworld_don't we all23:39

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!