[00:16] bigjools: wtf. bootstrap on your maas box works now [00:21] bigjools: so roger and i were debugging and trying things. then the server got shutdown. now, it seems it has just started working. i tried with and without all the debug logging [00:21] maybe the power cycle on the server helped, not sure [00:22] wallyworld_: dafuq [00:23] haha [00:24] bigjools: so looks like we have wasted 2 man days when we should just have listened to Roy ans Moss and "turned it off and on again" [00:24] \o/ [00:24] I still think it's a bug [00:25] well, not much juju can do if maas/apache closes the connection [00:25] from underneath it [00:25] i guess it could retry [00:25] but wtf [00:25] that's the only place that needs such logic [00:34] network code should never ever assume connections will stay open [00:34] that's for the http lib to worry about [00:35] yes [00:35] I think the Go http lib is a little crazy [00:35] exposing that Close setting is one thing, but requiring it before it can cope with the other end closing it is a bug [00:35] hard to argue that i think [00:36] anyway can someone land this please, I am not in the juju team any more: https://code.launchpad.net/~allenap/juju-core/maas-environment-uuid-use/+merge/191249 [00:36] i'll add you [00:36] please don;t :) [00:36] what's it worth to you [00:36] coffee and lunch at the Tavern? [00:36] tempting [00:37] lower latency to my maas server? [00:37] who knows [00:37] I'm slobbing it on the outdoor sofas today [00:38] bigjools: so that branch, does it duplicate tools? [00:38] not started it yet [00:38] I do not intend to duplicate them [00:38] ah, sorry, i thought the one you wanted landed was it [00:39] no, it's gavin's agent_name fixes [00:39] i should have read the description [00:39] bigjools: do you intend to propose this against 1.16 too? [00:39] failing the tavern, $5 big mac? :) [00:39] or just land in trunk? [00:40] honestly NFI what's best [00:40] I am not familiar with the release plans [00:40] thumper: when is 1.18 due out? [00:40] * thumper shrugs [00:40] someone said yesterday that there's another release for saucy [00:40] there is [00:40] 1.18 i think [00:40] so trunk then [00:41] bigjools: so gavin's fix, how critial is it [00:41] very [00:41] and the one I am about to do [00:41] i wonder if we need a 1.16.1 then [00:41] fwereade: any idea on release plans as per the backscroll? [00:42] are you going to approve the MP then? [00:44] yeah, sorry saw something shiny and got distracted [00:44] should be in the bot now [00:45] heh [00:45] thanks [00:46] bigjools: about lunch, is there a place to buy decent coffee beans out your way? [00:46] wallyworld_: the little bean in Kenmore [00:46] that's more out my way :-) [00:46] it's on the way :) [00:47] that's the nearest [00:47] though there's a new coffee shop coming apparently \o/ [00:47] i need coffee. i could drive out to you and get beans also. kill two birds with the one stone [00:47] poyfekt [00:48] remember that the cafe closed down, you have to go to the smaller place on the other side of the road now [00:48] what time? [00:48] yes [00:48] wallyworld_: I have a feeling that we'll only be able to put 1.16 point releases directly into saucy [00:48] anhy time you want [00:48] however [00:48] but continue as normal with trunk [00:48] I have a call from 12-1 [00:48] and the ppa [00:49] bigjools: i'll leave soon i guess [00:49] sure [00:49] thumper: that sucks [00:49] wallyworld_: that's working with distro [00:49] i thought 1.18 was going into saucy [00:49] they will only take cherry picks into saucy now [00:49] wallyworld_: unlikely at this stage [00:49] cause i've done a bunch of stuff in trunk [00:49] assuming it would be in saucy [00:50] this is very bad [00:50] 1.16 is not ready [00:50] there's still the tools repository to do [00:50] and the ongoing maas stuff [00:50] and lots of other tooling stuff [00:51] if we are forced to cherry pick stuff, it will be like the whole fucking cherry tree [00:58] bigjools: leaving now [00:58] wallyworld_: righto [01:10] every time I leave the juju code base for a while and then come back to work on it, I struggle to get everything compiling. I presume this is because of mismatched dependencies. What's the best way of dealing with this? [01:12] or, I suspect, branches moving and Go has the bug of using the wrong url for a branch :/ [01:12] * thumper nods [01:12] that is one [01:12] yeah goamz moved it seems [01:13] apparently jam had a proposal to get golang to use lp: urls for launchpad [01:13] no interest [01:13] oh dear [01:18] bigjools: yup, if the owner of goamz has moved [01:19] the go get'd branch is probably pointing at the wrong place [01:19] indeed it was [01:20] bigjools: niemeyer added support for bzr to go get [01:20] if you can show me what is wrong, i can try to get it fixed [01:20] thumper can explain it better than me [01:20] but the upshot is that it needs to pull from lp:project [01:20] not the actual branch url [01:21] davecheney: when the go tool resolves bzr branches to launchpad [01:21] it expands the project name into the full http url with unique name [01:21] this is very slow [01:21] the old url is http://bazaar.launchpad.net/~gophers/goamz/trunk/ [01:21] most LP users have their lp identiy set in bzr [01:21] the new url is bzr+ssh://bazaar.launchpad.net/+branch/goamz/ [01:21] which means lp: urls resolve to bzr+ssh [01:21] the latter is owner-agnostic [01:21] if you don't lp urls resolve to http [01:21] so lp: is better [01:22] also [01:22] bzr+ssh://bazaar.launchpad.net/+branch/project [01:22] always resolves to the development focus trunk of the project [01:22] even if the owner changes [01:22] but go get will turn "launchpad.net/loggo" into http://bazaar.launchpad.net/~thumper/loggo/trunk [01:23] instead of bzr+ssh://bazaar.launchpad.net/+branch/loggo [01:23] if go get passed "lp:loggo" to bzr [01:23] bzr translates to the best it knows [01:23] which is bzr+ssh if it has your id [01:23] and http if not [01:23] thumper: i'm pretty sure the choice of http is deliberate [01:23] davecheney: deliberate and stupid [01:23] IMO [01:24] fair [01:24] it is a choice made by someone who doesn't understand the bzr tool [01:24] and when jam suggested a patch to golang, they ignored it [01:24] * davecheney has no comment [01:24] even though he is probably the best person to make such a suggestion [01:25] * thumper goes back to reviewing wallyworld's brach [01:25] \o/ [01:34] * thumper needs to go pick up the car from the garage [01:34] bbs === thumper is now known as thumper-afk === thumper-afk is now known as thumper [02:12] axw: how are you doin? [02:14] thumper: heya [02:14] not too shabby [02:14] working on fixing null provider bugs [02:15] the apt repo one's a bit of a pain, need to extract the key from the keyserver... cloud-init would normally take care of that [02:15] * thumper nods [02:15] there isn't a handy command we can use? [02:16] doesn't add-apt-repository download the key? [02:17] thumper: only for ppas [02:17] bummer [02:17] I'm looking at the cloud-archive case [02:17] * thumper nods [02:23] * thumper goes to pick up the wife [02:23] geez [02:23] broken day [02:23] bbs [02:27] davecheney: are you aware of any tools for looking for unused functions/vars/types/etc.? [02:28] or, how can I identify all functions that are only ever used in tests [02:36] axw: I think there is a mode for go vet in 1.2 [02:37] and kamil kissel has written a tool [02:37] davecheney: thanks, I'll take a look [02:53] thumper: ping [02:54] thumper: i did some fixes for axw's review in the wrong branch in the pipeline. i'm fixing now so ignore the new diff in your review. [03:09] ok [03:30] thumper: what i did do though is reply to your comments on both merge proposals. i'll fix the issues like gc.HasLen etc but there's also a few things i've replied back to [03:32] * thumper nods [03:32] wallyworld: I feel I may pop down to harvey norman to look at the coffee machine [03:33] really need one that doesn't make me angry [03:33] yes indeed [03:33] get the dual boiler! [03:33] thumper: do all tests really need to extend loggingsuite even if they don't requite the base functionaity [03:34] seems like a waste [03:34] wallyworld: the logging suite captures the logging [03:34] without it, tests become noisy [03:34] if someone decides to add logging somewhere [03:34] that is in the testing path [03:34] fair point [03:34] that's all it really does [03:35] i guess we should fix all existing test suites at some point then [03:35] * thumper nods [04:40] thumper: if you still have any spare bandwidth left today, i've done fixes for those 2 mp's [04:48] wallyworld: when you have a moment, I've updated https://codereview.appspot.com/14527043/ [04:48] sure [04:48] wallyworld: sync no longer resolves metadata, but "juju metadata generate-tools" will still [04:56] axw_: i think everything should calc the sha etc, drop the option to allow it not to be done. the sha256 and size is absolutely needed for sync tools [04:57] and generate metadata is typically run using local files so it can always be done for that as well [04:57] wallyworld: it's only for existing tools with no metadata - I thought the conclusion was that it would be okay after 1.16? [04:58] after 1.16, there should be no metadata without size/checksum [04:59] so if this mp is to go into trunk, then drop the resolve option altogether [04:59] imo [04:59] always just do the checksum/size [04:59] wallyworld: as in, behave as if the call were specified with fetch/resolve==true all the time? [05:00] yeah [05:00] what's the point if there's no metadata without size/checksum? [05:00] the fetch=true tells the command to read the tarball data to do the size/checksum when the metadata is generated [05:00] and that's what we always want now [05:00] since we don't want to produce metadata without size/checksum [05:01] so fetch=false should be verboten [05:02] make sense? or am i missing something? [05:02] wallyworld: with the change, metadata is still beign generated with size/hash. It's populated when the tools are copied to storage [05:02] wallyworld: the only thing that's affected is tools that are in storage, but either don't have metadata, or have metadata without size or hash [05:04] so - if i have some tarballs locally, and i just want to generate metadata json, and not copy the tarballs anywhere - that's what the generate-metadata command does - that should always happen with size/hash [05:04] and even if the tarballs are not local, ie on a cloud, the same applies [05:04] the generate-metadata command should always produce json with size/hash now [05:04] wallyworld: yes, it does and will continue to do so with this change [05:05] generate-tools only [05:05] as of 1.16, there should be no metadata without size/hash [05:05] so i'm not sure if your comment above holds? [05:05] right [05:05] this one i mean [05:05] [15:02:48] wallyworld: the only thing that's affected is tools that are in storage, but either don't have metadata, or have metadata without size or hash [05:06] right, so my point is - the change won't break anything :) [05:06] sure, but why cater for a forbidden scenario [05:06] as in, it only affects a scenario that won't occur [05:06] it just complicates the code base [05:06] I'm explicitly not catering for it now [05:06] but there's still the fetch option etc [05:07] that is no longer needed [05:07] fetch=true always [05:07] sorry, wallyworld are you talking just about the metadata plugin? [05:07] as in, get rid of the command line option and have *that* always fetch? [05:07] i was just looking quite narrowly at the diff in the code review [05:07] and saw the option to resolve or not still there [05:08] wallyworld: yeah, that's *only* in the plugin now. [05:08] i do think we should always fetch, but we can do that as a separate mp [05:08] I can make it always do it [05:08] sorry, my brain hadn't made the distinction of what was where when reading the diff [05:08] can I just confirm that it's okay *not* to resolve metadata for syncing? [05:09] we do need to resolve for syncing [05:09] when I say resolve metadata, I mean fill in size/hash [05:09] cause we may have new tools [05:09] that need to be copied [05:09] wallyworld: heh, I mean for existing tools [05:09] sorry [05:09] not for newly copied ones [05:10] newly copied ones will always get it, there's no option to disable it [05:10] ok, i think it's reasonable, in trunk, to assume existing tools will have size.hash [05:10] agree? [05:10] okay, cool [05:10] yes [05:10] by brain hurts :-) [05:10] my [05:10] sorry :) [05:10] not your fault [05:10] wallyworld: and I'll update the generate-tools command to always fetch [05:11] ok, that would be great. i like leaving less legacy / tech-debt :-) [05:11] thaks :-) [05:15] wallyworld: updated [05:15] looking [05:19] axw_: looks good, land that fucker :-) [05:19] sweet, thanks [05:19] thank you for making it all work :-) [05:20] heh nps [05:34] pls to be reviewerating https://code.launchpad.net/~julian-edwards/juju-core/maas-uuid-file-prefix/+merge/191336 [05:34] sorry no Blofeld [05:43] bigjools, so how's the environment-uuid config field hooked up to the actual environment uuid? [05:44] fwereade: don't know the details, allenap did that already [05:46] fwereade: the UUID is allocated randomly, at prepare time [05:46] so... pointing at the same env requires sharing the UUID [05:50] axw_, bigjools: looks like that's not an environment UUID at all, it's just somemade-up shit :/ [05:50] yeah [05:51] * fwereade sighs deeply [05:51] bigjools, your branch looks fine [05:51] fwereade: ok thanks [05:52] and wow are you working late or in a different TZ? [05:52] bigjools, early [05:52] bigjools, flying to the US later today [05:52] fwereade: it calls utils.NewUUID() in gavin's branch [05:52] bigjools, need to go and see laura fora bit though, might not be back [05:53] so what is the somemadeup-shit you're talking about? [05:53] hoho [05:53] bigjools, the problem is the overwhelming bugfuck insanity of naming that thing "environment-uuid" when we already have an "environement uuid" that is not at all the same thing [05:53] bigjools, how to write unmaintainable code vol 1 ch1page 1 [05:54] bigjools, but I cannot deal with this now,I might be back shortly [05:54] it's very easy to criticise [05:54] but at least it got done [05:56] so do you guys still need two +1s or can I land on one now? [05:57] bigjools: just one [05:57] thanks axw_ === axw_ is now known as axw [05:59] axw: can you approve it please, I am ont in the juju team so I can't do it [05:59] bigjools: sure [05:59] thank you sir [06:42] bigjools, ok, I did not express myself in a helpful way and I apologise for that [06:42] bigjools, but I think it really is a problem that some environments now have two UUIDs and there's no clear distinction between them [06:43] bigjools, would it be possible to do a quick branch that just s/environment-uuid/maas-agent-name/ and eliminates this source of confusion? [06:43] fwereade: I'm not sure where Gavin received his advice from, but I believe it was mostly under the direction of someone in the core team and that whoever it was had a plan to resolve this [06:44] bigjools, yeah, I just read the review :( [06:45] sadly this is what happens when stuff needs to go in quickly before a release [06:47] bigjools, yeah, I would kinda like to figure out how the api-key fiction got created and then propagated so widely in the first place [06:47] bigjools, it never even crossed my mind that it was completely made up [06:47] bigjools, because it's persisted all the way through back from python days [06:48] bigjools, and we never even had a maas environment to check against for such a long time [07:04] mornin' all [07:04] fwereade: the environment-uuid thing is all my fault [07:05] fwereade: i don't really see what harm it can cause tbh [07:05] rogpeppe, heyhey, I saw the review, and I think I see the reasoning... but ISTM that now we have two "environment uuids" for maas environments, and I don't see how we're ever going to be able to pull them back together [07:05] fwereade: they don't join up [07:06] fwereade: the environment-uuid in the config doesn't make anywhere else, does it? [07:06] rogpeppe, then why do they have the same name? it looked like it was justified on the strength of being step 1 towards picking one at prepare time rather than bootstrap time [07:07] rogpeppe, which would be great, if we did it [07:07] rogpeppe, but now we have an environment config with one value, used by some parts of the system, and an environment doc with another used by different parts of the system [07:07] rogpeppe, and to imagine that never the twain shall meet strikes me as... optimistic [07:08] fwereade: well, currently maas has a private attribute called environment-uuid; the environment uuid in state doesn't come from or go into the config [07:09] fwereade: given that state.Initialize takes an environ config, we can easily change that at a later stage to put the environ-uuid from that into the current uuid doc [07:09] fwereade: and likewise we can easily change environs.Prepare to create it [07:10] fwereade: and when we do that, i *think* everything will just work, and the maas environ-uuid will then join up with the state uuid [07:11] wallyworld_: after sleeping on it, i *think* i know what's going on with the maas EOF bug [07:11] rogpeppe, that's fine for new environments, but existing environments will need to keep both around [07:12] fwereade: is that a problem? [07:12] rogpeppe, I think so, yes, because there is no longer a singular concept of environment uuid [07:13] rogpeppe, and I don't see how an existing environment can ever be brought in line [07:13] fwereade: is that a problem? [07:13] rogpeppe, well, yes, because an environment uuid is the only thing we have for globally identifying an environment [07:14] rogpeppe, and the last thing I want is to have to respond to bug reports by saying "ah, yes, it doesn't work because you should have used the *other* environment uuid" [07:14] fwereade: is it any worse than if maas created a new attribute, for example maas-machine-identifier ? [07:15] rogpeppe, yes, I think it is much worse [07:15] rogpeppe, a new identifier would have been great [07:15] rogpeppe, I thought I even saw you advocating that yesterday morning as I rushed by, and I thought "ah cool, everything's undercontrol" [07:15] fwereade: i advocated one or the other [07:16] fwereade: i quite liked the idea of just using environment-uuid, because i *don't* think there's a great problem currently - the maas attribute is not really visible to the user [07:17] rogpeppe, you think nobody looking at the environ config is going to be fooled? [07:18] rogpeppe, the environ config is most certainly visible [07:18] rogpeppe, it's *more* visible to the user than the one in the environ doc [07:18] fwereade: i actually think that fixing it properly is going to be quite a small change. [07:19] rogpeppe, what do we do about all the environments that have two uuids then? [07:19] fwereade: we just need to change environs/config to add UUID, change environs.Prepare to create it and change state.Initialize to use it [07:19] rogpeppe, apart from the fact that we have to carry code FOREVER to handle the fact that sometimes they're different [07:19] fwereade: really? [07:20] fwereade: what code would we need? [07:20] rogpeppe, code to figure out which one is "meant" at any given time [07:20] rogpeppe, as it is today we will be starting envs with two uuids [07:20] rogpeppe, both of which are exposed to external systems [07:21] fwereade: the other side of the coin is that in the future, we *would* like maas to use the environ uuid to tag its machines [07:21] rogpeppe, and which we therefore cannot change [07:21] fwereade: and if we don't make it use environ-uuid, it will forever use some other identifier [07:22] fwereade: well, some other attribute anyway [07:22] fwereade: because it could still take its value from environ-uuid [07:22] rogpeppe, yeah, that would be nice, we would be able to derive the differently-named attribute from the real uuid if a legacy one werenot already set [07:23] rogpeppe, bigjools: is there *any* way we can get this fixed without releasing in this state? [07:23] fwereade: well, it's just a naming issue right? [07:24] fwereade: so we just need to change the name [07:24] rogpeppe, yeah, but I am out of the loop and have no idea what timelines etc are in play [07:25] fwereade: we are at the mercy of the release managers in ubuntu [07:26] rogpeppe, if you can fix it, or ask someone else to, in time to not release with it in place, please please do so... but I have about half an hour to get up, pack, and catch a taxi to the airport [07:26] this is a major flaw in juju and maas and really needs to at least be a zero-day fix [07:26] so there is time to change it I think [07:26] fwereade: ok. how about i just fix it properly? i *think* it's quite a small change, though i may be wrong [07:27] rogpeppe, if you were to use environment-uuid in InitializeState that would be fine with me too [07:27] but one of you needs to do it AFAIC because my engineers have done enough already [07:27] fwereade: i'll give it a go [07:27] rogpeppe, can you do that please? and coordinate with jamespage I guess? tyvm [07:28] fwereade: i know what's going with the MAAS bootstrap EOF bug BTW, i'm pretty sure [07:28] fwereade: it's a very interesting conjunction of issues [07:40] rogpeppe: hi [07:40] wallyworld_: hiya [07:41] sorry, i was out getting my presecription filled before i go away [07:41] pwd [07:41] rogpeppe: a reboot of the server fixed everything [07:41] wallyworld_: of the MAAS server? [07:41] yep [07:41] i think juju's http is flawed [07:41] wallyworld_: i don't believe the problem is fixed [07:41] it should cope with disappearing connections [07:42] any networking stack needs to be robust [07:42] to connections going away [07:42] wallyworld_: i think the real problem is an underlying problem with the http protocol itself [07:42] sure, ut the http lib needs to hide that [07:42] wallyworld_: i'm not entirely sure whether it's possible [07:42] http libs from python et al do [07:43] wallyworld_: i wonder how they cope with this race: [07:43] wallyworld_: you use an existing connection and send a request, but the remote end drops the connection before it reads your request [07:43] hi juju devs: is it safe to use ~/.juju/current-environment as a reliable way to retrieve the current default env name? or should we just consider it an internal detail? [07:43] wallyworld_: then it looks like you're getting EOF in response to your request [07:44] why is a request data object dealing with protocols? [07:44] bigjools: ? [07:44] request has a Close on it [07:44] seems odd [07:44] bigjools: it's an http header [07:44] frankban: the value in that file can be overridden by JUJU_ENV i think [07:45] frankban: so i would not rely on it [07:45] wallyworld_: when the above scenario happens, should the http client resend the http request on a new connection (possibly duplicating side-effects) or just return the error? [07:46] not sure. i'd like to know how other libs handle it [07:46] wallyworld_: me too [07:46] wallyworld_: sure, I am trying to implement this logic: if JUJU_ENV is set, use it, otherwise, retrieve the default env as set by "juju switch". So my question is: how to reliably grab that value in the second code path? [07:46] but i've never seen this sort of behaviour elsewhere [07:47] wallyworld_: the thing is, it's usually a race with a very narrow window [07:47] wallyworld_: but in this case, an unfortunate set of circumstances conspire to make it happen every time [07:47] rogpeppe: in that case I'd expect the transport to deal with headers that affect its operation [07:47] frankban: what if juju switch has not been called yet? [07:48] bigjools: where should the user be able to tell the http package whether connections should be reused or not? [07:48] not on the request object that is for sure :-) [07:48] wallyworld_: it's ok, we tried and failed, and we have no default value. [07:49] wallyworld_: the last chance could be looking for environments.yaml[default] actually [07:49] frankban: is this a python script or something? [07:49] wallyworld_: yes it is [07:50] wallyworld_: the reason (i'm pretty sure, though i haven't had time this morning to verify) why we were seeing the problem every time, is that just before we send the request that fails, we do some very cpu-intensive operations for more than 5 seconds [07:50] frankban: so i think the order juju-core checks is: juju_env, juju switch file, env.yaml [07:51] frankban: so if you do that, you should be ok [07:51] wallyworld_: so, in order: JUJU_ENV -> juju switch -> environments.yaml[default] -> error "please specify an env name". [07:51] frankban: i think so [07:51] wallyworld_: and that meant that the goroutine that usually sees the remote connection being dropped was not being scheduled in that time [07:51] heh [07:51] rogpeppe: I'd have a higher level function on the transport rather than exposing protocol details on a request object [07:51] bigjools: the transport is actually lower level here, no? [07:52] bigjools: and most http clients don't see it [07:52] rogpeppe: not in that sense, I mean a function on the transport to say whether to do it or not. manipulating headers is low-level [07:52] rogpeppe, hey, change of heart -- please *don't* use environment-uuid, just change the name to something maas-specific [07:52] wallyworld_: yes my question is about the "juju switch" part: parsing the output seems fragile, and I was wondering if ~/.juju/current-environment is considered an internal detail. anyway, implementing something like "juju switch --format json" could be a good idea [07:52] rogpeppe, I'm not convinced we have properly thought through the issues witrh setting it early [07:52] rogpeppe, and I don't want maas/juju collisions [07:52] fwereade: really? [07:53] fwereade: I chatted to wallyworld_ about this earlier and we concluded that its akin to a private bucket name [07:53] rogpeppe, really really [07:53]