[00:00] ericsnow: has it been like that forever? i didn't think lxc had that issue? [00:01] wallyworld: this was just the lxd provider [00:01] ok, that makes sense since that provider was re-written from scratch [00:01] i didn't think we had this issue in 1.25 [00:02] with lxc [00:02] wallyworld: I'm guessing no-one had tried to use the LXD provider with non-amd64 before [00:02] not till now :-) [00:02] wallyworld: anyway, I gotta run; natefinch can fill in the rest :) [00:03] ok, ty [00:03] ttyl === Spads_ is now known as Spads [00:17] menn0: if you were going the first route, I'd just supply a names.Tag rather than a GlobalEntity [00:18] axw: yep, fair enough [00:18] menn0: I'm ambivalent though. I've found having the methods on state make it a bit easier to mock in tests [00:18] but at the same time, I don't like throwing it all on the one type [00:18] that ship has sailed I think :) [00:19] menn0: :) [00:19] but I guess I don't have to make it worse [00:21] axw: you're right about mocking in tests though... given that we only have concrete Machines I'm setting myself up for difficult testing if I add methods to Machine [00:21] difficult / dumb [00:22] menn0: some packages go to lengths to mock Machine out too, but it is a PITA [00:23] axw: especially for something so simple [00:23] menn0: not sure if you've looked at state/volume.go or state/filesystem.go, but this is why I put everything at the top level [00:23] it made testing much easier [00:24] menn0: I ended up having to have some methods on Volume/Filesystem to support, e.g. StatusSetter/StatusGetter [00:24] yep, makes sense [00:31] axw: howdy, wanted to chat on the model statuses and get your thought on something [00:31] axw: since the end state is archived, what about destroying is archiving? [00:32] rick_h_: it did occur to me. I would say it's not really archiving at that point. It's destroying the resources within the model, it's just the the model docs that are archived [00:32] axw: right, but it's working toward the archived state. And it's not used elsewhere because of that idea [00:33] axw: I guess it's not quite 'factual' but as far as state transitions it goes through a few moving bits on the way to archived. Picking any one points out a bit of what's up? [00:34] rick_h_: so I kinda agree that you should have through "archiving" to get to "archived", but at the same time I don't think "archiving" implies that anything is being cleaned up [00:34] axw: do you have a link to the bug there? I wanted to go back and refer to the list you had but I'm failing to see the bug [00:34] but more ... put away [00:34] 1 sec [00:35] rick_h_: https://bugs.launchpad.net/bugs/1534627. I've currently got it as "active", "destroying", "archived" in my branch [00:35] Bug #1534627: Destroyed models still show up in list-models <2.0-count>

[00:36] axw: so what caused us to keep it around by default anyway? [00:36] axw: vs a cleanup all the wya and removal? [00:36] rick_h_: I don't know, that predates my involvement. thumper? [00:36] axw: I mean it seems useful, but it seems like the exception to the rule. A destroy-model --archive that has this seems like the cleaner wa [00:37] it definitely is an exception to the rule [00:37] I guess I assumed we had a good reason from some stakeholder but I'm coming up empty for making this the default behavior. [00:37] wallyworld: ^ [00:38] rick_h_: predates me too [00:38] hmm... [00:38] rick_h_: it doesn't seem useful to me, TBH. I'd prefer to just lean on external logging to keep records [00:38] was there mostly so we don't suddenly error [00:38] axw: witht he rsyslog logging work going in I'm +1 to that idea [00:38] because the way people watch an environment being destroyed is status [00:39] if you had "watch juju status" running [00:39] you would watch the services go down [00:39] machines being removed [00:39] thumper: oic, so this was so that you could see it die vs go away and then if it failed to destroy you had no good way to get at it? [00:39] then "error: unknown environment" [00:39] mostly so you could watch it die without errors [00:40] and grab logs... maybe [00:40] axw: can you investigate if we can just destroy in a clean way and do away with the kept around models at all please? [00:40] axw: I feel like we're chasing the wrong end of the problem atm [00:40] rick_h_: our general pattern has been to shepard entities through dying -> dead. until they are reaped by a cleaner, they will still be there [00:41] rick_h_: sure [00:41] wallyworld: right, so can we do that better vs keep things around like this? [00:41] the 24 hours thing was entirely arbitrary [00:41] and something I picked out of the air [00:41] we could have a sensible default... [00:41] of a much smaller number [00:41] thumper: yea, I mean can we just watch it until it's dead cleanly and then remove it right away? [00:41] rick_h_: oh, i didn't realise they were kept for 24hrs [00:41] e.g. check every 5s or something? [00:41] rick_h_: i thought they were removed immediately once dead, sorry [00:42] I'm all for giving the user confidence/observability in what they're doing [00:42] it is the client going "juju status" [00:42] and us wanting to say "environment is dead" [00:42] rather than "error, you suck" [00:42] right, so at some point it fails that "model is not available" or the like [00:42] eventually [00:42] you asked for it to go away, what do you expect? [00:42] right now, 24 hours later [00:42] yea, that's gotta go imo [00:42] this is the job of the undertaker [00:42] if we can tell from status it's dead, then we should be able to reap it right then and there [00:43] +1 [00:43] and if status says it's not dead, then we don't reap it and you can get at it and diagnose [00:43] that's ow i thought it worked [00:43] heh no [00:43] didn't realise models were special [00:43] so we were looking to do all this renaming and such which :/ [00:43] little snowflakes in the wind :) [00:45] oh shit [00:45] I think I need to work out this juju 2.0 cli thing [00:45] * thumper needs to add a maas cloud [00:45] perhaps after a dog walk [00:45] * rick_h_ grumbles [00:45] thumper: on that please fix the whole maas as a cloud vs not needing credentials/etc kthx [00:46] wat? [00:46] * thumper has no idea about that [00:46] rick_h_: as in maas showing up in list-clouds? [00:46] thumper: there's a thing in that maas setup that's confusing users [00:46] maas is a cloud isn't it? [00:46] maas is a type of cloud [00:46] axw: yea, and if you do add it and then you try to add-credential it tells you maas does't need credentials [00:46] wat [00:46] but then users don't know how to use it since it's different [00:47] wat? [00:47] * redir goes eod [00:47] see you tomorrow juju-dev [00:47] night redir [00:47] rick_h_: I think that might be covered by anastasiamac_'s branch, http://reviews.vapour.ws/r/4573/ [00:47] night redir [00:47] rick_h_: iwat axw said - it's beeing fixed ^^ [00:47] (landing, really) === redir is now known as redir_afk [00:48] thumper: about cloud vs cloud type, see https://bugs.launchpad.net/juju-core/+bug/1564054 [00:48] Bug #1564054: lxd, maas and manual do not make sense in list-clouds

[00:49] rick_h_: what do you mean? juju add-credential maas works fine [00:49] wallyworld: hmm, a user was hitting it and gave me a pastbin the other day [00:50] rick_h_: that would have been an old beta [00:50] wallyworld: I'm trying to find it, they got an error out of juju along the lines of "xxx does not need a credential" [00:50] this is all wip [00:50] :) [00:50] heh [00:50] it used to be that [00:50] we have been delivering things as threy are finished in each beta [00:50] ah ok [00:50] never mind then [00:50] add credentials came late in the beta cycle [00:51] rick_h_: stakeholders want maas included in list clouds [00:51] wallyworld: did we have plans for an add-cloud as wel? [00:51] that's why we did it we know maas is not a cloud [00:51] wallyworld: well, maas clouds need to be there. The trouble is the lack of an add-cloud command for now [00:51] wallyworld: a user would hit a "boom" if they specify cloud as part of command arguments.. [00:51] wallyworld: yea, I was one of those, but mark is right [00:51] rick_h_: we have add-cloud, but not interactive [00:51] anastasiamac_: wallyworld so I +1 going with Mark's feedback there [00:51] axw: right, interactive is what I'm thinking [00:51] you just point at a YAML file [00:52] axw: not for 2.0 but we should add it down the road [00:52] rick_h_: will you break the news to adam? [00:52] it would be helpful [00:52] axw: right, but folks are confused as to what goes in the clousd.yaml, the credentials, and the config [00:52] * axw nods [00:52] wallyworld: stokachu? [00:52] yeah [00:52] he needs maas in list clouds for his app [00:52] wallyworld: heh ok, will do [00:53] wallyworld: I'm still -1 on the maas:/ special thing where you don't have to add it to the clouds.yaml [00:53] rick_h_: that was also well received and asked for by users :-) [00:54] wallyworld: bah, it just adds a 'differnet way to do it' that's special and unique and causes confusion folks have to look it upo [00:54] rick_h_: you mean like lxd :-P [00:54] there's no lxd in clouds.yaml also [00:55] wallyworld: heh [00:55] ah crap... [00:56] rick_h_: the use case driving this is - people want to know what they can put with juju bootstrap [00:56] I don't think my 1gb kvm maas instances are up to running juju are they? [00:56] wallyworld: remember when I said the whole cloud credential spec would be a big pile of work? [00:56] wallyworld: yea...but they can't do maas without config info (in this case the IP address) so we've changed the bootstap command to make it work [00:58] rick_h_: altered slightly yeah, but still inituitive imo [00:58] * thumper goes to walk the dog and get away from the copmuter a bit [00:58] thumper: oh i know [00:58] i never doubted it [00:59] we piled in so much stuff in such a short time, and delivered incrementally over several betas and got beat up when it wasn't all there beta 1 :-/ [01:09] wallyworld: no one got beat up :P [01:09] wallyworld: I couldn't reach you! [01:09] lol [01:09] rick_h_: not beat up directly [01:10] :P [01:10] , more complaints :-) [01:15] wallyworld: we just had an opinion that add-credential should have come first :P [01:15] not complaints, opinions...I hear it's like something else everyone has :) [01:16] rick_h_: sure, but it was harder to add and was icing - we needed the core functionality with credentials added by hand. having fancy add credentials with nothing working would not have been cool [01:16] wallyworld: understand completely...after I stopped and thought about it :P [01:16] :-) [01:17] rick_h_: next time we'll start the work at the beginning of the cycle, not a month or 2 out :-) [01:17] wallyworld: psh, don't go changing everything on me [01:17] great, now i've got that song stuck in my head [01:17] lol, glad to be of service [01:18] i love you just the way you are.... ta de dum [01:43] axw: got 5 mintes, standup? [01:43] wallyworld: sure, brt [01:50] should juju2 be able to bootstrap against a private openstack cloud? I'm getting: [01:50] 2016-04-14 01:30:25 ERROR cmd supercommand.go:448 failed to bootstrap model: model "admin" of type openstack does not support instances running on "amd64" [01:50] my index.json does list 14.04 images... [02:09] ah, when I don't use --upload-tools it gives: [02:09] 2016-04-14 02:05:56 ERROR cmd supercommand.go:448 failed to bootstrap model: cannot start bootstrap instance: no "trusty" images in bootstack-canonistack-bos01 matching instance types [m1.small m1.medium m1.large m1.xlarge] [02:20] wallyworld: ping [02:21] wot [02:22] thumper: ? [02:22] wallyworld: got a few minutes to chat? [02:22] sure [02:23] bradm: juju 2 should work similar to juju 1 for private clouds [02:23] it's all about getting the correct streams metadata which can be tricky [02:23] wallyworld: just filed LP#1570162 about it [02:24] but if it works for juju 1 it should work for juju 2 [02:24] ok, we'll look at the bug [02:24] let me know if you need any more info about it [02:24] wallyworld: 1:1 ? [02:24] yup [02:24] wallyworld: this is mitaka on trusty, and it has different arch compute nodes too [02:25] grabbing some lunch, back in a while [02:25] Bug #1570162 opened: juju2 openstack private cloud cannot start bootstrap instance

[02:45] wallyworld: juju-mongo-tools3.2 just got accepted \o/ [02:45] * mwhudson runs away for a bit [02:45] mwhudson: awesome [03:00] axw: I was thinking that the machiner would send the SSH host key to the state server (via the machiner facade) [03:01] menn0: sounds fine to me [03:01] axw: seem reasonable? [03:04] wallyworld: btw, this is the fix for the lxd arch bug that I'm pretty sure works: https://github.com/juju/juju/pull/5116/files. But I still can't quite fully test due to not being able to fully disable upload-tools.. I haven't figured out how to disable saving the tools (I've found a couple likely spots and commented out code, but to no avail). [03:04] ok, i'll look at pr, thanks [03:05] there's a toolsstorage that manages the tools in gridfs [03:07] wallyworld: I commented out this whole loop: https://github.com/juju/juju/blob/master/apiserver/tools.go#L262 which looks like the place where we store the tools after uploading them... but it didn't seems to change anything [03:09] natefinch: why not just make storage.Add() itself a no op [03:10] natefinch: anyway, here's what you wnated [03:10] fetchAndCacheTools [03:10] or maybe not, that is when we download then from streams [03:11] it's in jujud/bootstrap.go [03:11] logger.Debugf("Adding tools: %v", toolsVersion) [03:11] if err := toolstorage.Add(bytes.NewReader(data), metadata); err != nil { [03:11] i just searched for usages of toolsstorage.Add() [03:11] there's only a few to check [03:13] wallyworld: thanks... I really wasn't sure what to search for [03:13] just the .Add() [03:13] and see what calls it [03:13] as that's where tools are added to state [03:13] wallyworld: yes but, I didn't know it was called .Add [03:13] but you commented it out so you must have ? [03:14] in that place [03:14] anyways, you'll need to talk to john about that todo [03:14] wallyworld: I looked in the server code for where we were handling the http post for tools upload [03:14] as he is making lxd work on provisioned machones [03:14] sure, and toolsstorage.Add() was right there [03:14] wallyworld: it never occurred to me that there would be code in the client that would be identical [03:15] bootstrap is special [03:15] wallyworld: indeed [03:15] it has to have logic built in as there's no server running yet [03:15] so to recap, we'll need to get that todo sorted asap [03:15] i think [03:15] not really [03:16] this is the lxd provider [03:16] we only support localhost [03:16] actually, yeah maybe not [03:16] it's only local yeah [03:16] some day we maybe might support a remote host. Today's not that day :) [03:16] i was confusing the prtovisioner [03:16] yeah, it's confusing [03:17] good that it's fixed, ty [03:17] well, I'll go test it, but I'm pretty certain that fixes it [03:18] yeah, the code looks correct [03:18] there was a suspicious log message on boot that we were saving amd64 tools, even though we were bootstrapping with arm64... and that message changes to arm64 now... but I have to do the full test with upload-tools disabled to know for sure. [03:19] natefinch: maybe not - cloud init gets tools via the controller which acts as a proxy [03:20] so even if upload tools is used, you just trace requests into the controller [03:20] i didn't think of that previously [03:20] so just trace the tools download hander gets [03:28] I'm building on the arm64 machine which is just about the slowest machine in existence, I'm pretty sure [03:29] quick [03:30] someone [03:30] where is the cloudinit data stored [03:30] on the cloud machine [03:30] I need to get it off before the code kills the machine [03:31] dunno, sorry [03:31] menn0: ^^? [03:32] juju brought the machine up but can't ssh in [03:32] axw: hey... [03:33] this may be part your history [03:33] 2016-04-14 03:33:01 DEBUG juju.provider.common bootstrap.go:328 connection attempt for 192.168.100.3 failed: /var/lib/juju/nonce.txt does not exist [03:33] axw: we just bring up a machine with ssh keys [03:33] and I'm guessing that file [03:34] thumper: yes, so we know we've connected to the right machine [03:34] axw: but that file isn't there [03:34] thumper: something's not doing the right thing with cloud-init then I guess? [03:35] axw: know where the cloud init file is? [03:35] thumper: /var/lib/cloud I think? [03:35] * axw rummages [03:35] I have user_data.txt from there [03:35] but it is base64 encoded [03:35] and decoded looks binary [03:35] thumper: there should be a plaintext file nearby [03:36] thumper: instance/cloud-config.txt maybe [03:36] doesn't exist [03:36] oh [03:36] zero bytes [03:37] thumper: interesting, I don't think I've ever seen a zero-byte one. maybe part of the problem. [03:37] hmm... [03:37] I think I'll add logging to the creation of the user data [03:37] Bug #1570175 opened: juju2 kill-controller doesn't work when bootstrap server is unreachable

[03:38] axw: we are this |-| close to having maas2 boot [03:38] thumper: cool :) [03:42] ahh man, the fact we can't cross compile is killing me on this dumb arm64 machine [03:42] my laptop compiles juju in like 10 seconds. This machine takes like 3 minutes [03:43] thumper: sorry, missed your message... I don't know the answer either (cloud-init newbie) [03:43] well, I've added some debugging code to the cloud init renderer [03:44] which I found... [03:44] thumper, axw: would the cloudinit logs give some clues? (/var/log/cloud-init.output and cloud-init.log) [03:44] so I can catch it as it is yaml-ified, before gzip, base64 [03:45] ok, got the cloud init [03:45] but it is kinda big... [03:45] * thumper looks deeper [03:45] menn0 thumper: *is* there a /var/log/cloud-init-output.log? [03:45] I'll look this time [03:45] last time 10 minutes passed and machine was released [03:46] - install -D -m 644 /dev/null '/var/lib/juju/nonce.txt' [03:46] - printf '%s\n' 'user-admin:bootstrap' > '/var/lib/juju/nonce.txt' [03:46] from the local userdata it is sending down [03:46] not much of a nonce :) [03:47] thumper: heh yeah, but it's better than connecting to some random machine on your local network and running juju setup on it :) [03:48] true [03:50] 2016-04-14 03:47:40,431 - __init__.py[WARNING]: Unhandled non-multipart (text/x-not-multipart) userdata: 'H4sIAAAJbogA/+w7aXPbuJLf...' [03:50] WTF... [03:50] basically it stopped cloudinit [03:50] thumper: nice :) [03:50] thumper: so MAAS is just dropping it on the floor? [03:50] * thumper shrugs [03:50] I'll have to dig a bit more [03:51] thumper: people have complained about the nonce.txt file being missing before, I never could repro tho [03:51] maybe it depends on contents/padding/something [04:00] wallyworld: huzzah, success [04:01] great [04:01] after waiting ages for a compile [04:01] wallyworld: well, I kept getting stupid compile errors... finally realized I could just slap a return nil before the code I wanted to avoid [04:07] thumper: anastasiamac_ committed a fix for that maas bug you saw with listing credentials [04:09] wallyworld: hmm, I can bootstrap with juju2 beta 3 on canonistack-lcy01, but not on this new mitaka cloud [04:09] bradm: it may not have the flavours set up correctly? [04:10] juju uses those plus arch to determine the instance id [04:10] and matches it all with simplestreams [04:10] it all needs to match up correctly [04:12] wallyworld: it has flavours, I'm not sure how you have to set them up, we've never done anything about mapping them before [04:12] wallyworld: I can certainly boot instances, although that is done by specifying an image id [04:13] right, it appears the image id selection algorithm fails due to reaons [04:13] reasons [04:13] it would need investigation to see what's wrong with the set up [04:13] I can see all 3 arches in the index.json [04:14] i personally have not internalised the algorithm - would need to trace it all out and see what's needed where [04:14] something must have changed with mitaka [04:14] or juju2 [04:15] hm, thats a point, I should try juju1 on it [04:15] bradm: also anastasiamac_ thinks there could be related (same?) issue that is fixed in beta 4 [04:16] wallyworld: yeah, maybe I should just wait until beta 4 is out [04:16] are we there yet? are we there yet? ;) [04:16] bradm: in maybe 24 hrs tops [04:17] in time for xenial cut off [04:17] wallyworld: awesome. I'll try out juju1 on this to see how it goes, might narrow things down a bit. [04:17] bradm: yes please, that data point would be helpful in case there's still an issue [04:17] juju1 and 2 should behave the same AFAIK [04:18] that bug mentioned above may well be an existing issue [04:18] i don't know the detials [04:18] a newly released mitaka being deployed via unreleased charms and trying to use a beta juju on top of it? nah, couldn't be any problems there. :) [04:18] so we'll take it a step at a time: try juju1, wait till beta4 [04:18] of course not [04:18] what could possibly go wrong [04:19] its a house of unreleased cards [04:19] but thats why we're doing it, to help bash out any bugs early [04:25] wallyworld: aha, it fails in the same way [04:25] \o/ [04:25] not juju2 :-) [04:28] the index.json is pretty simple [04:30] oho [04:30] arch: x86_64 [04:30] why is that there [04:35] whelp, I have a PR up for that fix here: http://reviews.vapour.ws/r/4555/ I haven't been able to successfully change the tests such that they actually fail with the old code, but it's time for bed. [04:40] wallyworld: hah, that was it. for some reason the version of glance-simplestreams-sync we have was canonicalizing the arch from amd64 to x86_64. now I get a different error. :) [04:40] bradm: progress! [04:40] wallyworld: now it wants me to give it a network id [04:41] what is "it"? [04:41] the error from the bootstrap [04:42] i haven't done much with networking, not sure [04:42] Multiple possible networks found, use a Network ID to be more specific. [04:42] that feels like a nova error, though. [04:42] yes it does [04:43] could be propagated back from start instance [04:43] how do you tell juju2 what the default network is, like the network setting in juju1 [04:45] just trying to add it where you define the endpoint [04:47] nope, no go :-/ [04:48] do I just mark the bug as invalid? [04:50] bradm: juju has spaces and things. but if you are asking about config, instead of foo=bar in environments.yaml, you use --config "foo-bar" on the bootstrap cmd line [04:50] juju2 i mean [04:51] if it's a set up issue with openstack, then yeah, bug may be invalid [04:53] wallyworld: thats got it [04:53] wallyworld: at least, its trying to get an address now, so its much further than before. looks like we need a way to set the default network [04:53] that worked? [04:53] ok [04:53] that bit i am not sure of [04:54] bradm: dimiter and andy are the folks to ask about that [04:57] oh boy, not quite. now it errored out with: [04:57] error: flag provided but not defined: --model-config [05:00] bradm: you need upload-tools if you are running from trunk [05:00] wallyworld: that was with upload-tools... [05:00] trying now without [05:01] the error looks like it is because the jujud binary from tools is old [05:05] it seems to be getting further without the upload tools [05:05] oh, still fails. [05:05] https://pastebin.canonical.com/154283/ <- error message [05:09] maybe I should just wait for beta4 [05:09] axw: here's the state part and some of the API work for storing SSH host keys. http://reviews.vapour.ws/r/4586/ [05:09] menn0: cool, looking [05:09] bradm: do you have the latest code checked out? that could explain why upload tools give poor results. also that error - is the auth url correct? just aguess [05:09] wallyworld: nope, I'm just using teh beta3 version from the ppa [05:10] bradm: in that case upload tools does nothing [05:10] wallyworld: that can't be true, I saw very different things [05:10]