[00:00] <wallyworld> ericsnow: has it been like that forever? i didn't think lxc had that issue?
[00:01] <ericsnow> wallyworld: this was just the lxd provider
[00:01] <wallyworld> ok, that makes sense since that provider was re-written from scratch
[00:01] <wallyworld> i didn't think we had this issue in 1.25
[00:02] <wallyworld> with lxc
[00:02] <ericsnow> wallyworld: I'm guessing no-one had tried to use the LXD provider with non-amd64 before
[00:02] <wallyworld> not till now :-)
[00:02] <ericsnow> wallyworld: anyway, I gotta run; natefinch can fill in the rest :)
[00:03] <wallyworld> ok, ty
[00:03] <wallyworld> ttyl
[00:17] <axw> menn0: if you were going the first route, I'd just supply a names.Tag rather than a GlobalEntity
[00:18] <menn0> axw: yep, fair enough
[00:18] <axw> menn0: I'm ambivalent though. I've found having the methods on state make it a bit easier to mock in tests
[00:18] <axw> but at the same time, I don't like throwing it all on the one type
[00:18] <menn0> that ship has sailed I think :)
[00:19] <axw> menn0: :)
[00:19] <menn0> but I guess I don't have to make it worse
[00:21] <menn0> axw: you're right about mocking in tests though... given that we only have concrete Machines I'm setting myself up for difficult testing if I add methods to Machine
[00:21] <menn0> difficult / dumb
[00:22] <axw> menn0: some packages go to lengths to mock Machine out too, but it is a PITA
[00:23] <menn0> axw: especially for something so simple
[00:23] <axw> menn0: not sure if you've looked at state/volume.go or state/filesystem.go, but this is why I put everything at the top level
[00:23] <axw> it made testing much easier
[00:24] <axw> menn0: I ended up having to have some methods on Volume/Filesystem to support, e.g. StatusSetter/StatusGetter
[00:24] <menn0> yep, makes sense
[00:31] <rick_h_> axw: howdy, wanted to chat on the model statuses and get your thought on something
[00:31] <rick_h_> axw: since the end state is archived, what about destroying is archiving?
[00:32] <axw> rick_h_: it did occur to me. I would say it's not really archiving at that point. It's destroying the resources within the model, it's just the the model docs that are archived
[00:32] <rick_h_> axw: right, but it's working toward the archived state. And it's not used elsewhere because of that idea
[00:33] <rick_h_> axw: I guess it's not quite 'factual' but as far as state transitions it goes through a few moving bits on the way to archived. Picking any one points out a bit of what's up?
[00:34] <axw> rick_h_: so I kinda agree that you should have through "archiving" to get to "archived", but at the same time I don't think "archiving" implies that anything is being cleaned up
[00:34] <rick_h_> axw: do you have a link to the bug there? I wanted to go back and refer to the list you had but I'm failing to see the bug
[00:34] <axw> but more ... put away
[00:34] <axw> 1 sec
[00:35] <axw> rick_h_: https://bugs.launchpad.net/bugs/1534627. I've currently got it as "active", "destroying", "archived" in my branch
[00:35] <mup> Bug #1534627: Destroyed models still show up in list-models <2.0-count> <conjure> <juju-release-support> <juju-core:In Progress by axwalk> <https://launchpad.net/bugs/1534627>
[00:36] <rick_h_> axw: so what caused us to keep it around by default anyway?
[00:36] <rick_h_> axw: vs a cleanup all the wya and removal?
[00:36] <axw> rick_h_: I don't know, that predates my involvement. thumper?
[00:36] <rick_h_> axw: I mean it seems useful, but it seems like the exception to the rule. A destroy-model --archive that has this seems like the cleaner wa
[00:37] <axw> it definitely is an exception to the rule
[00:37] <rick_h_> I guess I assumed we had a good reason from some stakeholder but I'm coming up empty for making this the default behavior.
[00:37] <rick_h_> wallyworld: ^
[00:38] <wallyworld> rick_h_: predates me too
[00:38] <thumper> hmm...
[00:38] <axw> rick_h_: it doesn't seem useful to me, TBH. I'd prefer to just lean on external logging to keep records
[00:38] <thumper> was there mostly so we don't suddenly error
[00:38] <rick_h_> axw: witht he rsyslog logging work going in I'm +1 to that idea
[00:38] <thumper> because the way people watch an environment being destroyed is status
[00:39] <thumper> if you had "watch juju status" running
[00:39] <thumper> you would watch the services go down
[00:39] <thumper> machines being removed
[00:39] <rick_h_> thumper: oic, so this was so that you could see it die vs go away and then if it failed to destroy you had no good way to get at it?
[00:39] <thumper> then "error: unknown environment"
[00:39] <thumper> mostly so you could watch it die without errors
[00:40] <thumper> and grab logs... maybe
[00:40] <rick_h_> axw: can you investigate if we can just destroy in a clean way and do away with the kept around models at all please?
[00:40] <rick_h_> axw: I feel like we're chasing the wrong end of the problem atm
[00:40] <wallyworld> rick_h_: our general pattern has been to shepard entities through dying -> dead. until they are reaped by a cleaner, they will still be there
[00:41] <axw> rick_h_: sure
[00:41] <rick_h_> wallyworld: right, so can we do that better vs keep things around like this?
[00:41] <thumper> the 24 hours thing was entirely arbitrary
[00:41] <thumper> and something I picked out of the air
[00:41] <thumper> we could have a sensible default...
[00:41] <thumper> of a much smaller number
[00:41] <rick_h_> thumper: yea, I mean can we just watch it until it's dead cleanly and then remove it right away?
[00:41] <wallyworld> rick_h_: oh, i didn't realise they were kept for 24hrs
[00:41] <rick_h_> e.g. check every 5s or something?
[00:41] <wallyworld> rick_h_: i thought they were removed immediately once dead, sorry
[00:42] <rick_h_> I'm all for giving the user confidence/observability in what they're doing
[00:42] <thumper> it is the client going "juju status"
[00:42] <thumper> and us wanting to say "environment is dead"
[00:42] <thumper> rather than "error, you suck"
[00:42] <rick_h_> right, so at some point it fails that "model is not available" or the like
[00:42] <thumper> eventually
[00:42] <rick_h_> you asked for it to go away, what do you expect?
[00:42] <thumper> right now, 24 hours later
[00:42] <rick_h_> yea, that's gotta go imo
[00:42] <thumper> this is the job of the undertaker
[00:42] <rick_h_> if we can tell from status it's dead, then we should be able to reap it right then and there
[00:43] <wallyworld> +1
[00:43] <rick_h_> and if status says it's not dead, then we don't reap it and you can get at it and diagnose
[00:43] <wallyworld> that's ow i thought it worked
[00:43] <rick_h_> heh no
[00:43] <wallyworld> didn't realise models were special
[00:43] <rick_h_> so we were looking to do all this renaming and such which :/
[00:43] <rick_h_> little snowflakes in the wind :)
[00:45] <thumper> oh shit
[00:45] <thumper> I think I need to work out this juju 2.0 cli thing
[00:45]  * thumper needs to add a maas cloud
[00:45] <thumper> perhaps after a dog walk
[00:45]  * rick_h_ grumbles
[00:45] <rick_h_> thumper: on that please fix the whole maas as a cloud vs not needing credentials/etc kthx
[00:46] <thumper> wat?
[00:46]  * thumper has no idea about that
[00:46] <axw> rick_h_: as in maas showing up in list-clouds?
[00:46] <rick_h_> thumper: there's a thing in that maas setup that's confusing users
[00:46] <thumper> maas is a cloud isn't it?
[00:46] <axw> maas is a type of cloud
[00:46] <rick_h_> axw: yea, and if you do add it and then you try to add-credential it tells you maas does't need credentials
[00:46] <axw> wat
[00:46] <rick_h_> but then users don't know how to use it since it's different
[00:47] <thumper> wat?
[00:47]  * redir goes eod
[00:47] <redir> see you tomorrow juju-dev
[00:47] <rick_h_> night redir
[00:47] <axw> rick_h_: I think that might be covered by anastasiamac_'s branch, http://reviews.vapour.ws/r/4573/
[00:47] <axw> night redir
[00:47] <anastasiamac_> rick_h_: iwat axw said - it's beeing fixed ^^
[00:47] <anastasiamac_> (landing, really)
[00:48] <anastasiamac_> thumper: about cloud vs cloud type, see https://bugs.launchpad.net/juju-core/+bug/1564054
[00:48] <mup> Bug #1564054: lxd, maas and manual do not make sense in list-clouds <juju-release-support> <juju-core:In Progress by anastasia-macmood> <https://launchpad.net/bugs/1564054>
[00:49] <wallyworld> rick_h_: what do you mean? juju add-credential maas works fine
[00:49] <rick_h_> wallyworld: hmm, a user was hitting it and gave me a pastbin the other day
[00:50] <wallyworld> rick_h_: that would have been an old beta
[00:50] <rick_h_> wallyworld: I'm trying to find it, they got an error out of juju along the lines of "xxx does not need a credential"
[00:50] <wallyworld> this is all wip
[00:50] <thumper> :)
[00:50] <rick_h_> heh
[00:50] <wallyworld> it used to be that
[00:50] <wallyworld> we have been delivering things as threy are finished in each beta
[00:50] <rick_h_> ah ok
[00:50] <rick_h_> never mind then
[00:50] <wallyworld> add credentials came late in the beta cycle
[00:51] <wallyworld> rick_h_: stakeholders want maas included in list clouds
[00:51] <rick_h_> wallyworld: did we have plans for an add-cloud as wel?
[00:51] <wallyworld> that's why we did it we know maas is not a cloud
[00:51] <rick_h_> wallyworld: well, maas clouds need to be there. The trouble is the lack of an add-cloud command for now
[00:51] <anastasiamac_> wallyworld: a user would hit a "boom" if they specify cloud as part of command arguments..
[00:51] <rick_h_> wallyworld: yea, I was one of those, but mark is right
[00:51] <axw> rick_h_: we have add-cloud, but not interactive
[00:51] <rick_h_> anastasiamac_: wallyworld so I +1 going with Mark's feedback there
[00:51] <rick_h_> axw: right, interactive is what I'm thinking
[00:51] <axw> you just point at a YAML file
[00:52] <rick_h_> axw: not for 2.0 but we should add it down the road
[00:52] <wallyworld> rick_h_: will you break the news to adam?
[00:52] <axw> it would be helpful
[00:52] <rick_h_> axw: right, but folks are confused as to what goes in the clousd.yaml, the credentials, and the config
[00:52]  * axw nods
[00:52] <rick_h_> wallyworld: stokachu?
[00:52] <wallyworld> yeah
[00:52] <wallyworld> he needs maas in list clouds for his app
[00:52] <rick_h_> wallyworld: heh ok, will do
[00:53] <rick_h_> wallyworld: I'm still -1 on the maas:/ special thing where you don't have to add it to the clouds.yaml
[00:53] <wallyworld> rick_h_: that was also well received and asked for by users :-)
[00:54] <rick_h_> wallyworld: bah, it just adds a 'differnet way to do it' that's special and unique and causes confusion folks have to look it upo
[00:54] <wallyworld> rick_h_: you mean like lxd :-P
[00:54] <wallyworld> there's no lxd in clouds.yaml also
[00:55] <rick_h_> wallyworld: heh
[00:55] <thumper> ah crap...
[00:56] <wallyworld> rick_h_: the use case driving this is - people want to know what they can put with juju bootstrap <controllername> <what goes here>
[00:56] <thumper> I don't think my 1gb kvm maas instances are up to running juju are they?
[00:56] <thumper> wallyworld: remember when I said the whole cloud credential spec would be a big pile of work?
[00:56] <rick_h_> wallyworld: yea...but they can't do maas without config info (in this case the IP address) so we've changed the bootstap command to make it work
[00:58] <wallyworld> rick_h_: altered slightly yeah, but still inituitive imo
[00:58]  * thumper goes to walk the dog and get away from the copmuter a bit
[00:58] <wallyworld> thumper: oh i know
[00:58] <wallyworld> i never doubted it
[00:59] <wallyworld> we piled in so much stuff in such a short time, and delivered incrementally over several betas and got beat up when it wasn't all there beta 1 :-/
[01:09] <rick_h_> wallyworld: no one got beat up :P
[01:09] <rick_h_> wallyworld: I couldn't reach you!
[01:09] <wallyworld> lol
[01:09] <wallyworld> rick_h_: not beat up directly
[01:10] <rick_h_> :P
[01:10] <wallyworld> , more complaints :-)
[01:15] <rick_h_> wallyworld: we just had an opinion that add-credential should have come first :P
[01:15] <rick_h_> not complaints, opinions...I hear it's like something else everyone has :)
[01:16] <wallyworld> rick_h_: sure, but it was harder to add and was icing - we needed the core functionality with credentials added by hand. having fancy add credentials with nothing working would not have been cool
[01:16] <rick_h_> wallyworld: understand completely...after I stopped and thought about it :P
[01:16] <wallyworld> :-)
[01:17] <wallyworld> rick_h_: next time we'll start the work at the beginning of the cycle, not a month or 2 out :-)
[01:17] <rick_h_> wallyworld: psh, don't go changing everything on me
[01:17] <wallyworld> great, now i've got that song stuck in my head
[01:17] <rick_h_> lol, glad to be of service
[01:18] <wallyworld> i love you just the way you are.... ta de dum
[01:43] <wallyworld> axw: got 5 mintes, standup?
[01:43] <axw> wallyworld: sure, brt
[01:50] <bradm> should juju2 be able to bootstrap against a private openstack cloud? I'm getting:
[01:50] <bradm> 2016-04-14 01:30:25 ERROR cmd supercommand.go:448 failed to bootstrap model: model "admin" of type openstack does not support instances running on "amd64"
[01:50] <bradm> my index.json does list 14.04 images...
[02:09] <bradm> ah, when I don't use --upload-tools it gives:
[02:09] <bradm> 2016-04-14 02:05:56 ERROR cmd supercommand.go:448 failed to bootstrap model: cannot start bootstrap instance: no "trusty" images in bootstack-canonistack-bos01 matching instance types [m1.small m1.medium m1.large m1.xlarge]
[02:20] <thumper> wallyworld: ping
[02:21] <wallyworld> wot
[02:22] <wallyworld> thumper: ?
[02:22] <thumper> wallyworld: got a few minutes to chat?
[02:22] <wallyworld> sure
[02:23] <wallyworld> bradm: juju 2 should work similar to juju 1 for private clouds
[02:23] <wallyworld> it's all about getting the correct streams metadata which can be tricky
[02:23] <bradm> wallyworld: just filed LP#1570162 about it
[02:24] <wallyworld> but if it works for juju 1 it should work for juju 2
[02:24] <wallyworld> ok, we'll look at the bug
[02:24] <bradm> let me know if you need any more info about it
[02:24] <thumper> wallyworld: 1:1 ?
[02:24] <wallyworld> yup
[02:24] <bradm> wallyworld: this is mitaka on trusty, and it has different arch compute nodes too
[02:25] <bradm> grabbing some lunch, back in a while
[02:25] <mup> Bug #1570162 opened: juju2 openstack private cloud cannot start bootstrap instance <canonical-bootstack> <juju-core:New> <https://launchpad.net/bugs/1570162>
[02:45] <mwhudson> wallyworld: juju-mongo-tools3.2 just got accepted \o/
[02:45]  * mwhudson runs away for a bit
[02:45] <wallyworld> mwhudson: awesome
[03:00] <menn0> axw: I was thinking that the machiner would send the SSH host key to the state server (via the machiner facade)
[03:01] <axw> menn0: sounds fine to me
[03:01] <menn0> axw: seem reasonable?
[03:04] <natefinch> wallyworld: btw, this is the fix for the lxd arch bug that I'm pretty sure works: https://github.com/juju/juju/pull/5116/files. But I still can't quite fully test due to not being able to fully disable upload-tools.. I haven't figured out how to disable saving the tools (I've found a couple likely spots and commented out code, but to no avail).
[03:04] <wallyworld> ok, i'll look at pr, thanks
[03:05] <wallyworld> there's a toolsstorage that manages the tools in gridfs
[03:07] <natefinch> wallyworld: I commented out this whole loop: https://github.com/juju/juju/blob/master/apiserver/tools.go#L262 which looks like the place where we store the tools after uploading them... but it didn't seems to change anything
[03:09] <wallyworld> natefinch: why not just make storage.Add() itself a no op
[03:10] <wallyworld> natefinch: anyway, here's what you wnated
[03:10] <wallyworld> fetchAndCacheTools
[03:10] <wallyworld> or maybe not, that is when we download then from streams
[03:11] <wallyworld> it's in jujud/bootstrap.go
[03:11] <wallyworld> 		logger.Debugf("Adding tools: %v", toolsVersion)
[03:11] <wallyworld> 		if err := toolstorage.Add(bytes.NewReader(data), metadata); err != nil {
[03:11] <wallyworld> i just searched for usages of toolsstorage.Add()
[03:11] <wallyworld> there's only a few to check
[03:13] <natefinch> wallyworld: thanks... I really wasn't sure what to search for
[03:13] <wallyworld> just the .Add()
[03:13] <wallyworld> and see what calls it
[03:13] <wallyworld> as that's where tools are added to state
[03:13] <natefinch> wallyworld: yes but, I didn't know it was called .Add
[03:13] <wallyworld> but you commented it out so you must have ?
[03:14] <wallyworld> in that place
[03:14] <wallyworld> anyways, you'll need to talk to john about that todo
[03:14] <natefinch> wallyworld: I looked in the server code for where we were handling the http post for tools upload
[03:14] <wallyworld> as he is making lxd work on provisioned machones
[03:14] <wallyworld> sure, and toolsstorage.Add() was right there
[03:14] <natefinch> wallyworld: it never occurred to me that there would be code in the client that would be identical
[03:15] <wallyworld> bootstrap is special
[03:15] <natefinch> wallyworld: indeed
[03:15] <wallyworld> it has to have logic built in as there's no server running yet
[03:15] <wallyworld> so to recap, we'll need to get that todo sorted asap
[03:15] <wallyworld> i think
[03:15] <natefinch> not really
[03:16] <natefinch> this is the lxd provider
[03:16] <natefinch> we only support localhost
[03:16] <wallyworld> actually, yeah maybe not
[03:16] <wallyworld> it's only local yeah
[03:16] <natefinch> some day we maybe might support a remote host. Today's not that day :)
[03:16] <wallyworld> i was confusing the prtovisioner
[03:16] <natefinch> yeah, it's confusing
[03:17] <wallyworld> good that it's fixed, ty
[03:17] <natefinch> well, I'll go test it, but I'm pretty certain that fixes it
[03:18] <wallyworld> yeah, the code looks correct
[03:18] <natefinch> there was a suspicious log message on boot that we were saving amd64 tools, even though we were bootstrapping with arm64... and that message changes to arm64 now... but I have to do the full test with upload-tools disabled to know for sure.
[03:19] <wallyworld> natefinch: maybe not - cloud init gets tools via the controller which acts as a proxy
[03:20] <wallyworld> so even if upload tools is used, you just trace requests into the controller
[03:20] <wallyworld> i didn't think of that previously
[03:20] <wallyworld> so just trace the tools download hander gets
[03:28] <natefinch> I'm building on the arm64 machine which is just about the slowest machine in existence, I'm pretty sure
[03:29] <thumper> quick
[03:30] <thumper> someone
[03:30] <thumper> where is the cloudinit data stored
[03:30] <thumper> on the cloud machine
[03:30] <thumper> I need to get it off before the code kills the machine
[03:31] <natefinch> dunno, sorry
[03:31] <thumper> menn0: ^^?
[03:32] <thumper> juju brought the machine up but can't ssh in
[03:32] <thumper> axw: hey...
[03:33] <thumper> this may be part your history
[03:33] <thumper> 2016-04-14 03:33:01 DEBUG juju.provider.common bootstrap.go:328 connection attempt for 192.168.100.3 failed: /var/lib/juju/nonce.txt does not exist
[03:33] <thumper> axw: we just bring up a machine with ssh keys
[03:33] <thumper> and I'm guessing that file
[03:34] <axw> thumper: yes, so we know we've connected to the right machine
[03:34] <thumper> axw: but that file isn't there
[03:34] <axw> thumper: something's not doing the right thing with cloud-init then I guess?
[03:35] <thumper> axw: know where the cloud init file is?
[03:35] <axw> thumper: /var/lib/cloud I think?
[03:35]  * axw rummages
[03:35] <thumper> I have user_data.txt from there
[03:35] <thumper> but it is base64 encoded
[03:35] <thumper> and decoded looks binary
[03:35] <axw> thumper: there should be a plaintext file nearby
[03:36] <axw> thumper: instance/cloud-config.txt maybe
[03:36] <thumper> doesn't exist
[03:36] <thumper> oh
[03:36] <thumper> zero bytes
[03:37] <axw> thumper: interesting, I don't think I've ever seen a zero-byte one. maybe part of the problem.
[03:37] <thumper> hmm...
[03:37] <thumper> I think I'll add logging to the creation of the user data
[03:37] <mup> Bug #1570175 opened: juju2 kill-controller doesn't work when bootstrap server is unreachable <canonical-bootstack> <juju-core:New> <https://launchpad.net/bugs/1570175>
[03:38] <thumper> axw: we are this |-| close to having maas2 boot
[03:38] <axw> thumper: cool :)
[03:42] <natefinch> ahh man, the fact we can't cross compile is killing me on this dumb arm64 machine
[03:42] <natefinch> my laptop compiles juju in like 10 seconds.  This machine takes like 3 minutes
[03:43] <menn0> thumper: sorry, missed your message... I don't know the answer either (cloud-init newbie)
[03:43] <thumper> well, I've added some debugging code to the cloud init renderer
[03:44] <thumper> which I found...
[03:44] <menn0> thumper, axw: would the cloudinit logs give some clues? (/var/log/cloud-init.output and cloud-init.log)
[03:44] <thumper> so I can catch it as it is yaml-ified, before gzip, base64
[03:45] <thumper> ok, got the cloud init
[03:45] <thumper> but it is kinda big...
[03:45]  * thumper looks deeper
[03:45] <axw> menn0 thumper: *is* there a /var/log/cloud-init-output.log?
[03:45] <thumper> I'll look this time
[03:45] <thumper> last time 10 minutes passed and machine was released
[03:46] <thumper> - install -D -m 644 /dev/null '/var/lib/juju/nonce.txt'
[03:46] <thumper> - printf '%s\n' 'user-admin:bootstrap' > '/var/lib/juju/nonce.txt'
[03:46] <thumper> from the local userdata it is sending down
[03:46] <thumper> not much of a nonce :)
[03:47] <axw> thumper: heh yeah, but it's better than connecting to some random machine on your local network and running juju setup on it :)
[03:48] <thumper> true
[03:50] <thumper> 2016-04-14 03:47:40,431 - __init__.py[WARNING]: Unhandled non-multipart (text/x-not-multipart) userdata: 'H4sIAAAJbogA/+w7aXPbuJLf...'
[03:50] <thumper> WTF...
[03:50] <thumper> basically it stopped cloudinit
[03:50] <axw> thumper: nice :)
[03:50] <axw> thumper: so MAAS is just dropping it on the floor?
[03:50]  * thumper shrugs
[03:50] <thumper> I'll have to dig a bit more
[03:51] <axw> thumper: people have complained about the nonce.txt file being missing before, I never could repro tho
[03:51] <axw> maybe it depends on contents/padding/something
[04:00] <natefinch> wallyworld: huzzah, success
[04:01] <wallyworld> great
[04:01] <wallyworld> after waiting ages for a compile
[04:01] <natefinch> wallyworld: well, I kept getting stupid compile errors... finally realized I could just slap a return nil before the code I wanted to avoid
[04:07] <wallyworld> thumper: anastasiamac_ committed a fix for that maas bug you saw with listing credentials
[04:09] <bradm> wallyworld: hmm, I can bootstrap with juju2 beta 3 on canonistack-lcy01, but not on this new mitaka cloud
[04:09] <wallyworld> bradm: it may not have the flavours set up correctly?
[04:10] <wallyworld> juju uses those plus arch to determine the instance id
[04:10] <wallyworld> and matches it all with simplestreams
[04:10] <wallyworld> it all needs to match up correctly
[04:12] <bradm> wallyworld: it has flavours, I'm not sure how you have to set them up, we've never done anything about mapping them before
[04:12] <bradm> wallyworld: I can certainly boot instances, although that is done by specifying an image id
[04:13] <wallyworld> right, it appears the image id selection algorithm fails due to reaons
[04:13] <wallyworld> reasons
[04:13] <wallyworld> it would need investigation to see what's wrong with the set up
[04:13] <bradm> I can see all 3 arches in the index.json
[04:14] <wallyworld> i personally have not internalised the algorithm - would need to trace it all out and see what's needed where
[04:14] <bradm> something must have changed with mitaka
[04:14] <bradm> or juju2
[04:15] <bradm> hm, thats a point, I should try juju1 on it
[04:15] <wallyworld> bradm: also anastasiamac_ thinks there could be related (same?) issue that is fixed in beta 4
[04:16] <bradm> wallyworld: yeah, maybe I should just wait until beta 4 is out
[04:16] <bradm> are we there yet?  are we there yet? ;)
[04:16] <wallyworld> bradm: in maybe 24 hrs tops
[04:17] <wallyworld> in time for xenial cut off
[04:17] <bradm> wallyworld: awesome.  I'll try out juju1 on this to see how it goes, might narrow things down a bit.
[04:17] <wallyworld> bradm: yes please, that data point would be helpful in case there's still an issue
[04:17] <wallyworld> juju1 and 2 should behave the same AFAIK
[04:18] <wallyworld> that bug mentioned above may well be an existing issue
[04:18] <wallyworld> i don't know the detials
[04:18] <bradm> a newly released mitaka being deployed via unreleased charms and trying to use a beta juju on top of it?  nah, couldn't be any problems there. :)
[04:18] <wallyworld> so we'll take it a step at a time: try juju1, wait till beta4
[04:18] <wallyworld> of course not
[04:18] <wallyworld> what could possibly go wrong
[04:19] <bradm> its a house of unreleased cards
[04:19] <bradm> but thats why we're doing it, to help bash out any bugs early
[04:25] <bradm> wallyworld: aha, it fails in the same way
[04:25] <wallyworld> \o/
[04:25] <wallyworld> not juju2 :-)
[04:28] <bradm> the index.json is pretty simple
[04:30] <bradm> oho
[04:30] <bradm> arch: x86_64
[04:30] <bradm> why is that there
[04:35] <natefinch> whelp, I have a PR up for that fix here: http://reviews.vapour.ws/r/4555/  I haven't been able to successfully change the tests such that they actually fail with the old code, but it's time for bed.
[04:40] <bradm> wallyworld: hah, that was it.  for some reason the version of glance-simplestreams-sync we have was canonicalizing the arch from amd64 to x86_64.  now I get a different error. :)
[04:40] <wallyworld> bradm: progress!
[04:40] <bradm> wallyworld: now it wants me to give it a network id
[04:41] <wallyworld> what is "it"?
[04:41] <bradm> the error from the bootstrap
[04:42] <wallyworld> i haven't done much with networking, not sure
[04:42] <bradm> Multiple possible networks found, use a Network ID to be more specific.
[04:42] <bradm> that feels like a nova error, though.
[04:42] <wallyworld> yes it does
[04:43] <wallyworld> could be propagated back from start instance
[04:43] <bradm> how do you tell juju2 what the default network is, like the network setting in juju1
[04:45] <bradm> just trying to add it where you define the endpoint
[04:47] <bradm> nope, no go :-/
[04:48] <bradm> do I just mark the bug as invalid?
[04:50] <wallyworld> bradm: juju has spaces and things. but if you are asking about config, instead of foo=bar in environments.yaml, you use --config "foo-bar" on the bootstrap cmd line
[04:50] <wallyworld> juju2 i mean
[04:51] <wallyworld> if it's a set up issue with openstack, then yeah, bug may be invalid
[04:53] <bradm> wallyworld: thats got it
[04:53] <bradm> wallyworld: at least, its trying to get an address now, so its much further than before.  looks like we need a way to set the default network
[04:53] <wallyworld> that worked?
[04:53] <wallyworld> ok
[04:53] <wallyworld> that bit i am not sure of
[04:54] <wallyworld> bradm: dimiter and andy are the folks to ask about that
[04:57] <bradm> oh boy, not quite.  now it errored out with:
[04:57] <bradm> error: flag provided but not defined: --model-config
[05:00] <wallyworld> bradm: you need upload-tools if you are running from trunk
[05:00] <bradm> wallyworld: that was with upload-tools...
[05:00] <bradm> trying now without
[05:01] <wallyworld> the error looks like it is because the jujud binary from tools is old
[05:05] <bradm> it seems to be getting further without the upload tools
[05:05] <bradm> oh, still fails.
[05:05] <bradm> https://pastebin.canonical.com/154283/ <- error message
[05:09] <bradm> maybe I should just wait for beta4
[05:09] <menn0> axw: here's the state part and some of the API work for storing SSH host keys. http://reviews.vapour.ws/r/4586/
[05:09] <axw> menn0: cool, looking
[05:09] <wallyworld> bradm: do you have the latest code checked out? that could explain why upload tools give poor results. also that error - is the auth url correct? just aguess
[05:09] <bradm> wallyworld: nope, I'm just using teh beta3 version from the ppa
[05:10] <wallyworld> bradm: in that case upload tools does nothing
[05:10] <bradm> wallyworld: that can't be true, I saw very different things
[05:10] <wallyworld> ok, so it grabs the first juju binary from the search path and uses that
[05:10] <menn0> axw: next up... some routines in juju/utils for parsing host key files and generating known hosts files
[05:11] <wallyworld> bradm: or builds from source, but you don't have source checked out
[05:11] <bradm> wallyworld: and what auth url do you mean?  the endpoint in the clouds.yaml
[05:11] <wallyworld> bradm: so there may be another juju in the path
[05:11] <wallyworld> bradm: for openstack, yeas i think so
[05:11] <bradm> wallyworld: there's no juju source on this box at all
[05:11] <bradm> wallyworld: there is juju 1.25.5
[05:12] <wallyworld> bradm: so in that case upload tools is using those binaries
[05:12] <wallyworld> which explains a lot
[05:12] <wallyworld> the --model-config unknown etc
[05:12] <bradm> I used juju 1.25.5 to deploy openstack
[05:12] <wallyworld> sure but you said you wer eusing juju2 from a ppa
[05:12] <bradm> yeah, juju2 from ppa, juju1 from ppa
[05:12] <wallyworld> so if you us a juju2 client and upload tools picks the 1.25 binaries it wil screw up
[05:13] <wallyworld> don't use upload tools please :-)
[05:13] <wallyworld> unless you are a developer and have source
[05:13] <bradm> righto.
[05:14] <bradm> I'm just trying different things to work out what its doing :)
[05:14] <wallyworld> bradm: so now it may be that keystone is messing up
[05:14] <wallyworld> bradm: i would love upload tools to be removed tbh
[05:15] <wallyworld> it causes too many issues unless used under strict conditions
[05:15] <bradm> I'm sure we've had to use it to fix things in the past
[05:15] <bradm> but yeah, if its no longer of use
[05:15] <wallyworld> with source yes, or custom binaries put in the right place
[05:15] <wallyworld> it has a use, but yu need to be careful
[05:16] <bradm> wallyworld: I've been using juju since the python days, I'm sure I've got all sorts of redundant things in my brain about it :)
[05:16] <wallyworld> and in this case, it caused "weird" errors until i found out you didb't have source code and had 2 versins etc
[05:16] <wallyworld> bradm: understood. you deserve a medal :-)
[05:16] <wallyworld> for consuming all of our bugs for so long
[05:17] <mup> Bug #1570162 changed: juju2 openstack private cloud cannot start bootstrap instance <canonical-bootstack> <juju-core:Invalid> <https://launchpad.net/bugs/1570162>
[05:20] <bradm> oh my
[05:20] <bradm> that error message really is true, I can't reach the keystone IP from a VM
[05:20] <wallyworld> at least juju is not lying :-)
[05:21] <bradm> juju would never lie to us, would it?
[05:21] <wallyworld> *never*
[05:24] <bradm> missing a route.
[05:24] <axw> menn0: code looks good, but I have a question about the structure of the keys
[05:24] <axw> (in RB)
[05:26] <menn0> axw: can the SSH server really have multiple host keys for a given algorithm?
[05:26] <axw> menn0: I think so, but I'll test to make sure
[05:27] <menn0> axw: I just checked the man page... I don't think it's possible
[05:28] <axw> menn0: sshd_config? what part?
[05:28] <menn0> axw: man sshd
[05:29] <bradm> juju2 likes leaving secgroups around, just hit a quota limit
[05:29] <menn0> axw: the "-h host_key_file" part and the FILE section
[05:29] <menn0> FILES
[05:29] <menn0> axw: there's one host key file per key/algorithm type
[05:30] <axw> menn0: AFAIK they're just the default ones, referenced by /etc/ssh/sshd_config
[05:31] <menn0> axw: the wording in man sshd_config is more vague about it
[05:32] <menn0> axw: I guess it's safer to use []string (with no real downside)
[05:32] <menn0> axw: good catch
[05:32] <menn0> axw: in that case, I'll just store the key files in state verbatim (they're one line each)
[05:33] <menn0> axw: and handle the parsing and reformatting in the client when it generates the bespoke known_hosts file
[05:33] <axw> menn0: SGTM. FWIW, starting sshd with multiple RSA keys works fine
[05:33]  * axw nods
[05:33] <menn0> axw: you just tried it?
[05:33] <axw> menn0: yep
[05:33] <axw> menn0: BTW, there's a function that you can use to parse the public keys: https://godoc.org/golang.org/x/crypto/ssh#ParseAuthorizedKey
[05:33] <menn0> axw: ok cool.. that's definitely the right approach then
[05:35] <menn0> axw: good to know... I just found something in juju/utils/ssh which also does it :)
[05:35] <axw> heh
[05:36] <axw> can't have too many
[05:36] <menn0> axw: maybe the keys should be parsed to a (type, keydata, comment) struct and send and stored that way?
[05:36] <bradm> oh, you can just run juju2 enable-ha ?
[05:36] <bradm> er, can't just
[05:36] <menn0> axw: rather than the raw strig
[05:36] <menn0> string
[05:37] <axw> menn0: *shrug* if it's easy enough to do without losing any info, maybe
[05:37] <axw> I'm not sure it's worth the effort tho
[05:39] <axw> IOW, authorized key format is already perfect information, so probably not worth destructuring at that point unless we think we're going to query on the individual fields
[05:41] <menn0> axw: you're right
[05:42] <menn0> axw: what threw me a little was that the known_hosts file doesn't include the comment field on my machine
[05:42] <menn0> axw: so I was thinking it would need to be stripped
[05:42] <menn0> axw: but looking at the docs, it's fine if it's there
[05:42] <menn0> axw: []string it is
[05:43] <axw> menn0: cool. a comment saying that it's in authorized_keys format would be helpful
[05:43] <menn0> axw: yep, will add.
[05:57] <bradm> wallyworld: yeah, this network thing is going to be a blocker.  need a way to set it as a default somewhere.   just setup a multiuser env, tried to boot a VM and it errored out with the same thing about multiple networks
[05:57] <wallyworld> bradm: i recall conversations in this area but not any specifics, not sure of the status
[05:57] <wallyworld> aybe there's a solution already, i just don't know it
[06:09] <wallyworld> axw_: ping
[06:09] <axw_> wallyworld: pong
[06:10] <wallyworld> axw_: stupid quetion of the day, i'll make a dick of myself i'm sure. can look look at line 71 or environ_broker.go in the lxd provider
[06:11] <axw_> wallyworld: yep?
[06:11] <wallyworld> should be finishInstanceConfg()
[06:11] <wallyworld> the arg struct is passed by value
[06:11] <wallyworld> so how will it ever work
[06:11] <axw_> wallyworld: pretty sure InstanceConfig is a pointer
[06:11] <axw_> yep
[06:11] <wallyworld> ah right
[06:12] <wallyworld> yes i missed that
[06:13] <axw_> well shit. I just bootstrapped and it panicked on the server due to "send on closed channel" in the systemd package
[06:17] <wallyworld> yay
[06:20] <menn0> axw_: please take another look at http://reviews.vapour.ws/r/4586/diff/
[06:21] <menn0> axw_: no rush as I'm about to EOD. but if you could look before you finish that would be great
[06:21] <axw_> menn0: no worries, have a nice evening
[06:22] <menn0> axw_, wallyworld : it's feeling like this ssh host key handling issue is going to be easier to lick than it seemed (still a bit to do I realise)
[06:22] <wallyworld> win
[06:30] <axw_> wallyworld: is there an agenda for the sprint yet? formal or informal
[06:30] <axw_> wallyworld: I mean, topic list we're compiing
[06:30] <axw_> compiling*
[06:30] <axw_> wallyworld: CI for storage really needs to happen
[06:30] <wallyworld> axw_: sort of - right as of now, it's digesting the roadmap wish list
[06:30] <axw_> it's been broken in master since last year, in Malta...
[06:31] <axw_> yep
[06:31] <axw_> ok
[06:31] <wallyworld> axw_: agreed about CI for storage. can we discuss in 1:1 tomorrow?
[06:31] <axw_> wallyworld: sure
[07:17] <mwhudson> wallyworld: is there any other packaging stuff juju is waiting on?
[07:17] <mwhudson> other than juju itself ;-p
[07:17] <wallyworld> not that i know of
[07:17] <wallyworld> well do want mongo 3.2 in trusty and wily at some stage
[07:17] <wallyworld> soon hopefully :-)
[07:18] <mwhudson> wallyworld: somehow the deadlines on those don't seem so tight
[07:19] <mwhudson> eg i guess i should backport go 1.6.1 to trusty...
[07:20] <wallyworld> mwhudson: they are not as tight, but we do want a consistent mongo experience across series at some stage
[07:20] <mwhudson> i guess we can find out if the packages build for a start
[07:41] <wallyworld> axw_: could you look at http://reviews.vapour.ws/r/4587/ ? i want to land it because CI is failing with lxd on arm. i have taken nate's work and added tests
[07:42] <wallyworld> i want to try and get this in for beta4
[07:42] <axw_> wallyworld: looking
[07:42] <wallyworld> ta
[07:42] <bradm> I've filed LP#1570219 if anyone who knows more about the networking side of things could take a look, that'd be great, thanks.
[07:44] <axw_> wallyworld: done
[07:44] <wallyworld> ta
[07:47] <mup> Bug #1570216 opened: juju2 not cleaning up nova secgroups with openstack provider <canonical-bootstack> <juju-core:New> <https://launchpad.net/bugs/1570216>
[07:47] <mup> Bug #1570219 opened: juju2 openstack provider setting default network <canonical-bootstack> <juju-core:New> <https://launchpad.net/bugs/1570219>
[07:50] <fwereade__> ashipika, cmars: responded to a couple of bits of review, went into detail with the problems with the idiosyncratic approach to workers; we should probably all talk about this live today
[07:51] <ashipika> fwereade__:  thanks.. it's ok.. i was not going to land this PR as it is.. i still welcome any and all comments, but i'll be breaking it up into smaller managable bits, i expect..
[07:53] <fwereade__> ashipika, cool, thanks, let me know if you want to talk about any of it
[07:53] <ashipika> fwereade__: i expect i will.. many times along the way :)  but this week we have our priorities elsewhere, so this might have to wait a bit.. anyways it needs to land by may, if i understand the timeline..
[07:54] <fwereade__> ashipika, cool
[07:54] <ashipika> fwereade__: and the thing is: there was no spec.. it was just "oh, this needs to work.. "
[07:54] <fwereade__> ashipika, yeah, that was rather my reading of it
[07:55] <ashipika> fwereade__: we kept asking for more details, but nothing came back.. other than DTAG wants rsyslog forwarding
[08:50] <voidspace> babbageclunk: so, in my test for waitForNodeDeployment I'm seeing the same NotFound error
[08:50] <voidspace> babbageclunk: so looks like a straightforward bug
[08:50] <babbageclunk> voidspace: yay!
[08:50] <babbageclunk> voidspace: with my stuff landed now, what should I pick up?
[08:51] <voidspace> babbageclunk: maas2Instance.volumes would be good
[08:51] <babbageclunk> voidspace: ok
[08:51] <voidspace> babbageclunk: there's a list at the top of the status document of tasks
[08:52] <voidspace> babbageclunk: you could take on fixing the behaviour when you run against MAAS2 without the feature flag
[08:52] <voidspace> babbageclunk: currently it just panics
[08:52] <voidspace> babbageclunk: instead we should detect MAAS 2 and exit with an error instead
[08:52] <voidspace> babbageclunk: that's easy enough to do - might be good to get that in first
[08:52] <voidspace> babbageclunk: so attempt to create the controller even without the feature flag
[08:53] <voidspace> babbageclunk: if it succeeds then we're on MAAS 2
[08:53] <voidspace> babbageclunk: if we don't have the feature flag error out with a NotSupported error
[08:55] <babbageclunk> voidspace: yeah, I'll do that first.
[08:56] <mwhudson> wallyworld: juju-mongo* stuff builds on trusty with a bit of flailing for -tools https://launchpad.net/~mwhudson/+archive/ubuntu/devirt/+packages/?field.series_filter=trusty
[08:58] <voidspace> babbageclunk: ah no - my NotFound error is because my test doesn't give the fakeController.Machines method anything to return
[08:58] <voidspace> babbageclunk: so not the same bug...
[09:00] <voidspace> babbageclunk: ooh, see Tim's status update - he did some work for us
[09:00] <voidspace> babbageclunk: and got bootstrap further
[09:01]  * thumper wonders if he is in a different hangout to the others
[09:01] <voidspace> thumper: morning
[09:01] <voidspace> thumper: keen bean
[09:23] <voidspace> babbageclunk: just added a new task to the list - use maas2NetworkInterfaces from StartInstance
[09:23] <voidspace> babbageclunk: should be trivial, but needs a test as well
[09:24] <voidspace> babbageclunk: (a test at the StartInstance level)
[09:52] <voidspace> dimitern: frobware: babbageclunk: http://pastebin.ubuntu.com/15826362/
[09:52] <voidspace> dimitern: frobware: babbageclunk: that's real progress
[09:53] <dimitern> voidspace: awesome!
[09:54] <frobware> voidspace: indeed
[09:54] <dimitern> voidspace: I see you're possibly hitting the same ssh issue I had - my key is ssh-dss, apparently no longer considered secure
[09:55] <dimitern> voidspace: but I found a way around it, if you need
[09:55] <voidspace> dimitern: go ahead
[09:55] <voidspace> dimitern: I thought this was the issue tim had with not being able to ssh in
[09:56] <dimitern> might still be that, but try this:
[09:56] <voidspace> dimitern: can you join us in #maas on canonical
[09:56] <voidspace> dimitern: (as well)
[09:57] <dimitern> voidspace: http://paste.ubuntu.com/15826416/
[09:57] <dimitern> ah, I thought I'm there already
[09:57] <dimitern> voidspace: so replace the IP ranges to match yours - the important bits are the last 4 sections
[09:58] <voidspace> dimitern: thanks
[09:59] <voidspace> dimitern: /etc/ssh/ssh_config
[09:59] <dimitern> voidspace: ~/.ssh/config
[09:59] <voidspace> dimitern: cool, thanks
[09:59] <voidspace> dimitern: can you confirm that we gzip userdata for allenap in #maas on canonical irc
[10:01] <dimitern> voidspace: looking
[10:01] <voidspace> retrying
[10:01] <voidspace> (the bootstrap I mean)
[10:05] <mup> Bug #1570269 opened: state: ensure that Models are always paired with the correct State <tech-debt> <juju-core:Triaged> <https://launchpad.net/bugs/1570269>
[10:05] <babbageclunk> voidspace: awesome
[10:09] <thumper> voidspace: no, that is a different bug
[10:09] <TheMue> morning
[10:09] <thumper> 2016-04-14 03:57:54 ERROR cmd supercommand.go:448 failed to bootstrap model: waited for 10m0s without being able to connect: /var/lib/juju/nonce.txt does not exist
[10:09] <thumper> o/ TheMue
[10:09]  * thumper is outa here
[10:09] <voidspace> thumper: o/
[10:09] <TheMue> n8 thumper
[10:10] <TheMue> :)
[10:32] <mup> Bug #1570285 opened: worker/undertaker: update status with remaining resources <tech-debt> <juju-core:Triaged> <https://launchpad.net/bugs/1570285>
[10:47] <babbageclunk> dimitern: how do I bootstrap to a different region in AWS? If I use --to zone=eu-west-1 I get this: http://pastebin.ubuntu.com/15827282/
[10:47] <babbageclunk> dimitern: also, why do I always need to specify --upload-tools?
[10:47] <mwhudson> babbageclunk: i don't know but eu-west-1 is a region, not a zone
[10:48] <dimitern> babbageclunk: juju bootstrap <controller-name> aws/<region> ...
[10:48] <babbageclunk> mwhudson: Ah, ok - I was following some old docs, I think
[10:48] <dimitern> babbageclunk: since that changed I tend to keep this around for reference: http://paste.ubuntu.com/15127859/
[10:49] <babbageclunk> dimitern: thanks
[10:50] <babbageclunk> dimitern, mwhudson - also, juju help placement lead me astray too (I guess there are people writing new help messages).
[10:51] <dimitern> babbageclunk: docs can be improved indeed, but have you tried 'juju help bootstrap' ?
[10:51] <babbageclunk> dimitern: yup, that was how I got to 'juju help placement'
[10:52] <dimitern> babbageclunk: ah :) I see - "placement" is related to the --to argument
[10:55] <frobware> dimitern: can you drop into the sapphire standup HO?
[10:55] <dimitern> frobware: sure, just a sec
[10:55] <frobware> thx
[10:56] <babbageclunk> dimitern: Right - totally missed the bit at the top of the bootstrap help - was confused because the placement docs matched what I saw in the old docs on the web.
[10:57] <babbageclunk> dimitern: thanks!
[12:23] <voidspace> frobware: dimitern: babbageclunk: http://reviews.vapour.ws/r/4591/
[12:38] <dimitern> voidspace: LGTM
[12:38] <perrito666> bbl
[12:57] <mup> Bug #1570368 opened: juju commands timeout while a bootstrap is in process <conjure> <juju-core:New> <https://launchpad.net/bugs/1570368>
[13:12] <voidspace> dimitern: thanks
[13:12]  * voidspace lunches
[13:13] <mattyw> fwereade_, ping?
[13:16] <mattyw> fwereade_, I have some questions if you can spare 5 minutes?
[13:19] <babbageclunk> voidspace: what kind of error should I return for when the endpoint's MAAS 2 but the flag isn't set? errors.NotSupportedf?
[13:20] <babbageclunk> voidspace: struggling to give an error message that makes sense with " not supported" stuck on the end.
[13:22] <babbageclunk> voidspace: ok "unless the 'maas2' feature flag is set MAAS 2 is"
[13:43] <dimitern> babbageclunk: for those cases there's also a errors.NewNotSupported(nil, fmt.Sprintf("fmt str", args,...)) you can use
[13:43] <babbageclunk> dimitern: great, thanks¬
[13:43] <babbageclunk> oops, !
[14:00] <katco> morning all
[14:02] <natefinch> morning katco
[14:02] <katco> and actually need to reboot... brb
[14:08] <voidspace> babbageclunk: just errors.New and a sensible error message of your choice will be fine
[14:08] <babbageclunk> voidspace: hello
[14:09] <voidspace> babbageclunk: hello
[14:09] <babbageclunk> voidspace: so, obviously that change was trivial
[14:09] <voidspace> babbageclunk: cool
[14:09] <babbageclunk> voidspace: but working out why the tests weren't failing already in the same way wasn't
[14:09] <voidspace> babbageclunk: don't forget to update the status doc
[14:09] <voidspace> babbageclunk: hah
[14:09] <voidspace> babbageclunk: I added the feature flag to our tests at some point
[14:10] <babbageclunk> voidspace: it turns out maas2 just returns some html when you ask for /1.0/version/
[14:10] <babbageclunk> voidspace: rather than 404ing
[14:10] <voidspace> babbageclunk: ah, it used to return nul
[14:10] <voidspace> babbageclunk: they've changed it
[14:11] <voidspace> dimitern: babbageclunk: I'm leaving early today to go to a tatooist
[14:11] <voidspace> dimitern: babbageclunk: then I'm coming back in again later
[14:11] <babbageclunk> voidspace: well, it returns null when you parse it as json
[14:11] <dimitern> voidspace: ok, have phun ;)
[14:11] <voidspace> ah
[14:11] <voidspace> dimitern: I will
[14:11] <voidspace> babbageclunk: that makes sense
[14:11] <voidspace> babbageclunk: well, not returning a 404 doesn't make sense
[14:11] <voidspace> but there you go
[14:12] <babbageclunk> voidspace: have a nice tattoo appointment!
[14:13] <voidspace> babbageclunk: I'm sure I will, not going yet - but soonish
[14:13] <babbageclunk> voidspace: won't forget the doc this time, sorry!
[14:14] <voidspace> heh, np
[14:50] <mup> Bug #1453805 opened: Juju takes more than 20 minutes to enable voting <blocker> <ci> <ensure-availability> <intermittent-failure> <regression> <juju-core:Triaged
[14:50] <mup> by menno.smits> <juju-core 1.23:Fix Released by menno.smits> <juju-core 1.24:Fix Released by menno.smits> <https://launchpad.net/bugs/1453805>
[14:57] <babbageclunk> voidspace: still around?
[14:57] <voidspace> dimitern: ping
[14:57] <voidspace> babbageclunk: yes
[14:57] <voidspace> babbageclunk: I might have found the bug
[14:57] <voidspace> babbageclunk: gomaasapi does base64 encoding for us, and so do we
[14:58] <babbageclunk> voidspace, dimitern, frobware: http://reviews.vapour.ws/r/4595/
[14:58] <rick_h_> voidspace: do you know if frobware is around today?
[14:58] <babbageclunk> voidspace: Oops
[14:58] <voidspace> rick_h_: he was earlier, yes
[14:58] <babbageclunk> voidspace: nice
[14:58] <rick_h_> voidspace: k, ty
[14:58] <babbageclunk> voidspace: Is there an easy way to explore the maas api?
[14:58] <voidspace> babbageclunk: I use the CLI...
[14:59] <babbageclunk> voidspace: ah, I keep forgetting about that.
[14:59] <babbageclunk> voidspace: thanks
[15:00] <babbageclunk> voidspace: the docs are singularly unhelpful.
[15:00] <frobware> rick_h_: yep, here, but IRC dropping out a lot atm
[15:01] <rick_h_> frobware: ah ok, I asked stokachu to shoot you an email on a potential network/bridge issue he was seeing last night
[15:01] <katco> natefinch: standup time
[15:01] <rick_h_> frobware: wanted to let you know I asked him to and I know you've been doing MAAS2/bug stuff but wanted to see if you or something could poke at it and see if it's a bug or working as intended/etc
[15:03] <alexisb> rick_h_, we should have him open a bug so we can get it on the squad board
[15:03] <alexisb> that is where the full team is pulling priority bugs
[15:04] <rick_h_> alexisb: rgr, the question was "is this a bug?" so just wanted to make sure first
[15:06] <frobware> rick_h_: I semi-stalled on an answer to stokachu. tych0 is proposing a patch for the problems discussed in that email. I also owe tych0 a patch too.
[15:06] <katco> natefinch: ping?
[15:06] <rick_h_> frobware: ok, cool. Ignore me then.
[15:07] <natefinch> katco: sorry
[15:07] <natefinch> katco: lost track of time, coming
[15:08] <voidspace> dimitern: babbageclunk: frobware: removing the extra base64 encode from gomaasapi fixes the issue Tim reported this morning
[15:08] <voidspace> and now we die in a new way
[15:10] <frobware> voidspace: that was base64 on base64 then?
[15:10] <dimitern> voidspace: ah, good - too much encoding then :)
[15:11] <voidspace> frobware: yep
[15:11] <tych0> rick_h_: yeah, i know what the issues are with a bridge
[15:11] <tych0> gonna send some patches today
[15:11] <tych0> just need to catch up on email :)
[15:11] <rick_h_> tych0: ok cool, thanks for all the help in figuring it out!
[15:11] <voidspace> dimitern: where is that done - in the cloudinit package?
[15:13] <dimitern> voidspace: in cloudconfig/providerinit IIRC
[15:13] <voidspace> dimitern: you are correct
[15:14] <voidspace> dimitern: I've asked Tim to fix it in gomaasapi
[15:14] <dimitern> voidspace: sweet!
[15:15] <tych0> rick_h_: sure, np
[15:16] <voidspace> dimitern: do we propagate feature flags onto the juju controller machine?
[15:16] <voidspace> we must do
[15:17] <voidspace> however, the issue I'm seeing now kinda implies not
[15:19] <dimitern> voidspace: yeah
[15:19] <dimitern> voidspace: we do
[15:20] <voidspace> ok, kinda hard to see where this "requested map got nil" comes from
[15:20] <voidspace> I'm bootstrapping with debug to see
[15:20] <voidspace> maybe Subnets
[15:21] <dimitern> voidspace: this sounds like a GetMap() failed somewhere
[15:21] <voidspace> heh, possibly from space discoovery
[15:21] <voidspace> dimitern: well yes...
[15:21] <dimitern> but on a jsonobject, not a maasobject
[15:21] <dimitern> i.e. while processing a response
[15:22] <voidspace> that's the error message we usually get hitting a 1.0 endpoint against 2.0
[15:22] <voidspace> but I'm trying to work out where
[15:22] <voidspace> the error message is pointing me to a non existent line in supercommand.go and --debug provided no extra information
[15:23] <voidspace> although the debug line before it is immediately before a call to NewEnviron - which would report that error message if it thought the feature flag wasn't set
[15:24] <voidspace> dimitern: where in juju are feature flags set on the controller machine
[15:24] <voidspace> dimitern: if it's after we attempt to open an environ then we'll fail in this way
[15:24] <voidspace> when running jujud on the controller
[15:24] <dimitern> voidspace: let me check exactly
[15:25] <dimitern> voidspace: cmd/jujud/main_nix.go
[15:25] <voidspace> dimitern: I see a call to SetFlagsFromEnvironment in jujud/main_nix.go
[15:26] <voidspace> dimitern: right, but what puts them in the environment - cloud init?
[15:26] <dimitern> voidspace: no, they are part of the agent config we pass via the userdata
[15:27] <dimitern> voidspace: check also cmd/jujud/agent/machine.go - in the beginning or Run()
[15:27] <dimitern> you could grep for "developer feature flags enabled" in the logs
[15:28] <voidspace> ok
[15:28] <voidspace> dimitern: thanks
[15:29] <alexisb> ericsnow, when you have a moment can you kindly review http://reviews.vapour.ws/r/4583/
[15:44] <ericsnow> alexisb: will do
[15:47] <dimitern> babbageclunk: you've got a review btw
[16:06] <voidspace> dimitern: babbageclunk: as I suspected. With maas2 including babbageclunk's branch *and* my gomaasapi fix, bootstrap now dies with
[16:06] <voidspace> 2016-04-14 16:03:57 ERROR cmd supercommand.go:448 MAAS 2 is not supported unless the 'maas2' feature flag is set
[16:06] <voidspace> dimitern: babbageclunk: so the feature flag isn't being propagated correctly / early enough to the controller machine
[16:07] <dimitern> voidspace: you're not seeing the log saying they're enabled?
[16:09] <voidspace> dimitern: haven't checked yet - we haven't touched that code path though!
[16:09] <voidspace> off to the tatooist
[16:09] <voidspace> will look when I return
[16:09] <dimitern> ok
[16:26] <tych0> rick_h_: jam: frobware: https://github.com/juju/juju/pull/5164
[16:28] <dimitern> tych0: agent.LxdBridge is never ever set
[16:28] <dimitern> tych0: only agent.LxdBridge is
[16:28] <dimitern> if that
[16:28] <tych0> dimitern: sorry, i don't understand?
[16:29] <dimitern> tych0: e.g. here https://github.com/juju/juju/pull/5164/files#diff-7db54798352f1e675c4e2ecba7bc349dR57
[16:29] <dimitern> or the one below
[16:29] <dimitern> in MaintainInstance
[16:30] <tych0> dimitern: you said "agent.LxdBridge is never ever set only agent.LxdBridge is"
[16:30] <frobware> tych0: btw, the merge should be into next
[16:30] <dimitern> agent.LxcBridge is non-empty if explicitly set by a provider - MAAS and EC2 used to do that, but no longer do
[16:30] <dimitern> tych0: sorry :)
[16:31] <dimitern> so 'only agent.LxcBridge is'
[16:32] <dimitern> but as I said, now agent.LxcBridge is no longer set and is always empty
[16:33] <dimitern> the confusion comes from bad naming - agent.LxcBridge should've been called agent.ContainerBridge
[16:33] <dimitern> frobware: gofmt breaks alignment when it finds a blank line
[16:36] <tych0> dimitern: ok. so you're saying we should just delete that entirely and always use lxcbr0? or?
[16:36] <tych0> i don't actually know where that comes from, i just figured it was configuration from the user
[16:36] <dimitern> tych0: yes, I think that's correct
[16:37] <tych0> dimitern: i guess i'm a little gunshy about making that change
[16:37] <tych0> since i don't understand any of this very well :)
[16:37] <dimitern> tych0: long, long ago there was a "network-bridge" setting you could use to override agent.LxcBridge, but it's long gone
[16:38] <voidspace> babbageclunk: have you made much progress on volumes in gomaasapi?
[16:39] <tych0> dimitern: ok. it seems like that should be part of a larger change to get rid of it everywhere else then i guess?
[16:39] <voidspace> babbageclunk: it would be good to let Tim know where you got to in the status doc
[16:39] <tych0> i can drop that patch if you think it doesn't matter though
[16:39] <babbageclunk> voidspace: nope - struggling to understand how the current code works.
[16:39] <dimitern> tych0: so now unless the provider populates ContainerBridgeName in the BootstrapParams passed to providercommon.Bootstrap(), agent.LxcBridge won't be set in the agent config
[16:39] <babbageclunk> voidspace: It seems to rely on attrs that aren't in the 1.9 JSON.
[16:40] <voidspace> babbageclunk: can you write it up in the doc - Tim can look at it or we can feature request the maas guys
[16:40] <babbageclunk> voidspace: writing my own little test harness
[16:40] <voidspace> babbageclunk: cool
[16:40] <voidspace> babbageclunk: maybe there's another api to get the information
[16:44] <mup> Bug #1570473 opened: juju lxd bridge detection fallback is not reliable <conjure> <juju-core:New> <https://launchpad.net/bugs/1570473>
[16:50] <alexisb> ericsnow, or dimitern. or frobware : this is a high priority PR for review today: http://reviews.vapour.ws/r/4598/
[16:50] <ericsnow> alexisb: k
[16:51] <ericsnow> alexisb: already looking at it :)
[16:51] <alexisb> sweet :)
[16:52] <dimitern> alexisb: I've already added comments and discussed a few points with tych0
[16:52] <dimitern> tych0: apart from using the always empty agent.LxdBridge (or agent.LxcBride) - LGTM
[16:53] <tych0> dimitern: yeah. i guess i'm not super comfortable getting rid of that because i don't really know how it works
[16:53] <tych0> it seems like if we want to get rid of it, we should get rid of it everywhere
[16:53] <mup> Bug #1570473 changed: juju lxd bridge detection fallback is not reliable <conjure> <juju-core:New> <https://launchpad.net/bugs/1570473>
[16:53] <dimitern> tych0: sgtm
[16:59] <mup> Bug #1570473 opened: juju lxd bridge detection fallback is not reliable <conjure> <juju-core:New> <https://launchpad.net/bugs/1570473>
[17:00] <ericsnow> tych0: FYI, ship-it
[17:00] <ericsnow> tych0: (with one small comment)
[17:10] <tych0> ericsnow: no, that constant isn't exported in the LXD package; i moved it to lxdclient because we needed it there
[17:11] <ericsnow> tych0: sounds good
[17:11] <tych0> how do i change the branch target?
[17:11] <tych0> seems lik ei might need a new pr?
[17:11] <ericsnow> tych0: of the PR?  yeah, make a new PR and link to the old review request
[17:12] <tych0> ericsnow: ok, cool. and then i'm good to merge right away?
[17:12] <ericsnow> tych0: yep
[17:12] <tych0> ok, cool
[17:13] <tych0> ericsnow: wait, next is older than master?
[17:14] <ericsnow> tych0: no, though it may have temporarily diverged a little
[17:14] <tych0> ok
[17:21] <alexisb> tych0, remind me, what version of lxd did the switch to lxdbr0?
[17:21] <alexisb> was it rc9??
[17:22] <tych0> i think so
[17:22]  * tych0 looks
[17:23] <tych0> yeah
[17:23] <tych0> rc9
[17:24] <perrito666> mm, how long until its morning in nz?
[17:24]  * perrito666 needs a hand from menn0
[17:26] <alexisb> perrito666, you have about 2 hours
[17:26] <perrito666> one of the fun things of this job :p one question I hardly thought I would be making
[17:27] <redir> bbiab
[17:27] <alexisb> and tych0 do you have the link handy for your insights write-up on lxd init and bridge setup?
[17:28] <alexisb> cheryl linked me to is yesterday but now I cant find it :)
[17:28] <alexisb> tych0, nevermind
[17:28] <alexisb> found it
[17:28] <alexisb> sorry
[17:28] <perrito666> agh why are the tests that take the longer the ones that always fail
[17:28] <tych0> alexisb: cool, np
[17:35] <perrito666> dimitern: voidspace can any of you make anything of the first error in https://pastebin.canonical.com/154358/  ?
[17:37] <dimitern> perrito666: looks like map ordering issue?
[17:37] <alexisb> katco, did you add channels to the release notes?
[17:38] <katco> alexisb: no
[17:38] <katco> alexisb: we didn't do the front-end work... did it not make it in there?
[17:38] <alexisb> nope
[17:38] <dimitern> perrito666: in any case, feel free to skip/ignore or just delete this test, as it's no longer relevant - uses state.NetworkInterface which must be removed (no longer used) - just haven't got there yet myself
[17:38] <alexisb> katco, would you be up to adding soemthing?
[17:38] <alexisb> we need it to release
[17:38] <katco> alexisb: yeah adding now
[17:38] <alexisb> thanks
[17:39] <alexisb> heading under "Whats new for beta4" please
[17:42] <katco> alexisb: do you think this should be an overview of channels, or simply a blurb stating that they exist
[17:42] <alexisb> katco, I think an overview would be nice to have
[17:43] <alexisb> so that people know
[17:43] <alexisb> but it doesnt have to be overly detailed
[17:43] <katco> alexisb: ok, i'm going to ping someone from the CS side of things as they are way more familiar
[17:43] <alexisb> fair enough
[17:43] <perrito666> dimitern: tx, Just making sure everything tests properly with mongo3 and was not sure if I should pay attention to that test
[17:52] <katco> alexisb: are you fine with me linking to our already excellent documentation, and then providing info about juju's command line? https://jujucharms.com/docs/devel/authors-charm-store#entities-explained
[17:53] <alexisb> katco, yes that is fine
[17:53] <katco> alexisb: k
[17:59] <perrito666> hey, I suddenly have to go for like an hour, ill be back later, mail me if you need anything
[18:16] <voidspace> perrito666: no idea without digging into it, sorry
[18:28] <perrito666> Voidspace no worries dimitern told me what I needed
[19:42] <mup> Bug # changed: 1450299, 1538303, 1554675, 1556207, 1559099, 1560391, 1564694, 1567017, 1567020, 1568092, 1569982
[19:44] <perrito666> I see some bugs changed
[20:05] <katco> ericsnow: i'm in our 1:1 if you're ready
[20:30] <alexisb> wallyworld, when you are in please ping me
[20:31] <wallyworld> alexisb: give me 5
[20:31] <alexisb> lol
[20:31] <alexisb> it is not urgent
[20:31] <katco> wallyworld: you are a robot, i'm convinced
[20:34] <alexisb> katco, me too
[20:34] <alexisb> convinced that wallyworld is a robot
[20:35] <alexisb> fueled by hoity-toity coffee
[20:35] <redir> wall-eworld
[20:35] <alexisb> lol
[20:35] <alexisb> that was good redir
[20:35] <redir> :)
[20:39] <wallyworld> alexisb: zup?
[20:40] <alexisb> 1x1 HO
[20:42] <wallyworld> ok
[20:42] <mup> Bug # changed: 1426729, 1516668, 1524077, 1533262, 1537620, 1538735, 1543223, 1553272, 1554251, 1554687, 1555083, 1555248, 1556249, 1560201, 1560511, 1560520, 1560531, 1560595, 1560665, 1560667, 1563576, 1563615, 1563628, 1563762, 1563843, 1563845, 1563853, 1563923, 1563924, 1563927, 1563928,
[20:42] <mup> 1563938, 1563958, 1564057, 1566237, 1566589, 1566628, 1567182, 1567228, 1567683, 1568312, 1568390, 1569024, 1569097, 1569196, 1569408, 1569725
[20:47] <thumper> hi ho
[20:47] <thumper> hi ho
[20:47]  * thumper thinks he knows what's wrong with maas2 bootstrapping
[21:09] <mup> Bug #1570594 opened: read access to admin model allows grant <docteam> <juju-core:New> <https://launchpad.net/bugs/1570594>
[21:18] <alexisb> thumper, and it is your fault
[21:18] <alexisb> the bootstrap issue
[21:18] <thumper> alexisb: there is another one :)
[21:18] <thumper> probably also my fault
[21:18] <thumper> but from much earlier
[21:22] <thumper> wallyworld: would love a chat when you have a minute
[21:24] <thumper> damn...
[21:29] <wallyworld> thumper: sure, after release standup
[21:29] <thumper> wallyworld: s'ok, I think I've sorted it out
[21:30] <thumper> code has changed from what I remembered it being
[21:30] <thumper> and I was having to work through things
[21:41]  * thumper crosses his fingers
[21:45] <thumper> oh... getting close...
[21:47] <thumper> fuck yeah!!!
[21:47] <thumper> alexisb: bootstrap maas2 succeeded
[21:47] <perrito666> I take you found it?
[21:48]  * thumper tries deploy
[21:48] <thumper> hmm...
[21:48] <thumper> wat
[21:48] <mgz_> so, the correct final step after dpkg-reconfigure lxd on a fresh xenial,
[21:48] <mgz_> is `systemctl restart lxd`, right?
[21:48] <thumper> can't deploy?
[21:49] <thumper> wat?
[21:49] <perrito666> mgz_: dpkg should do that for you
[21:49] <perrito666> mgz_: in any case, if it doesnt, lxd-bridge
[21:49] <mgz_> it does, but that doesn't create the bridge
[21:49] <thumper> http://paste.ubuntu.com/15839571/
[21:49] <perrito666> mgz_: lxd-bridge is the service to restart
[21:49] <thumper> anyone had deploy issues?
[21:49] <mgz_> okay
[21:49] <thumper> trying to deploy ubuntu charm dies talking to charmstore
[21:50] <perrito666> mgz_: did you instruct dpkg to create the ipv4 network?
[21:50] <mgz_> yeah
[21:51] <thumper> WAT? debug-log not supported?
[21:51] <alexisb> \o/
[21:51] <alexisb> thumper, that is freak'n awesome!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[21:51] <thumper> alexisb: except debug log isn't working
[21:51] <wallyworld> perrito666: did you want to chat now?
[21:51] <thumper> deploy isn't working
[21:52] <thumper> and neither of these things are maas bugs
[21:52] <wallyworld> thumper: that error looks like juju not charm store
[21:52] <alexisb> heh baby steps
[21:52] <alexisb> and at least you know enough about the bug to know that
[21:55] <thumper> maybe it is maas's fault
[21:55] <thumper> jujud is panicing
[21:55] <alexisb> thumper, i jut deployed on lxd provider on latest next build
[21:56] <thumper> nice
[22:00] <menn0> thumper: I did think of this, and then forgot. Hooray for tests :) http://paste.ubuntu.com/15839652/
[22:01] <thumper> :)
[22:01] <thumper> alexisb: the failure was due to the maas provider needing to do subnet/space discovery the new way
[22:01] <thumper> so I'll leave that for voidspace
[22:02] <thumper> and get on to the filesystem bits of gomaasapi
[22:02] <perrito666> wallyworld: going
[22:04] <perrito666> wallyworld: ??
[22:04] <wallyworld> perrito666: cpu 100%, hangout frozen
[22:08] <alexisb> thumper == gomaasapi guy
[22:08] <alexisb> ;)
[22:09]  * thumper goes to make a coffee
[22:43]  * thumper looks to see who is on-call reviewer
[22:43] <thumper> ericsnow: still around?
[22:44] <ericsnow> thumper: yep
[22:44] <thumper> I'm just proposing a very simple branch that we need for maas2
[22:44] <ericsnow> thumper: k
[22:45] <thumper> damn it
[22:46] <thumper> proposed agains master , not next
[22:46]  * thumper redoes
[22:47] <thumper> ericsnow: http://reviews.vapour.ws/r/4603/diff/#
[22:48] <ericsnow> thumper: ah, bootstrap-state :)
[22:49] <thumper> ericsnow: were you around when voidspace was having these issues?
[22:49] <ericsnow> thumper: around but not involved
[22:49]  * thumper nods
[22:50] <ericsnow> thumper: Windows isn't a concern here, right?
[22:51] <thumper> ericsnow: no, because we only bootstrap on ubuntu
[22:51] <thumper> windows is currently workload only
[22:51] <thumper> not apiserver
[22:51] <ericsnow> katco: ^^^
[22:51] <ericsnow> thumper: sounds good
[22:52] <ericsnow> thumper: ship-it
[22:52] <thumper> ericsnow: ta
[23:46] <menn0> wallyworld: an old MongoDB HA bug has resurfaced (it's one of the blockers)
[23:46] <wallyworld> menn0: which branch? next?
[23:47] <wallyworld> bug number?
[23:47] <menn0> wallyworld: yep on next
[23:47] <wallyworld> menn0: it may be fixed in master
[23:47] <wallyworld> we fixed a bunch of ha stuff for beta4
[23:47] <mgz_> they will be reconverging shortly
[23:47] <menn0> wallyworld: bug 1453805
[23:47] <mup> Bug #1453805: Juju takes more than 20 minutes to enable voting <blocker> <ci> <ensure-availability> <intermittent-failure> <regression> <juju-core:Triaged by menno.smits> <juju-core 1.23:Fix Released by menno.smits> <juju-core 1.24:Fix Released by menno.smits> <https://launchpad.net/bugs/1453805>
[23:47] <menn0> wallyworld: ok that's good to know
[23:47] <wallyworld> oh i haven't see that bug
[23:48] <menn0> wallyworld: it's an old one that aaron reopened because the symptoms look the same
[23:48] <menn0> wallyworld: what happens is that after enable-ha the new controller hosts come up and the agents can connect to MongoDB but then get disconnected
[23:48] <menn0> wallyworld: we don't have the mongodb logs to confirm what's going on
[23:49] <wallyworld> joy
[23:49] <menn0> wallyworld: but off memory I think that can happen when the replicaset isn't ready yet
[23:49] <menn0> wallyworld: it's intermittment, I can't replicate it
[23:49] <wallyworld> sigh
[23:49] <menn0> wallyworld, mgz_ : we really need those MongoDB logs to know what's happening
[23:49] <wallyworld> if it's an existing bug why is it a regression?
[23:50] <menn0> wallyworld: it was fixed in 1.24 and 1.25 and has now come back
[23:50] <menn0> wallyworld: it could well be a completely different cause
[23:50] <wallyworld> i'd say so because i don't think we messed with those bits
[23:50] <wallyworld> but there were a lot of changes
[23:51] <menn0> wallyworld: what about those changes to mongodb setup in the machine agent that you made? (all that deleted code)
[23:51] <menn0> wallyworld: could that reorg have something to do with it?
[23:51] <wallyworld> the deleted code was for pre ha environments where stuff wasn't set up yet for replication
[23:51] <wallyworld> that setup is now done in bootstrap
[23:52] <menn0> wallyworld: yeah... seems unlikely
[23:52] <menn0> wallyworld: looking at the failures it's happening in master and next
[23:52] <wallyworld> and i think next was branched before my changes
[23:52] <wallyworld> but i did notice it took a while to transition to has-vote
[23:53] <wallyworld> i just thought it was mongo behaving as normal, because well, you know, mongo is web scale
[23:54] <menn0> wallyworld: not 20mins though right/
[23:54] <menn0> ?
[23:54] <wallyworld> not sure tbh
[23:54] <wallyworld> maybe 5?
[23:54] <redir> heading out for a while. I'll check back later this eve to see if things merged...
[23:55] <wallyworld> let's hope so
[23:56] <menn0> wallyworld: 5 is acceptable I think, 20+ is not
[23:56] <wallyworld> even 5 seems unforntunate
[23:56] <wallyworld> i mean, wtf is it doing