/srv/irclogs.ubuntu.com/2016/04/14/#juju-dev.txt

wallyworldericsnow: has it been like that forever? i didn't think lxc had that issue?00:00
ericsnowwallyworld: this was just the lxd provider00:01
wallyworldok, that makes sense since that provider was re-written from scratch00:01
wallyworldi didn't think we had this issue in 1.2500:01
wallyworldwith lxc00:02
ericsnowwallyworld: I'm guessing no-one had tried to use the LXD provider with non-amd64 before00:02
wallyworldnot till now :-)00:02
ericsnowwallyworld: anyway, I gotta run; natefinch can fill in the rest :)00:02
wallyworldok, ty00:03
wallyworldttyl00:03
=== Spads_ is now known as Spads
axwmenn0: if you were going the first route, I'd just supply a names.Tag rather than a GlobalEntity00:17
menn0axw: yep, fair enough00:18
axwmenn0: I'm ambivalent though. I've found having the methods on state make it a bit easier to mock in tests00:18
axwbut at the same time, I don't like throwing it all on the one type00:18
menn0that ship has sailed I think :)00:18
axwmenn0: :)00:19
menn0but I guess I don't have to make it worse00:19
menn0axw: you're right about mocking in tests though... given that we only have concrete Machines I'm setting myself up for difficult testing if I add methods to Machine00:21
menn0difficult / dumb00:21
axwmenn0: some packages go to lengths to mock Machine out too, but it is a PITA00:22
menn0axw: especially for something so simple00:23
axwmenn0: not sure if you've looked at state/volume.go or state/filesystem.go, but this is why I put everything at the top level00:23
axwit made testing much easier00:23
axwmenn0: I ended up having to have some methods on Volume/Filesystem to support, e.g. StatusSetter/StatusGetter00:24
menn0yep, makes sense00:24
rick_h_axw: howdy, wanted to chat on the model statuses and get your thought on something00:31
rick_h_axw: since the end state is archived, what about destroying is archiving?00:31
axwrick_h_: it did occur to me. I would say it's not really archiving at that point. It's destroying the resources within the model, it's just the the model docs that are archived00:32
rick_h_axw: right, but it's working toward the archived state. And it's not used elsewhere because of that idea00:32
rick_h_axw: I guess it's not quite 'factual' but as far as state transitions it goes through a few moving bits on the way to archived. Picking any one points out a bit of what's up?00:33
axwrick_h_: so I kinda agree that you should have through "archiving" to get to "archived", but at the same time I don't think "archiving" implies that anything is being cleaned up00:34
rick_h_axw: do you have a link to the bug there? I wanted to go back and refer to the list you had but I'm failing to see the bug00:34
axwbut more ... put away00:34
axw1 sec00:34
axwrick_h_: https://bugs.launchpad.net/bugs/1534627. I've currently got it as "active", "destroying", "archived" in my branch00:35
mupBug #1534627: Destroyed models still show up in list-models <2.0-count> <conjure> <juju-release-support> <juju-core:In Progress by axwalk> <https://launchpad.net/bugs/1534627>00:35
rick_h_axw: so what caused us to keep it around by default anyway?00:36
rick_h_axw: vs a cleanup all the wya and removal?00:36
axwrick_h_: I don't know, that predates my involvement. thumper?00:36
rick_h_axw: I mean it seems useful, but it seems like the exception to the rule. A destroy-model --archive that has this seems like the cleaner wa00:36
axwit definitely is an exception to the rule00:37
rick_h_I guess I assumed we had a good reason from some stakeholder but I'm coming up empty for making this the default behavior.00:37
rick_h_wallyworld: ^00:37
wallyworldrick_h_: predates me too00:38
thumperhmm...00:38
axwrick_h_: it doesn't seem useful to me, TBH. I'd prefer to just lean on external logging to keep records00:38
thumperwas there mostly so we don't suddenly error00:38
rick_h_axw: witht he rsyslog logging work going in I'm +1 to that idea00:38
thumperbecause the way people watch an environment being destroyed is status00:38
thumperif you had "watch juju status" running00:39
thumperyou would watch the services go down00:39
thumpermachines being removed00:39
rick_h_thumper: oic, so this was so that you could see it die vs go away and then if it failed to destroy you had no good way to get at it?00:39
thumperthen "error: unknown environment"00:39
thumpermostly so you could watch it die without errors00:39
thumperand grab logs... maybe00:40
rick_h_axw: can you investigate if we can just destroy in a clean way and do away with the kept around models at all please?00:40
rick_h_axw: I feel like we're chasing the wrong end of the problem atm00:40
wallyworldrick_h_: our general pattern has been to shepard entities through dying -> dead. until they are reaped by a cleaner, they will still be there00:40
axwrick_h_: sure00:41
rick_h_wallyworld: right, so can we do that better vs keep things around like this?00:41
thumperthe 24 hours thing was entirely arbitrary00:41
thumperand something I picked out of the air00:41
thumperwe could have a sensible default...00:41
thumperof a much smaller number00:41
rick_h_thumper: yea, I mean can we just watch it until it's dead cleanly and then remove it right away?00:41
wallyworldrick_h_: oh, i didn't realise they were kept for 24hrs00:41
rick_h_e.g. check every 5s or something?00:41
wallyworldrick_h_: i thought they were removed immediately once dead, sorry00:41
rick_h_I'm all for giving the user confidence/observability in what they're doing00:42
thumperit is the client going "juju status"00:42
thumperand us wanting to say "environment is dead"00:42
thumperrather than "error, you suck"00:42
rick_h_right, so at some point it fails that "model is not available" or the like00:42
thumpereventually00:42
rick_h_you asked for it to go away, what do you expect?00:42
thumperright now, 24 hours later00:42
rick_h_yea, that's gotta go imo00:42
thumperthis is the job of the undertaker00:42
rick_h_if we can tell from status it's dead, then we should be able to reap it right then and there00:42
wallyworld+100:43
rick_h_and if status says it's not dead, then we don't reap it and you can get at it and diagnose00:43
wallyworldthat's ow i thought it worked00:43
rick_h_heh no00:43
wallyworlddidn't realise models were special00:43
rick_h_so we were looking to do all this renaming and such which :/00:43
rick_h_little snowflakes in the wind :)00:43
thumperoh shit00:45
thumperI think I need to work out this juju 2.0 cli thing00:45
* thumper needs to add a maas cloud00:45
thumperperhaps after a dog walk00:45
* rick_h_ grumbles00:45
rick_h_thumper: on that please fix the whole maas as a cloud vs not needing credentials/etc kthx00:45
thumperwat?00:46
* thumper has no idea about that00:46
axwrick_h_: as in maas showing up in list-clouds?00:46
rick_h_thumper: there's a thing in that maas setup that's confusing users00:46
thumpermaas is a cloud isn't it?00:46
axwmaas is a type of cloud00:46
rick_h_axw: yea, and if you do add it and then you try to add-credential it tells you maas does't need credentials00:46
axwwat00:46
rick_h_but then users don't know how to use it since it's different00:46
thumperwat?00:47
* redir goes eod00:47
redirsee you tomorrow juju-dev00:47
rick_h_night redir00:47
axwrick_h_: I think that might be covered by anastasiamac_'s branch, http://reviews.vapour.ws/r/4573/00:47
axwnight redir00:47
anastasiamac_rick_h_: iwat axw said - it's beeing fixed ^^00:47
anastasiamac_(landing, really)00:47
=== redir is now known as redir_afk
anastasiamac_thumper: about cloud vs cloud type, see https://bugs.launchpad.net/juju-core/+bug/156405400:48
mupBug #1564054: lxd, maas and manual do not make sense in list-clouds <juju-release-support> <juju-core:In Progress by anastasia-macmood> <https://launchpad.net/bugs/1564054>00:48
wallyworldrick_h_: what do you mean? juju add-credential maas works fine00:49
rick_h_wallyworld: hmm, a user was hitting it and gave me a pastbin the other day00:49
wallyworldrick_h_: that would have been an old beta00:50
rick_h_wallyworld: I'm trying to find it, they got an error out of juju along the lines of "xxx does not need a credential"00:50
wallyworldthis is all wip00:50
thumper:)00:50
rick_h_heh00:50
wallyworldit used to be that00:50
wallyworldwe have been delivering things as threy are finished in each beta00:50
rick_h_ah ok00:50
rick_h_never mind then00:50
wallyworldadd credentials came late in the beta cycle00:50
wallyworldrick_h_: stakeholders want maas included in list clouds00:51
rick_h_wallyworld: did we have plans for an add-cloud as wel?00:51
wallyworldthat's why we did it we know maas is not a cloud00:51
rick_h_wallyworld: well, maas clouds need to be there. The trouble is the lack of an add-cloud command for now00:51
anastasiamac_wallyworld: a user would hit a "boom" if they specify cloud as part of command arguments..00:51
rick_h_wallyworld: yea, I was one of those, but mark is right00:51
axwrick_h_: we have add-cloud, but not interactive00:51
rick_h_anastasiamac_: wallyworld so I +1 going with Mark's feedback there00:51
rick_h_axw: right, interactive is what I'm thinking00:51
axwyou just point at a YAML file00:51
rick_h_axw: not for 2.0 but we should add it down the road00:52
wallyworldrick_h_: will you break the news to adam?00:52
axwit would be helpful00:52
rick_h_axw: right, but folks are confused as to what goes in the clousd.yaml, the credentials, and the config00:52
* axw nods00:52
rick_h_wallyworld: stokachu?00:52
wallyworldyeah00:52
wallyworldhe needs maas in list clouds for his app00:52
rick_h_wallyworld: heh ok, will do00:52
rick_h_wallyworld: I'm still -1 on the maas:/ special thing where you don't have to add it to the clouds.yaml00:53
wallyworldrick_h_: that was also well received and asked for by users :-)00:53
rick_h_wallyworld: bah, it just adds a 'differnet way to do it' that's special and unique and causes confusion folks have to look it upo00:54
wallyworldrick_h_: you mean like lxd :-P00:54
wallyworldthere's no lxd in clouds.yaml also00:54
rick_h_wallyworld: heh00:55
thumperah crap...00:55
wallyworldrick_h_: the use case driving this is - people want to know what they can put with juju bootstrap <controllername> <what goes here>00:56
thumperI don't think my 1gb kvm maas instances are up to running juju are they?00:56
thumperwallyworld: remember when I said the whole cloud credential spec would be a big pile of work?00:56
rick_h_wallyworld: yea...but they can't do maas without config info (in this case the IP address) so we've changed the bootstap command to make it work00:56
wallyworldrick_h_: altered slightly yeah, but still inituitive imo00:58
* thumper goes to walk the dog and get away from the copmuter a bit00:58
wallyworldthumper: oh i know00:58
wallyworldi never doubted it00:58
wallyworldwe piled in so much stuff in such a short time, and delivered incrementally over several betas and got beat up when it wasn't all there beta 1 :-/00:59
rick_h_wallyworld: no one got beat up :P01:09
rick_h_wallyworld: I couldn't reach you!01:09
wallyworldlol01:09
wallyworldrick_h_: not beat up directly01:09
rick_h_:P01:10
wallyworld, more complaints :-)01:10
rick_h_wallyworld: we just had an opinion that add-credential should have come first :P01:15
rick_h_not complaints, opinions...I hear it's like something else everyone has :)01:15
wallyworldrick_h_: sure, but it was harder to add and was icing - we needed the core functionality with credentials added by hand. having fancy add credentials with nothing working would not have been cool01:16
rick_h_wallyworld: understand completely...after I stopped and thought about it :P01:16
wallyworld:-)01:16
wallyworldrick_h_: next time we'll start the work at the beginning of the cycle, not a month or 2 out :-)01:17
rick_h_wallyworld: psh, don't go changing everything on me01:17
wallyworldgreat, now i've got that song stuck in my head01:17
rick_h_lol, glad to be of service01:17
wallyworldi love you just the way you are.... ta de dum01:18
wallyworldaxw: got 5 mintes, standup?01:43
axwwallyworld: sure, brt01:43
bradmshould juju2 be able to bootstrap against a private openstack cloud? I'm getting:01:50
bradm2016-04-14 01:30:25 ERROR cmd supercommand.go:448 failed to bootstrap model: model "admin" of type openstack does not support instances running on "amd64"01:50
bradmmy index.json does list 14.04 images...01:50
bradmah, when I don't use --upload-tools it gives:02:09
bradm2016-04-14 02:05:56 ERROR cmd supercommand.go:448 failed to bootstrap model: cannot start bootstrap instance: no "trusty" images in bootstack-canonistack-bos01 matching instance types [m1.small m1.medium m1.large m1.xlarge]02:09
thumperwallyworld: ping02:20
wallyworldwot02:21
wallyworldthumper: ?02:22
thumperwallyworld: got a few minutes to chat?02:22
wallyworldsure02:22
wallyworldbradm: juju 2 should work similar to juju 1 for private clouds02:23
wallyworldit's all about getting the correct streams metadata which can be tricky02:23
bradmwallyworld: just filed LP#1570162 about it02:23
wallyworldbut if it works for juju 1 it should work for juju 202:24
wallyworldok, we'll look at the bug02:24
bradmlet me know if you need any more info about it02:24
thumperwallyworld: 1:1 ?02:24
wallyworldyup02:24
bradmwallyworld: this is mitaka on trusty, and it has different arch compute nodes too02:24
bradmgrabbing some lunch, back in a while02:25
mupBug #1570162 opened: juju2 openstack private cloud cannot start bootstrap instance <canonical-bootstack> <juju-core:New> <https://launchpad.net/bugs/1570162>02:25
mwhudsonwallyworld: juju-mongo-tools3.2 just got accepted \o/02:45
* mwhudson runs away for a bit02:45
wallyworldmwhudson: awesome02:45
menn0axw: I was thinking that the machiner would send the SSH host key to the state server (via the machiner facade)03:00
axwmenn0: sounds fine to me03:01
menn0axw: seem reasonable?03:01
natefinchwallyworld: btw, this is the fix for the lxd arch bug that I'm pretty sure works: https://github.com/juju/juju/pull/5116/files. But I still can't quite fully test due to not being able to fully disable upload-tools.. I haven't figured out how to disable saving the tools (I've found a couple likely spots and commented out code, but to no avail).03:04
wallyworldok, i'll look at pr, thanks03:04
wallyworldthere's a toolsstorage that manages the tools in gridfs03:05
natefinchwallyworld: I commented out this whole loop: https://github.com/juju/juju/blob/master/apiserver/tools.go#L262 which looks like the place where we store the tools after uploading them... but it didn't seems to change anything03:07
wallyworldnatefinch: why not just make storage.Add() itself a no op03:09
wallyworldnatefinch: anyway, here's what you wnated03:10
wallyworldfetchAndCacheTools03:10
wallyworldor maybe not, that is when we download then from streams03:10
wallyworldit's in jujud/bootstrap.go03:11
wallyworldlogger.Debugf("Adding tools: %v", toolsVersion)03:11
wallyworldif err := toolstorage.Add(bytes.NewReader(data), metadata); err != nil {03:11
wallyworldi just searched for usages of toolsstorage.Add()03:11
wallyworldthere's only a few to check03:11
natefinchwallyworld: thanks... I really wasn't sure what to search for03:13
wallyworldjust the .Add()03:13
wallyworldand see what calls it03:13
wallyworldas that's where tools are added to state03:13
natefinchwallyworld: yes but, I didn't know it was called .Add03:13
wallyworldbut you commented it out so you must have ?03:13
wallyworldin that place03:14
wallyworldanyways, you'll need to talk to john about that todo03:14
natefinchwallyworld: I looked in the server code for where we were handling the http post for tools upload03:14
wallyworldas he is making lxd work on provisioned machones03:14
wallyworldsure, and toolsstorage.Add() was right there03:14
natefinchwallyworld: it never occurred to me that there would be code in the client that would be identical03:14
wallyworldbootstrap is special03:15
natefinchwallyworld: indeed03:15
wallyworldit has to have logic built in as there's no server running yet03:15
wallyworldso to recap, we'll need to get that todo sorted asap03:15
wallyworldi think03:15
natefinchnot really03:15
natefinchthis is the lxd provider03:16
natefinchwe only support localhost03:16
wallyworldactually, yeah maybe not03:16
wallyworldit's only local yeah03:16
natefinchsome day we maybe might support a remote host. Today's not that day :)03:16
wallyworldi was confusing the prtovisioner03:16
natefinchyeah, it's confusing03:16
wallyworldgood that it's fixed, ty03:17
natefinchwell, I'll go test it, but I'm pretty certain that fixes it03:17
wallyworldyeah, the code looks correct03:18
natefinchthere was a suspicious log message on boot that we were saving amd64 tools, even though we were bootstrapping with arm64... and that message changes to arm64 now... but I have to do the full test with upload-tools disabled to know for sure.03:18
wallyworldnatefinch: maybe not - cloud init gets tools via the controller which acts as a proxy03:19
wallyworldso even if upload tools is used, you just trace requests into the controller03:20
wallyworldi didn't think of that previously03:20
wallyworldso just trace the tools download hander gets03:20
natefinchI'm building on the arm64 machine which is just about the slowest machine in existence, I'm pretty sure03:28
thumperquick03:29
thumpersomeone03:30
thumperwhere is the cloudinit data stored03:30
thumperon the cloud machine03:30
thumperI need to get it off before the code kills the machine03:30
natefinchdunno, sorry03:31
thumpermenn0: ^^?03:31
thumperjuju brought the machine up but can't ssh in03:32
thumperaxw: hey...03:32
thumperthis may be part your history03:33
thumper2016-04-14 03:33:01 DEBUG juju.provider.common bootstrap.go:328 connection attempt for 192.168.100.3 failed: /var/lib/juju/nonce.txt does not exist03:33
thumperaxw: we just bring up a machine with ssh keys03:33
thumperand I'm guessing that file03:33
axwthumper: yes, so we know we've connected to the right machine03:34
thumperaxw: but that file isn't there03:34
axwthumper: something's not doing the right thing with cloud-init then I guess?03:34
thumperaxw: know where the cloud init file is?03:35
axwthumper: /var/lib/cloud I think?03:35
* axw rummages03:35
thumperI have user_data.txt from there03:35
thumperbut it is base64 encoded03:35
thumperand decoded looks binary03:35
axwthumper: there should be a plaintext file nearby03:35
axwthumper: instance/cloud-config.txt maybe03:36
thumperdoesn't exist03:36
thumperoh03:36
thumperzero bytes03:36
axwthumper: interesting, I don't think I've ever seen a zero-byte one. maybe part of the problem.03:37
thumperhmm...03:37
thumperI think I'll add logging to the creation of the user data03:37
mupBug #1570175 opened: juju2 kill-controller doesn't work when bootstrap server is unreachable <canonical-bootstack> <juju-core:New> <https://launchpad.net/bugs/1570175>03:37
thumperaxw: we are this |-| close to having maas2 boot03:38
axwthumper: cool :)03:38
natefinchahh man, the fact we can't cross compile is killing me on this dumb arm64 machine03:42
natefinchmy laptop compiles juju in like 10 seconds.  This machine takes like 3 minutes03:42
menn0thumper: sorry, missed your message... I don't know the answer either (cloud-init newbie)03:43
thumperwell, I've added some debugging code to the cloud init renderer03:43
thumperwhich I found...03:44
menn0thumper, axw: would the cloudinit logs give some clues? (/var/log/cloud-init.output and cloud-init.log)03:44
thumperso I can catch it as it is yaml-ified, before gzip, base6403:44
thumperok, got the cloud init03:45
thumperbut it is kinda big...03:45
* thumper looks deeper03:45
axwmenn0 thumper: *is* there a /var/log/cloud-init-output.log?03:45
thumperI'll look this time03:45
thumperlast time 10 minutes passed and machine was released03:45
thumper- install -D -m 644 /dev/null '/var/lib/juju/nonce.txt'03:46
thumper- printf '%s\n' 'user-admin:bootstrap' > '/var/lib/juju/nonce.txt'03:46
thumperfrom the local userdata it is sending down03:46
thumpernot much of a nonce :)03:46
axwthumper: heh yeah, but it's better than connecting to some random machine on your local network and running juju setup on it :)03:47
thumpertrue03:48
thumper2016-04-14 03:47:40,431 - __init__.py[WARNING]: Unhandled non-multipart (text/x-not-multipart) userdata: 'H4sIAAAJbogA/+w7aXPbuJLf...'03:50
thumperWTF...03:50
thumperbasically it stopped cloudinit03:50
axwthumper: nice :)03:50
axwthumper: so MAAS is just dropping it on the floor?03:50
* thumper shrugs03:50
thumperI'll have to dig a bit more03:50
axwthumper: people have complained about the nonce.txt file being missing before, I never could repro tho03:51
axwmaybe it depends on contents/padding/something03:51
natefinchwallyworld: huzzah, success04:00
wallyworldgreat04:01
wallyworldafter waiting ages for a compile04:01
natefinchwallyworld: well, I kept getting stupid compile errors... finally realized I could just slap a return nil before the code I wanted to avoid04:01
wallyworldthumper: anastasiamac_ committed a fix for that maas bug you saw with listing credentials04:07
bradmwallyworld: hmm, I can bootstrap with juju2 beta 3 on canonistack-lcy01, but not on this new mitaka cloud04:09
wallyworldbradm: it may not have the flavours set up correctly?04:09
wallyworldjuju uses those plus arch to determine the instance id04:10
wallyworldand matches it all with simplestreams04:10
wallyworldit all needs to match up correctly04:10
bradmwallyworld: it has flavours, I'm not sure how you have to set them up, we've never done anything about mapping them before04:12
bradmwallyworld: I can certainly boot instances, although that is done by specifying an image id04:12
wallyworldright, it appears the image id selection algorithm fails due to reaons04:13
wallyworldreasons04:13
wallyworldit would need investigation to see what's wrong with the set up04:13
bradmI can see all 3 arches in the index.json04:13
wallyworldi personally have not internalised the algorithm - would need to trace it all out and see what's needed where04:14
bradmsomething must have changed with mitaka04:14
bradmor juju204:14
bradmhm, thats a point, I should try juju1 on it04:15
wallyworldbradm: also anastasiamac_ thinks there could be related (same?) issue that is fixed in beta 404:15
bradmwallyworld: yeah, maybe I should just wait until beta 4 is out04:16
bradmare we there yet?  are we there yet? ;)04:16
wallyworldbradm: in maybe 24 hrs tops04:16
wallyworldin time for xenial cut off04:17
bradmwallyworld: awesome.  I'll try out juju1 on this to see how it goes, might narrow things down a bit.04:17
wallyworldbradm: yes please, that data point would be helpful in case there's still an issue04:17
wallyworldjuju1 and 2 should behave the same AFAIK04:17
wallyworldthat bug mentioned above may well be an existing issue04:18
wallyworldi don't know the detials04:18
bradma newly released mitaka being deployed via unreleased charms and trying to use a beta juju on top of it?  nah, couldn't be any problems there. :)04:18
wallyworldso we'll take it a step at a time: try juju1, wait till beta404:18
wallyworldof course not04:18
wallyworldwhat could possibly go wrong04:18
bradmits a house of unreleased cards04:19
bradmbut thats why we're doing it, to help bash out any bugs early04:19
bradmwallyworld: aha, it fails in the same way04:25
wallyworld\o/04:25
wallyworldnot juju2 :-)04:25
bradmthe index.json is pretty simple04:28
bradmoho04:30
bradmarch: x86_6404:30
bradmwhy is that there04:30
natefinchwhelp, I have a PR up for that fix here: http://reviews.vapour.ws/r/4555/  I haven't been able to successfully change the tests such that they actually fail with the old code, but it's time for bed.04:35
bradmwallyworld: hah, that was it.  for some reason the version of glance-simplestreams-sync we have was canonicalizing the arch from amd64 to x86_64.  now I get a different error. :)04:40
wallyworldbradm: progress!04:40
bradmwallyworld: now it wants me to give it a network id04:40
wallyworldwhat is "it"?04:41
bradmthe error from the bootstrap04:41
wallyworldi haven't done much with networking, not sure04:42
bradmMultiple possible networks found, use a Network ID to be more specific.04:42
bradmthat feels like a nova error, though.04:42
wallyworldyes it does04:42
wallyworldcould be propagated back from start instance04:43
bradmhow do you tell juju2 what the default network is, like the network setting in juju104:43
bradmjust trying to add it where you define the endpoint04:45
bradmnope, no go :-/04:47
bradmdo I just mark the bug as invalid?04:48
wallyworldbradm: juju has spaces and things. but if you are asking about config, instead of foo=bar in environments.yaml, you use --config "foo-bar" on the bootstrap cmd line04:50
wallyworldjuju2 i mean04:50
wallyworldif it's a set up issue with openstack, then yeah, bug may be invalid04:51
bradmwallyworld: thats got it04:53
bradmwallyworld: at least, its trying to get an address now, so its much further than before.  looks like we need a way to set the default network04:53
wallyworldthat worked?04:53
wallyworldok04:53
wallyworldthat bit i am not sure of04:53
wallyworldbradm: dimiter and andy are the folks to ask about that04:54
bradmoh boy, not quite.  now it errored out with:04:57
bradmerror: flag provided but not defined: --model-config04:57
wallyworldbradm: you need upload-tools if you are running from trunk05:00
bradmwallyworld: that was with upload-tools...05:00
bradmtrying now without05:00
wallyworldthe error looks like it is because the jujud binary from tools is old05:01
bradmit seems to be getting further without the upload tools05:05
bradmoh, still fails.05:05
bradmhttps://pastebin.canonical.com/154283/ <- error message05:05
bradmmaybe I should just wait for beta405:09
menn0axw: here's the state part and some of the API work for storing SSH host keys. http://reviews.vapour.ws/r/4586/05:09
axwmenn0: cool, looking05:09
wallyworldbradm: do you have the latest code checked out? that could explain why upload tools give poor results. also that error - is the auth url correct? just aguess05:09
bradmwallyworld: nope, I'm just using teh beta3 version from the ppa05:09
wallyworldbradm: in that case upload tools does nothing05:10
bradmwallyworld: that can't be true, I saw very different things05:10
wallyworldok, so it grabs the first juju binary from the search path and uses that05:10
menn0axw: next up... some routines in juju/utils for parsing host key files and generating known hosts files05:10
wallyworldbradm: or builds from source, but you don't have source checked out05:11
bradmwallyworld: and what auth url do you mean?  the endpoint in the clouds.yaml05:11
wallyworldbradm: so there may be another juju in the path05:11
wallyworldbradm: for openstack, yeas i think so05:11
bradmwallyworld: there's no juju source on this box at all05:11
bradmwallyworld: there is juju 1.25.505:11
wallyworldbradm: so in that case upload tools is using those binaries05:12
wallyworldwhich explains a lot05:12
wallyworldthe --model-config unknown etc05:12
bradmI used juju 1.25.5 to deploy openstack05:12
wallyworldsure but you said you wer eusing juju2 from a ppa05:12
bradmyeah, juju2 from ppa, juju1 from ppa05:12
wallyworldso if you us a juju2 client and upload tools picks the 1.25 binaries it wil screw up05:12
wallyworlddon't use upload tools please :-)05:13
wallyworldunless you are a developer and have source05:13
bradmrighto.05:13
bradmI'm just trying different things to work out what its doing :)05:14
wallyworldbradm: so now it may be that keystone is messing up05:14
wallyworldbradm: i would love upload tools to be removed tbh05:14
wallyworldit causes too many issues unless used under strict conditions05:15
bradmI'm sure we've had to use it to fix things in the past05:15
bradmbut yeah, if its no longer of use05:15
wallyworldwith source yes, or custom binaries put in the right place05:15
wallyworldit has a use, but yu need to be careful05:15
bradmwallyworld: I've been using juju since the python days, I'm sure I've got all sorts of redundant things in my brain about it :)05:16
wallyworldand in this case, it caused "weird" errors until i found out you didb't have source code and had 2 versins etc05:16
wallyworldbradm: understood. you deserve a medal :-)05:16
wallyworldfor consuming all of our bugs for so long05:16
mupBug #1570162 changed: juju2 openstack private cloud cannot start bootstrap instance <canonical-bootstack> <juju-core:Invalid> <https://launchpad.net/bugs/1570162>05:17
bradmoh my05:20
bradmthat error message really is true, I can't reach the keystone IP from a VM05:20
wallyworldat least juju is not lying :-)05:20
bradmjuju would never lie to us, would it?05:21
wallyworld*never*05:21
bradmmissing a route.05:24
axwmenn0: code looks good, but I have a question about the structure of the keys05:24
axw(in RB)05:24
menn0axw: can the SSH server really have multiple host keys for a given algorithm?05:26
axwmenn0: I think so, but I'll test to make sure05:26
menn0axw: I just checked the man page... I don't think it's possible05:27
axwmenn0: sshd_config? what part?05:28
menn0axw: man sshd05:28
bradmjuju2 likes leaving secgroups around, just hit a quota limit05:29
menn0axw: the "-h host_key_file" part and the FILE section05:29
menn0FILES05:29
menn0axw: there's one host key file per key/algorithm type05:29
axwmenn0: AFAIK they're just the default ones, referenced by /etc/ssh/sshd_config05:30
menn0axw: the wording in man sshd_config is more vague about it05:31
menn0axw: I guess it's safer to use []string (with no real downside)05:32
menn0axw: good catch05:32
menn0axw: in that case, I'll just store the key files in state verbatim (they're one line each)05:32
menn0axw: and handle the parsing and reformatting in the client when it generates the bespoke known_hosts file05:33
axwmenn0: SGTM. FWIW, starting sshd with multiple RSA keys works fine05:33
* axw nods05:33
menn0axw: you just tried it?05:33
axwmenn0: yep05:33
axwmenn0: BTW, there's a function that you can use to parse the public keys: https://godoc.org/golang.org/x/crypto/ssh#ParseAuthorizedKey05:33
menn0axw: ok cool.. that's definitely the right approach then05:33
menn0axw: good to know... I just found something in juju/utils/ssh which also does it :)05:35
axwheh05:35
axwcan't have too many05:36
menn0axw: maybe the keys should be parsed to a (type, keydata, comment) struct and send and stored that way?05:36
bradmoh, you can just run juju2 enable-ha ?05:36
bradmer, can't just05:36
menn0axw: rather than the raw strig05:36
menn0string05:36
axwmenn0: *shrug* if it's easy enough to do without losing any info, maybe05:37
axwI'm not sure it's worth the effort tho05:37
axwIOW, authorized key format is already perfect information, so probably not worth destructuring at that point unless we think we're going to query on the individual fields05:39
menn0axw: you're right05:41
menn0axw: what threw me a little was that the known_hosts file doesn't include the comment field on my machine05:42
menn0axw: so I was thinking it would need to be stripped05:42
menn0axw: but looking at the docs, it's fine if it's there05:42
menn0axw: []string it is05:42
axwmenn0: cool. a comment saying that it's in authorized_keys format would be helpful05:43
menn0axw: yep, will add.05:43
bradmwallyworld: yeah, this network thing is going to be a blocker.  need a way to set it as a default somewhere.   just setup a multiuser env, tried to boot a VM and it errored out with the same thing about multiple networks05:57
wallyworldbradm: i recall conversations in this area but not any specifics, not sure of the status05:57
wallyworldaybe there's a solution already, i just don't know it05:57
wallyworldaxw_: ping06:09
axw_wallyworld: pong06:09
wallyworldaxw_: stupid quetion of the day, i'll make a dick of myself i'm sure. can look look at line 71 or environ_broker.go in the lxd provider06:10
axw_wallyworld: yep?06:11
wallyworldshould be finishInstanceConfg()06:11
wallyworldthe arg struct is passed by value06:11
wallyworldso how will it ever work06:11
axw_wallyworld: pretty sure InstanceConfig is a pointer06:11
axw_yep06:11
wallyworldah right06:11
wallyworldyes i missed that06:12
axw_well shit. I just bootstrapped and it panicked on the server due to "send on closed channel" in the systemd package06:13
wallyworldyay06:17
menn0axw_: please take another look at http://reviews.vapour.ws/r/4586/diff/06:20
menn0axw_: no rush as I'm about to EOD. but if you could look before you finish that would be great06:21
axw_menn0: no worries, have a nice evening06:21
menn0axw_, wallyworld : it's feeling like this ssh host key handling issue is going to be easier to lick than it seemed (still a bit to do I realise)06:22
wallyworldwin06:22
axw_wallyworld: is there an agenda for the sprint yet? formal or informal06:30
axw_wallyworld: I mean, topic list we're compiing06:30
axw_compiling*06:30
axw_wallyworld: CI for storage really needs to happen06:30
wallyworldaxw_: sort of - right as of now, it's digesting the roadmap wish list06:30
axw_it's been broken in master since last year, in Malta...06:30
axw_yep06:31
axw_ok06:31
wallyworldaxw_: agreed about CI for storage. can we discuss in 1:1 tomorrow?06:31
axw_wallyworld: sure06:31
=== bradm_ is now known as bradm
mwhudsonwallyworld: is there any other packaging stuff juju is waiting on?07:17
mwhudsonother than juju itself ;-p07:17
wallyworldnot that i know of07:17
wallyworldwell do want mongo 3.2 in trusty and wily at some stage07:17
wallyworldsoon hopefully :-)07:17
mwhudsonwallyworld: somehow the deadlines on those don't seem so tight07:18
=== blahdeblah_ is now known as blahdeblah
mwhudsoneg i guess i should backport go 1.6.1 to trusty...07:19
wallyworldmwhudson: they are not as tight, but we do want a consistent mongo experience across series at some stage07:20
mwhudsoni guess we can find out if the packages build for a start07:20
wallyworldaxw_: could you look at http://reviews.vapour.ws/r/4587/ ? i want to land it because CI is failing with lxd on arm. i have taken nate's work and added tests07:41
wallyworldi want to try and get this in for beta407:42
axw_wallyworld: looking07:42
wallyworldta07:42
bradmI've filed LP#1570219 if anyone who knows more about the networking side of things could take a look, that'd be great, thanks.07:42
axw_wallyworld: done07:44
wallyworldta07:44
mupBug #1570216 opened: juju2 not cleaning up nova secgroups with openstack provider <canonical-bootstack> <juju-core:New> <https://launchpad.net/bugs/1570216>07:47
mupBug #1570219 opened: juju2 openstack provider setting default network <canonical-bootstack> <juju-core:New> <https://launchpad.net/bugs/1570219>07:47
fwereade__ashipika, cmars: responded to a couple of bits of review, went into detail with the problems with the idiosyncratic approach to workers; we should probably all talk about this live today07:50
ashipikafwereade__:  thanks.. it's ok.. i was not going to land this PR as it is.. i still welcome any and all comments, but i'll be breaking it up into smaller managable bits, i expect..07:51
fwereade__ashipika, cool, thanks, let me know if you want to talk about any of it07:53
ashipikafwereade__: i expect i will.. many times along the way :)  but this week we have our priorities elsewhere, so this might have to wait a bit.. anyways it needs to land by may, if i understand the timeline..07:53
fwereade__ashipika, cool07:54
ashipikafwereade__: and the thing is: there was no spec.. it was just "oh, this needs to work.. "07:54
fwereade__ashipika, yeah, that was rather my reading of it07:54
ashipikafwereade__: we kept asking for more details, but nothing came back.. other than DTAG wants rsyslog forwarding07:55
=== terje is now known as Guest76597
voidspacebabbageclunk: so, in my test for waitForNodeDeployment I'm seeing the same NotFound error08:50
voidspacebabbageclunk: so looks like a straightforward bug08:50
babbageclunkvoidspace: yay!08:50
babbageclunkvoidspace: with my stuff landed now, what should I pick up?08:50
voidspacebabbageclunk: maas2Instance.volumes would be good08:51
babbageclunkvoidspace: ok08:51
voidspacebabbageclunk: there's a list at the top of the status document of tasks08:51
voidspacebabbageclunk: you could take on fixing the behaviour when you run against MAAS2 without the feature flag08:52
voidspacebabbageclunk: currently it just panics08:52
voidspacebabbageclunk: instead we should detect MAAS 2 and exit with an error instead08:52
voidspacebabbageclunk: that's easy enough to do - might be good to get that in first08:52
voidspacebabbageclunk: so attempt to create the controller even without the feature flag08:52
voidspacebabbageclunk: if it succeeds then we're on MAAS 208:53
voidspacebabbageclunk: if we don't have the feature flag error out with a NotSupported error08:53
babbageclunkvoidspace: yeah, I'll do that first.08:55
mwhudsonwallyworld: juju-mongo* stuff builds on trusty with a bit of flailing for -tools https://launchpad.net/~mwhudson/+archive/ubuntu/devirt/+packages/?field.series_filter=trusty08:56
voidspacebabbageclunk: ah no - my NotFound error is because my test doesn't give the fakeController.Machines method anything to return08:58
voidspacebabbageclunk: so not the same bug...08:58
voidspacebabbageclunk: ooh, see Tim's status update - he did some work for us09:00
voidspacebabbageclunk: and got bootstrap further09:00
* thumper wonders if he is in a different hangout to the others09:01
voidspacethumper: morning09:01
voidspacethumper: keen bean09:01
voidspacebabbageclunk: just added a new task to the list - use maas2NetworkInterfaces from StartInstance09:23
voidspacebabbageclunk: should be trivial, but needs a test as well09:23
voidspacebabbageclunk: (a test at the StartInstance level)09:24
=== terje is now known as Guest32809
=== meetingology` is now known as meetingology
voidspacedimitern: frobware: babbageclunk: http://pastebin.ubuntu.com/15826362/09:52
voidspacedimitern: frobware: babbageclunk: that's real progress09:52
dimiternvoidspace: awesome!09:53
frobwarevoidspace: indeed09:54
dimiternvoidspace: I see you're possibly hitting the same ssh issue I had - my key is ssh-dss, apparently no longer considered secure09:54
dimiternvoidspace: but I found a way around it, if you need09:55
voidspacedimitern: go ahead09:55
voidspacedimitern: I thought this was the issue tim had with not being able to ssh in09:55
dimiternmight still be that, but try this:09:56
voidspacedimitern: can you join us in #maas on canonical09:56
voidspacedimitern: (as well)09:56
dimiternvoidspace: http://paste.ubuntu.com/15826416/09:57
dimiternah, I thought I'm there already09:57
dimiternvoidspace: so replace the IP ranges to match yours - the important bits are the last 4 sections09:57
voidspacedimitern: thanks09:58
voidspacedimitern: /etc/ssh/ssh_config09:59
dimiternvoidspace: ~/.ssh/config09:59
voidspacedimitern: cool, thanks09:59
voidspacedimitern: can you confirm that we gzip userdata for allenap in #maas on canonical irc09:59
dimiternvoidspace: looking10:01
voidspaceretrying10:01
voidspace(the bootstrap I mean)10:01
mupBug #1570269 opened: state: ensure that Models are always paired with the correct State <tech-debt> <juju-core:Triaged> <https://launchpad.net/bugs/1570269>10:05
babbageclunkvoidspace: awesome10:05
thumpervoidspace: no, that is a different bug10:09
TheMuemorning10:09
thumper2016-04-14 03:57:54 ERROR cmd supercommand.go:448 failed to bootstrap model: waited for 10m0s without being able to connect: /var/lib/juju/nonce.txt does not exist10:09
thumpero/ TheMue10:09
* thumper is outa here10:09
voidspacethumper: o/10:09
TheMuen8 thumper10:09
TheMue:)10:10
mupBug #1570285 opened: worker/undertaker: update status with remaining resources <tech-debt> <juju-core:Triaged> <https://launchpad.net/bugs/1570285>10:32
babbageclunkdimitern: how do I bootstrap to a different region in AWS? If I use --to zone=eu-west-1 I get this: http://pastebin.ubuntu.com/15827282/10:47
babbageclunkdimitern: also, why do I always need to specify --upload-tools?10:47
mwhudsonbabbageclunk: i don't know but eu-west-1 is a region, not a zone10:47
dimiternbabbageclunk: juju bootstrap <controller-name> aws/<region> ...10:48
babbageclunkmwhudson: Ah, ok - I was following some old docs, I think10:48
dimiternbabbageclunk: since that changed I tend to keep this around for reference: http://paste.ubuntu.com/15127859/10:48
babbageclunkdimitern: thanks10:49
babbageclunkdimitern, mwhudson - also, juju help placement lead me astray too (I guess there are people writing new help messages).10:50
dimiternbabbageclunk: docs can be improved indeed, but have you tried 'juju help bootstrap' ?10:51
babbageclunkdimitern: yup, that was how I got to 'juju help placement'10:51
dimiternbabbageclunk: ah :) I see - "placement" is related to the --to argument10:52
frobwaredimitern: can you drop into the sapphire standup HO?10:55
dimiternfrobware: sure, just a sec10:55
frobwarethx10:55
babbageclunkdimitern: Right - totally missed the bit at the top of the bootstrap help - was confused because the placement docs matched what I saw in the old docs on the web.10:56
babbageclunkdimitern: thanks!10:57
voidspacefrobware: dimitern: babbageclunk: http://reviews.vapour.ws/r/4591/12:23
dimiternvoidspace: LGTM12:38
perrito666bbl12:38
mupBug #1570368 opened: juju commands timeout while a bootstrap is in process <conjure> <juju-core:New> <https://launchpad.net/bugs/1570368>12:57
voidspacedimitern: thanks13:12
* voidspace lunches13:12
mattywfwereade_, ping?13:13
mattywfwereade_, I have some questions if you can spare 5 minutes?13:16
babbageclunkvoidspace: what kind of error should I return for when the endpoint's MAAS 2 but the flag isn't set? errors.NotSupportedf?13:19
babbageclunkvoidspace: struggling to give an error message that makes sense with " not supported" stuck on the end.13:20
babbageclunkvoidspace: ok "unless the 'maas2' feature flag is set MAAS 2 is"13:22
dimiternbabbageclunk: for those cases there's also a errors.NewNotSupported(nil, fmt.Sprintf("fmt str", args,...)) you can use13:43
babbageclunkdimitern: great, thanks¬13:43
babbageclunkoops, !13:43
katcomorning all14:00
natefinchmorning katco14:02
katcoand actually need to reboot... brb14:02
voidspacebabbageclunk: just errors.New and a sensible error message of your choice will be fine14:08
babbageclunkvoidspace: hello14:08
voidspacebabbageclunk: hello14:09
babbageclunkvoidspace: so, obviously that change was trivial14:09
voidspacebabbageclunk: cool14:09
babbageclunkvoidspace: but working out why the tests weren't failing already in the same way wasn't14:09
voidspacebabbageclunk: don't forget to update the status doc14:09
voidspacebabbageclunk: hah14:09
voidspacebabbageclunk: I added the feature flag to our tests at some point14:09
babbageclunkvoidspace: it turns out maas2 just returns some html when you ask for /1.0/version/14:10
babbageclunkvoidspace: rather than 404ing14:10
voidspacebabbageclunk: ah, it used to return nul14:10
voidspacebabbageclunk: they've changed it14:10
voidspacedimitern: babbageclunk: I'm leaving early today to go to a tatooist14:11
voidspacedimitern: babbageclunk: then I'm coming back in again later14:11
babbageclunkvoidspace: well, it returns null when you parse it as json14:11
dimiternvoidspace: ok, have phun ;)14:11
voidspaceah14:11
voidspacedimitern: I will14:11
voidspacebabbageclunk: that makes sense14:11
voidspacebabbageclunk: well, not returning a 404 doesn't make sense14:11
voidspacebut there you go14:11
babbageclunkvoidspace: have a nice tattoo appointment!14:12
voidspacebabbageclunk: I'm sure I will, not going yet - but soonish14:13
babbageclunkvoidspace: won't forget the doc this time, sorry!14:13
voidspaceheh, np14:14
mupBug #1453805 opened: Juju takes more than 20 minutes to enable voting <blocker> <ci> <ensure-availability> <intermittent-failure> <regression> <juju-core:Triaged14:50
mupby menno.smits> <juju-core 1.23:Fix Released by menno.smits> <juju-core 1.24:Fix Released by menno.smits> <https://launchpad.net/bugs/1453805>14:50
babbageclunkvoidspace: still around?14:57
voidspacedimitern: ping14:57
voidspacebabbageclunk: yes14:57
voidspacebabbageclunk: I might have found the bug14:57
voidspacebabbageclunk: gomaasapi does base64 encoding for us, and so do we14:57
babbageclunkvoidspace, dimitern, frobware: http://reviews.vapour.ws/r/4595/14:58
rick_h_voidspace: do you know if frobware is around today?14:58
babbageclunkvoidspace: Oops14:58
voidspacerick_h_: he was earlier, yes14:58
babbageclunkvoidspace: nice14:58
rick_h_voidspace: k, ty14:58
babbageclunkvoidspace: Is there an easy way to explore the maas api?14:58
voidspacebabbageclunk: I use the CLI...14:58
babbageclunkvoidspace: ah, I keep forgetting about that.14:59
babbageclunkvoidspace: thanks14:59
babbageclunkvoidspace: the docs are singularly unhelpful.15:00
frobwarerick_h_: yep, here, but IRC dropping out a lot atm15:00
rick_h_frobware: ah ok, I asked stokachu to shoot you an email on a potential network/bridge issue he was seeing last night15:01
katconatefinch: standup time15:01
rick_h_frobware: wanted to let you know I asked him to and I know you've been doing MAAS2/bug stuff but wanted to see if you or something could poke at it and see if it's a bug or working as intended/etc15:01
alexisbrick_h_, we should have him open a bug so we can get it on the squad board15:03
alexisbthat is where the full team is pulling priority bugs15:03
rick_h_alexisb: rgr, the question was "is this a bug?" so just wanted to make sure first15:04
frobwarerick_h_: I semi-stalled on an answer to stokachu. tych0 is proposing a patch for the problems discussed in that email. I also owe tych0 a patch too.15:06
katconatefinch: ping?15:06
rick_h_frobware: ok, cool. Ignore me then.15:06
natefinchkatco: sorry15:07
natefinchkatco: lost track of time, coming15:07
voidspacedimitern: babbageclunk: frobware: removing the extra base64 encode from gomaasapi fixes the issue Tim reported this morning15:08
voidspaceand now we die in a new way15:08
frobwarevoidspace: that was base64 on base64 then?15:10
dimiternvoidspace: ah, good - too much encoding then :)15:10
voidspacefrobware: yep15:11
tych0rick_h_: yeah, i know what the issues are with a bridge15:11
tych0gonna send some patches today15:11
tych0just need to catch up on email :)15:11
rick_h_tych0: ok cool, thanks for all the help in figuring it out!15:11
voidspacedimitern: where is that done - in the cloudinit package?15:11
dimiternvoidspace: in cloudconfig/providerinit IIRC15:13
voidspacedimitern: you are correct15:13
voidspacedimitern: I've asked Tim to fix it in gomaasapi15:14
dimiternvoidspace: sweet!15:14
tych0rick_h_: sure, np15:15
voidspacedimitern: do we propagate feature flags onto the juju controller machine?15:16
voidspacewe must do15:16
voidspacehowever, the issue I'm seeing now kinda implies not15:17
dimiternvoidspace: yeah15:19
dimiternvoidspace: we do15:19
voidspaceok, kinda hard to see where this "requested map got nil" comes from15:20
voidspaceI'm bootstrapping with debug to see15:20
voidspacemaybe Subnets15:20
dimiternvoidspace: this sounds like a GetMap() failed somewhere15:21
voidspaceheh, possibly from space discoovery15:21
voidspacedimitern: well yes...15:21
dimiternbut on a jsonobject, not a maasobject15:21
dimiterni.e. while processing a response15:21
voidspacethat's the error message we usually get hitting a 1.0 endpoint against 2.015:22
voidspacebut I'm trying to work out where15:22
voidspacethe error message is pointing me to a non existent line in supercommand.go and --debug provided no extra information15:22
voidspacealthough the debug line before it is immediately before a call to NewEnviron - which would report that error message if it thought the feature flag wasn't set15:23
voidspacedimitern: where in juju are feature flags set on the controller machine15:24
voidspacedimitern: if it's after we attempt to open an environ then we'll fail in this way15:24
voidspacewhen running jujud on the controller15:24
dimiternvoidspace: let me check exactly15:24
dimiternvoidspace: cmd/jujud/main_nix.go15:25
voidspacedimitern: I see a call to SetFlagsFromEnvironment in jujud/main_nix.go15:25
voidspacedimitern: right, but what puts them in the environment - cloud init?15:26
dimiternvoidspace: no, they are part of the agent config we pass via the userdata15:26
dimiternvoidspace: check also cmd/jujud/agent/machine.go - in the beginning or Run()15:27
dimiternyou could grep for "developer feature flags enabled" in the logs15:27
voidspaceok15:28
voidspacedimitern: thanks15:28
alexisbericsnow, when you have a moment can you kindly review http://reviews.vapour.ws/r/4583/15:29
ericsnowalexisb: will do15:44
dimiternbabbageclunk: you've got a review btw15:47
voidspacedimitern: babbageclunk: as I suspected. With maas2 including babbageclunk's branch *and* my gomaasapi fix, bootstrap now dies with16:06
voidspace2016-04-14 16:03:57 ERROR cmd supercommand.go:448 MAAS 2 is not supported unless the 'maas2' feature flag is set16:06
voidspacedimitern: babbageclunk: so the feature flag isn't being propagated correctly / early enough to the controller machine16:06
dimiternvoidspace: you're not seeing the log saying they're enabled?16:07
voidspacedimitern: haven't checked yet - we haven't touched that code path though!16:09
voidspaceoff to the tatooist16:09
voidspacewill look when I return16:09
dimiternok16:09
=== Spads_ is now known as Spads
tych0rick_h_: jam: frobware: https://github.com/juju/juju/pull/516416:26
dimiterntych0: agent.LxdBridge is never ever set16:28
dimiterntych0: only agent.LxdBridge is16:28
dimiternif that16:28
tych0dimitern: sorry, i don't understand?16:28
dimiterntych0: e.g. here https://github.com/juju/juju/pull/5164/files#diff-7db54798352f1e675c4e2ecba7bc349dR5716:29
dimiternor the one below16:29
dimiternin MaintainInstance16:29
tych0dimitern: you said "agent.LxdBridge is never ever set only agent.LxdBridge is"16:30
frobwaretych0: btw, the merge should be into next16:30
dimiternagent.LxcBridge is non-empty if explicitly set by a provider - MAAS and EC2 used to do that, but no longer do16:30
dimiterntych0: sorry :)16:30
dimiternso 'only agent.LxcBridge is'16:31
dimiternbut as I said, now agent.LxcBridge is no longer set and is always empty16:32
dimiternthe confusion comes from bad naming - agent.LxcBridge should've been called agent.ContainerBridge16:33
dimiternfrobware: gofmt breaks alignment when it finds a blank line16:33
tych0dimitern: ok. so you're saying we should just delete that entirely and always use lxcbr0? or?16:36
tych0i don't actually know where that comes from, i just figured it was configuration from the user16:36
dimiterntych0: yes, I think that's correct16:36
tych0dimitern: i guess i'm a little gunshy about making that change16:37
tych0since i don't understand any of this very well :)16:37
dimiterntych0: long, long ago there was a "network-bridge" setting you could use to override agent.LxcBridge, but it's long gone16:37
voidspacebabbageclunk: have you made much progress on volumes in gomaasapi?16:38
tych0dimitern: ok. it seems like that should be part of a larger change to get rid of it everywhere else then i guess?16:39
voidspacebabbageclunk: it would be good to let Tim know where you got to in the status doc16:39
tych0i can drop that patch if you think it doesn't matter though16:39
babbageclunkvoidspace: nope - struggling to understand how the current code works.16:39
dimiterntych0: so now unless the provider populates ContainerBridgeName in the BootstrapParams passed to providercommon.Bootstrap(), agent.LxcBridge won't be set in the agent config16:39
babbageclunkvoidspace: It seems to rely on attrs that aren't in the 1.9 JSON.16:39
voidspacebabbageclunk: can you write it up in the doc - Tim can look at it or we can feature request the maas guys16:40
babbageclunkvoidspace: writing my own little test harness16:40
voidspacebabbageclunk: cool16:40
voidspacebabbageclunk: maybe there's another api to get the information16:40
mupBug #1570473 opened: juju lxd bridge detection fallback is not reliable <conjure> <juju-core:New> <https://launchpad.net/bugs/1570473>16:44
alexisbericsnow, or dimitern. or frobware : this is a high priority PR for review today: http://reviews.vapour.ws/r/4598/16:50
ericsnowalexisb: k16:50
ericsnowalexisb: already looking at it :)16:51
alexisbsweet :)16:51
dimiternalexisb: I've already added comments and discussed a few points with tych016:52
dimiterntych0: apart from using the always empty agent.LxdBridge (or agent.LxcBride) - LGTM16:52
tych0dimitern: yeah. i guess i'm not super comfortable getting rid of that because i don't really know how it works16:53
tych0it seems like if we want to get rid of it, we should get rid of it everywhere16:53
mupBug #1570473 changed: juju lxd bridge detection fallback is not reliable <conjure> <juju-core:New> <https://launchpad.net/bugs/1570473>16:53
dimiterntych0: sgtm16:53
mupBug #1570473 opened: juju lxd bridge detection fallback is not reliable <conjure> <juju-core:New> <https://launchpad.net/bugs/1570473>16:59
ericsnowtych0: FYI, ship-it17:00
ericsnowtych0: (with one small comment)17:00
tych0ericsnow: no, that constant isn't exported in the LXD package; i moved it to lxdclient because we needed it there17:10
ericsnowtych0: sounds good17:11
tych0how do i change the branch target?17:11
tych0seems lik ei might need a new pr?17:11
ericsnowtych0: of the PR?  yeah, make a new PR and link to the old review request17:11
tych0ericsnow: ok, cool. and then i'm good to merge right away?17:12
ericsnowtych0: yep17:12
tych0ok, cool17:12
tych0ericsnow: wait, next is older than master?17:13
ericsnowtych0: no, though it may have temporarily diverged a little17:14
tych0ok17:14
alexisbtych0, remind me, what version of lxd did the switch to lxdbr0?17:21
alexisbwas it rc9??17:21
tych0i think so17:22
* tych0 looks17:22
tych0yeah17:23
tych0rc917:23
perrito666mm, how long until its morning in nz?17:24
* perrito666 needs a hand from menn017:24
alexisbperrito666, you have about 2 hours17:26
perrito666one of the fun things of this job :p one question I hardly thought I would be making17:26
redirbbiab17:27
alexisband tych0 do you have the link handy for your insights write-up on lxd init and bridge setup?17:27
alexisbcheryl linked me to is yesterday but now I cant find it :)17:28
alexisbtych0, nevermind17:28
alexisbfound it17:28
alexisbsorry17:28
perrito666agh why are the tests that take the longer the ones that always fail17:28
tych0alexisb: cool, np17:28
perrito666dimitern: voidspace can any of you make anything of the first error in https://pastebin.canonical.com/154358/  ?17:35
dimiternperrito666: looks like map ordering issue?17:37
alexisbkatco, did you add channels to the release notes?17:37
katcoalexisb: no17:38
katcoalexisb: we didn't do the front-end work... did it not make it in there?17:38
alexisbnope17:38
dimiternperrito666: in any case, feel free to skip/ignore or just delete this test, as it's no longer relevant - uses state.NetworkInterface which must be removed (no longer used) - just haven't got there yet myself17:38
alexisbkatco, would you be up to adding soemthing?17:38
alexisbwe need it to release17:38
katcoalexisb: yeah adding now17:38
alexisbthanks17:38
alexisbheading under "Whats new for beta4" please17:39
katcoalexisb: do you think this should be an overview of channels, or simply a blurb stating that they exist17:42
alexisbkatco, I think an overview would be nice to have17:42
alexisbso that people know17:43
alexisbbut it doesnt have to be overly detailed17:43
katcoalexisb: ok, i'm going to ping someone from the CS side of things as they are way more familiar17:43
alexisbfair enough17:43
perrito666dimitern: tx, Just making sure everything tests properly with mongo3 and was not sure if I should pay attention to that test17:43
katcoalexisb: are you fine with me linking to our already excellent documentation, and then providing info about juju's command line? https://jujucharms.com/docs/devel/authors-charm-store#entities-explained17:52
alexisbkatco, yes that is fine17:53
katcoalexisb: k17:53
perrito666hey, I suddenly have to go for like an hour, ill be back later, mail me if you need anything17:59
=== redir_afk is now known as redir
voidspaceperrito666: no idea without digging into it, sorry18:16
perrito666Voidspace no worries dimitern told me what I needed18:28
mupBug # changed: 1450299, 1538303, 1554675, 1556207, 1559099, 1560391, 1564694, 1567017, 1567020, 1568092, 156998219:42
perrito666I see some bugs changed19:44
katcoericsnow: i'm in our 1:1 if you're ready20:05
alexisbwallyworld, when you are in please ping me20:30
wallyworldalexisb: give me 520:31
alexisblol20:31
alexisbit is not urgent20:31
katcowallyworld: you are a robot, i'm convinced20:31
alexisbkatco, me too20:34
alexisbconvinced that wallyworld is a robot20:34
alexisbfueled by hoity-toity coffee20:35
redirwall-eworld20:35
alexisblol20:35
alexisbthat was good redir20:35
redir:)20:35
wallyworldalexisb: zup?20:39
alexisb1x1 HO20:40
wallyworldok20:42
mupBug # changed: 1426729, 1516668, 1524077, 1533262, 1537620, 1538735, 1543223, 1553272, 1554251, 1554687, 1555083, 1555248, 1556249, 1560201, 1560511, 1560520, 1560531, 1560595, 1560665, 1560667, 1563576, 1563615, 1563628, 1563762, 1563843, 1563845, 1563853, 1563923, 1563924, 1563927, 1563928,20:42
mup1563938, 1563958, 1564057, 1566237, 1566589, 1566628, 1567182, 1567228, 1567683, 1568312, 1568390, 1569024, 1569097, 1569196, 1569408, 156972520:42
thumperhi ho20:47
thumperhi ho20:47
* thumper thinks he knows what's wrong with maas2 bootstrapping20:47
mupBug #1570594 opened: read access to admin model allows grant <docteam> <juju-core:New> <https://launchpad.net/bugs/1570594>21:09
alexisbthumper, and it is your fault21:18
alexisbthe bootstrap issue21:18
thumperalexisb: there is another one :)21:18
thumperprobably also my fault21:18
thumperbut from much earlier21:18
thumperwallyworld: would love a chat when you have a minute21:22
thumperdamn...21:24
wallyworldthumper: sure, after release standup21:29
thumperwallyworld: s'ok, I think I've sorted it out21:29
thumpercode has changed from what I remembered it being21:30
thumperand I was having to work through things21:30
* thumper crosses his fingers21:41
thumperoh... getting close...21:45
thumperfuck yeah!!!21:47
thumperalexisb: bootstrap maas2 succeeded21:47
perrito666I take you found it?21:47
* thumper tries deploy21:48
thumperhmm...21:48
thumperwat21:48
mgz_so, the correct final step after dpkg-reconfigure lxd on a fresh xenial,21:48
mgz_is `systemctl restart lxd`, right?21:48
thumpercan't deploy?21:48
thumperwat?21:49
perrito666mgz_: dpkg should do that for you21:49
perrito666mgz_: in any case, if it doesnt, lxd-bridge21:49
mgz_it does, but that doesn't create the bridge21:49
thumperhttp://paste.ubuntu.com/15839571/21:49
perrito666mgz_: lxd-bridge is the service to restart21:49
thumperanyone had deploy issues?21:49
mgz_okay21:49
thumpertrying to deploy ubuntu charm dies talking to charmstore21:49
perrito666mgz_: did you instruct dpkg to create the ipv4 network?21:50
mgz_yeah21:50
thumperWAT? debug-log not supported?21:51
alexisb\o/21:51
alexisbthumper, that is freak'n awesome!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!21:51
thumperalexisb: except debug log isn't working21:51
wallyworldperrito666: did you want to chat now?21:51
thumperdeploy isn't working21:51
thumperand neither of these things are maas bugs21:52
wallyworldthumper: that error looks like juju not charm store21:52
alexisbheh baby steps21:52
alexisband at least you know enough about the bug to know that21:52
thumpermaybe it is maas's fault21:55
thumperjujud is panicing21:55
alexisbthumper, i jut deployed on lxd provider on latest next build21:55
thumpernice21:56
menn0thumper: I did think of this, and then forgot. Hooray for tests :) http://paste.ubuntu.com/15839652/22:00
thumper:)22:01
thumperalexisb: the failure was due to the maas provider needing to do subnet/space discovery the new way22:01
thumperso I'll leave that for voidspace22:01
thumperand get on to the filesystem bits of gomaasapi22:02
perrito666wallyworld: going22:02
perrito666wallyworld: ??22:04
wallyworldperrito666: cpu 100%, hangout frozen22:04
alexisbthumper == gomaasapi guy22:08
alexisb;)22:08
* thumper goes to make a coffee22:09
* thumper looks to see who is on-call reviewer22:43
thumperericsnow: still around?22:43
ericsnowthumper: yep22:44
thumperI'm just proposing a very simple branch that we need for maas222:44
ericsnowthumper: k22:44
thumperdamn it22:45
thumperproposed agains master , not next22:46
* thumper redoes22:46
thumperericsnow: http://reviews.vapour.ws/r/4603/diff/#22:47
ericsnowthumper: ah, bootstrap-state :)22:48
thumperericsnow: were you around when voidspace was having these issues?22:49
ericsnowthumper: around but not involved22:49
* thumper nods22:49
ericsnowthumper: Windows isn't a concern here, right?22:50
thumperericsnow: no, because we only bootstrap on ubuntu22:51
thumperwindows is currently workload only22:51
thumpernot apiserver22:51
ericsnowkatco: ^^^22:51
ericsnowthumper: sounds good22:51
ericsnowthumper: ship-it22:52
thumperericsnow: ta22:52
menn0wallyworld: an old MongoDB HA bug has resurfaced (it's one of the blockers)23:46
wallyworldmenn0: which branch? next?23:46
wallyworldbug number?23:47
menn0wallyworld: yep on next23:47
wallyworldmenn0: it may be fixed in master23:47
wallyworldwe fixed a bunch of ha stuff for beta423:47
mgz_they will be reconverging shortly23:47
menn0wallyworld: bug 145380523:47
mupBug #1453805: Juju takes more than 20 minutes to enable voting <blocker> <ci> <ensure-availability> <intermittent-failure> <regression> <juju-core:Triaged by menno.smits> <juju-core 1.23:Fix Released by menno.smits> <juju-core 1.24:Fix Released by menno.smits> <https://launchpad.net/bugs/1453805>23:47
menn0wallyworld: ok that's good to know23:47
wallyworldoh i haven't see that bug23:47
menn0wallyworld: it's an old one that aaron reopened because the symptoms look the same23:48
menn0wallyworld: what happens is that after enable-ha the new controller hosts come up and the agents can connect to MongoDB but then get disconnected23:48
menn0wallyworld: we don't have the mongodb logs to confirm what's going on23:48
wallyworldjoy23:49
menn0wallyworld: but off memory I think that can happen when the replicaset isn't ready yet23:49
menn0wallyworld: it's intermittment, I can't replicate it23:49
wallyworldsigh23:49
menn0wallyworld, mgz_ : we really need those MongoDB logs to know what's happening23:49
wallyworldif it's an existing bug why is it a regression?23:49
menn0wallyworld: it was fixed in 1.24 and 1.25 and has now come back23:50
menn0wallyworld: it could well be a completely different cause23:50
wallyworldi'd say so because i don't think we messed with those bits23:50
wallyworldbut there were a lot of changes23:50
menn0wallyworld: what about those changes to mongodb setup in the machine agent that you made? (all that deleted code)23:51
menn0wallyworld: could that reorg have something to do with it?23:51
wallyworldthe deleted code was for pre ha environments where stuff wasn't set up yet for replication23:51
wallyworldthat setup is now done in bootstrap23:51
menn0wallyworld: yeah... seems unlikely23:52
menn0wallyworld: looking at the failures it's happening in master and next23:52
wallyworldand i think next was branched before my changes23:52
wallyworldbut i did notice it took a while to transition to has-vote23:52
wallyworldi just thought it was mongo behaving as normal, because well, you know, mongo is web scale23:53
menn0wallyworld: not 20mins though right/23:54
menn0?23:54
wallyworldnot sure tbh23:54
wallyworldmaybe 5?23:54
redirheading out for a while. I'll check back later this eve to see if things merged...23:54
wallyworldlet's hope so23:55
=== redir is now known as redir_afk
menn0wallyworld: 5 is acceptable I think, 20+ is not23:56
wallyworldeven 5 seems unforntunate23:56
wallyworldi mean, wtf is it doing23:56

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!