[00:00] ericsnow: i've pushed $$merge$$ - perhaps you could take it over if there are some silly errors from the bot? [00:00] ericsnow: indeed [00:00] rogpeppe: sure [00:00] ericsnow: but this should actually work reasonably for most use cases [00:00] ericsnow: thanks [00:00] ty, ericsnow [00:00] * rogpeppe beds headward [00:00] rogpeppe: np [00:00] g'night all [00:32] Bug #1560262 opened: relation visibility rules different between service/service and service/subordinate relations

[01:02] axw: with the restore issue - reading the code leads me to believe the issue is because we no longer store admin-secret in bootstrap config. we could add it back but wouldn't it be better to store it in the backup metadata? [01:02] wallyworld: admin-secret is the password for admin@local, should just grab it out of accounts.yaml instead [01:02] ah, yes, good point [01:03] that does ite the restore to the user who did the backup [01:03] wallyworld: (it would be better to pull something out of backup metadata tho) [01:03] agreed [01:03] i may fix the easy way for now [01:03] wallyworld: +1 [01:03] i'd rather the backup be stand alone [01:03] wallyworld: needs an overhaul, so yeah [01:22] axw: joy, we also need ca-cert which is fine, but also ca-private-key which we don't currently store with the controller metadata [01:23] wallyworld: what do we need that for? [01:23] I thought we would generate new certs [01:24] i'll have to look deeper - the current code attempts to create an env config and it complains [01:26] thumper: cherylj https://github.com/juju/juju/pull/4836 [01:26] axw: atm we don't prepare or anything - the code just assumes we had a complete config available via bootstrap config, so it needs to be reworked [01:27] wallyworld: axw: last change for openstack virt type http://reviews.vapour.ws/r/4276/ [01:27] anastasiamac: looking [01:28] axw: tyvm \o/ [01:28] wallyworld: yeah, it does assume we have a bootstrap config already. it always did, so the only potential issue is to do with info we may have thrown away, like ca-private-key [01:28] wallyworld: only immediate problem I mean [01:29] axw: yeah, not having the same ca-private-key seems problematic i think? [01:29] maybe not [01:30] wallyworld: sorry, I don't know what's newly generated, and what needs to be the same [01:30] i guess if we are rebootstrapping, there's no existing machine sout there [01:30] so no need for existing agents to be able to reconnect [01:31] which i think would be the main issue [01:31] wallyworld: part of the restore process involves SSHing to the agents and fixing them up, IIANM [01:32] wallyworld: the controller's going to have a new IP, after all [01:32] right but doesn't the restore assume everything is gone if we use the -b option [01:32] maybe not [01:33] i'll have to read the code [01:51] anastasiamac: just a couple of comment changes, otherwise LGTM [01:51] axw: awesome \o/ thnx :D [01:51] anastasiamac: BTW, "code is kinda weird" is not your fault, just how it is / has to be due to openstack limitation [01:55] wallyworld: you said there was another bug about permission denied? I don't see it on LP against admin-controller-model. did you say you still had to add it? [01:55] axw: it's just not targetted to that branch, let me get it [01:55] https://launchpad.net/bugs/1461561 [01:56] Bug #1461561: juju run fails with "Permission denied (publickey)" on manual provider

[01:56] ta [01:56] Bug #1441302 changed: Vivid unit tests are not reliable enough

[01:58] wallyworld: I suspect it's just a race with the worker to update authorized_keys [01:58] sounds likely yeah [02:11] Bug #1496032 changed: backups restore won't create bootstrap on GCE

[02:36] axw: reviewed [02:36] wallyworld: thanks [03:01] wallyworld: please see reply about type name [03:01] ok [03:02] axw: no reply yet, did you publish? [03:03] wallyworld: derp. published now [03:05] axw: GenericResource I think is ok, but I'm not strong;y opinionated either way [03:05] wallyworld: in that case I'll leave it. if it becomes a pattern, we should strongly consider changing the framework to allow injecting things directly [03:06] sgtm [03:06] wallyworld: do you want me to wait before merging to admin-controller-model? [03:06] wallyworld: wait for CI to be a bit happier? [03:06] may as well merge now, what could possibly go wrong :-) [03:07] we should have a run in say 5 hours [03:07] still time to fix stuff before tomorrow and better to find issues early at this stage [03:07] we could merge the other 2 fixes first i guess [03:46] gah, I see the problem with the joyent provider [03:46] they're ignoring the tags passed into startinstance [03:46] cherylj: :D [03:47] thumper mwhudson https://github.com/golang/go/issues/14904 [03:48] i'm detouring into this bug this arvo [03:48] cmd/juju/status trips it up 100% of the time [03:56] axw: this backup thing is messy - apart from old bootstrap config, the only place we store the admin-secret aka oldpassword is in the agent.conf file, and the same is true for the ca-private-key. So assuming we have that file, which the backup does include in the archive, we can parse it on restore, but i think it would be better to do that stuff server side and include in the json metadata for the backup. but that sucks because [03:56] nothing so far in apiserver needs to know about agentconfig and i'd rather keep it that way. so it's all a bit yuck [03:58] axw: maybe we should just include the private key and admin secret in bootstrap config [03:59] wallyworld: it would be nice not to, but I guess we could as a stop gap at least [04:00] axw: yeah, i don't really want to. so what would we prefer - parsing the conf files on restore? or extracting the info srver side on backup [04:00] wallyworld: we definitely can't generate new ones at restore time? [04:00] i'm not sure. the private key i think needs to be paired with the cacert [04:01] wallyworld: yeah, we would need to update that too. [04:01] davecheney: interesting... [04:02] axw: and a new admin secret would not be the end of the world i guess. but we do have all the info in the backup archive [04:02] so we'd really need to look to use that info if availab;e [04:06] wallyworld: for a quick solution, I'd go with parsing agent.conf on restore. then we can look at improving. [04:07] wallyworld: we also need to think about how we're going to persist user auth details to log back into the controller after restore, in lieu of an accounts.yaml that's valid with the creds at time of backup [04:08] wallyworld: options are to store in the backup metadata, or to require the user to login. but these are things we could look into at the same time as the rest [04:09] ok, was thinking the same thing, also we only need this if we are rebootstrapping [04:10] wallyworld: I can't make heads or tails of the SSH thing. the system key is added to authorized-keys at bootstrap, and I've confirmed they're in the hosted model config; and that is present in cloud-config when machines are started [04:10] so there's no race AFAICT [04:10] fark [04:10] * axw disappears to make lunch [05:28] axw: i think adding admin-secret to bootstrap config is ok. it's used for the gui login having it accessible like that makes sense to me [05:29] wallyworld: I don't understand why we need it there. it's already in accounts.yaml. [05:29] wallyworld: admin-secret == password for admin@local [05:29] ah true [05:30] there's not a nice way that avoids yuk to get it server side, so i'll read from there for now [05:30] parsing the agent conf files client side is also no go [05:30] so i'll get that ingo srrver side and include with backup metadata [05:31] wallyworld: why is it no go? [05:32] all the code to open the archive and extract the files is all server side. there's a lot of it. and there's no guarantee what you'll get in terms of file names - it depends on machine id. [05:32] i'd have to refactor a lot of code [05:32] wallyworld: ok [05:33] restore works by uploading the archive to state server and server side code does the work [05:33] so none of that code is client side [05:34] wallyworld: ah, I think I figured out when the race could happen with authorized-keys. if you specify it explicitly in --config [05:35] wallyworld: that'll go into hosted model config, whereas admin model's config will have the system's public key added [05:35] * axw works on a patch [05:35] ah i see [05:35] bbiab, school pickup [06:10] axw: wallyworld: tag fixes for joyent http://reviews.vapour.ws/r/4282/ [06:23] anastasiamac: is there some significance to the "tag." prefix? [06:23] anastasiamac: do we need to add that to the prepopulated tags? [06:23] axw: I am fixing the broken code. what ever was there or whatever it was copied from, i've kept [06:24] axw: but i would have to say that probably not.. [06:24] anastasiamac: I guess if you tested and it worked, there's no significance. alrighty [06:26] anastasiamac: LGTM, thanks [06:27] axw: thnx. I'll re-google tag names in joyent but m pretty it's an arbitrary prefix :D [06:48] Bug #1560331 opened: juju-br0 fails to be up when no gateway is set on interface [07:12] axw: are you around for a bit? I'd like to figure out what we need for storage and LXD containers [07:13] especially for normal deployments (maas/ec2) but also for LXD provider. [07:22] jam: sorry, didn't see notification - not on usual machine (desktop's on the fritz) [07:22] jam: still around for a while [07:23] axw: so the LXC code has something about passing through the loop device [07:23] and I need to know what we actually use, and how we need to do that with LXD [07:23] AIUI if we want a device, then we talk directly to LXD and ask it to pass a device through (might need to be in the lxd profile, I need to investigate that side) [07:24] axw: but I'd like to know what we're using and how I can measure that I've done it correctly. [07:25] jam: the existing LXC storage code wasn't great. it was simply allowing access to all loop devices to all containers [07:25] axw: who was creating the loop device to then pass in? [07:25] jam: the container does it internally. they have access to the /dev/loopX files [07:26] jam: (it would be much better if the host passed it in) [07:27] axw: but who on the outside would create the fil to then mount? [07:27] you'd ask the container broker ? [07:28] jam: there's several places where we can create storage: dynamically on the machine (e.g. loop); dynamically on the controller (e.g. ebs); at machine provisioning time (e.g. maas) [07:28] jam: the last one might fit with creating in the container broker [07:29] jam: but it would be nice if there's some way we could do it dynamically through the LXD API [07:34] axw: LXD would let us pass the device through (supposedly), but it wouldn't do the mkfs, etc. stuff [07:35] I don't believe [07:40] jam: all we need is a device. juju will create filesystems as necessary [07:40] jam: does LXD have an API for allocating block devices? I don't see it in the REST API doc, but that may just be outdated [07:41] wallyworld: https://github.com/juju/juju/pull/4844 [07:41] looking [07:44] axw: what if the user has explictly set their auth keys for the hosted model [07:45] wallyworld: how? [07:45] ah, i was thinking we had a capability for host model config on cli [07:45] wallyworld: you can only specify the config for both admin and hosted model [07:46] one day we should add that separate host model config perhaps [07:46] axw: lgtm === menn0 is now known as menn0-afk [08:05] axw: off to soccer, restore tests are a bitch, need to refactor a bit, will finish when i get back and hopefully land tonight [08:05] wallyworld: okey dokey [08:06] axw: i'm so not happy with the solution, but there's little other option right now [08:06] that we can do quickly [08:09] jam: so thinking about it a bit more, if we were to pass disks through to the containers, I think we'd need to do that in two places: an environment-level storage provider for the lxd provider; and a machine-level storage provider for lxd-type containers [08:10] jam: for hte latter, we'd need to update machine-level storage to manage disks for containers as well as disks for the machine itself [08:11] jam: AFAICT there's no help from LXD for dynamically allocating volumes from ZFS or BTRS for use in containers (apart from root volume), which would be ideal [08:11] at worst we could do what we were doing with LXC: open up loop devices to all containers and let them have at them. not ideal tho [08:12] jam: I'd prefer if we could allocate any volumes the host supports to containers, though [08:26] aaargh, building a shell script generating a shell script but then doing wrong escaping. wonderful, costed me some time. [08:27] morning btw [09:30] Bug #1560391 opened: apt-mirror is not used in containers with MAAS provider [09:48] don't suppose anyone who did backup/restore work is online? [09:51] because apparently I have perpetrated lp:1559712 and I need a bit of context about why this is an error [09:52] perrito666, ericsnow perhaps? ^ [09:54] Well i am having breakfast and chatting from a phone [09:54] But shoot [09:54] Fwereade^ [10:00] In short, prepare restore puts juju in a pseudo ro mode for that iirc, it writes info in mongo so I presume that is what is failing hence the transaction aborted [10:02] dooferlad: standup? [10:09] Fwereade gtg need to take my wife to work + not feeling so well [10:31] Bug #1560428 opened: cmd/juju/common depends on gopkg/check.v1 [10:43] Bug #1560428 changed: cmd/juju/common depends on gopkg/check.v1 [10:46] perrito666, np, let me know if/when you're available [10:46] again :) [10:49] Bug #1560428 opened: cmd/juju/common depends on gopkg/check.v1 [11:41] fwereade, davecheney, FYI - updated the pprof wiki page with instructions how to access it from a browser [11:41] awesome tool btw [11:41] https://github.com/juju/juju/wiki/pprof-facility [12:13] Bug #1560457 opened: help text for juju bootstrap needs improving [12:25] axw: not sure if you have a moment to look at http://reviews.vapour.ws/r/4287/ [12:26] axw: you may need to hold your nose, but we need to re-write backup/restore somewhat for multi-model [12:30] wallyworld: hold on that though. I want to talk through that and what's in 2.0 vs 2.1 and how we go about that please [12:30] rick_h_: oh, no fear, we have no much for anything right now :-) [12:31] s/much/time [12:31] wallyworld: k [12:32] rick_h_: what we need to do to get 2.0 out the door given how the cloud credentials stuff now removes the need for storing an entire bootstrap config means that there's a few things we need to clean up for 2.1 [12:33] wallyworld: rgr [13:17] katco natefinch how does deploying a charm with resources local work? [13:18] like, what's the command line work like? [13:18] s/work/look/ [13:18] marcoceppi: yep [13:18] marcoceppi: juju help deploy ;) [13:19] marcoceppi: uju deploy foo --resource bar=/some/file.tgz --resource baz=./docs/cfg.xml [13:19] marcoceppi: you just specify the resources you want to have uploaded when you deploy the charm. You can do that for store resources too [13:20] natefinch: I DON'T HAVE TIME FOR YOUR REASONABLE SUGGESTIONS [13:20] marcoceppi: er, store charms that is [13:20] marcoceppi: lol [13:20] natefinch: yeah, but I'm waiting for charmstore update, etc [13:20] marcoceppi: right, just wanted to clarify that it's not just for local charms [13:20] natefinch: yeah, lookikng forward to that, but I've got a charm now with resources [13:21] natefinch: I know there's a filename param, etc [13:21] natefinch: and I remember that it's mostly ignored [13:22] marcoceppi: the extension of the file you upload needs to match the extension of the filename in the charm metadata, it'll complain if you don't have them match [13:22] natefinch: okay, one file is literally `python-jujusvg` with no ext [13:22] what happens then? [13:23] as in, it's just a binary [13:23] marcoceppi: then we enforce that what you upload doesn't have an extension [13:23] cool [13:23] <3 [13:24] marcoceppi: yeah, the filename in the metadata is really just used for the extension for uploads, and its' what we store the file data as on the units... though since we return the full path from resource-get, it shouldn't matter too much to the charm itself [13:24] natefinch: perfect, I remember that being shook out in capetown happy to see it implemented as such [13:25] marcoceppi: yeah, I'm mostly really happy with how the feature turned out. I think it'll be super useful. [13:26] natefinch: a few days ago the instance containing svg.juju.solutions died, so I'm charming that up with resources. I'll let you know how it goes [13:26] marcoceppi: cool cool. Please let us know any pain points or other sharp edges [13:43] Bug #1560487 opened: local provider fails to create lxc container from template [13:45] does anyone know the difference between jujuclient.Controller.Servers and jujuclient.Controller.APIEndpoints ? [13:46] it's a new type, so *presumably* they both have a role to play, but in my limited experimentation they both hold exactly the same thing. [13:47] fwereade, dimitern, natefinch, axw, wallyworld: ^ [13:48] rogpeppe: no idea, I haven't used that package [13:48] natefinch: np [13:48] rogpeppe: they mean the same as when used with the jenv stuff - servers are the host names, api endpoints the ip addresses [13:48] wallyworld: what's the difference? both seem to contain host names and port numbers [13:49] wallyworld: ah, you mean one has resolved IP addresss + port and the other has the equivalent hostnames ? [13:49] rogpeppe: as i understand it, if the hostnames are known they are used [13:49] yes [13:49] wallyworld: that is *really* confusing! [13:49] wallyworld: just the naming really [13:49] rogpeppe: that stuff goes back a looong time [13:49] we're really bad at naming [13:49] wallyworld: it's a new type! [13:49] which we ported [13:49] copied even [13:49] wallyworld: hrmph [13:50] we didn't want to change the semantics [13:50] as people would have groked the meaning b now [13:50] wallyworld: the semantics perhaps didn't need to change, but at least it could be documented that a) they both hold host:port pairs and b) one is resolved and the other isn't [13:50] wallyworld: ...or not :) [13:50] rogpeppe: it was probaly around your time on juju when this was first done :-) [13:51] wallyworld: i still can't work out which is resolved and which isn't [13:51] it's as documented now as it evr was [13:51] wallyworld: you're just digging your hole deeper [13:51] why? [13:51] we can't change the world in a day [13:51] wallyworld: if it's been the same for forever and it's been this well documented for forever, that just means it's bad and should be fixed. [13:51] wallyworld: just about everything else has changed about that stuff [13:52] yes it should, let me get 25 hours in a day and i'll do it [13:52] wallyworld: so i'd've thought that aspect could probably change too [13:52] not everything has changed, that bit hasn't :-) [13:53] how is anyone meant to guess what the intended relationship between these two things? [13:53] // Servers contains the addresses of hosts that form the Juju controller cluster. [13:53] // APIEndpoints is the collection of API endpoints running in this controller. [13:53] they sound very different from one another. [13:53] guess so, that doc was as we founf it [13:53] a lot of this stuff has been ported, not reimplemented [13:54] you don't need to understand everything to the nth degree if you are porting it [13:54] wallyworld: it's actually got worse since the original, which was this: http://paste.ubuntu.com/15472129/ [13:55] wallyworld: that at least explained the situation [13:55] we should copy that doc back across [13:55] wallyworld: +1 [13:57] wallyworld: BTW both the field names *and* the comments have changed from the original [13:58] yes, can't recall why now, may have been to comply with the spec we were given actually [14:01] morning all [14:01] wallyworld: evening o/ [14:01] hi [14:01] wallyworld: doing ok? [14:01] no, life sucks [14:02] someone's not living the dream today :/ [14:02] wallyworld: understandable atm... but the sun will rise on a better day :) [14:02] * wallyworld is grumpy [14:02] wallyworld: anything i can do to help? [14:02] wave a magic wand and solve all the merge conflicts and test failures [14:02] wallyworld: sorry if i excacerbated the situation :) [14:02] rogpeppe: np, i have copied the text across already [14:02] wallyworld: i can work on merges today if you'd like [14:02] wallyworld: thanks [14:03] wallyworld: changing the names would be good too. Something like: APIEndpoints and ResolvedAPIEndpoints ? [14:03] katco: thanks, bu there's a lot of ingrained knowledge needed to resolve the conlficts [14:03] frobware: is there no maas call today? [14:03] rogpeppe: i'll see what we can do [14:03] wallyworld: ok. well, lmk [14:04] will do [14:04] katco: you could review http://reviews.vapour.ws/r/4287/ - it's a temporary fix to get restore working, still a long way to go to fix it all properly [14:04] wallyworld: tal [14:05] wallyworld: thanks, that's appreciated [14:06] rogpeppe: is your 070-use-charmstore-v5-api branch something we need to get into the beta this week? [14:07] cherylj: no, sorry, it was a temporary hack to get a PR in a mutually dependent repo landed. I can delete it now. [14:07] rogpeppe: ah, ok, thanks! [14:08] cherylj: deleted [14:08] thanks! [14:09] cherylj: and BTW if you ever need to land something in a repository which both depends on an is depended on by juju-core, that's the way to do it and still have godeps work. [14:09] s/on an is/on and is/ [14:09] good to know. Thanks :) [14:09] rogpeppe: FYI, that merge went through on the first try :) [14:09] ericsnow: yeah, i saw in the morning and was happy [14:09] :) [14:10] ericsnow: thanks for keeping an eye on it [14:10] rogpeppe: np [14:11] rogpeppe: hey thanks for working so hard to land that patch :) work continues i guess? [14:11] katco, if you know backup: ISTM that creating a RestoreInfoSetter should not create a document; or, if it should, that it ought to handle concurrent creations of the same type [14:11] katco, is there some synchronisation mechanism somewhere in restore that I've missed? [14:11] fwereade: eh? is this regarding wallyworld's patch? [14:12] katco: i'm leaving as is for the time being, as i have to make progress in other areas. [14:12] katco, not at all I'm afraid [14:12] katco: it should be sufficient for most purposes. there may be some bugs :) [14:12] fwereade: ah. i don't know hardly anything about backup/restore i'm afraid :( but ericsnow and natefinch should [14:12] katco, np, thanks [14:13] fwereade: i'm reviewing a patch now, so maybe i'll become an expert in the next bit of time ;) [14:13] i don't recall doing anything with RestoreInfSetter in my patch? [14:13] katco: hope that's ok. did you have any other particularly pressing things? [14:13] ericsnow, natefinch: ^ ? (RestoreInfoSetter) [14:13] rogpeppe, katco: there are a few things left (like sending the channel through the AddCharm API endpoint) [14:13] wallyworld, nothing to do with you [14:13] perrito666: is Mr. Restore [14:13] ericsnow: file a bug [14:13] fwereade: perrito666 did all the work on restore, though I did review much of the work [14:13] wallyworld, my problem entirely :) [14:13] rogpeppe: the only horse i have in that race is natefinch being able to determine what channel a charm was deployed from [14:13] \o/ [14:13] rogpeppe: k [14:14] * katco head explodes from the different threads of convo [14:14] perrito666, last I heard, was not feeling well [14:14] gah.. people talk about sending channels around in go and I get all confused [14:14] fwereade: :( [14:14] ericsnow, I will do my best and ping you for a review [14:14] fwereade: sounds good; thanks! [14:15] natefinch: ha [14:15] ericsnow: ty [14:15] katco: yeah, currently it'll probably always use stable. [14:16] fwereade: I am back fully restored [14:16] fwereade: how can I help you? [14:16] rogpeppe, katco: we still need to sort out the long-lived macaroon situation, but I don't think it's as urgent [14:18] where can i find a list of all the valid Request types for each api facade? [14:19] tvansteenburgh: I don't believe that is cataloged anywhere, meaning you have to read through the facades code and apiserver/params/*.go [14:20] tvansteenburgh: yeah, unfortunately, what ericsnow said. We really should document all that stuff. [14:20] tvansteenburgh: godoc might help make it more manageable though [14:20] do we have something like this for stderr? https://github.com/juju/cmd/blob/master/output.go#L157 [14:21] ericsnow, natefinch: ok, well if i know the name of the facade, where in the source should i go to find the Request types for it? [14:21] tvansteenburgh: all the API data types are lumped together in several files in apiserver/params.go [14:22] tvansteenburgh: so you have to see what types are in the API methods and then find them in those params files [14:22] perrito666, so the heart of it is that creating a RestoreInfoSetter will sometimes fail [14:23] it being the bug you mentioned early today? [14:23] perrito666, which is presumably happening now because of timing changes in my branch [14:23] perrito666, yeah [14:23] perrito666, and I am worried that just fixing the current symptom is going to leave us unstable all the same [14:24] fwereade: see priv msg [14:28] wallyworld, ping [14:28] yo [14:28] hey, I've found a storage issue with bootstrapping on maas now [14:28] http://paste.ubuntu.com/15472360/ [14:29] looks like related to removing provider storage ? [14:29] ericsnow: i don't see a apiserver/params.go? [14:30] tvansteenburgh: apiserver/params/params.go [14:30] (in core) [14:31] dimitern: maas still has provider storage [14:31] it can't use tags the same way as the other providers, so we still need ito [14:32] are you referring to this line? DEBUG juju.provider.common state.go:36 putting "provider-state" to bootstrap storage *maas.maasStorage [14:32] wallyworld, nope - it fails like joyent - see the paste above - just after it connected once client logins were unblocked [14:33] dimitern: didnt remove storage in maas [14:33] oh wallyworld said that already [14:33] dimitern: so why do you think it's a storage issue? [14:33] oh :/ I was hoping it was something known.. I'll keep digging [14:34] dimitern: cherylj had a tgeory [14:34] theory [14:34] can't recall exactly what now [14:34] wallyworld, joyent precise jobs failed very similarly and intermittently and it was due to missing tags [14:34] * cherylj reads backscroll [14:34] dimitern: joyent iirc, was a porblem with tags not being populated as fast as they where requested [14:35] perrito666 - the problem with joyent was that they were not setting tags passed in as StartInstance args [14:35] ah, I got it wrong then :) [14:36] perrito666: no worries, I didn't discover that until adding in the logic to retry the api connection in bootstrap if the error was "not bootstrapped" [14:36] I saw that even after several minutes, we never got tagged instances [14:36]