[00:00] cherylj: so, we'll need to get that fix for bug 1567170 fixed [00:00] Bug #1567170: Disallow upgrading with --upload-tools for hosted models

[00:00] yes, it landed in next, so we have some time [00:00] ah phew ok [00:02] wallyworld: so juju is looking for mongodump in /usr/lib/juju/bin/mongodump? [00:02] mwhudson: from memory, yreah [00:02] ERROR while creating backup archive: while dumping juju state database: error dumping databases: error executing "/usr/lib/juju/bin/mongodump": 2016-04-13T00:02:21.072+0000 error parsing command line options: --dbpath and related flags are not supported in 3.0 tools.; See http://dochub.mongodb.org/core/tools-dbpath-deprecated for more information; 2016-04-13T00:02:21.072+0000 try 'mongodump --help' for more information; [00:02] sounds like juju needs moar fixing [00:03] but well no reason not to upload my package [00:03] mwhudson: yeah, juju is currently hard coded to look in the mongo 2.4 binary path for mongodump [00:03] wallyworld: I fixed my patch, can you look at it so I can merge it after dinner? [00:03] perrito666: will do [00:04] mwhudson: it should be a quick fix to sort out that mongodump path, i'll do that today [00:04] wallyworld: that was after i copied the 3.2 tools to /usr/lib/juju/bin/ [00:04] wallyworld: looks like the command line needs to change too [00:04] oh right, yes, i didn't read the error [00:04] damn it [00:04] BTW MIR for this is going to be a doozy [00:05] yes [00:05] but that's not today's problem either [00:05] mwhudson: so we have today to get a blessed beta4 for includsion in xenial main [00:05] is my understanding [00:06] wallyworld: well i guess Someone (tm) should get to working filing MIRs for the deps [00:06] mwhudson: yeah, i must admit i have nfi about the process with all that [00:06] it is being driven by others [00:07] mwhudson: but i thought we had permission to put mongo packages needed for juju into main [00:07] along with juju itself [00:07] isn;t that already approved? [00:08] wallyworld: oh maybe [00:08] i thought so but am out of the loop a little on all that [00:29] wallyworld, axw: I'm thinking about picking up bug 1456916. this also relates to these old tickets: bug 892552 and 802117 [00:29] Bug #892552: juju does not extract system ssh fingerprints

[00:30] menn0: it will be a fair bit of work [00:30] wallyworld, axw: to me, the easiest win seems to be to use per-model known hosts files for juju ssh and juju scp (instead of /dev/null) [00:31] wallyworld: why would we embed base suite? [00:31] perrito666: because that supresses logging output to console and other things [00:31] k [00:32] wallyworld, axw: that doesn't help with the "first connect" problem but does prevent MITM after the first connect while avoiding the scary warnings when addresses are reused between models [00:32] menn0: seems ok to me - so long as we can extract the fingerprints from cloud init, but i am not an expert [00:32] I guess there's still the problem of killing a machine and recreating it in the same model which could lead to the same address being reused [00:32] that is the issue [00:33] one of [00:33] wallyworld: yeah, one of [00:36] wallyworld: ok so the ultimate solution would be for juju to extract the ssh key for each new machine from cloud-init and store it... and then have juju ssh and juju scp get told the key before connecting so it can be written into the client's known_hosts [00:36] wallyworld: something like that? [00:37] menn0: yep [00:37] menn0: +1 [00:37] menn0: i assume we pass pass a bespoke known hosts file to the client each time [00:39] mwhudson: are we putting juju-mongo3.2 into trusty? [00:41] cherylj: i think you did a fix so that if mongo 3.2 were not found, it would not retry? ill need to retest, but it appeared not to work when i bootstrapped a trusty controller. took ages to bootstrap and cloud init was filled with mongo3.2 retries [00:42] i may be able to get a fix done today if i can sort out some other issues [00:46] wallyworld: we don't pass any known_hosts, we pass /dev/null. that's part of the problem [00:47] I railed against that at the time it was done, but the change went in anyway [00:47] sigh [00:47] axw: awesome, i removed juju-mongodb (only have mongo 3.2 installed) and tests fail \o/ [00:48] wallyworld: :/ [00:48] wallyworld: I'm guessing testing/mgo.go needs updating to look in the right spot [00:48] yep [00:48] i might worry about that later [00:53] perrito666: btw, removing ensureAdminUser is fine; need to double check restore, but bootstrap etc all works as expected [00:53] sweeeet, this is a great week [01:08] see you tomorrow juju-dev === redir is now known as redir_eod [01:16] wallyworld: yeah, I did. The fix was in juju/utils, so you'll need to make sure your deps are up to date to pull it in [01:18] ok, i'll double check [01:19] cherylj: ah, i changed my scripts (long story) and my godeps one had an issue, utils just got updated, so i'll retry [01:20] ok, phew [01:20] :) [01:20] menn0: if you get a minute https://github.com/juju/gomaasapi/pull/33 [01:22] mwhudson: when you upload the mongo 3.2 tools, the package will get installed automatically with mongo 3.2 db right? [01:23] * thumper needs food [01:23] * thumper goes to hunt in the kitchen [01:24] mwhudson: i'm testing a fix for that mongodump issue (the path and the args) and have no tools available - can you provide me wit the deb i can install manually? [01:24] Bug #1569632 opened: status: decide on statuses for migration [01:30] Bug #1569632 changed: status: decide on statuses for migration [01:33] wallyworld: it's in ppa:juju/experimental [01:33] ta [01:33] wallyworld: juju-mongo-tools32 [01:33] wallyworld: juju-mongo-tools3.2 [01:33] rather [01:38] mwhudson: yay, my fix worked, backups good now [01:38] wallyworld: nice [01:39] mwhudson: but we'll need that tools deb in the repos etc before beta4 goes out [01:39] wallyworld: slangasek said he'd look at it today [01:39] awesome [01:45] Bug #1569632 opened: status: decide on statuses for migration === alexisb-afk is now known as alexisb [01:48] wallyworld, menn0, axw is one of you available to look at: https://bugs.launchpad.net/juju-core/+bug/1569467 [01:48] Bug #1569467: backup-restore loses the hosted model

[01:49] I suspect that it has come from the made-model-workers branch [01:49] alexisb: this is a known deficiency of backup/restore, meant to be fixedin 2.1 or later IIRC [01:49] but it's just a theory at this point [01:49] I don't think it is [01:49] I can do this test manually and it works [01:49] It's just failing in CI [01:49] for some reason [01:49] cherylj: oh? I was under the impression that backup/restore didn't work with hosted models [01:50] axw: the backup is being created from the admin model [01:50] axw, BR should work on the admin model [01:50] ok [01:50] the problem that's happening in CI is that the dummy hosted model that is created to rebootstrap with is not removed [01:50] and the existing models don't seem to get "hooked up" === natefinch-afk is now known as natefinch [01:51] I'm wondering if the restore is actually dying in the controller [01:51] i'm working on some other backup fixes for mongo 3.2 (with tools), can look a bit later [01:51] and it doesn't complete [01:51] I think we need another run with the environment kept after the failure [01:51] thank you wallyworld we leave it in your capable hands [01:51] I'll update the bug with where I am with things [02:07] anyone seen messages like this when a unit gets a hook error? https://paste.ubuntu.com/15803425/ [02:08] juju resolved --retry doesn't seem to get picked up unless i reboot the machine that it's happening on [02:08] should i open a bug? [02:09] cmars: probably [02:09] ok [02:13] d'oh, i already did. ok, attached a full log to LP:#1566130 [02:13] Bug #1566130: awaiting error resolution for "install" hook [02:28] wallyworld, anastasiamac: I have a question about the imagemetadata api endpoint. It has code to default the arch to amd64: https://github.com/juju/juju/blob/master/apiserver/imagemetadata/metadata.go#L184 ...when would that be triggered? Why would we ever get image metadata that doesn't specify an arch? Isn't the arch like one of the most important pieces of information? [02:29] natefinch: hopefully never... [02:29] (as in hopefully nver triggered) [02:29] anastasiamac: the reason I ask is because of bug #1564791 [02:29] Bug #1564791: 2.0-beta3: LXD provider, jujud architecture mismatch

[02:30] natefinch: so when u reproduce the bug, u fall into this code? [02:30] anastasiamac: when using LXD, machines after the bootstrap machine on non-amd64 hosts are downloading amd64 juju binaries for some reason, which obviously don't work [02:31] anastasiamac: it's easy to reproduce the bug, it's very hard to add debugging changes, because it only happens if you *don't* use upload-tools. I'm actually not sure how to test this with my own binary [02:32] natefinch: I *think* u will only have no arch by this stage if ur image metadat does not have arch... which would b weird [02:32] natefinch: i think some early simplestream metadata may have omitted the arch, not sure now. but now a days, it should never be "" [02:32] wallyworld: ok, this may be a red herring then. [02:33] natefinch: well, depends on the simplestreams metadata [02:33] if they generate metadata without an explicit arch, then boom [02:34] actually, looks like they're explicitly requesting the wrong tools, now that I re-read the logs ni the bug [02:35] Attempt 1 to download tools from https://10.0.3.164:17070/tools/2.0-beta3-xenial-amd64 [02:36] so, not really the fault of the streams that it gave them what they were asking for [02:36] I'm having a heck of a time trying to figure out how we determine the arch that eventually goes into that download URL, though [02:41] natefinch: it should be the arch of the host on which the tools are to be run [02:50] menn0: can i bother you for a 99% red one? http://reviews.vapour.ws/r/4562/ [02:52] wallyworld: looking [02:54] wallyworld: best. change. ever. [02:55] yay :-) [02:58] wallyworld: I know the arch should be the arch of the host on which it is run, but in this case, the bug is that it's not :) [02:59] i need to read the bug [03:00] wallyworld: basically just that non-amd64 LXD environments are downloading amd64 tools for some reason [03:00] Bug #1554677 changed: Provider help topics need to be updated for 2.0 <2.0-count>

[03:00] Bug #1569652 opened: help text for juju grant needs improving

[03:00] Bug #1569654 opened: help text for juju revoke needs improving

[03:01] were they bootstrapped on amd64 [03:01] with --upload-tools [03:02] wallyworld: no. it's been seen on arm64 and s390x [03:02] wallyworld: upload-tools actually fixes the problem [03:05] hmmm, not sure then, i'd have to debug [03:05] you'd need to trace the simplestreams request and response [03:06] wallyworld: do you have tips on how to debug? Since upload-tools avoids the bug, I don't really know how to get my own jujud into the system, but still have it look to simplestreams to get tools for further machinoes === menn0 is now known as menn0-afk [03:07] i'd just hack the code somewhere, eg don't store the uploaded tools in state to force a simplestreams check or something [03:08] not sure off hand, i'd need to read the code [03:08] wallyworld: ok, np [03:12] Bug #1567104 changed: Unable to connect to API

[03:19] hi, I'm trying to understand when config.yaml is parsed when deploying a charm. [03:19] I'd like to use some values stored in config.yaml when my install, start and stop hooks are executed. [03:20] Do I have the right idea/ [03:20] ? [03:23] terje: yep.... anything the user sets in the charm's config will be available to the charm code by having the charm call the config-get command, which will any configuration set on the charm (default in yaml) [03:23] terje: https://jujucharms.com/docs/stable/authors-hook-environment#config-get [03:25] ok, great. [03:25] wallyworld: looks like perrito666's fix for the mongo path has some genuine test failures: http://juju-ci.vapour.ws:8080/job/github-merge-juju/7344/console [03:25] cherylj: already on it :-) [03:26] thanks :) [03:26] a couple of missing "." [03:26] natefinch: is config-get a shell function, sourced prior to executing my hooks? [03:26] (I don't see a shell command on my system) [03:27] terje: yep, there's a bunch of shell commands that get installed with a charm. They're called "hook tools" in juju terminology: https://jujucharms.com/docs/stable/authors-hook-environment#hook-tools [03:27] terje: they are in the path when a hook is executing [03:27] awesome, thank you. [03:27] ok [03:30] axw: https://bugs.launchpad.net/bugs/1551779 is fixed right? [03:30] Bug #1551779: New azure provider ignores agent mirrors <2.0-count>

[03:31] wallyworld: no, didn't make the cut: https://github.com/juju/juju/pull/5068 [03:31] ok, np [03:31] it's ready to go whenever [03:31] jus saw the bug get changed, i thought it was fix committed [03:40] axw: i have to go afk for 45 minutes or so, if you see any landing failures, could you please retry? [03:40] wallyworld: okey dokey [03:41] ta [03:43] Bug #1568177 changed: configFunctionalSuite.TestUsingTCPRemote lxdclient no addresses match

[04:05] sweet [04:06] well that was out of context... [04:06] it was for this: mwhudson: yay, my fix worked, backups good now [04:06] :)\ [04:06] irc window was scrolled [04:21] man, juju debug-log is sloooooooow [04:45] wallyworld: http://reviews.vapour.ws/r/4563/ reviewed [04:45] menn0: ty [04:45] menn0: the previous PR was actually lgtm, i just fixed the tests, i'll see if i can pull the job [04:46] or i can tweak those method names in a driveby next time [04:48] wallyworld: it's really not that important [04:48] wallyworld: don't pull the current job [04:48] menn0: ok, i'll fix real soon in another branch [04:59] wallyworld, axw: regarding SSH host key management... looks like there's 2 ways to tackle it [04:59] you can have cloud-init post them to a URL [04:59] or your can pre-generated them and have cloud-init install them for you [05:00] I prefer the latter as it's simpler, but it does mean a little extra work for the server to do for every new machine [05:01] menn0: we did explorer having a listener on the cloud init url at one point (for provisioning status and errors), but it is more moving parts [05:01] what's the cpu cost of generating a key? [05:01] wallyworld: pre-generating them server side does seem better [05:01] wallyworld: I will check [05:04] menn0: I don't think we can do option 1 without a lot of added complexity [05:05] as in, where do we post to [05:05] menn0: my intention was to generate client-side for bootstrap, then have the server generate a new key and publish it into state [05:05] menn0: then we'd query state for public keys when we want to SSH [05:06] no prompting needed for verifying fingerprints [05:06] axw: I don't like the complexity of #1 either. we'd need a endpoint on the API server. [05:06] axw: I think we're thinking the same thing. [05:07] axw: what do you mean generate the key client side and then have the server generate a new key though? [05:08] axw: what i'm thinking is that when a new machine is to be created, a the host keys are generated on the server and inserted into state, and also passed to cloud-init [05:08] when you pass the keys to cloud-init, ssh keys aren't generated on the machine and the one that were passed are used instead [05:08] axw: http://cloudinit.readthedocs.org/en/latest/topics/examples.html#configure-instances-ssh-keys [05:08] axw: "ssh_keys" [05:11] menn0: sec, trying to find something. there's something about cloud-config not being completely secure. I think it might just be because anything on the machine can query the metadata [05:11] axw: ok, that's worth following up [05:12] menn0: eh can't find it, I'm sure there's a bug aobut it tho. but basically on AWS and others, you can get the cloud-config metadata by GETting a statically defined URL [05:13] menn0: no ACLs other than being on the machine [05:13] axw: ok that would be bad [05:13] so if you wanted to run something non-privileged on the machine, it could easily become privileged [05:13] menn0: so, what I was thinking is that we'd pass through ssh_keys, and then have the machine agent generate a new one on startup [05:14] menn0: the initial ssh_keys one is really only necessary for the bootstrap machine tho [05:14] axw, wallyworld: FWIW generating both the RSA and DSA keys at the default (recommended) bit sizes takes less that 0.2s (combined) on my machine [05:15] menn0: and for dense openstack deployments with containers, there may be several spun up at once [05:15] (and containers can access the metadata URL) [05:16] i guess the host controller needing to run that many containers would cope [05:16] menn0: not sure if what I said about bootstrap & ssh_keys made any sense, let me know if a hangout would help [05:17] axw: it mostly makes sense but a hangout would be helpful [05:17] menn0: ok just eating lunch, give me 15 mins or so [05:17] axw: np [05:22] wallyworld: sorry, got distracted and forgot to check your merge again [05:23] axw: tis ok. there was a spurious failure, i resubmitted [05:23] would be good to have the CI bot sending messages in here [05:24] axw: +1 === mup_ is now known as mup [05:38] menn0: https://plus.google.com/hangouts/_/canonical.com/juju-ssh-keys?authuser=1 [05:38] if you're free now [05:40] axw: coming! [05:50] menn0: wrong button [05:50] brb [05:50] LOL [05:50] what have I done [06:01] yay! master can bootstrap on xenial again :) [06:13] there's another issue today though [06:13] 2016-04-13 06:12:08 ERROR juju.worker.dependency engine.go:526 "proxy-config-updater" manifold worker returned unexpected error: unknown watcher [06:13] id (not found) [06:48] axw: this shit just keeps popping up everywhere :-( http://reviews.vapour.ws/r/4564/ [06:49] wallyworld: :/ looking [06:49] ta [06:54] wallyworld: LGTM [06:54] ty [06:59] wallyworld: hey, thanks for fixing that juju-mongodb3.2 issue! [06:59] dimitern: np, one more to go, landing now, affects restore [07:00] wasa joint effort with horatio [07:00] wallyworld: I've found a new proxyupdater issue on master, which I'm testing a fix for - such a fix should qualify for landing on master, but first I'll file a bug report for it [07:01] ok. we need to get a bless within 6 hours or so, maybe a bit more [07:13] Bug #1569725 opened: proxyupdater api facade does not set NotifyWatcherId in the result [07:23] frobware, voidspace, dooferlad: http://reviews.vapour.ws/r/4565/ fixes the bug above, please take a look [07:40] dimitern: looking [07:44] dimitern: apart from the test changes is the fix just in proxyupdater.go? [07:44] dimitern: ah, I see it. [07:50] dimitern: +1 from me. [07:51] * dooferlad will be back a bit later [07:51] frobware: the fix is mostly in the api/proxyupdater changes [07:52] dooferlad: are we doing 1:1 in 10? [07:53] dooferlad: oh, it's 9:30... [07:53] frobware: indeed [07:53] * dooferlad really is going now [07:53] frobware: I've observed at last what you're describing - lxd containers coming up with 1 NIC (as reported by lxc info), but inside having multiple NICs; applying the machine profile we create directly on the container manually seems to fix this (lxc info shows 8 NICs as expected) [07:55] dimitern: it depends... the output from lxc list (with netinfo) can dribble in [07:55] dimitern: if you can get to the same stage then don't apply the profile manually, just sit on watch lxc list for a bit [07:55] frobware: so not seeing all nics there is ok as long as they actually work - that's what you're saying? [07:55] :) [07:56] dimitern: there's the two case: 1) we don't get anything. 2) as you've descibed, but I'm not 100% sure this is because the info from lxc list can dribble in over time. [07:57] dimitern: I want to change the way we apply the profile anyway. it can be applied to the container directly (need to expose some LXD api) [07:57] frobware: I've seen the dribbling you're talking about [07:58] dimitern: want to sync for 30 mins? [07:58] frobware: don't you have another call? [07:58] dimitern: would be interested in any results from LXD bashing [07:58] dimitern: yep, but in 30mins [07:58] frobware: I'm not there yet (heavily testing lxd deployments in w/ w/o charms) [07:59] dimitern: sure, but (gentle nudge) let's sync anyway. :) [07:59] frobware: so if you don't mind I'd like to get the AC removal well tested first and finish it, so I can focus on LXD testing [07:59] i.e after standup will be better [07:59] dimitern: sure [08:00] frobware: cheers [08:02] jam: ping [08:02] frobware: hi [08:02] jam: I'm guessing my PR for detectSubnet() does the wrong thing... [08:03] frobware: so its entirely possible that we're using it incorrectly, but the intent of detectSubnet is find a subnet that is not in use [08:04] and the highest one we found +1 is not in use, but clearly the highest one we find *is* in use. [08:04] jam: so then my fix is not a fix... [08:04] jam: it would read better as detectSubnetNotInUse() :- [08:04] frobware: renaming "detectSubnet" to findUnusedSubnet would probably be a step in figuring out what is wrong [08:04] frobware: jinx? [08:05] paraphrase jinx [08:05] 2016-04-13 08:04:42 DEBUG juju.cmd.juju.commands ssh.go:180 proxy-ssh is false [08:05] jam: OK, I'll revert and look at that this morning. [08:05] wtf?! since when [08:05] can't juju ssh 1/lxd/0 because of that [08:05] dimitern: there was a big discussion about it recently [08:05] only juju ssh 0 works [08:06] jam: and it was decided proxy-ssh is bad? [08:06] dimitern, jam: interesting. pretty sure I was ssh'ing to 0/lxd/0 yesterday [08:06] dimitern: the main motivation was that in multi-model controllers, proxy-ssh means you have to have ssh access to the API server. [08:06] which is very bad for security [08:06] dimitern: frobware: is "next" unblocked yet? (as in - do lxd tests pass again, can we land branches) [08:06] If I let you create a model on my controller, that doesn't mean I want you to have SSH access to root my API server. [08:07] ah, I see updates on master and next [08:07] jam: it depends whether 0/lxd/0 has a dns-name coming from a vlan (e.g. 10.100.19.0/24), which might only be accessible via the controller [08:07] so probably [08:07] frobware: dimitern: well aren't X/lxd/Y supposed to be on the routable subnets anyway? [08:08] dimitern: so for people that have complex networking setups, they can do the jumps themselves, which is slightly unfortunate, or they can pass "--proxy" to juju ssh [08:08] jam: yes, though it's nominally easier (for me) to think of them as logical machines (i.e., 0/lxd/3) then to dig (ho) out its IP addr [08:08] but the default is that you can only use it if you are an admin on the controller. [08:08] jam: they are, but what we deem the machine's private address for ssh still uses the legacy scoped selection + sorting [08:08] nice! --proxy it is then [08:08] yay! works [08:09] dimitern: so for people that aren't admins "--proxy" will fail (which is why we set it to default false) [08:09] but if you have access to the API server its available for you [08:10] jam: yeah, this makes sense, but unfortunately also means we have to sort out the private address picking now [08:10] dimitern: well, we should generally fix that anyway, right ? :) [08:10] jam: indeed :) [08:13] dimitern, jam: http://reviews.vapour.ws/r/4566/ [08:14] frobware: shipit [08:14] ty [08:14] 'this change is not incorrect' is confusing [08:14] but the change itself is good [08:14] frobware: LGTM [08:16] jam: :) the change is incorrect (was in my head, but clearly not my fingers). [08:16] wallyworld: hey, is there a way to disable the model switching to "default" from "admin" after bootstrap? [08:17] it's a bit frustrating to do parallel bootstrap tests on different controllers [08:17] axw: ^^ [08:17] dimitern: no, that was the intended behaviour [08:18] dimitern: not at the moment. that shouldn't stop parallel bootstrap though [08:18] we don't wanted users in general doing things with the admin model [08:18] wallyworld: it's an odd experience first time round though [08:18] wallyworld, axw: makes sense, but it's a bit inconvenient to have to pass -m local.xxx:yyy each time.. I guess needs to be scripted [08:19] frobware: why is it odd? you deploy workloads to hosted models [08:19] dimitern: juju switch is your friend [08:19] the only reason I do it it's because I don't want to wait yet another 5m after bootstrap to be able to add a container to a machine in the default model [08:20] wallyworld: as a new user I bootstrap and then try and add-machine lxd:0 and it doesn't. I guess most people deploy workloads. [08:20] wallyworld: switch doesn't work well with parallel bootstraps - each does a switch at the end [08:20] wallyworld: heh, dimitern's comment is what I try to avoid too [08:21] frobware: the default hosted model is empty that is true [08:21] machine 0, the old so called bootstrap machine, is not something we want users to think about [08:21] wallyworld: and I still don't understand how I can get upgrade-juju to work in the default model [08:21] the controller is an inrernal detail [08:22] I almost wish to have a --use-admin-model flag to bootstrap :) i.e. I want you to not switch to "default" as I know what I'm doing [08:23]