[00:19] thumper: poke regarding http://reviews.vapour.ws/r/5543/ :) [01:08] menn0: sorry, got distracted [01:08] thumper: no worries [01:25] menn0: do you recall how to get the engine as a dependency? [01:25] thumper: only vaguely [01:26] * menn0 checks [01:29] thumper: is worker/dependency.SelfManifold what you want? [01:29] probably [01:29] I'm halfway through your review [01:29] wallyworld: with you shortly, just want to finish this review [01:29] thumper: looks like it's currently only used in some tests [01:30] sure [01:30] * wallyworld is doing a review too [01:38] thumper: regarding version.Number over the API, that's how it's done elsewhere and it's fine. version.Number has custom JSON marshalling defined which turns it into a string. [01:43] menn0: if we are already doing it then fine [01:44] drop em [01:46] natefinch: hi, i am now [01:47] wallyworld: can you please take a look at http://reviews.vapour.ws/r/5534/diff/3-4/, made some changes in response to mhilton's review [01:48] sure [01:56] tych0: hey. I'm having difficulty figuring out this bug: https://bugs.launchpad.net/juju-core/1.25/+bug/1610880 [01:56] Bug #1610880: Downloading container templates fails in manual environment [01:57] tych0: the crux seems to be that lcx-create is trying a wget for the image ... and for some reason on a manual juju environment it fails and on a normal juju environment it works [01:58] tych0: I know this is only tangentially related to anything you've worked on, but I was wondering if you had any ideas [02:05] why is the lxd provider downloading a new ubuntu-xenial image? [02:05] does it auto update? [02:06] new as in - it already has one and it's getting another? [02:08] and the answer is.. AFAIK, LXD maintains its own list of images, what it does with those is mysterious magic. I don't *think* we tell it to auto-update, but I could be wrong. [02:18] logging added, ci test running again, time to get food [02:29] ugh [02:29] ffs [02:29] rerunning [02:38] oh interesting.... so, in a manual environment, we're not adding the cloud-local address for the machine to the cert [02:44] wallyworld: I like the new bootstrap output. looks much better [02:45] wallyworld: shouldn't these 2 things be the other way around though: [02:45] Installing curl, cpu-checker, bridge-utils, cloud-utils, tmux [02:45] Running apt-get update [03:06] pjdc: do you have an env that's currently exhibiting CPU/mem spikes as in https://bugs.launchpad.net/juju/+bug/1587644? I'm after a CPU profile in addition to what's been provided already [03:06] Bug #1587644: jujud and mongo cpu/ram usage spike

[03:07] axw: not right now. but if you can tell us how to capture what you'd like captured, i'll add it to our ticket [03:08] pjdc: you should just use the same command as for the heap profile, but use /debug/pprof/profile instead of /debug/pprof/heap [03:08] axw: righto - will update the ticket [03:08] pjdc: which is described here for 1.25: https://github.com/juju/juju/wiki/pprof-facility [03:08] pjdc: thanks [03:15] axw: just testing it here; does this look right? https://pastebin.canonical.com/164042/plain/ [03:18] pjdc: hrm, nope, that doesn't look right. should be much bigger. odd, it worked for me just now [03:18] pjdc: looks like the right command line invocation though ... [03:19] the single quote character seems pretty odd [03:31] menn0: sorry, was at lunch. you could be right, i'll have to check [03:38] thumper: since u r in the manul area, ci also seem to have observed long bootstrap (>45min)... if u have any thoughts on this would be awesome to have them in this bug [03:38] https://bugs.launchpad.net/juju/+bug/1617137 [03:38] Bug #1617137: Timing out fetching agent (xenial/s390x) [03:42] anastasiamac: that's not related, that's due to closed network [03:42] curtis needs to reimport the images [03:42] wallyworld: if u can comment in the bug, m sure QA and veebers will apprecaite [03:43] anastasiamac: curits already knows - he and i disucssed [03:43] he has more detial than i do [03:43] as to exactly what needs to be done [03:43] as he did it originally [03:43] wallyworld, anastasiamac ah ok, seems I missfiled that then. I'll get that fixed [03:43] wallyworld: sure, veebers filed the bug and this info is useful for anyone who was not in on ur discussion :D [03:43] awesome! tyvm :D [03:44] but i don't have all the detail [03:44] i only know generalities, i'd rather get the person who knows to comment [03:45] sounds likes the QA folks need to talk to each other more :-) [03:45] anastasiamac, wallyworld: If we can keep the bug for now (maybe not marked critical) I've updated the rule/issue to indicate it's a ci infra issue and we'll get those in the know to comment/remove/etc. thebug [03:45] no worries [03:45] having a bug means it can be tracked [03:45] veebers: feel free to adjust priority on it :D [03:49] anastasiamac: sweet, have done. also added affects ci with a (vague) comment about network [03:49] veebers: wallyworld \o/ [03:51] the bug should be retargetted to CI, as curtis confirm it's not juju [03:53] wallyworld: it has been [03:53] awesome, tyvm [03:53] wallyworld: oh no wait, I didn't remove juju, just added juju-ci [03:54] perhaps I should have lied just now and said that I had removed juju too :-) [03:54] we can add juju back if needed, but for now the best info is that it's a CI issue - "someone" needs to import LXD images [03:54] veebers: with the changes landed today, there a bunch more prechecks in place [03:54] wallyworld: removed now [03:54] so that LXD uses those importd instead of calling out to cloud-images [03:54] \o/ [03:54] menn0: oh neat :-) Does it break any assumptions the test makes currently? [03:55] veebers: the main one you'll be interested in is that it isn't possible to upgrade if the target controller tools version is less than the model tools version [03:55] veebers: no it shouldn't break the existing tests [03:56] veebers: the tests to ensure the source and target aren't controller are done too (mostly landed) [03:56] veebers: i'm implementing the prechecks to ensure that the source controller, model and target controller machines are healthy now [03:58] menn0: nice. Might look at getting a test for between versions going next week [03:58] veebers: sounds good. [03:58] veebers: just keeping you informed :) [03:59] menn0: :-) [04:16] wallyworld: is it me, or does this seem dangerous? https://github.com/juju/juju/blob/master/worker/certupdater/certupdater.go#L119 We're updating the saved addresses before knowing if we've actually successfully updated the cert. [04:17] yeah, seems suboptimal [04:18] wallyworld: I'm looking at this bug about lxc containers on manual provider, and it seems like we're not adding the cloud-local address to the cert for some reason [04:19] we use c.addresses to short circuit any future updates, i guess if things fail, that means we'll never process those addresses again [04:19] wallyworld: I don't think that's the actual problem [04:19] wallyworld: it just looks suspicious [04:20] i can't recall enough about the code to know why cloud local addresses are not arriving at the cert updater [04:20] they must be filtered out upstream somewhere [04:20] the machine address setting code is a bit gnarly [04:21] wallyworld: yeah... we call state.APIHostPorts() when the cert updater starts, and add any local-cloud addresses... on manual, it gets no addresses, on gce, it gets the correct local-cloud address [04:21] oh, manual [04:21] we won't get any [04:22] why not? [04:22] the cloud local addresses come from the instance poller from memory [04:22] and there's no such thing as an instance poller for manual IIANM [04:22] ug [04:22] this is a bit hand wavy [04:23] i could be wrong [04:23] but generally the instance poller is a major source of our knoweledge of machine addresses [04:23] you're at least slightly wrong... I see juju.state address.go:137 setting API hostPorts: [[104.196.3.75:17070 10.142.0.2:17070 127.0.0.1:17070 [::1]:17070]] on manual [04:23] with manual, we wil get machine local addresses [04:24] right, but it depends on how those are classified [04:24] that 10.142 address is what lxc-create is trying to wget the image from [04:24] by our address heuristics [04:24] hmm [04:24] we label addresses as machine local, cloud local, public etc [04:25] what is the machine foe which those host ports are being set above> [04:25] ? [04:25] a controller or a worker machine? [04:25] controller [04:25] anyways, 127.0.0.1 looks wrong [04:26] cause if that is handed out as a controller address, it can't work [04:26] works from that machine ;) [04:26] might not be the same issue as your seeing, but looks wrong to me [04:27] it gets set that way on my gce controller as well, so probably not the problem [04:27] so from memory, set addreses happens and we filter somehow and then hand out to cert updater, but i can't really recall exactly [04:28] maybe the filtering takes care of localhost [05:33] axw: i've updated http://reviews.vapour.ws/r/5533/ if you could PTAL, a bit of code was moved around [05:34] okey dokey [05:34] axw: added a fix and a test for the tranitive permission etc [05:38] well, with menn0's help, debugging is progressing [05:38] it is in the proxy updater where it is trying to set the lxd proxies [05:38] it is just blocking forever [05:38] holding workers up [05:38] anyway [05:39] more debugging for monday [05:39] * thumper is done for now [05:39] laters folks [05:43] wallyworld: found an existing bug, but otherwise LGTM [05:44] ta, looking [05:46] axw: yeah, that was existing behaviour before I started this PR. it does seem wrong doesn't it [05:46] actually [05:47] it is as per the all users case above, but it makes more sense there [05:47] i'll change to errperm [05:59] axw: ah, the existing client code will bail out with an error if *any* of the requested users results in an error [05:59] wallyworld: :/ [05:59] so that's why the server side was skipping over unpermitted users [06:00] wallyworld: we should do that filtering on the client, not on the server [06:00] agreed [06:01] axw: for now, how about i just modify the api caller to skip err perms [06:01] but error for other things [06:01] wallyworld: sounds OK I guess [06:01] or the other things is we could print the errors for tabular output [06:02] as well as the users it can find [06:02] we don't have such a good pattern for this [06:05] bah, too much churn, i'll add a todo [06:09] and the CLI only passes one at a time anyway === frankban|afk is now known as frankban [08:47] mhilton: thanks for the review [09:10] axw_: np [12:11] natefinch: replied on the bug [12:11] but it looks like wget is refusing to download something because the certs don't match? === freyes__ is now known as freyes [12:54] interesting new format of juju debug-log in beta16, it appears it rolled over to json formatting by default? [13:20] mgz: around? [13:20] babbageclunk: yo [13:21] Trying to cleanup instances from running CI tests with --keep-env [13:22] but I can't find my AWS credentials - I may have lost them when my machine died. [13:22] so, what I do is JUJU_DATA=~/cloud-city/jes-homes/gz-test-env-name $GOPATH/bin/juju destroy-controller [13:23] oh, actually, I have the user name and password, just don't have the URL for the canonical console [13:23] depends if you have the env on disk still? you don't need seperate creds unless you wiped the env from disk [13:23] which creds were you using to test? [13:23] I don't think that'll work - the api isn't up because the restore failed. [13:23] the shared dev ones, or CI's? [13:24] Hmm - I guess the CI's ones. [13:24] babbageclunk: then fall back to kill-controller - which again should get the aws details out of JUJU_DATA [13:24] opk [13:24] ok [13:24] try that, yell if you have problems [13:24] the aws console details for ci are in the consoles.txt file [13:25] be careful poking around there, if you do use [13:25] ok, thanks mgz [13:50] andrey-mp: Hi. w.r.t your information yesterday, I created "cinder-storagedriver" charm which will pass config data to relation to modify the configuration. I have taken your scaleio-openstack as reference. How do you certify scaleio-openstack charm? Please provide me the juu charm certification requirements. [14:01] natefinch: standup time [14:02] katco: oops, thanks [14:23] looks like juju has overwritten authorized_keys rather than appending to it, is that normal? (1.25) [14:24] admcleod: yes [14:24] mgz: oh. [14:25] mgz: thanks [14:25] admcleod: juju expects to manage the file, so it hass list/add/delete/import [14:25] to manage [14:28] right fair enough [14:29] that is a surprise if you've edited it manually on a machine yourself perhaps :) [14:30] babbageclunk: yell if you have any questions about my ssh key branch [14:42] frobware: dimitern: thank you for all your hard work for ivoks [14:43] katco: :) cheers [14:57] mgz: LGTM! [14:58] babbageclunk: merci [15:06] dimitern: lmk when you have that hotfix ready so i can pass it on [15:08] katco: will do [15:08] dimitern: ta [15:19] is someone able to talk to me about the new upload-tools logic? [15:20] mattyw: it just works, as long as jujud is in the same directory as the juju you're running and they have the exact same version number [15:21] natefinch, that's the thing - for me it didn't just work [15:21] natefinch, so I'm trying to work out why [15:29] mattyw: is it uploading when it's not supposed to, or vice versa? [15:30] perrito666: ping? [15:31] babbageclunk: pong [15:31] katco: I've sent this patch to ivoks - tested locally and seems to work, waiting on feedback later: http://paste.ubuntu.com/23093530/ [15:31] perrito666: I'm trying to understand backup and restore! :) [15:31] dimitern: yay! ty! hope the feedback is good [15:31] perrito666: various people said you were the one to talk to. [15:31] babbageclunk: sadly I am [15:32] dimitern, frobware you guys are rockstars thank you! [15:33] perrito666: don't you wish you had a... backup? [15:33] perrito666: it would... restore your sanity [15:33] * babbageclunk lols [15:33] * katco groans [15:33] katco: you should do that for a living [15:33] :) let's see if it'll be any good on site [15:33] babbageclunk, perrito666 I actually wanted to talk to both of you about the bugs youa reworking [15:34] do you guys have time for a quik HO? [15:34] alexisb: sure [15:34] alexisb: sure! [15:34] sorry got distracted by some other fires [15:34] perrito666, babbageclunk https://hangouts.google.com/hangouts/_/canonical.com/a-team-standup [15:44] its a trap [15:47] could anyone http://reviews.vapour.ws/r/5546/ ? [15:47] its a quick short one [15:47] I can look [15:48] natefinch: just added QA steps [15:50] ahh crap, forgot about QA steps [15:50] well that quintuples the amount of time I have to spend reviewing it :/ [16:02] why does the Juju client still hard code support for OS? === mup_ is now known as mup [16:07] https://bugs.launchpad.net/juju/+bug/1616531 [16:07] Bug #1616531: "panic: unknown OS for series" when running client on Fedora === frankban is now known as frankban|afk [16:29] lol fedora [16:31] natefinch: oh, that is not nice [16:31] fedora is a fine distro [16:32] I am sure it is. [16:32] also, why in the everloving hell does the client give a crap what OS it's running on? [16:33] I thought we had fixed that bug ages ago [16:33] no [16:33] we talked about it in december and never did it [16:34] :sadpanda: [19:08] perrito666, is this still an issue: https://bugs.launchpad.net/juju/+bug/1530840 [19:08] Bug #1530840: juju status-history too noisy with update-status [19:11] alexisb: priv [19:59] * redir lunches [20:28] so quiet on fridays:) [20:34] yep [20:38] ok, I give up [20:38] see you all next week [23:31] I didn' miss anyone at the stand-up did I? [23:32] I mean I just didn't join this week because there hasn't been anyone there since we rearranged === zeestrat_ is now known as zeestrat [23:44] ok [23:44] * redir goes EoW soon-ish [23:57] see you next week juju