[00:52] wallyworld: I think you misunderstood me about the API [00:52] I just meant to combine them in the CLI code [00:52] ah, damn [00:52] i.e. create an interface that you can satisfy by embedding both the facades into one struct [00:53] oh well, what's there is ok i think? [00:53] we use that same pattern elsewhere too [00:53] wallyworld: I think it's fine [00:53] wallyworld: ModelManager makes more sense for this command anyway I think [00:53] yeah, agree [01:23] wallyworld: is the model destroy blocking something communicated to other teams? [01:23] wallyworld: coukd tgat imoact things like the gui? [01:23] rick_h_: it was a stakeholder bug from OIL I think. I am not sure who else knows [01:24] i am not sure about GUI - I'll send an emil [01:24] rick_h_: but the blocking is ony CLI [01:24] wallyworld: right, i'm +1000 on the change, but want to make sure we chat with ither clients [01:24] wallyworld: ah ok [01:24] the api behaves the same [01:24] wallyworld: nvm then [01:25] fair enough question to asl [01:25] *ask [01:48] thumper: i've fixed the racy pingerSuite tests: https://bugs.launchpad.net/juju/+bug/1625214 [01:48] Bug #1625214: pingerSuite.TestAgentConnectionsShutDownWhenAPIServerDies write: broken pipe

[01:48] thumper: what I meant to paste: https://github.com/juju/juju/pull/6410/commits/f9ee5dba43e0fa032fe67d14d1c89cdc1c1e1a55 [01:49] looking [01:52] approve [01:52] thumper: cheers [02:01] wallyworld: https://github.com/juju/juju/pull/6411 please [02:02] menn0: sure, just finishing another [02:02] wallyworld: ignore the first commit. that one is merging as part of another PR now. [02:02] ok [02:19] menn0: could you take a look at this one by christian? i think your +1 would be valuable. I've already had a look https://github.com/juju/juju/pull/6408 [02:20] also, your 6411 pr has conflicts [02:20] wallyworld: will do. he emailed me about it too. [02:20] ta [02:21] wallyworld: conflicts fixed [02:21] ta [02:53] menn0: are you investigating that cert issue? [02:54] axw: I was about to take a short break and then stuck into it [02:54] axw: I still have a controller up where that's happened [02:54] axw: have you seen it too? [02:54] menn0: ok. would it be helpful if I looked in parallel? [02:54] menn0: no, but it didn't sound hard to repro [02:55] axw: could be. if you could take a look too that would be great. [02:55] menn0: ok. I'll let you know if I find anything [02:55] axw: here's what the failure looks like: http://paste.ubuntu.com/23301342/ [02:56] okey dokey [02:56] menn0: and all you did was bounce the controlelr agent? [02:57] axw: I stopped the lxd machine and started it again [02:57] ah yes [02:57] ok [02:57] the controller instance [02:57] * menn0 will be back in 15 mins [03:17] wallyworld: what did you want me to test with add-model? something like this? bootstrap rc2, upgrade to my branch, then add-model foo && add-model bar ap-southeast-2 [03:17] axw: yeah, check that the endpoint fixing is done for addmodel also [03:18] axw: or even first confirm that it is broken without your fix [03:18] so we can be sure that the fix is relvant nd valid [03:18] wallyworld: it would be broken if you tried to upgrade from rc2 [03:18] wallyworld: but if you added a model to a running rc2 without upgrading, it should work. anyway, I'll try what I described [03:19] ta, i think we need to be 10000% sure [03:40] menn0: I strongly suspect -> https://github.com/juju/juju/commit/0d64508f8005ea84c141ed392389fbaef0e70b30 [03:40] menn0: in particular, "info.Cert = existing.Cert" without also setting Key [03:51] Hi everyone, which ppa should I be using to get the RC version of juju 2.0 so I can try https://docs.google.com/document/d/1yT5pvS38g9Z9SviI9NPmhIZrsO8drwHDPNWAMSmCedE/edit# ? [03:52] axw: looks like there was already a ticket for this issue: https://bugs.launchpad.net/juju/+bug/1631145 [03:52] Bug #1631145: rc2 upgrade to rc3 failed with cannot start api server worker: cannot set initial certificate: cannot create new TLS certificate: crypto/tls: private key does not match public key [03:52] axw: i've just bumped the priority etc [03:54] okey dokey [03:54] axw: any luck replicating? [03:55] menn0: not yet [03:55] blahdeblah: according to release email, ppa:juju/devel [03:55] axw: cool - thanks [03:58] axw: with maas, do you know what happens if you attempt to acquire a node without specifying an architecture? does it just pick some arbitary machine so long as any other constraints match? [03:58] menn0: hrm. I am seeing "getsockopt...connection refused" without the other error though [03:58] and status isn't working now [03:58] wallyworld: I don't recall, sorry [03:58] axw: that's what you'll see at the client [03:58] axw: what about in the controller's logs? [03:59] maybe menn0 recalls? [03:59] menn0: this is in the controller machine log [03:59] axw: hmm ok [03:59] menn0: it's failing to connect to mongo... [03:59] axw: possibily a different problem? [03:59] yes, possibly [04:00] wallyworld: r:maas arch.. yes, i beleve so.. at lest this s what i ve seen [04:00] axw: not ideal [04:00] menn0: although... do we use the same cert/key for mongo? [04:00] menn0: ta, ok [04:00] wallyworld: sorry, not sure about maas [04:00] np, i'll ask the maas qguys [04:00] guys [04:01] axw: maybe we do use the same cert/key... worth checking [04:03] menn0: it does appear to be the case [04:04] when we update the cert, we write out to server.pem [04:04] axw: ok, so probably a different manifestation of the same issue [04:04] yup [04:04] axw: i'm tracing through where the cert and key that apiserver.NewServer is given come from [04:06] axw: ok so that's out of StateServingInfo in the agent config [04:06] axw: perhaps one is being updated without the other? [04:14] axw: this could be it: https://github.com/juju/juju/blob/master/cmd/jujud/agent/machine/servinginfo_setter.go#L67-L74 [04:15] axw: should existing.PrivateKey get copied too? [04:19] axw: I'd also be a lot happier if most of what the stateservinginfo_setter did happened /inside/ the ChangeConfig [04:19] axw: same for the method which does cert updates for the apiserver [04:19] looks racy otherwise [04:24] axw: the more I look, the more I think this is the bug [04:24] * menn0 fixes [04:25] menn0: sorry, I just realised you lost connection after I pasted the commit before :/ [04:25] axw: did you come to the same conclusion? [04:25] * menn0 has been having wifi problems [04:25] [11:40:32] menn0: I strongly suspect -> https://github.com/juju/juju/commit/0d64508f8005ea84c141ed392389fbaef0e70b30 [04:25] [11:40:57] menn0: in particular, "info.Cert = existing.Cert" without also setting Key [04:25] axw: ha! did see that. we independently found the same thing [04:26] axw: s/did/didn't/ [04:27] axw: do you also agree that much of the logic in stateservinginfo_setter should be /inside/ the ChangeConfig func? [04:28] axw: same for MachineAgent.upgradeCertificateDNSNames [04:28] menn0: yes, I think so [04:28] anything looking at current config [04:29] axw: ok, let's fix it. [04:29] axw: have you already started? [04:29] menn0: nope, eating lunch atm [04:30] blahdeblah: i so want to mark your bug as invalid. using firefox as default browser should be invalid :-) [04:30] axw: ok, i'll start on servinginfo_setter.go. [04:30] axw: i'm going to have to stop soon but will be back on later this evening [04:32] wallyworld: Feel like taking the gloves off today, eh? :-) [04:32] menn0: ok. let me know where you leave off, I can pick it up [04:32] axw: sounds good [04:32] blahdeblah: i used to run firefox until it started to grind everything to a halt with many tabs open and other performance things [04:33] wallyworld: It has never done that for me. I've heard many claims of such, but never seen it personally, despite having 4 windows with 100-odd tabs open right now. I thank NoScript for that. [04:34] blahdeblah: yeah, that may well be it. sadly lots of sites require js, including lp. well lp doesn't need it but uscks without it [04:34] I use js on lots of sites; I just don't let it do that by default [04:35] i find it too hard to whitelist everything. well maybe am too lazy [04:35] anyhew, my bug is valid, and I'll fight you for it! :-P [04:35] adblock works a treat though [04:35] oh i know it is valid [04:35] was just stirring [04:36] I see your stirring, and raise you lp:1587644, which got me out of bed at least once this weekend [04:37] menn0: wallyworld: considering u've re-viewed the fix for bug 1625774 [04:37] Bug #1625774: memory leak after repeated model creation/destruction

[04:38] menn0: wallyworld: is it possible/wanted/needed on 1.25 to solve some of the memory leakage there too? [04:39] not sure, depends on when state pool was introduced. not sure if it was for multi model or not [04:39] would have to dig in and look [04:39] anastasiamac: it could be applied there but I don't think multi-model was supported without a feature flag was it? the fix would have minimal utility if that's the case. [04:45] yes, multi model in 1.25 is ff [04:46] wallyworld: even server-side stuff? we usually ff at cli level... [04:46] not for multi model [04:46] IIRC [04:47] yeah, I'm pretty sure the feature flag was applied throughout the server too [04:52] axw: I have the ssi setter fixes done, pushing now [04:52] cool [04:52] axw: can I leave the MachineAgent.upgradeCertificateDNSNames fix to you? [04:53] menn0: yep sure [04:54] axw: https://github.com/juju/juju/pull/6413 [04:55] menn0: I think the only thing to do there is to not use "si.Cert", but get fresh value inside ChangeConfig? [04:55] ta [04:55] Bug #1630728 changed: remove user needs better message that user is made inactive [04:57] axw: shouldn't cert.NewDefaultServer be called inside the ChangeConfig though? [04:57] menn0: yes, I mean everything from "Parse the current certificate to get the current dns names." down should be inside [04:58] axw: +1 that's what I was thinking [04:58] should be easy [04:58] axw: just trying to QA this PR before I get called to help out with other stuff [04:58] menn0: except maybe the mongo bit. might want to do that after writing agent conf [04:58] menn0: okey dokey, thanks [04:59] axw: maybe... it might be ok to do it in the changeconfig though [04:59] axw: actually, i'm out of time. if you could review that PR, I'll check back in later and do the QA before merging [05:00] menn0: approved [05:01] menn0: with one late comment [06:30] axw: I made that change and QAed. Merging now. [06:33] menn0: thanks [06:33] menn0: I'm just QAing my change now [06:33] axw: cool. i can review when it's up. i've got to do dishes etc but will check back in later. [06:33] menn0: https://github.com/juju/juju/compare/master...axw:lp1631145-upgradecertificatednsnames?expand=1 if you want to take quick look [06:33] ok, no woprries [07:14] wallyworld: got anything smallish you'd like help? [07:16] axw: not off hand. I am finishing a partial list-clouds fix pending input from rick et al. i think there's one remaining bug on alexis' list to do with agent restart but not sure how big it is [07:16] wallyworld: sounded big, I'll take a look [07:17] yeah, it did at first look [07:52] axw: is your cert fix just because we need to recover from a borked cert cause by rc3? [07:52] wallyworld: yes. and also we should be doing stuff inside ConfigChanged in general. doesn't matter in this case because the function is only called at startup [07:53] but good to keep it clean, to avoid copy&paste errors [07:53] +1 [07:53] lgtm [08:05] could someone review my PR ? https://github.com/juju/juju/pull/6414 [08:06] also note that this PR will be the foundations of a new feature I wish to unlock on juju, enabling manual provisioning for windows machines. [08:07] foundation* [08:11] Bug #1420996 opened: Default secgroup reset periodically to allow 0.0.0.0/0 for 22, 17070, 37017

=== akhavr1 is now known as akhavr === stokachu_ is now known as stokachu === meetingology` is now known as meetingology === ejat_ is now known as ejat === Ursinha_ is now known as Ursinha [08:58] mgz: is the github jujubot happy? It seems to be ignoring my $$merge$$ on https://github.com/juju/juju/pull/6406 [09:20] wallyworld: ping? [09:20] yo [09:20] wallyworld: I think I agree with you about Release vs Put - do you think menn0 will sulk if I change it? [09:21] babbageclunk: he can sulk, i'll get the popcorn ready :-D [09:21] :) [09:21] babbageclunk: or even Done() [09:25] wallyworld: Done isn't verby enough for my taste. Get/Done doesn't seem right. [09:25] fair point, i was trying to find an alternative to Release() to aleviate objections :-) [09:27] always the peacemaker [09:27] oh, i have been known to enjoy poking the hornet's nest :-D [09:42] wallyworld: do you think I should rip the non-strict model uuid checking out of validateModelUUID? [09:43] wallyworld: There are still a few places that use it. [09:43] babbageclunk: i *think* so - maybe in a followup after checking with tim/menno. i am pretty sure it was just to support really old clients [09:44] wallyworld: Ok - I'll do it as a separate change. [09:44] sgtm [10:02] Bug #1631899 opened: juju show-controller --show-password does not show the password [10:21] wallyworld: Want to take another look at that PR, or are you ok for me to merge it? [10:22] babbageclunk: i can take a quick look, menno was happy with it [10:25] wallyworld: cool thanks - I think I've gone through all of your comments. [10:28] babbageclunk: i think it looks good to go. thanks for doing the fix; was a challenege to get all the bit lined up so you did well [10:28] wallyworld: cheers! [11:02] hi all, last week we started seeing sha256 mismatches when units try to download the charm from the controller (1.25.6). it's prevalent in openstack ci. ie. shutting down: ModeInstalling ... failed to download charm ... expected sha256 FOO, got BAR [11:02] is there a known issue or are we special? :) more detail @ http://pastebin.ubuntu.com/23302619/ [11:11] Bug #1631899 changed: juju show-controller --show-password does not show the password [11:23] Bug #1631899 opened: juju show-controller --show-password does not show the password [11:26] Bug #1631899 changed: juju show-controller --show-password does not show the password [11:26] Bug #1541482 opened: unable to download local: charm due to hash mismatch in multi-model deployment <2.0-count>

[12:17] Bug #1629951 changed: cannot specify subnet to create controller in on bootstrap [12:17] Bug #1630029 changed: models should inherit vpc-id from controller [13:00] rick_h_: 1:1 today? [13:01] dooferlad: /me checks thought he was in it [13:01] dooferlad: oh hmm, stuck at "requesting to join the video call" [14:00] katco``: dimitern voidspace mgz natefinch ping for standup [14:00] omw [14:00] omw [14:00] oh right, US away so ignore me katco`` and nate [14:02] rick_h_: omw [14:49] jamespage: do you often run into this issue: https://bugs.launchpad.net/juju/+bug/1600546 [14:49] Bug #1600546: lxd subnet setup by juju will interfere with openstack instance traffic <2.0>

[14:51] rick_h_: ping [14:51] voidspace: pong [15:21] dooferlad: ping [15:21] rick_h_: hello [15:21] dooferlad: have more time to chat? [15:22] rick_h_: yes [15:22] dooferlad: k, meet you back in the 1-1 room [15:32] rick_h_: 30 seconds for a bikeshed needed if you have it [15:34] rick_h_: last comment on bug 1602192 is the output I've implemented [15:34] Bug #1602192: when starting many LXD containers, they start failing to boot with "Too many open files" [15:38] voidspace: otp [15:38] voidspace: will look in a sec [15:38] rick_h_: ok, np - I'll PR it and the reviewer can bikeshed it [16:00] frobware, I've never run into that issue [16:00] jamespage: thanks. just trying to understand the severity and whether we address it "now". [16:00] but apparently trent has hit it alot [16:01] jamespage: any feeling whether this is just happening for just the training setup? [16:01] frobware, I'm not quite close enough to know the answer to that [16:01] sorry [16:02] rick_h_: ^^ fyi [16:02] dimitern: if you have any ideas on how to test this, I'm all ears [16:02] dimitern: https://github.com/juju/juju/pull/6419 [16:02] voidspace: looking [16:02] dimitern: except providing something that stubs out PrintLn. [16:03] dimitern: the important bit is in provider/common/bootstrap.go [16:04] voidspace: how about adding an interface with BootstrapMessage() method and then check in common/bootstrap if it's implemented by the Environ ? [16:04] voidspace: then you could test it with the dummy provider, but no need to change all providers? [16:05] dimitern: right, but what does that enable me to test? [16:05] dimitern: I could then pass in a fake environ with a custom message, but what do I test? [16:05] dimitern: unless I replace the call to fmt.Println with something that can be mocked - which doesn't seem very useful [16:05] dimitern: I'm kind of arguing that as this is a ui change it needn't be tested [16:06] dimitern: maybe I can look in featuretests to see if something like this is covered [16:06] voidspace: only that common.Bootstrap() calls the optional method, when it's there, and checking the ctx.Stdout() to ensure it's there? [16:06] voidspace: wfm for now, thank you [16:06] rick_h_: cool, thanks [16:06] frobware: yea, so I'm -1 on that being the biggest issue atm [16:06] * rick_h_ goes to get lunchables [16:06] dimitern: in which case I can test that *already* [16:07] dimitern: and put a non-empty string in dummy provider to test it [16:07] dimitern: let me look - thanks [16:07] dimitern: if we're calling fmt.Println will that be in ctx.Stdout ? [16:07] voidspace: alternatively, you could go with BootstrapContext.Infof() only called in provider/lxd ? [16:08] voidspace: since it's very specific and not needed everywhere [16:09] voidspace: check e.g. PrepareForBootstrap in provider/ec2 and the code around validateBootstrapVPC() [16:10] voidspace: (bootstrap)ctx.Infof() is already used for similar messages during bootstrap in some providers [16:10] kk [16:33] dimitern: using Stderr on the context I can test that the message is output [16:34] voidspace: \o/ nice! === frankban is now known as frankban|afk === dames is now known as thedac [17:55] katco``: i've made a bunch of comments and changes in response to your review of https://github.com/juju/juju/pull/6407. PTAL. [17:55] rogpeppe1: she's on US holiday today [17:56] rogpeppe1: and with the EU folks EOD'ing will have to catch tomorrow it looks like sorry for the delay [18:18] If any folks are have access to s390 we are seeing https://bugs.launchpad.net/juju/+bug/1632030 [18:18] Bug #1632030: juju-db fails to start -- WiredTiger reports Input/output error

[19:09] * rick_h_ goes to get the boy home from school, biab [20:23] alexisb: Ina normal (test) run, should I see the log message: "ERROR cmd supercommand.go:458 creating API connection: ..."? (I'm seeing this error in the failing tests: ERROR cmd supercommand.go:458 creating API connection: EOF) [20:24] not on a normal test run you are expecting to pass without error [20:24] I never see "creating API connection" in a passing test run (for grant-revoke) [20:24] veebers, when did you start seeing the new failures? [20:24] alexisb: what is it a symptom of? One sec will check times [20:25] alexisb: looks to be about Oct 7 (Jenkins time, so UTC?) had 2 passes of 12 runs since then, the failures all include that error message [20:28] menn0, ^^^ [20:28] I am wondering if this lines up with this merge: https://github.com/juju/juju/pull/6400 [20:32] veebers: where are you see that message? from the juju client or in agent logs? [20:33] veebers: a certain amount of such messages are expected as when controllers come up and when workers restart after new controllers are added [20:33] menn0: um, I'm pretty sure the client? (I'm seeing it in the log output from juju command) http://reports.vapour.ws/releases/4467/job/functional-grant-revoke/attempt/721 if you search for "connection: EOF" [20:34] * menn0 looks [20:34] menn0: this is after a controller comes up, users have been added etc. [20:40] menn0: Would this be related to the ping changes to? http://reports.vapour.ws/releases/4467/job/run-unit-tests-race/attempt/1949 (FAIL: monitor_internal_test.go:69: MonitorSuite.TestLaterPingFails) [20:41] veebers: that's a new test. i'll take a look. [20:41] veebers: the other issue appears to be macaroons related. here's the related bit in the controller's logs: [20:41] 2016-10-10 16:17:39 INFO juju.apiserver request_notifier.go:70 [1E] API connection from 10.0.8.1:33086 [20:41] 2016-10-10 16:17:39 INFO juju.apiserver admin.go:102 login failed with discharge-required error: verification failed: no macaroons [20:41] 2016-10-10 16:17:39 INFO bakery service.go:366 server attempting to discharge "eyJUaGlyZFBhcnR5UHVibGljS2V5IjoieVZlUEU3cUNJMm5nZWZuUXo2NlFPWVEwanlMVnVDanMwZGhMa0VUa3dRST0iLCJGaXJzdFBhcnR5UHVibGljS2V5IjoiNzF5NnNIOExNNUU1OHYxaGpGek9JdCtCZGRLRHR6SE5pN0ExQ1dBWHZqND0iLCJOb25jZSI6InpoMllTNnZ1bFFHNUlPdVRucXY4U0hVNW5MQWlSaE43IiwiSWQiOiJlSHM0Y3dERDRiOCs5TXBDUG1RakwvenVFdXZhV3dSbmhkY0tXWmtGQjA5MENaMXYxcnBYZ0RNczRsTTUrc2FSS2l2RGt0OFhlOG5BZTNxVG5 [20:41] 4Y1YwMXhIUWZxTXRmWHk4TUdnNmRvSWZlZWpSZmJnby9pdjIxSmZiU2FkQjNwdXR6ZmJ2ekRvbGh1aDNPcGdCbTRQd2pSRGx4cGFEQnZaL0lwZGhoL0lQb3dMdXc9PSJ9" [20:42] 2016-10-10 16:17:39 INFO juju.apiserver request_notifier.go:80 [1E] API connection terminated after 28.244335ms [20:42] 2016-10-10 16:17:39 INFO juju.apiserver request_notifier.go:70 [1F] API connection from 10.0.8.1:33096 [20:42] 2016-10-10 16:17:39 INFO juju.apiserver request_notifier.go:80 [1F] user-admin API connection terminated after 43.24719ms [20:43] veebers: i'll dig a bit more into that one to see if I can figure it out [20:43] menn0: I see a macroons change about 3 days ago, "remove macaroons on logout" [20:44] veebers: ok, could be related. those messages might also be normal. I'll do some digging. [20:46] menn0: Hmm, it's possible that the test misbehaves with those recent changes. Just looking now and if it's running on lxd (which this test is) it doesn't handle entering the password. [20:47] menn0: Did that behaviour change recently? (i.e. lxd not need password) [20:47] veebers: hmm, ok. that could be it. [20:47] veebers: I have no idea. [20:47] * veebers builds latest juju to test [20:47] menn0: let me test that and I'll get back to you [20:47] veebers: hmm, yea the branch nate did made sure to clear cookies when you logout [20:47] veebers: menn0 which meant that you had to actually login again after a logout for a change [20:48] rick_h_: that's probably it then, I should be able to confirm shortly [20:49] veebers: is there a ticket yet for the TestLaterPingFails intermittent failure? [20:49] menn0: not yet, I can make one right now [20:50] veebers: ok thanks. i've just reproed it here [20:51] grrr. even with injected clocks it's still too easy to introduce these intermittent timing issues .... [20:52] menn0: fyi https://bugs.launchpad.net/juju/+bug/1632105 [20:52] Bug #1632105: Test MonitorSuite.TestLaterPingFails fails

[20:53] veebers: cheers [21:14] veebers: ok i've got the fix for TestLaterPingFails done... will propose shortly [21:15] menn0: sweetbix [21:16] veebers: thanks for pointing it out [21:17] menn0: heh, no worries, it popped up in the 'unknowns' that I'm wrangling [21:18] good to get on top of these intermittent failures quickly [21:18] agreed [21:23] thumper: easy review pls: https://github.com/juju/juju/pull/6420 [21:23] * thumper looking [21:24] +1 [21:27] menn0: ugh, sorry for the noise earlier, the fix for the grant/revoke/macroons stuff was in the CI test. [21:27] veebers: ok good to know [21:40] thumper: it's probably too late to expose the HighAvailabilty facade for controller logins isn't it? [21:41] menn0: yes and no [21:41] you could create a new one [21:41] but still need to support the old one [21:41] thumper: i'm working on the ticket to allow juju enable-ha to just work regardless of the current model [21:42] I guessed [21:42] thumper: the client side work is a piece of cake but you then end up with a controller login [21:42] thumper: the alternative is to do something tricky in the client so it logs into the controller model [21:42] Add the new facade [21:42] * thumper thinks [21:43] I don't think that would be hard to do and it's possibly lower risk [21:43] you need to have admin access in the controller model to do it [21:44] * menn0 checks what the current story with access is [21:44] or perhaps just write? [21:45] you currently need superuser access [21:46] that probably makes sense TBH [21:50] thumper: i don't *think* a new facade is required, the existing HighAvailability facade just needs to be exposed for controller logins and (controller) model logins (for compatibility) [21:50] ok [21:50] thumper: the client will have to try both approaches (in case a newer client is talking to an older server) [21:50] * thumper nods [21:50] thumper: with a preference for a controller login [21:51] ok [21:51]