[00:09] thumper, can you please do the technical reviews on wallyworlds list-clouds and list-regions PRs [00:11] sure [01:32] wallyworld or thumper: here's the "juju enable-ha" improvement: https://github.com/juju/juju/pull/6421 [01:32] thumper: thanks for review, i've updated list-regions pr [01:32] looking [01:40] menn0: done. i also just wanted to make sure you tested using an rc3 client to check the model facade call works still [01:41] wallyworld: good call, will do [01:59] wallyworld: rc3 client works as expected although there are some errors due to unqualified user names: http://paste.ubuntu.com/23305923/ [01:59] wallyworld: is this ok? [02:01] menn0: those errors should not come up in practice for other clients. the main takeaway is tha the enable ha call worked. the error is because the 2.0 server returned a user without @local which rc3 client did not understand [02:02] wallyworld: yep that's what I figured. just wanted to check with you that that's expected. all good. merging now. [02:02] (other issues fixed) [02:02] awesome ty [02:02] menn0: if you are stuck for work, i have a small improvement i can pass on. not sure what you have in your plate [02:03] wallyworld: I was going to tackle that not-so-intermittent cert update test failure which was introduced yesterday [02:03] ok [02:03] wallyworld: but what else do you have? [02:03] wallyworld: menn0: thumper: I feel like bug 1631438 is not for point release (2.0.1) but for minor (2.1.0) since it inverts current behavior? agree? [02:03] Bug #1631438: Switch default behaviour of 'gui' w.r.t. browser [02:03] i need for executive 2nd opinion :D [02:04] menn0: easier via quick hangout to give context [02:04] standup? [02:04] wallyworld: see you there [02:05] anastasiamac: yep, that seems like more of a 2.1 thing [02:05] menn0: u r awesome \o/ [02:07] menn0, anastasiamac: hmm... I'm not so certain, given that it is a very 2.0 thing [02:08] personally it isn't something that I've had to deal with yet [02:08] and I feel that most others won't have hit it either [02:08] however it is very late in the day to be changing this [02:08] thumper: i can propose a change if we are just talking about code.. dunno who it'll break.. considering 2.0.0 is freezing soon, there are only couple of hrs eft really.. [02:08] my preference is to change nothing [02:08] left* [02:09] I think I talked myself out of saying we should change now [02:09] thumper: mine too... hence 2.1.0 [02:09] :D [02:12] thumper: since I have u... isn't this a dupe of the one u did last week? https://bugs.launchpad.net/juju/+bug/1576342 [02:12] Bug #1576342: `juju status` should show leadership primitives

[02:13] yes, fixed the bug [02:14] thumper: \o/ [02:19] Bug #1632159 opened: juju 1.25.6 HA - Causing unit state issues.

[02:29] thumper: list-regions PR? you happy with it now? [02:35] getting there, just dropped kiddo down to BJJ [03:05] wallyworld: PTAL - https://github.com/juju/juju/pull/6422 thank you for the wording [03:06] give me a minute [03:06] * anastasiamac is very giving today [03:06] :D [03:40] thumper: i added details to regions for yaml and json because in general with juju, yaml and json can provide extras not ordinarily excluded from list. i realise show-cloud can be used, but my thinking was 1. that gives more than just region details, 2. I wanted to not have 2 commands to present region info (it's more natural for the user to flip output format for the regions command). I originally didn't have yaml or json because [03:40] just having []string for those didn't seem to pay its way so adding those formats I wanted to make it worth it [03:41] s/excluded/included [03:42] ok\ [05:10] anastasiamac: not sure if you might have time to review the table header changes? https://github.com/juju/juju/pull/6423 [05:45] wallyworld: looking [05:48] ta [06:01] nps. let me know when u want a stamp :) [06:07] anastasiamac: thanks for picking up those things i missed, i think it's good now? [06:08] looking [06:11] anastasiamac: gotta run to do school pickup, if it's ok, can you $$merge$$ for me? [06:12] bbiab [06:12] wallyworld: run coz I think "paylod class" needs to b changed too... [06:12] k \o/ [06:12] i'll finish another review by the time u r back [06:31] anastasiamac: ty, fixed that last one [06:32] wallyworld: \o/ === frankban|afk is now known as frankban === rmcall_ is now known as rmcall [08:17] it's the same rule that one two +1 = $$merge$$ no? [08:18] Hey, thumper, can I bug you for a quick opinion? [08:18] sure [08:18] * thumper is online to talk to uros [08:18] if that rule is the same, could someone take a look on my latest PR? https://github.com/juju/juju/pull/6414 [08:18] Ah ok - no worries if you're busy [08:19] babbageclunk: not busy, he isn't around [08:19] consolation chat! [08:19] let me know if I have something to improve even further or it's safe to merge. [08:19] hoenir: to be honest, we probably won't be looking at things for a few days as we prep for 2.0 GA [08:19] hoenir: we won't merge it until after GA [08:19] bug fixes only just now [08:19] GA is Thursday, US time [08:20] so soon [08:20] thedac, GA? what is GA? could you explain a little more please? and thanks for the response [08:20] thumper: So I've worked out the remaining goroutine leaks - they're the read- and write-loops of http transports created by the lxd provider. [08:20] GA stands for? [08:21] general availability [08:21] so the full 2.0 release [08:21] no more release candidates [08:21] it's finally come to that.. [08:21] thumper, thanks for clearing this up, I should ping here before the GA release. [08:21] o/ dimitern [08:22] probably just after :) [08:22] hey thumper :) [08:22] thumper, thanks a lot ! [08:22] thumper: If I disable keepalives they go away, but that's probably bad. [08:22] hmm [08:22] how long is the keep alive set for? [08:23] can we just tweak that? [08:23] No, they don't timeout [08:23] and is it just lxd that we leak? [08:23] oh [08:23] Probably not [08:23] at all? [08:24] Actually, I'll try putting a timeout in to see what happens. Would be easier than adding a .Close to EnvironProvider. [08:24] where does the http transport live? [08:25] inside the lxd client, which is in the lxd provider. [08:25] and does the code create a new transport if the old has gone? [08:26] I think when a new model added is we create a new client/new transport. [08:26] * babbageclunk yoda'd [08:26] heh [08:27] The docs say we should keep one http.Client to avoid leaking - I think the problem is that we create a new one for each lxd client. [08:27] hmm... [08:27] So I could contrive to pass it in somehoe [08:28] uh, somehow [08:28] that sounds like a good idea [08:29] The other option would be to call .CloseIdleConnections which will stop the loops - but that probably requires EnvironProvider.Close. [08:29] I'll try injecting the http client in from further up. [08:30] ok thanks! [08:30] np [08:31] had fun mangling the stdlib to work out where the leak was coming from. === jamespag` is now known as jamespage [08:32] hey dimitern, how was Florence?! [08:33] babbageclunk: awesome! :) and all of Tuscany in general - better than Rome for sure hehe === StoneTable is now known as aisrael === jcsacket- is now known as jcsackett [09:49] menn0: So it turned out that the leaked connections are from the lxd clients inside environs. [09:49] menn0: also, hi! [09:49] babbageclunk: hi :) [09:50] babbageclunk: ok, interesting. easily fixable? [09:51] menn0: Well, setting IdleConnTimeout in the transport when creating the http client works. I'm going to talk to the lxd people to see if they mind doing that. Do you think there'd be a downside to just doing that all the time? [09:52] menn0: Or should it be configurable? [09:52] babbageclunk: depends if we expect to have connections not doing anything for a long time I guess [09:52] babbageclunk: configurable might be best [09:53] menn0: it'll reopen them if they timeout and there's another request - it's purely a performance thing. [09:54] babbageclunk: ok, in that case, I don't see any downside, especially if the timeout is a few minutes or something [09:57] menn0: Cool - I'll discuss it with Stephane and see if he has any objections. [09:57] menn0: Thanks! [10:04] babbageclunk: np. nice work figuring it out. [10:08] babbageclunk: reivew please: https://github.com/juju/juju/pull/6425 [10:08] review even [10:08] I'm off for now [10:08] so no rush [10:08] * menn0 is hoping he can still land this before the release branch [10:09] menn0: I can look at it quickly now if you want? [10:09] babbageclunk: no need. I still need to QA the changes with MAAS. [10:10] menn0: ah, ok [10:11] babbageclunk: just updated the description to show what the new output looks like [10:11] the change is the last line [10:12] * menn0 is exhausted [10:12] babbageclunk: good night [10:12] menn0: night! [10:34] jamespage: bug 1580418 [10:34] Bug #1580418: Cached local charms should be deleted when their service is removed

[10:41] mgz, ta [11:03] dimitern: running a sec late [11:03] dimitern: omw [11:08] dimitern: ok, ping if you're around later [11:08] rick_h_: oops sorry - omw [12:01] dooferlad: ping [12:02] voidspace: pong [12:02] voidspace: sorry, I have been in meetings for the last 24 hours [12:02] dooferlad: ouch :-) [12:02] voidspace: I guess you want me to look at some code! [12:02] voidspace: it's fine [12:03] dooferlad: please, it's already merged - but it should have had a follow up review [12:03] voidspace: in one now [12:03] dooferlad: https://github.com/juju/juju/pull/6419 [12:03] dooferlad: if there are changes needed I'll do a follow up [12:03] dooferlad: when you have time - thank you very much! [12:03] voidspace: thanks, will do [12:10] mgz: ping [12:21] dooferlad: did you get anywhere with the test failures in your PR https://github.com/juju/juju/pull/6406 ? === cmars` is now known as cmars [12:33] dooferlad, voidspace, frobware, babbageclunk: I'd appreciate a review on https://github.com/juju/juju/pull/6426 guys - needed to fix bug 1616098 [12:33] Bug #1616098: Juju 2.0 uses random IP for 'PUBLIC-ADDRESS' with MAAS 2.0 <4010> [12:35] morning [12:35] perrito666: o/ [12:35] * dimitern steps out for <1h [12:35] * voidspace launches 30 containers and goes for coffee [12:36] mgz: ping again if you're around [12:36] perrito666: hi :D [12:55] voidspace: yo [13:10] natefinch: i replied to https://github.com/juju/utils/pull/245 if you wanna take another look === frankban is now known as frankban|afk [13:26] well, it was really contrived and took some effort on my part, but i found a way to break the juju controller in a way that i dont know how to recover from [13:26] so, let me check some assertions first before i label it as such - does the juju controller probe interfaces on the host and add those to the controller config? specifically the juju-db service? [13:27] rogpeppe1: cool [13:36] voidspace: have looked at your branch - no issues from me [13:37] rick_h_: there was a Windows build failure. I didn't have a Windows VM around so I have been setting that up. It should be a case of providing a stub for Windows, but I don't want to code by throwing changes at the merge bot! [13:37] dooferlad: ok, just making sure we're on top of it since today is the last day to get code in for GA [13:37] dooferlad: so want to make sure we can get this change in [13:37] you can also just ssh into the ci windows machine and build stuff there [13:38] rick_h_: we will be lucky to. [13:38] mgz: details? [13:39] dooferlad: in the cloud-city branch [13:39] mgz: ohh, thanks! [13:45] dooferlad: cool, thanks [14:01] katco: dimitern frobware dooferlad ping for standup [14:01] mgz: ^ [14:01] rick_h_: I'm deep in rebuilding packages so don't want to reboot for ho [14:02] can give update here [14:04] omw [14:06] alexisb: ping? [14:15] frobware: interestingly port 49151 is indeed unreachable on windows2012r2, but wininit.exe does listen on 49152-49159 [14:16] dimitern: heh === frankban|afk is now known as frankban [14:16] macgreagoir: we have a parent-teacher thing, but when I'm back from that I'll grab a review [14:17] voidspace: Cheers! [14:18] mgz: anything you can do to help voidspace and his vsphere connection woes is <3 so we can validate this isn't a bigger 2.0 issue. [14:18] mgz: understand on the packaging and thanks for pushing through on that [14:18] dimitern: heads up this is showing conflicts now: https://github.com/juju/juju/pull/6426 [14:18] dimitern: oh, nvm...read that too fast heh [14:19] here's a PR that fixes this critical juju bug: https://bugs.launchpad.net/juju/+bug/1631449. a review would be much appreciated please: https://github.com/juju/juju/pull/6427 [14:19] Bug #1631449: External users don't have access to models unless everyone@external is granted [14:20] rick_h_: gotcha, thanks [14:36] perrito666, can you review rogpeppe1 PR [14:36] sure I can, havent seen his message sorry [14:38] macgreagoir: +1 on your pr [14:38] katco: woo hoo! :-) [14:38] Cheers [14:39] macgreagoir: great job sticking with it! i think we got it to a much better place [14:39] Thanks for the help. [14:55] perrito666: thanks [14:56] katco: would you be able to take another look at https://github.com/juju/juju/pull/6407 ? i've addressed the most significant issues but i haven't added fixes for unrelated tests. [14:57] rogpeppe1: yes sure... will be just a bit but i will get to it today [14:57] rogpeppe1: sorry for the wait [14:57] katco: thanks [14:57] rogpeppe1: and thanks for the discussion [14:57] katco: np [15:24] rogpeppe1: I am not completely sold on the allow model access thing, what kind of access are you allowing? [15:24] perrito666: if you've been granted permission to a model, you get access to that model [15:25] perrito666: even if you haven't got any access rights to the controller itself [15:34] voidspace / dimitern / frobware / babbageclunk: https://github.com/juju/juju/pull/6406 has been updated to exclude LXD code on Windows. The rest of the code already has a +1. [15:34] dooferlad: looking [15:34] rick_h_: maybe that container fix will land after all... ^^ [15:35] dooferlad: oooh [15:46] rick_h_: it seems I won't manage to fix the public address bug tonight - as discussed with frobware, more detailed and specific connection checker tests are needed [15:46] rick_h_: but that should be OK, right? I mean even once in GA mode we can still fix bugs? [15:46] dimitern: ok, yes we can fix bugs but we're not going to be doing weekly releases now [15:46] dimitern: so it'll be some time before a follow up and such [15:47] dimitern: that's why my push to get things in so that users don't feel stuck as much as possible, but we do what we can [15:47] dimitern: right fix > quick fix [15:47] rick_h_: +100 [15:47] :) [15:48] ok then, cheers rick_h_ [16:03] * rick_h_ goes to get lunch [16:10] new kernel bbiam [16:28] voidspace: what was the other 'thing' you changed for LXD? I have the examples from bug #1631038 but have since close the HO chat... [16:28] Bug #1631038: Need /etc/sysctl.d/10-juju.conf [17:00] frobware: fs.max-files [17:05] voidspace: ty [17:06] voidspace: any particular value that will take me over 18 containers? [17:06] frobware: I read the current value and then added a zero... :-) [17:09] voidspace: should this be: fs.file-max ? [17:10] frobware: uhm, yes. Sorry. Dodgy memory. [17:10] voidspace: and what's the setting that requires a reboot? [17:11] alexisb: thanks for hitting $$merge$$ again for me! :) [17:11] alexisb: i filed another "sporadic test failure" bug... [17:11] rogpeppe1, np, I will keeo an eye on it [17:11] rogpeppe1, yeah we have a bug for that one [17:12] but thank you [17:12] alexisb: it's just landed, yay! [17:12] thank you rogpeppe1! [17:12] voidspace: do you have a link to the bug you created for tracking the packaging? [17:12] alexisb: ah, i didn't find the relevant bug, perhaps i should mark mine as dupe [17:13] yeah when a have a moment, I wil go cross check with our list [17:13] rick_h_: I'll do it now... [17:13] rick_h_: https://bugs.launchpad.net/juju-release-tools/+bug/1631038 [17:13] Bug #1631038: Need /etc/sysctl.d/10-juju.conf [17:13] voidspace: is that it? [17:14] voidspace: frobware ty [17:14] frobware: plus sysctl -p [17:14] frobware: and you need the max_user_instances set higher than the default 128 too [17:14] frobware: but that *should* work (worked for me) [17:14] alexisb: p'raps we should have a standard format for the subject line of sporadic test failures [17:14] alexisb: my bug is here FWIW https://bugs.launchpad.net/juju/+bug/1632412 [17:14] Bug #1632412: apiserver: sporadic test failure in pingerSuite.TestAgentConnectionDelaysShutdownWithPing [17:15] rogpeppe1: can we not dot it with tags? [17:15] rick_h_: https://bugs.launchpad.net/juju/+bug/1632414 [17:15] Bug #1632414: Package sysctl.conf file with juju [17:15] frobware: i guess so, although a standard form for the subject line would help too [17:16] frobware: tbh i didn't even know that bugs could have a set of tags [17:17] rogpeppe1: I add network often enough... :) [17:17] rick_h_: there is morre we could do on bug 1602192 [17:17] Bug #1602192: when starting many LXD containers, they start failing to boot with "Too many open files" [17:17] frobware: where do you see the tags in the bugs list? [17:17] voidspace: right, just want to get what we're doing for 2.0 to move it forward [17:17] rick_h_: but should we mark is fix committed now we have the warning in and a separate bug for the system settings [17:17] voidspace: and we'll look at next steps from there [17:17] frobware: ah, i see on the lower right [17:17] voidspace: hmm, not sure I want to mark it fix-committed [17:17] rick_h_: kk [17:18] rick_h_, voidspace: given that I've been asking for your additional settings can we update the bug to contain more than the two entries currenty listed? [17:18] frobware: there are enough "intermittent-failure" bugs that it's still hard to find the one that applies [17:18] frobware: voidspace please update the bug to have all the info needed so they can move that [17:18] frobware: i think the subject should at least mention the failing package and test. [17:18] rogpeppe1: rather like a (good) commit message... :) [17:19] frobware: exactly [17:19] frobware: anyway, now that you've brought it to my attention, i've added the intermittent-failure tag [17:19] frobware: thanks for that [17:20] rick_h_: I want to punt the completion of that to voidspace as I'm cargo-culting... but I think voidspace and jam have the absolute things to change... [17:20] frobware: k [17:22] rick_h_: happy to validate because ATM, conjure-up openstack localhost fails for me. [17:24] * frobware EOD [17:28] macgreagoir: looking at #6424 [17:29] macgreagoir: actually, #6424 has existing review comments that need addressing [17:30] macgreagoir: looking at 6393 [17:31] macgreagoir: I would go with rick_h_ for that PR :-) [17:32] macgreagoir: that one LGTM [17:32] rick_h_: there's a chance macgreagoir is EOD [17:33] rick_h_: shall I set "Change the grant admmodel permission to add-model" to merge? [17:33] rick_h_: https://github.com/juju/juju/pull/6393 [17:33] voidspace: looking, did it get the fallback going? [17:33] rick_h_: yep, there's a test for it too [17:33] voidspace: then yes please === frankban is now known as frankban|afk [17:57] voidspace / rick_h_: https://github.com/juju/juju/pull/6406 -- last change was a comment fix so it should be good to go. [17:58] dooferlad: looking [17:58] then EOD [17:59] voidspace: ty [17:59] dooferlad: ty as well [17:59] voidspace: thanks. EOD + 1h for me. This is back after the toddler has been put to bed for a last minute code poke. [18:00] rick_h_: you got there before I could review it [18:00] g'night all [18:02] frobware: whats the conjure-up failure from? [18:02] ive run it on yakkety and the charms fail because of ipv6 i think [18:03] stokachu: is the ipv6 issue noted? I thought we had the ipv6 issues licked where you'd get an ipv6 default address? [18:03] rick_h_: so with a fresh yakkety and juju rc3 a default deploy assigns ipv6 to applications [18:06] rick_h_: i think juju does right by giving it ipv6 but the charms fail? [18:06] haven't had a chance to dig further into it [18:10] stokachu: so ipv4 should be preferred, but if it has ipv6 then I guess that's ok except juju doesn't support ipv6 yet [18:11] rick_h_: i can boot another yakkety system here and run it again [18:11] rick_h_: will juju status show ipv6 even if it uses ipv4? [18:12] stokachu: so it should pick ipv4 if it's there as the preferred and not show ipv6. It shouldn't show ipv6 if it's using v4 [18:12] * rick_h_ is trying to find the bug/pr we fixed along those lines recently [18:12] rick_h_: ok [18:12] rick_h_: also would that be in rc3 [18:14] stokachu: https://bugs.launchpad.net/juju/+bug/1624495 yes, would be in there. [18:14] Bug #1624495: operations fails on rackspace because of ipv6 address in dns-name

[18:14] rick_h_: ohhh [18:14] so i should setup xenial to configure both ipv4 and ipv6 with lxd init [18:14] so ill do that in another test in addition to a default yakkety run [18:15] stokachu: maybe? I don't want to have issues with juju 2.0 not working with conjure-up I guess [18:15] stokachu: that's why I ask if you've filed a bug or something or what's up here. That's news to me that it's not working for you. [18:15] stokachu: and we're in lock down for GA at EOD today so kind of :/ [18:15] rick_h_: i just ran it this morning [18:15] stokachu: ah ok [18:16] stokachu: yea, please let me know if there was any ipv4 or if it was pure ipv6 and what versions of the stack we're looking at there. lxd tests on yakkety pass CI so curious if things aren't there for you as there's no father bug fixes in that area on the plate for tomorrow's freeze [18:17] rick_h_: ok im testing right now for yakkety and xenial [18:17] stokachu: ty [18:17] rick_h_: normally on xenial i dont choose to configure ipv6 [18:17] ill enable it though and see what happens [18:17] stokachu: ah ok, so it's working sans ipv6 then? [18:18] yea that works [18:18] stokachu: ah ok, I feel better then [18:18] but yakkety may be different i want to verify [18:18] stokachu: and it should work ok with ipv6 and ipv4 enabled, but pure v6 won't work [18:18] stokachu: k, ty [19:43] sinzui: I'm not really sure what the correct behavior for this bug is: https://bugs.launchpad.net/juju/+bug/1613858 [19:43] Bug #1613858: generate-tools ignores 2.x agents and creates bad path [19:45] natefinch: we have a lot of Wiggle room. the goal is to allow a private cloud to assemple strreams to boostrap with. 1. stop ignoring agents with major version greater than 1, then make a sensible dir structure as I show 1.x generates [19:46] sinzui: I don't know what you mean when you say the tool ignores them. as I said, I don't know what the intended behavior is [19:47] natefinch: with juju 1, run "juju metadata generate-tools -d mystreams --stream devel". it makes streams with the agents in the dir. it also makes the correct dir tree. [19:47] natefinch: doing that with juju 2, it should also recognise the 1x and 2x agents. [19:48] sinzui: I don't know what the correct directory tree should look like. Is it just a v2 where we have a v1 now? [19:48] natefinch: I think juju 1x should also recognise 2.x agents, but that is not the critical path to enbling a provate cloud [19:48] natefinch: The rules have not changed in 3 years! [19:49] natefinch: 2,x's output sould match 1x's output [19:49] natefinch: both jujus should include 2.x agents, but for today, we just need to get 2.x to make streams [19:50] sinzui: I really don't know how to say in any other ways that I don't know what the expected output should be. When I run juju emtadata-tools, I get this: http://pastebin.ubuntu.com/23309748/ [19:50] sinzui: what part of that is wrong, and what should it look like instead? [19:51] sinzui: assume I have no clue what streams stuff is supposed to look like... because I don't. [19:53] natefinch: that is why I keep suggesting to run with 1.x we used juju 1.15.0 to 1.25.2 to make streams with that command. the output works for juju. Juju 2 needs to make the same tree. Juju 1 and 2 need to bootstrap with that tree [19:53] ok [19:53] natefinch: that paste lookss right, BTW, except there are not agents shown in devel. [19:54]