[00:52] Bug #1632530 opened: 1.25.6 cannot deploy a charm to a Yakkety lxc unit (image-stream: daily)

[02:17] menn0: here's that fix for model-defaults https://github.com/juju/juju/pull/6442 [02:17] i want to make one more change which i'll do now in a separate pr [03:28] menn0: sorry, one more, follow up to the previous one https://github.com/juju/juju/pull/6443 [03:29] wallyworld: ok [03:37] wallyworld: review done [03:38] tyvm === thumper is now known as thumper-cooking [04:47] anastasiamac: pretty please? https://github.com/juju/juju/pull/6444 [05:16] wallyworld: what does bootstrap lxd do? [05:16] as long as it is just "lxd" I'm ok [05:16] thumper-cooking: lxd-localhost === thumper-cooking is now known as thumper-dogwalk [05:16] ick [05:16] as localhost is the region [05:16] i don't mind it tbh [05:16] you are always free to specify controller name [05:17] cause we could lxd lxd not localhost later [06:02] menn0: not sure if you're around? i need a huge favour - a review of the PR to swap bootstrap args https://github.com/juju/juju/pull/6444 . CI has been updated so I just need to land this and it should be good [06:03] wallyworld: was school pick-uping.. u always ping around this time... m thinking on purpose? :D [06:03] wallyworld: looking now [06:03] yay, tyvm [06:04] anastasiamac: i ping because the NZ folks are EOD (with daylight saving) and with other away on leave, there's no one else to bother :-) [06:04] and the europeans aren't on yet [06:08] wallyworld: reviewed :D [06:08] tyvm === thumper-dogwalk is now known as thumper [06:53] wallyworld: perhaps just not add a region if there is only one? [06:53] anyway [06:53] I don't really care too much [06:54] thumper: yeah, that could work too. i'm ambivalent about that bit [06:54] thumper: i guess update-clouds could add another region and then you'd be confused if you had already bootstrapped [06:55] wallyworld: I don't care that much [06:55] if I did, I could provide a name [06:55] wallyworld: see ya tomorrow === frankban|afk is now known as frankban [08:24] mgz: the latest build seems to be a mess. What is the plan? Want me to take a look at anything? [08:48] wallyworld: I'm here now [08:49] menn0: hey. it got reviewed and landed and is almost through CI [08:49] wallyworld: excellent [08:49] lots of green :-) [08:49] wallyworld: I tried to repo the functional-storage failure but can't get assess_storage.py to get past the bootstrap [08:49] wallyworld: so no local repro yet [08:50] hmmm, is it intermittent? I think so? [08:50] wallyworld: I fear the functional-storage ain't going to pass as it hasn't in ages [08:51] wallyworld: http://juju-ci.vapour.ws/job/functional-storage/ [08:51] wallyworld: and it's a strange/alarming one - not being able to find a trusty image on AWS apparently [08:51] but why only that test? [08:51] yeah, have no idea off hand [08:52] but surely that error is not related to storage functionality [08:53] we probably need to audit the bootstrap set up [08:54] wallyworld: the bootstrap is pretty conventional. I've checked the bootstrap config and it's all pretty standard stuff. [08:54] wallyworld: the test fails when deploying a charm which uses storage [08:54] and yet that test's bootstrap fails and others don't [08:54] so bootstrap works [08:54] wallyworld: bootstrap only failed for me locally so that could be environment [08:55] i'll take a look [08:55] wallyworld: when the test runs in CI it fails deploying that charm [08:55] but it's the machine that fails to come up [08:55] ok [08:55] so before the charm even gets used [08:55] so hard to see then based on that how storage is involved [08:56] wallyworld: indeed [08:56] wallyworld: the test also runs create-storage-pool a few times before deploying the charm [08:57] wallyworld: hard to see why that would matter though [08:57] yeah [08:57] and it seems when the machine does come up, the test pases [08:57] wallyworld: and when I do roughly what the test does manually, it all works [08:57] yeah, sure does seem like a test artifact [11:28] dooferlad: fwiw, both the failures in 'run-unit-tests-race/build-1967' look like obvious races on magic numbers in time. [11:36] wallyworld, rick_h_: trivial https://github.com/juju/juju/pull/6445 [11:36] sure [11:37] frankban: doh! i missed that [11:37] thanks! [11:37] wallyworld: np, it was quite confusing earlier ;-) [11:37] was all done in a bit of a rush [11:38] shipping it [11:38] wallyworld: oh you did, thanks [11:38] yeah, keen to see it land [11:38] so we can get another good Ci run [12:20] dooferlad: seems 4485 is back to something closer to normal again [12:33] wallyworld: ci merge job failed: not sure if it's spurious: http://juju-ci.vapour.ws:8080/job/github-merge-juju/9504/ [12:33] probs is let me look [12:33] yup [12:33] remerge [12:34] wallyworld: ok done [12:36] [LOG] 0:10.015 ERROR juju.worker.dependency "metric-sender" manifold worker returned unexpected error: Access is denied. [12:38] wallyworld: we don't seem to have a bug filed for this? [12:38] i haven't seen one [12:38] oh, we do [12:38] bug 1632006 [12:38] Bug #1632006: cmd/jujud/agent: sporadic test failure in UnitSuite.TestWorkers [12:38] oh, an intermittent failure [12:39] ^you can metoo that frankban [12:39] it is a pretty new one [13:06] meh, the docs for https://github.com/juju/loggo are out of date, some of the samples dont work anymore [13:06] Given that Go has runnable examples, that's a shame === freyes__ is now known as freyes [13:41] natefinch: call? [13:41] rick_h_: oops, sorry, coming [13:52] mgz: hiya [13:52] rogpeppe: yo [13:53] mgz: i've been looking at network.ResolveOrDropHostnames (part of poring over PrepareEndpointsForCaching logic yet again) and see that it doesn't preserve address scope information. is that deliberate? [13:54] it does not sound deliberate [13:55] mgz: i thought not too. i'll add a TODO and raise a bug. [13:55] but I don't think the logic there has changed in a good while either [13:55] mgz: it's only used in one place [13:56] a TODO and bug sounds good [14:03] mgz: https://bugs.launchpad.net/juju/+bug/1633089 [14:03] Bug #1633089: network: ResolveOrDropHostnames doesn't preserve address meta-information [14:03] rogpeppe: thanks! [14:04] dimitern: ^also of interest I think [14:05] rogpeppe, mgz: what's the case you're hitting? [14:05] dimitern: i've just been trying to understand the code, and that logic just seems wrong to me [14:05] rogpeppe: Scope is auto-discovered usually, it should only affect tests [14:05] dimitern: if we already have scope information, why should we throw it away? [14:06] rogpeppe: because it might be different? [14:07] dimitern: why would the scope be different after we've resolved an address? [14:07] rogpeppe: e.g. resolving "www.google.com" which was using ScopeCloudLocal (for some reason), will yield multiple [14:07] rogpeppe: IPs, with likely different scopes [14:09] dimitern: if the scope is always derived from the address, why store it in the Address structure at all? [14:10] rogpeppe: well, it's not always derived - only when using NewAddress() (which calls NewScopedAddress() with ScopeUnknown) [14:10] dimitern: well, it's derived when we call ResolveOrDropHostnames - the original info is just thrown away [14:10] rogpeppe: and it seems for Type==HostName Scope is preserved as-is [14:11] rogpeppe: so, fair enough - ResolveOrDropHostnames() should be [14:11] (why I keep pressing enter :/) [14:11] rogpeppe: should be preserved [14:12] dimitern: so for example if we've got a numeric IP address with Scope=CloudLocal, that information will be thrown away unless it's actually an address in the private-ip-address range [14:12] yeah, that's the wrong bit [14:12] rogpeppe: nope, we only try to resolve when Type==HostName (line 154) [14:14] dimitern: ha, that assumes that Type is set correctly (I think that the Type field should be dropped and made into a method tbh) [14:14] dimitern: actually, no you're right, it'll work even when Type=="" [14:15] rogpeppe: well, network.Address and HostPort were always intended to be plain old data structs [14:15] dimitern: nothing wrong with a plain old data struct with some methods that return derived information [14:15] dimitern: there's no need to denormalise info [14:16] dimitern: it just provides more ways for things to go wrong [14:17] dimitern: do you think there's any way a cloud-local host name can legitimately resolve to a public ip address? [14:17] rogpeppe: of course :) [14:17] rogpeppe: it's whatever the dns resolver returns [14:18] dimitern, mgz: hmm, ok, well maybe ResolveOrDropHostnames is correct then. [14:18] rogpeppe: but usually, without specially configured DNS it won't happen [14:18] hm, I don't know of any case that should happen [14:19] the (almost) same thing happens if you put "www.google.com 127.0.0.1" in /etc/hosts [14:19] but we do get hostnames in aws that resolve to differently scoped ip addresses [14:19] depending on the context of the request [14:19] dimitern: so i think what you're saying is that for host names, the scope is important because we can't tell any other way, but for IP addresses, we can always derive the scope from the address itself. [14:20] rogpeppe: overriding the scope is useful in tests, but it's almost always derived otherwise [14:20] dimitern: what about when we get the addresses from a provider instance? do providers always derive the scope too [14:20] ? [14:21] rogpeppe, mgz: yeah, good point - split-horizon DNS servers like on AWS can resolve "ip-x-y-z.ec2-compute.internal" to local-cloud address inside the VPC or to the public IP outside of it [14:22] rogpeppe: not all of them, some are explicit (vsphere, joyent, a few others) - but then they don't use NewAddress(), they use NewScopedAddress() or create the Address directly [14:22] dimitern: an address like that will never resolve to the public ip address outside, will it? 'cos ".internal" isn't a valid TLD. [14:22] dimitern: ok, right [14:22] dimitern: so the scope can be important information that's not derived [14:23] rogpeppe: it's not a gTLD, but you still *can* change your AWS DNS config to resolve "xx.yy.ec2-compute.internal" to e.g. "8.8.8.8" [14:24] dimitern: sure [14:24] rogpeppe: there was no easy way to figure out if a hostname is public or not - there are a few golang/x/net packages that potentially can be used for such detection, but wasn't needed so far [14:26] * dimitern afk for a bit [14:27] dimitern: thanks for the help [14:29] mgz: i closed the bug [14:30] rogpeppe: gotcha, sorry, some of the subtleties here had gone past me [14:30] mgz: me too :) [14:32] dimitern, mgz: i'm thinking that we should get rid of "unresolved-api-endpoints" too, and replace it with "resolved-api-endpoints". [14:36] rogpeppe: as long as it doesn't slow down each run of `$ juju ...` +1 [14:37] rogpeppe: as the reason to have both was to lazy-resolve-when-changed [14:37] dimitern: well, tbh the DNS resolution thing should be done in a totally different place. it really clutters up that higher level logic. [14:37] dimitern: anyway, what's wrong with using the usual DNS cache? [14:38] rogpeppe: the reason for resolving before trying to connect to the apiserver was mostly to work around broken maas setups [14:39] dimitern: how does it help there? [14:39] rogpeppe: it somewhat does.. not too much though [14:40] dimitern: how? surely net.Dial does the resolve anyway? [14:40] rogpeppe: it works because we have both IPs and hostnames coming from the maas provider [14:41] dimitern: i'm still not getting it [14:42] dimitern: (it's very useful to have the conversation now because i'm really really trying hard to simplify the impenetrable logic in PrepareEndpointsForCaching...) [14:42] rogpeppe: but (IIRC) if your nameserver(s) in /etc/resolv.conf are misconfigured - e.g. unreachable or worse - blackholed, the first time net.Dial tries to resolve a hostname causes at minimum 3s delay (TCP-stack-related quirk) and some times up to several minutes [14:43] dimitern: it causes a delay even to a concurrent dial request made to an IP address a moment later? [14:43] rogpeppe: I'm actually working on fixing the juju ssh (and later bootstrap) behavior with multiple HPs available to choose from [14:44] rogpeppe: and I'm doing this by proactively trying to connect to all HPs in parallel [14:44] (unfortunately your parallel.Try didn't work as I expected) [14:44] dimitern: so, if slow-DNS-resolution is the problem, a better solution would be to implement a DNS cache at the bottom level rather than cluttering up the top level, I think. [14:45] dimitern: what didn't work as expected about parallel.Try? [14:46] dimitern: with a DNS cache at the bottom level, we wouldn't need to bother munging the addresses in juju/api.go at all [14:47] rogpeppe: I'll try to explain another day :) [14:47] dimitern: Try was designed to fit just this kinda problem FWIW [14:47] * dimitern really needs to finish that connection checker [14:47] dimitern: np [14:48] rogpeppe: thanks for the discussion though - it's useful and we should chat more about it soon [14:48] ;) [14:49] dimitern: i should have a PR for your delectation before too long [14:50] rogpeppe: nice! (learned a new word today ;) [15:46] uh, what does this mean? [15:46] juju enable-ha [15:46] ERROR unsupported with hosted models [15:46] wat [15:46] marcoceppi: juju switch controller && juju enable-ha ? [15:47] marcoceppi: I thought that was fixed to auto do the right thing though, is this rc3? [15:47] y [15:47] yes [15:47] BLEH [15:47] switch worked, this is rc3 [15:47] marcoceppi: k, maybe it landed post-rc3, but yea models aren't HA, controllers are [15:48] * rick_h_ goes to make sure that was landed post-rc3 [15:48] lame [15:48] marcoceppi: totally, why we have landed a chance to correct that [15:48] I know models aren't ha ;) I want my controller ha bruv ;) [15:48] cool, thanks [15:48] totally [15:48] bruh [15:54] * rick_h_ goes to grab lunchables [15:54] marcoceppi: yeah, I hate that error message [15:55] marcoceppi: also, it shouldn't even be an error... HA only applies to the controller, not models [15:55] marcoceppi: so it should just do the right thing. [15:55] I think an INFO might be better [15:57] marcoceppi: natefinch https://github.com/juju/juju/pull/6421 [15:57] at the very least it should say "please run this command on the controller model" [15:58] "juju enable-ha now just works without requiring the controller model be selected." [15:58] rick_h_: awesome! [15:58] man I hated that error message... "hosted model" is just such a confusing term [15:59] it caught me off guard, someone asked about HA [15:59] and I was like I WILL SHOW YOU [15:59] and I couldn't show him :'( [15:59] but I made up some other bullshit [15:59] so it's fine [16:02] dooferlad: ping [16:05] frobware: sorry, about to go out. Super urgent? [16:07] dooferlad: which PR fixed https://bugs.launchpad.net/juju/+bug/1623480 [16:07] Bug #1623480: Cannot resolve own hostname in LXD container [16:12] frobware: sorry, don't have that to hand (on phone). Only two PRs from me in the last week or so - should be findable [16:23] rick_h_: fyi - https://bugs.launchpad.net/juju/+bug/1633126 [16:23] Bug #1633126: can't resolve lxd containers by fqdn [16:46] I need a voluteer[told} to help with relesae blocker bug [16:46] anyone available? [16:46] * perrito666 sighs [16:46] o/ [16:46] alexisb: [17:00] frobware: :( [17:00] I need another voluteer for a blocker bug [17:01] natefinch: ^ [17:01] alexisb: what's up? [17:01] https://bugs.launchpad.net/juju/+bug/1626187 [17:01] Bug #1626187: metricsManagerSuite.TestMeterStatusSuccessfulSend got false

[17:01] ^^^ is no longer intermitten [17:01] and minimum we need to show it is a required test update [17:02] I can take it [17:02] thanks natefinch [17:12] Bug #1632909 opened: ceilometer-api fails to start on xenial-newton (maas + lxd)

=== frankban is now known as frankban|afk [17:21] Bug #1632909 changed: ceilometer-api fails to start on xenial-newton (maas + lxd)

[17:24] Bug #1632909 opened: ceilometer-api fails to start on xenial-newton (maas + lxd)

[17:50] natefinch, anything thoughts on the test failure? [17:50] cmars: ping, got a sec? [17:51] cmars: can you pair with natefinch on the https://bugs.launchpad.net/juju/+bug/1626187 failure to help unblock GA? [17:51] Bug #1626187: metricsManagerSuite.TestMeterStatusSuccessfulSend got false

[17:51] rick_h_: yeah, looking through it... I'm sure it's sjust a timing issue. Might be able to restructure the test so that we don't depend on the time. [17:52] natefinch: k, I just want to make sure that if there's anything in that topic that cmars might be able to help [17:53] rick_h_: definitely [18:02] need to reboot, brb [18:06] rick_h_, natefinch sure [18:07] one sec [19:07] cmars: btw, I was able to reproduce that failure if I added a 2 second sleep to the beginning of that test... testing with your fix and a 2 second sleep to make sure it does the right thing (which I'm sure it does, since the mock clock doesn't care about real-world time) [19:08] natefinch, awesome, that supports my hypothesis about the sequence of events leading to failure [19:08] natefinch, initial tests have passed, i've $$merge$$'d [19:10] ug! model-defaults is still the old table heading format [19:12] cmars: yeah, double checked that the fix works even with a sleep at the beginning of the test. Awesome. [19:12] natefinch, thanks! [19:12] natefinch, is the build bot stuck? it doesn't seem to pick up my $$merge$$ [19:13] rick_h_, any idea why this isn't landing? https://github.com/juju/juju/pull/6446 [19:13] cmars, master is blocked for the release [19:13] lol [19:14] release is blocked on this PR [19:14] so... deadlock? :) [19:14] cmars, I spoke with QA and given you have shown it is a test failure they are good wth the release as is [19:14] cmars, natefinch thank you! [19:14] __JFDI__ ? [19:14] oh, ok, sure [19:22] brb [19:40] redir: back from school duties. you still heading up from the bottom? [19:40] redir: should I start at the top and work down? [19:41] rick_h_: I started at the top and am working down [19:41] redir: ah ok, should I start at the bottom then? [19:42] sure [19:43] k, /me starts cutting/pasting [19:46] rick_h_, redir, I am going to change locations while you cut and paste :) [19:46] then finish my audit of the table [19:46] word [19:46] alexisb: rgr safe travels [19:46] i mean ack [20:02] rick_h_: back in HO [20:02] redir: k, omw [20:29] veebers: are you aware of this: https://docs.google.com/document/d/1sDPlv7AxyPTfaVfh0Yj6Pwr1I_jKOcbh4fQs-yB8GtY/edit [20:29] veebers: sorry wrong link :) [20:29] veebers: http://juju-ci.vapour.ws:8080/job/github-merge-juju/9506/console [20:30] * veebers looks [20:30] menn0: I'll bring it up with the team now [20:32] menn0: balloons will take care of you :-) [20:32] menn0, no more PR's against master. But since you made one, I'll let you play guenia pig on the message [20:33] give me 30 mins, I'll ping you.. otp [20:33] balloons: I have a PR against master? [20:33] menn0, ohh I just assumed === rmcall_ is now known as rmcall [20:33] ahh it is develop.. no worries [20:34] balloons: and it's not mine, I just noticed it [20:34] anyway, it's disabled for release still [20:34] balloons: it's a fix for the only thing that failed in the last CI run :) [20:35] I was going to be happy that juju bootstrap lxd works.... except that juju bootstrap lxd && juju show-controller lxd ... does not work *sad trombone* [20:39] natefinch, what do you mean? [20:39] I have been bootstraping all day [20:39] alexisb: it's called localhost vs lxd [20:39] alexisb: so you can boostrap lxd, but the name will localhost [20:39] well yeah unless you rename it [20:40] I may be coming into th emiddle of a conversation where I dont have the context [20:41] nope, that was basically it [20:41] if I juju bootstrap azure... what's will the name of the controller be? I have no clue. [20:42] natefinch: do try, you have credentials for it [20:42] natefinch: you can always specify the name if you want to be sure [20:42] I'm not saying I can't figure it out, I'm saying it's bad UX [20:42] menn0: I have OSs for answers like that :p [20:42] but I'm also aware that there are Those Who Disagree™ [20:50] natefinch: the controller name is cloud-region [20:51] if you don't want that, just supply the name as was the case before, your choice [20:51] and it tells you what the name of the controller is when it bootstraps, so you would have seen the controller was not called just lxd [20:59] wallyworld: I stick by my point that it's bas ux. I don't know what the default region for every cloud is. I shouldn't have to analyze the mound of output generated from bootstrap to figure out what the controller name is. And the region is not such critical information that it deserves to go into the name, IMO... especially since it's always the same per cloud. [20:59] s/bas/bad [20:59] but anyway, I gotta run for dinner. I know it's not going to change. [21:01] dinner? man, I just had afternoon tea and I am one hour ahead of you [21:06] can I get a +1? https://github.com/juju/juju/pull/6448 [21:10] I need a volunteer who enjoys reading [21:11] OMG, a bless [21:11] * thumper hides from alexisb [21:11] NICE!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! [21:13] ok thumper, menn0 you guys need to play rock paper scissors [21:13] * perrito666 throws holy wather at the build [21:13] * thumper just played, and he won [21:13] * thumper hands to menn0 [21:14] :) [21:14] I see menn0 is hiding [21:15] menn0: is writing a carefully craft bike shedding email [21:16] menn0, when you have a moment: https://hangouts.google.com/hangouts/_/canonical.com/core [21:19] wallyworld you are quiet. Can you +1 this to bump the revision number? https://github.com/juju/juju/pull/6448 [21:19] sure [21:20] menn0, ?? [21:21] balloons: lgtm [21:27] congrats everyone on the release! [21:27] thanks for all your hard work! [21:27] party time [21:27] Bug # changed: 1318378, 1328269, 1349854, 1632530 [21:28] Bug #1589680 opened: Upgrading to cloud-archive:mitaka breaks lxc creation

[21:37] Bug #1589680 changed: Upgrading to cloud-archive:mitaka breaks lxc creation

[21:37] Bug # opened: 1318378, 1328269, 1349854, 1632530 [21:40] Bug # changed: 1318378, 1328269, 1349854, 1632530 [21:40] Bug #1589680 opened: Upgrading to cloud-archive:mitaka breaks lxc creation

[21:55] cmars, if you're still about, I restarted your merge. Things should be open now [21:56] balloons, no rush, but thanks. just thought y'all needed it earlier for the release [21:56] cmars, ack. We had to cut and go with the info we had it [21:57] balloons, sure, it was just mocks and clocks :) [22:17] wallyworld: the release notes say that Juju now uses the AWS instance type API directly but that's not quite right is it? [22:17] no [22:17] wallyworld: it looks like axw introduced some tooling to automatically generated the hardcoded instance types [22:17] wallyworld: i'll remove that [22:18] ok, ta [22:19]