[00:17] axw: I said in the standup that the blobstore changes to allow it support io.{ReaderAt,Seeker} turned out to be bigger than I thought they would be [00:17] axw: I just pushed through with it and it's not too bad [00:18] axw: although the requirements for the ReaderAt interface are a little tricky to get right, especially efficiently [00:18] axw: I've gone for correctness over performance [00:18] axw: parking for now though, will hopefully pick it up later today [00:19] thumper: dumb question, if I create a model, deploy a charm and add a unit, if I just destroy the model it'll cleanup that unit right? Or do I need to specifically remoev that unit myself? [00:22] * thumper afk for lunch [00:22] yes [00:22] it will clean up that unit [00:22] awesome, thanks th [00:22] thumper* [00:47] axw: https://github.com/juju/juju/pull/6308 when you have some time [00:49] anastasiamac: I also applied your recommendation wrt --reset as []string in https://github.com/juju/juju/pull/6308 [00:50] anastasiamac: apologies though, as I squashed it in with the other changes. [00:50] redir: \o/ u r wonderful! tyvm :D [00:51] menn0: ok, thanks. no drama if you don't get to it until later - it's just a nice to have [00:51] redir: looking [00:51] np , I'm on to make dinner, so won't be back until this eve. [00:51] axw: it definitely would be a lot better that way, and I am going to get to it [00:52] Hopefully, this is really close with trivials or less:) [01:57] thumper: axw: menn0: if u have any ideas/suggestions for "server not reachable" and/or "replicaset" intermittent failures that we are seeing, it'd be great to share them now... i'd love to improve this area as part of the sprint :D [02:11] interesting, is there a known bug currently where you have to model-config enable-os-update && enable-os-refresh-update to false? I'm seeing this on both gce and aws, investigating further as it pops up while testing. [02:24] lazyPower: yes [02:24] lazyPower: some lxd update is failing [02:25] natefinch: ta, i was looking for it and didn't find a bug. i was about to reproduce and file one [02:26] lazyPower: it's like the apt upgrade of lxd is failing or something, I forget the details, but it was not our fault, per se. [02:27] right, bit by an SRU or something [02:27] right [02:27] not even mad just trying to be helpful :) [02:28] :) [02:28] anastasiamac: you around? [02:29] natefinch: it depends on how difficult ur question is :D [02:30] heh [02:30] https://bugs.launchpad.net/juju/+bug/1625243 [02:30] mup: https://bugs.launchpad.net/juju/+bug/1625243 ? [02:30] natefinch: I apologize, but I'm pretty strict about only responding to known commands. [02:30] mup help [02:30] natefinch: yep, i'llb mup and got it :D [02:31] anastasiamac: anyway.... do you know what they're talking about? :) [02:31] natefinch: mostly about format of simplestreams files [02:31] juju should fix its code to know that https://identity.api.rackspacecloud.com/v2.0 Is a global URL expected to be accessed first before going to any regionalized service. [02:31] natefinch: tech board had fun discussing format this week too :D [02:31] what does "accessed first" mean? [02:32] natefinch: if u look at endpoint url, the one we understand have v2 [02:33] without region in it [02:33] the data that was supplied had region [02:34] comment 10 :) [02:34] yes [02:34] the comment that says "we could fix this, but nah" [02:34] yep :( [02:34] so, what I don't understand is their suggested fix [02:34] so rackpsace provider need to be modified to handle region in endpoint url .. [02:35] I guess I don't understand. It's a URL. Are we parsing it? What does it matter if the URL has a region in it? [02:35] my biggest disappointment with htis, if they've fixed data, all our current implementations (including 1.25) would work [02:36] since we ar fixing the provider only release with the fix will work :( [02:36] right [02:37] dunno more than that.. u'll need to try it out with both our current (residing in CI) data without region and supplied data with regions.. not fun but this what I would have been doing [02:37] to see what's wrong and why it matters... [02:37] the thing I don't understand is that the URLs they're listing in the bug don't have any data in them [02:37] for example https://iad.images.api.rackspacecloud.com/v2 [02:39] and the url that curtis says is correct (https://identity.api.rackspacecloud.com/v2.0) just returns some xml that does not look pertinent to juju at all [02:40] I also don't see a difference between the rackspace data and the rest of the clouds' data here: https://cloud-images.ubuntu.com/releases/streams/v1/index.json [02:41] they *all* have region in the endpoint url [02:41] "region": "eu-central-1", [02:41] "endpoint": "https://ec2.eu-central-1.amazonaws.com" [02:41] If I shrug any harder I'm gonna pull a muscle [02:42] natefinch: this looks like ec2 endpoint not rackspace.. curtis is the best person to talk about this bug :) [02:42] oh no - don't pull muscles \o/ [02:42] talk to curtis :D [02:42] no no, I know... I was showing an endpoint from another cloud that also had region in the endpoint, which they were saying was bad somehow [02:43] I will talk to curtis when he gets back on. [02:43] \o/ [02:43] m sooo happy that u r being thrown into simplestreams too :) [02:43] rofl [02:43] misery loves company [02:44] :D === rmcall_ is now known as rmcall [03:52] thanks thumper [03:52] np [03:52] thumper: I replied to your q, but was perhaps a bit terse. we don't care about the lifecycle state of the units, just that none have been added or removed [03:53] ok [03:53] fair enough [03:53] I was just wondering [03:53] you are good to land [03:53] ta [04:08] axw: https://github.com/juju/juju/pull/6309 pls [04:08] axw: reviewing your other PR now [04:19] menn0, thumper: list-disabled-commands now returns "''" instead of "[]" if there are no disabled commands, I imagine this is a bug and not expected? [04:20] no [04:20] I think I changed that on purpose [04:20] thumper: sorry, this is in format yaml/json too [04:20] s/too// [04:21] not sure now, sorry [04:21] that's a bug. It should only be for tabular [04:21] natefinch: cool, filing now. Thanks :-) [04:29] veebers: did you get that bug filed, 'cause I have the fix ready ;) thumper? +8 -2 https://github.com/juju/juju/pull/6310 [04:30] menn0: looking [04:30] natefinch: lgtm [04:31] thumper: thanks [04:33] natefinch: bug 1626824 [04:33] Bug #1626824: list-disabled-commands returns "''" instead of "[]" for json/yaml format [04:33] menn0: LGTM [04:33] axw: thanks. I now it's a backport but it was different enough that I thought it needed a review [04:36] natefinch: yep, https://bugs.launchpad.net/juju/+bug/1626824 [04:36] Bug #1626824: list-disabled-commands returns "''" instead of "[]" for json/yaml format [04:36] anastasiamac: hah, I'm to slow [04:38] veebers: :) nps [04:40] axw: just a few minor things for yours. ship it! [04:40] menn0: thanks [04:42] menn0: you really want me to add docs to each method on Backend and friends? can I just say "go look at the state package"? [04:43] axw: just describe the interfaces briefly (referring to state if that makes sense) [04:43] menn0: ok [04:43] axw: and maybe " implements " for the method implementation (although I'm fairly +0 on that) [04:43] implementations [04:44] menn0: okey dokey [04:59] veebers: fix landed, will be in next release [05:11] I get this periodically [05:11] ERROR cannot resolve charm URL "cs:xenial/ubuntu": cannot get "/xenial/ubuntu/meta/any?include=id&include=supported-series&include=published": Get https://api.jujucharms.com/charmstore/v5/xenial/ubuntu/meta/any?include=id&include=supported-series&include=published: dial tcp: lookup api.jujucharms.com on 127.0.1.1:53: server misbehaving [05:11] I think we should have a retry around the charmstore access [05:14] why does the ubuntu charm install g++ ? [05:14] it is getting build-essential [05:15] and python3.5-dev [05:15] seems weird [05:16] * thumper waits for them all to finish [05:20] well that didn't work [05:22] ah fuck... [05:40] * thumper calls it a day [05:40] later folks [05:47] anastasiamac: do you have time for a review? [05:48] axw: i wish but not really ;( m only marginally here - ghossting.. [05:48] anastasiamac: np [05:48] menn0: are you EOD yet? [05:48] axw: not quite [05:49] menn0: if you have time, would appreciate a review on https://github.com/juju/juju/pull/6311. [05:49] doh, doc string updates snuck in [05:49] axw: looking [05:55] axw: done [05:55] axw: all good [05:59] menn0: thanks === frankban|afk is now known as frankban [07:41] axw: Clean ups of uniter temporary download files: https://github.com/juju/juju/pull/6314 [07:42] menn0: cool, looking [07:45] axw: I just added some comments to the change about possible improvements. pls review anyway though. [07:50] Bug #1626878 opened: ERROR juju.worker.dependency engine.go [07:58] menn0: reviewed [08:05] Bug #1626878 changed: ERROR juju.worker.dependency engine.go [08:08] Bug #1626878 opened: ERROR juju.worker.dependency engine.go === jamespag` is now known as jamespage [09:53] hey menn0, I think I found it! [09:54] https://github.com/juju/juju/blob/master/state/cleanup.go#L155 [09:54] menn0: ^ [09:59] * menn0 is about to watch something :) [09:59] babbageclunk: that's awesome though! [10:00] I don't see the problem though :-/ [10:05] menn0: It deletes settings non-transactionally. If you're unlucky, you get this sequence: cleanup queued, undertaker runs, makes remove settings txn, cleanup runs, txn gets stuck. [10:05] menn0: I think [10:06] menn0: Making a binary now to get Jason to try it. [10:07] menn0: Isn't it a bit late to start watching something!? Kids don't sleep later on weekends. [10:07] :) [10:08] I guess he went to bed [10:16] frobware: ping [10:16] perrito666: ping [10:16] voidspace: pong [10:17] voidspace: be kind I am barely conscious [10:17] perrito666: babbageclunk says you had an issue with maas2Instance.Status being effectively cached and not updated per call [10:17] perrito666: heh [10:17] perrito666: are you fixing that or working around it? [10:17] voidspace: i thought it was that, but it might be something else, I fixed the call to be not cached anyway [10:18] perrito666: I wonder if it is behind the problem at Telefonica that frobware, macgreagoir and I have been looking at [10:18] voidspace: pong [10:18] perrito666: is it landed? [10:18] frobware: I'm wondering whether an issue that perrito666 found is behind the telefonica issue [10:18] frobware: it doesn't *entirely* fit with the logs, so I'm still trying to dig into it [10:18] voidspace: it is not landed, can you describe the problem? [10:19] perrito666: basically a maas2Instance.Status call always returns the same value (the status when the maas2Instance was created) [10:19] frobware: basically a maas2Instance.Status call always returns the same value (the status when the maas2Instance was created) [10:19] perrito666: oops, that was meant to go to frobware [10:19] perrito666: we're seeing delayed provisioning on maas [10:19] perrito666: making a large delay very slow [10:20] perrito666: frobware: so if somewhere in the code is polling Status waiting for it to change, then it won't [10:20] well I have both problems, not enough polling and the status being cached anyway [10:20] so I started by fixing the not enough polling [10:20] voidspace: heh, interesting. in trying to repro locally I think my MAAS setup is truly borked atm [10:20] perrito666: frobware: I can't find any polling in the code yet, but the code paths are quite convoluted [10:20] frobware: :-( [10:20] perrito666: when do you think your fix will land? [10:21] perrito666: happy to be a guinea pig before then [10:21] perrito666: I wonder if we can get binaries to them to check if this fixes the issue [10:21] frobware: it's provider/maas/maas2Instance.go:#L68 [10:21] voidspace: let me push up my changes [10:22] frobware: perrito666: it looks like this is called by the provisioner_task in classifyMachine [10:22] frobware: hard to see if it's polling though [10:23] frobware: the Status is checked if InstanceId errors, which it does if there is no real maas machine yet (so no instance data) [10:23] frobware: and we do have "found machine pending provisioning" log lines from that code path [10:23] voidspace: its something cherilj added some time ago [10:23] perrito666: what, polling? [10:24] polling of the underlying status for useful return iirc [10:24] right [10:24] so it *could* be the problem [10:24] voidspace: that is half of my problem solved https://github.com/perrito666/juju/tree/fix_1604965 [10:24] I am now trying to figure where are we exactly refreshing since my problem is that failed machines never gets marked as so [10:25] right [10:30] perrito666: you probably want to delete the TODO as well. [10:30] frobware: worth trying that branch to see if it fixes the issue [10:31] voidspace: yep, need to get a working MAAS2 atm [10:31] cool [10:32] voidspace: true, sorry was not thiking on uploading that :p [10:32] heh [11:04] frobware: You've done this lots - is sending someone a binary to run just zipping up juju and jujud? Anything else to send? [11:05] babbageclunk: I do [11:05] babbageclunk: cd ~/go [11:05] babbageclunk: tar -cvf ~/juju-2-0-rc1-for-brad.tar --transform='s!^!juju-2.0-rc1-for-brad/!' bin/juju* [11:05] babbageclunk: at least then it has a path in it when you untar [11:06] frobware: Awesome, thanks. [11:07] babbageclunk: it's also handy to s/bard/for-git-commit-sha1/ [11:07] babbageclunk: and to complete the metacircular evaluator, s/bard/brad/ [11:08] frobware: lol. In words, you mean, it's handy to include the git hash so you know what code they're running? [11:08] babbageclunk: yup. [11:08] frobware: Will do [11:09] babbageclunk: we should change our build to include this [11:31] frobware: Do I need to tell the recipient to use -build-agent? Will that work without the source? [11:32] babbageclunk: I haven't mentioned it for binaries I've given away [11:33] frobware: So hopefully it just does the right thing? I'm a bit nervous because I'm building off the beta18 tag and in my testing it kept trying to use the stream version unless I passed --build-version [11:34] babbageclunk: if you bootstrap with debug does it not show that it's using the juju from your $PATH? [11:38] No, it says http://paste.ubuntu.com/23219686/ [11:39] frobware: ^. I might rebase onto tip just to be sure. [11:41] babbageclunk: doesn't it say all the anyway; I think if you do: $(which jujud) -- and grep for that in the debug output [11:48] frobware: No, that definitely doesn't appear. [11:48] babbageclunk: hmm. so, with -build-agent? [11:49] frobware: The sha1sums don't match either (although I guess they might not with --build-agent either). [11:49] frobware: Huh, with build-agent they do. [11:50] babbageclunk: TIL [11:50] frobware: That could just be that go didn't touch the binary because everything was up to date. [11:57] frobware: Ok, a quick scan of the code suggests that it'll use jujud from beside the running juju binary as long as it matches the version. So I think --build-agent is the right thing. [13:18] Bug #1627015 opened: no way to specify additional storage for controller(s) [13:24] Bug #1627015 changed: no way to specify additional storage for controller(s) [14:01] frobware: voidspace natefinch ping for standup [14:01] omw [14:02] rick_h_: gah, browser hang [14:03] hangout just dropped me... [14:04] rick_h_: browser hang again - I can hear everyone but currently can't unmute! [15:41] morninf [15:41] had one of those mornings where system wouldn't display fonts after resume. [15:41] lol [15:41] then wouldn't resolve dns after reboot [15:42] it was a two reboot morning [15:42] dang [15:43] speaking of which, my "hangs for 7 minutes after reboot" problem mysteriously went away after 2 months of me never rebooting. :) [15:43] redir, you system is trying to tell you it is friday [15:43] right. I better update and reboot everything before I leave tomorrow [15:44] I feel a little lenient toward this machine not always working right. I't has been getting every upgrade since 2013, including betas [15:45] speaking of, i it time to upgrade to yakkety yet? [15:53] also have this ongoing issue where chromium doesn't display jpegs, on multiple different systems. Works on arch though. [15:53] * redir quits whinging and gets to being productive [16:44] frobware: ping [16:45] voidspace: pong - otp [16:46] frobware: ok, answer when you get a chance (you may not know anyway) [16:46] frobware: I'm trying to deploy a lxd (to the juju controller machine for convenience) and I get this error in the logs: [16:46] machine-0: 17:39:57 ERROR juju.worker.proxyupdater lxdbr0 has no ipv4 or ipv6 subnet enabled [16:46] It looks like your lxdbr0 has not yet been configured. Please configure it via: [16:46] sudo dpkg-reconfigure -p medium lxd [16:46] and then bootstrap again. [16:46] frobware: there is no lxdbr0 of course [16:47] frobware: and so the container is forever stuck in pending [17:02] voidspace: i'm back [17:02] voidspace: on MAAS? [17:03] voidspace: are you using rc1/ === frankban is now known as frankban|afk [17:36] natefinch: ping, did you get your info and unblocked on things? [17:38] rick_h_: yeah, I got it. The descriptions in the bug were evidently only one side of a conversation, more or less :) [17:38] natefinch: ok cool, thanks [18:16] Bug #1627138 opened: unable to bootstrap on openstack provider [18:31] Bug #1627138 changed: unable to bootstrap on openstack provider [18:40] Bug #1627138 opened: unable to bootstrap on openstack provider [18:46] Bug #1627138 changed: unable to bootstrap on openstack provider [19:42] good lord bzr + launchpand is slow. [19:46] anyone online know how to set up credentials for rackspace? It's asking for domain name, and that isn't described in our docs [19:49] gah, when you bootstrap rackspace, all the error messages talk about Openstack. [20:22] holy hell. bootstrapCommand.Run is an almost 500 line function. [20:23] only 500? [20:24] :/ [20:25] sorry natefinch I don't know about bootstrapping rackspace. [20:25] redir: it's ok.... either it doesn't matter or my bug is occurring before that field is checked :) [20:26] I am using delve for the first time ever, and it's amazing. Running it in Visual Studio Code is almost as good as debugging C# with the real Visual Studio.... and that's saying a lot for me. [21:48] natefinch: that is saying something. [21:48] but I've not debugged C# or used visual studio [21:51] natefinch: leave off the domain name? thought that was not needed [21:52] natefinch: search our bugs for domain name, thought there was one about removing it