[02:15] babbageclunk: https://github.com/juju/1.25-upgrade/issues/32 <- huh? what do you mean we don't use the local juju client? the plugin is called by it... [02:15] babbageclunk: or do you just mean, other than the calling into the plugin [02:16] axw: I mean, the plugin doesn't run juju - so it doesn't matter how they're installed. [02:16] babbageclunk: okey dokey [02:16] axw: sorry, those are mostly notes for myself for things that have occurred to me while testing with xavpaice [02:17] babbageclunk: np [02:17] axw: although, one I wanted to ask you about was the lxc-lxd conversion - it seemed like it didn't leave the containers running? [02:18] babbageclunk: yeah just saw that, that's odd - it's always worked for me... [02:18] axw: I haven't had a chance to dig in there - he's had to go do some real work [02:18] you can see in the code that we start the LXD containers and wait for them to be accessible [02:18] babbageclunk: in fact, the migrate-lxc command actually runs an SSH command to check that they're acccessible [02:19] Hmm [02:19] He ran that without showing me output. Next time I do it I'll get him to pastebin it so we can see. [02:20] babbageclunk: okey dokey [02:20] axw: how was assembly? Heather and I had a fun chat-meeting. [02:23] babbageclunk: not bad, just went because Charlotte was getting a certificate. the class item was a bit meh :) [02:23] stink [02:23] babbageclunk: did I miss anything? how're things going? [02:25] Going through an upgrade with xav has been super helpful - no blockers yet, I've been adding issues for things that could be smoothed out as they come up. [02:26] veebers: what's up with the "Can one of the admins verify this patch?" message from the bot? what does that even mean? [02:27] The lxc migration and import's done, but upgrade-agents hasn't been finished yet - we were just manually restarting the lxds and about to retry when he got interrupted by other stuff. [02:27] I'm getting set up with canonistack to test the fixes I'll make to openstack tagging. [02:40] axw: where are you seeing that? [02:40] axw: I'm trying out some jenkins things, I had assumed I had correctly limited it to just my one test branch [02:42] axw: to answer your question, the plugin I'm testing apparently has a whitelist of 'admin' that can actually trigger a build (and thus asks for one of them to comment on the PR) [02:44] Oh, I see where that might have leaked though. Sorry, that should be plugged now [02:53] veebers: no worries, was just curious - thank you [02:54] axw: I think I've removed all instances of the message. Sorry for the noise :-P [02:54] all good :) [03:29] axw: w00t - blahdeblah is happy with his upgraded canonistack model (modulo the security group problem) [03:29] * blahdeblah does a little dance [03:29] babbageclunk: sweet :) nice work [03:29] \o\ /o/ \o\ /o/ [03:30] blahdeblah: was there any lxc in your environment? [03:30] no [03:30] okey dokey [03:30] axw: zarroo lxcs present! [03:30] that's the thing i'm working on now with xavpaice [03:30] * axw nods [03:31] axw: actually, any ideas about this? https://pastebin.canonical.com/198968/ [03:31] specifically the ssh errors at the end [03:32] babbageclunk: the ssh_exchange_identification? [03:32] yup [03:32] babbageclunk: might be that the container's still initialising. try running migrate-lxc again, should resume from the right place [03:32] oh but... already migrated [03:32] so I guess that's what you/he just did [03:33] I can ask him to try it again again though [03:34] babbageclunk: doesn't hurt I suppose... but if that's what was already done, I wouldn't expect a change [03:35] babbageclunk: strange that it only happens on a subset of containers tho [03:35] maybe they are still coming up, and those ones are lagging behind the others [03:38] babbageclunk: the loop in waitLXDContainerReady was supposed to prevent this ... :/ [03:58] axw: Looking at this, https://unix.stackexchange.com/questions/156908/intermittent-ssh-exchange-identification-connection-closed-by-remote-host-err [03:58] What if we're creating too many connections to the host and overloading it? [04:00] babbageclunk: you mean to the host of the lxc container? possible [04:00] yeah [04:00] babbageclunk: get him to try with --match=x/lxc/y [04:00] to match just one container that's failing [04:01] axw: he got this before (not with --match) https://pastebin.canonical.com/198977/ [04:01] better [04:01] but still one failure even though it didn't need to migrate any [04:02] babbageclunk: yeah it still needs to stop the juju agents though [04:02] ok [04:02] Oh, for idempotence? [04:02] babbageclunk: yup [04:03] yeah, makes sense although wasn't obvious to me at first. [04:11] axw: If I want to throttle the max connections per host, what's a good way to structure that in Go? [04:11] axw: might be easier to just do a max number of concurrent exec tasks, rather than per-host. [04:17] babbageclunk: not sure there's a one size fits all, I think I'd just batch the errgroup.Go's and call Wait when the limit is hit [04:18] axw: ok, thanks [04:20] axw: I was looking at this: http://jmoiron.net/blog/limiting-concurrency-in-go/ but it seems a bit lower level - mixing it with errgroup seems clumsy. [04:22] babbageclunk: actually that's probably not a bad approach. fill up a channel to whatever capacity, take one on entry to Go, immediately after defer adding one back [04:23] babbageclunk: you could have a map of channels keyed on host addr too [04:24] yeah, makes sense - I might try that. [04:24] s/ might/'ll/ [05:30] axw: ha ha, fast [05:30] :) [05:30] reviewing the other now [05:31] axw: Thanks [05:32] axw: mostly wondering about the panicking/timing out stuff. Actually I think I'll change the panics to errors. [05:32] babbageclunk: sounds fine, I don't mind either way - it's a case where panic is reasonable, but we don't really do panics [05:34] just feels a bit icky to have panics and lots of goroutines [05:35] babbageclunk: your call, code looks sound anyway [05:35] cool, thanks [05:41] babbageclunk: did you find out about your flight yet? [05:48] axw: no, not yet [05:50] babbageclunk: hope you're not too tired for the tour. I guess we'll all be jet lagged anyway though. [05:51] I think the worst-case scenario is that thumper and I have to get the flight that gets into NYC at about 5 - in that case I'll probably bail on the tour. [05:52] ouch, yeah [05:54] the advisory at the moment lists our flight NZ8 as one of the ones not needing a refuelling stop - but it doesn't say whether that's for all days or just for now === frankban|afk is now known as frankban === frankban is now known as frankban|afk [20:49] thumper, what happened to github.com/juju/loggo/loggocolor [20:49] um... what do you mean? [20:50] github.com/juju/loggo/loggocolor is a 404 now [20:51] it is there for me [20:52] thumper, can you check the repo settings perhaps? unless I'm crazy [20:52] https://github.com/juju/loggo has it [20:52] where are you seeing it missing? [20:52] I can't import it [20:53] last commit into loggo was 5 june [20:53] so my build fails [20:53] I guess it's me.. has been all along [20:57] thumper, specifically cmd/juju/commands/debuglog.go import fails, which actually is the only place it's imported [20:58] which branch? [20:58] and where? [20:58] thumper, this is the archive package. I'm going to assume it's just more annoyingness [20:59] balloons: check the dependencies are updated properly [20:59] thumper, ack, I don't doubt it because we're building debs, snaps, and I can build manually myself. [20:59] but no dice for archive builders [20:59] does the commit hash of loggo match the dependencies.tsv [21:01] thumper, I'm willing to bet it's the debian rules messing with things [21:02] thumper, anyways, I'll head back to it [21:02] * thumper nods [21:02] ok [21:02] and yes, it matches [21:02] wat?... [21:02] FFS [21:02] ohh yep, it's debian/rules [21:33] * thumper feels his head is close to exploding [21:36] Oh dear [21:37] I mean, I feel like that too, but just because I'm getting so smart these days! [21:53] babbageclunk: care to be a teddybear? [21:53] thumper: of course! [21:53] https://hangouts.google.com/hangouts/_/canonical.com/stuff [21:54] babbageclunk: ^^ [23:37] did you just ragequit [23:37] thumper: ? [23:39] back [23:40] no it went to sleep (or thought it did) [23:40] I was thinking and talking, and didn't realise you weren't there for a while [23:40] babbageclunk: ^^ [23:40] thumper: I dropped anyway - team meeting soon [23:40] yeah [23:40] ok [23:46] babbageclunk: meeting?