[00:04] thumper: I am about eod and haven't been able to reproce the race in bug #1579051 . Looking through the test my best guess at this point is https://goo.gl/zmw7u8 [00:04] Bug #1579051: Race in juju/controller/destroy and TestDestroyCommandConfirmation

[00:05] redir_: ta [00:05] thumper: I understand from wallyworld you would be looking at that today [00:05] redir_: will do [00:06] redir_: not sure it is that [00:06] pushing onto a channel doesn't need a lock [00:06] unless the channel was created after the last sync point [00:06] which isn't the case here [00:10] thumper: OK. best wrong guess. [00:15] redir_: it seems to be related on the fact that the tests are writing on dummy.dummy.ops [00:15] I am just guessing from a pass over the code [00:16] that might explain the fact that its intermittent, the order in which tests are run [00:17] redir_: but someone that knows a bit more about the underlying machinery in go should chime on that [00:17] davecheney: ? [00:20] ok, using davecheney's stress script with the race flag, I have the race condition regenerated [00:20] * thumper looks deeper [00:21] thumper: I put my coins in too many people suspiciously writing on that shared object plz share the result if you find it :p [00:27] perrito666: I figured it was the estate.name read in the anonymous func [00:27] * redir_ wants davecheney's stress script [00:28] ugh... reminded that there is global state [00:33] ok, I got this [00:37] * redir_ lurks [00:39] redir_: wanna see? [00:40] thumper: yes [00:40] pushing now [00:40] cool [00:40] thumper: there's another that I was solving by pure WAG at http://reviews.vapour.ws/r/4813/ [00:41] http://reviews.vapour.ws/r/4815/diff/# [00:42] ugh [00:42] spotted a typo [00:43] menn0: http://reviews.vapour.ws/r/4815/diff/# [00:43] Um, that looks a lot like https://goo.gl/zmw7u8 [00:43] thumper: ^ but with a fine comment [00:44] but whatevs [00:44] is there any docs or pointers to the stress script you used? [00:44] thumper: ^ [00:45] and thanks for sharing. [00:45] thumper: I remember reviewing redir_'s change recently which is the same as yours [00:46] heh [00:46] I am so tired [00:46] menn0 yesterdays was a different one, which didn't entirely fix it. There's an additional change for that at http://reviews.vapour.ws/r/4813/ [00:47] a different race [00:47] you ar right... [00:47] redir_'s fix is the same [00:47] I just added more of a comment :) [00:47] and I have confirmed it fixed it [00:47] ok cool [00:47] well the comment is good [00:48] sadly I don't know if I am fixing them, because I can't repro them [00:48] and redir_'s additional changes need to go in too [00:48] * redir_ goes EoD [00:49] see you tomorrow #juju-dev [00:56] mwhudson: ha, thanks for proposing that fix [00:56] mwhudson: I don't know how they can live like that [00:57] natefinch: i'm pretty sure last time i looked at it, half the packages in git didn't compile [00:58] the problem in that issue makes the packaging fiddly, which is how i ran into it [01:00] mwhudson: I can understand having a lot of junk shoved in closets, but when the window in the front door is broken... geez, just fix it. [01:11] lol, 2 failed uniter tests overflowed the default 3000 line buffer in my windows command prompt [01:12] and by lol, I mean, goodbye 15 minutes [01:21] natefinch: oh, never run tests in windows without > blah.log [01:21] perrito666: yeah, evidently [01:21] then you can read that with notepad :p [01:22] it would be funny if it weren't so sad [01:23] perrito666: I'm just going to install vs code on this machine [01:24] sinzui: what's with the windows apiserver/service bug getting marked fixed released? [01:25] this one: https://bugs.launchpad.net/juju-core/+bug/1580184 [01:25] Bug #1580184: Timeout in github.com/juju/juju/apiserver/service on windows

[01:26] I actually code on windows when fixing windows bug [01:26] perrito666: exactly, which is why I am going to put vscode on that VM :) [01:27] I did the same :) [01:27] I've got a juju subordinate charm stuck in allocating, I can't even see references to it in the juju logs - any ideas how I can get it moving again? or destroy it so I can redeploy? [01:30] bradm: juju destroy-service should work, I would think? [01:31] redir: http://dave.cheney.net/2013/06/19/stress-test-your-go-packages [01:31] sorry for the delay, the internet was totally crapped out here [01:31] natefinch: its a subordinate, and I want the rest of the others to stay [01:32] bradm: so just one unit of it is stuck? [01:32] natefinch: I tried juju destroy-unit and it told me to bugger off, its a subordinate [01:32] natefinch: yeah [01:32] natefinch: landscape-client, so there's basically subordinates everywhere [01:32] bradm: ahh, yeah, I see a caveat in the bottom of the docs that says you can't do that. Is it feasible to destroy the host unit? [01:32] natefinch: no. [01:32] natefinch: the host unit is the juju state server [01:34] bradm: not the host machine, the unit that the subordinate is attached to. I don't know that it'll help, but it's possible [01:35] natefinch: oh, I see what you're saying. hmmm. [01:35] thumper: any ideas? ^ [01:35] natefinch: we don't have juju ha due to bug 1575448 so I need to be super careful [01:35] Bug #1575448: trusty juju 1.25.5 HA availability issues

[01:39] anastasiamac: are you looking that that one? ^ [01:39] natefinch: hang on, m reading backscroll [01:40] the jujuds are as flakey as anything, constant errors in it [01:41] natefinch: it's with me but atm I need to reliably reproduce it. As part of this, m building vMAAS with 1.9... and m not there yet [01:41] natefinch: do u want the bug? [01:41] anastasiamac: no no no no no [01:41] natefinch: and here i was holding my breath \o/ [01:42] anastasiamac: it's honestly probably nothing to do with maas, but I suppose you never know [01:43] natefinch: yes, u may be right - our CI passes with maas and ha... but it does not help bradm ;) [01:44] natefinch: and i still would like to have my own maas 1.9 :/ [01:44] natefinch: consider it my bday wish :D [01:44] anastasiamac: I tried that at a sprint one, with the maas guys in the same room, still never got it working. But that was a while ago, maybe it's better now [01:44] anastasiamac: tap your heels 3 times and think on an orange box [01:45] what we need is vmaas that rides on top of lxd... lxd generally Just Works [01:45] I just don't get what we're doing different to cause this much instability [01:45] perrito666: it only works with red shoes and every time i try to go shoe shopping, wallyworld says i need to work :-P [01:48] bradm: I wish I knew. [01:50] so the only suggestion is to destroy the parent service? [01:51] the only evidence I see of the subordinate is a directory /var/lib/juju/agents/unit- with an agent.conf in it, but its not even fully filled out, when comparing it to others on the same node [01:52] natefinch: i had a maas controller runnign yesterday, could not add nodes to it. today my machine keeps dying, i can;t reach maas controller anymore, m back to rebuilding it [01:52] aghhhhh I need to test something in 1.25 and my environments.yaml is no more [01:53] perrito666: what do u need? i can email u mine [01:53] found a backup [01:53] anastasiamac: I dont think you have my lxd conf :p [01:54] bradm: I honestly have not dealt with subordinates hardly at all. Others might have more information. wallyworld? [01:54] perrito666: u r on ur own \o/ hope u swim well and r not afraid of heights [01:54] i've not used them much, other than to be under the impression their lifecycle is bound to the primary unit [01:54] (or dark) [01:59] might spin up a test env and see what I can work out [01:59] I don't think destroying the service will help, since there appears to be no real trace of the subordinate on the problem node [01:59] as in, the subordinate service [02:02] ahoy, anyone for a spot of team meeting ? [02:02] davecheney: coming [02:02] thumper: coming? [02:03] bradm: destroy/recreate *should help* I'd think, since a new subordinate should be created for the new unit [02:03] natefinch: wallyworld:thumper: team meeting? [02:03] axw: ping [02:09] wallyworld: hangout? [02:09] rick_h_: sure, give me a couple of mins? i'll ping you in a sec [02:09] k [02:14] rick_h_: https://hangouts.google.com/hangouts/_/canonical.com/ian-rick [02:15] wallyworld: can you invite me please? [02:15] ok [02:15] ty [02:21] thumper, wallyworld can one of you find someone to take bug 1580802? [02:21] Bug #1580802: NoContextWithLock fails on windows because of another process

[02:22] anastasiamac, davecheney - are you guys okay with the week of june 13 for a sprint at that same hotel near brisbane? [02:22] if so, I'll submit the sprint request [02:23] cherylj: i am \o/ [02:24] cherylj: it was summer and nice to be by the oocean in December... m not sure it'd b just as great in June (winter here) [02:26] oh ha, yeah, btw guys - -race is now run with the landing bot [02:26] fun === rodlogic is now known as Guest47582 [02:37] cherylj: awesome about -race [02:38] not so much, looking at the last few runs [02:45] cherylj: it has to be in sydney [02:45] i've been travelling to much [02:46] i'll be in the doghouse if I go away for another week [02:46] anastasiamac: your thoughts on ^^? [02:46] cherylj: sydney sounds good [02:47] sydney it is, then [02:48] 👍 [02:48] coverage overhead on my machine, 18.5 minutes instead of 14.5 minutes [02:48] thumper: that's not really all that bad [02:49] thumper: and what is mean (or average) coverage that u r seeing? [02:50] I varies [02:50] could u pastebin the output? [02:50] I've just closed up to go to a meeting off site, I'll let you know later [02:50] is there someone who can take bug 1580802? [02:50] Bug #1580802: NoContextWithLock fails on windows because of another process

[02:51] cherylj: I just got my windows environment all set up, I could look at it in the morning [02:52] is there someone who could take it now (in their daytime)? I'm looking at you, Aussies and Kiwis :) [02:53] cherylj: m not sure who in Aus and NZ has windows test environments... [02:54] sinzui: are you still around to provide access to the windows machine? [02:58] cherylj: if no one took it tomorrow AM I can also take it [02:58] for the moment working on https://bugs.launchpad.net/juju-core/+bug/1577939 [02:58] Bug #1577939: Backup-restore failed on xenial because service "juju-db": No such file or directory

[02:59] thanks, perrito666 [03:01] k brain is failing, going to sleep, see you al tomorrow earlier than I would like [03:02] ditto [04:13] anastasiamac: coverage from master http://pastebin.ubuntu.com/16372159/ [04:14] wallyworld: any idea what is happening here: http://juju-ci.vapour.ws:8080/job/github-merge-juju/7709/console ? [04:14] waaaat [04:15] why did we have to turn on the race detector *now* [04:15] when we are trying to get a blessed build for beta7 [04:15] surely it could have waited a couple of days [04:15] haven't seen those errors before [04:16] me neither [04:16] but it looks like they are trying to run with -race [04:25] wallyworld they should turn it off again [04:26] +1 from me [04:26] or at least update the package with the fixes that michael landed recently [04:26] we weren't told it had happened till after the fact === rodlogic is now known as Guest11700 [05:15] wallyworld: would you please take a look at https://github.com/juju/juju/pull/5383 [05:15] wallyworld: ignoring the first commit, which is already reviewed [05:15] sure [05:15] just give me 5 [05:15] wallyworld: no worries, take your time [05:40] axw: lgtm, thanks [05:40] good luck landing it though [05:41] wallyworld: thanks. why's that? [05:41] axw: see backscroll abut race detector [05:41] wallyworld: ah :/ [05:41] it seems they need upstream fixes, plus our tests have races [05:43] wallyworld: I've just had another look at your API doc, it looks good to me. I'm not sure if there is anything we need to change. probably just keeping the details like media types at the end as you've done is enough [05:43] axw: ta, yeah i'm hoping that's the case too [05:44] wallyworld: also I do like the option of fetching all service details at once (with _embedded), but I wonder if that should be toggled with a query option? [05:44] so I did a juju add-unit to the state server, and a juju destroy-unit to the one with the failing subordinate, and it won't die, its still waiting for the failed subordinate to come up I think [05:44] axw: if you GET against the collection, you'll get a list of services matching any query params passed in. or am i misunderstanding [05:45] whereas GET against the service url itself returns just the one [05:45] wallyworld: what are the query params? I don't see mention in the doc [05:45] wallyworld: ah, further down [05:45] yeah [05:46] the top bit is just a summary [05:46] hides the gory details [05:46] bradm: not sure if --force is an option? [05:46] wallyworld: so with the REST approach you'd just hit "/services", and traverse the _links right? but the response is also going to contain all the services in _embedded [05:46] wallyworld: nope, its not [05:47] wallyworld: at least, according to the help its not [05:47] axw: oh i see, we want a way to get *just* the links [05:47] wallyworld: yeah I think so [05:47] wallyworld: yup, I get the "error: flag provided but not defined: --force" if I try [05:48] axw: yeah ok, we can do that, i might leave that off for now as i expect/hope we won't get that far. if i get time i'll add something [05:48] bradm: so do you know what's wrong with the subordinate? [05:49] wallyworld: it just didn't start up [05:49] wallyworld: the only evidence of it on the node is a partially filled out agent.conf [05:50] oh the agent didn't come up [05:50] "Waiting for agent initialization to finish" is the message [05:50] i have no idea how to beat a subordinate unit into submission [05:50] time for another bug then, I guess [05:50] i fear it may require a db hack and an agent restart [05:51] ugh [05:51] this is about to go live too, its one of the things holding it up [05:51] but that's just me, there may be a better answer [05:51] william would be a good person to ask [05:52] a post to juju-dev mailing list would be good [05:56] juju-dev@lists.u.c ? [06:20] axw: interesting. I would have thought each API request would be a separate span. rather than each connection (given agents stay connected for days) [06:20] and then you'd just tag the spans with the username that is connected. [06:20] jam: yeah, I think each request would be better (after looking at the results) [06:20] and then you definitely want to start breaking down spans so that you can do things like how long the DB request took, etc. [06:20] jam: it would be sometimes helpful to have a span per connection, e.g. for user connections [06:21] i.e. user commands making multiple API requests [06:21] axw: it sounds like zipkin/whatever is going to be reasonably similar, probably much flashier with represntation and ability to dig into it. [06:21] jam: yes, I think so [06:26] wallyworld: email away to juju-dev, hopefully someone knows how to fix it === rodlogic is now known as Guest493 === frankban|afk is now known as frankban === rodlogic is now known as Guest85705 === thumper is now known as thumper-afk [09:02] dooferlad: standup? [09:06] voidspace: for some reason my mic and camera won't work today. Great. [09:06] dooferlad: ok dude [09:13] dooferlad: dimitern: frobware: firefox crash - will have to rejoin [09:13] dooferlad: oops, not you [09:13] voidspace: :-) [09:14] dooferlad: ah, you are there [09:14] dooferlad: o/ [09:14] voidspace: I rebooted [09:14] :-) [09:14] the classic fix... [09:29] dimitern: I have the email suggesting "ID := :" [09:29] e.g. "uuid:space#foo" (for a space provider ID) [09:29] dimitern: so I'll do that [09:29] dimitern: may ping you about it later [09:31] voidspace: just forwarded the last thing I sent on that thread [09:31] voidspace: sounds good, thanks! [09:31] dimitern: kk [09:47] Bug #1580946 opened: Juju 2 help commands for constraints or placement return ERROR unknown command [10:36] dimitern: FYI I am writing some stuff for thumper to give him some context about networks. If you could start on the state of Juju and what we can combine our efforts later. [10:37] ...Juju and what we have tried... that is [10:37] dooferlad: ok, I'll look into it later today [10:59] Bug #1580964 opened: LXD network container profiles need not be discrete entries === rodlogic is now known as Guest71281 [11:08] axw, menn0: hey, is anyone of you around by chance? [11:36] * dimitern wish in addition to restricted, there were auto-inherited-from-controller model attributes === rodlogic is now known as Guest50981 === freyes__ is now known as freyes [13:16] wallyworld: thumper-afk: I removed the --race from the lander and your branches are requeued [13:16] yay, tyvm [13:16] ^ alexisb [13:17] thank you sinzui! === dimitern is now known as dimitern|away [14:33] Bug #1581069 opened: controller leaking mongodb connections [14:36] jam: do we have a cloud feature / juju feature matrix that you know of? Is that something you could help put together? [14:37] jam: thumper was asking for it as well as some networking background. I am handling the networking background at the moment... [14:45] Bug #1581069 changed: controller leaking mongodb connections [14:48] jcastro: ping got a sec? [14:50] rick_h_: shoot [14:50] jcastro: you played with the add-user in 2.0? and registering/etc? [14:50] I have not yet, no [14:51] jcastro: so I'm setting it up for a demo and just tried it out and was awesome to email a command to myself, go to another machine, paste it, and then run juju-gui to have access to a subset of the controller [14:51] jcastro: I've got a school event in the AM tomorrow, but curious if you'd have time to incldue that in the office hours? [14:51] oooh, we have office hours this week [14:51] yeah [14:51] I'll work it in somehow [14:51] jcastro: or someone else there [14:51] I'll sent you notes/etc if you want before EOD, but this is cool/worth sharing imo [14:52] * rick_h_ volunteered to make school popcorn day [14:55] * perrito666 has pavlovian conditioning that onlly allows him to eat popcorn while watching movies [14:57] Bug #1581069 opened: controller leaking mongodb connections [15:03] Bug #1581074 opened: LXD container interface names don't align with parent interface names

[15:05] ericsnow: standup time [15:12] redir: http://dave.cheney.net/2013/06/19/stress-test-your-go-packages [15:13] natefinch: dave gave him the link last night [15:16] perrito666: hey would you mind joining moonstone rq? https://hangouts.google.com/hangouts/_/canonical.com/moonstone?authuser=1 [15:16] katco: sure, gimme a sec [15:35] dimitern|away: is it true legacyipaddressesC is used for "address allocation" on aws? [15:40] alexisb, cherylj: bug https://bugs.launchpad.net/juju-core/+bug/1580802 was diagnosed by davecheney as yet another fslock bug. Fixing it should probably involve not using file locks on windows, given that there's a syscall specifically for application-wide locks like this (CreateMutex). It's not a huge lift, but it's not a minor bugfix either. Not sure what you want to do. [15:40] Bug #1580802: NoContextWithLock fails on windows because of another process

[15:40] natefinch, otp one sec [16:10] katco: ping? [16:10] cherylj: otp, what's up? [16:11] katco: just coming up for air from my meetings, wanted to follow up on providing that binary [16:11] cherylj: other channel? [16:17] ok natefinch sorry reading back [16:17] natefinch, we agree [16:18] we need to fix that but the fslock issue is more then what is doable in beta timeframe [16:42] alexisb: ok, looking into what we can do in a limited time frame. [16:45] anyone have time for a quick review? http://reviews.vapour.ws/r/4817/diff/# [16:46] perrito666: tests? [17:04] Bug #1568895 changed: Cannot add MAAS-based LXD containers in 2.0beta4 on trusty

[17:04] Bug #1581138 opened: Juju20: websockets API unhelpful error message when not specifying support facade Version parameter

[17:04] cherylj, sinzui: updated the cloud-images and ran some tests, both centos with new tls and trusty with backports verified working [17:04] cherylj, sinzui: feel like I'm forgetting something else thouhg? [17:05] you know your lock object is buggy when it needs a BreakLock method :/ [17:05] mgz: I don't think you are. the windows jobs are a separate issue [17:06] mgz: huzzah about centos :) [17:06] mgz: and we are also testing centos in aws [17:06] I have a question in #juju regarding "chaining" subordinates that I think I might need a core dev to answer [17:06] If someone has a moment [17:07] mgz: cherylj I have hacked the registry of the windows host. I am replaying the win tests, I am not certain go the keys/items right. [17:07] sinzui: bogdanteleaga reckons the real problem is the tests actually building/using jujud [17:08] cory_fu: we can try... but most of us aren't too familiar with subordinates. And chaining them sounds like dangerous waters. [17:08] cory_fu: I'm pretty sure the answer is just "that doesn't work" [17:08] but you can have a subordinate that only operates if it has a sibling subordinate [17:08] seems like kind of a bad thing, but hey [17:09] I don't see why it shouldn't. There's a princpal at some level, and the relationship I actually want to model is between the two subordinates [17:10] In my case, the plugin wants to provide Hadoop, but it requires Java to do so. And Java is provided by another subordinate, but it's providing that to the plugin, not the principal [17:10] mgz yeah. I used the powershell script from windows_userdata_test.go as my guide. === frankban is now known as frankban|afk [17:11] cory_fu: what your operating is probably doing in practice is saying if there's a new service named the same as your existing subordinate, then deploy the sub-subordinate along with it [17:11] which isn't what you mean at all [17:11] -ing +ion [17:12] natefinch: where is your fix for destroy environment landed? [17:12] katco: no where, because 1.25 us blocked, and this was not a blocking bug [17:13] natefinch: k [17:13] katco: here's the PR: https://github.com/juju/juju/pull/5363 [17:13] Bug #1581138 changed: Juju20: websockets API unhelpful error message when not specifying support facade Version parameter