[00:02] Bug #1519128 opened: lxd provider fails to cleanup on failed bootstrap

[00:21] thumper: mwhudson go 1.5.2 release status https://groups.google.com/d/msg/golang-dev/JcZNxZgRR04/yjDXSO_RAQAJ [00:22] thumper: why does fslock have an IsLocked method ? [00:22] raaaaaaaaaaaaaaaaaaacy [00:23] davechen1y: very very close then [00:23] pyup [00:24] i just saw the ppc32 rel cl land [00:24] looks like austin has got another horrid gc bug [00:24] but i don't think it'll hold up the release [00:24] i asked russ last week if there was value in doing a release candidate for the rc [00:25] but he reminded me that they can always relase go 1.5.3, so there is no value in waiting to get 1.5.2 perfect [00:26] Bug #1519133 opened: cmd/jujud/agent: data race [00:26] Found 7 data race(s) [00:26] FAIL github.com/juju/juju/provider/ec2 531.204s [00:32] thumper: mgz, I should have a PR ready to disable the -race build on the remaining failing packages [00:33] would it be pssible to kick off another race job as soon as I check that in, or should I wait for the overnight run ? [00:56] Bug # opened: 1519141, 1519144, 1519145, 1519147 [00:59] Bug #1519149 opened: worker/uniter/remotestate: data race [01:02] axw: thumper menn0 https://github.com/juju/juju/pull/3806 [01:05] davechen1y: LGTM [01:05] Bug #1519149 changed: worker/uniter/remotestate: data race [01:08] Bug #1519149 opened: worker/uniter/remotestate: data race [01:11] ^ eventual consistency eh mup ? [01:17] Bug #1519149 changed: worker/uniter/remotestate: data race [01:26] Bug #1519149 opened: worker/uniter/remotestate: data race [01:41] oh shit [01:42] ? [01:42] wallyworld: is there any way to kick off http://reports.vapour.ws/releases/3346/job/run-unit-tests-race/attempt/597 [01:42] this job immediately ? [01:43] davechen1y: not sure, i'll see if i can [01:45] thanks [01:45] i'd like to get started on restoring the tests i just commented out [01:46] davechen1y: it needs a revision_build param, i'm not sure if that is a git sha or something else [01:46] i can look at previous logs [01:46] unless sinzui is still around? [01:47] davechen1y: to re-run that same attempt? [01:49] sinzui: no, a new attempt please, at 7652514bc7127bf9e1c283479b32733933708da7 [01:50] davechen1y: which branch is that. That will trigger a full round of tests [01:50] oh [01:51] davechen1y: CI got clobbered again. YES I will make CI pickup master tip now [01:52] ericsnow: I forced the ubuntu alias to point to the wily lxd image. We have a pass. I will update the bug with what I did to make lxd and Juju happy http://juju-ci.vapour.ws:8080/view/Juju%20Revisions/job/lxd-deploy-wily-amd64/51/console [01:54] sinzui: yes, that's on master now [01:54] thanks [01:55] davechen1y: your job will start in about 7 minutes. [01:55] I'm finished being mean to anastasiamac, your turn now [01:56] wallyworld: ^^ [01:56] axw: \o/ [01:56] oh, alright [01:56] axw: ur patience is phenominal and greatly appreciated!!! [02:00] sinzui: tahnks [02:02] axw: wallyworld: m hitting "land-me-quickly",... all bruised and battered \o/ [02:02] master is blocked [02:03] just joking!! :-D [02:03] wallyworld: it's k. m on feature branch :D [02:03] wallyworld: which somehow feels even more painful \o/ [02:18] axw: got a second? [02:18] natefinch: heya, what's up? [02:18] axw: I was looking at bug 1517344, but not sure what the actual repro steps are. [02:18] Bug #1517344: state: initially assigned units don't get storage attachments

[02:19] * thumper pulls a sadface [02:19] investigating one bug, and found at least four others [02:19] natefinch: you'll need a charm that requires storage. just deploy it, and the storage doesn't get added anymore. I'll push my testing charm to LP [02:20] sinzui: if you see the race job pass, please make it voting :) [02:20] axw: I saw the storage docs mention cs:~axwalk/postgresql ... is that still a valid test case? [02:20] natefinch: I think so, but I haven't tried using it in a while [02:20] * natefinch tries [02:20] thumper: that job cannot be voting for 1.25 though. That will be tricky [02:21] sinzui: no, just master [02:21] sinzui: and the upcoming 1.26 [02:21] thumper: yeah, we don't have branch level voting. it will be tricky [02:21] natefinch: otherwise there's cs:~axwalk/trusty/storagetest [02:21] sinzui: oh? [02:21] axw: cool, thanks [02:21] * thumper is surprised [02:22] natefinch: just "juju deploy cs:~axwalk/trusty/storagetest" *should* give you a unit of that service with a single "filesystem" storage instance [02:22] how do you go about adding ci tests for new branches that don't work on old branches? [02:22] sinzui: ^^ [02:22] natefinch: I think if you deploy the service and then add the unit (separate step), the second unit will get storage [02:23] thumper: the test exit early with pass, I can do that for 1.25 and older, but we also not run the test. [02:23] sinzui: can I watch the progress of the job ? [02:23] davechen1y: yes [02:23] sinzui: that is probably fine [02:23] sinzui: could you tell me the link please [02:24] davechen1y: http://juju-ci.vapour.ws:8080/job/run-unit-tests-race/599/console [02:24] ahh, it's a jenkins job [02:24] that's what I needed to know [02:24] now I can solve my own question [02:24] thumper: Yeah, I thiknk so. We weren't learning anything from the 1.25 runs [02:24] poop [02:24] 404 [02:25] and an empty stack trace for good measure [02:25] davechen1y: you have to be logged in [02:25] how do I log in ? [02:25] i've never logged in [02:25] can the job be changed to public [02:25] i don't think itneeds to be private [02:32] lol "WARNING: config not found or invalid" wait... what? Which is it? [02:33] q [02:35] natefinch: try double negative [02:35] config not not found or not invalid == success! [02:41] I just ... how do you not know if it doesn't exist? Why does the code not differentiate? [02:43] also, juju thinks some of the random juju-* processes in my bin directory are juju plugins and tries to run them when I use tab complete and ends up panicking because that's not what they are [02:44] wallyworld: left some comments on your PR [02:44] s/processes/executables [02:44] axw: ty [02:44] ok github.com/juju/juju/api/uniter1532.421s [02:44] yay cloud [02:44] wow [02:45] davechen1y: the CI job takes >60s for the joyent tests, takes ~2s on my desktop :/ [02:45] yay cloud [02:46] the raison d'etre of false economies [02:48] if only we had some way to run a cloud on top of our own fast bare metal.... [02:48] (not that we don't need to run cloud specific tests, obv...) [02:50] natefinch: you mean like a laptop [02:50] with ubuntu installed on it ? [02:51] davechen1y: I know they're hard to find in this company, but maybe we could scrounge up a few [03:00] thumper: whelp. One more thing to add to my airing of grievances against YAML this year. [03:00] \o/ [03:05] YAML, the configuration format so flexible, no matter what the issue -- it's your fault. [03:05] lol [03:07] davechen1y: https://bugs.launchpad.net/juju-core/+bug/1517632 [03:07] Bug #1517632: juju upgrade-juju after upload-tools fails [03:07] davechen1y: hangout to chat about it? [03:07] thumper: i'll me you in the 1:1 [03:07] k [03:09] axw: replied. since there's doubt, i can ask for clarification from rick etc [03:09] my favorite is still base 60 notation... so a value of 8080:22 is interpreted as 484822.0 but a value of 8080:61 is interpreted as a string "8080:61" [03:14] wallyworld: yes, please [03:14] ok [03:15] Bug #1519176 opened: apiserver/provisioner: tests unreliable under -race [03:18] Bug #1519176 changed: apiserver/provisioner: tests unreliable under -race [03:21] Bug #1519176 opened: apiserver/provisioner: tests unreliable under -race [03:27] Bug #1519176 changed: apiserver/provisioner: tests unreliable under -race [03:30] Bug #1519176 opened: apiserver/provisioner: tests unreliable under -race [03:42] FAILgithub.com/juju/juju/featuretests1411.997s [03:43] axw: i was wrong in my reply - deploy --series wily will cause subsequent add unit commands to use wily [03:43] wallyworld: with your branch? [03:43] wallyworld: or that's the expected outcome? [03:43] axw: with my branch and master [03:43] that's te current behavour [03:44] but we still need --force to get such a unit on a non wily machine each time [03:44] that's what i was trying to say [03:44] but got confused [03:44] wallyworld: ok, so can you just record the Force flag on the service then? [03:44] that's what i don't want to do [03:44] juju add-unit will still used wily without --force [03:45] but it will use a new wily machine [03:45] Bug #1519183 opened: featuretests: tests fail under -race because of crappy timing issues [03:45] --force is needed if we want to use a non wily clean/empty machne [03:45] wallyworld: non capisco [03:45] quick hangout? [03:45] ok [03:45] stndup [03:46] axw: am in standup hangout fwiw [04:03] Bug #1519189 opened: worker/leadership: FAIL: TrackerSuite.TestGainLeadership [04:08] davechen1y I see you've inherited bug 1517632. Congrats :) [04:08] Bug #1517632: juju upgrade-juju after upload-tools fails [04:08] davechen1y: could you put in an update before you EOD? [04:08] davechen1y: I'll need to follow up with bootstack in the morning [04:11] cherylj: i'm working on something else today [04:11] i have no update [04:11] feel free to unasign me [04:11] (thumper only gave me the bug 20 minutes ago) [04:11] Bug #20: Sort translatable packages by popcon popularity and nearness to completion

[04:11] ha, funny mup [04:11] cherylj: short answer is nothing on this one so far [04:12] Bug #1519189 changed: worker/leadership: FAIL: TrackerSuite.TestGainLeadership [04:12] wallyworld: did you say you had an environment where you can reproduce bug 1517632? [04:12] Bug #1517632: juju upgrade-juju after upload-tools fails [04:13] cherylj: one sec [04:15] Bug #1519189 opened: worker/leadership: FAIL: TrackerSuite.TestGainLeadership [04:15] Bug #1519190 opened: worker/addresser: FAIL: worker_test.go:260: workerEnabledSuite.TestWorkerAcceptsBrokenRelease [04:15] Bug #1519191 opened: worker/addresser: FAIL: worker_test.go:260: workerEnabledSuite.TestWorkerAcceptsBrokenRelease [04:16] cherylj: i think i saw it once on a stock 1.25 deployment i ran to specifically test the issue but thay env is long gone [04:17] but in general, i was not able to repro again after that [04:17] but i am pretty sure wayne said he could repro on 1.25 [04:18] wallyworld: okay, thanks. [04:18] sorry :-( [04:19] maybe 1.25 is ok after all [04:19] and it nly affects 1.22 [04:25] wallyworld: forgot to ask: any issues with adding URL to RemoteService? [04:26] wallyworld: then it'll be possible to have a different service name locally than what's in the service directory [04:27] axw: i don't think so offhand. i havn't got that branch open but i would have thought wed be recording ServiceURL in the remote service data model. we are not? [04:27] wallyworld: nope, just name, endpoints, life, relation-count [04:28] axw: hmmm, i'm sure i meant to add it. sigh [04:28] wallyworld: I'll add it in my branch [04:28] ty [04:32] wallyworld: in another upgrade bug.... Have you ever seen this message" "upgrader.go:185 desired version is 1.24.7, but current version is 1.23.3 and agent is not a manager node" [04:32] cherylj: wow, no :-( [04:32] I think the state server doesn't think it's the state server [04:32] yeah, that's really sucky [04:33] think there's any chance of recovery around that? [04:33] Bug #1498968 changed: ERROR environment destruction failed: destroying storage: listing volumes: An internal error has occurred (InternalError)

[04:33] cherylj: looks like something hasn't replicated [04:33] cherylj: we'd need to see juju status to see what version each agent is [04:34] to et an idea of where to start [04:34] I don't think any have upgraded. [04:36] cherylj: ah right ok. hmmm, do retrying the upgrade work? [04:37] wallyworld: no :( [04:37] we'd need logs etc then sadly :-( [04:37] and juju status [04:37] I have machine-0 log. Should I also ask for syslog? [04:38] would be good to see allmacines.log [04:38] or at least logs from all the HA state servers [04:38] and syslog for mongo won't hurt [04:38] here's the worrisome part, though. They went from 1.20 -> 1.23.3->1.24.7. And I know 1.23 had issues. [04:38] and possibly --debug from client [04:38] wondering if that had something to do with it [04:38] if they got off 1.23 that's a good sign [04:39] they didn't. It's the step to 1.24.7 that hit this problem [04:39] 1.23 tended to hang and not be able to be upgraded [04:39] Bug #1498968 opened: ERROR environment destruction failed: destroying storage: listing volumes: An internal error has occurred (InternalError)

[04:39] well... [04:39] cherylj: the symptom i was aware of was that agents got dealocked, but this seems different [04:40] if you use juju ssh to log into an lxd machine, then go 'less /var/log/juju/machine-3.log' you might not get things in the right order [04:40] NFI why [04:40] but if you copy the file to the host, then look, the file is all good [04:40] wallyworld: yeah... [04:41] cherylj: it may well be a related problem and the scripts meno did may help, but because that error message i have not seen before, can't be sure [04:41] hence logs etc [04:41] okay, thanks wallyworld [04:41] thumper: lxd hates you. they put in that easter egg just for you and now you'v found it. well done [04:42] \o/ [04:42] * thumper goes to look for wine [04:45] Bug #1498968 changed: ERROR environment destruction failed: destroying storage: listing volumes: An internal error has occurred (InternalError)

[04:46] sinzui: can you kick the job off again with 8b4e8b7d037c52c9a0df00d8227366033eea04d9 [04:46] i tried to do it myself but http://juju-ci.vapour.ws:8080/job/run-unit-tests-race/600/console [04:47] davechen1y: CI is still testing the last revision. Ci Will automaticlly start the next master or 1.25 revision [04:48] ok, thanks [04:49] davechen1y: I think the next round of testing will be in 15 minutes. master tip will be selected [04:55] sinzui: thanks, I think I have now excluded all the troublesome packages [04:56] davechen1y: you rock [05:17] davechen1y: I am off to sleep: Your job is running. http://juju-ci.vapour.ws:8080/view/Juju%20Revisions/job/run-unit-tests-race/601/console [05:23] wallyworld: I thought you were going to undo all the state changes? aren't they all unnecessary for this branch, since the series is just encoded in the charm URL? [05:24] wallyworld: i.e. the only place we check series is when resolving the URL [05:24] axw: i left in the ones needed for unit placement [05:24] ok [05:24] sorry, but i did remove the clean policy ones [05:32] axw: ahh, I see the problem with storage at service creation time. Service.unitStorageOps tries to read the database to get the service constraints, but obviously they haven't been written yet. [05:35] natefinch: yep, that'll be part of it. assignToNewMachine isn't even calling that though? [05:36] axw: it's during unit creation that I see the difference, not assignment [05:37] natefinch: ok. so the unit's not getting storage associated, and then of course when you go to assign it's got no storage so doesn't create attachments.. makes sense [05:37] correct [05:38] easy enough to do what I did with the rest of the stuff, which is factor out how the constraints are discovered and just pass them in [05:41] yay, unused variable compiler error saves me again [05:43] and fixed. [05:44] natefinch: excellent, thank you [05:44] wallyworld: you're going to be adding support for add-unit --series right? [05:45] wallyworld: or is it just add-unit --to --force [05:45] axw: not in very next branch. am wondering if i should just hold off on that [05:45] thinking about just "juju add-unit --to 1 --force" [05:45] but --to implies force i guess [05:45] as we discussed [05:45] wallyworld: right, so I'm wondering if propagating Force is required at all [05:46] wallyworld: if you specify --series, you'll need to record that on the unit doc [05:46] hmm... I'd prefer a force there.. to keep me from screwing myself if I typo or just forget that a machine is the wrong series [05:46] i am still vascilating on whether we want --force [05:46] or just an "are you sure?" or something [05:46] natefinch: --to already overrides everything else [05:47] as in, your constraints may be ignored if you use --to (e.g. in MAAS if you specify --to ) [05:47] yeah, but almost everything else is a nice to have for performance etc. ... series is like "this stuff may totally not function" [05:47] axw: actually, i forgot to mention - --to does not override series mismatch [05:47] currently [05:48] see assignToMachineOps [05:48] so i think that's what convices me we want --force [05:48] wallyworld: right, so the question is do we change --to from meaning "do what I say" to "do what I say (except if it's series, in which case I have to force you to do what I say)" [05:49] so changing semantics for a non 2.0 release would not be good [05:49] i think being conservative here is good and then we get feedback from feature buddies [05:50] that's IMHO [05:50] i can see both sides [05:50] hopefully it'll generally be a non-issue as charms become more multi-series compatible [05:50] you're changing semantics one way or the other [05:50] we aren't changing default --to behaviour [05:50] just adding a new option [05:51] wallyworld: no, but you are changing what --to means [05:51] really? [05:51] axw: --to can't override a charm's series now [05:51] due to an existing limitation, yes [05:51] i'm not sure we are changing what --to means. --to means "do not choose a machine for me or create one, use this one" [05:52] right. but why do I have to --force for series and not anything else? [05:52] partly because that's the current semantics, and also --to now overrides placement, not compatability per se [05:52] seems a bit arbitrary. I may have a charm that requires 64GiB RAM, but I've said to deploy to punydevice and it'll happily do that [05:53] overriding series is more of a compatability issue potentially [05:53] i can se the point about memory [05:53] you can't use --to to put cs:trusty/mysql on a vivid machine in 1.25, can you? [05:53] no [05:54] then having it also fail with series in metadata seems completely consistent [05:54] adding a flag that changes the behavior is certainly not breaking backwards compatibility of the CLI [05:54] that's my argument also [05:54] but i can also see the other side [05:55] but for me, the backwards compatability argument wins [05:55] --to may be inconsistent with respect to constraints, but that's the way it has been. I don't think changing it at this point is a good idea. [05:56] backwards compatibility matters when you're changing something that was possible that something that is not possible. we're doing the opposite. [05:56] anyway, past bedtime for me [05:56] * axw continues review anyway [05:56] natefinch: good night [05:57] wallyworld: what I'm getting at is: why would anyone care if they previously couldn't go "--to " and now they can? [05:57] people may have script that check errors etc [05:58] or expect errors [05:58] wallyworld: I cannot fathom why anyone would script that, except in CI to test for failures [05:59] me either, but then again as we see every day, customers do weird shit with juju [05:59] wallyworld: https://xkcd.com/1172/ [06:00] very relevant [06:00] ok, i'll change it [06:01] WINNER [06:01] I call that argument "appeal to XKCD" [06:01] not sure i fully agree, but we can iterate [06:01] i can honestly see both sides [06:02] wallyworld: yes, this is my opinion of course. I think you should at least bring it up with fwereade [06:02] my not-very-humble opinion [06:02] after i land the branch so that it will be too late [06:02] heh :) up to you [06:03] it was 2v1, maybe he will make it 2v2 [06:12] wallyworld: oh shit. looks like azure-sdk-for-go is using a too-new version of x/crypto :/ [06:12] \o/ [06:12] wallyworld: can't build on 1.2. are we updating soon? [06:12] sigh [07:28] axw: so, this test TestDeployBundleInvalidSeries now fails with --to now not complaining about series mismatch. I think it's valid to accept the bundle case as highlighted in the test as something that should now work [07:28] agree? === urulama__ is now known as urulama [07:40] mgz: the -race build finally passed !! [08:48] dooferlad, do you have a moment to look at http://paste.ubuntu.com/13479943/ please? [09:08] fwereade: on it [09:09] wallyworld: sorry, I missed your message. I think so, yes [09:09] axw: np, i've pushed changes [09:24] fwereade: please review: http://reviews.vapour.ws/r/3221/ [10:02] jam, fwereade, voidspace, standup? [10:04] dimitern: omw [10:05] dimitern: gah, 2fa dance [10:20] dooferlad: http://pastebin.ubuntu.com/13491666/ [10:21] dooferlad: fails with http://pastebin.ubuntu.com/13491674/ [10:40] dooferlad: from that output the space name seems to be being serialised as "string" not "space" [10:41] next issue, unreserved ranges is a map not an array [10:41] just testing with real maas to see if that's my bug - probably is :-) [10:43] voidspace: looking [10:48] dooferlad: hmmm... unreserved-ip-ranges should return an array [10:48] dooferlad: looks like it's returning a map [10:49] dooferlad: I haven't looked at the tests for unreserved ranges, just observing the error in my code (requested array got map) [10:50] dooferlad: although your test is deserialising into an array [10:51] dooferlad: I'll look again, it's possible there's a bug that only sends a map when the array is empty (I didn't populate the ranges first - haven't got that far in my test) [10:51] voidspace: OK [10:53] hmmm... my code looks good [10:54] unless it should be "unreserved_ip_ranges" [10:54] checking the maas docs :-) [10:54]