[01:26] axw: I commented through the resources spec. Please let me know if anything I say makes little to no sense or a chat through would help make things clear/etc. [01:27] axw: well, not the spec I guess, but the use case/design doc which had a lot more interesting details [01:27] rick_h_: ok, wallyworld has been doing that mostly, but I can take a look if you like [01:27] axw: ah np, I was going to ping you both but he's afk from irc atm [01:28] axw: if he comments just let him know I'm happy to be schooled in person/etc. [01:28] rick_h_: heh :) no worries [02:41] wallyworld: damn hangouts [02:41] I could hear you, but you weren't moving [02:41] yeah, tis ok. i'll reply [02:41] and stuff I typed into the chat window didn't come through [02:41] cheers [02:50] axw, wallyworld: i just noticed that our tests prefer the system installed mongod over the Juju one. is that intentional? [02:50] menn0: hrm, I didn't think so. I thought we were meant to use juju-db if it existed, otherwise whatever's available [02:51] um, don't think so [02:51] axw, wallyworld: getMongod (in juju/testing) tries $PATH first so that one wins if you have the system mongodb package installed [02:52] menn0: oh sorry, the *tests*. I think the reason for that is so we get the javascript-enabled version if it's there [02:52] menn0: IIRC juju-db doesn't have javascript on some arches [02:52] possibly all [02:54] axw: why do we need a JS enabled monogdb? [02:54] menn0: for charmstore I think [02:54] axw: ah ok [02:54] axw: it's just unfortunate that we can end up running tests with the mongodb version that isn't actually used in production [02:55] axw: it probably doesn't matter most of the time [02:55] menn0: true. perhaps it should be juju-db by default, and configurable (for charmstore) [02:55] axw: except if we unwittingly take advantage of some newer feature of mongodb [03:22] davecheney: o/ [03:23] davecheney: what were the problems with importing "time"? [03:24] anastasiamac: it is a symptom if a flakey test [03:24] kicking off an actoin then waiting for some time to pass [03:24] is likely to be flakey [03:25] so it's only a "symptom" if imported in tests? [03:25] it's k to import into actual code.. [03:25] davecheney: ? [03:27] yes [03:27] but imo test files that import "time" tend to be flakey [03:45] davecheney: :D ty for ur passion! (and answers) [03:55] no [03:55] np [03:56] i say they are flakey because there is in herent uncertainty [03:56] the test is usually saing somthing like "I'm going to hope that X happens in under NNN milliseconds, or Y shouldnt take more than NNN milliseconds to happen" [03:56] and if uou look at our build bot [03:56] neither of those tend to be true [04:01] :) [04:03] I am going to add a "r u absolutely sure?!" to a review I am doing based on ur opinion. I would love for us to move away from using actual "time" in tests... whether it will ever happen is a separate question... [04:04] * davecheney bursts into tears of applause [04:06] * anastasiamac can't believe she made Dave cry.. of joy nonetheless.. [04:06] :D [04:07] it doesn't take much these days [04:07] http://gophercon.com/ [04:07] note the helpful countdown, telling me how much work I haven't done [04:08] davecheney: it's impressive! well done :D m not going but it's looking to be really exciting!! [05:21] can i trouble someone for an easy review? http://reviews.vapour.ws/r/1940/ [05:59] menn0: looking ... [05:59] anastasiamac: thanks [06:01] menn0: rhetorical question :D... [06:02] so the deque with max length as 0 will behave as before? [06:02] dropping from the other side? [06:03] menn0: neevr mind - re-read and re-understood :D [06:03] anastasiamac: with a max len of 0, there is no maximum length (as before) [06:03] ok :) [06:09] anastasiamac: thanks for the review! [06:24] axw, anastasiamac: thanks for the reviews [06:24] anastasiamac, about the exported test suites [06:24] fwereade: nps [06:25] anastasiamac, do you know why people *avoid* exporting test suites? [06:25] anastasiamac, I've seen people do it [06:25] anastasiamac, and I've always thought it's a bit weird, but AFAICS it makes no practical difference at all [06:26] anastasiamac, so I always chalked it up to matter-of-taste-don't-worry-about-it [06:26] fwereade: well, I am kind of on the fence of this one... [06:26] fwereade: i just don not see the point of exporting suite [06:27] fwereade: and all suites that i have "accidentally" written with Capitals, I was told to change [06:27] fwereade: :D [06:27] fwereade: it doesn't really make a difference. I would argue if it's an internal test package (i.e. not _test), then not exporting keeps the exports clean for the _test package (not the case here) [06:27] otherwise matter of taste [06:28] anastasiamac, (fwiw, exporting "feels" right to me because the tests *are* the reason for the package to exist -- and I know it *really* works by passing it as an interface{} into gc.Suite, but the semantic payload of "this component is important and part of the public expression of the package" feel relevant) [06:28] fwereade: from practical perspective, I use autocompletion a lot [06:28] fwereade: and exported tests are visible [06:29] fwereade: they tempt me to re-use them [06:29] fwereade: I would rather they were not exported if noone else is meantt o use them :D [06:29] anastasiamac, when do you import a foo_test package? I thought you couldn't [06:30] anastasiamac, I submit that your autocomplete is buggy if it's pulling in stuff exported from _test packages ;p [06:30] fwereade: i do not import... I auto-complete.. which tries to import if I select something [06:30] fwereade: possibly... but it's not a deal breaker either way... [06:31] fwereade: i guess i wonder why export if it's not re-used? [06:31] anastasiamac, semantic payload [06:32] fwereade: so mayb we should have a consensus - export all suites :D [06:32] anastasiamac, possibly a bit of habit from python? I wouldn't keep a test private because I'd expect something like nose to find it for me, and my views on how tests shoudl be expressed are probably coloured by that [06:33] fwereade: m happy to let it go.. altho "semantics" does not really give an argument a tangible benefit IMHO :D [06:34] anastasiamac, heh, I feel it's terribly important [06:34] fwereade: :D if u feel "terrible" importance, I am happy to go with ur gut feeling [06:34] anastasiamac, the set of associations triggered in the developer's mind have a massive effect on how they use the code, and what expectations they have of it [06:35] and take it up next sprint :D [06:35] anastasiamac, the trouble is that it's hard to measure those effects objectively [06:35] anastasiamac, and even if you could they'd diiffer across sets of developers [06:36] anastasiamac, (sorry I'm not saying that test-exports are terribly important -- but I think the semantic associations of the names we pick *are*) [06:36] fwereade: mayb because I do not have python goggles on, I m a bit reserved about exporting un-reusable bits of code (or tests for that matter) [06:37] fwereade: semantics assosciations of names r important... [06:38] fwereade: exporting or not exporting test suites is an approach (pattern?) not a naming semantics... no? ;)) [06:38] anastasiamac, (...and I guess the extension of that is that exporting a type itself has semantic associations; and in my mind the "these are the parts someone using this package cares about" associations are stronger, in my own mind, than the "someone will actually want to import this type" ones) [06:38] anastasiamac, that's only true because you can't import test packages though [06:39] fwereade: :D [06:39] anastasiamac, if you could then the choice would not be completely academic [06:40] fwereade: i am happy for an executive decision here :D what would u like to see as a convention - export or not export? to be or not to be? [06:41] * anastasiamac @ school run [06:41] anastasiamac, my personal preference would be to export, but I also feel like tastes can legitimately differ, which is why I never made a fuss when I saw people starting to not-export-by-habit [06:44] anastasiamac, if we can predict harm caused by unpredictably-exported tests, yeah, we should pick a convention, but I can't really see anyone being inconvenienced either way? [07:10] fwereade: tyvm for an amazing discussion point :D it's always re-freshing to hear ur thoughts and read ur reasoning :)) [07:20] anastasiamac, thank you, always a pleasure :) [07:45] dimitern: ping [07:45] TheMue, pong [07:46] anastasiamac, what'd be the advantage of enums for fieldnames? [07:46] dimitern: one question regarding API errors. they have a message and a code. if an operation on state returns with an error it's clear to put it, sometimes with additional information, into the message. but do we have a common strategy which codes to use? e.g. when ipaddress.Remove() fails. [07:47] oops brb [07:47] TheMue, yes, use common.ServerError() to wrap an error as *params.Error [07:48] TheMue, btw I couldn't see an update in any of your branches [07:48] dimitern: aaaah, the missing puzzle piece [07:48] dimitern: yeah, just wanted to finish the remove method [07:49] TheMue, ok, cool [07:50] dimitern: feeling better with the current state of the code, even if I'm still not happy with our API segmentation :D [07:52] TheMue, well, one step at a time :) [07:52] dimitern: hehe, yes [07:58] wrapping an error into an error result into error results, seems to be a very brittle piece of data [08:37] ah, when wanting to use a mock state from inside a mock entity, this field should not be nil *facepalm* [08:46] axw, anastasiamac: have several responses on http://reviews.vapour.ws/r/1934/ ; thoughts appreciated [08:46] fwereade: thanks, responding now. [08:50] fwereade: BTW, it makes it much easier to follow up if you reply on the review page, rather than in the diff. your responses came up as new issues [09:01] voidspace, fwereade, standup? [09:01] dimitern: omw [09:15] axw, yeah, sorry about that, will try to remember -- I tend to click through for context and forget to go back [09:15] dimitern, oops omw [09:16] fwereade: yeah, it's kind of an annoying interface when replying to comments. [09:18] fwereade: responded. thanks, my primary issues were just misunderstandings. LGTM, thanks! [09:19] axw, awesome [09:19] axw, it's not ready yet though. not enough tests :) [09:19] fwereade: I know, I just meant the logic itself [09:19] and interface [09:19] axw, perfect, tyvm [09:39] voidspace: 7m30s in a trusty vm on my laptop with an SSD, but modest CPU [09:45] axw, hey, one thought: I'm becoming convinced that info,EarliestExpiry is an internal implementation detail, and exposing it can only encourage people to DTWT [09:45] axw, opinion? [09:45] axw, we still need it internally ofc [09:45] fwereade: I think I agree, let me take another look [09:46] axw, and then Info is just Holder;Expiry;AssertOp [09:46] fwereade: yes, SGTM [09:46] axw, cool, thanks [10:00] dooferlad, as you drop that test about allowing loop mounts not returning an error, you might close bug 1465404 as a side-effect [10:00] Bug #1465404: worker/provisioner: fail lxcBrokerSuite.TestStartInstanceLoopMountsDisallowed [10:01] dimitern: thanks, will do. [10:01] * dimitern ffs.. it takes more than 30m to run all tests on my machine, even with gt [10:02] dimitern: gt will only help once you have run once. It should now run in about a second [10:02] dimitern: until you change something that is, but it is still much faster not re-running tests on code that hasn't changed. [10:02] dooferlad, I can't wait for it to complete so I can see if there will be an improvement the second run [10:04] dimitern: :-) [10:05] dimitern: guess a new machine is needed! [10:05] dimitern: not that I have a problem and buy new computers way too often... [10:06] dooferlad, it's high time for a new one; I might not even wait until october for my laptop refresh :) [10:07] dimitern: well, the laptop refresh is just money right? If you can afford to spend now then you still recover the money in october. [10:08] dooferlad, yeah, of course - it's now *yet* bugging me that much, but it's close [10:09] wow! [10:10] it does indeed run in a couple of seconds after the first run - and prints out nice cached panics as well :) [10:12] dimitern: so, perhaps gt needs a cloud option - share the cached results for a particular file hash... [10:12] dimitern: should be really easy to add... [10:12] dooferlad, that'll make it reusable a lot easier among teams [10:13] dimitern: exactly. All of juju (or any project) could basically share CPU time across all their machines [10:14] dimitern: new friday labs project then? [10:14] dooferlad, sounds like it, yeah - I was just about to say the same :) [10:16] dimitern: https://github.com/rsc/gt/blob/master/main.go already uses sha1 to hash files and packages, so just storing them somewhere that isn't ~/.cache/go-test-cache is all you need [10:16] dimitern: ok, that looks just too simple not to do [10:20] dooferlad, :) [10:21] dooferlad, but if it takes more time, please put it on hold [10:26] dimitern: oh, I am not doing it *right now* [10:26] dimitern: I have some self control! [10:28] dooferlad, I know it's tempting :) [10:57] jam: just seen your message [10:57] jam: I still don't know the full runtime on this desktop - it timed out after 40 minutes so trying with an even more ridiculous timeout (2 hours) [10:57] jam: there's something up on this machine [10:58] voidspace: that sounds more like just a simple deadlock. If you do "go test -timeout" won't it print a panic for you? [10:58] jam: well I know *one* test takes 9 minutes so it may not be a deadlock [10:58] jam: (one specific test) [10:59] jam: yeah, I get the panic - not dug into it [11:44] dimitern: hi [11:49] wallyworld, hey [11:50] dimitern: i had on my todo list today bug 1464616 for 1.24.1 but i've only just finished some work in progress. maybe one of your guys could take a look and i'll fix it tomorrow if needed if you don't get to it? [11:50] Bug #1464616: destroy-machine --force no longer forceably destroys machine

[11:52] wallyworld, Iooking [11:53] looking even [11:53] ty, see how you go, i'll fix tommorrow if needed [11:53] wallyworld, cheers, have a good one [11:53] fwereade: re enums :D [11:54] jam: less than an hour is the answer... 2990.665s :-) definitely something up there. [11:54] fwereade: my personal guiding force is 1. type protection vs magic strings/numbers; 2. grouping of related constants [11:56] fwereade: hey, could you take a look at perrito666's latest uniter hook work to see if you're happy for the pr to land? [11:56] wallyworld, sorry, will do [11:56] ty [11:57] fwereade: of course, since most of ur consts are unexported and internal to the package, u could say that putting enums might be a bit on the paranoid side... [11:58] anastasiamac, sorry phone [11:59] fwereade: nps :) can be another of these topics we discuss informally over drinks ;D [12:00] anastasiamac, I never felt that golang really helped that much there -- and when they're all internal you tend to just trust yourself to use the symbolic constants and not the magic, rather than adding another whole layer of types and validation methods and so forth [12:02] fwereade: i agree - internally it may not be as useful (unless u forsee more values coming in the future under the same "group", or the need to be exported, etc) [12:02] fwereade: feel free to drop the issue :) [12:02] anastasiamac, yeah -- I'm hoping those will stay static and internal and forgottenn for many years ;p [12:03] fwereade: (and just for the record, no I don't trust myself as much u trust urself or I trust u).. [12:12] voidspace: your machine got scared when 50 minutes rolled around :) [12:12] makes me wonder if something like shortWait got set to 50s instead of 50ms or something. [12:12] jam: no-one else is seeing the same thing [12:13] jam: I have this on trunk [12:13] voidspace: you're sure you don't have any local changes, right? [12:13] jam: yep [12:13] jam: I'll dig in and see where the slowdown is actually occurring and see if I can work out why [12:13] voidspace: so the only thing *I* would get out of the panic is what test was running, you could use that to see what test is slow [12:14] I generally find golangs 200 tracebacks hard to debug since we use so many goroutines [12:14] jam: I know at least one test that takes 9 minutes [12:14] jam: yep :-/ [12:14] jam: so I have somewhere to start === anthonyf is now known as Guest89967 [13:28] internet over 3g here feels a lot like dial up, I am suddenly invaded by nostalgia... and lag, lots of lag [14:01] wwitzel3: standup [14:02] katco: yes [14:34] Bug #1465694 opened: TestDiesOnFatalError fails

[14:34] Bug #1465695 opened: StorageAddSuite setup fails

[14:34] first internet goes off and now the lights... I feel like in the prelude of a b movie about catastrophes [14:46] "the first reports came from south america" [14:48] katco: well I had cellphone signal and 3g inside my house, that was quite apocalyptic [14:48] maybe the usual electrical interference was gone [14:48] fantastic, the power outage reset the ISP box for the block and I am back online [14:48] :) [14:48] what's the environment variable for setting logging level during tests (alongside -check.vv) ? [14:49] katco: also I live in a suburb mostly populated by families or working age people, I am most likely the only person in 100m around me [15:30] dimitern: ping [15:32] voidspace, pong [15:36] gsamfira: are you free to talk windows maas images in 10 mins maybe? [15:36] mgz: sure [15:37] gsamfira: ace, will invite you in a min [15:42] dimitern: how do I enable trace logging for a test run? [15:46] voidspace, TEST_LOGGING_CONFIG='=TRACE' go test ... [15:52] mgz, ping [15:52] dimitern: hey [15:52] mgz, have a look at this http://paste.ubuntu.com/11725493/ [15:53] mgz, I believe that test should be dropped entirely [15:53] mgz, as it will only work as expected when type==manual and AWS_ACCESS_KEY is in os.environ [15:54] btw all, we have a customer on #juju who might have hit #1464304 in a production environment [15:54] Bug #1464304: Sending a SIGABRT to jujud process causes jujud to uninstall (wiping /var/lib/juju)

[15:54] and so juju has happily uninstalled itself, and he sorta needs juju on that machine, since there are services running there... [15:54] dimitern: thanks [15:55] natefinch, by "uninstalled" what do you mean? apt? upstart? [15:56] dimitern: if you send sigabrt to jujud, it wipes out /var/lib/juju/ I presume it also stops the service [15:56] this is a "feature" to help manual provider clean up [15:57] dimitern: do you mean drop the test, or drop the wrapper with the special case? [15:58] it's intended to just be an extra safety net after destroy-enviroment [15:58] the test could easily be fixed by overriding the envvar [15:59] perrito666: sounds like the guy mostly needs to recover the apipassword... do you happen to know where we can get that for the machine? [15:59] mgz, well, the code this tests checks requires both AWS_ACCESS_KEY in os.environ and config,type==manual, so unless both are set in that and the previous test, it will fail === kadams54 is now known as kadams54-away [16:00] dimitern: yup, not overriding the env is wrong [16:00] natefinch: context? [16:00] mgz, ok, but if it's overridden the it's almost the same as the previous one (safe for e.g. setting the aws_access_key to '') [16:01] perrito666: sorry... guy on #juju had his jujud uninstall itself... he's mostly recovered by copying files from other units, but the agent keeps failing saying it has a bad API password.... [16:01] dimitern: but that's the intention of the test I believe [16:01] perrito666: wasn't sure if in your work with restore, you might know a way to hack that. [16:01] ie, assert d_j_i is not called if the envvar is not set [16:01] natefinch: well if he is copying the agent.conf from another unit he will have problems because the tag will not be the right one [16:02] perrito666: right. [16:02] natefinch: and even then I am not sure he could recover the old password [16:03] he might need to do some state server side magic [16:03] *nod* he said he's been poking at mongo,, which is scary [16:05] mgz, sorry, I'm apparently confused :) [16:05] mgz, are you saying this test, as written is expected to fail always? [16:06] natefinch: when someone says he has been poking at mongo I just raise my hands like a soccer player after a foul [16:06] dimitern: no, I'm saying it should make sure the envvar is unset, then it would do what it intends [16:06] perrito666: yeah.... [16:06] gsamfira: ready to roll? [16:07] but yeah, we lack a way to recover it is an interesting feature though [16:07] mgz: in about 10-15 minutes. Got pulled into another meeting [16:07] mgz: will ping you soon [16:07] gsamfira: cool, thanks [16:07] juju --Iswearthatisaunitjustreaprovisionthenecessaryfiles [16:09] perrito666: yeah, I mentioned on bug #1464304 that we should probably not use a signal as our way of telling jujud to commit seppuku and instead use some really arcane jujud command like `jujud DIEPLEASEDIE -y --yesIreallyMeanIt --deleteAllMyStuffForReal` [16:09] Bug #1464304: Sending a SIGABRT to jujud process causes jujud to uninstall (wiping /var/lib/juju)

[16:09] natefinch: why on the universe we have a commit seppuku command on the first place? [16:10] perrito666: that's how manual provider tells a unit to clean itself up (since it can't just de-provision the machine). [16:11] and why does it work in any other provider :p [16:11] that was going to be my next point, too [16:11] but it shouldn't even work on the manual provider... it's just too easy to do by accident, regardless of what signal we use [16:11] mgz, you mean instead of 'AWS_ACCESS_KEY' in os.environ; add something like "and os.environ['AWS_ACCESS_KEY'] != '' ? [16:14] dimitern: I mean like, patch(os.environ, {}) [16:15] mgz: patch.dict(os.environ, clear=True) [16:16] or zat. [16:17] mgz, that won't work [16:18] dimitern: I don't see why not? [16:18] natefinch: thanks for the look yesterday. I'll do more work to narrow down the problem. If I get it live, can you hop on and take a look? [16:20] mgz: ready [16:20] gsamfira: jog pasting you hangout link [16:22] mgz, hmm, ok, it works, sorry for the noise :) [16:22] dimitern: :P *hugs* [16:24] mgz, I rather like the dummy stack actually - I'm extending how it's deployed so it can be used concurrently in pairs [16:25] dimitern: neat [16:25] mgz, e.g. deploy one pair sink/source in a couple of lxc containers, then deploy another pair in a kvm and on a machine (using different service names ofc) === natefinch_ is now known as natefinch [16:32] katco: I've been trying to help MrOJ over on #juju but have to run to pick up my daughter at preschool. Handing off to perrito666 for now. But wanted you to know I'd spent some time on it. === kadams54-away is now known as kadams54 === kadams54 is now known as kadams54-away [19:15] ericsnow: you around? [19:15] natefinch: yep [19:15] natefinch: just wrapping up a review on your patch [19:16] ericsnow: cool. We have a user on #juju that evidently had a problem with restoring his state server [19:16] ericsnow: or rather, he restored and then the agents on all his machines killed themselves [19:16] (on all the non-state-servers) [19:17] natefinch: that sounds familiar [19:17] natefinch: perrito666 fixed (?) something like that recently, I believe [19:18] ericsnow: he's on 1.23.3, so pretty recent. [19:18] perrito666: was helping out, but had to go run an errand... unfortunately that was before I thought to ask how the guy got in this state [19:20] natefinch: well, 1.23 is the first version with the new restore implementation [19:21] well at least that [19:22] natefinch: what info do we have? [19:22] he said he had some DNS issues during restore, and restore never finished, so he had to kill it. [19:23] I presume he has fixed the DNS issues and reran restore (verifying now) [19:26] natefinch: which provider? [19:26] ericsnow: maas [19:26] natefinch: lovely [19:26] iknorite? [19:27] he updated his state servers to 1.24 before trying restore after fixing DNS [19:41] I wonder if ^ was the cause of a problem, since now the agents are on the wrong version [19:46] natefinch: I expect it depends on at what point they canceled the restore === anthonyf is now known as Guest41199 [19:47] natefinch: restore really shouldn't impact the agents on other machines (other than to make the API unavailable) [19:48] natefinch: so the fact that those agents died implies to me that something in mongo got corrupted (or there was some API timeout that triggered the grim reaper) [19:49] ericsnow: yeah, it seems like a manifestation of the bug where if the agent can't get to the state server, it kills itself. Except that I thought we'd fixed that. [19:51] natefinch: yeah, that's what I was thinking of [19:51] natefinch: it may be that it's not fixed in the version(s) he is running [19:52] ericsnow: he's on 1.23.3 ... that's the latest release version. If it's fixed anywhere, it should be fixed there. [19:53] natefinch: it seems like we haven't been backporting as many (any?) fixes to 1.23 as we have to 1.24 [19:55] ericsnow: looks like it was fixed in 1.20.1: https://bugs.launchpad.net/juju-core/+bug/1339770 [19:55] Bug #1339770: Machines are killed if mongo fails

[19:57] natefinch: ah, fixed last year so it should be fixed in 1.23 [19:59] thank god juju uninstalling itself doesn't also blow away its logs [20:01] natefinch: so, we have logs for the restore issue? [20:01] natefinch: FYI, https://github.com/juju/juju/commit/b4e37a8d31bedb56ce46318779c04387f8333026 [20:04] natefinch: from what I can tell, the actual fix is on 1 line: https://github.com/juju/juju/commit/b4e37a8d31bedb56ce46318779c04387f8333026#diff-edccfba67a01587c9faca9185781e5dbR289 [20:44] Bug #1465844 opened: juju action-set skips values with underscore in the key [20:56] fwereade_: ping? [21:01] thumper, that is wishful thinking at this hour :) [21:01] alexisb: I know, but sometimes he is around [21:20] thumper, pong [21:20] thumper, how's it going? [21:20] fwereade_: good, [21:20] fwereade_: many of the CI failures seem to be jujud/agent tests getting deadlocked [21:20] I *think* this is lease related... [21:21] you are fixing this yes? [21:21] goddammit that is not at all unlikely [21:21] some of this is due to the agent not stopping [21:21] that is one of the features of the current implementation that I intend to do differently yes [21:21] thumper, I might even have a fix for it inn a rotted branch somewhere that I didn't realise I hadn't finished [21:22] fwereade_: was just poking to see if that could be fixed soonish [21:22] thumper, in the short term the core of it is *similar* to the following (and axw made most recent changes so can verfiy I think) [21:22] ericsnow: you still around? [21:23] thumper, there's an implicit dependency between the api server and the lease manager; and between the lease manager and state [21:23] Bug #1465694 changed: TestDiesOnFatalError fails

[21:23] thumper, if you shut them down in the right order (apiserver; lease manager; state) then the tests should be safe [21:24] thumper, but to do that you need to get all the way down to the dummy provider [21:24] fwereade_: jujud/agent calls a.Stop(nil) [21:24] i think [21:24] thumper, which is what creates the apiserver and state and shuts them down [21:24] thumper, yeah, I can well believe it [21:25] thumper, what I saw was deadlocks on test shutdown [21:25] thumper, but I didn't have a repro that wasn't tied up with other changes [21:26] thumper, specifically what I saw was that the apiserver had started to respond to a leadership request [21:27] thumper, but the lease manager had been shut down already [21:27] thumper, so the api call was blocking forever and preventing the apiserver from completing [21:28] fwereade_: this timeout is effecting everyone [21:28] can we priorities the fixing of it? [21:28] sinzui: have a moment to help me not make a fool of myself? [21:29]