[00:08] axw: do you think you could backport https://github.com/juju/juju/pull/3841 to 1.25 ? [00:08] to see if we can get the xenial tests passing on the 1.25 branch ? [00:26] thumper: http://reviews.vapour.ws/r/3278/ fixed and verified it builds under windows, linux, and darwin [00:41] davecheney: 1.25 doesn't have LXD to begin with [00:41] so must be a different issue [00:47] ok [00:47] thanks [01:07] davecheney: I'm pretty sure that I was trying to get us away from fslocks in the linux code at least [01:07] but then I hit places where people were caring about the message [01:07] and I rage quit [01:07] however [01:07] I think if we had a proper lock that went away with the process [01:08] we could simplify quite a bit of code [01:10] i don't think we want, lock.Lock("message") [01:10] we want lock.TryLock(timeout) [01:15] thumper: i just said to axw that I think at least some cases of fslock / filelock [01:15] we don't want locking [01:15] just atomic replacement [01:15] axw, what were the two case of fslock you knew of ? [01:16] env cache I think (thumper? is that right), and serialising hook execution on a machine [01:16] i.e. so two units (different processes) cannot execute hooks at the same time [01:17] I don't think it makes any sense to have a Lock function that doesn't lock. TryLock makes much more sense [01:17] boom, and all works sopts, https://bugs.launchpad.net/juju-core/+bug/1471941 [01:17] Bug #1471941: windows unit tests fail because handles are not available

[01:17] client side cache, template creation, uniter lock [01:17] thumper: those all sound like atomic replacement [01:17] also I believe something in reboot [01:17] and maybe actions? [01:17] axw: serialised hook execution [01:18] can we not make that ownership of the uniter socket ? [01:18] davecheney: different unit agents on the same machine [01:18] davecheney: only one unit agent should be executing a hook at the same time [01:18] why not just open a known unix socket on localhost [01:18] if the host doens't support unix sockets, tcp will do [01:19] davecheney: most places it is process serialisation, not just atomic replacement [01:19] if hook execution is per machine [01:19] note that juju run goes through the uniter execution code [01:19] although I think that axw [01:19] 's fix [01:19] make it more in sync with a hook type [01:28] thumper: sinzui https://bugs.launchpad.net/juju-core/+bug/1471941 [01:28] Bug #1471941: windows unit tests fail because handles are not available

[01:28] why did this bug suddently become blocking ? [01:29] davecheney: when it suddenlly startd failing reliably a few hour ago [01:30] The bug has a long history, but it recently became a problem in master http://reports.vapour.ws/releases/issue/559acfbe749a565572d289f1 [01:30] WARNING: Error cleaning up temporaries: [01:30] ERORS LOCGED AT WARNIGN!! [01:32] heh [01:32] sinzui: i see the bug [01:32] given it will require a third party to land any fix [01:32] can I implore you to not make it blocking ? [01:34] davecheney: how do we encourage someone to fix it, can we revert? [01:35] davecheney: or is there something I can do to mitigate the issue so that other regressions don't appear in the window's tests? [01:38] sinzui: the problem is whatever is writig that log file hasn't shut down in time [01:39] and windows won't let you unlink open files [01:39] my solution is to fix gocheck to not fail on a warning [01:40] davecheney: could a reboot to recycle space and mem help the log file to shutdown in time? [01:40] * sinzui dreds rebooting every day [01:41] hmm... we could make it easier by making the tests clean up after themselves [01:41] but the way to force this is to do what dave says [01:41] and have gocheck fail if it can't delete the files [01:44] thumper: gocheck fails now [01:44] but it calls it a warning :grrr: [01:56] davecheney: 1.25 looks like it will pass. I can try rebooting the machine hoping to recover something. the machine looks clean. I will then retest the last master rev. [02:09] sinzui: what is the 1.20 version that we tell people to upgrade through? [02:09] thumper: 1.20.14 is final [02:10] sinzui: ta [02:23] davecheney: thumper : the windows unit tests just failed again. Looking at the history of thus job against master, I think something since commit da5ec85 has upset the test suite. [02:31] sinzui: don't suppose you have a link to that commit? [02:31] * thumper looks at the git cli [02:32] sinzui: really? [02:32] that commit doesn't really do much [02:32] Merge pull request #3841 from axw/lp1520380-remove-lxd-containertype [02:32] sinzui: that one right? [02:32] thumper: that commit was the last pass. since then master fails [02:33] oh, something since that [02:33] that makes more sense [02:34] wallyworld: ping? [02:35] hey [02:35] wallyworld: quick question for you on the unit agent / unit workload status [02:35] sure [02:36] was the idea that the old status docs from before the split happened would just become the unit agent status docs? [02:36] and there was no need to update them during an upgrade? [02:36] yes [02:36] waigani_: ^^ [02:36] hence the #charm suffix for worklaod status [02:36] wallyworld: that's what I thought, but wanted to sanity check [02:36] sure [02:36] thanks! [02:37] cherylj, wallyworld: okay cool, will do [02:37] given that, what upgrade step is missing? [02:37] creating the status doc for u##charm [02:38] wallyworld: http://reviews.vapour.ws/r/3279/ [02:38] hmm, i had thought that we treated missing as unknown [02:38] sinzui: i'll take another look [02:38] that's how it was originally done i think [02:38] wallyworld: I can look at the history [02:39] maybe not, can't recal now [02:39] but now it fails with a "not found" [02:39] ok, clearly a problem then [02:39] it's been a while so i could be misremembering [02:39] wallyworld: so, is the solution to treat it as unknown? or insert a doc for each during upgrade? [02:40] I was assuming that it should be inserted as an upgrade step [02:40] a doc is cleaner [02:40] ok [02:40] so that's what I've currently got: a new agent status with status unknown inserted on upgrade [02:41] wallyworld: any chance if you could take a quick look at the upgrade step, see if it makes sense? [02:41] link above [02:41] looking [02:42] cheers [02:43] waigani_: comment doesn't make sense, u#name is not invalid [02:44] wallyworld: but it's invalid for a unit [02:44] not for an agent [02:44] we need both [02:45] wallyworld: that's what this upgrade step does [02:45] wallyworld: do you mean it's just the wording of the comment? [02:45] ok, i'll read the code, the comment was just confusing [02:48] waigani_: comment is plain wrong, u#name is not invalid. units are not misidentified. the issues is that in settings, there is an entry for u#name ony, when there should also be one for u#name#charm [02:49] sinzui: http://reports.vapour.ws/releases/2859/job/run-unit-tests-win2012-amd64/attempt/838 [02:50] this file, base_windows.go doesn't appear to exist on master [02:50] i'm probably looking in the wrong place [02:50] sinzui: nope, not on master [02:50] how come this bug is blocking master ? [02:51] davecheney: because only master is failing because of it [02:51] github.com/juju/juju/testing/base_windows.go [02:51] ^ this file does not exist on windows [02:51] sorry [02:51] in master [02:51] at least that I can see [02:52] is this for another branch ? [02:52] this file exists in juju-1.25.1 [02:52] but not on master [02:53] davecheney: that attempt above is on a feature branch [02:53] the resources one [02:54] sinzui: this doesn't sound like it's on master [02:54] can you please unblock master [02:54] davecheney: this issue is simple. The windows unit tests passed a few commits ago. Now they do not. The job passes on other branches, ergo, something has changed master to make the failure conistent. The issue matches an existing bug seen intermitttently in other brances. We escallated the bug a few hours ago [02:54] waigani_: you need to take another look - it's not correct as is [02:54] sinzui: got a link to a failing run on master? [02:55] sinzui: please can you point me to the jenkins run and i'll look into it [02:55] menn0: I think this may be your one [02:55] it is an upgrade featuretest [02:55] uhhh, thumper, the issue linked int eh bug diescription points to everything, and all the recent failures are in master [02:56] sinzui: i'm not very familar with how to navigate reports.sw [02:56] wallyworld: okay, will do [02:56] menn0: ping [02:56] i'd apprecaite it if ou could provide me with a link to the jenkins failure [02:56] waigani_: cherylj suggested the correct approach [02:56] davecheney: thumper http://reports.vapour.ws/releases/issue/559acfbe749a565572d289f1 from the bug will help. the last six failures are recent and affect master [02:57] thumper: pong [02:57] menn0: I believe the blocker above is due to the re-enabled upgrade feature tests [02:57] menn0: see http://reports.vapour.ws/releases/3376/job/run-unit-tests-win2012-amd64/attempt/1657 [02:57] sinzui: is this the failure you are talking about ? http://juju-ci.vapour.ws:8080/job/run-unit-tests-win2012-amd64/1659/console [02:58] sigh [02:58] * menn0 looks [02:58] davecheney: yes [02:58] thank you for confirming [02:58] ok, this is a different issue [02:58] this is the failure i see [02:58] upgrade_test.go:109: [02:58] go func() { c.Check(a.Run(nil), jc.ErrorIsNil) }() [02:58] ... value *errors.Err = &errors.Err{message:"failed to create C:/Juju/bin/juju-run.exe symlink", cause:(*os.LinkError)(0xc084427f00), previous:(*os.LinkError)(0xc084427f00), file:"github.com/juju/juju/cmd/jujud/agent/machine.go", line:1775} ("failed to create C:/Juju/bin/juju-run.exe symlink: symlink c:\\users\\admini~1\\appdata\\local\\temp\\tmpivnvca\\gogo\\tmp-juju-testtxo382\\check-5796262061010820532\\7\\var\\lib\\juju\\tools\\machin [02:59] ... error stack: [02:59] http://paste.ubuntu.com/13590291/ [02:59] sorry for the fat finger [02:59] somethign is trying to create a symlink on windows [02:59] ain't nobody got time for that [02:59] thumper: i'll take a look [02:59] davecheney: i'll take a look at that one too since I can guess what it's a bout [02:59] about [02:59] menn0: ta [03:00] menn0: want me to create a new bug ? [03:00] should make a nice change from the massive and horrible merge I've been working on all day [03:00] menn0: almost certainly, some of these shouldn't be running on windows... [03:01] menn0: all yours, https://bugs.launchpad.net/juju-core/+bug/1521446 [03:01] Bug #1521446: featuretests: failure on windows due to ill informed symlink [03:02] davecheney, thumper: are these both with master? [03:02] the feature test failures are [03:03] Bug #1521446 opened: featuretests: failure on windows due to ill informed symlink [03:03] menn0: any test that is deailing with running upgrade steps can be skipped on windows IMO [03:03] as they will never be executed on windows [03:04] menn0: I believe this is the source of the issue [03:04] thumper: fair enough... but I think they *did* run on windows previously [03:05] thumper: and I'm surprised the symlink issue isn't happening on Linux [03:05] * thumper shrugs [03:05] I've not looked deeply at it [03:05] just looked at the errors that were being spewed [03:06] * menn0 looks [03:06] and the commits that happened since the last bless [03:06] and put 1 and 1 together [03:06] I may have gotten 3 [03:06] but this appears to be the issue [03:06] thumper: davecheney I updated the bug description; https://bugs.launchpad.net/juju-core/+bug/1471941 [03:06] Bug #1471941: windows unit tests fail because handles are not available

[03:06] for extremely large valyes of 1 [03:06] sinzui: thank you [03:07] sinzui: i think that is incorrect [03:07] this faiure does not occur on master [03:07] that file is not present [03:07] this cannot be the same issue [03:08] davecheney: interesting, because http://reports.vapour.ws/releases/issue/559acfbe749a565572d289f1 master was tested [03:09] davecheney: The last run in that issue is http://reports.vapour.ws/releases/3376/job/run-unit-tests-win2012-amd64/attempt/1657 and it says master. Do you suspect a stale tarfile? [03:10] sinzui: is it possible when disucssion these failures to refer to the jenkins build [03:10] that is how I know how to diagnose those failures [03:10] is there a link on the reviews.ws page to the jenkins failure ? [03:11] nm [03:11] that page has enough details [03:11] davecheney: http://reports.vapour.ws/releases/3376/job/run-unit-tests-win2012-amd64/attempt/1657 jas a link that says "jenkins" [03:11] sinzui: http://paste.ubuntu.com/13590518/ [03:11] this looks like the failure from that build [03:12] sinzui: thanks [03:12] which is a different bug [03:12] sigh [03:14] davecheney: I am not particullary interested in this bug or taht bug. We are certain this master and no other branch tested today fails and the failure is the windows job, and the failure is consistent in 7 tries. [03:21] * thumper is making juju jump through flaming upgrade hoops [03:35] wallyworld: http://reviews.vapour.ws/r/3281/diff/# [03:36] looking [03:36] Bug # changed: 1497809, 1497810, 1509747, 1519994 [03:36] Bug #1521453 opened: worker/upgrader: test failure on windows due to file being in use [03:41] davecheney: regarding that symlink error... do you have the full error text? it's truncated in the backscroll above [03:41] thumper: just a wording quibble [03:46] davecheney: never mind... the error thumper asked me to look at also has the symlink failure. I think they're the same thing. Not being able to delete some log file is just a side effect. [03:46] wallyworld: hmm... I think we should change "We" [03:47] * menn0 doesn't have time for this. Skipping this test on windows. As thumper indicated, this test doesn't matter unless we plan to support state servers on windows. [03:47] wallyworld: how about "Please upgrade to the latest 1.25 release" [03:49] menn0: +1 [03:49] 'aint going to be a state server for a long time [03:50] along with the complete sellout of FOSS principals [03:50] so skipping the test sounds like a reasonable middle ground [04:13] davecheney: http://reviews.vapour.ws/r/3282/ [04:15] davecheney: actually, hold the phone. I just noticed this: https://bugs.launchpad.net/juju-core/+bug/1446885 [04:15] Bug #1446885: Skipped cmd/jujud/agent/upgrade_test.go tests on windows

[04:19] davecheney: *now* PTAL: http://reviews.vapour.ws/r/3282/diff/ [04:23] menn0: looking [04:24] menn0: why not [04:24] // +build !windows [04:24] just skip 'em all [04:24] meh, ship it [04:25] davecheney: I'd it this way as there may well be non-states server related upgrade tests in the future [04:26] sure, i didn't mean to overcomplicate things [04:26] davecheney: merging nw [04:26] now [04:27] and now for more pain... [04:28] \o/ [04:30] menn0: there are only two tests in that suite [04:31] the skip should have probably gone to setupTest or something [04:31] whatever [04:31] davecheney: nope, the way I did it was intentional [04:32] fair enough [04:32] zero ducks [04:33] menn0: here is a small review in return [04:33] http://reviews.vapour.ws/r/3283/ [04:36] cherylj, wallyworld: how does this look: http://reviews.vapour.ws/r/3279 [04:37] cherylj, wallyworld: sorry for the misunderstanding. I thought we had unit statuses disguised as agent statuses - not simply that unit statuses were missing. [04:39] davecheney: unreserved Ship It :) [04:39] ta [04:39] interstingly the race only happens on go 1.2 [04:39] something in later versions obscures it [04:39] that's gotta be a fluke [04:39] even with a long stress time [04:39] it's a race, sure and certain [04:40] what is likely that in go 1.2, the machiner worker starts _after_ the test has completed [04:40] but in later versions the machienr worker runs immediatley [04:40] immedaitely [04:40] the race is on the patch'ed net.GetInterfaces method [04:41] PatchValue will be the first against the wall when the revolution comes [05:17] why do we have a utils repo? [05:17] why are all those packages not just in top level repos by themselves? [05:43] natefinch: +1 from me [05:43] if they cannot justify a repo of their own [05:43] kill 'em with fire [05:43] yep [05:49] waigani_: sorry, was at doctor, reviewed, a couple of small changes to make [06:18] wallyworld: cool, just finished dinner, will update now [10:01] dimitern: jam: frobware: fwereade: standup? [10:03] omw [11:20] oh great, my bug is caused by a lock, where have I seen this discussion before... [11:36] frobware, ping [11:38] frobware, now that I have working IPv6 setup on MAAS, I found a small issue with the bridge script in 1.25 - get_gateway needs to handle the case where you have multiple default gateways (e.g. fc00::.. and fe80::.. - the latter is always added) [11:40] only needed change was to pipe the ip route list exact default output through "head -n1" first before piping that through cut [11:45] dimitern, ack [12:08] dimitern: ping [12:08] voidspace, pong [12:09] dimitern: inside the discoverspacesWorker I need access to both the Spaces facade and the Subnets facade [12:09] dimitern: can I create a single API struct with two facades [12:09] dimitern: or should I have two separate API structs [12:10] I need two FacadeCallers either way, I just wonder if it's good practise to concatenate them into a single type [12:10] voidspace, why 2 facades? [12:10] dimitern: Subnets and Spaces are separate facades and I need to create both spaces and subnets [12:10] voidspace, both of these are client facades, not agent [12:11] dimitern: do we have a different facade I should be using? [12:11] dimitern: or should I just use an agent facade [12:11] voidspace, nope I think we need a new facade including both of these methods, ideally also reusing the methods implementation in the 2 client and the new agent facades, using apiserver.common mixins [12:12] dimitern: is there a difference in how agent facades are implemented? [12:13] this card just got bigger if we can't use the existing api :-) [12:13] voidspace, facades are client-specific [12:14] voidspace, they're responsible for auth and params translation -- and if they're the only thing that needs the underlying capabilities they're often implemented there too [12:14] fwereade: time to go duplicate some stuff then ;-) [12:15] voidspace, if you have capabilities needed by multiple facades, great, move them into (a subpackage of) apiserver/common [12:15] yep [12:15] I'll do that first in a clean branch, then come back to the worker [12:15] voidspace, yeah, what fwereade said; also we can reuse existing code, with some refactoring - see apiserver.common.StatusSetter for example how to do it [12:15] dimitern: fwereade: thanks [12:16] voidspace, fwiw, it is likely worth just finishing the worker in terms of (an) interface(s) that will be implemented by the client [12:16] voidspace, the context switch will probably hurt more [12:16] ? [12:17] just looking at the code [12:17] fwereade: you're probably right [13:01] Bug #1521610 opened: Upgrade hung when moving from 1.18.4.3 to 1.24.7 [13:35] it's a whole new experience being able to run make check *and* at the same time live test on 3 versions of virtual maas (in lxc) bootstrapping juju in kvms [13:56] dimitern: seems you're really happy about your new machine. ;) [13:57] TheMue, :) oh yeah! [13:57] dimitern: I've got a MBP here, i7, 2.5 MHz, 16 Gigs, and 512 Gigs SSD. only hard part is switching from German to US keyboard :) [13:58] TheMue, sounds nice! :) how's erlang treating you? [13:59] dimitern: aaahh, pretty nice again. but after that longer pause and doing go for so long I have to change some habits back again. [14:00] dimitern: but right now I'm doing API design for the clients, will use JSON via WebSockets [14:02] dimitern, voidspace, dooferlad: maas interlock? [14:13] * fwereade has found his test bug and is really annoyed about it [14:23] frobware, oops sorry, got carried away with maas testing here [14:23] TheMue, ;) sounds interesting [14:23] * fwereade is coming to the realisation that go 1.5 is going to make clock tests decidedly unfun [14:24] fwereade: oh, you have a version of go that makes them fun? [14:24] which was good, because I've discovered yet another issue, this time with precise [14:25] perrito666, 1.2 doesn't seem to exhibit a particular unhelpful scheduling behaviour [14:25] that, for me, is a few kilometers left of the fun mark [14:25] perrito666, haha [14:30] * fwereade is worrying that we'll need to have setup tombs, or something... crib/tomb? cf cradle/grave? [14:32] the infuriating thing is that the test that I *know* fails can be easily modified to handle it [14:32] wonderful upgrade has a stale lock in agentconfig [14:32] but I *also* know that my other tests are now vulnerable too even if it hasn't shown up yet [14:36] fwereade: any idea if deleting charms out swift mid-upgrade is likely to hose anything? [14:38] mgz, hmm, we have something that copies them into gridfs, don't we? [14:39] we do... this is a bad idea for some old IS deployments [14:39] mgz, if that's already done it should be fine; if it's not I would be nervous about deleting them [14:39] mgz, ha, right, fat charms [14:39] one of which is now very slowly copying fat charms onto local disk mid upgrade from 1.18 to 1.24 [14:40] mgz, right [14:40] feck [14:40] mgz, I have no idea how it'll react if they're not there [14:40] or were there when it did a list at the start but then vanish when it comes to copying... [14:41] mgz, but if you replace them with a byte of garbage each, I don't think anything depends on their *content* until we deploy new units that expect to use them [14:41] mgz, I do still consider that pretty serious breakage [14:41] hm, I like that though [14:42] we can see the version numbers in swift I believe [14:42] mgz, so long as there's some other way to get the charms into gridfs before they're needed it feels least likely to confuse everything [14:43] mgz, ...unless we're smart enough to check the hashes at some point in that process [14:45] pretty sure we're dumb [14:46] if it panics because the charm contents don't match when it fetches them I'll cry [15:04] dimitern: ping [15:06] dimitern: so it's good that we have a new facade for creating spaces, because creating them from the provider is slightly different [15:06] dimitern: i.e. we need the provider id and we generate the juju name in the apiserver [15:06] dimitern: so it's not an exact duplicate of the existing one [15:07] dimitern: this also means that the space collection in state will need to grow a ProviderId field [15:07] frobware: see above - the "simple" task I'm on of importing spaces has grown a bit [15:08] frobware: we need an entirely new api facade for adding spaces and the state representation of spaces need to change [15:08] frobware: not quite just the "add a simple worker" we anticipated yesterday [15:08] voidspace, :) [15:08] frobware: just a heads up... [15:08] voidspace, thanks. [15:09] voidspace, hey [15:09] voidspace, yes, that sounds good [15:14] dimitern: what does "IsPublic" mean for spaces - this is to do with mapping public IP addresses in AWS, right? [15:14] dimitern: we don't need this parameter for the new AddSpace [15:15] voidspace, public needs to be enforced for subnets added to a public space (to be themselves public) [15:15] voidspace, it expresses intended use, not AWS-specific thing [15:15] dimitern: ok, how does that correspond to spaces we discover from MAAS [15:15] where we are modelling the outside world, so are not in a position to "enforce" anything [15:16] and the concept of public/non-public is not part of the world we are modelling here [15:16] (i.e. even if it isn't intended to be AWS specific - it is really... ;-) [15:16] voidspace, in MAAS I think all spaces are private by default [15:17] ok, I'll just set IsPublic to false [15:17] voidspace, +1 [16:21]