[00:07] thumper: wanna shoot for a 1:1 today ? [00:26] hi davecheney - what do you think? are we trying something too crazy? [00:30] beisner: deploying anything to machine 0 is crazy [00:30] but that isn't the problme [00:30] best I can tell the error is coming from upstart [00:30] davecheney, lol. [00:30] deploying units to machine 0 has been responsible for 100% of customer data loss [00:31] davecheney, good to know. fwiw, this is related to automated deployment testing, and we do that only for density. the whole enviro gets torn down after openstack tempest tests run. [00:32] what else do you think, aside from our crazy unit 0 ways? ;-) [00:34] the error is coming from upstart [00:34] can you make an issue and attach the raw file from upstart [00:34] the one you pasted [00:34] it has to be the raw file [00:35] i suspect there is some control characters or other whitespace crap in there that is throwing off upstart [00:36] davecheney, yes, i'd be happy to. will ping you in a bit. [00:45] davecheney: yeah, how about after lunch your time? [01:00] davecheney, so that's getting interesting (copying the raw file) with the 1/lxc/3 having "no internal address." [01:02] ahh never fear. there's always /var/lib/lxc/juju-machine-1-lxc-3/rootfs/etc/ ... grabbing it. [01:07] davecheney, https://bugs.launchpad.net/juju-core/+bug/1420049 [01:07] Bug #1420049: jujud: Syntax error: word unexpected (expecting ")")

=== kadams54-away is now known as kadams54 [01:29] menn0: the more I think about it, the more I reckon the services running on the api server machines on behalf of another environment should not appear in the other environment log, but in the state server environment log [01:30] thumper: I guess that sort of makes sense [01:30] which does keep things much simpler [01:31] thumper: but if I was investigating a problem within an environment I might want to see those logs... [01:31] well, you'd have to ask [01:31] :) [01:31] I think we should definitely have some way to record which env certain logs are for... [01:31] thumper: but if I was a user with access to an env, but not the state server env then I wouldn't be able to [01:31] but probably as part of the module [01:31] exactly [01:32] I don't think this is a bad thing [01:32] thumper: it's certainly safer that way [01:32] much safer [01:32] and a good starting position [01:32] thumper: for a user like that, the state server should probably be a black box [01:32] I'm prepared to be convinced that we may want to change in the future [01:33] but as a solid well defined starting position [01:33] the state server for other environments is as you just mentioned, a black box [01:33] thumper: you've convinced me :) [01:33] menn0: lets start with that assumption then [01:33] and document decisions [01:33] thumper: not state server... JES [01:33] sure [02:01] axw: hey, updated http://reviews.vapour.ws/r/864/diff/# [02:07] thumper: looking [02:14] thumper: LGTM [02:14] axw: ta [02:15] axw: the expected behaviour is that if you created a new environ config instance with New, then it will fail if there is already one with the same noe [02:15] name [02:16] thumper: ok, I'm just looking at the contract of EnvironInfo specifically [02:16] * thumper scratches his head [02:16] thumper: I think it needs clarification [02:16] not a big deal [02:16] also... all likely to change soon [02:16] :) [02:17] you have me wondering now... [02:17] * thumper looks [02:18] I had to change the memory implementation to check for created and !initialized in order to get the second write working while failing two news [02:19] and thought that the check should be the same with the disk implementation [02:19] * thumper looks again [02:21] hmm... [02:21] I thought I tested that [02:21] just using !iniitalized in both places does work [02:21] * thumper goes with simpler [03:10] morning all [03:11] davecheney, ping? === kadams54 is now known as kadams54-away [03:26] anastasiamac: https://github.com/juju/errors/pull/17 please [03:27] anastasiamac, and when you're done with that would you mind taking a look at this one http://reviews.vapour.ws/r/903/ :) [03:28] axw: done but m not graduated :D [03:28] thanks [03:28] mattyw: np - m going thru another one of axw [03:28] thumper: teeny weeny change to errors, would you mind? https://github.com/juju/errors/pull/17 [03:28] mattyw: it's a lot of fun - storage - it' s huge [03:29] mattyw: but will be done soon and will look at urs :D [03:29] anastasiamac, wnjoy that - mine is just a backport of a pr that landed yesterday so should be simpler :) [03:29] mattyw: sure thing :D [03:35] axw: done [03:35] davecheney: still want to do a 1:1? [03:37] anastasiamac, and one more - I've tried to make it as trivial as possible [03:37] anastasiamac, .... and here's the link: http://reviews.vapour.ws/r/904/ :) [03:37] mattyw: i will .. thnx :D [03:38] anastasiamac, you'll struggle to find a simpler one than /r/904 - ever I think [03:38] mattyw: :D i'll let u know where it is on my scale... [03:59] thumper: thanks [04:13] mattyw: out of interest, where r the credentials for metrics set? [04:14] mattyw: and yes, 904 is trivial. the only that would beat it, would a punctuation/typo correction ;D [04:14] mattyw: urs takes the rpize [04:15] prize* [04:16] anastasiamac, they're set here https://github.com/juju/juju/blob/master/state/unit.go#L1790 [04:17] anastasiamac, it's one level up from the todo in the scheme of things [04:17] mattyw: ah [04:17] anastasiamac, but it makes everything much easier [04:19] mattyw: thnx :D [04:23] hmmmm; so I'm told the table test approach is hard to debug and I shouldn't use it except for trivial cases? [04:23] just want to get confirmation of that [04:27] i.e., for i, test := range []struct{...}{...}{do some tests on test} [04:27] bodie_: hey there [04:27] o/ [04:27] bodie_: I'd say tables are fine if the body of the test in the loop is simple [04:27] ie [04:28] minor setup [04:28] err := some-test [04:28] if test.err { c.Assert(err, gc.ErrMatches, test.err); } [04:28] else { [04:28] c.Assert(err, jc.ErrorIsNil) [04:28] other assertions [04:29] if there are other if statements in the body, break the tests apart [04:29] there are some table based tests in our code that really shouldn't be [04:29] that makes sense [04:29] :) thanks thumper [04:29] np [04:31] anastasiamac: here's a good one for you: http://reviews.vapour.ws/r/902/ [04:31] * thumper runs away [04:31] thumper: i'd add to it, that u might not want to have tables if u need to take adavantage of setup/teardown [04:31] anastasiamac: agreed [04:33] anastasiamac: FYI, a previous branch of mine added PrepareForCreateEnvironment, and renamed Prepare to PrepareForBootstrap [04:33] some environments need to do additional checks for bootstrap [04:33] like the local provider makes sure the ports aren't in use [04:33] I wrote a super ugly one.. or two.. using a func(){}() to wrap a defer... :P [04:33] most other providers are much more simple [04:34] bodie_: while that isn't too terrible, it makes it harder to follow [04:34] one key indicator I have is "a test should be easy to follow and obviously correct" [04:35] it is hard enough to decode convoluted code [04:35] add convoluted tests to that and it just hurts more [04:35] * thumper looks at the uniter tests [04:35] oh god [04:36] I'll keep that in mind [04:36] night all [06:00] jam: thnx for ur elegant comments on RB894. i was hoping there was a convention :D [06:00] jam: or at least a precedence [06:00] anastasiamac: happy to, thanks for doing the review as well [08:23] ericsnow: fwereade, good morning [08:23] and good morning to every1 [08:29] morning [08:42] Muntaner, o/ [08:42] fwereade: can you have a look? https://bugs.launchpad.net/juju-core/+bug/1420155 [08:42] Bug #1420155: Juju fails to bootstrap on a private OpenStack cloud due to not finding image metadata

[08:43] since I'm italian, maybe my english made this launchpad bug not understandable :) [08:44] Muntaner, no, that's clear, and much appreciated [08:44] fwereade: I'm curious about how developers work [08:44] all of the confirmed bugs get fixed in a single release? [08:45] or you choose the most urgent and important and fix them ASAP? [08:45] Muntaner, afraid not -- there are many bugs, and somewhat harsh constraints on which ones actually get chosen at a given time [08:45] Muntaner, fwiw, I am rather grumpy that we have this particular bug and that it's lived so long [08:46] fwereade: maybe it came out because nobody had tried a configuration like mine [08:46] fwereade: is that possible? just thinking loud :) [08:47] Muntaner, most likely, yes -- we see an awful lot of use of juju *deploying* private openstack clouds with maas [08:47] Muntaner, and maas has its own image handling [08:48] fwereade: well, my situation is "more new" then - I don't have MAAS in my configuration [08:49] Muntaner, yeah -- thank you very much for your help in tracking down the problem [08:52] fwereade: no problem! btw, I noticed that dimitern had reported this situation in another launchpad -> https://bugs.launchpad.net/juju-quickstart/+bug/1411574 [08:52] Bug #1411574: quickstart should detect private clouds somehow and generate metadata url in the environments.yaml [08:52] dunno if it needs to be merged [08:53] Muntaner, quite possibly, and ericsnow found another that looked very closely related too [08:53] Muntaner, funny how many ways there are of characterising the exact same underlying issue ;) [08:54] fwereade: so, the problem is that the upload of the metadatas fails in a LAN environment? [09:01] Muntaner, I have a strong suspicion that the metadata-uploading is just *broken*, which is what I'm mainly so grumpy about [09:02] Muntaner, I suppose that in general, for a prod environment, I'd expect a separate server for the metadata anyway, so maybe that's why it hasn't been spotted [09:02] fwereade: yes - I agree that our configuration is somewhat "raw" :) [09:10] fwereade, ping? [09:12] jam, ping? [09:13] hey mattyw [10:01] dooferlad: TheMue: making coffee, be with you in a minute or two... === benji is now known as Guest47535 [11:05] morning all [11:08] perrito666: heya [11:11] perrito666: o/ [11:42] man, I am waay to far from europe to get an ubuntu phone [11:44] voidspace: the latest change on the Prepare... branch is pushed [12:00] TheMue: cool [12:03] voidspace: the current test for the exhausting of addresses knows the number of available IPs on a fresh dummy machine. that's no real good dependency. I added AddressesAvailable to subnet, but sadly I've got no access to the subnet from inside the test [12:03] voidspace: so if you know a good approach here it's welcome [12:10] is the build server feeling sick? [12:13] * TheMue is out to lunch [12:13] mattyw: ? [12:14] perrito666, seems to have been rejected branches for the last hour with all kinds of weird test failures [12:14] mattyw: mm odd... well not that odd [12:14] mgz ? [12:25] TheMue: why do you need AddressesAvailable? [12:25] TheMue: you have AllocatableIPLow and AllocatableIPHigh [12:26] TheMue: available is High - low + 1 - numberAlreadyAllocated [12:26] TheMue: I don't really understand your problem [12:26] TheMue: point me to the test you think has the bad dependency [12:36] * TheMue is back [12:37] voidspace: AddressesAvailable IS High - low + 1 - numberAlreadyAllocated ;) [12:38] voidspace: I only created it to have an external access to this information [12:38] voidspace: especially with the retrieval of the numberAlreadyAllocated [12:39] TheMue: I don't understand the problem you're trying to solve [12:39] TheMue: there is code that calculates this before attempting to pick a new address [12:40] TheMue: so it knows if the address space is exhausted [12:40] TheMue: and it has to fetch all the allocated IPs anyway [12:40] TheMue: so it's no extra work [12:40] voidspace: it's only for testing, I would like to know how many are still available opposite to simple pick them until they exhausted [12:41] voidspace: it's not for the productive code [12:41] TheMue: can't you create a subnet, then manually allocate specific addresses? [12:41] TheMue: sorry, I thought you said you wanted to add AddressesAvailable to Subnet [12:41] that *would* be external code [12:41] why not just create a known situation [12:42] TheMue: ah, but you don't have access to the subnet anyway? [12:42] is that the problem [12:42] so adding AddressesAvailable wouldn't help anyway [12:42] voidspace: yep, no access inside the test [12:42] voidspace: sadly currently not, exactly [12:42] I don't think it's needed anyway [12:42] TheMue: I'm about to leave :-/ [12:42] voidspace: ok, thanks anyway [12:43] TheMue: you'd have to manipulate the test setup [12:43] which with our test heirarchy is non-trivial [12:43] voidspace: hoped for a simpler answer :D [12:44] voidspace: currently my test relies on knowing the hard coded subnets of the dummy provider. but once somebody changes it the test would break. [12:45] TheMue: well they will most likely notice :p [12:45] TheMue: my answer would be that address space exhaustion is already tested [12:45] perrito666: yeah, indeed [12:45] TheMue: so it doesn't need testing at this level [12:45] TheMue: mock out the call that would produce the error and check the error is handled well [12:46] TheMue: but you don't need to simulate actual exhaustion [12:46] voidspace: ok [13:08] voidspace: do you have a maas setup? [13:09] wwitzel3: yes [13:09] voidspace: in all my attempts I am failing to reproduce https://bugs.launchpad.net/juju-core/+bug/1417875 [13:10] wwitzel3: I'll have to look at it on my return, I'm off out to a meeting [13:10] wwitzel3: but I'll give it a go [13:10] voidspace: all I need is for you to bootstrap and deploy something to using --to 0 and see if the logging gives you the x509 cert error. [13:10] voidspace: thanks, appreciate it [13:17] how did I not set my vim to run go fmt before saving before? this is glorious. [13:29] perrito666: indeed, there is no other appropriate way [13:30] wwitzel3: I used to, git diff, review, git commit, git push, notice go fmt is sad, scream, kick, insult, go fmt ./..., git commit --amend, git push [13:30] perrito666: sounds exhausting [14:53] restore 4 out of 6 merged :') [15:02] FYI, at some point Reviewboard will be down for (hopefully) at most an hour while we switch to a new environment [15:02] I'll be sure to notify everyone when we're close [15:10] ericsnow: thx for info [15:24] natefinch: one-on-one? [15:25] natefinch: (in 5 min I guess) === kadams54 is now known as kadams54-away === kadams54-away is now known as kadams54 === whit is now known as whit|review-q [15:50] really another backup/restore bug breaking 1.21? [15:51] perrito666, sorry, two in fact [15:51] o ffs [15:51] ok, on it [15:53] I wonder if we should add testing support for this in the already bloated CI test [15:55] hey there hazmat ! [15:55] alexisb: greetings === kadams54_ is now known as kadams54-away === benji_ is now known as benji [17:06] Hi natefinch, do you have a moment to review http://reviews.vapour.ws/r/906/ [17:07] sinzui: ship it@ [17:07] ! [17:07] thank you natefinch [17:07] sinzui: welcome === kadams54-away is now known as kadams54_ [18:12] * perrito666 is back, from a 3g network in a dentist's waiting room === ChanServ changed the topic of #juju-dev to: https://juju.ubuntu.com | On-call reviewer: see calendar | Open critical bugs: 1420426 [18:21] katco, perrito666, ericsnow: CI just found a cross-compile regressions. bug 1420426 [18:21] Bug #1420426: backups_nonlinux.go import error

[18:21] you have got to be kidding me [18:22] perrito666: perhaps we should just disable all tests in state/backups on non-linux [18:23] perrito666: (in package_test.go add c.Skip() if runtime.GOOS != "linux") [18:23] ericsnow: could you tacle this one? I have my hands full with the other two restore blockers [18:23] perrito666: sure [18:23] tx a lot, I owe a beverage of your chooice [18:23] :) [18:29] sinzui: I apologize, that was most likely my [18:29] my fault [18:55] perrito666: http://reviews.vapour.ws/r/907/ [19:28] g'night all [19:28] EOD === fuzzy__ is now known as Ponyo === Spads_ is now known as Spads [19:57] orry I was suddenly trapped for the past hour in my mother in law's home, its a good thing I know how to tear apart a door [20:00] ericsnow: that diff is correct? [20:00] I am extremely curious how that got merged to begin === kadams54_ is now known as kadams54-away [20:05] perrito666: because it would only have failed under non-linux (which I presume the merge bot does not try) [20:05] ah makes sense === kadams54-away is now known as kadams54_ [20:18] davecheney: regarding that backups fix, I had considered checking runtime.GOOS, but version.Current.OS gives us better granularity [20:34] ericsnow: replied on the review [20:34] davecheney: thanks [20:41] sinzui: is it possible to backport (and release off course) a fix to 1.20? [20:43] perrito666, we can, though it is disabled at this time [20:44] perrito666, sinzui 1.21 is OK to me, if you can confirm that juju upgrade-juju works from 1.20.8.1 to 1.21.x [20:44] :) [20:45] niedbalski, 1.20.8.1 is either a custom build or juju built using upload-tools, it is not an official release and not in streams [20:46] niedbalski, but I can confirm 1.20.x can upgrade to 1.21.x. [20:46] ok, that is all I needed to hear [20:47] niedbalski, but I will call your attention to https://bugs.launchpad.net/juju-core/+bug/1416928 which murdered my beloved juju-ci3 env [20:47] Bug #1416928: juju agent using lxcbr0 address as apiaddress instead of juju-br0 breaks agents

[20:47] niedbalski, suspect t juju flipped out when it found lxc installed in the bootstrap server [20:51] cmars: can I defer our call? [20:51] so, I'm trying to invoke an rpc of addmachine and I want to pass an arg that will be converted to a placement.instance, much lik what the juju cli does when you do juju machine add foo (it converts it to {env-uuid: foo}) ... I'm sending over a Placement value in the MachineParams structure, and the response is that it cannot unmarshal strings to an instance.Placement ; so the question then is how do I pass the argument to the AddMachines [20:51] request such that it does what the juju cli machine add does ? [20:51] sinzui, this is the simpler way to reproduce ( A. start an instance in a cloud and install lxc on it. With the manual provider, attempt to bootstrap it. ) [20:51] ? [20:51] I'm using the python-jujuclient as the sender FWIW; === kadams54_ is now known as kadams54-away [20:57] rharper: placement expects to be scope:directive [20:57] rharper: so, like lxc:1 [20:57] except I can do juju add-machine foo [20:57] and it will do a env-uuid:foo as an instance.Placement [20:57] object [20:57] this allows me to have the provide add that machine (maas backend) [20:58] I want to do the same via rpc, but I can't see how to pass the args that the Init() in cmd/juju/machine/add.go does [20:58] it attempts to construct an placement object from the args value, and if it fails, it prepends 'env-uuid' [20:59] rharper: this is the code that does the parsing of the string: https://github.com/juju/juju/blob/master/instance/placement.go#L53 [20:59] yep [20:59] https://github.com/juju/juju/blob/master/cmd/juju/machine/add.go#L131 [21:00] that's the bit where it takes extra args from machine add and prepends 'env-uuid' and then calls instance.Placement() [21:00] thumper, sgtm [21:00] that's the code that appears to allow juju add-machine foo.maas to pass foo.maas (placement ) to the provide which in turns starts up foo.maas [21:02] natefinch: what I don't know is how incoming json rpc requests are unmarshal; it appears from the response that it can't convert strings to instance.Placement objects ... http://paste.ubuntu.com/10163887/ [21:03] niedbalski, That is one scenario [21:05] niedbalski, for my case. I think I need to bootstrap 1.20.14, install lxc, reboot the machine, upgrade to 1.21. I think those are the events that happened on my server [21:05] rharper: oh, I see, ok. so for just normal json unmarshalling you just need to make placement an object with Scope and Directive string values so, "Placement": { "Scope": "foo", "Directive": "bar"} [21:06] rharper: I actually think that placement parsing code is for translation from the CLI [21:06] sorry, my mistake [21:06] no worries; just trying to figure out how to do the same thing the cli is doing via rpc; [21:06] lemme play with that now [21:28] natefinch, do you have a minute to review http://reviews.vapour.ws/r/910/ [21:38] sinzui: the 2 restore ones are now committed to 1.21 [21:39] perrito666, you rock [21:39] sinzui: in this case niedbalski rocks, because he put up with all the "try this" ideas I had and added some of his own :p [21:40] thank you niedbalski [21:47] natefinch: thanks for the tip; I've got it working, I need to extract the environment's config uuid, pass that in as the scope, and the directive uses the machine name; [21:48] rharper: cool, glad you got it working [21:56] sinzui, np [21:56] perrito666, thanks for the fixes :) === whit|review-q is now known as whit [22:17] I really hate juju bot [22:18] perrito666: tsk tst [22:21] jw4: we all do [22:21] admit it [22:27] perrito666: lol [22:30] * hazmat pats juju bot on the head [22:31] heh [22:32] ericsnow, release call? === kadams54 is now known as kadams54-away === kadams54-away is now known as kadams54 [23:03] wallyworld_, can you review https://github.com/juju/juju/pull/1580 [23:03] sure [23:04] well that was hard [23:08] alexisb: sorry, had a dentist appt. [23:36] waigani: morning! as an OCR,could u plz cast ur eyes over http://reviews.vapour.ws/r/888/ [23:37] anastasiamac: yep, just finishing a rather big one off, you're next on the list [23:37] waigani: thnx :D