[00:00] axw_: question, we volumes are listed, i would expect that we'd want to see the unit and storage name to which the volume may be attached, so that we can see this 100GB volume here serves the data-dir storage for mysql/0, right? [00:00] at the moment, the PR just shows storage tag [00:22] hi, im having some strange problem after " juju upgrade-juju --version 1.23-beta2" [00:23] all units are loging "unit-mongodb-production-0[25074]: 2015-03-31 00:22:51 ERROR juju.worker runner.go:219 exited "uniter": failed to initialize uniter for "unit-mongodb-production-0": cannot read "/var/lib/juju/agents/unit-mongodb-production-0/state/uniter": invalid operation state: unexpected hook info with Kind Continue" [00:34] ok, now stop working :P [00:49] wallyworld_: volumes are machine-level, hence we see machine attachment. you could argue that since we know the assigned storage instance, we could also show the unit-storage attachments... but we have "juju storage show" if people really want to see that [00:49] axw_: i think that distinction will be lost on the user [00:50] storage will also show fs etc [00:51] if i list volumes, i'd like to see where they're attached (ie what unit) if i were a user [00:51] s/attached/in use [00:51] wallyworld_: I'm not opposed to adding an optional UNIT column [00:52] ok, let's do it [00:52] we could if required (which i'm not sure we will be) add a CLI flag to exclude/include volumes attached to units [00:56] redelmann: i'm not sure what's happened with your upgrade, can you raise a bug and attach as many logs as possible from the affected machine [01:12] wallyworld_: ok [01:13] redelmann: the error is an internal one that indicates that something got out of sync, but we'd need more info [01:14] wallyworld_: im collecting log from machine-0 [01:14] even if you zip the /var/log/juju dir [01:15] but we'd need logs from the machine running the affected unit [02:00] wallyworld_, where i can send logs? [02:00] redelmann: attach them to the bug [02:24] redelmann: that looks like an upgrade issue where the upgrade steps for 1.23 didn't run. Let me know when you've created the bug and attached the logs [02:25] (one of the upgrade steps) [02:26] jw4: ok, first im trying to remove "some way" upgrade from mongo in machine0, need this running for today :P [02:26] thanks redelmann [02:26] jw4, i keep a copy of log for bug [02:26] gracias [02:26] jw4: thank to you [02:27] wallyworld_ thumper: pyfmt - https://plus.google.com/+EliBenderskyGplus/posts/SzKrvzvUuyc [02:28] * thumper looks [02:28] I can't see it being as effective as gofmt, since whitespace matters in Python [02:29] * thumper guesses yet another python fromatter [02:29] indeed :) [02:29] yeah, first paragraph [02:29] can't have too many [02:30] axw_: interesting [02:30] I want a js version too [02:30] resharper does a good job with visual studio [02:30] and C# [02:44] axw_, wallyworld_ : https://bugs.launchpad.net/juju-core/+bug/1438489 [02:44] Bug #1438489: juju stop responding after juju-upgrade [02:45] sorry axw_: that no was for you [02:45] jw4: https://bugs.launchpad.net/juju-core/+bug/1438489 [02:45] thanks redelmann :) [02:45] np [02:45] tellm if you need more details, and how to get it. [02:46] now im going to keep trying to remove upgrade from ¿queue? mongo [02:47] redelmann: I don't see that error in that log [02:47] redelmann: do you have other logs? [02:47] jw4, that error is in "juju debug-log" [02:47] redelmann: I see [02:47] cant run debug-log again, it stuck [02:48] seems like that text should be in one of the other log files under ~/.juju/ [02:48] that's why im looking in machine0 logs [02:49] that error would have come from one of the units; what other machine logs do you have under ~/.juju ? [02:49] jw4, i cant see any log in ~/.juju/ [02:49] jw4, all unit say the same [02:50] jw4, actually im deploying to aws [02:50] redelmann: ya, I see that environment [02:50] jw4, juju keep log in ~/.juju if deploying to aws? [02:51] redelmann: I think it'd be on the machine that's having trouble upgrading [02:58] redelmann: how many machines do you have deployed? [02:59] jw4, seven [02:59] redelmann: which machine is unit-pgsql-medium-0 on? [03:00] redelmann: also, machine-0 should have all-machines.log in the /var/log/juju folder [03:01] jw4, you are right [03:01] give me a second [03:02] redelmann: certainly, thanks! [03:02] im scping it [03:02] Bug #1438489 was opened: juju stop responding after juju-upgrade [03:04] jw4, im uploading complete file, this happend today. [03:04] redelmann: thanks; I'll review that. I'm going to be out for 30 minutes or so, but I'll brb [03:06] jw4, ok, i attached it to https://bugs.launchpad.net/juju-core/+bug/1438489 [03:06] Bug #1438489: juju stop responding after juju-upgrade [03:06] thanks [03:10] cherylj: do you need some reviews done ? [03:16] OH MY GOD [03:16] why does juju depend on ajstarks/svgo !?!? [03:16] how is that possibly necessary for the core operation of juju ? [03:23] axw_: only if you feel like it/have time, there's a juju status review which is the first 3 or so branches which would be good to get looked at http://reviews.vapour.ws/r/1334, otherwise i'll bug william when he's net online [03:24] wallyworld_: just started looking, it's quite involved. I'd like to get some of my stuff proposed, hope to take a look later on [03:24] sure np [03:24] i an talk you through anything if needed [03:25] i'll have another proposed straight after, and then a 3rd hopefully later today [03:27] jw4, ok, i "fix" it [03:30] jw4, http://paste.ubuntu.com/10710367/ [03:30] jw4, after that it upgrade fine [04:55] so, it keep saying cannot read "/var/lib/juju/agents/unit-xxx-0/state/uniter": invalid operation state: unexpected hook info with Kind Continue [04:55] redelmann: yeah [04:55] jw4, but now i can juju status ;) [04:56] the basic issue is that in 1.23 the uniter does not expect there to be hook information in the uniter state except when it's actually running a hook [04:56] jw4, any workaround? like deleting /var/lib/juju/agents/unit-xxx-0/state/uniter? [04:56] redelmann: the upgrade steps were supposed to remove the unnecessary hook info from the uniter state file, but apparently it didn't work [04:57] redelmann: if you can find the uniter state file (yaml) and just delete the hook and then restart the uniter, it *might* fix it :) [04:58] jw4, i will try [05:00] redelmann: it should just be called uniter I believe [05:01] mh... [05:01] filter.go:137 cannot retrieve meter status for unit ganglia/0: not found [05:02] maybe writing a working uniter [05:02] redelmann: I'm not sure about that one (possibly upgrade related too, but I'm not familiar with that part) [05:02] jw4, thats happend after deleting uniter file [05:02] redelmann: oh. yeah. I wouldn't delete the uniter file [05:02] jw4, this one: /var/lib/juju/agents/unit-xxx-0/state/uniter [05:03] oh [05:03] redelmann: inside that file there should be a line (in yaml format) signifying the hook [05:03] if you just update the yaml so there is no hook key/value [05:03] that *might* work [05:06] redelmann: unfortunately it's past my bedtime, and my wife has been waiting for me to come to bed for 45 minutes now :( [05:06] redelmann: :) I'll check in again in the morning [05:06] jw4, thank you [05:06] jw4, have a good night [05:07] thanks redelmann you have a great morning/afternoon/night day :) [06:23] morning everyone [07:44] dimitern, thanks for going through my reviews, much appreciated [07:45] tasdomas, no worries [07:45] tasdomas, aren't you ready to graduate btw ? :) [07:46] dimitern, well, that's not really up to me to decide ;-] [07:46] tasdomas, I think you are, but I'll have a chat with cmars as well [07:47] dimitern, but what I find really holding me back in some reviews is lack of knowledge of certain parts of juju [07:48] tasdomas, that will come in time, but when unsure about some part of the code, you should ask questions :) [08:00] morning all [08:00] morning voidspace [08:00] o/ [08:00] morning o/ [08:07] dimitern: the described network problem isn't visible in the clients log. sadly I cannot grab the log of the VM [08:07] dimitern: here you can see that downloads from the maas controller fail [08:09] dimitern: also connections to 169.254.169.254 timeout [08:09] TheMue: o/ [08:09] dimitern: so that's why I think the network setup on the node fails after the changes [08:09] voidspace: o/ [08:12] TheMue, hmm - can you ssh into the node after it failed and check /var/log/cloud-init-output.log? [08:12] dimitern: with the written release we've added the gathering of a primary interface out of the lshw data which is used in newCloudinitConfig() [08:13] dimitern: I'll see if I can [08:13] TheMue, yes, in order to determine which interface to bind into the juju-br0 bridge at cloud-init initial boot [08:14] dimitern: yep, this has been with 1.20.12 [08:14] dimitern: only checked back where it has been changed [08:15] TheMue, yeah, I remember I did this to fix another bug about not using the correct primary interface for the bridge [08:16] ok, while client is still waiting the new node at least gave up [08:17] TheMue, ok, I think this can be faked - if getting the ifaceInfo and the primary interface fails, try using "eth0" as primary interface for newCloudinitConfig() [08:17] now I only have the problem to log in [08:17] TheMue, maybe the ssh keys where not set? [08:17] dimitern: same setting as yesterday [08:18] dimitern: but don't know how far they are distributed to the node [08:18] TheMue, no, I mean if cloud-init failed to complete for some reason, the ssh keys might not be set on the node [08:18] TheMue, cloud-init sets the keys [08:19] dimitern: yep, that could be the reason [08:19] dimitern: my node dislikes me because I corrupted its data :,( [08:20] dimitern: :) [08:20] dimitern: will do the eth0 approach, should be quickly done [08:21] TheMue, it's not the node - that's a case of juju not generating the userdata correctly so the node cannot boot properly :) [08:21] dimitern: damn security, thought we're building OPEN source *rofl* [08:21] TheMue, try that workaround I suggested - destroy and cleanup to start afresh, then use "eth0" as primaryNIC if lshw cannot be parsed [08:26] * TheMue btw loves his MAAS environment [08:28] wallyworld_: published some comments on your PR === axw_ is now known as axw [08:37] dimitern: no real change, the missing interface information leads to >> 2015-03-31 08:29:49 DEBUG juju.provider.maas environ.go:807 node "/MAAS/api/1.0/nodes/node-4a6120e6-d6ea-11e4-a9b5-000c2975eaae/" network information: []network.Info(nil) [08:38] dimitern: already a bit earlier >> 2015-03-31 08:29:49 DEBUG juju.provider.maas environ.go:760 node "/MAAS/api/1.0/nodes/node-4a6120e6-d6ea-11e4-a9b5-000c2975eaae/" has network interfaces map[] [08:39] TheMue, that could be fine, all it matters is the generated userdata for cloud-init and the node booting ok [08:39] TheMue, did it finish? [08:40] dimitern: looks like before, no action anymore on the node while the client is still "deploying" (repeating this log entry until timeout) [08:40] TheMue, did you use --debug with bootstrap? [08:41] TheMue, it will help if you can paste the output :) [08:41] dimitern: yes, that's where those two lines above are coming from [08:42] dimitern: will create a paste when the timeout is done so that the whole debug log is in [08:42] TheMue, ok [08:43] dimitern: the one I've mailed to you is also debug, see the top line how I'm deploying [08:43] dimitern: fresh-ducks.local is my corrupted node ;) [08:45] TheMue, hmm.. well it seems to me some more debug logging is needed around newCloudinitUserdata to show what's generated [08:47] dimitern: will log it [08:48] * TheMue just receive a warning by the national weather service, winds will get really strong today :/ [08:56] TheMue: we've had strong winds all night, continuing now [08:57] voidspace: it will reach its top here between 2 and 6pm, and our neighbor has still two very high trees on her ground which always sway extreme. during the last storm one felt, thankfully not into our direction [08:59] dimitern, ping? [08:59] mattyw, hey [09:01] voidspace, standup? [09:01] dimitern: omw [09:27] Bug #1438590 was opened: juju version tries to make directories and files in my home [09:44] dooferlad, so just calling c.SpaceCommandBase.SetFlags(f) in c.SetFlags() works [10:24] dimitern: String method sadly had same troubles with references as values in map[string]interface{}, so had to use the renderer way [10:24] dimitern: here is the according output http://paste.ubuntu.com/10711305/ [10:26] TheMue, ok, this looks fine to me [10:26] TheMue, does it still fail to boot? [10:26] dimitern: yep, still the same failing behavior [10:27] dimitern: while when using a not corrupted node it works [10:27] TheMue, can you try ssh into it? [10:27] TheMue, while the bootstrap goes on? it should be possible after retrying for a while (might not be possible immediately) [10:28] dimitern: yes, currently rejected [10:30] dimitern: still Permission denied (publickey). [10:31] TheMue, try: ssh -i ~/.juju/ssh/id_rsa ubuntu@IP [10:32] dimitern: still the same [10:34] dimitern: at least it's pingable [10:34] TheMue, can you see the console of the VM ? [10:34] dimitern: if the ubuntu user would have a standard password I could login directly on the console [10:35] TheMue, nope, that won't work unfortunately [10:36] dimitern: I know, sadly [10:36] TheMue, maybe the VM is still booting - do you see any messages on the console? [10:37] dimitern: only the login screen [10:38] TheMue, try with your ssh key configured in maas? [10:38] dimitern: the last interesting part have been messages that downloads from controller and this 169... failed [10:38] dimitern: that's what I'm doing, using that key [10:44] dimitern: I'll bootstrap to a valid node and see what the cloudinit looks there [10:45] TheMue, did you try both the juju_id_rsa key *any* your key? [10:46] s/*any*/*and*/ [10:46] dimitern: my key [10:47] dimitern: http://paste.ubuntu.com/10711389/ [10:47] TheMue, well, that's why I wrote this earlier: TheMue, try: ssh -i ~/.juju/ssh/id_rsa ubuntu@IP [10:47] TheMue, that's the juju ssh key [10:48] dimitern: eh, sorry, sure, took it too [10:48] TheMue, so both you couldn't ssh with either key [10:49] s/both/both/ [10:49] aargh [10:49] s/both//\ [10:49] dimitern: hehe, yes [10:49] dimitern: and you're right, the cloudinit looks similar, only the host names [11:01] dimitern: I send you a screenshot of the console. does this help to get in? sadly I cannot cut'n'paste, but enter the key by hand [11:03] wife calls for lunch, bbiab [11:03] Code comments: https://twitter.com/nzkoz/status/538892801941848064/photo/1 [11:30] voidspace, dooferlad, please have a look - juju subnet CLI http://reviews.vapour.ws/r/1339/ [11:37] TheMue, that screenshot is too small to read, unfortunately [11:37] * fwereade swings by quickly to say: sorry sick yesterday, public holiday today, should be around properly later to catch up a bit [11:45] dimitern: looking after coffee [11:57] dimitern: hmm, it is sent in full size, strange [11:59] voidspace, I'm in a call with spakiegeek now investingating an issue with maas and juju [12:00] voidspace, it seems we're not out of the rabbit hole yet [12:00] voidspace, I'm getting logs to help debugging the issue [12:01] voidspace, he'll file a bug with details [12:02] voidspace, it's a separate issue than the one you fixed, but also revolves around parsing lshw properly [12:04] TheMue, ah, so it's just that thunderbird is dump - I managed to zoom in using a copy of the screenshot in gimp [12:05] TheMue, that screen is from too early in the boot process [12:06] TheMue, we should do a screensharing session I think to diagnose the issue [12:06] TheMue, it won't work by just describing what you see :) [12:14] dimitern: just doing some preparations [12:19] dimitern: so, sent you an invitation, we'll see how it works [12:19] TheMue, brt [12:26] dimitern: ah, right :-/ [12:33] dimitern: so after #1339 there will be a "subnets" super command, but it will do nothing? [12:33] dimitern: wouldn't it be better to add the functionality first and do the CLI as the last bit - or have I misunderstood? [12:33] "subnet create" anyway [12:34] voidspace: the idea is to do CLI before API, so we know what the API needs to provide. We know what commands we want already (see Juju Network Model document) [12:35] dooferlad: right, but then we're adding public facing brokenness [12:35] dooferlad: if we know what commands we want, don't we also know what API we need? [12:35] voidspace: only to a feature branch [12:35] ah yes, of course [12:35] I still prefer to work the other way round [12:35] then we don't *need* a long lived feature branch [12:36] voidspace: We have a pretty good idea about what the API needs to provide, but this way we don't end up designing endpoints that are a bad fit for the code that is using them. [12:37] voidspace, sorry, in a call again [12:37] dooferlad: ok [12:37] dimitern: np [12:39] voidspace, the CLI returns an error when you try to use it [12:40] dimitern: well exactly :-) [12:40] dimitern: CLI commands that do nothing but error out shouldn't exist... [12:40] dimitern: why do you need the --private option? [12:41] dimitern: if it's the default it does nothing [12:41] dimitern: flag I mean [12:41] specifying it does nothing and it's existence is causing you pain in the code [12:41] dimitern: why not just have --public ? [12:42] voidspace, we always went with the approach "state / api first, cli last" and that bit us on more than one occasion [12:42] dimitern: ok, interesting [12:42] I'm definitely in favour of *designing* the CLI, just not necessarily implementing it first [12:43] voidspace, so starting from the CLI with stub API is the new way I'd like to try now [12:43] but as it's on a feature branch it doesn't matter so much anyway [12:43] dimitern: ok [12:43] voidspace, so by the time we reach state /api we already know what's needed [12:43] voidspace, the --private being the default comes as a requirement from mark [12:44] dimitern: fine. In that case the "--private" flag is superfluous, it does nothing. [12:44] voidspace, it's part of the spec though [12:44] dimitern: just say, "networks are private by default, unless you pass the --public flag" [12:44] dimitern: heh [12:44] it does *nothing* [12:45] voidspace, yeah that's correct [12:45] voidspace: well, it is an exciting way of starting an IRC conversation :-) [12:46] heh [12:46] ... just like the name "space" [12:47] dimitern: other than that (and feel free to drop my issue - I meant it as a comment) it all looks good to me [12:47] dimitern: mostly boilerplate except for the parameter checking, which all looks fine [13:02] voidspace, sorry, I'm looking at your review now [13:03] voidspace, yes, having --private seems redundant, but it's part of the spec [13:03] voidspace, I'll add a comment [13:06] A large tree crashed our car, shit. I'm off for some time. [13:06] TheMue, oh that's bad :( [13:06] TheMue: shoot [13:07] TheMue: after all that felling of trees one still got your car!? [13:11] TheMue, I hope the damage is not extensive [13:12] fwereade: andrew has looked at http://reviews.vapour.ws/r/1334/, it you get time would love it if you could look too [13:12] the rear is totally crashed [13:12] TheMue, oh dear :( [13:13] TheMue, well, the only thing I could do is offer you to take a day off if you need to sort things out :/ [13:16] TheMue: Yikes! I hope your insurer is efficient :-| [13:22] Bug #1438683 was opened: Containers stuck allocating, interface not up [13:32] voidspace, dooferlad, I've replied to your review comments; thanks! [13:33] dimitern: who knew one little flag would cause so much typing :-) [13:34] voidspace, I've also triaged that maas bug ^^ and added a comment capturing what me and sparkiegeek discovered so far, which will possibly help finding the issue [13:34] dimitern: yeah, I've been reading it [13:34] dooferlad, indeed :) [13:34] sounds like a mess... :-/ [13:34] voidspace, yep :( workarounds biting us back, as usual [13:34] if only maas had usable api :) [13:40] Hi folks and esteemed juju devs, apologies for the impertinent question but are there any known issues getting juju to work in Azure (aside from the bug reports)? [13:45] alexisb: sorry, I had hoped the baby wasn't audible over the headset mic.... guess it was [13:46] yes, and sorry I muted you but the baby cry still triggers the mommy instinct and made it hard to concentrate on what I wanted to say :) [13:46] heh I understand [14:03] voidspace, you should really do something to start noticing your PMs :) [14:25] dimitern: tyvm for reviewing my PR [14:25] dimitern: PTAL http://reviews.vapour.ws/r/1234/ [14:25] dimitern: i had to add unit info and change sort order for tabular format [14:25] dimitern: also http://reviews.vapour.ws/r/1332/ and http://reviews.vapour.ws/r/1333/ [14:26] ericsnow, ok, I have a few already, will add them to my queue [14:26] dimitern: if u have a chance if u could PTAL it again despite ur prev shipit - would be amazing! [14:26] dimitern: thanks [14:26] dimitern: http://reviews.vapour.ws/r/1323/ [14:26] anastasiamac, np - remind me which one? [14:26] anastasiamac, ok, cheers [14:27] dimitern: thnx ;D m EODing now 00:27 on the clock... [14:27] dimitern: http://reviews.vapour.ws/r/1332/ shouild be easy and is a blocker for the 1.23 release, so perhaps you could "nice" it up :) [14:27] anastasiamac, oh that's late - have a good evening then :) [14:27] ericsnow, ack, will start with it then [14:27] dimitern: thanks! [14:28] dimitern: or early :D ttyl [14:28] cherylj: it looks like the LICENCE fixes are all done for 1.22 \o/ [14:28] cherylj: is that right? [14:29] ericsnow: yes, as long as the patch for dependencies.tsv passes CI :) [14:30] cherylj: awesome! Thanks for tackling all that! [14:30] ericsnow: np! [14:30] cherylj: be sure to mark the bug as "fix committed" for 1.22 [14:31] thanks for the reminder :) [14:32] hooray! it passed CI [14:34] Bug #1438721 was opened: sync-tools does not support patch versions [14:40] abentley: where can I find the code for the CI tests? [14:40] dimitern, voidspace: ping [14:40] aznashwan, pong [14:40] natefinch: lp:juju-ci-tools [14:41] aznashwan: pong [14:41] abentley: sigh.. launchpad, huh? [14:41] dimitern: could I bother you guys with what looks like a horrendously huge review but there's actually little to it: http://reviews.vapour.ws/r/1330/ :D [14:41] aznashwan, I'm about 33% done with that monster branch of yours about cloudinit [14:41] :) [14:41] natefinch: of course! [14:41] dimitern: great [14:42] dimitern: as said there, most of it's relocated code [14:42] aznashwan, why did you decide to rename MachineConfig to InstanceConfig? [14:42] aznashwan, that definitely blew up the diff considerably [14:43] dimitern: that was bogdanteleaga's doing at the suggestion of one of you guys [14:44] aznashwan, I see [14:44] ok then [14:44] dimitern: he'd be the better half to ask about this [14:45] aznashwan, ok, thanks [14:47] dimitern, it was done after some consulting with william; in hindsight it might not have been the best idea exactly because it ended up making the diff humongous, but there's not much to see outside of /cloudconfig [14:51] bogdanteleaga, right, I agree it could've waited, but well .. :) [14:51] dimitern: dooferlad: voidspace: wow, what an organizational stuff with insurance, fire department, neighbor. will post pics later, car seems to be totally crashed [14:52] TheMue: time for another whisky tasting? [14:52] dooferlad: will these eve definitely need one [14:52] TheMue, :( too bad - hopefully the insurance will cover all [14:53] dimitern: currently all signs are positive, looks like a 100% coverage [14:54] better than many of our unit tests *lol* [14:54] TheMue, good! [14:55] TheMue: I saw that about your car - You've already had one tree fall this year, and your neighbor was felling more trees a couple weeks ago... what gives? [14:55] * natefinch tries to remember how to use bzr [14:56] TheMue: ouch :-/ [15:02] abentley: do we have tests trying out HA? [15:02] (in CI) [15:03] natefinch: Yes. See http://juju-ci.vapour.ws/job/functional-ha-backup-restore/ and http://juju-ci.vapour.ws/job/functional-ha-recovery/ [15:07] dimitern: port is done, but not run manual tests yet [15:07] dimitern: doing a full normal test run first [15:07] dimitern: then can switch to the other bug [15:07] voidspace, ok, sounds good [15:08] natefinch, ericsnow: I'm updating the dependencies for 1.23, and I see that updating juju/names pulls in this change: https://github.com/juju/names/pull/43 [15:08] natefinch, ericsnow: It brings in a new tag, but doesn't change any functionality. [15:08] space tag sounds so much better than freeze tag [15:08] natefinch, ericsnow: Will that be okay? or do we need a new branch? [15:08] haha [15:09] cherylj: that's expanding the API in a backwards compatible way, so I think it's fine to leave it in the same branch [15:10] cherylj: wait sorry [15:10] cherylj: misunderstood... yes, I think making a new branch there is the right thing to do [15:10] cherylj: it's simple, but doesn't really belong in 1.23, and none of the rest of the code will know what to do with it [15:11] natefinch: okay, glad I asked :) Can you create the branch? [15:11] cherylj: do you have the right commit hash handy? [15:11] (and yes) [15:12] is it just the one before that space tag change? [15:12] natefinch: ce4ecb2967822062fc606e733919c677c584ab7e [15:13] cherylj: thanks [15:13] np! [15:15] cherylj: there you go [15:18] dimitern: hah, a few calls to network.NewAddress to fix first... [15:18] natefinch: thanks! [15:20] voidspace, :) I dread the moment when I have to merge master into net-cli already [15:21] dimitern: heh [15:21] dimitern: it wasn't so many I had to do [15:28] Bug #1438748 was opened: Use of /tmp/discover_init_system.sh is a security vulnerability. [15:38] dimitern: so, for the "multiple networks with no index" - manually increment the index? [15:38] dimitern: we still risk duplicating an index if there are some with suffix and some without [15:38] dimitern: no idea if that's possible though [15:38] voidspace, I guess that's the only thing to do [15:39] ok [15:39] voidspace, we should add more logging at debug level [15:39] dimitern: ok [15:39] voidspace, cheers [15:39] dimitern: and I'll look through the code seeing if I can work out why we pick the disabled nic [15:40] dimitern: it's almost certainly due to them having the same subnet (network) *or* due to the index issue [15:40] dimitern: but it would be good to know [15:40] dimitern: running a manual test of my 1.23 branch in parallel to doing this [15:40] abentley: I'm trying to write a new CI test to test using ensure availability to turn existing machines in the environment into state servers. Is there documentation on how to write CI tests? It looks fairly straight forward, but docs are always helpful :) [15:42] voidspace, yeah - it was hard to tell what's going on, but the logs should tell [15:49] natefinch: Sorry, was on a call. [15:49] abentley: np [15:51] natefinch: There isn't documentation. Basically you need to return 0 on success and 1 on failure, and you should allow the runtime environment name to be customized, and jujupy may be helpful. [15:52] abentley: ok, cool. It looks pretty easy, except perhaps how to add the extra jenkins job [15:52] * natefinch knows jack all about jenkins [15:53] natefinch: when you've got a script that works locally, I can help you with getting it onto Jenkins. [15:54] natefinch: When you say "existing machines", do you mean machines that aren't provisioned by juju? [15:55] natefinch, can we land in the HA stuff at this stage? [15:57] dimitern: ok, so manually testing this has revealed a problem with the upgrade step... [15:57] dimitern: and I think trunk is screwed too - I just didn't notice before [16:00] dimitern: http://pastebin.ubuntu.com/10712911/ [16:01] dimitern: the best fix is to have ReleaseAddress not require an instance iD [16:01] voidspace, oh sh*t [16:01] dimitern: I'll repeat the test for trunk, but I bet it has the same issue and I just didn't notice it before :-/ [16:01] voidspace, this is because we need the instance id? [16:01] dimitern: yep [16:01] voidspace, yeah [16:02] dimitern: but I don't think either MaaS or ec2 actually require it [16:02] voidspace, I should've thought of that [16:02] dimitern: so we should just drop it from the provider method signature [16:02] if it's really not used [16:02] voidspace, well, how do you suggest fixing it? [16:02] dimitern: yeah, me too :-) [16:02] dimitern: stop requiring instance ID [16:02] voidspace, don't you need it to get the machine? [16:03] voidspace, or we added it only because we needed it for ReleaseAddress? [16:03] dimitern: I think we just thought we might need it [16:04] dimitern: ah no [16:04] dimitern: ec2 does use it [16:04] voidspace, *whew* ok, then it's easy :) [16:04] nope [16:04] voidspace, oh? [16:04] dimitern: it actually needs a nic id [16:04] dimitern: but we get the nic id from the instance id [16:04] voidspace, yeah [16:04] voidspace, ok, sorry I'm a bit confused [16:05] Can I get a couple license related reviews? http://reviews.vapour.ws/r/1342/ http://reviews.vapour.ws/r/1343/ [16:05] voidspace, you need instance id in order to call ReleaseAddress in the worker [16:05] dimitern: yes [16:05] dimitern: maas doesn't use the instance id [16:05] voidspace, you don't need instance id in the provisioner api [16:05] dimitern: but ec2 does [16:06] dimitern: the issue here is specifically when the upgrade runs and there are dead ip addresses [16:06] dimitern: nothing to do with the provisioner api [16:06] dimitern: the worker notices the dead addresses when it starts and tries to remove them [16:06] voidspace, ok [16:06] voidspace, so we do need to pass instance id, even if it's unused [16:06] dimitern: the addresses have a MachineId() [16:07] dimitern: the worker tries to get the instance id from the machine - but the machine is already gone [16:07] dimitern: Environ.ReleaseAddress takes an instance id [16:07] dimitern: that is not used by maas, but unfortunately *is* used by ec2 [16:08] alexisb, abentley: sorry, left for a minute to make lunch.... [16:08] voidspace, but fortunately, in ec2 it doesn't matter [16:08] voidspace, because if the instance is gone, all its nics and ips are gone with it [16:09] alexisb: I can land it by EOD. I have to do a few trivial code review cleanup things and write a couple trivial unit tests, and then I can start working on the CI tests [16:09] dimitern: ok, so we need a way to release the address for maas [16:09] voidspace, so I guess it only matters for maas - we need to release the addresses, but instance id can be empty [16:09] voidspace, it can be a special value like instance.UnknownId ? [16:09] dimitern: ok, fine [16:09] dimitern: and ec2 will make that a no-op [16:09] abentley: I mean machines that have been added to the juju environment, either through deploy or add-machine. So, like, you could do juju ensure-availability --to 1,2 and convert machines 1 and 2 into state machines [16:09] dimitern: whereas maas will release the address [16:10] dimitern: fine - thanks [16:10] voidspace, exactly - the upgrade step should try a best effort, but not fail if instance id cannot be found [16:11] natefinch: Cool. I was thinking this might be something like the manual provider. We have a solution for testing the manual provider, but it's not the cleanest. [16:12] abentley: nope, pretty easy, really... just like the other HA tests./.. just bootstrap, juju add-machine -n 2, wait for them to come up, then juju ensure-availability --to 1,2 and then wait for HA. [16:12] ericsnow, re http://reviews.vapour.ws/r/1332/ - it seems it just drops upstart-specific confs [16:12] natefinch: great. [16:12] dimitern: that is correct [16:12] ericsnow, shouldn't it only drop them when systemd is used? [16:13] dimitern: we don't need them for upstart either [16:13] ericsnow, i.e. isn't that strictly worse than before? [16:13] ericsnow, ok, if they're not used for upstart fair enough [16:13] dimitern: they are just clutter since we generate a new upstart conf when we bootstrap [16:13] ericsnow, I'll take your word for it though :) [16:14] ericsnow, ok, that makes sense [16:14] ericsnow, thanks! [16:14] dimitern: if restore were more than replace-the-old-state server then we would have a different story :) [16:14] ericsnow, LGTM then [16:14] dimitern: thanks [16:17] I love it when writing documentation for code reveals a bug. [16:18] natefinch, it's worse when it reveals a bug in the spec :) [16:18] dimitern: heh [16:19] dimitern: I actually wasn't being sarcastic. I actually do like it when that happens. Better than not finding the bug until later :) [16:19] dimitern: btw, are watchers fired off serially, or do I need a lock in my Handle() method in case two get fired off in quick succession? [16:23] natefinch, you're using the same handler for 2 watchers? brave! :) [16:23] dimitern: no no... but if there's two changes back to back [16:24] natefinch, the watcher should coalesce events properly [16:24] dimitern: not exactly filling me with confidence ;) [16:25] natefinch, well, it depends if the watcher is working properly :) [16:25] lol [16:25] ok [16:34] natefinch: we also need to branch juju/utils for 1.23 at commit 3efdaa3f15cc47ee83f6c03f640dc14f5915e289 [16:35] cherylj: done [16:36] * natefinch is getting the hang of this git thing. ;) [16:37] natefinch: thanks! I'm not sure if we need to branch testing too... It pulls in https://github.com/juju/testing/pull/55 and https://github.com/juju/testing/pull/56 [16:37] natefinch, awesome thank you [16:38] cherylj: to be on the safe side, I think I would [16:38] natefinch: okay, the commit to branch from there is: c07aef9cb4f2ff1db7ddc9d43a0ea88eca39c9ea [16:39] cherylj: done [16:39] natefinch: thanks! [16:41] cherylj: welcome :) === natefinch is now known as natefinch-afk [16:58] Bug #1438798 was opened: juju sync-tools requires default environment [17:11] voidspace, you still around? [17:35] natefinch-afk: I think we can get away with not branching juju/charm for 1.23. It just pulls in this: https://github.com/juju/charm/pull/82 [17:35] natefinch-afk: what do you think? === natefinch-afk is now known as natefinch [17:37] cherylj: lgtm [17:37] natefinch: thanks1 [17:39] alexisb: yep [17:39] alexisb: just, about to go [17:39] is there an example of putting a provider behind a feature flag? not exactly sure how to approach this [17:39] natefinch: could you review a few more of the license changes? http://reviews.vapour.ws/r/1342/ http://reviews.vapour.ws/r/1343/ http://reviews.vapour.ws/r/1345/ http://reviews.vapour.ws/r/1346/ [17:39] voidspace, looks like you picked up a new bug today for 1.23 [17:39] alexisb: yep [17:39] I was just curious the scope of work [17:40] alexisb: two I think :-/ [17:40] yay! [17:40] alexisb: should be able to land both in the next two days - one per day, maybe quicker [17:40] alexisb: I need to file the second one [17:40] great thank you that is what I needed [17:40] alexisb: the one that is filed is a *little bit* unknown [17:40] alexisb: so it *could* explode :-) [17:40] alexisb: but we don't think so [17:41] voidspace, ok, keep me posted [17:41] alexisb: will do [17:42] cherylj: I'm kinda slammed getting some critical code into 1.23, maybe someone else could do the review, like voidspace or ericsnow or cmars or wwitzel3 [17:42] ah, okay, thanks [17:43] cherylj, about to head out for lunch, can take a look in a little bit [17:44] thanks, cmars [17:46] alexisb: actually 3 bugs [17:46] alexisb: for 1.23 [17:46] ah, ericsnow got 'em. Thanks, ericsnow! [17:46] cherylj: they all LGTM [17:46] thanks!! [17:46] alexisb: bug 1438820 [17:46] Bug #1438820: IP address life field upgrade step and addresser worker don't play well together [17:46] cherylj: good catch on the blobstore dep :) [17:46] bug 1438683 [17:46] Bug #1438683: Containers stuck allocating, interface not up [17:46] bug 1438168 [17:46] Bug #1438168: juju 1.23 doesn't release IP addresses for containers [17:47] voidspace, are you thinking a day for each? [17:47] any you think could be deferred to a point release? [17:47] alexisb: one of them is done but blocked on the other two [17:47] alexisb: so still two days for all of them is my thinking [17:48] alexisb: bug 1438168 is complete but in testing it I discovered bug 1438820 that also affects trunk [17:48] Bug #1438168: juju 1.23 doesn't release IP addresses for containers [17:48] Bug #1438820: IP address life field upgrade step and addresser worker don't play well together [17:48] alexisb: but it is 3 bugs / 3 branches [17:50] voidspace, ok, now go enjoy your evening [17:50] thanks for the info [17:52] ericsnow: I'm glad you commented that you couldn't see the review on RB. I saw that I created that pull request against the wrong branch! [17:53] ericsnow: here's the correct branch: http://reviews.vapour.ws/r/1347/ [17:53] cherylj: LGTM [17:53] ericsnow: thanks again! [17:53] cherylj: np [17:58] Bug #1438820 was opened: IP address life field upgrade step and addresser worker don't play well together [17:59] thumper: are you actually on? === joec1 is now known as joec1afk === joec1afk is now known as joec1 [18:50] so, you're supposed to use UnitsWatcher to watch Machines... makes total sense [18:57] natefinch, fwereade would have lots to say about that (this is one of the areas he wants to clean up really badly) [19:00] alexisb: heh... yeah, this is the first time I've delved into watchers and stuff, and there's some interesting choices being made in there... I'm sure they each made sense at the time [19:55] natefinch: no, I wasn't [19:55] obviously forgot to close IRC last night [19:55] thumper: heh, figured [19:55] thumper: 7am isn't your usual start time :) [19:56] nope, it isn't [19:56] thumper: I had some questions about watchers and the API, but I think I figured it out from reading other API facades [19:56] awesome [19:58] davecheney: in case you missed it (re: gccgo in vivid): https://lists.ubuntu.com/archives/ubuntu-devel/2015-March/038740.html [19:59] ericsnow: yeah, I saw that, but uh, joining ubuntu-devel to respond is kinda a high barrier of entry. I think putting gccgo and go as different effective versions is a really bad idea. [19:59] natefinch: yeah, I was thinking that too :) [19:59] ericsnow: dave is off now for a while [20:00] thumper: k [20:00] off until he flies to eurpoe, next week sprinting on arm64 go stuff, then meeting us in Nuremberg [20:00] thumper: ah [20:57] oh amazon, you're so slow... [21:02] bbl === natefinch is now known as natefinch-dinnne === natefinch-dinnne is now known as natefinch-dinner [21:17] Bug #1436655 was opened: gce provider should stop using deprecated zone europe-west1-a [21:24] wwitzel3: would you mind assigning the GCE issues you are working on to yourself [21:25] wwitzel3: there are 2 right? [21:40] * thumper moves to the coffee shop to see if a change of scenery will help with focus [22:13] cherylj: so all 1.23 changes are wrapped up now, right? :) [22:13] ericsnow: yes :) [22:14] cherylj: lol, too many channels [22:14] hehe :) [22:44] natefinch-dinner: In updating the dependencies for juju master, I see a new repo, juju/jujusvg which is using the LGPL, but doesn't mention the exception found in other repos. Should I add it in? [22:44] could anyone spare me a review on http://reviews.vapour.ws/r/1349/ and http://reviews.vapour.ws/r/1350/? [23:15] Hello, is safe to have two enviroments in same AWS account? [23:16] juju destroy-enviroment --force will kill everithing in aws? or just instances of enviroment? [23:25] redelmann_: interesting question... [23:25] redelmann_: in theory, it should be fine [23:25] redelmann_: in practice, I'd test it first [23:26] redelmann_: it should be fine, so if it isn't, please make sure you file a bug [23:26] could just be my paranoia [23:28] thumper, I tried it a while ago and juju terminate all my instances (juju 1.20 i think) [23:28] thumper, thats why i asking about this [23:29] redelmann_: can you file a bug please? [23:29] also though... [23:30] you shouldn't need to use --force [23:30] that is a very "last resort" mechanism for cleaning up an environment [23:38] axw: after school drop off did we need to catch up about storage since we missed the standup? [23:39] wallyworld_: I don't have much to talk about. I'm going to propose storageprovisioner changes today, assuming I get enough test coverage done [23:39] can do if you like tho [23:40] all good, maybe we can touch base this arvo if needed [23:40] wallyworld_: ok. gotta go help get the kids ready, bbl [23:40] ttyl [23:49] wallyworld_: if you have a few minutes, could you take a look at http://reviews.vapour.ws/r/1349/ and http://reviews.vapour.ws/r/1350/? [23:49] wallyworld_: they are both for 1.23 [23:50] ok [23:50] wallyworld_: thanks [23:56] thumper, ok