=== akhavr1 is now known as akhavr [01:07] Bug #1505460 opened: Image consistency with containers can be broken [01:10] Bug #1505460 changed: Image consistency with containers can be broken [01:13] Bug #1505460 opened: Image consistency with containers can be broken [01:16] Bug #1505460 changed: Image consistency with containers can be broken [01:19] Bug #1505460 opened: Image consistency with containers can be broken === akhavr1 is now known as akhavr === akhavr1 is now known as akhavr === akhavr1 is now known as akhavr [03:42] Get:1 http://ports.ubuntu.com/ubuntu-ports/ trusty-updates/main linux-image-3.19.0-30-generic arm64 3.19.0-30.34~14.04.1 [51.2 MB] [03:42] 16% [1 linux-image-3.19.0-30-generic 10.0 MB/51.2 MB 20%] 56.8 kB/s 15min 1s^Z [03:42] wheee [03:42] cloud scale === meetingology` is now known as meetingology === urulama_ is now known as urulama [05:23] Bug #1505504 opened: error when destroying current environment in a multi-environment scenario === akhavr1 is now known as akhavr [05:32] Bug #1505504 changed: error when destroying current environment in a multi-environment scenario [05:47] Bug #1505504 opened: error when destroying current environment in a multi-environment scenario === akhavr1 is now known as akhavr [06:57] mgz: it looks like reviewboard has stopped adding the "Review request" comment to github PR titles [06:57] mgz: (i just had to add one manually to https://github.com/juju/juju/pull/3485 and the two before it haven't been labeled) === akhavr1 is now known as akhavr === akhavr1 is now known as akhavr === akhavr1 is now known as akhavr [08:07] rogpeppe: yeah, I broke eric the other day, will fix now and poke him again later to sort out [08:08] * rogpeppe hopes eric can be fixed === akhavr1 is now known as akhavr === akhavr1 is now known as akhavr === akhavr1 is now known as akhavr [09:01] voidspace: hangout time [09:04] jam: hangout? [09:36] frobware: I've turned it into a PR so it can be reviewed, but I won't land it until we've confirmed it actually fixes the bug! [09:36] frobware: https://github.com/juju/juju/pull/3489 [09:36] frobware: bootstrapping against 1.9 now [09:38] bug updated as well [09:38] voidspace, thanks - bbiab (rebooting)... [10:01] frobware: hmmm... two failed bootstraps [10:01] frobware: failing with 2015-10-13 10:00:33 WARNING juju.replicaset replicaset.go:98 Initiate: fetching replication status failed: cannot get replica set status: can't get local.system.replset config from self or any seed (EMPTYCONFIG) [10:02] frobware: switched to 1.25 and trying to see if it fails in the same way [10:02] voidspace, worked first time for me - using your branch [10:02] odd :-) [10:02] it could just be dumb luck [10:02] voidspace, the times I see this I force a really clean build... [10:03] frobware: that's certainly not the failure I would expect [10:03] frobware: after I've tried vanilla 1.25 I'll rebuild and do a clean attempt [10:03] I would expect everything after the ifdown/ifup to fail [10:08] frobware: yeah, vanilla 1.25 fails much harder [10:08] frobware: so I'll call that a win ;-) [10:08] (fails earlier) [10:08] voidspace, still going through the motions with multiple NICs [10:09] frobware: kk [10:09] in the meantime [10:10] dooferlad: TheMue: fancy a review? http://reviews.vapour.ws/r/2882/ [10:28] frobware: ah, a godeps made some changes, maybe that was it [10:28] trying yet again === urulama is now known as urulama|afk [10:29] voidspace, pretty sure that's why/when I see the replicaset issue. [10:29] voidspace, posted a few comments and still doing some manual testing. [10:31] frobware: thanks [10:31] frobware: we can't create unit tests for machines with multiple nics I don't think [10:31] frobware: as the tests really just do text manipulation [10:32] voidspace, I was thinking more of the case where we have multiple entries in the expected input/output tests. === urulama|afk is now known as urulama [10:33] frobware: we use the primary interface as found from the "ip route list exact 0/0" command [10:33] voidspace, so maybe I misread the test - let me take another look [10:35] frobware: nice improvement in those bash functions though [10:35] pushing now [10:49] frobware:can I drop that issue? [10:52] voidspace, line 219 in environ_test.go - can we not simulate in there that there might be multiple NICs? [10:53] frobware: that's the script we generate - which is the same for multiple nics [10:53] frobware: we don't simulate the machine at all [10:53] that script has to *work* on machines with multiple nics - but is itself unchanged [10:53] voidspace, that's just expected output though [10:53] that's the script itself [10:53] the output of generating the script [10:54] voidspace, to me that's just test input/output, no? [10:54] frobware: well yes [10:55] frobware: but that input / output will be the same on a machine with multiple nics [10:55] we do not simulate the machine [10:55] voidspace, so does it make sense to have a test that has multiple ethN entries? [10:55] we only generate the script and check the generated output matches what we expect [10:55] frobware: the script will never have multiple ethN entries [10:56] *running* the script on a machine with multiple nics will do something [10:56] but the unit tests never run that part of the code [10:57] frobware: see how that script doesn't mention ethN at all - but sets and uses $PRIMARY_IFACE [10:57] frobware: or are you talking about networkStaticInitial versus networkStaticFinal [10:57] voidspace, sure, but the stuff I thought we were talking about was just text matching/replacement [10:58] frobware: and adding extra ethN entries in the canned /etc/network/interfaces ? [10:58] frobware: well it is [10:58] voidspace, exactly [10:58] ah [10:58] frobware: I can add an extra entry and we can test it is left untouched [10:58] some value in that I guess [10:58] voidspace, agreed [10:59] I'll add an extra test [10:59] voidspace, I think there should be two: 1 with eth0, 1 with eth...ethN. [10:59] frobware: well I'll add one extra with an eth1 [11:00] frobware: if eth1 is untouched then adding ...ethN adds no extra value [11:00] we just want to test that "not the primary interface" is left intact [11:05] frobware: sorry for the misunderstanding :-) [11:06] Bug #1505617 opened: juju backup intermittent failure

[11:09] Bug #1505617 changed: juju backup intermittent failure

[11:10] frobware: additional test pushed [11:12] Bug #1505617 opened: juju backup intermittent failure

[11:15] voidspace, thanks [11:19] voidspace, mgz: do we have an easy or existing mechanism to build a ppa for a proposed fix like this? [11:26] frobware_: I don't know, I don't think deploying a ppa is "simple" [11:26] voidspace, heh, no. I've always used fpm when I wanted to do this. [11:27] frobware_: are you using trusty or vivid? [11:27] frobware_: on trusty I can bootstrap but then get the replset failure - reliably it seems [11:27] voidspace, trusty [11:27] frobware_: I'm trying changing the images to see if that helps [11:27] probably a different mongo package [11:28] but if it works for you it shouldn't be that [11:28] voidspace, to rebuild I use: alias rebuild-juju='[ -f $PWD/juju/api.go ] && { set -e; nukegopkg; make clean; godeps -u dependencies.tsv; make install; }' [11:29] frobware_: people ask for ppas, but generally that's not actually useful for juju [11:29] we're better off giving them some built binaries [11:29] mgz: frobware_ : in this case it's Mark Shuttleworth asking for a ppa... [11:29] yeah, well that's always a fun conversation [11:29] oh no its not [11:29] frobware: additional test pushed [11:29] Bug # [11:30] voidspace: you killed him [11:30] https://bugs.launchpad.net/juju-core/+bug/1494476 [11:30] Bug #1494476: MAAS provider with MAAS 1.9 - /etc/network/interfaces "auto eth0" gets removed and bridge is not setup

[11:30]

[11:30] it's andreserl asking [11:30] prebuild binaries should be enough [11:31] yup. [11:36] mgz, anyone else: reviews much appreciated for this PR which fixes a critical issue: http://reviews.vapour.ws/r/2878/diff/# [11:38] rogpeppe: I'll +1, looked at it yesterday [11:38] mgz: it's changed quite a bit since then [11:38] mgz: yesterday's didn't actually fix the problem [11:39] mgz: because we actually need to be able to log in [11:39] mgz: and the Login method i had fetched the environment config, which failed [11:39] mgz: so i've refactored things so agent login can be done without doing that [11:39] ;_; [11:40] mgz: (which incidentally did simplify things a bit which was nice) [11:48] Bug #1505648 opened: meter-status worker spins if charm doesn't have meter-status-changed hook

[12:05] frobware: shall I land this branch [12:06] voidspace, I was just about try / validate on 1.8 [12:06] voidspace, give me 30 mins. OK? [12:19] frobware: sure [12:26] voidspace, I ran into the replicaset issue too [12:33] mgz, ping? [12:44] frobware: 1.8 or 1.9? [12:44] or both :-( [12:46] voidspace, 1.8. third time lucky?? now [12:46] frobware: did you work out how to get ssh access? [12:46] voidspace, backdoor access? [12:46] yeah [12:46] voidspace, nope :) [12:46] I'll try hacking cloud init config and ssh into the machine after a failed bootstrap [12:46] the machine is still running [12:46] frobware: I'm going to have lunch first [12:47] voidspace, me too. though the patch seems fine for the combos I have tried on 1.8 and 1.9 [12:47] frobware: if it works sometimes that implies a timing issue [12:47] yeah, odd :-/ [12:47] voidspace, and I tried 1.9 way more [12:47] annoying [13:00] belated good morning all [13:26] mattyw: hey, what's up? [13:27] mgz, hey hey - sorry you can ignore me [13:27] mgz, I found some docs I could read instead of pestering others :) [13:27] :) [14:00] Bug #1505617 changed: juju backup intermittent failure

[14:30] bogdanteleaga: wotcha, you around? need to bother you about centos testing [14:32] ericsnow: also poké, when you have a mo, to talk about reviewboard github persm [14:32] mgz: k (OTP) [14:41] mgz: did you forget to publish your review of http://reviews.vapour.ws/r/2878/ ? [14:47] rogpeppe: yeah, need to finish that [14:47] mgz: thanks [14:50] rogpeppe: okay, the thing I am hung up on, and why I didn't finish looking through the code yesterday, [14:51] is one we've started the server, what's to stop something connecting before all the upgrades have actually happened [14:51] the ordering around this hurts my head [14:54] mgz: the machine agent has logic in it to stop that [14:54] mgz: and i agree, it's a pain to reason about and should really be fixed at a more fundamental level [14:55] mgz: but this at least works around the issue for now and is no worse than what we had before [14:55] mgz: i think that probably mongo schema upgrades should be treated entirely separately from the other upgrades [14:56] mgz: but that's too big a change for me to do right now (and it's not really my responsibility to fix old juju tech debt tbh) [14:56] rogpeppe: I agree this is better, will plus one and give in to the fact I don't comprehend all of cmd/jujud/agent/ [15:00] mgz: y'know, looking at the code, i'm no longer convinced that it does [15:01] ehehe ;_; [15:03] mgz: but again, it's no different to how it was before [15:04] rogpeppe: you have reviews. [15:04] mgz: ta! [15:18] frobware: this is the /etc/network/interfaces I have [15:18] juju-dev [15:18] * gberginc has quit (Quit: -sigh-) [15:18] frobware: http://pastebin.ubuntu.com/12773634/ [15:18] a bit weird [15:18] ah [15:18] auto eth0:1 [15:19] has been replaced with "auto juju-br0:1" [15:19] funnily enough it occured to me that the sed we were using would clobber interface names where the primary interface was *part* of another interface [15:19] dammit [15:19] ok, easy enough to fix and test I think [15:20] mgz, here's my yaml migration pr http://reviews.vapour.ws/r/2883/ [15:26] dooferlad: ping [15:27] mattyw: thanks! [15:27] mattyw: anything particularly fun? [15:28] mgz, there was one place where yaml setters and getters were changed and I think that reduced line count [15:28] mgz, and also there was some validation that just wan't needed anymore [15:29] mgz, https://github.com/juju/juju/pull/3490/files#diff-992668ccf77c55eabded974107a15a3bL94 [15:30] voidspace, how did you get it into that state? [15:30] mattyw: is it worth wrapping errors from marshal funcs, or will we get enough context anyway? [15:30] frobware: that's after a fresh bootstrap [15:30] frobware: for some reason it has an "auto eth0:1" in it initially [15:30] voidspace, I did not run into that case [15:31] frobware: were you able to ssh in? [15:31] voidspace, this through adding additional NICs via MAAS? [15:31] frobware: that's ssh'ing into the failed bootstrap node [15:31] voidspace, ah. without your change? [15:31] frobware: the sed code (pre-existing I might add) badly mangles "ssh eth0:1" into "auto juju-br0:1" [15:31] frobware: no, that's with my change [15:32] mgz, I think we get enough context - but opinions may vary [15:32] frobware: so I can fix that problem and see if it fixes the replicaset issue [15:32] voidspace, I never saw that generated with multiple NICs. [15:32] just bootstrapping now to see if I've fixed it [15:32] mgz, for example for long sequences or maps you don't get much feedback [15:32] frobware: even when bootstrap failed? [15:32] mgz, but in the cases I've changed you're likely to know what was input anyway [15:32] voidspace, the only time I saw bootstrap fail (on 1.9) was when I was not running your branch [15:33] voidspace, btw - I'm currently running 1.25 in a bootstrap/destroy loop and have not seen the replicaset failure. [15:34] frobware: with my fix in place I still have an eth0:1, but it isn't mangled by sed [15:34] so maybe it will work this time [15:34] bootstrap still in progress [15:34] voidspace, it's not clear to me how you got :1 added [15:35] frobware: that's the *initial* /etc/network/interfaces that cloud init modifies [15:35] so it comes from maas [15:35] voidspace, have not seen that. [15:35] it has a different address! [15:35] the machine has two addresses [15:35] voidspace, two addresses, 1 NIC? [15:35] yep [15:36] voidspace, aha. Thought you were getting this on 2 NICs- confused. [15:36] I have no idea why it is there [15:36] but it is, and juju is no longer mangling it [15:37] frobware: so the fix is good I think [15:37] frobware: I'll push shortly with an extra test [15:37] well, if the bootstrap works that is... [15:37] nope, same problem [15:37] so I think this additional interface in /etc/network/interfaces must be the problem [15:38] I wonder if bootstrapping to a different node, not screwed up, would help [15:38] I've acquired that node to force maas to pick a different one for bootstrap [15:38] trying again [15:39] voidspace, and the new node has an alias too? [15:39] frobware: don't know - will see shortly [15:39] voidspace, I just added one to my node... previously I was only concentrating on two NICs. [15:39] frobware: as far as I can *see* the alias (if that's what it is) isn't defined in virt-manager [15:40] no idea where it came from [15:40] voidspace, I added the alias in the MAAS screen [15:40] voidspace, so I see eth0 and eth0:1 each with their own addrs [15:40] ah, ok - looking at the node in maas [15:40] ah yes, it's there [15:41] frobware: the other nodes don't have aliases [15:41] so that's the problem [15:41] voidspace, but it is a legitimate combo. [15:41] frobware: well, it is [15:41] frobware: but it's an existing problem [15:41] let's fix the reported bug [15:41] and file a new one for the alias [15:41] as the current bug is urgent [15:42] voidspace, fair enough [15:42] voidspace, but I think we would have to get the new bug fix into 1.25 as well [15:42] that would depend on priority of that bug :-) [15:43] I also don't immediately know the right fix [15:43] I think we would probably need to add bridge_ports to all aliases too [15:43] frobware: I'll leave the fix to not *mangle* aliases in [15:43] as it's more correct [15:43] so I'll write the extra test [15:44] but that scenario still doesn't work [15:44] I assume because the alias isn't routable and mongo is picking that address [15:44] voidspace, agreed to not mangling aliases [15:44] cool [15:48] frobware: bootstrap just worked [15:48] so that was it [15:49] voidspace, if it's obvious I'm inclined to make sure it works with aliases too [15:54] frobware: it's not obvious to me [15:54] frobware: but it's *probably* not hard [15:54] let's land this and I can work on it alongside [15:54] voidspace, ack [15:55] voidspace, whilst testing your bits I may have made some progress with addressable containers and macvlan... [15:55] frobware: latest version pushed [15:55] frobware: cool [15:55] frobware: if you fancy a quick look over and giving me a ShipIt I'll land it [15:56] voidspace, looking and trying now... [15:56] and then I'll do back/forward ports [15:56] although dimitern said he has a simpler fix for 1.24 [15:56] not sure what that is [15:57] voidspace, heh - the node I'm using is entitled 'elegant-belief'. :) [15:57] voidspace, I think the simper fix is just 2 invocations of sed. one to replace, one to insert the bridge bits. [15:58] ah, ok [15:59] maas node names are lovely [15:59] I'm on imaginative-hose [15:59] I wonder which meaning of hose they had in mind [16:00] heh [16:00] all of them [16:07] voidspace, I wonder whether as debug we should $(cat /e/n/i) before and after our mods. At least the former may show up before we nuke any access... [16:08] frobware: to the client? [16:09] voidspace, wherever debug goes [16:09] frobware: I don't think there is any debug output from cloud init [16:09] frobware: we could save a copy before mangling it [16:09] voidspace, but this is juju doing this bit [16:10] frobware: this is cloud init [16:10] frobware: this is a bash script we send to the machine using cloud init [16:10] voidspace, oh. I though juju did this in place [16:10] no, that's why we have to use bash [16:10] it's not the juju executable doing this [16:11] it's bash code executed as part of machine setup [16:11] voidspace, so it could still be logged. [16:12] frobware: using syslog? [16:12] it's before there's a juju log file [16:12] voidspace, cloud-init can log stuff too [16:13] ok [16:13] voidspace, just to be clear - your latest change is not alias aware, correct? [16:13] frobware: it has changes to not mangle aliases [16:13] but it doesn't work with aliases [16:13] so basically correct, yes [16:14] voidspace, so no bootstrap possible with aliases (eth0:1)? [16:14] frobware: that's correct, but it's also correct with trunk [16:14] the new code is marginally better [16:14] as trunk code will mangle aliases [16:15] voidspace, because it does bootstrap OK for me with aliases [16:15] frobware: no replicaset issue? [16:15] voidspace, nope. here with aliases: http://pastebin.ubuntu.com/12773981/ [16:15] frobware: deleting the alias solved the replicaset problem for me [16:16] frobware: using my branch? [16:16] frobware: right, I guet that output - but bootstrap fails [16:16] it maybe intermittent [16:16] it probably depends which address of the two that mongo reports first [16:16] voidspace, 7a13a40365df2aa5f13572955783a8ac60fb0dd6 Don't mangle aliases [16:17] frobware: cool [16:17] frobware: I'll add an alias and try again [16:17] frobware: in the meantime, care to give me a ShipIt? [16:19] right, bootstrap in progress [16:24] voidspace, I added one more alias and it no longer bootstraps for me. [16:24] heh [16:26] frobware: really we would want mongo to use the primary interface not the alias [16:26] so I wonder if that's the real problem here [16:27] voidspace, so the script output ends up in /var/lib/juju/nonce.txt - so we could add the cat before/after to see what we did... [16:27] voidspace, correction in /var/log/cloud-init-output.log [16:27] ah, right [16:27] we don't need after, as that's what's in /e/n/i anyway [16:27] just before [16:28] why the $(cat ...) ? [16:28] why not just "cat ..." [16:29] voidspace, the $(...) was just be delimiting in the written prose... [16:29] ah :-) [16:30] voidspace, the after is a sanity check if the cloud-init is logged to a console and e/n/i is borked. there's then a chance we could see why. [16:31] ok... [16:34] voidspace, so your most recent change does handle the alias -- or so it seems to me. [16:35] frobware: ok, that's good [16:35] frobware: we can topic it in tomorrow's standup [16:38] voidspace, added a shipit - do you plan to post any binaries? [16:39] frobware: I can create some I guess [16:39] wonder where they should go [16:40] voidspace, http://people.canonical.com/~voidspace [16:41] frobware: ah [16:41] never used that... [16:41] I'll look into it [16:42] heh, 404 [16:42] voidspace, nor me. I just saw dimiter publish some stuff there the other day... not sure I can login atm. [16:42] voidspace, I think you first need to create public_html [16:42] right [16:42] I'll look into it in a bit - some exercise first [16:42] I've set my branch to land on 1.25 [16:42] voidspace: you can (with vpn or chinstrap) scp to lillypilly.canonical.com:~/public_html [16:43] though it likes scri [16:43] mgz: thanks [16:43] ..screwing up file persm so you have to ssh in to make apache like it anywa [16:43] heh, right [16:46]