[00:23] Bug #1589736 opened: BootstrapSuite.TestBootstrapPrintClouds unqeual fred and mary

[00:53] Bug #1589372 changed: state: state test failure during stress test [01:02] that feel when cpu drops to 0% during tests and you know someone has used time.Sleep [01:16] ? github.com/juju/juju/state/statetest [no test files] [01:16] ok github.com/juju/juju/state/storage 0.209s [01:16] ? github.com/juju/juju/state/testing [no test files] [01:16] why does the state set of package have _two_ testing helper packages ?!?!? [01:16] davecheney: just in case? [01:17] better add a third, that's the juju way [01:17] davecheney: belt and suspenders [01:18] and two pairs of underpants [01:21] anastasiamac: could you +1 this for me? http://reviews.vapour.ws/r/4995/ [01:23] thumper: hey, have a moment I think I migt be reading a test wrongly [01:23] or the test migt be wrong [01:23] perrito666: my head is in the middle of something else just now [01:23] gimmie 15-20? [01:23] sure [01:23] ill ping back [01:24] perrito666: could you +1 the above trival pr while you are waiting? [01:24] wallyworld: checking [01:26] wallyworld: lgtm... maybe we should have a test for it tho \o/ [01:26] thre is [01:26] a test [01:26] it failed [01:26] hence the fix [01:26] it failed sometimes [01:26] wallyworld: how much of an ass am I being if I say that your for defines I before but it could get it from unpacking the slice [01:27] damn yeah, will fix that was left over from previous code [01:27] tnks [01:27] lgtm for the rest [01:44] thumper: ping? [01:44] perrito666: actually, this will take longer L( [01:44] :( [01:44] sorry [01:44] with natefinch on a hangout [01:44] lol, no worries [01:44] bitching about networking [01:44] uh, bitching, I wish I was invited to that [01:44] I am a great complainer [01:47] Bug #1589748 opened: commands should prompt you to "juju login" if your password has expired [02:27] mpf, is it possible that acl tests where broken? [02:31] * perrito666 does not want to be right about this [02:36] thumper: still running stress tests of my patch [02:36] i hope to submit it this afternoon [03:17] Bug #1589774 opened: Ghost models exist after being destroyed [03:31] thumper: dagnabit, i think there is still one test case left [03:31] just had another "session closed" failure in myu stress tets [03:37] :( [03:38] anyone else having trouble with the charmstore? [03:39] thumper: not I [03:45] axw_: PR for no cotntroller/model on cli.. http://reviews.vapour.ws/r/4997/ [03:45] anastasiamac: thanks, will take a look a bit later on [03:46] axw_: no rush.. i will need to make sure that unit tests are aligned.. if there are any that are checking err cause and/or msgs [04:17] Bug # changed: 1466629, 1496143, 1522090, 1531886, 1534804 [04:32] thumper: I'm off to bed. Good luck with that bug. [05:14] thumper: http://reviews.vapour.ws/r/4990/ [05:14] Patch set 4 should be the good one [05:14] I've been stress testing it for an hour and I think I've found all the places we were leaking watchers [05:38] thumper: charm migration done: http://reviews.vapour.ws/r/4998/ === menn0 is now known as menn0-afk [05:40] wallyworld: axw_ either of you around? [05:40] redir: yup [05:40] maybe :-) [05:41] got a minute to look at something? [05:41] an offer i can't refuse [05:41] tanzanite? [05:42] sure [05:58] night [05:58] and thanks wallyworld axw_ === redir is now known as redir_afk [05:58] see you later alligator [06:04] the state/presence tests run in 5 seconds in a variety of machines [06:04] someone's put a sleep in there [06:09] axw_: once you land your current work, and i land the initial controller config branch, i have another ready to propose (after resolving any conflicts) which does the shared config thing from bootstrap. i also realised the current PR does correctly reject UpdateModelConfig attrs that are controller ones, so I'll remove that unnecessary TODO in this next branch [06:10] wallyworld: ok, but I'm not really comfortable with shared model config anymore. I think it may need more thorough design and discussion [06:10] (see my comments in review) [06:11] ok, looking [06:12] wallyworld: I'm not really convinced that it's that worthwhile either. how often will or should a controller admin be enforcing shared config for all models? [06:12] a lot in maas [06:12] apt mirror for example [06:12] mainly for private cloud case [06:12] or tools url etc [06:13] wallyworld: so then I think it might make more sense as cloud config, rather than controller-wide [06:14] wallyworld: tools url will be going away [06:14] wallyworld: either way, I think it needs a bit more thought before we go changing something so fundamental [06:15] it can be made cloud specific fairly easily [06:15] either way, the PR to pull out controller config should be good [06:15] wallyworld: absolutely 100% agreed [06:16] wallyworld: just finishing up my branch now, it's going to be big... [06:16] tis ok [06:16] i got to do school pickup, bbiab [06:52] axw_: i think shared cloud (not controller) config is worth persuing. it will work now with one cloud per controller, and later too. it will be easy to add a global clouds collection and store the settings docs on that, keyed on cloud name. the vast majority of the other code in the branch remains the same. this then allows maas specific apt mirrors etc to be easily set up across hosted models, based on what's in clouds.yaml [06:55] wallyworld: sure, just so long as we can do it safely [06:55] wallyworld: and without confusing semantics around updating/removing config [06:56] axw_: yeah, i guess it depend on how confusing is defined. we can certainly warn if a user deletes a model attr that is also shared [06:56] and tell them that the shared value will now be is use [07:00] wallyworld: can you please write down what you think the end solution should look like, in terms of user commands, and we can discuss at the tech board again [07:00] I don't think we really covered the inheritance side of things well before [07:00] and it was just you me and william [07:00] ok. that bit doesn't necessarily need to land before beta9 as it won't affect upgradability [07:01] wallyworld: yep, +1 [07:01] my focus today and tomorrow is very tightly focused on beta9 sadly [07:01] wallyworld: gotta go get charlotte in a moment, will then do a live test and propose my branch [07:01] ok, i'll look either before or after soccer [07:01] wallyworld: then there's a tonne of other stuff to do as follow ups :/ [07:02] axw_: and my PR needs another look too when you get done proposing [07:02] yep [07:02] okey dokey [07:02] so long as what we land is upgradable [07:04] wallyworld: food for thought: I wonder if things like apt-mirror would be better suited as being cloud-specific, and you *cannot* set them at the model level [07:04] wallyworld: then there would be one place for each thing [07:04] axw_: maybe, but what if i wany *my* model to use something else [07:05] and I would be much happier at least ;p [07:05] wallyworld: why would you? [07:05] eg i set up my own mirror for testing or whatever [07:05] or to get my own packages [07:05] wallyworld: if it's just for testing, set up a test cloud? [07:05] (you're asking legitimate questions, I'm just wondering if we can/should go down that route) [07:05] that seems unwieldy just to use a different setting to one that's shared [07:05] eep, gtg [07:05] ttyl [07:28] wallyworld: it's a more work, yes, but I expect that's a very uncommon thing to do (setting apt-mirror on a per-model basis). if we have one and only one place for each config attribute, then we can have very clear semantics for updating/removing/etc. [07:28] wallyworld: BTW I think we want to add identity-url and identity-public-key to the controller-specific config? [07:36] axw_: we problaby should, i was just going by what was currenty ordained as controller specific, seems like those ones were missing [07:38] axw_: although, unless those are set at bootstrap, there would be no way of setting them after at the moment [07:38] do we error in validation? [07:38] i can ping uros about how he sets that stuff up [07:38] yes [07:38] if you try and set a controller attr via set-model-config it will error [07:38] wallyworld: ok. even still, I think we should add it to the list in case we need to change that behaviour [07:39] so if i add those, it just eans they need to be set at bootstrap and are then invariant [07:39] for now [07:39] wallyworld: that's the same as controller-uuid, ca-cert, etc. [07:40] yep [07:40] i just don't know the workflow for setting those [07:40] i'll check with uros [07:40] okey dokey [07:40] if they set them after bootstrap, then boom [07:41] i do disagree with you about the benefit of shared config, so we'll see what others think [07:41] wallyworld: sure, I've added my 2c to the thread [07:43] axw_: whether we go for inheritance or not, the bootstrap code in the wip branch will still work - we'd just count stuff in clouds.yaml as cloud config [07:43] all the serialisation etc is there now, and the backend storage [07:44] just need to remove the inheriance bit if that's what we decide [07:44] so i should be able to get all this landed tomorrow [07:57] Bug #1589736 changed: BootstrapSuite.TestBootstrapPrintClouds unequal fred and mary

[08:02] hey all [08:03] dimitern: ping, 1:1? [08:03] frobware: sorry, omw [08:03] voidspace: o/ [08:03] wallyworld: sorry for the delay, LGTM [08:04] axw_: nw, tyvm. am landing the feature branch before i head off to soccer [08:04] will have to jfdi it [08:09] wallyworld: finally, http://reviews.vapour.ws/r/5000/ [08:09] as you'll see from the description, still lots TODO [08:09] axw_: ty, i'll have to look after soccer now [08:09] wallyworld: no worries [08:10] wallyworld: I'll be tackling the references next I think, to avoid upgrade steps [08:10] +1 [08:10] and then removing cloud endpoints/creds/etc. from model config [08:10] probably after your branch lands [08:51] dooferlad: ping [08:51] dimitern: hi [08:51] dooferlad: so I tried testing my patch against LACP bonds on the NUCs ... and failed miserably [08:52] dooferlad: morning :) [08:52] dimitern: I am not entirely surprised. [08:52] dooferlad: can I ask you to try it on your hw setup? [08:52] dooferlad: it's here http://reviews.vapour.ws/r/4959/ [08:53] dimitern: sure, but can we discuss it with Andy and see how it meshes with our plan for the iproute2 / rebooting / ifupdown in the standup? [08:54] dooferlad: sure, ok [09:01] dimitern: frobware voidspace hangout time! [10:00] Bug #1589890 opened: juju2 azure fail with error 409 network conflict

[10:01] https://bugs.launchpad.net/juju-core/+bug/1588143 [10:01] Bug #1588143: cmd/juju/controller: send on a closed channel panic

[10:07] it is because we have defer in a defer stmt? [10:08] because defers saves the state of the program and it mangles up the program? [10:09] who not delete the defer inside that defer and put the estate.mu.Unlock() before the <-OpDestroy{//code} [10:09] ? [10:09] why not* [10:13] https://paste.ubuntu.com/17085985/ [10:14] so why not like this.. anyway one go rutine will access the channel because the estate.mu.Lock() and after unlock it. [10:15] thoughts on this? [10:15] anyone? [10:16] and I'm reffering at this bug https://bugs.launchpad.net/juju-core/+bug/1588143 [10:16] Bug #1588143: cmd/juju/controller: send on a closed channel panic

[11:31] back [11:31] man a long wait at the hospital :-( [11:33] voidspace: o/ [11:33] dimitern: hi [11:33] voidspace: everything went ok? [11:34] dimitern: yeah, routine tests for my wife - just checking something out [11:34] dimitern: mostly her being paranoid I think [11:34] voidspace: I see, ok [11:35] voidspace: btw I'd appreciate a second review on http://reviews.vapour.ws/r/4959/ [11:41] dimitern: looking [11:42] voidspace: ta! [12:04] voidspace: wives are never paranoid \o/ [12:12] dimitern: why are you removing all the pre-up (etc) parts of /e/n/i ? [12:12] dimitern: why were they needed and why are they no longer needed? [12:13] voidspace: they aren't necessary anymore [12:13] dimitern: why were they needed and why are they no longer needed? [12:13] I'd like to understand if you don't mind [12:14] voidspace: they were there to work around known issues when trying to ifup multiple static interfaces [12:15] dimitern: and how do we work around it now? [12:16] Bug #1589890 changed: juju2 azure fail with error 409 network conflict

[12:16] voidspace: well, we don't :) all my tests on trusty and xenial in the past weeks confirm the initial boot slowdown is gone with simple, statically configured interfaces [12:16] dimitern: ok, have you talked it through with dooferlad? [12:17] dimitern: he added that stuff IIRC [12:20] voidspace: we did talk; but those the pre-up and etc. steps were my doing, which I'm glad to drop actually :) [12:20] dimitern: ah right, ok - my mistake [12:20] LGTM then [12:20] voidspace: tyvm! [12:25] dooferlad: any update on testing with bonds btw? [13:03] dimitern: was having lunch after my second stand up of the day. On it now. [13:03] dooferlad: ok, np - just checking [13:05] dimitern: why bridge_maxwait 0? [13:05] surely we want the bridge to enter forwarding mode before we continue? [13:06] dooferlad: that's the intent [13:07] dimitern: http://manpages.ubuntu.com/manpages/precise/man5/bridge-utils-interfaces.5.html says it won't wait for the bridge to enter forwarding mode [13:07] dooferlad: otherwise maxwait is 32s by default (although in most of my tests it's a lot shorter in reality) [13:07] dimitern: yea, waiting is good. [13:08] dooferlad: with multiple bridges it gets very slow very quickly [13:08] dimitern: if you want that change locally when you are iterating on something that is one thing, but landing it in production code seems wrong. [13:09] dooferlad: how about a compromise? [13:09] dooferlad: e.g. 5s [13:10] dimitern: I am not comfortable with that either. I assume that 32s was chosen for a reason. If we want to change it we need a better reason. [13:10] dooferlad: I can compare the boot times with different values of maxwait, but not specifying it is bad pretty much every time you have >1 br [13:11] dimitern: why? Shouldn't the bridge come up and boot continue? It doesn't always wait 32s right? [13:11] dooferlad: what's there to wait for? the port was up and running just before the script was run [13:12] dimitern: if that is the case, why does it not just continue anyway? [13:12] dooferlad: it does come up, eventually, but with 7 VLANs => 7 bridges, it can take more than 75s for some bridges [13:13] dimitern: that really doesn't seem bad to me [13:13] dimitern: and, as I said, if we are going to change a default we need to justify it [13:14] dimitern: I would assume that the default is very widely tested and setting it to 0 isn't. [13:15] dooferlad: ok, fair enough (will still test with the default maxwait to compare); not having addresses on both nics and bridges seems to be the most important part for reliability, along with stp [13:15] dimitern: Agreed. I just don't want any surprises :-| [13:16] dimitern: though I would love to have it as 0 if it didn't make any difference other than booting faster. Perhaps when we aren't trying to get a release out and we can throw some CI resources at it! [13:17] dooferlad: +1 [13:21] dimitern: so what do you want me to run as a test? Just see if a bonded interface still works on boot? [13:21] dimitern: then stick a lxc on a machine and check it works/ [13:21] ? [13:22] dooferlad: I'd suggest - bootstrap on bonded dual-nic with no vlans first [13:23] dooferlad: then tear it down, add a couple of VLANs on the bond and bootstrap + add a couple of LXDs to machine 0 (switch controller first) [13:23] dooferlad: and in both cases before tearing down, try rebooting and see if it still works [13:24] if you can ssh into the node, you should be able to also ssh into the containers [13:25] dooferlad: and I'd at least check 'ping google.com' and 'ip r' from inside the container [13:25] that should cover most cases [13:25] (common ones) [13:27] dimitern: about stp - it was disabled for security reasons according to http://manpages.ubuntu.com/manpages/xenial/man5/bridge-utils-interfaces.5.html so do we have a good reason for turning it on? [13:27] dimitern: is that another boot time thing? [13:29] dooferlad: that sounds terribly handwavy [13:29] dooferlad: what security concerns? [13:30] dimitern: that is what it said in the man page. [13:30] dimitern: I would like to know more too. [13:30] dooferlad: yeah, so far I haven't seen references to such issues [13:31] dooferlad: but I did notice improved UX and stability with STP on (I was having terrible broadcast storms with incorrectly configured switches) [13:32] dimitern: http://www.linuxfoundation.org/collaborate/workgroups/networking/bridge_stp [13:32] dimitern: looks like a win on trusted networks [13:33] dimitern: were you having any problems on correctly configured switches? [13:33] dooferlad: I see, ok - makes sense and our networks are usually trusted [13:34] dooferlad: nope [13:34] dooferlad: it might be a problem with certain setups (on a shared substrate) [13:34] (you know where..) [13:34] dimitern: so... is there any advantage really? I don't care about poorly configured networks. That is somebody else's problem. [13:35] dooferlad: the issue is entirely not apparent when it happens [13:36] dooferlad: i.e. you might have initial connectivity, which suddenly drops due to e.g. arping an unknown IP [13:36] dimitern: that isn't STP fixing your network - that is broken. [13:36] dooferlad: it does fix it though [13:36] dimitern: and Juju deploys servers that are public, i.e. HTTP[S] [13:36] dooferlad: by blocking the loops that otherwise happen [13:37] dimitern: I think it really should be something that users can turn on, but shouldn't be on by default. [13:37] dooferlad: why? [13:38] dooferlad: it affects MAAS setups a lot more than SDNy clouds [13:38] dimitern: turning it on turns on a security hole. [13:38] dimitern: see the linuxfoundation.org page [13:39] The Spanning Tree Protocol has no authentication; all participants are assumed to be trustworthy and correct. This assumption is not true if bridging between a hostile environment like the Internet and a private network. For this reason, STP is turned off by default on the recent versions of Linux. [13:39] dooferlad: how does that apply here? we're not bridging hostile networks [13:40] dooferlad: all of them are managed by maas [13:40] dimitern: a Juju deployed node could have a public interface [13:40] dooferlad: on MAAS? [13:40] dimitern: why not? [13:41] dooferlad: not the typical case [13:41] dimitern: a MAAS network could have some machines in a DMZ that are just open to the net. There is no reason why not. [13:41] dooferlad: I'm not saying it's impossible, but your maas is likely firewalled deep inside your network [13:42] dimitern: I don't really care about typical for MAAS. I care about changing a default that is that way for a good reason. Doing something unexpected that can result in security problems is bad. [13:44] dimitern: I think it is perfectly reasonable to schedule work to turn STP on if a user asks for it. I think it being a space or subnet wide setting would be a good fit. [13:45] dooferlad: pretty much every guide on networking I've read strongly recommends enabling STP especially in complex setups like with MAAS where it's quite easy to have multiple redundant links across [13:45] dimitern: fine, so lets do the right thing please? Not just turn it on everywhere and hope we haven't screwed someone. [13:47] dooferlad: oh alright [13:48] dooferlad: in the interest of getting something else done, sure [14:13] Bug #1589890 opened: juju2 azure fail with error 409 network conflict

=== redir_afk is now known as redir_voting [14:47] dooferlad: any issues btw? [14:48] dimitern: sorry, I thought you were updating your patch. [14:48] dooferlad: oh, sorry I wasn't sure it was needed for that test [14:49] dimitern: since all your bridge script changes are going away I don't think a bond will make any difference for what it's worth. [14:49] dooferlad: but ok, will push an update dropping stp on and maxwait 0 [14:49] dimitern: unless I missed something! [14:49] dooferlad: no, not all [14:49] dooferlad: only the extra bridge settings [14:50] dimitern: in that case I would get that change landed if it works for you. The bond won't make any difference. [14:50] dooferlad: without testing it at least once? [14:51] dimitern: 'if it works for you' implied you could at least do a little test :-) [14:52] dooferlad: well, not with bonds :) [14:52] dimitern: yea, it won't care [14:52] dimitern: it is a tube with bits going in and out of it :-) [14:52] dooferlad: but otherwise I've done enough tests to be reasonably certain it works well [14:53] dimitern: in the container, why eth0 being special? [14:53] dooferlad: the thing is, I didn't change how we bridge bond slaves (e.g. removing duplicated IPs from there and leaving the on the bridge) [14:54] dimitern: MAAS only gives an address to one interface IIRC [14:54] dimitern: from the containers POV (which this change is about) it is just a connection to a network. [14:55] dooferlad: it's not special, just the first one is connected to the bridge which took over the NIC on the default route [14:56] dimitern: so eth0 is always connected to that has the default route on it? [14:56] dooferlad: true, having more than 1 gateway rendered in /e/n/i doesn't work [14:57] dimitern: I was more thinking about eth1 having the default route on the host. [14:57] dooferlad, dimitern: I just asked larry for more details regarding this ^^ [14:57] dooferlad: "eth0" can be connected to e.g. br-eth3 on the hosty [14:58] dooferlad: the names don't have to match [14:58] dooferlad, dimitern: I wanted to clarify his setup so that we could reproduce and confirm the bug fixes that [14:58] frobware: ok, it's good to know that [14:58] dimitern: I would be tempted to copy /etc/network/interfaces from the host to the guest and change IP and MAC addresses to match the container config. If guest-ethx is always connected to host-br-ethx that would work. [14:59] frobware: have you tried rebooting a machine with deployed lxd containers? [14:59] dimitern: that sounds ominuous [14:59] frobware: I've just discovered none of the lxds come up [14:59] oh la la [14:59] frobware: they seem to be not set to auto-start on boot, as starting them manually otherwise works fine [15:00] dimitern: lxc config show -- should show autostart state [15:00] * dooferlad goes to get a cup of tea and spend a few minutes not thinking about routes. [15:01] frobware: it does say `user.boot.autostart: "true"` [15:01] hmm.. looking at the logs [15:03] dimitern: what's the state of your patch w.r.t. backing out bridge script changes? I see stuff in there for the stp and explicitly matching "vlan_id" [15:05] voidspace: ping? [15:05] babbageclunk: pong [15:06] voidspace: if a juju deploy failed because the image wasn't available in my maas, how can I ask it to try again now that the image is available? [15:06] frobware: as discussed with dooferlad, I'm dropping the added stp on and maxwait 0 options [15:06] babbageclunk: just try again with the deploy command? [15:06] voidspace: I tried retry-provisioning, but it doesn't seem to have done anything. [15:06] babbageclunk: I'm assuming you tried that and it didn't work [15:07] frobware: and the 2 rm calls for eth0.cfg and 50-cloud-init.. [15:07] babbageclunk: destroy-service (or application) first? [15:07] voidspace: ok, just destroy the applications and the deploy again? [15:07] frobware: and I think we should land the rest [15:07] babbageclunk: I would *expect* that to work [15:08] frobware: `error: open /var/lib/lxd/containers: no such file or directory` << in /v/l/syslog on one of the hosts.. [15:09] voidspace: sweet, seems like it. [15:16] alexisb: just verifying... is the lxc to lxd work higher priority than that ipaddress borked bug? https://bugs.launchpad.net/juju-core/+bug/1537585 Ian gave me the impression I should work on the lxc to lxd stuff instead of that bug. [15:16] Bug #1537585: machine agent failed to register IP addresses, borks agent <2.0-count>

=== redir_voting is now known as redir [15:24] frobware: ping [15:31] Bug #1590045 opened: Uniter could not recover from failed juju run [15:37] dimitern: pong [15:37] frobware: I've updated the PR as agreed and set it to merge [15:38] (it's not picked up yet for some reason though) [15:39] dimitern: oh. because I wanted to talk about matching on vlan_id [15:39] frobware: there's no such setting 'vlan_id' [15:39] dimitern: so vlan_is no longer propgated to the bridge device, correct? [15:40] frobware: see vlan-interfaces(5) [15:40] frobware: correct - it should've been there to begin with [15:40] frobware: only vlan-raw-device is needed [15:40]