[01:01] wallyworld: I'm ready whenever you are [01:02] axw: ok, just finishing an email [01:11] axw: in 1:1 now [01:48] ericsnow: it occurs to me that for local charms, all we have at deploy time is all we'll ever have for info on resources. [02:52] wallyworld: I should be able to do the aliases immediately after controller-rename lands. [02:52] cherylj: :-) tyvm [02:53] I think there would be changes needed if I did it earlier. I'd need to create aliases for different commands [02:54] Bug #1532063 opened: Unit seems unable to proceed after watcher died in the middle of hook execution [02:55] cherylj: the controller rename is othogonal to the environment->model changes [02:55] the controler rename is state server / jes stuff -> controller [02:55] the other is environment -> model [02:56] unless i'm missing something [02:56] yes, but the controller commands use "environment" in them [02:56] ah ok [02:57] axw: we can ignore bug 1531886 in getting sm to land. that issue comes from master and doesn't need fixing in the feature beanch [02:57] Bug #1531886: Status fails during upgrade

[02:57] Bug #1532063 changed: Unit seems unable to proceed after watcher died in the middle of hook execution [02:58] wallyworld: ok [02:58] it's the unit tests one that needs doing [02:58] the tools not found [02:58] wallyworld: the BootstrapImage unit test fix is up for review [02:58] cool, looking [02:59] axw: i won't make team meeting, got things to finish, out of here soon [02:59] sure, safe travels [03:04] menn0 waigani_: I think it's just us here, I vote we cancel the team meeting [03:04] axw: fine by me. [03:04] axw: okay [03:05] sold [03:06] Bug #1532063 opened: Unit seems unable to proceed after watcher died in the middle of hook execution [03:25] gah, sorry I missed the team meeting... [03:26] although sounds like me making it probably wouldn't have changed the outcome [04:19] axw: can I ask your opinion on some upgrade-juju behavior? [04:20] cherylj: certainly [04:20] axw: I'm trying to write the changes to allow a 2.0 client to upgrade a 1.25.2 env [04:20] and I'm trying to make things generic [04:20] so that if there comes a time for there to be a 3.0 we don't run into this again [04:20] we can just keep a map of major versions to the minimum version to upgrade from [04:21] and reference that when we do our upgrade checks [04:21] * axw nods [04:21] but the problem I'm running into is we allow people using --upload-tools to put in whatever for the version [04:21] so we could have --upload-tools --version 4.4.4 [04:21] and that would pretend that we're at 4.4.4 [04:21] and we obviously don't have "4" in this map [04:21] so [04:22] either we don't check minimum levels for --upload-tools, or we don't allow those shenanigans [04:23] any thoughts? [04:23] cherylj: I think we should just reject the tools if they're beyond the latest major version [04:23] axw: yay, thanks! [04:31] cherylj: why do we care if people pretend they're on 4.4 if they use --upload-tools? [04:33] natefinch: I thought I explained it ok in the statements above. Was it not clear? [04:33] cherylj: why would we stop anyone from upgrading a 1.25.2 environment? [04:34] natefinch: the idea was not to stop them, but to modify the 2.0 upgrade-juju command to actually do it. [04:35] but we need to enforce that people upgrading to 2.0 are coming from 1.25.2 [04:35] or later [04:35] ahh, that's the trick [04:35] I would hope we could come up with a way for 2.0 that we'd never need such a restriction again [04:36] natefinch: I'm trying to make things generic, so we can just keep a map of major: minimum version to get there [04:36] like 2: 1.25.2 [04:36] and update that when we hit 3, etc [04:36] I thought the reason you had to be on 1.25.2 to upgrade was because previous versions just wouldn't let you upgrade to a higher major version [04:36] if we just remove that code... then it should be fine [04:37] but I'm probably missing something :) [04:37] natefinch: no, we want people on 1.25.2 because we're removing upgrade steps to *get* there from 2.0 [04:37] so 2.0 won't know what the upgrade steps are for 1.24.7 -> 1.25.2 [04:38] I am going on the assumption that future major releases will operate the same way where we want a known place where users are coming from. [04:38] this "minimum upgrade version", like 1.25.2 is to 2.0 [04:39] I guess I really hoped we could prevent people from having to upgrade twice [04:39] it's even more fun when you go past 2.0! [04:39] The decision was that you can't go from 1.25.2 -> 2.1 [04:40] has to be 1.25.2 -> 2.0.* -> 2.1 [04:40] 2.1 won't have *any* 1.x upgrade steps in it [04:41] here's my ideal scenario, though it's probably more complicated than it's worth: you run upgrade with 2.3 on a 1.17 env, 2.3 only knows how to run an upgrade from 2.0, so it downloads 2.0, and tells it to upgrade first... 2.0 only knows how to upgrade from 1.25.2, so it downloads the 1.25.2 tools, and tells them to upgrade the environment, which works, and the rest plays out. [04:41] would be nice if we had the resources to put to that [04:42] yeah.... I guess we'll have to wait until enough people complain :) [04:43] no, no, that doesn't mean we need more people to do this work. It's just means we're all shit at our jobs if we can't get it all done, right? [04:43] sorry, that was a bit too pessimistic :) [04:44] my team lost 25-33% of its developer resources and no one's changing our delivery date, which was already like a month earlier than our estimates think is likely. So... yeah, I think you were right. [04:44] it's late, and dave cheney's not online to play grumpy dev, so someone's gotta pick up the slack [04:44] hahaha, nice! [04:45] heh [04:47] There's a station on google play music called "Code Your Face Off" which is pretty awesome, btw (and not just because of the name). The description is "Work your magic while listening to these motivational electro-bangers -- and stop slacking off!" https://play.google.com/music/r/m/Ltnhqsx7baik6dlsft6wdfchdly?t=Code_Your_Face_Off [04:50] that sounds painful [04:53] unrelated - anyone know if canonical pays for any of the travel to gophercon, or is it all out of pocket? Convincing my wife to let me go away for ~4 days would be easier if it weren't also ~$1000 :) [05:00] natefinch: are you talking about for next year? [05:00] wait [05:00] this year [05:00] haha [05:01] in Denver [05:01] haha [05:01] yes [05:01] alexisb-afk: was going to work on getting sponsorship again [05:01] I' [05:01] so I'd let her know you're interested [05:01] will do [05:02] I'd love to go, last year was impossible because we already had a vacation booked that week. Year before was impossible because we had a 2 week old. [05:02] (baby) [05:02] I figured :) [05:02] :) [05:03] aight, it's time for bed for me. see you guys tomorrow [05:03] g'night [05:55] Bug #1532085 opened: Guarantee leadership revoked from departing unit before departed hooks run [06:04] Bug #1532085 changed: Guarantee leadership revoked from departing unit before departed hooks run [06:07] Bug #1532085 opened: Guarantee leadership revoked from departing unit before departed hooks run [09:43] Bug #1532130 opened: Config item 'version' vanishes with 1.26

[09:46] Bug #1532130 changed: Config item 'version' vanishes with 1.26

[09:52] Bug #1532130 opened: Config item 'version' vanishes with 1.26

=== dooferlad_ is now known as dooferlad [10:24] dimitern, hey - I'm having trouble with the MAAS spaces branch and reproducing thedac's deployments from yesterday [10:25] dimitern, I can't bootstrap an environment - the machines deploy ok, but when juju configures all of the bridges post deployment, it completely hoses the network configuration on the server... [10:25] :( [10:26] dimitern, help! [10:27] jamespage: we are in our standup right now. I am sure he will respond soon. [10:27] dooferlad, ack [10:27] dimitern, http://paste.ubuntu.com/14436922/ [10:32] jamespage, uh oh .. that looks like the addresses are missing [10:32] dimitern, uh-oh was my reaction [10:32] jamespage, would you mind joining our standup to discuss this? [10:32] dimitern, sure [10:33] https://plus.google.com/hangouts/_/canonical.com/sapphire [10:33] jamespage: ^^ [10:33] jamespage, can you paste the original /e/n/i -- juju creates a copy in /etc/network/ [10:33] http://paste.ubuntu.com/14436933/ === frobware_ is now known as frobware [10:33] frobware, dimitern ^^ [10:34] dooferlad, can you direct invite me from the hangout - still using iphone atm [10:37] http://paste.ubuntu.com/14436967/ [10:57] dooferlad, sorry my iphone drops me from g+ now as well :( [10:58] thedac, when you awake ^^ [11:09] jamespage, will fix this in maas-spaces branch [11:10] dimitern, can you raise the bug for the /e/n/i issue on 1.25 please [11:10] dimitern, and if you have an /e/n/i that is bust can you attach it to the bug. thx. [11:14] frobware, ack - I think I can hack the add-bridges script for now to workaround [11:14] trying that now [11:17] jamespage, http://pastebin.ubuntu.com/14437158/ [11:19] frobware, I actuall specialized to dns-search [11:20] jamespage, reading resolvconf and interfaces man pages indicates that dns- is not the beginning of any stanza, just options on the iface. [11:21] frobware, so infact those last two get appended to the last interface stanza right? [11:21] jamespage, yep [11:23] jamespage, otherwise you'll run into the same issue on dns-search - the script will think this a top-level stanza and break as it currently does. [11:27] frobware, hmm but dns-search is not part of the iface stanza's as generated right now [11:27] that might not matter if curtin also generates resolv.conf [11:29] jamespage, the combination of ifup running resolvconf which plucks the dns-search from /e/n/i should end up in resolv.conf [11:34] jamespage, I would still like to understand how/when/where the change to /e/n/i happened. Quite a few of the unit tests I have were cut+paste /e/n/i files from a deployed MAAS 1.9 node. [11:35] frobware, so would I - I also think the /e/n/i generated by maas/curtin is a little malformed right? [11:35] frobware, will do [11:36] jamespage, the indentation and the separation of the dns-* entries confuse matters. [11:39] * jamespage crosses fingers for network access [11:47] frobware, ok better - I'm only expecting a bridge for eth0 right now? [11:47] jamespage, how many interfaces do you have? You should get a bridge per active NIC now. [11:48] frobware, eth0 is the only active physical nic - all others are vlans on eth1 [11:48] jamespage, can you paste your original /e/n/i [11:52] frobware, I've filed bug 1532167 [11:52] Bug #1532167: maas bridge script handles VLAN NICs incorrectly

[11:52] dimitern, thanks! [11:54] dimitern, dooferlad, voidspace: should we propagate dns-* from the original iface to the new bridge stanza? [11:55] frobware: yes [11:55] frobware, yeah - because the bridge is taking over the nic and all its options, just adding the bridge-utils ones to them [11:56] dimitern, dooferlad: which is what I am/was doing, just confirming. [11:56] frobware: great :-) [11:57] dooferlad, dimitern: think this only holds true for bonds. e.g., http://pastebin.ubuntu.com/ [11:57] http://pastebin.ubuntu.com/14437347/ [11:59] frobware: that seems reasonable, but I don't know for sure. Since it is a rule that says "add this nameserver to resolv.conf when I am activated" as long as the rule will run under the same conditions, that is fine. [12:00] dooferlad, otherwise we have to special case which options we carry forward. Which is not a big deal because we already do it for the bond anyway. [12:04] dimitern, actually for the bond it doesn't take all its options. I deliberately omit anything starting with bond. [12:04] jamespage, any joy? [12:05] frobware, yeah [12:05] Bug #1532167 opened: maas bridge script handles VLAN NICs incorrectly

[12:05] frobware, that's because we verified what needs to stay for the bond-bridge(-vlan) device to still work [12:06] dimitern, I'll actively try leaving them in. I think the fewer special cases the better. [12:06] frobware, dimitern - ok I have a bound ceph deployment running! [12:06] w00t! [12:07] dimitern, the new iface (i..e, the bridge) is not a bond so I didn't carry those options forward. [12:07] jamespage, excellent! [12:08] frobware, ah, right [12:08] dimitern, these are all just arbitrary rules. :-) [12:11] Bug #1532167 changed: maas bridge script handles VLAN NICs incorrectly

[12:14] Bug #1532167 opened: maas bridge script handles VLAN NICs incorrectly

[12:23] frobware, also filed bug 1532176 [12:23] Bug #1532176: maas bridge script in maas-spaces feature branch handles dns-* options incorrectly

[12:24] frobware, added the expected /e/n/i in jamespage's case (AIUI) [12:24] jamespage, awesome! [12:25] dimitern, do we want to raise bugs for feature branches? [12:25] frobware, we can - whether we should is another thing :) [12:26] dimitern, seems overkill to me [12:26] frobware, but since I started composing it, esp. took care with the expected config, seemed good to have it filed [12:29] but I might've just as well added a card [12:35] dimitern: ping [12:37] perrito666, hey there [12:38] hey, ill privmsg you [13:17] Bug #1532186 opened: lxd container creation fails semi - regularly

[13:26] Bug #1532186 changed: lxd container creation fails semi - regularly

[13:35] Bug #1532186 opened: lxd container creation fails semi - regularly

[13:55] dimitern, dooferlad, voidspace: http://reviews.vapour.ws/r/3481/ - network/interfaces fix for demo [13:57] frobware, looking [14:02] frobware, LGTM with an extra test for the jamespage's case [14:11] dimitern, done and thanks for the review [14:12] frobware, cheers [14:14] frobware, hmm wait a moment.. [14:14] dimitern, oh... [14:14] frobware, looking at the test now [14:14] frobware, why auto eth1.2667 and the others still remain? [14:14] frobware, also I can only see one bridge [14:15] dimitern, I think we've danced this dance before. [14:15] frobware, either the expected is wrong or the script doesn't quite handle that as I'd have expected [14:15] dimitern, the underlying device is not active. [14:16] dimitern, I believe we previously said inactive devices don't get bridged. [14:16] dimitern, however, what should the behaviour be... [14:16] frobware, but it's auto and it has auto children which are active [14:17] dimitern, so the current test is... is the underlying vlan-raw-device eth1 active? [14:17] frobware, if this is the /e/n/i we'll use, how will multi-NIC containers work on that host node? [14:17] dimitern, they obviously won't... :( [14:17] frobware, well, it looks active - "auto eth1", despite being "manual" [14:17] it will be up when running ip link show [14:18] otherwise the vlans won't work either [14:18] dimitern, so I think we should nail down what we consider an active interface [14:18] dimitern, btw, I don't believe this is a demo problem. [14:19] frobware, yeah, it's not but only because we're not using containers [14:19] frobware, and don't get me wrong - your fix is till valid [14:19] dimitern, but let's fix containers whilst we're here. [14:19] frobware, but for the multi-NIC containers we'll need to fix that (for the os demo) [14:20] dimitern, because some of this could go back into 1.25 too [14:20] frobware, I think "active" should mean either the NIC itself is up and has address (can send/receive packets) or any of it children are active [14:21] dimitern, let's let the fix for the demo land and do another for multi-NICs [14:21] frobware, sgtm [14:21] dimitern, so it's the "and has an address" - this is where we have stumbled before. [14:21] dimitern, the eth1 in that test has no address. [14:22] frobware, yeah, but it has dependents which have [14:22] dimitern, our parsing is too simplistic [14:22] frobware, eth1 just needs to be up [14:22] dimitern, we don't really model it - we just do some pretty high-level text transformations. [14:22] frobware, yeah - one pass only [14:23] dimitern, let me work on this now... [14:24] frobware, we need to get a simple model from the /e/n/i with a dict of NIC names to NIC info which includes an "active" flag and any children - similarly [14:25] well, it can also include all the options so you don't have to re-parse them when rendering [14:26] dimitern, parsing is so cheap though. [14:26] dimitern, open a file, read line by line. [14:30] frobware, just tried with the initial e/n/i, but changed eth1 to dhcp instead of manual and added "a" to each dns-nameservers options except the last one and this works better:http://paste.ubuntu.com/14438110/ [14:30] frobware, what was the issue with just just always creating a bridge for each iface? [14:31] bonds I presume.. or boot slowdowns? [14:31] dimitern, heh. so that's an alternative. [14:31] dimitern, not sure there is one. [14:32] dimitern, I was trying to cater for the case where it is unconfigured. [14:32] dimitern, if it is unconfigured and we create a bridge, what happens for the VIP? [14:34] frobware, well, it needs to have "auto" at least perhaps? [14:34] frobware, I don't mind creating an extra bridge we'll not use, unless that causes problems [14:35] frobware, the VIP is not involved I think [14:35] frobware,do you mean the VIPs added by corosync ? [14:36] frobware, with or without a bridge that will still work - adding an extra IP to an interface that's not up - but it simply won't pass any traffic [14:36] dimitern, it shouldn't matter. I was thinking of the case where we want to give back a device [14:37] dimitern, the dellstack case --> http://pastebin.ubuntu.com/14438169/ [14:37] frobware, in that case we'll see that e.g. eth3 on node1 is linked to a subnet but in "link_up" state only, and that's what we'll give neutron-gateway [14:38] (well, once network-get returns more than an address, but the full NIC info) [14:38] dimitern, what did you think of ignoring is_active and just bridge the lot?! [14:39] frobware, I think we should try as the simplest thing we can do for multi-NIC containers [14:40] frobware, and seriously test it in an automated fashion with permutations of different NIC configs on different MAAS versions [14:40] dimitern, my hack should you want this: http://pastebin.ubuntu.com/14438191/ [14:40] frobware, cheers [14:41] frobware, I'll come up with something next week to automate setting up lxc-based maas of a given revision with a given set of networks [14:41] dimitern, there's a certain amount of orthogonality that comes with that [14:42] frobware, yeah, but we need to at least make sure it works with bonds and vlans (and vlans onto a bond) [14:43] dimitern, there are unit tests for that [14:43] the latter case is common in telcos [14:44] frobware, unit testing what we generate is a must, but live testing whether it will work (and didn't break across releases) is also needed when we can get it to work [14:44] (get the automated testing to work I mean) [14:44] dimitern, sure. I tried the vlans onto a bond a little while ago, but not with all the world bridged. [14:46] dimitern, but from a parsing and creating bridges POV I believe we have some of these permutations. what happens in the wild... stays in the wild. :-) [14:49] frobware, yeah, we'll just keep updating those tests as we discover issues [14:50] dooferlad, I just dumped 1531932 your way. Happy to answer questions here, in the card or wherever works. [14:51] frobware: ok [14:51] I assume that is bug #1531932 [14:51] Bug #1531932: Unit agents unable to connect with MAAS 1.8 due to lack of default gateway in MAAS [14:53] dooferlad, yep. Before weds next week we need to land that, #1525280. And port #1528217 to master. [14:53] Bug #1525280: 1.25.1 with maas 1.8: devices dns allocation uses non-unique hostname

[14:53] Bug #1528217: 1.25.2 doesn't set up DNS information with MAAS

[14:55] dooferlad, to tickle the lack of gateway in 1.8 you just need to remove any entry you may have in MAAS already in the network (in my case maas-eth0). [15:00] dimitern, if we don't have uniques hostnames on master I'm guessing we don't run into either lack of default gw and/or lack of DNS info - correct? It looks like that patch switches the provisioner to use configType.static. [15:02] dimitern, not sure if you saw that ^^ - formatting looks odd in my client [15:04] frobware, we don't have unique hostnames, but we still use static and I don't remember whether the dns and gateways were discovered when missing [15:10] dimitern, just trying to see what's the easiest environment to initially fix this in. 1.25 is the best place today as we know it fails there. [15:10] dooferlad, ^^ [15:12] frobware, well, 1.25 can be fixed after the revert happens so I guess we should fix dns + gateway on master and also unique hostnames first [15:12] dimitern, the reason why I'm asking is ... why doesn't it fail today in the CI tests. [15:13] dimitern, if there's a default gw entry, then it will be fine. but it popped up in CI because there was no entry. [15:13] frobware, because there are hardly any serious tests that use containers and/or multiple envs bootstrapped concurrently [15:14] dimitern, well 1.25.2 tripped over it as soon as we fixed dns [15:14] dimitern, so there are some [15:14] frobware, and for the dns / GW - I don't believe any CI tests (except likely for dooferlad's) check containers networking accessibility etc. [15:14] dimitern, anyway, I'm not suggesting we fix in 1.25.2 first, just that's a playground you can use today because it fails there atm [15:15] frobware, well, they started passing for the first time I guess :) [15:15] frobware, right, I see [15:16] dimitern, was trying to lower dooferlad's entry point... :) [15:20] Bug #1528259 changed: undertakerSuite.TestAPICalls nil deref panic

[15:20] Bug #1532232 opened: maas 1.9 container networking broken

[15:21] dimitern, ^^ LOL. sigh. [15:21] * frobware takes up smoking... [15:26] Bug #1532232 changed: maas 1.9 container networking broken

[15:26] Bug #1528259 opened: undertakerSuite.TestAPICalls nil deref panic

[15:26] :)( [15:32] sinzui, dimitern, jog: when I looked at John's email earlier today he indicated that it had worked and the failure may be intermittent. Has that changed? [15:33]