| axw | wallyworld: I'm ready whenever you are | 01:01 |
|---|---|---|
| wallyworld | axw: ok, just finishing an email | 01:02 |
| wallyworld | axw: in 1:1 now | 01:11 |
| natefinch | ericsnow: it occurs to me that for local charms, all we have at deploy time is all we'll ever have for info on resources. | 01:48 |
| cherylj | wallyworld: I should be able to do the aliases immediately after controller-rename lands. | 02:52 |
| wallyworld | cherylj: :-) tyvm | 02:52 |
| cherylj | I think there would be changes needed if I did it earlier. I'd need to create aliases for different commands | 02:53 |
| mup | Bug #1532063 opened: Unit seems unable to proceed after watcher died in the middle of hook execution <juju-core:New> <https://launchpad.net/bugs/1532063> | 02:54 |
| wallyworld | cherylj: the controller rename is othogonal to the environment->model changes | 02:55 |
| wallyworld | the controler rename is state server / jes stuff -> controller | 02:55 |
| wallyworld | the other is environment -> model | 02:55 |
| wallyworld | unless i'm missing something | 02:56 |
| cherylj | yes, but the controller commands use "environment" in them | 02:56 |
| wallyworld | ah ok | 02:56 |
| wallyworld | axw: we can ignore bug 1531886 in getting sm to land. that issue comes from master and doesn't need fixing in the feature beanch | 02:57 |
| mup | Bug #1531886: Status fails during upgrade <upgrade-juju> <juju-core:Triaged> <juju-core structured-metadata:In Progress by axwalk> <https://launchpad.net/bugs/1531886> | 02:57 |
| mup | Bug #1532063 changed: Unit seems unable to proceed after watcher died in the middle of hook execution <juju-core:New> <https://launchpad.net/bugs/1532063> | 02:57 |
| axw | wallyworld: ok | 02:58 |
| wallyworld | it's the unit tests one that needs doing | 02:58 |
| wallyworld | the tools not found | 02:58 |
| axw | wallyworld: the BootstrapImage unit test fix is up for review | 02:58 |
| wallyworld | cool, looking | 02:58 |
| wallyworld | axw: i won't make team meeting, got things to finish, out of here soon | 02:59 |
| axw | sure, safe travels | 02:59 |
| axw | menn0 waigani_: I think it's just us here, I vote we cancel the team meeting | 03:04 |
| menn0 | axw: fine by me. | 03:04 |
| waigani_ | axw: okay | 03:04 |
| axw | sold | 03:05 |
| mup | Bug #1532063 opened: Unit seems unable to proceed after watcher died in the middle of hook execution <juju-core:New> <https://launchpad.net/bugs/1532063> | 03:06 |
| natefinch | gah, sorry I missed the team meeting... | 03:25 |
| natefinch | although sounds like me making it probably wouldn't have changed the outcome | 03:26 |
| cherylj | axw: can I ask your opinion on some upgrade-juju behavior? | 04:19 |
| axw | cherylj: certainly | 04:20 |
| cherylj | axw: I'm trying to write the changes to allow a 2.0 client to upgrade a 1.25.2 env | 04:20 |
| cherylj | and I'm trying to make things generic | 04:20 |
| cherylj | so that if there comes a time for there to be a 3.0 we don't run into this again | 04:20 |
| cherylj | we can just keep a map of major versions to the minimum version to upgrade from | 04:20 |
| cherylj | and reference that when we do our upgrade checks | 04:21 |
| * axw nods | 04:21 | |
| cherylj | but the problem I'm running into is we allow people using --upload-tools to put in whatever for the version | 04:21 |
| cherylj | so we could have --upload-tools --version 4.4.4 | 04:21 |
| cherylj | and that would pretend that we're at 4.4.4 | 04:21 |
| cherylj | and we obviously don't have "4" in this map | 04:21 |
| cherylj | so | 04:21 |
| cherylj | either we don't check minimum levels for --upload-tools, or we don't allow those shenanigans | 04:22 |
| cherylj | any thoughts? | 04:23 |
| axw | cherylj: I think we should just reject the tools if they're beyond the latest major version | 04:23 |
| cherylj | axw: yay, thanks! | 04:23 |
| natefinch | cherylj: why do we care if people pretend they're on 4.4 if they use --upload-tools? | 04:31 |
| cherylj | natefinch: I thought I explained it ok in the statements above. Was it not clear? | 04:33 |
| natefinch | cherylj: why would we stop anyone from upgrading a 1.25.2 environment? | 04:33 |
| cherylj | natefinch: the idea was not to stop them, but to modify the 2.0 upgrade-juju command to actually do it. | 04:34 |
| cherylj | but we need to enforce that people upgrading to 2.0 are coming from 1.25.2 | 04:35 |
| cherylj | or later | 04:35 |
| natefinch | ahh, that's the trick | 04:35 |
| natefinch | I would hope we could come up with a way for 2.0 that we'd never need such a restriction again | 04:35 |
| cherylj | natefinch: I'm trying to make things generic, so we can just keep a map of major: minimum version to get there | 04:36 |
| cherylj | like 2: 1.25.2 | 04:36 |
| cherylj | and update that when we hit 3, etc | 04:36 |
| natefinch | I thought the reason you had to be on 1.25.2 to upgrade was because previous versions just wouldn't let you upgrade to a higher major version | 04:36 |
| natefinch | if we just remove that code... then it should be fine | 04:36 |
| natefinch | but I'm probably missing something :) | 04:37 |
| cherylj | natefinch: no, we want people on 1.25.2 because we're removing upgrade steps to *get* there from 2.0 | 04:37 |
| cherylj | so 2.0 won't know what the upgrade steps are for 1.24.7 -> 1.25.2 | 04:37 |
| cherylj | I am going on the assumption that future major releases will operate the same way where we want a known place where users are coming from. | 04:38 |
| cherylj | this "minimum upgrade version", like 1.25.2 is to 2.0 | 04:38 |
| natefinch | I guess I really hoped we could prevent people from having to upgrade twice | 04:39 |
| cherylj | it's even more fun when you go past 2.0! | 04:39 |
| cherylj | The decision was that you can't go from 1.25.2 -> 2.1 | 04:39 |
| cherylj | has to be 1.25.2 -> 2.0.* -> 2.1 | 04:40 |
| cherylj | 2.1 won't have *any* 1.x upgrade steps in it | 04:40 |
| natefinch | here's my ideal scenario, though it's probably more complicated than it's worth: you run upgrade with 2.3 on a 1.17 env, 2.3 only knows how to run an upgrade from 2.0, so it downloads 2.0, and tells it to upgrade first... 2.0 only knows how to upgrade from 1.25.2, so it downloads the 1.25.2 tools, and tells them to upgrade the environment, which works, and the rest plays out. | 04:41 |
| cherylj | would be nice if we had the resources to put to that | 04:41 |
| natefinch | yeah.... I guess we'll have to wait until enough people complain :) | 04:42 |
| cherylj | no, no, that doesn't mean we need more people to do this work. It's just means we're all shit at our jobs if we can't get it all done, right? | 04:43 |
| cherylj | sorry, that was a bit too pessimistic :) | 04:43 |
| natefinch | my team lost 25-33% of its developer resources and no one's changing our delivery date, which was already like a month earlier than our estimates think is likely. So... yeah, I think you were right. | 04:44 |
| natefinch | it's late, and dave cheney's not online to play grumpy dev, so someone's gotta pick up the slack | 04:44 |
| cherylj | hahaha, nice! | 04:44 |
| waigani_ | heh | 04:45 |
| natefinch | There's a station on google play music called "Code Your Face Off" which is pretty awesome, btw (and not just because of the name). The description is "Work your magic while listening to these motivational electro-bangers -- and stop slacking off!" https://play.google.com/music/r/m/Ltnhqsx7baik6dlsft6wdfchdly?t=Code_Your_Face_Off | 04:47 |
| cherylj | that sounds painful | 04:50 |
| natefinch | unrelated - anyone know if canonical pays for any of the travel to gophercon, or is it all out of pocket? Convincing my wife to let me go away for ~4 days would be easier if it weren't also ~$1000 :) | 04:53 |
| cherylj | natefinch: are you talking about for next year? | 05:00 |
| cherylj | wait | 05:00 |
| cherylj | this year | 05:00 |
| cherylj | haha | 05:00 |
| cherylj | in Denver | 05:01 |
| natefinch | haha | 05:01 |
| natefinch | yes | 05:01 |
| cherylj | alexisb-afk: was going to work on getting sponsorship again | 05:01 |
| natefinch | I' | 05:01 |
| cherylj | so I'd let her know you're interested | 05:01 |
| natefinch | will do | 05:01 |
| natefinch | I'd love to go, last year was impossible because we already had a vacation booked that week. Year before was impossible because we had a 2 week old. | 05:02 |
| natefinch | (baby) | 05:02 |
| cherylj | I figured :) | 05:02 |
| natefinch | :) | 05:02 |
| cherylj | aight, it's time for bed for me. see you guys tomorrow | 05:03 |
| natefinch | g'night | 05:03 |
| mup | Bug #1532085 opened: Guarantee leadership revoked from departing unit before departed hooks run <juju-core:New> <https://launchpad.net/bugs/1532085> | 05:55 |
| mup | Bug #1532085 changed: Guarantee leadership revoked from departing unit before departed hooks run <juju-core:New> <https://launchpad.net/bugs/1532085> | 06:04 |
| mup | Bug #1532085 opened: Guarantee leadership revoked from departing unit before departed hooks run <juju-core:New> <https://launchpad.net/bugs/1532085> | 06:07 |
| mup | Bug #1532130 opened: Config item 'version' vanishes with 1.26 <regression> <juju-core:New> <https://launchpad.net/bugs/1532130> | 09:43 |
| mup | Bug #1532130 changed: Config item 'version' vanishes with 1.26 <regression> <juju-core:New> <https://launchpad.net/bugs/1532130> | 09:46 |
| mup | Bug #1532130 opened: Config item 'version' vanishes with 1.26 <regression> <juju-core:New> <https://launchpad.net/bugs/1532130> | 09:52 |
| === dooferlad_ is now known as dooferlad | ||
| jamespage | dimitern, hey - I'm having trouble with the MAAS spaces branch and reproducing thedac's deployments from yesterday | 10:24 |
| jamespage | dimitern, I can't bootstrap an environment - the machines deploy ok, but when juju configures all of the bridges post deployment, it completely hoses the network configuration on the server... | 10:25 |
| jamespage | :( | 10:25 |
| jamespage | dimitern, help! | 10:26 |
| dooferlad | jamespage: we are in our standup right now. I am sure he will respond soon. | 10:27 |
| jamespage | dooferlad, ack | 10:27 |
| jamespage | dimitern, http://paste.ubuntu.com/14436922/ | 10:27 |
| dimitern | jamespage, uh oh .. that looks like the addresses are missing | 10:32 |
| jamespage | dimitern, uh-oh was my reaction | 10:32 |
| dimitern | jamespage, would you mind joining our standup to discuss this? | 10:32 |
| jamespage | dimitern, sure | 10:32 |
| dooferlad | https://plus.google.com/hangouts/_/canonical.com/sapphire | 10:33 |
| dooferlad | jamespage: ^^ | 10:33 |
| frobware_ | jamespage, can you paste the original /e/n/i -- juju creates a copy in /etc/network/ | 10:33 |
| jamespage | http://paste.ubuntu.com/14436933/ | 10:33 |
| === frobware_ is now known as frobware | ||
| jamespage | frobware, dimitern ^^ | 10:33 |
| jamespage | dooferlad, can you direct invite me from the hangout - still using iphone atm | 10:34 |
| jamespage | http://paste.ubuntu.com/14436967/ | 10:37 |
| jamespage | dooferlad, sorry my iphone drops me from g+ now as well :( | 10:57 |
| jamespage | thedac, when you awake ^^ | 10:58 |
| frobware | jamespage, will fix this in maas-spaces branch | 11:09 |
| frobware | dimitern, can you raise the bug for the /e/n/i issue on 1.25 please | 11:10 |
| frobware | dimitern, and if you have an /e/n/i that is bust can you attach it to the bug. thx. | 11:10 |
| jamespage | frobware, ack - I think I can hack the add-bridges script for now to workaround | 11:14 |
| jamespage | trying that now | 11:14 |
| frobware | jamespage, http://pastebin.ubuntu.com/14437158/ | 11:17 |
| jamespage | frobware, I actuall specialized to dns-search | 11:19 |
| frobware | jamespage, reading resolvconf and interfaces man pages indicates that dns- is not the beginning of any stanza, just options on the iface. | 11:20 |
| jamespage | frobware, so infact those last two get appended to the last interface stanza right? | 11:21 |
| frobware | jamespage, yep | 11:21 |
| frobware | jamespage, otherwise you'll run into the same issue on dns-search - the script will think this a top-level stanza and break as it currently does. | 11:23 |
| jamespage | frobware, hmm but dns-search is not part of the iface stanza's as generated right now | 11:27 |
| jamespage | that might not matter if curtin also generates resolv.conf | 11:27 |
| frobware | jamespage, the combination of ifup running resolvconf which plucks the dns-search from /e/n/i should end up in resolv.conf | 11:29 |
| frobware | jamespage, I would still like to understand how/when/where the change to /e/n/i happened. Quite a few of the unit tests I have were cut+paste /e/n/i files from a deployed MAAS 1.9 node. | 11:34 |
| jamespage | frobware, so would I - I also think the /e/n/i generated by maas/curtin is a little malformed right? | 11:35 |
| dimitern | frobware, will do | 11:35 |
| frobware | jamespage, the indentation and the separation of the dns-* entries confuse matters. | 11:36 |
| * jamespage crosses fingers for network access | 11:39 | |
| jamespage | frobware, ok better - I'm only expecting a bridge for eth0 right now? | 11:47 |
| frobware | jamespage, how many interfaces do you have? You should get a bridge per active NIC now. | 11:47 |
| jamespage | frobware, eth0 is the only active physical nic - all others are vlans on eth1 | 11:48 |
| frobware | jamespage, can you paste your original /e/n/i | 11:48 |
| dimitern | frobware, I've filed bug 1532167 | 11:52 |
| mup | Bug #1532167: maas bridge script handles VLAN NICs incorrectly <addressability> <maas-provider> <network> <juju-core:Triaged> <juju-core 2.0:Triaged> <https://launchpad.net/bugs/1532167> | 11:52 |
| frobware | dimitern, thanks! | 11:52 |
| frobware | dimitern, dooferlad, voidspace: should we propagate dns-* from the original iface to the new bridge stanza? | 11:54 |
| dooferlad | frobware: yes | 11:55 |
| dimitern | frobware, yeah - because the bridge is taking over the nic and all its options, just adding the bridge-utils ones to them | 11:55 |
| frobware | dimitern, dooferlad: which is what I am/was doing, just confirming. | 11:56 |
| dooferlad | frobware: great :-) | 11:56 |
| frobware | dooferlad, dimitern: think this only holds true for bonds. e.g., http://pastebin.ubuntu.com/ | 11:57 |
| frobware | http://pastebin.ubuntu.com/14437347/ | 11:57 |
| dooferlad | frobware: that seems reasonable, but I don't know for sure. Since it is a rule that says "add this nameserver to resolv.conf when I am activated" as long as the rule will run under the same conditions, that is fine. | 11:59 |
| frobware | dooferlad, otherwise we have to special case which options we carry forward. Which is not a big deal because we already do it for the bond anyway. | 12:00 |
| frobware | dimitern, actually for the bond it doesn't take all its options. I deliberately omit anything starting with bond. | 12:04 |
| frobware | jamespage, any joy? | 12:04 |
| jamespage | frobware, yeah | 12:05 |
| mup | Bug #1532167 opened: maas bridge script handles VLAN NICs incorrectly <addressability> <maas-provider> <network> <juju-core:Triaged> <juju-core 2.0:Triaged> <https://launchpad.net/bugs/1532167> | 12:05 |
| dimitern | frobware, that's because we verified what needs to stay for the bond-bridge(-vlan) device to still work | 12:05 |
| frobware | dimitern, I'll actively try leaving them in. I think the fewer special cases the better. | 12:06 |
| jamespage | frobware, dimitern - ok I have a bound ceph deployment running! | 12:06 |
| jamespage | w00t! | 12:06 |
| frobware | dimitern, the new iface (i..e, the bridge) is not a bond so I didn't carry those options forward. | 12:07 |
| frobware | jamespage, excellent! | 12:07 |
| dimitern | frobware, ah, right | 12:08 |
| frobware | dimitern, these are all just arbitrary rules. :-) | 12:08 |
| mup | Bug #1532167 changed: maas bridge script handles VLAN NICs incorrectly <addressability> <maas-provider> <network> <juju-core:Triaged> <juju-core 2.0:Triaged> <https://launchpad.net/bugs/1532167> | 12:11 |
| mup | Bug #1532167 opened: maas bridge script handles VLAN NICs incorrectly <addressability> <maas-provider> <network> <juju-core:Triaged> <juju-core 2.0:Triaged> <https://launchpad.net/bugs/1532167> | 12:14 |
| dimitern | frobware, also filed bug 1532176 | 12:23 |
| mup | Bug #1532176: maas bridge script in maas-spaces feature branch handles dns-* options incorrectly <addressability> <maas-provider> <juju-core maas-spaces:Triaged> <https://launchpad.net/bugs/1532176> | 12:23 |
| dimitern | frobware, added the expected /e/n/i in jamespage's case (AIUI) | 12:24 |
| dimitern | jamespage, awesome! | 12:24 |
| frobware | dimitern, do we want to raise bugs for feature branches? | 12:25 |
| dimitern | frobware, we can - whether we should is another thing :) | 12:25 |
| frobware | dimitern, seems overkill to me | 12:26 |
| dimitern | frobware, but since I started composing it, esp. took care with the expected config, seemed good to have it filed | 12:26 |
| dimitern | but I might've just as well added a card | 12:29 |
| perrito666 | dimitern: ping | 12:35 |
| dimitern | perrito666, hey there | 12:37 |
| perrito666 | hey, ill privmsg you | 12:38 |
| mup | Bug #1532186 opened: lxd container creation fails semi - regularly <local-provider> <juju-core:Triaged> <https://launchpad.net/bugs/1532186> | 13:17 |
| mup | Bug #1532186 changed: lxd container creation fails semi - regularly <local-provider> <juju-core:Triaged> <https://launchpad.net/bugs/1532186> | 13:26 |
| mup | Bug #1532186 opened: lxd container creation fails semi - regularly <local-provider> <juju-core:Triaged> <https://launchpad.net/bugs/1532186> | 13:35 |
| frobware | dimitern, dooferlad, voidspace: http://reviews.vapour.ws/r/3481/ - network/interfaces fix for demo | 13:55 |
| dimitern | frobware, looking | 13:57 |
| dimitern | frobware, LGTM with an extra test for the jamespage's case | 14:02 |
| frobware | dimitern, done and thanks for the review | 14:11 |
| dimitern | frobware, cheers | 14:12 |
| dimitern | frobware, hmm wait a moment.. | 14:14 |
| frobware | dimitern, oh... | 14:14 |
| dimitern | frobware, looking at the test now | 14:14 |
| dimitern | frobware, why auto eth1.2667 and the others still remain? | 14:14 |
| dimitern | frobware, also I can only see one bridge | 14:14 |
| frobware | dimitern, I think we've danced this dance before. | 14:15 |
| dimitern | frobware, either the expected is wrong or the script doesn't quite handle that as I'd have expected | 14:15 |
| frobware | dimitern, the underlying device is not active. | 14:15 |
| frobware | dimitern, I believe we previously said inactive devices don't get bridged. | 14:16 |
| frobware | dimitern, however, what should the behaviour be... | 14:16 |
| dimitern | frobware, but it's auto and it has auto children which are active | 14:16 |
| frobware | dimitern, so the current test is... is the underlying vlan-raw-device eth1 active? | 14:17 |
| dimitern | frobware, if this is the /e/n/i we'll use, how will multi-NIC containers work on that host node? | 14:17 |
| frobware | dimitern, they obviously won't... :( | 14:17 |
| dimitern | frobware, well, it looks active - "auto eth1", despite being "manual" | 14:17 |
| dimitern | it will be up when running ip link show | 14:17 |
| dimitern | otherwise the vlans won't work either | 14:18 |
| frobware | dimitern, so I think we should nail down what we consider an active interface | 14:18 |
| frobware | dimitern, btw, I don't believe this is a demo problem. | 14:18 |
| dimitern | frobware, yeah, it's not but only because we're not using containers | 14:19 |
| dimitern | frobware, and don't get me wrong - your fix is till valid | 14:19 |
| frobware | dimitern, but let's fix containers whilst we're here. | 14:19 |
| dimitern | frobware, but for the multi-NIC containers we'll need to fix that (for the os demo) | 14:19 |
| frobware | dimitern, because some of this could go back into 1.25 too | 14:20 |
| dimitern | frobware, I think "active" should mean either the NIC itself is up and has address (can send/receive packets) or any of it children are active | 14:20 |
| frobware | dimitern, let's let the fix for the demo land and do another for multi-NICs | 14:21 |
| dimitern | frobware, sgtm | 14:21 |
| frobware | dimitern, so it's the "and has an address" - this is where we have stumbled before. | 14:21 |
| frobware | dimitern, the eth1 in that test has no address. | 14:21 |
| dimitern | frobware, yeah, but it has dependents which have | 14:22 |
| frobware | dimitern, our parsing is too simplistic | 14:22 |
| dimitern | frobware, eth1 just needs to be up | 14:22 |
| frobware | dimitern, we don't really model it - we just do some pretty high-level text transformations. | 14:22 |
| dimitern | frobware, yeah - one pass only | 14:22 |
| frobware | dimitern, let me work on this now... | 14:23 |
| dimitern | frobware, we need to get a simple model from the /e/n/i with a dict of NIC names to NIC info which includes an "active" flag and any children - similarly | 14:24 |
| dimitern | well, it can also include all the options so you don't have to re-parse them when rendering | 14:25 |
| frobware | dimitern, parsing is so cheap though. | 14:26 |
| frobware | dimitern, open a file, read line by line. | 14:26 |
| dimitern | frobware, just tried with the initial e/n/i, but changed eth1 to dhcp instead of manual and added "a" to each dns-nameservers options except the last one and this works better:http://paste.ubuntu.com/14438110/ | 14:30 |
| dimitern | frobware, what was the issue with just just always creating a bridge for each iface? | 14:30 |
| dimitern | bonds I presume.. or boot slowdowns? | 14:31 |
| frobware | dimitern, heh. so that's an alternative. | 14:31 |
| frobware | dimitern, not sure there is one. | 14:31 |
| frobware | dimitern, I was trying to cater for the case where it is unconfigured. | 14:32 |
| frobware | dimitern, if it is unconfigured and we create a bridge, what happens for the VIP? | 14:32 |
| dimitern | frobware, well, it needs to have "auto" at least perhaps? | 14:34 |
| dimitern | frobware, I don't mind creating an extra bridge we'll not use, unless that causes problems | 14:34 |
| dimitern | frobware, the VIP is not involved I think | 14:35 |
| dimitern | frobware,do you mean the VIPs added by corosync ? | 14:35 |
| dimitern | frobware, with or without a bridge that will still work - adding an extra IP to an interface that's not up - but it simply won't pass any traffic | 14:36 |
| frobware | dimitern, it shouldn't matter. I was thinking of the case where we want to give back a device | 14:36 |
| frobware | dimitern, the dellstack case --> http://pastebin.ubuntu.com/14438169/ | 14:37 |
| dimitern | frobware, in that case we'll see that e.g. eth3 on node1 is linked to a subnet but in "link_up" state only, and that's what we'll give neutron-gateway | 14:37 |
| dimitern | (well, once network-get returns more than an address, but the full NIC info) | 14:38 |
| frobware | dimitern, what did you think of ignoring is_active and just bridge the lot?! | 14:38 |
| dimitern | frobware, I think we should try as the simplest thing we can do for multi-NIC containers | 14:39 |
| dimitern | frobware, and seriously test it in an automated fashion with permutations of different NIC configs on different MAAS versions | 14:40 |
| frobware | dimitern, my hack should you want this: http://pastebin.ubuntu.com/14438191/ | 14:40 |
| dimitern | frobware, cheers | 14:40 |
| dimitern | frobware, I'll come up with something next week to automate setting up lxc-based maas of a given revision with a given set of networks | 14:41 |
| frobware | dimitern, there's a certain amount of orthogonality that comes with that | 14:41 |
| dimitern | frobware, yeah, but we need to at least make sure it works with bonds and vlans (and vlans onto a bond) | 14:42 |
| frobware | dimitern, there are unit tests for that | 14:43 |
| dimitern | the latter case is common in telcos | 14:43 |
| dimitern | frobware, unit testing what we generate is a must, but live testing whether it will work (and didn't break across releases) is also needed when we can get it to work | 14:44 |
| dimitern | (get the automated testing to work I mean) | 14:44 |
| frobware | dimitern, sure. I tried the vlans onto a bond a little while ago, but not with all the world bridged. | 14:44 |
| frobware | dimitern, but from a parsing and creating bridges POV I believe we have some of these permutations. what happens in the wild... stays in the wild. :-) | 14:46 |
| dimitern | frobware, yeah, we'll just keep updating those tests as we discover issues | 14:49 |
| frobware | dooferlad, I just dumped 1531932 your way. Happy to answer questions here, in the card or wherever works. | 14:50 |
| dooferlad | frobware: ok | 14:51 |
| dooferlad | I assume that is bug #1531932 | 14:51 |
| mup | Bug #1531932: Unit agents unable to connect with MAAS 1.8 due to lack of default gateway in MAAS <juju-core:Triaged by dooferlad> <https://launchpad.net/bugs/1531932> | 14:51 |
| frobware | dooferlad, yep. Before weds next week we need to land that, #1525280. And port #1528217 to master. | 14:53 |
| mup | Bug #1525280: 1.25.1 with maas 1.8: devices dns allocation uses non-unique hostname <maas-provider> <network> <regression> <juju-core:Triaged> <juju-core 1.25:Fix Committed by dimitern> <https://launchpad.net/bugs/1525280> | 14:53 |
| mup | Bug #1528217: 1.25.2 doesn't set up DNS information with MAAS <blocker> <ci> <maas> <network> <regression> <juju-core:Incomplete> <juju-core 1.25:Triaged> <https://launchpad.net/bugs/1528217> | 14:53 |
| frobware | dooferlad, to tickle the lack of gateway in 1.8 you just need to remove any entry you may have in MAAS already in the network (in my case maas-eth0). | 14:55 |
| frobware | dimitern, if we don't have uniques hostnames on master I'm guessing we don't run into either lack of default gw and/or lack of DNS info - correct? It looks like that patch switches the provisioner to use configType.static. | 15:00 |
| frobware | dimitern, not sure if you saw that ^^ - formatting looks odd in my client | 15:02 |
| dimitern | frobware, we don't have unique hostnames, but we still use static and I don't remember whether the dns and gateways were discovered when missing | 15:04 |
| frobware | dimitern, just trying to see what's the easiest environment to initially fix this in. 1.25 is the best place today as we know it fails there. | 15:10 |
| frobware | dooferlad, ^^ | 15:10 |
| dimitern | frobware, well, 1.25 can be fixed after the revert happens so I guess we should fix dns + gateway on master and also unique hostnames first | 15:12 |
| frobware | dimitern, the reason why I'm asking is ... why doesn't it fail today in the CI tests. | 15:12 |
| frobware | dimitern, if there's a default gw entry, then it will be fine. but it popped up in CI because there was no entry. | 15:13 |
| dimitern | frobware, because there are hardly any serious tests that use containers and/or multiple envs bootstrapped concurrently | 15:13 |
| frobware | dimitern, well 1.25.2 tripped over it as soon as we fixed dns | 15:14 |
| frobware | dimitern, so there are some | 15:14 |
| dimitern | frobware, and for the dns / GW - I don't believe any CI tests (except likely for dooferlad's) check containers networking accessibility etc. | 15:14 |
| frobware | dimitern, anyway, I'm not suggesting we fix in 1.25.2 first, just that's a playground you can use today because it fails there atm | 15:14 |
| dimitern | frobware, well, they started passing for the first time I guess :) | 15:15 |
| dimitern | frobware, right, I see | 15:15 |
| frobware | dimitern, was trying to lower dooferlad's entry point... :) | 15:16 |
| mup | Bug #1528259 changed: undertakerSuite.TestAPICalls nil deref panic <ci> <i386> <panic> <regression> <test-failure> <juju-core:Fix Released by axwalk> <https://launchpad.net/bugs/1528259> | 15:20 |
| mup | Bug #1532232 opened: maas 1.9 container networking broken <ci> <lxc> <maas-provider> <regression> <juju-core:Incomplete> <juju-core 1.25:Triaged> <https://launchpad.net/bugs/1532232> | 15:20 |
| frobware | dimitern, ^^ LOL. sigh. | 15:21 |
| * frobware takes up smoking... | 15:21 | |
| mup | Bug #1532232 changed: maas 1.9 container networking broken <ci> <lxc> <maas-provider> <regression> <juju-core:Incomplete> <juju-core 1.25:Triaged> <https://launchpad.net/bugs/1532232> | 15:26 |
| mup | Bug #1528259 opened: undertakerSuite.TestAPICalls nil deref panic <ci> <i386> <panic> <regression> <test-failure> <juju-core:Fix Released by axwalk> <https://launchpad.net/bugs/1528259> | 15:26 |
| dimitern | :)( | 15:26 |
| frobware | sinzui, dimitern, jog: when I looked at John's email earlier today he indicated that it had worked and the failure may be intermittent. Has that changed? | 15:32 |
| dimitern | frobware, I haven't looked into it since then - want to finish the last 2 cards around interface_set parsing | 15:33 |
| sinzui | frobware: dimitern: I cannot say exactly what is wrong. I see that non of the containers called home, which looks like container networking. | 15:34 |
| dimitern | sinzui, if we have the cloud-init-output.log of the containers it will tell us what went wrong | 15:35 |
| frobware | sinzui, any of these nodes still up? | 15:35 |
| sinzui | dimitern: It is all there http://reports.vapour.ws/releases/3487/job/maas-1_9-OS-deployer/attempt/130 | 15:35 |
| frobware | sinzui, as in can we login to the node and look at the container setup, et al? | 15:35 |
| sinzui | frobware: no. | 15:35 |
| dimitern | sinzui, well, notably cloud-init-output.log from the container's rootfs is not there | 15:36 |
| dimitern | I might have reported a bug for that some time ago | 15:36 |
| sinzui | dimitern: damn. Thzt is a bug in in CI. We need to change the rules. I think the globe is wrong | 15:37 |
| dimitern | https://bugs.launchpad.net/juju-ci-tools/+bug/1424357 | 15:38 |
| mup | Bug #1424357: copy_local_logs should also get container logs <juju-ci-tools:Fix Released by sinzui> <https://launchpad.net/bugs/1424357> | 15:38 |
| sinzui | frobware: structured-metadata is testing now. It is possible to re-run the failing job, or just deploy a smaller bundle that uses containers. | 15:38 |
| dimitern | it's marked as fix released | 15:38 |
| frobware | sinzui, can we do that next? | 15:38 |
| mup | Bug #1528259 changed: undertakerSuite.TestAPICalls nil deref panic <ci> <i386> <panic> <regression> <test-failure> <juju-core:Fix Released by axwalk> <https://launchpad.net/bugs/1528259> | 15:38 |
| mup | Bug #1532232 opened: maas 1.9 container networking broken <ci> <lxc> <maas-provider> <regression> <juju-core:Incomplete> <juju-core 1.25:Triaged> <https://launchpad.net/bugs/1532232> | 15:38 |
| sinzui | dimitern: yes, but it is clear one log is missing | 15:38 |
| dimitern | sinzui, ah, sorry - I've now actually read what I posted originally, and I don't mention to get /var/lib/lxc/juju-*/rootfs/var/log/cloud-init*.log | 15:40 |
| dimitern | unlike with a broken kvm you can't get to, lxc rootfs is accessible from the host usually | 15:40 |
| sinzui | dimitern: I am going to make a one line change to get the missing log, then run a simpler job to get more information | 15:47 |
| dimitern | sinzui, thanks | 15:48 |
| thedac | jamespage: dimitern: frobware: Odd I did not run into that problem. Do things look ok for the demo now? | 15:50 |
| frobware | thedac, yes - I believe jamespage reported a working ceph deployment. | 15:50 |
| thedac | ok, great. | 15:51 |
| frobware | thedac, so it did trigger a genuine problem and we were curious as to how/why we haven't seen this to date. | 15:51 |
| frobware | thedac, the fix has been merged into the maas-spaces branch | 15:52 |
| thedac | Yeah, me too. I have done dozens of deplys and never saw that issue. | 15:52 |
| thedac | s/deplys/deploys | 15:52 |
| frobware | thedac, we were speculating that it may have been an external change in curtin/maas/trusty/...? | 15:53 |
| thedac | hmm, well I am not going to touch those hosts in MAAS so as not to introduce new problems ;) | 15:53 |
| sinzui | dimitern: the cloud-init-ouput.log is not in /var/lib/juju/containers. I need larger hack to get the log. | 16:02 |
| dimitern | sinzui, yeah, it's in /var/lib/lxc/juju-*/rootfs/var/log/cloud-init-*.log | 16:27 |
| mattyw | dimitern, ping? | 16:40 |
| mattyw | ^^ also maybe sinzui ping? | 16:40 |
| frobware | dooferlad, you about? | 16:42 |
| dooferlad | frobware: yep | 16:43 |
| frobware | dooferlad, can we do a quick HO... | 16:43 |
| dooferlad | frobware: sure | 16:43 |
| dimitern | frobware, dooferlad, voidspace, I'd appreciate a review on http://reviews.vapour.ws/r/3482/ when you can | 16:43 |
| dimitern | mattyw, pong | 16:43 |
| mattyw | dimitern, hey hey - as I understand it juju doesn't support bootstrapping on CentOs yet - is that right? | 16:44 |
| dimitern | mattyw, I've never tried bootstrapping, but some time ago I fixed an issue with manual provisioning to a centos machine | 16:45 |
| dimitern | mattyw, but looking at the cloudconfig code it should be possible - except if mongodb is borked there or something | 16:46 |
| mattyw | dimitern, I'm trying to help a guy out on the list who is bootstrapping on centos, I'm sure I remember some discussion about it not working yet, but can't find any evidence yet | 16:46 |
| mattyw | dimitern, there seem to be some selinux issues coming up | 16:46 |
| cherylj | mattyw: see also bug 1474386 | 16:47 |
| mup | Bug #1474386: Problems bootstrapping the manual provider with CentOS <centos> <manual-provider> <juju-core:Triaged> <https://launchpad.net/bugs/1474386> | 16:47 |
| dimitern | mattyw, right well, sorry I can't help with that | 16:47 |
| mattyw | dimitern, no problem - just wanted to see if you had any experience, as you were around | 16:48 |
| cherylj | mattyw: I also recall a recent change to handle the selinux problem? maybe? | 16:48 |
| cherylj | sounds familiar anyway | 16:48 |
| mattyw | cherylj, I'm not sure, I'm sure I remembered hearing something about it not being implemented ages ago, I don't think we have a ci test for it? | 16:48 |
| mattyw | cherylj, if we think bootstrapping on centos should work I'll help the guy raise a bug for us to look at | 16:49 |
| cherylj | mattyw: you can also attach more info that the one bug I linked to | 16:50 |
| mattyw | cherylj, the log I've got is totally different, so I'm going to raise a new one | 16:51 |
| cherylj | mattyw: k | 16:51 |
| natefinch | katco, ericsnow: putting juju show-service-resources back and making juju show-charm-resources into juju resource reminds me.... I think this is exactly backwards. I think people will use show-service-resources 1000x as often as show-charm-resources. You'll be constantly checking your environments to make sure they have the correct resources deployed, but how often will you really want to look at what's in the charm store from the CLI? Hardly ev | 16:51 |
| natefinch | er? Why not just use the nice website we've spent so much time on? | 16:51 |
| ericsnow | natefinch: yep | 16:51 |
| natefinch | katco, ericsnow: but hopefully thats something we can talk about during/after the demo | 16:52 |
| natefinch | katco, ericsnow: just make john type show-service-resources like 20 times during the demo, and we'll get him on our side ;) | 16:52 |
| mattyw | cherylj, https://bugs.launchpad.net/juju-core/+bug/1532266 | 16:54 |
| mup | Bug #1532266: bootstrapping on Centos fails with SELinux policy denies access <juju-core:New> <https://launchpad.net/bugs/1532266> | 16:54 |
| mup | Bug #1532266 opened: bootstrapping on Centos fails with SELinux policy denies access <juju-core:Triaged> <https://launchpad.net/bugs/1532266> | 17:17 |
| frobware | dooferlad, qemu-img create -f qcow2 -o preallocation=metadata vm-100-disk-3.qcow2 1G | 17:21 |
| natefinch | anyone online familiar with the new charm upload/publish? I can't get my server to deploy the charm I uploaded | 18:28 |
| natefinch | ericsnow or katco: trivial 2 line review of command rename? | 18:53 |
| natefinch | http://reviews.vapour.ws/r/3483/diff/# | 18:54 |
| ericsnow | natefinch: done | 18:55 |
| natefinch | early afternoon is the best time to have a lot of stuff to land... no competition for the landing bot | 18:57 |
| natefinch | I do wish our CI was a little more continuous than it actually is.... like telling me pre-emptively when my code won't compile after someone changes master out from under me.... ericsnow. | 18:58 |
| natefinch | ;) | 18:58 |
| ericsnow | natefinch: I thought I warned you :) | 18:58 |
| natefinch | ericsnow: probably, I don't know... who can keep track? | 18:59 |
| ericsnow | natefinch: if you have a sec, could you take a look at https://github.com/juju/charm/pull/188? | 19:03 |
| ericsnow | natefinch: thanks! | 19:22 |
| natefinch | ericsnow: welcome | 19:22 |
| katco | ericsnow: just finishing up some admin things | 19:33 |
| katco | ericsnow: and then we can pair agian | 19:33 |
| ericsnow | katco: k | 19:33 |
| katco | ericsnow: if you want that is | 19:33 |
| ericsnow | katco: yep | 19:33 |
| natefinch | ericsnow, katco: 40 lines of demo code to save resource metadata to state on deploy (most of this will probably be reused after the demo): http://reviews.vapour.ws/r/3484/diff/# | 20:05 |
| ericsnow | natefinch: sweet | 20:05 |
| natefinch | wrong diff: http://reviews.vapour.ws/r/3484/diff/# | 20:05 |
| natefinch | or not.. sorry, confusing myself between PR number and diff number | 20:06 |
| natefinch | holy crap it works | 20:48 |
| ericsnow | natefinch: :) | 20:49 |
| natefinch | realized why the code wasn't working for saving resources at deploy time.... I was saving them using the serviceID for the doc, which includes the environment UUID, but we query them with the service name | 20:49 |
| natefinch | easy fix | 20:49 |
| === gberginc_ is now known as gberginc | ||
| katco | ericsnow: ok sorry i'm ready if you're available | 20:52 |
| ericsnow | katco: sure | 20:53 |
| natefinch | ericsnow, katco: very quick review of a bugfix in the output for show-service-resources? 100% relevant to the demo: http://reviews.vapour.ws/r/3485/diff/# | 21:33 |
| natefinch | katco, ericsnow: btw: | 21:41 |
| natefinch | $ juju show-service-resources starsay | 21:41 |
| natefinch | RESOURCE ORIGIN REV USED COMMENT | 21:41 |
| natefinch | store-resource upload - no One line that is useful when operators need to push it. | 21:41 |
| natefinch | upload-resource upload - no Who uses xml anymore? | 21:41 |
| natefinch | real output stored in the database, derived from the metadata.yaml at deploy time | 21:41 |
| natefinch | ...once my last two patches land :) | 21:42 |
| natefinch | https://media.giphy.com/media/b3u8anVaWFQ9G/giphy.gif | 21:42 |
| natefinch | there are a lot more hits for grep "vespene gas" in the juju codebase than I would have suspected | 21:53 |
| natefinch | ericsnow, katco: gotta run for dinner, but I'll be back after for a few hours. Feel free to shipit and $$merge$$ my branches if you wish, otherwise I will when I get back. I started looking at eric's patches up for review, but they're big and will take a bit. I'll do that if there's nothing else pressing when I get back. | 21:58 |
| === natefinch is now known as natefinch-afk | ||
| katco | natefinch-afk: sorry for radio silence; ericsnow is a demanding pairing partner (sob) | 22:45 |
| ericsnow | katco: lol | 22:45 |
| katco | i feel so defeated | 22:45 |
| ericsnow | katco: whatever! | 22:45 |
| katco | natefinch-afk: thx for patches, glad to see it's working :) | 22:45 |
| ericsnow | katco: I thought we did well :) | 22:45 |
| === ericsnow is now known as ericsnow-afk | ||
| katco | ericsnow: yeah jk that was awesome | 22:45 |
| katco | ericsnow-afk: you even used emacs! | 22:46 |
| katco | ericsnow-afk: i'm going to call you an emacs user now | 22:46 |
| lazyPower | oh god, they're multiplying | 22:55 |
| katco | lazyPower: you don't know the power of the emacs | 23:00 |
| lazyPower | you're right | 23:00 |
| katco | lazyPower: give in to your fear and hatred. let it make you stronger | 23:01 |
| lazyPower | katco: i'll think about out... but my 8ball says the outlook is not good | 23:04 |
| lazyPower | s/out/it/ | 23:04 |
| lazyPower | i mean, why shop at walmart (emacs) when you've got amazon (vim) :) | 23:05 |
| perrito666 | Ericsnow if you are still around pls take a look at the gce patch for storage uuid on description | 23:29 |
| perrito666 | Eow bye all | 23:29 |
| mup | Bug #1532354 opened: Juju 1.26-Alpha4 reports incorrect public-address with MaaS 1.9 <juju-core:New> <https://launchpad.net/bugs/1532354> | 23:30 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!
