[13:25] <larsks> smoser: fyi, this just merged in openstack nova: https://review.openstack.org/#/c/467699/
[13:25] <larsks> That has implications for how cloud-init handles nameservers from openstack data sources.
[13:27] <smoser> larsks, thanks for the heads up.. .reading.
[13:27] <smoser> interesting.
[13:27] <smoser> fwiw, no linux networking configuration system that i'im aware of actually does dns servers specific to interfaces very well.
[13:28] <larsks> I dunno, my fedora environment seems to handle it pretty well (I have vpn-specific nameservers that are only used when the vpn is up).
[13:28] <larsks> Although I'm using a local dnsmasq proxy, which means that nothing ever touches /etc/resolv.conf.
[13:28] <smoser> and you send queries for *.<internal> and network/prefix correctly ?
[13:29] <larsks> Definitely the former (all *.redhat.com stuff is handled by the vpn nameservers).  Haven't checked the latter, honestly.
[13:29] <smoser> i suspect you're getting all stuff sent there :)
[13:30] <smoser> and yeah, network manager does a *reasonable* job.
[13:30] <larsks> No, only *.redhat.com goes to the vpn nameservers (as verified w/ tcpdump).
[13:30] <larsks> Requests for other domains go to my local router.
[13:31] <smoser> nice.
[13:31] <smoser> note... in the bug there.
[13:31] <larsks> This works via the dnsmasq --server option ("If one or more optional domains are given, that server is used only for those domains...")
[13:31] <smoser> it does not show any domain wildcards
[13:31] <smoser> right?
[13:31] <smoser> is there information that would say "send *.redhat.com queries here"
[13:32] <larsks> smoser: in my case, it's part of the vpn configuration.  The bigger use case w/r/t cloud-init is simply the fact that nameservers that are only reachable via a particular interface can be removed from the resolver configuration when the interface goes down, preventing hangs and other related behavior.
[13:33] <smoser> sure.
[13:33] <smoser> but if both are up?
[13:33] <larsks> Then it doesn't matter.
[13:33] <smoser> yes it does
[13:33] <smoser> if one has an answer and the other doesnt
[13:33] <smoser> or conflicting answers
[13:34] <smoser> i think cloud-init will handle this fine... have you tried ?
[13:34] <smoser> can you give me an example of the network_config.json ?
[13:34] <larsks> cloud-init modifies only the global /etc/resolv.conf and does not populate dns information in the interface config files.
[13:34] <larsks> Because until this change merged, that information simply wasn't available.,
[13:34] <larsks> There was no way to configure it.
[13:34] <larsks> We only received global information from openstack.
[13:35] <smoser> right... ok. it does do that for ENI (puts dns name servers on the interface that htey camne from)
[13:35] <smoser> do you have an example ?
[13:35] <smoser> and i guess best thing to do is just make a cloud-init task for that bug
[13:35] <smoser> i can open that, unless you disagree.
[13:36] <larsks> Not handy, but the change is minimal: in addition to the global "services" section, there would be a per-network "services" section as well.
[13:36] <larsks> Yeah, I was thinking of opening a cloud-init bug for it.  So go ahead.
[13:36] <larsks> I can spin up an environment running the latest nova and produce an example for a bug sometime this week.
[13:46] <smoser> larsks, just to clarify, it almost certainly does matter which queries go to which server.
[13:46] <smoser> if you have all the same information in all the dns servers, then it seems like additional services with no value.
[13:47] <smoser> i realize there are some cases... but it seems that scoping those queries is pretty quickly goign to be important, so you can do things like *.redhat.com -> $REDHAT_DNS.
[13:48] <larsks> smoser:  assuming that all dns servers are equally able to answer questions, the value is that if you have dns servers that are only reachable via one interface, you can remove them from the resolver configuration if they are unreachable because the interface is down.
[13:48] <smoser> yeah.
[13:48] <larsks> I don't think we recieve sufficient information to make domain related decisions.
[13:49] <smoser> we dont' in the provided network information for sure
[13:49] <smoser> thats what i'm saying :)
[13:49] <larsks> That's not even possible to configure in openstack.
[13:49] <larsks> So this is primarily an issue of reachability, and that is the value.
[14:03] <smoser> larsks, http://paste.ubuntu.com/25220242/
[14:03] <smoser> that adds a unit test that i think represents what you're change did.
[14:03] <larsks> Right, like that.
[14:03] <smoser> so if you're intereted in telling me what that *hsould* represent in sysconfig ... .patches are quite welcome :)
[14:04] <larsks> Yup, I figured :)
[14:10] <smoser> larsks, when you do make that change i suspect you'll break some of the other unit tests cases even test_small_config
[14:10] <smoser> as that has another network config format that does have dns entries per subnet
[14:10] <larsks> I think that's okay. I mean, both need to work.
[14:10] <smoser> right
[14:11] <smoser> you'll just have to fix the expected output of those tests too
[14:12] <smoser> larsks, hm..
[14:12] <smoser> wondering how sysconfig handles this
[14:14] <rharper> there is per ifcfg DNS_ values
[14:14] <smoser> https://serverfault.com/questions/423882/configure-a-dns-server-per-nic-interface-eth0-eth1
[14:14] <rharper> someone filed a bug about getting those in
[14:15] <rharper> DNS{1,2,3}
[14:15] <smoser> well, we write IPV4DNS0 and IPV4DNS1
[14:15] <smoser> but that is per nic
[14:15] <smoser> not per address
[14:15] <rharper> it would need the index
[14:15] <rharper> just like there's IPADDR1, 2, 3
[14:15] <smoser> the index of what ?
[14:15] <larsks> smoser: right now, we don't write per-nic dns servers (because they weren't available).
[14:15] <rharper> address index
[14:15] <smoser> i dont think it works like that :)
[14:15] <larsks> That's what this change enabled.
[14:15] <rharper> it does
[14:16] <smoser> larsks, right. openstack now provides per *address* dns information
[14:16] <smoser> (i think)
[14:16] <smoser> thats what i put in that patch at least.
[14:16] <smoser> ie, the 'services' are now on a 'network' entry
[14:17] <smoser> and multiple networks per nic
[14:17] <larsks> So pick 3, and have it.
[14:17] <smoser> but the best i can tell is that sysconfig supports only per NIC dns information.
[14:17] <larsks> Right.
[14:18] <larsks> DNS config is per NIC.  And the most common config I've seen with openstack is one network/one nic, but sure, you can have multiple ones attached.
[14:18] <larsks> The assumption there would be that they are all equally valid.
[14:19] <larsks> I don't think we receive information that would permit us to do per-prefix or per-domain routing of dns requests.
[14:19] <smoser> well, i guess the thing that makes this ok is that sysconfig only really does things on a per NIC basis anyway
[14:19] <larsks> Right.
[14:19] <smoser> rather than a per-address-per-nic
[14:19] <smoser> so openstack is providing information that technically could represent a superset of what sysconfig can support
[14:45] <smoser> larsks, http://paste.ubuntu.com/25220399/ . i think that includes the change for openstack.py that would be needed. but need changes in the sysconfig rendere to write the DNS* fields.
[14:45] <smoser> rather than rendering resolv.conf
[17:43] <powersj> rharper: All the errors from last night point to epel mirror issues. Was there another failure that I missed since you mentioned copr? I want to make sure I didn't miss a failure.
[17:44] <powersj> "Cannot retrieve metalink for repository: epel/x86_64. Please verify its path and try again"
[17:44] <powersj> from all 3 failures
[17:59] <smoser> powersj, i saw that too
[17:59] <smoser> i dont know.
[18:00] <smoser> its hard to believe that epel would be unreliable
[18:00] <smoser> even if copr was.
[18:00] <powersj> Right, re-running just now also resulted in failure
[18:00] <smoser> rharper suggested we can turn off the yum fastestmirror thing possibly to improve
[18:00] <powersj> so makes me wonder if something changed
[18:00] <powersj> I agree with that suggestion
[18:00] <powersj> as you saw we get very, very odd mirrors (e.g. japan?!)
[18:12]  * blackboxsw pulls his head out of the AWS-init-local sand and gets onto SRU bug template writeups. The 2nd part (2 of 3) IPv6 AWS branches is up for review https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/328241. 
[18:18] <rharper> powersj: there was one related to the copr repo timing out
[18:18] <powersj> hmm I didn't see that last night
[18:18] <rharper> Timeout on https://copr-be.cloud.fedoraproject.org/results/@cloud-init/el-stable/epel-7-x86_64/repodata/repomd.xml: (28, 'Connection timed out after 30001 milliseconds')
[18:19] <rharper> disabling fastmirror won't help there
[18:19] <rharper> I think copr might have been down at the time
[18:20] <powersj> oh.. I was looking at jenkins runs, we don't have one for that repo
[18:20] <rharper> right
[18:20] <powersj> sorry I heard "runs" and immediately jump to jenkins
[18:25] <rharper> no worries
[18:25] <rharper> I don't think we have much choice but to add some retries in there
[19:03] <blackboxsw> smoser: so how do we work this SRU for jsonschema optional dependency,  the upstream comit still contains the dependency defined https://git.launchpad.net/cloud-init/diff/requirements.txt?id=0a448dd034   But, I don't see a comparable filter defined in bddeb to prevent that python-jsonschema dependency from being added to the deb-requires.
[19:04] <blackboxsw> don't we need to prevent jsonschema from 'leaking' into package deps on xenial/zesty
[19:13] <rharper> blackboxsw: IIRC, the downstream packaging branch (ubuntu/xenial) will not update the debian/control file with the deps
[19:13] <rharper> or applies a patch that disabled it (removes it from a lest); but it will essentially always care some "Delta" from master
[19:13] <rharper> s/a lest/the list
[19:25] <smoser> blackboxsw, you're right.
[19:25] <smoser> that is applied to the ubuntu packaging branches at
[19:25] <smoser>  debian/patches/stable-release-no-jsonschema-dep.patch
[19:25] <smoser> ah. and look at that... i actually even referenced it in the debian changelog ;)
[19:26] <smoser> (i figured i'd missed it)
[19:26] <smoser>   * debian/patches/stable-release-no-jsonschema-dep.patch:
[19:26] <smoser>     add patch to remove optional dependency on jsonschema.
[19:29] <smoser> blackboxsw, for bug 1701097, bug 1705147 and bug 1706752 i suggest we verify using tools/net-convert.py
[19:29] <ubot5> bug 1701097 in cloud-init (Ubuntu Zesty) "eni rendering of ipv6 gateways fails" [Medium,Confirmed] https://launchpad.net/bugs/1701097
[19:29] <ubot5> bug 1705147 in cloud-init (Ubuntu Zesty) "cloud-init interface renaming should apply .lower() to mac_address values to match sysfs entries" [Medium,Confirmed] https://launchpad.net/bugs/1705147
[19:29] <ubot5> bug 1706752 in cloud-init (Ubuntu Zesty) "eni rendering broken for bridge params that require repeated key for values" [Medium,Confirmed] https://launchpad.net/bugs/1706752
[19:29] <smoser> similar template to
[19:29] <smoser>  https://bugs.launchpad.net/cloud-init/+bug/1684349
[19:29] <ubot5> Ubuntu bug 1684349 in cloud-init " mask2cidr error with integer value - argument of type 'int' is not iterable" [Medium,Fix committed]
[19:29] <smoser> we just have to come up with appropriate network config that would show these changes.
[19:42] <rharper> smoser: blackboxsw: most of the network configs that show it are taken from curtin's vmtests
[19:42] <rharper> I think many of the bugs include an example config that's broken as well
[19:52] <blackboxsw> smoser: thanks for confirmation on the patch preventing jsonschema dep.
[19:52] <blackboxsw> and agreed on using  net_convert