[06:29] <magicalChicken> smoser, rharper: Hey, I have fallback configuration working in (what I believe) is a mostly sane way
[06:29] <magicalChicken> lp:~wesley-wiedenmeier/cloud-init/net-fallback
[06:29] <smoser> reading
[06:29] <magicalChicken> Currently I am not attempting to rename the device to eth0, as that fails usually, but the code to do that is in place, a parameter just has to be changed
[06:31] <smoser> merge lines
[06:31] <magicalChicken> sure, just a second
[06:34] <magicalChicken> just pushed
[06:34] <smoser> ok
[06:34] <smoser> sitll ther ei think
[06:35] <magicalChicken> there's one or two that kidna make sense for separating sections, I can pop them out too
[06:35] <smoser> no. i'm complaining about the merge conflict lines
[06:35] <magicalChicken> oh, whoops
[06:35] <smoser> :)
[06:35] <magicalChicken> haha sorry about that i'm pretty tired :)
[06:36] <smoser> ok, so comments on what is see there.
[06:37] <smoser> find_fallback_network_device()
[06:37] <smoser> what id' like is somethign more like:
[06:37] <smoser>  generate_fallback_config()
[06:37] <magicalChicken> sure, that makes sense
[06:38] <smoser> and that would simply return {'config': {....}, 'version': 1}
[06:38] <smoser> the dictionary
[06:38] <magicalChicken> okay, and we could generate the .link in distros.debian?
[06:39] <magicalChicken> it might be okay to just omit the .link file for now until we can figure out the veth naming thing
[06:39] <smoser> right!
[06:39] <magicalChicken> haha awesome, okay
[06:39] <smoser> here... let me show you the diff i have.
[06:40] <smoser> http://paste.ubuntu.com/15470570/
[06:40] <smoser> so basically we call init.apply_networking()
[06:40] <smoser> and it decides which networking source to look at (via find_networking_config)
[06:41] <smoser> selects the right one, and then calls the distro's "apply" for that.
[06:41] <magicalChicken> yeah, that makes sense
[06:42] <smoser> the transfer stuff is sort of unrelated
[06:42] <magicalChicken> is that from trunk?
[06:42] <smoser> that diff is versus  my net1
[06:42] <magicalChicken> ah, okay
[06:42] <magicalChicken> i can merge in from there and apply the rest of the diff
[06:43] <smoser> dont worry about grabbing my paste that isnt in my branch yet
[06:44] <magicalChicken> okay sure
[06:44] <smoser> you get 'fallback_config()' to just return the config
[06:44] <magicalChicken> okay
[06:44] <smoser> make sense ?
[06:45] <magicalChicken> yeah, I'll get that done real quick
[06:53] <smoser> oh. and magicalChicken yrou return value should be a dict with 'config' and 'version'
[06:53] <magicalChicken> Oh, right yeah
[06:53] <magicalChicken> I've got the change made but something went wrong and the file didn't get written, so debugging rn
[06:54] <smoser> k
[06:57] <smoser> you had started taht from my net1 branch, right?
[06:58] <magicalChicken> yeah, but from around 1191, I think I merged at 1199 though
[07:04] <smoser> well, the one thing we wanat from your branch right now is the fallback_config()
[07:05] <smoser> so if you want to just quick re-do that, that'd be fine.
[07:05] <magicalChicken> just pushed, I think it has everything
[07:05] <magicalChicken> I renamed the function, and it generates the config fully formatted with the 'config' and 'version' keys
[07:08] <smoser> ok. i'll get it pulled into my branch and then push up. i will drop some of the other changes though for now.
[07:08] <magicalChicken> Awesome okay
[07:08] <magicalChicken> yeah, some of the other stuff was mainly for testing
[07:09] <magicalChicken> it was cool to see it boot and get a dev configured without the timeout, that'll be pretty nice in the future
[07:10] <smoser> i think you need to push
[07:10] <smoser> oh. i needed rto reload
[07:10] <magicalChicken> haha yeah, sometimes launchpad doesn't show new commits for a couple minutes
[07:11] <magicalChicken> oh, wait, I'm gonna remove the merge lines, I forgot to push that
[07:12] <magicalChicken> actually nvm, I think that was just launchpad failing to generate a diff
[07:12] <smoser> http://paste.ubuntu.com/15470624/
[07:12] <smoser> that is because i have enp0s25
[07:13] <magicalChicken> Aah, whoops, I thought str.strip would get the ones in the middle too, but I guess it's only at the beginning and end
[07:13] <magicalChicken> I can fix that real quick
[07:17] <smoser> k
[07:25] <magicalChicken> just pushed, it's fixed now, although the fix is kinda ugly, but I couldn't think of another way to do it on one line
[07:27] <smoser> k
[07:29] <magicalChicken> I think I'm gonna sign off for the night, that's pretty much everything I was working on done. I'll look into vmtests for this tomorrow
[07:44] <smoser> magicalChicken, thanks.
[07:54] <smoser> magicalChicken, it still needs some work, but good job
[07:54] <smoser> $ python3 -c 'from cloudinit import net; print(net.generate_fallback_config())'
[07:54] <smoser> {'config': {'routes': [], 'interfaces': {'vethVJLV0G': {'subnets': [{'type': 'dhcp4'}, {'type': 'dhcp6'}], 'type': 'physical', 'mode': 'manual', 'name': 'vethVJLV0G', 'mac_address': 'fe:15:28:38:df:27', 'inet': 'inet'}}, 'dns': {'search': [], 'nameservers': []}}, 'version': 1}
[07:54] <smoser> pulled yours into my branch
[09:38] <smoser> man...
[13:54] <rharper> smoser: if I'm reading trunk.net1 right , now with fallback from magicalChicken , we can use nocloud-net instead of nocloud for ds mode ?
[14:22] <rharper> smoser: which openstack did you use to get network_data via config drive ?
[16:13] <smoser> rharper, sorry. in now.
[16:13] <smoser> openstack that reads config drive is ConfigDrive
[16:13] <smoser> i had forgotten that yesterday too.
[16:13] <rharper> smoser: I meant which cloud
[16:13] <smoser> serverstack or canonistack will provide a config drive.
[16:13] <rharper> as you said, serverstack doesn't populate the metadata service with 'network_data' link
[16:13] <smoser> right.
[16:13] <rharper> yeah, I saw the config drive
[16:14] <smoser> so that sucks.
[16:14] <rharper> so I'm using the example json from the link
[16:14] <smoser> i dont know where i can get one....
[16:14] <rharper> but Ideally we'd be able to test this parser out
[16:14] <smoser> yeah
[16:14] <rharper> we'd have to standup one and enable it
[16:14] <smoser> let me search a couple public clouds and see if we can get network md anywhere for openstack.
[16:14] <rharper> or poke around any of the stacks you have access to
[16:14] <rharper> yeah
[16:14] <smoser> i'm not sure why we dont get it
[16:14] <rharper> so, one thing, I think the routes handling in the network stuff needs an adjustment
[16:15] <smoser> can you bother someone on opentsack team to see if you can get them to make it go on serverstack ?
[16:15] <rharper> yeah
[16:17] <rharper> I need to add support to the subnets section to allow one to specify a list of routes (which we already emit the 'up route ' stanzas for)
[16:17] <rharper> the 'networks' section in the config drive network bits is made just for that (network netmask gateway)
[16:44] <smoser> rharper, wel...
[16:44] <smoser> i got half
[16:45] <smoser> ssh root@162.253.53.94
[16:45] <smoser> ok. so half way good news.
[16:46] <smoser> serverstack has the network config in the MD , but not on the config drive
[16:46] <smoser> which is actually the same as i'm seeing on vexxhost where iw as able to launch an  instance after adding some networks.
[16:51] <smoser> suro-patz, you had some questions about networking and such, right ?
[17:02] <suro-patz> smoser: yes regarding https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1549403
[17:02] <suro-patz> basically I am working on MaaS equivalent in OpenStack, i.e. Ironic - and looking for configuring multiple IP (v4 & v6) on the target node through cloud-init
[17:04] <smoser> ah. ok.
[17:04] <smoser> is it config drive datasource ?
[17:05] <smoser> where will the network info come from
[17:06] <suro-patz> smoser: yes, config drive data source
[17:06] <suro-patz> Neutron
[17:07] <smoser> rharper, ^
[17:08] <smoser> that is exactly what we were wanting to work on next.
[17:09] <smoser> suro-patz, so in https://code.launchpad.net/~smoser/cloud-init/trunk.net1/
[17:09] <smoser> that branch, the code is mostly there to do this.
[17:09] <suro-patz> smoser: I will look into the diff
[17:09] <smoser> well at least the infrastructure.
[17:09] <suro-patz> that would be helpful
[17:10] <suro-patz> I made some changes to get it working for RHEL for us
[17:10] <smoser> suro-patz, the thing that is missing is the ConfigDrive datasource needs a 'network_config' method (see the one in DataSourceNoCloud)
[17:10] <suro-patz> but I would love to adhere to the infrastructure
[17:10] <suro-patz> smoser: I will check
[17:10] <smoser> and taht network_config basically needs to return a "network config" dictionary
[17:10] <smoser> and then, in theory, it'd all "just work"
[17:11] <smoser> ah.
[17:11] <smoser> there are 3 bits of work
[17:12] <smoser> a.) getting DataSourceCondfigDrive.network_config to return the right stuff.  That will require a parser/converter from openstack entworking format into 'network_config'
[17:12] <smoser> b.) some improvements to cloudinit/net/ for rendering that network config on ubuntu
[17:12] <smoser> c.) support for rendering that into Centos style networking
[17:12] <smoser> because i think you want centos, suro-patz
[17:13] <smoser> so now i have a question for you...
[17:13] <smoser> do you know how to convince nvoa to put a 'network_cofnig.json' on a libvirt config drive ?
[17:14] <suro-patz> smoser: I will work on that as I get things working end-to-end for me, I expect to start on that next week
[17:15] <smoser> --config-drive=1 gets me networking information in the metadata service but no network_data.json on the config drive
[17:15] <suro-patz> nova-compute uses a template to parse using jinja2
[17:15] <suro-patz> we can pass new template/conditionally
[17:15] <suro-patz> or we can modify the template too
[17:15] <suro-patz> I had to modify some of the nova-compute code too
[17:16] <suro-patz> once I get things working for me, which I expect by end of this week, I will work with the community to upstream
[17:17] <smoser> suro-patz, ?
[17:17] <smoser> template to parse?
[17:18] <suro-patz> nova-compute uses a template for the content file
[17:18] <smoser> i dont want interfaces
[17:19] <smoser> here. let me show.
[17:20] <suro-patz> smoser:https://github.com/openstack/nova/blob/master/nova/virt/netutils.py#L76
[17:20] <smoser> http://paste.ubuntu.com/15473498/
[17:21] <smoser> suro-patz, so i dont want the "/etc/network/intefaces" style file.
[17:21] <suro-patz> smoser: got your point
[17:21] <smoser> i want the more structured network_config.json
[17:22] <smoser> so the above came from an instance.
[17:22] <smoser> but the config drive does not have that file
[17:22] <smoser> :-(
[17:22] <suro-patz> I am not aware of nova-community's argument on why we should not put network_data.json
[17:22] <smoser> there is an argument ?
[17:22] <smoser> it really looksl ike its just busted
[17:23] <suro-patz> I am not aware of
[17:23] <smoser> i poked through the code, and it seems to be *trying* to do it.
[17:23] <suro-patz> we can fix it
[17:23] <suro-patz> even I felt the same
[17:23] <suro-patz> we do process meta_data.json
[17:23] <suro-patz> but the network config is in old format
[17:23] <smoser> right. i want to dump the old code.
[17:24] <smoser> not support the network_interfaces reading... well, leave it for dead i guess.
[17:24] <smoser> try to keep it working.
[17:24] <smoser> but in cloud-dinit want to focus on reading the network_data.json
[17:24] <suro-patz> smoser: will work in that direction
[17:24] <smoser> suro-patz, ggreat. one last thing...
[17:25] <smoser> for now the target is to read and apply network information from a local data source only
[17:25] <smoser> ie, config drive tells me what networking should look like, and we apply it before networking ever comes up
[17:26] <smoser> the more complex scenario of where you need *some* networking to get to network provided network configuration is for later.
[17:26] <smoser> that make sense ?
[17:26] <suro-patz> are you referring to metadata service sort of scenario
[17:27] <smoser> that branch does a good job (on ubuntu at least) of enforcing the NICs dont just come up before cloud-init has had a chance to write the system config.
[17:27] <smoser> suro-patz, yes,
[17:27] <smoser> metadata service.
[17:27] <suro-patz> its like bootstrap and download further config to apply
[17:27] <smoser> ie, for openstack metadata service, wed' have to first dhcp on some device (or bring up local networking) and then read the network config.
[17:27] <smoser> right.
[17:27] <suro-patz> only catch is they should not be conflicting
[17:28] <smoser> right.
[17:28] <smoser> the first step (config drive) is pretty easy
[17:28] <smoser> and almost done there on ubuntu. we write all the config before the OS gets a chance to bring it all up.  so we dont have to worry about changing it.
[17:29] <smoser> rharper, http://paste.ubuntu.com/15473498/ is a real world example.
[17:47] <rharper> smoser: nice!
[17:49] <rharper> smoser: nice and scary with 3 interfaces with dhcp =)
[17:50] <rharper> smoser: is there a definitive markup for what will get populated there?  services is new;  would like to see bridge examples as well
[17:52] <smoser> rharper, i dont know.
[17:52] <rharper> looks like the code
[17:53] <rharper> I looked at the above nova link, I can dig through there
[17:53] <smoser> i kind of doubt anything in openstack will be declaring bridges and such at this point
[17:53] <rharper> almost done parsing the json into network state so we can render eni
[17:53] <smoser> oh. /me remembers
[18:06] <smoser> how was it... jayf . works at rackspace on bare metal. they might have somethign like that.
[18:11] <smoser> rharper, JayF is going to get us some baremetal data
[18:11] <rharper> smoser: cool!
[18:11] <smoser> and suro-patz https://bugs.launchpad.net/nova/+bug/1513267 <-- that seems to be maybe what is missing
[18:12] <JayF> Just don't mine any dogecoin while you're on the box :P
[18:13]  * smoser googles dogecoin and assumes he should probably stop the bitcoin mining that he was doing :)
[18:14] <JayF> lol
[18:26] <smoser> rharper, jump on that box and look around.
[18:27] <smoser> JayF, mentions that the vlan stuff is patched in in baremetal
[18:27] <smoser> not in upstream.
[18:27] <suro-patz> smoser: with "sudo rm -rf /var/lib/instances/*; sudo rm -rf /var/log/cloud-init.log; sudo cloud-init init —local' did not reconfigure network from content/0000 - I had changed the content of the file, but it did not apply
[18:28] <smoser> hm..
[18:28] <suro-patz> any trick, how can I re-run cloud-init to reconfigure network, without reboot
[18:28] <suro-patz> even reboot also din't help
[18:28] <smoser> i dont knwo why that wouldnt have done it.
[18:29] <smatzek> rm -rf stuff under  /var/lib/cloud/data as well?
[18:30] <suro-patz> smatzek: no, I had not done that
[18:30] <suro-patz> will give a shot
[18:30] <JayF> smoser: I can probably even supply the nova patch we use if oyu want
[18:31] <smatzek> at a minimum you need to take  /var/lib/cloud/data/instance-id or change the UUID in it, or cloud-init sees the UUID hasn't changed and doesn't run
[18:31] <smatzek> take > take out
[18:32] <smoser> JayF, your metdata is odd
[18:32] <smoser> the config drive layout
[18:32] <smoser> has 'latest/' that has network_data.json in it
[18:32] <smoser> but the LIBERTY version (2015-10-15) does not
[18:32] <smoser> is that by design ?
[18:32] <JayF> probably means we haven't deployed that upstream bugfix you linked me in pm
[18:33] <smoser> well, JayF the bug was that it did not appear *anywhere*
[18:33] <JayF> oh. hm
[18:33] <JayF> I don't know why :)
[18:33] <JayF> Asking about stuff we did, at this point, 2y ago :)
[18:36] <smoser> rharper, for reference http://paste.ubuntu.com/15474058/
[18:37] <rharper> smoser: thanks!
[18:38] <smoser> rharper, and you might as well grab config-drive.img.gz and cnonfig-drive.tar.gz of that system. just in case.
[18:39] <rharper> k
[18:39] <rharper> got it
[18:42] <suro-patz> smatzek: works, after removing the instance-id! Cool!
[19:05] <smoser> JayF, we're done if you want to kill that thing.
[19:20] <randeffects__> nocloud question: if I use nocloud ds on a vmware vm does the disk image have to stay attached after the initial boot?  if I reboot the vm and the disk image is not attached, cloud init runs again with ds none.
[19:25] <smoser> randeffects__, yes
[19:25] <smoser> you're right :)
[19:25] <smoser> randeffects__, https://launchpad.net/bugs/1553815
[19:25] <smoser> that is fixed in trunk, and should soon be in xenial
[19:27] <smoser> but actually the fix i added for nocloud was to only check the seed direcdtories or the kernel command line (not mounting the disk)
[19:27] <randeffects__> smoser thanks
[19:27] <smoser> so... but you can get the behavior you want. you just have to be explicit
[19:27] <smoser> and set 'manual_cache_clean' in the image or in user data
[19:28] <smoser> actually, that wont work from user-data
[19:29] <smoser> i just put a comment in that bug
[19:38] <JayF> smoser: cool, hopefully it helped?
[19:40] <smoser> yeah. thank  you very much JayF .
[19:42] <JayF> np anytime I can help with something like that feel free to ask
[20:21] <vox_clamantis> hi all ... anyone know where to find info on configuring cloud-init to use OpenStack metadata service?
[20:27] <smoser> vox_clamantis, it should "just work".
[20:28] <vox_clamantis> :) i was afraid of that
[20:29] <vox_clamantis> it always picks another like configdrive or e2c. if i specify OpenStack as the only option in datasource_list it never finds it. although i have no problem accessing with curl
[20:30] <smoser> hm.
[20:30] <smoser> definitely if config drive is enabled it will select it.
[20:30] <smoser> but openstack shoudl run before ec2
[20:30] <smoser> check the order
[20:31] <vox_clamantis> let me try again. thanks for the suggestion
[20:38] <smoser> magicalChicken, around ?
[20:38] <magicalChicken> smoser: Yeah
[20:38] <smoser> hey....
[20:39] <smoser> shoot. sorry to ping you .
[20:39] <magicalChicken> it's okay
[20:39] <magicalChicken> is there anything I should be working on today aside from tests?
[20:39] <smoser> heres one
[20:39] <smoser> kernel command line
[20:40] <smoser> with your chagnes for kenrel command line in theory curtin should not need any patching of the image
[20:40] <smoser> ie, the ENI should be able to be left in tact
[20:40] <magicalChicken> Right yeah
[20:40] <smoser> so you could use your kernel command line branch
[20:40] <smoser> and integrate with net1
[20:40] <smoser> and then try vmtest for curtin
[20:40] <smoser> the other thing there...
[20:40] <magicalChicken> cool, okay
[20:40] <smoser> we should no longer need persistent names
[20:40] <smoser> in the curtin xenial command line
[20:40] <smoser> that make sense ? kernel cmdline ...
[20:41] <smoser>  net.ifnames=0
[20:41] <smoser> i never rememberr that string
[20:41] <magicalChicken> Okay yeah, that makes sense
[20:41] <smoser> i have to run *right now*
[20:41] <smoser> i'll check in later.
[20:41] <smoser> bye
[20:41] <magicalChicken> bye
[21:17] <rharper> smoser: ok, I've got something that converts the 3 different network-data json into network_state, when then we can render_eni etc;  I'm looking at cloudinit/source/DataSourceConfigDrive ; where do we determine which dir of the config drive to use? for example, the network_data.json is only in latest  ;
[21:20] <rharper> openstack helpers
[21:21] <rharper> OS_HAVANA is the latest, prolly should update that
[21:30]  * rharper relocates
[21:31] <smoser> rharper, did you actually see it in latest ?
[21:31] <smoser> because i've not seen it anywhere. althought hat file sys it will be.
[21:31] <smoser> i think we should assume correct working order of the cloud though and use the LIBERTY date
[21:32] <smoser> 'latest' is not really something you can use. they're explicilty stating there is a ABI. using latest/ is guaranteed to fail at some point.