[06:17] <felipe1982> where is the output from "echo" in a "boothook" script sent?
[06:18] <felipe1982> not seen in /var/log/cloud-init-output.log
=== cpaelzer__ is now known as cpaelzer
[12:22] <otubo> I have a small question about ds-identify: on line 1447 checks $DI_DSNAME, which is the data source specified on the configuration file. If it's specified, that's enough reason to end the script and return FOUND to the systemd generator. Am I correct?
[12:22] <otubo> I mean, it doesn't really checks if the data source specified in the configuration file really exists.
[12:25] <otubo> I see lots of commits form smoser, so I'll just ping :-)
[12:36] <smoser> otubo: yes, that is what it does
[12:38] <smoser> that would be specified as 'datasource=' in the ds-identify config.
[12:38] <smoser> which is not documented, somewhat by design.
[12:39] <smoser> https://bugs.launchpad.net/cloud-init/+bug/1838092
[12:39] <ubot5> Ubuntu bug 1838092 in cloud-init "Add documentation for ds-identify configuration" [Wishlist,Triaged]
[12:39] <smoser> and /me goes afk for a bit.
[12:40] <otubo> smoser, shouldn't it really check if the data source exists? Let me explain my issue: I'm deploying a VM with vSphere with network customization (aka data source). First boot ok, network is configured. Second boot, ds-identify fails to check that the data source is gone (because of this check I mentioned) and cloud-init is still enabled. With cloud-init enabled and no data source, it falls back to default configuration (dhcp in my case)
[12:51] <rharper> otubo: generally if the command line (or config on disk) is telling cloud-init which datasource to use, then there's nothing to check.  If you do not hardcode the datasource in cmdline or in config, then ds-identify will do the checks to find the datasource
[13:03] <lfain> The cloud-init.service  that is provided by cloud-init-19.2+10.g7f674256-1.el7.noarch.rpm for Centos7 doesn't contain "ConditionKernelCommandLine=!cloud-init=disabled"
[13:03] <lfain> Is it a bug?
[13:05] <Odd_Bloke> lfain: I believe the cloud-init generator reads the kernel cmdline and disables all the cloud-init services if that line exists.
[13:07] <Odd_Bloke> On an Ubuntu system, that generator is installed to /lib/systemd/system-generators/cloud-init-generator.
[13:07] <lfain> Unfortunately, cloud-init 19.2 ignores presence of "cloud-init=disabled" in the kernel command line in Centos7...
[13:08] <Odd_Bloke> lfain: Do you have that generator installed, do you know?
[13:09] <Odd_Bloke> (If it's not exactly in that path, try `find / -name cloud-init-generator`.)
[13:09] <lfain> Yes, it is installed. I just checked.
[13:10] <lfain> "/lib/systemd/system-generators/cloud-init-generator"
[13:12] <Odd_Bloke> lfain: OK, this is probably worth a bug report.  Could you file one using https://bugs.launchpad.net/cloud-init/+filebug and make sure you attach the output of `cloud-init collect-logs` on a failing instance, please?
[13:13] <lfain> Ok, I will
[13:13] <otubo> rharper, doesn't that increases the boot time? On the second boot of my use case I get 1 min and 34 seconds of boot time just for cloud-init-loca.service. Also, even if I don't set the data source on ds-identify.cfg, ds-identify recognizes OVF (vmware) but resets the network configuration to dhcp
[13:14] <lfain> Odd_Bloke: Is it safe to add the "ConditionKernelCommandLine=!cloud-init=disabled" meantime?
[13:17] <rharper> otubo: ds-identify runs fast, it's reading data from /proc and sys;  it does invoke blkid once; but the overhead isn't high at all.   you can run cloud-init analyze show to see where time was spent.  depdenting on the datasource, local may attempt to extract metadata from a network source, which means cloud-init will do a dhcp and then crawl metadata for network config.
[13:17] <rharper> 1m30 seconds *sounds* like a network timeout though
[13:17] <rharper> logs will tell the story
[13:18] <rharper> otubo: if someone specified a datasource that wasn't there but is network-based, then it's going to try to read a url without networking ... or wait for networkd-wait-online.service to complete
[13:25] <Odd_Bloke> lfain: You'd need to do it for every cloud-init unit.  But, to be clear, I don't know for sure that it will do what you want.
[13:26] <otubo> rharper, I'm a little bit puzzled from this command, look at the output: https://pastebin.com/hHtSapG4
[13:26] <otubo> rharper, on line 18 it says it couldn't find OVF data source, even if ds-identify recognizes it.
[13:27] <otubo> rharper, and on line 38 and 39 it spends a lot of time looking for network based configuration from EC2 and OpenStack (as you said above)
[13:27] <rharper> otubo: it does not look like ds-identify found an OVF, it looks completely disabled since cloud-init is searching all datasources
[13:28] <rharper> we recently fixed a but with the generator on RHEL systems where ds-identify didn't have the correct path
[13:28] <rharper> do you have a /run/cloud-init/ds-identify.log ?
[13:31] <otubo> rharper, ok, two important things to mention here: 1) I'm using version 18.5 (but we're open to upgrade if necessary); 2) I can see from ds-identify.log and from the function dscheck_OVF() that it returns FOUND if disable_vmware_customization is set to false (which is in this case)
[13:32] <rharper> otubo: on (2) do you see a /run/cloud-init/cloud.cfg  ?
[13:32] <rharper> ds-identify should write out a simple config of datasource: [<Found ds name>, None]
[13:33] <rharper> then cloud-init will only search that list of datasources rather than the complete list;
[13:33] <otubo> rharper, yes datasource_list: [ OVF, None ]
[13:33] <rharper> and if you've a cloud-init.log  then let's take a look
[13:33] <otubo> rharper, sure, hang on
[13:37] <otubo> rharper, You have exceeded the maximum paste size of 512 kilobytes per paste. hahahaha do you really need it?
[13:37] <rharper> that's  a lot of reboots =)
[13:37] <rharper> just the first boot or two
[13:38] <otubo> rharper, as I understand ds-identify finds OVF because it is set on cloud.cfg (disable_vmware_customization to false) and it triggers it to NOT find the data source hence resetting to dhcp. Now the reason why cloud-init tries all other data sources I don't know.
[13:40] <rharper> what a mess;  ds-identify can find OVF without it being set in config
[13:40] <rharper> disable_vmware_customization doesn't have anything to do with which datasource is selected, so that wouldn't make cloud-init search all datasources again
[13:40] <rharper> something else is going on; so the first boot or two logs
[13:41] <otubo> rharper, ok, hang on
[13:43] <otubo> rharper, this is the last reboot https://pastebin.com/80atbhBZ
[13:43] <otubo> rharper, disclaimer: there's some curse words in Portuguese as I was doing some debugs.
[13:44] <rharper> oh man, so .. OVF has a really rotten bug right now w.r.t instance-id, https://bugs.launchpad.net/cloud-init/+bug/1835205
[13:44] <ubot5> Ubuntu bug 1835205 in cloud-init "OVF datasource should check if instant id is still on VMware Platform" [Medium,Triaged]
[13:45] <rharper> 2019-08-15 13:06:35,076 - util.py[DEBUG]: waiting for configuration file took 90.096 seconds
[13:45] <rharper> that's your long boot time
[13:46] <otubo> rharper, yep, just saw it too.
[13:46] <rharper> the OVF datasource _resets_ it's instance-id on each boot so it could apply customizations after a first boot;  I'm not sure how we let that slip into the OVF datasource but it's just wrong from a cloud-init perspective
[13:47] <otubo> rharper, and I just realized that `cloud-init analyze show' also shows other boots not only the current one. So hopefully ds-identify is doing its job correctly. So the only problem now is fixing this bug you just mentioned.
[13:48] <rharper> at least that bug needs a fix;  there could be more issues; but it's hard to say without sorting out what to do.
[13:49] <otubo> rharper, I'm gonna try to contact Pengpeng Sun, perhaps we could work together.
[13:49] <rharper> otubo: if you have any vmware contacts, it would be great to reach out to them on this issue;  if we don't have anything before the summit, I'd really like to get together then and walk through the vmware scenarios to see how we can support as many and without this instance-id trick
[13:50] <otubo> rharper, sounds good. BTW, I'll be at the Summit, just got my flight tickets.
[13:51] <rharper> o/
[13:51]  * rharper raises other hand
[13:51] <rharper> \o/
[13:52] <otubo> rharper, I think I've been working with you for *at least* 8 years (since IBM..) and I have no idea how's your face :-D
[13:52] <rharper> hehe
[13:52] <rharper> I think I put something up on linkedin a while back
[13:53] <otubo> rharper, anyway, I'll work on this issue and will keep launchpad updated. Thanks for the help :-)
[13:53] <rharper> otubo: great!
[13:54] <rharper> if you have an existing RHBZ, feel free to link to that from launchpad as well, via the  Also affects distribution/package  link, you can specify distro and bug url
[13:58] <otubo> will do!
[14:27] <Odd_Bloke> rharper: So I'm looking at the dracut network configuration, and I was assuming this would be relatively easy as this is sysconfig.  However, it looks like we don't have support for parsing sysconfig network configuration, only for rendering it.  Am I missing something, or is that the case?
[14:57] <blackboxsw> rharper: do you have access to an OpenStack intsance w/  multi-nic and or  bonds bridges where we can dump curl http://169.254.169.254/openstack/latest/network_data.json   I was looking to avoid setting that up myself if you had a reference (this would be for my testing of OpenStack networking v2
[14:58] <blackboxsw> s/a reference/an example/  ...  I'm working on handling the unit test example coverage for v2, but wondered if you had any more complex examples in mind too
[15:37] <blackboxsw> Odd_Bloke: objections to my uploading to eoan? we have a couple of fixes, azure & oracle that could use some 'exposure' before feature freeze (and the rest of the oracle work is likely going to be a day or two)
[15:38] <Odd_Bloke> blackboxsw: Nope, except to observe we'll probably want to do an upload right before FF anyway.
[15:38] <blackboxsw> right +1. no concerns with more uploads :P
[15:39] <blackboxsw> ok kicking one off
[15:52] <blackboxsw> ok done [ubuntu/eoan-proposed] cloud-init 19.2-13-g2f3bb764-0ubuntu1 (Accepted)
[16:34] <rharper> blackboxsw: I was talking with jamespage about that; and it's not very common; I think maybe ironic backend;  the closest to that we have would be something on rax baremetal; they deploy with bonds and vlans;
[16:34] <rharper> I have some older network_data.json we used in the original parsing, but the upstream "format" should still show all of the details
[16:36] <rharper> https://specs.openstack.org/openstack/nova-specs/specs/liberty/implemented/metadata-service-network-info.html and https://specs.openstack.org/openstack/nova-specs/specs/rocky/approved/multiple-fixed-ips-network-information.html (proposed but not  yet landed I think)
[16:37] <rharper> Odd_Bloke: that's correct (we don't have a read-from-sysconfig -> network_state) path;  there are sysconfig reader helpers in tree used for updating and rendering sysconfig from distro classes;
[16:44] <blackboxsw> thanks for the reference rharper
[16:44] <Odd_Bloke> rharper: OK, thanks, I'll look around for those.
[16:45] <rharper>  cloudinit/distros/parsers/sys_conf.py
[16:45] <rharper> and cloudinit/distros/rhel_util.py
[16:46] <Odd_Bloke> Thanks!
[18:03] <blackboxsw> minor fix for ubuntu-drivers for review https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/371369
[18:37] <Odd_Bloke> blackboxsw: Reviewed.
[18:37] <blackboxsw> Odd_Bloke: just saw. wrapping up comments now thanks
[19:59] <blackboxsw> Odd_Bloke: did you mention no sysconfig parsers in cloud-init? cloudinit/distros/parsers/sys_conf.py
[19:59] <blackboxsw> not sure if you already ran into that case
[20:00] <blackboxsw> rharper: ^?
[20:00] <rharper> blackboxsw: I passed it along
[20:00] <Odd_Bloke> blackboxsw: Yeah, rharper pointed me at it. :)
[20:00] <Odd_Bloke> Thanks!
[20:00] <blackboxsw> sorry missed that response
[20:00] <rharper> np
[20:01] <blackboxsw> just ran into it while looking at all our uses of ConfigObj in existing cloudinit
[20:02] <rharper> blackboxsw: I hope you saw my comment w.r.t the drivers branch
[20:03] <blackboxsw> hadn't yet, refreshing rharper
[20:03] <rharper> IIUC, the /etc/default/* stuff is shell sourceable, so you can't use spaces in variable assignment
[20:03] <blackboxsw> ahh
[20:04] <blackboxsw> will use load_shell_content instead
[20:04] <blackboxsw> thx
[20:06] <blackboxsw> hrm but as Odd_Bloke mentioned, once we load_shell_content and attempt re-writing the file for amendments, we'll lose pre-existing comments if added by someone outside of cloud-init
[20:07] <blackboxsw> not sure if we care or not, but if we are re-writing the file, then I guess we could add the # written by cloud-init header
[20:07] <blackboxsw> and we can cope with bugs and improve it if need be
[20:10] <rharper> we can load the content to see what's there and then decide how to update/modify
[20:10] <rharper> we could render via append with #header\n<content>\n#footer
[20:12] <blackboxsw> and I see we use         return shlex.split(blob, comments=True)
[20:12] <blackboxsw> so I think we do have the ability to retain comments
[20:12] <blackboxsw> but maybe that's just on the specific line which contains a value
[20:12] <blackboxsw> will test
[20:13] <Odd_Bloke> My original comment doesn't stand as it's shell content.
[20:13] <Odd_Bloke> So we may just be able to stick with what you had?
[20:24] <blackboxsw> Odd_Bloke: rharper I think we do need to stick with what I had as load_shell_content actually strips comments, and trying to preserve comments across the shlex.split use is more work than just parsing line by line I feel.
[20:27] <rharper> blackboxsw: agreed;  I really don't expect that file to have a lot of manual changes
[20:31] <blackboxsw> rharper: Odd_Bloke force pushed, comments addressed I think.
[21:23]  * blackboxsw force-pushed set of fixes to https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/371369
[21:36] <blackboxsw> woot! thx Odd_Bloke rharper on https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/370970
[21:36]  * blackboxsw celebrates with a victory lap for the trello card
[21:36] <rharper> nice
[21:39] <Odd_Bloke> \o/
[22:44] <felipe1982> where is the output from "echo" in a "boothook" script sent? not seen in /var/log/cloud-init-output.log
[22:55] <blackboxsw> felipe1982: it could be an error in the script preventing you from getting to execution. try running /var/lib/cloud/instance/boothooks/part-001 to see if you get an error or your echo