| felipe1982 | where is the output from "echo" in a "boothook" script sent? | 06:17 |
|---|---|---|
| felipe1982 | not seen in /var/log/cloud-init-output.log | 06:18 |
| === cpaelzer__ is now known as cpaelzer | ||
| otubo | I have a small question about ds-identify: on line 1447 checks $DI_DSNAME, which is the data source specified on the configuration file. If it's specified, that's enough reason to end the script and return FOUND to the systemd generator. Am I correct? | 12:22 |
| otubo | I mean, it doesn't really checks if the data source specified in the configuration file really exists. | 12:22 |
| otubo | I see lots of commits form smoser, so I'll just ping :-) | 12:25 |
| smoser | otubo: yes, that is what it does | 12:36 |
| smoser | that would be specified as 'datasource=' in the ds-identify config. | 12:38 |
| smoser | which is not documented, somewhat by design. | 12:38 |
| smoser | https://bugs.launchpad.net/cloud-init/+bug/1838092 | 12:39 |
| ubot5 | Ubuntu bug 1838092 in cloud-init "Add documentation for ds-identify configuration" [Wishlist,Triaged] | 12:39 |
| smoser | and /me goes afk for a bit. | 12:39 |
| otubo | smoser, shouldn't it really check if the data source exists? Let me explain my issue: I'm deploying a VM with vSphere with network customization (aka data source). First boot ok, network is configured. Second boot, ds-identify fails to check that the data source is gone (because of this check I mentioned) and cloud-init is still enabled. With cloud-init enabled and no data source, it falls back to default configuration (dhcp in my case) | 12:40 |
| rharper | otubo: generally if the command line (or config on disk) is telling cloud-init which datasource to use, then there's nothing to check. If you do not hardcode the datasource in cmdline or in config, then ds-identify will do the checks to find the datasource | 12:51 |
| lfain | The cloud-init.service that is provided by cloud-init-19.2+10.g7f674256-1.el7.noarch.rpm for Centos7 doesn't contain "ConditionKernelCommandLine=!cloud-init=disabled" | 13:03 |
| lfain | Is it a bug? | 13:03 |
| Odd_Bloke | lfain: I believe the cloud-init generator reads the kernel cmdline and disables all the cloud-init services if that line exists. | 13:05 |
| Odd_Bloke | On an Ubuntu system, that generator is installed to /lib/systemd/system-generators/cloud-init-generator. | 13:07 |
| lfain | Unfortunately, cloud-init 19.2 ignores presence of "cloud-init=disabled" in the kernel command line in Centos7... | 13:07 |
| Odd_Bloke | lfain: Do you have that generator installed, do you know? | 13:08 |
| Odd_Bloke | (If it's not exactly in that path, try `find / -name cloud-init-generator`.) | 13:09 |
| lfain | Yes, it is installed. I just checked. | 13:09 |
| lfain | "/lib/systemd/system-generators/cloud-init-generator" | 13:10 |
| Odd_Bloke | lfain: OK, this is probably worth a bug report. Could you file one using https://bugs.launchpad.net/cloud-init/+filebug and make sure you attach the output of `cloud-init collect-logs` on a failing instance, please? | 13:12 |
| lfain | Ok, I will | 13:13 |
| otubo | rharper, doesn't that increases the boot time? On the second boot of my use case I get 1 min and 34 seconds of boot time just for cloud-init-loca.service. Also, even if I don't set the data source on ds-identify.cfg, ds-identify recognizes OVF (vmware) but resets the network configuration to dhcp | 13:13 |
| lfain | Odd_Bloke: Is it safe to add the "ConditionKernelCommandLine=!cloud-init=disabled" meantime? | 13:14 |
| rharper | otubo: ds-identify runs fast, it's reading data from /proc and sys; it does invoke blkid once; but the overhead isn't high at all. you can run cloud-init analyze show to see where time was spent. depdenting on the datasource, local may attempt to extract metadata from a network source, which means cloud-init will do a dhcp and then crawl metadata for network config. | 13:17 |
| rharper | 1m30 seconds *sounds* like a network timeout though | 13:17 |
| rharper | logs will tell the story | 13:17 |
| rharper | otubo: if someone specified a datasource that wasn't there but is network-based, then it's going to try to read a url without networking ... or wait for networkd-wait-online.service to complete | 13:18 |
| Odd_Bloke | lfain: You'd need to do it for every cloud-init unit. But, to be clear, I don't know for sure that it will do what you want. | 13:25 |
| otubo | rharper, I'm a little bit puzzled from this command, look at the output: https://pastebin.com/hHtSapG4 | 13:26 |
| otubo | rharper, on line 18 it says it couldn't find OVF data source, even if ds-identify recognizes it. | 13:26 |
| otubo | rharper, and on line 38 and 39 it spends a lot of time looking for network based configuration from EC2 and OpenStack (as you said above) | 13:27 |
| rharper | otubo: it does not look like ds-identify found an OVF, it looks completely disabled since cloud-init is searching all datasources | 13:27 |
| rharper | we recently fixed a but with the generator on RHEL systems where ds-identify didn't have the correct path | 13:28 |
| rharper | do you have a /run/cloud-init/ds-identify.log ? | 13:28 |
| otubo | rharper, ok, two important things to mention here: 1) I'm using version 18.5 (but we're open to upgrade if necessary); 2) I can see from ds-identify.log and from the function dscheck_OVF() that it returns FOUND if disable_vmware_customization is set to false (which is in this case) | 13:31 |
| rharper | otubo: on (2) do you see a /run/cloud-init/cloud.cfg ? | 13:32 |
| rharper | ds-identify should write out a simple config of datasource: [<Found ds name>, None] | 13:32 |
| rharper | then cloud-init will only search that list of datasources rather than the complete list; | 13:33 |
| otubo | rharper, yes datasource_list: [ OVF, None ] | 13:33 |
| rharper | and if you've a cloud-init.log then let's take a look | 13:33 |
| otubo | rharper, sure, hang on | 13:33 |
| otubo | rharper, You have exceeded the maximum paste size of 512 kilobytes per paste. hahahaha do you really need it? | 13:37 |
| rharper | that's a lot of reboots =) | 13:37 |
| rharper | just the first boot or two | 13:37 |
| otubo | rharper, as I understand ds-identify finds OVF because it is set on cloud.cfg (disable_vmware_customization to false) and it triggers it to NOT find the data source hence resetting to dhcp. Now the reason why cloud-init tries all other data sources I don't know. | 13:38 |
| rharper | what a mess; ds-identify can find OVF without it being set in config | 13:40 |
| rharper | disable_vmware_customization doesn't have anything to do with which datasource is selected, so that wouldn't make cloud-init search all datasources again | 13:40 |
| rharper | something else is going on; so the first boot or two logs | 13:40 |
| otubo | rharper, ok, hang on | 13:41 |
| otubo | rharper, this is the last reboot https://pastebin.com/80atbhBZ | 13:43 |
| otubo | rharper, disclaimer: there's some curse words in Portuguese as I was doing some debugs. | 13:43 |
| rharper | oh man, so .. OVF has a really rotten bug right now w.r.t instance-id, https://bugs.launchpad.net/cloud-init/+bug/1835205 | 13:44 |
| ubot5 | Ubuntu bug 1835205 in cloud-init "OVF datasource should check if instant id is still on VMware Platform" [Medium,Triaged] | 13:44 |
| rharper | 2019-08-15 13:06:35,076 - util.py[DEBUG]: waiting for configuration file took 90.096 seconds | 13:45 |
| rharper | that's your long boot time | 13:45 |
| otubo | rharper, yep, just saw it too. | 13:46 |
| rharper | the OVF datasource _resets_ it's instance-id on each boot so it could apply customizations after a first boot; I'm not sure how we let that slip into the OVF datasource but it's just wrong from a cloud-init perspective | 13:46 |
| otubo | rharper, and I just realized that `cloud-init analyze show' also shows other boots not only the current one. So hopefully ds-identify is doing its job correctly. So the only problem now is fixing this bug you just mentioned. | 13:47 |
| rharper | at least that bug needs a fix; there could be more issues; but it's hard to say without sorting out what to do. | 13:48 |
| otubo | rharper, I'm gonna try to contact Pengpeng Sun, perhaps we could work together. | 13:49 |
| rharper | otubo: if you have any vmware contacts, it would be great to reach out to them on this issue; if we don't have anything before the summit, I'd really like to get together then and walk through the vmware scenarios to see how we can support as many and without this instance-id trick | 13:49 |
| otubo | rharper, sounds good. BTW, I'll be at the Summit, just got my flight tickets. | 13:50 |
| rharper | o/ | 13:51 |
| * rharper raises other hand | 13:51 | |
| rharper | \o/ | 13:51 |
| otubo | rharper, I think I've been working with you for *at least* 8 years (since IBM..) and I have no idea how's your face :-D | 13:52 |
| rharper | hehe | 13:52 |
| rharper | I think I put something up on linkedin a while back | 13:52 |
| otubo | rharper, anyway, I'll work on this issue and will keep launchpad updated. Thanks for the help :-) | 13:53 |
| rharper | otubo: great! | 13:53 |
| rharper | if you have an existing RHBZ, feel free to link to that from launchpad as well, via the Also affects distribution/package link, you can specify distro and bug url | 13:54 |
| otubo | will do! | 13:58 |
| Odd_Bloke | rharper: So I'm looking at the dracut network configuration, and I was assuming this would be relatively easy as this is sysconfig. However, it looks like we don't have support for parsing sysconfig network configuration, only for rendering it. Am I missing something, or is that the case? | 14:27 |
| blackboxsw | rharper: do you have access to an OpenStack intsance w/ multi-nic and or bonds bridges where we can dump curl http://169.254.169.254/openstack/latest/network_data.json I was looking to avoid setting that up myself if you had a reference (this would be for my testing of OpenStack networking v2 | 14:57 |
| blackboxsw | s/a reference/an example/ ... I'm working on handling the unit test example coverage for v2, but wondered if you had any more complex examples in mind too | 14:58 |
| blackboxsw | Odd_Bloke: objections to my uploading to eoan? we have a couple of fixes, azure & oracle that could use some 'exposure' before feature freeze (and the rest of the oracle work is likely going to be a day or two) | 15:37 |
| Odd_Bloke | blackboxsw: Nope, except to observe we'll probably want to do an upload right before FF anyway. | 15:38 |
| blackboxsw | right +1. no concerns with more uploads :P | 15:38 |
| blackboxsw | ok kicking one off | 15:39 |
| blackboxsw | ok done [ubuntu/eoan-proposed] cloud-init 19.2-13-g2f3bb764-0ubuntu1 (Accepted) | 15:52 |
| rharper | blackboxsw: I was talking with jamespage about that; and it's not very common; I think maybe ironic backend; the closest to that we have would be something on rax baremetal; they deploy with bonds and vlans; | 16:34 |
| rharper | I have some older network_data.json we used in the original parsing, but the upstream "format" should still show all of the details | 16:34 |
| rharper | https://specs.openstack.org/openstack/nova-specs/specs/liberty/implemented/metadata-service-network-info.html and https://specs.openstack.org/openstack/nova-specs/specs/rocky/approved/multiple-fixed-ips-network-information.html (proposed but not yet landed I think) | 16:36 |
| rharper | Odd_Bloke: that's correct (we don't have a read-from-sysconfig -> network_state) path; there are sysconfig reader helpers in tree used for updating and rendering sysconfig from distro classes; | 16:37 |
| blackboxsw | thanks for the reference rharper | 16:44 |
| Odd_Bloke | rharper: OK, thanks, I'll look around for those. | 16:44 |
| rharper | cloudinit/distros/parsers/sys_conf.py | 16:45 |
| rharper | and cloudinit/distros/rhel_util.py | 16:45 |
| Odd_Bloke | Thanks! | 16:46 |
| blackboxsw | minor fix for ubuntu-drivers for review https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/371369 | 18:03 |
| Odd_Bloke | blackboxsw: Reviewed. | 18:37 |
| blackboxsw | Odd_Bloke: just saw. wrapping up comments now thanks | 18:37 |
| blackboxsw | Odd_Bloke: did you mention no sysconfig parsers in cloud-init? cloudinit/distros/parsers/sys_conf.py | 19:59 |
| blackboxsw | not sure if you already ran into that case | 19:59 |
| blackboxsw | rharper: ^? | 20:00 |
| rharper | blackboxsw: I passed it along | 20:00 |
| Odd_Bloke | blackboxsw: Yeah, rharper pointed me at it. :) | 20:00 |
| Odd_Bloke | Thanks! | 20:00 |
| blackboxsw | sorry missed that response | 20:00 |
| rharper | np | 20:00 |
| blackboxsw | just ran into it while looking at all our uses of ConfigObj in existing cloudinit | 20:01 |
| rharper | blackboxsw: I hope you saw my comment w.r.t the drivers branch | 20:02 |
| blackboxsw | hadn't yet, refreshing rharper | 20:03 |
| rharper | IIUC, the /etc/default/* stuff is shell sourceable, so you can't use spaces in variable assignment | 20:03 |
| blackboxsw | ahh | 20:03 |
| blackboxsw | will use load_shell_content instead | 20:04 |
| blackboxsw | thx | 20:04 |
| blackboxsw | hrm but as Odd_Bloke mentioned, once we load_shell_content and attempt re-writing the file for amendments, we'll lose pre-existing comments if added by someone outside of cloud-init | 20:06 |
| blackboxsw | not sure if we care or not, but if we are re-writing the file, then I guess we could add the # written by cloud-init header | 20:07 |
| blackboxsw | and we can cope with bugs and improve it if need be | 20:07 |
| rharper | we can load the content to see what's there and then decide how to update/modify | 20:10 |
| rharper | we could render via append with #header\n<content>\n#footer | 20:10 |
| blackboxsw | and I see we use return shlex.split(blob, comments=True) | 20:12 |
| blackboxsw | so I think we do have the ability to retain comments | 20:12 |
| blackboxsw | but maybe that's just on the specific line which contains a value | 20:12 |
| blackboxsw | will test | 20:12 |
| Odd_Bloke | My original comment doesn't stand as it's shell content. | 20:13 |
| Odd_Bloke | So we may just be able to stick with what you had? | 20:13 |
| blackboxsw | Odd_Bloke: rharper I think we do need to stick with what I had as load_shell_content actually strips comments, and trying to preserve comments across the shlex.split use is more work than just parsing line by line I feel. | 20:24 |
| rharper | blackboxsw: agreed; I really don't expect that file to have a lot of manual changes | 20:27 |
| blackboxsw | rharper: Odd_Bloke force pushed, comments addressed I think. | 20:31 |
| * blackboxsw force-pushed set of fixes to https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/371369 | 21:23 | |
| blackboxsw | woot! thx Odd_Bloke rharper on https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/370970 | 21:36 |
| * blackboxsw celebrates with a victory lap for the trello card | 21:36 | |
| rharper | nice | 21:36 |
| Odd_Bloke | \o/ | 21:39 |
| felipe1982 | where is the output from "echo" in a "boothook" script sent? not seen in /var/log/cloud-init-output.log | 22:44 |
| blackboxsw | felipe1982: it could be an error in the script preventing you from getting to execution. try running /var/lib/cloud/instance/boothooks/part-001 to see if you get an error or your echo | 22:55 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!