| === gschanuel216 is now known as gschanuel21 | ||
| cjp256 | in 21.3, would we expect to see hotplug events for query? log: `hotplug-hook called with the following arguments: {hotplug_action: query, subsystem: net, udevaction: None, devpath: None}` | 20:20 |
|---|---|---|
| cjp256 | is that this? https://github.com/canonical/cloud-init/commit/1d01da5d9916d97ef463ba61a36b3f98f8911419#diff-de5dd7a5e8b755c029c220ce990fddd60abc9a24a860e60369d8866b85566a8fL12 | 20:21 |
| ubottu | Commit 1d01da5 in canonical/cloud-init "Add 'install hotplug' module (SC-476) (#1069)" | 20:21 |
| blackboxsw | cjp256: I believe you should see logs. but not actual queries unless someone opted in via user-data | 20:22 |
| blackboxsw | checking the version and commit where the necessary opt-in was added | 20:23 |
| cjp256 | I think the "query" may have come from [ "$(cloud-init devel hotplug-hook -s "${SUBSYSTEM}" query)" == "enabled" ]. I just looked at the udev rule and only expected to see add/remove events, not query | 20:25 |
| blackboxsw | 21.3 looks like had the expected "hotplug not enabled for event of type %s" checks I was curious about as in 21.2 something we didn't block events appropriately I believe. Ahh, and your question is more around how/why did we generate that log and yes your suggestion is correct. the initial query hotplug_enabled() in the hook-hotplug script is what generates that log trail. | 20:35 |
| blackboxsw | Ultimately upstream should do something more efficient than invoking python from those udev events if we can avoid it and determine hot-plug is not configured, but we haven't gotten back to that yet. | 20:36 |
| blackboxsw | the hotplug functionality has evolved a bit since the initial commit you referenced in 21.3. But I don't think we've gotten back to clean up and efficiency concerns yet for systems with high-load udev add/remove events | 20:37 |
| blackboxsw | I know specifically that post 21.3 we dropped that [ "$(cloud-init devel hotplug-hook -s "${SUBSYSTEM}" query)" == "enabled" ] as it was too costly per udev event. | 20:41 |
| blackboxsw | related to fixing this bug https://bugs.launchpad.net/bugs/1946003 | 20:42 |
| ubottu | Launchpad bug 1946003 in cloud-init "hotplug causing cloud-init to spike CPU usage" [Critical, Fix Released] | 20:42 |
| orndorffgrant | blackboxsw: quick question: in what systemd unit does the new /run/cloud-init/cloud-id file get written? (cloud-init-local.service or cloud-init.service or something else?) | 20:56 |
| orndorffgrant | Oh wait is it cloud-config.service? | 20:57 |
| blackboxsw | @orndorffgrant it depends on the datasource unfortunately as it happens during a successful completion DataSource.get_data() (#datasource discovery in python). Some datasources get detected in init-local . and some detected in cloud-init.service | 20:58 |
| blackboxsw | orndorffgrant for reference you can tell if a datasource support local stage discovery versus "network" stage based on what dependecies it claims to support | 21:00 |
| orndorffgrant | ahhh interesting okay cool. It's not a big deal as its looking like an `After: cloud-config.target` is working to make sure the cloud-id file is in place. I'm just investigating trying to create a scenario where the the cloud-id is _not_ in place in time e.g. `Before: cloud-init.service` | 21:00 |
| blackboxsw | With GCE case: the datasource could be detected in cloud-init-local.service timeframe or cloud-init.service timeframe as described by its two datasource dependency lines (one with just FILESYSTEM dependency, and the other datasource class gets triggered if for discovery once networking is up (in cloud-init.service) https://github.com/canonical/cloud-init/blob/main/cloudinit/sources/DataSourceGCE.py#L304-L312 | 21:02 |
| blackboxsw | orndorffgrant: right, the scenario where cloud-id not in place would be a datasource that is only detected at network timeframe (during cloud-init.service run). | 21:03 |
| orndorffgrant | cool, cool that makes sense - thanks for the info and pointers! | 21:04 |
| holmanb | curious that centos-stream is including as a dependency "prettytable" in their dependencies - that isn't a requirement imposed by upstream, right? | 21:04 |
| holmanb | ^^ just an oddity I noticed while poking at the missing httpretty dependencies in the failed copr builds | 21:05 |
| kags2100 | Hey, I seem to be unable to use the power_state statment if I also use "package_reboot_if_required: true". Nothing happens after it reboots? | 21:26 |
| blackboxsw | kags2100: what's `sudo cloud-init query user-data` show on your system (be wary if you have passwords in it) | 21:32 |
| kags2100 | blackboxsw Let me check, hold on. | 21:33 |
| blackboxsw | kags2100: I'd also check grep "Reboot" /var/log/cloud-init.log to make sure you got appripriate messages related to whether reboot was going to be triggered or ignored | 21:35 |
| blackboxsw | `grep Reboot /var/og/cloud-init.log` rather | 21:35 |
| blackboxsw | typo-central... | 21:36 |
| kags2100 | blackboxsw Starting the VM now. Here is my UserConfig file https://pastebin.com/QzMAdiZc | 21:38 |
| kags2100 | blackboxsw Running the grep command only results in this: 2022-02-07 23:45:20,468 - cc_package_update_upgrade_install.py[WARNING]: Rebooting after upgrade or install per /var/run/reboot-required | 21:40 |
| kags2100 | blackboxsw Running the query command just returns this: ERROR: Undefined instance-data key user-data | 21:44 |
| blackboxsw | kags2100: quick first note `sudo cloud-init devel schema --system` should raise a couple of errors for you in the format. | 21:44 |
| blackboxsw | sorry `sudo cloud-init query userdata` # no hyphen | 21:44 |
| kags2100 | blackboxsw Right yeah that looks like the UserConfig, everything is there. | 21:45 |
| blackboxsw | the errors from schema validation in the attached user-data I don't think apply to your specific problem but.... you should quote '0600' for permissions: and 'arches: [amd64]' instead of 'arches: amd64' | 21:46 |
| kags2100 | blackboxsw Thanks yeah, don't think it's what causing my problem. Since everything else works fine. Also saw I forgot to add the power_state to my paste bin, but it looks like this: | 21:48 |
| kags2100 | power_state: | 21:48 |
| kags2100 | delay: "+1" | 21:48 |
| kags2100 | mode: poweroff | 21:48 |
| kags2100 | timeout: 120 | 21:48 |
| kags2100 | condition: True | 21:48 |
| blackboxsw | yeah, hrm. I'd expect if reboot never happened to see "Rebooted, but still running after %s seconds or Reboot did not happen after %s seconds! | 21:49 |
| kags2100 | However I notice, that If I use this: | 21:49 |
| kags2100 | cloud_final_modules: | 21:49 |
| kags2100 | - power-state-change | 21:49 |
| blackboxsw | you said ubuntu 20.04 right? | 21:49 |
| kags2100 | But then it don't reboot for package upgrades instead. | 21:49 |
| kags2100 | Yup | 21:49 |
| kags2100 | To me (a linux and cloud-init novis) It feels like the first reboot breaks the power_state command. | 21:50 |
| blackboxsw | yeah like timing issue resetting things. ok, if possible kags2100 would you be able to capture /var/log/cloud-init.log for us somehow (you could run `sudo cloud-init collect-logs` if you wanted to an attach that content to a new bug for cloud-init at https://bugs.launchpad.net/cloud-init/+filebug | 21:52 |
| blackboxsw | otherwise you can paste that cloud-init log somewhere and we can grok what's happened there | 21:52 |
| blackboxsw | relative to the user-data you've provided | 21:52 |
| blackboxsw | holmanb: we at some point had a prettytable dependency in cloud-init but upstream ended up rolling our own to avoid the dependency | 21:54 |
| blackboxsw | holmanb: it might be stale | 21:54 |
| blackboxsw | holmanb: https://github.com/canonical/cloud-init/commit/f010594beb75e146091db47b7d72d1fc1d763e98 | 21:54 |
| ubottu | Commit f010594 in canonical/cloud-init "Remove prettytable dependency, introduce simpletable" | 21:54 |
| blackboxsw | where we dropped it upstream | 21:54 |
| blackboxsw | so centos just stale on pkging deps front | 21:55 |
| holmanb | blackboxsw: that makes sense | 21:56 |
| holmanb | their build process doesn't use the in-tree spec file, so that's on their end | 21:56 |
| kags2100 | blackboxsw Sure, just need to make an account :) | 21:56 |
| kags2100 | blackboxsw Just filed the bug report | 22:09 |
| kags2100 | Btw, is there a way to manually set the user-data file while inside the VM? Right now I have to re-create the VM each time to pass the user-data file. Would be great if I could use that schema verification tool you showed me! | 22:11 |
| kags2100 | blackboxsw Here is the bugreport btw: https://bugs.launchpad.net/cloud-init/+bug/1960360 | 22:13 |
| ubottu | Launchpad bug 1960360 in cloud-init "package_reboot_if_required breaks power_state: shutdown" [Undecided, New] | 22:13 |
| blackboxsw | thanks kags2100 ! | 22:17 |
| blackboxsw | will check your logs and provide context in a few | 22:17 |
| kags2100 | Thanks! | 22:18 |
| kags2100 | blackboxsw Gonna to to bed now, but I'll check in in the morning again! Night! | 22:53 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!