| blackboxsw | holmanb: I can confirm password authentication still works on jammy for "tom" per our integration tests. So we can be sure something didn't degrade as far as functionality or behavior in that respect. Digging further to assess the mechanics of the integration test failure and what useradd is doing under the hood that converts the hashed password to yescrypt | 00:21 |
|---|---|---|
| blackboxsw | on Jammy | 00:21 |
| holmanb | blackboxsw: +1 sounds good thanks for digging into that | 00:22 |
| esv | hey folks, I am trying to deploy a vm with a populated home directory, unfortunately the cloud-init version my deployment is using does not support the defer opion yet for write_files | 00:57 |
| esv | is there an alternate way to delay write_files? | 00:58 |
| esv | I am using 20.2 | 00:58 |
| kags2100 | blackboxsw Hey, did you find something in the logs? | 09:54 |
| Clouduser | Hi, I am looking for ansible role for cloudinit or cloud cfg jinja2 template | 14:13 |
| holmanb | Clouduser: Making sure I understand: You are trying to use ansible to create cloud-configs? | 15:37 |
| holmanb | Clouduser: I don't know if you have seen this already, but I see an ansible role one galaxy related to cloud init that points to this github repo: https://github.com/mrlesmithjr/ansible-cloud-init | 15:38 |
| holmanb | Clouduser: if you are asking about launching ansible on an instance using cloud-init, we don't have a module for that at this time. | 15:39 |
| esv | hey folks, about creating once per instance scripts, are these created using write_files? | 16:00 |
| esv | I don't see a key in the documentation, just making sure I'm not missing anything | 16:01 |
| falcojr | esv: yes, that's one way. There's also an open PR to allow creating such scripts in userdata as part of a multipart mime message | 16:05 |
| esv | thnx | 16:05 |
| falcojr | blackboxsw: I noticed while playing with https://hackmd.io/ilz5fDx_RpiofaMsO1B8KA?view (great write-up btw!) that I think LXD VMs aren't detecting the LXD datasource when they should | 16:16 |
| falcojr | on a jammy VM, ds-identify says it only found NoCloud, even though /dev/lxd/sock exists | 16:17 |
| falcojr | and LXD is in the datasource list | 16:17 |
| falcojr | in /etc/cloud | 16:18 |
| falcojr | don't really know the details of that particular file, but wondering if it doesn't exist until after ds-identify looks for it | 16:20 |
| blackboxsw | falcojr: thanks, as we just chatted, I was under the impression that no systems should actually come up from cloud-init as discovering LXD from the python code because cloudimage metadata still defines NoCloud template configuration which forces NoCloud datasource in front of LXD. As you clarified, we still want ds-identify to detect maybe/yes for LXD as a possible datasource to datasource match. | 17:38 |
| blackboxsw | As we discussed, we think LXD VM specifically not detecting /dev/lxd/sock is a race condition due to a latent lxd-agent setup and configuration in the client (which also prevents us from ssh ing into the VM early in boot as well). I don't think this affects the container images because the lxd-agent is activated in the images sooner. But yes let's card or bug it up and dig into that for LXD datasource support of VMs. | 17:40 |
| minimal | anyone free to review my cloud-utils PR #35? | 17:43 |
| blackboxsw | falcojr: holmanb LXD VM might be racy with the fact that /dev/lxd/sock file looks to get removed/recrated during lxd handler setup here https://github.com/lxc/lxd/blob/master/lxd-agent/devlxd.go#L226 ? | 17:47 |
| blackboxsw | minimal: what's the link sorry? | 17:48 |
| minimal | https://github.com/canonical/cloud-utils/pull/35 | 17:48 |
| ubottu | Pull 35 in canonical/cloud-utils "growpart: Add support for overprovisioning" [Open] | 17:48 |
| blackboxsw | minimal: ahh thx I misread cloud-init PR #35 and got lost :/ | 17:57 |
| blackboxsw | will get an initial review on that today | 17:58 |
| blackboxsw | context intent and docs look good. will test it out. | 18:00 |
| minimal | blackboxsw: thanks | 18:03 |
| kags2100 | blackboxsw Hey, sorry I can't see if you replied to my bug-report logs, did you have to look it over? | 18:19 |
| blackboxsw | kags2100: no worries I saw you dropped out of IRC earlier and was waiting for a ping. Your cloud-init.tar.gz isn't attached to the bug https://bugs.launchpad.net/cloud-init/+bug/1960360 I was going to launch an instance with some semblance of your user-data but didn't get around to it yet today. Do you have /var/log/cloud-init.log handy? If not I'll launch a VM today and double check | 18:21 |
| ubottu | Launchpad bug 1960360 in cloud-init "package_reboot_if_required breaks power_state: shutdown" [Undecided, New] | 18:21 |
| kags2100 | blackboxsw It isn't? I can see it as being attatched: Bug attachments | 18:25 |
| kags2100 | cloud-init.tar.gz (edit) | 18:25 |
| kags2100 | Add attachment | 18:25 |
| kags2100 | But I also added it as a comment, if that helps? | 18:25 |
| kags2100 | I can get it for you, if you can't use the log I already attached? | 18:26 |
| blackboxsw | kags2100: for your other comment "Btw, is there a way to manually set the user-data file while inside the VM? Right now I have to re-create the VM each time to pass the user-data file. Would be great if I could use that schema verification tool you showed me!" You should generally be able to use `sudo cloud-init clean --logs;` to make sure cloud-init now views the machine as green-field, (fresh boot of a new system) so all | 18:26 |
| blackboxsw | cloud-init config operations would/should run again | 18:26 |
| blackboxsw | kags2100: if you wanted to walk through all cloud-init's boot stages individually without rebooting I think it's something like `sudo cloud-nit init --local; sudo cloud-init init; sudo cloud-init modules --mode=config; sudo cloud-init modues --mode=final` These are the stages cloud-init systemd services exec during boot process as seen by `cat /lib/systemd/system/cloud-*` | 18:28 |
| blackboxsw | rechecking the bug again | 18:28 |
| blackboxsw | strange, now I see the attachement thanks | 18:28 |
| kags2100 | blackboxsw Yeah, did try that before, but had some issues where it didn't take the new UserData properly. But I'll try again, might just be that I had a formatting error, didn't really look into it much. | 18:28 |
| kags2100 | blackboxsw Hm, I'll try it out later this week then, thanks! | 18:30 |
| blackboxsw | kags2100: ok cloud-init analyze show -i <your_cloud-init_logs> shows three reboots on that system. I'll dig in now to see what's up | 18:30 |
| blackboxsw | kags2100: during first boot cloud-init marked a semaphore file due to successful run of |`->config-power-state-change ran successfully @111.93800s +00.01900s | 18:31 |
| blackboxsw | the 2nd boot it says is skipped due to that semaphore (because >config-power-state-change runs only once per instance-id change) `|`->config-power-state-change previously ran @15.37300s +00.00100s` | 18:32 |
| blackboxsw | and 3rd boot it also ignores it `|`->config-power-state-change previously ran @15.04100s +00.00000s` | 18:32 |
| blackboxsw | so during the first boot I'm thinking the /var/run/reboot-required flag didn't exist on the system. | 18:33 |
| kags2100 | Well that's weird that it reboots 3 times. It should only reboot once? That's the behavior I've seen at least | 18:33 |
| blackboxsw | I'll check the more details logs now to see what happened first boot | 18:34 |
| kags2100 | Yeah great | 18:34 |
| blackboxsw | quick review minimal on your PR. not sure what thoughts/preferences you have for typical over-provisioning use-cases. | 18:37 |
| blackboxsw | I dropped comments in the PR | 18:37 |
| blackboxsw | kags2100: ok I see tracebacks there in logs pointing to IO errors from invoking /sbin/reboot | 18:42 |
| blackboxsw | cloudinit.subp.ProcessExecutionError: Unexpected error while running command. | 18:42 |
| blackboxsw | Command: ['/sbin/reboot'] | 18:42 |
| blackboxsw | Exit code: - | 18:42 |
| blackboxsw | Reason: [Errno 5] Input/output error | 18:42 |
| blackboxsw | that's the symptom/problem on this system | 18:42 |
| blackboxsw | grep Traceback /var/log/cloud-init.log | 18:43 |
| kags2100 | Hm okay | 18:43 |
| kags2100 | So what does mean? | 18:43 |
| blackboxsw | cloud-init status --long on the system generally should also surface these errors to you. | 18:45 |
| blackboxsw | kags2100: I'm grasping at straws but your journalctl -b 0 shows "kernel reports TIME_ERROR: 0x41: Clock Unsynchronized" which can cause a mess of issues on a system if you have bogus clock values | 18:46 |
| kags2100 | blackboxsw Hm, true, but dosn't the system sync on boot? I did set the time service correct right? Also the VM has all outbound ports open, so that shouldn't be an issue either. | 18:48 |
| blackboxsw | kags2100: sorry am being pulled into interview prep at the moment. But investigating why /sbin/reboot isn't firing in that environment is likely our source of probs | 18:48 |
| kags2100 | blackboxsw No worries, thanks for all the help so far! Yeah, I'll try to look into it too later! | 18:49 |
| blackboxsw | no worries, any findings lets put in the bug | 18:50 |
| kags2100 | Sure thing! | 18:50 |
| minimal | blackboxsw: re: typical over-provisioning use cases - I use it for physical machines (e.g. Raspberry Pis with SD cards) where I only partition 90% of the SD card | 19:06 |
| kags2100 | blackboxsw Was just able to fix this! Needed to add a serial socket to the VM, so the Input/output error makes sense now! Added the fix to the report, but not sure I close it. Thanks for all your help though! | 20:57 |
| blackboxsw | woot!. thanks kags2100 for getting to the bottom of it ! | 21:37 |
| kags2100 | blackboxsw Well thanks for the support :) | 21:52 |
| kags2100 | blackboxsw Another question, my plan is to deploy a base template using the User-Data file. Then after run cloud-init clean, from that template I can then provision new VMs that are then setup for their specific purpose, like Docker Server, etc. Is this a good idea? | 21:56 |
| apple-corps[m] | I'm having trouble getting a user-data script deployed by terraform to a custom ami executed or seeing debugging information. I had a look at /var/lib/cloud/instance/user-data.txt and see the contents of the script deployed. There seems to be a cloud-init systemd unit on the system. When I set an invalid hostname in the script I recieved an error. However after removing I don't see any of the script actions take place, no error | 22:14 |
| apple-corps[m] | logs, etc. | 22:14 |
| apple-corps[m] | Can any cloud-init folks help me debug? | 22:14 |
| apple-corps[m] | Posted a greater question to the mailing list | 22:32 |
| blackboxsw | apple-corps[m]: so `sudo cloud-init devel schema --system` should tell you if your user-data is valid for the system | 22:35 |
| apple-corps[m] | Vlid cloud-config: system userdata | 22:35 |
| blackboxsw | +1. step 1 unlocked :) any script output that is run will be emitted to /var/log/cloud-init-output.log by default. So errors or traces might show up there | 22:36 |
| apple-corps[m] | So how can I tell what user the cloud-config is executed as? For example I added a line - echo "" > /tmp/cc under the runcmd: . I don't see the file created but almost any user should be able to write to /tmp/ | 22:38 |
| apple-corps[m] | Not seeing any logs in journalctl related to this | 22:38 |
| blackboxsw | apple-corps[m]: would have the ability to past the user-data so we can see the intent? `sudo cloud-init query userdata | pastebinit` should drop your entire userdata to paste.ubuntu.com | 22:41 |
| blackboxsw | apple-corps[m]: beware if you have passwords in user-data^ | 22:41 |
| blackboxsw | apple-corps[m]: runcmd section should write out a runcmd file in /var/ somewhere that cloud-init executes after converting the YAML content to shell via a shellify function. you'd be able to find that script output written with `grep runcmd /var/log/cloud-init.log` | 22:43 |
| blackboxsw | I'll fire off a container to double check the script file written by cloud-init. you'd be able to attempt to re-run that runcmd script on the live system after cloud-init boot to see if it fails as you expect or has bad formatting or something else | 22:43 |
| blackboxsw | /var/lib/cloud/instances/t-j/scripts/runcmd is what you want | 22:45 |
| blackboxsw | where "t-j" is your instance name | 22:45 |
| blackboxsw | if that script doesn't run the way you expect, something is off with your runcmd: YAML content | 22:46 |
| blackboxsw | and/or your might have a traceback or warning in /var/log/cloud-init.log that tells "Failed to shellify" | 22:48 |
| blackboxsw | this error would be seen on a system with `cloud-init status --long` | 22:48 |
| apple-corps[m] | blackboxsw: I'll get back with this in a short while. | 22:59 |
| blackboxsw | no worries reponded in mailinglist too | 23:01 |
| blackboxsw | *responded* | 23:02 |
| apple-corps[m] | I can't send the file from the host but I can reproduce and share here | 23:44 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!