[00:42] i'm sure this is an excellent start into this venture: 74 failed, 4551 passed, 5 skipped, 30 warnings in 60.18 seconds === ananke_ is now known as ananke === esv_ is now known as esv [18:19] Despite setting the 'mode' for 'growpart' to 'off' in my multi-part configs, I see a debug message informing me that "No 'growpart' entry in cfg. Using default...". [18:20] I can very clearly see it in /var/lib/cloud/instance/cloud-config.txt [18:21] Any ideas as to what might be going on here? [18:32] benbrown: can you paste the output of `sudo cloud-init query userdata` (Be wary before your paste if you have passwords or senstive config in your user-data [18:32] I'm guessing offhand that it's a whitespace/YAML issue [18:38] benbrown: another thought is that your full multi-part user-data is being ignored for one reason or another. You might see WARNINGS in /var/log/cloud-init.log like ` Unhandled non-multipart (text/x-not-multipart) userdata` telling you user-data is off [18:38] https://pastebin.com/rhK42Yb1 [18:39] I don't see the mentioned warning. [18:51] hrm. mim message and boundary looks good... though I typically have base64encoded all the userdata in the mime message. What does /var/lib/cloud/instances//cloud-config.txt show you? [18:52] There are two UUIDs [18:52] hopefully the same #cloud-config header and content. [18:53] benbrown: ahh also there seems to be a whitespace/indent problem in the mime #cloud-config content (or just in your paste of the content) [18:54] in the paste I see a line starting with 'content:' which should be indented but isn't. [18:54] same goes for 'encoding:' those should be indented within the scope of write_files: [18:55] same w/ lines 15-16, indent under `growpart:` scope (yet paste doesn't show the inden [18:56] Hmm, paste shows the indentation for me. [18:56] blackboxsw: they show intended as fine when viewing in "raw" mode [18:56] One of the cloud-config.txt files for one of the UUIDs is empty [18:56] ahh yep, I reviewed. the paste link it was being rendered in my IRC client without spaces. all good ther [18:56] The one the /var/lib/cloud/instance symlink points to seems fine. [18:57] benbrown: given that you see two UUIDs, I'm wondering if your datasource metadata switched the 'instance-id' represented in metadata forcing cloud-init to re-run again at a point in time when your user-data was no longer available/attached. [18:58] The latter looks like https://pastebin.com/svTChuZb [19:00] benbrown: what datasource/cloud/environment are you running in? I also wonder if `grep -i 'previous iid' /var/log/cloud-init.log` will show us a progression from NO_PREVIOUS_INSTANCE_ID, to the UUID that have your valid MIME user-data to a different UUID which doesn't. [19:01] benbrown: `cloud-init query v1` will tell us your cloud/ platform information [19:02] Hmm, yeah that grep shows NO_PREVIOUS_INSTANCE_ID, the invalid UUID twice, then the valid one. [19:03] This is with openstack [19:06] I'd guess the UUID with no userdata is likely the result of a packer build. [19:08] Although based on the timestamps on those directories that guess seems incorrect. [19:10] The cmd cloud-init query userdata should be representing the latest/current user-data provided to the instance at launch. and the symlink /var/lib/cloud/instance should be pointing to the 'right' current instance-UUID provided from Openstack [19:11] benbrown: I think this may need a bit more debugging to properly triage the problem there. Would it be possible for you to file a bug at https://bugs.launchpad.net/cloud-init/+filebug and attach the tar file from cloud-init collect-logs? [19:12] it'd be helpful in the bug to also note the fact that you are starting from packer build. [19:12] So some of this is my confusion. The logs where it is running growpart are from the packer build, there are then later logs where growpart is correctly skipped "cc_growpart.py[DEBUG]: growpart disabled: mode=False". [19:12] ultimately I want to walk through /var/log/cloud-init.log to understand the ordering if instance-id detection and what happened when vs the jounalctl output. [19:13] That doesn't quite explain why the partition appears to have been extended to fill the disk though. But again perhaps I am misunderstanding. [19:13] understandable confusion.... and cloud-init logs are very busy/unwieldy [19:13] :) [19:15] The image build via packer ends up being approx 15GiB, when I am running the snapshot as a new very large instance, `parted -l` shows that the partition fully takes up the 300GiB disk. [19:16] blackboxsw: Cheers for the help. I'll submit a bug tomorrow after I've poked things about a bit more. [19:17] benbrown: perhaps at the end of the packer job you should delete the cloud-init.log and cloud-init-output.log? [19:18] BTW that one off script provided to kick off ansible-playbook could be used in #cloud-config runcmd if needed (as write_files happens before runcmd section). Also there is a new 'ansible:' module that may let you use git or some other SCM to download and apply specific playbooks from a remote repo (if helpful) https://cloudinit.readthedocs.io/en/latest/reference/modules.html#ansible and example here too [19:18] https://cloudinit.readthedocs.io/en/latest/reference/examples.html#install-and-run-ansible-pull [19:18] Yeah, I was just thinking that would help reduce the confusion. [19:18] +1 minimal benbrown on `sudo cloud-init clean --logs` [19:18] it'll drop those logs and artifacts from the built image [19:19] the ansible comment is just FYI in case interesting for your use-case [19:27] Yeah, I did stumble across that module recently. I may well change it to use that, though I haven't entirely made my mind up. [19:31] Ta for the runcmd tip. [22:15] this gonna be a lot of work: https://github.com/canonical/cloud-init/pull/2003 [22:15] -ubottu:#cloud-init- Pull 2003 in canonical/cloud-init "move get_ib_* functions to cloudinit.distros.networking" [Open] [22:16] holmanb: ok looks like openstack/pycloudlib is currently unhappy as you mentioned. Something is amiss with pycloudlib trying to hit the action URL via python openstacksdk at 'servers//action' with the {'addFloatingIp': {'address': 'W.X.Y.Z.Z'}} payload. I'm going to touch base w/ our openstack folks to see if a recent version uprev has incompatibilities with this operation as called by the python openstacksdk [22:17] +1 thanks blackboxsw [22:18] strange as I can invoke the CLI directly via `openstack server add floating ip ` without issue. [22:18] (this especially gonna be a lot of work, because I have no access to InfiniBand hardware…) [22:18] fyi I just merged https://github.com/canonical/cloud-init/pull/1923, our (then) second-longest-running PR [22:18] -ubottu:#cloud-init- Pull 1923 in canonical/cloud-init "Added skip_detect_openstack option for OpenStack" [Merged] [22:20] holmanb: can't believe i haven't topped that yet… [22:21] holmanb: and, yeah, I just pulled and saw that and added it to my report [22:21] meena: developing in a sandbox :/ I'm not sure if we had access locally for infiniband hw for the other than I thought Azure had IB available on some HPC instances [22:22] blackboxsw: i think Azure is definitely a maybe [22:22] but I can't quite afford that… [22:23] not sure. we might be able to work out spot testing once we have viable PR to ensure we avoid degrading on those instance types [22:23] *nod [23:42] holmanb: darn, just noticed you added/updated downstream packaging links, must add Alpine