[00:42] <meena> i'm sure this is an excellent start into this venture: 74 failed, 4551 passed, 5 skipped, 30 warnings in 60.18 seconds
[18:19] <benbrown> Despite setting the 'mode' for 'growpart' to 'off' in my multi-part configs, I see a debug message informing me that "No 'growpart' entry in cfg.  Using default...".
[18:20] <benbrown> I can very clearly see it in /var/lib/cloud/instance/cloud-config.txt
[18:21] <benbrown> Any ideas as to what might be going on here?
[18:32] <blackboxsw> benbrown: can you paste the output of `sudo cloud-init query userdata` (Be wary before your paste if you have passwords or senstive config in your user-data
[18:32] <blackboxsw> I'm guessing offhand that it's a whitespace/YAML issue
[18:38] <blackboxsw> benbrown: another thought is that your full multi-part user-data is being ignored for one reason or another. You might see WARNINGS in /var/log/cloud-init.log like ` Unhandled non-multipart (text/x-not-multipart) userdata` telling you user-data is off
[18:38] <benbrown> https://pastebin.com/rhK42Yb1
[18:39] <benbrown> I don't see the mentioned warning.
[18:51] <blackboxsw> hrm. mim message and boundary looks good... though I typically have base64encoded all the userdata in the mime message.  What does /var/lib/cloud/instances/<some_UUID>/cloud-config.txt show you?
[18:52] <benbrown> There are two UUIDs
[18:52] <blackboxsw> hopefully the same #cloud-config header and content.
[18:53] <blackboxsw> benbrown: ahh also there seems to be a whitespace/indent problem in the mime #cloud-config content (or just in your paste of the content)
[18:54] <blackboxsw> in the paste I see a line starting with 'content:'  which should be indented but isn't.
[18:54] <blackboxsw> same goes for 'encoding:' those should be indented within the scope of write_files:
[18:55] <blackboxsw> same w/ lines 15-16, indent under `growpart:` scope (yet paste doesn't show the inden
[18:56] <benbrown> Hmm, paste shows the indentation for me.
[18:56] <minimal> blackboxsw: they show intended as fine when viewing in "raw" mode
[18:56] <benbrown> One of the cloud-config.txt files for one of the UUIDs is empty
[18:56] <blackboxsw> ahh yep, I reviewed. the paste link it was being rendered in my IRC client without spaces. all good ther 
[18:56] <benbrown> The one the /var/lib/cloud/instance symlink points to seems fine.
[18:57] <blackboxsw> benbrown: given that you see two UUIDs, I'm wondering if your datasource metadata switched the 'instance-id' represented in metadata forcing cloud-init to re-run again at a point in time when your user-data was no longer available/attached.
[18:58] <benbrown> The latter looks like https://pastebin.com/svTChuZb
[19:00] <blackboxsw> benbrown: what datasource/cloud/environment are you running in? I also wonder if  `grep -i 'previous iid' /var/log/cloud-init.log`  will  show us a progression from NO_PREVIOUS_INSTANCE_ID, to the UUID that have your valid MIME user-data to a different UUID which doesn't.
[19:01] <blackboxsw> benbrown: `cloud-init query v1` will tell us your cloud/ platform information
[19:02] <benbrown> Hmm, yeah that grep shows NO_PREVIOUS_INSTANCE_ID, the invalid UUID twice, then the valid one.
[19:03] <benbrown> This is with openstack
[19:06] <benbrown> I'd guess the UUID with no userdata is likely the result of a packer build.
[19:08] <benbrown> Although based on the timestamps on those directories that guess seems incorrect.
[19:10] <blackboxsw> The cmd cloud-init query userdata should be representing the latest/current user-data provided to the  instance at launch. and the symlink /var/lib/cloud/instance should be pointing to the 'right' current instance-UUID provided from Openstack
[19:11] <blackboxsw> benbrown: I think this may need a bit more debugging to properly triage the problem there. Would it be possible for you to file a bug at  https://bugs.launchpad.net/cloud-init/+filebug  and attach the tar file from cloud-init collect-logs?
[19:12] <blackboxsw> it'd be helpful in the bug to also note the fact that you are starting from packer build.
[19:12] <benbrown> So some of this is my confusion. The logs where it is running growpart are from the packer build, there are then later logs where growpart is correctly skipped "cc_growpart.py[DEBUG]: growpart disabled: mode=False".
[19:12] <blackboxsw> ultimately I want to walk through /var/log/cloud-init.log to understand the ordering if instance-id detection and what happened when vs the jounalctl output.
[19:13] <benbrown> That doesn't quite explain why the partition appears to have been extended to fill the disk though. But again perhaps I am misunderstanding.
[19:13] <blackboxsw> understandable confusion.... and cloud-init logs are very busy/unwieldy 
[19:13] <benbrown> :)
[19:15] <benbrown> The image build via packer ends up being approx 15GiB, when I am running the snapshot as a new very large instance, `parted -l` shows that the partition fully takes up the 300GiB disk.
[19:16] <benbrown> blackboxsw: Cheers for the help. I'll submit a bug tomorrow after I've poked things about a bit more.
[19:17] <minimal> benbrown: perhaps at the end of the packer job you should delete the cloud-init.log and cloud-init-output.log?
[19:18] <blackboxsw> BTW that one off script provided to kick off ansible-playbook could be used in #cloud-config runcmd if needed (as write_files happens before runcmd section). Also there is a new 'ansible:' module that may let you use git or some other SCM to download and apply specific playbooks from a remote repo  (if helpful) https://cloudinit.readthedocs.io/en/latest/reference/modules.html#ansible and example here too  
[19:18] <blackboxsw> https://cloudinit.readthedocs.io/en/latest/reference/examples.html#install-and-run-ansible-pull
[19:18] <benbrown> Yeah, I was just thinking that would help reduce the confusion.
[19:18] <blackboxsw> +1 minimal benbrown on `sudo cloud-init clean  --logs`
[19:18] <blackboxsw> it'll drop those logs and artifacts from the built image
[19:19] <blackboxsw> the ansible comment is just FYI in case interesting for your use-case
[19:27] <benbrown> Yeah, I did stumble across that module recently. I may well change it to use that, though I haven't entirely made my mind up.
[19:31] <benbrown> Ta for the runcmd tip.
[22:15] <meena> this gonna be a lot of work: https://github.com/canonical/cloud-init/pull/2003
[22:15] -ubottu:#cloud-init- Pull 2003 in canonical/cloud-init "move get_ib_* functions to cloudinit.distros.networking" [Open]
[22:16] <blackboxsw> holmanb: ok looks like openstack/pycloudlib is currently unhappy as you mentioned. Something is amiss with pycloudlib trying to hit the action URL via python openstacksdk at 'servers/<MY_UUID>/action' with the {'addFloatingIp': {'address': 'W.X.Y.Z.Z'}}  payload. I'm going to touch base w/ our openstack folks to see if a recent version uprev has incompatibilities with this operation as called by the python openstacksdk
[22:17] <holmanb> +1 thanks blackboxsw
[22:18] <blackboxsw> strange as I can invoke the CLI directly via  `openstack server add floating ip <SERVER> <IP>` without issue.
[22:18] <meena> (this especially gonna be a lot of work, because I have no access to InfiniBand hardware…)
[22:18] <holmanb> fyi I just merged https://github.com/canonical/cloud-init/pull/1923, our (then) second-longest-running PR
[22:18] -ubottu:#cloud-init- Pull 1923 in canonical/cloud-init "Added skip_detect_openstack option for OpenStack" [Merged]
[22:20] <meena> holmanb: can't believe i haven't topped that yet…
[22:21] <meena> holmanb: and, yeah, I just pulled and saw that and added it to my report
[22:21] <blackboxsw> meena: developing in a sandbox :/ I'm not sure if we had access locally for infiniband hw for the other than I thought Azure had IB available on some HPC instances
[22:22] <meena> blackboxsw: i think Azure is definitely a maybe
[22:22] <meena> but I can't quite afford that…
[22:23] <blackboxsw> not sure. we might be able to work out spot testing once we have viable PR to ensure we avoid degrading on those instance types
[22:23] <meena> *nod
[23:42] <minimal> holmanb: darn, just noticed you added/updated downstream packaging links, must add Alpine