| meena | i'm sure this is an excellent start into this venture: 74 failed, 4551 passed, 5 skipped, 30 warnings in 60.18 seconds | 00:42 |
|---|---|---|
| === ananke_ is now known as ananke | ||
| === esv_ is now known as esv | ||
| benbrown | Despite setting the 'mode' for 'growpart' to 'off' in my multi-part configs, I see a debug message informing me that "No 'growpart' entry in cfg. Using default...". | 18:19 |
| benbrown | I can very clearly see it in /var/lib/cloud/instance/cloud-config.txt | 18:20 |
| benbrown | Any ideas as to what might be going on here? | 18:21 |
| blackboxsw | benbrown: can you paste the output of `sudo cloud-init query userdata` (Be wary before your paste if you have passwords or senstive config in your user-data | 18:32 |
| blackboxsw | I'm guessing offhand that it's a whitespace/YAML issue | 18:32 |
| blackboxsw | benbrown: another thought is that your full multi-part user-data is being ignored for one reason or another. You might see WARNINGS in /var/log/cloud-init.log like ` Unhandled non-multipart (text/x-not-multipart) userdata` telling you user-data is off | 18:38 |
| benbrown | https://pastebin.com/rhK42Yb1 | 18:38 |
| benbrown | I don't see the mentioned warning. | 18:39 |
| blackboxsw | hrm. mim message and boundary looks good... though I typically have base64encoded all the userdata in the mime message. What does /var/lib/cloud/instances/<some_UUID>/cloud-config.txt show you? | 18:51 |
| benbrown | There are two UUIDs | 18:52 |
| blackboxsw | hopefully the same #cloud-config header and content. | 18:52 |
| blackboxsw | benbrown: ahh also there seems to be a whitespace/indent problem in the mime #cloud-config content (or just in your paste of the content) | 18:53 |
| blackboxsw | in the paste I see a line starting with 'content:' which should be indented but isn't. | 18:54 |
| blackboxsw | same goes for 'encoding:' those should be indented within the scope of write_files: | 18:54 |
| blackboxsw | same w/ lines 15-16, indent under `growpart:` scope (yet paste doesn't show the inden | 18:55 |
| benbrown | Hmm, paste shows the indentation for me. | 18:56 |
| minimal | blackboxsw: they show intended as fine when viewing in "raw" mode | 18:56 |
| benbrown | One of the cloud-config.txt files for one of the UUIDs is empty | 18:56 |
| blackboxsw | ahh yep, I reviewed. the paste link it was being rendered in my IRC client without spaces. all good ther | 18:56 |
| benbrown | The one the /var/lib/cloud/instance symlink points to seems fine. | 18:56 |
| blackboxsw | benbrown: given that you see two UUIDs, I'm wondering if your datasource metadata switched the 'instance-id' represented in metadata forcing cloud-init to re-run again at a point in time when your user-data was no longer available/attached. | 18:57 |
| benbrown | The latter looks like https://pastebin.com/svTChuZb | 18:58 |
| blackboxsw | benbrown: what datasource/cloud/environment are you running in? I also wonder if `grep -i 'previous iid' /var/log/cloud-init.log` will show us a progression from NO_PREVIOUS_INSTANCE_ID, to the UUID that have your valid MIME user-data to a different UUID which doesn't. | 19:00 |
| blackboxsw | benbrown: `cloud-init query v1` will tell us your cloud/ platform information | 19:01 |
| benbrown | Hmm, yeah that grep shows NO_PREVIOUS_INSTANCE_ID, the invalid UUID twice, then the valid one. | 19:02 |
| benbrown | This is with openstack | 19:03 |
| benbrown | I'd guess the UUID with no userdata is likely the result of a packer build. | 19:06 |
| benbrown | Although based on the timestamps on those directories that guess seems incorrect. | 19:08 |
| blackboxsw | The cmd cloud-init query userdata should be representing the latest/current user-data provided to the instance at launch. and the symlink /var/lib/cloud/instance should be pointing to the 'right' current instance-UUID provided from Openstack | 19:10 |
| blackboxsw | benbrown: I think this may need a bit more debugging to properly triage the problem there. Would it be possible for you to file a bug at https://bugs.launchpad.net/cloud-init/+filebug and attach the tar file from cloud-init collect-logs? | 19:11 |
| blackboxsw | it'd be helpful in the bug to also note the fact that you are starting from packer build. | 19:12 |
| benbrown | So some of this is my confusion. The logs where it is running growpart are from the packer build, there are then later logs where growpart is correctly skipped "cc_growpart.py[DEBUG]: growpart disabled: mode=False". | 19:12 |
| blackboxsw | ultimately I want to walk through /var/log/cloud-init.log to understand the ordering if instance-id detection and what happened when vs the jounalctl output. | 19:12 |
| benbrown | That doesn't quite explain why the partition appears to have been extended to fill the disk though. But again perhaps I am misunderstanding. | 19:13 |
| blackboxsw | understandable confusion.... and cloud-init logs are very busy/unwieldy | 19:13 |
| benbrown | :) | 19:13 |
| benbrown | The image build via packer ends up being approx 15GiB, when I am running the snapshot as a new very large instance, `parted -l` shows that the partition fully takes up the 300GiB disk. | 19:15 |
| benbrown | blackboxsw: Cheers for the help. I'll submit a bug tomorrow after I've poked things about a bit more. | 19:16 |
| minimal | benbrown: perhaps at the end of the packer job you should delete the cloud-init.log and cloud-init-output.log? | 19:17 |
| blackboxsw | BTW that one off script provided to kick off ansible-playbook could be used in #cloud-config runcmd if needed (as write_files happens before runcmd section). Also there is a new 'ansible:' module that may let you use git or some other SCM to download and apply specific playbooks from a remote repo (if helpful) https://cloudinit.readthedocs.io/en/latest/reference/modules.html#ansible and example here too | 19:18 |
| blackboxsw | https://cloudinit.readthedocs.io/en/latest/reference/examples.html#install-and-run-ansible-pull | 19:18 |
| benbrown | Yeah, I was just thinking that would help reduce the confusion. | 19:18 |
| blackboxsw | +1 minimal benbrown on `sudo cloud-init clean --logs` | 19:18 |
| blackboxsw | it'll drop those logs and artifacts from the built image | 19:18 |
| blackboxsw | the ansible comment is just FYI in case interesting for your use-case | 19:19 |
| benbrown | Yeah, I did stumble across that module recently. I may well change it to use that, though I haven't entirely made my mind up. | 19:27 |
| benbrown | Ta for the runcmd tip. | 19:31 |
| meena | this gonna be a lot of work: https://github.com/canonical/cloud-init/pull/2003 | 22:15 |
| -ubottu:#cloud-init- Pull 2003 in canonical/cloud-init "move get_ib_* functions to cloudinit.distros.networking" [Open] | 22:15 | |
| blackboxsw | holmanb: ok looks like openstack/pycloudlib is currently unhappy as you mentioned. Something is amiss with pycloudlib trying to hit the action URL via python openstacksdk at 'servers/<MY_UUID>/action' with the {'addFloatingIp': {'address': 'W.X.Y.Z.Z'}} payload. I'm going to touch base w/ our openstack folks to see if a recent version uprev has incompatibilities with this operation as called by the python openstacksdk | 22:16 |
| holmanb | +1 thanks blackboxsw | 22:17 |
| blackboxsw | strange as I can invoke the CLI directly via `openstack server add floating ip <SERVER> <IP>` without issue. | 22:18 |
| meena | (this especially gonna be a lot of work, because I have no access to InfiniBand hardware…) | 22:18 |
| holmanb | fyi I just merged https://github.com/canonical/cloud-init/pull/1923, our (then) second-longest-running PR | 22:18 |
| -ubottu:#cloud-init- Pull 1923 in canonical/cloud-init "Added skip_detect_openstack option for OpenStack" [Merged] | 22:18 | |
| meena | holmanb: can't believe i haven't topped that yet… | 22:20 |
| meena | holmanb: and, yeah, I just pulled and saw that and added it to my report | 22:21 |
| blackboxsw | meena: developing in a sandbox :/ I'm not sure if we had access locally for infiniband hw for the other than I thought Azure had IB available on some HPC instances | 22:21 |
| meena | blackboxsw: i think Azure is definitely a maybe | 22:22 |
| meena | but I can't quite afford that… | 22:22 |
| blackboxsw | not sure. we might be able to work out spot testing once we have viable PR to ensure we avoid degrading on those instance types | 22:23 |
| meena | *nod | 22:23 |
| minimal | holmanb: darn, just noticed you added/updated downstream packaging links, must add Alpine | 23:42 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!