| blackboxsw | ananke: sorry vacation earlier this week. That user-data appears to upgrade all packages per `package_upgrade: true` those logs also appear to suggest something external to cloud-init in systemd ordering space (or package postinstalls) is disabling cloud-config.service and cloud-init.target. | 19:44 |
|---|---|---|
| blackboxsw | ananke: Is it possible that a package upgrade has some sort of systemd dependency loop pthat effectively disables cloud-init services or some upgraded deb package postinstall is trigger deactivation of cloud-init services? | 19:44 |
| ananke | blackboxsw: thanks for responding! I found the root cause this morning, and haven't had a chance to post a follow up. Come to find out the problem was caused by packer recipe missing our usual 'cloud-init --wait' very first step, and I'm baffled how it ever worked in the first place | 19:50 |
| ananke | later in the packer recipe there are apt commands/etc, which might have been the ones causing cloud-init to fail. I just could not pinpoint the connection | 19:51 |
| blackboxsw | ananke: ahh thanks for the circle back. Yes we'd seen a few cases in terraform and/or packer with early exit during pre-provisioning stages | 19:58 |
| blackboxsw | we've also seen semi-long timeouts from apt against some remote repos that exacerbated the problem with timing or multiple-package upgrades which were unable to complete before packer or terraform reset the node. | 19:59 |
| blackboxsw | like this one https://github.com/hashicorp/packer/issues/2639#issuecomment-493164778 | 20:02 |
| -ubottu:#cloud-init- Issue 2639 in hashicorp/packer "Option for builder to wait on cloud-init to complete" [Closed] | 20:02 | |
| ananke | it was the strangest thing that it worked just fine with last year's ami, repeatedly and 100%, but then it got uncovered in the new AMI. | 20:02 |
| ananke | yeah, it would be nice if packer had this feature. in the meantime, we make sure to have it as one of the earliest steps | 20:02 |
| blackboxsw | yeah, ❤️ the newer images breaking with the same user-data/meta-data. | 20:02 |
| ananke | and it would be different if those were newer versions of a given OS, but it's not the first time we've seen a linux distro introduce something midstream. I think it was also debian that caught us off-guard when they went from MBR based AMI to GPT one during one of their updates for 10 or 11 | 20:05 |
| minimal | a hybrid MBR/GPT AMI would be the obvious solution to that | 20:17 |
| minimal | though I'm not sure if that would work together with UEFI SecureBoot | 20:18 |
| falcojr | community notice: cloud-init chat will be moving to Matrix at https://matrix.to/#/#cloud-init:ubuntu.com ! You can see the full announcement at https://github.com/canonical/cloud-init/discussions/6418 . From now on, expect to find the upstream maintainers on Matrix and not here. | 20:45 |
| ananke | in this scenario the issue was fairly specific to our use case, where we copy the contents of processed/customized AMIs from various linux distros and produce smaller images. this particular step varies between distros and their major releases, but we typically do not expect it to change midstream | 20:46 |
| ananke | uhmm, bummer | 20:46 |
| === falcojr changed the topic of #cloud-init to: Join us at https://matrix.to/#/#cloud-init:ubuntu.com | ||
| blackboxsw | We'll definitely be floating in both platforms and channels for a while. But most projects at Canonical are moving over to Matrix for better searching, data persistence and lower barrier of entry for new users. Thanks @falcojr | 21:09 |
| minimal | I've not used Matrix at all so far for anything | 21:12 |
| minimal | my own "experience" of it is spam arriving in IRC channels from Matrix gateways :-( | 21:13 |
| minimal | s/own/only/ | 21:13 |
| blackboxsw | and same for some of the projects in canonical which migrated from IRC -> matrix. I think the company rejected the idea of bridging due to that spamming annoyance across bridges. | 21:18 |
| blackboxsw | I think fedora folks hit the same type of issue too https://discussion.fedoraproject.org/t/matrix-irc-bridge-causes-annoying-join-part-spam/29956/13 | 21:19 |
| minimal | falcojr: not having used Matrix before, is there some way to use an existing Launchpad account? Thunderbird wants a Username and Server for Matrix | 22:36 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!