=== MrGeneral_ is now known as MrGeneral
[12:29] <andrew76> Recently I've noticed problematic behavior using cloud-init's "packages" functionality to install some apt packages on Ubuntu 20.04. It seems to install the packages fine, but then if I look after the fact, the packages contain no files at all
[12:29] <andrew76> if I run "dpkg -L <package>", I get "Package '<package>' does not contain any files (!)"; so the apt package is installed, but there are no files associated with it; any suggestions on how to fix this? Thanks!
[12:35] <meena> andrew76: what happens when you run `apt install -f`?
[12:36] <andrew76> meena it says "0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded."
[12:36] <meena> odd. which packages are we taking about?
[12:37] <andrew76> whatever I specify in the cloud-init config; for example, I install openssh-server so I can SSH into the system after cloud-init finishes, but this leaves it in an inaccessible state since sshd is not actually installed at all
[12:37] <andrew76> but I can reproduce it with any package I put in the cloud-init config
[12:38] <meena> openssh-server should really be part of your stock image
[12:39] <meena> there's no point making cloud-init install that, if you're going to need it on ca 99-101% (±1) of your machines 
[12:39] <andrew76> agreed, but I'm consuming the upstream LXD image "images:ubuntu/focal/desktop" or "images:ubuntu/jammy/desktop", so it's up to them whether or not it includes it by default (and in this case it does not so I need to install it)
[12:40] <meena> ah, okay
[12:40] <andrew76> so you're saying I should just take the upstream image, make my modifcations, and host a local "better" version with that already done?
[12:40] <meena> LXD is kinda different, right, cuz you can just lxd console into the machines
[12:40] <andrew76> yep
[12:40] <meena> but that doesn't change the fact that packages is broken.
[12:41] <andrew76> yes
[12:41] <meena> what does the debug log say?
[12:44] <andrew76> "util.py[DEBUG]: apt-install [eatmydata apt-get --option=Dpkg::Options::=--force-confold --option=Dpkg::options::=--force-unsafe-io --assume-yes --quiet install openssh-server ... took 25.031 seconds"
[12:44] <andrew76> "handlers.py[DEBUG]: finish: modules-final/config-package-update-upgrade-install: SUCCESS: config-package-update-upgrade-install ran successfully"
[12:44] <waldi> eatmydata…
[12:44] <andrew76> yeah I don't know what that is
[12:44] <waldi> something broken
[12:44] <meena> took 25.031 seconds
[12:44] <meena> wild
[12:45] <meena> can you rerun those commands in a container without --quiet?
[12:45] <andrew76> with "eatmydata" in the mix or not?
[12:48] <andrew76> running it exactly as-is with "eatmydata" and no --quiet, it says "openssh-server is already the newest version (1:8.9p1-3ubuntu0.1)."
[12:49] <andrew76> so it really does think the package is installed, but commands like "dpkg -L openssh-server" show that there are no files in the package
[12:49] <andrew76> I believe an "apt-get install --reinstall openssh-server" will fix it, but obviously this shouldn't be necessary
[12:57] <meena> what's eatmydata?
[12:57] <andrew76> https://launchpad.net/libeatmydata
[12:58] <meena> 12:48 <andrew76> running it exactly as-is with "eatmydata" and no --quiet, it says "openssh-server is already the newest version (1:8.9p1-3ubuntu0.1)." ⬅️ that's why I said on a fresh container
[12:58] <andrew76> ah okay, one sec
[13:00] <meena> Once we have it reproduced, we can remove eatmydata and see if that was the culprit 
[13:00] <andrew76> also I forgot but this is using LXD with "--vm", so this is actually a virtual machine, not a container
[13:01] <meena> you can test if that makes any difference after establishing *something*
[13:01] <meena> right now, we know nothing
[13:02] <andrew76> I had to manually run "apt-get install eatmydata" first to get it installed (I guess cloud-init must be doing this automatically) but once done I was then able to run the same above command and the files are successfully installed (dpkg -L returns all of them)
[13:03] <andrew76> in case it's relevant, I did see this error in the "apt-get install" output: "ERROR: ld.so: object 'libeatmydata.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored."
[13:04] <meena> can we get the full pastes in a pastebin of what you did, how exactly, age what, exactly the result was
[13:09] <meena> Also, is this: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1016422 the bug you're hitting?
[13:09] -ubottu:#cloud-init- Debian bug 1016422 in eatmydata "eatmydata: wrapper script has missed library path change" [Normal, Open]
[13:09] <andrew76> https://gist.github.com/amartin3225/49a47f28beb9ac75f7b30bbb4e62af4c
[13:11] <andrew76> that looks like it could be the bug; it seems like that's not actually affecting this problem though since it gets printed but the package installs successfully anyway
[13:15] <meena> why are you installing eatmydata, exactly? 
[13:16] <andrew76> I did in this case so I could run the exact same command as when using cloud-init. Normally I don't install eatmydata, but it seems to be present automatically when using cloud-init
[13:17] <andrew76> e.g. I don't know where it's coming from
[13:19] <andrew76> here is the cloud-init config I'm passing via LXD's "user.user-data" field: https://gist.github.com/amartin3225/c431769a61562e8094a68689782d8f36
[13:19] <meena> I wouldn't be too surprised if I installed a piece of software called eatmydata, and it… ate my data
[13:19] <meena> anyway, this isn't actually new
[13:19] <meena>  - support running apt with 'eatmydata' via configuration token apt_get_wrapper (LP: #1236531). 
[13:19] <andrew76> yeah, I don't want it, but for some reason it's being forced on these servers
[13:19] -ubottu:#cloud-init- Launchpad bug 1236531 in cloud-init (Ubuntu) "support running apt with eatmydata" [Undecided, Fix Released] https://launchpad.net/bugs/1236531
[13:21] <andrew76> there's no documentation about what it does, but maybe providing this "apt_get_wrapper" option in the cloud-init config will disable eatmydata? https://cloudinit.readthedocs.io/en/18.5/topics/examples.html?highlight=eatmydata#additional-apt-configuration
[13:21] <meena> i should hope so
[13:22] <meena> by it would be default, is beyond my imagination and research ability while only on the phone
[13:22] <meena> but also: I'm a FreeBSD developer, mostly
[13:22] <andrew76> yeah I don't understand why it's enabled by default either (or why it's not documented)
[13:23] <meena> We have things like ZFS, and don't like to lose data. maybe Linux folks tick differently 🤷🏻‍♀️ 
[13:24] <meena> try disabling it, see what that gets you, then file a bug
[13:34] <meena> I've been involved with cloud-init since 2017, and there's still corners here that surprise me
[13:36] <andrew76> it looks like setting apt_get_wrapper to disable eatmydata worked!
[13:39] <meena> please report this as, at least documentation Bug, and also, talk to the people in #ubuntu-server? #lxd? i dunno… maybe this is some (new?) anomaly in lxd
[13:41] <meena> bug: undocumented eatmydata wrapper actually eats my data!
[13:45] <andrew76> I'll work on making a bug report for cloud-init
[13:45] <andrew76> it seems like eatmydata should be disabled by default and you have opt-in if you want to risk losing your data (rather than being unaware of it)
[13:45] <andrew76> the default should be getting a usable server, not best performance IMO
[13:45] <andrew76> thank you for the help!
[13:47] <meena> come to the dark side (FreeBSD). we have cookies (GDPR compliant)
[14:10] <holmanb> andrew76: why are you using a desktop image? Cloud images should have the bits you want / need preinstalled.
[14:16] <meena> (including eatmydata) 
[14:18] <holmanb> eatmydata just doesn't fsync during package install, which yes, makes it possible to loose data, but I wouldn't expect it to cause the symptoms you describe
[14:20] <holmanb> From what you've mentioned about disabling eatmydata fixing your bug on a system that doesn't have it installed, I'm guessing that this issue is maybe somehow due to the fact that you don't have eatmydata installed, which would mean a packaging bug in cloud-init (missing dependency)
[14:27] <meena> holmanb: from the log, eatmydata was already installed, because it's a dependency to cloud-init 
[14:29] <holmanb> Ah, okay I haven't gotten to the log yet, I'll go back and read 
[14:31] <meena> took me five times to spot it
[14:35] <holmanb> Okay that's hilarious, I didn't realize eatmydata was just an ld_preload wrapper. I bet the whole library is 3 lines of code.
[14:36] <waldi> holmanb: it is not much more. and it's super fragile
[14:55] <andrew76> holmanb I'm using the desktop image because that's the particular flavor I'm interested in working with for this use case
[15:04] <holmanb> Shoveling, then a meeting - I'll take a look more after
[15:38] <SuperLag> I tryed --> runcmd: with some lines, in my userdata file. but that didn't work so well. 
[15:38] <SuperLag> (i.e. it didn't work at all)
[15:38] <SuperLag> I was trying to add an APT repo that wasn't clear how I'd get it to work with the apt: module, because of the GPG key.
[15:48] <meena> SuperLag: does your #cloud-config user-data validate? 
[15:50] <SuperLag> cloud-init schema -c userdata.yml 
[15:50] <holmanb> SuperLag: what's the problem with the GPG key and the apt module?
[15:50] <SuperLag> it returns --> Valid cloud-config: userdata.yaml
[15:51] <SuperLag> sorry, I missed the a in "yaml" in the first line
[15:51] <holmanb> SuperLag: cloud-init has ways of installing gpg keys, do the current ways not work for your use case?
[15:52] <SuperLag> https://www.hashicorp.com/blog/announcing-the-hashicorp-linux-repository#wget-o-https-apt-releases-hashicorp-com-gpg-gpg-dearmor-sudo-tee-usr-share-keyrings-hashicorp-archive-keyring-gpg
[15:53] <SuperLag> the directions there have you pull down the GPG key and install it, then there's some... what's the word? "interpolation"? in the command that sets up the repo.
[15:54] <meena> waldi: looks a bit more than three lines, https://git.launchpad.net/libeatmydata/tree/libeatmydata/libeatmydata.c
[15:54] <SuperLag> What I tried was just to have each of those two lines, exactly as written there for the APT users, in a runcmd: block
[15:55] <holmanb> andrew76: trying to repro, "images:ubuntu/jammy/desktop" isn't found, did you accidentally s/default/desktop/?
[15:55] <holmanb> and it's not listed here https://us.lxd.images.canonical.com/
[15:58] <meena> holmanb: they're out
[16:00] <holmanb> SuperLag: got it. Just wondering why not use  source: "deb [signed-by=$KEY_FILE] https://apt.releases.hashicorp.com $RELEASE main" with the apt module (under `sources`), and grab a copy of the gpg key and drop that in `key` (also under `sources`)
[16:00] <holmanb> thanks meena
[16:01] <SuperLag> holmanb: is "source:" the module in this case?
[16:02] <holmanb> SuperLag: no, it's a subkey under the apt module
[16:04] <holmanb> SuperLag: see "ignored5" and "my-repo4.list" in https://cloudinit.readthedocs.io/en/latest/reference/examples.html#additional-apt-configuration-and-repositories
[16:05] <holmanb> combining the source line in ignored 5 and the key in the my-repo4.list should be pretty close to what you want
[16:07] <holmanb> the interpolated release in hashicorp's example is handled with $RELEASE by cloud-init under that key
[16:15] <meena> 15:55 <holmanb> andrew76: trying to repro, "images:ubuntu/jammy/desktop" isn't found, did you accidentally s/default/desktop/?
[16:15] <meena> 15:55 <holmanb> and it's not listed here https://us.lxd.images.canonical.com/
[16:15] <holmanb> I found the bug and was able to launch with the invocation there
[16:16] <andrew76> yep if you do "lxc image list images: | grep ubuntu/jammy/desktop" you should see the image
[16:16] <andrew76> where the "images" remote's URL is https://images.linuxcontainers.org
[16:33] <SuperLag> Sigh.
[16:33] <SuperLag> Error:
[16:33] <SuperLag> Cloud config schema errors: apt: Additional properties are not allowed ('key', 'source' were unexpected)
[16:33] <SuperLag> https://pastebin.ubuntu.com/p/5NBkzVrr4v/
[16:34] <SuperLag> initially, right under "apt:" I had "hashicorp.list:" but it didn't like that either
[16:38] <meena> SuperLag: you sure this is cloud-init, and not something else, like, Ubuntu autoinstall?
[16:38] <SuperLag> I don't know what you mean
[16:39] <SuperLag> Aside from adding this repo, so far, I've gotten everything else to work with cloud-init. I don't even know what Ubuntu autoinstall is.
[16:41] <meena> yeah, me neither…
[16:43] <SuperLag> I'm confused.
[16:49] <holmanb> SuperLag: under the apt key you'll need the sources key, and under the sources key multiple dictionaries are accepted (the dict key can be hashicorp.list or whatever), under that dict is where the `source` and `key` keys  go
[16:52] <holmanb> so the content of the apt module in more of a json format would be something like this {apt: {sources: {hashicorp.list: {source: <src content>, key: <content>}}}}
[16:53] <SuperLag> if only I knew how to indent that massive key block with vim :D
[16:53] <holmanb> there's a couple of different ways
[16:54] <SuperLag> yessir, it's called VScode 😂
[16:54] <holmanb> easiest is to highlight the lines with visual block and then `shift - >`
[16:56] <SuperLag> yeah, it's not passing the validation, though
[16:58] <holmanb> it should tell you why
[16:58] <holmanb> for better details of what's going wrong try passing --annotate
[17:00] <SuperLag> :https://pastebin.ubuntu.com/p/YN8WhSj297/
[17:02] <SuperLag> ohhhh
[17:02] <SuperLag> I think I know what I got wrong
[17:02] <SuperLag> derp
[17:03] <SuperLag> yep, got it
[17:03] <SuperLag> had to reread what you said a couple times, then *click* 💡
[17:05] <holmanb> nice
[17:05] <SuperLag> So meena... what did you mean? 
[17:07] <holmanb> autoinstall is a completely different project that overloads the cloud-init user data cloud config with their own stuff (and also uses cloud-init unders the hood), so folks looking for autoinstall support often confuse cloud-init for autoinstall and find themselves here asking the wrong questions to the wrong people
[17:12] <SuperLag> oh crap... well, I broke *something* because I can't log in now 
[17:16] <SuperLag> Does attempting to add another repo somehow override the defaults?
[17:16] <holmanb> shouldn't
[17:20] <SuperLag> well, that's odd
[17:20] <SuperLag> add the block to add the hashi repo, and I can no longer log in, comment it out, and I'm good again
[17:21] <SuperLag> and *with* the block included, once I finally got it formatted right, it validates
[17:31] <SuperLag> yeah, adding that apt: block breaks everything
[19:29] <blackboxsw> SuperLag: sorry for jumping in late on this conversation.....  I'm wondering if your sources: line in apt in line 4 of https://pastebin.ubuntu.com/p/5NBkzVrr4v/ is getting variable expanded. ....  I think you'll want to single quote the value source: 'deb [signed-by=$KEYFILE] https://apt.releases.hashicorp.com $RELEASE main'
[19:33] <blackboxsw> SuperLag: sorry, I'm backing up further on that invalid user-data the top-level keys below apt:sources: are opaque names: https://cloudinit.readthedocs.io/en/latest/reference/modules.html#apt-configure
[19:35] <SuperLag> opaque names?
[19:37] <blackboxsw> SuperLag: sorry. Clarification needed. I'd expect your apt config user-data to be something like `apt:sources:somefriendlynamedoesnotmatter:source: 'deb ...'`. Here's the resulting paste: https://pastebin.ubuntu.com/p/tC9Bjpjytk/
[19:39] <blackboxsw> the docs try to say (albeit not very clearly) a key name is required for each sources entry. 
[19:39] <blackboxsw> ```
[19:39] <blackboxsw> sources: (object) Source list entries can be specified as a dictionary under the sources config key, with each key in the dict representing a different source file. The key of each source entry will be used as an id that can be referenced in other config entries, as well as the filename for the source’s configuration under /etc/apt/sources.list.d. If the name does not end with .list, it will be appended. If there is no configuration 
[19:39] <blackboxsw> for a key in sources, no file will be written, but the key may still be referred to as an id in other sources entries.
[19:39] <blackboxsw> ```
[19:40] <SuperLag> so no quotes around the source? 🤔
[19:40] <SuperLag> (looking at the resulting paste, and comparing)
[19:41] <blackboxsw> good catch. I think you want the single quotes around the r-value of source: 'deb .....' and per the inline example in the modules page it gives the same type of example
[19:41] <blackboxsw> ```
[19:41] <blackboxsw>   sources:
[19:41] <blackboxsw>       source1:
[19:41] <blackboxsw>           keyid: 'keyid'
[19:41] <blackboxsw>           keyserver: 'keyserverurl'
[19:41] <blackboxsw>           source: 'deb [signed-by=$KEY_FILE] http://<url>/ bionic main'
[19:41] <blackboxsw> ```
[19:48] <SuperLag> Hmm... one step forward, two steps back. 
[19:49] <SuperLag> I can log in now, but looking at the output in the logs, it's having issues adding the GPG key. "Decode error, failed to add key"
[19:51] <blackboxsw> good news is it is now seeing that config properly :/
[19:51] <SuperLag> 2023-02-15 19:45:54,082 - util.py[WARNING]: Running module apt-configure (<module 'cloudinit.config.cc_apt_configure' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_apt_configure.py'>) failed
=== lagash_ is now known as lagash
[20:14] <meena> is yaml literally the worst, or could we do worse? an ongoing question that DevOps settles every year with: YAML *is* the worst (and because we can't think of anything worse, we're sticking to it)
[20:15] <holmanb> I'd take yaml over xml for user configs always
[20:16] <holmanb> and yaml's not that bad esp when you get an environment that points out all the dumb errors for you
[20:16] <meena> holmanb: ever had to reconfigure a JBoss at 02:00 in the morning, with only Solaris vi (and no syntax highlighting)? 
[20:17] <holmanb> I've successfully avoided JBoss altogether ;)
[20:17] <holmanb> so no
[20:18] <holmanb> SuperLag: gotta be super careful with whitespace and format of the gpg key
[20:53] <meena> holmanb: imagine tomcat, but ca 40× more