=== MrGeneral_ is now known as MrGeneral
andrew76Recently I've noticed problematic behavior using cloud-init's "packages" functionality to install some apt packages on Ubuntu 20.04. It seems to install the packages fine, but then if I look after the fact, the packages contain no files at all12:29
andrew76if I run "dpkg -L <package>", I get "Package '<package>' does not contain any files (!)"; so the apt package is installed, but there are no files associated with it; any suggestions on how to fix this? Thanks!12:29
meenaandrew76: what happens when you run `apt install -f`?12:35
andrew76meena it says "0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded."12:36
meenaodd. which packages are we taking about?12:36
andrew76whatever I specify in the cloud-init config; for example, I install openssh-server so I can SSH into the system after cloud-init finishes, but this leaves it in an inaccessible state since sshd is not actually installed at all12:37
andrew76but I can reproduce it with any package I put in the cloud-init config12:37
meenaopenssh-server should really be part of your stock image12:38
meenathere's no point making cloud-init install that, if you're going to need it on ca 99-101% (±1) of your machines 12:39
andrew76agreed, but I'm consuming the upstream LXD image "images:ubuntu/focal/desktop" or "images:ubuntu/jammy/desktop", so it's up to them whether or not it includes it by default (and in this case it does not so I need to install it)12:39
meenaah, okay12:40
andrew76so you're saying I should just take the upstream image, make my modifcations, and host a local "better" version with that already done?12:40
meenaLXD is kinda different, right, cuz you can just lxd console into the machines12:40
meenabut that doesn't change the fact that packages is broken.12:40
meenawhat does the debug log say?12:41
andrew76"util.py[DEBUG]: apt-install [eatmydata apt-get --option=Dpkg::Options::=--force-confold --option=Dpkg::options::=--force-unsafe-io --assume-yes --quiet install openssh-server ... took 25.031 seconds"12:44
andrew76"handlers.py[DEBUG]: finish: modules-final/config-package-update-upgrade-install: SUCCESS: config-package-update-upgrade-install ran successfully"12:44
andrew76yeah I don't know what that is12:44
waldisomething broken12:44
meenatook 25.031 seconds12:44
meenacan you rerun those commands in a container without --quiet?12:45
andrew76with "eatmydata" in the mix or not?12:45
andrew76running it exactly as-is with "eatmydata" and no --quiet, it says "openssh-server is already the newest version (1:8.9p1-3ubuntu0.1)."12:48
andrew76so it really does think the package is installed, but commands like "dpkg -L openssh-server" show that there are no files in the package12:49
andrew76I believe an "apt-get install --reinstall openssh-server" will fix it, but obviously this shouldn't be necessary12:49
meenawhat's eatmydata?12:57
meena12:48 <andrew76> running it exactly as-is with "eatmydata" and no --quiet, it says "openssh-server is already the newest version (1:8.9p1-3ubuntu0.1)." ⬅️ that's why I said on a fresh container12:58
andrew76ah okay, one sec12:58
meenaOnce we have it reproduced, we can remove eatmydata and see if that was the culprit 13:00
andrew76also I forgot but this is using LXD with "--vm", so this is actually a virtual machine, not a container13:00
meenayou can test if that makes any difference after establishing *something*13:01
meenaright now, we know nothing13:01
andrew76I had to manually run "apt-get install eatmydata" first to get it installed (I guess cloud-init must be doing this automatically) but once done I was then able to run the same above command and the files are successfully installed (dpkg -L returns all of them)13:02
andrew76in case it's relevant, I did see this error in the "apt-get install" output: "ERROR: ld.so: object 'libeatmydata.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored."13:03
meenacan we get the full pastes in a pastebin of what you did, how exactly, age what, exactly the result was13:04
meenaAlso, is this: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1016422 the bug you're hitting?13:09
-ubottu:#cloud-init- Debian bug 1016422 in eatmydata "eatmydata: wrapper script has missed library path change" [Normal, Open]13:09
andrew76that looks like it could be the bug; it seems like that's not actually affecting this problem though since it gets printed but the package installs successfully anyway13:11
meenawhy are you installing eatmydata, exactly? 13:15
andrew76I did in this case so I could run the exact same command as when using cloud-init. Normally I don't install eatmydata, but it seems to be present automatically when using cloud-init13:16
andrew76e.g. I don't know where it's coming from13:17
andrew76here is the cloud-init config I'm passing via LXD's "user.user-data" field: https://gist.github.com/amartin3225/c431769a61562e8094a68689782d8f3613:19
meenaI wouldn't be too surprised if I installed a piece of software called eatmydata, and it… ate my data13:19
meenaanyway, this isn't actually new13:19
meena - support running apt with 'eatmydata' via configuration token apt_get_wrapper (LP: #1236531). 13:19
andrew76yeah, I don't want it, but for some reason it's being forced on these servers13:19
-ubottu:#cloud-init- Launchpad bug 1236531 in cloud-init (Ubuntu) "support running apt with eatmydata" [Undecided, Fix Released] https://launchpad.net/bugs/123653113:19
andrew76there's no documentation about what it does, but maybe providing this "apt_get_wrapper" option in the cloud-init config will disable eatmydata? https://cloudinit.readthedocs.io/en/18.5/topics/examples.html?highlight=eatmydata#additional-apt-configuration13:21
meenai should hope so13:21
meenaby it would be default, is beyond my imagination and research ability while only on the phone13:22
meenabut also: I'm a FreeBSD developer, mostly13:22
andrew76yeah I don't understand why it's enabled by default either (or why it's not documented)13:22
meenaWe have things like ZFS, and don't like to lose data. maybe Linux folks tick differently 🤷🏻‍♀️ 13:23
meenatry disabling it, see what that gets you, then file a bug13:24
meenaI've been involved with cloud-init since 2017, and there's still corners here that surprise me13:34
andrew76it looks like setting apt_get_wrapper to disable eatmydata worked!13:36
meenaplease report this as, at least documentation Bug, and also, talk to the people in #ubuntu-server? #lxd? i dunno… maybe this is some (new?) anomaly in lxd13:39
meenabug: undocumented eatmydata wrapper actually eats my data!13:41
andrew76I'll work on making a bug report for cloud-init13:45
andrew76it seems like eatmydata should be disabled by default and you have opt-in if you want to risk losing your data (rather than being unaware of it)13:45
andrew76the default should be getting a usable server, not best performance IMO13:45
andrew76thank you for the help!13:45
meenacome to the dark side (FreeBSD). we have cookies (GDPR compliant)13:47
holmanbandrew76: why are you using a desktop image? Cloud images should have the bits you want / need preinstalled.14:10
meena(including eatmydata) 14:16
holmanbeatmydata just doesn't fsync during package install, which yes, makes it possible to loose data, but I wouldn't expect it to cause the symptoms you describe14:18
holmanbFrom what you've mentioned about disabling eatmydata fixing your bug on a system that doesn't have it installed, I'm guessing that this issue is maybe somehow due to the fact that you don't have eatmydata installed, which would mean a packaging bug in cloud-init (missing dependency)14:20
meenaholmanb: from the log, eatmydata was already installed, because it's a dependency to cloud-init 14:27
holmanbAh, okay I haven't gotten to the log yet, I'll go back and read 14:29
meenatook me five times to spot it14:31
holmanbOkay that's hilarious, I didn't realize eatmydata was just an ld_preload wrapper. I bet the whole library is 3 lines of code.14:35
waldiholmanb: it is not much more. and it's super fragile14:36
andrew76holmanb I'm using the desktop image because that's the particular flavor I'm interested in working with for this use case14:55
holmanbShoveling, then a meeting - I'll take a look more after15:04
SuperLagI tryed --> runcmd: with some lines, in my userdata file. but that didn't work so well. 15:38
SuperLag(i.e. it didn't work at all)15:38
SuperLagI was trying to add an APT repo that wasn't clear how I'd get it to work with the apt: module, because of the GPG key.15:38
meenaSuperLag: does your #cloud-config user-data validate? 15:48
SuperLagcloud-init schema -c userdata.yml 15:50
holmanbSuperLag: what's the problem with the GPG key and the apt module?15:50
SuperLagit returns --> Valid cloud-config: userdata.yaml15:50
SuperLagsorry, I missed the a in "yaml" in the first line15:51
holmanbSuperLag: cloud-init has ways of installing gpg keys, do the current ways not work for your use case?15:51
SuperLagthe directions there have you pull down the GPG key and install it, then there's some... what's the word? "interpolation"? in the command that sets up the repo.15:53
meenawaldi: looks a bit more than three lines, https://git.launchpad.net/libeatmydata/tree/libeatmydata/libeatmydata.c15:54
SuperLagWhat I tried was just to have each of those two lines, exactly as written there for the APT users, in a runcmd: block15:54
holmanbandrew76: trying to repro, "images:ubuntu/jammy/desktop" isn't found, did you accidentally s/default/desktop/?15:55
holmanband it's not listed here https://us.lxd.images.canonical.com/15:55
meenaholmanb: they're out15:58
holmanbSuperLag: got it. Just wondering why not use  source: "deb [signed-by=$KEY_FILE] https://apt.releases.hashicorp.com $RELEASE main" with the apt module (under `sources`), and grab a copy of the gpg key and drop that in `key` (also under `sources`)16:00
holmanbthanks meena16:00
SuperLagholmanb: is "source:" the module in this case?16:01
holmanbSuperLag: no, it's a subkey under the apt module16:02
holmanbSuperLag: see "ignored5" and "my-repo4.list" in https://cloudinit.readthedocs.io/en/latest/reference/examples.html#additional-apt-configuration-and-repositories16:04
holmanbcombining the source line in ignored 5 and the key in the my-repo4.list should be pretty close to what you want16:05
holmanbthe interpolated release in hashicorp's example is handled with $RELEASE by cloud-init under that key16:07
meena15:55 <holmanb> andrew76: trying to repro, "images:ubuntu/jammy/desktop" isn't found, did you accidentally s/default/desktop/?16:15
meena15:55 <holmanb> and it's not listed here https://us.lxd.images.canonical.com/16:15
holmanbI found the bug and was able to launch with the invocation there16:15
andrew76yep if you do "lxc image list images: | grep ubuntu/jammy/desktop" you should see the image16:16
andrew76where the "images" remote's URL is https://images.linuxcontainers.org16:16
SuperLagCloud config schema errors: apt: Additional properties are not allowed ('key', 'source' were unexpected)16:33
SuperLaginitially, right under "apt:" I had "hashicorp.list:" but it didn't like that either16:34
meenaSuperLag: you sure this is cloud-init, and not something else, like, Ubuntu autoinstall?16:38
SuperLagI don't know what you mean16:38
SuperLagAside from adding this repo, so far, I've gotten everything else to work with cloud-init. I don't even know what Ubuntu autoinstall is.16:39
meenayeah, me neither…16:41
SuperLagI'm confused.16:43
holmanbSuperLag: under the apt key you'll need the sources key, and under the sources key multiple dictionaries are accepted (the dict key can be hashicorp.list or whatever), under that dict is where the `source` and `key` keys  go16:49
holmanbso the content of the apt module in more of a json format would be something like this {apt: {sources: {hashicorp.list: {source: <src content>, key: <content>}}}}16:52
SuperLagif only I knew how to indent that massive key block with vim :D16:53
holmanbthere's a couple of different ways16:53
SuperLagyessir, it's called VScode 😂16:54
holmanbeasiest is to highlight the lines with visual block and then `shift - >`16:54
SuperLagyeah, it's not passing the validation, though16:56
holmanbit should tell you why16:58
holmanbfor better details of what's going wrong try passing --annotate16:58
SuperLagI think I know what I got wrong17:02
SuperLagyep, got it17:03
SuperLaghad to reread what you said a couple times, then *click* 💡17:03
SuperLagSo meena... what did you mean? 17:05
holmanbautoinstall is a completely different project that overloads the cloud-init user data cloud config with their own stuff (and also uses cloud-init unders the hood), so folks looking for autoinstall support often confuse cloud-init for autoinstall and find themselves here asking the wrong questions to the wrong people17:07
SuperLagoh crap... well, I broke *something* because I can't log in now 17:12
SuperLagDoes attempting to add another repo somehow override the defaults?17:16
SuperLagwell, that's odd17:20
SuperLagadd the block to add the hashi repo, and I can no longer log in, comment it out, and I'm good again17:20
SuperLagand *with* the block included, once I finally got it formatted right, it validates17:21
SuperLagyeah, adding that apt: block breaks everything17:31
blackboxswSuperLag: sorry for jumping in late on this conversation.....  I'm wondering if your sources: line in apt in line 4 of https://pastebin.ubuntu.com/p/5NBkzVrr4v/ is getting variable expanded. ....  I think you'll want to single quote the value source: 'deb [signed-by=$KEYFILE] https://apt.releases.hashicorp.com $RELEASE main'19:29
blackboxswSuperLag: sorry, I'm backing up further on that invalid user-data the top-level keys below apt:sources: are opaque names: https://cloudinit.readthedocs.io/en/latest/reference/modules.html#apt-configure19:33
SuperLagopaque names?19:35
blackboxswSuperLag: sorry. Clarification needed. I'd expect your apt config user-data to be something like `apt:sources:somefriendlynamedoesnotmatter:source: 'deb ...'`. Here's the resulting paste: https://pastebin.ubuntu.com/p/tC9Bjpjytk/19:37
blackboxswthe docs try to say (albeit not very clearly) a key name is required for each sources entry. 19:39
blackboxswsources: (object) Source list entries can be specified as a dictionary under the sources config key, with each key in the dict representing a different source file. The key of each source entry will be used as an id that can be referenced in other config entries, as well as the filename for the source’s configuration under /etc/apt/sources.list.d. If the name does not end with .list, it will be appended. If there is no configuration 19:39
blackboxswfor a key in sources, no file will be written, but the key may still be referred to as an id in other sources entries.19:39
SuperLagso no quotes around the source? 🤔19:40
SuperLag(looking at the resulting paste, and comparing)19:40
blackboxswgood catch. I think you want the single quotes around the r-value of source: 'deb .....' and per the inline example in the modules page it gives the same type of example19:41
blackboxsw  sources:19:41
blackboxsw      source1:19:41
blackboxsw          keyid: 'keyid'19:41
blackboxsw          keyserver: 'keyserverurl'19:41
blackboxsw          source: 'deb [signed-by=$KEY_FILE] http://<url>/ bionic main'19:41
SuperLagHmm... one step forward, two steps back. 19:48
SuperLagI can log in now, but looking at the output in the logs, it's having issues adding the GPG key. "Decode error, failed to add key"19:49
blackboxswgood news is it is now seeing that config properly :/19:51
SuperLag2023-02-15 19:45:54,082 - util.py[WARNING]: Running module apt-configure (<module 'cloudinit.config.cc_apt_configure' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_apt_configure.py'>) failed19:51
=== lagash_ is now known as lagash
meenais yaml literally the worst, or could we do worse? an ongoing question that DevOps settles every year with: YAML *is* the worst (and because we can't think of anything worse, we're sticking to it)20:14
holmanbI'd take yaml over xml for user configs always20:15
holmanband yaml's not that bad esp when you get an environment that points out all the dumb errors for you20:16
meenaholmanb: ever had to reconfigure a JBoss at 02:00 in the morning, with only Solaris vi (and no syntax highlighting)? 20:16
holmanbI've successfully avoided JBoss altogether ;)20:17
holmanbso no20:17
holmanbSuperLag: gotta be super careful with whitespace and format of the gpg key20:18
meenaholmanb: imagine tomcat, but ca 40× more20:53

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!