| MICROburst | how can I specify the system locale in a yaml file? Using "locale: en_DK.UTF-8" does not work. | 00:48 |
|---|---|---|
| minimal | MICROburst: you mean in user-data? In theory, yes, that should work, I use it myself. However you haven't indicated with cloud-init version you are using, which distro (and version of that), and which cloud-init datasource | 01:17 |
| MICROburst | minimal: I'm using debian bullseye and testing. I did not put it below user-dara since I would like to have it set system-wide. Is it required to list locales-all in the packages section? | 07:54 |
| yuvi | Hey, can I talk to anyone here, I'm trying to understand some basic things and google isn't helpful | 10:40 |
| aciba | yuvi: Hello, I can try to help | 10:44 |
| smoser | ananke: if you're still around... | 13:46 |
| smoser | a couple of weeks ago i looked at the cloud-initramfs-tools package in ubuntu for a overlayroot issue. | 13:46 |
| ananke | I'm here, thanks for responding! I'm right now not sure where to report this issue. Debian's bug tracker has it, but there seems to be no action | 13:47 |
| ananke | I've been able to work around the missing flock (and other missing tools such as sed/rm/grep), but right now I'm stumped by missing wait-for-root. Debian's packages do not include it in initramfs-tools | 13:48 |
| smoser | if i recall correctly, debian's is very out of date compared to ubuntu. They grabbed it, made a change or two (again, from memory, i disagreed with), and then really never synced in changes. | 13:48 |
| smoser | https://paste.opendev.org/show/bYpMFNvCIqqz0Bm3YDIX/ | 13:50 |
| ananke | I'm baffled by their changes honestly. They took out tools from the hooks file | 13:50 |
| smoser | it doesn't surprise me that growroot would "mostly work" without flock , espeically in the initramfs. | 13:51 |
| smoser | it was racy, and worked for *years* reliably before it started misbehaving (probably some udev or kernel thing changed timing) | 13:51 |
| ananke | lack of flock seemed to result in fatal failure, but lack of rm/grep/sed didn't | 13:51 |
| smoser | initramfs-tools is a mess in that regard. as are all initreamfs creation packages i'm aware of. | 13:52 |
| smoser | you have a module that says "i need program x". the initramfs script runs, it fails, and then the tool goes on. | 13:53 |
| smoser | the only way you know about the error is when it fails or if you happened to look at logs. | 13:53 |
| ananke | so I'm wondering what can be done at this point. is wait-for-root available in ubuntu? | 13:53 |
| ananke | and the changes to /usr/share/initramfs-tools/hooks/growroot they did were baffling. their 0.18.debian8 version included more tools (rm, readlink, sed, grep), but 0.18.debian11 removed them | 13:57 |
| ananke | I wish debian had a bit friendlier bug tracker, rather this mailing list approach | 13:58 |
| smoser | i'd be happy to take a patch to "upstreaM" cloud-initramfs-tools that identified that growpart depends on sed,awk, grep | 14:11 |
| smoser | but even then, its non-trivial because the busybox versions are sufficient, and you don't want to cause a copy of "the real sed" if busybox is present with that. | 14:12 |
| smoser | and ubuntu's busybox-initramfs *does* have those https://paste.opendev.org/show/bXSm70ar2pG8FQgZnnY4/ | 14:13 |
| smoser | hm.. reading https://manpages.ubuntu.com/manpages/bionic/en/man8/initramfs-tools.8.html i guess maybe growroot should/could set BUSYBOX=Y ? i dont know. | 14:17 |
| ananke | I'm going to do some testing on vanilla debian 11, using the official AMI, and see if I can replicate the issue there | 14:17 |
| smoser | oh, one question | 14:17 |
| smoser | why do you need it / want it in the initramfs ? | 14:18 |
| ananke | smoser: I personally do not care where it's located. my org is using official Kali images in AWS, which are impacted by this problematic change. Kali seems to be using upstream directly from Debian | 14:21 |
| smoser | and that makes sense... | 14:21 |
| ananke | the problem is simple: you make an ec2 instance with that kali AMI, you apply the updates (including that package), and then try to grow the disk. your instance will fail to boot | 14:21 |
| smoser | if i had a lot of time that i had nothing to do with, then i might try to get cloud-initramfs-tools to be a proper upstream (not a native package on ubuntu) and then get that into debian and have ubuntu just sync from debian. | 14:22 |
| ananke | looking at the debian bug report, this likely affects anybody else using virtualization images | 14:22 |
| smoser | and i think if i worked hard at that, it could get done in 6 montsh of calendar time. | 14:22 |
| ananke | yeah, that doesn't seem like a good ROI :) | 14:23 |
| ananke | if it wasn't for the fact that it used to work just fine, I wouldn't be diving so deep in it. if I don't find a good fix, and ideally get somebody to accept it upstream (Kali, or Debian), then we'll have to keep integrating that fix for the foreseable future. We very much rely on our images being small, and deploying them with larger disk size. rinse & repeat for thousands of users | 14:25 |
| smoser | for sure, i wouldnt want to do that. | 14:25 |
| smoser | here is what i'd suggest: | 14:25 |
| smoser | * start working from the ubuntu version https://git.launchpad.net/cloud-initramfs-tools | 14:28 |
| smoser | * add some way to copy grep sed awk as you showed | 14:28 |
| smoser | * suggest debian pull from that. | 14:28 |
| smoser | if you hyave a relationship with kali, you might just suggest they remove cloud-initramfs-tools from their packages | 14:29 |
| ananke | thank you | 14:30 |
| ananke | so that brings up a good point, which I haven't considered yet: growing the root filesystem doesn't need to be done with initramfs | 14:32 |
| ananke | so one potential way would be to remove cloud-initramfs-growroot package, and redo initramfs? I can test that | 14:33 |
| ananke | hah, it worked on debian 11. removed the package, shut it down, increased disk size, started it up. cloud-init was able to grow the filesystem on boot. I personally don't care how early in the boot process that filesystem is grown, so who knows - this may be our best option for right now | 14:39 |
| ananke | I'll do some more testing this afternoon, gotta take kids to the dentist. thanks again! | 14:41 |
| smoser | the only reason that it was done in the initramfs was because > 10 years ago, there was no option | 14:42 |
| smoser | if your kernel is nerwer than something like 3.18 you can do it while the filesystem is mounted. | 14:42 |
| smoser | *3.18* seriously. | 14:43 |
| ananke | I vaguely recall doing online resizing back in 2.4 or 2.6 series | 14:43 |
| ananke | either way, that may be a good stop gap measure for us, so any images we create now will be resizable in the near future, regardless of updates. Part of my issue was timing: I have to roll out few new images this/next week, and any kind of fix would need to survive whatever upstream rolls out | 14:45 |
| smoser | ananke: yeah, filesystem resize is older than that. but 'ptupdate' (partition update) was new in 3.8. | 15:18 |
| smoser | 3.8 becomes 10 years old next month . | 15:19 |
| minimal | anake: "growing the root filesystem doesn't need to be done with initramfs" - that was the recently I commented "I've never seen growpart run in an initramfs before" yesterday ;-) | 15:37 |
| minimal | s/the recently/why/ | 15:37 |
| minimal | ananke: ^^^ | 15:38 |
| smoser | kids | 16:10 |
| meena | smoser: whew | 16:34 |
| meena | OpenBSD and NetBSD still use the initramfs hack. online growfs for UFS/FFS2 has been in FreeBSD for… a long time. but different FFS flavours have somewhat diverged in the past 30 years… | 16:37 |
| meena | I wanted to… borrow Puppet's facter code for is_virtual, and promptly found a bug | 17:10 |
| ananke | smoser: yeah, re-reading partition table was notoriously problematic with linux. one often had to try different tools to eventually get it to acknowledge changes: blockdev --rereadpt, sfdisk -r, hdparm, partprobe, etc. | 18:38 |
| ananke | funny enough, just few weeks ago I came across a phantom problem, when building our images would fail because a block device was not found. the culprit? we still don't know why, but for some reason, that particular day executing 'partprobe' on EC2 instances would result in /dev/<insert nvme device> to disappear for less than 1 second. that was enough to cause our process to fail, because right after | 18:41 |
| ananke | partprobe we were trying to use dd against it | 18:41 |
| ananke | never happened before, we've had that process in place for months, running on various instances. that day it stopped working. we put in gratuitous waits, and re-ordered few things to avoid that issue, and by the time we were done testing the fix, the problem went away | 18:43 |
| minimal | ananke: yeah recently I wanted to remove parted package (for partprobe) as a dependancy of the Alpine cloud-init package...then I found Busybox blockdev didn't work reliably for cc_disk_setup lol | 18:47 |
| smoser | ananke: fwiw, running 'udevadm settle' usually fixes things. the problem with it is that most of the time so would 'sleep 1'. so it is very easy to say "oh, the fix is to run udevadm settle" and then you'll actually see it "fixed". but that is very likely no different than adding 'sleep 1' | 19:02 |
| smoser | s/very likely/possibly/ | 19:02 |
| * meena wonders if that's how udevadm is implemented | 19:04 | |
| minimal | I run without udev (and therefore without "udevadm settle") on some systems ;-) | 19:19 |
| meena | alias "udevadm settle" "sleep 1" | 19:21 |
| minimal | meena? | 19:23 |
| falcojr | I'm assuming it was a joke...otherwise I'm very worried XD | 19:23 |
| minimal | oops, meena: no need, cc_disk_setup only runs udevadm if it is present | 19:23 |
| minimal | falcojr: are you referring to me? | 19:24 |
| falcojr | no, the alias | 19:24 |
| meena | falcojr: mostly a joke, yes. mostly ;) | 19:27 |
| meena | five minutes in a ruby pry session, and i have forgotten half of python again | 19:30 |
| minimal | MICROburst: your response re your locale problem wasn't clear. Where exactly are you specifying the "locale:"? | 19:34 |
| xtalkme | Good evening all, I am trying to debug the cloud-init module cc_puppet, because of the following error. | 20:28 |
| xtalkme | cloudinit.url_helper.UrlError: HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Max retries exceeded with url: /puppetlabs/install-puppet/main/install.sh (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)'))) | 20:28 |
| xtalkme | A local wget works fine, certficate is valid enabled debugging on logger_cloud-init but no luck. | 20:28 |
| xtalkme | Any tips where to look ? | 20:28 |
| MICROburst | minimal: below "timezone:" (not indented, if you meant that) | 20:32 |
| minimal | MICROburst: I meant where as in if it is not in user-data (you said) then where exactly have you specified it? | 20:39 |
| minimal | in /etc/cloud/cloud.cfg ? somewhere else? | 20:39 |
| MICROburst | minimal: virt-install ... --cloud-init user-data=../etc/${NAME}.yaml | 21:16 |
| MICROburst | thought you were talking about the users section in the file... | 21:17 |
| minimal | MICROburst: so you *are* providing user-data, your earlier response said "I did not put it below user-data" | 21:17 |
| minimal | so did you check if the /etc/default/locale file was updated by cloud-init? That's the file cc_locale changes for Debian | 21:19 |
| minimal | do you also have the Debian locales package installed? | 21:20 |
| smoser | xtalkme: 2 options | 21:31 |
| smoser | 1. cloud-init/python is not looking for certs the same place as wget is | 21:31 |
| smoser | 2. when cloud-init tried the clock was not right | 21:32 |
| smoser | to do the same request with cloud-init, something like: | 21:33 |
| smoser | https://paste.openstack.org/show/bvAqYfyonGtxNQFytU3e/ | 21:36 |
| smoser | if that works at the same time wget works, then i think its clock related or (just thought of this), you're behind some sort of closed portal | 21:37 |
| smoser | your https request to github is going to some local page and https is crying about that. | 21:38 |
| minimal | https://raw.githubusercontent.com/puppetlabs/install-puppet/main/install.sh is defined in cc_puppet.py so that's where the url portions referenced above come from | 21:44 |
| minimal | the "local page" is actually the non-scheme, non-host part of that url | 21:45 |
| xtalkme | smoser Thanks did not think about time as a issue. Will have a look at that. And with closed portal i guess you mean something like a wifi portal ? | 21:55 |
| xtalkme | smoser when i import the urlhelper, i get the same error, time seems correct. | 22:02 |
| xtalkme | minimal not sure what you mean with the "Local page"? | 22:19 |
| xtalkme | smoser certifi.where() also seems to return the expected location for its certificates: '/etc/ssl/certs/ca-certificates.crt' | 22:19 |
| xtalkme | r = requests.get("https://raw.githubusercontent.com/puppetlabs/install-puppet/main/install.sh", verify=False) works so i guess something certificates... | 22:28 |
| xtalkme | The issue is on 22.04 but not on 20.04, both using 22.4.2 but different python versions | 22:40 |
| MICROburst | minimal: the locales-all package was not added to the list. I'll test this... | 22:51 |
| meena | https://github.com/canonical/cloud-init/pull/1957 | 22:58 |
| -ubottu:#cloud-init- Pull 1957 in canonical/cloud-init "add function is_virtual to distro/FreeBSD" [Open] | 22:58 | |
| minimal | xtalkme: I was referring to smoser's reference to "some local page" | 23:01 |
| === esv_ is now known as esv | ||
| MICROburst | minimal: No success: /etc/default/locale -> LANG=C.UTF-8 and should be LANG=en_DK.UTF-8 -- log file claims "invalid locale", however "localectl set-locale LANG=en_DK.UTF-8" works | 23:23 |
| minimal | MICROburst: have you turned on debugging for cloud-init and then looked in /var/log/cloud-init.log to see what it shows for cc_locale? | 23:27 |
| minimal | actually, hang on, is there such as thing as en_DK? Danish English? | 23:28 |
| minimal | oh there is, first I've ever seen it mentioned, wow | 23:29 |
| minimal | so you have the "locales" Debian package installed? | 23:30 |
| meena | minimal: best locale in the world. | 23:30 |
| minimal | probably ;-) | 23:30 |
| meena | minimal: gives you english language stuff with a sensible time and date format | 23:30 |
| minimal | meena: sensible time/date format? you mean like en_GB and en_IE? | 23:31 |
| meena | minimal: i don't know what those look like, but en_US is incomprehensible. | 23:34 |
| minimal | MICROburst: cc_locale on Debian doesn't run "localctl", it runs "update-locale" or "locale-gen" | 23:34 |
| meena | date on my system with LC_TIME=en_DK.utf-8 gives: 2023-01-12T23:34:27 GMT | 23:34 |
| minimal | meena: well you're in Ireland now, you should be using en_IE ;-) | 23:35 |
| meena | minimal: true | 23:35 |
| minimal | it's like en_GB but with Euros not Pounds lol | 23:35 |
| meena | https://gist.github.com/79424923d80e70213ceef8084b61222f | 23:36 |
| minimal | MICROburst: both those commands come from the "locales" package which is why I asked you earlier if you had it installed | 23:36 |
| MICROburst | minimal: locales-all IS installed | 23:37 |
| minimal | MICROburst: for the 2nd time, I didn't ask you about locales-all, I asked about locales | 23:37 |
| minimal | they are 2 different packages | 23:37 |
| MICROburst | minimal: package 'locale' is installed too. | 23:38 |
| minimal | so perhaps test manually if the "update-locale" and "locale-gen" commands work as that's what cloud-init is using | 23:39 |
| minimal | meena: you're using those settings? ;-) | 23:40 |
| meena | minimal: i have no idea. I assume the system knows what it's doing. I set this via the GUI. | 23:41 |
| meena | this makes me wonder if there's ga_IE? | 23:42 |
| minimal | meena: so your part of Ireland is using Danish Krone for currency? lol | 23:42 |
| meena | damn, forgot about that bit | 23:43 |
| minimal | yupe, there is ga_IE indeed | 23:43 |
| MICROburst | minimal: "update-locale" and "locale-gen" do not have an impact on /etc/default/locale | 23:43 |
| minimal | though you're not in a Gaeltacht are you? | 23:44 |
| meena | minimal: very close to one | 23:44 |
| minimal | MICROburtst: I'm looking at the cloud-init distro/debian.py functions for locale handling and they do not use localctl (apart from keymap), they use either update-locale or locale-gen | 23:44 |
| meena | several people in my partner's family speak Irish, and my daughter is learning it from them, too, i hope. She's certainly learning more Irish than Bosnian. | 23:45 |
| minimal | and if yuo have debugging enabled in cloud-init you could then see the exactly commands run by cloud-init when trying to set the locale for you | 23:45 |
| minimal | meena: did 1 year of Irish decades ago and forgotten most of it | 23:46 |
| MICROburst | minimal: I'm using 'localectl ..." on shell to check. The .yaml file contains "locale: en_DK.UTF-8" -- So what do I need to change? | 23:47 |
| minimal | MICROburst: so you're using a command to test things that the cloud-init code does not use | 23:47 |
| meena | minimal: https://glitch.social/@meena/108444208010644055 | 23:47 |
| minimal | MICROburst: look at cloudinit/distros/debian.py for the definition of LOCALE_CONF_FN and also at the get_locale and apply_locale functions in there | 23:49 |
| MICROburst | minimal: I can use for testing whatever I want - I has to work, since the system is installed. cloud-init has to set up things according to the .yaml file(s). I looked in /etc/default/locale as you recommended earlier - still "LANG=C.UTF-8" | 23:51 |
| minimal | MICROburst: "I can use for testing whatever I want" - as cloud-init does not use the localectl command then whether it works or not is irrelevant as cloud-init does not use it | 23:52 |
| minimal | or more specifically, cloud-init on Debian does not use it | 23:52 |
| MICROburst | minimal: So why is /etc/default/locale unchanged? - What do I need to put in user-data ... yaml? | 23:53 |
| minimal | I assume that file is unchanged because of the "invalid locale" you mentioned earlier | 23:54 |
| minimal | but I'm guess as you haven't shown any output from cloud-init.log when debugging is enabled | 23:54 |
| minimal | as the current contents of the /etc/default/locale file don't match the specified locale in your user-data then cloud-init should be running "locale-gen en_DK.UTF-8" and I assume that is what is failing | 23:58 |
| minimal | but I'm guessing whereas seeing the relevant excerpt from cloud-init.log would give specific information | 23:58 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!