MICRObursthow can I specify the system locale in a yaml file? Using "locale: en_DK.UTF-8" does not work.00:48
minimalMICROburst: you mean in user-data? In theory, yes, that should work, I use it myself. However you haven't indicated with cloud-init version you are using, which distro (and version of that), and which cloud-init datasource01:17
MICROburstminimal: I'm using debian bullseye and testing. I did not put it below user-dara since I would like to have it set system-wide. Is it required to list locales-all in the packages section?07:54
yuviHey, can I talk to anyone here, I'm trying to understand some basic things and google isn't helpful10:40
acibayuvi: Hello, I can try to help10:44
smoserananke: if you're still around...13:46
smosera couple of weeks ago i looked at the cloud-initramfs-tools package in ubuntu for a overlayroot issue.13:46
anankeI'm here, thanks for responding! I'm right now not sure where to report this issue. Debian's bug tracker has it, but there seems to be no action13:47
anankeI've been able to work around the missing flock (and other missing tools such as sed/rm/grep), but right now I'm stumped by missing wait-for-root. Debian's packages do not include it in initramfs-tools13:48
smoserif i recall correctly, debian's is very out of date compared to ubuntu.  They grabbed it, made a change or two (again, from memory, i disagreed with), and then really never synced in changes.13:48
anankeI'm baffled by their changes honestly. They took out tools from the hooks file13:50
smoserit doesn't surprise me that growroot would "mostly work" without flock , espeically in the initramfs.13:51
smoserit was racy, and worked for *years* reliably before it started misbehaving (probably some udev or kernel thing changed timing)13:51
anankelack of flock seemed to result in fatal failure, but lack of rm/grep/sed didn't13:51
smoserinitramfs-tools is a mess in that regard. as are all initreamfs creation packages i'm aware of.13:52
smoseryou have a module that says "i need program x". the initramfs script runs, it fails, and then the tool goes on.13:53
smoserthe only way you know about the error is when it fails or if you happened to look at logs.13:53
anankeso I'm wondering what can be done at this point. is wait-for-root available in ubuntu?13:53
anankeand the changes to /usr/share/initramfs-tools/hooks/growroot they did were baffling. their 0.18.debian8 version included more tools (rm, readlink, sed, grep), but 0.18.debian11 removed them13:57
anankeI wish debian had a bit friendlier bug tracker, rather this mailing list approach13:58
smoseri'd be happy to take a patch to "upstreaM" cloud-initramfs-tools that identified that growpart depends on sed,awk, grep14:11
smoserbut even then, its non-trivial because the busybox versions are sufficient, and you don't want to cause a copy of "the real sed" if busybox is present with that.14:12
smoserand ubuntu's busybox-initramfs *does* have those https://paste.opendev.org/show/bXSm70ar2pG8FQgZnnY4/14:13
smoserhm.. reading https://manpages.ubuntu.com/manpages/bionic/en/man8/initramfs-tools.8.html i guess maybe growroot should/could set BUSYBOX=Y ? i dont know.14:17
anankeI'm going to do some testing on vanilla debian 11, using the official AMI, and see if I can replicate the issue there14:17
smoseroh, one question 14:17
smoserwhy do you need it / want it in the initramfs ?14:18
anankesmoser: I personally do not care where it's located. my org is using official Kali images in AWS, which are impacted by this problematic change. Kali seems to be using upstream directly from Debian14:21
smoserand that makes sense...14:21
anankethe problem is simple: you make an ec2 instance with that kali AMI, you apply the updates (including that package), and then try to grow the disk. your instance will fail to boot14:21
smoserif i had a lot of time that i had nothing to do with, then i might try to get cloud-initramfs-tools to be a proper upstream (not a native package on ubuntu) and then get that into debian and have ubuntu just sync from debian.14:22
anankelooking at the debian bug report, this likely affects anybody else using virtualization images14:22
smoserand i think if i worked hard at that, it could get done in 6 montsh of calendar time.14:22
anankeyeah, that doesn't seem like a good ROI :)14:23
anankeif it wasn't for the fact that it used to work just fine, I wouldn't be diving so deep in it. if I don't find a good fix, and ideally get somebody to accept it upstream (Kali, or Debian), then we'll have to keep integrating that fix for the foreseable future. We very much rely on our images being small, and deploying them with larger disk size. rinse & repeat for thousands of users14:25
smoserfor sure, i wouldnt want to do that.14:25
smoserhere is what i'd suggest:14:25
smoser * start working from the ubuntu version https://git.launchpad.net/cloud-initramfs-tools14:28
smoser * add some way to copy grep sed awk as you showed14:28
smoser * suggest debian pull from that. 14:28
smoserif you hyave a relationship with kali, you  might just suggest they remove cloud-initramfs-tools from their packages14:29
anankethank you14:30
anankeso that brings up a good point, which I haven't considered yet: growing the root filesystem doesn't need to be done with initramfs14:32
anankeso one potential way would be to remove cloud-initramfs-growroot package, and redo initramfs? I can test that14:33
anankehah, it worked on debian 11. removed the package, shut it down, increased disk size, started it up. cloud-init was able to grow the filesystem on boot. I personally don't care how early in the boot process that filesystem is grown, so who knows - this may be our best option for right now14:39
anankeI'll do some more testing this afternoon, gotta take kids to the dentist. thanks again!14:41
smoserthe only reason that it was done in the initramfs was because > 10 years ago, there was no option14:42
smoserif your kernel is nerwer than something like 3.18 you can do it while the filesystem is mounted. 14:42
smoser*3.18* seriously.14:43
anankeI vaguely recall doing online resizing back in 2.4 or 2.6 series14:43
anankeeither way, that may be a good stop gap measure for us, so any images we create now will be resizable in the near future, regardless of updates. Part of my issue was timing: I have to roll out few new images this/next week, and any kind of fix would need to survive whatever upstream rolls out14:45
smoserananke: yeah, filesystem resize is older than that. but 'ptupdate' (partition update) was new in 3.8.15:18
smoser3.8 becomes 10 years old next month .15:19
minimalanake: "growing the root filesystem doesn't need to be done with initramfs" - that was the recently I commented "I've never seen growpart run in an initramfs before" yesterday ;-)15:37
minimals/the recently/why/15:37
minimalananke: ^^^15:38
meenasmoser: whew16:34
meenaOpenBSD and NetBSD still use the initramfs hack. online growfs for UFS/FFS2 has been in FreeBSD for… a long time. but different FFS flavours have somewhat diverged in the past 30 years… 16:37
meenaI wanted to… borrow Puppet's facter code for is_virtual, and promptly found a bug17:10
anankesmoser: yeah, re-reading partition table was notoriously problematic with linux. one often had to try different tools to eventually get it to acknowledge changes: blockdev --rereadpt, sfdisk -r, hdparm, partprobe, etc. 18:38
anankefunny enough, just few weeks ago I came across a phantom problem, when building our images would fail because a block device was not found. the culprit? we still don't know why, but for some reason, that particular day executing 'partprobe' on EC2 instances would result in /dev/<insert nvme device> to disappear for less than 1 second. that was enough to cause our process to fail, because right after 18:41
anankepartprobe we were trying to use dd against it18:41
anankenever happened before, we've had that process in place for months, running on various instances. that day it stopped working. we put in gratuitous waits, and re-ordered few things to avoid that issue, and by the time we were done testing the fix, the problem went away18:43
minimalananke: yeah recently I wanted to remove parted package (for partprobe) as a dependancy of the Alpine cloud-init package...then I found Busybox blockdev didn't work reliably for cc_disk_setup lol18:47
smoserananke: fwiw, running 'udevadm settle' usually fixes things.  the problem with it is that most of the time so would 'sleep 1'. so it is very easy to say "oh, the fix is to run udevadm settle" and then you'll actually see it "fixed". but that is very likely no different than adding 'sleep 1'19:02
smosers/very likely/possibly/19:02
* meena wonders if that's how udevadm is implemented19:04
minimalI run without udev (and therefore without "udevadm settle") on some systems ;-)19:19
meenaalias "udevadm settle" "sleep 1"19:21
falcojrI'm assuming it was a joke...otherwise I'm very worried XD19:23
minimaloops, meena: no need, cc_disk_setup only runs udevadm if it is present19:23
minimalfalcojr: are you referring to me?19:24
falcojrno, the alias19:24
meenafalcojr: mostly a joke, yes. mostly ;)19:27
meenafive minutes in a ruby pry session, and i have forgotten half of python again19:30
minimalMICROburst: your response re your locale problem wasn't clear. Where exactly are you specifying the "locale:"?19:34
xtalkmeGood evening all, I am trying to debug the cloud-init module cc_puppet, because of the following error.20:28
xtalkmecloudinit.url_helper.UrlError: HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Max retries exceeded with url: /puppetlabs/install-puppet/main/install.sh (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)')))20:28
xtalkmeA local wget works fine, certficate is valid enabled debugging on logger_cloud-init but no luck.20:28
xtalkmeAny tips where to look ?20:28
MICROburstminimal: below "timezone:" (not indented, if you meant that)20:32
minimalMICROburst: I meant where as in if it is not in user-data (you said) then where exactly have you specified it?20:39
minimalin /etc/cloud/cloud.cfg ? somewhere else?20:39
MICROburstminimal: virt-install ...     --cloud-init user-data=../etc/${NAME}.yaml21:16
MICROburstthought you were talking about the users section in the file...21:17
minimalMICROburst: so you *are* providing user-data, your earlier response said "I did not put it below user-data"21:17
minimalso did you check if the /etc/default/locale file was updated by cloud-init?  That's the file cc_locale changes for Debian21:19
minimaldo you also have the Debian locales package installed?21:20
smoserxtalkme: 2 options21:31
smoser1. cloud-init/python is not looking for certs the same place as wget is21:31
smoser2. when cloud-init tried the clock was not right21:32
smoserto do the same request with cloud-init, something like:21:33
smoserif that works at the same time wget works, then i think its clock related or (just thought of this), you're behind some sort of closed portal21:37
smoseryour https request to github is going to some local page and https is crying about that.21:38
minimalhttps://raw.githubusercontent.com/puppetlabs/install-puppet/main/install.sh is defined in cc_puppet.py so that's where the url portions referenced above come from21:44
minimalthe "local page" is actually the non-scheme, non-host part of that url21:45
xtalkmesmoser Thanks did not think about time as a issue. Will have a look at that. And with closed portal i guess you mean something like a wifi portal ?21:55
xtalkmesmoser when i import the urlhelper, i get the same error, time seems correct.22:02
xtalkmeminimal not sure what you mean with the "Local page"?22:19
xtalkmesmoser certifi.where() also seems to return the expected location for its certificates: '/etc/ssl/certs/ca-certificates.crt'22:19
xtalkmer = requests.get("https://raw.githubusercontent.com/puppetlabs/install-puppet/main/install.sh", verify=False) works so i guess something certificates...22:28
xtalkmeThe issue is on 22.04 but not on 20.04, both using 22.4.2 but different python versions22:40
MICROburstminimal: the locales-all package was not added to the list. I'll test this...22:51
-ubottu:#cloud-init- Pull 1957 in canonical/cloud-init "add function is_virtual to distro/FreeBSD" [Open]22:58
minimalxtalkme: I was referring to smoser's reference to "some local page"23:01
=== esv_ is now known as esv
MICROburstminimal: No success: /etc/default/locale -> LANG=C.UTF-8 and should be LANG=en_DK.UTF-8 -- log file claims "invalid locale", however "localectl set-locale LANG=en_DK.UTF-8" works23:23
minimalMICROburst: have you turned on debugging for cloud-init and then looked in /var/log/cloud-init.log to see what it shows for cc_locale?23:27
minimalactually, hang on, is there such as thing as en_DK? Danish English?23:28
minimaloh there is, first I've ever seen it mentioned, wow23:29
minimalso you have the "locales" Debian package installed?23:30
meenaminimal: best locale in the world.23:30
minimalprobably ;-)23:30
meenaminimal: gives you english language stuff with a sensible time and date format23:30
minimalmeena: sensible time/date format? you mean like en_GB and en_IE?23:31
meenaminimal: i don't know what those look like, but en_US is incomprehensible.23:34
minimalMICROburst: cc_locale on Debian doesn't run "localctl", it runs "update-locale" or "locale-gen"23:34
meenadate on my system with LC_TIME=en_DK.utf-8 gives: 2023-01-12T23:34:27 GMT23:34
minimalmeena: well you're in Ireland now, you should be using en_IE ;-)23:35
meenaminimal: true23:35
minimalit's like en_GB but with Euros not Pounds lol23:35
minimalMICROburst: both those commands come from the "locales" package which is why I asked you earlier if you had it installed23:36
MICROburstminimal: locales-all IS installed23:37
minimalMICROburst: for the 2nd time, I didn't ask you about locales-all, I asked about locales23:37
minimalthey are 2 different packages23:37
MICROburstminimal: package 'locale' is installed too.23:38
minimalso perhaps test manually if the "update-locale" and "locale-gen" commands work as that's what cloud-init is using23:39
minimalmeena: you're using those settings? ;-)23:40
meenaminimal: i have no idea. I assume the system knows what it's doing. I set this via the GUI.23:41
meenathis makes me wonder if there's ga_IE?23:42
minimalmeena: so your part of Ireland is using Danish Krone for currency? lol23:42
meenadamn, forgot about that bit23:43
minimalyupe, there is ga_IE indeed23:43
MICROburstminimal: "update-locale" and "locale-gen" do not have an impact on /etc/default/locale23:43
minimalthough you're not in a Gaeltacht are you?23:44
meenaminimal: very close to one23:44
minimalMICROburtst: I'm looking at the cloud-init distro/debian.py functions for locale handling and they do not use localctl (apart from keymap), they use either update-locale or locale-gen23:44
meenaseveral people in my partner's family speak Irish, and my daughter is learning it from them, too, i hope. She's certainly learning more Irish than Bosnian.23:45
minimaland if yuo have debugging enabled in cloud-init you could then see the exactly commands run by cloud-init when trying to set the locale for you23:45
minimalmeena: did 1 year of Irish decades ago and forgotten most of it23:46
MICROburstminimal: I'm using 'localectl ..." on shell to check. The .yaml file contains "locale: en_DK.UTF-8" -- So what do I need to change?23:47
minimalMICROburst: so you're using a command to test things that the cloud-init code does not use23:47
meenaminimal: https://glitch.social/@meena/10844420801064405523:47
minimalMICROburst: look at cloudinit/distros/debian.py for the definition of LOCALE_CONF_FN and also at the get_locale and apply_locale functions in there23:49
MICROburstminimal: I can use for testing whatever I want - I has to work, since the system is installed. cloud-init has to set up things according to the .yaml file(s). I looked in /etc/default/locale as you recommended earlier - still "LANG=C.UTF-8"23:51
minimalMICROburst: "I can use for testing whatever I want" - as cloud-init does not use the localectl command then whether it works or not is irrelevant as cloud-init does not use it23:52
minimalor more specifically, cloud-init on Debian does not use it23:52
MICROburstminimal: So why is /etc/default/locale unchanged? - What do I need to put in user-data ... yaml?23:53
minimalI assume that file is unchanged because of the "invalid locale" you mentioned earlier23:54
minimalbut I'm guess as you haven't shown any output from cloud-init.log when debugging is enabled23:54
minimalas the current contents of the /etc/default/locale file don't match the specified locale in your user-data then cloud-init should be running "locale-gen en_DK.UTF-8" and I assume that is what is failing23:58
minimalbut I'm guessing whereas seeing the relevant excerpt from cloud-init.log would give specific information23:58

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!