threepistonsGood afternoon.15:25
threepistonsI am trying to use cloud-init with a Server 22.04 ISO image to build PCs (so bare metal) with PXE.  When I create a user-data file that starts with #cloud-config, I can build successfully.  When I try to use the #include format for my user-data file, I get:15:28
threepistons(timestamp) - main.py[INFO]: contents of 'http://myserver/user-data' did not start with b'#cloud-config'15:29
threepistonsand cloud-init does not attempt to load the URLs listed in the user-data file.15:30
threepistonshttps://github.com/canonical/cloud-init suggested coming here for help.15:31
threepistonsWe are using iPXE.  My menu says:15:39
threepistonskernel http://myserver/pathto/vmlinuz15:40
threepistonsinitrd http://myserver/pathto/initrd15:40
threepistonsimgargs vmlinuz initrd=initrd url=http://myserver/pathto/ubuntu2204.iso url=http://myserver/pathto/user-data autoinstall ds=nocloud-net\;s=http://myserver/pathto ip=dhcp net.ifnames=015:40
threepistonsIs what I'm trying to do even supported using bare metal and nocloud-net?15:41
threepistonsI mean, is #include and a list of URLs supported with nocloud-net?15:43
meenahi threepistons 16:36
meenawhat's " the #include format "? 16:37
threepistonsExample at https://github.com/canonical/cloud-init/blob/main/doc/examples/include.txt16:38
threepistonsDescribed in #4 of https://help.ubuntu.com/community/CloudInit16:39
threepistons"This content is a "include" file. The file contains a list of urls, one per line. Each of the URLs will be read, and their content will be passed through this same set of rules. Ie, the content read from the URL can be gzipped, mime-multi-part, or plain text."16:42
meenaI've never seen those before 16:43
meena (last edited 2021-07-25 16:34:51 by danielbowers)16:44
meenathat's from this century16:44
threepistonsthe cloud-init on the PC that won't build is version 22.1-14-g2e17a0d6-0ubuntu1~22.04.5, it came with the ISO.16:47
minimalthreepistons: nocloud-net loads meta-data, user-data, and vendor-data from the base url17:00
minimalI don't think you can make it load any other urls17:00
threepistonsI'm looking at https://github.com/canonical/cloud-init/blob/2134b0487fa4d453d75ca83bd0d3cf008541ebcb/cloudinit/cmd/main.py#L201  It looks like by design #include will not work.  That's unfortunate because I have many different user-data files and they have a lot of data in common.17:01
threepistons> many different user-data files17:02
threepistonsSATA vs NVMe, LUKS vs plain, GUI vs headless...17:02
threepistonscombinatorial explosion17:02
minimalthreepistons: the function that line is in refers to MAAS DataSource, not NoCloud17:03
minimalalso notice line 222 indicates "url" is deprecated17:05
threepistonsI searched the repo for the error message I got and that was what I found.17:05
threepistonshttps://github.com/canonical/cloud-init/search?q=%22did+not+start+with%22&type=code only one result.17:06
minimalfrom my recollections of trying using NoCloud-Net there are 2 stages, first the URLs (meta-data, user-data, vendor-data) and downloaded, then 2nd stage they are parsed - if they try to include from another url I don't think there's code to then download from more urls (I could be wrong)17:07
minimalactually, looking at your cmdline options I'm confused - you have BOTH url=<where to find user-data> AND s=<base url for cloud-init files>17:08
threepistonsYeah, it seemed to need both to work when I first set up the iPXE menus for 18.04, and "if it ain't broke don't fix it" after that.17:10
minimalI haven't yet tried nocloud-net with PXE17:12
threepistonsNow that I want to refactor my list of staff SSH keys into *one* file instead of 36, maybe time to review that set of arguments.17:12
minimalor dynamically create the user-data contents when it is requested?17:12
threepistonsCGIing the user-data is my plan B.17:15
threepistonsI don't manage the OS of the server that the iPXE service and user-data files sit on, so negotiating the installation of a CGI engine is something I'd rather avoid.17:17
minimalactually you don't even necessarily need dynamic/CGI creation, you could prepare the content in advance (i.e. use jinja or similar templates to create files) whenever "provisioning" is done for a new machine17:18
threepistonsI was thinking of using something like make, never heard of jinja.17:19
minimalits what Python code (including cloud-init) tends to use17:19
minimale.g. cloud-init has template files in /etc/cloud/templates used to create chrony.conf and /etc/hosts17:21
threepistonsRemoving the s= and replacing the url= with cloud-config-url= leaves me with the same "did not start with b'#cloud-config'" message.  But the PC does load the user-data file, so I can at least remove the deprecated arguments from my iPXE menus.17:23
minimalall I know is that nocloud-net docs indicate using "s=" to specify the base url for it to retrieve meta-data, user-data, and vendor-data17:24
threepistonswe don't use meta-data and vendor-data.  I've removed the s= and replacing the url= with cloud-config-url= with a known good user-data file and now have a desktop happily building with a laptop's full disk encryption config.17:32
threepistonsThat helps me if I decide to go down the CGI route because I don't have to try to set parameters twice in the arguments that iPXE sends to the kernel.17:35
threepistonsAnd I don't have to try to set parameters in the path section of a URL.17:35
minimalthreepiston: I mean that whatever url you specify for "s=" is a BASE URL - nocloud appends "/meta-data", "/user-data", and "/vendor-data" to it and tried to fetch all 317:35
threepistonsYeah.  If I was trying to use CGI generated files, I'd have to somehow put the CGI arguments into the base URL because nocloud-net does that appending thing.17:37
minimaldid you read this? https://cloudinit.readthedocs.io/en/latest/reference/datasources/nocloud.html#file-formats17:38
threepistonsI'm building now with nocloud-net and no s=baseurl because I've used cloud-config-url instead and the PC accepted that.17:38
minimal"Bost files must be present for it to be considered a valid seed ISO."17:38
minimalthe use of the word "ISO" is misleading, it should say something like "location"17:39
threepistonsYes, the *empty* meta-data file is there next to the user-data file.  I just didn't put the s=baseurl in.17:40
minimalif you have "ds=nocloud s=" and exists but does not exist then NoCloud won't work17:40
threepistonsmeta-data is there, but I take your point that it needs to stay there.17:41
minimalI'm not sure what happens with an *empty* meta-data file, whether that is valid or not17:41
minimalas usually meta-data provides at least an instance-id17:41
threepistonsIt's worked for me for a timeframe measured in years.17:41
threepistonsthis is nocloud-net.17:42
minimalindeed that same part of the doc says "meta-data is a YAML-formatted file representing what you'd find in the EC2 metadata service" which implies a blank file is not valid17:42
minimalyes there were changes made to git recently to change nocloud and nocloud-net handling so that "ds=nocloud" covers both with the "s=" value distinguishing between them (file: vs http/https)17:43
minimalyou didn't indicate which cloud-init version you're using17:44
threepistons17:47:20 the cloud-init on the PC that won't build is version 22.1-14-g2e17a0d6-0ubuntu1~22.04.5, it came with the ISO.17:44
threepistonsI appreciate that you might have missed that post17:44
minimalthe docs for 22.1 still say the same things about meta-data being YAML formatted file etc17:46
minimalhave you debugging enabled so that /var/log/cloud-init.log gives more info about what is happening?17:46
threepistonsAnd yet I'm looking at a built PC that I've just fed a LUKS passphrase to...17:47
minimalthere may be a difference between what is documented/intended to work and what does work17:48
minimalin general just because something works doesn't mean it is intended to work and therefore is likely to keep on working in future versions ;-)17:49
threepistonsThe undocumented feature...17:49
minimalor unnoticed bug lol17:49
threepistonsI mean, the issue for me is that nocloud doesn't seem to support #include at the start of a user-data file.17:50
minimalprobably because, as I mentioned previously, that by the time it looks at the user-data file it has already fetched any urls it is going to fetch...17:50
threepistonsI know that you said that https://github.com/canonical/cloud-init/blob/2134b0487fa4d453d75ca83bd0d3cf008541ebcb/cloudinit/cmd/main.py#L201 applies to MAAS but there is no other instance of that error message in that repo.17:51
threepistonsWell, not error, but that WARN/INFO text.17:51
minimalalso I don't see "#include" in the 22.1 docs, I think it was only added a couple of releases ago17:52
minimalah, found it, my mistake "include file"17:53
minimalyes I found it, the search didn't pick it up17:54
threepistonsso omitting s= might be an undocumented feature but #include on nocloud is vapourware.17:54
minimalI've never used include files, only cloud-config files17:55
minimalI don't think nocloud-net is used by many people, I've already found some issues with it myself17:58
threepistonsYeah, I think the tool is called "cloud-init" because it's meant for cloud platforms, I think using it for PCs is a bit of an edge case.18:01
minimalwell I do use it myself for physical machines, though in my case I've already prepared the disk image for things like LVM, LUKS, etc, and I'm (currently) using a very small partition for the YAML config files18:02
minimalplus OVH are using it for some of their bare-metal provisioning18:02
threepistonsI'm using to replace preseed.18:03
minimalalso that's what Ubuntu's curtin/subiquity installer based is AFAIK18:04
threepistonsWhen Ubuntu stopped letting me inject preseed into Ubuntu Desktop installations, that was for me a disaster because suddenly this fire-and-forget, can get anyone with the BIOS password to do it task became manual and error-prone and we had laptops going out without full disk encryption, root passwords miskeyed, all sorts of problems.18:06
minimalhaven't used preseed for some time as I'm mainly Alpine these days18:07
threepistonsI'm at a university, we have a research and teaching need for Ubuntu on the desk, but also we have to comply with GDPR.18:07
minimalwell pretty much every org (in UK/EU) "has" to comply with GDPR but many/most doesn't know or don't care :-(18:08
threepistonsInterestingly, if I omit cloud-config-url and just use ds=nocloud\;s=myurl, cloud-init will not even download the user-data file.18:09
minimalif you're in the UK don't worry the ICO is pretty much unlikely to do anything no matter how badly you breach UK GDPR :-(18:09
threepistonsOur chief information security officer takes it quite seriously.18:10
minimal"ds=nocloud\;s=myurl" isn't correct for 22.1, it would be "ds=nocloud-net;s=url" (yeah I noticed issues with ";" on cmdline in the past))18:10
minimalwow, must be one of only a handful of people in UK who care (speaking from experience unfortunately)18:11
minimalon 22.1 "ds=nocloud" is for file-based seeds, "ds=nocloud-net" is for network-based seeds18:12
threepistonshe's fighting an uphill battle with the academics, who really don't care.18:12
minimal<getting off-topic> I spent 2+ years highlighting large scale breach of special category personal data by 2 local government bodies, ICO didn't care, investigated little of the matter and despite even then finding the orgs had breached GDPR and previous law they decided to take no actions :-(18:14
threepistonsThat's universal though, academics dislike anything that interferes with their research.18:14
threepistons<staying off-topic whilst I knock this PC over again> Can data subjects sue?18:16
minimalI'm about to spend a large amount of personal money taking the 2 orgs to court myself18:16
minimalbut think I can only take action about their handling of *my* personal data and not the same (ongoing) mishandling of almost 2 million other peoples' data18:17
threepistonsClass action?18:17
minimalanyway talking about GDPR lack of compliance is depressing :-(18:18
threepistonsProblem is, not enough data subjects care.18:18
minimalthat's exactly the issue - class action would require finding how many other people? part of ICO's failure to act on my complaints was "well no one else has complained about this"18:19
threepistonsIf you have a crowdjustice fundraiser for your case, DM me the link.18:19
meenaLet's instead talk about a willful misinterpretation if GDPR to mean: We can't share any data, ever. (Germany)18:19
minimalthreepistons: DM?18:21
minimalmeena: including "we can't share any data about how we can't share any data"? ;-)18:21
threepistonsThe ${thing} are set at the top of the menu file.  `imgargs ${mykernel} initrd=${myinitrd} url=${isourl} autoinstall ds=nocloud-net\;seedfrom=http://${nfs-host}/preseed/ubuntu/22.04/laptop-sda ip=dhcp net.ifnames=0` does not work.18:22
meenaminimal: i lied. to share data in Germany, all you need is: 📠 18:22
minimalmeena: what's that emoji?18:22
meenaa fax18:23
threepistonspocket calculator?18:23
threepistonsI've not seen one of those in about ten years.18:23
minimalmeena: its been a few years since I left Germany, have they still not phased out faxes? lol18:23
threepistonsThey are so secure, not.  They spew out the pages with no assurance as to who will see them.18:24
meenadespite the fact that some people have caught on to the fact that these days faxes are sent via email, you still can't just use email, but you can still use a fax18:25
minimalhave you checked "dmesg" do see that the cmdline values passed from kernel on to initramfs etc is correct, I found in past the "ds=" value was truncated at semicolon unless I quoted the whole "ds=" section18:25
threepistonstrying quotes now.18:31
threepistonsDM = direct message18:31
minimalyeah, was trying to direct message you18:32
threepistonsI just spotted that.18:33
threepistonsPutting quotes on ds= did not help.18:33
minimaldid "dmesg" show the cmdline looking correct?18:34
threepistonswhen you say check dmesg, you mean "go behind the Help of the locale menu and launch a terminal"?18:34
threepistonsit has the ; there and the full argument.  I replaced s with seedfrom for readabiity, will change that back.18:37
minimalon any machine that has booted if I run "dmesg" then the output includes the likes of: "Command line: BOOT_IMAGE=......."18:38
minimalwhich shows the cmdline the kernel got from the bootloader18:38
minimalwhen using nocloud-net in the past I has the problem that the cmdline passed to kernel (and therefore to initramfs and full OS) was being truncated that the semicolon in the "ds=nocloud-net;s=..." section18:39
threepistonsyes, it's more that I was checking that "from the locale menu in the middle of the by-hand installer wizard that the installer dropped into" was where you meant me to run dmesg from18:39
threepistonsit dropped into the wizard again and says the full ds=nocloud;s=http:... in quotes18:44
threepistonsI put s= and url= for a reason, probably that the one alone was not working.18:45
minimalit's been about 2 months since I last looked at/tried to debug nocloud-net stuff18:46
threepistonsgit log says I had issues and added url= to get around s= not working.18:48
minimalnever tried that, but also not tried PXE either to-date18:49
threepistonsIt wouldn't be the first time that I've had a globally-unique use case.19:08
threepistonsHow about Puppet-managing 312 Raspberry Pis?19:08
minimalI use cloud-init for RPIs initial bootstrapping (with prepared SDcard OS image) and then Ansible for any further config19:09
threepistonsNot so unique after all then.19:10
minimalhaving finished the full arch - which is where cloud-init's phone-home support is used to sign to central provisioning server19:11
minimalthreepistons: BTW <again off-topic> when is a Joint Data Controller not a Joint Data Controller?19:11
threepistonsWhen the JDC has a really good lawyer who can shift the blame onto the other DC?19:12
minimalwhen the alleged Joint Data Controllers (the vast majority of them) have never agreed to or signed the Data Sharing Agreement that defines the Joint DC relationship between the 600+ orgs for the sharing yet they've been sharing personal allegedly in the JDC relationship with those other orgs for 10 years...19:15
threepistonsThe fun part will be updating the OS on them this summer.  The Pis are fixed to the desks and getting them off to replace the SD cards requires dismantling the desks.  Or I can write a new image over the network, but I get one chance to get that right.  These are 3Bs, they don't support PXE without some weirdness.19:15
threepistonsSo how did they acquire the data without someone breaking the law?19:16
minimalyeah, I remember the fun of network booting RPIs (I was part of the original beta trial way back when)19:16
threepistonsGuessing you're also in academia.19:17
threepistonss= doesn't work ,cloud-config-data= does work.  I suspect that the docs are wrong.19:46

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!