| === hackedhead1 is now known as hackedhead | ||
| lanefu | Okay i've tried to RTFM a few dozen times but I have an order of operations issue with user-data trying to include before networking is up | 15:11 |
|---|---|---|
| lanefu | i've got a iso payload with user-data,meta-data,network-config, (nocloud) userdata strictly includes a remote resource.. btu its failing becuase network isn't setting up first | 15:12 |
| lanefu | configs, notes here: https://armbian.lane-fu.com/linx/m3zebigz.md | 15:12 |
| lanefu | Q: how do i get network-config to execute before #include in users data? | 15:13 |
| blackboxsw | lanefu: not sure if this answers your question. But it looks like the launch is trying to detect nocloud "local" datasource which happens before your network config is "up". Offhand if you are using qemu to launch that ISO I think you'd want to specify `nocloud-net` datasource with smbios directives https://cloudinit.readthedocs.io/en/latest/topics/datasources/nocloud.html?highlight=nocloud-net. That'll keep cloud-init from | 15:29 |
| blackboxsw | trying to run with your user-data in pre-networking timeframe | 15:30 |
| lanefu | blackboxsw: need to avoid doing smbios because eventually will be using some UEFI only images with qemu | 15:32 |
| lanefu | for the record : In my previous config i was doing bios payloasd for the user-data url + dhcp and it was aweosme | 15:33 |
| blackboxsw | lanefu: can you explain for me how you are launching this ISO currently with nocloud metadata? Just so I make sure I'm understanding the problem fully | 15:35 |
| lanefu | eyah. did you happen to take a look at the link I shared? it has all my configs documented | 15:38 |
| lanefu | will give you raw qemu command in a sec | 15:38 |
| lanefu | to establish a baseline real quick: the ISO _is_ working.. the network is getting configured, hostname is getting updated.. it's just the #include remoteuserdata is happening too early | 15:39 |
| lanefu | blackboxsw: okay this will make your eyes bleed a bit, but the raw command and templated https://armbian.lane-fu.com/linx/kjfuylbl.json | 15:40 |
| lanefu | the extra stuff you see in there, which you're probably WTFing about is I have to overload some stuff to make it do what i want. | 15:41 |
| lanefu | the iso generation, and deployment is all done via hashicorp nomda | 15:41 |
| lanefu | nomad | 15:41 |
| lanefu | it's qemu stuff works, but you have to sometimes twist its arm a bit | 15:42 |
| blackboxsw | thanks lanefu, it'll take me a bit to digest this and try a wrap my head around the use-case here as I specifically haven't tried pushing `#includes` into nocloud user-data. If unable to set ds=nocloud-net in smbios directives to the launch. | 15:47 |
| lanefu | I'm the king of unearthing edge-cases that are reasonable, but edge cases nontheless. :P | 15:48 |
| lanefu | thank you for taking a look | 15:48 |
| lanefu | someone suggesed that LEGACY network config might solve the order of operations issue.. but was trying to stick with the more modern option for purity purposes | 15:50 |
| blackboxsw | lanefu: I'm just throwing out an untested idea here. I think in your provided meta-data you may be able to replace `dsmode: local` with `dsmode: net` | 16:01 |
| lanefu | okay let me try that! | 16:02 |
| blackboxsw | that should tell the NoCloud datasource to mandate that it run in init-network timeframe instead of init-local (before networking is setup) | 16:02 |
| lanefu | naturally having to unwind another mess i created first :P | 16:11 |
| lanefu | will report shortly | 16:11 |
| blackboxsw | falcojr: also if we are adding test dependencies we might be able to quickly validate something like `./tools/run-container --unittest rockylinux/8` | 16:11 |
| blackboxsw | lanefu: story of my life | 16:12 |
| falcojr | blackboxsw: I checked and it's in rocky 8 | 16:12 |
| blackboxsw | cool | 16:12 |
| blackboxsw | yeah whatever we do as long as our copr builds don't break we have a prettty good smoke test | 16:12 |
| lanefu | YES! | 16:25 |
| lanefu | blackboxsw: worked! | 16:25 |
| lanefu | thank you! | 16:25 |
| blackboxsw | booom, I'll take that to the bank. Thanks. I might be able to put some docs together on this for future | 16:26 |
| lanefu | nice.. any snippets I can offer? | 16:27 |
| lanefu | i think that pastebin thing i shared pretty much is what i have tho | 16:27 |
| blackboxsw | lanefu: have a github useraccount? | 16:28 |
| blackboxsw | I'll send you a PR review request. I think we want to add some docs to https://cloudinit.readthedocs.io/en/latest/topics/datasources/nocloud.html | 16:28 |
| lanefu | blackboxsw: @lanefu | 16:28 |
| lanefu | sure | 16:28 |
| blackboxsw | GHuser makes complete sense :). I think we also want to provide a dsmode: net/local section at the top and a `::note ` about any remote #include resources require net mode instead of local | 16:29 |
| lanefu | sweet yeah as I move further along on the project i'll try to circle back and see what other lessons i can share | 16:30 |
| lanefu | fun triva fact.. my user-data and meta-data are in a consul KV store and i just have a small nginx write sitting in front of it | 16:30 |
| lanefu | s/write/rewrite | 16:31 |
| blackboxsw | +1 lanefu if you'd like to adapt the current text a bit and provide a PR, that'd be awesome. the content is here https://github.com/canonical/cloud-init/blob/main/doc/rtd/topics/datasources/nocloud.rst | 16:31 |
| blackboxsw | we can walk you through getting into the project and shepherding that PR in | 16:31 |
| blackboxsw | the more the merrier | 16:32 |
| lanefu | tis the way | 16:32 |
| lanefu | okay lemme drop that in ym notes so i dont forget | 16:32 |
| blackboxsw | lanefu: great. to test doc generation it's basically just edit the rst file and `tox -re doc; xdg-open doc/rtd_html/topics/datasources/nocloud.html` | 16:33 |
| lanefu | roger-roger | 16:34 |
| lanefu | thanks so much... uncorked blocker in my long-term science project that i only rarely get to do deep work on.. aka only on holiday lol | 16:35 |
| blackboxsw | nice | 16:38 |
| minimal | lanefu: interesting to see someone using nomad+consul with cloud-init, I'm heading in that direction myself | 16:38 |
| minimal | lanefu: BTW as a side note you're using virtio-blk, I assume that Ubuntu ARM should support virtio-scsi which is recommended and likely more performant | 16:39 |
| lanefu | minimal: let me sanitize a job file and i'll share you some insanity | 16:39 |
| lanefu | minimal: yeah..baby steps.. step one was get it to use virtio... i had to twist the qemu driver's arm with overload tos to od it | 16:39 |
| minimal | lanufu: my end goal is completely throwaway VMs using Alpine Linux run-from-ram combined with cloud-init | 16:40 |
| lanefu | love it.. yeah lots of reallyt cool lean options these days especially with some microvm tech | 16:44 |
| lanefu | which even qemu has a flavor now.. so you dont need firecracker etc | 16:44 |
| lanefu | anyway yeah i'll drop some stuff in a repo shortly | 16:44 |
| minimal | lanefu: I hadn't thought as far as keeping the user-data inside consul (was thinking of some sort of script serving it up) | 16:45 |
| minimal | lanefu: my own script for cloud-init enabled Alpine images: https://github.com/dermotbradley/create-alpine-disk-image | 16:46 |
| minimal | not currently run-from-ram, is using "classic" disk install mode | 16:46 |
| minimal | blackboxsw: re lanefu's problem, is the issue not that networking config hasn't yeat been read from ISO but rather that the interface hasn't yet been brought up before the user-data #include is referenced? | 16:48 |
| blackboxsw | minimal: yes that's ultimately the base problem. cloud-init has already rendered the network-config v1 that the seed ISO presented. But the networking-online.target in systemd hasn't been reached (and setup) yet. | 16:50 |
| blackboxsw | because NoCloud detected during the boot local stage, basically happens before a `pre-networking.target` systemd unit which is before systemd-networkd actually brings up the configured network | 16:51 |
| minimal | blackboxsw: whereas for other DS the ephimeral DHCP would be up to give some network access... | 16:52 |
| blackboxsw | `Before=network-pre.target` rather | 16:52 |
| blackboxsw | minimal: I think yes. The ephemeral DHCP context manager for other datasources wouldn't get affected by this I believe because the cloud/backplane/environment DHCP servers would provide the necessary connectivity to access and pull down the remote #include resources. | 16:56 |
| blackboxsw | I think I need to refresh on context there too, as I though in most cases user-data #includes generally shouldn't be processed until the init-network boot stage https://cloudinit.readthedocs.io/en/latest/topics/boot.html#network | 16:57 |
| blackboxsw | *as I thought* | 16:57 |
| lanefu | minimal: https://github.com/lanefu/lanecloud-snippets | 17:03 |
| minimal | lanefu: rather than using ISO have you though of using seed-urls? | 17:08 |
| lanefu | minimal: yes. :P | 17:08 |
| lanefu | i started from seed urls and came to ISO | 17:09 |
| minimal | lanefu: the only problem I had with seed-urls is that you cannot provide a network-config, its using "fallback" DHCP, and so any things like VLANs etc can't be configured | 17:09 |
| lanefu | yeah | 17:10 |
| lanefu | so my first iteration was great.. seed urls on DHCP nodes and rock n roll | 17:10 |
| lanefu | but now I'm actually doing stuff like static public IPs | 17:10 |
| lanefu | so need static config | 17:10 |
| lanefu | and then with the custom armbian images we're doing... the x86 ones are UEFI only | 17:11 |
| lanefu | so sadly its just cleaner to do the ISO | 17:11 |
| lanefu | and then send remote payload later | 17:11 |
| minimal | lanefu: you might find my script interesting - lightweight OS images :-) | 17:11 |
| minimal | lanefu: have you looked at firecracker support in nomad? It seems promising | 17:14 |
| lanefu | minimal: starred your script looks cool.... I typically distance myself from Alpine strictly due to relgious issues, BUT I RESPECT IT :P | 17:17 |
| blackboxsw | same on those repos. TIL a bit about consul && nomad thanks for the chat here | 17:19 |
| minimal | lanefu: religious issues? you mean OS-specific opinions? (glibc vs musl, systemd vs non-systemd etc) | 17:22 |
| lanefu | yeah musl and just the package manager | 17:22 |
| lanefu | alpine just annoys me cuz i'm in too much of a hurry to think differently | 17:22 |
| minimal | lol | 17:22 |
| lanefu | debian brain is just easier for me code for | 17:23 |
| lanefu | especially if i wanna use ansible etc | 17:23 |
| lanefu | also i'm an Armbian dev | 17:23 |
| lanefu | so just sometimes you gotta stick with your team even if they lose a lot of games | 17:23 |
| lanefu | re firecracker: yeah def been on my radar, just gonna be a while before i get there | 17:25 |
| lanefu | order of operations for project is: gracefully deploy any cloud-init generic images, with appropriate payloads sourced from netbox, consul, vault | 17:27 |
| lanefu | then containers | 17:27 |
| lanefu | then microvims | 17:27 |
| minimal | lanefu: I know the feeling - I've got a long list of stuff like that to try and integrate | 17:27 |
| lanefu | so there's some additional context for all of this of course | 17:28 |
| lanefu | but TLDR; a joke got out of hands after i discovered fiverr and a LOT of random cheap VPS nodes | 17:29 |
| lanefu | and now i'm in the joking, but not joking phase | 17:29 |
| lanefu | https://youtu.be/ptseUMG8VKY | 17:29 |
| lanefu | so really going "make my own cloud provider" but an opinionated modern take as opposed to the mom and pop shops using proxmox and a billing/control panel platform on top of it for provisioning | 17:30 |
| lanefu | and eventually mimic a lot of fly.io | 17:31 |
| lanefu | but running on cheaper infra | 17:31 |
| lanefu | cascading prerequisites lol | 17:31 |
| minimal | lanefu: interesting you mention fly.io - was looking at their docker->firecracker stuff and their use of Nomad recently | 17:31 |
| lanefu | yeah... i was really mad when I found out they existed and already were doing "my idea" | 17:32 |
| minimal | I think though we're getting a bit off-topic for this channel ;-) | 17:32 |
| lanefu | fair | 17:32 |
| lanefu | lol | 17:32 |
| blackboxsw | 'tis friday | 17:36 |
| EvanCarroll | https://devops.stackexchange.com/questions/15678/how-come-my-hostname-isnt-set-to-the-fqdn-with-cloud-init-when-using-prefer-fq | 23:57 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!