| holmanb | minimal: question doesn't apply to alpine :P | 05:34 |
|---|---|---|
| holmanb | minimal: but basically I'm proposing that cloud-init shouldn't set a locale by default | 05:37 |
| holmanb | minimal: here's the bug I filed if you're curious https://github.com/canonical/cloud-init/issues/5649 | 05:38 |
| -ubottu:#cloud-init- Issue 5649 in canonical/cloud-init "[enhancement]: stop setting a default locale" [Open] | 05:38 | |
| holmanb | my claim is that there is not a sufficiently good enough reason for cloud-init to set a default locale, so cloud-init should only set locale in response to user-data received | 05:42 |
| === arif-ali_ is now known as arif-ali | ||
| === SirScott3 is now known as SirScott | ||
| txtsd | How do I prevent cloud-init from injecting DigitalOcean's DNS servers on my VPS? | 08:34 |
| txtsd | systemd-resolved has 1.1.1.1 by default and also has DNSSEC on. DigitalOcean's DNS servers don't support DNSSEC and so DNS resolution is either mad slow or straight-up fails. | 08:35 |
| txtsd | I found `resolver-fix` in `/var/lib/cloud/instances/<snip>/boothooks/` that's doing this. I suppose this file is created during bootup etc, so how do I stop it from running? | 08:38 |
| falcojr | txtsd: I don't know how that particular boothook is provided...whether it is baked into their image vs being provided as vendor data. If it is a part of vendor data, you could pass user data with "vendor_data: {enabled: false}" as part of your cloud-config (see example 3 at | 12:42 |
| falcojr | https://cloudinit.readthedocs.io/en/latest/reference/modules.html#scripts-vendor) | 12:42 |
| falcojr | though I do think that DO provides additional vendor data that you'd be disabling | 12:42 |
| falcojr | if it's baked into the image, there's no good way that I can think of via cloud-init...you'd probably need to create a custom image where you remove that particular script | 12:43 |
| Odd_Bloke | I believe it is provided via vendor-data (though, yeah, there is a fair amount of other vendor data you'd also be disabling). | 13:14 |
| minimal | whichever DataSource is being used by the VM image may also be relevant - the Dataocean DS is deprecated and so I'd assume ConfigDrive is being used instead | 13:17 |
| minimal | s/Dataocean/Digitalocean/ | 13:17 |
| Odd_Bloke | Will /var/lib/cloud/instance have been set correctly by the time boothooks run? I'm looking at https://cloudinit.readthedocs.io/en/latest/explanation/format.html#example-of-once-per-instance-script and wondering if I can use a `/var/lib/cloud/instance/sem` file to simplify the changes. | 13:45 |
| falcojr | yes, the instance link should be correct by then, but I'm not sure I'm following about a sem file | 13:47 |
| Odd_Bloke | If I insert `[ -f /var/lib/cloud/instance/sem/do-dns ] && exit 0; touch /var/lib/cloud/instance/sem/do-dns` at the top of the boothook, I think it should only run the rest of the script once per instance? | 13:49 |
| falcojr | I see what you mean. Yeah, that should work for first boot and then it could be updated manually by others or in a later stage for subsequent boots. I wouldn't recommend vendors writing any data directly into /var/lib/cloud and would consider that unsupported in case we modify anything there, but the concept makes sense | 13:58 |
| falcojr | I realize the example does that...but I should also update the example :D | 13:59 |
| Odd_Bloke | Haha, yeah, I was about to observe the same. I know of `cloud-init-per` to execute a separate script per-instance; is there anything analogous which I could call from within a script? (Which I could use something like `cloud-init-per --check instance do-dns || exit 0` in this context.) | 14:03 |
| Odd_Bloke | Hmm, something like `cloud-init-per instance do-dns false` might be a rough facsimile of that. If it exits 1, then we haven't run before. | 14:11 |
| blackboxsw | funny as well I was just writing up an example for that. But if the do-dns operation is complex you'd have that script writing it's separate script every boot anyway just to ensure the script is there. How about a header command in the script `cloud-init-per instance do-dns exit 1 | 14:11 |
| Odd_Bloke | Hah, great minds etc. | 14:13 |
| blackboxsw | sorry thinko on the exit 1 don't want early exit, but yes some clause that'd low us to introspect "return value" and proceed on first run, exit on subsequent run | 14:14 |
| Odd_Bloke | Huh, `cloud-init-per instance do-dns false` exits 1 the first time and 0 subsequent times; `... exit 1` exits 1 every time and doesn't seem to write the sem file. | 14:16 |
| falcojr | you could technically throw the entire script under `cloud-init-per`. E.g., `cloud-init-per instance do-dns bash -c ' ... ' ` | 14:16 |
| Odd_Bloke | (`cloud-init-per instance do-dns bash -c 'exit 1'` does behave as I would expect.) | 14:17 |
| blackboxsw | Odd_Bloke: your `cloud-init-per instance do-dns /bin/false || exit 0` would do it too I think | 14:18 |
| blackboxsw | sorry typo `cloud-init-per instance do-dns /bin/false && exit 0 | 14:20 |
| blackboxsw | so every run after first boot would exit 0 | 14:21 |
| blackboxsw | until instance-id changes | 14:21 |
| Odd_Bloke | Yeah, I think I'll try that. The way we're generating this makes the `bash -c` approach complicated, else I think I would lean towards that. | 14:28 |
| === kuraudo1 is now known as kuraudo | ||
| txtsd | Y'all I went down the rabbithole to fix it: https://ihavea.quest/posts/2024-09-04-digital-ocean-arch-dns/ | 15:24 |
| minimal | txtsd: the systemd network files are created by cloud-init based on the network-config provided by DigitalOcean to cloud-init | 15:40 |
| minimal | which is either provided by the DigitalOcean metadata server (if cloud-init used the DigitalOcean datasource) or a file in the ISO (if cloud-init used the ConfigDrive datasource) | 15:41 |
| minimal | basically the principle in a cloud environment is that the Cloud Provider specifies network-related settings (which would include DNS) | 15:42 |
| === kuraudo1 is now known as kuraudo | ||
| holmanb | txtsd: glad you figured it out | 18:25 |
| holmanb | This highlights one of the major downsides of vendors providing vendor-data as scripts - it makes it difficult for users to override specific behaviors. | 18:30 |
| holmanb | if vendors used cloud-config instead that would allow users to override vendor-data with more nuance than "turn it off" | 18:35 |
| Odd_Bloke | The problem in cases like this is (IIRC!) that a vendor-data provided cloud-config `bootcmd:` would be overriden by any bootcmd provided by the user, who may not even be aware that a bootcmd is being run until their instance doesn't work as expected. | 18:44 |
| Odd_Bloke | (And if there are multiple bootcmds, they still I think only have the "all or none" option even specifying merging configuration.) | 18:45 |
| holmanb | Odd_Bloke: the problem still applies to shoving arbitrary code into bootcmd, agreed | 19:09 |
| holmanb | Odd_Bloke: but there is a cloud-config module for setting resolvconf | 19:16 |
| Odd_Bloke | I don't believe it's enabled by default, though (as `manage_resolv_conf` defaults to False, I think): so we'd have to opt users into cloud-init-generated resolvconf (and have those values written into a single global file) vs. leaving distro config intact and writing a .d file with just the DNS servers. | 19:26 |
| minimal | holmanb: there is cc_resolv_conf for setting /etc/resolv.conf, which is NOT resolvconf/openresolve | 19:28 |
| holmanb | @Odd_Bloke: It would be trivial to set `manage_resolv_conf: true` as part of the vendor configuration (indeed - that's how a user would override this setting) | 19:29 |
| holmanb | but you're right that overriding the distro config is not ideal | 19:29 |
| minimal | s/openresolve/openresolv/ | 19:29 |
| Odd_Bloke | Right, but then we change the behaviour of _all_ instances just to support the very limited subset of our users who don't want to use our DNS servers (plus there isn't a clue in the .d filename as to where these magic IP addresses have come from). | 19:30 |
| holmanb | but it would probably actually be a better cloud-init default to not override the vendor's `/etc/systemd/resolved.conf` | 19:30 |
| minimal | aren't you getting 2 distinct things mixed up? cc_resolv_conf is only for creating a single /etc/resolv.conf file whereas resolvconf/openresolv is an alternative mechanism where *multiple* config files can be used | 19:31 |
| holmanb | so a `/etc/systemd/resolved.conf.d/cloud-init.conf` would give the breadcrumb to users about where the config came from - and the template could include a note about where it came from | 19:32 |
| holmanb | minimal: cc_resolv_conf actually supports systemd too | 19:32 |
| holmanb | it's just not well documented | 19:32 |
| minimal | hmm, just seeing that too, ok, I'm not familiar with that, only with "classic" /etc/resolv.conf | 19:32 |
| minimal | I've been meaning to add openresolv support for Alpine to cloud-init | 19:33 |
| holmanb | yeah it just uses a different template and file path | 19:33 |
| Odd_Bloke | Oh, I think cc_resolv_conf doesn't even run on Debian/Ubuntu, actually. | 19:34 |
| minimal | Odd_Bloke: yupe, they're not listed in the "distros" | 19:35 |
| holmanb | oh, yeah that's true | 19:35 |
| holmanb | but no reason that it couldn't | 19:35 |
| minimal | because I assume they use resolvconf instead | 19:35 |
| holmanb | as for change in behavior - stable distros patch out breaking changes as needed | 19:35 |
| holmanb | minimal: ubuntu uses systemd-resolved | 19:36 |
| Odd_Bloke | Yeah, resolvconf has been in universe since bionic. | 19:36 |
| minimal | holmanb: so looking at cc_resolv_conf, for systemd OS' it only writes to /etc/systemd/resolved.conf rather than dtopping a file into /etc/systemd/resolved.conf.d/ and so it overrides the default/global config | 19:37 |
| holmanb | minimal: yeah, earlier I mentioned that | 19:38 |
| holmanb | > but it would probably actually be a better cloud-init default to not override the vendor's `/etc/systemd/resolved.conf` | 19:38 |
| minimal | well any file in /etc/systemd/resolved.conf.d/ would override the global file | 19:39 |
| holmanb | I think we're talking about different things - overriding settings vs overriding the file | 19:40 |
| minimal | yeah, I meant override settings ;-) | 19:41 |
| holmanb | we don't want cloud-init to overwrite the file | 19:41 |
| holmanb | based on the upstream default config only alpine and photon use this module | 19:55 |
| minimal | from memory on Alpine if DNS servers are specified in network-config v1/v2 then those end up in the c-i generated eni file but they're not actually used by Alpine | 20:00 |
| minimal | whereas those specified in user_data for "resolv_conf:" end up in /etc/resolv.conf | 20:01 |
| holmanb | interesting | 20:01 |
| minimal | so effectively there are 2 conflicting places for nameservers to be specified to give to cloud-init | 20:01 |
| holmanb | yeah, it's not ideal | 20:02 |
| minimal | I *think* if I get openresolv going on Alpine with cloud-init then there'd be no need to use cc_resolv_conf | 20:03 |
| minimal | do note the docs for cc_resolv_conf say "Many distros have moved away from manually editing resolv.conf so please verify that this is the preferred nameserver management method for your distro before using this module." | 20:04 |
| holmanb | however, network config can't be modified by users at runtime (though maybe it should?) | 20:04 |
| holmanb | yeah I'm pretty sure the git blame points back at me for that sentance | 20:04 |
| minimal | oops, after that it says "Note that using Network configuration is preferred, rather than using this module, when possible." | 20:05 |
| minimal | so that explains why you only found Alpine and Photon having the module in cloud.cfg | 20:05 |
| holmanb | because we were getting questions about how to use it and at the time I wasn't awaare that it supported systemd | 20:06 |
| holmanb | (but also it's still better to steer people away from a module that isn't universally available) | 20:06 |
| holmanb | minimal: maybe adding resolv.conf support to ENI would be easier if that's the default resolver on alpine? | 20:07 |
| minimal | re: network config "though maybe it should?" - I always think (in a cloud) of meta-data and network-config as being controlled by the Provider (it's *their* network after all) though the presented network config can be "influenced" by your actions (i.e. you create a VPC and put a VM in it, AWS then presents network info relating to that VPC) | 20:07 |
| minimal | holmanb: now that 24.3 is out I'm starting to look at my notes for what to work on next. Ideally I'd want to support openresolv as well as /etc/resolv.conf. I'll have to re-familiarise myself with the eni.py code to see what's involved in get either/both resolv.conf and openresolv populated from ENI | 20:11 |
| holmanb | re: can be "influenced" by your actions | 20:12 |
| holmanb | sure, but do vendors always expose the knobs users want? | 20:13 |
| minimal | which goes back to "it is their network after all" ;-) | 20:13 |
| holmanb | but as the user it's _my_ instance | 20:13 |
| minimal | yupe but running on _their_ network | 20:14 |
| holmanb | an when presented with no better alternatives, users may look elsewhere - at competitors, or going on-prem | 20:15 |
| minimal | so actions such as adding a 2nd interface to a VM plumbed into another VPC with DHCP enabled etc then automagically revise the network-config presented by the IMDS | 20:15 |
| minimal | DNS settings are an "outlier", the vast majority of network-related settings in a Cloud infra reflect config defined in the Cloud (VPC, interface, subnets, IPv4/v6 enabled, etc)] | 20:18 |
| holmanb | yeah, that's a nice mental model but in this case the vendor isn't even using network-config to spray their DNS settings | 20:22 |
| holmanb | Odd_Bloke: is there some reason why network-config isn't used for DNS instead of a vendor script? | 20:26 |
| minimal | amd it's not even clear *which~* DS is being used | 20:26 |
| minimal | there's also DHCP option for DNS Nameservers to throw into the mix too.... ;-) | 20:30 |
| minimal | a cloud/hypervisor could use that to specify nameserver(s) to use | 20:31 |
| minimal | holmanb: BTW re: Alpine, the issue wasn't with c-i's eni.py (it writes the nameserver entries fine), it is that Alpine's ifupdown-ng code doesn't look for/use those ENI entries, that's an issue/bug I need to look into | 20:33 |
| minimal | the same applied for the ifupdown package Alpine used to have which was a fork of the Debian code (but the Debian code had a /etc/if-up.d/resolved hook to talk to systemd-resolved) | 20:34 |
| Odd_Bloke | holmanb: I'm unsure: I don't actually own our vendor-data, I just saw the issue in here and picked it up. | 20:44 |
| Odd_Bloke | It does look like the servers are _also_ in network-config. | 20:46 |
| holmanb | huh | 20:54 |
| holmanb | that's odd | 20:54 |
| holmanb | minimal: gotcha | 20:54 |
| holmanb | minimal: were you still looking at lvm for cloud-init / growpart? | 21:38 |
| minimal | holmanb: yes it (as well as LUKS) are on my list - I was working on growpart & LUKS several months ago before I got diverted by the big lock_passwd stuff | 21:49 |
| minimal | I wanted to get the growpart & LUKS sorted first before moving onto LVM | 21:50 |
| holmanb | gotcha | 22:37 |
| minimal | oholmanb: BTW AWS does let you control DHCP options for VPC and so yes a VM could get DNS servers from DHCP as well as from network-config and/or user-data resolv_conf | 22:49 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!