[05:34] <holmanb> minimal: question doesn't apply to alpine :P
[05:37] <holmanb> minimal: but basically I'm proposing that cloud-init shouldn't set a locale by default
[05:38] <holmanb> minimal: here's the bug I filed if you're curious https://github.com/canonical/cloud-init/issues/5649
[05:38] -ubottu:#cloud-init- Issue 5649 in canonical/cloud-init "[enhancement]: stop setting a default locale" [Open]
[05:42] <holmanb> my claim is that there is not a sufficiently good enough reason for cloud-init to set a default locale, so cloud-init should only set locale in response to user-data received
=== arif-ali_ is now known as arif-ali
=== SirScott3 is now known as SirScott
[08:34] <txtsd> How do I prevent cloud-init from injecting DigitalOcean's DNS servers on my VPS?
[08:35] <txtsd> systemd-resolved has 1.1.1.1 by default and also has DNSSEC on. DigitalOcean's DNS servers don't support DNSSEC and so DNS resolution is either mad slow or straight-up fails.
[08:38] <txtsd> I found `resolver-fix` in `/var/lib/cloud/instances/<snip>/boothooks/` that's doing this. I suppose this file is created during bootup etc, so how do I stop it from running?
[12:42] <falcojr> txtsd: I don't know how that particular boothook is provided...whether it is baked into their image vs being provided as vendor data. If it is a part of vendor data, you could pass user data with "vendor_data: {enabled: false}" as part of your cloud-config (see example 3 at
[12:42] <falcojr> https://cloudinit.readthedocs.io/en/latest/reference/modules.html#scripts-vendor)
[12:42] <falcojr> though I do think that DO provides additional vendor data that you'd be disabling
[12:43] <falcojr> if it's baked into the image, there's no good way that I can think of via cloud-init...you'd probably need to create a custom image where you remove that particular script
[13:14] <Odd_Bloke> I believe it is provided via vendor-data (though, yeah, there is a fair amount of other vendor data you'd also be disabling).
[13:17] <minimal> whichever DataSource is being used by the VM image may also be relevant - the Dataocean DS is deprecated and so I'd assume ConfigDrive is being used instead
[13:17] <minimal> s/Dataocean/Digitalocean/
[13:45] <Odd_Bloke> Will /var/lib/cloud/instance have been set correctly by the time boothooks run?  I'm looking at https://cloudinit.readthedocs.io/en/latest/explanation/format.html#example-of-once-per-instance-script and wondering if I can use a `/var/lib/cloud/instance/sem` file to simplify the changes.
[13:47] <falcojr> yes, the instance link should be correct by then, but I'm not sure I'm following about a sem file
[13:49] <Odd_Bloke> If I insert `[ -f /var/lib/cloud/instance/sem/do-dns ] && exit 0; touch /var/lib/cloud/instance/sem/do-dns` at the top of the boothook, I think it should only run the rest of the script once per instance?
[13:58] <falcojr> I see what you mean. Yeah, that should work for first boot and then it could be updated manually by others or in a later stage for subsequent boots. I wouldn't recommend vendors writing any data directly into /var/lib/cloud and would consider that unsupported in case we modify anything there, but the concept makes sense
[13:59] <falcojr> I realize the example does that...but I should also update the example :D
[14:03] <Odd_Bloke> Haha, yeah, I was about to observe the same.  I know of `cloud-init-per` to execute a separate script per-instance; is there anything analogous which I could call from within a script?  (Which I could use something like `cloud-init-per --check instance do-dns || exit 0` in this context.)
[14:11] <Odd_Bloke> Hmm, something like `cloud-init-per instance do-dns false` might be a rough facsimile of that.  If it exits 1, then we haven't run before.
[14:11] <blackboxsw> funny as well I was just writing up an example for that. But if the do-dns operation is complex you'd have that script writing it's separate script every boot anyway just to ensure the script is there.   How about a header command in the script  `cloud-init-per instance do-dns exit 1
[14:13] <Odd_Bloke> Hah, great minds etc.
[14:14] <blackboxsw> sorry thinko on the exit 1 don't want early exit, but yes some clause that'd low us to introspect "return value" and proceed on first run, exit on subsequent run
[14:16] <Odd_Bloke> Huh, `cloud-init-per instance do-dns false` exits 1 the first time and 0 subsequent times; `... exit 1` exits 1 every time and doesn't seem to write the sem file.
[14:16] <falcojr> you could technically throw the entire script under `cloud-init-per`. E.g., `cloud-init-per instance do-dns bash -c ' ... ' `
[14:17] <Odd_Bloke> (`cloud-init-per instance do-dns bash -c 'exit 1'` does behave as I would expect.)
[14:18] <blackboxsw> Odd_Bloke: your `cloud-init-per instance do-dns /bin/false || exit 0` would do it too I think
[14:20] <blackboxsw> sorry typo `cloud-init-per instance do-dns /bin/false && exit 0
[14:21] <blackboxsw> so every run after first boot would exit 0 
[14:21] <blackboxsw> until instance-id changes
[14:28] <Odd_Bloke> Yeah, I think I'll try that.  The way we're generating this makes the `bash -c` approach complicated, else I think I would lean towards that.
=== kuraudo1 is now known as kuraudo
[15:24] <txtsd> Y'all I went down the rabbithole to fix it: https://ihavea.quest/posts/2024-09-04-digital-ocean-arch-dns/
[15:40] <minimal> txtsd: the systemd network files are created by cloud-init based on the network-config provided by DigitalOcean to cloud-init
[15:41] <minimal> which is either provided by the DigitalOcean metadata server (if cloud-init used the DigitalOcean datasource) or a file in the ISO (if cloud-init used the ConfigDrive datasource)
[15:42] <minimal> basically the principle in a cloud environment is that the Cloud Provider specifies network-related settings (which would include DNS)
=== kuraudo1 is now known as kuraudo
[18:25] <holmanb> txtsd: glad you figured it out
[18:30] <holmanb> This highlights one of the major downsides of vendors providing vendor-data as scripts - it makes it difficult for users to override specific behaviors.
[18:35] <holmanb> if vendors used cloud-config instead that would allow users to override vendor-data with more nuance than "turn it off"
[18:44] <Odd_Bloke> The problem in cases like this is (IIRC!) that a vendor-data provided cloud-config `bootcmd:` would be overriden by any bootcmd provided by the user, who may not even be aware that a bootcmd is being run until their instance doesn't work as expected.
[18:45] <Odd_Bloke> (And if there are multiple bootcmds, they still I think only have the "all or none" option even specifying merging configuration.)
[19:09] <holmanb> Odd_Bloke: the problem still applies to shoving arbitrary code into bootcmd, agreed
[19:16] <holmanb> Odd_Bloke: but there is a cloud-config module for setting resolvconf
[19:26] <Odd_Bloke> I don't believe it's enabled by default, though (as `manage_resolv_conf` defaults to False, I think): so we'd have to opt users into cloud-init-generated resolvconf (and have those values written into a single global file) vs. leaving distro config intact and writing a .d file with just the DNS servers.
[19:28] <minimal> holmanb: there is cc_resolv_conf for setting /etc/resolv.conf, which is NOT resolvconf/openresolve
[19:29] <holmanb> @Odd_Bloke:  It would be trivial to set `manage_resolv_conf: true` as part of the vendor configuration (indeed - that's how a user would override this setting)
[19:29] <holmanb> but you're right that overriding the distro config is not ideal
[19:29] <minimal> s/openresolve/openresolv/
[19:30] <Odd_Bloke> Right, but then we change the behaviour of _all_ instances just to support the very limited subset of our users who don't want to use our DNS servers (plus there isn't a clue in the .d filename as to where these magic IP addresses have come from).
[19:30] <holmanb> but it would probably actually be a better cloud-init default to not override the vendor's `/etc/systemd/resolved.conf`
[19:31] <minimal> aren't you getting 2 distinct things mixed up? cc_resolv_conf is only for creating a single /etc/resolv.conf file whereas resolvconf/openresolv is an alternative mechanism where *multiple* config files can be used
[19:32] <holmanb> so a `/etc/systemd/resolved.conf.d/cloud-init.conf` would give the breadcrumb to users about where the config came from - and the template could include a note about where it came from
[19:32] <holmanb> minimal: cc_resolv_conf actually supports systemd too
[19:32] <holmanb> it's just not well documented
[19:32] <minimal> hmm, just seeing that too, ok, I'm not familiar with that, only with "classic" /etc/resolv.conf
[19:33] <minimal> I've been meaning to add openresolv support for Alpine to cloud-init
[19:33] <holmanb> yeah it just uses a different template and file path
[19:34] <Odd_Bloke> Oh, I think cc_resolv_conf doesn't even run on Debian/Ubuntu, actually.
[19:35] <minimal> Odd_Bloke: yupe, they're not listed in the "distros"
[19:35] <holmanb> oh, yeah that's true
[19:35] <holmanb> but no reason that it couldn't
[19:35] <minimal> because I assume they use resolvconf instead
[19:35] <holmanb> as for change in behavior - stable distros patch out breaking changes as needed
[19:36] <holmanb> minimal: ubuntu uses systemd-resolved 
[19:36] <Odd_Bloke> Yeah, resolvconf has been in universe since bionic.
[19:37] <minimal> holmanb: so looking at cc_resolv_conf, for systemd OS' it only writes to /etc/systemd/resolved.conf rather than dtopping a file into /etc/systemd/resolved.conf.d/ and so it overrides the default/global config
[19:38] <holmanb> minimal: yeah, earlier I mentioned that
[19:38] <holmanb> > but it would probably actually be a better cloud-init default to not override the vendor's `/etc/systemd/resolved.conf`
[19:39] <minimal> well any file in /etc/systemd/resolved.conf.d/ would override the global file
[19:40] <holmanb> I think we're talking about different things - overriding settings vs overriding the file
[19:41] <minimal> yeah, I meant override settings ;-)
[19:41] <holmanb> we don't want cloud-init to overwrite the file
[19:55] <holmanb> based on the upstream default config only alpine and photon use this module
[20:00] <minimal> from memory on Alpine if DNS servers are specified in network-config v1/v2 then those end up in the c-i generated eni file but they're not actually used by Alpine
[20:01] <minimal> whereas those specified in user_data for "resolv_conf:" end up in /etc/resolv.conf
[20:01] <holmanb> interesting
[20:01] <minimal> so effectively there are 2 conflicting places for nameservers to be specified to give to cloud-init
[20:02] <holmanb> yeah, it's not ideal
[20:03] <minimal> I *think* if I get openresolv going on Alpine with cloud-init then there'd be no need to use cc_resolv_conf
[20:04] <minimal> do note the docs for cc_resolv_conf say "Many distros have moved away from manually editing resolv.conf so please verify that this is the preferred nameserver management method for your distro before using this module."
[20:04] <holmanb> however, network config can't be modified by users at runtime (though maybe it should?)
[20:04] <holmanb> yeah I'm pretty sure the git blame points back at me for that sentance
[20:05] <minimal> oops, after that it says "Note that using Network configuration is preferred, rather than using this module, when possible."
[20:05] <minimal> so that explains why you only found Alpine and Photon having the module in cloud.cfg
[20:06] <holmanb> because we were getting questions about how to use it and at the time I wasn't awaare that it supported systemd 
[20:06] <holmanb> (but also it's still better to steer people away from a module that isn't universally available)
[20:07] <holmanb> minimal: maybe adding resolv.conf support to ENI would be easier if that's the default resolver on alpine?
[20:07] <minimal> re: network config "though maybe it should?" - I always think (in a cloud) of meta-data and network-config as being controlled by the Provider (it's *their* network after all) though the presented network config can be "influenced" by your actions (i.e. you create a VPC and put a VM in it, AWS then presents network info relating to that VPC)
[20:11] <minimal> holmanb: now that 24.3 is out I'm starting to look at my notes for what to work on next. Ideally I'd want to support openresolv as well as /etc/resolv.conf. I'll have to re-familiarise myself with the eni.py code to see what's involved in get either/both resolv.conf and openresolv populated from ENI
[20:12] <holmanb> re: can be "influenced" by your actions
[20:13] <holmanb> sure, but do vendors always expose the knobs users want?
[20:13] <minimal> which goes back to "it is their network after all" ;-)
[20:13] <holmanb> but as the user it's _my_ instance
[20:14] <minimal> yupe but running on _their_ network
[20:15] <holmanb> an when presented with no better alternatives, users may look elsewhere - at competitors, or going on-prem
[20:15] <minimal> so actions such as adding a 2nd interface to a VM plumbed into another VPC with DHCP enabled etc then automagically revise the network-config presented by the IMDS
[20:18] <minimal> DNS settings are an "outlier", the vast majority of network-related settings in a Cloud infra reflect config defined in the Cloud (VPC, interface, subnets, IPv4/v6 enabled, etc)]
[20:22] <holmanb> yeah, that's a nice mental model but in this case the vendor isn't even using network-config to spray their DNS settings
[20:26] <holmanb> Odd_Bloke: is there some reason why network-config isn't used for DNS instead of a vendor script?
[20:26] <minimal> amd it's not even clear *which~* DS is being used
[20:30] <minimal> there's also DHCP option for DNS Nameservers to throw into the mix too.... ;-)
[20:31] <minimal> a cloud/hypervisor could use that to specify nameserver(s) to use
[20:33] <minimal> holmanb: BTW re: Alpine, the issue wasn't with c-i's eni.py (it writes the nameserver entries fine), it is that Alpine's ifupdown-ng code doesn't look for/use those ENI entries, that's an issue/bug I need to look into
[20:34] <minimal> the same applied for the ifupdown package Alpine used to have which was a fork of the Debian code (but the Debian code had a /etc/if-up.d/resolved hook to talk to systemd-resolved)
[20:44] <Odd_Bloke> holmanb: I'm unsure: I don't actually own our vendor-data, I just saw the issue in here and picked it up.
[20:46] <Odd_Bloke> It does look like the servers are _also_ in network-config.
[20:54] <holmanb> huh
[20:54] <holmanb> that's odd
[20:54] <holmanb> minimal: gotcha
[21:38] <holmanb> minimal: were you still looking at lvm for cloud-init / growpart?
[21:49] <minimal> holmanb: yes it (as well as LUKS) are on my list - I was working on growpart & LUKS several months ago before I got diverted by the big lock_passwd stuff
[21:50] <minimal> I wanted to get the growpart & LUKS sorted first before moving onto LVM
[22:37] <holmanb> gotcha
[22:49] <minimal> oholmanb: BTW AWS does let you control DHCP options for VPC and so yes a VM could get DNS servers from DHCP as well as from network-config and/or user-data resolv_conf