[12:21] Hello, first time playing around with cloud init. Things are going great in Ubuntu, but having some problems configuring networking in Debian. I'm using a nocloud database with a v2 network configuration file to set a static ip address. The user-data and meta-data is applied successfully, but after the machine comes up, it's falling back to DHCP. [12:21] Upon further inspection, this is because /etc/network/interfaces does a source-directory for both /etc/network/interfaces.d/ (which has the correctly configured file), and /run/network/interfaces.d/ which has a file that conflicts and falls back to DHCP. Is there a way to turn off the latter? [12:25] s/database/datastore [12:36] hmm, there's also something weird going on similar to https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=867921 [12:36] Debian bug 867921 in openstack-debian-images "cloud-init: Network configuration doesn't work" [Normal,Fixed] [12:38] So to recap, when using nocloud with a v2 network config that sets a static ip address, when the Debian host comes up, a few things need to happen in order for the machine to get networking: 1) `rm /run/network/interfaces.d/50-cloud-init.cfg` (which includes the DHCP setting), and 2) `mv /etc/network/interfaces.d/50-cloud-init.cfg [12:38] /etc/network/interfaces.d/50-cloud-init`, once those have been completed, a `systemctl restart networking` can be run and everything works fine. [12:39] This is cloud-init 18.3 [13:46] bswinnerton: have you tried a more recent version to see if this is fixed? [13:47] I haven't, I'm using what's built into the debian cloud images [13:56] In what scenarios would cloud-init write to `/run/network/interfaces.d/`? Is that the fallback described in https://cloudinit.readthedocs.io/en/latest/topics/network-config.html#fallback-network-configuration? [13:57] I'm trying to remember if we have deb packages pre-built anyhere [13:58] my brain isn't working very well today, so you might be quicker looking yourself [13:59] https://packages.debian.org/buster/cloud-init [13:59] They're quite old, which is why the debian images include them, I assume [14:00] no backports? [14:00] ah yes: https://packages.debian.org/source/buster-backports/cloud-init [14:00] there's https://launchpad.net/ubuntu/+source/cloud-init but i don't know how well those will work on Debian [14:03] bswinnerton: so let's try at version, and if that still fails, we'll file a bug report [14:03] and by we, i mean, can you please [14:04] s/at /that / [14:05] Sounds good, installing 20.2 now [14:07] Is there an effective way to test this with cloud-init on the guest without rebuilding the image? [14:07] Or does the image need to be rebuilt? [14:07] install, run cloud-init clean --logs --reboot [14:08] Neat, wasn't aware of `clean`. Doing that now [14:08] that will regenerate everything on the server, including ssh server keys [14:10] Sigh, that fixed it [14:11] Okay, off to pester the Debian folks. Thanks meena [14:11] 😽 [14:12] if i really were useful, I'd help you pinpoint where that was fixed, exactly [14:12] but screw that, let them upgrade to the latest version :P [14:13] A quick clarifying question, even after bumping to 20.2 I notice a very long delay when trying to bring up the interface, it appears that it's trying to get a DHCP lease with "Started ifup for ens3". It then eventually times out and continues along, and sets the static IP. Any idea how I could tell it not to try and DHCP? Is that the difference [14:13] between a dsstore of local vs net? [14:13] I have a feeling it was this: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=867921 [14:13] Debian bug 867921 in openstack-debian-images "cloud-init: Network configuration doesn't work" [Normal,Fixed] [14:13] But the dates don't quite align [14:14] I'd have to see your full config to figure that out [14:14] maybe there's a v1 config sent with dhcp? [14:14] It's using v2 and no shouldn't be DHCP, but let me throw it in a gist for a second set of eyes [14:15] cloud-init query [14:15] something or the other [14:18] Here's the user-data, meta-data, network-config that's being thrown into the nocloud datastore, as well as `cloud-init query -a`: https://gist.github.com/bswinnerton/86e2e44d4dee4b5271be92580afd056c [14:28] Started a conversation with the debian cloud folks here: https://lists.debian.org/debian-cloud/2020/06/msg00073.html [14:34] Answered my own question: it's not related to the local vs net dsstore option of cloud-localds. It still tries to get a lease on boot even with the value set to local [14:34] I bet it's because of the dhcp config that lives in /run/network/interfaces.d/ [14:34] meena: any idea where files in that directory come from? Is that part of cloud-init? [14:35] Oh yeah, that's definitely it. Removing the file leads to super quick boot times [14:36] God what a mess [14:50] that a patch from Debian? [14:51] https://github.com/canonical/cloud-init/commit/a6faf3acef02bd8cd4d46ac9efeebf24b3f21d81.patch [14:53] That's the first problem, the second is the fact that a file gets added to `/run/network/interfaces.d/` that defaults to DHCP [14:53] And I'm not sure where that comes from, but am working on terrible hacks with user-data to get rid of it [14:55] logs should be helpful [14:55] maybe [15:00] Not seeing any references to that directory in the log, so it must be coming from the Debian cloud image [15:01] Is runcmd the best hook for modifying network configurations early in the boot process? It only needs to run once [17:15] network is usually the first thing to be run, so there's almost no chance of doing anything before that dhcp thing [17:15] unless [17:17] https://cloudinit.readthedocs.io/en/latest/topics/modules.html#bootcmd [17:17] bswinnerton: this could be worth a shot, if you Anna rm that file, so you don't have to modify the image :D [17:21] wanna [17:22] i am now second stage tired / exhausted. [17:22] Good evening.