[12:21] <bswinnerton> Hello, first time playing around with cloud init. Things are going great in Ubuntu, but having some problems configuring networking in Debian. I'm using a nocloud database with a v2 network configuration file to set a static ip address. The user-data and meta-data is applied successfully, but after the machine comes up, it's falling back to DHCP.
[12:21] <bswinnerton> Upon further inspection, this is because /etc/network/interfaces does a source-directory for both /etc/network/interfaces.d/ (which has the correctly configured file), and /run/network/interfaces.d/ which has a file that conflicts and falls back to DHCP. Is there a way to turn off the latter?
[12:25] <bswinnerton> s/database/datastore
[12:36] <bswinnerton> hmm, there's also something weird going on similar to https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=867921
[12:38] <bswinnerton> So to recap, when using nocloud with a v2 network config that sets a static ip address, when the Debian host comes up, a few things need to happen in order for the machine to get networking: 1) `rm /run/network/interfaces.d/50-cloud-init.cfg` (which includes the DHCP setting), and 2) `mv /etc/network/interfaces.d/50-cloud-init.cfg
[12:38] <bswinnerton> /etc/network/interfaces.d/50-cloud-init`, once those have been completed, a `systemctl restart networking` can be run and everything works fine.
[12:39] <bswinnerton> This is cloud-init 18.3
[13:46] <meena> bswinnerton: have you tried a more recent version to see if this is fixed?
[13:47] <bswinnerton> I haven't, I'm using what's built into the debian cloud images
[13:56] <bswinnerton> In what scenarios would cloud-init write to `/run/network/interfaces.d/`? Is that the fallback described in https://cloudinit.readthedocs.io/en/latest/topics/network-config.html#fallback-network-configuration?
[13:57] <meena> I'm trying to remember if we have deb packages pre-built anyhere
[13:58] <meena> my brain isn't working very well today, so you might be quicker looking yourself
[13:59] <bswinnerton> https://packages.debian.org/buster/cloud-init
[13:59] <bswinnerton> They're quite old, which is why the debian images include them, I assume
[14:00] <meena> no backports?
[14:00] <bswinnerton> ah yes: https://packages.debian.org/source/buster-backports/cloud-init
[14:00] <meena> there's https://launchpad.net/ubuntu/+source/cloud-init but i don't know how well those will work on Debian
[14:03] <meena> bswinnerton: so let's try at version, and if that still fails, we'll file a bug report
[14:03] <meena> and by we, i mean, can you please
[14:04] <meena> s/at /that /
[14:05] <bswinnerton> Sounds good, installing 20.2 now
[14:07] <bswinnerton> Is there an effective way to test this with cloud-init on the guest without rebuilding the image?
[14:07] <bswinnerton> Or does the image need to be rebuilt?
[14:07] <meena> install, run cloud-init clean --logs --reboot
[14:08] <bswinnerton> Neat, wasn't aware of `clean`. Doing that now
[14:08] <meena> that will regenerate everything on the server, including ssh server keys
[14:10] <bswinnerton> Sigh, that fixed it
[14:11] <bswinnerton> Okay, off to pester the Debian folks. Thanks meena
[14:11] <meena> 😽
[14:12] <meena> if i really were useful, I'd help you pinpoint where that was fixed, exactly
[14:12] <meena> but screw that, let them upgrade to the latest version :P
[14:13] <bswinnerton> A quick clarifying question, even after bumping to 20.2 I notice a very long delay when trying to bring up the interface, it appears that it's trying to get a DHCP lease with "Started ifup for ens3". It then eventually times out and continues along, and sets the static IP. Any idea how I could tell it not to try and DHCP? Is that the difference
[14:13] <bswinnerton> between a dsstore of local vs net?
[14:13] <bswinnerton> I have a feeling it was this: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=867921
[14:13] <bswinnerton> But the dates don't quite align
[14:14] <meena> I'd have to see your full config to figure that out
[14:14] <meena> maybe there's a v1 config sent with dhcp?
[14:14] <bswinnerton> It's using v2 and no shouldn't be DHCP, but let me throw it in a gist for a second set of eyes
[14:15] <meena> cloud-init query
[14:15] <meena> something or the other
[14:18] <bswinnerton> Here's the user-data, meta-data, network-config that's being thrown into the nocloud datastore, as well as `cloud-init query -a`: https://gist.github.com/bswinnerton/86e2e44d4dee4b5271be92580afd056c
[14:28] <bswinnerton> Started a conversation with the debian cloud folks here: https://lists.debian.org/debian-cloud/2020/06/msg00073.html
[14:34] <bswinnerton> Answered my own question: it's not related to the local vs net dsstore option of cloud-localds. It still tries to get a lease on boot even with the value set to local
[14:34] <bswinnerton> I bet it's because of the dhcp config that lives in /run/network/interfaces.d/
[14:34] <bswinnerton> meena: any idea where files in that directory come from? Is that part of cloud-init?
[14:35] <bswinnerton> Oh yeah, that's definitely it. Removing the file leads to super quick boot times
[14:36] <bswinnerton> God what a mess
[14:50] <meena> that a patch from Debian?
[14:51] <bswinnerton> https://github.com/canonical/cloud-init/commit/a6faf3acef02bd8cd4d46ac9efeebf24b3f21d81.patch
[14:53] <bswinnerton> That's the first problem, the second is the fact that a file gets added to `/run/network/interfaces.d/` that defaults to DHCP
[14:53] <bswinnerton> And I'm not sure where that comes from, but am working on terrible hacks with user-data to get rid of it
[14:55] <meena> logs should be helpful
[14:55] <meena> maybe
[15:00] <bswinnerton> Not seeing any references to that directory in the log, so it must be coming from the Debian cloud image
[15:01] <bswinnerton> Is runcmd the best hook for modifying network configurations early in the boot process? It only needs to run once
[17:15] <meena> network is usually the first thing to be run, so there's almost no chance of doing anything before that dhcp thing
[17:15] <meena> unless
[17:17] <meena> https://cloudinit.readthedocs.io/en/latest/topics/modules.html#bootcmd
[17:17] <meena> bswinnerton: this could be worth a shot, if you Anna rm that file, so you don't have to modify the image :D
[17:21] <meena> wanna
[17:22] <meena> i am now second stage tired / exhausted.
[17:22] <meena> Good evening.