spetrosiHi folks, I am looking at the cloud-init feature of creating a /etc/NetworkManager/conf.d/99-cloud-init.conf file to ensure that DNS is managed by cloud-init. The only workaround I see to this is to add a `/etc/NetworkManager/conf.d/9A-override-99-cloud-init.conf` file with `dns=default` to override cloud-init's configuration. Is there some more valid fix? Btw I do not have a `resolve` section in my /etc/cloud/cloud.cfg15:09
minimalspetrosi: which DataSource are you using?15:24
spetrosiminimal, it is recommended in https://access.redhat.com/solutions/475776115:26
minimalI can't read that as I don't have a login. However I did ask which DataSource you are using in order to understand your problem...15:28
minimalas for some DataSources any DNS server configuration will come from network-config and in other cases via user-data15:30
spetrosiminimal, It's a VM provisioned from OpenStack15:38
minimalok, so network settings including DNS are coming from either OpenStack DS or ConfigDrive DS15:39
minimaland you want NetworkManager to use the cloud-init DNS setting or to *not* use the cloud-init DNS settings?15:40
spetrosiminimal, to *not* use the cloud-init DNS settings15:56
smosermeena: be nice. https://bugs.launchpad.net/cloud-init/+bug/2006784 ;)16:15
-ubottu:#cloud-init- Launchpad bug 2006784 in cloud-init "dhcp lookups depend on end-of-life dhclient" [Undecided, New]16:15
smoserwrt to dhcp for ephemeral. i really think cloud-init should have a builtin (python) dhcp client that it can interact with inn this case rather than relying on some binary to execute.16:16
minimalsmoser: I talk with holmanb in the past about adding both dhcpcd and udhcpc support17:01
minimalsmoser: interesting that the report is regarding NixOS, which is *not* a currently supported distro...17:03
holmanbalso nixos's design isn't a good fit for cloud-init imo17:04
minimalwell it is more "your software doesn't support a situation in a distro that you don't support (and therefore haven't tested)"17:05
minimalthough they seem to have added it themselves: https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/services/system/cloud-init.nix17:06
minimalthere selection of enabled modules is "interesting" as some of them may not work as the modules don't define NixOS (or any family for it) as supported...17:09
holmanbsmoser: Current plans are to add fallback support for dhcpcd in the near term - other projects require an external dhcp client as well, and dhcpcd seems to meet the most needs.17:09
holmanbsmoser: I explored some manually byte packed dhcp frames a while back to test out the idea - it seems doable, especially if limited to the functionality currently used rather than trying to implement a fully-featured client.17:10
smoseryeah, i kind of looked at that too. it was really surprising to me though. i found a number of dhcp clients in python, but they all required rthat the interfacxe you were using already have an ip address.17:11
smoserwhich, seems, surprising to me. and was then not useful for cloud-init. but i could have been missing something.17:11
smoserhttps://github.com/lvfrazao/dhcppython looks like it  might fit the bill. i dont think google turned that up so esaily when i last looked.17:13
minimalsmoser: from a quick scan it doesn't appear to do DHCPv6 though17:18
smoserdoes anything use that in ephemeral ?17:18
holmanbnot currently17:18
smoserif it magically worked, then its a huge improvement over what we have there. it has no dependencies.17:19
minimalperhaps not at present, but there is the future scenario of a IPv6-only environment ot consider17:19
smoserbut probably not packaged.17:19
smoserwell, it didn't "just work" for me in a container test. but that coudl be user error.17:32
smoserbut i really do think that such an approach is preferable compared to trying to kick muiltiple crappy dhcp clients into doing exactly what you want.  where "what you want" is not really a normal thing.17:33
meenasmoser: im always nice17:39
holmanbsmoser: in that case what would be the story for distro support? "use dhclient until you can get this new dependency packaged"?17:39
smoseror vendor i guess. 17:40
minimalstill need to support dhcpcd and/or another client outside of the ephemeral stuff - i.e. the client used by the distro - currently cloud-init writes a dhclient "hook" script (from memory)17:50
minimalwhich writes /etc/dhcp/dhclient-exit-hooks.d/hook-dhclient17:52
meenasmoser: i think we discussed adding scapy as dependency a few years back17:52
holmanbscapy is pretty big iirc17:53
holmanbwhich I think was impetus why I was looking into crafting byte-packed frames17:53
meenaand probably overpowered for most things we need17:54
smoseryeah, scapy is huge.17:54
holmanbminimal: What's the background on dhclient_hook? I see what it does, but I'm missing context on why we support this, and the docs for it are.... on par with many of our more obscure features17:57
meenaFreeBSD has a package, https://www.freshports.org/net/scapy/ but disables a lot of features by default17:59
minimalholmanb: not sure myself - looking on a NoCloud based box the hook script appears to be designed to only active on Azure to run "cloud-init dhclient-hook up/down <interface>"18:02
holmanbminimal: I see the initial commit message for that file explains it18:05
minimalI should have looked for that ;-)18:06
minimalmy point being the need to create equivalents hooks for dhcpcd and udhcpc18:09
holmanbThe hook is currently executed by dhclient, which is executed by cloud-init, right? 18:40
* holmanb boots on azure to poke at it18:44
minimalthe hook is run by dhclient, not sure if that is when dhclient is run by cloud-init as part of ephemeral DHCP or if it is later in boot when the OS brings up network interfaces18:44
holmanbminimal: dhclient-hook looks vestigal to me19:31
holmanbminimal: azure doesn't appear to be consuming that env var json file anymore19:33
minimalholmanb: cool. Less work to handle other dhclient alternatives then19:34
meenaholmanb: 🪓❓19:49
holmanbmeena: mayhaps19:55
holmanbmeena: Something that's a top level cloud-init command should probably be deprecated first, but since this looked to be cloud-specific cmd, and is unlikely to have external consumers (all it does is filter some specific env vars and stuff them in a file as json), it might be something we could axe without deprecation.19:56
holmanbEither way, it would be best to check with msft to make sure this file isn't used in some other way (i.e. WALinuxAgent directly), and to make sure I didn't miss something.19:57
minimalholmanb: there's nothing in the hotplug network interface functionality (i.e. AWS and Openstack currently) that relies on dhclient and/or its hooks?20:01
holmanbminimal: I'm not sure on that one. I thought that was udev-based.20:03
jchittumdefinitely want to be careful with Azure and hotplugging networking. hotplug networking is done to setup the Advanced Networking, which is a core functional requirement of AKS20:09
jchittumthere's a dragon in there somewhere, as we found out with the systemd / udev shenanigans in August20:09
=== arita- is now known as arita
holmanbjchittum: +1 more than one dragon I suspect20:34
blackboxswminimal/holmanb sorry for delay on separate CA certs PR. I'm almost done on my end, couple minor tweaks to unittest/docs/integrationtests20:39
blackboxswPR looks really good thx20:40
holmanbcjp256: github won't let me request you as a reviewer for some reason, but if you get a chance sometime https://github.com/canonical/cloud-init/pull/201520:40
-ubottu:#cloud-init- Pull 2015 in canonical/cloud-init "[RFC] remove vestigal dhclient_hook command" [Open]20:40
holmanbcjp256: which is followup to https://github.com/canonical/cloud-init/commit/5ad0768a796bc07232476d0d29b5225f1e6e131c20:40
-ubottu:#cloud-init- Commit 5ad0768 in canonical/cloud-init "sources/azure: remove lease file parsing (#1302)"20:40
blackboxswok done on https://github.com/canonical/cloud-init/pull/1962 think this'll make 23.1 release.22:01
-ubottu:#cloud-init- Pull 1962 in canonical/cloud-init "overhaul cc_ca_certs functionality" [Open]22:01
minimalblackboxsw: Thanks, I'll have a look shortly22:03
blackboxswthere's a secondary bug with SSH host keys floating around.... that I also want to take a look at to see if we can tackle it for this release. https://bugs.launchpad.net/cloud-init/+bug/199916422:05
-ubottu:#cloud-init- Launchpad bug 1999164 in cloud-init "when multiple SSH host key certificates are defined, only one HostCertificate is referenced in sshd_config" [High, Confirmed]22:05
blackboxswIt's now in our selected for development queue, so hopefully we can make some substantive progress there shortly22:05
blackboxswnote that I may be incorrect on the CA certs behavior suggestion from RHEL(esposem). holmanb/minimal do you understand that they wanted to still wait on us setting this behavior in general and landing it in tip of main until they change RHEL behavior. Or did minimal's backing out RHEL changes avoid this concern (because redhat will still continue to fully `remove_default_ca_certs`22:08
blackboxswminimal/holmanb Let's restate that question clearly. Do you think were waiting on a fix in redhat before we could land PR 1962 still?22:09
holmanbblackboxsw: I don't sense a change in behavior comming from RHEL, just a change in docs (which already landed).22:10
blackboxswThank you. I wanted to confirm I thought minimal backed out any rhel-related changes.22:11
holmanbblackboxsw: minimal previously had rhel "not implemented" in hopes of using a RHEL dedicated utility, but from RHEL it seems that their utility isn't intended to manage system certs, so that part has been reverted to former behavior, iirc, which is to delete the certs22:13
minimalblackboxsw: yes I reverted RHEL and FreeBSD related changes as things were unclear22:13
emper0rhi.. i'm getting a little problem with cloud-init when boot an ec2 into aws..22:16
emper0ris taking 6 hours 22:16
emper0rstucking the boot until get login prompt22:17
blackboxswemper0r: that's a problem :/ `systemd-analyze blame` or `cloud-init analyze show`?22:17
blackboxswshould tell you the blocking service that is taking so long in boot22:17
emper0rin fact i take a svg image22:17
emper0rand say22:18
emper0rcloud-init.service (6h 21min 55.264s)22:18
blackboxswcloud-init analyze blame may tell you what part of cloud-init is blocking.22:19
emper0rwant see the full blame output?22:19
blackboxswsure pastebin.com works22:20
emper0rwait let me paste in some site 1 sec22:20
blackboxswgreat and 'cloud-init analyze blame' and 'cloud-init analyze show' if possible please22:22
emper0r1 sec22:22
blackboxswsomething is likely blocking cloud-init.service from running... but not sure yet what that is .... also surpising that mysqld.service is taking an hour to come up BTW22:22
blackboxsw*2 hours rather22:23
blackboxswmakes me thing network throughput,retries and timeouts with misconfigured network  or proxy issues etc22:23
emper0ryes we have many database using engine InnoDB so when start mysql start check internal to rollback and start ok i need to login meanwhile is testing that...22:23
emper0r1 sec to get output 22:24
emper0rblackboxsw: both output here https://dpaste.com/8EPJRFCBH22:26
blackboxswemper0r: thanks your cloud-init analyze blame (and show) both show on latest boot, whatever bootcmd script is provided to cloud-init as user-data or vendor-data in these images took a  really long time to complete. line 264 of your paste -- Boot Record 07 --22:33
blackboxsw     22913.76400s (init-network/config-bootcmd)22:33
blackboxswI'd check `sudo cloud-init query userdata` and see what script your image is running and debug there22:33
blackboxswin a `bootcmd` section.22:34
blackboxswas that's what took the 6 hrs22:35
emper0rblackboxsw: https://dpaste.com/AKJ95THPC22:39
emper0ris a simple script we create to auto-registrate any ec2 into our private hosted zone automaticly22:39
emper0rwait.. now checking that script log22:43
emper0ri see the timeout registering into our zone22:43
emper0rCreate/Update EC2 record22:44
emper0rError: RequestError: send request failed22:44
emper0rcaused by: Get "https://route53.amazonaws.com/2013-04-01/hostedzonesbyname?dnsname=22:44
emper0rdial tcp i/o timeout22:44
emper0rand the start and stop logs are22:44
emper0r20230209-15:01 and ends 20230209-21:2322:45
emper0rwith exactly that 6 hour in the middle22:45
blackboxsw+1. yeah I haven't played w/ route 53 offerings much, just read a couple docs on it.... but interesting little side script there and falling over on the setup/config I think22:45
emper0ri guess that is the cause22:45
emper0rhmm have to check if have some option about timeout to set not more than x min to avoid this 22:46
emper0rthanks for all blackboxsw i guess I can handle from this..22:46
blackboxswtake care22:47

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!