| holmanb | minimal: ahh interesting | 02:34 |
|---|---|---|
| holmanb | I saw something else today that had a bash shebang line that made me wonder if it was a hard requirement | 02:34 |
| holmanb | Since POSIX shell is more portable that probably makes more sense anyway | 02:39 |
| holmanb | I'd have to dig a little to know what our hotplug testing looks like - I haven't touched that | 02:40 |
| === cpaelzer_ is now known as cpaelzer | ||
| minimal | any particular reason why the NoCloud seed-from functionality doesn't fetch a network-config URL? After all eni network config can be embedded in the meta-data requested so why not support network-config also? | 15:41 |
| gjolly | Hello cloud-init people! I was looking at boot times on Azure and especially for minimal Ubuntu instances (so without snapd running). Most of the boot time is spent on cloud-init-local.service (13s over 18s to reach ssh.service) and I would like to kindly ask you if you could have a look at those logs: https://pastebin.ubuntu.com/p/vCG8ztgJjd/ to see if there is anything that could be optimized there. Thanks! | 16:23 |
| gjolly | The "Timeout too large reducing to: 2147483646 (TIME_MAX - 1)" made me wonder if something could be optimized here. | 16:25 |
| blackboxsw | gjolly hi. looking that over, the timeout too large is actually the dhclient saying that the timeout Azure DHCP servers are sending us to too large, so dropping it locally to one less than the TIME_MAX. cloud-init in local boot stage on Azure relies on the DHCP backplane to give enough routing information for cloud-init to 'find' the instance metadata service on Azure in order to crawl it and collect desired network config for the vm. | 16:30 |
| blackboxsw | gjolly: and as your logs capture given the time stamps, near the entirety of this boot time cost in init-local looks to be the dhclient request waiting on a DHCP response from Azure DHCP servers on this region. We could opt to set a tighter dhclient timeout if we don't like waiting 12 seconds..... but that will trigger cloud-init to give up on detecting Azure in init-local stage and fallback to "init network" boot stage. | 16:32 |
| blackboxsw | it feels a bit like this DHCP server is overloaded possibly and not responding in a timely manner to DHCP requests. | 16:33 |
| gjolly | blackboxsw: Very interesting thank you for you help. I will try in other locations to see if it helps! | 16:35 |
| blackboxsw | gjolly: please do, that is an incredibly long time to wait on init-local on Azure for DHCP response | 16:37 |
| gjolly | blackboxsw: cloud-init-local.service @673ms +4.243s indeed it's way better in this other DC. | 16:38 |
| blackboxsw | thx, typical times we've seen for init-local are below 2 seconds generally | 16:41 |
| meena | Infiniband is my arch nemesis. | 17:24 |
| holmanb | minimal: Not sure why it doesn't - I would assume historical reasons. | 20:04 |
| holmanb | as in, never got around to it or had a request for it | 20:04 |
| holmanb | Sounds like a reasonable addition to me (and a better interface too). | 20:04 |
| holmanb | We may want to deprecate the eni-specific key from meta-data too if we do add support for network-config at a url. | 20:06 |
| minimal | yeah, this area needs some work, I need to find some time to get back to testing it some more. I think it should be using ephimeral DHCP rather than configuring a network interface (via eni on Alpine/Debian) as that messes things up later (as the interface never has "ifdown eth0" performed and then the 2nd "ifup eth0" (with the IP settings passed via meta-info's eni-settings) fails | 20:12 |
| holmanb | that sounds like a bug to me | 20:13 |
| minimal | using "-f" for the ifup command acts as a workaround for ifupdown but doesn't work for ifupdown-ng | 20:14 |
| holmanb | gotcha | 20:16 |
| holmanb | meena: stuck on something? | 20:16 |
| minimal | yeah it shouldn't we writing an /e/n/i file in the first place for its temporary interface config | 20:17 |
| minimal | looking at my old notes on this, its doing "Apply network configuration from fallback", then "Writing to /etc/network/interfaces" | 20:20 |
| minimal | in "init-local" | 20:20 |
| minimal | then the os bring up the interfaces as defined in /e/n/i | 20:22 |
| minimal | then c-i in "init" downloads from the seed-urls (include meta-data with the network settings) | 20:22 |
| minimal | then it does "applying net config names for...." and "Applying network configuration", and "Writing to /etc/network/interfaces" (where this is the config obtained from meta-info) | 20:23 |
| minimal | then then "Bringing up newly configured network interfaces" (which is already up as the OS did so between "init-local" and "init" stages) | 20:24 |
| meena | holmanb: i got access to an HPC Azure machine with infiniband devices, and I have no idea how to even make it show it up in ifconfig as NIC | 20:32 |
| minimal | meena: you have the hyper-v drivers active? | 20:37 |
| meena | minimal: hn0 is up and running; | 20:38 |
| minimal | so ifconfig hn0 shows nothing? | 20:43 |
| meena | minimal: ifconfig hn0 shows the default config, otherwise i wouldn't be able to connect | 20:49 |
| holmanb | this is using freebsd I assume? | 20:50 |
| minimal | so you should be seeing 2 interfaces per "virtual NIC" apparently, one via the hyper-v netvsc (which I assume is hn0 in your case) and the 2nd is a SR-IOV Virtual Function which shows up as a PCI device and uses the mlx driver | 20:57 |
| minimal | at least according to this: https://learn.microsoft.com/en-us/azure/virtual-network/accelerated-networking-how-it-works | 20:57 |
| meena | minimal: that's on the more or less regular instances. This is an HPC instance, and the mlx nic is an infiniband device. | 23:11 |
| minimal | meena: ok, seems like Mellanox OFED drivers may be required | 23:31 |
| meena | minimal: they are loaded; the stuff shows up in pciconf, and in ibstats, and so on, but not in infconfig; | 23:32 |
| meena | I am finding that somewhat confusing | 23:32 |
| meena | I'm currently building FreeBSD 14.0-CURRENT, WITH_OFED_EXTRA to see if that gets me even a cm further. | 23:33 |
| minimal | the documentation links I found on Nvidia's site for FreeBSD OFED don't work | 23:33 |
| meena | Love me a good acquisition… | 23:34 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!