[08:08] <meena> so why would you want infiniband if 400G ethernet exists
[14:22] <meena> aciba: can I please get the stale flag removed from https://github.com/canonical/cloud-init/pull/1957 ? 
[14:22] -ubottu:#cloud-init- Pull 1957 in canonical/cloud-init "add function is_virtual to distro/FreeBSD" [Open]
[14:29] <aciba> meena: done
[14:30] <meena> there really needs to be some kind mechanism for saying this depends on that
[14:41] <minimal> meena: infiniband has been around a long time, and certainly when it first came out it was faster than ethernet
[14:43] <minimal> infiniband has advantages other than speed
[14:54] <holmanb> RDMA is low latency and doesn't eat CPU cycles the way ethernet does - perfect for scientific computing.
[14:54] <holmanb>  You can pin a dma buffer and the nic drops (potentially really big) chunks of data into the buffer before letting the kernel know it's arrived (major oversimplification). 
[14:55] <holmanb> IB is used in many of the biggest HPC clusters, and many storage appliances and parallel filesystems (i.e. lustre) use it for the same reason.
[14:58] <holmanb> frankly our IB code is really weird and very openstack specific - I would argue that it should be refactored into openstack datasource
[14:59] <holmanb> *the weird parts of it should be refactored
[15:02] <minimal> the only time I ever worked with Infiniband was on a (very large) bare metal private cloud for very large in-memory databases, obviously low latency and CPU load was important there
[15:12] <holmanb> lots of fun tech came out of the early 2000s :)
[15:12] <minimal> holmanb: it was less than 5 years ago I had to deal with IB though lol
[15:13] <holmanb> ah, fair
[15:26] <meena> aye. I feel like i knew most of these things, but also keep forgetting them, because my Hardware knowledge is severely limited by the fact that I'm not allowed to touch hardware, lest it explode.
[15:27] <meena> it's really just my talent for exposing bugs, but if you transpose exposing bugs onto hardware it should just add easily catch fire
[15:28] <meena> could. could! not should
[15:31] <meena> but yeah, holmanb, when i was looking into refactoring some of most important parts of cloudinit.net (get_interfaces and friends) some of the dependent functions where ib, and the only place they're called is in OpenStack
[15:33] <meena> but even if they weren't OpenStack specific, the way they are written is just a waste of CPU
[15:34] <minimal> the Azure MANA network interfaces are based on Infiniband AFAIK
[15:37] <holmanb> minimal: I don't think we actually support configuring them though, right?
[15:39] <minimal> holmanb: no idea, don't know muxch about them, however the Linux driver for them has been around for almost 2 years and MANA == Microsoft Azure Network Adaptor so I'd expect at least some of their instance types to have them
[15:39] <minimal> I think of MANA as the Azure equivalent of AWS's ENA adaptors
[15:40] <minimal> meena: there even seems to be FreeBSD support for MANA ;-)
[15:40] <waldi> i have yet to seen any instance types with MANA
[15:42]  * meena just wants to replay secret of Mana now
[15:43] <minimal> wadi: seems strange to introduce a new network adaptor and a couple of years later not to actually use it.....
[15:44] <waldi> yes
[15:45] <waldi> in addition to virtual function (aka for vms), microsoft also added support for the physical function part of this device. i wonder if they want to run linux as vm host
[15:49] <minimal> AWS also have EFA interfaces but I'm not sure if they're infiniband or something else
[15:49] <holmanb> minimal: I think efa is its own silicon
[15:50] <minimal> holmanb: I saw some mentioned of RDMA so wondered
[15:51] <holmanb> +1 they have their own driver and userspace bits iirc
[15:51] <minimal> right, I was trying to find the source for it to check as it is not part of stock kernel...
[15:57] <holmanb> minimal: see infiniband/hw/efa/ in the kernel source
[15:58] <holmanb> https://elixir.bootlin.com/linux/v6.2-rc6/source/drivers/infiniband/hw/efa/efa.h
[15:59] <minimal> holmanb: right, so it is sort of IB-based/derived as its inside drivers/infiniband...
[16:00] <minimal> I thought it was still an out-of-kernel driver, obviously not
[16:01] <minimal> efa_main.c:     ibdev_info(&dev->ibdev, "IB device registered\n");
[16:01] <minimal> so that refers to it as a IB device ;-)
[16:06] <meena> I'm anyways so impressed when people can read Linux kernel source. i can't even find where to look for the right version
[16:07] <meena> honestly, that's most of the reason I moved to FreeBSD
[16:08] <holmanb> yeah, it's in the infiniband family, but I'd be curious how much of the spec (or compatibility with other devices) they actually support
[16:12] <minimal> holmanb: the efa_verbs.c file make a lot of use of IB_* defintions, the other files don't at all
[16:13] <minimal> meena: reading the kernel source is easy, understanding it is a completely different matter however ;-)
[16:14] <meena> but that's the thing: I have an easy enough time *understanding* FreeBSD source most of the time
[16:17] <holmanb> minimal: iirc there was some integration with nvidia gpus that efa was capable of or intended for - bit of a funny collaboration since nvidia bought mellanox
[16:21] <minimal> holmanb: well Nvidia have been on a shopping spree the past few years relating to DC tech - Mellanox for DPU and networking, they tried Arm for CPUs, 
[16:21] <holmanb> don't forget cumulus
[16:22] <holmanb> and bright
[16:23] <meena> I can't believe i actually got a response to my email https://lists.freebsd.org/archives/freebsd-infiniband/2023-February/000005.html
[16:23] <minimal> had missed the cumulus deal, bright I'd never heard of
[16:23] <meena> this is the fifth email to this list, so that's how active it is
[16:24] <minimal> meena: well infiniband is low latency ;-)
[16:24] <meena> lol
[16:25] <meena> you're in a fabulous mood, minimal. did you throw away the FreeBSD and RHEL code from your PR yet?
[16:25] <minimal> add that to the corny IT jokes list eh? "I could tell you a joke about UDP but I'm not sure you'd get it" :-)
[16:25] <holmanb> meena: was the response direct? I don't see it on the list
[16:25] <minimal> meena: am in the midst of expunging that currently
[16:26] <meena> holmanb: yes
[16:26] <holmanb> meena: I know on linux, ifconfig never grew the ability to see infiniband hw addresses - iproute2 did
[16:27] <meena> holmanb: I will never tire of explaining to people that BSD ifconfig is nothing like Linux ifconfig.
[16:28] <holmanb> :D
[16:28] <minimal> meena: I haven't used ifconfig in Linux in maybe 7+ years now...
[16:28] <meena> BSD ifconfig was always more like Linux ip a & ip link
[16:29] <meena> whereas Linux ifconfig was always like an unloved step child, who was eventually forgotten at a gas station during a vacation trip
=== cpaelzer_ is now known as cpaelzer
[16:30] <holmanb> meena: like it in which ways?
[16:30] <holmanb> bnf-esq cli?
[16:32] <meena> the CLI is backwards compatible, slowly changes, but exposes every bit of info and every bit of configurability
[17:10] <minimal> meena: I don't know if cloud-init supports any Linux distros that have ifconfig but not ip - holmanb, any thoughts?
[17:10] <minimal> perhaps RHEL and derivatives?
[17:10] <holmanb> rhel is on iproute2 for any version we still support
[17:12] <meena> yeah, all the Linux ifconfig code in cloud-init is super crusty. it's basically really just in netinfo
[17:13] <meena> and, honestly, netinfo itself is pretty awful code, that, but the end of this refactor could hopefully be deleted
[17:24] <minimal> holmanb: wondering whether after 23.1 there's any point in removing non-BSD ifconfig support?
[17:30] <holmanb> minimal: probably worth a mailing list comment to give any stragglers a chance to speak up, but I think that would probably be fine
[17:37] <meena> like, all the Linux ifconfig code is vestigial, at best. I doubt any of it is in any way still fully functional
[17:39] <meena> and, honestly, even if our Linux ifconfig code isn't, Linux' ifconfig probably can't do half of what we use iproute2 for
[17:40] <holmanb> true, our ephemeral dhcp setup is all iproute2
[17:46] <minimal> meena: that was the reason that ip/iproute originally came into being - settings like bridge/vlan were done using other tools separate from ifconfig and "ip" was developed to have a unified tool
[17:48] <meena> aye.
[17:49] <meena> again, BSD ifconfig does all that, which makes sense, cuz it was always developed in line with the kernel features
[18:33] <meena> who here is an OpenStack contributor?
[18:34] <waldi> security fixes many years ago don't count
[18:35] <meena> https://freebsdfoundation.org/project/openstack-on-freebsd/ wonder if this person has access to infiniband hardware lol
[18:45] <meena> wow, how… how do you end up with a university email address of sexbear@?
[19:35] <meena> https://cs.github.com/canonical/cloud-init?q=infiniband Search failed: undefined
[19:35] <meena> cool cool cool
[19:36] <meena> well, it is beta, I guess