| meena | so why would you want infiniband if 400G ethernet exists | 08:08 |
|---|---|---|
| meena | aciba: can I please get the stale flag removed from https://github.com/canonical/cloud-init/pull/1957 ? | 14:22 |
| -ubottu:#cloud-init- Pull 1957 in canonical/cloud-init "add function is_virtual to distro/FreeBSD" [Open] | 14:22 | |
| aciba | meena: done | 14:29 |
| meena | there really needs to be some kind mechanism for saying this depends on that | 14:30 |
| minimal | meena: infiniband has been around a long time, and certainly when it first came out it was faster than ethernet | 14:41 |
| minimal | infiniband has advantages other than speed | 14:43 |
| holmanb | RDMA is low latency and doesn't eat CPU cycles the way ethernet does - perfect for scientific computing. | 14:54 |
| holmanb | You can pin a dma buffer and the nic drops (potentially really big) chunks of data into the buffer before letting the kernel know it's arrived (major oversimplification). | 14:54 |
| holmanb | IB is used in many of the biggest HPC clusters, and many storage appliances and parallel filesystems (i.e. lustre) use it for the same reason. | 14:55 |
| holmanb | frankly our IB code is really weird and very openstack specific - I would argue that it should be refactored into openstack datasource | 14:58 |
| holmanb | *the weird parts of it should be refactored | 14:59 |
| minimal | the only time I ever worked with Infiniband was on a (very large) bare metal private cloud for very large in-memory databases, obviously low latency and CPU load was important there | 15:02 |
| holmanb | lots of fun tech came out of the early 2000s :) | 15:12 |
| minimal | holmanb: it was less than 5 years ago I had to deal with IB though lol | 15:12 |
| holmanb | ah, fair | 15:13 |
| meena | aye. I feel like i knew most of these things, but also keep forgetting them, because my Hardware knowledge is severely limited by the fact that I'm not allowed to touch hardware, lest it explode. | 15:26 |
| meena | it's really just my talent for exposing bugs, but if you transpose exposing bugs onto hardware it should just add easily catch fire | 15:27 |
| meena | could. could! not should | 15:28 |
| meena | but yeah, holmanb, when i was looking into refactoring some of most important parts of cloudinit.net (get_interfaces and friends) some of the dependent functions where ib, and the only place they're called is in OpenStack | 15:31 |
| meena | but even if they weren't OpenStack specific, the way they are written is just a waste of CPU | 15:33 |
| minimal | the Azure MANA network interfaces are based on Infiniband AFAIK | 15:34 |
| holmanb | minimal: I don't think we actually support configuring them though, right? | 15:37 |
| minimal | holmanb: no idea, don't know muxch about them, however the Linux driver for them has been around for almost 2 years and MANA == Microsoft Azure Network Adaptor so I'd expect at least some of their instance types to have them | 15:39 |
| minimal | I think of MANA as the Azure equivalent of AWS's ENA adaptors | 15:39 |
| minimal | meena: there even seems to be FreeBSD support for MANA ;-) | 15:40 |
| waldi | i have yet to seen any instance types with MANA | 15:40 |
| * meena just wants to replay secret of Mana now | 15:42 | |
| minimal | wadi: seems strange to introduce a new network adaptor and a couple of years later not to actually use it..... | 15:43 |
| waldi | yes | 15:44 |
| waldi | in addition to virtual function (aka for vms), microsoft also added support for the physical function part of this device. i wonder if they want to run linux as vm host | 15:45 |
| minimal | AWS also have EFA interfaces but I'm not sure if they're infiniband or something else | 15:49 |
| holmanb | minimal: I think efa is its own silicon | 15:49 |
| minimal | holmanb: I saw some mentioned of RDMA so wondered | 15:50 |
| holmanb | +1 they have their own driver and userspace bits iirc | 15:51 |
| minimal | right, I was trying to find the source for it to check as it is not part of stock kernel... | 15:51 |
| holmanb | minimal: see infiniband/hw/efa/ in the kernel source | 15:57 |
| holmanb | https://elixir.bootlin.com/linux/v6.2-rc6/source/drivers/infiniband/hw/efa/efa.h | 15:58 |
| minimal | holmanb: right, so it is sort of IB-based/derived as its inside drivers/infiniband... | 15:59 |
| minimal | I thought it was still an out-of-kernel driver, obviously not | 16:00 |
| minimal | efa_main.c: ibdev_info(&dev->ibdev, "IB device registered\n"); | 16:01 |
| minimal | so that refers to it as a IB device ;-) | 16:01 |
| meena | I'm anyways so impressed when people can read Linux kernel source. i can't even find where to look for the right version | 16:06 |
| meena | honestly, that's most of the reason I moved to FreeBSD | 16:07 |
| holmanb | yeah, it's in the infiniband family, but I'd be curious how much of the spec (or compatibility with other devices) they actually support | 16:08 |
| minimal | holmanb: the efa_verbs.c file make a lot of use of IB_* defintions, the other files don't at all | 16:12 |
| minimal | meena: reading the kernel source is easy, understanding it is a completely different matter however ;-) | 16:13 |
| meena | but that's the thing: I have an easy enough time *understanding* FreeBSD source most of the time | 16:14 |
| holmanb | minimal: iirc there was some integration with nvidia gpus that efa was capable of or intended for - bit of a funny collaboration since nvidia bought mellanox | 16:17 |
| minimal | holmanb: well Nvidia have been on a shopping spree the past few years relating to DC tech - Mellanox for DPU and networking, they tried Arm for CPUs, | 16:21 |
| holmanb | don't forget cumulus | 16:21 |
| holmanb | and bright | 16:22 |
| meena | I can't believe i actually got a response to my email https://lists.freebsd.org/archives/freebsd-infiniband/2023-February/000005.html | 16:23 |
| minimal | had missed the cumulus deal, bright I'd never heard of | 16:23 |
| meena | this is the fifth email to this list, so that's how active it is | 16:23 |
| minimal | meena: well infiniband is low latency ;-) | 16:24 |
| meena | lol | 16:24 |
| meena | you're in a fabulous mood, minimal. did you throw away the FreeBSD and RHEL code from your PR yet? | 16:25 |
| minimal | add that to the corny IT jokes list eh? "I could tell you a joke about UDP but I'm not sure you'd get it" :-) | 16:25 |
| holmanb | meena: was the response direct? I don't see it on the list | 16:25 |
| minimal | meena: am in the midst of expunging that currently | 16:25 |
| meena | holmanb: yes | 16:26 |
| holmanb | meena: I know on linux, ifconfig never grew the ability to see infiniband hw addresses - iproute2 did | 16:26 |
| meena | holmanb: I will never tire of explaining to people that BSD ifconfig is nothing like Linux ifconfig. | 16:27 |
| holmanb | :D | 16:28 |
| minimal | meena: I haven't used ifconfig in Linux in maybe 7+ years now... | 16:28 |
| meena | BSD ifconfig was always more like Linux ip a & ip link | 16:28 |
| meena | whereas Linux ifconfig was always like an unloved step child, who was eventually forgotten at a gas station during a vacation trip | 16:29 |
| === cpaelzer_ is now known as cpaelzer | ||
| holmanb | meena: like it in which ways? | 16:30 |
| holmanb | bnf-esq cli? | 16:30 |
| meena | the CLI is backwards compatible, slowly changes, but exposes every bit of info and every bit of configurability | 16:32 |
| minimal | meena: I don't know if cloud-init supports any Linux distros that have ifconfig but not ip - holmanb, any thoughts? | 17:10 |
| minimal | perhaps RHEL and derivatives? | 17:10 |
| holmanb | rhel is on iproute2 for any version we still support | 17:10 |
| meena | yeah, all the Linux ifconfig code in cloud-init is super crusty. it's basically really just in netinfo | 17:12 |
| meena | and, honestly, netinfo itself is pretty awful code, that, but the end of this refactor could hopefully be deleted | 17:13 |
| minimal | holmanb: wondering whether after 23.1 there's any point in removing non-BSD ifconfig support? | 17:24 |
| holmanb | minimal: probably worth a mailing list comment to give any stragglers a chance to speak up, but I think that would probably be fine | 17:30 |
| meena | like, all the Linux ifconfig code is vestigial, at best. I doubt any of it is in any way still fully functional | 17:37 |
| meena | and, honestly, even if our Linux ifconfig code isn't, Linux' ifconfig probably can't do half of what we use iproute2 for | 17:39 |
| holmanb | true, our ephemeral dhcp setup is all iproute2 | 17:40 |
| minimal | meena: that was the reason that ip/iproute originally came into being - settings like bridge/vlan were done using other tools separate from ifconfig and "ip" was developed to have a unified tool | 17:46 |
| meena | aye. | 17:48 |
| meena | again, BSD ifconfig does all that, which makes sense, cuz it was always developed in line with the kernel features | 17:49 |
| meena | who here is an OpenStack contributor? | 18:33 |
| waldi | security fixes many years ago don't count | 18:34 |
| meena | https://freebsdfoundation.org/project/openstack-on-freebsd/ wonder if this person has access to infiniband hardware lol | 18:35 |
| meena | wow, how… how do you end up with a university email address of sexbear@? | 18:45 |
| meena | https://cs.github.com/canonical/cloud-init?q=infiniband Search failed: undefined | 19:35 |
| meena | cool cool cool | 19:35 |
| meena | well, it is beta, I guess | 19:36 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!