| ahasenack | ddstreet: cjwatson wrt uring, mariadb and memlock, check "ulimit settings" in this doc: https://git.kernel.dk/cgit/liburing/tree/README#n19 | 13:31 |
|---|---|---|
| ahasenack | this is not mariadb calling mlock specifically, it's just a side effect of using uring | 13:32 |
| ahasenack | what we can tweak maybe is the size of the ring | 13:32 |
| ddstreet | ah that's interesting, i should have realized it from the name of the lib 'uring', makes sense. if the rlimit can't be increased then does seem like no other choice than to limit the uring size | 13:41 |
| cjwatson | The rlimit can be increased, I just advocate doing it in the distro rather than specifically for buildds | 13:44 |
| cjwatson | If that's thought to be the right thing to do | 13:45 |
| ahasenack | it's not even clear to me how the default memlock limit is calculated or set | 13:45 |
| cjwatson | Limiting the ring size doesn't seem silly if it's possible though | 13:45 |
| ahasenack | systemd manpages say one thing, but experiments show another | 13:45 |
| ahasenack | ddstreet: and fyi, the way things are currently, a focal host cannot run a jammy mariadb lxd without tweaking this on the host | 13:47 |
| ahasenack | the mariadb install will fail in the jammy lxd | 13:47 |
| ahasenack | mysql seems fine. I haven't checked if it's even using uring, but it installs fine in the same lxd at least | 13:48 |
| ddstreet | ahasenack there were some significant problems before around the rlimit size inside a container, as it was varying between host releases | 13:48 |
| ahasenack | / io_uring instance queue depth | 13:51 |
| ahasenack | const unsigned int kIoUringDepth = 256; | 13:51 |
| ahasenack | I guess I know where the 256 magic number comes from | 13:51 |
| * ahasenack runs some experiments | 13:52 | |
| ahasenack | hm, don't the units here seem odd: https://pastebin.ubuntu.com/p/94dhcWvcBj/ | 14:12 |
| ahasenack | I either get 65536 kbytes | 14:12 |
| ahasenack | or 64 kbytes | 14:12 |
| ahasenack | same host | 14:12 |
| ahasenack | it's a big difference | 14:12 |
| ahasenack | and I can't raise it inside the lxd, as expected | 14:12 |
| ahasenack | root@j-mysql:~# ulimit -l 65536 | 14:12 |
| ahasenack | -bash: ulimit: max locked memory: cannot modify limit: Operation not permitted | 14:12 |
| ahasenack | I wonder if there is some confusion about the unit, whether it's bytes or kbytes | 14:18 |
| ahasenack | ok, each ring entry costs 128 bytes | 14:26 |
| ahasenack | mariadb is setting it up with 2048 entries | 14:26 |
| ddstreet | ahasenack if i'm reading it correctly, it looks like lxd is defaulting to 64 for RLIMIT_MEMLOCK in containers... https://github.com/lxc/lxd/blob/master/lxd/main_forklimits.go#L103 | 14:30 |
| ahasenack | in /proc/<pid>/limits, the unit is bytes | 14:31 |
| ahasenack | in the ulimit tool, the unit is kbytes | 14:31 |
| ahasenack | ddstreet: but above that line, there is | 14:31 |
| ahasenack | if limit.name == "memlock" { | 14:31 |
| ahasenack | resource = unix.RLIMIT_MEMLOCK | 14:31 |
| ahasenack | ah, n/m | 14:32 |
| ddstreet | ah the '64' there isn't the default sorry | 14:32 |
| ddstreet | it's the 'bitsize' | 14:32 |
| ddstreet | i'm not very familiar with golang :) | 14:32 |
| stgraber | limits.kernel.memlock can likely be set to bump that | 14:33 |
| stgraber | https://linuxcontainers.org/lxd/docs/latest/instances/#resource-limits-via-limits-kernel-limit-name | 14:34 |
| ahasenack | stgraber: was wondering if there isn't a 1024 error somewhere | 14:34 |
| stgraber | ahasenack: LXD doesn't set any limits by default | 14:35 |
| stgraber | ahasenack: so you get what the kernel gets you | 14:35 |
| ahasenack | or if you really intended to have 64kbytes for the container, when the host has 64Mb | 14:35 |
| ahasenack | https://pastebin.ubuntu.com/p/94dhcWvcBj/ | 14:35 |
| ahasenack | anyway, that's a different issue | 14:37 |
| stgraber | ahasenack: that 64k is what the LXD snap is given | 14:38 |
| stgraber | ahasenack: we don't set it ourselves | 14:38 |
| ddstreet | ahasenack stgraber not sure if you noticed but all lxd containers (maybe all containers in general?) seem to be getting the reduced limit, e.g. https://pastebin.ubuntu.com/p/FfChSRxbrR/ | 14:39 |
| stgraber | stgraber@dakara:~$ cat /proc/$(cat /var/snap/lxd/common/lxd.pid)/limits | grep -i locked | 14:39 |
| stgraber | Max locked memory 65536 65536 bytes | 14:39 |
| ddstreet | maybe a snap or apparmor limit? | 14:39 |
| stgraber | ddstreet, ahasenack: yes, that's normal. The LXD snap is started with 64k apparently, don't know if it's a snapd, snap-confine or systemd thing and so everything that it spawns itself will have that limit applied too. | 14:39 |
| ahasenack | ok | 14:40 |
| stgraber | looks like it's the default for systemd units | 14:41 |
| stgraber | stgraber@dakara:~$ sudo cat /proc/$(pgrep sshd)/limits | grep -i locked | 14:42 |
| stgraber | Max locked memory 65536 65536 bytes | 14:42 |
| ahasenack | yeah, just checked another daemon, also that value | 14:42 |
| cjwatson | fwiw builders use chroots rather than lxd containers for .deb builds, so lxd wouldn't be the sole solution here | 14:42 |
| cjwatson | (I haven't been following in detail though) | 14:43 |
| ahasenack | right, I used lxd just because it was easier to use than a chroot | 14:43 |
| ahasenack | I needed something to have jammy on a focal kernel | 14:43 |
| ahasenack | it's good enough to test what I can tweak to get mariadb running | 14:44 |
| ahasenack | cjwatson: the default ulimit -l I get in a focal with 8Gb of RAM is 64Mb, can you check what you get in a focal builder host? `ulimit -a | grep locked` is best because it prints out the unit, so there is no confusion between 64kb and 64Mb | 15:00 |
| ahasenack | 64Mb would be more than enough for mariadb, it just needs 256kb | 15:00 |
| ahasenack | stgraber: > looks like it's the default for systemd units | 15:11 |
| ahasenack | one consequence is that mariadb in a jammy lxd container won't start | 15:11 |
| ahasenack | unsure yet how to address that | 15:11 |
| ahasenack | an unsuspecting user would have quite an investigation to find out what to tweak | 15:12 |
| ddstreet | maybe lxd should ask snapd to set a higher memlock limit for the lxd process? | 15:15 |
| ddstreet | 64k might be ok for general system services that don't typically need to lock memory, but containers probably should have a higher limit | 15:15 |
| ahasenack | we can also drop uring support in mariadb for the time being, go back to linux aio | 15:17 |
| ahasenack | or rather, the former. I think they mix aio with uring and aio with something-else | 15:17 |
| ahasenack | I'm not too versed in that | 15:18 |
| cjwatson | ahasenack: max locked memory (kbytes, -l) 65536 | 15:43 |
| ahasenack | 64Mb then | 15:55 |
| * ahasenack scratches head | 15:55 | |
| ahasenack | cjwatson: https://launchpadlibrarian.net/596606723/buildlog_ubuntu-jammy-amd64.mariadb-10.6_1%3A10.6.7-3_BUILDING.txt.gz shows 65536 bytes | 15:56 |
| ahasenack | search for "locked memory", there is a `ulimit -a` output at the end of the build | 15:56 |
| ddstreet | ahasenack i think this commit probably should be sru'ed for systemd https://github.com/systemd/systemd/commit/852b62507b2 | 16:02 |
| ubottu | Commit 852b625 in systemd/systemd "pid1,nspawn: raise default RLIMIT_MEMLOCK to 8M" | 16:02 |
| cjwatson | ahasenack: I don't know, sorry. Maybe schroot is doing something odd? | 16:02 |
| ahasenack | cjwatson: did you run it as your user, root, or something else? | 16:03 |
| cjwatson | ubuntu@ | 16:04 |
| ddstreet | ahasenack the LWN article linked from the commit is especially interesting and relevant to what you're looking at ;-) | 16:04 |
| cjwatson | I didn't set up a chroot though | 16:04 |
| cjwatson | ubuntu@dogfood-lcy02-amd64-004:~$ sudo cat /proc/$(pgrep -f launchpad-buildd)/limits | grep -i locked | 16:04 |
| cjwatson | Max locked memory 65536 65536 bytes | 16:04 |
| cjwatson | Maybe more relevant ... | 16:05 |
| cjwatson | That's launched directly from a systemd service | 16:05 |
| cjwatson | So maybe a systemd default? | 16:05 |
| ddstreet | cjwatson see the commit i referenced above | 16:05 |
| ahasenack | ah, ok, that's the missing piece then | 16:06 |
| ahasenack | launched by systemd, 64k is the default for systemd services | 16:06 |
| ddstreet | no | 16:06 |
| ddstreet | it's the kernel default | 16:06 |
| cjwatson | ah yes, looks plausible | 16:06 |
| ddstreet | sigh | 16:06 |
| ahasenack | my local user has a much higher limit | 16:07 |
| ahasenack | so something raised it, if it's the kernel default | 16:07 |
| cjwatson | SRUing that to focal would filter through to Launchpad with no intervention needed from us | 16:08 |
| cjwatson | (via cloud images) | 16:08 |
| ahasenack | ah, I'm on impish, forgot that | 16:12 |
| ahasenack | but focal, brand new 1Gb VM, got 65536 kbytes | 16:12 |
| ahasenack | $ ulimit -a|grep locked | 16:12 |
| ahasenack | max locked memory (kbytes, -l) 65536 | 16:12 |
| ahasenack | $ cat /proc/$$/limits|grep locked | 16:12 |
| ahasenack | Max locked memory 67108864 67108864 bytes | 16:12 |
| * ahasenack reached the part in the article where it says "The big new consumer of locked memory, though, is io_uring" | 16:19 | |
| ahasenack | cjwatson: is it unreasonable to add LimitMEMLOCK=524288 to the launchpad-buildd systemd service? Or even half that? DOes it come from a deb? | 18:23 |
| ahasenack | a systemd sru is scary | 18:24 |
| === bdmurray changed the topic of #ubuntu-devel to: Archive: Kinetic open! | Devel of Ubuntu (not support) | Build failures: http://qa.ubuntuwire.com/ftbfs/ | #ubuntu for support and discussion of Bionic-Jammy | If you can't send messages here, authenticate to NickServ first | Patch Pilots: | ||
| ahasenack | bdmurray: what about these packages? https://launchpad.net/ubuntu/kinetic/+queue?queue_state=1&queue_text= | 20:24 |
| ahasenack | nmap, in my case | 20:25 |
| vorlon | ahasenack: part of the opening process will be batch-accepting these from unapproved | 20:44 |
| ahasenack | k | 20:44 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!