| fenugrec | Hi, I'm looking for suggestions to figure out "where'd all my RAM go" . 'free -h' shows 47M available, 65M free (out of 2G). 'ps -aux' only adds up to about <10% of mem; /tmp is empty | 02:23 |
|---|---|---|
| fenugrec | ooh, 'df' shows a few /dev/loopN devices with 1-200MB used... they're not tmpfs but squashfs, does that add up into RAM total ? | 02:25 |
| sarnold | fenugrec: what's the free -h report for buff/cached ? that's a common place for memory to go | 02:25 |
| sarnold | fenugrec: the smem tool in the smem package is handy for figuring out how much memory to blame each process for consuming | 02:26 |
| fenugrec | sarnold, that was my first idea, but not in this case " Mem: 1.9Gi 1.8Gi 63Mi 0.0Ki 99Mi 40Mi " (99M for buff/cache) | 02:26 |
| fenugrec | noted for smem. Can't do anything right now, apt-get is running and system is slowed to a crawl | 02:27 |
| sarnold | yeah that looks like an unhappy system :/ | 02:27 |
| fenugrec | can't tell if OOM is doing anything because UFW is polluting dmesg... vmstat shows a ton of block io, no idea how much of it is due to apt-get | 02:28 |
| sarnold | fenugrec: it might be useful to grab a copy of /proc/meminfo and /proc/slabinfo to get a snapshot of how kernel memory is being used | 02:29 |
| dasm | fenugrec: try top or htop (if it's installed). it should show where the memory is | 02:32 |
| fenugrec | sarnold, dumped both https://pastebin.com/e681HhkX . Not used to looking at those raw, I'm just a peasant P ) | 02:33 |
| fenugrec | dasm, same issue, RSS and %MEM numbers in top don't add up (htop not installed) | 02:33 |
| sarnold | fenugrec: same, I usually just use the slabtop program, but if you're already low on memory, throwing that onto the fire too wouldn't help | 02:33 |
| dasm | fenugrec: what shows MiB Mem row? | 02:35 |
| dasm | it should add up to first column total | 02:35 |
| fenugrec | dasm, MiB Mem : 1978.2 total, 79.5 free, 1817.6 used, 81.2 buff/cache . Right, 1978 = 79+1817+81, but I'm taking issue with 1817 | 02:36 |
| dasm | if your system allows, try installing htop. it's very neat tool, i prefer it over top. it allows to sort by the biggest offender consuming the most memory | 02:36 |
| sarnold | fenugrec: check that kmalloc-128 line, it's vastly out of proportion to all the other kmalloc-* slabs | 02:37 |
| fenugrec | dasm, I agree 100% and use htop usually. But serverbox is currently running apt-get so I'm blocked for X more minutes/hours | 02:37 |
| sarnold | fenugrec: this feels a lot like a kernel memory leak | 02:37 |
| fenugrec | sarnold, good observation, slabtop is also showing kmalloc-128 way at the top | 02:37 |
| sarnold | if i've done the math correctly, that's .. | 02:38 |
| dasm | or in top you should be able to shift+m to see what's going on | 02:38 |
| sarnold | 1565.97070312500000000000 | 02:38 |
| sarnold | megabytes | 02:38 |
| fenugrec | (shift+m = sort by %mem ? nice, I can never remember). Right, localedef is at the top with 4% MEM. | 02:40 |
| sarnold | fenugrec: unfortunately it'll be pretty difficult to figure out what's consuming your kmalloc-128 slab from here :( -- rebooting is going to be the best way to get that memory back | 02:40 |
| dasm | sarnold: in the middle of apt update it's rather dangerous | 02:41 |
| sarnold | yes, good point | 02:41 |
| fenugrec | sarnold, yeah makes sense. I could spend hours killing stuff and not find the culprit... | 02:41 |
| dasm | treat it like windows, lol | 02:41 |
| fenugrec | uh oh my ssh session running apt just timed out | 02:42 |
| sarnold | fenugrec: it's a bit interesting to me that you've got enough ufw / iptables logging events to spot ooms, I wonder if that code path is affected | 02:42 |
| sarnold | :( :( :( | 02:42 |
| dasm | oh. | 02:42 |
| fenugrec | no no I'm *not seeing OOMs , all I see is UFW noise | 02:43 |
| sarnold | oof, don't mind me, just typos.. | 02:43 |
| fenugrec | apt-get is still running ? so it doesnt have a pty ... not sure what's going to happen. I guess future me will deal with it tomorrow ! | 02:44 |
| fenugrec | dasm, sarnold, thanks for the assistance | 02:44 |
| dasm | fenugrec: good call ;) | 02:44 |
| sarnold | fenugrec: one sec.. | 02:44 |
| sarnold | https://github.com/iovisor/bcc/blob/master/tools/slabratetop_example.txt | 02:44 |
| sarnold | this tool *might* be able to help you spot kernel features that cause those 128 byte allocations | 02:44 |
| dasm | fenugrec: i'm logging off. good luck with looking for the culprit! | 02:45 |
| sarnold | if you enable the firewall logging, and see that grow; disable firewall logging, and see those allocations stop, then you might have a really good guess | 02:45 |
| === de-facto_ is now known as de-facto | ||
| sarnold | aww crud. our bpfcc-tools package version of slabratetop-bpfcc fails for me :( | 02:46 |
| fenugrec | hmm let's see | 02:46 |
| fenugrec | oh, I need to install / compile it ? | 02:46 |
| sarnold | yeah, this one is crazy, it even does compile *at run time* with kernel headers or something similar | 02:47 |
| fenugrec | heh guess how well that's going to work on this cadaver P ) | 02:47 |
| sarnold | not well | 02:47 |
| sarnold | this is for post-reboot :) | 02:47 |
| fenugrec | aah yes | 02:47 |
| sarnold | I mean, I hope your leak is fixed in a newer kernel | 02:47 |
| sarnold | but in case it's still there :( this might be useful | 02:48 |
| fenugrec | huh it's on 5.4.0, sounds kinda old. Def will look into it. Not me though, future me | 02:48 |
| fenugrec | cheers, ttyl if I'm still hosed tomorrow | 02:49 |
| sarnold | gnight, good luck fenugrec :) | 02:50 |
| samba35 | <pre>make[1]: *** No rule to make target 'arch/x86/entry/syscalls/syscall_32.tbl', needed by 'arch/x86/include/generated/asm/syscalls_32.h'. Stop. | 09:29 |
| samba35 | arch/x86/Makefile:264: recipe for target 'archheaders' failed | 09:29 |
| samba35 | make: *** [archheaders] Error 2 | 09:29 |
| samba35 | </pre> | 09:29 |
| samba35 | i am getting this error when i try to make kernel compile with make command | 09:29 |
| samba35 | i am in /usr/src/linux-headers-4.15.0-135-generic | 09:29 |
| MICROburst | What are the kernel options for booting 20.04.2 live-server via PXE? Getting complains from /init about not finding /dev/sr0. | 11:18 |
| MICROburst | ping | 15:06 |
| teward | patience is a virtue MICROburst, and nobody'll answer you unless they have an idea of a solution. | 15:10 |
| MICROburst | teward: no traffic at all. | 17:46 |
| MICROburst | Getting the error "/init: line 49: can’t open /dev/sr0: No medium found" like here: https://community.theforeman.org/t/installing-ubuntu-via-pxe/20109 | 17:47 |
| TJ- | MICROburst: sr0 == Scsi Removable device (CD-ROM) - so presumably the installer is looking for the mounted image where it thought it had booted from (thinking it is booting locally ) | 17:49 |
| === ijohnson is now known as ijohnson|lunch | ||
| === vlm_ is now known as vlm | ||
| xibalba | when i configure an ipv6 address with /128 on my loopback interface via netplan it keeps coming up as a /64. should that be expected? | 19:48 |
| MICROburst | TJ-: I know that sr0 is the CDROM, vmlinuz and initrd are taken from an .iso-file. Do I need *.seed or how can I fix it? | 20:09 |
| rbasak | MICROburst: have you tried https://ubuntu.com/server/docs/install/netboot-amd64? | 20:18 |
| genii | There's also http://archive.ubuntu.com/ubuntu/dists/focal-updates/main/installer-amd64/current/legacy-images/netboot/ubuntu-installer/amd64/ | 20:20 |
| === ijohnson|lunch is now known as ijohnson | ||
| === blueeagle is now known as BlueEagle | ||
| gotcha | hey guys, im trying to add a static ip to a second card in 20.04, its different than it was in 18.04, this is what i have it set to https://i.imgur.com/Tk1a6bQ.png | 23:02 |
| gotcha | any idea why it doesnt work? | 23:02 |
| Ussat | initial thoughts are your indents are wrong | 23:03 |
| Ussat | YALM is VERY picky | 23:03 |
| Ussat | TAML | 23:03 |
| Ussat | YAML | 23:03 |
| Ussat | .... | 23:03 |
| Ussat | ya that | 23:04 |
| gotcha | this is what it gets me when i run netplan apply https://i.imgur.com/fX7rwBZ.png | 23:04 |
| gotcha | im using screenshots cuz its a VM and i cant copy/paste to pastebin | 23:04 |
| sarnold | gotcha: wild guess territory, move that version: 2 up above the ethernets: line | 23:05 |
| Ussat | Well, there ya go, indentation | 23:05 |
| Ussat | and what sarnold said | 23:05 |
| Ussat | I have gotten REAL familiar with YAML because of ansible | 23:05 |
| gotcha | ok.. the first part was set to static while i was installing the OS so i didnt touch that | 23:06 |
| sarnold | heh I'm still in the 'flail around wildly trying to make it work' stage of yaml | 23:06 |
| gotcha | i added the eth1: and below | 23:06 |
| Ussat | That stage never ends sarnold | 23:06 |
| * sarnold sobs | 23:07 | |
| Ussat | either does the sobbing stage | 23:07 |
| sarnold | lol | 23:07 |
| gotcha | ok, 'netplan apply' worked and the new static ip is showing | 23:09 |
| gotcha | but i still cant get online.. | 23:09 |
| gotcha | just so you guys know what ive done is this ubuntu has 2 virtual cards | 23:09 |
| gotcha | one is local and has no internet access and the second does | 23:09 |
| Ussat | OK.....here is a paste example of what I am useing: https://paste.centos.org/view/634bd00c | 23:12 |
| Ussat | Note how I define nameservers | 23:12 |
| Ussat | and yes, thats how you do bonding in Ubuntu :) | 23:14 |
| sarnold | gotcha: it probably doesn't make sense to put two gateway4: in the thing | 23:15 |
| sarnold | man does 'find in page' work for anyone on https://netplan.io/reference/ ? | 23:16 |
| Ussat | Well, they are seperate nic's so yes, you need to define it for both | 23:16 |
| gotcha | true.. i missed that | 23:18 |
| gotcha | here is a dumb question, why did they change to netplan? | 23:18 |
| sarnold | systemd-networkd files aren't super-fun to write | 23:19 |
| Ussat | ^^ | 23:19 |
| Ussat | Netplan is a really nice front end | 23:19 |
| Ussat | 'its just, YAML sucks | 23:19 |
| sarnold | there's a bit in that page that says "There should only be a single gateway | 23:19 |
| sarnold | set in your global config, to make it unambiguous. If you need multiple | 23:19 |
| sarnold | default routes, please define them via routing-policy." | 23:19 |
| sarnold | -- it's located about a quarter of the way through the page | 23:19 |
| Ussat | eh.... | 23:19 |
| Ussat | I find that page...lacking | 23:19 |
| sarnold | amen | 23:19 |
| Ussat | Here is a good example: | 23:21 |
| Ussat | https://paste.centos.org/view/754ed4e6 | 23:21 |
| Ussat | Oick and choose from that | 23:21 |
| Ussat | also: https://netplan.io/examples/ | 23:23 |
| sarnold | that's a great example Ussat | 23:25 |
| Ussat | also https://codebeautify.org/yaml-validator/cbd763fb | 23:25 |
| === Napsterbater is now known as Guest81744 | ||
| === Napsterbater_ is now known as Napsterbater | ||
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!
