[02:23] Hi, I'm looking for suggestions to figure out "where'd all my RAM go" . 'free -h' shows 47M available, 65M free (out of 2G). 'ps -aux' only adds up to about <10% of mem; /tmp is empty [02:25] ooh, 'df' shows a few /dev/loopN devices with 1-200MB used... they're not tmpfs but squashfs, does that add up into RAM total ? [02:25] fenugrec: what's the free -h report for buff/cached ? that's a common place for memory to go [02:26] fenugrec: the smem tool in the smem package is handy for figuring out how much memory to blame each process for consuming [02:26] sarnold, that was my first idea, but not in this case " Mem: 1.9Gi 1.8Gi 63Mi 0.0Ki 99Mi 40Mi " (99M for buff/cache) [02:27] noted for smem. Can't do anything right now, apt-get is running and system is slowed to a crawl [02:27] yeah that looks like an unhappy system :/ [02:28] can't tell if OOM is doing anything because UFW is polluting dmesg... vmstat shows a ton of block io, no idea how much of it is due to apt-get [02:29] fenugrec: it might be useful to grab a copy of /proc/meminfo and /proc/slabinfo to get a snapshot of how kernel memory is being used [02:32] fenugrec: try top or htop (if it's installed). it should show where the memory is [02:33] sarnold, dumped both https://pastebin.com/e681HhkX . Not used to looking at those raw, I'm just a peasant P ) [02:33] dasm, same issue, RSS and %MEM numbers in top don't add up (htop not installed) [02:33] fenugrec: same, I usually just use the slabtop program, but if you're already low on memory, throwing that onto the fire too wouldn't help [02:35] fenugrec: what shows MiB Mem row? [02:35] it should add up to first column total [02:36] dasm, MiB Mem : 1978.2 total, 79.5 free, 1817.6 used, 81.2 buff/cache . Right, 1978 = 79+1817+81, but I'm taking issue with 1817 [02:36] if your system allows, try installing htop. it's very neat tool, i prefer it over top. it allows to sort by the biggest offender consuming the most memory [02:37] fenugrec: check that kmalloc-128 line, it's vastly out of proportion to all the other kmalloc-* slabs [02:37] dasm, I agree 100% and use htop usually. But serverbox is currently running apt-get so I'm blocked for X more minutes/hours [02:37] fenugrec: this feels a lot like a kernel memory leak [02:37] sarnold, good observation, slabtop is also showing kmalloc-128 way at the top [02:38] if i've done the math correctly, that's .. [02:38] or in top you should be able to shift+m to see what's going on [02:38] 1565.97070312500000000000 [02:38] megabytes [02:40] (shift+m = sort by %mem ? nice, I can never remember). Right, localedef is at the top with 4% MEM. [02:40] fenugrec: unfortunately it'll be pretty difficult to figure out what's consuming your kmalloc-128 slab from here :( -- rebooting is going to be the best way to get that memory back [02:41] sarnold: in the middle of apt update it's rather dangerous [02:41] yes, good point [02:41] sarnold, yeah makes sense. I could spend hours killing stuff and not find the culprit... [02:41] treat it like windows, lol [02:42] uh oh my ssh session running apt just timed out [02:42] fenugrec: it's a bit interesting to me that you've got enough ufw / iptables logging events to spot ooms, I wonder if that code path is affected [02:42] :( :( :( [02:42] oh. [02:43] no no I'm *not seeing OOMs , all I see is UFW noise [02:43] oof, don't mind me, just typos.. [02:44] apt-get is still running ? so it doesnt have a pty ... not sure what's going to happen. I guess future me will deal with it tomorrow ! [02:44] dasm, sarnold, thanks for the assistance [02:44] fenugrec: good call ;) [02:44] fenugrec: one sec.. [02:44] https://github.com/iovisor/bcc/blob/master/tools/slabratetop_example.txt [02:44] this tool *might* be able to help you spot kernel features that cause those 128 byte allocations [02:45] fenugrec: i'm logging off. good luck with looking for the culprit! [02:45] if you enable the firewall logging, and see that grow; disable firewall logging, and see those allocations stop, then you might have a really good guess === de-facto_ is now known as de-facto [02:46] aww crud. our bpfcc-tools package version of slabratetop-bpfcc fails for me :( [02:46] hmm let's see [02:46] oh, I need to install / compile it ? [02:47] yeah, this one is crazy, it even does compile *at run time* with kernel headers or something similar [02:47] heh guess how well that's going to work on this cadaver P ) [02:47] not well [02:47] this is for post-reboot :) [02:47] aah yes [02:47] I mean, I hope your leak is fixed in a newer kernel [02:48] but in case it's still there :( this might be useful [02:48] huh it's on 5.4.0, sounds kinda old. Def will look into it. Not me though, future me [02:49] cheers, ttyl if I'm still hosed tomorrow [02:50] gnight, good luck fenugrec :) [09:29]

make[1]: *** No rule to make target 'arch/x86/entry/syscalls/syscall_32.tbl', needed by 'arch/x86/include/generated/asm/syscalls_32.h'.  Stop.
[09:29]  arch/x86/Makefile:264: recipe for target 'archheaders' failed
[09:29]  make: *** [archheaders] Error 2
[09:29]

[09:29] i am getting this error when i try to make kernel compile with make command [09:29] i am in /usr/src/linux-headers-4.15.0-135-generic [11:18] What are the kernel options for booting 20.04.2 live-server via PXE? Getting complains from /init about not finding /dev/sr0. [15:06] ping [15:10] patience is a virtue MICROburst, and nobody'll answer you unless they have an idea of a solution. [17:46] teward: no traffic at all. [17:47] Getting the error "/init: line 49: can’t open /dev/sr0: No medium found" like here: https://community.theforeman.org/t/installing-ubuntu-via-pxe/20109 [17:49] MICROburst: sr0 == Scsi Removable device (CD-ROM) - so presumably the installer is looking for the mounted image where it thought it had booted from (thinking it is booting locally ) === ijohnson is now known as ijohnson|lunch === vlm_ is now known as vlm [19:48] when i configure an ipv6 address with /128 on my loopback interface via netplan it keeps coming up as a /64. should that be expected? [20:09] TJ-: I know that sr0 is the CDROM, vmlinuz and initrd are taken from an .iso-file. Do I need *.seed or how can I fix it? [20:18] MICROburst: have you tried https://ubuntu.com/server/docs/install/netboot-amd64? [20:20] There's also http://archive.ubuntu.com/ubuntu/dists/focal-updates/main/installer-amd64/current/legacy-images/netboot/ubuntu-installer/amd64/ === ijohnson|lunch is now known as ijohnson === blueeagle is now known as BlueEagle [23:02] hey guys, im trying to add a static ip to a second card in 20.04, its different than it was in 18.04, this is what i have it set to https://i.imgur.com/Tk1a6bQ.png [23:02] any idea why it doesnt work? [23:03] initial thoughts are your indents are wrong [23:03] YALM is VERY picky [23:03] TAML [23:03] YAML [23:03] .... [23:04] ya that [23:04] this is what it gets me when i run netplan apply https://i.imgur.com/fX7rwBZ.png [23:04] im using screenshots cuz its a VM and i cant copy/paste to pastebin [23:05] gotcha: wild guess territory, move that version: 2 up above the ethernets: line [23:05] Well, there ya go, indentation [23:05] and what sarnold said [23:05] I have gotten REAL familiar with YAML because of ansible [23:06] ok.. the first part was set to static while i was installing the OS so i didnt touch that [23:06] heh I'm still in the 'flail around wildly trying to make it work' stage of yaml [23:06] i added the eth1: and below [23:06] That stage never ends sarnold [23:07] * sarnold sobs [23:07] either does the sobbing stage [23:07] lol [23:09] ok, 'netplan apply' worked and the new static ip is showing [23:09] but i still cant get online.. [23:09] just so you guys know what ive done is this ubuntu has 2 virtual cards [23:09] one is local and has no internet access and the second does [23:12] OK.....here is a paste example of what I am useing: https://paste.centos.org/view/634bd00c [23:12] Note how I define nameservers [23:14] and yes, thats how you do bonding in Ubuntu :) [23:15] gotcha: it probably doesn't make sense to put two gateway4: in the thing [23:16] man does 'find in page' work for anyone on https://netplan.io/reference/ ? [23:16] Well, they are seperate nic's so yes, you need to define it for both [23:18] true.. i missed that [23:18] here is a dumb question, why did they change to netplan? [23:19] systemd-networkd files aren't super-fun to write [23:19] ^^ [23:19] Netplan is a really nice front end [23:19] 'its just, YAML sucks [23:19] there's a bit in that page that says "There should only be a single gateway [23:19] set in your global config, to make it unambiguous. If you need multiple [23:19] default routes, please define them via routing-policy." [23:19] -- it's located about a quarter of the way through the page [23:19] eh.... [23:19] I find that page...lacking [23:19] amen [23:21] Here is a good example: [23:21] https://paste.centos.org/view/754ed4e6 [23:21] Oick and choose from that [23:23] also: https://netplan.io/examples/ [23:25] that's a great example Ussat [23:25] also https://codebeautify.org/yaml-validator/cbd763fb === Napsterbater is now known as Guest81744 === Napsterbater_ is now known as Napsterbater