[02:23] <fenugrec> Hi, I'm looking for suggestions to figure out "where'd all my RAM go" . 'free -h' shows 47M available, 65M free (out of 2G).  'ps -aux' only adds up to about <10% of mem;  /tmp is empty
[02:25] <fenugrec> ooh, 'df' shows a few /dev/loopN devices with 1-200MB used... they're not tmpfs but squashfs, does that add up into RAM total ?
[02:25] <sarnold> fenugrec: what's the free -h report for buff/cached ? that's  a common place for memory to go
[02:26] <sarnold> fenugrec: the smem tool in the smem package is handy for figuring out how much memory to blame each process for consuming
[02:26] <fenugrec> sarnold, that was my first idea, but not in this case " Mem:  1.9Gi   1.8Gi   63Mi  0.0Ki    99Mi    40Mi " (99M for buff/cache)
[02:27] <fenugrec> noted for smem. Can't do anything right now, apt-get is running and system is slowed to a crawl
[02:27] <sarnold> yeah that looks like an unhappy system :/
[02:28] <fenugrec> can't tell if OOM is doing anything because UFW is polluting dmesg... vmstat shows a ton of block io, no idea how much of it is due to apt-get
[02:29] <sarnold> fenugrec: it might be useful to grab a copy of /proc/meminfo and /proc/slabinfo to get a snapshot of how kernel memory is being used
[02:32] <dasm> fenugrec: try top or htop (if it's installed). it should show where the memory is
[02:33] <fenugrec> sarnold, dumped both https://pastebin.com/e681HhkX . Not used to looking at those raw, I'm just a peasant P )
[02:33] <fenugrec> dasm, same issue, RSS and %MEM numbers in top don't add up (htop not installed)
[02:33] <sarnold> fenugrec: same, I usually just use the slabtop program, but if you're already low on memory, throwing that onto the fire too wouldn't help
[02:35] <dasm> fenugrec: what shows MiB Mem row?
[02:35] <dasm> it should add up to first column total
[02:36] <fenugrec> dasm, MiB Mem :   1978.2 total,     79.5 free,   1817.6 used,     81.2 buff/cache . Right, 1978 = 79+1817+81, but I'm taking issue with 1817
[02:36] <dasm> if your system allows, try installing htop. it's very neat tool, i prefer it over top. it allows to sort by the biggest offender consuming the most memory
[02:37] <sarnold> fenugrec: check that kmalloc-128 line, it's vastly out of proportion to all the other kmalloc-* slabs
[02:37] <fenugrec> dasm, I agree 100% and use htop usually. But serverbox is currently running apt-get so I'm blocked for X more minutes/hours
[02:37] <sarnold> fenugrec: this feels a lot like a kernel memory leak
[02:37] <fenugrec> sarnold, good observation, slabtop is also showing kmalloc-128 way at the top
[02:38] <sarnold> if i've done the math correctly, that's ..
[02:38] <dasm> or in top you should be able to shift+m to see what's going on
[02:38] <sarnold> 1565.97070312500000000000
[02:38] <sarnold> megabytes
[02:40] <fenugrec> (shift+m = sort by %mem ? nice, I can never remember). Right, localedef is at the top with 4% MEM.
[02:40] <sarnold> fenugrec: unfortunately it'll be pretty difficult to figure out what's consuming your kmalloc-128 slab from here :( -- rebooting is going to be the best way to get that memory back
[02:41] <dasm> sarnold: in the middle of apt update it's rather dangerous
[02:41] <sarnold> yes, good point
[02:41] <fenugrec> sarnold, yeah makes sense. I could spend hours killing stuff and not find the culprit...
[02:41] <dasm> treat it like windows, lol
[02:42] <fenugrec> uh oh my ssh session running apt just timed out
[02:42] <sarnold> fenugrec: it's a bit interesting to me that you've got enough ufw / iptables logging events to spot ooms, I wonder if that code path is affected
[02:42] <sarnold> :( :( :(
[02:42] <dasm> oh.
[02:43] <fenugrec> no no I'm *not seeing OOMs , all I see is UFW noise
[02:43] <sarnold> oof, don't mind me, just typos..
[02:44] <fenugrec> apt-get is still running ? so it doesnt have a pty ... not sure what's going to happen. I guess future me will deal with it tomorrow !
[02:44] <fenugrec> dasm, sarnold, thanks for the assistance
[02:44] <dasm> fenugrec: good call ;)
[02:44] <sarnold> fenugrec: one sec..
[02:44] <sarnold> https://github.com/iovisor/bcc/blob/master/tools/slabratetop_example.txt
[02:44] <sarnold> this tool *might* be able to help you spot kernel features that cause those 128 byte allocations
[02:45] <dasm> fenugrec: i'm logging off. good luck with looking for the culprit!
[02:45] <sarnold> if you enable the firewall logging, and see that grow; disable firewall logging, and see those allocations stop, then you might have a really good guess
[02:46] <sarnold> aww crud. our bpfcc-tools package version of slabratetop-bpfcc fails for me :(
[02:46] <fenugrec> hmm let's see
[02:46] <fenugrec> oh, I need to install / compile it ?
[02:47] <sarnold> yeah, this one is crazy, it even does compile *at run time* with kernel headers or something similar
[02:47] <fenugrec> heh guess how well that's going to work on this cadaver P )
[02:47] <sarnold> not well
[02:47] <sarnold> this is for post-reboot :)
[02:47] <fenugrec> aah yes
[02:47] <sarnold> I mean, I hope your leak is fixed in a newer kernel
[02:48] <sarnold> but in case it's still there :( this might be useful
[02:48] <fenugrec> huh it's on 5.4.0, sounds kinda old. Def will look into it. Not me though, future me
[02:49] <fenugrec> cheers, ttyl if I'm still hosed tomorrow
[02:50] <sarnold> gnight, good luck fenugrec :)
make[1]: *** No rule to make target &apos;arch/x86/entry/syscalls/syscall_32.tbl&apos;, needed by &apos;arch/x86/include/generated/asm/syscalls_32.h&apos;.  Stop.
[09:29] <samba35> arch/x86/Makefile:264: recipe for target &apos;archheaders&apos; failed
[09:29] <samba35> make: *** [archheaders] Error 2

[09:29] <samba35> i am getting this error when i try to make kernel compile with make command
[09:29] <samba35>  i am in /usr/src/linux-headers-4.15.0-135-generic
[11:18] <MICROburst> What are the kernel options for booting 20.04.2 live-server via PXE? Getting complains from /init about not finding /dev/sr0.
[15:06] <MICROburst> ping
[15:10] <teward> patience is a virtue MICROburst, and nobody'll answer you unless they have an idea of a solution.
[17:46] <MICROburst> teward:  no traffic at all.
[17:47] <MICROburst> Getting the error "/init: line 49: can’t open /dev/sr0: No medium found" like here: https://community.theforeman.org/t/installing-ubuntu-via-pxe/20109
[17:49] <TJ-> MICROburst: sr0 == Scsi Removable device (CD-ROM) - so presumably the installer is looking for the mounted image where it thought it had booted from (thinking it is booting locally )
[19:48] <xibalba> when i configure an ipv6 address with /128 on my loopback interface via netplan it keeps coming up as a /64. should that be expected?
[20:09] <MICROburst> TJ-: I know that sr0 is the CDROM, vmlinuz and initrd are taken from an .iso-file. Do I need *.seed or how can I fix it?
[20:18] <rbasak> MICROburst: have you tried https://ubuntu.com/server/docs/install/netboot-amd64?
[20:20] <genii> There's also http://archive.ubuntu.com/ubuntu/dists/focal-updates/main/installer-amd64/current/legacy-images/netboot/ubuntu-installer/amd64/
[23:02] <gotcha> hey guys, im trying to add a static ip to a second card in 20.04, its different than it was in 18.04, this is what i have it set to https://i.imgur.com/Tk1a6bQ.png
[23:02] <gotcha> any idea why it doesnt work?
[23:03] <Ussat> initial thoughts are your indents are wrong
[23:03] <Ussat> YALM is VERY picky
[23:03] <Ussat> TAML
[23:03] <Ussat> YAML
[23:03] <Ussat> ....
[23:04] <Ussat> ya that
[23:04] <gotcha> this is what it gets me when i run netplan apply https://i.imgur.com/fX7rwBZ.png
[23:04] <gotcha> im using screenshots cuz its a VM and i cant copy/paste to pastebin
[23:05] <sarnold> gotcha: wild guess territory, move that version: 2 up above the ethernets: line
[23:05] <Ussat> Well, there ya go, indentation
[23:05] <Ussat> and what sarnold said
[23:05] <Ussat> I have gotten REAL familiar with YAML because of ansible
[23:06] <gotcha> ok.. the first part was set to static while i was installing the OS so i didnt touch that
[23:06] <sarnold> heh I'm still in the 'flail around wildly trying to make it work' stage of yaml
[23:06] <gotcha> i added the eth1: and below
[23:06] <Ussat> That stage never ends sarnold
[23:07]  * sarnold sobs
[23:07] <Ussat> either does the sobbing stage
[23:07] <sarnold> lol
[23:09] <gotcha> ok, 'netplan apply' worked and the new static ip is showing
[23:09] <gotcha> but i still cant get online..
[23:09] <gotcha> just so you guys know what ive done is this ubuntu has 2 virtual cards
[23:09] <gotcha> one is local and has no internet access and the second does
[23:12] <Ussat> OK.....here is a paste example of what I am useing: https://paste.centos.org/view/634bd00c
[23:12] <Ussat> Note how I define nameservers
[23:14] <Ussat> and yes, thats how you do bonding in Ubuntu :)
[23:15] <sarnold> gotcha: it probably doesn't make sense to put two gateway4: in the thing
[23:16] <sarnold> man does 'find in page' work for anyone on https://netplan.io/reference/ ?
[23:16] <Ussat> Well, they are seperate nic's so yes, you need to define it for both
[23:18] <gotcha> true.. i missed that
[23:18] <gotcha> here is a dumb question, why did they change to netplan?
[23:19] <sarnold> systemd-networkd files aren't super-fun to write
[23:19] <Ussat> ^^
[23:19] <Ussat> Netplan is a really nice front end
[23:19] <Ussat> 'its just, YAML sucks
[23:19] <sarnold> there's a bit in that page that says "There should only be a single gateway
[23:19] <sarnold> set in your global config, to make it unambiguous. If you need multiple
[23:19] <sarnold> default routes, please define them via routing-policy."
[23:19] <sarnold> -- it's located about a quarter of the way through the page
[23:19] <Ussat> eh....
[23:19] <Ussat> I find that page...lacking
[23:19] <sarnold> amen
[23:21] <Ussat> Here is a good example:
[23:21] <Ussat> https://paste.centos.org/view/754ed4e6
[23:21] <Ussat> Oick and choose from that
[23:23] <Ussat> also: https://netplan.io/examples/
[23:25] <sarnold> that's a great example Ussat
[23:25] <Ussat> also https://codebeautify.org/yaml-validator/cbd763fb