=== gschanuel211 is now known as gschanuel21 [15:58] how do I find out why my ram usage is growing and not stop growing, even if I kill stuff it does not make a dent [16:25] because my server when I reboot just keeps using ram and not stopping until it is full, then I have to restart it [16:25] then it starts the entire thing over again [16:26] FESTIVUS-MAXIMUS: https://www.linuxatemyram.com/ [16:27] on my server screen it was giving me killing messages [16:27] because it was out of ram [16:27] I have been watching htop and nothing is using that much ram [16:27] then you have an application that behaves badly. it will show up in a monitoring tool [16:28] it usually also kills the application that uses the most ram first [16:28] so that can be a pointer [16:28] htop did not show it [16:37] FESTIVUS-MAXIMUS: Run top(1), press f, move to RES, press s, press q [16:38] Now top(1) is displaying processes ordered by how much actual RAM they're consuming. Should give you some idea what's going on. [16:38] Right-arrow if you can't see whole process names. [16:38] ram is still growing and nothing is changing [16:38] 397 root 19 -1 84356 28740 27612 S 0.0 0.4 0:00.63 systemd-journal [16:38] I am at 1 GB of ram [16:40] FESTIVUS-MAXIMUS: Alright, something is logging if that's the biggest consumer. [16:40] it is now 5745 mysql 20 0 2061040 390444 36120 S 0.3 4.9 0:00.84 mysqld [16:41] Oh, doesn't seem like you've sorted by resident usage then. [16:41] I did what you said [16:41] f goto RES s q [16:44] kk, top entries should be the biggest RAM consumers [16:44] top one is 5745 mysql 20 0 2062072 403908 36484 S 0.0 5.0 0:01.13 mysqld [16:45] It's possible there is an explosion of smaller things eating RAM too, but top is a good place to start [16:45] I have tried restarting mysql, and it does not stop the ram [16:45] *drop [16:47] Next stop might be understanding what it's doing that's problematic, and you might look at things talking to it to start (sudo netstat -nap maybe) and see if there's some logging to note activity, maybe as compared with another box doing the same thing but not running out of RAM [16:48] ok [16:48] In short, is it being used normally or is there something funny going on? What's it doing? Has it simply exceeded its design expectations in some way? [16:48] If you have a DBA they might be able to help with this, or maybe someone involved with developing whatever the database is doing. [16:48] ok [16:50] thank you for your help mason [16:51] FESTIVUS-MAXIMUS: Such as it was. Sorry I don't have more detailed ideas. I can use databases but I'm no DBA, so there might be easy things to check that I don't know about. But look at who's using it and how it's being used and you should be headed in the right direction. [16:51] ok [17:27] FESTIVUS-MAXIMUS: if you look over the past OOM records in the systemd journal, which are those processes which got killed? [17:28] make a list, see which ones got killed first (after rebooting), and which ones got killed the most [17:28] ok [17:29] also, is this a container, a VM, or a dedicated server? if a container, RAM might be 'stolen' by the host, too [17:30] i.e. they could overcommit and if you have noisy neighbour that might cause you problems [17:32] if you don't kow exactly what this environment is, you can install package virt-what and run the virt-what command and it will try to tell which environment you're operating in. [17:38] actually, ram might also be taken from your system if you're on a VM. you'd see messages in systemd-journald about "balloon" then (the technique is called ballooning) [17:51] decaded server [17:51] and I found the issue [17:51] someone is trying to brute force there way into my smtp server [18:18] FESTIVUS-MAXIMUS: that's entirely normal as a situation, but should not cause high ram allocation, unless there's a misconfiguration [18:19] I am looking in to the config