[15:58] <FESTIVUS-MAXIMUS> how do I find out why my ram usage is growing and not stop growing, even if I kill stuff it does not make a dent
[16:25] <FESTIVUS-MAXIMUS> because my server when I reboot just keeps using ram and not stopping until it is full, then I have to restart it 
[16:25] <FESTIVUS-MAXIMUS> then it starts the entire thing over again
[16:26] <ravage> FESTIVUS-MAXIMUS: https://www.linuxatemyram.com/
[16:27] <FESTIVUS-MAXIMUS> on my server screen it was giving me killing messages
[16:27] <FESTIVUS-MAXIMUS> because it was out of ram 
[16:27] <FESTIVUS-MAXIMUS> I have been watching htop and nothing is using that much ram
[16:27] <ravage> then you have an application that behaves badly. it will show up in a monitoring tool
[16:28] <ravage> it usually also kills the application that uses the most ram first
[16:28] <ravage> so that can be a pointer
[16:28] <FESTIVUS-MAXIMUS> htop did not show it
[16:37] <mason> FESTIVUS-MAXIMUS: Run top(1), press f, move to RES, press s, press q
[16:38] <mason> Now top(1) is displaying processes ordered by how much actual RAM they're consuming. Should give you some idea what's going on.
[16:38] <mason> Right-arrow if you can't see whole process names.
[16:38] <FESTIVUS-MAXIMUS> ram is still growing and nothing is changing
[16:38] <FESTIVUS-MAXIMUS>  397 root      19  -1   84356  28740  27612 S   0.0   0.4   0:00.63 systemd-journal  
[16:38] <FESTIVUS-MAXIMUS> I am at 1 GB of ram
[16:40] <mason> FESTIVUS-MAXIMUS: Alright, something is logging if that's the biggest consumer.
[16:40] <FESTIVUS-MAXIMUS> it is now    5745 mysql     20   0 2061040 390444  36120 S   0.3   4.9   0:00.84 mysqld
[16:41] <mason> Oh, doesn't seem like you've sorted by resident usage then.
[16:41] <FESTIVUS-MAXIMUS> I did what you said
[16:41] <FESTIVUS-MAXIMUS> f goto RES s q
[16:44] <mason> kk, top entries should be the biggest RAM consumers
[16:44] <FESTIVUS-MAXIMUS> top one is  5745 mysql     20   0 2062072 403908  36484 S   0.0   5.0   0:01.13 mysqld  
[16:45] <mason> It's possible there is an explosion of smaller things eating RAM too, but top is a good place to start
[16:45] <FESTIVUS-MAXIMUS> I have tried restarting mysql, and it does not stop the ram
[16:45] <FESTIVUS-MAXIMUS> *drop
[16:47] <mason> Next stop might be understanding what it's doing that's problematic, and you might look at things talking to it to start (sudo netstat -nap maybe) and see if there's some logging to note activity, maybe as compared with another box doing the same thing but not running out of RAM
[16:48] <FESTIVUS-MAXIMUS> ok
[16:48] <mason> In short, is it being used normally or is there something funny going on? What's it doing? Has it simply exceeded its design expectations in some way?
[16:48] <mason> If you have a DBA they might be able to help with this, or maybe someone involved with developing whatever the database is doing.
[16:48] <FESTIVUS-MAXIMUS> ok
[16:50] <FESTIVUS-MAXIMUS> thank you for your help mason
[16:51] <mason> FESTIVUS-MAXIMUS: Such as it was. Sorry I don't have more detailed ideas. I can use databases but I'm no DBA, so there might be easy things to check that I don't know about. But look at who's using it and how it's being used and you should be headed in the right direction.
[16:51] <FESTIVUS-MAXIMUS> ok
[17:27] <tomreyn> FESTIVUS-MAXIMUS: if you look over the past OOM records in the systemd journal, which are those processes which got killed?
[17:28] <tomreyn> make a list, see which ones got killed first (after rebooting), and which ones got killed the most
[17:28] <FESTIVUS-MAXIMUS> ok
[17:29] <tomreyn> also, is this a container, a VM, or a dedicated server? if a container, RAM might be 'stolen' by the host, too
[17:30] <tomreyn> i.e. they could overcommit and if you have noisy neighbour that might cause you problems
[17:32] <tomreyn> if you don't kow exactly what this environment is, you can install package    virt-what    and run the    virt-what   command and it will try to tell which environment you're operating in.
[17:38] <tomreyn> actually, ram might also be taken from your system if you're on a VM. you'd see messages in systemd-journald about "balloon" then (the technique is called ballooning)
[17:51] <FESTIVUS-MAXIMUS> decaded server
[17:51] <FESTIVUS-MAXIMUS> and I found the issue
[17:51] <FESTIVUS-MAXIMUS> someone is trying to brute force there way into my smtp server
[18:18] <tomreyn> FESTIVUS-MAXIMUS: that's entirely normal as a situation, but should not cause high ram allocation, unless there's a misconfiguration
[18:19] <FESTIVUS-MAXIMUS> I am looking in to the config