| === gschanuel211 is now known as gschanuel21 | ||
| FESTIVUS-MAXIMUS | how do I find out why my ram usage is growing and not stop growing, even if I kill stuff it does not make a dent | 15:58 |
|---|---|---|
| FESTIVUS-MAXIMUS | because my server when I reboot just keeps using ram and not stopping until it is full, then I have to restart it | 16:25 |
| FESTIVUS-MAXIMUS | then it starts the entire thing over again | 16:25 |
| ravage | FESTIVUS-MAXIMUS: https://www.linuxatemyram.com/ | 16:26 |
| FESTIVUS-MAXIMUS | on my server screen it was giving me killing messages | 16:27 |
| FESTIVUS-MAXIMUS | because it was out of ram | 16:27 |
| FESTIVUS-MAXIMUS | I have been watching htop and nothing is using that much ram | 16:27 |
| ravage | then you have an application that behaves badly. it will show up in a monitoring tool | 16:27 |
| ravage | it usually also kills the application that uses the most ram first | 16:28 |
| ravage | so that can be a pointer | 16:28 |
| FESTIVUS-MAXIMUS | htop did not show it | 16:28 |
| mason | FESTIVUS-MAXIMUS: Run top(1), press f, move to RES, press s, press q | 16:37 |
| mason | Now top(1) is displaying processes ordered by how much actual RAM they're consuming. Should give you some idea what's going on. | 16:38 |
| mason | Right-arrow if you can't see whole process names. | 16:38 |
| FESTIVUS-MAXIMUS | ram is still growing and nothing is changing | 16:38 |
| FESTIVUS-MAXIMUS | 397 root 19 -1 84356 28740 27612 S 0.0 0.4 0:00.63 systemd-journal | 16:38 |
| FESTIVUS-MAXIMUS | I am at 1 GB of ram | 16:38 |
| mason | FESTIVUS-MAXIMUS: Alright, something is logging if that's the biggest consumer. | 16:40 |
| FESTIVUS-MAXIMUS | it is now 5745 mysql 20 0 2061040 390444 36120 S 0.3 4.9 0:00.84 mysqld | 16:40 |
| mason | Oh, doesn't seem like you've sorted by resident usage then. | 16:41 |
| FESTIVUS-MAXIMUS | I did what you said | 16:41 |
| FESTIVUS-MAXIMUS | f goto RES s q | 16:41 |
| mason | kk, top entries should be the biggest RAM consumers | 16:44 |
| FESTIVUS-MAXIMUS | top one is 5745 mysql 20 0 2062072 403908 36484 S 0.0 5.0 0:01.13 mysqld | 16:44 |
| mason | It's possible there is an explosion of smaller things eating RAM too, but top is a good place to start | 16:45 |
| FESTIVUS-MAXIMUS | I have tried restarting mysql, and it does not stop the ram | 16:45 |
| FESTIVUS-MAXIMUS | *drop | 16:45 |
| mason | Next stop might be understanding what it's doing that's problematic, and you might look at things talking to it to start (sudo netstat -nap maybe) and see if there's some logging to note activity, maybe as compared with another box doing the same thing but not running out of RAM | 16:47 |
| FESTIVUS-MAXIMUS | ok | 16:48 |
| mason | In short, is it being used normally or is there something funny going on? What's it doing? Has it simply exceeded its design expectations in some way? | 16:48 |
| mason | If you have a DBA they might be able to help with this, or maybe someone involved with developing whatever the database is doing. | 16:48 |
| FESTIVUS-MAXIMUS | ok | 16:48 |
| FESTIVUS-MAXIMUS | thank you for your help mason | 16:50 |
| mason | FESTIVUS-MAXIMUS: Such as it was. Sorry I don't have more detailed ideas. I can use databases but I'm no DBA, so there might be easy things to check that I don't know about. But look at who's using it and how it's being used and you should be headed in the right direction. | 16:51 |
| FESTIVUS-MAXIMUS | ok | 16:51 |
| tomreyn | FESTIVUS-MAXIMUS: if you look over the past OOM records in the systemd journal, which are those processes which got killed? | 17:27 |
| tomreyn | make a list, see which ones got killed first (after rebooting), and which ones got killed the most | 17:28 |
| FESTIVUS-MAXIMUS | ok | 17:28 |
| tomreyn | also, is this a container, a VM, or a dedicated server? if a container, RAM might be 'stolen' by the host, too | 17:29 |
| tomreyn | i.e. they could overcommit and if you have noisy neighbour that might cause you problems | 17:30 |
| tomreyn | if you don't kow exactly what this environment is, you can install package virt-what and run the virt-what command and it will try to tell which environment you're operating in. | 17:32 |
| tomreyn | actually, ram might also be taken from your system if you're on a VM. you'd see messages in systemd-journald about "balloon" then (the technique is called ballooning) | 17:38 |
| FESTIVUS-MAXIMUS | decaded server | 17:51 |
| FESTIVUS-MAXIMUS | and I found the issue | 17:51 |
| FESTIVUS-MAXIMUS | someone is trying to brute force there way into my smtp server | 17:51 |
| tomreyn | FESTIVUS-MAXIMUS: that's entirely normal as a situation, but should not cause high ram allocation, unless there's a misconfiguration | 18:18 |
| FESTIVUS-MAXIMUS | I am looking in to the config | 18:19 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!