=== zzlatev is now known as ZZlatev
JamesBenson@gbkersey:  Thanks for the info.  Even a 7-8Gbps is better than a 1Gbps.  We are using the ga-16.04-lowlatency kernel for what it's worth03:09
lordievaderGood morning06:01
MrMojit0I have an fresh Ubuntu machine running and I installed Nagios using the following website: https://kifarunix.com/how-to-install-and-configure-nagios-core-on-ubuntu-18-04/07:29
MrMojit0Installation seems to be successfully because I can access the website. But now comes to problem, when I reboot the machine it won´t work anymore and I need to follow the complete tutorial again to get it up. But then again after a reboot its not working.07:29
MrMojit0My Linux experience is zero. So I was hoping if anyone has an idea what could be wrong on that tutorial or how I can get Nagios to work again.07:30
lordievaderIs the Nagios service enabled?07:31
MrMojit0lordievader: How can I check this on Ubuntu?07:47
lordievader`systemctl status nagios` (assuming here the nagios service is called that way)07:48
MrMojit0Active: active (running)07:49
MrMojit0Just did a check on Apache2 if that is running and that is also good07:53
lordievaderIs it the same after a reboot?07:53
MrMojit0Let me restart to be sure07:54
MrMojit0The IP address is also reachable. Maybe it has something to do with the firewall07:55
MrMojit0Both Nagios and Apache are running successfully after a reboot07:57
MrMojit0Found the issue! Thank you07:58
MrMojit0I need to readd the ¨ufw allow apache;ufw reload¨07:58
MrMojit0How can I make sure those settings will stay stored and not be gone after a reboot?07:58
lordievaderI think ufw takes care of that. Haven't used ufw in ages. Dislike the way it does things.08:01
MrMojit0I am just rebooting the machine again to see if ufw is enabled or not08:02
MrMojit0It is enabled, but still need to add the rules. I will look how to make those rules mandatory08:14
=== msmarcal|eod is now known as msmarcal
gbkerseyJamesBenson: thanks for that....13:23
* foo attempts to figure out what is causing oom to murder processes14:05
* foo reads https://serverfault.com/questions/134669/how-to-diagnose-causes-of-oom-killer-killing-processes14:05
fooI wonder if oom-killer can be too aggressive? What's strange is I rarely see this system swapping14:24
vahnxHi all, I'm looking to setup a ticketing system on 18.04 LTS and looking for recommendations. I was using Spiceworks on Windows but have since moved to Linux and do not have a Windows license.14:39
Ussatticketing system, like help desk thing /14:39
vahnxSomething that I will mainly use, maybe 1 or 2 more users. Yeah for tech support.14:40
cryptodanvahnx: look at osticket14:40
vahnxOk thanks, will do!14:40
cryptodanvahnx: https://osticket.com/14:40
codefriarany Traefik experts here?14:41
admin0hi .. my server / is 100% full, but i am not able to see what is causing it ..    / is 80G ..  du -sh /* grep G does not even come near 80G15:09
admin0is it possible it can be something that is in memory or an open file handler . and if such is there, how do I find it out ?15:10
codefriaradmin0 I once ran out of inodes, and it showed as full15:11
admin0inodes is only 4% used15:11
admin0df -h =>  /dev/mapper/cloud-root   75G   75G     0 100% /   |  df -i => /dev/mapper/cloud-root  5005312 174582  4830730    4% /15:12
leftyfbadmin0: cd / ; sudo du -hs .[^.]*15:14
leftyfbadmin0: that'll run against any hidden directories15:14
leftyfbadmin0: cd / ; sudo du -hs .[^.]* * |grep G # this will run on everything15:15
leftyfbadmin0: once you get some space, I recommend using ncdu15:15
whislockadmin0: If a process is holding a file open, that space will still be consumed for filesystem allocation purposes until the process releases the lock.15:15
admin09.0G .15:15
admin0that is what i get15:15
admin0but df is 100% full15:15
admin0wishlock , how do I locate such process or such file15:16
leftyfbadmin0: can you pastebin exactly what commands you are running and the output please?15:16
admin0sure .. one moement15:17
admin0whislock, thanks for the pointer .. a cron was rm -rf a file while the process was not stopped15:19
admin0rebooting that process ( libvirtd) cleared up the space15:19
admin0instead of rm -rf the file, will do cp /dev/null instead15:20
admin0thanks guys for helping15:35
* admin0 sends pizza (virtual) to leftyfb and whislock :D15:36
leftyfbadmin0: future reference, install ncdu15:36
DammitJimhow can I figure out what blocked processes I have on a server?17:30
DammitJimmy monitoring system is telling me I have on average 5 blocked processes, but I don't see a D in the S column on top17:31
tomreynis uninterruptable sleep what your monitoring system means by "blocked process", though?17:34
DammitJimtomreyn, good question... not sure17:38
DammitJimndicates the number of processes blocked for I/O, paging, etc.17:39
tomreynhmm yes, sounds like it should be that17:42
sarnoldDammitJim: procfs(5) /proc/pid/syscall sounds vaguely enough like a blocked vs not-blocked measure for such a tool17:43
sarnoldit'd be a bit silly to open, read, and close, a few thousand files for this information every N seconds of course, but maybe that's what it's doing17:43
tomreynif its source code is available to you, you could inspect what it actually does.17:44
DammitJimso, how do I get the process that is blocked?17:46
sarnoldwell, the thing with these kinds of measurements, is that it's all very transitory and racy17:47
DammitJimso, hard to "catch?"17:47
sarnoldafter all it takes ~20ms to handle a read IO operation from a spinning metal hard drive, by the time top or similar tool has crawled through all the processes on the system, the information it has on a process is likely already out of date17:48
DammitJimoh yeah, here I"m talking about an all flash array17:48
DammitJimand the blocked process stats from the monitoring system are reported every 5 minutes and I had this "problem" for about an hour17:49
tomreynaccording to google your quote's source is https://docs.eginnovations.com/Unix_and_Windows_Servers/System_Details_Test_1.htm17:49
DammitJimI'm on hold with them asking them what they are actually polling17:50
tomreynno source code there, i assume.17:50
sarnoldyou could perf trace or strace the thing. it'd be drinking from a firehose though17:51
dlloydyou can filter to specific sysclals with strace17:59
fooWhen oom starts killing stuff, per syslog, it's not always clear what that is, correct?18:01
sarnoldhmm? I'm accustomed to seeing it saying which process it killed18:02
tomreynboth pid and process name should be listed18:03
sarnoldof course if it kills X11 and then all your X clients *also* die because the other end of their socket went away, that might feel a lot like the oom killer not reporting what died .. when really, it was just responsible for one process going away18:03
footomreyn / sarnold - thanks, but that's not *always* the process that is consuming the memory right? eg. X can consume a ton of memory, Y will get killed off a result, correct? Or am I misunderstanding?18:04
sarnoldfoo: yeah, there's also some per-process scoring involved; and depending upon how much memory is shared among processes, killing "huge" ones may not actually free up much memory18:05
foosarnold: ok, so whatever gets killed is not always the culprit. eg. I've seen a ton of different things killed off now that I think about it18:06
fooSystem runs nginx, postgres and a few python scripts. Attempting to figure out what is causing this18:06
sarnoldyeah, the kernel tries to balance (a) killing something quickly (b) killing as little as possible (c) while also still getting as much memory for the pain18:07
tomreynthe journal will report which process was killed. processes which depend on this process may also fail as a result, and wont be listed individually as part of the oom kill record..18:07
foosarnold: thanks18:08
foonginx looks ok, checking postgres right now too.18:08
tomreynyou can actually influence the kernels' decision making a little. but, much more reasonably, you don't want the OOM killing to happen in the first place.18:08
fooAlso going to enable query logs for slower queries18:09
foo./postgresqltuner.pl says [URGENT] set vm.overcommit_memory=2 in /etc/sysctl.conf and run sysctl -p to reload it. This will disable memory overcommitment and avoid postgresql killed by OOM killer. - I've been tracking down a memory issue with something, not sure what it is. Are we in agreement this is suggested? I assume it is but thought I'd ask18:09
tomreynfirst identify which of the processes allocated more memory than they should have according to your planning, then try to see how to tune them.18:10
tomreynif you start increasing debugging / verbosity now you already change their resource allocation18:10
footomreyn: "first identify which of the processes allocated more memory" - I can only do this by checking conf files, right? Is there another way?18:12
footomreyn: you have suggested tools? It's so sporadic, I haven't been able to narrow it down. Running a top and sysstat and what not now18:16
tomreynyou run some services on your server. ideally as few as possible, and move others to separate servers (or VMs). you think about how much memory you want each of them (as well as the OS itself) to consume, and calculate the total memory allocation. you configure services to allocate only the amount of memory you want them to allocate (which is not always possible, but it often is more or less possible, especially with DB servers).18:16
tomreynand you do monitoring in short enough intervals to determine what may have consumed more memory than planned. and when this happens you review its logs (maybe increase verbosity), configuration, do the tuning.18:18
footomreyn: yeah, I thought about splitting things about a bit more... namely moving postgres onto it's own system. Right now postgres + nginx + various python scripts all on one server... and thus fine-tuning isn't an exact science since each fluctuates18:18
tomreynright, DB servers should always be run just by themselves IMO.18:18
tomreynpostgresql is actually quite configurable in terms of memory allocation, nginx also, but there i find it not to be so plannable.18:20
sarnoldthe downside to running databases on different servers is that can add milliseconds to latency. that's probably better than minutes of latency if the oom killer has decided your database is a hog :) but still, something to keep in mind18:21
tomreynso can a lot of other factors, yes.18:22
footomreyn / sarnold - yeah, I'm not opposed to that. Would definitely help control resources better18:22
sdezieltomreyn: do you recommend to always separate the DB backend from the web frontend for security? performance? upgradability? all those?18:23
fooI know amazon has RDS. I wonder if Digital Ocean has something.18:29
fooDoes anyone have any commentary on this suggestion: [URGENT] set vm.overcommit_memory=2 in /etc/sysctl.conf and run sysctl -p to reload it. This will disable memory overcommitment and avoid postgresql killed by OOM killer.18:29
sarnoldfoo: in isolation, I don't like the suggestion. if, after doing the analysis tomreyn suggested, you may realize it makes sense or it may not make sense18:30
sarnoldfoo: yes, that should drastically reduce the chances of hitting OOM, but it might also make the machine nearly unusable.18:31
foosarnold: thank you. Part of my challenge is little to nothing meaningful has changed in the past month that I can see. I'm almost wondering if some library had some API change and there's some obscure threading issue due to some change which is causing some resource issue... but meh, OOM killed stuff once in feb, once in march, and 4 times his month (already). Traffic all looks nearly the same18:34
sarnoldfoo: that sounds a lot like the machine just isn't sized correctly for the workload18:35
foosarnold: thank you. it's been online for 3 months. It was a recent migration from ubuntu 14.04 to 18.04. Not much has changed in the past few months but nonetheless, I agree something isn't tuned properly. I don't think gunicorn can be tuned, leaving nginx + postgres, namely. Django also runs on here.18:37
BrianBlazehow do I get an older version of mysql? everytime I try to install a deb it tells me dependency issues and install -f just gives me the latest version18:37
BrianBlazewondering if anyone can poitn me in a direction :)18:38
sarnoldBrianBlaze: can you pastebin the whole thing? (pastebinit package has an easy pastbinit tool that can help this)18:38
* foo sets up pg_stat_statements18:38
sarnoldwhy do you want to install that specific version?18:40
sarnoldwhere did you get it?18:40
BrianBlazebecause this app needs mysql version between 5.5 and 5.2.2418:40
BrianBlazesorry 5.7.2418:40
sarnolddoes 5.7.25 break something? or does their documentation just not know about 5.7.25 yet?18:41
BrianBlazewhen I go through the install it tells me it won't work with the newest version of MYSQL and won't let me go farther18:41
BrianBlazeso yeah the latter sarnold18:41
sarnoldalright then18:41
sarnolddo you have any data in the database that you care about?18:41
BrianBlazenah this is a fresh install18:42
BrianBlazebasically we use orangeHRM at work18:42
BrianBlazeopen surce18:42
BrianBlazeand I am trying to go to the latest version18:42
BrianBlazeI will worry about getting the data there after18:42
sarnoldalright, cool. I think you'd be best served by apt-get purge mysql-server  -- maybe you'll need to purge other mysql packages while you're at it -- and download the 5.7.24 packages from https://launchpad.net/ubuntu/+source/mysql-5.7/5.7.24-0ubuntu0.18.04.118:43
BrianBlazethanks so much I will give it a shot18:45
tomreynsdeziel: not always, not necessarily for a small test / dev / hobby project. but for anything 'serious', yes.18:52
sdezieltomreyn: OK. I myself usually put it on the same machine to remove the network from potential source of failure. I also think that since the web app has the DB password, security-wise it isn't much worse18:53
sdezieltomreyn: but for a bigger deployment, I guess you are right it's best to separate them18:54
tomreynsdeziel: sure, networking is always a possible hazard (still but not neccessarily as much in a more controlled environment than the Internet), and there is latency, as sarnold mentioned. but if you run a webserver on the same system as a database server, it already rules out a serious HA setup. (definitely but not neccessarily only) if there's server side scripting involed on the webserver it also also means you're adding additional19:02
tomreynattack vectors against a local vs remote database server (vectors and attacks which involve the local (e.g. file) system, such as remote file include, privilege escalation, directory traversal).19:02
sdezieltomreyn: right, good point. It's harder to secure when both are on the same machine19:03
sarnoldBrianBlaze: don't forget to dpkg hold the mysql packages to prevent security updates from replacing the specific versions you're installing19:03
sarnoldBrianBlaze: apt-mark(8) can do that19:05
sdezieltomreyn: that said, the only valuable thing on the DB server is usually the DB itself19:05
BrianBlazehow true19:05
tomreynsdeziel: which is the big secret trove, the crown jewels, though, right? surely not always, but in many cases DB leaks are worse than, say, application code leaks (though those can be very bad, too, exposing malpractive, dodgy policies which carried into code)19:07
sdezieltomreyn: agreed but since the web app already has access to the DB...19:07
tomreynsdeziel: database user access, yes, not file system access19:08
tomreynthose are very different19:08
sdezieltomreyn: that's probably what I fail to understan19:08
sdezielmind elaborating a little on the security implications?19:09
tomreynif you can "select into outfile" on a backend DB server but have no means to access the data it stored into a file that is now local to the DB server, such as thorugh a remote file include attacks against PHP, then this attack vector doesn't help you at all.19:09
sdezieland such case, the source of the select would have to be something else than the DB itself, is that even possible?19:11
BrianBlazeI really appreciate the input sarnold I am on my way :)19:11
sdeziel(I know very little about DBs... just enough to drop a table/DB ;) )19:11
sarnoldBrianBlaze: great! :) have fun19:11
sarnoldlittle sdeziel tables :)19:12
tomreynsdeziel: so imagine this scenario: there is a php application running on the weserver which is both vulnerable to remote file includes and SQL injection, and you have a mysql server as the backend. and the SQL injection is limited in that the application prevents it almost except that you can still run INTO OUTFILE sql queries successfully, where mysql qould write the result of a query into a file on the local file system.19:15
sdezieltomreyn: so far I understand from the above that you could extract stuff the mysql user has access to.19:16
tomreynsdeziel: in this scenario, if the DB server runs on the same system as the vulnerable web application, you can access this file via remote file include. not so if the database server runs on a different system and wrote the file on this systems' file system but not that of the web application19:16
sdezieltomreyn: I (think I) understand that part but what I fail to understand is how would that be a bigger threat than leaking the full DB the web app has access to anyways?19:17
tomreynsdeziel: it is only marginally greater. but in the scenario discussed, you can't make the web application leak the full DB its DB user has access to by any other means.19:19
tomreynnormally web applications are not meant to just read the full DB and dump it to the internet ;-)19:19
tomreynwe'Re well beyond the scope of this channel by the way. if anyone thinks we should move elsewhere please say so.19:20
sdezielI'm not worried about the normal case ;) but I'd assume someone with SQL injection and interested in the DB data would simply leak it without the intermediate file19:20
sdezielyeah, that's OT, sorry19:20
tomreynsdeziel: it's all a matter of what the attacker can control. if they can just run any SQL statement they like against the database within the scope of the web applications' database, then surely that means they can dump it.19:22
tomreynthe scenario i meant to describe only allows the attacker very limited control over how sql statements can be modified.19:23
tomreyntv time now, but we can talk later in #ubuntu-offtopic or elsewhere, just ping me.19:25
sdezieltomreyn: thanks19:25
sarnoldthanks for the discussion, it's been fun reading19:25
tomreyn:) and fun for me learning to understand how i can express myself better, and not mixing up the proper terms so much. i bet sarnold would have explained it much better. ;-)19:27
sarnoldI wouldn't be so sure of that -- actually *using* computers isn't my forte :)19:27
sdezielonce again, I get to the conclusion I should learn more stuff to better understand things..19:28
sarnoldheh, yes :)19:28
JamesBenson@gbkersey: FYI: Linux 4.4.0-145-lowlatency #171-Ubuntu SMP PREEMPT Tue Mar 26 13:17:00 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux20:44
gbkerseyJamesBenson: any luck with the 10Gb ?21:26
tewardehehehehehehehehe i feel privileged... xD21:29
tewardI have TWO cable hookups here xD21:29
tewardsarnold: mind helping me test something?21:33

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!