| xsl | hello all ... before my server freezes im detecting this message in logs "TCP Peer: {ip} unexpectedly shrunk window 954892853:954895757 (repaired)" | 00:42 |
|---|---|---|
| xsl | any ideas on what it might be? | 00:42 |
| hikenboot | is there ubuntu server (latest stable) settings I can implement that make up for slow drives I have battery backed up everything and a UPS and data doesnt change often. So A large cache might be helpful or other settings I dont know of. Thanks for the pointers! | 00:46 |
| SpaceBass | hikenboot, can you say more? are you losing data to power outages? | 00:49 |
| hikenboot | no data is being lost...everything is fine just I notice the VM (uunder esx5i) has slow typing into the ubuntu server guest and also that the admin panel in wordpress is slow responing | 00:50 |
| SpaceBass | xsl, thats probably normal, what else appears in the log? | 00:50 |
| hikenboot | s/responing/responding/ | 00:50 |
| SpaceBass | hikenboot, ah. What's the VM's config? how much RAM? what's the host box like? | 00:50 |
| xsl | SpaceBass, after that i only see the reboot it self | 00:51 |
| hikenboot | host box only has one hard drive but I have 24 gig of memory for two vms ( a windows 2008 R2 SP1 Domain controller) and (a ubuntu latest stable server running apache and wordpress website) | 00:51 |
| xsl | Feb 23 05:19:03 andy kernel: [30601.301962] usb 2-1.8: USB disconnect, device number 5 | 00:51 |
| xsl | Feb 23 05:20:03 andy kernel: imklog 5.8.6, log source = /proc/kmsg started. | 00:51 |
| hikenboot | hold on let me see how much memory i have assigned to guest to be sure | 00:51 |
| hikenboot | 8 gig to the ubuntu guest | 00:52 |
| xsl | I have lxcontainers on that server... i'm starting to believe its some sort of misconfiguration ( i'm in a dead end .... :( ) | 00:52 |
| hikenboot | 4 virtual cpu's on an 8 core system | 00:52 |
| xsl | do you guys know if lxcontainers support ext4 as a backend lvm? | 00:52 |
| hikenboot | open source guest tools installed | 00:53 |
| SpaceBass | xsl, that TCP window error is pretty common, but usually triggered by lots of data and too little cache on the NIC?still nothing to worry about. It does make one wonder if the NIC itself may be going bad and causing a panic? but that's a stretch | 00:53 |
| xsl | but i dont see anything on the logs | 00:53 |
| xsl | could be a faulty sysctl config? | 00:54 |
| SpaceBass | hikenboot, sounds like quite the box! and 8gb is plenty (at least enough to avoid input lag) ? you might be on to something re disk lag. | 00:54 |
| xsl | its weird because i have several servers like this one ... and only this one gives me problems ( tough its the one with the highest load ) | 00:54 |
| SpaceBass | hikenboot, I'm outta my league past that ? I'd be tempted to research disk caching and your VM provider?and then maybe test with an SSD on the main bus, if for no other reason than to test throughput | 00:55 |
| SpaceBass | xsl, highest load sounds suspicious? I'd start at the most basic level: new/different ethernet cable, different port on the switch, then maybe confirm correct kernel module for the NIC is loaded, and then maybe different NIC ?if only for trouble shooting | 00:56 |
| xsl | SpaceBass, i understand... i already requested a hardware test and the ISP says its all ok | 00:57 |
| xsl | its a rented server | 00:57 |
| SpaceBass | xsl, oh wow, doubly complicated in that case. | 00:57 |
| xsl | i'm so lost that i'm starting to doubt my setup ... | 00:58 |
| xsl | open files problem ... maybe disk out of inodes? | 00:58 |
| xsl | but could not be... that way it didnt hang | 00:58 |
| xsl | i have separate partitions | 00:59 |
| SpaceBass | if you were out of inodes, it'd throw errors in the log long before a crash | 00:59 |
| SpaceBass | xsl, can you throttle the traffic to see if it increases uptime? | 01:00 |
| xsl | i have to check how to do that ... this is a high load webserver | 01:01 |
| SpaceBass | hikenboot, the input delay is suspicious ? with that kind of ram and horsepower, it does sound like disk lag. But I'm not aware of any settings to tine that (though I'm sure some exist). | 01:01 |
| SpaceBass | xsl, maybe on the router, upstream? also, confirm the basics like the NIC in full duplex 100 or 1000 mbs mode | 01:02 |
| xsl | and the proof that it rly hangs is that the software raid ... needs to rebuild sometime | 01:02 |
| xsl | SpaceBass, thx for the tips ... i'm gonna try and see what i can do about the nic | 01:02 |
| SpaceBass | xsl, good luck?I'm curious to know what you learn | 01:03 |
| SpaceBass | xsl, woah,?software raid? mdadmin? | 01:03 |
| xsl | i will report it... its been a mysterious issue to solf | 01:03 |
| xsl | yes | 01:03 |
| SpaceBass | xsl, gig ethernet? | 01:04 |
| xsl | yes | 01:04 |
| SpaceBass | wonder if you are flooding the write buffer on the software raid | 01:04 |
| xsl | whats new to me.. hmmm | 01:05 |
| xsl | how can i check that? | 01:05 |
| SpaceBass | used to happen to me w/ a software raid 5 all the time. | 01:05 |
| xsl | this one is raid1 | 01:05 |
| xsl | so ... now im thinking... maybe i pushed the nofiles too high... | 01:06 |
| SpaceBass | that could do it | 01:07 |
| xsl | in each container i have like ... 65536 | 01:07 |
| xsl | for hard limit of nginx user | 01:07 |
| xsl | for mysql user | 01:07 |
| xsl | for php user | 01:07 |
| xsl | and the default is 1024 | 01:08 |
| SpaceBass | have seen systems with 75000, so 65000 doesn't seem too high, but that could very well be it | 01:08 |
| xsl | i have like 18G of ram .. and its allways at 30% of its capacity, couldn't i use ram to tweak this out? | 01:09 |
| SpaceBass | you could tune mysql to use more ram | 01:10 |
| SpaceBass | assuming it's DB writes thats the issue | 01:11 |
| xsl | i'm using a Innodb buffer pool size of 6Gb | 01:12 |
| xsl | you might be right | 01:12 |
| xsl | i'm pushing the disks | 01:12 |
| SpaceBass | could it be heat? | 01:14 |
| SpaceBass | are your running lm-sensors ? | 01:14 |
| xsl | and i have innodb_flush_log_at_trx_commit=1 ... maybe i should set it to 2 | 01:14 |
| xsl | no, but i can install it | 01:14 |
| SpaceBass | I had an overheating issue for a while?set up a cron to push CPU temps to my iPhone every 15 mins (if over critical)? ended up buying a $15 fan off amazon and it solved the problem. | 01:15 |
| xsl | well this is in a rented server, i want to believe that they have a good ventilation | 01:16 |
| SpaceBass | 2 is each commit, right? | 01:16 |
| xsl | but non the less.. its a good thing to keep track | 01:16 |
| xsl | yesterday it rebooted itself .. maybe its rly heat | 01:17 |
| SpaceBass | those drives could be cranking out some heat | 01:17 |
| xsl | i have noted down all the ideas you gave me ... its been rly helpfull ... i will tell you my findings | 01:19 |
| xsl | thx | 01:19 |
| SpaceBass | in fairness, I'm no expert. But enjoyed thinking through the troubleshooting. | 01:21 |
| SpaceBass | keep us posted! | 01:21 |
| xsl | sure ty once again | 01:21 |
| === paddymahoney1 is now known as paddymahoney | ||
| rurufufuss | if hdparm shows fast speeds (e.g around 100MB/s) but cp etc is ridiculously slow (1MB/s), what are the possible reasons? | 02:03 |
| === paddymahoney1 is now known as paddymahoney | ||
| === sweettea is now known as Guest48399 | ||
| jetole | Hey guys. I hope someones around because I need some help. Not as in my system is about to explode but post recovery forensics to determine what caused a server issue. I have some theories and I have what data I thought to collect prior to the reboot. I don't know where to begin other then I guess explain the situation | 11:56 |
| jetole | we have a Linux firewall we use in production, in a rack at the data center. It's actually one of two which provides high availability via conntrackd | 11:56 |
| jetole | the server stopped accepting ssh requests mid Nov. It showed the port was open and sshd actually gave a error which I don't have in front of right now but I'll pull that up in a minute | 11:57 |
| jetole | anyways, I went to the data center the other day and I saw the server clearly had a issue | 11:57 |
| jetole | it's unix load average was 7000+ | 11:58 |
| RoyK | I guess without the error, no fun | 11:58 |
| RoyK | oops | 11:58 |
| RoyK | that's a bit ;) | 11:58 |
| jetole | the error isn't relavent but | 11:58 |
| jetole | yeah... | 11:58 |
| jetole | so as I was saying | 11:58 |
| jetole | I did some pre-reboot checks and I found one of the main causes seemed to be cron | 11:58 |
| RoyK | high load is usually because of threads hanging in D state, because of bad i/o | 11:58 |
| jetole | oh and by the way the high load is what causes the error on ssh but ssh isn't the problem here | 11:58 |
| jetole | which is the case | 11:59 |
| jetole | and I am suspecting this may be driver related | 11:59 |
| jetole | at first I thought it was batched cron jobs | 11:59 |
| RoyK | check if processes are in D state | 11:59 |
| RoyK | that is - have you rebooted it yet? | 11:59 |
| jetole | but then I noticed we had some sshd instances that were also hung and netstat said they were in wait close I believe it was but they have been hung for several months | 11:59 |
| jetole | RoyK: I did but I saved a lot of stats prior and yes @ d state | 12:00 |
| RoyK | which processes were in D state? | 12:00 |
| jetole | which is making me suspect it's a driver issue. If sshd is hung on a tcp close for several months... well it makes me think it is in D waiting on the NIC to realease the uninterupptable lock | 12:00 |
| jetole | many, let me pull up the PS log I did before the reboot | 12:01 |
| RoyK | I've never seen D state be network related | 12:01 |
| RoyK | always disk related the times I've seen it climb | 12:01 |
| RoyK | but I don't know the internals well enough to say for sure | 12:02 |
| maswan | also, did you save dmesg output? | 12:02 |
| maswan | and df? | 12:02 |
| jetole | well I don't want to sound bias but we had some network issues with the broadcom nics when we deployed these servers prior to upgrading the driver from the broadcom site | 12:02 |
| jetole | it's using bridge on bond | 12:02 |
| RoyK | maswan: df??? | 12:02 |
| maswan | RoyK: yes, filesystems don't always play nice when you fill them up | 12:02 |
| jetole | maswan: I wish I did but I just pulled copies of the dmesg log as well as others from the server now and this server had been in this state for several months now | 12:03 |
| maswan | RoyK: a full /var/log could stick lots of processes in D | 12:03 |
| RoyK | maswan: erm - a full filesystem making load exceed 7k? never seen that ;) | 12:03 |
| RoyK | maswan: why? | 12:03 |
| RoyK | they get an error writing to disk | 12:03 |
| RoyK | if the filesystem is full | 12:04 |
| maswan | RoyK: Not necessarily | 12:04 |
| RoyK | they aren't put in d state | 12:04 |
| maswan | RoyK: Sometimes they just get stuck instead | 12:04 |
| jetole | I don't think the FS is full but let me check. I also note that the server seemed very responsive when I logged in via the console despite the 7000+ PIDs and load avg | 12:04 |
| RoyK | I'd love to see that demonstrated | 12:04 |
| maswan | RoyK: xfs is nutorious for that, but can happen to other filesystems too | 12:04 |
| jetole | no xfs | 12:04 |
| maswan | RoyK: Only happens in certain circumstances, but we see it happen a couple of times per year | 12:05 |
| RoyK | maswan: got a reference for that? | 12:05 |
| jetole | df is good | 12:05 |
| jetole | nothing listed above 10% usage | 12:05 |
| RoyK | maswan: I've never seen that... | 12:05 |
| jetole | now let me look at the PS file as RoyK asked which procs were in D | 12:05 |
| jetole | and I know cron has 7000+ procs where most were in D but don't know what else | 12:06 |
| RoyK | jetole: that usually means cron is trying to write to a dangling filesystem. I've seen that with NFS | 12:06 |
| RoyK | then it's stuck in D state and can't be killed until the I/O transaction is completed | 12:07 |
| RoyK | meaning *high* load may occur | 12:07 |
| jetole | RoyK: but sshd hung while waiting for a tcp close since Nov? | 12:09 |
| RoyK | jetole: do you have dmesg output? | 12:09 |
| maswan | RoyK: when most allocation groups are full and you do many concurrent writes the last few blocks might become wedged instead. "xfs full filesystem hang" seem to find some of those refernces | 12:09 |
| RoyK | maswan: it seems like a very rare case - still, this isn't xfs, as jetole said. | 12:10 |
| jetole | I don't think this server has nfs but lets go back to which procs. This is a big PS file as I used ps -o x,x,x,x,x,x,x,x,x,x,x,x specifying every little detail from the ps man page I could think might be important. does anyone know the awk syntax for multiple columns? I typically only use it for one column | 12:10 |
| jetole | RoyK: one thing at a time here. I'm only human | 12:10 |
| RoyK | dmesg? | 12:10 |
| jetole | so dmesg first? ok | 12:10 |
| jetole | one min | 12:10 |
| jetole | well... | 12:12 |
| jetole | the last dmesg seems to be wrote at 44.67xxxxx on the one saved to /var/log/dmesg. I wish I got the live one but this looks like we have bnx2 issues already | 12:13 |
| jetole | ... or not. It looks like it's writing the allocations | 12:13 |
| jetole | irq allocations. | 12:13 |
| jetole | my mistake | 12:13 |
| jetole | @ RoyK | 12:13 |
| RoyK | will need the live one to see the errors | 12:14 |
| jetole | the system has been rebooted already | 12:14 |
| RoyK | iirc /var/log/dmesg is just the one from the bootup | 12:14 |
| jetole | it may be | 12:14 |
| jetole | one sec and let me tell you what I have | 12:14 |
| RoyK | it is | 12:14 |
| RoyK | just checked | 12:14 |
| jetole | I have lsmod, lspci, lsof, ps with the following columns: PID,PPID,STARTED,S,BLOCKED,CAUGHT,CLS,TIME,F,IGNORED,LWP,NI,NLWP,PENDING,PGID,PRI,PSR,RSS,SCH,SESS,RSS,SZ,STACKP,STAT,SZ,TT,VSZ,WCHAN,USER,GROUP,CMD,CMD | 12:16 |
| jetole | I also have logs from newest to oldest pre log rotate for: conntrackd, dmesg, kern, messages, syslog | 12:17 |
| RoyK | still doesn't help, since what's needed, is the live dmesg at the time of the problem | 12:18 |
| RoyK | I guess I/O was hanging | 12:18 |
| RoyK | that is, the disk or subsystem | 12:18 |
| jetole | so you're saying I'll never be able to figure it out since I don't have the dmesg? you don't think syslog or lsof may hold some clues? it was in this state from Oct 8th until last night | 12:19 |
| jetole | I'm skeptical | 12:19 |
| jetole | on disk | 12:19 |
| jetole | also, the system was booted on apr 2nd and didn't start to have these back logged / hung procs till oct 8th | 12:19 |
| RoyK | pastebin the syslog (or put it somewhere) | 12:20 |
| jetole | I really, really want to | 12:20 |
| RoyK | if I/O was hanging, this will probably happen again | 12:20 |
| jetole | but | 12:20 |
| jetole | this is corporate | 12:20 |
| jetole | I can't | 12:20 |
| RoyK | didn't you say this was one of two in a cluster? | 12:20 |
| jetole | I could be tarred, feathered and hung if I did | 12:20 |
| jetole | RoyK: it will probably happen again but it took 7 months before it started and yes @ one of two | 12:21 |
| RoyK | jetole: there's no way of finding a lost dmesg. period. so if there's nothing in the logs, there's nothing in the logs | 12:21 |
| * RoyK thinks jetole will remember dmesg next time | 12:21 | |
| jetole | RoyK: who says there's nothing in the logs | 12:22 |
| RoyK | well, post the logs | 12:22 |
| jetole | I'm just starting forensics now. I'm hoping something is in the logs | 12:22 |
| RoyK | I can scan through them | 12:22 |
| jetole | I wish I could but I can't. I'm sorry. I just can't. Appreciate any hints you can give though since this is a lot of logs | 12:22 |
| RoyK | then use egrep -v 'unimportant|blah|blah' logfile | 12:23 |
| RoyK | and you'll end up with whatever you don't understand, which may be interesting | 12:24 |
| RoyK | but if processes are stuck in D state, they *hang* and can't write to logs | 12:24 |
| RoyK | they won't notice they're hanging | 12:24 |
| jetole | yeah I'm about to do something similar. I just changed to the syslog dir and ran while read file; do cat "$file" >> master.syslog; done < <(ls -1 | tac) # | 12:24 |
| RoyK | so you probably won't find anything | 12:24 |
| jetole | about to start vim'ing the master file and :g /pattern/d for all unimportant | 12:24 |
| jetole | oh | 12:25 |
| jetole | ... well that sucks | 12:25 |
| RoyK | just wait | 12:25 |
| RoyK | monitor the server regularly | 12:25 |
| jetole | yeah I'm also going to start writing a montoring script this weekend to help us catch this earlier next time unless I can prove what the failure is first | 12:25 |
| RoyK | use icinga or something to generate alerts if the load gets too high | 12:25 |
| jetole | right | 12:26 |
| * jetole prefers nagios but I get the point | 12:26 | |
| jetole | I'm gonna go hop in the shower. I'll be back in a bit | 12:28 |
| === highvolt1ge is now known as highvoltage | ||
| jetole | RoyK: I had 20 minutes to collect this information before I had to perform a scheduled and planned fail over and reboot. We just recently found out about this issue and while we use Nagios, this server is... I don't know how to phrase it without breaking NDA's so let's just say a different class then the rest but in the future it's going to be added to nagios. Anyways, I had 20 minutes where I had attempted to somehow ... | 12:56 |
| jetole | ... recover the server before the reboot and during the last 5 mins when I realized this wasn't possible, off the top of my head I thought what do I need to save before the reboot, let's get it. Anyways, yes, I'll remember dmesg last time but this was just a different situation then you may be used to so please don't be too quick to judge | 12:56 |
| RoyK | setup syslog to log to a different server | 12:57 |
| jetole | we will | 12:58 |
| RoyK | the kernel log should hold whatever comes to dmesg | 12:58 |
| jetole | like I said, it's hard to explain but not in your typical class of how we keep servers normally | 12:58 |
| jetole | it's kind of new to us to access it but not new as in just been deployed. It's complex | 12:58 |
| jetole | and NDA's | 12:58 |
| RoyK | ok | 13:00 |
| jetole | I know | 13:01 |
| jetole | I wish I could say more but I can't | 13:01 |
| * jetole sighs | 13:01 | |
| jetole | joy to corporate politics but they do keep the pay checks comming :-) | 13:01 |
| morfeo_81 | hi | 13:48 |
| morfeo_81 | how can I found file on flashplayer | 13:48 |
| morfeo_81 | lsof!grep flash | 13:49 |
| === nixon is now known as Guest14856 | ||
| === Guest14856 is now known as n1xon | ||
| xsl | hello all, i cannot find the /sys/block/md0/md/stripe_cache_size file .. is this been removed ? how will i know the stripe cache size of my mdadm device? | 16:21 |
| RoyK | what linux version? | 16:22 |
| RoyK | works for my machines - on ubuntu 12.04 or later | 16:23 |
| xsl | i have ubuntu 12.04... weird | 16:25 |
| xsl | Description: Ubuntu 12.04.2 LTS | 16:25 |
| xsl | cat /sys/block/md1/md/stripe_cache_size | 16:26 |
| xsl | cat: /sys/block/md1/md/stripe_cache_size: No such file or directory | 16:26 |
| xsl | it was not md0 sorry ... i want the second partition of the disks | 16:26 |
| xsl | that is built into a raid1 | 16:26 |
| RoyK | xsl: do you have anything under /sys/block? | 16:38 |
| RoyK | and is your md dev named md1? | 16:38 |
| RoyK | pastebin /dev/mdstats | 16:39 |
| RoyK | pastebin /dev/mdstat even | 16:40 |
| xsl | http://pastebin.com/KGYGgjUS | 16:47 |
| xsl | ty RoyK for the time | 16:47 |
| _jfb | RoyK are you around?? | 16:48 |
| * RoyK is | 16:49 | |
| RoyK | _jfb: long time no see :) | 16:49 |
| _jfb | RoyK: indeed! | 16:50 |
| _jfb | busy days! You? | 16:50 |
| RoyK | well, somewhat busy, but I'm not sweathing | 16:50 |
| _jfb | my home theater PC was just hacked!!! We were just sitting here and the mouse started moving around, they opened a browser and pointed to ip2location.com before I could shut it off... the IP (looking at my router) is coming from Egypt. Suggestions? What the F%#$ to do to be sure my home network is 'cleased'?? :o | 16:51 |
| _jfb | cleansed... | 16:52 |
| _jfb | I've taken that computer offline for now, but our others are still online... | 16:52 |
| RoyK | rkhunter and chkrootkit is a good start | 16:53 |
| RoyK | if the box is rooted, well, reinstall it - you never know what they left | 16:53 |
| RoyK | oh, in terms of rooting, check out this book - it's just *brilliant* http://craphound.com/rotn/ | 16:54 |
| RoyK | comes in dead tree versions too | 16:54 |
| RoyK | _jfb: any windows machines on that network? | 16:55 |
| _jfb | we don't know for how long they've been here... so yes, there's one. | 16:55 |
| RoyK | check last -10 for unknown ssh logins | 16:56 |
| RoyK | check for rootkits | 16:56 |
| RoyK | check the system logs | 16:56 |
| RoyK | in that order, usually | 16:56 |
| _jfb | what do you mean if the box is rooted? The user that was logged on has sudo. | 16:56 |
| RoyK | use rkhunter *and* chkrootkit to check if there's a rootkit around | 16:57 |
| RoyK | rootkits will let the intruder access the system without futher logins | 16:57 |
| RoyK | if the account used had or has sudo access without password, better reinstall the box | 16:58 |
| _jfb | ok. | 16:58 |
| _jfb | freaking annoying. | 16:59 |
| RoyK | I know | 16:59 |
| funkyHat | Do you have remote desktop enabled, and have the port for it forwarded from your router? | 16:59 |
| _jfb | I'll take it as a learning experience. | 16:59 |
| xsl | _jfb, do you use java on your system? | 16:59 |
| Alienhead | i have a windows server 2003 box with an ntfs formatted raid5 array on a softraid card. is there a way to assemble the array in ubuntu and mount it? | 17:00 |
| RoyK | _jfb: first machine rooted is always inconvenient | 17:00 |
| _jfb | I have a router port forwarding to ssh port | 17:00 |
| _jfb | xsl: yes, java was recently installed... in fact, I think for some remote android ap I was playing with! | 17:00 |
| RoyK | java doesn't open new ports | 17:00 |
| RoyK | and the router in front should stop access unless you browse from it | 17:01 |
| xsl | java executes anything you want :P | 17:01 |
| xsl | there have been several exploits on java | 17:01 |
| funkyHat | _jfb: and your'e sure no-one else in the house might have messed around with your remote android app? | 17:01 |
| xsl | it can log keystrokes | 17:01 |
| xsl | send to hacker | 17:01 |
| xsl | and then ... | 17:01 |
| xsl | you get the picture | 17:01 |
| RoyK | xsl: not unless you browse from the system | 17:02 |
| xsl | dont allow plain text passwords on your ssh .. user rsa certs | 17:02 |
| xsl | s/user/use | 17:02 |
| RoyK | xsl: "plaintext" on ssh is rather safe if your passwords are good | 17:02 |
| xsl | RoyK, not necessarly.. you can visit a website that offers "free something" and your being compromised | 17:02 |
| xsl | passwords are easy to get logged | 17:03 |
| RoyK | xsl: erm - you have to browse from that server for that to work | 17:03 |
| RoyK | or perhaps use the same username and password for that service | 17:03 |
| _jfb | funkyHat: certain. | 17:03 |
| RoyK | which means you're doing something stupid | 17:04 |
| funkyHat | RoyK: xsl is talking about a java web applet on the client machine logging keystrokes | 17:04 |
| RoyK | can really a web applet log keystrokes? | 17:04 |
| xsl | its very common these days | 17:04 |
| _jfb | funkyHat: RoyK: xsl: fearing I may have done "something stupid"... carelessly playing around looking for these android remotes. | 17:05 |
| _jfb | was feeling a little suspicious at times. | 17:05 |
| Alienhead | its only stupid if you knew better at the time and did it anyway | 17:06 |
| xsl | i never install android apps that have only "2 or 3" reviews | 17:06 |
| RoyK | _jfb: did you find a rootkit? | 17:06 |
| xsl | if even a coder on CM project was caught loggin stuff... imagine people that give away "game cheats for android games" "free very good apps that dotn have ads" | 17:07 |
| xsl | if you use the simple clamav you might find virus on your temporary files | 17:07 |
| funkyHat | _jfb: as others have said, the safest thing to do is reinstall. You might find that something quite benign went on though | 17:07 |
| xsl | firefoxx or chromium or whatever | 17:07 |
| xsl | if you dont reinstall you will never be 100% sure... trust me .. the first time is a killer one :D | 17:08 |
| xsl | and using RSA files to auth yourself is a good idea ... it prevents the need to install fail2ban or something | 17:08 |
| xsl | for ssh i mean | 17:09 |
| xsl | RoyK, did you take a look at http://pastebin.com/KGYGgjUS ? | 17:10 |
| xsl | and i'm using Ubuntu 12.04 | 17:10 |
| RoyK | xsl: sorry - don't know | 17:12 |
| _jfb | Searching for suspicious files and dirs, it may take a while... The following suspicious files and directories were found: | 17:12 |
| _jfb | /usr/lib/jvm/.java-1.6.0-openjdk-amd64.jinfo /usr/lib/pymodules/python2.7/.path | 17:13 |
| _jfb | result of chkroot. | 17:13 |
| xsl | first thing :( imho dont use openjdk .. and install oracle java 7 | 17:13 |
| _jfb | and rkhunter: | 17:16 |
| _jfb | /usr/bin/whoami [ OK ] | 17:16 |
| _jfb | /usr/bin/unhide.rb [ Warning ] | 17:16 |
| _jfb | /usr/bin/mawk [ OK ] | 17:16 |
| xsl | unhide is from a package you have installed | 17:23 |
| RoyK | (hopefully) | 17:24 |
| RoyK | _jfb: for your new setup, use fail2ban or perhaps denyhosts to block ssh connection attempts | 17:25 |
| RoyK | or use key-based login | 17:25 |
| xsl | :) | 17:25 |
| RoyK | the latter is more secure, but doesn't allow you to login from everywhere | 17:25 |
| _jfb | yes, I guess key based ... | 17:25 |
| xsl | just have your key with a passphrase in a USB disk and you will be fine | 17:26 |
| RoyK | we have some hosts at work requiring both key and password | 17:26 |
| RoyK | that's pretty secure | 17:26 |
| RoyK | RequiredAuthentications2 publickey,password | 17:28 |
| RoyK | put that in sshd_config | 17:28 |
| xsl | that way he needs to have both auth to login | 17:29 |
| xsl | nice | 17:29 |
| _jfb | This hurts! What a pain it's going to be... :/ | 17:29 |
| RoyK | _jfb: first time rooted? :) | 17:29 |
| _jfb | yup. | 17:29 |
| RoyK | it hurts badly, but you learn a bit from it | 17:30 |
| xsl | the biggest pain will be that your going to start building the new server... and you wanna harden each step :) | 17:30 |
| _jfb | I've always been a little suspicious of the level of Paul's security... but I guess now I understand! | 17:30 |
| _jfb | yup. | 17:30 |
| RoyK | hehehe | 17:31 |
| RoyK | I guess Paul has had a box rooted, then | 17:31 |
| _jfb | hehe, perhaps. One thing for certain, he's going to enjoy hearing about this! | 17:32 |
| RoyK | probably ;) | 17:32 |
| RoyK | I guess you made two mistakes | 17:33 |
| RoyK | one: a bad password, or someone sniffed it | 17:33 |
| RoyK | two: sudo without password | 17:33 |
| xsl | RoyK, that file i was chasing at .. does not exist on raid0 or raid1 | 17:34 |
| xsl | its for raid5 and raid6 | 17:34 |
| _jfb | three: installing all these stupid android remotes... I'm pretty convinced. | 17:34 |
| RoyK | xsl: ah - that makes sense | 17:34 |
| xsl | i need to increase write buffer for my mdadm devices i have a mysql server with a large innodb pool and my server freezes each 2 days :( | 17:34 |
| RoyK | xsl: I only have raid6 here | 17:34 |
| xsl | im thinking its a disk problem since i dont have nothing ( rly nothing ) on my logs | 17:35 |
| RoyK | xsl: the main issue there, is that you're using mysql ;) | 17:35 |
| xsl | lol | 17:35 |
| xsl | i dont know that much of postgres | 17:35 |
| RoyK | it works far better | 17:35 |
| RoyK | sql syntax is about the same | 17:35 |
| xsl | and i used a online tool from percona website... and i believe they push too much out of the hardware... and i dont have a raid controler .. its 2 disks doing all the job | 17:36 |
| xsl | well the problem is i dont know how to administer it that well | 17:36 |
| xsl | mysql i know all the syntax to create, view, bla bla bla | 17:36 |
| xsl | give permissions, take, etc... | 17:37 |
| RoyK | mysql is a pile of * | 17:37 |
| xsl | and this is from a community of 1000 concurrent users accessing a ipb forum ... | 17:37 |
| RoyK | well | 17:37 |
| RoyK | mysql works well for reads | 17:37 |
| xsl | this has alot of writes | 17:38 |
| RoyK | but don't use mysql in something that uses transactional databases | 17:38 |
| RoyK | just my opinion | 17:38 |
| RoyK | postgresql is faster for various workloads | 17:38 |
| RoyK | mysql for read-mostly | 17:38 |
| RoyK | and if you're just using simple databases without stored procedures or other hacks, moving to psql will be easy | 17:39 |
| xsl | i will take a look in to it | 17:42 |
| xsl | since i have my server with lxcontainers and each has its own software.. like a nginx.lxc php.lxc mysql.lxc | 17:42 |
| xsl | i can create a container and migrate the data | 17:42 |
| xsl | then i will just change in the php.lxc with php-fpm the socket and ip of the data | 17:43 |
| RoyK | postgres uses the system buffer for caching | 17:43 |
| RoyK | instead of allocating memory of its own | 17:43 |
| RoyK | that helps out a bit | 17:43 |
| RoyK | _jfb: what did those android remotes do? | 17:43 |
| xsl | have you tried linux containers and running postgres inside of them ? | 17:43 |
| RoyK | no, but since postgres leaves the OS to do the caching, I'm pretty sure it will perform better than the dedicated memory caching in mysql | 17:44 |
| RoyK | s/leaves the/leaves to the/ | 17:44 |
| _jfb | RoyK: let me access ubuntu using my phone... | 18:37 |
| _jfb | via a java server. | 18:37 |
| RoyK | ok | 18:37 |
| RoyK | was that open from the internet? | 18:37 |
| _jfb | no. | 18:37 |
| RoyK | then that shouldn't be the problem, really | 18:37 |
| _jfb | but it required jre/java... so who knows what was lurking. | 18:38 |
| RoyK | well, java doesn't open any ports | 18:38 |
| _jfb | well, like I said, I don't *know* that it wasn't open. | 18:38 |
| RoyK | and so far you have said only ssh was open | 18:38 |
| RoyK | in the router | 18:38 |
| _jfb | What was weird, is we were just sitting here... and the mouse started to move. | 18:38 |
| _jfb | yes, that's correct. | 18:38 |
| _jfb | one port on my router directing to 22 | 18:39 |
| _jfb | on this box. | 18:39 |
| RoyK | perhaps someone pulled your leg? | 18:39 |
| _jfb | ?? | 18:39 |
| _jfb | my two year old son? | 18:40 |
| RoyK | it's rather uncommon for a hacker to engage in interactive takeover of a system | 18:40 |
| _jfb | like I was saying, then they opened a browser (chrome) and opened the url: ip2location | 18:40 |
| RoyK | not a javascript doing that? | 18:41 |
| _jfb | yes, probably not a very savvy hacker -- maybe just a kid messing around... but freaky none the less. | 18:41 |
| _jfb | I don't think javascript can move a mouse around or launch two seperate browsers (they tried firefox first, but it started updating)... then they chose chrome | 18:43 |
| _jfb | like I said, we were using the box, it just happened that we were sitting here and had our tv on (the monitor)... | 18:43 |
| _jfb | anyway, definitely going to scrub this box. | 18:44 |
| RoyK | _jfb: did you see the same behaviour from a different client? | 18:48 |
| RoyK | might be your mac is rooted | 18:49 |
| _jfb | RoyK: what do you mean mac? | 19:01 |
| _jfb | I've never had anything like this happen before... | 19:02 |
| ikonia | mac address | 19:02 |
| _jfb | how can a mac address be rooted? | 19:02 |
| _jfb | RoyK: back to your comment about it being 'uncommon', now I wish I had let them keep playing... just to see what they were up to ;) | 19:03 |
| _jfb | the one fortunate thing of all this, it wouldn't be too easy to connect that box to me. | 19:04 |
| xsl | _jfb, you have X forward on in you sshd? | 19:04 |
| xsl | *your | 19:04 |
| xsl | thats enough to "move your mouse" and "see your desktop" | 19:05 |
| xsl | but to be honest... if an hacker is good enough to root you.. he does not need to move the mouse to check a website to know from where is your connection | 19:05 |
| xsl | gtg | 19:05 |
| RoyK | no chance x forward would make it though ssh without authentication | 19:06 |
| _jfb | RoyK: what did you mean that my mac might be rooted? | 19:06 |
| RoyK | it seems unlikely that the server with only ssh access in should be compromised | 19:08 |
| _jfb | so you think my router is compromised? | 19:09 |
| _jfb | I'm not following... | 19:10 |
| RoyK | no | 19:14 |
| RoyK | just check last -10 | 19:15 |
| RoyK | or -100 | 19:15 |
| RoyK | on that server | 19:15 |
| === marahin is now known as system | ||
| === system is now known as marahin | ||
| Oblivion1500 | hello, i was wondering how you add permissions to a user to edit a file say the user is username@localhost and the dir is /example | 23:58 |
| Oblivion1500 | or i mean edit a directory | 23:59 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!