| smackusrevival | possible attack! | 00:00 |
|---|---|---|
| bekks | So maybe it want the webserver which was under attack. | 00:01 |
| bekks | *it wasnt | 00:01 |
| genii | Maybe check the auth log for bruteforce password attempts | 00:02 |
| teward | ^ that | 00:03 |
| teward | smackusrevival: your company isn't stupid enough to have your DB servers listening on public internet addresses for traffic from everything is it? | 00:03 |
| smackusrevival | hold, i think i found problem. | 00:07 |
| smackusrevival | got it. hardware issue. router somewhere had changed ip address routes. reverted back and everything good now. | 00:10 |
| quantic | smackusrevival: what have we learned? | 00:10 |
| smackusrevival | check this first next time. | 00:10 |
| quantic | smackusrevival: more generally - unless you have evidence pointing to an attack, malfunctions or misconfigurations are far more likely. | 00:11 |
| teward | agree with quantic there, smackusrevival :) | 00:12 |
| smackusrevival | thanks for the help. but it doesn't discredit the fact there was major traffic at one point which made server ram 100% during this time. | 00:13 |
| keithzg | The old saying "cockup before conspiracy" ;) | 00:13 |
| teward | smackusrevival: how are you determining RAM usage? | 00:13 |
| teward | smackusrevival: if your system shows 100% but `free -m` shows a lot free with the '+/- buffers/cache' line, then it's not really using all the RAM like you think | 00:13 |
| teward | (caching is at play, and that's not really 'in active use') | 00:14 |
| smackusrevival | like this. http://imgur.com/uCqzGuP | 00:20 |
| smackusrevival | my mistake, not 100% but close. red in graph is ram in use. almost maxed out. | 00:20 |
| smackusrevival | green is cached. | 00:21 |
| smackusrevival | graph pulled for monitorix. | 00:21 |
| bekks | So basically cached memory increased. | 00:21 |
| teward | smackusrevival: so your cached memory increased, but active usage didn't increase | 00:21 |
| teward | smackusrevival: that's not uncommon | 00:21 |
| teward | and you don't really need to be worried about cached memory being consumed up to the max on your RAM | 00:21 |
| teward | though if that were entirely RED that'd be a different story | 00:21 |
| teward | i had a rogue ruby utility that did that once | 00:21 |
| ponyofdeath | hi, what calles dhclient on ubuntu server 12.04 what is the init script | 00:22 |
| smackusrevival | why only at this one time during the day though? | 00:22 |
| sarnold | ponyofdeath: /etc/network/interfaces configures the interfaces; called via the /etc/init/networking.conf upstart script or /etc/init.d/networking sysv-init script.. | 00:23 |
| bekks | smackusrevival: look at the logs. | 00:23 |
| teward | smackusrevival: logs are your friend | 00:23 |
| teward | read them | 00:23 |
| smackusrevival | yeah ok. fair enough. | 00:24 |
| sarnold | backups? updatedb? | 00:24 |
| teward | smackusrevival: also, don't worry if mostly green/cached is taking up a lot of RAM | 00:25 |
| teward | smackusrevival: http://paste.ubuntu.com/13609973/ | 00:25 |
| teward | granted this is my Ubuntu laptop | 00:25 |
| bekks | smackusrevival: http://www.linuxatemyram.com/ | 00:25 |
| teward | but point not withstanding, the site bekks gave is important, AND... | 00:25 |
| teward | you can see most of the 'memory' is used on mine with buffers/cache, not active use | 00:25 |
| teward | (only about 2.4 GB is in use, which is usual for this system since I do a lot of resource-eating stuff) | 00:26 |
| smackusrevival | performing manual backup of server as i write this, as precaution. | 00:27 |
| teward | smackusrevival: 99.999% sure the RAM usage is not a concern :) | 00:27 |
| bekks | smackusrevival: that backup is mostly worthless, since you cant tell for sure wether it is compromised or not. | 00:27 |
| teward | AND bekks is right | 00:27 |
| teward | if you can't tell if your machine is compromised, chances are it may be | 00:27 |
| teward | so a backup won't help you | 00:27 |
| teward | and if you aren't already doing regular backups you should have been | 00:28 |
| bekks | For further investigation purposes, that backup may be helpful, but you should strongly consider setting up that server from scratch. | 00:29 |
| bekks | And - literally - document every change you are doing to that new server. | 00:29 |
| teward | ^ | 00:30 |
| smackusrevival | teward: yeah we have many backups. this was only precautionry, but regular backups are made all the time | 00:31 |
| teward | 'this was only precautionary' is wrong | 00:32 |
| teward | smackusrevival: if your server *was* compromised then it's not precautionary | 00:32 |
| teward | it's recklessly | 00:32 |
| quantic | The only good taking a backup will do is to have an image to analyze later, and taking a normal backup to do so is useless. You need a bit-level image of the system for proper forensics. | 00:32 |
| smackusrevival | looking through /var/log/auth.log there has been MANY unauthorized login attempts. | 00:32 |
| quantic | smackusrevival: Is the system connected to the internet? | 00:33 |
| bekks | The only purpose of that backup *may* be restoring files which have changed since the last know not-compromised backup AND known to be not altered by an attacker. | 00:33 |
| bekks | *known | 00:33 |
| teward | smackusrevival: is the system internet-connected, and do you do any hardening on your servers? | 00:33 |
| smackusrevival | quantic: yes | 00:33 |
| teward | (such as disable pure password auth, etc) | 00:34 |
| quantic | smackusrevival: Then yeah, you're going to have login attempts. Get used to it. | 00:34 |
| quantic | smackusrevival: You should be looking for login SUCCESSES, not attempts. | 00:34 |
| teward | ^ that | 00:34 |
| teward | but you should also be considering hardening your servers a little if you haven't already, such as disable password-based authentication and enforce SSH Key authentication, etc. | 00:34 |
| smackusrevival | quantic: ok, usually ignore logs unless something like this happens. but need to start paying more attention i think. | 00:35 |
| smackusrevival | teward: ok thanks. | 00:35 |
| teward | smackusrevival: NEVER IGNORE LOGS | 00:35 |
| teward | they're IMPORTANT | 00:35 |
| teward | REALLY important | 00:35 |
| bekks | and - for the sake of reducing the amount of failed attempts - chaning the default ssh port. | 00:35 |
| teward | ^ that | 00:35 |
| teward | bekks: though I just IP-restrict SSH xD | 00:35 |
| teward | bekks: in some cases changing the defaults won't help, but yes | 00:35 |
| bekks | teward: that helps too :D | 00:35 |
| teward | I also have Duo Security 2FA enabled so... | 00:36 |
| teward | that PAM layer adds additional authentication reqs to get in | 00:36 |
| sarnold | ... and don't forget, logs are often scrubbeds after systems are compromised | 00:36 |
| teward | indeed | 00:36 |
| teward | sarnold: even better reason to have a syslog server that receives all logs from the server | 00:36 |
| bekks | log time inconsistencies. great thing to script :) | 00:36 |
| teward | so even if they're scrubbed locally | 00:36 |
| teward | they're not necessarily scrubbed at the syslog location xD | 00:36 |
| sarnold | teward: yeah | 00:36 |
| teward | at the *real* syslog location | 00:36 |
| bekks | rsyslog ftw :) | 00:37 |
| teward | sarnold: speaking of which, know any good Ubuntu syslog solutions that make it nice and readable and searchable, etc. including colorcoded by severity levels and stuff? | 00:37 |
| bekks | which reminds me to setup that for a bunch of server regularly used by other employees. | 00:37 |
| sarnold | teward: I keep hearing things about kibana and elasticsearch and on and on but i've never been too impressed with what I see, compared to just tail -F :( | 00:38 |
| sarnold | teward: there's some tool named something like czze that apparently does termina-based log coloring | 00:38 |
| teward | i mean with a syslog server and such included in the list | 00:39 |
| teward | because i want one location NOT on the servers themselves | 00:39 |
| teward | but meh | 00:39 |
| teward | sarnold: i've heard kibana/elasticsearch too but dislike it | 00:39 |
| teward | splunk's always an option, but meh | 00:39 |
| teward | i'll go do more research xD | 00:39 |
| bekks | teward: keep me informed on your proceedings please :) | 00:40 |
| teward | bekks: probably will do so once I poke my other workplace again xD | 00:40 |
| bekks | ;) | 00:40 |
| smackusrevival | gotta go guys. thank you all so much for your help. a sys admin is never alone when he has irc. ;-) | 00:53 |
| teward | smackusrevival: that's why we're here :p | 00:53 |
| sarnold | :) | 00:53 |
| ponyofdeath | sarnold: thanks! | 01:05 |
| sarnold | ponyofdeath: got everything sorted? :) | 01:07 |
| ponyofdeath | sarnold: yup :) | 01:07 |
| sarnold | woohoo :) | 01:07 |
| jvwjgames | I am needing my network interface persistence file fixed or regenerated | 03:10 |
| jvwjgames | Cause I have no network connectivity on that server | 03:11 |
| jvwjgames | Can anyone please help | 03:11 |
| tsimonq2 | jvwjgames: the people in #ubuntu could probably provide help with that | 03:22 |
| jvwjgames | Ya but it is a Ubuntu server witch is why I am here | 03:23 |
| sarnold | have you figured out what config options need to change to get the connectivity back? | 03:23 |
| rbasak | jvwjgames: it's in /etc/udev/rules.d/70-persistent-net.rules on 14.04 I think. | 03:23 |
| rbasak | I believe that if you delete it it'll get regenerated on next boot, but don't rely on my knowledge if it'll break your server - test first. | 03:24 |
| jvwjgames | I know it is but it is messed up is there a way to regenerate it | 03:24 |
| sarnold | rbasak: ahhh that hting I forgot all about it :) | 03:24 |
| rbasak | It has changed since 14.04 and I am no longer up to date with the current stuff. | 03:24 |
| JakeTheAfroPedob | hi guys | 05:07 |
| JakeTheAfroPedob | need some help regarding manaual installation | 05:08 |
| JakeTheAfroPedob | on 14.04 | 05:08 |
| JakeTheAfroPedob | LTS that is | 05:08 |
| === cpaelzer_ is now known as cpaelzer | ||
| orogor | hi | 08:34 |
| orogor | anyone here use/know dell open manage essentials ? | 08:34 |
| OerHeks | orogor, i read about it, but have no PowerEdge Server, the repo instructions is here http://linux.dell.com/repo/community/ubuntu/ | 08:37 |
| lordievader | Good morning. | 09:19 |
| === Lcawte|Away is now known as Lcawte | ||
| hackeron | Anyone has any ideas about this boot issue and how to diagnose further? < https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1521749 | 11:30 |
| ubottu | Launchpad bug 1521749 in linux (Ubuntu) "NUC NUC5CPYH Does not boot on 4.2.0-19 (Ubuntu 15.10)" [Undecided,Confirmed] | 11:30 |
| TJ- | hackeron: did the systemd analysis not help narrow it down? | 11:56 |
| hackeron | TJ-: I added comments about that in the bug. I tried starting various services just 1 by 1 after getting into the emergency.target - I can start networking, I can start several random things - but as soon as I start something like ssh or even apport, I see this and it freezes: https://www.dropbox.com/s/gmpse7bx0c70e4v/IMG_1479.jpg?dl=0 | 12:01 |
| TJ- | ahhh, I still had the bug open from yesterday so couldn't see the updates! | 12:05 |
| TJ- | hackeron: can you add to the bug report a dmesg captured from a 4.2 emergency.target boot? probably need to copy it off to a USB device manually mounted | 12:10 |
| TJ- | hackeron: also, have you tried booting it with 4.2 when *all* USb devices are disconnected? | 12:11 |
| hackeron | TJ-: sure, let me do that | 12:11 |
| hackeron | TJ-: yes, I have | 12:11 |
| TJ- | hackeron: so we can rule out udev stuck due to external devices? | 12:13 |
| TJ- | I notice the iwlwifi firmware isn't found for 3.19... I wonder if 4.2 gets stuck due to that not being available - unlikey, but in the absense of any other evidence...! | 12:14 |
| hackeron | TJ-: yeh, I noticed that as well, but seems unlikely as I can bring networking up without problems | 12:30 |
| TJ- | hackeron: OK, that's the only thing that stands out from the 3.19 logs | 12:31 |
| hackeron | TJ-: I downgraded to ubuntu 15.04 (well, installed from scratch) - I have another unit here, just trying to install 15.10 on it now | 12:44 |
| === Guest50543 is now known as hxm | ||
| rbasak | cpaelzer: I've pushed some suggested changes to the nis merge in https://git.launchpad.net/~racb/ubuntu/+source/nis. Please could you check that you're happy with them? | 13:09 |
| rbasak | cpaelzer: I presume you've re-tested the upgrade paths? If so I'm happy to upload this. | 13:09 |
| rbasak | cpaelzer: my changes are only really minor or couldn't be anticipated by pitti's upload though. I'm completely happy with your solution for the upgrade path - good job on that. | 13:12 |
| hackeron | TJ-: ah! - it seems the older bios does in fact boot on the NUC - but the latest does not, so will check the changelog what Intel changed in the latest bios | 13:16 |
| cpaelzer | rbasak - I tested the upgrade paths, with the new version it continued working | 13:18 |
| cpaelzer | rbasak: no more config file fallout | 13:18 |
| rbasak | Great! | 13:18 |
| cpaelzer | rbasak: let me look into your upload, but I'm generally happy :-) | 13:19 |
| cpaelzer | rbasak: damn I knew I missed the ~ in that version check, but I couldn't find where exactly in the version string to add it - thanks for fixing | 13:20 |
| cpaelzer | rbasak: thanks for fixing my typos and whitespace :-/ | 13:21 |
| cpaelzer | rbasak: about the Tabs I had spaces, but took tabs as the rest of the file had tabs | 13:22 |
| cpaelzer | rbasak: but I'm totally fine with that | 13:22 |
| cpaelzer | rbasak: so as predicted - yes I'm happy | 13:22 |
| rbasak | Tabs are fine, but if you want hanging indent to align with something on the previous line, then leading tabs need to match the upper line's tabs and the rest has to be spaces. | 13:22 |
| rbasak | Otherwise it doesn't match up visually unless you match the tab size setting. And since less and "git diff" uses eight space tabs, it all looks wrong immediately to me :-/ | 13:23 |
| rbasak | OK, thanks. I'll upload! | 13:23 |
| rbasak | Oh, I didn't fix all the version comparisons. I'll do that now. | 13:25 |
| TJ- | hackeron: interesting... did you do a complete factory reset of the BIOS config? | 13:26 |
| rbasak | [ubuntu/xenial-proposed] nis 3.17-34ubuntu3 (Accepted) | 14:07 |
| rbasak | cpaelzer: ^^ do you get that email, out of interest?] | 14:07 |
| cpaelzer | rbasak: no I don't | 14:08 |
| cpaelzer | rbasak: I'm not yet subscribed to all lists that I could be to keep the inbox manageable | 14:08 |
| cpaelzer | rbasak: should that have come through https://lists.ubuntu.com/mailman/listinfo/ubuntu-release? | 14:09 |
| cpaelzer | rbasak: or just because my commits are in there? | 14:09 |
| rbasak | cpaelzer: I just get that email after a dput. I guess it gets sent to the package signer only. | 14:10 |
| rbasak | I thought it might also be copied to you as you're the "uploader" but I guess not. | 14:10 |
| hackeron | TJ-: I didn't reset bios, but this is a brand new unit out of the box - trying now | 14:45 |
| hackeron | TJ-: Not sure if it helped - I rebooted 5 times, 3 of those times it froze, 2 it booted | 14:51 |
| TJ- | hackeron: sounds like a race condition | 14:52 |
| hackeron | TJ-: with 3.19 it reboots every time - I tried via ssh on a loop to reboot 50 times | 14:52 |
| TJ- | hackeron: if you can capture a dmesg when it boots with 4.2 | 14:52 |
| === CiPi is now known as cipi | ||
| rbasak | cpaelzer: I've pushed the nis git branch to ~ubuntu-server-dev/ubuntu/+source/nis for the next merge. I've deliberately not pushed the previous reconstruct and logical tags though as they were mismatched. We shouldn't need them for a future merge anyway, as the current master branch has the logical changes (not fully squashed though). | 15:15 |
| rbasak | It should be enough to transfer forward to the next merge without losing anything or duplicating work I think. | 15:15 |
| cpaelzer | rbasak: I agree, thank you | 15:15 |
| === Piper-Off is now known as Monthrect | ||
| Raboo | Does anyone know if QLogic 57840S is included in the kernel for ubuntu 14.04? | 15:50 |
| Raboo | I'm guessing it's bnx2x | 15:51 |
| TJ- | Raboo: you can search for its PCI ID in /usr/share/misc/pci.ids and then check for a modalias that matches under /lib/modules/<VERSION>/* | 15:53 |
| Raboo | TJ- thanks | 15:56 |
| urthmover | I have been tasked with creating a redundant openstack environment that utilizes docker containers. I do not have physical machines, but I do have a vmware 5.5 environment that I intend to use for the controller and compute nodes. I DO NOT have shell access to the vmware hosts though. What should I install first? What software should I skip because I'm not using physical openstack servers? I did create a ubuntu autopilot env | 17:00 |
| === Pici is now known as Guest87554 | ||
| === jelly-home is now known as Guest98020 | ||
| === Guest87554 is now known as Pici | ||
| === Guest98020 is now known as jelly-effnfn | ||
| === sarnold_ is now known as sarnold | ||
| === Jare_ is now known as Jare | ||
| === jelly-effnfn is now known as jelly | ||
| thomaslnx | hi everyone... | 19:01 |
| thomaslnx | what should I worry when setting up ubuntu server secure? | 19:02 |
| thomaslnx | any suggestions? | 19:02 |
| sarnold | #1 don't use ssh passwords, require ssh keys #2 a firewall is nice to make sure you only expose what you intend to expose | 19:03 |
| genii | Make sure to set up fail2ban, move remote login things to other ports | 19:03 |
| sarnold | get those two right and the rest gets a lot easier | 19:03 |
| thomaslnx | thanks sarnold and genii | 19:06 |
| jcastro | roaksoax: this one's for you guys! http://askubuntu.com/questions/687325/maas-integration-with-ipam-solution | 19:42 |
| foo | I want to change my system time. By default, server is in New York, so all system time is in EST. However, I'm in PST timezone. If I change system timezone to PST, I imagine I shouldn't have any problems since *everything* should change (as opposed to only changing mysql timezone and leaving system in EST). Is this correct? | 20:01 |
| genii | Going forwards in time is fine. When you go backwards the system sometimes becomes confused that some files are from the future | 20:04 |
| sarnold | and don't sleep with or kill your grandparents | 20:05 |
| sarnold | foo: the kernel keeps track of time by counting the number of seconds since jan 01, 00:00:00 1970, in UTC | 20:06 |
| sarnold | foo: the "timezones" are just some libraries that convert those seconds to friendly human display | 20:06 |
| sarnold | foo: different applications have run in different timezones without issue | 20:07 |
| foo | sarnold: thought so, so if I change local timezone, and I have time sensitive software, 3pm will just change to 2pm in both mysql timezone fields and in the software checking against these fields. | 20:07 |
| foo | sarnold: eg. nothing should break, am I understanding this correctly? | 20:07 |
| sarnold | foo: you might run into some poorly written software that doesn't follow the rules, but I can't name any off-hand | 20:08 |
| foo | sarnold: I wrote it all, it's just a matter of using local time from PHP to compare against timestamp in mysql field, so if this all changes, I think I'm good. Oh, and Drupal, but I doubt that'll have any issues (nothing time sensitive there) | 20:09 |
| sarnold | foo: last I looked at sql times I think they either store times via utc or combined with a tz offset.. | 20:10 |
| === rharper` is now known as rharper | ||
| foo | sarnold: yup, tz offset. I set local time, restarted mysql, and I'm golden. Thank you | 20:12 |
| sarnold | foo: woot ;) nice work :) | 20:12 |
| foo | sarnold++ | 20:13 |
| pvl | hey all. im stuck. trying to install 15.10 from usb (which i DD'd), and i keep getting an error that isolinux.bin isnt found | 20:44 |
| pvl | or corrupt | 20:44 |
| === cipi is now known as CiPi | ||
| === apb1963_ is now known as apb1963 | ||
| keithzg | pvl: I'd suggest using usb-creator or mkusb instead of directly DD'ing it, see if that works for you. | 22:09 |
| === Monthrect is now known as Piper-Off | ||
| sadmin | Hey, is there a security/software update schdule? I want to create a patch policy and a schedule | 23:44 |
| bekks | there is no schedule, no. | 23:51 |
| bekks | Patches are released whenever they are released. | 23:51 |
| bekks | Basic schedule might be: patch your systems once a week, on saturday. | 23:51 |
| bekks | At least thats how I am patching things. | 23:54 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!
