| === Monthrect is now known as Piper-Off | ||
| tsimonq2 | hi, where would the last meeting's logs be? I can't seem to find them... | 00:27 |
|---|---|---|
| === m1dnight1 is now known as m1dnight_ | ||
| PCatinean | Hello everyone, I have received an email from the hosting service that my server was sending out spam.Looking at the server logs I see hundrets of email being sent constantly | 07:10 |
| PCatinean | I've shut the postfix server down for now but now I need to investigate and see the cause, can anyone please help me debug this so I can take the proper course of action? | 07:11 |
| cpaelzer | PCatinean: Hi, is that the same discussion as started on #ubunutu ? | 07:11 |
| PCatinean | cpaelzer, yes it's the same, I was refered to this chan for ubuntu server, but yeah | 07:12 |
| cpaelzer | PCatinean: IHO - since the discussion there already started to drag in more people lets keep it there for now | 07:12 |
| PCatinean | Ok will do, thanks cpaelzer | 07:13 |
| cpaelzer | PCatinean: we are fine to get here again if they refer you to go "off" the chan :-) | 07:13 |
| PCatinean | Haha, ok thanks :D | 07:13 |
| PCatinean | what is your take on this btw cpaelzer ? | 07:15 |
| cpaelzer | PCatinean: I'm not an expert, but I'm fine giving you my take :-) | 07:17 |
| cpaelzer | PCatinean: in general I'd have two ways of attack #1 stop it spreading #2 analysis | 07:17 |
| PCatinean | Done first by shutting down postfix server I guess | 07:17 |
| cpaelzer | PCatinean: #1 would mean I take the server off the network entirely (if possible) and keep it for later debugging | 07:17 |
| cpaelzer | PCatinean: replacing it with a totally fresh and healthy new one for a while at least | 07:18 |
| cpaelzer | PCatinean: then there would be time to do #2 and find out what happened - with that you have an idea if/what to search in your further environment | 07:18 |
| cpaelzer | PCatinean: there are a few good links I found on the first search - probably those guys have thought way more about it - e.g. | 07:18 |
| cpaelzer | https://wiki.ubuntu.com/BasicSecurity/DidIJustGetOwned http://serverfault.com/questions/218005/how-do-i-deal-with-a-compromised-server http://www.cert.org/historical/tech_tips/win-UNIX-system_compromise.cfm (old) | 07:19 |
| PCatinean | Thanks a lot cpaelzer | 07:20 |
| PCatinean | Btw I have ispconfig installed if that makes a difference | 07:20 |
| cpaelzer | PCatinean: honestly If I'm in your situation I'd go to people who did that more often - I expect there are a lot things that can be done right or wrong :-/ | 07:21 |
| cpaelzer | PCatinean: are you in the lucky case that you have some kind of official support contract like http://www.ubuntu.com/management/ubuntu-advantage (or others) - so you could tap on these ressources? | 07:21 |
| cpaelzer | PCatinean: if not my personal way would go to some friends with an IT Sec background, but well you need to find those right :-) | 07:22 |
| cpaelzer | PCatinean: if nothing of that is an option for you my (personal) direction would be as stated above, take it off the net, replace it so your service works and read all the links (and more) to start breaking down what happened | 07:23 |
| cpaelzer | PCatinean: I never used ispconfig, so I don't know if it would help you in any way in this case | 07:23 |
| cpaelzer | PCatinean: all of that https://wiki.ubuntu.com/BasicSecurity is good, but most is about preparing, IMHO only the part I linked before is for "after-the-fact" cases | 07:24 |
| PCatinean | cpaelzer, thank you for the detailed answer.It's clear that I just have to read through links, break them down one by one and In the meantime I've contacted a sysadmin | 07:24 |
| cpaelzer | PCatinean: yeah thats probably wise, so you are not owning the systems but only the services on them? | 07:25 |
| PCatinean | I do own the system I have full root access | 07:25 |
| cpaelzer | PCatinean: but it is hosted somewhere for you then regarding "I contacted a sysadmin"? | 07:26 |
| PCatinean | Ah yes he is a general sysadmin does not belong to the hosting company where the admin is | 07:27 |
| cpaelzer | PCatinean: ok, sorry I couldn't help more - I hope you quickly find what happened and it isn't too widespread | 07:27 |
| PCatinean | Sysadmin as in he works with this stuff on a regualar basis not like me that I just need them when I need to put my programming work online | 07:27 |
| PCatinean | Me too, thanks a lot for your patience :D | 07:27 |
| cpaelzer | PCatinean: ok, I see that is your friend to ask then | 07:27 |
| === Lcawte|Away is now known as Lcawte | ||
| PCatinean | cpaelzer, found this in the log, I think this is the first time they broke in | 08:59 |
| PCatinean | 82.211.31.232 | 08:59 |
| PCatinean | oops no that's the ip | 08:59 |
| PCatinean | http://hastebin.com/izuwihohon.pl | 08:59 |
| cpaelzer | PCatinean: well then it is at least not too old yet | 09:03 |
| cpaelzer | PCatinean: this can be false positives in case the IP is actually one of you and missing in @local_domains_maps | 09:04 |
| PCatinean | It's sending out emails to trolololo, that wasn't any of our users for sure :)) | 09:04 |
| PCatinean | So it does not specify which email address was used? | 09:05 |
| cpaelzer | PCatinean: it seems it doesn't specify, but since with SMTP the sending addr could easily be faked it might only lead you astray anyway | 09:06 |
| cpaelzer | maybe that is why they didn't report it | 09:06 |
| PCatinean | Sorry not to familiar with the terms and how it works | 09:07 |
| PCatinean | Doesn't that line imply that the email server received a order to send to those email addresses which means somebody had a username? | 09:07 |
| cpaelzer | PCatinean: well they faked the submitter to example.com didn't they | 09:08 |
| cpaelzer | PCatinean: yes I read it as it got an order to send, but authentication and SMTP are two things that don't stick together all too well | 09:09 |
| cpaelzer | PCatinean: just read the first paragraph https://en.wikipedia.org/wiki/Email_spoofing :-) | 09:09 |
| cpaelzer | PCatinean: so I'm not saying that alone | 09:09 |
| cpaelzer | PCatinean: so far you only know that someone was able to connect to your SMTP - "if and which" authentication you set in front of that is up to you as SMTP itself doesn't imply one | 09:10 |
| cpaelzer | PCatinean: https://en.wikipedia.org/wiki/SMTP_Authentication for basics, you have to find what of that or similar your setup uses | 09:11 |
| cpaelzer | PCatinean: and then you are right to assume that somebody went past that | 09:11 |
| PCatinean | hmm hmm hmm | 09:11 |
| cpaelzer | PCatinean: the reported IP could also be one path for you to take a look | 09:15 |
| cpaelzer | PCatinean: while it might be likely that this wasn't the origin, but just an hop in between | 09:16 |
| cpaelzer | PCatinean: it might still be worth to send a mail to the owner so they can check - maybe they are compromised as well | 09:16 |
| PCatinean | pfuuu this is complex stuff when you have no idea | 09:16 |
| cpaelzer | PCatinean: "seems" to be from http://www.accelerated.de/en/ and they do hosting | 09:17 |
| cpaelzer | PCatinean: so maybe just one of the systems they host is compromised as well | 09:17 |
| cpaelzer | PCatinean: yet I'd consider it nice if you let them know about it | 09:17 |
| cpaelzer | PCatinean: you can (more or less) check IPs with e.g. https://www.whoismyisp.org/ip/82.211.31.232 | 09:18 |
| cpaelzer | PCatinean: but then be aware that mostly you only finde the ISP with that - stil lin this case it seems to point at a particula data center service provider which is good | 09:18 |
| cpaelzer | PCatinean: as the have likely personal to deal with such stuff | 09:18 |
| cpaelzer | PCatinean: on their end at least | 09:18 |
| cpaelzer | PCatinean: you still have to find how they passed your security/auth but that depends too much on what you actually had set up | 09:19 |
| cpaelzer | PCatinean: I talked to a few more people and - while I don't know if that is an option for you - other than taking it just off the network we weould have a few more hints | 09:27 |
| cpaelzer | PCatinean: that would be - before you start looking around on that system clone off the disks (in case the rootkit or whatever removes itself to not be found you can go back and try again) | 09:28 |
| cpaelzer | PCatinean: also any analysis should be done from "the outside" so not on that system booted, but from another system mounting the disks | 09:28 |
| cpaelzer | PCatinean: otherwise the rootkits can hide themselve too good | 09:28 |
| cpaelzer | PCatinean: but as I said, I don't know if that is an option for your hosted environment | 09:29 |
| PCatinean | cpaelzer, this is eeary | 09:31 |
| PCatinean | http://hastebin.com/tamuhijoji.rb | 09:31 |
| PCatinean | I think the test email username has been hijacked? | 09:32 |
| PCatinean | This is the first time things started acting up in the logs | 09:32 |
| cpaelzer | PCatinean: another hint from some friends http://www.gfi.com/blog/top-20-free-digital-forensic-investigation-tools-for-sysadmins/ | 09:36 |
| cpaelzer | PCatinean: but almost all efforts start with cloning and taking it offline | 09:36 |
| cpaelzer | PCatinean: yeah that looks like the auth for the test account | 09:37 |
| cpaelzer | PCatinean: is that still in the default setup? | 09:37 |
| PCatinean | Can't take it offline as it is a production server :( | 09:37 |
| cpaelzer | PCatinean: that might be the most likely issue | 09:37 |
| PCatinean | The default setup of postfix? yes | 09:37 |
| PCatinean | and ispconfig | 09:37 |
| cpaelzer | PCatinean: well than IMO you WILL take it offline at some day - there is no way keeping a compromised system in the long run | 09:37 |
| cpaelzer | PCatinean: you never know "what else" might be in there | 09:38 |
| PCatinean | Well I can make a fresh install in the weekend I guess and config everything from scratch | 09:38 |
| PCatinean | Just I need to know what went wrong and fix it the next time | 09:38 |
| cpaelzer | PCatinean: but do it on a different system if you can, so you have time to analyze this one more thoroughly | 09:39 |
| cpaelzer | PCatinean: follow all of this and likely more http://serverfault.com/questions/644219/postfix-and-compromised-accounts | 09:40 |
| cpaelzer | PCatinean: but I guess you already did the first time - at least you have amavis and such | 09:40 |
| PCatinean | Not sure if I want to wipe the entire system because of something like for example: having test@site.com with a weak password | 09:42 |
| cpaelzer | PCatinean: if you are sure it was just that it - after all it is your call to make anyway | 09:42 |
| PCatinean | I can never be sure but the evidence tends to point at this | 09:45 |
| peetaur | Hi, when trying to do "apt-get update" using either de.archive.ubuntu.com or us.archive.ubuntu.com (didn't try others), I get this error: W: Failed to fetch http://de.archive.ubuntu.com/ubuntu/dists/precise-updates/main/binary-i386/Packages Hash Sum mismatch | 10:01 |
| peetaur | seems to happen on any server. So ... what can I do to get it fixed? | 10:01 |
| peetaur | only affects precise, not trusty. | 10:02 |
| peetaur | and not just main repo, but also universe | 10:02 |
| PCatinean | Is this bad? smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated defer_unauth_destination | 10:09 |
| peetaur | also not sure if related... but I get some bzip2 errors also, also only for precise, not trusty... but if I delete these repos from my apt-cacher-ng server, the errors go away https://bpaste.net/show/6ae2b0ac1497 | 10:10 |
| peetaur | (and the first error happens with or without using apt-cacher-ng) | 10:11 |
| lordievader | Good morning. | 10:47 |
| MacroMan | Wanted to check that calling debchange -nmu on a package that I am building will prevent apt-get from overwriting my installed package? | 12:22 |
| ikonia | I wouldn't say thats the best way to do it | 12:25 |
| ikonia | I'd look at pinning it pre-install | 12:25 |
| ikonia | or changing the package name to significy it's your custom version assuming nothing depends on it | 12:25 |
| MacroMan | ikonia, Thanks. So I'd use debchange --create for that ? | 12:27 |
| ikonia | doesn't that just create a diff | 12:28 |
| MacroMan | Efectively creating my own unique package | 12:28 |
| rbasak | If the package in the archive is version 1.0-1, I usually use 1.0-1.1~local1 or something like that. Then any security or bugfix updates will trump my local version, but I can set up a package hold to avoidthat if I wish. | 12:30 |
| * MacroMan is reading more docs on debchange | 12:30 | |
| rbasak | MacroMan: you want to focus on how apt works and how Debian version strings are compared (defined in Debian policy, test with dpkg --compare-versions) | 12:30 |
| MacroMan | rbasak, Cool thanks. | 12:31 |
| rbasak | MacroMan: debchange is really for the common cases only. It just changes the version string in certain preset ways. | 12:31 |
| rbasak | Maintainers can always trump debchange by changing the version string manually | 12:31 |
| MacroMan | OK. Well my use case is adding a config string onto nginx and repackaging for install. I really just want apt to not overwrite my package and let me upgrade it manually. | 12:32 |
| MacroMan | My use it important that it remains my version and not the maintainers, even for security as my setup completely relies on my change. | 12:32 |
| rbasak | Is this for HTTP/2 support? | 12:32 |
| MacroMan | No, image_filter. | 12:33 |
| MacroMan | I'm already using their repo which has http/2 enabled. | 12:33 |
| MacroMan | This for an image server only and so if image_filter stops working, so will all my image resizing | 12:34 |
| rbasak | Say on Trusty, where nginx is 1.4.6-1ubuntu3. I would rebuild as 1.4.6-1ubuntu3+local1. Then I'd add a package hold using dpkg --set-selections. | 12:35 |
| rbasak | I'm not absolutely sure this works as you want. Please check before using! | 12:35 |
| rbasak | Also, obviously you want to be keeping a very close eye on any nginx security notices. | 12:35 |
| === Lcawte is now known as Lcawte|Away | ||
| MacroMan | Presumibly calling `dpkg --install` will still allow me to install a package even with a hold? | 12:37 |
| MacroMan | It'll just prevent apt from upgrading it? | 12:37 |
| MacroMan | Found the answer. And yes it does. | 12:43 |
| MacroMan | rbasak, Thanks for that. It's set me on the right track. | 12:43 |
| === quantic_ is now known as quantic | ||
| MacroMan | I'm looking to run something to test that one of my servers is online, but sending it a ping every second somehow seems absurd. Is there a better solution? | 14:41 |
| MacroMan | This is to run almost continuously for the next 2 weeks. | 14:41 |
| hateball | MacroMan: what's absurd about that? if you need to know it responds to ping every second, what else would you do? | 14:53 |
| hateball | Do you need to monitor if a certain service is running? | 14:53 |
| lordievader | Zabbix does precisely this to see if something is up. | 14:53 |
| hateball | Yes, or nagios or plenty of others | 14:55 |
| lordievader | Uhum | 14:55 |
| peetaur | MacroMan: why not ping? does it use too much bandwidth? :P | 15:00 |
| OerHeks | make a cronjob that pings every minute? | 15:01 |
| MacroMan | Just seemed a little absurd, but I suppose any keep-alive tool will do just that. | 15:02 |
| MacroMan | I'm basically having and argument with a data centre over connectivity dropping and them insisting that it's fine. So I want to run a continuous test to show it the connection goes down. | 15:03 |
| hateball | oh they left | 15:07 |
| peetaur | "failure is fine" :D | 15:09 |
| hateball | I was going to suggest them to install smokeping | 15:10 |
| hateball | produces nice graphs they could hand over | 15:10 |
| === roaksoax is now known as roaksoax-brb | ||
| jpastore | Can I get a recommendation on where to place a drop rule? the ufw rules files seem a little confusing for something that's supposed to be uncomplicated. basically I want to drop anything not allowed | 16:08 |
| thebwt | if that were a rule, it should be last. But give me a second, there should be a way to make that default (if it isn't already) | 16:11 |
| jpastore | thebwt, I agree it should be the last rule. I think it should go in ufw-after-input chain in the after.rules file is that correct? | 16:12 |
| === Lcawte|Away is now known as Lcawte | ||
| thebwt | ohhh you mean direct iptables manipulations. Yes that sounds correct | 16:13 |
| jpastore | well I'm modifying the ufw config file for persistence. | 16:14 |
| thebwt | gotcha, I thought you where messing with the 'ufw default deny INPUT' type thing. I've not done it that way. | 16:15 |
| jpastore | so if I add to the /etc/ufw/after.rules as the last line before the commit: -A ufw-after-input -i p5p1 -j DROP it shoudl drop anything not matched prior right? | 16:15 |
| jpastore | I know p5p1 is weird. | 16:15 |
| thebwt | yes, but it also skips the ufw-reject-input, ufw-after-logging-input, and ufw-track-input chains | 16:16 |
| jpastore | wouldn't that be better in the case of some type of a DoS attack? by rejecting and/or logging, it will cause more of a problem. is it not better to just drop off problems? | 16:17 |
| thebwt | I mean, if that's the objective. | 16:19 |
| thebwt | not enough context to say, my infra usually sits under load balancer, so only the ports allowed pass through | 16:20 |
| thebwt | if it's a DoS to some non http protocol, then yea that would help. But usually they're going to be over the stuff you've already allowed | 16:20 |
| thebwt | and if it's a ddos (not a just a Dos like you said) iptables ain't gonna save you | 16:21 |
| jpastore | thebwt, fair enough. thanks for the input | 16:34 |
| jpastore | I feel like ufw is far more complicated than iptables solo. | 16:35 |
| ^King | How to check if http is installed on my ubuntu vps? | 16:48 |
| ^King | via ssh/putty | 16:48 |
| rbasak | nacc: https://bugs.launchpad.net/ubuntu/+source/php5 | 16:53 |
| pmatulis | stgraber, hallyn: are there iptables issues with using LXD (especially in regards to using it with Juju)? can iptables rules get in the way somehow? | 16:53 |
| rbasak | nacc: https://bugs.launchpad.net/ubuntu/+source/php5/+bug/1315888 | 16:55 |
| ubottu | Launchpad bug 1315888 in php5 (Ubuntu Trusty) "Zlib functions (gzopen etc.) are undefined while gzopen64 etc. exist" [High,Confirmed] | 16:55 |
| rbasak | https://bugs.launchpad.net/ubuntu/+source/php5/+bug/74647 | 16:56 |
| ubottu | Launchpad bug 74647 in php "php5-gd not using bundled GD library" [Undecided,Fix released] | 16:56 |
| rbasak | https://bugs.launchpad.net/ubuntu/+source/php5/+bug/1184252 | 16:57 |
| ubottu | Launchpad bug 1184252 in php5 (Ubuntu) "php5: non-free files in upstream tarball ("The Software shall be used for Good, not Evil")" [High,Fix released] | 16:57 |
| rbasak | https://bugs.launchpad.net/ubuntu/+source/php5/+bug/1069529 | 16:57 |
| ubottu | Launchpad bug 1069529 in php5 (Ubuntu Raring) "Regression in system fallback for date_default_timezone_get()" [Medium,Fix released] | 16:57 |
| === Piper-Off is now known as Monthrect | ||
| coreycb` | zul, can you sync python-os-win 0.1.1-1 from experimental | 18:14 |
| zul | coreycb`: 0.0.6-1ubuntu1 is in -proposed | 18:15 |
| coreycb` | zul, it's ok for a sync, debian has py3 support now | 18:15 |
| lucidguy | need to upgrade a 10.04 server to 12.04. Can't seem to accomplish this via apt etc.. guess due to eof. Any predicitons to how successful I'll be using a 12.04 ISO instead? | 19:17 |
| sarnold | do-release-upgrade ought to do it | 19:18 |
| lucidguy | sarnold: I doesnt, you get erorr 404 on some of the repos. Tried switchout out to archive ones, but still not luck. | 19:19 |
| patdk-wk | only if he enabled the archives repos for 10.04 | 19:19 |
| lucidguy | actually using old-releases.ubuntu.com etc | 19:20 |
| bekks | !eolupgrade | lucidguy | 19:23 |
| ubottu | lucidguy: End-Of-Life is the time when security updates and support for an Ubuntu release stop, see https://wiki.ubuntu.com/Releases for more information. Looking to upgrade from an EOL release? See https://help.ubuntu.com/community/EOLUpgrades | 19:23 |
| lucidguy | Yeah, followed those instructions, the install pukes | 19:24 |
| bekks | lucidguy: So your computer starts to smell? | 19:26 |
| bekks | lucidguy: Or do you get specific error messages instead? | 19:26 |
| gQuigs | where can I find what's blocking ceph 0.8.10 from the cloud archive? (or the cloud archive bug version of 1477174 or 1535278) | 20:14 |
| gQuigs | I'm really looking for when 0.8.11 would be released (or where I can track it) for the cloud archive for precise... | 20:15 |
| jpastore | hi, quick question, Is there a limit to the number of physical cores the stock kernel can support? someone told me it was 16. | 22:04 |
| jpastore | looking for info to substantiate that | 22:04 |
| shauno | jpastore: there is, but it's way past 16. try: grep NR_CPUS /boot/config-`uname -r` | 22:10 |
| jpastore | thank you | 22:12 |
| sarnold | jpastore: you could also use something like num_cpus kernel boot parameter, taskset(1) or cpuset(7) | 23:08 |
| jpastore | sarnold, well I'm trying to figure out how many cores postgres will support and if I need to do anything special to the kernel or postgres. like recompile with other switches...though I'm considering seeing if the intel primatives would be of benefit as well | 23:13 |
| sarnold | jpastore: the handful of google results that looked like they'd be worth interpreting (with a grain of salt) suggests postgresql scales at least through 64 cores well, but that your workload needs to be parallel enough for it to work -- postgresql will use one task per connection, so if you've just got four connections, that might use four cores, and leave the rest of the cores more or less idle | 23:17 |
| nacc | jpastore: do you actually mean physical cores? or do you mean logical cpus? | 23:26 |
| nacc | jpastore: iirc, NR_CPUS=256 in the ubuntu kernel (at least glancing at one of my systems) -- and that's the number of *logical* cpus the kernel supports | 23:26 |
| nacc | jpastore: so let's say you had HT2 on an Intel CPU, I believe that would mean you had 128 cores enumerable, aiui | 23:28 |
| nacc | jpastore: it then depends on how many sockets are in your physical machine, etc | 23:29 |
| nacc | the only way it would be 16, though is, if you had HT4, which I don't think is possible on any of the current generation of Intel CPUs (skylake) | 23:30 |
| ianorlin | I don't think there are multisocket skylake out yet or anything more than say a quad core for skylake as xeons only the E3 have even been anounced | 23:34 |
| === Lcawte is now known as Lcawte|Away | ||
| nacc | ianorlin: ah true | 23:48 |
| ianorlin | I don't think there are any with four threads per core | 23:49 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!