| lotuspsychje | good morning to all | 01:18 |
|---|---|---|
| sarnold | morning lotuspsychje :) | 01:20 |
| lotuspsychje | hey sarnold | 01:20 |
| lotuspsychje | reboot after kernel first :p | 01:21 |
| sarnold | tomreyn: jfyi :) https://lists.ubuntu.com/archives/ubuntu-release/2019-August/004788.html | 01:22 |
| tomreyn | sarnold: thanks ;) i noticed it posted earlier in -release | 01:25 |
| sarnold | woot | 01:25 |
| tomreyn | i'm already testing live-server | 01:26 |
| sarnold | hahaha | 01:26 |
| sarnold | awesome :D thanks | 01:26 |
| tomreyn | 2 bugs so far, but nothing serious | 01:26 |
| lotuspsychje | morning tomreyn | 01:27 |
| tomreyn | good middle of the night, lotuspsychje | 01:27 |
| tomreyn | or morning, whichever you prefer :) | 01:27 |
| sarnold | good "I hope your air isn't still on fire" | 01:27 |
| lotuspsychje | hehe | 01:28 |
| * tomreyn waiting for 20 snaps to be installed after first boot (which were selected to be installed during installation) | 01:43 | |
| lotuspsychje | uh-oh, more affects on my bug #1838644 | 01:43 |
| ubot5 | bug 1838644 in linux-hwe (Ubuntu) "Booting into desktop results in flickering" [Undecided,Incomplete] https://launchpad.net/bugs/1838644 | 01:43 |
| tomreyn | carl fletcher also has a clevo | 01:49 |
| lotuspsychje | oh really | 01:51 |
| lotuspsychje | tnx for notice tomreyn | 01:51 |
| tomreyn | model: N7x0WU, (uefi) bios date: 06/11/2018 | 01:51 |
| lotuspsychje | N7x0WU v: 7.009 date: 05/14/2018 mine | 01:53 |
| tomreyn | same model even, just different bios | 01:54 |
| lotuspsychje | not sure its related but this came yesterday aswell tomreyn https://bugs.launchpad.net/bugs/1838818 | 02:04 |
| ubot5 | Ubuntu bug 1838818 in xorg (Ubuntu) "intel graphic" [Undecided,New] | 02:04 |
| OerHeks | Package: xorg (not installed) ... | 02:04 |
| tomreyn | lol | 02:05 |
| OerHeks | just wondered .. | 02:05 |
| tomreyn | it's also i386 and LXDE | 02:05 |
| tomreyn | or would be LXDE rather | 02:06 |
| TJ- | The source package is xorg, but there isn't a binary package of the same name | 02:06 |
| OerHeks | yes, and response on lotus bugreport is on KDE | 02:06 |
| lotuspsychje | yeah description isnt the best here lol | 02:06 |
| tomreyn | version.xserver-xorg-core: xserver-xorg-core 2:1.19.6-1ubuntu4.3 | 02:06 |
| sarnold | Uname: Linux 5.0.0-050000-generic i686 | 02:07 |
| TJ- | [ 0.169959] ** You are using 32-bit PTI on a 64-bit PCID-capable CPU. ** | 02:07 |
| lotuspsychje | oO | 02:07 |
| tomreyn | is that a mainline ppa kernel | 02:07 |
| lotuspsychje | tested something similiar yesterday | 02:08 |
| lotuspsychje | 5.0.0-050000rc1-generic | 02:08 |
| sarnold | status installed linux-image-5.0.0-050000-generic:i386 5.0.0-050000.201903032031 | 02:09 |
| sarnold | is that really a five month old build? or.. I'm very confused | 02:09 |
| lotuspsychje | not sure this user has it installed though? | 02:09 |
| lotuspsychje | i was helping bisect | 02:09 |
| lotuspsychje | *why | 02:09 |
| OerHeks | i hope not LINUXMINT-19.1/5.0.0-050000-GENERIC/I686 | 02:10 |
| lotuspsychje | heh | 02:11 |
| lotuspsychje | currently testing Linux Rootbox 5.2.0+ #23 with fastboot=0 and no flickering with the dev | 02:12 |
| TJ- | I can't see anything wrong in that report, Xorg.log looks fine | 02:13 |
| lotuspsychje | TJ-: didnt find intel errors in his dmesg neither | 02:14 |
| lotuspsychje | no underrun like mine | 02:14 |
| TJ- | lotuspsychje: I wonder if the actual issue is a login loop or similar | 02:15 |
| OerHeks | with me, it happened before boot, i could perform ctrl alt del to reboot. | 02:16 |
| lotuspsychje | yeah could be, his description is too vague | 02:16 |
| lotuspsychje | OerHeks: i only get in trouble on desktop after few sec, at gdm3 and suspend too | 02:17 |
| OerHeks | i did 2 things, boot with live iso, and fix filesystem that appeared not to have issues, then booting in recovery, dpkg option to fix packages. | 02:17 |
| OerHeks | i see no simular questions .. https://askubuntu.com/questions/tagged/boot | 02:27 |
| === guiverc2 is now known as guiverc | ||
| marcoagpinto | [11:42] <marcoagpinto> Buaaaaaaaa... I didn't go to work... I am depressed | 10:43 |
| lotuspsychje | drink a cola, it all will be better | 10:44 |
| daftykins | xD | 10:44 |
| marcoagpinto | lotuspsychje: I have already drank ~3 litres and no effec | 10:44 |
| marcoagpinto | effect* | 10:44 |
| daftykins | aside from you bouncing off the walls? | 10:44 |
| daftykins | ;) | 10:44 |
| lotuspsychje | oh, you can mix rum into it :p | 10:45 |
| lotuspsychje | cuba libre cola! | 10:45 |
| marcoagpinto | lotuspsychje: I can't drink alcohool because of the medicins I take | 10:45 |
| jeremy31 | lotuspsychje: I tired the 5.0.0-23 kernel and no flickering on this HP laptop | 11:24 |
| jeremy31 | tried | 11:24 |
| lotuspsychje | jeremy31: ok tnx for the test | 11:26 |
| lotuspsychje | jeremy31: might be related to clevo panel only? | 11:28 |
| jeremy31 | It could be because of the display used, can you connect to external monitor? | 11:29 |
| lotuspsychje | jeremy31: didnt test that yet | 11:29 |
| lotuspsychje | good idea | 11:30 |
| lotuspsychje | jeremy31: lol, this bug gets even weirder | 11:34 |
| lotuspsychje | when i type laptop screen gets black | 11:35 |
| lotuspsychje | and external screen on hdmi flickers | 11:36 |
| jeremy31 | can you switch to hdmi only? | 11:36 |
| lotuspsychje | when i use mouse, laptop screen is normal | 11:36 |
| lotuspsychje | hdmi only flickers the tv too | 11:38 |
| BluesKaj | Howdy folks | 11:47 |
| marcoagpinto | BluesKaj! Hello! | 12:34 |
| BluesKaj | hi marcoagpinto | 12:35 |
| marcoagpinto | BluesKaj: I haven't gone to my weekend job as I am depressed... I spent the whole week working on the thesis, day and night, and I am feeling terrible | 12:36 |
| marcoagpinto | :) | 12:36 |
| BluesKaj | have some cola :-) | 12:37 |
| marcoagpinto | last night I finished another revision | 12:37 |
| marcoagpinto | I already drank 3,5 litres | 12:37 |
| marcoagpinto | no effect | 12:37 |
| BluesKaj | no wonder you feel terrible, that's waaay too much | 12:37 |
| marcoagpinto | I know... it is so hard to deal with the pression | 12:38 |
| lotuspsychje | wb TJ- | 15:09 |
| lotuspsychje | tomreyn: 1838851 looks interesting heh, we might need a dmesg there | 15:44 |
| tomreyn | lotuspsychje: right thats the bug i was just looking at | 15:45 |
| lotuspsychje | oh nvm its in log.txt | 15:45 |
| tomreyn | there's mesg there but of 4.18 | 15:45 |
| tomreyn | *dmesg | 15:45 |
| lotuspsychje | Linux version 5.0.0-23-generic (buildd@lgw01-amd64-030) in his log.txt | 15:46 |
| tomreyn | its an amdgpu driven gpu though | 15:46 |
| lotuspsychje | lets have a look | 15:46 |
| tomreyn | oh right log.txt, i was looking at dmesg.txt | 15:48 |
| tomreyn | so this system went into suspend, then power cycled | 15:50 |
| lotuspsychje | yeah doesnt sound its related to mine | 15:50 |
| tomreyn | AMD Ryzen 5 2600 | 15:51 |
| lotuspsychje | !info linux-image-generic disco | 18:04 |
| ubot5 | linux-image-generic (source: linux-meta): Generic Linux kernel image. In component main, is optional. Version 5.0.0.23.24 (disco), package size 2 kB, installed size 15 kB | 18:04 |
| Kevin199 | Hiya | 18:46 |
| lotuspsychje | Kevin199: what are you looking for in an Os? | 18:46 |
| Kevin199 | I want it to be as fast as possible, I want to click 1 button for email, browser, etc | 18:47 |
| Kevin199 | Preferably run most my services in the terminal | 18:48 |
| tomreyn | Kevin199: i guess you can do this on either. | 18:55 |
| tomreyn | the major differences will be that ubuntu goes for more stability, less things breaking, is a more vendor supported / tested platform, whereas arch does not usually hold back on version upgrades, tries to always run the very latest of everything, which probably means more things will break. | 18:58 |
| tomreyn | personally, i mostly just want my desktop system to work reliably, don't always need the very latest software / features, and if i do i use !HWE and !PPA -s or build it myself. | 18:59 |
| tomreyn | and on servers i definitely want stability. | 18:59 |
| Kevin199 | I like stability too, I'm very new to Linux so I don't 100% understand the variety in distros | 19:00 |
| Kevin199 | There's so many of them and it's quite bizzare | 19:00 |
| tomreyn | so to me the tradeoff of going from ubuntu to arch is you get the latest versions faster, at the cost of having to fiddle more. | 19:01 |
| tomreyn | if you have backups, by all means try more distros, at least a rolling and a non rolling one, and maybe the large ones. | 19:02 |
| tomreyn | you can do it in a VM but it's not really the same as doing it as your primary OS on your primary computer | 19:03 |
| Kevin199 | I might try Arch in a VM once I figure out the installation process | 19:05 |
| daftykins | heh oh dear in at the deep end ;) | 19:07 |
| marcoagpinto | my dear beloved brothers! | 19:29 |
| wasanzy | Hello | 19:40 |
| marcoagpinto | hey hey | 19:40 |
| marcoagpinto | >:) | 19:40 |
| wasanzy | with the help of tomreyn yesterday I discovered the malware that was consuming the system's CPU at a high rate. In short, the info about the malware is here: https://www.virustotal.com/gui/file/ed3b7209ee905cc5a2a2b33f351511c895ea6913428536b9e162eb487a24528f/detection | 19:43 |
| wasanzy | I am not able to determine the attack vector yet. Running file /var/lib/postgresql/10/main/postgresq1 output this : https://paste.debian.net/1094265/ | 19:45 |
| tomreyn | hi wasanzy, did your clamav scan bring up any other files? | 19:47 |
| wasanzy | No | 19:48 |
| wasanzy | that was the only file it brought out | 19:48 |
| tomreyn | i think you had not yet an answer to my question of whether the postgrsql server may have been directly accessible from the internet - have you learnt anything about this situation since? | 19:49 |
| wasanzy | the postgresql server doesn't have direct access from the internet | 19:49 |
| tomreyn | so you have you checked your firewall configurations and concluded there are no rules / policies set that would allow for this? | 19:50 |
| tomreyn | i understand that it will have a LAN IP address and is therefore not immediately reachable form the internet, but you never know whether there was a freelancer who 'temporarily' 'needed direct access' so a port forwarding or other firewall rule was set up. | 19:52 |
| tomreyn | (thus check it) | 19:52 |
| wasanzy | yes, let me paste you the rules | 19:52 |
| tomreyn | https://www.postgresql-archive.org/posgresql-log-tp6021877p6021904.html - what we looked at yesterday - also has this statement: "I've also noticed there is a n596tx.so which is not a part of standard installation." you could search your file system for a file named like this: sudo find / -type f -name 'n596tx.so' | 19:55 |
| wasanzy | let me check that | 19:57 |
| tomreyn | you can also install 'debsums' (apt package) and have it list any files present on this system which deviate from files provided by debian packages at the same location: sudo debsums -sc | 19:58 |
| wasanzy | that file does not exist | 19:58 |
| wasanzy | have you taken a look at the firewall rules? | 20:00 |
| tomreyn | wasanzy: i missed the private message | 20:02 |
| tomreyn | looking now, but i'm not that good with iptables | 20:02 |
| tomreyn | oh this is simple enough, ok | 20:02 |
| wasanzy | ok | 20:02 |
| tomreyn | wasanzy: so this server did have a public ip address with no NAT or hardware firewall in front? | 20:03 |
| wasanzy | yes | 20:03 |
| TJ- | wasanzy: you can use "sudo dpkg --verify" to verify all package-installed files haven't been conpromised | 20:06 |
| TJ- | wasanzy: but as I said yesterday the best place to look is the apache web-server log files plus any the application running on apache creates. Knowing what web-applications are being used might help us figure it out | 20:07 |
| wasanzy | tomreyn: debsums -sc -r / return nothing | 20:08 |
| wasanzy | TJ-: OK am running that | 20:08 |
| wasanzy | I check the nginx logs actually | 20:08 |
| wasanzy | we don't run apache on the server | 20:09 |
| TJ- | wasanzy: oh, I thought it was apache. Same thing anyhow... figure out the time frame between which the infection must have occurred then look for clues between those times | 20:10 |
| TJ- | wasanzy: is the web-application something standard/open-source or custom ? | 20:10 |
| tomreyn | yes, we should discuss all of the web applicationS | 20:11 |
| wasanzy | TJ-: https://paste.debian.net/1094270/ that is output of dpkg --verify | 20:11 |
| wasanzy | The web application is custom build. they are java applications actually | 20:12 |
| TJ- | wasanzy: is nginx acting as a proxy ? | 20:13 |
| TJ- | wasanzy: most java web apps uses Tomcat or some J2EE container | 20:13 |
| tomreyn | i'd also be interested in the output of ls -l /var/lib/postgresql/10/main/ if you can share this | 20:14 |
| TJ- | wasanzy: you need to run that command as root (sudo) so it can access everything... but generally most changes under /etc/ will be local admin config changes... but its worth triple-checking | 20:14 |
| wasanzy | Nginx is serving as a proxy. the app is a stand-alone no tomcat | 20:15 |
| tomreyn | we could also search for all files changed since when the intrusion likely happened. we know the miner started yesterday, which means it must have been before then, but we do not yet know any more than that | 20:15 |
| wasanzy | https://paste.debian.net/1094271/ | 20:16 |
| wasanzy | tomreyn: miner started on the 1st of August, thus 3 days a go | 20:17 |
| tomreyn | thanks for clarifying | 20:18 |
| TJ- | wasanzy: so should be easier to track how it got added then | 20:18 |
| TJ- | wasanzy: I presume your web-application (Java) is using raw SQL statements in talking to Postgresql ? | 20:19 |
| wasanzy | unless I confirm from the developers | 20:20 |
| TJ- | wasanzy: so at some point I'm guessing that some front-facing public form input is not being sanitised and is able to add arbitrary SQL commands with a "; ..." - this would explain how/why the malware executes as the postgres user account | 20:20 |
| wasanzy | the only entry to the db to the best of my knowledge is user/password login on the app | 20:21 |
| tomreyn | wasanzy: please also show stat /var/lib/postgresql/10/main/postgresq1 and getfattr --dump /var/lib/postgresql/10/main/postgresq1 | 20:22 |
| tomreyn | getfattr is provided by the attr package | 20:24 |
| TJ- | wasanzy: I really wouldn't keep that server operating. I'd replace it with a known clean backup | 20:25 |
| TJ- | wasanzy: you can always keep the VM alive for forensics but right now you're risking whatever business/service relies on it | 20:26 |
| wasanzy | tomreyn" https://paste.debian.net/1094274/ | 20:26 |
| tomreyn | TJ-: i'm not sure this info got to you the other day, but root owned audit logs in /var/log/audit* were also deleted. so worst case the fact the miner was postgres owned could be an attempt to mislead. | 20:27 |
| wasanzy | TJ-*: The server is no more processing services, taken it off yesterday | 20:28 |
| tomreyn | wasanzy: i second this, you should really not boot this system anymore. you should only access the file system from a frehly installed ubuntu system. | 20:28 |
| wasanzy | tomreyn: I can only access the server remotely | 20:29 |
| TJ- | tomreyn: really? you're sure of that? | 20:29 |
| wasanzy | user.xdg.origin.url="http://207.148.118.183/post0120/post" could this mean this is the source of the malware? | 20:30 |
| TJ- | wasanzy: It's Linode I noticed, so you could access the VM through the Finnix recovery option | 20:30 |
| tomreyn | yeay | 20:30 |
| tomreyn | exactly | 20:30 |
| tomreyn | this server is down, though | 20:31 |
| wasanzy | tomreyn: were you referring to my question? | 20:31 |
| wasanzy | I mean the source of the malware question | 20:32 |
| tomreyn | wasanzy: yes, postgresq1 was downloaded from http://207.148.118.183/post0120/post using wget (unless this information was falsified) | 20:32 |
| tomreyn | another VULTR system. | 20:33 |
| wasanzy | which could mean the attacker got access to our system and used wget to download? | 20:34 |
| tomreyn | sure, that's the hypothesis i'm operating on since reading https://www.postgresql-archive.org/posgresql-log-td6021877.html yesterday | 20:35 |
| tomreyn | wasanzy: back to the point that you should not be booting this server anymore: we have to assume ti was rooted. after that, a rootkit may have been installed. you can no longer know what this system does, it may spread malware further in your network, and on the internet. it may have logged your login credentials when you logged in using ssh. and worse. | 20:37 |
| tomreyn | other approaches i can think of now is to see if you can use undeletion utilities to recover files from /var/tmp and /var/log | 20:41 |
| tomreyn | and to look for suid + sgid files across the entire file system | 20:41 |
| tomreyn | i mean across all the file systems | 20:42 |
| wasanzy | if what I pasted here is true: https://paste.debian.net/1094274/ then my case is the same as https://www.postgresql-archive.org/posgresql-log-td6021877.html | 20:43 |
| wasanzy | tomreyn: do you have any undeletion utility in mind? | 20:44 |
| wasanzy | note: I rebooted the system | 20:44 |
| tomreyn | which file system do you hav there? | 20:45 |
| tomreyn | extundelete for ext3/4 | 20:46 |
| tomreyn | or ext4magic | 20:46 |
| tomreyn | here, too, you introduce more problems by running the server from the original installation: you're overwriting data, making data recovery much less likely | 20:47 |
| tomreyn | whenever you do forensics, any possibly compromised storage locations must only be mounted read-only. | 20:48 |
| wasanzy | ext4 | 20:51 |
| TJ- | wasanzy: if it was me I'd create a local guest VM with the same release/architecture and packages as the Linode, then use rsync to copy from Linode all the *changed/different* files and log which files are transferred by rsync. Then I can test the identical set-up locally AND know which files need checking | 20:51 |
| tomreyn | sorry TJ, i somehow missed this, thought you were asking wasanzy for confirmation. what were you referring to here? <TJ-> tomreyn: really? you're sure of that? | 20:55 |
| TJ- | tomreyn: sure something deleted audit logs? | 20:55 |
| TJ- | tomreyn: because those same messages will be in the journald logs | 20:55 |
| tomreyn | that's what wasanzy implied | 20:55 |
| tomreyn | i think he said that audit logs are in /var/(log/audit* but those logs are missing for the very day the intrusion occurred. | 20:56 |
| tomreyn | if they're also in the systemd journal this should be investigated | 20:56 |
| wasanzy | how do I find them in the system journal? | 20:57 |
| tomreyn | journalctl is the utility used to access those logs | 20:58 |
| TJ- | wasanzy: which ubuntu release is it? | 20:58 |
| wasanzy | 18.04 | 20:58 |
| tomreyn | i think you said 18.04.2 fully up to date yesterday? | 20:59 |
| wasanzy | yes | 20:59 |
| tomreyn | journalctl --list-boots could be of interest to tell which uptime the system had | 20:59 |
| TJ- | wasanzy: try "journalctl -u auditd.service" | 20:59 |
| tomreyn | maybe also --verify to see if there's anything missing | 21:00 |
| TJ- | wasanzy: it could depend on how auditd configures its logging | 21:00 |
| wasanzy | Aug 01 18:47:34 minex360-linode-prod-1 systemd[1]: Stopping Security Auditing Service... | 21:02 |
| TJ- | well, I have to go out. It's almost midnight and we've just started combining the oil-seed rape | 21:02 |
| wasanzy | I think that was when I rebooted the server | 21:02 |
| wasanzy | thanks a lot TJ- | 21:02 |
| TJ- | "minex360-" ? ironic! | 21:03 |
| tomreyn | good luck on the old-seed rape. | 21:06 |
| tomreyn | brinmg flashlights | 21:06 |
| wasanzy | tomreyn: --verify is running | 21:08 |
| tomreyn | this can take a while. can you show the latest boots, too? | 21:09 |
| wasanzy | https://paste.debian.net/1094280/ -> ---verify | 21:13 |
| tomreyn | so they seem to be complete, or coherent | 21:15 |
| wasanzy | reboot system boot 4.15.0-55-generi Thu Aug 1 18:49 still running | 21:15 |
| tomreyn | wasanzy: that's just the latest boot. the ones before that would be more interesting | 21:17 |
| wasanzy | last reboot => that is the only entry returned | 21:19 |
| tomreyn | by --list-boots ? | 21:20 |
| tomreyn | what you posted lookd more like output of the "last" command | 21:22 |
| tomreyn | so sourced from /var/log/wtmp | 21:22 |
| tomreyn | but i'm asking about the output of journalctl --list-boots which would provide output in a different format | 21:22 |
| wasanzy | ext4magic -m -d /var/log | 21:24 |
| wasanzy | -1 36e9fc446ca743de91fc3aeafa76b2eb Mon 2019-07-29 12:00:01 UTC�<80><94>Thu 2019-08-01 18:47:34 UTC | 21:25 |
| wasanzy | 0 ad41338ef7424c76a7f1ba629918d6c9 Thu 2019-08-01 18:49:23 UTC�<80><94>Sat 2019-08-03 21:24:22 UTC | 21:25 |
| wasanzy | ext4magic -m -d /var/log => doesn't seem to be recovering anything | 21:26 |
| tomreyn | wasanzy: you did not specify the file system to operate on, see <filesystem> on the ext4magic man page | 21:29 |
| tomreyn | wasanzy: are those the only boots journalctl --list-boots reported, or the only ones you shared with us? | 21:29 |
| tomreyn | it'd help to get a bit more context from you in general. | 21:29 |
| wasanzy | the only ones reported | 21:30 |
| tomreyn | ok | 21:30 |
| tomreyn | wasanzy: can you share the full log from boot -1 using jounrctl -b -1 | nc termbin.com 9999 or, if that's not an option (I'd understand), can you share this: journalctl -b -1 | grep 'Linux version' | 21:33 |
| tomreyn | sorry, the first command was journalctl -b -1 | nc termbin.com 9999 | 21:33 |
| wasanzy | ext4magic /dev/sda3 -R -a $(date -d "-5day" +%s) => you think this could help? | 21:33 |
| wasanzy | jounrctl -b -1 | nc termbin.com 999 => no output | 21:35 |
| wasanzy | journalctl -b -1 | grep 'Linux version' => no output | 21:35 |
| tomreyn | wasanzy: you missed one 9 | 21:35 |
| wasanzy | journalctl -b -1 | nc termbin.com 9999 => no output | 21:36 |
| tomreyn | so this would post the log from Mon 2019-07-29 12:00:01 UTC till Thu 2019-08-01 18:47:34 UTC online: journalctl -b -1 | nc termbin.com 9999 | 21:36 |
| tomreyn | hmm weird | 21:37 |
| tomreyn | if you just run this? journalctl -b -1 | 21:37 |
| tomreyn | maybe some logs were actually deleted there, too. can you show ls -lahR /var/log/journal/ | nc termbin.com 9999 | 21:38 |
| wasanzy | journalctl -b -1 => print a lot of entries | 21:39 |
| wasanzy | https://termbin.com/0zw9 | 21:39 |
| tomreyn | wasanzy: the output of journalctl -b -1 may copntains logs on what the intruder did, and which they failed to delete. | 21:41 |
| wasanzy | ok I will look a that | 21:42 |
| tomreyn | the oldest system logs you still have are from Jul 29 18:24 | 21:42 |
| tomreyn | what does this say ? getent passwd postgres | 21:43 |
| wasanzy | am trying to run this: ext4magic /dev/sda -R -a $(date -d "-4day" +%s) | 21:43 |
| wasanzy | postgres:x:111:116:PostgreSQL administrator,,,:/var/lib/postgresql:/bin/bash | 21:44 |
| tomreyn | okay this looks normal | 21:46 |
| wasanzy | ok | 21:48 |
| wasanzy | ERROR: can not use "RECOVERDIR" for recover directory. It's the same filesystem : "/dev/sda" | 21:49 |
| tomreyn | right, this is an intelligent utility, it would not write data to the same storage it is trying to recover data from. | 21:49 |
| tomreyn | after all, thois could prevent recovery of this data | 21:50 |
| tomreyn | can you tell when this server was actually installed? the oldest journalctl logs date back to only 2019-06-29 | 21:50 |
| wasanzy | the server was installed before I even joined the organization last year | 21:51 |
| tomreyn | how much disk space is there available on /var/log? df -h /var/log | 21:58 |
| wasanzy | 117G | 21:59 |
| tomreyn | systemd will automatically delete old logs when it reaches certain storage thresholds. in your case, it reached the 4 GB fixed threshold, thus old system logs are gone | 22:01 |
| tomreyn | apparently this system was really logging a *lot* | 22:01 |
| tomreyn | more than ususal | 22:01 |
| wasanzy | am coming to boot into rescue mode | 22:02 |
| tomreyn | it created 10 log files each sized 128 MB *after compression* per day | 22:03 |
| wasanzy | right | 22:03 |
| tomreyn | this suggests something was not properly configured / misbehaving in the first place. | 22:03 |
| tomreyn | as you can surely see when looking at journalctl -b -1 -e | 22:04 |
| tomreyn | (-e goes to the end of the log period) | 22:04 |
| wasanzy | output starts from Aug 01 | 22:05 |
| tomreyn | i assume you can't share this log? | 22:07 |
| TJ- | whatever the custom java web-application is I'd expect it to be using the java.logger interface to do its own logging, which might be a better place to investigate | 22:09 |
| Ben64 | bleh, still no window controls in chrome | 22:12 |
| Ben64 | not sure what update broke it, but it's kinda annoying | 22:12 |
| tomreyn | TJ-: we do have audit logs via journalctl starting Aug 01, but i don't know how to work with them | 22:16 |
| tomreyn | (also this may well be too late) | 22:16 |
| TJ- | tomreyn: if the attack vector is the java web-app > postgresql raw SQL input, then I'd put more reliance on the java-app's own logs (if it does log!) | 22:17 |
| wasanzy | TJ-: Yea the java app has logs. I can see a lot of direct sql statements in the logs. "Select statements though" | 22:17 |
| tomreyn | selects would work | 22:17 |
| tomreyn | select * into outfile (i think this is mysql specific, not sure, but postgres may have something similar) | 22:18 |
| TJ- | wasanzy: I'd look for anything matching the regexp "..*;[^$]" (in other words, any statement that has a semi-colon NOT at the end of the line | 22:19 |
| TJ- | wasanzy: that'd mean the bit after the semi-colon was an additional statement, which if raw input is allowed, would be the obvious vector | 22:19 |
| tomreyn | that's %3B or %3b is url encoded | 22:19 |
| tomreyn | that's %3B or %3b IF url encoded | 22:19 |
| TJ- | wasanzy: the other possibility is the java application itself, and whatever user account it runs as, is compromised | 22:20 |
| TJ- | this assumes it's a high level attack via services and not a kernel-level compromise | 22:21 |
| wasanzy | zgrep "..*;[^$]" /path/to/logdir/* | grep -i "SELECT" | 22:24 |
| wasanzy | nothing yet | 22:26 |
| TJ- | wasanzy: I'm not guaranteeing my syntax was correct! | 22:28 |
| TJ- | seems to work in a test though: echo -e "select * from table where x=y;\nselect * from table where x=y;select input >/var/tmp/test" | grep '..*;[^$]' | 22:30 |
| wasanzy | yea I got some outputs but not much to think is a hack. looks like some exceptions | 22:34 |
| tomreyn | how about nginx logs, it would onl yhave GET requests logged though, and only if you did log succesfull requests. | 22:35 |
| wasanzy | I could look for post requests though | 22:37 |
| tomreyn | those wouldn't be logged by a web server normally | 22:38 |
| tomreyn | it appears that the "search" web app allows for self-registration, possibly providing extended access? | 22:40 |
| TJ- | ? did we find out what the web-application is? | 22:43 |
| tomreyn | i did find some, i think | 22:44 |
| tomreyn | but won't discuss it unless wasanzy wants me to | 22:44 |
| tomreyn | he passed me some syslog in private | 22:44 |
| wasanzy | TJ- and tomreyn: I shared paste in private with you two | 22:48 |
| wasanzy | Please study that and see if you can make any sense out of it | 22:49 |
| tomreyn | thats sshd, running as root, accessing /etc/passwd to verify a connecting user exists | 22:50 |
| tomreyn | ...i think | 22:50 |
| TJ- | wasanzy: ahhh, I have messages blocked | 22:50 |
| wasanzy | blocked? | 22:50 |
| wasanzy | you mean you blocked private messages? | 22:50 |
| TJ- | wasanzy: Yes | 22:50 |
| wasanzy | ok | 22:51 |
| tomreyn | a rather common setup actually ;) | 22:51 |
| TJ- | wasanzy: otherwise people think they can bother me for personal support constantly | 22:51 |
| wasanzy | ok | 22:52 |
| tomreyn | TJ-: did you receive those 5 lines from syslog since? they give away the server's public IP address, which you can then use to determine the application host names based on the SSL certificate on 443 | 22:52 |
| tomreyn | (HTTPS) | 22:53 |
| TJ- | tomreyn: no, and I have to go anyhow, we're working through the night | 22:53 |
| tomreyn | yes, you said so, ok, was just wondering | 22:53 |
| TJ- | I've been in and out as the loads arrive from the combine | 22:54 |
| tomreyn | cool | 22:54 |
| tomreyn | wasanzy: so i registered a new user on the 'search' web app, and was then able to connect a database to it. i connected the postgresql database on 127.0.0.1 (the server itself) , authenticating as the postgres user. tested the connection - it succeeded. | 22:56 |
| tomreyn | it doesn't seem possible to exfiltrate data this way directly, but this is probably not sound. | 22:56 |
| tomreyn | oh actually i can access all databases | 22:58 |
| tomreyn | and exfiltrate data | 22:58 |
| wasanzy | that is my server? | 22:58 |
| tomreyn | so i'll stop here | 22:58 |
| tomreyn | the one which is now running at the ip address listed as the ssh server destination in the latest log excerpt you shared | 22:59 |
| tomreyn | i assume this is the replacement production server you setup? | 22:59 |
| wasanzy | no | 23:01 |
| tomreyn | ahem, actually thios hostname points to a different server of the same organization. i'll send you details in private message. | 23:01 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!