| hehehe | solved | 02:39 |
|---|---|---|
| cpaelzer | good morning | 05:30 |
| jnollette | howdy | 05:36 |
| lordievader | Good morning | 07:40 |
| TafThorne | morning | 07:41 |
| lordievader | Hey TafThorne, how are you doing? | 07:42 |
| TafThorne | TafThorne: OK thank you. Busy, busy, busy at work but that is better than having nothing to do. How are you? | 07:54 |
| lordievader | Doing good here :) Busy too. | 08:02 |
| cpaelzer | Hi TafThorne, nice to see you again | 08:15 |
| cpaelzer | the day we are not busy will be boring | 08:15 |
| cpaelzer | so embrace it the way it is :-) | 08:16 |
| TafThorne | TafThorne: Hi. I am often around, I just do not have much to say. | 08:20 |
| lordievader | TafThorne: You know, you don't need to mention your own name ;) | 08:50 |
| lordievader | s/name/nick | 08:50 |
| TafThorne | ops, that was meant to be a cpaelzer: :-D | 08:57 |
| TafThorne | TafThorne: Feels as if he should address statements about TafThorne in the third person to help him stand out. | 08:59 |
| cpaelzer | hehe | 09:03 |
| necrophcodr | I'm not sure if this is the right place or not, but I am looking for a way of reading a LOT of files and applying programmatic rules to them. Is there an application that is suitable for this task? | 09:05 |
| necrophcodr | We're talking millions of files ranging from few bytes to gigabytes. The rules I'm looking to apply are for determining wether the files contain specific sequences or strings matching regular expressions, but also on file metadata such as site, filetype even, and so on. | 09:07 |
| TafThorne | necrophcodr: grep will do that. | 09:14 |
| rbasak | powersj: that's great. Thanks! | 09:14 |
| TafThorne | necrophcodr: I am not saying it is fast and when you start talking about filesystems with multiple GB of data you begin to find other interesting issues. You might benifit from using GNU parallel to run your grep inside. | 09:15 |
| necrophcodr | TafThorne, grep won't do that unfortunately. Grep will go some of the way, but it will also take too long. | 09:15 |
| TafThorne | necrophcodr: it and sed will so that but it will take along time. If your list of files is static you can save some time by getting "cached" lists of the filesystems and diving up the jobs and so forth. | 09:17 |
| necrophcodr | I think it's about 200GB of mixed data, and it's currently taking around 6-8 hours, depending on the data. | 09:17 |
| necrophcodr | That's using yara for the file matching, but it's not doing a great job. | 09:17 |
| necrophcodr | The filelist is not static unfortunately. I could probably run some jobs in parallel, but grep is out the window. | 09:18 |
| necrophcodr | I guess technically I could use grep but the amount of work to have it match all of our hundreds of rules would take a long time to implement properly. | 09:19 |
| TafThorne | necrophcodr: At my old job, that would be a small filesystem. Buy a BlueArc or a HDS HNAS :-D More seriously though, is this all stored on a single linear access disk? Does the system doing the search (and edit?) have lots of RAM to store things in cache? Is the disk networked or directly connected via SATA, eSATA, USB? | 09:19 |
| necrophcodr | It's on a VM that's on a SAN system, and there's more than 100 of these servers with that kind of storage that I need to scan preferably in less than 5 hours per server. | 09:20 |
| necrophcodr | And as I mentioned, the file list is not static, nor is the content. | 09:20 |
| TafThorne | necrophcodr: Sounds like a horrible problem to solve. Can you scan the files on creation and keep a list of hits? Can you determine modified files since your last scan to make it so you just re-run your results? | 09:21 |
| necrophcodr | Those are all optimizations for later, I don't have a proper scanning system yet, which is what I'm looking for. | 09:22 |
| TafThorne | necrophcodr: The short answer is that I know nothing that will do that for you. A server with a 100GB of RAM and direct storage access should not take that long to crunch the data. You cannot always throw hardware at it though. Sorry I cannot be of more help. | 09:22 |
| necrophcodr | I'm okay with having to develop one myself as well, but I'd need some idea of what to use, and I'm a bit stumped atm. | 09:22 |
| necrophcodr | TafThorne, it's no problem, I'm glad I got a quick response! That's not usually the case on some of the irc channels i frequent. | 09:23 |
| necrophcodr | And I can surely determine modified files and so on, that's part of the plan as well, but it's not yet feasible to do. It might turn out to not be a feasible optimization anyway, I'm not sure. | 09:24 |
| necrophcodr | Oh, and determining the modification might not be possible with a date alone either, as the data can change but the metadata might be re-written to it's original state, so even the modified date could be reset. | 09:24 |
| TafThorne | necrophcodr: doodle? | 09:27 |
| necrophcodr | TafThorne, what do you mean? | 09:27 |
| TafThorne | necrophcodr: You are welcome to the quick response. Just happened to be a topic I had some scant knowledge of. | 09:27 |
| TafThorne | https://linux.die.net/man/1/doodle | 09:28 |
| TafThorne | necrophcodr: I did some Googling and it suggested you might want to do some doodle(ing). | 09:28 |
| necrophcodr | TafThorne, thanks, but it doesn't allow me to set up complex rules | 09:32 |
| necrophcodr | I'll look into what can be done to do complicated rule matching on individual files at high performance | 09:32 |
| lordievader | You might want to write some perl script for that. | 09:46 |
| lordievader | Perl is great at doing regex stuff. | 09:46 |
| TafThorne | necrophcodr: There are some possibilities in https://askubuntu.com/questions/29483/software-for-text-search-in-files too I think. | 09:47 |
| TafThorne | necrophcodr: doodled, the doodle daemon might do a lot of what you want https://linux.die.net/man/1/doodled Initial indexing would probably be a pain but things might be OK afer that. Although I can see the caviate that doodled " uses libfam and is thus limited to monitoring less than 1024 directories for changes" which might make it not suitable for you. Manually re-running the stnadard `doodle -b <path_to_index>` does not include such a lim | 10:52 |
| TafThorne | Looks like a helpful command. I'll probably make a lot of use of it. Nice when trying to help others leads to learning sometihng useful for yourself. | 10:53 |
| Raboo | If anyone here is a junior sysadmin in Stockholm that is looking to change job, PM me.. | 12:27 |
| hateball | :o | 12:28 |
| === JanC is now known as Guest43740 | ||
| === JanC_ is now known as JanC | ||
| === dpawlik is now known as danpawlik_ | ||
| === danpawlik_ is now known as dpawlik_ | ||
| === drab_ is now known as drab | ||
| docmur | Hey guys, I'm trying to setup logstash and filebeat, when I run filebeat I'm getting ERR Failed to publish events caused by: read tcp 192.168.154.155:49128->192.168.154.168:5443: i/o timeout | 15:31 |
| docmur | I have the iptable entry to forward 5443 to the server from the client | 15:32 |
| docmur | I've disabled my firewall to test | 15:32 |
| docmur | and this is set net.ipv4.ip_forward=1 | 15:32 |
| drab | docmur: have you tried to tcpdump at dst? do you see any traffic at all? | 15:46 |
| docmur | Doing that right nw :) | 15:46 |
| drab | :) | 15:46 |
| docmur | Yep the dest is getting what it needs | 15:48 |
| drab | ok, what about tcpdump on client? does it see any traffic back?\ | 15:48 |
| drab | it may be a problem of SNAT | 15:48 |
| drab | in fact, what does dst see the traffic as coming from? | 15:48 |
| docmur | IP 192.168.154.155.49148 > elk-master.5443 | 15:50 |
| docmur | and the port from the src is changing | 15:50 |
| drab | eeer, that makes no sense... I was gonna say, ip looks ok, but yeah if the return pkts go to the wrong port that is indeed not going to work | 15:52 |
| drab | can you pastebin your nat/fw rules? | 15:52 |
| docmur | Ha, I was just going to do that :P https://pastebin.com/1dK7ik76 | 15:52 |
| drab | but yeah, also tcpdump on the client would be good, just to see if it sees the return traffic | 15:53 |
| drab | (even if it ends up discarding it because of unmatching port) | 15:53 |
| docmur | Ya no traffic on the client :( at port 5443 at least | 15:54 |
| drab | you mean the filter is source port 5443? | 15:55 |
| docmur | ya tcpdump port 5443 on the client I'm sending the logs from | 15:55 |
| drab | what if you tcpdump for the dst host's ip? | 15:56 |
| docmur | I might see the issue actually | 15:56 |
| docmur | I think it's a host name configuraton issue :S | 15:57 |
| docmur | I just saw this | 15:57 |
| docmur | IP elk.domain.net.5443 > 192.168.154.155.49164: Flags [S.], seq 4047232613, ack 332584761, win 28960, options [mss 1460,sackOK,TS val 16123230 ecr 3165914291,nop,wscale 7], length 0 | 15:57 |
| docmur | it should come baack from elk-master | 15:57 |
| docmur | I thin kthe domain is mismatched | 15:57 |
| drab | mmmh, maybe, maybe not, if the ips are correct and there's no PTR verification or other TLS thing where the cn must match, then it shouldn't matter | 15:58 |
| drab | what if you run tcpdump without dns resolution? | 15:58 |
| docmur | ya that was't the issue | 15:59 |
| drab | so the client, the fw and the elk-master are 3 diff boxes? on 2 diff networks? | 15:59 |
| drab | can you share a little more about that pls | 15:59 |
| docmur | So the client is the server itself (192.168.154.155), the elk setup is on a VM (192.168.154.168), I'm doing the routing on the server to the VM | 16:00 |
| docmur | The firewall is off right now on the serfver, I'm just using the iptables | 16:01 |
| drab | the VM is on the same server? | 16:01 |
| docmur | yes | 16:01 |
| drab | so basically everything on the same hw box, correct? | 16:01 |
| docmur | It's network interface is routed via br0 | 16:01 |
| docmur | yes | 16:01 |
| drab | I'm confused, why do you need iptables? aren't the VM and the server on the same subnet? ie, can't they talk to each other? | 16:03 |
| drab | oh, I think I see a possible problem | 16:04 |
| docmur | There are other VM's that don't come into play that I'm routing to, so I added the rules to route the elk server ports. | 16:04 |
| drab | docmur: look at your iptables, the SNAT part, shouldn't the POSTROUTING have a -s 192... you have -d | 16:05 |
| drab | so the postrouting is not matching | 16:05 |
| drab | at least if I remmber my iptables right, which I may not | 16:06 |
| drab | don't mess with that stuff as often anymore | 16:06 |
| drab | docmur: you can verify with counters, just do a -L -v with iptables and see if the numbers are incrementing as traffic is fired off, they shouldn't if there's no match | 16:07 |
| drab | or just try to change to -s and see if it works :P | 16:07 |
| drab | also it's not --dport at that point, it's a source port | 16:08 |
| docmur | oh okay | 16:08 |
| docmur | changing it to -s didn't seem to work | 16:08 |
| drab | docmur: yeah chjange the --dport to --sport too | 16:09 |
| docmur | Just did that, trying it now | 16:10 |
| docmur | Didn't work :S | 16:12 |
| docmur | I might try the logstash forum | 16:13 |
| docmur | thanks for your help :) | 16:14 |
| drab | to be sure since it's all on the same server, the server isn't considering the elk router's ip as local, is it? | 16:14 |
| docmur | ip r gives | 16:14 |
| docmur | 192.168.154.0/24 dev br0 proto kernel scope link src 192.168.154.155 | 16:14 |
| docmur | so now | 16:14 |
| drab | that's it? no default route? | 16:15 |
| docmur | default via 149.56.240.254 dev eth0 onlink | 16:15 |
| docmur | My default is the external ip | 16:16 |
| drab | k | 16:16 |
| drab | do you have any other rules in the fw? | 16:16 |
| docmur | it's off right now actually just to be sure it's not cauing an issue | 16:16 |
| drab | so what I'd do to rule out any other problems is to just test with netcat | 16:18 |
| docmur | kk | 16:18 |
| drab | stop elk for a second, start netcat in listening mode on the server | 16:18 |
| drab | and fireoff the client | 16:18 |
| drab | using the same port on the server of course so that the firewall rules get tested | 16:18 |
| drab | and you can run iptables -L -v -t nat before and after running netcat | 16:18 |
| drab | and see if the counters have changes, ie if pkts went through those rules | 16:19 |
| docmur | doing that now | 16:19 |
| drab | btw I'm assuming that for other things connectivity with that box work just fine, correct? ie you can ssh to the VM from the server or something | 16:20 |
| docmur | I can ssh to it, telnet to port 5443, I can access it's webportal ,etc... | 16:20 |
| docmur | The pkt counter is going up, about once per 30 seconds | 16:26 |
| drab | tbh I'm still not really understanding why iptables is involved...with that routing table/net scenario a pkt for 192.168.154.168 will be routed through the bridge where the VM's interface is also listeinng on and it'll just pick it up | 16:35 |
| === Guest75213 is now known as med_ | ||
| docmur | I culd remove it but that problem doesn't go away. I have other VM's that are listening for an external port, which is why I have them in the first place | 16:42 |
| drab | oh, I see, I didn't get that, this is just a testing setup | 16:42 |
| docmur | Anyway, thanks for the help, I posted on the logstash forum | 16:43 |
| drab | k, let me know ifyou figure it out, I'm curious now :) | 16:45 |
| docmur | ya I'll totally post the solution :) | 16:45 |
| drab | if you can test with requests from an ip outside of that network, I think it may shed some light on it | 16:45 |
| drab | something I still have a feeling part of the issue is the contrived example | 16:45 |
| drab | the other thing you could do is to setup another VM on say 192.168.153 or whatever, and use that as a client | 16:46 |
| ahasenack | ah, lovely whitespace delta | 20:28 |
| ahasenack | Suggests: libnss-ldapd | libnss-ldap $ | 20:28 |
| ahasenack | (output of cat -vet) | 20:29 |
| ahasenack | and here I was scratching my head why a patch wasn't applying | 20:29 |
| ahasenack | - Suggests: libnss-ldapd | libnss-ldap | 20:29 |
| ahasenack | + Suggests: libnss-ldapd | libnss-ldap | 20:29 |
| sarnold | set list and set listchars=tab:\ \ ,trail:$ in ~/.vimrc can make those stand out | 20:31 |
| ahasenack | nice | 20:33 |
| hehehe | hi | 21:55 |
| hehehe | desktop wfi doez not work | 21:55 |
| hehehe | lol | 21:55 |
| hehehe | connects and nothing | 21:55 |
| hehehe | works on a phone | 21:56 |
| tomreyn | and that is an #ubuntu-servertopic because? | 21:57 |
| michr | Hey guys, quick question -- our VPS has been going down intermittently throughout the day for 5 minutes at a time | 22:50 |
| michr | Is there anything I should be looking for to track what's causing the outage | 22:50 |
| sarnold | logs on both endpoints? | 22:51 |
| sarnold | oh VPS not VPN.. uh.. | 22:51 |
| sarnold | logs on the VPS? :) | 22:51 |
| hehehe | fixthatshit.com | 22:52 |
| hehehe | :) | 22:52 |
| michr | I've looked at the logs and we're getting spikes of traffic, which is causing some of the requests to go into a queue, and that's what's pulling our server down | 22:52 |
| michr | I just haven't been able to track where the requests are coming from | 22:52 |
| hehehe | well beef up server | 22:53 |
| hehehe | or host with ddos protection | 22:53 |
| michr | I updated it to 4 cores from 2 | 22:53 |
| michr | it's still having outages, just not as frequently | 22:53 |
| hehehe | just see who is flooding you | 22:54 |
| hehehe | or its legit traffic? | 22:54 |
| michr | could it be possibly that we have malware or spyware that's doing this? | 22:54 |
| hehehe | nah | 22:54 |
| hehehe | check access.log | 22:54 |
| hehehe | and see | 22:54 |
| hehehe | stop inventing | 22:54 |
| sarnold | it's certainly possible that you've been compromised; maybe the provider would have network usage graphs that could indicate if you're joined a botnet or something similar | 22:56 |
| hehehe | seems like many dns servers under attack | 22:56 |
| hehehe | hehe | 22:56 |
| hehehe | had to switch to fucking google | 22:56 |
| hehehe | atm | 22:57 |
| michr | I'm checking the access.log now | 22:57 |
| sarnold | google's servers have the advantage of doing lookups for 250M users. that means whatever you want is probably already cached. | 22:57 |
| sarnold | hot dns servers are happy dns servers | 22:57 |
| hehehe | 58.6.115.42 was down | 22:57 |
| hehehe | and 43 | 22:57 |
| hehehe | most issues arent a hack | 22:58 |
| hehehe | its a bug | 22:58 |
| hehehe | 99% is bug 1% hack | 22:58 |
| ubottu | bug 1 in Ubuntu Malaysia LoCo Team "Microsoft has a majority market share" [Critical,In progress] https://launchpad.net/bugs/1 | 22:58 |
| hehehe | instant fixes of all | 22:58 |
| sarnold | michr: if there's nothing obvious in logs or hosting provider's usage graphs, you could fire up smokeping to make sure it's still online, collect netstat and similar stats periodically, and try to find patterns when it has trouble | 22:59 |
| hehehe | dude sarnold sometimesw people simply ddos | 22:59 |
| hehehe | or more visitors | 22:59 |
| hehehe | seems free dns servers are under attack | 23:01 |
| hehehe | :D | 23:01 |
| hehehe | comodo is up | 23:02 |
| trippeh | a site I operate periodically have 1-2 minute spikes every hour even ~64 cores + cloudflare doesnt fix ;) | 23:02 |
| hehehe | cloudfare is shit | 23:02 |
| hehehe | its just waf + cdn | 23:02 |
| hehehe | and as someone said servers rangers can be determined easily | 23:02 |
| hehehe | for ddos etc | 23:02 |
| trippeh | for most setups, yes | 23:03 |
| trippeh | people leak addresses everywhere | 23:03 |
| hehehe | you can use aws they got waf too now :D | 23:03 |
| hehehe | cheaper | 23:03 |
| trippeh | most ddosers never get to that stage however | 23:03 |
| hehehe | wtf | 23:03 |
| hehehe | why are they so dumb lol | 23:04 |
| hehehe | well in fact I met police today also not much smarter | 23:04 |
| hehehe | :) | 23:04 |
| hehehe | so ye its cool | 23:04 |
| hehehe | I think soon I will simply use ram only cd r os and keeppasx on air gapped phone | 23:05 |
| hehehe | dont have to worry about browser exploits etc :) | 23:05 |
| trippeh | we had to blacklist a ton of datacenter/VPS operators, so many abusive bots | 23:05 |
| michr | @sarnold, @hehehe thanks guys. Gonna try to see if I can figure out what the heck is going on here | 23:05 |
| trippeh | too bad about people rolling their own vpns | 23:05 |
| hehehe | michr: post access.log here | 23:05 |
| sarnold | michr: good luck | 23:05 |
| hehehe | also do u have fail2bank | 23:06 |
| hehehe | fail2ban | 23:06 |
| hehehe | and which firewall you use? | 23:06 |
| hehehe | did you check syslog and firewall log? | 23:06 |
| hehehe | I have to say I am pretty new to linux, just applying common sense | 23:07 |
| hehehe | trippeh: many people are angry | 23:09 |
| hehehe | and been passive agressive they do bots etc | 23:10 |
| hehehe | run scripts | 23:10 |
| hehehe | it will only get more and more | 23:10 |
| hehehe | who dont want to deal with some 0 days :D | 23:11 |
| hehehe | as I said before ram only no write access OS like tails or subgraph seems to be sufficient for most desktop users | 23:11 |
| hehehe | trippeh: I wonder whats going on with my box lol | 23:12 |
| hehehe | DST=224.0.0.1 LEN=32 TOS=0x00 PREC=0xC0 TTL=1 ID=1843 DF PROTO=2 blocked | 23:12 |
| hehehe | and without it dns resolving does not work lol | 23:12 |
| hehehe | emm | 23:12 |
| trippeh | thats just multicast, and you prob have mdns enabled | 23:13 |
| RoyK | just allow multicast in ufw | 23:13 |
| RoyK | it won't hurt you | 23:13 |
| hehehe | cool | 23:14 |
| hehehe | I usually allow 53 80 443 | 23:14 |
| hehehe | thats it :D | 23:14 |
| RoyK | well, multicast is a set of addresses on ipv4 and ipv6, not port numbers | 23:15 |
| RoyK | something like ufw allow to ff00::/8 | 23:17 |
| RoyK | and similar to 224.0.0.0/4 | 23:17 |
| RoyK | those won't cross a router boundary without igmp snooping | 23:17 |
| hehehe | ty | 23:18 |
| hehehe | sudo ufw allow out proto tcp to 224.0.0.1 + udp works now | 23:18 |
| hehehe | also folks any ideas how to figure site hidden api? | 23:18 |
| hehehe | my next project :) | 23:18 |
| RoyK | no idea - pretty vague question | 23:19 |
| hehehe | 1 moment and I will solidify it | 23:19 |
| hehehe | it seems to be client side js site | 23:21 |
| hehehe | http://www.gregreda.com/2015/02/15/web-scraping-finding-the-api/ | 23:25 |
| hehehe | :) | 23:25 |
| hehehe | well I guess I just to read output more carefully | 23:26 |
| hehehe | :D | 23:27 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!