| lordievader | Good morning | 06:05 |
|---|---|---|
| punkgeek | is there anyway to solve this problem? https://paste.ubuntu.com/p/XKBpjmcHMS/ | 07:32 |
| lordievader | You might want to check the mysql/mariadb error log to see what the actual error is. | 07:40 |
| punkgeek | now i have this error | 07:51 |
| punkgeek | https://paste.ubuntu.com/p/fmjmZkG2D5/ | 07:51 |
| data_ | Hi. I'm looking for a tool to get a server up and running repeatably. E.g. I have a clean LTS ubuntu server, which I want to hav A, B and C packages installed, X and Y users created, SSH keys installed, etc etc etc. Are tools like Ansible/Chef/Puppet what I'm looking for? They look very complicated - I'm looking for something very simple. Is my current approach of a bash script the best solution? Thanks for any help. | 11:50 |
| blackflow | data_: Start with Ansible. doesn't require any client/server setup, only ssh. | 11:52 |
| data_ | Ansible seemed really complicated | 11:53 |
| blackflow | simpler than writing a bash script which woul dhave to do all the logistics of checking whether files exist, packages are installed etc... | 11:54 |
| lordievader | Other automation tool you can look at is Puppet. | 11:54 |
| blackflow | which is way more complicated. | 11:55 |
| data_ | Any tool for editing ansible yml files you could recommend? | 11:57 |
| blackflow | any text editor | 11:57 |
| data_ | I guess I'm looking for something with auto-complete, discoverable modules and all that | 11:58 |
| blackflow | like, how complex do you think your set up will grow that you need all that? | 11:59 |
| data_ | well, not that complex, to be honest - and if this was just a local home server i wouldnt give it a second thought: php (composer), python (pip), postgres, apache, some users, some databases. basically ~20 lines of a bash script | 12:01 |
| data_ | but it feels "dirty" to do it that way - "there must be a better way!" | 12:02 |
| data_ | im thinking ansible would be more resistant to things going haywire in the middle of the script and doing lots of weird stuff | 12:03 |
| blackflow | well if it's just bootstrapping (imperative, no state), then I suppose a shell script is better. | 12:03 |
| blackflow | data_: and the key to that is writing idempotent rules | 12:03 |
| blackflow | and those require state checking so something like Ansible is really the best thing to do. | 12:03 |
| data_ | right, because you tell ansible what you want the end result to be, not what to do (i.e. creating databases) | 12:04 |
| blackflow | more or less, yeah. | 12:06 |
| blackflow | primarily it is like that yes. you describe a state. but you also describe handlers that trigger when state changes and those are, for example, places where idempotency can break if you're not careful. | 12:06 |
| data_ | blackflow: here's a an example script of what i'm talking about: https://gist.github.com/iivmok/2c6224fa33a4c4ac51207fa601017514 | 12:09 |
| blackflow | data_: well if that works for you.... and I'm totally not gonna say anything about running that php -r thing and fetching files to exec, as root :) | 12:14 |
| data_ | that's from composers installation guide, and it does check the hash of the downloaded script, so i think it should be safe from MITM | 12:16 |
| lordievader | The download of the script can't be MITM'ed? | 12:22 |
| blackflow | data_: you'll also need set -e in that script so it exits when a command fails. see, logistic requirements start to pile up if you want it done properly. just use Ansible ;) | 12:24 |
| data_ | lordievader: if understand that line right, it removes the composer-install file if it fails a hash check | 12:26 |
| lordievader | data_: No, you are saying that because the script checks the hash of a downloaded file you should be fine against MITMs. But what if the script has been tempered with while you where downloading through a MITM attack? (I know very theoretical). | 12:27 |
| blackflow | btw why are you using php for things like unlink.... | 12:28 |
| blackflow | this it TheDailyWTF material :) | 12:28 |
| data_ | blackflow: as I said, that's from the composer installation docs - theres a link in the comment there | 12:29 |
| data_ | here https://getcomposer.org/download/ | 12:29 |
| blackflow | that's so much full of fail..... | 12:30 |
| data_ | should've used the script from here: https://getcomposer.org/doc/faqs/how-to-install-composer-programmatically.md | 12:30 |
| data_ | but that's beside the point | 12:30 |
| data_ | blackflow: eh. your perl is my python is their php. i mean sure, rm would be more portable, but i don't remember a builtin md5 tool in *nix | 12:31 |
| blackflow | well, not blindly running stuff found online is one of the steps toward proper server maintenance :) Have to be harh. Unfortunately we live in the age of IT pwnage, massive data breaches, infosec-pocalypse :) | 12:31 |
| blackflow | *harsh | 12:32 |
| blackflow | md5sum | 12:32 |
| blackflow | also sha384 sum, if you really want that one. | 12:32 |
| blackflow | but anyway... what I would do there, with Ansible, is prepare all such files in the config repo itself, not download ad-hoc on the machines. | 12:33 |
| data_ | blackflow: why blindly? it does check for the hash, and i mean, it has to start somewhere. if its not in the distros ppa, its all as unsafe | 12:33 |
| blackflow | beside better security, you also make sure that all the machines are running exactly the same setup procedure. | 12:33 |
| data_ | any npm/pypa/packagist package could be compromised | 12:34 |
| blackflow | technically so can distro repos | 12:35 |
| blackflow | but really, that's why you downlaod those files yourself, chekc them, test them, and upload to the server for use. with Ansible, that's very easy to do. | 12:35 |
| RoyK | blackflow: md5 has been cracked years ago | 12:54 |
| blackflow | RoyK: that's why we use sha, yes. | 12:55 |
| blackflow | (256 or better) | 12:55 |
| RoyK | sh512 is good - and faster than 256 on 64bit machines | 12:56 |
| blackflow | though I have yet to see a viable misuse of md5 for checksum validation. that's quite different from crypto where all the collisions make md5 very bad choice for, say, password hashing. | 12:59 |
| === RoyK is now known as Guest82073 | ||
| === RoyK^ is now known as RoyK | ||
| DirtyCajun | my nfsd is running 16 threads on a 16 thread system and i am sitting at a 18 load often. Should i increase the threads or decrease the threads to remedy? | 15:35 |
| compdoc | 18% cpu use, or aht? | 15:37 |
| compdoc | what | 15:37 |
| DirtyCajun | nono. 18 load on a 16 core server | 15:38 |
| DirtyCajun | so about 120% load | 15:38 |
| compdoc | whew | 15:38 |
| DirtyCajun | but low cpu/ram utilization. its all nfsd waiting to do stuff | 15:38 |
| DirtyCajun | pretty low iowait too | 15:38 |
| dlloyd | where is the time spent if iowait is low? | 15:39 |
| DirtyCajun | its all nfs threads waiting | 15:40 |
| DirtyCajun | ps -e v shows the stat for all the threads as D | 15:40 |
| DirtyCajun | and thats uninterruptable sleep. man page says (usually IO) but io_wait is low like i said before | 15:42 |
| dlloyd | then there doesn't seem to really be an issue? loadavg is a composite metric that doesn't necessarily correlate to an 'overloaded' system | 15:42 |
| DirtyCajun | you dont see an issue with a loadavg metric above the threadcount of your system? | 15:43 |
| DirtyCajun | ive never heard that opinion in my 20 years in the industry (not saying that agressively just saying)( | 15:44 |
| blackflow | DirtyCajun: btw is it 16 thread or 16 core system? | 15:47 |
| dlloyd | it entirely depends on the workload. unless there are metrics indicating either processing delays or latency in handling requests, i would say that its not actionable. without knowing more about the precise state of the threads, i would say if you want to reduce the load average, reduce the count of threads. no idea what impact that will have on the performance | 15:47 |
| DirtyCajun | 2 4core 8thread systems | 15:47 |
| DirtyCajun | s/systems/system | 15:47 |
| blackflow | 2 x 4 core HT cpus, right? | 15:48 |
| DirtyCajun | dlloyd, i would agree with you and that was my initial thought, but everything online seems to say to increase which is why i hopped on here in the first place haha | 15:48 |
| DirtyCajun | blackflow, yes | 15:48 |
| dlloyd | state D is counted against 'running' in load average calculation, but isn't necessarily consuming actual cpu time | 15:48 |
| dlloyd | right, i was saying if you want to game load average down vs handling more throughput/connections | 15:49 |
| RoyK | DirtyCajun: how much of the load is I/O wait? | 16:20 |
| RoyK | DirtyCajun: sar/systat or top or whatever will tell - I guess it's most of it | 16:20 |
| DirtyCajun | So after upping the threads to 64, it looks like the load has gone down to 10ish out of 16 | 16:54 |
| DirtyCajun | i do see that context switches are really high (ctx_sw). LIke 100K of them | 16:55 |
| steve3245 | Anyone running critical production web-facing servers....do you use ufw or manually set rules with iptables directly? other? | 18:13 |
| steve3245 | **for establishing host firewall rules | 18:13 |
| _KaszpiR_ | steve3245 firewalld | 18:17 |
| steve3245 | _KaszpiR_, from fedora-land? | 18:17 |
| _KaszpiR_ | or plain iptables or ferm | 18:17 |
| steve3245 | _KaszpiR_, do you administer ubuntu and fedora-derived servers? | 18:18 |
| _KaszpiR_ | both | 18:18 |
| _KaszpiR_ | (well it's centos, not fedora but close) | 18:18 |
| steve3245 | (a second question i'm looking into now is if i can develop a unified approach to firewalling centos/debian/ubuntu machines) | 18:18 |
| steve3245 | right | 18:18 |
| steve3245 | i like just using plain IPtables on centos | 18:19 |
| whislock | firewalld. | 18:19 |
| steve3245 | but i didn't know if that is a kind of lame hack for ubuntu best practices | 18:19 |
| _KaszpiR_ | at first I was not happy with firewalld till I just beat the shit out of it and started to like it | 18:19 |
| whislock | Honestly, unless you're extremely good at firewalls, you shouldn't be mucking with iptables directly. | 18:19 |
| _KaszpiR_ | there are some pretty decent things in it | 18:19 |
| whislock | I know a lot of people will disagree with me, but I've seen a lot of large security holes left open or created because people wanted to directly manipulate iptables. | 18:19 |
| steve3245 | I'm no expert but what i like about plain IPtables is i just basically set default DENY on incoming, and then add very restrictive incoming allow rules | 18:20 |
| _KaszpiR_ | do you hhave WAF (web app firewall wich actually filters out and validates requests before they reach application)? | 18:20 |
| steve3245 | not at this time | 18:21 |
| steve3245 | possibly in the future but we are kind of a small shop | 18:21 |
| _KaszpiR_ | then don't spend too much time on firewall | 18:21 |
| steve3245 | most of this stuff is just LAMP type servers etc | 18:21 |
| _KaszpiR_ | most attacks are just on app layer itself | 18:21 |
| _KaszpiR_ | such as exploiting app vunerabilities | 18:22 |
| steve3245 | i hear you - we got owned recently due to web devs not keeping drupal updated | 18:22 |
| steve3245 | ~10 compromised sites | 18:22 |
| _KaszpiR_ | where's your firewall now? :D | 18:22 |
| steve3245 | haha | 18:23 |
| _KaszpiR_ | ssure it's good practice to fliter unwanted stuff as close to the source as possible | 18:23 |
| steve3245 | Yes I think they were all "drupalgeddon" attacks....it looks like all it took was someone stuffing some bogus strings into a URL and they had write permissions to the webroot dir sigh | 18:24 |
| steve3245 | I will look into that | 18:24 |
| steve3245 | is that some kind of hardware appliance you run? | 18:24 |
| _KaszpiR_ | I'd rather focus on processing logs and adding something like fail2ban on too many errors for given app | 18:24 |
| steve3245 | yes i'm definitely hoping to get fail2ban going on these soon | 18:25 |
| steve3245 | i inhereted a fair amount of kind of not so well kept servers | 18:25 |
| steve3245 | and just trying to tidy them up a bit | 18:25 |
| steve3245 | do you know if fail2ban directly depends on my using any particular firewall? | 18:26 |
| steve3245 | (related to my earlier question about ufw/iptables/firewalld) | 18:26 |
| _KaszpiR_ | otherwise just making very strict rules for web servers of what is acceptable by the app (this may be tricky but usually boils down to what kind of requests are allowed, what vars are allowed with what kind of type, for example page can be number only) - usually iuts just a lines and lines of regexp rules ;) | 18:26 |
| _KaszpiR_ | fail2ban can run with different stuff, by default it talks with iptables by creating dedicated rules which it manages | 18:27 |
| whislock | Consider snort/suricata. | 18:28 |
| steve3245 | yes i'm looking into running snort also to keep eyes on network stuff | 18:28 |
| steve3245 | what is flying around the network | 18:28 |
| steve3245 | do you need realy fast hardware to run that? | 18:28 |
| whislock | No. | 18:28 |
| _KaszpiR_ | nope | 18:28 |
| steve3245 | it sounded like it needed a dedicated bare metal fast CPU host | 18:28 |
| whislock | But it's not just network stuff. IDS can alert/IPS can mitigate some app exploits. | 18:28 |
| steve3245 | oh awesome | 18:28 |
| whislock | Depends on the engine, the rules its fed, etc. | 18:29 |
| steve3245 | yeah i'd love that....it sounded like it could inspect packets and notice "bad" signatures flying around and alert in real time? | 18:29 |
| steve3245 | like "hey this tcp header looks like you might have some ransomware stuff in the network"? | 18:29 |
| whislock | It's more interested in the content than in TCP. | 18:30 |
| steve3245 | great - are you using snort now in production | 18:30 |
| whislock | "This request coming into your web server looks like a SQL injection attack" kind of idea. | 18:31 |
| steve3245 | oh awesome | 18:31 |
| steve3245 | yes after this recent drupal incident i'm hoping to be a little more proactive in the future | 18:31 |
| steve3245 | and not depending on the web devs to keep drupal updated | 18:31 |
| _KaszpiR_ | 'nah that the devs that made a typo' ;D | 18:31 |
| steve3245 | hey i'd rather have false alerts than not know :] | 18:32 |
| whislock | Snort is the most widely deployed IDS/IPS engine in the world. | 18:32 |
| whislock | It's maintained by Talos - Cisco's threat intelligence arm. | 18:32 |
| whislock | https://www.talosintelligence.com/snort | 18:33 |
| whislock | I was using it for my house, before my employer transitioned me to Palo Alto firewalls. | 18:33 |
| steve3245 | thanks I will check it out | 18:35 |
| steve3245 | appreciate the input whislock and _KaszpiR_ | 18:35 |
| _KaszpiR_ | there is one more thing you may consider | 18:36 |
| whislock | Zoom. | 18:36 |
| _KaszpiR_ | https://en.wikipedia.org/wiki/Open_Source_Tripwire | 18:36 |
| _KaszpiR_ | for validating files on host, so that you can detect if file signatures change | 18:37 |
| * trippeh_ trips | 18:37 | |
| trippeh_ | clear | 18:37 |
| trippeh_ | nm, I'm not tripwire on this IRC network it turns out | 18:38 |
| oerheks | info suckless-tools | 19:25 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!