| === Lcawte is now known as Lcawte|Away | ||
| delinquentme | hey all. I'm designing some analytical chemistry instruments on a linux system and I'm trying to sort out the best way to handle persisting / storage of 50 milisecond events -- each event is a really tiny numpy float16 | 00:16 |
|---|---|---|
| delinquentme | a friend suggested syslog ... but I cant help but think that buffering in memory ( not sure if syslog does this ) and then writing to a file | 00:17 |
| delinquentme | would be a better solution | 00:17 |
| === bilde2910 is now known as bilde2910|away | ||
| === markthomas is now known as markthomas|away | ||
| === zz_DenBeiren is now known as DenBeiren | ||
| === Lcawte|Away is now known as Lcawte | ||
| === blanoz is now known as Blanoz | ||
| === gblfxt_ is now known as gblfxt | ||
| Thorn | hello | 12:16 |
| Thorn | I've got a new 14.04 server where SSL doesn't work at all (server vert verification fails). can't git clone from github, pip install doesn't work, etc. | 12:17 |
| Thorn | update-ca-certificates --fresh didn't help | 12:18 |
| Thorn | any idea how it can be fixed? | 12:19 |
| spicypixel | Thorn: is it because openssl hasn't been updated? | 13:06 |
| Thorn | I ran aptitude update; aptitude upgrade, it didn't update openssl | 13:07 |
| Thorn | (this is a dedicated server from leaseweb so I didn't install ubuntu myself. apparently it was updated from a previous version, and there were other weird things that I fixed. maybe I should reinstall the OS using ILO) | 13:08 |
| devster31 | hi, can I avoid restarting the server after updating openssl package? this command cat /var/run/reboot-required.pkgs tells me it's the only package that's requesting a reboot but I'd avoid that if possible | 13:32 |
| devster31 | the package is libssl1.0.0 | 13:40 |
| Meerkat | restart apache. and anything else using openssl. | 13:43 |
| devster31 | and the warning goes away? | 13:45 |
| devster31 | I have nginx, i suppose it's the same | 13:45 |
| ikonia | devster31: if you want server security - restart the server | 14:15 |
| ikonia | devster31: if you don't take it seriously, try to work around it | 14:15 |
| ikonia | so many things are linked against ssl - even if they are not using them | 14:15 |
| devster31 | I don't take it very seriously, I run a very small server for 2 or 3 people without any kind of sensitive data, I'm making myself more comfortable with the admin tasks | 14:35 |
| === bilde2910|away is now known as bilde2910 | ||
| === DrStrangeRoy is now known as RoyK | ||
| === Lcawte is now known as Lcawte|Away | ||
| === Lcawte|Away is now known as Lcawte | ||
| === Blanoz is now known as blanoz | ||
| === blanoz is now known as Blanoz | ||
| OliPicard | Greetings everyone, I'm having problems with my Nginx instance. I have setup the correct SSL information however if the config is setup to listen to port 443 nginx refuses to bind to the port. Nothing else is binding to the same port and ive tried killing all nginx proceses to no avail. | 19:19 |
| TJ- | OliPicard: what error message do you get? to bind to a privileged port (port number < 1024) the process needs netadmin capabilities, or run as root | 19:20 |
| OliPicard | one second, i'll paste the error | 19:21 |
| OliPicard | https://www.digitalocean.com/community/questions/nginx-is-unable-to-bind-to-443 | 19:21 |
| RoyK | TJ-: nginx does that by default :P | 19:22 |
| OliPicard | id be happy to provide my nginx config as well. | 19:22 |
| OliPicard | https://gist.github.com/anonymous/cda414038cb8de41b86f <thats my config at the moment | 19:23 |
| TJ- | RoyK: Indeed... but if it is being launched under a non-privileged user ID, it won't be able to | 19:23 |
| RoyK | TJ-: well, if it can bind to port 80, it certainly can bind to port 443 | 19:23 |
| OliPicard | the master process is being run by root. | 19:24 |
| TJ- | OliPicard: What does it report in the log ? | 19:24 |
| OliPicard | root has full access perms | 19:24 |
| OliPicard | nginx: [emerg] bind() to 0.0.0.0:443 failed (98: Address already in use) | 19:24 |
| OliPicard | yet when i run a netstat it only shows nginx as running. | 19:25 |
| OliPicard | tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 3494/nginx | 19:25 |
| TJ- | OliPicard: The way I read the docs, if you want to bind for IPv4 and IPv6, you just need a single "listen [::]:443" without "ipv6only=on" and no separate "listen 443 ssl" for IPv4 | 19:31 |
| OliPicard | TJ-: tried it but still wont bind | 19:34 |
| OliPicard | im thinking of just using http, too much fuss for https. | 19:34 |
| TJ- | OliPicard: Any other nginx config files being read that might be conflicting? It sounds to me that it has already bound to the port. With the single "listen [::]:443" do you get the same error report? | 19:35 |
| OliPicard | Yup | 19:35 |
| OliPicard | and it's only 1 nginx instance running according to netstat. | 19:36 |
| OliPicard | which is the IPV6 version we just added. | 19:36 |
| TJ- | OliPicard: then I'm going to point at there being multiple configs being read | 19:36 |
| OliPicard | the default config has been deleted | 19:37 |
| OliPicard | the only config running is the current one i've been using. | 19:37 |
| TJ- | OliPicard: comment out that "listen" line so there are none you know of, and then try starting it, see what happens, whether something is bound to 443 in that case | 19:38 |
| OliPicard | TJ-: Nothing else running 443 after uncommenting that config. | 19:39 |
| OliPicard | when i comment it back and restart the only instance is the correct 443 one however nginx is saying it wont bind. | 19:40 |
| OliPicard | even thought i can see in inside ubuntu that it's binding. | 19:40 |
| OliPicard | decided to rebuild the instance on a different host to see if it improves. | 20:28 |
| TJ- | OliPicard: is it possible 2 instances of nginx are being started and the 2nd therefore fails? Any clues as to which PIDs generate the warning message? | 21:02 |
| teward | TJ-: was going to say he only needs a `Listen 443` and it should autobind to everything, but I see a bigger problem in his config | 21:38 |
| teward | TJ-: https://gist.github.com/anonymous/cda414038cb8de41b86f#file-gistfile1-txt-L9 Line 9 is ***BAD*** | 21:39 |
| teward | (POODLE) | 21:39 |
| teward | and i'm wrong, that'd be v4 only. he needs at least the two listens, the one with ipv6only would bind to v6 | 21:41 |
| teward | but the bigger issue I see is POODLE | 21:41 |
| TJ- | teward: According to the docs, a single "listen" can bind to both IPv4 and IPv6 of the form "listen [::]:443"" | 21:42 |
| teward | TJ-: which docs? If you say the nginx wiki i'mma smack you that's old | 21:42 |
| teward | (not really, but i'm going to reiterate that wiki hasn't been updated in eons) | 21:42 |
| teward | if ipv6only=off is specified, then probably | 21:43 |
| teward | but it's default-on apparently | 21:44 |
| teward | http://nginx.org/en/docs/http/ngx_http_core_module.html#listen | 21:44 |
| teward | TJ-: whether that's the case or not, he's got a POODLE exploit path with that config there with SSLv3 | 21:44 |
| teward | so he needs to update that if he shows up again | 21:44 |
| OliPicard | Ive just tested my node.js https stack on a new host. The nginx problem still persists across platforms. | 21:51 |
| teward | OliPicard: you have a bigger problem first | 21:52 |
| teward | OliPicard: your setup and config open you up to POODLE - the vulnerability that effectively killed SSLv3 | 21:53 |
| teward | OliPicard: paste *all* your configs for the site, you can sanitize auth and IP addresses, but put something in place as placeholders. I mean all the configs. | 21:53 |
| teward | nginx.conf | 21:53 |
| teward | site config | 21:53 |
| teward | default.conf | 21:53 |
| OliPicard | Roger that, I'm all years. | 21:53 |
| teward | everything listed in /etc/nginx/sites-enabled | 21:53 |
| OliPicard | ops ears* | 21:53 |
| teward | OliPicard: and while we're at it, change line 9 in your site configs to this: `ssl_protocols TLSv1 TLSv1.1 TLSv1.2;` | 21:55 |
| teward | remove the backticks in your conf too | 21:55 |
| teward | I apologize if i'm sounding bossy, but debugging nginx loadtime problems? Configs all help :) | 21:55 |
| * teward yawns | 21:57 | |
| OliPicard | no it's aboslutely fine :) | 21:57 |
| OliPicard | https://gist.github.com/anonymous/9523914622e2a574b3db | 21:57 |
| teward | staring at code all day didn't help either :) | 21:57 |
| teward | OliPicard: that's your site configuration - i also need the other site(s) in /etc/nginx/sites-enabled (if any exist), and your nginx.conf | 21:57 |
| teward | because i think you have a conflict in the one instance with something else listening | 21:57 |
| teward | also this will only listen on v4 with `listen 443 ssl` or similar | 21:58 |
| OliPicard | under sites-enabled theres a default vhost (the default nginx config) | 21:58 |
| OliPicard | posting the nginx.conf shoftly :) | 21:58 |
| teward | OliPicard: remove that if you don't need it | 21:59 |
| teward | BTW, lets strive to use indentation for ease of reading - https://gist.github.com/teward/09313a3c16f29ea1cf4a | 21:59 |
| OliPicard | teward: https://gist.github.com/anonymous/d6fa7025fc9ffa9a1688 < thats the conf | 22:03 |
| teward | OliPicard: remove the default config from sites-enabled (it's just a symlink, the original default config is in sites-available). i assume your config is in sites-available and symlinked into sites-enabled? | 22:04 |
| OliPicard | teward: yup have gone ahead and done that. Yes the config is symlinked | 22:05 |
| teward | OliPicard: okay, `sudo service nginx stop; sudo service nginx configtest` | 22:06 |
| teward | see whether there's any errors triggered | 22:06 |
| teward | (the reason i want you to stop is so it can forget the default config) | 22:06 |
| OliPicard | Have gone ahead and run both commands checking the logs now | 22:06 |
| teward | it'd output any config errors | 22:07 |
| teward | said "Config test passed" or "Failed" or something | 22:07 |
| OliPicard | hmm binding is still failing | 22:07 |
| teward | worst case test: sudo nginx -t | 22:07 |
| OliPicard | config test passed | 22:07 |
| teward | OliPicard: sudo netstat -tulpn | grep 443 | 22:07 |
| teward | pastebin the output | 22:07 |
| teward | (if its one line you can probably paste it here, if its more than one... well, pastebin) | 22:07 |
| OliPicard | roger that doing that now :) | 22:08 |
| teward | OliPicard: was the binding error on 80 or 443? | 22:08 |
| OliPicard | https://gist.github.com/anonymous/5478aac9e10223936988 | 22:08 |
| OliPicard | port 443 | 22:08 |
| OliPicard | teward: sorry port 443 and the paste has been included above. | 22:10 |
| teward | OliPicard: okay, it looks like nginx is running somewhere else as well... hmm... | 22:12 |
| teward | OliPicard: did you stop the nginx service? | 22:12 |
| OliPicard | i did indeed :) | 22:12 |
| teward | OliPicard: okay then you have a rogue nginx process | 22:12 |
| teward | OliPicard: killall -15 nginx | 22:13 |
| OliPicard | i did a ps aux | grep nginx | 22:13 |
| teward | OliPicard: it might be stale - still run the nuke to kill the processes | 22:13 |
| OliPicard | ok then use the service again to boot up? | 22:13 |
| teward | yes | 22:13 |
| teward | see if that solves the binding problem | 22:13 |
| === bilde2910 is now known as bilde2910|away | ||
| OliPicard | teward: unfortunately it hasn't the strange part was that the default nginx page displayed for 5 seconds, upon refreshing the server had terminated the connect. | 22:16 |
| OliPicard | *connection | 22:16 |
| TJ- | :q | 22:16 |
| teward | OliPicard: I'm not sure what you just said? Are you saying that it was the default page and not your app instead of what you expected? | 22:17 |
| teward | blah gotta run to take the girlfriend on the date i promised >.< | 22:18 |
| OliPicard | teward: the default nginx html page, refreshed and the server error still persists unfortunately | 22:18 |
| teward | OliPicard: okay, i have to run, but if you don't get a resolution here, try in #nginx here on IRC, that's the main nginx support | 22:19 |
| OliPicard | Thanks Teward! | 22:19 |
| teward | gotta take the girlfriend out :) | 22:19 |
| TJ- | OliPicard: if it is any consolation I can reproduce your issue here | 22:30 |
| OliPicard | TJ-: Phew glad i'm not the only one! | 22:31 |
| TJ- | Makes it easier to diaagnose, for sure | 22:31 |
| OliPicard | Yup, im wondering if it's a dependcy | 22:32 |
| TJ- | $ sudo netstat -tnlp | grep 443 | 22:34 |
| TJ- | tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 15282/nginx | 22:34 |
| TJ- | tcp6 0 0 :::443 :::* LISTEN 15282/nginx | 22:34 |
| * TJ- laughs ... I think I know what you've *not* done! | 22:35 | |
| TJ- | OliPicard: The config file you're editing - what is the exact path to it? | 22:35 |
| OliPicard | TJ-: the path is /etc/nginx/sites-available/host | 22:37 |
| OliPicard | did an ls no other files in the same area | 22:37 |
| TJ- | OliPicard: That's it bingo! | 22:37 |
| TJ- | OliPicard: Did you forget to create the *symlink* to it in "/etc/nginx/sites-enabled/" ? | 22:38 |
| OliPicard | TJ-; Did create a syslink at that location :) | 22:39 |
| TJ- | OliPicard: I got your exact issue with no symlink... soon as it was created, nginx started correctly | 22:39 |
| OliPicard | sudo ln -s /etc/nginx/sites-available/hostconf /etc/nginx/sites-enabled/ | 22:40 |
| OliPicard | thats what i used :) | 22:40 |
| OliPicard | TJ-: Not sure if that was the same as the link you did? | 22:42 |
| TJ- | OliPicard: Weird. I deleted the link again, and now the failure doesn't occur | 22:43 |
| OliPicard | how did you delete the link? | 22:45 |
| OliPicard | i'll try again to create a link and see if that works | 22:45 |
| OliPicard | TJ-: Nope sadly the link didn't work for me | 22:48 |
| TJ- | OliPicard: Now, without the link nginx starts but doesn't bind to any sockets. With the link it starts and listens correctly. | 22:49 |
| TJ- | OliPicard: This is the config file: http://paste.ubuntu.com/9984494/ | 22:50 |
| OliPicard | 1 sec ill grab my current config :) | 22:52 |
| TJ- | OliPicard: Have you checked the SSL key and certs are reachable by nginx process? | 22:53 |
| OliPicard | TJ-: How do i check that? | 22:53 |
| OliPicard | TJ-: current config - http://paste.ubuntu.com/9984542/ | 22:54 |
| TJ- | It probably won't cause that error - I tested it here, it gives a different specific missing file warning | 22:55 |
| TJ- | OliPicard: Has nginx ever started since installation? | 22:57 |
| OliPicard | TJ-: Yup it has just for some reason it struggles to bind on port 443. if i set it to port 80 its fine. | 22:58 |
| TJ- | I'm wondering if it might be caused by a default installation config issue, such as something not created/written under /var/lib/nginx/ for example. I'm purging and reinstalling it here, to test that, since I know I was able to reproduce the issue | 22:58 |
| spicypixel | so, bit weird, using winbind and 2012 AD, I've got to the point where I can log into any user on the AD list, and it shows the groups it's in, but brings up "groups: cannot find name for group ID 5001" on login, and groups confirms it's in the right group + 5001 which has no name, has no local entry not AD entry I can see | 23:00 |
| OliPicard | TJ-: Ok, i shall await to see if you can recreate the bug :) | 23:01 |
| TJ- | OliPicard: Grrr, no, I can't | 23:02 |
| OliPicard | TJ-: did you use a pass phrase on the certificate? | 23:04 |
| TJ- | OliPicard: No | 23:04 |
| TJ- | OliPicard: Hmmm, that's a good point! Did you? | 23:04 |
| OliPicard | I did! | 23:04 |
| OliPicard | let me see if that's the problem | 23:05 |
| TJ- | OliPicard: It isn't - I added a passphrase to my private key and tested it. It prompts for the passphrase, and gives "[emerg] 17145#0: SSL_CTX_use_PrivateKey_file("/etc/ssl/private/iam.tj.key") failed (SSL: error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt error:0906A065:PEM routines:PEM_do_header:bad decrypt error:140B0009:SSL routines:SSL_CTX_use_PrivateKey_file:PEM lib)" | 23:07 |
| OliPicard | hmm mine never prompts me for a passphrase | 23:08 |
| TJ- | OliPicard: Then it doesn't have one :) | 23:08 |
| OliPicard | TJ-; But during the CSR request i input a unique certificate passphrase. | 23:09 |
| TJ- | OliPicard: Use openssl to check it | 23:09 |
| TJ- | OliPicard: "openssl rsa -noout -text -in /srv/ssl/domain.io.key" | 23:11 |
| OliPicard | TJ-: Nope still not working. did decrypt the passphrase. | 23:14 |
| TJ- | OliPicard: Let's go back to basics. Ensure no nginx processes are running (use ps to confirm) then start it under strace: "strace -o /tmp/nginx.log -f nginx -c /etc/nginx/nginx.conf" then "pastebinit /tmp/nginx.log" | 23:16 |
| OliPicard | TJ-: https://gist.github.com/anonymous/25350d2b06007aaa01ee | 23:21 |
| TJ- | OliPicard: line 487 shows it worked | 23:22 |
| OliPicard | Yup yet if i type sudo nginx the log shows it's not binding | 23:23 |
| OliPicard | i know it's binding from what i can see, just nginx doesn't think it is. | 23:24 |
| TJ- | OliPicard: how do you mean "if i type sudo nginx" ? | 23:24 |
| TJ- | OliPicard: If the service/process is already started that command would be expected to report the socket is already in use | 23:24 |
| OliPicard | TJ-: I'm seeing the msgs coming up also in my nginx logs. | 23:25 |
| OliPicard | sudo nginx nginx: [emerg] bind() to 0.0.0.0:443 failed (98: Address already in use) nginx: [emerg] bind() to 0.0.0.0:443 failed (98: Address already in use) nginx: [emerg] bind() to 0.0.0.0:443 failed (98: Address already in use) nginx: [emerg] bind() to 0.0.0.0:443 failed (98: Address already in use) nginx: [emerg] bind() to 0.0.0.0:443 failed (98: Address already in use) nginx: [emerg] still could not bind() | 23:25 |
| TJ- | OliPicard: If it is already running that's what you'd expect. The strace log you showed, proves it is bound and listening correctly | 23:26 |
| TJ- | exit | 23:27 |
| OliPicard | TJ-: I'm seeing the same msg showing on /var/log/nginx/error.log | 23:27 |
| TJ- | OliPicard: But you'd expect to! the process is running until you kill it | 23:28 |
| OliPicard | I have! | 23:28 |
| OliPicard | Using the killall command serval times | 23:28 |
| TJ- | OliPicard: When you run "nginx" it forks into the background. If you use "service nginx start" then you can "service nginx stop" to stop it, otherwise you have to "killall nginx" | 23:28 |
| OliPicard | every time i boot up it gives the same error message about being unable to bind. | 23:28 |
| TJ- | OliPicard: You mean when the init system starts nginx automatically? | 23:29 |
| OliPicard | i'll try another start | 23:29 |
| OliPicard | Yes TJ-: | 23:29 |
| TJ- | OliPicard: Is the network 'up' at that point? | 23:29 |
| OliPicard | I believe it is | 23:30 |
| OliPicard | I'm running the box off of a linode. | 23:30 |
| TJ- | OliPicard: OK, I'd expect it to be active then | 23:30 |
| TJ- | The thing is we've just proved, with strace, that it does start correctly, so there's something else going on | 23:31 |
| OliPicard | i also tried using a KVM based server to just to rule out it being an Xen issue. | 23:32 |
| TJ- | OliPicard: Is apache also installed/starting on that host? | 23:32 |
| OliPicard | i had to the exact same problems on the KVM box as i did on the Xen box | 23:32 |
| OliPicard | Nope i checked :) | 23:32 |
| OliPicard | if i try to visit the site it's still down | 23:33 |
| TJ- | it's got to be something you are (not) doing :) | 23:33 |
| === Blanoz is now known as blanoz | ||
| TJ- | OliPicard: OK, let's separate the site-down from whether the process is listening - DNS misconfiguration could cause site-down issues | 23:34 |
| OliPicard | ok | 23:35 |
| TJ- | OliPicard: Silly question, but, the "already in use" messages in the log file - they *are* repeating, with new current timestamps? | 23:35 |
| OliPicard | TJ-: Not that i can see, the last timestamped bind error was 23:24pm | 23:36 |
| TJ- | OliPicard: If in a 2nd ssh shell you run "sudo tail -f /var/log/nginx/error.log", and then on the 1st ssh shell so "sudo service nginx restart" ... does the log tail show those errors | 23:36 |
| OliPicard | ill check 1 second :) | 23:37 |
| TJ- | OliPicard: OK! So, those are *old* reports and do not represent what is happening *now*. My point being, it has likely been working correctly for a while now but your testing - via trying to access the site - may fail due to other reasons | 23:37 |
| OliPicard | TJ-: Ok, so normally you would see a gateway error at the least? | 23:39 |
| TJ- | OliPicard: That depends on a lot of variables! No.1 is to ensure after the process has started, it continues to run, and is listening on the port. USe "sudo netstat -tnlp | grep 443" for that | 23:40 |
| TJ- | OliPicard: If that shows it is listening... then proceed to figure out why a user agent can't access it. Likely to do with the proxy config, or firewall, or routing, or any number of other things | 23:41 |
| OliPicard | The firewall is open to all on port 443 | 23:42 |
| OliPicard | The DNS has an A record pointing to the server. | 23:42 |
| TJ- | OliPicard: OK, but the config is to proxy to localhost:9001 | 23:42 |
| TJ- | OliPicard: the error could be coming from that | 23:43 |
| TJ- | OliPicard: Sorry, port 9000 now I see. It was 9001 earlier though, could that be an issue? | 23:43 |
| TJ- | OliPicard: Is it proxying for a Tomcat instance, or similar? | 23:44 |
| OliPicard | nope just a single nginx server running :) | 23:48 |
| OliPicard | its preoxying node .js | 23:48 |
| OliPicard | brb gotta reconnect | 23:48 |
| === zz_DenBeiren is now known as DenBeiren | ||
| OliPicard_ | Im back sorry about that. | 23:58 |
| OliPicard_ | Yeah my server app is running but nginx isn't processing the request | 23:58 |
| TJ- | OliPicard: test a connection locally using 'openssl s_client' | 23:59 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!