/srv/irclogs.ubuntu.com/2016/08/20/#ubuntu-server.txt

thekryn__any nginx users in the room?01:34
patdk-lap!ask01:41
ubottuPlease don't ask to ask a question, simply ask the question (all on ONE line and in the channel, so that others can read and follow it easily). If anyone knows the answer they will most likely reply. :-) See also !patience01:41
patdk-lapthe question you just asked is the most annoying one ever to use on irc01:41
patdk-lapcause if someone does, it annoys them waiting for you to respond01:42
thekryn__is there any downside to setting limit_req_zone rate to 1r/s to ensure that the lim_req's are always fielding requests01:45
thekryn__i get screamed more about asking the incorrect questions in a channel than asking if people know about the subject01:46
thekryn__10:101:46
patdk-lapdownside would be it wouldn't serve someone more than once per second01:47
patdk-lapit will make your site look *slow* to me01:47
patdk-lapand I won't use it01:47
thekryn__tehcnically it does as of right now, which is what i dont understand01:48
patdk-lapwell, this is why people say you ask incorrect questions01:48
thekryn__limit_req_zone $uri zone=dsp_per_ex:10m rate=1r/s;01:48
thekryn__   limit_req zone=dsp_per_ex burst=10 nodelay;01:48
thekryn__alows 10 requests per second01:48
thekryn__whowever, if i do rate=10r/s and burst = 1, it ignores the burst01:48
patdk-lapsee how you just change that from, I have a question, to, I have a problem I didn't want to tell you about, but let me hint at it01:48
thekryn__hence the 1r/s is the only way to do it01:49
thekryn__but im not sure about potential implications of performance01:49
thekryn__and dont understand why it would have been designed that way01:49
patdk-laphow did you test?01:50
patdk-lapand what does, ignores the burst, mean?01:50
patdk-lapyour skipping steps01:50
patdk-lapwhat you did, how you tested, what the test results where, what you expected the results to be01:50
thekryn__limit_req_zone req 10,000r/s, server{ location /a {limit_req burst 5,000}, location /b {limit_req burst 1}}01:51
thekryn__when i hit domain/a 5000 per sec, i get 5000 204 responses01:52
thekryn__when i hit domain/b 5000 per sec, i get 5000 204 responses01:52
patdk-lapyes, that all looks right01:52
thekryn__if i drop limit_req_zone rate to 1r/s, i get 5000 204 responses for /a, 1 204 response for /b, which is what i want01:53
patdk-lapthat sounds odd01:54
thekryn__but im not sure if there are underlying latency or ram issues associated with that approach, as it seems like that shouldnt work01:54
patdk-lapwhy would you set a burst so high?01:54
patdk-lapbursts should be kept low01:54
thekryn__the traffic i have hitting my cluster is around 10mil req/s01:54
thekryn__from 10 companies01:54
patdk-lapyes, so you want /a to handle 5000 for one second01:55
patdk-lapthen go on at a rate of 1 per second after that first 5000?01:55
thekryn__i want say 10 of those kind of endpoints, say /1, /2, /3, etc...01:55
thekryn__and then i want to have a /debug which i set to something low01:55
thekryn__or.. i want to say choose one of those companies, say /701:56
thekryn__and force it to burst 1 which would basically be like a block01:56
patdk-lapyou do know what burst means right?01:56
thekryn__how many req / s?01:56
patdk-lapno01:56
thekryn__thats how i read it from the docs01:56
patdk-lapheh?01:56
patdk-laphow many requests BEFORE it uses the limit_req_zone r/s value01:56
patdk-lapas it's name says, it's the burst setting01:57
patdk-lapa normal webpage has say, 50 pictures01:57
patdk-lapthe burst allows you to load all the pictures01:57
thekryn__interesting01:57
patdk-lapbefore you hit the limit and it slows you down01:57
thekryn__oh so that makes more sense then01:58
patdk-lapso something like 10r/s with a 50 burst01:58
patdk-lapload page and images01:58
thekryn__ok, so if i set the burst to 10,00001:58
patdk-lapthen as users goes page to page using the same images, they won't go over that 10r/s again01:58
thekryn__that means it will handle 10k a sec until they exceed that01:58
patdk-lapno01:58
patdk-lapit means it will handle 10k01:58
patdk-lapnot per second01:58
thekryn__oh... so there's no way to set a rate then01:59
thekryn__using lim_req01:59
patdk-lapthat 10k should be refilled at the rate of r/s setting, 1r/s01:59
thekryn__so if i need one endpoint to fire at 10k/s and the other at max 1/s01:59
patdk-laphmm, that is what the zone setting does01:59
patdk-lapyou didn't set a zone01:59
thekryn__limit_req_zone $uri zone=dsp_per_ex:10m rate=1r/s;01:59
thekryn__so if i were to set to, say02:00
thekryn__limit_req_zone $uri zone=dsp_per_ex:10m rate=10000r/s;02:00
patdk-lapI don't see a limit_req zone=02:00
patdk-lapI don't see a limit_req zone=dsp_per_ex02:00
thekryn__well, then i have this:02:00
thekryn__   limit_req zone=dsp_per_ex burst=1 nodelay;02:00
thekryn__which does absolutely nothing02:00
thekryn__when rate=10000r/s02:01
patdk-lapwell, that would say 10k per second is allowed02:01
patdk-lapno bursting02:01
patdk-lapso just straight 10k/sec02:01
thekryn__so then maybe the question should be... is there a way to set separate zone rates per endpoint02:01
thekryn__as i tried wrapping limit_req_zone in if clauses, and that didnt work02:01
patdk-lapwhy?02:02
patdk-lapwhy would you do that?02:02
thekryn__because i need to limit the number of requests per endpoint02:02
patdk-lapyes, did you look at the manual?02:02
patdk-laphttp://nginx.org/en/docs/http/ngx_http_limit_req_module.html02:02
patdk-laplook at the example right at the top02:02
patdk-lapdefine each limit_req_zone02:02
patdk-lapthen below assign that zone to a location02:02
patdk-lapsince each zone has it's own tracking and rates02:03
patdk-lapeach location gets whatever you set02:03
thekryn__so you mean something like02:03
patdk-lapif you need to limit 50 locations, define 50 limit_req_zones02:03
thekryn__limit_req_zone "endpoint1" ....; limit_req_zone "endpoint2" ....; etc?02:04
patdk-lapbut I don't see that limit_req_zone wrapped at all do you?02:04
patdk-lapno02:04
patdk-laplimit_req_zone zone=endpoint1 ....02:04
thekryn__ohhhh02:04
* patdk-lap has never used or touched nginx before02:05
patdk-lapanother reason your first question was pointless02:05
thekryn__limit_req_zone $uri zone=e1 rate=10000r/s; limit_req_zone $uri zone=e2 rate=1r/s02:05
patdk-lapno02:05
patdk-lapmaybe02:05
patdk-lapwhat $uri?02:06
patdk-lapwhat are you attempting to limit?02:06
thekryn__number of requests per $uri02:06
patdk-lapI guess02:07
patdk-lapseems strange to me :)02:07
thekryn__i have: domain.com/(16charhash)02:07
thekryn__company1 has hash1, and they have say a pool of 10 ips making 1000 qps each02:07
thekryn__my issue was is that company3 had a machine at some ip that started wrecking me at 40k qps02:08
thekryn__the limit_req is sitting in front of a proxy_pass02:08
thekryn__and it was choking the proxy pass02:08
thekryn__we told them to cut it out.. it took then 8 hours to turn it down02:09
patdk-lapif you want to limit the 16charhas to 10k per second, use the e1 limit_req_zone02:09
patdk-lapthen use e2 for the one you want to be lower02:09
thekryn__yup.. thats what i was thinking.. basically e2 i would prob name the hash02:09
thekryn__like zone=hash_0123456789abcdef02:09
thekryn__so i can keep track of fairly easily02:09
patdk-lapsee where it says, limitation is done using leaky bucket method02:11
thekryn__i was trying to set the $uri thing.. totally didnt realize you could just make different zones by zone name and query them directly02:11
patdk-lapthe burst is how many it can handle, the r/s is how fast the burst is refilled02:11
thekryn__yup, see that02:11
patdk-lapso you probably will want to set burst=r/s02:11
patdk-lapwell >=02:12
thekryn__got it... so if the rate is at 10,000r/s02:12
thekryn__and we let them say burst at slightly higher than that02:12
patdk-lapfor your 1r/s, probably set it to something sane, maybe 10, or 1 if you really really want02:12
thekryn__if they go over, then theyr'e at th emercy of the r/s catching up02:12
patdk-lapyep02:12
thekryn__so if they sustain higher than 10k, it takes longer nad longer to adjust02:12
thekryn__that makes perfect sense02:13
thekryn__thanks, appriciate it02:13
thekryn__we've been doing consistant hash upstream stuff and all sorts of other stuff.. been literally at it for 2 weeks, so im a bit burnt out02:13
thekryn__we've got about 200 4 core machines in the router/upstream config by now02:14
thekryn__so its a bit brutal to keep track of02:14
thekryn__and im assuming i could make something like02:17
thekryn__zone= production rate=10000r/s; zone=choked rate=1/rs02:17
thekryn__and just assigned like 10 endpoints to production and 2 to choked and update configs as necessary if people misbehave?02:18
patdk-lapsure02:18
patdk-lapyou might need to increase that 10M to soemthing larger when using $uri though02:19
patdk-lapyou will know when you start serving up lots of 503's02:20
thekryn__got it... the 10M im guessing is for sessions?02:22
patdk-lapfor the hash table02:22
patdk-lapit will add $uri and how many r/s it can have02:23
patdk-lapso the longer the $uri and the more uniq $uri's, the more space it will use02:23
thekryn__on our entire system, we only have 10 $uri02:23
thekryn__since its just a list of 10 companies, 1 hash per02:23
patdk-lapso not likely to be an issue now02:23
thekryn__we do have a similar setup, where its IP address based02:24
patdk-lapip's are more predictable, 128bytes per entry02:24
thekryn__since those requests come in by uri, and they have POST json which we decode, get an IP address out of, proxy pass and use the consistant hash upstream02:24
patdk-lapso 10M will allow you like 82k ip's02:24
thekryn__and the machiens catching those are going to have limits on them as well per ip.. so that'll happen then02:24
thekryn__perfect... we allocated about 4G per machine ram wise for scaling out02:25
patdk-lapso if your rejecting non-hash uri's before it hits that system, then you don't have to worry about some vaunerability scanner filling that $uri limit02:26
thekryn__thats a great point.. we havent seen anything like that as of yet, at least at scale02:27
patdk-lapit shouldn't matter02:27
thekryn__but we are 204'ing those at the moment and dont have a lock on them02:28
patdk-lapunless you have a *not used much* client02:28
patdk-lapand they attempt to use it, after the scan was going on and filled it02:28
patdk-lapfor the ones active before the scan, they would be fine02:28
thekryn__yeah, we manually load all the hash directives first before we let anything else hit02:29
patdk-lapno, you can't control this02:29
patdk-lapit will load and remove them as they are used02:29
thekryn__how so?02:29
patdk-lapthe only way to control it, is some other limit or server blocking it02:29
patdk-laplimit_req_zone $uri zone=e1:10M rate=10000r/s02:30
patdk-lapit will load the $uri into e1, when something hits the webpage02:30
patdk-lapif the rate is completely fill, it will likely remove it, to make room for others02:30
patdk-lapsince it's at the full rate, no need to track what the rate is anymore02:31
patdk-lapthink of it more, if you where limiting by ip address02:31
patdk-lapif it didn't remove them02:31
patdk-laponly the first 82k ip's to use your server would EVER be able to view your website02:31
patdk-laptill you restarted nginx02:31
thekryn__ahh, got it02:32
thekryn__well, the good news about this entire setup is that we purposefully hash so the endpoints are not public02:34
thekryn__its not a website per se02:34
thekryn__so most of the rogue fires will end up in a sort of blackhole02:35
thekryn__thanks again, really appriciate it02:39
pclerie sarnold: Hi! Re nfs problem we discussed the other day, I filed bug #1614261. Got a work around, that did not help much. But thanks for the tip.03:09
ubottubug 1614261 in nfs-utils (Ubuntu) "RPCSVCGSSDOPT is ignored by boot script" [Undecided,New] https://launchpad.net/bugs/161426103:09
ilivbosco, joing #yourhttpserver03:58
=== JanC is now known as Guest65624
=== JanC_ is now known as JanC
jonahHi can anyone help with constant intermittend email/dovecot imap authentication problems. I keep getting errors: pam_unix(dovecot:auth): authentication failure; logname= uid=0 euid=0 tty=dovecot ruser=info-website.co.uk rhost=::1  user=info-website.co.uk05:56
jonahalso this error: auth: PAM unable to dlopen(pam_systemd.so): /lib/security/pam_systemd.so: cannot open shared object file: No such file or directory05:56
jonahI don't know if that has anything to do with the auth problem but I get this too: auth: PAM adding faulty module: pam_systemd.so05:57
jonahI'm running ubuntu 14.04 LTS05:57
=== markthomas_ is now known as markthomas
=== andyjones2001_ is now known as andyjones2001
=== froike- is now known as froike
=== beisner- is now known as beisner
=== fidothe_ is now known as fidothe
=== NetworkingPro_ is now known as NetworkingPro
=== masACC is now known as maswan
=== Sling_ is now known as Sling
=== Pici` is now known as Pici
sikunquestion, if you were setting up a HA/Load Balancing web server (two hosts) would you setup the MySQL server on the master and configure the slave for replication or host the MySQL server on a third host?13:57
=== JanC_ is now known as JanC
RoyKsikun: I'm working on a HA setup of MariaDB (the MySQL fork) and my plan is a MariaDB Galera cluster with a couple of haproxy servers in front with a master/slave setup with Corosync/Pacemaker14:33
RoyKsikun: this will be in an all-virtualised environment (we have 250ish VMs currently, physical machines are down to a small fraction of that)14:36
sikunRoyK, ah nice14:42
sikunRoyK, what I'm building I am utilizing physical boxes until the funds are available to build a proper virtualization environment that is capable of HA14:43
sikunor I should say until I can prove the need for the equipment to obtain a loan or whatnot14:44
RoyKtwo machines should be sufficient for HA with KVM - I setup a test system with that a few years back14:45
RoyKseparate, shared storage is recommended14:45
sikunsadly, not the hypervisor I use but still two machines is good enough for what I do use.14:46
RoyKwhich hypervisor is that?14:46
sikunHyper-V14:46
RoyK*blargh*14:46
sikunlol14:46
RoyKI used to work with Hyper-V, and although that was four years ago and I guess a lot has happened since then, I really didn't like it14:48
sikunI have one host right now, it is a bit old.. but it still performs amazingly and with the hardware upgrades I ordered that should be here next week it'll at least last me a good 6 months to a year14:48
sikunI hear that a lot, and I have worked with it since 2008 when it was total garbage.. but it really has become a very good hypervisor14:49
RoyKI even setup a KVM system along it to run Linux VM's, since any ubuntu VM we tried to put on Hyper-V lost its network connection on high load, no error messages, nothing in the logs, neither on the ubuntu machines nor on the hyper-v hosts14:49
RoyKthis was on win2k8, yes14:50
sikunthe data center I work at, they used Hyper-V way back when and would have VMs just disappear14:50
RoyKHAHA14:50
sikunand that was also on 2k814:50
sikun2k12r2 is fantastic, I'm very excited for 2k16 to release14:51
sikunI try to use the Hyper-V 2012R2 core install for a vm host whenever I can14:51
RoyKI started working with vmware some 3-4 years back and I'm rather excited on what it can do and how things just work14:51
RoyKhow many hosts?14:52
sikunVMware is good, I'm not saying it isn't by any means.. but I extremely dislike the licensing.. how much you have to pay to be able to do certain things14:52
RoyKI didn't say it was cheap14:52
sikunlol, true14:52
sikunwell I downsized the hosts to one at the moment I had 414:53
sikunthe one remaining host was purchased outright so that's why I still have it14:53
RoyKwhat sort of hw?14:54
sikunI'm working on getting an environment ready for a possible client, I have a meeting with him next week and if he decides he wants to move all of his services to my infrastructure, I'll be extremely happy but also stressed as hell, lol14:55
sikunHP Proliant14:55
RoyKblade things?14:55
sikunit's dated... it does need to be decommissioned but I'm going to offset the load on it by using other servers14:55
sikunno, it is a DL160 G614:56
RoyKwe only use blade servers these days14:56
sikundual xeon quad core, 96GB of RAM with more RAM that should be coming next week along with all new hard drives.14:56
sikunthat's what I'm looking into for a replacement for everything14:57
RoyKthat is, we actually bought a 4U server a few months back, since some of our scientists insist of using Stata on Windows when they really should have been using R on a supercomputer14:57
sikunthrow a Intel Xeon Phi or a Kepler in a server and let em go to town14:57
RoyKquad socket dell thing with four 8-core CPUs clocked the highest we got and half a terabyte of RAM14:58
sikunvery nice14:58
RoyK$50k or so :P14:58
sikunI was checking out the specs of a Lenovo rack server, I think it was Lenovo, but had dual 24 core Xeons14:58
sikunthe data center I work at, their VMware cluster is total garbage..14:59
RoyKwe have three clusters15:00
RoyKone for test/lab/etc15:00
RoyKone for the important stuff with new hosts15:00
RoyKand one with older machines (and thus older instruction sets, say, 4-5 yo) for medium importance machines15:01
sikunresources are so low of this cluster... I can't even get my requests for a test VM to be spun up, oh wow.. it can't afford a 512MB, 4vCPU and 20GB for 3 days?15:01
sikunpathetic15:01
sikunI end up having to utilize my personal equipment to spin up test VMs15:02
sikunall my personal hardware is old, don't get me wrong.. but even when it was at 90% utilization it would outperform that VMware cluster15:03
RoyKhehe - perhaps they should get a few new hosts ;)15:05
sikunha... we have 9.2% free space on the SAN, we're screwed because getting a $50k SAN isn't going to happen anytime soon15:05
RoyKbeefed up with a ton of memory, since that's where the bottleneck usually is15:05
sikunthey are actually loaded with RAM15:06
RoyKwhat sort of SAN?15:06
sikunthe SAN is garbage15:06
sikunDell EqualLogic15:06
RoyKhaha - we have EQL as well15:06
RoyKand I know exactly what you mean15:06
sikunthe whole load balancing question is actually a potential client of mine, not the company I work for. ha15:08
sikuneven colo'ing in the data center where I work isn't cheap... I have occasionally got some bigger discounts by allowing them to temporarily utilize my hardware15:10
RoyKapropos load balancing... we have two shelves with 100TiB net storage each in an equallogic storage group, and they're supposed to stripe across the two. Curiously, lately one of them has been running at 100% utilisation while the other is at 60%15:13
sikunslightly odd15:14
sikunnot throwing any errors?15:14
RoyKno, and the only debug reports you can get out of an equallogic system, are encrypted with Dell's public key so they can only be read by Dell15:14
sikunomg I hate that bull shit.. I want to rip the two Barracuda spam filters out of the rack and go all office space on them because of that same reason... oh hey, they the twins are pegged at 100% CPU utilization and the queue is now up to 3k messages but can I diagnose what's wrong? nope15:16
sikunand of course barracuda support when the remote in are always like, lets reboot these quick... reboot it and I will hunt you down and beat you to death with a keyboard15:17
sikunhmm... three EMC Isilon IQ36000X 36TB units for $2,50015:19
RoyKI'd rather use something homegrown15:21
RoyKlike some boxes with ZFS and iSCSI15:22
sikunah15:22
sikunI have yet to play around with ZFS in detail15:22
RoyKI've worked with it for 6-7 years15:22
sikunnice15:23
sikunon my Proliant DL160 G6, damn RAID controller failed.. hopefully the one I ordered shows up Monday15:25
RoyKsikun: I guess we should take this to #ubuntu-offtopic before someone complains ;)15:26
sikungood idea15:26
ducassei've noticed a weird thing with an lxc container on 16.04. inside the container 'free -m' reports 350mb used, while lxc-info reports "memory use" as 20gb. for other containers the numbers match.15:58
stonerlhi hope someone can help me. I try to install ejabberd on Ubuntu 16.04 from the repos. But installation always fails because of a missing pid file:16:50
stonerlejabberd.service: PID file /run/ejabberd/ejabberd.pid not readable (yet?) after start: No such file or directory16:50
stonerlhas anybody a clou what to do?16:50
reefoctopusanyone know of a reason why my system could be completely ignoring my sysctl time_wait setting?17:07
patdk-lapit's just not interested in using that value17:07
reefoctopus?17:08
=== Daviey_ is now known as Daviey
LostSoulHi20:46
LostSoulI'm observing strange ssh issue20:46
LostSoulWhen I try to login as root it works like a charm20:46
LostSoulBut when I'm trying to login using ldap + active directory + pam - it takes up to 30-60 seconds20:46
=== RoyK^_ is now known as RoyK
bekksDid you check DNS connectivity?20:51
LostSoulYeah bekks, I think DNS is working fine21:04
bekksDo you think or did you check? :)21:05
jonahHi has anyone upgraded from Ubuntu server 14.04.3 to 16.04 - going to do it shortly and wanted to check any problems21:09
jonahalso I'm running an older kernel on my server 14.04, does anyone know if linux-image-generic or linux-image-server is installed as default and I could reinstall to get the latest stable kernel again? Or do server ubuntu installs not have this package installed by default?21:33
LostSoulbekks: I don't know how to verify this :(22:05
bekksyou can do that using nslookup and/or dig22:06
=== med_ is now known as Guest25180
LostSoulbekks: But what should I check? :)22:23
LostSoulI mean what can causing problem?22:23
bekksFailed DNS responses, timeouts while waiting, switching to the next DNS server available, etc.22:23

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!