[02:53] I can't start Dovecot because somehow it doesn't have permission to make /var/run/dovecot. Everything was fine before I restarted for a kernel update. What happened and how do I fix it? [02:56] Aug 20 19:52:31 andariel dovecot[12222]: Fatal: mkdir(/var/run/dovecot) failed: Permission denied <-- this is the only thing in the log [06:00] good morning === cpaelzer_ is now known as cpaelzer [06:29] Good morning [07:16] anyone know if citadel is still under development? [09:42] anyone know if citadel is still under development? [09:43] !patience | calcmandan [09:43] calcmandan: Don't feel ignored and repeat your question quickly; if nobody knows your answer, nobody will answer you. While you wait, try searching https://help.ubuntu.com or http://ubuntuforums.org or http://askubuntu.com/ [09:55] hi [09:55] who here runs own server? [09:56] I am thinking here hmm do I really need aws for my say windows server [09:56] no I dont [09:56] :) [09:56] the only reason I use hosting is anonimity :) [09:56] however for none anon stuff I could self host [10:02] thanks lordievader, alot of people joined the channel in the 2 hours i waited for an answer. [10:03] hehehe: i host my own servers to control/own my data as well as the privacy. [10:03] going to bed. gonight [10:03] calcmandan: It is probably better to look at git commits or alike. [10:04] hehehe: raises hand [10:28] kl [10:28] well likes of piratebay is pointless on home connection [10:28] however e commerce site if ping is good [10:28] maybe [10:29] as to privacy you could true crypt hd right? [10:29] :D [10:29] or I am missing something [10:36] fuck off silent people [10:36] :D [10:36] for real [10:36] hehehe: please stop the language [10:36] there is no need for it, I've spoken to you about this before, please adjust your attitude if you wish to continue using the channel === dpawlik_ is now known as danpawlik [14:18] have 4 servers that i was able to ssh into last night, and this morning i cant but they are all running. not sure wtf happened. [14:20] madLyfe: Power outage? [14:20] dont think so, and if so, i have it set to power back on as the default in bios. and they are in fact running [14:21] i cant ping them [14:21] host unreachable [14:29] Perhaps a router in the middle failed? [14:30] they are behind a switch am able to ping a win machine that is behind that. [14:31] they are showing up under my wired table in my router admin panel. well 3 are and 1 isnt. [14:31] I suppose a port scan reports all ports as closed? [14:32] sorry im pretty green. not sure what you mean === danpawlik is now known as danpawlik_ === danpawlik_ is now known as dpawlik_ [14:44] madLyfe: Run nmap on your host to see if ports report to be open. [14:52] it cant be pinged so scan will do nothing? [15:01] hmm i can ping them now [15:01] wait, nvm [15:03] still getting Destination host unreachable. [15:09] ok to just hard power them down? [15:35] hmm wasnt able to get a connection to them unless i did a hard reboot. [18:10] madLyfe: About the no ping -> no ports, that is not necesarily true. A firewall could simply block icmp and allow all other traffic. [18:12] i dunno what happened over night or why it didnt recover properly, it should be find after a power outage or loss of network but i rebooted all of them and they are good to go. [18:32] it's too early for me to know for sure, but changing vm.swappiness to 1 actually seems to have made a big difference to IO and overall system performance, it just wasn't noticeable for a while because the system is slowly swapping in all swapped out memory, over the last day it went from about 2.5% to 1% swap usage [18:34] what is interesting is that all processes which were doing infrequent reads are doing no reads at all now, only one of my read time processes does very infrequent reads of 8KB now, and the only other process that did a disk read very recently is SSH [18:34] so all the chunks that were being read from disk were not even the processes themselves but just the kernel swapping [18:34] Hexian: if you know your working set size will fit into memory and you never overcommit, just turn off swap [18:35] IO wait times peak at 0.5% instead of like 20% now [18:35] Hexian: that's almost what you're doing by setting it to a near-zero value -- only swap when the kernel absolutely must [18:37] if swapping really turns out to have been the cause of these major intermittent issues, the default swappiness is far too high for servers [18:38] Hexian: how well do you know your workload? [18:38] Hexian: as in, do you even need swap? [18:39] I don't need swap, but I'd rather not disable it completely just in case there is ever a memory leak on the box [18:39] if there is a memory leak on the box, swap won't technically help you, it just delays the inevitable... [18:40] but in any case [18:40] Hexian: yes, it's assumed you'd tune the value based upon workload [18:40] yeah I guess I'd rather have processes crash if there is a leak, than have performance issues [18:40] right, so you don't want swap :) [18:40] so maybe disabling swap is a good idea, though it's never been necessary in the past for any of my boxes [18:40] it may be necessary with the newer kernel [18:41] Hexian: i'm not sure there is an distinction made between the default sysctl value on server and desktop, i think it's 60 everywhere? [18:41] it seems to assume all servers have SSDs [18:41] Hexian: but it sounds like you do have some overcommit then, if you did hit swap at all? [18:42] reducing swappiness to 0 has increased memory usage a bit, but there is still plenty free ram on the box at all times [18:42] like 50% [18:42] the free 16+GB is being used for IO caching [18:42] Hexian: i *believe* swappiness serves as a bit of toggle over the watermarks [18:43] i'd need to go reread the code to be sure [18:44] my problem is that these major issues that I've had for months only happen severely at random times, the box can have no performance drops large enough to matter for a week or 2, and then 2 days of constant issues every few minutes or a few times an hour [18:45] so I will have to monitor the box for weeks to be sure that it's even the swapping [18:45] sounds like unstable workloads and pretty typical for untuned systmes [18:45] *systems [18:45] Hexian: as in, nothing you say is inherently surprising to me :) [18:45] my processes don't write to disk, they write to a ram disk to avoid IO overhead [18:46] they do little reads (now that I've reduced swappiness it seems they do virtually no reads at all) [18:46] yet when the issue is severe, they can freeze for anywhere from 500ms to 20 seconds straight while waiting on IO [18:47] writing to a ram disk puts *more* pressure on memory [18:47] sure, but the writes are small and infrequent, and more than 50% ram is free usually [18:48] writing to ram disk should be completely unnecessary, but it prevents ever blocking on the infrequent disk writes [18:48] I moved writes to a ram disk to try to solve the issue that I believe is swap related now [20:38] drab: netdata does support backends for historical stats these days - https://github.com/firehol/netdata/wiki/netdata-backends [22:00] zesty is out now? [22:10] Epx998-: it has been for a while (since april) === Epx998- is now known as Epx998 [22:31] rbasak: i'm thinking we may want a wrapper around build's logic for just obtaining the orig tarballs (I might already ahve filed a bug for this), e.g. git ubuntu extract-orig. The reason being some commonly-used tools, e.g., dpkg-source, expect to find the orig tarball in the parent directory === Epx998- is now known as Epx998 [23:05] nacc: i've been in channel since then ;P [23:10] Epx998: :) [23:28] cpaelzer: do you want me to sync the three packages from the pad? (irqbalance, mod-wsgi, pwgen)? [23:33] ahasenack: do yhou have room for another merge? net-snmp should be relatively straightforward, as several bits of delta seem to be cherry-picks from debian in the first place? [23:34] teward: are you planning on updating nginx in artful? sorry if you answered before, my brain has already context-switched [23:47] lamont: do you know why src:python-formencode's orig tarball is different in Ubuntu from Debian's? [23:48] (for 1.3.0)