| tannewt | maxb: ah, I see, is it because versions are compared alphabetically? | 00:14 |
|---|---|---|
| johanbr | tannewt: if you're doing this automatically, you may want to do it via git, since that'll carry all the version tags | 00:17 |
| johanbr | both ubuntu + upstream | 00:17 |
| JanC | tannewt: basically alphabetically with some added quirks ;) | 00:17 |
| tannewt | johanbr: hmm, I just do it based on tarballs for the kernel. I can't spend too much time on it because I'm tracking all packages not just the kernel | 00:19 |
| johanbr | oh, I see | 00:19 |
| tannewt | johanbr: probably not the smartest decision I've made :-) | 00:20 |
| tannewt | johanbr: for 8 distributions at this point too, oswatershed.org | 00:20 |
| johanbr | wow :) | 00:21 |
| johanbr | sounds like a pretty ambitious project | 00:21 |
| tannewt | johanbr: yeah, it is, its pretty cool data though | 00:21 |
| johanbr | "Parse error: syntax error, unexpected $end, expecting ')' in /var/www/oswatershed.org/htdocs/gen/data_2009032523.php on line 3" | 00:21 |
| tannewt | johanbr: hmm, let me look | 00:22 |
| tannewt | johanbr: I've been messing with things :-) | 00:23 |
| johanbr | :) | 00:23 |
| tannewt | johanbr: not much to see, I'm redoing it anyway | 00:23 |
| tannewt | johanbr: http://abstract.cs.washington.edu/~tannewt/ is another copy of it going | 00:23 |
| johanbr | alright | 00:24 |
| dtchen | how does it account for different distributions' stable/development release policies? | 00:25 |
| tannewt | dtchen: it compares stable releases versus different stable releases | 00:28 |
| tannewt | dtchen: in other words, it crawls multiple repos and classifies them as different branches of a single distro | 00:29 |
| tannewt | dtchen: past, lts, current, future and experimental are the current branch classifications | 00:29 |
| tannewt | dtchen: however, there is no independent notion of when a distro is stable | 00:31 |
| domas | hi! I'm staring at 2.6.24-18-server being idiotic: http://p.defau.lt/?WB6QRUQKJK19nVoZNQlNCA ;-) | 10:55 |
| domas | it swapped out a working process for 2G of 'cache' that isn't really used in any way | 10:55 |
| soren | domas: From here it looks like you're used all your RAM. | 11:11 |
| soren | 47048k free is not much, and you have less than 1 MB cached? | 11:11 |
| domas | soren: thats 1G cached | 11:14 |
| soren | In swap. | 11:22 |
| IntuitiveNipple | domas: 2G (almost) cached :) ... can you show the complete ps -efly - we can see mysqld is using almost all RAM already, so it wouldn't be surprising if it swaps, especially if the vm's 'optimistic' allocation has been caught out | 11:24 |
| domas | mysql is using 15G on 16G machine | 11:24 |
| domas | see, on most machines you'd say "1G for OS is enough" | 11:24 |
| domas | if I had _lots_ of processes eating 1G each, it wouldn't be an issue | 11:24 |
| domas | but it seems to be an issue with single big process | 11:25 |
| IntuitiveNipple | how big are the databases mysqld is handling? | 11:25 |
| domas | 200G | 11:26 |
| domas | (thats why I prefer mysqld to handle the dataset, not OS :) | 11:26 |
| IntuitiveNipple | Is it using temporary tables during processing? | 11:27 |
| domas | nope, none at all | 11:27 |
| domas | just writing few logs | 11:27 |
| domas | append-only operation, but they seem to take a fair share in the cache :) | 11:27 |
| domas | I can probably flush any cache with fadvise(MDONTNEED) :) | 11:27 |
| mdz | apw: I just got a new false positive with the apport suspend check | 11:28 |
| mdz | apw: is there anything I can check to help diagnose? | 11:29 |
| IntuitiveNipple | mdz: Is it because the resume took slightly over 5 seconds? | 11:29 |
| IntuitiveNipple | mdz: grep 'PM: resume devices took' /var/log/kern.log usually identifies that | 11:30 |
| soren | domas: "ps -efly" would still be convenient... | 11:31 |
| domas | http://p.defau.lt/?J_3IkiT90__D6PXxReo_EA | 11:32 |
| IntuitiveNipple | domas: the xargs -P 16 spawning the simultaneous jobs would explain the cache usage because all those gzip jobs need memory | 11:36 |
| domas | few megs each | 11:37 |
| domas | each | 11:37 |
| domas | but that would be not 'cached: 1G' | 11:37 |
| IntuitiveNipple | how much data is being sent in as a result of those? | 11:38 |
| domas | ~1MB packets | 11:38 |
| IntuitiveNipple | how many threads is mysqld using? | 11:43 |
| domas | 16 | 11:43 |
| IntuitiveNipple | what does this report: SHOW STATUS LIKE '%key_read%'; | 11:47 |
| domas | this is innodb mostly | 11:47 |
| domas | all i/o is single file | 11:47 |
| domas | 10000 for key_read ;-) | 11:48 |
| IntuitiveNipple | ok. | 11:48 |
| IntuitiveNipple | what does it show for mem usage then? SHOW INNODB STATUS; | 11:49 |
| domas | Total memory allocated 15965703304; in additional pool allocated 7059968 | 11:50 |
| domas | thats 14.8G | 11:51 |
| IntuitiveNipple | domas: I saw a reference to that which said "...Note that the real usage is maybe 2 - 4 times bigger than what it reports..." | 11:53 |
| domas | probably bad reference then | 11:53 |
| IntuitiveNipple | domas: in reference to the reported additional pool value | 11:53 |
| IntuitiveNipple | I doubt it, it was from one of the innodb dev's | 11:53 |
| domas | additional pool doesn't matter much | 11:54 |
| IntuitiveNipple | It will when the memory limit is hitting the RAM limit... if the additional pool is larger than that value it would explain the swap | 11:54 |
| domas | no, it is not | 11:55 |
| domas | :) | 11:55 |
| domas | mysql real rss should be ~15G | 11:55 |
| soren | I'm still not completely sure what the problem is. free says you've used all your RAM, so naturally, there's some swapping going on. | 11:55 |
| domas | yup, but then there's lots of caching | 11:55 |
| domas | which isn't needed | 11:55 |
| domas | I don't want that to have any VM pressure :) | 11:55 |
| soren | Swap caching. | 11:55 |
| soren | It's not caching files from the filesystem. | 11:56 |
| domas | Cached: 637916 kB | 11:56 |
| domas | SwapCached: 76608 kB | 11:56 |
| domas | well, I changed swapiness to 1 now | 11:56 |
| soren | domas: I'm still looking at http://p.defau.lt/?WB6QRUQKJK19nVoZNQlNCA | 11:57 |
| domas | thats file cache | 11:57 |
| soren | No. | 11:57 |
| soren | Well.. "that"? | 11:58 |
| soren | Which "that"? | 11:58 |
| domas | the 2G number | 11:58 |
| soren | I disagree. | 11:58 |
| domas | I checked meminfo, it was in 'cached:' line, not 'swapcached' | 11:58 |
| soren | Mind you, this is ancient knowledge, so it might have changed, but back in the good old days, what was reported as cached in swap was stuff that had been in swap recently, but was loaded back in to RAM, but hadn't been written to, so if it were to be swapped back out, it wouldn't have to be rewritten to disk. | 11:58 |
| domas | mhm | 11:59 |
| soren | I'm happy to be corrected if that's not the case anymore. | 11:59 |
| IntuitiveNipple | That is my understanding of it too | 12:01 |
| domas | right now after the swapiness decreased it seems that swap usage has fallen down, so does kswapd weirdness | 12:02 |
| domas | it is max 33% cpu now | 12:02 |
| domas | though the workload changed a bit | 12:03 |
| domas | ghm, maybe I didn't set O_DIRECT :) | 12:04 |
| IntuitiveNipple | what does vmstat show? lots of active swapping? | 12:04 |
| domas | nah | 12:04 |
| domas | around ~1MB/s swap-oin | 12:04 |
| domas | swap-in | 12:04 |
| domas | occasional swapout | 12:05 |
| IntuitiveNipple | O_DIRECT would help | 12:06 |
| IntuitiveNipple | soren: interesting overview: http://feedblog.org/2007/09/29/using-o_direct-on-linux-and-innodb-to-fix-swap-insanity/ | 12:06 |
| domas | I'm usually using O_DIRECT, the problem is that filesystems are stupid a bit | 12:06 |
| domas | only XFS allows parallel writing to O_DIRECT files, and even then it has limits (there have to be _no_ cached pages for that file at all) | 12:07 |
| domas | so you have to either flush oages with fadvise() or remount filesystems | 12:07 |
| domas | gotta patch mysql to do that for me | 12:08 |
| domas | btw, when process really goes oom, linux panics nowaday | 12:09 |
| domas | instead of killing the process | 12:09 |
| soren | domas: You can flush caches using /proc/sys/vm/drop_caches | 12:11 |
| domas | yep, not selective though | 12:11 |
| domas | (on the other hand, who cares about selectivity ;-) | 12:11 |
| domas | I have seen few more problems with kswapd behavior - especially with LVM. it seems that LVM marks some data required for snapshots as 'reclaimable', so kswapd goes crazy when gets some VM pressure and tries to free them up | 12:12 |
| IntuitiveNipple | I've just been looking at the Innodb tuning stuff, and the discussion of this particular issue. There it says, "...Make sure however the swapping is not happening ie your VMSTAT “si/so” columns are zero on Linux. Couple of swaps per minute would not hurt bad but if you’re doing 100+ pages per second of swap IO you’re in trouble." | 12:24 |
| IntuitiveNipple | See: http://www.mysqlperformanceblog.com/2007/11/03/choosing-innodb_buffer_pool_size/ | 12:24 |
| domas | IntuitiveNipple: I know that :) but I want OS not to swap my reasonably sized buffers | 12:29 |
| domas | I just want that 15G dataset would stay in memory :) | 12:30 |
| domas | on 16GB machine | 12:30 |
| domas | (or 30GB dataset on 32GB machine) | 12:31 |
| IntuitiveNipple | The recommendation suggests 14G for a 16G machine, so maybe your expectations aren't realistic | 12:34 |
| domas | this is 13G btw | 12:34 |
| Kano | hi rtg , could you build a 2.6.29 server kernel? | 12:34 |
| domas | IntuitiveNipple: I have worked on InnoDB memory consumption more than actual InnoDB devs, or mysqlperformanceblog people, or nearly anyone in the industry | 12:35 |
| dandel | domas, read this... http://www.novell.com/coolsolutions/feature/18990.html | 12:35 |
| domas | dandel: that link has too simplistic information | 12:35 |
| dandel | what does cat /proc/sys/vm/swappiness reveal tho? | 12:36 |
| domas | for now I solved it way easier, my process does not do any non O_DIRECT operation | 12:36 |
| domas | dandel: right now it is at 1 | 12:36 |
| domas | our default is 10 | 12:36 |
| dandel | could always do a ramdisk swap instead of hd based swap. | 12:37 |
| domas | dandel: the problem is not actual swapping, but kswapd taking extra cycles to manage the buffers | 12:37 |
| amitk | rtg: is the -11.37 tag set correctly? | 12:41 |
| rtg | amitk: checking | 12:42 |
| domas | IntuitiveNipple: actually, if anyone would set 14G buffer pool, they'd definitely go OOM | 12:42 |
| apw | Kano, cna't you build that from the karmic tree? | 12:42 |
| domas | IntuitiveNipple: that setting made sense on earlier versions, which didn't have high per-page mutex overhead :) | 12:42 |
| Kano | apw: i could only build the generic one | 12:42 |
| rtg | amitk: it looks close, why? | 12:43 |
| apw | Kano, why so? | 12:43 |
| Kano | i always got errors with the server one, not finding some modules or so | 12:43 |
| rtg | Kano: I'm running 2.6.29-1-server | 12:43 |
| amitk | rtg: 'close' ? I am tracking some regression in ARM flavours (versatile) due to AA being enabled. | 12:43 |
| Kano | rtg: hmm, maybe i need more free space then, but i deleted already much... | 12:44 |
| rtg | amitk: AFAICT the tag is planted right on the commit message that I would expect. | 12:44 |
| Kano | will test again. | 12:45 |
| amitk | rtg: but the commit after it (updateconfigs) was required | 12:45 |
| rtg | amitk: hmm, you might be right. I don't actually remember doing that, but I might have. (wouldn't be the first time) | 12:46 |
| rtg | amitk: I'm about to push some other stuff, so I'll fix the tag. | 12:46 |
| amitk | rtg: thanks. | 12:47 |
| amitk | rtg: any uploads planned? | 12:47 |
| rtg | amitk: yeah, I'd like to get one uploaded today. | 12:48 |
| rtg | amitk: ok, fixed the tag. the other commits I'm still testing. | 12:50 |
| amitk | rtg: could I push this trivial fix to linux-meta onto you then when you upload? https://bugs.edge.launchpad.net/ubuntu/+source/linux-meta/+bug/345992 | 12:51 |
| ubot3 | Malone bug 345992 in linux-meta "The one-line package description for the linux-image-imx51 and linux-image-versatile meta-packages appears wrong" [Undecided,Confirmed] | 12:51 |
| rtg | sure | 12:51 |
| domas | right, once all I/O is O_DIRECT, no swapping/caching is done | 13:35 |
| rtg | amitk: were you wanting to get something in today's upload? | 14:55 |
| === popey_ is now known as popey | ||
| amitk | rtg: I'm not done yet. Please go ahead. I'll commit tomorrow | 14:58 |
| rtg | amitk: I'm out tomorrow, so smb_tp will have to help if its something urgent. Otherwise, Monday | 14:58 |
| amitk | sure | 14:58 |
| smb_tp | Yessir :) | 14:58 |
| rtg | amitk: I'm doing a scorched earth Jaunty update on my dev box, so perhaps Monday is optimistic :) | 14:59 |
| amitk | rtg: :) | 15:08 |
| BUGabundo | apw: so I don't keep confusing things | 15:23 |
| BUGabundo | what's is different with backports kernel? | 15:23 |
| apw | backports modules not kernel | 15:23 |
| BUGabundo | ahh | 15:23 |
| apw | its a collection of updated drivers for things. currently wireless things which are moving fast | 15:23 |
| BUGabundo | so that's it | 15:24 |
| BUGabundo | so just about any one needs to have it? | 15:24 |
| BUGabundo | or only ppl with modules not compiled into kernel? | 15:24 |
| apw | people only need it if they have trouble with the main kernel drivers, or there is none | 15:25 |
| rtg | BUGabundo: only ppl that want the features offered by 2.6.29 wireless drivers and protocols | 15:25 |
| BUGabundo | ah so when I installe it my self without the need for it, am I doing it wrong? | 15:25 |
| apw | wrong is a relative term, you are putting together things which are less suited to each other | 15:26 |
| apw | they may work better or worse, better than nothing clearly | 15:26 |
| rtg | apw: actually, I think 2.6.29 wireless is beginning to work pretty well. | 15:27 |
| BUGabundo | since I have and intel 4965 | 15:27 |
| BUGabundo | I guess the kernel support should be fine | 15:28 |
| apw | yeah i am more saying that your mixing things, that is likely to add risk, but is the only way for people who need it | 15:28 |
| BUGabundo | no great changes recently | 15:28 |
| rtg | BUGabundo: if the 2.6.28 driver is working for you, then why change? | 15:28 |
| BUGabundo | ehe | 15:28 |
| BUGabundo | don't fix what aint broken | 15:28 |
| apw | indeed | 15:29 |
| * BUGabundo looks at installed stuff | 15:29 | |
| BUGabundo | 'cause I remember that change to kernel to fix kill_switch | 15:29 |
| BUGabundo | came on backports | 15:29 |
| apw | that'd be the definition of not working for you | 15:29 |
| laga | jau | 15:30 |
| laga | oops | 15:30 |
| laga | wrong window, sorry | 15:30 |
| BUGabundo | actually haven't tested the soft switch | 15:30 |
| kees | amitk: if you need help with AA, ask jjohansen on #apparmor on oftc, he's usually around and very helpful. | 15:49 |
| amitk | kees: we got it fixed for now. Thanks | 15:53 |
| amitk | kees: I did get in touch with him btw. | 15:53 |
| kees | amitk: ah, excellent | 16:30 |
| === cropalato_ is now known as cropalato | ||
| apw | dtchen, those two alsa write pointer fixes ... do they fix up the issues with pa and glitch-free do you know? | 17:16 |
| bradf | bradf: test | 18:42 |
| bradf | bradf: another | 18:43 |
| bradf | bradf: another | 18:43 |
| === bradf_afk is now known as bradf | ||
| === calc_ is now known as calc | ||
| === tjaalton_ is now known as tjaalton | ||
| dtchen | apw: yes | 22:18 |
| apw | dtchen, most excellent | 22:18 |
| dtchen | apw: note that jaunty's pulseaudio version is a bit dated, so performance will be less optimal than with 0.9.15ish (or whatever Fedora 11 will have) | 22:18 |
| dtchen | still, users have been reporting markedly improved stability and performance | 22:19 |
| TheMuso | It seems that Lennart aims his releases for Fedora's release schedule, which is always going to mean we're behind the 8 ball so to speak. | 22:20 |
| TheMuso | As far as I understand things anyway. | 22:21 |
| dtchen | right, unless we obtain an exception for PA similar to what the GNOME desktop has | 22:22 |
| dtchen | of course that would mean updating the ALSA stack, too, which uh... | 22:22 |
| dtchen | anyhow, happy with what we're given | 22:23 |
| TheMuso | dtchen: Yep totally agree. | 22:26 |
| TheMuso | cd | 22:26 |
| TheMuso | woops wrong tab | 22:26 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!