[10:20] <Wavesonics> does anyone know if Ubuntu 8.10 will have ext4 enabled?
[10:38] <Keybuk> it does
[10:38] <Keybuk> not by default though
[10:39] <Keybuk> the kernel module is there, and the userland tools are too
[14:53] <Keybuk> rtg: so Corey Minyard replied and said Signed-off-by: him ...
[14:53] <Keybuk> does that mean it's in the upstream kernel now?
[14:53] <Keybuk> who has to commit it?
[14:54] <rtg> Keybuk: no, I still have to send it to akpm.
[14:54] <rtg> I though they'd have a maintainer's tree, but I guess not.
[14:54] <Keybuk> how do you do that?
[14:55] <rtg> Keybuk: I'll have to figure it out. Its been awhile since I sent anything to Andrew.
[14:56] <rtg> Keybuk: once I do send it, you'll get 1 or more automated responses that tell you when the patch transitions states
[14:59] <Keybuk> so, let me work this out
[14:59] <Keybuk> you first send the patch to the subsystem maintainer
[14:59] <Keybuk> cc lkml, just for kicks
[14:59] <Keybuk> and _if_ you think it might be worth a stable update later, add a Cc: stable@kernel.org in the body
[14:59] <Keybuk> the subsystem maintainer might commit it themselves to their own tree
[14:59] <Keybuk> or they might just reply with another signed-off-by line?
[15:00] <Keybuk> at that point, I guess you *reformat* the patch to include that new signed-off-by line, and then send it to someone else?
[15:00] <rtg> Keybuk: the subsystem maintainer _should_ commit to their tree. Seems like Cory is being lazy, or they don't actually have a subsystem tree. After all, isn't that the role of a subsystem maintainer?
[15:01] <Keybuk> why would you have a subsystem tree for every single driver?
[15:01] <rtg> Anyway, the Cc is to notify stable when the patch is included in Linus tree.
[15:02] <Keybuk> what if there's no subsystem?
[15:02] <rtg> Reformatting the patch with an extended SOB would normally happen when he maintainer added it to their tree.
[15:03] <rtg> Keybuk: All drivers belong to a subsystem at some layer.
[15:03] <Keybuk> let's say I get around to writing waitfd()
[15:03] <Keybuk> who would I send that patch to ?
[15:04] <rtg> Keybuk: you're getting pretty high up the stack. You'll have to get new syscalls ack'd by Linus' first level lieutenants. Guys like  Garzik, Miller, Morton, etc.
[15:05] <rtg> suffice it to say, there  are no hard and fast rules for patch submission.
[15:05] <rtg> its kind of fluid :)
[15:05] <rtg> generally, someone will tell you what to do if you've transgressed.
[15:51] <rtg> just setup a Netgear WNDR3300 (RangeMax Dual Band Wireless-N Router). Seems like it's working pretty good.
[16:22]  * apw wonders if you are using it as a router, or have installed intrepid on it :)
[16:30] <rtg> nope, just as a router. I'm using it to test wireless stuff, so I don't want to introduxce any more problems to the mix.
[16:36] <hallyn> so is there any chance at all of the freezer control group (just pulled into Linus' tree a few days ago) making it into the intrepid kernel?
[16:38] <rtg> hallyn: this close to release? I don't think so.
[16:39] <hallyn> drat
[16:41] <lamont> NCommander: around?
[16:41] <NCommander> lamont, yup
[16:42] <hallyn> guess i'll see how trivially i can pull those patches into intrepid.git
[16:45] <NCommander> lamont, what's up?
[18:58] <Ng> do any of you guys have USB 3g modems? they pretend to be CD drives until you eject them, then they become modems. I just tried two different models on two laptops and they just spew loads of scsi errors
[18:59] <Ng> ./win 245
[18:59] <Ng> gar
[19:19] <calc> 100 for /proc/sys/vm/vfs_cache_pressure seems too low at least on laptops
[19:19] <calc> anyone around for a discussion about it?
[19:23] <jdong> calc: you mean too high?
[19:23] <calc> jdong: needs to be a larger number it seems or something else is causing problems during high disk io
[19:23] <calc> jdong: my system is almost completely unresponsive when i do a transmission torrent check
[19:23] <calc> jdong: can't use evolution or even pidgin at times, not even repainting the screen
[19:24] <jdong> calc: isn't that more of an IO scheduler issue?
[19:24] <calc> jdong: probably so, someone pointed me at this tuning option but i think its probably a io scheduler issue
[19:24] <calc> jdong: i don't know enough about how disk io works under linux to know what is the problem
[19:25] <jdong> calc: do you see any evidence of swapping?
[19:25] <jdong> calc: if you're swapping it could possibly be a vfs_cache_pressure or vm.swappiness issue
[19:25] <calc> 4MB usage in swap
[19:25] <calc> ~ 2300MB in buffers
[19:25] <jdong> calc: yeah that's not cache pressure then
[19:25] <calc> er not buffers
[19:25] <calc> 2300MB in cached, sorry about that
[19:25] <calc> i'm checking a 28GB torrent
[19:25] <jdong> calc: that's your IO scheduler not prioritizing the smaller requests needed to check your e-mail vs the 5GB of torrent data :)
[19:25] <jdong> err, 28GB :)
[19:25] <calc> heh
[19:26] <calc> as this channel is logged i'll leave it at that ;-)
[19:26] <calc> but it seems in general not to fairly divide up disk io between apps
[19:27] <calc> to the point that pidgin didn't repaint for what seemed like at least 30s, i finally switched to my terminal window, so i don't know how long it actually took
[19:27] <jdong> calc: well it fairly divides between IO requests
[19:27] <jdong> calc: doesn't help if your torrent client is making a lot of said requests :)
[19:27] <jdong> calc: have you tried to see if the deadline IO scheduler works any better for your workload?
[19:28] <jdong> echo deadline > /sys/block/sda/queue/scheduler
[19:28] <calc> jdong: how do i switch to it?
[19:28] <calc> oh ok
[19:28] <calc> is that safe to do while under heavy disk io?
[19:28] <jdong> yeah
[19:28] <jdong> they're designed to be online-switchable
[19:28] <calc> ok
[19:29] <calc> that is amazingly better
[19:29] <calc> completely responsive
[19:30] <calc> hmm maybe i should switch back to verify its actually not a placebo thing
[19:30] <calc> what was the other one called?
[19:31] <calc> it definitely seems more responsive than it was, but it might be due to other reasons, need to try flipping back and forth
[19:32] <steve__> hrm, it appears deadline is the default scheduler here: 
[19:32] <steve__> [denisovich ~]$ cat /sys/block/sda/queue/scheduler
[19:32] <steve__> noop anticipatory [deadline] cfq
[19:33] <calc> i switched to cfq to see what it does
[19:33] <paran> calc: cfq is the default for -generic kernel
[19:34] <calc> ok
[19:35] <sbeattie> ah, right, that machine is running the -kernel server
[19:35] <paran> calc: in my experience excessive disk-io when using bittorrent is often caused by fragmentation in the (ext3) file system
[19:35] <calc> well deadline does seem more responsive to a non-scientific click on my mailboxes in evolution test
[19:35] <calc> paran: well doing a torrent check causes a lot of disk io also
[19:36] <calc> and apparently helps to expose holes in the scheduler
[19:36] <calc> hmm when i waited a bit to let cache purge my mailboxes it seems deadline might not help much after all
[19:37] <calc> so i really can't tell :\
[19:37] <paran> calc: of course, but the disk IO will hurt a lot more if it is mostly seeks, cause of fragmentation
[19:37] <calc> yea
[19:37] <calc> so is there any way to defrag ext3? :)
[19:37] <calc> everyone claimed that ext3 never needed to be defraged in the past
[19:37] <paran> calc: you can use "filefrag <filename>" to see how fragmented a file is
[19:38] <calc> really fraged
[19:38] <paran> calc: as far as I know you can't. the problem is that ext3 do not like the usage pattern of a typical torrent client
[19:38] <calc> eg 9915 extents found perfection would be 3
[19:39] <paran> calc: you should get a client that pre allocates (writes the whole file full with zeroes) before starting the download
[19:39] <calc> paran: well the default one in ubuntu that is preinstalled doesn't do that, so maybe i should file a bug :\
[19:40] <calc> though i think filing a bug about the lack of a defragger when it is clearly really needed would also be good
[19:40] <calc> jdong: you made one i see in google ;-)
[19:41] <jdong> calc: haha :)
[19:42] <jdong> calc: something else to consider with CFQ is the use of ionice priorities
[19:42] <paran> calc: the problem is that most people would then file bugs complaining about their torrents not starting right away
[19:42] <jdong> calc: you can punt your torrent client to best-effort priority 8
[19:42] <calc> jdong: ah
[19:42] <jdong> paran: ext4 will support preallocation right?
[19:42] <jdong> i.e. KTorrent with XFS uses the XFS preallocation calls.
[19:43] <jdong> takes a fraction of a second to reserve optimal contiguous space for the entire torrent
[19:43] <calc> jdong: so is your defrag tool safe? :)
[19:43] <jdong> calc: it is basically a cp a b; mv b a;
[19:43] <jdong> calc: as long as your filesystem treats mv b a atomically it should be safe
[19:43] <jdong> worst case is leaving behind a tempfile
[19:43] <calc> ok
[19:44] <calc> where is the official copy of this tool? :)
[19:44] <paran> jdong: I haven't really followed ext4, but I think so
[19:44] <jdong> calc: you're a pretty experienced hand at UNIX, you probably can just do it by hand for problematic files :)
[19:44] <jdong> calc: it should be in bzr somewhere on launchpad
[19:44] <jdong> (lol)
[19:44] <calc> lol
[19:44] <calc> ah i see it
[19:44] <calc> 3rd hit on google
[19:45] <calc> it seems you have lots of empty branches on lp that you forgot to delete
[19:45] <jdong> haha that could very well be
[19:46] <jdong> calc: you know, I've always considered writing a hackish script that's the equivalent of Windows's foreground-IO-booster
[19:46] <jdong> the default behavior of workstation builds of Windows is to allocate a higher priority for the foreground app
[19:46] <jdong> that might arguably help interactiveness a bit if you are runing a torrent client and switch to your e-mail client
[19:49] <calc> yea
[19:49] <paran> do the defragmenting program you are talking about now simply moves files between file systems?
[19:50] <calc> jdong: defrag is pretty cool :)
[19:50] <jdong> paran: pretty much it makes a copy of the file and checks if the new copy has fewer fragments according to filefrags
[19:50] <calc> of course it would be nice if ext3/ext4 or something like that had built in defrag option
[19:50] <jdong> paran: it's the same strategy xfs_fsr uses, except XFS has a efficient way to sort by most fragmented files
[19:50] <calc> iirc XFS has something like that
[19:50] <jdong> yup
[19:51] <calc> when i tried it was buggy though and ate my fs ;-)
[19:51] <jdong> calc: well it's as safe as XFS is ;-)
[19:51] <jdong> what it does is basically equivalent of cp and mv operations except it uses an atomic mv
[19:51] <calc> jdong: heh
[19:51] <jdong> (XFS mv is NOT atomic by default)
[19:51] <jdong> that IMO is insane
[19:51] <jdong> but technically POSIX-compliant :D
[19:51] <paran> jdong: that is pretty much the only thing you can do from user space. I have defragmented files by moving from ext3 to a tmpfs and back :)
[19:52] <jdong> paran: yeah, I don't know (or WANt to know!) what that does to free space fragmentation
[19:52] <jdong> but empirically it helps a lot with file fragmentation and read performance
[19:52] <calc> jdong: does defrag have an option to skip files that are mostly defraged already?
[19:52] <jdong> calc: there should be a frags/MB threshold commandline flag
[19:52] <calc> ok
[19:53] <jdong> calc: by default IIRC it is pretty sane, like around 2frags/MB?
[19:53] <calc> oh ok
[19:53] <jdong> paran: another neat trick that's worked wonders for me: tar and remove all the files mentioned in readahead/boot, then restore them at once from the tar
[19:53] <calc> i didn't look to see what it defaulted to
[19:53] <jdong> paran: that's managed to shave 10-15% time off readahead for me
[19:54] <jdong> I'm guessing EXT3 is more likely to allocate files in the same area of the disk if created at the same time?
[19:54] <calc> what would be neat is a defrag that could work like norton defrag on windows (or how it used to work) where you could even select file order, like what you are basically doing with tar
[19:55] <paran> jdong: yeah, that should work.
[19:56] <jdong> calc: yeah I wish we actually had a defrag API call!
[19:56] <jdong> (ext4 feature)
[19:56] <calc> and have option to defrag order most recent to oldest accessed files on the disk, that would help for non ssd's
[19:56] <calc> jdong: oh its in ext4?
[19:56] <paran> in the mean time you could always use XFS.
[19:56] <jdong> calc: Windows is nice that it gives an atomic defrag function that basically takes a inode and place to put it
[19:57] <calc> paran: XFS is too scary for me after it eating my disk multiple times in the past :\
[19:58] <paran> calc: oh? we have many TiB on XFS at work, no problems and much better performance than ext3.
[19:59] <calc> paran: well the last time i used XFS was quite a while back > 4 years ago so it probably has improved
[19:59] <calc> but it was very very unreliable back then
[19:59] <jdong> calc: XFS used to have a lot of out-of-order write hazard bugs back then
[19:59] <jdong> well s/bugs/'features'/
[19:59] <jdong> it's better
[19:59] <jdong> not as reliable as ext3 for irresponsible plug-pulling but should be good enough in gneeral
[19:59] <calc> after seeing its bugs and reiserfs bugs i decided to go with what everyone uses since it tends to be more debugged, eg ext3 for now
[19:59] <jdong> I put all my torrents on XFS for performance
[19:59] <jdong> my data and OS is on ext3
[19:59] <jdong> XFS's small-file performance still sucks
[20:01] <paran> the only thing I remember is that XFS on 32 bit x86 used to be problematic, but it was years since we installed anything not x86_64 :)
[20:01] <calc> jdong: is defrag smart enough not to cross mount points?
[20:02]  * calc considers running it on /
[20:02] <calc> paran: i'm still having to use i386 since resume doesn't work right on my laptop on amd64
[20:05] <calc> wow transmission manages to fairly consistently frag files ~ 32frag/MB
[20:37] <ARCKEDA> WTF/
[20:46] <calc> so ext4 became 'stable' as of yesterday?
[20:47] <calc> apparently fedora 10 is going to use it
[20:48] <calc> hmm actually that happened on oct 11, wikipedia seems to be conflicting itself
[20:57] <NCommander> calc, well, fix the confliction