| BenC | if 2.6.20-14 works for him, then there's no reason 2.6.20-15 shouldn't | 12:24 |
|---|---|---|
| BenC | mjg59: is the dont_disable_c3 patch still needed for ipw2100? | 12:25 |
| BenC | mjg59: I don't recall any bug reports that benefited from it | 12:25 |
| mjg59 | It was never a bug-fixing measure | 12:26 |
| mjg59 | It was a "I'm willing to accept some packet corruption in return for better battery life" one | 12:26 |
| BenC | do we really want it in there? | 12:29 |
| mjg59 | I find it useful | 12:29 |
| mjg59 | By default it does nothing | 12:30 |
| BenC | Can you get it upstream? | 12:30 |
| mjg59 | I'll try to, when I push the other ipw2100 stuff I have | 12:33 |
| BenC | Great, thanks | 12:33 |
| === marcin_ant [n=marcin@194.114.146.126] has joined #ubuntu-kernel | ||
| kylem | win 23 | 01:28 |
| === defendguin_ [n=supertux@cpe-72-181-7-135.houston.res.rr.com] has joined #ubuntu-kernel | ||
| BenC | I hate when laptop manufacturers use 2x512Meg for 1G of mem, means you can't just add another 1G stick to get 2G | 01:47 |
| kylem | pretty common cost saving measure... | 01:47 |
| BenC | my gateway came with 1G stick | 01:47 |
| BenC | or wait, maybe it was the HP | 01:47 |
| === gnomefreak [n=gnomefre@ubuntu/member/gnomefreak] has joined #ubuntu-kernel | ||
| BenC | one of them did anyway :) | 01:48 |
| === m0rg0th [n=manugarg@219.64.120.181] has joined #ubuntu-kernel | ||
| === johanbr [n=j@blk-224-156-151.eastlink.ca] has joined #ubuntu-kernel | ||
| === marcin_ant [n=marcin@194.114.146.126] has joined #ubuntu-kernel | ||
| === vimalg2 [n=test@202.56.231.116] has joined #ubuntu-kernel | ||
| === jdong [n=jdong@ubuntu/member/jdong] has joined #ubuntu-kernel | ||
| === vimalg2 [n=test@202.56.231.116] has left #ubuntu-kernel [] | ||
| jdong | BenC: ping | 02:29 |
| jdong | Can someone look at bug https://bugs.launchpad.net/ubuntu/+source/linux-restricted-modules-2.6.20/+bug/101984 | 02:30 |
| jdong | "firegl_public.c: Ubuntu modification uses obsolete __syscall_return" | 02:30 |
| mjg59 | jdong: Given that it's all a huge hack in the first place, what's the impact of this? | 02:34 |
| jdong | I'm not sure what the impact is | 02:35 |
| jdong | that's why I asked | 02:35 |
| jdong | some user bugged me about it.... | 02:35 |
| mjg59 | Why? | 02:36 |
| === jdong shrugs | ||
| jdong | he referred to it as an 'annoying bug' | 02:37 |
| mjg59 | Yeah, but how? | 02:38 |
| BenC | from what I can tell it means people have a harder time using custom kernels and fglrx-kernel-source | 02:41 |
| BenC | which means I don't care | 02:41 |
| === jdong shrugs | ||
| jdong | oh well :) | 02:42 |
| BenC | the patch we applied came from fglrx source, so it's not a patch we created | 02:43 |
| BenC | s/from/for/ | 02:43 |
| BenC | nm, I meant from | 02:43 |
| BenC | mjg59: the dongle_id = 0x9 patch for IBM0071 in nsc-ircc, is that a patch you authored? | 02:45 |
| mjg59 | Yeah | 02:45 |
| BenC | mjg59: Can I send it upstream with your signed-off-by? | 02:45 |
| mjg59 | My experience is that all the IBM ones have that dongle type | 02:45 |
| mjg59 | So sure - I don't /absolutely/ guarantee that it's correct, but I think it's better than autodetection | 02:46 |
| BenC | mjg59: what do you think we should do with the raw keycode patch for input.c, was there ever any discussion about that on lkml or something? | 02:47 |
| BenC | we've been carrying it around for several releases now | 02:49 |
| mjg59 | Oh, to pass through whether it was in the filter or not? | 02:51 |
| mjg59 | Can't remember. I think it came up on lkml at some point, but I'll look into it. | 02:52 |
| BenC | mjg59: right, that one...thanks | 03:04 |
| kylem | moo. | 03:04 |
| === rushfan_ [n=g2@cpe-066-057-009-238.nc.res.rr.com] has joined #ubuntu-kernel | ||
| === jdong [n=jdong@ubuntu/member/jdong] has left #ubuntu-kernel ["FCKGW-RHQQ2-YXRKT-8TG6W-2B7Q8"] | ||
| === MrNOKIA1 [n=user@86.121.182.171] has joined #ubuntu-kernel | ||
| === m0rg0th [n=manugarg@219.64.120.181] has joined #ubuntu-kernel | ||
| rushfan_ | BenC is .27 supposed to have the sata_nv fix in it ? | 03:32 |
| rushfan_ | I'm [g2] | 03:32 |
| BenC | yes | 03:41 |
| rushfan_ | BenC I was running .27 and I think I had a sata command response failure on my NForce4 board | 03:42 |
| rushfan_ | with regard to the command time outs | 03:42 |
| rushfan_ | I just loaded your .26 kernel and it seems to be working so far | 03:43 |
| rushfan_ | unfortunately this is my pvr box and the .26 kernel doesn't have the ivtv drivers | 03:44 |
| === rushfan_ fires up bonnie++ | ||
| BenC | it has the drivers, just not the firmware | 03:49 |
| BenC | copy the firmware | 03:49 |
| BenC | rushfan_: my .26 kernel is _exactly_ the same as the archive's -15.27 kernel | 03:49 |
| BenC | brb, doing some hw juggling | 03:50 |
| === m0rg0th [n=manugarg@219.64.120.181] has joined #ubuntu-kernel | ||
| === kkubasik [n=kjk38@kjk38-laptop.STUDENT.CWRU.Edu] has joined #ubuntu-kernel | ||
| === philwyett [n=philip@bb-87-81-146-45.ukonline.co.uk] has joined #ubuntu-kernel | ||
| === BenC [n=bcollins@debian/developer/bcollins] has joined #ubuntu-kernel | ||
| === m0rg0th [n=manugarg@219.64.120.181] has joined #ubuntu-kernel | ||
| === defendguin_ [n=supertux@cpe-72-181-7-135.houston.res.rr.com] has joined #ubuntu-kernel | ||
| === macd [n=d@68-190-68-87.dhcp.slid.la.charter.com] has joined #ubuntu-kernel | ||
| === m0rg0th [n=manugarg@72.14.231.89] has joined #ubuntu-kernel | ||
| === ivoks [n=ivoks@3-178.dsl.iskon.hr] has joined #ubuntu-kernel | ||
| === bdgraue [n=bdgraue@dyndsl-085-016-081-162.ewe-ip-backbone.de] has joined #ubuntu-kernel | ||
| === m0rg0th [n=manugarg@72.14.231.89] has joined #ubuntu-kernel | ||
| === doko [n=doko@dslb-088-073-122-210.pools.arcor-ip.net] has joined #ubuntu-kernel | ||
| === bronson_ [n=bronson@adsl-76-202-197-80.dsl.pltn13.sbcglobal.net] has joined #ubuntu-kernel | ||
| === Kream [n=karim@122.163.179.157] has joined #ubuntu-kernel | ||
| === dholbach [n=daniel@i59F7635B.versanet.de] has joined #ubuntu-kernel | ||
| dholbach | good morning | 08:19 |
| === Lure [n=lure@ubuntu/member/lure] has joined #ubuntu-kernel | ||
| === ivoks [n=ivoks@20-32.dsl.iskon.hr] has joined #ubuntu-kernel | ||
| === anibal [n=anibal@debian/developer/anibal] has joined #ubuntu-kernel | ||
| === abogani [n=abogani@adsl203-157-083.mclink.it] has joined #ubuntu-kernel | ||
| === ivoks_ [n=ivoks@36-61.dsl.iskon.hr] has joined #ubuntu-kernel | ||
| === jml [n=jml@59.167.203.115] has joined #ubuntu-kernel | ||
| === cassidy [n=cassidy@host-213-189-171-21.brutele.be] has joined #ubuntu-kernel | ||
| === kkubasik [n=kjk38@kjk38-laptop.STUDENT.CWRU.Edu] has left #ubuntu-kernel [] | ||
| === reitblatt [n=mark@w-mob101-128-62-91-44.public.utexas.edu] has joined #ubuntu-kernel | ||
| === kkubasik [n=kjk38@kjk38-laptop.STUDENT.CWRU.Edu] has joined #ubuntu-kernel | ||
| === kkubasik_ [n=kjk38@kjk38-laptop.STUDENT.CWRU.Edu] has joined #ubuntu-kernel | ||
| === smurfix [n=smurf@debian/developer/smurf] has joined #ubuntu-kernel | ||
| === tijsco [n=matthijs@82.73.208.11] has joined #ubuntu-kernel | ||
| === Keybuk [n=scott@yttrium.canonical.com] has joined #ubuntu-kernel | ||
| === marcin_ant [n=marcin@194.114.146.126] has joined #ubuntu-kernel | ||
| === kkubasik_ [n=kjk38@kjk38-laptop.STUDENT.CWRU.Edu] has joined #ubuntu-kernel | ||
| === philwyett [n=philip@bb-87-81-146-45.ukonline.co.uk] has joined #ubuntu-kernel | ||
| === gortiz [n=gortiz@88.87.105.4] has joined #ubuntu-kernel | ||
| === EtienneG [n=etienne@ubuntu/member/EtienneG] has joined #ubuntu-kernel | ||
| === saispo [n=saispo@ryu.zarb.org] has joined #ubuntu-kernel | ||
| saispo | hi | 01:47 |
| === dholbach_ [n=daniel@i59F707C1.versanet.de] has joined #ubuntu-kernel | ||
| === abogani [n=abogani@adsl203-157-083.mclink.it] has joined #ubuntu-kernel | ||
| === AlinuxOS [n=vsichi@host177-127-dynamic.2-79-r.retail.telecomitalia.it] has joined #ubuntu-kernel | ||
| === reitblatt [n=mark@w-mob400-128-62-216-13.public.utexas.edu] has joined #ubuntu-kernel | ||
| === infinity2 [n=adconrad@cerberus.0c3.net] has joined #ubuntu-kernel | ||
| === _MMA_ [n=mma@cpe-071-070-203-016.nc.res.rr.com] has joined #ubuntu-kernel | ||
| === zul_ [n=chuck@CPE0006258ec6c1-CM000a73655d0e.cpe.net.cable.rogers.com] has joined #ubuntu-kernel | ||
| === hans [n=hans@5634185c.rev.stofanet.dk] has joined #ubuntu-kernel | ||
| === kkubasik [n=kjk38@kjk38-laptop.STUDENT.CWRU.Edu] has joined #ubuntu-kernel | ||
| === d1uluv2h8 [i=0@89.33.166.2] has joined #ubuntu-kernel | ||
| === ivoks [n=ivoks@3-126.dsl.iskon.hr] has joined #ubuntu-kernel | ||
| zul | is there still a meeting going on today? | 04:54 |
| === tuxmaniac [n=aanjhan@unaffiliated/tuxmaniac] has joined #ubuntu-kernel | ||
| === rajeevindus [n=rajeev@124.125.162.178] has joined #ubuntu-kernel | ||
| === rajeevindus [n=rajeev@124.125.162.178] has left #ubuntu-kernel [] | ||
| === Eruantalon [n=hans@5634185c.rev.stofanet.dk] has joined #ubuntu-kernel | ||
| Keybuk | so | 05:24 |
| Keybuk | you remember that "VMware takes an hour to install feisty" thing? | 05:24 |
| === Keybuk can replicate that every time | ||
| Keybuk | AND I can make it take only 15-20 min | 05:25 |
| kkubasik | there all in a meeting right now | 05:29 |
| mjg59 | Keybuk: Oh? | 05:29 |
| === pmjdebruijn [n=pmjdebru@pmjdebruijn.xs4all.nl] has joined #ubuntu-kernel | ||
| === mjg59_ [n=mjg59@cavan.codon.org.uk] has joined #ubuntu-kernel | ||
| Keybuk | let me just reverse the change, and make sure this goes back to 1h+ installs | 05:40 |
| mjg59_ | Keybuk: Sorry, missed the last few lines. What was the change? | 05:40 |
| Keybuk | I increased the amount of memory I gave to the VM | 05:41 |
| Keybuk | from 256MB to 512MB | 05:41 |
| Nafallo | mjg59_: no you didn't miss anything :-) | 05:41 |
| mjg59_ | So, uh. Ubuntu installs slowly in 256MB of RAM. This is unsurprising, surely? | 05:41 |
| Keybuk | no, it's very surprising | 05:42 |
| Keybuk | it's a very recent change | 05:42 |
| Keybuk | the LiveCD is almost unusable in 256MB | 05:42 |
| Keybuk | where as beta is fine | 05:43 |
| mjg59_ | Well, it certainly doesn't sound like a kernel issue... | 05:43 |
| Keybuk | unless it's caused by the squashfs change using more memory? | 05:43 |
| Keybuk | (or it could indeed be completely unrelated) | 05:43 |
| mjg59_ | Well, how much free ram do you have after boot, compared with beta? | 05:44 |
| Keybuk | that's what I'm about to look at | 05:44 |
| Keybuk | our original thought was that this was an atapi problem | 05:44 |
| Keybuk | but pure I/O seems fine | 05:44 |
| === Keybuk is still waiting for current to boot | ||
| === mjg59 [n=mjg59@cavan.codon.org.uk] has joined #ubuntu-kernel | ||
| === mjg59_ [n=mjg59@cavan.codon.org.uk] has joined #ubuntu-kernel | ||
| mjg59 | Hey wow -powerpc still actually boots Mac hardware | 05:54 |
| mjg59 | Reassuring | 05:55 |
| cjwatson | 256MB was fine with edgy | 05:57 |
| cjwatson | not exactly stellar or anything, but not crushing | 05:57 |
| Keybuk | free doesn't return bad numbers, 128MB free before page cache | 05:59 |
| kylem | argh. | 05:59 |
| kylem | i'm a muppet. | 05:59 |
| Keybuk | cjwatson: did you revert your system to a snapshot before you tried your test install? | 06:00 |
| cjwatson | Keybuk: I think it was clean boot + install | 06:01 |
| cjwatson | certainly no snapshot involved | 06:01 |
| Keybuk | was the drive clean though? | 06:02 |
| Keybuk | casper mounts any swap partition it can find on boot, remember | 06:03 |
| === johanbr [n=j@JBrannlund.MathStat.Dal.Ca] has joined #ubuntu-kernel | ||
| Keybuk | so if you already had an install, you would've had some swap | 06:03 |
| cjwatson | Keybuk: that VM is set to 700MB ... | 06:03 |
| cjwatson | I suspect it doesn't matter ;-) | 06:03 |
| Keybuk | ahh | 06:03 |
| Keybuk | mdz and I both leave ours at 256MB as a matter of course | 06:04 |
| Keybuk | could you try with 256 and see how it boots/installs? | 06:04 |
| cjwatson | that may well explain it | 06:04 |
| cjwatson | sure | 06:04 |
| cjwatson | wah, none of my shift keys work any more | 06:05 |
| cjwatson | hang on while I sort this out ;-) | 06:06 |
| cjwatson | boots in 2:13 | 06:10 |
| cjwatson | subjectively, the slowness is primarily after X starts | 06:10 |
| === lmierzej [n=lmierzej@drc242.neoplus.adsl.tpnet.pl] has joined #ubuntu-kernel | ||
| Keybuk | agree | 06:11 |
| Keybuk | right when GNOME hauls its fat, bloaty arse into memory | 06:11 |
| cjwatson | free after cache is 155MB | 06:11 |
| Keybuk | cache includes loaded apps though | 06:11 |
| Keybuk | iirc | 06:11 |
| Keybuk | ie. mapped bits of .so files, binaries | 06:11 |
| cjwatson | ubiquity is liable to eat at least 50MB, possibly more | 06:11 |
| cjwatson | most of which I think is debconf | 06:12 |
| Keybuk | I get 104MB plus cache with ubiq | 06:12 |
| cjwatson | what's the c-a-f1 rune in vmware again? | 06:12 |
| Keybuk | c-a-space-f1 | 06:13 |
| Keybuk | ie, hold ctrl alt, press space, then f1 | 06:13 |
| Keybuk | I think | 06:13 |
| cjwatson | thanks | 06:13 |
| cjwatson | 129MB right after starting ubiquity | 06:13 |
| Keybuk | 97MB once install begins | 06:15 |
| === Keybuk just hit zero | ||
| cjwatson | ubiquity itself has 24MB res 10MB of which shared | 06:15 |
| Keybuk | so now it's swapping to the same disk it's installing *to* to unpack the squashfs of the disk it's reading *from* | 06:16 |
| cjwatson | debconf-communicate has 15MB res | 06:16 |
| cjwatson | can you look at top and see what's using memory? | 06:16 |
| Keybuk | u6y with 10.3% | 06:17 |
| cjwatson | growing as files are copied? | 06:18 |
| Keybuk | no | 06:18 |
| cjwatson | 10.3% sounds like about baseline | 06:18 |
| cjwatson | that's similar to the 24MB I had above right after starting it | 06:19 |
| cjwatson | so what's using MORE memory? | 06:19 |
| cjwatson | I'm around 90MB+ free after cache, copying files | 06:21 |
| cjwatson | if anything, it's growing slightly | 06:21 |
| Keybuk | right | 06:21 |
| Keybuk | nothing's growing in userland | 06:21 |
| Keybuk | how do I find out how much memory the kernel is hogging for itself? | 06:21 |
| cjwatson | where are you getting your zero number? | 06:21 |
| Keybuk | free | 06:21 |
| Keybuk | it hit zero free after cache, and sat and swapped for a few seconds | 06:21 |
| cjwatson | yeah, see I'm not getting zero free after cache | 06:22 |
| cjwatson | this is the -/+ buffers/cache: ... (free) column? | 06:22 |
| === d1uluv2h8 [i=0@89.33.166.2] has left #ubuntu-kernel [] | ||
| cjwatson | that's settled down at 106MB | 06:23 |
| Keybuk | it only hit it once | 06:23 |
| Keybuk | then it swapped a huge amount and has stayed around 70MB since | 06:23 |
| cjwatson | maybe I should look at this in gnome-terminal rather than a console | 06:23 |
| cjwatson | just worried about g-t's own bloatiness | 06:23 |
| cjwatson | I'm not seeing your memory problems, but I do have to agree that this isn't fast | 06:25 |
| cjwatson | it's 5+ minutes in by vmware's clock and only at 29% | 06:25 |
| === ^^CatTuX^^ [n=Shoaibi@mbl-65-129-173.dsl.net.pk] has joined #UBUNTU-KERNEL | ||
| === FOAD_ [n=dok@dinah.blub.net] has joined #ubuntu-kernel | ||
| Keybuk | I can't see why the memory is a problem | 06:31 |
| Keybuk | all I know is that if I increase the memory available to it, it's not slow anymore | 06:31 |
| Keybuk | I'm trying to work out what is using that extra memory | 06:31 |
| === rajeevindus [n=rajeev@124.125.162.178] has joined #ubuntu-kernel | ||
| pkl_ | Keybuk: increase the memory available to what? the vmware VM? | 06:53 |
| Keybuk | yes | 06:54 |
| === pulkit [n=root@221.134.114.125] has joined #ubuntu-kernel | ||
| === rajeevindus [n=rajeev@124.125.162.178] has left #ubuntu-kernel [] | ||
| pkl_ | if the VM is low on memory, it will be shrinking the caches - dentry cache, inode cache and the buffer/page-table caches. If it shrinks the dentry/inode cache and gets rid of inodes subsequently needed, this will cause a lot of extra seeking in the Squashfs image | 06:56 |
| Keybuk | what I can't see is why it needs to shrink it that much | 06:57 |
| Keybuk | free says it rarely goes below 100MB | 06:57 |
| pkl_ | there's a parameter which controls whether the kernel prefers to shrink caches or prefers to move stuff out to swap. | 06:57 |
| pulkit | BenC ping | 06:57 |
| BenC | pulkit: yes | 06:58 |
| BenC | Keybuk: squashfs, tmpfs, etc.. | 06:58 |
| pulkit | BenC: are you free..I wanted some advice on my rejected Google SOC application | 06:58 |
| pkl_ | the policies relating to when and what get shrunk, or swapped are a bit of a black art, and are constantly being tweaked between kernels. | 06:58 |
| BenC | pulkit: I don't handle the SoC stuff | 06:58 |
| Keybuk | BenC: 100MB of them?! | 06:59 |
| BenC | Keybuk: then there's the whole desktop thing running :) | 06:59 |
| pulkit | BenC: O sorry, I just saw your name for a project, and wanted to ask what mistakes I had made...Thanks nevertheless | 06:59 |
| Keybuk | BenC: was I right that the page cache figure includes mapped portions of binaries and libraries? | 06:59 |
| BenC | Keybuk: increasing mem allows for more things to stay cached | 07:00 |
| BenC | of there's not enough memory, then the kernel will spend a lot of time dumping cached pages for disk reads, and so almost any page read will result in an actual IO op to the device | 07:00 |
| BenC | *if | 07:00 |
| BenC | I think 256meg is the bare minimum for livecd installs, but could do with a suggested min of 384M or even 512M | 07:02 |
| pkl_ | squashfs doesn't use a lot of memory itself, but it will be asked to re-read anything dropped from the page-cache and/or dentry/inode caches. | 07:04 |
| cjwatson | yeah, I think that's really a bit ridiculous though | 07:04 |
| cjwatson | we have a *lot* of users on 256MB (subjectively, from bug reports) | 07:04 |
| === pulkit [n=root@221.134.114.125] has left #ubuntu-kernel [] | ||
| pkl_ | the big question is does the installer (Ubquity?) re-access files it previously accessed? Especially files which it read sometime ago and which may have been removed from caches on low-mem systems- and can this be improved? | 07:06 |
| === bronson_ [n=bronson@66.237.74.66.ptr.us.xo.net] has joined #ubuntu-kernel | ||
| pkl_ | In general as far as the installer is concerned, it is better to swap anonymous memory to hard-disk rather than delete cached data/metadata from the Squashfs filesystem, because it is more expensive to re-read from the Squashfs image than the swap file (on slower processors anyway). But I doubt the kernel does this. | 07:13 |
| BenC | pkl_: we have no swap in the installer | 07:13 |
| BenC | unless the installer could use the swap it is setting up for the installed system early | 07:14 |
| pkl_ | BenC: that's good to know. So, on a low-memory system the kernel is going to be aggresively shrinking the caches for data it thinks can be re-read. Very bad for Squashfs performance. | 07:15 |
| BenC | it would seem so | 07:16 |
| BenC | and there's like a million possible things that could cause an increase in memory pressure on the feisty livecd as opposed to the edgy one | 07:16 |
| pkl_ | and, as I said, the policies on what and when stuff gets deleted can have changed between the kernels. | 07:17 |
| pkl_ | A lot of installers (obviously not Ubuntu) on low-memory systems immediately create swap and mount it. Could that be done? | 07:20 |
| === blueyed_ [n=daniel@i5387DD1E.versanet.de] has joined #ubuntu-kernel | ||
| BenC | I don't know that we could do it immediately, but maybe once the user goes through the setup process and creates the partitions, the installer could use the one the user defined | 07:21 |
| BenC | that would be just before the actual install process starts | 07:21 |
| mdz | BenC: the installer does activate swap as soon as it's available | 07:21 |
| BenC | oh does it? | 07:21 |
| mdz | BenC: (after it's been partitioned and mkswap'd) | 07:21 |
| BenC | ok, then there goes that idea | 07:21 |
| === pulkit [n=root@221.134.114.125] has joined #ubuntu-kernel | ||
| mdz | pkl_: by far the lion's share of the I/O done by the installer is copying the contents of the squashfs to the hard disk | 07:22 |
| mdz | pkl_: i.e., mostly reading files once and never touching them again | 07:22 |
| BenC | mdz: what program is used to copy those files? Is it cp, rsync, or some custom thing? | 07:23 |
| mdz | something like O_STREAMING would be appropriate if it were available in python | 07:23 |
| mdz | BenC: python | 07:23 |
| pkl_ | this is one of my annoyances with the kernel, it doesn't have any concept of relative costs of I/O from difference devices/filesystems, at least the time I looked. | 07:23 |
| BenC | ok | 07:23 |
| === kkubasik [n=kjk38@kjk38-laptop.STUDENT.CWRU.Edu] has joined #ubuntu-kernel | ||
| tuxmaniac | pkl_, what do you mean by relative costs ofI/O? | 07:26 |
| cjwatson | pkl_: while it's copying files, it basically just goes straight through in os.walk() order | 07:26 |
| cjwatson | Keybuk: hmm, just out of interest, did you create swap in your test install | 07:26 |
| cjwatson | ? | 07:26 |
| mdz | I almost always do erase disk, which last I checked created swap | 07:27 |
| pkl_ | tuxmaniac: it rather more expensive, for example, to re-read a block off CDROM than the hard-disk. Once the page goes into the page-cache this knowledge isn't there. | 07:27 |
| BenC | tuxmaniac: like the kernel should be less likely to drop page caches for a squashfs filesystem on cdrom that it would for ext3 fs on a hd | 07:27 |
| Keybuk | cjwatson: yes | 07:27 |
| cjwatson | although I did create swap, and it was still 44 minutes by vmware's clock versus the more usual 9 or so | 07:27 |
| cjwatson | mdz: yes, it does | 07:28 |
| cjwatson | BenC: it's shutil.copyfile() in python | 07:28 |
| cjwatson | which is basically read()/write() slurping in 16K chunks | 07:28 |
| BenC | I wonder if python is doing anything whacky that would cause extra IO | 07:28 |
| Keybuk | http://people.ubuntu.com/~scott/meminfo.png | 07:29 |
| BenC | decent strace would probably find something out | 07:29 |
| pkl_ | cjwatson: probably because the kernel is preferring to shrink caches than swap. | 07:29 |
| cjwatson | last time I straced I think it was just read()/write(), though I could check | 07:29 |
| cjwatson | I really doubt it's python | 07:30 |
| BenC | not so much the read/write, but maybe interim things that it may do between files or something, like reread dentry or something dumb like that :) | 07:30 |
| tuxmaniac | BenC, is that available on BSD? I mean the page cache drops depending on the FS | 07:30 |
| BenC | tuxmaniac: no idea, just explaining pkl's statement :) | 07:30 |
| tuxmaniac | aah ok :-) | 07:31 |
| cjwatson | I can certainly have a look | 07:31 |
| === kkubasik [n=kjk38@kjk38-laptop.STUDENT.CWRU.Edu] has left #ubuntu-kernel [] | ||
| === kkubasik [n=kjk38@kjk38-laptop.STUDENT.CWRU.Edu] has joined #ubuntu-kernel | ||
| Keybuk | so whatever the two cyanish lines are, they bounce around between 250MB and 300MB during install | 07:32 |
| Keybuk | (that's with 512mb btw) | 07:33 |
| cjwatson | BenC: http://people.ubuntu.com/~cjwatson/tmp/ubiquity-copy.trace.bz2 | 07:34 |
| cjwatson | the chmod/utimes/stat etc. stuff in between files is mostly ubiquity | 07:35 |
| cjwatson | I wonder if mmapping would be faster than reading | 07:36 |
| cjwatson | wonder what the mmaps it does do are for | 07:37 |
| cjwatson | oh, maybe just memory allocation | 07:37 |
| BenC | lstat64("/rofs/bin/uname", {st_mode=S_IFREG|0755, st_size=14000, ...}) = 0 | 07:38 |
| BenC | stat64("/target/bin/uname", 0xbf9f17d8) = -1 ENOENT (No such file or directory) | 07:38 |
| BenC | stat64("/rofs/bin/uname", {st_mode=S_IFREG|0755, st_size=14000, ...}) = 0 | 07:38 |
| BenC | stat64("/target/bin/uname", 0xbf9f14f8) = -1 ENOENT (No such file or directory) | 07:38 |
| BenC | open("/rofs/bin/uname", O_RDONLY|O_LARGEFILE) = 3 | 07:38 |
| BenC | fstat64(3, {st_mode=S_IFREG|0755, st_size=14000, ...}) = 0 | 07:38 |
| BenC | open("/target/bin/uname", O_WRONLY|O_CREAT|O_TRUNC|O_LARGEFILE, 0666) = 4 | 07:38 |
| BenC | fstat64(4, {st_mode=S_IFREG|0666, st_size=0, ...}) = 0 | 07:38 |
| BenC | fstat64(3, {st_mode=S_IFREG|0755, st_size=14000, ...}) = 0 | 07:38 |
| BenC | lot of stats going on in there | 07:38 |
| BenC | it fstats the source twice after opening it, and twice before opening | 07:39 |
| BenC | just normal stat before open, but you get what I mean | 07:39 |
| cjwatson | I'll cough to the inefficiency (the first is in shutil.copyfile, the second must be somewhere in python I think), but is it likely to be triggering this slowdown? | 07:42 |
| cjwatson | surely one stat or two is much the same, goes to the dentry cache either way | 07:42 |
| pkl_ | if the dentry and inode caches have been shrunk, they'll go to the filesystem again. | 07:43 |
| cjwatson | they'd have to be shrunk so that they could only hold one item | 07:43 |
| cjwatson | seems like pretty serious shrinkage | 07:44 |
| cjwatson | the stray fstats shouldn't go to the filesystem | 07:44 |
| cjwatson | shouldn't stat information on open files be accessible without that? | 07:44 |
| pkl_ | ok, so the multiple stats are stat file, stat file again, rather than stat the entire filesystem (or a substantial part), and then stat the entire filesystem again. | 07:45 |
| cjwatson | (this is where pkl tells me I know nothing about filesystems) | 07:45 |
| pkl_ | stat info on open files will never go to the filesystem. | 07:45 |
| cjwatson | pkl_: right, it's two files (source and target; the source on squashfs, the target not) but a pair at a time rather than the whole filesystem, yes | 07:45 |
| cjwatson | it's "does the source file exist? what is it? does the target file exist? are they the same thing already? copy" | 07:46 |
| cjwatson | the logic is obviously a bit more competent than that but you get the idea | 07:46 |
| cjwatson | I suppose it's possible that memory pressure forces /rofs/bin out of the dentry cache while /rofs/bin/umount is being copied and before starting on /rofs/bin/uname | 07:49 |
| cjwatson | (e.g.) | 07:49 |
| cjwatson | is there any reasonable way to test a hypothesis like that? | 07:49 |
| pkl_ | A hook in Squashfs (a printk in the Squashfs lookup and readdir routines) would show that happening, or hooks into VFS. | 07:51 |
| BenC | cjwatson: I had to do stat caching in dpkg to fix a test case I had created so that it took 30 seconds to install a package instead of 30 minutes | 07:53 |
| === Lure [n=lure@clj46-234.dial-up.arnes.si] has joined #ubuntu-kernel | ||
| BenC | granted it was a huge test case, much more stat's than the install case we're talking about | 07:53 |
| BenC | but they do add up, so I was just pointing them out | 07:53 |
| pkl_ | Doing find with and without -noleaf is an easy way to see how file stating slows things down... | 07:55 |
| BenC | I also don't know if there's negative cache for the ENOENT case | 07:56 |
| BenC | no idea if an ENOENT has much of a performance inpact | 07:56 |
| BenC | *impact | 07:56 |
| BenC | internally a stat() on non-existent file has to get dentry for every dir leading up to the last component, right? | 07:57 |
| pkl_ | BenC: yes, and on the leaf directory the directory has to be searched for the non-existent entry. Depending on the filesystem this can be quite expensive. | 07:58 |
| BenC | right, even on ext3, that can consume a good chunk of cache, especially in a dir like /usr/share/doc/ | 07:59 |
| pkl_ | Squashfs uses directory hashing, and the impact is much lower than otherwise (searching the entire directory). | 07:59 |
| BenC | mdz, cjwatson: I just did another feisty x86 install in vmware, this time with a 512M VM, and start to finish was 9 minutes | 08:32 |
| === infinity [n=adconrad@cerberus.0c3.net] has joined #ubuntu-kernel | ||
| BenC | in fact, even the initial boot to to livecd took less than 1 minute | 08:33 |
| BenC | so it definitely looks like we have memory issues | 08:33 |
| cjwatson | for feisty, my inclination is to document this as best we can (e.g. change the memory threshold below which releases.ubuntu.com advises using alternate) | 08:34 |
| cjwatson | but for gutsy I would really like to get the threshold back down | 08:35 |
| BenC | going to be a very thorough investigation to get this nailed down | 08:36 |
| cjwatson | yeah, that's why I don't want to rush it into feisty now | 08:38 |
| mdz | cjwatson: agreed, can you add it to FeistyKnownIssues? | 08:40 |
| mdz | I'm quite happy to chalk it up to memory usage changing in subtle ways | 08:40 |
| mdz | we should see if we can't get it down again for gutsy, rather than increase the standard requirements | 08:41 |
| cjwatson | added | 08:43 |
| cjwatson | seems like a good developer sprint topic if we don't nail it before then | 08:43 |
| === doko_ [n=doko@dslb-088-073-094-188.pools.arcor-ip.net] has joined #ubuntu-kernel | ||
| === philwyett [n=philip@bb-87-81-146-45.ukonline.co.uk] has joined #ubuntu-kernel | ||
| === m0rg0th [n=manugarg@219.64.120.181] has joined #ubuntu-kernel | ||
| === m0rg0th [n=manugarg@219.64.120.181] has joined #ubuntu-kernel | ||
| === mjg59 [n=mjg59@cavan.codon.org.uk] has joined #ubuntu-kernel | ||
| === sacater [n=sacater@colchester-lug/member/sacater] has joined #ubuntu-kernel | ||
| === ivoks [n=ivoks@3-170.dsl.iskon.hr] has joined #ubuntu-kernel | ||
| === pulkit [n=root@221.134.114.125] has left #ubuntu-kernel [] | ||
| === mjg59 [n=mjg59@cavan.codon.org.uk] has joined #ubuntu-kernel | ||
| === mjg59_ [n=mjg59@cavan.codon.org.uk] has joined #ubuntu-kernel | ||
| === defendguin [n=jmsunser@216-136-108-250.static.twtelecom.net] has joined #ubuntu-kernel | ||
| === asac [n=asac@debian/developer/asac] has joined #ubuntu-kernel | ||
| === lmierzej [n=lmierzej@drc242.neoplus.adsl.tpnet.pl] has left #ubuntu-kernel [] | ||
| === cox [n=cox@client-81-107-200-114.glfd.adsl.virgin.net] has joined #ubuntu-kernel | ||
| === rpereira [n=rpereira@ubuntu/member/rpereira] has joined #ubuntu-kernel | ||
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!
