[12:24] <BenC> if 2.6.20-14 works for him, then there's no reason 2.6.20-15 shouldn't
[12:25] <BenC> mjg59: is the dont_disable_c3 patch still needed for ipw2100?
[12:25] <BenC> mjg59: I don't recall any bug reports that benefited from it
[12:26] <mjg59> It was never a bug-fixing measure
[12:26] <mjg59> It was a "I'm willing to accept some packet corruption in return for better battery life" one
[12:29] <BenC> do we really want it in there?
[12:29] <mjg59> I find it useful
[12:30] <mjg59> By default it does nothing
[12:30] <BenC> Can you get it upstream?
[12:33] <mjg59> I'll try to, when I push the other ipw2100 stuff I have
[12:33] <BenC> Great, thanks
[01:28] <kylem> win 23
[01:47] <BenC> I hate when laptop manufacturers use 2x512Meg for 1G of mem, means you can't just add another 1G stick to get 2G
[01:47] <kylem> pretty common cost saving measure...
[01:47] <BenC> my gateway came with 1G stick
[01:47] <BenC> or wait, maybe it was the HP
[01:48] <BenC> one of them did anyway :)
[02:29] <jdong> BenC: ping
[02:30] <jdong> Can someone look at bug https://bugs.launchpad.net/ubuntu/+source/linux-restricted-modules-2.6.20/+bug/101984
[02:30] <jdong> "firegl_public.c: Ubuntu modification uses obsolete __syscall_return"
[02:34] <mjg59> jdong: Given that it's all a huge hack in the first place, what's the impact of this?
[02:35] <jdong> I'm not sure what the impact is
[02:35] <jdong> that's why I asked
[02:35] <jdong> some user bugged me about it....
[02:36] <mjg59> Why?
[02:37] <jdong> he referred to it as an 'annoying bug'
[02:38] <mjg59> Yeah, but how?
[02:41] <BenC> from what I can tell it means people have a harder time using custom kernels and fglrx-kernel-source
[02:41] <BenC> which means I don't care
[02:42] <jdong> oh well :)
[02:43] <BenC> the patch we applied came from fglrx source, so it's not a patch we created
[02:43] <BenC> s/from/for/
[02:43] <BenC> nm, I meant from
[02:45] <BenC> mjg59: the dongle_id = 0x9 patch for IBM0071 in nsc-ircc, is that a patch you authored?
[02:45] <mjg59> Yeah
[02:45] <BenC> mjg59: Can I send it upstream with your signed-off-by?
[02:45] <mjg59> My experience is that all the IBM ones have that dongle type
[02:46] <mjg59> So sure - I don't /absolutely/ guarantee that it's correct, but I think it's better than autodetection
[02:47] <BenC> mjg59: what do you think we should do with the raw keycode patch for input.c, was there ever any discussion about that on lkml or something?
[02:49] <BenC> we've been carrying it around for several releases now
[02:51] <mjg59> Oh, to pass through whether it was in the filter or not?
[02:52] <mjg59> Can't remember. I think it came up on lkml at some point, but I'll look into it.
[03:04] <BenC> mjg59: right, that one...thanks
[03:04] <kylem> moo.
[03:32] <rushfan_> BenC is .27 supposed to have the sata_nv fix in it ?
[03:32] <rushfan_> I'm [g2] 
[03:41] <BenC> yes
[03:42] <rushfan_> BenC I was running .27 and I think I had a sata command response failure on my NForce4 board
[03:42] <rushfan_> with regard to the command time outs
[03:43] <rushfan_> I just loaded your .26 kernel and it seems to be working so far
[03:44] <rushfan_> unfortunately this is my pvr box and the .26 kernel doesn't have the ivtv drivers
[03:49] <BenC> it has the drivers, just not the firmware
[03:49] <BenC> copy the firmware
[03:49] <BenC> rushfan_: my .26 kernel is _exactly_ the same as the archive's -15.27 kernel
[03:50] <BenC> brb, doing some hw juggling
[08:19] <dholbach> good morning
[01:47] <saispo> hi
[04:54] <zul> is there still a meeting going on today?
[05:24] <Keybuk> so
[05:24] <Keybuk> you remember that "VMware takes an hour to install feisty" thing?
[05:25] <Keybuk> AND I can make it take only 15-20 min
[05:29] <kkubasik> there all in a meeting right now
[05:29] <mjg59> Keybuk: Oh?
[05:40] <Keybuk> let me just reverse the change, and make sure this goes back to 1h+ installs
[05:40] <mjg59_> Keybuk: Sorry, missed the last few lines. What was the change?
[05:41] <Keybuk> I increased the amount of memory I gave to the VM
[05:41] <Keybuk> from 256MB to 512MB
[05:41] <Nafallo> mjg59_: no you didn't miss anything :-)
[05:41] <mjg59_> So, uh. Ubuntu installs slowly in 256MB of RAM. This is unsurprising, surely?
[05:42] <Keybuk> no, it's very surprising
[05:42] <Keybuk> it's a very recent change
[05:42] <Keybuk> the LiveCD is almost unusable in 256MB
[05:43] <Keybuk> where as beta is fine
[05:43] <mjg59_> Well, it certainly doesn't sound like a kernel issue...
[05:43] <Keybuk> unless it's caused by the squashfs change using more memory?
[05:43] <Keybuk> (or it could indeed be completely unrelated)
[05:44] <mjg59_> Well, how much free ram do you have after boot, compared with beta?
[05:44] <Keybuk> that's what I'm about to look at
[05:44] <Keybuk> our original thought was that this was an atapi problem
[05:44] <Keybuk> but pure I/O seems fine
[05:54] <mjg59> Hey wow -powerpc still actually boots Mac hardware
[05:55] <mjg59> Reassuring
[05:57] <cjwatson> 256MB was fine with edgy
[05:57] <cjwatson> not exactly stellar or anything, but not crushing
[05:59] <Keybuk> free doesn't return bad numbers, 128MB free before page cache
[05:59] <kylem> argh.
[05:59] <kylem> i'm a muppet.
[06:00] <Keybuk> cjwatson: did you revert your system to a snapshot before you tried your test install?
[06:01] <cjwatson> Keybuk: I think it was clean boot + install
[06:01] <cjwatson> certainly no snapshot involved
[06:02] <Keybuk> was the drive clean though?
[06:03] <Keybuk> casper mounts any swap partition it can find on boot, remember
[06:03] <Keybuk> so if you already had an install, you would've had some swap
[06:03] <cjwatson> Keybuk: that VM is set to 700MB ...
[06:03] <cjwatson> I suspect it doesn't matter ;-)
[06:03] <Keybuk> ahh
[06:04] <Keybuk> mdz and I both leave ours at 256MB as a matter of course
[06:04] <Keybuk> could you try with 256 and see how it boots/installs?
[06:04] <cjwatson> that may well explain it
[06:04] <cjwatson> sure
[06:05] <cjwatson> wah, none of my shift keys work any more
[06:06] <cjwatson> hang on while I sort this out ;-)
[06:10] <cjwatson> boots in 2:13
[06:10] <cjwatson> subjectively, the slowness is primarily after X starts
[06:11] <Keybuk> agree
[06:11] <Keybuk> right when GNOME hauls its fat, bloaty arse into memory
[06:11] <cjwatson> free after cache is 155MB
[06:11] <Keybuk> cache includes loaded apps though
[06:11] <Keybuk> iirc
[06:11] <Keybuk> ie. mapped bits of .so files, binaries
[06:11] <cjwatson> ubiquity is liable to eat at least 50MB, possibly more
[06:12] <cjwatson> most of which I think is debconf
[06:12] <Keybuk> I get 104MB plus cache with ubiq
[06:12] <cjwatson> what's the c-a-f1 rune in vmware again?
[06:13] <Keybuk> c-a-space-f1
[06:13] <Keybuk> ie, hold ctrl alt, press space, then f1
[06:13] <Keybuk> I think
[06:13] <cjwatson> thanks
[06:13] <cjwatson> 129MB right after starting ubiquity
[06:15] <Keybuk> 97MB once install begins
[06:15] <cjwatson> ubiquity itself has 24MB res 10MB of which shared
[06:16] <Keybuk> so now it's swapping to the same disk it's installing *to* to unpack the squashfs of the disk it's reading *from*
[06:16] <cjwatson> debconf-communicate has 15MB res
[06:16] <cjwatson> can you look at top and see what's using memory?
[06:17] <Keybuk> u6y with 10.3%
[06:18] <cjwatson> growing as files are copied?
[06:18] <Keybuk> no
[06:18] <cjwatson> 10.3% sounds like about baseline
[06:19] <cjwatson> that's similar to the 24MB I had above right after starting it
[06:19] <cjwatson> so what's using MORE memory?
[06:21] <cjwatson> I'm around 90MB+ free after cache, copying files
[06:21] <cjwatson> if anything, it's growing slightly
[06:21] <Keybuk> right
[06:21] <Keybuk> nothing's growing in userland
[06:21] <Keybuk> how do I find out how much memory the kernel is hogging for itself?
[06:21] <cjwatson> where are you getting your zero number?
[06:21] <Keybuk> free
[06:21] <Keybuk> it hit zero free after cache, and sat and swapped for a few seconds
[06:22] <cjwatson> yeah, see I'm not getting zero free after cache
[06:22] <cjwatson> this is the -/+ buffers/cache: ... (free) column?
[06:23] <cjwatson> that's settled down at 106MB
[06:23] <Keybuk> it only hit it once
[06:23] <Keybuk> then it swapped a huge amount and has stayed around 70MB since
[06:23] <cjwatson> maybe I should look at this in gnome-terminal rather than a console
[06:23] <cjwatson> just worried about g-t's own bloatiness
[06:25] <cjwatson> I'm not seeing your memory problems, but I do have to agree that this isn't fast
[06:25] <cjwatson> it's 5+ minutes in by vmware's clock and only at 29%
[06:31] <Keybuk> I can't see why the memory is a problem
[06:31] <Keybuk> all I know is that if I increase the memory available to it, it's not slow anymore
[06:31] <Keybuk> I'm trying to work out what is using that extra memory
[06:53] <pkl_> Keybuk: increase the memory available to what? the vmware VM?
[06:54] <Keybuk> yes
[06:56] <pkl_> if the VM is low on memory, it will be shrinking the caches - dentry cache, inode cache and the buffer/page-table caches.  If it shrinks the dentry/inode cache and gets rid of inodes subsequently needed, this will cause a lot of extra seeking in the Squashfs image
[06:57] <Keybuk> what I can't see is why it needs to shrink it that much
[06:57] <Keybuk> free says it rarely goes below 100MB
[06:57] <pkl_> there's a parameter which controls whether the kernel prefers to shrink caches or prefers to move stuff out to swap.
[06:57] <pulkit> BenC ping
[06:58] <BenC> pulkit: yes
[06:58] <BenC> Keybuk: squashfs, tmpfs, etc..
[06:58] <pulkit> BenC: are you free..I wanted some advice on my rejected Google SOC application
[06:58] <pkl_> the policies relating to when and what get shrunk, or swapped are a bit of a black art, and are constantly being tweaked between kernels.
[06:58] <BenC> pulkit: I don't handle the SoC stuff
[06:59] <Keybuk> BenC: 100MB of them?!
[06:59] <BenC> Keybuk: then there's the whole desktop thing running :)
[06:59] <pulkit> BenC: O sorry, I just saw your name for a project, and wanted to ask what mistakes I had made...Thanks nevertheless
[06:59] <Keybuk> BenC: was I right that the page cache figure includes mapped portions of binaries and libraries?
[07:00] <BenC> Keybuk: increasing mem allows for more things to stay cached
[07:00] <BenC> of there's not enough memory, then the kernel will spend a lot of time dumping cached pages for disk reads, and so almost any page read will result in an actual IO op to the device
[07:00] <BenC> *if
[07:02] <BenC> I think 256meg is the bare minimum for livecd installs, but could do with a suggested min of 384M or even 512M
[07:04] <pkl_> squashfs doesn't use a lot of memory itself, but it will be asked to re-read anything dropped from the page-cache and/or dentry/inode caches.
[07:04] <cjwatson> yeah, I think that's really a bit ridiculous though
[07:04] <cjwatson> we have a *lot* of users on 256MB (subjectively, from bug reports)
[07:06] <pkl_> the big question is does the installer (Ubquity?) re-access files it previously accessed?  Especially files which it read sometime ago and which may have been removed from caches on low-mem systems-  and can this be improved?
[07:13] <pkl_> In general as far as the installer is concerned, it is better to swap anonymous memory to hard-disk rather than delete cached data/metadata from the Squashfs filesystem, because it is more expensive to re-read from the Squashfs image than the swap file (on slower processors anyway).  But I doubt the kernel does this.
[07:13] <BenC> pkl_: we have no swap in the installer
[07:14] <BenC> unless the installer could use the swap it is setting up for the installed system early
[07:15] <pkl_> BenC: that's good to know.  So, on a low-memory system the kernel is going to be aggresively shrinking the caches for data it thinks can be re-read.  Very bad for Squashfs performance.
[07:16] <BenC> it would seem so
[07:16] <BenC> and there's like a million possible things that could cause an increase in memory pressure on the feisty livecd as opposed to the edgy one
[07:17] <pkl_> and, as I said, the policies on what and when stuff gets deleted can have changed between the kernels.
[07:20] <pkl_> A lot of installers (obviously not Ubuntu) on low-memory systems immediately create swap and mount it.  Could that be done?
[07:21] <BenC> I don't know that we could do it immediately, but maybe once the user goes through the setup process and creates the partitions, the installer could use the one the user defined
[07:21] <BenC> that would be just before the actual install process starts
[07:21] <mdz> BenC:  the installer does activate swap as soon as it's available
[07:21] <BenC> oh does it?
[07:21] <mdz> BenC: (after it's been partitioned and mkswap'd)
[07:21] <BenC> ok, then there goes that idea
[07:22] <mdz> pkl_: by far the lion's share of the I/O done by the installer is copying the contents of the squashfs to the hard disk
[07:22] <mdz> pkl_: i.e., mostly reading files once and never touching them again
[07:23] <BenC> mdz: what program is used to copy those files? Is it cp, rsync, or some custom thing?
[07:23] <mdz> something like O_STREAMING would be appropriate if it were available in python
[07:23] <mdz> BenC: python
[07:23] <pkl_> this is one of my annoyances with the kernel, it doesn't have any concept of relative costs of I/O from difference devices/filesystems, at least the time I looked.
[07:23] <BenC> ok
[07:26] <tuxmaniac> pkl_, what do you mean by relative costs ofI/O?
[07:26] <cjwatson> pkl_: while it's copying files, it basically just goes straight through in os.walk() order
[07:26] <cjwatson> Keybuk: hmm, just out of interest, did you create swap in your test install
[07:26] <cjwatson> ?
[07:27] <mdz> I almost always do erase disk, which last I checked created swap
[07:27] <pkl_> tuxmaniac: it rather more expensive, for example, to re-read a block off CDROM than the hard-disk.  Once the page goes into the page-cache this knowledge isn't there.
[07:27] <BenC> tuxmaniac: like the kernel should be less likely to drop page caches for a squashfs filesystem on cdrom that it would for ext3 fs on a hd
[07:27] <Keybuk> cjwatson: yes
[07:27] <cjwatson> although I did create swap, and it was still 44 minutes by vmware's clock versus the more usual 9 or so
[07:28] <cjwatson> mdz: yes, it does
[07:28] <cjwatson> BenC: it's shutil.copyfile() in python
[07:28] <cjwatson> which is basically read()/write() slurping in 16K chunks
[07:28] <BenC> I wonder if python is doing anything whacky that would cause extra IO
[07:29] <Keybuk> http://people.ubuntu.com/~scott/meminfo.png
[07:29] <BenC> decent strace would probably find something out
[07:29] <pkl_> cjwatson: probably because the kernel is preferring to shrink caches than swap.
[07:29] <cjwatson> last time I straced I think it was just read()/write(), though I could check
[07:30] <cjwatson> I really doubt it's python
[07:30] <BenC> not so much the read/write, but maybe interim things that it may do between files or something, like reread dentry or something dumb like that :)
[07:30] <tuxmaniac> BenC, is that available on BSD? I mean the page cache drops depending on the FS
[07:30] <BenC> tuxmaniac: no idea, just explaining pkl's statement :)
[07:31] <tuxmaniac> aah ok :-)
[07:31] <cjwatson> I can certainly have a look
[07:32] <Keybuk> so whatever the two cyanish lines are, they bounce around between 250MB and 300MB during install
[07:33] <Keybuk> (that's with 512mb btw)
[07:34] <cjwatson> BenC: http://people.ubuntu.com/~cjwatson/tmp/ubiquity-copy.trace.bz2
[07:35] <cjwatson> the chmod/utimes/stat etc. stuff in between files is mostly ubiquity
[07:36] <cjwatson> I wonder if mmapping would be faster than reading
[07:37] <cjwatson> wonder what the mmaps it does do are for
[07:37] <cjwatson> oh, maybe just memory allocation
[07:38] <BenC> lstat64("/rofs/bin/uname", {st_mode=S_IFREG|0755, st_size=14000, ...}) = 0
[07:38] <BenC> stat64("/target/bin/uname", 0xbf9f17d8) = -1 ENOENT (No such file or directory)
[07:38] <BenC> stat64("/rofs/bin/uname", {st_mode=S_IFREG|0755, st_size=14000, ...}) = 0
[07:38] <BenC> stat64("/target/bin/uname", 0xbf9f14f8) = -1 ENOENT (No such file or directory)
[07:38] <BenC> open("/rofs/bin/uname", O_RDONLY|O_LARGEFILE) = 3
[07:38] <BenC> fstat64(3, {st_mode=S_IFREG|0755, st_size=14000, ...}) = 0
[07:38] <BenC> open("/target/bin/uname", O_WRONLY|O_CREAT|O_TRUNC|O_LARGEFILE, 0666) = 4
[07:38] <BenC> fstat64(4, {st_mode=S_IFREG|0666, st_size=0, ...}) = 0
[07:38] <BenC> fstat64(3, {st_mode=S_IFREG|0755, st_size=14000, ...}) = 0
[07:38] <BenC> lot of stats going on in there
[07:39] <BenC> it fstats the source twice after opening it, and twice before opening
[07:39] <BenC> just normal stat before open, but you get what I mean
[07:42] <cjwatson> I'll cough to the inefficiency (the first is in shutil.copyfile, the second must be somewhere in python I think), but is it likely to be triggering this slowdown?
[07:42] <cjwatson> surely one stat or two is much the same, goes to the dentry cache either way
[07:43] <pkl_> if the dentry and inode caches have been shrunk, they'll go to the filesystem again.
[07:43] <cjwatson> they'd have to be shrunk so that they could only hold one item
[07:44] <cjwatson> seems like pretty serious shrinkage
[07:44] <cjwatson> the stray fstats shouldn't go to the filesystem
[07:44] <cjwatson> shouldn't stat information on open files be accessible without that?
[07:45] <pkl_> ok, so the multiple stats are stat file, stat file again, rather than stat the entire filesystem (or a substantial part), and then stat the entire filesystem again.
[07:45] <cjwatson> (this is where pkl tells me I know nothing about filesystems)
[07:45] <pkl_> stat info on open files will never go to the filesystem.
[07:45] <cjwatson> pkl_: right, it's two files (source and target; the source on squashfs, the target not) but a pair at a time rather than the whole filesystem, yes
[07:46] <cjwatson> it's "does the source file exist? what is it? does the target file exist? are they the same thing already? copy"
[07:46] <cjwatson> the logic is obviously a bit more competent than that but you get the idea
[07:49] <cjwatson> I suppose it's possible that memory pressure forces /rofs/bin out of the dentry cache while /rofs/bin/umount is being copied and before starting on /rofs/bin/uname
[07:49] <cjwatson> (e.g.)
[07:49] <cjwatson> is there any reasonable way to test a hypothesis like that?
[07:51] <pkl_> A hook in Squashfs (a printk in the Squashfs lookup and readdir routines) would show that happening, or hooks into VFS.
[07:53] <BenC> cjwatson: I had to do stat caching in dpkg to fix a test case I had created so that it took 30 seconds to install a package instead of 30 minutes
[07:53] <BenC> granted it was a huge test case, much more stat's than the install case we're talking about
[07:53] <BenC> but they do add up, so I was just pointing them out
[07:55] <pkl_> Doing find with and without -noleaf is an easy way to see how file stating slows things down...
[07:56] <BenC> I also don't know if there's negative cache for the ENOENT case
[07:56] <BenC> no idea if an ENOENT has much of a performance inpact
[07:56] <BenC> *impact
[07:57] <BenC> internally a stat() on non-existent file has to get dentry for every dir leading up to the last component, right?
[07:58] <pkl_> BenC: yes, and on the leaf directory the directory has to be searched for the non-existent entry.   Depending on the filesystem this can be quite expensive.
[07:59] <BenC> right, even on ext3, that can consume a good chunk of cache, especially in a dir like /usr/share/doc/
[07:59] <pkl_> Squashfs uses directory hashing, and the impact is much lower than otherwise (searching the entire directory).
[08:32] <BenC> mdz, cjwatson: I just did another feisty x86 install in vmware, this time with a 512M VM, and start to finish was 9 minutes
[08:33] <BenC> in fact, even the initial boot to to livecd took less than 1 minute
[08:33] <BenC> so it definitely looks like we have memory issues
[08:34] <cjwatson> for feisty, my inclination is to document this as best we can (e.g. change the memory threshold below which releases.ubuntu.com advises using alternate)
[08:35] <cjwatson> but for gutsy I would really like to get the threshold back down
[08:36] <BenC> going to be a very thorough investigation to get this nailed down
[08:38] <cjwatson> yeah, that's why I don't want to rush it into feisty now
[08:40] <mdz> cjwatson: agreed, can you add it to FeistyKnownIssues?
[08:40] <mdz> I'm quite happy to chalk it up to memory usage changing in subtle ways
[08:41] <mdz> we should see if we can't get it down again for gutsy, rather than increase the standard requirements
[08:43] <cjwatson> added
[08:43] <cjwatson> seems like a good developer sprint topic if we don't nail it before then