| === zz_gouki is now known as gouki | ||
| dholbach | hiya | 07:59 |
|---|---|---|
| dholbach | can anybody tell me if this looks like a kernel problem or more of a Keybuk-boot problem: | 08:00 |
| dholbach | the last thing when I boot on the other machine is "ACPI: I/O resource nForce2_smbus [0x1c40-0x1c7f] conflicts with ACPI region SM01 ............. Error probing SMB2" | 08:00 |
| dholbach | but I get the feeling that that message has nothing to do with the hang on the machine I'm seeing | 08:00 |
| dholbach | if I boot with bin=/bin/bash the last thing I see is "bash: cannot set terminal process group (-1): Inappropriate iotcl for device" "bash: no job control in this shell" | 08:00 |
| dholbach | this is the latest karmic with the ubuntu-boot ppa enabled | 08:00 |
| dholbach | Keybuk: ^ any idea what I could do? | 08:00 |
| jk- | dholbach: you mean init=/bin/bash ? | 08:06 |
| dholbach | jk-: oh yeah, that's right | 08:06 |
| jk- | looks like bash is starting up OK, just giving you a warning. | 08:06 |
| dholbach | ok... so it might actually be something boot-related breaking | 08:06 |
| jk- | it may not have correct access to the console device | 08:06 |
| dholbach | and not the kernel | 08:07 |
| apw | tend to agree, would be suspicious its boot and not the kernel | 08:07 |
| dholbach | the ACPI smbus thing seems to be unrelated? | 08:07 |
| apw | you may even find it will respond to commands even if it does not look like its prompting | 08:07 |
| jk- | yeah, make sure you have /dev/ populated | 08:07 |
| apw | which with init as bash you won't without devtmpfs | 08:08 |
| dholbach | erm... I think I'm seeing a very very weird thing right now - after booting from a livecd and running fsck and smartctl it works again now (both reported no errors and didn't say anything about fixing something?!) | 08:11 |
| dholbach | Keybuk: nevermind | 08:11 |
| dholbach | apw, jk-: thanks guys - I have no idea what "fixed it now" | 08:12 |
| apw | hrm ... computers ... | 08:12 |
| dholbach | might be a good idea to get a replacement disk :-) | 08:12 |
| apw | heh ... lets hope not | 08:13 |
| dholbach | at least have it ready and do a backup again :-) | 08:14 |
| thux | hi, with one hardware my jaunty waits long time 'starting to boot' on screen. acpi=off reduces that time but complaint pnpbios. and if acpi=off and pnpbios=off again 'starting to boot' show long time. is there a way around this? | 08:30 |
| smb | thux, Its hard to say something on that. It might be some subsystem that just takes long to initialize. And with the combination that seems to be faster it might not be there at all. I assume you removed the "quiet" from the command line and still there is nothing between the first line and the boot? | 08:38 |
| apw | thux one would need to get the dmesg output from the boot and see if there is an obvious time delay in the timestampts there. may give a clue as to cause | 08:38 |
| thux | ok I try first remove quiet and then put dmesg to pastebin | 08:39 |
| apw | smb, as a heads up, karmic intel graphics are looking like they have hang issues ... | 08:44 |
| smb | apw, I noticed from comments last week. Seems mostly ok on the aa1 | 08:45 |
| apw | i am suspicious it only triggers after a suspend/resume | 08:45 |
| smb | I had, not yet, a complete lockup as you had. There is a strange case of the netbook launcher sometimes ignoring the mouse after a while. But that is rather something else | 08:47 |
| smb | apw, What model do you have? 945GME? | 08:48 |
| thux | here is my dmesg http://pastebin.com/m5185a0b0 | 08:50 |
| smb | [ 9.224019] pci 0000:00:1d.7: EHCI: BIOS handoff failed (BIOS bug?) 01010001 | 08:51 |
| thux | yes | 08:51 |
| smb | This seems to be the problem | 08:51 |
| thux | before that line it waits now | 08:52 |
| smb | Is USB working after this for you? | 08:52 |
| thux | yes | 08:52 |
| smb | If I read this message correctly there is a problem transferring control of usb from the bios to the kernel | 08:53 |
| thux | i have many usb devices plugged only hp laser sometimes not detected | 08:54 |
| smb | If there are settings in the bios to control usb support, you might play around with them. But often you want early usb support for usb keyboards | 08:54 |
| thux | yes i have that | 08:54 |
| apw | does the problem occur if you boot without any usb devices attached | 08:55 |
| apw | smb, i have both 945GME and GM45E's showing the issue | 09:00 |
| smb | apw, Ok, I also try to keep an eye on it. Though I am running mixed stuff, Jaunty userspace with -10.31 kernel and x-org-edgers X | 09:02 |
| apw | heh ... franenstein | 09:02 |
| smb | :) Ubuntu, Halloween edition ;-) | 09:03 |
| apw | actually thats interesting as these hangs can just as easily be a bad gpu usage from mesa as they can be kernel issues ... so ... a different mesa combination should be an interesting control | 09:03 |
| apw | i like the sounds of that ... Ubuntu, halloween | 09:03 |
| apw | horrific halloween | 09:03 |
| apw | shame we have used up HH | 09:03 |
| smb | Hehe, and its no animal... | 09:04 |
| smb | I should compare the mesa versions against yours. The edgers should in theory be near Karmic, shouldnt' they? | 09:04 |
| apw | halloween hyena | 09:04 |
| apw | yeah near, but i'd expect them to be ahead a little | 09:05 |
| smb | Mine are all 7.7.0-git20090911.a79eecb9-0ubuntu0tormod-jaunty | 09:06 |
| thux | there was a setting in bios for usb hand off and it was disabled, i enabled it and now it boots normally, thanks :) | 09:09 |
| thux | wonder why it was default disabled | 09:09 |
| smb | you're welcome. good to hear. Meh would not think about it... Maybe works better with the other OS... | 09:11 |
| thux | ah that i see | 09:11 |
| === ogra_ is now known as ogra | ||
| apw | cking, gnarl, did either of you report the sd card not mounting thingy | 10:08 |
| cking | nope | 10:08 |
| smb | apw, Don't think so... Though not mounting thingy is a bit broad | 10:08 |
| cking | it was steve who hit this issue | 10:09 |
| apw | oh welll i've filed 5 today, wahts one more | 10:10 |
| apw | if only i knew what the heck was meant to handle it | 10:10 |
| smb | apw, does it only affect sdhc or all or is the reader not detected? | 10:11 |
| smb | normally the reader would issue a udev event for a device getting ready (I think) | 10:11 |
| apw | the card is detected and the devices are all made, the partition table is read, and the p1 device is made | 10:17 |
| apw | its the next step, the mounting it and popping up boxes to show it that don't occur | 10:17 |
| smb | Is /media/bla created? | 10:18 |
| smb | Err well likely not | 10:18 |
| smb | as you say it does not get mounted | 10:18 |
| smb | In theory the partition scanning is also done in udev ... Do you get any error message in dmesg? | 10:20 |
| smb | I believe each partition should also do a uevent, then udev call blkid and create the symlinks /dev/disk/by-... The mount step might be device-kit... | 10:22 |
| apw | smb, no it doesn't get mounted | 10:35 |
| apw | the p1 device does get mounted | 10:35 |
| smb | apw, ??? huh | 10:36 |
| apw | _made_ sorry | 10:37 |
| apw | not mounted | 10:37 |
| apw | there are by-id files as well | 10:37 |
| smb | The id files at least show the partition got through blkid. I just cannot grep the statement about the p1 device. So nothing gets mounted, correct? | 10:39 |
| apw | i meant that it does partition discovery correctly and finds it has a partition and makes mmblk0p1 as well | 10:43 |
| apw | nothing is mounted as a result. manual mount triggers gnome to find piccies etc | 10:44 |
| apw | i've filed it on gnome-volume-manager, bug #429257 | 10:44 |
| ubot3 | Malone bug 429257 in gnome-volume-manager "MMC cards no longer auto mount" [Undecided,New] https://launchpad.net/bugs/429257 | 10:44 |
| apw | i am a bug filing machine ... 4 today alone | 10:44 |
| smb | apw, Shame, you should fix them. ;-) | 10:45 |
| apw | na, i file them _you_ get to fix them | 10:45 |
| apw | most are these intel graphics hangs ... bah | 10:45 |
| smb | If its Karmic, I can' see it (yet) :) | 10:45 |
| smb | apw, But seriously, I am not completely sure, the automounting thing seemed to be tied to a desktop (so gnome) but (might check up on this later) might have moved to something more generic to work without a running desktop too... | 10:47 |
| apw | smb, yeah figured i file with gnome thingy as thats description included automounting, someone else can move it | 11:07 |
| cking | apw, I'm seeing a lot of messages: "TKIP: RX tkey->key_idx=2 frame keyidx=1 priv=de21d780" - is this new with the latest kernel? | 11:49 |
| apw | cking, hmmm not sure | 11:49 |
| apw | [ 10.393047] lib80211_crypt: registered algorithm 'TKIP' | 11:50 |
| apw | changed your AP setup recently? | 11:50 |
| apw | not getting it here, but that is a WPA2 thing iirc ? | 11:50 |
| cking | yep, WPA2 | 11:51 |
| cking | apw, I've rebooted my router today, surely, that's not caused this message? | 11:52 |
| apw | whats your driver here, that message appears to be in the rtl8187se driver | 11:53 |
| apw | looks to always have been in there | 11:54 |
| cking | it's the broadcom wl driver | 11:54 |
| cking | :-( | 11:54 |
| apw | yay | 11:54 |
| cking | nggg | 11:55 |
| === sap is now known as csurbhi | ||
| * amitk welcomes csurbhi to the kernel team | 12:58 | |
| smb | Hi csurbhi | 12:58 |
| === pgraner` is now known as pgraner | ||
| csurbhi | Hi ! | 13:02 |
| csurbhi | thanks Amit | 13:02 |
| * apw waves to csurbhi | 13:09 | |
| csurbhi | hello apw :) | 13:10 |
| apw | csurbhi, so hows it going so far | 13:20 |
| csurbhi | reading up the docs and setting up accounts.. other than that.. descent | 13:20 |
| csurbhi | :) | 13:20 |
| smb | Luckily there is help close. :) | 13:21 |
| csurbhi | yes.. thats right | 13:21 |
| rtg | apw, I wonder why we have CONFIG_LIB80211_DEBUG enabled ? | 13:25 |
| rtg | csurbhi, howdy | 13:25 |
| csurbhi | okie dokie | 13:25 |
| apw | debian.master/config/config.common.ports:# CONFIG_LIB80211_DEBUG is not set | 13:26 |
| apw | debian.master/config/config.common.ubuntu:# CONFIG_LIB80211_DEBUG is not set | 13:26 |
| apw | rtg ? | 13:26 |
| smb | backports? | 13:27 |
| rtg | apw: hmm, then what kernel is pgraner running that he would get the 'CCMP: replay detected' message? | 13:27 |
| smb | updates/compat-wireless-2.6/config.mk:# CONFIG_LIB80211_DEBUG=y | 13:28 |
| apw | rtg ahh we jut hit that with cking, a couple of the WL drivers contain the whole damn 80211 stack in them with mods | 13:28 |
| apw | and it may be on there | 13:28 |
| rtg | apw: it is indeed enabled in LBM | 13:28 |
| smb | rtg, apw, Or he has LBM installed | 13:28 |
| cking | I have LBM installed | 13:29 |
| apw | i think pete has the cranky wl driver doesn't he? | 13:29 |
| rtg | apw, rt73 IIRC | 13:30 |
| rtg | apw: actually, rtl8187se | 13:30 |
| apw | cking, which rt driver did you have, it was that rtlxxxse one wasn;t it? | 13:30 |
| apw | that definatly has the 80211 in it en-toto | 13:31 |
| apw | a staging driver horror show and no mistake | 13:31 |
| cking | a;w | 13:34 |
| cking | apw, I saw this on my wl broadcom driver | 13:34 |
| apw | gibber | 13:34 |
| apw | its everywhere, and spreading :) | 13:35 |
| cking | it's on my HP mini 1000 ;-) | 13:38 |
| rtg | apw, how about Keybuk's ext4 patch? 'ext4: Don't update superblock write time when filesystem is read-only' | 14:07 |
| apw | i've not heard back from him that i know of on its effacacy | 14:08 |
| apw | Keybuk, did you get to test that kernel i did? | 14:08 |
| Keybuk | apw: did you ever e-mail it me? | 14:09 |
| apw | i pointed you to it on here ... | 14:10 |
| Keybuk | see, that never works :p | 14:10 |
| * Keybuk doesn't read IRC scrollback | 14:10 | |
| apw | ahh ... damn | 14:10 |
| * apw finds it | 14:10 | |
| * Keybuk finds his laptop under things | 14:10 | |
| apw | http://people.canonical.com/~apw/lp427822-karmic/ | 14:11 |
| apw | mine is burried in bugs | 14:11 |
| ogra | Keybuk, hmm, you only talk abour east of UTC | 14:11 |
| ogra | *about | 14:11 |
| ogra | Keybuk, (see #ubuntu-devel, loic-m is surely in france but seems to see a similar thing) | 14:12 |
| Keybuk | ogra: france is very much east of UTC | 14:12 |
| Keybuk | I realise that, historically speaking, they are very unhappy about that | 14:12 |
| Keybuk | and feel that the prime meridian of the world, and thus all timekeeping, should pass through Paris | 14:12 |
| * ogra must have gotten his sense of east and west wrong | 14:12 | |
| apw | heh | 14:13 |
| Keybuk | but that was not the consensus of the rest of the world | 14:13 |
| Keybuk | and if they're still clinging to that, we can't help them :p | 14:13 |
| rtg | Keybuk, you're poking the hornets nest | 14:13 |
| apw | Keybuk, sorry about the email, i'll remember that for next time | 14:13 |
| Keybuk | rtg: it's a hobby ;) | 14:13 |
| * apw files his 7th bug of the day, sigh | 14:14 | |
| Keybuk | apw: if I'm actually paying attention to IRC, it's usually ok - also /msg is always good - I save those and read them back | 14:14 |
| Keybuk | but things-on-a-channel I don't tend to notice if they go off the top of the screen while I'm not looking | 14:14 |
| apw | ahh right, fair enough | 14:14 |
| rtg | My use of IRC isn't much different. | 14:15 |
| rtg | its a lousy bug tracking mechanism | 14:15 |
| Keybuk | ogra: the recovery-while-read-only bug affects everyone from Britain (in Summer time) through Europe, across Eastern Europe, the Middle East, Asia, Indionesia and Australia and (I think) NZ | 14:15 |
| apw | all true | 14:15 |
| Keybuk | rtg: for me it's a getting-things-done defence | 14:15 |
| Keybuk | IRC is a productivity black hole | 14:16 |
| rtg | indeed | 14:16 |
| Keybuk | if I wake up and start reading through things like e-mail or IRC, then I may as well write off being productive in the morning | 14:16 |
| ogra | Keybuk, yeah, i always get the sides wrong here :P | 14:16 |
| Keybuk | and I find that people tend to notice when you're back and remind you about things anyway | 14:16 |
| Keybuk | ogra: what's nice is that it's the exact opposite of the init scripts bug | 14:16 |
| Keybuk | which only affected Merkins | 14:17 |
| ogra | heh | 14:17 |
| rtg | Keybuk, new topic. anything changed recently that I should investigate that slows the boot time on my SSD laptop? It used tyo be 28 seconds, but has gotten much longer since late last week. | 14:17 |
| Keybuk | rtg: I've noticed some slowdown with recent kernels myself | 14:17 |
| apw | my main laptop seems to take hours to login these days | 14:17 |
| Keybuk | both on SDD and HDD | 14:17 |
| apw | specially now as its hidden the feeling of length is even worse | 14:17 |
| Keybuk | I figured that the IO elevator of the week wasn't working as well as last weeks' | 14:17 |
| rtg | Keybuk, you think its kernel related? | 14:18 |
| Keybuk | rtg: I'm not sure what else has changed | 14:18 |
| apw | we added xsplash | 14:18 |
| apw | osd seems to have lost its mind | 14:18 |
| rtg | I'm watching the GDM throbber just sit there and flash at me, no disk activity | 14:18 |
| Keybuk | hmm | 14:19 |
| Keybuk | odd | 14:19 |
| Keybuk | couchdb? | 14:19 |
| rtg | dunno what couchdb is | 14:20 |
| Keybuk | silly desktop database stuff they added for Ubuntu One | 14:20 |
| Keybuk | it's quite heavy | 14:20 |
| apw | nnng | 14:20 |
| Keybuk | and seems to sit there for a few minutes | 14:20 |
| ogra | isnt it erlang too ? | 14:20 |
| apw | that reminds me it exploded on my other laptop | 14:20 |
| apw | ubuntu one | 14:20 |
| rtg | some serious bitching is in order if thats what it is | 14:20 |
| * ogra wonders how fast/slow erlang is compared to other langs | 14:20 | |
| apw | can we tell from a bootchart | 14:21 |
| ogra | loading another interpreter during boot is surely not speeding up things | 14:21 |
| Keybuk | probably | 14:21 |
| apw | Keybuk, are we using sreadahead by default now, could that be contributing to slowness | 14:22 |
| apw | in the sense we are upgrading a lot, it will be loading old junk? | 14:22 |
| ogra | if you got a new apparmor profile it will definately slow down the next boot after that | 14:24 |
| Keybuk | apw: one of the main reasons we switched to sreadahead is that it reprofiles often | 14:24 |
| Keybuk | it may be simply that you upgrade before every reboot | 14:24 |
| Keybuk | so every boot sreadahead is profiling | 14:24 |
| Keybuk | (rather than assisting) | 14:24 |
| apw | now that i almost always do do yes | 14:24 |
| Keybuk | I've also noticed a strange issue that I think is ext4, but can't place it | 14:24 |
| Keybuk | after people upgraded to my PPA, and rebooted | 14:25 |
| Keybuk | they have a 4-6s "exe" process that seems to be somewhere around the point the root filesystem was mounted | 14:25 |
| Keybuk | it's almost as if, after major upgrades, ext4 has to do lots of work before it can mount the filesystem next time | 14:25 |
| Keybuk | that needs investigation | 14:25 |
| apw | its called 'exe' in the profiles? | 14:26 |
| Keybuk | yes | 14:26 |
| Keybuk | that's what confuses me | 14:26 |
| apw | how does the name get generated for the profiles, from the executable link? | 14:27 |
| apw | interestingly that link is call 'exe', so perhaps its what things get called which are not convertable back to a name | 14:28 |
| apw | i wonder if the chrooting madness and rm madness going on at root mount time leaves us with bad /proc/NNNN/exe links and bad names in the profile | 14:28 |
| apw | ie. it could be anything | 14:28 |
| Keybuk | right, that's what I think | 14:28 |
| Keybuk | that it's something run from the initramfs, just before it's cleaned up, so the link never exists | 14:29 |
| apw | hmmmmm | 14:29 |
| * apw goes for a couple of boots to see if that speeds things up | 14:30 | |
| apw | Keybuk, i am getting 1m frm bios to X, 1:22 to bios to gdm, 34s gdm to desktop. now this is a HDD, so i'd expect it to be what 30s total? | 14:39 |
| ogra | try a second one :) | 14:39 |
| apw | (that was second boot, so i assume sreadahead has done its thing) | 14:40 |
| apw | ogra, that was the second one | 14:40 |
| ogra | ah | 14:40 |
| apw | still seems like an insane length of time | 14:40 |
| ogra | i see a few added seconds every time i get an updated apparmor profile for an app | 14:40 |
| apw | ouch | 14:40 |
| ogra | which i was told is normal now | 14:41 |
| Keybuk | can you get a bootchart of that | 14:41 |
| ogra | since apparmor turns the profile textfiles into binary thingies | 14:41 |
| Keybuk | use bootchart=nostop on the kernel command line | 14:41 |
| Keybuk | then once you're a desktop, run sudo /etc/init.d/stop-bootchart start | 14:41 |
| apw | Keybuk, sure | 14:41 |
| apw | yay java ... sigh | 14:42 |
| ogra | apw, whats wrong with java ? ... as long as you dont have to touch its code it's usually fine :) | 14:51 |
| apw | Keybuk, ok got it, seems sreadahead runs for a full 90s, that seems unreasonable to me | 15:05 |
| Keybuk | apw: really? | 15:06 |
| Keybuk | when you login do "status sreadahead" | 15:06 |
| Keybuk | what does it say? | 15:06 |
| apw | apw@dm$ status sreadahead | 15:06 |
| apw | sreadahead stop/waiting | 15:06 |
| * apw pushed up the data to rookery | 15:08 | |
| apw | if this image is to be believed, then it sustained like 44MB/s for that 90 odd seconds | 15:08 |
| apw | which is like 4GB of poop | 15:08 |
| apw | http://people.canonical.com/~apw/bootchart/ | 15:09 |
| apw | does the green line on the bar there show instantaneough thorugh put? of so its like 100k/s over that time? | 15:11 |
| * apw has a nother go | 15:12 | |
| apw | Keybuk, ok the second boot was a little better, so lets assume the first one was a reconfigure or something | 15:20 |
| apw | device discovery takes like 20s alone | 15:20 |
| Keybuk | status sreadahead - if that says "stop/waiting" then your boot was assisted by sreadahead | 15:21 |
| Keybuk | if it says "stop/killed" or similar, then your boot was *reprofiled* | 15:22 |
| Keybuk | apw: the -2 | 15:22 |
| Keybuk | that shows the exact same I/O weirdness that I see | 15:22 |
| apw | what bit? | 15:22 |
| Keybuk | the entire system spends most of its time in I/O wait | 15:22 |
| Keybuk | http://people.canonical.com/~apw/bootchart/dm-karmic-20090914-2.png | 15:22 |
| Keybuk | the two graphs along the top | 15:22 |
| Keybuk | the first graph, blue is CPU usage, red is I/O wait | 15:23 |
| Keybuk | the second graph, red is disk utilization, the green lines are throughput | 15:23 |
| apw | yep, so that would imply its nailed waiting for disk | 15:23 |
| Keybuk | so that's showing | 15:23 |
| Keybuk | 1. your system is spending almost all of its time in I/O wait (BAD BAD BAD BAD) | 15:23 |
| Keybuk | 2. your disk is continually utilized | 15:23 |
| Keybuk | 3. almost no throughput is being achieved, despite the continuous utilization | 15:23 |
| Keybuk | I'd also point out where this starts | 15:23 |
| apw | so read ahead is sucking? | 15:23 |
| Keybuk | deep inside the initramfs | 15:23 |
| Keybuk | in fact, last time I debugged this, it started as soon as you loaded the disk controller | 15:24 |
| Keybuk | which implies something seriously wrong in the kernel | 15:24 |
| rtg | apw, do you still have older kernels installed? | 15:24 |
| * apw would have a bunch of them yes | 15:24 | |
| apw | i'll go back to something older and see | 15:25 |
| rtg | apw, IIRC -8 worked well. | 15:25 |
| apw | if its the kernel it would imply its more like kernel readahead sucking | 15:25 |
| apw | anyhow ... i'll try -8 | 15:25 |
| Keybuk | apw: could be, though again, it starts in the initramfs | 15:26 |
| Keybuk | this is quite important | 15:26 |
| Keybuk | it starts *before* we mount the root filesystem | 15:26 |
| Keybuk | it starts before we even know which block device is the root filesystem | 15:26 |
| apw | could it just be bad stats rather than actual anything | 15:27 |
| apw | enough to throw off sreadahead | 15:27 |
| Keybuk | sreadahead doesn't look at stats | 15:28 |
| Keybuk | but I don't think it is bad stats | 15:28 |
| Keybuk | my Dell laptop has exactly this problem | 15:29 |
| Keybuk | and has had for a while now | 15:29 |
| apw | though my disk light is nailed ... and its rattling like hell | 15:29 |
| Keybuk | this is why you hear me whining continuously about I/O performance | 15:29 |
| Keybuk | *but* other laptops and hardware I have *do not* | 15:29 |
| apw | but there is io perf and io perf | 15:29 |
| apw | mine is a dell | 15:29 |
| Keybuk | I suspect it's controller related | 15:30 |
| apw | a horrible thouught | 15:30 |
| apw | -5 is as bad | 15:33 |
| === bjf-afk is now known as bjf | ||
| Keybuk | apw: I have tested your kernel packages, and they are good | 16:29 |
| apw | Keybuk, most excellent | 16:52 |
| apw | Keybuk, i am assuming you will be going back to ted, and we'll get a pair of patches from you when thats ready? | 16:53 |
| apw | Keybuk, do you know waht this 'utilisation' on the disk actually is, which stat it is | 17:04 |
| apw | we don't seem to have any form of actual 'how utilised is the disk' number in the numbers collected by bootchart | 17:04 |
| erichammond | jjohansen, pgraner: Is there an EC2 kernel status meeting now? | 17:06 |
| pgraner | jjohansen: ^^^^^^^^???? | 17:07 |
| apw | its at this time in my calender | 17:07 |
| apw | (modulo google being pants) | 17:08 |
| jjohansen | ah yes | 17:08 |
| jjohansen | just a sec | 17:09 |
| pgraner | apw: pants, pants, pants... that gcal is | 17:09 |
| apw | so very true | 17:09 |
| rtg | erichammond, lool has approved the linux-ec2 package for main inclusion, so perhaps I can get it built today. | 17:09 |
| smoser | here | 17:09 |
| jjohansen | alright lets begin | 17:10 |
| jjohansen | This is the EC2 kernel status update meeting | 17:10 |
| jjohansen | as far as I am aware the only change since thursday/friday is the state of bug #427288 | 17:11 |
| ubot3 | Malone bug 427288 in eglibc "Karmic i386 EC2 kernel emulating unsupported memory accesses" [High,Fix released] https://launchpad.net/bugs/427288 | 17:11 |
| jjohansen | which was fixed in glibc | 17:11 |
| erichammond | smoser: Can a new AMI be created to test this fix? | 17:12 |
| jjohansen | it looks like there is some packaging work to be done as well | 17:12 |
| rtg | smoser, have you been able to test this? | 17:12 |
| smoser | i have not tested this yet, but will do that today. | 17:13 |
| smoser | erichammond, i think publishing a new ami for it doesn't make sense given alpha 6 on thursday | 17:13 |
| jjohansen | smoser: so are we then planning on rolling out a new ami with the karmic kernel | 17:14 |
| jjohansen | minus this glibc fix | 17:14 |
| smoser | i'm planing on alpha6 using the karmic kernel | 17:15 |
| smoser | jjohansen, also, i opened bug 428692 | 17:15 |
| ubot3 | Malone bug 428692 in ubuntu "ec2 kernel needs CONFIG_BLK_DEV_LOOP=y and other config changes" [Medium,Confirmed] https://launchpad.net/bugs/428692 | 17:15 |
| jjohansen | smoser: thanks | 17:15 |
| rtg | smoser, why _wouldn't_ you use the new glibc? Isn't that what the alphas are for? | 17:15 |
| smoser | i did not intend to say i wouldnt | 17:16 |
| smoser | i hope to get 427288 "all the way fixed" for alpha6 | 17:16 |
| jjohansen | ah, I missed interpreted, no point in rolling out a new ami before alpha6, right? | 17:16 |
| erichammond | smoser: My opinion is that if we can test a fix, we should do it sooner rather than later. Are we saying that the fix won't be in glibc until alpha6? | 17:17 |
| smoser | I'll test it. it can easily be tested just by apt-get update && apt-get install libc6-xen and reboot | 17:18 |
| erichammond | smoser: nice, thanks. | 17:19 |
| smoser | then, i'll make sure that we get libc6-xen into the ami, and that that ami will not get the error. but i dont plan on publishing another ami with all that incorporated prior to alpha6 | 17:19 |
| erichammond | smoser: When you publish an AMI are you building it from scratch with vmbuilder or are you simply uploading and registering the daily image which is automatically created? | 17:19 |
| smoser | i much prefer to use the nightly | 17:20 |
| smoser | if that werent' sufficient, then i'd re-spin the nightly in the same way, just at a later date | 17:20 |
| erichammond | smoser: Ok. How often is the code updated which is used to build the nightly? I thought I saw that you pinged soren to manually update it recently. | 17:21 |
| lool | rtg: It was still in NEW when I looked though | 17:21 |
| smoser | erichammond, you're correct | 17:21 |
| smoser | we do need a long term solution for that. | 17:22 |
| lool | rtg: (Thanks for clarifying the doc question BTW) | 17:22 |
| rtg | lool, right. how long will it sit there before getting accepted? | 17:22 |
| lool | rtg: I'm not archive admin; it's up to them | 17:22 |
| rtg | lool, so, just find the Monday guy and hassle him? | 17:22 |
| lool | Digging the Ubuntu wiki page | 17:22 |
| lool | https://wiki.ubuntu.com/ArchiveAdministration | 17:22 |
| lool | # | 17:22 |
| lool | Monday: SteveLangasek; JamesWestby | 17:22 |
| lool | rtg: You could I guess | 17:22 |
| lool | rtg: slangasek seems like a good target; he knows about the source package and how it's constructed | 17:23 |
| lool | rtg: I expect that if the copyright was properly updated for the Xen patches it should be ok | 17:23 |
| rtg | lool, uh, I didn't see that mentioned anywhere. | 17:24 |
| lool | rtg: Well it's not MIR related, it's NEW related | 17:24 |
| lool | rtg: Source packages have to document their copyright properly before they enter the archive, right? | 17:24 |
| lool | My role was just in approving the move of this package from universe to main | 17:24 |
| rtg | lool, AFAIK there are no copyright issues. | 17:25 |
| lool | Since it's built like the other kernel packages which are in main already and since the kernel team will care for it, I dont have a big issue with another one | 17:25 |
| rtg | everything is gplv2 | 17:25 |
| lool | k | 17:25 |
| Keybuk | apw: yes, I've already sent ted a note | 17:25 |
| erichammond | (I'm lost, is this still part of the EC2 status?) | 17:25 |
| Keybuk | apw: I think it comes from /proc/diskstats | 17:26 |
| lool | rtg: debian/copyright usually mentions where stuff was taken from, which license it's under and who owns copyright; that said this last bit probably isn't maintainable in the copyright file so I guess you would refer to the list of AUTHORS or something similar or just omit it | 17:26 |
| rtg | erichammond, sorry, lool and I were discussing linux-ec2 acceptance. | 17:26 |
| lool | (Is this a meeting?) | 17:26 |
| rtg | lool, 'tis | 17:26 |
| Keybuk | apw: bootchart logs /proc/stat, /proc/diskstats and /proc/*/stat | 17:27 |
| Keybuk | all chart information is derived from that | 17:27 |
| rtg | jjohansen, are we done? | 17:27 |
| jjohansen | I believe so | 17:27 |
| jjohansen | anyone else? | 17:28 |
| erichammond | smoser's second bug has not been addressed. | 17:28 |
| lool | Ok /me disappears then, sorry for interrupting | 17:28 |
| erichammond | bug 428692 | 17:28 |
| ubot3 | Malone bug 428692 in ubuntu "ec2 kernel needs CONFIG_BLK_DEV_LOOP=y and other config changes" [Medium,Confirmed] https://launchpad.net/bugs/428692 | 17:28 |
| Keybuk | isn't DEV_LOOP=y already? | 17:28 |
| rtg | Keybuk, it is for the master branch | 17:29 |
| jjohansen | Keybuk: not in EC2 build currently | 17:29 |
| smoser | the bug is really "make the ec2 kernel more like -server" | 17:29 |
| rtg | smoser, I see that. I'll see what I can do | 17:29 |
| smoser | but explicitly, users have already asked for CONFIG_BLK_DEV_LOOP=y | 17:29 |
| jjohansen | smoser: you also wanted KEXEC :) | 17:30 |
| apw | Keybuk, from waht i can see the utilisation is a number of ticks that there was an io on the queue... so that means 1 io has the basic same effect on the table as 10 | 17:30 |
| Keybuk | apw: it doesn't seem to be translated into binary though, it appears as an analogue graph | 17:31 |
| smoser | jjohansen, yes, but you dont want to support it. so i wasn't going to explicitly ask :) | 17:31 |
| Keybuk | and, most importantly | 17:31 |
| Keybuk | I have machines which *don't* have that I/O graph | 17:31 |
| Keybuk | and they boot much faster, with identical installs | 17:31 |
| jjohansen | smoser: right | 17:32 |
| apw | Keybuk, right if the io is very empty then it would be empty, or if the queue is empty any time in the sampling tick the answer will be less than 1.0, but if its got 1 io for all the time then its 1.0 same as if its 100 io for all the time | 17:32 |
| apw | ie, its pretty much saying if read ahead worked even badly the block should be 'full' utilisation wise | 17:32 |
| jjohansen | all right if there is nothing else, we will call the EC2 kernel meeting adjourned | 17:32 |
| apw | seems a bit pointless | 17:33 |
| erichammond | This may also be the place and time to discuss bug 429169 as some of the alternatives affect the kernel. | 17:33 |
| ubot3 | Malone bug 429169 in vm-builder "ec2: Include kernel modules in AMIs" [Undecided,New] https://launchpad.net/bugs/429169 | 17:33 |
| Keybuk | apw: but the output doesn't support that | 17:33 |
| Keybuk | we have graphs which are quite fluid | 17:33 |
| Keybuk | suggesting it doesn't behave that way | 17:33 |
| apw | suggesting the data doesn't always mean the same perhaps | 17:33 |
| Keybuk | http://people.canonical.com/~scott/boot-performance/sam-karmic-20090721-8.png | 17:33 |
| Keybuk | for example | 17:33 |
| apw | but thats what they are according to the docs | 17:34 |
| Keybuk | I tend to disbelieve any documentation of things in /proc | 17:34 |
| Keybuk | they're hopelessly out of date | 17:34 |
| Keybuk | especially on that graph above | 17:34 |
| Keybuk | note how the I/O wait goes *flat* | 17:34 |
| apw | well no that graph looks pretty the same, mostly high when read ahead is high | 17:34 |
| Keybuk | you don't see that on machines "affected" by this I/O issue | 17:34 |
| Keybuk | right | 17:34 |
| Keybuk | but the whole point is that on my Dell, the graph goes high when readahead isn't even running | 17:35 |
| Keybuk | and remains high after it finished | 17:35 |
| apw | yeah i think the graph we have on my machine literlally only says IO isn't going very fast | 17:35 |
| ogra | amitk, any chance we see a new imx51 kernel before the freeze ? | 17:35 |
| Keybuk | and | 17:35 |
| Keybuk | key | 17:35 |
| apw | and so the IO wait is high all the time | 17:35 |
| Keybuk | *the throughput on the disk is virtually zero* | 17:35 |
| Keybuk | that to me suggests that the I/O queue is getting stuck | 17:35 |
| Keybuk | that things are waiting for I/O | 17:35 |
| Keybuk | and the kernel isn't actually bothering to service them | 17:35 |
| Keybuk | since I've seen bugs exactly like this, where the kernel stops doing I/O, I'm inclined to believe that there is a bug here and it's not a charting issue | 17:36 |
| erichammond | jjohansen, smoser: Did I not speak in time to keep the meeting going? | 17:36 |
| jjohansen | erichammond, smoser: I think it would be appropriate to take that discussion over to #ubuntu-server | 17:36 |
| smoser | ok | 17:36 |
| Keybuk | (just looking up the bug number) | 17:37 |
| Keybuk | https://bugs.edge.launchpad.net/ubuntu/+source/linux/+bug/336652 | 17:37 |
| ubot3 | Malone bug 336652 in linux "Poor system performance under I/O load" [Medium,Triaged] | 17:37 |
| * apw notes that the green line is very low agian even in your one ... odd | 17:37 | |
| erichammond | jjohansen: Two of the options relate to building things into the kernel and/or ramdisk, but it is quieter over on #ubuntu-server | 17:38 |
| jjohansen | erichammond: that is why I suggested it | 17:38 |
| Keybuk | apw: on the sam one, the green line is at least trying to lift when there is I/O wait | 17:38 |
| Keybuk | what worries me about yours (and my other laptop) is that the green line stays flat even when there is I/O wait | 17:38 |
| apw | the green line in mine is the same pattern | 17:39 |
| apw | its that there is no gaps in the wait that its not so obvious | 17:39 |
| apw | not that that means there isn't an issue, it taking 100years to boot demonstrates that on its own | 17:39 |
| Keybuk | no, yours shows a different pattern | 17:39 |
| Keybuk | you have red I/O wait almost all the time | 17:40 |
| Keybuk | with a flat green line | 17:40 |
| apw | what i don't really understand is why sreadahead isn't able to get its shit onto the queues | 17:40 |
| Keybuk | that implies waiting for I/O without any disk throughput | 17:40 |
| apw | nope it has two distinct lists | 17:40 |
| Keybuk | which leans me to I/O queue stall | 17:40 |
| apw | lifts one where you first block is, and the second where you 51mb/s is | 17:40 |
| Keybuk | yes | 17:40 |
| Keybuk | but it's otherwise flat | 17:40 |
| Keybuk | even though you have I/O wait | 17:40 |
| Keybuk | on http://people.canonical.com/~scott/boot-performance/sam-karmic-20090721-8.png | 17:41 |
| Keybuk | there is | 17:41 |
| Keybuk | a) *far* less I/O wait | 17:41 |
| apw | yep, that is cause sreadahead is take soooo long to do its thing | 17:41 |
| Keybuk | in fact, much of the top of the graph is blue, not red | 17:41 |
| apw | its pushing in a dribble of io which accounts for the queue | 17:41 |
| Keybuk | and every point there is I/O wait, there is continual lift | 17:41 |
| Keybuk | apw: please disregard sreadahead | 17:41 |
| apw | the question is why is read ahead not occuring in 2-4 s in mine than yours | 17:41 |
| Keybuk | if you like, comment out the "start on" from /etc/init/sreadahead.conf | 17:41 |
| Keybuk | and you'll see the same pattern | 17:42 |
| Keybuk | right | 17:42 |
| apw | Keybuk, i can't as its likely the cuase of constant wait | 17:42 |
| Keybuk | sreadahead being unable to push I/O is a symptom of the bug | 17:42 |
| apw | likely yes | 17:42 |
| Keybuk | not the bug | 17:42 |
| apw | indeed ... | 17:42 |
| Keybuk | and if you remove sreadahead, you still see the exact same pattern | 17:42 |
| Keybuk | which fits | 17:42 |
| Keybuk | because | 17:42 |
| Keybuk | as I do keep pointing out | 17:42 |
| Keybuk | your problems start several seconds before sreadahead is even started ;) | 17:42 |
| Keybuk | sreadhead starts at 5s | 17:42 |
| apw | well you have to be careful with that statement, as all yo ucan tell is that there is 1 page on the queue during that perios | 17:43 |
| Keybuk | but you've already been in large I/O wait for a couple of seconds with no throughput by then | 17:43 |
| Keybuk | there shouldn't be any pages on the queue | 17:43 |
| apw | we have no idea if thats 1 block or 10000 blocks for a short time | 17:43 |
| Keybuk | or there should be disk throughput | 17:43 |
| apw | on -2 there is a lift in throughput with the fvery first spike of red | 17:43 |
| apw | its small but there | 17:43 |
| Keybuk | so why is there I/O wait after the lieft | 17:44 |
| apw | there isn't the lift drops as does the wait | 17:44 |
| Keybuk | ok | 17:44 |
| apw | there is a similar width gap then it starts again, lift + wait | 17:44 |
| Keybuk | try something for me | 17:44 |
| Keybuk | boot without sreadahead | 17:44 |
| apw | yep will do that, seems most logical to confirm your theory | 17:44 |
| apw | how did i do that? | 17:44 |
| Keybuk | edit /etc/init/sreadhead.conf | 17:44 |
| Keybuk | comment out the "start on" line | 17:44 |
| apw | (and i do believe it in principle | 17:44 |
| apw | just its not proven to me yet) and if its right the debug will be nasty | 17:45 |
| Keybuk | I can't find the bug I was looking for though yet | 17:45 |
| Keybuk | I had an actual test case where I could stall "dd" on this machine | 17:45 |
| apw | launchpad hates us all | 17:45 |
| Keybuk | in a way that the kernel had decided that dd was doing no more I/O and could block on read() forever | 17:46 |
| apw | that sounds like the sort of thing you see with barriers in the queue | 17:47 |
| Keybuk | annoyingly, I have a feeling it was when I was trying to back up my old laptop hard drive | 17:47 |
| Keybuk | so I don't even have the logs, because the hard drive has obviously been replaced ;) | 17:47 |
| Keybuk | but basically, yes, on this machine I find that any attempt to do large amounts of I/O get extremely slow | 17:48 |
| Keybuk | readahead included | 17:48 |
| Keybuk | pitti has a similar laptop, and confirmed my results | 17:48 |
| Keybuk | but the XPS 1330 for example *does not* have the same problem | 17:48 |
| pgraner | Keybuk: anything similar to https://bugs.launchpad.net/ubuntu/+source/linux/+bug/392288 | 17:54 |
| ubot3 | Malone bug 392288 in linux "dd extremely slow writing to usb key without oflag=dsync" [Medium,Triaged] | 17:54 |
| Keybuk | pgraner: I don't think so, on the basis that I was having these problems on jaunty | 17:56 |
| Keybuk | (it was during Berlin that I had the dd-stall issue) | 17:56 |
| pgraner | Keybuk: ack | 17:56 |
| Keybuk | (I've had the abysmal I/O performance on this laptop for several releases now, it's not a "new" bug | 17:57 |
| Keybuk | if I were to guess, I'd say it was edgy or feisty that it started) | 17:58 |
| apw | Keybuk, ok ... the readahead less case is faster booting | 18:09 |
| apw | there is wait along the way as predicted, though depending how the driver calcs that stat we might expect that | 18:10 |
| apw | its a poor metric at best | 18:10 |
| apw | of course readahead getting behind would account for the worse with readahead mode too | 18:11 |
| apw | so all pointing to poor io performance full stpo | 18:12 |
| apw | stop | 18:12 |
| Keybuk | yeah | 18:12 |
| Keybuk | don't suppose you could time how long sreadahead takes? | 18:12 |
| Keybuk | time sreadahead --debug --no-fork | 18:13 |
| apw | on my machine it was 90s | 18:13 |
| apw | if i run it once booted it takes like 0 | 18:13 |
| Keybuk | ok | 18:13 |
| Keybuk | boot with init=/bin/bash | 18:13 |
| Keybuk | then run it (with those args) | 18:13 |
| * apw tries your incantation | 18:14 | |
| * apw notes it running for some time, no IO obvious | 18:14 | |
| apw | 15s | 18:14 |
| Keybuk | weird | 18:14 |
| Keybuk | so that took 15s to do nothing? | 18:15 |
| apw | and also notes it complains about not being oable to open things | 18:15 |
| apw | sda/queue/read_ahead_kb for one | 18:15 |
| apw | which sounds bad to me | 18:15 |
| apw | Keybuk, that does seem about how it felt yes | 18:17 |
| apw | Keybuk, does your machine (thats slow) have its root in an extended partition? | 18:19 |
| Keybuk | no | 18:21 |
| Keybuk | /dev/sda1 | 18:21 |
| apw | Keybuk, i can't get it to do any actual readahead calls in debug mode | 18:22 |
| Keybuk | you need both arguments | 18:26 |
| apw | Keybuk, have both as in --debug --no-fork | 18:27 |
| Keybuk | yes | 18:27 |
| apw | Keybuk, i am worried by mdz's bug #429001 ... that warn on there is triggered by what appears to be ftrace on opens, and if i am reading sreadahead right it turns that on... could it be its failing to turn it off again ... as he had it on still when he suspend/resumed much later | 18:28 |
| ubot3 | Malone bug 429001 in linux "WARNING: at /build/buildd/linux-2.6.31/kernel/trace/ring_buffer.c:1393 rb_add_time_stamp+0x79/0x200()" [Low,Incomplete] https://launchpad.net/bugs/429001 | 18:28 |
| Keybuk | I don't think it should fail to turn it off | 18:30 |
| Keybuk | unless he killed it;) | 18:30 |
| apw | he is running your boot ppa's so anything possible :) | 18:31 |
| Keybuk | mdz's laptop is a strange beast | 18:32 |
| Keybuk | also sreadahead only turns on ftrace when profiling | 18:32 |
| Keybuk | not in "normal operation" | 18:32 |
| apw | bah it just fooked its own db | 18:45 |
| apw | Keybuk, ok i am finding that running the read ahead on my laptop after dropping caches takes 30s, if i make it single thread not 4 threads it takes 15s | 19:10 |
| Keybuk | right, you have a HDD | 19:11 |
| Keybuk | but I don't see what about HDDs makes that need to happen, when on SDD you have the exact opposite behaviour | 19:12 |
| apw | the threads can out of order the requests badly and make the disk thrash i guess | 19:12 |
| apw | i guess we can actually tell now, there is a rotational flag in the device so we could use that to check | 19:12 |
| apw | Keybuk, /sys/block/sda/queue/rotational seems to tell us | 19:14 |
| apw | as sreadahead is in there already it could check and change behavior | 19:15 |
| Keybuk | that's handy | 19:15 |
| apw | one assumes its cause seek latency is essentially 0 on ssd | 19:15 |
| Keybuk | I actually already have the sreadahead patches to use a single thread ;) | 19:16 |
| Keybuk | but I also found in testing that you then need sreadhead to *block* the boot | 19:16 |
| Keybuk | ie. single thread and other stuff in background = bad | 19:16 |
| Keybuk | (which fits with readahead-list) | 19:16 |
| Keybuk | but then I noticed slow down unless the readahead pack was not sorted ideally for HDD | 19:16 |
| apw | hmmm, thats not hot either is it | 19:16 |
| Keybuk | and I hadn't done the "ideal sort" bit yet | 19:16 |
| apw | this is all a bit of a nightmare | 19:17 |
| apw | just trying a single thread one on my machine | 19:17 |
| apw | but still in parallell | 19:17 |
| Keybuk | I wrote a very long mail about this | 19:17 |
| apw | i know ... i remember reading it | 19:17 |
| apw | i wonder if letting it run single in parallel but skipping the first few files deliberatly would work | 19:18 |
| apw | as that would hold the boot in the sense those required io's would get the boot behind readahead | 19:18 |
| Keybuk | I don't follow | 19:18 |
| Keybuk | still don't follow | 19:19 |
| apw | the risk is if the boot ever gets ahead of where the read ahead is we can make things much worse | 19:19 |
| apw | as its reading stuff we no longer need and already ready | 19:19 |
| Keybuk | still don't follow | 19:19 |
| Keybuk | we never do that | 19:19 |
| Keybuk | sreadahead is the first thing started | 19:19 |
| Keybuk | the problem on HDD isn't that the boot gets ahead | 19:20 |
| Keybuk | it's that we're inherently seeking the disk all over the place | 19:20 |
| Keybuk | back after dinner | 19:21 |
| Keybuk | will try to use that rotational flag and merge my other sreadahead updates to do the foreground stuff | 19:21 |
| Keybuk | see if it makes a difference | 19:21 |
| apw | yeah ... tricky | 19:25 |
| apw | ok from 15 -> 10s if i stop it using low IO priority | 19:25 |
| amitk | ogra: yes, I am hoping for a imx51 upload tomorrow. | 19:29 |
| ogra | amitk, oh, ok | 19:30 |
| amitk | had already discussed it with rtg | 19:30 |
| ogra | yeah i saw that last week but couldnt remember the outcome | 19:31 |
| amitk | ogra: http://people.canonical.com/~amitk/mx51/linux-image-2.6.31-100-imx51_2.6.31-101.8_armel.deb is a preview if you want to play | 19:32 |
| Keybuk | apw: interesting on the scheduler | 20:29 |
| Keybuk | is that changeable per-device or ? | 20:30 |
| === Seeker`_ is now known as Seeker` | ||
| apw | Keybuk, yep per device on the fly, look for queue/scheduler in the device | 21:48 |
| Keybuk | if we stayed in the foreground, and blasted the readahead list for an HDD before letting the boot continue | 21:52 |
| Keybuk | whilst setting the io priority to realtime, and the scheduler to deadline | 21:52 |
| Keybuk | that might work better? | 21:52 |
| === bjf is now known as bjf-afk | ||
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!
