[00:06] hi, im using lucid rc, i just removed a package using apt-get purge and the process ends up with a segmentation fault, the package seems to be removed tho [00:54] slangasek: yay, I literally don't need my GPG key anymore ;) === dmb is now known as NullPointerExcep [00:54] you sign and upload for me [00:55] :-) [01:08] james_w: random question; how is the lp:ubuntu/lucid -> lp:ubuntu/maverick transition handled? [01:20] psusi: I've read your patch [01:39] Keybuk, and? what did you think? [01:46] psusi: I replied to your mail, but basically I like the idea [01:46] it's what I was trying to do with that code before, but I didn't know how to get the disk locations [01:47] I didn't know you could call readahead() on the actual disk like that, does it actually work? [01:48] Keybuk, seems to... it sped up my boot a bit... the way you had it before the calls into e2fslibs to get the next inode just did a synchronous read() of the inode table of the current block group, then once it had returned all the used inodes there, read() the next group and returns the first entry there... the read() calls read into the buffer cache, then copy the requested inode to the user buffer [01:48] readahead() just queues the read into buffer cache and doesn't bother taking a copy [01:51] after some discussion on the mailing lists it seems that there just isn't much you can do about readahead() blocking when it has to read the indirect block or external extent tree node to locate the rest of the blocks [01:51] so it seems the best thing to do is to have ureadahead map where all the data blocks PLUS indirect blocks are during profile, and call ureadahead() for all of those blocks on the block device, rather than going through the filesystem [01:52] of course if defrag manages to reduce the extents a file occupies to 4 or less so they all fit in the inode, then it doesn't make much difference [01:53] but I still have some work to do to have defrag properly expand and contract the size of the extent tree [01:57] right [01:57] if we can readahead() directly on the block devices, then we can avoid the overhead of open()ing every single file [02:00] exactly [02:01] but to avoid the overhead of the open() later when the file is actually needed we want to readahead() the blocks of the directories on the path to the files [02:02] calling readahead() on the block dev allows us to queue the reads in the background without blocking, and hopefully gets the directory blocks read, and in optimal order, so that they are there when the normal boot process tries to open() those files [02:05] right [02:05] makes sense [02:08] need to find an e2fslibs call to map the indirect blocks or extent tree blocks used by the file though [02:08] and add the to the block list in the pack file [02:12] right [02:12] may need to write one of those [02:13] can I just break a moment [02:13] I need to go out of the balcony and start screaming into the San Francisco air [02:13] because some fucking idiot has broken lvm ;) [02:13] (an upstream fucking idiot) [02:14] right [02:14] back [02:14] * Keybuk understands why lvm2 device nodes fail to get mounted during boot [02:17] why? [02:18] ? [02:18] because some idiot merged the broken package I'd deliberately not merged [02:18] well [02:18] no [02:18] that's unfair [02:18] the idiot is the upstream who wrote the broke software in the first place :p [02:18] * psusi is just pissed that lvm does not handle mirrors for shit [02:19] It doesn't actually expose the events? Will your fix mean that we can actually autodetect and autoload dm_mod as a bonus? [02:20] I don't think I can fix it [02:21] I simply know how it's broken [02:21] dm_mod is built-in [02:21] *that's* not a problem [02:22] Ah. I'll stop asking then, so you can get past the pain of understanding the breakage but not the fix. [02:22] I'm not going to bother understanding the fix [02:23] we need to hire an lvm expert [02:23] or drop it [02:23] but I understand why it's broken [02:24] what's broken about it now? [02:24] it doesn't respond to "add" events anymore [02:24] almost the inverse of the change you made to dm-raid ;) [02:27] so basically, if an lvm device gets brought up before cold-plugging (say in the initramfs) [02:27] the "add" event caused by the cold plug gets ignored by lvm [02:27] so udev never learns about it [02:27] Ah, that explains the wonkiness that happens when a USB cable falls out when one has a VG that spans multiple USB drives. [02:27] so blkid is never run [02:28] so permissions are never set [02:28] so mountall never sees it [02:28] so upstart never sees it [02:28] etc. [02:28] yeah, probably other bugs too [02:30] oooh [02:30] no, this is an easy one line fix [02:30] wait, I dropped the change event from dmraid and left it only responding to add... upstream changed lvm to not respond to add? [02:31] yes [02:31] changed devmapper in fact [02:31] wtf? when the hell is it supposed to detect the drive then? [02:31] magic [02:31] apparently devmapper "always changes the device" [02:31] except, say, if udev wasn't running when that happened [02:32] sure... ignoring CHANGE is one thing... ADD is another... [02:32] I can understand why they thought that [02:32] you always get "add /block/dm-0" when the devmapper device is created [02:32] but that's always before the table is loaded (which is a separate ioctl) [02:32] loading the devmapper table means you get "change /block/dm-0" [02:32] (this is one of the reasons I think your dm-raid patch is wrong, fwiw) [02:33] so devmapper only really needs to respond to "change" events [02:33] at least [02:33] when you look at it that way [02:33] yea, but who cares unless you are trying to use a logical volume as a physical volume? [02:33] this is always the case for devmapper and lvm [02:34] eh? add event for /dev/sda1 needs to trigger an lvm scan to see if it is a pv.. [02:34] you're misunderstanding [02:34] I'm not talking about the underlying block devices [02:34] I'm talking about the devmapper devices [02:35] they're block devices too [02:35] called /block/dm-N [02:35] yea, don't need to scan that unless you are trying to use an lv as a pv... [02:35] no [02:35] again [02:35] you're misunderstanding [02:35] I'm not talking about *lvm* [02:35] I'm talking about *devmapper* itself [02:35] or a dmraid array as a pv... [02:35] when a dm-N block device is created [02:35] we need to query it to find out what kind of devmapper device it is [02:35] we need to *make a device node in /dev* for it [02:36] (so we need to know what it's name is, e.g. /dev/mapper/foo-bar) [02:36] we then need to run blkid on it, to see what kind of filesystem is inside [02:36] then [02:36] maybe [02:36] * psusi nods [02:36] it's something else like lvm, in which case we add /dev/vg/lv symlinks [02:36] or maybe the filesystem is actually an lv or something sure [02:36] but that's just stacking [02:36] what I mean is some idiot disabled all those rules (create device node for dm devices, run blkid on them, etc.) for the "add" event [02:37] I thought it didn't matter for the add event, since the device is empty at that point... have to do all that during change [02:37] that's clearly what the person with no brain thought [02:37] but [02:37] during boot [02:38] what if the devmapper device is created before, say, udev is run? [02:38] then udev never knows about it [02:38] so we "coldplug" udev [02:38] ohh, then you only get the coldplug add event [02:38] right [02:38] so [02:38] on boot [02:38] all the dm devices that existed before udev were run ... [02:38] now don't get processed [02:38] and that neatly explains this bug [02:38] yep... [02:39] Hans did this? [02:39] no idea who did it [02:39] but it's wrong [02:39] badly wrong [02:39] (that being said, it partially explains a change to udev that will break dm-raid with your patch :p) [02:40] well, I still think the right thing to do is remove the partition table support from the kernel entirely and have udev invoke partx or similar to detect partition table and create partition devices ;) [02:41] I don't disagree [02:41] except that would make an initramfs mandatory [02:41] yep [02:41] but Ubuntu does not support installs that don't use an initramfs right? so... [02:42] There are many folks who want them though. [02:42] yes we do [02:42] it's a perfectly acceptable variation [02:42] we put a not unreasonable amount of effort into continuing to support it [02:42] well, acceptable sure if you want do deal with the config changes needed.. but not the default... [02:42] it's not the default, but it's *entirely* supported [02:43] in fact, it's very little config change [02:43] sed -i -e "/do_initrd/s/yes/no/" /etc/kernel-img.conf iirc [02:43] (and change the root= line) [02:44] well, imho if you want to continue to have the kernel support reading partition tables on dmraid disks then the kernel partition code needs extended to detect dmraid signatures and ignore disks that have tnem [02:44] I think we should round up everyone who knows about dmraid [02:44] and kill them [02:45] and burn the evidence [02:45] and then round up everyone who knows how to recreate dmraid [02:45] and kill them too [02:45] but at the same time, that is more complexity than I feel belongs in the kernel, hence, fuck it, move it to udev rules and get the partition table code out of the kernel ;) [02:45] hehe [02:46] yea... especially dmraid level 1 arrays [02:46] stripe is one thing, but people using fakeraid mirrors need slapped silly [02:46] since it does not have write intent bitmaps or handle errors [02:51] heh [02:51] I like my good old raid-1 mdadm mirror [02:51] it's nice and simple [02:52] except with the --incremental bug reassembling diverged legs of a mirror [02:53] I don't know about that bug [02:53] (and don't want to :p) [02:53] heh... plug in one disk, boot degraded, reboot with the other disk only, boot degraded... which is now the default... [02:54] huh? [02:54] and now you have to degraded arrays that have diverged from the original unified mirrored state... boot with both disks, and mdadm can put them back together assuming they are still in sync even though they are not and you end up with a fucked up filesystem that contains parts of both [02:54] the default is "don't boot at all" [02:54] at least it was [02:54] in case of disk failure, we always used to fail [02:55] not automatically repair or continue without the other half [02:55] naw, default now seems to be wait to see if the other disk shows up for a while, and if it times out, boot degraded [02:55] fail [02:55] Could I get an xserver-xorg-video-intel upload sponsored on bug #541511? It's just to drop the patch to disable DRI on 855 and 845 which didn't help as much as we thought it would. [02:55] Launchpad bug 541511 in xserver-xorg-video-intel "MASTER: [i855] GPU lockup (apport-crash)" [Unknown,Confirmed] https://launchpad.net/bugs/541511 [02:55] not a problem there in and of itself... if one disk has actually failed, then booting degraded can be better than not at all for a remote machine [02:56] problem is that should the next boot only find the second disk, now youh ave both degraded and mdadm incorrect glues the two back together if the third boot sees both disks [02:56] right [02:57] which is why we didn't boot degraded [02:57] a server should never come up broken [02:57] when the metadata on disk 1 days disk 2 failed and was removed, and the metadata on disk 2 says the reverse... mdadm --incremental happily reinserts disk 2 into the running array when invoked from udev on add [02:57] a sysadmin should be despatched to your DC to investigate [02:57] and if the event count happens to be the same mdadm assums they are in sync and does not bother doing a resylc when it recombines the disks [02:58] so you sometimes can read data from disk1, and sometimes from disk 2, and they are not the same [02:58] dispatched too probably [02:58] I suspect despatched is a completely different work ;P [02:58] yeah [02:58] raid failure is bad [02:58] naw, booting degraded the first time is good... since at least you have a server up and running, and emailing the admin that a degrade has occured [02:58] it's a manual fix, not an automatic one [02:58] no it isn't [02:59] you should never bring a server up with a known failure [02:59] that server is no longer redundant [02:59] it should stay off [02:59] a sysadmin should pull the affected disk [02:59] and replace it [02:59] yea, but it's no worse off than it would be if you weren't using raid 1 [02:59] then the server should be brought up [02:59] you never bring up a server *without* raid [02:59] or if the second disk had failed while the system was already up and running [02:59] then you shutdown the system [03:00] naw, you rebuilt using a spare disk hopefully, if you have one ;) [03:00] if not, then the admin hopefully gets the email about the failure quicly and does something about it [03:00] my thinking on this is probably more "enterprise" [03:00] systems *do not* come up broken [03:00] you have engineers employed to work at the data centre in case of a remote failure [03:01] why not? is there really any difference between coming up degraded, and becoming degraded while already up? [03:01] it sounds like "small town ISP" thinking is creeping into the server team [03:01] "if my remote server fails, I have to drive all the way downtown to fix it" [03:01] psusi: no, but becoming degraded while up is an instant shutdown event [03:01] not usually [03:02] that's one of the big reasons people use raid 1... if a disk fails you don't crash and have down time while you replace it [03:02] default config is to email root telling them HEY! DISK WENT BAD... AND I AM OR AM NOT AUTOMATICALLY REBULDING WITH SPARE [03:03] hopefully you have the spare so the system automatically returns to redundant status quickly [03:06] Of course we go to great lengths to avoid MTAs ending up in systems unless someone asks for it, so the odds of that email getting anywhere useful are not so great. [03:08] probably should not do that... [03:41] psusi: http://jdong.mit.edu/~jdong/ureadahead-pack.svg [03:41] psusi: y...yeah, some defragging needed (tm) [04:04] jdong: heh, what did you do to generate that? [04:05] jdong, neat, what do the colors mean? [04:07] different files I guess? [04:13] * ccheney thinks he found a bug or something wrt networkmanager and ipv4/ipv6 support [04:14] might already be reported though [04:14] if you have ipv4 and ipv6 both set to automatic but you don't have something to provide the ipv6 address it doesn't work at all, ie no ipv4 address either [04:14] is that supposed to happen? [04:22] ccheney: can't quite follow. Can you find it (with more details, screenshots, etc) [04:22] ccheney: certainly one of {ipv6,ipv4} failing should not be preventing the other from working, if there is the wherewithall to do so [04:22] ccheney: s/find/file a big/ [04:22] ccheney: s/find/file a bug/ [04:27] Keybuk: indeed different files; the svg is generated by a hackish python script that reads ureadahead --dump [04:27] the code is at bzr branch http://bradmisik.com/trunk/fragraph/ [04:27] jdong: I think the hackish python should be published ;) [04:27] ah, sweet [04:28] my friend wrote the code, lol I tossed him the idea early last week and he ran with it I guess :) [04:28] I just wanted code to generate tree-ring graphs for an unrelated project, but this is pretty neat too :) [04:29] but yeah, now that you're here, you can do a quick sanity check to make sure we're interpreting ureadahead's output correctly [04:29] each character is a page [04:29] @### represents a fragment of the file that's read in [04:29] with @ marking the first page, and # marking the rest of the pages read in that block [04:30] @##@## means an apparently contiguous part of the file is actually in two fragments [04:30] the numbers below are the same, but in terms of bytes and blocks [04:30] so yea [04:30] looks like you are [04:30] temugen, meet Keybuk; Keybuk, meet temugen :) [04:31] Nice to meet you. [04:31] temugen: congrats. you just got the Keybuk stamp of approval (tm) :) [04:31] and for the record, this is his ext3 disk on his messy thinkpad ;-) [04:31] so yeah [04:31] it's something like [04:31] Heh. Thanks. It's not entirely accurate, but I thought it was reasonable for one slide of a presentation [04:31] offset, len bytes at diskpos [04:32] you grab the len and the diskpos [04:32] so that'd be correct afaik [04:32] *nods* [04:32] pretty > accurate ;) [04:32] so wow, bootup files are virtually scattered across the disk then [04:32] jdong: now you know why I'm so keen for us to switch to btrfs [04:32] Keybuk: hehe I look forward to that :) [04:33] jdong: It's still slightly skewed since I don't draw the entire disk - only from the lowest read bootup block to the highest [04:33] Keybuk: speaking of that, *driveby bug* do you know what ureadahead doesn't seem to pick up a btrfs root fs? [04:33] and depending on the macroblock size it'll obviously look more clustered [04:34] jdong: doesn't pick up? how do you mean? [04:34] Keybuk: I don't get a pack generated for my / [04:34] what error do you get? [04:34] Keybuk: I don't get any error, just I only have packs for my boot and var/run mountpoints [04:34] jdong: kooky [04:35] I suspect it's yet another can't-match-the-UUID style bug? [04:35] jdong: wouldn't have thought so, if it gets mounted! [04:35] ureadahead doesn't care about such things [04:35] maybe btrfs doesn't support FIEMAP? [04:35] it should do though [04:35] what's a quick way to test that? [04:35] temugen: ooi, what are your plans for this code long-term? [04:35] jdong: err, strace ureadahead I guess ;) [04:36] it's about the only thing that uses that ioctl [04:36] Keybuk: my plans long-term were to give jdong a pretty slide for his presentation XD [04:36] haha [04:36] hahaha :) [04:36] Keybuk: what are *your* long-term wishes for this code? :) [04:36] would you have any objection if I included it in ureadahead? and maybe in the future ported it to C as another --dump mode? (with full credit to you, of course) [04:37] No objections at all [04:38] I think it could use a little bit of cleanup before that, but I'm never satisfied with my own code regardless [04:39] I'd need to know your licence terms for that ;-) [04:39] ureadahead is GPLv2 - is that to your liking? [04:40] jdong: OOI, if you're running btrfs, how do you get around http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=575891 ? [04:40] Debian bug 575891 in dpkg "dpkg makes wrong assumption about readdir() and lose metadata files with btrfs" [Important,Fixed] [04:41] oh, because apparently Colin backported it to lucid [04:41] in the _most_recent_ update ;) [04:41] Keybuk: (1) hehe was that why for several months I was randomly losing gconf default schema [04:41] (2) I saw the commit fly around the btrfs mailing list, and built my own btrfs from dkms [04:41] jdong: yeah, maintainer files had a tendency to go missing [04:42] ah, fun :) [04:42] because who cares about data as long as you've got the latest crack FS? :) [04:42] ajmitch: hey, I consider myself sacrificed for the science of testing :) [04:42] ajmitch: If it's not jdong, it's not crack. [04:42] Keybuk: That is fine. [04:42] ScottK: he's been known for such things for a long, long time [04:42] Yep. [04:43] There was something he recommended against because it was too insane and everyone immediately backed away. [04:43] hahaha I vaguely recall that :) [04:44] temugen: sweet, thanks :) [04:45] Keybuk: hiya ... have you tested raid much at all in 10.04? [04:45] Keybuk: is "ureadahead: Unable to obtain rotationalness for device 0:17: No such file or directory [04:45] as soon as we get a grub2 patch for btrfs, I'm going to switch again [04:45] normal? [04:45] kirkland: I don't test raid [04:45] jdong: hah, devmapper? [04:45] Keybuk: i've spent the last two days fighting lucid/raid [04:45] kirkland: raid sounds like the kind of thing the server team should test ;-) [04:45] Keybuk: oh yeah, LVM [04:45] Keybuk: is that why? [04:46] Keybuk: we usually cover it in vm's [04:46] Keybuk: No problem. If you have any ideas for it, I'm all ears. [04:46] Keybuk: my experience this weekend on real hardware hasn't been good :-/ [04:46] jdong: yeah, LVM have no underlying extends/devices/etc. [04:47] you end up with a slew of ureadahead warnings/errors followed by it ignoring the device [04:47] iirc [04:47] Keybuk: oh okay. awww so lvm and LUKS users don't get ureadahead love for now? [04:47] kirkland: not really something I care about, sorry [04:47] jdong: I don't think it's "for now" so much as "ever" [04:47] Keybuk: hehe okay :) [04:48] figuring out the on-disk extends and ordering of an encrypted filesystem ... [04:48] that sounds more like a *science* project! [04:48] Keybuk: when at an initramfs prompt, I can see a /dev/md0 and a /proc/mdstat that shows it operational; when i try to "mount /dev/md0 /root" in (initramfs), it fails, with Invalid Argument [04:48] Keybuk: and btw, have you seen ibuclaw's forward-porting of legacy grub device detection to grub2? [04:48] or the kind of thing where, if I succeeded, I'd be immediately hired by the NSA [04:48] Keybuk: http://ubuntuforums.org/showthread.php?t=1389279 "Hack GRUB2" [04:48] jdong: no, source? [04:48] kirkland: someone on the server team should be able to help with that ;-) [04:48] Keybuk: that's the patch I'm using so that update-grub behaves on btrfs [04:49] Keybuk: origin is bug 450260 [04:49] Launchpad bug 450260 in grub2 "grub-probe fails with btrfs root (and ext3 /boot)" [Medium,Triaged] https://launchpad.net/bugs/450260 [04:49] jdong: ah, that's different to having grub root device support [04:49] Keybuk: sorry, it's not the server team that has reinvented the boot process ;-) [04:49] Keybuk: oh yeah, right, I misunderstood your complaint :) [04:49] hmm [04:49] jdong: that bug is actually really quite interesting [04:49] Keybuk: none of the old tricks that use to work, work any more [04:49] Keybuk: yeah, that patch is also needed for a btrfs root even with ext3 boot [04:50] kirkland: if you're at an initramfs prompt, you're not into the boot process yet - and you're in an area that's barely changed since dapper [04:50] jdong: right [04:50] now [04:50] that's REALLY interesting [04:50] you're on btrfs root right now? [04:50] yeah, I've got a Karmic system up, and can boot up a Lucid VM. [04:50] * Keybuk joins up two dots and sees a dog [04:50] both btrfs root [04:50] jdong: can you grab the output of "stat /" for me? [04:51] Keybuk: http://paste.ubuntu.com/422540/ [04:51] aha! [04:52] so that ureadahead warning/error wasn't from devmapper [04:52] it was from your btrfs root, right? [04:53] Keybuk: correct [04:53] riiiight [04:53] sorry [04:53] I didn't join the dots [04:53] yeah [04:53] now that makes sense ;) [04:53] cool [04:54] ureadahead assumes that it can get the major/minor of a given filesystem by stat() on the mountpoint [04:54] yeah my lucid VM is devmapper-less and still doesn't generate a pack :) [04:54] ah [04:54] and convert that into a /sys/dev/block/%d:%d lookup [04:54] which it then uses to get at the queue/rotational file ;) [04:54] so for you it's looking for /sys/dev/block/0:17/queue/rotational [04:54] which doesn't exist [04:54] when it should be looking for /sys/dev/block/8:0/queue/rotational [04:55] ah [04:55] so yeah [04:56] exactly the same bug as update-grub actually ;-) [04:56] just in different code [04:56] makes sense. I was assuming the two were related [04:57] do I need to file a bug about this, or is this in the back of your mind? :) [04:57] yes, please file a bug [04:57] the back of my mind keeps falling off [04:57] I know what you mean :) Will do! [05:01] temugen: http://bazaar.launchpad.net/~scott/ureadahead/trunk/revision/109 [05:03] Keybuk: haha, I like the summary. Awesome. :) [05:04] :) awesome [05:04] temugen: ^5 [05:28] Keybuk: just a heads up - there's a template.svg script that the visualizer uses for building the svg file. I might submit a patch later to inline it, but I'm letting you know anyway [05:34] ah [05:34] I missed that? [05:35] added [05:44] Laibsch: the Takao fonts are derivatives of the official IPA fonts, which contain bug fixes that may go back into the IPA fonts. The IPA fonts are the industry standard fonts in Japan and it is highly desired that we ship and use them by default in order to gain market share in Japan. [06:00] Good morning [06:00] pitti: Good morning. [06:07] good morning #kernel [06:16] kees, jdstrand: ugh, qa-regression-testing is > 500 MB now? what have you guys done? :-) [06:17] sladen: oh ok, sorry i got plugged away from irc for a couple hours, so what i did was enabled ipv6 'automatic' just like is default for ipv4 in networkmanager 'edit connections' for my particular wifi setup [06:18] sladen: when i do that it refuses to connect at all since my router doesn't actually support ipv6 or at least it seems to do that (checking again to verify) [06:19] sladen: yea i have to set ipv6 'ignore' before it will let me connect again [06:20] sladen: i saw someone mention this issue on the comcast ipv6 test forum, they are about to start testing for widespread ipv6 rollout on their cable network in the US [06:20] ccheney: finally [06:21] i think the rollout probably won't happen until late next year, some of the hardware manufacturers still don't have ipv6 support aiui [06:21] but the testing is happening now to determine what bugs are still left [06:22] one thing that would keep it from being seamless for ubuntu appears to be ipv6 is disabled by default and if you enabled it and don't have ipv6 on your network it keeps the connection down completely even for ipv4 (which seems odd) [06:22] unless i am somehow doing something wrong, which if that is the case it seems non-intuitive [06:24] maco: if you have comcast you can sign up for ipv6 test group still :) [06:25] There's an open bug against (IIRC) eglibc about delays in this case, but not oughtright failure [06:25] hmm they appear to have removed the sign up page now, maybe its not open any more [06:25] * ScottK is a former Comcast customer due to a desire to have reliable internet [06:25] ccheney: Your experience mirrors mine - network manager bails if it can't get an ipv6 address, even though the dhcp server has provided a perfectly acceptable ipv4 address. [06:25] http://www.comcast6.net/volunteer.php [06:25] still open just didn't see it [06:26] RAOF: ok [06:26] who took over NM from asac, anyone happen to know? [06:26]