[10:56] hrm [10:56] with the latest dist-upgrade my babbage hangs somewhere in the init-bottom scripts [11:37] wow, i cant even chroot into the disk anymore from initramfs [12:04] you broke it [12:06] yeah, definately [12:06] no way to get it booting again [13:21] sigh [13:24] grrr and cdimage isnt reachable through rsync [13:24] * ogra goes mad [13:35] * persia provides ogra with a tea in which to dip himself [13:55] heh [13:56] i really dont get what broke it [13:56] looks like upstart hangs [14:03] upstart hangs? [14:03] Maybe set init to strace upstart? :) [14:04] well, if i boot without initramfs its hangs at Freeing init memory: 176K [14:04] no output afterwards at all [14:09] OK. What if you boot with init=/bin/bash without initramfs? [14:09] didnt try that yet, i'm just checking what updates come on a 1day old livefs [14:09] since it stopped booting after apt-get dist-upgrade [14:10] but there is nothing noticeable ... [14:10] i removed plymouth from disk already [14:10] busybox-initramfs cant be it since i did boot without initramfs [14:10] and the rest are desktop libs [14:10] That shouldn't have done much: I heard it didn't work for non-intel FBs right now. [14:11] well, plars said he sees a splash [14:11] (on dove) [14:11] I know. I don't understand why. [14:11] But slangasek said he didn't (on some non-intel x86 box) [14:11] And that he was trying to sort it, and suspected it was intel-only. [14:12] But my knowledge of plymouth stops after configure runs :) [14:12] http://paste.ubuntu.com/361263/ thats what apt-get dist-upgrade on a running livefs gets me [14:12] not sure what libbsd0 does [14:13] seems its in standard [14:13] It's some random BSD-only stdlib stuff that doesn't appear in linux. [14:14] * ogra tries what happens if he uses the redboot setup from teh livefs to boot [14:14] I don't see anything there that should break it either :( [14:16] Begin: Running /scripts/init-bottom ... [14:16] Done. [14:16] hangs ... [14:16] the funbs stuff is, it doesnt actually hang hard or something [14:17] i have a cursor and pressing enter moves it on the screen [14:18] Cool! [14:18] its just that init doesnt seem to proceed [14:18] set -x ? [14:18] in init ? [14:18] its a binary, not a script [14:18] Ah, I missed the "Done." [14:19] yeah [14:19] Freeing init memory: 176K doesnt show up if using an initramfs [14:19] i'll try /bin/bash [14:20] bash: cannot set terminal process group (-1): Inappropriate ioctl for device [14:20] bash: no job control in this shell [14:20] root@(none):/# [14:21] works [14:21] OK, so it's not the kernel. [14:21] no, else i wouldnt get into busybox either [14:21] i can properly break and the like [14:22] its also not uboot [14:22] since redboot shows the same bahavior [14:22] *be [14:22] bootloader shouldn't be affecting anything post-kernel load anyway. [14:23] if the HW is wrongly initialized it does [14:23] else we wouldnt have issues with the NIC ion uboot :) [14:23] *in [14:24] Well, that's HW interfaces. I like to live in a happy little world where userspace apps just talk to each other. Sometimes USB HID is permitted. [14:25] But yeah, blame upstart. See if there's some debug mode: perhaps you can get it to output which job it's trying when it hangs. [14:26] well, either upstart or the first script that runs or so [14:26] i dont get to fsck [14:26] and i dont see the mount /dev to none error message [14:27] Right, but for both cases you need upstart debug messages. [14:27] Or strace [14:27] gah, running dhclient in init=/bin/bash mode isnt such a good idea [14:27] (can one strace init on boot? That would be very helpful to determine the issue, since it's likely to be either exec(...) or some HW call. [14:27] no job control ... no ctrl-c [14:27] heh. No! [14:28] * ogra reboots [14:28] i think you can start init from bash in that setup somehow, Keybuk has magic commands for that [14:28] but i dont know them [14:29] oh [14:29] `strace /sbin/init` ? [14:29] my mtab is totally messy [14:30] write(2, "init: Failed to connect to socke"..., 74init: Failed to connect to socket /com/ubuntu/upstart: Connection refused [14:30] ) = 74 [14:30] pfft [14:30] Try asking in #upstart. Might be some leftover upstream around. [14:31] i can wait for monday too :) [14:31] i wouldnt see why upstart should suddenly stop working [14:31] especially since it works in the live image [14:32] Are there any update-initramfs calls that happen with apt-get dist-upgrade? [14:32] yes, plymouth enforces them [14:33] but i'm currently booting a live initramfs [14:33] Then get plymouth out before you upgrade, and see if that works. [14:33] already done [14:33] long ago [14:33] mount, chroot :) [14:33] that was my first setp :) [14:33] *step [14:33] Ah. [14:33] since indeed i did install plymouth yesterday [14:33] But you've already messed up the initramfs. [14:33] and then ran a dist-upgrade later [14:34] Thanks for that, by the way. I'd think it'd be cool if it worked. [14:34] no, i have plenty of initramfses around [14:34] lots of backups [14:34] Well, stuff in a working initramfs, and try to boot :) [14:34] its definately nothing in the initramfs [14:34] the live initramfs boots the live system just fine [14:35] so why shouldnt it boot my SATA disk [14:35] and in the livefs plymouth is the older version [14:35] The *same* initramfs can't boot your drive? Something is seriously wonky. [14:35] yes [14:35] but the issue has to be on the drive [14:36] * ogra removes the messy mtab [14:36] and rebbots ... [14:37] iirc the first thing that runs on boot is mountall [14:38] nope, mtab didnt help [14:39] hey ... [14:39] weekend reminder ;) [14:39] bah .. [14:39] fix my board and i can sleep tonight :) [14:46] root@(none):/# cat /etc/init/mountall.conf [14:46] root@(none):/# [14:46] hmm [14:47] heh [14:48] ogra: you shouldnt have tried to upgrade just today ;) [14:49] j.k. [14:49] dont know whats going on [14:49] asac_, i left it running yesterday [14:49] still wrestling ffox [14:49] when i got up today i just wanted to take a look [14:49] Done. [14:49] mountall: Could not connect to Plymouth [14:49] fsck from util-linux-ng 2.16 [14:49] e2fsck 1.41.9 (22-Aug-2009) [14:49] /dev/sda6: clean, 146974/2321984 files, 1007361/9277521 blocks [14:49] * ogra dances [14:49] finally ! [14:49] cheers [14:50] https://edge.launchpad.net/ubuntu/lucid/+queue?queue_text=chromium-browser# [14:50] ;) [14:50] doesnt seem to move further though [14:50] archive admins are laggers :) [14:50] heh [14:50] 12h [14:50] ;) [14:50] hmm, i suspect there is more trashed in /etc/init/ [14:50] crap [14:55] asac_: None of the archive admins have signed up for saturday or sunday as archive admin days. You'll probably have to wait until Monday. [14:57] i know [14:58] So, how are they laggers? [14:58] You could volunteer to be an archive admin :) [14:58] persia, they didnt approve it last night before finishing their day [14:58] and then you could pick one of the unclaimed days ... [14:58] indeed they are laggers :) [14:59] * persia does a timezone check [14:59] it surely was in the queue for at least 1h before the last AA left :) [14:59] Friday needs a backup archive-admin, it only has one. [14:59] Tuesday would also qualify for a backup, but doesn't need one. [15:20] bah, i'll do a reinstall on monday ... there is more broken than i thought ... [15:21] always hitting the reset button when working on bootloaders doesnt do good to a filesystem :P [15:21] though i thought ext4 was more robust [15:43] ogra: what error are you getting? [15:43] my usb-storage issues also complained about ext4 issues [15:44] i havent got any errors [15:44] i have some zero byte files [15:44] but its to late now, i wiped the partition [15:45] i threatened the disk really hard the last two weeks though ... its no wonder that there are corrupted files [15:45] about 50 resets per day or so [15:46] never shot it down properly [15:47] asac_, our uboot installation will cause a lot more issues than we had yet btw [15:48] it just struck me that we need to teach ubiquity that /boot can *only* live on an SD card [15:48] i dont think there is any code in partman that could do that yet [15:48] that will be a huge task to implement [15:49] unless we also want to put the images directly to the boot floppy as a backup [15:49] well, still the "boot floppy" has to be the SD [15:49] no [15:49] well [15:49] yes [15:49] ? [15:49] the babbage can only boot from Sd or flash [15:49] not on fs ... just behind the master boot record [15:49] raw data [15:50] and flash is to small to hold more than a bootloader [15:50] so flash-kernel will do it [15:50] right [15:50] but how does redboot do it? [15:50] redboot doesnt cerate /boot [15:50] raw fis partition [15:50] but where does it put the stuff? [15:50] right [15:50] which we dont have [15:50] so thats the equivalent we could do as a fall back [15:50] with uboot [15:51] hrm [15:51] we already have a /boot partition [15:51] why use two [15:51] we need to re-use that /boot partition anyway [15:51] yes. what i mean is: make the non-fs data partition bigger and put it in there as raw data if we dont find it through the boot.scr looping [15:51] but we need to teach partman and ubiquity that this is the only option [15:52] does redboot support usb? [15:52] no [15:52] thats why we took the bootfloppy approach with it [15:52] anyway [15:52] uboot will just make everything a lot more complicated [15:52] ogra: right. i mean we can do both [15:52] would be waste sure [15:52] sure, but that doesnt go into flash-kernel [15:52] but if ubiquity fix is too hard [15:53] well, it has to go into ubiquity [15:53] flash-kernel updates the data in the fis partition, right? [15:53] right [15:53] and not right :) [15:53] thats waht flash-kernel would do for uboot ... just put it in the no-fs data [15:53] depends where you look at [15:53] ok [15:53] in the installer it uses the postinst from the package instead [15:53] that would be the complex part [15:54] additionally /boot as vfat in the installed system wont work [15:54] something else ... can we grep sed etc. something in hushshell? [15:54] i did some tests and update-initramfs gets very unhappy [15:54] e.g. basically parsing numberes out of command output? [15:54] no, no grep, no sed [15:54] hush only has loops and conditionals [15:55] and the test binary [15:55] i didnt expect them to be there ... just hoped it had some syntax for parsing stuff [15:55] ok [15:55] why do we go for uboot again? [15:55] ;) [15:55] because david wants it :P [15:55] whats his motivation? [15:55] he thinks we gain anything if we can put multiple kernels into /boot [15:55] i thoguth we told fsl to move there and now they moved there and we have to follow [15:56] which is very hard to do (not impossible though) with a fis partition [15:56] yeah i see that [15:56] you can do it indeed [15:56] but you wont be able to easily select [15:56] i mean the gain from multiple kernels [15:56] fallback [15:56] but we need framebuffer in uboot to make that really useful for normal users [15:56] also if you write to fis and power off the board, your bootloader setup is trashed [15:57] (as you properly mentioned at some point) [15:57] yeah [15:57] the power-off scenario is the one that bothers david most [15:57] thats true. imo you would just another set of fallback fis partitions [15:57] and then you have at least 2 [15:57] but effectively, the more i work on our uboot i notice drawbacks [15:57] which should resolve that [15:58] uboot is definately slower in loading [15:58] well. uboot is buggy. that lets one think its all crap ;) [15:58] likely because of the FS inbetween [15:58] if it works and has usb support that would be great [15:58] yeah [15:58] i dont say its bad [15:58] ogra: uboot is probably slower because the mmc code is busted [15:58] loading the raw dat wasnt fast either [15:58] but loading kernel and initramfs takes abouot twice as long [15:58] i managed to load a 2M kernel with mmc read [15:58] and that took about 5 seconds (optimistically measured) [15:59] did you compare to redboot [15:59] but since we get hangs i wonder if thats really a generic mmc bug [15:59] yes, it should be really fast [15:59] < 1s [15:59] the time we load the kernel and initramfs the user sits in front of a black screen [15:59] there is something fishy with the mmc part i am 98% sure [15:59] i noticed i get impatient since using uboot [15:59] i doubt that [15:59] you simply have an additional layer between you and the data [15:59] how can you doubt that? loading 2M with rad mmc read [16:00] taking 5 seconds means that the mmc driver is busted [16:00] ogra: i am not talking about fatload [16:00] hmm, k [16:00] with the 2009.01 uboot ? [16:00] do this: type mmc read 0x90008000 3000 [16:00] or something [16:00] dont have help mmc at hand [16:00] i'm only takling about .01 [16:00] ogra: i only tried it on .08 [16:00] currently i cant, just running a reinstall [16:01] the blk number * 512 is the size [16:01] so 2048 is 1M [16:01] try to load that to some random address [16:01] if its fast then its fat [16:01] will do [16:01] we cant use fat in the installation anyway [16:01] we cant? [16:01] update-initramfs gets very unhappy [16:02] how? [16:02] we should fix that [16:02] it tries to do hardlinks in /boot for the backup files [16:02] isnt fat even included in default initram? [16:02] i'm talking about *update*-initramfs :) [16:02] not the initramfs itself [16:03] the hardlinks are temporary? [16:03] i dont se them in my /boot [16:03] yup [16:03] its during the .bak file creation [16:03] but surely has a speed reason that hardlinks are used [16:04] beyond that se should have vmlinuz and initrd.img symlinks that point to the current kernel [16:04] which isnt possible either on vfat [16:04] so during install we should reformat the SD /boot partition to ext2 [16:05] which sadly means we wipe it, which in turn adds another risk like the power-off scenario david fears so much [16:06] (only during install indeed, not for every kernel update or re-rolled initramfs) [16:06] so dove uboot supports ext3/4 or what? [16:06] no, it uses ext2 [16:06] but that can be anywhere (ide, usb, MMC) [16:07] but it uses fat on the live image? [16:07] and it doesnt re-use the installation media [16:07] or ext2 there too? [16:07] i think so [16:07] fat [16:07] but as i said, you dont re-use the same media [16:07] its differnt [16:07] we have at least 1 spare place for another partition on our live image in worst case [16:08] sure [16:08] but they will show up on your desktop :) [16:08] so we could simulate that, just that we have a special partition on the live disk because we cant do usb etc. [16:08] i will ask freescale whats up with usb support [16:08] maybe its really just filling in right numbers etc [16:08] far out ... last time i asked [16:08] no direver was written yet [16:08] hmm ok [16:09] but indeed we should ask [16:09] i asked at UDS time last time [16:10] maybe we should stop shipping images, but rather a image builder that allows to reproduce our exact "release" images and many more thing ;) [16:11] heh [16:11] that was the initial approach of rootstock [16:11] yes. something like that [16:11] just much more professional [16:11] that was the initial approach of rootstock :) [16:11] well ... professional in a way that i covers the whole stack [16:11] rotostock was only a quick first approach [16:11] from fs production to image for target device [16:12] i know [16:12] right [16:12] that were my plans [16:12] but they always fell over because of other more important specs [16:12] or because the images didnt work or... or ... [16:12] always something that stopped work on it [16:12] right. so we should kill the images and focus on that ;) j.k. [16:13] heh [16:13] but a good idea imo [16:13] we might be able next release, who knows [16:13] hrm, whats up with that installer [16:13] doesnt run [16:13] sigh [16:14] i thought partman was fixed [16:14] now that dove is fixed, imx51 is broken? ;) [16:14] its not arch specific [16:14] joking [16:14] the partman bug was there in A2 already [16:15] not sure if you were here still ... the images work again now ... the bootchartgui hangs turned out to be triggered by outdated .pyc files of the previous python ;) [16:16] the import uno hang feels like its caused because of our libgcc3_uno.so.jaunty to me [16:16] which i found out is pulled in by that import [16:17] uno == ooo [16:17] dove really feels like a bloody minefield to me :-P [16:17] yeah, i read the backlog [16:17] it is [16:18] the prob is that the silicon often doesnt fulfill the actually promoted features [16:18] seems if everything is working right, then all is fine [16:18] but if stuff that isnt expected happens, it goes mad [16:18] imx51 has its own issues though [16:18] like being behind on kernel or software being not ready (see uboot) [16:18] too bad ... so the fact that there were still firefox packages in hardy/dapper didnt help [16:19] the source is in new :( [16:19] that hit us way harder in the last reelases [16:19] the luck with imx51 vs dove this time is that the silicon doesnt change anymore [16:20] so for you it feels like dove is a minefield :) [16:20] but it was both arches behaving like that in the last releases [16:20] yeah. i can bet that [16:21] i think we could have done better [16:21] doko for instance need both boards [16:21] if thta was the case he would have noticed that dove hangs before uploading the thumb2 switch [16:21] yeah [16:21] otherwise ncommander or someone else should have checked that [16:21] well, but thzumb2 was advertised by the vendor [16:21] ;) [16:22] so whom do you trust :) [16:24] grumble ... [16:24] install doesnt want to start [16:31] ogra: so how was lost? [16:31] ;) [16:31] great :) [16:31] is that on kabel 1? [16:31] yeah, the new episodes are on K! [16:31] K1 [16:31] the new season starts while we are in portland :) [16:31] unfortunately final season [16:32] no, thats one season ahead of us [16:32] we have 5 here [16:32] yes, i am tracking US ;) [16:32] when we're in portland 6 starts in the US [16:32] ah [16:32] already have seen all ;) [16:32] heh [16:32] unforunately [16:32] but fortunately too [16:32] i'm pondering a HD+ reciever to watch them in HD [16:32] the voices are really better in original [16:32] as usual [16:33] but i watch them with susie [16:33] heh [16:33] he english doesnt suffice :) [16:33] i can give you the full collection ;) [16:33] but not dubbed :) [16:33] noo [16:33] subtitles are good for her ;) [16:34]