=== emma is now known as em [06:40] moin === smb` is now known as smb [07:04] ppisati, morning === fmasi_afk is now known as fmasi [07:45] infinity, Still got a bunch of Xen uploads on my mind which are giving me headaches. For Saucy there is a FFE as well... [07:55] xnox, when did we switch to upstart user sessions by default ? [07:56] apw: week1 of saucy [07:56] apw: raring had it available (~2-3 weeks after feature freeze), but needed a manual configuration file change. [07:56] to activate. [07:57] xnox, i mean as a default option on a new image [07:57] in saucy [08:02] apw: yeah, saucy desktop cd booted very early in the cycle into upstart user sessions by default. On touch it's different story, as there was no lightdm, so it took some time before it gained upstart user session support. [08:02] (there is a package done by ogra integrating that in) [08:02] on server/tty there is still no user sessions. [08:02] At least it would not explain the sudden rise of boot time [08:03] smb, no? would not early in teh cycle be june ? [08:03] Oh hm [08:05] smb: apw: in /etc/upstart-xsessions comment out your session (e.g. ubuntu) and reboot. If there is no automatic user login enabled, then upstart user sessions do not kick in at all.... [08:05] xnox, we don't use upstart if you type a password ? [08:06] xnox, Hm, right, so I type in my password then its not upstart sessions? [08:06] apw: we use it, but it doesn't affect the bootchart. [08:06] xnox, boot charts continue through a password ? [08:06] smb: apw: upstart user session is kicked off, after a graphical login. [08:06] xnox, well to build the graphical login right ? [08:07] xn [08:07] xnox, it runs gnome-session etc [08:07] no. lightdm is running under system init, not session init. [08:07] right .... in smb's case the new 'delay' is all after he hits return in the password prompt [08:07] maybe i'm not clear.... [08:07] apw: oh. [08:08] so in /etc/X11/Xsession.d there is 00upstart & 99upstart, which is what starts user session. [08:09] i don't think we did any benchmarks / charts on how long the login takes with & without upstart user session. [08:10] xnox, I just commented out what you said and rebooted the machine. Need to grab the new chart though [08:11] From my feeling it still toook a while [08:12] apw: what's "pumbing" vs "xorg +plubming" vs "desktop" on http://reports.qa.ubuntu.com/bootspeed/machine/2/amd64/ [08:13] xnox, so basically plumbing is done when root is mounted and we move to real /, Xorg is started when X the process is running, desktop starts when the session starts [08:14] xnox, you can see by following the lines down to where they coincide with a specific process start [08:14] xnox, Ok, so it seems its still about 15s [08:14] maybe 2s less [08:15] well one can twiddle the init line in the 99upstart script to increase verbosity (e.g. --debug or --verbose). [08:16] to get all events... but the culprit might not be an upstart managed process =/ [08:17] xnox, indeeed, smb lets do a few runs on each to make sure it is real i guess [08:22] apw, ack. from the bootchart it seesm a bit like compiz is doing something a lot and I can see something about the time that gets less frantic and the gtk-decorators start [08:23] err [08:23] * smb looks at the empty space where all the indicators *should* be [08:26] hm. we added upstart support startpar, thus for each init.d script we take penalty of it checking weather there is upstart jobs or init.d script. [08:27] good: from june https://jenkins.qa.ubuntu.com/job/bootspeed-saucy-desktop-amd64-acer-veriton-02/27/artifact/11/bootchart.svg [08:27] bad: current https://jenkins.qa.ubuntu.com/job/bootspeed-saucy-desktop-amd64-acer-veriton-02/117/artifact/11/bootchart.svg === fmasi is now known as fmasi_afk [08:36] xnox, Just to make sure, after commenting out the "ubuntu" in /etc/upstart-xsessions, is none of the indicators coming up an expected result? [08:37] smb: yeap, looks like tedg migrated those to be upstart jobs only without any other way to activate them (e.g. no dbus activation) [08:38] xnox, Ok, so its not a complete fair comparison, but at least it seems to indicate that the indicators are not the main reason for the delay [08:40] somehow on the bootchart.svg i linked above Xorgs comes up 5 seconds later, despite the first inital stage (networking? it's aligned with hostname job) finishing 2 seconds earlier. [08:42] lool: any chance we can get an up to date flash-kernel in ubuntu? we are still stuck at 3.0 while debian has 3.10 [08:43] lool: and they have rudimentary support for contatenating kernel+dtb, something we need too [08:46] xnox, One of the "advantages" I got with having to log in is that I get that a bit more separated between coming up to the greeter and when hitting enter and see those ending and the session to start [08:48] xnox, It looks at least like 9s between compiz starting and until gtk-windows-decorators starting and I believe that is about when the screen is turned on [08:50] ppisati: I haven't had time to work on it in the Debian side for a while, and the Ubuntu one had tons of delta that I couldn't safely rework without a lot of time [08:51] lool: ok, then i'll import the dtb thing support in ubuntu [08:51] ppisati: ok [08:51] lool: actually what debian has is _really_ trivial, but we can plug on it [08:51] ppisati: and I would like too to see the two versions being closer or identical even :-) [08:52] ppisati: cool [08:54] actually the ubuntu version is at 3.4/3.5 [08:55] but pkg version was never updated [09:49] oh goody [11:52] * henrix -> food [12:32] what happened to xfs? it seems to have been removed from a kernel some weeks ago making my laptop unbootable [12:48] nfk, from which kernel [12:48] in which series even [12:49] nfk, it is enabled in raring and saucy according to the configs for example [12:57] nfk, ? === psivaa is now known as psivaa-lunch === kentb-out is now known as kentb === kentb is now known as kentb-afk === psivaa-lunch is now known as psivaa [13:41] latest kubuntu [13:41] 3.8.30 or thereabout i think [13:42] apw, ^ [13:45] debian.master/config/config.common.ubuntu:CONFIG_XFS_FS=m [13:45] nfk, so it is enabled there [13:45] nfk, how can you tell the kernel doesn't supportit [13:45] and it was never disabled? [13:46] i think i managed to boot that kernel manually and it didn't list xfs partitions or something like that, let me try again [13:46] indeed, it seems to have always been on for all of raring [13:46] while the previous kernel at least got to emergency shell (although didn't mount / as i didn't know how to specify that) [13:47] also unlike gentoo's dracut initrd that one seemed so basic it was unusable for recovery, derp [13:47] then again, maybe that was because it couldn't mount / [13:50] nfk, it has enough bits to mount root in general and little more [13:50] why would it have more [13:51] so the previous kernel didn't work either ? [13:51] has this system ever worked? [13:51] and if so which kernel works there [13:51] nfk, ^ [13:51] yes, it's an upgrade from the last version that had alternative cd [13:51] the only way to install on lvm2 [13:54] why is grub so hard to boot manually [13:54] for starters it's not even possible to read all of help [13:54] nfk, i thought the live CD now did lvm [13:54] pretty sure that is how i installed a machine on my desk here [13:55] anyhow is this breaking at grub or at initrd [13:56] i have no idea at this point [13:56] i can't even manage to boot any kernel at this point [13:56] they have rather different prompts [13:56] grub i am pretty sure has grub> in its prompt [13:56] and initramfs has a big splash when it drops there [13:56] even though i'm also glancing at the grub config of this gentoo box and have booted grub1 many times manually and at least once also grub2 [13:56] but if you arn't able to tell perhaps a photo? [13:56] ah, that, yeah, i'm stopping at grub [13:57] and ls from there says what [13:57] nothing [13:57] it's the fucking grub [13:57] i bet it can't read its config or something [13:57] so you ahve an emergency grub prompt ? [13:57] would be nice if i could tell it to load one [13:57] yes [13:57] what sort of LVM config do you have [13:58] ie, what is /boot [13:58] i have no idea, it's legacy from fedora 16 or something [13:58] and last time grub at least read out lvm volumes but not this time [13:58] horrible thing [13:58] making this easy arn't we [13:59] grub can grok lvm nominally it does on many of mine [13:59] it did so last time for me too [13:59] but not this time [13:59] nfk, so, your prompt looks like grub> yes ? [13:59] yes [13:59] and ls produces absolutly no output [14:00] and i'd apply a brick by now to that promt if i didn't feel sorry for the laptop [14:00] let's restart [14:00] it does produce output, it's listing files on / [14:00] ok ls for me shows me the partitions only [14:00] no, ls lists files on / [14:01] ls with no parameters doesn't do that for me on my grub> prompt her [14:01] it says like * [14:01] (hd0) (hd1) ... [14:01] rebooted, grub only states the version and gives generic information about the prompt [14:01] yeah, now it does that for me too [14:01] where's my brick? [14:02] and does it list your lvm things [14:02] yeah [14:03] arp, if i do ls then it starts to list / [14:04] nfk, so i htink you can then do 'configfile /boot/grub/grub.cfg' [14:04] and then perhaps hitting ESC will get you your menu [14:04] so i managed to make it read the configfile and nothing really happened except now it only shows grub> [14:04] ah [14:04] let's try esc [14:05] nope, esc didn't help [14:05] nothing at all ? [14:05] yeah, boot tells me to load a kernel first [14:05] so i guess it didn't like the configfile [14:05] i mean when you hit ESC you didnt get any sort of output [14:05] assuming cat works... the fucking file is empty [14:06] yeah, grubenv prints ############## so i guess the config is empty [14:06] sorry for bothering you [14:06] i guess i may have booted those last two kernels differently back then [14:07] cat works ok here [14:07] so ... you need to make up the config by hand [14:08] linux /vmlinuz [14:08] initrd /initrd [14:08] boot [14:08] shouldn't kernel something root=label=herpes [14:08] might work [14:08] initrd something [14:08] yeah [14:08] i'm fairly sure it won't boot without / [14:08] but let's try [14:09] root= which you likely need to know [14:09] yep [14:09] didn't work out without root [14:09] if it is lvm it might be easu [14:09] i have no idea how to specify it except for label or uuid (no way) but ic can't make grub print them out this time [14:09] root=/dev/mapper/- [14:10] if you are using lvm [14:10] oh, i see why i thought xfs was removed [14:10] (which it cannot be as we are at grub not the kernel) [14:10] the latest kernel says that it tried ext3 ext4 vfat fuseblk [14:11] apw, actually grub should be able to do it, i just don't know how [14:11] i thought grub didn't have a working config [14:12] how did it even get to the kernel to say that [14:12] i set kernel and initrd [14:12] bear in mind i can see the 26 lines of what you typed and nothing else [14:13] i think that implies it didn't load an initrd [14:13] as you don't get that error unless the kernel is itself trying to mount / [14:13] and from its point of view / is the initramfs [14:13] nfk, ^^ [14:14] and it will only try the ones it can do direct which are the builtins, and wouldn't include anything in lvm regardless [14:14] nfk, if your grub.cfg is 0 length then perhaps so is your initrd [14:14] try rebooting to grub and loading a much older kernel [14:15] leet's see [14:15] and its associated initrd [14:15] that might explain why trying to boot the previous kernel at least got me to the emergency shell [14:15] if you are unable to boot that, i would then say you need a recovery image, a live USB stick or something [14:15] so we have some more tooling to look at the image [14:16] if i had that, i'd be installing opensuse about now [14:16] i don't have capabilities to write cd/dvds at this point [14:16] and my only usb stick is a known buggy kingston that boots only when threated with violence and 220V [14:17] shame you are making life hard for oneself [14:17] anyhow try and older kernel [14:18] with the matched initramfs [14:18] this box had a cd/dvd writer but it literally bite the dust and went crazy [14:18] had to remove it [14:19] *bit [14:19] the *old in / didn't work out, let's try the ones in /boot [14:19] yeah pick a pair in /boot [14:20] nfk, now, i assume the mesasge you saw with the latest kernel was: [14:20] List of all partitions: [14:20] stuff [14:20] No filesystem could mount root, tried: [14:21] kernel panic - not syncing... [14:21] after that [14:21] yeah [14:21] yeah so that is kernel side, which means it did not load initrd correctly [14:21] so that implies initramfs is also broke, perhaps 0 length [14:22] anyway, even 3.8.28 didn't work and i'm certain it got to emergency shell last time [14:22] i guess it didn't like root=/dev/mapper/vg_my-root [14:22] did it fail with the same message as before or by dopping to busybox [14:22] if you didn't get a busy box you didn't get an initrd [14:22] maybe ubuntu uses real_root not root for the actual /? [14:22] yeah, same message as before [14:23] then your issue is your initramfs somehow [14:23] you would not get that message with a valid one [14:23] you would get a busybox shell, and if you do not have an initrd [14:23] you cannot ever boot an lvm system, full stop [14:23] as it is the proggies in there which find it [14:23] so we [14:23] so, we have to be suspicious of your initrd contents there [14:24] let's cat it [14:24] how many kernels do you have in there [14:24] printing a lot [14:24] still cat'ing [14:24] but its not loading right somehow [14:24] and still [14:24] else the kernel would mount it [14:24] and not spew its root not found thing [14:24] (as root in its world should be the initrd) [14:25] stil lgoing and not even all zeroes [14:25] yep [14:25] i have quite a fe [14:25] w [14:25] including something like 3.2 === kentb-afk is now known as kentb [14:25] if it is real then its 40MB [14:25] what? [14:25] if the initrd is real then it is not tiny [14:26] 2,8M /boot/initramfs-3.9.10-gentoo.img [14:26] 2,8M /boot/initramfs-3.9.11-gentoo-r1.img [14:26] 5,6M total [14:26] those boot this system [14:26] depends if they are minimal ones, or default ones [14:26] this /boot is merely 30 MB [14:26] minimal ones are much smaller [14:26] look i don't want to argue about how big the world can or is [14:27] though i'm migrating to 100M one someday this month [14:27] anyway, still cat'ing [14:27] i'd say the initrd looks legit [14:28] ls -l would have worked btw [14:28] and which kernels have you tried, what filenames [14:28] cat is more fun and i never knew they were that large [14:28] i'm used to few MB files [14:28] even 1MB is going to take 'some time' through that emulated output channel [14:29] the kernels are larger than their initrds [14:29] right, ^C didn't work, fun times [14:29] thanks ibm for the 3 finger salute [14:30] oh, ls -l actually prints UUIDs [14:30] so... root or real_root or what? [14:30] as i keep saying, if you are seeing the kernel panic on mount [14:30] you are not getting an initrd [14:31] root is a kernel command line, and not going to help [14:31] and we use root= [14:31] does boot show it loading both at the top ? [14:31] both? [14:31] vmlinx and initrd [14:31] it read RAMDISK and then EOF and go uncompression error [14:32] and uncompression error [14:32] that sounds bad [14:32] how big did you say it was, ls -l on it [14:32] let's reboot again [14:32] an uncompresson error from the kernle, will bin the initrd as if it isn't there [14:33] did we have a nasty crash or odd behavior before the boot which didn't work ? [14:33] 9.62k if i'm worth any salt [14:34] the system was often low on free space on / but i'm fairly sure after latest updates it had 400M free [14:34] 9.62 k ? [14:34] that is definatly not an initrd [14:34] although i had to do them via apt-* as GUI just outright crashed because it ran out of free space (removing older kernels freed up 2G, btw) [14:34] nfk, so do you have any initrd's which are bigger [14:35] -rw-r--r-- 1 root root 31M Aug 24 22:23 initrd.img-3.8.0-29-generic [14:35] that is a generci one, one which is portable, and the default options make these sort of sizes [14:35] both -28 and -29 are around 32M [14:35] ok boot one of that pair with its paired kerenl image [14:36] and then rebuild your initrd for the latest kernel [14:36] i'm fairly sure i have tried them [14:36] no dice [14:36] but let's try again [14:36] as a test i just booted a kernel with just [14:36] no root=, correct? [14:36] linux /boot/vmlinux-VER [14:36] initrd /boot/initrd-VER [14:36] boot [14:36] and it fails, because there is no root= [14:36] but ... it drops me correclty to the (initramfs) busybox prompt [14:37] erp [14:37] i may have cocked up, didn't find a valid RAMDISk [14:39] wow [14:39] got to the emergency shell [14:39] finally [14:40] anyway, where's the fstab? shouldn't an initrd have one so that it knows what to mount? [14:41] heh no [14:42] so you should be able to use lvm commands now [14:42] via the lvm wrapper [14:42] to confirm the name of the disk [14:42] disk? [14:42] the voluem [14:42] grub already told me both pv and lv names [14:43] well the key i was trying to achieve was finding a working initrd [14:43] so you have a pair there, i believe if you reboot usingf those [14:43] and i did set them as root=/dev/mapper/vg_my-root [14:43] while there have a look in /dev/mapper [14:43] ls /dev/mapper [14:43] and you can see the actual name you need to use in root= [14:44] yeah [14:44] vg_my-root [14:44] there is some odd mapping with s/-/-- [14:44] so you booted these linux/initrd combination with root=/dev/mapper/vg_my-root [14:44] and still ended up at (initramfs) [14:45] i used it for linux command [14:45] cat /proc/cmdline [14:45] should i have set it for initrd too? [14:45] nope you did it right in theory [14:45] not this time [14:45] else i'd get the lovely not syncing error [14:45] ? [14:46] what? [14:46] nfk you could try mounting it now [14:46] mount /dev/mapper/vg_my-root /root [14:46] to where? [14:46] and if that works you could hit ^D [14:46] and see if it boots [14:48] invalid argument [14:48] wtf [14:48] replacing mount with ls seems to work so the path have to be right [14:48] wait [14:48] are you sure you know lvm? [14:49] ? [14:49] * xnox raises an eyebrow and goes to fetch more coffee. [14:49] i mean, is this really how lvm is mounted? [14:49] nfk, it and i are not intimate in the bedroom departement but i have perhaps 6 systems with lvm on it [14:49] i'm about to lower my head and fetch something stronger [14:50] yes lvm is mounted out of initrd, cause the kernel doesn't know how [14:50] ** [14:50] ** Ubuntu Kernel Team Meeting - Today @ 17:00 UTC - #ubuntu-meeting [14:50] ** [14:50] which is why you have dmsetup in your initrd to hadle it [14:50] nfk, what did cat /proc/cmdline say before you tried that [14:51] the expected [14:51] BOOT_IMAGE=blablabla [14:51] nothing about root as i didn't set it [14:51] ok so .. redo the boot with the same pair but adding root= [14:51] and set to what? [14:52] root=/dev/mapper/vg_my-root [14:52] if that is what your root is called [14:52] my is actually something more personal but yeah [14:53] kernel panick [14:53] -k [14:53] as usual [14:53] and couldn't find a valid ramdisk [14:53] what on earth [14:54] and without root= it worked, that makes no sense [14:54] let's try again, maybe i'm just too tired [14:57] so it worked out this time [14:57] so if you get logged in [14:57] i may have set the wrong initrd by accident [14:57] so slow that it might be booting [14:57] then you want to rebuild the initrd, adn grub cfg [14:57] update-initramfs -u; update-grub [14:57] then you might be able to reboot again [14:58] if that works we know it was the space issue [14:58] ah, so it's network that's making my startup so slow always [14:58] anyway, apw, hi5 [14:58] it booted [14:58] now time to regenerate the config [14:58] it was only grub being horrible, at lesat i hope so [14:58] *least [14:59] note to self: don't reboot after runnig out of space, without checvking you ahve a valid kernle/initrd/grub.cfg [15:00] i did check that there were like 400M free and that there were no errors while finishing up updating [15:00] oh [15:00] my bad, 1.3G free [15:00] that can't be from running out of free space [15:01] depends how many old kernels you ripped [15:01] they are pretty huge including headers [15:01] unless it ran out of free space on the GUI update, crapped itself but marked updates as finished (can't happen on gentoo but maybe it can on kubunu/ubuntu) [15:01] anyhow, i presume you can recover this into normality [15:01] not an installer expert, though dpkg is pretty careful normally [15:01] yeah, though i'm wondering if it won't repeat itself [15:02] though initramfs and grub.cfg are not 'updated' per-se [15:02] those are generated on the system [15:02] you don't have a separate /boot here [15:02] oh my HARUHI [15:02] the grub config now cats just fine [15:02] what on... i dunno, hell? [15:03] lvm tended to have a separae partition for /boot [15:03] it previously didn't print anything from grub prompt but now it's looking fine [15:03] which would show up in df obviously [15:03] it's on / [15:03] also i could access kernels and initrds just fine [15:04]