bjfkano:  git://kernel.ubuntu.com/ubuntu/linux.git linux-3.5.y00:10
=== Guest54733 is now known as nuxninja
=== zequence_ is now known as zequence
smbgood morning apw ;-P10:02
psivaaapw: smb: hello morning, would you know if anyone is looking at the issue in bug 1100386 ?10:03
ubot2Launchpad bug 1100386 in linux (Ubuntu) "Raring server installations on VMs fail to reboot after the installations" [High,Confirmed] https://launchpad.net/bugs/110038610:03
smbpsivaa, I think I had a quick look there yesterday and it seemed Joe was on it to some degree (or I got distracted and just forgot about it)10:05
psivaasmb: ok thanks, it would be helpful, daily smoke is getting hit with this bug10:13
smbpsivaa, btw, just for future reference, I find it more helpful to have text files attached uncompressed. That gives you a bigger chance that I will actually look at them because I don't have to save them before that. ;)10:17
psivaasmb: ack, ill remember that10:18
apwpsivaa, so this is server images only ?10:24
apwppisati, and is this xen or kvm, virtmanager can be either10:25
smbOh and is that virt-manager with xen or kvm or what?10:25
smbpsivaa, Hm, and not sure it is already in the report: how are those installs done? netboot /maas or really using any isos?10:27
apwjodh, hey we have a case here where upstart is aborting, and dumping core, that seems utterly specious as the machine will panic instantly and not sync the dump10:35
psivaaapw: smb: that is only for server images10:48
psivaait could be observed on both virt-manager and virtualbox as stated in the bug and the installation is using isos i did not use netboot or mass but preseeded and manual iso installations10:49
apwvirt-manager using KVM or virt-manager using Xen as the virtualisation tech ?10:49
psivaaapw: virt-manager using KVM10:50
rbasakapw: hey, any known bugs with overlayfs and armadaxp? I have a regression between precise and quantal, looks like it affects raring as well.11:07
rbasakDoesn't affect highbank11:07
rbasak(in precise and quantal, not checked raring)11:08
apwrbasak, nope, overlayfs is generic and used on other arm platforms ok11:18
rbasakapw: I'll file a bug then. Thanks!11:19
apwrbasak, what sort of regression is it ?11:21
rbasakapw: it's broken :) http://paste.ubuntu.com/1558509/ - "mv" doesn't work for existing files in lowerdir. This breaks sbuild which is how I noticed it.11:21
apwrbasak, what filesystem type is / in this case ?11:27
rbasakapw: aha. That might be different between my armadaxp and highbank installs!11:27
* rbasak looks11:27
rbasakapw: ext4 on both, but with slightly different options11:28
rbasakarmadaxp: /dev/sda2 / ext4 rw,relatime,data=ordered 0 011:28
rbasakhighbank: /dev/disk/by-uuid/02fcfe14-c5ec-4371-9d59-7da564b2b428 / ext4 rw,relatime,errors=remount-ro,user_xattr,barrier=1,data=ordered 0 011:28
apwrbasak, user_xattr is likely your problem11:29
rbasakI'll try ripping that out now11:29
apwperhaps the armadaxp is missing support for them in the kernel config (assuming you cannot just turn them on)11:29
rbasakapw: one of us has this backwards. armadaxp does not have user_xattr but does have the issue. highbank does have user_xattr but does not have the issue11:30
apwas the last thing it does before that error is:11:30
apw        err = vfs_setxattr(newdentry, ovl_whiteout_xattr, "y", 1, 0);11:30
rbasakBoth fstabs are identical11:30
rbasakI think I might follow you now.11:30
apwrbasak, that is consistent, you likely need xattrs for userspace enabled which is what that flag enabled11:31
apwso i suspect that some XATTR related config option is off in the kernel config11:31
rbasakI see11:31
rbasak# CONFIG_EXT4_FS_XATTR is not set11:31
rbasakShould overlayfs be #ifdef'ing that vfs_setxattr line?11:32
apwnope, it need to mark it as a whiteout, that is what i uses, xattrs11:33
apwit should be on, your config is spack11:33
rbasakThen overlayfs should refuse to mount with an upper dir without xattr enabled surely?11:34
apwwell some of its functionality works, so perhaps, perhaps not11:34
rbasakOK fair enough11:34
apwif it hurts, stop11:34
rbasakSo what do we do to fix this? Raring - enable it? And Quantal?11:34
apwfix the config11:34
apwit is wrong in both11:35
rbasakIs an config change SRU acceptable for quantal?11:35
apwit fixes a bug, so i can't see a reason to say no, it is enableing something whihc is on in _all_ other configurations11:35
rbasakIt is nice for sbuild to work on armhf for development work as reinstalls take ages.11:35
rbasakOK. Do you need a bug filed?11:35
apwso its not something to be scared of11:36
apwbug> yes, as all srus need them11:36
rbasakOK I'll do that now.11:36
rbasakapw: thanks. You just saved me hours of tracking this down. I would have started a bisection :-/11:36
apwrbasak, always worth asking11:38
apwpsivaa, just did an install using virtmanager with xen and that worked just fine11:38
psivaaapw: ok, i have not tried with xen though, could it be then KVM related?11:39
apwpsivaa, could be, though less likely when you say virtual box has the issue too11:40
apwrbasak, as overlayfs doesn't work that means we have never tested livecds with this h/w11:40
psivaaapw: yes virtual box had the issue, and is your installation amd64 or i386 server?11:40
smbpsivaa, Perhaps you could provide us with the seed file and the steps to apply that to the install in the bug report?11:40
rbasakapw: not sure there is a live cd for armadaxp :)11:40
apwpsivaa, i did a 64 bit install on a 64bit raring host11:41
einonmHi, is there an easy way to remove old self-compiled kernel files from an ubuntu install? I used 'make modules_install install' to install them.11:46
psivaasmb: apw: ok, the panic occurred with both amd64 and i386 installs on amd64 host. Although the occurrence is more frequent on i386 installs. Ill add the preseed and steps for auto installs in the bug anyway11:48
smbeinonm, Depends on whether you made the version numbers simple to be recognized... rm -rf is simple12:00
einonmsmb: It's a linux-next tree, and I know where most of the files are, like /lib/modules and in /boot. But there's grub2 scripts that have changed also.12:02
smbeinonm, No, just re-run update-grub after you removed the files from /boot12:03
einonmok, I've been doing that - I hoped there was a ready made way. thanks12:04
=== henrix_ is now known as henrix
rbasakapw: filed https://bugs.launchpad.net/ubuntu/+source/linux-meta-armadaxp/+bug/1102970 - should that be linux-armadaxp or linux-meta-armadaxp or something else?12:20
ubot2Ubuntu bug 1102970 in linux-meta-armadaxp (Ubuntu) "Missing CONFIG_EXT{3,4}_FS_XATTR breaks overlayfs" [Undecided,New]12:20
rbasakubuntu-bug led me to linux-meta-armadaxp which seems wrong to me12:20
* henrix -> lunch12:42
apwjodh, hey ... so are we using upstart in the initramfs now ?13:02
jodhapw: no.13:41
jodhapw: the problem is bug 1096531. I've fixed this upstream, but it's not in Ubuntu yet.13:42
ubot2Launchpad bug 1096531 in upstart (Ubuntu) "After touch /forcefsck and reboot: Assertion failed in log_clear_unflushed" [High,In progress] https://launchpad.net/bugs/109653113:42
psivaasmb: apw: i have just attached a preseed file to the bug 1100386 with steps 13:48
ubot2Launchpad bug 1100386 in linux (Ubuntu) "Raring server installations on VMs fail to reboot after the installations" [High,Confirmed] https://launchpad.net/bugs/110038613:48
jodhpsivaa/apw: I'm looking at doing a cherry-pick fix for bug 1096531 today.14:02
ubot2Launchpad bug 1096531 in upstart (Ubuntu) "After touch /forcefsck and reboot: Assertion failed in log_clear_unflushed" [High,In progress] https://launchpad.net/bugs/109653114:02
psivaajodh: ack, thanks14:03
apwjodh, ok so the issue is actually reproducible on random boots of the installed image, could that still be that bug ?14:10
jodhapw: I'm sure this is the same issue. See comment https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/1096531/comments/4 for the triggering factors.14:12
ubot2Ubuntu bug 1096531 in upstart (Ubuntu) "After touch /forcefsck and reboot: Assertion failed in log_clear_unflushed" [High,In progress]14:12
jodhpsivaa/apw: ... but to be convinced, could you please try the workarounds in #5 to see if you can still reproduce the problem?14:14
apwjodh, ok just added --no-log to the kernel command line, and it is still reproducible14:17
jodhapw: are you sure? the failure is coming from the logger code which should not be used with that option.14:18
apwjodh, it is definatly getting to upstart as --debug emits lots of upstart output14:18
apwjodh, yes, confirmed that --no-log is in the boot options always (added to /etc/default/grub)14:18
apwand on 1 boot in 2 (ish) i get this hang, so this is probabally not the issue you see i recon14:19
apwjodh, so my symptoms are the boot just stopping, immediatly after the fsck output14:19
jodhapw: that must be a different issue. What environment are you seeing this problem in?14:23
apwjodh, i've only seen it with server cd installs booted under kvm14:23
apwjodh, this is occuring in my installed image on around half my boots14:24
jodhapw: what's the cpu usage for the kvm process when it hangs? Is it spinning do you think?14:25
apwjodh, if i strace the kvm process it is waking up on a 1s basis and doing a small ammount, not spinning14:27
apwjodh, just did a boot with --verbose and it definatly is in upstart, i see the upstart bridge start then it clears the console and does the fsck, and wedged14:28
sforsheeapw, I bisected the wireless problem and found a commit from broadcom that introduced a bug, so it's not a ucode issue after all14:28
sforsheeapw, they've supplied a patch, care to test it?14:28
apwsforshee, sweet, it is pissing me off regularly14:28
apwsforshee, YES :)14:28
sforsheeapw, http://people.canonical.com/~sforshee/brcmsmac-txstatus-build/14:28
sforsheeI'm about to give it a spin on my machine, but it looks like it should get the job done14:28
ckingnice one14:29
jodhapw: can you try booting to a shell, remounting rw, removing /forcefsck, then rebooting?14:29
apwjodh, i doubt it even has that file, but will check14:29
apwjodh, yeah that forcefsck is _not_ there, so this is not that, whatever it is14:32
jodhapw: when did your problem start?14:32
apwjodh, i am booting todays images, i am only aware of it from today, i have no information from before14:33
apwjodh, i am only aware because it was reported as above by psivaa 14:33
apwjodh, be i think they imply it is a recent issue14:33
apwjodh, is there any further debug i can ask for above --debug ?14:35
jodhapw: no. can you get a serial console log though?14:36
apwjodh, maybe14:37
=== cmagina_away is now known as cmagina
=== henrix_ is now known as henrix
apwjodh, ok it is possible to add the serial being logs14:52
apwjodh, but when i log the output to there as well it doesn't reproduce any more14:53
apwjodh, any other ideas ?14:53
jodhapw: you could boot to a shell, remount / rw then create tty8.conf to "start on startup". That should allow you to login and see the status of the existing jobs.14:54
jodhapw: does friendly-recovery work btw?14:55
apwjodh, i am a kernel engineer :)  what the heck is that14:56
apwjodh, i am just using the rare good boot to change things14:57
jodhapw: select the "recovery" option on the boot menu :)14:57
apwoh that i've used a few times, always successfully14:57
jodhapw: it would be worth trying as that *is* actually being started by upstart.14:57
apwjodh, ok i have done two of those and get to the menu and select 'continue normal boot'14:59
apwjodh, and it boots fine, i will do a coupkle more, but i just got 14:59
apwinitctl: Event failed14:59
apwon the console14:59
apwjodh, could that be indicative of a lost event?15:00
jodhapw: probably related to a failed job. We really need to see the console log.15:04
apwjodh, cannot get it for you without losing the issue15:05
jodhapw: can you try the tty8 trick? atleast then you should be able to login and sniff around.15:05
apwjodh, that sounds good, what the heck is it15:06
jodhapw: see above. Either create tty8.conf or modify tty[1-6].conf to 'start on startup' and reboot.15:06
jodhapw: make that tty[2-6].conf.15:07
apwjodh, ok i seemed to have a login on tty8 when it booted normally, but in the 'broken' state i do not15:10
apwthough i did see init: debug for many 100s of lines before the occurance15:10
apwjodh, so now i am very confused15:11
jodhapw: how about booting without --debug and seeing if you can login in the wedged state: we can still see that job state then.15:11
jodhapw: I've got a fully up-to-date server and can't reproduce what you're seeing. A list of jobs would also be useful. I'm going to do the cherry-pick now, but could you raise a bug on this so we can track it?15:13
davmor2hey guys is there a guide on debugging bluetooth issues anywhere I have a new Lenovo Ideapad y580 and the bluetooth constantly say Bluetooth is disabled whether it is on or off.15:15
apwjodh, ok i still have nothing on /dev/tty8 odd15:16
jodhapw: could this be a weird udev issue maybe?15:17
* ogasawara back in 2015:28
apwjodh, it could indeed, bloody thing15:39
thotypoushi, which should be marked duplicated of which? https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1101797  https://bugs.launchpad.net/ubuntu/+source/linux/+bug/110135515:48
ubot2Ubuntu bug 1101797 in linux (Ubuntu) "On shutdown: “umount2: Device or resource busy, umount:/dev/sda2 busy - remounted read-only” appears" [Undecided,Confirmed]15:48
ubot2Ubuntu bug 1101355 in linux (Ubuntu) "can't umount home partition on 3.5.0-22-generic (works on 3.5.0-21-generic)" [Medium,Confirmed]15:48
jsalisburythotypous, 1101797 should probably be marked as a dup of 110135515:54
* jsalisbury notices a famous guest on http://ubuntuonair.com/15:59
=== pgraner is now known as pgraner-afk
=== pgraner-afk is now known as pgraner
* ppisati goes out for a bit...17:11
=== jsalisbury changed the topic of #ubuntu-kernel to: Home: https://wiki.ubuntu.com/Kernel/ || Ubuntu Kernel Team Meeting - Tues January 29th, 2013 - 17:00 UTC || If you have a question just ask, and do wait around for an answer!
jodhpsivaa: I'm just testing the fix for bug 1100386, but having issues on my SSD system. Looks like a kernel bug as opposed to an Upstart issue, but it's looking now like it'll be tomorrow realistically before the fix is uploaded to allow time to investigate.17:28
ubot2Launchpad bug 1096531 in upstart (Ubuntu) "duplicate for #1100386 After touch /forcefsck and reboot: Assertion failed in log_clear_unflushed" [High,In progress] https://launchpad.net/bugs/109653117:28
psivaajodh: ok thanks for the information, i assume you've seen the nomodeset related information that was added to the bug17:31
smbProbably something in the cirrus drm module (or related). Not sure whether that can explain the virtualbox hang as well. 17:33
smbDepends a bit what gfx hw they emulate. Xen for example has uses some different pci ids which do not cause the drm module to load. And I could not make it hang there.17:34
jodhpsivaa: yes - there seem to be 2 problems here.17:36
=== yofel_ is now known as yofel
bguthroI'm trying to google for the answer to this, but probably haven't gotten the right keywords. Is the ubuntu-raring.git repo planned on shipping with 3.8 - or will development continue to 3.9?19:31
ogasawarabguthro: we'll ship Raring with a v3.8 kernel19:32
bguthroogasawara, thank you - much appreciated.19:33
ogasawarabguthro: we'll eventually open a Raring+1, ie a 13.10 kernel, which will continue tracking the latest kernel, eg a 3.9 kernel once it opens19:33
bguthroogasawara, thanks. That seems to be a trend (or at least has been for the past 3, or 4 releases where I've been paying attention) - Is this done for all releases, or just the .04 releases?19:36
ogasawarabguthro: it's something we try do for all releases, just makes maintenance going forward easier19:37
bguthroogasawara, understood. Thanks for the explanation.19:37
jsalisburybjf, herton, bug 1101666 appears to be a regression in Precise and Quantal related to inotify.  A bisect may be possible, just wanted to give you a heads up.19:46
ubot2Launchpad bug 1101666 in linux (Ubuntu) "inotify fd leak" [Medium,Incomplete] https://launchpad.net/bugs/110166619:46
bjfjsalisbury, if you can get that identified "soonish" we can get that fixed for the point release. we (ogasawara) is working another regression at this time.19:49
jsalisburybjf, ack.  I'll kick off a bisect19:49
hertonjsalisbury, also this looks like related to the sauce/upstream fsnotify patches. Someone posted two day refering to two other bugs (1101355, 1101797). It looks related, so perhaps to cut time you can revert just the fsnotify patches and ask for testing.19:53
jsalisburyherton, ack, I'll do that.19:54
bjfarges, you have any thoughts on bug 110166621:03
ubot2Launchpad bug 1101666 in linux (Ubuntu Quantal) "inotify fd leak" [Medium,In progress] https://launchpad.net/bugs/110166621:03
argesbjf: hi looking21:03
argesbjf: jsalisbury : so which set of fsnotify patches are you reverting? the last set i cherry-picked from upstream21:05
jsalisburyarges, yes, but just to build a test kernel for bug 1101666.21:05
ubot2Launchpad bug 1101666 in linux (Ubuntu Quantal) "inotify fd leak" [Medium,In progress] https://launchpad.net/bugs/110166621:05
jsalisburybjf, arges, there seems to be another set of patches in comment #6 of bug 1101355 21:08
ubot2Launchpad bug 1101355 in linux (Ubuntu) "can't umount home partition on 3.5.0-22-generic (works on 3.5.0-21-generic)" [Medium,Confirmed] https://launchpad.net/bugs/110135521:08
bjfjsalisbury, are you chasing both Precise and Quantal or focusing on one?21:11
jsalisburybjf, I figured I'd build a test kernel for both since there are reports for both21:12
bjfjsalisbury, ack21:12
jsalisburybjf, for the revert anyway21:12
jsalisburybjf, hmm, I'm getting an ABI error during the build after reverting the patches:21:13
jsalisburyII: Checking for changes to ABI...21:13
jsalisbury    HASH : fsnotify_destroy_mark                    : 0x6f87a911 => 0xe80f761021:13
jsalisbury    HASH : fsnotify_alloc_group                     : 0xef67e06f => 0x81e5619921:13
jsalisbury    HASH : fsnotify_init_mark                       : 0xfbfedde7 => 0x0098ed6721:13
jsalisbury    HASH : fsnotify_add_mark                        : 0x82789f61 => 0x72e1be1221:13
jsalisbury    HASH : fsnotify_put_group                       : 0x9ad3a0b6 => 0x95f2960e21:13
jsalisbury    HASH : fsnotify_put_mark                        : 0x67507862 => 0x9e5033ad21:13
jsalisburyEE: 6 symbols changed hash and weren't ignored21:13
jsalisburyII: Module hash change summary...21:13
jsalisbury    vmlinux                                 : 621:13
jsalisburyII: Done21:13
jsalisburymake: *** [abi-check-generic] Error 121:13
bjfjsalisbury, i can believe that though i've not looked at the patches21:15
bjfjsalisbury, is this P or Q that you are talking about?21:15
jsalisburybjf, what I posted above was from P.  But it happens to both P and Q21:15
bjfjsalisbury, is 3.2.0-36.56 good or bad?21:15
jsalisburybjf, bad21:17
bjfjsalisbury, and 3.2.0-36.57 is also bad?21:18
jsalisburybjf, actually 3.2.0-36.57 is bad.  I'm not sure about .56, but I can check21:19
bjfjsalisbury, that would be best to try first21:19
jsalisburybjf, ack, testing 3.2.0-36.56 should be the same as reverting the fsnotify patches, correct?21:20
bjfjsalisbury, it changes one set of fsnotify patches for another21:21
jsalisburybjf, right, got it.21:21
=== sabayonuser2 is now known as Tuxkalle
=== lborda_ is now known as lborda

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!