[02:16] <chatter29> hey guys
[02:16] <chatter29> allah is doing
[02:16] <chatter29> sun is not doing allah is doing
[02:16] <chatter29> to accept Islam say that i bear witness that there is no deity worthy of worship except Allah and Muhammad peace be upon him is his slave and messenger
=== zyga_ is now known as zyga
[13:36] <tisch> Hi, all. I'm new to reporting kernel bugs, but could I try my luck here?
[13:55] <apw> tisch, you are not in the wrong place
[14:23] <tisch> thats good to know, i just wanted to gather some general information for submitting a bug regarding a mainline kernel (4.10.12)
[14:25] <tisch> i've previously sumbitted to systemd, but the maintainer pointed me to the ubuntu kernel maintainers. See my report here: https://github.com/systemd/systemd/issues/5790#event-1054663503
[14:27] <apw> generally they are keen for you to talk to whoever packaged what you are using
[14:27] <apw> as we might have broke it.  in the case of mainline kernel-ppa builds they are in theory
[14:27] <apw> raw upstream bits, so the testing is more relevant there
=== JanC is now known as Guest16266
=== JanC_ is now known as JanC
[14:29] <tisch> ok, thanks apw. could you point me to a website where i would best submit this bug?
[14:32] <apw> what sort of bug is it ?
[14:37] <tisch> using the 4.10.12 kernel i'm unable to login on a gnome 16.04 system. The system freezes after putting in the login information. Same happens when I try to login from the shell
[14:41] <tisch> The following ouptut appears on the shell repeatedly:
[14:41] <tisch> MI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [systemd-logind:913y]
[15:05] <apw> tisch, sounds like a real bug indeed
[15:11] <tisch> on 4.10.11 the issue does not appear. I also have the same issue on 4.11.0-rc8
[15:12] <apw> sounds very bisectable then
[15:13] <tisch> apw, i've never done a git bisect before. how can i aid the kernel team further to fix this bug. do you need hardware information?
[15:40] <LocutusOfBorg> tisch, can you reproduce with a different pc?
[15:41] <tisch> LocutusOfBorg,yes, i could try to boot my old laptop and install the kernel version. it'll take a little momemt
[15:42] <LocutusOfBorg> does this happen with a vanilla kernel?
[15:42] <LocutusOfBorg> where did you get that kernel?
[15:43] <tisch> i installed the kernel using the ukuu kernel updated gui tool on gnome 16.04
[15:44] <tisch> so, it should be vanilla i guess
[15:47] <LocutusOfBorg> does this happen with 4.11.0-rc7?
[15:48] <tisch> i just tried 4.10.11 (OK), 4.10.12 (error) and 4.11.0-rc8 (error) so far, but i can go ahead and try with 4.11.0-rc7 if it helps
[15:48] <LocutusOfBorg> yes
[15:49] <tisch> ok, one sec. i'll be back in a minute
[15:54] <tisch> same error on 4.11.0-rc7
[15:54] <LocutusOfBorg> tisch, also, please test this packages http://kernel.ubuntu.com/~kernel-ppa/mainline/
[15:54] <LocutusOfBorg> oh, interesting
[15:54] <LocutusOfBorg> so it is needed to understand if rc5, rc6 works
[15:55] <LocutusOfBorg> to bisect we need to know the first rc broken
[15:55] <tisch> ok, i'll work my way down the kernel rcs for 4.11 then
[15:55] <LocutusOfBorg> yes, and also understanding if the kernel.ubuntu.com version works would be nice
[15:55] <LocutusOfBorg> maybe this is a known and already patched/fixed issue
[15:56] <LocutusOfBorg> if the problem is on the rc kernel, it has been backported on the stable branch
[15:56] <LocutusOfBorg> and this is bad
[15:57] <tisch> ok,  understand, how do i install the kernel from the kernel ppa mainline website you sent? So far I've really just used a gui tool like ukuu
[15:58] <LocutusOfBorg> download the deb files from the website
[15:58] <LocutusOfBorg> and sudo dpkg -i *.deb
[15:58] <LocutusOfBorg> download only for your arch :)
[15:59] <tisch> the ones for 4.11.0-rc8, i guess then  (http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.11-rc8/)
[15:59] <LocutusOfBorg> exactly
[15:59] <LocutusOfBorg> or just dpkg -i various versions, so you can reboot a few times and quickly find the one that is bad
[15:59] <LocutusOfBorg> (if canonical version suffers from the same issue)
[16:00] <tisch> just out of my curiosity, those are not the same kernels which i already installed through ukuu?
[16:00] <LocutusOfBorg> they carry some Ubuntu patch
[16:00] <tisch> ok, i see
[16:00] <LocutusOfBorg> with a big value of "some"
[16:00] <LocutusOfBorg> :)
[16:03] <tisch> ok, so i downloaded the files for amd64 headers and generic without the lowlatency ones. i guess this is correct. 
[16:05] <LocutusOfBorg> two generic and one _all.deb you need
[16:06] <LocutusOfBorg> headers-generic image-generic and _all.deb
[16:06] <tisch> yeah, looks right
[16:06] <tisch> and now sudo dpkg -i *.deb 
[16:06] <LocutusOfBorg> exactly
[16:07]  * LocutusOfBorg will leave shortly
[16:07] <tisch> ok, it's installed. i'll reboot and be back in a minute
[16:10] <tisch> same issue with the downloaded kernel
[16:10] <LocutusOfBorg> wonderful
[16:10] <LocutusOfBorg> so this is an upstream issue not fixed by canonical
[16:11] <LocutusOfBorg> I would complain with upstream kernel devs in this case
[16:11] <tisch> should i report somewhere else?
[16:11] <LocutusOfBorg> but bisecting and understanding when it has been introduced (at least the first bad rc) is required
[16:11] <LocutusOfBorg> so, please try to understand that
[16:12] <LocutusOfBorg> in case we find the commit, we can just email the author
[16:12] <tisch> ok, so i'll just work my way down the rcs
[16:12] <LocutusOfBorg> or send to kernel mail list
[16:12] <LocutusOfBorg> yep
[16:12] <LocutusOfBorg> you can use the ppa versions, easier to install/remove
[16:12] <tisch> can i use the ukuu tool or d i have to download from the ubuntu kernel ppa website?
[16:13] <tisch> who should i talk to when i found the rc which introduced the regression?
[16:13] <LocutusOfBorg> to me it is fine
[16:13] <LocutusOfBorg> I will try to have a quick look
[16:13] <tisch> thanks
[16:13]  * LocutusOfBorg is going to take the train, will disconnect and reconnect
[16:14] <tisch> ok, so i'll work my way down. i'll write when i found the highest working rc version
[16:30] <tisch> i just checked the ukuu documentation which repository the tool uses, the doc says: "Fetches list of kernels from kernel.ubuntu.com", so i guess those are the same as i manually downloaded and installed then
[16:35] <tisch> LocutusOfBorg: kernel 4.11.0-rc6 works fine
[16:35] <tisch> so the regression is introduced in rc7 i guess
[16:43] <LocutusOfBorg> architecture is amd64?
[16:44] <LocutusOfBorg> cpu is intel I would guess
[16:44] <tisch> yes
[16:44] <tisch> yes it's an i7-7700
[16:44] <LocutusOfBorg> interesting
[16:44] <LocutusOfBorg> does this happen with other amd64 laptops?
[16:45] <tisch> i haven't tried yet. but the other laptop i have at hand is pretty old. i guess it's first generation mobile i7
[16:46] <tisch> build date 2009
[16:48] <tisch> it's also an amd64 architecture though
[16:50] <LocutusOfBorg> not sure, I see two commits that have been backported and are related to cpu
[16:52] <LocutusOfBorg> this commit is really suspicious https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=7f00f388712b29005782bad7e4b25942620f3b9c
[16:52] <apw> LocutusOfBorg, ugg, thanks
[16:53] <LocutusOfBorg> apw, do you agree? I would like to do a test-build for him without that commit
[16:53] <LocutusOfBorg> but I never looked at the ubuntu kernel
[16:54] <apw> why is it suspicious ?
[16:54] <apw> it sounds like it fixes something legitimate
[16:55] <LocutusOfBorg> because git diff reveals only two changes in the cpu code
[16:55] <LocutusOfBorg> and the second one is not so interesting
[16:55] <LocutusOfBorg> this one is about locks and race conditions
[16:55] <LocutusOfBorg> and both changes have been backported to stable, between 4.10.11-4.10.12
[16:55] <LocutusOfBorg> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=f2200ac311302fcdca6556fd0c5127eab6c65a3e
[16:55] <LocutusOfBorg> this is the other one
[16:56]  * LocutusOfBorg did grep for cpu
[16:56] <apw> well cirtianly you could try reverting it, but we can likely spin you a test kernel if there is a bu
[16:56] <apw> bug associated with the issue
[16:58] <LocutusOfBorg> tisch, can you please open a bug and subscribe me?
[16:58] <LocutusOfBorg> I'm downloading the artful kernel, patch it, and upload into xenial ppa
[16:58] <LocutusOfBorg> and see if the stable+ the two commits is broken or not
[16:58] <tisch> yes, sure. can you point me to a webpage where to open the bug
[17:00] <LocutusOfBorg> ubuntu-bug linux should work
[17:00] <LocutusOfBorg> even if opening bugs for not yet packaged linux versions is... strange
[17:01] <apw> it may not let you file a bug, with a non-official kernle installed, it is kinda picky
[17:02] <LocutusOfBorg> if my connection allows, I'm uploading a  4.10.0-19.21~ppa1 to my ppa
[17:02] <LocutusOfBorg> with the two commits above
[17:02] <LocutusOfBorg> otherwise, git bisect ftw
[17:15] <jsalisbury> LocutusOfBorg, I can build some test kernels for you, if you need.
[17:17] <LocutusOfBorg> jsalisbury, http://paste.debian.net/929141/
[17:17] <LocutusOfBorg> this patch on top of the latest kernel code you have (zesty?artful?)
[17:17] <LocutusOfBorg> if my assumption is correct, that will introduce the issue
[17:17] <LocutusOfBorg> build for xenial please
[17:19] <jsalisbury> LocutusOfBorg, that commit is upstream as of 4.11-rc7.  It's not in artful.  You want me to build you a Xenial test kernel with a pick of that commit?
[17:19] <LocutusOfBorg> jsalisbury, the artful kernel, with that upstream cherry-pick for xenial
[17:20] <LocutusOfBorg> I'm trying to bisect a regression between 4.11-rc6 and 4.11-rc7, and I would like to cherry-pick the two interesting commits, to see if an older kernel crashes with them
[17:20] <jsalisbury> LocutusOfBorg, will do.  did you happen to open a bug yet?  
[17:20] <LocutusOfBorg> no
[17:20] <LocutusOfBorg> it is a mainline issue right now :)
[17:21] <LocutusOfBorg> in case we reproduce the problem, either the first or the second commit will be the culprit, in that case I'll ask tisch to send emails to who wrote the patch and mail llists
[17:21] <jsalisbury> LocutusOfBorg, it would be helpful if you could open a bug, that way we can keep track of everything.  I should have a test kernel ready in about 20 minutes or so
[17:21] <LocutusOfBorg> I already asked tisch to do it, I'm not the person with that issue :)
[17:21]  * LocutusOfBorg is leaving shortly
[17:22] <tisch> yes, i can open the bug report
[17:22] <jsalisbury> LocutusOfBorg, ok.  
[17:22] <jsalisbury> tisch, thanks
[17:22] <tisch> no, guys, thanks goes to you for all the heavy lifting
[17:23] <tisch> the bug report should go here? https://bugs.launchpad.net/ubuntu/+source/linux
[17:24] <LocutusOfBorg> yep
[17:26] <tisch> any suggestiosn for the summary and further information fields for the bug report?
[17:27] <jsalisbury> tisch, just a basic description of the bug.
[17:27] <tisch> ok, will do
[17:34] <tisch> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1685865
[17:34] <ubot5> Ubuntu bug 1685865 in linux (Ubuntu) "Can't login after boot with Kernel 4.11.0-rc7, soft lockup in systemd-logind" [Undecided,New]
[17:34] <tisch> thats the bug report, i've subscribed LocutusOfBorg and jsalisbury 
[17:35] <jsalisbury> tisch, great, thanks
[17:35] <LocutusOfBorg> yep thanks
[17:35] <jsalisbury> tisch, I have a test kernel building now.  I'll let you know when it's done.
[17:36] <tisch> thanks joseph
[17:36] <jsalisbury> np
[17:36] <tisch> i'll try to reproduce the bug on my other eold laptop (first gen mobile i7)
[18:32] <jsalisbury> tisch, The test kernel is available for downloaded from:
[18:32] <jsalisbury> http://kernel.ubuntu.com/~jsalisbury/lp1685865/
[18:49] <tisch> thanks, i'll install it right away. do i need to download and install all deb packages? 
[18:50] <tisch> jsalisbury: or can i leave something like cloud-tools out?
[18:50] <jsalisbury> tisch, just the linux-image and linux-image-extra .deb packages.
[18:59] <tisch> jsalisbury: the issue seems to be fixed with the compiled test kernel
[19:00] <ivan> http://www.openwall.com/lists/oss-security/2017/04/24/4
[19:01] <jsalisbury> tisch, hmm, I think LocutusOfBorg believed this bug would happen if commit f2200ac311302f was added.
[19:01] <jsalisbury> tisch, that might mean that commit is not the cause of the bug.
[19:02] <tisch> ok, thats unfortunate
[19:02] <jsalisbury> tisch, I can perform a bisect to try an narrow down the offending commit
[19:02] <jsalisbury> tisch, Just to confirm the bug happens with -rc7 but not -rc6?
[19:03] <jsalisbury> tisch, also, have you tried -rc8 to see if the bug was already fixed?
[19:03] <tisch> jsalisbury: sure, just to confirm, the kernel i booted right now is: uname -a
[19:03] <tisch> Linux tisch-XPS-15-9560 4.10.0-19-generic #21~TestKernelWithCommitf2200ac3113 SMP Mon Apr 24 17:28:58 UTC  x86_64 x86_64 x86_64 GNU/Linux
[19:03] <jsalisbury> tisch, yes, that looks correct
[19:03] <tisch> jsalisbury: yes, the bug is present in all version higher than rc6 and 4.10.12
[19:04] <jsalisbury> tisch, ok, I can start a bisect between -rc6 and -rc7.  It will require testing of about 7 - 10 test kernels.  Would you be able to test?
[19:05] <tisch> sure, i'll be of as much help as i can.
[19:06] <jsalisbury> tisch, I'll build the first kernel .  I'll post links to the test kernels in the bug report
[19:07] <tisch> ok, i'll grab the links from there and install and test those kernels
[19:07] <jsalisbury> tisch, thanks
[19:07] <tisch> thanks to you man!
[19:08] <tisch> as is said, you're doing the heavy lifting compiling kernels here
[19:08] <jsalisbury> tisch, its no problem at all
[20:07] <CarlFK> I likely did something wrong, but in case anyone wants a panic - pxe booted vmlinuz-4.10.0-19-generic.efi   https://veyepar.nextdayvideo.com/static/temp/pxe_efi_panic1.png
[20:15] <tisch> jsalisbury: there is no linux-image-extra in http://kernel.ubuntu.com/~jsalisbury/lp1685865/82f1faa86727de976e38eade5e96a1846742d71e/ so ig uess i'll just download linux-headers*_amd64.deb and linux-image*_amd64.deb?
[20:26] <jsalisbury> tisch, correct, that is a mainline kernel, so there is no -extra package.  You just need to install the linux-image pacakge.
[20:27] <tisch> jsalisbury: ok, thanks. i'll also leave out the linux-image*_all.deb i guess?
[20:28] <tisch> jsalisbury: sorry i meant linux-headers*_all.deb
[20:28] <jsalisbury> tisch, correct.  you just need this file: linux-image-4.11.0-041100rc6-generic_4.11.0-041100rc6.201704241508_amd64.deb
[20:31] <tisch> jsalisbury: thanks for the clarification. and sorry for me being not really literate about installing kernels
[20:31] <jsalisbury> tisch, sure, no problem
[20:56] <LocutusOfBorg> thanks jsalisbury for the kernel help!
[20:56] <LocutusOfBorg> and sorry for not finding the right one :(
[21:20] <LocutusOfBorg> sigh maybe the culprit is c4a3fa261b16858416f1fd7db03a33d7ef5fc0b3