[01:20] <xcyclist> Ubuntu kernel builds appear to put out a bunch of stuff, .deb files, .udeb files, and a .tar.gz file, that I don't see specified in a document accurately.  Is there a document I SHOULD  be reading, or can I help updating one that is old?
[01:20] <xcyclist> I have been using https://wiki.ubuntu.com/Kernel/BuildYourOwnKernel, and for instance it says I should see 3 .deb files instead of dozens of files.
[01:20] <xcyclist> Please advise how I can be most helpful in either asking a question, reading other documents, or updating inaccuracies.
[01:31] <xcyclist> I apologize.  I lost my connection.
[07:13] <juergh> xcyclist, running 'fakeroot debian/rules binary' (from the wiki page you reference) simply does a build of the binary packages for your currently running architecture (which results in 3 debs). A full official build will result in a whole lot more packages: binaries for all the supported architecture, indep packages, source package, ...
[08:26] <apw> udebs for the installer and for x86 a signing tarball
[09:21] <tjaalton> sforshee: ping bug 1752507
[09:21] <ubot5`> bug 1752507 in linux-firmware (Ubuntu Artful) "add i915/glk_dmc_ver1_04.bin" [Undecided,New] https://launchpad.net/bugs/1752507
[10:01] <thresh> hello.
[10:05] <thresh> I'm on 4.13.0-36-generic, backported to 16.04, and see this crash/lockup ocasionnally: http://thre.sh/stuff/v05.crash.txt
[10:05] <thresh> does that sound similar to something already known?
[13:04] <sforshee> jjohansen, tyhicks: either of you have any ideas about this? https://paste.ubuntu.com/p/JBHVsV9SsZ/
[13:04] <sforshee> it's happening during adt, but only on ppc64el
[13:16] <tyhicks> sforshee: hey - what's being tested when that's happening?
[13:20] <sforshee> tyhicks: I don't think we know, cascardo?
[13:23] <cascardo> I am not sure, but it might be apparmor
[13:23] <tyhicks> that sounds reasonable
[13:23] <cascardo> or something way before that, cause we don't know exactly when it starts
[13:24] <tyhicks> it looks like something is testing/fuzzing the /proc/<PID>/attr/current interface
[13:25] <sforshee> the audit backlog messages start about 1.5 minutes later, possibly these two things aren't related
[13:27] <tyhicks> the "AppArmor: change_hat: ..." messages don't go through the audit subsystem
[13:27] <tyhicks> so they're unrelated in the sense that they're not causing the audit backlog issue
[13:28] <tyhicks> but it is still possible that AppArmor is generating the audit messages that are causing the audit backlog issue
[13:28] <cascardo> I would disagree
[13:28] <cascardo> [121737.681674] audit: type=1400 audit(1520969595.539:16276): apparmor="DENIED" operation="setprocattr" info="current" error=-22 profile="unconfined" pid=7572 comm="bash"
[13:28] <cascardo> but, yeah, I also agree that might be a red herring
[13:29] <tyhicks> cascardo: to be clear, I suspect that something is testing apparmor and apparmor is generating a bunch of audit messages
[13:30] <tyhicks> what problems is it causing? is the testbed failing because of this?
[13:30] <sforshee> tyhicks: the test does fail, we're not sure that it's related to all of those messages but it's one of the few leads we've got right now
[13:31] <sforshee> rather the vm seems to get hung somehow
[13:32] <sforshee> but I'm also now suspecting that it may be that a test is trying to suspend/resume the vm and that it's dying there
[13:33] <tyhicks> neither of those sets of error messages (the ones from AppArmor nor the ones from audit) indicate a dire situation that would take the vm down
[13:33] <tyhicks> they're error conditions for sure but they're handled error conditions
[13:34] <sforshee> ok, thanks tyhicks
[13:34] <cascardo> from my little experience with autopkgtest, sometimes the backend will look for the login prompt on the console
[13:34] <cascardo> and I thought that was the problem, that so many errors on the console would prevent it from finding it
[13:35] <cascardo> but maybe the nova backend is not that broken
[13:35] <tyhicks> that's possible
[13:35] <tyhicks> I'd argue that the apparmor error message is mostly useless and should go away
[13:35] <tyhicks> the audit message is important and can't go away
[13:35] <tyhicks> so, getting a clear console might not be possible
[13:36] <cascardo> that's okay
[13:36] <cascardo> I think the problem here is how it would generate that so many messages in that small amount of time
[13:36] <cascardo> when trying to sigstop auditd, I couldn't get that to happen even when writing to attr/current in a loop
[13:37] <tyhicks> huh, why is auditd running in the VM?
[13:37] <tyhicks> that's surprising to me
[13:37] <cascardo> I got a slightly different behavior as well, a "lost" message as well
[13:37] <cascardo> ah, I am not saying it was
[13:37] <tyhicks> oh, I see what you're saying
[13:37] <cascardo> that's just the only thing I thought would have caused those backlog messages
[13:38] <cascardo> like: there need to be an open auditd socket, right?
[13:38] <tyhicks> the audit subsystem forwards to the syslog when an auditd hasn't registered itself with the kernel
[13:39] <tyhicks> that's the default configuration on ubuntu
[13:39] <tyhicks> I would expect that the test runners would be in that configuration (auditd isn't installed by default)
[13:40] <cascardo> but how does it forward that? by the kernel log/printk only?
[13:40] <tyhicks> yeah, forwarding is in-kernel
[13:42] <cascardo> okay, so I got that same TAP13 message as from the ADT log
[13:46] <cascardo> right after running the step_after_suspend_test
[13:46] <cascardo> and my VM can't respond anymore
[13:46] <cascardo> what is supposed to be waking it up, by the way?
[13:46] <cascardo> and nothing seemed to have changed on that test, either from our last kernel or on the testing repos
[13:47] <cascardo> more likely, we broke suspend on ppc
[13:47] <apw> or worse we made it do something, and the wake up does not work
[13:48] <apw> i have this odd feeling we don't run those in adt in some sense
[13:49] <cascardo> I could just test it with the older kernel and see how it behaves
[13:49] <apw> yeah, the problem is we modify the kernel self tests etc to turn them off, and that can stop working
[13:49] <apw> has done before
[13:49] <cascardo> just have to set it up all over again, cause this openstack is broken
[13:50] <cascardo> I looked it up, we didn't just turn it on again recently
[13:50] <cascardo> so, it should have been running before
[13:50] <cking> hrm, i'm confused, I was sure it was disabled for x86
[13:50] <apw> cking thinks we haven;t turned it back on
[13:50] <apw> well this is ppc which is breaking on suspend
[13:50] <cking> maybe I'm mistaken, a lot changes around here
[13:50] <apw> so i am suspicious we should not be running it there either
[13:50] <apw> as an unrelaible to wake up test
[13:51] <apw> in a VM environment
[13:51] <cking> it maybe that the RTC wakeup is broken in a VM, or it the RTC wakeup is too short, or a bunch of other reasons
[13:52] <cascardo> hum... it was commented out on 4.15.0-10.11
[13:52] <cascardo> and clone didn't work on 4.15.0-11.12
[13:53] <sforshee> from a previous log which did run to completion, I don't see that test being run
[13:54] <sforshee> https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-bionic/bionic/ppc64el/l/linux/20180221_002951_88802@/log.gz
[13:55] <cascardo> okay
[13:55] <cascardo> that file has changed
[13:55] <cascardo> and the sed line is not working anymore
[13:56] <cascardo> it's not a += anymore, it's a :=
[13:56] <sforshee> \o/
[13:56] <cascardo> I am fixing that on the testsuite, then will retrigger it
[13:57] <sforshee> awesome, thanks cascardo 
[13:57] <cascardo> gotta make sure I write something that will work on new and old kernels
[13:58] <cascardo> thank you for making me look past the audit backlog thing
[14:02] <tyhicks> nice!
[14:03] <cascardo> now, let's hope the only problem we see is the kallsyms one
[14:04] <juergh> apw, can you please accept the nomination for #1755817?
[14:05] <cascardo> juergh: approved
[14:10] <juergh> cascardo, thanks
[15:06] <xcyclist> Thank you juergh.  Is there a document with the taxonomy of the different build combinations, both setups, and product files?
[15:12]  * apw doubts we have anything that detailed ... the intent of packages is one thing and one thing only to build in the defined debian/ubuntu build environment
[17:11] <sforshee> cascardo: your ppc64el retry finished, the only failure is the expected kernel_symbols_missing
[17:12] <sforshee> I'll add a hint for that, then we should be in good shape to promote
[17:45] <sforshee> apw: the testing for our bionic kernel is all in order but it looks like we need to clear NBS
[17:49] <apw> sforshee, i think i did that already, about 15:30 my time
[17:49] <apw> but as it is 17:50 i am suspicious
[17:50] <sforshee> apw: the adt matrix still shows it is NBS
[17:50] <apw> sforshee, hmmm, should be resolved there soon, cirtainly britney is now showing it only held by the bug
[17:51] <sforshee> ack, ta
[18:40] <cascardo> sforshee: thanks!