[04:30] <tjaalton> apw: the one on xenial is skl/kbl/bxt
[08:56] <apw> tjaalton, hmm i don't think i know what those three all are; skylate, erm
[09:04] <smb> If one would not know better one would guess keyboard failure
[09:13] <tjaalton> apw: heh, kabylake and broxton (aka apollo lake)
[09:26] <apw> tjaalton, those i have not even heard of
[09:27] <apw> tjaalton, ok so i am seeing lockups on resume in what feels like graphics and some visual corruption occasionally in icons in firefox, on broadwell
[09:27] <apw> tjaalton, with the xenial kernel -12 at least, i just upgraded to -13 and have not been on it long enough to know if it is sad too
[09:29] <tjaalton> apw: ok, is the corruption right from the start, or only after a resume or such? check dmesg if so
[09:30] <tjaalton> there could be a gpu hang and recovery isn't complete
[09:30] <apw> tjaalton, you know how these bugs are you are never quite sure till they happen a few times, i currently feel, only after a resume
[09:30] <tjaalton> right
[09:30] <apw> but i'd not want to swear to it in court.  i will check dmesg if i see such corruption again on -13
[09:32] <tjaalton> cool
[09:52]  * apw wonders if xnox has tested his second debug kernel yet
[11:38] <xnox> apw, not yet
[11:45] <xnox> apw, i did not hit any of those printk... http://paste.ubuntu.com/15400970/ unless i'm not collecting them at all.
[11:48] <apw> xnox, literally none of them, which is perplexing
[11:48] <apw> xnox, that is the generic kernel not an apw build
[11:48] <apw> [    0.008770] Linux version 4.4.0-13-generic (buildd@z13-012) (gcc version 5.3.1 20160225 (Ubuntu 5.3.1-10ubuntu2) ) #29-Ubuntu SMP Fri Mar 11 19:30:41 UTC 2016 (Ubuntu 4.4.0-13.29-generic 4.4.5)
[11:48] <xnox> boom
[12:01] <xnox> apw, http://paste.ubuntu.com/15400999/
[12:02] <xnox> apw, btw the .debs are suspiciously the same as I tested yesterday, you didn't bump version number timestamp at all?
[12:02] <apw> xnox, yes, they would have had new times
[12:03] <xnox> it downloaded for me as linux-image-4.4.0-14-generic_4.4.0-14.30lp1557250v201603151737_s390x.deb.1 hence i don't think so.
[12:03] <xnox> http://people.canonical.com/~apw/lp1557250-xenial/ has timestamps 17:41 from yesterday, which is from before i left, no?!
[12:04] <apw> xnox, i mean i cannot make a build with the same time, so if it is the same time, that is not a new build, its is ... erm ... wrong
[12:04] <apw> xnox, ok something is screwey, the only safe thing for me to do is rebuild it
[12:05] <apw> xnox, oh i think i may have built a wrong branch and pushed binaries but useless ones for another bug, sigh, be 10m
[12:09] <xnox> more coffee
[12:10] <apw> xnox, ok some actually new binaries pushing, sigh, sorry about the confusion there, should be up in ... now
[12:11] <xnox> ack
[12:25] <xnox> apw, http://paste.ubuntu.com/15401110/
[12:25] <xnox> [    0.734265] Loading compiled-in X.509 certificates
[12:25] <xnox> [    0.734266] x509_cert_parse called
[12:25] <xnox> [    0.735100] x509_check_signature -> -448754944
[12:25] <xnox> [    0.735101] preparse -> ret=-74
[12:29] <apw> xnox, ta ...
[12:38] <apw> xnox, ... and again
[13:19] <xnox> apw, 
[13:19] <xnox> [    0.715208] Loading compiled-in X.509 certificates
[13:19] <xnox> [    0.715209] x509_cert_parse called
[13:19] <xnox> [    0.715230] APW: crypto_shash_finup 0
[13:19] <xnox> [    0.715231] APW: RSA_verify_signature called
[13:19] <xnox> [    0.716088] APW: public_key_verify_signature ret=-74
[13:19] <xnox> [    0.716089] x509_check_signature -> -74
[13:19] <xnox> [    0.716090] preparse -> ret=-74
[13:19] <xnox> http://paste.ubuntu.com/15401110/
[13:20] <xnox> wrong paste.
[13:20] <xnox> i mean
[13:20] <xnox> http://paste.ubuntu.com/15401255/
[13:20] <xnox> (yeah thats the 1234 build)
[14:17] <apw> xnox, ok making progress, its tricky to add all teh debug we need in one run
[14:17] <apw> xnox, as we jump through callbacks where there are 3-4 options
[14:17] <apw> RSA_verify_signature was one of those
[15:24] <tyrog> Hi guys, do you think Kernel 4.5 is still coming as default for Ubuntu 16.04?
[15:24] <ogra_> no
[15:25] <tyrog> ogra_: I know for now it is on Kernel 4.4, can that decision change until Final?
[15:25] <ogra_> nope ... 4.4 is an LTS kernel ... 
[15:25] <tyrog> ogra_: ok, tnx :)
[15:26] <ogra_> there will be hwe kernels later ...  like you can alwas use newer kernels on ubuntu LTS releases... 
[15:26] <tyrog> ogra_: 3.13 on 14.04 wasn't a LTS kernel too...
[15:27] <ogra_> yeah, that means that the ubuntu kernel team needs to maintain this version for 5 years themselves
[15:28] <ogra_> if you can have a kernel that is upstream maintained instead it means you free developers for other work
[15:28] <tyrog> I see
[15:29] <tyrog> yes, then probably the right decision is to keep 4.4 :D
[15:29] <ogra_> it will just make the 4.10 (or whatever will be current at 16.10 release) hwe kernel better ;)
[15:39] <apw> tyrog, yeah waht ogra said
[15:39] <tyrog> apw: Fine as 4.4 should be very stable when 16.04 Final is released ;)
[15:40] <apw> tyrog, yeah we try and hold back a bit for these lts' than we would in an interim one, and that upstream is doing stable for 4.4 is a major boon
[15:47] <apw> xnox, another test kernel for you, i hope
[15:50] <xnox> tyrog, if you want things in 16.04 kernel, make those things to be backported to 4.4. into stable tree, or other trees. Backports of bits & pieces of 4.5 drivers are in our 4.4 kernel...
[15:51] <xnox> or e.g. bugfixes should be taken into stable tree.
[16:35] <xnox> apw, http://paste.ubuntu.com/15402573/
[16:35] <xnox> 1424 kernel
[17:22] <apw> xnox, another one uploading ...
[18:04] <xnox> apw, off by one?! http://paste.ubuntu.com/15403281/
[18:04] <apw> xnox, hrm ... odd ... indeed
[18:05] <xnox> there is extra garbage on non-s390x? because it's using.... weird encoding?!
[18:05] <xnox> anyway, i should stop guessing and go back in fixing something useful.
[18:10] <apw> i'll poke seom more indeed
[18:58] <apw> xnox, antoher kernel pushing, this one might work or it might explodes, that will be interesting
[19:34] <xnox> apw, thank you for powerpc kernel?!
[19:34] <apw> xnox, the last s390x is the same one, that is for someone else :)
[19:34] <apw> is the same contents as the ppx
[19:34] <apw> ppc
[19:34] <xnox> ack.
[19:34] <xnox> will test 1848 kernel
[19:34] <apw> ta
[19:37] <xnox> fail
[19:37] <xnox> http://paste.ubuntu.com/15403933/
[19:37] <xnox> missing prefix, hence off-by-one?
[19:37] <xnox> did we check by the way that e.g. powerpc loads the cert fine?
[19:38] <xnox> i guess we don't have powerpc (big endian) machines to test the kernels on, do we?
[19:40] <apw> yes, a powerpc BE also fails in the same way
[19:40] <apw> bah this is an endian issue, and i need to think about how to fix it ... prolly will punt it at anton :)
[19:42] <xnox> because the cert is read in host-endian order, rather than little-endian order.
[19:42] <xnox> we should be able to find upstream regression too.
[19:42] <xnox> cause we had powerpc forever.
[19:43] <apw> xnox, its not quite like that, its handled in an endian sensible way, its not handling the packing right it seems
[19:43] <apw> debugging it is tricky
[19:43] <xnox> https://bugs.launchpad.net/ubuntu-mate/+bug/1541151
[19:46] <apw> why would that not be able to boot
[19:46] <apw> that does not prevent booting
[19:46] <xnox> also
[19:46] <xnox> https://lists.debian.org/debian-powerpc/2016/02/msg00010.html
[19:47] <apw> i don't believe the mate issue is actually related, but ...
[19:47] <xnox> it's a data point that in alpha the bug was present.
[19:47] <apw> i don't believe that is the reason the system does not boot
[19:47] <apw> but it is clearly there for a long time
[19:48] <xnox> should i open kernel.org bug?
[19:48] <apw> which we would never notice because it is benign
[19:48] <apw> xnox, for the moment no, i am working on it
[19:48] <xnox> ok.
[19:56] <infinity> apw: antonb and benh should be around (and maybe even awake shortly) ... mpe might already be, I can never remember where he lives.
[19:56] <apw> ack
[20:06] <apw> xnox, another one pushing now for testing if you are about
[20:09] <xnox> apw, patch looks plausible.
[20:11] <apw> xnox, yeah, plausible, but i bet wrong again :)
[20:25] <apw> xnox, fun ?
[21:09] <xnox> apw, http://paste.ubuntu.com/15404387/
[21:09] <xnox> not sure about the size length reversion.
[21:10] <apw> xnox, so made no differance at all ?
[21:11] <xnox> apw, well we don't know. you have reverted the size compare, so we don't know if a later [1] check succeeds or not.
[21:11] <xnox> i'd like a kernel without the 0008 patch
[21:12] <apw> ack
[21:17] <apw> xnox, uploading now ...
[21:17] <apw> ... and done
[21:27] <xnox> apw, so your fix didn't help as per previous stuff. it may have helped with something/somewhere else.
[21:27] <xnox> http://paste.ubuntu.com/15404438/
[21:28] <apw> hmmm
[21:29] <apw> xnox, ok i need to dump the thing en-toto
[21:37] <xnox> apw, in the mean time, i win - LD [M]  /home/ubuntu/linux/debian/build/build-generic/zfs/module/zfs/zfs.ko
[21:38] <apw> xnox, heheh, if you have it building, we might have some tests somewhere
[21:45] <xnox> apw, can you test-build s390x kernel proper for me with https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1519814 ?
[21:45] <apw> xnox, yep in a few
[21:46] <xnox> i only got it to do ./debian/rules build to complete. And i have tried to fake the patch to mimic other commits you people do in a strange way.
[21:54] <apw> ok
[21:54] <apw> as it is zfs i'llneed to upload it to a PPA
[21:59] <xnox> yeap.
[22:00] <xnox> i bet abi checker or some such will tell me to shove my zfs back were it was
[22:02] <apw> xnox, likely
[22:04] <apw> xnox, ok pushing a possible fix found for the signature problem ifyou could test
[22:04] <apw> and done
[22:06] <xnox> i just don't want to even know anything about limbs in the kernel
[22:09] <xnox> [    0.678605] Loaded X.509 cert 'Build time autogenerated kernel key: 8d0653ca20551eb8f7822151a14011cc64a15ed4'
[22:09] <xnox> we are good.
[22:09] <xnox> http://paste.ubuntu.com/15404698/
[22:09] <xnox> apw, ^
[22:09] <infinity> \o/
[22:10] <apw> xnox, sweet, i'll apply that then for the next upload
[22:10] <apw> xnox, and i'll upload your zfs to a ppa
[22:11] <xnox> apw, cooling having that 509 and zfs fixed would be nice for the beta next week.
[22:12] <xnox> apw, which / where is the ppa upload?
[22:12] <xnox> or shall i call it a night
[22:18] <apw> xnox, i am uploading it now, will send you a link
[22:23] <apw> xnox, https://launchpad.net/~canonical-kernel-team/+archive/ubuntu/bootstrap/+packages
[22:24] <xnox> tah
[22:24] <apw> should be about 30m
[22:28] <apw> xnox, ok that sig fix now pushed to master-next for the upload, i'll wait on your testing for the zfs one
[22:52] <apw> xnox, ok its uploading binaries now
[22:58] <apw> xnox, well the linux-image has the .ko's you would want to find in there ...
[22:58] <apw> xnox, let me know if it explodes in your face or not
[23:00] <xnox> why do we have zfs-dkms and zfs in the kernel?
[23:00] <xnox> shouldn't we retire zfs-dkms package from zfsutils package?
[23:03] <xnox> apw, the module loads
[23:04] <apw> xnox, we make the kernel modules from the dkms package
[23:04] <xnox> huh?!
[23:04] <apw> and the kernel provides that package, so we can drop it from the kernel
[23:05]  * xnox is confused.... anyway.
[23:05] <xnox> the module loads.
[23:05] <apw> xnox, we need to fix this in the -dkms package to do this right, and update the kernel from it
[23:06] <apw> but that is all deatails, if it works that would be interesting to us
[23:06] <xnox> ok.
[23:06] <xnox> well, i have upstream pull request open too.
[23:06] <apw> yeah so that is good
[23:06] <xnox> Package zfsutils-linux is not available,
[23:06] <xnox> right.
[23:07] <apw> yes, that will be a problem
[23:08] <xnox> apw, do kernel people manage zfs-linux or may i upload it?
[23:09] <apw> xnox, cking nominally looks after the delta to debian
[23:09] <apw> xnox, bug i don't see any problem uploading a simple delta like enabling s390x for it
[23:10] <xnox> why do we only compile on 64-bit arches?
[23:10]  * xnox thought upstream totally supports 32bit arches....
[23:14] <apw> xnox, mostly for support coverage, if you are using zfs it is unlikely to be 32bit
[23:15] <xnox> apw, well, the question did come up as to why it's not available on 32bit.
[23:15] <xnox> uploaded into https://launchpad.net/~xnox/+archive/ubuntu/nonvirt/+packages
[23:15] <xnox> with arch:any, to see what happens.
[23:15] <xnox> dkms module is arch:all, so no reason not to attempt to compile the user space tools.
[23:16] <xnox> plus e.g. i hope that 32bit tools, can be used on 64bit kernel, for example.
[23:16] <apw> yep
[23:16] <apw> lets hope that
[23:20] <apw> xnox, kaboom
[23:21] <xnox> apw, so..... this is not module building is it?
[23:22] <xnox> but tools exploading?!
[23:22] <apw> yep the tools going blammo, something about VTOCs
[23:23] <xnox> oh i think i am meant to either define _SUNOS_VTOC_8 or _SUNOS_VTOC_16