vieirais there a fix for meltdown comming out today for linux-hwe-16.04?00:28
TJ-vieira: no patches for v4.10, you need hwe-16.04-edge (4.13)00:29
vieiraOk, I just installed linux-virtual-hwe-16.04-edge and rebooted00:29
TJ-vieira: 4.10 kernel from 17.04 reaches end of life on Jan 13th, so wasn't worth the effort to backport patches00:30
vieirabut have 4.13.0-21-generic00:30
vieiraI think it should be 4.13.0-25.29?00:30
TJ-vieira: waiting for it to appear in the -security archive00:30
vieiraahh ok! Thanks :)00:31
keithzgOver in #ubuntu-server it was mentioned that I should mention this here: unable to boot past the RAM image loading on 4.4.0-108.131, meanwhile 4.4.0-104.127 continues to work fine. CPU is an Intel i5-4670K, motherboard is an ASUS Z87-A.01:13
TJ-thanks keithzg 01:16
TJ-possibly related Ask Ubuntu, with stack-trace photo: https://askubuntu.com/questions/994067/kernel-panic-after-spectre-meltdown-update-16-0401:17
bjfkeithzg, are you still around?01:31
keithzgbjf: Yup01:31
Odd_Blokebjf: He's been active in... there he is.01:31
bjfkeithzg, ok, may have a new kernel for you to try "soon"01:32
TJ-bjf: we're hoping keithzg can capture some log messages in the meantime ... got your camera handy, keithzg ?01:32
keithzgYup, camera's ready although it's not much to see currently!01:33
TJ-hehehe, common problem01:33
keithzgWhat little I have so far is now at https://photos.app.goo.gl/nqKrJHknd11NuCeO201:37
TJ-one last test, try adding "nopti" on that kernel's command line01:39
keithzgTJ-: I haven't tried adding "debug" yet, should I try both, or just "debug" first, or?01:39
TJ-keithzg: both, why not!01:41
TJ-keithzg: it looks like 'debug' on it's own won't capture anything; looks like the kernel's initial setup is failing01:41
keithzgTJ-: hehe, shall do01:42
bjfkeithzg, sorry, i won't have a new kernel for you to try. please file a bug and them come back here and tell me what the bug id is01:47
keithzgTJ-: No dice, I just get the loading lines and nothing more.01:47
keithzgbjf: Fair enough, shall do01:47
TJ-keithzg: that's with 'nopti' ?01:48
keithzgTJ-: Yeah.01:48
TJ-keithzg: hmmmph, that IS strange01:48
keithzgWell, that *and* debug01:48
keithzgIt's conceivable I don't know what I'm doing with editing GRUB entries on the fly01:49
TJ-keithzg: which suggests some of the other x86/mm sub-system patches not directly related to the pti patch-set may be implicated01:49
TJ-keithzg: you're editing the kernel command line via GRUB? pressing E on a highlighted entry, navigate to the line starting "linux ...", add the options towards the end of the line, e.g. " debug nopti" then IMPORTANTLY press Ctrl+X too boot with that change - don't return to the menu to start the entry else changes will be lost01:50
keithzgTJ-: For whatever reason Ctrl-x just writes x to the editor, so I went with F10 as the alternate option, yeah. But now I'm wondering if there were quotation marks around the whole thing and I didn't notice and put my options outside of them? The much smaller window you get when running GRUB in a non-graphical mode makes things harder to visually parse, heh. I'm just going to try adding nopti to GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub, regenerate 01:52
keithzggrub, and then try another reboot.01:52
TJ-keithzg: no quotation marks required when editing the GRUB lines01:53
keithzgTJ-: Yeah I was just wondering in retrospect. And yeah, I verified that it's definitely trying to boot with that option, and definitely still hanging there. Bug report time!01:56
bjfkeithzg, ok, i will shortly have something for you to try... i'm waiting for LP to complete copying something01:57
bjfkeithzg, https://launchpad.net/~canonical-kernel-team/+archive/ubuntu/pti01:59
bjfkeithzg, in that ppa there will be "real soon now" a newer 4.4 kernel01:59
bjfkeithzg, it's telling me the kernel is there now02:00
* bjf is stepping away for a bit02:02
keithzgWell it's a banner day for errors for me, I just tried opening a bug on Launchpad and got Timeout Error ((Error ID: OOPS-809eb4fff20040fcf023ecf9bda0f5d3))02:03
keithzgbjf: Only newer package I'm seeing is linux-libc-dev?02:04
TJ-keithzg: maybe the LP server just got it's PTI patches :p02:05
keithzgTJ-: hah!02:05
bjfkeithzg, 4.4.0-109.132 .. you probably need to pull down the .debs and dpkg -i them02:06
TJ-see https://launchpad.net/~canonical-kernel-team/+archive/ubuntu/pti/+sourcepub/8710767/+listing-archive-extra 4.4.0-109-132 02:06
TJ-apt-cache policy shows it Candidate: 4.4.0-109.13202:07
keithzgHmm, apt-cache policy is still just showing me "" as the latest for linux-image-generic. Manual .deb time then I guess!02:09
TJ-keithzg: did you "apt update" ?02:09
TJ-keithzg: presumably you added the ppa via apt-add-repository ?02:09
keithzgTJ-: Yes and yes02:09
TJ-keithzg: strange; I did the update and it shows 02:10
TJ-keithzg: try "apt-cache policy linux-image-4.4.0-109-lowlatency"02:10
keithzgTJ-: Yeah, I see it there then, as I do too with `apt-cache policy linux-image-4.4.0-109-generic`. I guess it makes sense to just apt install *that*.02:11
keithzgAlthough I might need linux-signed-* correspondingly?02:12
TJ-keithzg: if using Secure Boot, yes.02:13
keithzgTJ-: I remain unsure, heh, been too long since I set up this server and I don't remember if I disabled that or not. Running `apt install linux-signed-image-4.4.0-109-generic linux-image-extra-4.4.0-109-generic linux-headers-4.4.0-109-generic` to be sure.02:15
keithzgInstalled, lets give this a shot.02:20
keithzgBooted with 4.4.0-109-generic! That's with "nopti" still on the kernel command line, I'm going to try removing that and seeing if it's still fine.02:25
keithzgExcellent, that seems to have done the trick. Many thanks, bjf and TJ- :)02:31
TJ-keithzg: so do we know what was wrong? 02:32
keithzgTJ-: I certainly have no idea, all I know is that the newer kernel packages from the PPA have resulted in everything appearing to be fine now on my system.02:33
TJ-keithzg: Of course, I'd forgotten you've just updated to the very new build! I'm getting confused with the regression I'm investigating here02:34
keithzgTJ-: It's a busy time :)02:35
bjfkeithzg, thanks!02:38
keithzgbjf: :)02:50
keithzgWorth mentioning, I ran into this again on another server, also using an ASUS motherboard (different, however: P8Z77-V LX in this case) and an older Intel CPU (i5-3550).02:52
keithzgGoing to check in a bit if the newer PPA packages fix this one too (it does also reboot fine with the -104 kernel, so odds seem good). First I'm going to pick up a pizza, though, since this may be a long night ;)02:53
bjfkeithzg, we are rushing out the kernel you tested for us03:23
keithzgbjf: Good to hear :) Hopefully there aren't any more issues lurking!05:03
f_gTJ-, bjf: that keithzg issue was one I pointed out twice in here already (including the fix), apw said he'll look at it on Monday already :-/ seems like I should have filed an LP bug earlier? I was under the impression that nobody would look at those atm while being busy backporting the huge patch sets.. going forward, I assume LP is the way to go again to report improvements/regressions?07:16
apwf_g: is that the _ds panic?07:54
f_g(I know it's already fixed in the latest 4.4)07:55
apwif so reporting that here was the right thing, iirc you were going to test if that fix was the fix, and we had people scrambling to test too but positive reports didn't make it in time to hit the first updates07:56
apwreporting it here was worthwhile07:56
f_gokay. you're right, I did not explicitly state that it actually fixed the issue, I just pinged a second time about it not yet being applied in the pti branch. I think I just assumed that it would get in in any case since it was an obviously incomplete backport - could have stated it more clearly.07:59
f_ggoing ahead, should I file bugs at LP in general, or just for already released kernels? are there any plans for Spectre / IBRS / RETPOLINE / .. already (or ways to contribute to that?)08:01
f_g(FWIW, we haven't received any other negative feedback, so things are looking good atm)08:02
apwjust bad luck on timing, we were juggling all the series at once08:03
apwit will be in the next update, and that will soon, we are in free release mode08:03
apwSpectre is in progress for sure08:04
apwf_g, oh indeed it looks like there is a build in proposed with that fix right no08:18
apwf_g, and i hear it was already approved for -updates, so it should be there 'soon'08:19
f_gyes :) we only had to juggle two kernel releases here, and still missed some stuff in the first iteration. that's just how it is ;) gotta catch up on LKML now - if there is anything that can be done from over here regarding Spectre backporting/testing/.. just ping08:21
* ogra didnt notice this channel became +r11:52
ograit seems the just released kernel doesnt boot on my desktop ... 11:53
ograi'm greeted with an oops right after grub loads it11:53
f_gogra: which kernel / distro?11:54
ograheh, ubuntu 16.0411:54
f_g4.4.0-109 likely fixes it, if the oops looks like https://imgur.com/a/vVbR011:55
ograi had to go back to 4.4.0-104-generic11:55
f_gsee LP#174193411:55
ograah, i'm just behind one version then11:56
* ogra upgrades11:56
acheronukapw et al. thanks for the hard work11:59
* alkisg also had an issue with 4.4.0-108 today... 4.4.0-104 booted, but -108 displayed a black screen without even loading the initramfs12:49
alkisgI solved it by installing the -hwe stack, maybe I should have filed a bug report instead...12:50
apwalkisg, if you get a chance try -10912:55
alkisgapw: thanks, if they call me again from that school I will do so12:55
threshany known issues with linux-image-generic-hwe-16.04 and kernel panics?13:40
apwthresh: best to just tell us the symptoms you are seeing13:46
threshapw, as it's a hetzner server, I have limited visibility (and it seems IP KVM has issues, too).  I'm trying to boot it into rescue now to maybe have more info.13:52
threshhere's what I got from the support: "But it seems as the server shows no video output anymore (or responds to key strokes) once the boot process is nearly finished and the login promt should show up"13:52
threshit's an AMD Ryzen system, too.13:52
threshI'm sorry I don't have a lot of info atm, still waiting for it to boot into rescue system.13:53
threshAnd it turns out I dont have a netconsole set up on that server :|13:53
threshthe kernel version I've got installed is 4.13.0-26.29~16.04.213:55
threshI was able to boot into the rescue system, and there are no logs, sadly.14:38
threshchanged grub default back to the 4.8.0-56 and system booted up just fine.  I understand that that does not help.14:45
threshok, tried booting back to 4.13.0-26.29~16.04.2, after some kernel initialization it freezes, and goes to "no signal" on KVM again.  Sadly, it seems to early for netconsole to kick in.15:04
threshand no panic stack trace on the screen either.15:05
metaphysicianOn 16.04, xtables-addons dkms fails to build with the new 4.13 kernel.16:09
threshI do have another AMD system locally, although it's AMD Turion(tm) II, not Ryzen.  I'm going to test the upgrades tomorrow.16:10
thresh(with physical access, too)16:10
sbeattiethresh: given the 'had to go back to a 4.8 kernel', I'm guessing you hadn't run any of the 4.10 linux-hwe kernels? Do any of the 4.10 linux-hwe kernels work for you? Also you could manually install the linux-hwe-edge 4.13.0-21.24~16.04.1 to try to narrow it down to just being the meltdown/kpti patchset that's breaking things for your hardware.16:24
threshsbeattie, yeah, I've been using 4.8.0 since, well, it ran fine for me for quite some time.16:25
threshI'm going to try those kernels now.16:25
threshsbeattie, linux-image-4.13.0-21-generic=4.13.0-21.24~16.04.1 booted up fine (except did not bring up the network interfaces).16:36
ctennissorry for the question if it's already been addressed, but the MeltdownAndSpectre wiki page says the linux-aws kernel version for Xenial is 4.4.0-1047.56 whereas the linked 3522-1 says it is  Any clarity on which is correct?16:38
ctennisUSN-3522-1 that is16:38
sbeattiethresh: likely need linux-image-extra-4.13.0-21-generic=4.13.0-21.24~16.04.1 too16:41
threshyeah, r816916:42
threshbut, I guess, that kinda sorta means that kpti patches are the problem16:42
sbeattiectennis: is the linux-meta version, which update-manager uses to know to pull in the specific linux-image 4.4.0-1047.56 package.16:43
ctennisthat's seemingly really confusing16:44
sbeattie(more correctly, linux-meta-aws version, and linux-image-4.4.0-1047-aws)16:45
sbeattieconfusing> a bit yes, but it's what allows you to have multiple kernel images installed so that you can rollback if a specific version fails to work for you.16:46
ctennisright, it's just the different #s between the two pages16:47
ctennisI had 1047.49 installed, but it did NOT have 4.4.0-1048.57 with it16:48
ctennis4.4.0.1047.49 I mean16:48
ctennisbut just 30 minutes ago it became available, so now i have and the corresponding 4.4.0-1048.5716:48
ctennisanyway, looks like its sorted now, and I was just in a small window of time between updates16:50
sbeattiectennis: uhhhh, it should not be possible to have linux-image-aws installed without also having linux-image-4.4.0-1047-aws (4.4.0-1048.57) installed, as the former has a direct dependency on the latter.16:54
sbeattiealso, to be explicit, both are referred to in USN-3522-116:55
ctennisThis is what I had about 30 minutes ago: linux-aws is already the newest version (
ctennisThe Meldown page says just says "4.4.0-1047.56"16:57
ctennisso I recognize maybe that's a different package, but at first glance it looks like...I'm not up to date enough16:57
ctennisanyway, thanks for the response, and don't worry about me, I just wanted to get clarity.  I still think it's confusing, but it sounds like you're all on top of it.17:01
threshwell, apparently, linux-image-extra-4.13.0-21-generic=4.13.0-21.24~16.04.1 failed to boot.17:03
threshcould be a r8169 which is crashes it, then, too..17:05
jsalisburythresh, would it be possible for you to open a launchpad bug?  This will allow allow all this info to be kept together and serachable by others.17:14
threshjsalisbury, sure - I'm just trying to single out the exact problem which causes the issue.17:14
jsalisburythresh, thanks!  I can work with you on the problem.  Can you post the bug number once you have it?17:15
threshyup, but I need to go home soon - will hopefully be able to work on it after a couple of hours.17:16
jsalisburythresh, ok, thanks17:17
threshthe fact that it's on hetzner does not help a lot to debug, too :-(17:17
=== JanC is now known as Guest37549
=== JanC_ is now known as JanC
ricotzklebers, hi, I noticed that the kernel metapackage for raspi2 depends on linux-firmware which seems not right and linux-firmware-raspi2 would be more fitting17:54
chilukis linux-firmware a metapackage that points a linux-firmware-raspi2?19:52

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!