[07:42] <SvenKieske> hey there, someone here familiar with spectre_v2 attacks/mitigations? if I understand this report correctly it is not possible currently to allow virtualized guests on ubuntu 20.04. to take advantage of the IBRS mitigation? https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1921754
[07:44] <SvenKieske> sorry, I meant 18.04
[07:44] <cpaelzer> ah, I just wondered as it is fix released for focal and later
[07:57] <cpaelzer> There are various ways to get newer virt stacks onto 18.04 like https://wiki.ubuntu.com/OpenStack/CloudArchive or https://launchpad.net/~canonical-server/+archive/ubuntu/server-backports - but at some point things start to show its age (and it isn't broken, just not able to use the speedup of the mitigation). And that also only on very recent chips as e.g. a bunch of -IBRS chips are even in bionic
[07:57] <cpaelzer> back in the day there was no reasonable backport as AMD also changed a lot of other structural things, maybe there is something nowadays and it would be good to re-evaluate
[07:59] <cpaelzer> I have not much time, but I see your bug update which is great to give this a new look
[07:59] <tomreyn> isn't the issue there just that there is no EPYC-Rome CPU model for 18.04 (and thus no need to provide Epyc-Rome-IBRS either)?
[08:00] <cpaelzer> yeah, that is part of the truth tomreyn. But the follow up would be how would I IBRS on an EPYC* then?
[08:00] <cpaelzer> I'd need to have a look ...
[08:03] <cpaelzer> upstreams 2.11.2 stable tree also has only the intel IBRS models (we have that) and EPYC-IBPB (mind the characters, not the same) we have that as well
[08:22] <tomreyn> Hmm, apparently CVE-2017-5715 is generally non-mitigated on 18.04 amd64 with HWE (5.4.0-104-generic #118~18.04.1-Ubuntu). "cat /sys/devices/system/cpu/vulnerabilities/spectre_v2" reports "Mitigation: LFENCE, IBPB: conditional, STIBP: disabled, RSB filling" (on this Ryzen 7 1800X), so it seems the kernel has not been compiled with a retpoline-aware compiler
[09:04] <apw> tomreyn, that would be supprising, i am relativly sure that bionic was current when spectre was first in the frame, and it got retpolines added.  i am fairly sure the kernel won't compile if retpoline is not available as it always builds those as alternatives.
[09:06] <tomreyn> apw: do you have any means to check this on your end, or is there a good test i could do?
[09:07] <tomreyn> i guess i could boot a current 18.04 installer iso
[09:08] <apw> tomreyn, a good question.  though i had the feeling that we don't expect retpoline turned on in the default case do we?  where you have ibrs it is assume good enough if you have user-bpf turned off.  but if you are more paranoid you can select ibrs+retpoline
[09:09] <apw> as no other gadgets are known but that does not mean they are not present.
[09:10] <tomreyn> i don't have IBRS either, though, just LFENCE
[09:11] <apw> yeah sorry, lfence+retpoline for amd
[09:17] <tomreyn> apw: a Debian 11.2 running Linux 5.10.0-11-amd64 #1 SMP Debian 5.10.92-1 (2022-01-18) x86_64 (on EPYC 7351P) *does* have "Mitigation: Full AMD retpoline, IBPB: conditional, STIBP: disabled, RSB filling"
[09:17] <tomreyn> so i was assuming ubuntu would default to this as well
[09:18] <tomreyn> newer kernel there, though, different cpu (though roughly the same generation)
[09:18] <apw> tomreyn, you can tell if it is availble by selecting spectre_v2=retpoline,lfence by my reading.
[09:22] <apw> tomreyn, the CVE you quote there is the original spectre issue, and that is mean to be mitigated by just lfence, right?  obviously time has moved on and the new issues exposed by CVE-2022-0001 change the equation for some people.
[09:24] <SvenKieske> tomreyn: sorry was hung up in some other kernel sec bugs.. I got: "cat /sys/devices/system/cpu/vulnerabilities/spectre_v2" reports "Mitigation: Full AMD retpoline, IBPB: conditional, IBRS_FW, STIBP: always-on, RSB filling" on a "AMD EPYC 7713P 64-Core Processor" with kernel: 5.4.0-97-generic
[09:27] <tomreyn> SvenKieske: which gcc was this built with, which ubuntu version is it?
[09:27] <apw> tomreyn, i should say just lfence, assuming you have appropriate cpu-firmware loaded.
[09:28] <tomreyn> apw: yes, the info CVE page is basically irrelevant by now for current HWE on 18.04
[09:28] <SvenKieske> apw|tomreyn: the performance impact is one thing; I'm a bit more worried about retpoline also not being as effective as IBRS? that's what I read from the kernel docs, but they are also not very clear: "For a full mitigation against BHB attacks, it’s recommended to use retpolines (or eIBRS combined with retpolines)."
[09:30] <tomreyn> in case it's of any use, here's spectre-meldown-checker output: https://termbin.com/22op , with --verbose: https://termbin.com/n1e2 , with --explain: https://termbin.com/buef6
[09:30] <tomreyn> i'll try booting with the kernel cmdline options apw mentioned
[09:30] <SvenKieske> tomreyn: we don't built kernels ourselves anymore (I hope I don't have to go back to those times..) that is linux-hwe-5.4 on ubuntu 18.04 from official ubuntu repositories afaik.
[09:30] <apw> SvenKieske, the CVE update at -104 carried some relaxing of option combinations and allowing manual override for those options.
[09:31] <tomreyn> SvenKieske: thanks.
[09:31] <apw> SvenKieske, specifically allowing selection of eibrs and retpoline for instance.
[09:31] <SvenKieske> apw: a longstanding issue I got with ubuntu btw is, that uname -r does not report ubuntu patchlevels.. always need to parse uname -a, very annoying.. this is running 5.4.0-97-generic #110~18.04
[09:32] <SvenKieske> we also go with stock kernel cmd line atm (except for our ceph clusters), well not really true, we have hugepages and stuff enabled, but no custom mitigation flags currently
[09:33] <apw> SvenKieske, that is an interaction with kmod which uses uname -r to find the modules.  that said now that we have to alway bump abi (for secureboot and signed modules) we could change the form of uname -r more readily.
[09:33] <SvenKieske> for completeness sake, here are the custom kernel options, but they shouldn't interfere here:  no_timer_check nofb nomodeset gfxpayload=text numa_balancing=disable transparent_hugepage=never default_hugepagesz=1GB hugepagesz=1GB
[09:34] <SvenKieske> apw: makes sense, many scripts use the short kernel versions provided via uname -r
[09:34] <SvenKieske> tomreyn: is that spectre checker part of stock ubuntu repos by chance? :)
[09:34] <apw> SvenKieske, as i understand the security position (and i am only on the periphery) it is believed that unless someone finds new gadgets that disabling uBPF avoids the need to enable more (expensive) mitigations in the general case; but this depends on your threat model.
[09:36] <SvenKieske> apw: wasn't there a recent paper that showed that not only unprivileged ebpf was affected? let me dig up a link (I'm also not an expert in C-Programming or Security, just an interested bystander for over 10 years ;) )...
[09:37] <apw> SvenKieske, i believe there are theortically others, i have not see a real one; yet.  but that i think is where the stance of not enabling but making-possible the additional mitigations.
[09:37] <SvenKieske> apw: there you go, a real one ;) https://www.vusec.net/projects/bhi-spectre-bhb/
[09:37] <apw> for this machine right here, none makes any sense.
[09:38] <apw> SvenKieske, that is the eBPF exploit
[09:39] <SvenKieske> to be fair: I didn't read the whole paper, just the abstract and the webpage for mere mortals like me ( I aborted my comp sci studies..)
[09:39] <tomreyn> SvenKieske: no, from their github
[09:39] <apw> SvenKieske, and which is why the security update ensures the default for use BPF is off.
[09:39] <SvenKieske> apw: from the website: "So, is disabling unprivileged eBPF sufficient?" "As hinted in the answer above, we believe this is not the case. However, we believe unprivileged eBPF does  tremendously facilitate speculative execution (and other) attacks."
[09:39] <tomreyn> apw: $ cat /proc/version /proc/cmdline /sys/devices/system/cpu/vulnerabilities/spectre_v2 | nc termbin.com 9999
[09:39] <tomreyn> https://termbin.com/gbhc
[09:40] <apw> SvenKieske, right, but that contention is "we believe" and that is reasonable, as their attack relies on having a cirtain shaped code gadget in kernel mode, to attack the branch buffers.
[09:41] <SvenKieske> apw: ok, granted; I'm afk for a bit..
[09:41] <apw> SvenKieske, and they are saying, they are not convinced there isn't other ways to make one, and they are likely right, but, none is currently known.  so one makes a trade off between actual and possible risk.
[09:43] <apw> SvenKieske, which is why it really comes down to your threat model. on my laptop none of these mitigations make any sense at all; and the 10% or whatever performance cost is not warrentd.  if you run potentially hostile payloads (such as occurs in a cloud) of course the cost is necessary.
[09:45] <tomreyn> apw: you mentioned "assuming you have appropriate cpu-firmware loaded" - i believe this to be the case based on spectre-meltdown-checker v0.44-18-g16f2160 (git head) output:   * CPU microcode is the latest known available version:  YES  (latest version is 0x8001138 dated 2019/02/04 according to builtin firmwares DB v220+i20220208)
[09:45] <apw> tomreyn, right, but for example in your debian test, it might have different loaded.
[09:46] <tomreyn> it loaded with debian defaults there, which could differ from ubuntu defaults
[09:53] <tomreyn> apw: FWIW, jjohansen in -security pointed out that: "Until this last week lfence/jump was amd's recommended mitigation. You could force generic retpoline using the kernel boot parameter spectre_v2=retpoline,generic - see https://www.amd.com/en/corporate/product-security/bulletin/amd-sb-1036
[09:53] <tomreyn> "
[09:54] <tomreyn> however, this adivory does not list Ryzen 1000 series Desktop CPUs (such as this one). which could mean those are not affected, or are not being handled.
[09:54] <tomreyn> maybe those are just EOL.
[09:55] <apw> interesting they say just retpolines, and not retpoline,lfence which is what was known as retpoline,amd ...
[09:55] <tomreyn> "retpolines" (plural 's')?
[09:55] <apw> tomreyn, indeed.
[09:57] <SvenKieske> apw: yeah I should've mentioned my threadmodel (I agree it's important): running a basically public cloud, so untrusted kvm guests in openstack
[09:57] <SvenKieske> always mixing up threads and threats :D
[09:57] <tomreyn> apw: not for me, unless you don't mean it verbatim: curl -s https://www.amd.com/en/corporate/product-security/bulletin/amd-sb-1036 | grep -i retpolines
[10:00] <apw> tomreyn, they say retpoline,generic which is retpolines only, not their own ,amd variant.  this seems odd, given it exists for a reason.
[10:01] <tomreyn> ok, got you now.
[10:01] <SvenKieske> I can run the spectre checker on the above amd epyc, if it helps?
[10:02] <tomreyn> SvenKieske: we've already seen that yours has retpoline enabled, so i don't think it helps us
[10:02] <tomreyn> "Full AMD retpoline" specifically
[10:03] <SvenKieske> okay
[10:04] <SvenKieske> the bottomline for me is: I want to make sure I can run untrusted guests, to the best current knowledge, safely. the cherry on top would be if it would be more performant, does anybody has seen performance comparisons between retpoline and IBRS?
[10:05] <tomreyn> i think phoronix had some for amd
[10:05] <SvenKieske> one thing I don't get is, why I have "IBRS_FW" but not "IBRS"; I mean, okay, I know why I don't got IBRS, but how can IBRS_FW work without IBRS? 
[10:07] <SvenKieske> tomreyn: right, I read it at the time, if anyone wants to reread: https://www.phoronix.com/scan.php?page=article&item=3-years-specmelt&num=1
[10:08] <SvenKieske> there is also a new performance comparison, but only intel so far for the new mitigations against the above mentioned vusec paper: https://www.phoronix.com/scan.php?page=article&item=spectre-bhi-retpoline&num=2
[10:15] <tomreyn> apw: so i rebooted again, this time with spectre_v2=retpoline,generic , and got "Mitigation: Retpolines, IBPB: conditional, STIBP: disabled, RSB filling"
[10:16] <tomreyn> which i guess is better than LFENCE, worse than "Full AMD retpoline", if i'm getting this right
[11:54] <apw> tomreyn, at least you know it has retpolines in it; which is what we expected.
[12:02] <tomreyn> now, with the non default cmdline, it does, yes. :)
[12:03] <tomreyn> thanks for your help in figuring it out, apw.
[12:04] <apw> tomreyn, np.
[13:49] <SvenKieske> to update you guys: we in fact are already using the ubuntu cloud archive to get updated packages, but the package list was frozen due to internal organzational challenges, so the IBRS problem will be resolved once we can update our docker images, thanks!