SMTDOC 0

Add a document describing when and how to disable L1TF for KVM. The goal is to avoid disabling as much as possible to keep the performance benefits from SMT.

The warning message from KVM when SMT is on can point to this document.

---

An open item is still how to handle interrupts for the "confined cores" case. Right now it describes the sledge hammer of changing the affinity of all interrupts, but that's likely not needed in all cases. I'll send another email on this.

---
 Documentation/virtual/kvm/l1tf-mitigation.txt | 68 +++++++++++++++++++++++++++
 1 file changed, 68 insertions(+)
 create mode 100644 Documentation/virtual/kvm/l1tf-mitigation.txt

new file mode 100644
index 000000000000..876a28499973
@@ -0,0 +1,68 @@
+L1TF mitigation strategies in KVM
+
+On some Intel CPUs a side channel attack may make it possible to read
+arbitrary data in the L1D cache of the CPU core in the guest, potentially allowing
+to read data from outside the VM.
+
+The Linux kernel automatically detects the conditions for this problem
+and reports the status in /sys/devices/system/cpu/vulnerabilities/l1tf.
+
+With HyperThreading (SMT) enabled the L1D cache of a core is shared between the two
+CPU threads in a core.
+
+KVM mitigates this problems transparently whe SMT is disabled. With SMT
+enabled additional actions may be needed.  This document describes
+these additional actions.
+
+- When the system does not support HyperThreading no further
+effort is needed.
+
+This can be determined with
+
+% lscpu | grep Thread
+
+If the reported Thread number is 1 then HyperThreading is not
+available.
+
+- When the guest is not running any untrusted code then
+no further action is needed.
+
+- When the guest kernel is controlled by the host owner the only
+mitigation needed is to update the guest kernel to a kernel version
+with L1TF page table mitigation integrated.
+
+This will prevent any untrusted user programs running in the guest attacking
+the host, other guests, or other processes running inside the guest.
+
+When this is done no further action is needed.
+
+- When the guest is already confined to an dedicated core that
+does not run any host processes or other guests on either of its threads
+no further action is needed.
+
+The core could still in some cases process interrupts.
+
+The interrupts can be redirected to other cores using the
+/proc/irq/NUMBER/smp_affinity interface.
+
+XXX discuss interrupt security here
+
+- When the guest is not confined it can be confined to dedicated cores
+through exclusive cpusets. This does not necessarily need to be dedicated
+cores for each guest, but could be also dedicated cores for a group
+of VMs controlled by the same entity.
+
+For more details on cpusets please see Documentation/cgroup-v1/cpusets.txt
+
+- If this all fails SMT can be disabled on all cores
+
+SMT improves performance so it should be only disabled when absolutely
+needed, and reenabled after the untrusted guest is finished.
+
+To disable SMT:
+
+echo off > /sys/devices/system/cpu/smt/control
+
+To reenable after running untrusted guest
+
+echo on > /sys/devices/system/cpu/smt/control

2.14.4

What?!? Of course L1 flushing in KVM be always enabled. Why would you ever want to disable it?

AFAIK it doesn't make much performance difference anyways.

I thought novirt just means keep SMT on.

Jiri?

-Andi

Well, you don't want the flushes if you trust the VMs not to snoop on each other (and the hypervisor). That's *exactly the same* argument why we are not disabling HT by default, isn't it?

If the VMs are not trusted, enabling conditionally just half of the mitigations doesn't bring any additional security.

So the *whole and only* point of ('machine/system independent') 'l1tf=' parameter really is to let the system owner/admin to globally take a simple choice between

- I absolutely do want to isolate VMs and am OK with paying the performance price if it's needed on a particular HW (l1tf=full[,novirt])

- My VMs+hypervisor absolutely are friends with each other no matter what (l1tf="novirt,nowarn")

- dear kernel, please let me know in case something potentially dangerous is happening, but keep going (l1tf="novirt" or l1tf="full" with SMT on) -- the default

I am afraid that if we don't start from this point, the state space explosion happens immediately.

Or are you suggesting to flush L1D cache on *every* VM boundary crossing, unconditionally, by default? If so, please provide the full decision matrix understandable for end users (admins) so that each and every of them is able to make a qualified decision on the combination of smt/flushes/process afinity/irq afinity parameters they have to use so that they are "feeling proven safe".
I am afraid the error rate will be enormous, while with the proposed semantics I am trying to get to is as close as possible to clearly understandable "untrusted VMs->full mitigation" message as possible.

I also believe (Linus, if I am interpreting you in a wrong way, please speak up) that was why Linus (and others) originally suggested not to disable HT by default -- exactly because the 'default' case (with exception to the cloud providers) is that you *don't* require the full VM isolation (ignoring all the "HT, interrupts, caches, flushes" details) *by design*.

Okay, so if there is a solid proof that all the added vmenter/vmexit overhead is completely (or even in amortized sense) for free, so be it. Has anyone really done such analysis yet?

I can't really see us just adding L1D flush to every VM boundary without having even hand-wawy numbers available. Heck, we're accompanyning every tiny scheduler change by almost-a-master-thesis-in-statistics changelog, and now we'd just do this all of a sudden?

Then please read a changelog, and the text being added to kernel-parameters.txt, and the code.

Anyway, a simple question: which of the statements below you believe should be the default kernel view of the world, if it detects L1TF vulnerable CPU?

- the VMs are mutually not trusted (wrt. other VMs, and *also* wrt. the hypervisor)

- the VMs and the hypervisor mutually trust each other

- something else (please explain in terms of decision matrix / tree)

Thanks,

Jiri Kosina
SUSE Labs

full[,force] here, sorry for the noise.

Jiri Kosina
SUSE Labs

No, the *true* default is that you don't do VM's at all, even if you have the capability.

And those people who do the occasional "run some windows VM for xyz" don't care about the isolation much, not "by design", but simply because it's not an issue. They don't run crazy untrusted code.

The cloud people really are special. They can't trust their customers not to be actively evil. Almost nobody else cares.

Linus

HT off can be very expensive, L1 flush on entry is not.

So the trade offs are very different here. iirc the flush overhead was low enough that it's probably ok to just have it on by default,

Maybe an obscure tuanble somewhere to turn it off if someone is extremely sensitive to performance. But most users probably don't need to bother with that.

Everything helps, it slows down attackers and since the L1 cache is very transistent it may well make things impossible.

-Andi

What?

What were the 50-100% slowdown reports I saw? I think they were for disk-IO intensive stuff, but I seem to recall a huge slowdown from the L1 flush.

No?

Linus

I posted numbers once:

Bootup of a stock debian guest is 10% slower with L1D Flush on every VMENTER

A kernel compile in the same guest (all virtio) slows down by < 1%. The number of VMEXITs is minimal, so no surprise.

With a testcase which took an interrupt every 25us, the slowdown depends on the computational work done in the guest and ranges between 20 and 70%. The number of L1D misses increases by a factor of 13. noflush 2e6, flush 260e6.

And yes, we should clearly describe facts and not the wishful thinking of Intel. I'm so tired of this marketing drivel.

Thanks,

tglx

Yeah, compiles are generally fairly easy, and 90% of the time just CPU-intensive in user space.

I assum eyou tested the non-MSR one, doing the nasty flush by hand. That's probably worse, because not only does the flush (obviously) cause a lot of subsequent D$ fills, but the flush itself basically forces the same number of L1 misses in order to do the flush (and probably a lot of L2 misses too, to read the stupid pointless flush area).

So the "just invalidate the cache with an MSR write" is probably quite noticeably better.

But it's probably better by a factor of 2x in the best case, not a factor of 13x ;)

Linus

No, that was with the MSR based flush. The SW one is roughly in the same ballpark on that machine. It's slightly worse, but not a huge difference.

Thanks,

tglx

Most of that is the PIO overhead, which is not terrible interesting. It does an exit for every PIO to load kernel+initrd.

This one should go away anyways with Paolo's optimizations to not do L1 flush for such "fast exit" cases

The benchmarks I saw didn't show significant overhead for "real" workloads.

Of course you can always construct some worst case, but basing your decisions purely on worst cases is usually not a good strategy.

-Andi

It's still wrong not to clearly tell that this can have significant impact under certain conditions. VMEXIT heavy scenarios exist in reality.

Thanks,

tglx

Just to add, according to the numbers I've seen, it takes upwards of 512 cycles (double on older parts) to refill the L1D on Skylake. That's not huge, but it's not free. And it's not a complete mitigation on its own. It seems to make most sense to go with what Jiri and Linus have said.

In case it's helpful to have a comparison example though, and because not everyone here may be aware of it, POWER9 is vulnerable to original Meltdown and the currently upstream mitigation is to do an L1D$ flush on every return to userspace or hypervisor exit. Essentially, the same as is being described here, with also a similarly fast L2->L1 refill. They added a fast L1D$ flush instruction via millicode similar to Intel's.

Jon.

Computer Architect | Sent from my Fedora powered laptop

It's easy enough to make that automatic. There was an issue when I tried first, but that's not longer the case and I simply forgot to write the patch. I'll whip that up later today.

Thanks,

tglx