Optimize nSVM TLB flushes
Subsystems (5)
v2 cover Optimize nSVM TLB flushes
A second RFC series (or third) for optimizing TLB flushes for nested SVM, over a year later. This series optimizes TLB flushes by using and maintaining a separate ASID for L2 for each vCPU (similar to VMX's handling of VPIDs), rather than sharing the same ASID and flushing everything on every nested transition.
The series also drops the dynamic ASID allocation scheme and uses a static ASID for vCPU (or two if using nested), brining it further closer to how VMX handles VPIDs. Modern CPUs (checked Milan, Genoa, and Turin) advertise 32K ASIDs, so there is no point.
The series is mostly in a good shape, but it is an RFC because I only did basic testing (booted an L2 and an L3, run netperf in L2). I am mainly sending this out to get some feedback on the code while I do more extensive testing. Ideally I would test this with a memory overcommitted VM in L0 (to trigger KVM unmapping and flushing guest memory) and a CPU overcommitted VM in L1 (to trigger L1 KVM multiplexing multiple L2 ASIDs on the same L1 vCPU) -- let's see if AI can help with this.
Admittedly, the SEV and HV bits only got build tested, and I doubt that I would be able to do more than this going forward.
Since over a year passed and the code changed non-trivially, I dropped all Reviewed-by tags (sorry Maxim!), even on patches that are relatively the same. I won't go too much into the details for what changed because over a year has passed, but I will describe the main changes.
Breakdown:
- Patch 1 is a bug fix that can be taken separately.
- Patches 2-9 implement the change to use static ASIDs per vCPU for SVM, and can also be taken separately if needed.
- The rest of the patches (up to patch 24) add an L2 ASID and proper handling for it, leading to ultimately dropping the unconditional flushes on nested transitions.
- Patch 25 is purely for testing convenience. It adds a module param that adds back the unconditional flushes. Although if we are really paranoid we can keep it for a while? Idk if we usually do that.
RFC v1 -> RFC v2:
- Changed from a static ASID per VM to a static ASID per vCPU, dropping the needed per-CPU ASID -> vCPU tracking and dropping all SEV changes.
- Fixed a missing flush in the current code when forcefully leaving nested (patch 1).
- Expose more ASIDs to L1 (patch 2), otherwise the usefulness of this series is limited when L1 runs a bunch of L2s.
- Drop passing is_guest_mode around (past me, EW!).
- Reuse more of kvm_mmu_invlpg() for INVLPGA emulation.
- Fix INVLPGA emulation in case L1 runs on a different CPU than L2 (but L2 never switches its CPU).
- Resync the nested NPT when L1 changes L2's ASID (missed in RFC v1).
- Shuffled some patches around, combined/split patches differently, and rewrote some changlogs.
RFC v1: https://lore.kernel.org/lkml/[email protected]/
Yosry Ahmed (25):
KVM: nSVM: Flush the TLB after forcefully leaving nested
KVM: SVM: Passthrough the number of supported ASIDs
KVM: VMX: Generalize VPID allocation to be vendor-neutral
KVM: x86/mmu: Support specifying a minimum TLB tag
KVM: SVM: Add helpers to set/clear ASID flush in VMCB
KVM: SVM: Fallback to flush everything if FLUSHBYASID is not available
KVM: SVM: Duplicate pre-run ASID check for SEV and non-SEV guests
KVM: SEV: Stop using per-vCPU ASID for SEV VMs
KVM: SVM: Use a static ASID per vCPU
KVM: nSVM: Add a placeholder ASID for L2
KVM: x86: hyper-v: Rename kvm_hv_vcpu_purge_flush_tlb()
KVM: x86: hyper-v: Allow puring all TLB flush FIFOs
KVM: nSVM: Flush both L1 and L2 ASIDs on KVM_REQ_TLB_FLUSH
KVM: nSVM: Move svm_switch_vmcb() to nested.c
KVM: nSVM: Call nested_svm_transition_tlb_flush() on every VMCB switch
KVM: nSVM: Split nested_svm_transition_tlb_flush() into entry/exit fns
KVM: nSVM: Service local TLB flushes before nested transitions
KVM: nSVM: Handle nested TLB flush requests through TLB_CONTROL
KVM: nSVM: Flush the TLB if L1 changes L2's ASID in vmcb12
KVM: nSVM: Do not reset TLB_CONTROL in vmcb02 on nested VM-Enter
KVM: x86/mmu: rename __kvm_mmu_invalidate_addr()
KVM: x86/mmu: Refactor kvm_mmu_invlpg() to allow skipping the gva
flush
KVM: nSVM: Flush L2's ASID when emulating INVLPGA
KVM: nSVM: Use different ASIDs for L1 and L2
DO NOT MERGE: Add nested_tlb_force_flush
arch/x86/include/asm/kvm_host.h | 3 + arch/x86/kvm/cpuid.c | 2 - arch/x86/kvm/hyperv.h | 33 ++++++-- arch/x86/kvm/mmu.h | 8 ++ arch/x86/kvm/mmu/mmu.c | 106 +++++++++++++++++++++--- arch/x86/kvm/svm/nested.c | 106 ++++++++++++++++++------ arch/x86/kvm/svm/sev.c | 12 ++- arch/x86/kvm/svm/svm.c | 137 ++++++++++++++++++++------------ arch/x86/kvm/svm/svm.h | 41 +++++++--- arch/x86/kvm/vmx/vmx.c | 40 +++------- arch/x86/kvm/vmx/vmx.h | 29 ++++++- arch/x86/kvm/x86.c | 2 +- 12 files changed, 377 insertions(+), 142 deletions(-)
base-commit: c1f7303302927f9cbf4efedf70f0512cde168c65
Signature · 1 line
2.54.0.1136.gdb2ca164c4-goog
RFC PATCH v2 1/25 KVM: nSVM: Flush the TLB after forcefully leaving nested
KVM flushes the TLB on nested VM-Enter and nested VM-Exit, but not when forcefully leaving nested. In this case, L2 TLB entries can leak into L1. Flush the TLB after forcefully exiting L2, similar to nested VM-Exits.
Note that vmx_leave_nested() handles this correctly, as it reuses nested_vmx_vmexit(), which handles the necessary TLB flushes on an L2 -> L1 transition.
--- arch/x86/kvm/svm/nested.c | 2 ++ 1 file changed, 2 insertions(+)
arch/x86/kvm/svm/nested.c +2 -0
@@ -1550,6 +1550,8 @@ void svm_leave_nested(struct kvm_vcpu *vcpu)svm_switch_vmcb (svm, &svm->vmcb01); +nested_svm_transition_tlb_flush (vcpu); +nested_svm_uninit_mmu_context (vcpu);vmcb_mark_all_dirty (svm->vmcb);
Signature · 1 line
2.54.0.1136.gdb2ca164c4-goog
RFC PATCH v2 2/25 KVM: SVM: Passthrough the number of supported ASIDs
In reply to Yosry AhmedKVM currently hardcodes the number of supported ASIDs in CPUID to 8. A KVM guest (L1) would then do a full TLB flush (i.e. TLB_CONTROL_FLUSH_ALL_ASID) every time it runs out of ASIDs on a vCPU and updates the generation (see new_asid()).
This is currently harmless, as KVM (L0) uses the same ASID for both L1 and L2, and flushes that ASID on nested transitions. However, following changes will add proper ASID emulation and a separate ASID for L2, minimizing the TLB flushes on nested transitions. At that point, a full TLB flush from a KVM guest (L1) would flush both L1 and L2 ASIDs, so should be avoided as much as possible.
Passthrough the number of ASIDs in hardware instead of hardcoding 8, to reduce the chances of an L1 guest flushing its own TLB entries unnecessarily on a nested VMRUN.
In practice, there is no harm in exposing a large number of ASIDs to the guest, even larger than what hardware supports, as KVM never actually uses the value of ASID from vmcb12. Even with a separate L2 ASID, KVM would allocate a (supported) ASID for L2, and just flush that same ASID every time L1 changes the ASID in vmcb12.
That being said, avoid the temptation of just advertising the maximum possible number of ASIDs (i.e. 0xFFFFFFFF), in case any peculiar guest OS does not handle that properly.
Note: QEMU currently hardcodes the number of ASIDs to 16, so this change doesn't help QEMU VMs (without making a similar change in QEMU).
--- arch/x86/kvm/cpuid.c | 2 -- 1 file changed, 2 deletions(-)
arch/x86/kvm/cpuid.c +0 -2
@@ -1824,8 +1824,6 @@ static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function)break ; } entry->eax =1 ;/* SVM revision 1 */ - entry->ebx =8 ;/* Lets support 8 ASIDs in case we add proper -ASID emulation to nested SVM */ entry->ecx =0 ;/* Reserved */ cpuid_entry_override (entry,CPUID_8000_000A_EDX );break ;
Signature · 1 line
2.54.0.1136.gdb2ca164c4-goog
Re: [RFC PATCH v2 02/25] KVM: SVM: Passthrough the number of supported ASIDs
Quoted 23 lines · Yosry Ahmed
KVM currently hardcodes the number of supported ASIDs in CPUID to 8. A KVM guest (L1) would then do a full TLB flush (i.e.
TLB_CONTROL_FLUSH_ALL_ASID) every time it runs out of ASIDs on a vCPU and updates the generation (seenew_asid()).This is currently harmless, as KVM (L0) uses the same ASID for both L1 and L2, and flushes that ASID on nested transitions. However, following changes will add proper ASID emulation and a separate ASID for L2, minimizing the TLB flushes on nested transitions. At that point, a full TLB flush from a KVM guest (L1) would flush both L1 and L2 ASIDs, so should be avoided as much as possible.
Passthrough the number of ASIDs in hardware instead of hardcoding 8, to reduce the chances of an L1 guest flushing its own TLB entries unnecessarily on a nested VMRUN.
In practice, there is no harm in exposing a large number of ASIDs to the guest, even larger than what hardware supports, as KVM never actually uses the value of ASID from vmcb12. Even with a separate L2 ASID, KVM would allocate a (supported) ASID for L2, and just flush that same ASID every time L1 changes the ASID in vmcb12.
Quoted 6 lines
That being said, avoid the temptation of just advertising the maximum possible number of ASIDs (i.e.
0xFFFFFFFF), in case any peculiar guest OS does not handle that properly.Note: QEMU currently hardcodes the number of ASIDs to 16, so this change doesn't help QEMU VMs (without making a similar change in QEMU).
I had a discussion with Jim internally about what our VMM should advertise as the number of ASIDs, and based on that I think passing through the hardware value may not be the best idea, at least not without some disclaimers.
Jim pointed out that AMD CPUs will ignore unsupported ASID bits, or at least earlier ones did, even though it's not mentioned in the APM (last time I checked). This means that technically KVM can ignore the higher bits of the ASID set by L1 beyond supported ones (e.g. if it uses a hashtable to multiplex L1 ASIDs onto several physical ASIDs). KVM doesn't currently do so with or without this series, but it could.
The problem happens if a migration pool contains multiple AMD CPUs with a different number of ASIDs in hardware. If the VMM advertises the greater value of the two, it would technically be advertising an unsupported number of ASIDs on one of the CPUs. If KVM ignores the upper bits based on what it supports, not what userspace advertises in CPUID, then userspace would be shooting itself in the foot.
I think technically speaking userspace would be in the wrong for using a number of ASIDs above what KVM_GET_SUPPORTED_CPUID provides, and it should use a number of ASIDs that is supported by all CPUs in a migration pool. However, this makes the VMM's life difficult, say if AMD decides to reduce the number of ASIDs in a newer CPU just for kicks.
All of this is hypothetical in nature, but it's a very valid concern imo.
I think there are several ways we can handle this so that VMMs can reasonably start advertising more ASIDs to guests without these complications:
1. We can just advertise a larger ASID value instead of 8 (say 16K or 32K), rather than passthrough what's in hardware. I assume this would mean that KVM is more-or-less committing to supporting that number of ASIDs regardless of underlying hardware (i.e. KVM wouldn't reduce it in the future). In this case, a VMM can just advertise the same value.
2. We can advertise the maximum number of ASIDs (2^32 - 1?), which is the same as option (1), except that we're biting the bullet and preventing the possibility of KVM ignoring upper bits completely. Might be too aggressive.
3. Document (somewhere) that KVM should use the userspace advertised value as the source of truth for ignorable bits, even if KVM technically supports a lower value. As I am writing this I am realizing it doesn't make much sense as it means that the value supported by KVM is meaningless. Maybe that comes naturally with the "don't break userspace" rule, but it could also fall under the category of userspace shooting itself in the foot and KVM letting it.
4. Do nothing, let this be userspace's problem, and hope that AMD doesn't reduce the number of ASIDs in a future CPU.
I think option (1) is probably the most reasonable, we advertise 32K ASIDs, which is what recent AMD CPUs advertise anyway, but it's more-or-less KVM committing to 32K ASIDs even if a newer AMD CPU doesn't support them.
Sean, WDYT?
Re: [RFC PATCH v2 02/25] KVM: SVM: Passthrough the number of supported ASIDs
Quoted 83 lines · Yosry Ahmed
Quoted 23 lines · Yosry Ahmed
KVM currently hardcodes the number of supported ASIDs in CPUID to 8. A KVM guest (L1) would then do a full TLB flush (i.e.
TLB_CONTROL_FLUSH_ALL_ASID) every time it runs out of ASIDs on a vCPU and updates the generation (seenew_asid()).This is currently harmless, as KVM (L0) uses the same ASID for both L1 and L2, and flushes that ASID on nested transitions. However, following changes will add proper ASID emulation and a separate ASID for L2, minimizing the TLB flushes on nested transitions. At that point, a full TLB flush from a KVM guest (L1) would flush both L1 and L2 ASIDs, so should be avoided as much as possible.
Passthrough the number of ASIDs in hardware instead of hardcoding 8, to reduce the chances of an L1 guest flushing its own TLB entries unnecessarily on a nested VMRUN.
In practice, there is no harm in exposing a large number of ASIDs to the guest, even larger than what hardware supports, as KVM never actually uses the value of ASID from vmcb12. Even with a separate L2 ASID, KVM would allocate a (supported) ASID for L2, and just flush that same ASID every time L1 changes the ASID in vmcb12.
Quoted 6 lines
That being said, avoid the temptation of just advertising the maximum possible number of ASIDs (i.e.
0xFFFFFFFF), in case any peculiar guest OS does not handle that properly.Note: QEMU currently hardcodes the number of ASIDs to 16, so this change doesn't help QEMU VMs (without making a similar change in QEMU).
I had a discussion with Jim internally about what our VMM should advertise as the number of ASIDs, and based on that I think passing through the hardware value may not be the best idea, at least not without some disclaimers.
Jim pointed out that AMD CPUs will ignore unsupported ASID bits, or at least earlier ones did, even though it's not mentioned in the APM (last time I checked). This means that technically KVM can ignore the higher bits of the ASID set by L1 beyond supported ones (e.g. if it uses a hashtable to multiplex L1 ASIDs onto several physical ASIDs). KVM doesn't currently do so with or without this series, but it could.
The problem happens if a migration pool contains multiple AMD CPUs with a different number of ASIDs in hardware. If the VMM advertises the greater value of the two, it would technically be advertising an unsupported number of ASIDs on one of the CPUs. If KVM ignores the upper bits based on what it supports, not what userspace advertises in CPUID, then userspace would be shooting itself in the foot.
I think technically speaking userspace would be in the wrong for using a number of ASIDs above what
KVM_GET_SUPPORTED_CPUIDprovides, and it should use a number of ASIDs that is supported by all CPUs in a migration pool. However, this makes the VMM's life difficult, say if AMD decides to reduce the number of ASIDs in a newer CPU just for kicks.All of this is hypothetical in nature, but it's a very valid concern imo.
I think there are several ways we can handle this so that VMMs can reasonably start advertising more ASIDs to guests without these complications:
1. We can just advertise a larger ASID value instead of 8 (say 16K or 32K), rather than passthrough what's in hardware. I assume this would mean that KVM is more-or-less committing to supporting that number of ASIDs regardless of underlying hardware (i.e. KVM wouldn't reduce it in the future). In this case, a VMM can just advertise the same value.
2. We can advertise the maximum number of ASIDs (2^32 - 1?), which is the same as option (1), except that we're biting the bullet and preventing the possibility of KVM ignoring upper bits completely. Might be too aggressive.
3. Document (somewhere) that KVM should use the userspace advertised value as the source of truth for ignorable bits, even if KVM technically supports a lower value. As I am writing this I am realizing it doesn't make much sense as it means that the value supported by KVM is meaningless. Maybe that comes naturally with the "don't break userspace" rule, but it could also fall under the category of userspace shooting itself in the foot and KVM letting it.
Quoted 7 lines
4. Do nothing, let this be userspace's problem, and hope that AMD doesn't reduce the number of ASIDs in a future CPU.
I think option (1) is probably the most reasonable, we advertise 32K ASIDs, which is what recent AMD CPUs advertise anyway, but it's more-or-less KVM committing to 32K ASIDs even if a newer AMD CPU doesn't support them.
I think Bulldozer (family 15H) first introduced support for 32K ASIDs. KVM still supports CPUs older than that, doesn't it?
The tricky part about advertising more ASIDs than the host supports on pre-Bulldozer CPUs is that those CPUs don't have FlushByASID. So, do you flush all ASIDs when one is reused?
Re: [RFC PATCH v2 02/25] KVM: SVM: Passthrough the number of supported ASIDs
Quoted 93 lines · Jim Mattson
Quoted 83 lines · Yosry Ahmed
Quoted 23 lines · Yosry Ahmed
KVM currently hardcodes the number of supported ASIDs in CPUID to 8. A KVM guest (L1) would then do a full TLB flush (i.e.
TLB_CONTROL_FLUSH_ALL_ASID) every time it runs out of ASIDs on a vCPU and updates the generation (seenew_asid()).This is currently harmless, as KVM (L0) uses the same ASID for both L1 and L2, and flushes that ASID on nested transitions. However, following changes will add proper ASID emulation and a separate ASID for L2, minimizing the TLB flushes on nested transitions. At that point, a full TLB flush from a KVM guest (L1) would flush both L1 and L2 ASIDs, so should be avoided as much as possible.
Passthrough the number of ASIDs in hardware instead of hardcoding 8, to reduce the chances of an L1 guest flushing its own TLB entries unnecessarily on a nested VMRUN.
In practice, there is no harm in exposing a large number of ASIDs to the guest, even larger than what hardware supports, as KVM never actually uses the value of ASID from vmcb12. Even with a separate L2 ASID, KVM would allocate a (supported) ASID for L2, and just flush that same ASID every time L1 changes the ASID in vmcb12.
Quoted 6 lines
That being said, avoid the temptation of just advertising the maximum possible number of ASIDs (i.e.
0xFFFFFFFF), in case any peculiar guest OS does not handle that properly.Note: QEMU currently hardcodes the number of ASIDs to 16, so this change doesn't help QEMU VMs (without making a similar change in QEMU).
I had a discussion with Jim internally about what our VMM should advertise as the number of ASIDs, and based on that I think passing through the hardware value may not be the best idea, at least not without some disclaimers.
Jim pointed out that AMD CPUs will ignore unsupported ASID bits, or at least earlier ones did, even though it's not mentioned in the APM (last time I checked). This means that technically KVM can ignore the higher bits of the ASID set by L1 beyond supported ones (e.g. if it uses a hashtable to multiplex L1 ASIDs onto several physical ASIDs). KVM doesn't currently do so with or without this series, but it could.
The problem happens if a migration pool contains multiple AMD CPUs with a different number of ASIDs in hardware. If the VMM advertises the greater value of the two, it would technically be advertising an unsupported number of ASIDs on one of the CPUs. If KVM ignores the upper bits based on what it supports, not what userspace advertises in CPUID, then userspace would be shooting itself in the foot.
I think technically speaking userspace would be in the wrong for using a number of ASIDs above what
KVM_GET_SUPPORTED_CPUIDprovides, and it should use a number of ASIDs that is supported by all CPUs in a migration pool. However, this makes the VMM's life difficult, say if AMD decides to reduce the number of ASIDs in a newer CPU just for kicks.All of this is hypothetical in nature, but it's a very valid concern imo.
I think there are several ways we can handle this so that VMMs can reasonably start advertising more ASIDs to guests without these complications:
1. We can just advertise a larger ASID value instead of 8 (say 16K or 32K), rather than passthrough what's in hardware. I assume this would mean that KVM is more-or-less committing to supporting that number of ASIDs regardless of underlying hardware (i.e. KVM wouldn't reduce it in the future). In this case, a VMM can just advertise the same value.
2. We can advertise the maximum number of ASIDs (2^32 - 1?), which is the same as option (1), except that we're biting the bullet and preventing the possibility of KVM ignoring upper bits completely. Might be too aggressive.
3. Document (somewhere) that KVM should use the userspace advertised value as the source of truth for ignorable bits, even if KVM technically supports a lower value. As I am writing this I am realizing it doesn't make much sense as it means that the value supported by KVM is meaningless. Maybe that comes naturally with the "don't break userspace" rule, but it could also fall under the category of userspace shooting itself in the foot and KVM letting it.
Quoted 7 lines
4. Do nothing, let this be userspace's problem, and hope that AMD doesn't reduce the number of ASIDs in a future CPU.
I think option (1) is probably the most reasonable, we advertise 32K ASIDs, which is what recent AMD CPUs advertise anyway, but it's more-or-less KVM committing to 32K ASIDs even if a newer AMD CPU doesn't support them.
Quoted 6 lines
I think Bulldozer (family 15H) first introduced support for 32K ASIDs. KVM still supports CPUs older than that, doesn't it?
The tricky part about advertising more ASIDs than the host supports on pre-Bulldozer CPUs is that those CPUs don't have FlushByASID. So, do you flush all ASIDs when one is reused?
Currently KVM will flush all ASIDs when running out, then start reusing.
With my patches, KVM allocates a single ASID as the fallback ASID, and keeps reusing that one after running out of ASIDs. Without FlushByASID, KVM falls back to flushing all ASIDs. Performance will probably suck, but I don't think we care about performance that much on these CPUs?
Sean Christopherson
maintainerRe: [RFC PATCH v2 02/25] KVM: SVM: Passthrough the number of supported ASIDs
Quoted 9 lines · Yosry Ahmed
Quoted 8 lines · Jim Mattson
Quoted 8 lines
Quoted 2 lines
Note: QEMU currently hardcodes the number of ASIDs to 16, so this change doesn't help QEMU VMs (without making a similar change in QEMU).
I had a discussion with Jim internally about what our VMM should advertise as the number of ASIDs, and based on that I think passing through the hardware value may not be the best idea, at least not without some disclaimers.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Jim pointed out that AMD CPUs will ignore unsupported ASID bits, or at least earlier ones did, even though it's not mentioned in the APM (last time I checked). This means that technically KVM can ignore the higher bits of the ASID set by L1 beyond supported ones (e.g. if it uses a hashtable to multiplex L1 ASIDs onto several physical ASIDs). KVM doesn't currently do so with or without this series, but it could.
Deliberately ignoring ASID bits would certainly be a choice. I can see why hardware might make that compromise, but I don't see any reason why KVM should do the same.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
The problem happens if a migration pool contains multiple AMD CPUs with a different number of ASIDs in hardware. If the VMM advertises the greater value of the two, it would technically be advertising an unsupported number of ASIDs on one of the CPUs. If KVM ignores the upper bits based on what it supports, not what userspace advertises in CPUID, then userspace would be shooting itself in the foot.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
I think technically speaking userspace would be in the wrong for using a number of ASIDs above what
KVM_GET_SUPPORTED_CPUIDprovides, and it should use a number of ASIDs that is supported by all CPUs in a migration pool. However, this makes the VMM's life difficult, say if AMD decides to reduce the number of ASIDs in a newer CPU just for kicks.All of this is hypothetical in nature, but it's a very valid concern imo.
Eh, I disagree. It's hypothetical, and not a concern for KVM.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
I think there are several ways we can handle this so that VMMs can reasonably start advertising more ASIDs to guests without these complications:
What complications? Advertise the lowest common feature set for the pool, just like userspace has to do for literally every other feature.
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
1. We can just advertise a larger ASID value instead of 8 (say 16K or 32K), rather than passthrough what's in hardware. I assume this would mean that KVM is more-or-less committing to supporting that number of ASIDs regardless of underlying hardware (i.e. KVM wouldn't reduce it in the future). In this case, a VMM can just advertise the same value.
2. We can advertise the maximum number of ASIDs (2^32 - 1?), which is the same as option (1), except that we're biting the bullet and preventing the possibility of KVM ignoring upper bits completely. Might be too aggressive.
And potentially suboptimal for performance. There might be a legitimate reason why a CPU generation advertises X instead of Y.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
3. Document (somewhere) that KVM should use the userspace advertised value as the source of truth for ignorable bits, even if KVM technically supports a lower value. As I am writing this I am realizing it doesn't make much sense as it means that the value supported by KVM is meaningless.
As above, I don't think KVM should have "ignorable bits" in the first place.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Maybe that comes naturally with the "don't break userspace" rule, but it could also fall under the category of userspace shooting itself in the foot and KVM letting it.
4. Do nothing, let this be userspace's problem, and hope that AMD doesn't reduce the number of ASIDs in a future CPU.
By "Do nothing", I assume you mean "advertise what's reported in CPUID"?
If so there's a fifth option.
5. Be super paranoid and zero the entry, like KVM does for the topology leaves. See commit 45e966fcca03 ("KVM: x86: Do not return host topology information from KVM_GET_SUPPORTED_CPUID").
IMO, the only sane options are #4 and #5, and #5 feels unnecessarily paranoid. I genuinely don't see how this is any different than any other feature where userspace needs to not over-advertise features across its migration pool.
"Simple" use cases that don't do migration and just reflect KVM's CPUID into the guest Just Work, and more sophisticated use cases get an explicit hint from KVM as to the optimal number of ASIDs to advertise. I don't see why we need to make it more complex than that.
Re: [RFC PATCH v2 02/25] KVM: SVM: Passthrough the number of supported ASIDs
Quoted 2 lines · Sean Christopherson
What complications? Advertise the lowest common feature set for the pool, just like userspace has to do for literally every other feature.
The problem with passing through the number of host ASIDs to the guest is what to do when the combined host plus guest usage exceeds the number of available ASIDs, particularly when FlushByASID is not available.
Re: [RFC PATCH v2 02/25] KVM: SVM: Passthrough the number of supported ASIDs
Quoted 10 lines · Jim Mattson
Quoted 2 lines · Sean Christopherson
What complications? Advertise the lowest common feature set for the pool, just like userspace has to do for literally every other feature.
The problem with passing through the number of host ASIDs to the guest is what to do when the combined host plus guest usage exceeds the number of available ASIDs, particularly when FlushByASID is not available.
I think that particular scenario is fine because they are two separate domains. KVM assigns 1 ASIDs per vCPU for L2 (in this series), regardless of what L1 assigns L2. We flush that ASID if L1 changes the ASID it uses. We can use more than 1 ASID in the future, but I think KVM will still allocate a certain number of ASIDs per L2 and the domains will remain separate.
If your concern is the fact that without FlushByASID KVM will do a lot of full flushes, you are right, and this series aggressively falls back to full flushes when FlushByASID isn't available, rather than using a different ASID like the current approach. My assumption has been that we don't care about performance that much on these old CPUs.
Re: [RFC PATCH v2 02/25] KVM: SVM: Passthrough the number of supported ASIDs
Quoted 13 lines · Yosry Ahmed
Quoted 10 lines · Jim Mattson
Quoted 2 lines · Sean Christopherson
What complications? Advertise the lowest common feature set for the pool, just like userspace has to do for literally every other feature.
The problem with passing through the number of host ASIDs to the guest is what to do when the combined host plus guest usage exceeds the number of available ASIDs, particularly when FlushByASID is not available.
Quoted 6 lines
I think that particular scenario is fine because they are two separate domains. KVM assigns 1 ASIDs per vCPU for L2 (in this series), regardless of what L1 assigns L2. We flush that ASID if L1 changes the ASID it uses. We can use more than 1 ASID in the future, but I think KVM will still allocate a certain number of ASIDs per L2 and the domains will remain separate.
IIRC, Hyper-V uses two ASIDs per vCPU--or, at least, it used to. It might be worth checking to see if that is still the case.
Quoted 5 lines
If your concern is the fact that without FlushByASID KVM will do a lot of full flushes, you are right, and this series aggressively falls back to full flushes when FlushByASID isn't available, rather than using a different ASID like the current approach. My assumption has been that we don't care about performance that much on these old CPUs.
...until someone complains? :)
Re: [RFC PATCH v2 02/25] KVM: SVM: Passthrough the number of supported ASIDs
Quoted 25 lines · Jim Mattson
Quoted 13 lines · Yosry Ahmed
Quoted 10 lines · Jim Mattson
Quoted 2 lines · Sean Christopherson
What complications? Advertise the lowest common feature set for the pool, just like userspace has to do for literally every other feature.
The problem with passing through the number of host ASIDs to the guest is what to do when the combined host plus guest usage exceeds the number of available ASIDs, particularly when FlushByASID is not available.
Quoted 6 lines
I think that particular scenario is fine because they are two separate domains. KVM assigns 1 ASIDs per vCPU for L2 (in this series), regardless of what L1 assigns L2. We flush that ASID if L1 changes the ASID it uses. We can use more than 1 ASID in the future, but I think KVM will still allocate a certain number of ASIDs per L2 and the domains will remain separate.
IIRC, Hyper-V uses two ASIDs per vCPU--or, at least, it used to. It might be worth checking to see if that is still the case.
Quoted 7 lines
Quoted 5 lines
If your concern is the fact that without FlushByASID KVM will do a lot of full flushes, you are right, and this series aggressively falls back to full flushes when FlushByASID isn't available, rather than using a different ASID like the current approach. My assumption has been that we don't care about performance that much on these old CPUs.
...until someone complains? :)
I am not sure how many people run VMs on pre-Bulldozer CPUs and care about performance, but practically it's only a problem if the full flush actually makes a difference. It is possible that the TLB is not big enough; flushing the ASID vs flushing the entire TLB makes a difference.
If we really care, it's a problem not specifically for nested, but for assigning a fixed ASID to each vCPU in general (similar to VPIDs). It is a nice simplification, but AFAICT it's not necessary, the very first version of the RFC didn't have it [1], it was suggested by Sean (well, we went to per-VM ASIDs first, then per-vCPU ASIDs).
Re: [RFC PATCH v2 02/25] KVM: SVM: Passthrough the number of supported ASIDs
Quoted 33 lines · Yosry Ahmed
Quoted 31 lines · Jim Mattson
Quoted 13 lines · Yosry Ahmed
Quoted 10 lines · Jim Mattson
Quoted 2 lines · Sean Christopherson
What complications? Advertise the lowest common feature set for the pool, just like userspace has to do for literally every other feature.
The problem with passing through the number of host ASIDs to the guest is what to do when the combined host plus guest usage exceeds the number of available ASIDs, particularly when FlushByASID is not available.
Quoted 6 lines
I think that particular scenario is fine because they are two separate domains. KVM assigns 1 ASIDs per vCPU for L2 (in this series), regardless of what L1 assigns L2. We flush that ASID if L1 changes the ASID it uses. We can use more than 1 ASID in the future, but I think KVM will still allocate a certain number of ASIDs per L2 and the domains will remain separate.
IIRC, Hyper-V uses two ASIDs per vCPU--or, at least, it used to. It might be worth checking to see if that is still the case.
Quoted 5 lines
If your concern is the fact that without FlushByASID KVM will do a lot of full flushes, you are right, and this series aggressively falls back to full flushes when FlushByASID isn't available, rather than using a different ASID like the current approach. My assumption has been that we don't care about performance that much on these old CPUs.
Quoted 7 lines
Quoted 1 line
...until someone complains? :)
I am not sure how many people run VMs on pre-Bulldozer CPUs and care about performance, but practically it's only a problem if the full flush actually makes a difference. It is possible that the TLB is not big enough; flushing the ASID vs flushing the entire TLB makes a difference.
Well, the current implementation (one ASID per L2 per L1) already flushes the entire TLB every time that L2 ASID is reused. Performance can't get worse than that, I imagine.
Sean Christopherson
maintainerRe: [RFC PATCH v2 02/25] KVM: SVM: Passthrough the number of supported ASIDs
In reply to Jim MattsonQuoted 9 lines · Jim Mattson
Quoted 2 lines · Sean Christopherson
What complications? Advertise the lowest common feature set for the pool, just like userspace has to do for literally every other feature.
The problem with passing through the number of host ASIDs to the guest is what to do when the combined host plus guest usage exceeds the number of available ASIDs, particularly when FlushByASID is not available.
But that doesn't have anything to do with the number of ASIDs that KVM enumerates to L1. As of this series, the number ASIDs KVM will consume in hardware is a property of the total number of vCPUs in the system (one ASID per vCPU, plus one more for nested usage). E.g. if KVM advertises 10000 ASIDs to L1, all 10000 (10001, if ASID=0 counts?) of the those ASIDs will map to svm->nested.asid02.
Of course, that completely undermines my statement about suboptimal performance:
And potentially suboptimal for performance. There might be a legitimate reason why a CPU generation advertises X instead of Y.
My bogus assertion about performance notwithstanding, I still think advertising what hardware supports is the simple, sane approach. Because at the end of the day it's just that: advertising. Userspace can do whatever it wants, including peeking at raw CPUID. E.g. if we want to pull a stupid and emulate the behavior of ignoring "unsupported" ASIDs, then we'd need to do that based on userspace's defined CPUID model, not KVM's advertised support.
Re: [RFC PATCH v2 02/25] KVM: SVM: Passthrough the number of supported ASIDs
Quoted 12 lines · Sean Christopherson
Quoted 9 lines · Jim Mattson
Quoted 2 lines · Sean Christopherson
What complications? Advertise the lowest common feature set for the pool, just like userspace has to do for literally every other feature.
The problem with passing through the number of host ASIDs to the guest is what to do when the combined host plus guest usage exceeds the number of available ASIDs, particularly when FlushByASID is not available.
Quoted 5 lines
But that doesn't have anything to do with the number of ASIDs that KVM enumerates to L1. As of this series, the number ASIDs KVM will consume in hardware is a property of the total number of vCPUs in the system (one ASID per vCPU, plus one more for nested usage). E.g. if KVM advertises 10000 ASIDs to L1, all 10000 (10001, if ASID=0 counts?) of the those ASIDs will map to
svm->nested.asid02.
That's not the only conceivable implementation. KVM could advertise 64 ASIDs to L1 and reserve the corresponding non-zero ASIDs for L2 VMs (i.e. the first L1 ASID is 64). Then, vmcb02.asid = vmcb12.asid & 0x3f.
Quoted 11 lines
Of course, that completely undermines my statement about suboptimal performance:
And potentially suboptimal for performance. There might be a legitimate reason why a CPU generation advertises X instead of Y.
My bogus assertion about performance notwithstanding, I still think advertising what hardware supports is the simple, sane approach. Because at the end of the day it's just that: advertising. Userspace can do whatever it wants, including peeking at raw CPUID. E.g. if we want to pull a stupid and emulate the behavior of ignoring "unsupported" ASIDs, then we'd need to do that based on userspace's defined CPUID model, not KVM's advertised support.
But you could refuse a userspace CPUID model that claims more ASIDs than KVM supports.
And __nested_copy_vmcb_control_to_cache() already ignores a bunch of unsupported bits. I don't see how this is any different. That's how the hardware is implemented. Intel likes to fail VM-entry. AMD prefers to just ignore hypervisor stupidity and move on.
Sean Christopherson
maintainerRe: [RFC PATCH v2 02/25] KVM: SVM: Passthrough the number of supported ASIDs
Quoted 13 lines · Jim Mattson
Quoted 12 lines · Sean Christopherson
Quoted 9 lines · Jim Mattson
Quoted 2 lines · Sean Christopherson
What complications? Advertise the lowest common feature set for the pool, just like userspace has to do for literally every other feature.
The problem with passing through the number of host ASIDs to the guest is what to do when the combined host plus guest usage exceeds the number of available ASIDs, particularly when FlushByASID is not available.
Quoted 7 lines
Quoted 5 lines
But that doesn't have anything to do with the number of ASIDs that KVM enumerates to L1. As of this series, the number ASIDs KVM will consume in hardware is a property of the total number of vCPUs in the system (one ASID per vCPU, plus one more for nested usage). E.g. if KVM advertises 10000 ASIDs to L1, all 10000 (10001, if ASID=0 counts?) of the those ASIDs will map to
svm->nested.asid02.That's not the only conceivable implementation.
But it is what is proposed in this series.
Quoted 16 lines
KVM could advertise 64 ASIDs to L1 and reserve the corresponding non-zero ASIDs for L2 VMs (i.e. the first L1 ASID is 64). Then, vmcb02.asid = vmcb12.asid &
0x3f.Quoted 11 lines
Of course, that completely undermines my statement about suboptimal performance:
And potentially suboptimal for performance. There might be a legitimate reason why a CPU generation advertises X instead of Y.
My bogus assertion about performance notwithstanding, I still think advertising what hardware supports is the simple, sane approach. Because at the end of the day it's just that: advertising. Userspace can do whatever it wants, including peeking at raw CPUID. E.g. if we want to pull a stupid and emulate the behavior of ignoring "unsupported" ASIDs, then we'd need to do that based on userspace's defined CPUID model, not KVM's advertised support.
Quoted 2 lines
But you could refuse a userspace CPUID model that claims more ASIDs than KVM supports.
Nah, that's not the KVM way. For this one in particular, there's zero reason to enforce anything. Practically speaking, *if* we want KVM to act like hardware and ignore unsupported ASIDs, then KVM *must* allow userspace to define a max ASID other than exactly what's reported by KVM_GET_SUPPORTED_CPUID, otherwise migration pools become impossible. And at that point, disallowing a larger max ASID adds zero value.
If KVM claiming support for an explicit number of ASIDs is a sticking point, then I vote to zero the output in KVM_GET_SUPPORTED_CPUID, but I'd prefer not to do that because it will break reflecting KVM_GET_SUPPORTED_CPUID directly back into KVM_SET_CPUID2.
Quoted 4 lines
And
__nested_copy_vmcb_control_to_cache()already ignores a bunch of unsupported bits. I don't see how this is any different. That's how the hardware is implemented. Intel likes to fail VM-entry. AMD prefers to just ignore hypervisor stupidity and move on.
The issue is that KVM has played nice with unsupported ASIDs for years. If we want to change that, it needs to be quirked. And I just don't see the point, because KVM's behavior isn't outright wrong: the APM doesn't say what will happen if the hypervisor is being stupid.
Re: [RFC PATCH v2 02/25] KVM: SVM: Passthrough the number of supported ASIDs
Quoted 51 lines · Sean Christopherson
Quoted 13 lines · Jim Mattson
Quoted 12 lines · Sean Christopherson
Quoted 9 lines · Jim Mattson
Quoted 2 lines · Sean Christopherson
What complications? Advertise the lowest common feature set for the pool, just like userspace has to do for literally every other feature.
The problem with passing through the number of host ASIDs to the guest is what to do when the combined host plus guest usage exceeds the number of available ASIDs, particularly when FlushByASID is not available.
Quoted 7 lines
Quoted 5 lines
But that doesn't have anything to do with the number of ASIDs that KVM enumerates to L1. As of this series, the number ASIDs KVM will consume in hardware is a property of the total number of vCPUs in the system (one ASID per vCPU, plus one more for nested usage). E.g. if KVM advertises 10000 ASIDs to L1, all 10000 (10001, if ASID=0 counts?) of the those ASIDs will map to
svm->nested.asid02.That's not the only conceivable implementation.
But it is what is proposed in this series.
Quoted 16 lines
KVM could advertise 64 ASIDs to L1 and reserve the corresponding non-zero ASIDs for L2 VMs (i.e. the first L1 ASID is 64). Then, vmcb02.asid = vmcb12.asid &
0x3f.Quoted 11 lines
Of course, that completely undermines my statement about suboptimal performance:
And potentially suboptimal for performance. There might be a legitimate reason why a CPU generation advertises X instead of Y.
My bogus assertion about performance notwithstanding, I still think advertising what hardware supports is the simple, sane approach. Because at the end of the day it's just that: advertising. Userspace can do whatever it wants, including peeking at raw CPUID. E.g. if we want to pull a stupid and emulate the behavior of ignoring "unsupported" ASIDs, then we'd need to do that based on userspace's defined CPUID model, not KVM's advertised support.
Quoted 2 lines
But you could refuse a userspace CPUID model that claims more ASIDs than KVM supports.
Nah, that's not the KVM way. For this one in particular, there's zero reason to enforce anything. Practically speaking, *if* we want KVM to act like hardware and ignore unsupported ASIDs, then KVM *must* allow userspace to define a max ASID other than exactly what's reported by
KVM_GET_SUPPORTED_CPUID, otherwise migration pools become impossible. And at that point, disallowing a larger max ASID adds zero value.
Quoted 4 lines
If KVM claiming support for an explicit number of ASIDs is a sticking point, then I vote to zero the output in
KVM_GET_SUPPORTED_CPUID, but I'd prefer not to do that because it will break reflectingKVM_GET_SUPPORTED_CPUIDdirectly back intoKVM_SET_CPUID2.
If KVM supports any userspace-provided value for NASIDS, then shouldn't KVM_GET_SUPPORTED_CPUID return 0xffffffff?
Quoted 9 lines
Quoted 4 lines
And
__nested_copy_vmcb_control_to_cache()already ignores a bunch of unsupported bits. I don't see how this is any different. That's how the hardware is implemented. Intel likes to fail VM-entry. AMD prefers to just ignore hypervisor stupidity and move on.The issue is that KVM has played nice with unsupported ASIDs for years. If we want to change that, it needs to be quirked. And I just don't see the point, because KVM's behavior isn't outright wrong: the APM doesn't say what will happen if the hypervisor is being stupid.
The APM doesn't say that an out-of-bounds ASID causes VM-entry failure. What other possibilities remain?
Re: [RFC PATCH v2 02/25] KVM: SVM: Passthrough the number of supported ASIDs
Quoted 53 lines · Jim Mattson
Quoted 51 lines · Sean Christopherson
Quoted 13 lines · Jim Mattson
Quoted 12 lines · Sean Christopherson
Quoted 9 lines · Jim Mattson
Quoted 2 lines · Sean Christopherson
What complications? Advertise the lowest common feature set for the pool, just like userspace has to do for literally every other feature.
The problem with passing through the number of host ASIDs to the guest is what to do when the combined host plus guest usage exceeds the number of available ASIDs, particularly when FlushByASID is not available.
Quoted 7 lines
Quoted 5 lines
But that doesn't have anything to do with the number of ASIDs that KVM enumerates to L1. As of this series, the number ASIDs KVM will consume in hardware is a property of the total number of vCPUs in the system (one ASID per vCPU, plus one more for nested usage). E.g. if KVM advertises 10000 ASIDs to L1, all 10000 (10001, if ASID=0 counts?) of the those ASIDs will map to
svm->nested.asid02.That's not the only conceivable implementation.
But it is what is proposed in this series.
Quoted 16 lines
KVM could advertise 64 ASIDs to L1 and reserve the corresponding non-zero ASIDs for L2 VMs (i.e. the first L1 ASID is 64). Then, vmcb02.asid = vmcb12.asid &
0x3f.Quoted 11 lines
Of course, that completely undermines my statement about suboptimal performance:
And potentially suboptimal for performance. There might be a legitimate reason why a CPU generation advertises X instead of Y.
My bogus assertion about performance notwithstanding, I still think advertising what hardware supports is the simple, sane approach. Because at the end of the day it's just that: advertising. Userspace can do whatever it wants, including peeking at raw CPUID. E.g. if we want to pull a stupid and emulate the behavior of ignoring "unsupported" ASIDs, then we'd need to do that based on userspace's defined CPUID model, not KVM's advertised support.
Quoted 2 lines
But you could refuse a userspace CPUID model that claims more ASIDs than KVM supports.
Nah, that's not the KVM way. For this one in particular, there's zero reason to enforce anything. Practically speaking, *if* we want KVM to act like hardware and ignore unsupported ASIDs, then KVM *must* allow userspace to define a max ASID other than exactly what's reported by
KVM_GET_SUPPORTED_CPUID, otherwise migration pools become impossible. And at that point, disallowing a larger max ASID adds zero value.
Quoted 7 lines
Quoted 4 lines
If KVM claiming support for an explicit number of ASIDs is a sticking point, then I vote to zero the output in
KVM_GET_SUPPORTED_CPUID, but I'd prefer not to do that because it will break reflectingKVM_GET_SUPPORTED_CPUIDdirectly back intoKVM_SET_CPUID2.If KVM supports any userspace-provided value for NASIDS, then shouldn't
KVM_GET_SUPPORTED_CPUIDreturn0xffffffff?
Having given it some more thought, 0xffffffff might induce buggy guest behavior.
However, since KVM completely ignores NASID, it would be sad if migration were blocked because the target has a lower NASID value than the source.
That's an argument against changing it without some kind of userspace opt-in. Increasing it seems safe, but if userspace just reflects the value into KVM_SET_CPUID2, then an increase would block kernel rollback.
In any case, I think the new value should be a constant so it doesn't imply a migration constraint. Perhaps we should adopt the highest value reported on a physical CPU to date? I think that would be Bulldozer's 0x10000.
Quoted 12 lines
Quoted 9 lines
Quoted 4 lines
And
__nested_copy_vmcb_control_to_cache()already ignores a bunch of unsupported bits. I don't see how this is any different. That's how the hardware is implemented. Intel likes to fail VM-entry. AMD prefers to just ignore hypervisor stupidity and move on.The issue is that KVM has played nice with unsupported ASIDs for years. If we want to change that, it needs to be quirked. And I just don't see the point, because KVM's behavior isn't outright wrong: the APM doesn't say what will happen if the hypervisor is being stupid.
The APM doesn't say that an out-of-bounds ASID causes VM-entry failure. What other possibilities remain?
Sean Christopherson
maintainerRe: [RFC PATCH v2 02/25] KVM: SVM: Passthrough the number of supported ASIDs
Quoted 59 lines · Jim Mattson
Quoted 58 lines · Jim Mattson
Quoted 55 lines · Sean Christopherson
Quoted 13 lines · Jim Mattson
Quoted 12 lines · Sean Christopherson
Quoted 9 lines · Jim Mattson
Quoted 2 lines · Sean Christopherson
What complications? Advertise the lowest common feature set for the pool, just like userspace has to do for literally every other feature.
The problem with passing through the number of host ASIDs to the guest is what to do when the combined host plus guest usage exceeds the number of available ASIDs, particularly when FlushByASID is not available.
Quoted 7 lines
Quoted 5 lines
But that doesn't have anything to do with the number of ASIDs that KVM enumerates to L1. As of this series, the number ASIDs KVM will consume in hardware is a property of the total number of vCPUs in the system (one ASID per vCPU, plus one more for nested usage). E.g. if KVM advertises 10000 ASIDs to L1, all 10000 (10001, if ASID=0 counts?) of the those ASIDs will map to
svm->nested.asid02.That's not the only conceivable implementation.
But it is what is proposed in this series.
Quoted 16 lines
KVM could advertise 64 ASIDs to L1 and reserve the corresponding non-zero ASIDs for L2 VMs (i.e. the first L1 ASID is 64). Then, vmcb02.asid = vmcb12.asid &
0x3f.Quoted 11 lines
Of course, that completely undermines my statement about suboptimal performance:
And potentially suboptimal for performance. There might be a legitimate reason why a CPU generation advertises X instead of Y.
My bogus assertion about performance notwithstanding, I still think advertising what hardware supports is the simple, sane approach. Because at the end of the day it's just that: advertising. Userspace can do whatever it wants, including peeking at raw CPUID. E.g. if we want to pull a stupid and emulate the behavior of ignoring "unsupported" ASIDs, then we'd need to do that based on userspace's defined CPUID model, not KVM's advertised support.
Quoted 2 lines
But you could refuse a userspace CPUID model that claims more ASIDs than KVM supports.
Nah, that's not the KVM way. For this one in particular, there's zero reason to enforce anything. Practically speaking, *if* we want KVM to act like hardware and ignore unsupported ASIDs, then KVM *must* allow userspace to define a max ASID other than exactly what's reported by
KVM_GET_SUPPORTED_CPUID, otherwise migration pools become impossible. And at that point, disallowing a larger max ASID adds zero value.If KVM claiming support for an explicit number of ASIDs is a sticking point, then I vote to zero the output in
KVM_GET_SUPPORTED_CPUID, but I'd prefer not to do that because it will break reflectingKVM_GET_SUPPORTED_CPUIDdirectly back intoKVM_SET_CPUID2.
Quoted 4 lines
Quoted 2 lines
If KVM supports any userspace-provided value for NASIDS, then shouldn't
KVM_GET_SUPPORTED_CPUIDreturn0xffffffff?Having given it some more thought,
0xffffffffmight induce buggy guest behavior.
Ya.
Quoted 9 lines
However, since KVM completely ignores NASID, it would be sad if migration were blocked because the target has a lower NASID value than the source.
That's an argument against changing it without some kind of userspace opt-in. Increasing it seems safe, but if userspace just reflects the value into
KVM_SET_CPUID2, then an increase would block kernel rollback.
Quoted 4 lines
In any case, I think the new value should be a constant so it doesn't imply a migration constraint. Perhaps we should adopt the highest value reported on a physical CPU to date? I think that would be Bulldozer's
0x10000.
Or keep KVM's enumeration at 8, then find somewhere in KVM's documentation to describe KVM's behavior and provide guidance for userspace, but otherwise make it userspace's problem to solve? I'd be a-ok with that.
In practice, I highly doubt any production setups that support nSVM regurgitate 0x8000000A.EBX from KVM_GET_SUPPORTED_CPUID back into the guest. And if they do, very arguably keeping KVM's existing behavior is the safest option for them.
E.g. somewhat amusingly, QEMU hardcodes the number of ASIDs to 16:
*ebx = 0x00000010; /* nr of ASIDs */
I think we'd want to update KVM selftests to validate a broader range of ASIDs, but I'd want that coverage no matter what.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
And
__nested_copy_vmcb_control_to_cache()already ignores a bunch of unsupported bits. I don't see how this is any different. That's how the hardware is implemented. Intel likes to fail VM-entry. AMD prefers to just ignore hypervisor stupidity and move on.
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
The issue is that KVM has played nice with unsupported ASIDs for years. If we want to change that, it needs to be quirked. And I just don't see the point, because KVM's behavior isn't outright wrong: the APM doesn't say what will happen if the hypervisor is being stupid.
The APM doesn't say that an out-of-bounds ASID causes VM-entry failure. What other possibilities remain?
As real CPUs apparently do, the unsupported/unused bits can be ignored. But that's only a viable option if the number of ASIDs is a power-of-2, which KVM can't guarantee (without yet more semi-arbitrary restrictions on what virtual CPU model can be enumerated to the guest).
ASIDs above the max could be capped to the max ASID.
Hardware could detect the impossible-to-track ASID and effectively treat it like VMX handles VPID-disabled, i.e. flush on every VMRUN and #VMEXIT.
And obviously "hardware" could pretend the max is just a hint, and make it all work, as KVM does.
Re: [RFC PATCH v2 02/25] KVM: SVM: Passthrough the number of supported ASIDs
Quoted 11 lines · Sean Christopherson
Quoted 9 lines · Jim Mattson
However, since KVM completely ignores NASID, it would be sad if migration were blocked because the target has a lower NASID value than the source.
That's an argument against changing it without some kind of userspace opt-in. Increasing it seems safe, but if userspace just reflects the value into
KVM_SET_CPUID2, then an increase would block kernel rollback.
Quoted 8 lines
Quoted 4 lines
In any case, I think the new value should be a constant so it doesn't imply a migration constraint. Perhaps we should adopt the highest value reported on a physical CPU to date? I think that would be Bulldozer's
0x10000.Or keep KVM's enumeration at 8, then find somewhere in KVM's documentation to describe KVM's behavior and provide guidance for userspace, but otherwise make it userspace's problem to solve? I'd be a-ok with that.
That sounds best to me. Document that KVM ignores the guest NASID and commits to always supporting 2^32 ASIDs. Advise userspace to populate NASID with a power of 2, and perhaps mention that hardware has only reported three different values over the last 20 years: 0x40, 0x10000, and 0x8000.
Re: [RFC PATCH v2 02/25] KVM: SVM: Passthrough the number of supported ASIDs
Quoted 22 lines · Jim Mattson
Quoted 11 lines · Sean Christopherson
Quoted 9 lines · Jim Mattson
However, since KVM completely ignores NASID, it would be sad if migration were blocked because the target has a lower NASID value than the source.
That's an argument against changing it without some kind of userspace opt-in. Increasing it seems safe, but if userspace just reflects the value into
KVM_SET_CPUID2, then an increase would block kernel rollback.Quoted 8 lines
Quoted 4 lines
In any case, I think the new value should be a constant so it doesn't imply a migration constraint. Perhaps we should adopt the highest value reported on a physical CPU to date? I think that would be Bulldozer's
0x10000.Or keep KVM's enumeration at 8, then find somewhere in KVM's documentation to describe KVM's behavior and provide guidance for userspace, but otherwise make it userspace's problem to solve? I'd be a-ok with that.
Quoted 5 lines
That sounds best to me. Document that KVM ignores the guest NASID and commits to always supporting 2^32 ASIDs. Advise userspace to populate NASID with a power of 2, and perhaps mention that hardware has only reported three different values over the last 20 years:
0x40,0x10000, and0x8000.
I agree that documenting that KVM supports any NASID value from userspace is needed here, so that VMMs can set reasonable values and migration pools are sane.
That being said, I still think KVM's enumeration should be increased from 8. If KVM's enumeration is just a hint, then it should be a good one imo. Current L1 KVM will flush everything (L1 + L2) every time it runs out of ASIDs and pumps the generation, so advertising a small number of ASIDs leads to L1 unnecessarily hurting its performance. Newer KVM (with this series) won't suffer as much.
I think we probably want to pump KVM enumeration (e.g. to 0x8000 or even 0x40) *and* document that it's just a hint and KVM effectively allows any value.
Re: [RFC PATCH v2 02/25] KVM: SVM: Passthrough the number of supported ASIDs
Quoted 41 lines · Yosry Ahmed
Quoted 22 lines · Jim Mattson
Quoted 11 lines · Sean Christopherson
Quoted 9 lines · Jim Mattson
However, since KVM completely ignores NASID, it would be sad if migration were blocked because the target has a lower NASID value than the source.
That's an argument against changing it without some kind of userspace opt-in. Increasing it seems safe, but if userspace just reflects the value into
KVM_SET_CPUID2, then an increase would block kernel rollback.Quoted 8 lines
Quoted 4 lines
In any case, I think the new value should be a constant so it doesn't imply a migration constraint. Perhaps we should adopt the highest value reported on a physical CPU to date? I think that would be Bulldozer's
0x10000.Or keep KVM's enumeration at 8, then find somewhere in KVM's documentation to describe KVM's behavior and provide guidance for userspace, but otherwise make it userspace's problem to solve? I'd be a-ok with that.
Quoted 5 lines
That sounds best to me. Document that KVM ignores the guest NASID and commits to always supporting 2^32 ASIDs. Advise userspace to populate NASID with a power of 2, and perhaps mention that hardware has only reported three different values over the last 20 years:
0x40,0x10000, and0x8000.I agree that documenting that KVM supports any NASID value from userspace is needed here, so that VMMs can set reasonable values and migration pools are sane.
That being said, I still think KVM's enumeration should be increased from 8. If KVM's enumeration is just a hint, then it should be a good one imo. Current L1 KVM will flush everything (L1 + L2) every time it runs out of ASIDs and pumps the generation, so advertising a small number of ASIDs leads to L1 unnecessarily hurting its performance. Newer KVM (with this series) won't suffer as much.
Quoted 3 lines
I think we probably want to pump KVM enumeration (e.g. to
0x8000or even0x40) *and* document that it's just a hint and KVM effectively allows any value.
How do you do that without breaking rollback?
Re: [RFC PATCH v2 02/25] KVM: SVM: Passthrough the number of supported ASIDs
Quoted 43 lines · Jim Mattson
Quoted 41 lines · Yosry Ahmed
Quoted 22 lines · Jim Mattson
Quoted 11 lines · Sean Christopherson
Quoted 9 lines · Jim Mattson
However, since KVM completely ignores NASID, it would be sad if migration were blocked because the target has a lower NASID value than the source.
That's an argument against changing it without some kind of userspace opt-in. Increasing it seems safe, but if userspace just reflects the value into
KVM_SET_CPUID2, then an increase would block kernel rollback.Quoted 8 lines
Quoted 4 lines
In any case, I think the new value should be a constant so it doesn't imply a migration constraint. Perhaps we should adopt the highest value reported on a physical CPU to date? I think that would be Bulldozer's
0x10000.Or keep KVM's enumeration at 8, then find somewhere in KVM's documentation to describe KVM's behavior and provide guidance for userspace, but otherwise make it userspace's problem to solve? I'd be a-ok with that.
Quoted 5 lines
That sounds best to me. Document that KVM ignores the guest NASID and commits to always supporting 2^32 ASIDs. Advise userspace to populate NASID with a power of 2, and perhaps mention that hardware has only reported three different values over the last 20 years:
0x40,0x10000, and0x8000.I agree that documenting that KVM supports any NASID value from userspace is needed here, so that VMMs can set reasonable values and migration pools are sane.
That being said, I still think KVM's enumeration should be increased from 8. If KVM's enumeration is just a hint, then it should be a good one imo. Current L1 KVM will flush everything (L1 + L2) every time it runs out of ASIDs and pumps the generation, so advertising a small number of ASIDs leads to L1 unnecessarily hurting its performance. Newer KVM (with this series) won't suffer as much.
Quoted 5 lines
Quoted 3 lines
I think we probably want to pump KVM enumeration (e.g. to
0x8000or even0x40) *and* document that it's just a hint and KVM effectively allows any value.How do you do that without breaking rollback?
KVM's enumeration is just a hint :)
Also, in practice, even if you rollback the CPUID presented to an existing VM by userspace should never change, right?
Re: [RFC PATCH v2 02/25] KVM: SVM: Passthrough the number of supported ASIDs
Quoted 45 lines · Yosry Ahmed
Quoted 43 lines · Jim Mattson
Quoted 41 lines · Yosry Ahmed
Quoted 22 lines · Jim Mattson
Quoted 11 lines · Sean Christopherson
Quoted 9 lines · Jim Mattson
However, since KVM completely ignores NASID, it would be sad if migration were blocked because the target has a lower NASID value than the source.
That's an argument against changing it without some kind of userspace opt-in. Increasing it seems safe, but if userspace just reflects the value into
KVM_SET_CPUID2, then an increase would block kernel rollback.Quoted 8 lines
Quoted 4 lines
In any case, I think the new value should be a constant so it doesn't imply a migration constraint. Perhaps we should adopt the highest value reported on a physical CPU to date? I think that would be Bulldozer's
0x10000.Or keep KVM's enumeration at 8, then find somewhere in KVM's documentation to describe KVM's behavior and provide guidance for userspace, but otherwise make it userspace's problem to solve? I'd be a-ok with that.
Quoted 5 lines
That sounds best to me. Document that KVM ignores the guest NASID and commits to always supporting 2^32 ASIDs. Advise userspace to populate NASID with a power of 2, and perhaps mention that hardware has only reported three different values over the last 20 years:
0x40,0x10000, and0x8000.I agree that documenting that KVM supports any NASID value from userspace is needed here, so that VMMs can set reasonable values and migration pools are sane.
That being said, I still think KVM's enumeration should be increased from 8. If KVM's enumeration is just a hint, then it should be a good one imo. Current L1 KVM will flush everything (L1 + L2) every time it runs out of ASIDs and pumps the generation, so advertising a small number of ASIDs leads to L1 unnecessarily hurting its performance. Newer KVM (with this series) won't suffer as much.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
I think we probably want to pump KVM enumeration (e.g. to
0x8000or even0x40) *and* document that it's just a hint and KVM effectively allows any value.How do you do that without breaking rollback?
KVM's enumeration is just a hint :)
Yes, but existing userspace VMMs do not know that.
Quoted 2 lines
Also, in practice, even if you rollback the CPUID presented to an existing VM by userspace should never change, right?
Today's KVM claims support for only 8 ASIDs. If tomorrow's KVM supports 0x8000 ASIDS, and we start a VM on tomorrow's KVM with a userspace that copies NASID from KVM_GET_SUPPORTED_CPUID, that VM will look like it can't be supported on the old KVM if we have to roll back.
Hence, until you get an ACK from *all* userspace VMMs that they understand that KVM's reported NASID is just a hint, you can't change the reported ASID—unless, of course, you make that change an opt-in. But any VMM that opts in already knows that the value is just a hint, that it can set any NASID value it wants in the guest CPUID, and that all values are actually supported by all versions of KVM that report any non-zero value. So, then, what's the point of the opt-in? What's the point of changing the hint at all? Documentation solves the problem completely.
Sean Christopherson
maintainerRe: [RFC PATCH v2 02/25] KVM: SVM: Passthrough the number of supported ASIDs
Quoted 59 lines · Jim Mattson
Quoted 45 lines · Yosry Ahmed
Quoted 43 lines · Jim Mattson
Quoted 41 lines · Yosry Ahmed
Quoted 22 lines · Jim Mattson
Quoted 11 lines · Sean Christopherson
Quoted 9 lines · Jim Mattson
However, since KVM completely ignores NASID, it would be sad if migration were blocked because the target has a lower NASID value than the source.
That's an argument against changing it without some kind of userspace opt-in. Increasing it seems safe, but if userspace just reflects the value into
KVM_SET_CPUID2, then an increase would block kernel rollback.Quoted 8 lines
Quoted 4 lines
In any case, I think the new value should be a constant so it doesn't imply a migration constraint. Perhaps we should adopt the highest value reported on a physical CPU to date? I think that would be Bulldozer's
0x10000.Or keep KVM's enumeration at 8, then find somewhere in KVM's documentation to describe KVM's behavior and provide guidance for userspace, but otherwise make it userspace's problem to solve? I'd be a-ok with that.
Quoted 5 lines
That sounds best to me. Document that KVM ignores the guest NASID and commits to always supporting 2^32 ASIDs. Advise userspace to populate NASID with a power of 2, and perhaps mention that hardware has only reported three different values over the last 20 years:
0x40,0x10000, and0x8000.I agree that documenting that KVM supports any NASID value from userspace is needed here, so that VMMs can set reasonable values and migration pools are sane.
That being said, I still think KVM's enumeration should be increased from 8. If KVM's enumeration is just a hint, then it should be a good one imo. Current L1 KVM will flush everything (L1 + L2) every time it runs out of ASIDs and pumps the generation, so advertising a small number of ASIDs leads to L1 unnecessarily hurting its performance. Newer KVM (with this series) won't suffer as much.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
I think we probably want to pump KVM enumeration (e.g. to
0x8000or even0x40) *and* document that it's just a hint and KVM effectively allows any value.How do you do that without breaking rollback?
KVM's enumeration is just a hint :)
Yes, but existing userspace VMMs do not know that.
Quoted 2 lines
Also, in practice, even if you rollback the CPUID presented to an existing VM by userspace should never change, right?
Quoted 8 lines
Today's KVM claims support for only 8 ASIDs. If tomorrow's KVM supports
0x8000ASIDS, and we start a VM on tomorrow's KVM with a userspace that copies NASID fromKVM_GET_SUPPORTED_CPUID, that VM will look like it can't be supported on the old KVM if we have to roll back.Hence, until you get an ACK from *all* userspace VMMs that they understand that KVM's reported NASID is just a hint,
Way off topic, I keep reading NASID as NSAID and can't help but wonder if you've found another way to imply that KVM is diseased and needs drugs ;-)
Quoted 6 lines
you can't change the reported ASID—unless, of course, you make that change an opt-in. But any VMM that opts in already knows that the value is just a hint, that it can set any NASID value it wants in the guest CPUID, and that all values are actually supported by all versions of KVM that report any non-zero value. So, then, what's the point of the opt-in? What's the point of changing the hint at all? Documentation solves the problem completely.
+1000
Re: [RFC PATCH v2 02/25] KVM: SVM: Passthrough the number of supported ASIDs
Quoted 73 lines · Sean Christopherson
Quoted 59 lines · Jim Mattson
Quoted 45 lines · Yosry Ahmed
Quoted 43 lines · Jim Mattson
Quoted 41 lines · Yosry Ahmed
Quoted 22 lines · Jim Mattson
Quoted 11 lines · Sean Christopherson
Quoted 9 lines · Jim Mattson
However, since KVM completely ignores NASID, it would be sad if migration were blocked because the target has a lower NASID value than the source.
That's an argument against changing it without some kind of userspace opt-in. Increasing it seems safe, but if userspace just reflects the value into
KVM_SET_CPUID2, then an increase would block kernel rollback.Quoted 8 lines
Quoted 4 lines
In any case, I think the new value should be a constant so it doesn't imply a migration constraint. Perhaps we should adopt the highest value reported on a physical CPU to date? I think that would be Bulldozer's
0x10000.Or keep KVM's enumeration at 8, then find somewhere in KVM's documentation to describe KVM's behavior and provide guidance for userspace, but otherwise make it userspace's problem to solve? I'd be a-ok with that.
Quoted 5 lines
That sounds best to me. Document that KVM ignores the guest NASID and commits to always supporting 2^32 ASIDs. Advise userspace to populate NASID with a power of 2, and perhaps mention that hardware has only reported three different values over the last 20 years:
0x40,0x10000, and0x8000.I agree that documenting that KVM supports any NASID value from userspace is needed here, so that VMMs can set reasonable values and migration pools are sane.
That being said, I still think KVM's enumeration should be increased from 8. If KVM's enumeration is just a hint, then it should be a good one imo. Current L1 KVM will flush everything (L1 + L2) every time it runs out of ASIDs and pumps the generation, so advertising a small number of ASIDs leads to L1 unnecessarily hurting its performance. Newer KVM (with this series) won't suffer as much.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
I think we probably want to pump KVM enumeration (e.g. to
0x8000or even0x40) *and* document that it's just a hint and KVM effectively allows any value.How do you do that without breaking rollback?
KVM's enumeration is just a hint :)
Yes, but existing userspace VMMs do not know that.
Quoted 2 lines
Also, in practice, even if you rollback the CPUID presented to an existing VM by userspace should never change, right?
Quoted 8 lines
Today's KVM claims support for only 8 ASIDs. If tomorrow's KVM supports
0x8000ASIDS, and we start a VM on tomorrow's KVM with a userspace that copies NASID fromKVM_GET_SUPPORTED_CPUID, that VM will look like it can't be supported on the old KVM if we have to roll back.Hence, until you get an ACK from *all* userspace VMMs that they understand that KVM's reported NASID is just a hint,
Way off topic, I keep reading NASID as NSAID and can't help but wonder if you've found another way to imply that KVM is diseased and needs drugs ;-)
Quoted 8 lines
Quoted 6 lines
you can't change the reported ASID—unless, of course, you make that change an opt-in. But any VMM that opts in already knows that the value is just a hint, that it can set any NASID value it wants in the guest CPUID, and that all values are actually supported by all versions of KVM that report any non-zero value. So, then, what's the point of the opt-in? What's the point of changing the hint at all? Documentation solves the problem completely.
+1000
Pointed taken. I will drop this patch in the next version and add a patch documenting this instead.
Re: [RFC PATCH v2 02/25] KVM: SVM: Passthrough the number of supported ASIDs
In reply to Sean ChristophersonQuoted 18 lines · Sean Christopherson
Quoted 9 lines · Yosry Ahmed
Quoted 8 lines · Jim Mattson
Quoted 8 lines
Quoted 2 lines
Note: QEMU currently hardcodes the number of ASIDs to 16, so this change doesn't help QEMU VMs (without making a similar change in QEMU).
I had a discussion with Jim internally about what our VMM should advertise as the number of ASIDs, and based on that I think passing through the hardware value may not be the best idea, at least not without some disclaimers.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Jim pointed out that AMD CPUs will ignore unsupported ASID bits, or at least earlier ones did, even though it's not mentioned in the APM (last time I checked). This means that technically KVM can ignore the higher bits of the ASID set by L1 beyond supported ones (e.g. if it uses a hashtable to multiplex L1 ASIDs onto several physical ASIDs). KVM doesn't currently do so with or without this series, but it could.
Quoted 3 lines
Deliberately ignoring ASID bits would certainly be a choice. I can see why hardware might make that compromise, but I don't see any reason why KVM should do the same.
Right, but KVM could decide to start doing that in the future and still be architecturally sound.
Quoted 17 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
The problem happens if a migration pool contains multiple AMD CPUs with a different number of ASIDs in hardware. If the VMM advertises the greater value of the two, it would technically be advertising an unsupported number of ASIDs on one of the CPUs. If KVM ignores the upper bits based on what it supports, not what userspace advertises in CPUID, then userspace would be shooting itself in the foot.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
I think technically speaking userspace would be in the wrong for using a number of ASIDs above what
KVM_GET_SUPPORTED_CPUIDprovides, and it should use a number of ASIDs that is supported by all CPUs in a migration pool. However, this makes the VMM's life difficult, say if AMD decides to reduce the number of ASIDs in a newer CPU just for kicks.All of this is hypothetical in nature, but it's a very valid concern imo.
Quoted 8 lines
Eh, I disagree. It's hypothetical, and not a concern for KVM.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
I think there are several ways we can handle this so that VMMs can reasonably start advertising more ASIDs to guests without these complications:
What complications? Advertise the lowest common feature set for the pool, just like userspace has to do for literally every other feature.
See below.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
1. We can just advertise a larger ASID value instead of 8 (say 16K or 32K), rather than passthrough what's in hardware. I assume this would mean that KVM is more-or-less committing to supporting that number of ASIDs regardless of underlying hardware (i.e. KVM wouldn't reduce it in the future). In this case, a VMM can just advertise the same value.
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
2. We can advertise the maximum number of ASIDs (2^32 - 1?), which is the same as option (1), except that we're biting the bullet and preventing the possibility of KVM ignoring upper bits completely. Might be too aggressive.
And potentially suboptimal for performance. There might be a legitimate reason why a CPU generation advertises X instead of Y.
Right, for the CPU, but why would it be suboptimal for KVM to advertise the max number of ASIDs. It's actually more optimal with the current implementation of KVM, because L1 KVM has the least chance of doing a full flush and flushing its own TLB entries unnecessarily.
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
3. Document (somewhere) that KVM should use the userspace advertised value as the source of truth for ignorable bits, even if KVM technically supports a lower value. As I am writing this I am realizing it doesn't make much sense as it means that the value supported by KVM is meaningless.
As above, I don't think KVM should have "ignorable bits" in the first place.
I agree, but nothing guarantees that, unless it falls under "don't break userspace"?
Quoted 9 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Maybe that comes naturally with the "don't break userspace" rule, but it could also fall under the category of userspace shooting itself in the foot and KVM letting it.
4. Do nothing, let this be userspace's problem, and hope that AMD doesn't reduce the number of ASIDs in a future CPU.
By "Do nothing", I assume you mean "advertise what's reported in CPUID"?
Yeah, like this patch essentially.
Quoted 10 lines
If so there's a fifth option.
5. Be super paranoid and zero the entry, like KVM does for the topology leaves. See commit
45e966fcca03("KVM: x86: Do not return host topology information fromKVM_GET_SUPPORTED_CPUID").IMO, the only sane options are #4 and #5, and #5 feels unnecessarily paranoid. I genuinely don't see how this is any different than any other feature where userspace needs to not over-advertise features across its migration pool.
I think the ASIDs case is more nuanced, at least in my mind, because:
- It's more likely for future CPUs to advertise a lower number of ASIDs than stop supporting a feature.
- What KVM advertises to userspace is not really a hard cap, it's actually meaningless today, and userspace can just ignore it and advertise whatever and it would work. But, there is no guarantee this will continue to work if KVM decides otherwise.
Sure, KVM can still punt this to userspace, but it might just make VMM's lives more difficult for no good reason. This came up internally when we discussed what our internal VMM should advertise. If we just advertise the same number of ASIDs advertised by HW for the virtual CPU, we risk not being able to add future AMD CPUs to the migration pool if they have a lower number of ASIDs.
Quoted 4 lines
"Simple" use cases that don't do migration and just reflect KVM's CPUID into the guest Just Work, and more sophisticated use cases get an explicit hint from KVM as to the optimal number of ASIDs to advertise. I don't see why we need to make it more complex than that.
As long as it's "a hint" it should be fine. The problem happens if KVM decides to make it a hard limit and ignore upper bits based on it. AFAICT, nothing prevents KVM from doing this in the future.
RFC PATCH v2 3/25 KVM: VMX: Generalize VPID allocation to be vendor-neutral
In reply to Yosry AhmedIn preparation for sharing with SVM, generalize the VMX VPID allocation code and move it to common code as a TLB tags allocator. Parameterize the TLB tags allocator by the number of tags, and allocate the bitmap dynamically. Opportunisitcally use guards to acquire the lock instead of spin_{lock/unlock}().
The number of tags includes tag=0, which is not usable. The interface is a little confusing in that regard, but this will be changed with the introducing of a minimum tag later.
Initialize the TLB tags allocator during hardware setup/unsetup, and reserve tag=0 during initialziation, similar to how VPID=0 is currently reserved in the VMX-specific bitmap during hardware setup.
Keep allocate_vpid() and free_vpid() as wrapper that check enable_vpid to avoid checking at all callsites, and add init_vpids() and destroy_vpids() to wrap init/destroy calls as well.
No functional change intended.
--- arch/x86/kvm/mmu.h | 8 ++++++ arch/x86/kvm/mmu/mmu.c | 64 ++++++++++++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/vmx.c | 40 ++++++-------------------- arch/x86/kvm/vmx/vmx.h | 28 +++++++++++++++--- 4 files changed, 105 insertions(+), 35 deletions(-)
arch/x86/kvm/mmu.h +8 -0
@@ -334,4 +334,12 @@ static inline bool kvm_is_gfn_alias(struct kvm *kvm, gfn_t gfn) {return gfn &kvm_gfn_direct_bits (kvm); } + +typedef unsigned int kvm_tlb_tag_t ; + +int kvm_init_tlb_tags (unsigned int nr); +void kvm_destroy_tlb_tags (void ); +kvm_tlb_tag_t kvm_alloc_tlb_tag (void ); +void kvm_free_tlb_tag (kvm_tlb_tag_t tag); +#endif
arch/x86/kvm/mmu/mmu.c +64 -0
68 lines not on this page · raw message
arch/x86/kvm/vmx/vmx.c +9 -31
@@ -594,9 +594,6 @@ DEFINE_PER_CPU(struct vmcs *, current_vmcs);*/ static DEFINE_PER_CPU(struct list_head, loaded_vmcss_on_cpu); -static DECLARE_BITMAP(vmx_vpid_bitmap, VMX_NR_VPIDS); -static DEFINE_SPINLOCK (vmx_vpid_lock); -struct vmcs_config vmcs_config__ro_after_init ;struct vmx_capability vmx_capability__ro_after_init ;
@@ -4067,31 +4064,6 @@ static void seg_setup(int seg)vmcs_write32 (sf->ar_bytes, ar); } -int allocate_vpid (void ) -{ -int vpid; - -if (!enable_vpid) -return 0 ; -spin_lock (&vmx_vpid_lock); - vpid =find_first_zero_bit (vmx_vpid_bitmap,VMX_NR_VPIDS ); -if (vpid <VMX_NR_VPIDS ) -__set_bit (vpid, vmx_vpid_bitmap); -else - vpid =0 ; -spin_unlock (&vmx_vpid_lock); -return vpid; -} - -void free_vpid (int vpid) -{ -if (!enable_vpid || vpid ==0 ) -return ; -spin_lock (&vmx_vpid_lock); -__clear_bit (vpid, vmx_vpid_bitmap); -spin_unlock (&vmx_vpid_lock); -} -static void vmx_msr_bitmap_l01_changed (struct vcpu_vmx *vmx) {/*
@@ -8474,6 +8446,8 @@ void vmx_hardware_unsetup(void)if (nested)nested_vmx_hardware_unsetup (); + +destroy_vpids (); }void vmx_vm_destroy (struct kvm *kvm)
@@ -8699,8 +8673,6 @@ __init int vmx_hardware_setup(void) kvm_caps.has_bus_lock_exit =cpu_has_vmx_bus_lock_detection (); kvm_caps.has_notify_vmexit =cpu_has_notify_vmexit (); -set_bit (0 , vmx_vpid_bitmap);/* 0 is reserved for host */ -if (enable_ept)kvm_mmu_set_ept_masks (enable_ept_ad_bits);else
@@ -8765,6 +8737,10 @@ __init int vmx_hardware_setup(void)vmx_set_cpu_caps (); + r =init_vpids (); +if (r) +return r; +/* * Configure nested capabilities after core CPU capabilities so that * nested support can be conditional on base support, e.g. so that KVM
@@ -8772,8 +8748,10 @@ __init int vmx_hardware_setup(void)*/ if (nested) { r =nested_vmx_hardware_setup (kvm_vmx_exit_handlers); -if (r) +if (r) { +destroy_vpids ();return r; + } }kvm_set_posted_intr_wakeup_handler (pi_wakeup_handler);
arch/x86/kvm/vmx/vmx.h +24 -4
@@ -175,7 +175,7 @@ struct nested_vmx {u64 pre_vmenter_ssp;u64 pre_vmenter_ssp_tbl; -u16 vpid02; +kvm_tlb_tag_t vpid02;u16 last_vpid;int tsc_autostore_slot;
@@ -249,7 +249,7 @@ struct vcpu_vmx {u32 ar; } seg[8 ]; } segment_cache; -int vpid; +kvm_tlb_tag_t vpid;/* Support for a guest hypervisor (nested VMX) */ struct nested_vmx nested;
@@ -334,9 +334,29 @@ static __always_inline u32 vmx_get_intr_info(struct kvm_vcpu *vcpu)return vt->exit_intr_info; } +static __always_inline int init_vpids(void) +{ +return enable_vpid ?kvm_init_tlb_tags (VMX_NR_VPIDS ) :0 ; +} + +static __always_inline void destroy_vpids(void) +{ +if (enable_vpid) +kvm_destroy_tlb_tags (); +} + +static __always_inline kvm_tlb_tag_t allocate_vpid(void) +{ +return enable_vpid ?kvm_alloc_tlb_tag () :0 ; +} + +static __always_inline void free_vpid(kvm_tlb_tag_t vpid) +{ +if (enable_vpid) +kvm_free_tlb_tag (vpid); +} +void vmx_vcpu_load_vmcs (struct kvm_vcpu *vcpu,int cpu); -int allocate_vpid (void ); -void free_vpid (int vpid);void vmx_set_constant_host_state (struct vcpu_vmx *vmx);void vmx_prepare_switch_to_guest (struct kvm_vcpu *vcpu); void vmx_set_host_fs_gs(struct vmcs_host_state *host, u16 fs_sel, u16 gs_sel,
Signature · 1 line
2.54.0.1136.gdb2ca164c4-goog
Sean Christopherson
maintainerRe: [RFC PATCH v2 03/25] KVM: VMX: Generalize VPID allocation to be vendor-neutral
Quoted 8 lines · Yosry Ahmed
arch/x86/kvm/mmu/mmu.c +0
-0@@ -8192,4 +8192,68 @@ void kvm_mmu_init_memslot_memory_attributes(struct kvm *kvm, } } }
arch/x86/kvm/mmu/mmu.c void kvm_mmu_init_memslot_memory_attributes(struct kvm *kvm, · line 8204
Quoted 10 lines
arch/x86/kvm/mmu/mmu.c +10
-0+ +static struct { +spinlock_t lock; +unsigned long *bitmap; +unsigned int nr; +} tlb_tags; + +int kvm_init_tlb_tags (unsigned int nr) +{ +if (WARN_ON_ONCE (!nr))
I think we should cap @nr at 0xffff, i.e. at VMX_NR_VPIDS -1. If we end up on a system (e.g. in a VM) that supports 4 billion ASIDs, KVM will burn 64MiB for the bitmap, without any reasonable hope of actually consuming anywhere near that many ASIDs. Burning at most 1024 bytes is far more reasonable.
To provide some defence against future systems, maybe pr_warn() or something if the number of ASIDs is capped?
Re: [RFC PATCH v2 03/25] KVM: VMX: Generalize VPID allocation to be vendor-neutral
Quoted 14 lines · Sean Christopherson
Quoted 12 lines · Yosry Ahmed
arch/x86/kvm/mmu/mmu.c +4
-0@@ -8192,4 +8192,68 @@ void kvm_mmu_init_memslot_memory_attributes(struct kvm *kvm, } } } + +static struct { + spinlock_t lock; + unsigned long *bitmap;
Quoted 8 lines
arch/x86/kvm/mmu/mmu.c void kvm_mmu_init_memslot_memory_attributes(struct kvm *kvm, · line 8204
Quoted 6 lines
arch/x86/kvm/mmu/mmu.c +6
-0+unsigned int nr; +} tlb_tags; + +int kvm_init_tlb_tags (unsigned int nr) +{ +if (WARN_ON_ONCE (!nr))I think we should cap @nr at
0xffff, i.e. atVMX_NR_VPIDS-1. If we end up on
Why not VMX_NR_VPIDS (i.e. 0x10000) like the current implementation? Is it to avoid allocating an extra long just for one 1 bit? It would be a change of behavior for VMX tho, not that anyone would care.
If we do this, I'd rather replace VMX_NR_VPIDS with a generic MAX_NR_TLB_TAGS, and just have the VMX code use that. WDYT?
Quoted 6 lines
a system (e.g. in a VM) that supports 4 billion ASIDs, KVM will burn 64MiB for the bitmap, without any reasonable hope of actually consuming anywhere near that many ASIDs. Burning at most 1024 bytes is far more reasonable.
To provide some defence against future systems, maybe
pr_warn()or something if the number of ASIDs is capped?
Sounds good.
Sean Christopherson
maintainerRe: [RFC PATCH v2 03/25] KVM: VMX: Generalize VPID allocation to be vendor-neutral
Quoted 24 lines · Yosry Ahmed
Quoted 14 lines · Sean Christopherson
Quoted 12 lines · Yosry Ahmed
arch/x86/kvm/mmu/mmu.c +4
-0@@ -8192,4 +8192,68 @@ void kvm_mmu_init_memslot_memory_attributes(struct kvm *kvm, } } } + +static struct { + spinlock_t lock; + unsigned long *bitmap;Quoted 8 lines
arch/x86/kvm/mmu/mmu.c void kvm_mmu_init_memslot_memory_attributes(struct kvm *kvm, · line 8204
Quoted 6 lines
arch/x86/kvm/mmu/mmu.c +6
-0+unsigned int nr; +} tlb_tags; + +int kvm_init_tlb_tags (unsigned int nr) +{ +if (WARN_ON_ONCE (!nr))I think we should cap @nr at
0xffff, i.e. atVMX_NR_VPIDS-1. If we end up on
Quoted 1 line
Why not
VMX_NR_VPIDS(i.e.0x10000) like the current implementation?
Purely because I wrote that suggestion when looking at that final code in this series that passed "VMX_NR_VPIDS - 1" as @nr, and didn't think too hard about the math.
Quoted 1 line
Is it to avoid allocating an extra long just for one 1 bit?
FWIW, it's not actually an extra long, __KERNEL_DIV_ROUND_UP(0xffff, 64) and __KERNEL_DIV_ROUND_UP(0x10000, 64) both come out as 1024.
Quoted 4 lines
It would be a change of behavior for VMX tho, not that anyone would care.
If we do this, I'd rather replace
VMX_NR_VPIDSwith a genericMAX_NR_TLB_TAGS, and just have the VMX code use that. WDYT?
The VMX code should use VMX_NR_VPIDS, that's its architectural max. I say avoid a #define and keep the limit internal to kvm_init_tlb_tags(). I can't think of any reason to expose that limit outside of the allocation. E.g.
int kvm_init_tlb_tags(unsigned int nr, unsigned int nr_reserved)
{
/*
* Limit the number of TLB tags to VMX's hardcoded maximum of 0x10000
* to avoid wasting memory for the bitmap in the unlikely scenario the
* CPU supports an inordinate number of ASIDs (on AMD). If userspace
* wants to concurrently run tens of thousands of vCPUs, they'll likely
* need a solution that works for both Intel and AMD.
*/
const unsigned int MAX_NR_TLB_TAGS = VMX_NR_VPIDS;
Quoted 8 lines
Quoted 6 lines
a system (e.g. in a VM) that supports 4 billion ASIDs, KVM will burn 64MiB for the bitmap, without any reasonable hope of actually consuming anywhere near that many ASIDs. Burning at most 1024 bytes is far more reasonable.
To provide some defence against future systems, maybe
pr_warn()or something if the number of ASIDs is capped?Sounds good.
RFC PATCH v2 4/25 KVM: x86/mmu: Support specifying a minimum TLB tag
In reply to Yosry AhmedIn preparation for using the TLB tags allocator for SVM, which has a range of ASIDs allocated for SEV/SNP, pass in a minimum TLB tag when initializing the TLB tags allocator. The bitmap is conceptually shifted such that bit=0 corresponds to tag=min.
Specifying the minimum value during initialization also makes the API clearer, as the passed number of tags becomes the actual number of *usable* tags, and tag=0 is explicitly excluded by the caller.
No functional change intended for VMX as VPID=0 is not used anyway.
--- arch/x86/kvm/mmu.h | 2 +- arch/x86/kvm/mmu/mmu.c | 38 +++++++++++++++++++++++--------------- arch/x86/kvm/vmx/vmx.h | 3 ++- 3 files changed, 26 insertions(+), 17 deletions(-)
arch/x86/kvm/mmu.h +1 -1
@@ -337,7 +337,7 @@ static inline bool kvm_is_gfn_alias(struct kvm *kvm, gfn_t gfn)typedef unsigned int kvm_tlb_tag_t ; -int kvm_init_tlb_tags (unsigned int nr); +int kvm_init_tlb_tags (kvm_tlb_tag_t min,unsigned int nr);void kvm_destroy_tlb_tags (void );kvm_tlb_tag_t kvm_alloc_tlb_tag (void );void kvm_free_tlb_tag (kvm_tlb_tag_t tag);
arch/x86/kvm/mmu/mmu.c +23 -15
@@ -8197,24 +8197,26 @@ static struct {spinlock_t lock;unsigned long *bitmap;unsigned int nr; +kvm_tlb_tag_t min; } tlb_tags; -int kvm_init_tlb_tags (unsigned int nr) +int kvm_init_tlb_tags (kvm_tlb_tag_t min,unsigned int nr) { -if (WARN_ON_ONCE (!nr)) -return -EINVAL ; - - tlb_tags.bitmap =bitmap_zalloc (nr,GFP_KERNEL ); -if (!tlb_tags.bitmap) -return -ENOMEM ; +unsigned int end;/* * 0 is the host's TLB tag for both VMX's VPID and SVM's ASID, and is * returned on failed allocations (e.g. no more tags left). */ -__set_bit (0 , tlb_tags.bitmap); +if (WARN_ON_ONCE (!min || !nr ||check_add_overflow (min, nr, &end))) +return -EINVAL ; + + tlb_tags.bitmap =bitmap_zalloc (nr,GFP_KERNEL ); +if (!tlb_tags.bitmap) +return -ENOMEM ; tlb_tags.nr = nr; + tlb_tags.min = min;spin_lock_init (&tlb_tags.lock);return 0 ; }
@@ -8229,30 +8231,36 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_destroy_tlb_tags);kvm_tlb_tag_t kvm_alloc_tlb_tag (void ) { -kvm_tlb_tag_t tag; +unsigned int bit;if (WARN_ON_ONCE (!tlb_tags.bitmap))return 0 ;guard (spinlock)(&tlb_tags.lock); - tag =find_first_zero_bit (tlb_tags.bitmap, tlb_tags.nr); -if (tag >= tlb_tags.nr) + bit =find_first_zero_bit (tlb_tags.bitmap, tlb_tags.nr); +if (bit >= tlb_tags.nr)return 0 ; -__set_bit (tag, tlb_tags.bitmap); -return tag; +__set_bit (bit, tlb_tags.bitmap); +return tlb_tags.min + bit; }EXPORT_SYMBOL_FOR_KVM_INTERNAL (kvm_alloc_tlb_tag);void kvm_free_tlb_tag (kvm_tlb_tag_t tag) { -if (!tag ||WARN_ON_ONCE (tag >= tlb_tags.nr)) +unsigned int bit; + +if (!tag ||WARN_ON_ONCE (tag < tlb_tags.min)) +return ; + + bit = tag - tlb_tags.min; +if (WARN_ON_ONCE (bit >= tlb_tags.nr))return ;guard (spinlock)(&tlb_tags.lock); -__clear_bit (tag, tlb_tags.bitmap); +__clear_bit (bit, tlb_tags.bitmap); }EXPORT_SYMBOL_FOR_KVM_INTERNAL (kvm_free_tlb_tag);
arch/x86/kvm/vmx/vmx.h +2 -1
@@ -336,7 +336,8 @@ static __always_inline u32 vmx_get_intr_info(struct kvm_vcpu *vcpu) static __always_inline int init_vpids(void) { -return enable_vpid ?kvm_init_tlb_tags (VMX_NR_VPIDS ) :0 ; +/* Exclude VPID=0 as it is used for the host */ +return enable_vpid ?kvm_init_tlb_tags (1 ,VMX_NR_VPIDS -1 ) :0 ; } static __always_inline void destroy_vpids(void)
Signature · 1 line
2.54.0.1136.gdb2ca164c4-goog
Sean Christopherson
maintainerRe: [RFC PATCH v2 04/25] KVM: x86/mmu: Support specifying a minimum TLB tag
Quoted 36 lines · Yosry Ahmed
In preparation for using the TLB tags allocator for SVM, which has a range of ASIDs allocated for SEV/SNP, pass in a minimum TLB tag when initializing the TLB tags allocator. The bitmap is conceptually shifted such that bit=0 corresponds to tag=min.
Specifying the minimum value during initialization also makes the API clearer, as the passed number of tags becomes the actual number of *usable* tags, and tag=0 is explicitly excluded by the caller.
No functional change intended for VMX as VPID=0 is not used anyway.
Signed-off-by Yosry Ahmed <[email protected]>--- arch/x86/kvm/mmu.h | 2 +- arch/x86/kvm/mmu/mmu.c | 38 +++++++++++++++++++++++--------------- arch/x86/kvm/vmx/vmx.h | 3 ++- 3 files changed, 26 insertions(+), 17 deletions(-)arch/x86/kvm/mmu.h +1
-1@@ -337,7 +337,7 @@ static inline bool kvm_is_gfn_alias(struct kvm *kvm, gfn_t gfn)typedef unsigned int kvm_tlb_tag_t ; -int kvm_init_tlb_tags (unsigned int nr); +int kvm_init_tlb_tags (kvm_tlb_tag_t min,unsigned int nr);void kvm_destroy_tlb_tags (void );kvm_tlb_tag_t kvm_alloc_tlb_tag (void );void kvm_free_tlb_tag (kvm_tlb_tag_t tag);arch/x86/kvm/mmu/mmu.c +0
-0
arch/x86/kvm/mmu/mmu.c static struct { · line 8203
Quoted 8 lines
arch/x86/kvm/mmu/mmu.c +2
-1@@ -8197,24 +8197,26 @@ static struct {spinlock_t lock;unsigned long *bitmap;unsigned int nr; +kvm_tlb_tag_t min; } tlb_tags; -int kvm_init_tlb_tags (unsigned int nr) +int kvm_init_tlb_tags (kvm_tlb_tag_t min,unsigned int nr)
Hmm, I think this code should handle "nr - min" instead of requiring the caller to do so. Because people that know how SEV+ works, but aren't familiar with this code (like I am right now), will probably find it confusing that nothing in here appears to account for the "reserved" ASIDs.
Maybe this? With documentation that reserved ASIDs start at 0, inclusive, and are contiguous.
int kvm_init_tlb_tags(unsigned int nr, unsigned nr_reserved)
{
}And then tlb_tags.min becomes something like tlb_tags.offset (that's a terrible name, but you probably get the gist).
Re: [RFC PATCH v2 04/25] KVM: x86/mmu: Support specifying a minimum TLB tag
Quoted 51 lines · Sean Christopherson
Quoted 36 lines · Yosry Ahmed
In preparation for using the TLB tags allocator for SVM, which has a range of ASIDs allocated for SEV/SNP, pass in a minimum TLB tag when initializing the TLB tags allocator. The bitmap is conceptually shifted such that bit=0 corresponds to tag=min.
Specifying the minimum value during initialization also makes the API clearer, as the passed number of tags becomes the actual number of *usable* tags, and tag=0 is explicitly excluded by the caller.
No functional change intended for VMX as VPID=0 is not used anyway.
Signed-off-by Yosry Ahmed <[email protected]>--- arch/x86/kvm/mmu.h | 2 +- arch/x86/kvm/mmu/mmu.c | 38 +++++++++++++++++++++++--------------- arch/x86/kvm/vmx/vmx.h | 3 ++- 3 files changed, 26 insertions(+), 17 deletions(-)arch/x86/kvm/mmu.h +1
-1@@ -337,7 +337,7 @@ static inline bool kvm_is_gfn_alias(struct kvm *kvm, gfn_t gfn)typedef unsigned int kvm_tlb_tag_t ; -int kvm_init_tlb_tags (unsigned int nr); +int kvm_init_tlb_tags (kvm_tlb_tag_t min,unsigned int nr);void kvm_destroy_tlb_tags (void );kvm_tlb_tag_t kvm_alloc_tlb_tag (void );void kvm_free_tlb_tag (kvm_tlb_tag_t tag);arch/x86/kvm/mmu/mmu.c +0
-0arch/x86/kvm/mmu/mmu.c static struct { · line 8203
Quoted 8 lines
arch/x86/kvm/mmu/mmu.c +2
-1@@ -8197,24 +8197,26 @@ static struct {spinlock_t lock;unsigned long *bitmap;unsigned int nr; +kvm_tlb_tag_t min; } tlb_tags; -int kvm_init_tlb_tags (unsigned int nr) +int kvm_init_tlb_tags (kvm_tlb_tag_t min,unsigned int nr)Hmm, I think this code should handle "nr - min" instead of requiring the caller to do so. Because people that know how SEV+ works, but aren't familiar with this code (like I am right now), will probably find it confusing that nothing in here appears to account for the "reserved" ASIDs.
Quoted 7 lines
Maybe this? With documentation that reserved ASIDs start at 0, inclusive, and are contiguous.
int kvm_init_tlb_tags(unsigned int nr, unsigned nr_reserved) { }
So essentially make the reserved tags included in 'nr', and min becomes nr_reserved. I wanted to keep it generic by not factoring in the assumptions about the reserved tags, but we can revisit later if ever needed (very unlikely).
I am fine with this suggestion.
Quoted 3 lines
And then tlb_tags.min becomes something like tlb_tags.offset (that's a terrible name, but you probably get the gist).
Sean Christopherson
maintainerRe: [RFC PATCH v2 04/25] KVM: x86/mmu: Support specifying a minimum TLB tag
Quoted 55 lines · Yosry Ahmed
Quoted 54 lines · Sean Christopherson
Quoted 36 lines · Yosry Ahmed
In preparation for using the TLB tags allocator for SVM, which has a range of ASIDs allocated for SEV/SNP, pass in a minimum TLB tag when initializing the TLB tags allocator. The bitmap is conceptually shifted such that bit=0 corresponds to tag=min.
Specifying the minimum value during initialization also makes the API clearer, as the passed number of tags becomes the actual number of *usable* tags, and tag=0 is explicitly excluded by the caller.
No functional change intended for VMX as VPID=0 is not used anyway.
Signed-off-by Yosry Ahmed <[email protected]>--- arch/x86/kvm/mmu.h | 2 +- arch/x86/kvm/mmu/mmu.c | 38 +++++++++++++++++++++++--------------- arch/x86/kvm/vmx/vmx.h | 3 ++- 3 files changed, 26 insertions(+), 17 deletions(-)arch/x86/kvm/mmu.h +1
-1@@ -337,7 +337,7 @@ static inline bool kvm_is_gfn_alias(struct kvm *kvm, gfn_t gfn)typedef unsigned int kvm_tlb_tag_t ; -int kvm_init_tlb_tags (unsigned int nr); +int kvm_init_tlb_tags (kvm_tlb_tag_t min,unsigned int nr);void kvm_destroy_tlb_tags (void );kvm_tlb_tag_t kvm_alloc_tlb_tag (void );void kvm_free_tlb_tag (kvm_tlb_tag_t tag);arch/x86/kvm/mmu/mmu.c +0
-0arch/x86/kvm/mmu/mmu.c static struct { · line 8203
Quoted 8 lines
arch/x86/kvm/mmu/mmu.c +2
-1@@ -8197,24 +8197,26 @@ static struct {spinlock_t lock;unsigned long *bitmap;unsigned int nr; +kvm_tlb_tag_t min; } tlb_tags; -int kvm_init_tlb_tags (unsigned int nr) +int kvm_init_tlb_tags (kvm_tlb_tag_t min,unsigned int nr)Hmm, I think this code should handle "nr - min" instead of requiring the caller to do so. Because people that know how SEV+ works, but aren't familiar with this code (like I am right now), will probably find it confusing that nothing in here appears to account for the "reserved" ASIDs.
Maybe this? With documentation that reserved ASIDs start at 0, inclusive, and are contiguous.
Quoted 8 lines
Quoted 4 lines
int kvm_init_tlb_tags(unsigned int nr, unsigned nr_reserved) { }So essentially make the reserved tags included in 'nr', and min becomes
nr_reserved. I wanted to keep it generic by not factoring in the assumptions about the reserved tags,
Eh, by having a @min you've already lost that battle though. It's only "generic" in the sense that it obfuscates why there is a minimum in the first place. E.g. if SEV+ used ASIDs starting that the top and counting backwards, taking only the min wouldn't suffice.
Quoted 1 line
but we can revisit later if ever needed (very unlikely).
RFC PATCH v2 5/25 KVM: SVM: Add helpers to set/clear ASID flush in VMCB
In reply to Yosry AhmedIntroduce set/clear helpers to set tlb_ctl to TLB_CONTROL_FLUSH_ASID or TLB_CONTROL_DO_NOTHING. Incoming changes will eliminate the use of TLB_CONTROL_FLUSH_ALL_ASID except as a fallback when FLUSHBYASID is not available, so the helpers will create a common path for this.
--- arch/x86/kvm/svm/nested.c | 2 +- arch/x86/kvm/svm/sev.c | 2 +- arch/x86/kvm/svm/svm.c | 4 ++-- arch/x86/kvm/svm/svm.h | 10 ++++++++++ 4 files changed, 14 insertions(+), 4 deletions(-)
arch/x86/kvm/svm/nested.c +1 -1
@@ -928,7 +928,7 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm)/* Done at vmrun: asid. */ /* Also overwritten later if necessary. */ - vmcb02->control.tlb_ctl =TLB_CONTROL_DO_NOTHING ; +vmcb_clr_flush_asid (vmcb02);/* Use vmcb01 MMU and format if guest does not use nNPT */ if (nested_npt_enabled (svm)) {
arch/x86/kvm/svm/sev.c +1 -1
@@ -3569,7 +3569,7 @@ int pre_sev_run(struct vcpu_svm *svm, int cpu)return 0 ; sd->sev_vmcbs[asid] = svm->vmcb; - svm->vmcb->control.tlb_ctl =TLB_CONTROL_FLUSH_ASID ; +vmcb_set_flush_asid (svm->vmcb);vmcb_mark_dirty (svm->vmcb,VMCB_ASID );return 0 ; }
arch/x86/kvm/svm/svm.c +2 -2
@@ -4186,7 +4186,7 @@ static void svm_flush_tlb_asid(struct kvm_vcpu *vcpu)* VM-Exit (via kvm_mmu_reset_context()). */ if (static_cpu_has (X86_FEATURE_FLUSHBYASID )) - svm->vmcb->control.tlb_ctl =TLB_CONTROL_FLUSH_ASID ; +vmcb_set_flush_asid (svm->vmcb);else svm->current_vmcb->asid_generation--; }
@@ -4599,7 +4599,7 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags) vcpu->arch.nested_run_pending =0 ; } - svm->vmcb->control.tlb_ctl =TLB_CONTROL_DO_NOTHING ; +vmcb_clr_flush_asid (svm->vmcb);/* * Unconditionally mask off the CLEAR_RAP bit, the AND is just as cheap
arch/x86/kvm/svm/svm.h +10 -0
@@ -476,6 +476,16 @@ static inline void vmcb_set_gpat(struct vmcb *vmcb, u64 data)vmcb_mark_dirty (vmcb,VMCB_NPT ); } +static inline void vmcb_set_flush_asid (struct vmcb *vmcb) +{ + vmcb->control.tlb_ctl =TLB_CONTROL_FLUSH_ASID ; +} + +static inline void vmcb_clr_flush_asid (struct vmcb *vmcb) +{ + vmcb->control.tlb_ctl =TLB_CONTROL_DO_NOTHING ; +} + static __always_inline struct vcpu_svm *to_svm(struct kvm_vcpu *vcpu) { return container_of(vcpu, struct vcpu_svm, vcpu);
Signature · 1 line
2.54.0.1136.gdb2ca164c4-goog
RFC PATCH v2 6/25 KVM: SVM: Fallback to flush everything if FLUSHBYASID is not available
In reply to Yosry AhmedCurrently, if FLUSHBYASID is not available when performing a TLB flush, the fallback is decrementing the ASID generation to trigger allocating a new ASID. In preparation for using a static ASID per vCPU (like VMX), just fallback to flushing everything if FLUSHBYASID is not available. This is probably worse from a performance perspective, but FLUSHBYASID has been around for ~15 years and it's not worth carrying the complexity.
The fallback logic is moved within vmcb_set_flush_asid(), as more callers will be added and will need the fallback as well. The only other current caller is from SEV code, and no CPUs support SEV but not FLUSHBYASID (at least according to AI).
--- arch/x86/kvm/svm/svm.c | 5 +---- arch/x86/kvm/svm/svm.h | 5 ++++- 2 files changed, 5 insertions(+), 5 deletions(-)
arch/x86/kvm/svm/svm.c +1 -4
@@ -4185,10 +4185,7 @@ static void svm_flush_tlb_asid(struct kvm_vcpu *vcpu)* unconditionally does a TLB flush on both nested VM-Enter and nested * VM-Exit (via kvm_mmu_reset_context()). */ -if (static_cpu_has (X86_FEATURE_FLUSHBYASID )) -vmcb_set_flush_asid (svm->vmcb); -else - svm->current_vmcb->asid_generation--; +vmcb_set_flush_asid (svm->vmcb); }static void svm_flush_tlb_current (struct kvm_vcpu *vcpu)
arch/x86/kvm/svm/svm.h +4 -1
@@ -478,7 +478,10 @@ static inline void vmcb_set_gpat(struct vmcb *vmcb, u64 data)static inline void vmcb_set_flush_asid (struct vmcb *vmcb) { - vmcb->control.tlb_ctl =TLB_CONTROL_FLUSH_ASID ; +if (static_cpu_has (X86_FEATURE_FLUSHBYASID )) + vmcb->control.tlb_ctl =TLB_CONTROL_FLUSH_ASID ; +else + vmcb->control.tlb_ctl =TLB_CONTROL_FLUSH_ALL_ASID ; }static inline void vmcb_clr_flush_asid (struct vmcb *vmcb)
Signature · 1 line
2.54.0.1136.gdb2ca164c4-goog
Sean Christopherson
maintainerRe: [RFC PATCH v2 06/25] KVM: SVM: Fallback to flush everything if FLUSHBYASID is not available
Quoted 9 lines · Yosry Ahmed
Currently, if FLUSHBYASID is not available when performing a TLB flush, the fallback is decrementing the ASID generation to trigger allocating a new ASID. In preparation for using a static ASID per vCPU (like VMX), just fallback to flushing everything if FLUSHBYASID is not available. This is probably worse from a performance perspective, but FLUSHBYASID has been around for ~15 years and it's not worth carrying the complexity.
The fallback logic is moved within
vmcb_set_flush_asid(), as more
Phrase this as a command, not a description, i.e. "Move the fallback logic into blah blah blah...".
Quoted 3 lines
callers will be added and will need the fallback as well. The only other current caller is from SEV code, and no CPUs support SEV but not FLUSHBYASID (at least according to AI).
RFC PATCH v2 7/25 KVM: SVM: Duplicate pre-run ASID check for SEV and non-SEV guests
In reply to Yosry AhmedIn preparation for dropping the check for non-SEV VMs, duplicate the check between pre_sev_run() and pre_svm_run() (for the non-SEV code path).
Opportunistically drop the unconditional dirtying of VMCB_ASID, as it's only needed when the ASID is actually updated in the VMCB, not when TLB_CONTROL is updated (as the code currently reads).
No functional change intended.
--- arch/x86/kvm/svm/sev.c | 6 +++++- arch/x86/kvm/svm/svm.c | 9 +++++---- 2 files changed, 10 insertions(+), 5 deletions(-)
arch/x86/kvm/svm/sev.c +5 -1
@@ -3558,6 +3558,11 @@ int pre_sev_run(struct vcpu_svm *svm, int cpu)/* Assign the asid allocated with this SEV guest */ svm->asid = asid; +if (unlikely (svm->asid != svm->vmcb->control.asid)) { + svm->vmcb->control.asid = asid; +vmcb_mark_dirty (svm->vmcb,VMCB_ASID ); + } +/* * Flush guest TLB: *
@@ -3570,7 +3575,6 @@ int pre_sev_run(struct vcpu_svm *svm, int cpu) sd->sev_vmcbs[asid] = svm->vmcb;vmcb_set_flush_asid (svm->vmcb); -vmcb_mark_dirty (svm->vmcb,VMCB_ASID );return 0 ; }
arch/x86/kvm/svm/svm.c +5 -4
@@ -3766,6 +3766,11 @@ static int pre_svm_run(struct kvm_vcpu *vcpu)if (svm->current_vmcb->asid_generation != sd->asid_generation)new_asid (svm, sd); +if (unlikely (svm->asid != svm->vmcb->control.asid)) { + svm->vmcb->control.asid = svm->asid; +vmcb_mark_dirty (svm->vmcb,VMCB_ASID ); + } +return 0 ; }
@@ -4502,10 +4507,6 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags)sync_lapic_to_cr8 (vcpu); -if (unlikely (svm->asid != svm->vmcb->control.asid)) { - svm->vmcb->control.asid = svm->asid; -vmcb_mark_dirty (svm->vmcb,VMCB_ASID ); - } svm->vmcb->save.cr2 = vcpu->arch.cr2; if (guest_cpu_cap_has(vcpu, X86_FEATURE_ERAPS) &&
Signature · 1 line
2.54.0.1136.gdb2ca164c4-goog
RFC PATCH v2 8/25 KVM: SEV: Stop using per-vCPU ASID for SEV VMs
In reply to Yosry Ahmedsvm->asid is only used by SEV in the pre-run check, to update the ASID in the VMCB. Otherwise, svm->asid is only used for non-SEV VMs to keep track of the per-vCPU ASID, as SEV VMs use a per-VM fixed ASID instead.
Initialize the ASID in the VMCB one time in sev_init_vmcb() instead of checking it on every vCPU run, and change the pre-run check to a stopgap with a WARNING.
--- arch/x86/kvm/svm/sev.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
arch/x86/kvm/svm/sev.c +4 -4
@@ -3555,10 +3555,7 @@ int pre_sev_run(struct vcpu_svm *svm, int cpu)if (!cpumask_test_cpu (cpu,to_kvm_sev_info (kvm)->have_run_cpus))cpumask_set_cpu (cpu,to_kvm_sev_info (kvm)->have_run_cpus); -/* Assign the asid allocated with this SEV guest */ - svm->asid = asid; - -if (unlikely (svm->asid != svm->vmcb->control.asid)) { +if (WARN_ON_ONCE (asid != svm->vmcb->control.asid)) { svm->vmcb->control.asid = asid;vmcb_mark_dirty (svm->vmcb,VMCB_ASID ); }
@@ -4763,6 +4760,9 @@ void sev_init_vmcb(struct vcpu_svm *svm, bool init_event) svm->vmcb->control.misc_ctl |=SVM_MISC_ENABLE_SEV ;clr_exception_intercept (svm,UD_VECTOR ); + svm->vmcb->control.asid =sev_get_asid (vcpu->kvm); +vmcb_mark_dirty (svm->vmcb,VMCB_ASID ); +/* * Don't intercept #GP for SEV guests, e.g. for the VMware backdoor, as * KVM can't decrypt guest memory to decode the faulting instruction.
Signature · 1 line
2.54.0.1136.gdb2ca164c4-goog
[email protected]
AI reviewRe: [RFC PATCH v2 08/25] KVM: SEV: Stop using per-vCPU ASID for SEV VMs
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] A malicious SEV guest or an unprivileged host-userspace process can trigger a host WARN_ON_ONCE by…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] A malicious SEV guest or an unprivileged host-userspace process can trigger a host WARN_ON_ONCE by executing VMRUN (entering nested virtualization) or via KVM_SET_NESTED_STATE. --
commit c3ac1dbd3331e2991b6bd808b207ce3395a73130
Author: Yosry Ahmed <[email protected]>
KVM: SEV: Stop using per-vCPU ASID for SEV VMs
This commit updates KVM to initialize the SEV VM ASID once in sev_init_vmcb() instead of checking and setting it on every vCPU run. It replaces the run-time assignment check in pre_sev_run() with a warning.
arch/x86/kvm/svm/sev.c int pre_sev_run(struct vcpu_svm *svm, int cpu) · line 3558
Quoted 12 lines
arch/x86/kvm/svm/sev.c +1
-4@@ -3555,10 +3555,7 @@ int pre_sev_run(struct vcpu_svm *svm, int cpu)if (!cpumask_test_cpu (cpu,to_kvm_sev_info (kvm)->have_run_cpus))cpumask_set_cpu (cpu,to_kvm_sev_info (kvm)->have_run_cpus); -/* Assign the asid allocated with this SEV guest */ - svm->asid = asid; - -if (unlikely (svm->asid != svm->vmcb->control.asid)) { +if (WARN_ON_ONCE (asid != svm->vmcb->control.asid)) {
[Severity: High]
Can a malicious SEV guest or an unprivileged host-userspace process trigger this warning by entering nested virtualization or via KVM_SET_NESTED_STATE?
KVM does not strictly prohibit userspace from exposing CPUID.SVM to an SEV guest. If an SEV guest sets EFER.SVME and executes VMRUN, KVM intercepts it and handles the nested transition via nested_svm_vmrun(), which switches the active VMCB pointer to the nested vmcb02.
The nested vmcb02 does not inherit the SEV ASID assignment from sev_init_vmcb(), and its control.asid is initialized separately. During the subsequent svm_vcpu_run() -> pre_sev_run() call, the VM's valid SEV ASID will no longer match the nested vmcb02's ASID, causing this warning to trigger.
Since this condition is guest-reachable and controllable by userspace, does this create a host denial of service vector when panic_on_warn is set?
Quoted 3 lines
arch/x86/kvm/svm/sev.c +0
-0svm->vmcb->control.asid = asid;vmcb_mark_dirty (svm->vmcb,VMCB_ASID ); }
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=8
Re: [RFC PATCH v2 08/25] KVM: SEV: Stop using per-vCPU ASID for SEV VMs
Quoted 36 lines · On Mon, Jun 15, 2026 at 6:06 PM <[email protected]> wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] A malicious SEV guest or an unprivileged host-userspace process can trigger a hostWARN_ON_ONCEby executing VMRUN (entering nested virtualization) or viaKVM_SET_NESTED_STATE. --commit
c3ac1dbd3331e2991b6bd808b207ce3395a73130
Author: Yosry Ahmed <[email protected]>KVM: SEV: Stop using per-vCPU ASID for SEV VMs
This commit updates KVM to initialize the SEV VM ASID once in
sev_init_vmcb()instead of checking and setting it on every vCPU run. It replaces the run-time assignment check inpre_sev_run()with a warning.arch/x86/kvm/svm/sev.c int pre_sev_run(struct vcpu_svm *svm, int cpu) · line 3558
Quoted 12 lines
arch/x86/kvm/svm/sev.c +1
-4@@ -3555,10 +3555,7 @@ int pre_sev_run(struct vcpu_svm *svm, int cpu)if (!cpumask_test_cpu (cpu,to_kvm_sev_info (kvm)->have_run_cpus))cpumask_set_cpu (cpu,to_kvm_sev_info (kvm)->have_run_cpus); -/* Assign the asid allocated with this SEV guest */ - svm->asid = asid; - -if (unlikely (svm->asid != svm->vmcb->control.asid)) { +if (WARN_ON_ONCE (asid != svm->vmcb->control.asid)) {[Severity: High]
Can a malicious SEV guest or an unprivileged host-userspace process trigger this warning by entering nested virtualization or viaKVM_SET_NESTED_STATE?KVM does not strictly prohibit userspace from exposing CPUID.SVM to an SEV guest. If an SEV guest sets EFER.SVME and executes VMRUN, KVM intercepts it and handles the nested transition via
nested_svm_vmrun(), which switches the active VMCB pointer to the nested vmcb02.
Quoted 4 lines
The nested vmcb02 does not inherit the SEV ASID assignment from
sev_init_vmcb(), and its control.asid is initialized separately. During the subsequentsvm_vcpu_run()->pre_sev_run()call, the VM's valid SEV ASID will no longer match the nested vmcb02's ASID, causing this warning to trigger.
Hmm I looked around the code and I can't see what prevents SEV guests from setting EFER_SVME or using KVM_SET_NESTED_STATE. However, I think a VMRUN from an SEV VM will fail, at least due to not being able to read the vmcb12 from guest memory?
That being said, I can drop this stopgap if preferred, I was going to drop it before sending but I left it in just to be cautious.
Quoted 10 lines
Since this condition is guest-reachable and controllable by userspace, does this create a host denial of service vector when
panic_on_warnis set?arch/x86/kvm/svm/sev.c int pre_sev_run(struct vcpu_svm *svm, int cpu) · line 3561
Quoted 3 lines
arch/x86/kvm/svm/sev.c +0
-0svm->vmcb->control.asid = asid;vmcb_mark_dirty (svm->vmcb,VMCB_ASID ); }--
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=8
Re: [RFC PATCH v2 08/25] KVM: SEV: Stop using per-vCPU ASID for SEV VMs
Quoted 43 lines · Yosry Ahmed
Quoted 36 lines · On Mon, Jun 15, 2026 at 6:06 PM <[email protected]> wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] A malicious SEV guest or an unprivileged host-userspace process can trigger a hostWARN_ON_ONCEby executing VMRUN (entering nested virtualization) or viaKVM_SET_NESTED_STATE. --commit
c3ac1dbd3331e2991b6bd808b207ce3395a73130
Author: Yosry Ahmed <[email protected]>KVM: SEV: Stop using per-vCPU ASID for SEV VMs
This commit updates KVM to initialize the SEV VM ASID once in
sev_init_vmcb()instead of checking and setting it on every vCPU run. It replaces the run-time assignment check inpre_sev_run()with a warning.arch/x86/kvm/svm/sev.c int pre_sev_run(struct vcpu_svm *svm, int cpu) · line 3558
Quoted 12 lines
arch/x86/kvm/svm/sev.c +1
-4@@ -3555,10 +3555,7 @@ int pre_sev_run(struct vcpu_svm *svm, int cpu)if (!cpumask_test_cpu (cpu,to_kvm_sev_info (kvm)->have_run_cpus))cpumask_set_cpu (cpu,to_kvm_sev_info (kvm)->have_run_cpus); -/* Assign the asid allocated with this SEV guest */ - svm->asid = asid; - -if (unlikely (svm->asid != svm->vmcb->control.asid)) { +if (WARN_ON_ONCE (asid != svm->vmcb->control.asid)) {[Severity: High]
Can a malicious SEV guest or an unprivileged host-userspace process trigger this warning by entering nested virtualization or viaKVM_SET_NESTED_STATE?KVM does not strictly prohibit userspace from exposing CPUID.SVM to an SEV guest. If an SEV guest sets EFER.SVME and executes VMRUN, KVM intercepts it and handles the nested transition via
nested_svm_vmrun(), which switches the active VMCB pointer to the nested vmcb02.Quoted 4 lines
The nested vmcb02 does not inherit the SEV ASID assignment from
sev_init_vmcb(), and its control.asid is initialized separately. During the subsequentsvm_vcpu_run()->pre_sev_run()call, the VM's valid SEV ASID will no longer match the nested vmcb02's ASID, causing this warning to trigger.
Quoted 7 lines
Hmm I looked around the code and I can't see what prevents SEV guests from setting
EFER_SVMEor usingKVM_SET_NESTED_STATE. However, I think a VMRUN from an SEV VM will fail, at least due to not being able to read the vmcb12 from guest memory?That being said, I can drop this stopgap if preferred, I was going to drop it before sending but I left it in just to be cautious.
From an offline discussion with Sean, seems like it's technically possible to actually run nested in SEV. So I need to actually handle this in this series, by making sure L2 uses the same ASID as L1 (required for SEV), and always flushing that ASID on nested transitions. I will also drop this stopgap, given that it's likely that using the wrong ASID will result in a loud failure anyway.
Sean Christopherson
maintainerRe: [RFC PATCH v2 08/25] KVM: SEV: Stop using per-vCPU ASID for SEV VMs
Quoted 34 lines · Yosry Ahmed
Quoted 33 lines · Yosry Ahmed
Quoted 22 lines · On Mon, Jun 15, 2026 at 6:06 PM <[email protected]> wrote:
arch/x86/kvm/svm/sev.c int pre_sev_run(struct vcpu_svm *svm, int cpu) · line 3558
Quoted 12 lines
arch/x86/kvm/svm/sev.c +1
-4@@ -3555,10 +3555,7 @@ int pre_sev_run(struct vcpu_svm *svm, int cpu)if (!cpumask_test_cpu (cpu,to_kvm_sev_info (kvm)->have_run_cpus))cpumask_set_cpu (cpu,to_kvm_sev_info (kvm)->have_run_cpus); -/* Assign the asid allocated with this SEV guest */ - svm->asid = asid; - -if (unlikely (svm->asid != svm->vmcb->control.asid)) { +if (WARN_ON_ONCE (asid != svm->vmcb->control.asid)) {[Severity: High]
Can a malicious SEV guest or an unprivileged host-userspace process trigger this warning by entering nested virtualization or viaKVM_SET_NESTED_STATE?KVM does not strictly prohibit userspace from exposing CPUID.SVM to an SEV guest. If an SEV guest sets EFER.SVME and executes VMRUN, KVM intercepts it and handles the nested transition via
nested_svm_vmrun(), which switches the active VMCB pointer to the nested vmcb02.Quoted 4 lines
The nested vmcb02 does not inherit the SEV ASID assignment from
sev_init_vmcb(), and its control.asid is initialized separately. During the subsequentsvm_vcpu_run()->pre_sev_run()call, the VM's valid SEV ASID will no longer match the nested vmcb02's ASID, causing this warning to trigger.Hmm I looked around the code and I can't see what prevents SEV guests from setting
EFER_SVMEor usingKVM_SET_NESTED_STATE. However, I think a VMRUN from an SEV VM will fail, at least due to not being able to read the vmcb12 from guest memory?
Quoted 5 lines
Quoted 2 lines
That being said, I can drop this stopgap if preferred, I was going to drop it before sending but I left it in just to be cautious.
From an offline discussion with Sean, seems like it's technically possible to actually run nested in SEV.
It is, past me actually got it working (I was curious). :-)
Quoted 4 lines
So I need to actually handle this in this series, by making sure L2 uses the same ASID as L1 (required for SEV), and always flushing that ASID on nested transitions. I will also drop this stopgap, given that it's likely that using the wrong ASID will result in a loud failure anyway.
FWIW, "need" is debatable. If supporting nested+SEV adds a lot of complexity, I'd probably be ok dropping support for that, e.g. by failing VMRUN with a software error. But if it's roughly the same complexity overall, I'd probably vote to allow it, mainly to avoid making semi-arbitrary support/policy decisions in KVM.
Re: [RFC PATCH v2 08/25] KVM: SEV: Stop using per-vCPU ASID for SEV VMs
Quoted 48 lines · Sean Christopherson
Quoted 34 lines · Yosry Ahmed
Quoted 33 lines · Yosry Ahmed
Quoted 22 lines · On Mon, Jun 15, 2026 at 6:06 PM <[email protected]> wrote:
arch/x86/kvm/svm/sev.c int pre_sev_run(struct vcpu_svm *svm, int cpu) · line 3558
Quoted 12 lines
arch/x86/kvm/svm/sev.c +1
-4@@ -3555,10 +3555,7 @@ int pre_sev_run(struct vcpu_svm *svm, int cpu)if (!cpumask_test_cpu (cpu,to_kvm_sev_info (kvm)->have_run_cpus))cpumask_set_cpu (cpu,to_kvm_sev_info (kvm)->have_run_cpus); -/* Assign the asid allocated with this SEV guest */ - svm->asid = asid; - -if (unlikely (svm->asid != svm->vmcb->control.asid)) { +if (WARN_ON_ONCE (asid != svm->vmcb->control.asid)) {[Severity: High]
Can a malicious SEV guest or an unprivileged host-userspace process trigger this warning by entering nested virtualization or viaKVM_SET_NESTED_STATE?KVM does not strictly prohibit userspace from exposing CPUID.SVM to an SEV guest. If an SEV guest sets EFER.SVME and executes VMRUN, KVM intercepts it and handles the nested transition via
nested_svm_vmrun(), which switches the active VMCB pointer to the nested vmcb02.Quoted 4 lines
The nested vmcb02 does not inherit the SEV ASID assignment from
sev_init_vmcb(), and its control.asid is initialized separately. During the subsequentsvm_vcpu_run()->pre_sev_run()call, the VM's valid SEV ASID will no longer match the nested vmcb02's ASID, causing this warning to trigger.Hmm I looked around the code and I can't see what prevents SEV guests from setting
EFER_SVMEor usingKVM_SET_NESTED_STATE. However, I think a VMRUN from an SEV VM will fail, at least due to not being able to read the vmcb12 from guest memory?Quoted 5 lines
Quoted 2 lines
That being said, I can drop this stopgap if preferred, I was going to drop it before sending but I left it in just to be cautious.
From an offline discussion with Sean, seems like it's technically possible to actually run nested in SEV.
It is, past me actually got it working (I was curious). :-)
Quoted 4 lines
So I need to actually handle this in this series, by making sure L2 uses the same ASID as L1 (required for SEV), and always flushing that ASID on nested transitions. I will also drop this stopgap, given that it's likely that using the wrong ASID will result in a loud failure anyway.
Quoted 4 lines
FWIW, "need" is debatable. If supporting nested+SEV adds a lot of complexity, I'd probably be ok dropping support for that, e.g. by failing VMRUN with a software error. But if it's roughly the same complexity overall, I'd probably vote to allow it, mainly to avoid making semi-arbitrary support/policy decisions in KVM.
It's mostly fine. Actually I dropped this patch in the next version and kept using svm->asid for SEV (but I moved its initialization to VMCB init). With that, handling SEV TLB flushes boils down to flushing on nested transitions if svm->asid == svm->nested.asid02, which we need to do anyway in case both L1 and L2 are using the fallback ASID.
RFC PATCH v2 9/25 KVM: SVM: Use a static ASID per vCPU
In reply to Yosry AhmedSwitch from dynamic ASID allocation to a static per-vCPU ASID. The ASID dynamic ASID allocation logic is now only effectively used when running out of ASIDs on a CPU (uncommon on modern hardware), or switching CPUs.
The per-CPU ASID generation is initialized to 1, and the per-VMCB ASID generation is initialized to 0. This leads to a TLB flush of the ASID on the first VMRUN, allocating a new ASID, and pumping the per-VMCB generation to 1 (matching the per-CPU generation). The ASID remains static until either:
- The vCPU runs on a new CPU, in which case KVM resets the per-VMCB generation to allocate a new ASID on the new CPU.
- KVM hits the maximum ASID on a CPU and increments the per-CPU ASID generation, at which point all vCPUs on this CPU will allocate a new ASID.
Drop the complexity and make the ASID static for each vCPU. This makes SVM's handling of ASIDs closer to VMX's handling of VPIDs, and simplifies the code. It also completely avoids full TLB flushes (i.e. TLB_CONTROL_FLUSH_ALL_ASID) on systems with FLUSHBYASID. Full flushes previously happened when updating the generation, so not so common, but should generally be avoided as they cause a VMRUN to invalidate the TLB entries for other VMs as well as the host.
When a vCPU is migrated to a new physical CPU, flush the (now static) ASID instead of allocating a new one. This might cause extra TLB flushes when switching CPUs (compared to just using a new ASID), but odds are that the TLB is cold on the new CPU anyway.
As using ASIDs cannot be disabled like VPIDs, allocate a fallback ASID to be shared by all vCPUs after running out of ASIDs (and flushed on all VMRUNs), in the very unlikely scenario that more than 32K vCPUs are active at the same time (as ASIDs are recycled when vCPUs are deleted). Add allocate_asid() and free_asid() wrappers to handle the fallback ASID logic, as it will be reused for nested in following changes.
Note #1, svm->asid can technically be dropped in favor of directly using svm->vmcb->control.asid. However, that would require allocating the ASID in init_vmcb(), and there is no corresponding cleanup function to free the ASID. That would also require avoiding reallocation of a new ASID during a vCPU reset. Opt for simplicity and keep svm->asid, which is allocated and free in the vCPU creation/freeing paths like VMX VPIDs.
Note #2, a nice side-effect is reading the min/max ASIDs once during initialization, instead of once per-CPU.
--- arch/x86/kvm/svm/nested.c | 4 +-- arch/x86/kvm/svm/svm.c | 68 +++++++++++++++++++++------------------ arch/x86/kvm/svm/svm.h | 22 +++++++++---- 3 files changed, 53 insertions(+), 41 deletions(-)
arch/x86/kvm/svm/nested.c +2 -2
@@ -925,9 +925,9 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm)else vmcb02->control.bus_lock_counter =0 ; -/* Done at vmrun: asid. */ + vmcb02->control.asid = vmcb01->control.asid; -/* Also overwritten later if necessary. */ +/* Overwritten later if necessary. */ vmcb_clr_flush_asid (vmcb02);/* Use vmcb01 MMU and format if guest does not use nNPT */
arch/x86/kvm/svm/svm.c +36 -32
139 lines not on this page · raw message
arch/x86/kvm/svm/svm.h +15 -7
@@ -26,6 +26,7 @@#include "regs.h" #include "x86.h" #include "pmu.h" +#include "mmu.h" /* * Helpers to convert to/from physical addresses for pages whose address is
@@ -143,7 +144,6 @@ struct kvm_vmcb_info {struct vmcb *ptr;unsigned long pa;int cpu; -uint64_t asid_generation; };struct vmcb_save_area_cached {
@@ -282,7 +282,7 @@ struct vcpu_svm {struct vmcb *vmcb;struct kvm_vmcb_info vmcb01;struct kvm_vmcb_info *current_vmcb; -u32 asid; +kvm_tlb_tag_t asid;u32 sysenter_esp_hi;u32 sysenter_eip_hi;uint64_t tsc_aux;
@@ -369,11 +369,6 @@ struct vcpu_svm { };struct svm_cpu_data { -u64 asid_generation; -u32 max_asid; -u32 next_asid; -u32 min_asid; -bool bp_spec_reduce_set;struct vmcb *save_area;
@@ -476,6 +471,19 @@ static inline void vmcb_set_gpat(struct vmcb *vmcb, u64 data)vmcb_mark_dirty (vmcb,VMCB_NPT ); } +extern kvm_tlb_tag_t fallback_asid; + +static inline kvm_tlb_tag_t allocate_asid (void ) +{ +return kvm_alloc_tlb_tag () ?: fallback_asid; +} + +static inline void free_asid (kvm_tlb_tag_t asid) +{ +if (likely (asid != fallback_asid)) +kvm_free_tlb_tag (asid); +} +static inline void vmcb_set_flush_asid (struct vmcb *vmcb) {if (static_cpu_has (X86_FEATURE_FLUSHBYASID ))
Signature · 1 line
2.54.0.1136.gdb2ca164c4-goog
[email protected]
AI reviewRe: [RFC PATCH v2 09/25] KVM: SVM: Use a static ASID per vCPU
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] KVM fails to load on systems where all guest ASIDs are assigned to SEV (`nr_asids == max_sev_asid…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] KVM fails to load on systems where all guest ASIDs are assigned to SEV (nr_asids == max_sev_asid + 1), breaking SEV-only environments. --
commit 6e1ef9a91f738606e7d8f9e36c1659d383c52061
Author: Yosry Ahmed <[email protected]>
KVM: SVM: Use a static ASID per vCPU
This patch switches KVM SVM from dynamic ASID allocation to a static per-vCPU ASID to avoid full TLB flushes and simplify the code. It introduces a fallback ASID logic and relies on TLB tag tracking for the pool of available ASIDs.
Quoted 4 lines
arch/x86/kvm/svm/svm.c +0
-0
[ ... ]
arch/x86/kvm/svm/svm.c static __init int svm_hardware_setup(void) · line 5750
Quoted 8 lines
arch/x86/kvm/svm/svm.c +4
-0@@ -5751,6 +5744,17 @@ static __init int svm_hardware_setup(void) kvm_caps.inapplicable_quirks &= ~KVM_X86_QUIRK_CD_NW_CLEARED ; +/* Consumes max_sev_asid initialized by sev_hardware_setup() */ + min_asid = max_sev_asid +1 ; + nr_asids =cpuid_ebx (SVM_CPUID_FUNC ); + r =kvm_init_tlb_tags (min_asid, nr_asids - min_asid);
[Severity: High]
Will this cause KVM to fail loading on systems where the BIOS configures all available guest ASIDs for SEV?
If max_sev_asid equals nr_asids - 1 (a common configuration for cloud servers dedicated to Confidential VMs), then min_asid will equal nr_asids. This makes nr_asids - min_asid evaluate to 0.
When kvm_init_tlb_tags() is called with a count of 0, it returns -EINVAL, which causes svm_hardware_setup() to fail and prevents the KVM module from loading entirely. Prior to this patch, KVM would successfully load in this configuration and allow SEV guests to run normally.
Quoted 10 lines
arch/x86/kvm/svm/svm.c +7
-0+if (r) + goto err; + + fallback_asid =kvm_alloc_tlb_tag (); +if (!fallback_asid) + goto err; +for_each_possible_cpu (cpu) { r =svm_cpu_init (cpu);if (r)
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=9
Re: [RFC PATCH v2 09/25] KVM: SVM: Use a static ASID per vCPU
Quoted 35 lines · On Mon, Jun 15, 2026 at 6:08 PM <[email protected]> wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] KVM fails to load on systems where all guest ASIDs are assigned to SEV (nr_asids == max_sev_asid + 1), breaking SEV-only environments. --commit
6e1ef9a91f738606e7d8f9e36c1659d383c52061
Author: Yosry Ahmed <[email protected]>KVM: SVM: Use a static ASID per vCPU
This patch switches KVM SVM from dynamic ASID allocation to a static per-vCPU ASID to avoid full TLB flushes and simplify the code. It introduces a fallback ASID logic and relies on TLB tag tracking for the pool of available ASIDs.
Quoted 4 lines
arch/x86/kvm/svm/svm.c +0
-0[ ... ]
arch/x86/kvm/svm/svm.c static __init int svm_hardware_setup(void) · line 5750
Quoted 8 lines
arch/x86/kvm/svm/svm.c +4
-0@@ -5751,6 +5744,17 @@ static __init int svm_hardware_setup(void) kvm_caps.inapplicable_quirks &= ~KVM_X86_QUIRK_CD_NW_CLEARED ; +/* Consumes max_sev_asid initialized by sev_hardware_setup() */ + min_asid = max_sev_asid +1 ; + nr_asids =cpuid_ebx (SVM_CPUID_FUNC ); + r =kvm_init_tlb_tags (min_asid, nr_asids - min_asid);[Severity: High]
Will this cause KVM to fail loading on systems where the BIOS configures all available guest ASIDs for SEV?
Quoted 8 lines
If
max_sev_asidequalsnr_asids- 1 (a common configuration for cloud servers dedicated to Confidential VMs), thenmin_asidwill equalnr_asids. This makesnr_asids-min_asidevaluate to 0.When
kvm_init_tlb_tags()is called with a count of 0, it returns-EINVAL, which causessvm_hardware_setup()to fail and prevents the KVM module from loading entirely. Prior to this patch, KVM would successfully load in this configuration and allow SEV guests to run normally.
Is this a realistic scenario?
I can make it such that kvm_init_tlb_tags() doesn't fail with 0 ASIDs, and that failures to allocate the fallback_asid are also allowed. This will allow configuring all the ASIDs for SEV, and then allocate_asid() will fail vCPU creation for any non-ASID VMs. But I am not sure if we need to actually handle this.
Looking at the current code, I think initialization will succeed if all ASIDs are allocated to SEV, but then new_asid() will allocate an ASID larger than the max supported ASID. Not sure if that will fail a VMRUN, or if the higher bits will be ignored.
Quoted 14 lines
arch/x86/kvm/svm/svm.c static __init int svm_hardware_setup(void) · line 5760
Quoted 11 lines
arch/x86/kvm/svm/svm.c +7
-0+if (r) + goto err; + + fallback_asid =kvm_alloc_tlb_tag (); +if (!fallback_asid) + goto err; +for_each_possible_cpu (cpu) { r =svm_cpu_init (cpu);if (r)--
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=9
Re: [RFC PATCH v2 09/25] KVM: SVM: Use a static ASID per vCPU
Quoted 46 lines · Yosry Ahmed
Quoted 35 lines · On Mon, Jun 15, 2026 at 6:08 PM <[email protected]> wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] KVM fails to load on systems where all guest ASIDs are assigned to SEV (nr_asids == max_sev_asid + 1), breaking SEV-only environments. --commit
6e1ef9a91f738606e7d8f9e36c1659d383c52061
Author: Yosry Ahmed <[email protected]>KVM: SVM: Use a static ASID per vCPU
This patch switches KVM SVM from dynamic ASID allocation to a static per-vCPU ASID to avoid full TLB flushes and simplify the code. It introduces a fallback ASID logic and relies on TLB tag tracking for the pool of available ASIDs.
Quoted 4 lines
arch/x86/kvm/svm/svm.c +0
-0[ ... ]
arch/x86/kvm/svm/svm.c static __init int svm_hardware_setup(void) · line 5750
Quoted 8 lines
arch/x86/kvm/svm/svm.c +4
-0@@ -5751,6 +5744,17 @@ static __init int svm_hardware_setup(void) kvm_caps.inapplicable_quirks &= ~KVM_X86_QUIRK_CD_NW_CLEARED ; +/* Consumes max_sev_asid initialized by sev_hardware_setup() */ + min_asid = max_sev_asid +1 ; + nr_asids =cpuid_ebx (SVM_CPUID_FUNC ); + r =kvm_init_tlb_tags (min_asid, nr_asids - min_asid);[Severity: High]
Will this cause KVM to fail loading on systems where the BIOS configures all available guest ASIDs for SEV?Quoted 8 lines
If
max_sev_asidequalsnr_asids- 1 (a common configuration for cloud servers dedicated to Confidential VMs), thenmin_asidwill equalnr_asids. This makesnr_asids-min_asidevaluate to 0.When
kvm_init_tlb_tags()is called with a count of 0, it returns-EINVAL, which causessvm_hardware_setup()to fail and prevents the KVM module from loading entirely. Prior to this patch, KVM would successfully load in this configuration and allow SEV guests to run normally.
Quoted 7 lines
Is this a realistic scenario?
I can make it such that
kvm_init_tlb_tags()doesn't fail with 0 ASIDs, and that failures to allocate thefallback_asidare also allowed. This will allow configuring all the ASIDs for SEV, and thenallocate_asid()will fail vCPU creation for any non-ASID VMs. But I am not sure if we need to actually handle this.
I will do this for the next version, it shouldn't add significant complexity.
RFC PATCH v2 10/25 KVM: nSVM: Add a placeholder ASID for L2
In reply to Yosry AhmedIn preparation for introducing a separate ASID for L2, introduce a 'placeholder' ASID that is still the same as L1's ASID. This will facilitate future changes that need to distinguish L1 and L2's ASIDs, before actually using a different ASID for L1 and L2.
No functional change intended.
--- arch/x86/kvm/svm/nested.c | 5 +++-- arch/x86/kvm/svm/svm.h | 2 ++ 2 files changed, 5 insertions(+), 2 deletions(-)
arch/x86/kvm/svm/nested.c +3 -2
@@ -698,7 +698,6 @@ static void nested_svm_transition_tlb_flush(struct kvm_vcpu *vcpu)* - Honor L1's request to flush an ASID on nested VMRUN * - Sync nested NPT MMU on VMRUN that flushes L2's ASID[*] * - Don't crush a pending TLB flush in vmcb02 on nested VMRUN -* - Flush L1's ASID on KVM_REQ_TLB_FLUSH_GUEST * * [*] Unlike nested EPT, SVM's ASID management can invalidate nested * NPT guest-physical mappings on VMRUN.
@@ -925,7 +924,7 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm)else vmcb02->control.bus_lock_counter =0 ; - vmcb02->control.asid = vmcb01->control.asid; + vmcb02->control.asid = svm->nested.asid02;/* Overwritten later if necessary. */ vmcb_clr_flush_asid (vmcb02);
@@ -1495,6 +1494,8 @@ int svm_allocate_nested(struct vcpu_svm *svm)if (!svm->nested.msrpm) goto err_free_vmcb02; + svm->nested.asid02 = svm->asid; + svm->nested.initialized = true;return 0 ;
arch/x86/kvm/svm/svm.h +2 -0
@@ -232,6 +232,8 @@ struct svm_nested_state {*/ struct vmcb_save_area_cached save; +kvm_tlb_tag_t asid02; +bool initialized;/*
Signature · 1 line
2.54.0.1136.gdb2ca164c4-goog
RFC PATCH v2 11/25 KVM: x86: hyper-v: Rename kvm_hv_vcpu_purge_flush_tlb()
In reply to Yosry AhmedRename kvm_hv_vcpu_purge_flush_tlb() to kvm_hv_purge_tlb_flush_fifo() to clarify that it purges the TLB flush FIFO, not purge *and* flush the TLB.
No functional change intended.
--- arch/x86/kvm/hyperv.h | 4 ++-- arch/x86/kvm/svm/svm.c | 2 +- arch/x86/kvm/x86.c | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-)
arch/x86/kvm/hyperv.h +2 -2
@@ -204,7 +204,7 @@ static inline struct kvm_vcpu_hv_tlb_flush_fifo *kvm_hv_get_tlb_flush_fifo(strucreturn &hv_vcpu->tlb_flush_fifo[i]; } -static inline void kvm_hv_vcpu_purge_flush_tlb (struct kvm_vcpu *vcpu) +static inline void kvm_hv_purge_tlb_flush_fifo (struct kvm_vcpu *vcpu) {struct kvm_vcpu_hv_tlb_flush_fifo *tlb_flush_fifo;
@@ -286,7 +286,7 @@ static inline int kvm_hv_hypercall(struct kvm_vcpu *vcpu) {return HV_STATUS_ACCESS_DENIED ; } -static inline void kvm_hv_vcpu_purge_flush_tlb (struct kvm_vcpu *vcpu) {} +static inline void kvm_hv_purge_tlb_flush_fifo (struct kvm_vcpu *vcpu) {}static inline bool kvm_hv_synic_has_vector (struct kvm_vcpu *vcpu,int vector) {return false;
arch/x86/kvm/svm/svm.c +1 -1
@@ -4173,7 +4173,7 @@ static void svm_flush_tlb_asid(struct kvm_vcpu *vcpu)* A TLB flush for the current ASID flushes both "host" and "guest" TLB * entries, and thus is a superset of Hyper-V's fine grained flushing. */ -kvm_hv_vcpu_purge_flush_tlb (vcpu); +kvm_hv_purge_tlb_flush_fifo (vcpu);/* * Flush only the current ASID even if the TLB flush was invoked via
arch/x86/kvm/x86.c +1 -1
@@ -3690,7 +3690,7 @@ static void kvm_vcpu_flush_tlb_guest(struct kvm_vcpu *vcpu)* Flushing all "guest" TLB is always a superset of Hyper-V's fine * grained flushing. */ -kvm_hv_vcpu_purge_flush_tlb (vcpu); +kvm_hv_purge_tlb_flush_fifo (vcpu); }
Signature · 1 line
2.54.0.1136.gdb2ca164c4-goog
RFC PATCH v2 12/25 KVM: x86: hyper-v: Allow puring all TLB flush FIFOs
In reply to Yosry AhmedRefactor kvm_hv_purge_tlb_flush_fifo() to introduce an inner helper, __kvm_hv_purge_tlb_flush_fifo(), parameterized by an optional FIFO. If a FIFO is not passed in, all FIFOs are purged.
Note that KVM_REQ_HV_TLB_FLUSH is consumed by __kvm_hv_purge_tlb_flush_fifo(), so a selective purge on one FIFO will render any subsequent non-selective purge useless. This is not a problem because purging TLB flush FIFOs is an optimization (and there are currently no users of the non-selective purges).
No functional change intended as a FIFO is always being passed in now.
--- arch/x86/kvm/hyperv.h | 25 ++++++++++++++++++++----- 1 file changed, 20 insertions(+), 5 deletions(-)
arch/x86/kvm/hyperv.h +20 -5
@@ -204,16 +204,31 @@ static inline struct kvm_vcpu_hv_tlb_flush_fifo *kvm_hv_get_tlb_flush_fifo(strucreturn &hv_vcpu->tlb_flush_fifo[i]; } -static inline void kvm_hv_purge_tlb_flush_fifo (struct kvm_vcpu *vcpu) +/* Purge pending TLB flushes in @fifo, or in all FIFOs if @fifo is NULL */ +static inline void __kvm_hv_purge_tlb_flush_fifo (struct kvm_vcpu *vcpu, +struct kvm_vcpu_hv_tlb_flush_fifo *fifo) { -struct kvm_vcpu_hv_tlb_flush_fifo *tlb_flush_fifo; +struct kvm_vcpu_hv *hv_vcpu =to_hv_vcpu (vcpu); +int i; -if (!to_hv_vcpu (vcpu) || !kvm_check_request (KVM_REQ_HV_TLB_FLUSH , vcpu)) +if (!hv_vcpu || !kvm_check_request (KVM_REQ_HV_TLB_FLUSH , vcpu))return ; - tlb_flush_fifo =kvm_hv_get_tlb_flush_fifo (vcpu,is_guest_mode (vcpu)); +for (i =0 ; i <ARRAY_SIZE (hv_vcpu->tlb_flush_fifo); i++) { +if (!fifo || fifo == &hv_vcpu->tlb_flush_fifo[i]) +kfifo_reset_out (&hv_vcpu->tlb_flush_fifo[i].entries); + } +} + +static inline void kvm_hv_purge_tlb_flush_fifo (struct kvm_vcpu *vcpu) +{ +struct kvm_vcpu_hv_tlb_flush_fifo *fifo; + +if (!to_hv_vcpu (vcpu)) +return ; -kfifo_reset_out (&tlb_flush_fifo->entries); + fifo =kvm_hv_get_tlb_flush_fifo (vcpu,is_guest_mode (vcpu)); +__kvm_hv_purge_tlb_flush_fifo (vcpu, fifo); }static inline bool guest_hv_cpuid_has_l2_tlb_flush (struct kvm_vcpu *vcpu)
Signature · 1 line
2.54.0.1136.gdb2ca164c4-goog
RFC PATCH v2 13/25 KVM: nSVM: Flush both L1 and L2 ASIDs on KVM_REQ_TLB_FLUSH
In reply to Yosry AhmedFlush both L1 and L2 ASIDs in svm_flush_tlb_all() to appropriately handle KVM_REQ_TLB_FLUSH by flushing all TLB entries in all contexts (e.g. for kvm_flush_remote_tlbs()). Since both L1 and L2 currently share an ASID, this is effectively a noop, but it won't be once L2 has a separate ASID.
Purge all Hyper-V TLB FIFOs (for both L1 and L2), since both ASIDs are flushed, and an ASID flush is a superset of Hyper-V's fine-grained flushing (see comment in svm_flush_tlb_asid()).
Note that if one TLB flush FIFO is purged (e.g. as a result of KVM_REQ_TLB_FLUSH_CURRENT), it will consume KVM_REQ_HV_TLB_FLUSH, and a subsequent KVM_REQ_TLB_FLUSH will not flush the other FIFO. This is alright as flushing both FIFOs is a (newly introduced) optimization anyway. The other FIFO will be checked after a nested transition, as KVM_REQ_HV_TLB_FLUSH is always set on nested transitions.
--- arch/x86/kvm/hyperv.h | 6 ++++++ arch/x86/kvm/svm/nested.c | 1 - arch/x86/kvm/svm/svm.c | 8 +++++++- 3 files changed, 13 insertions(+), 2 deletions(-)
arch/x86/kvm/hyperv.h +6 -0
@@ -231,6 +231,11 @@ static inline void kvm_hv_purge_tlb_flush_fifo(struct kvm_vcpu *vcpu)__kvm_hv_purge_tlb_flush_fifo (vcpu, fifo); } +static inline void kvm_hv_purge_all_tlb_flush_fifos (struct kvm_vcpu *vcpu) +{ +__kvm_hv_purge_tlb_flush_fifo (vcpu,NULL ); +} +static inline bool guest_hv_cpuid_has_l2_tlb_flush (struct kvm_vcpu *vcpu) {struct kvm_vcpu_hv *hv_vcpu =to_hv_vcpu (vcpu);
@@ -302,6 +307,7 @@ static inline int kvm_hv_hypercall(struct kvm_vcpu *vcpu)return HV_STATUS_ACCESS_DENIED ; }static inline void kvm_hv_purge_tlb_flush_fifo (struct kvm_vcpu *vcpu) {} +static inline void kvm_hv_purge_all_tlb_flush_fifos (struct kvm_vcpu *vcpu) {}static inline bool kvm_hv_synic_has_vector (struct kvm_vcpu *vcpu,int vector) {return false;
arch/x86/kvm/svm/nested.c +0 -1
@@ -694,7 +694,6 @@ static void nested_svm_transition_tlb_flush(struct kvm_vcpu *vcpu)* TODO: optimize unconditional TLB flush/MMU sync. A partial list of * things to fix before this can be conditional: * -* - Flush TLBs for both L1 and L2 remote TLB flush * - Honor L1's request to flush an ASID on nested VMRUN * - Sync nested NPT MMU on VMRUN that flushes L2's ASID[*] * - Don't crush a pending TLB flush in vmcb02 on nested VMRUN
arch/x86/kvm/svm/svm.c +7 -1
@@ -4203,6 +4203,8 @@ static void svm_flush_tlb_current(struct kvm_vcpu *vcpu)static void svm_flush_tlb_all (struct kvm_vcpu *vcpu) { +struct vcpu_svm *svm =to_svm (vcpu); +/* * When running on Hyper-V with EnlightenedNptTlb enabled, remote TLB * flushes should be routed to hv_flush_remote_tlbs() without requesting
@@ -4213,7 +4215,11 @@ static void svm_flush_tlb_all(struct kvm_vcpu *vcpu)if (WARN_ON_ONCE (svm_hv_is_enlightened_tlb_enabled (vcpu)))hv_flush_remote_tlbs (vcpu->kvm); -svm_flush_tlb_asid (vcpu); +kvm_hv_purge_all_tlb_flush_fifos (vcpu); + +vmcb_set_flush_asid (svm->vmcb01.ptr); +if (svm->nested.vmcb02.ptr) +vmcb_set_flush_asid (svm->nested.vmcb02.ptr); }static void svm_flush_tlb_gva (struct kvm_vcpu *vcpu,gva_t gva)
Signature · 1 line
2.54.0.1136.gdb2ca164c4-goog
[email protected]
AI reviewRe: [RFC PATCH v2 13/25] KVM: nSVM: Flush both L1 and L2 ASIDs on KVM_REQ_TLB_FLUSH
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] Dereferencing `svm->nested.vmcb02.ptr` without verifying `svm->nested.initialized` causes a…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Dereferencing svm->nested.vmcb02.ptr without verifying svm->nested.initialized causes a Use-After-Free write if svm_allocate_nested() previously failed. --
commit 88a69b7a032d088ddb452c91fba161d26dbed0ec
Author: Yosry Ahmed <[email protected]>
KVM: nSVM: Flush both L1 and L2 ASIDs on KVM_REQ_TLB_FLUSH
This commit modifies svm_flush_tlb_all() to flush both L1 and L2 ASIDs to properly handle KVM_REQ_TLB_FLUSH across all contexts. It also purges all Hyper-V TLB FIFOs for both L1 and L2.
Quoted 4 lines
arch/x86/kvm/svm/svm.c +0
-0
[ ... ]
arch/x86/kvm/svm/svm.c static void svm_flush_tlb_all(struct kvm_vcpu *vcpu) · line 4222
Quoted 10 lines
arch/x86/kvm/svm/svm.c +5
-1@@ -4213,7 +4215,11 @@ static void svm_flush_tlb_all(struct kvm_vcpu *vcpu)if (WARN_ON_ONCE (svm_hv_is_enlightened_tlb_enabled (vcpu)))hv_flush_remote_tlbs (vcpu->kvm); -svm_flush_tlb_asid (vcpu); +kvm_hv_purge_all_tlb_flush_fifos (vcpu); + +vmcb_set_flush_asid (svm->vmcb01.ptr); +if (svm->nested.vmcb02.ptr) +vmcb_set_flush_asid (svm->nested.vmcb02.ptr);
[Severity: High]
Could this dereference of svm->nested.vmcb02.ptr cause a use-after-free?
If svm_allocate_nested() successfully allocates vmcb02_page and assigns it to svm->nested.vmcb02.ptr, but a subsequent allocation (such as msrpm) fails, the error path frees the page but does not clear the pointer:
arch/x86/kvm/svm/nested.c:svm_allocate_nested() {
...
err_free_vmcb02:
__free_page(vmcb02_page);
return -ENOMEM;
}If a TLB flush is later triggered and calls svm_flush_tlb_all(), the check for if (svm->nested.vmcb02.ptr) would evaluate to true on the dangling pointer, and vmcb_set_flush_asid() would write to freed memory.
Would it be safer to check svm->nested.initialized here instead, to ensure the nested state is fully allocated and valid?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=13
Re: [RFC PATCH v2 13/25] KVM: nSVM: Flush both L1 and L2 ASIDs on KVM_REQ_TLB_FLUSH
Quoted 44 lines · On Mon, Jun 15, 2026 at 6:05 PM <[email protected]> wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Dereferencingsvm->nested.vmcb02.ptrwithout verifyingsvm->nested.initializedcauses a Use-After-Free write ifsvm_allocate_nested()previously failed. --commit
88a69b7a032d088ddb452c91fba161d26dbed0ec
Author: Yosry Ahmed <[email protected]>KVM: nSVM: Flush both L1 and L2 ASIDs on
KVM_REQ_TLB_FLUSHThis commit modifies
svm_flush_tlb_all()to flush both L1 and L2 ASIDs to properly handleKVM_REQ_TLB_FLUSHacross all contexts. It also purges all Hyper-V TLB FIFOs for both L1 and L2.Quoted 4 lines
arch/x86/kvm/svm/svm.c +0
-0[ ... ]
arch/x86/kvm/svm/svm.c static void svm_flush_tlb_all(struct kvm_vcpu *vcpu) · line 4222
Quoted 10 lines
arch/x86/kvm/svm/svm.c +5
-1@@ -4213,7 +4215,11 @@ static void svm_flush_tlb_all(struct kvm_vcpu *vcpu)if (WARN_ON_ONCE (svm_hv_is_enlightened_tlb_enabled (vcpu)))hv_flush_remote_tlbs (vcpu->kvm); -svm_flush_tlb_asid (vcpu); +kvm_hv_purge_all_tlb_flush_fifos (vcpu); + +vmcb_set_flush_asid (svm->vmcb01.ptr); +if (svm->nested.vmcb02.ptr) +vmcb_set_flush_asid (svm->nested.vmcb02.ptr);[Severity: High]
Could this dereference ofsvm->nested.vmcb02.ptrcause a use-after-free?If
svm_allocate_nested()successfully allocatesvmcb02_pageand assigns it tosvm->nested.vmcb02.ptr, but a subsequent allocation (such as msrpm) fails, the error path frees the page but does not clear the pointer:arch/x86/kvm/svm/nested.c:svm_allocate_nested() { ... err_free_vmcb02: __free_page(vmcb02_page); return -ENOMEM; }
Quoted 6 lines
If a TLB flush is later triggered and calls
svm_flush_tlb_all(), the check for if (svm->nested.vmcb02.ptr) would evaluate to true on the dangling pointer, andvmcb_set_flush_asid()would write to freed memory.Would it be safer to check
svm->nested.initializedhere instead, to ensure the nested state is fully allocated and valid?
Yes. But if no one objects, I'd rather add a patch to drop svm->nested.initialized completely, and properly NULLify svm->nested.vmcb02.ptr in the destruction paths. Keying off the pointer makes more sense semantically imo.
Sean Christopherson
maintainerRe: [RFC PATCH v2 13/25] KVM: nSVM: Flush both L1 and L2 ASIDs on KVM_REQ_TLB_FLUSH
Quoted 35 lines · Yosry Ahmed
Quoted 34 lines · On Mon, Jun 15, 2026 at 6:05 PM <[email protected]> wrote:
Quoted 4 lines
arch/x86/kvm/svm/svm.c +0
-0[ ... ]
arch/x86/kvm/svm/svm.c static void svm_flush_tlb_all(struct kvm_vcpu *vcpu) · line 4222
Quoted 10 lines
arch/x86/kvm/svm/svm.c +5
-1@@ -4213,7 +4215,11 @@ static void svm_flush_tlb_all(struct kvm_vcpu *vcpu)if (WARN_ON_ONCE (svm_hv_is_enlightened_tlb_enabled (vcpu)))hv_flush_remote_tlbs (vcpu->kvm); -svm_flush_tlb_asid (vcpu); +kvm_hv_purge_all_tlb_flush_fifos (vcpu); + +vmcb_set_flush_asid (svm->vmcb01.ptr); +if (svm->nested.vmcb02.ptr) +vmcb_set_flush_asid (svm->nested.vmcb02.ptr);[Severity: High]
Could this dereference ofsvm->nested.vmcb02.ptrcause a use-after-free?If
svm_allocate_nested()successfully allocatesvmcb02_pageand assigns it tosvm->nested.vmcb02.ptr, but a subsequent allocation (such as msrpm) fails, the error path frees the page but does not clear the pointer:arch/x86/kvm/svm/nested.c:svm_allocate_nested() { ... err_free_vmcb02: __free_page(vmcb02_page); return -ENOMEM; }If a TLB flush is later triggered and calls
svm_flush_tlb_all(), the check for if (svm->nested.vmcb02.ptr) would evaluate to true on the dangling pointer, andvmcb_set_flush_asid()would write to freed memory.
Quoted 7 lines
Quoted 2 lines
Would it be safer to check
svm->nested.initializedhere instead, to ensure the nested state is fully allocated and valid?Yes. But if no one objects, I'd rather add a patch to drop
svm->nested.initializedcompletely, and properly NULLifysvm->nested.vmcb02.ptrin the destruction paths. Keying off the pointer makes more sense semantically imo.
+1
RFC PATCH v2 14/25 KVM: nSVM: Move svm_switch_vmcb() to nested.c
In reply to Yosry Ahmedsvm_switch_vmcb() is only used outside of nested.c to initialize svm->current_vmcb and svm->vmcb. Open-code the initialization (which is arguably not a "switch" to begin with) in the vCPU creation path, and move svm_switch_vmcb() to nested.c in preparation for adding more nested logic.
No fcuntional change intended.
--- arch/x86/kvm/svm/nested.c | 6 ++++++ arch/x86/kvm/svm/svm.c | 10 +++------- arch/x86/kvm/svm/svm.h | 2 -- 3 files changed, 9 insertions(+), 9 deletions(-)
arch/x86/kvm/svm/nested.c +6 -0
@@ -705,6 +705,12 @@ static void nested_svm_transition_tlb_flush(struct kvm_vcpu *vcpu)kvm_make_request (KVM_REQ_TLB_FLUSH_CURRENT , vcpu); } +static void svm_switch_vmcb (struct vcpu_svm *svm,struct kvm_vmcb_info *target_vmcb) +{ + svm->current_vmcb = target_vmcb; + svm->vmcb = target_vmcb->ptr; +} +/* * Load guest's/host's cr3 on nested vmentry or vmexit. @nested_npt is true * if we are emulating VM-Entry into a guest with NPT enabled.
arch/x86/kvm/svm/svm.c +3 -7
@@ -1300,12 +1300,6 @@ static void svm_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event)__svm_vcpu_reset (vcpu); } -void svm_switch_vmcb (struct vcpu_svm *svm,struct kvm_vmcb_info *target_vmcb) -{ - svm->current_vmcb = target_vmcb; - svm->vmcb = target_vmcb->ptr; -} -static int svm_vcpu_precreate (struct kvm *kvm) {return avic_alloc_physical_id_table (kvm);
@@ -1353,7 +1347,9 @@ static int svm_vcpu_create(struct kvm_vcpu *vcpu) svm->vmcb01.ptr =page_address (vmcb01_page); svm->vmcb01.pa =__sme_set (page_to_pfn (vmcb01_page) <<PAGE_SHIFT ); -svm_switch_vmcb (svm, &svm->vmcb01); + + svm->current_vmcb = &svm->vmcb01; + svm->vmcb = svm->vmcb01.ptr; svm->guest_state_loaded = false;
arch/x86/kvm/svm/svm.h +0 -2
@@ -922,8 +922,6 @@ void nested_copy_vmcb_control_to_cache(struct vcpu_svm *svm,void nested_copy_vmcb_save_to_cache (struct vcpu_svm *svm,struct vmcb_save_area *save);void nested_sync_control_from_vmcb02 (struct vcpu_svm *svm); -void svm_switch_vmcb (struct vcpu_svm *svm,struct kvm_vmcb_info *target_vmcb); -static inline void __svm_pmu_handle_nested_transition (struct vcpu_svm *svm,bool defer)
Signature · 1 line
2.54.0.1136.gdb2ca164c4-goog
RFC PATCH v2 15/25 KVM: nSVM: Call nested_svm_transition_tlb_flush() on every VMCB switch
In reply to Yosry AhmedMove the calls to nested_svm_transition_tlb_flush() in different transition code paths to svm_switch_vmcb(). This ensures that TLB flushes are not missed during switches, and makes it clearer that the TLB flushes are directly related to the active VMCB.
The ordering currently doesn't matter, but as more TLB handling is added for nested, requesting the TLB flushes *after* the VMCB switch will become a requirement.
No functional change intended (for now).
--- arch/x86/kvm/svm/nested.c | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-)
arch/x86/kvm/svm/nested.c +4 -6
@@ -707,8 +707,12 @@ static void nested_svm_transition_tlb_flush(struct kvm_vcpu *vcpu)static void svm_switch_vmcb (struct vcpu_svm *svm,struct kvm_vmcb_info *target_vmcb) { +struct kvm_vcpu *vcpu = &svm->vcpu; + svm->current_vmcb = target_vmcb; svm->vmcb = target_vmcb->ptr; + +nested_svm_transition_tlb_flush (vcpu); }/*
@@ -860,8 +864,6 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm)struct vmcb *vmcb01 = svm->vmcb01.ptr;struct kvm_vcpu *vcpu = &svm->vcpu; -nested_svm_transition_tlb_flush (vcpu); -/* Enter Guest-Mode */ enter_guest_mode (vcpu);svm_pmu_handle_nested_transition (svm);
@@ -1435,8 +1437,6 @@ void nested_svm_vmexit(struct vcpu_svm *svm) svm->vcpu.arch.dr7 =DR7_FIXED_1 ;kvm_update_dr7 (&svm->vcpu); -nested_svm_transition_tlb_flush (vcpu); -nested_svm_uninit_mmu_context (vcpu);if (nested_svm_load_cr3 (vcpu, vmcb01->save.cr3, false, true))
@@ -1556,8 +1556,6 @@ void svm_leave_nested(struct kvm_vcpu *vcpu)svm_switch_vmcb (svm, &svm->vmcb01); -nested_svm_transition_tlb_flush (vcpu); -nested_svm_uninit_mmu_context (vcpu);vmcb_mark_all_dirty (svm->vmcb);
Signature · 1 line
2.54.0.1136.gdb2ca164c4-goog
Sean Christopherson
maintainerRe: [RFC PATCH v2 15/25] KVM: nSVM: Call nested_svm_transition_tlb_flush() on every VMCB switch
Quoted 21 lines · Yosry Ahmed
Move the calls to
nested_svm_transition_tlb_flush()in different transition code paths tosvm_switch_vmcb(). This ensures that TLB flushes are not missed during switches, and makes it clearer that the TLB flushes are directly related to the active VMCB.The ordering currently doesn't matter, but as more TLB handling is added for nested, requesting the TLB flushes *after* the VMCB switch will become a requirement.
No functional change intended (for now).
Signed-off-by Yosry Ahmed <[email protected]>--- arch/x86/kvm/svm/nested.c | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-)arch/x86/kvm/svm/nested.c +0
-0
arch/x86/kvm/svm/nested.c static void nested_svm_transition_tlb_flush(struct kvm_vcpu *vcpu) · line 715
Quoted 9 lines
arch/x86/kvm/svm/nested.c +4
-0@@ -707,8 +707,12 @@ static void nested_svm_transition_tlb_flush(struct kvm_vcpu *vcpu)static void svm_switch_vmcb (struct vcpu_svm *svm,struct kvm_vmcb_info *target_vmcb) { +struct kvm_vcpu *vcpu = &svm->vcpu; + svm->current_vmcb = target_vmcb; svm->vmcb = target_vmcb->ptr; + +nested_svm_transition_tlb_flush (vcpu);
I'm not convinced this is the right place to do the flushing. Yes, manually doing the flushes requires more code, and arguably makes KVM less robust, but IMO there's value in mimicking architectural flows because it's the architecture that dictates the flushing.
Re: [RFC PATCH v2 15/25] KVM: nSVM: Call nested_svm_transition_tlb_flush() on every VMCB switch
Quoted 23 lines · Sean Christopherson
Quoted 22 lines · Yosry Ahmed
Move the calls to
nested_svm_transition_tlb_flush()in different transition code paths tosvm_switch_vmcb(). This ensures that TLB flushes are not missed during switches, and makes it clearer that the TLB flushes are directly related to the active VMCB.The ordering currently doesn't matter, but as more TLB handling is added for nested, requesting the TLB flushes *after* the VMCB switch will become a requirement.
No functional change intended (for now).
Signed-off-by Yosry Ahmed <[email protected]>--- arch/x86/kvm/svm/nested.c | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-)arch/x86/kvm/svm/nested.c +0
-0@@ -707,8 +707,12 @@ static void nested_svm_transition_tlb_flush(struct kvm_vcpu *vcpu)
Quoted 13 lines
arch/x86/kvm/svm/nested.c static void nested_svm_transition_tlb_flush(struct kvm_vcpu *vcpu) · line 715
Quoted 8 lines
arch/x86/kvm/svm/nested.c +4
-0static void svm_switch_vmcb (struct vcpu_svm *svm,struct kvm_vmcb_info *target_vmcb) { +struct kvm_vcpu *vcpu = &svm->vcpu; + svm->current_vmcb = target_vmcb; svm->vmcb = target_vmcb->ptr; + +nested_svm_transition_tlb_flush (vcpu);I'm not convinced this is the right place to do the flushing. Yes, manually doing the flushes requires more code, and arguably makes KVM less robust, but IMO there's value in mimicking architectural flows because it's the architecture that dictates the flushing.
Responded to the same comment in patch 17.
RFC PATCH v2 16/25 KVM: nSVM: Split nested_svm_transition_tlb_flush() into entry/exit fns
In reply to Yosry AhmedThe handling for the entry and exit TLB flushes will diverge significantly in the following changes. Instead of adding an 'is_vmenter' argument like nested_vmx_transition_tlb_flush(), just split the function into two variants for 'entry' and 'exit'.
No functional change intended.
--- arch/x86/kvm/svm/nested.c | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-)
arch/x86/kvm/svm/nested.c +13 -2
@@ -685,7 +685,7 @@ static void nested_save_pending_event_to_vmcb12(struct vcpu_svm *svm, vmcb12->control.exit_int_info = exit_int_info; } -static void nested_svm_transition_tlb_flush (struct kvm_vcpu *vcpu) +static void nested_svm_entry_tlb_flush (struct kvm_vcpu *vcpu) {/* Handle pending Hyper-V TLB flush requests */ kvm_hv_nested_transtion_tlb_flush (vcpu, npt_enabled);
@@ -705,6 +705,14 @@ static void nested_svm_transition_tlb_flush(struct kvm_vcpu *vcpu)kvm_make_request (KVM_REQ_TLB_FLUSH_CURRENT , vcpu); } +static void nested_svm_exit_tlb_flush (struct kvm_vcpu *vcpu) +{ +kvm_hv_nested_transtion_tlb_flush (vcpu, npt_enabled); + +kvm_make_request (KVM_REQ_MMU_SYNC , vcpu); +kvm_make_request (KVM_REQ_TLB_FLUSH_CURRENT , vcpu); +} +static void svm_switch_vmcb (struct vcpu_svm *svm,struct kvm_vmcb_info *target_vmcb) {struct kvm_vcpu *vcpu = &svm->vcpu;
@@ -712,7 +720,10 @@ static void svm_switch_vmcb(struct vcpu_svm *svm, struct kvm_vmcb_info *target_v svm->current_vmcb = target_vmcb; svm->vmcb = target_vmcb->ptr; -nested_svm_transition_tlb_flush (vcpu); +if (target_vmcb == &svm->nested.vmcb02) +nested_svm_entry_tlb_flush (vcpu); +else +nested_svm_exit_tlb_flush (vcpu); }/*
Signature · 1 line
2.54.0.1136.gdb2ca164c4-goog
RFC PATCH v2 17/25 KVM: nSVM: Service local TLB flushes before nested transitions
In reply to Yosry AhmedKVM does not track TLB flush requests for L1 vs. L2. Hence, service local flush that target the current context before switching to a new one. Since TLB flushes are performed through the VMCB's TLB_CONTROL field, service the flushes before every VMCB switch.
Note that nested_svm_{entry/exit}_tlb_flush() must be called after kvm_service_local_tlb_flush_requests(), otherwise the TLB flushes will be immediately serviced in the "old" VMCB rather than the new one.
This is conceptually similar to how nVMX calls kvm_service_local_tlb_flush_requests() with a few differences:
1. VMX performs TLB flushes through INVVPID or INVEPT. The VPID is determined based on guest mode, and the EPT pointer is determined based on the active MMU. Hence, local TLB flushes are serviced before switching guest mode and switching the MMU. On the other hand, SVM performs TLB flushes by updating the VMCB, hence local TLB flushes are serviced before switching the VMCB.
2. VMX has a single code path for entering guest mode (i.e. nested_vmx_enter_non_root_mode()) and a single code path for exiting guest mode (i.e. nested_vmx_vmexit()). Other code paths like vmx_set_nested_state() and vmx_leave_nested() call into these functions. On the other hand, SVM open codes the switches in several places, so call kvm_service_local_tlb_flush_requests() from svm_switch_svm() to more-or-less guarantee it is not missed.
--- arch/x86/kvm/svm/nested.c | 7 +++++++ 1 file changed, 7 insertions(+)
arch/x86/kvm/svm/nested.c +7 -0
@@ -717,9 +717,16 @@ static void svm_switch_vmcb(struct vcpu_svm *svm, struct kvm_vmcb_info *target_v {struct kvm_vcpu *vcpu = &svm->vcpu; +/* +* TLB flushes are applied to the VMCB, so apply any pending TLB flushes +* on the current VMCB before switching to a new one.. +*/ +kvm_service_local_tlb_flush_requests (vcpu); + svm->current_vmcb = target_vmcb; svm->vmcb = target_vmcb->ptr; +/* .. then request TLB flushes needed for the new VMCB */ if (target_vmcb == &svm->nested.vmcb02)nested_svm_entry_tlb_flush (vcpu);else
Signature · 1 line
2.54.0.1136.gdb2ca164c4-goog
[email protected]
AI reviewRe: [RFC PATCH v2 17/25] KVM: nSVM: Service local TLB flushes before nested transitions
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] Context mismatch in `svm_switch_vmcb()` incorrectly applies global TLB flushes to the outgoing…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Context mismatch in svm_switch_vmcb() incorrectly applies global TLB flushes to the outgoing VMCB and clears the request bit, causing the incoming VMCB to miss the flush. Furthermore, on nested VM-Exit, leave_guest_mode() is called before svm_switch_vmcb(), causing kvm_hv_purge_tlb_flush_fifo() to incorrectly reset L1's Hyper-V TLB flush FIFO while hardware flushes are applied to L2's vmcb02. --
commit 5fb35fcca4bb6e1756cc67fdc65ffe16ab6c194f
Author: Yosry Ahmed <[email protected]>
KVM: nSVM: Service local TLB flushes before nested transitions
This patch updates nSVM to service local TLB flush requests before switching VMCBs during nested transitions. It aims to ensure flushes are applied to the correct context, similar to how nVMX handles them.
arch/x86/kvm/svm/nested.c static void svm_switch_vmcb(struct vcpu_svm *svm, struct kvm_vmcb_info *target_v · line 724
Quoted 13 lines
arch/x86/kvm/svm/nested.c +5
-0@@ -717,9 +717,16 @@ static void svm_switch_vmcb(struct vcpu_svm *svm, struct kvm_vmcb_info *target_v {struct kvm_vcpu *vcpu = &svm->vcpu; +/* +* TLB flushes are applied to the VMCB, so apply any pending TLB flushes +* on the current VMCB before switching to a new one.. +*/ +kvm_service_local_tlb_flush_requests (vcpu);
[Severity: High]
Does this clear global TLB flush requests (like KVM_REQ_TLB_FLUSH_GUEST) while applying them only to the outgoing VMCB?
If a host-initiated flush occurs (e.g., from page reclaim), the request bit will be consumed and applied to the outgoing VMCB, leaving the incoming VMCB to miss the flush and run with a stale TLB.
Also, is there a context mismatch here during nested VM-Exit?
In nested_svm_vmexit(), leave_guest_mode(vcpu) is called before svm_switch_vmcb(svm, &svm->vmcb01).
Because of this, kvm_service_local_tlb_flush_requests() will see is_guest_mode(vcpu) as false. If Hyper-V is enabled, this means kvm_hv_purge_tlb_flush_fifo() will incorrectly target L1's FIFO while the hardware flushes are actually being applied to L2's vmcb02.
Quoted 3 lines
arch/x86/kvm/svm/nested.c +1
-0+ svm->current_vmcb = target_vmcb; svm->vmcb = target_vmcb->ptr;
[ ... ]
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=17
Re: [RFC PATCH v2 17/25] KVM: nSVM: Service local TLB flushes before nested transitions
Quoted 32 lines · On Mon, Jun 15, 2026 at 6:20 PM <[email protected]> wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Context mismatch insvm_switch_vmcb()incorrectly applies global TLB flushes to the outgoing VMCB and clears the request bit, causing the incoming VMCB to miss the flush. Furthermore, on nested VM-Exit,leave_guest_mode()is called beforesvm_switch_vmcb(), causingkvm_hv_purge_tlb_flush_fifo()to incorrectly reset L1's Hyper-V TLB flush FIFO while hardware flushes are applied to L2'svmcb02. --commit
5fb35fcca4bb6e1756cc67fdc65ffe16ab6c194f
Author: Yosry Ahmed <[email protected]>KVM: nSVM: Service local TLB flushes before nested transitions
This patch updates nSVM to service local TLB flush requests before switching VMCBs during nested transitions. It aims to ensure flushes are applied to the correct context, similar to how nVMX handles them.
arch/x86/kvm/svm/nested.c static void svm_switch_vmcb(struct vcpu_svm *svm, struct kvm_vmcb_info *target_v · line 724
Quoted 13 lines
arch/x86/kvm/svm/nested.c +5
-0@@ -717,9 +717,16 @@ static void svm_switch_vmcb(struct vcpu_svm *svm, struct kvm_vmcb_info *target_v {struct kvm_vcpu *vcpu = &svm->vcpu; +/* +* TLB flushes are applied to the VMCB, so apply any pending TLB flushes +* on the current VMCB before switching to a new one.. +*/ +kvm_service_local_tlb_flush_requests (vcpu);[Severity: High]
Does this clear global TLB flush requests (likeKVM_REQ_TLB_FLUSH_GUEST) while applying them only to the outgoing VMCB?
Quoted 3 lines
If a host-initiated flush occurs (e.g., from page reclaim), the request bit will be consumed and applied to the outgoing VMCB, leaving the incoming VMCB to miss the flush and run with a stale TLB.
Yeah it is intentional to apply the flushes to the current/old/outgoing context. If there are flushes needed in the new context, they are queued by nested_svm_{entry/exit}_tlb_flush(). That's similar to what VMX does.
Quoted 10 lines
Also, is there a context mismatch here during nested VM-Exit?
In
nested_svm_vmexit(),leave_guest_mode(vcpu)is called beforesvm_switch_vmcb(svm, &svm->vmcb01).Because of this,
kvm_service_local_tlb_flush_requests()will seeis_guest_mode(vcpu)as false. If Hyper-V is enabled, this meanskvm_hv_purge_tlb_flush_fifo()will incorrectly target L1's FIFO while the hardware flushes are actually being applied to L2's vmcb02.
Ugh.. yes. This is annoying. kvm_service_local_tlb_flush_requests() needs to be called on both the current/old/outgoing VMCB *and* guest mode. So we'll need to open-code the call in a bunch of places before svm_switch_vmcb() and {enter/leave}_guest_mode(). I really liked putting it in svm_switch_vmcb() together with nested_svm_{entry/exit}_tlb_flush() so that all the TLB flushing logic for nested transitions live in one place and the ordering needs to be handled in one place.
I guess there's already a lot of ordering dependencies in all code entering/leaving guest mode. We really should refactor the code, if possible, to have one helper for entering nested and one helper for leaving nested, but I guess that's outside the scope of this series.
Re: [RFC PATCH v2 17/25] KVM: nSVM: Service local TLB flushes before nested transitions
Quoted 54 lines · Yosry Ahmed
Quoted 32 lines · On Mon, Jun 15, 2026 at 6:20 PM <[email protected]> wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Context mismatch insvm_switch_vmcb()incorrectly applies global TLB flushes to the outgoing VMCB and clears the request bit, causing the incoming VMCB to miss the flush. Furthermore, on nested VM-Exit,leave_guest_mode()is called beforesvm_switch_vmcb(), causingkvm_hv_purge_tlb_flush_fifo()to incorrectly reset L1's Hyper-V TLB flush FIFO while hardware flushes are applied to L2'svmcb02. --commit
5fb35fcca4bb6e1756cc67fdc65ffe16ab6c194f
Author: Yosry Ahmed <[email protected]>KVM: nSVM: Service local TLB flushes before nested transitions
This patch updates nSVM to service local TLB flush requests before switching VMCBs during nested transitions. It aims to ensure flushes are applied to the correct context, similar to how nVMX handles them.
arch/x86/kvm/svm/nested.c static void svm_switch_vmcb(struct vcpu_svm *svm, struct kvm_vmcb_info *target_v · line 724
Quoted 13 lines
arch/x86/kvm/svm/nested.c +5
-0@@ -717,9 +717,16 @@ static void svm_switch_vmcb(struct vcpu_svm *svm, struct kvm_vmcb_info *target_v {struct kvm_vcpu *vcpu = &svm->vcpu; +/* +* TLB flushes are applied to the VMCB, so apply any pending TLB flushes +* on the current VMCB before switching to a new one.. +*/ +kvm_service_local_tlb_flush_requests (vcpu);[Severity: High]
Does this clear global TLB flush requests (likeKVM_REQ_TLB_FLUSH_GUEST) while applying them only to the outgoing VMCB?Quoted 3 lines
If a host-initiated flush occurs (e.g., from page reclaim), the request bit will be consumed and applied to the outgoing VMCB, leaving the incoming VMCB to miss the flush and run with a stale TLB.
Yeah it is intentional to apply the flushes to the current/old/outgoing context. If there are flushes needed in the new context, they are queued by nested_svm_{entry/exit}_tlb_flush(). That's similar to what VMX does.
Quoted 10 lines
Also, is there a context mismatch here during nested VM-Exit?
In
nested_svm_vmexit(),leave_guest_mode(vcpu)is called beforesvm_switch_vmcb(svm, &svm->vmcb01).Because of this,
kvm_service_local_tlb_flush_requests()will seeis_guest_mode(vcpu)as false. If Hyper-V is enabled, this meanskvm_hv_purge_tlb_flush_fifo()will incorrectly target L1's FIFO while the hardware flushes are actually being applied to L2's vmcb02.
Quoted 8 lines
Ugh.. yes. This is annoying.
kvm_service_local_tlb_flush_requests()needs to be called on both the current/old/outgoing VMCB *and* guest mode. So we'll need to open-code the call in a bunch of places beforesvm_switch_vmcb()and {enter/leave}_guest_mode(). I really liked putting it insvm_switch_vmcb()together with nested_svm_{entry/exit}_tlb_flush() so that all the TLB flushing logic for nested transitions live in one place and the ordering needs to be handled in one place.
Maybe we can just re-order the code to always call svm_switch_vmcb() before {enter/leave}_guest_mode(). We already do that on the entry side, and seems to be straightforward on the exit side.
With that, we can keep kvm_service_local_tlb_flush_requests() and nested_svm_{entry/exit}_tlb_flush() inside svm_switch_vmcb(). We can add a WARNING to kvm_service_local_tlb_flush_requests() to make sure the calls remains in the correct context (i.e. VMCB matches guest mode).
Sean Christopherson
maintainerRe: [RFC PATCH v2 17/25] KVM: nSVM: Service local TLB flushes before nested transitions
Quoted 48 lines · Yosry Ahmed
Quoted 38 lines · Yosry Ahmed
Quoted 18 lines
arch/x86/kvm/svm/nested.c static void svm_switch_vmcb(struct vcpu_svm *svm, struct kvm_vmcb_info *target_v · line 724
Quoted 13 lines
arch/x86/kvm/svm/nested.c +5
-0@@ -717,9 +717,16 @@ static void svm_switch_vmcb(struct vcpu_svm *svm, struct kvm_vmcb_info *target_v {struct kvm_vcpu *vcpu = &svm->vcpu; +/* +* TLB flushes are applied to the VMCB, so apply any pending TLB flushes +* on the current VMCB before switching to a new one.. +*/ +kvm_service_local_tlb_flush_requests (vcpu);[Severity: High]
Does this clear global TLB flush requests (likeKVM_REQ_TLB_FLUSH_GUEST) while applying them only to the outgoing VMCB?Quoted 3 lines
If a host-initiated flush occurs (e.g., from page reclaim), the request bit will be consumed and applied to the outgoing VMCB, leaving the incoming VMCB to miss the flush and run with a stale TLB.
Yeah it is intentional to apply the flushes to the current/old/outgoing context. If there are flushes needed in the new context, they are queued by nested_svm_{entry/exit}_tlb_flush(). That's similar to what VMX does.
Quoted 10 lines
Also, is there a context mismatch here during nested VM-Exit?
In
nested_svm_vmexit(),leave_guest_mode(vcpu)is called beforesvm_switch_vmcb(svm, &svm->vmcb01).Because of this,
kvm_service_local_tlb_flush_requests()will seeis_guest_mode(vcpu)as false. If Hyper-V is enabled, this meanskvm_hv_purge_tlb_flush_fifo()will incorrectly target L1's FIFO while the hardware flushes are actually being applied to L2's vmcb02.Quoted 8 lines
Ugh.. yes. This is annoying.
kvm_service_local_tlb_flush_requests()needs to be called on both the current/old/outgoing VMCB *and* guest mode. So we'll need to open-code the call in a bunch of places beforesvm_switch_vmcb()and {enter/leave}_guest_mode(). I really liked putting it insvm_switch_vmcb()together with nested_svm_{entry/exit}_tlb_flush() so that all the TLB flushing logic for nested transitions live in one place and the ordering needs to be handled in one place.
Quoted 3 lines
Maybe we can just re-order the code to always call
svm_switch_vmcb()before {enter/leave}_guest_mode(). We already do that on the entry side, and seems to be straightforward on the exit side.
As stated earlier, I'd prefer to explicitly do flushing stuff where it fits from an architectural perspective.
Quoted 5 lines
With that, we can keep
kvm_service_local_tlb_flush_requests()and nested_svm_{entry/exit}_tlb_flush() insidesvm_switch_vmcb(). We can add a WARNING tokvm_service_local_tlb_flush_requests()to make sure the calls remains in the correct context (i.e. VMCB matches guest mode).
Re: [RFC PATCH v2 17/25] KVM: nSVM: Service local TLB flushes before nested transitions
Quoted 49 lines · Sean Christopherson
Quoted 48 lines · Yosry Ahmed
Quoted 46 lines · Yosry Ahmed
Quoted 18 lines
arch/x86/kvm/svm/nested.c static void svm_switch_vmcb(struct vcpu_svm *svm, struct kvm_vmcb_info *target_v · line 724
Quoted 13 lines
arch/x86/kvm/svm/nested.c +5
-0@@ -717,9 +717,16 @@ static void svm_switch_vmcb(struct vcpu_svm *svm, struct kvm_vmcb_info *target_v {struct kvm_vcpu *vcpu = &svm->vcpu; +/* +* TLB flushes are applied to the VMCB, so apply any pending TLB flushes +* on the current VMCB before switching to a new one.. +*/ +kvm_service_local_tlb_flush_requests (vcpu);[Severity: High]
Does this clear global TLB flush requests (likeKVM_REQ_TLB_FLUSH_GUEST) while applying them only to the outgoing VMCB?Quoted 3 lines
If a host-initiated flush occurs (e.g., from page reclaim), the request bit will be consumed and applied to the outgoing VMCB, leaving the incoming VMCB to miss the flush and run with a stale TLB.
Yeah it is intentional to apply the flushes to the current/old/outgoing context. If there are flushes needed in the new context, they are queued by nested_svm_{entry/exit}_tlb_flush(). That's similar to what VMX does.
Quoted 10 lines
Also, is there a context mismatch here during nested VM-Exit?
In
nested_svm_vmexit(),leave_guest_mode(vcpu)is called beforesvm_switch_vmcb(svm, &svm->vmcb01).Because of this,
kvm_service_local_tlb_flush_requests()will seeis_guest_mode(vcpu)as false. If Hyper-V is enabled, this meanskvm_hv_purge_tlb_flush_fifo()will incorrectly target L1's FIFO while the hardware flushes are actually being applied to L2's vmcb02.Ugh.. yes. This is annoying.
kvm_service_local_tlb_flush_requests()needs to be called on both the current/old/outgoing VMCB *and* guest mode. So we'll need to open-code the call in a bunch of places beforesvm_switch_vmcb()and {enter/leave}_guest_mode(). I really liked putting it insvm_switch_vmcb()together with nested_svm_{entry/exit}_tlb_flush() so that all the TLB flushing logic for nested transitions live in one place and the ordering needs to be handled in one place.
Quoted 6 lines
Quoted 3 lines
Maybe we can just re-order the code to always call
svm_switch_vmcb()before {enter/leave}_guest_mode(). We already do that on the entry side, and seems to be straightforward on the exit side.As stated earlier, I'd prefer to explicitly do flushing stuff where it fits from an architectural perspective.
I did it this way because (as you also stated) it's more robust, and it also documents the ordering requirements:
- We need to service local flushes before switching to the new VMCB.
- We need to queue new flushes after servicing local flushes.
I like that it's all in one place. All that being said, I did go back-and-forth on this so I am not opposed to open-coding it. Let me know if the above argument swayed you or if you still prefer open-coding.
Sean Christopherson
maintainerRe: [RFC PATCH v2 17/25] KVM: nSVM: Service local TLB flushes before nested transitions
Quoted 54 lines · Yosry Ahmed
Quoted 53 lines · Sean Christopherson
Quoted 51 lines · Yosry Ahmed
Quoted 38 lines · Yosry Ahmed
Quoted 18 lines
arch/x86/kvm/svm/nested.c static void svm_switch_vmcb(struct vcpu_svm *svm, struct kvm_vmcb_info *target_v · line 724
Quoted 13 lines
arch/x86/kvm/svm/nested.c +5
-0@@ -717,9 +717,16 @@ static void svm_switch_vmcb(struct vcpu_svm *svm, struct kvm_vmcb_info *target_v {struct kvm_vcpu *vcpu = &svm->vcpu; +/* +* TLB flushes are applied to the VMCB, so apply any pending TLB flushes +* on the current VMCB before switching to a new one.. +*/ +kvm_service_local_tlb_flush_requests (vcpu);[Severity: High]
Does this clear global TLB flush requests (likeKVM_REQ_TLB_FLUSH_GUEST) while applying them only to the outgoing VMCB?Quoted 3 lines
If a host-initiated flush occurs (e.g., from page reclaim), the request bit will be consumed and applied to the outgoing VMCB, leaving the incoming VMCB to miss the flush and run with a stale TLB.
Yeah it is intentional to apply the flushes to the current/old/outgoing context. If there are flushes needed in the new context, they are queued by nested_svm_{entry/exit}_tlb_flush(). That's similar to what VMX does.
Quoted 10 lines
Also, is there a context mismatch here during nested VM-Exit?
In
nested_svm_vmexit(),leave_guest_mode(vcpu)is called beforesvm_switch_vmcb(svm, &svm->vmcb01).Because of this,
kvm_service_local_tlb_flush_requests()will seeis_guest_mode(vcpu)as false. If Hyper-V is enabled, this meanskvm_hv_purge_tlb_flush_fifo()will incorrectly target L1's FIFO while the hardware flushes are actually being applied to L2's vmcb02.Quoted 8 lines
Ugh.. yes. This is annoying.
kvm_service_local_tlb_flush_requests()needs to be called on both the current/old/outgoing VMCB *and* guest mode. So we'll need to open-code the call in a bunch of places beforesvm_switch_vmcb()and {enter/leave}_guest_mode(). I really liked putting it insvm_switch_vmcb()together with nested_svm_{entry/exit}_tlb_flush() so that all the TLB flushing logic for nested transitions live in one place and the ordering needs to be handled in one place.Maybe we can just re-order the code to always call
svm_switch_vmcb()before {enter/leave}_guest_mode(). We already do that on the entry side, and seems to be straightforward on the exit side.
Quoted 6 lines
Quoted 2 lines
As stated earlier, I'd prefer to explicitly do flushing stuff where it fits from an architectural perspective.
I did it this way because (as you also stated) it's more robust, and it also documents the ordering requirements:
- We need to service local flushes before switching to the new VMCB.
No, that's not the requirement. The requirement is that KVM faithlyfully emulates the SVM architecture. For KVM's implementation, that _mostly_ aligns with switching between vmcb01 and vmcb02, but that's not a hard guarantee. Unlike VMX, SVM doesn't force KVM doesn't need to switch the active VMCB in order to make changes to a VMCB, so nSVM may never end up with as many switches as nVMX that don't need to trigger a TLB flush, but conceptually it's still inaccurate.
And regarding robustness, the flaw Sashiko pointed out regarding servicing pending flushes after leave_guest_mode(vcpu) highlights that burying architectural behaviors in what are effectively utility functions can be dangerous. The counter-argument is that we ended up with a similar bug in nVMX where KVM straight up forgot to service the flushes, but my point is that handling this in svm_switch_vmcb() isn't a silver bullet. And I really don't like adding an arbitrary constraint that the "new" VMCB needs to "match" the current L1 vs. L2 mode.
Quoted 5 lines
- We need to queue new flushes after servicing local flushes.
I like that it's all in one place. All that being said, I did go back-and-forth on this so I am not opposed to open-coding it. Let me know if the above argument swayed you or if you still prefer open-coding.
I still prefer open-coding the calls to perform/request/serivce flushes.
Re: [RFC PATCH v2 17/25] KVM: nSVM: Service local TLB flushes before nested transitions
Quoted 77 lines · Sean Christopherson
Quoted 54 lines · Yosry Ahmed
Quoted 53 lines · Sean Christopherson
Quoted 51 lines · Yosry Ahmed
Quoted 38 lines · Yosry Ahmed
Quoted 18 lines
arch/x86/kvm/svm/nested.c static void svm_switch_vmcb(struct vcpu_svm *svm, struct kvm_vmcb_info *target_v · line 724
Quoted 13 lines
arch/x86/kvm/svm/nested.c +5
-0@@ -717,9 +717,16 @@ static void svm_switch_vmcb(struct vcpu_svm *svm, struct kvm_vmcb_info *target_v {struct kvm_vcpu *vcpu = &svm->vcpu; +/* +* TLB flushes are applied to the VMCB, so apply any pending TLB flushes +* on the current VMCB before switching to a new one.. +*/ +kvm_service_local_tlb_flush_requests (vcpu);[Severity: High]
Does this clear global TLB flush requests (likeKVM_REQ_TLB_FLUSH_GUEST) while applying them only to the outgoing VMCB?Quoted 3 lines
If a host-initiated flush occurs (e.g., from page reclaim), the request bit will be consumed and applied to the outgoing VMCB, leaving the incoming VMCB to miss the flush and run with a stale TLB.
Yeah it is intentional to apply the flushes to the current/old/outgoing context. If there are flushes needed in the new context, they are queued by nested_svm_{entry/exit}_tlb_flush(). That's similar to what VMX does.
Quoted 10 lines
Also, is there a context mismatch here during nested VM-Exit?
In
nested_svm_vmexit(),leave_guest_mode(vcpu)is called beforesvm_switch_vmcb(svm, &svm->vmcb01).Because of this,
kvm_service_local_tlb_flush_requests()will seeis_guest_mode(vcpu)as false. If Hyper-V is enabled, this meanskvm_hv_purge_tlb_flush_fifo()will incorrectly target L1's FIFO while the hardware flushes are actually being applied to L2's vmcb02.Quoted 8 lines
Ugh.. yes. This is annoying.
kvm_service_local_tlb_flush_requests()needs to be called on both the current/old/outgoing VMCB *and* guest mode. So we'll need to open-code the call in a bunch of places beforesvm_switch_vmcb()and {enter/leave}_guest_mode(). I really liked putting it insvm_switch_vmcb()together with nested_svm_{entry/exit}_tlb_flush() so that all the TLB flushing logic for nested transitions live in one place and the ordering needs to be handled in one place.Maybe we can just re-order the code to always call
svm_switch_vmcb()before {enter/leave}_guest_mode(). We already do that on the entry side, and seems to be straightforward on the exit side.Quoted 6 lines
Quoted 2 lines
As stated earlier, I'd prefer to explicitly do flushing stuff where it fits from an architectural perspective.
I did it this way because (as you also stated) it's more robust, and it also documents the ordering requirements:
- We need to service local flushes before switching to the new VMCB.No, that's not the requirement. The requirement is that KVM faithlyfully emulates the SVM architecture. For KVM's implementation, that _mostly_ aligns with switching between vmcb01 and vmcb02, but that's not a hard guarantee. Unlike VMX, SVM doesn't force KVM doesn't need to switch the active VMCB in order to make changes to a VMCB, so nSVM may never end up with as many switches as nVMX that don't need to trigger a TLB flush, but conceptually it's still inaccurate.
And regarding robustness, the flaw Sashiko pointed out regarding servicing pending flushes after
leave_guest_mode(vcpu)highlights that burying architectural behaviors in what are effectively utility functions can be dangerous. The counter-argument is that we ended up with a similar bug in nVMX where KVM straight up forgot to service the flushes, but my point is that handling this insvm_switch_vmcb()isn't a silver bullet. And I really don't like adding an arbitrary constraint that the "new" VMCB needs to "match" the current L1 vs. L2 mode.
Quoted 7 lines
Quoted 5 lines
- We need to queue new flushes after servicing local flushes.
I like that it's all in one place. All that being said, I did go back-and-forth on this so I am not opposed to open-coding it. Let me know if the above argument swayed you or if you still prefer open-coding.
I still prefer open-coding the calls to perform/request/serivce flushes.
For the record, I had locally added a warning to svm_switch_vmcb() to make sure that guest_mode and the active VMCB are consistent when kvm_service_local_tlb_flush_requests() is called. Given that we are going with open coding the calls, I will add a wrapper for the warn+call:
static void nested_svm_service_local_tlb_flushes(struct vcpu *vcpu)
{
struct vcpu_svm *svm = to_svm(vcpu);
/*
* TLB flushes are applied to the VMCB, so apply any pending TLB flushes
* on the outgoing VMCB before switching to a new one. A TLB flush could
* purge the relevant HV TLB flush FIFO depending on guest_mode, so make
* sure the VMCB and guest_mode context is consistent.
*/
WARN_ON_ONCE(is_guest_mode(vcpu) != (svm->vmcb == svm->nested.vmcb02.ptr));
kvm_service_local_tlb_flush_requests(vcpu);
}RFC PATCH v2 18/25 KVM: nSVM: Handle nested TLB flush requests through TLB_CONTROL
In reply to Yosry AhmedHandle L1's requests to flush L2's TLB through the TLB_CONTROL field of VMCB12.
On nested VM-Enter, flush L2's ASID if any flush is specified in TLB_CONTROL. This handles TLB_CONTROL_FLUSH_ASID and TLB_CONTROL_FLUSH_ASID_LOCAL equally for simplicity.
On nested VM-Exit, flush L1's own ASID if L1 requested a *full* TLB flush (i.e. TLB_CONTROL_FLUSH_ALL_ASID).
Essentially, TLB_CONTROL_FLUSH_ASID[_LOCAL] cause a TLB flush on nested VM-Enter only, while TLB_CONTROL_FLUSH_ALL_ASID causes a TLB flush on both nested VM-Enter and nested VM-Exit.
Additionally, sync the nested NPTs on nested VM-Enter on an ASID flush, as ASID flushes also invalidate guest-physical translations on SVM (unlike VMX, which has separate VPID-based and EPT-based flushing).
All TLB_CONTROL values can be handled by KVM regardless of FLUSHBYASID support on the underlying CPU, so keep advertising FLUSHBYASID to the guest unconditionally.
Note, TLB_CONTROL_FLUSH_ALL_ASID is never propagated from the vmcb12 to the vmcb02 (unless FLUSHBYASID is not available), as this gives the guest the power to flush the entire physical TLB (including translations for the host and other VMs).
--- arch/x86/kvm/svm/nested.c | 30 +++++++++++++++++++++++++----- arch/x86/kvm/svm/svm.c | 5 ++--- 2 files changed, 27 insertions(+), 8 deletions(-)
arch/x86/kvm/svm/nested.c +25 -5
@@ -687,19 +687,33 @@ static void nested_save_pending_event_to_vmcb12(struct vcpu_svm *svm,static void nested_svm_entry_tlb_flush (struct kvm_vcpu *vcpu) { +struct vcpu_svm *svm =to_svm (vcpu); +/* Handle pending Hyper-V TLB flush requests */ kvm_hv_nested_transtion_tlb_flush (vcpu, npt_enabled); +/* +* If L1 requested a TLB flush for L2, flush L2's TLB on nested entry +* and sync the nested NPT MMU, as TLB_CONTROL also flushes NPT +* guest-physical mappings. +* +* If L1 requested a full TLB flush for all ASIDs (including its own), +* L1's own ASID is also flushed on nested VM-Exit, before running L1. +* +* TLB_CONTROL_FLUSH_ASID and TLB_CONTROL_FLUSH_ASID_LOCAL are handled +* equally for simplicity. +*/ +if (svm->nested.ctl.tlb_ctl !=TLB_CONTROL_DO_NOTHING ) { +if (nested_npt_enabled (svm)) +kvm_make_request (KVM_REQ_MMU_SYNC , vcpu); +kvm_make_request (KVM_REQ_TLB_FLUSH_GUEST , vcpu); + } +/* * TODO: optimize unconditional TLB flush/MMU sync. A partial list of * things to fix before this can be conditional: * -* - Honor L1's request to flush an ASID on nested VMRUN -* - Sync nested NPT MMU on VMRUN that flushes L2's ASID[*] * - Don't crush a pending TLB flush in vmcb02 on nested VMRUN -* -* [*] Unlike nested EPT, SVM's ASID management can invalidate nested -* NPT guest-physical mappings on VMRUN. */ kvm_make_request (KVM_REQ_MMU_SYNC , vcpu);kvm_make_request (KVM_REQ_TLB_FLUSH_CURRENT , vcpu);
@@ -707,8 +721,14 @@ static void nested_svm_entry_tlb_flush(struct kvm_vcpu *vcpu)static void nested_svm_exit_tlb_flush (struct kvm_vcpu *vcpu) { +struct vcpu_svm *svm =to_svm (vcpu); +kvm_hv_nested_transtion_tlb_flush (vcpu, npt_enabled); +/* Flush L1's own ASID if it request a *full* TLB flush on VMRUN */ +if (svm->nested.ctl.tlb_ctl ==TLB_CONTROL_FLUSH_ALL_ASID ) +kvm_make_request (KVM_REQ_TLB_FLUSH_GUEST , vcpu); +kvm_make_request (KVM_REQ_MMU_SYNC , vcpu);kvm_make_request (KVM_REQ_TLB_FLUSH_CURRENT , vcpu); }
arch/x86/kvm/svm/svm.c +2 -3
@@ -5512,9 +5512,8 @@ static __init void svm_set_cpu_caps(void)kvm_cpu_cap_set (X86_FEATURE_VMCBCLEAN );/* -* KVM currently flushes TLBs on *every* nested SVM transition, -* and so for all intents and purposes KVM supports flushing by -* ASID, i.e. KVM is guaranteed to honor every L1 ASID flush. +* KVM handles all TLB_CONTROL values set by L1, even if the +* underlying CPU does not. */ kvm_cpu_cap_set (X86_FEATURE_FLUSHBYASID );
Signature · 1 line
2.54.0.1136.gdb2ca164c4-goog
RFC PATCH v2 19/25 KVM: nSVM: Flush the TLB if L1 changes L2's ASID in vmcb12
In reply to Yosry AhmedKVM uses a single ASID for L2 guests per-vCPU. Hence, when L1 changes L2's ASID in vmcb12 (e.g. due to switching a different L2 vCPU or simply to avoid flushing an existing ASID), KVM needs to flush the L2 ASID to correctly emulate different ASIDs as different TLB domains.
Additionally, the nested NPT needs to be resync'd, as the MMU context is not tagged by ASID, and KVM cannot reuse nested NPT entries across different L2 ASIDs.
Essentially, L1 switching L2's ASID is treated exactly the same as L1 flushing L2's ASID, which is consistent with the APM:
Software may effectively flush the guest's TLB entries by allocating a new ASID for the guest and not reusing the old ASID until the entire TLB has been flushed at least once.
This is similar to nVMX's handling of last_vpid, except that when L1 changes VPID12, KVM does *not* need to resync the nested EPT, because VMX VPIDs , unlike SVM ASIDs, do not tag guest-physical translations (i.e. nGPA to GPA translations).
Drop the commentary about vmcb12's ASID being copied around only for the consistency checks, as they no longer apply, and opportunistically fix whitespace alignment.
This is currently functionally a noop, as a full flush and sync is triggered on every nested transition, but is a step toward eliminating that.
--- arch/x86/kvm/svm/nested.c | 22 ++++++++++++++++------ arch/x86/kvm/svm/svm.h | 2 ++ 2 files changed, 18 insertions(+), 6 deletions(-)
arch/x86/kvm/svm/nested.c +16 -6
@@ -553,10 +553,8 @@ void __nested_copy_vmcb_control_to_cache(struct kvm_vcpu *vcpu, to->misc_ctl2 = from->misc_ctl2; to->pause_filter_count = from->pause_filter_count; to->pause_filter_thresh = from->pause_filter_thresh; - -/* Copy asid here because nested_vmcb_check_controls() will check it */ - to->asid = from->asid; - to->clean = from->clean; + to->asid = from->asid; + to->clean = from->clean;#ifdef CONFIG_KVM_HYPERV /* Hyper-V extensions (Enlightened VMCB) */
@@ -688,22 +686,34 @@ static void nested_save_pending_event_to_vmcb12(struct vcpu_svm *svm,static void nested_svm_entry_tlb_flush (struct kvm_vcpu *vcpu) {struct vcpu_svm *svm =to_svm (vcpu); +bool new_asid = false;/* Handle pending Hyper-V TLB flush requests */ kvm_hv_nested_transtion_tlb_flush (vcpu, npt_enabled); +if (svm->nested.ctl.asid != svm->nested.last_asid) { + svm->nested.last_asid = svm->nested.ctl.asid; + new_asid = true; + } +/* * If L1 requested a TLB flush for L2, flush L2's TLB on nested entry * and sync the nested NPT MMU, as TLB_CONTROL also flushes NPT * guest-physical mappings. * +* Handle L1 changing L2's ASID12 similarly, as KVM only uses one ASID +* for L2 in hardware (per vCPU), so it must start fresh when L1 changes +* ASID12 to emulate different ASIDs correctly. Additionally, the MMU +* context is not tagged by the ASID, so the shadow NPTs cannot be +* reused across different L2 ASIDs. +* * If L1 requested a full TLB flush for all ASIDs (including its own), * L1's own ASID is also flushed on nested VM-Exit, before running L1. * * TLB_CONTROL_FLUSH_ASID and TLB_CONTROL_FLUSH_ASID_LOCAL are handled * equally for simplicity. */ -if (svm->nested.ctl.tlb_ctl !=TLB_CONTROL_DO_NOTHING ) { +if (new_asid || (svm->nested.ctl.tlb_ctl !=TLB_CONTROL_DO_NOTHING )) {if (nested_npt_enabled (svm))kvm_make_request (KVM_REQ_MMU_SYNC , vcpu);kvm_make_request (KVM_REQ_TLB_FLUSH_GUEST , vcpu);
@@ -1882,7 +1892,7 @@ void nested_svm_update_tsc_ratio_msr(struct kvm_vcpu *vcpu)svm_write_tsc_multiplier (vcpu); } -/* Inverse operation of nested_copy_vmcb_control_to_cache(). asid is copied too. */ +/* Inverse operation of nested_copy_vmcb_control_to_cache() */ static void nested_copy_vmcb_cache_to_control (struct vmcb_control_area *dst,struct vmcb_ctrl_area_cached *from) {
arch/x86/kvm/svm/svm.h +2 -0
@@ -244,6 +244,8 @@ struct svm_nested_state {* on its side. */ bool force_msr_bitmap_recalc; + +u32 last_asid; };struct vcpu_sev_es_state {
Signature · 1 line
2.54.0.1136.gdb2ca164c4-goog
RFC PATCH v2 20/25 KVM: nSVM: Do not reset TLB_CONTROL in vmcb02 on nested VM-Enter
In reply to Yosry AhmedStop clearing TLB_CONTROL when preparing the control area of vmcb02, as this potentially undos pending TLB flushes for L2 (e.g. through KVM_REQ_TLB_FLUSH while L1 is running), and remove the associated TODO comment.
This is currently harmless, because nested_svm_entry_tlb_flush() always requests KVM_REQ_TLB_FLUSH_CURRENT on nested VM-Enter, which sets TLB_CONTROL again before L2 is actually run. However, always flushing will soon go away with proper TLB handling for L2, at which point always clearing TLB_CONTROL would be a bug.
Clearing TLB_CONTROL on nested VM-Enter was probably done because TLB_CONTROL is not cleared by the CPU on VM-Exit. However, KVM always clears TLB_CONTROL in the active VMCB after VMRUN. Hence, at nested VM-Enter, TLB_CONTROL in vmcb02 can only be non-zero if a TLB flush is queued for L2 while L1 is running (i.e. KVM_REQ_TLB_FLUSH), and that should never be ignored.
--- arch/x86/kvm/svm/nested.c | 10 +--------- 1 file changed, 1 insertion(+), 9 deletions(-)
arch/x86/kvm/svm/nested.c +1 -9
@@ -719,12 +719,7 @@ static void nested_svm_entry_tlb_flush(struct kvm_vcpu *vcpu)kvm_make_request (KVM_REQ_TLB_FLUSH_GUEST , vcpu); } -/* -* TODO: optimize unconditional TLB flush/MMU sync. A partial list of -* things to fix before this can be conditional: -* -* - Don't crush a pending TLB flush in vmcb02 on nested VMRUN -*/ +/* TODO: optimize unconditional TLB flush/MMU sync */ kvm_make_request (KVM_REQ_MMU_SYNC , vcpu);kvm_make_request (KVM_REQ_TLB_FLUSH_CURRENT , vcpu); }
@@ -981,9 +976,6 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm) vmcb02->control.asid = svm->nested.asid02; -/* Overwritten later if necessary. */ -vmcb_clr_flush_asid (vmcb02); -/* Use vmcb01 MMU and format if guest does not use nNPT */ if (nested_npt_enabled (svm)) { vmcb02->control.misc_ctl &= ~SVM_MISC_ENABLE_GMET ;
Signature · 1 line
2.54.0.1136.gdb2ca164c4-goog
RFC PATCH v2 21/25 KVM: x86/mmu: rename __kvm_mmu_invalidate_addr()
In reply to Yosry AhmedIn preparation for creating another helper for kvm_mmu_invalidate_addr(), rename __kvm_mmu_invalidate_addr() to kvm_mmu_invalidate_addr_in_root().
No functional change intended.
--- arch/x86/kvm/mmu/mmu.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
arch/x86/kvm/mmu/mmu.c +5 -4
@@ -6577,8 +6577,9 @@ void kvm_mmu_print_sptes(struct kvm_vcpu *vcpu, gpa_t gpa, const char *msg) }EXPORT_SYMBOL_FOR_KVM_INTERNAL (kvm_mmu_print_sptes); -static void __kvm_mmu_invalidate_addr (struct kvm_vcpu *vcpu,struct kvm_mmu *mmu, -u64 addr,hpa_t root_hpa) +static void kvm_mmu_invalidate_addr_in_root (struct kvm_vcpu *vcpu, +struct kvm_mmu *mmu, +u64 addr,hpa_t root_hpa) {struct kvm_shadow_walk_iterator iterator;
@@ -6634,11 +6635,11 @@ void kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu,return ;if (roots &KVM_MMU_ROOT_CURRENT ) -__kvm_mmu_invalidate_addr (vcpu, mmu, addr, mmu->root.hpa); +kvm_mmu_invalidate_addr_in_root (vcpu, mmu, addr, mmu->root.hpa);for (i =0 ; i <KVM_MMU_NUM_PREV_ROOTS ; i++) {if (roots &KVM_MMU_ROOT_PREVIOUS (i)) -__kvm_mmu_invalidate_addr (vcpu, mmu, addr, mmu->prev_roots[i].hpa); +kvm_mmu_invalidate_addr_in_root (vcpu, mmu, addr, mmu->prev_roots[i].hpa); } }EXPORT_SYMBOL_FOR_KVM_INTERNAL (kvm_mmu_invalidate_addr);
Signature · 1 line
2.54.0.1136.gdb2ca164c4-goog
RFC PATCH v2 22/25 KVM: x86/mmu: Refactor kvm_mmu_invlpg() to allow skipping the gva flush
In reply to Yosry AhmedRefactor helpers out of kvm_mmu_invalidate_addr() and kvm_mmu_invlpg() that take in an extra argument to skip the GVA flush.
This will be used when invalidating GVAs in a different context than the correct one (i.e. invalidating an L2 GVA from L1), so flushing the current context would flush the wrong TLB entries.
No functional change intended.
--- arch/x86/kvm/mmu/mmu.c | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-)
arch/x86/kvm/mmu/mmu.c +17 -6
@@ -6615,15 +6615,15 @@ static void kvm_mmu_invalidate_addr_in_root(struct kvm_vcpu *vcpu,write_unlock (&vcpu->kvm->mmu_lock); } -void kvm_mmu_invalidate_addr (struct kvm_vcpu *vcpu,struct kvm_mmu *mmu, -u64 addr,unsigned long roots) +static void __kvm_mmu_invalidate_addr (struct kvm_vcpu *vcpu,struct kvm_mmu *mmu, +u64 addr,unsigned long roots,bool flush_gva) {int i;WARN_ON_ONCE (roots & ~KVM_MMU_ROOTS_ALL );/* It's actually a GPA for vcpu->arch.guest_mmu. */ -if (mmu != &vcpu->arch.guest_mmu) { +if (flush_gva && mmu != &vcpu->arch.guest_mmu) {/* INVLPG on a non-canonical address is a NOP according to the SDM. */ if (is_noncanonical_invlpg_address (addr, vcpu))return ;
@@ -6642,9 +6642,15 @@ void kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu,kvm_mmu_invalidate_addr_in_root (vcpu, mmu, addr, mmu->prev_roots[i].hpa); } } + +void kvm_mmu_invalidate_addr (struct kvm_vcpu *vcpu,struct kvm_mmu *mmu, +u64 addr,unsigned long roots) +{ +__kvm_mmu_invalidate_addr (vcpu, mmu, addr, roots, true); +}EXPORT_SYMBOL_FOR_KVM_INTERNAL (kvm_mmu_invalidate_addr); -void kvm_mmu_invlpg (struct kvm_vcpu *vcpu,gva_t gva) +static void __kvm_mmu_invlpg (struct kvm_vcpu *vcpu,gva_t gva,bool flush_gva) {/* * INVLPG is required to invalidate any global mappings for the VA,
@@ -6656,11 +6662,16 @@ void kvm_mmu_invlpg(struct kvm_vcpu *vcpu, gva_t gva)* be synced when switching to that new cr3, so nothing needs to be * done here for them. */ -kvm_mmu_invalidate_addr (vcpu, vcpu->arch.walk_mmu, gva,KVM_MMU_ROOTS_ALL ); +__kvm_mmu_invalidate_addr (vcpu, vcpu->arch.walk_mmu, gva, +KVM_MMU_ROOTS_ALL , flush_gva); ++vcpu->stat.invlpg; } -EXPORT_SYMBOL_FOR_KVM_INTERNAL (kvm_mmu_invlpg); +void kvm_mmu_invlpg (struct kvm_vcpu *vcpu,gva_t gva) +{ +__kvm_mmu_invlpg (vcpu, gva, true); +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL (kvm_mmu_invlpg);void kvm_mmu_invpcid_gva (struct kvm_vcpu *vcpu,gva_t gva,unsigned long pcid) {
Signature · 1 line
2.54.0.1136.gdb2ca164c4-goog
Sean Christopherson
maintainerRe: [RFC PATCH v2 22/25] KVM: x86/mmu: Refactor kvm_mmu_invlpg() to allow skipping the gva flush
Quoted 38 lines · Yosry Ahmed
Refactor helpers out of
kvm_mmu_invalidate_addr()andkvm_mmu_invlpg()that take in an extra argument to skip the GVA flush.This will be used when invalidating GVAs in a different context than the correct one (i.e. invalidating an L2 GVA from L1), so flushing the current context would flush the wrong TLB entries.
No functional change intended.
Signed-off-by Yosry Ahmed <[email protected]>--- arch/x86/kvm/mmu/mmu.c | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-)arch/x86/kvm/mmu/mmu.c +3
-3@@ -6615,15 +6615,15 @@ static void kvm_mmu_invalidate_addr_in_root(struct kvm_vcpu *vcpu,write_unlock (&vcpu->kvm->mmu_lock); } -void kvm_mmu_invalidate_addr (struct kvm_vcpu *vcpu,struct kvm_mmu *mmu, -u64 addr,unsigned long roots) +static void __kvm_mmu_invalidate_addr (struct kvm_vcpu *vcpu,struct kvm_mmu *mmu, +u64 addr,unsigned long roots,bool flush_gva) {int i;WARN_ON_ONCE (roots & ~KVM_MMU_ROOTS_ALL );/* It's actually a GPA for vcpu->arch.guest_mmu. */ -if (mmu != &vcpu->arch.guest_mmu) { +if (flush_gva && mmu != &vcpu->arch.guest_mmu) {/* INVLPG on a non-canonical address is a NOP according to the SDM. */ if (is_noncanonical_invlpg_address (addr, vcpu))return ;
arch/x86/kvm/mmu/mmu.c void kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, · line 6647
Quoted 6 lines
arch/x86/kvm/mmu/mmu.c +3
-0@@ -6642,9 +6642,15 @@ void kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu,kvm_mmu_invalidate_addr_in_root (vcpu, mmu, addr, mmu->prev_roots[i].hpa); } } + +void kvm_mmu_invalidate_addr (struct kvm_vcpu *vcpu,struct kvm_mmu *mmu, +u64 addr,unsigned long roots)
Rather than kvm_mmu_invalidate_addr() for the wrapper, what if we call this kvm_mmu_invalidate_gva()? And then kvm_mmu_invlpg_gva(). Then we don't need to have the "in_root" version to a quad-underscores helper, and IMO it's more obvious what's different between the one-line wrappers and the inner helpers.
arch/x86/kvm/mmu/mmu.c void kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, · line 6650
Quoted 3 lines
arch/x86/kvm/mmu/mmu.c +3
-0+{ +__kvm_mmu_invalidate_addr (vcpu, mmu, addr, roots, true); +}
Make these one-liners static inlines to avoid the export, and because they're trivial wrappers that should be inlined.
Quoted 29 lines
arch/x86/kvm/mmu/mmu.c +8
-3EXPORT_SYMBOL_FOR_KVM_INTERNAL (kvm_mmu_invalidate_addr); -void kvm_mmu_invlpg (struct kvm_vcpu *vcpu,gva_t gva) +static void __kvm_mmu_invlpg (struct kvm_vcpu *vcpu,gva_t gva,bool flush_gva) {/* * INVLPG is required to invalidate any global mappings for the VA, @@ -6656,11 +6662,16 @@ void kvm_mmu_invlpg(struct kvm_vcpu *vcpu, gva_t gva)* be synced when switching to that new cr3, so nothing needs to be * done here for them. */ -kvm_mmu_invalidate_addr (vcpu, vcpu->arch.walk_mmu, gva,KVM_MMU_ROOTS_ALL ); +__kvm_mmu_invalidate_addr (vcpu, vcpu->arch.walk_mmu, gva, +KVM_MMU_ROOTS_ALL , flush_gva); ++vcpu->stat.invlpg; } -EXPORT_SYMBOL_FOR_KVM_INTERNAL (kvm_mmu_invlpg); +void kvm_mmu_invlpg (struct kvm_vcpu *vcpu,gva_t gva) +{ +__kvm_mmu_invlpg (vcpu, gva, true); +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL (kvm_mmu_invlpg);void kvm_mmu_invpcid_gva (struct kvm_vcpu *vcpu,gva_t gva,unsigned long pcid) {Signature · 1 line
2.54.0.1136.gdb2ca164c4-goog
Sean Christopherson
maintainerRe: [RFC PATCH v2 22/25] KVM: x86/mmu: Refactor kvm_mmu_invlpg() to allow skipping the gva flush
Quoted 42 lines · Sean Christopherson
Quoted 41 lines · Yosry Ahmed
Refactor helpers out of
kvm_mmu_invalidate_addr()andkvm_mmu_invlpg()that take in an extra argument to skip the GVA flush.This will be used when invalidating GVAs in a different context than the correct one (i.e. invalidating an L2 GVA from L1), so flushing the current context would flush the wrong TLB entries.
No functional change intended.
Signed-off-by Yosry Ahmed <[email protected]>--- arch/x86/kvm/mmu/mmu.c | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-)arch/x86/kvm/mmu/mmu.c +3
-3@@ -6615,15 +6615,15 @@ static void kvm_mmu_invalidate_addr_in_root(struct kvm_vcpu *vcpu,write_unlock (&vcpu->kvm->mmu_lock); } -void kvm_mmu_invalidate_addr (struct kvm_vcpu *vcpu,struct kvm_mmu *mmu, -u64 addr,unsigned long roots) +static void __kvm_mmu_invalidate_addr (struct kvm_vcpu *vcpu,struct kvm_mmu *mmu, +u64 addr,unsigned long roots,bool flush_gva) {int i;WARN_ON_ONCE (roots & ~KVM_MMU_ROOTS_ALL );/* It's actually a GPA for vcpu->arch.guest_mmu. */ -if (mmu != &vcpu->arch.guest_mmu) { +if (flush_gva && mmu != &vcpu->arch.guest_mmu) {/* INVLPG on a non-canonical address is a NOP according to the SDM. */ if (is_noncanonical_invlpg_address (addr, vcpu))return ;@@ -6642,9 +6642,15 @@ void kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu,kvm_mmu_invalidate_addr_in_root (vcpu, mmu, addr, mmu->prev_roots[i].hpa); } }
Quoted 8 lines
arch/x86/kvm/mmu/mmu.c void kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, · line 6647
Quoted 3 lines
arch/x86/kvm/mmu/mmu.c +3
-0+ +void kvm_mmu_invalidate_addr (struct kvm_vcpu *vcpu,struct kvm_mmu *mmu, +u64 addr,unsigned long roots)Rather than
kvm_mmu_invalidate_addr()for the wrapper, what if we call thiskvm_mmu_invalidate_gva()? And thenkvm_mmu_invlpg_gva(). Then we don't need to have the "in_root" version to a quad-underscores helper, and IMO it's more obvious what's different between the one-line wrappers and the inner helpers.
Hrm, or maybe I'm not understanding what "flush_gva" means. At first glance, I was assuming you were using it to differentiate between GVA and GPA, but IIUC, it's literally skipping the flush for the current context, which just so happens to be done only for GVAs. I'd still like to avoid the "in_root" helper, if at all possible.
Re: [RFC PATCH v2 22/25] KVM: x86/mmu: Refactor kvm_mmu_invlpg() to allow skipping the gva flush
Quoted 52 lines · Sean Christopherson
Quoted 42 lines · Sean Christopherson
Quoted 41 lines · Yosry Ahmed
Refactor helpers out of
kvm_mmu_invalidate_addr()andkvm_mmu_invlpg()that take in an extra argument to skip the GVA flush.This will be used when invalidating GVAs in a different context than the correct one (i.e. invalidating an L2 GVA from L1), so flushing the current context would flush the wrong TLB entries.
No functional change intended.
Signed-off-by Yosry Ahmed <[email protected]>--- arch/x86/kvm/mmu/mmu.c | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-)arch/x86/kvm/mmu/mmu.c +3
-3@@ -6615,15 +6615,15 @@ static void kvm_mmu_invalidate_addr_in_root(struct kvm_vcpu *vcpu,write_unlock (&vcpu->kvm->mmu_lock); } -void kvm_mmu_invalidate_addr (struct kvm_vcpu *vcpu,struct kvm_mmu *mmu, -u64 addr,unsigned long roots) +static void __kvm_mmu_invalidate_addr (struct kvm_vcpu *vcpu,struct kvm_mmu *mmu, +u64 addr,unsigned long roots,bool flush_gva) {int i;WARN_ON_ONCE (roots & ~KVM_MMU_ROOTS_ALL );/* It's actually a GPA for vcpu->arch.guest_mmu. */ -if (mmu != &vcpu->arch.guest_mmu) { +if (flush_gva && mmu != &vcpu->arch.guest_mmu) {/* INVLPG on a non-canonical address is a NOP according to the SDM. */ if (is_noncanonical_invlpg_address (addr, vcpu))return ;@@ -6642,9 +6642,15 @@ void kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu,kvm_mmu_invalidate_addr_in_root (vcpu, mmu, addr, mmu->prev_roots[i].hpa); } }Quoted 8 lines
arch/x86/kvm/mmu/mmu.c void kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, · line 6647
Quoted 3 lines
arch/x86/kvm/mmu/mmu.c +3
-0+ +void kvm_mmu_invalidate_addr (struct kvm_vcpu *vcpu,struct kvm_mmu *mmu, +u64 addr,unsigned long roots)Rather than
kvm_mmu_invalidate_addr()for the wrapper, what if we call thiskvm_mmu_invalidate_gva()? And thenkvm_mmu_invlpg_gva(). Then we don't need to have the "in_root" version to a quad-underscores helper, and IMO it's more obvious what's different between the one-line wrappers and the inner helpers.
Quoted 5 lines
Hrm, or maybe I'm not understanding what "
flush_gva" means. At first glance, I was assuming you were using it to differentiate between GVA and GPA, but IIUC, it's literally skipping the flush for the current context, which just so happens to be done only for GVAs. I'd still like to avoid the "in_root" helper, if at all possible.
Yes, it's skipping the TLB flush that is only needed for GVAs. The alternative I had in mind (but thought was worse) was to refactor the TLB flush part out of kvm_mmu_invalidate_addr() (or __kvm_mmu_invalidate_addr()) in this patch instead of adding a boolean, but this still requires adding a wrapper and the possibility of a quad-underscore helper.
What's the main objection to kvm_mmu_invalidate_addr_in_root()?
Sean Christopherson
maintainerRe: [RFC PATCH v2 22/25] KVM: x86/mmu: Refactor kvm_mmu_invlpg() to allow skipping the gva flush
Quoted 59 lines · Yosry Ahmed
Quoted 52 lines · Sean Christopherson
Quoted 42 lines · Sean Christopherson
Quoted 41 lines · Yosry Ahmed
Refactor helpers out of
kvm_mmu_invalidate_addr()andkvm_mmu_invlpg()that take in an extra argument to skip the GVA flush.This will be used when invalidating GVAs in a different context than the correct one (i.e. invalidating an L2 GVA from L1), so flushing the current context would flush the wrong TLB entries.
No functional change intended.
Signed-off-by Yosry Ahmed <[email protected]>--- arch/x86/kvm/mmu/mmu.c | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-)arch/x86/kvm/mmu/mmu.c +3
-3@@ -6615,15 +6615,15 @@ static void kvm_mmu_invalidate_addr_in_root(struct kvm_vcpu *vcpu,write_unlock (&vcpu->kvm->mmu_lock); } -void kvm_mmu_invalidate_addr (struct kvm_vcpu *vcpu,struct kvm_mmu *mmu, -u64 addr,unsigned long roots) +static void __kvm_mmu_invalidate_addr (struct kvm_vcpu *vcpu,struct kvm_mmu *mmu, +u64 addr,unsigned long roots,bool flush_gva) {int i;WARN_ON_ONCE (roots & ~KVM_MMU_ROOTS_ALL );/* It's actually a GPA for vcpu->arch.guest_mmu. */ -if (mmu != &vcpu->arch.guest_mmu) { +if (flush_gva && mmu != &vcpu->arch.guest_mmu) {/* INVLPG on a non-canonical address is a NOP according to the SDM. */ if (is_noncanonical_invlpg_address (addr, vcpu))return ;@@ -6642,9 +6642,15 @@ void kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu,kvm_mmu_invalidate_addr_in_root (vcpu, mmu, addr, mmu->prev_roots[i].hpa); } }Quoted 8 lines
arch/x86/kvm/mmu/mmu.c void kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, · line 6647
Quoted 3 lines
arch/x86/kvm/mmu/mmu.c +3
-0+ +void kvm_mmu_invalidate_addr (struct kvm_vcpu *vcpu,struct kvm_mmu *mmu, +u64 addr,unsigned long roots)Rather than
kvm_mmu_invalidate_addr()for the wrapper, what if we call thiskvm_mmu_invalidate_gva()? And thenkvm_mmu_invlpg_gva(). Then we don't need to have the "in_root" version to a quad-underscores helper, and IMO it's more obvious what's different between the one-line wrappers and the inner helpers.Quoted 5 lines
Hrm, or maybe I'm not understanding what "
flush_gva" means. At first glance, I was assuming you were using it to differentiate between GVA and GPA, but IIUC, it's literally skipping the flush for the current context, which just so happens to be done only for GVAs. I'd still like to avoid the "in_root" helper, if at all possible.
Quoted 8 lines
Yes, it's skipping the TLB flush that is only needed for GVAs. The alternative I had in mind (but thought was worse) was to refactor the TLB flush part out of
kvm_mmu_invalidate_addr()(or__kvm_mmu_invalidate_addr()) in this patch instead of adding a boolean, but this still requires adding a wrapper and the possibility of a quad-underscore helper.What's the main objection to
kvm_mmu_invalidate_addr_in_root()?
I don't love the __kvm_mmu_invalidate_addr() => kvm_mmu_invalidate_addr_in_root() callchain. It's not at all obvious that the in_root() helper shouldn't be called directly. I don't hate it, but I do think we need better clarity on what all this is doing.
E.g. when looking at __kvm_inject_emulated_page_fault(), since it hardcodes a single root, it's a bit headscratching to use kvm_mmu_invalidate_addr() instead of kvm_mmu_invalidate_addr_in_root.
Hmm, and arguably, the way invlpga_interception() handles the ASID is flat out wrong. KVM doesn't need to flush *all* roots, rather it needs to flush L1 roots for ASID=0, and L2 roots for ASID!=0. If we can figure out an elegant way to express and handle that, it should naturally handle the "flush GVA" aspect.
Actually, isn't there a pre-existing over-flush when handling kvm_mmu_invpcid_gva()? Oof, and a missed flush?
To fix the over-flush, I think we want this?
x86/kvm/mmu/mmu.c +2 -1
@@ -6672,7 +6672,8 @@ void kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_pagewalk *w,if (is_noncanonical_invlpg_address (addr, vcpu))return ; -kvm_x86_call (flush_tlb_gva)(vcpu, addr); +if (roots &KVM_MMU_ROOT_CURRENT ) +kvm_x86_call (flush_tlb_gva)(vcpu, addr);if (tdp_enabled)return ;
And that highlights the missed flush: if the PCID isn't the current PCID, then flush_tlb_gva() neglects to flush the hardware TLB for the target PCID, which could leave a stale entry in the TLB if the guest switches to the new PCID with MOV CR3 + X86_CR3_PCID_NOFLUSH.
Re: [RFC PATCH v2 22/25] KVM: x86/mmu: Refactor kvm_mmu_invlpg() to allow skipping the gva flush
Quoted 10 lines
Actually, isn't there a pre-existing over-flush when handling
kvm_mmu_invpcid_gva()? Oof, and a missed flush?To fix the over-flush, I think we want this?
x86/kvm/mmu/mmu.c +0
-0
x86/kvm/mmu/mmu.c void kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_pagewalk *w, · line 6676
Quoted 6 lines
x86/kvm/mmu/mmu.c +2
-1@@ -6672,7 +6672,8 @@ void kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_pagewalk *w,if (is_noncanonical_invlpg_address (addr, vcpu))return ; -kvm_x86_call (flush_tlb_gva)(vcpu, addr); +if (roots &KVM_MMU_ROOT_CURRENT ) +kvm_x86_call (flush_tlb_gva)(vcpu, addr);
Hmmm why?
AFAICT, if the gva has a shadow mapping, then the rest of kvm_mmu_invalidate_addr() will make sure the shadow mapping is up-to-date and do any necessary TLB flushes in the process. So I am assuming the flush here is intended to cover some cases where the gva does not have a shadow mapping. Not sure how this could happen, I assume we always flush the TLB when a shadow PTE is freed.
If there are cases where we can have TLB entries but no shadow mappings, then shouldn't we flush regardless of which cached roots match the target PCID? I assume if none of the cached roots match the PCID it's not an issue because KVM will create a new root and flush the TLB before switching into the new PCID.
Quoted 9 lines
x86/kvm/mmu/mmu.c +0
-0if (tdp_enabled)return ;And that highlights the missed flush: if the PCID isn't the current PCID, then
flush_tlb_gva()neglects to flush the hardware TLB for the target PCID, which could leave a stale entry in the TLB if the guest switches to the new PCID with MOV CR3 +X86_CR3_PCID_NOFLUSH.
I also don't follow this part flush_tlb_gva() ends up executing INVVPID or INVLPGA. In both cases, all PCIDs for the VPID/ASID are flushed, right?
Sean Christopherson
maintainerRe: [RFC PATCH v2 22/25] KVM: x86/mmu: Refactor kvm_mmu_invlpg() to allow skipping the gva flush
Quoted 26 lines · Yosry Ahmed
Quoted 10 lines
Actually, isn't there a pre-existing over-flush when handling
kvm_mmu_invpcid_gva()? Oof, and a missed flush?To fix the over-flush, I think we want this?
x86/kvm/mmu/mmu.c +0
-0x86/kvm/mmu/mmu.c void kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_pagewalk *w, · line 6676
Quoted 6 lines
x86/kvm/mmu/mmu.c +2
-1@@ -6672,7 +6672,8 @@ void kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_pagewalk *w,if (is_noncanonical_invlpg_address (addr, vcpu))return ; -kvm_x86_call (flush_tlb_gva)(vcpu, addr); +if (roots &KVM_MMU_ROOT_CURRENT ) +kvm_x86_call (flush_tlb_gva)(vcpu, addr);Hmmm why?
AFAICT, if the gva has a shadow mapping, then the rest of
kvm_mmu_invalidate_addr()will make sure the shadow mapping is up-to-date and do any necessary TLB flushes in the process. So I am assuming the flush here is intended to cover some cases where the gva does not have a shadow mapping. Not sure how this could happen, I assume we always flush the TLB when a shadow PTE is freed.
Quoted 5 lines
If there are cases where we can have TLB entries but no shadow mappings, then shouldn't we flush regardless of which cached roots match the target PCID? I assume if none of the cached roots match the PCID it's not an issue because KVM will create a new root and flush the TLB before switching into the new PCID.
Gah, I conflated PCIDs and VPIDs.
Quoted 4 lines
Quoted 4 lines
x86/kvm/mmu/mmu.c +0
-0if (tdp_enabled)return ;
Quoted 8 lines
Quoted 4 lines
And that highlights the missed flush: if the PCID isn't the current PCID, then
flush_tlb_gva()neglects to flush the hardware TLB for the target PCID, which could leave a stale entry in the TLB if the guest switches to the new PCID with MOV CR3 +X86_CR3_PCID_NOFLUSH.I also don't follow this part
flush_tlb_gva()ends up executing INVVPID or INVLPGA. In both cases, all PCIDs for the VPID/ASID are flushed, right?
Ignore me, I had it in my head that flush_tlb_gva() is wired up to flush PCIDs, not VPIDs.
Re: [RFC PATCH v2 22/25] KVM: x86/mmu: Refactor kvm_mmu_invlpg() to allow skipping the gva flush
Quoted 28 lines · Sean Christopherson
Quoted 26 lines · Yosry Ahmed
Quoted 10 lines
Actually, isn't there a pre-existing over-flush when handling
kvm_mmu_invpcid_gva()? Oof, and a missed flush?To fix the over-flush, I think we want this?
x86/kvm/mmu/mmu.c +0
-0x86/kvm/mmu/mmu.c void kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_pagewalk *w, · line 6676
Quoted 6 lines
x86/kvm/mmu/mmu.c +2
-1@@ -6672,7 +6672,8 @@ void kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_pagewalk *w,if (is_noncanonical_invlpg_address (addr, vcpu))return ; -kvm_x86_call (flush_tlb_gva)(vcpu, addr); +if (roots &KVM_MMU_ROOT_CURRENT ) +kvm_x86_call (flush_tlb_gva)(vcpu, addr);Hmmm why?
AFAICT, if the gva has a shadow mapping, then the rest of
kvm_mmu_invalidate_addr()will make sure the shadow mapping is up-to-date and do any necessary TLB flushes in the process. So I am assuming the flush here is intended to cover some cases where the gva does not have a shadow mapping. Not sure how this could happen, I assume we always flush the TLB when a shadow PTE is freed.
Quoted 7 lines
Quoted 5 lines
If there are cases where we can have TLB entries but no shadow mappings, then shouldn't we flush regardless of which cached roots match the target PCID? I assume if none of the cached roots match the PCID it's not an issue because KVM will create a new root and flush the TLB before switching into the new PCID.
Gah, I conflated PCIDs and VPIDs.
Disregarding the alleged over-flush, I am actually trying to figure out why flush_tlb_gva here is needed in the INVLPG and INVPCID emulation paths. I think understanding this will help us figure out how to best handle it for INVLPGA.
Sean Christopherson
maintainerRe: [RFC PATCH v2 22/25] KVM: x86/mmu: Refactor kvm_mmu_invlpg() to allow skipping the gva flush
Quoted 35 lines · Yosry Ahmed
Quoted 34 lines · Sean Christopherson
Quoted 31 lines · Yosry Ahmed
Quoted 10 lines
Actually, isn't there a pre-existing over-flush when handling
kvm_mmu_invpcid_gva()? Oof, and a missed flush?To fix the over-flush, I think we want this?
x86/kvm/mmu/mmu.c +0
-0x86/kvm/mmu/mmu.c void kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_pagewalk *w, · line 6676
Quoted 6 lines
x86/kvm/mmu/mmu.c +2
-1@@ -6672,7 +6672,8 @@ void kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_pagewalk *w,if (is_noncanonical_invlpg_address (addr, vcpu))return ; -kvm_x86_call (flush_tlb_gva)(vcpu, addr); +if (roots &KVM_MMU_ROOT_CURRENT ) +kvm_x86_call (flush_tlb_gva)(vcpu, addr);Hmmm why?
AFAICT, if the gva has a shadow mapping, then the rest of
kvm_mmu_invalidate_addr()will make sure the shadow mapping is up-to-date and do any necessary TLB flushes in the process. So I am assuming the flush here is intended to cover some cases where the gva does not have a shadow mapping. Not sure how this could happen, I assume we always flush the TLB when a shadow PTE is freed.If there are cases where we can have TLB entries but no shadow mappings, then shouldn't we flush regardless of which cached roots match the target PCID? I assume if none of the cached roots match the PCID it's not an issue because KVM will create a new root and flush the TLB before switching into the new PCID.
Quoted 6 lines
Quoted 1 line
Gah, I conflated PCIDs and VPIDs.
Disregarding the alleged over-flush, I am actually trying to figure out why
flush_tlb_gvahere is needed in the INVLPG and INVPCID emulation paths. I think understanding this will help us figure out how to best handle it for INVLPGA.
That emits an actual hardware TLB flush if there are no unsync SPTEs. Everything below this point:
/* Invalidate shadow pages, whether GPA->GVA or nGPA->GPA. */ if (!mmu->sync_spte) return;
is specific to unsync SPTEs. __kvm_mmu_invalidate_addr() is one big nop if there aren't any unsync SPTEs. E.g. before unsync support came along, KVM didn't intercept INVLPG: a7052897b3bc ("KVM: x86: trap invlpg").
Wwith TDP enabled, KVM definitely needs to flush guest mappings, e.g. if KVM synthesizes a #PF (which are architecturally guaranteed to flush any relevant mappings), because KVM isn't controlling GVA=>GPA (obviously). When using shadow paging, it's less obvious when KVM would need to flush hardware TLBs even if KVM's SPTEs haven't changed since the last flush. The only things that jumps to mind are TLB entries that were preserved across a MOV CR3 switch, e.g. global entries or MOV CR3 with X86_CR3_PCID_NOFLUSH, and the new CR3 contains a stale mapping (or none at all).
Re: [RFC PATCH v2 22/25] KVM: x86/mmu: Refactor kvm_mmu_invlpg() to allow skipping the gva flush
In reply to Sean ChristophersonQuoted 51 lines
Quoted 42 lines
Quoted 36 lines
Quoted 27 lines
Quoted 27 lines
arch/x86/kvm/mmu/mmu.c +3
-3@@ -6615,15 +6615,15 @@ static void kvm_mmu_invalidate_addr_in_root(struct kvm_vcpu *vcpu,write_unlock (&vcpu->kvm->mmu_lock); } -void kvm_mmu_invalidate_addr (struct kvm_vcpu *vcpu,struct kvm_mmu *mmu, -u64 addr,unsigned long roots) +static void __kvm_mmu_invalidate_addr (struct kvm_vcpu *vcpu,struct kvm_mmu *mmu, +u64 addr,unsigned long roots,bool flush_gva) {int i;WARN_ON_ONCE (roots & ~KVM_MMU_ROOTS_ALL );/* It's actually a GPA for vcpu->arch.guest_mmu. */ -if (mmu != &vcpu->arch.guest_mmu) { +if (flush_gva && mmu != &vcpu->arch.guest_mmu) {/* INVLPG on a non-canonical address is a NOP according to the SDM. */ if (is_noncanonical_invlpg_address (addr, vcpu))return ;@@ -6642,9 +6642,15 @@ void kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu,kvm_mmu_invalidate_addr_in_root (vcpu, mmu, addr, mmu->prev_roots[i].hpa); } }Quoted 8 lines
arch/x86/kvm/mmu/mmu.c void kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, · line 6647
Quoted 3 lines
arch/x86/kvm/mmu/mmu.c +3
-0+ +void kvm_mmu_invalidate_addr (struct kvm_vcpu *vcpu,struct kvm_mmu *mmu, +u64 addr,unsigned long roots)Rather than
kvm_mmu_invalidate_addr()for the wrapper, what if we call thiskvm_mmu_invalidate_gva()? And thenkvm_mmu_invlpg_gva(). Then we don't need to have the "in_root" version to a quad-underscores helper, and IMO it's more obvious what's different between the one-line wrappers and the inner helpers.Quoted 5 lines
Hrm, or maybe I'm not understanding what "
flush_gva" means. At first glance, I was assuming you were using it to differentiate between GVA and GPA, but IIUC, it's literally skipping the flush for the current context, which just so happens to be done only for GVAs. I'd still like to avoid the "in_root" helper, if at all possible.Quoted 8 lines
Yes, it's skipping the TLB flush that is only needed for GVAs. The alternative I had in mind (but thought was worse) was to refactor the TLB flush part out of
kvm_mmu_invalidate_addr()(or__kvm_mmu_invalidate_addr()) in this patch instead of adding a boolean, but this still requires adding a wrapper and the possibility of a quad-underscore helper.What's the main objection to
kvm_mmu_invalidate_addr_in_root()?
Quoted 8 lines
I don't love the
__kvm_mmu_invalidate_addr()=>kvm_mmu_invalidate_addr_in_root()callchain. It's not at all obvious that thein_root()helper shouldn't be called directly. I don't hate it, but I do think we need better clarity on what all this is doing.E.g. when looking at
__kvm_inject_emulated_page_fault(), since it hardcodes a single root, it's a bit headscratching to usekvm_mmu_invalidate_addr()instead ofkvm_mmu_invalidate_addr_in_root.
Coming back to this, I agree it's confusing. I think this can be fixed with a better name though. Looking at kvm_mmu_invalidate_addr_in_root() (or __kvm_mmu_invalidate_addr() in current code), seems like what it does is find SPTEs for that address, sync them, and flush the TLB if needed.
So maybe mmu_sync_addr_sptes() or mmu_sync_and_flush_addr_sptes()?
I think this would make it obvious it is not a direct alternative to kvm_mmu_invalidate_addr() that just operates on a single root.
Quoted 3 lines
Hmm, and arguably, the way
invlpga_interception()handles the ASID is flat out wrong. KVM doesn't need to flush *all* roots, rather it needs to flush L1 roots for ASID=0, and L2 roots for ASID!=0.
I wouldn't say it's wrong. It syncs more roots than needed, yes, and I did document that in the comment. I didn't want to spend time optimizing this because it only matters if NPT=0 in the host (and probably L1 will only use INVLPGA if it also has NPT=0).
I think KVM generally over-syncs in multiple cases. For example, emulating INVVPID on a single address flushes the entire root (see handle_invvpid()), and I don't think anyone cared enough to fix that. Also, emulating INVLPG and INVPCID syncs all roots, arguably it should only sync the ones' matching the vCPUs current guest_mode?
Quoted 2 lines
If we can figure out an elegant way to
express and handle that, it should naturally handle the "flush GVA" aspect.
Hmm how? I think the way to properly do this is to filter roots based on role.guest_mode, like kvm_mmu_free_guest_mode_roots(). I'd probably refactor that logic out of kvm_mmu_free_guest_mode_roots() and reuse it if we go this route. But this doesn't really help with the "flush GVA" aspect.
One way to actually tackle both is to pass "guest_mode" to kvm_invalidate_mmu_addr() as a boolean (yuck). Then, it can do the root filtering inside kvm_invalidate_mmu_addr(). We can then plump guest_mode into flush_tlb_gva, and handle flushing L1's ASID vs L2's ASID completely within flush_tlb_gva.
But then we'll have to figure out what's the right value of that boolean to pass from all callers of kvm_invalidate_mmu_addr() and there's more things we can get wrong.
Sean Christopherson
maintainerRe: [RFC PATCH v2 22/25] KVM: x86/mmu: Refactor kvm_mmu_invlpg() to allow skipping the gva flush
Quoted 60 lines · Yosry Ahmed
Quoted 51 lines
Quoted 42 lines
Quoted 36 lines
Quoted 27 lines
Quoted 27 lines
arch/x86/kvm/mmu/mmu.c +3
-3@@ -6615,15 +6615,15 @@ static void kvm_mmu_invalidate_addr_in_root(struct kvm_vcpu *vcpu,write_unlock (&vcpu->kvm->mmu_lock); } -void kvm_mmu_invalidate_addr (struct kvm_vcpu *vcpu,struct kvm_mmu *mmu, -u64 addr,unsigned long roots) +static void __kvm_mmu_invalidate_addr (struct kvm_vcpu *vcpu,struct kvm_mmu *mmu, +u64 addr,unsigned long roots,bool flush_gva) {int i;WARN_ON_ONCE (roots & ~KVM_MMU_ROOTS_ALL );/* It's actually a GPA for vcpu->arch.guest_mmu. */ -if (mmu != &vcpu->arch.guest_mmu) { +if (flush_gva && mmu != &vcpu->arch.guest_mmu) {/* INVLPG on a non-canonical address is a NOP according to the SDM. */ if (is_noncanonical_invlpg_address (addr, vcpu))return ;@@ -6642,9 +6642,15 @@ void kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu,kvm_mmu_invalidate_addr_in_root (vcpu, mmu, addr, mmu->prev_roots[i].hpa); } }Quoted 8 lines
arch/x86/kvm/mmu/mmu.c void kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, · line 6647
Quoted 3 lines
arch/x86/kvm/mmu/mmu.c +3
-0+ +void kvm_mmu_invalidate_addr (struct kvm_vcpu *vcpu,struct kvm_mmu *mmu, +u64 addr,unsigned long roots)Rather than
kvm_mmu_invalidate_addr()for the wrapper, what if we call thiskvm_mmu_invalidate_gva()? And thenkvm_mmu_invlpg_gva(). Then we don't need to have the "in_root" version to a quad-underscores helper, and IMO it's more obvious what's different between the one-line wrappers and the inner helpers.Quoted 5 lines
Hrm, or maybe I'm not understanding what "
flush_gva" means. At first glance, I was assuming you were using it to differentiate between GVA and GPA, but IIUC, it's literally skipping the flush for the current context, which just so happens to be done only for GVAs. I'd still like to avoid the "in_root" helper, if at all possible.Quoted 8 lines
Yes, it's skipping the TLB flush that is only needed for GVAs. The alternative I had in mind (but thought was worse) was to refactor the TLB flush part out of
kvm_mmu_invalidate_addr()(or__kvm_mmu_invalidate_addr()) in this patch instead of adding a boolean, but this still requires adding a wrapper and the possibility of a quad-underscore helper.What's the main objection to
kvm_mmu_invalidate_addr_in_root()?Quoted 8 lines
I don't love the
__kvm_mmu_invalidate_addr()=>kvm_mmu_invalidate_addr_in_root()callchain. It's not at all obvious that thein_root()helper shouldn't be called directly. I don't hate it, but I do think we need better clarity on what all this is doing.E.g. when looking at
__kvm_inject_emulated_page_fault(), since it hardcodes a single root, it's a bit headscratching to usekvm_mmu_invalidate_addr()instead ofkvm_mmu_invalidate_addr_in_root.
Quoted 7 lines
Coming back to this, I agree it's confusing. I think this can be fixed with a better name though. Looking at
kvm_mmu_invalidate_addr_in_root()(or__kvm_mmu_invalidate_addr()in current code), seems like what it does is find SPTEs for that address, sync them, and flush the TLB if needed.So maybe
mmu_sync_addr_sptes()ormmu_sync_and_flush_addr_sptes()?
kvm_mmu_sync_addr()? The "sptes" part is implied in things like kvm_sync_page() and mmu_sync_children().
Re: [RFC PATCH v2 22/25] KVM: x86/mmu: Refactor kvm_mmu_invlpg() to allow skipping the gva flush
Quoted 68 lines · Sean Christopherson
Quoted 66 lines · Yosry Ahmed
Quoted 51 lines
Quoted 42 lines
Quoted 36 lines
Quoted 27 lines
Quoted 27 lines
arch/x86/kvm/mmu/mmu.c +3
-3@@ -6615,15 +6615,15 @@ static void kvm_mmu_invalidate_addr_in_root(struct kvm_vcpu *vcpu,write_unlock (&vcpu->kvm->mmu_lock); } -void kvm_mmu_invalidate_addr (struct kvm_vcpu *vcpu,struct kvm_mmu *mmu, -u64 addr,unsigned long roots) +static void __kvm_mmu_invalidate_addr (struct kvm_vcpu *vcpu,struct kvm_mmu *mmu, +u64 addr,unsigned long roots,bool flush_gva) {int i;WARN_ON_ONCE (roots & ~KVM_MMU_ROOTS_ALL );/* It's actually a GPA for vcpu->arch.guest_mmu. */ -if (mmu != &vcpu->arch.guest_mmu) { +if (flush_gva && mmu != &vcpu->arch.guest_mmu) {/* INVLPG on a non-canonical address is a NOP according to the SDM. */ if (is_noncanonical_invlpg_address (addr, vcpu))return ;@@ -6642,9 +6642,15 @@ void kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu,kvm_mmu_invalidate_addr_in_root (vcpu, mmu, addr, mmu->prev_roots[i].hpa); } }Quoted 8 lines
arch/x86/kvm/mmu/mmu.c void kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, · line 6647
Quoted 3 lines
arch/x86/kvm/mmu/mmu.c +3
-0+ +void kvm_mmu_invalidate_addr (struct kvm_vcpu *vcpu,struct kvm_mmu *mmu, +u64 addr,unsigned long roots)Rather than
kvm_mmu_invalidate_addr()for the wrapper, what if we call thiskvm_mmu_invalidate_gva()? And thenkvm_mmu_invlpg_gva(). Then we don't need to have the "in_root" version to a quad-underscores helper, and IMO it's more obvious what's different between the one-line wrappers and the inner helpers.Quoted 5 lines
Hrm, or maybe I'm not understanding what "
flush_gva" means. At first glance, I was assuming you were using it to differentiate between GVA and GPA, but IIUC, it's literally skipping the flush for the current context, which just so happens to be done only for GVAs. I'd still like to avoid the "in_root" helper, if at all possible.Quoted 8 lines
Yes, it's skipping the TLB flush that is only needed for GVAs. The alternative I had in mind (but thought was worse) was to refactor the TLB flush part out of
kvm_mmu_invalidate_addr()(or__kvm_mmu_invalidate_addr()) in this patch instead of adding a boolean, but this still requires adding a wrapper and the possibility of a quad-underscore helper.What's the main objection to
kvm_mmu_invalidate_addr_in_root()?Quoted 8 lines
I don't love the
__kvm_mmu_invalidate_addr()=>kvm_mmu_invalidate_addr_in_root()callchain. It's not at all obvious that thein_root()helper shouldn't be called directly. I don't hate it, but I do think we need better clarity on what all this is doing.E.g. when looking at
__kvm_inject_emulated_page_fault(), since it hardcodes a single root, it's a bit headscratching to usekvm_mmu_invalidate_addr()instead ofkvm_mmu_invalidate_addr_in_root.Coming back to this, I agree it's confusing. I think this can be fixed with a better name though. Looking at
kvm_mmu_invalidate_addr_in_root()(or__kvm_mmu_invalidate_addr()in current code), seems like what it does is find SPTEs for that address, sync them, and flush the TLB if needed.
Quoted 4 lines
Quoted 1 line
So maybe
mmu_sync_addr_sptes()ormmu_sync_and_flush_addr_sptes()?
kvm_mmu_sync_addr()? The "sptes" part is implied in things likekvm_sync_page()andmmu_sync_children().
Sounds good. I assume you're okay with keeping everything else the same way? I assume you didn't like any of the alternatives?
Sean Christopherson
maintainerRe: [RFC PATCH v2 22/25] KVM: x86/mmu: Refactor kvm_mmu_invlpg() to allow skipping the gva flush
Quoted 19 lines · Yosry Ahmed
Quoted 18 lines · Sean Christopherson
Quoted 17 lines · Yosry Ahmed
Quoted 10 lines
Quoted 1 line
What's the main objection to
kvm_mmu_invalidate_addr_in_root()?I don't love the
__kvm_mmu_invalidate_addr()=>kvm_mmu_invalidate_addr_in_root()callchain. It's not at all obvious that thein_root()helper shouldn't be called directly. I don't hate it, but I do think we need better clarity on what all this is doing.E.g. when looking at
__kvm_inject_emulated_page_fault(), since it hardcodes a single root, it's a bit headscratching to usekvm_mmu_invalidate_addr()instead ofkvm_mmu_invalidate_addr_in_root.Coming back to this, I agree it's confusing. I think this can be fixed with a better name though. Looking at
kvm_mmu_invalidate_addr_in_root()(or__kvm_mmu_invalidate_addr()in current code), seems like what it does is find SPTEs for that address, sync them, and flush the TLB if needed.
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
So maybe
mmu_sync_addr_sptes()ormmu_sync_and_flush_addr_sptes()?
kvm_mmu_sync_addr()? The "sptes" part is implied in things likekvm_sync_page()andmmu_sync_children().Sounds good. I assume you're okay with keeping everything else the same way? I assume you didn't like any of the alternatives?
Ya, I don't want risk delaying this series chasing down flaws that no one cares about in practice.
RFC PATCH v2 23/25 KVM: nSVM: Flush L2's ASID when emulating INVLPGA
In reply to Yosry AhmedKVM currently handles INVLPGA in the same way as INVLPG, flushing L1's own ASID. This is currently correct because L1 and L2 share an ASID, but it doesn't work once they have separate ASIDs.
If L1 is flushing a different L2 ASID than the one KVM is tracking, do nothing, as KVM will flush L2's ASID in hardware (and sync the MMU if needed) when L1 switches to the target ASID.
If L1 is flushing its own ASID, handle the flush the same as INVLPG. Otherwise, skip the GVA flush for the current context (L1's ASID), and flush the L2 ASID in hardware using INVLPGA if running on the same CPU as L2. If not, fallback to a VMCB-based ASID flush. Note that if L2 then runs on a different CPU KVM will flush the ASID anyway.
Either way, sync the MMU if NPT is disabled, which is handled by __kvm_mmu_invlpg(). Note that all MMU roots are sync'd when NPT is disabled, which can be optimized by keying off guest_mode to only sync the appropriate context (L1 vs. L2).
--- arch/x86/include/asm/kvm_host.h | 3 +++ arch/x86/kvm/mmu/mmu.c | 8 ++++--- arch/x86/kvm/svm/svm.c | 38 +++++++++++++++++++++++++++++++-- 3 files changed, 44 insertions(+), 5 deletions(-)
arch/x86/include/asm/kvm_host.h +3 -0
@@ -2385,7 +2385,10 @@ static inline void kvm_dec_apicv_irq_window_req(struct kvm *kvm)int kvm_mmu_page_fault (struct kvm_vcpu *vcpu,gpa_t cr2_or_gpa,u64 error_code,void *insn,int insn_len);void kvm_mmu_print_sptes (struct kvm_vcpu *vcpu,gpa_t gpa,const char *msg); +void __kvm_mmu_invlpg (struct kvm_vcpu *vcpu,gva_t gva,bool flush_gva);void kvm_mmu_invlpg (struct kvm_vcpu *vcpu,gva_t gva); +void __kvm_mmu_invalidate_addr (struct kvm_vcpu *vcpu,struct kvm_mmu *mmu, +u64 addr,unsigned long roots,bool flush_gva);void kvm_mmu_invalidate_addr (struct kvm_vcpu *vcpu,struct kvm_mmu *mmu,u64 addr,unsigned long roots);void kvm_mmu_invpcid_gva (struct kvm_vcpu *vcpu,gva_t gva,unsigned long pcid);
arch/x86/kvm/mmu/mmu.c +5 -3
@@ -6615,8 +6615,8 @@ static void kvm_mmu_invalidate_addr_in_root(struct kvm_vcpu *vcpu,write_unlock (&vcpu->kvm->mmu_lock); } -static void __kvm_mmu_invalidate_addr (struct kvm_vcpu *vcpu,struct kvm_mmu *mmu, -u64 addr,unsigned long roots,bool flush_gva) +void __kvm_mmu_invalidate_addr (struct kvm_vcpu *vcpu,struct kvm_mmu *mmu, +u64 addr,unsigned long roots,bool flush_gva) {int i;
@@ -6642,6 +6642,7 @@ static void __kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_mmu *mmukvm_mmu_invalidate_addr_in_root (vcpu, mmu, addr, mmu->prev_roots[i].hpa); } } +EXPORT_SYMBOL_FOR_KVM_INTERNAL (__kvm_mmu_invalidate_addr);void kvm_mmu_invalidate_addr (struct kvm_vcpu *vcpu,struct kvm_mmu *mmu,u64 addr,unsigned long roots)
@@ -6650,7 +6651,7 @@ void kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, }EXPORT_SYMBOL_FOR_KVM_INTERNAL (kvm_mmu_invalidate_addr); -static void __kvm_mmu_invlpg (struct kvm_vcpu *vcpu,gva_t gva,bool flush_gva) +void __kvm_mmu_invlpg (struct kvm_vcpu *vcpu,gva_t gva,bool flush_gva) {/* * INVLPG is required to invalidate any global mappings for the VA,
@@ -6666,6 +6667,7 @@ static void __kvm_mmu_invlpg(struct kvm_vcpu *vcpu, gva_t gva, bool flush_gva) KVM_MMU_ROOTS_ALL, flush_gva); ++vcpu->stat.invlpg; } +EXPORT_SYMBOL_FOR_KVM_INTERNAL (__kvm_mmu_invlpg);void kvm_mmu_invlpg (struct kvm_vcpu *vcpu,gva_t gva) {
arch/x86/kvm/svm/svm.c +36 -2
@@ -2404,17 +2404,51 @@ static int clgi_interception(struct kvm_vcpu *vcpu)static int invlpga_interception (struct kvm_vcpu *vcpu) { +struct vcpu_svm *svm =to_svm (vcpu);/* FIXME: Handle an address size prefix. */ gva_t gva =kvm_rax_read (vcpu);u32 asid =kvm_ecx_read (vcpu); +int cpu;if (nested_svm_check_permissions (vcpu))return 1 ;trace_kvm_invlpga (to_svm (vcpu)->vmcb->save.rip, asid, gva); -/* Let's treat INVLPGA the same as INVLPG (can be optimized!) */ -kvm_mmu_invlpg (vcpu, gva); +/* +* INVLPG on a non-canonical address is a NOP according to the SDM, +* assumethe same behavior from INVLPGA since the APM doesn't specify. +*/ +if (is_noncanonical_invlpg_address (gva, vcpu)) +return kvm_skip_emulated_instruction (vcpu); + +/* +* Do nothing if L1 is flushing a different L2 ASID than the one KVM is +* currently tracking. KVM tracks a single L2 ASID, and performs a TLB +* flush (and MMU resync if needed) when L1 switches ASIDs anyway. +*/ +if (asid && asid != svm->nested.last_asid) +return kvm_skip_emulated_instruction (vcpu); + +/* +* Handle INVLPGA similar to INVLPG, with one caveat. If the specified +* ASID is non-zero (i.e. L1 is not flushing it's own ASID), skip +* flushing the TLB for the current context (L1's), and use INVLPGA to +* flush L2's ASID in hardware if running on the same CPU (otherwise +* fallback to a full ASID flush). +* +* Note, if NPT is disabled, this will sync all the shadow page tables. +* This can be optimized by keying off guest_mode. +*/ +__kvm_mmu_invlpg (vcpu, gva, !asid); +if (asid) { + cpu =get_cpu (); +if (cpu == svm->nested.vmcb02.cpu) +invlpga (gva, svm->nested.asid02); +else +vmcb_set_flush_asid (svm->nested.vmcb02.ptr); +put_cpu (); + }return kvm_skip_emulated_instruction (vcpu); }
Signature · 1 line
2.54.0.1136.gdb2ca164c4-goog
RFC PATCH v2 24/25 KVM: nSVM: Use different ASIDs for L1 and L2
In reply to Yosry AhmedNow that TLB flushes are properly handled and tracked for L1 vs L2 ASIDs, allocate a separate new ASID for L2 for each vCPU, similar to how VMX handles VPIDs. Drop the unconditional flushes and syncs on nested transitions.
On a Turin CPU, this results in 8-15% performance boost in CPUID rate microbenchmark [1] and netperf TCP_RR latency/throughput.
[1]https://lore.kernel.org/kvm/[email protected]/
--- arch/x86/kvm/svm/nested.c | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-)
arch/x86/kvm/svm/nested.c +3 -8
@@ -718,10 +718,6 @@ static void nested_svm_entry_tlb_flush(struct kvm_vcpu *vcpu)kvm_make_request (KVM_REQ_MMU_SYNC , vcpu);kvm_make_request (KVM_REQ_TLB_FLUSH_GUEST , vcpu); } - -/* TODO: optimize unconditional TLB flush/MMU sync */ -kvm_make_request (KVM_REQ_MMU_SYNC , vcpu); -kvm_make_request (KVM_REQ_TLB_FLUSH_CURRENT , vcpu); }static void nested_svm_exit_tlb_flush (struct kvm_vcpu *vcpu)
@@ -733,9 +729,6 @@ static void nested_svm_exit_tlb_flush(struct kvm_vcpu *vcpu)/* Flush L1's own ASID if it request a *full* TLB flush on VMRUN */ if (svm->nested.ctl.tlb_ctl ==TLB_CONTROL_FLUSH_ALL_ASID )kvm_make_request (KVM_REQ_TLB_FLUSH_GUEST , vcpu); - -kvm_make_request (KVM_REQ_MMU_SYNC , vcpu); -kvm_make_request (KVM_REQ_TLB_FLUSH_CURRENT , vcpu); }static void svm_switch_vmcb (struct vcpu_svm *svm,struct kvm_vmcb_info *target_vmcb)
@@ -1539,7 +1532,7 @@ int svm_allocate_nested(struct vcpu_svm *svm)if (!svm->nested.msrpm) goto err_free_vmcb02; - svm->nested.asid02 = svm->asid; + svm->nested.asid02 =allocate_asid (); svm->nested.initialized = true;return 0 ;
@@ -1557,6 +1550,8 @@ void svm_free_nested(struct vcpu_svm *svm)if (WARN_ON_ONCE (svm->vmcb != svm->vmcb01.ptr))svm_switch_vmcb (svm, &svm->vmcb01); +free_asid (svm->nested.asid02); +svm_vcpu_free_msrpm (svm->nested.msrpm); svm->nested.msrpm =NULL ;
Signature · 1 line
2.54.0.1136.gdb2ca164c4-goog
[email protected]
AI reviewRe: [RFC PATCH v2 24/25] KVM: nSVM: Use different ASIDs for L1 and L2
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
New issues:
- [Critical] Failure to reset svm->nested.last_asid during nested SVM teardown allows L2 guests to run with stale TLB entries from recycled ASIDs.
Pre-existing issues:
- [Critical] Dangling pointer svm->nested.vmcb02.ptr leaked on the svm_allocate_nested() error path leads to a Use-After-Free write during TLB flushes. --
commit 0fdb61625f72b8737a6f7fe184ab78e2c15d5576
Author: Yosry Ahmed <[email protected]>
KVM: nSVM: Use different ASIDs for L1 and L2
This patch modifies nested SVM to allocate a separate hardware ASID for L2 for each vCPU. It drops the unconditional TLB flushes and MMU syncs on nested transitions, replacing them with proper tracking for L1 vs L2 ASIDs.
Quoted 4 lines
arch/x86/kvm/svm/nested.c +0
-0
[ ... ]
Quoted 11 lines
arch/x86/kvm/svm/nested.c +1
-1@@ -1539,7 +1532,7 @@ int svm_allocate_nested(struct vcpu_svm *svm)if (!svm->nested.msrpm) goto err_free_vmcb02; - svm->nested.asid02 = svm->asid; + svm->nested.asid02 =allocate_asid (); svm->nested.initialized = true;return 0 ;
err_free_vmcb02:
arch/x86/kvm/svm/nested.c
Quoted 3 lines
arch/x86/kvm/svm/nested.c +0
-0__free_page (vmcb02_page);return -ENOMEM ; }
[Severity: Critical]
This isn't a bug introduced by this patch, but does the error path here leave a dangling pointer in svm->nested.vmcb02.ptr?
When msrpm allocation fails, the vmcb02_page is freed, but svm->nested.vmcb02.ptr is not explicitly cleared. If a subsequent TLB flush occurs on this vCPU, for example via MMU notifiers:
svm_flush_tlb_all() {
...
if (svm->nested.vmcb02.ptr)
vmcb_set_flush_asid(svm->nested.vmcb02.ptr);
}Could this result in a use-after-free write that corrupts the freed memory?
[ ... ]
arch/x86/kvm/svm/nested.c void svm_free_nested(struct vcpu_svm *svm) · line 1554
Quoted 6 lines
arch/x86/kvm/svm/nested.c +2
-0@@ -1557,6 +1550,8 @@ void svm_free_nested(struct vcpu_svm *svm)if (WARN_ON_ONCE (svm->vmcb != svm->vmcb01.ptr))svm_switch_vmcb (svm, &svm->vmcb01); +free_asid (svm->nested.asid02); +
[Severity: Critical]
When the hardware ASID is freed here during nested SVM teardown, does svm->nested.last_asid also need to be reset to 0?
If L1 disables SVM, svm_free_nested() is called, freeing asid02. This ASID tag can then be reallocated. If L1 later re-enables SVM and uses the same L1-managed ASID, nested_svm_entry_tlb_flush() will see:
nested_svm_entry_tlb_flush() {
...
if (svm->nested.ctl.asid != svm->nested.last_asid) {
svm->nested.last_asid = svm->nested.ctl.asid;
new_asid = true;
}
...
}This check would fail to set new_asid = true. Since this patch removes the unconditional TLB flush on nested entry, would this regression allow the L2 guest to run with stale TLB entries from the previous owner of the reallocated hardware ASID?
Quoted 3 lines
arch/x86/kvm/svm/nested.c +0
-0svm_vcpu_free_msrpm (svm->nested.msrpm); svm->nested.msrpm =NULL ;
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=24
Re: [RFC PATCH v2 24/25] KVM: nSVM: Use different ASIDs for L1 and L2
[Omitting the previously reported issue]
Quoted 11 lines
arch/x86/kvm/svm/nested.c void svm_free_nested(struct vcpu_svm *svm) · line 1554
Quoted 6 lines
arch/x86/kvm/svm/nested.c +2
-0@@ -1557,6 +1550,8 @@ void svm_free_nested(struct vcpu_svm *svm)if (WARN_ON_ONCE (svm->vmcb != svm->vmcb01.ptr))svm_switch_vmcb (svm, &svm->vmcb01); +free_asid (svm->nested.asid02); +[Severity: Critical]
When the hardware ASID is freed here during nested SVM teardown, doessvm->nested.last_asidalso need to be reset to 0?
Quoted 3 lines
If L1 disables SVM,
svm_free_nested()is called, freeing asid02. This ASID tag can then be reallocated. If L1 later re-enables SVM and uses the same L1-managed ASID,nested_svm_entry_tlb_flush()will see:
Hmm if L1 reuses the same ASID, it's up to L1 to explicitly flush it if needed.
That being said, KVM needs to flush ASIDs when first allocated anyway, otherwise we risk reusing TLB entries from a completely different vCPU or VM. This series correctly does so for L1 (calls vmcb_set_flush_asid() when initializing vmcb01's ASID), but not for L2.
So yes, we need to make sure asid02 is always initialized (and reset) to 0, but not for the reason Sashiko mentioned. It's so that the first nested VMRUN using an L2 ASID always does a flush (as L1 is not allowed to use ASID 0).
Quoted 21 lines
nested_svm_entry_tlb_flush() { ... if (svm->nested.ctl.asid != svm->nested.last_asid) { svm->nested.last_asid = svm->nested.ctl.asid; new_asid = true; } ... }This check would fail to set
new_asid= true. Since this patch removes the unconditional TLB flush on nested entry, would this regression allow the L2 guest to run with stale TLB entries from the previous owner of the reallocated hardware ASID?arch/x86/kvm/svm/nested.c void svm_free_nested(struct vcpu_svm *svm) · line 1557
Quoted 3 lines
arch/x86/kvm/svm/nested.c +0
-0svm_vcpu_free_msrpm (svm->nested.msrpm); svm->nested.msrpm =NULL ;--
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=24
Re: [RFC PATCH v2 24/25] KVM: nSVM: Use different ASIDs for L1 and L2
Quoted 20 lines · Yosry Ahmed
[Omitting the previously reported issue]
Quoted 11 lines
arch/x86/kvm/svm/nested.c void svm_free_nested(struct vcpu_svm *svm) · line 1554
Quoted 6 lines
arch/x86/kvm/svm/nested.c +2
-0@@ -1557,6 +1550,8 @@ void svm_free_nested(struct vcpu_svm *svm)if (WARN_ON_ONCE (svm->vmcb != svm->vmcb01.ptr))svm_switch_vmcb (svm, &svm->vmcb01); +free_asid (svm->nested.asid02); +[Severity: Critical]
When the hardware ASID is freed here during nested SVM teardown, doessvm->nested.last_asidalso need to be reset to 0?Quoted 3 lines
If L1 disables SVM,
svm_free_nested()is called, freeing asid02. This ASID tag can then be reallocated. If L1 later re-enables SVM and uses the same L1-managed ASID,nested_svm_entry_tlb_flush()will see:Hmm if L1 reuses the same ASID, it's up to L1 to explicitly flush it if needed.
Quoted 7 lines
That being said, KVM needs to flush ASIDs when first allocated anyway, otherwise we risk reusing TLB entries from a completely different vCPU or VM. This series correctly does so for L1 (calls
vmcb_set_flush_asid()when initializing vmcb01's ASID), but not for L2.So yes, we need to make sure asid02 is always initialized (and reset)
last_asid* not asid02.
Quoted 3 lines
to 0, but not for the reason Sashiko mentioned. It's so that the first nested VMRUN using an L2 ASID always does a flush (as L1 is not allowed to use ASID 0).
Re: [RFC PATCH v2 24/25] KVM: nSVM: Use different ASIDs for L1 and L2
In reply to Yosry AhmedQuoted 26 lines · Yosry Ahmed
[Omitting the previously reported issue]
Quoted 11 lines
arch/x86/kvm/svm/nested.c void svm_free_nested(struct vcpu_svm *svm) · line 1554
Quoted 6 lines
arch/x86/kvm/svm/nested.c +2
-0@@ -1557,6 +1550,8 @@ void svm_free_nested(struct vcpu_svm *svm)if (WARN_ON_ONCE (svm->vmcb != svm->vmcb01.ptr))svm_switch_vmcb (svm, &svm->vmcb01); +free_asid (svm->nested.asid02); +[Severity: Critical]
When the hardware ASID is freed here during nested SVM teardown, doessvm->nested.last_asidalso need to be reset to 0?Quoted 3 lines
If L1 disables SVM,
svm_free_nested()is called, freeing asid02. This ASID tag can then be reallocated. If L1 later re-enables SVM and uses the same L1-managed ASID,nested_svm_entry_tlb_flush()will see:Hmm if L1 reuses the same ASID, it's up to L1 to explicitly flush it if needed.
That being said, KVM needs to flush ASIDs when first allocated anyway, otherwise we risk reusing TLB entries from a completely different vCPU or VM. This series correctly does so for L1 (calls
vmcb_set_flush_asid()when initializing vmcb01's ASID), but not for L2.
Quoted 4 lines
So yes, we need to make sure asid02 is always initialized (and reset) to 0, but not for the reason Sashiko mentioned. It's so that the first nested VMRUN using an L2 ASID always does a flush (as L1 is not allowed to use ASID 0).
I think VMX might have the same problem.
Re: [RFC PATCH v2 24/25] KVM: nSVM: Use different ASIDs for L1 and L2
Quoted 28 lines · Yosry Ahmed
Quoted 26 lines · Yosry Ahmed
[Omitting the previously reported issue]
Quoted 11 lines
arch/x86/kvm/svm/nested.c void svm_free_nested(struct vcpu_svm *svm) · line 1554
Quoted 6 lines
arch/x86/kvm/svm/nested.c +2
-0@@ -1557,6 +1550,8 @@ void svm_free_nested(struct vcpu_svm *svm)if (WARN_ON_ONCE (svm->vmcb != svm->vmcb01.ptr))svm_switch_vmcb (svm, &svm->vmcb01); +free_asid (svm->nested.asid02); +[Severity: Critical]
When the hardware ASID is freed here during nested SVM teardown, doessvm->nested.last_asidalso need to be reset to 0?Quoted 3 lines
If L1 disables SVM,
svm_free_nested()is called, freeing asid02. This ASID tag can then be reallocated. If L1 later re-enables SVM and uses the same L1-managed ASID,nested_svm_entry_tlb_flush()will see:Hmm if L1 reuses the same ASID, it's up to L1 to explicitly flush it if needed.
That being said, KVM needs to flush ASIDs when first allocated anyway, otherwise we risk reusing TLB entries from a completely different vCPU or VM. This series correctly does so for L1 (calls
vmcb_set_flush_asid()when initializing vmcb01's ASID), but not for L2.
Quoted 6 lines
Quoted 4 lines
So yes, we need to make sure asid02 is always initialized (and reset) to 0, but not for the reason Sashiko mentioned. It's so that the first nested VMRUN using an L2 ASID always does a flush (as L1 is not allowed to use ASID 0).
I think VMX might have the same problem.
This code *should* take care of it, since vmcs12->virtual_processor_id can't be 0 if it's being used, and vmx->nested_last_vpid is initialized to 0:
Quoted 5 lines
if (is_vmenter && vmcs12->virtual_processor_id != vmx->nested.last_vpid) {
vmx->nested.last_vpid = vmcs12->virtual_processor_id;
kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu);
return;
}However, Gemini points out that last_vpid is not cleared when L1 does a VMXOFF, even though vmx->nested.vpid02 is freed.
Per the exhortation in security-bugs.rst, I am treating this issue as public.
Re: [RFC PATCH v2 24/25] KVM: nSVM: Use different ASIDs for L1 and L2
Quoted 47 lines · Jim Mattson
Quoted 28 lines · Yosry Ahmed
Quoted 26 lines · Yosry Ahmed
[Omitting the previously reported issue]
Quoted 11 lines
arch/x86/kvm/svm/nested.c void svm_free_nested(struct vcpu_svm *svm) · line 1554
Quoted 6 lines
arch/x86/kvm/svm/nested.c +2
-0@@ -1557,6 +1550,8 @@ void svm_free_nested(struct vcpu_svm *svm)if (WARN_ON_ONCE (svm->vmcb != svm->vmcb01.ptr))svm_switch_vmcb (svm, &svm->vmcb01); +free_asid (svm->nested.asid02); +[Severity: Critical]
When the hardware ASID is freed here during nested SVM teardown, doessvm->nested.last_asidalso need to be reset to 0?Quoted 3 lines
If L1 disables SVM,
svm_free_nested()is called, freeing asid02. This ASID tag can then be reallocated. If L1 later re-enables SVM and uses the same L1-managed ASID,nested_svm_entry_tlb_flush()will see:Hmm if L1 reuses the same ASID, it's up to L1 to explicitly flush it if needed.
That being said, KVM needs to flush ASIDs when first allocated anyway, otherwise we risk reusing TLB entries from a completely different vCPU or VM. This series correctly does so for L1 (calls
vmcb_set_flush_asid()when initializing vmcb01's ASID), but not for L2.Quoted 6 lines
Quoted 4 lines
So yes, we need to make sure asid02 is always initialized (and reset) to 0, but not for the reason Sashiko mentioned. It's so that the first nested VMRUN using an L2 ASID always does a flush (as L1 is not allowed to use ASID 0).
I think VMX might have the same problem.
This code *should* take care of it, since
vmcs12->virtual_processor_idcan't be 0 if it's being used, andvmx->nested_last_vpidis initialized to 0:Quoted 5 lines
if (is_vmenter && vmcs12->virtual_processor_id != vmx->nested.last_vpid) { vmx->nested.last_vpid = vmcs12->virtual_processor_id; kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu); return; }
Quoted 4 lines
However, Gemini points out that
last_vpidis not cleared when L1 does a VMXOFF, even thoughvmx->nested.vpid02is freed.Per the exhortation in
security-bugs.rst, I am treating this issue as public.
Yeah both last_vpid (and last_asid) should be reset to 0 when nested is turned off (VMXOFF or clearing EFER.SVME), we should probably also document that this is required to ensure a TLB flush on the first usage of the ASID/VPID.
Re: [RFC PATCH v2 24/25] KVM: nSVM: Use different ASIDs for L1 and L2
Quoted 52 lines · Yosry Ahmed
Quoted 50 lines · Jim Mattson
Quoted 28 lines · Yosry Ahmed
Quoted 26 lines · Yosry Ahmed
[Omitting the previously reported issue]
Quoted 11 lines
arch/x86/kvm/svm/nested.c void svm_free_nested(struct vcpu_svm *svm) · line 1554
Quoted 6 lines
arch/x86/kvm/svm/nested.c +2
-0@@ -1557,6 +1550,8 @@ void svm_free_nested(struct vcpu_svm *svm)if (WARN_ON_ONCE (svm->vmcb != svm->vmcb01.ptr))svm_switch_vmcb (svm, &svm->vmcb01); +free_asid (svm->nested.asid02); +[Severity: Critical]
When the hardware ASID is freed here during nested SVM teardown, doessvm->nested.last_asidalso need to be reset to 0?Quoted 3 lines
If L1 disables SVM,
svm_free_nested()is called, freeing asid02. This ASID tag can then be reallocated. If L1 later re-enables SVM and uses the same L1-managed ASID,nested_svm_entry_tlb_flush()will see:Hmm if L1 reuses the same ASID, it's up to L1 to explicitly flush it if needed.
That being said, KVM needs to flush ASIDs when first allocated anyway, otherwise we risk reusing TLB entries from a completely different vCPU or VM. This series correctly does so for L1 (calls
vmcb_set_flush_asid()when initializing vmcb01's ASID), but not for L2.Quoted 6 lines
Quoted 4 lines
So yes, we need to make sure asid02 is always initialized (and reset) to 0, but not for the reason Sashiko mentioned. It's so that the first nested VMRUN using an L2 ASID always does a flush (as L1 is not allowed to use ASID 0).
I think VMX might have the same problem.
This code *should* take care of it, since
vmcs12->virtual_processor_idcan't be 0 if it's being used, andvmx->nested_last_vpidis initialized to 0:Quoted 5 lines
if (is_vmenter && vmcs12->virtual_processor_id != vmx->nested.last_vpid) { vmx->nested.last_vpid = vmcs12->virtual_processor_id; kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu); return; }However, Gemini points out that
last_vpidis not cleared when L1 does a VMXOFF, even thoughvmx->nested.vpid02is freed.
Quoted 6 lines
Quoted 1 line
Per the exhortation in
security-bugs.rst, I am treating this issue as public.Yeah both
last_vpid(andlast_asid) should be reset to 0 when nested is turned off (VMXOFF or clearing EFER.SVME), we should probably also document that this is required to ensure a TLB flush on the first usage of the ASID/VPID.
I tried reproducing the problem by running an L2, killing it, removing and re-adding KVM in L1, then running an L2 with another kernel (i.e. different kernel addresses). That didn't cause any problems. Nonetheless, I think the fix is correct (famous last words). I sent it out (with another TLB flush bug in nVMX):
https://lore.kernel.org/kvm/[email protected]/.
RFC PATCH v2 25/25 DO NOT MERGE: Add nested_tlb_force_flush
In reply to Yosry AhmedPurely for testing only, add a knob that brings back unconditional force flushes (and resync) on nested transitions.
--- arch/x86/kvm/svm/nested.c | 15 +++++++++++++++ 1 file changed, 15 insertions(+)
arch/x86/kvm/svm/nested.c +15 -0
@@ -32,6 +32,9 @@#include "hyperv.h" #include "pmu.h" +static bool nested_tlb_force_flush; +module_param (nested_tlb_force_flush, bool,0644 ); +#define CC KVM_NESTED_VMENTER_CONSISTENCY_CHECK static void nested_svm_inject_npf_exit(struct kvm_vcpu *vcpu,
@@ -691,6 +694,12 @@ static void nested_svm_entry_tlb_flush(struct kvm_vcpu *vcpu)/* Handle pending Hyper-V TLB flush requests */ kvm_hv_nested_transtion_tlb_flush (vcpu, npt_enabled); +if (nested_tlb_force_flush) { +kvm_make_request (KVM_REQ_MMU_SYNC , vcpu); +kvm_make_request (KVM_REQ_TLB_FLUSH_CURRENT , vcpu); +return ; + } +if (svm->nested.ctl.asid != svm->nested.last_asid) { svm->nested.last_asid = svm->nested.ctl.asid; new_asid = true;
@@ -726,6 +735,12 @@ static void nested_svm_exit_tlb_flush(struct kvm_vcpu *vcpu)kvm_hv_nested_transtion_tlb_flush (vcpu, npt_enabled); +if (nested_tlb_force_flush) { +kvm_make_request (KVM_REQ_MMU_SYNC , vcpu); +kvm_make_request (KVM_REQ_TLB_FLUSH_CURRENT , vcpu); +return ; + } +/* Flush L1's own ASID if it request a *full* TLB flush on VMRUN */ if (svm->nested.ctl.tlb_ctl ==TLB_CONTROL_FLUSH_ALL_ASID )kvm_make_request (KVM_REQ_TLB_FLUSH_GUEST , vcpu);
Signature · 1 line
2.54.0.1136.gdb2ca164c4-goog
[email protected]
AI reviewRe: [RFC PATCH v2 25/25] DO NOT MERGE: Add nested_tlb_force_flush
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: - [Critical] State Desynchronization (ASID tracking) / Isolation Breach - [High] Weakened Flush Semantics /…
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [Critical] State Desynchronization (ASID tracking) / Isolation Breach
- [High] Weakened Flush Semantics / Stale MMU Roots
- [Medium] Guideline Violation / Mutable State
--
commit 3fa5ceb50c23aa3458e6cdaf0dc109608a8b6192
Author: Yosry Ahmed <[email protected]>
DO NOT MERGE: Add nested_tlb_force_flush
Purely for testing only, add a knob that brings back unconditional force flushes (and resync) on nested transitions.
arch/x86/kvm/svm/nested.c line 36
Quoted 9 lines
arch/x86/kvm/svm/nested.c +2
-0@@ -32,6 +32,9 @@#include "hyperv.h" #include "pmu.h" +static bool nested_tlb_force_flush; +module_param (nested_tlb_force_flush, bool,0644 );
[Severity: Medium]
Does this module parameter need to be mutable at runtime (0644)?
KVM guidelines suggest new flags should default to immutable (0444) because toggling them creates unexpected state-machine transitions that can be difficult to handle safely.
arch/x86/kvm/svm/nested.c static void nested_svm_entry_tlb_flush(struct kvm_vcpu *vcpu) · line 701
Quoted 10 lines
[ ... ]
arch/x86/kvm/svm/nested.c +5
-0@@ -691,6 +694,12 @@ static void nested_svm_entry_tlb_flush(struct kvm_vcpu *vcpu)/* Handle pending Hyper-V TLB flush requests */ kvm_hv_nested_transtion_tlb_flush (vcpu, npt_enabled); +if (nested_tlb_force_flush) { +kvm_make_request (KVM_REQ_MMU_SYNC , vcpu); +kvm_make_request (KVM_REQ_TLB_FLUSH_CURRENT , vcpu); +return ; + }
[Severity: Critical]
If this parameter is toggled off at runtime, can it leave svm->nested.last_asid out of sync?
Because the early return skips updating last_asid, if the feature is later disabled, KVM might see the stale last_asid matching a newly assigned L1 ASID:
nested_svm_entry_tlb_flush() {
...
if (svm->nested.ctl.asid != svm->nested.last_asid) {
svm->nested.last_asid = svm->nested.ctl.asid;
new_asid = true;
}
...
}Would this incorrectly evaluate new_asid to false and skip the required flush, allowing the L2 guest to run with mappings from a different context?
[Severity: High]
Does returning early here bypass the full guest flush guarantees?
When replacing KVM_REQ_TLB_FLUSH_GUEST with KVM_REQ_MMU_SYNC and KVM_REQ_TLB_FLUSH_CURRENT, it appears to skip synchronizing the previous mmu roots (via kvm_mmu_sync_prev_roots) and guest-specific flush logic.
If L1 explicitly requests a full TLB flush, could a guest switching CR3 to a cached previous root end up using stale shadow page tables?
arch/x86/kvm/svm/nested.c static void nested_svm_exit_tlb_flush(struct kvm_vcpu *vcpu) · line 742
Quoted 13 lines
arch/x86/kvm/svm/nested.c +5
-0if (svm->nested.ctl.asid != svm->nested.last_asid) { svm->nested.last_asid = svm->nested.ctl.asid; new_asid = true;@@ -726,6 +735,12 @@ static void nested_svm_exit_tlb_flush(struct kvm_vcpu *vcpu)kvm_hv_nested_transtion_tlb_flush (vcpu, npt_enabled); +if (nested_tlb_force_flush) { +kvm_make_request (KVM_REQ_MMU_SYNC , vcpu); +kvm_make_request (KVM_REQ_TLB_FLUSH_CURRENT , vcpu); +return ; + }
[Severity: High]
Could this early return on VM-Exit similarly replace an explicit TLB_CONTROL_FLUSH_ALL_ASID request with a weaker current-only flush?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=25