KVM: x86: Event/exception fixes and cleanups
Sean Christopherson
maintainercover KVM: x86: Event/exception fixes and cleanups
The main goal of this series is to fix KVM's longstanding bug of not honoring L1's exception intercepts wants when handling an exception that occurs during delivery of a different exception. E.g. if L0 and L1 are using shadow paging, and L2 hits a #PF, and then hits another #PF while vectoring the first #PF due to _L1_ not having a shadow page for the IDT, KVM needs to check L1's intercepts before morphing the #PF => #PF => #DF so that the #PF is routed to L1, not injected into L2 as a #DF.
nVMX has hacked around the bug for years by overriding the #PF injector for shadow paging to go straight to VM-Exit, and nSVM has started doing the same. The hacks mostly work, but they're incomplete, confusing, and lead to other hacky code, e.g. bailing from the emulator because #PF injection forced a VM-Exit and suddenly KVM is back in L1.
Everything leading up to that are related fixes and cleanups I encountered along the way; some through code inspection, some through tests (I truly thought this series was finished 10 commits and 3 days ago...).
Nothing in here is all that urgent; all bugs tagged for stable have been around for multiple releases (years in most cases).
Sean Christopherson (21):
KVM: x86: Return immediately from x86_emulate_instruction() on code
#DB
KVM: nVMX: Unconditionally purge queued/injected events on nested
"exit"
KVM: VMX: Drop bits 31:16 when shoving exception error code into VMCS
KVM: x86: Don't check for code breakpoints when emulating on exception
KVM: nVMX: Treat General Detect #DB (DR7.GD=1) as fault-like
KVM: nVMX: Prioritize TSS T-flag #DBs over Monitor Trap Flag
KVM: x86: Treat #DBs from the emulator as fault-like (code and
DR7.GD=1) KVM: x86: Use DR7_GD macro instead of open coding check in emulator
KVM: nVMX: Ignore SIPI that arrives in L2 when vCPU is not in WFS
KVM: nVMX: Unconditionally clear mtf_pending on nested VM-Exit
KVM: VMX: Inject #PF on ENCLS as "emulated" #PF
KVM: x86: Rename kvm_x86_ops.queue_exception to inject_exception
KVM: x86: Make kvm_queued_exception a properly named, visible struct
KVM: x86: Formalize blocking of nested pending exceptions
KVM: x86: Use kvm_queue_exception_e() to queue #DF
KVM: x86: Hoist nested event checks above event injection logic
KVM: x86: Evaluate ability to inject SMI/NMI/IRQ after potential
VM-Exit
KVM: x86: Morph pending exceptions to pending VM-Exits at queue time
KVM: VMX: Update MTF and ICEBP comments to document KVM's subtle
behavior
KVM: selftests: Use uapi header to get VMX and SVM exit reasons/codes
KVM: selftests: Add an x86-only test to verify nested exception
queueing
arch/x86/include/asm/kvm-x86-ops.h | 2 +- arch/x86/include/asm/kvm_host.h | 33 +- arch/x86/kvm/emulate.c | 3 +- arch/x86/kvm/svm/nested.c | 100 ++--- arch/x86/kvm/svm/svm.c | 18 +- arch/x86/kvm/vmx/nested.c | 322 +++++++++----- arch/x86/kvm/vmx/sgx.c | 2 +- arch/x86/kvm/vmx/vmx.c | 53 ++- arch/x86/kvm/x86.c | 409 ++++++++++++------ arch/x86/kvm/x86.h | 10 +- tools/testing/selftests/kvm/.gitignore | 1 + tools/testing/selftests/kvm/Makefile | 1 + .../selftests/kvm/include/x86_64/svm_util.h | 5 +- .../selftests/kvm/include/x86_64/vmx.h | 51 +-- .../kvm/x86_64/nested_exceptions_test.c | 307 +++++++++++++ 15 files changed, 914 insertions(+), 403 deletions(-) create mode 100644 tools/testing/selftests/kvm/x86_64/nested_exceptions_test.c
base-commit: 4a204f7895878363ca8211f50ec610408c8c70aa
Signature · 1 line
2.35.1.723.g4982287a31-goog
Sean Christopherson
maintainerPATCH 1/21 KVM: x86: Return immediately from x86_emulate_instruction() on code #DB
Return immediately if a code #DB is encountered during instruction emulation, code #DBs have fault-like behavior and are higher priority than any exceptions that occur on the code fetch itself, and obviously should prevent decode/execution.
--- arch/x86/kvm/x86.c | 31 +++++++++++++++++++------------ 1 file changed, 19 insertions(+), 12 deletions(-)
arch/x86/kvm/x86.c +19 -12
@@ -8212,7 +8212,7 @@ int kvm_skip_emulated_instruction(struct kvm_vcpu *vcpu) }EXPORT_SYMBOL_GPL (kvm_skip_emulated_instruction); -static bool kvm_vcpu_check_breakpoint (struct kvm_vcpu *vcpu,int *r) +static bool kvm_vcpu_check_code_breakpoint (struct kvm_vcpu *vcpu,int *r) {if (unlikely (vcpu->guest_debug &KVM_GUESTDBG_USE_HW_BP ) && (vcpu->arch.guest_debug_dr7 &DR7_BP_EN_MASK )) {
@@ -8281,25 +8281,23 @@ static bool is_vmware_backdoor_opcode(struct x86_emulate_ctxt *ctxt) }/* -* Decode to be emulated instruction. Return EMULATION_OK if success. +* Decode an instruction for emulation. The caller is responsible for handling +* code breakpoints. Note, manually detecting code breakpoints is unnecessary +* (and wrong) when emulating on an intercepted fault-like exception[*], as +* code breakpoints have higher priority and thus have already been done by +* hardware. +* +* [*] Except #MC, which is higher priority, but KVM should never emulate in +* response to a machine check. */ int x86_decode_emulated_instruction (struct kvm_vcpu *vcpu,int emulation_type,void *insn,int insn_len) { -int r =EMULATION_OK ;struct x86_emulate_ctxt *ctxt = vcpu->arch.emulate_ctxt; +int r;init_emulate_ctxt (vcpu); -/* -* We will reenter on the same instruction since we do not set -* complete_userspace_io. This does not handle watchpoints yet, -* those would be handled in the emulate_ops. -*/ -if (!(emulation_type &EMULTYPE_SKIP ) && -kvm_vcpu_check_breakpoint (vcpu, &r)) -return r; - r =x86_decode_insn (ctxt, insn, insn_len, emulation_type);trace_kvm_emulate_insn_start (vcpu);
@@ -8332,6 +8330,15 @@ int x86_emulate_instruction(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,if (!(emulation_type &EMULTYPE_NO_DECODE )) {kvm_clear_exception_queue (vcpu); +/* +* Return immediately if RIP hits a code breakpoint, such #DBs +* are fault-like and are higher priority than any faults on +* the code fetch itself. +*/ +if (!(emulation_type &EMULTYPE_SKIP ) && +kvm_vcpu_check_code_breakpoint (vcpu, &r)) +return r; + r =x86_decode_emulated_instruction (vcpu, emulation_type, insn, insn_len);if (r !=EMULATION_OK ) {
Signature · 1 line
2.35.1.723.g4982287a31-goog
Sean Christopherson
maintainerPATCH 2/21 KVM: nVMX: Unconditionally purge queued/injected events on nested "exit"
In reply to Sean ChristophersonDrop pending exceptions and events queued for re-injection when leaving nested guest mode, even if the "exit" is due to VM-Fail, SMI, or forced by host userspace. Failure to purge events could result in an event belonging to L2 being injected into L1.
This _should_ never happen for VM-Fail as all events should be blocked by nested_run_pending, but it's possible if KVM, not the L1 hypervisor, is the source of VM-Fail when running vmcs02.
SMI is a nop (barring unknown bugs) as recognition of SMI and thus entry to SMM is blocked by pending exceptions and re-injected events.
Forced exit is definitely buggy, but has likely gone unnoticed because userspace probably follows the forced exit with KVM_SET_VCPU_EVENTS (or some other ioctl() that purges the queue).
--- arch/x86/kvm/vmx/nested.c | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-)
arch/x86/kvm/vmx/nested.c +11 -8
@@ -4233,14 +4233,6 @@ static void prepare_vmcs12(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12,nested_vmx_abort (vcpu,VMX_ABORT_SAVE_GUEST_MSR_FAIL ); } - -/* -* Drop what we picked up for L2 via vmx_complete_interrupts. It is -* preserved above and would only end up incorrectly in L1. -*/ - vcpu->arch.nmi_injected = false; -kvm_clear_exception_queue (vcpu); -kvm_clear_interrupt_queue (vcpu); }/*
@@ -4582,6 +4574,17 @@ void nested_vmx_vmexit(struct kvm_vcpu *vcpu, u32 vm_exit_reason,WARN_ON_ONCE (nested_early_check); } +/* +* Drop events/exceptions that were queued for re-injection to L2 +* (picked up via vmx_complete_interrupts()), as well as exceptions +* that were pending for L2. Note, this must NOT be hoisted above +* prepare_vmcs12(), events/exceptions queued for re-injection need to +* be captured in vmcs12 (see vmcs12_save_pending_event()). +*/ + vcpu->arch.nmi_injected = false; +kvm_clear_exception_queue (vcpu); +kvm_clear_interrupt_queue (vcpu); +vmx_switch_vmcs (vcpu, &vmx->vmcs01);/* Update any VMCS fields that might have changed while L2 ran */
Signature · 1 line
2.35.1.723.g4982287a31-goog
Sean Christopherson
maintainerPATCH 3/21 KVM: VMX: Drop bits 31:16 when shoving exception error code into VMCS
In reply to Sean ChristophersonDeliberately truncate the exception error code when shoving it into the VMCS (VM-Entry field for vmcs01 and vmcs02, VM-Exit field for vmcs12). Intel CPUs are incapable of handling 32-bit error codes and will never generate an error code with bits 31:16, but userspace can provide an arbitrary error code via KVM_SET_VCPU_EVENTS. Failure to drop the bits on exception injection results in failed VM-Entry, as VMX disallows setting bits 31:16. Setting the bits on VM-Exit would at best confuse L1, and at worse induce a nested VM-Entry failure, e.g. if L1 decided to reinject the exception back into L2.
--- arch/x86/kvm/vmx/nested.c | 9 ++++++++- arch/x86/kvm/vmx/vmx.c | 11 ++++++++++- 2 files changed, 18 insertions(+), 2 deletions(-)
arch/x86/kvm/vmx/nested.c +8 -1
@@ -3808,7 +3808,14 @@ static void nested_vmx_inject_exception_vmexit(struct kvm_vcpu *vcpu,u32 intr_info = nr |INTR_INFO_VALID_MASK ;if (vcpu->arch.exception.has_error_code) { - vmcs12->vm_exit_intr_error_code = vcpu->arch.exception.error_code; +/* +* Intel CPUs will never generate an error code with bits 31:16 +* set, and more importantly VMX disallows setting bits 31:16 +* in the injected error code for VM-Entry. Drop the bits to +* mimic hardware and avoid inducing failure on nested VM-Entry +* if L1 chooses to inject the exception back to L2. +*/ + vmcs12->vm_exit_intr_error_code = (u16 )vcpu->arch.exception.error_code; intr_info |=INTR_INFO_DELIVER_CODE_MASK ; }
arch/x86/kvm/vmx/vmx.c +10 -1
@@ -1613,7 +1613,16 @@ static void vmx_queue_exception(struct kvm_vcpu *vcpu)kvm_deliver_exception_payload (vcpu);if (has_error_code) { -vmcs_write32 (VM_ENTRY_EXCEPTION_ERROR_CODE , error_code); +/* +* Despite the error code being architecturally defined as 32 +* bits, and the VMCS field being 32 bits, Intel CPUs and thus +* VMX don't actually supporting setting bits 31:16. Hardware +* will (should) never provide a bogus error code, but KVM's +* ABI lets userspace shove in arbitrary 32-bit values. Drop +* the upper bits to avoid VM-Fail, losing information that +* does't really exist is preferable to killing the VM. +*/ +vmcs_write32 (VM_ENTRY_EXCEPTION_ERROR_CODE , (u16 )error_code); intr_info |=INTR_INFO_DELIVER_CODE_MASK ; }
Signature · 1 line
2.35.1.723.g4982287a31-goog
Sean Christopherson
maintainerPATCH 4/21 KVM: x86: Don't check for code breakpoints when emulating on exception
In reply to Sean ChristophersonDon't check for code breakpoints during instruction emulation if the emulation was triggered by exception interception. Code breakpoints are the highest priority fault-like exception, and KVM only emulates on exceptions that are fault-like. Thus, if hardware signaled a different exception, then the vCPU is already passed the stage of checking for hardware breakpoints.
This is likely a glorified nop in terms of functionality, and is more for clarification and is technically an optimization. Intel's SDM explicitly states vmcs.GUEST_RFLAGS.RF on exception interception is the same as the value that would have been saved on the stack had the exception not been intercepted, i.e. will be '1' due to all fault-like exceptions setting RF to '1'. AMD says "guest state saved ... is the processor state as of the moment the intercept triggers", but that begs the question, "when does the intercept trigger?".
--- arch/x86/kvm/x86.c | 21 ++++++++++++++++++--- 1 file changed, 18 insertions(+), 3 deletions(-)
arch/x86/kvm/x86.c +18 -3
@@ -8212,8 +8212,24 @@ int kvm_skip_emulated_instruction(struct kvm_vcpu *vcpu) }EXPORT_SYMBOL_GPL (kvm_skip_emulated_instruction); -static bool kvm_vcpu_check_code_breakpoint (struct kvm_vcpu *vcpu,int *r) +static bool kvm_vcpu_check_code_breakpoint (struct kvm_vcpu *vcpu, +int emulation_type,int *r) { +WARN_ON_ONCE (emulation_type &EMULTYPE_NO_DECODE ); + +/* +* Do not check for code breakpoints if hardware has already done the +* checks, as inferred from the emulation type. On NO_DECODE and SKIP, +* the instruction has passed all exception checks, and all intercepted +* exceptions that trigger emulation have lower priority than code +* breakpoints, i.e. the fact that the intercepted exception occurred +* means any code breakpoints have already been serviced. +*/ +if (emulation_type & (EMULTYPE_NO_DECODE |EMULTYPE_SKIP | +EMULTYPE_TRAP_UD |EMULTYPE_TRAP_UD_FORCED | +EMULTYPE_VMWARE_GP |EMULTYPE_PF )) +return false; +if (unlikely (vcpu->guest_debug &KVM_GUESTDBG_USE_HW_BP ) && (vcpu->arch.guest_debug_dr7 &DR7_BP_EN_MASK )) {struct kvm_run *kvm_run = vcpu->run;
@@ -8335,8 +8351,7 @@ int x86_emulate_instruction(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,* are fault-like and are higher priority than any faults on * the code fetch itself. */ -if (!(emulation_type &EMULTYPE_SKIP ) && -kvm_vcpu_check_code_breakpoint (vcpu, &r)) +if (kvm_vcpu_check_code_breakpoint (vcpu, emulation_type, &r))return r; r = x86_decode_emulated_instruction(vcpu, emulation_type,
Signature · 1 line
2.35.1.723.g4982287a31-goog
Sean Christopherson
maintainerPATCH 5/21 KVM: nVMX: Treat General Detect #DB (DR7.GD=1) as fault-like
In reply to Sean ChristophersonExclude General Detect #DBs, which have fault-like behavior but also have a non-zero payload (DR6.BD=1), from nVMX's handling of pending debug traps. Opportunistically rewrite the comment to better document what is being checked, i.e. "has a non-zero payload" vs. "has a payload", and to call out the many caveats surrounding #DBs that KVM dodges one way or another.
--- arch/x86/kvm/vmx/nested.c | 36 +++++++++++++++++++++++++----------- 1 file changed, 25 insertions(+), 11 deletions(-)
arch/x86/kvm/vmx/nested.c +25 -11
@@ -3832,16 +3832,29 @@ static void nested_vmx_inject_exception_vmexit(struct kvm_vcpu *vcpu, }/* -* Returns true if a debug trap is pending delivery. +* Returns true if a debug trap is (likely) pending delivery. Infer the class +* of a #DB (trap-like vs. fault-like) from the exception payload (to-be-DR6). +* Using the payload is flawed because code breakpoints (fault-like) and data +* breakpoints (trap-like) set the same bits in DR6 (breakpoint detected), i.e. +* this will return false positives if a to-be-injected code breakpoint #DB is +* pending (from KVM's perspective, but not "pending" across an instruction +* boundary). ICEBP, a.k.a. INT1, is also not reflected here even though it +* too is trap-like. * -* In KVM, debug traps bear an exception payload. As such, the class of a #DB -* exception may be inferred from the presence of an exception payload. +* KVM "works" despite these flaws as ICEBP isn't currently supported by the +* emulator, Monitor Trap Flag is not marked pending on intercepted #DBs (the +* #DB has already happened), and MTF isn't marked pending on code breakpoints +* from the emulator (because such #DBs are fault-like and thus don't trigger +* actions that fire on instruction retire). */ -static inline bool vmx_pending_dbg_trap (struct kvm_vcpu *vcpu) +static inline unsigned long vmx_get_pending_dbg_trap (struct kvm_vcpu *vcpu) { -return vcpu->arch.exception.pending && - vcpu->arch.exception.nr ==DB_VECTOR && - vcpu->arch.exception.payload; +if (!vcpu->arch.exception.pending || + vcpu->arch.exception.nr !=DB_VECTOR ) +return 0 ; + +/* General Detect #DBs are always fault-like. */ +return vcpu->arch.exception.payload & ~DR6_BD ; }/*
@@ -3853,9 +3866,10 @@ static inline bool vmx_pending_dbg_trap(struct kvm_vcpu *vcpu)*/ static void nested_vmx_update_pending_dbg (struct kvm_vcpu *vcpu) { -if (vmx_pending_dbg_trap (vcpu)) -vmcs_writel (GUEST_PENDING_DBG_EXCEPTIONS , - vcpu->arch.exception.payload); +unsigned long pending_dbg =vmx_get_pending_dbg_trap (vcpu); + +if (pending_dbg) +vmcs_writel (GUEST_PENDING_DBG_EXCEPTIONS , pending_dbg); }static bool nested_vmx_preemption_timer_pending (struct kvm_vcpu *vcpu)
@@ -3912,7 +3926,7 @@ static int vmx_check_nested_events(struct kvm_vcpu *vcpu)* while delivering the pending exception. */ -if (vcpu->arch.exception.pending && !vmx_pending_dbg_trap (vcpu)) { +if (vcpu->arch.exception.pending && !vmx_get_pending_dbg_trap (vcpu)) {if (vmx->nested.nested_run_pending)return -EBUSY ;if (!nested_vmx_check_exception (vcpu, &exit_qual))
Signature · 1 line
2.35.1.723.g4982287a31-goog
Sean Christopherson
maintainerPATCH 6/21 KVM: nVMX: Prioritize TSS T-flag #DBs over Monitor Trap Flag
In reply to Sean ChristophersonService TSS T-flag #DBs prior to pending MTFs, as such #DBs are higher priority than MTF. KVM itself doesn't emulate TSS #DBs, and any such exceptions injected from L1 will be handled by hardware (or morphed to a fault-like exception if injection fails), but theoretically userspace could pend a TSS T-flag #DB in conjunction with a pending MTF.
Note, there's no known use case this fixes, it's purely to be technically correct with respect to Intel's SDM.
--- arch/x86/kvm/vmx/nested.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
arch/x86/kvm/vmx/nested.c +5 -3
@@ -3918,15 +3918,17 @@ static int vmx_check_nested_events(struct kvm_vcpu *vcpu) }/* -* Process any exceptions that are not debug traps before MTF. +* Process exceptions that are higher priority than Monitor Trap Flag: +* fault-like exceptions, TSS T flag #DB (not emulated by KVM, but +* could theoretically come in from userspace), and ICEBP (INT1). * * Note that only a pending nested run can block a pending exception. * Otherwise an injected NMI/interrupt should either be * lost or delivered to the nested hypervisor in the IDT_VECTORING_INFO, * while delivering the pending exception. */ - -if (vcpu->arch.exception.pending && !vmx_get_pending_dbg_trap (vcpu)) { +if (vcpu->arch.exception.pending && + !(vmx_get_pending_dbg_trap (vcpu) & ~DR6_BT )) {if (vmx->nested.nested_run_pending)return -EBUSY ;if (!nested_vmx_check_exception (vcpu, &exit_qual))
Signature · 1 line
2.35.1.723.g4982287a31-goog
Sean Christopherson
maintainerPATCH 7/21 KVM: x86: Treat #DBs from the emulator as fault-like (code and DR7.GD=1)
In reply to Sean ChristophersonAdd a dedicated "exception type" for #DBs, as #DBs can be fault-like or trap-like depending the sub-type of #DB, and effectively defer the decision of what to do with the #DB to the caller.
For the emulator's two calls to exception_type(), treat the #DB as fault-like, as the emulator handles only code breakpoint and general detect #DBs, both of which are fault-like.
For event injection, which uses exception_type() to determine whether to set EFLAGS.RF=1 on the stack, keep the current behavior of not setting RF=1 for #DBs. Intel and AMD explicitly state RF isn't set on code #DBs, so exempting by failing the "== EXCPT_FAULT" check is correct. The only other fault-like #DB is General Detect, and despite Intel and AMD both strongly implying (through omission) that General Detect #DBs should set RF=1, hardware (multiple generations of both Intel and AMD), in fact does not. Through insider knowledge, extreme foresight, sheer dumb luck, or some combination thereof, KVM correctly handled RF for General Detect #DBs.
--- arch/x86/kvm/x86.c | 27 +++++++++++++++++++++++++-- 1 file changed, 25 insertions(+), 2 deletions(-)
arch/x86/kvm/x86.c +25 -2
@@ -535,6 +535,7 @@ static int exception_class(int vector)#define EXCPT_TRAP 1#define EXCPT_ABORT 2#define EXCPT_INTERRUPT 3 +#define EXCPT_DB 4static int exception_type (int vector) {
@@ -545,8 +546,14 @@ static int exception_type(int vector) mask =1 << vector; -/* #DB is trap, as instruction watchpoints are handled elsewhere */ -if (mask & ((1 <<DB_VECTOR ) | (1 <<BP_VECTOR ) | (1 <<OF_VECTOR ))) +/* +* #DBs can be trap-like or fault-like, the caller must check other CPU +* state, e.g. DR6, to determine whether a #DB is a trap or fault. +*/ +if (mask & (1 <<DB_VECTOR )) +return EXCPT_DB ; + +if (mask & ((1 <<BP_VECTOR ) | (1 <<OF_VECTOR )))return EXCPT_TRAP ;if (mask & ((1 <<DF_VECTOR ) | (1 <<MC_VECTOR )))
@@ -8480,6 +8487,12 @@ int x86_emulate_instruction(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,unsigned long rflags =static_call (kvm_x86_get_rflags)(vcpu);toggle_interruptibility (vcpu, ctxt->interruptibility); vcpu->arch.emulate_regs_need_sync_to_vcpu = false; + +/* +* Note, EXCPT_DB is assumed to be fault-like as the emulator +* only supports code breakpoints and general detect #DB, both +* of which are fault-like. +*/ if (!ctxt->have_exception ||exception_type (ctxt->exception.vector) ==EXCPT_TRAP ) {kvm_pmu_trigger_event (vcpu,PERF_COUNT_HW_INSTRUCTIONS );
@@ -9361,6 +9374,16 @@ static int inject_pending_event(struct kvm_vcpu *vcpu, bool *req_immediate_exit) vcpu->arch.exception.pending = false; vcpu->arch.exception.injected = true; +/* +* Fault-class exceptions, except #DBs, set RF=1 in the RFLAGS +* value pushed on the stack. Trap-like exception and all #DBs +* leave RF as-is (KVM follows Intel's behavior in this regard; +* AMD states that code breakpoint #DBs excplitly clear RF=0). +* +* Note, most versions of Intel's SDM and AMD's APM incorrectly +* describe the behavior of General Detect #DBs, which are +* fault-like. They do _not_ set RF, a la code breakpoints. +*/ if (exception_type (vcpu->arch.exception.nr) ==EXCPT_FAULT )__kvm_set_rflags (vcpu,kvm_get_rflags (vcpu) |X86_EFLAGS_RF );
Signature · 1 line
2.35.1.723.g4982287a31-goog
Sean Christopherson
maintainerPATCH 8/21 KVM: x86: Use DR7_GD macro instead of open coding check in emulator
In reply to Sean ChristophersonUse DR7_GD in the emulator instead of open coding the check, and drop a comically wrong comment.
--- arch/x86/kvm/emulate.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
arch/x86/kvm/emulate.c +1 -2
@@ -4148,8 +4148,7 @@ static int check_dr7_gd(struct x86_emulate_ctxt *ctxt) ctxt->ops->get_dr (ctxt,7 , &dr7); -/* Check if DR7.Global_Enable is set */ -return dr7 & (1 <<13 ); +return dr7 &DR7_GD ; }static int check_dr_read (struct x86_emulate_ctxt *ctxt)
Signature · 1 line
2.35.1.723.g4982287a31-goog
Sean Christopherson
maintainerPATCH 9/21 KVM: nVMX: Ignore SIPI that arrives in L2 when vCPU is not in WFS
In reply to Sean ChristophersonFall through to handling other pending exception/events for L2 if SIPI is pending while the CPU is not in Wait-for-SIPI. KVM correctly ignores the event, but incorrectly returns immediately, e.g. a SIPI coincident with another event could lead to KVM incorrectly routing the event to L1 instead of L2.
--- arch/x86/kvm/vmx/nested.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
arch/x86/kvm/vmx/nested.c +4 -2
@@ -3911,10 +3911,12 @@ static int vmx_check_nested_events(struct kvm_vcpu *vcpu)return -EBUSY ;clear_bit (KVM_APIC_SIPI , &apic->pending_events); -if (vcpu->arch.mp_state ==KVM_MP_STATE_INIT_RECEIVED ) +if (vcpu->arch.mp_state ==KVM_MP_STATE_INIT_RECEIVED ) {nested_vmx_vmexit (vcpu,EXIT_REASON_SIPI_SIGNAL ,0 , apic->sipi_vector &0xFFUL ); -return 0 ; +return 0 ; + } +/* Fallthrough, the SIPI is completely ignored. */ }/*
Signature · 1 line
2.35.1.723.g4982287a31-goog
Sean Christopherson
maintainerPATCH 10/21 KVM: nVMX: Unconditionally clear mtf_pending on nested VM-Exit
In reply to Sean ChristophersonClear mtf_pending on nested VM-Exit instead of handling the clear on a case-by-case basis in vmx_check_nested_events(). The pending MTF should never survive nested VM-Exit, as it is a property of KVM's run of the current L2, i.e. should never affect the next L2 run by L1. In practice, this is likely a nop as getting to L1 with nested_run_pending is impossible, and KVM doesn't correctly handle morphing a pending exception that occurs on a prior injected exception (need for re-injected exception being the other case where MTF isn't cleared). However, KVM will hopefully soon correctly deal with a pending exception on top of an injected exception.
--- arch/x86/kvm/vmx/nested.c | 16 +++++++--------- 1 file changed, 7 insertions(+), 9 deletions(-)
arch/x86/kvm/vmx/nested.c +7 -9
@@ -3884,16 +3884,8 @@ static int vmx_check_nested_events(struct kvm_vcpu *vcpu)unsigned long exit_qual;bool block_nested_events = vmx->nested.nested_run_pending ||kvm_event_needs_reinjection (vcpu); -bool mtf_pending = vmx->nested.mtf_pending;struct kvm_lapic *apic = vcpu->arch.apic; -/* -* Clear the MTF state. If a higher priority VM-exit is delivered first, -* this state is discarded. -*/ -if (!block_nested_events) - vmx->nested.mtf_pending = false; -if (lapic_in_kernel (vcpu) &&test_bit (KVM_APIC_INIT , &apic->pending_events)) {if (block_nested_events)
@@ -3902,6 +3894,9 @@ static int vmx_check_nested_events(struct kvm_vcpu *vcpu)clear_bit (KVM_APIC_INIT , &apic->pending_events);if (vcpu->arch.mp_state !=KVM_MP_STATE_INIT_RECEIVED )nested_vmx_vmexit (vcpu,EXIT_REASON_INIT_SIGNAL ,0 ,0 ); + +/* MTF is discarded if the vCPU is in WFS. */ + vmx->nested.mtf_pending = false;return 0 ; }
@@ -3939,7 +3934,7 @@ static int vmx_check_nested_events(struct kvm_vcpu *vcpu)return 0 ; } -if (mtf_pending) { +if (vmx->nested.mtf_pending) {if (block_nested_events)return -EBUSY ;nested_vmx_update_pending_dbg (vcpu);
@@ -4532,6 +4527,9 @@ void nested_vmx_vmexit(struct kvm_vcpu *vcpu, u32 vm_exit_reason,struct vcpu_vmx *vmx =to_vmx (vcpu);struct vmcs12 *vmcs12 =get_vmcs12 (vcpu); +/* Pending MTF traps are discarded on VM-Exit. */ + vmx->nested.mtf_pending = false; +/* trying to cancel vmlaunch/vmresume is a bug */ WARN_ON_ONCE (vmx->nested.nested_run_pending);
Signature · 1 line
2.35.1.723.g4982287a31-goog
Sean Christopherson
maintainerPATCH 11/21 KVM: VMX: Inject #PF on ENCLS as "emulated" #PF
In reply to Sean ChristophersonTreat #PFs that occur during emulation of ENCLS as, wait for it, emulated page faults. Practically speaking, this is a glorified nop as the exception is never of the nested flavor, and it's extremely unlikely the guest is relying on the side effect of an implicit INVLPG on the faulting address.
--- arch/x86/kvm/vmx/sgx.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kvm/vmx/sgx.c +1 -1
@@ -129,7 +129,7 @@ static int sgx_inject_fault(struct kvm_vcpu *vcpu, gva_t gva, int trapnr) ex.address = gva; ex.error_code_valid = true; ex.nested_page_fault = false; -kvm_inject_page_fault (vcpu, &ex); +kvm_inject_emulated_page_fault (vcpu, &ex); }else {kvm_inject_gp (vcpu,0 ); }
Signature · 1 line
2.35.1.723.g4982287a31-goog
Sean Christopherson
maintainerPATCH 12/21 KVM: x86: Rename kvm_x86_ops.queue_exception to inject_exception
In reply to Sean ChristophersonRename the kvm_x86_ops hook for exception injection to better reflect reality, and to align with pretty much every other related function name in KVM.
No functional change intended.
--- arch/x86/include/asm/kvm-x86-ops.h | 2 +- arch/x86/include/asm/kvm_host.h | 2 +- arch/x86/kvm/svm/svm.c | 4 ++-- arch/x86/kvm/vmx/vmx.c | 4 ++-- arch/x86/kvm/x86.c | 2 +- 5 files changed, 7 insertions(+), 7 deletions(-)
arch/x86/include/asm/kvm-x86-ops.h +1 -1
@@ -66,7 +66,7 @@ KVM_X86_OP(get_interrupt_shadow) KVM_X86_OP(patch_hypercall) KVM_X86_OP(inject_irq) KVM_X86_OP(inject_nmi) -KVM_X86_OP(queue_exception) +KVM_X86_OP(inject_exception) KVM_X86_OP(cancel_injection) KVM_X86_OP(interrupt_allowed) KVM_X86_OP(nmi_allowed)
arch/x86/include/asm/kvm_host.h +1 -1
@@ -1391,7 +1391,7 @@ struct kvm_x86_ops { unsigned char *hypercall_addr);void (*inject_irq)(struct kvm_vcpu *vcpu);void (*inject_nmi)(struct kvm_vcpu *vcpu); -void (*queue_exception)(struct kvm_vcpu *vcpu); +void (*inject_exception)(struct kvm_vcpu *vcpu);void (*cancel_injection)(struct kvm_vcpu *vcpu);int (*interrupt_allowed)(struct kvm_vcpu *vcpu,bool for_injection);int (*nmi_allowed)(struct kvm_vcpu *vcpu,bool for_injection);
arch/x86/kvm/svm/svm.c +2 -2
@@ -382,7 +382,7 @@ static int svm_skip_emulated_instruction(struct kvm_vcpu *vcpu)return 1 ; } -static void svm_queue_exception (struct kvm_vcpu *vcpu) +static void svm_inject_exception (struct kvm_vcpu *vcpu) {struct vcpu_svm *svm =to_svm (vcpu);unsigned nr = vcpu->arch.exception.nr;
@@ -4580,7 +4580,7 @@ static struct kvm_x86_ops svm_x86_ops __initdata = { .patch_hypercall = svm_patch_hypercall, .inject_irq = svm_inject_irq, .inject_nmi = svm_inject_nmi, - .queue_exception = svm_queue_exception, + .inject_exception = svm_inject_exception, .cancel_injection = svm_cancel_injection, .interrupt_allowed = svm_interrupt_allowed, .nmi_allowed = svm_nmi_allowed,
arch/x86/kvm/vmx/vmx.c +2 -2
@@ -1602,7 +1602,7 @@ static void vmx_clear_hlt(struct kvm_vcpu *vcpu)vmcs_write32 (GUEST_ACTIVITY_STATE ,GUEST_ACTIVITY_ACTIVE ); } -static void vmx_queue_exception (struct kvm_vcpu *vcpu) +static void vmx_inject_exception (struct kvm_vcpu *vcpu) {struct vcpu_vmx *vmx =to_vmx (vcpu);unsigned nr = vcpu->arch.exception.nr;
@@ -7783,7 +7783,7 @@ static struct kvm_x86_ops vmx_x86_ops __initdata = { .patch_hypercall = vmx_patch_hypercall, .inject_irq = vmx_inject_irq, .inject_nmi = vmx_inject_nmi, - .queue_exception = vmx_queue_exception, + .inject_exception = vmx_inject_exception, .cancel_injection = vmx_cancel_injection, .interrupt_allowed = vmx_interrupt_allowed, .nmi_allowed = vmx_nmi_allowed,
arch/x86/kvm/x86.c +1 -1
@@ -9312,7 +9312,7 @@ static void kvm_inject_exception(struct kvm_vcpu *vcpu) {if (vcpu->arch.exception.error_code && !is_protmode (vcpu)) vcpu->arch.exception.error_code = false; -static_call (kvm_x86_queue_exception)(vcpu); +static_call (kvm_x86_inject_exception)(vcpu); }static int inject_pending_event (struct kvm_vcpu *vcpu,bool *req_immediate_exit)
Signature · 1 line
2.35.1.723.g4982287a31-goog
Sean Christopherson
maintainerPATCH 13/21 KVM: x86: Make kvm_queued_exception a properly named, visible struct
In reply to Sean ChristophersonMove the definition of "struct kvm_queued_exception" out of kvm_vcpu_arch in anticipation of adding a second instance in kvm_vcpu_arch to handle exceptions that occur when vectoring an injected exception and are morphed to VM-Exit instead of leading to #DF.
Opportunistically take advantage of the churn to rename "nr" to "vector".
No functional change intended.
--- arch/x86/include/asm/kvm_host.h | 23 +++++----- arch/x86/kvm/svm/nested.c | 45 ++++++++++--------- arch/x86/kvm/svm/svm.c | 14 +++--- arch/x86/kvm/vmx/nested.c | 44 +++++++++--------- arch/x86/kvm/vmx/vmx.c | 20 ++++----- arch/x86/kvm/x86.c | 80 ++++++++++++++++----------------- arch/x86/kvm/x86.h | 3 +- 7 files changed, 112 insertions(+), 117 deletions(-)
arch/x86/include/asm/kvm_host.h +13 -10
@@ -615,6 +615,17 @@ struct kvm_vcpu_xen {unsigned long evtchn_pending_sel; }; +struct kvm_queued_exception { +bool pending; +bool injected; +bool has_error_code; +u8 vector; +u32 error_code; +unsigned long payload; +bool has_payload; +u8 nested_apf; +}; +struct kvm_vcpu_arch {/* * rip and regs accesses must go through
@@ -713,16 +724,8 @@ struct kvm_vcpu_arch {u8 event_exit_inst_len; -struct kvm_queued_exception { -bool pending; -bool injected; -bool has_error_code; -u8 nr; -u32 error_code; -unsigned long payload; -bool has_payload; -u8 nested_apf; - } exception; +/* Exceptions to be injected to the guest. */ +struct kvm_queued_exception exception;struct kvm_queued_interrupt {bool injected;
arch/x86/kvm/svm/nested.c +23 -22
78 lines not on this page · raw message
arch/x86/kvm/svm/svm.c +6 -8
@@ -384,14 +384,12 @@ static int svm_skip_emulated_instruction(struct kvm_vcpu *vcpu)static void svm_inject_exception (struct kvm_vcpu *vcpu) { +struct kvm_queued_exception *ex = &vcpu->arch.exception;struct vcpu_svm *svm =to_svm (vcpu); -unsigned nr = vcpu->arch.exception.nr; -bool has_error_code = vcpu->arch.exception.has_error_code; -u32 error_code = vcpu->arch.exception.error_code; -kvm_deliver_exception_payload (vcpu); +kvm_deliver_exception_payload (vcpu, ex); -if (nr ==BP_VECTOR && !nrips) { +if (ex->vector ==BP_VECTOR && !nrips) {unsigned long rip, old_rip =kvm_rip_read (vcpu);/*
@@ -407,11 +405,11 @@ static void svm_inject_exception(struct kvm_vcpu *vcpu) svm->int3_injected = rip - old_rip; } - svm->vmcb->control.event_inj = nr + svm->vmcb->control.event_inj = ex->vector |SVM_EVTINJ_VALID - | (has_error_code ?SVM_EVTINJ_VALID_ERR :0 ) + | (ex->has_error_code ?SVM_EVTINJ_VALID_ERR :0 ) |SVM_EVTINJ_TYPE_EXEPT ; - svm->vmcb->control.event_inj_err = error_code; + svm->vmcb->control.event_inj_err = ex->error_code; }static void svm_init_erratum_383 (void )
arch/x86/kvm/vmx/nested.c +21 -23
87 lines not on this page · raw message
arch/x86/kvm/vmx/vmx.c +9 -11
@@ -1577,7 +1577,7 @@ static void vmx_update_emulated_instruction(struct kvm_vcpu *vcpu)*/ if (nested_cpu_has_mtf (vmcs12) && (!vcpu->arch.exception.pending || - vcpu->arch.exception.nr ==DB_VECTOR )) + vcpu->arch.exception.vector ==DB_VECTOR )) vmx->nested.mtf_pending = true;else vmx->nested.mtf_pending = false;
@@ -1604,15 +1604,13 @@ static void vmx_clear_hlt(struct kvm_vcpu *vcpu)static void vmx_inject_exception (struct kvm_vcpu *vcpu) { +struct kvm_queued_exception *ex = &vcpu->arch.exception; +u32 intr_info = ex->vector |INTR_INFO_VALID_MASK ;struct vcpu_vmx *vmx =to_vmx (vcpu); -unsigned nr = vcpu->arch.exception.nr; -bool has_error_code = vcpu->arch.exception.has_error_code; -u32 error_code = vcpu->arch.exception.error_code; -u32 intr_info = nr |INTR_INFO_VALID_MASK ; -kvm_deliver_exception_payload (vcpu); +kvm_deliver_exception_payload (vcpu, ex); -if (has_error_code) { +if (ex->has_error_code) {/* * Despite the error code being architecturally defined as 32 * bits, and the VMCS field being 32 bits, Intel CPUs and thus
@@ -1622,21 +1620,21 @@ static void vmx_inject_exception(struct kvm_vcpu *vcpu)* the upper bits to avoid VM-Fail, losing information that * does't really exist is preferable to killing the VM. */ -vmcs_write32 (VM_ENTRY_EXCEPTION_ERROR_CODE , (u16 )error_code); +vmcs_write32 (VM_ENTRY_EXCEPTION_ERROR_CODE , (u16 )ex->error_code); intr_info |=INTR_INFO_DELIVER_CODE_MASK ; }if (vmx->rmode.vm86_active) {int inc_eip =0 ; -if (kvm_exception_is_soft (nr)) +if (kvm_exception_is_soft (ex->vector)) inc_eip = vcpu->arch.event_exit_inst_len; -kvm_inject_realmode_interrupt (vcpu, nr, inc_eip); +kvm_inject_realmode_interrupt (vcpu, ex->vector, inc_eip);return ; }WARN_ON_ONCE (vmx->emulation_required); -if (kvm_exception_is_soft (nr)) { +if (kvm_exception_is_soft (ex->vector)) {vmcs_write32 (VM_ENTRY_INSTRUCTION_LEN , vmx->vcpu.arch.event_exit_inst_len); intr_info |=INTR_TYPE_SOFT_EXCEPTION ;
arch/x86/kvm/x86.c +38 -42
175 lines not on this page · raw message
arch/x86/kvm/x86.h +2 -1
@@ -270,7 +270,8 @@ int kvm_write_guest_virt_system(struct kvm_vcpu *vcpu,int handle_ud (struct kvm_vcpu *vcpu); -void kvm_deliver_exception_payload (struct kvm_vcpu *vcpu); +void kvm_deliver_exception_payload (struct kvm_vcpu *vcpu, +struct kvm_queued_exception *ex);void kvm_vcpu_mtrr_init (struct kvm_vcpu *vcpu);u8 kvm_mtrr_get_guest_memory_type (struct kvm_vcpu *vcpu,gfn_t gfn);
Signature · 1 line
2.35.1.723.g4982287a31-goog
Sean Christopherson
maintainerPATCH 14/21 KVM: x86: Formalize blocking of nested pending exceptions
In reply to Sean ChristophersonCapture nested_run_pending as block_pending_exceptions so that the logic of why exceptions are blocked only needs to be documented once instead of at every place that employs the logic.
No functional change intended.
--- arch/x86/kvm/svm/nested.c | 20 ++++++++++---------- arch/x86/kvm/vmx/nested.c | 23 ++++++++++++----------- 2 files changed, 22 insertions(+), 21 deletions(-)
arch/x86/kvm/svm/nested.c +10 -10
@@ -1201,10 +1201,16 @@ static inline bool nested_exit_on_init(struct vcpu_svm *svm)static int svm_check_nested_events (struct kvm_vcpu *vcpu) { -struct vcpu_svm *svm =to_svm (vcpu); -bool block_nested_events = -kvm_event_needs_reinjection (vcpu) || svm->nested.nested_run_pending;struct kvm_lapic *apic = vcpu->arch.apic; +struct vcpu_svm *svm =to_svm (vcpu); +/* +* Only a pending nested run blocks a pending exception. If there is a +* previously injected event, the pending exception occurred while said +* event was being delivered and thus needs to be handled. +*/ +bool block_nested_exceptions = svm->nested.nested_run_pending; +bool block_nested_events = block_nested_exceptions || +kvm_event_needs_reinjection (vcpu);if (lapic_in_kernel (vcpu) &&test_bit (KVM_APIC_INIT , &apic->pending_events)) {
@@ -1217,13 +1223,7 @@ static int svm_check_nested_events(struct kvm_vcpu *vcpu) }if (vcpu->arch.exception.pending) { -/* -* Only a pending nested run can block a pending exception. -* Otherwise an injected NMI/interrupt should either be -* lost or delivered to the nested hypervisor in the EXITINTINFO -* vmcb field, while delivering the pending exception. -*/ -if (svm->nested.nested_run_pending) +if (block_nested_exceptions)return -EBUSY ;if (!nested_exit_on_exception (svm))return 0 ;
arch/x86/kvm/vmx/nested.c +12 -11
@@ -3878,11 +3878,17 @@ static bool nested_vmx_preemption_timer_pending(struct kvm_vcpu *vcpu)static int vmx_check_nested_events (struct kvm_vcpu *vcpu) { -struct vcpu_vmx *vmx =to_vmx (vcpu); -unsigned long exit_qual; -bool block_nested_events = - vmx->nested.nested_run_pending ||kvm_event_needs_reinjection (vcpu);struct kvm_lapic *apic = vcpu->arch.apic; +struct vcpu_vmx *vmx =to_vmx (vcpu); +unsigned long exit_qual; +/* +* Only a pending nested run blocks a pending exception. If there is a +* previously injected event, the pending exception occurred while said +* event was being delivered and thus needs to be handled. +*/ +bool block_nested_exceptions = vmx->nested.nested_run_pending; +bool block_nested_events = block_nested_exceptions || +kvm_event_needs_reinjection (vcpu);if (lapic_in_kernel (vcpu) &&test_bit (KVM_APIC_INIT , &apic->pending_events)) {
@@ -3916,15 +3922,10 @@ static int vmx_check_nested_events(struct kvm_vcpu *vcpu)* Process exceptions that are higher priority than Monitor Trap Flag: * fault-like exceptions, TSS T flag #DB (not emulated by KVM, but * could theoretically come in from userspace), and ICEBP (INT1). -* -* Note that only a pending nested run can block a pending exception. -* Otherwise an injected NMI/interrupt should either be -* lost or delivered to the nested hypervisor in the IDT_VECTORING_INFO, -* while delivering the pending exception. */ if (vcpu->arch.exception.pending && !(vmx_get_pending_dbg_trap (vcpu) & ~DR6_BT )) { -if (vmx->nested.nested_run_pending) +if (block_nested_exceptions)return -EBUSY ;if (!nested_vmx_check_exception (vcpu, &exit_qual)) goto no_vmexit;
@@ -3941,7 +3942,7 @@ static int vmx_check_nested_events(struct kvm_vcpu *vcpu) }if (vcpu->arch.exception.pending) { -if (vmx->nested.nested_run_pending) +if (block_nested_exceptions)return -EBUSY ;if (!nested_vmx_check_exception (vcpu, &exit_qual)) goto no_vmexit;
Signature · 1 line
2.35.1.723.g4982287a31-goog
Sean Christopherson
maintainerPATCH 15/21 KVM: x86: Use kvm_queue_exception_e() to queue #DF
In reply to Sean ChristophersonQueue #DF by recursing on kvm_multiple_exception() by way of kvm_queue_exception_e() instead of open coding the behavior. This will allow KVM to Just Work when a future commit moves exception interception checks (for L2 => L1) into kvm_multiple_exception().
No functional change intended.
--- arch/x86/kvm/x86.c | 21 +++++++++------------ 1 file changed, 9 insertions(+), 12 deletions(-)
arch/x86/kvm/x86.c +9 -12
@@ -669,25 +669,22 @@ static void kvm_multiple_exception(struct kvm_vcpu *vcpu, } class1 =exception_class (prev_nr); class2 =exception_class (nr); -if ((class1 ==EXCPT_CONTRIBUTORY && class2 ==EXCPT_CONTRIBUTORY ) - || (class1 ==EXCPT_PF && class2 !=EXCPT_BENIGN )) { +if ((class1 ==EXCPT_CONTRIBUTORY && class2 ==EXCPT_CONTRIBUTORY ) || + (class1 ==EXCPT_PF && class2 !=EXCPT_BENIGN )) {/* -* Generate double fault per SDM Table 5-5. Set -* exception.pending = true so that the double fault -* can trigger a nested vmexit. +* Synthesize #DF. Clear the previously injected or pending +* exception so as not to incorrectly trigger shutdown. */ - vcpu->arch.exception.pending = true; vcpu->arch.exception.injected = false; - vcpu->arch.exception.has_error_code = true; - vcpu->arch.exception.vector =DF_VECTOR ; - vcpu->arch.exception.error_code =0 ; - vcpu->arch.exception.has_payload = false; - vcpu->arch.exception.payload =0 ; - }else + vcpu->arch.exception.pending = false; + +kvm_queue_exception_e (vcpu,DF_VECTOR ,0 ); + }else {/* replace previous exception with a new one in a hope that instruction re-execution will regenerate lost exception */ goto queue; + } }void kvm_queue_exception (struct kvm_vcpu *vcpu,unsigned nr)
Signature · 1 line
2.35.1.723.g4982287a31-goog
Sean Christopherson
maintainerPATCH 16/21 KVM: x86: Hoist nested event checks above event injection logic
In reply to Sean ChristophersonPerform nested event checks before re-injecting exceptions/events into L2. If a pending exception causes VM-Exit to L1, re-injecting events into vmcs02 is premature and wasted effort. Take care to ensure events that need to be re-injected are still re-injected if checking for nested events "fails", i.e. if KVM needs to force an immediate entry+exit to complete the to-be-re-injecteed event.
Keep the "can_inject" logic the same for now; it too can be pushed below the nested checks, but is a slightly riskier change (see past bugs about events not being properly purged on nested VM-Exit).
Add and/or modify comments to better document the various interactions. Of note is the comment regarding "blocking" previously injected NMIs and IRQs if an exception is pending. The old comment isn't wrong strictly speaking, but it failed to capture the reason why the logic even exists.
--- arch/x86/kvm/x86.c | 89 +++++++++++++++++++++++++++------------------- 1 file changed, 53 insertions(+), 36 deletions(-)
arch/x86/kvm/x86.c +53 -36
@@ -9310,53 +9310,70 @@ static void kvm_inject_exception(struct kvm_vcpu *vcpu)static int inject_pending_event (struct kvm_vcpu *vcpu,bool *req_immediate_exit) { +bool can_inject = !kvm_event_needs_reinjection (vcpu);int r; -bool can_inject = true; -/* try to reinject previous events if any */ +/* +* Process nested events first, as nested VM-Exit supercedes event +* re-injection. If there's an event queued for re-injection, it will +* be saved into the appropriate vmc{b,s}12 fields on nested VM-Exit. +*/ +if (is_guest_mode (vcpu)) + r =kvm_check_nested_events (vcpu); +else + r =0 ; -if (vcpu->arch.exception.injected) { +/* +* Re-inject exceptions and events *especially* if immediate entry+exit +* to/from L2 is needed, as any event that has already been injected +* into L2 needs to complete its lifecycle before injecting a new event. +* +* Don't re-inject an NMI or interrupt if there is a pending exception. +* This collision arises if an exception occurred while vectoring the +* injected event, KVM intercepted said exception, and KVM ultimately +* determined the fault belongs to the guest and queues the exception +* for injection back into the guest. +* +* "Injected" interrupts can also collide with pending exceptions if +* userspace ignores the "ready for injection" flag and blindly queues +* an interrupt. In that case, prioritizing the exception is correct, +* as the exception "occurred" before the exit to userspace. Trap-like +* exceptions, e.g. most #DBs, have higher priority than interrupts. +* And while fault-like exceptions, e.g. #GP and #PF, are the lowest +* priority, they're only generated (pended) during instruction +* execution, and interrupts are recognized at instruction boundaries. +* Thus a pending fault-like exception means the fault occurred on the +* *previous* instruction and must be serviced prior to recognizing any +* new events in order to fully complete the previous instruction. +*/ +if (vcpu->arch.exception.injected)kvm_inject_exception (vcpu); - can_inject = false; - } +else if (vcpu->arch.exception.pending) + ;/* see above */ +else if (vcpu->arch.nmi_injected) +static_call (kvm_x86_inject_nmi)(vcpu); +else if (vcpu->arch.interrupt.injected) +static_call (kvm_x86_inject_irq)(vcpu); +/* -* Do not inject an NMI or interrupt if there is a pending -* exception. Exceptions and interrupts are recognized at -* instruction boundaries, i.e. the start of an instruction. -* Trap-like exceptions, e.g. #DB, have higher priority than -* NMIs and interrupts, i.e. traps are recognized before an -* NMI/interrupt that's pending on the same instruction. -* Fault-like exceptions, e.g. #GP and #PF, are the lowest -* priority, but are only generated (pended) during instruction -* execution, i.e. a pending fault-like exception means the -* fault occurred on the *previous* instruction and must be -* serviced prior to recognizing any new events in order to -* fully complete the previous instruction. +* Exceptions that morph to VM-Exits are handled above, and pending +* exceptions on top of injected exceptions that do not VM-Exit should +* either morph to #DF or, sadly, override the injected exception. */ -else if (!vcpu->arch.exception.pending) { -if (vcpu->arch.nmi_injected) { -static_call (kvm_x86_inject_nmi)(vcpu); - can_inject = false; - }else if (vcpu->arch.interrupt.injected) { -static_call (kvm_x86_inject_irq)(vcpu); - can_inject = false; - } - } -WARN_ON_ONCE (vcpu->arch.exception.injected && vcpu->arch.exception.pending);/* -* Call check_nested_events() even if we reinjected a previous event -* in order for caller to determine if it should require immediate-exit -* from L2 to L1 due to pending L1 events which require exit -* from L2 to L1. +* Bail if immediate entry+exit to/from the guest is needed to complete +* nested VM-Enter or event re-injection so that a different pending +* event can be serviced (or if KVM needs to exit to userspace). +* +* Otherwise, continue processing events even if VM-Exit occurred. The +* VM-Exit will have cleared exceptions that were meant for L2, but +* there may now be events that can be injected into L1. */ -if (is_guest_mode (vcpu)) { - r =kvm_check_nested_events (vcpu); -if (r <0 ) - goto out; - } +if (r <0 ) + goto out;/* try to inject new event if pending */ if (vcpu->arch.exception.pending) {
Signature · 1 line
2.35.1.723.g4982287a31-goog
Sean Christopherson
maintainerPATCH 17/21 KVM: x86: Evaluate ability to inject SMI/NMI/IRQ after potential VM-Exit
In reply to Sean ChristophersonDetermine whether or not new events can be injected after checking nested events. If a VM-Exit occurred during nested event handling, any previous event that needed re-injection is gone from's KVM perspective; the event is captured in the vmc*12 VM-Exit information, but doesn't exist in terms of what needs to be done for entry to L1.
--- arch/x86/kvm/x86.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)
arch/x86/kvm/x86.c +8 -2
@@ -9310,7 +9310,7 @@ static void kvm_inject_exception(struct kvm_vcpu *vcpu)static int inject_pending_event (struct kvm_vcpu *vcpu,bool *req_immediate_exit) { -bool can_inject = !kvm_event_needs_reinjection (vcpu); +bool can_inject;int r;/*
@@ -9375,7 +9375,13 @@ static int inject_pending_event(struct kvm_vcpu *vcpu, bool *req_immediate_exit)if (r <0 ) goto out; -/* try to inject new event if pending */ +/* +* New events, other than exceptions, cannot be injected if KVM needs +* to re-inject a previous event. See above comments on re-injecting +* for why pending exceptions get priority. +*/ + can_inject = !kvm_event_needs_reinjection (vcpu); +if (vcpu->arch.exception.pending) { trace_kvm_inj_exception(vcpu->arch.exception.vector, vcpu->arch.exception.has_error_code,
Signature · 1 line
2.35.1.723.g4982287a31-goog
Sean Christopherson
maintainerPATCH 18/21 KVM: x86: Morph pending exceptions to pending VM-Exits at queue time
In reply to Sean ChristophersonMorph pending exceptions to pending VM-Exits (due to interception) when the exception is queued instead of waiting until nested events are checked at VM-Entry. This fixes a longstanding bug where KVM fails to handle an exception that occurs during delivery of a previous exception, KVM (L0) and L1 both want to intercept the exception (e.g. #PF for shadow paging), and KVM determines that the exception is in the guest's domain, i.e. queues the new exception for L2. Deferring the interception check causes KVM to esclate various combinations of injected+pending exceptions to double fault (#DF) without consulting L1's interception desires, and ends up injecting a spurious #DF into L2.
KVM has fudged around the issue for #PF by special casing emulated #PF injection for shadow paging, but the underlying issue is not unique to shadow paging in L0, e.g. if KVM is intercepting #PF because the guest has a smaller maxphyaddr and L1 (but not L0) is using shadow paging. Other exceptions are affected as well, e.g. if KVM is intercepting #GP for one of SVM's workaround or for the VMware backdoor emulation stuff. The other cases have gone unnoticed because the #DF is spurious if and only if L1 resolves the exception, e.g. KVM's goofs go unnoticed if L1 would have injected #DF anyways.
The hack-a-fix has also led to ugly code, e.g. bailing from the emulator if #PF injection forced a nested VM-Exit and the emulator finds itself back in L1. Allowing for direct-to-VM-Exit queueing also neatly solves the async #PF in L2 mess; no need to set a magic flag and token, simply queue a #PF nested VM-Exit.
Deal with event migration by flagging that a pending exception was queued by userspace and check for interception at the next KVM_RUN, e.g. so that KVM does the right thing regardless of the order in which userspace restores nested state vs. event state.
When "getting" events from userspace, simply drop any pending excpetion that is destined to be intercepted if there is also an injected exception to be migrated. Ideally, KVM would migrate both events, but that would require new ABI, and practically speaking losing the event is unlikely to be noticed, let alone fatal. The injected exception is captured, RIP still points at the original faulting instruction, etc... So either the injection on the target will trigger the same intercepted exception, or the source of the intercepted exception was transient and/or non-deterministic, thus dropping it is ok-ish.
Opportunistically add a gigantic comment above vmx_check_nested_events() to document the priorities of all known events on Intel CPUs. Kudos to Jim Mattson for doing the hard work of collecting and interpreting the priorities from various locations throughtout the SDM (because putting them all in one place in the SDM would be too easy).
--- arch/x86/include/asm/kvm_host.h | 10 +- arch/x86/kvm/svm/nested.c | 39 ++---- arch/x86/kvm/vmx/nested.c | 223 +++++++++++++++++++++----------- arch/x86/kvm/vmx/vmx.c | 6 +- arch/x86/kvm/x86.c | 142 +++++++++++++++----- arch/x86/kvm/x86.h | 7 + 6 files changed, 288 insertions(+), 139 deletions(-)
arch/x86/include/asm/kvm_host.h +7 -3
@@ -623,7 +623,6 @@ struct kvm_queued_exception {u32 error_code;unsigned long payload;bool has_payload; -u8 nested_apf; };struct kvm_vcpu_arch {
@@ -724,8 +723,12 @@ struct kvm_vcpu_arch {u8 event_exit_inst_len; +bool exception_from_userspace; +/* Exceptions to be injected to the guest. */ struct kvm_queued_exception exception; +/* Exception VM-Exits to be synthesized to L1. */ +struct kvm_queued_exception exception_vmexit;struct kvm_queued_interrupt {bool injected;
@@ -836,7 +839,6 @@ struct kvm_vcpu_arch {u32 id;bool send_user_only;u32 host_apf_flags; -unsigned long nested_apf_token;bool delivery_as_pf_vmexit;bool pageready_pending; } apf;
@@ -1500,6 +1502,8 @@ struct kvm_x86_ops {struct kvm_x86_nested_ops {void (*leave_nested)(struct kvm_vcpu *vcpu); +bool (*is_exception_vmexit)(struct kvm_vcpu *vcpu,u8 vector, +u32 error_code);int (*check_events)(struct kvm_vcpu *vcpu);bool (*hv_timer_pending)(struct kvm_vcpu *vcpu);void (*triple_fault)(struct kvm_vcpu *vcpu);
@@ -1754,7 +1758,7 @@ void kvm_queue_exception_p(struct kvm_vcpu *vcpu, unsigned nr, unsigned long payvoid kvm_requeue_exception (struct kvm_vcpu *vcpu,unsigned nr);void kvm_requeue_exception_e (struct kvm_vcpu *vcpu,unsigned nr,u32 error_code);void kvm_inject_page_fault (struct kvm_vcpu *vcpu,struct x86_exception *fault); -bool kvm_inject_emulated_page_fault (struct kvm_vcpu *vcpu, +void kvm_inject_emulated_page_fault (struct kvm_vcpu *vcpu,struct x86_exception *fault);bool kvm_require_cpl (struct kvm_vcpu *vcpu,int required_cpl);bool kvm_require_dr (struct kvm_vcpu *vcpu,int dr);
arch/x86/kvm/svm/nested.c +12 -27
@@ -54,24 +54,6 @@ static void nested_svm_inject_npf_exit(struct kvm_vcpu *vcpu,nested_svm_vmexit (svm); } -static void svm_inject_page_fault_nested (struct kvm_vcpu *vcpu,struct x86_exception *fault) -{ -struct vcpu_svm *svm =to_svm (vcpu); -WARN_ON (!is_guest_mode (vcpu)); - -if (vmcb12_is_intercept (&svm->nested.ctl, -INTERCEPT_EXCEPTION_OFFSET +PF_VECTOR ) && - !svm->nested.nested_run_pending) { - svm->vmcb->control.exit_code =SVM_EXIT_EXCP_BASE +PF_VECTOR ; - svm->vmcb->control.exit_code_hi =0 ; - svm->vmcb->control.exit_info_1 = fault->error_code; - svm->vmcb->control.exit_info_2 = fault->address; -nested_svm_vmexit (svm); - }else { -kvm_inject_page_fault (vcpu, fault); - } -} -static u64 nested_svm_get_tdp_pdptr (struct kvm_vcpu *vcpu,int index) {struct vcpu_svm *svm =to_svm (vcpu);
@@ -680,9 +662,6 @@ int enter_svm_guest_mode(struct kvm_vcpu *vcpu, u64 vmcb12_gpa,if (ret)return ret; -if (!npt_enabled) - vcpu->arch.mmu->inject_page_fault = svm_inject_page_fault_nested; -if (!from_vmrun)kvm_make_request (KVM_REQ_GET_NESTED_STATE_PAGES , vcpu);
@@ -1152,16 +1131,17 @@ int nested_svm_check_permissions(struct kvm_vcpu *vcpu)return 0 ; } -static bool nested_exit_on_exception (struct vcpu_svm *svm) +static bool nested_svm_is_exception_vmexit (struct kvm_vcpu *vcpu,u8 vector, +u32 error_code) { -unsigned int vector = svm->vcpu.arch.exception.vector; +struct vcpu_svm *svm =to_svm (vcpu);return (svm->nested.ctl.intercepts[INTERCEPT_EXCEPTION ] &BIT (vector)); }static void nested_svm_inject_exception_vmexit (struct kvm_vcpu *vcpu) { -struct kvm_queued_exception *ex = &vcpu->arch.exception; +struct kvm_queued_exception *ex = &vcpu->arch.exception_vmexit;struct vcpu_svm *svm =to_svm (vcpu); svm->vmcb->control.exit_code =SVM_EXIT_EXCP_BASE + ex->vector;
@@ -1222,15 +1202,19 @@ static int svm_check_nested_events(struct kvm_vcpu *vcpu)return 0 ; } -if (vcpu->arch.exception.pending) { +if (vcpu->arch.exception_vmexit.pending) {if (block_nested_exceptions)return -EBUSY ; -if (!nested_exit_on_exception (svm)) -return 0 ;nested_svm_inject_exception_vmexit (vcpu);return 0 ; } +if (vcpu->arch.exception.pending) { +if (block_nested_exceptions) +return -EBUSY ; +return 0 ; + } +if (vcpu->arch.smi_pending && !svm_smi_blocked (vcpu)) {if (block_nested_events)return -EBUSY ;
@@ -1567,6 +1551,7 @@ static bool svm_get_nested_state_pages(struct kvm_vcpu *vcpu)struct kvm_x86_nested_ops svm_nested_ops = { .leave_nested = svm_leave_nested, + .is_exception_vmexit = nested_svm_is_exception_vmexit, .check_events = svm_check_nested_events, .triple_fault = nested_svm_triple_fault, .get_nested_state_pages = svm_get_nested_state_pages,
arch/x86/kvm/vmx/nested.c +147 -76
299 lines not on this page · raw message
arch/x86/kvm/vmx/vmx.c +4 -2
@@ -1577,7 +1577,9 @@ static void vmx_update_emulated_instruction(struct kvm_vcpu *vcpu)*/ if (nested_cpu_has_mtf (vmcs12) && (!vcpu->arch.exception.pending || - vcpu->arch.exception.vector ==DB_VECTOR )) + vcpu->arch.exception.vector ==DB_VECTOR ) && + (!vcpu->arch.exception_vmexit.pending || + vcpu->arch.exception_vmexit.vector ==DB_VECTOR )) vmx->nested.mtf_pending = true;else vmx->nested.mtf_pending = false;
@@ -5479,7 +5481,7 @@ static bool vmx_emulation_required_with_pending_exception(struct kvm_vcpu *vcpu)struct vcpu_vmx *vmx =to_vmx (vcpu);return vmx->emulation_required && !vmx->rmode.vm86_active && - vcpu->arch.exception.pending; +kvm_is_exception_pending (vcpu); }static int handle_invalid_guest_state (struct kvm_vcpu *vcpu)
arch/x86/kvm/x86.c +111 -31
280 lines not on this page · raw message
arch/x86/kvm/x86.h +7 -0
@@ -61,10 +61,17 @@ static inline unsigned int __shrink_ple_window(unsigned int val,void kvm_service_local_tlb_flush_requests (struct kvm_vcpu *vcpu);int kvm_check_nested_events (struct kvm_vcpu *vcpu); +static inline bool kvm_is_exception_pending (struct kvm_vcpu *vcpu) +{ +return vcpu->arch.exception.pending || + vcpu->arch.exception_vmexit.pending; +} +static inline void kvm_clear_exception_queue (struct kvm_vcpu *vcpu) { vcpu->arch.exception.pending = false; vcpu->arch.exception.injected = false; + vcpu->arch.exception_vmexit.pending = false; } static inline void kvm_queue_interrupt(struct kvm_vcpu *vcpu, u8 vector,
Signature · 1 line
2.35.1.723.g4982287a31-goog
Sean Christopherson
maintainerPATCH 19/21 KVM: VMX: Update MTF and ICEBP comments to document KVM's subtle behavior
In reply to Sean ChristophersonDocument the oddities of ICEBP interception (trap-like #DB is intercepted as a fault-like exception), and how using VMX's inner "skip" helper deliberately bypasses the pending MTF and single-step #DB logic.
No functional change intended.
--- arch/x86/kvm/vmx/vmx.c | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-)
arch/x86/kvm/vmx/vmx.c +11 -5
@@ -1570,9 +1570,13 @@ static void vmx_update_emulated_instruction(struct kvm_vcpu *vcpu)/* * Per the SDM, MTF takes priority over debug-trap exceptions besides -* T-bit traps. As instruction emulation is completed (i.e. at the -* instruction boundary), any #DB exception pending delivery must be a -* debug-trap. Record the pending MTF state to be delivered in +* TSS T-bit traps and ICEBP (INT1). KVM doesn't emulate T-bit traps +* or ICEBP (in the emulator proper), and skipping of ICEBP after an +* intercepted #DB deliberately avoids single-step #DB and MTF updates +* as ICEBP is higher priority than both. As instruction emulation is +* completed at this point (i.e. KVM is at the instruction boundary), +* any #DB exception pending delivery must be a debug-trap of lower +* priority than MTF. Record the pending MTF state to be delivered in * vmx_check_nested_events(). */ if (nested_cpu_has_mtf(vmcs12) &&
@@ -4924,8 +4928,10 @@ static int handle_exception_nmi(struct kvm_vcpu *vcpu)* instruction. ICEBP generates a trap-like #DB, but * despite its interception control being tied to #DB, * is an instruction intercept, i.e. the VM-Exit occurs -* on the ICEBP itself. Note, skipping ICEBP also -* clears STI and MOVSS blocking. +* on the ICEBP itself. Use the inner "skip" helper to +* avoid single-step #DB and MTF updates, as ICEBP is +* higher priority. Note, skipping ICEBP still clears +* STI and MOVSS blocking. * * For all other #DBs, set vmcs.PENDING_DBG_EXCEPTIONS.BS * if single-step is enabled in RFLAGS and STI or MOVSS
Signature · 1 line
2.35.1.723.g4982287a31-goog
Sean Christopherson
maintainerPATCH 20/21 KVM: selftests: Use uapi header to get VMX and SVM exit reasons/codes
In reply to Sean ChristophersonInclude the vmx.h and svm.h uapi headers that KVM so kindly provides instead of manually defining all the same exit reasons/code.
--- .../selftests/kvm/include/x86_64/svm_util.h | 5 +- .../selftests/kvm/include/x86_64/vmx.h | 51 +------------------ 2 files changed, 4 insertions(+), 52 deletions(-)
tools/testing/selftests/kvm/include/x86_64/svm_util.h +2 -3
@@ -9,6 +9,8 @@#ifndef SELFTEST_KVM_SVM_UTILS_H #define SELFTEST_KVM_SVM_UTILS_H +#include <asm/svm.h> +#include <stdint.h> #include "svm.h" #include "processor.h"
@@ -16,9 +18,6 @@#define CPUID_SVM_BIT 2#define CPUID_SVM BIT_ULL(CPUID_SVM_BIT) -#define SVM_EXIT_MSR 0x07c -#define SVM_EXIT_VMMCALL 0x081 -struct svm_test_data {/* VMCB */ struct vmcb *vmcb;/* gva */
tools/testing/selftests/kvm/include/x86_64/vmx.h +2 -49
63 lines not on this page · raw message
Signature · 1 line
2.35.1.723.g4982287a31-goog
Sean Christopherson
maintainerPATCH 21/21 KVM: selftests: Add an x86-only test to verify nested exception queueing
In reply to Sean ChristophersonAdd a test to verify that KVM_{G,S}ET_EVENTS play nice with pending vs. injected exceptions when an exception is being queued for L2, and that KVM correctly handles L1's exception intercept wants.
--- tools/testing/selftests/kvm/.gitignore | 1 + tools/testing/selftests/kvm/Makefile | 1 + .../kvm/x86_64/nested_exceptions_test.c | 307 ++++++++++++++++++ 3 files changed, 309 insertions(+) create mode 100644 tools/testing/selftests/kvm/x86_64/nested_exceptions_test.c
tools/testing/selftests/kvm/.gitignore +1 -0
@@ -23,6 +23,7 @@ /x86_64/hyperv_features /x86_64/mmio_warning_test /x86_64/mmu_role_test +/x86_64/nested_exceptions_test /x86_64/platform_info_test /x86_64/pmu_event_filter_test /x86_64/set_boot_cpu_id
tools/testing/selftests/kvm/Makefile +1 -0
@@ -56,6 +56,7 @@ TEST_GEN_PROGS_x86_64 += x86_64/kvm_clock_test TEST_GEN_PROGS_x86_64 += x86_64/kvm_pv_test TEST_GEN_PROGS_x86_64 += x86_64/mmio_warning_test TEST_GEN_PROGS_x86_64 += x86_64/mmu_role_test +TEST_GEN_PROGS_x86_64 += x86_64/nested_exceptions_test TEST_GEN_PROGS_x86_64 += x86_64/platform_info_test TEST_GEN_PROGS_x86_64 += x86_64/pmu_event_filter_test TEST_GEN_PROGS_x86_64 += x86_64/set_boot_cpu_id
tools/testing/selftests/kvm/x86_64/nested_exceptions_test.c new file +307 -0
307 lines not on this page · raw message
Signature · 1 line
2.35.1.723.g4982287a31-goog
Hi Sean,
Quoted 8 lines · Sean Christopherson
The main goal of this series is to fix KVM's longstanding bug of not honoring L1's exception intercepts wants when handling an exception that occurs during delivery of a different exception. E.g. if L0 and L1 are using shadow paging, and L2 hits a #PF, and then hits another #PF while vectoring the first #PF due to _L1_ not having a shadow page for the IDT, KVM needs to check L1's intercepts before morphing the #PF => #PF => #DF so that the #PF is routed to L1, not injected into L2 as a #DF.
Quoted 5 lines
nVMX has hacked around the bug for years by overriding the #PF injector for shadow paging to go straight to VM-Exit, and nSVM has started doing the same. The hacks mostly work, but they're incomplete, confusing, and lead to other hacky code, e.g. bailing from the emulator because #PF injection forced a VM-Exit and suddenly KVM is back in L1.
Looks like we were working on similar KVM area recently [1].
It look like parts of our patch sets touch the same code. Since your patch set is much bigger and comprehensive I will base mine on top of yours once there are no more incoming review comments for your patch set (in other words, once it is in its final form).
Thanks,
Maciej
Quoted 69 lines · Sean Christopherson
The main goal of this series is to fix KVM's longstanding bug of not honoring L1's exception intercepts wants when handling an exception that occurs during delivery of a different exception. E.g. if L0 and L1 are using shadow paging, and L2 hits a #PF, and then hits another #PF while vectoring the first #PF due to _L1_ not having a shadow page for the IDT, KVM needs to check L1's intercepts before morphing the #PF => #PF => #DF so that the #PF is routed to L1, not injected into L2 as a #DF.
nVMX has hacked around the bug for years by overriding the #PF injector for shadow paging to go straight to VM-Exit, and nSVM has started doing the same. The hacks mostly work, but they're incomplete, confusing, and lead to other hacky code, e.g. bailing from the emulator because #PF injection forced a VM-Exit and suddenly KVM is back in L1.
Everything leading up to that are related fixes and cleanups I encountered along the way; some through code inspection, some through tests (I truly thought this series was finished 10 commits and 3 days ago...).
Nothing in here is all that urgent; all bugs tagged for stable have been around for multiple releases (years in most cases).
Sean Christopherson (21):
KVM: x86: Return immediately fromx86_emulate_instruction()on code#DB KVM: nVMX: Unconditionally purge queued/injected events on nested "exit" KVM: VMX: Drop bits 31:16 when shoving exception error code into VMCS KVM: x86: Don't check for code breakpoints when emulating on exception KVM: nVMX: Treat General Detect #DB (DR7.GD=1) as fault-like KVM: nVMX: Prioritize TSS T-flag #DBs over Monitor Trap Flag KVM: x86: Treat #DBs from the emulator as fault-like (code and DR7.GD=1)KVM: x86: Use
DR7_GDmacro instead of open coding check in emulator
KVM: nVMX: Ignore SIPI that arrives in L2 when vCPU is not in WFS
KVM: nVMX: Unconditionally clearmtf_pendingon nested VM-Exit
KVM: VMX: Inject #PF on ENCLS as "emulated" #PF
KVM: x86: Rename kvm_x86_ops.queue_exception toinject_exception
KVM: x86: Makekvm_queued_exceptiona properly named, visible struct
KVM: x86: Formalize blocking of nested pending exceptions
KVM: x86: Usekvm_queue_exception_e()to queue #DF
KVM: x86: Hoist nested event checks above event injection logic
KVM: x86: Evaluate ability to inject SMI/NMI/IRQ after potential
VM-Exit
KVM: x86: Morph pending exceptions to pending VM-Exits at queue time
KVM: VMX: Update MTF and ICEBP comments to document KVM's subtle
behavior
KVM: selftests: Use uapi header to get VMX and SVM exit reasons/codes
KVM: selftests: Add an x86-only test to verify nested exception
queueingarch/x86/include/asm/kvm-x86-ops.h | 2 +- arch/x86/include/asm/kvm_host.h | 33 +- arch/x86/kvm/emulate.c | 3 +- arch/x86/kvm/svm/nested.c | 100 ++--- arch/x86/kvm/svm/svm.c | 18 +- arch/x86/kvm/vmx/nested.c | 322 +++++++++----- arch/x86/kvm/vmx/sgx.c | 2 +- arch/x86/kvm/vmx/vmx.c | 53 ++- arch/x86/kvm/x86.c | 409 ++++++++++++------ arch/x86/kvm/x86.h | 10 +- tools/testing/selftests/kvm/.gitignore | 1 + tools/testing/selftests/kvm/Makefile | 1 + .../selftests/kvm/include/x86_64/svm_util.h | 5 +- .../selftests/kvm/include/x86_64/vmx.h | 51 +-- .../kvm/x86_64/nested_exceptions_test.c | 307 +++++++++++++ 15 files changed, 914 insertions(+), 403 deletions(-) create mode 100644 tools/testing/selftests/kvm/x86_64/nested_exceptions_test.c
Quoted 1 line
base-commit:
4a204f7895878363ca8211f50ec610408c8c70aa
I am just curious. Are you aware that I worked on this few months ago? I am sure that you even reviewed some of my code back then.
If so, could you have had at least mentioned this and/or pinged me to continue working on this instead of re-implementing it?
Best regards,
Maxim Levitsky
Sean Christopherson
maintainerQuoted 15 lines · Maxim Levitsky
Quoted 14 lines · Sean Christopherson
The main goal of this series is to fix KVM's longstanding bug of not honoring L1's exception intercepts wants when handling an exception that occurs during delivery of a different exception. E.g. if L0 and L1 are using shadow paging, and L2 hits a #PF, and then hits another #PF while vectoring the first #PF due to _L1_ not having a shadow page for the IDT, KVM needs to check L1's intercepts before morphing the #PF => #PF => #DF so that the #PF is routed to L1, not injected into L2 as a #DF.
nVMX has hacked around the bug for years by overriding the #PF injector for shadow paging to go straight to VM-Exit, and nSVM has started doing the same. The hacks mostly work, but they're incomplete, confusing, and lead to other hacky code, e.g. bailing from the emulator because #PF injection forced a VM-Exit and suddenly KVM is back in L1.
Quoted 8 lines
Quoted 7 lines
Everything leading up to that are related fixes and cleanups I encountered along the way; some through code inspection, some through tests (I truly thought this series was finished 10 commits and 3 days ago...).
Nothing in here is all that urgent; all bugs tagged for stable have been around for multiple releases (years in most cases).
I am just curious. Are you aware that I worked on this few months ago?
Ah, so that's why I had a feeling of deja vu when factoring out kvm_queued_exception. I completely forgot about it :-/ In my defense, that was nearly a year ago[1][2], though I suppose one could argue 11 == "a few" :-)
[1] https://lore.kernel.org/all/[email protected]
[2] https://lore.kernel.org/all/[email protected]
Quoted 1 line
I am sure that you even reviewed some of my code back then.
Yep, now that I've found the threads I remember discussing the mechanics.
Quoted 2 lines
If so, could you have had at least mentioned this and/or pinged me to continue working on this instead of re-implementing it?
I'm invoking Hanlon's razor[*]; I certainly didn't intended to stomp over your work, I simply forgot.
As for the technical aspects, looking back at your series, I strongly considered taking the same approach of splitting pending vs. injected (again, without any recollection of your work). I ultimately opted to go with the "immediated morph to pending VM-Exit" approach as it allows KVM to do the right thing in almost every case without requiring new ABI, and even if KVM screws up, e.g. queues multiple pending exceptions. It also neatly handles one-off things like async #PF in L2.
However, I hadn't considered your approach, which addresses the ABI conundrum by processing pending=>injected immediately after handling the VM-Exit. I can't think of any reason that wouldn't work, but I really don't like splitting the event priority logic, nor do I like having two event injection sites (getting rid of the extra calls to kvm_check_nested_events() is still on my wish list). If we could go back in time, I would likely vote for properly tracking injected vs. pending, but since we're mostly stuck with KVM's ABI, I prefer the "immediately morph to pending VM-Exit" hack over the "immediately morph to 'injected' exception" hack.
Quoted 49 lines · Sean Christopherson
Quoted 15 lines · Maxim Levitsky
Quoted 14 lines · Sean Christopherson
The main goal of this series is to fix KVM's longstanding bug of not honoring L1's exception intercepts wants when handling an exception that occurs during delivery of a different exception. E.g. if L0 and L1 are using shadow paging, and L2 hits a #PF, and then hits another #PF while vectoring the first #PF due to _L1_ not having a shadow page for the IDT, KVM needs to check L1's intercepts before morphing the #PF => #PF => #DF so that the #PF is routed to L1, not injected into L2 as a #DF.
nVMX has hacked around the bug for years by overriding the #PF injector for shadow paging to go straight to VM-Exit, and nSVM has started doing the same. The hacks mostly work, but they're incomplete, confusing, and lead to other hacky code, e.g. bailing from the emulator because #PF injection forced a VM-Exit and suddenly KVM is back in L1.
Quoted 8 lines
Quoted 7 lines
Everything leading up to that are related fixes and cleanups I encountered along the way; some through code inspection, some through tests (I truly thought this series was finished 10 commits and 3 days ago...).
Nothing in here is all that urgent; all bugs tagged for stable have been around for multiple releases (years in most cases).
I am just curious. Are you aware that I worked on this few months ago?
Ah, so that's why I had a feeling of deja vu when factoring out
kvm_queued_exception. I completely forgot about it :-/ In my defense, that was nearly a year ago[1][2], though I suppose one could argue 11 == "a few" :-)[1] https://lore.kernel.org/all/[email protected]
[2] https://lore.kernel.org/all/[email protected]Quoted 1 line
I am sure that you even reviewed some of my code back then.
Yep, now that I've found the threads I remember discussing the mechanics.
Quoted 2 lines
If so, could you have had at least mentioned this and/or pinged me to continue working on this instead of re-implementing it?
I'm invoking Hanlon's razor[*]; I certainly didn't intended to stomp over your work, I simply forgot.
As for the technical aspects, looking back at your series, I strongly considered taking the same approach of splitting pending vs. injected (again, without any recollection of your work). I ultimately opted to go with the "immediated morph to pending VM-Exit" approach as it allows KVM to do the right thing in almost every case without requiring new ABI, and even if KVM screws up, e.g. queues multiple pending exceptions. It also neatly handles one-off things like async #PF in L2.
Quoted 8 lines
However, I hadn't considered your approach, which addresses the ABI conundrum by processing pending=>injected immediately after handling the VM-Exit. I can't think of any reason that wouldn't work, but I really don't like splitting the event priority logic, nor do I like having two event injection sites (getting rid of the extra calls to
kvm_check_nested_events()is still on my wish list). If we could go back in time, I would likely vote for properly tracking injected vs. pending, but since we're mostly stuck with KVM's ABI, I prefer the "immediately morph to pending VM-Exit" hack over the "immediately morph to 'injected' exception" hack.
So, what's the plan here: is your patch set Sean considered to supersede Maxim's earlier proposed changes or will you post an updated patch set incorporating at least some of them?
I am asking because I have a series that touches the same general area of KVM [1] and would preferably have it based on the final form of the event injection code to avoid unforeseen negative interactions between these changes.
Thanks,
Maciej
Sean Christopherson
maintainerQuoted 3 lines · Maciej S. Szmigiero
So, what's the plan here: is your patch set Sean considered to supersede Maxim's earlier proposed changes or will you post an updated patch set incorporating at least some of them?
Next step is to reach a consensus on how we want to solve the problem (or if we can't reach consensus, until Paolo uses his special powers). I definitely won't post anything new until there's more conversation.
Quoted 4 lines
I am asking because I have a series that touches the same general area of KVM [1] and would preferably have it based on the final form of the event injection code to avoid unforeseen negative interactions between these changes.
I don't think you need to do anything, at a glance your changes are orthogonal even though they have similar themes. Any conflicts should be minor.
Quoted 8 lines · Sean Christopherson
Quoted 3 lines · Maciej S. Szmigiero
So, what's the plan here: is your patch set Sean considered to supersede Maxim's earlier proposed changes or will you post an updated patch set incorporating at least some of them?
Next step is to reach a consensus on how we want to solve the problem (or if we can't reach consensus, until Paolo uses his special powers). I definitely won't post anything new until there's more conversation.
Ack.
Quoted 7 lines
Quoted 4 lines
I am asking because I have a series that touches the same general area of KVM [1] and would preferably have it based on the final form of the event injection code to avoid unforeseen negative interactions between these changes.
I don't think you need to do anything, at a glance your changes are orthogonal even though they have similar themes. Any conflicts should be minor.
All right - I will try to keep an eye on the developments related to the event injection code just to be sure there are no obvious conflicts here.
Thanks,
Maciej
Quoted 34 lines · Sean Christopherson
Quoted 15 lines · Maxim Levitsky
Quoted 14 lines · Sean Christopherson
The main goal of this series is to fix KVM's longstanding bug of not honoring L1's exception intercepts wants when handling an exception that occurs during delivery of a different exception. E.g. if L0 and L1 are using shadow paging, and L2 hits a #PF, and then hits another #PF while vectoring the first #PF due to _L1_ not having a shadow page for the IDT, KVM needs to check L1's intercepts before morphing the #PF => #PF => #DF so that the #PF is routed to L1, not injected into L2 as a #DF.
nVMX has hacked around the bug for years by overriding the #PF injector for shadow paging to go straight to VM-Exit, and nSVM has started doing the same. The hacks mostly work, but they're incomplete, confusing, and lead to other hacky code, e.g. bailing from the emulator because #PF injection forced a VM-Exit and suddenly KVM is back in L1.
Quoted 8 lines
Quoted 7 lines
Everything leading up to that are related fixes and cleanups I encountered along the way; some through code inspection, some through tests (I truly thought this series was finished 10 commits and 3 days ago...).
Nothing in here is all that urgent; all bugs tagged for stable have been around for multiple releases (years in most cases).
I am just curious. Are you aware that I worked on this few months ago?
Ah, so that's why I had a feeling of deja vu when factoring out
kvm_queued_exception. I completely forgot about it :-/ In my defense, that was nearly a year ago[1][2], though I suppose one could argue 11 == "a few" :-)[1] https://lore.kernel.org/all/[email protected]
[2] https://lore.kernel.org/all/[email protected]Quoted 1 line
I am sure that you even reviewed some of my code back then.
Quoted 7 lines
Yep, now that I've found the threads I remember discussing the mechanics.
Quoted 2 lines
If so, could you have had at least mentioned this and/or pinged me to continue working on this instead of re-implementing it?
I'm invoking Hanlon's razor[*]; I certainly didn't intended to stomp over your work, I simply forgot.
Thank you very much for the explanation, and I am glad that it was a honest mistake.
Other than that I am actually very happy that you posted this patch series, as this gives more chance that this long standing issue will be fixed,
and if your patches are better/simpler/less invasive to KVM and still address the issue, I fully support using them instead of mine.
Totally agree with you about your thoughts about splitting pending/injected exception, I also can't say I liked my approach that much, for the same reasons you mentioned.
It is also the main reason I put the whole thing on the backlog lately, because I was feeling that I am changing too much of the KVM, for a relatively theoretical issue.
I will review your patches, compare them to mine, and check if you or I missed something.
PS:
Back then, I also did an extensive review on few cases when qemu injects exceptions itself, which it does thankfully rarely. There are several (theoretical) issues there. I don't remember those details, I need to refresh my memory.
AFAIK, qemu injects #MC sometimes when it gets it from the kernel in form of a signal, if I recall this correctly, and it also reflects back #DB, when guest debug was enabled (and that is the reason for some work I did in this area, like the KVM_GUESTDBG_BLOCKIRQ thing)
Qemu does this without considering nested and/or pending exception/etc. It just kind of abuses the KVM_SET_VCPU_EVENTS for that.
Best regards,
Maxim Levitsky
Quoted 19 lines
As for the technical aspects, looking back at your series, I strongly considered taking the same approach of splitting pending vs. injected (again, without any recollection of your work). I ultimately opted to go with the "immediated morph to pending VM-Exit" approach as it allows KVM to do the right thing in almost every case without requiring new ABI, and even if KVM screws up, e.g. queues multiple pending exceptions. It also neatly handles one-off things like async #PF in L2.
However, I hadn't considered your approach, which addresses the ABI conundrum by processing pending=>injected immediately after handling the VM-Exit. I can't think of any reason that wouldn't work, but I really don't like splitting the event priority logic, nor do I like having two event injection sites (getting rid of the extra calls to
kvm_check_nested_events()is still on my wish list). If we could go back in time, I would likely vote for properly tracking injected vs. pending, but since we're mostly stuck with KVM's ABI, I prefer the "immediately morph to pending VM-Exit" hack over the "immediately morph to 'injected' exception" hack.
Sean Christopherson
maintainerQuoted 4 lines · Maxim Levitsky
Other than that I am actually very happy that you posted this patch series, as this gives more chance that this long standing issue will be fixed,
and if your patches are better/simpler/less invasive to KVM and still address the issue, I fully support using them instead of mine.
I highly doubt they're simpler or less invasive, but I do hope that the approach wil be easier to maintain.
Quoted 16 lines
Totally agree with you about your thoughts about splitting pending/injected exception, I also can't say I liked my approach that much, for the same reasons you mentioned.
It is also the main reason I put the whole thing on the backlog lately, because I was feeling that I am changing too much of the KVM, for a relatively theoretical issue.
I will review your patches, compare them to mine, and check if you or I missed something.
PS:
Back then, I also did an extensive review on few cases when qemu injects exceptions itself, which it does thankfully rarely. There are several (theoretical) issues there. I don't remember those details, I need to refresh my memory.
Quoted 6 lines
AFAIK, qemu injects #MC sometimes when it gets it from the kernel in form of a signal, if I recall this correctly, and it also reflects back #DB, when guest debug was enabled (and that is the reason for some work I did in this area, like the
KVM_GUESTDBG_BLOCKIRQthing)Qemu does this without considering nested and/or pending exception/etc. It just kind of abuses the
KVM_SET_VCPU_EVENTSfor that.
I wouldn't call that abuse, the ioctl() isn't just for migration. Not checking for a pending exception is firmly a userspace bug and not something KVM should try to fix.
For #DB, I suspect it's a non-issue. The exit is synchronous, so unless userspace is deferring the reflection, which would be architecturally wrong in and of itself, there can never be another pending exception.
For #MC, I think the correct behavior would be to defer the synthesized #MC if there's a pending exception and resume the guest until the exception is injected. E.g. if a different task encounters the real #MC, the synthesized #MC will be fully asynchronous and may be coincident with a pending exception that is unrelated to the #MC. That would require to userspace to enable KVM_CAP_EXCEPTION_PAYLOAD, otherwise userspace won't be able to differentiate between a pending and injected exception, e.g. if the #MC occurs during exception vectoring, userspace should override the injected exception and synthesize #MC, otherwise it would likely soft hang the guest.
Quoted 29 lines · Sean Christopherson
Quoted 4 lines · Maxim Levitsky
Other than that I am actually very happy that you posted this patch series, as this gives more chance that this long standing issue will be fixed,
and if your patches are better/simpler/less invasive to KVM and still address the issue, I fully support using them instead of mine.I highly doubt they're simpler or less invasive, but I do hope that the approach wil be easier to maintain.
Quoted 20 lines
Totally agree with you about your thoughts about splitting pending/injected exception, I also can't say I liked my approach that much, for the same reasons you mentioned.
It is also the main reason I put the whole thing on the backlog lately, because I was feeling that I am changing too much of the KVM, for a relatively theoretical issue.
I will review your patches, compare them to mine, and check if you or I missed something.
PS:
Back then, I also did an extensive review on few cases when qemu injects exceptions itself, which it does thankfully rarely. There are several (theoretical) issues there. I don't remember those details, I need to refresh my memory.
AFAIK, qemu injects #MC sometimes when it gets it from the kernel in form of a signal, if I recall this correctly, and it also reflects back #DB, when guest debug was enabled (and that is the reason for some work I did in this area, like the
KVM_GUESTDBG_BLOCKIRQthing)
Quoted 6 lines
Quoted 2 lines
Qemu does this without considering nested and/or pending exception/etc. It just kind of abuses the
KVM_SET_VCPU_EVENTSfor that.I wouldn't call that abuse, the
ioctl()isn't just for migration. Not checking for a pending exception is firmly a userspace bug and not something KVM should try to fix.
yes, but to make the right decision, the userspace has to know if there is a pending exception, and if there is, then merge it (which might even involve triple fault),
On top of that it is possible that pending excpetion is not intercepted by L1, but merged result is, so injecting the exception will cause nested VMexit, which is something that is hard for userspace to model.
I think that the cleanest way to do this is to add new ioctl, KVM_INJECT_EXCEPTION, which can do the right thing in the kernel, but I am not sure that it is worth it, knowing that thankfully userspace doesn't inject exceptions much.
Quoted 4 lines
For #DB, I suspect it's a non-issue. The exit is synchronous, so unless userspace is deferring the reflection, which would be architecturally wrong in and of itself, there can never be another pending exception.
Could very be, but still there could be corner cases. Like what if you set data fetch breakpoint on a IDT entry of some exception? I guess during delivery of that exception there might be #DB, but I am not 100% expert on when and how #DB is generated, so I can't be sure. Anyway #DB isn't a big deal because qemu only re-injects it when guest debug is enabled and that is broken anyway, and does worse things like leaking EFLAGS.TF to the guest stack and such.
Quoted 9 lines
For #MC, I think the correct behavior would be to defer the synthesized #MC if there's a pending exception and resume the guest until the exception is injected. E.g. if a different task encounters the real #MC, the synthesized #MC will be fully asynchronous and may be coincident with a pending exception that is unrelated to the #MC. That would require to userspace to enable
KVM_CAP_EXCEPTION_PAYLOAD, otherwise userspace won't be able to differentiate between a pending and injected exception, e.g. if the #MC occurs during exception vectoring, userspace should override the injected exception and synthesize #MC, otherwise it would likely soft hang the guest.
Something like that, I don't remember all the details.
Best regards,
Maxim Levitsky
Quoted 1 line
Sean Christopherson
maintainerQuoted 7 lines · Maxim Levitsky
Quoted 3 lines · Sean Christopherson
I wouldn't call that abuse, the
ioctl()isn't just for migration. Not checking for a pending exception is firmly a userspace bug and not something KVM should try to fix.yes, but to make the right decision, the userspace has to know if there is a pending exception, and if there is, then merge it (which might even involve triple fault),
There's no need for userspace to ever merge exceptions unless KVM supports either exiting to userspace on an exception that can occur during exception delivery, or userspace itself is emulating exception delivery. Outside of debug scenarios, #PF is likely the only exception that might ever be forwarded to userspace. But in those scenarios, userspace is almost always going to fix the #PF and resume the guest. If userspace doesn't fix the #PF, the guest is completely hosed because its IDT will trigger #PF, i.e. it's headed to shutdown regardless of KVM's ABI.
VM introspection is the only use case I can think of that might possibly want to emulate exception delivery in userspace, and VMI is a completely new set of APIs, in no small part because supporting something like this in KVM would require far more hooks than KVM provides.
Quoted 9 lines
On top of that it is possible that pending excpetion is not intercepted by L1, but merged result is, so injecting the exception will cause nested VMexit, which is something that is hard for userspace to model.
I think that the cleanest way to do this is to add new ioctl,
KVM_INJECT_EXCEPTION, which can do the right thing in the kernel, but I am not sure that it is worth it, knowing that thankfully userspace doesn't inject exceptions much.Quoted 1 line
Quoted 7 lines
Quoted 3 lines
For #DB, I suspect it's a non-issue. The exit is synchronous, so unless userspace is deferring the reflection, which would be architecturally wrong in and of itself, there can never be another pending exception.
Could very be, but still there could be corner cases. Like what if you set data fetch breakpoint on a IDT entry of some exception? I guess during delivery of that exception there might be #DB, but I am not 100% expert on when and how #DB is generated, so I can't be sure.
Data #DBs are trap-like. The #DB will arrive after exception delivery completes, i.e. will occur "on" the first instruction in the exception handler.