FAILED: patch "[PATCH] KVM: VMX: Preserve host's DEBUGCTLMSR_FREEZE_IN_SMM while" failed to apply to 6.16-stable tree
Subsystems (3)
[email protected]
maintainerThe patch below does not apply to the 6.16-stable tree.
If someone wants it applied there, or to any other stable or longterm tree, then please email the backport, including the original git commit id to <[email protected]>.
To reproduce the conflict and resubmit, you may use the following commands:
git fetch https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/ linux-6.16.y
git checkout FETCH_HEAD
git cherry-pick -x 6b1dd26544d045f6a79e8c73572c0c0db3ef3c1a
# <resolve conflicts, build, test, etc.>
git commit -s
git send-email --to '<[email protected]>' --in-reply-to '2025081231-vengeful-creasing-d789@gregkh' --subject-prefix 'PATCH 6.16.y' HEAD^..
Possible dependencies:
thanks,
greg k-h
------------------ original commit in Linus's tree ------------------
From 6b1dd26544d045f6a79e8c73572c0c0db3ef3c1a Mon Sep 17 00:00:00 2001
From: Maxim Levitsky <[email protected]>
Date: Tue, 10 Jun 2025 16:20:10 -0700
Subject: [PATCH] KVM: VMX: Preserve host's DEBUGCTLMSR_FREEZE_IN_SMM while
running the guest
Set/clear DEBUGCTLMSR_FREEZE_IN_SMM in GUEST_IA32_DEBUGCTL based on the host's pre-VM-Enter value, i.e. preserve the host's FREEZE_IN_SMM setting while running the guest. When running with the "default treatment of SMIs" in effect (the only mode KVM supports), SMIs do not generate a VM-Exit that is visible to host (non-SMM) software, and instead transitions directly from VMX non-root to SMM. And critically, DEBUGCTL isn't context switched by hardware on SMI or RSM, i.e. SMM will run with whatever value was resident in hardware at the time of the SMI.
Failure to preserve FREEZE_IN_SMM results in the PMU unexpectedly counting events while the CPU is executing in SMM, which can pollute profiling and potentially leak information into the guest.
Check for changes in FREEZE_IN_SMM prior to every entry into KVM's inner run loop, as the bit can be toggled in IRQ context via IPI callback (SMP function call), by way of /sys/devices/cpu/freeze_on_smi.
Add a field in kvm_x86_ops to communicate which DEBUGCTL bits need to be preserved, as FREEZE_IN_SMM is only supported and defined for Intel CPUs, i.e. explicitly checking FREEZE_IN_SMM in common x86 is at best weird, and at worst could lead to undesirable behavior in the future if AMD CPUs ever happened to pick up a collision with the bit.
Exempt TDX vCPUs, i.e. protected guests, from the check, as the TDX Module owns and controls GUEST_IA32_DEBUGCTL.
WARN in SVM if KVM_RUN_LOAD_DEBUGCTL is set, mostly to document that the lack of handling isn't a KVM bug (TDX already WARNs on any run_flag).
Lastly, explicitly reload GUEST_IA32_DEBUGCTL on a VM-Fail that is missed by KVM but detected by hardware, i.e. in nested_vmx_restore_host_state(). Doing so avoids the need to track host_debugctl on a per-VMCS basis, as GUEST_IA32_DEBUGCTL is unconditionally written by prepare_vmcs02() and load_vmcs12_host_state(). For the VM-Fail case, even though KVM won't have actually entered the guest, vcpu_enter_guest() will have run with vmcs02 active and thus could result in vmcs01 being run with a stale value.
arch/x86/include/asm/kvm_host.h +7 -0
@@ -1677,6 +1677,7 @@ static inline u16 kvm_lapic_irq_dest_mode(bool dest_mode_logical)enum kvm_x86_run_flags {KVM_RUN_FORCE_IMMEDIATE_EXIT =BIT (0 ),KVM_RUN_LOAD_GUEST_DR6 =BIT (1 ), +KVM_RUN_LOAD_DEBUGCTL =BIT (2 ), };struct kvm_x86_ops {
@@ -1707,6 +1708,12 @@ struct kvm_x86_ops {void (*vcpu_load)(struct kvm_vcpu *vcpu,int cpu);void (*vcpu_put)(struct kvm_vcpu *vcpu); +/* +* Mask of DEBUGCTL bits that are owned by the host, i.e. that need to +* match the host's value even while the guest is active. +*/ +const u64 HOST_OWNED_DEBUGCTL ; +void (*update_exception_bitmap)(struct kvm_vcpu *vcpu);int (*get_msr)(struct kvm_vcpu *vcpu,struct msr_data *msr);int (*set_msr)(struct kvm_vcpu *vcpu,struct msr_data *msr);
arch/x86/kvm/vmx/main.c +2 -0
@@ -915,6 +915,8 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .vcpu_load = vt_op(vcpu_load), .vcpu_put = vt_op(vcpu_put), + .HOST_OWNED_DEBUGCTL = DEBUGCTLMSR_FREEZE_IN_SMM, + .update_exception_bitmap = vt_op(update_exception_bitmap), .get_feature_msr = vmx_get_feature_msr, .get_msr = vt_op(get_msr),
arch/x86/kvm/vmx/nested.c +3 -0
@@ -4861,6 +4861,9 @@ static void nested_vmx_restore_host_state(struct kvm_vcpu *vcpu)WARN_ON (kvm_set_dr (vcpu,7 ,vmcs_readl (GUEST_DR7 ))); } +/* Reload DEBUGCTL to ensure vmcs01 has a fresh FREEZE_IN_SMM value. */ +vmx_reload_guest_debugctl (vcpu); +/* * Note that calling vmx_set_{efer,cr0,cr4} is important as they * handle a variety of side effects to KVM's software model.
arch/x86/kvm/vmx/vmx.c +3 -0
@@ -7377,6 +7377,9 @@ fastpath_t vmx_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags)if (run_flags &KVM_RUN_LOAD_GUEST_DR6 )set_debugreg (vcpu->arch.dr6,6 ); +if (run_flags &KVM_RUN_LOAD_DEBUGCTL ) +vmx_reload_guest_debugctl (vcpu); +/* * Refresh vmcs.HOST_CR3 if necessary. This must be done immediately * prior to VM-Enter, as the kernel may load a new ASID (PCID) any time
arch/x86/kvm/vmx/vmx.h +14 -1
@@ -419,12 +419,25 @@ bool vmx_is_valid_debugctl(struct kvm_vcpu *vcpu, u64 data, bool host_initiated)static inline void vmx_guest_debugctl_write (struct kvm_vcpu *vcpu,u64 val) { +WARN_ON_ONCE (val &DEBUGCTLMSR_FREEZE_IN_SMM ); + + val |= vcpu->arch.host_debugctl &DEBUGCTLMSR_FREEZE_IN_SMM ;vmcs_write64 (GUEST_IA32_DEBUGCTL , val); }static inline u64 vmx_guest_debugctl_read (void ) { -return vmcs_read64 (GUEST_IA32_DEBUGCTL ); +return vmcs_read64 (GUEST_IA32_DEBUGCTL ) & ~DEBUGCTLMSR_FREEZE_IN_SMM ; +} + +static inline void vmx_reload_guest_debugctl (struct kvm_vcpu *vcpu) +{ +u64 val =vmcs_read64 (GUEST_IA32_DEBUGCTL ); + +if (!((val ^ vcpu->arch.host_debugctl) &DEBUGCTLMSR_FREEZE_IN_SMM )) +return ; + +vmx_guest_debugctl_write (vcpu, val & ~DEBUGCTLMSR_FREEZE_IN_SMM ); }/*
arch/x86/kvm/x86.c +12 -2
@@ -10779,7 +10779,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)dm_request_for_irq_injection (vcpu) &&kvm_cpu_accept_dm_intr (vcpu);fastpath_t exit_fastpath; -u64 run_flags; +u64 run_flags, debug_ctl;bool req_immediate_exit = false;
@@ -11051,7 +11051,17 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)set_debugreg (0 ,7 ); } - vcpu->arch.host_debugctl =get_debugctlmsr (); +/* +* Refresh the host DEBUGCTL snapshot after disabling IRQs, as DEBUGCTL +* can be modified in IRQ context, e.g. via SMP function calls. Inform +* vendor code if any host-owned bits were changed, e.g. so that the +* value loaded into hardware while running the guest can be updated. +*/ + debug_ctl =get_debugctlmsr (); +if ((debug_ctl ^ vcpu->arch.host_debugctl) & kvm_x86_ops.HOST_OWNED_DEBUGCTL && + !vcpu->arch.guest_state_protected) + run_flags |=KVM_RUN_LOAD_DEBUGCTL ; + vcpu->arch.host_debugctl = debug_ctl;guest_timing_enter_irqoff ();
Sasha Levin
maintainerPATCH 1/6 KVM: x86: Convert vcpu_run()'s immediate exit param into a generic bitmap
From: Sean Christopherson <[email protected]>
[ Upstream commit 2478b1b220c49d25cb1c3f061ec4f9b351d9a131 ]
Convert kvm_x86_ops.vcpu_run()'s "force_immediate_exit" boolean parameter into an a generic bitmap so that similar "take action" information can be passed to vendor code without creating a pile of boolean parameters.
This will allow dropping kvm_x86_ops.set_dr6() in favor of a new flag, and will also allow for adding similar functionality for re-loading debugctl in the active VMCS.
Opportunistically massage the TDX WARN and comment to prepare for adding more run_flags, all of which are expected to be mutually exclusive with TDX, i.e. should be WARNed on.
No functional change intended.
Stable-dep-of: 6b1dd26544d0 ("KVM: VMX: Preserve host's DEBUGCTLMSR_FREEZE_IN_SMM while running the guest")
--- arch/x86/include/asm/kvm_host.h | 6 +++++- arch/x86/kvm/svm/svm.c | 4 ++-- arch/x86/kvm/vmx/main.c | 6 +++--- arch/x86/kvm/vmx/tdx.c | 18 +++++++++--------- arch/x86/kvm/vmx/vmx.c | 3 ++- arch/x86/kvm/vmx/x86_ops.h | 4 ++-- arch/x86/kvm/x86.c | 11 ++++++++--- 7 files changed, 31 insertions(+), 21 deletions(-)
arch/x86/include/asm/kvm_host.h +5 -1
@@ -1680,6 +1680,10 @@ static inline u16 kvm_lapic_irq_dest_mode(bool dest_mode_logical)return dest_mode_logical ?APIC_DEST_LOGICAL :APIC_DEST_PHYSICAL ; } +enum kvm_x86_run_flags { +KVM_RUN_FORCE_IMMEDIATE_EXIT =BIT (0 ), +}; +struct kvm_x86_ops {const char *name;
@@ -1761,7 +1765,7 @@ struct kvm_x86_ops {int (*vcpu_pre_run)(struct kvm_vcpu *vcpu);enum exit_fastpath_completion (*vcpu_run)(struct kvm_vcpu *vcpu, -bool force_immediate_exit); +u64 run_flags);int (*handle_exit)(struct kvm_vcpu *vcpu,enum exit_fastpath_completion exit_fastpath);int (*skip_emulated_instruction)(struct kvm_vcpu *vcpu);
arch/x86/kvm/svm/svm.c +2 -2
@@ -4389,9 +4389,9 @@ static noinstr void svm_vcpu_enter_exit(struct kvm_vcpu *vcpu, bool spec_ctrl_inguest_state_exit_irqoff (); } -static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu, -bool force_immediate_exit) +static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags) { +bool force_immediate_exit = run_flags &KVM_RUN_FORCE_IMMEDIATE_EXIT ;struct vcpu_svm *svm =to_svm (vcpu);bool spec_ctrl_intercepted =msr_write_intercepted (vcpu,MSR_IA32_SPEC_CTRL );
arch/x86/kvm/vmx/main.c +3 -3
@@ -175,12 +175,12 @@ static int vt_vcpu_pre_run(struct kvm_vcpu *vcpu)return vmx_vcpu_pre_run (vcpu); } -static fastpath_t vt_vcpu_run (struct kvm_vcpu *vcpu,bool force_immediate_exit) +static fastpath_t vt_vcpu_run (struct kvm_vcpu *vcpu,u64 run_flags) {if (is_td_vcpu (vcpu)) -return tdx_vcpu_run (vcpu, force_immediate_exit); +return tdx_vcpu_run (vcpu, run_flags); -return vmx_vcpu_run (vcpu, force_immediate_exit); +return vmx_vcpu_run (vcpu, run_flags); } static int vt_handle_exit(struct kvm_vcpu *vcpu,
arch/x86/kvm/vmx/tdx.c +9 -9
@@ -1025,20 +1025,20 @@ static void tdx_load_host_xsave_state(struct kvm_vcpu *vcpu) DEBUGCTLMSR_FREEZE_PERFMON_ON_PMI | \ DEBUGCTLMSR_FREEZE_IN_SMM) -fastpath_t tdx_vcpu_run (struct kvm_vcpu *vcpu,bool force_immediate_exit) +fastpath_t tdx_vcpu_run (struct kvm_vcpu *vcpu,u64 run_flags) {struct vcpu_tdx *tdx =to_tdx (vcpu);struct vcpu_vt *vt =to_vt (vcpu);/* -* force_immediate_exit requires vCPU entering for events injection with -* an immediately exit followed. But The TDX module doesn't guarantee -* entry, it's already possible for KVM to _think_ it completely entry -* to the guest without actually having done so. -* Since KVM never needs to force an immediate exit for TDX, and can't -* do direct injection, just warn on force_immediate_exit. +* WARN if KVM wants to force an immediate exit, as the TDX module does +* not guarantee entry into the guest, i.e. it's possible for KVM to +* _think_ it completed entry to the guest and forced an immediate exit +* without actually having done so. Luckily, KVM never needs to force +* an immediate exit for TDX (KVM can't do direct event injection, so +* just WARN and continue on. */ -WARN_ON_ONCE (force_immediate_exit); +WARN_ON_ONCE (run_flags);/* * Wait until retry of SEPT-zap-related SEAMCALL completes before
@@ -1048,7 +1048,7 @@ fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu, bool force_immediate_exit)if (unlikely (READ_ONCE (to_kvm_tdx (vcpu->kvm)->wait_for_sept_zap)))return EXIT_FASTPATH_EXIT_HANDLED ; -trace_kvm_entry (vcpu, force_immediate_exit); +trace_kvm_entry (vcpu, run_flags &KVM_RUN_FORCE_IMMEDIATE_EXIT );if (pi_test_on (&vt->pi_desc)) { apic->send_IPI_self (POSTED_INTR_VECTOR );
arch/x86/kvm/vmx/vmx.c +2 -1
@@ -7323,8 +7323,9 @@ static noinstr void vmx_vcpu_enter_exit(struct kvm_vcpu *vcpu,guest_state_exit_irqoff (); } -fastpath_t vmx_vcpu_run (struct kvm_vcpu *vcpu,bool force_immediate_exit) +fastpath_t vmx_vcpu_run (struct kvm_vcpu *vcpu,u64 run_flags) { +bool force_immediate_exit = run_flags &KVM_RUN_FORCE_IMMEDIATE_EXIT ;struct vcpu_vmx *vmx =to_vmx (vcpu);unsigned long cr3, cr4;
arch/x86/kvm/vmx/x86_ops.h +2 -2
@@ -21,7 +21,7 @@ void vmx_vm_destroy(struct kvm *kvm);int vmx_vcpu_precreate (struct kvm *kvm);int vmx_vcpu_create (struct kvm_vcpu *vcpu);int vmx_vcpu_pre_run (struct kvm_vcpu *vcpu); -fastpath_t vmx_vcpu_run (struct kvm_vcpu *vcpu,bool force_immediate_exit); +fastpath_t vmx_vcpu_run (struct kvm_vcpu *vcpu,u64 run_flags);void vmx_vcpu_free (struct kvm_vcpu *vcpu);void vmx_vcpu_reset (struct kvm_vcpu *vcpu,bool init_event);void vmx_vcpu_load (struct kvm_vcpu *vcpu,int cpu);
@@ -133,7 +133,7 @@ void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event);void tdx_vcpu_free (struct kvm_vcpu *vcpu);void tdx_vcpu_load (struct kvm_vcpu *vcpu,int cpu);int tdx_vcpu_pre_run (struct kvm_vcpu *vcpu); -fastpath_t tdx_vcpu_run (struct kvm_vcpu *vcpu,bool force_immediate_exit); +fastpath_t tdx_vcpu_run (struct kvm_vcpu *vcpu,u64 run_flags);void tdx_prepare_switch_to_guest (struct kvm_vcpu *vcpu);void tdx_vcpu_put (struct kvm_vcpu *vcpu);bool tdx_protected_apic_has_interrupt (struct kvm_vcpu *vcpu);
arch/x86/kvm/x86.c +8 -3
@@ -10785,6 +10785,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)dm_request_for_irq_injection (vcpu) &&kvm_cpu_accept_dm_intr (vcpu);fastpath_t exit_fastpath; +u64 run_flags;bool req_immediate_exit = false;
@@ -11029,8 +11030,11 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) goto cancel_injection; } -if (req_immediate_exit) + run_flags =0 ; +if (req_immediate_exit) { + run_flags |=KVM_RUN_FORCE_IMMEDIATE_EXIT ;kvm_make_request (KVM_REQ_EVENT , vcpu); + }fpregs_assert_state_consistent ();if (test_thread_flag (TIF_NEED_FPU_LOAD ))
@@ -11067,8 +11071,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)WARN_ON_ONCE ((kvm_vcpu_apicv_activated (vcpu) !=kvm_vcpu_apicv_active (vcpu)) && (kvm_get_apic_mode (vcpu) !=LAPIC_MODE_DISABLED )); - exit_fastpath =kvm_x86_call (vcpu_run)(vcpu, - req_immediate_exit); + exit_fastpath =kvm_x86_call (vcpu_run)(vcpu, run_flags);if (likely (exit_fastpath !=EXIT_FASTPATH_REENTER_GUEST ))break ;
@@ -11080,6 +11083,8 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)break ; } + run_flags =0 ; +/* Note, VM-Exits that go down the "slow" path are accounted below. */ ++vcpu->stat.exits; }
Signature · 1 line
2.39.5
Sasha Levin
maintainerPATCH 2/6 KVM: x86: Drop kvm_x86_ops.set_dr6() in favor of a new KVM_RUN flag
From: Sean Christopherson <[email protected]>
[ Upstream commit 80c64c7afea1da6a93ebe88d3d29d8a60377ef80 ]
Instruct vendor code to load the guest's DR6 into hardware via a new KVM_RUN flag, and remove kvm_x86_ops.set_dr6(), whose sole purpose was to load vcpu->arch.dr6 into hardware when DR6 can be read/written directly by the guest.
Note, TDX already WARNs on any run_flag being set, i.e. will yell if KVM thinks DR6 needs to be reloaded. TDX vCPUs force KVM_DEBUGREG_AUTO_SWITCH and never clear the flag, i.e. should never observe KVM_RUN_LOAD_GUEST_DR6.
Stable-dep-of: 6b1dd26544d0 ("KVM: VMX: Preserve host's DEBUGCTLMSR_FREEZE_IN_SMM while running the guest")
--- arch/x86/include/asm/kvm-x86-ops.h | 1 - arch/x86/include/asm/kvm_host.h | 2 +- arch/x86/kvm/svm/svm.c | 10 ++++++---- arch/x86/kvm/vmx/main.c | 9 --------- arch/x86/kvm/vmx/vmx.c | 9 +++------ arch/x86/kvm/x86.c | 2 +- 6 files changed, 11 insertions(+), 22 deletions(-)
arch/x86/include/asm/kvm-x86-ops.h +0 -1
@@ -49,7 +49,6 @@ KVM_X86_OP(set_idt) KVM_X86_OP(get_gdt) KVM_X86_OP(set_gdt) KVM_X86_OP(sync_dirty_debug_regs) -KVM_X86_OP(set_dr6) KVM_X86_OP(set_dr7) KVM_X86_OP(cache_reg) KVM_X86_OP(get_rflags)
arch/x86/include/asm/kvm_host.h +1 -1
@@ -1682,6 +1682,7 @@ static inline u16 kvm_lapic_irq_dest_mode(bool dest_mode_logical)enum kvm_x86_run_flags {KVM_RUN_FORCE_IMMEDIATE_EXIT =BIT (0 ), +KVM_RUN_LOAD_GUEST_DR6 =BIT (1 ), };struct kvm_x86_ops {
@@ -1734,7 +1735,6 @@ struct kvm_x86_ops {void (*get_gdt)(struct kvm_vcpu *vcpu,struct desc_ptr *dt);void (*set_gdt)(struct kvm_vcpu *vcpu,struct desc_ptr *dt);void (*sync_dirty_debug_regs)(struct kvm_vcpu *vcpu); -void (*set_dr6)(struct kvm_vcpu *vcpu,unsigned long value);void (*set_dr7)(struct kvm_vcpu *vcpu,unsigned long value);void (*cache_reg)(struct kvm_vcpu *vcpu,enum kvm_reg reg);unsigned long (*get_rflags)(struct kvm_vcpu *vcpu);
arch/x86/kvm/svm/svm.c +6 -4
@@ -4438,10 +4438,13 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags)svm_hv_update_vp_id (svm->vmcb, vcpu);/* -* Run with all-zero DR6 unless needed, so that we can get the exact cause -* of a #DB. +* Run with all-zero DR6 unless the guest can write DR6 freely, so that +* KVM can get the exact cause of a #DB. Note, loading guest DR6 from +* KVM's snapshot is only necessary when DR accesses won't exit. */ -if (likely (!(vcpu->arch.switch_db_regs &KVM_DEBUGREG_WONT_EXIT ))) +if (unlikely (run_flags &KVM_RUN_LOAD_GUEST_DR6 )) +svm_set_dr6 (vcpu, vcpu->arch.dr6); +else if (likely (!(vcpu->arch.switch_db_regs &KVM_DEBUGREG_WONT_EXIT )))svm_set_dr6 (vcpu,DR6_ACTIVE_LOW );clgi ();
@@ -5252,7 +5255,6 @@ static struct kvm_x86_ops svm_x86_ops __initdata = { .set_idt = svm_set_idt, .get_gdt = svm_get_gdt, .set_gdt = svm_set_gdt, - .set_dr6 = svm_set_dr6, .set_dr7 = svm_set_dr7, .sync_dirty_debug_regs = svm_sync_dirty_debug_regs, .cache_reg = svm_cache_reg,
arch/x86/kvm/vmx/main.c +0 -9
@@ -489,14 +489,6 @@ static void vt_set_gdt(struct kvm_vcpu *vcpu, struct desc_ptr *dt)vmx_set_gdt (vcpu, dt); } -static void vt_set_dr6 (struct kvm_vcpu *vcpu,unsigned long val) -{ -if (is_td_vcpu (vcpu)) -return ; - -vmx_set_dr6 (vcpu, val); -} -static void vt_set_dr7 (struct kvm_vcpu *vcpu,unsigned long val) {if (is_td_vcpu (vcpu))
@@ -943,7 +935,6 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .set_idt = vt_op(set_idt), .get_gdt = vt_op(get_gdt), .set_gdt = vt_op(set_gdt), - .set_dr6 = vt_op(set_dr6), .set_dr7 = vt_op(set_dr7), .sync_dirty_debug_regs = vt_op(sync_dirty_debug_regs), .cache_reg = vt_op(cache_reg),
arch/x86/kvm/vmx/vmx.c +3 -6
@@ -5606,12 +5606,6 @@ void vmx_sync_dirty_debug_regs(struct kvm_vcpu *vcpu)set_debugreg (DR6_RESERVED ,6 ); } -void vmx_set_dr6 (struct kvm_vcpu *vcpu,unsigned long val) -{ -lockdep_assert_irqs_disabled (); -set_debugreg (vcpu->arch.dr6,6 ); -} -void vmx_set_dr7 (struct kvm_vcpu *vcpu,unsigned long val) {vmcs_writel (GUEST_DR7 , val);
@@ -7370,6 +7364,9 @@ fastpath_t vmx_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags)vmcs_writel (GUEST_RIP , vcpu->arch.regs[VCPU_REGS_RIP ]); vcpu->arch.regs_dirty =0 ; +if (run_flags &KVM_RUN_LOAD_GUEST_DR6 ) +set_debugreg (vcpu->arch.dr6,6 ); +/* * Refresh vmcs.HOST_CR3 if necessary. This must be done immediately * prior to VM-Enter, as the kernel may load a new ASID (PCID) any time
arch/x86/kvm/x86.c +1 -1
@@ -11052,7 +11052,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)set_debugreg (vcpu->arch.eff_db[3 ],3 );/* When KVM_DEBUGREG_WONT_EXIT, dr6 is accessible in guest. */ if (unlikely (vcpu->arch.switch_db_regs &KVM_DEBUGREG_WONT_EXIT )) -kvm_x86_call (set_dr6)(vcpu, vcpu->arch.dr6); + run_flags |=KVM_RUN_LOAD_GUEST_DR6 ; }else if (unlikely (hw_breakpoint_active ())) {set_debugreg (DR7_FIXED_1 ,7 ); }
Signature · 1 line
2.39.5
Re: [PATCH 6.16.y 2/6] KVM: x86: Drop kvm_x86_ops.set_dr6() in favor of a new KVM_RUN flag
Quoted 13 lines · Sasha Levin
From: Sean Christopherson <[email protected]>
[ Upstream commit
80c64c7afea1da6a93ebe88d3d29d8a60377ef80]Instruct vendor code to load the guest's DR6 into hardware via a new
KVM_RUNflag, and removekvm_x86_ops.set_dr6(), whose sole purpose was to loadvcpu->arch.dr6into hardware when DR6 can be read/written directly by the guest.Note, TDX already WARNs on any
run_flagbeing set, i.e. will yell if KVM thinks DR6 needs to be reloaded. TDX vCPUs forceKVM_DEBUGREG_AUTO_SWITCHand never clear the flag, i.e. should never observeKVM_RUN_LOAD_GUEST_DR6.
Quoted 6 lines
Signed-off-by Sean Christopherson <[email protected]>Stable-dep-of:
6b1dd26544d0("KVM: VMX: Preserve host'sDEBUGCTLMSR_FREEZE_IN_SMMwhile running the guest")Signed-off-by Sasha Levin <[email protected]>---
Sasha Levin
maintainerPATCH 3/6 KVM: VMX: Extract checking of guest's DEBUGCTL into helper
In reply to Sasha LevinFrom: Sean Christopherson <[email protected]>
[ Upstream commit 8a4351ac302cd8c19729ba2636acfd0467c22ae8 ]
Move VMX's logic to check DEBUGCTL values into a standalone helper so that the code can be used by nested VM-Enter to apply the same logic to the value being loaded from vmcs12.
KVM needs to explicitly check vmcs12->guest_ia32_debugctl on nested VM-Enter, as hardware may support features that KVM does not, i.e. relying on hardware to detect invalid guest state will result in false negatives. Unfortunately, that means applying KVM's funky suppression of BTF and LBR to vmcs12 so as not to break existing guests.
No functional change intended.
Stable-dep-of: 6b1dd26544d0 ("KVM: VMX: Preserve host's DEBUGCTLMSR_FREEZE_IN_SMM while running the guest")
--- arch/x86/kvm/vmx/vmx.c | 29 +++++++++++++++++------------ 1 file changed, 17 insertions(+), 12 deletions(-)
arch/x86/kvm/vmx/vmx.c +17 -12
@@ -2189,6 +2189,19 @@ static u64 vmx_get_supported_debugctl(struct kvm_vcpu *vcpu, bool host_initiatedreturn debugctl; } +static bool vmx_is_valid_debugctl (struct kvm_vcpu *vcpu,u64 data, +bool host_initiated) +{ +u64 invalid; + + invalid = data & ~vmx_get_supported_debugctl (vcpu, host_initiated); +if (invalid & (DEBUGCTLMSR_BTF |DEBUGCTLMSR_LBR )) { +kvm_pr_unimpl_wrmsr (vcpu,MSR_IA32_DEBUGCTLMSR , data); + invalid &= ~(DEBUGCTLMSR_BTF |DEBUGCTLMSR_LBR ); + } +return !invalid; +} +/* * Writes msr value into the appropriate "register". * Returns 0 on success, non-0 otherwise.
@@ -2257,19 +2270,12 @@ int vmx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) }vmcs_writel (GUEST_SYSENTER_ESP , data);break ; -case MSR_IA32_DEBUGCTLMSR : { -u64 invalid; - - invalid = data & ~vmx_get_supported_debugctl (vcpu, msr_info->host_initiated); -if (invalid & (DEBUGCTLMSR_BTF |DEBUGCTLMSR_LBR )) { -kvm_pr_unimpl_wrmsr (vcpu, msr_index, data); - data &= ~(DEBUGCTLMSR_BTF |DEBUGCTLMSR_LBR ); - invalid &= ~(DEBUGCTLMSR_BTF |DEBUGCTLMSR_LBR ); - } - -if (invalid) +case MSR_IA32_DEBUGCTLMSR : +if (!vmx_is_valid_debugctl (vcpu, data, msr_info->host_initiated))return 1 ; + data &=vmx_get_supported_debugctl (vcpu, msr_info->host_initiated); +if (is_guest_mode (vcpu) &&get_vmcs12 (vcpu)->vm_exit_controls &VM_EXIT_SAVE_DEBUG_CONTROLS )get_vmcs12 (vcpu)->guest_ia32_debugctl = data;
@@ -2279,7 +2285,6 @@ int vmx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) (data & DEBUGCTLMSR_LBR))intel_pmu_create_guest_lbr_event (vcpu);return 0 ; - }case MSR_IA32_BNDCFGS : if (!kvm_mpx_supported() || (!msr_info->host_initiated &&
Signature · 1 line
2.39.5
Re: [PATCH 6.16.y 3/6] KVM: VMX: Extract checking of guest's DEBUGCTL into helper
Quoted 14 lines · Sasha Levin
From: Sean Christopherson <[email protected]>
[ Upstream commit
8a4351ac302cd8c19729ba2636acfd0467c22ae8]Move VMX's logic to check DEBUGCTL values into a standalone helper so that the code can be used by nested VM-Enter to apply the same logic to the value being loaded from vmcs12.
KVM needs to explicitly check
vmcs12->guest_ia32_debugctlon nested VM-Enter, as hardware may support features that KVM does not, i.e. relying on hardware to detect invalid guest state will result in false negatives. Unfortunately, that means applying KVM's funky suppression of BTF and LBR to vmcs12 so as not to break existing guests.
Quoted 8 lines
No functional change intended.
Reviewed-by Dapeng Mi <[email protected]>Signed-off-by Sean Christopherson <[email protected]>Stable-dep-of:
6b1dd26544d0("KVM: VMX: Preserve host'sDEBUGCTLMSR_FREEZE_IN_SMMwhile running the guest")Signed-off-by Sasha Levin <[email protected]>---
Sasha Levin
maintainerPATCH 4/6 KVM: nVMX: Check vmcs12->guest_ia32_debugctl on nested VM-Enter
In reply to Sasha LevinFrom: Maxim Levitsky <[email protected]>
[ Upstream commit 095686e6fcb4150f0a55b1a25987fad3d8af58d6 ]
Add a consistency check for L2's guest_ia32_debugctl, as KVM only supports a subset of hardware functionality, i.e. KVM can't rely on hardware to detect illegal/unsupported values. Failure to check the vmcs12 value would allow the guest to load any harware-supported value while running L2.
Take care to exempt BTF and LBR from the validity check in order to match KVM's behavior for writes via WRMSR, but without clobbering vmcs12. Even if VM_EXIT_SAVE_DEBUG_CONTROLS is set in vmcs12, L1 can reasonably expect that vmcs12->guest_ia32_debugctl will not be modified if writes to the MSR are being intercepted.
Arguably, KVM _should_ update vmcs12 if VM_EXIT_SAVE_DEBUG_CONTROLS is set *and* writes to MSR_IA32_DEBUGCTLMSR are not being intercepted by L1, but that would incur non-trivial complexity and wouldn't change the fact that KVM's handling of DEBUGCTL is blatantly broken. I.e. the extra complexity is not worth carrying.
Stable-dep-of: 6b1dd26544d0 ("KVM: VMX: Preserve host's DEBUGCTLMSR_FREEZE_IN_SMM while running the guest")
--- arch/x86/kvm/vmx/nested.c | 12 ++++++++++-- arch/x86/kvm/vmx/vmx.c | 5 ++--- arch/x86/kvm/vmx/vmx.h | 3 +++ 3 files changed, 15 insertions(+), 5 deletions(-)
arch/x86/kvm/vmx/nested.c +10 -2
@@ -2663,7 +2663,8 @@ static int prepare_vmcs02(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12,if (vmx->nested.nested_run_pending && (vmcs12->vm_entry_controls &VM_ENTRY_LOAD_DEBUG_CONTROLS )) {kvm_set_dr (vcpu,7 , vmcs12->guest_dr7); -vmcs_write64 (GUEST_IA32_DEBUGCTL , vmcs12->guest_ia32_debugctl); +vmcs_write64 (GUEST_IA32_DEBUGCTL , vmcs12->guest_ia32_debugctl & +vmx_get_supported_debugctl (vcpu, false)); }else {kvm_set_dr (vcpu,7 , vcpu->arch.dr7);vmcs_write64 (GUEST_IA32_DEBUGCTL , vmx->nested.pre_vmenter_debugctl);
@@ -3156,7 +3157,8 @@ static int nested_vmx_check_guest_state(struct kvm_vcpu *vcpu,return -EINVAL ;if ((vmcs12->vm_entry_controls &VM_ENTRY_LOAD_DEBUG_CONTROLS ) && -CC (!kvm_dr7_valid (vmcs12->guest_dr7))) + (CC (!kvm_dr7_valid (vmcs12->guest_dr7)) || +CC (!vmx_is_valid_debugctl (vcpu, vmcs12->guest_ia32_debugctl, false))))return -EINVAL ; if ((vmcs12->vm_entry_controls & VM_ENTRY_LOAD_IA32_PAT) &&
@@ -4608,6 +4610,12 @@ static void sync_vmcs02_to_vmcs12(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12) (vmcs12->vm_entry_controls & ~VM_ENTRY_IA32E_MODE ) | (vm_entry_controls_get (to_vmx (vcpu)) &VM_ENTRY_IA32E_MODE ); +/* +* Note! Save DR7, but intentionally don't grab DEBUGCTL from vmcs02. +* Writes to DEBUGCTL that aren't intercepted by L1 are immediately +* propagated to vmcs12 (see vmx_set_msr()), as the value loaded into +* vmcs02 doesn't strictly track vmcs12. +*/ if (vmcs12->vm_exit_controls &VM_EXIT_SAVE_DEBUG_CONTROLS ) vmcs12->guest_dr7 = vcpu->arch.dr7;
arch/x86/kvm/vmx/vmx.c +2 -3
@@ -2174,7 +2174,7 @@ static u64 nested_vmx_truncate_sysenter_addr(struct kvm_vcpu *vcpu,return (unsigned long )data; } -static u64 vmx_get_supported_debugctl (struct kvm_vcpu *vcpu,bool host_initiated) +u64 vmx_get_supported_debugctl (struct kvm_vcpu *vcpu,bool host_initiated) {u64 debugctl =0 ;
@@ -2189,8 +2189,7 @@ static u64 vmx_get_supported_debugctl(struct kvm_vcpu *vcpu, bool host_initiatedreturn debugctl; } -static bool vmx_is_valid_debugctl (struct kvm_vcpu *vcpu,u64 data, -bool host_initiated) +bool vmx_is_valid_debugctl (struct kvm_vcpu *vcpu,u64 data,bool host_initiated) {u64 invalid;
arch/x86/kvm/vmx/vmx.h +3 -0
@@ -414,6 +414,9 @@ static inline void vmx_set_intercept_for_msr(struct kvm_vcpu *vcpu, u32 msr,void vmx_update_cpu_dirty_logging (struct kvm_vcpu *vcpu); +u64 vmx_get_supported_debugctl (struct kvm_vcpu *vcpu,bool host_initiated); +bool vmx_is_valid_debugctl (struct kvm_vcpu *vcpu,u64 data,bool host_initiated); +/* * Note, early Intel manuals have the write-low and read-high bitmap offsets * the wrong way round. The bitmaps control MSRs 0x00000000-0x00001fff and
Signature · 1 line
2.39.5
Re: [PATCH 6.16.y 4/6] KVM: nVMX: Check vmcs12->guest_ia32_debugctl on nested VM-Enter
Quoted 21 lines · Sasha Levin
From: Maxim Levitsky <[email protected]>
[ Upstream commit
095686e6fcb4150f0a55b1a25987fad3d8af58d6]Add a consistency check for L2's
guest_ia32_debugctl, as KVM only supports a subset of hardware functionality, i.e. KVM can't rely on hardware to detect illegal/unsupported values. Failure to check the vmcs12 value would allow the guest to load any harware-supported value while running L2.Take care to exempt BTF and LBR from the validity check in order to match KVM's behavior for writes via WRMSR, but without clobbering vmcs12. Even if
VM_EXIT_SAVE_DEBUG_CONTROLSis set in vmcs12, L1 can reasonably expect thatvmcs12->guest_ia32_debugctlwill not be modified if writes to the MSR are being intercepted.Arguably, KVM _should_ update vmcs12 if
VM_EXIT_SAVE_DEBUG_CONTROLSis set *and* writes toMSR_IA32_DEBUGCTLMSRare not being intercepted by L1, but that would incur non-trivial complexity and wouldn't change the fact that KVM's handling of DEBUGCTL is blatantly broken. I.e. the extra complexity is not worth carrying.
Quoted 8 lines
Signed-off-by Maxim Levitsky <[email protected]>Co-developed-by Sean Christopherson <[email protected]>Signed-off-by Sean Christopherson <[email protected]>Stable-dep-of:
6b1dd26544d0("KVM: VMX: Preserve host'sDEBUGCTLMSR_FREEZE_IN_SMMwhile running the guest")Signed-off-by Sasha Levin <[email protected]>---
Sasha Levin
maintainerPATCH 5/6 KVM: VMX: Wrap all accesses to IA32_DEBUGCTL with getter/setter APIs
In reply to Sasha LevinFrom: Maxim Levitsky <[email protected]>
[ Upstream commit 7d0cce6cbe71af6e9c1831bff101a2b9c249c4a2 ]
Introduce vmx_guest_debugctl_{read,write}() to handle all accesses to vmcs.GUEST_IA32_DEBUGCTL. This will allow stuffing FREEZE_IN_SMM into GUEST_IA32_DEBUGCTL based on the host setting without bleeding the state into the guest, and without needing to copy+paste the FREEZE_IN_SMM logic into every patch that accesses GUEST_IA32_DEBUGCTL.
No functional change intended.
Stable-dep-of: 6b1dd26544d0 ("KVM: VMX: Preserve host's DEBUGCTLMSR_FREEZE_IN_SMM while running the guest")
--- arch/x86/kvm/vmx/nested.c | 10 +++++----- arch/x86/kvm/vmx/pmu_intel.c | 8 ++++---- arch/x86/kvm/vmx/vmx.c | 8 +++++--- arch/x86/kvm/vmx/vmx.h | 10 ++++++++++ 4 files changed, 24 insertions(+), 12 deletions(-)
arch/x86/kvm/vmx/nested.c +5 -5
@@ -2663,11 +2663,11 @@ static int prepare_vmcs02(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12,if (vmx->nested.nested_run_pending && (vmcs12->vm_entry_controls &VM_ENTRY_LOAD_DEBUG_CONTROLS )) {kvm_set_dr (vcpu,7 , vmcs12->guest_dr7); -vmcs_write64 (GUEST_IA32_DEBUGCTL , vmcs12->guest_ia32_debugctl & -vmx_get_supported_debugctl (vcpu, false)); +vmx_guest_debugctl_write (vcpu, vmcs12->guest_ia32_debugctl & +vmx_get_supported_debugctl (vcpu, false)); }else {kvm_set_dr (vcpu,7 , vcpu->arch.dr7); -vmcs_write64 (GUEST_IA32_DEBUGCTL , vmx->nested.pre_vmenter_debugctl); +vmx_guest_debugctl_write (vcpu, vmx->nested.pre_vmenter_debugctl); }if (kvm_mpx_supported () && (!vmx->nested.nested_run_pending || !(vmcs12->vm_entry_controls &VM_ENTRY_LOAD_BNDCFGS )))
@@ -3532,7 +3532,7 @@ enum nvmx_vmentry_status nested_vmx_enter_non_root_mode(struct kvm_vcpu *vcpu,if (!vmx->nested.nested_run_pending || !(vmcs12->vm_entry_controls &VM_ENTRY_LOAD_DEBUG_CONTROLS )) - vmx->nested.pre_vmenter_debugctl =vmcs_read64 (GUEST_IA32_DEBUGCTL ); + vmx->nested.pre_vmenter_debugctl =vmx_guest_debugctl_read ();if (kvm_mpx_supported () && (!vmx->nested.nested_run_pending || !(vmcs12->vm_entry_controls &VM_ENTRY_LOAD_BNDCFGS )))
@@ -4806,7 +4806,7 @@ static void load_vmcs12_host_state(struct kvm_vcpu *vcpu,__vmx_set_segment (vcpu, &seg,VCPU_SREG_LDTR );kvm_set_dr (vcpu,7 ,0x400 ); -vmcs_write64 (GUEST_IA32_DEBUGCTL ,0 ); +vmx_guest_debugctl_write (vcpu,0 );if (nested_vmx_load_msr (vcpu, vmcs12->vm_exit_msr_load_addr, vmcs12->vm_exit_msr_load_count))
arch/x86/kvm/vmx/pmu_intel.c +4 -4
@@ -653,11 +653,11 @@ static void intel_pmu_reset(struct kvm_vcpu *vcpu)*/ static void intel_pmu_legacy_freezing_lbrs_on_pmi (struct kvm_vcpu *vcpu) { -u64 data =vmcs_read64 (GUEST_IA32_DEBUGCTL ); +u64 data =vmx_guest_debugctl_read ();if (data &DEBUGCTLMSR_FREEZE_LBRS_ON_PMI ) { data &= ~DEBUGCTLMSR_LBR ; -vmcs_write64 (GUEST_IA32_DEBUGCTL , data); +vmx_guest_debugctl_write (vcpu, data); } }
@@ -730,7 +730,7 @@ void vmx_passthrough_lbr_msrs(struct kvm_vcpu *vcpu)if (!lbr_desc->event) {vmx_disable_lbr_msrs_passthrough (vcpu); -if (vmcs_read64 (GUEST_IA32_DEBUGCTL ) &DEBUGCTLMSR_LBR ) +if (vmx_guest_debugctl_read () &DEBUGCTLMSR_LBR ) goto warn;if (test_bit (INTEL_PMC_IDX_FIXED_VLBR , pmu->pmc_in_use)) goto warn;
@@ -752,7 +752,7 @@ void vmx_passthrough_lbr_msrs(struct kvm_vcpu *vcpu)static void intel_pmu_cleanup (struct kvm_vcpu *vcpu) { -if (!(vmcs_read64 (GUEST_IA32_DEBUGCTL ) &DEBUGCTLMSR_LBR )) +if (!(vmx_guest_debugctl_read () &DEBUGCTLMSR_LBR ))intel_pmu_release_guest_lbr_event (vcpu); }
arch/x86/kvm/vmx/vmx.c +5 -3
@@ -2149,7 +2149,7 @@ int vmx_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) msr_info->data = vmx->pt_desc.guest.addr_a[index /2 ];break ;case MSR_IA32_DEBUGCTLMSR : - msr_info->data =vmcs_read64 (GUEST_IA32_DEBUGCTL ); + msr_info->data =vmx_guest_debugctl_read ();break ;default : find_uret_msr:
@@ -2279,7 +2279,8 @@ int vmx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) VM_EXIT_SAVE_DEBUG_CONTROLS) get_vmcs12(vcpu)->guest_ia32_debugctl = data; - vmcs_write64(GUEST_IA32_DEBUGCTL, data); + vmx_guest_debugctl_write(vcpu, data); + if (intel_pmu_lbr_is_enabled(vcpu) && !to_vmx(vcpu)->lbr_desc.event && (data & DEBUGCTLMSR_LBR)) intel_pmu_create_guest_lbr_event(vcpu);
@@ -4794,7 +4795,8 @@ static void init_vmcs(struct vcpu_vmx *vmx)vmcs_write32 (GUEST_SYSENTER_CS ,0 );vmcs_writel (GUEST_SYSENTER_ESP ,0 );vmcs_writel (GUEST_SYSENTER_EIP ,0 ); -vmcs_write64 (GUEST_IA32_DEBUGCTL ,0 ); + +vmx_guest_debugctl_write (&vmx->vcpu,0 );if (cpu_has_vmx_tpr_shadow ()) {vmcs_write64 (VIRTUAL_APIC_PAGE_ADDR ,0 );
arch/x86/kvm/vmx/vmx.h +10 -0
@@ -417,6 +417,16 @@ void vmx_update_cpu_dirty_logging(struct kvm_vcpu *vcpu);u64 vmx_get_supported_debugctl (struct kvm_vcpu *vcpu,bool host_initiated);bool vmx_is_valid_debugctl (struct kvm_vcpu *vcpu,u64 data,bool host_initiated); +static inline void vmx_guest_debugctl_write (struct kvm_vcpu *vcpu,u64 val) +{ +vmcs_write64 (GUEST_IA32_DEBUGCTL , val); +} + +static inline u64 vmx_guest_debugctl_read (void ) +{ +return vmcs_read64 (GUEST_IA32_DEBUGCTL ); +} +/* * Note, early Intel manuals have the write-low and read-high bitmap offsets * the wrong way round. The bitmaps control MSRs 0x00000000-0x00001fff and
Signature · 1 line
2.39.5
Re: [PATCH 6.16.y 5/6] KVM: VMX: Wrap all accesses to IA32_DEBUGCTL with getter/setter APIs
Quoted 18 lines · Sasha Levin
From: Maxim Levitsky <[email protected]>
[ Upstream commit
7d0cce6cbe71af6e9c1831bff101a2b9c249c4a2]Introduce vmx_guest_debugctl_{read,write}() to handle all accesses to vmcs.GUEST_IA32_DEBUGCTL. This will allow stuffing
FREEZE_IN_SMMintoGUEST_IA32_DEBUGCTLbased on the host setting without bleeding the state into the guest, and without needing to copy+paste theFREEZE_IN_SMMlogic into every patch that accessesGUEST_IA32_DEBUGCTL.No functional change intended.
Signed-off-by Maxim Levitsky <[email protected]>[sean: massage changelog, make inline, use in all prepare_vmcs02() cases]Reviewed-by Dapeng Mi <[email protected]>Signed-off-by Sean Christopherson <[email protected]>
Quoted 3 lines
Stable-dep-of:
6b1dd26544d0("KVM: VMX: Preserve host'sDEBUGCTLMSR_FREEZE_IN_SMMwhile running the guest")Signed-off-by Sasha Levin <[email protected]>---
Sasha Levin
maintainerPATCH 6/6 KVM: VMX: Preserve host's DEBUGCTLMSR_FREEZE_IN_SMM while running the guest
In reply to Sasha LevinFrom: Maxim Levitsky <[email protected]>
[ Upstream commit 6b1dd26544d045f6a79e8c73572c0c0db3ef3c1a ]
Set/clear DEBUGCTLMSR_FREEZE_IN_SMM in GUEST_IA32_DEBUGCTL based on the host's pre-VM-Enter value, i.e. preserve the host's FREEZE_IN_SMM setting while running the guest. When running with the "default treatment of SMIs" in effect (the only mode KVM supports), SMIs do not generate a VM-Exit that is visible to host (non-SMM) software, and instead transitions directly from VMX non-root to SMM. And critically, DEBUGCTL isn't context switched by hardware on SMI or RSM, i.e. SMM will run with whatever value was resident in hardware at the time of the SMI.
Failure to preserve FREEZE_IN_SMM results in the PMU unexpectedly counting events while the CPU is executing in SMM, which can pollute profiling and potentially leak information into the guest.
Check for changes in FREEZE_IN_SMM prior to every entry into KVM's inner run loop, as the bit can be toggled in IRQ context via IPI callback (SMP function call), by way of /sys/devices/cpu/freeze_on_smi.
Add a field in kvm_x86_ops to communicate which DEBUGCTL bits need to be preserved, as FREEZE_IN_SMM is only supported and defined for Intel CPUs, i.e. explicitly checking FREEZE_IN_SMM in common x86 is at best weird, and at worst could lead to undesirable behavior in the future if AMD CPUs ever happened to pick up a collision with the bit.
Exempt TDX vCPUs, i.e. protected guests, from the check, as the TDX Module owns and controls GUEST_IA32_DEBUGCTL.
WARN in SVM if KVM_RUN_LOAD_DEBUGCTL is set, mostly to document that the lack of handling isn't a KVM bug (TDX already WARNs on any run_flag).
Lastly, explicitly reload GUEST_IA32_DEBUGCTL on a VM-Fail that is missed by KVM but detected by hardware, i.e. in nested_vmx_restore_host_state(). Doing so avoids the need to track host_debugctl on a per-VMCS basis, as GUEST_IA32_DEBUGCTL is unconditionally written by prepare_vmcs02() and load_vmcs12_host_state(). For the VM-Fail case, even though KVM won't have actually entered the guest, vcpu_enter_guest() will have run with vmcs02 active and thus could result in vmcs01 being run with a stale value.
--- arch/x86/include/asm/kvm_host.h | 7 +++++++ arch/x86/kvm/vmx/main.c | 2 ++ arch/x86/kvm/vmx/nested.c | 3 +++ arch/x86/kvm/vmx/vmx.c | 3 +++ arch/x86/kvm/vmx/vmx.h | 15 ++++++++++++++- arch/x86/kvm/x86.c | 14 ++++++++++++-- 6 files changed, 41 insertions(+), 3 deletions(-)
arch/x86/include/asm/kvm_host.h +7 -0
@@ -1683,6 +1683,7 @@ static inline u16 kvm_lapic_irq_dest_mode(bool dest_mode_logical)enum kvm_x86_run_flags {KVM_RUN_FORCE_IMMEDIATE_EXIT =BIT (0 ),KVM_RUN_LOAD_GUEST_DR6 =BIT (1 ), +KVM_RUN_LOAD_DEBUGCTL =BIT (2 ), };struct kvm_x86_ops {
@@ -1713,6 +1714,12 @@ struct kvm_x86_ops {void (*vcpu_load)(struct kvm_vcpu *vcpu,int cpu);void (*vcpu_put)(struct kvm_vcpu *vcpu); +/* +* Mask of DEBUGCTL bits that are owned by the host, i.e. that need to +* match the host's value even while the guest is active. +*/ +const u64 HOST_OWNED_DEBUGCTL ; +void (*update_exception_bitmap)(struct kvm_vcpu *vcpu);int (*get_msr)(struct kvm_vcpu *vcpu,struct msr_data *msr);int (*set_msr)(struct kvm_vcpu *vcpu,struct msr_data *msr);
arch/x86/kvm/vmx/main.c +2 -0
@@ -915,6 +915,8 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .vcpu_load = vt_op(vcpu_load), .vcpu_put = vt_op(vcpu_put), + .HOST_OWNED_DEBUGCTL = DEBUGCTLMSR_FREEZE_IN_SMM, + .update_exception_bitmap = vt_op(update_exception_bitmap), .get_feature_msr = vmx_get_feature_msr, .get_msr = vt_op(get_msr),
arch/x86/kvm/vmx/nested.c +3 -0
@@ -4861,6 +4861,9 @@ static void nested_vmx_restore_host_state(struct kvm_vcpu *vcpu)WARN_ON (kvm_set_dr (vcpu,7 ,vmcs_readl (GUEST_DR7 ))); } +/* Reload DEBUGCTL to ensure vmcs01 has a fresh FREEZE_IN_SMM value. */ +vmx_reload_guest_debugctl (vcpu); +/* * Note that calling vmx_set_{efer,cr0,cr4} is important as they * handle a variety of side effects to KVM's software model.
arch/x86/kvm/vmx/vmx.c +3 -0
@@ -7373,6 +7373,9 @@ fastpath_t vmx_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags)if (run_flags &KVM_RUN_LOAD_GUEST_DR6 )set_debugreg (vcpu->arch.dr6,6 ); +if (run_flags &KVM_RUN_LOAD_DEBUGCTL ) +vmx_reload_guest_debugctl (vcpu); +/* * Refresh vmcs.HOST_CR3 if necessary. This must be done immediately * prior to VM-Enter, as the kernel may load a new ASID (PCID) any time
arch/x86/kvm/vmx/vmx.h +14 -1
@@ -419,12 +419,25 @@ bool vmx_is_valid_debugctl(struct kvm_vcpu *vcpu, u64 data, bool host_initiated)static inline void vmx_guest_debugctl_write (struct kvm_vcpu *vcpu,u64 val) { +WARN_ON_ONCE (val &DEBUGCTLMSR_FREEZE_IN_SMM ); + + val |= vcpu->arch.host_debugctl &DEBUGCTLMSR_FREEZE_IN_SMM ;vmcs_write64 (GUEST_IA32_DEBUGCTL , val); }static inline u64 vmx_guest_debugctl_read (void ) { -return vmcs_read64 (GUEST_IA32_DEBUGCTL ); +return vmcs_read64 (GUEST_IA32_DEBUGCTL ) & ~DEBUGCTLMSR_FREEZE_IN_SMM ; +} + +static inline void vmx_reload_guest_debugctl (struct kvm_vcpu *vcpu) +{ +u64 val =vmcs_read64 (GUEST_IA32_DEBUGCTL ); + +if (!((val ^ vcpu->arch.host_debugctl) &DEBUGCTLMSR_FREEZE_IN_SMM )) +return ; + +vmx_guest_debugctl_write (vcpu, val & ~DEBUGCTLMSR_FREEZE_IN_SMM ); }/*
arch/x86/kvm/x86.c +12 -2
@@ -10785,7 +10785,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)dm_request_for_irq_injection (vcpu) &&kvm_cpu_accept_dm_intr (vcpu);fastpath_t exit_fastpath; -u64 run_flags; +u64 run_flags, debug_ctl;bool req_immediate_exit = false;
@@ -11057,7 +11057,17 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)set_debugreg (DR7_FIXED_1 ,7 ); } - vcpu->arch.host_debugctl =get_debugctlmsr (); +/* +* Refresh the host DEBUGCTL snapshot after disabling IRQs, as DEBUGCTL +* can be modified in IRQ context, e.g. via SMP function calls. Inform +* vendor code if any host-owned bits were changed, e.g. so that the +* value loaded into hardware while running the guest can be updated. +*/ + debug_ctl =get_debugctlmsr (); +if ((debug_ctl ^ vcpu->arch.host_debugctl) & kvm_x86_ops.HOST_OWNED_DEBUGCTL && + !vcpu->arch.guest_state_protected) + run_flags |=KVM_RUN_LOAD_DEBUGCTL ; + vcpu->arch.host_debugctl = debug_ctl;guest_timing_enter_irqoff ();
Signature · 1 line
2.39.5
Re: [PATCH 6.16.y 6/6] KVM: VMX: Preserve host's DEBUGCTLMSR_FREEZE_IN_SMM while running the guest
Quoted 41 lines · Sasha Levin
From: Maxim Levitsky <[email protected]>
[ Upstream commit
6b1dd26544d045f6a79e8c73572c0c0db3ef3c1a]Set/clear
DEBUGCTLMSR_FREEZE_IN_SMMinGUEST_IA32_DEBUGCTLbased on the host's pre-VM-Enter value, i.e. preserve the host'sFREEZE_IN_SMMsetting while running the guest. When running with the "default treatment of SMIs" in effect (the only mode KVM supports), SMIs do not generate a VM-Exit that is visible to host (non-SMM) software, and instead transitions directly from VMX non-root to SMM. And critically, DEBUGCTL isn't context switched by hardware on SMI or RSM, i.e. SMM will run with whatever value was resident in hardware at the time of the SMI.Failure to preserve
FREEZE_IN_SMMresults in the PMU unexpectedly counting events while the CPU is executing in SMM, which can pollute profiling and potentially leak information into the guest.Check for changes in
FREEZE_IN_SMMprior to every entry into KVM's inner run loop, as the bit can be toggled in IRQ context via IPI callback (SMP function call), by way of/sys/devices/cpu/freeze_on_smi.Add a field in
kvm_x86_opsto communicate which DEBUGCTL bits need to be preserved, asFREEZE_IN_SMMis only supported and defined for Intel CPUs, i.e. explicitly checkingFREEZE_IN_SMMin common x86 is at best weird, and at worst could lead to undesirable behavior in the future if AMD CPUs ever happened to pick up a collision with the bit.Exempt TDX vCPUs, i.e. protected guests, from the check, as the TDX Module owns and controls
GUEST_IA32_DEBUGCTL.WARN in SVM if
KVM_RUN_LOAD_DEBUGCTLis set, mostly to document that the lack of handling isn't a KVM bug (TDX already WARNs on anyrun_flag).Lastly, explicitly reload
GUEST_IA32_DEBUGCTLon a VM-Fail that is missed by KVM but detected by hardware, i.e. innested_vmx_restore_host_state(). Doing so avoids the need to trackhost_debugctlon a per-VMCS basis, asGUEST_IA32_DEBUGCTLis unconditionally written byprepare_vmcs02()andload_vmcs12_host_state(). For the VM-Fail case, even though KVM won't have actually entered the guest,vcpu_enter_guest()will have run with vmcs02 active and thus could result in vmcs01 being run with a stale value.
Quoted 7 lines
Signed-off-by Maxim Levitsky <[email protected]>Co-developed-by Sean Christopherson <[email protected]>Signed-off-by Sean Christopherson <[email protected]>Signed-off-by Sasha Levin <[email protected]>---
Re: [PATCH 6.16.y 1/6] KVM: x86: Convert vcpu_run()'s immediate exit param into a generic bitmap
In reply to Sasha LevinQuoted 16 lines · Sasha Levin
From: Sean Christopherson <[email protected]>
[ Upstream commit
2478b1b220c49d25cb1c3f061ec4f9b351d9a131]Convert kvm_x86_ops.vcpu_run()'s "
force_immediate_exit" boolean parameter into an a generic bitmap so that similar "take action" information can be passed to vendor code without creating a pile of boolean parameters.This will allow dropping
kvm_x86_ops.set_dr6()in favor of a new flag, and will also allow for adding similar functionality for re-loading debugctl in the active VMCS.Opportunistically massage the TDX WARN and comment to prepare for adding more
run_flags, all of which are expected to be mutually exclusive with TDX, i.e. should be WARNed on.
Quoted 8 lines
No functional change intended.
Signed-off-by Sean Christopherson <[email protected]>Stable-dep-of:
6b1dd26544d0("KVM: VMX: Preserve host'sDEBUGCTLMSR_FREEZE_IN_SMMwhile running the guest")Signed-off-by Sasha Levin <[email protected]>---