Enable notify VM exit
v8 cover Enable notify VM exit
Notify VM exit is introduced to mitigate the potential DOS attach from malicious VM. This series is the userspace part to enable this feature through a new KVM capability KVM_CAP_X86_NOTIFY_VMEXIT. The detailed info can be seen in Patch 4.
The corresponding KVM support can be found in linux 6.0-rc:
(2f4073e08f4c KVM: VMX: Enable Notify VM exit)
---
Change logs:
v7 -> v8
- Add triple_fault_pending field transmission on migration (Paolo)
- Change the notify-vmexit and notify-window to the accelerator property. Add it as a x86-specific property. (Paolo)
- Add a preparation patch to expose struct KVMState in order to add target-specific property.
- Define three option for notify-vmexit. Make it on by default. (Paolo)
- Raise a KVM internal error instead of triple fault if invalid context of guest VMCS detected.
- v7: https://lore.kernel.org/qemu-devel/[email protected]/
v6 -> v7
- Add a warning message when exiting to userspace (Peter Xu)
- v6: https://lore.kernel.org/all/[email protected]/
v5 -> v6
- Add some info related to the valid range of notify_window in patch 2. (Peter Xu)
- Add the doc in qemu-options.hx. (Peter Xu)
- v5: https://lore.kernel.org/qemu-devel/[email protected]/
---
Chenyi Qiang (3):
i386: kvm: extend kvm_{get, put}_vcpu_events to support pending triple
fault
kvm: expose struct KVMState
i386: add notify VM exit support
Paolo Bonzini (1):
kvm: allow target-specific accelerator properties
accel/kvm/kvm-all.c | 78 ++----------------------- include/sysemu/kvm.h | 2 + include/sysemu/kvm_int.h | 75 ++++++++++++++++++++++++ qapi/run-state.json | 17 ++++++ qemu-options.hx | 11 ++++ target/arm/kvm.c | 4 ++ target/i386/cpu.c | 1 + target/i386/cpu.h | 1 + target/i386/kvm/kvm.c | 121 +++++++++++++++++++++++++++++++++++++++ target/i386/machine.c | 20 +++++++ target/mips/kvm.c | 4 ++ target/ppc/kvm.c | 4 ++ target/riscv/kvm.c | 4 ++ target/s390x/kvm/kvm.c | 4 ++ 14 files changed, 272 insertions(+), 74 deletions(-)
Signature · 1 line
2.17.1
PATCH v8 1/4 i386: kvm: extend kvm_{get, put}_vcpu_events to support pending triple fault
For the direct triple faults, i.e. hardware detected and KVM morphed to VM-Exit, KVM will never lose them. But for triple faults sythesized by KVM, e.g. the RSM path, if KVM exits to userspace before the request is serviced, userspace could migrate the VM and lose the triple fault.
A new flag KVM_VCPUEVENT_VALID_TRIPLE_FAULT is defined to signal that the event.triple_fault_pending field contains a valid state if the KVM_CAP_X86_TRIPLE_FAULT_EVENT capability is enabled.
--- target/i386/cpu.c | 1 + target/i386/cpu.h | 1 + target/i386/kvm/kvm.c | 20 ++++++++++++++++++++ target/i386/machine.c | 20 ++++++++++++++++++++ 4 files changed, 42 insertions(+)
target/i386/cpu.c +1 -0
@@ -6017,6 +6017,7 @@ static void x86_cpu_reset(DeviceState *dev) env->exception_has_payload = false; env->exception_payload =0 ; env->nmi_injected = false; + env->triple_fault_pending = false;#if !defined(CONFIG_USER_ONLY )/* We hard-wire the BSP to the first CPU. */ apic_designate_bsp (cpu->apic_state, s->cpu_index ==0 );
target/i386/cpu.h +1 -0
@@ -1739,6 +1739,7 @@ typedef struct CPUArchState {uint8_t has_error_code;uint8_t exception_has_payload;uint64_t exception_payload; +uint8_t triple_fault_pending;uint32_t ins_len;uint32_t sipi_vector;bool tsc_valid;
target/i386/kvm/kvm.c +20 -0
@@ -132,6 +132,7 @@ static int has_xcrs;static int has_pit_state2;static int has_sregs2;static int has_exception_payload; +static int has_triple_fault_event;static bool has_msr_mcg_ext_ctl;
@@ -2483,6 +2484,16 @@ int kvm_arch_init(MachineState *ms, KVMState *s) } } + has_triple_fault_event =kvm_check_extension (s,KVM_CAP_X86_TRIPLE_FAULT_EVENT ); +if (has_triple_fault_event) { + ret =kvm_vm_enable_cap (s,KVM_CAP_X86_TRIPLE_FAULT_EVENT ,0 , true); +if (ret <0 ) { +error_report ("kvm: Failed to enable triple fault event cap: %s" , +strerror (-ret)); +return ret; + } + } + ret =kvm_get_supported_msrs (s);if (ret <0 ) {return ret;
@@ -4299,6 +4310,11 @@ static int kvm_put_vcpu_events(X86CPU *cpu, int level) } } +if (has_triple_fault_event) { + events.flags |=KVM_VCPUEVENT_VALID_TRIPLE_FAULT ; + events.triple_fault.pending = env->triple_fault_pending; + } +return kvm_vcpu_ioctl (CPU (cpu),KVM_SET_VCPU_EVENTS , &events); }
@@ -4368,6 +4384,10 @@ static int kvm_get_vcpu_events(X86CPU *cpu) } } +if (events.flags &KVM_VCPUEVENT_VALID_TRIPLE_FAULT ) { + env->triple_fault_pending = events.triple_fault.pending; + } + env->sipi_vector = events.sipi_vector;return 0 ;
target/i386/machine.c +20 -0
@@ -1562,6 +1562,25 @@ static const VMStateDescription vmstate_arch_lbr = { } }; +static bool triple_fault_needed (void *opaque) +{ +X86CPU *cpu = opaque; +CPUX86State *env = &cpu->env; + +return env->triple_fault_pending; +} + +static const VMStateDescription vmstate_triple_fault = { + .name ="cpu/triple_fault" , + .version_id =1 , + .minimum_version_id =1 , + .needed = triple_fault_needed, + .fields = (VMStateField []) { +VMSTATE_UINT8 (env.triple_fault_pending,X86CPU ), +VMSTATE_END_OF_LIST () + } +}; +const VMStateDescription vmstate_x86_cpu = { .name ="cpu" , .version_id =12 ,
@@ -1706,6 +1725,7 @@ const VMStateDescription vmstate_x86_cpu = { &vmstate_amx_xtile, #endif &vmstate_arch_lbr, + &vmstate_triple_fault,NULL } };
Signature · 1 line
2.17.1
PATCH v8 2/4 kvm: allow target-specific accelerator properties
In reply to Chenyi QiangFrom: Paolo Bonzini <[email protected]>
Several hypervisor capabilities in KVM are target-specific. When exposed to QEMU users as accelerator properties (i.e. -accel kvm,prop=value), they should not be available for all targets.
Add a hook for targets to add their own properties to -accel kvm, for now no such property is defined.
--- accel/kvm/kvm-all.c | 2 ++ include/sysemu/kvm.h | 2 ++ target/arm/kvm.c | 4 ++++ target/i386/kvm/kvm.c | 4 ++++ target/mips/kvm.c | 4 ++++ target/ppc/kvm.c | 4 ++++ target/riscv/kvm.c | 4 ++++ target/s390x/kvm/kvm.c | 4 ++++ 8 files changed, 28 insertions(+)
accel/kvm/kvm-all.c +2 -0
@@ -3737,6 +3737,8 @@ static void kvm_accel_class_init(ObjectClass *oc, void *data) NULL, NULL);object_class_property_set_description (oc,"dirty-ring-size" ,"Size of KVM dirty page ring buffer (default: 0, i.e. use bitmap)" ); + +kvm_arch_accel_class_init (oc); }static const TypeInfo kvm_accel_type = {
include/sysemu/kvm.h +2 -0
@@ -353,6 +353,8 @@ bool kvm_device_supported(int vmfd, uint64_t type);extern const KVMCapabilityInfo kvm_arch_required_capabilities[]; +void kvm_arch_accel_class_init (ObjectClass *oc); +void kvm_arch_pre_run (CPUState *cpu,struct kvm_run *run);MemTxAttrs kvm_arch_post_run (CPUState *cpu,struct kvm_run *run);
target/arm/kvm.c +4 -0
@@ -1056,3 +1056,7 @@ bool kvm_arch_cpu_check_are_resettable(void) {return true; } + +void kvm_arch_accel_class_init (ObjectClass *oc) +{ +}
target/i386/kvm/kvm.c +4 -0
@@ -5472,3 +5472,7 @@ void kvm_request_xsave_components(X86CPU *cpu, uint64_t mask) mask &= ~BIT_ULL (bit); } } + +void kvm_arch_accel_class_init (ObjectClass *oc) +{ +}
target/mips/kvm.c +4 -0
@@ -1294,3 +1294,7 @@ bool kvm_arch_cpu_check_are_resettable(void) {return true; } + +void kvm_arch_accel_class_init (ObjectClass *oc) +{ +}
target/ppc/kvm.c +4 -0
@@ -2966,3 +2966,7 @@ bool kvm_arch_cpu_check_are_resettable(void) {return true; } + +void kvm_arch_accel_class_init (ObjectClass *oc) +{ +}
target/riscv/kvm.c +4 -0
@@ -532,3 +532,7 @@ bool kvm_arch_cpu_check_are_resettable(void) {return true; } + +void kvm_arch_accel_class_init (ObjectClass *oc) +{ +}
target/s390x/kvm/kvm.c +4 -0
@@ -2581,3 +2581,7 @@ int kvm_s390_get_zpci_op(void) {return cap_zpci_op; } + +void kvm_arch_accel_class_init (ObjectClass *oc) +{ +}
Signature · 1 line
2.17.1
PATCH v8 3/4 kvm: expose struct KVMState
In reply to Chenyi QiangExpose struct KVMState out of kvm-all.c so that the field of struct KVMState can be accessed when defining target-specific accelerator properties.
--- accel/kvm/kvm-all.c | 74 --------------------------------------- include/sysemu/kvm_int.h | 75 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 75 insertions(+), 74 deletions(-)
accel/kvm/kvm-all.c +0 -74
@@ -77,86 +77,12 @@ do { } while (0)#endif -#define KVM_MSI_HASHTAB_SIZE 256 -struct KVMParkedVcpu {unsigned long vcpu_id;int kvm_fd;QLIST_ENTRY (KVMParkedVcpu) node; }; -enum KVMDirtyRingReaperState { -KVM_DIRTY_RING_REAPER_NONE =0 , -/* The reaper is sleeping */ -KVM_DIRTY_RING_REAPER_WAIT , -/* The reaper is reaping for dirty pages */ -KVM_DIRTY_RING_REAPER_REAPING , -}; - -/* -* KVM reaper instance, responsible for collecting the KVM dirty bits -* via the dirty ring. -*/ -struct KVMDirtyRingReaper { -/* The reaper thread */ -QemuThread reaper_thr; -volatile uint64_t reaper_iteration;/* iteration number of reaper thr */ -volatile enum KVMDirtyRingReaperState reaper_state;/* reap thr state */ -}; - -struct KVMState -{ -AccelState parent_obj; - -int nr_slots; -int fd; -int vmfd; -int coalesced_mmio; -int coalesced_pio; -struct kvm_coalesced_mmio_ring *coalesced_mmio_ring; -bool coalesced_flush_in_progress; -int vcpu_events; -int robust_singlestep; -int debugregs; -#ifdef KVM_CAP_SET_GUEST_DEBUG - QTAILQ_HEAD(, kvm_sw_breakpoint) kvm_sw_breakpoints; -#endif -int max_nested_state_len; -int many_ioeventfds; -int intx_set_mask; -int kvm_shadow_mem; -bool kernel_irqchip_allowed; -bool kernel_irqchip_required; -OnOffAuto kernel_irqchip_split; -bool sync_mmu; -uint64_t manual_dirty_log_protect; -/* The man page (and posix) say ioctl numbers are signed int, but -* they're not. Linux, glibc and *BSD all treat ioctl numbers as -* unsigned, and treating them as signed here can break things */ -unsigned irq_set_ioctl; -unsigned int sigmask_len; -GHashTable *gsimap; -#ifdef KVM_CAP_IRQ_ROUTING -struct kvm_irq_routing *irq_routes; -int nr_allocated_irq_routes; -unsigned long *used_gsi_bitmap; -unsigned int gsi_count; - QTAILQ_HEAD(, KVMMSIRoute) msi_hashtab[KVM_MSI_HASHTAB_SIZE]; -#endif -KVMMemoryListener memory_listener; - QLIST_HEAD(, KVMParkedVcpu) kvm_parked_vcpus; - -/* For "info mtree -f" to tell if an MR is registered in KVM */ -int nr_as; -struct KVMAs { -KVMMemoryListener *ml; -AddressSpace *as; - } *as; -uint64_t kvm_dirty_ring_bytes;/* Size of the per-vcpu dirty ring */ -uint32_t kvm_dirty_ring_size;/* Number of dirty GFNs per ring */ -struct KVMDirtyRingReaper reaper; -}; -KVMState *kvm_state;bool kvm_kernel_irqchip;bool kvm_split_irqchip;
include/sysemu/kvm_int.h +75 -0
81 lines not on this page · raw message
Signature · 1 line
2.17.1
PATCH v8 4/4 i386: add notify VM exit support
In reply to Chenyi QiangThere are cases that malicious virtual machine can cause CPU stuck (due to event windows don't open up), e.g., infinite loop in microcode when nested #AC (CVE-2015-5307). No event window means no event (NMI, SMI and IRQ) can be delivered. It leads the CPU to be unavailable to host or other VMs. Notify VM exit is introduced to mitigate such kind of attacks, which will generate a VM exit if no event window occurs in VM non-root mode for a specified amount of time (notify window).
A new KVM capability KVM_CAP_X86_NOTIFY_VMEXIT is exposed to user space so that the user can query the capability and set the expected notify window when creating VMs. The format of the argument when enabling this capability is as follows:
Bit 63:32 - notify window specified in qemu command
Bit 31:0 - some flags (e.g. KVM_X86_NOTIFY_VMEXIT_ENABLED is set to
enable the feature.)
Users can configure the feature by a new (x86 only) accel property:
qemu -accel kvm,notify-vmexit=run|internal-error|disable,notify-window=n
The default option of notify-vmexit is run, which will enable the capability and do nothing if the exit happens. The internal-error option raises a KVM internal error if it happens. The disable option does not enable the capability. The default value of notify-window is 0. It is valid only when notify-vmexit is not disabled. The valid range of notify-window is non-negative. It is even safe to set it to zero since there's an internal hardware threshold to be added to ensure no false positive.
Because a notify VM exit may happen with VM_CONTEXT_INVALID set in exit qualification (no cases are anticipated that would set this bit), which means VM context is corrupted. It would be reflected in the flags of KVM_EXIT_NOTIFY exit. If KVM_NOTIFY_CONTEXT_INVALID bit is set, raise a KVM internal error unconditionally.
--- accel/kvm/kvm-all.c | 2 + qapi/run-state.json | 17 ++++++++ qemu-options.hx | 11 +++++ target/i386/kvm/kvm.c | 97 +++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 127 insertions(+)
accel/kvm/kvm-all.c +2 -0
@@ -3636,6 +3636,8 @@ static void kvm_accel_instance_init(Object *obj) s->kernel_irqchip_split =ON_OFF_AUTO_AUTO ;/* KVM dirty ring is by default off */ s->kvm_dirty_ring_size =0 ; + s->notify_vmexit =NOTIFY_VMEXIT_OPTION_RUN ; + s->notify_window =0 ; }static void kvm_accel_class_init (ObjectClass *oc,void *data)
qapi/run-state.json +17 -0
@@ -643,3 +643,20 @@ { 'struct': 'MemoryFailureFlags', 'data': { 'action-required': 'bool', 'recursive': 'bool'} } + +## +# @NotifyVmexitOption: +# +# An enumeration of the options specified when enabling notify VM exit +# +# @run: enable the feature, do nothing and continue if the notify VM exit happens. +# +# @internal-error: enable the feature, raise a internal error if the notify +# VM exit happens. +# +# @disable: disable the feature. +# +# Since: 7.2 +## +{ 'enum': 'NotifyVmexitOption', + 'data': [ 'run', 'internal-error', 'disable' ] }
qemu-options.hx +11 -0
@@ -191,6 +191,7 @@ DEF("accel", HAS_ARG, QEMU_OPTION_accel, " split-wx=on|off (enable TCG split w^x mapping)\n" " tb-size=n (TCG translation block cache size)\n" " dirty-ring-size=n (KVM dirty ring GFN count, default 0)\n" + " notify-vmexit=run|internal-error|disable,notify-window=n (enable notify VM exit and set notify window, x86 only)\n" " thread=single|multi (enable multi-threaded TCG)\n", QEMU_ARCH_ALL) SRST ``-accel name[,prop=value[,...]]``
@@ -242,6 +243,16 @@ SRST is disabled (dirty-ring-size=0). When enabled, KVM will instead record dirty pages in a bitmap. + ``notify-vmexit=run|internal-error|disable,notify-window=n`` + Enables or disables notify VM exit support on x86 host and specify + the corresponding notify window to trigger the VM exit if enabled. + ``run`` option enables the feature. It does nothing and continue + if the exit happens. ``internal-error`` option enables the feature. + It raises a internal error. ``disable`` option doesn't enable the feature. + This feature can mitigate the CPU stuck issue due to event windows don't + open up for a specified of time (i.e. notify-window). + Default: notify-vmexit=run,notify-window=0. + ERST DEF("smp", HAS_ARG, QEMU_OPTION_smp,
target/i386/kvm/kvm.c +97 -0
127 lines not on this page · raw message
Signature · 1 line
2.17.1
Paolo Bonzini
maintainerQuoted 7 lines · Chenyi Qiang
Notify VM exit is introduced to mitigate the potential DOS attach from malicious VM. This series is the userspace part to enable this feature through a new KVM capability
KVM_CAP_X86_NOTIFY_VMEXIT. The detailed info can be seen in Patch 4.The corresponding KVM support can be found in linux 6.0-rc:
(2f4073e08f4cKVM: VMX: Enable Notify VM exit)
Thanks, I will queue this in my next pull request.
Paolo
Quoted 47 lines
---
Change logs:
v7 -> v8
- Addtriple_fault_pendingfield transmission on migration (Paolo)
- Change the notify-vmexit and notify-window to the accelerator property. Add it as a x86-specific property. (Paolo)
- Add a preparation patch to expose struct KVMState in order to add target-specific property.
- Define three option for notify-vmexit. Make it on by default. (Paolo)
- Raise a KVM internal error instead of triple fault if invalid context of guest VMCS detected.
- v7: https://lore.kernel.org/qemu-devel/[email protected]/v6 -> v7
- Add a warning message when exiting to userspace (Peter Xu)
- v6: https://lore.kernel.org/all/[email protected]/v5 -> v6
- Add some info related to the valid range ofnotify_windowin patch 2. (Peter Xu)
- Add the doc in qemu-options.hx. (Peter Xu)
- v5: https://lore.kernel.org/qemu-devel/[email protected]/---
Chenyi Qiang (3):
i386: kvm: extend kvm_{get, put}_vcpu_events to support pending triple
fault
kvm: expose struct KVMState
i386: add notify VM exit supportPaolo Bonzini (1):
kvm: allow target-specific accelerator propertiesaccel/kvm/kvm-all.c | 78 ++----------------------- include/sysemu/kvm.h | 2 + include/sysemu/kvm_int.h | 75 ++++++++++++++++++++++++ qapi/run-state.json | 17 ++++++ qemu-options.hx | 11 ++++ target/arm/kvm.c | 4 ++ target/i386/cpu.c | 1 + target/i386/cpu.h | 1 + target/i386/kvm/kvm.c | 121 +++++++++++++++++++++++++++++++++++++++ target/i386/machine.c | 20 +++++++ target/mips/kvm.c | 4 ++ target/ppc/kvm.c | 4 ++ target/riscv/kvm.c | 4 ++ target/s390x/kvm/kvm.c | 4 ++ 14 files changed, 272 insertions(+), 74 deletions(-)
Quoted 6 lines · Paolo Bonzini
Quoted 5 lines · Chenyi Qiang
Notify VM exit is introduced to mitigate the potential DOS attach from malicious VM. This series is the userspace part to enable this feature through a new KVM capability
KVM_CAP_X86_NOTIFY_VMEXIT. The detailed info can be seen in Patch 4.
Quoted 7 lines
Quoted 2 lines
The corresponding KVM support can be found in linux 6.0-rc:
(2f4073e08f4cKVM: VMX: Enable Notify VM exit)Thanks, I will queue this in my next pull request.
Paolo
Thanks Paolo!
Please take the resend version at https://lore.kernel.org/qemu-devel/[email protected]/
There's a minor compile issue in this one.
Chenyi
Quoted 55 lines
Quoted 54 lines
---
Change logs:
v7 -> v8
- Addtriple_fault_pendingfield transmission on migration (Paolo)
- Change the notify-vmexit and notify-window to the accelerator property. Add it as
a x86-specific property. (Paolo)
- Add a preparation patch to expose struct KVMState in order to add target-specific property.
- Define three option for notify-vmexit. Make it on by default. (Paolo)
- Raise a KVM internal error instead of triple fault if invalid context of guest VMCS detected.
- v7:
https://lore.kernel.org/qemu-devel/[email protected]/v6 -> v7
- Add a warning message when exiting to userspace (Peter Xu)
- v6:
https://lore.kernel.org/all/[email protected]/v5 -> v6
- Add some info related to the valid range ofnotify_windowin patch
2. (Peter Xu)
- Add the doc in qemu-options.hx. (Peter Xu)
- v5:
https://lore.kernel.org/qemu-devel/[email protected]/---
Chenyi Qiang (3):
i386: kvm: extend kvm_{get, put}_vcpu_events to support pending triple fault
kvm: expose struct KVMState
i386: add notify VM exit supportPaolo Bonzini (1):
kvm: allow target-specific accelerator propertiesaccel/kvm/kvm-all.c | 78 ++----------------------- include/sysemu/kvm.h | 2 +
include/sysemu/kvm_int.h| 75 ++++++++++++++++++++++++ qapi/run-state.json | 17 ++++++
qemu-options.hx | 11 ++++
target/arm/kvm.c | 4 ++
target/i386/cpu.c | 1 +
target/i386/cpu.h | 1 +
target/i386/kvm/kvm.c | 121 +++++++++++++++++++++++++++++++++++++++ target/i386/machine.c | 20 +++++++
target/mips/kvm.c | 4 ++
target/ppc/kvm.c | 4 ++
target/riscv/kvm.c | 4 ++
target/s390x/kvm/kvm.c | 4 ++14 files changed, 272 insertions(+), 74 deletions(-)