KVM: arm64: Confine protected VM vCPU state to EL2
Subsystems (4)
Fuad Tabba
reviewerv4 cover KVM: arm64: Confine protected VM vCPU state to EL2
Hi folks,
Changes since v3 [1]:
- A protected vCPU that EL2 holds powered off reads as KVM_MP_STATE_UNINITIALIZED, in place of a new bool. (Marc)
- The marshalling and PSCI patches access the host copy through the vCPU accessors, with every assignment on one line. (Marc)
- The PC adjustment pair is kvm_adjust_pc_get()/put() and tests for the hyp vCPU directly. (Marc)
- CPU_OFF clears the reset flag before its release of OFF, CPU_ON's relaxed cmpxchg is ordered by its control dependency, and the CPU_ON rollback stores OFF with a release. pkvm_reset_vcpu() carries the CPU_ON/CPU_OFF sequence as a comment and drops its WARN_ON(). (Vincent, Will)
- Collected Reviewed-bys.
Following the vCPU state-sync series [3], this series completes the job for protected VMs: a protected guest's register state stays at EL2, and the host sees only what handling each exit requires.
EL2 marshals a protected vCPU's state per exception class instead of copying the whole context both ways. It owns the vCPU's trap configuration, system register reset and HVC handling, and implements PSCI itself: AFFINITY_INFO is left to the host, and CPU_ON and CPU_OFF are resolved at EL2 with the host only scheduling the target or stopping it. A protected guest's TRNG calls, which the host handled until now, return NOT_SUPPORTED until TRNG for protected guests follows. EL2 implements PSCI 1.1, so a protected guest also loses the functions the host supports above that version, SYSTEM_OFF2 included. Host ioctls that would reach the state EL2 owns fail with -EPERM, so a protected VM's state isn't save/restorable. All of this is scoped to KVM_VM_TYPE_ARM_PROTECTED, and pkvm.rst describes the resulting API.
The kvmtool changes that go with this are posted separately [4].
Patches 1 to 3 go out separately: the HCR_EL2.VSE fix [5], and the host vCPU VM read and pin fixes, patches 4 and 5 of the host hypercall fixes series [2]. None of the three is part of this series. They're carried so the series applies as is and Sashiko can run on it.
The KVM_ARM_PREFERRED_TARGET documentation fix [6] went out just ahead of v1. Nothing here depends on it to apply, but patch 18 documents vCPU feature availability as something the capabilities report, while api.rst 4.83 still points userspace at a bitmap that has always been empty.
The series is structured as follows:
01: The HCR_EL2.VSE fix, posted separately. 02-03: The host vCPU VM read and pin fixes, posted separately. 04: Steal time disabled for protected VMs. 05-06: Per-exception-class entry handlers; EL2 owns a protected
vCPU's trap configuration.
07-09: Timer state, system register reset and HVC handling at EL2. 10-11: PSCI at EL2, and the KVM_ARM_VCPU_INIT and PSCI version
restrictions.
12-14: Host PC adjustments blocked; an UNDEF at EL2 for exit
classes the host doesn't emulate; per-class state marshalling.
15-16: Host changes to private state, and host power-on of a vCPU
EL2 holds powered off, rejected.
17: Capability allowlist.
18: Documentation.
Still to come: selftests, TRNG, self-hosted debug and SVE for protected guests, and much more, as separate series.
Based on v7.3-rc3 (fd73f4a665989).
Cheers,
/fuad
[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/
[3] https://lore.kernel.org/all/[email protected]/
[4] https://lore.kernel.org/all/[email protected]/
[5] https://lore.kernel.org/all/[email protected]/
[6] https://lore.kernel.org/all/[email protected]/
Fuad Tabba (16):
KVM: arm64: Sync HCR_EL2.VSE back to the host vCPU under pKVM
KVM: arm64: Validate the host vCPU's VM before reading it under pKVM
KVM: arm64: Pin the host vCPU before adjusting its PC under pKVM
KVM: arm64: Disable steal time for protected VMs
KVM: arm64: Skip fixed-feature state flush for protected vCPUs
KVM: arm64: Add system register reset framework for protected VMs
KVM: arm64: Implement HVC handling for protected guests at EL2
KVM: arm64: Handle PSCI calls for protected VMs at EL2
KVM: arm64: Restrict KVM_ARM_VCPU_INIT and PSCI version for protected
VMs
KVM: arm64: Prevent host PC adjustments for protected vCPUs
KVM: arm64: Inject an UNDEF at EL2 for unhandled protected guest exits
KVM: arm64: Add per-EC entry/exit state marshalling for protected
guests
KVM: arm64: Reject host access to protected VM private state
KVM: arm64: Reject host power-on of a vCPU that EL2 holds powered off
KVM: arm64: Advertise the capabilities that protected VMs support
KVM: arm64: Document the protected VM userspace API
Marc Zyngier (2):
KVM: arm64: Introduce per-EC entry handlers for pKVM
KVM: arm64: Add {flush,sync}_hyp_timer_state() primitives
Documentation/virt/kvm/api.rst | 27 +- .../virt/kvm/arm/fw-pseudo-registers.rst | 2 + Documentation/virt/kvm/arm/pkvm.rst | 164 ++++- Documentation/virt/kvm/devices/vcpu.rst | 3 +- arch/arm64/include/asm/kvm_host.h | 22 +- arch/arm64/include/asm/kvm_hyp.h | 4 + arch/arm64/include/asm/kvm_pkvm.h | 33 + arch/arm64/kvm/arm.c | 76 ++- arch/arm64/kvm/guest.c | 11 + arch/arm64/kvm/hyp/exception.c | 26 +- arch/arm64/kvm/hyp/include/hyp/adjust_pc.h | 48 ++ arch/arm64/kvm/hyp/include/nvhe/pkvm.h | 20 + arch/arm64/kvm/hyp/nvhe/hyp-main.c | 595 +++++++++++++++++- arch/arm64/kvm/hyp/nvhe/pkvm.c | 422 ++++++++++++- arch/arm64/kvm/hyp/nvhe/switch.c | 29 +- arch/arm64/kvm/hyp/nvhe/sys_regs.c | 60 ++ arch/arm64/kvm/hypercalls.c | 7 + arch/arm64/kvm/pkvm.c | 21 +- arch/arm64/kvm/psci.c | 11 +- arch/arm64/kvm/pvtime.c | 10 +- 20 files changed, 1503 insertions(+), 88 deletions(-)
base-commit: fd73f4a6659897191fa0d40695fe370925dd3780
Signature · 1 line
2.39.5
Fuad Tabba
reviewerPATCH v4 1/18 KVM: arm64: Sync HCR_EL2.VSE back to the host vCPU under pKVM
flush_hyp_vcpu() copies HCR_EL2.VSE from the host vCPU into the hyp vCPU on every entry, and nothing copies it back. Once the guest takes the vSError the hyp vCPU's copy clears with the hardware bit while the host's stays set, so the next entry pends the same SError again, KVM_GET_VCPU_EVENTS keeps reporting it, and kvm_arch_vcpu_runnable() never lets the vCPU block in WFI.
Reflect VSE back on every exit, which the flush-side comment already states.
--- arch/arm64/kvm/hyp/nvhe/hyp-main.c | 4 ++++ 1 file changed, 4 insertions(+)
arch/arm64/kvm/hyp/nvhe/hyp-main.c +4 -0
@@ -276,6 +276,10 @@ static void sync_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu) host_vcpu->arch.iflags = hyp_vcpu->vcpu.arch.iflags; +/* Cleared by hardware once the guest takes the vSError. */ + host_vcpu->arch.hcr_el2 &= ~HCR_VSE ; + host_vcpu->arch.hcr_el2 |= hyp_vcpu->vcpu.arch.hcr_el2 &HCR_VSE ; +sync_hyp_vgic_state (hyp_vcpu); }
Signature · 1 line
2.39.5
Fuad Tabba
reviewerPATCH v4 2/18 KVM: arm64: Validate the host vCPU's VM before reading it under pKVM
In reply to Fuad TabbaOn an MTE-capable host under pKVM, enter_exception64() reads the VM's MTE flag through vcpu->kvm, which for a host vCPU is a host-writable pointer nothing validates. The host can point it at any address in the hyp linear map and read back bit 1 of that word through PSR_TCO in the vCPU's CPSR, or panic the hypervisor with an unmapped one.
Get the VM through a get/put pair around the read: a loaded vCPU's is the hyp VM, an unloaded host vCPU's is read once and pinned, and a pointer the host never shared leaves TCO clear.
--- arch/arm64/kvm/hyp/exception.c | 5 ++- arch/arm64/kvm/hyp/include/hyp/adjust_pc.h | 28 ++++++++++++++++ arch/arm64/kvm/hyp/include/nvhe/pkvm.h | 2 ++ arch/arm64/kvm/hyp/nvhe/pkvm.c | 37 ++++++++++++++++++++++ 4 files changed, 71 insertions(+), 1 deletion(-)
arch/arm64/kvm/hyp/exception.c +4 -1
@@ -70,6 +70,7 @@ static void enter_exception64(struct kvm_vcpu *vcpu, unsigned long target_mode, enum exception_type type) {unsigned long sctlr, vbar, old, new, mode; +struct kvm *kvm;u64 exc_offset; mode = *vcpu_cpsr (vcpu) & (PSR_MODE_MASK |PSR_MODE32_BIT );
@@ -109,8 +110,10 @@ static void enter_exception64(struct kvm_vcpu *vcpu, unsigned long target_mode, new |= (old &PSR_C_BIT ); new |= (old &PSR_V_BIT ); -if (kvm_has_mte (kern_hyp_va (vcpu->kvm))) + kvm =vcpu_get_kvm (vcpu); +if (kvm &&kvm_has_mte (kvm)) new |=PSR_TCO_BIT ; +vcpu_put_kvm (vcpu, kvm); new |= (old &PSR_DIT_BIT );
arch/arm64/kvm/hyp/include/hyp/adjust_pc.h +28 -0
@@ -13,6 +13,34 @@#include <asm/kvm_emulate.h> #include <asm/kvm_host.h> +#ifdef __KVM_NVHE_HYPERVISOR__ +#include <nvhe/pkvm.h> + +/* Under pKVM a host vCPU's ->kvm is host-writable. */ +static inline struct kvm *vcpu_get_kvm (struct kvm_vcpu *vcpu) +{ +if (is_protected_kvm_enabled ()) +return pkvm_vcpu_get_kvm (vcpu); + +return kern_hyp_va (vcpu->kvm); +} + +static inline void vcpu_put_kvm (struct kvm_vcpu *vcpu,struct kvm *kvm) +{ +if (is_protected_kvm_enabled ()) +pkvm_vcpu_put_kvm (vcpu, kvm); +} +#else +static inline struct kvm *vcpu_get_kvm (struct kvm_vcpu *vcpu) +{ +return vcpu->kvm; +} + +static inline void vcpu_put_kvm (struct kvm_vcpu *vcpu,struct kvm *kvm) +{ +} +#endif +static inline void kvm_skip_instr (struct kvm_vcpu *vcpu) {if (vcpu_mode_is_32bit (vcpu)) {
arch/arm64/kvm/hyp/include/nvhe/pkvm.h +2 -0
@@ -83,6 +83,8 @@ struct pkvm_hyp_vcpu *pkvm_load_hyp_vcpu(pkvm_handle_t handle, unsigned int vcpu_idx);void pkvm_put_hyp_vcpu (struct pkvm_hyp_vcpu *hyp_vcpu);struct pkvm_hyp_vcpu *pkvm_get_loaded_hyp_vcpu (void ); +struct kvm *pkvm_vcpu_get_kvm (struct kvm_vcpu *vcpu); +void pkvm_vcpu_put_kvm (struct kvm_vcpu *vcpu,struct kvm *kvm);struct pkvm_hyp_vm *get_pkvm_hyp_vm (pkvm_handle_t handle);struct pkvm_hyp_vm *get_np_pkvm_hyp_vm (pkvm_handle_t handle);
arch/arm64/kvm/hyp/nvhe/pkvm.c +37 -0
@@ -304,6 +304,43 @@ struct pkvm_hyp_vcpu *pkvm_get_loaded_hyp_vcpu(void) } +static struct pkvm_hyp_vm *pkvm_get_loaded_hyp_vm (struct kvm_vcpu *vcpu) +{ +struct pkvm_hyp_vcpu *hyp_vcpu =pkvm_get_loaded_hyp_vcpu (); + +if (hyp_vcpu && + (vcpu == &hyp_vcpu->vcpu || vcpu == hyp_vcpu->host_vcpu)) +return pkvm_hyp_vcpu_to_hyp_vm (hyp_vcpu); + +return NULL ; +} + +/* +* A loaded vCPU's VM is the hyp VM. An unloaded host vCPU's is mapped at +* EL2 only while pinned, so it's read once and pinned; the pin fails for +* memory the host isn't sharing. +*/ +struct kvm *pkvm_vcpu_get_kvm (struct kvm_vcpu *vcpu) +{ +struct pkvm_hyp_vm *hyp_vm =pkvm_get_loaded_hyp_vm (vcpu); +struct kvm *kvm; + +if (hyp_vm) +return &hyp_vm->kvm; + + kvm =kern_hyp_va (READ_ONCE (vcpu->kvm)); +if (hyp_pin_shared_mem (kvm, kvm +1 )) +return NULL ; + +return kvm; +} + +void pkvm_vcpu_put_kvm (struct kvm_vcpu *vcpu,struct kvm *kvm) +{ +if (kvm && !pkvm_get_loaded_hyp_vm (vcpu)) +hyp_unpin_shared_mem (kvm, kvm +1 ); +} +struct pkvm_hyp_vm *get_pkvm_hyp_vm (pkvm_handle_t handle) {struct pkvm_hyp_vm *hyp_vm;
Signature · 1 line
2.39.5
Fuad Tabba
reviewerPATCH v4 3/18 KVM: arm64: Pin the host vCPU before adjusting its PC under pKVM
In reply to Fuad TabbaUnder pKVM, a page the host shares with EL2 is mapped at EL2 only while it's pinned, and the host vCPU is pinned at its first KVM_RUN. KVM_SET_VCPU_EVENTS with ext_dabt_pending reaches __kvm_adjust_pc() at EL2 before that, and dereferencing the unmapped host vCPU panics the hypervisor. Any process holding /dev/kvm on a pKVM host can trigger it.
Pin the host vCPU around the adjustment when no hyp vCPU is loaded for it. A loaded hyp vCPU implies it's pinned. A pin fails only for memory the host isn't sharing, and the request is then dropped like any other bad host pointer.
--- arch/arm64/kvm/hyp/nvhe/hyp-main.c | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-)
arch/arm64/kvm/hyp/nvhe/hyp-main.c +20 -1
@@ -546,8 +546,27 @@ static void handle___pkvm_host_mkyoung_guest(struct kvm_cpu_context *host_ctxt)static void handle___kvm_adjust_pc (struct kvm_cpu_context *host_ctxt) { DECLARE_REG(struct kvm_vcpu *, vcpu, host_ctxt, 1); +struct pkvm_hyp_vcpu *hyp_vcpu; +struct kvm_vcpu *host_vcpu; -__kvm_adjust_pc (kern_hyp_va (vcpu)); + host_vcpu =__get_host_hyp_vcpus (vcpu, &hyp_vcpu); +if (host_vcpu) { +__kvm_adjust_pc (host_vcpu); +return ; + } + +/* +* With no hyp vCPU loaded for it, the host vCPU may be unpinned, +* and so unmapped at EL2: its first run pins it. A pin fails only +* for memory the host isn't sharing, a bad pointer, so the request +* is dropped. +*/ + host_vcpu =kern_hyp_va (vcpu); +if (hyp_pin_shared_mem (host_vcpu, host_vcpu +1 )) +return ; + +__kvm_adjust_pc (host_vcpu); +hyp_unpin_shared_mem (host_vcpu, host_vcpu +1 ); }static void handle___kvm_flush_vm_context (struct kvm_cpu_context *host_ctxt)
Signature · 1 line
2.39.5
Fuad Tabba
reviewerPATCH v4 4/18 KVM: arm64: Disable steal time for protected VMs
In reply to Fuad TabbapKVM doesn't support steal time for protected guests, and KVM_CAP_STEAL_TIME already reads 0 for them on the VM file descriptor, but kvm_arm_pvtime_supported() takes no VM, so the host still accepts the KVM_ARM_VCPU_PVTIME_CTRL attribute, whose IPA would point kvm_update_stolen_time() at the guest's private memory on every vCPU load. Pass the VM in and return false for a protected one, so the attribute returns -ENXIO as it does where steal time isn't implemented.
--- arch/arm64/include/asm/kvm_host.h | 2 +- arch/arm64/kvm/arm.c | 2 +- arch/arm64/kvm/pvtime.c | 10 +++++----- 3 files changed, 7 insertions(+), 7 deletions(-)
arch/arm64/include/asm/kvm_host.h +1 -1
@@ -1346,7 +1346,7 @@ long kvm_hypercall_pv_features(struct kvm_vcpu *vcpu);gpa_t kvm_init_stolen_time (struct kvm_vcpu *vcpu);void kvm_update_stolen_time (struct kvm_vcpu *vcpu); -bool kvm_arm_pvtime_supported (void ); +bool kvm_arm_pvtime_supported (struct kvm *kvm);int kvm_arm_pvtime_set_attr (struct kvm_vcpu *vcpu,struct kvm_device_attr *attr); int kvm_arm_pvtime_get_attr(struct kvm_vcpu *vcpu,
arch/arm64/kvm/arm.c +1 -1
@@ -447,7 +447,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) r =system_supports_mte ();break ;case KVM_CAP_STEAL_TIME : - r =kvm_arm_pvtime_supported (); + r =kvm_arm_pvtime_supported (kvm);break ;case KVM_CAP_ARM_EL1_32BIT : r =cpus_have_final_cap (ARM64_HAS_32BIT_EL1 );
arch/arm64/kvm/pvtime.c +5 -5
@@ -67,9 +67,9 @@ gpa_t kvm_init_stolen_time(struct kvm_vcpu *vcpu)return base; } -bool kvm_arm_pvtime_supported (void ) +bool kvm_arm_pvtime_supported (struct kvm *kvm) { -return !!sched_info_on (); +return !!sched_info_on () && (!kvm || !kvm_vm_is_protected (kvm)); } int kvm_arm_pvtime_set_attr(struct kvm_vcpu *vcpu,
@@ -81,7 +81,7 @@ int kvm_arm_pvtime_set_attr(struct kvm_vcpu *vcpu,int ret =0 ;int idx; -if (!kvm_arm_pvtime_supported () || +if (!kvm_arm_pvtime_supported (kvm) || attr->attr !=KVM_ARM_VCPU_PVTIME_IPA )return -ENXIO ;
@@ -110,7 +110,7 @@ int kvm_arm_pvtime_get_attr(struct kvm_vcpu *vcpu,u64 __user *user = (u64 __user *)attr->addr;u64 ipa; -if (!kvm_arm_pvtime_supported () || +if (!kvm_arm_pvtime_supported (vcpu->kvm) || attr->attr !=KVM_ARM_VCPU_PVTIME_IPA )return -ENXIO ;
@@ -126,7 +126,7 @@ int kvm_arm_pvtime_has_attr(struct kvm_vcpu *vcpu, {switch (attr->attr) {case KVM_ARM_VCPU_PVTIME_IPA : -if (kvm_arm_pvtime_supported ()) +if (kvm_arm_pvtime_supported (vcpu->kvm))return 0 ; }return -ENXIO ;
Signature · 1 line
2.39.5
Fuad Tabba
reviewerPATCH v4 5/18 KVM: arm64: Introduce per-EC entry handlers for pKVM
In reply to Fuad TabbaFrom: Marc Zyngier <[email protected]>
Add an ESR_EL2.EC-indexed handler table, entry_hyp_vm_handlers[], read from flush_hyp_vcpu() on re-entry when the previous exit was a trap: sync_hyp_vcpu() records the exit reason in the hyp vCPU as exit_code, and the trap's exception class comes from its ESR_EL2. Its single generic handler stands in until the per-EC marshalling patch replaces the table with protected-only ones.
Entry carries the host's requested PC updates to the hyp vCPU: add vcpu_copy_flag() to copy a masked set of flags between the two vCPU structures, with PC_UPDATE_REQ covering INCREMENT_PC and the pending-exception flags. The wholesale iflags copy moves into the non-protected branch; a protected vCPU takes only PC_UPDATE_REQ.
--- arch/arm64/include/asm/kvm_host.h | 19 +++++++++++++ arch/arm64/kvm/hyp/include/nvhe/pkvm.h | 3 ++ arch/arm64/kvm/hyp/nvhe/hyp-main.c | 39 +++++++++++++++++++++++--- 3 files changed, 57 insertions(+), 4 deletions(-)
arch/arm64/include/asm/kvm_host.h +19 -0
@@ -1033,11 +1033,28 @@ struct kvm_vcpu_arch { set; \ }) +#define __vcpu_copy_flag (vt, vs, flagset, f, m) \ + do { \ + typeof(vs->arch.flagset) tmp, val; \ + \ + __build_check_flag(vs, flagset, f, m); \ + \ + val = READ_ONCE(vs->arch.flagset); \ + val &= (m); \ + __vcpu_flags_preempt_disable(); \ + tmp = READ_ONCE(vt->arch.flagset); \ + tmp &= ~(m); \ + tmp |= val; \ + WRITE_ONCE(vt->arch.flagset, tmp); \ + __vcpu_flags_preempt_enable(); \ + } while (0) +#define vcpu_get_flag (v, ...) __vcpu_get_flag((v), __VA_ARGS__)#define vcpu_set_flag (v, ...) __vcpu_set_flag((v), __VA_ARGS__)#define vcpu_clear_flag (v, ...) __vcpu_clear_flag((v), __VA_ARGS__)#define vcpu_test_and_clear_flag (v, ...) \ __vcpu_test_and_clear_flag((v), __VA_ARGS__) +#define vcpu_copy_flag (vt, vs, ...) __vcpu_copy_flag((vt), (vs), __VA_ARGS__)/* KVM_ARM_VCPU_INIT completed */ #define VCPU_INITIALIZED __vcpu_single_flag(cflags, BIT(0))
@@ -1055,6 +1072,8 @@ struct kvm_vcpu_arch {#define INCREMENT_PC __vcpu_single_flag(iflags, BIT(1))/* Target EL/MODE (not a single flag, but let's abuse the macro) */ #define EXCEPT_MASK __vcpu_single_flag(iflags, GENMASK(3, 1)) +/* Cover both PENDING_EXCEPTION and EXCEPT_MASK for global operations */ +#define PC_UPDATE_REQ __vcpu_single_flag(iflags, GENMASK(3, 0))/* Host-set: the hyp flushes the non-protected vCPU state in on entry */ #define PKVM_HOST_STATE_DIRTY __vcpu_single_flag(iflags, BIT(4))
arch/arm64/kvm/hyp/include/nvhe/pkvm.h +3 -0
@@ -26,6 +26,9 @@ struct pkvm_hyp_vcpu {* per-cpu pointer tracking us. Otherwise, NULL if not loaded. */ struct pkvm_hyp_vcpu **loaded_hyp_vcpu; + +/* The previous exit's ARM_EXCEPTION_* code. */ +u32 exit_code; };/*
arch/arm64/kvm/hyp/nvhe/hyp-main.c +35 -4
@@ -31,6 +31,17 @@ unsigned int hyp_gicv3_nr_lr;void __kvm_hyp_host_forward_smc (struct kvm_cpu_context *host_ctxt); +typedef void (*hyp_entry_exit_handler_fn )(struct pkvm_hyp_vcpu *); + +static void handle_vm_entry_generic (struct pkvm_hyp_vcpu *hyp_vcpu) +{ +vcpu_copy_flag (&hyp_vcpu->vcpu, hyp_vcpu->host_vcpu,PC_UPDATE_REQ ); +} + +static const hyp_entry_exit_handler_fn entry_hyp_vm_handlers[] = { + [0 ... ESR_ELx_EC_MAX] = handle_vm_entry_generic, +}; +static void __hyp_sve_save_guest (struct kvm_vcpu *vcpu) {__vcpu_assign_sys_reg (vcpu,ZCR_EL1 ,read_sysreg_el1 (SYS_ZCR ));
@@ -216,6 +227,8 @@ static void sync_debug_state(struct pkvm_hyp_vcpu *hyp_vcpu)static void flush_hyp_vcpu (struct pkvm_hyp_vcpu *hyp_vcpu) {struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu; +hyp_entry_exit_handler_fn ec_handler; +u8 esr_ec;fpsimd_sve_flush ();flush_debug_state (hyp_vcpu);
@@ -228,6 +241,7 @@ static void flush_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu)if (!pkvm_hyp_vcpu_is_protected (hyp_vcpu)) {if (vcpu_get_flag (host_vcpu,PKVM_HOST_STATE_DIRTY ))flush_hyp_vcpu_state (hyp_vcpu); + hyp_vcpu->vcpu.arch.iflags = host_vcpu->arch.iflags; }else { hyp_vcpu->vcpu.arch.ctxt = host_vcpu->arch.ctxt; }
@@ -245,16 +259,31 @@ static void flush_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu) hyp_vcpu->vcpu.arch.hcr_el2 |=READ_ONCE (host_vcpu->arch.hcr_el2) & (HCR_TWI |HCR_TWE |HCR_VSE ); - hyp_vcpu->vcpu.arch.iflags = host_vcpu->arch.iflags; - hyp_vcpu->vcpu.arch.vsesr_el2 = host_vcpu->arch.vsesr_el2;flush_hyp_vgic_state (hyp_vcpu); hyp_vcpu->vcpu.arch.pid = host_vcpu->arch.pid; + +switch (ARM_EXCEPTION_CODE (hyp_vcpu->exit_code)) { +case ARM_EXCEPTION_IRQ : +case ARM_EXCEPTION_EL1_SERROR : +case ARM_EXCEPTION_IL : +break ; +case ARM_EXCEPTION_TRAP : + esr_ec =ESR_ELx_EC (kvm_vcpu_get_esr (&hyp_vcpu->vcpu)); + ec_handler = entry_hyp_vm_handlers[esr_ec]; +if (ec_handler) +ec_handler (hyp_vcpu); +break ; +default : +BUG (); + } + + hyp_vcpu->exit_code =0 ; } -static void sync_hyp_vcpu (struct pkvm_hyp_vcpu *hyp_vcpu) +static void sync_hyp_vcpu (struct pkvm_hyp_vcpu *hyp_vcpu,u32 exit_reason) {struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu;
@@ -281,6 +310,8 @@ static void sync_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu) host_vcpu->arch.hcr_el2 |= hyp_vcpu->vcpu.arch.hcr_el2 &HCR_VSE ;sync_hyp_vgic_state (hyp_vcpu); + + hyp_vcpu->exit_code = exit_reason; }static void handle___pkvm_vcpu_load (struct kvm_cpu_context *host_ctxt)
@@ -395,7 +426,7 @@ static void handle___kvm_vcpu_run(struct kvm_cpu_context *host_ctxt) ret =__kvm_vcpu_run (&hyp_vcpu->vcpu); -sync_hyp_vcpu (hyp_vcpu); +sync_hyp_vcpu (hyp_vcpu, ret); }else {/* The host is fully trusted, run its vCPU directly. */ fpsimd_lazy_switch_to_guest (host_vcpu);
Signature · 1 line
2.39.5
Fuad Tabba
reviewerPATCH v4 6/18 KVM: arm64: Skip fixed-feature state flush for protected vCPUs
In reply to Fuad Tabbaflush_hyp_vcpu() copies the host's mdcr_el2, TWI/TWE and debug state into the hyp vCPU on every entry, so for a protected vCPU the host's trap configuration overrides the one pkvm_vcpu_init_traps() computed at EL2. Move the mdcr_el2 and TWI/TWE copies into the non-protected branch, and return early from flush_debug_state() and sync_debug_state() for a protected vCPU: its debug registers are hypervisor-owned, and it takes the host's TWI/TWE at vCPU load. MDCR_EL2.TDA traps only the guest's own accesses; it doesn't stop the world switch from loading host-supplied breakpoints and MDSCR_EL1.
EL2's mdcr_el2 starts from 0, and HPMN == 0 is reserved without FEAT_HPMN0. Set HPMN at vCPU load to PMCR_EL0.N on a PE whose ID_AA64DFR0_EL1 implements PMUv3, the value the host's copy carried, read as kvm_init_host_debug_data() reads it: the PMUv3 cpucap is system-wide and off without CONFIG_HW_PERF_EVENTS. A protected guest's PMU accesses trap regardless.
--- arch/arm64/kvm/hyp/nvhe/hyp-main.c | 34 ++++++++++++++++++++++-------- arch/arm64/kvm/hyp/nvhe/pkvm.c | 2 +- 2 files changed, 26 insertions(+), 10 deletions(-)
arch/arm64/kvm/hyp/nvhe/hyp-main.c +25 -9
@@ -200,6 +200,9 @@ static void flush_debug_state(struct pkvm_hyp_vcpu *hyp_vcpu) {struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu; +if (pkvm_hyp_vcpu_is_protected (hyp_vcpu)) +return ; + hyp_vcpu->vcpu.arch.debug_owner = host_vcpu->arch.debug_owner;if (kvm_guest_owns_debug_regs (&hyp_vcpu->vcpu)) {
@@ -218,6 +221,9 @@ static void sync_debug_state(struct pkvm_hyp_vcpu *hyp_vcpu) {struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu; +if (pkvm_hyp_vcpu_is_protected (hyp_vcpu)) +return ; +if (kvm_guest_owns_debug_regs (&hyp_vcpu->vcpu)) host_vcpu->arch.vcpu_debug_state = hyp_vcpu->vcpu.arch.vcpu_debug_state;else if (kvm_host_owns_debug_regs (&hyp_vcpu->vcpu))
@@ -241,6 +247,12 @@ static void flush_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu)if (!pkvm_hyp_vcpu_is_protected (hyp_vcpu)) {if (vcpu_get_flag (host_vcpu,PKVM_HOST_STATE_DIRTY ))flush_hyp_vcpu_state (hyp_vcpu); + + hyp_vcpu->vcpu.arch.hcr_el2 &= ~(HCR_TWI |HCR_TWE ); + hyp_vcpu->vcpu.arch.hcr_el2 |=READ_ONCE (host_vcpu->arch.hcr_el2) & + (HCR_TWI |HCR_TWE ); + + hyp_vcpu->vcpu.arch.mdcr_el2 = host_vcpu->arch.mdcr_el2; hyp_vcpu->vcpu.arch.iflags = host_vcpu->arch.iflags; }else { hyp_vcpu->vcpu.arch.ctxt = host_vcpu->arch.ctxt;
@@ -249,17 +261,13 @@ static void flush_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu)/* __hyp_running_vcpu must be NULL in a guest context. */ hyp_vcpu->vcpu.arch.ctxt.__hyp_running_vcpu =NULL ; - hyp_vcpu->vcpu.arch.mdcr_el2 = host_vcpu->arch.mdcr_el2;/* -* HCR_EL2.VSE is host-owned (a pending virtual SError to inject), not a -* trap-control bit, so it must flow to the hyp vCPU alongside TWI/TWE -* for the vSError to be delivered. sync_hyp_vcpu() reflects it back. +* A host-injected vSError is masked by the guest's own PSTATE.A, so it +* applies to protected guests too. */ - hyp_vcpu->vcpu.arch.hcr_el2 &= ~(HCR_TWI |HCR_TWE |HCR_VSE ); - hyp_vcpu->vcpu.arch.hcr_el2 |=READ_ONCE (host_vcpu->arch.hcr_el2) & - (HCR_TWI |HCR_TWE |HCR_VSE ); - - hyp_vcpu->vcpu.arch.vsesr_el2 = host_vcpu->arch.vsesr_el2; + hyp_vcpu->vcpu.arch.hcr_el2 &= ~HCR_VSE ; + hyp_vcpu->vcpu.arch.hcr_el2 |=READ_ONCE (host_vcpu->arch.hcr_el2) &HCR_VSE ; + hyp_vcpu->vcpu.arch.vsesr_el2 = host_vcpu->arch.vsesr_el2;flush_hyp_vgic_state (hyp_vcpu);
@@ -326,9 +334,17 @@ static void handle___pkvm_vcpu_load(struct kvm_cpu_context *host_ctxt)return ;if (pkvm_hyp_vcpu_is_protected (hyp_vcpu)) { +u64 dfr0 =read_sysreg (id_aa64dfr0_el1); +/* Propagate WFx trapping flags */ hyp_vcpu->vcpu.arch.hcr_el2 &= ~(HCR_TWE |HCR_TWI ); hyp_vcpu->vcpu.arch.hcr_el2 |= hcr_el2 & (HCR_TWE |HCR_TWI ); + +/* HPMN == 0 is reserved without FEAT_HPMN0. */ +if (pmuv3_implemented (SYS_FIELD_GET (ID_AA64DFR0_EL1 , PMUVer, dfr0))) +u64p_replace_bits (&hyp_vcpu->vcpu.arch.mdcr_el2, +FIELD_GET (ARMV8_PMU_PMCR_N ,read_sysreg (pmcr_el0)), +MDCR_EL2_HPMN ); }else {memcpy (&hyp_vcpu->vcpu.arch.fgt, hyp_vcpu->host_vcpu->arch.fgt,sizeof (hyp_vcpu->vcpu.arch.fgt));
arch/arm64/kvm/hyp/nvhe/pkvm.c +1 -1
@@ -102,7 +102,7 @@ static void pvm_init_traps_mdcr(struct kvm_vcpu *vcpu)if (!kvm_has_feat (kvm,ID_AA64DFR0_EL1 , PMUVer,IMP )) { val |=MDCR_EL2_TPM |MDCR_EL2_TPMCR ; - val &= ~(MDCR_EL2_HPME |MDCR_EL2_MTPME |MDCR_EL2_HPMN_MASK ); + val &= ~(MDCR_EL2_HPME |MDCR_EL2_MTPME ); }if (!kvm_has_feat (kvm,ID_AA64DFR0_EL1 , DebugVer,IMP ))
Signature · 1 line
2.39.5
[email protected]
AI reviewRe: [PATCH v4 06/18] KVM: arm64: Skip fixed-feature state flush for protected vCPUs
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] kvm: arm64: host hardware breakpoints leak into protected guests --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] kvm: arm64: host hardware breakpoints leak into protected guests --
commit 678dfb9fd1576272693dba360820510bcb1ec938
Author: Fuad Tabba <[email protected]>
KVM: arm64: Skip fixed-feature state flush for protected vCPUs
This commit moves the mdcr_el2 and TWI/TWE copies into the non-protected branch. It returns early from flush_debug_state() and sync_debug_state() for protected vCPUs, as their debug registers are hypervisor-owned, and configures HPMN at vCPU load.
arch/arm64/kvm/hyp/nvhe/hyp-main.c static void flush_debug_state(struct pkvm_hyp_vcpu *hyp_vcpu) · line 205
Quoted 11 lines
arch/arm64/kvm/hyp/nvhe/hyp-main.c +3
-0@@ -200,6 +200,9 @@ static void flush_debug_state(struct pkvm_hyp_vcpu *hyp_vcpu) {struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu; +if (pkvm_hyp_vcpu_is_protected (hyp_vcpu)) +return ; +
[Severity: High]
Could this early return cause a regression by allowing the host's hardware breakpoints to leak into the protected guest?
By returning early here, the hyp vCPU's debug_owner remains VCPU_DEBUG_FREE (0) for protected VMs.
During the world switch, this causes the hypervisor to skip context-switching the hardware debug registers because kvm_debug_regs_in_use() evaluates to false:
arch/arm64/kvm/hyp/include/hyp/debug-sr.h:__debug_switch_to_guest_common() {
...
if (!kvm_debug_regs_in_use(vcpu))
return;
...
}As a result, the host's physical breakpoints remain active in hardware.
If the guest implements DebugVer, MDCR_EL2.TDA is not set:
arch/arm64/kvm/hyp/nvhe/pkvm.c:pvm_init_traps_mdcr() {
...
if (!kvm_has_feat(kvm, ID_AA64DFR0_EL1, DebugVer, IMP))
val |= MDCR_EL2_TDRA | MDCR_EL2_TDA;
...
}This gives the guest untrapped access to MDSCR_EL1. If the guest sets MDSCR_EL1.MDE=1, the host's hardware breakpoints could fire inside the guest at EL1, circumventing guest isolation.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
Fuad Tabba
reviewerRe: [PATCH v4 06/18] KVM: arm64: Skip fixed-feature state flush for protected vCPUs
Quoted 3 lines · On Thu, 1 Oct 2026 at 15:19, <[email protected]> wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] kvm: arm64: host hardware breakpoints leak into protected guests
No hardware breakpoint/watchpoint support for protected guests yet. A protected guest sees DebugVer as 0.
/fuad
Quoted 58 lines
--
commit
678dfb9fd1576272693dba360820510bcb1ec938
Author: Fuad Tabba <[email protected]>KVM: arm64: Skip fixed-feature state flush for protected vCPUs
This commit moves the
mdcr_el2and TWI/TWE copies into the non-protected branch. It returns early fromflush_debug_state()andsync_debug_state()for protected vCPUs, as their debug registers are hypervisor-owned, and configures HPMN at vCPU load.arch/arm64/kvm/hyp/nvhe/hyp-main.c static void flush_debug_state(struct pkvm_hyp_vcpu *hyp_vcpu) · line 205
Quoted 11 lines
arch/arm64/kvm/hyp/nvhe/hyp-main.c +3
-0@@ -200,6 +200,9 @@ static void flush_debug_state(struct pkvm_hyp_vcpu *hyp_vcpu) {struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu; +if (pkvm_hyp_vcpu_is_protected (hyp_vcpu)) +return ; +[Severity: High]
Could this early return cause a regression by allowing the host's hardware breakpoints to leak into the protected guest?By returning early here, the hyp vCPU's
debug_ownerremainsVCPU_DEBUG_FREE(0) for protected VMs.During the world switch, this causes the hypervisor to skip context-switching the hardware debug registers because
kvm_debug_regs_in_use()evaluates to false:arch/arm64/kvm/hyp/include/hyp/debug-sr.h:__debug_switch_to_guest_common() { ... if (!kvm_debug_regs_in_use(vcpu)) return; ... }As a result, the host's physical breakpoints remain active in hardware.
If the guest implements DebugVer, MDCR_EL2.TDA is not set:
arch/arm64/kvm/hyp/nvhe/pkvm.c:pvm_init_traps_mdcr() { ... if (!kvm_has_feat(kvm, ID_AA64DFR0_EL1, DebugVer, IMP)) val |= MDCR_EL2_TDRA | MDCR_EL2_TDA; ... }This gives the guest untrapped access to
MDSCR_EL1. If the guest sets MDSCR_EL1.MDE=1, the host's hardware breakpoints could fire inside the guest at EL1, circumventing guest isolation.--
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
Fuad Tabba
reviewerPATCH v4 7/18 KVM: arm64: Add {flush,sync}_hyp_timer_state() primitives
In reply to Fuad TabbaFrom: Marc Zyngier <[email protected]>
A protected guest's virtual timer is programmed from EL2's own copy of the state and read back there, so the host can't change the guest's deadline. cntvoff_el2 is zeroed, so the guest's virtual time is the physical time.
The flush also preserves the guest's CNTV and CNTP CVAL/CTL across the host's full-context copy into the hyp vCPU, until the per-EC marshalling patch removes that copy.
--- arch/arm64/kvm/hyp/nvhe/hyp-main.c | 41 ++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+)
arch/arm64/kvm/hyp/nvhe/hyp-main.c +41 -0
@@ -154,6 +154,35 @@ static void sync_hyp_vgic_state(struct pkvm_hyp_vcpu *hyp_vcpu) host_cpu_if->vgic_lr[i] = hyp_cpu_if->vgic_lr[i]; } +static void flush_hyp_timer_state (struct pkvm_hyp_vcpu *hyp_vcpu) +{ +struct kvm_vcpu *vcpu = &hyp_vcpu->vcpu; + +if (!pkvm_hyp_vcpu_is_protected (hyp_vcpu)) +return ; + +/* A hyp vcpu has no offset, and sees vtime == ptime. */ +write_sysreg (0 , cntvoff_el2); +write_sysreg_el0 (__vcpu_sys_reg (vcpu,CNTV_CVAL_EL0 ),SYS_CNTV_CVAL ); +isb (); +write_sysreg_el0 (__vcpu_sys_reg (vcpu,CNTV_CTL_EL0 ),SYS_CNTV_CTL ); +} + +static void sync_hyp_timer_state (struct pkvm_hyp_vcpu *hyp_vcpu) +{ +struct kvm_vcpu *vcpu = &hyp_vcpu->vcpu; + +if (!pkvm_hyp_vcpu_is_protected (hyp_vcpu)) +return ; + +/* +* Preserve the vtimer state so that it is always correct, +* even if the host tries to make a mess. +*/ +__vcpu_assign_sys_reg (vcpu,CNTV_CVAL_EL0 ,read_sysreg_el0 (SYS_CNTV_CVAL )); +__vcpu_assign_sys_reg (vcpu,CNTV_CTL_EL0 ,read_sysreg_el0 (SYS_CNTV_CTL )); +} +static void __copy_vcpu_state (const struct kvm_vcpu *from_vcpu,struct kvm_vcpu *to_vcpu) {
@@ -255,7 +284,17 @@ static void flush_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu) hyp_vcpu->vcpu.arch.mdcr_el2 = host_vcpu->arch.mdcr_el2; hyp_vcpu->vcpu.arch.iflags = host_vcpu->arch.iflags; }else { +u64 v_cval = hyp_vcpu->vcpu.arch.ctxt.sys_regs[CNTV_CVAL_EL0 ]; +u64 v_ctl = hyp_vcpu->vcpu.arch.ctxt.sys_regs[CNTV_CTL_EL0 ]; +u64 p_cval = hyp_vcpu->vcpu.arch.ctxt.sys_regs[CNTP_CVAL_EL0 ]; +u64 p_ctl = hyp_vcpu->vcpu.arch.ctxt.sys_regs[CNTP_CTL_EL0 ]; + hyp_vcpu->vcpu.arch.ctxt = host_vcpu->arch.ctxt; + + hyp_vcpu->vcpu.arch.ctxt.sys_regs[CNTV_CVAL_EL0 ] = v_cval; + hyp_vcpu->vcpu.arch.ctxt.sys_regs[CNTV_CTL_EL0 ] = v_ctl; + hyp_vcpu->vcpu.arch.ctxt.sys_regs[CNTP_CVAL_EL0 ] = p_cval; + hyp_vcpu->vcpu.arch.ctxt.sys_regs[CNTP_CTL_EL0 ] = p_ctl; }/* __hyp_running_vcpu must be NULL in a guest context. */
@@ -270,6 +309,7 @@ static void flush_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu) hyp_vcpu->vcpu.arch.vsesr_el2 = host_vcpu->arch.vsesr_el2;flush_hyp_vgic_state (hyp_vcpu); +flush_hyp_timer_state (hyp_vcpu); hyp_vcpu->vcpu.arch.pid = host_vcpu->arch.pid;
@@ -318,6 +358,7 @@ static void sync_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu, u32 exit_reason) host_vcpu->arch.hcr_el2 |= hyp_vcpu->vcpu.arch.hcr_el2 &HCR_VSE ;sync_hyp_vgic_state (hyp_vcpu); +sync_hyp_timer_state (hyp_vcpu); hyp_vcpu->exit_code = exit_reason; }
Signature · 1 line
2.39.5
Fuad Tabba
reviewerPATCH v4 8/18 KVM: arm64: Add system register reset framework for protected VMs
In reply to Fuad TabbaAdd kvm_reset_pvm_sys_regs() and the pvm_sys_reg_reset_vals[] table that drives it, and call it from init_pkvm_hyp_vcpu() for protected vCPUs. The table holds the registers KVM resets that EL2 loads for a protected vCPU: the ones __sysreg_restore_state_nvhe() restores, and CNTV_CTL_EL0, which flush_hyp_timer_state() loads from a copy the host's kvm_timer_vcpu_reset() doesn't write. The ptrauth keys, which entry.S loads, are not in the table: sys_regs.c resets them to UNKNOWN, and a vCPU that CPU_ON brings back keeps the keys the guest last set, one UNKNOWN value in place of another.
A register that resets to 0 still takes an entry: a later patch calls kvm_reset_pvm_sys_regs() again for PSCI CPU_ON at EL2, on a context that has run.
A poison value stands in where sys_regs.c resets to UNKNOWN, and for VBAR_EL1 and CONTEXTIDR_EL1 in place of the 0 it uses.
MPIDR_EL1 is derived from vcpu_id, as for any KVM guest.
AMAIR_EL1 resets to 0 where sys_regs.c reads the hardware: the host writes it freely before the hypercall, and it's loaded on every guest entry.
Until the per-EC marshalling patch removes the host's full-context copy from flush_hyp_vcpu(), that copy overwrites these values on every entry. It preserves CNTV_CTL_EL0, whose entry writes the zero the page already holds. So this patch has no observable effect on its own.
No functional change intended.
--- arch/arm64/kvm/hyp/include/nvhe/pkvm.h | 1 + arch/arm64/kvm/hyp/nvhe/pkvm.c | 5 +++ arch/arm64/kvm/hyp/nvhe/sys_regs.c | 60 ++++++++++++++++++++++++++ 3 files changed, 66 insertions(+)
arch/arm64/kvm/hyp/include/nvhe/pkvm.h +1 -0
@@ -97,6 +97,7 @@ bool kvm_handle_pvm_hvc64(struct kvm_vcpu *vcpu, u64 *exit_code);bool kvm_handle_pvm_sysreg (struct kvm_vcpu *vcpu,u64 *exit_code);bool kvm_handle_pvm_restricted (struct kvm_vcpu *vcpu,u64 *exit_code);void kvm_init_pvm_id_regs (struct kvm_vcpu *vcpu); +void kvm_reset_pvm_sys_regs (struct kvm_vcpu *vcpu);int kvm_check_pvm_sysreg_table (void );#endif /* __ARM64_KVM_NVHE_PKVM_H__ */
arch/arm64/kvm/hyp/nvhe/pkvm.c +5 -0
@@ -588,6 +588,11 @@ static int init_pkvm_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu, goto done; ret =pkvm_vcpu_init_sve (hyp_vcpu, host_vcpu); +if (ret) + goto done; + +if (pkvm_hyp_vcpu_is_protected (hyp_vcpu)) +kvm_reset_pvm_sys_regs (&hyp_vcpu->vcpu); done:if (ret)unpin_host_vcpu (host_vcpu);
arch/arm64/kvm/hyp/nvhe/sys_regs.c +60 -0
66 lines not on this page · raw message
Signature · 1 line
2.39.5
Fuad Tabba
reviewerPATCH v4 9/18 KVM: arm64: Implement HVC handling for protected guests at EL2
In reply to Fuad TabbaExtend kvm_handle_pvm_hvc64() to handle SMCCC_VERSION, SMCCC_ARCH_FEATURES and the vendor hypervisor call UID at EL2, so these queries don't reach the host. ARCH_FEATURES is mandatory from SMCCC 1.1, the version EL2 implements: it returns SUCCESS for itself and for SMCCC_VERSION, NOT_SUPPORTED for anything else, and handles the ARCH_WORKAROUND_1/2/3 queries as the host does, from a copy of the host's mitigation state taken at init with the sanitised ID registers. The workaround calls themselves are already handled on hyp entry.
--- arch/arm64/include/asm/kvm_hyp.h | 4 ++ arch/arm64/kvm/arm.c | 4 ++ arch/arm64/kvm/hyp/nvhe/pkvm.c | 68 ++++++++++++++++++++++++++++++++ 3 files changed, 76 insertions(+)
arch/arm64/include/asm/kvm_hyp.h +4 -0
@@ -10,6 +10,7 @@#include <linux/compiler.h> #include <linux/kvm_host.h> #include <asm/alternative.h> +#include <asm/spectre.h> #include <asm/sysreg.h> DECLARE_PER_CPU(struct kvm_cpu_context, kvm_hyp_ctxt);
@@ -150,6 +151,9 @@ extern u64 kvm_nvhe_sym(id_aa64smfr0_el1_sys_val);extern unsigned long kvm_nvhe_sym (__icache_flags );extern unsigned int kvm_nvhe_sym (kvm_arm_vmid_bits ); +extern enum mitigation_state kvm_nvhe_sym (spectre_v2_state ); +extern enum mitigation_state kvm_nvhe_sym (spectre_v4_state ); +extern enum mitigation_state kvm_nvhe_sym (spectre_bhb_state );extern unsigned int kvm_nvhe_sym (kvm_host_sve_max_vl );extern unsigned long kvm_nvhe_sym (hyp_nr_cpus );extern unsigned int kvm_nvhe_sym (hyp_gicv3_nr_lr );
arch/arm64/kvm/arm.c +4 -0
@@ -2622,6 +2622,10 @@ static void kvm_hyp_init_symbols(void)kvm_nvhe_sym (__icache_flags) = __icache_flags;kvm_nvhe_sym (kvm_arm_vmid_bits) = kvm_arm_vmid_bits; +kvm_nvhe_sym (spectre_v2_state) =arm64_get_spectre_v2_state (); +kvm_nvhe_sym (spectre_v4_state) =arm64_get_spectre_v4_state (); +kvm_nvhe_sym (spectre_bhb_state) =arm64_get_spectre_bhb_state (); +/* Propagate the FGT state to the nVHE side */ kvm_nvhe_sym (hfgrtr_masks) = hfgrtr_masks;kvm_nvhe_sym (hfgwtr_masks) = hfgwtr_masks;
arch/arm64/kvm/hyp/nvhe/pkvm.c +68 -0
88 lines not on this page · raw message
Signature · 1 line
2.39.5
Re: [PATCH v4 09/18] KVM: arm64: Implement HVC handling for protected guests at EL2
Hi Faud,
I have some drive-by comments for this patch, having stumbled upon it while poking around in SSBS code today...
Quoted 43 lines · Fuad Tabba
Extend
kvm_handle_pvm_hvc64()to handleSMCCC_VERSION,SMCCC_ARCH_FEATURESand the vendor hypervisor call UID at EL2, so these queries don't reach the host.ARCH_FEATURESis mandatory from SMCCC 1.1, the version EL2 implements: it returns SUCCESS for itself and forSMCCC_VERSION,NOT_SUPPORTEDfor anything else, and handles the ARCH_WORKAROUND_1/2/3 queries as the host does, from a copy of the host's mitigation state taken at init with the sanitised ID registers. The workaround calls themselves are already handled on hyp entry.Signed-off-by Fuad Tabba <[email protected]>--- arch/arm64/include/asm/kvm_hyp.h | 4 ++ arch/arm64/kvm/arm.c | 4 ++ arch/arm64/kvm/hyp/nvhe/pkvm.c | 68 ++++++++++++++++++++++++++++++++ 3 files changed, 76 insertions(+)arch/arm64/include/asm/kvm_hyp.h +4
-0@@ -10,6 +10,7 @@#include <linux/compiler.h> #include <linux/kvm_host.h> #include <asm/alternative.h> +#include <asm/spectre.h> #include <asm/sysreg.h> DECLARE_PER_CPU(struct kvm_cpu_context, kvm_hyp_ctxt);@@ -150,6 +151,9 @@ extern u64 kvm_nvhe_sym(id_aa64smfr0_el1_sys_val);extern unsigned long kvm_nvhe_sym (__icache_flags );extern unsigned int kvm_nvhe_sym (kvm_arm_vmid_bits ); +extern enum mitigation_state kvm_nvhe_sym (spectre_v2_state ); +extern enum mitigation_state kvm_nvhe_sym (spectre_v4_state ); +extern enum mitigation_state kvm_nvhe_sym (spectre_bhb_state );extern unsigned int kvm_nvhe_sym (kvm_host_sve_max_vl );extern unsigned long kvm_nvhe_sym (hyp_nr_cpus );extern unsigned int kvm_nvhe_sym (hyp_gicv3_nr_lr );arch/arm64/kvm/arm.c +0
-0
arch/arm64/kvm/arm.c static void kvm_hyp_init_symbols(void) · line 2627
Quoted 6 lines
arch/arm64/kvm/arm.c +3
-0@@ -2622,6 +2622,10 @@ static void kvm_hyp_init_symbols(void)kvm_nvhe_sym (__icache_flags) = __icache_flags;kvm_nvhe_sym (kvm_arm_vmid_bits) = kvm_arm_vmid_bits; +kvm_nvhe_sym (spectre_v2_state) =arm64_get_spectre_v2_state (); +kvm_nvhe_sym (spectre_v4_state) =arm64_get_spectre_v4_state (); +kvm_nvhe_sym (spectre_bhb_state) =arm64_get_spectre_bhb_state ();
Is it really correct to snapshot this state then use it at runtime assuming its still accurate? See below...
Quoted 58 lines
arch/arm64/kvm/arm.c +1
-0+/* Propagate the FGT state to the nVHE side */ kvm_nvhe_sym (hfgrtr_masks) = hfgrtr_masks;kvm_nvhe_sym (hfgwtr_masks) = hfgwtr_masks;arch/arm64/kvm/hyp/nvhe/pkvm.c +31
-0@@ -10,12 +10,18 @@#include <linux/mm.h> #include <asm/kvm_emulate.h> +#include <asm/spectre.h> #include <nvhe/mem_protect.h> #include <nvhe/memory.h> #include <nvhe/pkvm.h> #include <nvhe/trap_handler.h> +/* The host's Spectre mitigation state, for SMCCC_ARCH_FEATURES. */ +enum mitigation_state spectre_v2_state; +enum mitigation_state spectre_v4_state; +enum mitigation_state spectre_bhb_state; +/* Used by icache_is_aliasing(). */ unsigned long __icache_flags;@@ -1181,8 +1187,70 @@ bool kvm_handle_pvm_hvc64(struct kvm_vcpu *vcpu, u64 *exit_code) {u64 val[4 ] = {SMCCC_RET_INVALID_PARAMETER };bool handled = true; +u32 feature; +uuid_t uuid;switch (smccc_get_function (vcpu)) { +case ARM_SMCCC_VERSION_FUNC_ID : +/* Nothing to be handled by the host. Go back to the guest. */ + val[0 ] =ARM_SMCCC_VERSION_1_1 ; +break ; +case ARM_SMCCC_ARCH_FEATURES_FUNC_ID : +/* The workaround calls themselves are handled on hyp entry. */ + val[0 ] =SMCCC_RET_NOT_SUPPORTED ; + feature =smccc_get_arg1 (vcpu); +switch (feature) { +case ARM_SMCCC_VERSION_FUNC_ID : +case ARM_SMCCC_ARCH_FEATURES_FUNC_ID : + val[0 ] =SMCCC_RET_SUCCESS ; +break ; +case ARM_SMCCC_ARCH_WORKAROUND_1 : +switch (spectre_v2_state) { +case SPECTRE_VULNERABLE : +break ; +case SPECTRE_MITIGATED : + val[0 ] =SMCCC_RET_SUCCESS ; +break ; +case SPECTRE_UNAFFECTED : + val[0 ] =SMCCC_ARCH_WORKAROUND_RET_UNAFFECTED ; +break ;
arch/arm64/kvm/hyp/nvhe/pkvm.c bool kvm_handle_pvm_hvc64(struct kvm_vcpu *vcpu, u64 *exit_code) · line 1226
Quoted 10 lines
arch/arm64/kvm/hyp/nvhe/pkvm.c +10
-0+ } +break ; +case ARM_SMCCC_ARCH_WORKAROUND_2 : +switch (spectre_v4_state) { +case SPECTRE_VULNERABLE : +break ; +case SPECTRE_MITIGATED : +/* With SSBS advertised the guest's default is safe. */ +if (kvm_has_feat (vcpu->kvm,ID_AA64PFR1_EL1 ,SSBS ,IMP )) +break ;
For this case (and the break immediately above) you return val[0] = SMCCC_RET_INVALID_PARAMETER (from init at top of this function). But the kernel's kvm_smccc_call_handler() returns val[0] = SMCCC_RET_NOT_SUPPORTED in these cases (which seems more likely to be correct). Does this violate the ABI?
arch/arm64/kvm/hyp/nvhe/pkvm.c bool kvm_handle_pvm_hvc64(struct kvm_vcpu *vcpu, u64 *exit_code) · line 1229
Quoted 3 lines
arch/arm64/kvm/hyp/nvhe/pkvm.c +3
-0+ fallthrough; +case SPECTRE_UNAFFECTED : + val[0 ] =SMCCC_RET_NOT_REQUIRED ;
Is it possible that the (compromised) kernel could have disabled the mitigation after the state snapshot was taken? I.e. spectre_v4_state == SPECTRE_MITIGATED and the VM doesn't advertise SSBS so it returns SMCCC_RET_NOT_REQUIRED here. But the kernel has since turned off the mitigation?
Thanks,
Ryan
Quoted 27 lines
arch/arm64/kvm/hyp/nvhe/pkvm.c +24
-0+break ; + } +break ; +case ARM_SMCCC_ARCH_WORKAROUND_3 : +switch (spectre_bhb_state) { +case SPECTRE_VULNERABLE : +break ; +case SPECTRE_MITIGATED : + val[0 ] =SMCCC_RET_SUCCESS ; +break ; +case SPECTRE_UNAFFECTED : + val[0 ] =SMCCC_ARCH_WORKAROUND_RET_UNAFFECTED ; +break ; + } +break ; + } +break ; +case ARM_SMCCC_VENDOR_HYP_CALL_UID_FUNC_ID : + uuid =ARM_SMCCC_VENDOR_HYP_UID_KVM ; + val[0 ] =smccc_uuid_to_reg (&uuid,0 ); + val[1 ] =smccc_uuid_to_reg (&uuid,1 ); + val[2 ] =smccc_uuid_to_reg (&uuid,2 ); + val[3 ] =smccc_uuid_to_reg (&uuid,3 ); +break ;case ARM_SMCCC_VENDOR_HYP_KVM_FEATURES_FUNC_ID : val[0 ] =BIT (ARM_SMCCC_KVM_FUNC_FEATURES ); val[0 ] |=BIT (ARM_SMCCC_KVM_FUNC_HYP_MEMINFO );
Fuad Tabba
reviewerRe: [PATCH v4 09/18] KVM: arm64: Implement HVC handling for protected guests at EL2
Hi Ryan,
On Wed, 11 Nov 2026 13:59:17 +0000, Ryan Roberts <[email protected]> wrote:
[...]
Quoted 1 line
Quoted 1 line
arch/arm64/kvm/arm.c +0
-0
[...]
Quoted 10 lines
arch/arm64/kvm/arm.c static void kvm_hyp_init_symbols(void) · line 2627
Quoted 7 lines
arch/arm64/kvm/arm.c +3
-0@@ -2622,6 +2622,10 @@ static void kvm_hyp_init_symbols(void)kvm_nvhe_sym (__icache_flags) = __icache_flags;kvm_nvhe_sym (kvm_arm_vmid_bits) = kvm_arm_vmid_bits; +kvm_nvhe_sym (spectre_v2_state) =arm64_get_spectre_v2_state (); +kvm_nvhe_sym (spectre_v4_state) =arm64_get_spectre_v4_state (); +kvm_nvhe_sym (spectre_bhb_state) =arm64_get_spectre_bhb_state ();Is it really correct to snapshot this state then use it at runtime assuming its still accurate? See below...
These can't change after this point: update_mitigation_state() doesn't update a state once system_capabilities_finalized() is true, and kvm_arm_init() runs after that.
[...]
Quoted 1 line
Quoted 1 line
arch/arm64/kvm/hyp/nvhe/pkvm.c +0
-0
[...]
Quoted 1 line
Quoted 1 line
arch/arm64/kvm/hyp/nvhe/pkvm.c +0
-0@@ -1181,8 +1187,70 @@ bool kvm_handle_pvm_hvc64(struct kvm_vcpu *vcpu, u64 *exit_code)
[...]
Quoted 3 lines
Quoted 3 lines
arch/arm64/kvm/hyp/nvhe/pkvm.c +3
-0+case ARM_SMCCC_ARCH_FEATURES_FUNC_ID : +/* The workaround calls themselves are handled on hyp entry. */ + val[0 ] =SMCCC_RET_NOT_SUPPORTED ;
[...]
Quoted 9 lines
arch/arm64/kvm/hyp/nvhe/pkvm.c bool kvm_handle_pvm_hvc64(struct kvm_vcpu *vcpu, u64 *exit_code) · line 1226
Quoted 8 lines
arch/arm64/kvm/hyp/nvhe/pkvm.c +8
-0+case ARM_SMCCC_ARCH_WORKAROUND_2 : +switch (spectre_v4_state) { +case SPECTRE_VULNERABLE : +break ; +case SPECTRE_MITIGATED : +/* With SSBS advertised the guest's default is safe. */ +if (kvm_has_feat (vcpu->kvm,ID_AA64PFR1_EL1 ,SSBS ,IMP )) +break ;
Quoted 4 lines
For this case (and the break immediately above) you return val[0] =
SMCCC_RET_INVALID_PARAMETER(from init at top of this function). But the kernel'skvm_smccc_call_handler()returns val[0] =SMCCC_RET_NOT_SUPPORTEDin these cases (which seems more likely to be correct). Does this violate the ABI?
Both return NOT_SUPPORTED. val[0] is set to it at the top of the ARM_SMCCC_ARCH_FEATURES_FUNC_ID case.
Quoted 9 lines
arch/arm64/kvm/hyp/nvhe/pkvm.c bool kvm_handle_pvm_hvc64(struct kvm_vcpu *vcpu, u64 *exit_code) · line 1229
Quoted 4 lines
arch/arm64/kvm/hyp/nvhe/pkvm.c +3
-0+ fallthrough; +case SPECTRE_UNAFFECTED : + val[0 ] =SMCCC_RET_NOT_REQUIRED ;Is it possible that the (compromised) kernel could have disabled the mitigation after the state snapshot was taken? I.e.
spectre_v4_state==SPECTRE_MITIGATEDand the VM doesn't advertise SSBS so it returnsSMCCC_RET_NOT_REQUIREDhere. But the kernel has since turned off the mitigation?
Yes, where the mitigation is the firmware one. The host can disable it with an ARM_SMCCC_ARCH_WORKAROUND_2 call, which EL2 forwards, and EL2 doesn't enable it again before running the guest. With TF-A that's only on a Cortex-A76 without SSBS, the only CPU it implements the dynamic workaround for.
I'll send out a fix. Thanks for spotting this!
Cheers,
/fuad
Re: [PATCH v4 09/18] KVM: arm64: Implement HVC handling for protected guests at EL2
Quoted 39 lines · Fuad Tabba
Hi Ryan,
On Wed, 11 Nov 2026 13:59:17 +0000, Ryan Roberts <[email protected]> wrote:
[...]Quoted 1 line
Quoted 1 line
arch/arm64/kvm/arm.c +0
-0[...]
Quoted 10 lines
arch/arm64/kvm/arm.c static void kvm_hyp_init_symbols(void) · line 2627
Quoted 7 lines
arch/arm64/kvm/arm.c +3
-0@@ -2622,6 +2622,10 @@ static void kvm_hyp_init_symbols(void)kvm_nvhe_sym (__icache_flags) = __icache_flags;kvm_nvhe_sym (kvm_arm_vmid_bits) = kvm_arm_vmid_bits; +kvm_nvhe_sym (spectre_v2_state) =arm64_get_spectre_v2_state (); +kvm_nvhe_sym (spectre_v4_state) =arm64_get_spectre_v4_state (); +kvm_nvhe_sym (spectre_bhb_state) =arm64_get_spectre_bhb_state ();Is it really correct to snapshot this state then use it at runtime assuming its still accurate? See below...
These can't change after this point:
update_mitigation_state()doesn't update a state oncesystem_capabilities_finalized()is true, andkvm_arm_init()runs after that.[...]
Quoted 1 line
Quoted 1 line
arch/arm64/kvm/hyp/nvhe/pkvm.c +0
-0[...]
Quoted 1 line
Quoted 1 line
arch/arm64/kvm/hyp/nvhe/pkvm.c +0
-0@@ -1181,8 +1187,70 @@ bool kvm_handle_pvm_hvc64(struct kvm_vcpu *vcpu, u64 *exit_code)[...]
Quoted 3 lines
Quoted 3 lines
arch/arm64/kvm/hyp/nvhe/pkvm.c +3
-0+case ARM_SMCCC_ARCH_FEATURES_FUNC_ID : +/* The workaround calls themselves are handled on hyp entry. */ + val[0 ] =SMCCC_RET_NOT_SUPPORTED ;[...]
Quoted 9 lines
Quoted 8 lines
arch/arm64/kvm/hyp/nvhe/pkvm.c +8
-0+case ARM_SMCCC_ARCH_WORKAROUND_2 : +switch (spectre_v4_state) { +case SPECTRE_VULNERABLE : +break ; +case SPECTRE_MITIGATED : +/* With SSBS advertised the guest's default is safe. */ +if (kvm_has_feat (vcpu->kvm,ID_AA64PFR1_EL1 ,SSBS ,IMP )) +break ;
Quoted 7 lines
Quoted 4 lines
For this case (and the break immediately above) you return val[0] =
SMCCC_RET_INVALID_PARAMETER(from init at top of this function). But the kernel'skvm_smccc_call_handler()returns val[0] =SMCCC_RET_NOT_SUPPORTEDin these cases (which seems more likely to be correct). Does this violate the ABI?Both return
NOT_SUPPORTED. val[0] is set to it at the top of theARM_SMCCC_ARCH_FEATURES_FUNC_IDcase.
Ahh missed that oops!
Quoted 11 lines
Quoted 10 lines
arch/arm64/kvm/hyp/nvhe/pkvm.c bool kvm_handle_pvm_hvc64(struct kvm_vcpu *vcpu, u64 *exit_code) · line 1229
Quoted 5 lines
arch/arm64/kvm/hyp/nvhe/pkvm.c +3
-0+ fallthrough; +case SPECTRE_UNAFFECTED : + val[0 ] =SMCCC_RET_NOT_REQUIRED ;Is it possible that the (compromised) kernel could have disabled the mitigation after the state snapshot was taken? I.e.
spectre_v4_state==SPECTRE_MITIGATEDand the VM doesn't advertise SSBS so it returnsSMCCC_RET_NOT_REQUIREDhere. But the kernel has since turned off the mitigation?
Quoted 5 lines
Yes, where the mitigation is the firmware one. The host can disable it with an
ARM_SMCCC_ARCH_WORKAROUND_2call, which EL2 forwards, and EL2 doesn't enable it again before running the guest. With TF-A that's only on a Cortex-A76 without SSBS, the only CPU it implements the dynamic workaround for.
If the mitigation is via SSBS but SSBS is not exposed to the guest, I'm guessing this is safe because the guest is expected to set SSBS=0 in this case?
Thanks,
Ryan
Quoted 5 lines
I'll send out a fix. Thanks for spotting this!
Cheers,
/fuad
Fuad Tabba
reviewerRe: [PATCH v4 09/18] KVM: arm64: Implement HVC handling for protected guests at EL2
On Wed, 11 Nov 2026 at 17:50, Ryan Roberts <[email protected]> wrote:
[...]
Quoted 8 lines
Quoted 5 lines
Yes, where the mitigation is the firmware one. The host can disable it with an
ARM_SMCCC_ARCH_WORKAROUND_2call, which EL2 forwards, and EL2 doesn't enable it again before running the guest. With TF-A that's only on a Cortex-A76 without SSBS, the only CPU it implements the dynamic workaround for.If the mitigation is via SSBS but SSBS is not exposed to the guest, I'm guessing this is safe because the guest is expected to set SSBS=0 in this case?
Yes. A vCPU is reset with PSTATE.SSBS and SCTLR_EL1.DSSBS clear, so the guest doesn't have to set it.
Cheers,
/fuad
Fuad Tabba
reviewerPATCH v4 10/18 KVM: arm64: Handle PSCI calls for protected VMs at EL2
In reply to Fuad TabbaEL2 implements PSCI 1.1 for protected VMs: CPU_ON, CPU_OFF, PSCI_VERSION and PSCI_FEATURES are resolved at EL2 (CPU_ON and CPU_OFF still exit to the host, which only schedules the target or stops scheduling it), AFFINITY_INFO, CPU_SUSPEND and the platform power operations are forwarded to the host, and anything else returns NOT_SUPPORTED, including the TRNG calls and the functions above 1.1, SYSTEM_OFF2 among them, that the host handled for a protected guest until now. TRNG for protected guests is a follow-up. AFFINITY_INFO stays with the host, which returns OFF only once it has stopped scheduling the target: the host is what a guest polls to see a CPU_OFF complete before it issues the next CPU_ON, as Linux does on hotplug.
Three consequences follow:
- A protected VM has one primary vCPU, the first whose hyp vCPU is created with mp_state RUNNABLE. A second one, or an mp_state other than RUNNABLE or STOPPED, fails that vCPU's first KVM_RUN with -EINVAL.
- CPU_ON finds its target among the hyp vCPUs, which exist from the target's first KVM_RUN; before that the guest gets INVALID_PARAMETERS.
- A vCPU EL2 holds powered off doesn't run: handle___kvm_vcpu_run() returns ARM_EXCEPTION_IL, reported as KVM_EXIT_FAIL_ENTRY. Its existing bail-outs return the same code, for every hyp vCPU, instead of an -EINVAL whose bit 31 the host read as a pending SError.
Non-protected VMs keep power_state ON and accept any mp_state.
Each protected vCPU is OFF, ON_PENDING or ON. CPU_ON moves the target to ON_PENDING, and the target's next run resets it and moves it to ON. The racing transitions are cmpxchg, and the reset state is published with a release/acquire pair, documented at each site. CPU_OFF clears the reset flag and stores OFF with a release, and CPU_ON's writes are control dependent on its cmpxchg reading OFF, so the target's accesses to reset_state are ordered before the winner's writes. Rolling a CPU_ON the host failed back to OFF requires the host's return value, which the per-EC marshalling patch delivers along with the rollback. Until then such a target stays ON_PENDING, and the reset has no observable
effect: flush_hyp_vcpu() copies the host's context in on every entry until that patch removes the copy, so the target enters on the host's values rather than the ones EL2 reset.
--- arch/arm64/kvm/hyp/include/nvhe/pkvm.h | 14 ++ arch/arm64/kvm/hyp/nvhe/hyp-main.c | 25 ++- arch/arm64/kvm/hyp/nvhe/pkvm.c | 293 ++++++++++++++++++++++++- 3 files changed, 321 insertions(+), 11 deletions(-)
arch/arm64/kvm/hyp/include/nvhe/pkvm.h +14 -0
@@ -29,6 +29,12 @@ struct pkvm_hyp_vcpu {/* The previous exit's ARM_EXCEPTION_* code. */ u32 exit_code; + +/* +* PSCI_0_2_AFFINITY_LEVEL_{OFF, ON_PENDING, ON}. A non-protected +* vCPU is always ON. +*/ +int power_state; };/*
@@ -46,6 +52,12 @@ struct pkvm_hyp_vm {struct hyp_pool pool;hyp_spinlock_t lock; +/* +* The vCPU initialised RUNNABLE: claimed under vm_table_lock, +* released only if its own init fails. +*/ +struct pkvm_hyp_vcpu *primary_vcpu; +/* Array of the hyp vCPU structures for this VM. */ struct pkvm_hyp_vcpu *vcpus[]; };
@@ -100,4 +112,6 @@ void kvm_init_pvm_id_regs(struct kvm_vcpu *vcpu);void kvm_reset_pvm_sys_regs (struct kvm_vcpu *vcpu);int kvm_check_pvm_sysreg_table (void ); +int pkvm_reset_vcpu (struct pkvm_hyp_vcpu *hyp_vcpu); +struct pkvm_hyp_vcpu *pkvm_mpidr_to_hyp_vcpu (struct pkvm_hyp_vm *vm,unsigned long mpidr);#endif /* __ARM64_KVM_NVHE_PKVM_H__ */
arch/arm64/kvm/hyp/nvhe/hyp-main.c +19 -6
@@ -8,6 +8,7 @@#include <hyp/switch.h> #include <linux/irqchip/arm-gic-v3.h> +#include <uapi/linux/psci.h> #include <asm/pgtable-types.h> #include <asm/kvm_asm.h>
@@ -458,14 +459,12 @@ static void handle___kvm_vcpu_run(struct kvm_cpu_context *host_ctxt) {struct pkvm_hyp_vcpu *hyp_vcpu;struct kvm_vcpu *host_vcpu; -int ret; +int ret =ARM_EXCEPTION_IL ; host_vcpu =get_host_hyp_vcpus (host_ctxt,1 , &hyp_vcpu); -if (!host_vcpu) { - ret = -EINVAL ; +if (!host_vcpu) goto out; - }if (unlikely (hyp_vcpu)) {/*
@@ -474,8 +473,22 @@ static void handle___kvm_vcpu_run(struct kvm_cpu_context *host_ctxt)* loading a vcpu. Therefore, if SME features enabled the host * is misbehaving. */ -if (unlikely (system_supports_sme () &&read_sysreg_s (SYS_SVCR ))) { - ret = -EINVAL ; +if (unlikely (system_supports_sme () &&read_sysreg_s (SYS_SVCR ))) + goto out; + +/* +* ON has a single writer, pkvm_reset_vcpu() on this CPU, so +* READ_ONCE suffices. ON_PENDING takes the reset; -ECANCELED +* is a rollback that raced it. +*/ +switch (READ_ONCE (hyp_vcpu->power_state)) { +case PSCI_0_2_AFFINITY_LEVEL_ON : +break ; +case PSCI_0_2_AFFINITY_LEVEL_ON_PENDING : +if (pkvm_reset_vcpu (hyp_vcpu)) + goto out; +break ; +default : goto out; }
arch/arm64/kvm/hyp/nvhe/pkvm.c +288 -5
365 lines not on this page · raw message
Signature · 1 line
2.39.5
Fuad Tabba
reviewerPATCH v4 11/18 KVM: arm64: Restrict KVM_ARM_VCPU_INIT and PSCI version for protected VMs
In reply to Fuad TabbaEL2 masks a protected VM's vCPU features down to the set pKVM allows, so a KVM_ARM_VCPU_INIT that requests more succeeds and produces a guest without them. Reject such an init with -EINVAL. The host and EL2 both take the allowed set from kvm_pkvm_vcpu_allowed_features().
EL2 returns 1.1 for a protected guest's PSCI_VERSION whatever the host is configured for, and forwards the platform calls to the host. Require KVM_ARM_VCPU_PSCI_0_2: without it the host dispatches the forwarded calls as PSCI 0.1. Reject a KVM_REG_ARM_PSCI_VERSION write below 1.1: below it the host rejects the SYSTEM_RESET2 that EL2 advertises.
--- arch/arm64/include/asm/kvm_pkvm.h | 24 ++++++++++++++++++++++++ arch/arm64/kvm/arm.c | 13 +++++++++++++ arch/arm64/kvm/hyp/nvhe/pkvm.c | 17 ++--------------- arch/arm64/kvm/hypercalls.c | 7 +++++++ 4 files changed, 46 insertions(+), 15 deletions(-)
arch/arm64/include/asm/kvm_pkvm.h +24 -0
@@ -53,6 +53,30 @@ static inline bool kvm_pkvm_ext_allowed(struct kvm *kvm, long ext) } } +/* +* The vCPU features a protected VM may use: checked by the host at +* KVM_ARM_VCPU_INIT, applied by EL2 when the hyp VM is created. +*/ +static inline void kvm_pkvm_vcpu_allowed_features (struct kvm *kvm, +unsigned long *allowed) +{ +bitmap_zero (allowed,KVM_VCPU_MAX_FEATURES ); + +set_bit (KVM_ARM_VCPU_PSCI_0_2 , allowed); + +if (kvm_pkvm_ext_allowed (kvm,KVM_CAP_ARM_PMU_V3 )) +set_bit (KVM_ARM_VCPU_PMU_V3 , allowed); + +if (kvm_pkvm_ext_allowed (kvm,KVM_CAP_ARM_PTRAUTH_ADDRESS )) +set_bit (KVM_ARM_VCPU_PTRAUTH_ADDRESS , allowed); + +if (kvm_pkvm_ext_allowed (kvm,KVM_CAP_ARM_PTRAUTH_GENERIC )) +set_bit (KVM_ARM_VCPU_PTRAUTH_GENERIC , allowed); + +if (kvm_pkvm_ext_allowed (kvm,KVM_CAP_ARM_SVE )) +set_bit (KVM_ARM_VCPU_SVE , allowed); +} +/* * Check whether the KVM VM IOCTL is allowed in pKVM. *
arch/arm64/kvm/arm.c +13 -0
@@ -1620,6 +1620,19 @@ static int kvm_vcpu_init_check_features(struct kvm_vcpu *vcpu,if (features & ~system_supported_vcpu_features ())return -EINVAL ; +/* Reject features EL2 would drop when it creates the hyp VM. */ +if (vcpu_is_protected (vcpu)) { +DECLARE_BITMAP (allowed,KVM_VCPU_MAX_FEATURES ); + +kvm_pkvm_vcpu_allowed_features (vcpu->kvm, allowed); +if (!bitmap_subset (&features, allowed,KVM_VCPU_MAX_FEATURES )) +return -EINVAL ; + +/* EL2 implements PSCI 1.1; the host must not dispatch as 0.1. */ +if (!test_bit (KVM_ARM_VCPU_PSCI_0_2 , &features)) +return -EINVAL ; + } +/* * For now make sure that both address/generic pointer authentication * features are requested by the userspace together.
arch/arm64/kvm/hyp/nvhe/pkvm.c +2 -15
@@ -410,23 +410,10 @@ static void pkvm_init_features_from_host(struct pkvm_hyp_vm *hyp_vm, const strucif (kvm_pkvm_ext_allowed (kvm,KVM_CAP_ARM_MTE )) kvm->arch.flags |= host_arch_flags &BIT (KVM_ARCH_FLAG_MTE_ENABLED ); -bitmap_zero (allowed_features,KVM_VCPU_MAX_FEATURES ); +kvm_pkvm_vcpu_allowed_features (kvm, allowed_features); -set_bit (KVM_ARM_VCPU_PSCI_0_2 , allowed_features); - -if (kvm_pkvm_ext_allowed (kvm,KVM_CAP_ARM_PMU_V3 )) -set_bit (KVM_ARM_VCPU_PMU_V3 , allowed_features); - -if (kvm_pkvm_ext_allowed (kvm,KVM_CAP_ARM_PTRAUTH_ADDRESS )) -set_bit (KVM_ARM_VCPU_PTRAUTH_ADDRESS , allowed_features); - -if (kvm_pkvm_ext_allowed (kvm,KVM_CAP_ARM_PTRAUTH_GENERIC )) -set_bit (KVM_ARM_VCPU_PTRAUTH_GENERIC , allowed_features); - -if (kvm_pkvm_ext_allowed (kvm,KVM_CAP_ARM_SVE )) { -set_bit (KVM_ARM_VCPU_SVE , allowed_features); +if (kvm_pkvm_ext_allowed (kvm,KVM_CAP_ARM_SVE )) kvm->arch.flags |= host_arch_flags &BIT (KVM_ARCH_FLAG_GUEST_HAS_SVE ); - }bitmap_and (kvm->arch.vcpu_features, host_kvm->arch.vcpu_features, allowed_features,KVM_VCPU_MAX_FEATURES );
arch/arm64/kvm/hypercalls.c +7 -0
@@ -580,6 +580,13 @@ int kvm_arm_set_fw_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg) wants_02 =vcpu_has_feature (vcpu,KVM_ARM_VCPU_PSCI_0_2 ); +/* +* EL2 advertises PSCI 1.1; the host handles the forwarded calls +* by this version. +*/ +if (vcpu_is_protected (vcpu) && val <KVM_ARM_PSCI_1_1 ) +return -EINVAL ; +switch (val) {case KVM_ARM_PSCI_0_1 :if (wants_02)
Signature · 1 line
2.39.5
Fuad Tabba
reviewerPATCH v4 12/18 KVM: arm64: Prevent host PC adjustments for protected vCPUs
In reply to Fuad Tabba__kvm_adjust_pc() lets the host advance a vCPU's PC or inject an exception, which for a protected vCPU would let the host redirect guest execution. Leave the request on the host copy there: the entry handlers apply the host's PC_UPDATE_REQ at the next entry, where EL2 allows it.
__kvm_adjust_pc() adjusts the vCPU a get/put pair returns, the one it was given outside pKVM. Both host callers hold the vCPU mutex, so a hyp vCPU loaded for the vCPU is loaded on the calling CPU, and EL2's own calls pass the hyp vCPU itself. For a loaded protected vCPU, kvm_adjust_pc_get() returns NULL and the request stays on the host copy. For a loaded non-protected vCPU, PKVM_HOST_STATE_DIRTY selects the copy, since adjusting the hyp vCPU while the host copy is authoritative loses the update at the next flush. Adjusting the hyp vCPU copies PC_UPDATE_REQ in and back out again: without the copy back, INCREMENT_PC outlives the adjustment and the next KVM_SET_VCPU_EVENTS trips WARN_ON(INCREMENT_PC) in kvm_pend_exception(). With no hyp vCPU loaded, as under KVM_SET_VCPU_EVENTS, the host copy is adjusted as before.
Until the marshalling patch clears PC_UPDATE_REQ on the host copy at exit, a KVM_RUN that returns to userspace with INCREMENT_PC set leaves it on the host copy of a loaded protected vCPU, and a KVM_SET_VCPU_EVENTS before the next run then trips that WARN_ON.
--- arch/arm64/kvm/hyp/exception.c | 21 +++++++++----- arch/arm64/kvm/hyp/include/hyp/adjust_pc.h | 20 ++++++++++++++ arch/arm64/kvm/hyp/nvhe/hyp-main.c | 32 ++++++++++++++++++++++ 3 files changed, 66 insertions(+), 7 deletions(-)
arch/arm64/kvm/hyp/exception.c +14 -7
@@ -353,12 +353,19 @@ static void kvm_inject_exception(struct kvm_vcpu *vcpu)*/ void __kvm_adjust_pc (struct kvm_vcpu *vcpu) { -if (vcpu_get_flag (vcpu,PENDING_EXCEPTION )) { -kvm_inject_exception (vcpu); -vcpu_clear_flag (vcpu,PENDING_EXCEPTION ); -vcpu_clear_flag (vcpu,EXCEPT_MASK ); - }else if (vcpu_get_flag (vcpu,INCREMENT_PC )) { -kvm_skip_instr (vcpu); -vcpu_clear_flag (vcpu,INCREMENT_PC ); +struct kvm_vcpu *target =kvm_adjust_pc_get (vcpu); + +if (!target) +return ; + +if (vcpu_get_flag (target,PENDING_EXCEPTION )) { +kvm_inject_exception (target); +vcpu_clear_flag (target,PENDING_EXCEPTION ); +vcpu_clear_flag (target,EXCEPT_MASK ); + }else if (vcpu_get_flag (target,INCREMENT_PC )) { +kvm_skip_instr (target); +vcpu_clear_flag (target,INCREMENT_PC ); } + +kvm_adjust_pc_put (vcpu, target); }
arch/arm64/kvm/hyp/include/hyp/adjust_pc.h +20 -0
@@ -78,4 +78,24 @@ static inline void kvm_skip_host_instr(void) write_sysreg_el2(read_sysreg_el2(SYS_ELR) + 4, SYS_ELR); } +/* +* Under pKVM, the vCPU __kvm_adjust_pc() adjusts for @vcpu (NULL leaves the +* request on @vcpu for its next entry), and the copy of the consumed +* PC_UPDATE_REQ back to @vcpu. +*/ +#ifdef __KVM_NVHE_HYPERVISOR__ +struct kvm_vcpu *kvm_adjust_pc_get (struct kvm_vcpu *vcpu); +void kvm_adjust_pc_put (struct kvm_vcpu *vcpu,struct kvm_vcpu *target); +#else +static inline struct kvm_vcpu *kvm_adjust_pc_get (struct kvm_vcpu *vcpu) +{ +return vcpu; +} + +static inline void kvm_adjust_pc_put (struct kvm_vcpu *vcpu, +struct kvm_vcpu *target) +{ +} +#endif +#endif
arch/arm64/kvm/hyp/nvhe/hyp-main.c +32 -0
@@ -644,6 +644,38 @@ static void handle___pkvm_host_mkyoung_guest(struct kvm_cpu_context *host_ctxt)cpu_reg (host_ctxt,1 ) = ret; } +/* +* PKVM_HOST_STATE_DIRTY names the authoritative copy, the host's when set. +* A loaded protected vCPU takes the request at its next entry instead. +*/ +struct kvm_vcpu *kvm_adjust_pc_get (struct kvm_vcpu *vcpu) +{ +struct pkvm_hyp_vcpu *hyp_vcpu; + +if (!is_protected_kvm_enabled ()) +return vcpu; + + hyp_vcpu =pkvm_get_loaded_hyp_vcpu (); +if (!hyp_vcpu || vcpu == &hyp_vcpu->vcpu) +return vcpu; + +if (pkvm_hyp_vcpu_is_protected (hyp_vcpu)) +return NULL ; + +if (vcpu_get_flag (vcpu,PKVM_HOST_STATE_DIRTY )) +return vcpu; + +vcpu_copy_flag (&hyp_vcpu->vcpu, vcpu,PC_UPDATE_REQ ); +return &hyp_vcpu->vcpu; +} + +/* Reflect the consumed request back, otherwise it stays pending. */ +void kvm_adjust_pc_put (struct kvm_vcpu *vcpu,struct kvm_vcpu *target) +{ +if (target != vcpu) +vcpu_copy_flag (vcpu, target,PC_UPDATE_REQ ); +} +static void handle___kvm_adjust_pc (struct kvm_cpu_context *host_ctxt) { DECLARE_REG(struct kvm_vcpu *, vcpu, host_ctxt, 1);
Signature · 1 line
2.39.5
[email protected]
AI reviewRe: [PATCH v4 12/18] KVM: arm64: Prevent host PC adjustments for protected vCPUs
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [Critical] kvm: host DoS via KVM_SET_VCPU_EVENTS and uncleared INCREMENT_PC - [High] kvm: silently…
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Critical] kvm: host DoS via KVM_SET_VCPU_EVENTS and uncleared INCREMENT_PC
- [High] kvm: silently dropped exception injections on non-trap exits for protected guests --
commit 8bf814727f8778e45c8fb25a3c96ee3b58fc462e
Author: Fuad Tabba <[email protected]>
KVM: arm64: Prevent host PC adjustments for protected vCPUs
This commit modifies __kvm_adjust_pc() to leave PC_UPDATE_REQ (including INCREMENT_PC and PENDING_EXCEPTION) on the host copy for protected VMs. It introduces kvm_adjust_pc_get() which returns NULL for protected vCPUs, bypassing the adjustment and leaving the flag clearing to entry/exit handlers.
Quoted 9 lines
arch/arm64/kvm/hyp/exception.c +0
-1@@ -353,12 +353,19 @@ static void kvm_inject_exception(struct kvm_vcpu *vcpu)*/ void __kvm_adjust_pc (struct kvm_vcpu *vcpu) { -if (vcpu_get_flag (vcpu,PENDING_EXCEPTION )) {
arch/arm64/kvm/hyp/exception.c static void kvm_inject_exception(struct kvm_vcpu *vcpu) · line 359
Quoted 10 lines
arch/arm64/kvm/hyp/exception.c +4
-6-kvm_inject_exception (vcpu); -vcpu_clear_flag (vcpu,PENDING_EXCEPTION ); -vcpu_clear_flag (vcpu,EXCEPT_MASK ); - }else if (vcpu_get_flag (vcpu,INCREMENT_PC )) { -kvm_skip_instr (vcpu); -vcpu_clear_flag (vcpu,INCREMENT_PC ); +struct kvm_vcpu *target =kvm_adjust_pc_get (vcpu); + +if (!target) +return ;
[Severity: Critical]
Can this early return leave INCREMENT_PC uncleared, leading to a host panic?
When an unprivileged host userspace VMM manages a protected VM and sets SCTLR2_EL1.NMEA=1, a guest MMIO exit causes kvm_handle_mmio_return() to set INCREMENT_PC on the host vCPU copy.
If entry is then aborted due to a pending signal, we exit to userspace with -EINTR. Because target is NULL for protected VMs, we return early here without clearing INCREMENT_PC.
If userspace then calls KVM_SET_VCPU_EVENTS to inject an SError (with serror_pending=1), serror_is_masked() returns false, which causes pend_serror_exception() to call kvm_pend_exception(). This hits the assertion:
WARN_ON(vcpu_get_flag(v, INCREMENT_PC));
crashing the host if panic_on_warn is set. While Patch 14 attempts to fix this by clearing PC_UPDATE_REQ on exit from EL2, INCREMENT_PC is set at EL1 on MMIO return after the EL2 exit.
[ ... ]
Quoted 15 lines
arch/arm64/kvm/hyp/nvhe/hyp-main.c +7
-0@@ -644,6 +644,38 @@ static void handle___pkvm_host_mkyoung_guest(struct kvm_cpu_context *host_ctxt)cpu_reg (host_ctxt,1 ) = ret; } +/* +* PKVM_HOST_STATE_DIRTY names the authoritative copy, the host's when set. +* A loaded protected vCPU takes the request at its next entry instead. +*/ +struct kvm_vcpu *kvm_adjust_pc_get (struct kvm_vcpu *vcpu) +{ +struct pkvm_hyp_vcpu *hyp_vcpu;
arch/arm64/kvm/hyp/nvhe/hyp-main.c static void handle___pkvm_host_mkyoung_guest(struct kvm_cpu_context *host_ctxt) · line 663
Quoted 10 lines
arch/arm64/kvm/hyp/nvhe/hyp-main.c +10
-0+ +if (!is_protected_kvm_enabled ()) +return vcpu; + + hyp_vcpu =pkvm_get_loaded_hyp_vcpu (); +if (!hyp_vcpu || vcpu == &hyp_vcpu->vcpu) +return vcpu; + +if (pkvm_hyp_vcpu_is_protected (hyp_vcpu)) +return NULL ;
[Severity: High]
Does deferring exception injection to the entry path cause host-injected exceptions to be silently dropped on non-trap exits?
If a protected guest exits due to an IRQ (ARM_EXCEPTION_IRQ) and userspace calls KVM_SET_VCPU_EVENTS with serror_pending=1, PENDING_EXCEPTION is set on the host vCPU.
On KVM_RUN, flush_hyp_vcpu() evaluates the exit code and bypasses the ec_handler for non-trap exits:
case ARM_EXCEPTION_IRQ: case ARM_EXCEPTION_EL1_SERROR: case ARM_EXCEPTION_IL: break;
This ignores the PENDING_EXCEPTION flag during entry. When the guest exits again, sync_hyp_vcpu() unconditionally clears PC_UPDATE_REQ (which includes PENDING_EXCEPTION), permanently losing the injected exception.
[ ... ]
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=12
Fuad Tabba
reviewerRe: [PATCH v4 12/18] KVM: arm64: Prevent host PC adjustments for protected vCPUs
On Thu, 1 Oct 2026 at 15:21, <[email protected]> wrote:
[...]
Quoted 3 lines
[Severity: High]
Does deferring exception injection to the entry path cause host-injected exceptions to be silently dropped on non-trap exits?
Yes, by design. EL2 completes a host-pended exception only as the trapped class's own exception.
/fuad
Quoted 21 lines
If a protected guest exits due to an IRQ (
ARM_EXCEPTION_IRQ) and userspace callsKVM_SET_VCPU_EVENTSwith serror_pending=1,PENDING_EXCEPTIONis set on the host vCPU.On
KVM_RUN,flush_hyp_vcpu()evaluates the exit code and bypasses theec_handlerfor non-trap exits:case ARM_EXCEPTION_IRQ: case ARM_EXCEPTION_EL1_SERROR: case ARM_EXCEPTION_IL: break;This ignores the
PENDING_EXCEPTIONflag during entry. When the guest exits again,sync_hyp_vcpu()unconditionally clearsPC_UPDATE_REQ(which includesPENDING_EXCEPTION), permanently losing the injected exception.[ ... ]
--
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=12
Fuad Tabba
reviewerPATCH v4 13/18 KVM: arm64: Inject an UNDEF at EL2 for unhandled protected guest exits
In reply to Fuad TabbaMake pvm_exit_handlers[] an allow-list: a protected guest's exit class is handled at EL2, forwarded where the host emulates it, or results in an UNDEF at EL2. Any other class, once forwarded, would re-execute forever. The host resolves such an exit by injecting an exception or incrementing the PC. The marshalling that follows allows neither: a protected vCPU takes only the UNDEF on a SYS64 trap and the external abort on a forwarded IABT or DABT that EL2 builds for it.
WFxT takes the UNDEF, as it isn't advertised to protected guests. FP/SIMD keeps the lazy switch, and the watchpoint entry is dropped: a protected vCPU runs with TDE clear.
--- arch/arm64/kvm/hyp/nvhe/switch.c | 29 +++++++++++++++++++++++++---- 1 file changed, 25 insertions(+), 4 deletions(-)
arch/arm64/kvm/hyp/nvhe/switch.c +25 -4
@@ -203,15 +203,36 @@ static const exit_handler_fn hyp_exit_handlers[] = { [ESR_ELx_EC_MOPS] = kvm_hyp_handle_mops, }; +/* WFI/WFE exit to the host, which emulates them. WFxT is not advertised. */ +static bool kvm_handle_pvm_wfx (struct kvm_vcpu *vcpu,u64 *exit_code) +{ +if (kvm_vcpu_get_esr (vcpu) & ESR_ELx_WFx_ISS_WFxT) +return kvm_handle_pvm_restricted (vcpu, exit_code); + +return false; +} + +/* Lazy FP/SIMD switch, or the UNDEF that would otherwise be left to the host. */ +static bool kvm_handle_pvm_fpsimd (struct kvm_vcpu *vcpu,u64 *exit_code) +{ +if (kvm_hyp_handle_fpsimd (vcpu, exit_code)) +return true; + +return kvm_handle_pvm_restricted (vcpu, exit_code); +} + +/* +* A class not listed takes an UNDEF at EL2: the host has no way to +* inject one into a protected vCPU. +*/ static const exit_handler_fn pvm_exit_handlers[] = { - [0 ... ESR_ELx_EC_MAX] =NULL , + [0 ... ESR_ELx_EC_MAX] = kvm_handle_pvm_restricted, + [ESR_ELx_EC_WFx] = kvm_handle_pvm_wfx, [ESR_ELx_EC_HVC64] = kvm_handle_pvm_hvc64, [ESR_ELx_EC_SYS64] = kvm_handle_pvm_sys64, - [ESR_ELx_EC_SVE] = kvm_handle_pvm_restricted, - [ESR_ELx_EC_FP_ASIMD] = kvm_hyp_handle_fpsimd, + [ESR_ELx_EC_FP_ASIMD] = kvm_handle_pvm_fpsimd, [ESR_ELx_EC_IABT_LOW] = kvm_hyp_handle_iabt_low, [ESR_ELx_EC_DABT_LOW] = kvm_hyp_handle_dabt_low, - [ESR_ELx_EC_WATCHPT_LOW] = kvm_hyp_handle_watchpt_low, [ESR_ELx_EC_MOPS] = kvm_hyp_handle_mops, };
Signature · 1 line
2.39.5
Fuad Tabba
reviewerPATCH v4 14/18 KVM: arm64: Add per-EC entry/exit state marshalling for protected guests
In reply to Fuad TabbaMove a protected guest's state between the hyp vCPU and the host per exception class instead of copying the whole context. Add entry_hyp_pvm_handlers[] and exit_hyp_pvm_handlers[] for WFx, SYS64, IABT, DABT and HVC64, and route protected guests through them: on exit each handler copies out only what its class requires, and on re-entry only what the host may have changed: the PC update or exception it requested (PC_UPDATE_REQ), the value of a read it emulated, and a forwarded PSCI call's return value. entry_hyp_vm_handlers[] is
removed: a non-protected vCPU's iflags are copied wholesale.
The host's fault view is EL2's own syndrome with the guest register index withheld, the deferred SError syndrome (DISR_EL1), and the addresses each class requires. The value of a written register is passed in r0. MMIO data is clamped to the access width and, for a load, sign-extended at EL2 from EL2's syndrome. Endianness stays with the host. A data abort's PC update is taken for a completed MMIO access, and for a cache maintenance operation the host skips on unbacked memory, as it does for any guest.
The host's copy of a protected vCPU's PSTATE is a view the guest never runs from: its reset value has PSTATE.A set, and EL2 sets PSTATE.A in the mode it copies out regardless of the guest's, so, unless the VMM wrote the host copy's PSTATE or SCTLR2_EL1 before the first run, serror_is_masked() is true and kvm_inject_serror_esr() pends a host-injected SError through HCR_EL2.VSE, masked by the guest's own PSTATE.A, instead of emulating the entry on that copy.
Neither dispatch runs for a trap taken with an SError pending: EL2 doesn't handle it, and the guest replays it once the host has injected the SError. The exit handlers would otherwise marshal a trap EL2 never handled, and handle_pvm_exit_hvc64() would panic on an unfiltered function id.
For HVC64, only the PSCI calls EL2 forwards reach the host: the exit handler passes the function id and the arguments each call takes, and the entry handler returns the host's result. A CPU_ON the host failed is rolled back to OFF and returned as INTERNAL_FAILURE, or as ALREADY_ON when that's what the host returned: PSCI defines it as the retry signal for a CPU_ON that reaches the implementation before the target's CPU_OFF has been processed (DEN0022 section 6.6), which a guest that doesn't poll AFFINITY_INFO first can do. A target that already ran returns SUCCESS. The rollback leaves the published reset state in place: clearing it races a fresh CPU_ON's publication and leaves the target stuck at ON_PENDING, and the entry point left behind is one the guest supplied. The rollback stores OFF with a release, as CPU_OFF does, so this vCPU's writes to the reset state are ordered before the next winner's. The rollback's cmpxchg carries no generation, so one that lands after a later CPU_ON has republished ON_PENDING cancels that cycle too. The target is then OFF at EL2 while the host has it runnable, and its CPU_ONs return ALREADY_ON until the VMM stops it again.
--- arch/arm64/kvm/hyp/nvhe/hyp-main.c | 435 +++++++++++++++++++++++++++-- 1 file changed, 412 insertions(+), 23 deletions(-)
arch/arm64/kvm/hyp/nvhe/hyp-main.c +412 -23
@@ -4,6 +4,8 @@* Author: Andrew Scull <[email protected]> */ +#include <kvm/arm_hypercalls.h> +#include <hyp/adjust_pc.h> #include <hyp/switch.h>
@@ -34,13 +36,371 @@ void __kvm_hyp_host_forward_smc(struct kvm_cpu_context *host_ctxt);typedef void (*hyp_entry_exit_handler_fn )(struct pkvm_hyp_vcpu *); -static void handle_vm_entry_generic (struct pkvm_hyp_vcpu *hyp_vcpu) +static bool pvm_sys64_is_write (u64 esr) { -vcpu_copy_flag (&hyp_vcpu->vcpu, hyp_vcpu->host_vcpu,PC_UPDATE_REQ ); +return (esr & ESR_ELx_SYS64_ISS_DIR_MASK) == ESR_ELx_SYS64_ISS_DIR_WRITE; } -static const hyp_entry_exit_handler_fn entry_hyp_vm_handlers[] = { - [0 ... ESR_ELx_EC_MAX] = handle_vm_entry_generic, +static void handle_pvm_entry_wfx (struct pkvm_hyp_vcpu *hyp_vcpu) +{ +struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu; +struct kvm_vcpu *vcpu = &hyp_vcpu->vcpu; + +/* Exceptions have priority; the host injects none on WFx. */ +if (vcpu_get_flag (host_vcpu,PENDING_EXCEPTION )) +return ; + +if (vcpu_get_flag (host_vcpu,INCREMENT_PC )) { +vcpu_clear_flag (vcpu,PC_UPDATE_REQ ); +kvm_incr_pc (vcpu); + } +} + +static void handle_pvm_entry_sys64 (struct pkvm_hyp_vcpu *hyp_vcpu) +{ +struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu; +struct kvm_vcpu *vcpu = &hyp_vcpu->vcpu; +bool pc_update; + +/* Exceptions have priority over anything else */ +if (vcpu_get_flag (host_vcpu,PENDING_EXCEPTION )) { +/* A host-requested exception on SYS64 is always an UNDEF. */ +u32 esr = (ESR_ELx_EC_UNKNOWN << ESR_ELx_EC_SHIFT) | ESR_ELx_IL; + +__vcpu_assign_sys_reg (vcpu,ESR_EL1 , esr); +kvm_pend_exception (vcpu,EXCEPT_AA64_EL1_SYNC ); +return ; + } + +/* Handle PC increment on a host-emulated access */ + pc_update =vcpu_get_flag (host_vcpu,INCREMENT_PC ); +if (pc_update) { +vcpu_clear_flag (vcpu,PC_UPDATE_REQ ); +kvm_incr_pc (vcpu); + } + +/* If the host emulated a read access, update the register */ +if (pc_update && !pvm_sys64_is_write (kvm_vcpu_get_esr (vcpu))) { +/* r0 as transfer register between the guest and the host. */ +u64 rt_val =READ_ONCE (vcpu_gp_regs (host_vcpu)->regs[0 ]); +int rt =kvm_vcpu_sys_get_rt (vcpu); + +vcpu_set_reg (vcpu, rt, rt_val); + } +} + +static void handle_pvm_entry_iabt (struct pkvm_hyp_vcpu *hyp_vcpu) +{ +struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu; +struct kvm_vcpu *vcpu = &hyp_vcpu->vcpu; +unsigned long cpsr = *vcpu_cpsr (vcpu); +u32 esr = ESR_ELx_IL; + +if (!vcpu_get_flag (host_vcpu,PENDING_EXCEPTION )) +return ; + +/* The host's only IABT injection: an external abort. */ +if ((cpsr &PSR_MODE_MASK ) == PSR_MODE_EL0t) + esr |= (ESR_ELx_EC_IABT_LOW << ESR_ELx_EC_SHIFT); +else + esr |= (ESR_ELx_EC_IABT_CUR << ESR_ELx_EC_SHIFT); + + esr |= ESR_ELx_FSC_EXTABT; + +__vcpu_assign_sys_reg (vcpu,ESR_EL1 , esr); +__vcpu_assign_sys_reg (vcpu,FAR_EL1 ,kvm_vcpu_get_hfar (vcpu)); + +/* Injected by __kvm_adjust_pc() on entry. */ +kvm_pend_exception (vcpu,EXCEPT_AA64_EL1_SYNC ); +} + +/* +* Clamp MMIO data to the access width, so a write does not leak the +* register's upper bits and a read takes no bits beyond the load. The +* host applies endianness. +*/ +static inline u64 kvm_mmio_clamp_data (struct kvm_vcpu *vcpu,u64 val) +{ +unsigned int len =kvm_vcpu_dabt_get_as (vcpu); + +return val &GENMASK_U64 (len *8 -1 ,0 ); +} + +/* +* Complete an MMIO load: sign-extend from EL2's own syndrome, as the +* architecture does. +*/ +static inline u64 kvm_mmio_read_data (struct kvm_vcpu *vcpu,u64 val) +{ + val =kvm_mmio_clamp_data (vcpu, val); + +if (kvm_vcpu_dabt_issext (vcpu)) + val =sign_extend64 (val,kvm_vcpu_dabt_get_as (vcpu) *8 -1 ); + +if (!kvm_vcpu_dabt_issf (vcpu)) + val &=GENMASK_U64 (31 ,0 ); + +return val; +} + +static void handle_pvm_entry_dabt (struct pkvm_hyp_vcpu *hyp_vcpu) +{ +struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu; +struct kvm_vcpu *vcpu = &hyp_vcpu->vcpu; +bool pc_update; + +/* Exceptions have priority over anything else */ +if (vcpu_get_flag (host_vcpu,PENDING_EXCEPTION )) { +unsigned long cpsr = *vcpu_cpsr (vcpu); +u32 esr = ESR_ELx_IL; + +if ((cpsr &PSR_MODE_MASK ) == PSR_MODE_EL0t) + esr |= (ESR_ELx_EC_DABT_LOW << ESR_ELx_EC_SHIFT); +else + esr |= (ESR_ELx_EC_DABT_CUR << ESR_ELx_EC_SHIFT); + + esr |= ESR_ELx_FSC_EXTABT; + +__vcpu_assign_sys_reg (vcpu,ESR_EL1 , esr); +__vcpu_assign_sys_reg (vcpu,FAR_EL1 ,kvm_vcpu_get_hfar (vcpu)); + +/* Injected by __kvm_adjust_pc() on entry. */ +kvm_pend_exception (vcpu,EXCEPT_AA64_EL1_SYNC ); + +/* Cancel any in-flight MMIO */ + vcpu->mmio_needed = false; +return ; + } + +/* Handle PC increment on MMIO, or on a CMO the host skipped */ + pc_update =vcpu_get_flag (host_vcpu,INCREMENT_PC ) && + (vcpu->mmio_needed ||kvm_vcpu_dabt_is_cm (vcpu)); +if (pc_update) { +vcpu_clear_flag (vcpu,PC_UPDATE_REQ ); +kvm_incr_pc (vcpu); + } + +/* If the host emulated an MMIO read, update the register */ +if (pc_update && vcpu->mmio_needed && !kvm_vcpu_dabt_iswrite (vcpu)) { +/* r0 as transfer register between the guest and the host. */ +u64 rd_val =READ_ONCE (vcpu_gp_regs (host_vcpu)->regs[0 ]); +int rd =kvm_vcpu_dabt_get_rd (vcpu); + + rd_val =kvm_mmio_read_data (vcpu, rd_val); +vcpu_set_reg (vcpu, rd, rd_val); + } + + vcpu->mmio_needed = false; +} + +static void handle_pvm_entry_hvc64 (struct pkvm_hyp_vcpu *hyp_vcpu) +{ +struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu; +struct kvm_vcpu *vcpu = &hyp_vcpu->vcpu; +u64 ret =READ_ONCE (vcpu_gp_regs (host_vcpu)->regs[0 ]); +u32 psci_fn =smccc_get_function (vcpu); + +switch (psci_fn) { +case PSCI_0_2_FN_CPU_ON : +case PSCI_0_2_FN64_CPU_ON : +/* +* Roll back a CPU_ON the host failed, unless the target +* already reached ON: it is running, and the guest sees +* SUCCESS. +*/ +if (ret !=PSCI_RET_SUCCESS ) { +unsigned long cpu_id =smccc_get_arg1 (vcpu); +struct pkvm_hyp_vcpu *target_vcpu; +struct pkvm_hyp_vm *hyp_vm; +int prev; + + hyp_vm =pkvm_hyp_vcpu_to_hyp_vm (hyp_vcpu); + target_vcpu =pkvm_mpidr_to_hyp_vcpu (hyp_vm, cpu_id); + +/* +* pvm_psci_vcpu_on() resolved this MPIDR and vcpus[] +* entries are never removed, so the lookup cannot miss. +* The release orders this vCPU's reset_state writes +* before OFF, for the next CPU_ON. +*/ + prev =cmpxchg_release (&target_vcpu->power_state, +PSCI_0_2_AFFINITY_LEVEL_ON_PENDING , +PSCI_0_2_AFFINITY_LEVEL_OFF ); +switch (prev) { +case PSCI_0_2_AFFINITY_LEVEL_ON_PENDING : +/* +* Leave reset_state.reset set: a clear races a +* fresh CPU_ON's publish. The stale pc/r0/be are +* the guest's own. ALREADY_ON is PSCI's retry +* signal for a CPU_ON that raced the CPU_OFF. +*/ +if (ret !=PSCI_RET_ALREADY_ON ) + ret =PSCI_RET_INTERNAL_FAILURE ; +break ; +case PSCI_0_2_AFFINITY_LEVEL_ON : +case PSCI_0_2_AFFINITY_LEVEL_OFF : +/* Target already ran (and may have stopped). */ + ret =PSCI_RET_SUCCESS ; +break ; +default : + ret =PSCI_RET_INTERNAL_FAILURE ; +break ; + } + } + +break ; +default : +break ; + } + +vcpu_set_reg (vcpu,0 , ret); +} + +/* The host's view of a syndrome: the guest register index is withheld. */ +static u64 pvm_host_esr (u64 esr) +{ +switch (ESR_ELx_EC (esr)) { +case ESR_ELx_EC_WFx: +return esr & ~ESR_ELx_WFx_ISS_RN; +case ESR_ELx_EC_SYS64: +return esr & ~ESR_ELx_SYS64_ISS_RT_MASK; +case ESR_ELx_EC_DABT_LOW: +return esr & ~ESR_ELx_SRT_MASK; +default : +return esr; + } +} + +/* +* The host's view of PSTATE: the mode, with SErrors masked so that the host +* pends an SError through HCR_EL2.VSE rather than emulating the entry. +*/ +static unsigned long pvm_host_pstate (unsigned long pstate) +{ +return (pstate &PSR_MODE_MASK ) |PSR_A_BIT ; +} + +static void handle_pvm_exit_wfx (struct pkvm_hyp_vcpu *hyp_vcpu) +{ +struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu; +struct kvm_vcpu *vcpu = &hyp_vcpu->vcpu; + + *vcpu_cpsr (host_vcpu) =pvm_host_pstate (*vcpu_cpsr (vcpu)); +} + +static void handle_pvm_exit_sys64 (struct pkvm_hyp_vcpu *hyp_vcpu) +{ +struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu; +struct kvm_vcpu *vcpu = &hyp_vcpu->vcpu; +u64 esr =kvm_vcpu_get_esr (vcpu); + +/* The mode is required for the host to emulate some sysregs */ + *vcpu_cpsr (host_vcpu) =pvm_host_pstate (*vcpu_cpsr (vcpu)); + +/* r0 as transfer register between the guest and the host. */ +if (pvm_sys64_is_write (esr)) { +int rt =kvm_vcpu_sys_get_rt (vcpu); +u64 rt_val =vcpu_get_reg (vcpu, rt); + +vcpu_set_reg (host_vcpu,0 , rt_val); + } +} + +static void handle_pvm_exit_iabt (struct pkvm_hyp_vcpu *hyp_vcpu) +{ +struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu; +struct kvm_vcpu *vcpu = &hyp_vcpu->vcpu; + + host_vcpu->arch.fault.hpfar_el2 = vcpu->arch.fault.hpfar_el2; +} + +static void handle_pvm_exit_dabt (struct pkvm_hyp_vcpu *hyp_vcpu) +{ +struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu; +struct kvm_vcpu *vcpu = &hyp_vcpu->vcpu; +u64 sctlr; + +/* +* EL2 has no memslot view: a decodable data abort is prepared as MMIO +* for the host to resolve. On unbacked memory the host injects an SEA +* for one with ISV clear (LDP/STP, atomics), which EL2 does not +* decode, and skips a cache maintenance operation, as for any guest. +*/ + vcpu->mmio_needed =kvm_vcpu_dabt_isvalid (vcpu); + +/* r0 as transfer register between the guest and the host. */ +if (vcpu->mmio_needed &&kvm_vcpu_dabt_iswrite (vcpu)) { +int rt =kvm_vcpu_dabt_get_rd (vcpu); +u64 rt_val =vcpu_get_reg (vcpu, rt); + + rt_val =kvm_mmio_clamp_data (vcpu, rt_val); +vcpu_set_reg (host_vcpu,0 , rt_val); + } + + *vcpu_cpsr (host_vcpu) =pvm_host_pstate (*vcpu_cpsr (vcpu)); + host_vcpu->arch.fault.far_el2 = vcpu->arch.fault.far_el2 &GENMASK (11 ,0 ); + host_vcpu->arch.fault.hpfar_el2 = vcpu->arch.fault.hpfar_el2; + sctlr =__vcpu_sys_reg (vcpu,SCTLR_EL1 ) & (SCTLR_ELx_EE |SCTLR_EL1_E0E ); +__vcpu_assign_sys_reg (host_vcpu,SCTLR_EL1 , sctlr); +} + +static void handle_pvm_exit_hvc64 (struct pkvm_hyp_vcpu *hyp_vcpu) +{ +struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu; +struct kvm_vcpu *vcpu = &hyp_vcpu->vcpu; +int n, i; + +switch (smccc_get_function (vcpu)) { +/* +* CPU_ON: the host uses only the target MPIDR (x1). EL2 resets the +* target from its own copy of the entry point and context id. +*/ +case PSCI_0_2_FN_CPU_ON : +case PSCI_0_2_FN64_CPU_ON : + n =2 ; +break ; + +case PSCI_0_2_FN_CPU_OFF : +case PSCI_0_2_FN_SYSTEM_OFF : +case PSCI_0_2_FN_SYSTEM_RESET : +case PSCI_0_2_FN_CPU_SUSPEND : +case PSCI_0_2_FN64_CPU_SUSPEND : + n =1 ; +break ; + +case PSCI_0_2_FN_AFFINITY_INFO : +case PSCI_0_2_FN64_AFFINITY_INFO : +case PSCI_1_1_FN_SYSTEM_RESET2 : +case PSCI_1_1_FN64_SYSTEM_RESET2 : + n =3 ; +break ; + +/* Unreachable: kvm_handle_pvm_hvc64() forwards only the calls above. */ +default : +hyp_panic (); + } + +/* Pass the HVC function id (r0) and its arguments. */ +for (i =0 ; i < n; i++) +vcpu_set_reg (host_vcpu, i,vcpu_get_reg (vcpu, i)); +} + +static const hyp_entry_exit_handler_fn entry_hyp_pvm_handlers[] = { + [0 ... ESR_ELx_EC_MAX] =NULL , + [ESR_ELx_EC_WFx] = handle_pvm_entry_wfx, + [ESR_ELx_EC_SYS64] = handle_pvm_entry_sys64, + [ESR_ELx_EC_IABT_LOW] = handle_pvm_entry_iabt, + [ESR_ELx_EC_DABT_LOW] = handle_pvm_entry_dabt, + [ESR_ELx_EC_HVC64] = handle_pvm_entry_hvc64, +}; + +static const hyp_entry_exit_handler_fn exit_hyp_pvm_handlers[] = { + [0 ... ESR_ELx_EC_MAX] =NULL , + [ESR_ELx_EC_WFx] = handle_pvm_exit_wfx, + [ESR_ELx_EC_SYS64] = handle_pvm_exit_sys64, + [ESR_ELx_EC_IABT_LOW] = handle_pvm_exit_iabt, + [ESR_ELx_EC_DABT_LOW] = handle_pvm_exit_dabt, + [ESR_ELx_EC_HVC64] = handle_pvm_exit_hvc64, };static void __hyp_sve_save_guest (struct kvm_vcpu *vcpu)
@@ -284,18 +644,6 @@ static void flush_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu) hyp_vcpu->vcpu.arch.mdcr_el2 = host_vcpu->arch.mdcr_el2; hyp_vcpu->vcpu.arch.iflags = host_vcpu->arch.iflags; - }else { -u64 v_cval = hyp_vcpu->vcpu.arch.ctxt.sys_regs[CNTV_CVAL_EL0 ]; -u64 v_ctl = hyp_vcpu->vcpu.arch.ctxt.sys_regs[CNTV_CTL_EL0 ]; -u64 p_cval = hyp_vcpu->vcpu.arch.ctxt.sys_regs[CNTP_CVAL_EL0 ]; -u64 p_ctl = hyp_vcpu->vcpu.arch.ctxt.sys_regs[CNTP_CTL_EL0 ]; - - hyp_vcpu->vcpu.arch.ctxt = host_vcpu->arch.ctxt; - - hyp_vcpu->vcpu.arch.ctxt.sys_regs[CNTV_CVAL_EL0 ] = v_cval; - hyp_vcpu->vcpu.arch.ctxt.sys_regs[CNTV_CTL_EL0 ] = v_ctl; - hyp_vcpu->vcpu.arch.ctxt.sys_regs[CNTP_CVAL_EL0 ] = p_cval; - hyp_vcpu->vcpu.arch.ctxt.sys_regs[CNTP_CTL_EL0 ] = p_ctl; }/* __hyp_running_vcpu must be NULL in a guest context. */
@@ -320,10 +668,16 @@ static void flush_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu)case ARM_EXCEPTION_IL :break ;case ARM_EXCEPTION_TRAP : - esr_ec =ESR_ELx_EC (kvm_vcpu_get_esr (&hyp_vcpu->vcpu)); - ec_handler = entry_hyp_vm_handlers[esr_ec]; -if (ec_handler) -ec_handler (hyp_vcpu); +/* Nothing was marshalled for this trap, see sync_hyp_vcpu(). */ +if (ARM_SERROR_PENDING (hyp_vcpu->exit_code)) +break ; + +if (pkvm_hyp_vcpu_is_protected (hyp_vcpu)) { + esr_ec =ESR_ELx_EC (kvm_vcpu_get_esr (&hyp_vcpu->vcpu)); + ec_handler = entry_hyp_pvm_handlers[esr_ec]; +if (ec_handler) +ec_handler (hyp_vcpu); + }break ;default :BUG ();
@@ -335,13 +689,26 @@ static void flush_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu)static void sync_hyp_vcpu (struct pkvm_hyp_vcpu *hyp_vcpu,u32 exit_reason) {struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu; +hyp_entry_exit_handler_fn ec_handler; +u8 esr_ec;fpsimd_sve_sync (&hyp_vcpu->vcpu);sync_debug_state (hyp_vcpu);if (pkvm_hyp_vcpu_is_protected (hyp_vcpu)) { - host_vcpu->arch.ctxt = hyp_vcpu->vcpu.arch.ctxt; +/* +* Protected: the host sees ESR_EL2 as EL2 took it, register +* index withheld; the fault addresses stay withheld unless the +* EC handler below adds them. +*/ + host_vcpu->arch.fault = (struct kvm_vcpu_fault_info ) { + .esr_el2 =pvm_host_esr (hyp_vcpu->vcpu.arch.fault.esr_el2), + .disr_el1 = hyp_vcpu->vcpu.arch.fault.disr_el1, + }; }else { +/* Non-protected: the host gets the full fault. */ + host_vcpu->arch.fault = hyp_vcpu->vcpu.arch.fault; + host_vcpu->arch.iflags = hyp_vcpu->vcpu.arch.iflags;/* * PC feeds trace_kvm_exit(), PSTATE.SS the host software-step * machine, and both run before the next on-demand ctxt sync.
@@ -350,9 +717,28 @@ static void sync_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu, u32 exit_reason) host_vcpu->arch.ctxt.regs.pstate = hyp_vcpu->vcpu.arch.ctxt.regs.pstate; } - host_vcpu->arch.fault = hyp_vcpu->vcpu.arch.fault; +switch (ARM_EXCEPTION_CODE (exit_reason)) { +case ARM_EXCEPTION_IRQ : +break ; +case ARM_EXCEPTION_TRAP : +/* SError pending: not handled at EL2, the guest replays it. */ +if (ARM_SERROR_PENDING (exit_reason)) +break ; - host_vcpu->arch.iflags = hyp_vcpu->vcpu.arch.iflags; +/* Per-EC marshalling is for protected guests only. */ +if (pkvm_hyp_vcpu_is_protected (hyp_vcpu)) { + esr_ec =ESR_ELx_EC (kvm_vcpu_get_esr (&hyp_vcpu->vcpu)); + ec_handler = exit_hyp_pvm_handlers[esr_ec]; +if (ec_handler) +ec_handler (hyp_vcpu); + } +break ; +case ARM_EXCEPTION_EL1_SERROR : +case ARM_EXCEPTION_IL : +break ; +default : +BUG (); + }/* Cleared by hardware once the guest takes the vSError. */ host_vcpu->arch.hcr_el2 &= ~HCR_VSE ;
@@ -361,6 +747,9 @@ static void sync_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu, u32 exit_reason)sync_hyp_vgic_state (hyp_vcpu);sync_hyp_timer_state (hyp_vcpu); +if (pkvm_hyp_vcpu_is_protected (hyp_vcpu)) +vcpu_clear_flag (host_vcpu,PC_UPDATE_REQ ); + hyp_vcpu->exit_code = exit_reason; }
Signature · 1 line
2.39.5
Fuad Tabba
reviewerPATCH v4 15/18 KVM: arm64: Reject host access to protected VM private state
In reply to Fuad TabbaA protected vCPU's register and debug state is no longer exposed to the host, and the host ioctls that would reach it fail instead:
- KVM_GET_ONE_REG and KVM_SET_ONE_REG return -EPERM once the vCPU has
run: the host copy then holds the boot state the VMM wrote plus what the exit handlers copy out, which the VMM can't tell apart. Pre-run access still builds the guest's boot state.
- KVM_ARM_VCPU_INIT returns -EPERM once the vCPU has run: it would reset the host copy alone and rewrite mp_state, which EL2 reads only at hyp vCPU creation, so a vCPU the guest powered off would come back RUNNABLE.
- KVM_SET_VCPU_EVENTS rejects external-abort injection with -EPERM; SError injection is forwarded and stays permitted.
- KVM_SET_GUEST_DEBUG returns -EPERM: a protected guest's debug state is hypervisor-owned.
The KVM_{GET,SET}_ONE_REG and KVM_ARM_VCPU_INIT checks are one filter on the ioctl number in kvm_arch_vcpu_ioctl(), the only caller of the three functions they were in. Its -EPERM now precedes the cases' -EFAULT and the ONE_REG case's pending-reset handling, which the next KVM_RUN performs. The external-abort check reads the payload, and KVM_SET_GUEST_DEBUG has its own case in kvm_vcpu_ioctl(), so it never reaches kvm_arch_vcpu_ioctl(): those two stay in their handlers.
KVM_CHECK_EXTENSION returns 0 for KVM_CAP_ARM_INJECT_EXT_DABT and KVM_CAP_SET_GUEST_DEBUG on a protected VM: kvm_pkvm_ext_allowed() returns false on every capability it doesn't list, and the patch that advertises the capabilities protected VMs support leaves these two out. The two ioctl checks stay for a VMM that doesn't query KVM_CHECK_EXTENSION.
--- arch/arm64/kvm/arm.c | 22 ++++++++++++++++++++++ arch/arm64/kvm/guest.c | 11 +++++++++++ 2 files changed, 33 insertions(+)
arch/arm64/kvm/arm.c +22 -0
@@ -1865,6 +1865,24 @@ static int kvm_arm_vcpu_set_events(struct kvm_vcpu *vcpu,return __kvm_arm_vcpu_set_events (vcpu, events); } +/* +* Once a protected vCPU has run, the host copy holds the boot state the VMM +* wrote plus what the exit handlers copy out, and EL2 has read mp_state, +* which it does only at hyp vCPU creation. +*/ +static long pkvm_filter_vcpu_ioctl (struct kvm_vcpu *vcpu,unsigned int ioctl) +{ +switch (ioctl) { +case KVM_ARM_VCPU_INIT : +case KVM_SET_ONE_REG : +case KVM_GET_ONE_REG : +if (vcpu_is_protected (vcpu) &&vcpu_has_run_once (vcpu)) +return -EPERM ; + } + +return 0 ; +} +long kvm_arch_vcpu_ioctl (struct file *filp,unsigned int ioctl,unsigned long arg) {
@@ -1873,6 +1891,10 @@ long kvm_arch_vcpu_ioctl(struct file *filp,struct kvm_device_attr attr;long r; + r =pkvm_filter_vcpu_ioctl (vcpu, ioctl); +if (r) +return r; +switch (ioctl) {case KVM_ARM_VCPU_INIT : {struct kvm_vcpu_init init;
arch/arm64/kvm/guest.c +11 -0
@@ -786,6 +786,13 @@ int __kvm_arm_vcpu_set_events(struct kvm_vcpu *vcpu,u64 esr = events->exception.serror_esr;int ret =0 ; +/* +* EL2 injects an external abort only to complete a forwarded abort. +* SError injection is forwarded. +*/ +if (vcpu_is_protected (vcpu) && ext_dabt_pending) +return -EPERM ; +/* * Immediately commit the pending SEA to the vCPU's architectural * state which is necessary since we do not return a pending SEA
@@ -883,6 +890,10 @@ int kvm_arch_vcpu_ioctl_set_guest_debug(struct kvm_vcpu *vcpu, {trace_kvm_set_guest_debug (vcpu, dbg->control); +/* A protected guest's debug state is not exposed to the host. */ +if (vcpu_is_protected (vcpu)) +return -EPERM ; +if (dbg->control & ~KVM_GUESTDBG_VALID_MASK )return -EINVAL ;
Signature · 1 line
2.39.5
Fuad Tabba
reviewerPATCH v4 16/18 KVM: arm64: Reject host power-on of a vCPU that EL2 holds powered off
In reply to Fuad TabbaA protected vCPU's power state belongs to EL2, which changes it only on the guest's own PSCI calls. KVM_SET_MP_STATE(RUNNABLE) on a vCPU EL2 holds powered off changes the host's copy alone, and the guest's next CPU_ON of that vCPU then fails: mp_state is no longer STOPPED, so the host returns ALREADY_ON, and the guest's retry fails the same way until the VMM stops the vCPU again. SUSPENDED has the same effect.
Record a vCPU EL2 holds powered off as KVM_MP_STATE_UNINITIALIZED, an encoding arm64 didn't use: the guest's CPU_OFF writes it, as does hyp vCPU creation for a vCPU that starts STOPPED, and only the guest's CPU_ON writes RUNNABLE over it. RUNNABLE and SUSPENDED from the VMM return -EPERM on it, the VMM can't write the value itself, and KVM_ARM_VCPU_INIT, which would write STOPPED or RUNNABLE over it, is rejected once the vCPU has run.
The VMM can still pause and resume a vCPU EL2 has powered on. Neither STOPPED nor kvm_prepare_system_event() overwrites the value: a vCPU at it is already off, and a STOPPED in its place would turn the guest's next CPU_ON into ALREADY_ON, after a SYSTEM_RESET exit the VMM runs on from. kvm_arm_vcpu_stopped() covers it, so the run loop sleeps the vCPU as before.
The guest's CPU_ON is gated on the value rather than on STOPPED. A STOPPED written between the target's CPU_OFF exit and the host's power-off of it, which kvm_prepare_system_event() can do from another vCPU's SYSTEM_OFF or SYSTEM_RESET, would otherwise let the CPU_ON through, and the power-off would then record the target powered off while EL2 holds it ON_PENDING, with no way back before VM teardown. Gated on the value, that CPU_ON returns ALREADY_ON and the retry succeeds once the power-off has run.
--- arch/arm64/include/asm/kvm_host.h | 1 + arch/arm64/kvm/arm.c | 35 +++++++++++++++++++++++++------ arch/arm64/kvm/pkvm.c | 21 +++++++++++++++---- arch/arm64/kvm/psci.c | 11 ++++++++-- 4 files changed, 56 insertions(+), 12 deletions(-)
arch/arm64/include/asm/kvm_host.h +1 -0
@@ -1564,6 +1564,7 @@ static inline void kvm_hyp_reserve(void) { }void kvm_arm_vcpu_power_off (struct kvm_vcpu *vcpu);bool kvm_arm_vcpu_stopped (struct kvm_vcpu *vcpu); +bool kvm_pkvm_vcpu_is_powered_off (struct kvm_vcpu *vcpu);static inline u64 *__vm_id_reg (struct kvm_arch *ka,u32 reg) {
arch/arm64/kvm/arm.c +29 -6
@@ -770,23 +770,28 @@ void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu) vcpu->cpu =-1 ; } -static void __kvm_arm_vcpu_power_off (struct kvm_vcpu *vcpu) +static void __kvm_arm_vcpu_power_off (struct kvm_vcpu *vcpu,u32 mp_state) { -WRITE_ONCE (vcpu->arch.mp_state.mp_state,KVM_MP_STATE_STOPPED ); +WRITE_ONCE (vcpu->arch.mp_state.mp_state, mp_state);kvm_make_request (KVM_REQ_SLEEP , vcpu);kvm_vcpu_kick (vcpu); } +/* The guest's own CPU_OFF: EL2 has already powered the vCPU off. */ void kvm_arm_vcpu_power_off (struct kvm_vcpu *vcpu) { +u32 mp_state =vcpu_is_protected (vcpu) ?KVM_MP_STATE_UNINITIALIZED :KVM_MP_STATE_STOPPED ; +spin_lock (&vcpu->arch.mp_state_lock); -__kvm_arm_vcpu_power_off (vcpu); +__kvm_arm_vcpu_power_off (vcpu, mp_state);spin_unlock (&vcpu->arch.mp_state_lock); }bool kvm_arm_vcpu_stopped (struct kvm_vcpu *vcpu) { -return READ_ONCE (vcpu->arch.mp_state.mp_state) ==KVM_MP_STATE_STOPPED ; +u32 mp_state =READ_ONCE (vcpu->arch.mp_state.mp_state); + +return mp_state ==KVM_MP_STATE_STOPPED || mp_state ==KVM_MP_STATE_UNINITIALIZED ; }static void kvm_arm_vcpu_suspend (struct kvm_vcpu *vcpu)
@@ -801,6 +806,12 @@ static bool kvm_arm_vcpu_suspended(struct kvm_vcpu *vcpu)return READ_ONCE (vcpu->arch.mp_state.mp_state) ==KVM_MP_STATE_SUSPENDED ; } +/* Only the guest's CPU_ON may start a vCPU EL2 holds powered off. */ +bool kvm_pkvm_vcpu_is_powered_off (struct kvm_vcpu *vcpu) +{ +return READ_ONCE (vcpu->arch.mp_state.mp_state) ==KVM_MP_STATE_UNINITIALIZED ; +} +int kvm_arch_vcpu_ioctl_get_mpstate (struct kvm_vcpu *vcpu,struct kvm_mp_state *mp_state) {
@@ -818,12 +829,24 @@ int kvm_arch_vcpu_ioctl_set_mpstate(struct kvm_vcpu *vcpu,switch (mp_state->mp_state) {case KVM_MP_STATE_RUNNABLE : +if (kvm_pkvm_vcpu_is_powered_off (vcpu)) { + ret = -EPERM ; +break ; + } +WRITE_ONCE (vcpu->arch.mp_state, *mp_state);break ;case KVM_MP_STATE_STOPPED : -__kvm_arm_vcpu_power_off (vcpu); +/* Keep EL2's off record: STOPPED over it would let RUNNABLE through. */ +if (!kvm_pkvm_vcpu_is_powered_off (vcpu)) +__kvm_arm_vcpu_power_off (vcpu,KVM_MP_STATE_STOPPED );break ;case KVM_MP_STATE_SUSPENDED : +if (kvm_pkvm_vcpu_is_powered_off (vcpu)) { + ret = -EPERM ; +break ; + } +kvm_arm_vcpu_suspend (vcpu);break ;default :
@@ -1788,7 +1811,7 @@ static int kvm_arch_vcpu_ioctl_vcpu_init(struct kvm_vcpu *vcpu,spin_lock (&vcpu->arch.mp_state_lock);if (power_off) -__kvm_arm_vcpu_power_off (vcpu); +__kvm_arm_vcpu_power_off (vcpu,KVM_MP_STATE_STOPPED );else WRITE_ONCE (vcpu->arch.mp_state.mp_state,KVM_MP_STATE_RUNNABLE );
arch/arm64/kvm/pkvm.c +17 -4
@@ -118,12 +118,25 @@ static int __pkvm_create_hyp_vcpu(struct kvm_vcpu *vcpu)return -ENOMEM ; ret =kvm_call_hyp_nvhe (__pkvm_init_vcpu, handle, vcpu, hyp_vcpu); -if (!ret) -vcpu_set_flag (vcpu,VCPU_PKVM_FINALIZED ); -else +if (ret) {free_pages_exact (hyp_vcpu, hyp_vcpu_sz); +return ret; + } -return ret; +/* +* Mirror EL2's seeding of power_state from mp_state. The hyp vCPU is +* published, so take mp_state_lock against kvm_psci_vcpu_on(). +*/ +if (vcpu_is_protected (vcpu)) { +spin_lock (&vcpu->arch.mp_state_lock); +if (kvm_arm_vcpu_stopped (vcpu)) +WRITE_ONCE (vcpu->arch.mp_state.mp_state,KVM_MP_STATE_UNINITIALIZED ); +spin_unlock (&vcpu->arch.mp_state_lock); + } + +vcpu_set_flag (vcpu,VCPU_PKVM_FINALIZED ); + +return 0 ; }/*
arch/arm64/kvm/psci.c +9 -2
@@ -63,7 +63,12 @@ static unsigned long kvm_psci_vcpu_on(struct kvm_vcpu *source_vcpu)return PSCI_RET_INVALID_PARAMS ;spin_lock (&vcpu->arch.mp_state_lock); -if (!kvm_arm_vcpu_stopped (vcpu)) { +/* +* A protected vCPU is off once the host has acted on EL2's CPU_OFF. +* STOPPED before that is the VMM's pause. +*/ +if (!kvm_arm_vcpu_stopped (vcpu) || + (vcpu_is_protected (vcpu) && !kvm_pkvm_vcpu_is_powered_off (vcpu))) {if (kvm_psci_version (source_vcpu) !=KVM_ARM_PSCI_0_1 ) ret =PSCI_RET_ALREADY_ON ;else
@@ -161,7 +166,9 @@ static void kvm_prepare_system_event(struct kvm_vcpu *vcpu, u32 type, u64 flags)*/ kvm_for_each_vcpu (i, tmp, vcpu->kvm) {spin_lock (&tmp->arch.mp_state_lock); -WRITE_ONCE (tmp->arch.mp_state.mp_state,KVM_MP_STATE_STOPPED ); +/* Keep EL2's off record for the guest's next CPU_ON. */ +if (!kvm_pkvm_vcpu_is_powered_off (tmp)) +WRITE_ONCE (tmp->arch.mp_state.mp_state,KVM_MP_STATE_STOPPED );spin_unlock (&tmp->arch.mp_state_lock); }kvm_make_all_cpus_request (vcpu->kvm,KVM_REQ_SLEEP );
Signature · 1 line
2.39.5
[email protected]
AI reviewRe: [PATCH v4 16/18] KVM: arm64: Reject host power-on of a vCPU that EL2 holds powered off
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [High] arm64: PSCI CPU_ON spuriously fails for protected secondary vCPUs - [Medium] arm64…
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] arm64: PSCI CPU_ON spuriously fails for protected secondary vCPUs
- [Medium] arm64: KVM_SET_MP_STATE rejects KVM_MP_STATE_UNINITIALIZED with -EINVAL --
commit f4c00ca4b159d4b7954978f9de7770cc8b67b029
Author: Fuad Tabba <[email protected]>
KVM: arm64: Reject host power-on of a vCPU that EL2 holds powered off
This patch records a protected vCPU held powered off by EL2 as KVM_MP_STATE_UNINITIALIZED. It restricts KVM_SET_MP_STATE from overwriting this value with RUNNABLE or SUSPENDED, and updates the guest CPU_ON checks to gate on this state rather than STOPPED.
Quoted 4 lines
arch/arm64/kvm/arm.c +0
-0
[ ... ]
Quoted 26 lines
arch/arm64/kvm/arm.c +13
-1@@ -818,12 +829,24 @@ int kvm_arch_vcpu_ioctl_set_mpstate(struct kvm_vcpu *vcpu,switch (mp_state->mp_state) {case KVM_MP_STATE_RUNNABLE : +if (kvm_pkvm_vcpu_is_powered_off (vcpu)) { + ret = -EPERM ; +break ; + } +WRITE_ONCE (vcpu->arch.mp_state, *mp_state);break ;case KVM_MP_STATE_STOPPED : -__kvm_arm_vcpu_power_off (vcpu); +/* Keep EL2's off record: STOPPED over it would let RUNNABLE through. */ +if (!kvm_pkvm_vcpu_is_powered_off (vcpu)) +__kvm_arm_vcpu_power_off (vcpu,KVM_MP_STATE_STOPPED );break ;case KVM_MP_STATE_SUSPENDED : +if (kvm_pkvm_vcpu_is_powered_off (vcpu)) { + ret = -EPERM ; +break ; + } +kvm_arm_vcpu_suspend (vcpu);break ;default :
arch/arm64/kvm/arm.c
Quoted 2 lines
arch/arm64/kvm/arm.c +0
-0ret = -EINVAL ; }
[Severity: Medium]
Does this break the userspace API for KVM_SET_MP_STATE?
The switch statement lacks a case for KVM_MP_STATE_UNINITIALIZED, defaulting to -EINVAL for a state that KVM now actively exposes. If a userspace VMM attempts to save and restore the state by reading via KVM_GET_MP_STATE and later passing that exact state back via KVM_SET_MP_STATE, it will fail with -EINVAL. Can we add a case to properly handle the uninitialized state?
[ ... ]
arch/arm64/kvm/arm.c static int kvm_arch_vcpu_ioctl_vcpu_init(struct kvm_vcpu *vcpu, · line 1814
Quoted 6 lines
arch/arm64/kvm/arm.c +1
-1@@ -1788,7 +1811,7 @@ static int kvm_arch_vcpu_ioctl_vcpu_init(struct kvm_vcpu *vcpu,spin_lock (&vcpu->arch.mp_state_lock);if (power_off) -__kvm_arm_vcpu_power_off (vcpu); +__kvm_arm_vcpu_power_off (vcpu,KVM_MP_STATE_STOPPED );
[Severity: High]
Can this cause a spurious PSCI_RET_ALREADY_ON failure when booting secondary protected vCPUs?
When the VMM initializes a secondary protected vCPU with KVM_ARM_VCPU_POWER_OFF, it explicitly hardcodes its mp_state to KVM_MP_STATE_STOPPED here.
Quoted 3 lines
arch/arm64/kvm/arm.c +0
-0else WRITE_ONCE (vcpu->arch.mp_state.mp_state,KVM_MP_STATE_RUNNABLE );
[ ... ]
Quoted 8 lines
arch/arm64/kvm/psci.c +0
-0@@ -63,7 +63,12 @@ static unsigned long kvm_psci_vcpu_on(struct kvm_vcpu *source_vcpu)return PSCI_RET_INVALID_PARAMS ;spin_lock (&vcpu->arch.mp_state_lock);
arch/arm64/kvm/psci.c static unsigned long kvm_psci_vcpu_on(struct kvm_vcpu *source_vcpu) · line 74
Quoted 10 lines
arch/arm64/kvm/psci.c +6
-1-if (!kvm_arm_vcpu_stopped (vcpu)) { +/* +* A protected vCPU is off once the host has acted on EL2's CPU_OFF. +* STOPPED before that is the VMM's pause. +*/ +if (!kvm_arm_vcpu_stopped (vcpu) || + (vcpu_is_protected (vcpu) && !kvm_pkvm_vcpu_is_powered_off (vcpu))) {if (kvm_psci_version (source_vcpu) !=KVM_ARM_PSCI_0_1 ) ret =PSCI_RET_ALREADY_ON ;else
[Severity: High]
Following up on the KVM_ARM_VCPU_POWER_OFF initialization above, does this condition incorrectly reject guest CPU_ON requests?
If the primary vCPU executes a PSCI CPU_ON call before the secondary vCPU thread has executed its first KVM_RUN, __pkvm_create_hyp_vcpu() hasn't yet upgraded the state to KVM_MP_STATE_UNINITIALIZED.
Consequently, when kvm_psci_vcpu_on() evaluates the condition above, it sees the vCPU as STOPPED but not UNINITIALIZED. The condition evaluates to true, causing it to return PSCI_RET_ALREADY_ON to the guest. This deterministically breaks SMP boot for protected VMs on VMMs that block secondary vCPU threads prior to wakeup.
Should kvm_arch_vcpu_ioctl_vcpu_init() directly set the state to KVM_MP_STATE_UNINITIALIZED for protected vCPUs to avoid this race?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=16
Fuad Tabba
reviewerRe: [PATCH v4 16/18] KVM: arm64: Reject host power-on of a vCPU that EL2 holds powered off
On Thu, 01 Oct 2026 15:18:27 +0100, [email protected] wrote:
[...]
Quoted 1 line
Does this break the userspace API for
KVM_SET_MP_STATE?
No. KVM_SET_MP_STATE has never accepted KVM_MP_STATE_UNINITIALIZED on arm64, and the host can't restore a power state that EL2 owns.
[...]
Quoted 2 lines
Following up on the
KVM_ARM_VCPU_POWER_OFFinitialization above, does this condition incorrectly reject guestCPU_ONrequests?
No. Before the target's first KVM_RUN it has no hyp vCPU, so pvm_psci_vcpu_on() returns INVALID_PARAMETERS at EL2 and the host never sees the CPU_ON.
/fuad
Fuad Tabba
reviewerPATCH v4 17/18 KVM: arm64: Advertise the capabilities that protected VMs support
In reply to Fuad TabbaFor a protected VM, kvm_pkvm_ext_allowed() returns false on every capability it doesn't name, so KVM_CHECK_EXTENSION returns 0 for interfaces the VM implements: it denies KVM_CAP_ONE_REG while KVM_{GET,SET}_ONE_REG stage the guest's boot state.
Allow the capabilities that work for a protected guest under the restrictions the preceding patches add, the vGIC and the I/O bus being host-managed.
--- arch/arm64/include/asm/kvm_pkvm.h | 9 +++++++++ 1 file changed, 9 insertions(+)
arch/arm64/include/asm/kvm_pkvm.h +9 -0
@@ -42,6 +42,15 @@ static inline bool kvm_pkvm_ext_allowed(struct kvm *kvm, long ext)case KVM_CAP_ARM_VM_IPA_SIZE :case KVM_CAP_ARM_PTRAUTH_ADDRESS :case KVM_CAP_ARM_PTRAUTH_GENERIC : +case KVM_CAP_ONE_REG : +case KVM_CAP_MP_STATE : +case KVM_CAP_VCPU_EVENTS : +case KVM_CAP_VCPU_ATTRIBUTES : +case KVM_CAP_IMMEDIATE_EXIT : +case KVM_CAP_IOEVENTFD : +case KVM_CAP_IRQFD_RESAMPLE : +case KVM_CAP_ARM_IRQ_LINE_LAYOUT_2 : +case KVM_CAP_ARM_INJECT_SERROR_ESR :return true;case KVM_CAP_ARM_MTE :return false;
Signature · 1 line
2.39.5
Fuad Tabba
reviewerPATCH v4 18/18 KVM: arm64: Document the protected VM userspace API
In reply to Fuad TabbaEL2 owns a protected vCPU's state once it has run, and the ioctls that would reach it fail rather than operate on the host copy, which holds the boot state the VMM wrote plus what the exit handlers copy out. Describe the resulting API in pkvm.rst, point api.rst at it from each ioctl that behaves differently, and say beside the PVTIME attribute in devices/vcpu.rst and the PSCI version register in fw-pseudo-registers.rst how protected VMs differ. The overview also states the GIC support pKVM enforces: GICv2 is not supported in protected mode (vgic-v2.c) and GICv5-based guests aren't registered under pKVM (vgic-v5.c).
--- Documentation/virt/kvm/api.rst | 27 ++- .../virt/kvm/arm/fw-pseudo-registers.rst | 2 + Documentation/virt/kvm/arm/pkvm.rst | 164 +++++++++++++++++- Documentation/virt/kvm/devices/vcpu.rst | 3 +- 4 files changed, 185 insertions(+), 11 deletions(-)
Documentation/virt/kvm/api.rst +24 -3
@@ -1322,6 +1322,9 @@ guests, across different userspace implementations. Nevertheless, userspace can still emulate all Arm exceptions by manipulating individual registers using the KVM_SET_ONE_REG API. +For a protected VM, setting ext_dabt_pending returns -EPERM; see +Documentation/virt/kvm/arm/pkvm.rst. Injecting an SError is unaffected. + See KVM_GET_VCPU_EVENTS for the data structure. Calling this ioctl on a vCPU that hasn't been initialized will return
@@ -1545,7 +1548,9 @@ Possible values are: KVM_MP_STATE_RUNNABLE the vcpu is currently running [x86,arm64,riscv,loongarch] KVM_MP_STATE_UNINITIALIZED the vcpu is an application processor (AP) - which has not yet received an INIT signal [x86] + which has not yet received an INIT signal + [x86], or a protected VM's vcpu that the + guest holds powered off [arm64] KVM_MP_STATE_INIT_RECEIVED the vcpu has received an INIT signal, and is now ready for a SIPI [x86] KVM_MP_STATE_HALTED the vcpu has executed a HLT instruction and
@@ -1624,6 +1629,12 @@ For arm64/riscv: The only states that are valid are KVM_MP_STATE_STOPPED and KVM_MP_STATE_RUNNABLE which reflect if the vcpu should be paused or not. +On arm64, once a protected VM's vcpu has run, it reads as +KVM_MP_STATE_UNINITIALIZED while the guest holds it powered off, with +CPU_OFF or by not yet bringing it online with CPU_ON, and +KVM_MP_STATE_RUNNABLE and KVM_MP_STATE_SUSPENDED return -EPERM on it: only +the guest can power it on. See Documentation/virt/kvm/arm/pkvm.rst. + On LoongArch, only the KVM_MP_STATE_RUNNABLE state is used to reflect whether the vcpu is runnable.
@@ -2324,7 +2335,8 @@ Errors: ENOENT no such register EINVAL invalid register ID, or no such register or used with VMs in protected virtualization mode on s390 - EPERM (arm64) register access not allowed before vcpu finalization + EPERM (arm64) register access not allowed before vcpu + finalization, or after a protected VM's vcpu has run EBUSY (riscv) changing register value not allowed after the vcpu has run at least once ====== ============================================================
@@ -2949,7 +2961,8 @@ Errors include: ENOENT no such register EINVAL invalid register ID, or no such register or used with VMs in protected virtualization mode on s390 - EPERM (arm64) register access not allowed before vcpu finalization + EPERM (arm64) register access not allowed before vcpu + finalization, or after a protected VM's vcpu has run ======== ============================================================ (These error codes are indicative only: do not rely on a specific error
@@ -3484,6 +3497,7 @@ Errors: ====== ================================================================= EINVAL the target is unknown, or the combination of features is invalid. ENOENT a features bit specified is unknown. + EPERM the vcpu belongs to a protected VM and has already run. ====== ================================================================= This tells KVM what type of CPU to present to the guest, and what
@@ -3512,6 +3526,10 @@ after the vcpu has been run. This will reset the vcpu to its initial state. All calls to this function after the initial call must use the same target and same set of feature flags, otherwise EINVAL will be returned. +For a protected VM this ioctl returns EPERM once the vcpu has run, and the +features it accepts are restricted: KVM_ARM_VCPU_PSCI_0_2 is required. See +Documentation/virt/kvm/arm/pkvm.rst. + Possible features: - KVM_ARM_VCPU_POWER_OFF: Starts the CPU in a power-off state.
@@ -3790,6 +3808,9 @@ can be determined by querying the KVM_CAP_GUEST_DEBUG_HW_BPS and KVM_CAP_GUEST_DEBUG_HW_WPS capabilities which return a positive number indicating the number of supported registers. +On arm64, this ioctl returns -EPERM for a protected VM: debugging a +protected guest is not supported. See Documentation/virt/kvm/arm/pkvm.rst. + For ppc, the KVM_CAP_PPC_GUEST_DEBUG_SSTEP capability indicates whether the single-step debug event (KVM_GUESTDBG_SINGLESTEP) is supported.
Documentation/virt/kvm/arm/fw-pseudo-registers.rst +2 -0
@@ -33,6 +33,8 @@ The following registers are defined: highest PSCI version implemented by KVM and compatible with v0.2) - Allows any PSCI version implemented by KVM and compatible with v0.2 to be set with SET_ONE_REG + - Returns -EINVAL for a version below v1.1 on a protected VM (see + Documentation/virt/kvm/arm/pkvm.rst) - Affects the whole VM (even if the register view is per-vcpu) * KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_1:
Documentation/virt/kvm/arm/pkvm.rst +157 -7
204 lines not on this page · raw message
Documentation/virt/kvm/devices/vcpu.rst +2 -1
@@ -231,7 +231,8 @@ Returns: Specifies the base address of the stolen time structure for this VCPU. The base address must be 64 byte aligned and exist within a valid guest memory region. See Documentation/virt/kvm/arm/pvtime.rst for more information -including the layout of the stolen time structure. +including the layout of the stolen time structure. Stolen time is not +supported for protected VMs (see Documentation/virt/kvm/arm/pkvm.rst). 4. GROUP: KVM_VCPU_TSC_CTRL ===========================
Signature · 1 line
2.39.5
Fuad Tabba
reviewerHi Marc,
Here's a link to the series based on kvmarm/next, conflicts resolved:
https://android-kvm.googlesource.com/linux/+/refs/heads/tabba/pkvm-core-b-v4-kvmarm-next
Cheers,
/fuad
Quoted 131 lines · Fuad Tabba
Hi folks,
Changes since v3 [1]:
- A protected vCPU that EL2 holds powered off reads asKVM_MP_STATE_UNINITIALIZED, in place of a new bool. (Marc)
- The marshalling and PSCI patches access the host copy through the vCPU accessors, with every assignment on one line. (Marc)
- The PC adjustment pair is kvm_adjust_pc_get()/put() and tests for the hyp vCPU directly. (Marc)
-CPU_OFFclears the reset flag before its release of OFF, CPU_ON's relaxed cmpxchg is ordered by its control dependency, and theCPU_ONrollback stores OFF with a release.pkvm_reset_vcpu()carries the CPU_ON/CPU_OFF sequence as a comment and drops itsWARN_ON(). (Vincent, Will)
- Collected Reviewed-bys.Following the vCPU state-sync series [3], this series completes the job for protected VMs: a protected guest's register state stays at EL2, and the host sees only what handling each exit requires.
EL2 marshals a protected vCPU's state per exception class instead of copying the whole context both ways. It owns the vCPU's trap configuration, system register reset and HVC handling, and implements PSCI itself:
AFFINITY_INFOis left to the host, andCPU_ONandCPU_OFFare resolved at EL2 with the host only scheduling the target or stopping it. A protected guest's TRNG calls, which the host handled until now, returnNOT_SUPPORTEDuntil TRNG for protected guests follows. EL2 implements PSCI 1.1, so a protected guest also loses the functions the host supports above that version,SYSTEM_OFF2included. Host ioctls that would reach the state EL2 owns fail with-EPERM, so a protected VM's state isn't save/restorable. All of this is scoped toKVM_VM_TYPE_ARM_PROTECTED, andpkvm.rstdescribes the resulting API.The kvmtool changes that go with this are posted separately [4].
Patches 1 to 3 go out separately: the HCR_EL2.VSE fix [5], and the host vCPU VM read and pin fixes, patches 4 and 5 of the host hypercall fixes series [2]. None of the three is part of this series. They're carried so the series applies as is and Sashiko can run on it.
The
KVM_ARM_PREFERRED_TARGETdocumentation fix [6] went out just ahead of v1. Nothing here depends on it to apply, but patch 18 documents vCPU feature availability as something the capabilities report, whileapi.rst4.83 still points userspace at a bitmap that has always been empty.The series is structured as follows:
01: The HCR_EL2.VSE fix, posted separately. 02-03: The host vCPU VM read and pin fixes, posted separately. 04: Steal time disabled for protected VMs. 05-06: Per-exception-class entry handlers; EL2 owns a protectedvCPU's trap configuration.
07-09: Timer state, system register reset and HVC handling at EL2. 10-11: PSCI at EL2, and theKVM_ARM_VCPU_INITand PSCI version
restrictions.
12-14: Host PC adjustments blocked; an UNDEF at EL2 for exit
classes the host doesn't emulate; per-class state marshalling.
15-16: Host changes to private state, and host power-on of a vCPU
EL2 holds powered off, rejected.
17: Capability allowlist.
18: Documentation.Still to come: selftests, TRNG, self-hosted debug and SVE for protected guests, and much more, as separate series.
Based on v7.3-rc3 (
fd73f4a665989).Cheers,
/fuad[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/
[3] https://lore.kernel.org/all/[email protected]/
[4] https://lore.kernel.org/all/[email protected]/
[5] https://lore.kernel.org/all/[email protected]/
[6] https://lore.kernel.org/all/[email protected]/Fuad Tabba (16):
KVM: arm64: Sync HCR_EL2.VSE back to the host vCPU under pKVM
KVM: arm64: Validate the host vCPU's VM before reading it under pKVM
KVM: arm64: Pin the host vCPU before adjusting its PC under pKVM
KVM: arm64: Disable steal time for protected VMs
KVM: arm64: Skip fixed-feature state flush for protected vCPUs
KVM: arm64: Add system register reset framework for protected VMs
KVM: arm64: Implement HVC handling for protected guests at EL2
KVM: arm64: Handle PSCI calls for protected VMs at EL2
KVM: arm64: RestrictKVM_ARM_VCPU_INITand PSCI version for protected
VMs
KVM: arm64: Prevent host PC adjustments for protected vCPUs
KVM: arm64: Inject an UNDEF at EL2 for unhandled protected guest exits
KVM: arm64: Add per-EC entry/exit state marshalling for protected
guests
KVM: arm64: Reject host access to protected VM private state
KVM: arm64: Reject host power-on of a vCPU that EL2 holds powered off
KVM: arm64: Advertise the capabilities that protected VMs support
KVM: arm64: Document the protected VM userspace APIMarc Zyngier (2):
KVM: arm64: Introduce per-EC entry handlers for pKVM
KVM: arm64: Add {flush,sync}_hyp_timer_state() primitivesDocumentation/virt/kvm/api.rst | 27 +- .../virt/kvm/arm/fw-pseudo-registers.rst | 2 + Documentation/virt/kvm/arm/pkvm.rst | 164 ++++- Documentation/virt/kvm/devices/vcpu.rst | 3 +- arch/arm64/include/asm/kvm_host.h | 22 +- arch/arm64/include/asm/kvm_hyp.h | 4 + arch/arm64/include/asm/kvm_pkvm.h | 33 + arch/arm64/kvm/arm.c | 76 ++- arch/arm64/kvm/guest.c | 11 + arch/arm64/kvm/hyp/exception.c | 26 +- arch/arm64/kvm/hyp/include/hyp/adjust_pc.h | 48 ++ arch/arm64/kvm/hyp/include/nvhe/pkvm.h | 20 + arch/arm64/kvm/hyp/nvhe/hyp-main.c | 595 +++++++++++++++++- arch/arm64/kvm/hyp/nvhe/pkvm.c | 422 ++++++++++++- arch/arm64/kvm/hyp/nvhe/switch.c | 29 +- arch/arm64/kvm/hyp/nvhe/sys_regs.c | 60 ++ arch/arm64/kvm/hypercalls.c | 7 + arch/arm64/kvm/pkvm.c | 21 +- arch/arm64/kvm/psci.c | 11 +- arch/arm64/kvm/pvtime.c | 10 +- 20 files changed, 1503 insertions(+), 88 deletions(-)base-commit:
fd73f4a6659897191fa0d40695fe370925dd3780--
2.39.5
Quoted 14 lines · Fuad Tabba
Changes since v3 [1]:
- A protected vCPU that EL2 holds powered off reads asKVM_MP_STATE_UNINITIALIZED, in place of a new bool. (Marc)
- The marshalling and PSCI patches access the host copy through the vCPU accessors, with every assignment on one line. (Marc)
- The PC adjustment pair is kvm_adjust_pc_get()/put() and tests for the hyp vCPU directly. (Marc)
-CPU_OFFclears the reset flag before its release of OFF, CPU_ON's relaxed cmpxchg is ordered by its control dependency, and theCPU_ONrollback stores OFF with a release.pkvm_reset_vcpu()carries the CPU_ON/CPU_OFF sequence as a comment and drops itsWARN_ON(). (Vincent, Will)
- Collected Reviewed-bys.
Quoted 1 line
[...]
Applied to next, thanks!
[01/18] KVM: arm64: Sync HCR_EL2.VSE back to the host vCPU under pKVM
commit: 3708f6342f010d66a767b587dc881f18d95ceb3e
[02/18] KVM: arm64: Validate the host vCPU's VM before reading it under pKVM
commit: cf533b4d3630b229d78ad55567c5db2f365052a2
[03/18] KVM: arm64: Pin the host vCPU before adjusting its PC under pKVM
commit: a2bb5f5c67a95ac97fcd6859dab51be421cf26fe
[04/18] KVM: arm64: Disable steal time for protected VMs
commit: a5b0f36a045e0e96f8e2e044403ff8e6464995e1
[05/18] KVM: arm64: Introduce per-EC entry handlers for pKVM
commit: 66a5e82d2a8d113d5953d04e008447d6b779e03e
[06/18] KVM: arm64: Skip fixed-feature state flush for protected vCPUs
commit: 06499de8f998e0ec975cb73adde936c12a8a1200
[07/18] KVM: arm64: Add {flush,sync}_hyp_timer_state() primitives
commit: a8b2a7d0fe9c0eb493542bb1d306b68313e6a828
[08/18] KVM: arm64: Add system register reset framework for protected VMs
commit: ceaf9e57f88711fa3c7775ee4fea4f6731f1a71c
[09/18] KVM: arm64: Implement HVC handling for protected guests at EL2
commit: fe40ff87ead5c541ac4b42aa149b98052a1abca1
[10/18] KVM: arm64: Handle PSCI calls for protected VMs at EL2
commit: 9d4ec80be2eb4745d877c6989f1e59c62a2e4da4
[11/18] KVM: arm64: Restrict KVM_ARM_VCPU_INIT and PSCI version for protected VMs
commit: 0b7d843ef3bf0a791d9e92c8a85178fe2a5ee9ff
[12/18] KVM: arm64: Prevent host PC adjustments for protected vCPUs
commit: fc519dabd86b07b7f7e285ce68b2993d370f64d3
[13/18] KVM: arm64: Inject an UNDEF at EL2 for unhandled protected guest exits
commit: 3cfbad49ededd985555820d97b62a27f92506b52
[14/18] KVM: arm64: Add per-EC entry/exit state marshalling for protected guests
commit: 872383bd12e116fc79cb4487b1740c84f23c4c23
[15/18] KVM: arm64: Reject host access to protected VM private state
commit: ad4ebc3decaa19fad2c00868e237561b6ac004c6
[16/18] KVM: arm64: Reject host power-on of a vCPU that EL2 holds powered off
commit: 25d726dddc237043327aed9a2b51a5b6336bc31a
[17/18] KVM: arm64: Advertise the capabilities that protected VMs support
commit: 4957c24559e90592cfb60b6d4425296c2a7cd0e0
[18/18] KVM: arm64: Document the protected VM userspace API
commit: 61ace86519fb830a7b271c2a1037c8db5f2f3333Given how busy this merge window is, there were plenty of conflicts, most of them with your own patches. Please have a look at the way I resolved them and let me know if anything is on the bogus side of wrong...
Cheers,
M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
Fuad Tabba
reviewerHi Marc,
On Fri, 02 Oct 2026 16:55:26 +0100, Marc Zyngier <[email protected]> wrote:
[...]
Quoted 3 lines
Given how busy this merge window is, there were plenty of conflicts, most of them with your own patches. Please have a look at the way I resolved them and let me know if anything is on the bogus side of wrong...
Thanks for taking these, and for sorting out the conflicts. Most of the resolution matches mine, but two things in the pkvm-state merge are off:
- In __kvm_vcpu_run(), the !host_vcpu and SVCR bail-outs still return -EINVAL. Patch 10 changed them to ARM_EXCEPTION_IL, since the host reads bit 31 of -EINVAL as a pending SError.
- In __kvm_adjust_pc(), the pin path uses kern_hyp_va() on the __kern pointer, which sparse flags. It needs kern_hyp_va_host().
The diff below fixes both. It also drops the set_bit()s in pkvm_init_features_from_host() that kvm_pkvm_vcpu_allowed_features() already does, and realigns the esr_dabt_is_cm() line. Could you fold it into the merge resolution?
With it applied on next, I booted protected and non-protected guests, ran CPU hotplug in both, and ran the pKVM selftest (from my tree), all passing.
arch/arm64/kvm/hyp/nvhe/hyp-main.c +4 -4
@@ -244,7 +244,7 @@ static void handle_pvm_entry_dabt(struct pkvm_hyp_vcpu *hyp_vcpu)/* Handle PC increment on MMIO, or on a CMO the host skipped */ pc_update =vcpu_get_flag (host_vcpu,INCREMENT_PC ) && - (vcpu->mmio_needed ||esr_dabt_is_cm (kvm_vcpu_get_esr (vcpu))); + (vcpu->mmio_needed ||esr_dabt_is_cm (kvm_vcpu_get_esr (vcpu)));if (pc_update) {vcpu_clear_flag (vcpu,PC_UPDATE_REQ );kvm_incr_pc (vcpu);
@@ -914,7 +914,7 @@ DEFINE_KVM_HOST_HCALL(int, __kvm_vcpu_run, struct kvm_vcpu __kern *, vcpu) host_vcpu =__get_host_hyp_vcpus (kern_hyp_va_host (vcpu), &hyp_vcpu);if (!host_vcpu) -return -EINVAL ; +return ret;if (unlikely (hyp_vcpu)) {/*
@@ -924,7 +924,7 @@ DEFINE_KVM_HOST_HCALL(int, __kvm_vcpu_run, struct kvm_vcpu __kern *, vcpu)* is misbehaving. */ if (unlikely (system_supports_sme () &&read_sysreg_s (SYS_SVCR ))) -return -EINVAL ; +return ret;/* * ON has a single writer, pkvm_reset_vcpu() on this CPU, so
@@ -1122,7 +1122,7 @@ DEFINE_KVM_HOST_HCALL(void, __kvm_adjust_pc,* for memory the host isn't sharing, a bad pointer, so the request * is dropped. */ - host_vcpu =kern_hyp_va (vcpu); + host_vcpu =kern_hyp_va_host (vcpu);if (hyp_pin_shared_mem (host_vcpu, host_vcpu +1 ))return ;
arch/arm64/kvm/hyp/nvhe/pkvm.c +0 -14
@@ -413,20 +413,6 @@ static void pkvm_init_features_from_host(struct pkvm_hyp_vm *hyp_vm, const struckvm_pkvm_vcpu_allowed_features (kvm, allowed_features); -set_bit (KVM_ARM_VCPU_PSCI_0_2 , allowed_features); - -if (kvm_pkvm_ext_allowed (kvm,KVM_CAP_ARM_PMU_V3 )) -set_bit (KVM_ARM_VCPU_PMU_V3 , allowed_features); - -if (kvm_pkvm_ext_allowed (kvm,KVM_CAP_ARM_PTRAUTH_ADDRESS )) -set_bit (KVM_ARM_VCPU_PTRAUTH_ADDRESS , allowed_features); - -if (kvm_pkvm_ext_allowed (kvm,KVM_CAP_ARM_PTRAUTH_GENERIC )) -set_bit (KVM_ARM_VCPU_PTRAUTH_GENERIC , allowed_features); - -if (kvm_pkvm_ext_allowed (kvm,KVM_CAP_ARM_SVE )) -set_bit (KVM_ARM_VCPU_SVE , allowed_features); -bitmap_and (kvm->arch.vcpu_features, host_kvm->arch.vcpu_features, allowed_features,KVM_VCPU_MAX_FEATURES ); out:
Cheers,
/fuad
Marc Zyngier
maintainerQuoted 21 lines · Fuad Tabba
Hi Marc,
On Fri, 02 Oct 2026 16:55:26 +0100, Marc Zyngier <[email protected]> wrote:
[...]
Quoted 3 lines
Given how busy this merge window is, there were plenty of conflicts, most of them with your own patches. Please have a look at the way I resolved them and let me know if anything is on the bogus side of wrong...
Thanks for taking these, and for sorting out the conflicts. Most of the resolution matches mine, but two things in the pkvm-state merge are off:
- In
__kvm_vcpu_run(), the !host_vcpu and SVCR bail-outs still return-EINVAL. Patch 10 changed them toARM_EXCEPTION_IL, since the host reads bit 31 of-EINVALas a pending SError.- In
__kvm_adjust_pc(), the pin path useskern_hyp_va()on the__kernpointer, which sparse flags. It needskern_hyp_va_host().
Quoted 8 lines
The diff below fixes both. It also drops the set_bit()s in
pkvm_init_features_from_host()thatkvm_pkvm_vcpu_allowed_features()already does, and realigns theesr_dabt_is_cm()line. Could you fold it into the merge resolution?With it applied on next, I booted protected and non-protected guests, ran CPU hotplug in both, and ran the pKVM selftest (from my tree), all passing.
Thanks for having a look at it. I've now applied the new resolution and pushed out the updated branch,
Cheers,
M.
Signature · 1 line
Without deviation from the norm, progress is not possible.