Skip to contentSkip to conversation

KVM: arm64: Rework pKVM vCPU state synchronisation

PATCH In review Fuad Tabba 4 participants 25 messages Jun 19 – Jun 22

Subsystems (2)
lore
Share & tools
lore

Fuad Tabba

author
#0 lore raw

v2 cover KVM: arm64: Rework pKVM vCPU state synchronisation

Hi folks,

Changes since v1 [2]:
- Dropped the guard()/scoped_guard() conversion patches: standalone churn on code this series does not otherwise rework. (Marc)
- Rebased onto kvmarm/next. The VGIC flush primitive now bounds used_lrs using the cached hyp_gicv3_nr_lr instead of reading ICH_VTR_EL2 on every entry. (Marc)
- Grouped the PKVM_HOST_STATE_DIRTY flag with the other iflags and clarified its comment. (Marc)
- Sync PSTATE alongside PC on every non-protected exit, and sync+dirty before host-side SError injection so the syndrome is not dropped. (sashiko)
- Various cleanups and tidying up. (Vincent)

Building on Will's pKVM infrastructure series [1], this series reworks how pKVM moves vCPU state between the host and EL2, and stops copying a non-protected guest's state on every world switch.

EL2 gains proper primitives for the state it transfers: vCPU lookup helpers, and VGIC flush/sync that reduces how much host state EL2 dereferences. The series also moves some preparatory code (such as sys reg access and PSCI helpers) to shared headers and HYP, and implements lazy copying of a non-protected guest's register state back to the host until the host actually needs it, instead of on every exit.

This is the first of two series moving pKVM vCPU state management to EL2. The follow-up completes the job for protected VMs: state isolation, PSCI handling at EL2, and the resulting API behaviour.

The series is structured as follows:

01-04: Preparatory refactoring (MPIDR, sys reg access, vCPU reset, PSCI
helpers) to shared headers and HYP.

  05:     Host and hypervisor vCPU lookup primitives.
  06-07:  VGIC: reduce EL2's exposure to host state, add flush/sync primitives.
  08:     Lazy state sync for non-protected guests.

Based on kvmarm/next.

[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/

Cheers,
/fuad

Fuad Tabba (5):
KVM: arm64: Extract MPIDR computation into a shared header
KVM: arm64: Make vcpu_{read,write}_sys_reg available to HYP code
KVM: arm64: Factor out reusable vCPU reset helpers
KVM: arm64: Move PSCI helper functions to a shared header
KVM: arm64: Implement lazy vCPU state sync for non-protected guests

Marc Zyngier (3):
KVM: arm64: Add host and hypervisor vCPU lookup primitives
KVM: arm64: Minimise EL2's exposure of host VGIC state during world
switch
KVM: arm64: Add primitives to flush/sync the VGIC state at EL2

 arch/arm64/include/asm/kvm_arm.h     |  12 ++
 arch/arm64/include/asm/kvm_asm.h     |   1 +
 arch/arm64/include/asm/kvm_emulate.h |  79 +++++++-
 arch/arm64/include/asm/kvm_host.h    |   2 +
 arch/arm64/kvm/arm.c                 |   7 +
 arch/arm64/kvm/handle_exit.c         |  30 ++++
 arch/arm64/kvm/hyp/exception.c       |  34 +---
 arch/arm64/kvm/hyp/nvhe/hyp-main.c   | 258 +++++++++++++++++++++++----
 arch/arm64/kvm/psci.c                |  30 +---
 arch/arm64/kvm/reset.c               |  60 +------
 arch/arm64/kvm/sys_regs.c            |  14 +-
 arch/arm64/kvm/sys_regs.h            |  19 ++
 include/kvm/arm_psci.h               |  27 +++
 13 files changed, 410 insertions(+), 163 deletions(-)
Signature · 1 line
2.55.0.rc0.738.g0c8ab3ebcc-goog
author
#1 lore raw

PATCH v2 1/8 KVM: arm64: Extract MPIDR computation into a shared header

Extract the vCPU MPIDR computation embedded in reset_mpidr() into a kvm_calculate_mpidr() inline in sys_regs.h, so it can be computed without duplicating the logic. A follow-up series reuses it to reset protected vCPUs at EL2.

No functional change intended.

---
 arch/arm64/kvm/sys_regs.c | 14 +-------------
 arch/arm64/kvm/sys_regs.h | 19 +++++++++++++++++++
 2 files changed, 20 insertions(+), 13 deletions(-)
arch/arm64/kvm/sys_regs.c +1 -13
index 33c921df19b5..674fabe1d40d 100644
@@ -976,21 +976,9 @@ static u64 reset_actlr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r)
 
 static u64 reset_mpidr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r)
 {
-	u64 mpidr;
+	u64 mpidr = kvm_calculate_mpidr(vcpu);
 
-	/*
-	 * Map the vcpu_id into the first three affinity level fields of
-	 * the MPIDR. We limit the number of VCPUs in level 0 due to a
-	 * limitation to 16 CPUs in that level in the ICC_SGIxR registers
-	 * of the GICv3 to be able to address each CPU directly when
-	 * sending IPIs.
-	 */
-	mpidr = (vcpu->vcpu_id & 0x0f) << MPIDR_LEVEL_SHIFT(0);
-	mpidr |= ((vcpu->vcpu_id >> 4) & 0xff) << MPIDR_LEVEL_SHIFT(1);
-	mpidr |= ((vcpu->vcpu_id >> 12) & 0xff) << MPIDR_LEVEL_SHIFT(2);
-	mpidr |= (1ULL << 31);
 	vcpu_write_sys_reg(vcpu, mpidr, MPIDR_EL1);
-
 	return mpidr;
 }
 
arch/arm64/kvm/sys_regs.h +19 -0
index 2a983664220c..bd56a45abbf9 100644
@@ -222,6 +222,25 @@ find_reg(const struct sys_reg_params *params, const struct sys_reg_desc table[],
 	return __inline_bsearch((void *)pval, table, num, sizeof(table[0]), match_sys_reg);
 }
 
+static inline u64 kvm_calculate_mpidr(const struct kvm_vcpu *vcpu)
+{
+	u64 mpidr;
+
+	/*
+	 * Map the vcpu_id into the first three affinity level fields of
+	 * the MPIDR. We limit the number of VCPUs in level 0 due to a
+	 * limitation to 16 CPUs in that level in the ICC_SGIxR registers
+	 * of the GICv3 to be able to address each CPU directly when
+	 * sending IPIs.
+	 */
+	mpidr = (vcpu->vcpu_id & 0x0f) << MPIDR_LEVEL_SHIFT(0);
+	mpidr |= ((vcpu->vcpu_id >> 4) & 0xff) << MPIDR_LEVEL_SHIFT(1);
+	mpidr |= ((vcpu->vcpu_id >> 12) & 0xff) << MPIDR_LEVEL_SHIFT(2);
+	mpidr |= (1ULL << 31);
+
+	return mpidr;
+}
+
 const struct sys_reg_desc *get_reg_by_id(u64 id,
 					 const struct sys_reg_desc table[],
 					 unsigned int num);
Signature · 1 line
2.55.0.rc0.738.g0c8ab3ebcc-goog
Reviewed-by on v2 1/8
#2 lore raw

Re: [PATCH v2 1/8] KVM: arm64: Extract MPIDR computation into a shared header

Quoted 8 lines · Fuad Tabba

Extract the vCPU MPIDR computation embedded in reset_mpidr() into a kvm_calculate_mpidr() inline in sys_regs.h, so it can be computed without duplicating the logic. A follow-up series reuses it to reset protected vCPUs at EL2.

No functional change intended.

Quoted 65 lines
---
 arch/arm64/kvm/sys_regs.c | 14 +-------------
 arch/arm64/kvm/sys_regs.h | 19 +++++++++++++++++++
 2 files changed, 20 insertions(+), 13 deletions(-)
arch/arm64/kvm/sys_regs.c +1 -13
index 33c921df19b5..674fabe1d40d 100644
@@ -976,21 +976,9 @@ static u64 reset_actlr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r)
 
 static u64 reset_mpidr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r)
 {
-	u64 mpidr;
+	u64 mpidr = kvm_calculate_mpidr(vcpu);
 
-	/*
-	 * Map the vcpu_id into the first three affinity level fields of
-	 * the MPIDR. We limit the number of VCPUs in level 0 due to a
-	 * limitation to 16 CPUs in that level in the ICC_SGIxR registers
-	 * of the GICv3 to be able to address each CPU directly when
-	 * sending IPIs.
-	 */
-	mpidr = (vcpu->vcpu_id & 0x0f) << MPIDR_LEVEL_SHIFT(0);
-	mpidr |= ((vcpu->vcpu_id >> 4) & 0xff) << MPIDR_LEVEL_SHIFT(1);
-	mpidr |= ((vcpu->vcpu_id >> 12) & 0xff) << MPIDR_LEVEL_SHIFT(2);
-	mpidr |= (1ULL << 31);
 	vcpu_write_sys_reg(vcpu, mpidr, MPIDR_EL1);
-
 	return mpidr;
 }
 
arch/arm64/kvm/sys_regs.h +19 -0
index 2a983664220c..bd56a45abbf9 100644
@@ -222,6 +222,25 @@ find_reg(const struct sys_reg_params *params, const struct sys_reg_desc table[],
 	return __inline_bsearch((void *)pval, table, num, sizeof(table[0]), match_sys_reg);
 }
 
+static inline u64 kvm_calculate_mpidr(const struct kvm_vcpu *vcpu)
+{
+	u64 mpidr;
+
+	/*
+	 * Map the vcpu_id into the first three affinity level fields of
+	 * the MPIDR. We limit the number of VCPUs in level 0 due to a
+	 * limitation to 16 CPUs in that level in the ICC_SGIxR registers
+	 * of the GICv3 to be able to address each CPU directly when
+	 * sending IPIs.
+	 */
+	mpidr = (vcpu->vcpu_id & 0x0f) << MPIDR_LEVEL_SHIFT(0);
+	mpidr |= ((vcpu->vcpu_id >> 4) & 0xff) << MPIDR_LEVEL_SHIFT(1);
+	mpidr |= ((vcpu->vcpu_id >> 12) & 0xff) << MPIDR_LEVEL_SHIFT(2);
+	mpidr |= (1ULL << 31);
+
+	return mpidr;
+}
+
 const struct sys_reg_desc *get_reg_by_id(u64 id,
 					 const struct sys_reg_desc table[],
 					 unsigned int num);
Signature · 1 line
2.55.0.rc0.738.g0c8ab3ebcc-goog
author
#3 lore raw

PATCH v2 2/8 KVM: arm64: Make vcpu_{read,write}_sys_reg available to HYP code

In reply to Fuad Tabba

The vcpu_{read,write}_sys_reg() accessors are host-only, so helpers built on them such as kvm_vcpu_set_be()/kvm_vcpu_is_be() cannot be shared with hyp code. exception.c already wraps them in __vcpu_{read,write}_sys_reg(), which pick the host- or hyp-side accessor via has_vhe() and so are valid in any context.

Move those wrappers to kvm_emulate.h as kvm_vcpu_{read,write}_sys_reg() and switch the callers over, so a follow-up series can share that emulation code at EL2.

No functional change intended.

---
 arch/arm64/include/asm/kvm_emulate.h | 22 +++++++++++++++---
 arch/arm64/kvm/hyp/exception.c       | 34 ++++++++--------------------
 2 files changed, 28 insertions(+), 28 deletions(-)
arch/arm64/include/asm/kvm_emulate.h +19 -3
index 5bf3d7e1d92c..80b30fead3d1 100644
@@ -506,6 +506,22 @@ static inline unsigned long kvm_vcpu_get_mpidr_aff(struct kvm_vcpu *vcpu)
 	return __vcpu_sys_reg(vcpu, MPIDR_EL1) & MPIDR_HWID_BITMASK;
 }
 
+static inline u64 kvm_vcpu_read_sys_reg(const struct kvm_vcpu *vcpu, int reg)
+{
+	if (has_vhe())
+		return vcpu_read_sys_reg(vcpu, reg);
+
+	return __vcpu_sys_reg(vcpu, reg);
+}
+
+static inline void kvm_vcpu_write_sys_reg(struct kvm_vcpu *vcpu, u64 val, int reg)
+{
+	if (has_vhe())
+		vcpu_write_sys_reg(vcpu, val, reg);
+	else
+		__vcpu_assign_sys_reg(vcpu, reg, val);
+}
+
 static inline void kvm_vcpu_set_be(struct kvm_vcpu *vcpu)
 {
 	if (vcpu_mode_is_32bit(vcpu)) {
@@ -516,9 +532,9 @@ static inline void kvm_vcpu_set_be(struct kvm_vcpu *vcpu)
 
 		r = vcpu_has_nv(vcpu) ? SCTLR_EL2 : SCTLR_EL1;
 
-		sctlr = vcpu_read_sys_reg(vcpu, r);
+		sctlr = kvm_vcpu_read_sys_reg(vcpu, r);
 		sctlr |= SCTLR_ELx_EE;
-		vcpu_write_sys_reg(vcpu, sctlr, r);
+		kvm_vcpu_write_sys_reg(vcpu, sctlr, r);
 	}
 }
 
@@ -533,7 +549,7 @@ static inline bool kvm_vcpu_is_be(struct kvm_vcpu *vcpu)
 	r = is_hyp_ctxt(vcpu) ? SCTLR_EL2 : SCTLR_EL1;
 	bit = vcpu_mode_priv(vcpu) ? SCTLR_ELx_EE : SCTLR_EL1_E0E;
 
-	return vcpu_read_sys_reg(vcpu, r) & bit;
+	return kvm_vcpu_read_sys_reg(vcpu, r) & bit;
 }
 
 static inline unsigned long vcpu_data_guest_to_host(struct kvm_vcpu *vcpu,
arch/arm64/kvm/hyp/exception.c +9 -25
index bef40ddb16db..2cb68dc7d441 100644
@@ -20,22 +20,6 @@
 #error Hypervisor code only!
 #endif
 
-static inline u64 __vcpu_read_sys_reg(const struct kvm_vcpu *vcpu, int reg)
-{
-	if (has_vhe())
-		return vcpu_read_sys_reg(vcpu, reg);
-
-	return __vcpu_sys_reg(vcpu, reg);
-}
-
-static inline void __vcpu_write_sys_reg(struct kvm_vcpu *vcpu, u64 val, int reg)
-{
-	if (has_vhe())
-		vcpu_write_sys_reg(vcpu, val, reg);
-	else
-		__vcpu_assign_sys_reg(vcpu, reg, val);
-}
-
 static void __vcpu_write_spsr(struct kvm_vcpu *vcpu, unsigned long target_mode,
 			      u64 val)
 {
@@ -101,14 +85,14 @@ static void enter_exception64(struct kvm_vcpu *vcpu, unsigned long target_mode,
 
 	switch (target_mode) {
 	case PSR_MODE_EL1h:
-		vbar = __vcpu_read_sys_reg(vcpu, VBAR_EL1);
-		sctlr = __vcpu_read_sys_reg(vcpu, SCTLR_EL1);
-		__vcpu_write_sys_reg(vcpu, *vcpu_pc(vcpu), ELR_EL1);
+		vbar = kvm_vcpu_read_sys_reg(vcpu, VBAR_EL1);
+		sctlr = kvm_vcpu_read_sys_reg(vcpu, SCTLR_EL1);
+		kvm_vcpu_write_sys_reg(vcpu, *vcpu_pc(vcpu), ELR_EL1);
 		break;
 	case PSR_MODE_EL2h:
-		vbar = __vcpu_read_sys_reg(vcpu, VBAR_EL2);
-		sctlr = __vcpu_read_sys_reg(vcpu, SCTLR_EL2);
-		__vcpu_write_sys_reg(vcpu, *vcpu_pc(vcpu), ELR_EL2);
+		vbar = kvm_vcpu_read_sys_reg(vcpu, VBAR_EL2);
+		sctlr = kvm_vcpu_read_sys_reg(vcpu, SCTLR_EL2);
+		kvm_vcpu_write_sys_reg(vcpu, *vcpu_pc(vcpu), ELR_EL2);
 		break;
 	default:
 		/* Don't do that */
@@ -185,7 +169,7 @@ static void enter_exception64(struct kvm_vcpu *vcpu, unsigned long target_mode,
  */
 static unsigned long get_except32_cpsr(struct kvm_vcpu *vcpu, u32 mode)
 {
-	u32 sctlr = __vcpu_read_sys_reg(vcpu, SCTLR_EL1);
+	u32 sctlr = kvm_vcpu_read_sys_reg(vcpu, SCTLR_EL1);
 	unsigned long old, new;
 
 	old = *vcpu_cpsr(vcpu);
@@ -281,7 +265,7 @@ static void enter_exception32(struct kvm_vcpu *vcpu, u32 mode, u32 vect_offset)
 {
 	unsigned long spsr = *vcpu_cpsr(vcpu);
 	bool is_thumb = (spsr & PSR_AA32_T_BIT);
-	u32 sctlr = __vcpu_read_sys_reg(vcpu, SCTLR_EL1);
+	u32 sctlr = kvm_vcpu_read_sys_reg(vcpu, SCTLR_EL1);
 	u32 return_address;
 
 	*vcpu_cpsr(vcpu) = get_except32_cpsr(vcpu, mode);
@@ -305,7 +289,7 @@ static void enter_exception32(struct kvm_vcpu *vcpu, u32 mode, u32 vect_offset)
 	if (sctlr & (1 << 13))
 		vect_offset += 0xffff0000;
 	else /* always have security exceptions */
-		vect_offset += __vcpu_read_sys_reg(vcpu, VBAR_EL1);
+		vect_offset += kvm_vcpu_read_sys_reg(vcpu, VBAR_EL1);
 
 	*vcpu_pc(vcpu) = vect_offset;
 }
Signature · 1 line
2.55.0.rc0.738.g0c8ab3ebcc-goog
Reviewed-by on v2 2/8
#4 lore raw

Re: [PATCH v2 2/8] KVM: arm64: Make vcpu_{read,write}_sys_reg available to HYP code

Quoted 6 lines · Fuad Tabba

The vcpu_{read,write}_sys_reg() accessors are host-only, so helpers built on them such as kvm_vcpu_set_be()/kvm_vcpu_is_be() cannot be shared with hyp code. exception.c already wraps them in __vcpu_{read,write}_sys_reg(), which pick the host- or hyp-side accessor via has_vhe() and so are valid in any context.

Quoted 7 lines

Move those wrappers to kvm_emulate.h as kvm_vcpu_{read,write}_sys_reg() and switch the callers over, so a follow-up series can share that emulation code at EL2.

No functional change intended.

Quoted 131 lines
---
 arch/arm64/include/asm/kvm_emulate.h | 22 +++++++++++++++---
 arch/arm64/kvm/hyp/exception.c       | 34 ++++++++--------------------
 2 files changed, 28 insertions(+), 28 deletions(-)
arch/arm64/include/asm/kvm_emulate.h +19 -3
index 5bf3d7e1d92c..80b30fead3d1 100644
@@ -506,6 +506,22 @@ static inline unsigned long kvm_vcpu_get_mpidr_aff(struct kvm_vcpu *vcpu)
 	return __vcpu_sys_reg(vcpu, MPIDR_EL1) & MPIDR_HWID_BITMASK;
 }
 
+static inline u64 kvm_vcpu_read_sys_reg(const struct kvm_vcpu *vcpu, int reg)
+{
+	if (has_vhe())
+		return vcpu_read_sys_reg(vcpu, reg);
+
+	return __vcpu_sys_reg(vcpu, reg);
+}
+
+static inline void kvm_vcpu_write_sys_reg(struct kvm_vcpu *vcpu, u64 val, int reg)
+{
+	if (has_vhe())
+		vcpu_write_sys_reg(vcpu, val, reg);
+	else
+		__vcpu_assign_sys_reg(vcpu, reg, val);
+}
+
 static inline void kvm_vcpu_set_be(struct kvm_vcpu *vcpu)
 {
 	if (vcpu_mode_is_32bit(vcpu)) {
@@ -516,9 +532,9 @@ static inline void kvm_vcpu_set_be(struct kvm_vcpu *vcpu)
 
 		r = vcpu_has_nv(vcpu) ? SCTLR_EL2 : SCTLR_EL1;
 
-		sctlr = vcpu_read_sys_reg(vcpu, r);
+		sctlr = kvm_vcpu_read_sys_reg(vcpu, r);
 		sctlr |= SCTLR_ELx_EE;
-		vcpu_write_sys_reg(vcpu, sctlr, r);
+		kvm_vcpu_write_sys_reg(vcpu, sctlr, r);
 	}
 }
 
@@ -533,7 +549,7 @@ static inline bool kvm_vcpu_is_be(struct kvm_vcpu *vcpu)
 	r = is_hyp_ctxt(vcpu) ? SCTLR_EL2 : SCTLR_EL1;
 	bit = vcpu_mode_priv(vcpu) ? SCTLR_ELx_EE : SCTLR_EL1_E0E;
 
-	return vcpu_read_sys_reg(vcpu, r) & bit;
+	return kvm_vcpu_read_sys_reg(vcpu, r) & bit;
 }
 
 static inline unsigned long vcpu_data_guest_to_host(struct kvm_vcpu *vcpu,
arch/arm64/kvm/hyp/exception.c +9 -25
index bef40ddb16db..2cb68dc7d441 100644
@@ -20,22 +20,6 @@
 #error Hypervisor code only!
 #endif
 
-static inline u64 __vcpu_read_sys_reg(const struct kvm_vcpu *vcpu, int reg)
-{
-	if (has_vhe())
-		return vcpu_read_sys_reg(vcpu, reg);
-
-	return __vcpu_sys_reg(vcpu, reg);
-}
-
-static inline void __vcpu_write_sys_reg(struct kvm_vcpu *vcpu, u64 val, int reg)
-{
-	if (has_vhe())
-		vcpu_write_sys_reg(vcpu, val, reg);
-	else
-		__vcpu_assign_sys_reg(vcpu, reg, val);
-}
-
 static void __vcpu_write_spsr(struct kvm_vcpu *vcpu, unsigned long target_mode,
 			      u64 val)
 {
@@ -101,14 +85,14 @@ static void enter_exception64(struct kvm_vcpu *vcpu, unsigned long target_mode,
 
 	switch (target_mode) {
 	case PSR_MODE_EL1h:
-		vbar = __vcpu_read_sys_reg(vcpu, VBAR_EL1);
-		sctlr = __vcpu_read_sys_reg(vcpu, SCTLR_EL1);
-		__vcpu_write_sys_reg(vcpu, *vcpu_pc(vcpu), ELR_EL1);
+		vbar = kvm_vcpu_read_sys_reg(vcpu, VBAR_EL1);
+		sctlr = kvm_vcpu_read_sys_reg(vcpu, SCTLR_EL1);
+		kvm_vcpu_write_sys_reg(vcpu, *vcpu_pc(vcpu), ELR_EL1);
 		break;
 	case PSR_MODE_EL2h:
-		vbar = __vcpu_read_sys_reg(vcpu, VBAR_EL2);
-		sctlr = __vcpu_read_sys_reg(vcpu, SCTLR_EL2);
-		__vcpu_write_sys_reg(vcpu, *vcpu_pc(vcpu), ELR_EL2);
+		vbar = kvm_vcpu_read_sys_reg(vcpu, VBAR_EL2);
+		sctlr = kvm_vcpu_read_sys_reg(vcpu, SCTLR_EL2);
+		kvm_vcpu_write_sys_reg(vcpu, *vcpu_pc(vcpu), ELR_EL2);
 		break;
 	default:
 		/* Don't do that */
@@ -185,7 +169,7 @@ static void enter_exception64(struct kvm_vcpu *vcpu, unsigned long target_mode,
  */
 static unsigned long get_except32_cpsr(struct kvm_vcpu *vcpu, u32 mode)
 {
-	u32 sctlr = __vcpu_read_sys_reg(vcpu, SCTLR_EL1);
+	u32 sctlr = kvm_vcpu_read_sys_reg(vcpu, SCTLR_EL1);
 	unsigned long old, new;
 
 	old = *vcpu_cpsr(vcpu);
@@ -281,7 +265,7 @@ static void enter_exception32(struct kvm_vcpu *vcpu, u32 mode, u32 vect_offset)
 {
 	unsigned long spsr = *vcpu_cpsr(vcpu);
 	bool is_thumb = (spsr & PSR_AA32_T_BIT);
-	u32 sctlr = __vcpu_read_sys_reg(vcpu, SCTLR_EL1);
+	u32 sctlr = kvm_vcpu_read_sys_reg(vcpu, SCTLR_EL1);
 	u32 return_address;
 
 	*vcpu_cpsr(vcpu) = get_except32_cpsr(vcpu, mode);
@@ -305,7 +289,7 @@ static void enter_exception32(struct kvm_vcpu *vcpu, u32 mode, u32 vect_offset)
 	if (sctlr & (1 << 13))
 		vect_offset += 0xffff0000;
 	else /* always have security exceptions */
-		vect_offset += __vcpu_read_sys_reg(vcpu, VBAR_EL1);
+		vect_offset += kvm_vcpu_read_sys_reg(vcpu, VBAR_EL1);
 
 	*vcpu_pc(vcpu) = vect_offset;
 }
Signature · 1 line
2.55.0.rc0.738.g0c8ab3ebcc-goog
author
#5 lore raw

PATCH v2 3/8 KVM: arm64: Factor out reusable vCPU reset helpers

In reply to Fuad Tabba

Pull the reusable pieces out of kvm_reset_vcpu(): expose the reset PSTATE values in kvm_arm.h, and split the core register reset and the PSCI-driven reset into kvm_reset_vcpu_core() and kvm_reset_vcpu_psci(). A follow-up series reuses these to reset protected vCPUs at EL2.

No functional change intended.

---
 arch/arm64/include/asm/kvm_arm.h     | 12 ++++++
 arch/arm64/include/asm/kvm_emulate.h | 57 ++++++++++++++++++++++++++
 arch/arm64/kvm/reset.c               | 60 ++--------------------------
 3 files changed, 72 insertions(+), 57 deletions(-)
arch/arm64/include/asm/kvm_arm.h +12 -0
index 3f9233b5a130..aba4ec09acd2 100644
@@ -348,4 +348,16 @@
 	{ PSR_AA32_MODE_UND,	"32-bit UND" },	\
 	{ PSR_AA32_MODE_SYS,	"32-bit SYS" }
 
+/*
+ * ARMv8 Reset Values
+ */
+#define VCPU_RESET_PSTATE_EL1	(PSR_MODE_EL1h | PSR_A_BIT | PSR_I_BIT | \
+				 PSR_F_BIT | PSR_D_BIT)
+
+#define VCPU_RESET_PSTATE_EL2	(PSR_MODE_EL2h | PSR_A_BIT | PSR_I_BIT | \
+				 PSR_F_BIT | PSR_D_BIT)
+
+#define VCPU_RESET_PSTATE_SVC	(PSR_AA32_MODE_SVC | PSR_AA32_A_BIT | \
+				 PSR_AA32_I_BIT | PSR_AA32_F_BIT)
+
 #endif /* __ARM64_KVM_ARM_H__ */
arch/arm64/include/asm/kvm_emulate.h +57 -0

61 lines not on this page · raw message

arch/arm64/kvm/reset.c +3 -57

85 lines not on this page · raw message

Signature · 1 line
2.55.0.rc0.738.g0c8ab3ebcc-goog
Reviewed-by on v2 3/8
#6 lore raw

Re: [PATCH v2 3/8] KVM: arm64: Factor out reusable vCPU reset helpers

Quoted 8 lines · Fuad Tabba

Pull the reusable pieces out of kvm_reset_vcpu(): expose the reset PSTATE values in kvm_arm.h, and split the core register reset and the PSCI-driven reset into kvm_reset_vcpu_core() and kvm_reset_vcpu_psci(). A follow-up series reuses these to reset protected vCPUs at EL2.

No functional change intended.

Quoted 189 lines
---
 arch/arm64/include/asm/kvm_arm.h     | 12 ++++++
 arch/arm64/include/asm/kvm_emulate.h | 57 ++++++++++++++++++++++++++
 arch/arm64/kvm/reset.c               | 60 ++--------------------------
 3 files changed, 72 insertions(+), 57 deletions(-)
arch/arm64/include/asm/kvm_arm.h +12 -0
index 3f9233b5a130..aba4ec09acd2 100644
@@ -348,4 +348,16 @@
 	{ PSR_AA32_MODE_UND,	"32-bit UND" },	\
 	{ PSR_AA32_MODE_SYS,	"32-bit SYS" }
 
+/*
+ * ARMv8 Reset Values
+ */
+#define VCPU_RESET_PSTATE_EL1	(PSR_MODE_EL1h | PSR_A_BIT | PSR_I_BIT | \
+				 PSR_F_BIT | PSR_D_BIT)
+
+#define VCPU_RESET_PSTATE_EL2	(PSR_MODE_EL2h | PSR_A_BIT | PSR_I_BIT | \
+				 PSR_F_BIT | PSR_D_BIT)
+
+#define VCPU_RESET_PSTATE_SVC	(PSR_AA32_MODE_SVC | PSR_AA32_A_BIT | \
+				 PSR_AA32_I_BIT | PSR_AA32_F_BIT)
+
 #endif /* __ARM64_KVM_ARM_H__ */
arch/arm64/include/asm/kvm_emulate.h +57 -0
index 80b30fead3d1..2385d8855fcf 100644
@@ -704,4 +704,61 @@ static inline void vcpu_set_hcrx(struct kvm_vcpu *vcpu)
 			vcpu->arch.hcrx_el2 |= HCRX_EL2_EnASR;
 	}
 }
+
+/* Reset a vcpu's core registers. */
+static inline void kvm_reset_vcpu_core(struct kvm_vcpu *vcpu)
+{
+	u32 pstate;
+
+	if (vcpu_el1_is_32bit(vcpu))
+		pstate = VCPU_RESET_PSTATE_SVC;
+	else if (vcpu_has_nv(vcpu))
+		pstate = VCPU_RESET_PSTATE_EL2;
+	else
+		pstate = VCPU_RESET_PSTATE_EL1;
+
+	/* Reset core registers */
+	memset(vcpu_gp_regs(vcpu), 0, sizeof(*vcpu_gp_regs(vcpu)));
+	memset(&vcpu->arch.ctxt.fp_regs, 0, sizeof(vcpu->arch.ctxt.fp_regs));
+	vcpu->arch.ctxt.spsr_abt = 0;
+	vcpu->arch.ctxt.spsr_und = 0;
+	vcpu->arch.ctxt.spsr_irq = 0;
+	vcpu->arch.ctxt.spsr_fiq = 0;
+	vcpu_gp_regs(vcpu)->pstate = pstate;
+}
+
+/* PSCI reset handling for a vcpu. */
+static inline void kvm_reset_vcpu_psci(struct kvm_vcpu *vcpu,
+				       struct vcpu_reset_state *reset_state)
+{
+	unsigned long target_pc = reset_state->pc;
+
+	/* Gracefully handle Thumb2 entry point */
+	if (vcpu_mode_is_32bit(vcpu) && (target_pc & 1)) {
+		target_pc &= ~1UL;
+		vcpu_set_thumb(vcpu);
+	}
+
+	/* Propagate caller endianness */
+	if (reset_state->be)
+		kvm_vcpu_set_be(vcpu);
+
+	*vcpu_pc(vcpu) = target_pc;
+
+	/*
+	 * We may come from a state where either a PC update was
+	 * pending (SMC call resulting in PC being increpented to
+	 * skip the SMC) or a pending exception. Make sure we get
+	 * rid of all that, as this cannot be valid out of reset.
+	 *
+	 * Note that clearing the exception mask also clears PC
+	 * updates, but that's an implementation detail, and we
+	 * really want to make it explicit.
+	 */
+	vcpu_clear_flag(vcpu, PENDING_EXCEPTION);
+	vcpu_clear_flag(vcpu, EXCEPT_MASK);
+	vcpu_clear_flag(vcpu, INCREMENT_PC);
+	vcpu_set_reg(vcpu, 0, reset_state->r0);
+}
+
 #endif /* __ARM64_KVM_EMULATE_H__ */
arch/arm64/kvm/reset.c +3 -57
index b963fd975aac..10eb7249aa9e 100644
@@ -34,18 +34,6 @@
 static u32 __ro_after_init kvm_ipa_limit;
 unsigned int __ro_after_init kvm_host_sve_max_vl;
 
-/*
- * ARMv8 Reset Values
- */
-#define VCPU_RESET_PSTATE_EL1	(PSR_MODE_EL1h | PSR_A_BIT | PSR_I_BIT | \
-				 PSR_F_BIT | PSR_D_BIT)
-
-#define VCPU_RESET_PSTATE_EL2	(PSR_MODE_EL2h | PSR_A_BIT | PSR_I_BIT | \
-				 PSR_F_BIT | PSR_D_BIT)
-
-#define VCPU_RESET_PSTATE_SVC	(PSR_AA32_MODE_SVC | PSR_AA32_A_BIT | \
-				 PSR_AA32_I_BIT | PSR_AA32_F_BIT)
-
 unsigned int __ro_after_init kvm_sve_max_vl;
 
 int __init kvm_arm_init_sve(void)
@@ -191,7 +179,6 @@ void kvm_reset_vcpu(struct kvm_vcpu *vcpu)
 {
 	struct vcpu_reset_state reset_state;
 	bool loaded;
-	u32 pstate;
 
 	spin_lock(&vcpu->arch.mp_state_lock);
 	reset_state = vcpu->arch.reset_state;
@@ -210,21 +197,8 @@ void kvm_reset_vcpu(struct kvm_vcpu *vcpu)
 		kvm_vcpu_reset_sve(vcpu);
 	}
 
-	if (vcpu_el1_is_32bit(vcpu))
-		pstate = VCPU_RESET_PSTATE_SVC;
-	else if (vcpu_has_nv(vcpu))
-		pstate = VCPU_RESET_PSTATE_EL2;
-	else
-		pstate = VCPU_RESET_PSTATE_EL1;
-
 	/* Reset core registers */
-	memset(vcpu_gp_regs(vcpu), 0, sizeof(*vcpu_gp_regs(vcpu)));
-	memset(&vcpu->arch.ctxt.fp_regs, 0, sizeof(vcpu->arch.ctxt.fp_regs));
-	vcpu->arch.ctxt.spsr_abt = 0;
-	vcpu->arch.ctxt.spsr_und = 0;
-	vcpu->arch.ctxt.spsr_irq = 0;
-	vcpu->arch.ctxt.spsr_fiq = 0;
-	vcpu_gp_regs(vcpu)->pstate = pstate;
+	kvm_reset_vcpu_core(vcpu);
 
 	/* Reset system registers */
 	kvm_reset_sys_regs(vcpu);
@@ -233,36 +207,8 @@ void kvm_reset_vcpu(struct kvm_vcpu *vcpu)
 	 * Additional reset state handling that PSCI may have imposed on us.
 	 * Must be done after all the sys_reg reset.
 	 */
-	if (reset_state.reset) {
-		unsigned long target_pc = reset_state.pc;
-
-		/* Gracefully handle Thumb2 entry point */
-		if (vcpu_mode_is_32bit(vcpu) && (target_pc & 1)) {
-			target_pc &= ~1UL;
-			vcpu_set_thumb(vcpu);
-		}
-
-		/* Propagate caller endianness */
-		if (reset_state.be)
-			kvm_vcpu_set_be(vcpu);
-
-		*vcpu_pc(vcpu) = target_pc;
-
-		/*
-		 * We may come from a state where either a PC update was
-		 * pending (SMC call resulting in PC being increpented to
-		 * skip the SMC) or a pending exception. Make sure we get
-		 * rid of all that, as this cannot be valid out of reset.
-		 *
-		 * Note that clearing the exception mask also clears PC
-		 * updates, but that's an implementation detail, and we
-		 * really want to make it explicit.
-		 */
-		vcpu_clear_flag(vcpu, PENDING_EXCEPTION);
-		vcpu_clear_flag(vcpu, EXCEPT_MASK);
-		vcpu_clear_flag(vcpu, INCREMENT_PC);
-		vcpu_set_reg(vcpu, 0, reset_state.r0);
-	}
+	if (reset_state.reset)
+		kvm_reset_vcpu_psci(vcpu, &reset_state);
 
 	/* Reset timer */
 	kvm_timer_vcpu_reset(vcpu);
Signature · 1 line
2.55.0.rc0.738.g0c8ab3ebcc-goog
author
#7 lore raw

PATCH v2 4/8 KVM: arm64: Move PSCI helper functions to a shared header

In reply to Fuad Tabba

Move kvm_psci_valid_affinity() and kvm_psci_narrow_to_32bit() from psci.c to include/kvm/arm_psci.h, and move psci_affinity_mask() there too, renaming it kvm_psci_affinity_mask() now that it is no longer file-local. A follow-up series handles some protected-guest PSCI calls at EL2 using these helpers.

No functional change intended.

---
 arch/arm64/kvm/psci.c  | 30 +-----------------------------
 include/kvm/arm_psci.h | 27 +++++++++++++++++++++++++++
 2 files changed, 28 insertions(+), 29 deletions(-)
arch/arm64/kvm/psci.c +1 -29
index 3b5dbe9a0a0e..e3db84400d1f 100644
@@ -21,16 +21,6 @@
  * as described in ARM document number ARM DEN 0022A.
  */
 
-#define AFFINITY_MASK(level)	~((0x1UL << ((level) * MPIDR_LEVEL_BITS)) - 1)
-
-static unsigned long psci_affinity_mask(unsigned long affinity_level)
-{
-	if (affinity_level <= 3)
-		return MPIDR_HWID_BITMASK & AFFINITY_MASK(affinity_level);
-
-	return 0;
-}
-
 static unsigned long kvm_psci_vcpu_suspend(struct kvm_vcpu *vcpu)
 {
 	/*
@@ -51,12 +41,6 @@ static unsigned long kvm_psci_vcpu_suspend(struct kvm_vcpu *vcpu)
 	return PSCI_RET_SUCCESS;
 }
 
-static inline bool kvm_psci_valid_affinity(struct kvm_vcpu *vcpu,
-					   unsigned long affinity)
-{
-	return !(affinity & ~MPIDR_HWID_BITMASK);
-}
-
 static unsigned long kvm_psci_vcpu_on(struct kvm_vcpu *source_vcpu)
 {
 	struct vcpu_reset_state *reset_state;
@@ -135,7 +119,7 @@ static unsigned long kvm_psci_vcpu_affinity_info(struct kvm_vcpu *vcpu)
 		return PSCI_RET_INVALID_PARAMS;
 
 	/* Determine target affinity mask */
-	target_affinity_mask = psci_affinity_mask(lowest_affinity_level);
+	target_affinity_mask = kvm_psci_affinity_mask(lowest_affinity_level);
 	if (!target_affinity_mask)
 		return PSCI_RET_INVALID_PARAMS;
 
@@ -220,18 +204,6 @@ static void kvm_psci_system_suspend(struct kvm_vcpu *vcpu)
 	run->exit_reason = KVM_EXIT_SYSTEM_EVENT;
 }
 
-static void kvm_psci_narrow_to_32bit(struct kvm_vcpu *vcpu)
-{
-	int i;
-
-	/*
-	 * Zero the input registers' upper 32 bits. They will be fully
-	 * zeroed on exit, so we're fine changing them in place.
-	 */
-	for (i = 1; i < 4; i++)
-		vcpu_set_reg(vcpu, i, lower_32_bits(vcpu_get_reg(vcpu, i)));
-}
-
 static unsigned long kvm_psci_check_allowed_function(struct kvm_vcpu *vcpu, u32 fn)
 {
 	/*
include/kvm/arm_psci.h +27 -0
index cbaec804eb83..f86a006d6713 100644
@@ -38,6 +38,33 @@ static inline int kvm_psci_version(struct kvm_vcpu *vcpu)
 	return KVM_ARM_PSCI_0_1;
 }
 
+/* Narrow the PSCI register arguments (r1 to r3) to 32 bits. */
+static inline void kvm_psci_narrow_to_32bit(struct kvm_vcpu *vcpu)
+{
+	int i;
+
+	/*
+	 * Zero the input registers' upper 32 bits. They will be fully
+	 * zeroed on exit, so we're fine changing them in place.
+	 */
+	for (i = 1; i < 4; i++)
+		vcpu_set_reg(vcpu, i, lower_32_bits(vcpu_get_reg(vcpu, i)));
+}
+
+static inline bool kvm_psci_valid_affinity(struct kvm_vcpu *vcpu,
+					   unsigned long affinity)
+{
+	return !(affinity & ~MPIDR_HWID_BITMASK);
+}
+
+static inline unsigned long kvm_psci_affinity_mask(unsigned long affinity_level)
+{
+	if (affinity_level <= 3)
+		return MPIDR_HWID_BITMASK &
+			~((0x1UL << (affinity_level * MPIDR_LEVEL_BITS)) - 1);
+
+	return 0;
+}
 
 int kvm_psci_call(struct kvm_vcpu *vcpu);
 
Signature · 1 line
2.55.0.rc0.738.g0c8ab3ebcc-goog
Reviewed-by on v2 4/8
#8 lore raw

Re: [PATCH v2 4/8] KVM: arm64: Move PSCI helper functions to a shared header

Quoted 9 lines · Fuad Tabba

Move kvm_psci_valid_affinity() and kvm_psci_narrow_to_32bit() from psci.c to include/kvm/arm_psci.h, and move psci_affinity_mask() there too, renaming it kvm_psci_affinity_mask() now that it is no longer file-local. A follow-up series handles some protected-guest PSCI calls at EL2 using these helpers.

No functional change intended.

Quoted 108 lines
---
 arch/arm64/kvm/psci.c  | 30 +-----------------------------
 include/kvm/arm_psci.h | 27 +++++++++++++++++++++++++++
 2 files changed, 28 insertions(+), 29 deletions(-)
arch/arm64/kvm/psci.c +1 -29
index 3b5dbe9a0a0e..e3db84400d1f 100644
@@ -21,16 +21,6 @@
  * as described in ARM document number ARM DEN 0022A.
  */
 
-#define AFFINITY_MASK(level)	~((0x1UL << ((level) * MPIDR_LEVEL_BITS)) - 1)
-
-static unsigned long psci_affinity_mask(unsigned long affinity_level)
-{
-	if (affinity_level <= 3)
-		return MPIDR_HWID_BITMASK & AFFINITY_MASK(affinity_level);
-
-	return 0;
-}
-
 static unsigned long kvm_psci_vcpu_suspend(struct kvm_vcpu *vcpu)
 {
 	/*
@@ -51,12 +41,6 @@ static unsigned long kvm_psci_vcpu_suspend(struct kvm_vcpu *vcpu)
 	return PSCI_RET_SUCCESS;
 }
 
-static inline bool kvm_psci_valid_affinity(struct kvm_vcpu *vcpu,
-					   unsigned long affinity)
-{
-	return !(affinity & ~MPIDR_HWID_BITMASK);
-}
-
 static unsigned long kvm_psci_vcpu_on(struct kvm_vcpu *source_vcpu)
 {
 	struct vcpu_reset_state *reset_state;
@@ -135,7 +119,7 @@ static unsigned long kvm_psci_vcpu_affinity_info(struct kvm_vcpu *vcpu)
 		return PSCI_RET_INVALID_PARAMS;
 
 	/* Determine target affinity mask */
-	target_affinity_mask = psci_affinity_mask(lowest_affinity_level);
+	target_affinity_mask = kvm_psci_affinity_mask(lowest_affinity_level);
 	if (!target_affinity_mask)
 		return PSCI_RET_INVALID_PARAMS;
 
@@ -220,18 +204,6 @@ static void kvm_psci_system_suspend(struct kvm_vcpu *vcpu)
 	run->exit_reason = KVM_EXIT_SYSTEM_EVENT;
 }
 
-static void kvm_psci_narrow_to_32bit(struct kvm_vcpu *vcpu)
-{
-	int i;
-
-	/*
-	 * Zero the input registers' upper 32 bits. They will be fully
-	 * zeroed on exit, so we're fine changing them in place.
-	 */
-	for (i = 1; i < 4; i++)
-		vcpu_set_reg(vcpu, i, lower_32_bits(vcpu_get_reg(vcpu, i)));
-}
-
 static unsigned long kvm_psci_check_allowed_function(struct kvm_vcpu *vcpu, u32 fn)
 {
 	/*
include/kvm/arm_psci.h +27 -0
index cbaec804eb83..f86a006d6713 100644
@@ -38,6 +38,33 @@ static inline int kvm_psci_version(struct kvm_vcpu *vcpu)
 	return KVM_ARM_PSCI_0_1;
 }
 
+/* Narrow the PSCI register arguments (r1 to r3) to 32 bits. */
+static inline void kvm_psci_narrow_to_32bit(struct kvm_vcpu *vcpu)
+{
+	int i;
+
+	/*
+	 * Zero the input registers' upper 32 bits. They will be fully
+	 * zeroed on exit, so we're fine changing them in place.
+	 */
+	for (i = 1; i < 4; i++)
+		vcpu_set_reg(vcpu, i, lower_32_bits(vcpu_get_reg(vcpu, i)));
+}
+
+static inline bool kvm_psci_valid_affinity(struct kvm_vcpu *vcpu,
+					   unsigned long affinity)
+{
+	return !(affinity & ~MPIDR_HWID_BITMASK);
+}
+
+static inline unsigned long kvm_psci_affinity_mask(unsigned long affinity_level)
+{
+	if (affinity_level <= 3)
+		return MPIDR_HWID_BITMASK &
+			~((0x1UL << (affinity_level * MPIDR_LEVEL_BITS)) - 1);
+
+	return 0;
+}
 
 int kvm_psci_call(struct kvm_vcpu *vcpu);
 
Signature · 1 line
2.55.0.rc0.738.g0c8ab3ebcc-goog
author
#9 lore raw

PATCH v2 5/8 KVM: arm64: Add host and hypervisor vCPU lookup primitives

In reply to Fuad Tabba

From: Marc Zyngier <[email protected]>

The nVHE hypervisor repeatedly resolves a host vCPU into the EL2 address space and validates that the loaded hyp vCPU matches it, with that logic open-coded in each handler.

Add __get_host_hyp_vcpus() and the get_host_hyp_vcpus() macro, which translate the host vCPU into the hypervisor's address space and, when pKVM is enabled, also return the loaded hyp vCPU if it matches. If pKVM is enabled but the loaded hyp vCPU does not correspond to the requested host vCPU, both the host and hyp vCPU are returned as NULL. Convert handle___kvm_vcpu_run() to use it.

No functional change intended.

---
 arch/arm64/kvm/hyp/nvhe/hyp-main.c | 52 ++++++++++++++++++++++--------
 1 file changed, 38 insertions(+), 14 deletions(-)
arch/arm64/kvm/hyp/nvhe/hyp-main.c +38 -14
index 1d01c6e547f5..8923f594c264 100644
@@ -212,14 +212,45 @@ static void handle___pkvm_vcpu_put(struct kvm_cpu_context *host_ctxt)
 		pkvm_put_hyp_vcpu(hyp_vcpu);
 }
 
-static void handle___kvm_vcpu_run(struct kvm_cpu_context *host_ctxt)
+static struct kvm_vcpu *__get_host_hyp_vcpus(struct kvm_vcpu *arg,
+					     struct pkvm_hyp_vcpu **hyp_vcpup)
 {
-	DECLARE_REG(struct kvm_vcpu *, host_vcpu, host_ctxt, 1);
-	int ret;
+	struct kvm_vcpu *host_vcpu = kern_hyp_va(arg);
+	struct pkvm_hyp_vcpu *hyp_vcpu = NULL;
 
 	if (unlikely(is_protected_kvm_enabled())) {
-		struct pkvm_hyp_vcpu *hyp_vcpu = pkvm_get_loaded_hyp_vcpu();
+		hyp_vcpu = pkvm_get_loaded_hyp_vcpu();
 
+		if (!hyp_vcpu || hyp_vcpu->host_vcpu != host_vcpu) {
+			hyp_vcpu = NULL;
+			host_vcpu = NULL;
+		}
+	}
+
+	*hyp_vcpup = hyp_vcpu;
+	return host_vcpu;
+}
+
+#define get_host_hyp_vcpus(ctxt, regnr, hyp_vcpup)			\
+	({								\
+		DECLARE_REG(struct kvm_vcpu *, __vcpu, ctxt, regnr);	\
+		__get_host_hyp_vcpus(__vcpu, hyp_vcpup);		\
+	})
+
+static void handle___kvm_vcpu_run(struct kvm_cpu_context *host_ctxt)
+{
+	struct pkvm_hyp_vcpu *hyp_vcpu;
+	struct kvm_vcpu *host_vcpu;
+	int ret;
+
+	host_vcpu = get_host_hyp_vcpus(host_ctxt, 1, &hyp_vcpu);
+
+	if (!host_vcpu) {
+		ret = -EINVAL;
+		goto out;
+	}
+
+	if (unlikely(hyp_vcpu)) {
 		/*
 		 * KVM (and pKVM) doesn't support SME guests for now, and
 		 * ensures that SME features aren't enabled in pstate when
@@ -231,23 +262,16 @@ static void handle___kvm_vcpu_run(struct kvm_cpu_context *host_ctxt)
 			goto out;
 		}
 
-		if (!hyp_vcpu) {
-			ret = -EINVAL;
-			goto out;
-		}
-
 		flush_hyp_vcpu(hyp_vcpu);
 
 		ret = __kvm_vcpu_run(&hyp_vcpu->vcpu);
 
 		sync_hyp_vcpu(hyp_vcpu);
 	} else {
-		struct kvm_vcpu *vcpu = kern_hyp_va(host_vcpu);
-
 		/* The host is fully trusted, run its vCPU directly. */
-		fpsimd_lazy_switch_to_guest(vcpu);
-		ret = __kvm_vcpu_run(vcpu);
-		fpsimd_lazy_switch_to_host(vcpu);
+		fpsimd_lazy_switch_to_guest(host_vcpu);
+		ret = __kvm_vcpu_run(host_vcpu);
+		fpsimd_lazy_switch_to_host(host_vcpu);
 	}
 out:
 	cpu_reg(host_ctxt, 1) =  ret;
Signature · 1 line
2.55.0.rc0.738.g0c8ab3ebcc-goog
Reviewed-by on v2 5/8
#10 lore raw

Re: [PATCH v2 5/8] KVM: arm64: Add host and hypervisor vCPU lookup primitives

Quoted 13 lines · Fuad Tabba

From: Marc Zyngier <[email protected]>

The nVHE hypervisor repeatedly resolves a host vCPU into the EL2 address space and validates that the loaded hyp vCPU matches it, with that logic open-coded in each handler.

Add __get_host_hyp_vcpus() and the get_host_hyp_vcpus() macro, which translate the host vCPU into the hypervisor's address space and, when pKVM is enabled, also return the loaded hyp vCPU if it matches. If pKVM is enabled but the loaded hyp vCPU does not correspond to the requested host vCPU, both the host and hyp vCPU are returned as NULL. Convert handle___kvm_vcpu_run() to use it.

Quoted 5 lines

No functional change intended.

Quoted 88 lines
---
 arch/arm64/kvm/hyp/nvhe/hyp-main.c | 52 ++++++++++++++++++++++--------
 1 file changed, 38 insertions(+), 14 deletions(-)
arch/arm64/kvm/hyp/nvhe/hyp-main.c +38 -14
index 1d01c6e547f5..8923f594c264 100644
@@ -212,14 +212,45 @@ static void handle___pkvm_vcpu_put(struct kvm_cpu_context *host_ctxt)
 		pkvm_put_hyp_vcpu(hyp_vcpu);
 }
 
-static void handle___kvm_vcpu_run(struct kvm_cpu_context *host_ctxt)
+static struct kvm_vcpu *__get_host_hyp_vcpus(struct kvm_vcpu *arg,
+					     struct pkvm_hyp_vcpu **hyp_vcpup)
 {
-	DECLARE_REG(struct kvm_vcpu *, host_vcpu, host_ctxt, 1);
-	int ret;
+	struct kvm_vcpu *host_vcpu = kern_hyp_va(arg);
+	struct pkvm_hyp_vcpu *hyp_vcpu = NULL;
 
 	if (unlikely(is_protected_kvm_enabled())) {
-		struct pkvm_hyp_vcpu *hyp_vcpu = pkvm_get_loaded_hyp_vcpu();
+		hyp_vcpu = pkvm_get_loaded_hyp_vcpu();
 
+		if (!hyp_vcpu || hyp_vcpu->host_vcpu != host_vcpu) {
+			hyp_vcpu = NULL;
+			host_vcpu = NULL;
+		}
+	}
+
+	*hyp_vcpup = hyp_vcpu;
+	return host_vcpu;
+}
+
+#define get_host_hyp_vcpus(ctxt, regnr, hyp_vcpup)			\
+	({								\
+		DECLARE_REG(struct kvm_vcpu *, __vcpu, ctxt, regnr);	\
+		__get_host_hyp_vcpus(__vcpu, hyp_vcpup);		\
+	})
+
+static void handle___kvm_vcpu_run(struct kvm_cpu_context *host_ctxt)
+{
+	struct pkvm_hyp_vcpu *hyp_vcpu;
+	struct kvm_vcpu *host_vcpu;
+	int ret;
+
+	host_vcpu = get_host_hyp_vcpus(host_ctxt, 1, &hyp_vcpu);
+
+	if (!host_vcpu) {
+		ret = -EINVAL;
+		goto out;
+	}
+
+	if (unlikely(hyp_vcpu)) {
 		/*
 		 * KVM (and pKVM) doesn't support SME guests for now, and
 		 * ensures that SME features aren't enabled in pstate when
@@ -231,23 +262,16 @@ static void handle___kvm_vcpu_run(struct kvm_cpu_context *host_ctxt)
 			goto out;
 		}
 
-		if (!hyp_vcpu) {
-			ret = -EINVAL;
-			goto out;
-		}
-
 		flush_hyp_vcpu(hyp_vcpu);
 
 		ret = __kvm_vcpu_run(&hyp_vcpu->vcpu);
 
 		sync_hyp_vcpu(hyp_vcpu);
 	} else {
-		struct kvm_vcpu *vcpu = kern_hyp_va(host_vcpu);
-
 		/* The host is fully trusted, run its vCPU directly. */
-		fpsimd_lazy_switch_to_guest(vcpu);
-		ret = __kvm_vcpu_run(vcpu);
-		fpsimd_lazy_switch_to_host(vcpu);
+		fpsimd_lazy_switch_to_guest(host_vcpu);
+		ret = __kvm_vcpu_run(host_vcpu);
+		fpsimd_lazy_switch_to_host(host_vcpu);
 	}
 out:
 	cpu_reg(host_ctxt, 1) =  ret;
Signature · 1 line
2.55.0.rc0.738.g0c8ab3ebcc-goog
author
#11 lore raw

PATCH v2 6/8 KVM: arm64: Minimise EL2's exposure of host VGIC state during world switch

In reply to Fuad Tabba

From: Marc Zyngier <[email protected]>

The host passes a vgic_v3_cpu_if pointer to the __vgic_v3_save_aprs and __vgic_v3_restore_vmcr_aprs hypercalls, which EL2 dereferences wholesale. That exposes the host's full VGIC emulation state to the hypervisor, against pKVM's isolation goals.

Recover the host vCPU from the supplied cpu_if via container_of() and copy only vgic_vmcr and the active priority registers between EL2's hyp-side state and the host vCPU, so EL2 no longer dereferences the host's vgic_v3_cpu_if directly.

---
 arch/arm64/kvm/hyp/nvhe/hyp-main.c | 67 ++++++++++++++++++++++++++++--
 1 file changed, 63 insertions(+), 4 deletions(-)
arch/arm64/kvm/hyp/nvhe/hyp-main.c +63 -4
index 8923f594c264..f25ee3971528 100644
@@ -7,6 +7,8 @@
 #include <hyp/adjust_pc.h>
 #include <hyp/switch.h>
 
+#include <linux/irqchip/arm-gic-v3.h>
+
 #include <asm/pgtable-types.h>
 #include <asm/kvm_asm.h>
 #include <asm/kvm_emulate.h>
@@ -237,6 +239,16 @@ static struct kvm_vcpu *__get_host_hyp_vcpus(struct kvm_vcpu *arg,
 		__get_host_hyp_vcpus(__vcpu, hyp_vcpup);		\
 	})
 
+#define get_host_hyp_vcpus_from_vgic_v3_cpu_if(ctxt, regnr, hyp_vcpup)		\
+	({									\
+		DECLARE_REG(struct vgic_v3_cpu_if *, cif, ctxt, regnr);\
+		struct kvm_vcpu *__vcpu = container_of(cif,			\
+						       struct kvm_vcpu,		\
+						       arch.vgic_cpu.vgic_v3);	\
+										\
+		__get_host_hyp_vcpus(__vcpu, hyp_vcpup);			\
+	})
+
 static void handle___kvm_vcpu_run(struct kvm_cpu_context *host_ctxt)
 {
 	struct pkvm_hyp_vcpu *hyp_vcpu;
@@ -506,16 +518,63 @@ static void handle___vgic_v3_init_lrs(struct kvm_cpu_context *host_ctxt)
 
 static void handle___vgic_v3_save_aprs(struct kvm_cpu_context *host_ctxt)
 {
-	DECLARE_REG(struct vgic_v3_cpu_if *, cpu_if, host_ctxt, 1);
+	struct pkvm_hyp_vcpu *hyp_vcpu;
+	struct kvm_vcpu *host_vcpu;
 
-	__vgic_v3_save_aprs(kern_hyp_va(cpu_if));
+	host_vcpu = get_host_hyp_vcpus_from_vgic_v3_cpu_if(host_ctxt, 1,
+							   &hyp_vcpu);
+	if (!host_vcpu)
+		return;
+
+	if (unlikely(hyp_vcpu)) {
+		struct vgic_v3_cpu_if *hyp_cpu_if, *host_cpu_if;
+		int i;
+
+		hyp_cpu_if = &hyp_vcpu->vcpu.arch.vgic_cpu.vgic_v3;
+		__vgic_v3_save_aprs(hyp_cpu_if);
+
+		host_cpu_if = &host_vcpu->arch.vgic_cpu.vgic_v3;
+		host_cpu_if->vgic_vmcr = hyp_cpu_if->vgic_vmcr;
+		for (i = 0; i < ARRAY_SIZE(host_cpu_if->vgic_ap0r); i++) {
+			host_cpu_if->vgic_ap0r[i] = hyp_cpu_if->vgic_ap0r[i];
+			host_cpu_if->vgic_ap1r[i] = hyp_cpu_if->vgic_ap1r[i];
+		}
+	} else {
+		__vgic_v3_save_aprs(&host_vcpu->arch.vgic_cpu.vgic_v3);
+	}
 }
 
 static void handle___vgic_v3_restore_vmcr_aprs(struct kvm_cpu_context *host_ctxt)
 {
-	DECLARE_REG(struct vgic_v3_cpu_if *, cpu_if, host_ctxt, 1);
+	struct pkvm_hyp_vcpu *hyp_vcpu;
+	struct kvm_vcpu *host_vcpu;
 
-	__vgic_v3_restore_vmcr_aprs(kern_hyp_va(cpu_if));
+	host_vcpu = get_host_hyp_vcpus_from_vgic_v3_cpu_if(host_ctxt, 1,
+							   &hyp_vcpu);
+	if (!host_vcpu)
+		return;
+
+	if (unlikely(hyp_vcpu)) {
+		struct vgic_v3_cpu_if *hyp_cpu_if, *host_cpu_if;
+		int i;
+
+		hyp_cpu_if = &hyp_vcpu->vcpu.arch.vgic_cpu.vgic_v3;
+		host_cpu_if = &host_vcpu->arch.vgic_cpu.vgic_v3;
+
+		hyp_cpu_if->vgic_vmcr = host_cpu_if->vgic_vmcr;
+		/* Should be a one-off */
+		hyp_cpu_if->vgic_sre = (ICC_SRE_EL1_DIB |
+					ICC_SRE_EL1_DFB |
+					ICC_SRE_EL1_SRE);
+		for (i = 0; i < ARRAY_SIZE(host_cpu_if->vgic_ap0r); i++) {
+			hyp_cpu_if->vgic_ap0r[i] = host_cpu_if->vgic_ap0r[i];
+			hyp_cpu_if->vgic_ap1r[i] = host_cpu_if->vgic_ap1r[i];
+		}
+
+		__vgic_v3_restore_vmcr_aprs(hyp_cpu_if);
+	} else {
+		__vgic_v3_restore_vmcr_aprs(&host_vcpu->arch.vgic_cpu.vgic_v3);
+	}
 }
 
 static void handle___pkvm_init(struct kvm_cpu_context *host_ctxt)
Signature · 1 line
2.55.0.rc0.738.g0c8ab3ebcc-goog
#12 lore raw

Re: [PATCH v2 6/8] KVM: arm64: Minimise EL2's exposure of host VGIC state during world switch

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

New issues:
- [High] handle___vgic_v3_save_aprs incorrectly clobbers the host's vgic_vmcr with a stale value.

Pre-existing issues:
- [Critical] The patch fixes arbitrary host-pointer dereferences in the VGICv3 save/restore hypercalls but leaves the exact same isolation bypass vulnerability open for GICv5 equivalents and handle___kvm_adjust_pc. --

commit a6f1c910e601736a7f5d9f267e781560d10966dc
Author: Marc Zyngier <[email protected]>

KVM: arm64: Minimise EL2's exposure of host VGIC state during world switch

The host passes a vgic_v3_cpu_if pointer to the __vgic_v3_save_aprs and __vgic_v3_restore_vmcr_aprs hypercalls, which EL2 dereferences wholesale. This patch recovers the host vCPU from the supplied cpu_if via container_of() and copies only vgic_vmcr and the active priority registers between EL2's hyp-side state and the host vCPU.

Quoted 4 lines
arch/arm64/kvm/hyp/nvhe/hyp-main.c +0 -0
index 45d717889f6ea..f3233ee343a39 100644

[ ... ]

Quoted 2 lines
arch/arm64/kvm/hyp/nvhe/hyp-main.c +0 -0
@@ -508,16 +520,63 @@ static void handle___vgic_v3_save_aprs(struct kvm_cpu_context *host_ctxt)
 {

[ ... ]

arch/arm64/kvm/hyp/nvhe/hyp-main.c static void handle___vgic_v3_init_lrs(struct kvm_cpu_context *host_ctxt) · line 537

Quoted 2 lines
arch/arm64/kvm/hyp/nvhe/hyp-main.c +2 -0
+		host_cpu_if = &host_vcpu->arch.vgic_cpu.vgic_v3;
+		host_cpu_if->vgic_vmcr = hyp_cpu_if->vgic_vmcr;

[Severity: High]
Does this overwrite the host's updated vgic_vmcr with a stale hyp copy?

Since __vgic_v3_save_aprs() does not read the hardware ICH_VMCR_EL2 register, hyp_cpu_if->vgic_vmcr retains the value it held at the last guest exit.

If the host kernel emulated a write to the guest's VMCR (for example, via vgic_v3_set_vmcr() during a trapped sysreg access) before the vCPU thread is preempted, this explicit copy back to host_cpu_if will discard the host's emulated state changes.

[ ... ]

arch/arm64/kvm/hyp/nvhe/hyp-main.c static void handle___vgic_v3_init_lrs(struct kvm_cpu_context *host_ctxt) · line 545

Quoted 4 lines
arch/arm64/kvm/hyp/nvhe/hyp-main.c +3 -0
+	} else {
+		__vgic_v3_save_aprs(&host_vcpu->arch.vgic_cpu.vgic_v3);
+	}
 }

[Severity: Critical]
This is a pre-existing issue, but since this patch secures the VGICv3 save/restore hypercalls against arbitrary host-pointer dereferences, should the same protections be extended to handle___kvm_adjust_pc and the v5 equivalents?

Currently, handlers like handle___kvm_adjust_pc() still pass unvalidated host registers directly into kern_hyp_va():

arch/arm64/kvm/hyp/nvhe/hyp-main.c:handle___kvm_adjust_pc() {
    DECLARE_REG(struct kvm_vcpu *, vcpu, host_ctxt, 1);
    __kvm_adjust_pc(kern_hyp_va(vcpu));
}

Because kern_hyp_va() is idempotent for addresses that already carry the hyp linear-map tag, an untrusted host could supply an EL2 linear-map pointer that is returned unchanged.

Passing an arbitrary EL2 memory pointer here will cause __kvm_adjust_pc() to read and write flags directly at that address, giving the host an arbitrary EL2 memory corruption primitive.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
author
#13 lore raw

Re: [PATCH v2 6/8] KVM: arm64: Minimise EL2's exposure of host VGIC state during world switch

Quoted 5 lines · On Fri, 19 Jun 2026 at 08:25, <[email protected]> wrote:

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

New issues:
- [High] handle___vgic_v3_save_aprs incorrectly clobbers the host's vgic_vmcr with a stale value.

The VMCR copy in the hyp_vcpu path is redundant with sync_hyp_vgic_state, not a clobber. __vgic_v3_save_state reads ICH_VMCR_EL2 from HW at guest exit, then sync_hyp_vgic_state copies it to the host. The save_aprs handler writes the same value again. The only writer to host_cpu_if->vgic_vmcr is vgic_v3_set_vmcr (GICv2 distributor MMIO emulation), which is userspace-driven and runs outside the vcpu_load/vcpu_put window.

Quoted 3 lines

Pre-existing issues:
- [Critical] The patch fixes arbitrary host-pointer dereferences in the VGICv3 save/restore hypercalls but leaves the exact same isolation bypass vulnerability open for GICv5 equivalents and handle___kvm_adjust_pc.

GICv5 is not supported in protected mode (is_protected_kvm_enabled() gate in vgic-v5.c skips device registration entirely), so the v5 handlers are unreachable. handle___kvm_adjust_pc and handle___kvm_tlb_flush_vmid_ipa are pre-existing upstream unvalidated-pointer hypercalls, not introduced or modified by this commit. They are fixed as a group in a subsequent series.

Cheers,
/fuad

Quoted 70 lines

--

commit a6f1c910e601736a7f5d9f267e781560d10966dc
Author: Marc Zyngier <[email protected]>

KVM: arm64: Minimise EL2's exposure of host VGIC state during world switch

The host passes a vgic_v3_cpu_if pointer to the __vgic_v3_save_aprs and __vgic_v3_restore_vmcr_aprs hypercalls, which EL2 dereferences wholesale. This patch recovers the host vCPU from the supplied cpu_if via container_of() and copies only vgic_vmcr and the active priority registers between EL2's hyp-side state and the host vCPU.

Quoted 4 lines
arch/arm64/kvm/hyp/nvhe/hyp-main.c +0 -0
index 45d717889f6ea..f3233ee343a39 100644

[ ... ]

Quoted 2 lines
arch/arm64/kvm/hyp/nvhe/hyp-main.c +0 -0
@@ -508,16 +520,63 @@ static void handle___vgic_v3_save_aprs(struct kvm_cpu_context *host_ctxt)
 {

[ ... ]

arch/arm64/kvm/hyp/nvhe/hyp-main.c static void handle___vgic_v3_init_lrs(struct kvm_cpu_context *host_ctxt) · line 537

Quoted 2 lines
arch/arm64/kvm/hyp/nvhe/hyp-main.c +2 -0
+             host_cpu_if = &host_vcpu->arch.vgic_cpu.vgic_v3;
+             host_cpu_if->vgic_vmcr = hyp_cpu_if->vgic_vmcr;

[Severity: High]
Does this overwrite the host's updated vgic_vmcr with a stale hyp copy?

Since __vgic_v3_save_aprs() does not read the hardware ICH_VMCR_EL2 register, hyp_cpu_if->vgic_vmcr retains the value it held at the last guest exit.

If the host kernel emulated a write to the guest's VMCR (for example, via vgic_v3_set_vmcr() during a trapped sysreg access) before the vCPU thread is preempted, this explicit copy back to host_cpu_if will discard the host's emulated state changes.

[ ... ]

arch/arm64/kvm/hyp/nvhe/hyp-main.c static void handle___vgic_v3_init_lrs(struct kvm_cpu_context *host_ctxt) · line 545

Quoted 4 lines
arch/arm64/kvm/hyp/nvhe/hyp-main.c +3 -0
+     } else {
+             __vgic_v3_save_aprs(&host_vcpu->arch.vgic_cpu.vgic_v3);
+     }
 }

[Severity: Critical]
This is a pre-existing issue, but since this patch secures the VGICv3 save/restore hypercalls against arbitrary host-pointer dereferences, should the same protections be extended to handle___kvm_adjust_pc and the v5 equivalents?

Currently, handlers like handle___kvm_adjust_pc() still pass unvalidated host registers directly into kern_hyp_va():

arch/arm64/kvm/hyp/nvhe/hyp-main.c:handle___kvm_adjust_pc() {
    DECLARE_REG(struct kvm_vcpu *, vcpu, host_ctxt, 1);
    __kvm_adjust_pc(kern_hyp_va(vcpu));
}

Because kern_hyp_va() is idempotent for addresses that already carry the hyp linear-map tag, an untrusted host could supply an EL2 linear-map pointer that is returned unchanged.

Passing an arbitrary EL2 memory pointer here will cause __kvm_adjust_pc() to read and write flags directly at that address, giving the host an arbitrary EL2 memory corruption primitive.

--
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6

author
#14 lore raw

PATCH v2 7/8 KVM: arm64: Add primitives to flush/sync the VGIC state at EL2

In reply to Fuad Tabba

From: Marc Zyngier <[email protected]>

pKVM performs its own world switch for protected VMs but has no primitives to move the per-vCPU VGIC state between the host and hypervisor vCPU contexts.

Add flush_hyp_vgic_state() and sync_hyp_vgic_state(). Flush copies vgic_hcr, the in-use list registers and used_lrs from the host into the hyp vCPU and pins vgic_sre to a fixed value; sync copies vgic_hcr, vgic_vmcr and the in-use list registers back. The active priority registers are handled separately by the save/restore-aprs path.

Bound used_lrs by hyp_gicv3_nr_lr, the cached implemented-LR count, instead of reading ICH_VTR_EL2 on each entry. That clamps the host-supplied value and avoids a per-entry sysreg read that is costly under NV.

---
 arch/arm64/kvm/hyp/nvhe/hyp-main.c | 55 ++++++++++++++++++++++--------
 1 file changed, 41 insertions(+), 14 deletions(-)
arch/arm64/kvm/hyp/nvhe/hyp-main.c +41 -14
index f25ee3971528..0194965930e6 100644
@@ -102,6 +102,45 @@ static void fpsimd_sve_sync(struct kvm_vcpu *vcpu)
 	*host_data_ptr(fp_owner) = FP_STATE_HOST_OWNED;
 }
 
+static void flush_hyp_vgic_state(struct pkvm_hyp_vcpu *hyp_vcpu)
+{
+	struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu;
+	struct vgic_v3_cpu_if *host_cpu_if, *hyp_cpu_if;
+	unsigned int used_lrs, i;
+
+	host_cpu_if	= &host_vcpu->arch.vgic_cpu.vgic_v3;
+	hyp_cpu_if	= &hyp_vcpu->vcpu.arch.vgic_cpu.vgic_v3;
+
+	used_lrs	= host_cpu_if->used_lrs;
+	used_lrs	= min(used_lrs, hyp_gicv3_nr_lr);
+
+	hyp_cpu_if->vgic_hcr	= host_cpu_if->vgic_hcr;
+	/* Should be a one-off */
+	hyp_cpu_if->vgic_sre	= (ICC_SRE_EL1_DIB |
+				   ICC_SRE_EL1_DFB |
+				   ICC_SRE_EL1_SRE);
+	hyp_cpu_if->used_lrs	= used_lrs;
+
+	for (i = 0; i < used_lrs; i++)
+		hyp_cpu_if->vgic_lr[i] = host_cpu_if->vgic_lr[i];
+}
+
+static void sync_hyp_vgic_state(struct pkvm_hyp_vcpu *hyp_vcpu)
+{
+	struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu;
+	struct vgic_v3_cpu_if *host_cpu_if, *hyp_cpu_if;
+	unsigned int i;
+
+	host_cpu_if	= &host_vcpu->arch.vgic_cpu.vgic_v3;
+	hyp_cpu_if	= &hyp_vcpu->vcpu.arch.vgic_cpu.vgic_v3;
+
+	host_cpu_if->vgic_hcr = hyp_cpu_if->vgic_hcr;
+	host_cpu_if->vgic_vmcr = hyp_cpu_if->vgic_vmcr;
+
+	for (i = 0; i < hyp_cpu_if->used_lrs; i++)
+		host_cpu_if->vgic_lr[i] = hyp_cpu_if->vgic_lr[i];
+}
+
 static void flush_debug_state(struct pkvm_hyp_vcpu *hyp_vcpu)
 {
 	struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu;
@@ -150,13 +189,7 @@ static void flush_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu)
 
 	hyp_vcpu->vcpu.arch.vsesr_el2	= host_vcpu->arch.vsesr_el2;
 
-	hyp_vcpu->vcpu.arch.vgic_cpu.vgic_v3 = host_vcpu->arch.vgic_cpu.vgic_v3;
-
-	/* Bound used_lrs by the number of implemented list registers. */
-	hyp_vcpu->vcpu.arch.vgic_cpu.vgic_v3.used_lrs =
-		min_t(unsigned int,
-		      hyp_vcpu->vcpu.arch.vgic_cpu.vgic_v3.used_lrs,
-		      hyp_gicv3_nr_lr);
+	flush_hyp_vgic_state(hyp_vcpu);
 
 	hyp_vcpu->vcpu.arch.pid = host_vcpu->arch.pid;
 }
@@ -164,9 +197,6 @@ static void flush_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu)
 static void sync_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu)
 {
 	struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu;
-	struct vgic_v3_cpu_if *hyp_cpu_if = &hyp_vcpu->vcpu.arch.vgic_cpu.vgic_v3;
-	struct vgic_v3_cpu_if *host_cpu_if = &host_vcpu->arch.vgic_cpu.vgic_v3;
-	unsigned int i;
 
 	fpsimd_sve_sync(&hyp_vcpu->vcpu);
 	sync_debug_state(hyp_vcpu);
@@ -179,10 +209,7 @@ static void sync_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu)
 
 	host_vcpu->arch.iflags		= hyp_vcpu->vcpu.arch.iflags;
 
-	host_cpu_if->vgic_hcr		= hyp_cpu_if->vgic_hcr;
-	host_cpu_if->vgic_vmcr		= hyp_cpu_if->vgic_vmcr;
-	for (i = 0; i < hyp_cpu_if->used_lrs; ++i)
-		host_cpu_if->vgic_lr[i] = hyp_cpu_if->vgic_lr[i];
+	sync_hyp_vgic_state(hyp_vcpu);
 }
 
 static void handle___pkvm_vcpu_load(struct kvm_cpu_context *host_ctxt)
Signature · 1 line
2.55.0.rc0.738.g0c8ab3ebcc-goog
#15 lore raw

Re: [PATCH v2 7/8] KVM: arm64: Add primitives to flush/sync the VGIC state at EL2

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

New issues:
- [High] flush_hyp_vgic_state() fails to copy vgic_vmcr from the host vCPU to the hyp vCPU, causing host-side VMCR updates to be silently lost and subsequently clobbered.

Pre-existing issues:
- [High] flush_hyp_vcpu() unconditionally copies mdcr_el2 from the untrusted host vCPU, allowing the host to control debug trap configurations for protected VMs. --

commit 5eaa1440f7e2f022736f913bec34aaa0cbd53a5d
Author: Marc Zyngier <[email protected]>

KVM: arm64: Add primitives to flush/sync the VGIC state at EL2

pKVM performs its own world switch for protected VMs but has no primitives to move the per-vCPU VGIC state between the host and hypervisor vCPU contexts. Add flush_hyp_vgic_state() and sync_hyp_vgic_state() to handle the required copying.

Quoted 4 lines
arch/arm64/kvm/hyp/nvhe/hyp-main.c +0 -0
index f3233ee343a39..2db56146ec493 100644

[ ... ]

Quoted 8 lines
arch/arm64/kvm/hyp/nvhe/hyp-main.c +4 -0
@@ -104,6 +104,45 @@ static void fpsimd_sve_sync(struct kvm_vcpu *vcpu)
 	*host_data_ptr(fp_owner) = FP_STATE_HOST_OWNED;
 }
 
+static void flush_hyp_vgic_state(struct pkvm_hyp_vcpu *hyp_vcpu)
+{
+	struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu;
+	struct vgic_v3_cpu_if *host_cpu_if, *hyp_cpu_if;

arch/arm64/kvm/hyp/nvhe/hyp-main.c static void fpsimd_sve_sync(struct kvm_vcpu *vcpu) · line 120

Quoted 10 lines
arch/arm64/kvm/hyp/nvhe/hyp-main.c +10 -0
+	unsigned int used_lrs, i;
+
+	host_cpu_if	= &host_vcpu->arch.vgic_cpu.vgic_v3;
+	hyp_cpu_if	= &hyp_vcpu->vcpu.arch.vgic_cpu.vgic_v3;
+
+	used_lrs	= host_cpu_if->used_lrs;
+	used_lrs	= min(used_lrs, hyp_gicv3_nr_lr);
+
+	hyp_cpu_if->vgic_hcr	= host_cpu_if->vgic_hcr;
+	/* Should be a one-off */

[Severity: High]
Does flush_hyp_vgic_state() need to copy vgic_vmcr from the host vCPU to the hyp vCPU here?

For GICv2-on-GICv3 emulation, guest writes to GICC_VMCR and GICC_CTLR trap to the host (EL1) as MMIO.

The host updates the host_vcpu's vgic_vmcr and resumes the guest loop without a full vcpu_put/load cycle.

By omitting the copy here, the hypervisor will re-enter the guest with the old hardware VMCR state.

Quoted 10 lines
arch/arm64/kvm/hyp/nvhe/hyp-main.c +10 -0
+	hyp_cpu_if->vgic_sre	= (ICC_SRE_EL1_DIB |
+				   ICC_SRE_EL1_DFB |
+				   ICC_SRE_EL1_SRE);
+	hyp_cpu_if->used_lrs	= used_lrs;
+
+	for (i = 0; i < used_lrs; i++)
+		hyp_cpu_if->vgic_lr[i] = host_cpu_if->vgic_lr[i];
+}
+
+static void sync_hyp_vgic_state(struct pkvm_hyp_vcpu *hyp_vcpu)

arch/arm64/kvm/hyp/nvhe/hyp-main.c static void fpsimd_sve_sync(struct kvm_vcpu *vcpu) · line 138

Quoted 10 lines
arch/arm64/kvm/hyp/nvhe/hyp-main.c +10 -0
+{
+	struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu;
+	struct vgic_v3_cpu_if *host_cpu_if, *hyp_cpu_if;
+	unsigned int i;
+
+	host_cpu_if	= &host_vcpu->arch.vgic_cpu.vgic_v3;
+	hyp_cpu_if	= &hyp_vcpu->vcpu.arch.vgic_cpu.vgic_v3;
+
+	host_cpu_if->vgic_hcr = hyp_cpu_if->vgic_hcr;
+	host_cpu_if->vgic_vmcr = hyp_cpu_if->vgic_vmcr;

[Severity: High]
Will this unconditionally overwrite the host's vgic_vmcr with the hypervisor's stale copy?

Since flush_hyp_vgic_state() doesn't bring the updated vgic_vmcr into the hyp vCPU on entry, syncing it back to the host here appears to permanently clobber the guest's MMIO write.

This missing synchronization could break GICv2 priority masking and group enables.

Quoted 4 lines
arch/arm64/kvm/hyp/nvhe/hyp-main.c +4 -0
+
+	for (i = 0; i < hyp_cpu_if->used_lrs; i++)
+		host_cpu_if->vgic_lr[i] = hyp_cpu_if->vgic_lr[i];
+}

[ ... ]

arch/arm64/kvm/hyp/nvhe/hyp-main.c static void flush_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu)

Quoted 1 line
arch/arm64/kvm/hyp/nvhe/hyp-main.c +0 -0
@@ -152,13 +191,7 @@ static void flush_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu)

[Severity: High]
This is a pre-existing issue, but does flush_hyp_vcpu() unconditionally copy mdcr_el2 from the untrusted host vCPU?

Currently, flush_hyp_vcpu() contains:

    hyp_vcpu->vcpu.arch.mdcr_el2 = host_vcpu->arch.mdcr_el2;

In pKVM, the host memory is untrusted. By manipulating the host_vcpu's mdcr_el2, could a compromised host configure debug traps (e.g., MDCR_EL2.TDA, TDE) for protected VMs?

This would bypass the pKVM threat model, which requires trap configuration registers to be initialized in an EL2-private context to ensure guest confidentiality and integrity.

Quoted 14 lines
arch/arm64/kvm/hyp/nvhe/hyp-main.c +1 -7
 	hyp_vcpu->vcpu.arch.vsesr_el2	= host_vcpu->arch.vsesr_el2;
 
-	hyp_vcpu->vcpu.arch.vgic_cpu.vgic_v3 = host_vcpu->arch.vgic_cpu.vgic_v3;
-
-	/* Bound used_lrs by the number of implemented list registers. */
-	hyp_vcpu->vcpu.arch.vgic_cpu.vgic_v3.used_lrs =
-		min_t(unsigned int,
-		      hyp_vcpu->vcpu.arch.vgic_cpu.vgic_v3.used_lrs,
-		      hyp_gicv3_nr_lr);
+	flush_hyp_vgic_state(hyp_vcpu);
 
 	hyp_vcpu->vcpu.arch.pid = host_vcpu->arch.pid;
 }
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=7
maintainer
#16 lore raw

Re: [PATCH v2 7/8] KVM: arm64: Add primitives to flush/sync the VGIC state at EL2

Quoted 48 lines · On Fri, 19 Jun 2026 08:21:20 +0100, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

New issues:
- [High] flush_hyp_vgic_state() fails to copy vgic_vmcr from the host vCPU to the hyp vCPU, causing host-side VMCR updates to be silently lost and subsequently clobbered.

Pre-existing issues:
- [High] flush_hyp_vcpu() unconditionally copies mdcr_el2 from the untrusted host vCPU, allowing the host to control debug trap configurations for protected VMs. --

commit 5eaa1440f7e2f022736f913bec34aaa0cbd53a5d
Author: Marc Zyngier <[email protected]>

KVM: arm64: Add primitives to flush/sync the VGIC state at EL2

pKVM performs its own world switch for protected VMs but has no primitives to move the per-vCPU VGIC state between the host and hypervisor vCPU contexts. Add flush_hyp_vgic_state() and sync_hyp_vgic_state() to handle the required copying.

Quoted 4 lines
arch/arm64/kvm/hyp/nvhe/hyp-main.c +0 -0
index f3233ee343a39..2db56146ec493 100644

[ ... ]

Quoted 8 lines
arch/arm64/kvm/hyp/nvhe/hyp-main.c +4 -0
@@ -104,6 +104,45 @@ static void fpsimd_sve_sync(struct kvm_vcpu *vcpu)
 	*host_data_ptr(fp_owner) = FP_STATE_HOST_OWNED;
 }
 
+static void flush_hyp_vgic_state(struct pkvm_hyp_vcpu *hyp_vcpu)
+{
+	struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu;
+	struct vgic_v3_cpu_if *host_cpu_if, *hyp_cpu_if;

arch/arm64/kvm/hyp/nvhe/hyp-main.c static void fpsimd_sve_sync(struct kvm_vcpu *vcpu) · line 120

Quoted 10 lines
arch/arm64/kvm/hyp/nvhe/hyp-main.c +10 -0
+	unsigned int used_lrs, i;
+
+	host_cpu_if	= &host_vcpu->arch.vgic_cpu.vgic_v3;
+	hyp_cpu_if	= &hyp_vcpu->vcpu.arch.vgic_cpu.vgic_v3;
+
+	used_lrs	= host_cpu_if->used_lrs;
+	used_lrs	= min(used_lrs, hyp_gicv3_nr_lr);
+
+	hyp_cpu_if->vgic_hcr	= host_cpu_if->vgic_hcr;
+	/* Should be a one-off */

[Severity: High]
Does flush_hyp_vgic_state() need to copy vgic_vmcr from the host vCPU to the hyp vCPU here?

Quoted 2 lines

For GICv2-on-GICv3 emulation, guest writes to GICC_VMCR and GICC_CTLR trap to the host (EL1) as MMIO.

GICC_VMCR doesn't exist, so the guest can't write to it. There is no MMIO involved with pKVM, since it is GICv3 only. And the whole point of having a VGIC is to avoid trapping on each access.

More importantly, VMCR is not allowed to change behind the guest's back, because it contains the state that the *guest* has programmed there.

Can someone *please* unplug Sashiko? I've had enough of it.

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
author
#17 lore raw

Re: [PATCH v2 7/8] KVM: arm64: Add primitives to flush/sync the VGIC state at EL2

Quoted 58 lines · Marc Zyngier
Quoted 48 lines · On Fri, 19 Jun 2026 08:21:20 +0100, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

New issues:
- [High] flush_hyp_vgic_state() fails to copy vgic_vmcr from the host vCPU to the hyp vCPU, causing host-side VMCR updates to be silently lost and subsequently clobbered.

Pre-existing issues:
- [High] flush_hyp_vcpu() unconditionally copies mdcr_el2 from the untrusted host vCPU, allowing the host to control debug trap configurations for protected VMs. --

commit 5eaa1440f7e2f022736f913bec34aaa0cbd53a5d
Author: Marc Zyngier <[email protected]>

KVM: arm64: Add primitives to flush/sync the VGIC state at EL2

pKVM performs its own world switch for protected VMs but has no primitives to move the per-vCPU VGIC state between the host and hypervisor vCPU contexts. Add flush_hyp_vgic_state() and sync_hyp_vgic_state() to handle the required copying.

Quoted 4 lines
arch/arm64/kvm/hyp/nvhe/hyp-main.c +0 -0
index f3233ee343a39..2db56146ec493 100644

[ ... ]

Quoted 8 lines
arch/arm64/kvm/hyp/nvhe/hyp-main.c +4 -0
@@ -104,6 +104,45 @@ static void fpsimd_sve_sync(struct kvm_vcpu *vcpu)
    *host_data_ptr(fp_owner) = FP_STATE_HOST_OWNED;
 }
 
+static void flush_hyp_vgic_state(struct pkvm_hyp_vcpu *hyp_vcpu)
+{
+   struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu;
+   struct vgic_v3_cpu_if *host_cpu_if, *hyp_cpu_if;

arch/arm64/kvm/hyp/nvhe/hyp-main.c static void fpsimd_sve_sync(struct kvm_vcpu *vcpu) · line 120

Quoted 10 lines
arch/arm64/kvm/hyp/nvhe/hyp-main.c +10 -0
+   unsigned int used_lrs, i;
+
+   host_cpu_if     = &host_vcpu->arch.vgic_cpu.vgic_v3;
+   hyp_cpu_if      = &hyp_vcpu->vcpu.arch.vgic_cpu.vgic_v3;
+
+   used_lrs        = host_cpu_if->used_lrs;
+   used_lrs        = min(used_lrs, hyp_gicv3_nr_lr);
+
+   hyp_cpu_if->vgic_hcr    = host_cpu_if->vgic_hcr;
+   /* Should be a one-off */

[Severity: High]
Does flush_hyp_vgic_state() need to copy vgic_vmcr from the host vCPU to the hyp vCPU here?

Quoted 2 lines

For GICv2-on-GICv3 emulation, guest writes to GICC_VMCR and GICC_CTLR trap to the host (EL1) as MMIO.

GICC_VMCR doesn't exist, so the guest can't write to it. There is no MMIO involved with pKVM, since it is GICv3 only. And the whole point of having a VGIC is to avoid trapping on each access.

Quoted 5 lines

More importantly, VMCR is not allowed to change behind the guest's back, because it contains the state that the *guest* has programmed there.

Can someone *please* unplug Sashiko? I've had enough of it.

I am trying to at least fix it :) Key phrase I am adding in my PR:

## Grounding: you do not have the ARM ARM

along with other things, so it wouldn't hallucinate it :)

Testing this now locally before I send the PR.

Cheers,
/fuad

Quoted 5 lines
        M.

--
Without deviation from the norm, progress is not possible.

author
#18 lore raw

PATCH v2 8/8 KVM: arm64: Implement lazy vCPU state sync for non-protected guests

In reply to Fuad Tabba

pKVM copies a non-protected guest's register context between the host and the hypervisor on every world switch, even when the host never inspects it. Defer the copy: on entry, flush the host context into the hyp vCPU only when the host marked it dirty (PKVM_HOST_STATE_DIRTY); on exit, leave it in the hyp vCPU and copy it back only when the host needs it, via a __pkvm_vcpu_sync_state hypercall on trap handling or at vcpu put. A protected guest's context is copied as before, since lazy sync only helps where the host is trusted to see the guest's registers.

PC and PSTATE are the exception: they are copied back on every exit so the kvm_exit tracepoint reports the guest's real exit PC, and the run loop's vcpu_mode_is_bad_32bit() and SError-masking checks evaluate the guest's current PSTATE rather than the value left by the previous sync.

handle_exit_early() can also inject an SError, which writes the guest context (ESR_EL1) outside the trap-handling path. For a non-protected guest it therefore syncs the context from the hyp vCPU and marks it dirty, as handle_trap_exceptions() does, so the injection reaches the hyp vCPU on re-entry rather than being dropped.

---
 arch/arm64/include/asm/kvm_asm.h   |  1 +
 arch/arm64/include/asm/kvm_host.h  |  2 +
 arch/arm64/kvm/arm.c               |  7 +++
 arch/arm64/kvm/handle_exit.c       | 30 +++++++++++
 arch/arm64/kvm/hyp/nvhe/hyp-main.c | 86 ++++++++++++++++++++++++++++--
 5 files changed, 121 insertions(+), 5 deletions(-)
arch/arm64/include/asm/kvm_asm.h +1 -0
index 043495f7fc78..6e1135b3ded4 100644
@@ -113,6 +113,7 @@ enum __kvm_host_smccc_func {
 	__KVM_HOST_SMCCC_FUNC___pkvm_finalize_teardown_vm,
 	__KVM_HOST_SMCCC_FUNC___pkvm_vcpu_load,
 	__KVM_HOST_SMCCC_FUNC___pkvm_vcpu_put,
+	__KVM_HOST_SMCCC_FUNC___pkvm_vcpu_sync_state,
 	__KVM_HOST_SMCCC_FUNC___pkvm_tlb_flush_vmid,
 
 	MARKER(__KVM_HOST_SMCCC_FUNC_MAX)
arch/arm64/include/asm/kvm_host.h +2 -0
index 2faa60df847d..caa39ee5125f 100644
@@ -1068,6 +1068,8 @@ struct kvm_vcpu_arch {
 #define INCREMENT_PC		__vcpu_single_flag(iflags, BIT(1))
 /* Target EL/MODE (not a single flag, but let's abuse the macro) */
 #define EXCEPT_MASK		__vcpu_single_flag(iflags, GENMASK(3, 1))
+/* Host-set: the hyp flushes the non-protected vCPU state in on entry */
+#define PKVM_HOST_STATE_DIRTY	__vcpu_single_flag(iflags, BIT(4))
 
 /* Helpers to encode exceptions with minimum fuss */
 #define __EXCEPT_MASK_VAL	unpack_vcpu_flag(EXCEPT_MASK)
arch/arm64/kvm/arm.c +7 -0
index 3732ee9eb0d4..4e89558d8027 100644
@@ -733,6 +733,10 @@ void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu)
 	if (is_protected_kvm_enabled()) {
 		kvm_call_hyp(__vgic_v3_save_aprs, &vcpu->arch.vgic_cpu.vgic_v3);
 		kvm_call_hyp_nvhe(__pkvm_vcpu_put);
+
+		/* __pkvm_vcpu_put implies a sync of the state */
+		if (!kvm_vm_is_protected(vcpu->kvm))
+			vcpu_set_flag(vcpu, PKVM_HOST_STATE_DIRTY);
 	}
 
 	kvm_vcpu_put_debug(vcpu);
@@ -964,6 +968,9 @@ int kvm_arch_vcpu_run_pid_change(struct kvm_vcpu *vcpu)
 		return ret;
 
 	if (is_protected_kvm_enabled()) {
+		/* Start with the vcpu in a dirty state */
+		if (!kvm_vm_is_protected(vcpu->kvm))
+			vcpu_set_flag(vcpu, PKVM_HOST_STATE_DIRTY);
 		ret = pkvm_create_hyp_vm(kvm);
 		if (ret)
 			return ret;
arch/arm64/kvm/handle_exit.c +30 -0
index 54aedf93c78b..8963621bcdd1 100644
@@ -422,6 +422,20 @@ static int handle_trap_exceptions(struct kvm_vcpu *vcpu)
 {
 	int handled;
 
+	/*
+	 * If we run a non-protected VM when protection is enabled
+	 * system-wide, resync the state from the hypervisor and mark
+	 * it as dirty on the host side if it wasn't dirty already
+	 * (which could happen if preemption has taken place).
+	 */
+	if (is_protected_kvm_enabled() && !kvm_vm_is_protected(vcpu->kvm)) {
+		guard(preempt)();
+		if (!(vcpu_get_flag(vcpu, PKVM_HOST_STATE_DIRTY))) {
+			kvm_call_hyp_nvhe(__pkvm_vcpu_sync_state);
+			vcpu_set_flag(vcpu, PKVM_HOST_STATE_DIRTY);
+		}
+	}
+
 	/*
 	 * See ARM ARM B1.14.1: "Hyp traps on instructions
 	 * that fail their condition code check"
@@ -489,6 +503,22 @@ int handle_exit(struct kvm_vcpu *vcpu, int exception_index)
 /* For exit types that need handling before we can be preempted */
 void handle_exit_early(struct kvm_vcpu *vcpu, int exception_index)
 {
+	bool inject_serror = ARM_SERROR_PENDING(exception_index) ||
+		ARM_EXCEPTION_CODE(exception_index) == ARM_EXCEPTION_EL1_SERROR;
+
+	/*
+	 * An SError injected below writes the host ctxt; for a non-protected
+	 * guest, sync from the hyp vCPU and keep it dirty so it isn't dropped.
+	 */
+	if (is_protected_kvm_enabled()) {
+		vcpu_clear_flag(vcpu, PKVM_HOST_STATE_DIRTY);
+
+		if (inject_serror && !kvm_vm_is_protected(vcpu->kvm)) {
+			kvm_call_hyp_nvhe(__pkvm_vcpu_sync_state);
+			vcpu_set_flag(vcpu, PKVM_HOST_STATE_DIRTY);
+		}
+	}
+
 	if (ARM_SERROR_PENDING(exception_index)) {
 		if (this_cpu_has_cap(ARM64_HAS_RAS_EXTN)) {
 			u64 disr = kvm_vcpu_get_disr(vcpu);
arch/arm64/kvm/hyp/nvhe/hyp-main.c +81 -5

117 lines not on this page · raw message

Signature · 1 line
2.55.0.rc0.738.g0c8ab3ebcc-goog
#19 lore raw

Re: [PATCH v2 8/8] KVM: arm64: Implement lazy vCPU state sync for non-protected guests

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [High] Dropped register updates for non-protected guests due to missing PKVM_HOST_STATE_DIRTY flag.…

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] Dropped register updates for non-protected guests due to missing PKVM_HOST_STATE_DIRTY flag.
- [High] Missing HCR_EL2 sync on guest exit causes an infinite loop of virtual SError injections. --

commit b6999f3ffe5ca70d542b029d1a2d4a9e5e9d8276
Author: Fuad Tabba <[email protected]>

KVM: arm64: Implement lazy vCPU state sync for non-protected guests

This commit defers copying a non-protected guest's register context between the host and hypervisor. It introduces a PKVM_HOST_STATE_DIRTY flag to only sync the state when modified by the host.

Quoted 3 lines
arch/arm64/kvm/handle_exit.c +0 -0

[ ... ]

arch/arm64/kvm/handle_exit.c int handle_exit(struct kvm_vcpu *vcpu, int exception_index) · line 514

Quoted 13 lines
arch/arm64/kvm/handle_exit.c +9 -0
@@ -489,6 +503,22 @@ int handle_exit(struct kvm_vcpu *vcpu, int exception_index)
 /* For exit types that need handling before we can be preempted */
 void handle_exit_early(struct kvm_vcpu *vcpu, int exception_index)
 {
+	bool inject_serror = ARM_SERROR_PENDING(exception_index) ||
+		ARM_EXCEPTION_CODE(exception_index) == ARM_EXCEPTION_EL1_SERROR;
+
+	/*
+	 * An SError injected below writes the host ctxt; for a non-protected
+	 * guest, sync from the hyp vCPU and keep it dirty so it isn't dropped.
+	 */
+	if (is_protected_kvm_enabled()) {
+		vcpu_clear_flag(vcpu, PKVM_HOST_STATE_DIRTY);

[Severity: High]
Does unconditionally clearing this flag on every exit drop register updates for non-trap exits?

For non-trap exits like ARM_EXCEPTION_IRQ or ARM_EXCEPTION_IL, handle_trap_exceptions() will not be called to set the flag again.

If an asynchronous request like KVM_REQ_VCPU_RESET (triggered by PSCI CPU_ON) modifies the host vcpu->arch.ctxt in the kvm_arch_vcpu_ioctl_run() loop, or if an undefined exception is injected via handle_exit(), flush_hyp_vcpu() will skip copying these updates because the dirty flag is clear.

Could this cause the guest to resume execution from its old pc with stale state?

Quoted 3 lines
arch/arm64/kvm/hyp/nvhe/hyp-main.c +0 -0

[ ... ]

arch/arm64/kvm/hyp/nvhe/hyp-main.c static void sync_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu) · line 208

Quoted 7 lines
arch/arm64/kvm/hyp/nvhe/hyp-main.c +0 -3
@@ -203,9 +255,13 @@ static void sync_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu)
 	fpsimd_sve_sync(&hyp_vcpu->vcpu);
 	sync_debug_state(hyp_vcpu);
 
-	host_vcpu->arch.ctxt		= hyp_vcpu->vcpu.arch.ctxt;
-
-	host_vcpu->arch.hcr_el2		= hyp_vcpu->vcpu.arch.hcr_el2;

[Severity: High]
Does removing the hcr_el2 sync cause an infinite loop of virtual SError injections?

To inject a virtual SError, the host sets HCR_VSE in host_vcpu->arch.hcr_el2. When the guest takes the SError, the hardware clears the VSE bit, which is captured in hyp_vcpu->vcpu.arch.hcr_el2 on exit.

Since this patch removes the copy-back of hcr_el2 to the host for both protected and non-protected guests, the host's HCR_VSE bit remains set permanently.

Will this cause flush_hyp_vcpu() to endlessly re-inject the SError on every subsequent entry?

Quoted 9 lines
arch/arm64/kvm/hyp/nvhe/hyp-main.c +7 -0
+	if (pkvm_hyp_vcpu_is_protected(hyp_vcpu)) {
+		host_vcpu->arch.ctxt = hyp_vcpu->vcpu.arch.ctxt;
+	} else {
+		/* Keep PC (tracepoint) and PSTATE (vcpu_mode_is_bad_32bit) current. */
+		host_vcpu->arch.ctxt.regs.pc = hyp_vcpu->vcpu.arch.ctxt.regs.pc;
+		host_vcpu->arch.ctxt.regs.pstate = hyp_vcpu->vcpu.arch.ctxt.regs.pstate;
+	}
 
 	host_vcpu->arch.fault		= hyp_vcpu->vcpu.arch.fault;
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=8
author
#20 lore raw

Re: [PATCH v2 8/8] KVM: arm64: Implement lazy vCPU state sync for non-protected guests

Quoted 4 lines · On Fri, 19 Jun 2026 at 08:28, <[email protected]> wrote:

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] Dropped register updates for non-protected guests due to missing PKVM_HOST_STATE_DIRTY flag.
- [High] Missing HCR_EL2 sync on guest exit causes an infinite loop of virtual SError injections.

VSE is explicitly synced back, not dropped. The stale-SW-copy concern is pre-existing upstream nVHE (no HCR_EL2 readback from HW); the old full copy synced the same stale value. PENDING_EXCEPTION rides iflags, copied unconditionally (not DIRTY-gated). EL2 re-derives the exception via enter_exception64. KVM_REQ_VCPU_RESET targets the OFF->ON vCPU, not the running one.

/fuad

Quoted 83 lines

--

commit b6999f3ffe5ca70d542b029d1a2d4a9e5e9d8276
Author: Fuad Tabba <[email protected]>

KVM: arm64: Implement lazy vCPU state sync for non-protected guests

This commit defers copying a non-protected guest's register context between the host and hypervisor. It introduces a PKVM_HOST_STATE_DIRTY flag to only sync the state when modified by the host.

Quoted 3 lines
arch/arm64/kvm/handle_exit.c +0 -0

[ ... ]

arch/arm64/kvm/handle_exit.c int handle_exit(struct kvm_vcpu *vcpu, int exception_index) · line 514

Quoted 13 lines
arch/arm64/kvm/handle_exit.c +9 -0
@@ -489,6 +503,22 @@ int handle_exit(struct kvm_vcpu *vcpu, int exception_index)
 /* For exit types that need handling before we can be preempted */
 void handle_exit_early(struct kvm_vcpu *vcpu, int exception_index)
 {
+     bool inject_serror = ARM_SERROR_PENDING(exception_index) ||
+             ARM_EXCEPTION_CODE(exception_index) == ARM_EXCEPTION_EL1_SERROR;
+
+     /*
+      * An SError injected below writes the host ctxt; for a non-protected
+      * guest, sync from the hyp vCPU and keep it dirty so it isn't dropped.
+      */
+     if (is_protected_kvm_enabled()) {
+             vcpu_clear_flag(vcpu, PKVM_HOST_STATE_DIRTY);

[Severity: High]
Does unconditionally clearing this flag on every exit drop register updates for non-trap exits?

For non-trap exits like ARM_EXCEPTION_IRQ or ARM_EXCEPTION_IL, handle_trap_exceptions() will not be called to set the flag again.

If an asynchronous request like KVM_REQ_VCPU_RESET (triggered by PSCI CPU_ON) modifies the host vcpu->arch.ctxt in the kvm_arch_vcpu_ioctl_run() loop, or if an undefined exception is injected via handle_exit(), flush_hyp_vcpu() will skip copying these updates because the dirty flag is clear.

Could this cause the guest to resume execution from its old pc with stale state?

Quoted 3 lines
arch/arm64/kvm/hyp/nvhe/hyp-main.c +0 -0

[ ... ]

arch/arm64/kvm/hyp/nvhe/hyp-main.c static void sync_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu) · line 208

Quoted 7 lines
arch/arm64/kvm/hyp/nvhe/hyp-main.c +0 -3
@@ -203,9 +255,13 @@ static void sync_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu)
      fpsimd_sve_sync(&hyp_vcpu->vcpu);
      sync_debug_state(hyp_vcpu);
 
-     host_vcpu->arch.ctxt            = hyp_vcpu->vcpu.arch.ctxt;
-
-     host_vcpu->arch.hcr_el2         = hyp_vcpu->vcpu.arch.hcr_el2;

[Severity: High]
Does removing the hcr_el2 sync cause an infinite loop of virtual SError injections?

To inject a virtual SError, the host sets HCR_VSE in host_vcpu->arch.hcr_el2. When the guest takes the SError, the hardware clears the VSE bit, which is captured in hyp_vcpu->vcpu.arch.hcr_el2 on exit.

Since this patch removes the copy-back of hcr_el2 to the host for both protected and non-protected guests, the host's HCR_VSE bit remains set permanently.

Will this cause flush_hyp_vcpu() to endlessly re-inject the SError on every subsequent entry?

arch/arm64/kvm/hyp/nvhe/hyp-main.c static void sync_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu) · line 264

Quoted 9 lines
arch/arm64/kvm/hyp/nvhe/hyp-main.c +7 -0
+     if (pkvm_hyp_vcpu_is_protected(hyp_vcpu)) {
+             host_vcpu->arch.ctxt = hyp_vcpu->vcpu.arch.ctxt;
+     } else {
+             /* Keep PC (tracepoint) and PSTATE (vcpu_mode_is_bad_32bit) current. */
+             host_vcpu->arch.ctxt.regs.pc = hyp_vcpu->vcpu.arch.ctxt.regs.pc;
+             host_vcpu->arch.ctxt.regs.pstate = hyp_vcpu->vcpu.arch.ctxt.regs.pstate;
+     }
 
      host_vcpu->arch.fault           = hyp_vcpu->vcpu.arch.fault;

--
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=8

#21 lore raw

Re: [PATCH v2 8/8] KVM: arm64: Implement lazy vCPU state sync for non-protected guests

In reply to Fuad Tabba
Quoted 91 lines · Fuad Tabba

pKVM copies a non-protected guest's register context between the host and the hypervisor on every world switch, even when the host never inspects it. Defer the copy: on entry, flush the host context into the hyp vCPU only when the host marked it dirty (PKVM_HOST_STATE_DIRTY); on exit, leave it in the hyp vCPU and copy it back only when the host needs it, via a __pkvm_vcpu_sync_state hypercall on trap handling or at vcpu put. A protected guest's context is copied as before, since lazy sync only helps where the host is trusted to see the guest's registers.

PC and PSTATE are the exception: they are copied back on every exit so the kvm_exit tracepoint reports the guest's real exit PC, and the run loop's vcpu_mode_is_bad_32bit() and SError-masking checks evaluate the guest's current PSTATE rather than the value left by the previous sync.

handle_exit_early() can also inject an SError, which writes the guest context (ESR_EL1) outside the trap-handling path. For a non-protected guest it therefore syncs the context from the hyp vCPU and marks it dirty, as handle_trap_exceptions() does, so the injection reaches the hyp vCPU on re-entry rather than being dropped.

---
 arch/arm64/include/asm/kvm_asm.h   |  1 +
 arch/arm64/include/asm/kvm_host.h  |  2 +
 arch/arm64/kvm/arm.c               |  7 +++
 arch/arm64/kvm/handle_exit.c       | 30 +++++++++++
 arch/arm64/kvm/hyp/nvhe/hyp-main.c | 86 ++++++++++++++++++++++++++++--
 5 files changed, 121 insertions(+), 5 deletions(-)
arch/arm64/include/asm/kvm_asm.h +1 -0
index 043495f7fc78..6e1135b3ded4 100644
@@ -113,6 +113,7 @@ enum __kvm_host_smccc_func {
 	__KVM_HOST_SMCCC_FUNC___pkvm_finalize_teardown_vm,
 	__KVM_HOST_SMCCC_FUNC___pkvm_vcpu_load,
 	__KVM_HOST_SMCCC_FUNC___pkvm_vcpu_put,
+	__KVM_HOST_SMCCC_FUNC___pkvm_vcpu_sync_state,
 	__KVM_HOST_SMCCC_FUNC___pkvm_tlb_flush_vmid,
 
 	MARKER(__KVM_HOST_SMCCC_FUNC_MAX)
arch/arm64/include/asm/kvm_host.h +2 -0
index 2faa60df847d..caa39ee5125f 100644
@@ -1068,6 +1068,8 @@ struct kvm_vcpu_arch {
 #define INCREMENT_PC		__vcpu_single_flag(iflags, BIT(1))
 /* Target EL/MODE (not a single flag, but let's abuse the macro) */
 #define EXCEPT_MASK		__vcpu_single_flag(iflags, GENMASK(3, 1))
+/* Host-set: the hyp flushes the non-protected vCPU state in on entry */
+#define PKVM_HOST_STATE_DIRTY	__vcpu_single_flag(iflags, BIT(4))
 
 /* Helpers to encode exceptions with minimum fuss */
 #define __EXCEPT_MASK_VAL	unpack_vcpu_flag(EXCEPT_MASK)
arch/arm64/kvm/arm.c +7 -0
index 3732ee9eb0d4..4e89558d8027 100644
@@ -733,6 +733,10 @@ void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu)
 	if (is_protected_kvm_enabled()) {
 		kvm_call_hyp(__vgic_v3_save_aprs, &vcpu->arch.vgic_cpu.vgic_v3);
 		kvm_call_hyp_nvhe(__pkvm_vcpu_put);
+
+		/* __pkvm_vcpu_put implies a sync of the state */
+		if (!kvm_vm_is_protected(vcpu->kvm))
+			vcpu_set_flag(vcpu, PKVM_HOST_STATE_DIRTY);
 	}
 
 	kvm_vcpu_put_debug(vcpu);
@@ -964,6 +968,9 @@ int kvm_arch_vcpu_run_pid_change(struct kvm_vcpu *vcpu)
 		return ret;
 
 	if (is_protected_kvm_enabled()) {
+		/* Start with the vcpu in a dirty state */
+		if (!kvm_vm_is_protected(vcpu->kvm))
+			vcpu_set_flag(vcpu, PKVM_HOST_STATE_DIRTY);
 		ret = pkvm_create_hyp_vm(kvm);
 		if (ret)
 			return ret;
arch/arm64/kvm/handle_exit.c +4 -0
index 54aedf93c78b..8963621bcdd1 100644
@@ -422,6 +422,20 @@ static int handle_trap_exceptions(struct kvm_vcpu *vcpu)
 {
 	int handled;
 
+	/*
+	 * If we run a non-protected VM when protection is enabled
+	 * system-wide, resync the state from the hypervisor and mark
+	 * it as dirty on the host side if it wasn't dirty already

arch/arm64/kvm/handle_exit.c static int handle_trap_exceptions(struct kvm_vcpu *vcpu) · line 438

Quoted 10 lines
arch/arm64/kvm/handle_exit.c +10 -0
+	 * (which could happen if preemption has taken place).
+	 */
+	if (is_protected_kvm_enabled() && !kvm_vm_is_protected(vcpu->kvm)) {
+		guard(preempt)();
+		if (!(vcpu_get_flag(vcpu, PKVM_HOST_STATE_DIRTY))) {
+			kvm_call_hyp_nvhe(__pkvm_vcpu_sync_state);
+			vcpu_set_flag(vcpu, PKVM_HOST_STATE_DIRTY);
+		}
+	}
+

Could we remove this update here and let handle_exit_early() do the sync regardless of the SError injection? One of the main point of handle_exit_early() is to do things under !prempt().

Quoted 5 lines
arch/arm64/kvm/handle_exit.c +0 -0
 	/*
 	 * See ARM ARM B1.14.1: "Hyp traps on instructions
 	 * that fail their condition code check"
@@ -489,6 +503,22 @@ int handle_exit(struct kvm_vcpu *vcpu, int exception_index)
 /* For exit types that need handling before we can be preempted */

arch/arm64/kvm/handle_exit.c int handle_exit(struct kvm_vcpu *vcpu, int exception_index) · line 513

Quoted 10 lines
arch/arm64/kvm/handle_exit.c +8 -0
 void handle_exit_early(struct kvm_vcpu *vcpu, int exception_index)
 {
+	bool inject_serror = ARM_SERROR_PENDING(exception_index) ||
+		ARM_EXCEPTION_CODE(exception_index) == ARM_EXCEPTION_EL1_SERROR;
+
+	/*
+	 * An SError injected below writes the host ctxt; for a non-protected
+	 * guest, sync from the hyp vCPU and keep it dirty so it isn't dropped.
+	 */
+	if (is_protected_kvm_enabled()) {

Should we test !kvm_vm_is_protected(vcpu->kvm) here, as the PKVM_HOST_STATE_DIRTY is only updated for p-guests everywhere else?

Quoted 11 lines
arch/arm64/kvm/handle_exit.c +8 -0
+		vcpu_clear_flag(vcpu, PKVM_HOST_STATE_DIRTY);
+
+		if (inject_serror && !kvm_vm_is_protected(vcpu->kvm)) {
+			kvm_call_hyp_nvhe(__pkvm_vcpu_sync_state);
+			vcpu_set_flag(vcpu, PKVM_HOST_STATE_DIRTY);
+		}
+	}
+
 	if (ARM_SERROR_PENDING(exception_index)) {
 		if (this_cpu_has_cap(ARM64_HAS_RAS_EXTN)) {
 			u64 disr = kvm_vcpu_get_disr(vcpu);

[...]

author
#22 lore raw

Re: [PATCH v2 8/8] KVM: arm64: Implement lazy vCPU state sync for non-protected guests

Quoted 99 lines · Vincent Donnefort
Quoted 97 lines · Fuad Tabba

pKVM copies a non-protected guest's register context between the host and the hypervisor on every world switch, even when the host never inspects it. Defer the copy: on entry, flush the host context into the hyp vCPU only when the host marked it dirty (PKVM_HOST_STATE_DIRTY); on exit, leave it in the hyp vCPU and copy it back only when the host needs it, via a __pkvm_vcpu_sync_state hypercall on trap handling or at vcpu put. A protected guest's context is copied as before, since lazy sync only helps where the host is trusted to see the guest's registers.

PC and PSTATE are the exception: they are copied back on every exit so the kvm_exit tracepoint reports the guest's real exit PC, and the run loop's vcpu_mode_is_bad_32bit() and SError-masking checks evaluate the guest's current PSTATE rather than the value left by the previous sync.

handle_exit_early() can also inject an SError, which writes the guest context (ESR_EL1) outside the trap-handling path. For a non-protected guest it therefore syncs the context from the hyp vCPU and marks it dirty, as handle_trap_exceptions() does, so the injection reaches the hyp vCPU on re-entry rather than being dropped.

---
 arch/arm64/include/asm/kvm_asm.h   |  1 +
 arch/arm64/include/asm/kvm_host.h  |  2 +
 arch/arm64/kvm/arm.c               |  7 +++
 arch/arm64/kvm/handle_exit.c       | 30 +++++++++++
 arch/arm64/kvm/hyp/nvhe/hyp-main.c | 86 ++++++++++++++++++++++++++++--
 5 files changed, 121 insertions(+), 5 deletions(-)
arch/arm64/include/asm/kvm_asm.h +1 -0
index 043495f7fc78..6e1135b3ded4 100644
@@ -113,6 +113,7 @@ enum __kvm_host_smccc_func {
      __KVM_HOST_SMCCC_FUNC___pkvm_finalize_teardown_vm,
      __KVM_HOST_SMCCC_FUNC___pkvm_vcpu_load,
      __KVM_HOST_SMCCC_FUNC___pkvm_vcpu_put,
+     __KVM_HOST_SMCCC_FUNC___pkvm_vcpu_sync_state,
      __KVM_HOST_SMCCC_FUNC___pkvm_tlb_flush_vmid,
 
      MARKER(__KVM_HOST_SMCCC_FUNC_MAX)
arch/arm64/include/asm/kvm_host.h +2 -0
index 2faa60df847d..caa39ee5125f 100644
@@ -1068,6 +1068,8 @@ struct kvm_vcpu_arch {
 #define INCREMENT_PC         __vcpu_single_flag(iflags, BIT(1))
 /* Target EL/MODE (not a single flag, but let's abuse the macro) */
 #define EXCEPT_MASK          __vcpu_single_flag(iflags, GENMASK(3, 1))
+/* Host-set: the hyp flushes the non-protected vCPU state in on entry */
+#define PKVM_HOST_STATE_DIRTY        __vcpu_single_flag(iflags, BIT(4))
 
 /* Helpers to encode exceptions with minimum fuss */
 #define __EXCEPT_MASK_VAL    unpack_vcpu_flag(EXCEPT_MASK)
arch/arm64/kvm/arm.c +7 -0
index 3732ee9eb0d4..4e89558d8027 100644
@@ -733,6 +733,10 @@ void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu)
      if (is_protected_kvm_enabled()) {
              kvm_call_hyp(__vgic_v3_save_aprs, &vcpu->arch.vgic_cpu.vgic_v3);
              kvm_call_hyp_nvhe(__pkvm_vcpu_put);
+
+             /* __pkvm_vcpu_put implies a sync of the state */
+             if (!kvm_vm_is_protected(vcpu->kvm))
+                     vcpu_set_flag(vcpu, PKVM_HOST_STATE_DIRTY);
      }
 
      kvm_vcpu_put_debug(vcpu);
@@ -964,6 +968,9 @@ int kvm_arch_vcpu_run_pid_change(struct kvm_vcpu *vcpu)
              return ret;
 
      if (is_protected_kvm_enabled()) {
+             /* Start with the vcpu in a dirty state */
+             if (!kvm_vm_is_protected(vcpu->kvm))
+                     vcpu_set_flag(vcpu, PKVM_HOST_STATE_DIRTY);
              ret = pkvm_create_hyp_vm(kvm);
              if (ret)
                      return ret;
arch/arm64/kvm/handle_exit.c +10 -0
index 54aedf93c78b..8963621bcdd1 100644
@@ -422,6 +422,20 @@ static int handle_trap_exceptions(struct kvm_vcpu *vcpu)
 {
      int handled;
 
+     /*
+      * If we run a non-protected VM when protection is enabled
+      * system-wide, resync the state from the hypervisor and mark
+      * it as dirty on the host side if it wasn't dirty already
+      * (which could happen if preemption has taken place).
+      */
+     if (is_protected_kvm_enabled() && !kvm_vm_is_protected(vcpu->kvm)) {
+             guard(preempt)();
+             if (!(vcpu_get_flag(vcpu, PKVM_HOST_STATE_DIRTY))) {
+                     kvm_call_hyp_nvhe(__pkvm_vcpu_sync_state);
Quoted 8 lines

arch/arm64/kvm/handle_exit.c static int handle_trap_exceptions(struct kvm_vcpu *vcpu) · line 438

Quoted 4 lines
arch/arm64/kvm/handle_exit.c +4 -0
+                     vcpu_set_flag(vcpu, PKVM_HOST_STATE_DIRTY);
+             }
+     }
+

Could we remove this update here and let handle_exit_early() do the sync regardless of the SError injection? One of the main point of handle_exit_early() is to do things under !prempt().

Agreed on the move: handle_exit_early() is already preempt-off, so the guard() goes away. Not on every exit though. handle_exit_early() runs on every exit, and sync_hyp_vcpu() only copies PC/PSTATE/fault back for a non-protected guest; the GPRs and sysregs cross solely via __pkvm_vcpu_sync_state. Syncing unconditionally would pull the full context back on plain IRQ exits, which is the copy this patch avoids. So I will gate it on trap-or-SError and drop the handle_trap_exceptions() block.

Quoted 12 lines
Quoted 12 lines
arch/arm64/kvm/handle_exit.c +3 -0
      /*
       * See ARM ARM B1.14.1: "Hyp traps on instructions
       * that fail their condition code check"
@@ -489,6 +503,22 @@ int handle_exit(struct kvm_vcpu *vcpu, int exception_index)
 /* For exit types that need handling before we can be preempted */
 void handle_exit_early(struct kvm_vcpu *vcpu, int exception_index)
 {
+     bool inject_serror = ARM_SERROR_PENDING(exception_index) ||
+             ARM_EXCEPTION_CODE(exception_index) == ARM_EXCEPTION_EL1_SERROR;
+
Quoted 8 lines

arch/arm64/kvm/handle_exit.c int handle_exit(struct kvm_vcpu *vcpu, int exception_index) · line 513

Quoted 5 lines
arch/arm64/kvm/handle_exit.c +5 -0
+     /*
+      * An SError injected below writes the host ctxt; for a non-protected
+      * guest, sync from the hyp vCPU and keep it dirty so it isn't dropped.
+      */
+     if (is_protected_kvm_enabled()) {

Should we test !kvm_vm_is_protected(vcpu->kvm) here, as the PKVM_HOST_STATE_DIRTY is only updated for p-guests everywhere else?

Yes. The flag is only ever set for non-protected guests, so clearing it for a protected one is a no-op, but gating it matches the invariant.

Both fold into one block in handle_exit_early():

      if (is_protected_kvm_enabled() && !kvm_vm_is_protected(vcpu->kvm)) {
              if (inject_serror ||
                  ARM_EXCEPTION_CODE(exception_index) == ARM_EXCEPTION_TRAP) {
                      kvm_call_hyp_nvhe(__pkvm_vcpu_sync_state);
                      vcpu_set_flag(vcpu, PKVM_HOST_STATE_DIRTY);
              } else {
                      vcpu_clear_flag(vcpu, PKVM_HOST_STATE_DIRTY);
              }
      }

I will fold this into the next respin.

Thanks for the reviews!
/fuad

Quoted 14 lines
Quoted 12 lines
arch/arm64/kvm/handle_exit.c +8 -0
+             vcpu_clear_flag(vcpu, PKVM_HOST_STATE_DIRTY);
+
+             if (inject_serror && !kvm_vm_is_protected(vcpu->kvm)) {
+                     kvm_call_hyp_nvhe(__pkvm_vcpu_sync_state);
+                     vcpu_set_flag(vcpu, PKVM_HOST_STATE_DIRTY);
+             }
+     }
+
      if (ARM_SERROR_PENDING(exception_index)) {
              if (this_cpu_has_cap(ARM64_HAS_RAS_EXTN)) {
                      u64 disr = kvm_vcpu_get_disr(vcpu);

[...]

#23 lore raw

Re: [PATCH v2 8/8] KVM: arm64: Implement lazy vCPU state sync for non-protected guests

[...]

Quoted 72 lines
Quoted 18 lines
Quoted 18 lines
arch/arm64/kvm/handle_exit.c +10 -0
index 54aedf93c78b..8963621bcdd1 100644
@@ -422,6 +422,20 @@ static int handle_trap_exceptions(struct kvm_vcpu *vcpu)
 {
      int handled;
 
+     /*
+      * If we run a non-protected VM when protection is enabled
+      * system-wide, resync the state from the hypervisor and mark
+      * it as dirty on the host side if it wasn't dirty already
+      * (which could happen if preemption has taken place).
+      */
+     if (is_protected_kvm_enabled() && !kvm_vm_is_protected(vcpu->kvm)) {
+             guard(preempt)();
+             if (!(vcpu_get_flag(vcpu, PKVM_HOST_STATE_DIRTY))) {
+                     kvm_call_hyp_nvhe(__pkvm_vcpu_sync_state);
Quoted 8 lines

arch/arm64/kvm/handle_exit.c static int handle_trap_exceptions(struct kvm_vcpu *vcpu) · line 438

Quoted 4 lines
arch/arm64/kvm/handle_exit.c +4 -0
+                     vcpu_set_flag(vcpu, PKVM_HOST_STATE_DIRTY);
+             }
+     }
+

Could we remove this update here and let handle_exit_early() do the sync regardless of the SError injection? One of the main point of handle_exit_early() is to do things under !prempt().

Agreed on the move: handle_exit_early() is already preempt-off, so the guard() goes away. Not on every exit though. handle_exit_early() runs on every exit, and sync_hyp_vcpu() only copies PC/PSTATE/fault back for a non-protected guest; the GPRs and sysregs cross solely via __pkvm_vcpu_sync_state. Syncing unconditionally would pull the full context back on plain IRQ exits, which is the copy this patch avoids. So I will gate it on trap-or-SError and drop the handle_trap_exceptions() block.

Quoted 12 lines
Quoted 12 lines
arch/arm64/kvm/handle_exit.c +3 -0
      /*
       * See ARM ARM B1.14.1: "Hyp traps on instructions
       * that fail their condition code check"
@@ -489,6 +503,22 @@ int handle_exit(struct kvm_vcpu *vcpu, int exception_index)
 /* For exit types that need handling before we can be preempted */
 void handle_exit_early(struct kvm_vcpu *vcpu, int exception_index)
 {
+     bool inject_serror = ARM_SERROR_PENDING(exception_index) ||
+             ARM_EXCEPTION_CODE(exception_index) == ARM_EXCEPTION_EL1_SERROR;
+
Quoted 8 lines

arch/arm64/kvm/handle_exit.c int handle_exit(struct kvm_vcpu *vcpu, int exception_index) · line 513

Quoted 5 lines
arch/arm64/kvm/handle_exit.c +5 -0
+     /*
+      * An SError injected below writes the host ctxt; for a non-protected
+      * guest, sync from the hyp vCPU and keep it dirty so it isn't dropped.
+      */
+     if (is_protected_kvm_enabled()) {

Should we test !kvm_vm_is_protected(vcpu->kvm) here, as the PKVM_HOST_STATE_DIRTY is only updated for p-guests everywhere else?

Yes. The flag is only ever set for non-protected guests, so clearing it for a protected one is a no-op, but gating it matches the invariant.

Both fold into one block in handle_exit_early():

      if (is_protected_kvm_enabled() && !kvm_vm_is_protected(vcpu->kvm)) {
              if (inject_serror ||
                  ARM_EXCEPTION_CODE(exception_index) == ARM_EXCEPTION_TRAP) {
                      kvm_call_hyp_nvhe(__pkvm_vcpu_sync_state);
                      vcpu_set_flag(vcpu, PKVM_HOST_STATE_DIRTY);
              } else {
                      vcpu_clear_flag(vcpu, PKVM_HOST_STATE_DIRTY);
              }
      }
Quoted 1 line

I will fold this into the next respin.

Ah yes of course, I was hoping we could just have a switch here, just like handle_exit() does, but that's not possible because of ARM_SERROR_PENDING().

Perhaps it would look cleaner if done in a separate function handle_exit_pkvm_state()?

Quoted 18 lines

Thanks for the reviews!
/fuad

Quoted 14 lines
Quoted 12 lines
arch/arm64/kvm/handle_exit.c +8 -0
+             vcpu_clear_flag(vcpu, PKVM_HOST_STATE_DIRTY);
+
+             if (inject_serror && !kvm_vm_is_protected(vcpu->kvm)) {
+                     kvm_call_hyp_nvhe(__pkvm_vcpu_sync_state);
+                     vcpu_set_flag(vcpu, PKVM_HOST_STATE_DIRTY);
+             }
+     }
+
      if (ARM_SERROR_PENDING(exception_index)) {
              if (this_cpu_has_cap(ARM64_HAS_RAS_EXTN)) {
                      u64 disr = kvm_vcpu_get_disr(vcpu);

[...]

author
#24 lore raw

Re: [PATCH v2 8/8] KVM: arm64: Implement lazy vCPU state sync for non-protected guests

Quoted 75 lines · Vincent Donnefort

[...]

Quoted 72 lines
Quoted 18 lines
Quoted 18 lines
arch/arm64/kvm/handle_exit.c +10 -0
index 54aedf93c78b..8963621bcdd1 100644
@@ -422,6 +422,20 @@ static int handle_trap_exceptions(struct kvm_vcpu *vcpu)
 {
      int handled;
 
+     /*
+      * If we run a non-protected VM when protection is enabled
+      * system-wide, resync the state from the hypervisor and mark
+      * it as dirty on the host side if it wasn't dirty already
+      * (which could happen if preemption has taken place).
+      */
+     if (is_protected_kvm_enabled() && !kvm_vm_is_protected(vcpu->kvm)) {
+             guard(preempt)();
+             if (!(vcpu_get_flag(vcpu, PKVM_HOST_STATE_DIRTY))) {
+                     kvm_call_hyp_nvhe(__pkvm_vcpu_sync_state);
Quoted 8 lines

arch/arm64/kvm/handle_exit.c static int handle_trap_exceptions(struct kvm_vcpu *vcpu) · line 438

Quoted 4 lines
arch/arm64/kvm/handle_exit.c +4 -0
+                     vcpu_set_flag(vcpu, PKVM_HOST_STATE_DIRTY);
+             }
+     }
+

Could we remove this update here and let handle_exit_early() do the sync regardless of the SError injection? One of the main point of handle_exit_early() is to do things under !prempt().

Agreed on the move: handle_exit_early() is already preempt-off, so the guard() goes away. Not on every exit though. handle_exit_early() runs on every exit, and sync_hyp_vcpu() only copies PC/PSTATE/fault back for a non-protected guest; the GPRs and sysregs cross solely via __pkvm_vcpu_sync_state. Syncing unconditionally would pull the full context back on plain IRQ exits, which is the copy this patch avoids. So I will gate it on trap-or-SError and drop the handle_trap_exceptions() block.

Quoted 12 lines
Quoted 12 lines
arch/arm64/kvm/handle_exit.c +3 -0
      /*
       * See ARM ARM B1.14.1: "Hyp traps on instructions
       * that fail their condition code check"
@@ -489,6 +503,22 @@ int handle_exit(struct kvm_vcpu *vcpu, int exception_index)
 /* For exit types that need handling before we can be preempted */
 void handle_exit_early(struct kvm_vcpu *vcpu, int exception_index)
 {
+     bool inject_serror = ARM_SERROR_PENDING(exception_index) ||
+             ARM_EXCEPTION_CODE(exception_index) == ARM_EXCEPTION_EL1_SERROR;
+
Quoted 8 lines

arch/arm64/kvm/handle_exit.c int handle_exit(struct kvm_vcpu *vcpu, int exception_index) · line 513

Quoted 5 lines
arch/arm64/kvm/handle_exit.c +5 -0
+     /*
+      * An SError injected below writes the host ctxt; for a non-protected
+      * guest, sync from the hyp vCPU and keep it dirty so it isn't dropped.
+      */
+     if (is_protected_kvm_enabled()) {

Should we test !kvm_vm_is_protected(vcpu->kvm) here, as the PKVM_HOST_STATE_DIRTY is only updated for p-guests everywhere else?

Yes. The flag is only ever set for non-protected guests, so clearing it for a protected one is a no-op, but gating it matches the invariant.

Both fold into one block in handle_exit_early():

      if (is_protected_kvm_enabled() && !kvm_vm_is_protected(vcpu->kvm)) {
              if (inject_serror ||
                  ARM_EXCEPTION_CODE(exception_index) == ARM_EXCEPTION_TRAP) {
                      kvm_call_hyp_nvhe(__pkvm_vcpu_sync_state);
                      vcpu_set_flag(vcpu, PKVM_HOST_STATE_DIRTY);
              } else {
                      vcpu_clear_flag(vcpu, PKVM_HOST_STATE_DIRTY);
              }
      }
Quoted 7 lines
Quoted 1 line

I will fold this into the next respin.

Ah yes of course, I was hoping we could just have a switch here, just like handle_exit() does, but that's not possible because of ARM_SERROR_PENDING().

Perhaps it would look cleaner if done in a separate function handle_exit_pkvm_state()?

Agreed, that reads better. Pulling it into handle_exit_pkvm_state() also gets inject_serror out of handle_exit_early(), which only used it for this. Done for the next respin.

Cheers,
/fuad

Quoted 20 lines
Quoted 20 lines

Thanks for the reviews!
/fuad

Quoted 14 lines
Quoted 12 lines
arch/arm64/kvm/handle_exit.c +8 -0
+             vcpu_clear_flag(vcpu, PKVM_HOST_STATE_DIRTY);
+
+             if (inject_serror && !kvm_vm_is_protected(vcpu->kvm)) {
+                     kvm_call_hyp_nvhe(__pkvm_vcpu_sync_state);
+                     vcpu_set_flag(vcpu, PKVM_HOST_STATE_DIRTY);
+             }
+     }
+
      if (ARM_SERROR_PENDING(exception_index)) {
              if (this_cpu_has_cap(ARM64_HAS_RAS_EXTN)) {
                      u64 disr = kvm_vcpu_get_disr(vcpu);

[...]