Skip to contentSkip to conversation

KVM: arm64: pKVM vCPU state management at EL2

PATCH Superseded Fuad Tabba 3 participants 16 messages Jul 15 – Jul 17

Subsystems (2)

v6 of 7 Superseded Latest is v7 →

lore
Share & tools
lore

Fuad Tabba

author reviewer
#0 lore raw

v6 cover KVM: arm64: pKVM vCPU state management at EL2

Hi folks,

Changes since v5 [1]:
- Reworked how vcpu_{read,write}_sys_reg() are made available to hyp code. Instead of renaming the host-only implementations, keep the canonical names and redirect them to the raw context accessors in nVHE hyp builds via the preprocessor. (Marc)

Building on Will's pKVM infrastructure series [2], this series reworks how pKVM moves vCPU state between the host and EL2, and stops copying a non-protected guest's state on every world switch.

EL2 gains proper primitives for the state it transfers: vCPU lookup helpers, and VGIC flush/sync that reduces how much host state EL2 dereferences. The series also moves some preparatory code (such as sys reg access and PSCI helpers) to shared headers and HYP, and implements lazy copying of a non-protected guest's register state back to the host until the host actually needs it, instead of on every exit.

This is the first of two series moving pKVM vCPU state management to EL2. The follow-up completes the job for protected VMs: state isolation, PSCI handling at EL2, and the resulting API behaviour.

The series is structured as follows:

01-04: Preparatory refactoring (MPIDR, sys reg access, vCPU reset, PSCI
helpers) to shared headers and HYP.

  05:     Host and hypervisor vCPU lookup primitives.
  06-07:  VGIC: reduce EL2's exposure to host state, add flush/sync primitives.
  08:     Lazy state sync for non-protected guests.

Based on v7.2-rc3 (a13c140cc289c).

Cheers,
/fuad

[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/

Fuad Tabba (5):
KVM: arm64: Extract MPIDR computation into a shared header
KVM: arm64: Make vcpu_{read,write}_sys_reg available to HYP code
KVM: arm64: Factor out reusable vCPU reset helpers
KVM: arm64: Move PSCI helper functions to a shared header
KVM: arm64: Implement lazy vCPU state sync for non-protected guests

Marc Zyngier (3):
KVM: arm64: Add host and hypervisor vCPU lookup primitives
KVM: arm64: Minimise EL2's exposure of host VGIC state during world
switch
KVM: arm64: Add primitives to flush/sync the VGIC state at EL2

 arch/arm64/include/asm/kvm_arm.h     |  12 ++
 arch/arm64/include/asm/kvm_asm.h     |   1 +
 arch/arm64/include/asm/kvm_emulate.h |  63 +++++++
 arch/arm64/include/asm/kvm_host.h    |   2 +
 arch/arm64/kvm/arm.c                 |   7 +
 arch/arm64/kvm/handle_exit.c         |  23 +++
 arch/arm64/kvm/hyp/exception.c       |  34 +---
 arch/arm64/kvm/hyp/nvhe/hyp-main.c   | 258 +++++++++++++++++++++++----
 arch/arm64/kvm/psci.c                |  30 +---
 arch/arm64/kvm/reset.c               |  60 +------
 arch/arm64/kvm/sys_regs.c            |  14 +-
 arch/arm64/kvm/sys_regs.h            |  19 ++
 include/kvm/arm_psci.h               |  27 +++
 13 files changed, 390 insertions(+), 160 deletions(-)

base-commit: a13c140cc289c0b7b3770bce5b3ad42ab35074aa

Signature · 1 line
2.39.5
author reviewer
#1 lore raw

PATCH v6 1/8 KVM: arm64: Extract MPIDR computation into a shared header

Extract the vCPU MPIDR computation embedded in reset_mpidr() into a kvm_calculate_mpidr() inline in sys_regs.h, so it can be computed without duplicating the logic. A follow-up series reuses it to reset protected vCPUs at EL2.

No functional change intended.

---
 arch/arm64/kvm/sys_regs.c | 14 +-------------
 arch/arm64/kvm/sys_regs.h | 19 +++++++++++++++++++
 2 files changed, 20 insertions(+), 13 deletions(-)
arch/arm64/kvm/sys_regs.c +1 -13
index 5d5c579d45790..08ba882799d48 100644
@@ -976,21 +976,9 @@ static u64 reset_actlr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r)
 
 static u64 reset_mpidr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r)
 {
-	u64 mpidr;
+	u64 mpidr = kvm_calculate_mpidr(vcpu);
 
-	/*
-	 * Map the vcpu_id into the first three affinity level fields of
-	 * the MPIDR. We limit the number of VCPUs in level 0 due to a
-	 * limitation to 16 CPUs in that level in the ICC_SGIxR registers
-	 * of the GICv3 to be able to address each CPU directly when
-	 * sending IPIs.
-	 */
-	mpidr = (vcpu->vcpu_id & 0x0f) << MPIDR_LEVEL_SHIFT(0);
-	mpidr |= ((vcpu->vcpu_id >> 4) & 0xff) << MPIDR_LEVEL_SHIFT(1);
-	mpidr |= ((vcpu->vcpu_id >> 12) & 0xff) << MPIDR_LEVEL_SHIFT(2);
-	mpidr |= (1ULL << 31);
 	vcpu_write_sys_reg(vcpu, mpidr, MPIDR_EL1);
-
 	return mpidr;
 }
 
arch/arm64/kvm/sys_regs.h +19 -0
index 2a983664220ce..bd56a45abbf9c 100644
@@ -222,6 +222,25 @@ find_reg(const struct sys_reg_params *params, const struct sys_reg_desc table[],
 	return __inline_bsearch((void *)pval, table, num, sizeof(table[0]), match_sys_reg);
 }
 
+static inline u64 kvm_calculate_mpidr(const struct kvm_vcpu *vcpu)
+{
+	u64 mpidr;
+
+	/*
+	 * Map the vcpu_id into the first three affinity level fields of
+	 * the MPIDR. We limit the number of VCPUs in level 0 due to a
+	 * limitation to 16 CPUs in that level in the ICC_SGIxR registers
+	 * of the GICv3 to be able to address each CPU directly when
+	 * sending IPIs.
+	 */
+	mpidr = (vcpu->vcpu_id & 0x0f) << MPIDR_LEVEL_SHIFT(0);
+	mpidr |= ((vcpu->vcpu_id >> 4) & 0xff) << MPIDR_LEVEL_SHIFT(1);
+	mpidr |= ((vcpu->vcpu_id >> 12) & 0xff) << MPIDR_LEVEL_SHIFT(2);
+	mpidr |= (1ULL << 31);
+
+	return mpidr;
+}
+
 const struct sys_reg_desc *get_reg_by_id(u64 id,
 					 const struct sys_reg_desc table[],
 					 unsigned int num);
Signature · 1 line
2.39.5
author reviewer
#2 lore raw

PATCH v6 2/8 KVM: arm64: Make vcpu_{read,write}_sys_reg available to HYP code

In reply to Fuad Tabba

The vcpu_{read,write}_sys_reg() accessors abstract where a system register lives, but their VHE implementation cannot be linked into the nVHE hypervisor, so exception.c open-codes has_vhe() wrappers instead.

Redirect the accessors to the raw context accessors for nVHE hyp builds, where registers are always in memory, and drop the local wrappers. This lets hyp code reuse helpers built on the canonical accessors.

No functional change intended.

---
 arch/arm64/include/asm/kvm_emulate.h |  6 +++++
 arch/arm64/kvm/hyp/exception.c       | 34 ++++++++--------------------
 2 files changed, 15 insertions(+), 25 deletions(-)
arch/arm64/include/asm/kvm_emulate.h +6 -0
index 5bf3d7e1d92c7..fb4062b3ce55a 100644
@@ -506,6 +506,12 @@ static inline unsigned long kvm_vcpu_get_mpidr_aff(struct kvm_vcpu *vcpu)
 	return __vcpu_sys_reg(vcpu, MPIDR_EL1) & MPIDR_HWID_BITMASK;
 }
 
+/* In nVHE hyp code, registers are always in memory: use the raw accessors. */
+#if defined(__KVM_NVHE_HYPERVISOR__)
+#define vcpu_read_sys_reg(v, r)		__vcpu_sys_reg(v, r)
+#define vcpu_write_sys_reg(v, x, r)	__vcpu_assign_sys_reg(v, r, x)
+#endif
+
 static inline void kvm_vcpu_set_be(struct kvm_vcpu *vcpu)
 {
 	if (vcpu_mode_is_32bit(vcpu)) {
arch/arm64/kvm/hyp/exception.c +9 -25
index bef40ddb16dbc..754e2dc1df54a 100644
@@ -20,22 +20,6 @@
 #error Hypervisor code only!
 #endif
 
-static inline u64 __vcpu_read_sys_reg(const struct kvm_vcpu *vcpu, int reg)
-{
-	if (has_vhe())
-		return vcpu_read_sys_reg(vcpu, reg);
-
-	return __vcpu_sys_reg(vcpu, reg);
-}
-
-static inline void __vcpu_write_sys_reg(struct kvm_vcpu *vcpu, u64 val, int reg)
-{
-	if (has_vhe())
-		vcpu_write_sys_reg(vcpu, val, reg);
-	else
-		__vcpu_assign_sys_reg(vcpu, reg, val);
-}
-
 static void __vcpu_write_spsr(struct kvm_vcpu *vcpu, unsigned long target_mode,
 			      u64 val)
 {
@@ -101,14 +85,14 @@ static void enter_exception64(struct kvm_vcpu *vcpu, unsigned long target_mode,
 
 	switch (target_mode) {
 	case PSR_MODE_EL1h:
-		vbar = __vcpu_read_sys_reg(vcpu, VBAR_EL1);
-		sctlr = __vcpu_read_sys_reg(vcpu, SCTLR_EL1);
-		__vcpu_write_sys_reg(vcpu, *vcpu_pc(vcpu), ELR_EL1);
+		vbar = vcpu_read_sys_reg(vcpu, VBAR_EL1);
+		sctlr = vcpu_read_sys_reg(vcpu, SCTLR_EL1);
+		vcpu_write_sys_reg(vcpu, *vcpu_pc(vcpu), ELR_EL1);
 		break;
 	case PSR_MODE_EL2h:
-		vbar = __vcpu_read_sys_reg(vcpu, VBAR_EL2);
-		sctlr = __vcpu_read_sys_reg(vcpu, SCTLR_EL2);
-		__vcpu_write_sys_reg(vcpu, *vcpu_pc(vcpu), ELR_EL2);
+		vbar = vcpu_read_sys_reg(vcpu, VBAR_EL2);
+		sctlr = vcpu_read_sys_reg(vcpu, SCTLR_EL2);
+		vcpu_write_sys_reg(vcpu, *vcpu_pc(vcpu), ELR_EL2);
 		break;
 	default:
 		/* Don't do that */
@@ -185,7 +169,7 @@ static void enter_exception64(struct kvm_vcpu *vcpu, unsigned long target_mode,
  */
 static unsigned long get_except32_cpsr(struct kvm_vcpu *vcpu, u32 mode)
 {
-	u32 sctlr = __vcpu_read_sys_reg(vcpu, SCTLR_EL1);
+	u32 sctlr = vcpu_read_sys_reg(vcpu, SCTLR_EL1);
 	unsigned long old, new;
 
 	old = *vcpu_cpsr(vcpu);
@@ -281,7 +265,7 @@ static void enter_exception32(struct kvm_vcpu *vcpu, u32 mode, u32 vect_offset)
 {
 	unsigned long spsr = *vcpu_cpsr(vcpu);
 	bool is_thumb = (spsr & PSR_AA32_T_BIT);
-	u32 sctlr = __vcpu_read_sys_reg(vcpu, SCTLR_EL1);
+	u32 sctlr = vcpu_read_sys_reg(vcpu, SCTLR_EL1);
 	u32 return_address;
 
 	*vcpu_cpsr(vcpu) = get_except32_cpsr(vcpu, mode);
@@ -305,7 +289,7 @@ static void enter_exception32(struct kvm_vcpu *vcpu, u32 mode, u32 vect_offset)
 	if (sctlr & (1 << 13))
 		vect_offset += 0xffff0000;
 	else /* always have security exceptions */
-		vect_offset += __vcpu_read_sys_reg(vcpu, VBAR_EL1);
+		vect_offset += vcpu_read_sys_reg(vcpu, VBAR_EL1);
 
 	*vcpu_pc(vcpu) = vect_offset;
 }
Signature · 1 line
2.39.5
author reviewer
#3 lore raw

PATCH v6 3/8 KVM: arm64: Factor out reusable vCPU reset helpers

In reply to Fuad Tabba

Pull the reusable pieces out of kvm_reset_vcpu(): expose the reset PSTATE values in kvm_arm.h, and split the core register reset and the PSCI-driven reset into kvm_reset_vcpu_core() and kvm_reset_vcpu_psci(). A follow-up series reuses these to reset protected vCPUs at EL2.

No functional change intended.

---
 arch/arm64/include/asm/kvm_arm.h     | 12 ++++++
 arch/arm64/include/asm/kvm_emulate.h | 57 ++++++++++++++++++++++++++
 arch/arm64/kvm/reset.c               | 60 ++--------------------------
 3 files changed, 72 insertions(+), 57 deletions(-)
arch/arm64/include/asm/kvm_arm.h +12 -0
index 3f9233b5a1308..aba4ec09acd23 100644
@@ -348,4 +348,16 @@
 	{ PSR_AA32_MODE_UND,	"32-bit UND" },	\
 	{ PSR_AA32_MODE_SYS,	"32-bit SYS" }
 
+/*
+ * ARMv8 Reset Values
+ */
+#define VCPU_RESET_PSTATE_EL1	(PSR_MODE_EL1h | PSR_A_BIT | PSR_I_BIT | \
+				 PSR_F_BIT | PSR_D_BIT)
+
+#define VCPU_RESET_PSTATE_EL2	(PSR_MODE_EL2h | PSR_A_BIT | PSR_I_BIT | \
+				 PSR_F_BIT | PSR_D_BIT)
+
+#define VCPU_RESET_PSTATE_SVC	(PSR_AA32_MODE_SVC | PSR_AA32_A_BIT | \
+				 PSR_AA32_I_BIT | PSR_AA32_F_BIT)
+
 #endif /* __ARM64_KVM_ARM_H__ */
arch/arm64/include/asm/kvm_emulate.h +57 -0

61 lines not on this page · raw message

arch/arm64/kvm/reset.c +3 -57

85 lines not on this page · raw message

Signature · 1 line
2.39.5
reviewer Reviewed-by on v6 3/8
#4 lore raw

Re: [PATCH v6 3/8] KVM: arm64: Factor out reusable vCPU reset helpers

Quoted 9 lines · Fuad Tabba

Pull the reusable pieces out of kvm_reset_vcpu(): expose the reset PSTATE values in kvm_arm.h, and split the core register reset and the PSCI-driven reset into kvm_reset_vcpu_core() and kvm_reset_vcpu_psci(). A follow-up series reuses these to reset protected vCPUs at EL2.

No functional change intended.

That looks good. There is another usecase for this. I did a very similar thing in [1] to reuse the reset functionality for arm64 on s390. I just did not the move the code to a different file.

FWIW:

[1] https://lore.kernel.org/all/[email protected]/

Steffen

author reviewer
#5 lore raw

PATCH v6 4/8 KVM: arm64: Move PSCI helper functions to a shared header

In reply to Fuad Tabba

Move kvm_psci_valid_affinity() and kvm_psci_narrow_to_32bit() from psci.c to include/kvm/arm_psci.h, and move psci_affinity_mask() there too, renaming it kvm_psci_affinity_mask() now that it is no longer file-local. A follow-up series handles some protected-guest PSCI calls at EL2 using these helpers.

No functional change intended.

---
 arch/arm64/kvm/psci.c  | 30 +-----------------------------
 include/kvm/arm_psci.h | 27 +++++++++++++++++++++++++++
 2 files changed, 28 insertions(+), 29 deletions(-)
arch/arm64/kvm/psci.c +1 -29
index 3b5dbe9a0a0ea..e3db84400d1f8 100644
@@ -21,16 +21,6 @@
  * as described in ARM document number ARM DEN 0022A.
  */
 
-#define AFFINITY_MASK(level)	~((0x1UL << ((level) * MPIDR_LEVEL_BITS)) - 1)
-
-static unsigned long psci_affinity_mask(unsigned long affinity_level)
-{
-	if (affinity_level <= 3)
-		return MPIDR_HWID_BITMASK & AFFINITY_MASK(affinity_level);
-
-	return 0;
-}
-
 static unsigned long kvm_psci_vcpu_suspend(struct kvm_vcpu *vcpu)
 {
 	/*
@@ -51,12 +41,6 @@ static unsigned long kvm_psci_vcpu_suspend(struct kvm_vcpu *vcpu)
 	return PSCI_RET_SUCCESS;
 }
 
-static inline bool kvm_psci_valid_affinity(struct kvm_vcpu *vcpu,
-					   unsigned long affinity)
-{
-	return !(affinity & ~MPIDR_HWID_BITMASK);
-}
-
 static unsigned long kvm_psci_vcpu_on(struct kvm_vcpu *source_vcpu)
 {
 	struct vcpu_reset_state *reset_state;
@@ -135,7 +119,7 @@ static unsigned long kvm_psci_vcpu_affinity_info(struct kvm_vcpu *vcpu)
 		return PSCI_RET_INVALID_PARAMS;
 
 	/* Determine target affinity mask */
-	target_affinity_mask = psci_affinity_mask(lowest_affinity_level);
+	target_affinity_mask = kvm_psci_affinity_mask(lowest_affinity_level);
 	if (!target_affinity_mask)
 		return PSCI_RET_INVALID_PARAMS;
 
@@ -220,18 +204,6 @@ static void kvm_psci_system_suspend(struct kvm_vcpu *vcpu)
 	run->exit_reason = KVM_EXIT_SYSTEM_EVENT;
 }
 
-static void kvm_psci_narrow_to_32bit(struct kvm_vcpu *vcpu)
-{
-	int i;
-
-	/*
-	 * Zero the input registers' upper 32 bits. They will be fully
-	 * zeroed on exit, so we're fine changing them in place.
-	 */
-	for (i = 1; i < 4; i++)
-		vcpu_set_reg(vcpu, i, lower_32_bits(vcpu_get_reg(vcpu, i)));
-}
-
 static unsigned long kvm_psci_check_allowed_function(struct kvm_vcpu *vcpu, u32 fn)
 {
 	/*
include/kvm/arm_psci.h +27 -0
index cbaec804eb839..f86a006d67136 100644
@@ -38,6 +38,33 @@ static inline int kvm_psci_version(struct kvm_vcpu *vcpu)
 	return KVM_ARM_PSCI_0_1;
 }
 
+/* Narrow the PSCI register arguments (r1 to r3) to 32 bits. */
+static inline void kvm_psci_narrow_to_32bit(struct kvm_vcpu *vcpu)
+{
+	int i;
+
+	/*
+	 * Zero the input registers' upper 32 bits. They will be fully
+	 * zeroed on exit, so we're fine changing them in place.
+	 */
+	for (i = 1; i < 4; i++)
+		vcpu_set_reg(vcpu, i, lower_32_bits(vcpu_get_reg(vcpu, i)));
+}
+
+static inline bool kvm_psci_valid_affinity(struct kvm_vcpu *vcpu,
+					   unsigned long affinity)
+{
+	return !(affinity & ~MPIDR_HWID_BITMASK);
+}
+
+static inline unsigned long kvm_psci_affinity_mask(unsigned long affinity_level)
+{
+	if (affinity_level <= 3)
+		return MPIDR_HWID_BITMASK &
+			~((0x1UL << (affinity_level * MPIDR_LEVEL_BITS)) - 1);
+
+	return 0;
+}
 
 int kvm_psci_call(struct kvm_vcpu *vcpu);
 
Signature · 1 line
2.39.5
#6 lore raw

Re: [PATCH v6 4/8] KVM: arm64: Move PSCI helper functions to a shared header

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] The header `include/kvm/arm_psci.h` is not self-contained because it uses `vcpu_set_reg()` and…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] The header include/kvm/arm_psci.h is not self-contained because it uses vcpu_set_reg() and vcpu_get_reg() without including <asm/kvm_emulate.h>. --

--- Patch [4]: [PATCH v6 4/8] KVM: arm64: Move PSCI helper functions to a shared header ---
(Note: The format of this report has been altered to a free-form summary without direct patch quotation due to automated system recitation restrictions.)

commit 41ab99d154503d49e14931a3e9bb6c067e760b44
Author: Fuad Tabba <[email protected]>

KVM: arm64: Move PSCI helper functions to a shared header

This patch moves several PSCI helper functions out of psci.c and into the shared include/kvm/arm_psci.h header. It renames the file-local psci_affinity_mask() to kvm_psci_affinity_mask() to support follow-up work for protected-guest PSCI calls.

[Severity: Low]
In the newly moved kvm_psci_narrow_to_32bit() function within include/kvm/arm_psci.h, the code makes use of vcpu_set_reg() and vcpu_get_reg(). Does this new header location need an explicit include for asm/kvm_emulate.h to ensure it remains self-contained?

If a C file includes kvm/arm_psci.h without having already included asm/kvm_emulate.h, it looks like it might lead to a build failure due to missing function declarations.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
author reviewer
#7 lore raw

Re: [PATCH v6 4/8] KVM: arm64: Move PSCI helper functions to a shared header

Quoted 3 lines · On Wed, 15 Jul 2026 at 09:24, <[email protected]> wrote:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] The header include/kvm/arm_psci.h is not self-contained because it uses vcpu_set_reg() and vcpu_get_reg() without including <asm/kvm_emulate.h>.

https://lore.kernel.org/all/CA+EHjTwpB2fmoYGfkP5RVZcB4sgbVPQ0=QR5dyUJnpQ_VQa+pA@mail.gmail.com/

/fuad

Quoted 27 lines

--

--- Patch [4]: [PATCH v6 4/8] KVM: arm64: Move PSCI helper functions to a shared header ---
(Note: The format of this report has been altered to a free-form summary without direct patch quotation due to automated system recitation restrictions.)

commit 41ab99d154503d49e14931a3e9bb6c067e760b44
Author: Fuad Tabba <[email protected]>

KVM: arm64: Move PSCI helper functions to a shared header

This patch moves several PSCI helper functions out of psci.c and into the shared include/kvm/arm_psci.h header. It renames the file-local psci_affinity_mask() to kvm_psci_affinity_mask() to support follow-up work for protected-guest PSCI calls.

[Severity: Low]
In the newly moved kvm_psci_narrow_to_32bit() function within include/kvm/arm_psci.h, the code makes use of vcpu_set_reg() and vcpu_get_reg(). Does this new header location need an explicit include for asm/kvm_emulate.h to ensure it remains self-contained?

If a C file includes kvm/arm_psci.h without having already included asm/kvm_emulate.h, it looks like it might lead to a build failure due to missing function declarations.

--
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4

author reviewer
#8 lore raw

PATCH v6 5/8 KVM: arm64: Add host and hypervisor vCPU lookup primitives

In reply to Fuad Tabba

From: Marc Zyngier <[email protected]>

The nVHE hypervisor repeatedly resolves a host vCPU into the EL2 address space and validates that the loaded hyp vCPU matches it, with that logic open-coded in each handler.

Add __get_host_hyp_vcpus() and the get_host_hyp_vcpus() macro, which translate the host vCPU into the hypervisor's address space and, when pKVM is enabled, also return the loaded hyp vCPU if it matches. If pKVM is enabled but the loaded hyp vCPU does not correspond to the requested host vCPU, both the host and hyp vCPU are returned as NULL. Convert handle___kvm_vcpu_run() to use it.

No functional change intended.

---
 arch/arm64/kvm/hyp/nvhe/hyp-main.c | 52 ++++++++++++++++++++++--------
 1 file changed, 38 insertions(+), 14 deletions(-)
arch/arm64/kvm/hyp/nvhe/hyp-main.c +38 -14
index d3c69de698f48..45d717889f6ea 100644
@@ -214,14 +214,45 @@ static void handle___pkvm_vcpu_put(struct kvm_cpu_context *host_ctxt)
 		pkvm_put_hyp_vcpu(hyp_vcpu);
 }
 
-static void handle___kvm_vcpu_run(struct kvm_cpu_context *host_ctxt)
+static struct kvm_vcpu *__get_host_hyp_vcpus(struct kvm_vcpu *arg,
+					     struct pkvm_hyp_vcpu **hyp_vcpup)
 {
-	DECLARE_REG(struct kvm_vcpu *, host_vcpu, host_ctxt, 1);
-	int ret;
+	struct kvm_vcpu *host_vcpu = kern_hyp_va(arg);
+	struct pkvm_hyp_vcpu *hyp_vcpu = NULL;
 
 	if (unlikely(is_protected_kvm_enabled())) {
-		struct pkvm_hyp_vcpu *hyp_vcpu = pkvm_get_loaded_hyp_vcpu();
+		hyp_vcpu = pkvm_get_loaded_hyp_vcpu();
 
+		if (!hyp_vcpu || hyp_vcpu->host_vcpu != host_vcpu) {
+			hyp_vcpu = NULL;
+			host_vcpu = NULL;
+		}
+	}
+
+	*hyp_vcpup = hyp_vcpu;
+	return host_vcpu;
+}
+
+#define get_host_hyp_vcpus(ctxt, regnr, hyp_vcpup)			\
+	({								\
+		DECLARE_REG(struct kvm_vcpu *, __vcpu, ctxt, regnr);	\
+		__get_host_hyp_vcpus(__vcpu, hyp_vcpup);		\
+	})
+
+static void handle___kvm_vcpu_run(struct kvm_cpu_context *host_ctxt)
+{
+	struct pkvm_hyp_vcpu *hyp_vcpu;
+	struct kvm_vcpu *host_vcpu;
+	int ret;
+
+	host_vcpu = get_host_hyp_vcpus(host_ctxt, 1, &hyp_vcpu);
+
+	if (!host_vcpu) {
+		ret = -EINVAL;
+		goto out;
+	}
+
+	if (unlikely(hyp_vcpu)) {
 		/*
 		 * KVM (and pKVM) doesn't support SME guests for now, and
 		 * ensures that SME features aren't enabled in pstate when
@@ -233,23 +264,16 @@ static void handle___kvm_vcpu_run(struct kvm_cpu_context *host_ctxt)
 			goto out;
 		}
 
-		if (!hyp_vcpu) {
-			ret = -EINVAL;
-			goto out;
-		}
-
 		flush_hyp_vcpu(hyp_vcpu);
 
 		ret = __kvm_vcpu_run(&hyp_vcpu->vcpu);
 
 		sync_hyp_vcpu(hyp_vcpu);
 	} else {
-		struct kvm_vcpu *vcpu = kern_hyp_va(host_vcpu);
-
 		/* The host is fully trusted, run its vCPU directly. */
-		fpsimd_lazy_switch_to_guest(vcpu);
-		ret = __kvm_vcpu_run(vcpu);
-		fpsimd_lazy_switch_to_host(vcpu);
+		fpsimd_lazy_switch_to_guest(host_vcpu);
+		ret = __kvm_vcpu_run(host_vcpu);
+		fpsimd_lazy_switch_to_host(host_vcpu);
 	}
 out:
 	cpu_reg(host_ctxt, 1) =  ret;
Signature · 1 line
2.39.5
author reviewer
#9 lore raw

PATCH v6 6/8 KVM: arm64: Minimise EL2's exposure of host VGIC state during world switch

In reply to Fuad Tabba

From: Marc Zyngier <[email protected]>

The host passes a vgic_v3_cpu_if pointer to the __vgic_v3_save_aprs and __vgic_v3_restore_vmcr_aprs hypercalls, which EL2 dereferences wholesale. That exposes the host's full VGIC emulation state to the hypervisor, against pKVM's isolation goals.

Recover the host vCPU from the supplied cpu_if via container_of() and copy only vgic_vmcr and the active priority registers between EL2's hyp-side state and the host vCPU, so EL2 no longer dereferences the host's vgic_v3_cpu_if directly.

---
 arch/arm64/kvm/hyp/nvhe/hyp-main.c | 67 ++++++++++++++++++++++++++++--
 1 file changed, 63 insertions(+), 4 deletions(-)
arch/arm64/kvm/hyp/nvhe/hyp-main.c +63 -4
index 45d717889f6ea..f3233ee343a39 100644
@@ -7,6 +7,8 @@
 #include <hyp/adjust_pc.h>
 #include <hyp/switch.h>
 
+#include <linux/irqchip/arm-gic-v3.h>
+
 #include <asm/pgtable-types.h>
 #include <asm/kvm_asm.h>
 #include <asm/kvm_emulate.h>
@@ -239,6 +241,16 @@ static struct kvm_vcpu *__get_host_hyp_vcpus(struct kvm_vcpu *arg,
 		__get_host_hyp_vcpus(__vcpu, hyp_vcpup);		\
 	})
 
+#define get_host_hyp_vcpus_from_vgic_v3_cpu_if(ctxt, regnr, hyp_vcpup)		\
+	({									\
+		DECLARE_REG(struct vgic_v3_cpu_if *, cif, ctxt, regnr);\
+		struct kvm_vcpu *__vcpu = container_of(cif,			\
+						       struct kvm_vcpu,		\
+						       arch.vgic_cpu.vgic_v3);	\
+										\
+		__get_host_hyp_vcpus(__vcpu, hyp_vcpup);			\
+	})
+
 static void handle___kvm_vcpu_run(struct kvm_cpu_context *host_ctxt)
 {
 	struct pkvm_hyp_vcpu *hyp_vcpu;
@@ -508,16 +520,63 @@ static void handle___vgic_v3_init_lrs(struct kvm_cpu_context *host_ctxt)
 
 static void handle___vgic_v3_save_aprs(struct kvm_cpu_context *host_ctxt)
 {
-	DECLARE_REG(struct vgic_v3_cpu_if *, cpu_if, host_ctxt, 1);
+	struct pkvm_hyp_vcpu *hyp_vcpu;
+	struct kvm_vcpu *host_vcpu;
 
-	__vgic_v3_save_aprs(kern_hyp_va(cpu_if));
+	host_vcpu = get_host_hyp_vcpus_from_vgic_v3_cpu_if(host_ctxt, 1,
+							   &hyp_vcpu);
+	if (!host_vcpu)
+		return;
+
+	if (unlikely(hyp_vcpu)) {
+		struct vgic_v3_cpu_if *hyp_cpu_if, *host_cpu_if;
+		int i;
+
+		hyp_cpu_if = &hyp_vcpu->vcpu.arch.vgic_cpu.vgic_v3;
+		__vgic_v3_save_aprs(hyp_cpu_if);
+
+		host_cpu_if = &host_vcpu->arch.vgic_cpu.vgic_v3;
+		host_cpu_if->vgic_vmcr = hyp_cpu_if->vgic_vmcr;
+		for (i = 0; i < ARRAY_SIZE(host_cpu_if->vgic_ap0r); i++) {
+			host_cpu_if->vgic_ap0r[i] = hyp_cpu_if->vgic_ap0r[i];
+			host_cpu_if->vgic_ap1r[i] = hyp_cpu_if->vgic_ap1r[i];
+		}
+	} else {
+		__vgic_v3_save_aprs(&host_vcpu->arch.vgic_cpu.vgic_v3);
+	}
 }
 
 static void handle___vgic_v3_restore_vmcr_aprs(struct kvm_cpu_context *host_ctxt)
 {
-	DECLARE_REG(struct vgic_v3_cpu_if *, cpu_if, host_ctxt, 1);
+	struct pkvm_hyp_vcpu *hyp_vcpu;
+	struct kvm_vcpu *host_vcpu;
 
-	__vgic_v3_restore_vmcr_aprs(kern_hyp_va(cpu_if));
+	host_vcpu = get_host_hyp_vcpus_from_vgic_v3_cpu_if(host_ctxt, 1,
+							   &hyp_vcpu);
+	if (!host_vcpu)
+		return;
+
+	if (unlikely(hyp_vcpu)) {
+		struct vgic_v3_cpu_if *hyp_cpu_if, *host_cpu_if;
+		int i;
+
+		hyp_cpu_if = &hyp_vcpu->vcpu.arch.vgic_cpu.vgic_v3;
+		host_cpu_if = &host_vcpu->arch.vgic_cpu.vgic_v3;
+
+		hyp_cpu_if->vgic_vmcr = host_cpu_if->vgic_vmcr;
+		/* Should be a one-off */
+		hyp_cpu_if->vgic_sre = (ICC_SRE_EL1_DIB |
+					ICC_SRE_EL1_DFB |
+					ICC_SRE_EL1_SRE);
+		for (i = 0; i < ARRAY_SIZE(host_cpu_if->vgic_ap0r); i++) {
+			hyp_cpu_if->vgic_ap0r[i] = host_cpu_if->vgic_ap0r[i];
+			hyp_cpu_if->vgic_ap1r[i] = host_cpu_if->vgic_ap1r[i];
+		}
+
+		__vgic_v3_restore_vmcr_aprs(hyp_cpu_if);
+	} else {
+		__vgic_v3_restore_vmcr_aprs(&host_vcpu->arch.vgic_cpu.vgic_v3);
+	}
 }
 
 static void handle___pkvm_init(struct kvm_cpu_context *host_ctxt)
Signature · 1 line
2.39.5
#10 lore raw

Re: [PATCH v6 6/8] KVM: arm64: Minimise EL2's exposure of host VGIC state during world switch

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

New issues:
- [High] handle___vgic_v3_save_aprs silently overwrites host modifications to vgic_vmcr with a stale value.

Pre-existing issues:
- [Critical] The patch addresses the host-pointer trust vulnerability for GICv3 but leaves the identical vulnerability open for GICv5. --

--- Patch [6]: [PATCH v6 6/8] KVM: arm64: Minimise EL2's exposure of host VGIC state during world switch --- commit d47381d6be1828e59e8906a9d69d3c104d673dc3
Author: Marc Zyngier <[email protected]>

KVM: arm64: Minimise EL2's exposure of host VGIC state during world switch

This commit modifies the world switch to stop EL2 from dereferencing the host's vgic_v3_cpu_if pointer wholesale. It copies only vgic_vmcr and active priority registers to the hyp-side state.

Quoted 4 lines
arch/arm64/kvm/hyp/nvhe/hyp-main.c +0 -0
index 45d717889f6ea..f3233ee343a39 100644

[ ... ]

Quoted 13 lines
arch/arm64/kvm/hyp/nvhe/hyp-main.c +6 -2
@@ -508,16 +520,63 @@ static void handle___vgic_v3_init_lrs(struct kvm_cpu_context *host_ctxt)
 
 static void handle___vgic_v3_save_aprs(struct kvm_cpu_context *host_ctxt)
 {
-	DECLARE_REG(struct vgic_v3_cpu_if *, cpu_if, host_ctxt, 1);
+	struct pkvm_hyp_vcpu *hyp_vcpu;
+	struct kvm_vcpu *host_vcpu;
 
-	__vgic_v3_save_aprs(kern_hyp_va(cpu_if));
+	host_vcpu = get_host_hyp_vcpus_from_vgic_v3_cpu_if(host_ctxt, 1,
+							   &hyp_vcpu);
+	if (!host_vcpu)
+		return;

arch/arm64/kvm/hyp/nvhe/hyp-main.c static void handle___vgic_v3_init_lrs(struct kvm_cpu_context *host_ctxt) · line 539

Quoted 10 lines
arch/arm64/kvm/hyp/nvhe/hyp-main.c +10 -0
+
+	if (unlikely(hyp_vcpu)) {
+		struct vgic_v3_cpu_if *hyp_cpu_if, *host_cpu_if;
+		int i;
+
+		hyp_cpu_if = &hyp_vcpu->vcpu.arch.vgic_cpu.vgic_v3;
+		__vgic_v3_save_aprs(hyp_cpu_if);
+
+		host_cpu_if = &host_vcpu->arch.vgic_cpu.vgic_v3;
+		host_cpu_if->vgic_vmcr = hyp_cpu_if->vgic_vmcr;

[Severity: High]
Does this assignment overwrite host modifications to vgic_vmcr with a stale value?

When userspace modifies a VMCR field (like SYS_ICC_PMR_EL1) via KVM_SET_ONE_REG, the execution flow modifies the host state:

kvm_vcpu_ioctl()
vcpu_load()

    handle___vgic_v3_restore_vmcr_aprs()
  // ioctl modifies host_cpu_if->vgic_vmcr
  vcpu_put()
    handle___vgic_v3_save_aprs()

Since __vgic_v3_save_aprs() does not read ICH_VMCR_EL2 from the hardware, hyp_cpu_if->vgic_vmcr contains the old value. Does copying it back to host_cpu_if->vgic_vmcr here cause the userspace changes to be lost?

arch/arm64/kvm/hyp/nvhe/hyp-main.c static void handle___vgic_v3_init_lrs(struct kvm_cpu_context *host_ctxt) · line 547

Quoted 8 lines
arch/arm64/kvm/hyp/nvhe/hyp-main.c +7 -0
+		for (i = 0; i < ARRAY_SIZE(host_cpu_if->vgic_ap0r); i++) {
+			host_cpu_if->vgic_ap0r[i] = hyp_cpu_if->vgic_ap0r[i];
+			host_cpu_if->vgic_ap1r[i] = hyp_cpu_if->vgic_ap1r[i];
+		}
+	} else {
+		__vgic_v3_save_aprs(&host_vcpu->arch.vgic_cpu.vgic_v3);
+	}
 }

[ ... ]

[Severity: Critical]
This is a pre-existing issue and was not introduced by the patch under review, but does the same pattern remain open for GICv5?

Looking at handle___vgic_v5_save_apr():

arch/arm64/kvm/hyp/nvhe/hyp-main.c:handle___vgic_v5_save_apr() {
	DECLARE_REG(struct vgic_v5_cpu_if *, cpu_if, host_ctxt, 1);
	__vgic_v5_save_apr(kern_hyp_va(cpu_if));
}

And similarly in handle___vgic_v5_restore_vmcr_apr():

arch/arm64/kvm/hyp/nvhe/hyp-main.c:handle___vgic_v5_restore_vmcr_apr() {
	DECLARE_REG(struct vgic_v5_cpu_if *, cpu_if, host_ctxt, 1);
	__vgic_v5_restore_vmcr_apr(kern_hyp_va(cpu_if));
}

Since these handlers apply kern_hyp_va() to an untrusted host pointer without validation, and kern_hyp_va() is idempotent for already-hyp-tagged pointers, could a malicious host provide a pointer to EL2 private metadata?

Could this allow the host to arbitrarily read or write EL2 memory?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
author reviewer
#11 lore raw

Re: [PATCH v6 6/8] KVM: arm64: Minimise EL2's exposure of host VGIC state during world switch

Quoted 8 lines · On Wed, 15 Jul 2026 at 09:37, <[email protected]> wrote:

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

New issues:
- [High] handle___vgic_v3_save_aprs silently overwrites host modifications to vgic_vmcr with a stale value.

Pre-existing issues:
- [Critical] The patch addresses the host-pointer trust vulnerability for GICv3 but leaves the identical vulnerability open for GICv5.

Both already covered:
https://lore.kernel.org/all/CA+EHjTyGULmVCgyoya3bXG4gRj0OYFE1gnJLhNE6kvCrZFtXyQ@mail.gmail.com/

/fuad

Quoted 95 lines

--

--- Patch [6]: [PATCH v6 6/8] KVM: arm64: Minimise EL2's exposure of host VGIC state during world switch --- commit d47381d6be1828e59e8906a9d69d3c104d673dc3
Author: Marc Zyngier <[email protected]>

KVM: arm64: Minimise EL2's exposure of host VGIC state during world switch

This commit modifies the world switch to stop EL2 from dereferencing the host's vgic_v3_cpu_if pointer wholesale. It copies only vgic_vmcr and active priority registers to the hyp-side state.

Quoted 4 lines
arch/arm64/kvm/hyp/nvhe/hyp-main.c +0 -0
index 45d717889f6ea..f3233ee343a39 100644

[ ... ]

Quoted 13 lines
arch/arm64/kvm/hyp/nvhe/hyp-main.c +6 -2
@@ -508,16 +520,63 @@ static void handle___vgic_v3_init_lrs(struct kvm_cpu_context *host_ctxt)
 
 static void handle___vgic_v3_save_aprs(struct kvm_cpu_context *host_ctxt)
 {
-     DECLARE_REG(struct vgic_v3_cpu_if *, cpu_if, host_ctxt, 1);
+     struct pkvm_hyp_vcpu *hyp_vcpu;
+     struct kvm_vcpu *host_vcpu;
 
-     __vgic_v3_save_aprs(kern_hyp_va(cpu_if));
+     host_vcpu = get_host_hyp_vcpus_from_vgic_v3_cpu_if(host_ctxt, 1,
+                                                        &hyp_vcpu);
+     if (!host_vcpu)
+             return;

arch/arm64/kvm/hyp/nvhe/hyp-main.c static void handle___vgic_v3_init_lrs(struct kvm_cpu_context *host_ctxt) · line 539

Quoted 10 lines
arch/arm64/kvm/hyp/nvhe/hyp-main.c +10 -0
+
+     if (unlikely(hyp_vcpu)) {
+             struct vgic_v3_cpu_if *hyp_cpu_if, *host_cpu_if;
+             int i;
+
+             hyp_cpu_if = &hyp_vcpu->vcpu.arch.vgic_cpu.vgic_v3;
+             __vgic_v3_save_aprs(hyp_cpu_if);
+
+             host_cpu_if = &host_vcpu->arch.vgic_cpu.vgic_v3;
+             host_cpu_if->vgic_vmcr = hyp_cpu_if->vgic_vmcr;

[Severity: High]
Does this assignment overwrite host modifications to vgic_vmcr with a stale value?

When userspace modifies a VMCR field (like SYS_ICC_PMR_EL1) via KVM_SET_ONE_REG, the execution flow modifies the host state:

kvm_vcpu_ioctl()
vcpu_load()

    handle___vgic_v3_restore_vmcr_aprs()
  // ioctl modifies host_cpu_if->vgic_vmcr
  vcpu_put()
    handle___vgic_v3_save_aprs()

Since __vgic_v3_save_aprs() does not read ICH_VMCR_EL2 from the hardware, hyp_cpu_if->vgic_vmcr contains the old value. Does copying it back to host_cpu_if->vgic_vmcr here cause the userspace changes to be lost?

arch/arm64/kvm/hyp/nvhe/hyp-main.c static void handle___vgic_v3_init_lrs(struct kvm_cpu_context *host_ctxt) · line 547

Quoted 8 lines
arch/arm64/kvm/hyp/nvhe/hyp-main.c +7 -0
+             for (i = 0; i < ARRAY_SIZE(host_cpu_if->vgic_ap0r); i++) {
+                     host_cpu_if->vgic_ap0r[i] = hyp_cpu_if->vgic_ap0r[i];
+                     host_cpu_if->vgic_ap1r[i] = hyp_cpu_if->vgic_ap1r[i];
+             }
+     } else {
+             __vgic_v3_save_aprs(&host_vcpu->arch.vgic_cpu.vgic_v3);
+     }
 }

[ ... ]

[Severity: Critical]
This is a pre-existing issue and was not introduced by the patch under review, but does the same pattern remain open for GICv5?

Looking at handle___vgic_v5_save_apr():

arch/arm64/kvm/hyp/nvhe/hyp-main.c:handle___vgic_v5_save_apr() {
        DECLARE_REG(struct vgic_v5_cpu_if *, cpu_if, host_ctxt, 1);
        __vgic_v5_save_apr(kern_hyp_va(cpu_if));
}

And similarly in handle___vgic_v5_restore_vmcr_apr():

arch/arm64/kvm/hyp/nvhe/hyp-main.c:handle___vgic_v5_restore_vmcr_apr() {
        DECLARE_REG(struct vgic_v5_cpu_if *, cpu_if, host_ctxt, 1);
        __vgic_v5_restore_vmcr_apr(kern_hyp_va(cpu_if));
}

Since these handlers apply kern_hyp_va() to an untrusted host pointer without validation, and kern_hyp_va() is idempotent for already-hyp-tagged pointers, could a malicious host provide a pointer to EL2 private metadata?

Could this allow the host to arbitrarily read or write EL2 memory?

--
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6

author reviewer
#12 lore raw

PATCH v6 7/8 KVM: arm64: Add primitives to flush/sync the VGIC state at EL2

In reply to Fuad Tabba

From: Marc Zyngier <[email protected]>

pKVM performs its own world switch for protected VMs but has no primitives to move the per-vCPU VGIC state between the host and hypervisor vCPU contexts.

Add flush_hyp_vgic_state() and sync_hyp_vgic_state(). Flush copies vgic_hcr, the in-use list registers and used_lrs from the host into the hyp vCPU and pins vgic_sre to a fixed value; sync copies vgic_hcr, vgic_vmcr and the in-use list registers back. The active priority registers are handled separately by the save/restore-aprs path.

Bound used_lrs by hyp_gicv3_nr_lr, the cached implemented-LR count, instead of reading ICH_VTR_EL2 on each entry. That clamps the host-supplied value and avoids a per-entry sysreg read that is costly under NV.

---
 arch/arm64/kvm/hyp/nvhe/hyp-main.c | 55 ++++++++++++++++++++++--------
 1 file changed, 41 insertions(+), 14 deletions(-)
arch/arm64/kvm/hyp/nvhe/hyp-main.c +41 -14
index f3233ee343a39..2db56146ec493 100644
@@ -104,6 +104,45 @@ static void fpsimd_sve_sync(struct kvm_vcpu *vcpu)
 	*host_data_ptr(fp_owner) = FP_STATE_HOST_OWNED;
 }
 
+static void flush_hyp_vgic_state(struct pkvm_hyp_vcpu *hyp_vcpu)
+{
+	struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu;
+	struct vgic_v3_cpu_if *host_cpu_if, *hyp_cpu_if;
+	unsigned int used_lrs, i;
+
+	host_cpu_if	= &host_vcpu->arch.vgic_cpu.vgic_v3;
+	hyp_cpu_if	= &hyp_vcpu->vcpu.arch.vgic_cpu.vgic_v3;
+
+	used_lrs	= host_cpu_if->used_lrs;
+	used_lrs	= min(used_lrs, hyp_gicv3_nr_lr);
+
+	hyp_cpu_if->vgic_hcr	= host_cpu_if->vgic_hcr;
+	/* Should be a one-off */
+	hyp_cpu_if->vgic_sre	= (ICC_SRE_EL1_DIB |
+				   ICC_SRE_EL1_DFB |
+				   ICC_SRE_EL1_SRE);
+	hyp_cpu_if->used_lrs	= used_lrs;
+
+	for (i = 0; i < used_lrs; i++)
+		hyp_cpu_if->vgic_lr[i] = host_cpu_if->vgic_lr[i];
+}
+
+static void sync_hyp_vgic_state(struct pkvm_hyp_vcpu *hyp_vcpu)
+{
+	struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu;
+	struct vgic_v3_cpu_if *host_cpu_if, *hyp_cpu_if;
+	unsigned int i;
+
+	host_cpu_if	= &host_vcpu->arch.vgic_cpu.vgic_v3;
+	hyp_cpu_if	= &hyp_vcpu->vcpu.arch.vgic_cpu.vgic_v3;
+
+	host_cpu_if->vgic_hcr = hyp_cpu_if->vgic_hcr;
+	host_cpu_if->vgic_vmcr = hyp_cpu_if->vgic_vmcr;
+
+	for (i = 0; i < hyp_cpu_if->used_lrs; i++)
+		host_cpu_if->vgic_lr[i] = hyp_cpu_if->vgic_lr[i];
+}
+
 static void flush_debug_state(struct pkvm_hyp_vcpu *hyp_vcpu)
 {
 	struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu;
@@ -152,13 +191,7 @@ static void flush_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu)
 
 	hyp_vcpu->vcpu.arch.vsesr_el2	= host_vcpu->arch.vsesr_el2;
 
-	hyp_vcpu->vcpu.arch.vgic_cpu.vgic_v3 = host_vcpu->arch.vgic_cpu.vgic_v3;
-
-	/* Bound used_lrs by the number of implemented list registers. */
-	hyp_vcpu->vcpu.arch.vgic_cpu.vgic_v3.used_lrs =
-		min_t(unsigned int,
-		      hyp_vcpu->vcpu.arch.vgic_cpu.vgic_v3.used_lrs,
-		      hyp_gicv3_nr_lr);
+	flush_hyp_vgic_state(hyp_vcpu);
 
 	hyp_vcpu->vcpu.arch.pid = host_vcpu->arch.pid;
 }
@@ -166,9 +199,6 @@ static void flush_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu)
 static void sync_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu)
 {
 	struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu;
-	struct vgic_v3_cpu_if *hyp_cpu_if = &hyp_vcpu->vcpu.arch.vgic_cpu.vgic_v3;
-	struct vgic_v3_cpu_if *host_cpu_if = &host_vcpu->arch.vgic_cpu.vgic_v3;
-	unsigned int i;
 
 	fpsimd_sve_sync(&hyp_vcpu->vcpu);
 	sync_debug_state(hyp_vcpu);
@@ -181,10 +211,7 @@ static void sync_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu)
 
 	host_vcpu->arch.iflags		= hyp_vcpu->vcpu.arch.iflags;
 
-	host_cpu_if->vgic_hcr		= hyp_cpu_if->vgic_hcr;
-	host_cpu_if->vgic_vmcr		= hyp_cpu_if->vgic_vmcr;
-	for (i = 0; i < hyp_cpu_if->used_lrs; ++i)
-		host_cpu_if->vgic_lr[i] = hyp_cpu_if->vgic_lr[i];
+	sync_hyp_vgic_state(hyp_vcpu);
 }
 
 static void handle___pkvm_vcpu_load(struct kvm_cpu_context *host_ctxt)
Signature · 1 line
2.39.5
author reviewer
#13 lore raw

PATCH v6 8/8 KVM: arm64: Implement lazy vCPU state sync for non-protected guests

In reply to Fuad Tabba

pKVM copies a non-protected guest's register context between the host and the hypervisor on every world switch, even when the host never inspects it. Defer the copy: on entry, flush the host context into the hyp vCPU only when the host marked it dirty (PKVM_HOST_STATE_DIRTY); on exit, leave it in the hyp vCPU and copy it back only when the host needs it, via a __pkvm_vcpu_sync_state hypercall or at vcpu put. A protected guest's context is copied as before, since lazy sync only helps where the host is trusted to see the guest's registers.

PC and PSTATE are the exception: they are copied back on every exit so the kvm_exit tracepoint reports the guest's real exit PC, and the run loop's vcpu_mode_is_bad_32bit() and SError-masking checks evaluate the guest's current PSTATE rather than the value left by the previous sync.

The host needs the full context when it is about to read it (trap handling) or write it (the SError injection that writes ESR_EL1). Sync both from handle_exit_early(), which runs non-preemptible so the loaded hyp vCPU is stable without a preempt guard.

---
 arch/arm64/include/asm/kvm_asm.h   |  1 +
 arch/arm64/include/asm/kvm_host.h  |  2 +
 arch/arm64/kvm/arm.c               |  7 +++
 arch/arm64/kvm/handle_exit.c       | 23 ++++++++
 arch/arm64/kvm/hyp/nvhe/hyp-main.c | 86 ++++++++++++++++++++++++++++--
 5 files changed, 114 insertions(+), 5 deletions(-)
arch/arm64/include/asm/kvm_asm.h +1 -0
index 043495f7fc78b..6e1135b3ded44 100644
@@ -113,6 +113,7 @@ enum __kvm_host_smccc_func {
 	__KVM_HOST_SMCCC_FUNC___pkvm_finalize_teardown_vm,
 	__KVM_HOST_SMCCC_FUNC___pkvm_vcpu_load,
 	__KVM_HOST_SMCCC_FUNC___pkvm_vcpu_put,
+	__KVM_HOST_SMCCC_FUNC___pkvm_vcpu_sync_state,
 	__KVM_HOST_SMCCC_FUNC___pkvm_tlb_flush_vmid,
 
 	MARKER(__KVM_HOST_SMCCC_FUNC_MAX)
arch/arm64/include/asm/kvm_host.h +2 -0
index bae2c4f92ef5c..86d45bc8f4959 100644
@@ -1051,6 +1051,8 @@ struct kvm_vcpu_arch {
 #define INCREMENT_PC		__vcpu_single_flag(iflags, BIT(1))
 /* Target EL/MODE (not a single flag, but let's abuse the macro) */
 #define EXCEPT_MASK		__vcpu_single_flag(iflags, GENMASK(3, 1))
+/* Host-set: the hyp flushes the non-protected vCPU state in on entry */
+#define PKVM_HOST_STATE_DIRTY	__vcpu_single_flag(iflags, BIT(4))
 
 /* Helpers to encode exceptions with minimum fuss */
 #define __EXCEPT_MASK_VAL	unpack_vcpu_flag(EXCEPT_MASK)
arch/arm64/kvm/arm.c +7 -0
index 50adfff75be82..ace7b67fe1cc8 100644
@@ -735,6 +735,10 @@ void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu)
 	if (is_protected_kvm_enabled()) {
 		kvm_call_hyp(__vgic_v3_save_aprs, &vcpu->arch.vgic_cpu.vgic_v3);
 		kvm_call_hyp_nvhe(__pkvm_vcpu_put);
+
+		/* __pkvm_vcpu_put implies a sync of the state */
+		if (!kvm_vm_is_protected(vcpu->kvm))
+			vcpu_set_flag(vcpu, PKVM_HOST_STATE_DIRTY);
 	}
 
 	kvm_vcpu_put_debug(vcpu);
@@ -966,6 +970,9 @@ int kvm_arch_vcpu_run_pid_change(struct kvm_vcpu *vcpu)
 		return ret;
 
 	if (is_protected_kvm_enabled()) {
+		/* Start with the vcpu in a dirty state */
+		if (!kvm_vm_is_protected(vcpu->kvm))
+			vcpu_set_flag(vcpu, PKVM_HOST_STATE_DIRTY);
 		ret = pkvm_create_hyp_vm(kvm);
 		if (ret)
 			return ret;
arch/arm64/kvm/handle_exit.c +23 -0
index 54aedf93c78b6..29108e5c0206e 100644
@@ -486,9 +486,32 @@ int handle_exit(struct kvm_vcpu *vcpu, int exception_index)
 	}
 }
 
+static void handle_exit_pkvm_state(struct kvm_vcpu *vcpu, int exception_index)
+{
+	int exception_code = ARM_EXCEPTION_CODE(exception_index);
+
+	if (!is_protected_kvm_enabled() || kvm_vm_is_protected(vcpu->kvm))
+		return;
+
+	/*
+	 * Sync the context back when the host will read (trap) or write
+	 * (SError) it. Preempt-off here, so the loaded hyp vCPU is stable.
+	 */
+	if (exception_code == ARM_EXCEPTION_TRAP ||
+	    exception_code == ARM_EXCEPTION_EL1_SERROR ||
+	    ARM_SERROR_PENDING(exception_index)) {
+		kvm_call_hyp_nvhe(__pkvm_vcpu_sync_state);
+		vcpu_set_flag(vcpu, PKVM_HOST_STATE_DIRTY);
+	} else {
+		vcpu_clear_flag(vcpu, PKVM_HOST_STATE_DIRTY);
+	}
+}
+
 /* For exit types that need handling before we can be preempted */
 void handle_exit_early(struct kvm_vcpu *vcpu, int exception_index)
 {
+	handle_exit_pkvm_state(vcpu, exception_index);
+
 	if (ARM_SERROR_PENDING(exception_index)) {
 		if (this_cpu_has_cap(ARM64_HAS_RAS_EXTN)) {
 			u64 disr = kvm_vcpu_get_disr(vcpu);
arch/arm64/kvm/hyp/nvhe/hyp-main.c +81 -5

117 lines not on this page · raw message

Signature · 1 line
2.39.5
#14 lore raw

Re: [PATCH v6 8/8] KVM: arm64: Implement lazy vCPU state sync for non-protected guests

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:

New issues:
- [High] Removal of hcr_el2 synchronization breaks virtual SError injection, causing an infinite loop of virtual SErrors.
- [High] Asynchronous host-side modifications to vCPU system registers (e.g., PMU overflow updating PMOVSSET_EL0) are permanently lost due to lazy state sync.

Pre-existing issues:
- [Critical] The hypervisor unconditionally overwrites a protected guest's register context with the untrusted host context on every entry, and leaks it back to the host on exit. --

--- Patch [8]: [PATCH v6 8/8] KVM: arm64: Implement lazy vCPU state sync for non-protected guests --- commit 89d46644a5cb6e03eee6a1ef5d530db27a9d6cdb
Author: Fuad Tabba <[email protected]>
Subject: KVM: arm64: Implement lazy vCPU state sync for non-protected guests

This commit defers the register context copy for non-protected guests between the host and hypervisor. The state is only flushed to the hypervisor if it is marked dirty by the host on entry, and copied back to the host on exit only when needed. Protected guests continue to have their state copied unconditionally.

Quoted 4 lines
arch/arm64/kvm/handle_exit.c +0 -0
index 54aedf93c78b6..29108e5c0206e 100644

[ ... ]

Quoted 7 lines
arch/arm64/kvm/handle_exit.c +3 -0
@@ -486,9 +486,32 @@ int handle_exit(struct kvm_vcpu *vcpu, int exception_index)
 	}
 }
 
+static void handle_exit_pkvm_state(struct kvm_vcpu *vcpu, int exception_index)
+{
+	int exception_code = ARM_EXCEPTION_CODE(exception_index);

[ ... ]

arch/arm64/kvm/handle_exit.c int handle_exit(struct kvm_vcpu *vcpu, int exception_index) · line 506

Quoted 7 lines
arch/arm64/kvm/handle_exit.c +7 -0
+	if (exception_code == ARM_EXCEPTION_TRAP ||
+	    exception_code == ARM_EXCEPTION_EL1_SERROR ||
+	    ARM_SERROR_PENDING(exception_index)) {
+		kvm_call_hyp_nvhe(__pkvm_vcpu_sync_state);
+		vcpu_set_flag(vcpu, PKVM_HOST_STATE_DIRTY);
+	} else {
+		vcpu_clear_flag(vcpu, PKVM_HOST_STATE_DIRTY);

[Severity: High]
Does clearing the dirty flag on IRQ exits result in dropping asynchronous host modifications to vCPU system registers?

When an external IRQ triggers during guest execution, handle_exit_pkvm_state() clears the dirty flag. If the host's IRQ handler asynchronously updates the host's vcpu->arch.ctxt (for example, kvm_pmu_perf_overflow() setting PMOVSSET_EL0), flush_hyp_vcpu() will skip syncing this updated state on the next guest entry.

When the guest later traps, handle_exit_pkvm_state() calls __pkvm_vcpu_sync_state(), which overwrites the host's updated context with the hypervisor's stale context, permanently erasing the asynchronous update.

Quoted 7 lines
arch/arm64/kvm/handle_exit.c +2 -0
+	}
+}
arch/arm64/kvm/hyp/nvhe/hyp-main.c +0 -0
index 2db56146ec493..4a8445b636e60 100644

[ ... ]

Quoted 5 lines
arch/arm64/kvm/hyp/nvhe/hyp-main.c +0 -1
@@ -172,7 +214,17 @@ static void flush_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu)
 	fpsimd_sve_flush();
 	flush_debug_state(hyp_vcpu);
 
-	hyp_vcpu->vcpu.arch.ctxt	= host_vcpu->arch.ctxt;

arch/arm64/kvm/hyp/nvhe/hyp-main.c static void flush_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu) · line 226

Quoted 10 lines
arch/arm64/kvm/hyp/nvhe/hyp-main.c +10 -0
+	/*
+	 * If we deal with a non-protected guest and the state is potentially
+	 * dirty (from a host perspective), copy the state back into the hyp
+	 * vcpu.
+	 */
+	if (!pkvm_hyp_vcpu_is_protected(hyp_vcpu)) {
+		if (vcpu_get_flag(host_vcpu, PKVM_HOST_STATE_DIRTY))
+			flush_hyp_vcpu_state(hyp_vcpu);
+	} else {
+		hyp_vcpu->vcpu.arch.ctxt = host_vcpu->arch.ctxt;

[Severity: Critical]
This is a pre-existing issue, but does this unconditionally overwrite the protected guest's register context with the untrusted host context?

In flush_hyp_vcpu(), this line allows the host EL1 to arbitrarily modify the protected guest's general purpose and system registers, bypassing pKVM's guest confidentiality and integrity guarantees.

Quoted 4 lines
arch/arm64/kvm/hyp/nvhe/hyp-main.c +1 -0
+	}
 
 	/* __hyp_running_vcpu must be NULL in a guest context. */
 	hyp_vcpu->vcpu.arch.ctxt.__hyp_running_vcpu = NULL;

[ ... ]

arch/arm64/kvm/hyp/nvhe/hyp-main.c static void sync_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu) · line 208

Quoted 7 lines
arch/arm64/kvm/hyp/nvhe/hyp-main.c +0 -3
@@ -203,9 +255,13 @@ static void sync_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu)
 	fpsimd_sve_sync(&hyp_vcpu->vcpu);
 	sync_debug_state(hyp_vcpu);
 
-	host_vcpu->arch.ctxt		= hyp_vcpu->vcpu.arch.ctxt;
-
-	host_vcpu->arch.hcr_el2		= hyp_vcpu->vcpu.arch.hcr_el2;

[Severity: High]
Does removing the synchronization of hcr_el2 break virtual SError injection and cause an infinite loop of virtual SErrors?

When the host injects a virtual SError by setting HCR_VSE in its local hcr_el2 copy, the guest takes the SError, and the hardware clears HCR_VSE in the hypervisor's active hcr_el2.

Without this line, the hypervisor's cleared state is never synchronized back to the host. The host's copy permanently retains HCR_VSE=1, and on every subsequent guest entry, flush_hyp_vcpu() will copy HCR_VSE=1 back to the hypervisor, endlessly re-injecting the SError.

arch/arm64/kvm/hyp/nvhe/hyp-main.c static void sync_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu) · line 259

Quoted 2 lines
arch/arm64/kvm/hyp/nvhe/hyp-main.c +2 -0
+	if (pkvm_hyp_vcpu_is_protected(hyp_vcpu)) {
+		host_vcpu->arch.ctxt = hyp_vcpu->vcpu.arch.ctxt;

[Severity: Critical]
This is a pre-existing issue, but does this leak the protected guest's registers back to the untrusted host?

Similar to the issue in flush_hyp_vcpu(), this line in sync_hyp_vcpu() copies the protected guest's register context to the untrusted host on exit, which violates pKVM's threat model guarantees.

Quoted 5 lines
arch/arm64/kvm/hyp/nvhe/hyp-main.c +5 -0
+	} else {
+		/* Keep PC (tracepoint) and PSTATE (vcpu_mode_is_bad_32bit) current. */
+		host_vcpu->arch.ctxt.regs.pc = hyp_vcpu->vcpu.arch.ctxt.regs.pc;
+		host_vcpu->arch.ctxt.regs.pstate = hyp_vcpu->vcpu.arch.ctxt.regs.pstate;
+	}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=8
author reviewer
#15 lore raw

Re: [PATCH v6 8/8] KVM: arm64: Implement lazy vCPU state sync for non-protected guests

Quoted 9 lines · On Wed, 15 Jul 2026 at 09:34, <[email protected]> wrote:

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:

New issues:
- [High] Removal of hcr_el2 synchronization breaks virtual SError injection, causing an infinite loop of virtual SErrors.
- [High] Asynchronous host-side modifications to vCPU system registers (e.g., PMU overflow updating PMOVSSET_EL0) are permanently lost due to lazy state sync.

Pre-existing issues:
- [Critical] The hypervisor unconditionally overwrites a protected guest's register context with the untrusted host context on every entry, and leaks it back to the host on exit.

https://lore.kernel.org/all/CA+EHjTxiyUSJqRcadSCOmR+tELSsBiGaTYH9xxXh-g-xyuHZdA@mail.gmail.com/

/fuad

Quoted 125 lines

--

--- Patch [8]: [PATCH v6 8/8] KVM: arm64: Implement lazy vCPU state sync for non-protected guests --- commit 89d46644a5cb6e03eee6a1ef5d530db27a9d6cdb
Author: Fuad Tabba <[email protected]>
Subject: KVM: arm64: Implement lazy vCPU state sync for non-protected guests

This commit defers the register context copy for non-protected guests between the host and hypervisor. The state is only flushed to the hypervisor if it is marked dirty by the host on entry, and copied back to the host on exit only when needed. Protected guests continue to have their state copied unconditionally.

Quoted 4 lines
arch/arm64/kvm/handle_exit.c +0 -0
index 54aedf93c78b6..29108e5c0206e 100644

[ ... ]

Quoted 7 lines
arch/arm64/kvm/handle_exit.c +3 -0
@@ -486,9 +486,32 @@ int handle_exit(struct kvm_vcpu *vcpu, int exception_index)
      }
 }
 
+static void handle_exit_pkvm_state(struct kvm_vcpu *vcpu, int exception_index)
+{
+     int exception_code = ARM_EXCEPTION_CODE(exception_index);

[ ... ]

arch/arm64/kvm/handle_exit.c int handle_exit(struct kvm_vcpu *vcpu, int exception_index) · line 506

Quoted 7 lines
arch/arm64/kvm/handle_exit.c +7 -0
+     if (exception_code == ARM_EXCEPTION_TRAP ||
+         exception_code == ARM_EXCEPTION_EL1_SERROR ||
+         ARM_SERROR_PENDING(exception_index)) {
+             kvm_call_hyp_nvhe(__pkvm_vcpu_sync_state);
+             vcpu_set_flag(vcpu, PKVM_HOST_STATE_DIRTY);
+     } else {
+             vcpu_clear_flag(vcpu, PKVM_HOST_STATE_DIRTY);

[Severity: High]
Does clearing the dirty flag on IRQ exits result in dropping asynchronous host modifications to vCPU system registers?

When an external IRQ triggers during guest execution, handle_exit_pkvm_state() clears the dirty flag. If the host's IRQ handler asynchronously updates the host's vcpu->arch.ctxt (for example, kvm_pmu_perf_overflow() setting PMOVSSET_EL0), flush_hyp_vcpu() will skip syncing this updated state on the next guest entry.

When the guest later traps, handle_exit_pkvm_state() calls __pkvm_vcpu_sync_state(), which overwrites the host's updated context with the hypervisor's stale context, permanently erasing the asynchronous update.

Quoted 7 lines
arch/arm64/kvm/handle_exit.c +2 -0
+     }
+}
arch/arm64/kvm/hyp/nvhe/hyp-main.c +0 -0
index 2db56146ec493..4a8445b636e60 100644

[ ... ]

Quoted 5 lines
arch/arm64/kvm/hyp/nvhe/hyp-main.c +0 -1
@@ -172,7 +214,17 @@ static void flush_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu)
      fpsimd_sve_flush();
      flush_debug_state(hyp_vcpu);
 
-     hyp_vcpu->vcpu.arch.ctxt        = host_vcpu->arch.ctxt;

arch/arm64/kvm/hyp/nvhe/hyp-main.c static void flush_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu) · line 226

Quoted 10 lines
arch/arm64/kvm/hyp/nvhe/hyp-main.c +10 -0
+     /*
+      * If we deal with a non-protected guest and the state is potentially
+      * dirty (from a host perspective), copy the state back into the hyp
+      * vcpu.
+      */
+     if (!pkvm_hyp_vcpu_is_protected(hyp_vcpu)) {
+             if (vcpu_get_flag(host_vcpu, PKVM_HOST_STATE_DIRTY))
+                     flush_hyp_vcpu_state(hyp_vcpu);
+     } else {
+             hyp_vcpu->vcpu.arch.ctxt = host_vcpu->arch.ctxt;

[Severity: Critical]
This is a pre-existing issue, but does this unconditionally overwrite the protected guest's register context with the untrusted host context?

In flush_hyp_vcpu(), this line allows the host EL1 to arbitrarily modify the protected guest's general purpose and system registers, bypassing pKVM's guest confidentiality and integrity guarantees.

Quoted 4 lines
arch/arm64/kvm/hyp/nvhe/hyp-main.c +1 -0
+     }
 
      /* __hyp_running_vcpu must be NULL in a guest context. */
      hyp_vcpu->vcpu.arch.ctxt.__hyp_running_vcpu = NULL;

[ ... ]

arch/arm64/kvm/hyp/nvhe/hyp-main.c static void sync_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu) · line 208

Quoted 7 lines
arch/arm64/kvm/hyp/nvhe/hyp-main.c +0 -3
@@ -203,9 +255,13 @@ static void sync_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu)
      fpsimd_sve_sync(&hyp_vcpu->vcpu);
      sync_debug_state(hyp_vcpu);
 
-     host_vcpu->arch.ctxt            = hyp_vcpu->vcpu.arch.ctxt;
-
-     host_vcpu->arch.hcr_el2         = hyp_vcpu->vcpu.arch.hcr_el2;

[Severity: High]
Does removing the synchronization of hcr_el2 break virtual SError injection and cause an infinite loop of virtual SErrors?

When the host injects a virtual SError by setting HCR_VSE in its local hcr_el2 copy, the guest takes the SError, and the hardware clears HCR_VSE in the hypervisor's active hcr_el2.

Without this line, the hypervisor's cleared state is never synchronized back to the host. The host's copy permanently retains HCR_VSE=1, and on every subsequent guest entry, flush_hyp_vcpu() will copy HCR_VSE=1 back to the hypervisor, endlessly re-injecting the SError.

arch/arm64/kvm/hyp/nvhe/hyp-main.c static void sync_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu) · line 259

Quoted 2 lines
arch/arm64/kvm/hyp/nvhe/hyp-main.c +2 -0
+     if (pkvm_hyp_vcpu_is_protected(hyp_vcpu)) {
+             host_vcpu->arch.ctxt = hyp_vcpu->vcpu.arch.ctxt;

[Severity: Critical]
This is a pre-existing issue, but does this leak the protected guest's registers back to the untrusted host?

Similar to the issue in flush_hyp_vcpu(), this line in sync_hyp_vcpu() copies the protected guest's register context to the untrusted host on exit, which violates pKVM's threat model guarantees.

arch/arm64/kvm/hyp/nvhe/hyp-main.c static void sync_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu) · line 264

Quoted 5 lines
arch/arm64/kvm/hyp/nvhe/hyp-main.c +5 -0
+     } else {
+             /* Keep PC (tracepoint) and PSTATE (vcpu_mode_is_bad_32bit) current. */
+             host_vcpu->arch.ctxt.regs.pc = hyp_vcpu->vcpu.arch.ctxt.regs.pc;
+             host_vcpu->arch.ctxt.regs.pstate = hyp_vcpu->vcpu.arch.ctxt.regs.pstate;
+     }

--
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=8