Skip to contentSkip to conversation

KVM: arm64: Sync SPSR_EL1 when injecting an exception into a pVM

PATCH Applied Fuad Tabba 4 participants 11 messages Jun 12 – Jun 17

Subsystems (2)
lore
Share & tools
lore

Fuad Tabba

author
#0 lore raw

PATCH 1/1 KVM: arm64: Sync SPSR_EL1 when injecting an exception into a pVM

When pKVM injects a synchronous exception into a protected guest, it re-enters without restoring the guest's EL1 sysregs and writes the EL1 exception registers to hardware by hand: ESR_EL1 and ELR_EL1, but not SPSR_EL1. enter_exception64() sets SPSR_EL1 (the interrupted PSTATE) only in memory, so the guest's handler reads a stale SPSR_EL1 and restores the wrong PSTATE on eret.

Write SPSR_EL1 alongside the other exception registers.

Fixes 6c30bfb18d0b ("KVM: arm64: Add handlers for protected VM System Registers")
---
 arch/arm64/kvm/hyp/nvhe/sys_regs.c | 1 +
 1 file changed, 1 insertion(+)
arch/arm64/kvm/hyp/nvhe/sys_regs.c +1 -0
index 8c3fbb413a06..1a7d5cd16d72 100644
@@ -268,6 +268,7 @@ static void inject_sync64(struct kvm_vcpu *vcpu, u64 esr)
 
 	write_sysreg_el1(esr, SYS_ESR);
 	write_sysreg_el1(read_sysreg_el2(SYS_ELR), SYS_ELR);
+	write_sysreg_el1(read_sysreg_el2(SYS_SPSR), SYS_SPSR);
 	write_sysreg_el2(*vcpu_pc(vcpu), SYS_ELR);
 	write_sysreg_el2(*vcpu_cpsr(vcpu), SYS_SPSR);
 }
Signature · 1 line
2.54.0.1136.gdb2ca164c4-goog
#1 lore raw
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

Pre-existing issues:
- [High] Stale SCTLR_EL1 used during synchronous exception injection in protected VMs leads to incorrect PSTATE computation.
- [High] pKVM's inject_undef64() injects an undefined instruction exception with the IL (Instruction Length) bit cleared in ESR_EL1, incorrectly indicating a 16-bit instruction to an AArch64 guest. --

commit 5157ad995c29fa359271604075525691515c2f95
Author: Fuad Tabba <[email protected]>

KVM: arm64: Sync SPSR_EL1 when injecting an exception into a pVM

When pKVM injects a synchronous exception into a protected guest, it re-enters without restoring the guest's EL1 sysregs. This commit ensures SPSR_EL1 is written alongside the other exception registers to avoid restoring a stale PSTATE on eret.

arch/arm64/kvm/hyp/nvhe/sys_regs.c static void inject_sync64(struct kvm_vcpu *vcpu, u64 esr) · line 270

Quoted 8 lines
arch/arm64/kvm/hyp/nvhe/sys_regs.c +0 -0
index 8c3fbb413a06d..1a7d5cd16d723 100644
@@ -268,6 +268,7 @@ static void inject_sync64(struct kvm_vcpu *vcpu, u64 esr)
 
 	write_sysreg_el1(esr, SYS_ESR);
 	write_sysreg_el1(read_sysreg_el2(SYS_ELR), SYS_ELR);

[Severity: High]
This isn't a bug introduced by this patch, but since we are fixing sysreg synchronization in inject_sync64(), is there also a missing synchronization for SCTLR_EL1 before pending the exception?

Earlier in inject_sync64(), VBAR_EL1 is explicitly synchronized to memory because traps are handled early:

arch/arm64/kvm/hyp/nvhe/sys_regs.c:inject_sync64()

	/*
	 * Make sure we have the latest update to VBAR_EL1, as pKVM
	 * handles traps very early, before sysregs are resync'ed
	 */
	__vcpu_assign_sys_reg(vcpu, VBAR_EL1, read_sysreg_el1(SYS_VBAR));

	kvm_pend_exception(vcpu, EXCEPT_AA64_EL1_SYNC);

	__kvm_adjust_pc(vcpu);

However, when __kvm_adjust_pc() eventually invokes enter_exception64(), it reads SCTLR_EL1 from memory to compute the new PSTATE:

arch/arm64/kvm/hyp/exception.c:enter_exception64()

	sctlr = __vcpu_read_sys_reg(vcpu, SCTLR_EL1);

Because SCTLR_EL1 wasn't synchronized from hardware like VBAR_EL1 was, could the guest's exception handler end up executing with a stale PSTATE.PAN and PSTATE.SSBS?

arch/arm64/kvm/hyp/nvhe/sys_regs.c static void inject_sync64(struct kvm_vcpu *vcpu, u64 esr) · line 274

Quoted 4 lines
arch/arm64/kvm/hyp/nvhe/sys_regs.c +1 -0
+	write_sysreg_el1(read_sysreg_el2(SYS_SPSR), SYS_SPSR);
 	write_sysreg_el2(*vcpu_pc(vcpu), SYS_ELR);
 	write_sysreg_el2(*vcpu_cpsr(vcpu), SYS_SPSR);
 }

[Severity: High]
This is a pre-existing issue, but looking at the related inject_undef64() function located just below inject_sync64():

arch/arm64/kvm/hyp/nvhe/sys_regs.c:inject_undef64() static void inject_undef64(struct kvm_vcpu *vcpu)

{
	inject_sync64(vcpu, (ESR_ELx_EC_UNKNOWN << ESR_ELx_EC_SHIFT));
}

Does this need to set the ESR_ELx_IL bit?

According to the ARM Architecture Reference Manual, for an Unknown reason exception (EC = 0x00) originating from AArch64, the Instruction Length (IL) bit must be set to 1 to indicate a 32-bit instruction.

Unconditionally leaving it as 0 by only bit-shifting ESR_ELx_EC_UNKNOWN exposes an architecturally invalid exception syndrome register to the guest.

Could this cause the guest kernel's exception decoding logic to fail or panic?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
author
#2 lore raw
Quoted 24 lines · On Fri, 12 Jun 2026 at 12:50, <[email protected]> wrote:

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

Pre-existing issues:
- [High] Stale SCTLR_EL1 used during synchronous exception injection in protected VMs leads to incorrect PSTATE computation.
- [High] pKVM's inject_undef64() injects an undefined instruction exception with the IL (Instruction Length) bit cleared in ESR_EL1, incorrectly indicating a 16-bit instruction to an AArch64 guest. --

commit 5157ad995c29fa359271604075525691515c2f95
Author: Fuad Tabba <[email protected]>

KVM: arm64: Sync SPSR_EL1 when injecting an exception into a pVM

When pKVM injects a synchronous exception into a protected guest, it re-enters without restoring the guest's EL1 sysregs. This commit ensures SPSR_EL1 is written alongside the other exception registers to avoid restoring a stale PSTATE on eret.

Quoted 6 lines
arch/arm64/kvm/hyp/nvhe/sys_regs.c +0 -0
index 8c3fbb413a06d..1a7d5cd16d723 100644
@@ -268,6 +268,7 @@ static void inject_sync64(struct kvm_vcpu *vcpu, u64 esr)
 
Quoted 7 lines

arch/arm64/kvm/hyp/nvhe/sys_regs.c static void inject_sync64(struct kvm_vcpu *vcpu, u64 esr) · line 270

Quoted 2 lines
arch/arm64/kvm/hyp/nvhe/sys_regs.c +0 -0
      write_sysreg_el1(esr, SYS_ESR);
      write_sysreg_el1(read_sysreg_el2(SYS_ELR), SYS_ELR);

[Severity: High]
This isn't a bug introduced by this patch, but since we are fixing sysreg synchronization in inject_sync64(), is there also a missing synchronization for SCTLR_EL1 before pending the exception?

How quickly you forget my AI friend. I submitted a fix here, when you flagged this very issue:
https://lore.kernel.org/all/[email protected]/

Quoted 46 lines

Earlier in inject_sync64(), VBAR_EL1 is explicitly synchronized to memory because traps are handled early:

arch/arm64/kvm/hyp/nvhe/sys_regs.c:inject_sync64()

        /*
         * Make sure we have the latest update to VBAR_EL1, as pKVM
         * handles traps very early, before sysregs are resync'ed
         */
        __vcpu_assign_sys_reg(vcpu, VBAR_EL1, read_sysreg_el1(SYS_VBAR));

        kvm_pend_exception(vcpu, EXCEPT_AA64_EL1_SYNC);

        __kvm_adjust_pc(vcpu);

However, when __kvm_adjust_pc() eventually invokes enter_exception64(), it reads SCTLR_EL1 from memory to compute the new PSTATE:

arch/arm64/kvm/hyp/exception.c:enter_exception64()

        sctlr = __vcpu_read_sys_reg(vcpu, SCTLR_EL1);

Because SCTLR_EL1 wasn't synchronized from hardware like VBAR_EL1 was, could the guest's exception handler end up executing with a stale PSTATE.PAN and PSTATE.SSBS?

arch/arm64/kvm/hyp/nvhe/sys_regs.c static void inject_sync64(struct kvm_vcpu *vcpu, u64 esr) · line 274

Quoted 4 lines
arch/arm64/kvm/hyp/nvhe/sys_regs.c +1 -0
+     write_sysreg_el1(read_sysreg_el2(SYS_SPSR), SYS_SPSR);
      write_sysreg_el2(*vcpu_pc(vcpu), SYS_ELR);
      write_sysreg_el2(*vcpu_cpsr(vcpu), SYS_SPSR);
 }

[Severity: High]
This is a pre-existing issue, but looking at the related inject_undef64() function located just below inject_sync64():

arch/arm64/kvm/hyp/nvhe/sys_regs.c:inject_undef64() static void inject_undef64(struct kvm_vcpu *vcpu)

{
        inject_sync64(vcpu, (ESR_ELx_EC_UNKNOWN << ESR_ELx_EC_SHIFT));
}

Does this need to set the ESR_ELx_IL bit?

According to the ARM Architecture Reference Manual, for an Unknown reason exception (EC = 0x00) originating from AArch64, the Instruction Length (IL) bit must be set to 1 to indicate a 32-bit instruction.

Quoted 5 lines

Unconditionally leaving it as 0 by only bit-shifting ESR_ELx_EC_UNKNOWN exposes an architecturally invalid exception syndrome register to the guest.

Could this cause the guest kernel's exception decoding logic to fail or panic?

You're right -- the architecture mandates setting IL here. Like you said, this is pre-existing, and I'll submit a separate patch to fix it. But first, Marc: was there a specific reason for not setting it, or is it just an omission?

Cheers,
/fuad

Quoted 3 lines

--
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1

maintainer
#3 lore raw
Quoted 10 lines · Fuad Tabba
Quoted 3 lines · On Fri, 12 Jun 2026 at 12:50, <[email protected]> wrote:

Could this cause the guest kernel's exception decoding logic to fail or panic?

You're right -- the architecture mandates setting IL here. Like you said, this is pre-existing, and I'll submit a separate patch to fix it. But first, Marc: was there a specific reason for not setting it, or is it just an omission?

Just a plain bug. The normal KVM code is usually careful to construct the new ESR based on the source of the exception, meaning that we always convert ESR_EL2 into ESR_EL1, and propagate IL amongst other things. This is obviously not the case here.

It'd be worth auditing the rest of the code and make sure this is the only case of IL not being correctly set.

Thanks,

	M.
Signature · 1 line
Jazz isn't dead. It just smells funny.
author
#4 lore raw
Quoted 14 lines · Marc Zyngier
Quoted 10 lines · Fuad Tabba
Quoted 3 lines · On Fri, 12 Jun 2026 at 12:50, <[email protected]> wrote:

Could this cause the guest kernel's exception decoding logic to fail or panic?

You're right -- the architecture mandates setting IL here. Like you said, this is pre-existing, and I'll submit a separate patch to fix it. But first, Marc: was there a specific reason for not setting it, or is it just an omission?

Quoted 7 lines

Just a plain bug. The normal KVM code is usually careful to construct the new ESR based on the source of the exception, meaning that we always convert ESR_EL2 into ESR_EL1, and propagate IL amongst other things. This is obviously not the case here.

It'd be worth auditing the rest of the code and make sure this is the only case of IL not being correctly set.

Thanks Marc. I'll do that and fix it on the respin.

Cheers,
/fuad

Quoted 7 lines

Thanks,

        M.

--
Jazz isn't dead. It just smells funny.

maintainer
#5 lore raw
In reply to Fuad Tabba
Quoted 21 lines · Fuad Tabba

When pKVM injects a synchronous exception into a protected guest, it re-enters without restoring the guest's EL1 sysregs and writes the EL1 exception registers to hardware by hand: ESR_EL1 and ELR_EL1, but not SPSR_EL1. enter_exception64() sets SPSR_EL1 (the interrupted PSTATE) only in memory, so the guest's handler reads a stale SPSR_EL1 and restores the wrong PSTATE on eret.

Write SPSR_EL1 alongside the other exception registers.

Fixes 6c30bfb18d0b ("KVM: arm64: Add handlers for protected VM System Registers")
---
 arch/arm64/kvm/hyp/nvhe/sys_regs.c | 1 +
 1 file changed, 1 insertion(+)
arch/arm64/kvm/hyp/nvhe/sys_regs.c +0 -0
index 8c3fbb413a06..1a7d5cd16d72 100644

arch/arm64/kvm/hyp/nvhe/sys_regs.c static void inject_sync64(struct kvm_vcpu *vcpu, u64 esr) · line 274

Quoted 7 lines
arch/arm64/kvm/hyp/nvhe/sys_regs.c +1 -0
@@ -268,6 +268,7 @@ static void inject_sync64(struct kvm_vcpu *vcpu, u64 esr)
 
 	write_sysreg_el1(esr, SYS_ESR);
 	write_sysreg_el1(read_sysreg_el2(SYS_ELR), SYS_ELR);
+	write_sysreg_el1(read_sysreg_el2(SYS_SPSR), SYS_SPSR);
 	write_sysreg_el2(*vcpu_pc(vcpu), SYS_ELR);
 	write_sysreg_el2(*vcpu_cpsr(vcpu), SYS_SPSR);
 }

Is SPSR_EL1 not set in enter_exception64() using vcpu_cpsr(vcpu), which *is* set here? I'm just a bit wary of the report, as I'd have expected fireworks if we weren't initialising the guest's SPSR on the exception injection path.

Will

author
#6 lore raw

Hi Will,

Quoted 24 lines · Will Deacon
Quoted 22 lines · Fuad Tabba

When pKVM injects a synchronous exception into a protected guest, it re-enters without restoring the guest's EL1 sysregs and writes the EL1 exception registers to hardware by hand: ESR_EL1 and ELR_EL1, but not SPSR_EL1. enter_exception64() sets SPSR_EL1 (the interrupted PSTATE) only in memory, so the guest's handler reads a stale SPSR_EL1 and restores the wrong PSTATE on eret.

Write SPSR_EL1 alongside the other exception registers.

Fixes 6c30bfb18d0b ("KVM: arm64: Add handlers for protected VM System Registers")
---
 arch/arm64/kvm/hyp/nvhe/sys_regs.c | 1 +
 1 file changed, 1 insertion(+)
arch/arm64/kvm/hyp/nvhe/sys_regs.c +0 -0
index 8c3fbb413a06..1a7d5cd16d72 100644
@@ -268,6 +268,7 @@ static void inject_sync64(struct kvm_vcpu *vcpu, u64 esr)
 
Quoted 11 lines

arch/arm64/kvm/hyp/nvhe/sys_regs.c static void inject_sync64(struct kvm_vcpu *vcpu, u64 esr) · line 274

Quoted 6 lines
arch/arm64/kvm/hyp/nvhe/sys_regs.c +1 -0
      write_sysreg_el1(esr, SYS_ESR);
      write_sysreg_el1(read_sysreg_el2(SYS_ELR), SYS_ELR);
+     write_sysreg_el1(read_sysreg_el2(SYS_SPSR), SYS_SPSR);
      write_sysreg_el2(*vcpu_pc(vcpu), SYS_ELR);
      write_sysreg_el2(*vcpu_cpsr(vcpu), SYS_SPSR);
 }

Is SPSR_EL1 not set in enter_exception64() using vcpu_cpsr(vcpu), which *is* set here? I'm just a bit wary of the report, as I'd have expected fireworks if we weren't initialising the guest's SPSR on the exception injection path.

Yes, enter_exception64() sets SPSR_EL1, but only in memory:
__vcpu_write_spsr() takes the nVHE path and calls __vcpu_assign_sys_reg(vcpu, SPSR_EL1, val), which writes vcpu->arch.ctxt.sys_regs[SPSR_EL1] without touching the hardware register.

In the normal nVHE entry path that memory value reaches hardware via __sysreg_restore_state_nvhe() before __guest_enter(). But inject_sync64() runs inside the fixup_guest_exit() loop, which re-enters the guest directly without a sysreg restore pass. ESR_EL1 and ELR_EL1 are already written to hardware by hand for this reason but SPSR_EL1 was missed.

I think we don't see fireworks because inject_sync64() only fires on error paths, e.g., accesses to restricted or undefined sysregs. A well-behaved guest shouldn't trigger these, so the stale SPSR_EL1 isn't observed.

Cheers,
/fuad

Quoted 2 lines

Will

maintainer
#7 lore raw
Quoted 43 lines · Fuad Tabba
Quoted 24 lines · Will Deacon
Quoted 22 lines · Fuad Tabba

When pKVM injects a synchronous exception into a protected guest, it re-enters without restoring the guest's EL1 sysregs and writes the EL1 exception registers to hardware by hand: ESR_EL1 and ELR_EL1, but not SPSR_EL1. enter_exception64() sets SPSR_EL1 (the interrupted PSTATE) only in memory, so the guest's handler reads a stale SPSR_EL1 and restores the wrong PSTATE on eret.

Write SPSR_EL1 alongside the other exception registers.

Fixes 6c30bfb18d0b ("KVM: arm64: Add handlers for protected VM System Registers")
---
 arch/arm64/kvm/hyp/nvhe/sys_regs.c | 1 +
 1 file changed, 1 insertion(+)
arch/arm64/kvm/hyp/nvhe/sys_regs.c +0 -0
index 8c3fbb413a06..1a7d5cd16d72 100644
@@ -268,6 +268,7 @@ static void inject_sync64(struct kvm_vcpu *vcpu, u64 esr)
 
Quoted 11 lines

arch/arm64/kvm/hyp/nvhe/sys_regs.c static void inject_sync64(struct kvm_vcpu *vcpu, u64 esr) · line 274

Quoted 6 lines
arch/arm64/kvm/hyp/nvhe/sys_regs.c +1 -0
      write_sysreg_el1(esr, SYS_ESR);
      write_sysreg_el1(read_sysreg_el2(SYS_ELR), SYS_ELR);
+     write_sysreg_el1(read_sysreg_el2(SYS_SPSR), SYS_SPSR);
      write_sysreg_el2(*vcpu_pc(vcpu), SYS_ELR);
      write_sysreg_el2(*vcpu_cpsr(vcpu), SYS_SPSR);
 }

Is SPSR_EL1 not set in enter_exception64() using vcpu_cpsr(vcpu), which *is* set here? I'm just a bit wary of the report, as I'd have expected fireworks if we weren't initialising the guest's SPSR on the exception injection path.

Yes, enter_exception64() sets SPSR_EL1, but only in memory:
__vcpu_write_spsr() takes the nVHE path and calls __vcpu_assign_sys_reg(vcpu, SPSR_EL1, val), which writes vcpu->arch.ctxt.sys_regs[SPSR_EL1] without touching the hardware register.

Quoted 6 lines

In the normal nVHE entry path that memory value reaches hardware via __sysreg_restore_state_nvhe() before __guest_enter(). But inject_sync64() runs inside the fixup_guest_exit() loop, which re-enters the guest directly without a sysreg restore pass. ESR_EL1 and ELR_EL1 are already written to hardware by hand for this reason but SPSR_EL1 was missed.

Ah, I see. Does that mean that all the in-memory updates performed by enter_exception64() (e.g. the construction of the cpsr) are ignored too?

Will

author
#8 lore raw
Quoted 52 lines · Will Deacon
Quoted 43 lines · Fuad Tabba
Quoted 24 lines · Will Deacon
Quoted 22 lines · Fuad Tabba

When pKVM injects a synchronous exception into a protected guest, it re-enters without restoring the guest's EL1 sysregs and writes the EL1 exception registers to hardware by hand: ESR_EL1 and ELR_EL1, but not SPSR_EL1. enter_exception64() sets SPSR_EL1 (the interrupted PSTATE) only in memory, so the guest's handler reads a stale SPSR_EL1 and restores the wrong PSTATE on eret.

Write SPSR_EL1 alongside the other exception registers.

Fixes 6c30bfb18d0b ("KVM: arm64: Add handlers for protected VM System Registers")
---
 arch/arm64/kvm/hyp/nvhe/sys_regs.c | 1 +
 1 file changed, 1 insertion(+)
arch/arm64/kvm/hyp/nvhe/sys_regs.c +0 -0
index 8c3fbb413a06..1a7d5cd16d72 100644
@@ -268,6 +268,7 @@ static void inject_sync64(struct kvm_vcpu *vcpu, u64 esr)
 
Quoted 11 lines

arch/arm64/kvm/hyp/nvhe/sys_regs.c static void inject_sync64(struct kvm_vcpu *vcpu, u64 esr) · line 274

Quoted 6 lines
arch/arm64/kvm/hyp/nvhe/sys_regs.c +1 -0
      write_sysreg_el1(esr, SYS_ESR);
      write_sysreg_el1(read_sysreg_el2(SYS_ELR), SYS_ELR);
+     write_sysreg_el1(read_sysreg_el2(SYS_SPSR), SYS_SPSR);
      write_sysreg_el2(*vcpu_pc(vcpu), SYS_ELR);
      write_sysreg_el2(*vcpu_cpsr(vcpu), SYS_SPSR);
 }

Is SPSR_EL1 not set in enter_exception64() using vcpu_cpsr(vcpu), which *is* set here? I'm just a bit wary of the report, as I'd have expected fireworks if we weren't initialising the guest's SPSR on the exception injection path.

Yes, enter_exception64() sets SPSR_EL1, but only in memory:
__vcpu_write_spsr() takes the nVHE path and calls __vcpu_assign_sys_reg(vcpu, SPSR_EL1, val), which writes vcpu->arch.ctxt.sys_regs[SPSR_EL1] without touching the hardware register.

Quoted 6 lines

In the normal nVHE entry path that memory value reaches hardware via __sysreg_restore_state_nvhe() before __guest_enter(). But inject_sync64() runs inside the fixup_guest_exit() loop, which re-enters the guest directly without a sysreg restore pass. ESR_EL1 and ELR_EL1 are already written to hardware by hand for this reason but SPSR_EL1 was missed.

Quoted 2 lines

Ah, I see. Does that mean that all the in-memory updates performed by enter_exception64() (e.g. the construction of the cpsr) are ignored too?

From looking at the code, no. inject_sync64() reads back the cpsr and PC that enter_exception64() computed in memory and writes them to hardware:

    write_sysreg_el2(*vcpu_pc(vcpu), SYS_ELR);
    write_sysreg_el2(*vcpu_cpsr(vcpu), SYS_SPSR);

So the guest re-enters with the right PSTATE. The only value left stranded in memory is SPSR_EL1.

Cheers,
/fuad

Quoted 2 lines

Will

maintainer Acked-by
#9 lore raw
Quoted 60 lines · Fuad Tabba
Quoted 52 lines · Will Deacon
Quoted 43 lines · Fuad Tabba
Quoted 24 lines · Will Deacon
Quoted 22 lines · Fuad Tabba

When pKVM injects a synchronous exception into a protected guest, it re-enters without restoring the guest's EL1 sysregs and writes the EL1 exception registers to hardware by hand: ESR_EL1 and ELR_EL1, but not SPSR_EL1. enter_exception64() sets SPSR_EL1 (the interrupted PSTATE) only in memory, so the guest's handler reads a stale SPSR_EL1 and restores the wrong PSTATE on eret.

Write SPSR_EL1 alongside the other exception registers.

Fixes 6c30bfb18d0b ("KVM: arm64: Add handlers for protected VM System Registers")
---
 arch/arm64/kvm/hyp/nvhe/sys_regs.c | 1 +
 1 file changed, 1 insertion(+)
arch/arm64/kvm/hyp/nvhe/sys_regs.c +0 -0
index 8c3fbb413a06..1a7d5cd16d72 100644
@@ -268,6 +268,7 @@ static void inject_sync64(struct kvm_vcpu *vcpu, u64 esr)
 
Quoted 11 lines

arch/arm64/kvm/hyp/nvhe/sys_regs.c static void inject_sync64(struct kvm_vcpu *vcpu, u64 esr) · line 274

Quoted 6 lines
arch/arm64/kvm/hyp/nvhe/sys_regs.c +1 -0
      write_sysreg_el1(esr, SYS_ESR);
      write_sysreg_el1(read_sysreg_el2(SYS_ELR), SYS_ELR);
+     write_sysreg_el1(read_sysreg_el2(SYS_SPSR), SYS_SPSR);
      write_sysreg_el2(*vcpu_pc(vcpu), SYS_ELR);
      write_sysreg_el2(*vcpu_cpsr(vcpu), SYS_SPSR);
 }

Is SPSR_EL1 not set in enter_exception64() using vcpu_cpsr(vcpu), which *is* set here? I'm just a bit wary of the report, as I'd have expected fireworks if we weren't initialising the guest's SPSR on the exception injection path.

Yes, enter_exception64() sets SPSR_EL1, but only in memory:
__vcpu_write_spsr() takes the nVHE path and calls __vcpu_assign_sys_reg(vcpu, SPSR_EL1, val), which writes vcpu->arch.ctxt.sys_regs[SPSR_EL1] without touching the hardware register.

Quoted 6 lines

In the normal nVHE entry path that memory value reaches hardware via __sysreg_restore_state_nvhe() before __guest_enter(). But inject_sync64() runs inside the fixup_guest_exit() loop, which re-enters the guest directly without a sysreg restore pass. ESR_EL1 and ELR_EL1 are already written to hardware by hand for this reason but SPSR_EL1 was missed.

Quoted 2 lines

Ah, I see. Does that mean that all the in-memory updates performed by enter_exception64() (e.g. the construction of the cpsr) are ignored too?

From looking at the code, no. inject_sync64() reads back the cpsr and PC that enter_exception64() computed in memory and writes them to hardware:

Quoted 5 lines
    write_sysreg_el2(*vcpu_pc(vcpu), SYS_ELR);
    write_sysreg_el2(*vcpu_cpsr(vcpu), SYS_SPSR);

So the guest re-enters with the right PSTATE. The only value left stranded in memory is SPSR_EL1.

Ah, thanks. I had pissed that inject_sync64() calls __kvm_adjust_pc() immediately after pending the exception. So you've convinced me:

Will

maintainer Applied
#10 lore raw
In reply to Fuad Tabba
Quoted 8 lines · Fuad Tabba

When pKVM injects a synchronous exception into a protected guest, it re-enters without restoring the guest's EL1 sysregs and writes the EL1 exception registers to hardware by hand: ESR_EL1 and ELR_EL1, but not SPSR_EL1. enter_exception64() sets SPSR_EL1 (the interrupted PSTATE) only in memory, so the guest's handler reads a stale SPSR_EL1 and restores the wrong PSTATE on eret.

[...]

Applied to fixes, thanks!

[1/1] KVM: arm64: Sync SPSR_EL1 when injecting an exception into a pVM

      commit: ec40342aaca8162bc8ab2607076535ebab1838b8

Cheers,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.