KVM: arm64: Fixes for resolving the fault IPA
v2 cover KVM: arm64: Fixes for resolving the fault IPA
KVM's heuristics for determining the fault IPA are a bit shaky and don't necessarily align with the letter of the architecture. On top of that, HPFAR_EL2 is UNKNOWN if an SEA occurred during a table walk. Re-walking the page tables will replay the SEA at EL2, becoming a panic in the hyp.
v1: https://lore.kernel.org/kvmarm/[email protected]/
v1 -> v2 (addressed Marc's feedback):
- Drop section reference in favor of the rule for HPFAR_EL2 validity
- Fold test for Address size fault into helper, reuse existing definitions
- Get rid of 'magic' shift for transforming FIPA field to an actual IPA
- Add BUG_ON() to handle_host_mem_abort() if we fail to resolve the fault IPA.
Oliver Upton (3):
KVM: arm64: Only read HPFAR_EL2 when value is architecturally valid
arm64: Convert HPFAR_EL2 to sysreg table
KVM: arm64: Don't translate FAR if invalid/unsafe
arch/arm64/include/asm/esr.h | 44 +++++++++++++++- arch/arm64/include/asm/kvm_emulate.h | 7 ++- arch/arm64/include/asm/kvm_ras.h | 2 +- arch/arm64/kvm/hyp/include/hyp/fault.h | 70 ++++++++++++++++++-------- arch/arm64/kvm/hyp/nvhe/mem_protect.c | 9 +++- arch/arm64/kvm/mmu.c | 31 +++++++----- arch/arm64/tools/sysreg | 7 +++ 7 files changed, 131 insertions(+), 39 deletions(-)
base-commit: 369c0122682c4468a69f2454614eded71c5348f3
Signature · 1 line
2.39.5
PATCH v2 1/3 KVM: arm64: Only read HPFAR_EL2 when value is architecturally valid
KVM's logic for deciding when HPFAR_EL2 is UNKNOWN doesn't align with the architecture. Most notably, KVM assumes HPFAR_EL2 contains the faulting IPA even in the case of an SEA.
Align the logic with the architecture rather than attempting to paraphrase it. Additionally, take the opportunity to improve the language around ARM erratum #834220 such that it actually describes the bug.
--- arch/arm64/include/asm/esr.h | 22 ++++++++++-- arch/arm64/kvm/hyp/include/hyp/fault.h | 46 ++++++++++++++++---------- 2 files changed, 48 insertions(+), 20 deletions(-)
arch/arm64/include/asm/esr.h +20 -2
@@ -121,6 +121,15 @@#define ESR_ELx_FSC_SEA_TTW (n) (0x14 + (n))#define ESR_ELx_FSC_SECC (0x18)#define ESR_ELx_FSC_SECC_TTW (n) (0x1c + (n)) +#define ESR_ELx_FSC_ADDRSZ (0x00) + +/* +* Annoyingly, the negative levels for Address size faults aren't laid out +* contiguously (or in the desired order) +*/ +#define ESR_ELx_FSC_ADDRSZ_nL (n) ((n) == -1 ? 0x25 : 0x2C) +#define ESR_ELx_FSC_ADDRSZ_L (n) ((n) < 0 ? ESR_ELx_FSC_ADDRSZ_nL(n) : \ + (ESR_ELx_FSC_ADDRSZ + (n)))/* Status codes for individual page table levels */ #define ESR_ELx_FSC_ACCESS_L (n) (ESR_ELx_FSC_ACCESS + (n))
@@ -161,8 +170,6 @@#define ESR_ELx_Xs_MASK (GENMASK_ULL(4, 0))/* ISS field definitions for exceptions taken in to Hyp */ -#define ESR_ELx_FSC_ADDRSZ (0x00) -#define ESR_ELx_FSC_ADDRSZ_L (n) (ESR_ELx_FSC_ADDRSZ + (n))#define ESR_ELx_CV (UL(1) << 24)#define ESR_ELx_COND_SHIFT (20)#define ESR_ELx_COND_MASK (UL(0xF) << ESR_ELx_COND_SHIFT)
@@ -464,6 +471,17 @@ static inline bool esr_fsc_is_access_flag_fault(unsigned long esr) (esr ==ESR_ELx_FSC_ACCESS_L (0 )); } +static inline bool esr_fsc_is_addr_sz_fault (unsigned long esr) +{ + esr &= ESR_ELx_FSC; + +return (esr ==ESR_ELx_FSC_ADDRSZ_L (3 )) || + (esr ==ESR_ELx_FSC_ADDRSZ_L (2 )) || + (esr ==ESR_ELx_FSC_ADDRSZ_L (1 )) || + (esr ==ESR_ELx_FSC_ADDRSZ_L (0 )) || + (esr ==ESR_ELx_FSC_ADDRSZ_L (-1 )); +} +/* Indicate whether ESR.EC==0x1A is for an ERETAx instruction */ static inline bool esr_iss_is_eretax (unsigned long esr) {
arch/arm64/kvm/hyp/include/hyp/fault.h +28 -18
59 lines not on this page · raw message
Signature · 1 line
2.39.5
Marc Zyngier
maintainerRe: [PATCH v2 1/3] KVM: arm64: Only read HPFAR_EL2 when value is architecturally valid
Quoted 23 lines · Oliver Upton
KVM's logic for deciding when
HPFAR_EL2is UNKNOWN doesn't align with the architecture. Most notably, KVM assumesHPFAR_EL2contains the faulting IPA even in the case of an SEA.Align the logic with the architecture rather than attempting to paraphrase it. Additionally, take the opportunity to improve the language around ARM erratum #834220 such that it actually describes the bug.
Signed-off-by Oliver Upton <[email protected]>--- arch/arm64/include/asm/esr.h | 22 ++++++++++-- arch/arm64/kvm/hyp/include/hyp/fault.h | 46 ++++++++++++++++---------- 2 files changed, 48 insertions(+), 20 deletions(-)arch/arm64/include/asm/esr.h +0
-0@@ -121,6 +121,15 @@#define ESR_ELx_FSC_SEA_TTW (n) (0x14 + (n))#define ESR_ELx_FSC_SECC (0x18)
arch/arm64/include/asm/esr.h line 132
Quoted 10 lines
arch/arm64/include/asm/esr.h +9
-0#define ESR_ELx_FSC_SECC_TTW (n) (0x1c + (n)) +#define ESR_ELx_FSC_ADDRSZ (0x00) + +/* +* Annoyingly, the negative levels for Address size faults aren't laid out +* contiguously (or in the desired order) +*/ +#define ESR_ELx_FSC_ADDRSZ_nL (n) ((n) == -1 ? 0x25 : 0x2C) +#define ESR_ELx_FSC_ADDRSZ_L (n) ((n) < 0 ? ESR_ELx_FSC_ADDRSZ_nL(n) : \ + (ESR_ELx_FSC_ADDRSZ + (n)))
Oh gawd, D128 and its level -2. Do we really want to add this now? I'd rather leave it to whoever already lives in th future. Specially given that the helper you introduce below doesn't make any use of it.
Otherwise looks OK.
M.
Signature · 1 line
Jazz isn't dead. It just smells funny.
Re: [PATCH v2 1/3] KVM: arm64: Only read HPFAR_EL2 when value is architecturally valid
Quoted 29 lines · Marc Zyngier
Quoted 27 lines · Oliver Upton
KVM's logic for deciding when
HPFAR_EL2is UNKNOWN doesn't align with the architecture. Most notably, KVM assumesHPFAR_EL2contains the faulting IPA even in the case of an SEA.Align the logic with the architecture rather than attempting to paraphrase it. Additionally, take the opportunity to improve the language around ARM erratum #834220 such that it actually describes the bug.
Signed-off-by Oliver Upton <[email protected]>--- arch/arm64/include/asm/esr.h | 22 ++++++++++-- arch/arm64/kvm/hyp/include/hyp/fault.h | 46 ++++++++++++++++---------- 2 files changed, 48 insertions(+), 20 deletions(-)arch/arm64/include/asm/esr.h +3
-0@@ -121,6 +121,15 @@#define ESR_ELx_FSC_SEA_TTW (n) (0x14 + (n))#define ESR_ELx_FSC_SECC (0x18)#define ESR_ELx_FSC_SECC_TTW (n) (0x1c + (n)) +#define ESR_ELx_FSC_ADDRSZ (0x00) + +/*
Quoted 8 lines
arch/arm64/include/asm/esr.h line 132
Quoted 6 lines
arch/arm64/include/asm/esr.h +6
-0+* Annoyingly, the negative levels for Address size faults aren't laid out +* contiguously (or in the desired order) +*/ +#define ESR_ELx_FSC_ADDRSZ_nL (n) ((n) == -1 ? 0x25 : 0x2C) +#define ESR_ELx_FSC_ADDRSZ_L (n) ((n) < 0 ? ESR_ELx_FSC_ADDRSZ_nL(n) : \ + (ESR_ELx_FSC_ADDRSZ + (n)))Oh gawd, D128 and its level -2. Do we really want to add this now?
I certainly don't want to, but the other FSC level macros (whether intentional or not) already cope with D128. So I did the same for the sake of consisency.
Thanks,
Oliver
Marc Zyngier
maintainerRe: [PATCH v2 1/3] KVM: arm64: Only read HPFAR_EL2 when value is architecturally valid
Quoted 40 lines · Oliver Upton
Quoted 29 lines · Marc Zyngier
Quoted 27 lines · Oliver Upton
KVM's logic for deciding when
HPFAR_EL2is UNKNOWN doesn't align with the architecture. Most notably, KVM assumesHPFAR_EL2contains the faulting IPA even in the case of an SEA.Align the logic with the architecture rather than attempting to paraphrase it. Additionally, take the opportunity to improve the language around ARM erratum #834220 such that it actually describes the bug.
Signed-off-by Oliver Upton <[email protected]>--- arch/arm64/include/asm/esr.h | 22 ++++++++++-- arch/arm64/kvm/hyp/include/hyp/fault.h | 46 ++++++++++++++++---------- 2 files changed, 48 insertions(+), 20 deletions(-)arch/arm64/include/asm/esr.h +3
-0@@ -121,6 +121,15 @@#define ESR_ELx_FSC_SEA_TTW (n) (0x14 + (n))#define ESR_ELx_FSC_SECC (0x18)#define ESR_ELx_FSC_SECC_TTW (n) (0x1c + (n)) +#define ESR_ELx_FSC_ADDRSZ (0x00) + +/* Quoted 8 lines
arch/arm64/include/asm/esr.h line 132
Quoted 6 lines
arch/arm64/include/asm/esr.h +6
-0+* Annoyingly, the negative levels for Address size faults aren't laid out +* contiguously (or in the desired order) +*/ +#define ESR_ELx_FSC_ADDRSZ_nL (n) ((n) == -1 ? 0x25 : 0x2C) +#define ESR_ELx_FSC_ADDRSZ_L (n) ((n) < 0 ? ESR_ELx_FSC_ADDRSZ_nL(n) : \ + (ESR_ELx_FSC_ADDRSZ + (n)))Oh gawd, D128 and its level -2. Do we really want to add this now?
Quoted 3 lines
I certainly don't want to, but the other FSC level macros (whether intentional or not) already cope with D128. So I did the same for the sake of consisency.
Ah, fair enough. Let's have it then.
Thanks,
M.
Signature · 1 line
Jazz isn't dead. It just smells funny.
PATCH v2 2/3 arm64: Convert HPFAR_EL2 to sysreg table
In reply to Oliver UptonSwitch over to the typical sysreg table for HPFAR_EL2 as we're about to start using more fields in the register.
--- arch/arm64/include/asm/kvm_emulate.h | 4 +++- arch/arm64/kvm/hyp/nvhe/mem_protect.c | 2 +- arch/arm64/tools/sysreg | 7 +++++++ 3 files changed, 11 insertions(+), 2 deletions(-)
arch/arm64/include/asm/kvm_emulate.h +3 -1
@@ -305,7 +305,9 @@ static __always_inline unsigned long kvm_vcpu_get_hfar(const struct kvm_vcpu *vc static __always_inline phys_addr_t kvm_vcpu_get_fault_ipa(const struct kvm_vcpu *vcpu) { -return ((phys_addr_t )vcpu->arch.fault.hpfar_el2 &HPFAR_MASK ) <<8 ; +u64 hpfar = vcpu->arch.fault.hpfar_el2; + +return FIELD_GET (HPFAR_EL2_FIPA , hpfar) <<12 ; }static inline u64 kvm_vcpu_get_disr (const struct kvm_vcpu *vcpu)
arch/arm64/kvm/hyp/nvhe/mem_protect.c +1 -1
@@ -578,7 +578,7 @@ void handle_host_mem_abort(struct kvm_cpu_context *host_ctxt)return ; } - addr = (fault.hpfar_el2 &HPFAR_MASK ) <<8 ; + addr =FIELD_GET (HPFAR_EL2_FIPA , fault.hpfar_el2) <<12 ; ret =host_stage2_idmap (addr);BUG_ON (ret && ret != -EAGAIN ); }
arch/arm64/tools/sysreg +7 -0
@@ -3433,3 +3433,10 @@ Field 5 F Field 4 P Field 3:0 Align EndSysreg + +Sysreg HPFAR_EL2 3 4 6 0 4 +Field 63 NS +Res0 62:48 +Field 47:4 FIPA +Res0 3:0 +EndSysreg
Signature · 1 line
2.39.5
PATCH v2 3/3 KVM: arm64: Don't translate FAR if invalid/unsafe
In reply to Oliver UptonDon't re-walk the page tables if an SEA occurred during the faulting page table walk to avoid taking a fatal exception in the hyp. Additionally, check that FAR_EL2 is valid for SEAs not taken on PTW as the architecture doesn't guarantee it contains the fault VA.
Finally, fix up the rest of the abort path by checking for SEAs early and bugging the VM if we get further along with an UNKNOWN fault IPA.
--- arch/arm64/include/asm/esr.h | 22 ++++++++++++++++++ arch/arm64/include/asm/kvm_emulate.h | 3 +++ arch/arm64/include/asm/kvm_ras.h | 2 +- arch/arm64/kvm/hyp/include/hyp/fault.h | 26 ++++++++++++++++----- arch/arm64/kvm/hyp/nvhe/mem_protect.c | 7 ++++++ arch/arm64/kvm/mmu.c | 31 ++++++++++++++++---------- 6 files changed, 73 insertions(+), 18 deletions(-)
arch/arm64/include/asm/esr.h +22 -0
@@ -482,6 +482,28 @@ static inline bool esr_fsc_is_addr_sz_fault(unsigned long esr) (esr ==ESR_ELx_FSC_ADDRSZ_L (-1 )); } +static inline bool esr_fsc_is_sea_ttw (unsigned long esr) +{ + esr = esr & ESR_ELx_FSC; + +return (esr ==ESR_ELx_FSC_SEA_TTW (3 )) || + (esr ==ESR_ELx_FSC_SEA_TTW (2 )) || + (esr ==ESR_ELx_FSC_SEA_TTW (1 )) || + (esr ==ESR_ELx_FSC_SEA_TTW (0 )) || + (esr ==ESR_ELx_FSC_SEA_TTW (-1 )); +} + +static inline bool esr_fsc_is_secc_ttw (unsigned long esr) +{ + esr = esr & ESR_ELx_FSC; + +return (esr ==ESR_ELx_FSC_SECC_TTW (3 )) || + (esr ==ESR_ELx_FSC_SECC_TTW (2 )) || + (esr ==ESR_ELx_FSC_SECC_TTW (1 )) || + (esr ==ESR_ELx_FSC_SECC_TTW (0 )) || + (esr ==ESR_ELx_FSC_SECC_TTW (-1 )); +} +/* Indicate whether ESR.EC==0x1A is for an ERETAx instruction */ static inline bool esr_iss_is_eretax (unsigned long esr) {
arch/arm64/include/asm/kvm_emulate.h +3 -0
@@ -307,6 +307,9 @@ static __always_inline phys_addr_t kvm_vcpu_get_fault_ipa(const struct kvm_vcpu {u64 hpfar = vcpu->arch.fault.hpfar_el2; +if (unlikely (!(hpfar &HPFAR_EL2_NS ))) +return INVALID_GPA ; +return FIELD_GET (HPFAR_EL2_FIPA , hpfar) <<12 ; }
arch/arm64/include/asm/kvm_ras.h +1 -1
@@ -14,7 +14,7 @@* Was this synchronous external abort a RAS notification? * Returns '0' for errors handled by some RAS subsystem, or -ENOENT. */ -static inline int kvm_handle_guest_sea (phys_addr_t addr,u64 esr) +static inline int kvm_handle_guest_sea (void ) {/* apei_claim_sea(NULL) expects to mask interrupts itself */ lockdep_assert_irqs_enabled ();
arch/arm64/kvm/hyp/include/hyp/fault.h +21 -5
@@ -12,6 +12,16 @@#include <asm/kvm_hyp.h> #include <asm/kvm_mmu.h> +static inline bool __fault_safe_to_translate (u64 esr) +{ +u64 fsc = esr & ESR_ELx_FSC; + +if (esr_fsc_is_sea_ttw (esr) ||esr_fsc_is_secc_ttw (esr)) +return false; + +return !(fsc == ESR_ELx_FSC_EXTABT && (esr & ESR_ELx_FnV)); +} +static inline bool __translate_far_to_hpfar (u64 far,u64 *hpfar) {int ret;
@@ -71,17 +81,23 @@ static inline bool __hpfar_valid(u64 esr)static inline bool __get_fault_info (u64 esr,struct kvm_vcpu_fault_info *fault) { -u64 hpfar, far; +u64 hpfar; - far =read_sysreg_el2 (SYS_FAR ); + fault->far_el2 =read_sysreg_el2 (SYS_FAR ); + fault->hpfar_el2 =0 ;if (__hpfar_valid (esr)) hpfar =read_sysreg (hpfar_el2); -else if (!__translate_far_to_hpfar (far, &hpfar)) +else if (unlikely (!__fault_safe_to_translate (esr))) +return true; +else if (!__translate_far_to_hpfar (fault->far_el2, &hpfar))return false; - fault->far_el2 = far; - fault->hpfar_el2 = hpfar; +/* +* Hijack HPFAR_EL2.NS (RES0 in Non-secure) to indicate a valid +* HPFAR value. +*/ + fault->hpfar_el2 = hpfar |HPFAR_EL2_NS ;return true; }
arch/arm64/kvm/hyp/nvhe/mem_protect.c +7 -0
@@ -578,7 +578,14 @@ void handle_host_mem_abort(struct kvm_cpu_context *host_ctxt)return ; } + +/* +* Yikes, we couldn't resolve the fault IPA. This should reinject an +* abort into the host when we figure out how to do that. +*/ +BUG_ON (!(fault.hpfar_el2 &HPFAR_EL2_NS )); addr =FIELD_GET (HPFAR_EL2_FIPA , fault.hpfar_el2) <<12 ; + ret =host_stage2_idmap (addr);BUG_ON (ret && ret != -EAGAIN ); }
arch/arm64/kvm/mmu.c +19 -12
@@ -1794,9 +1794,28 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu)gfn_t gfn;int ret, idx; +/* Synchronous External Abort? */ +if (kvm_vcpu_abt_issea (vcpu)) { +/* +* For RAS the host kernel may handle this abort. +* There is no need to pass the error into the guest. +*/ +if (kvm_handle_guest_sea ()) +kvm_inject_vabt (vcpu); + +return 1 ; + } + esr =kvm_vcpu_get_esr (vcpu); +/* +* The fault IPA should be reliable at this point as we're not dealing +* with an SEA. +*/ ipa = fault_ipa =kvm_vcpu_get_fault_ipa (vcpu); +if (KVM_BUG_ON (ipa ==INVALID_GPA , vcpu->kvm)) +return -EFAULT ; + is_iabt =kvm_vcpu_trap_is_iabt (vcpu);if (esr_fsc_is_translation_fault (esr)) {
@@ -1818,18 +1837,6 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu) } } -/* Synchronous External Abort? */ -if (kvm_vcpu_abt_issea (vcpu)) { -/* -* For RAS the host kernel may handle this abort. -* There is no need to pass the error into the guest. -*/ -if (kvm_handle_guest_sea (fault_ipa,kvm_vcpu_get_esr (vcpu))) -kvm_inject_vabt (vcpu); - -return 1 ; - } -trace_kvm_guest_fault (*vcpu_pc (vcpu),kvm_vcpu_get_esr (vcpu),kvm_vcpu_get_hfar (vcpu), fault_ipa);
Signature · 1 line
2.39.5
Quoted 31 lines · Oliver Upton
KVM's heuristics for determining the fault IPA are a bit shaky and don't necessarily align with the letter of the architecture. On top of that,
HPFAR_EL2is UNKNOWN if an SEA occurred during a table walk. Re-walking the page tables will replay the SEA at EL2, becoming a panic in the hyp.v1: https://lore.kernel.org/kvmarm/[email protected]/
v1 -> v2 (addressed Marc's feedback):
- Drop section reference in favor of the rule forHPFAR_EL2validity
- Fold test for Address size fault into helper, reuse existing definitions
- Get rid of 'magic' shift for transforming FIPA field to an actual IPA
- AddBUG_ON()tohandle_host_mem_abort()if we fail to resolve the fault IPA.Oliver Upton (3):
KVM: arm64: Only readHPFAR_EL2when value is architecturally valid
arm64: ConvertHPFAR_EL2to sysreg table
KVM: arm64: Don't translate FAR if invalid/unsafearch/arm64/include/asm/esr.h | 44 +++++++++++++++- arch/arm64/include/asm/kvm_emulate.h | 7 ++- arch/arm64/include/asm/kvm_ras.h | 2 +- arch/arm64/kvm/hyp/include/hyp/fault.h | 70 ++++++++++++++++++-------- arch/arm64/kvm/hyp/nvhe/mem_protect.c | 9 +++- arch/arm64/kvm/mmu.c | 31 +++++++----- arch/arm64/tools/sysreg | 7 +++ 7 files changed, 131 insertions(+), 39 deletions(-)
Quoted 1 line
base-commit:
369c0122682c4468a69f2454614eded71c5348f3
My comment on patch #1 notwithstanding, this looks pretty good.
M.
Signature · 1 line
Jazz isn't dead. It just smells funny.
Oliver Upton
AppliedQuoted 8 lines · Oliver Upton
KVM's heuristics for determining the fault IPA are a bit shaky and don't necessarily align with the letter of the architecture. On top of that,
HPFAR_EL2is UNKNOWN if an SEA occurred during a table walk. Re-walking the page tables will replay the SEA at EL2, becoming a panic in the hyp.v1: https://lore.kernel.org/kvmarm/[email protected]/
[...]
Applied to fixes, thanks!
[1/3] KVM: arm64: Only read HPFAR_EL2 when value is architecturally valid
https://git.kernel.org/kvmarm/kvmarm/c/fb8a3eba9c81 [2/3] arm64: Convert HPFAR_EL2 to sysreg table https://git.kernel.org/kvmarm/kvmarm/c/1cf3e126f152 [3/3] KVM: arm64: Don't translate FAR if invalid/unsafe https://git.kernel.org/kvmarm/kvmarm/c/26fbdf369227
--
Best,
Oliver