KVM: arm64: Work around Ampere1 erratum AC03_CPU_38
Subsystems (4)
cover KVM: arm64: Work around Ampere1 erratum AC03_CPU_38
Hi folks,
Small series to work around a CPU erratum on AmpereOne. While the implementation does not advertise support for FEAT_HAFDBS (due to another erratum), the associated control bits do not have RES0 behavior as required by the architecture.
Usage of HAFDBS at stage-1 is unaffected, since HA and HD are only enabled on implementations that advertise the feature. However, KVM relies on HA having RES0 semantics if the feature isn't implemented. The end result is that KVM enables a broken hardware access flag implementation that could lead to correctness issues.
Applies to 6.4-rc1. Tested with access_tracking_perf_test, verifying that KVM is indeed taking Access Flag faults.
Oliver Upton (3):
arm64: errata: Mitigate Ampere1 erratum AC03_CPU_38 at stage-2
KVM: arm64: Refactor HFGxTR configuration into separate helpers
KVM: arm64: Prevent guests from enabling HA/HD on Ampere1
Documentation/arm64/silicon-errata.rst | 3 + arch/arm64/Kconfig | 17 +++++ arch/arm64/kernel/cpu_errata.c | 7 ++ arch/arm64/kvm/hyp/include/hyp/switch.h | 99 ++++++++++++++++++++----- arch/arm64/kvm/hyp/pgtable.c | 14 +++- arch/arm64/tools/cpucaps | 1 + 6 files changed, 120 insertions(+), 21 deletions(-)
base-commit: ac9a78681b921877518763ba0e89202254349d1b
Signature · 1 line
2.41.0.162.gfafddb0af9-goog
PATCH 1/3 arm64: errata: Mitigate Ampere1 erratum AC03_CPU_38 at stage-2
AmpereOne has an erratum in its implementation of FEAT_HAFDBS that required disabling the feature on the design. This was done by reporting the feature as not implemented in the ID register, although the corresponding control bits were not actually RES0. This does not align well with the requirements of the architecture, which mandates these bits be RES0 if HAFDBS isn't implemented.
The kernel's use of stage-1 is unaffected, as the HA and HD bits are only set if HAFDBS is detected in the ID register. KVM, on the other hand, relies on the RES0 behavior at stage-2 to use the same value for VTCR_EL2 on any cpu in the system. Mitigate the non-RES0 behavior by leaving VTCR_EL2.HA clear on affected systems.
--- Documentation/arm64/silicon-errata.rst | 3 +++ arch/arm64/Kconfig | 17 +++++++++++++++++ arch/arm64/kernel/cpu_errata.c | 7 +++++++ arch/arm64/kvm/hyp/pgtable.c | 14 +++++++++++--- arch/arm64/tools/cpucaps | 1 + 5 files changed, 39 insertions(+), 3 deletions(-)
Documentation/arm64/silicon-errata.rst +3 -0
@@ -52,6 +52,9 @@ stable kernels. | Allwinner | A64/R18 | UNKNOWN1 | SUN50I_ERRATUM_UNKNOWN1 | +----------------+-----------------+-----------------+-----------------------------+ +----------------+-----------------+-----------------+-----------------------------+ +| Ampere | AmpereOne | AC03_CPU_38 | AMPERE_ERRATUM_AC03_CPU_38 | ++----------------+-----------------+-----------------+-----------------------------+ ++----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-A510 | #2457168 | ARM64_ERRATUM_2457168 | +----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-A510 | #2064142 | ARM64_ERRATUM_2064142 |
arch/arm64/Kconfig +17 -0
@@ -406,6 +406,23 @@ menu "Kernel Features" menu "ARM errata workarounds via the alternatives framework" +config AMPERE_ERRATUM_AC03_CPU_38 + bool "AmpereOne: AC03_CPU_38: Certain bits in the Virtualization Translation Control Register and Translation Control Registers do not follow RES0 semantics" + default y + help + This option adds an alternative code sequence to work around Ampere + erratum AC03_CPU_38 on AmpereOne. + + The affected design reports FEAT_HAFDBS as not implemented in + ID_AA64MMFR1_EL1.HAFDBS, but (V)TCR_ELx.{HA,HD} are not RES0 + as required by the architecture. + + The workaround forces KVM to explicitly set VTCR_EL2.HA to 0, + which avoids enabling unadvertised hardware Access Flag management + at stage-2. + + If unsure, say Y. + config ARM64_WORKAROUND_CLEAN_CACHE bool
arch/arm64/kernel/cpu_errata.c +7 -0
@@ -729,6 +729,13 @@ const struct arm64_cpu_capabilities arm64_errata[] = { MIDR_FIXED(MIDR_CPU_VAR_REV(1,1), BIT(25)), .cpu_enable = cpu_clear_bf16_from_user_emulation, }, +#endif +#ifdef CONFIG_AMPERE_ERRATUM_AC03_CPU_38 + { + .desc = "AmpereOne erratum AC03_CPU_38", + .capability = ARM64_WORKAROUND_AMPERE_AC03_CPU_38, + ERRATA_MIDR_ALL_VERSIONS(MIDR_AMPERE1), + }, #endif { }
arch/arm64/kvm/hyp/pgtable.c +11 -3
@@ -609,10 +609,18 @@ u64 kvm_get_vtcr(u64 mmfr0, u64 mmfr1, u32 phys_shift)#ifdef CONFIG_ARM64_HW_AFDBM /* * Enable the Hardware Access Flag management, unconditionally -* on all CPUs. The features is RES0 on CPUs without the support -* and must be ignored by the CPUs. +* on all CPUs. In systems that have asymmetric support for the feature +* this allows KVM to leverage hardware support on the subset of cores +* that implement the feature. +* +* The architecture requires VTCR_EL2.HA to be RES0 (thus ignored by +* hardware) on implementations that do not advertise support for the +* feature. As such, setting HA unconditionally is safe, unless you +* happen to be running on a design that has unadvertised support for +* HAFDBS. Here be dragons. */ - vtcr |=VTCR_EL2_HA ; +if (!cpus_have_final_cap (ARM64_WORKAROUND_AMPERE_AC03_CPU_38 )) + vtcr |=VTCR_EL2_HA ;#endif /* CONFIG_ARM64_HW_AFDBM */ /* Set the vmid bits */
arch/arm64/tools/cpucaps +1 -0
@@ -77,6 +77,7 @@ WORKAROUND_2077057 WORKAROUND_2457168 WORKAROUND_2645198 WORKAROUND_2658417 +WORKAROUND_AMPERE_AC03_CPU_38 WORKAROUND_TRBE_OVERWRITE_FILL_MODE WORKAROUND_TSB_FLUSH_FAILURE WORKAROUND_TRBE_WRITE_OUT_OF_RANGE
Signature · 1 line
2.41.0.162.gfafddb0af9-goog
Re: [PATCH 1/3] arm64: errata: Mitigate Ampere1 erratum AC03_CPU_38 at stage-2
Quoted 13 lines · Oliver Upton
AmpereOne has an erratum in its implementation of
FEAT_HAFDBSthat required disabling the feature on the design. This was done by reporting the feature as not implemented in the ID register, although the corresponding control bits were not actually RES0. This does not align well with the requirements of the architecture, which mandates these bits be RES0 if HAFDBS isn't implemented.The kernel's use of stage-1 is unaffected, as the HA and HD bits are only set if HAFDBS is detected in the ID register. KVM, on the other hand, relies on the RES0 behavior at stage-2 to use the same value for
VTCR_EL2on any cpu in the system. Mitigate the non-RES0 behavior by leaving VTCR_EL2.HA clear on affected systems.
Quoted 4 lines
Cc D Scott Phillips <[email protected]>Cc Darren Hart <[email protected]>Signed-off-by Oliver Upton <[email protected]>
For the non-KVM bits in here:
Re: [PATCH 1/3] arm64: errata: Mitigate Ampere1 erratum AC03_CPU_38 at stage-2
In reply to Oliver UptonQuoted 13 lines · Oliver Upton
AmpereOne has an erratum in its implementation of
FEAT_HAFDBSthat required disabling the feature on the design. This was done by reporting the feature as not implemented in the ID register, although the corresponding control bits were not actually RES0. This does not align well with the requirements of the architecture, which mandates these bits be RES0 if HAFDBS isn't implemented.The kernel's use of stage-1 is unaffected, as the HA and HD bits are only set if HAFDBS is detected in the ID register. KVM, on the other hand, relies on the RES0 behavior at stage-2 to use the same value for
VTCR_EL2on any cpu in the system. Mitigate the non-RES0 behavior by leaving VTCR_EL2.HA clear on affected systems.
Quoted 4 lines
Cc D Scott Phillips <[email protected]>Cc Darren Hart <[email protected]>Signed-off-by Oliver Upton <[email protected]>
PATCH 2/3 KVM: arm64: Refactor HFGxTR configuration into separate helpers
In reply to Oliver UptonA subsequent change will need to flip more trap bits in HFGWTR_EL2. Make room for this by factoring out the programming of the HFGxTR registers into helpers and using locals to build the set/clear masks.
--- arch/arm64/kvm/hyp/include/hyp/switch.h | 60 +++++++++++++++++-------- 1 file changed, 42 insertions(+), 18 deletions(-)
arch/arm64/kvm/hyp/include/hyp/switch.h +42 -18
@@ -70,6 +70,44 @@ static inline void __activate_traps_fpsimd32(struct kvm_vcpu *vcpu) } } +static inline bool __hfgxtr_traps_required (void ) +{ +if (cpus_have_final_cap (ARM64_SME )) +return true; + +return false; +} + +static inline void __activate_traps_hfgxtr (void ) +{ +u64 r_clr =0 , w_clr =0 , r_set =0 , w_set =0 , tmp; + +if (cpus_have_final_cap (ARM64_SME )) { + tmp = HFGxTR_EL2_nSMPRI_EL1_MASK | HFGxTR_EL2_nTPIDR2_EL0_MASK; + + r_clr |= tmp; + w_clr |= tmp; + } + +sysreg_clear_set_s (SYS_HFGRTR_EL2 , r_clr, r_set); +sysreg_clear_set_s (SYS_HFGWTR_EL2 , w_clr, w_set); +} + +static inline void __deactivate_traps_hfgxtr (void ) +{ +u64 r_clr =0 , w_clr =0 , r_set =0 , w_set =0 , tmp; + +if (cpus_have_final_cap (ARM64_SME )) { + tmp = HFGxTR_EL2_nSMPRI_EL1_MASK | HFGxTR_EL2_nTPIDR2_EL0_MASK; + + r_set |= tmp; + w_set |= tmp; + } + +sysreg_clear_set_s (SYS_HFGRTR_EL2 , r_clr, r_set); +sysreg_clear_set_s (SYS_HFGWTR_EL2 , w_clr, w_set); +} +static inline void __activate_traps_common (struct kvm_vcpu *vcpu) {/* Trap on AArch32 cp15 c15 (impdef sysregs) accesses (EL1 or EL0) */
@@ -89,16 +127,8 @@ static inline void __activate_traps_common(struct kvm_vcpu *vcpu) vcpu->arch.mdcr_el2_host =read_sysreg (mdcr_el2);write_sysreg (vcpu->arch.mdcr_el2, mdcr_el2); -if (cpus_have_final_cap (ARM64_SME )) { -sysreg_clear_set_s (SYS_HFGRTR_EL2 , - HFGxTR_EL2_nSMPRI_EL1_MASK | - HFGxTR_EL2_nTPIDR2_EL0_MASK, -0 ); -sysreg_clear_set_s (SYS_HFGWTR_EL2 , - HFGxTR_EL2_nSMPRI_EL1_MASK | - HFGxTR_EL2_nTPIDR2_EL0_MASK, -0 ); - } +if (__hfgxtr_traps_required ()) +__activate_traps_hfgxtr (); }static inline void __deactivate_traps_common (struct kvm_vcpu *vcpu)
@@ -109,14 +139,8 @@ static inline void __deactivate_traps_common(struct kvm_vcpu *vcpu)if (kvm_arm_support_pmu_v3 ())write_sysreg (0 , pmuserenr_el0); -if (cpus_have_final_cap (ARM64_SME )) { -sysreg_clear_set_s (SYS_HFGRTR_EL2 ,0 , - HFGxTR_EL2_nSMPRI_EL1_MASK | - HFGxTR_EL2_nTPIDR2_EL0_MASK); -sysreg_clear_set_s (SYS_HFGWTR_EL2 ,0 , - HFGxTR_EL2_nSMPRI_EL1_MASK | - HFGxTR_EL2_nTPIDR2_EL0_MASK); - } +if (__hfgxtr_traps_required ()) +__deactivate_traps_hfgxtr (); }static inline void ___activate_traps (struct kvm_vcpu *vcpu)
Signature · 1 line
2.41.0.162.gfafddb0af9-goog
PATCH 3/3 KVM: arm64: Prevent guests from enabling HA/HD on Ampere1
In reply to Oliver UptonAn erratum in the HAFDBS implementation in AmpereOne was addressed by clearing the feature in the ID register, with the expectation that software would not attempt to use the corresponding controls in TCR_EL1. The architecture, on the other hand, takes a much more pedantic stance on the subject, requiring the TCR bits behave as RES0.
Take an extremely conservative stance on the issue and leverage the precise write trap afforded by FGT. Handle guest writes by clearing HA and HD before writing the intended value to the EL1 register alias.
--- arch/arm64/kvm/hyp/include/hyp/switch.h | 39 +++++++++++++++++++++++++ 1 file changed, 39 insertions(+)
arch/arm64/kvm/hyp/include/hyp/switch.h +39 -0
@@ -75,6 +75,9 @@ static inline bool __hfgxtr_traps_required(void)if (cpus_have_final_cap (ARM64_SME ))return true; +if (cpus_have_final_cap (ARM64_WORKAROUND_AMPERE_AC03_CPU_38 )) +return true; +return false; }
@@ -89,6 +92,12 @@ static inline void __activate_traps_hfgxtr(void) w_clr |= tmp; } +/* +* Trap guest writes to TCR_EL1 to prevent it from enabling HA or HD. +*/ +if (cpus_have_final_cap (ARM64_WORKAROUND_AMPERE_AC03_CPU_38 )) + w_set |= HFGxTR_EL2_TCR_EL1_MASK; +sysreg_clear_set_s (SYS_HFGRTR_EL2 , r_clr, r_set);sysreg_clear_set_s (SYS_HFGWTR_EL2 , w_clr, w_set); }
@@ -104,6 +113,9 @@ static inline void __deactivate_traps_hfgxtr(void) w_set |= tmp; } +if (cpus_have_final_cap (ARM64_WORKAROUND_AMPERE_AC03_CPU_38 )) + w_clr |= HFGxTR_EL2_TCR_EL1_MASK; +sysreg_clear_set_s (SYS_HFGRTR_EL2 , r_clr, r_set);sysreg_clear_set_s (SYS_HFGWTR_EL2 , w_clr, w_set); }
@@ -400,12 +412,39 @@ static bool kvm_hyp_handle_cntpct(struct kvm_vcpu *vcpu)return true; } +static bool handle_ampere1_tcr (struct kvm_vcpu *vcpu) +{ +u32 sysreg =esr_sys64_to_sysreg (kvm_vcpu_get_esr (vcpu)); +int rt =kvm_vcpu_sys_get_rt (vcpu); +u64 val =vcpu_get_reg (vcpu, rt); + +if (sysreg !=SYS_TCR_EL1 ) +return false; + +/* +* Affected parts do not advertise support for hardware Access Flag / +* Dirty state management in ID_AA64MMFR1_EL1.HAFDBS, but the underlying +* control bits are still functional. The architecture requires these be +* RES0 on systems that do not implement FEAT_HAFDBS. +* +* Uphold the requirements of the architecture by masking guest writes +* to TCR_EL1.{HA,HD} here. +*/ + val &= ~(TCR_HD |TCR_HA ); +write_sysreg_el1 (val,SYS_TCR ); +return true; +} +static bool kvm_hyp_handle_sysreg (struct kvm_vcpu *vcpu,u64 *exit_code) {if (cpus_have_final_cap (ARM64_WORKAROUND_CAVIUM_TX2_219_TVM ) &&handle_tx2_tvm (vcpu))return true; +if (cpus_have_final_cap (ARM64_WORKAROUND_AMPERE_AC03_CPU_38 ) && +handle_ampere1_tcr (vcpu)) +return true; +if (static_branch_unlikely (&vgic_v3_cpuif_trap) &&__vgic_v3_perform_cpuif_access (vcpu) ==1 )return true;
Signature · 1 line
2.41.0.162.gfafddb0af9-goog
Catalin Marinas
maintainerQuoted 10 lines · Oliver Upton
Small series to work around a CPU erratum on AmpereOne. While the implementation does not advertise support for
FEAT_HAFDBS(due to another erratum), the associated control bits do not have RES0 behavior as required by the architecture.Usage of HAFDBS at stage-1 is unaffected, since HA and HD are only enabled on implementations that advertise the feature. However, KVM relies on HA having RES0 semantics if the feature isn't implemented. The end result is that KVM enables a broken hardware access flag implementation that could lead to correctness issues.
Just curious, what's the correctness issue here? The access flag is mostly indicative of which pages are old for swapping out/discarding. It's not like the dirty state which would be dangerous if we get wrong.
Signature · 1 line
Catalin
Hey Catalin,
Quoted 12 lines · Catalin Marinas
Quoted 10 lines · Oliver Upton
Small series to work around a CPU erratum on AmpereOne. While the implementation does not advertise support for
FEAT_HAFDBS(due to another erratum), the associated control bits do not have RES0 behavior as required by the architecture.Usage of HAFDBS at stage-1 is unaffected, since HA and HD are only enabled on implementations that advertise the feature. However, KVM relies on HA having RES0 semantics if the feature isn't implemented. The end result is that KVM enables a broken hardware access flag implementation that could lead to correctness issues.
Quoted 3 lines
Just curious, what's the correctness issue here? The access flag is mostly indicative of which pages are old for swapping out/discarding. It's not like the dirty state which would be dangerous if we get wrong.
I probably could have helped out by giving the full context.
The software-observable behavior on this system is that the A or D updates could arrive after a PTE has been marked as invalid, which could corrupt software metadata stuffed into the page tables. We do exactly that at stage-2 in KVM for parallel fault handling, where a magic value indicates a PTE is being updated by another thread.
Signature · 2 lines
Thanks, Oliver
Catalin Marinas
maintainerQuoted 19 lines · Oliver Upton
Hey Catalin,
Quoted 12 lines · Catalin Marinas
Quoted 10 lines · Oliver Upton
Small series to work around a CPU erratum on AmpereOne. While the implementation does not advertise support for
FEAT_HAFDBS(due to another erratum), the associated control bits do not have RES0 behavior as required by the architecture.Usage of HAFDBS at stage-1 is unaffected, since HA and HD are only enabled on implementations that advertise the feature. However, KVM relies on HA having RES0 semantics if the feature isn't implemented. The end result is that KVM enables a broken hardware access flag implementation that could lead to correctness issues.
Quoted 3 lines
Just curious, what's the correctness issue here? The access flag is mostly indicative of which pages are old for swapping out/discarding. It's not like the dirty state which would be dangerous if we get wrong.
Quoted 7 lines
I probably could have helped out by giving the full context.
The software-observable behavior on this system is that the A or D updates could arrive after a PTE has been marked as invalid, which could corrupt software metadata stuffed into the page tables. We do exactly that at stage-2 in KVM for parallel fault handling, where a magic value indicates a PTE is being updated by another thread.
Ah, ok, that's dangerous indeed. Thanks for the details (you may want to add them in the patch description or the erratum kconfig entry).
Signature · 1 line
Catalin
Quoted 8 lines · Oliver Upton
Hi folks,
Small series to work around a CPU erratum on AmpereOne. While the implementation does not advertise support for
FEAT_HAFDBS(due to another erratum), the associated control bits do not have RES0 behavior as required by the architecture.
Quoted 8 lines
Usage of HAFDBS at stage-1 is unaffected, since HA and HD are only enabled on implementations that advertise the feature. However, KVM relies on HA having RES0 semantics if the feature isn't implemented. The end result is that KVM enables a broken hardware access flag implementation that could lead to correctness issues.
Applies to 6.4-rc1. Tested with
access_tracking_perf_test, verifying that KVM is indeed taking Access Flag faults.
For the series:
M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
Quoted 5 lines · Oliver Upton
Small series to work around a CPU erratum on AmpereOne. While the implementation does not advertise support for
FEAT_HAFDBS(due to another erratum), the associated control bits do not have RES0 behavior as required by the architecture.
Quoted 7 lines
Usage of HAFDBS at stage-1 is unaffected, since HA and HD are only enabled on implementations that advertise the feature. However, KVM relies on HA having RES0 semantics if the feature isn't implemented. The end result is that KVM enables a broken hardware access flag implementation that could lead to correctness issues.
[...]
Applied w/ an expanded description of what's wrong with the unadvertised HAFDBS implementation, per Catalin's suggestion.
[1/3] arm64: errata: Mitigate Ampere1 erratum AC03_CPU_38 at stage-2
https://git.kernel.org/kvmarm/kvmarm/c/6df696cd9bc1 [2/3] KVM: arm64: Refactor HFGxTR configuration into separate helpers https://git.kernel.org/kvmarm/kvmarm/c/ce4a36225753 [3/3] KVM: arm64: Prevent guests from enabling HA/HD on Ampere1 https://git.kernel.org/kvmarm/kvmarm/c/082fdfd13841
--
Best,
Oliver