Skip to contentSkip to conversation

KVM: arm64: Work around Ampere1 erratum AC03_CPU_38

PATCH In review Oliver Upton 4 participants 11 messages Jun 9, 2023 – Jun 20, 2023

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

cover KVM: arm64: Work around Ampere1 erratum AC03_CPU_38

Hi folks,

Small series to work around a CPU erratum on AmpereOne. While the implementation does not advertise support for FEAT_HAFDBS (due to another erratum), the associated control bits do not have RES0 behavior as required by the architecture.

Usage of HAFDBS at stage-1 is unaffected, since HA and HD are only enabled on implementations that advertise the feature. However, KVM relies on HA having RES0 semantics if the feature isn't implemented. The end result is that KVM enables a broken hardware access flag implementation that could lead to correctness issues.

Applies to 6.4-rc1. Tested with access_tracking_perf_test, verifying that KVM is indeed taking Access Flag faults.

Oliver Upton (3):
arm64: errata: Mitigate Ampere1 erratum AC03_CPU_38 at stage-2
KVM: arm64: Refactor HFGxTR configuration into separate helpers
KVM: arm64: Prevent guests from enabling HA/HD on Ampere1

 Documentation/arm64/silicon-errata.rst  |  3 +
 arch/arm64/Kconfig                      | 17 +++++
 arch/arm64/kernel/cpu_errata.c          |  7 ++
 arch/arm64/kvm/hyp/include/hyp/switch.h | 99 ++++++++++++++++++++-----
 arch/arm64/kvm/hyp/pgtable.c            | 14 +++-
 arch/arm64/tools/cpucaps                |  1 +
 6 files changed, 120 insertions(+), 21 deletions(-)

base-commit: ac9a78681b921877518763ba0e89202254349d1b

Signature · 1 line
2.41.0.162.gfafddb0af9-goog
#1 lore raw

PATCH 1/3 arm64: errata: Mitigate Ampere1 erratum AC03_CPU_38 at stage-2

AmpereOne has an erratum in its implementation of FEAT_HAFDBS that required disabling the feature on the design. This was done by reporting the feature as not implemented in the ID register, although the corresponding control bits were not actually RES0. This does not align well with the requirements of the architecture, which mandates these bits be RES0 if HAFDBS isn't implemented.

The kernel's use of stage-1 is unaffected, as the HA and HD bits are only set if HAFDBS is detected in the ID register. KVM, on the other hand, relies on the RES0 behavior at stage-2 to use the same value for VTCR_EL2 on any cpu in the system. Mitigate the non-RES0 behavior by leaving VTCR_EL2.HA clear on affected systems.

---
 Documentation/arm64/silicon-errata.rst |  3 +++
 arch/arm64/Kconfig                     | 17 +++++++++++++++++
 arch/arm64/kernel/cpu_errata.c         |  7 +++++++
 arch/arm64/kvm/hyp/pgtable.c           | 14 +++++++++++---
 arch/arm64/tools/cpucaps               |  1 +
 5 files changed, 39 insertions(+), 3 deletions(-)
Documentation/arm64/silicon-errata.rst +3 -0
index 9e311bc43e05..cd46e2b20a81 100644
@@ -52,6 +52,9 @@ stable kernels.
 | Allwinner      | A64/R18         | UNKNOWN1        | SUN50I_ERRATUM_UNKNOWN1     |
 +----------------+-----------------+-----------------+-----------------------------+
 +----------------+-----------------+-----------------+-----------------------------+
+| Ampere         | AmpereOne       | AC03_CPU_38     | AMPERE_ERRATUM_AC03_CPU_38  |
++----------------+-----------------+-----------------+-----------------------------+
++----------------+-----------------+-----------------+-----------------------------+
 | ARM            | Cortex-A510     | #2457168        | ARM64_ERRATUM_2457168       |
 +----------------+-----------------+-----------------+-----------------------------+
 | ARM            | Cortex-A510     | #2064142        | ARM64_ERRATUM_2064142       |
arch/arm64/Kconfig +17 -0
index b1201d25a8a4..f853af10142b 100644
@@ -406,6 +406,23 @@ menu "Kernel Features"
 
 menu "ARM errata workarounds via the alternatives framework"
 
+config AMPERE_ERRATUM_AC03_CPU_38
+        bool "AmpereOne: AC03_CPU_38: Certain bits in the Virtualization Translation Control Register and Translation Control Registers do not follow RES0 semantics"
+	default y
+	help
+	  This option adds an alternative code sequence to work around Ampere
+	  erratum AC03_CPU_38 on AmpereOne.
+
+	  The affected design reports FEAT_HAFDBS as not implemented in
+	  ID_AA64MMFR1_EL1.HAFDBS, but (V)TCR_ELx.{HA,HD} are not RES0
+	  as required by the architecture.
+
+	  The workaround forces KVM to explicitly set VTCR_EL2.HA to 0,
+	  which avoids enabling unadvertised hardware Access Flag management
+	  at stage-2.
+
+	  If unsure, say Y.
+
 config ARM64_WORKAROUND_CLEAN_CACHE
 	bool
 
arch/arm64/kernel/cpu_errata.c +7 -0
index 307faa2b4395..be66e94a21bd 100644
@@ -729,6 +729,13 @@ const struct arm64_cpu_capabilities arm64_errata[] = {
 		MIDR_FIXED(MIDR_CPU_VAR_REV(1,1), BIT(25)),
 		.cpu_enable = cpu_clear_bf16_from_user_emulation,
 	},
+#endif
+#ifdef CONFIG_AMPERE_ERRATUM_AC03_CPU_38
+	{
+		.desc = "AmpereOne erratum AC03_CPU_38",
+		.capability = ARM64_WORKAROUND_AMPERE_AC03_CPU_38,
+		ERRATA_MIDR_ALL_VERSIONS(MIDR_AMPERE1),
+	},
 #endif
 	{
 	}
arch/arm64/kvm/hyp/pgtable.c +11 -3
index 3d61bd3e591d..9b5c8e6c08a0 100644
@@ -609,10 +609,18 @@ u64 kvm_get_vtcr(u64 mmfr0, u64 mmfr1, u32 phys_shift)
 #ifdef CONFIG_ARM64_HW_AFDBM
 	/*
 	 * Enable the Hardware Access Flag management, unconditionally
-	 * on all CPUs. The features is RES0 on CPUs without the support
-	 * and must be ignored by the CPUs.
+	 * on all CPUs. In systems that have asymmetric support for the feature
+	 * this allows KVM to leverage hardware support on the subset of cores
+	 * that implement the feature.
+	 *
+	 * The architecture requires VTCR_EL2.HA to be RES0 (thus ignored by
+	 * hardware) on implementations that do not advertise support for the
+	 * feature. As such, setting HA unconditionally is safe, unless you
+	 * happen to be running on a design that has unadvertised support for
+	 * HAFDBS. Here be dragons.
 	 */
-	vtcr |= VTCR_EL2_HA;
+	if (!cpus_have_final_cap(ARM64_WORKAROUND_AMPERE_AC03_CPU_38))
+		vtcr |= VTCR_EL2_HA;
 #endif /* CONFIG_ARM64_HW_AFDBM */
 
 	/* Set the vmid bits */
arch/arm64/tools/cpucaps +1 -0
index 40ba95472594..9f9a2d6652eb 100644
@@ -77,6 +77,7 @@ WORKAROUND_2077057
 WORKAROUND_2457168
 WORKAROUND_2645198
 WORKAROUND_2658417
+WORKAROUND_AMPERE_AC03_CPU_38
 WORKAROUND_TRBE_OVERWRITE_FILL_MODE
 WORKAROUND_TSB_FLUSH_FAILURE
 WORKAROUND_TRBE_WRITE_OUT_OF_RANGE
Signature · 1 line
2.41.0.162.gfafddb0af9-goog
#4 lore raw

PATCH 2/3 KVM: arm64: Refactor HFGxTR configuration into separate helpers

In reply to Oliver Upton

A subsequent change will need to flip more trap bits in HFGWTR_EL2. Make room for this by factoring out the programming of the HFGxTR registers into helpers and using locals to build the set/clear masks.

---
 arch/arm64/kvm/hyp/include/hyp/switch.h | 60 +++++++++++++++++--------
 1 file changed, 42 insertions(+), 18 deletions(-)
arch/arm64/kvm/hyp/include/hyp/switch.h +42 -18
index c41166f1a1dd..e5702c27a8b2 100644
@@ -70,6 +70,44 @@ static inline void __activate_traps_fpsimd32(struct kvm_vcpu *vcpu)
 	}
 }
 
+static inline bool __hfgxtr_traps_required(void)
+{
+	if (cpus_have_final_cap(ARM64_SME))
+		return true;
+
+	return false;
+}
+
+static inline void __activate_traps_hfgxtr(void)
+{
+	u64 r_clr = 0, w_clr = 0, r_set = 0, w_set = 0, tmp;
+
+	if (cpus_have_final_cap(ARM64_SME)) {
+		tmp = HFGxTR_EL2_nSMPRI_EL1_MASK | HFGxTR_EL2_nTPIDR2_EL0_MASK;
+
+		r_clr |= tmp;
+		w_clr |= tmp;
+	}
+
+	sysreg_clear_set_s(SYS_HFGRTR_EL2, r_clr, r_set);
+	sysreg_clear_set_s(SYS_HFGWTR_EL2, w_clr, w_set);
+}
+
+static inline void __deactivate_traps_hfgxtr(void)
+{
+	u64 r_clr = 0, w_clr = 0, r_set = 0, w_set = 0, tmp;
+
+	if (cpus_have_final_cap(ARM64_SME)) {
+		tmp = HFGxTR_EL2_nSMPRI_EL1_MASK | HFGxTR_EL2_nTPIDR2_EL0_MASK;
+
+		r_set |= tmp;
+		w_set |= tmp;
+	}
+
+	sysreg_clear_set_s(SYS_HFGRTR_EL2, r_clr, r_set);
+	sysreg_clear_set_s(SYS_HFGWTR_EL2, w_clr, w_set);
+}
+
 static inline void __activate_traps_common(struct kvm_vcpu *vcpu)
 {
 	/* Trap on AArch32 cp15 c15 (impdef sysregs) accesses (EL1 or EL0) */
@@ -89,16 +127,8 @@ static inline void __activate_traps_common(struct kvm_vcpu *vcpu)
 	vcpu->arch.mdcr_el2_host = read_sysreg(mdcr_el2);
 	write_sysreg(vcpu->arch.mdcr_el2, mdcr_el2);
 
-	if (cpus_have_final_cap(ARM64_SME)) {
-		sysreg_clear_set_s(SYS_HFGRTR_EL2,
-				   HFGxTR_EL2_nSMPRI_EL1_MASK |
-				   HFGxTR_EL2_nTPIDR2_EL0_MASK,
-				   0);
-		sysreg_clear_set_s(SYS_HFGWTR_EL2,
-				   HFGxTR_EL2_nSMPRI_EL1_MASK |
-				   HFGxTR_EL2_nTPIDR2_EL0_MASK,
-				   0);
-	}
+	if (__hfgxtr_traps_required())
+		__activate_traps_hfgxtr();
 }
 
 static inline void __deactivate_traps_common(struct kvm_vcpu *vcpu)
@@ -109,14 +139,8 @@ static inline void __deactivate_traps_common(struct kvm_vcpu *vcpu)
 	if (kvm_arm_support_pmu_v3())
 		write_sysreg(0, pmuserenr_el0);
 
-	if (cpus_have_final_cap(ARM64_SME)) {
-		sysreg_clear_set_s(SYS_HFGRTR_EL2, 0,
-				   HFGxTR_EL2_nSMPRI_EL1_MASK |
-				   HFGxTR_EL2_nTPIDR2_EL0_MASK);
-		sysreg_clear_set_s(SYS_HFGWTR_EL2, 0,
-				   HFGxTR_EL2_nSMPRI_EL1_MASK |
-				   HFGxTR_EL2_nTPIDR2_EL0_MASK);
-	}
+	if (__hfgxtr_traps_required())
+		__deactivate_traps_hfgxtr();
 }
 
 static inline void ___activate_traps(struct kvm_vcpu *vcpu)
Signature · 1 line
2.41.0.162.gfafddb0af9-goog
#5 lore raw

PATCH 3/3 KVM: arm64: Prevent guests from enabling HA/HD on Ampere1

In reply to Oliver Upton

An erratum in the HAFDBS implementation in AmpereOne was addressed by clearing the feature in the ID register, with the expectation that software would not attempt to use the corresponding controls in TCR_EL1. The architecture, on the other hand, takes a much more pedantic stance on the subject, requiring the TCR bits behave as RES0.

Take an extremely conservative stance on the issue and leverage the precise write trap afforded by FGT. Handle guest writes by clearing HA and HD before writing the intended value to the EL1 register alias.

---
 arch/arm64/kvm/hyp/include/hyp/switch.h | 39 +++++++++++++++++++++++++
 1 file changed, 39 insertions(+)
arch/arm64/kvm/hyp/include/hyp/switch.h +39 -0
index e5702c27a8b2..17d6bcd321d8 100644
@@ -75,6 +75,9 @@ static inline bool __hfgxtr_traps_required(void)
 	if (cpus_have_final_cap(ARM64_SME))
 		return true;
 
+	if (cpus_have_final_cap(ARM64_WORKAROUND_AMPERE_AC03_CPU_38))
+		return true;
+
 	return false;
 }
 
@@ -89,6 +92,12 @@ static inline void __activate_traps_hfgxtr(void)
 		w_clr |= tmp;
 	}
 
+	/*
+	 * Trap guest writes to TCR_EL1 to prevent it from enabling HA or HD.
+	 */
+	if (cpus_have_final_cap(ARM64_WORKAROUND_AMPERE_AC03_CPU_38))
+		w_set |= HFGxTR_EL2_TCR_EL1_MASK;
+
 	sysreg_clear_set_s(SYS_HFGRTR_EL2, r_clr, r_set);
 	sysreg_clear_set_s(SYS_HFGWTR_EL2, w_clr, w_set);
 }
@@ -104,6 +113,9 @@ static inline void __deactivate_traps_hfgxtr(void)
 		w_set |= tmp;
 	}
 
+	if (cpus_have_final_cap(ARM64_WORKAROUND_AMPERE_AC03_CPU_38))
+		w_clr |= HFGxTR_EL2_TCR_EL1_MASK;
+
 	sysreg_clear_set_s(SYS_HFGRTR_EL2, r_clr, r_set);
 	sysreg_clear_set_s(SYS_HFGWTR_EL2, w_clr, w_set);
 }
@@ -400,12 +412,39 @@ static bool kvm_hyp_handle_cntpct(struct kvm_vcpu *vcpu)
 	return true;
 }
 
+static bool handle_ampere1_tcr(struct kvm_vcpu *vcpu)
+{
+	u32 sysreg = esr_sys64_to_sysreg(kvm_vcpu_get_esr(vcpu));
+	int rt = kvm_vcpu_sys_get_rt(vcpu);
+	u64 val = vcpu_get_reg(vcpu, rt);
+
+	if (sysreg != SYS_TCR_EL1)
+		return false;
+
+	/*
+	 * Affected parts do not advertise support for hardware Access Flag /
+	 * Dirty state management in ID_AA64MMFR1_EL1.HAFDBS, but the underlying
+	 * control bits are still functional. The architecture requires these be
+	 * RES0 on systems that do not implement FEAT_HAFDBS.
+	 *
+	 * Uphold the requirements of the architecture by masking guest writes
+	 * to TCR_EL1.{HA,HD} here.
+	 */
+	val &= ~(TCR_HD | TCR_HA);
+	write_sysreg_el1(val, SYS_TCR);
+	return true;
+}
+
 static bool kvm_hyp_handle_sysreg(struct kvm_vcpu *vcpu, u64 *exit_code)
 {
 	if (cpus_have_final_cap(ARM64_WORKAROUND_CAVIUM_TX2_219_TVM) &&
 	    handle_tx2_tvm(vcpu))
 		return true;
 
+	if (cpus_have_final_cap(ARM64_WORKAROUND_AMPERE_AC03_CPU_38) &&
+	    handle_ampere1_tcr(vcpu))
+		return true;
+
 	if (static_branch_unlikely(&vgic_v3_cpuif_trap) &&
 	    __vgic_v3_perform_cpuif_access(vcpu) == 1)
 		return true;
Signature · 1 line
2.41.0.162.gfafddb0af9-goog
maintainer
#6 lore raw
In reply to Oliver Upton
Quoted 10 lines · Oliver Upton

Small series to work around a CPU erratum on AmpereOne. While the implementation does not advertise support for FEAT_HAFDBS (due to another erratum), the associated control bits do not have RES0 behavior as required by the architecture.

Usage of HAFDBS at stage-1 is unaffected, since HA and HD are only enabled on implementations that advertise the feature. However, KVM relies on HA having RES0 semantics if the feature isn't implemented. The end result is that KVM enables a broken hardware access flag implementation that could lead to correctness issues.

Just curious, what's the correctness issue here? The access flag is mostly indicative of which pages are old for swapping out/discarding. It's not like the dirty state which would be dangerous if we get wrong.

Signature · 1 line
Catalin
maintainer Acked-by on v1 1/3
#2 lore raw

Re: [PATCH 1/3] arm64: errata: Mitigate Ampere1 erratum AC03_CPU_38 at stage-2

In reply to Oliver Upton
Quoted 13 lines · Oliver Upton

AmpereOne has an erratum in its implementation of FEAT_HAFDBS that required disabling the feature on the design. This was done by reporting the feature as not implemented in the ID register, although the corresponding control bits were not actually RES0. This does not align well with the requirements of the architecture, which mandates these bits be RES0 if HAFDBS isn't implemented.

The kernel's use of stage-1 is unaffected, as the HA and HD bits are only set if HAFDBS is detected in the ID register. KVM, on the other hand, relies on the RES0 behavior at stage-2 to use the same value for VTCR_EL2 on any cpu in the system. Mitigate the non-RES0 behavior by leaving VTCR_EL2.HA clear on affected systems.

Quoted 4 lines

For the non-KVM bits in here:

Acked-by on v1 1/3
#3 lore raw

Re: [PATCH 1/3] arm64: errata: Mitigate Ampere1 erratum AC03_CPU_38 at stage-2

In reply to Oliver Upton
Quoted 13 lines · Oliver Upton

AmpereOne has an erratum in its implementation of FEAT_HAFDBS that required disabling the feature on the design. This was done by reporting the feature as not implemented in the ID register, although the corresponding control bits were not actually RES0. This does not align well with the requirements of the architecture, which mandates these bits be RES0 if HAFDBS isn't implemented.

The kernel's use of stage-1 is unaffected, as the HA and HD bits are only set if HAFDBS is detected in the ID register. KVM, on the other hand, relies on the RES0 behavior at stage-2 to use the same value for VTCR_EL2 on any cpu in the system. Mitigate the non-RES0 behavior by leaving VTCR_EL2.HA clear on affected systems.

Quoted 4 lines
#7 lore raw
In reply to Catalin Marinas

Hey Catalin,

Quoted 12 lines · Catalin Marinas
Quoted 10 lines · Oliver Upton

Small series to work around a CPU erratum on AmpereOne. While the implementation does not advertise support for FEAT_HAFDBS (due to another erratum), the associated control bits do not have RES0 behavior as required by the architecture.

Usage of HAFDBS at stage-1 is unaffected, since HA and HD are only enabled on implementations that advertise the feature. However, KVM relies on HA having RES0 semantics if the feature isn't implemented. The end result is that KVM enables a broken hardware access flag implementation that could lead to correctness issues.

Quoted 3 lines

Just curious, what's the correctness issue here? The access flag is mostly indicative of which pages are old for swapping out/discarding. It's not like the dirty state which would be dangerous if we get wrong.

I probably could have helped out by giving the full context.

The software-observable behavior on this system is that the A or D updates could arrive after a PTE has been marked as invalid, which could corrupt software metadata stuffed into the page tables. We do exactly that at stage-2 in KVM for parallel fault handling, where a magic value indicates a PTE is being updated by another thread.

Signature · 2 lines
Thanks,
Oliver
maintainer
#8 lore raw
Quoted 19 lines · Oliver Upton

Hey Catalin,

Quoted 12 lines · Catalin Marinas
Quoted 10 lines · Oliver Upton

Small series to work around a CPU erratum on AmpereOne. While the implementation does not advertise support for FEAT_HAFDBS (due to another erratum), the associated control bits do not have RES0 behavior as required by the architecture.

Usage of HAFDBS at stage-1 is unaffected, since HA and HD are only enabled on implementations that advertise the feature. However, KVM relies on HA having RES0 semantics if the feature isn't implemented. The end result is that KVM enables a broken hardware access flag implementation that could lead to correctness issues.

Quoted 3 lines

Just curious, what's the correctness issue here? The access flag is mostly indicative of which pages are old for swapping out/discarding. It's not like the dirty state which would be dangerous if we get wrong.

Quoted 7 lines

I probably could have helped out by giving the full context.

The software-observable behavior on this system is that the A or D updates could arrive after a PTE has been marked as invalid, which could corrupt software metadata stuffed into the page tables. We do exactly that at stage-2 in KVM for parallel fault handling, where a magic value indicates a PTE is being updated by another thread.

Ah, ok, that's dangerous indeed. Thanks for the details (you may want to add them in the patch description or the erratum kconfig entry).

Signature · 1 line
Catalin
maintainer Reviewed-by on the v1 series
#9 lore raw
In reply to Oliver Upton
Quoted 8 lines · Oliver Upton

Hi folks,

Small series to work around a CPU erratum on AmpereOne. While the implementation does not advertise support for FEAT_HAFDBS (due to another erratum), the associated control bits do not have RES0 behavior as required by the architecture.

Quoted 8 lines

Usage of HAFDBS at stage-1 is unaffected, since HA and HD are only enabled on implementations that advertise the feature. However, KVM relies on HA having RES0 semantics if the feature isn't implemented. The end result is that KVM enables a broken hardware access flag implementation that could lead to correctness issues.

Applies to 6.4-rc1. Tested with access_tracking_perf_test, verifying that KVM is indeed taking Access Flag faults.

For the series:

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#10 lore raw
In reply to Oliver Upton
Quoted 5 lines · Oliver Upton

Small series to work around a CPU erratum on AmpereOne. While the implementation does not advertise support for FEAT_HAFDBS (due to another erratum), the associated control bits do not have RES0 behavior as required by the architecture.

Quoted 7 lines

Usage of HAFDBS at stage-1 is unaffected, since HA and HD are only enabled on implementations that advertise the feature. However, KVM relies on HA having RES0 semantics if the feature isn't implemented. The end result is that KVM enables a broken hardware access flag implementation that could lead to correctness issues.

[...]

Applied w/ an expanded description of what's wrong with the unadvertised HAFDBS implementation, per Catalin's suggestion.

[1/3] arm64: errata: Mitigate Ampere1 erratum AC03_CPU_38 at stage-2

      https://git.kernel.org/kvmarm/kvmarm/c/6df696cd9bc1
[2/3] KVM: arm64: Refactor HFGxTR configuration into separate helpers
      https://git.kernel.org/kvmarm/kvmarm/c/ce4a36225753
[3/3] KVM: arm64: Prevent guests from enabling HA/HD on Ampere1
      https://git.kernel.org/kvmarm/kvmarm/c/082fdfd13841

--
Best,
Oliver