Initial BBML2 support for contpte_convert()
Subsystems (4)
v2 cover Initial BBML2 support for contpte_convert()
Hi All,
This patch series adds adding initial support for eliding break-before-make requirements on systems that support BBML2 and additionally guarantee to never raise a conflict abort.
This support reorders and optionally elides a TLB invalidation in contpte_convert(). The elision of said invalidation leads to a 12% improvement when executing a microbenchmark designed to force the pathological path where contpte_convert() gets called. This represents an 80% reduction in the cost of calling contpte_convert().
However, even without the elision, the reodering represents a performance improvement due to reducing thread contention, as there is a smaller time window for racing threads to see an invalid pagetable entry (especially if they already have a cached entry in their TLB that they are working off of).
This series is based on v6.14-rc3 (0ad2507d5d93).
Patch 1 implements an allow-list of cpus that support BBML2, but with the additional constraint of never causing TLB conflict aborts. We settled on this constraint because we will use the feature for kernel mappings in the future, for which we cannot handle conflict aborts safely.
Yang Shi has a series at [1] that aims to use BBML2 to enable splitting the linear map at runtime. This series partially overlaps with it to add the cpu feature. We beleive this series is fully compatible with Yang's requirements and could go first, given there is still a lot of discussion around the best way to manage the mechanics of splitting/collapsing the linear map.
[1]:
https://lore.kernel.org/linux-arm-kernel/[email protected]/
Mikołaj Lenczewski (4):
arm64: Add BBM Level 2 cpu feature
arm64/mm: Delay tlbi in contpte_convert() under BBML2 arm64/mm: Elide tlbi in contpte_convert() under BBML2 iommu/arm: Add BBM Level 2 smmu feature
arch/arm64/Kconfig | 11 +++ arch/arm64/include/asm/cpucaps.h | 2 + arch/arm64/include/asm/cpufeature.h | 5 ++ arch/arm64/kernel/cpufeature.c | 68 +++++++++++++++++++ arch/arm64/mm/contpte.c | 3 +- arch/arm64/tools/cpucaps | 1 + .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 3 + drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 3 + drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 4 ++ 9 files changed, 99 insertions(+), 1 deletion(-)
Signature · 1 line
2.45.3
PATCH v2 1/4 arm64: Add BBM Level 2 cpu feature
The Break-Before-Make cpu feature supports multiple levels (levels 0-2), and this commit adds a dedicated BBML2 cpufeature to test against support for.
This is a system feature as we might have a big.LITTLE architecture where some cores support BBML2 and some don't, but we want all cores to be available and BBM to default to level 0 (as opposed to having cores without BBML2 not coming online).
To support BBML2 in as wide a range of contexts as we can, we want not only the architectural guarantees that BBML2 makes, but additionally want BBML2 to not create TLB conflict aborts. Not causing aborts avoids us having to prove that no recursive faults can be induced in any path that uses BBML2, allowing its use for arbitrary kernel mappings. Support detection of such CPUs.
--- arch/arm64/Kconfig | 11 +++++ arch/arm64/include/asm/cpucaps.h | 2 + arch/arm64/include/asm/cpufeature.h | 5 +++ arch/arm64/kernel/cpufeature.c | 69 +++++++++++++++++++++++++++++ arch/arm64/tools/cpucaps | 1 + 5 files changed, 88 insertions(+)
arch/arm64/Kconfig +11 -0
@@ -2057,6 +2057,17 @@ config ARM64_TLB_RANGE The feature introduces new assembly instructions, and they were support when binutils >= 2.30. +config ARM64_ENABLE_BBML2_NOABORT + bool "Enable support for Break-Before-Make Level 2 detection and usage" + default y + help + FEAT_BBM provides detection of support levels for break-before-make + sequences. If BBM level 2 is supported, some TLB maintenance requirements + can be relaxed to improve performance. We additonally require the + property that the implementation cannot ever raise TLB Conflict Aborts. + Selecting N causes the kernel to fallback to BBM level 0 behaviour + even if the system supports BBM level 2. + endmenu # "ARMv8.4 architectural features" menu "ARMv8.5 architectural features"
arch/arm64/include/asm/cpucaps.h +2 -0
@@ -23,6 +23,8 @@ cpucap_is_possible(const unsigned int cap)return IS_ENABLED (CONFIG_ARM64_PAN );case ARM64_HAS_EPAN :return IS_ENABLED (CONFIG_ARM64_EPAN ); +case ARM64_HAS_BBML2_NOABORT : +return IS_ENABLED (CONFIG_ARM64_BBML2_NOABORT );case ARM64_SVE :return IS_ENABLED (CONFIG_ARM64_SVE );case ARM64_SME :
arch/arm64/include/asm/cpufeature.h +5 -0
@@ -866,6 +866,11 @@ static __always_inline bool system_supports_mpam_hcr(void)return alternative_has_cap_unlikely (ARM64_MPAM_HCR ); } +static inline bool system_supports_bbml2_noabort (void ) +{ +return alternative_has_cap_unlikely (ARM64_HAS_BBML2_NOABORT ); +} +int do_emulate_mrs (struct pt_regs *regs,u32 sys_reg,u32 rt);bool try_emulate_mrs (struct pt_regs *regs,u32 isn);
arch/arm64/kernel/cpufeature.c +69 -0
81 lines not on this page · raw message
arch/arm64/tools/cpucaps +1 -0
@@ -14,6 +14,7 @@ HAS_ADDRESS_AUTH_ARCH_QARMA5 HAS_ADDRESS_AUTH_IMP_DEF HAS_AMU_EXTN HAS_ARMv8_4_TTL +HAS_BBML2_NOABORT HAS_CACHE_DIC HAS_CACHE_IDC HAS_CNP
Signature · 1 line
2.45.3
Re: [PATCH v2 1/4] arm64: Add BBM Level 2 cpu feature
Hi Miko,
Thanks for getting this work. It is perfect timing. I'm going to have my series build on top of this patch.
Yang
Quoted 175 lines · Mikołaj Lenczewski
The Break-Before-Make cpu feature supports multiple levels (levels 0-2), and this commit adds a dedicated BBML2 cpufeature to test against support for.
This is a system feature as we might have a big.LITTLE architecture where some cores support BBML2 and some don't, but we want all cores to be available and BBM to default to level 0 (as opposed to having cores without BBML2 not coming online).
To support BBML2 in as wide a range of contexts as we can, we want not only the architectural guarantees that BBML2 makes, but additionally want BBML2 to not create TLB conflict aborts. Not causing aborts avoids us having to prove that no recursive faults can be induced in any path that uses BBML2, allowing its use for arbitrary kernel mappings. Support detection of such CPUs.
Signed-off-by Mikołaj Lenczewski <[email protected]>--- arch/arm64/Kconfig | 11 +++++ arch/arm64/include/asm/cpucaps.h | 2 + arch/arm64/include/asm/cpufeature.h | 5 +++ arch/arm64/kernel/cpufeature.c | 69 +++++++++++++++++++++++++++++ arch/arm64/tools/cpucaps | 1 + 5 files changed, 88 insertions(+)arch/arm64/Kconfig +11
-0@@ -2057,6 +2057,17 @@ config ARM64_TLB_RANGE The feature introduces new assembly instructions, and they were support when binutils >= 2.30. +config ARM64_ENABLE_BBML2_NOABORT + bool "Enable support for Break-Before-Make Level 2 detection and usage" + default y + help + FEAT_BBM provides detection of support levels for break-before-make + sequences. If BBM level 2 is supported, some TLB maintenance requirements + can be relaxed to improve performance. We additonally require the + property that the implementation cannot ever raise TLB Conflict Aborts. + Selecting N causes the kernel to fallback to BBM level 0 behaviour + even if the system supports BBM level 2. + endmenu # "ARMv8.4 architectural features" menu "ARMv8.5 architectural features"arch/arm64/include/asm/cpucaps.h +2
-0@@ -23,6 +23,8 @@ cpucap_is_possible(const unsigned int cap)return IS_ENABLED (CONFIG_ARM64_PAN );case ARM64_HAS_EPAN :return IS_ENABLED (CONFIG_ARM64_EPAN ); +case ARM64_HAS_BBML2_NOABORT : +return IS_ENABLED (CONFIG_ARM64_BBML2_NOABORT );case ARM64_SVE :return IS_ENABLED (CONFIG_ARM64_SVE );case ARM64_SME :arch/arm64/include/asm/cpufeature.h +5
-0@@ -866,6 +866,11 @@ static __always_inline bool system_supports_mpam_hcr(void)return alternative_has_cap_unlikely (ARM64_MPAM_HCR ); } +static inline bool system_supports_bbml2_noabort (void ) +{ +return alternative_has_cap_unlikely (ARM64_HAS_BBML2_NOABORT ); +} +int do_emulate_mrs (struct pt_regs *regs,u32 sys_reg,u32 rt);bool try_emulate_mrs (struct pt_regs *regs,u32 isn);arch/arm64/kernel/cpufeature.c +69
-0@@ -2176,6 +2176,68 @@ static bool hvhe_possible(const struct arm64_cpu_capabilities *entry,return arm64_test_sw_feature_override (ARM64_SW_FEATURE_OVERRIDE_HVHE ); } +static bool cpu_has_bbml2_noabort (unsigned int cpu_midr) +{ +/* We want to allow usage of bbml2 in as wide a range of kernel contexts +* as possible. This list is therefore an allow-list of known-good +* implementations that both support bbml2 and additionally, fulfill the +* extra constraint of never generating TLB conflict aborts when using +* the relaxed bbml2 semantics (such aborts make use of bbml2 in certain +* kernel contexts difficult to prove safe against recursive aborts). +* +* Note that implementations can only be considered "known-good" if their +* implementors attest to the fact that the implementation never raises +* TLBI conflict aborts for bbml2 mapping granularity changes. +*/ +static const struct midr_range supports_bbml2_noabort_list[] = { +MIDR_REV_RANGE (MIDR_CORTEX_X4 ,0 ,3 ,0xf ), +MIDR_REV_RANGE (MIDR_NEOVERSE_V3 ,0 ,2 ,0xf ), + {} + }; + +return is_midr_in_range_list (cpu_midr, supports_bbml2_noabort_list); +} + +static inline unsigned int __cpu_read_midr (int cpu) +{ +WARN_ON_ONCE (!cpu_online (cpu)); + +return per_cpu (cpu_data, cpu).reg_midr; +} + +static bool has_bbml2_noabort (const struct arm64_cpu_capabilities *caps,int scope) +{ +if (!IS_ENABLED (CONFIG_ARM64_ENABLE_BBML2_NOABORT )) +return false; + +if (scope &SCOPE_SYSTEM ) { +int cpu; + +/* We are a boot CPU, and must verify that all enumerated boot +* CPUs have MIDR values within our allowlist. Otherwise, we do +* not allow the BBML2 feature to avoid potential faults when +* the insufficient CPUs access memory regions using BBML2 +* semantics. +*/ +for_each_online_cpu (cpu) { +if (!cpu_has_bbml2_noabort (__cpu_read_midr (cpu))) +return false; + } + +return true; + }else if (scope &SCOPE_LOCAL_CPU ) { +/* We are a hot-plugged CPU, so only need to check our MIDR. +* If we have the correct MIDR, but the kernel booted on an +* insufficient CPU, we will not use BBML2 (this is safe). If +* we have an incorrect MIDR, but the kernel booted on a +* sufficient CPU, we will not bring up this CPU. +*/ +return cpu_has_bbml2_noabort (read_cpuid_id ()); + } + +return false; +} +#ifdef CONFIG_ARM64_PAN static void cpu_enable_pan (const struct arm64_cpu_capabilities *__unused) {@@ -2926,6 +2988,13 @@ static const struct arm64_cpu_capabilities arm64_features[] = { .matches = has_cpuid_feature,ARM64_CPUID_FIELDS (ID_AA64MMFR2_EL1 ,EVT ,IMP ) }, + { + .desc ="BBM Level 2 without conflict abort" , + .capability = ARM64_HAS_BBML2_NOABORT, + .type = ARM64_CPUCAP_SYSTEM_FEATURE, + .matches = has_bbml2_noabort, +ARM64_CPUID_FIELDS (ID_AA64MMFR2_EL1 ,BBM ,2 ) + }, { .desc ="52-bit Virtual Addressing for KVM (LPA2)" , .capability = ARM64_HAS_LPA2,arch/arm64/tools/cpucaps +1
-0@@ -14,6 +14,7 @@ HAS_ADDRESS_AUTH_ARCH_QARMA5 HAS_ADDRESS_AUTH_IMP_DEF HAS_AMU_EXTN HAS_ARMv8_4_TTL +HAS_BBML2_NOABORT HAS_CACHE_DIC HAS_CACHE_IDC HAS_CNP
Re: [PATCH v2 1/4] arm64: Add BBM Level 2 cpu feature
In reply to Mikołaj LenczewskiQuoted 65 lines · Mikołaj Lenczewski
The Break-Before-Make cpu feature supports multiple levels (levels 0-2), and this commit adds a dedicated BBML2 cpufeature to test against support for.
This is a system feature as we might have a big.LITTLE architecture where some cores support BBML2 and some don't, but we want all cores to be available and BBM to default to level 0 (as opposed to having cores without BBML2 not coming online).
To support BBML2 in as wide a range of contexts as we can, we want not only the architectural guarantees that BBML2 makes, but additionally want BBML2 to not create TLB conflict aborts. Not causing aborts avoids us having to prove that no recursive faults can be induced in any path that uses BBML2, allowing its use for arbitrary kernel mappings. Support detection of such CPUs.
Signed-off-by Mikołaj Lenczewski <[email protected]>--- arch/arm64/Kconfig | 11 +++++ arch/arm64/include/asm/cpucaps.h | 2 + arch/arm64/include/asm/cpufeature.h | 5 +++ arch/arm64/kernel/cpufeature.c | 69 +++++++++++++++++++++++++++++ arch/arm64/tools/cpucaps | 1 + 5 files changed, 88 insertions(+)arch/arm64/Kconfig +11
-0@@ -2057,6 +2057,17 @@ config ARM64_TLB_RANGE The feature introduces new assembly instructions, and they were support when binutils >= 2.30. +config ARM64_ENABLE_BBML2_NOABORT + bool "Enable support for Break-Before-Make Level 2 detection and usage" + default y + help + FEAT_BBM provides detection of support levels for break-before-make + sequences. If BBM level 2 is supported, some TLB maintenance requirements + can be relaxed to improve performance. We additonally require the + property that the implementation cannot ever raise TLB Conflict Aborts. + Selecting N causes the kernel to fallback to BBM level 0 behaviour + even if the system supports BBM level 2. + endmenu # "ARMv8.4 architectural features" menu "ARMv8.5 architectural features"arch/arm64/include/asm/cpucaps.h +2
-0@@ -23,6 +23,8 @@ cpucap_is_possible(const unsigned int cap)return IS_ENABLED (CONFIG_ARM64_PAN );case ARM64_HAS_EPAN :return IS_ENABLED (CONFIG_ARM64_EPAN ); +case ARM64_HAS_BBML2_NOABORT : +return IS_ENABLED (CONFIG_ARM64_BBML2_NOABORT );case ARM64_SVE :return IS_ENABLED (CONFIG_ARM64_SVE );case ARM64_SME :arch/arm64/include/asm/cpufeature.h +0
-0
arch/arm64/include/asm/cpufeature.h static __always_inline bool system_supports_mpam_hcr(void) · line 872
Quoted 7 lines
arch/arm64/include/asm/cpufeature.h +4
-0@@ -866,6 +866,11 @@ static __always_inline bool system_supports_mpam_hcr(void)return alternative_has_cap_unlikely (ARM64_MPAM_HCR ); } +static inline bool system_supports_bbml2_noabort (void ) +{ +return alternative_has_cap_unlikely (ARM64_HAS_BBML2_NOABORT ); +}
Hi Miko,
I added AmpereOne mdir on top of this patch. I can see BBML2 feature is detected via dmesg. But system_supports_bbml2_noabort() returns false. The warning in the below debug patch is triggered:
arch/arm64/kernel/cpufeature.c +0 -0
@@ -3814,6 +3814,9 @@ void __init setup_system_features(void)
{
setup_system_capabilities();
+ if (!system_supports_bbml2_noabort())
+ WARN_ON_ONCE(1);
+
kpti_install_ng_mappings();
sve_setup();
I thought it may be too early. But it seems other system features work well, for example, MPAM. I didn't figure out why. It is weird.
Thanks,
Yang
Quoted 103 lines
arch/arm64/include/asm/cpufeature.h +1
-0+int do_emulate_mrs (struct pt_regs *regs,u32 sys_reg,u32 rt);bool try_emulate_mrs (struct pt_regs *regs,u32 isn);arch/arm64/kernel/cpufeature.c +69
-0@@ -2176,6 +2176,68 @@ static bool hvhe_possible(const struct arm64_cpu_capabilities *entry,return arm64_test_sw_feature_override (ARM64_SW_FEATURE_OVERRIDE_HVHE ); } +static bool cpu_has_bbml2_noabort (unsigned int cpu_midr) +{ +/* We want to allow usage of bbml2 in as wide a range of kernel contexts +* as possible. This list is therefore an allow-list of known-good +* implementations that both support bbml2 and additionally, fulfill the +* extra constraint of never generating TLB conflict aborts when using +* the relaxed bbml2 semantics (such aborts make use of bbml2 in certain +* kernel contexts difficult to prove safe against recursive aborts). +* +* Note that implementations can only be considered "known-good" if their +* implementors attest to the fact that the implementation never raises +* TLBI conflict aborts for bbml2 mapping granularity changes. +*/ +static const struct midr_range supports_bbml2_noabort_list[] = { +MIDR_REV_RANGE (MIDR_CORTEX_X4 ,0 ,3 ,0xf ), +MIDR_REV_RANGE (MIDR_NEOVERSE_V3 ,0 ,2 ,0xf ), + {} + }; + +return is_midr_in_range_list (cpu_midr, supports_bbml2_noabort_list); +} + +static inline unsigned int __cpu_read_midr (int cpu) +{ +WARN_ON_ONCE (!cpu_online (cpu)); + +return per_cpu (cpu_data, cpu).reg_midr; +} + +static bool has_bbml2_noabort (const struct arm64_cpu_capabilities *caps,int scope) +{ +if (!IS_ENABLED (CONFIG_ARM64_ENABLE_BBML2_NOABORT )) +return false; + +if (scope &SCOPE_SYSTEM ) { +int cpu; + +/* We are a boot CPU, and must verify that all enumerated boot +* CPUs have MIDR values within our allowlist. Otherwise, we do +* not allow the BBML2 feature to avoid potential faults when +* the insufficient CPUs access memory regions using BBML2 +* semantics. +*/ +for_each_online_cpu (cpu) { +if (!cpu_has_bbml2_noabort (__cpu_read_midr (cpu))) +return false; + } + +return true; + }else if (scope &SCOPE_LOCAL_CPU ) { +/* We are a hot-plugged CPU, so only need to check our MIDR. +* If we have the correct MIDR, but the kernel booted on an +* insufficient CPU, we will not use BBML2 (this is safe). If +* we have an incorrect MIDR, but the kernel booted on a +* sufficient CPU, we will not bring up this CPU. +*/ +return cpu_has_bbml2_noabort (read_cpuid_id ()); + } + +return false; +} +#ifdef CONFIG_ARM64_PAN static void cpu_enable_pan (const struct arm64_cpu_capabilities *__unused) {@@ -2926,6 +2988,13 @@ static const struct arm64_cpu_capabilities arm64_features[] = { .matches = has_cpuid_feature,ARM64_CPUID_FIELDS (ID_AA64MMFR2_EL1 ,EVT ,IMP ) }, + { + .desc ="BBM Level 2 without conflict abort" , + .capability = ARM64_HAS_BBML2_NOABORT, + .type = ARM64_CPUCAP_SYSTEM_FEATURE, + .matches = has_bbml2_noabort, +ARM64_CPUID_FIELDS (ID_AA64MMFR2_EL1 ,BBM ,2 ) + }, { .desc ="52-bit Virtual Addressing for KVM (LPA2)" , .capability = ARM64_HAS_LPA2,arch/arm64/tools/cpucaps +1
-0@@ -14,6 +14,7 @@ HAS_ADDRESS_AUTH_ARCH_QARMA5 HAS_ADDRESS_AUTH_IMP_DEF HAS_AMU_EXTN HAS_ARMv8_4_TTL +HAS_BBML2_NOABORT HAS_CACHE_DIC HAS_CACHE_IDC HAS_CNP
Re: [PATCH v2 1/4] arm64: Add BBM Level 2 cpu feature
Quoted 99 lines · Yang Shi
Quoted 68 lines · Mikołaj Lenczewski
The Break-Before-Make cpu feature supports multiple levels (levels 0-2), and this commit adds a dedicated BBML2 cpufeature to test against support for.
This is a system feature as we might have a big.LITTLE architecture where some cores support BBML2 and some don't, but we want all cores to be available and BBM to default to level 0 (as opposed to having cores without BBML2 not coming online).
To support BBML2 in as wide a range of contexts as we can, we want not only the architectural guarantees that BBML2 makes, but additionally want BBML2 to not create TLB conflict aborts. Not causing aborts avoids us having to prove that no recursive faults can be induced in any path that uses BBML2, allowing its use for arbitrary kernel mappings. Support detection of such CPUs.
Signed-off-by Mikołaj Lenczewski <[email protected]>---
arch/arm64/Kconfig | 11 +++++ arch/arm64/include/asm/cpucaps.h | 2 +arch/arm64/include/asm/cpufeature.h| 5 +++ arch/arm64/kernel/cpufeature.c | 69 +++++++++++++++++++++++++++++ arch/arm64/tools/cpucaps | 1 +5 files changed, 88 insertions(+)arch/arm64/Kconfig +0
-0@@ -2057,6 +2057,17 @@ config ARM64_TLB_RANGEThe feature introduces new assembly instructions, and they were support when binutils >= 2.30.
+configARM64_ENABLE_BBML2_NOABORT
+ bool "Enable support for Break-Before-Make Level 2 detection and usage"
+ default y
+ help
+FEAT_BBMprovides detection of support levels for break-before-make
+ sequences. If BBM level 2 is supported, some TLB maintenance requirements
+ can be relaxed to improve performance. We additonally require the + property that the implementation cannot ever raise TLB Conflict Aborts.
+ Selecting N causes the kernel to fallback to BBM level 0 behaviour
+ even if the system supports BBM level 2.
+
endmenu # "ARMv8.4 architectural features"
menu "ARMv8.5 architectural features"arch/arm64/include/asm/cpucaps.h +0
-0@@ -23,6 +23,8 @@ cpucap_is_possible(const unsigned int cap) return IS_ENABLED(CONFIG_ARM64_PAN);case
ARM64_HAS_EPAN:
returnIS_ENABLED(CONFIG_ARM64_EPAN); + caseARM64_HAS_BBML2_NOABORT:
+ returnIS_ENABLED(CONFIG_ARM64_BBML2_NOABORT); caseARM64_SVE:
returnIS_ENABLED(CONFIG_ARM64_SVE); caseARM64_SME:arch/arm64/include/asm/cpufeature.h +0
-0arch/arm64/include/asm/cpufeature.h static __always_inline bool · line 872
Quoted 7 lines
arch/arm64/include/asm/cpufeature.h +3
-0@@ -866,6 +866,11 @@ static __always_inline boolsystem_supports_mpam_hcr (void ) return alternative_has_cap_unlikely(ARM64_MPAM_HCR); } +static inline bool system_supports_bbml2_noabort(void) +{ + return alternative_has_cap_unlikely(ARM64_HAS_BBML2_NOABORT); +}Hi Miko,
I added AmpereOne mdir on top of this patch. I can see BBML2 feature is detected via dmesg. But
system_supports_bbml2_noabort()returns false. The warning in the below debug patch is triggered:arch/arm64/kernel/cpufeature.c +3
-0@@ -3814,6 +3814,9 @@ void __init setup_system_features(void) { setup_system_capabilities(); + if (!system_supports_bbml2_noabort()) + WARN_ON_ONCE(1); + kpti_install_ng_mappings(); sve_setup();
Quoted 1 line
I thought it may be too early. But it seems other system features work well, for example, MPAM. I didn't figure out why. It is weird.
I just figured out the problem It is because the wrong kconfig name is used in cpucaps.h. The code is:
+ case ARM64_HAS_BBML2_NOABORT:
+ return IS_ENABLED(CONFIG_ARM64_BBML2_NOABORT);
But the kconfig name actually is:
+config ARM64_ENABLE_BBML2_NOABORT
IMHO, the "ENABLE" in kconfig name sounds unnecessary.
Thanks,
Yang
Quoted 118 lines
Thanks,
YangQuoted 112 lines
arch/arm64/include/asm/cpufeature.h +1
-0+ int do_emulate_mrs(struct pt_regs *regs, u32 sys_reg, u32 rt); bool try_emulate_mrs(struct pt_regs *regs, u32 isn);diff --git
a/arch/arm64/kernel/cpufeature.cb/arch/arm64/kernel/cpufeature.cindexd561cf3b8ac7..63f6d356dc77100644arch/arm64/kernel/cpufeature.c +2
-0@@ -2176,6 +2176,68 @@ static bool hvhe_possible(const structarm64_cpu_capabilities *entry, return arm64_test_sw_feature_override(ARM64_SW_FEATURE_OVERRIDE_HVHE); } +static bool cpu_has_bbml2_noabort(unsigned int cpu_midr) +{ + /* We want to allow usage of bbml2 in as wide a range of kernelcontexts
+ * as possible. This list is therefore an allow-list of known-good + * implementations that both support bbml2 and additionally, fulfill the
+ * extra constraint of never generating TLB conflict aborts when using
+ * the relaxed bbml2 semantics (such aborts make use of bbml2 in certain
+ * kernel contexts difficult to prove safe against recursive aborts).
+ *
+ * Note that implementations can only be considered "known-good" if their
+ * implementors attest to the fact that the implementation never raises
+ * TLBI conflict aborts for bbml2 mapping granularity changes. + */
+ static const structmidr_rangesupports_bbml2_noabort_list[] = { +MIDR_REV_RANGE(MIDR_CORTEX_X4, 0, 3, 0xf),
+MIDR_REV_RANGE(MIDR_NEOVERSE_V3, 0, 2, 0xf),
+ {}
+ };
+
+ returnis_midr_in_range_list(cpu_midr, supports_bbml2_noabort_list);arch/arm64/kernel/cpufeature.c +10
-0+} + +static inline unsigned int __cpu_read_midr (int cpu) +{ + WARN_ON_ONCE(!cpu_online(cpu)); + + return per_cpu(cpu_data, cpu).reg_midr; +} + +static bool has_bbml2_noabort(const struct arm64_cpu_capabilities*caps, int scope)
arch/arm64/kernel/cpufeature.c +32
-0+{ + if (!IS_ENABLED(CONFIG_ARM64_ENABLE_BBML2_NOABORT)) + return false; + + if (scope & SCOPE_SYSTEM) { + int cpu; + + /* We are a boot CPU, and must verify that all enumerated boot + * CPUs have MIDR values within our allowlist. Otherwise, we do + * not allow the BBML2 feature to avoid potential faults when + * the insufficient CPUs access memory regions using BBML2 + * semantics. + */ + for_each_online_cpu(cpu) { + if (!cpu_has_bbml2_noabort(__cpu_read_midr(cpu))) + return false; + } + + return true; + } else if (scope & SCOPE_LOCAL_CPU) { + /* We are a hot-plugged CPU, so only need to check our MIDR. + * If we have the correct MIDR, but the kernel booted on an + * insufficient CPU, we will not use BBML2 (this is safe). If + * we have an incorrect MIDR, but the kernel booted on a + * sufficient CPU, we will not bring up this CPU. + */ + return cpu_has_bbml2_noabort(read_cpuid_id()); + } + + return false; +} +#ifdef
CONFIG_ARM64_PAN
static voidcpu_enable_pan(const struct arm64_cpu_capabilities *__unused){arch/arm64/kernel/cpufeature.c +7
-0@@ -2926,6 +2988,13 @@ static const struct arm64_cpu_capabilities arm64_features[] = { .matches = has_cpuid_feature, ARM64_CPUID_FIELDS(ID_AA64MMFR2_EL1, EVT, IMP) }, + { + .desc = "BBM Level 2 without conflict abort", + .capability = ARM64_HAS_BBML2_NOABORT, + .type = ARM64_CPUCAP_SYSTEM_FEATURE, + .matches = has_bbml2_noabort, + ARM64_CPUID_FIELDS(ID_AA64MMFR2_EL1, BBM, 2) + }, { .desc = "52-bit Virtual Addressing for KVM (LPA2)",.capability =
ARM64_HAS_LPA2,arch/arm64/tools/cpucaps +0
-0@@ -14,6 +14,7 @@ HAS_ADDRESS_AUTH_ARCH_QARMA5
HAS_ADDRESS_AUTH_IMP_DEF
HAS_AMU_EXTN
HAS_ARMv8_4_TTL
+HAS_BBML2_NOABORT
HAS_CACHE_DIC
HAS_CACHE_IDC
HAS_CNP
Re: [PATCH v2 1/4] arm64: Add BBM Level 2 cpu feature
Quoted 91 lines · Yang Shi
Quoted 78 lines · Yang Shi
Quoted 48 lines
arch/arm64/Kconfig +0
-0@@ -2057,6 +2057,17 @@ config ARM64_TLB_RANGEThe feature introduces new assembly instructions, and they were support when binutils >= 2.30.
+configARM64_ENABLE_BBML2_NOABORT
+ bool "Enable support for Break-Before-Make Level 2 detection and usage"
+ default y
+ help
+FEAT_BBMprovides detection of support levels for break-before-make
+ sequences. If BBM level 2 is supported, some TLB maintenance requirements
+ can be relaxed to improve performance. We additonally require the + property that the implementation cannot ever raise TLB Conflict Aborts.
+ Selecting N causes the kernel to fallback to BBM level 0 behaviour
+ even if the system supports BBM level 2.
+
endmenu # "ARMv8.4 architectural features"
menu "ARMv8.5 architectural features"arch/arm64/include/asm/cpucaps.h +0
-0@@ -23,6 +23,8 @@ cpucap_is_possible(const unsigned int cap) return IS_ENABLED(CONFIG_ARM64_PAN);case
ARM64_HAS_EPAN:
returnIS_ENABLED(CONFIG_ARM64_EPAN); + caseARM64_HAS_BBML2_NOABORT:
+ returnIS_ENABLED(CONFIG_ARM64_BBML2_NOABORT); caseARM64_SVE:
returnIS_ENABLED(CONFIG_ARM64_SVE); caseARM64_SME:arch/arm64/include/asm/cpufeature.h +0
-0arch/arm64/include/asm/cpufeature.h static __always_inline bool · line 872
Quoted 7 lines
arch/arm64/include/asm/cpufeature.h +3
-0@@ -866,6 +866,11 @@ static __always_inline boolsystem_supports_mpam_hcr (void ) return alternative_has_cap_unlikely(ARM64_MPAM_HCR); } +static inline bool system_supports_bbml2_noabort(void) +{ + return alternative_has_cap_unlikely(ARM64_HAS_BBML2_NOABORT); +}Hi Miko,
I added AmpereOne mdir on top of this patch. I can see BBML2 feature is detected via dmesg. But
system_supports_bbml2_noabort()returns false. The warning in the below debug patch is triggered:arch/arm64/kernel/cpufeature.c +3
-0@@ -3814,6 +3814,9 @@ void __init setup_system_features(void) { setup_system_capabilities(); + if (!system_supports_bbml2_noabort()) + WARN_ON_ONCE(1); + kpti_install_ng_mappings(); sve_setup();Quoted 2 lines
I thought it may be too early. But it seems other system features work well, for example, MPAM. I didn't figure out why. It is weird.
I just figured out the problem It is because the wrong kconfig name is used in
cpucaps.h. The code is:+ case
ARM64_HAS_BBML2_NOABORT:
+ returnIS_ENABLED(CONFIG_ARM64_BBML2_NOABORT);
Quoted 10 lines
But the kconfig name actually is:
+config
ARM64_ENABLE_BBML2_NOABORTIMHO, the "ENABLE" in kconfig name sounds unnecessary.
Thanks,
Yang
Hi Yang,
Thank you for the review, and apologies for the slight delay.
Thanks again for the spot, I agree that ENABLE is probably redundant (and clearly, caused an issue here). Will remove this. Please let me know if there are any other issues with rebasing your patches on top of mine.
Signature · 2 lines
Kind regards, Mikołaj Lenczewski
Re: [PATCH v2 1/4] arm64: Add BBM Level 2 cpu feature
Quoted 98 lines · Mikołaj Lenczewski
Quoted 87 lines · Yang Shi
Quoted 76 lines · Yang Shi
Quoted 47 lines
arch/arm64/Kconfig +1
-0@@ -2057,6 +2057,17 @@ config ARM64_TLB_RANGE The feature introduces new assembly instructions, and they were support when binutils >= 2.30. +config ARM64_ENABLE_BBML2_NOABORT + bool "Enable support for Break-Before-Make Level 2 detectionand usage"
+ default y
+ help
+FEAT_BBMprovides detection of support levels for break-before-make
+ sequences. If BBM level 2 is supported, some TLB maintenance requirements
+ can be relaxed to improve performance. We additonally require the + property that the implementation cannot ever raise TLB Conflict Aborts.
+ Selecting N causes the kernel to fallback to BBM level 0 behaviour
+ even if the system supports BBM level 2.arch/arm64/Kconfig +1
-0+ endmenu # "ARMv8.4 architectural features" menu "ARMv8.5 architectural features"arch/arm64/include/asm/cpucaps.h +2
-0@@ -23,6 +23,8 @@ cpucap_is_possible(const unsigned int cap) return IS_ENABLED(CONFIG_ARM64_PAN); case ARM64_HAS_EPAN: return IS_ENABLED(CONFIG_ARM64_EPAN); + case ARM64_HAS_BBML2_NOABORT: + return IS_ENABLED(CONFIG_ARM64_BBML2_NOABORT); case ARM64_SVE: return IS_ENABLED(CONFIG_ARM64_SVE); case ARM64_SME:arch/arm64/include/asm/cpufeature.h +0
-0arch/arm64/include/asm/cpufeature.h static __always_inline bool · line 872
Quoted 7 lines
arch/arm64/include/asm/cpufeature.h +3
-0@@ -866,6 +866,11 @@ static __always_inline boolsystem_supports_mpam_hcr (void ) return alternative_has_cap_unlikely(ARM64_MPAM_HCR); } +static inline bool system_supports_bbml2_noabort(void) +{ + return alternative_has_cap_unlikely(ARM64_HAS_BBML2_NOABORT); +}Hi Miko,
I added AmpereOne mdir on top of this patch. I can see BBML2 feature is detected via dmesg. But
system_supports_bbml2_noabort()returns false. The warning in the below debug patch is triggered:arch/arm64/kernel/cpufeature.c +3
-0@@ -3814,6 +3814,9 @@ void __init setup_system_features(void) { setup_system_capabilities(); + if (!system_supports_bbml2_noabort()) + WARN_ON_ONCE(1); + kpti_install_ng_mappings(); sve_setup();Quoted 2 lines
I thought it may be too early. But it seems other system features work well, for example, MPAM. I didn't figure out why. It is weird.
I just figured out the problem It is because the wrong kconfig name is used in
cpucaps.h. The code is:+ case
ARM64_HAS_BBML2_NOABORT:
+ returnIS_ENABLED(CONFIG_ARM64_BBML2_NOABORT);Quoted 10 lines
But the kconfig name actually is:
+config
ARM64_ENABLE_BBML2_NOABORTIMHO, the "ENABLE" in kconfig name sounds unnecessary.
Thanks,
Yang
Quoted 8 lines
Hi Yang,
Thank you for the review, and apologies for the slight delay.
Thanks again for the spot, I agree that
ENABLEis probably redundant (and clearly, caused an issue here). Will remove this. Please let me know if there are any other issues with rebasing your patches on top of mine.
Thank you. I didn't spot any other problem on our machines. With this addressed, you can have Tested-by: Yang Shi <[email protected]>, although I can't test on asymmetric system.
Yang
Quoted 1 line
PATCH v2 2/4 arm64/mm: Delay tlbi in contpte_convert() under BBML2
In reply to Mikołaj LenczewskiWhen converting a region via contpte_convert() to use mTHP, we have two different goals. We have to mark each entry as contiguous, and we would like to smear the dirty and young (access) bits across all entries in the contiguous block. Currently, we do this by first accumulating the dirty and young bits in the block, using an atomic __ptep_get_and_clear() and the relevant pte_{dirty,young}() calls, performing a tlbi, and finally smearing the correct bits across the block using __set_ptes().
This approach works fine for BBM level 0, but with support for BBM level 2 we are allowed to reorder the tlbi to after setting the pagetable entries. This reordering means that other threads will not see an invalid pagetable entry, instead operating on stale data, until we have performed our smearing and issued the invalidation. Avoiding this invalid entry reduces faults in other threads, and thus improves performance marginally (more so when there are more threads).
--- arch/arm64/mm/contpte.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
arch/arm64/mm/contpte.c +5 -1
@@ -68,9 +68,13 @@ static void contpte_convert(struct mm_struct *mm, unsigned long addr, pte =pte_mkyoung (pte); } -__flush_tlb_range (&vma, start_addr, addr,PAGE_SIZE , true,3 ); +if (!system_supports_bbml2_noabort ()) +__flush_tlb_range (&vma, start_addr, addr,PAGE_SIZE , true,3 );__set_ptes (mm, start_addr, start_ptep, pte,CONT_PTES ); + +if (system_supports_bbml2_noabort ()) +__flush_tlb_range (&vma, start_addr, addr,PAGE_SIZE , true,3 ); } void __contpte_try_fold(struct mm_struct *mm, unsigned long addr,
Signature · 1 line
2.45.3
PATCH v2 3/4 arm64/mm: Elide tlbi in contpte_convert() under BBML2
In reply to Mikołaj LenczewskiIf we support bbml2 without conflict aborts, we can avoid the final flush and have hardware manage the tlb entries for us. Avoiding flushes is a win.
--- arch/arm64/mm/contpte.c | 3 --- 1 file changed, 3 deletions(-)
arch/arm64/mm/contpte.c +0 -3
@@ -72,9 +72,6 @@ static void contpte_convert(struct mm_struct *mm, unsigned long addr,__flush_tlb_range (&vma, start_addr, addr,PAGE_SIZE , true,3 );__set_ptes (mm, start_addr, start_ptep, pte,CONT_PTES ); - -if (system_supports_bbml2_noabort ()) -__flush_tlb_range (&vma, start_addr, addr,PAGE_SIZE , true,3 ); } void __contpte_try_fold(struct mm_struct *mm, unsigned long addr,
Signature · 1 line
2.45.3
Re: [PATCH v2 3/4] arm64/mm: Elide tlbi in contpte_convert() under BBML2
Quoted 15 lines · Mikołaj Lenczewski
If we support bbml2 without conflict aborts, we can avoid the final flush and have hardware manage the tlb entries for us. Avoiding flushes is a win.
Signed-off-by Mikołaj Lenczewski <[email protected]>Reviewed-by Ryan Roberts <[email protected]>--- arch/arm64/mm/contpte.c | 3 --- 1 file changed, 3 deletions(-)arch/arm64/mm/contpte.c +0
-0
arch/arm64/mm/contpte.c static void contpte_convert(struct mm_struct *mm, unsigned long addr, · line 77
Quoted 9 lines
arch/arm64/mm/contpte.c +0
-3@@ -72,9 +72,6 @@ static void contpte_convert(struct mm_struct *mm, unsigned long addr,__flush_tlb_range (&vma, start_addr, addr,PAGE_SIZE , true,3 );__set_ptes (mm, start_addr, start_ptep, pte,CONT_PTES ); - -if (system_supports_bbml2_noabort ()) -__flush_tlb_range (&vma, start_addr, addr,PAGE_SIZE , true,3 ); } void __contpte_try_fold(struct mm_struct *mm, unsigned long addr,
What's the point of not squashing this into #2? :)
If this split was requested during earlier review, at least seeing patch #2 on its own confused me.
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH v2 3/4] arm64/mm: Elide tlbi in contpte_convert() under BBML2
Hi David,
Thanks for taking the time to review.
Quoted 26 lines · David Hildenbrand
Quoted 15 lines · Mikołaj Lenczewski
If we support bbml2 without conflict aborts, we can avoid the final flush and have hardware manage the tlb entries for us. Avoiding flushes is a win.
Signed-off-by Mikołaj Lenczewski <[email protected]>Reviewed-by Ryan Roberts <[email protected]>--- arch/arm64/mm/contpte.c | 3 --- 1 file changed, 3 deletions(-)arch/arm64/mm/contpte.c +0
-0arch/arm64/mm/contpte.c static void contpte_convert(struct mm_struct *mm, unsigned long addr, · line 75
Quoted 7 lines
arch/arm64/mm/contpte.c +0
-3@@ -72,9 +72,6 @@ static void contpte_convert(struct mm_struct *mm, unsigned long addr,__flush_tlb_range (&vma, start_addr, addr,PAGE_SIZE , true,3 );__set_ptes (mm, start_addr, start_ptep, pte,CONT_PTES ); - -if (system_supports_bbml2_noabort ()) -__flush_tlb_range (&vma, start_addr, addr,PAGE_SIZE , true,3 ); } void __contpte_try_fold(struct mm_struct *mm, unsigned long addr,What's the point of not squashing this into #2? :)
Quoted 2 lines
If this split was requested during earlier review, at least seeing patch #2 on its own confused me.
This split is a holdover from an earlier patchset, where it was still unknown whether the removal of the second flush was permitted with BBML2. Partly this was due to us being worried about conflict aborts after the removal, and partly this was because the "delay" is a separate optimisation that we could apply even if it turned out the final patch was not architecturally sound.
Now that we do not handle conflict aborts (preferring only systems that handle BBML2 without ever raising aborts), the first issue is not a problem. The reasoning behind the second patch is also a little bit outdated, but I can see the logical split between a tlbi reorder, and the removal of the tlbi. If this is truly redundant though, I would be happy to squash the two into a single patch.
Signature · 2 lines
Kind regards, Mikołaj Lenczewski
Re: [PATCH v2 3/4] arm64/mm: Elide tlbi in contpte_convert() under BBML2
Quoted 41 lines · Mikołaj Lenczewski
Hi David,
Thanks for taking the time to review.
Quoted 26 lines · David Hildenbrand
Quoted 15 lines · Mikołaj Lenczewski
If we support bbml2 without conflict aborts, we can avoid the final flush and have hardware manage the tlb entries for us. Avoiding flushes is a win.
Signed-off-by Mikołaj Lenczewski <[email protected]>Reviewed-by Ryan Roberts <[email protected]>--- arch/arm64/mm/contpte.c | 3 --- 1 file changed, 3 deletions(-)arch/arm64/mm/contpte.c +0
-0arch/arm64/mm/contpte.c static void contpte_convert(struct mm_struct *mm, unsigned long addr, · line 75
Quoted 7 lines
arch/arm64/mm/contpte.c +0
-3@@ -72,9 +72,6 @@ static void contpte_convert(struct mm_struct *mm, unsigned long addr,__flush_tlb_range (&vma, start_addr, addr,PAGE_SIZE , true,3 );__set_ptes (mm, start_addr, start_ptep, pte,CONT_PTES ); - -if (system_supports_bbml2_noabort ()) -__flush_tlb_range (&vma, start_addr, addr,PAGE_SIZE , true,3 ); } void __contpte_try_fold(struct mm_struct *mm, unsigned long addr,What's the point of not squashing this into #2? :)
Quoted 2 lines
If this split was requested during earlier review, at least seeing patch #2 on its own confused me.
This split is a holdover from an earlier patchset, where it was still unknown whether the removal of the second flush was permitted with BBML2. Partly this was due to us being worried about conflict aborts after the removal, and partly this was because the "delay" is a separate optimisation that we could apply even if it turned out the final patch was not architecturally sound.
Quoted 6 lines
Now that we do not handle conflict aborts (preferring only systems that handle BBML2 without ever raising aborts), the first issue is not a problem. The reasoning behind the second patch is also a little bit outdated, but I can see the logical split between a tlbi reorder, and the removal of the tlbi. If this is truly redundant though, I would be happy to squash the two into a single patch.
Thanks for the information.
Does patch #2 (reordering the tlbi) have any benefit on its own? I read "other threads will not see an invalid pagetable entry", but I am not sure that is correct. A concurrent HW page table walker would still find the invalid PTE? It's just a matter of TLB state.
If there is no benefit in having patch #2 independently, I'd just squash them. Reordering to then remove is more complicated than just removing it IMHO.
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH v2 3/4] arm64/mm: Elide tlbi in contpte_convert() under BBML2
Quoted 49 lines · David Hildenbrand
Quoted 41 lines · Mikołaj Lenczewski
Hi David,
Thanks for taking the time to review.
Quoted 26 lines · David Hildenbrand
Quoted 15 lines · Mikołaj Lenczewski
If we support bbml2 without conflict aborts, we can avoid the final flush and have hardware manage the tlb entries for us. Avoiding flushes is a win.
Signed-off-by Mikołaj Lenczewski <[email protected]>Reviewed-by Ryan Roberts <[email protected]>--- arch/arm64/mm/contpte.c | 3 --- 1 file changed, 3 deletions(-)arch/arm64/mm/contpte.c +0
-0arch/arm64/mm/contpte.c static void contpte_convert(struct mm_struct *mm, unsigned long addr, · line 75
Quoted 7 lines
arch/arm64/mm/contpte.c +0
-3@@ -72,9 +72,6 @@ static void contpte_convert(struct mm_struct *mm, unsigned long addr,__flush_tlb_range (&vma, start_addr, addr,PAGE_SIZE , true,3 );__set_ptes (mm, start_addr, start_ptep, pte,CONT_PTES ); - -if (system_supports_bbml2_noabort ()) -__flush_tlb_range (&vma, start_addr, addr,PAGE_SIZE , true,3 ); } void __contpte_try_fold(struct mm_struct *mm, unsigned long addr,What's the point of not squashing this into #2? :)
Quoted 2 lines
If this split was requested during earlier review, at least seeing patch #2 on its own confused me.
This split is a holdover from an earlier patchset, where it was still unknown whether the removal of the second flush was permitted with BBML2. Partly this was due to us being worried about conflict aborts after the removal, and partly this was because the "delay" is a separate optimisation that we could apply even if it turned out the final patch was not architecturally sound.
Quoted 6 lines
Now that we do not handle conflict aborts (preferring only systems that handle BBML2 without ever raising aborts), the first issue is not a problem. The reasoning behind the second patch is also a little bit outdated, but I can see the logical split between a tlbi reorder, and the removal of the tlbi. If this is truly redundant though, I would be happy to squash the two into a single patch.
Quoted 6 lines
Thanks for the information.
Does patch #2 (reordering the tlbi) have any benefit on its own? I read "other threads will not see an invalid pagetable entry", but I am not sure that is correct. A concurrent HW page table walker would still find the invalid PTE? It's just a matter of TLB state.
I think I understand what you mean. I agree that it is possible for a concurrent walk to see an invalid TLBI state, if it is on the same TLB that the repaint is happening on. For other TLBs, the flush has not yet propagated our invalidated PTEs (from __ptep_get_and_clear()) though? That invalidation will only be seen by other TLBs after the __flush_tlb_range(), so we should save a few faults because only "local" threads will ever see the invalid entry, as opposed to all threads that try to read our modified range? Or it is the case that I have misunderstood something basic here, or that I have misinterpreted what you have written?
Quoted 9 lines
If there is no benefit in having patch #2 independently, I'd just squash them. Reordering to then remove is more complicated than just removing it IMHO.
Signature · 3 lines
Cheers, David / dhildenb
Signature · 2 lines
Kind regards, Mikołaj Lenczewski
Re: [PATCH v2 3/4] arm64/mm: Elide tlbi in contpte_convert() under BBML2
Quoted 57 lines · Mikołaj Lenczewski
Quoted 49 lines · David Hildenbrand
Quoted 41 lines · Mikołaj Lenczewski
Hi David,
Thanks for taking the time to review.
Quoted 26 lines · David Hildenbrand
Quoted 15 lines · Mikołaj Lenczewski
If we support bbml2 without conflict aborts, we can avoid the final flush and have hardware manage the tlb entries for us. Avoiding flushes is a win.
Signed-off-by Mikołaj Lenczewski <[email protected]>Reviewed-by Ryan Roberts <[email protected]>--- arch/arm64/mm/contpte.c | 3 --- 1 file changed, 3 deletions(-)arch/arm64/mm/contpte.c +0
-0arch/arm64/mm/contpte.c static void contpte_convert(struct mm_struct *mm, unsigned long addr, · line 75
Quoted 7 lines
arch/arm64/mm/contpte.c +0
-3@@ -72,9 +72,6 @@ static void contpte_convert(struct mm_struct *mm, unsigned long addr,__flush_tlb_range (&vma, start_addr, addr,PAGE_SIZE , true,3 );__set_ptes (mm, start_addr, start_ptep, pte,CONT_PTES ); - -if (system_supports_bbml2_noabort ()) -__flush_tlb_range (&vma, start_addr, addr,PAGE_SIZE , true,3 ); } void __contpte_try_fold(struct mm_struct *mm, unsigned long addr,What's the point of not squashing this into #2? :)
Quoted 2 lines
If this split was requested during earlier review, at least seeing patch #2 on its own confused me.
This split is a holdover from an earlier patchset, where it was still unknown whether the removal of the second flush was permitted with BBML2. Partly this was due to us being worried about conflict aborts after the removal, and partly this was because the "delay" is a separate optimisation that we could apply even if it turned out the final patch was not architecturally sound.
Quoted 6 lines
Now that we do not handle conflict aborts (preferring only systems that handle BBML2 without ever raising aborts), the first issue is not a problem. The reasoning behind the second patch is also a little bit outdated, but I can see the logical split between a tlbi reorder, and the removal of the tlbi. If this is truly redundant though, I would be happy to squash the two into a single patch.
Quoted 6 lines
Thanks for the information.
Does patch #2 (reordering the tlbi) have any benefit on its own? I read "other threads will not see an invalid pagetable entry", but I am not sure that is correct. A concurrent HW page table walker would still find the invalid PTE? It's just a matter of TLB state.
Quoted 4 lines
I think I understand what you mean. I agree that it is possible for a concurrent walk to see an invalid TLBI state, if it is on the same TLB that the repaint is happening on. For other TLBs, the flush has not yet propagated our invalidated PTEs (from
__ptep_get_and_clear()) though?
What I am saying is: if there is no TLB entry yet, HW will walk the page table to find no present PTE and trigger a fault.
Quoted 4 lines
That invalidation will only be seen by other TLBs after the
__flush_tlb_range(), so we should save a few faults because only "local" threads will ever see the invalid entry, as opposed to all threads that try to read our modified range?
So what you say is, that deferring the flush means that if there is already a TLB entry, flushing deferred reduces the likelihood that a page table walk is triggered that could find no present PTE: consequently, reducing the likelihood that a page fault is triggered.
(I use the word likelihood, because I assume other action could result in a TLB entry getting flushed in the meantime, such as TLB entry reuse)
Correct?
Or it is the case that I
Quoted 2 lines
have misunderstood something basic here, or that I have misinterpreted what you have written?
No, that makes it clearer, thanks.
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH v2 3/4] arm64/mm: Elide tlbi in contpte_convert() under BBML2
Quoted 7 lines
Quoted 4 lines
I think I understand what you mean. I agree that it is possible for a concurrent walk to see an invalid TLBI state, if it is on the same TLB that the repaint is happening on. For other TLBs, the flush has not yet propagated our invalidated PTEs (from
__ptep_get_and_clear()) though?What I am saying is: if there is no TLB entry yet, HW will walk the page table to find no present PTE and trigger a fault.
Yes, that is 100% correct. I believe that this is unavoidable.
Quoted 10 lines
Quoted 4 lines
That invalidation will only be seen by other TLBs after the
__flush_tlb_range(), so we should save a few faults because only "local" threads will ever see the invalid entry, as opposed to all threads that try to read our modified range?So what you say is, that deferring the flush means that if there is already a TLB entry, flushing deferred reduces the likelihood that a page table walk is triggered that could find no present PTE: consequently, reducing the likelihood that a page fault is triggered.
Quoted 4 lines
(I use the word likelihood, because I assume other action could result in a TLB entry getting flushed in the meantime, such as TLB entry reuse)
Correct?
Yes, and your language here is clearer than the original commit message (and cover letter). Will amend it to be closer to your wording.
Signature · 2 lines
Kind regards, Mikołaj Lenczewski
PATCH v2 4/4 iommu/arm: Add BBM Level 2 smmu feature
In reply to Mikołaj LenczewskiFor supporting BBM Level 2 for userspace mappings, we want to ensure that the smmu also supports its own version of BBM Level 2. Luckily, the smmu spec (IHI 0070G 3.21.1.3) is stricter than the aarch64 spec (DDI 0487K.a D8.16.2), so already guarantees that no aborts are raised when BBM level 2 is claimed.
Add the feature and testing for it under arm_smmu_sva_supported().
--- arch/arm64/kernel/cpufeature.c | 7 +++---- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 3 +++ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 3 +++ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 4 ++++ 4 files changed, 13 insertions(+), 4 deletions(-)
arch/arm64/kernel/cpufeature.c +3 -4
@@ -2223,8 +2223,6 @@ static bool has_bbml2_noabort(const struct arm64_cpu_capabilities *caps, int scoif (!cpu_has_bbml2_noabort (__cpu_read_midr (cpu)))return false; } - -return true; }else if (scope &SCOPE_LOCAL_CPU ) {/* We are a hot-plugged CPU, so only need to check our MIDR. * If we have the correct MIDR, but the kernel booted on an
@@ -2232,10 +2230,11 @@ static bool has_bbml2_noabort(const struct arm64_cpu_capabilities *caps, int sco* we have an incorrect MIDR, but the kernel booted on a * sufficient CPU, we will not bring up this CPU. */ -return cpu_has_bbml2_noabort (read_cpuid_id ()); +if (!cpu_has_bbml2_noabort (read_cpuid_id ())) +return false; } -return false; +return has_cpuid_feature (caps, scope); }#ifdef CONFIG_ARM64_PAN
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +3 -0
@@ -222,6 +222,9 @@ bool arm_smmu_sva_supported(struct arm_smmu_device *smmu) feat_mask |=ARM_SMMU_FEAT_VAX ; } +if (system_supports_bbml2_noabort ()) + feat_mask |=ARM_SMMU_FEAT_BBML2 ; +if ((smmu->features & feat_mask) != feat_mask)return false;
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +3 -0
@@ -4406,6 +4406,9 @@ static int arm_smmu_device_hw_probe(struct arm_smmu_device *smmu)if (FIELD_GET (IDR3_RIL , reg)) smmu->features |=ARM_SMMU_FEAT_RANGE_INV ; +if (FIELD_GET (IDR3_BBML , reg) ==IDR3_BBML2 ) + smmu->features |=ARM_SMMU_FEAT_BBML2 ; +/* IDR5 */ reg =readl_relaxed (smmu->base +ARM_SMMU_IDR5 );
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +4 -0
@@ -60,6 +60,9 @@ struct arm_smmu_device;#define ARM_SMMU_IDR3 0xc#define IDR3_FWB (1 << 8)#define IDR3_RIL (1 << 10) +#define IDR3_BBML GENMASK(12, 11) +#define IDR3_BBML1 (1 << 11) +#define IDR3_BBML2 (2 << 11)#define ARM_SMMU_IDR5 0x14#define IDR5_STALL_MAX GENMASK(31, 16)
@@ -754,6 +757,7 @@ struct arm_smmu_device {#define ARM_SMMU_FEAT_HA (1 << 21)#define ARM_SMMU_FEAT_HD (1 << 22)#define ARM_SMMU_FEAT_S2FWB (1 << 23) +#define ARM_SMMU_FEAT_BBML2 (1 << 24)u32 features;#define ARM_SMMU_OPT_SKIP_PREFETCH (1 << 0)
Signature · 1 line
2.45.3
Re: [PATCH v2 4/4] iommu/arm: Add BBM Level 2 smmu feature
Quoted 8 lines · Mikołaj Lenczewski
For supporting BBM Level 2 for userspace mappings, we want to ensure that the smmu also supports its own version of BBM Level 2. Luckily, the smmu spec (IHI 0070G 3.21.1.3) is stricter than the aarch64 spec (DDI 0487K.a D8.16.2), so already guarantees that no aborts are raised when BBM level 2 is claimed.
Add the feature and testing for it under
arm_smmu_sva_supported().
Quoted 7 lines
Signed-off-by Mikołaj Lenczewski <[email protected]>--- arch/arm64/kernel/cpufeature.c | 7 +++---- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 3 +++ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 3 +++ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 4 ++++ 4 files changed, 13 insertions(+), 4 deletions(-)
This patch looks good, for what it does. However for bisection safety it should be earlier, before the patches that change the page table algorithms to be unsafe for the SMMU.
However, I've heard people talking about shipping chips that have CPUs with BBML2 but SMMUs without.
On such a system it seems like your series would break previously working SVA support because this patch will end up disabling it?
Though I see your MIDR_REV list is limited, so perhaps that worry doesn't effect any real chips made with those families? I am trying to check some NVIDIA products against this list..
Jason
RE: [PATCH v2 4/4] iommu/arm: Add BBM Level 2 smmu feature
Quoted 42 lines
-----Original Message-----
From: Jason Gunthorpe <[email protected]>
Sent: Friday, February 28, 2025 7:32 PM
To: Mikołaj Lenczewski <[email protected]>; Shameerali Kolothum Thodi <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; linux- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH v2 4/4] iommu/arm: Add BBM Level 2 smmu featureQuoted 8 lines · Mikołaj Lenczewski
For supporting BBM Level 2 for userspace mappings, we want to ensure that the smmu also supports its own version of BBM Level 2. Luckily, the smmu spec (IHI 0070G 3.21.1.3) is stricter than the aarch64 spec (DDI 0487K.a D8.16.2), so already guarantees that no aborts are raised when BBM level 2 is claimed.
Add the feature and testing for it under
arm_smmu_sva_supported().Quoted 7 lines
Signed-off-by Mikołaj Lenczewski <[email protected]>--- arch/arm64/kernel/cpufeature.c | 7 +++---- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 3 +++ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 3 +++ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 4 ++++ 4 files changed, 13 insertions(+), 4 deletions(-)This patch looks good, for what it does. However for bisection safety it should be earlier, before the patches that change the page table algorithms to be unsafe for the SMMU.
However, I've heard people talking about shipping chips that have CPUs with BBML2 but SMMUs without.
Quoted 6 lines
On such a system it seems like your series would break previously working SVA support because this patch will end up disabling it?
Though I see your
MIDR_REVlist is limited, so perhaps that worry doesn't effect any real chips made with those families? I am trying to check some NVIDIA products against this list..
We do have implementations that support CPUs with BBLM2 with TLB conflict aborts and SMMUv3 with BBML2. So don't think those platforms be affected by this. Will check with our hardware folks if there is anything that will be affected by this.
Also we have plans to try to use SMMUv3 BBML2 during VM live migration to split block pages to 4K. I guess, in that case we can enable SMMU BBML2 independent of CPU side.
Thanks,
Shameer
Re: [PATCH v2 4/4] iommu/arm: Add BBM Level 2 smmu feature
Hi Both,
Thanks for review and for checking other implementations for this discrepancy.
Quoted 35 lines · Shameerali Kolothum Thodi
Quoted 35 lines
-----Original Message-----
From: Jason Gunthorpe <[email protected]>
Sent: Friday, February 28, 2025 7:32 PM
To: Mikołaj Lenczewski <[email protected]>; Shameerali Kolothum Thodi <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; linux- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH v2 4/4] iommu/arm: Add BBM Level 2 smmu featureQuoted 15 lines · Mikołaj Lenczewski
For supporting BBM Level 2 for userspace mappings, we want to ensure that the smmu also supports its own version of BBM Level 2. Luckily, the smmu spec (IHI 0070G 3.21.1.3) is stricter than the aarch64 spec (DDI 0487K.a D8.16.2), so already guarantees that no aborts are raised when BBM level 2 is claimed.
Add the feature and testing for it under
arm_smmu_sva_supported().Signed-off-by Mikołaj Lenczewski <[email protected]>--- arch/arm64/kernel/cpufeature.c | 7 +++---- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 3 +++ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 3 +++ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 4 ++++ 4 files changed, 13 insertions(+), 4 deletions(-)
Quoted 3 lines
Quoted 3 lines
This patch looks good, for what it does. However for bisection safety it should be earlier, before the patches that change the page table algorithms to be unsafe for the SMMU.
Right, I should have noticed this earlier. Will reorder the patch.
Quoted 5 lines
Quoted 5 lines
However, I've heard people talking about shipping chips that have CPUs with BBML2 but SMMUs without.
On such a system it seems like your series would break previously working SVA support because this patch will end up disabling it?
Perhaps my understanding is flawed here, but I was under the impression that with SVA both the core and smmu MUST support BBML2 to use it safely for core translations? Otherwise the smmu might experience page faults when it touches pages from the core that use BBML2, if it does not support BBML2 itself? Again, I could very well be wrong, will double check with the reference manuals.
Quoted 3 lines
Quoted 3 lines
Though I see your
MIDR_REVlist is limited, so perhaps that worry doesn't effect any real chips made with those families? I am trying to check some NVIDIA products against this list..
Hopefully, as you say, the MIDR list restricts the breakage to a limited (ideally, zero-size) set of implementations which advertise BBML2 without conflict aborts, but which do not support BBML2 on the smmu.
However, if my understanding of the BBML2 feature and how it interacts with SVA is flawed, this will obviously be something for me to fix.
Quoted 11 lines
We do have implementations that support CPUs with BBLM2 with TLB conflict aborts and SMMUv3 with BBML2. So don't think those platforms be affected by this. Will check with our hardware folks if there is anything that will be affected by this.
Also we have plans to try to use SMMUv3 BBML2 during VM live migration to split block pages to 4K. I guess, in that case we can enable SMMU BBML2 independent of CPU side.
Thanks,
Shameer
On independently enabling BBML2 on the smmu but not the CPU, this should be possible. My check was intended to catch the case where the CPU is on the MIDR allowlist, but the smmu does not support the feature. Whilst this might still be bad logic, if it is correct, then as long as the CPU is not in the allowlist sva should not be affected. And bbml2 itself on the smmu should be safe regardless, as it is stricted than the corresponding cpu-side feature.
Signature · 2 lines
Kind regards, Mikołaj Lenczewski
Re: [PATCH v2 4/4] iommu/arm: Add BBM Level 2 smmu feature
Quoted 6 lines · Mikołaj Lenczewski
Quoted 2 lines
Quoted 2 lines
On such a system it seems like your series would break previously working SVA support because this patch will end up disabling it?
Perhaps my understanding is flawed here, but I was under the impression that with SVA both the core and smmu MUST support BBML2 to use it safely for core translations?
Yes
But today's kernel does not use BBML2 in the CPU or the SMMU so it is compatible with everything.
So it is an upgrade issue, going from today's kernel without any BBML2 support to tomorrow's kernel that does then you loose SVA on previously working HW.
Quoted 6 lines
Hopefully, as you say, the MIDR list restricts the breakage to a limited (ideally, zero-size) set of implementations which advertise BBML2 without conflict aborts, but which do not support BBML2 on the smmu.
However, if my understanding of the BBML2 feature and how it interacts with SVA is flawed, this will obviously be something for me to fix.
Lets hope, I was not able to discover any NVIDIA platforms that have an issue with this series as is.
But every addition to the MIDR list will require some consideration :\
Quoted 2 lines
On independently enabling BBML2 on the smmu but not the CPU, this should be possible.
What about the reverse? Could we disable BBML2 on the CPU side on a per-mm basis? Ie when an old SMMU attaches with disable the incompatible feature? Not something for this series, but if we get into trouble down the road
Jason
Re: [PATCH v2 4/4] iommu/arm: Add BBM Level 2 smmu feature
Quoted 9 lines · Jason Gunthorpe
Quoted 6 lines · Mikołaj Lenczewski
Quoted 2 lines
Quoted 2 lines
On such a system it seems like your series would break previously working SVA support because this patch will end up disabling it?
Perhaps my understanding is flawed here, but I was under the impression that with SVA both the core and smmu MUST support BBML2 to use it safely for core translations?
Quoted 8 lines
Yes
But today's kernel does not use BBML2 in the CPU or the SMMU so it is compatible with everything.
So it is an upgrade issue, going from today's kernel without any BBML2 support to tomorrow's kernel that does then you loose SVA on previously working HW.
I see what you mean.
If you have hardware that supports cpu BBML2 but doesn't support smmu BBML2, either by virtue of having a pre-SMMUv3.2 smmu, or by only implementing BBML1, the simplest option I see is then to disable BBML2 globally in the kernel. A bad klude, but would at least leave current SVA working.
Quoted 11 lines
Quoted 6 lines
Hopefully, as you say, the MIDR list restricts the breakage to a limited (ideally, zero-size) set of implementations which advertise BBML2 without conflict aborts, but which do not support BBML2 on the smmu.
However, if my understanding of the BBML2 feature and how it interacts with SVA is flawed, this will obviously be something for me to fix.
Lets hope, I was not able to discover any NVIDIA platforms that have an issue with this series as is.
But every addition to the MIDR list will require some consideration :\
Yes, I agree. I believe that for larger cores, this is a guarantee that implementors will be able to make when adding themselves to the list. They know what smmu their core uses, and that tends to be performance-focused (hence more likely to use an smmu that support BBML2).
For smaller cores, i.e. mobile cores, this becomes a larger issue as you may have a higher likelyhood of pairing such a core with an older/insufficient smmu version. So, careful consideration is required.
Quoted 7 lines
Quoted 2 lines
On independently enabling BBML2 on the smmu but not the CPU, this should be possible.
What about the reverse? Could we disable BBML2 on the CPU side on a per-mm basis? Ie when an old SMMU attaches with disable the incompatible feature? Not something for this series, but if we get into trouble down the road
I agree in principle with having, as you point out, some mechanism to disable BBML2 when an smmu without BBML2 is used. However, I think that the complexity of such a mechanism depends on how the BBML2 feature is used by future patches.
For example, if we use BBML2 for kernel mappings, does that require us to repaint all kernel mappings when disabling BBML2 on smmu attach? I am not sure, but definitely something to be worked out.
Signature · 2 lines
Kind regards, Mikołaj Lenczewski
Re: [PATCH v2 4/4] iommu/arm: Add BBM Level 2 smmu feature
Quoted 3 lines · Mikołaj Lenczewski
For example, if we use BBML2 for kernel mappings, does that require us to repaint all kernel mappings when disabling BBML2 on smmu attach? I am not sure, but definitely something to be worked out.
No, it would be a per-mm_struct basis only if we did something like that
When the SMMU driver puts a SVA on top of the mm_struct it would disable BBML2 usage only for that mm_struct and it's contained VMAs.
Kernel mappings are never made into a SVA so are not effected. Userspace only.
Jason
Re: [PATCH v2 4/4] iommu/arm: Add BBM Level 2 smmu feature
Quoted 11 lines · Jason Gunthorpe
Quoted 3 lines · Mikołaj Lenczewski
For example, if we use BBML2 for kernel mappings, does that require us to repaint all kernel mappings when disabling BBML2 on smmu attach? I am not sure, but definitely something to be worked out.
No, it would be a per-mm_struct basis only if we did something like that
When the SMMU driver puts a SVA on top of the
mm_structit would disable BBML2 usage only for thatmm_structand it's contained VMAs.
I guess we would need to figure out some synchonization mechanism if disabling BBML2 dynaically per-mm. If there was already a BBML2 operation in flight would want to wait for it to end. But that's a problem to solve if/when it's shown to be needed, I think.
Quoted 5 lines
Kernel mappings are never made into a SVA so are not effected. Userspace only.
Jason
Re: [PATCH v2 4/4] iommu/arm: Add BBM Level 2 smmu feature
Quoted 10 lines · Ryan Roberts
Quoted 9 lines · Jason Gunthorpe
Quoted 3 lines · Mikołaj Lenczewski
For example, if we use BBML2 for kernel mappings, does that require us to repaint all kernel mappings when disabling BBML2 on smmu attach? I am not sure, but definitely something to be worked out.
No, it would be a per-mm_struct basis only if we did something like that
Quoted 7 lines
Quoted 2 lines
When the SMMU driver puts a SVA on top of the
mm_structit would disable BBML2 usage only for thatmm_structand it's contained VMAs.I guess we would need to figure out some synchonization mechanism if disabling BBML2 dynaically per-mm. If there was already a BBML2 operation in flight would want to wait for it to end. But that's a problem to solve if/when it's shown to be needed, I think.
I have a feeling we can piggyback on the mmu notifiers to achieve this as all the changes to the PTEs should be bracketed by notifier callbacks..
Let's hope it isn't needed.
Jasob
Robin Murphy
reviewerRe: [PATCH v2 4/4] iommu/arm: Add BBM Level 2 smmu feature
Quoted 19 lines · Jason Gunthorpe
Quoted 10 lines · Ryan Roberts
Quoted 9 lines · Jason Gunthorpe
Quoted 3 lines · Mikołaj Lenczewski
For example, if we use BBML2 for kernel mappings, does that require us to repaint all kernel mappings when disabling BBML2 on smmu attach? I am not sure, but definitely something to be worked out.
No, it would be a per-mm_struct basis only if we did something like that
Quoted 7 lines
Quoted 2 lines
When the SMMU driver puts a SVA on top of the
mm_structit would disable BBML2 usage only for thatmm_structand it's contained VMAs.I guess we would need to figure out some synchonization mechanism if disabling BBML2 dynaically per-mm. If there was already a BBML2 operation in flight would want to wait for it to end. But that's a problem to solve if/when it's shown to be needed, I think.
Quoted 5 lines
I have a feeling we can piggyback on the mmu notifiers to achieve this as all the changes to the PTEs should be bracketed by notifier callbacks..
Let's hope it isn't needed.
Yup, as mentioned previously, this is largely theoretical and at worst only a risk of affecting 3rd-party SMMU implementations. Arm's implementations from MMU-700 onwards do support BBML2; MMU-600 *might* actually be OK as well, but it predates the definition of the feature, and there are more practical reasons not to integrate a decade-old SMMU design with brand new CPUs anyway.
Thanks,
Robin.
Re: [PATCH v2 4/4] iommu/arm: Add BBM Level 2 smmu feature
In reply to Mikołaj LenczewskiQuoted 30 lines · Mikołaj Lenczewski
For supporting BBM Level 2 for userspace mappings, we want to ensure that the smmu also supports its own version of BBM Level 2. Luckily, the smmu spec (IHI 0070G 3.21.1.3) is stricter than the aarch64 spec (DDI 0487K.a D8.16.2), so already guarantees that no aborts are raised when BBM level 2 is claimed.
Add the feature and testing for it under
arm_smmu_sva_supported().Signed-off-by Mikołaj Lenczewski <[email protected]>--- arch/arm64/kernel/cpufeature.c | 7 +++---- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 3 +++ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 3 +++ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 4 ++++ 4 files changed, 13 insertions(+), 4 deletions(-)arch/arm64/kernel/cpufeature.c +0
-2@@ -2223,8 +2223,6 @@ static bool has_bbml2_noabort(const struct arm64_cpu_capabilities *caps, int scoif (!cpu_has_bbml2_noabort (__cpu_read_midr (cpu)))return false; } - -return true; }else if (scope &SCOPE_LOCAL_CPU ) {/* We are a hot-plugged CPU, so only need to check our MIDR. * If we have the correct MIDR, but the kernel booted on an
arch/arm64/kernel/cpufeature.c static bool has_bbml2_noabort(const struct arm64_cpu_capabilities *caps, int sco · line 2237
Quoted 10 lines
arch/arm64/kernel/cpufeature.c +3
-2@@ -2232,10 +2230,11 @@ static bool has_bbml2_noabort(const struct arm64_cpu_capabilities *caps, int sco* we have an incorrect MIDR, but the kernel booted on a * sufficient CPU, we will not bring up this CPU. */ -return cpu_has_bbml2_noabort (read_cpuid_id ()); +if (!cpu_has_bbml2_noabort (read_cpuid_id ())) +return false; } -return false; +return has_cpuid_feature (caps, scope);
Do we really need this? IIRC, it means the MIDR has to be in the allow list *AND* MMFR2 register has to be set too. AmpereOne doesn't have MMFR2 register set.
Thanks,
Yang
Quoted 53 lines
arch/arm64/kernel/cpufeature.c +0
-0}#ifdef CONFIG_ARM64_PAN drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +3
-0@@ -222,6 +222,9 @@ bool arm_smmu_sva_supported(struct arm_smmu_device *smmu) feat_mask |=ARM_SMMU_FEAT_VAX ; } +if (system_supports_bbml2_noabort ()) + feat_mask |=ARM_SMMU_FEAT_BBML2 ; +if ((smmu->features & feat_mask) != feat_mask)return false;drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +3
-0@@ -4406,6 +4406,9 @@ static int arm_smmu_device_hw_probe(struct arm_smmu_device *smmu)if (FIELD_GET (IDR3_RIL , reg)) smmu->features |=ARM_SMMU_FEAT_RANGE_INV ; +if (FIELD_GET (IDR3_BBML , reg) ==IDR3_BBML2 ) + smmu->features |=ARM_SMMU_FEAT_BBML2 ; +/* IDR5 */ reg =readl_relaxed (smmu->base +ARM_SMMU_IDR5 );drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +4
-0@@ -60,6 +60,9 @@ struct arm_smmu_device;#define ARM_SMMU_IDR3 0xc#define IDR3_FWB (1 << 8)#define IDR3_RIL (1 << 10) +#define IDR3_BBML GENMASK(12, 11) +#define IDR3_BBML1 (1 << 11) +#define IDR3_BBML2 (2 << 11)#define ARM_SMMU_IDR5 0x14#define IDR5_STALL_MAX GENMASK(31, 16)@@ -754,6 +757,7 @@ struct arm_smmu_device {#define ARM_SMMU_FEAT_HA (1 << 21)#define ARM_SMMU_FEAT_HD (1 << 22)#define ARM_SMMU_FEAT_S2FWB (1 << 23) +#define ARM_SMMU_FEAT_BBML2 (1 << 24)u32 features;#define ARM_SMMU_OPT_SKIP_PREFETCH (1 << 0)
Re: [PATCH v2 4/4] iommu/arm: Add BBM Level 2 smmu feature
Quoted 40 lines · Yang Shi
Quoted 36 lines · Mikołaj Lenczewski
For supporting BBM Level 2 for userspace mappings, we want to ensure that the smmu also supports its own version of BBM Level 2. Luckily, the smmu spec (IHI 0070G 3.21.1.3) is stricter than the aarch64 spec (DDI 0487K.a D8.16.2), so already guarantees that no aborts are raised when BBM level 2 is claimed.
Add the feature and testing for it under
arm_smmu_sva_supported().Signed-off-by Mikołaj Lenczewski <[email protected]>---
arch/arm64/kernel/cpufeature.c | 7 +++----drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c| 3 +++ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 3 +++ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 4 ++++4 files changed, 13 insertions(+), 4 deletions(-)arch/arm64/kernel/cpufeature.c +0
-0@@ -2223,8 +2223,6 @@ static bool has_bbml2_noabort(const struct
arm64_cpu_capabilities*caps, int sco
if (!cpu_has_bbml2_noabort(__cpu_read_midr(cpu))) return false;
}
-
- return true;
} else if (scope &SCOPE_LOCAL_CPU) {
/* We are a hot-plugged CPU, so only need to check our MIDR.
* If we have the correct MIDR, but the kernel booted on anarch/arm64/kernel/cpufeature.c +0
-0@@ -2232,10 +2230,11 @@ static bool has_bbml2_noabort(const struct
arm64_cpu_capabilities*caps, int sco
* we have an incorrect MIDR, but the kernel booted on a
* sufficient CPU, we will not bring up this CPU. */
- return cpu_has_bbml2_noabort(read_cpuid_id());
Quoted 8 lines
Quoted 5 lines
+ if (!cpu_has_bbml2_noabort(read_cpuid_id())) + return false;
}
- return false;
+ returnhas_cpuid_feature(caps, scope);Do we really need this? IIRC, it means the MIDR has to be in the allow list *AND* MMFR2 register has to be set too. AmpereOne doesn't have MMFR2 register set.
Miko, I think this should have been squashed into patch #1? It doesn't belong in this patch.
Yang, we discussed this internally and decided that we thought it was best to still require BBML2 being advertised in the feature register. That way if trying to use KVM to emulate a CPU that is in the allow list but doesn't really support BBML2, we won't try to use it.
But we still end up with the same problem if running on a physical CPU that supports BBML2 with conflict aborts, but emulating a CPU in the allow list. So given AmpereOne doesn't advertise BBML2 but does support it, I'd be happy to remove this check.
Thanks,
Ryan
Quoted 55 lines
Thanks,
YangQuoted 50 lines
}
#ifdefCONFIG_ARM64_PANdrivers/iommu/ renamed from drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +0
-0
arm/arm-smmu-v3/arm-smmu-v3-sva.c
index9ba596430e7c..6ba182572788100644drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +2
-0@@ -222,6 +222,9 @@ bool arm_smmu_sva_supported(struct arm_smmu_device *smmu) feat_mask |= ARM_SMMU_FEAT_VAX; } + if (system_supports_bbml2_noabort()) + feat_mask |= ARM_SMMU_FEAT_BBML2; + if ((smmu->features & feat_mask) != feat_mask) return false;diff --git
a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.cb/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index358072b4e293..dcee0bdec924100644drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +2
-0@@ -4406,6 +4406,9 @@ static int arm_smmu_device_hw_probe(struct arm_smmu_device *smmu) if (FIELD_GET(IDR3_RIL, reg)) smmu->features |= ARM_SMMU_FEAT_RANGE_INV; + if (FIELD_GET(IDR3_BBML, reg) == IDR3_BBML2) + smmu->features |= ARM_SMMU_FEAT_BBML2; + /* IDR5 */ reg = readl_relaxed(smmu->base + ARM_SMMU_IDR5);diff --git
a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.hb/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
indexbd9d7c85576a..85eaf3ab88c2100644drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +4
-0@@ -60,6 +60,9 @@ struct arm_smmu_device; #define ARM_SMMU_IDR3 0xc #define IDR3_FWB (1 << 8) #define IDR3_RIL (1 << 10) +#define IDR3_BBML GENMASK(12, 11) +#define IDR3_BBML1 (1 << 11) +#define IDR3_BBML2 (2 << 11) #define ARM_SMMU_IDR5 0x14 #define IDR5_STALL_MAX GENMASK(31, 16)@@ -754,6 +757,7 @@ struct arm_smmu_device { #define ARM_SMMU_FEAT_HA (1 << 21) #define ARM_SMMU_FEAT_HD (1 << 22) #define ARM_SMMU_FEAT_S2FWB (1 << 23) +#define ARM_SMMU_FEAT_BBML2 (1 << 24) u32 features; #define ARM_SMMU_OPT_SKIP_PREFETCH (1 << 0)
Re: [PATCH v2 4/4] iommu/arm: Add BBM Level 2 smmu feature
Quoted 50 lines · Ryan Roberts
Quoted 40 lines · Yang Shi
Quoted 36 lines · Mikołaj Lenczewski
For supporting BBM Level 2 for userspace mappings, we want to ensure that the smmu also supports its own version of BBM Level 2. Luckily, the smmu spec (IHI 0070G 3.21.1.3) is stricter than the aarch64 spec (DDI 0487K.a D8.16.2), so already guarantees that no aborts are raised when BBM level 2 is claimed.
Add the feature and testing for it under
arm_smmu_sva_supported().Signed-off-by Mikołaj Lenczewski <[email protected]>---
arch/arm64/kernel/cpufeature.c | 7 +++----drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c| 3 +++ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 3 +++ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 4 ++++4 files changed, 13 insertions(+), 4 deletions(-)arch/arm64/kernel/cpufeature.c +0
-0@@ -2223,8 +2223,6 @@ static bool has_bbml2_noabort(const struct
arm64_cpu_capabilities*caps, int sco
if (!cpu_has_bbml2_noabort(__cpu_read_midr(cpu))) return false;
}
-
- return true;
} else if (scope &SCOPE_LOCAL_CPU) {
/* We are a hot-plugged CPU, so only need to check our MIDR.
* If we have the correct MIDR, but the kernel booted on anarch/arm64/kernel/cpufeature.c +0
-0@@ -2232,10 +2230,11 @@ static bool has_bbml2_noabort(const struct
arm64_cpu_capabilities*caps, int sco
* we have an incorrect MIDR, but the kernel booted on a
* sufficient CPU, we will not bring up this CPU. */
- return cpu_has_bbml2_noabort(read_cpuid_id());Quoted 8 lines
Quoted 5 lines
+ if (!cpu_has_bbml2_noabort(read_cpuid_id())) + return false;
}
- return false;
+ returnhas_cpuid_feature(caps, scope);Do we really need this? IIRC, it means the MIDR has to be in the allow list *AND* MMFR2 register has to be set too. AmpereOne doesn't have MMFR2 register set.
Quoted 2 lines
Miko, I think this should have been squashed into patch #1? It doesn't belong in this patch.
Yes, 100%. Missed this, will put into patch #1.
Re: [PATCH v2 4/4] iommu/arm: Add BBM Level 2 smmu feature
In reply to Ryan RobertsQuoted 55 lines · Ryan Roberts
Quoted 38 lines · Yang Shi
Quoted 35 lines · Mikołaj Lenczewski
For supporting BBM Level 2 for userspace mappings, we want to ensure that the smmu also supports its own version of BBM Level 2. Luckily, the smmu spec (IHI 0070G 3.21.1.3) is stricter than the aarch64 spec (DDI 0487K.a D8.16.2), so already guarantees that no aborts are raised when BBM level 2 is claimed.
Add the feature and testing for it under
arm_smmu_sva_supported().Signed-off-by Mikołaj Lenczewski <[email protected]>--- arch/arm64/kernel/cpufeature.c | 7 +++---- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 3 +++ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 3 +++ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 4 ++++ 4 files changed, 13 insertions(+), 4 deletions(-)arch/arm64/kernel/cpufeature.c +0
-0@@ -2223,8 +2223,6 @@ static bool has_bbml2_noabort(const struct
arm64_cpu_capabilities*caps, int sco
if (!cpu_has_bbml2_noabort(__cpu_read_midr(cpu))) return false;
}arch/arm64/kernel/cpufeature.c +0
-2- - return true; } else if (scope & SCOPE_LOCAL_CPU) { /* We are a hot-plugged CPU, so only need to check our MIDR.* If we have the correct MIDR, but the kernel booted on an @@ -2232,10 +2230,11 @@ static bool has_bbml2_noabort(const struct
arm64_cpu_capabilities*caps, int sco
* we have an incorrect MIDR, but the kernel booted on a
* sufficient CPU, we will not bring up this CPU. */Quoted 8 lines
Quoted 6 lines
- return cpu_has_bbml2_noabort(read_cpuid_id()); + if (!cpu_has_bbml2_noabort(read_cpuid_id())) + return false;
}
- return false;
+ returnhas_cpuid_feature(caps, scope);Do we really need this? IIRC, it means the MIDR has to be in the allow list *AND* MMFR2 register has to be set too. AmpereOne doesn't have MMFR2 register set.
Miko, I think this should have been squashed into patch #1? It doesn't belong in this patch.
Yang, we discussed this internally and decided that we thought it was best to still require BBML2 being advertised in the feature register. That way if trying to use KVM to emulate a CPU that is in the allow list but doesn't really support BBML2, we won't try to use it.
Quoted 4 lines
But we still end up with the same problem if running on a physical CPU that supports BBML2 with conflict aborts, but emulating a CPU in the allow list. So given AmpereOne doesn't advertise BBML2 but does support it, I'd be happy to remove this check.
Thank you.
Yang
Quoted 58 lines
Thanks,
RyanQuoted 53 lines
Thanks,
YangQuoted 50 lines
}
#ifdefCONFIG_ARM64_PANdrivers/iommu/ renamed from drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +0
-0
arm/arm-smmu-v3/arm-smmu-v3-sva.c
index9ba596430e7c..6ba182572788100644drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +2
-0@@ -222,6 +222,9 @@ bool arm_smmu_sva_supported(struct arm_smmu_device *smmu) feat_mask |= ARM_SMMU_FEAT_VAX; } + if (system_supports_bbml2_noabort()) + feat_mask |= ARM_SMMU_FEAT_BBML2; + if ((smmu->features & feat_mask) != feat_mask) return false; diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/
arm/arm-smmu-v3/arm-smmu-v3.c
index358072b4e293..dcee0bdec924100644drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +2
-0@@ -4406,6 +4406,9 @@ static int arm_smmu_device_hw_probe(struct arm_smmu_device *smmu) if (FIELD_GET(IDR3_RIL, reg)) smmu->features |= ARM_SMMU_FEAT_RANGE_INV; + if (FIELD_GET(IDR3_BBML, reg) == IDR3_BBML2) + smmu->features |= ARM_SMMU_FEAT_BBML2; + /* IDR5 */ reg = readl_relaxed(smmu->base + ARM_SMMU_IDR5); diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/
arm/arm-smmu-v3/arm-smmu-v3.h
indexbd9d7c85576a..85eaf3ab88c2100644drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +4
-0@@ -60,6 +60,9 @@ struct arm_smmu_device; #define ARM_SMMU_IDR3 0xc #define IDR3_FWB (1 << 8) #define IDR3_RIL (1 << 10) +#define IDR3_BBML GENMASK(12, 11) +#define IDR3_BBML1 (1 << 11) +#define IDR3_BBML2 (2 << 11) #define ARM_SMMU_IDR5 0x14 #define IDR5_STALL_MAX GENMASK(31, 16)@@ -754,6 +757,7 @@ struct arm_smmu_device { #define ARM_SMMU_FEAT_HA (1 << 21) #define ARM_SMMU_FEAT_HD (1 << 22) #define ARM_SMMU_FEAT_S2FWB (1 << 23) +#define ARM_SMMU_FEAT_BBML2 (1 << 24) u32 features; #define ARM_SMMU_OPT_SKIP_PREFETCH (1 << 0)
Re: [PATCH v2 4/4] iommu/arm: Add BBM Level 2 smmu feature
In reply to Ryan RobertsQuoted 53 lines · Ryan Roberts
Quoted 40 lines · Yang Shi
Quoted 36 lines · Mikołaj Lenczewski
For supporting BBM Level 2 for userspace mappings, we want to ensure that the smmu also supports its own version of BBM Level 2. Luckily, the smmu spec (IHI 0070G 3.21.1.3) is stricter than the aarch64 spec (DDI 0487K.a D8.16.2), so already guarantees that no aborts are raised when BBM level 2 is claimed.
Add the feature and testing for it under
arm_smmu_sva_supported().Signed-off-by Mikołaj Lenczewski <[email protected]>--- arch/arm64/kernel/cpufeature.c | 7 +++---- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 3 +++ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 3 +++ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 4 ++++ 4 files changed, 13 insertions(+), 4 deletions(-)arch/arm64/kernel/cpufeature.c +0
-0@@ -2223,8 +2223,6 @@ static bool has_bbml2_noabort(const struct
arm64_cpu_capabilities*caps, int sco
if (!cpu_has_bbml2_noabort(__cpu_read_midr(cpu))) return false;
}arch/arm64/kernel/cpufeature.c +0
-2- - return true; } else if (scope & SCOPE_LOCAL_CPU) { /* We are a hot-plugged CPU, so only need to check our MIDR.* If we have the correct MIDR, but the kernel booted on an @@ -2232,10 +2230,11 @@ static bool has_bbml2_noabort(const struct
arm64_cpu_capabilities*caps, int sco
* we have an incorrect MIDR, but the kernel booted on a
* sufficient CPU, we will not bring up this CPU. */
- return cpu_has_bbml2_noabort(read_cpuid_id());Quoted 8 lines
Quoted 5 lines
+ if (!cpu_has_bbml2_noabort(read_cpuid_id())) + return false;
}
- return false;
+ returnhas_cpuid_feature(caps, scope);Do we really need this? IIRC, it means the MIDR has to be in the allow list *AND* MMFR2 register has to be set too. AmpereOne doesn't have MMFR2 register set.
Miko, I think this should have been squashed into patch #1? It doesn't belong in this patch.
Quoted 7 lines
Yang, we discussed this internally and decided that we thought it was best to still require BBML2 being advertised in the feature register. That way if trying to use KVM to emulate a CPU that is in the allow list but doesn't really support BBML2, we won't try to use it.
But we still end up with the same problem if running on a physical CPU that supports BBML2 with conflict aborts, but emulating a CPU in the allow list. So
I don't understand the problem here ? In the worst case, if we want to disable the BBML2 feature on a given CPU, we could provide an id- override to reset the value of BBML2. Or provide a kernel parameter to
disable this in case we want to absolutely disable the feature on a
"distro" kernel.
Suzuki
Quoted 63 lines
given AmpereOne doesn't advertise BBML2 but does support it, I'd be happy to remove this check.
Thanks,
RyanQuoted 55 lines
Thanks,
YangQuoted 50 lines
}
#ifdefCONFIG_ARM64_PANdrivers/iommu/ renamed from drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +0
-0
arm/arm-smmu-v3/arm-smmu-v3-sva.c
index9ba596430e7c..6ba182572788100644drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +2
-0@@ -222,6 +222,9 @@ bool arm_smmu_sva_supported(struct arm_smmu_device *smmu) feat_mask |= ARM_SMMU_FEAT_VAX; } + if (system_supports_bbml2_noabort()) + feat_mask |= ARM_SMMU_FEAT_BBML2; + if ((smmu->features & feat_mask) != feat_mask) return false; diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/
arm/arm-smmu-v3/arm-smmu-v3.c
index358072b4e293..dcee0bdec924100644drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +2
-0@@ -4406,6 +4406,9 @@ static int arm_smmu_device_hw_probe(struct arm_smmu_device *smmu) if (FIELD_GET(IDR3_RIL, reg)) smmu->features |= ARM_SMMU_FEAT_RANGE_INV; + if (FIELD_GET(IDR3_BBML, reg) == IDR3_BBML2) + smmu->features |= ARM_SMMU_FEAT_BBML2; + /* IDR5 */ reg = readl_relaxed(smmu->base + ARM_SMMU_IDR5); diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/
arm/arm-smmu-v3/arm-smmu-v3.h
indexbd9d7c85576a..85eaf3ab88c2100644drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +4
-0@@ -60,6 +60,9 @@ struct arm_smmu_device; #define ARM_SMMU_IDR3 0xc #define IDR3_FWB (1 << 8) #define IDR3_RIL (1 << 10) +#define IDR3_BBML GENMASK(12, 11) +#define IDR3_BBML1 (1 << 11) +#define IDR3_BBML2 (2 << 11) #define ARM_SMMU_IDR5 0x14 #define IDR5_STALL_MAX GENMASK(31, 16)@@ -754,6 +757,7 @@ struct arm_smmu_device { #define ARM_SMMU_FEAT_HA (1 << 21) #define ARM_SMMU_FEAT_HD (1 << 22) #define ARM_SMMU_FEAT_S2FWB (1 << 23) +#define ARM_SMMU_FEAT_BBML2 (1 << 24) u32 features; #define ARM_SMMU_OPT_SKIP_PREFETCH (1 << 0)
Re: [PATCH v2 4/4] iommu/arm: Add BBM Level 2 smmu feature
Quoted 60 lines · Suzuki K Poulose
Quoted 59 lines · Ryan Roberts
Quoted 41 lines · Yang Shi
Quoted 37 lines · Mikołaj Lenczewski
For supporting BBM Level 2 for userspace mappings, we want to ensure that the smmu also supports its own version of BBM Level 2. Luckily, the smmu spec (IHI 0070G 3.21.1.3) is stricter than the aarch64 spec (DDI 0487K.a D8.16.2), so already guarantees that no aborts are raised when BBM level 2 is claimed.
Add the feature and testing for it under
arm_smmu_sva_supported().Signed-off-by Mikołaj Lenczewski <[email protected]>---
arch/arm64/kernel/cpufeature.c | 7 +++----drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c| 3 +++ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 3 +++ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 4 ++++4 files changed, 13 insertions(+), 4 deletions(-)arch/arm64/kernel/cpufeature.c +0
-0@@ -2223,8 +2223,6 @@ static bool has_bbml2_noabort(const struct
arm64_cpu_capabilities*caps, int sco
if (!cpu_has_bbml2_noabort(__cpu_read_midr(cpu))) return false;
}
-
- return true;
} else if (scope &SCOPE_LOCAL_CPU) {
/* We are a hot-plugged CPU, so only need to check our MIDR.
* If we have the correct MIDR, but the kernel booted on anarch/arm64/kernel/cpufeature.c +0
-0@@ -2232,10 +2230,11 @@ static bool has_bbml2_noabort(const struct
arm64_cpu_capabilities*caps, int sco
* we have an incorrect MIDR, but the kernel booted on a
* sufficient CPU, we will not bring up this CPU. */
- return cpu_has_bbml2_noabort(read_cpuid_id()); + if (!cpu_has_bbml2_noabort(read_cpuid_id()))Quoted 8 lines
Quoted 4 lines
+ return false;
}
- return false;
+ returnhas_cpuid_feature(caps, scope);Do we really need this? IIRC, it means the MIDR has to be in the allow list *AND* MMFR2 register has to be set too. AmpereOne doesn't have MMFR2 register set.
Miko, I think this should have been squashed into patch #1? It doesn't belong in this patch.
Yang, we discussed this internally and decided that we thought it was best to still require BBML2 being advertised in the feature register. That way if trying to use KVM to emulate a CPU that is in the allow list but doesn't really support BBML2, we won't try to use it.
Quoted 8 lines
Quoted 2 lines
But we still end up with the same problem if running on a physical CPU that supports BBML2 with conflict aborts, but emulating a CPU in the allow list. So
I don't understand the problem here ? In the worst case, if we want to disable the BBML2 feature on a given CPU, we could provide an id- override to reset the value of BBML2. Or provide a kernel parameter to disable this in case we want to absolutely disable the feature on a "distro" kernel.
Hi Suzuki,
Sorry perhaps I'm confusing everyone; As I recall, we had a conversation before Miko posted this series where you were suggesting we should check BOTH that all the CPUs' MIDRs are in the allow list AND that BBML2 is advertised in MMFR2 in order to decide to enable the CPU feature. My understanding was that without the MMFR2 check, you were concerned that in a virtualization scenario, a CPU's MIDR could be overridden to emulate a CPU that is in the allow list, but in reality the CPU does not support BBML2. We would then enable BBML2 and BadThings (TM) will happen. So additionally checking the MMFR2 would solve this.
But Yang is saying that he plans to add the AmpereOne to the allow list because it does support BBML2+NOCONFLICT semantics and we want to benefit from that. But AmpereOne does not advertise BBML2 in it's MMFR2. So with the current approach, adding AmpereOne to the allow list is not sufficient to enable the feature.
But back to your original justification for checking the MMFR2; I don't think that really solves the problem in general, because we don't just require BBML2, we require BBML2+NOCONFLICT. And we can only determine that from the MIDR. So why bother checking MMFR2?
I guess we could provide an id-override on the kernel command line to *enable* BBML2 for AmpereOne, but that's not going to be suitable for mass deployment, I don't think?
Thanks,
Ryan
Quoted 68 lines
Suzuki
Quoted 63 lines
given AmpereOne doesn't advertise BBML2 but does support it, I'd be happy to remove this check.
Thanks,
RyanQuoted 55 lines
Thanks,
YangQuoted 50 lines
}
#ifdefCONFIG_ARM64_PANdrivers/iommu/ renamed from drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +0
-0
arm/arm-smmu-v3/arm-smmu-v3-sva.c
index9ba596430e7c..6ba182572788100644drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +2
-0@@ -222,6 +222,9 @@ bool arm_smmu_sva_supported(struct arm_smmu_device *smmu) feat_mask |= ARM_SMMU_FEAT_VAX; } + if (system_supports_bbml2_noabort()) + feat_mask |= ARM_SMMU_FEAT_BBML2; + if ((smmu->features & feat_mask) != feat_mask) return false;diff --git
a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.cb/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index358072b4e293..dcee0bdec924100644drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +2
-0@@ -4406,6 +4406,9 @@ static int arm_smmu_device_hw_probe(struct arm_smmu_device *smmu) if (FIELD_GET(IDR3_RIL, reg)) smmu->features |= ARM_SMMU_FEAT_RANGE_INV; + if (FIELD_GET(IDR3_BBML, reg) == IDR3_BBML2) + smmu->features |= ARM_SMMU_FEAT_BBML2; + /* IDR5 */ reg = readl_relaxed(smmu->base + ARM_SMMU_IDR5);diff --git
a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.hb/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
indexbd9d7c85576a..85eaf3ab88c2100644drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +4
-0@@ -60,6 +60,9 @@ struct arm_smmu_device; #define ARM_SMMU_IDR3 0xc #define IDR3_FWB (1 << 8) #define IDR3_RIL (1 << 10) +#define IDR3_BBML GENMASK(12, 11) +#define IDR3_BBML1 (1 << 11) +#define IDR3_BBML2 (2 << 11) #define ARM_SMMU_IDR5 0x14 #define IDR5_STALL_MAX GENMASK(31, 16)@@ -754,6 +757,7 @@ struct arm_smmu_device { #define ARM_SMMU_FEAT_HA (1 << 21) #define ARM_SMMU_FEAT_HD (1 << 22) #define ARM_SMMU_FEAT_S2FWB (1 << 23) +#define ARM_SMMU_FEAT_BBML2 (1 << 24) u32 features; #define ARM_SMMU_OPT_SKIP_PREFETCH (1 << 0)
Re: [PATCH v2 4/4] iommu/arm: Add BBM Level 2 smmu feature
Quoted 86 lines · Ryan Roberts
Quoted 60 lines · Suzuki K Poulose
Quoted 59 lines · Ryan Roberts
Quoted 41 lines · Yang Shi
Quoted 37 lines · Mikołaj Lenczewski
For supporting BBM Level 2 for userspace mappings, we want to ensure that the smmu also supports its own version of BBM Level 2. Luckily, the smmu spec (IHI 0070G 3.21.1.3) is stricter than the aarch64 spec (DDI 0487K.a D8.16.2), so already guarantees that no aborts are raised when BBM level 2 is claimed.
Add the feature and testing for it under
arm_smmu_sva_supported().Signed-off-by Mikołaj Lenczewski <[email protected]>--- arch/arm64/kernel/cpufeature.c | 7 +++---- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 3 +++ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 3 +++ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 4 ++++ 4 files changed, 13 insertions(+), 4 deletions(-)arch/arm64/kernel/cpufeature.c +0
-0@@ -2223,8 +2223,6 @@ static bool has_bbml2_noabort(const struct
arm64_cpu_capabilities*caps, int sco
if (!cpu_has_bbml2_noabort(__cpu_read_midr(cpu))) return false;
}arch/arm64/kernel/cpufeature.c +0
-2- - return true; } else if (scope & SCOPE_LOCAL_CPU) { /* We are a hot-plugged CPU, so only need to check our MIDR.* If we have the correct MIDR, but the kernel booted on an @@ -2232,10 +2230,11 @@ static bool has_bbml2_noabort(const struct
arm64_cpu_capabilities*caps, int sco
* we have an incorrect MIDR, but the kernel booted on a
* sufficient CPU, we will not bring up this CPU. */
- return cpu_has_bbml2_noabort(read_cpuid_id());
+ if (!cpu_has_bbml2_noabort(read_cpuid_id()))Quoted 8 lines
Quoted 4 lines
+ return false;
}
- return false;
+ returnhas_cpuid_feature(caps, scope);Do we really need this? IIRC, it means the MIDR has to be in the allow list *AND* MMFR2 register has to be set too. AmpereOne doesn't have MMFR2 register set.
Miko, I think this should have been squashed into patch #1? It doesn't belong in this patch.
Yang, we discussed this internally and decided that we thought it was best to still require BBML2 being advertised in the feature register. That way if trying to use KVM to emulate a CPU that is in the allow list but doesn't really support BBML2, we won't try to use it.
Quoted 8 lines
Quoted 2 lines
But we still end up with the same problem if running on a physical CPU that supports BBML2 with conflict aborts, but emulating a CPU in the allow list. So
I don't understand the problem here ? In the worst case, if we want to disable the BBML2 feature on a given CPU, we could provide an id- override to reset the value of BBML2. Or provide a kernel parameter to disable this in case we want to absolutely disable the feature on a "distro" kernel.
Hi Suzuki,
Sorry perhaps I'm confusing everyone; As I recall, we had a conversation before Miko posted this series where you were suggesting we should check BOTH that all the CPUs' MIDRs are in the allow list AND that BBML2 is advertised in MMFR2 in order to decide to enable the CPU feature. My understanding was that without the MMFR2 check, you were concerned that in a virtualization scenario, a CPU's MIDR could be overridden to emulate a CPU that is in the allow list, but in reality the CPU does not support BBML2. We would then enable BBML2 and BadThings (TM) will happen. So additionally checking the MMFR2 would solve this.
But Yang is saying that he plans to add the AmpereOne to the allow list because it does support BBML2+NOCONFLICT semantics and we want to benefit from that. But AmpereOne does not advertise BBML2 in it's MMFR2. So with the current approach, adding AmpereOne to the allow list is not sufficient to enable the feature.
Quoted 4 lines
But back to your original justification for checking the MMFR2; I don't think that really solves the problem in general, because we don't just require BBML2, we require BBML2+NOCONFLICT. And we can only determine that from the MIDR. So why bother checking MMFR2?
My concerns are not around enabling a CPU, but having a damage control with a "kernel" that a user has no control over (Read, standard distribution kernel).
1. If the combination of the above causes problem (in Virtualization)
2. If the combination of the above is detected to have problems in baremetal.
In (1), VMM could control the ID register and disable the feature.
For (2) we could provide an id-override on command line to disable it in the worst case.
So, having the id register case is a good way to get the system running
with a given kernel (in either world). Without that, we don't have a tunable to control the behavior at runtime.
May be I am being over paranoid about this.
Quoted 3 lines
I guess we could provide an id-override on the kernel command line to *enable* BBML2 for AmpereOne, but that's not going to be suitable for mass deployment, I
Unfortunately, we can't override to provide the "feature" that is missing (at least today).
One option is, run with a whitelist, but provide a kernel parameter to disable bbml2 feature.
Something like, arm64.bbml2=0
So the decision is based on the parameter && MIDR list.
Cheers
Suzuki
Quoted 74 lines
don't think?
Thanks,
RyanQuoted 68 lines
Suzuki
Quoted 63 lines
given AmpereOne doesn't advertise BBML2 but does support it, I'd be happy to remove this check.
Thanks,
RyanQuoted 55 lines
Thanks,
YangQuoted 50 lines
}
#ifdefCONFIG_ARM64_PANdrivers/iommu/ renamed from drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +0
-0
arm/arm-smmu-v3/arm-smmu-v3-sva.c
index9ba596430e7c..6ba182572788100644drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +2
-0@@ -222,6 +222,9 @@ bool arm_smmu_sva_supported(struct arm_smmu_device *smmu) feat_mask |= ARM_SMMU_FEAT_VAX; } + if (system_supports_bbml2_noabort()) + feat_mask |= ARM_SMMU_FEAT_BBML2; + if ((smmu->features & feat_mask) != feat_mask) return false; diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/
arm/arm-smmu-v3/arm-smmu-v3.c
index358072b4e293..dcee0bdec924100644drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +2
-0@@ -4406,6 +4406,9 @@ static int arm_smmu_device_hw_probe(struct arm_smmu_device *smmu) if (FIELD_GET(IDR3_RIL, reg)) smmu->features |= ARM_SMMU_FEAT_RANGE_INV; + if (FIELD_GET(IDR3_BBML, reg) == IDR3_BBML2) + smmu->features |= ARM_SMMU_FEAT_BBML2; + /* IDR5 */ reg = readl_relaxed(smmu->base + ARM_SMMU_IDR5); diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/
arm/arm-smmu-v3/arm-smmu-v3.h
indexbd9d7c85576a..85eaf3ab88c2100644drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +4
-0@@ -60,6 +60,9 @@ struct arm_smmu_device; #define ARM_SMMU_IDR3 0xc #define IDR3_FWB (1 << 8) #define IDR3_RIL (1 << 10) +#define IDR3_BBML GENMASK(12, 11) +#define IDR3_BBML1 (1 << 11) +#define IDR3_BBML2 (2 << 11) #define ARM_SMMU_IDR5 0x14 #define IDR5_STALL_MAX GENMASK(31, 16)@@ -754,6 +757,7 @@ struct arm_smmu_device { #define ARM_SMMU_FEAT_HA (1 << 21) #define ARM_SMMU_FEAT_HD (1 << 22) #define ARM_SMMU_FEAT_S2FWB (1 << 23) +#define ARM_SMMU_FEAT_BBML2 (1 << 24) u32 features; #define ARM_SMMU_OPT_SKIP_PREFETCH (1 << 0)
Re: [PATCH v2 4/4] iommu/arm: Add BBM Level 2 smmu feature
Quoted 90 lines · Suzuki K Poulose
Quoted 89 lines · Ryan Roberts
Quoted 63 lines · Suzuki K Poulose
Quoted 62 lines · Ryan Roberts
Quoted 41 lines · Yang Shi
Quoted 37 lines · Mikołaj Lenczewski
For supporting BBM Level 2 for userspace mappings, we want to ensure that the smmu also supports its own version of BBM Level 2. Luckily, the smmu spec (IHI 0070G 3.21.1.3) is stricter than the aarch64 spec (DDI 0487K.a D8.16.2), so already guarantees that no aborts are raised when BBM level 2 is claimed.
Add the feature and testing for it under
arm_smmu_sva_supported().Signed-off-by Mikołaj Lenczewski <[email protected]>---
arch/arm64/kernel/cpufeature.c | 7 +++----drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c| 3 +++ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 3 +++ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 4 ++++4 files changed, 13 insertions(+), 4 deletions(-)arch/arm64/kernel/cpufeature.c +0
-0@@ -2223,8 +2223,6 @@ static bool has_bbml2_noabort(const struct
arm64_cpu_capabilities*caps, int sco
if (!cpu_has_bbml2_noabort(__cpu_read_midr(cpu))) return false;
}
-
- return true;
} else if (scope &SCOPE_LOCAL_CPU) {
/* We are a hot-plugged CPU, so only need to check our MIDR.
* If we have the correct MIDR, but the kernel booted on anarch/arm64/kernel/cpufeature.c +0
-0@@ -2232,10 +2230,11 @@ static bool has_bbml2_noabort(const struct
arm64_cpu_capabilities*caps, int sco
* we have an incorrect MIDR, but the kernel booted on a
* sufficient CPU, we will not bring up this CPU. */
- return cpu_has_bbml2_noabort(read_cpuid_id());
+ if (!cpu_has_bbml2_noabort(read_cpuid_id()))Quoted 8 lines
Quoted 4 lines
+ return false;
}
- return false;
+ returnhas_cpuid_feature(caps, scope);Do we really need this? IIRC, it means the MIDR has to be in the allow list *AND* MMFR2 register has to be set too. AmpereOne doesn't have MMFR2 register set.
Miko, I think this should have been squashed into patch #1? It doesn't belong in
this patch.Yang, we discussed this internally and decided that we thought it was best to still require BBML2 being advertised in the feature register. That way if trying
to use KVM to emulate a CPU that is in the allow list but doesn't really support
BBML2, we won't try to use it.Quoted 8 lines
Quoted 2 lines
But we still end up with the same problem if running on a physical CPU that supports BBML2 with conflict aborts, but emulating a CPU in the allow list. So
I don't understand the problem here ? In the worst case, if we want to disable the BBML2 feature on a given CPU, we could provide an id- override to reset the value of BBML2. Or provide a kernel parameter to disable this in case we want to absolutely disable the feature on a "distro" kernel.
Hi Suzuki,
Sorry perhaps I'm confusing everyone; As I recall, we had a conversation before Miko posted this series where you were suggesting we should check BOTH that all the CPUs' MIDRs are in the allow list AND that BBML2 is advertised in MMFR2 in order to decide to enable the CPU feature. My understanding was that without the MMFR2 check, you were concerned that in a virtualization scenario, a CPU's MIDR could be overridden to emulate a CPU that is in the allow list, but in reality the CPU does not support BBML2. We would then enable BBML2 and BadThings (TM) will happen. So additionally checking the MMFR2 would solve this.
But Yang is saying that he plans to add the AmpereOne to the allow list because it does support BBML2+NOCONFLICT semantics and we want to benefit from that. But AmpereOne does not advertise BBML2 in it's MMFR2. So with the current approach, adding AmpereOne to the allow list is not sufficient to enable the feature.
Quoted 7 lines
Quoted 4 lines
But back to your original justification for checking the MMFR2; I don't think that really solves the problem in general, because we don't just require BBML2, we require BBML2+NOCONFLICT. And we can only determine that from the MIDR. So why bother checking MMFR2?
My concerns are not around enabling a CPU, but having a damage control with a "kernel" that a user has no control over (Read, standard distribution kernel).
Ahh I misunderstood then.
Quoted 19 lines
1. If the combination of the above causes problem (in Virtualization)
2. If the combination of the above is detected to have problems in baremetal.In (1), VMM could control the ID register and disable the feature.
For (2) we could provide an id-override on command line to disable it in the worst case.
So, having the id register case is a good way to get the system running with a given kernel (in either world). Without that, we don't have a tunable to control the behavior at runtime.
May be I am being over paranoid about this.
Quoted 3 lines
I guess we could provide an id-override on the kernel command line to *enable* BBML2 for AmpereOne, but that's not going to be suitable for mass deployment, I
Quoted 8 lines
Unfortunately, we can't override to provide the "feature" that is missing (at least today).
One option is, run with a whitelist, but provide a kernel parameter to disable bbml2 feature.
Something like, arm64.bbml2=0
Quoted 1 line
So the decision is based on the parameter && MIDR list.
This sounds like a good solution to me. I guess we would wire this up to the SW features "register".
Thanks,
Ryan
Quoted 80 lines
Cheers
SuzukiQuoted 74 lines
don't think?
Thanks,
RyanQuoted 68 lines
Suzuki
Quoted 63 lines
given AmpereOne doesn't advertise BBML2 but does support it, I'd be happy to remove this check.
Thanks,
RyanQuoted 55 lines
Thanks,
YangQuoted 50 lines
}
#ifdefCONFIG_ARM64_PANdrivers/iommu/ renamed from drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +0
-0
arm/arm-smmu-v3/arm-smmu-v3-sva.c
index9ba596430e7c..6ba182572788100644drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +2
-0@@ -222,6 +222,9 @@ bool arm_smmu_sva_supported(struct arm_smmu_device *smmu) feat_mask |= ARM_SMMU_FEAT_VAX; } + if (system_supports_bbml2_noabort()) + feat_mask |= ARM_SMMU_FEAT_BBML2; + if ((smmu->features & feat_mask) != feat_mask) return false;diff --git
a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.cb/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index358072b4e293..dcee0bdec924100644drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +2
-0@@ -4406,6 +4406,9 @@ static int arm_smmu_device_hw_probe(struct arm_smmu_device *smmu) if (FIELD_GET(IDR3_RIL, reg)) smmu->features |= ARM_SMMU_FEAT_RANGE_INV; + if (FIELD_GET(IDR3_BBML, reg) == IDR3_BBML2) + smmu->features |= ARM_SMMU_FEAT_BBML2; + /* IDR5 */ reg = readl_relaxed(smmu->base + ARM_SMMU_IDR5);diff --git
a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.hb/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
indexbd9d7c85576a..85eaf3ab88c2100644drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +4
-0@@ -60,6 +60,9 @@ struct arm_smmu_device; #define ARM_SMMU_IDR3 0xc #define IDR3_FWB (1 << 8) #define IDR3_RIL (1 << 10) +#define IDR3_BBML GENMASK(12, 11) +#define IDR3_BBML1 (1 << 11) +#define IDR3_BBML2 (2 << 11) #define ARM_SMMU_IDR5 0x14 #define IDR5_STALL_MAX GENMASK(31, 16)@@ -754,6 +757,7 @@ struct arm_smmu_device { #define ARM_SMMU_FEAT_HA (1 << 21) #define ARM_SMMU_FEAT_HD (1 << 22) #define ARM_SMMU_FEAT_S2FWB (1 << 23) +#define ARM_SMMU_FEAT_BBML2 (1 << 24) u32 features; #define ARM_SMMU_OPT_SKIP_PREFETCH (1 << 0)
Quoted 11 lines · Mikołaj Lenczewski
Hi All,
This patch series adds adding initial support for eliding break-before-make requirements on systems that support BBML2 and additionally guarantee to never raise a conflict abort.
This support reorders and optionally elides a TLB invalidation in
contpte_convert(). The elision of said invalidation leads to a 12% improvement when executing a microbenchmark designed to force the pathological path wherecontpte_convert()gets called. This represents an 80% reduction in the cost of callingcontpte_convert().
Just a general comment: Nice! :)
Signature · 3 lines
Cheers, David / dhildenb