Initial BBML2 support for contpte_convert()
Subsystems (5)
v6 cover Initial BBML2 support for contpte_convert()
Hi All,
This patch series adds initial support for eliding Break-Before-Make requirements on systems that support BBML2 and additionally guarantee to never raise a conflict abort.
This support reorders and elides both a TLB invalidation and a DSB in contpte_convert(), when BBML2 is supported. This leads to a 12% improvement when executing a microbenchmark designed to force the pathological path where contpte_convert() gets called. This represents an 80% reduction in the cost of calling contpte_convert().
We clarify both the correctness and performance benefits of this elision with respect to the relevant Arm ARM passages, via substantial comments in the contpte_convert() source.
This series is based on v6.15-rc3 (9c32cda43eb7).
Notes
======
Patch 1 implements an allow-list of cpus that support BBML2, but with the additional constraint of never causing TLB conflict aborts. We settled on this constraint because we will use the feature for kernel mappings in the future, for which we cannot handle conflict aborts safely.
Yang Shi has a series at [1] that aims to use BBML2 to enable splitting the linear map at runtime. This series partially overlaps with it to add the cpu feature. We believe this series is fully compatible with Yang's requirements and could go first.
Due to constraints with the current design of the cpufeature framework and the fact that our has_bbml2_noabort() check relies on both a MIDR allowlist and the exposed MMFR2 register value, if an implementation supports our desired BBML2+NOABORT semantics but fails to declare support for BBML2 via the id_aa64mmfr2.bbm field, the check will fail.
Not declaring base support for BBML2 when supporting BBML2+NOABORT should be considered an erratum [2], and a workaround can be applied in __cpuinfo_store_cpu() to patch in support for BBML2 for the sanitised register view used by SCOPE_SYSTEM. However, SCOPE_LOCAL_CPU bypasses this sanitised view and reads the MSRs directly by design, and so an additional workaround can be applied in __read_sysreg_by_encoding() for the MMFR2 case.
For situations where support for BBML2+NOABORT is claimed by an implementor and subsequently built into the kernel, but problems later arise that require user damage control [3], we introduce a kernel commandline parameter override for disabling all BBML2 support.
[1]:
https://lore.kernel.org/linux-arm-kernel/[email protected]/
[2]:
https://lore.kernel.org/linux-arm-kernel/[email protected]/
[3]:
https://lore.kernel.org/all/[email protected]/
Changelog
=========
v6:
- clarify correctness and performance of elision of __tlb_flush_range()
- rebase onto v6.15-rc3
v5:
- https://lore.kernel.org/all/[email protected]/
- fixup coding style nits
- document motivation for kernel commandline parameter
v4:
- https://lore.kernel.org/all/[email protected]/
- rebase onto v6.14-rc5
- switch from arm64 sw feature override to hw feature override
- reintroduce has_cpuid_feature() check in addition to MIDR check
v3:
- https://lore.kernel.org/all/[email protected]/
- rebase onto v6.14-rc4
- add arm64.nobbml2 commandline override
- squash "delay tlbi" and "elide tlbi" patches
v2:
- https://lore.kernel.org/all/[email protected]/
- fix buggy MIDR check to properly account for all boot+late cpus
- add smmu bbml2 feature check
v1:
- https://lore.kernel.org/all/[email protected]/
- rebase onto v6.14-rc3
- remove kvm bugfix patches from series
- strip out conflict abort handler code
- switch from blocklist to allowlist of bmml2+noabort implementations
- remove has_cpuid_feature() in favour of MIDR check
rfc-v1:
- https://lore.kernel.org/all/[email protected]/
- https://lore.kernel.org/all/[email protected]/
Mikołaj Lenczewski (3):
arm64: Add BBM Level 2 cpu feature iommu/arm: Add BBM Level 2 smmu feature arm64/mm: Elide tlbi in contpte_convert() under BBML2
.../admin-guide/kernel-parameters.txt | 3 + arch/arm64/Kconfig | 19 +++ arch/arm64/include/asm/cpucaps.h | 2 + arch/arm64/include/asm/cpufeature.h | 5 + arch/arm64/kernel/cpufeature.c | 71 +++++++++ arch/arm64/kernel/pi/idreg-override.c | 2 + arch/arm64/mm/contpte.c | 139 +++++++++++++++++- arch/arm64/tools/cpucaps | 1 + .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 3 + drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 3 + drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 4 + 11 files changed, 251 insertions(+), 1 deletion(-)
Signature · 1 line
2.49.0
PATCH v6 1/3 arm64: Add BBM Level 2 cpu feature
The Break-Before-Make cpu feature supports multiple levels (levels 0-2), and this commit adds a dedicated BBML2 cpufeature to test against support for, as well as a kernel commandline parameter to optionally disable BBML2 altogether.
This is a system feature as we might have a big.LITTLE architecture where some cores support BBML2 and some don't, but we want all cores to be available and BBM to default to level 0 (as opposed to having cores without BBML2 not coming online).
To support BBML2 in as wide a range of contexts as we can, we want not only the architectural guarantees that BBML2 makes, but additionally want BBML2 to not create TLB conflict aborts. Not causing aborts avoids us having to prove that no recursive faults can be induced in any path that uses BBML2, allowing its use for arbitrary kernel mappings. Support detection of such CPUs.
--- .../admin-guide/kernel-parameters.txt | 3 + arch/arm64/Kconfig | 19 +++++ arch/arm64/include/asm/cpucaps.h | 2 + arch/arm64/include/asm/cpufeature.h | 5 ++ arch/arm64/kernel/cpufeature.c | 71 +++++++++++++++++++ arch/arm64/kernel/pi/idreg-override.c | 2 + arch/arm64/tools/cpucaps | 1 + 7 files changed, 103 insertions(+)
Documentation/admin-guide/kernel-parameters.txt +3 -0
@@ -449,6 +449,9 @@ arm64.no32bit_el0 [ARM64] Unconditionally disable the execution of 32 bit applications. + arm64.nobbml2 [ARM64] Unconditionally disable Break-Before-Make Level + 2 support + arm64.nobti [ARM64] Unconditionally disable Branch Target Identification support
arch/arm64/Kconfig +19 -0
@@ -2070,6 +2070,25 @@ config ARM64_TLB_RANGE The feature introduces new assembly instructions, and they were support when binutils >= 2.30. +config ARM64_BBML2_NOABORT + bool "Enable support for Break-Before-Make Level 2 detection and usage" + default y + help + FEAT_BBM provides detection of support levels for break-before-make + sequences. If BBM level 2 is supported, some TLB maintenance requirements + can be relaxed to improve performance. We additonally require the + property that the implementation cannot ever raise TLB Conflict Aborts. + Selecting N causes the kernel to fallback to BBM level 0 behaviour + even if the system supports BBM level 2. + + To enable detection of BBML2 support, and to make use of it, say Y. + + Detection of and support for BBM level 2 can optionally be overridden + at runtime via the use of the arm64.nobbml2 kernel commandline + parameter. If your system claims support for BBML2, but is unstable + with this option enabled, either say N or make use of the commandline + parameter override to force BBML0. + endmenu # "ARMv8.4 architectural features" menu "ARMv8.5 architectural features"
arch/arm64/include/asm/cpucaps.h +2 -0
@@ -23,6 +23,8 @@ cpucap_is_possible(const unsigned int cap)return IS_ENABLED (CONFIG_ARM64_PAN );case ARM64_HAS_EPAN :return IS_ENABLED (CONFIG_ARM64_EPAN ); +case ARM64_HAS_BBML2_NOABORT : +return IS_ENABLED (CONFIG_ARM64_BBML2_NOABORT );case ARM64_SVE :return IS_ENABLED (CONFIG_ARM64_SVE );case ARM64_SME :
arch/arm64/include/asm/cpufeature.h +5 -0
@@ -848,6 +848,11 @@ static inline bool system_supports_pmuv3(void)return cpus_have_final_cap (ARM64_HAS_PMUV3 ); } +static inline bool system_supports_bbml2_noabort (void ) +{ +return alternative_has_cap_unlikely (ARM64_HAS_BBML2_NOABORT ); +} +int do_emulate_mrs (struct pt_regs *regs,u32 sys_reg,u32 rt);bool try_emulate_mrs (struct pt_regs *regs,u32 isn);
arch/arm64/kernel/cpufeature.c +71 -0
83 lines not on this page · raw message
arch/arm64/kernel/pi/idreg-override.c +2 -0
@@ -102,6 +102,7 @@ static const struct ftr_set_desc mmfr2 __prel64_initconst = { .override = &id_aa64mmfr2_override, .fields = {FIELD ("varange" , ID_AA64MMFR2_EL1_VARange_SHIFT, mmfr2_varange_filter), + FIELD("bbm", ID_AA64MMFR2_EL1_BBM_SHIFT, NULL), {} }, };
@@ -246,6 +247,7 @@ static const struct { { "rodata=off", "arm64_sw.rodataoff=1" }, { "arm64.nolva", "id_aa64mmfr2.varange=0" }, { "arm64.no32bit_el0", "id_aa64pfr0.el0=1" }, + { "arm64.nobbml2", "id_aa64mmfr2.bbm=0" }, };static int __init parse_hexdigit (const char *p,u64 *v)
arch/arm64/tools/cpucaps +1 -0
@@ -14,6 +14,7 @@ HAS_ADDRESS_AUTH_ARCH_QARMA5 HAS_ADDRESS_AUTH_IMP_DEF HAS_AMU_EXTN HAS_ARMv8_4_TTL +HAS_BBML2_NOABORT HAS_CACHE_DIC HAS_CACHE_IDC HAS_CNP
Signature · 1 line
2.49.0
PATCH v6 2/3 iommu/arm: Add BBM Level 2 smmu feature
In reply to Mikołaj LenczewskiFor supporting BBM Level 2 for userspace mappings, we want to ensure that the smmu also supports its own version of BBM Level 2. Luckily, the smmu spec (IHI 0070G 3.21.1.3) is stricter than the aarch64 spec (DDI 0487K.a D8.16.2), so already guarantees that no aborts are raised when BBM level 2 is claimed.
Add the feature and testing for it under arm_smmu_sva_supported().
--- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 3 +++ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 3 +++ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 4 ++++ 3 files changed, 10 insertions(+)
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +3 -0
@@ -222,6 +222,9 @@ bool arm_smmu_sva_supported(struct arm_smmu_device *smmu) feat_mask |=ARM_SMMU_FEAT_VAX ; } +if (system_supports_bbml2_noabort ()) + feat_mask |=ARM_SMMU_FEAT_BBML2 ; +if ((smmu->features & feat_mask) != feat_mask)return false;
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +3 -0
@@ -4430,6 +4430,9 @@ static int arm_smmu_device_hw_probe(struct arm_smmu_device *smmu)if (FIELD_GET (IDR3_RIL , reg)) smmu->features |=ARM_SMMU_FEAT_RANGE_INV ; +if (FIELD_GET (IDR3_BBML , reg) ==IDR3_BBML2 ) + smmu->features |=ARM_SMMU_FEAT_BBML2 ; +/* IDR5 */ reg =readl_relaxed (smmu->base +ARM_SMMU_IDR5 );
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +4 -0
@@ -60,6 +60,9 @@ struct arm_smmu_device;#define ARM_SMMU_IDR3 0xc#define IDR3_FWB (1 << 8)#define IDR3_RIL (1 << 10) +#define IDR3_BBML GENMASK(12, 11) +#define IDR3_BBML1 (1 << 11) +#define IDR3_BBML2 (2 << 11)#define ARM_SMMU_IDR5 0x14#define IDR5_STALL_MAX GENMASK(31, 16)
@@ -755,6 +758,7 @@ struct arm_smmu_device {#define ARM_SMMU_FEAT_HA (1 << 21)#define ARM_SMMU_FEAT_HD (1 << 22)#define ARM_SMMU_FEAT_S2FWB (1 << 23) +#define ARM_SMMU_FEAT_BBML2 (1 << 24)u32 features;#define ARM_SMMU_OPT_SKIP_PREFETCH (1 << 0)
Signature · 1 line
2.49.0
PATCH v6 3/3 arm64/mm: Elide tlbi in contpte_convert() under BBML2
In reply to Mikołaj LenczewskiWhen converting a region via contpte_convert() to use mTHP, we have two different goals. We have to mark each entry as contiguous, and we would like to smear the dirty and young (access) bits across all entries in the contiguous block. Currently, we do this by first accumulating the dirty and young bits in the block, using an atomic __ptep_get_and_clear() and the relevant pte_{dirty,young}() calls, performing a tlbi, and finally smearing the correct bits across the block using __set_ptes().
This approach works fine for BBM level 0, but with support for BBM level 2 we are allowed to reorder the tlbi to after setting the pagetable entries. We expect the time cost of a tlbi to be much greater than the cost of clearing and resetting the PTEs. As such, this reordering of the tlbi outside the window where our PTEs are invalid greatly reduces the duration the PTE are visibly invalid for other threads. This reduces the likelyhood of a concurrent page walk finding an invalid PTE, reducing the likelyhood of a fault in other threads, and improving performance (more so when there are more threads).
Because we support via allowlist only bbml2 implementations that never raise conflict aborts and instead invalidate the tlb entries automatically in hardware, we can avoid the final flush altogether.
However, avoiding the intermediate tlbi+dsb must be carefully considered to ensure that we remain both correct and performant. We document our reasoning and the expected interactions further in the contpte_convert() source. To do so we rely on the aarch64 spec (DDI 0487L.a D8.7.1.1) requirements RNGLXZ and RJQQTC to provide guarantees that the elision is correct.
--- arch/arm64/mm/contpte.c | 139 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 138 insertions(+), 1 deletion(-)
arch/arm64/mm/contpte.c +138 -1
@@ -68,7 +68,144 @@ static void contpte_convert(struct mm_struct *mm, unsigned long addr, pte =pte_mkyoung (pte); } -__flush_tlb_range (&vma, start_addr, addr,PAGE_SIZE , true,3 ); +/* +* On eliding the __tlb_flush_range() under BBML2+noabort: +* +* NOTE: Instead of using N=16 as the contiguous block length, we use +* N=4 for clarity. +* +* NOTE: 'n' and 'c' are used to denote the "contiguous bit" being +* unset and set, respectively. +* +* We worry about two cases where contiguous bit is used: +* - When folding N smaller non-contiguous ptes as 1 contiguous block. +* - When unfolding a contiguous block into N smaller non-contiguous ptes. +* +* Currently, the BBML0 folding case looks as follows: +* +* 0) Initial page-table layout: +* +* +----+----+----+----+ +* |RO,n|RO,n|RO,n|RW,n| <--- last page being set as RO +* +----+----+----+----+ +* +* 1) Aggregate AF + dirty flags using __ptep_get_and_clear(): +* +* +----+----+----+----+ +* | 0 | 0 | 0 | 0 | +* +----+----+----+----+ +* +* 2) __flush_tlb_range(): +* +* |____ tlbi + dsb ____| +* +* 3) __set_ptes() to repaint contiguous block: +* +* +----+----+----+----+ +* |RO,c|RO,c|RO,c|RO,c| +* +----+----+----+----+ +* +* 4) The kernel will eventually __flush_tlb() for changed page: +* +* |____| <--- tlbi + dsb +* +* As expected, the intermediate tlbi+dsb ensures that other PEs +* only ever see an invalid (0) entry, or the new contiguous TLB entry. +* The final tlbi+dsb will always throw away the newly installed +* contiguous TLB entry, which is a micro-optimisation opportunity, +* but does not affect correctness. +* +* In the BBML2 case, the change is avoiding the intermediate tlbi+dsb. +* This means a few things, but notably other PEs will still "see" any +* stale cached TLB entries. This could lead to a "contiguous bit +* misprogramming" issue until the final tlbi+dsb of the changed page, +* which would clear out both the stale (RW,n) entry and the new (RO,c) +* contiguous entry installed in its place. +* +* What this is saying, is the following: +* +* +----+----+----+----+ +* |RO,n|RO,n|RO,n|RW,n| <--- old page tables, all non-contiguous +* +----+----+----+----+ +* +* +----+----+----+----+ +* |RO,c|RO,c|RO,c|RO,c| <--- new page tables, all contiguous +* +----+----+----+----+ +* /\ +* || +* +* If both the old single (RW,n) and new contiguous (RO,c) TLB entries +* are present, and a write is made to this address, do we fault or +* is the write permitted (via amalgamation)? +* +* The relevant Arm ARM DDI 0487L.a requirements are RNGLXZ and RJQQTC, +* and together state that when BBML1 or BBML2 are implemented, either +* a TLB conflict abort is raised (which we expressly forbid), or will +* "produce an OA, access permissions, and memory attributes that are +* consistent with any of the programmed translation table values". +* +* That is to say, will either raise a TLB conflict, or produce one of +* the cached TLB entries, but never amalgamate. +* +* Thus, as the page tables are only considered "consistent" after +* the final tlbi+dsb (which evicts both the single stale (RW,n) TLB +* entry as well as the new contiguous (RO,c) TLB entry), omitting the +* initial tlbi+dsb is correct. +* +* It is also important to note that at the end of the BBML2 folding +* case, we are still left with potentially all N TLB entries still +* cached (the N-1 non-contiguous ptes, and the single contiguous +* block). However, over time, natural TLB pressure will cause the +* non-contiguous pte TLB entries to be flushed, leaving only the +* contiguous block TLB entry. This means that omitting the tlbi+dsb is +* not only correct, but also keeps our eventual performance benefits. +* +* For the unfolding case, BBML0 looks as follows: +* +* 0) Initial page-table layout: +* +* +----+----+----+----+ +* |RW,c|RW,c|RW,c|RW,c| <--- last page being set as RO +* +----+----+----+----+ +* +* 1) Aggregate AF + dirty flags using __ptep_get_and_clear(): +* +* +----+----+----+----+ +* | 0 | 0 | 0 | 0 | +* +----+----+----+----+ +* +* 2) __flush_tlb_range(): +* +* |____ tlbi + dsb ____| +* +* 3) __set_ptes() to repaint as non-contiguous: +* +* +----+----+----+----+ +* |RW,n|RW,n|RW,n|RW,n| +* +----+----+----+----+ +* +* 4) Update changed page permissions: +* +* +----+----+----+----+ +* |RW,n|RW,n|RW,n|RO,n| <--- last page permissions set +* +----+----+----+----+ +* +* 5) The kernel will eventually __flush_tlb() for changed page: +* +* |____| <--- tlbi + dsb +* +* For BBML2, we again remove the intermediate tlbi+dsb. Here, there +* are no issues, as the final tlbi+dsb covering the changed page is +* guaranteed to remove the original large contiguous (RW,c) TLB entry, +* as well as the intermediate (RW,n) TLB entry; the next access will +* install the new (RO,n) TLB entry and the page tables are only +* considered "consistent" after the final tlbi+dsb, so software must +* be prepared for this inconsistency prior to finishing the mm dance +* regardless. +*/ + +if (!system_supports_bbml2_noabort ()) +__flush_tlb_range (&vma, start_addr, addr,PAGE_SIZE , true,3 );__set_ptes (mm, start_addr, start_ptep, pte,CONT_PTES ); }
Signature · 1 line
2.49.0
PATCH v6 3/3 arm64/mm: Reorder tlbi in contpte_convert() under BBML2
In reply to Mikołaj LenczewskiWhen converting a region via contpte_convert() to use mTHP, we have two different goals. We have to mark each entry as contiguous, and we would like to smear the dirty and young (access) bits across all entries in the contiguous block. Currently, we do this by first accumulating the dirty and young bits in the block, using an atomic __ptep_get_and_clear() and the relevant pte_{dirty,young}() calls, performing a tlbi, and finally smearing the correct bits across the block using __set_ptes().
This approach works fine for BBM level 0, but with support for BBM level 2 we are allowed to reorder the tlbi to after setting the pagetable entries. We expect the time cost of a tlbi to be much greater than the cost of clearing and resetting the PTEs. As such, this reordering of the tlbi outside the window where our PTEs are invalid greatly reduces the duration the PTE are visibly invalid for other threads. This reduces the likelyhood of a concurrent page walk finding an invalid PTE, reducing the likelyhood of a fault in other threads, and improving performance (more so when there are more threads).
Because we support via allowlist only bbml2 implementations that never raise conflict aborts and instead invalidate the tlb entries automatically in hardware, we could avoid the final flush altogether.
Unfortunately, certain implementations might implement bbml2 in such a way that avoiding the tlbi leads to a performance degradation. Thus, to remain both correct and performant, we simply switch from __flush_tlbi_range() to __flush_tlbi_range_nosync() and are thus guaranteed correctness (via BBML2 semantics) and performance (leaving the actual flush to the next DSB) in all cases.
--- arch/arm64/mm/contpte.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-)
arch/arm64/mm/contpte.c +13 -1
@@ -68,9 +68,21 @@ static void contpte_convert(struct mm_struct *mm, unsigned long addr, pte =pte_mkyoung (pte); } -__flush_tlb_range (&vma, start_addr, addr,PAGE_SIZE , true,3 ); +if (!system_supports_bbml2_noabort ()) +__flush_tlb_range (&vma, start_addr, addr,PAGE_SIZE , true,3 );__set_ptes (mm, start_addr, start_ptep, pte,CONT_PTES ); + +/* +* Despite BBML2 allowing us to elide the following TLBI whilst remaining +* correct, there may be implementations where not issuing said TLBI +* is not performant. To handle this, we can issue a TLBI but delegate +* the flush to the next DSB (worst case, at the next context switch). +* This remains correct (due to BBML2 semantics) and fast (due to not +* waiting for a DSB) in all cases. +*/ +if (system_supports_bbml2_noabort ()) +__flush_tlb_range_nosync (&vma, start_addr, addr,PAGE_SIZE , true,3 ); } void __contpte_try_fold(struct mm_struct *mm, unsigned long addr,
Signature · 1 line
2.49.0
Re: [PATCH v6 3/3] arm64/mm: Reorder tlbi in contpte_convert() under BBML2
Embarassingly, managed to leave around a stale patch when renaming the commit. Please ignore this patch, the correct one is "Elide tlbi in contpte_convert() under BBML2". Am considering resending...
Signature · 2 lines
Kind regards, Mikołaj Lenczewski
Re: [PATCH v6 3/3] arm64/mm: Reorder tlbi in contpte_convert() under BBML2
Quoted 4 lines · Mikołaj Lenczewski
Embarassingly, managed to leave around a stale patch when renaming the commit. Please ignore this patch, the correct one is "Elide tlbi in
contpte_convert()under BBML2". Am considering resending...
I think it would be clearer to resend. Otherwise people are going to get confused.