arm64: Add support for new control bits CTR_EL0.DIC and CTR_EL0.IDC
PATCH v2 1/1 arm64: Add support for new control bits CTR_EL0.DIC and CTR_EL0.IDC
The DCache clean & ICache invalidation requirements for instructions to be data coherence are discoverable through new fields in CTR_EL0. The following two control bits DIC and IDC were defined for this purpose. No need to perform point of unification cache maintenance operations from software on systems where CPU caches are transparent.
This patch optimize the three functions __flush_cache_user_range(), clean_dcache_area_pou() and invalidate_icache_range() if the hardware reports CTR_EL0.IDC and/or CTR_EL0.IDC. Basically it skips the two instructions 'DC CVAU' and 'IC IVAU', and the associated loop logic in order to avoid the unnecessary overhead.
CTR_EL0.DIC: Instruction cache invalidation requirements for
instruction to data coherence. The meaning of this bit[29].
0: Instruction cache invalidation to the point of unification
is required for instruction to data coherence.
1: Instruction cache cleaning to the point of unification is
not required for instruction to data coherence.
CTR_EL0.IDC: Data cache clean requirements for instruction to data
coherence. The meaning of this bit[28].
0: Data cache clean to the point of unification is required for
instruction to data coherence, unless CLIDR_EL1.LoC == 0b000
or (CLIDR_EL1.LoUIS == 0b000 && CLIDR_EL1.LoUU == 0b000). 1: Data cache clean to the point of unification is not required
for instruction to data coherence.
---
Changes since v1:
-Reworded commit text.
-Used the alternatives framework as Catalin suggested. -Rebased on top of https://patchwork.kernel.org/patch/10227927/
arch/arm64/Kconfig | 21 +++++++++++++++++++ arch/arm64/include/asm/cache.h | 5 +++++ arch/arm64/include/asm/cpucaps.h | 4 +++- arch/arm64/kernel/cpufeature.c | 44 ++++++++++++++++++++++++++++++++++------ arch/arm64/mm/cache.S | 15 ++++++++++++++ 5 files changed, 82 insertions(+), 7 deletions(-)
arch/arm64/Kconfig +21 -0
@@ -1095,6 +1095,27 @@ config ARM64_RAS_EXTN and access the new registers if the system supports the extension. Platform RAS features may additionally depend on firmware support. +config ARM64_CACHE_IDC + bool "Enable support for DCache clean PoU optimization" + default y + help + The data cache clean to the point of unification is not required + for instruction to be data coherence if CTR_EL0.IDC has value 1. + + Selecting this feature will allow the kernel to optimize the POU + cache maintaince operations where it requires 'DC CVAU'. + +config ARM64_CACHE_DIC + bool "Enable support for ICache invalidation PoU optimization" + default y + help + Instruction cache invalidation to the point of unification is not + required for instruction to be data coherence if CTR_EL0.DIC has + value 1. + + Selecting this feature will allow the kernel to optimize the POU + cache maintaince operations where it requires 'IC IVAU'. + endmenu config ARM64_SVE
arch/arm64/include/asm/cache.h +5 -0
@@ -20,8 +20,13 @@#define CTR_L1IP_SHIFT 14#define CTR_L1IP_MASK 3 +#define CTR_DMLINE_SHIFT 16 +#define CTR_ERG_SHIFT 20#define CTR_CWG_SHIFT 24#define CTR_CWG_MASK 15 +#define CTR_IDC_SHIFT 28 +#define CTR_DIC_SHIFT 29 +#define CTR_B31_SHIFT 31#define CTR_L1IP (ctr) (((ctr) >> CTR_L1IP_SHIFT) & CTR_L1IP_MASK)
arch/arm64/include/asm/cpucaps.h +3 -1
@@ -45,7 +45,9 @@#define ARM64_HARDEN_BRANCH_PREDICTOR 24#define ARM64_HARDEN_BP_POST_GUEST_EXIT 25#define ARM64_HAS_RAS_EXTN 26 +#define ARM64_HAS_CACHE_IDC 27 +#define ARM64_HAS_CACHE_DIC 28 -#define ARM64_NCAPS 27 +#define ARM64_NCAPS 29#endif /* __ASM_CPUCAPS_H */
arch/arm64/kernel/cpufeature.c +38 -6
62 lines not on this page · raw message
arch/arm64/mm/cache.S +15 -0
@@ -50,6 +50,9 @@ ENTRY(flush_icache_range) */ ENTRY(__flush_cache_user_range) uaccess_ttbr0_enable x2, x3, x4 +alternative_if ARM64_HAS_CACHE_IDC + b 8f +alternative_else_nop_endif dcache_line_size x2, x3 sub x3, x2, #1 bic x4, x0, x3
@@ -60,6 +63,11 @@ user_alt 9f, "dc cvau, x4", "dc civac, x4", ARM64_WORKAROUND_CLEAN_CACHE b.lo 1b dsb ish +8: +alternative_if ARM64_HAS_CACHE_DIC + mov x0, #0 + b 1f +alternative_else_nop_endif invalidate_icache_by_line x0, x1, x2, x3, 9f mov x0, #0 1:
@@ -80,6 +88,10 @@ ENDPROC(__flush_cache_user_range) * - end - virtual end address of region */ ENTRY(invalidate_icache_range) +alternative_if ARM64_HAS_CACHE_DIC + mov x0, xzr + ret +alternative_else_nop_endif uaccess_ttbr0_enable x2, x3, x4 invalidate_icache_by_line x0, x1, x2, x3, 2f
@@ -116,6 +128,9 @@ ENDPIPROC(__flush_dcache_area) * - size - size in question */ ENTRY(__clean_dcache_area_pou) +alternative_if ARM64_HAS_CACHE_IDC + ret +alternative_else_nop_endif dcache_by_line_op cvau, ish, x0, x1, x2, x3 ret ENDPROC(__clean_dcache_area_pou)
Signature · 2 lines
Qualcomm Datacenter Technologies, Inc. on behalf of the Qualcomm Technologies, Inc. Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.
Catalin Marinas
maintainerQuoted 18 lines · Shanker Donthineni
arch/arm64/Kconfig +10
-0@@ -1095,6 +1095,27 @@ config ARM64_RAS_EXTN and access the new registers if the system supports the extension. Platform RAS features may additionally depend on firmware support. +config ARM64_CACHE_IDC + bool "Enable support for DCache clean PoU optimization" + default y + help + The data cache clean to the point of unification is not required + for instruction to be data coherence if CTR_EL0.IDC has value 1. + + Selecting this feature will allow the kernel to optimize the POU + cache maintaince operations where it requires 'DC CVAU'. +
arch/arm64/Kconfig config ARM64_RAS_EXTN · line 1117
Quoted 10 lines
arch/arm64/Kconfig +10
-0+config ARM64_CACHE_DIC + bool "Enable support for ICache invalidation PoU optimization" + default y + help + Instruction cache invalidation to the point of unification is not + required for instruction to be data coherence if CTR_EL0.DIC has + value 1. + + Selecting this feature will allow the kernel to optimize the POU + cache maintaince operations where it requires 'IC IVAU'.
A single Kconfig entry is sufficient for both features.
Quoted 9 lines
arch/arm64/Kconfig +5
-0@@ -864,6 +864,22 @@ static bool has_no_fpsimd(const struct arm64_cpu_capabilities *entry, int __unus ID_AA64PFR0_FP_SHIFT) < 0; } +#ifdef CONFIG_ARM64_CACHE_IDC +static bool has_cache_idc(const struct arm64_cpu_capabilities *entry, + int __unused) +{ + return !!(read_sanitised_ftr_reg(SYS_CTR_EL0) & (1UL << CTR_IDC_SHIFT));
arch/arm64/Kconfig static bool has_no_fpsimd(const struct arm64_cpu_capabilities *entry, int __unus · line 881
Quoted 10 lines
arch/arm64/Kconfig +10
-0+} +#endif + +#ifdef CONFIG_ARM64_CACHE_DIC +static bool has_cache_dic(const struct arm64_cpu_capabilities *entry, + int __unused) +{ + return !!(read_sanitised_ftr_reg(SYS_CTR_EL0) & (1UL << CTR_DIC_SHIFT)); +} +#endif
Nitpick: no need for !! since the function type is bool already.
Quoted 16 lines
arch/arm64/mm/cache.S +3
-0@@ -50,6 +50,9 @@ ENTRY(flush_icache_range) */ ENTRY(__flush_cache_user_range) uaccess_ttbr0_enable x2, x3, x4 +alternative_if ARM64_HAS_CACHE_IDC + b 8f +alternative_else_nop_endif dcache_line_size x2, x3 sub x3, x2, #1 bic x4, x0, x3@@ -60,6 +63,11 @@ user_alt 9f, "dc cvau, x4", "dc civac, x4", ARM64_WORKAROUND_CLEAN_CACHE b.lo 1b
arch/arm64/mm/cache.S user_alt 9f, "dc cvau, x4", "dc civac, x4", ARM64_WORKAROUND_CLEAN_CACHE · line 73
Quoted 10 lines
arch/arm64/mm/cache.S +5
-0dsb ish +8: +alternative_if ARM64_HAS_CACHE_DIC + mov x0, #0 + b 1f +alternative_else_nop_endif invalidate_icache_by_line x0, x1, x2, x3, 9f mov x0, #0 1:
You can add another label at mov x0, #0 below this hunk and keep a single instruction in the alternative path.
However, my worry is that in an implementation with DIC set, we also skip the DSB/ISB sequence in the invalidate_icache_by_line macro. For example, in an implementation with transparent PoU, we could have:
str <some instr>, [addr] // no cache maintenance or barrier br <addr>
Is an ISB required between the instruction store and execution? I would say yes but maybe Will has a better opinion here.
Signature · 1 line
Catalin
Hi Catalin,
Quoted 23 lines · Catalin Marinas
Quoted 22 lines · Shanker Donthineni
arch/arm64/Kconfig +14
-0@@ -1095,6 +1095,27 @@ config ARM64_RAS_EXTN and access the new registers if the system supports the extension. Platform RAS features may additionally depend on firmware support. +config ARM64_CACHE_IDC + bool "Enable support for DCache clean PoU optimization" + default y + help + The data cache clean to the point of unification is not required + for instruction to be data coherence if CTR_EL0.IDC has value 1. + + Selecting this feature will allow the kernel to optimize the POU + cache maintaince operations where it requires 'DC CVAU'. + +config ARM64_CACHE_DIC + bool "Enable support for ICache invalidation PoU optimization" + default y + help
Quoted 9 lines
arch/arm64/Kconfig config ARM64_RAS_EXTN · line 1117
Quoted 6 lines
arch/arm64/Kconfig +6
-0+ Instruction cache invalidation to the point of unification is not + required for instruction to be data coherence if CTR_EL0.DIC has + value 1. + + Selecting this feature will allow the kernel to optimize the POU + cache maintaince operations where it requires 'IC IVAU'.A single
Kconfigentry is sufficient for both features.
I'll do in v3 patch.
Quoted 13 lines
Quoted 13 lines
arch/arm64/Kconfig +9
-0@@ -864,6 +864,22 @@ static bool has_no_fpsimd(const struct arm64_cpu_capabilities *entry, int __unus ID_AA64PFR0_FP_SHIFT) < 0; } +#ifdef CONFIG_ARM64_CACHE_IDC +static bool has_cache_idc(const struct arm64_cpu_capabilities *entry, + int __unused) +{ + return !!(read_sanitised_ftr_reg(SYS_CTR_EL0) & (1UL << CTR_IDC_SHIFT)); +} +#endif + +#ifdef CONFIG_ARM64_CACHE_DIC
Quoted 9 lines
arch/arm64/Kconfig static bool has_no_fpsimd(const struct arm64_cpu_capabilities *entry, int __unus · line 881
Quoted 6 lines
arch/arm64/Kconfig +6
-0+static bool has_cache_dic(const struct arm64_cpu_capabilities *entry, + int __unused) +{ + return !!(read_sanitised_ftr_reg(SYS_CTR_EL0) & (1UL << CTR_DIC_SHIFT)); +} +#endifNitpick: no need for !! since the function type is bool already.
Sure, I'll remove '!!'.
Quoted 30 lines
Quoted 16 lines
arch/arm64/mm/cache.S +3
-0@@ -50,6 +50,9 @@ ENTRY(flush_icache_range) */ ENTRY(__flush_cache_user_range) uaccess_ttbr0_enable x2, x3, x4 +alternative_if ARM64_HAS_CACHE_IDC + b 8f +alternative_else_nop_endif dcache_line_size x2, x3 sub x3, x2, #1 bic x4, x0, x3@@ -60,6 +63,11 @@ user_alt 9f, "dc cvau, x4", "dc civac, x4", ARM64_WORKAROUND_CLEAN_CACHE b.lo 1barch/arm64/mm/cache.S user_alt 9f, "dc cvau, x4", "dc civac, x4", ARM64_WORKAROUND_CLEAN_CACHE · line 73
Quoted 10 lines
arch/arm64/mm/cache.S +5
-0dsb ish +8: +alternative_if ARM64_HAS_CACHE_DIC + mov x0, #0 + b 1f +alternative_else_nop_endif invalidate_icache_by_line x0, x1, x2, x3, 9f mov x0, #0 1:You can add another label at mov x0, #0 below this hunk and keep a single instruction in the alternative path.
Quoted 8 lines
However, my worry is that in an implementation with DIC set, we also skip the DSB/ISB sequence in the
invalidate_icache_by_linemacro. For example, in an implementation with transparent PoU, we could have:str <some instr>, [addr] // no cache maintenance or barrier br <addr>
Thanks for pointing out the missing barriers. I think it make sense to follow the existing barrier semantics in order to avoid the unknown things.
Quoted 3 lines
Is an ISB required between the instruction store and execution? I would say yes but maybe Will has a better opinion here.
Agree, an ISB is required especially for self-modifying code. I'll include in v3 patch.
Signature · 3 lines
Shanker Donthineni Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc. Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.
Quoted 10 lines · Shanker Donthineni
Quoted 8 lines · Catalin Marinas
However, my worry is that in an implementation with DIC set, we also skip the DSB/ISB sequence in the
invalidate_icache_by_linemacro. For example, in an implementation with transparent PoU, we could have:str <some instr>, [addr] // no cache maintenance or barrier br <addr>
Quoted 7 lines
Thanks for pointing out the missing barriers. I think it make sense to follow the existing barrier semantics in order to avoid the unknown things.
Quoted 3 lines
Is an ISB required between the instruction store and execution? I would say yes but maybe Will has a better opinion here.
Agree, an ISB is required especially for self-modifying code. I'll include in v3 patch.
I'd have thought you'd need a DSB too, before the ISB.
Will