Skip to contentSkip to conversation

arch_topology: Pre-allocate cacheinfo from primary CPU

PATCH Superseded Radu Rendec 2 participants 5 messages Apr 3, 2023 – Apr 6, 2023

Subsystems (2)

v2 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

v2 cover arch_topology: Pre-allocate cacheinfo from primary CPU

Commit 5944ce092b97 ("arch_topology: Build cacheinfo from primary CPU") tries to build the cacheinfo from the primary CPU prior to secondary CPUs boot, if the DT/ACPI description contains cache information. However, if such information is not present, it still reverts to the old behavior, which allocates the cacheinfo memory on each secondary CPU. On RT kernels, this triggers a "BUG: sleeping function called from invalid context" because the allocation is done before preemption is first enabled on the secondary CPU.

The solution is to add cache information to DT/ACPI, but at least on arm64 systems this can be avoided by leveraging automatic detection (through the CLIDR_EL1 register), which is already implemented but currently doesn't work on RT kernels for the reason described above.

This patch series attempts to enable automatic detection for RT kernels when no DT/ACPI cache information is available, by pre-allocating cacheinfo memory on the primary CPU.

The first patch adds an architecture independent infrastructure that allows architecture specific code to take an early guess at the number of cache leaves of the secodary CPUs, while it runs in preemptible context on the primary CPU. At the same time, it gives architecture specific code the opportunity to go back later, while it runs on the secondary CPU, and reallocate the cacheinfo memory if the initial guess proves to be wrong.

The second patch leverages the infrastructure implemented in the first patch and enables early cache depth detection for arm64.

The patch series is based on an RFC patch that was posted to the linux-arm-kernel mailing list and discussed with a smaller audience:
https://lore.kernel.org/all/[email protected]/

Radu Rendec (2):
cacheinfo: Add arch specific early level initializer
cacheinfo: Add arm64 early level initializer implementation

 arch/arm64/kernel/cacheinfo.c | 32 +++++++++++++++-----
 drivers/base/cacheinfo.c      | 57 +++++++++++++++++++++++------------
 include/linux/cacheinfo.h     |  2 ++
 3 files changed, 64 insertions(+), 27 deletions(-)
Signature · 1 line
2.39.2
#1 lore raw

PATCH v2 1/2 cacheinfo: Add arch specific early level initializer

This patch gives of architecture specific code the ability to initialize the cache level and allocate cacheinfo memory early, when cache level initialization runs on the primary CPU for all possible CPUs.

This is part of a patch series that attempts to further the work in commit 5944ce092b97 ("arch_topology: Build cacheinfo from primary CPU"). Previously, in the absence of any DT/ACPI cache info, architecture specific cache detection and info allocation for secondary CPUs would happen in non-preemptible context during early CPU initialization and trigger a "BUG: sleeping function called from invalid context" splat on an RT kernel.

More specifically, this patch adds the early_cache_level() function, which is called by fetch_cache_info() as a fallback when the number of cache leaves cannot be extracted from DT/ACPI. In the default generic (weak) implementation, this new function returns -ENOENT, which preserves the original behavior for architectures that do not implement the function.

Since early detection can get the number of cache leaves wrong in some cases*, additional logic is added to still call init_cache_level() later on the secondary CPU, therefore giving the architecture specific code an opportunity to go back and fix the initial guess. Again, the original behavior is preserved for architectures that do not implement the new function.

* For example, on arm64, CLIDR_EL1 detection works only when it runs on the current CPU. In other words, a CPU cannot detect the cache depth for any other CPU than itself.

---
 drivers/base/cacheinfo.c  | 57 ++++++++++++++++++++++++++-------------
 include/linux/cacheinfo.h |  2 ++
 2 files changed, 40 insertions(+), 19 deletions(-)
drivers/base/cacheinfo.c +38 -19
index f6573c335f4c..7f8ac0cb549f 100644
@@ -398,6 +398,11 @@ static void free_cache_attributes(unsigned int cpu)
 	cache_shared_cpu_map_remove(cpu);
 }
 
+int __weak early_cache_level(unsigned int cpu)
+{
+	return -ENOENT;
+}
+
 int __weak init_cache_level(unsigned int cpu)
 {
 	return -ENOENT;
@@ -423,51 +428,65 @@ int allocate_cache_info(int cpu)
 
 int fetch_cache_info(unsigned int cpu)
 {
-	struct cpu_cacheinfo *this_cpu_ci;
+	struct cpu_cacheinfo *this_cpu_ci = get_cpu_cacheinfo(cpu);
 	unsigned int levels = 0, split_levels = 0;
 	int ret;
 
-	if (acpi_disabled) {
+	if (acpi_disabled)
 		ret = init_of_cache_level(cpu);
-		if (ret < 0)
-			return ret;
-	} else {
+	else {
 		ret = acpi_get_cache_info(cpu, &levels, &split_levels);
-		if (ret < 0)
+		if (!ret) {
+			this_cpu_ci->num_levels = levels;
+			/*
+			 * This assumes that:
+			 * - there cannot be any split caches (data/instruction)
+			 *   above a unified cache
+			 * - data/instruction caches come by pair
+			 */
+			this_cpu_ci->num_leaves = levels + split_levels;
+		}
+	}
+
+	if (ret || !cache_leaves(cpu)) {
+		ret = early_cache_level(cpu);
+		if (ret)
 			return ret;
 
-		this_cpu_ci = get_cpu_cacheinfo(cpu);
-		this_cpu_ci->num_levels = levels;
-		/*
-		 * This assumes that:
-		 * - there cannot be any split caches (data/instruction)
-		 *   above a unified cache
-		 * - data/instruction caches come by pair
-		 */
-		this_cpu_ci->num_leaves = levels + split_levels;
+		if (!cache_leaves(cpu))
+			return -ENOENT;
+
+		this_cpu_ci->early_arch_info = true;
 	}
-	if (!cache_leaves(cpu))
-		return -ENOENT;
 
 	return allocate_cache_info(cpu);
 }
 
 int detect_cache_attributes(unsigned int cpu)
 {
+	unsigned int early_leaves = cache_leaves(cpu);
 	int ret;
 
 	/* Since early initialization/allocation of the cacheinfo is allowed
 	 * via fetch_cache_info() and this also gets called as CPU hotplug
 	 * callbacks via cacheinfo_cpu_online, the init/alloc can be skipped
 	 * as it will happen only once (the cacheinfo memory is never freed).
-	 * Just populate the cacheinfo.
+	 * Just populate the cacheinfo. However, if the cacheinfo has been
+	 * allocated early through the arch-specific early_cache_level() call,
+	 * there is a chance the info is wrong (this can happen on arm64). In
+	 * that case, call init_cache_level() anyway to give the arch-specific
+	 * code a chance to make things right.
 	 */
-	if (per_cpu_cacheinfo(cpu))
+	if (per_cpu_cacheinfo(cpu) && !ci_cacheinfo(cpu)->early_arch_info)
 		goto populate_leaves;
 
 	if (init_cache_level(cpu) || !cache_leaves(cpu))
 		return -ENOENT;
 
+	if (cache_leaves(cpu) <= early_leaves)
+		goto populate_leaves;
+
+	kfree(per_cpu_cacheinfo(cpu));
 	ret = allocate_cache_info(cpu);
 	if (ret)
 		return ret;
include/linux/cacheinfo.h +2 -0
index 908e19d17f49..c9d44308fc42 100644
@@ -76,9 +76,11 @@ struct cpu_cacheinfo {
 	unsigned int num_levels;
 	unsigned int num_leaves;
 	bool cpu_map_populated;
+	bool early_arch_info;
 };
 
 struct cpu_cacheinfo *get_cpu_cacheinfo(unsigned int cpu);
+int early_cache_level(unsigned int cpu);
 int init_cache_level(unsigned int cpu);
 int init_of_cache_level(unsigned int cpu);
 int populate_cache_leaves(unsigned int cpu);
Signature · 1 line
2.39.2
#2 lore raw

Re: [PATCH v2 1/2] cacheinfo: Add arch specific early level initializer

Hello Radu,

Quoted 58 lines · Radu Rendec

This patch gives of architecture specific code the ability to initialize the cache level and allocate cacheinfo memory early, when cache level initialization runs on the primary CPU for all possible CPUs.

This is part of a patch series that attempts to further the work in commit 5944ce092b97 ("arch_topology: Build cacheinfo from primary CPU"). Previously, in the absence of any DT/ACPI cache info, architecture specific cache detection and info allocation for secondary CPUs would happen in non-preemptible context during early CPU initialization and trigger a "BUG: sleeping function called from invalid context" splat on an RT kernel.

More specifically, this patch adds the early_cache_level() function, which is called by fetch_cache_info() as a fallback when the number of cache leaves cannot be extracted from DT/ACPI. In the default generic (weak) implementation, this new function returns -ENOENT, which preserves the original behavior for architectures that do not implement the function.

Since early detection can get the number of cache leaves wrong in some cases*, additional logic is added to still call init_cache_level() later on the secondary CPU, therefore giving the architecture specific code an opportunity to go back and fix the initial guess. Again, the original behavior is preserved for architectures that do not implement the new function.

* For example, on arm64, CLIDR_EL1 detection works only when it runs on the current CPU. In other words, a CPU cannot detect the cache depth for any other CPU than itself.

---
  drivers/base/cacheinfo.c  | 57 ++++++++++++++++++++++++++-------------
  include/linux/cacheinfo.h |  2 ++
  2 files changed, 40 insertions(+), 19 deletions(-)
drivers/base/cacheinfo.c +6 -1
index f6573c335f4c..7f8ac0cb549f 100644
@@ -398,6 +398,11 @@ static void free_cache_attributes(unsigned int cpu)
  	cache_shared_cpu_map_remove(cpu);
  }
 
+int __weak early_cache_level(unsigned int cpu)
+{
+	return -ENOENT;
+}
+
  int __weak init_cache_level(unsigned int cpu)
  {
  	return -ENOENT;
@@ -423,51 +428,65 @@ int allocate_cache_info(int cpu)
 
  int fetch_cache_info(unsigned int cpu)
  {
-	struct cpu_cacheinfo *this_cpu_ci;
+	struct cpu_cacheinfo *this_cpu_ci = get_cpu_cacheinfo(cpu);

drivers/base/cacheinfo.c int allocate_cache_info(int cpu) · line 437

Quoted 10 lines
drivers/base/cacheinfo.c +2 -4
  	unsigned int levels = 0, split_levels = 0;
  	int ret;
 
-	if (acpi_disabled) {
+	if (acpi_disabled)
  		ret = init_of_cache_level(cpu);
-		if (ret < 0)
-			return ret;
-	} else {
+	else {

NIT: I think braces should be used in the first branch if they are used in the second branch, cf. './scripts/checkpatch.pl --strict'

Quoted 31 lines
drivers/base/cacheinfo.c +18 -10
  		ret = acpi_get_cache_info(cpu, &levels, &split_levels);
-		if (ret < 0)
+		if (!ret) {
+			this_cpu_ci->num_levels = levels;
+			/*
+			 * This assumes that:
+			 * - there cannot be any split caches (data/instruction)
+			 *   above a unified cache
+			 * - data/instruction caches come by pair
+			 */
+			this_cpu_ci->num_leaves = levels + split_levels;
+		}
+	}
+
+	if (ret || !cache_leaves(cpu)) {
+		ret = early_cache_level(cpu);
+		if (ret)
  			return ret;
 
-		this_cpu_ci = get_cpu_cacheinfo(cpu);
-		this_cpu_ci->num_levels = levels;
-		/*
-		 * This assumes that:
-		 * - there cannot be any split caches (data/instruction)
-		 *   above a unified cache
-		 * - data/instruction caches come by pair
-		 */
-		this_cpu_ci->num_leaves = levels + split_levels;
+		if (!cache_leaves(cpu))
+			return -ENOENT;
+

drivers/base/cacheinfo.c int allocate_cache_info(int cpu) · line 466

Quoted 10 lines
drivers/base/cacheinfo.c +1 -2
+		this_cpu_ci->early_arch_info = true;
  	}
-	if (!cache_leaves(cpu))
-		return -ENOENT;
 
  	return allocate_cache_info(cpu);
  }
 
  int detect_cache_attributes(unsigned int cpu)
  {

[start]

Quoted 17 lines
drivers/base/cacheinfo.c +7 -2
+	unsigned int early_leaves = cache_leaves(cpu);
  	int ret;
 
  	/* Since early initialization/allocation of the cacheinfo is allowed
  	 * via fetch_cache_info() and this also gets called as CPU hotplug
  	 * callbacks via cacheinfo_cpu_online, the init/alloc can be skipped
  	 * as it will happen only once (the cacheinfo memory is never freed).
-	 * Just populate the cacheinfo.
+	 * Just populate the cacheinfo. However, if the cacheinfo has been
+	 * allocated early through the arch-specific early_cache_level() call,
+	 * there is a chance the info is wrong (this can happen on arm64). In
+	 * that case, call init_cache_level() anyway to give the arch-specific
+	 * code a chance to make things right.
  	 */
-	if (per_cpu_cacheinfo(cpu))
+	if (per_cpu_cacheinfo(cpu) && !ci_cacheinfo(cpu)->early_arch_info)
  		goto populate_leaves;

drivers/base/cacheinfo.c int allocate_cache_info(int cpu) · line 492

Quoted 10 lines
drivers/base/cacheinfo.c +4 -0
 
  	if (init_cache_level(cpu) || !cache_leaves(cpu))
  		return -ENOENT;
 
+	if (cache_leaves(cpu) <= early_leaves)
+		goto populate_leaves;
+
+	kfree(per_cpu_cacheinfo(cpu))>   	ret = allocate_cache_info(cpu);
  	if (ret)
  		return ret;

[stop]

Maybe this would be the occasion to put the code between the start/stop in a separate function and remove the 'populate_leaves' label. The code seems correct, but it was already a bit complex to read before the patch.

Quoted 16 lines
include/linux/cacheinfo.h +2 -0
index 908e19d17f49..c9d44308fc42 100644
@@ -76,9 +76,11 @@ struct cpu_cacheinfo {
  	unsigned int num_levels;
  	unsigned int num_leaves;
  	bool cpu_map_populated;
+	bool early_arch_info;
  };
 
  struct cpu_cacheinfo *get_cpu_cacheinfo(unsigned int cpu);
+int early_cache_level(unsigned int cpu);
  int init_cache_level(unsigned int cpu);
  int init_of_cache_level(unsigned int cpu);
  int populate_cache_leaves(unsigned int cpu);
#3 lore raw

Re: [PATCH v2 1/2] cacheinfo: Add arch specific early level initializer

Hello Pierre,

Quoted 27 lines · Pierre Gondois

[start]

Quoted 26 lines

+       unsigned int early_leaves = cache_leaves(cpu);
        int ret;

        /* Since early initialization/allocation of the cacheinfo is allowed
         * via fetch_cache_info() and this also gets called as CPU hotplug
         * callbacks via cacheinfo_cpu_online, the init/alloc can be skipped
         * as it will happen only once (the cacheinfo memory is never freed). -        * Just populate the cacheinfo.
+        * Just populate the cacheinfo. However, if the cacheinfo has been +        * allocated early through the arch-specific early_cache_level() call, +        * there is a chance the info is wrong (this can happen on arm64). In +        * that case, call init_cache_level() anyway to give the arch-specific +        * code a chance to make things right.
         */
-       if (per_cpu_cacheinfo(cpu))
+       if (per_cpu_cacheinfo(cpu) && !ci_cacheinfo(cpu)->early_arch_info) goto populate_leaves;

        if (init_cache_level(cpu) || !cache_leaves(cpu))
                return -ENOENT;

+       if (cache_leaves(cpu) <= early_leaves)
+               goto populate_leaves;
+
+       kfree(per_cpu_cacheinfo(cpu))

Quoted 8 lines
Quoted 3 lines

        ret = allocate_cache_info(cpu);
        if (ret)
                return ret;

[stop]

Maybe this would be the occasion to put the code between the start/stop in a separate function and remove the 'populate_leaves' label. The code seems correct, but it was already a bit complex to read before the patch.

Yes, that makes sense. I will address this (and your other comment about the braces) and post v3 shortly. Thanks for reviewing the patch and for the feedback!

After I had sent v2, I realized there was something missing from that code between start/stop. I think we should also set the early_arch_info flag back to false to prevent another detection/reallocation in case detect_cache_attributes() is called again (I'm thinking CPU hotplug). I will address this as well in v3.

Best regards,
Radu

#4 lore raw

PATCH v2 2/2 cacheinfo: Add arm64 early level initializer implementation

In reply to Radu Rendec

This patch adds an architecture specific early cache level detection handler for arm64. This is basically the CLIDR_EL1 based detection that was previously done (only) in init_cache_level().

This is part of a patch series that attempts to further the work in commit 5944ce092b97 ("arch_topology: Build cacheinfo from primary CPU"). Previously, in the absence of any DT/ACPI cache info, architecture specific cache detection and info allocation for secondary CPUs would happen in non-preemptible context during early CPU initialization and trigger a "BUG: sleeping function called from invalid context" splat on an RT kernel.

This patch does not solve the problem completely for RT kernels. It relies on the assumption that on most systems, the CPUs are symmetrical and therefore have the same number of cache leaves. The cacheinfo memory is allocated early (on the primary CPU), relying on the new handler. If later (when CLIDR_EL1 based detection runs again on the secondary CPU) the initial assumption proves to be wrong and the CPU has in fact more leaves, the cacheinfo memory is reallocated, and that still triggers a splat on an RT kernel.

In other words, asymmetrical CPU systems *must* still provide cacheinfo data in DT/ACPI to avoid the splat on RT kernels (unless secondary CPUs happen to have less leaves than the primary CPU). But symmetrical CPU systems (the majority) can now get away without the additional DT/ACPI data and rely on CLIDR_EL1 based detection.

---
 arch/arm64/kernel/cacheinfo.c | 32 ++++++++++++++++++++++++--------
 1 file changed, 24 insertions(+), 8 deletions(-)
arch/arm64/kernel/cacheinfo.c +24 -8
index c307f69e9b55..520d17e4ebe9 100644
@@ -38,21 +38,37 @@ static void ci_leaf_init(struct cacheinfo *this_leaf,
 	this_leaf->type = type;
 }
 
-int init_cache_level(unsigned int cpu)
+static void detect_cache_level(unsigned int *level, unsigned int *leaves)
 {
-	unsigned int ctype, level, leaves;
-	int fw_level, ret;
-	struct cpu_cacheinfo *this_cpu_ci = get_cpu_cacheinfo(cpu);
+	unsigned int ctype;
 
-	for (level = 1, leaves = 0; level <= MAX_CACHE_LEVEL; level++) {
-		ctype = get_cache_type(level);
+	for (*level = 1, *leaves = 0; *level <= MAX_CACHE_LEVEL; (*level)++) {
+		ctype = get_cache_type(*level);
 		if (ctype == CACHE_TYPE_NOCACHE) {
-			level--;
+			(*level)--;
 			break;
 		}
 		/* Separate instruction and data caches */
-		leaves += (ctype == CACHE_TYPE_SEPARATE) ? 2 : 1;
+		*leaves += (ctype == CACHE_TYPE_SEPARATE) ? 2 : 1;
 	}
+}
+
+int early_cache_level(unsigned int cpu)
+{
+	struct cpu_cacheinfo *this_cpu_ci = get_cpu_cacheinfo(cpu);
+
+	detect_cache_level(&this_cpu_ci->num_levels, &this_cpu_ci->num_leaves);
+
+	return 0;
+}
+
+int init_cache_level(unsigned int cpu)
+{
+	unsigned int level, leaves;
+	int fw_level, ret;
+	struct cpu_cacheinfo *this_cpu_ci = get_cpu_cacheinfo(cpu);
+
+	detect_cache_level(&level, &leaves);
 
 	if (acpi_disabled) {
 		fw_level = of_find_last_cache_level(cpu);
Signature · 7 lines
2.39.2


_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel