arm64: topology: Fix handling of multi-level cluster MPIDR-based detection

From: Mark Brown <[email protected]>

The only requirement the scheduler has on cluster IDs is that they must be unique. When enumerating the topology based on MPIDR information the kernel currently generates cluster IDs by using the first level of affinity above the core ID (either level one or two depending on if the core has multiple threads) however the ARMv8 architecture allows for up to three levels of affinity. This means that an ARMv8 system may contain cores which have MPIDRs identical other than affinity level three which with current code will cause us to report multiple cores with the same identification to the scheduler in violation of its uniqueness requirement.

Ensure that we do not violate the scheduler requirements on systems that uses all the affinity levels by incorporating both affinity levels two and three into the cluser ID when the cores are not threaded.

While no currently known hardware uses multi-level clusters it is better to program defensively, this will help ease bringup of systems that have them and will ensure that things like distribution install media do not need to be respun to replace kernels in order to deploy such systems. In the worst case the system will work but perform suboptimally until a kernel modified to handle the new topology better is installed, in the best case this will be an adequate description of such topologies for the scheduler to perform well.

Signed-off-by Mark Brown <[email protected]>

---
 arch/arm64/kernel/topology.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

arch/arm64/kernel/topology.c +0 -0

index b6ee26b..5752c1b 100644

--
2.1.0.rc1

Hi Mark,

Quoted 31 lines · Mark Brown

From: Mark Brown <[email protected]>

The only requirement the scheduler has on cluster IDs is that they must be unique. When enumerating the topology based on MPIDR information the kernel currently generates cluster IDs by using the first level of affinity above the core ID (either level one or two depending on if the core has multiple threads) however the ARMv8 architecture allows for up to three levels of affinity. This means that an ARMv8 system may contain cores which have MPIDRs identical other than affinity level three which with current code will cause us to report multiple cores with the same identification to the scheduler in violation of its uniqueness requirement.

Ensure that we do not violate the scheduler requirements on systems that uses all the affinity levels by incorporating both affinity levels two and three into the cluser ID when the cores are not threaded.

While no currently known hardware uses multi-level clusters it is better to program defensively, this will help ease bringup of systems that have them and will ensure that things like distribution install media do not need to be respun to replace kernels in order to deploy such systems. In the worst case the system will work but perform suboptimally until a kernel modified to handle the new topology better is installed, in the best case this will be an adequate description of such topologies for the scheduler to perform well.

Signed-off-by Mark Brown <[email protected]>

---
 arch/arm64/kernel/topology.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

Quoted 5 lines

arch/arm64/kernel/topology.c +0 -0

index b6ee26b..5752c1b 100644

arch/arm64/kernel/topology.c void store_cpu_topology(unsigned int cpuid) · line 262

Quoted 9 lines

arch/arm64/kernel/topology.c +2 -1

@@ -255,7 +255,8 @@ void store_cpu_topology(unsigned int cpuid)
                /* Multiprocessor system : Multi-threads per core */
                cpuid_topo->thread_id  = MPIDR_AFFINITY_LEVEL(mpidr, 0);
                cpuid_topo->core_id    = MPIDR_AFFINITY_LEVEL(mpidr, 1);
-               cpuid_topo->cluster_id = MPIDR_AFFINITY_LEVEL(mpidr, 2);
+               cpuid_topo->cluster_id = MPIDR_AFFINITY_LEVEL(mpidr, 2) |
+                                        MPIDR_AFFINITY_LEVEL(mpidr, 3) << 8;
        } else {
                /* Multiprocessor system : Single-thread per core */
                cpuid_topo->thread_id  = -1;

Can you please extend this for non-SMT case also?

To be honest with you all I would have preferred to sort this out with DT, that's already doable now without this patch.

Having said that, for platforms setting MPIDR_EL1 to reasonable values, I think that's what we should do, squash the upper MPIDR levels (for the non SMT case too). Please stick a proper comment for that to the code. We can always rely on DT to fix other cases that can't be treated properly with code above.

I took some time to think about that, since I am aware of tools (ie powertop) taking the socket id value verbatim, which can become a big number when we squash it. Since it is not a userspace API (or at least it is written nowhere that the socket id value must be a sequential, monotonic value starting from 0 and and on top of that that's an arch specific id that as far as I know must only be unique) I think the change above (plus additional code for non-SMT case) is acceptable.

Quoted 2 lines · Lorenzo Pieralisi

To be honest with you all I would have preferred to sort this out with DT, that's already doable now without this patch.

We'd still have to make sure there's a complete topology binding for ACPI and get through standardisation if any changes are needed. I guess it's probably mostly OK as the scenarios should all be shared with x86 systems but I've not looked to see if some of the subsetting that's being done for ACPI on ARMv8 causes issues.

Quoted 5 lines

Having said that, for platforms setting MPIDR_EL1 to reasonable values, I think that's what we should do, squash the upper MPIDR levels (for the non SMT case too). Please stick a proper comment for that to the code. We can always rely on DT to fix other cases that can't be treated properly with code above.

OK, I'll send a patch.

Quoted 6 lines · Mark Brown

Quoted 4 lines

I think that's what we should do, squash the upper MPIDR levels (for the non SMT case too). Please stick a proper comment for that to the code. We can always rely on DT to fix other cases that can't be treated properly with code above.

OK, I'll send a patch.

do you have time to send it ?