cacheinfo: CPU affinity and Devicetree 'id' support

Use the minimum CPU h/w id of the CPUs associated with the cache for the cache 'id'. This will provide a stable id value for a given system. As we need to check all possible CPUs, we can't use the shared_cpu_map which is just online CPUs. There's not a cache to CPUs mapping in DT, so we have to walk all CPU nodes and then walk cache levels.

Cc Greg Kroah-Hartman <[email protected]>

Cc "Rafael J. Wysocki" <[email protected]>

Signed-off-by Rob Herring <[email protected]>

---
v2:
- Loop with for_each_possible_cpu instead of for_each_of_cpu_node as we will need the logical cpu numbers.

---
  drivers/base/cacheinfo.c | 31 +++++++++++++++++++++++++++++++
  1 file changed, 31 insertions(+)

drivers/base/cacheinfo.c +31 -0

index 66d10bdb863b..21accddf8f5f 100644
@@ -136,6 +136,36 @@ static bool cache_node_is_unified(struct cacheinfo *this_leaf,
  	return of_property_read_bool(np, "cache-unified");
  }
 
+static void cache_of_set_id(struct cacheinfo *this_leaf, struct device_node *np)
+{
+	int cpu;
+	unsigned long min_id = ~0UL;
+
+	for_each_possible_cpu(cpu) {
+		u64 id;
+		struct device_node *cache_node, *cpu_node;
+
+		cache_node = cpu_node = of_get_cpu_node(cpu, NULL);
+		id = of_get_cpu_hwid(cpu_node, 0);
+		while ((cache_node = of_find_next_cache_node(cache_node))) {
+			if (cache_node == np) {
+				if (id < min_id) {
+					min_id = id;
+					of_node_put(cache_node);
+					break;
+				}
+			}
+			of_node_put(cache_node);
+		}
+		of_node_put(cpu_node);
+	}
+
+	if (min_id != ~0UL) {
+		this_leaf->id = min_id;
+		this_leaf->attributes |= CACHE_ID;
+	}
+}
+
  static void cache_of_set_props(struct cacheinfo *this_leaf,
  			       struct device_node *np)
  {

@@ -151,6 +181,7 @@ static void cache_of_set_props(struct cacheinfo *this_leaf,
  	cache_get_line_size(this_leaf, np);
  	cache_nr_sets(this_leaf, np);
  	cache_associativity(this_leaf);
+	cache_of_set_id(this_leaf, np);
  }
 
  static int cache_setup_of_node(unsigned int cpu)

Hi Rob,

Quoted 5 lines · Rob Herring

Use the minimum CPU h/w id of the CPUs associated with the cache for the cache 'id'. This will provide a stable id value for a given system. As we need to check all possible CPUs, we can't use the shared_cpu_map which is just online CPUs. There's not a cache to CPUs mapping in DT, so we have to walk all CPU nodes and then walk cache levels.

I believe another expected use of the cache ID exposed in sysfs is to program steering tags for cache stashing (typically in VFIO-based userspace drivers like DPDK so we can't realistically mediate it any other way). There were plans afoot last year to ensure that ACPI PPTT could provide the necessary ID values for arm64 systems which will typically be fairly arbitrary (but unique) due to reflecting underlying interconnect routing IDs. Assuming that there will eventually be some interest in cache stashing on DT-based systems too, we probably want to allow for an explicit ID property on DT cache nodes in a similar manner.

That said, I think it does make sense to have some kind of auto-generated fallback scheme *as well*, since I'm sure there will be plenty systems which care about MPAM but don't support stashing, and therefore wouldn't have a meaningful set of IDs to populate their DT with. Conversely I think that might also matter for ACPI too - one point I remember from previous discussions is that PPTT may use a compact representation where a single entry represents all equivalent caches at that level, so I'm not sure we can necessarily rely on IDs out of that path being unique either.

Quoted 6 lines · Robin Murphy

Hi Rob,

Quoted 2 lines · Rob Herring

Use the minimum CPU h/w id of the CPUs associated with the cache for the cache 'id'. This will provide a stable id value for a given system. As

Quoted 3 lines

Quoted 3 lines

we need to check all possible CPUs, we can't use the shared_cpu_map which is just online CPUs. There's not a cache to CPUs mapping in DT, so we have to walk all CPU nodes and then walk cache levels.

Quoted 10 lines

I believe another expected use of the cache ID exposed in sysfs is to program steering tags for cache stashing (typically in VFIO-based userspace drivers like DPDK so we can't realistically mediate it any other way). There were plans afoot last year to ensure that ACPI PPTT could provide the necessary ID values for arm64 systems which will typically be fairly arbitrary (but unique) due to reflecting underlying interconnect routing IDs. Assuming that there will eventually be some interest in cache stashing on DT-based systems too, we probably want to allow for an explicit ID property on DT cache nodes in a similar manner.

If you have a suggestion for ID values that correspond to the h/w, then we can add them. I'd like a bit more than just trusting that ID is something real.

While the ACPI folks may be willing to take an arbitrary index, it's something we (mostly) avoid for DT.

Quoted 9 lines

That said, I think it does make sense to have some kind of auto-generated fallback scheme *as well*, since I'm sure there will be plenty systems which care about MPAM but don't support stashing, and therefore wouldn't have a meaningful set of IDs to populate their DT with. Conversely I think that might also matter for ACPI too - one point I remember from previous discussions is that PPTT may use a compact representation where a single entry represents all equivalent caches at that level, so I'm not sure we can necessarily rely on IDs out of that path being unique either.

AIUI, cache ids break the compact representation.

Quoted 7 lines · Rob Herring

Quoted 6 lines · Robin Murphy

Hi Rob,

Quoted 2 lines · Rob Herring

Use the minimum CPU h/w id of the CPUs associated with the cache for the cache 'id'. This will provide a stable id value for a given system. As

I am trying to follow the code. IIUC, the level one(I$ and D$) are skipped in this logic and the private unified cache if any will get the cpu hwid as the cache id which is all fine. But what happens if there are 2 levels of unified private cache ? I am assuming we only care about shared caches for MPAM and ignore private caches which sounds OK but I just wanted to confirm.

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

we need to check all possible CPUs, we can't use the shared_cpu_map which is just online CPUs. There's not a cache to CPUs mapping in DT, so we have to walk all CPU nodes and then walk cache levels.

I would have preferred to add the cache IDs in DT similar to ACPI but I see you have certain concerns with that which are valid as well.

Quoted 5 lines · Rob Herring

Quoted 4 lines · Robin Murphy

Hi Rob,

On 2021-12-16 23:31, Rob Herring wrote:

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Use the minimum CPU h/w id of the CPUs associated with the cache for the cache 'id'. This will provide a stable id value for a given system. As

I am trying to follow the code. IIUC, the level one(I$ and D$) are skipped in this logic and the private unified cache if any will get the cpu hwid as the cache id which is all fine. But what happens if there are 2 levels of unified private cache ? I am assuming we only care about shared caches for MPAM and ignore private caches which sounds OK but I just wanted to confirm.

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

we need to check all possible CPUs, we can't use the shared_cpu_map which is just online CPUs. There's not a cache to CPUs mapping in DT, so we have to walk all CPU nodes and then walk cache levels.

I would have preferred to add the cache IDs in DT similar to ACPI but I see you have certain concerns with that which are valid as well.

Quoted 11 lines

Quoted 10 lines

I believe another expected use of the cache ID exposed in sysfs is to program steering tags for cache stashing (typically in VFIO-based userspace drivers like DPDK so we can't realistically mediate it any other way). There were plans afoot last year to ensure that ACPI PPTT could provide the necessary ID values for arm64 systems which will typically be fairly arbitrary (but unique) due to reflecting underlying interconnect routing IDs. Assuming that there will eventually be some interest in cache stashing on DT-based systems too, we probably want to allow for an explicit ID property on DT cache nodes in a similar manner.

Quoted 4 lines

If you have a suggestion for ID values that correspond to the h/w, then we can add them. I'd like a bit more than just trusting that ID is something real.

I agree, probably architecture must do better job at defining these. But generated IDs IMO might cause issues especial if we have to change the logic without breaking the backward compatibility.

Quoted 3 lines

While the ACPI folks may be willing to take an arbitrary index, it's something we (mostly) avoid for DT.

Not sure if we can call that *arbitrary* 😄, in that case we can imagine the same at several places in the firmware.

Quoted 7 lines · Sudeep Holla

Quoted 5 lines · Rob Herring

Quoted 4 lines · Robin Murphy

Hi Rob,

On 2021-12-16 23:31, Rob Herring wrote:

Quoted 8 lines

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Use the minimum CPU h/w id of the CPUs associated with the cache for the cache 'id'. This will provide a stable id value for a given system. As

I am trying to follow the code. IIUC, the level one(I$ and D$) are skipped in this logic and the private unified cache if any will get the cpu hwid as the cache id which is all fine. But what happens if there are 2 levels of unified private cache ? I am assuming we only care about shared caches for MPAM and ignore private caches which sounds OK but I just wanted to confirm.

The cacheinfo 'id' is only unique to the level and type. It's the type, level, and ID that gives a unique identifier:

* struct cacheinfo - represent a cache leaf node
* @id: This cache's id. It is unique among caches with the same (type, level).

Maybe ACPI's ID expects/allows globally unique cache IDs?

Quoted 28 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

we need to check all possible CPUs, we can't use the shared_cpu_map which is just online CPUs. There's not a cache to CPUs mapping in DT, so we have to walk all CPU nodes and then walk cache levels.

I would have preferred to add the cache IDs in DT similar to ACPI but I see you have certain concerns with that which are valid as well.

Quoted 11 lines

Quoted 10 lines

I believe another expected use of the cache ID exposed in sysfs is to program steering tags for cache stashing (typically in VFIO-based userspace drivers like DPDK so we can't realistically mediate it any other way). There were plans afoot last year to ensure that ACPI PPTT could provide the necessary ID values for arm64 systems which will typically be fairly arbitrary (but unique) due to reflecting underlying interconnect routing IDs. Assuming that there will eventually be some interest in cache stashing on DT-based systems too, we probably want to allow for an explicit ID property on DT cache nodes in a similar manner.

Quoted 4 lines

If you have a suggestion for ID values that correspond to the h/w, then we can add them. I'd like a bit more than just trusting that ID is something real.

I agree, probably architecture must do better job at defining these. But generated IDs IMO might cause issues especial if we have to change the logic without breaking the backward compatibility.

Quoted 6 lines

Quoted 3 lines

While the ACPI folks may be willing to take an arbitrary index, it's something we (mostly) avoid for DT.

Not sure if we can call that *arbitrary* 😄, in that case we can imagine the same at several places in the firmware.

By arbitrary, I mean made up by the binding/dts author or documentation convention (UART0, UART1, etc.). Certainly things like clock IDs are often made up number spaces, but I don't see how we avoid that. DT had 'cell-index' which I still see attempted. But that property traces back to h/w having a single power ctrl register and cell-index was the bit index for the register. If only h/w was still that simple.

Rob

Quoted 10 lines · Robin Murphy

Hi Rob,

Quoted 5 lines · Rob Herring

Use the minimum CPU h/w id of the CPUs associated with the cache for the cache 'id'. This will provide a stable id value for a given system. As we need to check all possible CPUs, we can't use the shared_cpu_map which is just online CPUs. There's not a cache to CPUs mapping in DT, so we have to walk all CPU nodes and then walk cache levels.

Quoted 9 lines

I believe another expected use of the cache ID exposed in sysfs is to program steering tags for cache stashing (typically in VFIO-based userspace drivers like DPDK so we can't realistically mediate it any other way). There were plans afoot last year to ensure that ACPI PPTT could provide the necessary ID values for arm64 systems which will typically be fairly arbitrary (but unique) due to reflecting underlying interconnect routing IDs. Assuming that there will eventually be some interest in cache stashing on DT-based systems too, we probably want to allow for an explicit ID property on DT cache nodes in a similar manner.

If you have a suggestion for ID values that correspond to the h/w, then we can add them. I'd like a bit more than just trusting that ID is something real.

While the ACPI folks may be willing to take an arbitrary index, it's something we (mostly) avoid for DT.

Quoted 9 lines

That said, I think it does make sense to have some kind of auto-generated fallback scheme *as well*, since I'm sure there will be plenty systems which care about MPAM but don't support stashing, and therefore wouldn't have a meaningful set of IDs to populate their DT with. Conversely I think that might also matter for ACPI too - one point I remember from previous discussions is that PPTT may use a compact representation where a single entry represents all equivalent caches at that level, so I'm not sure we can necessarily rely on IDs out of that path being unique either.

AIUI, cache ids break the compact representation.

Quoted 21 lines · Rob Herring

Quoted 10 lines · Robin Murphy

Hi Rob,

Quoted 5 lines · Rob Herring

Use the minimum CPU h/w id of the CPUs associated with the cache for the cache 'id'. This will provide a stable id value for a given system. As we need to check all possible CPUs, we can't use the shared_cpu_map which is just online CPUs. There's not a cache to CPUs mapping in DT, so we have to walk all CPU nodes and then walk cache levels.

Quoted 9 lines

I believe another expected use of the cache ID exposed in sysfs is to program steering tags for cache stashing (typically in VFIO-based userspace drivers like DPDK so we can't realistically mediate it any other way). There were plans afoot last year to ensure that ACPI PPTT could provide the necessary ID values for arm64 systems which will typically be fairly arbitrary (but unique) due to reflecting underlying interconnect routing IDs. Assuming that there will eventually be some interest in cache stashing on DT-based systems too, we probably want to allow for an explicit ID property on DT cache nodes in a similar manner.

Quoted 6 lines

If you have a suggestion for ID values that correspond to the h/w, then we can add them. I'd like a bit more than just trusting that ID is something real.

While the ACPI folks may be willing to take an arbitrary index, it's something we (mostly) avoid for DT.

Not really. On the CHI side there are two fields - StashNID, which could be any node ID value depending on the interconnect layout, plus (optionally) StashLPID to address a specific cache within that node if it's something like a CPU cluster. However, how a PCIe TLP steering tag translates to those fields in the resulting CHI flit is largely up to the root complex.

I think it's going to be more like a "reg" property than a nice validatable index.

Quoted 10 lines

Quoted 9 lines

That said, I think it does make sense to have some kind of auto-generated fallback scheme *as well*, since I'm sure there will be plenty systems which care about MPAM but don't support stashing, and therefore wouldn't have a meaningful set of IDs to populate their DT with. Conversely I think that might also matter for ACPI too - one point I remember from previous discussions is that PPTT may use a compact representation where a single entry represents all equivalent caches at that level, so I'm not sure we can necessarily rely on IDs out of that path being unique either.

Quoted 1 line

AIUI, cache ids break the compact representation.

Right, firmware authors can't use it if they do want to specify IDs, but that also means that if we find we *are* consuming a compact PPTT, then chances are we're not getting meaningful IDs out of it for MPAM to rely on.

Quoted 30 lines · Robin Murphy

Quoted 21 lines · Rob Herring

Quoted 10 lines · Robin Murphy

Hi Rob,

Quoted 5 lines · Rob Herring

Use the minimum CPU h/w id of the CPUs associated with the cache for the cache 'id'. This will provide a stable id value for a given system. As we need to check all possible CPUs, we can't use the shared_cpu_map which is just online CPUs. There's not a cache to CPUs mapping in DT, so we have to walk all CPU nodes and then walk cache levels.

Quoted 9 lines

I believe another expected use of the cache ID exposed in sysfs is to program steering tags for cache stashing (typically in VFIO-based userspace drivers like DPDK so we can't realistically mediate it any other way). There were plans afoot last year to ensure that ACPI PPTT could provide the necessary ID values for arm64 systems which will typically be fairly arbitrary (but unique) due to reflecting underlying interconnect routing IDs. Assuming that there will eventually be some interest in cache stashing on DT-based systems too, we probably want to allow for an explicit ID property on DT cache nodes in a similar manner.

Quoted 6 lines

If you have a suggestion for ID values that correspond to the h/w, then we can add them. I'd like a bit more than just trusting that ID is something real.

While the ACPI folks may be willing to take an arbitrary index, it's something we (mostly) avoid for DT.

Quoted 6 lines

Not really. On the CHI side there are two fields - StashNID, which could be any node ID value depending on the interconnect layout, plus (optionally) StashLPID to address a specific cache within that node if it's something like a CPU cluster. However, how a PCIe TLP steering tag translates to those fields in the resulting CHI flit is largely up to the root complex.

Knowing next to nothing about CHI, this means pretty much nothing to me. :(

I would guess there is a bit more to supporting CHI in DT systems than just a cache ID.

Quoted 13 lines

I think it's going to be more like a "reg" property than a nice validatable index.

Quoted 10 lines

Quoted 9 lines

That said, I think it does make sense to have some kind of auto-generated fallback scheme *as well*, since I'm sure there will be plenty systems which care about MPAM but don't support stashing, and therefore wouldn't have a meaningful set of IDs to populate their DT with. Conversely I think that might also matter for ACPI too - one point I remember from previous discussions is that PPTT may use a compact representation where a single entry represents all equivalent caches at that level, so I'm not sure we can necessarily rely on IDs out of that path being unique either.

Quoted 5 lines

Quoted 1 line

AIUI, cache ids break the compact representation.

Right, firmware authors can't use it if they do want to specify IDs, but that also means that if we find we *are* consuming a compact PPTT, then chances are we're not getting meaningful IDs out of it for MPAM to rely on.

Sounds like broken firmware is in our future. ;) Or ACPI can default to the same id scheme.

Quoted 30 lines · Robin Murphy

Quoted 21 lines · Rob Herring

Quoted 10 lines · Robin Murphy

Hi Rob,

Quoted 5 lines · Rob Herring

Use the minimum CPU h/w id of the CPUs associated with the cache for the cache 'id'. This will provide a stable id value for a given system. As we need to check all possible CPUs, we can't use the shared_cpu_map which is just online CPUs. There's not a cache to CPUs mapping in DT, so we have to walk all CPU nodes and then walk cache levels.

Quoted 9 lines

I believe another expected use of the cache ID exposed in sysfs is to program steering tags for cache stashing (typically in VFIO-based userspace drivers like DPDK so we can't realistically mediate it any other way). There were plans afoot last year to ensure that ACPI PPTT could provide the necessary ID values for arm64 systems which will typically be fairly arbitrary (but unique) due to reflecting underlying interconnect routing IDs. Assuming that there will eventually be some interest in cache stashing on DT-based systems too, we probably want to allow for an explicit ID property on DT cache nodes in a similar manner.

Quoted 6 lines

If you have a suggestion for ID values that correspond to the h/w, then we can add them. I'd like a bit more than just trusting that ID is something real.

While the ACPI folks may be willing to take an arbitrary index, it's something we (mostly) avoid for DT.

Quoted 6 lines

Not really. On the CHI side there are two fields - StashNID, which could be any node ID value depending on the interconnect layout, plus (optionally) StashLPID to address a specific cache within that node if it's something like a CPU cluster. However, how a PCIe TLP steering tag translates to those fields in the resulting CHI flit is largely up to the root complex.

Knowing next to nothing about CHI, this means pretty much nothing to me. :(

I would guess there is a bit more to supporting CHI in DT systems than just a cache ID.

Quoted 13 lines

I think it's going to be more like a "reg" property than a nice validatable index.

Quoted 10 lines

Quoted 9 lines

That said, I think it does make sense to have some kind of auto-generated fallback scheme *as well*, since I'm sure there will be plenty systems which care about MPAM but don't support stashing, and therefore wouldn't have a meaningful set of IDs to populate their DT with. Conversely I think that might also matter for ACPI too - one point I remember from previous discussions is that PPTT may use a compact representation where a single entry represents all equivalent caches at that level, so I'm not sure we can necessarily rely on IDs out of that path being unique either.

Quoted 5 lines

Quoted 1 line

AIUI, cache ids break the compact representation.

Right, firmware authors can't use it if they do want to specify IDs, but that also means that if we find we *are* consuming a compact PPTT, then chances are we're not getting meaningful IDs out of it for MPAM to rely on.

Sounds like broken firmware is in our future. ;) Or ACPI can default to the same id scheme.

For upcoming Arm MPAM support in resctrl, it is necessary to have the cacheinfo 'id' for MPAM enabled caches. The 'id' is part of the resctrl ABI. While this support already exists for ACPI based systems, it is missing for DT. This series adds the support.

The 'id' value used is the smallest CPU h/w id value associated with a cache. This requires walking the cache hierarchy from every CPU node to get all CPUs associated with a cache. As MPAM also needs to know this, the CPU affinity is also saved to avoid reimplementing and walking the firmware tables again.

Patches 1 and 2 are v2 from the prior series[1]. The rest are new.

Tested on arm64 with DT. ACPI changes are untested. I don't have a system with an appropriate PPTT nor do I know how to modify ACPI tables.

Rob

[1] https://lore.kernel.org/all/[email protected]/

Rob Herring (6):
cacheinfo: Allow for >32-bit cache 'id'
cacheinfo: Set cache 'id' based on DT data
cacheinfo: Add cpu_affinity_map to store affinity for all CPUs ACPI / PPTT: Populate the cacheinfo.cpu_affinity_map
cacheinfo: Use cpu_affinity_map for populating shared_cpu_map
cacheinfo: Add cacheinfo_get_cache_affinity() function

 drivers/acpi/pptt.c       | 29 +++++++++++++++--
 drivers/base/cacheinfo.c  | 65 ++++++++++++++++++++++++++-------------
 include/linux/cacheinfo.h | 29 +++++++++++++++--
 3 files changed, 97 insertions(+), 26 deletions(-)

Signature · 1 line

2.32.0