Heterogenous memory node attributes

System memory may have caches to help improve access speed to frequently requested address ranges. While the system provided cache is transparent to the software accessing these memory ranges, applications can optimize their own access based on cache attributes.

Provide a new API for the kernel to register these memory-side caches under the memory node that provides it.

The new sysfs representation is modeled from the existing cpu cacheinfo attributes, as seen from /sys/devices/system/cpu/<cpu>/cache/. Unlike CPU cacheinfo though, the node cache level is reported from the view of the memory. A higher level number is nearer to the CPU, while lower levels are closer to the last level memory.

The exported attributes are the cache size, the line size, associativity, and write back policy, and add the attributes for the system memory caches to sysfs stable documentation.

Signed-off-by Keith Busch <[email protected]>

---
 Documentation/ABI/stable/sysfs-devices-node |  35 +++++++
 drivers/base/node.c                         | 151 ++++++++++++++++++++++++++++
 include/linux/node.h                        |  34 +++++++
 3 files changed, 220 insertions(+)

Documentation/ABI/stable/sysfs-devices-node +18 -0

index cd64b62152ba..5c88cb9ca14e 100644
@@ -143,3 +143,38 @@ Contact:	Keith Busch <[email protected]>
 Description:
 		This node's write latency in nanoseconds when access
 		from nodes found in this class's linked initiators.
+
+What:		/sys/devices/system/node/nodeX/memory_side_cache/indexY/
+Date:		December 2018
+Contact:	Keith Busch <[email protected]>
+Description:
+		The directory containing attributes for the memory-side cache
+		level 'Y'.
+
+		The caches associativity: 0 for direct mapped, non-zero if
+What:		/sys/devices/system/node/nodeX/memory_side_cache/indexY/associativity
+Date:		December 2018
+Contact:	Keith Busch <[email protected]>
+Description:
+		The caches associativity: 0 for direct mapped, non-zero if
+		indexed.
+
+What:		/sys/devices/system/node/nodeX/memory_side_cache/indexY/line_size
+Date:		December 2018

Le 14/02/2019 à 18:10, Keith Busch a écrit :

Documentation/ABI/stable/sysfs-devices-node Contact: Keith Busch <[email protected]> · line 173

Quoted 5 lines

Documentation/ABI/stable/sysfs-devices-node +5 -0

+What:		/sys/devices/system/node/nodeX/memory_side_cache/indexY/size
+Date:		December 2018
+Contact:	Keith Busch <[email protected]>
+Description:
+		The size of this memory side cache in bytes.

Hello Keith,

CPU-side cache size is reported in kilobytes:

$ cat
/sys/devices/system/cpu/cpu0/cache/index*/size                                             

32K
32K
256K
4096K

Can you do the same of memory-side caches instead of reporting bytes?

Quoted 15 lines · Brice Goglin

Documentation/ABI/stable/sysfs-devices-node Contact: Keith Busch <[email protected]> · line 173

Quoted 5 lines · Keith Busch

Documentation/ABI/stable/sysfs-devices-node +5 -0

+What:              /sys/devices/system/node/nodeX/memory_side_cache/indexY/size
+Date:              December 2018
+Contact:   Keith Busch <[email protected]>
+Description:
+           The size of this memory side cache in bytes.

Hello Keith,

CPU-side cache size is reported in kilobytes:

$ cat
/sys/devices/system/cpu/cpu0/cache/index*/size

Quoted 6 lines

32K
32K
256K
4096K

Can you do the same of memory-side caches instead of reporting bytes?

Ok, will do.

System memory may have caches to help improve access speed to frequently requested address ranges. While the system provided cache is transparent to the software accessing these memory ranges, applications can optimize their own access based on cache attributes.

Provide a new API for the kernel to register these memory-side caches under the memory node that provides it.

The new sysfs representation is modeled from the existing cpu cacheinfo attributes, as seen from /sys/devices/system/cpu/<cpu>/cache/. Unlike CPU cacheinfo though, the node cache level is reported from the view of the memory. A higher level number is nearer to the CPU, while lower levels are closer to the last level memory.

The exported attributes are the cache size, the line size, associativity, and write back policy, and add the attributes for the system memory caches to sysfs stable documentation.

Signed-off-by Keith Busch <[email protected]>

---
 Documentation/ABI/stable/sysfs-devices-node |  35 +++++++
 drivers/base/node.c                         | 151 ++++++++++++++++++++++++++++
 include/linux/node.h                        |  34 +++++++
 3 files changed, 220 insertions(+)

Documentation/ABI/stable/sysfs-devices-node +5 -0

index cd64b62152ba..5c88cb9ca14e 100644
@@ -143,3 +143,38 @@ Contact:	Keith Busch <[email protected]>
 Description:
 		This node's write latency in nanoseconds when access
 		from nodes found in this class's linked initiators.
+
+What:		/sys/devices/system/node/nodeX/memory_side_cache/indexY/
+Date:		December 2018
+Contact:	Keith Busch <[email protected]>
+Description:

Le 14/02/2019 à 18:10, Keith Busch a écrit :

Documentation/ABI/stable/sysfs-devices-node Contact: Keith Busch <[email protected]> · line 160

Quoted 6 lines

Documentation/ABI/stable/sysfs-devices-node +6 -0

+What:		/sys/devices/system/node/nodeX/memory_side_cache/indexY/associativity
+Date:		December 2018
+Contact:	Keith Busch <[email protected]>
+Description:
+		The caches associativity: 0 for direct mapped, non-zero if
+		indexed.

Should we rename "associativity" into "indexing" or something else?

When I see "associativity" that contains 0, I tend to interpret this as the associativity value itself, which would mean fully-associative here (as in CPU-side cache "ways_of_associativity" attribute), while actually 0 means direct-mapped (ie 1-associative) with yout semantics.

Brice

If the HMAT Subsystem Address Range provides a valid processor proximity domain for a memory domain, or a processor domain matches the performance access of the valid processor proximity domain, register the memory target with that initiator so this relationship will be visible under the node's sysfs directory.

By registering only the best performing relationships, this provides the most useful information applications may want to know when considering which CPU they should run on for a given memory node, or which memory node they should allocate memory from for a given CPU.

Since HMAT requires valid address ranges have an equivalent SRAT entry, verify each memory target satisfies this requirement.

Signed-off-by Keith Busch <[email protected]>

---
 drivers/acpi/hmat/Kconfig |   1 +
 drivers/acpi/hmat/hmat.c  | 396 +++++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 396 insertions(+), 1 deletion(-)

drivers/acpi/hmat/Kconfig +0 -0

index c9637e2e7514..08e972ead159 100644

Quoted 5 lines

drivers/acpi/hmat/Kconfig +0 -0

index c9637e2e7514..08e972ead159 100644
@@ -2,6 +2,7 @@

drivers/acpi/hmat/Kconfig line 5

Quoted 4 lines

drivers/acpi/hmat/Kconfig +1 -0

 config ACPI_HMAT
        bool "ACPI Heterogeneous Memory Attribute Table Support"
        depends on ACPI_NUMA
+       select HMEM_REPORTING

If you want to do this here, I'm not sure that defining HMEM_REPORTING as a user-selectable option is a good idea. In particular, I don't really think that setting ACPI_HMAT without it makes a lot of sense. Apart from this, the patch looks reasonable to me.

Quoted 5 lines · Rafael J. Wysocki

Quoted 5 lines

drivers/acpi/hmat/Kconfig +0 -0

index c9637e2e7514..08e972ead159 100644
@@ -2,6 +2,7 @@

Quoted 8 lines

drivers/acpi/hmat/Kconfig line 5

Quoted 4 lines

drivers/acpi/hmat/Kconfig +1 -0

 config ACPI_HMAT
        bool "ACPI Heterogeneous Memory Attribute Table Support"
        depends on ACPI_NUMA
+       select HMEM_REPORTING

If you want to do this here, I'm not sure that defining HMEM_REPORTING as a user-selectable option is a good idea. In particular, I don't really think that setting ACPI_HMAT without it makes a lot of sense. Apart from this, the patch looks reasonable to me.

I guess the question is whether we would want to allow folks to consume the HMAT inside the kernel while not reporting it out via HMEM_REPORTING. We have some in-kernel users of the HMAT lined up like mitigations for memory-side caches.

It's certainly possible that folks would want to consume those mitigations without anything in sysfs. They might not even want or need NUMA support itself, for instance.

So, what should we do?

config HMEM_REPORTING
bool # no user-visible prompt
default y if ACPI_HMAT

So folks can override in their .config, but they don't see a prompt?

Quoted 27 lines · Dave Hansen

Quoted 5 lines · Rafael J. Wysocki

Quoted 5 lines

drivers/acpi/hmat/Kconfig +0 -0

index c9637e2e7514..08e972ead159 100644
@@ -2,6 +2,7 @@

Quoted 8 lines

drivers/acpi/hmat/Kconfig line 5

Quoted 4 lines

drivers/acpi/hmat/Kconfig +1 -0

 config ACPI_HMAT
        bool "ACPI Heterogeneous Memory Attribute Table Support"
        depends on ACPI_NUMA
+       select HMEM_REPORTING

If you want to do this here, I'm not sure that defining HMEM_REPORTING as a user-selectable option is a good idea. In particular, I don't really think that setting ACPI_HMAT without it makes a lot of sense. Apart from this, the patch looks reasonable to me.

I guess the question is whether we would want to allow folks to consume the HMAT inside the kernel while not reporting it out via HMEM_REPORTING. We have some in-kernel users of the HMAT lined up like mitigations for memory-side caches.

It's certainly possible that folks would want to consume those mitigations without anything in sysfs. They might not even want or need NUMA support itself, for instance.

So, what should we do?

Quoted 5 lines

config HMEM_REPORTING
bool # no user-visible prompt
default y if ACPI_HMAT

So folks can override in their .config, but they don't see a prompt?

I would add an "&& ACPI_NUMA" to that default as well.

Quoted 33 lines · Dan Williams

Quoted 31 lines · Dave Hansen

Quoted 5 lines · Rafael J. Wysocki

Quoted 5 lines

drivers/acpi/hmat/Kconfig +0 -0

index c9637e2e7514..08e972ead159 100644
@@ -2,6 +2,7 @@

Quoted 8 lines

drivers/acpi/hmat/Kconfig line 5

Quoted 4 lines

drivers/acpi/hmat/Kconfig +1 -0

 config ACPI_HMAT
        bool "ACPI Heterogeneous Memory Attribute Table Support"
        depends on ACPI_NUMA
+       select HMEM_REPORTING

If you want to do this here, I'm not sure that defining HMEM_REPORTING as a user-selectable option is a good idea. In particular, I don't really think that setting ACPI_HMAT without it makes a lot of sense. Apart from this, the patch looks reasonable to me.

I guess the question is whether we would want to allow folks to consume the HMAT inside the kernel while not reporting it out via HMEM_REPORTING. We have some in-kernel users of the HMAT lined up like mitigations for memory-side caches.

It's certainly possible that folks would want to consume those mitigations without anything in sysfs. They might not even want or need NUMA support itself, for instance.

So, what should we do?

config HMEM_REPORTING
bool # no user-visible prompt
default y if ACPI_HMAT

Quoted 3 lines

Quoted 1 line

So folks can override in their .config, but they don't see a prompt?

I would add an "&& ACPI_NUMA" to that default as well.

But ACPI_HMAT depends on ACPI_NUMA already, or am I missing anything?

Quoted 5 lines · Rafael J. Wysocki

Quoted 5 lines

drivers/acpi/hmat/Kconfig +0 -0

index c9637e2e7514..08e972ead159 100644
@@ -2,6 +2,7 @@

Quoted 8 lines

drivers/acpi/hmat/Kconfig line 5

Quoted 4 lines

drivers/acpi/hmat/Kconfig +1 -0

 config ACPI_HMAT
        bool "ACPI Heterogeneous Memory Attribute Table Support"
        depends on ACPI_NUMA
+       select HMEM_REPORTING

If you want to do this here, I'm not sure that defining HMEM_REPORTING as a user-selectable option is a good idea. In particular, I don't really think that setting ACPI_HMAT without it makes a lot of sense. Apart from this, the patch looks reasonable to me.

I guess the question is whether we would want to allow folks to consume the HMAT inside the kernel while not reporting it out via HMEM_REPORTING. We have some in-kernel users of the HMAT lined up like mitigations for memory-side caches.

It's certainly possible that folks would want to consume those mitigations without anything in sysfs. They might not even want or need NUMA support itself, for instance.

So, what should we do?

config HMEM_REPORTING
bool # no user-visible prompt
default y if ACPI_HMAT

So folks can override in their .config, but they don't see a prompt?

Quoted 30 lines · Dave Hansen

Quoted 5 lines · Rafael J. Wysocki

Quoted 5 lines

drivers/acpi/hmat/Kconfig +0 -0

index c9637e2e7514..08e972ead159 100644
@@ -2,6 +2,7 @@

Quoted 8 lines

drivers/acpi/hmat/Kconfig line 5

Quoted 4 lines

drivers/acpi/hmat/Kconfig +1 -0

 config ACPI_HMAT
        bool "ACPI Heterogeneous Memory Attribute Table Support"
        depends on ACPI_NUMA
+       select HMEM_REPORTING

If you want to do this here, I'm not sure that defining HMEM_REPORTING as a user-selectable option is a good idea. In particular, I don't really think that setting ACPI_HMAT without it makes a lot of sense. Apart from this, the patch looks reasonable to me.

I guess the question is whether we would want to allow folks to consume the HMAT inside the kernel while not reporting it out via HMEM_REPORTING. We have some in-kernel users of the HMAT lined up like mitigations for memory-side caches.

It's certainly possible that folks would want to consume those mitigations without anything in sysfs. They might not even want or need NUMA support itself, for instance.

So, what should we do?

config HMEM_REPORTING
bool # no user-visible prompt
default y if ACPI_HMAT

Quoted 1 line

So folks can override in their .config, but they don't see a prompt?

Maybe it would be better to make HMEM_REPORTING do "select ACPI_HMAT if ACPI".

The mitigations could then do that too if they depend on HMAT and ACPI_HMAT need not be user-visible at all.

Quoted 33 lines · Rafael J. Wysocki

Quoted 31 lines · Dave Hansen

Quoted 5 lines · Rafael J. Wysocki

Quoted 5 lines

drivers/acpi/hmat/Kconfig +0 -0

index c9637e2e7514..08e972ead159 100644
@@ -2,6 +2,7 @@

Quoted 8 lines

drivers/acpi/hmat/Kconfig line 5

Quoted 4 lines

drivers/acpi/hmat/Kconfig +1 -0

 config ACPI_HMAT
        bool "ACPI Heterogeneous Memory Attribute Table Support"
        depends on ACPI_NUMA
+       select HMEM_REPORTING

If you want to do this here, I'm not sure that defining HMEM_REPORTING as a user-selectable option is a good idea. In particular, I don't really think that setting ACPI_HMAT without it makes a lot of sense. Apart from this, the patch looks reasonable to me.

I guess the question is whether we would want to allow folks to consume the HMAT inside the kernel while not reporting it out via HMEM_REPORTING. We have some in-kernel users of the HMAT lined up like mitigations for memory-side caches.

It's certainly possible that folks would want to consume those mitigations without anything in sysfs. They might not even want or need NUMA support itself, for instance.

So, what should we do?

config HMEM_REPORTING
bool # no user-visible prompt
default y if ACPI_HMAT

So folks can override in their .config, but they don't see a prompt?

Quoted 4 lines

Maybe it would be better to make HMEM_REPORTING do "select ACPI_HMAT if ACPI".

The mitigations could then do that too if they depend on HMAT and ACPI_HMAT need not be user-visible at all.

That sounds okay, though it would create unreachable code if !ACPI since that's the only user for the new reporting interfaces.

Quoted 4 lines · Keith Busch

config ACPI_HMAT
bool "ACPI Heterogeneous Memory Attribute Table Support" depends on ACPI_NUMA
+ select HMEM_REPORTING

If you want to do this here, I'm not sure that defining HMEM_REPORTING as a user-selectable option is a good idea. In particular, I don't really think that setting ACPI_HMAT without it makes a lot of sense. Apart from this, the patch looks reasonable to me.

Quoted 10 lines · Rafael J. Wysocki

Quoted 4 lines · Keith Busch

config ACPI_HMAT
bool "ACPI Heterogeneous Memory Attribute Table Support" depends on ACPI_NUMA
+ select HMEM_REPORTING

If you want to do this here, I'm not sure that defining HMEM_REPORTING as a user-selectable option is a good idea. In particular, I don't really think that setting ACPI_HMAT without it makes a lot of sense. Apart from this, the patch looks reasonable to me.

I'm trying to implement based on the feedback, but I'm a little confused.

As I have it at the moment, HMEM_REPORTING is not user-prompted, so another option needs to turn it on. I have ACPI_HMAT do that here.

So when you say it's a bad idea to make HMEM_REPORTING user selectable, isn't it already not user selectable?

If I do it the other way around, that's going to make HMEM_REPORTING complicated if a non-ACPI implementation wants to report HMEM properties.

Quoted 21 lines · Keith Busch

Quoted 10 lines · Rafael J. Wysocki

Quoted 4 lines · Keith Busch

config ACPI_HMAT
bool "ACPI Heterogeneous Memory Attribute Table Support" depends on ACPI_NUMA
+ select HMEM_REPORTING

If you want to do this here, I'm not sure that defining HMEM_REPORTING as a user-selectable option is a good idea. In particular, I don't really think that setting ACPI_HMAT without it makes a lot of sense. Apart from this, the patch looks reasonable to me.

I'm trying to implement based on the feedback, but I'm a little confused.

As I have it at the moment, HMEM_REPORTING is not user-prompted, so another option needs to turn it on. I have ACPI_HMAT do that here.

So when you say it's a bad idea to make HMEM_REPORTING user selectable, isn't it already not user selectable?

Quoted 3 lines

If I do it the other way around, that's going to make HMEM_REPORTING complicated if a non-ACPI implementation wants to report HMEM properties.

Agree. If a platform supports these HMEM properties then they should be reported.

ACPI_HMAT is that opt-in for ACPI based platforms, and other archs can do something similar. It's not clear that one would ever want to opt-in to HMAT support and opt-out of reporting any of it to userspace.

Quoted 10 lines · Rafael J. Wysocki

Quoted 4 lines · Keith Busch

config ACPI_HMAT
bool "ACPI Heterogeneous Memory Attribute Table Support" depends on ACPI_NUMA
+ select HMEM_REPORTING

If you want to do this here, I'm not sure that defining HMEM_REPORTING as a user-selectable option is a good idea. In particular, I don't really think that setting ACPI_HMAT without it makes a lot of sense. Apart from this, the patch looks reasonable to me.

I'm trying to implement based on the feedback, but I'm a little confused.

As I have it at the moment, HMEM_REPORTING is not user-prompted, so another option needs to turn it on. I have ACPI_HMAT do that here.

So when you say it's a bad idea to make HMEM_REPORTING user selectable, isn't it already not user selectable?

If I do it the other way around, that's going to make HMEM_REPORTING complicated if a non-ACPI implementation wants to report HMEM properties.

Quoted 3 lines · Keith Busch

If I do it the other way around, that's going to make HMEM_REPORTING complicated if a non-ACPI implementation wants to report HMEM properties.

But the mitigations that Dave was talking about get in the way, don't they?

Say there is another Kconfig option,CACHE_MITIGATIONS, to enable them. Then you want ACPI_HMAT to be set when that it set and you also want ACPI_HMAT to be set when HMEM_REPORTING and ACPI_NUMA are both set.

OTOH, you may not want HMEM_REPORTING to be set when CACHE_MITIGATIONS is set, but that causes ACPI_HMAT to be set and which means that ACPI_HMAT alone will not be sufficient to determine the HMEM_REPORTING value.

Now, if you prompt for HMEM_REPORTING and make it depend on ACPI_NUMA, then ACPI_HMAT can be selected by that (regardless of the CACHE_MITIGATIONS value).

And if someone wants to use HMEM_REPORTING without ACPI_NUMA, it can be made depend on whatever new option is there for that non-ACPI mechanism.

There might be a problem if someone wanted to enable the alternative way of HMEM_REPORTING if ACPI_NUMA was set (in which case HMAT would have to be ignored even if it was present), but in that case there would need to be an explicit way to choose between HMAT and non-HMAT anyway.

In any case, I prefer providers to be selected by consumers and not the other way around, in case there are multiple consumers for one provider.

Quoted 7 lines · Rafael J. Wysocki

Quoted 3 lines · Keith Busch

If I do it the other way around, that's going to make HMEM_REPORTING complicated if a non-ACPI implementation wants to report HMEM properties.

But the mitigations that Dave was talking about get in the way, don't they?

Quoted 8 lines

Say there is another Kconfig option,CACHE_MITIGATIONS, to enable them. Then you want ACPI_HMAT to be set when that it set and you also want ACPI_HMAT to be set when HMEM_REPORTING and ACPI_NUMA are both set.

OTOH, you may not want HMEM_REPORTING to be set when CACHE_MITIGATIONS is set, but that causes ACPI_HMAT to be set and which means that ACPI_HMAT alone will not be sufficient to determine the HMEM_REPORTING value.

I can't think of when we'd want to suppress reporting these attributes to user space, but I can split HMAT enabling so it doesn't depend on HMEM_REPORTING just in case there really is an in-kernel user that definitely does not want the same attributes exported.

Quoted 14 lines

Now, if you prompt for HMEM_REPORTING and make it depend on ACPI_NUMA, then ACPI_HMAT can be selected by that (regardless of the CACHE_MITIGATIONS value).

And if someone wants to use HMEM_REPORTING without ACPI_NUMA, it can be made depend on whatever new option is there for that non-ACPI mechanism.

There might be a problem if someone wanted to enable the alternative way of HMEM_REPORTING if ACPI_NUMA was set (in which case HMAT would have to be ignored even if it was present), but in that case there would need to be an explicit way to choose between HMAT and non-HMAT anyway.

Quoted 3 lines

In any case, I prefer providers to be selected by consumers and not the other way around, in case there are multiple consumers for one provider.

Well, the HMEM_REPORTING fundamentally has no dependency on any of these things and I've put some effort into making this part provider agnostic.

I will change it if this concern is gating acceptance, but I don't think it's as intuitive for generic interfaces to be the selector for implementation specific providers.

If the HMAT Subsystem Address Range provides a valid processor proximity domain for a memory domain, or a processor domain matches the performance access of the valid processor proximity domain, register the memory target with that initiator so this relationship will be visible under the node's sysfs directory.

By registering only the best performing relationships, this provides the most useful information applications may want to know when considering which CPU they should run on for a given memory node, or which memory node they should allocate memory from for a given CPU.

Since HMAT requires valid address ranges have an equivalent SRAT entry, verify each memory target satisfies this requirement.

Signed-off-by Keith Busch <[email protected]>

drivers/acpi/hmat/hmat.c +8 -0

+static __init void alloc_memory_initiator(unsigned int cpu_pxm)
+{
+	struct memory_initiator *intitator;
+
+	if (pxm_to_node(cpu_pxm) == NUMA_NO_NODE)
+		return;
+
+	intitator = find_mem_initiator(cpu_pxm);

Save the best performance access attributes and register these with the memory's node if HMAT provides the locality table. While HMAT does make it possible to know performance for all possible initiator-target pairings, we export only the local pairings at this time.

Signed-off-by Keith Busch <[email protected]>

---
 drivers/acpi/hmat/hmat.c | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)

drivers/acpi/hmat/hmat.c +16 -2

index b29f7160c7bb..6833c4897ff4 100644
@@ -549,12 +549,27 @@ static __init void hmat_register_target_initiators(struct memory_target *target)
        }
 }
 
+static __init void hmat_register_target_perf(struct memory_target *target)
+{
+       unsigned mem_nid = pxm_to_node(target->memory_pxm);
+
+       if (!target->hmem_attrs.read_bandwidth &&
+           !target->hmem_attrs.read_latency &&
+           !target->hmem_attrs.write_bandwidth &&
+           !target->hmem_attrs.write_latency)
+               return;
+
+       node_set_perf_attrs(mem_nid, &target->hmem_attrs, 0);
+}
+
 static __init void hmat_register_targets(void)
 {
        struct memory_target *target;
 
-       list_for_each_entry(target, &targets, node)
+       list_for_each_entry(target, &targets, node) {
                hmat_register_target_initiators(target);
+               hmat_register_target_perf(target);
+       }
 }
 
 static __init void hmat_free_structures(void)
--

2.14.4

Register memory side cache attributes with the memory's node if HMAT provides the side cache iniformation table.

Signed-off-by Keith Busch <[email protected]>

---
 drivers/acpi/hmat/hmat.c | 32 ++++++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)

drivers/acpi/hmat/hmat.c +32 -1

index 6833c4897ff4..e2a15f53fe45 100644
@@ -314,6 +314,7 @@ static __init int hmat_parse_cache(union acpi_subtable_headers *header,
                                   const unsigned long end)
 {
        struct acpi_hmat_cache *cache = (void *)header;
+       struct node_cache_attrs cache_attrs;
        u32 attrs;
 
        if (cache->header.length < sizeof(*cache)) {

@@ -327,6 +328,37 @@ static __init int hmat_parse_cache(union acpi_subtable_headers *header,
                cache->memory_PD, cache->cache_size, attrs,
                cache->number_of_SMBIOShandles);
 
+       cache_attrs.size = cache->cache_size;
+       cache_attrs.level = (attrs & ACPI_HMAT_CACHE_LEVEL) >> 4;
+       cache_attrs.line_size = (attrs & ACPI_HMAT_CACHE_LINE_SIZE) >> 16;
+
+       switch ((attrs & ACPI_HMAT_CACHE_ASSOCIATIVITY) >> 8) {
+       case ACPI_HMAT_CA_DIRECT_MAPPED:
+               cache_attrs.associativity = NODE_CACHE_DIRECT_MAP;
+               break;
+       case ACPI_HMAT_CA_COMPLEX_CACHE_INDEXING:
+               cache_attrs.associativity = NODE_CACHE_INDEXED;
+               break;
+       case ACPI_HMAT_CA_NONE:
+       default:
+               cache_attrs.associativity = NODE_CACHE_OTHER;
+               break;
+       }
+
+       switch ((attrs & ACPI_HMAT_WRITE_POLICY) >> 12) {
+       case ACPI_HMAT_CP_WB:
+               cache_attrs.write_policy = NODE_CACHE_WRITE_BACK;
+               break;
+       case ACPI_HMAT_CP_WT:
+               cache_attrs.write_policy = NODE_CACHE_WRITE_THROUGH;
+               break;
+       case ACPI_HMAT_CP_NONE:
+       default:
+               cache_attrs.write_policy = NODE_CACHE_WRITE_OTHER;
+               break;
+       }
+
+       node_add_cache(pxm_to_node(cache->memory_PD), &cache_attrs);
        return 0;
 }
 
--

2.14.4

== Changes since v5 ==

Updated HMAT parsing to account for the recently released ACPI 6.3 changes.

HMAT attribute calculation overflow checks.

Fixed memory leak if HMAT parse fails.

Minor change to the patch order. All the base node attributes occur before HMAT usage for these new node attributes to resolve a dependency on a new struct.

Reporting failures to parse HMAT or allocate structures are elevated to a NOTICE level from DEBUG. Any failure will result in just one print so that it is obvious something may need to be investigated rather than silently fail, but also not to be too alarming either.

Determining the cpu and memory node local relationships is quite different this time (PATCH 7/10). The local relationship to a memory target will be either *only* the node from the Initiator Proximity Domain if provided, or if it is not provided, all the nodes that have the same highest performance. Latency was chosen to take prioirty over bandwidth when ranking performance.

Le 14/02/2019 à 18:10, Keith Busch a écrit :

Quoted 6 lines

Determining the cpu and memory node local relationships is quite different this time (PATCH 7/10). The local relationship to a memory target will be either *only* the node from the Initiator Proximity Domain if provided, or if it is not provided, all the nodes that have the same highest performance. Latency was chosen to take prioirty over bandwidth when ranking performance.

Hello Keith

I am trying to understand what this last paragraph means.

Let's say I have a machine with DDR and NVDIMM both attached to the same socket, and I use Dave Hansen's kmem patchs to make the NVDIMM appear as "normal memory" in an additional NUMA node. Let's call node0 the DDR and node1 the NVDIMM kmem node.

Now user-space wants to find out which CPUs are actually close to the NVDIMMs. My understanding is that SRAT says that CPUs are local to the DDR only. Hence /sys/devices/system/node/node1/cpumap says there are no CPU local to the NVDIMM. And HMAT won't change this, right?

Will node1 contain access0/initiators/node0 to clarify that CPUs local to the NVDIMM are those of node0? Even if latency from node0 to node1 latency is higher than node0 to node0?

Another way to ask this: Is the latency/performance only used for distinguishing the local initiator CPUs among multiple CPU nodes accesing the same memory node? Or is it also used to distinguish the local memory target among multiple memories access by a single CPU node?

The Intel machine I am currently testing patches on doesn't have a HMAT in 1-level-memory unfortunately.

Platforms may provide multiple types of cpu attached system memory. The memory ranges for each type may have different characteristics that applications may wish to know about when considering what node they want their memory allocated from.

It had previously been difficult to describe these setups as memory rangers were generally lumped into the NUMA node of the CPUs. New platform attributes have been created and in use today that describe the more complex memory hierarchies that can be created.

This series' objective is to provide the attributes from such systems that are useful for applications to know about, and readily usable with existing tools and libraries. Those applications may query performance attributes relative to a particular CPU they're running on in order to make more informed choices for where they want to allocate hot and cold data. This works with mbind() or the numactl library.

Keith Busch (10):
acpi: Create subtable parsing infrastructure
acpi: Add HMAT to generic parsing tables acpi/hmat: Parse and report heterogeneous memory
node: Link memory nodes to their compute nodes
node: Add heterogenous memory access attributes
node: Add memory-side caching attributes acpi/hmat: Register processor domain to its memory acpi/hmat: Register performance attributes acpi/hmat: Register memory side cache attributes doc/mm: New documentation for memory performance

 Documentation/ABI/stable/sysfs-devices-node   |  89 +++-
 Documentation/admin-guide/mm/numaperf.rst     | 164 +++++++
 arch/arm64/kernel/acpi_numa.c                 |   2 +-
 arch/arm64/kernel/smp.c                       |   4 +-
 arch/ia64/kernel/acpi.c                       |  12 +-
 arch/x86/kernel/acpi/boot.c                   |  36 +-
 drivers/acpi/Kconfig                          |   1 +
 drivers/acpi/Makefile                         |   1 +
 drivers/acpi/hmat/Kconfig                     |   9 +
 drivers/acpi/hmat/Makefile                    |   1 +
 drivers/acpi/hmat/hmat.c                      | 677 ++++++++++++++++++++++++++
 drivers/acpi/numa.c                           |  16 +-
 drivers/acpi/scan.c                           |   4 +-
 drivers/acpi/tables.c                         |  76 ++-
 drivers/base/Kconfig                          |   8 +
 drivers/base/node.c                           | 351 ++++++++++++-
 drivers/irqchip/irq-gic-v2m.c                 |   2 +-
 drivers/irqchip/irq-gic-v3-its-pci-msi.c      |   2 +-
 drivers/irqchip/irq-gic-v3-its-platform-msi.c |   2 +-
 drivers/irqchip/irq-gic-v3-its.c              |   6 +-
 drivers/irqchip/irq-gic-v3.c                  |  10 +-
 drivers/irqchip/irq-gic.c                     |   4 +-
 drivers/mailbox/pcc.c                         |   2 +-
 include/linux/acpi.h                          |   6 +-
 include/linux/node.h                          |  60 ++-
 25 files changed, 1480 insertions(+), 65 deletions(-)
 create mode 100644 Documentation/admin-guide/mm/numaperf.rst
 create mode 100644 drivers/acpi/hmat/Kconfig
 create mode 100644 drivers/acpi/hmat/Makefile
 create mode 100644 drivers/acpi/hmat/hmat.c