Externalize SLIT table
The SLIT table provides useful information on internode distances. Has anyone considered externalizing this table via /proc or some equivalent mechanism.
For example, something like the following would be useful:
# cat /proc/acpi/slit 010 066 046 066 066 010 066 046 046 066 010 020 066 046 020 010
If this looks ok (or something equivalent), I'll generate a patch....
Signature · 4 lines
Thanks Jack Steiner ([email protected]) 651-683-5302 Principal Engineer SGI - Silicon Graphics, Inc.
Hi,
For wider audience, added LKML.
From: Jack Steiner <[email protected]>
Subject: Externalize SLIT table
Date: Wed, 3 Nov 2004 14:56:56 -0600
Quoted 6 lines
The SLIT table provides useful information on internode distances. Has anyone considered externalizing this table via /proc or some equivalent mechanism.
For example, something like the following would be useful:
Quoted 7 lines
# cat /proc/acpi/slit 010 066 046 066 066 010 066 046 046 066 010 020 066 046 020 010If this looks ok (or something equivalent), I'll generate a patch....
For user space to manipulate scheduling domains, pinning processes to some cpu groups etc, that kind of information is very useful! Without this, users have no notion about how far between two nodes.
But ACPI SLIT table is too arch specific (ia64 and x86 only) and user-visible logical number and ACPI proximity domain number is not always identical.
Why not export node_distance() under sysfs?
I like (1).
(1) obey one-value-per-file sysfs principle
% cat /sys/devices/system/node/node0/distance0
10
% cat /sys/devices/system/node/node0/distance1
66
(2) one distance for each line
% cat /sys/devices/system/node/node0/distance
0:10
1:66
2:46
3:66
(3) all distances in one line like /proc/<PID>/stat
% cat /sys/devices/system/node/node0/distance
10 66 46 66
---
Takayoshi Kochi
Quoted 21 lines · Takayoshi Kochi
Hi,
For wider audience, added LKML.
From: Jack Steiner <[email protected]>
Subject: Externalize SLIT table
Date: Wed, 3 Nov 2004 14:56:56 -0600Quoted 13 lines
The SLIT table provides useful information on internode distances. Has anyone considered externalizing this table via /proc or some equivalent mechanism.
For example, something like the following would be useful:
# cat /proc/acpi/slit 010 066 046 066 066 010 066 046 046 066 010 020 066 046 020 010If this looks ok (or something equivalent), I'll generate a patch....
This isn't very useful without information about proximity domains. e.g. on x86-64 the proximity domain number is not necessarily the same as the node number.
Quoted 3 lines
For user space to manipulate scheduling domains, pinning processes to some cpu groups etc, that kind of information is very useful! Without this, users have no notion about how far between two nodes.
Also some reporting of _PXM for PCI devices is needed. I had a experimental patch for this on x86-64 (not ACPI based), that reported nearby nodes for PCI busses.
Quoted 4 lines
But ACPI SLIT table is too arch specific (ia64 and x86 only) and user-visible logical number and ACPI proximity domain number is not always identical.
Exactly.
Quoted 8 lines
Why not export
node_distance()under sysfs?
I like (1).(1) obey one-value-per-file sysfs principle
% cat
/sys/devices/system/node/node0/distance0
10
Surely distance from 0 to 0 is 0?
Quoted 12 lines
% cat
/sys/devices/system/node/node0/distance1
66(2) one distance for each line
% cat
/sys/devices/system/node/node0/distance
0:10
1:66
2:46
3:66
Quoted 4 lines
(3) all distances in one line like /proc/<PID>/stat
% cat
/sys/devices/system/node/node0/distance
10 66 46 66
I would prefer that.
-Andi
Hi,
From: Andi Kleen <[email protected]>
Subject: Re: Externalize SLIT table
Date: Thu, 4 Nov 2004 05:07:13 +0100
Quoted 9 lines
Quoted 7 lines
Why not export
node_distance()under sysfs?
I like (1).(1) obey one-value-per-file sysfs principle
% cat
/sys/devices/system/node/node0/distance0
10Surely distance from 0 to 0 is 0?
According to the ACPI spec, 10 means local and other values mean ratio to 10. But what the distance number should mean mean is ambiguous from the spec (e.g. some veondors interpret as memory access latency, others interpret as memory throughput etc.)
However relative distance just works for most of uses, I believe.
Anyway, we should clarify how the numbers should be interpreted to avoid confusion.
How about this?
"The distance to itself means the base value. Distance to other nodes are relative to the base value.
0 means unreachable (hot-removed or disabled) to that node."
(Just FYI, numbers 0-9 are reserved and 255 (unsigned char -1) means unreachable, according to the ACPI spec.)
Quoted 12 lines
Quoted 12 lines
% cat
/sys/devices/system/node/node0/distance1
66(2) one distance for each line
% cat
/sys/devices/system/node/node0/distance
0:10
1:66
2:46
3:66
Quoted 6 lines
Quoted 4 lines
(3) all distances in one line like /proc/<PID>/stat
% cat
/sys/devices/system/node/node0/distance
10 66 46 66I would prefer that.
Ah, I missed the following last sentence inDocumentation/filesystems/sysfs.txt:
|Attributes should be ASCII text files, preferably with only one value |per file. It is noted that it may not be efficient to contain only |value per file, so it is socially acceptable to express an array of |values of the same type.
If an array is acceptable, I would prefer (3), too.
---
Takayoshi Kochi
Quoted 11 lines · Takayoshi Kochi
Hi,
From: Andi Kleen <[email protected]>
Subject: Re: Externalize SLIT table
Date: Thu, 4 Nov 2004 05:07:13 +0100Quoted 5 lines
Quoted 5 lines
Why not export
node_distance()under sysfs?
I like (1).(1) obey one-value-per-file sysfs principle
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
% cat
/sys/devices/system/node/node0/distance0
10Surely distance from 0 to 0 is 0?
According to the ACPI spec, 10 means local and other values mean ratio to 10. But what the distance number should mean
Ah, missed that. ok I guess it makes sense to use the same encoding as ACPI, no need to be intentionally different.
Quoted 7 lines
mean is ambiguous from the spec (e.g. some veondors interpret as memory access latency, others interpret as memory throughput etc.)
However relative distance just works for most of uses, I believe.Anyway, we should clarify how the numbers should be interpreted to avoid confusion.
Defining it as "as defined in the ACPI spec" should be ok. I guess even non ACPI architectures will be able to live with that.
Anyways, since we seem to agree and so far nobody has complained it's just that somebody needs to do a patch? If possible make it generic code in drivers/acpi/numa.c, there won't be anything architecture specific in this and it should work for x86-64 too.
-Andi
Quoted 33 lines · Takayoshi Kochi
Hi,
For wider audience, added LKML.
From: Jack Steiner <[email protected]>
Subject: Externalize SLIT table
Date: Wed, 3 Nov 2004 14:56:56 -0600Quoted 6 lines
The SLIT table provides useful information on internode distances. Has anyone considered externalizing this table via /proc or some equivalent mechanism.
For example, something like the following would be useful:
Quoted 7 lines
# cat /proc/acpi/slit 010 066 046 066 066 010 066 046 046 066 010 020 066 046 020 010If this looks ok (or something equivalent), I'll generate a patch....
For user space to manipulate scheduling domains, pinning processes to some cpu groups etc, that kind of information is very useful! Without this, users have no notion about how far between two nodes.
But ACPI SLIT table is too arch specific (ia64 and x86 only) and user-visible logical number and ACPI proximity domain number is not always identical.
Why not export
node_distance()under sysfs?
I like (1).
Quoted 6 lines
(1) obey one-value-per-file sysfs principle
% cat
/sys/devices/system/node/node0/distance0
10
% cat/sys/devices/system/node/node0/distance1
66
I'm not familar with the internals of sysfs. For example, on a 256 node system, there will be 65536 instances of
/sys/devices/system/node/node<M>/distance<N>
Does this require any significant amount of kernel resources to maintain this amount of information.
Quoted 9 lines
(2) one distance for each line
% cat
/sys/devices/system/node/node0/distance
0:10
1:66
2:46
3:66
Quoted 5 lines
(3) all distances in one line like /proc/<PID>/stat
% cat
/sys/devices/system/node/node0/distance
10 66 46 66
I like (3) the best.
I think it would also be useful to have a similar cpu-to-cpu distance metric:
% cat /sys/devices/system/cpu/cpu0/distance 10 20 40 60
This gives the same information but is cpu-centric rather than node centric.
Signature · 4 lines
Thanks Jack Steiner ([email protected]) 651-683-5302 Principal Engineer SGI - Silicon Graphics, Inc.
Quoted 41 lines · Jack Steiner
Quoted 33 lines · Takayoshi Kochi
Hi,
For wider audience, added LKML.
From: Jack Steiner <[email protected]>
Subject: Externalize SLIT table
Date: Wed, 3 Nov 2004 14:56:56 -0600Quoted 6 lines
The SLIT table provides useful information on internode distances. Has anyone considered externalizing this table via /proc or some equivalent mechanism.
For example, something like the following would be useful:
Quoted 7 lines
# cat /proc/acpi/slit 010 066 046 066 066 010 066 046 046 066 010 020 066 046 020 010If this looks ok (or something equivalent), I'll generate a patch....
For user space to manipulate scheduling domains, pinning processes to some cpu groups etc, that kind of information is very useful! Without this, users have no notion about how far between two nodes.
But ACPI SLIT table is too arch specific (ia64 and x86 only) and user-visible logical number and ACPI proximity domain number is not always identical.
Why not export
node_distance()under sysfs?
I like (1).Quoted 6 lines
(1) obey one-value-per-file sysfs principle
% cat
/sys/devices/system/node/node0/distance0
10
% cat/sys/devices/system/node/node0/distance1
66
Quoted 6 lines
I'm not familar with the internals of sysfs. For example, on a 256 node system, there will be 65536 instances of
/sys/devices/system/node/node<M>/distance<N>Does this require any significant amount of kernel resources to maintain this amount of information.
Yes it does, even with the new sysfs backing store. And reading it would create all the inodes and dentries, which are quite bloated.
Quoted 8 lines
I think it would also be useful to have a similar cpu-to-cpu distance metric:
% cat /sys/devices/system/cpu/cpu0/distance 10 20 40 60This gives the same information but is cpu-centric rather than node centric.
And the same thing for PCI busses, like in this patch. However for strict ACPI systems this information would need to be gotten from _PXM first. x86-64 on Opteron currently reads it directly from the hardware and uses it to allocate DMA memory near the device.
-Andi
drivers/pci/pci-sysfs.c +11 -0
@@ -17,6 +17,7 @@#include <linux/kernel.h> #include <linux/pci.h> #include <linux/stat.h> +#include <linux/topology.h> #include "pci.h"
@@ -38,6 +39,15 @@ pci_config_attr(subsystem_device, "0x%04pci_config_attr (class,"0x%06x\n" );pci_config_attr (irq,"%u\n" ); +static ssize_t local_cpus_show (struct device *dev,char *buf) +{ +struct pci_dev *pdev =to_pci_dev (dev); +cpumask_t mask =pcibus_to_cpumask (pdev->bus->number); +int len =cpumask_scnprintf (buf,PAGE_SIZE -1 , mask); +strcat (buf,"\n" ); +return 1 +len; +} +/* show resources */ static ssize_t resource_show (struct device * dev,char * buf)
@@ -67,6 +77,7 @@ struct device_attribute pci_dev_attrs[]__ATTR_RO (subsystem_device),__ATTR_RO (class),__ATTR_RO (irq), +__ATTR_RO (local_cpus), __ATTR_NULL, };
Quoted 7 lines · Jack Steiner
I think it would also be useful to have a similar cpu-to-cpu distance metric:
% cat/sys/devices/system/cpu/cpu0/distance
10 20 40 60This gives the same information but is cpu-centric rather than node centric.
I don't see the use of that once you have some way to find the logical CPU to node number mapping. The "node distances" are meant to be proportional to the memory access latency ratios (20 means 2 times larger than local (intra-node) access, which is by definition 10). If the cpu_to_cpu distance is necessary because there is a hierarchy in the memory blocks inside one node, then maybe the definition of a node should be changed...
We currently have (at least in -mm kernels):
% ls /sys/devices/system/node/node0/cpu*
for finding out which CPUs belong to which nodes. Together with
/sys/devices/system/node/node0/distances
this should be enough for user space NUMA tools.
Regards,
Erich
Quoted 11 lines · Erich Focht
Quoted 7 lines · Jack Steiner
I think it would also be useful to have a similar cpu-to-cpu distance metric:
????????% cat/sys/devices/system/cpu/cpu0/distance????????10 20 40 60This gives the same information but is cpu-centric rather than node centric.
I don't see the use of that once you have some way to find the logical CPU to node number mapping. The "node distances" are meant to be
I think he wants it just to have a more convenient interface, which is not necessarily a bad thing. But then one could put the convenience into libnuma anyways.
-Andi
Quoted 10 lines · Andi Kleen
Quoted 9 lines · Erich Focht
Quoted 7 lines · Jack Steiner
I think it would also be useful to have a similar cpu-to-cpu distance metric:
????????% cat/sys/devices/system/cpu/cpu0/distance????????10 20 40 60This gives the same information but is cpu-centric rather than node centric.
Quoted 8 lines
Quoted 2 lines
I don't see the use of that once you have some way to find the logical CPU to node number mapping. The "node distances" are meant to be
I think he wants it just to have a more convenient interface, which is not necessarily a bad thing. But then one could put the convenience into libnuma anyways.
-Andi
Yes, strictly convenience. Most of the cases that I have seen deal with cpu placement & cpu distances from each other. I agree that cpu-to-cpu distances can be determined by converting to nodes & finding the node-to-node distance.
A second reason is symmetry. If there is a /sys/devices/system/node/node0/distance metric, it seems as though there should also be a /sys/devices/system/cpu/cpu0/distance metric.
Signature · 4 lines
Thanks Jack Steiner ([email protected]) 651-683-5302 Principal Engineer SGI - Silicon Graphics, Inc.
Based on the ideas from Andi & Takayoshi, I created a patch to add the SLIT distance information to the sysfs.
I've tested this on Altix/IA64 & it appears to work ok. I have not tried it on other architectures.
Andi also posted a related patch for adding similar information for PCI busses.
Comments, suggestions, .....
# cd /sys/devices/system # find . ./node ./node/node5 ./node/node5/cpu11 ./node/node5/cpu10 ./node/node5/distance ./node/node5/numastat ./node/node5/meminfo ./node/node5/cpumap ./node/node4 ./node/node4/cpu9 ./node/node4/cpu8 ./node/node4/distance ./node/node4/numastat ./node/node4/meminfo ./node/node4/cpumap .... ./cpu ./cpu/cpu11 ./cpu/cpu11/distance ./cpu/cpu10 ./cpu/cpu10/distance ./cpu/cpu9 ./cpu/cpu9/distance ./cpu/cpu8 ...
# cat ./node/node0/distance 10 20 64 42 42 22
# cat ./cpu/cpu8/distance 42 42 64 64 22 22 42 42 10 10 20 20
# cat node/*/distance 10 20 64 42 42 22 20 10 42 22 64 84 64 42 10 20 22 42 42 22 20 10 42 62 42 64 22 42 10 20 22 84 42 62 20 10
# cat cpu/*/distance 10 10 20 20 64 64 42 42 42 42 22 22 10 10 20 20 64 64 42 42 42 42 22 22 20 20 10 10 42 42 22 22 64 64 84 84 20 20 10 10 42 42 22 22 64 64 84 84 64 64 42 42 10 10 20 20 22 22 42 42 64 64 42 42 10 10 20 20 22 22 42 42 42 42 22 22 20 20 10 10 42 42 62 62 42 42 22 22 20 20 10 10 42 42 62 62 42 42 64 64 22 22 42 42 10 10 20 20 42 42 64 64 22 22 42 42 10 10 20 20 22 22 84 84 42 42 62 62 20 20 10 10 22 22 84 84 42 42 62 62 20 20 10 10
Index: linux/drivers/base/node.c
=================================--- linux.orig/drivers/base/node.c 2004-11-05 08:34:42.000000000 -0600
+++ linux/drivers/base/node.c 2004-11-05 09:00:01.000000000 -0600
@@ -111,6 +111,21 @@ static ssize_t node_read_numastat(struct
}
static SYSDEV_ATTR(numastat, S_IRUGO, node_read_numastat, NULL);
+static ssize_t node_read_distance(struct sys_device * dev, char * buf) +{
+ int nid = dev->id;
+ int len = 0;
+ int i;
+
+ for (i = 0; i < numnodes; i++)
+ len += sprintf(buf + len, "%s%d", i ? " " : "", node_distance(nid, i));
+
+ len += sprintf(buf + len, "\n");
+ return len;
+}
+static SYSDEV_ATTR(distance, S_IRUGO, node_read_distance, NULL); +
+
/*
* register_node - Setup a driverfs device for a node.
* @num - Node number to use when creating the device.
@@ -129,6 +144,7 @@ int __init register_node(struct node *no
sysdev_create_file(&node->sysdev, &attr_cpumap); sysdev_create_file(&node->sysdev, &attr_meminfo); sysdev_create_file(&node->sysdev, &attr_numastat);
+ sysdev_create_file(&node->sysdev, &attr_distance);
}
return error;
}
Index: linux/drivers/base/cpu.c
=================================--- linux.orig/drivers/base/cpu.c 2004-11-05 08:58:09.000000000 -0600
+++ linux/drivers/base/cpu.c 2004-11-05 08:59:25.000000000 -0600
@@ -8,6 +8,7 @@
#include <linux/cpu.h>
#include <linux/topology.h>
#include <linux/device.h>
+#include <linux/cpumask.h>
struct sysdev_class cpu_sysdev_class = {
@@ -58,6 +59,31 @@ static inline void register_cpu_control(
}
#endif /* CONFIG_HOTPLUG_CPU */
+#ifdef CONFIG_NUMA
+static ssize_t cpu_read_distance(struct sys_device * dev, char * buf) +{
+ int nid = cpu_to_node(dev->id);
+ int len = 0;
+ int i;
+
+ for (i = 0; i < num_possible_cpus(); i++)
+ len += sprintf(buf + len, "%s%d", i ? " " : "",
+ node_distance(nid, cpu_to_node(i)));
+ len += sprintf(buf + len, "\n");
+ return len;
+}
+static SYSDEV_ATTR(distance, S_IRUGO, cpu_read_distance, NULL); +
+static inline void register_cpu_distance(struct cpu *cpu)
+{
+ sysdev_create_file(&cpu->sysdev, &attr_distance);
+}
+#else /* !CONFIG_NUMA */
+static inline void register_cpu_distance(struct cpu *cpu)
+{
+}
+#endif
+
/*
* register_cpu - Setup a driverfs device for a CPU.
* @cpu - Callers can set the cpu->no_control field to 1, to indicate not to
@@ -81,6 +107,10 @@ int __init register_cpu(struct cpu *cpu,
kobject_name(&cpu->sysdev.kobj));
if (!error && !cpu->no_control)
register_cpu_control(cpu);
+
+ if (!error)
+ register_cpu_distance(cpu);
+
return error;
}
Quoted 65 lines · Takayoshi Kochi
Hi,
From: Andi Kleen <[email protected]>
Subject: Re: Externalize SLIT table
Date: Thu, 4 Nov 2004 05:07:13 +0100Quoted 9 lines
Quoted 7 lines
Why not export
node_distance()under sysfs?
I like (1).(1) obey one-value-per-file sysfs principle
% cat
/sys/devices/system/node/node0/distance0
10Surely distance from 0 to 0 is 0?
According to the ACPI spec, 10 means local and other values mean ratio to 10. But what the distance number should mean mean is ambiguous from the spec (e.g. some veondors interpret as memory access latency, others interpret as memory throughput etc.)
However relative distance just works for most of uses, I believe.Anyway, we should clarify how the numbers should be interpreted to avoid confusion.
How about this?
"The distance to itself means the base value. Distance to other nodes are relative to the base value.
0 means unreachable (hot-removed or disabled) to that node."(Just FYI, numbers 0-9 are reserved and 255 (unsigned char -1) means unreachable, according to the ACPI spec.)
Quoted 12 lines
Quoted 12 lines
% cat
/sys/devices/system/node/node0/distance1
66(2) one distance for each line
% cat
/sys/devices/system/node/node0/distance
0:10
1:66
2:46
3:66Quoted 6 lines
Quoted 4 lines
(3) all distances in one line like /proc/<PID>/stat
% cat
/sys/devices/system/node/node0/distance
10 66 46 66I would prefer that.
Ah, I missed the following last sentence in
Documentation/filesystems/sysfs.txt:|Attributes should be ASCII text files, preferably with only one value |per file. It is noted that it may not be efficient to contain only |value per file, so it is socially acceptable to express an array of |values of the same type.If an array is acceptable, I would prefer (3), too.
---
Takayoshi Kochi
Signature · 4 lines
Thanks Jack Steiner ([email protected]) 651-683-5302 Principal Engineer SGI - Silicon Graphics, Inc.
Review static ssize_t node_read_numastat(struct · line 121
Quoted 12 lines · Jack Steiner
(quoted diff) +8
-0@@ -111,6 +111,21 @@ static ssize_t node_read_numastat(struct } static SYSDEV_ATTR(numastat, S_IRUGO, node_read_numastat, NULL); +static ssize_t node_read_distance(struct sys_device * dev, char * buf) +{ + int nid = dev->id; + int len = 0; + int i; + + for (i = 0; i < numnodes; i++) + len += sprintf(buf + len, "%s%d", i ? " " : "", node_distance(nid, i));
Can this overflow the space allocated for buf?
Quoted 4 lines
(quoted diff) +0
-0@@ -58,6 +59,31 @@ static inline void register_cpu_control( } #endif /* CONFIG_HOTPLUG_CPU */
Review static inline void register_cpu_control( · line 71
Quoted 10 lines
(quoted diff) +10
-0+#ifdef CONFIG_NUMA +static ssize_t cpu_read_distance(struct sys_device * dev, char * buf) +{ + int nid = cpu_to_node(dev->id); + int len = 0; + int i; + + for (i = 0; i < num_possible_cpus(); i++) + len += sprintf(buf + len, "%s%d", i ? " " : "", + node_distance(nid, cpu_to_node(i)));
Or this?
Andreas.
Signature · 4 lines
Andreas Schwab, SuSE Labs, [email protected] SuSE Linux AG, Maxfeldstraße 5, 90409 Nürnberg, Germany Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5 "And now for something completely different."
Quoted 6 lines · Andreas Schwab
Quoted 4 lines · Jack Steiner
(quoted diff) +0
-0@@ -111,6 +111,21 @@ static ssize_t node_read_numastat(struct } static SYSDEV_ATTR(numastat, S_IRUGO, node_read_numastat, NULL);
Quoted 10 lines
Review static ssize_t node_read_numastat(struct · line 121
Quoted 8 lines
(quoted diff) +8
-0+static ssize_t node_read_distance(struct sys_device * dev, char * buf) +{ + int nid = dev->id; + int len = 0; + int i; + + for (i = 0; i < numnodes; i++) + len += sprintf(buf + len, "%s%d", i ? " " : "", node_distance(nid, i));Can this overflow the space allocated for buf?
Good point. I think we are ok for now. AFAIK, the largest cpu count currently supported is 512. That gives a max string of 2k (max of 3 digits + space per cpu).
However, I should probably add a BUILD_BUG_ON to check for overflow.
BUILD_BUG_ON(NR_NODES*4 > PAGE_SIZE/2); BUILD_BUG_ON(NR_CPUS*4 > PAGE_SIZE/2);
Quoted 25 lines
Quoted 5 lines
(quoted diff) +0
-0@@ -58,6 +59,31 @@ static inline void register_cpu_control( } #endif /* CONFIG_HOTPLUG_CPU */Review static inline void register_cpu_control( · line 71
Quoted 10 lines
(quoted diff) +10
-0+#ifdef CONFIG_NUMA +static ssize_t cpu_read_distance(struct sys_device * dev, char * buf) +{ + int nid = cpu_to_node(dev->id); + int len = 0; + int i; + + for (i = 0; i < num_possible_cpus(); i++) + len += sprintf(buf + len, "%s%d", i ? " " : "", + node_distance(nid, cpu_to_node(i)));Or this?
Andreas.
Signature · 4 lines
Andreas Schwab, SuSE Labs, [email protected] SuSE Linux AG, Maxfeldstra_e 5, 90409 N|rnberg, Germany Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5 "And now for something completely different."
Signature · 4 lines
Thanks Jack Steiner ([email protected]) 651-683-5302 Principal Engineer SGI - Silicon Graphics, Inc.
Hi Jack,
the patch looks fine, of course.
Quoted 2 lines
# cat ./node/node0/distance 10 20 64 42 42 22
Great!
But:
Quoted 2 lines
# cat ./cpu/cpu8/distance 42 42 64 64 22 22 42 42 10 10 20 20
...
what exactly do you mean by cpu_to_cpu distance? In analogy with the node distance I'd say it is the time (latency) for moving data from the register of one CPU into the register of another CPU:
cpu*/distance : cpu -> memory -> cpu
node1 node? node2
On most architectures this means flushing a cacheline to memory on one side and reading it on another side. What you actually implement is the latency from memory (one node) to a particular cpu (on some node).
memory -> cpu
node1 node2That's only half of the story and actually misleading. I don't think the complexity hiding is good in this place. Questions coming to my mind are: Where is the memory? Is the SLIT matrix really symmetric (cpu_to_cpu distance only makes sense for symmetric matrices)? I remember talking to IBM people about hardware where the node distance matrix was asymmetric.
Why do you want this distance anyway? libnuma offers you _node_ masks for allocating memory from a particular node. And when you want to arrange a complex MPI process structure you'll have to think about latency for moving data from one processes buffer to the other processes buffer. The buffers live on nodes, not on cpus.
Regards,
Erich
Quoted 10 lines · Erich Focht
Hi Jack,
the patch looks fine, of course.
Quoted 2 lines
# cat ./node/node0/distance 10 20 64 42 42 22Great!
But:
Quoted 2 lines
# cat ./cpu/cpu8/distance 42 42 64 64 22 22 42 42 10 10 20 20
Quoted 8 lines
...
what exactly do you mean by
cpu_to_cpudistance? In analogy with the node distance I'd say it is the time (latency) for moving data from the register of one CPU into the register of another CPU:
cpu*/distance : cpu -> memory -> cpu
node1 node? node2
I'm trying to create an easy-to-use metric for finding sets of cpus that are close to each other. By "close", I mean that the average offnode reference from a cpu to remote memory in the set is minimized.
The numbers in cpuN/distance represent the distance from cpu N to the memory that is local to each of the other cpus.
I agree that this can be derived from converting cpuN->node, finding internode distances, then finding the cpus on each remote node. The cpu metric is much easier to use.
Quoted 6 lines
On most architectures this means flushing a cacheline to memory on one side and reading it on another side. What you actually implement is the latency from memory (one node) to a particular cpu (on some node).
memory -> cpu node1 node2
I see how the term can be misleading. The metric is intended to represent ONLY the cost of remote access to another processor's local memory. Is there a better way to describe the cpu-to-remote-cpu's-memory metric OR should we let users contruct their own matrix from the node data?
Quoted 8 lines
That's only half of the story and actually misleading. I don't think the complexity hiding is good in this place. Questions coming to my mind are: Where is the memory? Is the SLIT matrix really symmetric (
cpu_to_cpudistance only makes sense for symmetric matrices)? I remember talking to IBM people about hardware where the node distance matrix was asymmetric.
Quoted 5 lines
Why do you want this distance anyway? libnuma offers you _node_ masks for allocating memory from a particular node. And when you want to arrange a complex MPI process structure you'll have to think about latency for moving data from one processes buffer to the other processes buffer. The buffers live on nodes, not on cpus.
One important use is in the creation of cpusets. The batch scheduler needs to pick a subset of cpus that are as close together as possible.
Signature · 4 lines
Thanks Jack Steiner ([email protected]) 651-683-5302 Principal Engineer SGI - Silicon Graphics, Inc.
Quoted 9 lines · Jack Steiner
Quoted 4 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ for (i = 0; i < numnodes; i++) + len += sprintf(buf + len, "%s%d", i ? " " : "", node_distance(nid, i));Can this overflow the space allocated for buf?
Good point. I think we are ok for now. AFAIK, the largest cpu count currently supported is 512. That gives a max string of 2k (max of 3 digits + space per cpu).
I always wondered why sysfs doesn't use the seq_file interface that makes life easier in the rest of them kernel.
Quoted 7 lines · Christoph Hellwig
Quoted 6 lines · Jack Steiner
Quoted 4 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ for (i = 0; i < numnodes; i++) + len += sprintf(buf + len, "%s%d", i ? " " : "", node_distance(nid, i));Can this overflow the space allocated for buf?
Quoted 6 lines
Quoted 3 lines
Good point. I think we are ok for now. AFAIK, the largest cpu count currently supported is 512. That gives a max string of 2k (max of 3 digits + space per cpu).
I always wondered why sysfs doesn't use the
seq_fileinterface that makes life easier in the rest of them kernel.
Most fields only output a single number, and seq_file would be extreme overkill for that.
-Andi
Quoted 12 lines · Andi Kleen
Quoted 11 lines · Christoph Hellwig
Quoted 9 lines · Jack Steiner
Quoted 4 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ for (i = 0; i < numnodes; i++) + len += sprintf(buf + len, "%s%d", i ? " " : "", node_distance(nid, i));Can this overflow the space allocated for buf?
Good point. I think we are ok for now. AFAIK, the largest cpu count currently supported is 512. That gives a max string of 2k (max of 3 digits + space per cpu).
Quoted 5 lines
Quoted 2 lines
I always wondered why sysfs doesn't use the
seq_fileinterface that makes life easier in the rest of them kernel.Most fields only output a single number, and
seq_filewould be extreme overkill for that.
Personally I think even a:
static voidshow_foo(struct device *dev, struct seq_file *s)
{
seq_printf(s, "blafcsvsdfg\n");
}static ssize_tshow_foo(struct device *dev, char *buf)
{
return snprintf(buf, 20, "blafcsvsdfg\n");
}would be a definitive improvement.
Quoted 9 lines · Andi Kleen
Quoted 4 lines · Takayoshi Kochi
(3) all distances in one line like /proc/<PID>/stat
% cat
/sys/devices/system/node/node0/distance
10 66 46 66I would prefer that.
-Andi
That would be my vote as well. One line, space delimited. Easy to parse... Plus you could easily reproduce the entire SLIT matrix by:
cd /sys/devices/system/node/
for i in ls node*; do cat $i/distance; done
-Matt
Quoted 17 lines · Erich Focht
Quoted 7 lines · Jack Steiner
I think it would also be useful to have a similar cpu-to-cpu distance metric:
% cat/sys/devices/system/cpu/cpu0/distance
10 20 40 60This gives the same information but is cpu-centric rather than node centric.
I don't see the use of that once you have some way to find the logical CPU to node number mapping. The "node distances" are meant to be proportional to the memory access latency ratios (20 means 2 times larger than local (intra-node) access, which is by definition 10). If the
cpu_to_cpudistance is necessary because there is a hierarchy in the memory blocks inside one node, then maybe the definition of a node should be changed...
Quoted 8 lines
We currently have (at least in -mm kernels):
% ls/sys/devices/system/node/node0/cpu*
for finding out which CPUs belong to which nodes. Together with
/sys/devices/system/node/node0/distances
this should be enough for user space NUMA tools.Regards,
Erich
I have to agree with Erich here. Node distances make sense, but adding 'cpu distances' which are just re-exporting the node distances in each cpu's directory in sysfs doesn't make much sense to me. Especially because it is so trivial to get a list of which CPUs are on which node. If you're looking for groups of CPUs which are close, simply look for groups of nodes that are close, then use the CPUs on those nodes. If we came up with some sort of different notion of 'distance' for CPUs and exported that, I'd be OK with it, because it'd be new information. I don't think we should export the *exact same* node distance information through the CPUs, though.
-Matt
Quoted 10 lines · Andi Kleen
Quoted 9 lines · Erich Focht
Quoted 7 lines · Jack Steiner
I think it would also be useful to have a similar cpu-to-cpu distance metric:
????????% cat/sys/devices/system/cpu/cpu0/distance????????10 20 40 60This gives the same information but is cpu-centric rather than node centric.
Quoted 8 lines
Quoted 2 lines
I don't see the use of that once you have some way to find the logical CPU to node number mapping. The "node distances" are meant to be
I think he wants it just to have a more convenient interface, which is not necessarily a bad thing. But then one could put the convenience into libnuma anyways.
-Andi
Using libnuma sounds fine to me. On a 512 CPU system, with 4 CPUs/node, we'd have 128 nodes. Re-exporting ALL the same data, those huge strings of node-to-node distances, 512 *additional* times in the per-CPU sysfs directories seems like a waste.
-Matt
Quoted 3 lines · Matthew Dobson
...
I don't think we should export the *exact same* node distance information through the CPUs, though.
We should still export cpu distances though because the distance between cpus on the same node may not be equal. e.g. consider a node with multiple cpu sockets, each socket with a hyperthreaded (or dual core) cpu.
Once again however, it depends on the definition of distance. For nodes, we've established it's the ACPI SLIT (relative distance to memory). For cpus, should it be distance to memory? Distance to cache? Registers? Or what?
-- Mark
Quoted 5 lines · Mark Goodwin
Quoted 3 lines · Matthew Dobson
...
I don't think we should export the *exact same* node distance information through the CPUs, though.
Quoted 8 lines
We should still export cpu distances though because the distance between cpus on the same node may not be equal. e.g. consider a node with multiple cpu sockets, each socket with a hyperthreaded (or dual core) cpu.
Once again however, it depends on the definition of distance. For nodes, we've established it's the ACPI SLIT (relative distance to memory). For cpus, should it be distance to memory? Distance to cache? Registers? Or what?
Yeah, that's a tough call. We should definitely get the node stuff in there now though, IMO. We can always add the CPU distances later if we figure out what they should mean.
Jesse
Quoted 8 lines · Mark Goodwin
Quoted 3 lines · Matthew Dobson
...
I don't think we should export the *exact same* node distance information through the CPUs, though.We should still export cpu distances though because the distance between cpus on the same node may not be equal. e.g. consider a node with multiple cpu sockets, each socket with a hyperthreaded (or dual core) cpu.
Well, I'm not sure that just because a CPU has two hyperthread units in the same core that those HT units have a different distance or latency to memory...? The fact that it is a HT unit and not a physical core has implications to the scheduler, but I thought that the 2 siblings looked identical to userspace, no? If 2 CPUs in the same node are on the same bus, then in all likelihood they have the same "distance".
Quoted 6 lines
Once again however, it depends on the definition of distance. For nodes, we've established it's the ACPI SLIT (relative distance to memory). For cpus, should it be distance to memory? Distance to cache? Registers? Or what?
-- Mark
That's the real issue. We need to agree upon a meaningful definition of CPU-to-CPU "distance". As Jesse mentioned in a follow-up, we can all agree on what Node-to-Node "distance" means, but there doesn't appear to be much consensus on what CPU "distance" means.
-Matt
Quoted 10 lines · Matthew Dobson
Quoted 5 lines · Mark Goodwin
Once again however, it depends on the definition of distance. For nodes, we've established it's the ACPI SLIT (relative distance to memory). For cpus, should it be distance to memory? Distance to cache? Registers? Or what?
That's the real issue. We need to agree upon a meaningful definition of CPU-to-CPU "distance". As Jesse mentioned in a follow-up, we can all agree on what Node-to-Node "distance" means, but there doesn't appear to be much consensus on what CPU "distance" means.
How about we define cpu-distance to be "relative distance to the lowest level cache on another CPU". On a system that has nodes with multiple sockets (each supporting multiple cores or HT "CPUs" sharing some level of cache), when the scheduler needs to migrate a task it would first choose a CPU sharing the same cache, then a CPU on the same node, then an off-node CPU (i.e. falling back to node distance).
Of course, I have no idea if that's anything like an optimal or desirable task migration policy. Probably depends on cache-trashiness of the task being migrated.
-- Mark
Quoted 8 lines · Mark Goodwin
Quoted 6 lines · Matthew Dobson
Quoted 5 lines · Mark Goodwin
Once again however, it depends on the definition of distance. For nodes, we've established it's the ACPI SLIT (relative distance to memory). For cpus, should it be distance to memory? Distance to cache? Registers? Or what?
Quoted 7 lines
Quoted 4 lines
That's the real issue. We need to agree upon a meaningful definition of CPU-to-CPU "distance". As Jesse mentioned in a follow-up, we can all agree on what Node-to-Node "distance" means, but there doesn't appear to be much consensus on what CPU "distance" means.
How about we define cpu-distance to be "relative distance to the lowest level cache on another CPU".
Several definitions are possible, this is really a source of confusion. Any of these can be reconstructed if one has access to the
constituents: node-to-node latency (SLIT), cache-to-cache latencies. The later ones aren't available and would anyhow be better placed in something like /proc/cpuinfo or similar. They are CPU or package specific and have nothing to do with NUMA.
Quoted 5 lines
On a system that has nodes with multiple sockets (each supporting multiple cores or HT "CPUs" sharing some level of cache), when the scheduler needs to migrate a task it would first choose a CPU sharing the same cache, then a CPU on the same node, then an off-node CPU (i.e. falling back to node distance).
This should be done by correctly setting up the sched domains. It's not a question of exporting useless or redundant information to user space.
The need for some (any) cpu-to-cpu metrics initially brought up by Jack seemed mainly motivated by existing user space tools for constructing cpusets (maybe in PBS). I think it is a tolerable effort to introduce in user space an inlined function or macro doing something like
cpu_metric(i,j) := node_metric(cpu_node(i),cpu_node(j))
It keeps the kernel free of misleading information which might just slightly make cpusets construction more comfortable. In user space you have the full freedom to enhance your metrics when getting more details about the next generation cpus.
Regards,
Erich
Quoted 18 lines · Erich Focht
Quoted 5 lines · Mark Goodwin
On a system that has nodes with multiple sockets (each supporting multiple cores or HT "CPUs" sharing some level of cache), when the scheduler needs to migrate a task it would first choose a CPU sharing the same cache, then a CPU on the same node, then an off-node CPU (i.e. falling back to node distance).
This should be done by correctly setting up the sched domains. It's not a question of exporting useless or redundant information to user space.
The need for some (any) cpu-to-cpu metrics initially brought up by Jack seemed mainly motivated by existing user space tools for constructing cpusets (maybe in PBS). I think it is a tolerable effort to introduce in user space an inlined function or macro doing something like
cpu_metric(i,j):= node_metric(cpu_node(i),cpu_node(j))
Quoted 4 lines
It keeps the kernel free of misleading information which might just slightly make cpusets construction more comfortable. In user space you have the full freedom to enhance your metrics when getting more details about the next generation cpus.
Good point, Erich. I don't think there is any desperate need for CPU-to-CPU distances to be exported to userspace right now. If that is incorrect and someone really needs a particular distance metric to be exported by the kernel, we can look into that and export the required info. For now I think the Node-to-Node distance information is enough. -Matt