Skip to contentSkip to conversation

Exporting physical topology information

PATCH Martin Hicks 4 participants 10 messages Mar 17, 2004 – May 8, 2004 Driver core

lore
Share & tools
lore
#0 lore raw

Hi,

I'm trying to figure out what the best way is to export a minimal amount of physical topology information to userland. Would it be acceptable to export this kind of information with sysfs?

I'm not proposing that we build an entire physical topology tree in sysfs, but just providing an attribute file. The two most obvious examples of where this would be useful is for nodes and pci busses. The Altix platform is a modular system with CPU bricks and IO bricks. We currently have no method for locating where "node0" is, nor do we have a method for locating pci bus 0000:20, for example.

If we could physically locate a PCI bus, then it would be much easier to (for example) locate our defective SCSI disk that is target4 on the SCSI controller that is on pci bus 0000:20.

The attached patch, care of Jesse Barnes, exports a physid attribute for each node, which indicates the physical location of the node. Altix specific.

thanks
mh

Signature · 2 lines
Martin Hicks                Wild Open Source Inc.
[email protected]     613-266-2296

===== arch/ia64/mm/numa.c 1.7 vs edited =====

arch/ia64/mm/numa.c +9 -0
@@ -19,6 +19,8 @@
 #include <linux/bootmem.h>
 #include <asm/mmzone.h>
 #include <asm/numa.h>
+#include <asm/sn/nodepda.h>
+#include <asm/sn/module.h>
 
 static struct node *sysfs_nodes;
 static struct cpu *sysfs_cpus;
@@ -48,6 +50,13 @@
 			break;
 
 	return (i < num_node_memblks) ? node_memblk[i].nid : (num_node_memblks ? -1 : 0);
+}
+
+void node_to_physid(int node, char *buf)
+{
+     struct nodepda_s *nodeinfo = NODEPDA(node);
+
+     format_module_id(buf, nodeinfo->module->id, MODULE_FORMAT_BRIEF);
 }
 
 static int __init topology_init(void)

===== drivers/base/node.c 1.18 vs edited =====

drivers/base/node.c +13 -0
@@ -56,6 +56,17 @@
 static SYSDEV_ATTR(meminfo,S_IRUGO,node_read_meminfo,NULL);
 
 
+static ssize_t node_read_physid(struct sys_device * dev, char * buf)
+{
+	struct node *node_dev = to_node(dev);
+	int len;
+
+	len = snprintf(buf, NODE_MAX_PHYSID + 1, "%s\n", node_dev->physid);
+	return len;
+}
+
+static SYSDEV_ATTR(physid,S_IRUGO,node_read_physid,NULL);
+
 /*
  * register_node - Setup a driverfs device for a node.
  * @num - Node number to use when creating the device.
@@ -67,6 +78,7 @@
 	int error;
 
 	node->cpumap = node_to_cpumask(num);
+	node_to_physid(num, node->physid);
 	node->sysdev.id = num;
 	node->sysdev.cls = &node_class;
 	error = sysdev_register(&node->sysdev);
@@ -74,6 +86,7 @@
 	if (!error){
 		sysdev_create_file(&node->sysdev, &attr_cpumap);
 		sysdev_create_file(&node->sysdev, &attr_meminfo);
+		sysdev_create_file(&node->sysdev, &attr_physid);
 	}
 	return error;
 }

===== include/asm-ia64/topology.h 1.10 vs edited =====

include/asm-ia64/topology.h +2 -0
@@ -45,6 +45,8 @@
 
 void build_cpu_to_node_map(void);
 
+extern void node_to_physid(int node, char *buf);
+
 #endif /* CONFIG_NUMA */
 
 #include <asm-generic/topology.h>

===== include/linux/node.h 1.5 vs edited =====

include/linux/node.h +3 -0
@@ -22,8 +22,11 @@
 #include <linux/sysdev.h>
 #include <linux/cpumask.h>
 
+#define NODE_MAX_PHYSID 80
+
 struct node {
 	cpumask_t cpumap;	/* Bitmap of CPUs on the Node */
+	char physid[NODE_MAX_PHYSID]; /* Physical ID of node */
 	struct sys_device	sysdev;
 };
 
#1 lore raw
Quoted 6 lines · Martin Hicks

I'm not proposing that we build an entire physical topology tree in sysfs, but just providing an attribute file. The two most obvious examples of where this would be useful is for nodes and pci busses. The Altix platform is a modular system with CPU bricks and IO bricks. We currently have no method for locating where "node0" is, nor do we have a method for locating pci bus 0000:20, for example.

I'm curious how other arches deal with this too. Like on ppc64 when you want to remove a CPU or set of CPUs, you have to bring it (or all of the cores on a given module) down via software, then go into the lab and find the module to pull it out. Is there a mapping somewhere that the user is expected to use? A hypervisor call of some sort to make some lights blink?

Quoted 3 lines

If we could physically locate a PCI bus, then it would be much easier to (for example) locate our defective SCSI disk that is target4 on the SCSI controller that is on pci bus 0000:20.

This seems like one of the main uses--find components that went bad. Physically locating a CPU, DIMM, PCI board, or disk would all be easier if we provided some sort of physical identifier and logical->physical mapping information. On IRIX, we actually expose the whole physical hierarchy of the system in /hw. One of the problems with that approach is that everytime a new system configuration is released the kernel has to be updated to know about it, resulting in /hw paths that change over time, and from system to system...

Jesse

#2 lore raw
In reply to Martin Hicks
Quoted 14 lines · Martin Hicks

Hi,

I'm trying to figure out what the best way is to export a minimal amount of physical topology information to userland. Would it be acceptable to export this kind of information with sysfs?

I'm not proposing that we build an entire physical topology tree in sysfs, but just providing an attribute file. The two most obvious examples of where this would be useful is for nodes and pci busses. The Altix platform is a modular system with CPU bricks and IO bricks. We currently have no method for locating where "node0" is, nor do we have a method for locating pci bus 0000:20, for example.

Quoted 3 lines

If we could physically locate a PCI bus, then it would be much easier to (for example) locate our defective SCSI disk that is target4 on the SCSI controller that is on pci bus 0000:20.

Um, what's wrong with the current /sys/class/pci_bus/*/cpuaffinity files for determining this topology information? That is why it was added.

thanks,

greg k-h

#3 lore raw
Quoted 10 lines · Greg KH
Quoted 6 lines · Martin Hicks

Hi,

If we could physically locate a PCI bus, then it would be much easier to (for example) locate our defective SCSI disk that is target4 on the SCSI controller that is on pci bus 0000:20.

Um, what's wrong with the current /sys/class/pci_bus/*/cpuaffinity files for determining this topology information? That is why it was added.

This gives us more logical topology information. It still doesn't tell us where in the room the specific piece of equipment is.

mh

Signature · 2 lines
Martin Hicks                Wild Open Source Inc.
[email protected]     613-266-2296
#5 lore raw
In reply to Greg KH
Quoted 6 lines · Greg KH
Quoted 3 lines

If we could physically locate a PCI bus, then it would be much easier to (for example) locate our defective SCSI disk that is target4 on the SCSI controller that is on pci bus 0000:20.

Um, what's wrong with the current /sys/class/pci_bus/*/cpuaffinity files for determining this topology information? That is why it was added.

Nothing, except that it only provides logical information. In a large system, it's really useful to be able to physically locate a component somehow. That was the idea behind adding 'physid'. For example:

[jbarnes@spamtin pci0000:02]$ pwd
/sys/devices/pci0000:02
[jbarnes@spamtin pci0000:02]$ cat physid
rack: 5
module: 12
slot: 3

or for nodes:

[jbarnes@spamtin node2]$ cat physid
rack: 1
module: 3
slot: 1

Then you could walk into the lab and know exactly which device to kick. Obviously, these values would be platform specific, though on ia64 and some x86 platforms, we could probably use the ACPI namespace to access some of the info, and on ppc the OF namespace might have it.

Thanks,
Jesse

#4 lore raw
In reply to Martin Hicks
Quoted 10 lines · Martin Hicks
Quoted 8 lines · Greg KH
Quoted 6 lines · Martin Hicks

Hi,

If we could physically locate a PCI bus, then it would be much easier to (for example) locate our defective SCSI disk that is target4 on the SCSI controller that is on pci bus 0000:20.

Quoted 5 lines
Quoted 2 lines

Um, what's wrong with the current /sys/class/pci_bus/*/cpuaffinity files for determining this topology information? That is why it was added.

This gives us more logical topology information. It still doesn't tell us where in the room the specific piece of equipment is.

True, but isn't that what labels on your CPU nodes are for?

:)

greg k-h

#6 lore raw
In reply to Jesse Barnes
Quoted 12 lines · Jesse Barnes
Quoted 6 lines · Greg KH
Quoted 3 lines

If we could physically locate a PCI bus, then it would be much easier to (for example) locate our defective SCSI disk that is target4 on the SCSI controller that is on pci bus 0000:20.

Um, what's wrong with the current /sys/class/pci_bus/*/cpuaffinity files for determining this topology information? That is why it was added.

Nothing, except that it only provides logical information. In a large system, it's really useful to be able to physically locate a component somehow. That was the idea behind adding 'physid'. For example:

Quoted 6 lines

[jbarnes@spamtin pci0000:02]$ pwd
/sys/devices/pci0000:02
[jbarnes@spamtin pci0000:02]$ cat physid
rack: 5
module: 12
slot: 3

Hm, that looks to violate the "one value per file" mandate of sysfs, right? Right now PCI Hotplug slots have a LED on them that you can flash from userspace to help locate the physical slot that you want to change. I also know of large PCI drawers that have LEDs that flash to locate them.

Also, this is _very_ hardware/platform specific. If you want to try to implement this, I'd be interested in what the patch would look like.

thanks,

greg k-h

#7 lore raw
Quoted 2 lines · Greg KH

Hm, that looks to violate the "one value per file" mandate of sysfs, right? Right now PCI Hotplug slots have a LED on them that you can

Yeah... my original patch to implement node physids used the Altix module id, which looks like rrrtss, where rrr is a rack id, t is a brick type, and ss is the rack slot, e.g. 001c12, so it was one value. The example above was just brainstorming, I'm sure there are better ways to do it.

Quoted 3 lines

flash from userspace to help locate the physical slot that you want to change. I also know of large PCI drawers that have LEDs that flash to locate them.

Yeah, that makes things easy, but it would be nice to cover CPUs and memory banks too, so you can go remove the DIMM with a persistent single or double bit error, or a CPU with a bad cache or whatever. I imagine some hardware has blinking lights for that too.

Quoted 2 lines

Also, this is _very_ hardware/platform specific. If you want to try to implement this, I'd be interested in what the patch would look like.

Here's the (very platform specific) patch I did for Altix, just to see what it would look like, and to solicit comments. There's some other per-node stuff that would be nice to have available to userspace too, mostly for administrative purposes, like the chipset revision and type, firmware revision, and other hardware specific details. One way to export that sort of thing is with some sort of arbitrary data blob, but like you said, that violates the sysfs one file, one value principle.

Thanks,
Jesse

===== arch/ia64/mm/numa.c 1.6 vs edited =====

arch/ia64/mm/numa.c +9 -0
@@ -20,6 +20,8 @@
 #include <linux/bootmem.h>
 #include <asm/mmzone.h>
 #include <asm/numa.h>
+#include <asm/sn/nodepda.h>
+#include <asm/sn/module.h>
 
 static struct memblk *sysfs_memblks;
 static struct node *sysfs_nodes;
@@ -50,6 +52,13 @@
 			break;
 
 	return (i < num_memblks) ? node_memblk[i].nid : (num_memblks ? -1 : 0);
+}
+
+void node_to_physid(int node, char *buf)
+{
+	struct nodepda_s *nodeinfo = NODEPDA(node);
+
+	format_module_id(buf, nodeinfo->module->id, MODULE_FORMAT_BRIEF);
 }
 
 static int __init topology_init(void)

===== drivers/base/node.c 1.16 vs edited =====

drivers/base/node.c +13 -0
@@ -56,6 +56,17 @@
 static SYSDEV_ATTR(meminfo,S_IRUGO,node_read_meminfo,NULL);
 
 
+static ssize_t node_read_physid(struct sys_device * dev, char * buf)
+{
+	struct node *node_dev = to_node(dev);
+	int len;
+
+	len = snprintf(buf, NODE_MAX_PHYSID + 1, "%s\n", node_dev->physid);
+	return len;
+}
+
+static SYSDEV_ATTR(physid,S_IRUGO,node_read_physid,NULL);
+
 /*
  * register_node - Setup a driverfs device for a node.
  * @num - Node number to use when creating the device.
@@ -67,6 +78,7 @@
 	int error;
 
 	node->cpumap = node_to_cpumask(num);
+	node_to_physid(num, node->physid);
 	node->sysdev.id = num;
 	node->sysdev.cls = &node_class;
 	error = sys_device_register(&node->sysdev);
@@ -74,6 +86,7 @@
 	if (!error){
 		sysdev_create_file(&node->sysdev, &attr_cpumap);
 		sysdev_create_file(&node->sysdev, &attr_meminfo);
+		sysdev_create_file(&node->sysdev, &attr_physid);
 	}
 	return error;
 }

===== include/asm-ia64/topology.h 1.9 vs edited =====

include/asm-ia64/topology.h +2 -0
@@ -60,6 +60,8 @@
 
 void build_cpu_to_node_map(void);
 
+extern void node_to_physid(int node, char *buf);
+
 #endif /* CONFIG_NUMA */
 
 #include <asm-generic/topology.h>

===== include/asm-ia64/sn/module.h 1.10 vs edited =====

include/asm-ia64/sn/module.h +1 -0
@@ -14,6 +14,7 @@
 
 
 #include <linux/config.h>
+#include <asm/sn/sgi.h>
 #include <asm/sn/klconfig.h>
 #include <asm/sn/ksys/elsc.h>
 

===== include/linux/node.h 1.5 vs edited =====

include/linux/node.h +3 -0
@@ -22,8 +22,11 @@
 #include <linux/sysdev.h>
 #include <linux/cpumask.h>
 
+#define NODE_MAX_PHYSID 80
+
 struct node {
 	cpumask_t cpumap;	/* Bitmap of CPUs on the Node */
+	char physid[NODE_MAX_PHYSID]; /* Physical ID of node */
 	struct sys_device	sysdev;
 };
 
#8 lore raw
Quoted 8 lines · Jesse Barnes

Here's the (very platform specific) patch I did for Altix, just to see what it would look like, and to solicit comments. There's some other per-node stuff that would be nice to have available to userspace too, mostly for administrative purposes, like the chipset revision and type, firmware revision, and other hardware specific details. One way to export that sort of thing is with some sort of arbitrary data blob, but like you said, that violates the sysfs one file, one value principle.

Greg, any comments on this? (Sorry I was gone for much of last month and lost track of this thread.) Maybe a whole subdirectory containing physical tag, version files and such would make more sense?

Jesse

#9 lore raw
Quoted 10 lines · Jesse Barnes
Quoted 8 lines · Jesse Barnes

Here's the (very platform specific) patch I did for Altix, just to see what it would look like, and to solicit comments. There's some other per-node stuff that would be nice to have available to userspace too, mostly for administrative purposes, like the chipset revision and type, firmware revision, and other hardware specific details. One way to export that sort of thing is with some sort of arbitrary data blob, but like you said, that violates the sysfs one file, one value principle.

Quoted 3 lines

Greg, any comments on this? (Sorry I was gone for much of last month and lost track of this thread.) Maybe a whole subdirectory containing physical tag, version files and such would make more sense?

I think my main comments was you should work with the other numa people so that you all agree on one common format to export this data in sysfs. If you all agree that this is the way to do it, I'll accept the patch :)

thanks,

greg k-h