mm: show node to memory section relationship with symlinks in sysfs

PATCH In review Gary Hade 3 participants 7 messages Sep 29, 2008 – Oct 1, 2008

Subsystems (5)

Memory management Documentation MM core Driver core +1 more subsystems

lore

Share & tools

lore

Gary Hade

author

2008-09-29 20:05 UTC #0 lore raw

PATCH 1/1 mm: show node to memory section relationship with symlinks in sysfs

Show node to memory section relationship with symlinks in sysfs

Add /sys/devices/system/node/nodeX/memoryY symlinks for all the memory sections located on nodeX. For example:
/sys/devices/system/node/node1/memory135 -> ../../memory/memory135 indicates that memory section 135 resides on node1.

Successfully tested with 2.6.27-rc7 source on 2-node x86_64, 2-node ppc64, and 2-node ia64 systems.

Also revises documentation to cover this change as well as updating Documentation/ABI/testing/sysfs-devices-memory to include descriptions of memory hotremove files 'phys_device', 'phys_index', and 'state' that were previously not described there.

Supersedes the "mm: show memory section to node relationship in sysfs" patch posted on 05 Sept 2008 which created node ID containing 'node' files in /sys/devices/system/memory/memoryX instead of symlinks. Changed from files to symlinks due to feedback that symlinks were more consistent with the sysfs way.

Signed-off-by Gary Hade <[email protected]>

Signed-off-by Badari Pulavarty <[email protected]>

---
 Documentation/ABI/testing/sysfs-devices-memory |   51 ++++++++++++
 Documentation/memory-hotplug.txt               |   16 +++
 drivers/base/memory.c                          |   10 ++
 drivers/base/node.c                            |   61 +++++++++++++++
 include/linux/memory.h                         |    4 
 include/linux/node.h                           |   11 ++
 6 files changed, 146 insertions(+), 7 deletions(-)

Documentation/ABI/testing/sysfs-devices-memory +50 -1

Index: linux-2.6.27-rc5/Documentation/ABI/testing/sysfs-devices-memory
===================================================================
@@ -6,7 +6,6 @@
 		internal state of the kernel memory blocks. Files could be
 		added or removed dynamically to represent hot-add/remove
 		operations.
-
 Users:		hotplug memory add/remove tools
 		https://w3.opensource.ibm.com/projects/powerpc-utils/

@@ -19,6 +18,56 @@
 		This is useful for a user-level agent to determine
 		identify removable sections of the memory before attempting
 		potentially expensive hot-remove memory operation
+Users:		hotplug memory remove tools
+		https://w3.opensource.ibm.com/projects/powerpc-utils/
+
+What:		/sys/devices/system/memory/memoryX/phys_device
+Date:		September 2008
+Contact:	Badari Pulavarty <[email protected]>
+Description:
+		The file /sys/devices/system/memory/memoryX/phys_device
+		is read-only and is designed to show the name of physical
+		memory device.  Implementation is currently incomplete.
 
+What:		/sys/devices/system/memory/memoryX/phys_index
+Date:		September 2008
+Contact:	Badari Pulavarty <[email protected]>
+Description:
+		The file /sys/devices/system/memory/memoryX/phys_index
+		is read-only and contains the section ID in hexadecimal
+		which is equivalent to decimal X contained in the
+		memory section directory name.
+
+What:		/sys/devices/system/memory/memoryX/state
+Date:		September 2008
+Contact:	Badari Pulavarty <[email protected]>
+Description:
+		The file /sys/devices/system/memory/memoryX/state
+		is read-write.  When read, it's contents show the
+		online/offline state of the memory section.  When written,
+		root can toggle the the online/offline state of a removable
+		memory section (see removable file description above)
+		using the following commands.
+		# echo online > /sys/devices/system/memory/memoryX/state
+		# echo offline > /sys/devices/system/memory/memoryX/state
+
+		For example, if /sys/devices/system/memory/memory22/removable
+		contains a value of 1 and
+		/sys/devices/system/memory/memory22/state contains the
+		string "online" the following command can be executed by
+		by root to offline that section.
+		# echo offline > /sys/devices/system/memory/memory22/state
 Users:		hotplug memory remove tools
 		https://w3.opensource.ibm.com/projects/powerpc-utils/
+
+What:		/sys/devices/system/node/nodeX/memoryY
+Date:		September 2008
+Contact:	Gary Hade <[email protected]>
+Description:
+		When CONFIG_NUMA is enabled
+		/sys/devices/system/node/nodeX/memoryY is a symbolic link that
+		points to the corresponding /sys/devices/system/memory/memoryY
+		memory section directory.  For example, the following symbolic
+		link is created for memory section 9 on node0.
+		/sys/devices/system/node/node0/memory9 -> ../../memory/memory9
+

Documentation/memory-hotplug.txt +13 -3

Index: linux-2.6.27-rc5/Documentation/memory-hotplug.txt
===================================================================
@@ -124,7 +124,7 @@
     This option can be kernel module too.
 
 --------------------------------
-3 sysfs files for memory hotplug
+4 sysfs files for memory hotplug
 --------------------------------
 All sections have their device information under /sys/devices/system/memory as

@@ -138,11 +138,12 @@
 (0x100000000 / 1Gib = 4)
 This device covers address range [0x100000000 ... 0x140000000)
 
-Under each section, you can see 3 files.
+Under each section, you can see 4 files.
 
 /sys/devices/system/memory/memoryXXX/phys_index
 /sys/devices/system/memory/memoryXXX/phys_device
 /sys/devices/system/memory/memoryXXX/state
+/sys/devices/system/memory/memoryXXX/removable
 
 'phys_index' : read-only and contains section id, same as XXX.
 'state'      : read-write

@@ -150,10 +151,20 @@
                at write: user can specify "online", "offline" command
 'phys_device': read-only: designed to show the name of physical memory device.
                This is not well implemented now.
+'removable'  : read-only: contains an integer value indicating
+               whether the memory section is removable or not
+               removable.  A value of 1 indicates that the memory
+               section is removable and a value of 0 indicates that
+               it is not removable.
 
 NOTE:
   These directories/files appear after physical memory hotplug phase.
 
+If CONFIG_NUMA is specified the
+/sys/devices/system/memory/memoryXXX memory section
+directories can also be accessed via symbolic links located in
+the /sys/devices/system/node/node* directories.  For example:
+/sys/devices/system/node/node0/memory9 -> ../../memory/memory9
 
 --------------------------------
 4. Physical memory hot-add phase

@@ -365,7 +376,6 @@
   - allowing memory hot-add to ZONE_MOVABLE. maybe we need some switch like
     sysctl or new control file.
   - showing memory section and physical device relationship.
-  - showing memory section and node relationship (maybe good for NUMA)
   - showing memory section is under ZONE_MOVABLE or not
   - test and make it better memory offlining.
   - support HugeTLB page migration and offlining.

drivers/base/memory.c +9 -1

Index: linux-2.6.27-rc5/drivers/base/memory.c
===================================================================
@@ -368,6 +368,13 @@
 		ret = mem_create_simple_file(mem, phys_device);
 	if (!ret)
 		ret = mem_create_simple_file(mem, removable);
+	if (!ret) {
+		ret = register_mem_sect_under_node(mem);
+		if (ret == -EFAULT) {
+			/* expected during boot if node not registered yet */
+			ret = 0;
+		}
+	}
 
 	return ret;
 }

@@ -380,7 +387,7 @@
  *
  * This could be made generic for all sysdev classes.
  */
-static struct memory_block *find_memory_block(struct mem_section *section)
+struct memory_block *find_memory_block(struct mem_section *section)
 {
 	struct kobject *kobj;
 	struct sys_device *sysdev;

@@ -409,6 +416,7 @@
 	struct memory_block *mem;
 
 	mem = find_memory_block(section);
+	unregister_mem_sect_under_node(mem);
 	mem_remove_simple_file(mem, phys_index);
 	mem_remove_simple_file(mem, state);
 	mem_remove_simple_file(mem, phys_device);

drivers/base/node.c +61 -0

79 lines not on this page · raw message

include/linux/memory.h +2 -2

Index: linux-2.6.27-rc5/include/linux/memory.h
===================================================================
@@ -84,9 +84,9 @@
 extern int memory_dev_init(void);
 extern int remove_memory_block(unsigned long, struct mem_section *, int);
 extern int memory_notify(unsigned long val, void *v);
+extern struct memory_block *find_memory_block(struct mem_section *);
 #define CONFIG_MEM_BLOCK_SIZE	(PAGES_PER_SECTION<<PAGE_SHIFT)
-
-
+#define section_nr_to_nid(section_nr) pfn_to_nid(section_nr_to_pfn(section_nr))
 #endif /* CONFIG_MEMORY_HOTPLUG_SPARSE */
 
 #ifdef CONFIG_MEMORY_HOTPLUG

include/linux/node.h +11 -0

Index: linux-2.6.27-rc5/include/linux/node.h
===================================================================
@@ -26,6 +26,7 @@
 	struct sys_device	sysdev;
 };
 
+struct memory_block;
 extern struct node node_devices[];
 
 extern int register_node(struct node *, int, struct node *);

@@ -35,6 +36,8 @@
 extern void unregister_one_node(int nid);
 extern int register_cpu_under_node(unsigned int cpu, unsigned int nid);
 extern int unregister_cpu_under_node(unsigned int cpu, unsigned int nid);
+extern int register_mem_sect_under_node(struct memory_block *mem_blk);
+extern int unregister_mem_sect_under_node(struct memory_block *mem_blk);
 #else
 static inline int register_one_node(int nid)
 {

@@ -52,6 +55,14 @@
 {
 	return 0;
 }
+static inline int register_mem_sect_under_node(struct memory_block *mem_blk)
+{
+	return 0;
+}
+static inline int unregister_mem_sect_under_node(struct memory_block *mem_blk)
+{
+	return 0;
+}
 #endif
 
 #define to_node(sys_device) container_of(sys_device, struct node, sysdev)

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Yasunori Goto

2008-09-30 08:06 UTC #1 lore raw

drivers/base/node.c line 230

Quoted 2 lines

drivers/base/node.c +2 -0

+#ifdef CONFIG_MEMORY_HOTPLUG_SPARSE
+int register_mem_sect_under_node(struct memory_block *mem_blk)

I think this patch is convenience even when memory hotplug is disabled. CONFIG_SPARSEMEM seems better than CONFIG_MEMORY_HOTPLUG_SPARSE.

Quoted 7 lines

drivers/base/node.c +7 -0

+int register_mem_sect_under_node(struct memory_block *mem_blk)
+{
+	unsigned int nid;
+
+	if (!mem_blk)
+		return -EFAULT;
+	nid = section_nr_to_nid(mem_blk->phys_index);

(snip)

Quoted 2 lines

+#define section_nr_to_nid(section_nr) pfn_to_nid(section_nr_to_pfn(section_nr))
#endif /* CONFIG_MEMORY_HOTPLUG_SPARSE */

If the first page of the section is not valid, then this section_nr_to_nid() doesn't return correct value.

I tested this patch. In my box, the start_pfn of node 1 is 1200400, but section_nr_to_pfn(mem_blk->phys_index) returns 1200000. As a result, the section is linked to node 0.

Bye.

Signature · 8 lines

Yasunori Goto 


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Dave Hansen

2008-09-30 15:50 UTC #2 lore raw

Quoted 9 lines · Yasunori Goto

Quoted 2 lines
+#define section_nr_to_nid(section_nr) pfn_to_nid(section_nr_to_pfn(section_nr))
#endif /* CONFIG_MEMORY_HOTPLUG_SPARSE */
If the first page of the section is not valid, then this section_nr_to_nid() doesn't return correct value.
I tested this patch. In my box, the start_pfn of node 1 is 1200400, but section_nr_to_pfn(mem_blk->phys_index) returns 1200000. As a result, the section is linked to node 0.

Crap, I was worried about that.

Gary, this means that we have a N:1 relationship between NUMA nodes and sections. This normally isn't a problem because sections don't really care about nodes and they layer underneath them.

We'll probably need multiple symlinks in each section directory.

-- Dave

Gary Hade

author

2008-09-30 19:42 UTC #3 lore raw

Quoted 11 lines · Dave Hansen

Quoted 9 lines · Yasunori Goto
Quoted 2 lines
+#define section_nr_to_nid(section_nr) pfn_to_nid(section_nr_to_pfn(section_nr))
#endif /* CONFIG_MEMORY_HOTPLUG_SPARSE */
If the first page of the section is not valid, then this section_nr_to_nid() doesn't return correct value.
I tested this patch. In my box, the start_pfn of node 1 is 1200400, but section_nr_to_pfn(mem_blk->phys_index) returns 1200000. As a result, the section is linked to node 0.

Quoted 5 lines

Crap, I was worried about that.
Gary, this means that we have a N:1 relationship between NUMA nodes and sections. This normally isn't a problem because sections don't really care about nodes and they layer underneath them.

So, using Yasunori-san's example the memory section starting at pfn 1200000 actually resides on both node 0 and node 1.

Quoted 2 lines

We'll probably need multiple symlinks in each section directory.

or perhaps symlinks to the same section directory from >1 node directory.

Gary

Signature · 12 lines

Gary Hade
System x Enablement
IBM Linux Technology Center
503-578-4503  IBM T/L: 775-4503
[email protected]
http://www.ibm.com/linux/ltc

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Yasunori Goto

2008-10-01 02:48 UTC #4 lore raw

Quoted 12 lines · Dave Hansen

Quoted 11 lines · Yasunori Goto
Quoted 9 lines
Quoted 2 lines
+#define section_nr_to_nid(section_nr) pfn_to_nid(section_nr_to_pfn(section_nr))
#endif /* CONFIG_MEMORY_HOTPLUG_SPARSE */
If the first page of the section is not valid, then this section_nr_to_nid() doesn't return correct value.
I tested this patch. In my box, the start_pfn of node 1 is 1200400, but section_nr_to_pfn(mem_blk->phys_index) returns 1200000. As a result, the section is linked to node 0.

Quoted 8 lines

Quoted 5 lines
Crap, I was worried about that.
Gary, this means that we have a N:1 relationship between NUMA nodes and sections. This normally isn't a problem because sections don't really care about nodes and they layer underneath them.
So, using Yasunori-san's example the memory section starting at pfn 1200000 actually resides on both node 0 and node 1.

It may be possible that one section is divided to different node in theory. (I don't know really there is...)

But, the cause of my trouble differs from it. There is a memory hole which is occupied by firmware. So, the memory map of my box is here.

----
early_node_map[3] active PFN ranges

    0: 0x00000100 -> 0x00006d00
    0: 0x00408000 -> 0x00410000
    1: 0x01200400 -> 0x01210000

----

memmap_init() initializes from start_pfn (to end_pfn).
So, the memmaps for this first hole (0x1200000 - 0x12003ff) are not initialized, and node id is not set for them. This is true cause.

Bye.

Signature · 8 lines

Yasunori Goto 


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Gary Hade

author

2008-10-01 16:51 UTC #5 lore raw

Quoted 31 lines · Yasunori Goto

Quoted 12 lines · Dave Hansen
Quoted 11 lines · Yasunori Goto
Quoted 9 lines
Quoted 2 lines
+#define section_nr_to_nid(section_nr) pfn_to_nid(section_nr_to_pfn(section_nr))
#endif /* CONFIG_MEMORY_HOTPLUG_SPARSE */
If the first page of the section is not valid, then this section_nr_to_nid() doesn't return correct value.
I tested this patch. In my box, the start_pfn of node 1 is 1200400, but section_nr_to_pfn(mem_blk->phys_index) returns 1200000. As a result, the section is linked to node 0.
Quoted 8 lines
Quoted 5 lines
Crap, I was worried about that.
Gary, this means that we have a N:1 relationship between NUMA nodes and sections. This normally isn't a problem because sections don't really care about nodes and they layer underneath them.
So, using Yasunori-san's example the memory section starting at pfn 1200000 actually resides on both node 0 and node 1.
It may be possible that one section is divided to different node in theory. (I don't know really there is...)
But, the cause of my trouble differs from it. There is a memory hole which is occupied by firmware. So, the memory map of my box is here.
----
early_node_map[3] active PFN ranges

Quoted 8 lines

    0: 0x00000100 -> 0x00006d00
    0: 0x00408000 -> 0x00410000
    1: 0x01200400 -> 0x01210000
----
memmap_init() initializes from start_pfn (to end_pfn).
So, the memmaps for this first hole (0x1200000 - 0x12003ff) are not initialized, and node id is not set for them. This is true cause.

Thanks for the clarification. I think we need to cover both the theoretical single memory section spanning multiple nodes case and your memory hole/memory section intersection case.

Gary

Signature · 12 lines

Gary Hade
System x Enablement
IBM Linux Technology Center
503-578-4503  IBM T/L: 775-4503
[email protected]
http://www.ibm.com/linux/ltc

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Gary Hade

author

2008-09-30 23:29 UTC #6 lore raw

In reply to Yasunori Goto

Quoted 7 lines · Yasunori Goto

drivers/base/node.c line 230
Quoted 3 lines
drivers/base/node.c +2 -0
+#ifdef CONFIG_MEMORY_HOTPLUG_SPARSE
+int register_mem_sect_under_node(struct memory_block *mem_blk)
        :
I think this patch is convenience even when memory hotplug is disabled. CONFIG_SPARSEMEM seems better than CONFIG_MEMORY_HOTPLUG_SPARSE.

Yes, this would be nice but unfortunately the presence of the memory section directories that are referenced by the symlinks also depend on CONFIG_MEMORY_HOTPLUG_SPARSE being enabled. Removal of the memory hotplug dependency for the code in drivers/base/memory.c will require more than a simple CONFIG_MEMORY_HOTPLUG_SPARSE to CONFIG_SPARSEMEM dependency change. I am still looking at this.

Thanks for the review and testing.

Gary

Signature · 12 lines

Gary Hade
System x Enablement
IBM Linux Technology Center
503-578-4503  IBM T/L: 775-4503
[email protected]
http://www.ibm.com/linux/ltc

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>