mm: show node to memory section relationship with symlinks in sysfs

Show node to memory section relationship with symlinks in sysfs

 arch/x86/mm/init_32.c                          |    2 
 arch/x86/mm/init_64.c                          |    2 

Show node to memory section relationship with symlinks in sysfs

Add /sys/devices/system/node/nodeX/memoryY symlinks for all the memory sections located on nodeX. For example:
/sys/devices/system/node/node1/memory135 -> ../../memory/memory135 indicates that memory section 135 resides on node1.

Also revises documentation to cover this change as well as updating Documentation/ABI/testing/sysfs-devices-memory to include descriptions of memory hotremove files 'phys_device', 'phys_index', and 'state' that were previously not described there.

In addition to it always being a good policy to provide users with the maximum possible amount of physical location information for resources that can be hot-added and/or hot-removed, the following are some (but likely not all) of the user benefits provided by this change.
Immediate:
- Provides information needed to determine the specific node on which a defective DIMM is located. This will reduce system downtime when the node or defective DIMM is swapped out.
- Prevents unintended onlining of a memory section that was previously offlined due to a defective DIMM. This could happen during node hot-add when the user or node hot-add assist script onlines _all_ offlined sections due to user or script inability to identify the specific memory sections located on the hot-added node. The consequences of reintroducing the defective memory could be ugly.
- Provides information needed to vary the amount and distribution of memory on specific nodes for testing or debugging purposes.
Future:
- Will provide information needed to identify the memory sections that need to be offlined prior to physical removal of a specific node.

Symlink creation during boot was tested on 2-node x86_64, 2-node ppc64, and 2-node ia64 systems. Symlink creation during physical memory hot-add tested on a 2-node x86_64 system.

Supersedes the "mm: show memory section to node relationship in sysfs" patch posted on 05 Sept 2008 which created node ID containing 'node' files in /sys/devices/system/memory/memoryX instead of symlinks. Changed from files to symlinks due to feedback that symlinks were more consistent with the sysfs way.

...

 Documentation/ABI/testing/sysfs-devices-memory |   51 +++++++
 Documentation/memory-hotplug.txt               |   16 +-
 arch/ia64/mm/init.c                            |    2 
 arch/powerpc/mm/mem.c                          |    2 
 arch/s390/mm/init.c                            |    2 
 arch/sh/mm/init.c                              |    3 
 arch/x86/mm/init_32.c                          |    2 
 arch/x86/mm/init_64.c                          |    2 
 drivers/base/memory.c                          |   19 +-
 drivers/base/node.c                            |  100 +++++++++++++++
 include/linux/memory.h                         |    6 
 include/linux/memory_hotplug.h                 |    2 
 include/linux/node.h                           |   13 +
 mm/memory_hotplug.c                            |    9 -
 14 files changed, 205 insertions(+), 24 deletions(-)

Dumb question: why do this with a symlink forest instead of, say, cat /proc/sys/vm/mem-sections?

Quoted 2 lines · Andrew Morton

Dumb question: why do this with a symlink forest instead of, say, cat /proc/sys/vm/mem-sections?

The basic problem is that we on/offline memory based on sections and not nodes. But, physically, people care about nodes.

So, the question we're answering is "to which sections does this node's memory belong?". We could just put all this data in one big file and have:

$ cat /proc/sys/vm/mem-sections?
node: section numbers
0: 1 2 3 4 5
1: 5 6 7 8
2: 99 100 101 102

But, we have the nodes in sysfs and we also have the sections in sysfs and I don't want Greg to be mean to me. He's scary. We could simply dump the section numbers in sysfs, but the first thing userspace is going to do is:

for section in /sys/devices/system/node/node1/memory*; do

	nr=$(cat $section)
	cat foo > /sys/devices/system/memory/memory$nr/bar

done

Making the symlinks makes it harder for us to screw this process up, both in the kernel and in userspace. Plus, symlinks are easy to code up in sysfs.

Show node to memory section relationship with symlinks in sysfs

Add /sys/devices/system/node/nodeX/memoryY symlinks for all the memory sections located on nodeX. For example:
/sys/devices/system/node/node1/memory135 -> ../../memory/memory135 indicates that memory section 135 resides on node1.

Also revises documentation to cover this change as well as updating Documentation/ABI/testing/sysfs-devices-memory to include descriptions of memory hotremove files 'phys_device', 'phys_index', and 'state' that were previously not described there.

In addition to it always being a good policy to provide users with the maximum possible amount of physical location information for resources that can be hot-added and/or hot-removed, the following are some (but likely not all) of the user benefits provided by this change.
Immediate:
- Provides information needed to determine the specific node on which a defective DIMM is located. This will reduce system downtime when the node or defective DIMM is swapped out.
- Prevents unintended onlining of a memory section that was previously offlined due to a defective DIMM. This could happen during node hot-add when the user or node hot-add assist script onlines _all_ offlined sections due to user or script inability to identify the specific memory sections located on the hot-added node. The consequences of reintroducing the defective memory could be ugly.
- Provides information needed to vary the amount and distribution of memory on specific nodes for testing or debugging purposes.
Future:
- Will provide information needed to identify the memory sections that need to be offlined prior to physical removal of a specific node.

Symlink creation during boot was tested on 2-node x86_64, 2-node ppc64, and 2-node ia64 systems. Symlink creation during physical memory hot-add tested on a 2-node x86_64 system.

Supersedes the "mm: show memory section to node relationship in sysfs" patch posted on 05 Sept 2008 which created node ID containing 'node' files in /sys/devices/system/memory/memoryX instead of symlinks. Changed from files to symlinks due to feedback that symlinks were more consistent with the sysfs way.

Supersedes the "mm: show node to memory section relationship with symlinks in sysfs" patch posted on 29 Sept 2008 to address a Yasunori Goto reported problem where an incorrect symlink was created due to a range of uninitialized pages at the beginning of a section. This problem which produced a symlink in /sys/devices/system/node/node0 that incorrectly referenced a mem section located on node1 is corrected in this version. This version also covers the case were a mem section could span multiple nodes.

Supersedes the "mm: show node to memory section relationship with symlinks in sysfs" patch posted on 09 Oct 2008 to add the Andrew Morton requested usefulness information and update to apply cleanly to 2.6.28-rc3 and 2.6-git. Code is unchanged.

Signed-off-by Gary Hade <[email protected]>

Signed-off-by Badari Pulavarty <[email protected]>

Quoted 54 lines · Gary Hade

Show node to memory section relationship with symlinks in sysfs

Add /sys/devices/system/node/nodeX/memoryY symlinks for all the memory sections located on nodeX. For example:
/sys/devices/system/node/node1/memory135 -> ../../memory/memory135 indicates that memory section 135 resides on node1.

Also revises documentation to cover this change as well as updating Documentation/ABI/testing/sysfs-devices-memory to include descriptions of memory hotremove files 'phys_device', 'phys_index', and 'state' that were previously not described there.

In addition to it always being a good policy to provide users with the maximum possible amount of physical location information for resources that can be hot-added and/or hot-removed, the following are some (but likely not all) of the user benefits provided by this change.
Immediate:
- Provides information needed to determine the specific node on which a defective DIMM is located. This will reduce system downtime when the node or defective DIMM is swapped out.
- Prevents unintended onlining of a memory section that was previously offlined due to a defective DIMM. This could happen during node hot-add when the user or node hot-add assist script onlines _all_ offlined sections due to user or script inability to identify the specific memory sections located on the hot-added node. The consequences of reintroducing the defective memory could be ugly.
- Provides information needed to vary the amount and distribution of memory on specific nodes for testing or debugging purposes.
Future:
- Will provide information needed to identify the memory sections that need to be offlined prior to physical removal of a specific node.

Symlink creation during boot was tested on 2-node x86_64, 2-node ppc64, and 2-node ia64 systems. Symlink creation during physical memory hot-add tested on a 2-node x86_64 system.

Supersedes the "mm: show memory section to node relationship in sysfs" patch posted on 05 Sept 2008 which created node ID containing 'node' files in /sys/devices/system/memory/memoryX instead of symlinks. Changed from files to symlinks due to feedback that symlinks were more consistent with the sysfs way.

Supersedes the "mm: show node to memory section relationship with symlinks in sysfs" patch posted on 29 Sept 2008 to address a Yasunori Goto reported problem where an incorrect symlink was created due to a range of uninitialized pages at the beginning of a section. This problem which produced a symlink in /sys/devices/system/node/node0 that incorrectly referenced a mem section located on node1 is corrected in this version. This version also covers the case were a mem section could span multiple nodes.

Quoted 8 lines

Supersedes the "mm: show node to memory section relationship with symlinks in sysfs" patch posted on 09 Oct 2008 to add the Andrew Morton requested usefulness information and update to apply cleanly to 2.6.28-rc3 and 2.6-git. Code is unchanged.

Signed-off-by Gary Hade <[email protected]>

Signed-off-by Badari Pulavarty <[email protected]>

Hi Gary,

While testing latest mmtom (which has this patch) ran into an issue with sysfs files. What I noticed was, with this patch "memoryXX" directories in /sys/devices/system/memory/ are not getting cleaned up. Backing out the patch seems to fix the problem.

When I tried to remove 64 blocks of memory, empty directories are stayed around. (look at memory151 - memory215). This is causing OOPS while trying to add memory block again. I think this could be because of the symlink added from node directory. Can you look ?

Quoted 71 lines · Badari Pulavarty

Quoted 54 lines · Gary Hade

Show node to memory section relationship with symlinks in sysfs

Add /sys/devices/system/node/nodeX/memoryY symlinks for all the memory sections located on nodeX. For example:
/sys/devices/system/node/node1/memory135 -> ../../memory/memory135 indicates that memory section 135 resides on node1.

Also revises documentation to cover this change as well as updating Documentation/ABI/testing/sysfs-devices-memory to include descriptions of memory hotremove files 'phys_device', 'phys_index', and 'state' that were previously not described there.

In addition to it always being a good policy to provide users with the maximum possible amount of physical location information for resources that can be hot-added and/or hot-removed, the following are some (but likely not all) of the user benefits provided by this change.
Immediate:
- Provides information needed to determine the specific node on which a defective DIMM is located. This will reduce system downtime when the node or defective DIMM is swapped out.
- Prevents unintended onlining of a memory section that was previously offlined due to a defective DIMM. This could happen during node hot-add when the user or node hot-add assist script onlines _all_ offlined sections due to user or script inability to identify the specific memory sections located on the hot-added node. The consequences of reintroducing the defective memory could be ugly.
- Provides information needed to vary the amount and distribution of memory on specific nodes for testing or debugging purposes.
Future:
- Will provide information needed to identify the memory sections that need to be offlined prior to physical removal of a specific node.

Symlink creation during boot was tested on 2-node x86_64, 2-node ppc64, and 2-node ia64 systems. Symlink creation during physical memory hot-add tested on a 2-node x86_64 system.

Supersedes the "mm: show memory section to node relationship in sysfs" patch posted on 05 Sept 2008 which created node ID containing 'node' files in /sys/devices/system/memory/memoryX instead of symlinks. Changed from files to symlinks due to feedback that symlinks were more consistent with the sysfs way.

Supersedes the "mm: show node to memory section relationship with symlinks in sysfs" patch posted on 29 Sept 2008 to address a Yasunori Goto reported problem where an incorrect symlink was created due to a range of uninitialized pages at the beginning of a section. This problem which produced a symlink in /sys/devices/system/node/node0 that incorrectly referenced a mem section located on node1 is corrected in this version. This version also covers the case were a mem section could span multiple nodes.

Quoted 8 lines

Supersedes the "mm: show node to memory section relationship with symlinks in sysfs" patch posted on 09 Oct 2008 to add the Andrew Morton requested usefulness information and update to apply cleanly to 2.6.28-rc3 and 2.6-git. Code is unchanged.

Signed-off-by Gary Hade <[email protected]>

Signed-off-by Badari Pulavarty <[email protected]>

Hi Gary,

While testing latest mmtom (which has this patch) ran into an issue with sysfs files. What I noticed was, with this patch "memoryXX" directories in /sys/devices/system/memory/ are not getting cleaned up. Backing out the patch seems to fix the problem.

Quoted 4 lines

When I tried to remove 64 blocks of memory, empty directories are stayed around. (look at memory151 - memory215). This is causing OOPS while trying to add memory block again. I think this could be because of the symlink added from node directory. Can you look ?

Badari, The call to unregister_mem_sect_under_nodes() in remove_memory_block() preceding the removal of the files in the memory section directory _should have_ removed all the symlinks referencing the memory section directory. Did you happen to check to see if the symlinks to memory151-memory215 were still present?

Quoted 71 lines · Badari Pulavarty

Quoted 54 lines · Gary Hade

Show node to memory section relationship with symlinks in sysfs

Add /sys/devices/system/node/nodeX/memoryY symlinks for all the memory sections located on nodeX. For example:
/sys/devices/system/node/node1/memory135 -> ../../memory/memory135 indicates that memory section 135 resides on node1.

Also revises documentation to cover this change as well as updating Documentation/ABI/testing/sysfs-devices-memory to include descriptions of memory hotremove files 'phys_device', 'phys_index', and 'state' that were previously not described there.

In addition to it always being a good policy to provide users with the maximum possible amount of physical location information for resources that can be hot-added and/or hot-removed, the following are some (but likely not all) of the user benefits provided by this change.
Immediate:
- Provides information needed to determine the specific node on which a defective DIMM is located. This will reduce system downtime when the node or defective DIMM is swapped out.
- Prevents unintended onlining of a memory section that was previously offlined due to a defective DIMM. This could happen during node hot-add when the user or node hot-add assist script onlines _all_ offlined sections due to user or script inability to identify the specific memory sections located on the hot-added node. The consequences of reintroducing the defective memory could be ugly.
- Provides information needed to vary the amount and distribution of memory on specific nodes for testing or debugging purposes.
Future:
- Will provide information needed to identify the memory sections that need to be offlined prior to physical removal of a specific node.

Symlink creation during boot was tested on 2-node x86_64, 2-node ppc64, and 2-node ia64 systems. Symlink creation during physical memory hot-add tested on a 2-node x86_64 system.

Supersedes the "mm: show memory section to node relationship in sysfs" patch posted on 05 Sept 2008 which created node ID containing 'node' files in /sys/devices/system/memory/memoryX instead of symlinks. Changed from files to symlinks due to feedback that symlinks were more consistent with the sysfs way.

Supersedes the "mm: show node to memory section relationship with symlinks in sysfs" patch posted on 29 Sept 2008 to address a Yasunori Goto reported problem where an incorrect symlink was created due to a range of uninitialized pages at the beginning of a section. This problem which produced a symlink in /sys/devices/system/node/node0 that incorrectly referenced a mem section located on node1 is corrected in this version. This version also covers the case were a mem section could span multiple nodes.

Quoted 8 lines

Supersedes the "mm: show node to memory section relationship with symlinks in sysfs" patch posted on 09 Oct 2008 to add the Andrew Morton requested usefulness information and update to apply cleanly to 2.6.28-rc3 and 2.6-git. Code is unchanged.

Signed-off-by Gary Hade <[email protected]>

Signed-off-by Badari Pulavarty <[email protected]>

Hi Gary,

While testing latest mmtom (which has this patch) ran into an issue with sysfs files. What I noticed was, with this patch "memoryXX" directories in /sys/devices/system/memory/ are not getting cleaned up. Backing out the patch seems to fix the problem.

Quoted 4 lines

When I tried to remove 64 blocks of memory, empty directories are stayed around. (look at memory151 - memory215). This is causing OOPS while trying to add memory block again. I think this could be because of the symlink added from node directory. Can you look ?

Badari, The call to unregister_mem_sect_under_nodes() in remove_memory_block() preceding the removal of the files in the memory section directory _should have_ removed all the symlinks referencing the memory section directory. Did you happen to check to see if the symlinks to memory151-memory215 were still present?

Gary

Quoted 84 lines · Gary Hade

Quoted 71 lines · Badari Pulavarty

Quoted 54 lines · Gary Hade

Show node to memory section relationship with symlinks in sysfs

Add /sys/devices/system/node/nodeX/memoryY symlinks for all the memory sections located on nodeX. For example:
/sys/devices/system/node/node1/memory135 -> ../../memory/memory135 indicates that memory section 135 resides on node1.

Also revises documentation to cover this change as well as updating Documentation/ABI/testing/sysfs-devices-memory to include descriptions of memory hotremove files 'phys_device', 'phys_index', and 'state' that were previously not described there.

In addition to it always being a good policy to provide users with the maximum possible amount of physical location information for resources that can be hot-added and/or hot-removed, the following are some (but likely not all) of the user benefits provided by this change.
Immediate:
- Provides information needed to determine the specific node on which a defective DIMM is located. This will reduce system downtime when the node or defective DIMM is swapped out.
- Prevents unintended onlining of a memory section that was previously offlined due to a defective DIMM. This could happen during node hot-add when the user or node hot-add assist script onlines _all_ offlined sections due to user or script inability to identify the specific memory sections located on the hot-added node. The consequences of reintroducing the defective memory could be ugly.
- Provides information needed to vary the amount and distribution of memory on specific nodes for testing or debugging purposes.
Future:
- Will provide information needed to identify the memory sections that need to be offlined prior to physical removal of a specific node.

Symlink creation during boot was tested on 2-node x86_64, 2-node ppc64, and 2-node ia64 systems. Symlink creation during physical memory hot-add tested on a 2-node x86_64 system.

Supersedes the "mm: show memory section to node relationship in sysfs" patch posted on 05 Sept 2008 which created node ID containing 'node' files in /sys/devices/system/memory/memoryX instead of symlinks. Changed from files to symlinks due to feedback that symlinks were more consistent with the sysfs way.

Supersedes the "mm: show node to memory section relationship with symlinks in sysfs" patch posted on 29 Sept 2008 to address a Yasunori Goto reported problem where an incorrect symlink was created due to a range of uninitialized pages at the beginning of a section. This problem which produced a symlink in /sys/devices/system/node/node0 that incorrectly referenced a mem section located on node1 is corrected in this version. This version also covers the case were a mem section could span multiple nodes.

Quoted 8 lines

Supersedes the "mm: show node to memory section relationship with symlinks in sysfs" patch posted on 09 Oct 2008 to add the Andrew Morton requested usefulness information and update to apply cleanly to 2.6.28-rc3 and 2.6-git. Code is unchanged.

Signed-off-by Gary Hade <[email protected]>

Signed-off-by Badari Pulavarty <[email protected]>

Hi Gary,

While testing latest mmtom (which has this patch) ran into an issue with sysfs files. What I noticed was, with this patch "memoryXX" directories in /sys/devices/system/memory/ are not getting cleaned up. Backing out the patch seems to fix the problem.

Quoted 4 lines

When I tried to remove 64 blocks of memory, empty directories are stayed around. (look at memory151 - memory215). This is causing OOPS while trying to add memory block again. I think this could be because of the symlink added from node directory. Can you look ?

Badari, The call to unregister_mem_sect_under_nodes() in remove_memory_block() preceding the removal of the files in the memory section directory _should have_ removed all the symlinks referencing the memory section directory. Did you happen to check to see if the symlinks to memory151-memory215 were still present?

Quoted 2 lines

Gary

Hi Gary,

As discussed earlier, patch is leaving an extra reference on the memoryX directory. Needs a kobject_put() to match the reference you get in find_memory_block().

Could you update the patch and resend it ?

Quoted 85 lines · Badari Pulavarty

Quoted 84 lines · Gary Hade

Quoted 71 lines · Badari Pulavarty

Quoted 54 lines · Gary Hade

Show node to memory section relationship with symlinks in sysfs

Add /sys/devices/system/node/nodeX/memoryY symlinks for all the memory sections located on nodeX. For example:
/sys/devices/system/node/node1/memory135 -> ../../memory/memory135 indicates that memory section 135 resides on node1.

Also revises documentation to cover this change as well as updating Documentation/ABI/testing/sysfs-devices-memory to include descriptions of memory hotremove files 'phys_device', 'phys_index', and 'state' that were previously not described there.

In addition to it always being a good policy to provide users with the maximum possible amount of physical location information for resources that can be hot-added and/or hot-removed, the following are some (but likely not all) of the user benefits provided by this change.
Immediate:
- Provides information needed to determine the specific node on which a defective DIMM is located. This will reduce system downtime when the node or defective DIMM is swapped out.
- Prevents unintended onlining of a memory section that was previously offlined due to a defective DIMM. This could happen during node hot-add when the user or node hot-add assist script onlines _all_ offlined sections due to user or script inability to identify the specific memory sections located on the hot-added node. The consequences of reintroducing the defective memory could be ugly.
- Provides information needed to vary the amount and distribution of memory on specific nodes for testing or debugging purposes.
Future:
- Will provide information needed to identify the memory sections that need to be offlined prior to physical removal of a specific node.

Symlink creation during boot was tested on 2-node x86_64, 2-node ppc64, and 2-node ia64 systems. Symlink creation during physical memory hot-add tested on a 2-node x86_64 system.

Supersedes the "mm: show memory section to node relationship in sysfs" patch posted on 05 Sept 2008 which created node ID containing 'node' files in /sys/devices/system/memory/memoryX instead of symlinks. Changed from files to symlinks due to feedback that symlinks were more consistent with the sysfs way.

Supersedes the "mm: show node to memory section relationship with symlinks in sysfs" patch posted on 29 Sept 2008 to address a Yasunori Goto reported problem where an incorrect symlink was created due to a range of uninitialized pages at the beginning of a section. This problem which produced a symlink in /sys/devices/system/node/node0 that incorrectly referenced a mem section located on node1 is corrected in this version. This version also covers the case were a mem section could span multiple nodes.

Quoted 8 lines

Supersedes the "mm: show node to memory section relationship with symlinks in sysfs" patch posted on 09 Oct 2008 to add the Andrew Morton requested usefulness information and update to apply cleanly to 2.6.28-rc3 and 2.6-git. Code is unchanged.

Signed-off-by Gary Hade <[email protected]>

Signed-off-by Badari Pulavarty <[email protected]>

Hi Gary,

While testing latest mmtom (which has this patch) ran into an issue with sysfs files. What I noticed was, with this patch "memoryXX" directories in /sys/devices/system/memory/ are not getting cleaned up. Backing out the patch seems to fix the problem.

Quoted 4 lines

When I tried to remove 64 blocks of memory, empty directories are stayed around. (look at memory151 - memory215). This is causing OOPS while trying to add memory block again. I think this could be because of the symlink added from node directory. Can you look ?

Badari, The call to unregister_mem_sect_under_nodes() in remove_memory_block() preceding the removal of the files in the memory section directory _should have_ removed all the symlinks referencing the memory section directory. Did you happen to check to see if the symlinks to memory151-memory215 were still present?

Quoted 8 lines

Quoted 2 lines

Gary

Hi Gary,

As discussed earlier, patch is leaving an extra reference on the memoryX directory. Needs a kobject_put() to match the reference you get in find_memory_block().

Badari, Thanks again for finding that!

Quoted 2 lines

Could you update the patch and resend it ?

Will do.