memory hotplug disable boot option
Subsystems (4)
PATCH 1/1 memory hotplug disable boot option
Proposed patch to disable memory hotplug via a boot option, mem_hotplug=[on|off]. The patch only disables memory hotplug in that it prevents the creation of the memory sysfs directories for memory sections.
This patch is meant to help alleviate very long boot times on systems with large memory (1+ TB) and many memory sections (10's of thousands).
--- drivers/base/memory.c | 33 +++++++++++++++++++++++++++++++++ drivers/base/node.c | 3 +++ include/linux/memory.h | 2 ++ 3 files changed, 38 insertions(+)
drivers/base/memory.c +33 -0
@@ -29,6 +29,24 @@#define MEMORY_CLASS_NAME "memory" +static int mem_hp_enabled__read_mostly =1 ; + +static int __setup_memory_hotplug (char *option) +{ +if (!strcmp (option,"off" )) + mem_hp_enabled =0 ; +else if (!strcmp (option,"on" )) + mem_hp_enabled =1 ; + +return 1 ; +} +__setup ("mem_hotplug=" , __setup_memory_hotplug); + +int memory_hotplug_enabled (void ) +{ +return mem_hp_enabled; +} +static struct sysdev_class memory_sysdev_class = { .name =MEMORY_CLASS_NAME , };
@@ -320,6 +338,9 @@static int block_size_init (void ) { +if (!memory_hotplug_enabled ()) +return 0 ; +return sysfs_create_file (&memory_sysdev_class.kset.kobj, &attr_block_size_bytes.attr); }
@@ -442,6 +463,9 @@unsigned long start_pfn;int ret =0 ; +if (!memory_hotplug_enabled ()) +return 0 ; +if (!mem)return -ENOMEM ;
@@ -483,6 +507,9 @@struct memory_block *mem;char name[sizeof (MEMORY_CLASS_NAME ) +9 +1 ]; +if (!memory_hotplug_enabled ()) +return NULL ; +/* * This only works because we know that section == sysdev->id * slightly redundant with sysdev_register()
@@ -504,6 +531,9 @@ {struct memory_block *mem; +if (!memory_hotplug_enabled ()) +return 0 ; + mem =find_memory_block (section);unregister_mem_sect_under_nodes (mem);mem_remove_simple_file (mem, phys_index);
@@ -541,6 +571,9 @@int ret;int err; +if (!memory_hotplug_enabled ()) +return 0 ; + memory_sysdev_class.kset.uevent_ops = &memory_uevent_ops; ret =sysdev_class_register (&memory_sysdev_class);if (ret)
drivers/base/node.c +3 -0
@@ -411,6 +411,9 @@unsigned long pfn;int err =0 ; +if (!memory_hotplug_enabled ()) +return 0 ; +for (pfn = start_pfn; pfn < end_pfn; pfn +=PAGES_PER_SECTION ) {unsigned long section_nr =pfn_to_section_nr (pfn);struct mem_section *mem_sect;
include/linux/memory.h +2 -0
@@ -128,6 +128,8 @@#define hotplug_memory_notifier (fn, pri) do { } while (0)#endif +extern int memory_hotplug_enabled (void ); +/* * 'struct memory_accessor' is a generic interface to provide * in-kernel access to persistent memory such as i2c or SPI EEPROMs
Quoted 6 lines
Proposed patch to disable memory hotplug via a boot option, mem_hotplug=[on|off]. The patch only disables memory hotplug in that it prevents the creation of the memory sysfs directories for memory sections.
This patch is meant to help alleviate very long boot times on systems with large memory (1+ TB) and many memory sections (10's of thousands).
Why making simple /sys file is so slowly? Couldn't we fix such performance problem?
Quoted 14 lines
Signed-off-by Nathan Fontenot <[email protected]>--- drivers/base/memory.c | 33 +++++++++++++++++++++++++++++++++ drivers/base/node.c | 3 +++ include/linux/memory.h | 2 ++ 3 files changed, 38 insertions(+)Index:
linux-2.6/drivers/base/memory.c
===================================================================
drivers/base/memory.c line 32
Quoted 4 lines
drivers/base/memory.c +1
-0@@ -29,6 +29,24 @@#define MEMORY_CLASS_NAME "memory" +static int mem_hp_enabled__read_mostly =1 ;
please don't use 'hp' acronym. 'hotplug' is not so long word.
drivers/base/memory.c line 43
Quoted 11 lines
drivers/base/memory.c +11
-0+ +static int __setup_memory_hotplug (char *option) +{ +if (!strcmp (option,"off" )) + mem_hp_enabled =0 ; +else if (!strcmp (option,"on" )) + mem_hp_enabled =1 ; + +return 1 ; +} +__setup ("mem_hotplug=" , __setup_memory_hotplug);
no documentation on Documentation/kernel-parameters.txt?
Quoted 9 lines · KOSAKI Motohiro
Quoted 6 lines
Proposed patch to disable memory hotplug via a boot option, mem_hotplug=[on|off]. The patch only disables memory hotplug in that it prevents the creation of the memory sysfs directories for memory sections.
This patch is meant to help alleviate very long boot times on systems with large memory (1+ TB) and many memory sections (10's of thousands).
Why making simple /sys file is so slowly? Couldn't we fix such performance problem?
Yes I agree this really needs to be fixed properly, not hacked around with an option. Nathan can you please post some profile logs of the long boot times?
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
Quoted 11 lines · Andi Kleen
Quoted 9 lines · KOSAKI Motohiro
Quoted 6 lines
Proposed patch to disable memory hotplug via a boot option, mem_hotplug=[on|off]. The patch only disables memory hotplug in that it prevents the creation of the memory sysfs directories for memory sections.
This patch is meant to help alleviate very long boot times on systems with large memory (1+ TB) and many memory sections (10's of thousands).
Why making simple /sys file is so slowly? Couldn't we fix such performance problem?
The issue is the large number of sysfs memory directories that get created. On a system with 1 TB of memory I am seeing ~63,00 directories. The long creation time is due to the string compare check in sysfs code to ensure we are not creating a directory with a duplicate name.
Quoted 5 lines
Yes I agree this really needs to be fixed properly, not hacked around with an option. Nathan can you please post some profile logs of the long boot times?
At this point the only profiling data I have is from adding a printk before and after the creation of the memory sysfs files in drivers/base/memory.c and booting with printk.time=1.
With 250 GB of memory: 10 seconds
[ 0.539562] Memory Start [ 10.450409] Memory End
With 1 TB of memory: 9.1 minutes
[ 31.680168] Memory Start [ 584.186500] Memory End
I am hoping to get access to a machine with 2 TB of memory sometime soon and can post data for boot times on an unpatched kernel.
I posted a patch earlier that updated sysfs to use reb-black trees to store the sysfs dirent structs (though the patch has issues with namespaces). Using this patch on a 1 TB system the memory sysfs dir creation dropped to 2.2 minutes.
[ 1.295874] Memory Start [ 137.293510] Memory End
On a system with 2 TB of memory, the reb-black tree patch dropped the sysfs file creation time down to 33 minutes and total boot time to 1 hour 15 minutes. I haven't measured an unpatched kernel, but total boot of an unpatched kernel is just over 8 hours.
With 2 TB and patched kernel: 33 minutes
[ 3.241679] Memory Start [ 1986.973324] Memory End
I am open to other ideas on a solution for this. With this many memory sections I feel that the sysfs memory directory really isn't human readable, not with 10's or 100's of thousands of entries. Perhaps moving to a flat file representation and not create all the directories?
-Nathan
Quoted 5 lines · Nathan Fontenot
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Proposed patch to disable memory hotplug via a boot option, mem_hotplug=[on|off]. The patch only disables memory hotplug in that it prevents the creation of the memory sysfs directories for memory sections.
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
This patch is meant to help alleviate very long boot times on systems with large memory (1+ TB) and many memory sections (10's of thousands).
Why making simple /sys file is so slowly? Couldn't we fix such performance problem?
The issue is the large number of sysfs memory directories that get created. On a system with 1 TB of memory I am seeing ~63,00 directories. The long
Maybe you could simply make the memory blocks larger on such large systems? I agree 6k blocks are probably not very useful, but maybe 10 or 100 are?
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
Quoted 9 lines · Andi Kleen
Quoted 8 lines · Nathan Fontenot
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Proposed patch to disable memory hotplug via a boot option, mem_hotplug=[on|off]. The patch only disables memory hotplug in that it prevents the creation of the memory sysfs directories for memory sections.
This patch is meant to help alleviate very long boot times on systems with large memory (1+ TB) and many memory sections (10's of thousands).
Quoted 9 lines
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
Why making simple /sys file is so slowly? Couldn't we fix such performance problem?
The issue is the large number of sysfs memory directories that get created. On a system with 1 TB of memory I am seeing ~63,00 directories. The long
Maybe you could simply make the memory blocks larger on such large systems? I agree 6k blocks are probably not very useful, but maybe 10 or 100 are?
Yes, this would work to reduce the number of memory sections created. The reason I have not gone this route is that increasing the memory section size would break DLPAR memory add and remove for powerpc pseries.
-Nathan
Quoted 1 line
Yes, this would work to reduce the number of memory sections created. The reason
Note it could be done on the sysfs interface level too, e.g. have one entry that gets a bitmap as input/output.
Quoted 2 lines
I have not gone this route is that increasing the memory section size would break DLPAR memory add and remove for powerpc pseries.
How would it break it?
But aren't you breaking this anyways by disabling the sysfs entries?!?
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
Quoted 4 lines · Andi Kleen
Quoted 1 line
Yes, this would work to reduce the number of memory sections created. The reason
Note it could be done on the sysfs interface level too, e.g. have one entry that gets a bitmap as input/output.
This was sort of the idea I was thinking of for a flat representation in sysfs. Instead of creating all of the directories for each memory section, have a single file for the memory section attributes. For example, an 'online' file. Users could read the file to get online memory section numbers and write to it to online a specific memory section.
Quoted 5 lines
Quoted 3 lines
I have not gone this route is that increasing the memory section size would break DLPAR memory add and remove for powerpc pseries.
How would it break it?
For the powerpc/pseries hardware memory is assigned in LMB size chunks. If the memory section size is larger than the LMB size then DLPAR remove would have to hotplug remove more memory than required to DLPAR remove the memory. For DLPAR memory add, we would fail because we could not hotplug add the larger memory section since that much memory had not actually been given to the machine.
Quoted 2 lines
But aren't you breaking this anyways by disabling the sysfs entries?!?
This is meant more for allowing people to bot systems without the long boot times (as mentioned before 8+ hours for a 2TB system) even if it means they will not be able to memory hotplug or memory DLPAR.
-Nathan
Quoted 10 lines · Andi Kleen
Quoted 9 lines
Quoted 7 lines · KOSAKI Motohiro
Quoted 6 lines
Proposed patch to disable memory hotplug via a boot option, mem_hotplug=[on|off]. The patch only disables memory hotplug in that it prevents the creation of the memory sysfs directories for memory sections.
This patch is meant to help alleviate very long boot times on systems with large memory (1+ TB) and many memory sections (10's of thousands).
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
Why making simple /sys file is so slowly? Couldn't we fix such performance problem?
The issue is the large number of sysfs memory directories that get created. On a system with 1 TB of memory I am seeing ~63,00 directories. The long creation time is due to the string compare check in sysfs code to ensure we are not creating a directory with a duplicate name.
Ah, I see. probably this is sysfs issue. So Let's cc Greg and Eric. Greg, I have dumb question. Why sysfs call strcmp() so heavily? I mean why sysfs don't have hash based name dupliation check?
- kosaki
Quoted 43 lines
Quoted 5 lines
Yes I agree this really needs to be fixed properly, not hacked around with an option. Nathan can you please post some profile logs of the long boot times?
At this point the only profiling data I have is from adding a printk before and after the creation of the memory sysfs files in
drivers/base/memory.cand booting with printk.time=1.With 250 GB of memory: 10 seconds
[ 0.539562] Memory Start [ 10.450409] Memory EndWith 1 TB of memory: 9.1 minutes
[ 31.680168] Memory Start [ 584.186500] Memory EndI am hoping to get access to a machine with 2 TB of memory sometime soon and can post data for boot times on an unpatched kernel.
I posted a patch earlier that updated sysfs to use reb-black trees to store the sysfs dirent structs (though the patch has issues with namespaces). Using this patch on a 1 TB system the memory sysfs dir creation dropped to 2.2 minutes.
[ 1.295874] Memory Start [ 137.293510] Memory EndOn a system with 2 TB of memory, the reb-black tree patch dropped the sysfs file creation time down to 33 minutes and total boot time to 1 hour 15 minutes. I haven't measured an unpatched kernel, but total boot of an unpatched kernel is just over 8 hours.
With 2 TB and patched kernel: 33 minutes
[ 3.241679] Memory Start [ 1986.973324] Memory EndI am open to other ideas on a solution for this. With this many memory sections I feel that the sysfs memory directory really isn't human readable, not with 10's or 100's of thousands of entries. Perhaps moving to a flat file representation and not create all the directories?
-Nathan
Quoted 13 lines · KOSAKI Motohiro
Quoted 13 lines · Andi Kleen
Quoted 11 lines
Quoted 9 lines · KOSAKI Motohiro
Quoted 6 lines
Proposed patch to disable memory hotplug via a boot option, mem_hotplug=[on|off]. The patch only disables memory hotplug in that it prevents the creation of the memory sysfs directories for memory sections.
This patch is meant to help alleviate very long boot times on systems with large memory (1+ TB) and many memory sections (10's of thousands).
Why making simple /sys file is so slowly? Couldn't we fix such performance problem?
Quoted 8 lines
Quoted 4 lines
The issue is the large number of sysfs memory directories that get created. On a system with 1 TB of memory I am seeing ~63,00 directories. The long creation time is due to the string compare check in sysfs code to ensure we are not creating a directory with a duplicate name.
Ah, I see. probably this is sysfs issue. So Let's cc Greg and Eric. Greg, I have dumb question. Why sysfs call
strcmp()so heavily? I mean why sysfs don't have hash based name dupliation check?
Simplicity of the current implementation.
I have a prototype patch sitting around somewhere. I think ultimately it makes sense to do something like extN's htree directory structure in sysfs. I wanted to get the tagged sysfs support in before I worked on scalability because that slightly changes the requirements.
Improving the scalability here is certainly worth doing, but I am slightly concerned there is something else algorithmically wrong if this is still going to take 33 minutes to boot with 2TB.
Eric
Quoted 8 lines
I have a prototype patch sitting around somewhere. I think ultimately it makes sense to do something like extN's htree directory structure in sysfs. I wanted to get the tagged sysfs support in before I worked on scalability because that slightly changes the requirements.
Improving the scalability here is certainly worth doing, but I am slightly concerned there is something else algorithmically wrong if this is still going to take 33 minutes to boot with 2TB.
I'm don't think thousands of entries in sysfs is really a good idea. Even if you fix the the insert algorithm issues a simple ls will still be very slow and there will be likely other issues too. And nobody can claim that's a good interface.
This really needs a better a interface.
e.g. add some wildcard entry as it was earlier proposed. and only generate the wildcard when some threshold is exceeded.
-Andi
Quoted 5 lines · Andi Kleen
Quoted 5 lines
I have a prototype patch sitting around somewhere. I think ultimately it makes sense to do something like extN's htree directory structure in sysfs. I wanted to get the tagged sysfs support in before I worked on scalability because that slightly changes the requirements.
Quoted 7 lines
Quoted 3 lines
Improving the scalability here is certainly worth doing, but I am slightly concerned there is something else algorithmically wrong if this is still going to take 33 minutes to boot with 2TB.
I'm don't think thousands of entries in sysfs is really a good idea. Even if you fix the the insert algorithm issues a simple ls will still be very slow and there will be likely other issues too. And nobody can claim that's a good interface.
Yes. I am much more interested in fixing lookup and stat performance, if people only look at insert performance it is a bit of a joke.
That said there are cases of people using a lot of network virtual network devices that are essentially sane that push sysfs.
This is only the second or third time sysfs scalability has come up this year. So at some level I think it is sane to fix sysfs regardless of what we do with memory hotplug.
Ensuring that our insert times are O(logN) for insert, O(N) for readdir, O(logN) for lookup and O(log1) for stat, seems useful if we can do it without other complications.
Eric
Quoted 13 lines · KOSAKI Motohiro
Quoted 13 lines · Andi Kleen
Quoted 11 lines
Quoted 9 lines · KOSAKI Motohiro
Quoted 6 lines
Proposed patch to disable memory hotplug via a boot option, mem_hotplug=[on|off]. The patch only disables memory hotplug in that it prevents the creation of the memory sysfs directories for memory sections.
This patch is meant to help alleviate very long boot times on systems with large memory (1+ TB) and many memory sections (10's of thousands).
Why making simple /sys file is so slowly? Couldn't we fix such performance problem?
Quoted 8 lines
Quoted 4 lines
The issue is the large number of sysfs memory directories that get created. On a system with 1 TB of memory I am seeing ~63,00 directories. The long creation time is due to the string compare check in sysfs code to ensure we are not creating a directory with a duplicate name.
Ah, I see. probably this is sysfs issue. So Let's cc Greg and Eric. Greg, I have dumb question. Why sysfs call
strcmp()so heavily? I mean why sysfs don't have hash based name dupliation check?
Because we have not needed such complexity before.
You might want to take a step back and asky why you are creating 63 thousand directories in sysfs for large memory systems. Are you really going to use all of those directories? What are they for? Perhaps someone created the wrong interface to memory and that needs to be fixed instead?
And as always, patches are welcome :)
thanks,
greg k-h
Quoted 23 lines · Greg KH
Quoted 13 lines · KOSAKI Motohiro
Quoted 13 lines · Andi Kleen
Quoted 11 lines
Quoted 9 lines · KOSAKI Motohiro
Quoted 6 lines
Proposed patch to disable memory hotplug via a boot option, mem_hotplug=[on|off]. The patch only disables memory hotplug in that it prevents the creation of the memory sysfs directories for memory sections.
This patch is meant to help alleviate very long boot times on systems with large memory (1+ TB) and many memory sections (10's of thousands).
Why making simple /sys file is so slowly? Couldn't we fix such performance problem?
Quoted 8 lines
Quoted 4 lines
The issue is the large number of sysfs memory directories that get created. On a system with 1 TB of memory I am seeing ~63,00 directories. The long creation time is due to the string compare check in sysfs code to ensure we are not creating a directory with a duplicate name.
Ah, I see. probably this is sysfs issue. So Let's cc Greg and Eric. Greg, I have dumb question. Why sysfs call
strcmp()so heavily? I mean why sysfs don't have hash based name dupliation check?
Quoted 8 lines
Because we have not needed such complexity before.
You might want to take a step back and asky why you are creating 63 thousand directories in sysfs for large memory systems. Are you really going to use all of those directories? What are they for? Perhaps someone created the wrong interface to memory and that needs to be fixed instead?
The directories being created are the standard directories, one for each of the memory sections present at boot. I think the most used files in each of these directories is the state and removable file used to do memory hotplug.
-Nathan
Quoted 33 lines · Nathan Fontenot
Quoted 23 lines · Greg KH
Quoted 13 lines · KOSAKI Motohiro
Quoted 13 lines · Andi Kleen
Quoted 11 lines
Quoted 9 lines · KOSAKI Motohiro
Quoted 6 lines
Proposed patch to disable memory hotplug via a boot option, mem_hotplug=[on|off]. The patch only disables memory hotplug in that it prevents the creation of the memory sysfs directories for memory sections.
This patch is meant to help alleviate very long boot times on systems with large memory (1+ TB) and many memory sections (10's of thousands).
Why making simple /sys file is so slowly? Couldn't we fix such performance problem?
Quoted 8 lines
Quoted 4 lines
The issue is the large number of sysfs memory directories that get created. On a system with 1 TB of memory I am seeing ~63,00 directories. The long creation time is due to the string compare check in sysfs code to ensure we are not creating a directory with a duplicate name.
Ah, I see. probably this is sysfs issue. So Let's cc Greg and Eric. Greg, I have dumb question. Why sysfs call
strcmp()so heavily? I mean why sysfs don't have hash based name dupliation check?Quoted 8 lines
Because we have not needed such complexity before.
You might want to take a step back and asky why you are creating 63 thousand directories in sysfs for large memory systems. Are you really going to use all of those directories? What are they for? Perhaps someone created the wrong interface to memory and that needs to be fixed instead?
Quoted 3 lines
The directories being created are the standard directories, one for each of the memory sections present at boot. I think the most used files in each of these directories is the state and removable file used to do memory hotplug.
And perhaps we shouldn't really be creating so many directories? Why not work with the memory hotplug developers to change their interface to not abuse sysfs in such a manner?
thanks,
greg k-h
Quoted 7 lines · Greg KH
Quoted 3 lines
The directories being created are the standard directories, one for each of the memory sections present at boot. I think the most used files in each of these directories is the state and removable file used to do memory hotplug.
And perhaps we shouldn't really be creating so many directories? Why not work with the memory hotplug developers to change their interface to not abuse sysfs in such a manner?
Heh, it wasn't abuse until we got this much memory. But, I think this one is pretty much 100% my fault.
Nathan, I think the right fix here is probably to untie sysfs from the sections a bit. We should be able to have sysfs dirs that represent more than one contiguous SECTION_SIZE area of memory.
It will mean re-teaching some of the tools how things work. They'll have to know that you can split mem sections, and we'll have to come up with a way to do the splitting.
Does ppc *really* remove 16MB sections of RAM these days? It's probably worth checking with the firmware folks to see what the limits are in practice.
-- Dave
Quoted 9 lines · Greg KH
Quoted 7 lines
Quoted 3 lines
The directories being created are the standard directories, one for each of the memory sections present at boot. I think the most used files in each of these directories is the state and removable file used to do memory hotplug.
And perhaps we shouldn't really be creating so many directories? Why not work with the memory hotplug developers to change their interface to not abuse sysfs in such a manner?
Quoted 6 lines
Heh, it wasn't abuse until we got this much memory. But, I think this one is pretty much 100% my fault.
Nathan, I think the right fix here is probably to untie sysfs from the sections a bit. We should be able to have sysfs dirs that represent more than one contiguous
SECTION_SIZEarea of memory.
Why do we need abi breakage? Yourself talked about we guess ppc don't actually need 16MB section. I think IBM folks have to confirm it. If our guessing is correct, the firmware fixing is only necessary.
Thats said, I don't 100% refuse your idea. it's interesting. but, In generical I hate _unncessary_ abi change.
Quoted 10 lines
It will mean re-teaching some of the tools how things work. They'll have to know that you can split mem sections, and we'll have to come up with a way to do the splitting.
Does ppc *really* remove 16MB sections of RAM these days? It's probably worth checking with the firmware folks to see what the limits are in practice.
-- Dave
Quoted 12 lines · KOSAKI Motohiro
Quoted 12 lines · Greg KH
Quoted 7 lines
Quoted 3 lines
The directories being created are the standard directories, one for each of the memory sections present at boot. I think the most used files in each of these directories is the state and removable file used to do memory hotplug.
And perhaps we shouldn't really be creating so many directories? Why not work with the memory hotplug developers to change their interface to not abuse sysfs in such a manner?
Heh, it wasn't abuse until we got this much memory. But, I think this one is pretty much 100% my fault.
Quoted 7 lines
Quoted 3 lines
Nathan, I think the right fix here is probably to untie sysfs from the sections a bit. We should be able to have sysfs dirs that represent more than one contiguous
SECTION_SIZEarea of memory.Why do we need abi breakage? Yourself talked about we guess ppc don't actually need 16MB section. I think IBM folks have to confirm it. If our guessing is correct, the firmware fixing is only necessary.
Yes, ppc still needs to support add/remove of 16MB sections. This correlates to the smallest lmb size on ppc that we need to support.
Quoted 3 lines
Thats said, I don't 100% refuse your idea. it's interesting. but, In generical I hate _unncessary_ abi change.
Me too, but I'm not sure the current sysfs layout of memory scales well for machines with huge amounts of memory.
How about providing an alternate sysfs layout for systems that have a large number of memory sections? Even on the machines I worked with that have 1 and 2 TB of memory, if we increase the memory sections size to equal the lmb size we still would be creating 6k+ directories for a 1 TB machine. This would alleviate much of the perfomrance issue but still leaves us with a directory of thousands (or tens of thousands for really big systems) of memoryXXX subdirectories, which is not really human readable.
Or some method of having a single memory XXX dir represent multiple sections, as Dave suggested would work. Perhaps there is a way to subdivide the memory section dirs into separate dirs based on their node.
At the point of dealing with this many memory sections would it make sense to not create directories for each of the memory sections? Perhaps just files to report information about the memory sections.
-Nathan
Quoted 17 lines · KOSAKI Motohiro
Quoted 16 lines
Quoted 15 lines · Greg KH
Quoted 7 lines
Quoted 3 lines
The directories being created are the standard directories, one for each of the memory sections present at boot. I think the most used files in each of these directories is the state and removable file used to do memory hotplug.
And perhaps we shouldn't really be creating so many directories? Why not work with the memory hotplug developers to change their interface to not abuse sysfs in such a manner?
Heh, it wasn't abuse until we got this much memory. But, I think this one is pretty much 100% my fault.
Nathan, I think the right fix here is probably to untie sysfs from the sections a bit. We should be able to have sysfs dirs that represent more than one contiguous
SECTION_SIZEarea of memory.
Quoted 6 lines
Quoted 3 lines
Why do we need abi breakage? Yourself talked about we guess ppc don't actually need 16MB section. I think IBM folks have to confirm it. If our guessing is correct, the firmware fixing is only necessary.
Yes, ppc still needs to support add/remove of 16MB sections. This correlates to the smallest lmb size on ppc that we need to support.
okey. I'm not against the change by strong reason. fortunatelly hotplug users are still few. If we maintain CONFIG_OLD_MEMHOTPLUG_LAYOUT a while time, I guess
nobody oppose new one.
Quoted 6 lines
Quoted 2 lines
Thats said, I don't 100% refuse your idea. it's interesting. but, In generical I hate _unncessary_ abi change.
Me too, but I'm not sure the current sysfs layout of memory scales well for machines with huge amounts of memory.
Quoted 7 lines
How about providing an alternate sysfs layout for systems that have a large number of memory sections? Even on the machines I worked with that have 1 and 2 TB of memory, if we increase the memory sections size to equal the lmb size we still would be creating 6k+ directories for a 1 TB machine. This would alleviate much of the perfomrance issue but still leaves us with a directory of thousands (or tens of thousands for really big systems) of memoryXXX subdirectories, which is not really human readable.
?? human readable?
As far as I observed, this dir have been no human readable since it was born. but nobody complained this one because only hotplug-script need to read this.
Am I missing something?
Quoted 7 lines
Or some method of having a single memory XXX dir represent multiple sections, as Dave suggested would work. Perhaps there is a way to subdivide the memory section dirs into separate dirs based on their node.
At the point of dealing with this many memory sections would it make sense to not create directories for each of the memory sections? Perhaps just files to report information about the memory sections.
Probably, It will works. I have one question. How many dir do the patch reduce on your machine? Do we need to combinate Eric's hash-dir patch?
Quoted 16 lines · KOSAKI Motohiro
Quoted 9 lines · Greg KH
Quoted 7 lines
Quoted 3 lines
The directories being created are the standard directories, one for each of the memory sections present at boot. I think the most used files in each of these directories is the state and removable file used to do memory hotplug.
And perhaps we shouldn't really be creating so many directories? Why not work with the memory hotplug developers to change their interface to not abuse sysfs in such a manner?
Quoted 6 lines
Heh, it wasn't abuse until we got this much memory. But, I think this one is pretty much 100% my fault.
Nathan, I think the right fix here is probably to untie sysfs from the sections a bit. We should be able to have sysfs dirs that represent more than one contiguous
SECTION_SIZEarea of memory.
Quoted 5 lines
Why do we need abi breakage? Yourself talked about we guess ppc don't actually need 16MB section. I think IBM folks have to confirm it. If our guessing is correct, the firmware fixing is only necessary.
From the mouth of the kernel dumbass who coded this up: it's not the
firmware's fault. We shouldn't punt this to them, and the proper fix _isn't_ in the firmware, plus they may have other more fundamental reasons to keep the LMB sizes what they are.
-- Dave
Quoted 17 lines · Dave Hansen
Quoted 16 lines · KOSAKI Motohiro
Quoted 15 lines · Greg KH
Quoted 7 lines
Quoted 3 lines
The directories being created are the standard directories, one for each of the memory sections present at boot. I think the most used files in each of these directories is the state and removable file used to do memory hotplug.
And perhaps we shouldn't really be creating so many directories? Why not work with the memory hotplug developers to change their interface to not abuse sysfs in such a manner?
Heh, it wasn't abuse until we got this much memory. But, I think this one is pretty much 100% my fault.
Nathan, I think the right fix here is probably to untie sysfs from the sections a bit. We should be able to have sysfs dirs that represent more than one contiguous
SECTION_SIZEarea of memory.
Quoted 8 lines
Quoted 5 lines
Why do we need abi breakage? Yourself talked about we guess ppc don't actually need 16MB section. I think IBM folks have to confirm it. If our guessing is correct, the firmware fixing is only necessary.
From the mouth of the kernel dumbass who coded this up: it's not the
firmware's fault. We shouldn't punt this to them, and the proper fix _isn't_ in the firmware, plus they may have other more fundamental reasons to keep the LMB sizes what they are.
I agree, this should be fixed in the api to userspace, having this many sysfs directories and/or files is just looney.
thanks,
greg k-h
Quoted 23 lines · On Tue, 29 Jun 2010 11:04:15 -0700 Greg KH <[email protected]> wrote:
Quoted 22 lines · Dave Hansen
Quoted 21 lines · KOSAKI Motohiro
Quoted 9 lines · Greg KH
Quoted 7 lines
Quoted 3 lines
The directories being created are the standard directories, one for each of the memory sections present at boot. I think the most used files in each of these directories is the state and removable file used to do memory hotplug.
And perhaps we shouldn't really be creating so many directories? Why not work with the memory hotplug developers to change their interface to not abuse sysfs in such a manner?
Quoted 6 lines
Heh, it wasn't abuse until we got this much memory. But, I think this one is pretty much 100% my fault.
Nathan, I think the right fix here is probably to untie sysfs from the sections a bit. We should be able to have sysfs dirs that represent more than one contiguous
SECTION_SIZEarea of memory.Why do we need abi breakage? Yourself talked about we guess ppc don't actually need 16MB section. I think IBM folks have to confirm it. If our guessing is correct, the firmware fixing is only necessary.
From the mouth of the kernel dumbass who coded this up: it's not the
Quoted 9 lines
Quoted 3 lines
firmware's fault. We shouldn't punt this to them, and the proper fix _isn't_ in the firmware, plus they may have other more fundamental reasons to keep the LMB sizes what they are.
I agree, this should be fixed in the api to userspace, having this many sysfs directories and/or files is just looney.
thanks,
Hmm, adding
CONFIG_NEW_MEMORY_SYSFS_LAYOUT or memory_sysfs_layout=small boot option
and adding a scalable interface for large scale machines ? I'd like to consider something..
Thanks,
-Kame
Quoted 30 lines · KAMEZAWA Hiroyuki
Quoted 27 lines · On Tue, 29 Jun 2010 11:04:15 -0700 Greg KH <[email protected]> wrote:
Quoted 25 lines · Dave Hansen
Quoted 16 lines · KOSAKI Motohiro
Quoted 9 lines · Greg KH
Quoted 7 lines
Quoted 3 lines
The directories being created are the standard directories, one for each of the memory sections present at boot. I think the most used files in each of these directories is the state and removable file used to do memory hotplug.
And perhaps we shouldn't really be creating so many directories? Why not work with the memory hotplug developers to change their interface to not abuse sysfs in such a manner?
Quoted 6 lines
Heh, it wasn't abuse until we got this much memory. But, I think this one is pretty much 100% my fault.
Nathan, I think the right fix here is probably to untie sysfs from the sections a bit. We should be able to have sysfs dirs that represent more than one contiguous
SECTION_SIZEarea of memory.Quoted 5 lines
Why do we need abi breakage? Yourself talked about we guess ppc don't actually need 16MB section. I think IBM folks have to confirm it. If our guessing is correct, the firmware fixing is only necessary.
From the mouth of the kernel dumbass who coded this up: it's not the
firmware's fault. We shouldn't punt this to them, and the proper fix _isn't_ in the firmware, plus they may have other more fundamental reasons to keep the LMB sizes what they are.
Quoted 8 lines
Quoted 5 lines
I agree, this should be fixed in the api to userspace, having this many sysfs directories and/or files is just looney.
thanks,
Hmm, adding
CONFIG_NEW_MEMORY_SYSFS_LAYOUTor
It will not be "NEW" in a year :)
Quoted 1 line
memory_sysfs_layout=small boot option
Yes, that sounds good.
Quoted 2 lines
and adding a scalable interface for large scale machines ? I'd like to consider something..
Dynamically changing the layout on big memory boxes makes sense to me, how about you?
thanks,
greg k-h
Quoted 6 lines · On Wed, 30 Jun 2010 08:47:55 -0700 Greg KH <[email protected]> wrote:
Quoted 2 lines
and adding a scalable interface for large scale machines ? I'd like to consider something..
Dynamically changing the layout on big memory boxes makes sense to me, how about you?
like this ?
==
boot option:
memory_sysfs_layout=compact
memory_sysfs_layout=auto (default)
memory_sysfs_layout=full
Considering briefly, how about this compact layout ?
/sys/devices/system/memory/:
list, hide, show, memoryX...
list: // show available memory index list.
#cat list
0 1 2 ....10000...
show: //an interface to enable the interface.
#echo INDEX > memory_index
will create memoryINDEX diretory.
hide: //an interface to hide the interface.
#echo INDEX > memory_hide
will remove memoryINDEX sysfs directory.
In compact mode, all memoryX interface are hidden at boot. In full mode, all memoryX interaface are shown.
The Boot option just affects status at boot. If users want, he can make all memory sysfs in shown state.
At hot-add event (via acpi) or probe-event, newly created memory section should be start from "shown" mode. hotplug scirpt can hide it after online.
At hot-remove, the users has to offline memory before hotplug. He'll has to do check list and show interface.
I think this change is not very difficult technically but can this kind of interface be allowed ?
Thanks,
-Kame
Quoted 34 lines · KAMEZAWA Hiroyuki
Quoted 6 lines · On Wed, 30 Jun 2010 08:47:55 -0700 Greg KH <[email protected]> wrote:
Quoted 2 lines
and adding a scalable interface for large scale machines ? I'd like to consider something..
Dynamically changing the layout on big memory boxes makes sense to me, how about you?
like this ?
==
boot option:
memory_sysfs_layout=compact
memory_sysfs_layout=auto (default)
memory_sysfs_layout=fullConsidering briefly, how about this compact layout ?
/sys/devices/system/memory/:
list, hide, show, memoryX...list: // show available memory index list.
#cat list
0 1 2 ....10000...show: //an interface to enable the interface.
#echo INDEX >memory_index
will create memoryINDEX diretory.hide: //an interface to hide the interface.
#echo INDEX >memory_hide
will remove memoryINDEX sysfs directory.
Quoted 4 lines
In compact mode, all memoryX interface are hidden at boot. In full mode, all memoryX interaface are shown.
The Boot option just affects status at boot. If users want, he can make all memory sysfs in shown state.
Do we need to make something as complicated as dynamically adding and removing the sysfs directories? Why not a compact layout that just takes the files that currently reside in the memoryXX dirs and move them up to the memory directory. This would be state (which should probably be split into an 'online' and 'offline' file), removable, phys_index, and phys_device.
Doing a cat on each of these files would simply report the appropriate information for all of the memory sections present. We could even go as far storing the online status and removable status as a bitmap instead of putting it in the memory_block struct and use th in-kernel routines for printing bitmaps.
Users could then do memory hotplug by echo'ing the memory section to online to the 'online' file, and echo'ing the section number to the 'offline' file to offline a section.
In this mode ew would skip the creation of all of the sysfs nodes and completely remove the performance issue seen.
Since this new layout and possible mdification to the memory_block structure would not easily allow switching between he different layouts, perhaps this should be set via a config option.
-Nathan
Quoted 40 lines · On Wed, 30 Jun 2010 22:17:35 -0500 Nathan Fontenot <[email protected]> wrote:
Quoted 34 lines · KAMEZAWA Hiroyuki
Quoted 6 lines · On Wed, 30 Jun 2010 08:47:55 -0700 Greg KH <[email protected]> wrote:
Quoted 2 lines
and adding a scalable interface for large scale machines ? I'd like to consider something..
Dynamically changing the layout on big memory boxes makes sense to me, how about you?
like this ?
==
boot option:
memory_sysfs_layout=compact
memory_sysfs_layout=auto (default)
memory_sysfs_layout=fullConsidering briefly, how about this compact layout ?
/sys/devices/system/memory/:
list, hide, show, memoryX...list: // show available memory index list.
#cat list
0 1 2 ....10000...show: //an interface to enable the interface.
#echo INDEX >memory_index
will create memoryINDEX diretory.hide: //an interface to hide the interface.
#echo INDEX >memory_hide
will remove memoryINDEX sysfs directory.Quoted 4 lines
In compact mode, all memoryX interface are hidden at boot. In full mode, all memoryX interaface are shown.
The Boot option just affects status at boot. If users want, he can make all memory sysfs in shown state.
Quoted 6 lines
Do we need to make something as complicated as dynamically adding and removing the sysfs directories? Why not a compact layout that just takes the files that currently reside in the memoryXX dirs and move them up to the memory directory. This would be state (which should probably be split into an 'online' and 'offline' file), removable,
phys_index, andphys_device.
I've considered several patterns.
with 4096 bytes buffer of sysfs, "printting bitmap" just covers small amount of sections even with smart ASCII format to show contiguous range as a chunk. That's my concern. (and 'list' file in above example is impossible to impelement.)
If I can use rmdir/mkdir interface, it's much simple rather than above "echo". But it seems impossible.
Because we have memory information in /proc/iomem already, I think hide/show interface (instead of mkdir/rmdir) is not very bad.
If you can implement highly scalable interface, please. I just stop dreaming and ack yours.
Thanks,
-Kame
Quoted 49 lines · KAMEZAWA Hiroyuki
Quoted 40 lines · On Wed, 30 Jun 2010 22:17:35 -0500 Nathan Fontenot <[email protected]> wrote:
Quoted 34 lines · KAMEZAWA Hiroyuki
Quoted 6 lines · On Wed, 30 Jun 2010 08:47:55 -0700 Greg KH <[email protected]> wrote:
Quoted 2 lines
and adding a scalable interface for large scale machines ? I'd like to consider something..
Dynamically changing the layout on big memory boxes makes sense to me, how about you?
like this ?
==
boot option:
memory_sysfs_layout=compact
memory_sysfs_layout=auto (default)
memory_sysfs_layout=fullConsidering briefly, how about this compact layout ?
/sys/devices/system/memory/:
list, hide, show, memoryX...list: // show available memory index list.
#cat list
0 1 2 ....10000...show: //an interface to enable the interface.
#echo INDEX >memory_index
will create memoryINDEX diretory.hide: //an interface to hide the interface.
#echo INDEX >memory_hide
will remove memoryINDEX sysfs directory.Quoted 4 lines
In compact mode, all memoryX interface are hidden at boot. In full mode, all memoryX interaface are shown.
The Boot option just affects status at boot. If users want, he can make all memory sysfs in shown state.Quoted 6 lines
Do we need to make something as complicated as dynamically adding and removing the sysfs directories? Why not a compact layout that just takes the files that currently reside in the memoryXX dirs and move them up to the memory directory. This would be state (which should probably be split into an 'online' and 'offline' file), removable,
phys_index, andphys_device.
Quoted 6 lines
I've considered several patterns.
with 4096 bytes buffer of sysfs, "printting bitmap" just covers small amount of sections even with smart ASCII format to show contiguous range as a chunk. That's my concern. (and 'list' file in above example is impossible to impelement.)
Yes, that's my concern as well. sysfs is "one value per file" and if you are worried about the size of the sysfs buffer, something is wrong with your interface.
Quoted 2 lines
If I can use rmdir/mkdir interface, it's much simple rather than above "echo". But it seems impossible.
That's what configfs is for :)
Quoted 5 lines
Because we have memory information in
/proc/iomemalready, I think hide/show interface (instead of mkdir/rmdir) is not very bad.If you can implement highly scalable interface, please. I just stop dreaming and ack yours.
Why not just create your own filesystem for this, "memoryfs", where you can do what you want and you don't have to worry about sysfs apis and buffer sizes.
thanks,
greg k-h
Quoted 3 lines · On Thu, 1 Jul 2010 09:31:30 +0900 KAMEZAWA Hiroyuki…
I think this change is not very difficult technically but can this kind of interface be allowed ?
Here is a patch.
But I'm not a specialist of memory hotplug in these days. So, please modify this as required even when you like this.
This patch is onto mmotm but will not hunk with mainline, I hope. If nonsense, sorry for noise ;)
-Kame
==
Memory hotplug has interfaces for offlining memory per section. With large memory/small section machine, we tend to have too many sysfs directory as
/sys/devices/system/memory/memoryXXX/ (XXX is section name)
But this interface is only necessary when we do hotplug.
This patch adds 2 new interfaces as
/sys/devieces/system/memory/hide
/sys/devieces/system/memory/show
'hide' will remove sysfs directroy of given number's section. 'show' will create sysfs directroy of given number's section.
# echo 120 > /sys/devices/system/memory/hide
....memory120 will be removed.This patch also adds hidememorysysfs boot option to hide all sysfs for memory section at boot time.
--- Documentation/kernel-parameters.txt | 2 Documentation/memory-hotplug.txt | 25 +++++ drivers/base/memory.c | 165 +++++++++++++++++++++++++++++++++--- drivers/base/node.c | 2 4 files changed, 184 insertions(+), 10 deletions(-)
drivers/base/memory.c +155 -10
@@ -23,10 +23,12 @@#include <linux/mutex.h> #include <linux/stat.h> #include <linux/slab.h> +#include <linux/radix-tree.h> #include <asm/atomic.h> #include <asm/uaccess.h> +DEFINE_MUTEX (mem_sysfs_lock);#define MEMORY_CLASS_NAME "memory"static struct sysdev_class memory_sysdev_class = {
@@ -324,6 +326,8 @@ static int block_size_init(void) &attr_block_size_bytes.attr); } + +/* * Some architectures will have custom drivers to do this, and * will not need to do it from userspace. The fake hot-add code
@@ -363,6 +367,7 @@ static inline int memory_probe_init(void }#endif +#ifdef CONFIG_MEMORY_FAILURE /* * Support for offlining pages of memory
@@ -505,13 +510,14 @@ int remove_memory_block(unsigned long nostruct memory_block *mem; mem =find_memory_block (section); +if (!mem)/* already hidden ? */ +return 0 ;unregister_mem_sect_under_nodes (mem);mem_remove_simple_file (mem, phys_index);mem_remove_simple_file (mem, state);mem_remove_simple_file (mem, phys_device);mem_remove_simple_file (mem, removable);unregister_memory (mem, section); -return 0 ; }
@@ -521,20 +527,142 @@ int remove_memory_block(unsigned long no*/ int register_new_memory (int nid,struct mem_section *section) { -return add_memory_block (nid, section,MEM_OFFLINE ,HOTPLUG ); +int ret; + +mutex_lock (&mem_sysfs_lock); + ret =add_memory_block (nid, section,MEM_OFFLINE ,HOTPLUG ); +mutex_unlock (&mem_sysfs_lock); +return ret; }int unregister_memory_section (struct mem_section *section) { +int ret; +if (!present_section (section))return -EINVAL ; +mutex_lock (&mem_sysfs_lock); + ret =remove_memory_block (0 , section,0 ); +mutex_unlock (&mem_sysfs_lock); +return ret; +} + +/* Remember memory online/offline status for _hidden_ memory */ + +RADIX_TREE (hidden_mems,GFP_KERNEL ); +static int record_mem_status (unsigned long section_nr,int status) +{ +int ret; +long lstat = status+1 ; +if (radix_tree_preload (GFP_KERNEL )) +return -ENOMEM ; + ret =radix_tree_insert (&hidden_mems, section_nr, (void *)lstat); +radix_tree_preload_end (); +return ret; +} + +static int lookup_mem_status (unsigned long section_nr) +{ +void *ptr; +/* we already have big mutex */ + ptr=radix_tree_lookup (&hidden_mems, section_nr); +/* treate not-recorded mems'state as ONLINE */ +if (!ptr) +return MEM_ONLINE ; +return (long )ptr -1 ; +} + +static void forget_mem_status (unsigned long section_nr) +{ +radix_tree_delete (&hidden_mems, section_nr); +} + +static ssize_t +memory_show_store (struct class *class,struct class_attribute *attr, +const char *buf,size_t count) +{ +struct mem_section *section; +unsigned long section_nr; +int nid, status; +ssize_t ret; + + section_nr =simple_strtoull (buf,NULL ,0 ); +if (!present_section_nr (section_nr)) +return -EINVAL ; + section =__nr_to_section (section_nr); +VM_BUG_ON (!section); + +mutex_lock (&mem_sysfs_lock); + +if (find_memory_block (section)) { + ret = -EEXIST ; + goto out; + } + + nid =pfn_to_nid (section_nr_to_pfn (section_nr)); + + status =lookup_mem_status (section_nr); + ret =add_memory_block (nid, section, status,HOTPLUG ); +if (ret) + goto out; +forget_mem_status (section_nr); + ret = count; +out: +mutex_unlock (&mem_sysfs_lock); +return ret; +} -return remove_memory_block (0 , section,0 ); +static CLASS_ATTR(show, S_IWUSR, NULL, memory_show_store); +static inline int memory_show_init (void ) +{ +return sysfs_create_file (&memory_sysdev_class.kset.kobj, + &class_attr_show.attr); } +static ssize_t +memory_hide_store (struct class *class,struct class_attribute *attr, +const char *buf,size_t count) +{ +struct mem_section *section; +struct memory_block *mem; +unsigned long section_nr; +ssize_t ret = -EINVAL ; + + section_nr =simple_strtoull (buf,NULL ,0 ); +if (!present_section_nr (section_nr)) +return ret; + section =__nr_to_section (section_nr); +VM_BUG_ON (!section); + +mutex_lock (&mem_sysfs_lock); + mem =find_memory_block (section); +if (!mem) + goto out; +record_mem_status (section_nr, mem->state); +unregister_mem_sect_under_nodes (mem); +mem_remove_simple_file (mem, phys_index); +mem_remove_simple_file (mem, state); +mem_remove_simple_file (mem, phys_device); +mem_remove_simple_file (mem, removable); +unregister_memory (mem, section); +mutex_unlock (&mem_sysfs_lock); + ret = count; +out: +return ret; +} + +static CLASS_ATTR(hide, S_IWUSR, NULL, memory_hide_store); +static inline int memory_hide_init (void ) +{ +return sysfs_create_file (&memory_sysdev_class.kset.kobj, + &class_attr_hide.attr); +} + +/* * Initialize the sysfs support for memory devices... */ +static int hide_memory_sysfs__initdata ;int __init memory_dev_init (void ) {unsigned int i;
@@ -550,13 +678,16 @@ int __init memory_dev_init(void)* Create entries for memory sections that were found * during boot and have been initialized */ -for (i =0 ; i <NR_MEM_SECTIONS ; i++) { -if (!present_section_nr (i)) -continue ; - err =add_memory_block (0 ,__nr_to_section (i),MEM_ONLINE , -BOOT ); -if (!ret) - ret = err; +if (!hide_memory_sysfs){ +for (i =0 ; i <NR_MEM_SECTIONS ; i++) { +if (!present_section_nr (i)) +continue ; + err =add_memory_block (0 ,__nr_to_section (i), +MEM_ONLINE , +BOOT ); +if (!ret) + ret = err; + } } err =memory_probe_init ();
@@ -568,8 +699,22 @@ int __init memory_dev_init(void) err =block_size_init ();if (!ret) ret = err; + err =memory_show_init (); +if (!ret) + ret = err; + err =memory_hide_init (); +if (!ret) + ret = err; out:if (ret)printk (KERN_ERR "%s() failed: %d\n" , __func__, ret);return ret; } + +static int __init hidememorysysfs (char *s) +{ + hide_memory_sysfs =1 ; +return 1 ; +} +__setup ("hidememorysysfs" , hidememorysysfs); +
drivers/base/node.c +2 -0
@@ -421,6 +421,8 @@ static int link_mem_sections(int nid)continue ; mem_sect =__nr_to_section (section_nr); mem_blk =find_memory_block (mem_sect); +if (!mem_blk)/* hidden ? */ +continue ; ret =register_mem_sect_under_node (mem_blk, nid);if (!err) err = ret;
Documentation/memory-hotplug.txt +25 -0
@@ -15,6 +15,7 @@ be changed often. 1.3. Unit of Memory online/offline operation 2. Kernel Configuration 3. sysfs files for memory hotplug + 3.1 hide and show sysfs 4. Physical memory hot-add phase 4.1 Hardware(Firmware) Support 4.2 Notify memory hot-add event by hand
@@ -169,6 +170,30 @@ For example: A backlink will also be created: /sys/devices/system/memory/memory9/node0 -> ../../node/node0 +3.1 Hide and Show for sysfs. + +On some big memory system, memory syfs may contain too much sysfs +directory and consume kernel resource too much. To handle that, +memory sysfs has hide and show interface. + +hide : remove memoryXXX directory on demand. + A user can remove sysfs entry by writing memory section number. + +echo 120 > /sys/devices/system/memory/hide +...then, memory120 disappears. + +show : create memory XXX directroy on demand. + A user can add sysfs entry by writing memory section number. + If memory doesn't exit, this fails. + +echo 120 > /sys/dev/ices/system/memory/show +...then, memory120 directory is available. + +And we have boot option as "hidememorysysfs". This makes all +memoryXXX sysfs at boot to be hidden. All hot-added sections +will be visible automatically. Users can make memoryXXX sysfs +entries by 'show' interface. + -------------------------------- 4. Physical memory hot-add phase --------------------------------
Documentation/kernel-parameters.txt +2 -0
@@ -853,6 +853,8 @@ and is between 256 and 4096 characters. corresponding firmware-first mode error processing logic will be disabled. + hidememorysyfs [KNL, BOOT] hides memoryXXX directory in sysfs at BOOT. + highmem=nn[KMG] [KNL,BOOT] forces the highmem zone to have an exact size of <nn>. This works even on boxes that have no highmem otherwise. This also works to reduce highmem
Quoted 25 lines · KAMEZAWA Hiroyuki
Quoted 6 lines · On Wed, 30 Jun 2010 08:47:55 -0700 Greg KH <[email protected]> wrote:
Quoted 2 lines
and adding a scalable interface for large scale machines ? I'd like to consider something..
Dynamically changing the layout on big memory boxes makes sense to me, how about you?
like this ?
==
boot option:
memory_sysfs_layout=compact
memory_sysfs_layout=auto (default)
memory_sysfs_layout=fullConsidering briefly, how about this compact layout ?
/sys/devices/system/memory/:
list, hide, show, memoryX...list: // show available memory index list.
#cat list
0 1 2 ....10000...
Quoted 7 lines
show: //an interface to enable the interface.
#echo INDEX >memory_index
will create memoryINDEX diretory.hide: //an interface to hide the interface.
#echo INDEX >memory_hide
will remove memoryINDEX sysfs directory.
Ick, that can get confusing very quickly, and not really solve any of your root problems, right?
Quoted 5 lines
In compact mode, all memoryX interface are hidden at boot. In full mode, all memoryX interaface are shown.
The Boot option just affects status at boot. If users want, he can make all memory sysfs in shown state.
Quoted 8 lines
At hot-add event (via acpi) or probe-event, newly created memory section should be start from "shown" mode. hotplug scirpt can hide it after online.
At hot-remove, the users has to offline memory before hotplug. He'll has to do check list and show interface.
I think this change is not very difficult technically but can this kind of interface be allowed ?
Not really, I don't like it.
Why not just simplify what you currently have to not use so many directories and files?
And maybe, this doesn't belong in sysfs at all...
thanks,
greg k-h
Quoted 30 lines · On Thu, 1 Jul 2010 16:26:34 -0700 Greg KH <[email protected]> wrote:
Quoted 29 lines · KAMEZAWA Hiroyuki
Quoted 6 lines · On Wed, 30 Jun 2010 08:47:55 -0700 Greg KH <[email protected]> wrote:
Quoted 2 lines
and adding a scalable interface for large scale machines ? I'd like to consider something..
Dynamically changing the layout on big memory boxes makes sense to me, how about you?
like this ?
==
boot option:
memory_sysfs_layout=compact
memory_sysfs_layout=auto (default)
memory_sysfs_layout=fullConsidering briefly, how about this compact layout ?
/sys/devices/system/memory/:
list, hide, show, memoryX...list: // show available memory index list.
#cat list
0 1 2 ....10000...show: //an interface to enable the interface.
#echo INDEX >memory_index
will create memoryINDEX diretory.
Quoted 7 lines
Quoted 3 lines
hide: //an interface to hide the interface.
#echo INDEX >memory_hide
will remove memoryINDEX sysfs directory.Ick, that can get confusing very quickly, and not really solve any of your root problems, right?
Why not ?
I'm sorry if I misunderstand the problem . It is tooo many entries under a directory, which makes the routine slow. No ?
By this interface, the sysfs entries pops up on demand if the user hides it at boot time (by boot options.)
IIUC, the most problematic IBM's 16MB-sections system has no notification from firmware and their system management middleware controls all memory hotplug information. IOW, all memory-hotplug actions are from user-land, not from firmware. They should know which mem_section should be visible under /memory before starting hotplug.
In other platforms, using ACPI, hot-added memory sections can be automatically visible. At memory hot-remove, management software should now which phys_address memory should be removed before it starts action. It can make visible section directory.
For many many users, who never wants memory hotplug, they can avoid creating sysfs for memory hotplug and will be happy.
Quoted 14 lines
Quoted 5 lines
In compact mode, all memoryX interface are hidden at boot. In full mode, all memoryX interaface are shown.
The Boot option just affects status at boot. If users want, he can make all memory sysfs in shown state.Quoted 8 lines
At hot-add event (via acpi) or probe-event, newly created memory section should be start from "shown" mode. hotplug scirpt can hide it after online.
At hot-remove, the users has to offline memory before hotplug. He'll has to do check list and show interface.
I think this change is not very difficult technically but can this kind of interface be allowed ?
Quoted 7 lines
Not really, I don't like it.
Why not just simplify what you currently have to not use so many directories and files?
And maybe, this doesn't belong in sysfs at all...
Because the system has topology and sysfs's /sys/devices/system/ shows system's topology by directories and files, we use it. Especially, "node" information is important in some system and we'd made use of it.
If you recommend guys to remake all NUMA-cpu-memory topology information in other fs, someone will do.
Fujitsu will not complain about that and welcome it because we have a long period until RHEL7 ages.
Thanks,
-Kame
Quoted 10 lines · KAMEZAWA Hiroyuki
Considering briefly, how about this compact layout ?
/sys/devices/system/memory/:
list, hide, show, memoryX...list: // show available memory index list.
#cat list
0 1 2 ....10000...
Quoted 7 lines
show: //an interface to enable the interface.
#echo INDEX >memory_index
will create memoryINDEX diretory.hide: //an interface to hide the interface.
#echo INDEX >memory_hide
will remove memoryINDEX sysfs directory.
I was thinking more along the lines of just taking adjacent sections and merging them. We'll need a new "end address" or size file. Maybe "end_phys_index" or something similar.
Such a beast would not fix all of the pathological cases, like where only every other 16MB section is populated with RAM, but I don't think those are very common at all, especially in cases where there's a lot of RAM. But, it also has a chance of being relatively backward-compatible. In most cases, we may even be able to calculate a new phys_block_size where everything fits evenly and be fully backward-compatible with the old ABI.
-- Dave
Quoted 5 lines · Dave Hansen
I was thinking more along the lines of just taking adjacent sections and merging them. We'll need a new "end address" or size file. Maybe "
end_phys_index" or something similar.
Quoted 8 lines
Such a beast would not fix all of the pathological cases, like where only every other 16MB section is populated with RAM, but I don't think those are very common at all, especially in cases where there's a lot of RAM. But, it also has a chance of being relatively backward-compatible. In most cases, we may even be able to calculate a new
phys_block_sizewhere everything fits evenly and be fully backward-compatible with the old ABI.
Under this scenario were you thinking that all of the memory sections that reside under this memory block would then be acted upon as a whole. For example would we allow users to hotplug individual memory sections included in th block, or would the memory block be acted upon as a whole?
-Nathan
Quoted 14 lines · Nathan Fontenot
Quoted 5 lines · Dave Hansen
I was thinking more along the lines of just taking adjacent sections and merging them. We'll need a new "end address" or size file. Maybe "
end_phys_index" or something similar.Quoted 7 lines
Such a beast would not fix all of the pathological cases, like where only every other 16MB section is populated with RAM, but I don't think those are very common at all, especially in cases where there's a lot of RAM. But, it also has a chance of being relatively backward-compatible. In most cases, we may even be able to calculate a new
phys_block_sizewhere everything fits evenly and be fully backward-compatible with the old ABI.
Quoted 4 lines
Under this scenario were you thinking that all of the memory sections that reside under this memory block would then be acted upon as a whole. For example would we allow users to hotplug individual memory sections included in th block, or would the memory block be acted upon as a whole?
I think we would need a mechanism that allowed the sysfs directories to be broken down somehow. If the merging is very successful, it could lead to a case where no existing sysfs dir is a reasonable size to remove. That's what we'd have to avoid, so I think we'd _need_ splitting of some kind.
-- Dave
Quoted 20 lines · Dave Hansen
Quoted 14 lines · Nathan Fontenot
Quoted 5 lines · Dave Hansen
I was thinking more along the lines of just taking adjacent sections and merging them. We'll need a new "end address" or size file. Maybe "
end_phys_index" or something similar.Quoted 7 lines
Such a beast would not fix all of the pathological cases, like where only every other 16MB section is populated with RAM, but I don't think those are very common at all, especially in cases where there's a lot of RAM. But, it also has a chance of being relatively backward-compatible. In most cases, we may even be able to calculate a new
phys_block_sizewhere everything fits evenly and be fully backward-compatible with the old ABI.Quoted 4 lines
Under this scenario were you thinking that all of the memory sections that reside under this memory block would then be acted upon as a whole. For example would we allow users to hotplug individual memory sections included in th block, or would the memory block be acted upon as a whole?
Quoted 6 lines
I think we would need a mechanism that allowed the sysfs directories to be broken down somehow. If the merging is very successful, it could lead to a case where no existing sysfs dir is a reasonable size to remove. That's what we'd have to avoid, so I think we'd _need_ splitting of some kind.
Agreed.
Perhaps something along the following lines. We create a sysfs directory for each memory block, where each memory block can contain multiple memory sections. The default being one memory section per memory block. All of the existing files that are currently under each memoryXXX directory in sysfs still appear along with a new file as Dave suggested to indicate the number of memory sections contained in that memory block. Additionally, a new file named 'split' would allow users to split the memory block in half, thus creating a new directory for the split-off half of the block.
A slight change in the state file behavior would also need to occur, in that writing 'online' or 'offline' to the file would perform the appropriate operation on all of the memory sections contained in the memory block.
Thoughts? I can start working on this if no one has any objections.
-Nathan