: ACPI: Automatically online hot-added memory
Subsystems (5)
RFC PATCH 1/1 : ACPI: Automatically online hot-added memory
New sockets have on-die memory controllers. This means that in certain HW configurations the memory behind the socket comes and goes as the socket is physically enabled and disabled.
Since the cpu bringup code does on node memory allocations, the memory on the added socket must be onlined first.
Add a .config option to automatically online hot added memory, and enable it in the acpi memory add path.
drivers/acpi/Kconfig +8 -0
@@ -350,6 +350,14 @@ config ACPI_HOTPLUG_MEMORY To compile this driver as a module, choose M here: the module will be called acpi_memhotplug. +config ACPI_HOTPLUG_MEMORY_AUTO_ONLINE + bool "Automatically online hotplugged memory" + depends on ACPI_HOTPLUG_MEMORY + default n + help + This forces memory that is brought into service by ACPI + to be automatically onlined. + config ACPI_SBS tristate "Smart Battery System" depends on X86
drivers/acpi/acpi_memhotplug.c +14 -0
@@ -30,6 +30,7 @@#include <linux/init.h> #include <linux/types.h> #include <linux/memory_hotplug.h> +#include <linux/memory.h> #include <acpi/acpi_drivers.h> #define ACPI_MEMORY_DEVICE_CLASS "memory"
@@ -252,6 +253,19 @@ static int acpi_memory_enable_device(struct acpi_memory_device *mem_device) result =add_memory (node, info->start_addr, info->length);if (result)continue ; +#ifdef CONFIG_ACPI_HOTPLUG_MEMORY_AUTO_ONLINE +/* +* New processors require memory to be online before cpus. +* No notifications are required here as "we" are the only +* ones who know about the new memory right now. +*/ + result =online_pages (info->start_addr >>PAGE_SHIFT , + info->length >>PAGE_SHIFT ); +if (!result) +set_memory_state (info->start_addr,MEM_ONLINE ); +else +printk ("Memory online failed.\n" ); +#endif info->enabled =1 ; num_enabled++; }
drivers/base/memory.c +14 -0
@@ -510,6 +510,20 @@ int remove_memory_block(unsigned long node_id, struct mem_section *section, }/* +* need an interface for the VM to mark sections on and offline when hot-adding +* memory. +*/ +void set_memory_state (unsigned long addr,unsigned long state) +{ +struct mem_section *section; +struct memory_block *mem; + + section =__pfn_to_section (addr >>PAGE_SHIFT ); + mem =find_memory_block (section); + mem->state = state; +} + +/* * need an interface for the VM to add new memory regions, * but without onlining it. */
include/linux/memory.h +1 -0
@@ -112,6 +112,7 @@ extern int remove_memory_block(unsigned long, struct mem_section *, int);extern int memory_notify (unsigned long val,void *v);extern int memory_isolate_notify (unsigned long val,void *v);extern struct memory_block *find_memory_block (struct mem_section *); +extern void set_memory_state (unsigned long ,unsigned long );#define CONFIG_MEM_BLOCK_SIZE (PAGES_PER_SECTION<<PAGE_SHIFT)enum mem_add_context {BOOT ,HOTPLUG };#endif /* CONFIG_MEMORY_HOTPLUG_SPARSE */
Quoted 9 lines · Prarit Bhargava
New sockets have on-die memory controllers. This means that in certain HW configurations the memory behind the socket comes and goes as the socket is physically enabled and disabled.
Since the cpu bringup code does on node memory allocations, the memory on the added socket must be onlined first.
Add a .config option to automatically online hot added memory, and enable it in the acpi memory add path.
This seems like the right thing to do.
drivers/acpi/Kconfig config ACPI_HOTPLUG_MEMORY · line 356
Quoted 4 lines
drivers/acpi/Kconfig +4
-0+config ACPI_HOTPLUG_MEMORY_AUTO_ONLINE + bool "Automatically online hotplugged memory" + depends on ACPI_HOTPLUG_MEMORY + default n
default !S390? default x86?
drivers/acpi/acpi_memhotplug.c static int acpi_memory_enable_device(struct acpi_memory_device *mem_device) · line 267
Quoted 6 lines
drivers/acpi/acpi_memhotplug.c +6
-0+ result =online_pages (info->start_addr >>PAGE_SHIFT , + info->length >>PAGE_SHIFT ); +if (!result) +set_memory_state (info->start_addr,MEM_ONLINE ); +else +printk ("Memory online failed.\n" );
That probably wants to be more descriptive and have a loglevel. What happens if this fails? The CPU presumably can't be brought up?
Signature · 1 line
Matthew Garrett | [email protected]
Quoted 16 lines · Matthew Garrett
Quoted 10 lines · Prarit Bhargava
New sockets have on-die memory controllers. This means that in certain HW configurations the memory behind the socket comes and goes as the socket is physically enabled and disabled.
Since the cpu bringup code does on node memory allocations, the memory on the added socket must be onlined first.
Add a .config option to automatically online hot added memory, and enable it in the acpi memory add path.
This seems like the right thing to do.
Quoted 8 lines
drivers/acpi/Kconfig config ACPI_HOTPLUG_MEMORY · line 356
Quoted 5 lines
drivers/acpi/Kconfig +4
-0+config ACPI_HOTPLUG_MEMORY_AUTO_ONLINE + bool "Automatically online hotplugged memory" + depends on ACPI_HOTPLUG_MEMORY + default ndefault !S390? default x86?
I've been inquiring about this. It seems that some vendors want the capability to manually bring memory online. I'm trying to get more details as this doesn't seem intuitive. It seems to me that if you add memory you want to online it right now.
Quoted 10 lines
drivers/acpi/acpi_memhotplug.c static int acpi_memory_enable_device(struct acpi_memory_device *mem_device) · line 267
Quoted 8 lines
drivers/acpi/acpi_memhotplug.c +6
-0+ result =online_pages (info->start_addr >>PAGE_SHIFT , + info->length >>PAGE_SHIFT ); +if (!result) +set_memory_state (info->start_addr,MEM_ONLINE ); +else +printk ("Memory online failed.\n" );That probably wants to be more descriptive and have a loglevel.
Oops. I'm very bad about setting the loglevel's on my printk's ;) ... I'll fix that in the next patch.
Quoted 2 lines
What happens if this fails? The CPU presumably can't be brought up?
That's a good question... After thinking about it I think that acpi_processor_add() must check the node to see if it has memory before bringing the cpu online.
I'll code that up, test, and repost.
P.
Quoted 43 lines · Prarit Bhargava
Quoted 16 lines · Matthew Garrett
Quoted 10 lines · Prarit Bhargava
New sockets have on-die memory controllers. This means that in certain HW configurations the memory behind the socket comes and goes as the socket is physically enabled and disabled.
Since the cpu bringup code does on node memory allocations, the memory on the added socket must be onlined first.
Add a .config option to automatically online hot added memory, and enable it in the acpi memory add path.
This seems like the right thing to do.
Quoted 8 lines
drivers/acpi/Kconfig config ACPI_HOTPLUG_MEMORY · line 356
Quoted 5 lines
drivers/acpi/Kconfig +4
-0+config ACPI_HOTPLUG_MEMORY_AUTO_ONLINE + bool "Automatically online hotplugged memory" + depends on ACPI_HOTPLUG_MEMORY + default ndefault !S390? default x86?
I've been inquiring about this. It seems that some vendors want the capability to manually bring memory online. I'm trying to get more details as this doesn't seem intuitive. It seems to me that if you add memory you want to online it right now.
Quoted 10 lines
drivers/acpi/acpi_memhotplug.c static int acpi_memory_enable_device(struct acpi_memory_device *mem_device) · line 267
Quoted 8 lines
drivers/acpi/acpi_memhotplug.c +6
-0+ result =online_pages (info->start_addr >>PAGE_SHIFT , + info->length >>PAGE_SHIFT ); +if (!result) +set_memory_state (info->start_addr,MEM_ONLINE ); +else +printk ("Memory online failed.\n" );That probably wants to be more descriptive and have a loglevel.
Quoted 2 lines
Oops. I'm very bad about setting the loglevel's on my printk's ;) ... I'll fix that in the next patch.
One note is that the hot-added memory will be automatically onlined. And we had better add an interface so that the user-space can quiry whether the hot-added memory is automatically onlined. Otherwise the user-space will receive the corresponding event and online hot-added memory again. In such case it will get some error messages.
Quoted 8 lines
Quoted 4 lines
What
happens if this fails? The CPU presumably can't be brought up?That's a good question... After thinking about it I think that
acpi_processor_add()must check the node to see if it has memory before bringing the cpu online.
Why do we need to see whether the memory is onlined before bringing cpu to online state? It seems that there is no dependency between cpu online and memory online.
Thanks.
Quoted 9 lines
I'll code that up, test, and repost.
P.
--
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 7 lines
Why do we need to see whether the memory is onlined before bringing cpu to online state? It seems that there is no dependency between cpu online and memory online.
Yakui,
Here's a deeper look into the issue. New Intel processors have an on-die memory controller and this means that as the socket comes and goes, so does the memory "behind" the socket.
ie) with new processors it is possible that an entire node which consists of memory and cpus comes and goes with the socket enable and disable.
The cpu bringup code does local node allocations for the cpu. If the memory connected to the node (which is "behind" the socket) isn't online, then these allocations fail, and then the cpu bringup fails.
P.
Quoted 9 lines · Prarit Bhargava
Quoted 6 lines
Why do we need to see whether the memory is onlined before bringing cpu to online state? It seems that there is no dependency between cpu online and memory online.
Yakui,
Thanks for the explanation.
Quoted 3 lines
Here's a deeper look into the issue. New Intel processors have an on-die memory controller and this means that as the socket comes and goes, so does the memory "behind" the socket.
Yes. The nehalem processor has the integrated memory controller. But it is not required that the hot-added memory should be onlined before bringing up CPU.
I do the following memory-hotplug test on one Machine. a. Before hot plugging memory, four CPUs socket are installed and
all the logical CPU are brought up. (Only one node has the memory)
b. The memory is hot-plugged and then the memory is onlined so that
it can be accessed by the system.
In the above testing case the CPU is brought up before onlining the hot-added memory. And the test shows that it can work well.
Quoted 8 lines
ie) with new processors it is possible that an entire node which consists of memory and cpus comes and goes with the socket enable and disable.
The cpu bringup code does local node allocations for the cpu. If the memory connected to the node (which is "behind" the socket) isn't online, then these allocations fail, and then the cpu bringup fails.
If the CPU can't allocate the memory from its own node, it can turn to other node and see whether the memory can be allocated. And this depends on the NUMA allocation policy.
Quoted 3 lines
P.
--
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 11 lines
Yes. The nehalem processor has the integrated memory controller. But it is not required that the hot-added memory should be onlined before bringing up CPU.
I do the following memory-hotplug test on one Machine. a. Before hot plugging memory, four CPUs socket are installed and
all the logical CPU are brought up. (Only one node has the memory)
b. The memory is hot-plugged and then the memory is onlined so that
it can be accessed by the system.
Quoted 3 lines
In the above testing case the CPU is brought up before onlining the hot-added memory. And the test shows that it can work well.
That doesn't work when you have multiple nodes AFAICT. The cpus do not come into service because of a lack of memory on the node.... per node allocations will fail.
Just curious, exactly what did you test with? 2.6.33 (or newer)?
Quoted 10 lines
Quoted 5 lines
ie) with new processors it is possible that an entire node which consists of memory and cpus comes and goes with the socket enable and disable.
The cpu bringup code does local node allocations for the cpu. If the memory connected to the node (which is "behind" the socket) isn't online, then these allocations fail, and then the cpu bringup fails.
If the CPU can't allocate the memory from its own node, it can turn to other node and see whether the memory can be allocated. And this depends on the NUMA allocation policy.
Maybe that could work, but I haven't gotten that to work. Even if it does work, it's a HUGE performance hit :(. I can't imagine incurring an extra hop just to get to per_cpu memory. I'd rather bring the memory on the local node up first.
I think it is much better to bring the memory up first.
Either way, it's a nice feature to have.
P.
Quoted 7 lines
Quoted 5 lines
P.
Quoted 10 lines · Prarit Bhargava
Quoted 10 lines
Yes. The nehalem processor has the integrated memory controller. But it is not required that the hot-added memory should be onlined before bringing up CPU.
I do the following memory-hotplug test on one Machine. a. Before hot plugging memory, four CPUs socket are installed and
all the logical CPU are brought up. (Only one node has the memory)
b. The memory is hot-plugged and then the memory is onlined so that
it can be accessed by the system.
Quoted 7 lines
Quoted 3 lines
In the above testing case the CPU is brought up before onlining the hot-added memory. And the test shows that it can work well.
That doesn't work when you have multiple nodes AFAICT. The cpus do not come into service because of a lack of memory on the node.... per node allocations will fail.
In the test the system has multiple nodes. The reason is that the cpu without memory can turn to other node and allocate the memory.
Quoted 2 lines
Just curious, exactly what did you test with? 2.6.33 (or newer)?
I test it on 2.6.32 kernel.
Quoted 34 lines
Quoted 6 lines
Quoted 6 lines
ie) with new processors it is possible that an entire node which consists of memory and cpus comes and goes with the socket enable and disable.
Quoted 9 lines
Quoted 4 lines
The cpu bringup code does local node allocations for the cpu. If the memory connected to the node (which is "behind" the socket) isn't online, then these allocations fail, and then the cpu bringup fails.
If the CPU can't allocate the memory from its own node, it can turn to other node and see whether the memory can be allocated. And this depends on the NUMA allocation policy.
Maybe that could work, but I haven't gotten that to work. Even if it does work, it's a HUGE performance hit :(. I can't imagine incurring an extra hop just to get to
per_cpumemory. I'd rather bring the memory on the local node up first.I think it is much better to bring the memory up first.
Either way, it's a nice feature to have.
P.
Quoted 7 lines
Quoted 5 lines
P.
Quoted 15 lines · ykzhao
Quoted 14 lines · Prarit Bhargava
Quoted 13 lines
Yes. The nehalem processor has the integrated memory controller. But it is not required that the hot-added memory should be onlined before bringing up CPU.
I do the following memory-hotplug test on one Machine. a. Before hot plugging memory, four CPUs socket are installed and
all the logical CPU are brought up. (Only one node has the memory)
b. The memory is hot-plugged and then the memory is onlined so that
it can be accessed by the system.In the above testing case the CPU is brought up before onlining the hot-added memory. And the test shows that it can work well.
Quoted 6 lines
Quoted 3 lines
That doesn't work when you have multiple nodes AFAICT. The cpus do not come into service because of a lack of memory on the node.... per node allocations will fail.
In the test the system has multiple nodes. The reason is that the cpu without memory can turn to other node and allocate the memory.
I agree with Yakui. The memory and CPU are irrelevant in some way. CPU can get memory from other nodes if it hasn't local memory, though for now it has some issues. Andi is working on it now (http://lkml.org/lkml/2010/2/3/343)
BTW, how about using UDEV rules to do this operation. It looks more smooth. I know some Novell guy is working on it.
Quoted 30 lines
Quoted 3 lines
Just curious, exactly what did you test with? 2.6.33 (or newer)?
I test it on 2.6.32 kernel.
Quoted 24 lines
Quoted 6 lines
Quoted 6 lines
ie) with new processors it is possible that an entire node which consists of memory and cpus comes and goes with the socket enable and disable.
Quoted 9 lines
Quoted 4 lines
The cpu bringup code does local node allocations for the cpu. If the memory connected to the node (which is "behind" the socket) isn't online, then these allocations fail, and then the cpu bringup fails.
If the CPU can't allocate the memory from its own node, it can turn to other node and see whether the memory can be allocated. And this depends on the NUMA allocation policy.
Maybe that could work, but I haven't gotten that to work. Even if it does work, it's a HUGE performance hit :(. I can't imagine incurring an extra hop just to get to
per_cpumemory. I'd rather bring the memory on the local node up first.I think it is much better to bring the memory up first.
Either way, it's a nice feature to have.
Quoted 19 lines · chen gong
Quoted 17 lines · ykzhao
Quoted 15 lines · Prarit Bhargava
Quoted 9 lines
Yes. The nehalem processor has the integrated memory controller. But it is not required that the hot-added memory should be onlined before bringing up CPU.
I do the following memory-hotplug test on one Machine. a. Before hot plugging memory, four CPUs socket are installed and
all the logical CPU are brought up. (Only one node has the memory)
b. The memory is hot-plugged and then the memory is onlined so that
it can be accessed by the system.Quoted 4 lines
In the above testing case the CPU is brought up before onlining the hot-added memory. And the test shows that it can work well.
That doesn't work when you have multiple nodes AFAICT. The cpus do not come into service because of a lack of memory on the node.... per node allocations will fail.
Quoted 8 lines
Quoted 3 lines
In the test the system has multiple nodes. The reason is that the cpu without memory can turn to other node and allocate the memory.
I agree with Yakui. The memory and CPU are irrelevant in some way. CPU can get memory from other nodes if it hasn't local memory, though for now it has some issues. Andi is working on it now (http://lkml.org/lkml/2010/2/3/343)
Right -- I'm working with Andi on this to get RHEL working. So I have his slab patches, etc.. That's one of the reasons it isn't working right now.
Quoted 4 lines
BTW, how about using UDEV rules to do this operation. It looks more smooth. I know some Novell guy is working on it.
I was thinking about that, but the CPU comes online automatically, so why not memory? It seems to be a discrepancy between the two. And ... the two of you still aren't considering the obvious performance hit which I would think makes this a must have.
P.
Quoted 1 line
Quoted 1 line · On Thursday 11 March 2010 09:32:09 chen gong wrote:
On 2010-3-11 16:07, ykzhao wrote:
...
Quoted 2 lines
BTW, how about using UDEV rules to do this operation. It looks more smooth. I know some Novell guy is working on it.
I also know this guy :)
These would be the udev rules to automatically add memory/cpus:
SUBSYSTEM=="cpu", ACTION=="add", TEST=="online", ATTR{online}=="0", ATTR{online}="1", RUN+="/bin/logger onlining cpu: $env{DEVPATH}" SUBSYSTEM=="memory", ACTION=="add", TEST=="state", ATTR{state}=="offline", ATTR{state}="online", RUN+="/bin/logger onlining memory: $env{DEVPATH}"
But this should be the same as you suggest (at least the memory rule) to do in the kernel:
automatically online the memory, once hotadded.
I would not add any udev rules before this does not work reliably and currently it is totally broken, mainly because:
- not being able to alloc memory on foreign nodes (at least with slab)
- C-state, throttling and cpufreq set up is done without valid cpu_data(new_cpu) resulting in wrong C-state (and other) info
Thomas
Quoted 19 lines · Thomas Renninger
Quoted 2 lines · On Thursday 11 March 2010 09:32:09 chen gong wrote:
On 2010-3-11 16:07, ykzhao wrote:
...
Quoted 3 lines
BTW, how about using UDEV rules to do this operation. It looks more smooth. I know some Novell guy is working on it.
I also know this guy :)
These would be the udev rules to automatically add memory/cpus:
SUBSYSTEM=="cpu", ACTION=="add", TEST=="online", ATTR{online}=="0", ATTR{online}="1", RUN+="/bin/logger onlining cpu: $env{DEVPATH}" SUBSYSTEM=="memory", ACTION=="add", TEST=="state", ATTR{state}=="offline", ATTR{state}="online", RUN+="/bin/logger onlining memory: $env{DEVPATH}"But this should be the same as you suggest (at least the memory rule) to do in the kernel:
automatically online the memory, once hotadded.
Quoted 7 lines
I would not add any udev rules before this does not work reliably and currently it is totally broken, mainly because:
- not being able to alloc memory on foreign nodes (at least with slab)
- C-state, throttling and cpufreq set up is done without validcpu_data(new_cpu)resulting in wrong C-state (and other) info
Thomas, forgive my ignorance of udev rules ... but can one udev rule block another? ... That's not the best way to explain it ... here's an example:
CPU hot add on Nehalem-EX. Memory is brought online. Memory udev events are generated. CPU udev events are generated.
Can the memory udev events block the cpu udev events? ie) can I be assured that memory will come online before the cpus?
P.
Quoted 7 lines
Thomas
--
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 32 lines · Prarit Bhargava
Quoted 21 lines · Thomas Renninger
Quoted 1 line · On Thursday 11 March 2010 09:32:09 chen gong wrote:
On 2010-3-11 16:07, ykzhao wrote:
...
Quoted 2 lines
BTW, how about using UDEV rules to do this operation. It looks more smooth. I know some Novell guy is working on it.
I also know this guy :)
These would be the udev rules to automatically add memory/cpus:
SUBSYSTEM=="cpu", ACTION=="add", TEST=="online", ATTR{online}=="0", ATTR{online}="1", RUN+="/bin/logger onlining cpu: $env{DEVPATH}" SUBSYSTEM=="memory", ACTION=="add", TEST=="state", ATTR{state}=="offline", ATTR{state}="online", RUN+="/bin/logger onlining
memory: $env{DEVPATH}"But this should be the same as you suggest (at least the memory rule) to do in the kernel:
automatically online the memory, once hotadded.Quoted 5 lines
I would not add any udev rules before this does not work reliably and currently it is totally broken, mainly because:
- not being able to alloc memory on foreign nodes (at least with slab)
- C-state, throttling and cpufreq set up is done without validcpu_data(new_cpu)resulting in wrong C-state (and other) infoThomas, forgive my ignorance of udev rules ... but can one udev rule block another? ... That's not the best way to explain it ... here's an example:
Quoted 5 lines
CPU hot add on Nehalem-EX. Memory is brought online. Memory udev events are generated. CPU udev events are generated.
Can the memory udev events block the cpu udev events? ie) can I be assured that memory will come online before the cpus?
Blocking does not work.
But I have an idea, whatabout:
- CPU add rule which only adds a CPU if the corresponding Numa node already has onlined memory
- Memory add rule which adds the memory and also onlines CPUs if the Numa node still has offlined CPUs
This has the side effect that you online a CPU which may have been offlined on purpose if you hotadd memory on the same node..., a rather uncommon case.
A quick try showed that it would be convenient to add the Numa node to the uevent which is a bit tricky...
Ok, I got this working, but messed up the cpu hotplugging with my patches:
...
Unable to map lapic 32 to logical cpu number
...
This already worked...
I wanted to post something today, but I couldn't made it. Still I more or less could prove that above works and I hopefully can show some results on Monday.
Comments?
Thomas
BTW: I was wrong with slub. It also has bad side effects. What is strange is that with deep C-states one CPU (not the onlined) gets stuck (softlock up), but with processor.max_cstate=1 it does not. It reproducable hangs (loops?) in mm code, looks like it tries to alloc foreign memory. What I do not understand is why this happens with C3, but not with C1, it shouldn't have anything to do with memory setup. Anyway, I go back to slab again, lets ensure somehow that memory hot add comes first.
Quoted 9 lines
Blocking does not work.
But I have an idea, whatabout:
- CPU add rule which only adds a CPU if the corresponding Numa node already has onlined memory
- Memory add rule which adds the memory and also onlines CPUs if the Numa node still has offlined CPUs
Quoted 4 lines
This has the side effect that you online a CPU which may have been offlined on purpose if you hotadd memory on the same node..., a rather uncommon case.
That actually might be more of a problem than you think. It's not atypical that in the evening a system's components are offlined in order to save power. Maintenance also is scheduled for downtime so when memory is added to the system we may bring cpus into service erroneously. I think the "end-user" may not be too happy with this result.
[OTOH, a little documentation could fix that issue]
Quoted 7 lines
A quick try showed that it would be convenient to add the Numa node to the uevent which is a bit tricky... Ok, I got this working, but messed up the cpu hotplugging with my patches:
...
Unable to map lapic 32 to logical cpu number
...
This already worked...
Quoted 5 lines
I wanted to post something today, but I couldn't made it. Still I more or less could prove that above works and I hopefully can show some results on Monday.
If it works then I'm all for it :)
Quoted 6 lines
Comments?
Thomas
BTW: I was wrong with slub. It also has bad side effects. What is strange is
FWIW, I'm testing with slab.
P.
Quoted 7 lines · Prarit Bhargava
Quoted 7 lines
Blocking does not work.
But I have an idea, whatabout:
- CPU add rule which only adds a CPU if the corresponding Numa node
already has onlined memory
- Memory add rule which adds the memory and also onlines CPUs if the
Numa node still has offlined CPUs
Quoted 7 lines
Quoted 3 lines
This has the side effect that you online a CPU which may have been offlined on purpose if you hotadd memory on the same node..., a rather uncommon case.
That actually might be more of a problem than you think. It's not atypical that in the evening a system's components are offlined in order to save power.
This shouldn't be a problem currently, because offlined CPUs drain more power than online ones:
http://bugzilla.kernel.org/show_bug.cgi?id=5471
Quoted 1 line
Maintenance also is scheduled for downtime
I don't understand this part.
Quoted 3 lines
so when
memory is added to the system we may bring cpus into service erroneously. I think the "end-user" may not be too happy with this result.
...
Quoted 5 lines
Quoted 3 lines
I wanted to post something today, but I couldn't made it. Still I more or less could prove that above works and I hopefully can show some results on Monday.
If it works then I'm all for it :)
Great. I'll hopefully post some patches in a new thread soon and try to collect all interested people in CC.
Thomas
Quoted 33 lines · On Wednesday 17 March 2010 19:47:55 Prarit Bhargava wrote:
Quoted 19 lines · Thomas Renninger
Quoted 2 lines · On Thursday 11 March 2010 09:32:09 chen gong wrote:
On 2010-3-11 16:07, ykzhao wrote:
...
Quoted 3 lines
BTW, how about using UDEV rules to do this operation. It looks more smooth. I know some Novell guy is working on it.
I also know this guy :)
These would be the udev rules to automatically add memory/cpus:
SUBSYSTEM=="cpu", ACTION=="add", TEST=="online", ATTR{online}=="0", ATTR{online}="1", RUN+="/bin/logger onlining cpu: $env{DEVPATH}" SUBSYSTEM=="memory", ACTION=="add", TEST=="state", ATTR{state}=="offline", ATTR{state}="online", RUN+="/bin/logger onlining memory: $env{DEVPATH}"But this should be the same as you suggest (at least the memory rule) to do in the kernel:
automatically online the memory, once hotadded.Quoted 7 lines
I would not add any udev rules before this does not work reliably and currently it is totally broken, mainly because:
- not being able to alloc memory on foreign nodes (at least with slab)
- C-state, throttling and cpufreq set up is done without validcpu_data(new_cpu)resulting in wrong C-state (and other) infoThomas, forgive my ignorance of udev rules ... but can one udev rule block another? ... That's not the best way to explain it ... here's an example:
Quoted 5 lines
CPU hot add on Nehalem-EX. Memory is brought online. Memory udev events are generated. CPU udev events are generated.
Can the memory udev events block the cpu udev events? ie) can I be assured that memory will come online before the cpus?
I give up, there are too many open issues.
Onlining cpus and memory in parallel currently is not a good idea.
If the cpu event comes first, you run into other issues. I accidently had the acpi_memhotplug driver not loaded, in this case a new node was not created at all and the cpus ended up on the wrong node.
Some comments on your patch:
Your patch misses EXPORT_SYMBOL(_GPL?)(set_memory_state); (I wonder why drivers/base/memory.c does not export symbols under gpl...) and EXPORT_SYMBOL_GPL(online_pages);
You also need to set each memory block MEM_ONLINE somehow, while all memory was active, only one memory block showed "online" here:/sys/devices/system/node/node3/memory*/state
which has side-effects.
Would be great if you add me to CC if you repost or on related topics.
Thanks,
Thomas
Quoted 35 lines · Thomas Renninger
Quoted 33 lines · On Wednesday 17 March 2010 19:47:55 Prarit Bhargava wrote:
Quoted 31 lines · Thomas Renninger
Quoted 3 lines · On Thursday 11 March 2010 09:32:09 chen gong wrote:
On 2010-3-11 16:07, ykzhao wrote:
...
Quoted 4 lines
BTW, how about using UDEV rules to do this operation. It looks more smooth. I know some Novell guy is working on it.
I also know this guy :)
These would be the udev rules to automatically add memory/cpus:
SUBSYSTEM=="cpu", ACTION=="add", TEST=="online", ATTR{online}=="0", ATTR{online}="1", RUN+="/bin/logger onlining cpu: $env{DEVPATH}" SUBSYSTEM=="memory", ACTION=="add", TEST=="state", ATTR{state}=="offline", ATTR{state}="online", RUN+="/bin/logger onlining memory: $env{DEVPATH}"But this should be the same as you suggest (at least the memory rule) to do in the kernel:
automatically online the memory, once hotadded.I would not add any udev rules before this does not work reliably and currently it is totally broken, mainly because:
- not being able to alloc memory on foreign nodes (at least with slab)
- C-state, throttling and cpufreq set up is done without validcpu_data(new_cpu)resulting in wrong C-state (and other) info
Quoted 10 lines
Quoted 6 lines
Thomas, forgive my ignorance of udev rules ... but can one udev rule block another? ... That's not the best way to explain it ... here's an example:
CPU hot add on Nehalem-EX. Memory is brought online. Memory udev events are generated. CPU udev events are generated.
Can the memory udev events block the cpu udev events? ie) can I be assured that memory will come online before the cpus?
I give up, there are too many open issues.
Onlining cpus and memory in parallel currently is not a good idea.
Thomas,
Quoted 5 lines
If the cpu event comes first, you run into other issues. I accidently had the
acpi_memhotplugdriver not loaded, in this case a new node was not created at all and the cpus ended up on the wrong node.
Right :(
Quoted 6 lines
Some comments on your patch:
Your patch misses EXPORT_SYMBOL(_GPL?)(set_memory_state); (I wonder whydrivers/base/memory.cdoes not export symbols under gpl...) andEXPORT_SYMBOL_GPL(online_pages);
Okay, I'll fix that up.
Quoted 6 lines
You also need to set each memory block
MEM_ONLINEsomehow, while all memory was active, only one memory block showed "online" here:/sys/devices/system/node/node3/memory*/state
which has side-effects.
Huh -- I wonder if that has to do with they way I'm calling online_pages. I wonder if I have to make that clal "deeper" in the call path and call it on EACH segment that I'm bringing into service...
Quoted 3 lines
Would be great if you add me to CC if you repost or on related topics.
Absolutely no problem :) I didn't mean to leave you out of the conversation :)
P.
Quoted 4 lines
Thanks,
Thomas
Quoted 18 lines · ykzhao
Quoted 9 lines · Prarit Bhargava
Quoted 9 lines
Yes. The nehalem processor has the integrated memory controller. But it is not required that the hot-added memory should be onlined before bringing up CPU.
I do the following memory-hotplug test on one Machine. a. Before hot plugging memory, four CPUs socket are installed and
all the logical CPU are brought up. (Only one node has the memory)
b. The memory is hot-plugged and then the memory is onlined so that
it can be accessed by the system.Quoted 6 lines
Quoted 4 lines
In the above testing case the CPU is brought up before onlining the hot-added memory. And the test shows that it can work well.
That doesn't work when you have multiple nodes AFAICT. The cpus do not come into service because of a lack of memory on the node.... per node allocations will fail.
Quoted 9 lines
In the test the system has multiple nodes. The reason is that the cpu without memory can turn to other node and allocate the memory.
Quoted 2 lines
Just curious, exactly what did you test with? 2.6.33 (or newer)?
I test it on 2.6.32 kernel.
2.6.32 doesn't work with memory and cpu.
2.6.33 doesn't work with memory and cpu.
Other patches (which have been posted upstream) are required.
I'll try to dig up the patches and you can retest.
I will try a memory-free cpu add as well to see if that works... but the performance hit due to the off-node allocations probably doesn't make it worthwhile.
P.
Quoted 1 line
Quoted 26 lines · Prarit Bhargava
Quoted 24 lines · ykzhao
Quoted 9 lines · Prarit Bhargava
Quoted 9 lines
Yes. The nehalem processor has the integrated memory controller. But it is not required that the hot-added memory should be onlined before bringing up CPU.
I do the following memory-hotplug test on one Machine. a. Before hot plugging memory, four CPUs socket are installed and
all the logical CPU are brought up. (Only one node has the memory)
b. The memory is hot-plugged and then the memory is onlined so that
it can be accessed by the system.Quoted 8 lines
Quoted 4 lines
In the above testing case the CPU is brought up before onlining the hot-added memory. And the test shows that it can work well.
That doesn't work when you have multiple nodes AFAICT. The cpus do not come into service because of a lack of memory on the node.... per node allocations will fail.
In the test the system has multiple nodes. The reason is that the cpu without memory can turn to other node and allocate the memory.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Just curious, exactly what did you test with? 2.6.33 (or newer)?
I test it on 2.6.32 kernel.
2.6.32 doesn't work with memory and cpu.
2.6.33 doesn't work with memory and cpu.
I don't test the scenario of both cpu and memory hot-plugging.
What I mean is that the 2.6.32 kernel can work if only memory is hot-plugged (The cpu is already brought-up in the boot phase).
Quoted 14 lines
Other patches (which have been posted upstream) are required.
I'll try to dig up the patches and you can retest.
I will try a memory-free cpu add as well to see if that works... but the performance hit due to the off-node allocations probably doesn't make it worthwhile.
P.
Quoted 1 line
Quoted 34 lines · On Thursday 11 March 2010 01:55:15 ykzhao wrote:
Quoted 9 lines · Prarit Bhargava
Quoted 6 lines
Why do we need to see whether the memory is onlined before bringing cpu to online state? It seems that there is no dependency between cpu online and memory online.
Yakui,
Thanks for the explanation.
Quoted 3 lines
Here's a deeper look into the issue. New Intel processors have an on-die memory controller and this means that as the socket comes and goes, so does the memory "behind" the socket.
Yes. The nehalem processor has the integrated memory controller. But it is not required that the hot-added memory should be onlined before bringing up CPU.
I do the following memory-hotplug test on one Machine. a. Before hot plugging memory, four CPUs socket are installed and
all the logical CPU are brought up. (Only one node has the memory)
b. The memory is hot-plugged and then the memory is onlined so that
it can be accessed by the system.In the above testing case the CPU is brought up before onlining the hot-added memory. And the test shows that it can work well.
Quoted 5 lines
ie) with new processors it is possible that an entire node which consists of memory and cpus comes and goes with the socket enable and disable.
Quoted 7 lines
Quoted 3 lines
The cpu bringup code does local node allocations for the cpu. If the memory connected to the node (which is "behind" the socket) isn't online, then these allocations fail, and then the cpu bringup fails.
If the CPU can't allocate the memory from its own node, it can turn to other node and see whether the memory can be allocated. And this depends on the NUMA allocation policy.
Yes and this is broken and needs fixing.
Yakui, I expect you miss this patch and wrongly online the cpus to existing nodes, therefore you do not run into "out of memory" conditions:0271f91003d3703675be13b8865618359a6caa1f
I know for sure that slab is broken.
slub behaves different, but I am not sure whether this is due to wrong CPU hotadd code (processor_core.c is also broken and you get wrong C-state info from BIOS tables on hotadded CPUs)
Prarit: Can you retest with slub and processor.max_cstate=1, this could/should work.
AFAIK vmware injects memory in the same way into clients, so you may have different behavior of virtualized Linux clients.
This is a work around for current memory management not being able to allocate from foreign nodes. That should not mean that I generally vote against this to get added. If it works reliably why not add a work around until the more complicated stuff works.
One question: You also want to automatically add the CPUs, once a CPU hotplug event got fired, right?
The fact that the memory hotplug driver adds the memory immediately once notified, does not ensure that the HW/BIOS fires this event first. Theoretically you need a logic to not add CPUs to memoryless nodes, poll/wait until memory got added, etc.
Thomas
--
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 35 lines · Thomas Renninger
Quoted 33 lines · On Thursday 11 March 2010 01:55:15 ykzhao wrote:
Quoted 9 lines · Prarit Bhargava
Quoted 6 lines
Why do we need to see whether the memory is onlined before bringing cpu to online state? It seems that there is no dependency between cpu online and memory online.
Yakui,
Thanks for the explanation.
Quoted 2 lines
Here's a deeper look into the issue. New Intel processors have an on-die memory controller and this means that as the socket comes and goes, so does the memory "behind" the socket.
Yes. The nehalem processor has the integrated memory controller. But it is not required that the hot-added memory should be onlined before bringing up CPU.
I do the following memory-hotplug test on one Machine. a. Before hot plugging memory, four CPUs socket are installed and
all the logical CPU are brought up. (Only one node has the memory)
b. The memory is hot-plugged and then the memory is onlined so that
it can be accessed by the system.In the above testing case the CPU is brought up before onlining the hot-added memory. And the test shows that it can work well.
Quoted 4 lines
ie) with new processors it is possible that an entire node which consists of memory and cpus comes and goes with the socket enable and disable.
The cpu bringup code does local node allocations for the cpu. If the memory connected to the node (which is "behind" the socket) isn't online, then these allocations fail, and then the cpu bringup fails.
Quoted 9 lines
Quoted 4 lines
If the CPU can't allocate the memory from its own node, it can turn to other node and see whether the memory can be allocated. And this depends on the NUMA allocation policy.
Yes and this is broken and needs fixing.
Yakui, I expect you miss this patch and wrongly online the cpus to existing nodes, therefore you do not run into "out of memory" conditions:0271f91003d3703675be13b8865618359a6caa1f
FWIW, I'm working on a 2.6.32 based tree, but I have that patch in (as well as several others). I'm also running the latest upstream (tip as of this morning). The issues I see in my 2.6.32 based tree are the same AFAICT that I see upstream: a cpu comes online and attempts to make a per_node allocation which fails, so the cpu bringup fails.
Quoted 2 lines
I know for sure that slab is broken.
Yes, but I believe Andi Kleen has added some patches that resolve (at least some of) the issues. I've been using slab (and occasionally testing slub).
Quoted 8 lines
slub behaves different, but I am not sure whether this is due to wrong CPU hotadd code (
processor_core.cis also broken and you get wrong C-state info from BIOS tables on hotadded CPUs)Prarit: Can you retest with slub and processor.max_cstate=1, this could/should work.
Two tests:
1. WITHOUT my auto online patch, the cpus fail to come into service because of a per_node allocation failure.
2. WITH my auto online patch, the cpus come into service
... I have NOT done any sort of testing to see if the cpus are really live ;)
Quoted 4 lines
AFAIK vmware injects memory in the same way into clients, so you may have different behavior of virtualized Linux clients.
Are you referring the vmware ballooning driver (or whatever they call it). IIRC (and I'm not saying I do ;) ), vmware adds memory and automatically onlines it in a guest. I'm not sure how that's done -- it could be via udev.
I'll see if anyone here knows.
Quoted 3 lines
One question: You also want to automatically add the CPUs, once a CPU hotplug event got fired, right?
Yes, That's correct.
Quoted 3 lines
The fact that the memory hotplug driver adds the memory immediately once notified, does not ensure that the HW/BIOS fires this event first.
Is that right? FWIW I always see this sequence of events:
ACPI memory added
ACPI cpus added
I never see them out-of-order. OTOH, I'm only testing on Intel's latest platform so maybe there are some older systems that don't do this in that order.
Quoted 3 lines
Theoretically you need a logic to not add CPUs to memoryless nodes, poll/wait until memory got added, etc.
Theoretically yes -- but are there any systems that generate cpu add events before memory add events?
Thanks for the input Thomas :)
P.
Quoted 2 lines
Thomas
--
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
I think if you're going to poke into drivers/base/memory.c we should let the -mm guys know that you're creating a new interface.
The ACPI part looks fine to me.
cc added.
/ac
* Prarit Bhargava <[email protected]>:
Quoted 104 lines
New sockets have on-die memory controllers. This means that in certain HW configurations the memory behind the socket comes and goes as the socket is physically enabled and disabled.
Since the cpu bringup code does on node memory allocations, the memory on the added socket must be onlined first.
Add a .config option to automatically online hot added memory, and enable it in the acpi memory add path.
Signed-off-by Prarit Bhargava <[email protected]>drivers/acpi/Kconfig +8
-0@@ -350,6 +350,14 @@ config ACPI_HOTPLUG_MEMORY To compile this driver as a module, choose M here: the module will be called acpi_memhotplug. +config ACPI_HOTPLUG_MEMORY_AUTO_ONLINE + bool "Automatically online hotplugged memory" + depends on ACPI_HOTPLUG_MEMORY + default n + help + This forces memory that is brought into service by ACPI + to be automatically onlined. + config ACPI_SBS tristate "Smart Battery System" depends on X86drivers/acpi/acpi_memhotplug.c +14
-0@@ -30,6 +30,7 @@#include <linux/init.h> #include <linux/types.h> #include <linux/memory_hotplug.h> +#include <linux/memory.h> #include <acpi/acpi_drivers.h> #define ACPI_MEMORY_DEVICE_CLASS "memory"@@ -252,6 +253,19 @@ static int acpi_memory_enable_device(struct acpi_memory_device *mem_device) result =add_memory (node, info->start_addr, info->length);if (result)continue ; +#ifdef CONFIG_ACPI_HOTPLUG_MEMORY_AUTO_ONLINE +/* +* New processors require memory to be online before cpus. +* No notifications are required here as "we" are the only +* ones who know about the new memory right now. +*/ + result =online_pages (info->start_addr >>PAGE_SHIFT , + info->length >>PAGE_SHIFT ); +if (!result) +set_memory_state (info->start_addr,MEM_ONLINE ); +else +printk ("Memory online failed.\n" ); +#endif info->enabled =1 ; num_enabled++; }drivers/base/memory.c +14
-0@@ -510,6 +510,20 @@ int remove_memory_block(unsigned long node_id, struct mem_section *section, }/* +* need an interface for the VM to mark sections on and offline when hot-adding +* memory. +*/ +void set_memory_state (unsigned long addr,unsigned long state) +{ +struct mem_section *section; +struct memory_block *mem; + + section =__pfn_to_section (addr >>PAGE_SHIFT ); + mem =find_memory_block (section); + mem->state = state; +} + +/* * need an interface for the VM to add new memory regions, * but without onlining it. */ include/linux/memory.h +1
-1@@ -112,6 +112,7 @@ extern int remove_memory_block(unsigned long, struct mem_section *, int);extern int memory_notify (unsigned long val,void *v);extern int memory_isolate_notify (unsigned long val,void *v);extern struct memory_block *find_memory_block (struct mem_section *); +extern void set_memory_state (unsigned long ,unsigned long );#define CONFIG_MEM_BLOCK_SIZE (PAGES_PER_SECTION<<PAGE_SHIFT)enum mem_add_context {BOOT ,HOTPLUG };#endif /* CONFIG_MEMORY_HOTPLUG_SPARSE */ --To unsubscribe from this list: send the line "unsubscribe linux-acpi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 4 lines · Alex Chiang
I think if you're going to poke into
drivers/base/memory.cwe should let the -mm guys know that you're creating a new interface.
Absolutely -- I was just RFC'ing for now ;)
Quoted 5 lines
The ACPI part looks fine to me.
cc added.
Thanks :)
P.
Quoted 109 lines
/ac
* Prarit Bhargava <[email protected]>:
Quoted 105 lines
New sockets have on-die memory controllers. This means that in certain HW configurations the memory behind the socket comes and goes as the socket is physically enabled and disabled.
Since the cpu bringup code does on node memory allocations, the memory on the added socket must be onlined first.
Add a .config option to automatically online hot added memory, and enable it in the acpi memory add path.
Signed-off-by Prarit Bhargava <[email protected]>drivers/acpi/Kconfig +8
-0@@ -350,6 +350,14 @@ config ACPI_HOTPLUG_MEMORY To compile this driver as a module, choose M here: the module will be called acpi_memhotplug. +config ACPI_HOTPLUG_MEMORY_AUTO_ONLINE + bool "Automatically online hotplugged memory" + depends on ACPI_HOTPLUG_MEMORY + default n + help + This forces memory that is brought into service by ACPI + to be automatically onlined. + config ACPI_SBS tristate "Smart Battery System" depends on X86drivers/acpi/acpi_memhotplug.c +14
-0@@ -30,6 +30,7 @@#include <linux/init.h> #include <linux/types.h> #include <linux/memory_hotplug.h> +#include <linux/memory.h> #include <acpi/acpi_drivers.h> #define ACPI_MEMORY_DEVICE_CLASS "memory"@@ -252,6 +253,19 @@ static int acpi_memory_enable_device(struct acpi_memory_device *mem_device) result =add_memory (node, info->start_addr, info->length);if (result)continue ; +#ifdef CONFIG_ACPI_HOTPLUG_MEMORY_AUTO_ONLINE +/* +* New processors require memory to be online before cpus. +* No notifications are required here as "we" are the only +* ones who know about the new memory right now. +*/ + result =online_pages (info->start_addr >>PAGE_SHIFT , + info->length >>PAGE_SHIFT ); +if (!result) +set_memory_state (info->start_addr,MEM_ONLINE ); +else +printk ("Memory online failed.\n" ); +#endif info->enabled =1 ; num_enabled++; }drivers/base/memory.c +14
-0@@ -510,6 +510,20 @@ int remove_memory_block(unsigned long node_id, struct mem_section *section, }/* +* need an interface for the VM to mark sections on and offline when hot-adding +* memory. +*/ +void set_memory_state (unsigned long addr,unsigned long state) +{ +struct mem_section *section; +struct memory_block *mem; + + section =__pfn_to_section (addr >>PAGE_SHIFT ); + mem =find_memory_block (section); + mem->state = state; +} + +/* * need an interface for the VM to add new memory regions, * but without onlining it. */ include/linux/memory.h +1
-1@@ -112,6 +112,7 @@ extern int remove_memory_block(unsigned long, struct mem_section *, int);extern int memory_notify (unsigned long val,void *v);extern int memory_isolate_notify (unsigned long val,void *v);extern struct memory_block *find_memory_block (struct mem_section *); +extern void set_memory_state (unsigned long ,unsigned long );#define CONFIG_MEM_BLOCK_SIZE (PAGES_PER_SECTION<<PAGE_SHIFT)enum mem_add_context {BOOT ,HOTPLUG };#endif /* CONFIG_MEMORY_HOTPLUG_SPARSE */ --To unsubscribe from this list: send the line "unsubscribe linux-acpi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html