Skip to contentSkip to conversation

: ACPI: Automatically online hot-added memory

RFC PATCH In review Prarit Bhargava 6 participants 23 messages Mar 9, 2010 – Mar 24, 2010

Subsystems (5)
lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 : ACPI: Automatically online hot-added memory

New sockets have on-die memory controllers. This means that in certain HW configurations the memory behind the socket comes and goes as the socket is physically enabled and disabled.

Since the cpu bringup code does on node memory allocations, the memory on the added socket must be onlined first.

Add a .config option to automatically online hot added memory, and enable it in the acpi memory add path.

drivers/acpi/Kconfig +8 -0
index 93d2c79..dece6bd 100644
@@ -350,6 +350,14 @@ config ACPI_HOTPLUG_MEMORY
 	  To compile this driver as a module, choose M here:
 	  the module will be called acpi_memhotplug.
 
+config ACPI_HOTPLUG_MEMORY_AUTO_ONLINE
+	bool "Automatically online hotplugged memory"
+	depends on ACPI_HOTPLUG_MEMORY
+	default n
+	help
+	  This forces memory that is brought into service by ACPI
+	  to be automatically onlined.
+
 config ACPI_SBS
 	tristate "Smart Battery System"
 	depends on X86
drivers/acpi/acpi_memhotplug.c +14 -0
index 3597d73..9814c50 100644
@@ -30,6 +30,7 @@
 #include <linux/init.h>
 #include <linux/types.h>
 #include <linux/memory_hotplug.h>
+#include <linux/memory.h>
 #include <acpi/acpi_drivers.h>
 
 #define ACPI_MEMORY_DEVICE_CLASS		"memory"
@@ -252,6 +253,19 @@ static int acpi_memory_enable_device(struct acpi_memory_device *mem_device)
 		result = add_memory(node, info->start_addr, info->length);
 		if (result)
 			continue;
+#ifdef CONFIG_ACPI_HOTPLUG_MEMORY_AUTO_ONLINE
+		/*
+		 * New processors require memory to be online before cpus.
+		 * No notifications are required here as "we" are the only
+		 * ones who know about the new memory right now.
+		 */
+		result = online_pages(info->start_addr >> PAGE_SHIFT,
+				      info->length >> PAGE_SHIFT);
+		if (!result)
+			set_memory_state(info->start_addr, MEM_ONLINE);
+		else
+			printk("Memory online failed.\n");
+#endif
 		info->enabled = 1;
 		num_enabled++;
 	}
drivers/base/memory.c +14 -0
index 2f86915..fb465d5 100644
@@ -510,6 +510,20 @@ int remove_memory_block(unsigned long node_id, struct mem_section *section,
 }
 
 /*
+ * need an interface for the VM to mark sections on and offline when hot-adding
+ * memory.
+ */
+void set_memory_state(unsigned long addr, unsigned long state)
+{
+	struct mem_section *section;
+	struct memory_block *mem;
+
+	section = __pfn_to_section(addr >> PAGE_SHIFT);
+	mem = find_memory_block(section);
+	mem->state = state;
+}
+
+/*
  * need an interface for the VM to add new memory regions,
  * but without onlining it.
  */
include/linux/memory.h +1 -0
index 1adfe77..5d8d78c 100644
@@ -112,6 +112,7 @@ extern int remove_memory_block(unsigned long, struct mem_section *, int);
 extern int memory_notify(unsigned long val, void *v);
 extern int memory_isolate_notify(unsigned long val, void *v);
 extern struct memory_block *find_memory_block(struct mem_section *);
+extern void set_memory_state(unsigned long, unsigned long);
 #define CONFIG_MEM_BLOCK_SIZE	(PAGES_PER_SECTION<<PAGE_SHIFT)
 enum mem_add_context { BOOT, HOTPLUG };
 #endif /* CONFIG_MEMORY_HOTPLUG_SPARSE */
#1 lore raw
Quoted 9 lines · Prarit Bhargava

New sockets have on-die memory controllers. This means that in certain HW configurations the memory behind the socket comes and goes as the socket is physically enabled and disabled.

Since the cpu bringup code does on node memory allocations, the memory on the added socket must be onlined first.

Add a .config option to automatically online hot added memory, and enable it in the acpi memory add path.

This seems like the right thing to do.

drivers/acpi/Kconfig config ACPI_HOTPLUG_MEMORY · line 356

Quoted 4 lines
drivers/acpi/Kconfig +4 -0
+config ACPI_HOTPLUG_MEMORY_AUTO_ONLINE
+	bool "Automatically online hotplugged memory"
+	depends on ACPI_HOTPLUG_MEMORY
+	default n

default !S390? default x86?

drivers/acpi/acpi_memhotplug.c static int acpi_memory_enable_device(struct acpi_memory_device *mem_device) · line 267

Quoted 6 lines
drivers/acpi/acpi_memhotplug.c +6 -0
+		result = online_pages(info->start_addr >> PAGE_SHIFT,
+				      info->length >> PAGE_SHIFT);
+		if (!result)
+			set_memory_state(info->start_addr, MEM_ONLINE);
+		else
+			printk("Memory online failed.\n");

That probably wants to be more descriptive and have a loglevel. What happens if this fails? The CPU presumably can't be brought up?

Signature · 1 line
Matthew Garrett | [email protected]
#2 lore raw
Quoted 16 lines · Matthew Garrett
Quoted 10 lines · Prarit Bhargava

New sockets have on-die memory controllers. This means that in certain HW configurations the memory behind the socket comes and goes as the socket is physically enabled and disabled.

Since the cpu bringup code does on node memory allocations, the memory on the added socket must be onlined first.

Add a .config option to automatically online hot added memory, and enable it in the acpi memory add path.

This seems like the right thing to do.

Quoted 8 lines

drivers/acpi/Kconfig config ACPI_HOTPLUG_MEMORY · line 356

Quoted 5 lines
drivers/acpi/Kconfig +4 -0
+config ACPI_HOTPLUG_MEMORY_AUTO_ONLINE
+	bool "Automatically online hotplugged memory"
+	depends on ACPI_HOTPLUG_MEMORY
+	default n

default !S390? default x86?

I've been inquiring about this. It seems that some vendors want the capability to manually bring memory online. I'm trying to get more details as this doesn't seem intuitive. It seems to me that if you add memory you want to online it right now.

Quoted 10 lines

drivers/acpi/acpi_memhotplug.c static int acpi_memory_enable_device(struct acpi_memory_device *mem_device) · line 267

Quoted 8 lines
drivers/acpi/acpi_memhotplug.c +6 -0
+		result = online_pages(info->start_addr >> PAGE_SHIFT,
+				      info->length >> PAGE_SHIFT);
+		if (!result)
+			set_memory_state(info->start_addr, MEM_ONLINE);
+		else
+			printk("Memory online failed.\n");

That probably wants to be more descriptive and have a loglevel.

Oops. I'm very bad about setting the loglevel's on my printk's ;) ... I'll fix that in the next patch.

Quoted 2 lines

What happens if this fails? The CPU presumably can't be brought up?

That's a good question... After thinking about it I think that acpi_processor_add() must check the node to see if it has memory before bringing the cpu online.

I'll code that up, test, and repost.

P.

#3 lore raw
Quoted 43 lines · Prarit Bhargava
Quoted 16 lines · Matthew Garrett
Quoted 10 lines · Prarit Bhargava

New sockets have on-die memory controllers. This means that in certain HW configurations the memory behind the socket comes and goes as the socket is physically enabled and disabled.

Since the cpu bringup code does on node memory allocations, the memory on the added socket must be onlined first.

Add a .config option to automatically online hot added memory, and enable it in the acpi memory add path.

This seems like the right thing to do.

Quoted 8 lines

drivers/acpi/Kconfig config ACPI_HOTPLUG_MEMORY · line 356

Quoted 5 lines
drivers/acpi/Kconfig +4 -0
+config ACPI_HOTPLUG_MEMORY_AUTO_ONLINE
+	bool "Automatically online hotplugged memory"
+	depends on ACPI_HOTPLUG_MEMORY
+	default n

default !S390? default x86?

I've been inquiring about this. It seems that some vendors want the capability to manually bring memory online. I'm trying to get more details as this doesn't seem intuitive. It seems to me that if you add memory you want to online it right now.

Quoted 10 lines

drivers/acpi/acpi_memhotplug.c static int acpi_memory_enable_device(struct acpi_memory_device *mem_device) · line 267

Quoted 8 lines
drivers/acpi/acpi_memhotplug.c +6 -0
+		result = online_pages(info->start_addr >> PAGE_SHIFT,
+				      info->length >> PAGE_SHIFT);
+		if (!result)
+			set_memory_state(info->start_addr, MEM_ONLINE);
+		else
+			printk("Memory online failed.\n");

That probably wants to be more descriptive and have a loglevel.

Quoted 2 lines

Oops. I'm very bad about setting the loglevel's on my printk's ;) ... I'll fix that in the next patch.

One note is that the hot-added memory will be automatically onlined. And we had better add an interface so that the user-space can quiry whether the hot-added memory is automatically onlined. Otherwise the user-space will receive the corresponding event and online hot-added memory again. In such case it will get some error messages.

Quoted 8 lines
Quoted 4 lines

What
happens if this fails? The CPU presumably can't be brought up?

That's a good question... After thinking about it I think that acpi_processor_add() must check the node to see if it has memory before bringing the cpu online.

Why do we need to see whether the memory is onlined before bringing cpu to online state? It seems that there is no dependency between cpu online and memory online.

Thanks.

Quoted 9 lines

I'll code that up, test, and repost.

P.

--
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#4 lore raw
Quoted 7 lines

Why do we need to see whether the memory is onlined before bringing cpu to online state? It seems that there is no dependency between cpu online and memory online.

Yakui,

Here's a deeper look into the issue. New Intel processors have an on-die memory controller and this means that as the socket comes and goes, so does the memory "behind" the socket.

ie) with new processors it is possible that an entire node which consists of memory and cpus comes and goes with the socket enable and disable.

The cpu bringup code does local node allocations for the cpu. If the memory connected to the node (which is "behind" the socket) isn't online, then these allocations fail, and then the cpu bringup fails.

P.

#5 lore raw
Quoted 9 lines · Prarit Bhargava
Quoted 6 lines

Why do we need to see whether the memory is onlined before bringing cpu to online state? It seems that there is no dependency between cpu online and memory online.

Yakui,

Thanks for the explanation.

Quoted 3 lines

Here's a deeper look into the issue. New Intel processors have an on-die memory controller and this means that as the socket comes and goes, so does the memory "behind" the socket.

Yes. The nehalem processor has the integrated memory controller. But it is not required that the hot-added memory should be onlined before bringing up CPU.
I do the following memory-hotplug test on one Machine. a. Before hot plugging memory, four CPUs socket are installed and
all the logical CPU are brought up. (Only one node has the memory)
b. The memory is hot-plugged and then the memory is onlined so that
it can be accessed by the system.

In the above testing case the CPU is brought up before onlining the hot-added memory. And the test shows that it can work well.

Quoted 8 lines

ie) with new processors it is possible that an entire node which consists of memory and cpus comes and goes with the socket enable and disable.

The cpu bringup code does local node allocations for the cpu. If the memory connected to the node (which is "behind" the socket) isn't online, then these allocations fail, and then the cpu bringup fails.

If the CPU can't allocate the memory from its own node, it can turn to other node and see whether the memory can be allocated. And this depends on the NUMA allocation policy.

Quoted 3 lines

P.

--
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#6 lore raw
Quoted 11 lines

Yes. The nehalem processor has the integrated memory controller. But it is not required that the hot-added memory should be onlined before bringing up CPU.
I do the following memory-hotplug test on one Machine. a. Before hot plugging memory, four CPUs socket are installed and
all the logical CPU are brought up. (Only one node has the memory)
b. The memory is hot-plugged and then the memory is onlined so that
it can be accessed by the system.

Quoted 3 lines

In the above testing case the CPU is brought up before onlining the hot-added memory. And the test shows that it can work well.

That doesn't work when you have multiple nodes AFAICT. The cpus do not come into service because of a lack of memory on the node.... per node allocations will fail.

Just curious, exactly what did you test with? 2.6.33 (or newer)?

Quoted 10 lines
Quoted 5 lines

ie) with new processors it is possible that an entire node which consists of memory and cpus comes and goes with the socket enable and disable.

The cpu bringup code does local node allocations for the cpu. If the memory connected to the node (which is "behind" the socket) isn't online, then these allocations fail, and then the cpu bringup fails.

If the CPU can't allocate the memory from its own node, it can turn to other node and see whether the memory can be allocated. And this depends on the NUMA allocation policy.

Maybe that could work, but I haven't gotten that to work. Even if it does work, it's a HUGE performance hit :(. I can't imagine incurring an extra hop just to get to per_cpu memory. I'd rather bring the memory on the local node up first.

I think it is much better to bring the memory up first.

Either way, it's a nice feature to have.

P.

Quoted 7 lines
Quoted 5 lines

P.

#7 lore raw
Quoted 10 lines · Prarit Bhargava
Quoted 10 lines

Yes. The nehalem processor has the integrated memory controller. But it is not required that the hot-added memory should be onlined before bringing up CPU.
I do the following memory-hotplug test on one Machine. a. Before hot plugging memory, four CPUs socket are installed and
all the logical CPU are brought up. (Only one node has the memory)
b. The memory is hot-plugged and then the memory is onlined so that
it can be accessed by the system.

Quoted 7 lines
Quoted 3 lines

In the above testing case the CPU is brought up before onlining the hot-added memory. And the test shows that it can work well.

That doesn't work when you have multiple nodes AFAICT. The cpus do not come into service because of a lack of memory on the node.... per node allocations will fail.

In the test the system has multiple nodes. The reason is that the cpu without memory can turn to other node and allocate the memory.

Quoted 2 lines

Just curious, exactly what did you test with? 2.6.33 (or newer)?

I test it on 2.6.32 kernel.

Quoted 34 lines
Quoted 6 lines
Quoted 6 lines

ie) with new processors it is possible that an entire node which consists of memory and cpus comes and goes with the socket enable and disable.

Quoted 9 lines
Quoted 4 lines

The cpu bringup code does local node allocations for the cpu. If the memory connected to the node (which is "behind" the socket) isn't online, then these allocations fail, and then the cpu bringup fails.

If the CPU can't allocate the memory from its own node, it can turn to other node and see whether the memory can be allocated. And this depends on the NUMA allocation policy.

Maybe that could work, but I haven't gotten that to work. Even if it does work, it's a HUGE performance hit :(. I can't imagine incurring an extra hop just to get to per_cpu memory. I'd rather bring the memory on the local node up first.

I think it is much better to bring the memory up first.

Either way, it's a nice feature to have.

P.

Quoted 7 lines
Quoted 5 lines

P.

#8 lore raw
Quoted 15 lines · ykzhao
Quoted 14 lines · Prarit Bhargava
Quoted 13 lines

Yes. The nehalem processor has the integrated memory controller. But it is not required that the hot-added memory should be onlined before bringing up CPU.
I do the following memory-hotplug test on one Machine. a. Before hot plugging memory, four CPUs socket are installed and
all the logical CPU are brought up. (Only one node has the memory)
b. The memory is hot-plugged and then the memory is onlined so that
it can be accessed by the system.

In the above testing case the CPU is brought up before onlining the hot-added memory. And the test shows that it can work well.

Quoted 6 lines
Quoted 3 lines

That doesn't work when you have multiple nodes AFAICT. The cpus do not come into service because of a lack of memory on the node.... per node allocations will fail.

In the test the system has multiple nodes. The reason is that the cpu without memory can turn to other node and allocate the memory.

I agree with Yakui. The memory and CPU are irrelevant in some way. CPU can get memory from other nodes if it hasn't local memory, though for now it has some issues. Andi is working on it now (http://lkml.org/lkml/2010/2/3/343)

BTW, how about using UDEV rules to do this operation. It looks more smooth. I know some Novell guy is working on it.

Quoted 30 lines
Quoted 3 lines

Just curious, exactly what did you test with? 2.6.33 (or newer)?

I test it on 2.6.32 kernel.

Quoted 24 lines
Quoted 6 lines
Quoted 6 lines

ie) with new processors it is possible that an entire node which consists of memory and cpus comes and goes with the socket enable and disable.

Quoted 9 lines
Quoted 4 lines

The cpu bringup code does local node allocations for the cpu. If the memory connected to the node (which is "behind" the socket) isn't online, then these allocations fail, and then the cpu bringup fails.

If the CPU can't allocate the memory from its own node, it can turn to other node and see whether the memory can be allocated. And this depends on the NUMA allocation policy.

Maybe that could work, but I haven't gotten that to work. Even if it does work, it's a HUGE performance hit :(. I can't imagine incurring an extra hop just to get to per_cpu memory. I'd rather bring the memory on the local node up first.

I think it is much better to bring the memory up first.

Either way, it's a nice feature to have.

#9 lore raw
Quoted 19 lines · chen gong
Quoted 17 lines · ykzhao
Quoted 15 lines · Prarit Bhargava
Quoted 9 lines

Yes. The nehalem processor has the integrated memory controller. But it is not required that the hot-added memory should be onlined before bringing up CPU.
I do the following memory-hotplug test on one Machine. a. Before hot plugging memory, four CPUs socket are installed and
all the logical CPU are brought up. (Only one node has the memory)
b. The memory is hot-plugged and then the memory is onlined so that
it can be accessed by the system.

Quoted 4 lines

In the above testing case the CPU is brought up before onlining the hot-added memory. And the test shows that it can work well.

That doesn't work when you have multiple nodes AFAICT. The cpus do not come into service because of a lack of memory on the node.... per node allocations will fail.

Quoted 8 lines
Quoted 3 lines

In the test the system has multiple nodes. The reason is that the cpu without memory can turn to other node and allocate the memory.

I agree with Yakui. The memory and CPU are irrelevant in some way. CPU can get memory from other nodes if it hasn't local memory, though for now it has some issues. Andi is working on it now (http://lkml.org/lkml/2010/2/3/343)

Right -- I'm working with Andi on this to get RHEL working. So I have his slab patches, etc.. That's one of the reasons it isn't working right now.

Quoted 4 lines

BTW, how about using UDEV rules to do this operation. It looks more smooth. I know some Novell guy is working on it.

I was thinking about that, but the CPU comes online automatically, so why not memory? It seems to be a discrepancy between the two. And ... the two of you still aren't considering the obvious performance hit which I would think makes this a must have.

P.

Quoted 1 line
#10 lore raw
In reply to chen gong
Quoted 1 line · On Thursday 11 March 2010 09:32:09 chen gong wrote:

On 2010-3-11 16:07, ykzhao wrote:

...

Quoted 2 lines

BTW, how about using UDEV rules to do this operation. It looks more smooth. I know some Novell guy is working on it.

I also know this guy :)

These would be the udev rules to automatically add memory/cpus:
SUBSYSTEM=="cpu", ACTION=="add", TEST=="online", ATTR{online}=="0", ATTR{online}="1", RUN+="/bin/logger onlining cpu: $env{DEVPATH}" SUBSYSTEM=="memory", ACTION=="add", TEST=="state", ATTR{state}=="offline", ATTR{state}="online", RUN+="/bin/logger onlining memory: $env{DEVPATH}"

But this should be the same as you suggest (at least the memory rule) to do in the kernel:
automatically online the memory, once hotadded.

I would not add any udev rules before this does not work reliably and currently it is totally broken, mainly because:
- not being able to alloc memory on foreign nodes (at least with slab)
- C-state, throttling and cpufreq set up is done without valid cpu_data(new_cpu) resulting in wrong C-state (and other) info

Thomas

#11 lore raw
Quoted 19 lines · Thomas Renninger
Quoted 2 lines · On Thursday 11 March 2010 09:32:09 chen gong wrote:

On 2010-3-11 16:07, ykzhao wrote:

...

Quoted 3 lines

BTW, how about using UDEV rules to do this operation. It looks more smooth. I know some Novell guy is working on it.

I also know this guy :)

These would be the udev rules to automatically add memory/cpus:
SUBSYSTEM=="cpu", ACTION=="add", TEST=="online", ATTR{online}=="0", ATTR{online}="1", RUN+="/bin/logger onlining cpu: $env{DEVPATH}" SUBSYSTEM=="memory", ACTION=="add", TEST=="state", ATTR{state}=="offline", ATTR{state}="online", RUN+="/bin/logger onlining memory: $env{DEVPATH}"

But this should be the same as you suggest (at least the memory rule) to do in the kernel:
automatically online the memory, once hotadded.

Quoted 7 lines

I would not add any udev rules before this does not work reliably and currently it is totally broken, mainly because:
- not being able to alloc memory on foreign nodes (at least with slab)
- C-state, throttling and cpufreq set up is done without valid cpu_data(new_cpu) resulting in wrong C-state (and other) info

Thomas, forgive my ignorance of udev rules ... but can one udev rule block another? ... That's not the best way to explain it ... here's an example:

CPU hot add on Nehalem-EX. Memory is brought online. Memory udev events are generated. CPU udev events are generated.

Can the memory udev events block the cpu udev events? ie) can I be assured that memory will come online before the cpus?

P.

Quoted 7 lines

Thomas
--
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#12 lore raw
Quoted 32 lines · Prarit Bhargava
Quoted 21 lines · Thomas Renninger
Quoted 1 line · On Thursday 11 March 2010 09:32:09 chen gong wrote:

On 2010-3-11 16:07, ykzhao wrote:

...

Quoted 2 lines

BTW, how about using UDEV rules to do this operation. It looks more smooth. I know some Novell guy is working on it.

I also know this guy :)

These would be the udev rules to automatically add memory/cpus:
SUBSYSTEM=="cpu", ACTION=="add", TEST=="online", ATTR{online}=="0", ATTR{online}="1", RUN+="/bin/logger onlining cpu: $env{DEVPATH}" SUBSYSTEM=="memory", ACTION=="add", TEST=="state", ATTR{state}=="offline", ATTR{state}="online", RUN+="/bin/logger onlining
memory: $env{DEVPATH}"

But this should be the same as you suggest (at least the memory rule) to do in the kernel:
automatically online the memory, once hotadded.

Quoted 5 lines

I would not add any udev rules before this does not work reliably and currently it is totally broken, mainly because:
- not being able to alloc memory on foreign nodes (at least with slab)
- C-state, throttling and cpufreq set up is done without valid cpu_data(new_cpu) resulting in wrong C-state (and other) info

Thomas, forgive my ignorance of udev rules ... but can one udev rule block another? ... That's not the best way to explain it ... here's an example:

Quoted 5 lines

CPU hot add on Nehalem-EX. Memory is brought online. Memory udev events are generated. CPU udev events are generated.

Can the memory udev events block the cpu udev events? ie) can I be assured that memory will come online before the cpus?

Blocking does not work.
But I have an idea, whatabout:
- CPU add rule which only adds a CPU if the corresponding Numa node already has onlined memory
- Memory add rule which adds the memory and also onlines CPUs if the Numa node still has offlined CPUs

This has the side effect that you online a CPU which may have been offlined on purpose if you hotadd memory on the same node..., a rather uncommon case.

A quick try showed that it would be convenient to add the Numa node to the uevent which is a bit tricky...
Ok, I got this working, but messed up the cpu hotplugging with my patches:
...
Unable to map lapic 32 to logical cpu number
...
This already worked...

I wanted to post something today, but I couldn't made it. Still I more or less could prove that above works and I hopefully can show some results on Monday.

Comments?

Thomas

BTW: I was wrong with slub. It also has bad side effects. What is strange is that with deep C-states one CPU (not the onlined) gets stuck (softlock up), but with processor.max_cstate=1 it does not. It reproducable hangs (loops?) in mm code, looks like it tries to alloc foreign memory. What I do not understand is why this happens with C3, but not with C1, it shouldn't have anything to do with memory setup. Anyway, I go back to slab again, lets ensure somehow that memory hot add comes first.

#13 lore raw
Quoted 9 lines

Blocking does not work.
But I have an idea, whatabout:
- CPU add rule which only adds a CPU if the corresponding Numa node already has onlined memory
- Memory add rule which adds the memory and also onlines CPUs if the Numa node still has offlined CPUs

Quoted 4 lines

This has the side effect that you online a CPU which may have been offlined on purpose if you hotadd memory on the same node..., a rather uncommon case.

That actually might be more of a problem than you think. It's not atypical that in the evening a system's components are offlined in order to save power. Maintenance also is scheduled for downtime so when memory is added to the system we may bring cpus into service erroneously. I think the "end-user" may not be too happy with this result.

[OTOH, a little documentation could fix that issue]

Quoted 7 lines

A quick try showed that it would be convenient to add the Numa node to the uevent which is a bit tricky... Ok, I got this working, but messed up the cpu hotplugging with my patches:
...
Unable to map lapic 32 to logical cpu number
...
This already worked...

Quoted 5 lines

I wanted to post something today, but I couldn't made it. Still I more or less could prove that above works and I hopefully can show some results on Monday.

If it works then I'm all for it :)

Quoted 6 lines

Comments?

Thomas

BTW: I was wrong with slub. It also has bad side effects. What is strange is

FWIW, I'm testing with slab.

P.

#14 lore raw
Quoted 7 lines · Prarit Bhargava
Quoted 7 lines

Blocking does not work.
But I have an idea, whatabout:
- CPU add rule which only adds a CPU if the corresponding Numa node
already has onlined memory
- Memory add rule which adds the memory and also onlines CPUs if the
Numa node still has offlined CPUs

Quoted 7 lines
Quoted 3 lines

This has the side effect that you online a CPU which may have been offlined on purpose if you hotadd memory on the same node..., a rather uncommon case.

That actually might be more of a problem than you think. It's not atypical that in the evening a system's components are offlined in order to save power.

This shouldn't be a problem currently, because offlined CPUs drain more power than online ones:
http://bugzilla.kernel.org/show_bug.cgi?id=5471

Quoted 1 line

Maintenance also is scheduled for downtime

I don't understand this part.

Quoted 3 lines

so when
memory is added to the system we may bring cpus into service erroneously. I think the "end-user" may not be too happy with this result.

...

Quoted 5 lines
Quoted 3 lines

I wanted to post something today, but I couldn't made it. Still I more or less could prove that above works and I hopefully can show some results on Monday.

If it works then I'm all for it :)

Great. I'll hopefully post some patches in a new thread soon and try to collect all interested people in CC.

Thomas

#15 lore raw
In reply to Prarit Bhargava
Quoted 33 lines · On Wednesday 17 March 2010 19:47:55 Prarit Bhargava wrote:
Quoted 19 lines · Thomas Renninger
Quoted 2 lines · On Thursday 11 March 2010 09:32:09 chen gong wrote:

On 2010-3-11 16:07, ykzhao wrote:

...

Quoted 3 lines

BTW, how about using UDEV rules to do this operation. It looks more smooth. I know some Novell guy is working on it.

I also know this guy :)

These would be the udev rules to automatically add memory/cpus:
SUBSYSTEM=="cpu", ACTION=="add", TEST=="online", ATTR{online}=="0", ATTR{online}="1", RUN+="/bin/logger onlining cpu: $env{DEVPATH}" SUBSYSTEM=="memory", ACTION=="add", TEST=="state", ATTR{state}=="offline", ATTR{state}="online", RUN+="/bin/logger onlining memory: $env{DEVPATH}"

But this should be the same as you suggest (at least the memory rule) to do in the kernel:
automatically online the memory, once hotadded.

Quoted 7 lines

I would not add any udev rules before this does not work reliably and currently it is totally broken, mainly because:
- not being able to alloc memory on foreign nodes (at least with slab)
- C-state, throttling and cpufreq set up is done without valid cpu_data(new_cpu) resulting in wrong C-state (and other) info

Thomas, forgive my ignorance of udev rules ... but can one udev rule block another? ... That's not the best way to explain it ... here's an example:

Quoted 5 lines

CPU hot add on Nehalem-EX. Memory is brought online. Memory udev events are generated. CPU udev events are generated.

Can the memory udev events block the cpu udev events? ie) can I be assured that memory will come online before the cpus?

I give up, there are too many open issues.
Onlining cpus and memory in parallel currently is not a good idea.

If the cpu event comes first, you run into other issues. I accidently had the acpi_memhotplug driver not loaded, in this case a new node was not created at all and the cpus ended up on the wrong node.

Some comments on your patch:
Your patch misses EXPORT_SYMBOL(_GPL?)(set_memory_state); (I wonder why drivers/base/memory.c does not export symbols under gpl...) and EXPORT_SYMBOL_GPL(online_pages);

You also need to set each memory block MEM_ONLINE somehow, while all memory was active, only one memory block showed "online" here:
/sys/devices/system/node/node3/memory*/state
which has side-effects.

Would be great if you add me to CC if you repost or on related topics.

Thanks,

Thomas

#16 lore raw
Quoted 35 lines · Thomas Renninger
Quoted 33 lines · On Wednesday 17 March 2010 19:47:55 Prarit Bhargava wrote:
Quoted 31 lines · Thomas Renninger
Quoted 3 lines · On Thursday 11 March 2010 09:32:09 chen gong wrote:

On 2010-3-11 16:07, ykzhao wrote:

...

Quoted 4 lines

BTW, how about using UDEV rules to do this operation. It looks more smooth. I know some Novell guy is working on it.

I also know this guy :)

These would be the udev rules to automatically add memory/cpus:
SUBSYSTEM=="cpu", ACTION=="add", TEST=="online", ATTR{online}=="0", ATTR{online}="1", RUN+="/bin/logger onlining cpu: $env{DEVPATH}" SUBSYSTEM=="memory", ACTION=="add", TEST=="state", ATTR{state}=="offline", ATTR{state}="online", RUN+="/bin/logger onlining memory: $env{DEVPATH}"

But this should be the same as you suggest (at least the memory rule) to do in the kernel:
automatically online the memory, once hotadded.

I would not add any udev rules before this does not work reliably and currently it is totally broken, mainly because:
- not being able to alloc memory on foreign nodes (at least with slab)
- C-state, throttling and cpufreq set up is done without valid cpu_data(new_cpu) resulting in wrong C-state (and other) info

Quoted 10 lines
Quoted 6 lines

Thomas, forgive my ignorance of udev rules ... but can one udev rule block another? ... That's not the best way to explain it ... here's an example:

CPU hot add on Nehalem-EX. Memory is brought online. Memory udev events are generated. CPU udev events are generated.

Can the memory udev events block the cpu udev events? ie) can I be assured that memory will come online before the cpus?

I give up, there are too many open issues.
Onlining cpus and memory in parallel currently is not a good idea.

Thomas,

Quoted 5 lines

If the cpu event comes first, you run into other issues. I accidently had the acpi_memhotplug driver not loaded, in this case a new node was not created at all and the cpus ended up on the wrong node.

Right :(

Quoted 6 lines

Some comments on your patch:
Your patch misses EXPORT_SYMBOL(_GPL?)(set_memory_state); (I wonder why drivers/base/memory.c does not export symbols under gpl...) and EXPORT_SYMBOL_GPL(online_pages);

Okay, I'll fix that up.

Quoted 6 lines

You also need to set each memory block MEM_ONLINE somehow, while all memory was active, only one memory block showed "online" here:
/sys/devices/system/node/node3/memory*/state
which has side-effects.

Huh -- I wonder if that has to do with they way I'm calling online_pages. I wonder if I have to make that clal "deeper" in the call path and call it on EACH segment that I'm bringing into service...

Quoted 3 lines

Would be great if you add me to CC if you repost or on related topics.

Absolutely no problem :) I didn't mean to leave you out of the conversation :)

P.

Quoted 4 lines

Thanks,

Thomas

#17 lore raw
In reply to ykzhao
Quoted 18 lines · ykzhao
Quoted 9 lines · Prarit Bhargava
Quoted 9 lines

Yes. The nehalem processor has the integrated memory controller. But it is not required that the hot-added memory should be onlined before bringing up CPU.
I do the following memory-hotplug test on one Machine. a. Before hot plugging memory, four CPUs socket are installed and
all the logical CPU are brought up. (Only one node has the memory)
b. The memory is hot-plugged and then the memory is onlined so that
it can be accessed by the system.

Quoted 6 lines
Quoted 4 lines

In the above testing case the CPU is brought up before onlining the hot-added memory. And the test shows that it can work well.

That doesn't work when you have multiple nodes AFAICT. The cpus do not come into service because of a lack of memory on the node.... per node allocations will fail.

Quoted 9 lines

In the test the system has multiple nodes. The reason is that the cpu without memory can turn to other node and allocate the memory.

Quoted 2 lines

Just curious, exactly what did you test with? 2.6.33 (or newer)?

I test it on 2.6.32 kernel.

2.6.32 doesn't work with memory and cpu.
2.6.33 doesn't work with memory and cpu.

Other patches (which have been posted upstream) are required.

I'll try to dig up the patches and you can retest.

I will try a memory-free cpu add as well to see if that works... but the performance hit due to the off-node allocations probably doesn't make it worthwhile.

P.

Quoted 1 line
#18 lore raw
Quoted 26 lines · Prarit Bhargava
Quoted 24 lines · ykzhao
Quoted 9 lines · Prarit Bhargava
Quoted 9 lines

Yes. The nehalem processor has the integrated memory controller. But it is not required that the hot-added memory should be onlined before bringing up CPU.
I do the following memory-hotplug test on one Machine. a. Before hot plugging memory, four CPUs socket are installed and
all the logical CPU are brought up. (Only one node has the memory)
b. The memory is hot-plugged and then the memory is onlined so that
it can be accessed by the system.

Quoted 8 lines
Quoted 4 lines

In the above testing case the CPU is brought up before onlining the hot-added memory. And the test shows that it can work well.

That doesn't work when you have multiple nodes AFAICT. The cpus do not come into service because of a lack of memory on the node.... per node allocations will fail.

In the test the system has multiple nodes. The reason is that the cpu without memory can turn to other node and allocate the memory.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Just curious, exactly what did you test with? 2.6.33 (or newer)?

I test it on 2.6.32 kernel.

2.6.32 doesn't work with memory and cpu.
2.6.33 doesn't work with memory and cpu.

I don't test the scenario of both cpu and memory hot-plugging.

What I mean is that the 2.6.32 kernel can work if only memory is hot-plugged (The cpu is already brought-up in the boot phase).

Quoted 14 lines

Other patches (which have been posted upstream) are required.

I'll try to dig up the patches and you can retest.

I will try a memory-free cpu add as well to see if that works... but the performance hit due to the off-node allocations probably doesn't make it worthwhile.

P.

Quoted 1 line
#19 lore raw
In reply to ykzhao
Quoted 34 lines · On Thursday 11 March 2010 01:55:15 ykzhao wrote:
Quoted 9 lines · Prarit Bhargava
Quoted 6 lines

Why do we need to see whether the memory is onlined before bringing cpu to online state? It seems that there is no dependency between cpu online and memory online.

Yakui,

Thanks for the explanation.

Quoted 3 lines

Here's a deeper look into the issue. New Intel processors have an on-die memory controller and this means that as the socket comes and goes, so does the memory "behind" the socket.

Yes. The nehalem processor has the integrated memory controller. But it is not required that the hot-added memory should be onlined before bringing up CPU.
I do the following memory-hotplug test on one Machine. a. Before hot plugging memory, four CPUs socket are installed and
all the logical CPU are brought up. (Only one node has the memory)
b. The memory is hot-plugged and then the memory is onlined so that
it can be accessed by the system.

In the above testing case the CPU is brought up before onlining the hot-added memory. And the test shows that it can work well.

Quoted 5 lines

ie) with new processors it is possible that an entire node which consists of memory and cpus comes and goes with the socket enable and disable.

Quoted 7 lines
Quoted 3 lines

The cpu bringup code does local node allocations for the cpu. If the memory connected to the node (which is "behind" the socket) isn't online, then these allocations fail, and then the cpu bringup fails.

If the CPU can't allocate the memory from its own node, it can turn to other node and see whether the memory can be allocated. And this depends on the NUMA allocation policy.

Yes and this is broken and needs fixing.
Yakui, I expect you miss this patch and wrongly online the cpus to existing nodes, therefore you do not run into "out of memory" conditions:
0271f91003d3703675be13b8865618359a6caa1f

I know for sure that slab is broken.
slub behaves different, but I am not sure whether this is due to wrong CPU hotadd code (processor_core.c is also broken and you get wrong C-state info from BIOS tables on hotadded CPUs)

Prarit: Can you retest with slub and processor.max_cstate=1, this could/should work.

AFAIK vmware injects memory in the same way into clients, so you may have different behavior of virtualized Linux clients.

This is a work around for current memory management not being able to allocate from foreign nodes. That should not mean that I generally vote against this to get added. If it works reliably why not add a work around until the more complicated stuff works.

One question: You also want to automatically add the CPUs, once a CPU hotplug event got fired, right?
The fact that the memory hotplug driver adds the memory immediately once notified, does not ensure that the HW/BIOS fires this event first. Theoretically you need a logic to not add CPUs to memoryless nodes, poll/wait until memory got added, etc.

Thomas
--
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#20 lore raw
Quoted 35 lines · Thomas Renninger
Quoted 33 lines · On Thursday 11 March 2010 01:55:15 ykzhao wrote:
Quoted 9 lines · Prarit Bhargava
Quoted 6 lines

Why do we need to see whether the memory is onlined before bringing cpu to online state? It seems that there is no dependency between cpu online and memory online.

Yakui,

Thanks for the explanation.

Quoted 2 lines

Here's a deeper look into the issue. New Intel processors have an on-die memory controller and this means that as the socket comes and goes, so does the memory "behind" the socket.

Yes. The nehalem processor has the integrated memory controller. But it is not required that the hot-added memory should be onlined before bringing up CPU.
I do the following memory-hotplug test on one Machine. a. Before hot plugging memory, four CPUs socket are installed and
all the logical CPU are brought up. (Only one node has the memory)
b. The memory is hot-plugged and then the memory is onlined so that
it can be accessed by the system.

In the above testing case the CPU is brought up before onlining the hot-added memory. And the test shows that it can work well.

Quoted 4 lines

ie) with new processors it is possible that an entire node which consists of memory and cpus comes and goes with the socket enable and disable.

The cpu bringup code does local node allocations for the cpu. If the memory connected to the node (which is "behind" the socket) isn't online, then these allocations fail, and then the cpu bringup fails.

Quoted 9 lines
Quoted 4 lines

If the CPU can't allocate the memory from its own node, it can turn to other node and see whether the memory can be allocated. And this depends on the NUMA allocation policy.

Yes and this is broken and needs fixing.
Yakui, I expect you miss this patch and wrongly online the cpus to existing nodes, therefore you do not run into "out of memory" conditions:
0271f91003d3703675be13b8865618359a6caa1f

FWIW, I'm working on a 2.6.32 based tree, but I have that patch in (as well as several others). I'm also running the latest upstream (tip as of this morning). The issues I see in my 2.6.32 based tree are the same AFAICT that I see upstream: a cpu comes online and attempts to make a per_node allocation which fails, so the cpu bringup fails.

Quoted 2 lines

I know for sure that slab is broken.

Yes, but I believe Andi Kleen has added some patches that resolve (at least some of) the issues. I've been using slab (and occasionally testing slub).

Quoted 8 lines

slub behaves different, but I am not sure whether this is due to wrong CPU hotadd code (processor_core.c is also broken and you get wrong C-state info from BIOS tables on hotadded CPUs)

Prarit: Can you retest with slub and processor.max_cstate=1, this could/should work.

Two tests:

1. WITHOUT my auto online patch, the cpus fail to come into service because of a per_node allocation failure.
2. WITH my auto online patch, the cpus come into service

... I have NOT done any sort of testing to see if the cpus are really live ;)

Quoted 4 lines

AFAIK vmware injects memory in the same way into clients, so you may have different behavior of virtualized Linux clients.

Are you referring the vmware ballooning driver (or whatever they call it). IIRC (and I'm not saying I do ;) ), vmware adds memory and automatically onlines it in a guest. I'm not sure how that's done -- it could be via udev.

I'll see if anyone here knows.

Quoted 3 lines

One question: You also want to automatically add the CPUs, once a CPU hotplug event got fired, right?

Yes, That's correct.

Quoted 3 lines

The fact that the memory hotplug driver adds the memory immediately once notified, does not ensure that the HW/BIOS fires this event first.

Is that right? FWIW I always see this sequence of events:

ACPI memory added
ACPI cpus added

I never see them out-of-order. OTOH, I'm only testing on Intel's latest platform so maybe there are some older systems that don't do this in that order.

Quoted 3 lines

Theoretically you need a logic to not add CPUs to memoryless nodes, poll/wait until memory got added, etc.

Theoretically yes -- but are there any systems that generate cpu add events before memory add events?

Thanks for the input Thomas :)

P.

Quoted 2 lines

Thomas

--
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#21 lore raw
In reply to Prarit Bhargava

I think if you're going to poke into drivers/base/memory.c we should let the -mm guys know that you're creating a new interface.

The ACPI part looks fine to me.

cc added.

/ac

* Prarit Bhargava <[email protected]>:

Quoted 104 lines

New sockets have on-die memory controllers. This means that in certain HW configurations the memory behind the socket comes and goes as the socket is physically enabled and disabled.

Since the cpu bringup code does on node memory allocations, the memory on the added socket must be onlined first.

Add a .config option to automatically online hot added memory, and enable it in the acpi memory add path.

drivers/acpi/Kconfig +8 -0
index 93d2c79..dece6bd 100644
@@ -350,6 +350,14 @@ config ACPI_HOTPLUG_MEMORY
 	  To compile this driver as a module, choose M here:
 	  the module will be called acpi_memhotplug.
 
+config ACPI_HOTPLUG_MEMORY_AUTO_ONLINE
+	bool "Automatically online hotplugged memory"
+	depends on ACPI_HOTPLUG_MEMORY
+	default n
+	help
+	  This forces memory that is brought into service by ACPI
+	  to be automatically onlined.
+
 config ACPI_SBS
 	tristate "Smart Battery System"
 	depends on X86
drivers/acpi/acpi_memhotplug.c +14 -0
index 3597d73..9814c50 100644
@@ -30,6 +30,7 @@
 #include <linux/init.h>
 #include <linux/types.h>
 #include <linux/memory_hotplug.h>
+#include <linux/memory.h>
 #include <acpi/acpi_drivers.h>
 
 #define ACPI_MEMORY_DEVICE_CLASS		"memory"
@@ -252,6 +253,19 @@ static int acpi_memory_enable_device(struct acpi_memory_device *mem_device)
 		result = add_memory(node, info->start_addr, info->length);
 		if (result)
 			continue;
+#ifdef CONFIG_ACPI_HOTPLUG_MEMORY_AUTO_ONLINE
+		/*
+		 * New processors require memory to be online before cpus.
+		 * No notifications are required here as "we" are the only
+		 * ones who know about the new memory right now.
+		 */
+		result = online_pages(info->start_addr >> PAGE_SHIFT,
+				      info->length >> PAGE_SHIFT);
+		if (!result)
+			set_memory_state(info->start_addr, MEM_ONLINE);
+		else
+			printk("Memory online failed.\n");
+#endif
 		info->enabled = 1;
 		num_enabled++;
 	}
drivers/base/memory.c +14 -0
index 2f86915..fb465d5 100644
@@ -510,6 +510,20 @@ int remove_memory_block(unsigned long node_id, struct mem_section *section,
 }
 
 /*
+ * need an interface for the VM to mark sections on and offline when hot-adding
+ * memory.
+ */
+void set_memory_state(unsigned long addr, unsigned long state)
+{
+	struct mem_section *section;
+	struct memory_block *mem;
+
+	section = __pfn_to_section(addr >> PAGE_SHIFT);
+	mem = find_memory_block(section);
+	mem->state = state;
+}
+
+/*
  * need an interface for the VM to add new memory regions,
  * but without onlining it.
  */
include/linux/memory.h +1 -1
index 1adfe77..5d8d78c 100644
@@ -112,6 +112,7 @@ extern int remove_memory_block(unsigned long, struct mem_section *, int);
 extern int memory_notify(unsigned long val, void *v);
 extern int memory_isolate_notify(unsigned long val, void *v);
 extern struct memory_block *find_memory_block(struct mem_section *);
+extern void set_memory_state(unsigned long, unsigned long);
 #define CONFIG_MEM_BLOCK_SIZE	(PAGES_PER_SECTION<<PAGE_SHIFT)
 enum mem_add_context { BOOT, HOTPLUG };
 #endif /* CONFIG_MEMORY_HOTPLUG_SPARSE */
--

To unsubscribe from this list: send the line "unsubscribe linux-acpi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#22 lore raw
Quoted 4 lines · Alex Chiang

I think if you're going to poke into drivers/base/memory.c we should let the -mm guys know that you're creating a new interface.

Absolutely -- I was just RFC'ing for now ;)

Quoted 5 lines

The ACPI part looks fine to me.

cc added.

Thanks :)

P.

Quoted 109 lines

/ac

* Prarit Bhargava <[email protected]>:

Quoted 105 lines

New sockets have on-die memory controllers. This means that in certain HW configurations the memory behind the socket comes and goes as the socket is physically enabled and disabled.

Since the cpu bringup code does on node memory allocations, the memory on the added socket must be onlined first.

Add a .config option to automatically online hot added memory, and enable it in the acpi memory add path.

drivers/acpi/Kconfig +8 -0
index 93d2c79..dece6bd 100644
@@ -350,6 +350,14 @@ config ACPI_HOTPLUG_MEMORY
 	  To compile this driver as a module, choose M here:
 	  the module will be called acpi_memhotplug.
 
+config ACPI_HOTPLUG_MEMORY_AUTO_ONLINE
+	bool "Automatically online hotplugged memory"
+	depends on ACPI_HOTPLUG_MEMORY
+	default n
+	help
+	  This forces memory that is brought into service by ACPI
+	  to be automatically onlined.
+
 config ACPI_SBS
 	tristate "Smart Battery System"
 	depends on X86
drivers/acpi/acpi_memhotplug.c +14 -0
index 3597d73..9814c50 100644
@@ -30,6 +30,7 @@
 #include <linux/init.h>
 #include <linux/types.h>
 #include <linux/memory_hotplug.h>
+#include <linux/memory.h>
 #include <acpi/acpi_drivers.h>
 
 #define ACPI_MEMORY_DEVICE_CLASS		"memory"
@@ -252,6 +253,19 @@ static int acpi_memory_enable_device(struct acpi_memory_device *mem_device)
 		result = add_memory(node, info->start_addr, info->length);
 		if (result)
 			continue;
+#ifdef CONFIG_ACPI_HOTPLUG_MEMORY_AUTO_ONLINE
+		/*
+		 * New processors require memory to be online before cpus.
+		 * No notifications are required here as "we" are the only
+		 * ones who know about the new memory right now.
+		 */
+		result = online_pages(info->start_addr >> PAGE_SHIFT,
+				      info->length >> PAGE_SHIFT);
+		if (!result)
+			set_memory_state(info->start_addr, MEM_ONLINE);
+		else
+			printk("Memory online failed.\n");
+#endif
 		info->enabled = 1;
 		num_enabled++;
 	}
drivers/base/memory.c +14 -0
index 2f86915..fb465d5 100644
@@ -510,6 +510,20 @@ int remove_memory_block(unsigned long node_id, struct mem_section *section,
 }
 
 /*
+ * need an interface for the VM to mark sections on and offline when hot-adding
+ * memory.
+ */
+void set_memory_state(unsigned long addr, unsigned long state)
+{
+	struct mem_section *section;
+	struct memory_block *mem;
+
+	section = __pfn_to_section(addr >> PAGE_SHIFT);
+	mem = find_memory_block(section);
+	mem->state = state;
+}
+
+/*
  * need an interface for the VM to add new memory regions,
  * but without onlining it.
  */
include/linux/memory.h +1 -1
index 1adfe77..5d8d78c 100644
@@ -112,6 +112,7 @@ extern int remove_memory_block(unsigned long, struct mem_section *, int);
 extern int memory_notify(unsigned long val, void *v);
 extern int memory_isolate_notify(unsigned long val, void *v);
 extern struct memory_block *find_memory_block(struct mem_section *);
+extern void set_memory_state(unsigned long, unsigned long);
 #define CONFIG_MEM_BLOCK_SIZE	(PAGES_PER_SECTION<<PAGE_SHIFT)
 enum mem_add_context { BOOT, HOTPLUG };
 #endif /* CONFIG_MEMORY_HOTPLUG_SPARSE */
--

To unsubscribe from this list: send the line "unsubscribe linux-acpi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html