Skip to contentSkip to conversation

Documentation for CPU hotplug support

DISCUSSION Ashok Raj 7 participants 8 messages Nov 10, 2005 – Nov 23, 2005 Documentation

lore
Share & tools
lore

Ashok Raj

author
#0 lore raw

Hi Andrew

CPU hotplug support didnt have a readme for general help.

Thanks to Andi for some feedback.

Consider for -mm so if people have updates they can do when it gets up the tree.

I added authors as much as i can remember, if roles have changed, please send updates to this document.

Signature · 286 lines
Cheers,
Ashok Raj
- Open Source Technology Center


First cut Documentation update for cpu hotplug.

Signed-off-by: Ashok Raj <[email protected]>
--------------------------------------------------------
 Documentation/cpu-hotplug.txt |  269 ++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 269 insertions(+)

Index: linux-2.6.14-mm1/Documentation/cpu-hotplug.txt
===================================================================
--- /dev/null
+++ linux-2.6.14-mm1/Documentation/cpu-hotplug.txt
@@ -0,0 +1,269 @@
+					CPU hotplug Support in Linux(tm) Kernel
+
+Authors:
+
+General Core Infrastructure:
+
+Rusty Russell: 		[email protected]
+Srivatsa Vaddagiri: [email protected]
+
+i386:
+Zwane:				[email protected]
+
+ppc64:
+Joel Schopp:		[email protected]
+Nathan Lynch:		[email protected]
+
+
+ia64/x86_64:
+Ashok Raj			[email protected]
+Anil (ACPI support)	[email protected]
+
+Introduction
+
+	Modern advances in system architectures have introduced advanced error
+reporting and correction capabilities in processors. CPU architectures permit
+partitioning support, where compute resources of a single CPU could be made
+available to virtual machine environments. There are couple OEMS that
+support NUMA hardware which are hot pluggable as well, where physical
+node insertion and removal require CPU hotplug support.
+
+Such advances require CPU available to a kernel to be removed either for
+provisioning reasons, or for RAS purposes to keep an offending CPU off
+system execution path. Hence the need for CPU hotplug support in the
+Linux kernel.
+
+A more novel use of CPU-hotplug support is its use in suspend
+resume support for SMP. Dual-core and HT support makes even
+a laptop run SMP kernels which didn't support these methods. SMP support
+for suspend/resume is a work in progress. This is a work in progress in
+ACPI for now.
+
+General Stuff about CPU Hotplug
+--------------------------------
+
+cpu_possible_map: Bitmap of possible CPUs that can ever be available in the
+system. This is used to allocated some boot time memory for per_cpu variables
+that aren't designed to grow/shrink as CPUs are made available or removed.
+Trimming it accurately for your system needs upfront can save some
+boot time memory. See below for how we use heuristics in x86_64 case
+to keep this under check.
+
+cpu_online_map: Bitmap of all CPUs currently online.
+
+cpu_present_map: Bitmap of CPUs currently present in the system. Not all
+of them may be online. One could choose to start with maxcpus=n boot time
+parameter to limit the number of boot time CPUs.
+
+When manipulating with cpu maps, please use one of the appropriate
+ones already defined.
+
+Never use anything other than cpumask_t to represent
+bitmap of CPUs.
+
+#include <linux/cpumask.h>
+
+for_each_cpu			- Iterate over cpu_possible_map
+for_each_online_cpu		- Iterate over cpu_online_map
+for_each_present_cpu	- Iterate over cpu_present_map
+for_each_cpu_mask(x,mask)	- Iterate over some random collection of cpu mask.
+
+#include <linux/cpu.h>
+lock_cpu_hotplug() and unlock_cpu_hotplug():
+Used to stop cpu hotplug from being started.
+
+CPU Hotplug - Frequently Asked Questions.
+
+Q: How to i enable my kernel to support CPU hotplug?
+A: When doing make defconfig, Enable CPU hotplug support
+
+   "Processor type and Features" -> Support for Hotpluggable CPUs
+
+Make sure that you have CONFIG_HOTPLUG, and CONFIG_SMP turned on as well.
+
+You would need to enable CONFIG_HOTPLUG_CPU for SMP suspend/resume support
+as well.
+
+Q: What architectures support CPU hotplug?
+A: As of 2.6.14, the following architectures support CPU hotplug.
+
+i386 (Intel), ppc, ppc64, parisc, s390, ia64 and x86_64
+
+Q: How to test if hotplug is supported on the newly built kernel?
+A: You should now notice an entry in sysfs.
+
+Check if sysfs is mounted, using the "mount" command. You should notice
+an entry as shown below in the output.
+
+....
+none on /sys type sysfs (rw)
+....
+
+if this is not mounted, do the following.
+
+#mkdir /sysfs
+#mount -t sysfs sys /sys
+
+now you should see entries for all present cpu, the following is an example
+in a 8-way system.
+
+#pwd
+#/sys/devices/system/cpu
+#ls -l
+total 0
+drwxr-xr-x  10 root root 0 Sep 19 07:44 .
+drwxr-xr-x  13 root root 0 Sep 19 07:45 ..
+drwxr-xr-x   3 root root 0 Sep 19 07:44 cpu0
+drwxr-xr-x   3 root root 0 Sep 19 07:44 cpu1
+drwxr-xr-x   3 root root 0 Sep 19 07:44 cpu2
+drwxr-xr-x   3 root root 0 Sep 19 07:44 cpu3
+drwxr-xr-x   3 root root 0 Sep 19 07:44 cpu4
+drwxr-xr-x   3 root root 0 Sep 19 07:44 cpu5
+drwxr-xr-x   3 root root 0 Sep 19 07:44 cpu6
+drwxr-xr-x   3 root root 0 Sep 19 07:48 cpu7
+
+Under each directory you would find an "online" file which is the control
+file to logically online/offline a processor.
+
+Q: What is logical CPU hotplug and Physical CPU hotplug?
+A: Logical CPU hotplug just removes the presence of the CPU from the
+kernel image. Physical CPU hotplug refers to the electrical isolation
+of the CPU from the system. For support of physical hotplug there needs
+to be a method such as ACPI that interfaces with the system BIOS to
+pass notification of removal to kernel, such as the Attention button in
+PCI based hotplug.
+
+Q: How do i logically offline a CPU?
+A: Do the following.
+
+#echo 0 > /sys/devices/system/cpu/cpuX/online
+
+once the logical offline is successful, check
+
+#cat /proc/interrupts
+
+you should now not see the CPU that you removed.
+
+Q: Why cant i remove CPU0 on some systems?
+A: Some architectures may have some special dependency on a certain CPU.
+
+For e.g in IA64 platforms we have ability to sent platform interrupts to the
+OS. a.k.a Corrected Platform Error Interrupts (CPEI). In current ACPI
+specifications, we dint have a way to change the target CPU. Hence if the
+current ACPI version doesn't support such re-direction, we disable that CPU
+by making it not-removable.
+
+Q: How do i find out if a particular CPU is not removable?
+A: Depending on the implementation, some architectures may show this by the
+absence of the "online" file. This is done if it can be determined ahead of
+time that this CPU cannot be removed.
+
+In some situations, this can be a run time check, i.e if you try to remove the
+last CPU, this will not be permitted. You can find such failures by
+investigating the return value of the "echo" command.
+
+Q: What happens when a CPU is being logically offlined?
+A: The following happen, listed in no particular order :-)
+
+- A notification is sent to in-kernel registered modules by sending an event
+  CPU_DOWN_PREPARE
+- All process is migrated away from this outgoing CPU to a new CPU
+- All interrupts targeted to this CPU is migrated to a new CPU
+- timers/bottom half/task lets are also migrated to a new CPU
+- Once all services are migrated, kernel calls an arch specific routine
+  __cpu_disable() to perform arch specific cleanup.
+- Once this is successful, an event for successful cleanup is sent by an event
+  CPU_DEAD.
+
+  "It is expected that each service cleans up when the CPU_DOWN_PREPARE
+  notifier is called, when CPU_DEAD is called its expected there is nothing
+  running on behalf of this CPU that was offlined"
+
+Q: If i have some kernel code that needs to be aware of CPU arrival and
+   departure, how to i arrange for proper notification?
+A: This is what you would need in your kernel code to receive notifications.
+
+    #include <linux/cpu.h>
+	static int __cpuinit foobar_cpu_callback(struct notifier_block *nfb,
+								unsigned long action, void *hcpu)
+	{
+		unsigned int cpu = (unsigned long)hcpu;
+
+		switch (action) {
+		case CPU_ONLINE:
+			foobar_online_action(cpu);
+			break;
+		case CPU_DEAD:
+			foobar_dead_action(cpu);
+			break;
+		}
+		return NOTIFY_OK;
+	}
+
+	static struct notifier_block foobar_cpu_notifer =
+	{
+	   .notifier_call = foobar_cpu_callback,
+	};
+
+
+In your init function,
+
+	register_cpu_notifier(&foobar_cpu_notifier);
+
+You can fail PREPARE notifiers if something doesn't work to prepare resources.
+This will stop the activity and send a following CANCELED event back.
+
+CPU_DEAD should not be failed, its just a goodness indication, but bad
+things will happen if a notifier in path sent a BAD notify code.
+
+Q: I don't see my action being called for all CPUs already up and running?
+A: Yes, CPU notifiers are called only when new CPUs are on-lined or offlined.
+   If you need to perform some action for each cpu already in the system, then
+
+  for_each_online_cpu(i) {
+		foobar_cpu_callback(&foobar_cpu_notifier, CPU_UP_PREPARE, i);
+		foobar_cpu_callback(&foobar-cpu_notifier, CPU_ONLINE, i);
+  }
+
+Q: If i would like to develop cpu hotplug support for a new architecture,
+   what do i need at a minimum?
+A: The following are what is required for CPU hotplug infrastructure to work
+   correctly.
+
+    - Make sure you have an entry in Kconfig to enable CONFIG_HOTPLUG_CPU
+	- __cpu_up() 		- Arch interface to bring up a CPU
+	- __cpu_disable()	- Arch interface to shutdown a CPU, no more interrupts
+						  can be handled by the kernel after the routine
+						  returns. Including local APIC timers etc are
+						  shutdown.
+	- __cpu_die()		- This actually supposed to ensure death of the CPU.
+						  Actually look at some example code in other arch
+						  that implement CPU hotplug. The processor is taken
+						  down from the idle() loop for that specific
+						  architecture. __cpu_die() typically waits for some
+						  per_cpu state to be set, to ensure the processor
+						  dead routine is called to be sure positively.
+
+Q: I need to ensure the a particular cpu is not removed when there is some
+   work specific to this cpu is in progress.
+A: Take a look at drivers/cpufreq/cpufreq.c. Typically you would acquire
+   the cpucontrol lock via lock_cpu_hotplug() and unlock_cpu_hotplug().
+
+   When you are in the notification callbacks, you don't need to acquire the
+   lock. use the current_in_hotplug() call to verify is the current
+   thread is performing the hotplug functions. This will ensure you don't
+   end up with deadlocks trying to acquire cpucontrol while its already held
+   by the current caller.
+
+Q: How do we determine how many CPUs are available for hotplug.
+A: There is no clear spec defined way from ACPI that can give us that
+   information today. Based on some input from Natalie of Unisys,
+   that the MADT ACPI tables marks those possible CPUs in a system with
+   disabled status.
+
+   Andi implemented some simple heuristics that count the number of disabled
+   CPUs in MADT as hotpluggable CPUS.  In the case there are no disabled CPUS
+   we assume 1/2 the number of CPUs currently present can be hotplugged.
+
+   Caveat: Today's MADT can only provide 256 entries since the apicid field
+   in MADT is only 8 bits.
Acked-by
#1 lore raw
Quoted 4 lines · Ashok Raj

Hi Andrew

CPU hotplug support didnt have a readme for general help.

Quoted 8 lines

Thanks to Andi for some feedback.

Consider for -mm so if people have updates they can do when it gets up the tree.

I added authors as much as i can remember, if roles have changed, please send updates to this document.

While far from complete (no mention of userspace notification for instance), it is a very good start.

#2 lore raw
In reply to Ashok Raj

Hi Ashok-

Thanks a lot for doing this; it's a good start. Some feedback and minor corrections below:

Quoted 45 lines · Ashok Raj

Index: linux-2.6.14-mm1/Documentation/cpu-hotplug.txt
===================================================================

Documentation/cpu-hotplug.txt new file +40 -0
@@ -0,0 +1,269 @@
+					CPU hotplug Support in Linux(tm) Kernel
+
+Authors:
+
+General Core Infrastructure:
+
+Rusty Russell: 		[email protected]
+Srivatsa Vaddagiri: [email protected]
+
+i386:
+Zwane:				[email protected]
+
+ppc64:
+Joel Schopp:		[email protected]
+Nathan Lynch:		[email protected]
+
+
+ia64/x86_64:
+Ashok Raj			[email protected]
+Anil (ACPI support)	[email protected]
+
+Introduction
+
+	Modern advances in system architectures have introduced advanced error
+reporting and correction capabilities in processors. CPU architectures permit
+partitioning support, where compute resources of a single CPU could be made
+available to virtual machine environments. There are couple OEMS that
+support NUMA hardware which are hot pluggable as well, where physical
+node insertion and removal require CPU hotplug support.
+
+Such advances require CPU available to a kernel to be removed either for
+provisioning reasons, or for RAS purposes to keep an offending CPU off
+system execution path. Hence the need for CPU hotplug support in the
+Linux kernel.
+
+A more novel use of CPU-hotplug support is its use in suspend
+resume support for SMP. Dual-core and HT support makes even
+a laptop run SMP kernels which didn't support these methods. SMP support
+for suspend/resume is a work in progress. This is a work in progress in
+ACPI for now.

Documentation/cpu-hotplug.txt line 50

Quoted 10 lines
Documentation/cpu-hotplug.txt new file +10 -0
+
+General Stuff about CPU Hotplug
+--------------------------------
+
+cpu_possible_map: Bitmap of possible CPUs that can ever be available in the
+system. This is used to allocated some boot time memory for per_cpu variables
+that aren't designed to grow/shrink as CPUs are made available or removed.
+Trimming it accurately for your system needs upfront can save some
+boot time memory. See below for how we use heuristics in x86_64 case
+to keep this under check.

It think it would be appropriate to stress that cpu_possible_map must not change after boot.

Documentation/cpu-hotplug.txt line 52

Quoted 2 lines
Documentation/cpu-hotplug.txt new file +2 -0
+
+cpu_online_map: Bitmap of all CPUs currently online.

I think a little discussion of what it means for a cpu to be marked online is in order. For instance, it is available for running tasks and servicing interrupts.

Documentation/cpu-hotplug.txt line 56

Quoted 4 lines
Documentation/cpu-hotplug.txt new file +4 -0
+
+cpu_present_map: Bitmap of CPUs currently present in the system. Not all
+of them may be online. One could choose to start with maxcpus=n boot time
+parameter to limit the number of boot time CPUs.

Shouldn't the maxcpus bit be somewhere else? maxcpus shouldn't affect the setting of cpu_present_map. Maybe it should go with the online map section.

Also mention that cpu_present_map may change during the system's lifetime (have we worked out the locking rules for that?)

Documentation/cpu-hotplug.txt line 59

Quoted 3 lines
Documentation/cpu-hotplug.txt new file +3 -0
+
+When manipulating with cpu maps, please use one of the appropriate
+ones already defined.

In general, most code should not manipulate or directly access cpu maps at all, especially not cpu_online_map or cpu_present_map. Per-cpu resources should almost always be set up according to cpu_possible_map/for_each_cpu.

Quoted 4 lines
Documentation/cpu-hotplug.txt new file +4 -0
+
+Never use anything other than cpumask_t to represent
+bitmap of CPUs.
+

Documentation/cpu-hotplug.txt line 73

Quoted 10 lines
Documentation/cpu-hotplug.txt new file +10 -0
+#include <linux/cpumask.h>
+
+for_each_cpu			- Iterate over cpu_possible_map
+for_each_online_cpu		- Iterate over cpu_online_map
+for_each_present_cpu	- Iterate over cpu_present_map
+for_each_cpu_mask(x,mask)	- Iterate over some random collection of cpu mask.
+
+#include <linux/cpu.h>
+lock_cpu_hotplug() and unlock_cpu_hotplug():
+Used to stop cpu hotplug from being started.

Used to inhibit cpu hotplug operations. While the cpucontrol mutex is held, cpu_online_map will not change.

If you merely need to keep any cpus from going offline, disabling preemption in your critical section is sufficient. This is true as long as cpu_down is implemented with stop_machine_run.

Quoted 22 lines
Documentation/cpu-hotplug.txt new file +22 -0
+
+CPU Hotplug - Frequently Asked Questions.
+
+Q: How to i enable my kernel to support CPU hotplug?
+A: When doing make defconfig, Enable CPU hotplug support
+
+   "Processor type and Features" -> Support for Hotpluggable CPUs
+
+Make sure that you have CONFIG_HOTPLUG, and CONFIG_SMP turned on as well.
+
+You would need to enable CONFIG_HOTPLUG_CPU for SMP suspend/resume support
+as well.
+
+Q: What architectures support CPU hotplug?
+A: As of 2.6.14, the following architectures support CPU hotplug.
+
+i386 (Intel), ppc, ppc64, parisc, s390, ia64 and x86_64
+
+Q: How to test if hotplug is supported on the newly built kernel?
+A: You should now notice an entry in sysfs.
+
+Check if sysfs is mounted, using the "mount" command. You should notice

Documentation/cpu-hotplug.txt line 105

Quoted 10 lines
Documentation/cpu-hotplug.txt new file +10 -0
+an entry as shown below in the output.
+
+....
+none on /sys type sysfs (rw)
+....
+
+if this is not mounted, do the following.
+
+#mkdir /sysfs
+#mount -t sysfs sys /sys

Eh, I think the mounting sysfs stuff is superfluous these days, but that's a nitpick.

Quoted 15 lines
Documentation/cpu-hotplug.txt new file +15 -0
+
+now you should see entries for all present cpu, the following is an example
+in a 8-way system.
+
+#pwd
+#/sys/devices/system/cpu
+#ls -l
+total 0
+drwxr-xr-x  10 root root 0 Sep 19 07:44 .
+drwxr-xr-x  13 root root 0 Sep 19 07:45 ..
+drwxr-xr-x   3 root root 0 Sep 19 07:44 cpu0
+drwxr-xr-x   3 root root 0 Sep 19 07:44 cpu1
+drwxr-xr-x   3 root root 0 Sep 19 07:44 cpu2
+drwxr-xr-x   3 root root 0 Sep 19 07:44 cpu3
+drwxr-xr-x   3 root root 0 Sep 19 07:44 cpu4

Documentation/cpu-hotplug.txt line 130

Quoted 10 lines
Documentation/cpu-hotplug.txt new file +10 -0
+drwxr-xr-x   3 root root 0 Sep 19 07:44 cpu5
+drwxr-xr-x   3 root root 0 Sep 19 07:44 cpu6
+drwxr-xr-x   3 root root 0 Sep 19 07:48 cpu7
+
+Under each directory you would find an "online" file which is the control
+file to logically online/offline a processor.
+
+Q: What is logical CPU hotplug and Physical CPU hotplug?
+A: Logical CPU hotplug just removes the presence of the CPU from the
+kernel image.

Well... I think the "hotplug" term is getting overused and is possibly confusing here. I'd say it is more precise to distinguish between online/offline operations and addition/removal.

Quoted 6 lines

... Physical CPU hotplug refers to the electrical isolation

Documentation/cpu-hotplug.txt new file +5 -0
+of the CPU from the system. For support of physical hotplug there needs
+to be a method such as ACPI that interfaces with the system BIOS to
+pass notification of removal to kernel, such as the Attention button in
+PCI based hotplug.
+

Documentation/cpu-hotplug.txt

Quoted 10 lines
Documentation/cpu-hotplug.txt new file +10 -0
+Q: How do i logically offline a CPU?
+A: Do the following.
+
+#echo 0 > /sys/devices/system/cpu/cpuX/online
+
+once the logical offline is successful, check
+
+#cat /proc/interrupts
+
+you should now not see the CPU that you removed.

Or cat /sys/devices/system/cpu/cpuX/online, it should print 0.

Quoted 89 lines
Documentation/cpu-hotplug.txt new file +89 -0
+
+Q: Why cant i remove CPU0 on some systems?
+A: Some architectures may have some special dependency on a certain CPU.
+
+For e.g in IA64 platforms we have ability to sent platform interrupts to the
+OS. a.k.a Corrected Platform Error Interrupts (CPEI). In current ACPI
+specifications, we dint have a way to change the target CPU. Hence if the
+current ACPI version doesn't support such re-direction, we disable that CPU
+by making it not-removable.
+
+Q: How do i find out if a particular CPU is not removable?
+A: Depending on the implementation, some architectures may show this by the
+absence of the "online" file. This is done if it can be determined ahead of
+time that this CPU cannot be removed.
+
+In some situations, this can be a run time check, i.e if you try to remove the
+last CPU, this will not be permitted. You can find such failures by
+investigating the return value of the "echo" command.
+
+Q: What happens when a CPU is being logically offlined?
+A: The following happen, listed in no particular order :-)
+- A notification is sent to in-kernel registered modules by sending an event
+  CPU_DOWN_PREPARE
+- All process is migrated away from this outgoing CPU to a new CPU
+- All interrupts targeted to this CPU is migrated to a new CPU
+- timers/bottom half/task lets are also migrated to a new CPU
+- Once all services are migrated, kernel calls an arch specific routine
+  __cpu_disable() to perform arch specific cleanup.
+- Once this is successful, an event for successful cleanup is sent by an event
+  CPU_DEAD.
+
+  "It is expected that each service cleans up when the CPU_DOWN_PREPARE
+  notifier is called, when CPU_DEAD is called its expected there is nothing
+  running on behalf of this CPU that was offlined"
+
+Q: If i have some kernel code that needs to be aware of CPU arrival and
+   departure, how to i arrange for proper notification?
+A: This is what you would need in your kernel code to receive notifications.
+
+    #include <linux/cpu.h>
+	static int __cpuinit foobar_cpu_callback(struct notifier_block *nfb,
+								unsigned long action, void *hcpu)
+	{
+		unsigned int cpu = (unsigned long)hcpu;
+
+		switch (action) {
+		case CPU_ONLINE:
+			foobar_online_action(cpu);
+			break;
+		case CPU_DEAD:
+			foobar_dead_action(cpu);
+			break;
+		}
+		return NOTIFY_OK;
+	}
+
+	static struct notifier_block foobar_cpu_notifer =
+	{
+	   .notifier_call = foobar_cpu_callback,
+	};
+
+
+In your init function,
+
+	register_cpu_notifier(&foobar_cpu_notifier);
+
+You can fail PREPARE notifiers if something doesn't work to prepare resources.
+This will stop the activity and send a following CANCELED event back.
+
+CPU_DEAD should not be failed, its just a goodness indication, but bad
+things will happen if a notifier in path sent a BAD notify code.
+
+Q: I don't see my action being called for all CPUs already up and running?
+A: Yes, CPU notifiers are called only when new CPUs are on-lined or offlined.
+   If you need to perform some action for each cpu already in the system, then
+
+  for_each_online_cpu(i) {
+		foobar_cpu_callback(&foobar_cpu_notifier, CPU_UP_PREPARE, i);
+		foobar_cpu_callback(&foobar-cpu_notifier, CPU_ONLINE, i);
+  }
+
+Q: If i would like to develop cpu hotplug support for a new architecture,
+   what do i need at a minimum?
+A: The following are what is required for CPU hotplug infrastructure to work
+   correctly.
+
+    - Make sure you have an entry in Kconfig to enable CONFIG_HOTPLUG_CPU
+	- __cpu_up() 		- Arch interface to bring up a CPU
+	- __cpu_disable()	- Arch interface to shutdown a CPU, no more interrupts

Documentation/cpu-hotplug.txt

Quoted 10 lines
Documentation/cpu-hotplug.txt new file +10 -0
+						  can be handled by the kernel after the routine
+						  returns. Including local APIC timers etc are
+						  shutdown.
+	- __cpu_die()		- This actually supposed to ensure death of the CPU.
+						  Actually look at some example code in other arch
+						  that implement CPU hotplug. The processor is taken
+						  down from the idle() loop for that specific
+						  architecture. __cpu_die() typically waits for some
+						  per_cpu state to be set, to ensure the processor
+						  dead routine is called to be sure positively.

Formatting? This section doesn't fit in my terminal.

Documentation/cpu-hotplug.txt

Quoted 11 lines
Documentation/cpu-hotplug.txt new file +11 -0
+
+Q: I need to ensure the a particular cpu is not removed when there is some
+   work specific to this cpu is in progress.
+A: Take a look at drivers/cpufreq/cpufreq.c. Typically you would acquire
+   the cpucontrol lock via lock_cpu_hotplug() and unlock_cpu_hotplug().
+
+   When you are in the notification callbacks, you don't need to acquire the
+   lock. use the current_in_hotplug() call to verify is the current
+   thread is performing the hotplug functions. This will ensure you don't
+   end up with deadlocks trying to acquire cpucontrol while its already held
+   by the current caller.

Argh, no. That current_in_cpuhotplug hack has to go.

And actually, the A: to that particular Q: should be "disable preemption in your critical section".

Quoted 12 lines
Documentation/cpu-hotplug.txt new file +12 -0
+Q: How do we determine how many CPUs are available for hotplug.
+A: There is no clear spec defined way from ACPI that can give us that
+   information today. Based on some input from Natalie of Unisys,
+   that the MADT ACPI tables marks those possible CPUs in a system with
+   disabled status.
+
+   Andi implemented some simple heuristics that count the number of disabled
+   CPUs in MADT as hotpluggable CPUS.  In the case there are no disabled CPUS
+   we assume 1/2 the number of CPUs currently present can be hotplugged.
+
+   Caveat: Today's MADT can only provide 256 entries since the apicid field
+   in MADT is only 8 bits.
#3 lore raw
Quoted 2 lines · Nathan

It think it would be appropriate to stress that cpu_possible_map must not change after boot.

This and some other aspects of cpu_online_map, cpu_possible_map and cpu_present_map are documented in a long comment in cpumask.h.

You are welcome to steal from or reference that comment.

You are also welcome to fix up any lies you find in that comment ;).

Signature · 3 lines
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401
#4 lore raw
In reply to Nathan Lynch
Quoted 2 lines · Nathan Lynch

Argh, no. That current_in_cpuhotplug hack has to go.

Yes, Ashok is busily working on removing that ;)

We shouldn't enshrine it in documentation. Please.

#5 lore raw
Quoted 5 lines · Andrew Morton
Quoted 2 lines · Nathan Lynch

Argh, no. That current_in_cpuhotplug hack has to go.

Yes, Ashok is busily working on removing that ;)

Ashok was my patch for the cpufreq driver *that* horrible? Or perhaps we just need to move things like the set_cpus_allowed further up in the calls and handle everything in one location. Interested?

Thanks,
Zwane

#6 lore raw
In reply to Andrew Morton
Quoted 5 lines · * Andrew Morton ([email protected]) wrote:
Quoted 2 lines · Nathan Lynch

Argh, no. That current_in_cpuhotplug hack has to go.

Yes, Ashok is busily working on removing that ;)

That's nice. As it is, it deadlocks on boot:

cpufreq_stats_init
lock_cpu_hotplug
cpufreq_stat_cpu_callback
... (trace below)
cpufreq_governor_performance
__cpufreq_driver_target
lock_cpu_hotplug <-- uh-oh

SysRq : Show State

sibling
task PC pid father child younger older
swapper D 003D08BC 764 1 0 2 (L-TLB)

       eff8ed78 ef832030 c184a5f0 003d08bc effc5180 000000d0 df810b00 003d08bc
       00000002 ef832030 c184a5a0 00000000 df810b00 003d08bc ef832030 eff8dab0
       eff8dbd8 c0637140 c0637148 00000282 eff8edb0 c058bad3 eff8dab0 00000001
Call Trace:
 [<c058bad3>] __down+0x76/0xde
 [<c058a4aa>] __down_failed+0xa/0x10
 [<c04609ef>] .text.lock.cpufreq+0xee/0x1ff
 [<c0461049>] cpufreq_governor_performance+0x5d/0x74
 [<c045ff1c>] __cpufreq_governor+0x70/0x106
 [<c04603d8>] __cpufreq_set_policy+0x214/0x2b9
 [<c04605db>] cpufreq_update_policy+0xca/0xf1
 [<c0460fd5>] cpufreq_stat_cpu_callback+0x27/0x3e
 [<c06f657d>] cpufreq_stats_init+0xf0/0x152
 [<c06d59c2>] do_initcalls+0x56/0xba
 [<c06d5a4a>] do_basic_setup+0x24/0x2a
 [<c01003cf>] init+0xb5/0x1fe
 [<c010148d>] kernel_thread_helper+0x5/0xb
author
#7 lore raw
In reply to Nathan Lynch
Quoted 6 lines · Nathan Lynch

Hi Ashok-

Thanks a lot for doing this; it's a good start. Some feedback and minor corrections below:

Thanks Nathan for the feedback. Beleive it has most of feedback i got todate. user space side is still missing, i will try to add it after some time once i have right data.

- spell checked (maybe ;-))
- Added some more code snips
- Some more details liks available_cpus in x86_64.

Andrew: Could you help queue this for next update, so we will have some starting point.

CPU Hotplug Documentation writeup.

------------------------------------------------------------

 Documentation/cpu-hotplug.txt |  315 ++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 315 insertions(+)
Documentation/cpu-hotplug.txt new file +315 -0
Index: linux-2.6.15-rc1-mm2/Documentation/cpu-hotplug.txt
===================================================================
@@ -0,0 +1,315 @@
+		CPU hotplug Support in Linux(tm) Kernel
+
+
+Introduction
+
+Modern advances in system architectures have introduced advanced error
+reporting and correction capabilities in processors. CPU architectures permit
+partitioning support, where compute resources of a single CPU could be made
+available to virtual machine environments. There are couple OEMS that
+support NUMA hardware which are hot pluggable as well, where physical
+node insertion and removal require support for CPU hotplug.
+
+Such advances require CPUs available to a kernel to be removed either for
+provisioning reasons, or for RAS purposes to keep an offending CPU off
+system execution path. Hence the need for CPU hotplug support in the
+Linux kernel.
+
+A more novel use of CPU-hotplug support is its use today in suspend
+resume support for SMP. Dual-core and HT support makes even
+a laptop run SMP kernels which didn't support these methods. SMP support
+for suspend/resume is a work in progress.
+
+General Stuff about CPU Hotplug
+--------------------------------
+
+Command Line Switches
+---------------------
+maxcpus=n    Restrict boot time cpus to n. Say if you have 4 cpus, using
+             maxcpus=2 will only boot 2. You can choose to bring the
+             other cpus later online, read FAQ's for more info.
+
+additional_cpus=n	[x86_64 only] use this to limit hotpluggable cpus.
+                        This option sets
+			cpu_possible_map = cpu_present_map + additional_cpus
+
+CPU maps and such
+-----------------
+[More on cpumaps and primitive to manipulate, please check
+include/linux/cpumask.h that has more descriptive text.]
+
+cpu_possible_map: Bitmap of possible CPUs that can ever be available in the
+system. This is used to allocate some boot time memory for per_cpu variables
+that aren't designed to grow/shrink as CPUs are made available or removed.
+Once set during boot time discovery phase, the map is static, i.e no bits
+are added or removed anytime.  Trimming it accurately for your system needs
+upfront can save some boot time memory. See below for how we use heuristics
+in x86_64 case to keep this under check.
+
+cpu_online_map: Bitmap of all CPUs currently online. Its set in __cpu_up()
+after a cpu is available for kernel scheduling and ready to receive
+interrupts from devices. Its cleared when a cpu is brought down using
+__cpu_disable(), before which all OS services including interrupts are
+migrated to another target CPU.
+
+cpu_present_map: Bitmap of CPUs currently present in the system. Not all
+of them may be online. When physical hotplug is processed by the relevant
+subsystem (e.g ACPI) can change and new bit either be added or removed
+from the map depending on the event is hot-add/hot-remove. There are currently
+no locking rules as of now. Typical usage is to init topology during boot,
+at which time hotplug is disabled.
+
+You really dont need to manipulate any of the system cpu maps. They should
+be read-only for most use. When setting up per-cpu resources almost always use
+cpu_possible_map/for_each_cpu() to iterate.
+
+Never use anything other than cpumask_t to represent bitmap of CPUs.
+
+#include <linux/cpumask.h>
+
+for_each_cpu              - Iterate over cpu_possible_map
+for_each_online_cpu       - Iterate over cpu_online_map
+for_each_present_cpu      - Iterate over cpu_present_map
+for_each_cpu_mask(x,mask) - Iterate over some random collection of cpu mask.
+
+#include <linux/cpu.h>
+lock_cpu_hotplug() and unlock_cpu_hotplug():
+
+The above calls are used to inhibit cpu hotplug operations. While holding the
+cpucontrol mutex, cpu_online_map will not change. If you merely need to avoid
+cpus going away, you could also use preempt_disable() and preempt_enable()
+for those sections. Just remember the critical section cannot call any
+function that can sleep or schedule this process away. The preempt_disable()
+will work as long as stop_machine_run() is used to take a cpu down.
+
+CPU Hotplug - Frequently Asked Questions.
+
+Q: How to i enable my kernel to support CPU hotplug?
+A: When doing make defconfig, Enable CPU hotplug support
+
+   "Processor type and Features" -> Support for Hotpluggable CPUs
+
+Make sure that you have CONFIG_HOTPLUG, and CONFIG_SMP turned on as well.
+
+You would need to enable CONFIG_HOTPLUG_CPU for SMP suspend/resume support
+as well.
+
+Q: What architectures support CPU hotplug?
+A: As of 2.6.14, the following architectures support CPU hotplug.
+
+i386 (Intel), ppc, ppc64, parisc, s390, ia64 and x86_64
+
+Q: How to test if hotplug is supported on the newly built kernel?
+A: You should now notice an entry in sysfs.
+
+Check if sysfs is mounted, using the "mount" command. You should notice
+an entry as shown below in the output.
+
+....
+none on /sys type sysfs (rw)
+....
+
+if this is not mounted, do the following.
+
+#mkdir /sysfs
+#mount -t sysfs sys /sys
+
+now you should see entries for all present cpu, the following is an example
+in a 8-way system.
+
+#pwd
+#/sys/devices/system/cpu
+#ls -l
+total 0
+drwxr-xr-x  10 root root 0 Sep 19 07:44 .
+drwxr-xr-x  13 root root 0 Sep 19 07:45 ..
+drwxr-xr-x   3 root root 0 Sep 19 07:44 cpu0
+drwxr-xr-x   3 root root 0 Sep 19 07:44 cpu1
+drwxr-xr-x   3 root root 0 Sep 19 07:44 cpu2
+drwxr-xr-x   3 root root 0 Sep 19 07:44 cpu3
+drwxr-xr-x   3 root root 0 Sep 19 07:44 cpu4
+drwxr-xr-x   3 root root 0 Sep 19 07:44 cpu5
+drwxr-xr-x   3 root root 0 Sep 19 07:44 cpu6
+drwxr-xr-x   3 root root 0 Sep 19 07:48 cpu7
+
+Under each directory you would find an "online" file which is the control
+file to logically online/offline a processor.
+
+Q: Does hot-add/hot-remove refer to physical add/remove of cpus?
+A: The usage of hot-add/remove may not be very consistently used in the code.
+CONFIG_CPU_HOTPLUG enables logical online/offline capability in the kernel.
+To support physical addition/removal, one would need some BIOS hooks and
+the platform should have something like an attention button in PCI hotplug.
+CONFIG_ACPI_HOTPLUG_CPU enables ACPI support for physical add/remove of CPUs.
+
+Q: How do i logically offline a CPU?
+A: Do the following.
+
+#echo 0 > /sys/devices/system/cpu/cpuX/online
+
+once the logical offline is successful, check
+
+#cat /proc/interrupts
+
+you should now not see the CPU that you removed. Also online file will report
+the state as 0 when a cpu if offline and 1 when its online.
+
+#To display the current cpu state.
+#cat /sys/devices/system/cpu/cpuX/online
+
+Q: Why cant i remove CPU0 on some systems?
+A: Some architectures may have some special dependency on a certain CPU.
+
+For e.g in IA64 platforms we have ability to sent platform interrupts to the
+OS. a.k.a Corrected Platform Error Interrupts (CPEI). In current ACPI
+specifications, we didn't have a way to change the target CPU. Hence if the
+current ACPI version doesn't support such re-direction, we disable that CPU
+by making it not-removable.
+
+In such cases you will also notice that the online file is missing under cpu0.
+
+Q: How do i find out if a particular CPU is not removable?
+A: Depending on the implementation, some architectures may show this by the
+absence of the "online" file. This is done if it can be determined ahead of
+time that this CPU cannot be removed.
+
+In some situations, this can be a run time check, i.e if you try to remove the
+last CPU, this will not be permitted. You can find such failures by
+investigating the return value of the "echo" command.
+
+Q: What happens when a CPU is being logically offlined?
+A: The following happen, listed in no particular order :-)
+
+- A notification is sent to in-kernel registered modules by sending an event
+  CPU_DOWN_PREPARE
+- All process is migrated away from this outgoing CPU to a new CPU
+- All interrupts targeted to this CPU is migrated to a new CPU
+- timers/bottom half/task lets are also migrated to a new CPU
+- Once all services are migrated, kernel calls an arch specific routine
+  __cpu_disable() to perform arch specific cleanup.
+- Once this is successful, an event for successful cleanup is sent by an event
+  CPU_DEAD.
+
+  "It is expected that each service cleans up when the CPU_DOWN_PREPARE
+  notifier is called, when CPU_DEAD is called its expected there is nothing
+  running on behalf of this CPU that was offlined"
+
+Q: If i have some kernel code that needs to be aware of CPU arrival and
+   departure, how to i arrange for proper notification?
+A: This is what you would need in your kernel code to receive notifications.
+
+    #include <linux/cpu.h>
+	static int __cpuinit foobar_cpu_callback(struct notifier_block *nfb,
+								unsigned long action, void *hcpu)
+	{
+		unsigned int cpu = (unsigned long)hcpu;
+
+		switch (action) {
+		case CPU_ONLINE:
+			foobar_online_action(cpu);
+			break;
+		case CPU_DEAD:
+			foobar_dead_action(cpu);
+			break;
+		}
+		return NOTIFY_OK;
+	}
+
+	static struct notifier_block foobar_cpu_notifer =
+	{
+	   .notifier_call = foobar_cpu_callback,
+	};
+
+
+In your init function,
+
+	register_cpu_notifier(&foobar_cpu_notifier);
+
+You can fail PREPARE notifiers if something doesn't work to prepare resources.
+This will stop the activity and send a following CANCELED event back.
+
+CPU_DEAD should not be failed, its just a goodness indication, but bad
+things will happen if a notifier in path sent a BAD notify code.
+
+Q: I don't see my action being called for all CPUs already up and running?
+A: Yes, CPU notifiers are called only when new CPUs are on-lined or offlined.
+   If you need to perform some action for each cpu already in the system, then
+
+  for_each_online_cpu(i) {
+		foobar_cpu_callback(&foobar_cpu_notifier, CPU_UP_PREPARE, i);
+		foobar_cpu_callback(&foobar-cpu_notifier, CPU_ONLINE, i);
+  }
+
+Q: If i would like to develop cpu hotplug support for a new architecture,
+   what do i need at a minimum?
+A: The following are what is required for CPU hotplug infrastructure to work
+   correctly.
+
+    - Make sure you have an entry in Kconfig to enable CONFIG_HOTPLUG_CPU
+    - __cpu_up()        - Arch interface to bring up a CPU
+    - __cpu_disable()   - Arch interface to shutdown a CPU, no more interrupts
+                          can be handled by the kernel after the routine
+                          returns. Including local APIC timers etc are
+                          shutdown.
+     - __cpu_die()      - This actually supposed to ensure death of the CPU.
+                          Actually look at some example code in other arch
+                          that implement CPU hotplug. The processor is taken
+                          down from the idle() loop for that specific
+                          architecture. __cpu_die() typically waits for some
+                          per_cpu state to be set, to ensure the processor
+                          dead routine is called to be sure positively.
+
+Q: I need to ensure that a particular cpu is not removed when there is some
+   work specific to this cpu is in progress.
+A: First switch the current thread context to preferred cpu
+
+   int my_func_on_cpu(int cpu)
+   {
+       cpumask_t saved_mask, new_mask = CPU_MASK_NONE;
+       int curr_cpu, err = 0;
+
+       saved_mask = current->cpus_allowed;
+       cpu_set(cpu, new_mask);
+       err = set_cpus_allowed(current, new_mask);
+
+       if (err)
+           return err;
+
+       /*
+        * If we got scheduled out just after the return from
+        * set_cpus_allowed() before running the work, this ensures
+        * we stay locked.
+        */
+       curr_cpu = get_cpu();
+
+       if (curr_cpu != cpu) {
+	   err = -EAGAIN;
+           goto ret;
+       } else {
+       	   /*
+	    * Do work : But cant sleep, since get_cpu() disables preempt
+	    */
+       }
+    ret:
+    	put_cpu();
+	set_cpus_allowed(current, saved_mask);
+	return err;
+    }
+
+
+Q: How do we determine how many CPUs are available for hotplug.
+A: There is no clear spec defined way from ACPI that can give us that
+   information today. Based on some input from Natalie of Unisys,
+   that the ACPI MADT (Multiple APIC Description Tables) marks those possible
+   CPUs in a system with disabled status.
+
+   Andi implemented some simple heuristics that count the number of disabled
+   CPUs in MADT as hotpluggable CPUS.  In the case there are no disabled CPUS
+   we assume 1/2 the number of CPUs currently present can be hotplugged.
+
+   Caveat: Today's ACPI MADT can only provide 256 entries since the apicid field
+   in MADT is only 8 bits.
+
+User Space Notification
+
+TBD: