NUMA hotplug emulator
Subsystems (6)
Hi!
Quoted 8 lines
hotplug emulator: Abstract cpu register functions
Abstract function
arch_register_cpuandregister_cpu, move the implementation details to a sub function with prefix "__".each of the sub function has an extra parameter nid, it can be used to register CPU under a fake NUMA node, it is a reserved interface for cpu hotplug emulation (CPU PROBE/RELEASE) in x86.
I don't get it. CPU hotplug can already be tested using echo 0/1 > online, and that works on 386. How is this different?
It seems to add some numa magic. Why is it important?
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
RFC 5/7 NUMA hotplug emulator
hotplug emulator: Abstract cpu register functions
Abstract function arch_register_cpu and register_cpu, move the implementation details to a sub function with prefix "__".
each of the sub function has an extra parameter nid, it can be used to register CPU under a fake NUMA node, it is a reserved interface for cpu hotplug emulation (CPU PROBE/RELEASE) in x86.
---
arch/x86/kernel/topology.c +10 -1
@@ -34,7 +34,11 @@ static DEFINE_PER_CPU(struct x86_cpu, cpu_devices);#ifdef CONFIG_HOTPLUG_CPU -int __ref arch_register_cpu (int num) +/* +* Add nid(NUMA node id) as parameter for cpu hotplug emulation. It supports +* to register a CPU to any nodes. +*/ +static int __ref __arch_register_cpu (int num,int nid) {/* * CPU0 cannot be offlined due to several
@@ -50,6 +54,11 @@ int __ref arch_register_cpu(int num)return register_cpu (&per_cpu (cpu_devices, num).cpu, num); } + +int __ref arch_register_cpu (int num) +{ +return __arch_register_cpu (num,NUMA_NO_NODE ); +}EXPORT_SYMBOL (arch_register_cpu);void arch_unregister_cpu (int num)
drivers/base/cpu.c +17 -5
@@ -208,17 +208,20 @@ static ssize_t print_cpus_offline(struct sysdev_class *class, static SYSDEV_CLASS_ATTR(offline, 0444, print_cpus_offline, NULL);/* -* register_cpu - Setup a sysfs device for a CPU. +* __register_cpu -Initialize and register the CPU device. +* * @cpu - cpu->hotpluggable field set to 1 will generate a control file in * sysfs for this CPU. * @num - CPU number to use when creating the device. +* @nid - numa node id * -* Initialize and register the CPU device. +* We do not calculate nid by funciton cpu_to_node(), and change it as a +* parameter, it is an reserved interface for CPU hotplug emulation. */ -int __cpuinit register_cpu (struct cpu *cpu,int num) +static int __cpuinit __register_cpu (struct cpu *cpu,int num,int nid) {int error; - cpu->node_id =cpu_to_node (num); + cpu->node_id = nid; cpu->sysdev.id = num; cpu->sysdev.cls = &cpu_sysdev_class;
@@ -229,7 +232,7 @@ int __cpuinit register_cpu(struct cpu *cpu, int num)if (!error)per_cpu (cpu_sys_devices, num) = &cpu->sysdev;if (!error) -register_cpu_under_node (num,cpu_to_node (num)); +register_cpu_under_node (num, nid);#ifdef CONFIG_KEXEC if (!error)
@@ -238,6 +241,15 @@ int __cpuinit register_cpu(struct cpu *cpu, int num)return error; } +/* +* register_cpu - Setup a sysfs device for a CPU. +* Initialize and register the CPU device. +*/ +int __cpuinit register_cpu (struct cpu *cpu,int num) +{ +return __register_cpu (cpu, num,cpu_to_node (num)); +} +struct sys_device *get_cpu_sysdev (unsigned cpu) {if (cpu < nr_cpu_ids &&cpu_possible (cpu))
Signature · 2 lines
Thanks & Regards, Shaohui
hotplug emulator: support cpu probe/release in x86
Add cpu interface probe/release under sysfs for x86. User can use this interface to emulate the cpu hot-add process, it is for cpu hotplug test purpose. Add a kernel option CONFIG_ARCH_CPU_PROBE_RELEASE for this feature.
This interface provides a mechanism to emulate cpu hotplug with software
methods, it becomes possible to do cpu hotplug automation and stress
testing.
Directive:
*) Reserve CPU throu grub parameter like:
maxcpus=4
the rest CPUs will not be initiliazed.
*) Probe CPU
we can use the probe interface to hot-add new CPUs:
echo nid > /sys/devices/system/cpu/probe
*) Release a CPU
echo cpu > /sys/devices/system/cpu/release
A reserved CPU will be hot-added to the specified node.
1) nid == 0, the CPU will be added to the real node which the CPU should be in
2) nid != 0, add the CPU to node nid even through it is a fake node.
---
arch/x86/Kconfig +14 -0
@@ -1228,6 +1228,17 @@ config NODE_HOTPLUG_EMU N is the number of hidden nodes, size is the memory size per hidden node. This is only useful for debugging. +config ARCH_CPU_PROBE_RELEASE + def_bool y + bool "CPU hotplug emulation" + depends on NUMA_HOTPLUG_EMU + ---help--- + Enable cpu hotplug emulation. Reserve cpu with grub parameter + "maxcpus=N", where N is the initial CPU number, the rest physical + CPUs will not be initialized; there is a probe/release interface + is for cpu hot-add/hot-remove to specified node in software method. + This is for debuging and testing purpose + config NODES_SHIFT int "Maximum NUMA Nodes (as a power of 2)" if !MAXSMP range 1 10
@@ -1651,6 +1662,9 @@ config HOTPLUG_CPU ( Note: power management support will enable this option automatically on SMP systems. ) Say N if you want to disable CPU hotplug. +config ARCH_CPU_PROBE_RELEASE + def_bool y + depends on HOTPLUG_CPU config COMPAT_VDSO def_bool y
arch/x86/include/asm/cpu.h +3 -0
@@ -28,6 +28,9 @@ struct x86_cpu {#ifdef CONFIG_HOTPLUG_CPU extern int arch_register_cpu (int num);extern void arch_unregister_cpu (int ); +#ifdef CONFIG_ARCH_CPU_PROBE_RELEASE +extern int arch_register_cpu_emu (int num,int nid); +#endif #endif DECLARE_PER_CPU (int, cpu_state);
arch/x86/kernel/acpi/boot.c +36 -0
@@ -592,8 +592,44 @@ int __ref acpi_map_lsapic(acpi_handle handle, int *pcpu) }EXPORT_SYMBOL (acpi_map_lsapic); +#ifdef CONFIG_ARCH_CPU_PROBE_RELEASE +static void acpi_map_cpu2node_emu (int cpu,int physid,int nid) +{ +#ifdef CONFIG_ACPI_NUMA +#ifdef CONFIG_X86_64 + apicid_to_node[physid] = nid; +numa_set_node (cpu, nid); +#else /* CONFIG_X86_32 */ + apicid_2_node[physid] = nid; + cpu_to_node_map[cpu] = nid; +#endif +#endif +} + +static u16 cpu_to_apicid_saved[CONFIG_NR_CPUS ]; +int __ref acpi_map_lsapic_emu (int pcpu,int nid) +{ +/* backup cpu apicid to array cpu_to_apicid_saved */ +if (cpu_to_apicid_saved[pcpu] ==0 && +per_cpu (x86_cpu_to_apicid, pcpu) !=BAD_APICID ) + cpu_to_apicid_saved[pcpu] =per_cpu (x86_cpu_to_apicid, pcpu); + +per_cpu (x86_cpu_to_apicid, pcpu) = cpu_to_apicid_saved[pcpu]; +acpi_map_cpu2node_emu (pcpu,per_cpu (x86_cpu_to_apicid, pcpu), nid); + +return pcpu; +} +EXPORT_SYMBOL (acpi_map_lsapic_emu); +#endif +int acpi_unmap_lsapic (int cpu) { +#ifdef CONFIG_ARCH_CPU_PROBE_RELEASE +/* backup cpu apicid to array cpu_to_apicid_saved */ +if (cpu_to_apicid_saved[cpu] ==0 && +per_cpu (x86_cpu_to_apicid, cpu) !=BAD_APICID ) + cpu_to_apicid_saved[cpu] =per_cpu (x86_cpu_to_apicid, cpu); +#endif per_cpu (x86_cpu_to_apicid, cpu) =-1 ;set_cpu_present (cpu, false); num_processors--;
arch/x86/kernel/topology.c +110 -3
145 lines not on this page · raw message
arch/x86/mm/numa_64.c +14 -0
@@ -12,6 +12,7 @@#include <linux/module.h> #include <linux/nodemask.h> #include <linux/sched.h> +#include <linux/cpu.h> #include <asm/e820.h> #include <asm/proto.h>
@@ -889,6 +890,19 @@ void __init init_cpu_to_node(void) }#endif +#ifdef CONFIG_ARCH_CPU_PROBE_RELEASE +static __init int cpu_hpe_setup (char *opt) +{ +if (!opt) +return -EINVAL ; + +if (!strncmp (opt,"on" ,2 ) || !strncmp (opt,"1" ,1 )) + cpu_hpe_on =1 ; + +return 0 ; +} +early_param ("cpu_hpe" , cpu_hpe_setup); +#endif /* CONFIG_ARCH_CPU_PROBE_RELEASE */ void __cpuinit numa_set_node (int cpu,int node) {
drivers/acpi/processor_driver.c +9 -0
@@ -604,6 +604,14 @@ static int __cpuinit acpi_processor_add(struct acpi_device *device) goto err_free_cpumask; sysdev =get_cpu_sysdev (pr->id); +/* +* Reserve cpu for hotplug emulation, the reserved cpu can be hot-added +* throu the cpu probe interface. Return directly. +*/ +if (sysdev ==NULL ) { + goto out; + } +if (sysfs_create_link (&device->dev.kobj, &sysdev->kobj,"sysdev" )) { result = -EFAULT ; goto err_remove_fs;
@@ -643,6 +651,7 @@ static int __cpuinit acpi_processor_add(struct acpi_device *device) goto err_remove_sysfs; } +out:return 0 ; err_remove_sysfs:
drivers/base/cpu.c +20 -1
@@ -22,9 +22,15 @@ struct sysdev_class cpu_sysdev_class = { };EXPORT_SYMBOL (cpu_sysdev_class); -static DEFINE_PER_CPU(struct sys_device *, cpu_sys_devices); +DEFINE_PER_CPU(struct sys_device *, cpu_sys_devices);#ifdef CONFIG_HOTPLUG_CPU +/* +* cpu_hpe_on is a switch to enable/disable cpu hotplug emulation. it is +* disabled in default, we can enable it throu grub parameter cpu_hpe=on +*/ +int cpu_hpe_on; +static ssize_t show_online (struct sys_device *dev,struct sysdev_attribute *attr,char *buf) {
@@ -80,6 +86,7 @@ void unregister_cpu(struct cpu *cpu) }#ifdef CONFIG_ARCH_CPU_PROBE_RELEASE + static ssize_t cpu_probe_store(struct sysdev_class *class, struct sysdev_class_attribute *attr, const char *buf,
@@ -250,6 +257,18 @@ int __cpuinit register_cpu(struct cpu *cpu, int num)return __register_cpu (cpu, num,cpu_to_node (num)); } +/* +* Register cpu to the specified NUMA node +* +* emulated version of function register_cpu, but is more flexible. it supports +* an extra parameter nid, We can register a CPU to any specified node throu +* this function. +*/ +int __cpuinit register_cpu_emu (struct cpu *cpu,int num,int nid) +{ +return __register_cpu (cpu, num, nid); +} +struct sys_device *get_cpu_sysdev (unsigned cpu) {if (cpu < nr_cpu_ids &&cpu_possible (cpu))
include/linux/acpi.h +1 -0
@@ -102,6 +102,7 @@ void acpi_numa_arch_fixup(void);#ifdef CONFIG_ACPI_HOTPLUG_CPU /* Arch dependent functions for cpu hotplug support */ int acpi_map_lsapic (acpi_handle handle,int *pcpu); +int acpi_map_lsapic_emu (int pcpu,int nid);int acpi_unmap_lsapic (int cpu);#endif /* CONFIG_ACPI_HOTPLUG_CPU */
include/linux/cpu.h +5 -0
@@ -30,7 +30,10 @@ struct cpu {struct sys_device sysdev; }; +DECLARE_PER_CPU(struct sys_device *, cpu_sys_devices); +extern int register_cpu (struct cpu *cpu,int num); +extern int register_cpu_emu (struct cpu *cpu,int num,int nid);extern struct sys_device *get_cpu_sysdev (unsigned cpu);extern int cpu_add_sysdev_attr (struct sysdev_attribute *attr);
@@ -116,6 +119,7 @@ extern void put_online_cpus(void);#define register_hotcpu_notifier (nb) register_cpu_notifier(nb)#define unregister_hotcpu_notifier (nb) unregister_cpu_notifier(nb)int cpu_down (unsigned int cpu); +extern int cpu_hpe_on;#ifdef CONFIG_ARCH_CPU_PROBE_RELEASE extern void cpu_hotplug_driver_lock (void );
@@ -138,6 +142,7 @@ static inline void cpu_hotplug_driver_unlock(void)/* These aren't inline functions due to a GCC bug. */ #define register_hotcpu_notifier (nb) ({ (void)(nb); 0; })#define unregister_hotcpu_notifier (nb) ({ (void)(nb); }) +static int cpu_hpe_on;#endif /* CONFIG_HOTPLUG_CPU */ #ifdef CONFIG_PM_SLEEP_SMP
Quoted 6 lines · Shaohui Zheng
hotplug emulator: Abstract cpu register functions
Abstract function
arch_register_cpuandregister_cpu, move the implementation details to a sub function with prefix "__".
Quoted 6 lines
each of the sub function has an extra parameter nid, it can be used to register CPU under a fake NUMA node, it is a reserved interface for cpu hotplug emulation (CPU PROBE/RELEASE) in x86.
Signed-off-by Shaohui Zheng <[email protected]>Signed-off-by Haicheng Li <[email protected]>
I don't know anything about this, please don't Cc me on these patches. Given the very long Cc list, I'm certain many other developers you have included are not interested. Please focus on the relevant lists next time!
Signature · 7 lines
Jean Delvare -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 6 lines · Shaohui Zheng
hotplug emulator: Abstract cpu register functions
Abstract function
arch_register_cpuandregister_cpu, move the implementation details to a sub function with prefix "__".
Quoted 6 lines
each of the sub function has an extra parameter nid, it can be used to register CPU under a fake NUMA node, it is a reserved interface for cpu hotplug emulation (CPU PROBE/RELEASE) in x86.
Signed-off-by Shaohui Zheng <[email protected]>Signed-off-by Haicheng Li <[email protected]>
I don't know anything about this, please don't Cc me on these patches. Given the very long Cc list, I'm certain many other developers you have included are not interested. Please focus on the relevant lists next time!
Signature · 1 line
Jean Delvare
Hi!
Quoted 8 lines
hotplug emulator: Abstract cpu register functions
Abstract function
arch_register_cpuandregister_cpu, move the implementation details to a sub function with prefix "__".each of the sub function has an extra parameter nid, it can be used to register CPU under a fake NUMA node, it is a reserved interface for cpu hotplug emulation (CPU PROBE/RELEASE) in x86.
I don't get it. CPU hotplug can already be tested using echo 0/1 > online, and that works on 386. How is this different?
It seems to add some numa magic. Why is it important?
Signature · 8 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 7 lines · Pavel Machek
Hi!
Quoted 5 lines
hotplug emulator: Abstract cpu register functions
Abstract function
arch_register_cpuandregister_cpu, move the implementation details to a sub function with prefix "__".
Quoted 8 lines
Quoted 3 lines
each of the sub function has an extra parameter nid, it can be used to register CPU under a fake NUMA node, it is a reserved interface for cpu hotplug emulation (CPU PROBE/RELEASE) in x86.
I don't get it. CPU hotplug can already be tested using echo 0/1 > online, and that works on 386. How is this different?
It seems to add some numa magic. Why is it important?
My guess is that he wants to test the software surrounding NUMA on a non-NUMA (or different-NUMA) machine, perhaps in order to shake out bugs before the corresponding hardware is available.
Thanx, Paul
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 7 lines · Pavel Machek
Hi!
Quoted 5 lines
hotplug emulator: Abstract cpu register functions
Abstract function
arch_register_cpuandregister_cpu, move the implementation details to a sub function with prefix "__".
Quoted 8 lines
Quoted 3 lines
each of the sub function has an extra parameter nid, it can be used to register CPU under a fake NUMA node, it is a reserved interface for cpu hotplug emulation (CPU PROBE/RELEASE) in x86.
I don't get it. CPU hotplug can already be tested using echo 0/1 > online, and that works on 386. How is this different?
It seems to add some numa magic. Why is it important?
My guess is that he wants to test the software surrounding NUMA on a non-NUMA (or different-NUMA) machine, perhaps in order to shake out bugs before the corresponding hardware is available.
Thanx, Paul
Quoted 5 lines · Paul E. McKenney
Quoted 4 lines · Pavel Machek
Hi!
Quoted 2 lines
hotplug emulator: Abstract cpu register functions
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
Abstract function
arch_register_cpuandregister_cpu, move the implementation details to a sub function with prefix "__".each of the sub function has an extra parameter nid, it can be used to register CPU under a fake NUMA node, it is a reserved interface for cpu hotplug emulation (CPU PROBE/RELEASE) in x86.
I don't get it. CPU hotplug can already be tested using echo 0/1 > online, and that works on 386. How is this different?
"echo 0/1 > online" is logical cpu online/offline. The emulator intends to emulate physical add/remove of cpus. They cover different code path.
You can get details of the terms via $KERN_SRC/Documentation/cpu-hotplug.txt.
Quoted 1 line
Quoted 1 line
It seems to add some numa magic. Why is it important?
In real world, numa affinity info of the cpus is required for physical cpu hotadd/remove , which finally affects related data structures and code path. Emulator need the ability to emulate it.
Quoted 3 lines
My guess is that he wants to test the software surrounding NUMA on a non-NUMA (or different-NUMA) machine, perhaps in order to shake out bugs before the corresponding hardware is available.
This is one of the purposes. Auto tests and debugging all can get benefits from such emulation.
-haicheng
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 5 lines · Paul E. McKenney
Quoted 4 lines · Pavel Machek
Hi!
Quoted 2 lines
hotplug emulator: Abstract cpu register functions
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
Abstract function
arch_register_cpuandregister_cpu, move the implementation details to a sub function with prefix "__".each of the sub function has an extra parameter nid, it can be used to register CPU under a fake NUMA node, it is a reserved interface for cpu hotplug emulation (CPU PROBE/RELEASE) in x86.
I don't get it. CPU hotplug can already be tested using echo 0/1 > online, and that works on 386. How is this different?
"echo 0/1 > online" is logical cpu online/offline. The emulator intends to emulate physical add/remove of cpus. They cover different code path.
You can get details of the terms via $KERN_SRC/Documentation/cpu-hotplug.txt.
Quoted 1 line
Quoted 1 line
It seems to add some numa magic. Why is it important?
In real world, numa affinity info of the cpus is required for physical cpu hotadd/remove , which finally affects related data structures and code path. Emulator need the ability to emulate it.
Quoted 3 lines
My guess is that he wants to test the software surrounding NUMA on a non-NUMA (or different-NUMA) machine, perhaps in order to shake out bugs before the corresponding hardware is available.
This is one of the purposes. Auto tests and debugging all can get benefits from such emulation.
-haicheng
Quoted 7 lines · Pavel Machek
Hi!
Quoted 5 lines
hotplug emulator: Abstract cpu register functions
Abstract function
arch_register_cpuandregister_cpu, move the implementation details to a sub function with prefix "__".
Quoted 8 lines
Quoted 3 lines
each of the sub function has an extra parameter nid, it can be used to register CPU under a fake NUMA node, it is a reserved interface for cpu hotplug emulation (CPU PROBE/RELEASE) in x86.
I don't get it. CPU hotplug can already be tested using echo 0/1 > online, and that works on 386. How is this different?
It seems to add some numa magic. Why is it important?
Pavel,
it is not an easy thing to understand the full story since you may not work on this project
so you have such question. Let me do a simpe introductions about the background.
We need to understand 2 differnets concepts if you wnat to know the reason why we develop
the hotplug emulaor.
- CPU logcial online/offline
it is the existed feature which you mentioned, we can online/offline CPUs throu sysfs
interface /sys/device/system/cpu/cpuX/online (X is an integer, it stands for the CPU number)
echo 0/1 > /sys/device/cpu/cpuX/online
This is is logical CPU online/offline, when we do such operation, the CPU is already pluged
into the motherboard, and the OS initialized the CPU. the data structure and CPU entries on sysfs are created, the CPU present mask and possible mask are setted, it does not refer to any physical hardware. the CPU status becomes online from offline, and ready to schedule to run process by scheduler.
CPU online/offline is control by the kernel option CONFIG_HOTPLUG_CPU.
- CPU hot-add/hot-remove
This is physical CPU hot-add/hot-remove into motherboard, without shutdown the machine, after
the hot-add operation, the new CPU will be powered on, and the OS recognize the new CPUs throu SCI interrupts, then OS intializes the new CPUs, create the related CPU structures, create sysfs entries
for the new CPUs. Once all done, the CPU is ready to logcial online.
The process to hot-add CPU:
1) Physical CPU hot-add to motherboard when after the machine is powered on
2) the BIOS send SCI interrupts to notice the OS
3) Linux hotplug handler parse the data from the acpi_handle data
4) hotplug handler initialize the CPU structure according the cpu ACPI data
Current situation:
1) Provides developers an envronment
Only very few hardware can support CPU hot-add/hot-remove, we need create an working environment for developers to write and debug hotplug code even through they do not has such hardward on hand. It is what NUMA hotplug emulator does exactly. Physcial hotplug emuator should be a better name.
We have 2 solutions to solve this problem, and this one is selected finally; if you want to know more about the solutions, we can continue to on this thread.
2) Offers an automation test inferface for Linux CPU hot-add/hot-remove code
Linux hot-add/hot-remove code has obvious bugs, but we do not see any automation test suite for it, even in LTP project(LTP has hotplug suite for logical CPU online/offline).
It is a know diffcult work to test physcial hot-add/hot-remove code in automation way, but the hotplug emualtor does a good job for it. We reproduce all the major hotlug bugs against the internal emulator v2 and v3.
We are sharing it to the community, wish more wisdoms and talents are included in it. We want to show an exmaple of software emualtion, and hopes more guys benifit from it, this is the purpose for this group patches.
PowerPC supporting
For ppc, it was added about half year ago by Nathan Fontenot, but x86 does not has such feature.
Thanks for lethal to mention it, we already did some researching about it, I will reply it in another thread.
commit 12633e803a2a556f6469e0933d08233d0844a2d9 Author: Nathan Fontenot <[email protected]> Date: Wed Nov 25 17:23:25 2009 +0000 commit 1a8061c46c46c960f715c597b9d279ea2ba42bd9 Author: Nathan Fontenot <[email protected]> Date: Tue Nov 24 21:13:32 2009 +0000
We inherit the name style from ppc, CPU hot-add/hot-remove is called CPU probe/release in kernel, it was control by kernel option CONFIG_ARCH_CPU_PROBE_RELEASE.
Signature · 8 lines
Thanks & Regards, Shaohui -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 7 lines · Pavel Machek
Hi!
Quoted 5 lines
hotplug emulator: Abstract cpu register functions
Abstract function
arch_register_cpuandregister_cpu, move the implementation details to a sub function with prefix "__".
Quoted 8 lines
Quoted 3 lines
each of the sub function has an extra parameter nid, it can be used to register CPU under a fake NUMA node, it is a reserved interface for cpu hotplug emulation (CPU PROBE/RELEASE) in x86.
I don't get it. CPU hotplug can already be tested using echo 0/1 > online, and that works on 386. How is this different?
It seems to add some numa magic. Why is it important?
Pavel,
it is not an easy thing to understand the full story since you may not work on this project
so you have such question. Let me do a simpe introductions about the background.
We need to understand 2 differnets concepts if you wnat to know the reason why we develop
the hotplug emulaor.
- CPU logcial online/offline
it is the existed feature which you mentioned, we can online/offline CPUs throu sysfs
interface /sys/device/system/cpu/cpuX/online (X is an integer, it stands for the CPU number)
echo 0/1 > /sys/device/cpu/cpuX/online
This is is logical CPU online/offline, when we do such operation, the CPU is already pluged
into the motherboard, and the OS initialized the CPU. the data structure and CPU entries on sysfs are created, the CPU present mask and possible mask are setted, it does not refer to any physical hardware. the CPU status becomes online from offline, and ready to schedule to run process by scheduler.
CPU online/offline is control by the kernel option CONFIG_HOTPLUG_CPU.
- CPU hot-add/hot-remove
This is physical CPU hot-add/hot-remove into motherboard, without shutdown the machine, after
the hot-add operation, the new CPU will be powered on, and the OS recognize the new CPUs throu SCI interrupts, then OS intializes the new CPUs, create the related CPU structures, create sysfs entries
for the new CPUs. Once all done, the CPU is ready to logcial online.
The process to hot-add CPU:
1) Physical CPU hot-add to motherboard when after the machine is powered on
2) the BIOS send SCI interrupts to notice the OS
3) Linux hotplug handler parse the data from the acpi_handle data
4) hotplug handler initialize the CPU structure according the cpu ACPI data
Current situation:
1) Provides developers an envronment
Only very few hardware can support CPU hot-add/hot-remove, we need create an working environment for developers to write and debug hotplug code even through they do not has such hardward on hand. It is what NUMA hotplug emulator does exactly. Physcial hotplug emuator should be a better name.
We have 2 solutions to solve this problem, and this one is selected finally; if you want to know more about the solutions, we can continue to on this thread.
2) Offers an automation test inferface for Linux CPU hot-add/hot-remove code
Linux hot-add/hot-remove code has obvious bugs, but we do not see any automation test suite for it, even in LTP project(LTP has hotplug suite for logical CPU online/offline).
It is a know diffcult work to test physcial hot-add/hot-remove code in automation way, but the hotplug emualtor does a good job for it. We reproduce all the major hotlug bugs against the internal emulator v2 and v3.
We are sharing it to the community, wish more wisdoms and talents are included in it. We want to show an exmaple of software emualtion, and hopes more guys benifit from it, this is the purpose for this group patches.
PowerPC supporting
For ppc, it was added about half year ago by Nathan Fontenot, but x86 does not has such feature.
Thanks for lethal to mention it, we already did some researching about it, I will reply it in another thread.
commit 12633e803a2a556f6469e0933d08233d0844a2d9 Author: Nathan Fontenot <[email protected]> Date: Wed Nov 25 17:23:25 2009 +0000 commit 1a8061c46c46c960f715c597b9d279ea2ba42bd9 Author: Nathan Fontenot <[email protected]> Date: Tue Nov 24 21:13:32 2009 +0000
We inherit the name style from ppc, CPU hot-add/hot-remove is called CPU probe/release in kernel, it was control by kernel option CONFIG_ARCH_CPU_PROBE_RELEASE.
Signature · 2 lines
Thanks & Regards, Shaohui
Quoted 7 lines · Pavel Machek
Hi!
Quoted 5 lines
hotplug emulator: Abstract cpu register functions
Abstract function
arch_register_cpuandregister_cpu, move the implementation details to a sub function with prefix "__".
Quoted 6 lines
Quoted 3 lines
each of the sub function has an extra parameter nid, it can be used to register CPU under a fake NUMA node, it is a reserved interface for cpu hotplug emulation (CPU PROBE/RELEASE) in x86.
I don't get it. CPU hotplug can already be tested using echo 0/1> online, and that works on 386. How is this different?
It tests a different code path.
Quoted 1 line
It seems to add some numa magic. Why is it important?
It tests memory and node hotadd too. Memory/node hotadd is a quite problematic feature and needs all the testing support it can get.
-Andi
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 7 lines · Pavel Machek
Hi!
Quoted 5 lines
hotplug emulator: Abstract cpu register functions
Abstract function
arch_register_cpuandregister_cpu, move the implementation details to a sub function with prefix "__".
Quoted 6 lines
Quoted 3 lines
each of the sub function has an extra parameter nid, it can be used to register CPU under a fake NUMA node, it is a reserved interface for cpu hotplug emulation (CPU PROBE/RELEASE) in x86.
I don't get it. CPU hotplug can already be tested using echo 0/1> online, and that works on 386. How is this different?
It tests a different code path.
Quoted 1 line
It seems to add some numa magic. Why is it important?
It tests memory and node hotadd too. Memory/node hotadd is a quite problematic feature and needs all the testing support it can get.
-Andi
Quoted 6 lines
PowerPC supporting
For ppc, it was added about half year ago by Nathan Fontenot, but x86 does not has such feature.
Thanks for lethal to mention it, we already did some researching about it, I will reply it in another thread.
Again the probe interface only covers part of the code, not ACPI for example.
-Andi
Quoted 6 lines
PowerPC supporting
For ppc, it was added about half year ago by Nathan Fontenot, but x86 does not has such feature.
Thanks for lethal to mention it, we already did some researching about it, I will reply it in another thread.
Again the probe interface only covers part of the code, not ACPI for example.
-Andi
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>