x86 generic hotplug changes for v3.8
Subsystems (4)
Linus,
Please pull the latest x86-bsp-hotplug-for-linus git tree from:
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git x86-bsp-hotplug-for-linus
HEAD: a71c8bc5dfefbbf80ef90739791554ef7ea4401b x86, topology: Debug CPU0 hotplug
This tree enables CPU#0 (the boot processor) to be onlined/offlined on x86, just like any other CPU. Enabled on Intel CPUs for now.
Allowing this required the identification and fixing of latent CPU#0 assumptions (such as CPU#0 initializations, etc.) in the x86 architecture code, plus the identification of barriers to BSP-offlining, such as active PIC interrupts which can only be serviced on the BSP.
It's behind a default-off option, and there's a debug option that allows the automatic testing of this feature.
The motivation of this feature is to allow and prepare for true CPU-hotplug hardware support: recent changes to MCE support enable us to detect a deteriorating but not yet hard-failing L1/L2 cache on a CPU that could be soft-unplugged - or a failing L3 cache on a multi-socket system.
Note that true hardware hot-plug is not yet fully enabled by this, because that requires a special platform wakeup sequence to be sent to the freshly powered up CPU#0. Future patches for this are planned, once such a platform exists. Chicken and egg
...
out-of-topic modifications in x86-bsp-hotplug-for-linus:
--------------------------------------------------------kernel/cpu.c # 6e32d47: kernel/cpu.c: Add comment for pri
Thanks,
Ingo
------------------>
Fenghua Yu (14):
doc: Add x86 CPU0 online/offline feature
x86, Kconfig: Add config switch for CPU0 hotplug
x86, topology: Don't offline CPU0 if any PIC irq can not be migrated out of it x86, hotplug: Support functions for CPU0 online/offline
x86, hotplug, suspend: Online CPU0 for suspend or hibernate kernel/cpu.c: Add comment for priority in cpu_hotplug_pm_callback x86-64, hotplug: Add start_cpu0() entry point to head_64.S x86-32, hotplug: Add start_cpu0() entry point to head_32.S
x86, hotplug: Wake up CPU0 via NMI instead of INIT, SIPI, SIPI x86, hotplug: During CPU0 online, enable x2apic, set_numa_node. x86, hotplug: The first online processor saves the MTRR state x86, hotplug: Handle retrigger irq by the first available CPU x86/i387.c: Initialize thread xstate only on CPU0 only once
x86, topology: Debug CPU0 hotplug
Documentation/cpu-hotplug.txt | 24 ++++++ Documentation/kernel-parameters.txt | 14 ++++ arch/x86/Kconfig | 44 +++++++++++ arch/x86/include/asm/cpu.h | 4 + arch/x86/include/asm/smp.h | 1 + arch/x86/kernel/apic/io_apic.c | 4 +- arch/x86/kernel/cpu/common.c | 5 +- arch/x86/kernel/cpu/mtrr/main.c | 9 ++- arch/x86/kernel/head_32.S | 13 ++++ arch/x86/kernel/head_64.S | 16 ++++ arch/x86/kernel/i387.c | 6 +- arch/x86/kernel/smpboot.c | 149 +++++++++++++++++++++++++++++------- arch/x86/kernel/topology.c | 101 ++++++++++++++++++++++-- arch/x86/power/cpu.c | 82 ++++++++++++++++++++ kernel/cpu.c | 5 ++ 15 files changed, 436 insertions(+), 41 deletions(-)
Documentation/cpu-hotplug.txt +24 -0
@@ -207,6 +207,30 @@ by making it not-removable. In such cases you will also notice that the online file is missing under cpu0. +Q: Is CPU0 removable on X86? +A: Yes. If kernel is compiled with CONFIG_BOOTPARAM_HOTPLUG_CPU0=y, CPU0 is +removable by default. Otherwise, CPU0 is also removable by kernel option +cpu0_hotplug. + +But some features depend on CPU0. Two known dependencies are: + +1. Resume from hibernate/suspend depends on CPU0. Hibernate/suspend will fail if +CPU0 is offline and you need to online CPU0 before hibernate/suspend can +continue. +2. PIC interrupts also depend on CPU0. CPU0 can't be removed if a PIC interrupt +is detected. + +It's said poweroff/reboot may depend on CPU0 on some machines although I haven't +seen any poweroff/reboot failure so far after CPU0 is offline on a few tested +machines. + +Please let me know if you know or see any other dependencies of CPU0. + +If the dependencies are under your control, you can turn on CPU0 hotplug feature +either by CONFIG_BOOTPARAM_HOTPLUG_CPU0 or by kernel parameter cpu0_hotplug. + +--Fenghua Yu <[email protected]> + Q: How do i find out if a particular CPU is not removable? A: Depending on the implementation, some architectures may show this by the absence of the "online" file. This is done if it can be determined ahead of
Documentation/kernel-parameters.txt +14 -0
@@ -1984,6 +1984,20 @@ bytes respectively. Such letter suffixes can also be entirely omitted. nox2apic [X86-64,APIC] Do not enable x2APIC mode. + cpu0_hotplug [X86] Turn on CPU0 hotplug feature when + CONFIG_BOOTPARAM_HOTPLUG_CPU0 is off. + Some features depend on CPU0. Known dependencies are: + 1. Resume from suspend/hibernate depends on CPU0. + Suspend/hibernate will fail if CPU0 is offline and you + need to online CPU0 before suspend/hibernate. + 2. PIC interrupts also depend on CPU0. CPU0 can't be + removed if a PIC interrupt is detected. + It's said poweroff/reboot may depend on CPU0 on some + machines although I haven't seen such issues so far + after CPU0 is offline on a few tested machines. + If the dependencies are under your control, you can + turn on cpu0_hotplug. + nptcg= [IA-64] Override max number of concurrent global TLB purges which is reported from either PAL_VM_SUMMARY or SAL PALO.
arch/x86/Kconfig +44 -0
50 lines not on this page · raw message
arch/x86/include/asm/cpu.h +4 -0
@@ -28,6 +28,10 @@ struct x86_cpu {#ifdef CONFIG_HOTPLUG_CPU extern int arch_register_cpu (int num);extern void arch_unregister_cpu (int ); +extern void __cpuinit start_cpu0 (void ); +#ifdef CONFIG_DEBUG_HOTPLUG_CPU0 +extern int _debug_hotplug_cpu (int cpu,int action); +#endif #endif DECLARE_PER_CPU (int, cpu_state);
arch/x86/include/asm/smp.h +1 -0
@@ -166,6 +166,7 @@ void native_send_call_func_ipi(const struct cpumask *mask);void native_send_call_func_single_ipi (int cpu);void x86_idle_thread_init (unsigned int cpu,struct task_struct *idle); +void smp_store_boot_cpu_info (void );void smp_store_cpu_info (int id);#define cpu_physical_id (cpu) per_cpu(x86_cpu_to_apicid, cpu)
arch/x86/kernel/apic/io_apic.c +3 -1
@@ -2199,9 +2199,11 @@ static int ioapic_retrigger_irq(struct irq_data *data) {struct irq_cfg *cfg = data->chip_data;unsigned long flags; +int cpu;raw_spin_lock_irqsave (&vector_lock, flags); - apic->send_IPI_mask (cpumask_of (cpumask_first (cfg->domain)), cfg->vector); + cpu =cpumask_first_and (cfg->domain, cpu_online_mask); + apic->send_IPI_mask (cpumask_of (cpu), cfg->vector);raw_spin_unlock_irqrestore (&vector_lock, flags);return 1 ;
arch/x86/kernel/cpu/common.c +2 -3
@@ -1237,7 +1237,7 @@ void __cpuinit cpu_init(void) oist = &per_cpu (orig_ist, cpu);#ifdef CONFIG_NUMA -if (cpu !=0 &&this_cpu_read (numa_node) ==0 && +if (this_cpu_read (numa_node) ==0 &&early_cpu_to_node (cpu) !=NUMA_NO_NODE )set_numa_node (early_cpu_to_node (cpu));#endif
@@ -1269,8 +1269,7 @@ void __cpuinit cpu_init(void)barrier ();x86_configure_nx (); -if (cpu !=0 ) -enable_x2apic (); +enable_x2apic ();/* * set up and load the per-CPU TSS
arch/x86/kernel/cpu/mtrr/main.c +7 -2
@@ -695,11 +695,16 @@ void mtrr_ap_init(void) }/** -* Save current fixed-range MTRR state of the BSP +* Save current fixed-range MTRR state of the first cpu in cpu_online_mask. */ void mtrr_save_state (void ) { -smp_call_function_single (0 , mtrr_save_fixed_ranges,NULL ,1 ); +int first_cpu; + +get_online_cpus (); + first_cpu =cpumask_first (cpu_online_mask); +smp_call_function_single (first_cpu, mtrr_save_fixed_ranges,NULL ,1 ); +put_online_cpus (); }void set_mtrr_aps_delayed_init (void )
arch/x86/kernel/head_32.S +13 -0
@@ -266,6 +266,19 @@ num_subarch_entries = (. - subarch_entries) / 4 jmp default_entry #endif /* CONFIG_PARAVIRT */ +#ifdef CONFIG_HOTPLUG_CPU +/* + * Boot CPU0 entry point. It's called from play_dead(). Everything has been set + * up already except stack. We just set up stack here. Then call + * start_secondary(). + */ +ENTRY(start_cpu0) + movl stack_start, %ecx + movl %ecx, %esp + jmp *(initial_code) +ENDPROC(start_cpu0) +#endif + /* * Non-boot CPU entry point; entered from trampoline.S * We can't lgdt here, because lgdt itself uses a data segment, but
arch/x86/kernel/head_64.S +16 -0
@@ -252,6 +252,22 @@ ENTRY(secondary_startup_64) pushq %rax # target address in negative space lretq +#ifdef CONFIG_HOTPLUG_CPU +/* + * Boot CPU0 entry point. It's called from play_dead(). Everything has been set + * up already except stack. We just set up stack here. Then call + * start_secondary(). + */ +ENTRY(start_cpu0) + movq stack_start(%rip),%rsp + movq initial_code(%rip),%rax + pushq $0 # fake return address to stop unwinder + pushq $__KERNEL_CS # set correct cs + pushq %rax # target address in negative space + lretq +ENDPROC(start_cpu0) +#endif + /* SMP bootup changes these two */ __REFDATA .align 8
arch/x86/kernel/i387.c +5 -1
@@ -175,7 +175,11 @@ void __cpuinit fpu_init(void) cr0 |=X86_CR0_EM ;write_cr0 (cr0); -if (!smp_processor_id ()) +/* +* init_thread_xstate is only called once to avoid overriding +* xstate_size during boot time or during CPU hotplug. +*/ +if (xstate_size ==0 )init_thread_xstate ();mxcsr_feature_mask_init ();
arch/x86/kernel/smpboot.c +122 -27
278 lines not on this page · raw message
arch/x86/kernel/topology.c +94 -7
117 lines not on this page · raw message
arch/x86/power/cpu.c +82 -0
91 lines not on this page · raw message
kernel/cpu.c +5 -0
@@ -601,6 +601,11 @@ cpu_hotplug_pm_callback(struct notifier_block *nb,static int __init cpu_hotplug_pm_sync_init (void ) { +/* +* cpu_hotplug_pm_callback has higher priority than x86 +* bsp_pm_callback which depends on cpu_hotplug_pm_callback +* to disable cpu hotplug to avoid cpu hotplug race. +*/ pm_notifier (cpu_hotplug_pm_callback,0 );return 0 ; }