i386 cpu hotplug bug - instant reboot when onlining secondary
Hi-
On a dual P3 Xeon machine, offlining and then onlining a cpu makes the box instantly reboot. I've been seeing this throughout the 2.6.16-rc series, but wasn't able to collect more information until now. Not sure when this last worked, unfortunately.
With the debugging patch below, I get this on serial console:
[17179681.704000] CPU 1 is now offline [17179686.908000] Booting processor 1/1 eip 3000 [17179686.912000] CPU 1 irqstacks, hard=78383000 soft=7837b000 [17179686.920000] Setting warm reset code and vector. [17179686.924000] 1. [17179686.924000] 2. [17179686.928000] 3. [17179686.928000] Asserting INIT. [17179686.932000] Waiting for send to finish... [17179686.936000] +<7>Deasserting INIT. [17179686.952000] Waiting for send to finish... [17179686.956000] +<7>#startup loops: 2. [17179686.960000] Sending STARTUP #1. [17179686.960000] After apic_write. [17179686.964000] Doing apic_write_around for target chip... [17179686.972000] Doing apic_write_around to kick the second...
Any suggestions?
arch/i386/kernel/smpboot.c +2 -0
@@ -801,10 +801,12 @@ wakeup_secondary_cpu(int phys_apicid, un*/ /* Target chip */ +Dprintk ("Doing apic_write_around for target chip...\n" );apic_write_around (APIC_ICR2 ,SET_APIC_DEST_FIELD (phys_apicid));/* Boot on the stack */ /* Kick the second */ +Dprintk ("Doing apic_write_around to kick the second...\n" );apic_write_around (APIC_ICR ,APIC_DM_STARTUP | (start_eip >>12 ));
include/asm-i386/apic.h +1 -1
@@ -8,7 +8,7 @@#include <asm/processor.h> #include <asm/system.h> -#define Dprintk(x...) +#define Dprintk(fmt,arg...) printk(KERN_DEBUG fmt,##arg)/* * Debugging macros
Quoted 6 lines · Nathan Lynch
On a dual P3 Xeon machine, offlining and then onlining a cpu makes the box instantly reboot. I've been seeing this throughout the 2.6.16-rc series, but wasn't able to collect more information until now. Not sure when this last worked, unfortunately.
With the debugging patch below, I get this on serial console:
Does 2.6.14 work? Also i wonder if it gets out of the trampoline...
Index: linux-2.6.16-rc2/arch/i386/kernel/smpboot.c
===================================================================
RCS file: /home/cvsroot/linux-2.6.16-rc2/arch/i386/kernel/smpboot.c,v retrieving revision 1.1.1.1
arch/i386/kernel/smpboot.c +1 -0
@@ -514,6 +514,7 @@ static void __devinit start_secondary(vocpu_init ();preempt_disable ();smp_callin (); +Dprintk ("startup_secondary\n" );while (!cpu_isset (smp_processor_id (), smp_commenced_mask))rep_nop ();setup_secondary_APIC_clock ();
Quoted 10 lines · Zwane Mwaikambo
Quoted 6 lines · Nathan Lynch
On a dual P3 Xeon machine, offlining and then onlining a cpu makes the box instantly reboot. I've been seeing this throughout the 2.6.16-rc series, but wasn't able to collect more information until now. Not sure when this last worked, unfortunately.
With the debugging patch below, I get this on serial console:
Does 2.6.14 work? Also i wonder if it gets out of the trampoline...
2.6.14 works (albeit with an APIC error reported). When retesting 2.6.16-rc4 with your patch on top of my debugging patch, I don't see the "startup_secondary" line:
[17179709.100000] CPU 1 is now offline [17179714.636000] Booting processor 1/1 eip 3000 [17179714.688000] CPU 1 irqstacks, hard=7837f000 soft=78377000 [17179714.756000] Setting warm reset code and vector. [17179714.812000] 1. [17179714.836000] 2. [17179714.860000] 3. [17179714.880000] Asserting INIT. [17179714.916000] Waiting for send to finish... [17179714.968000] +<7>Deasserting INIT. [17179715.024000] Waiting for send to finish... [17179715.072000] +<7>#startup loops: 2. [17179715.116000] Sending STARTUP #1. [17179715.160000] After apic_write. [17179715.196000] Doing apic_write_around for target chip... [17179715.260000] Doing apic_write_around to kick the second...
Quoted 16 lines
Index:
linux-2.6.16-rc2/arch/i386/kernel/smpboot.c
===================================================================
RCS file:/home/cvsroot/linux-2.6.16-rc2/arch/i386/kernel/smpboot.c,vretrieving revision 1.1.1.1
diff -u -p -B -r1.1.1.1smpboot.carch/i386/kernel/smpboot.c +1
-0@@ -514,6 +514,7 @@ static void __devinit start_secondary(vocpu_init ();preempt_disable ();smp_callin (); +Dprintk ("startup_secondary\n" );while (!cpu_isset (smp_processor_id (), smp_commenced_mask))rep_nop ();setup_secondary_APIC_clock ();
Quoted 8 lines · Nathan Lynch
Quoted 7 lines · Zwane Mwaikambo
Quoted 5 lines · Nathan Lynch
On a dual P3 Xeon machine, offlining and then onlining a cpu makes the box instantly reboot. I've been seeing this throughout the 2.6.16-rc series, but wasn't able to collect more information until now. Not sure when this last worked, unfortunately.
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
With the debugging patch below, I get this on serial console:
Does 2.6.14 work? Also i wonder if it gets out of the trampoline...
2.6.14 works (albeit with an APIC error reported). When retesting 2.6.16-rc4 with your patch on top of my debugging patch, I don't see the "
startup_secondary" line:
Hi Nathan,
Can you try the following patch? We can start moving the WARM_BOOT_HLT down until it triple faults (i'm assuming it at least gets this far).
Index: linux-2.6.16-rc2-mm1/arch/i386/kernel/head.S
===================================================================
RCS file: /home/cvsroot/linux-2.6.16-rc2-mm1/arch/i386/kernel/head.S,v retrieving revision 1.1.1.1
arch/i386/kernel/head.S +8 -0
@@ -146,6 +146,12 @@ page_pde_offset = (__PAGE_OFFSET >> 20); * we know the trampoline has already loaded the boot_gdt_table GDT * for us. */ +#define warm_boot tsc_sync_disabled-__PAGE_OFFSET +#define WARM_BOOT_HLT \ + cmpl $0, warm_boot; \ +10: \ + jne 10b + ENTRY(startup_32_smp) cld movl $(__BOOT_DS),%eax
@@ -168,6 +174,8 @@ ENTRY(startup_32_smp) * NOTE! We have to correct for the fact that we're * not yet offset PAGE_OFFSET.. */ + WARM_BOOT_HLT + #define cr4_bits mmu_cr4_features-__PAGE_OFFSET movl cr4_bits,%edx andl %edx,%edx
Index: linux-2.6.16-rc2-mm1/arch/i386/kernel/smpboot.c
===================================================================
RCS file: /home/cvsroot/linux-2.6.16-rc2-mm1/arch/i386/kernel/smpboot.c,v retrieving revision 1.1.1.1
arch/i386/kernel/smpboot.c +1 -1
@@ -102,7 +102,7 @@ static cpumask_t smp_commenced_mask;* is no way to resync one AP against BP. TBD: for prescott and above, we * should use IA64's algorithm */ -static int __devinitdata tsc_sync_disabled; +int __devinitdata tsc_sync_disabled;/* Per CPU bogomips and other parameters */ struct cpuinfo_x86 cpu_data[NR_CPUS] __cacheline_aligned;
Quoted 14 lines · Zwane Mwaikambo
Quoted 12 lines · Nathan Lynch
Quoted 10 lines · Zwane Mwaikambo
Quoted 6 lines · Nathan Lynch
On a dual P3 Xeon machine, offlining and then onlining a cpu makes the box instantly reboot. I've been seeing this throughout the 2.6.16-rc series, but wasn't able to collect more information until now. Not sure when this last worked, unfortunately.
With the debugging patch below, I get this on serial console:
Does 2.6.14 work? Also i wonder if it gets out of the trampoline...
Quoted 8 lines
Quoted 3 lines
2.6.14 works (albeit with an APIC error reported). When retesting 2.6.16-rc4 with your patch on top of my debugging patch, I don't see the "
startup_secondary" line:Hi Nathan,
Can you try the following patch? We can start moving the
WARM_BOOT_HLTdown until it triple faults (i'm assuming it at least gets this far).
Here's what I got with this one on top of a day-old -git (all debugging patches still applied):
[17179725.020000] CPU 1 is now offline [17179730.900000] Booting processor 1/1 eip 3000 [17179730.952000] CPU 1 irqstacks, hard=7837f000 soft=78377000 [17179731.020000] Setting warm reset code and vector. [17179731.076000] 1. [17179731.100000] 2. [17179731.120000] 3. [17179731.144000] Asserting INIT. [17179731.180000] Waiting for send to finish... [17179731.232000] +<7>Deasserting INIT. [17179731.284000] Waiting for send to finish... [17179731.336000] +<7>#startup loops: 2. [17179731.380000] Sending STARTUP #1. [17179731.420000] After apic_write. [17179731.460000] Doing apic_write_around for target chip... [17179731.524000] Doing apic_write_around to kick the second... [17179731.592000] Startup point 1. [17179731.632000] Waiting for send to finish... [17179731.680000] +<7>Sending STARTUP #2. [17179731.728000] After apic_write. [17179731.768000] Doing apic_write_around for target chip... [17179731.832000] Doing apic_write_around to kick the second... [17179731.900000] Startup point 1. [17179731.936000] Waiting for send to finish... [17179731.988000] +<7>After Startup. [17179732.028000] Before Callout 1. [17179732.068000] After Callout 1. [17179737.080000] Stuck ?? [17179737.108000] Inquiring remote APIC #1... [17179737.156000] ... APIC #1 ID: 01000000 [17179737.204000] ... APIC #1 VERSION: 00040011 [17179737.256000] ... APIC #1 SPIV: 000000ff
Quoted 47 lines
Index:
linux-2.6.16-rc2-mm1/arch/i386/kernel/head.S
===================================================================
RCS file:/home/cvsroot/linux-2.6.16-rc2-mm1/arch/i386/kernel/head.S,vretrieving revision 1.1.1.1
diff -u -p -B -r1.1.1.1head.Sarch/i386/kernel/head.S +8
-0@@ -146,6 +146,12 @@ page_pde_offset = (__PAGE_OFFSET >> 20); * we know the trampoline has already loaded the boot_gdt_table GDT * for us. */ +#define warm_boot tsc_sync_disabled-__PAGE_OFFSET +#define WARM_BOOT_HLT \ + cmpl $0, warm_boot; \ +10: \ + jne 10b + ENTRY(startup_32_smp) cld movl $(__BOOT_DS),%eax@@ -168,6 +174,8 @@ ENTRY(startup_32_smp) * NOTE! We have to correct for the fact that we're * not yet offset PAGE_OFFSET.. */ + WARM_BOOT_HLT + #define cr4_bits mmu_cr4_features-__PAGE_OFFSET movl cr4_bits,%edx andl %edx,%edxIndex:
linux-2.6.16-rc2-mm1/arch/i386/kernel/smpboot.c
===================================================================
RCS file:/home/cvsroot/linux-2.6.16-rc2-mm1/arch/i386/kernel/smpboot.c,vretrieving revision 1.1.1.1
diff -u -p -B -r1.1.1.1smpboot.carch/i386/kernel/smpboot.c +1
-1@@ -102,7 +102,7 @@ static cpumask_t smp_commenced_mask;* is no way to resync one AP against BP. TBD: for prescott and above, we * should use IA64's algorithm */ -static int __devinitdata tsc_sync_disabled; +int __devinitdata tsc_sync_disabled;/* Per CPU bogomips and other parameters */ struct cpuinfo_x86 cpu_data[NR_CPUS] __cacheline_aligned;
Quoted 19 lines · Nathan Lynch
Quoted 18 lines · Zwane Mwaikambo
Quoted 15 lines · Nathan Lynch
Quoted 10 lines · Zwane Mwaikambo
Quoted 6 lines · Nathan Lynch
On a dual P3 Xeon machine, offlining and then onlining a cpu makes the box instantly reboot. I've been seeing this throughout the 2.6.16-rc series, but wasn't able to collect more information until now. Not sure when this last worked, unfortunately.
With the debugging patch below, I get this on serial console:
Does 2.6.14 work? Also i wonder if it gets out of the trampoline...
2.6.14 works (albeit with an APIC error reported). When retesting 2.6.16-rc4 with your patch on top of my debugging patch, I don't see the "
startup_secondary" line:
Quoted 7 lines
Quoted 4 lines
Hi Nathan,
Can you try the following patch? We can start moving the
WARM_BOOT_HLTdown until it triple faults (i'm assuming it at least gets this far).Here's what I got with this one on top of a day-old -git (all debugging patches still applied):
Looks good, how about the following
Index: linux-2.6.16-rc2-mm1/arch/i386/kernel/head.S
===================================================================
RCS file: /home/cvsroot/linux-2.6.16-rc2-mm1/arch/i386/kernel/head.S,v retrieving revision 1.1.1.1
arch/i386/kernel/head.S +7 -0
@@ -146,6 +146,12 @@ page_pde_offset = (__PAGE_OFFSET >> 20); * we know the trampoline has already loaded the boot_gdt_table GDT * for us. */ +#define warm_boot tsc_sync_disabled-__PAGE_OFFSET +#define WARM_BOOT_HLT \ + cmpl $0, warm_boot; \ +10: \ + jne 10b + ENTRY(startup_32_smp) cld movl $(__BOOT_DS),%eax
@@ -324,6 +330,7 @@ is386: movl $2,%ecx # set MP cmpb $0,%cl je 1f # the first CPU calls start_kernel # all other CPUs call initialize_secondary + WARM_BOOT_HLT call initialize_secondary jmp L6 1:
Index: linux-2.6.16-rc2-mm1/arch/i386/kernel/smpboot.c
===================================================================
RCS file: /home/cvsroot/linux-2.6.16-rc2-mm1/arch/i386/kernel/smpboot.c,v retrieving revision 1.1.1.1
arch/i386/kernel/smpboot.c +1 -1
@@ -102,7 +102,7 @@ static cpumask_t smp_commenced_mask;* is no way to resync one AP against BP. TBD: for prescott and above, we * should use IA64's algorithm */ -static int __devinitdata tsc_sync_disabled; +int __devinitdata tsc_sync_disabled;/* Per CPU bogomips and other parameters */ struct cpuinfo_x86 cpu_data[NR_CPUS] __cacheline_aligned;
Quoted 23 lines · Zwane Mwaikambo
Quoted 21 lines · Nathan Lynch
Quoted 20 lines · Zwane Mwaikambo
Quoted 8 lines · Nathan Lynch
Quoted 7 lines · Zwane Mwaikambo
Quoted 5 lines · Nathan Lynch
On a dual P3 Xeon machine, offlining and then onlining a cpu makes the box instantly reboot. I've been seeing this throughout the 2.6.16-rc series, but wasn't able to collect more information until now. Not sure when this last worked, unfortunately.
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
With the debugging patch below, I get this on serial console:
Does 2.6.14 work? Also i wonder if it gets out of the trampoline...
2.6.14 works (albeit with an APIC error reported). When retesting 2.6.16-rc4 with your patch on top of my debugging patch, I don't see the "
startup_secondary" line:Hi Nathan,
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Can you try the following patch? We can start moving the
WARM_BOOT_HLTdown until it triple faults (i'm assuming it at least gets this far).Here's what I got with this one on top of a day-old -git (all debugging patches still applied):
Looks good, how about the following
I now get:
[17179687.244000] CPU 1 is now offline [17179693.164000] Booting processor 1/1 eip 3000 [17179693.216000] CPU 1 irqstacks, hard=7837f000 soft=78377000 [17179693.284000] Setting warm reset code and vector. [17179693.340000] 1. [17179693.364000] 2. [17179693.388000] 3. [17179693.408000] Asserting INIT. [17179693.448000] Waiting for send to finish... [17179693.496000] +<7>Deasserting INIT. [17179693.552000] Waiting for send to finish... [17179693.600000] +<7>#startup loops: 2. [17179693.644000] Sending STARTUP #1. [17179693.688000] After apic_write. [17179693.724000] Doing apic_write_around for target chip... [17179693.788000] Doing apic_write_around to kick the second...
Quoted 44 lines
Index:
linux-2.6.16-rc2-mm1/arch/i386/kernel/head.S
===================================================================
RCS file:/home/cvsroot/linux-2.6.16-rc2-mm1/arch/i386/kernel/head.S,vretrieving revision 1.1.1.1
diff -u -p -B -r1.1.1.1head.Sarch/i386/kernel/head.S +7
-0@@ -146,6 +146,12 @@ page_pde_offset = (__PAGE_OFFSET >> 20); * we know the trampoline has already loaded the boot_gdt_table GDT * for us. */ +#define warm_boot tsc_sync_disabled-__PAGE_OFFSET +#define WARM_BOOT_HLT \ + cmpl $0, warm_boot; \ +10: \ + jne 10b + ENTRY(startup_32_smp) cld movl $(__BOOT_DS),%eax@@ -324,6 +330,7 @@ is386: movl $2,%ecx # set MP cmpb $0,%cl je 1f # the first CPU calls start_kernel # all other CPUs call initialize_secondary + WARM_BOOT_HLT call initialize_secondary jmp L6 1:Index:
linux-2.6.16-rc2-mm1/arch/i386/kernel/smpboot.c
===================================================================
RCS file:/home/cvsroot/linux-2.6.16-rc2-mm1/arch/i386/kernel/smpboot.c,vretrieving revision 1.1.1.1
diff -u -p -B -r1.1.1.1smpboot.carch/i386/kernel/smpboot.c +1
-1@@ -102,7 +102,7 @@ static cpumask_t smp_commenced_mask;* is no way to resync one AP against BP. TBD: for prescott and above, we * should use IA64's algorithm */ -static int __devinitdata tsc_sync_disabled; +int __devinitdata tsc_sync_disabled;/* Per CPU bogomips and other parameters */ struct cpuinfo_x86 cpu_data[NR_CPUS] __cacheline_aligned;
Quoted 17 lines · Nathan Lynch
[17179687.244000] CPU 1 is now offline [17179693.164000] Booting processor 1/1 eip 3000 [17179693.216000] CPU 1 irqstacks, hard=7837f000 soft=78377000 [17179693.284000] Setting warm reset code and vector. [17179693.340000] 1. [17179693.364000] 2. [17179693.388000] 3. [17179693.408000] Asserting INIT. [17179693.448000] Waiting for send to finish... [17179693.496000] +<7>Deasserting INIT. [17179693.552000] Waiting for send to finish... [17179693.600000] +<7>#startup loops: 2. [17179693.644000] Sending STARTUP #1. [17179693.688000] After apic_write. [17179693.724000] Doing apic_write_around for target chip... [17179693.788000] Doing apic_write_around to kick the second...
Ok, could you apply only the following patch?
Index: linux-2.6.16-rc2-mm1/arch/i386/kernel/smpboot.c
===================================================================
RCS file: /home/cvsroot/linux-2.6.16-rc2-mm1/arch/i386/kernel/smpboot.c,v retrieving revision 1.1.1.1
arch/i386/kernel/smpboot.c +5 -0
@@ -535,9 +535,14 @@ static void __devinit start_secondary(vo* booting is too fragile that we want to limit the * things done here to the most necessary things. */ +Dprintk ("S1\n" );cpu_init (); +Dprintk ("S2\n" );preempt_disable (); +Dprintk ("S3\n" );smp_callin (); +Dprintk ("S4\n" ); +while (!cpu_isset (smp_processor_id (), smp_commenced_mask))rep_nop ();setup_secondary_APIC_clock ();
Quoted 20 lines · Zwane Mwaikambo
Quoted 17 lines · Nathan Lynch
[17179687.244000] CPU 1 is now offline [17179693.164000] Booting processor 1/1 eip 3000 [17179693.216000] CPU 1 irqstacks, hard=7837f000 soft=78377000 [17179693.284000] Setting warm reset code and vector. [17179693.340000] 1. [17179693.364000] 2. [17179693.388000] 3. [17179693.408000] Asserting INIT. [17179693.448000] Waiting for send to finish... [17179693.496000] +<7>Deasserting INIT. [17179693.552000] Waiting for send to finish... [17179693.600000] +<7>#startup loops: 2. [17179693.644000] Sending STARTUP #1. [17179693.688000] After apic_write. [17179693.724000] Doing apic_write_around for target chip... [17179693.788000] Doing apic_write_around to kick the second...
Quoted 1 line
Ok, could you apply only the following patch?
Sorry for the delay in getting back to you.
Applied your latest patch, (plus one-liner to make Dprintk actually print) -- I don't see any of the new print statements:
[17179687.744000] CPU 1 is now offline [17179693.032000] Booting processor 1/1 eip 3000 [17179693.084000] CPU 1 irqstacks, hard=783da000 soft=783d2000 [17179693.152000] Setting warm reset code and vector. [17179693.208000] 1. [17179693.232000] 2. [17179693.256000] 3. [17179693.276000] Asserting INIT. [17179693.316000] Waiting for send to finish... [17179693.364000] +<7>Deasserting INIT. [17179693.420000] Waiting for send to finish... [17179693.468000] +<7>#startup loops: 2. [17179693.512000] Sending STARTUP #1. [17179693.556000] After apic_write.
Quoted 23 lines
Index:
linux-2.6.16-rc2-mm1/arch/i386/kernel/smpboot.c
===================================================================
RCS file:/home/cvsroot/linux-2.6.16-rc2-mm1/arch/i386/kernel/smpboot.c,vretrieving revision 1.1.1.1
diff -u -p -B -r1.1.1.1smpboot.carch/i386/kernel/smpboot.c +5
-0@@ -535,9 +535,14 @@ static void __devinit start_secondary(vo* booting is too fragile that we want to limit the * things done here to the most necessary things. */ +Dprintk ("S1\n" );cpu_init (); +Dprintk ("S2\n" );preempt_disable (); +Dprintk ("S3\n" );smp_callin (); +Dprintk ("S4\n" ); +while (!cpu_isset (smp_processor_id (), smp_commenced_mask))rep_nop ();setup_secondary_APIC_clock ();