Skip to contentSkip to conversation

i386 cpu hotplug bug - instant reboot when onlining secondary

PATCH Nathan Lynch 2 participants 9 messages Feb 19, 2006 – Mar 6, 2006

lore
Share & tools
lore
#0 lore raw

Hi-

On a dual P3 Xeon machine, offlining and then onlining a cpu makes the box instantly reboot. I've been seeing this throughout the 2.6.16-rc series, but wasn't able to collect more information until now. Not sure when this last worked, unfortunately.

With the debugging patch below, I get this on serial console:

[17179681.704000] CPU 1 is now offline
[17179686.908000] Booting processor 1/1 eip 3000
[17179686.912000] CPU 1 irqstacks, hard=78383000 soft=7837b000
[17179686.920000] Setting warm reset code and vector.
[17179686.924000] 1.
[17179686.924000] 2.
[17179686.928000] 3.
[17179686.928000] Asserting INIT.
[17179686.932000] Waiting for send to finish...
[17179686.936000] +<7>Deasserting INIT.
[17179686.952000] Waiting for send to finish...
[17179686.956000] +<7>#startup loops: 2.
[17179686.960000] Sending STARTUP #1.
[17179686.960000] After apic_write.
[17179686.964000] Doing apic_write_around for target chip...
[17179686.972000] Doing apic_write_around to kick the second...

Any suggestions?

arch/i386/kernel/smpboot.c +2 -0
index fb00ab7..85aff00 100644
@@ -801,10 +801,12 @@ wakeup_secondary_cpu(int phys_apicid, un
 		 */
 
 		/* Target chip */
+		Dprintk("Doing apic_write_around for target chip...\n");
 		apic_write_around(APIC_ICR2, SET_APIC_DEST_FIELD(phys_apicid));
 
 		/* Boot on the stack */
 		/* Kick the second */
+		Dprintk("Doing apic_write_around to kick the second...\n");
 		apic_write_around(APIC_ICR, APIC_DM_STARTUP
 					| (start_eip >> 12));
 
include/asm-i386/apic.h +1 -1
index d30b857..2c8dcfa 100644
@@ -8,7 +8,7 @@
 #include <asm/processor.h>
 #include <asm/system.h>
 
-#define Dprintk(x...)
+#define Dprintk(fmt,arg...) printk(KERN_DEBUG fmt,##arg)
 
 /*
  * Debugging macros
#1 lore raw
Quoted 6 lines · Nathan Lynch

On a dual P3 Xeon machine, offlining and then onlining a cpu makes the box instantly reboot. I've been seeing this throughout the 2.6.16-rc series, but wasn't able to collect more information until now. Not sure when this last worked, unfortunately.

With the debugging patch below, I get this on serial console:

Does 2.6.14 work? Also i wonder if it gets out of the trampoline...

Index: linux-2.6.16-rc2/arch/i386/kernel/smpboot.c
===================================================================
RCS file: /home/cvsroot/linux-2.6.16-rc2/arch/i386/kernel/smpboot.c,v retrieving revision 1.1.1.1

arch/i386/kernel/smpboot.c +1 -0
diff -u -p -B -r1.1.1.1 smpboot.c
@@ -514,6 +514,7 @@ static void __devinit start_secondary(vo
 	cpu_init();
 	preempt_disable();
 	smp_callin();
+	Dprintk("startup_secondary\n");
 	while (!cpu_isset(smp_processor_id(), smp_commenced_mask))
 		rep_nop();
 	setup_secondary_APIC_clock();
#2 lore raw
Quoted 10 lines · Zwane Mwaikambo
Quoted 6 lines · Nathan Lynch

On a dual P3 Xeon machine, offlining and then onlining a cpu makes the box instantly reboot. I've been seeing this throughout the 2.6.16-rc series, but wasn't able to collect more information until now. Not sure when this last worked, unfortunately.

With the debugging patch below, I get this on serial console:

Does 2.6.14 work? Also i wonder if it gets out of the trampoline...

2.6.14 works (albeit with an APIC error reported). When retesting 2.6.16-rc4 with your patch on top of my debugging patch, I don't see the "startup_secondary" line:

[17179709.100000] CPU 1 is now offline
[17179714.636000] Booting processor 1/1 eip 3000
[17179714.688000] CPU 1 irqstacks, hard=7837f000 soft=78377000
[17179714.756000] Setting warm reset code and vector.
[17179714.812000] 1.
[17179714.836000] 2.
[17179714.860000] 3.
[17179714.880000] Asserting INIT.
[17179714.916000] Waiting for send to finish...
[17179714.968000] +<7>Deasserting INIT.
[17179715.024000] Waiting for send to finish...
[17179715.072000] +<7>#startup loops: 2.
[17179715.116000] Sending STARTUP #1.
[17179715.160000] After apic_write.
[17179715.196000] Doing apic_write_around for target chip...
[17179715.260000] Doing apic_write_around to kick the second...
Quoted 16 lines

Index: linux-2.6.16-rc2/arch/i386/kernel/smpboot.c
===================================================================
RCS file: /home/cvsroot/linux-2.6.16-rc2/arch/i386/kernel/smpboot.c,v retrieving revision 1.1.1.1
diff -u -p -B -r1.1.1.1 smpboot.c

arch/i386/kernel/smpboot.c +1 -0
@@ -514,6 +514,7 @@ static void __devinit start_secondary(vo
 	cpu_init();
 	preempt_disable();
 	smp_callin();
+	Dprintk("startup_secondary\n");
 	while (!cpu_isset(smp_processor_id(), smp_commenced_mask))
 		rep_nop();
 	setup_secondary_APIC_clock();
#3 lore raw
Quoted 8 lines · Nathan Lynch
Quoted 7 lines · Zwane Mwaikambo
Quoted 5 lines · Nathan Lynch

On a dual P3 Xeon machine, offlining and then onlining a cpu makes the box instantly reboot. I've been seeing this throughout the 2.6.16-rc series, but wasn't able to collect more information until now. Not sure when this last worked, unfortunately.

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

With the debugging patch below, I get this on serial console:

Does 2.6.14 work? Also i wonder if it gets out of the trampoline...

2.6.14 works (albeit with an APIC error reported). When retesting 2.6.16-rc4 with your patch on top of my debugging patch, I don't see the "startup_secondary" line:

Hi Nathan,

Can you try the following patch? We can start moving the WARM_BOOT_HLT down until it triple faults (i'm assuming it at least gets this far).

Index: linux-2.6.16-rc2-mm1/arch/i386/kernel/head.S
===================================================================
RCS file: /home/cvsroot/linux-2.6.16-rc2-mm1/arch/i386/kernel/head.S,v retrieving revision 1.1.1.1

arch/i386/kernel/head.S +8 -0
diff -u -p -B -r1.1.1.1 head.S
@@ -146,6 +146,12 @@ page_pde_offset = (__PAGE_OFFSET >> 20);
  * we know the trampoline has already loaded the boot_gdt_table GDT
  * for us.
  */
+#define warm_boot	tsc_sync_disabled-__PAGE_OFFSET
+#define WARM_BOOT_HLT		\
+	cmpl	$0, warm_boot;	\
+10:				\
+	jne	10b
+
 ENTRY(startup_32_smp)
 	cld
 	movl $(__BOOT_DS),%eax
@@ -168,6 +174,8 @@ ENTRY(startup_32_smp)
  *	NOTE! We have to correct for the fact that we're
  *	not yet offset PAGE_OFFSET..
  */
+	WARM_BOOT_HLT
+
 #define cr4_bits mmu_cr4_features-__PAGE_OFFSET
 	movl cr4_bits,%edx
 	andl %edx,%edx

Index: linux-2.6.16-rc2-mm1/arch/i386/kernel/smpboot.c
===================================================================
RCS file: /home/cvsroot/linux-2.6.16-rc2-mm1/arch/i386/kernel/smpboot.c,v retrieving revision 1.1.1.1

arch/i386/kernel/smpboot.c +1 -1
diff -u -p -B -r1.1.1.1 smpboot.c
@@ -102,7 +102,7 @@ static cpumask_t smp_commenced_mask;
  * is no way to resync one AP against BP. TBD: for prescott and above, we
  * should use IA64's algorithm
  */
-static int __devinitdata tsc_sync_disabled;
+int __devinitdata tsc_sync_disabled;
 
 /* Per CPU bogomips and other parameters */
 struct cpuinfo_x86 cpu_data[NR_CPUS] __cacheline_aligned;
#4 lore raw
Quoted 14 lines · Zwane Mwaikambo
Quoted 12 lines · Nathan Lynch
Quoted 10 lines · Zwane Mwaikambo
Quoted 6 lines · Nathan Lynch

On a dual P3 Xeon machine, offlining and then onlining a cpu makes the box instantly reboot. I've been seeing this throughout the 2.6.16-rc series, but wasn't able to collect more information until now. Not sure when this last worked, unfortunately.

With the debugging patch below, I get this on serial console:

Does 2.6.14 work? Also i wonder if it gets out of the trampoline...

Quoted 8 lines
Quoted 3 lines

2.6.14 works (albeit with an APIC error reported). When retesting 2.6.16-rc4 with your patch on top of my debugging patch, I don't see the "startup_secondary" line:

Hi Nathan,

Can you try the following patch? We can start moving the WARM_BOOT_HLT down until it triple faults (i'm assuming it at least gets this far).

Here's what I got with this one on top of a day-old -git (all debugging patches still applied):

[17179725.020000] CPU 1 is now offline
[17179730.900000] Booting processor 1/1 eip 3000
[17179730.952000] CPU 1 irqstacks, hard=7837f000 soft=78377000
[17179731.020000] Setting warm reset code and vector.
[17179731.076000] 1.
[17179731.100000] 2.
[17179731.120000] 3.
[17179731.144000] Asserting INIT.
[17179731.180000] Waiting for send to finish...
[17179731.232000] +<7>Deasserting INIT.
[17179731.284000] Waiting for send to finish...
[17179731.336000] +<7>#startup loops: 2.
[17179731.380000] Sending STARTUP #1.
[17179731.420000] After apic_write.
[17179731.460000] Doing apic_write_around for target chip...
[17179731.524000] Doing apic_write_around to kick the second...
[17179731.592000] Startup point 1.
[17179731.632000] Waiting for send to finish...
[17179731.680000] +<7>Sending STARTUP #2.
[17179731.728000] After apic_write.
[17179731.768000] Doing apic_write_around for target chip...
[17179731.832000] Doing apic_write_around to kick the second...
[17179731.900000] Startup point 1.
[17179731.936000] Waiting for send to finish...
[17179731.988000] +<7>After Startup.
[17179732.028000] Before Callout 1.
[17179732.068000] After Callout 1.
[17179737.080000] Stuck ??
[17179737.108000] Inquiring remote APIC #1...
[17179737.156000] ... APIC #1 ID: 01000000
[17179737.204000] ... APIC #1 VERSION: 00040011
[17179737.256000] ... APIC #1 SPIV: 000000ff
Quoted 47 lines

Index: linux-2.6.16-rc2-mm1/arch/i386/kernel/head.S
===================================================================
RCS file: /home/cvsroot/linux-2.6.16-rc2-mm1/arch/i386/kernel/head.S,v retrieving revision 1.1.1.1
diff -u -p -B -r1.1.1.1 head.S

arch/i386/kernel/head.S +8 -0
@@ -146,6 +146,12 @@ page_pde_offset = (__PAGE_OFFSET >> 20);
  * we know the trampoline has already loaded the boot_gdt_table GDT
  * for us.
  */
+#define warm_boot	tsc_sync_disabled-__PAGE_OFFSET
+#define WARM_BOOT_HLT		\
+	cmpl	$0, warm_boot;	\
+10:				\
+	jne	10b
+
 ENTRY(startup_32_smp)
 	cld
 	movl $(__BOOT_DS),%eax
@@ -168,6 +174,8 @@ ENTRY(startup_32_smp)
  *	NOTE! We have to correct for the fact that we're
  *	not yet offset PAGE_OFFSET..
  */
+	WARM_BOOT_HLT
+
 #define cr4_bits mmu_cr4_features-__PAGE_OFFSET
 	movl cr4_bits,%edx
 	andl %edx,%edx

Index: linux-2.6.16-rc2-mm1/arch/i386/kernel/smpboot.c
===================================================================
RCS file: /home/cvsroot/linux-2.6.16-rc2-mm1/arch/i386/kernel/smpboot.c,v retrieving revision 1.1.1.1
diff -u -p -B -r1.1.1.1 smpboot.c

arch/i386/kernel/smpboot.c +1 -1
@@ -102,7 +102,7 @@ static cpumask_t smp_commenced_mask;
  * is no way to resync one AP against BP. TBD: for prescott and above, we
  * should use IA64's algorithm
  */
-static int __devinitdata tsc_sync_disabled;
+int __devinitdata tsc_sync_disabled;
 
 /* Per CPU bogomips and other parameters */
 struct cpuinfo_x86 cpu_data[NR_CPUS] __cacheline_aligned;
#5 lore raw
Quoted 19 lines · Nathan Lynch
Quoted 18 lines · Zwane Mwaikambo
Quoted 15 lines · Nathan Lynch
Quoted 10 lines · Zwane Mwaikambo
Quoted 6 lines · Nathan Lynch

On a dual P3 Xeon machine, offlining and then onlining a cpu makes the box instantly reboot. I've been seeing this throughout the 2.6.16-rc series, but wasn't able to collect more information until now. Not sure when this last worked, unfortunately.

With the debugging patch below, I get this on serial console:

Does 2.6.14 work? Also i wonder if it gets out of the trampoline...

2.6.14 works (albeit with an APIC error reported). When retesting 2.6.16-rc4 with your patch on top of my debugging patch, I don't see the "startup_secondary" line:

Quoted 7 lines
Quoted 4 lines

Hi Nathan,

Can you try the following patch? We can start moving the WARM_BOOT_HLT down until it triple faults (i'm assuming it at least gets this far).

Here's what I got with this one on top of a day-old -git (all debugging patches still applied):

Looks good, how about the following

Index: linux-2.6.16-rc2-mm1/arch/i386/kernel/head.S
===================================================================
RCS file: /home/cvsroot/linux-2.6.16-rc2-mm1/arch/i386/kernel/head.S,v retrieving revision 1.1.1.1

arch/i386/kernel/head.S +7 -0
diff -u -p -B -r1.1.1.1 head.S
@@ -146,6 +146,12 @@ page_pde_offset = (__PAGE_OFFSET >> 20);
  * we know the trampoline has already loaded the boot_gdt_table GDT
  * for us.
  */
+#define warm_boot	tsc_sync_disabled-__PAGE_OFFSET
+#define WARM_BOOT_HLT		\
+	cmpl	$0, warm_boot;	\
+10:				\
+	jne	10b
+
 ENTRY(startup_32_smp)
 	cld
 	movl $(__BOOT_DS),%eax
@@ -324,6 +330,7 @@ is386:	movl $2,%ecx		# set MP
 	cmpb $0,%cl
 	je 1f			# the first CPU calls start_kernel
 				# all other CPUs call initialize_secondary
+	WARM_BOOT_HLT
 	call initialize_secondary
 	jmp L6
 1:

Index: linux-2.6.16-rc2-mm1/arch/i386/kernel/smpboot.c
===================================================================
RCS file: /home/cvsroot/linux-2.6.16-rc2-mm1/arch/i386/kernel/smpboot.c,v retrieving revision 1.1.1.1

arch/i386/kernel/smpboot.c +1 -1
diff -u -p -B -r1.1.1.1 smpboot.c
@@ -102,7 +102,7 @@ static cpumask_t smp_commenced_mask;
  * is no way to resync one AP against BP. TBD: for prescott and above, we
  * should use IA64's algorithm
  */
-static int __devinitdata tsc_sync_disabled;
+int __devinitdata tsc_sync_disabled;
 
 /* Per CPU bogomips and other parameters */
 struct cpuinfo_x86 cpu_data[NR_CPUS] __cacheline_aligned;
#6 lore raw
Quoted 23 lines · Zwane Mwaikambo
Quoted 21 lines · Nathan Lynch
Quoted 20 lines · Zwane Mwaikambo
Quoted 8 lines · Nathan Lynch
Quoted 7 lines · Zwane Mwaikambo
Quoted 5 lines · Nathan Lynch

On a dual P3 Xeon machine, offlining and then onlining a cpu makes the box instantly reboot. I've been seeing this throughout the 2.6.16-rc series, but wasn't able to collect more information until now. Not sure when this last worked, unfortunately.

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

With the debugging patch below, I get this on serial console:

Does 2.6.14 work? Also i wonder if it gets out of the trampoline...

2.6.14 works (albeit with an APIC error reported). When retesting 2.6.16-rc4 with your patch on top of my debugging patch, I don't see the "startup_secondary" line:

Hi Nathan,

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

Can you try the following patch? We can start moving the WARM_BOOT_HLT down until it triple faults (i'm assuming it at least gets this far).

Here's what I got with this one on top of a day-old -git (all debugging patches still applied):

Looks good, how about the following

I now get:

[17179687.244000] CPU 1 is now offline
[17179693.164000] Booting processor 1/1 eip 3000
[17179693.216000] CPU 1 irqstacks, hard=7837f000 soft=78377000
[17179693.284000] Setting warm reset code and vector.
[17179693.340000] 1.
[17179693.364000] 2.
[17179693.388000] 3.
[17179693.408000] Asserting INIT.
[17179693.448000] Waiting for send to finish...
[17179693.496000] +<7>Deasserting INIT.
[17179693.552000] Waiting for send to finish...
[17179693.600000] +<7>#startup loops: 2.
[17179693.644000] Sending STARTUP #1.
[17179693.688000] After apic_write.
[17179693.724000] Doing apic_write_around for target chip...
[17179693.788000] Doing apic_write_around to kick the second...
Quoted 44 lines

Index: linux-2.6.16-rc2-mm1/arch/i386/kernel/head.S
===================================================================
RCS file: /home/cvsroot/linux-2.6.16-rc2-mm1/arch/i386/kernel/head.S,v retrieving revision 1.1.1.1
diff -u -p -B -r1.1.1.1 head.S

arch/i386/kernel/head.S +7 -0
@@ -146,6 +146,12 @@ page_pde_offset = (__PAGE_OFFSET >> 20);
  * we know the trampoline has already loaded the boot_gdt_table GDT
  * for us.
  */
+#define warm_boot	tsc_sync_disabled-__PAGE_OFFSET
+#define WARM_BOOT_HLT		\
+	cmpl	$0, warm_boot;	\
+10:				\
+	jne	10b
+
 ENTRY(startup_32_smp)
 	cld
 	movl $(__BOOT_DS),%eax
@@ -324,6 +330,7 @@ is386:	movl $2,%ecx		# set MP
 	cmpb $0,%cl
 	je 1f			# the first CPU calls start_kernel
 				# all other CPUs call initialize_secondary
+	WARM_BOOT_HLT
 	call initialize_secondary
 	jmp L6
 1:

Index: linux-2.6.16-rc2-mm1/arch/i386/kernel/smpboot.c
===================================================================
RCS file: /home/cvsroot/linux-2.6.16-rc2-mm1/arch/i386/kernel/smpboot.c,v retrieving revision 1.1.1.1
diff -u -p -B -r1.1.1.1 smpboot.c

arch/i386/kernel/smpboot.c +1 -1
@@ -102,7 +102,7 @@ static cpumask_t smp_commenced_mask;
  * is no way to resync one AP against BP. TBD: for prescott and above, we
  * should use IA64's algorithm
  */
-static int __devinitdata tsc_sync_disabled;
+int __devinitdata tsc_sync_disabled;
 
 /* Per CPU bogomips and other parameters */
 struct cpuinfo_x86 cpu_data[NR_CPUS] __cacheline_aligned;
#7 lore raw
Quoted 17 lines · Nathan Lynch
[17179687.244000] CPU 1 is now offline
[17179693.164000] Booting processor 1/1 eip 3000
[17179693.216000] CPU 1 irqstacks, hard=7837f000 soft=78377000
[17179693.284000] Setting warm reset code and vector.
[17179693.340000] 1.
[17179693.364000] 2.
[17179693.388000] 3.
[17179693.408000] Asserting INIT.
[17179693.448000] Waiting for send to finish...
[17179693.496000] +<7>Deasserting INIT.
[17179693.552000] Waiting for send to finish...
[17179693.600000] +<7>#startup loops: 2.
[17179693.644000] Sending STARTUP #1.
[17179693.688000] After apic_write.
[17179693.724000] Doing apic_write_around for target chip...
[17179693.788000] Doing apic_write_around to kick the second...

Ok, could you apply only the following patch?

Index: linux-2.6.16-rc2-mm1/arch/i386/kernel/smpboot.c
===================================================================
RCS file: /home/cvsroot/linux-2.6.16-rc2-mm1/arch/i386/kernel/smpboot.c,v retrieving revision 1.1.1.1

arch/i386/kernel/smpboot.c +5 -0
diff -u -p -B -r1.1.1.1 smpboot.c
@@ -535,9 +535,14 @@ static void __devinit start_secondary(vo
 	 * booting is too fragile that we want to limit the
 	 * things done here to the most necessary things.
 	 */
+	Dprintk("S1\n");
 	cpu_init();
+	Dprintk("S2\n");
 	preempt_disable();
+	Dprintk("S3\n");
 	smp_callin();
+	Dprintk("S4\n");
+
 	while (!cpu_isset(smp_processor_id(), smp_commenced_mask))
 		rep_nop();
 	setup_secondary_APIC_clock();
#8 lore raw
Quoted 20 lines · Zwane Mwaikambo
Quoted 17 lines · Nathan Lynch
[17179687.244000] CPU 1 is now offline
[17179693.164000] Booting processor 1/1 eip 3000
[17179693.216000] CPU 1 irqstacks, hard=7837f000 soft=78377000
[17179693.284000] Setting warm reset code and vector.
[17179693.340000] 1.
[17179693.364000] 2.
[17179693.388000] 3.
[17179693.408000] Asserting INIT.
[17179693.448000] Waiting for send to finish...
[17179693.496000] +<7>Deasserting INIT.
[17179693.552000] Waiting for send to finish...
[17179693.600000] +<7>#startup loops: 2.
[17179693.644000] Sending STARTUP #1.
[17179693.688000] After apic_write.
[17179693.724000] Doing apic_write_around for target chip...
[17179693.788000] Doing apic_write_around to kick the second...
Quoted 1 line

Ok, could you apply only the following patch?

Sorry for the delay in getting back to you.

Applied your latest patch, (plus one-liner to make Dprintk actually print) -- I don't see any of the new print statements:

[17179687.744000] CPU 1 is now offline
[17179693.032000] Booting processor 1/1 eip 3000
[17179693.084000] CPU 1 irqstacks, hard=783da000 soft=783d2000
[17179693.152000] Setting warm reset code and vector.
[17179693.208000] 1.
[17179693.232000] 2.
[17179693.256000] 3.
[17179693.276000] Asserting INIT.
[17179693.316000] Waiting for send to finish...
[17179693.364000] +<7>Deasserting INIT.
[17179693.420000] Waiting for send to finish...
[17179693.468000] +<7>#startup loops: 2.
[17179693.512000] Sending STARTUP #1.
[17179693.556000] After apic_write.
Quoted 23 lines

Index: linux-2.6.16-rc2-mm1/arch/i386/kernel/smpboot.c
===================================================================
RCS file: /home/cvsroot/linux-2.6.16-rc2-mm1/arch/i386/kernel/smpboot.c,v retrieving revision 1.1.1.1
diff -u -p -B -r1.1.1.1 smpboot.c

arch/i386/kernel/smpboot.c +5 -0
@@ -535,9 +535,14 @@ static void __devinit start_secondary(vo
 	 * booting is too fragile that we want to limit the
 	 * things done here to the most necessary things.
 	 */
+	Dprintk("S1\n");
 	cpu_init();
+	Dprintk("S2\n");
 	preempt_disable();
+	Dprintk("S3\n");
 	smp_callin();
+	Dprintk("S4\n");
+
 	while (!cpu_isset(smp_processor_id(), smp_commenced_mask))
 		rep_nop();
 	setup_secondary_APIC_clock();