Skip to contentSkip to conversation

x86: support for new UV apic

REGRESSION Applied Priit Laes 4 participants 9 messages Apr 25, 2008 – Apr 28, 2008 x86

lore
Share & tools
lore

Priit Laes

author
#0 lore raw

Hey,

I am getting two following warnings (probable one per each CPU core) using the latest Linus's git:

WARNING: at arch/x86/kernel/genapic_64.c:86 read_apic_id+0x30/0x62()
Modules linked in:

Pid: 1, comm: swapper Not tainted 2.6.25-03563-gf7288b1 #20

Call Trace:
 [<ffffffff8022f1e1>] warn_on_slowpath+0x51/0x63
 [<ffffffff80282e48>] sys_ioctl+0x2d/0x5d
 [<ffffffff805185ff>] _spin_lock+0xe/0x24
 [<ffffffff80227459>] task_rq_lock+0x3d/0x73
 [<ffffffff805133c3>] set_cpu_sibling_map+0x336/0x350
 [<ffffffff8021c1b8>] read_apic_id+0x30/0x62
 [<ffffffff806d921d>] verify_local_APIC+0x90/0x138
 [<ffffffff806d84b5>] native_smp_prepare_cpus+0x1f9/0x305
 [<ffffffff806ce7b1>] kernel_init+0x59/0x2d9
 [<ffffffff80518a26>] _spin_unlock_irq+0x11/0x2b
 [<ffffffff8020bf48>] child_rip+0xa/0x12
 [<ffffffff806ce758>] kernel_init+0x0/0x2d9
 [<ffffffff8020bf3e>] child_rip+0x0/0x12

---[ end trace ca143223eefdc828 ]---

Few hours of bisecting gave me following commit as a suspect

ac23d4ee3f84de33c16ed7e68f9adee2386e74fb is first bad commit commit ac23d4ee3f84de33c16ed7e68f9adee2386e74fb

Author: Jack Steiner <[email protected]>
Date:   Fri Mar 28 14:12:16 2008 -0500

x86: support for new UV apic

#1 lore raw
Quoted 8 lines · Priit Laes

Hey,

I am getting two following warnings (probable one per each CPU core) using the latest Linus's git:

WARNING: at arch/x86/kernel/genapic_64.c:86 read_apic_id+0x30/0x62()
Modules linked in:

Pid: 1, comm: swapper Not tainted 2.6.25-03563-gf7288b1 #20

thanks - could you check the patch below, does it fix the warnings?

Ingo

--------------->
Subject: x86: disable preemption in native_smp_prepare_cpus
From: Ingo Molnar <[email protected]>
Date: Fri Apr 18 11:07:10 CEST 2008

Priit Laes reported the following warning:

Call Trace:
 [<ffffffff8022f1e1>] warn_on_slowpath+0x51/0x63
 [<ffffffff80282e48>] sys_ioctl+0x2d/0x5d
 [<ffffffff805185ff>] _spin_lock+0xe/0x24
 [<ffffffff80227459>] task_rq_lock+0x3d/0x73
 [<ffffffff805133c3>] set_cpu_sibling_map+0x336/0x350
 [<ffffffff8021c1b8>] read_apic_id+0x30/0x62
 [<ffffffff806d921d>] verify_local_APIC+0x90/0x138
 [<ffffffff806d84b5>] native_smp_prepare_cpus+0x1f9/0x305
 [<ffffffff806ce7b1>] kernel_init+0x59/0x2d9
 [<ffffffff80518a26>] _spin_unlock_irq+0x11/0x2b
 [<ffffffff8020bf48>] child_rip+0xa/0x12
 [<ffffffff806ce758>] kernel_init+0x0/0x2d9
 [<ffffffff8020bf3e>] child_rip+0x0/0x12

fix this by generally disabling preemption in native_smp_prepare_cpus().

---
 arch/x86/kernel/smpboot.c |    5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
arch/x86/kernel/smpboot.c +4 -1
Index: linux-x86.q/arch/x86/kernel/smpboot.c
===================================================================
@@ -1181,6 +1181,7 @@ static void __init smp_cpu_index_default
  */
 void __init native_smp_prepare_cpus(unsigned int max_cpus)
 {
+	preempt_disable();
 	nmi_watchdog_default();
 	smp_cpu_index_default();
 	current_cpu_data = boot_cpu_data;
@@ -1197,7 +1198,7 @@ void __init native_smp_prepare_cpus(unsi
 	if (smp_sanity_check(max_cpus) < 0) {
 		printk(KERN_INFO "SMP disabled\n");
 		disable_smp();
-		return;
+		goto out;
 	}
 
 	preempt_disable();
@@ -1237,6 +1238,8 @@ void __init native_smp_prepare_cpus(unsi
 	printk(KERN_INFO "CPU%d: ", 0);
 	print_cpu_info(&cpu_data(0));
 	setup_boot_clock();
+out:
+	preempt_enable();
 }
 /*
  * Early setup to make printk work.
#2 lore raw
Quoted 4 lines · Ingo Molnar
Quoted 2 lines · Priit Laes

Hey,

Quoted 8 lines
Quoted 6 lines

I am getting two following warnings (probable one per each CPU core) using the latest Linus's git:

WARNING: at arch/x86/kernel/genapic_64.c:86 read_apic_id+0x30/0x62()
Modules linked in:

Pid: 1, comm: swapper Not tainted 2.6.25-03563-gf7288b1 #20

thanks - could you check the patch below, does it fix the warnings?

Hi Ingo ,

I got the same warning on my Core2Quad Q9300 box , your patch fixes the warning for me.

Quoted 56 lines

Ingo

--------------->
Subject: x86: disable preemption in native_smp_prepare_cpus
From: Ingo Molnar <[email protected]>
Date: Fri Apr 18 11:07:10 CEST 2008

Priit Laes reported the following warning:

Call Trace:
 [<ffffffff8022f1e1>] warn_on_slowpath+0x51/0x63
 [<ffffffff80282e48>] sys_ioctl+0x2d/0x5d
 [<ffffffff805185ff>] _spin_lock+0xe/0x24
 [<ffffffff80227459>] task_rq_lock+0x3d/0x73
 [<ffffffff805133c3>] set_cpu_sibling_map+0x336/0x350
 [<ffffffff8021c1b8>] read_apic_id+0x30/0x62
 [<ffffffff806d921d>] verify_local_APIC+0x90/0x138
 [<ffffffff806d84b5>] native_smp_prepare_cpus+0x1f9/0x305
 [<ffffffff806ce7b1>] kernel_init+0x59/0x2d9
 [<ffffffff80518a26>] _spin_unlock_irq+0x11/0x2b
 [<ffffffff8020bf48>] child_rip+0xa/0x12
 [<ffffffff806ce758>] kernel_init+0x0/0x2d9
 [<ffffffff8020bf3e>] child_rip+0x0/0x12

fix this by generally disabling preemption in native_smp_prepare_cpus().

---
 arch/x86/kernel/smpboot.c |    5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

Index: linux-x86.q/arch/x86/kernel/smpboot.c
===================================================================

arch/x86/kernel/smpboot.c +2 -1
@@ -1181,6 +1181,7 @@ static void __init smp_cpu_index_default
  */
 void __init native_smp_prepare_cpus(unsigned int max_cpus)
 {
+	preempt_disable();
 	nmi_watchdog_default();
 	smp_cpu_index_default();
 	current_cpu_data = boot_cpu_data;
@@ -1197,7 +1198,7 @@ void __init native_smp_prepare_cpus(unsi
 	if (smp_sanity_check(max_cpus) < 0) {
 		printk(KERN_INFO "SMP disabled\n");
 		disable_smp();
-		return;
+		goto out;
 	}
 
 	preempt_disable();

arch/x86/kernel/smpboot.c void __init native_smp_prepare_cpus(unsi · line 1245

Quoted 8 lines
arch/x86/kernel/smpboot.c +2 -0
@@ -1237,6 +1238,8 @@ void __init native_smp_prepare_cpus(unsi
 	printk(KERN_INFO "CPU%d: ", 0);
 	print_cpu_info(&cpu_data(0));
 	setup_boot_clock();
+out:
+	preempt_enable();
 }
 /*
  * Early setup to make printk work.

Regards,

Gabriel

author
#3 lore raw
In reply to Ingo Molnar

Ühel kenal päeval, R, 2008-04-25 kell 08:44, kirjutas Ingo Molnar:

Quoted 12 lines · Priit Laes
Quoted 9 lines

Hey,

I am getting two following warnings (probable one per each CPU core) using the latest Linus's git:

WARNING: at arch/x86/kernel/genapic_64.c:86 read_apic_id+0x30/0x62()
Modules linked in:

Pid: 1, comm: swapper Not tainted 2.6.25-03563-gf7288b1 #20

thanks - could you check the patch below, does it fix the warnings?

Works for me!

Thanks :)

--
Priit Laes

#4 lore raw
In reply to Ingo Molnar
Quoted 4 lines · Ingo Molnar
Quoted 2 lines · Priit Laes

Hey,

Quoted 8 lines
Quoted 6 lines

I am getting two following warnings (probable one per each CPU core) using the latest Linus's git:

WARNING: at arch/x86/kernel/genapic_64.c:86 read_apic_id+0x30/0x62()
Modules linked in:

Pid: 1, comm: swapper Not tainted 2.6.25-03563-gf7288b1 #20

thanks - could you check the patch below, does it fix the warnings?

I think it would have been better to change the WARN_ON to

WARN_ON(preemptible() && num_online_cpus() > 1)

Preemption is actually ok here because this runs before the other CPUs are started

Alternative patch appended.

-Andi

---

Don't warn in read_apic_id() when preemptible but only one CPU online.

arch/x86/kernel/genapic_64.c +1 -1
Index: linux/arch/x86/kernel/genapic_64.c
===================================================================
@@ -83,7 +83,7 @@ unsigned int read_apic_id(void)
 {
 	unsigned int id;
 
-	WARN_ON(preemptible());
+	WARN_ON(preemptible() && num_online_cpus() > 1);
 	id = apic_read(APIC_ID);
 	if (uv_system_type >= UV_X2APIC)
 		id  |= __get_cpu_var(x2apic_extra_bits);
#5 lore raw
Quoted 9 lines · Andi Kleen
Quoted 7 lines · Ingo Molnar
Quoted 5 lines · Priit Laes

Hey,

I am getting two following warnings (probable one per each CPU core) using the latest Linus's git:

Quoted 8 lines
Quoted 4 lines
Quoted 3 lines
WARNING: at arch/x86/kernel/genapic_64.c:86 read_apic_id+0x30/0x62()
Modules linked in:

Pid: 1, comm: swapper Not tainted 2.6.25-03563-gf7288b1 #20

thanks - could you check the patch below, does it fix the warnings?

I think it would have been better to change the WARN_ON to

WARN_ON(preemptible() && num_online_cpus() > 1)

Hmm , I think you are right , Ingo's patch fixes the warning on boot but now I get one when I reboot the box.

Quoted 5 lines

Preemption is actually ok here because this runs before the other CPUs are started

Alternative patch appended.

I will test your patch in some minutes.

Quoted 2 lines

-Andi

Gabriel

Quoted 20 lines

---

Don't warn in read_apic_id() when preemptible but only one CPU online.

Index: linux/arch/x86/kernel/genapic_64.c
===================================================================

arch/x86/kernel/genapic_64.c +1 -1
@@ -83,7 +83,7 @@ unsigned int read_apic_id(void)
 {
 	unsigned int id;
 
-	WARN_ON(preemptible());
+	WARN_ON(preemptible() && num_online_cpus() > 1);
 	id = apic_read(APIC_ID);
 	if (uv_system_type >= UV_X2APIC)
 		id  |= __get_cpu_var(x2apic_extra_bits);
#6 lore raw
Quoted 18 lines · Gabriel C
Quoted 9 lines · Andi Kleen
Quoted 7 lines · Ingo Molnar
Quoted 5 lines · Priit Laes

Hey,

I am getting two following warnings (probable one per each CPU core) using the latest Linus's git:

Quoted 7 lines
Quoted 4 lines
Quoted 3 lines
WARNING: at arch/x86/kernel/genapic_64.c:86 read_apic_id+0x30/0x62()
Modules linked in:

Pid: 1, comm: swapper Not tainted 2.6.25-03563-gf7288b1 #20

thanks - could you check the patch below, does it fix the warnings?

I think it would have been better to change the WARN_ON to

WARN_ON(preemptible() && num_online_cpus() > 1)

Quoted 8 lines

Hmm , I think you are right , Ingo's patch fixes the warning on boot but now I get one when I reboot the box.

Quoted 4 lines

Preemption is actually ok here because this runs before the other CPUs are started

Alternative patch appended.

I will test your patch in some minutes.

Your patch works better , at least for my box , 4 reboots with your patch and boot/reboot warnings are all gone.

Quoted 20 lines
Quoted 18 lines

Don't warn in read_apic_id() when preemptible but only one CPU online.

Index: linux/arch/x86/kernel/genapic_64.c
===================================================================

arch/x86/kernel/genapic_64.c +1 -1
@@ -83,7 +83,7 @@ unsigned int read_apic_id(void)
 {
 	unsigned int id;
 
-	WARN_ON(preemptible());
+	WARN_ON(preemptible() && num_online_cpus() > 1);
 	id = apic_read(APIC_ID);
 	if (uv_system_type >= UV_X2APIC)
 		id  |= __get_cpu_var(x2apic_extra_bits);
author
#7 lore raw
In reply to Andi Kleen

Ühel kenal päeval, R, 2008-04-25 kell 11:45, kirjutas Andi Kleen:

Quoted 15 lines
Quoted 4 lines · Ingo Molnar
Quoted 2 lines · Priit Laes

Hey,

Quoted 8 lines
Quoted 6 lines

I am getting two following warnings (probable one per each CPU core) using the latest Linus's git:

WARNING: at arch/x86/kernel/genapic_64.c:86 read_apic_id+0x30/0x62()
Modules linked in:

Pid: 1, comm: swapper Not tainted 2.6.25-03563-gf7288b1 #20

thanks - could you check the patch below, does it fix the warnings?

Quoted 8 lines

I think it would have been better to change the WARN_ON to

WARN_ON(preemptible() && num_online_cpus() > 1)

Preemption is actually ok here because this runs before the other CPUs are started

Alternative patch appended.

Worksforme :)

--
Priit Laes

Applied
#8 lore raw
In reply to Andi Kleen
Quoted 3 lines · Andi Kleen

Alternative patch appended.

Don't warn in read_apic_id() when preemptible but only one CPU online.

applied, thanks Andi.

Ingo