Skip to contentSkip to conversation

x86: used used_vectors in init_IRQ

PATCH In review Yinghai Lu 2 participants 10 messages Apr 15, 2009 x86

lore
Share & tools
lore

Yinghai Lu

author
#0 lore raw

PATCH 1/1 x86: used used_vectors in init_IRQ

Impact: fix panic

Yinghai found:

[  552.616646] general protection fault: 0403 [#1] SMP
[  552.620013] last sysfs file:
/sys/devices/pci0000:00/0000:00:02.0/usb1/1-1/1-1:1.0/host13/target13:0:0/13:0:0:0/block/sr0/size
[  552.620013] CPU 0
[  552.620013] Modules linked in:
[  552.620013] Pid: 0, comm: swapper Not tainted 2.6.30-rc1-tip-01931-g8fcafd8-dirty #28 Sun Fire X4440
[  552.620013] RIP: 0010:[<ffffffff8023bada>]  [<ffffffff8023bada>] default_idle+0x7d/0xda
[  552.620013] RSP: 0018:ffffffff81345e68  EFLAGS: 00010246
[  552.620013] RAX: 0000000000000000 RBX: ffffffff8133d870 RCX: ffffc20000000000
[  552.620013] RDX: 00000000001d0620 RSI: ffffffff8023bad8 RDI: ffffffff802a3169
[  552.620013] RBP: ffffffff81345e98 R08: 0000000000000000 R09: ffffffff812244a0
[  552.620013] R10: ffffffff81345dc8 R11: 7ebe1b6fa0bcac50 R12: 4ec4ec4ec4ec4ec5
[  552.620013] R13: ffffffff813a54d0 R14: ffffffff813a7a40 R15: 0000000000000000
[  552.620013] FS:  00000000006d1880(0000) GS:ffffc20000000000(0000) knlGS:0000000000000000
[  552.620013] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
[  552.620013] CR2: 00007fec9d936a50 CR3: 000000007d1a9000 CR4: 00000000000006e0
[  552.620013] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  552.620013] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[  552.620013] Process swapper (pid: 0, threadinfo ffffffff81344000,task ffffffff812244a0)
[  552.620013] Stack:
[  552.620013]  0000000000000000 ffffc20000000000 00000000001d0620 7ebe1b6fa0bcac50
[  552.620013]  ffffffff8133d870 4ec4ec4ec4ec4ec5 ffffffff81345ec8 ffffffff8023bd84
[  552.620013]  4ec4ec4ec4ec4ec5 ffffffff813a54d0 7ebe1b6fa0bcac50 ffffffff8133d870
[  552.620013] Call Trace:
[  552.620013]  [<ffffffff8023bd84>] c1e_idle+0x109/0x124
[  552.620013]  [<ffffffff8023314b>] cpu_idle+0xb8/0x101
[  552.620013]  [<ffffffff80c16d6a>] rest_init+0x7e/0x94
[  552.620013]  [<ffffffff81357efc>] start_kernel+0x3dc/0x3fd
[  552.620013]  [<ffffffff813572a9>] x86_64_start_reservations+0xb9/0xd4
[  552.620013]  [<ffffffff813573b2>] x86_64_start_kernel+0xee/0x109
[  552.620013] Code: 48 8b 04 25 f8 b4 00 00 83 a0 3c e0 ff ff fb 0f ae f0 65 48 8b 04 25 f8 b4 00 00 f6 80 38 e0 ff ff 08 75 09 e8 71 76 06 00 fb f4 <eb> 06 e8 68 76 06 00 fb 65 48 8b 04 25 f8 b4 00 00 83 88 3c e0
[  552.620013] RIP  [<ffffffff8023bada>] default_idle+0x7d/0xda
[  552.620013]  RSP <ffffffff81345e68>
[  552.828646] ---[ end trace 4cbfc5c01382af7f ]---

Joerg Roedel said
"The 0403 error code means that there was an external interrupt with vector 0x80. Yinghai, my theory is that the kernel on this machine has no 32bit emulation compiled in, right? In this case the selector points to a zero entry which may cause the #gpf right after the hlt.
But I have no idea where the external int 0x80 comes from"

it turns out that we could use 0x80 for external device with 64bit but 32bit emulation is disabled.

but we forget to set gate for it.

try to set gate for it by checking with used_vectors. also move apic_intr_init() early to avoid set that gate two times.

---
 arch/x86/kernel/irqinit.c |    8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
arch/x86/kernel/irqinit.c +4 -4
Index: linux-2.6/arch/x86/kernel/irqinit.c
===================================================================
@@ -240,19 +240,19 @@ void __init native_init_IRQ(void)
 	/* Execute any quirks before the call gates are initialised: */
 	x86_quirk_pre_intr_init();
 
+	apic_intr_init();
+
 	/*
 	 * Cover the whole vector space, no vector can escape
 	 * us. (some of these will be overridden and become
 	 * 'special' SMP interrupts)
 	 */
 	for (i = FIRST_EXTERNAL_VECTOR; i < NR_VECTORS; i++) {
-		/* IA32_SYSCALL_VECTOR was reserved in trap_init. */
-		if (i != IA32_SYSCALL_VECTOR)
+		/* IA32_SYSCALL_VECTOR could be used in trap_init already. */
+		if (!test_bit(i, used_vectors))
 			set_intr_gate(i, interrupt[i-FIRST_EXTERNAL_VECTOR]);
 	}
 
-	apic_intr_init();
-
 	if (!acpi_ioapic)
 		setup_irq(2, &irq2);
 
#1 lore raw
Quoted 1 line · Yinghai Lu

Impact: fix panic

thanks.

Quoted 2 lines
 arch/x86/kernel/irqinit.c |    8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

Note, i removed the -stable tag because i dont think .29 nor .30-rc2 is affected. Your patch to irqinit*.c (and the later unification by Pekka) is something still pending in the perfcounters and x86 trees, queued for .31.

Ingo

author
#2 lore raw
Quoted 5 lines · Ingo Molnar

Note, i removed the -stable tag because i dont think .29 nor .30-rc2 is affected. Your patch to irqinit*.c (and the later unification by Pekka) is something still pending in the perfcounters and x86 trees, queued for .31.

not sure .29, but it seems we need this for 2.6.30

or you want me to produce one against to linus tree?

YH

#3 lore raw
Quoted 10 lines · Yinghai Lu
Quoted 4 lines · Ingo Molnar

Note, i removed the -stable tag because i dont think .29 nor .30-rc2 is affected. Your patch to irqinit*.c (and the later unification by Pekka) is something still pending in the perfcounters and x86 trees, queued for .31.

not sure .29, but it seems we need this for 2.6.30

or you want me to produce one against to linus tree?

Yes, do one please.

Ingo

author
#4 lore raw
Quoted 10 lines · Ingo Molnar
Quoted 8 lines · Yinghai Lu
Quoted 4 lines · Ingo Molnar

Note, i removed the -stable tag because i dont think .29 nor .30-rc2 is affected. Your patch to irqinit*.c (and the later unification by Pekka) is something still pending in the perfcounters and x86 trees, queued for .31.

not sure .29, but it seems we need this for 2.6.30

you are right...

in trap_init with linus tree

still have

#ifdef CONFIG_X86_64
        set_bit(IA32_SYSCALL_VECTOR, used_vectors);
#else
        set_bit(SYSCALL_VECTOR, used_vectors);

#endif

so one patch (merge irqinit_32/64) in tip for .31 expose that bug.

aka we don't back port ...

YH

#5 lore raw
Quoted 21 lines · Yinghai Lu
Quoted 10 lines · Ingo Molnar
Quoted 8 lines · Yinghai Lu
Quoted 4 lines · Ingo Molnar

Note, i removed the -stable tag because i dont think .29 nor .30-rc2 is affected. Your patch to irqinit*.c (and the later unification by Pekka) is something still pending in the perfcounters and x86 trees, queued for .31.

not sure .29, but it seems we need this for 2.6.30

you are right...

in trap_init with linus tree

still have

#ifdef CONFIG_X86_64
        set_bit(IA32_SYSCALL_VECTOR, used_vectors);
#else
        set_bit(SYSCALL_VECTOR, used_vectors);
Quoted 6 lines

#endif

so one patch (merge irqinit_32/64) in tip for .31 expose that bug.

aka we don't back port ...

ok, good!

Ingo

author
#6 lore raw
Quoted 25 lines · Ingo Molnar
Quoted 23 lines · Yinghai Lu
Quoted 10 lines · Ingo Molnar
Quoted 8 lines · Yinghai Lu
Quoted 4 lines · Ingo Molnar

Note, i removed the -stable tag because i dont think .29 nor .30-rc2 is affected. Your patch to irqinit*.c (and the later unification by Pekka) is something still pending in the perfcounters and x86 trees, queued for .31.

not sure .29, but it seems we need this for 2.6.30

you are right...

in trap_init with linus tree

still have

#ifdef CONFIG_X86_64
        set_bit(IA32_SYSCALL_VECTOR, used_vectors);
#else
        set_bit(SYSCALL_VECTOR, used_vectors);

#endif

Quoted 6 lines
Quoted 3 lines

so one patch (merge irqinit_32/64) in tip for .31 expose that bug.

aka we don't back port ...

ok, good!

for memo:
exposed by:
commit bb3f0b59ad005d2d2ecbbe9bd048eab6d1ecbd31

Author: Yinghai Lu <[email protected]>
Date:   Sun Jan 25 02:38:09 2009 -0800

x86: make irqinit_32.c more like irqinit_64.c, v2

Impact: cleanup

1. add smp_intr_init and apic_intr_init for 32bit, the same as 64bit
2. move the apic_intr_init() call before set gate with interrupt[i]
3. for 64bit, if ia32_emulation is not used, will make per_cpu to use 0x80 vector.

[ v2: should use !test_bit() instead of test_bit() with 32bit ]

arch/x86/kernel/traps.c +6 -9
index ed5aee5..d36a502 100644
@@ -979,8 +979,13 @@ void __init trap_init(void)
 #endif
        set_intr_gate(19, &simd_coprocessor_error);
 
+       /* Reserve all the builtin and the syscall vector: */
+       for (i = 0; i < FIRST_EXTERNAL_VECTOR; i++)
+               set_bit(i, used_vectors);
+
 #ifdef CONFIG_IA32_EMULATION
        set_system_intr_gate(IA32_SYSCALL_VECTOR, ia32_syscall);
+       set_bit(IA32_SYSCALL_VECTOR, used_vectors);
 #endif
 
 #ifdef CONFIG_X86_32
@@ -997,17 +1002,9 @@ void __init trap_init(void)
        }
 
        set_system_trap_gate(SYSCALL_VECTOR, &system_call);
-#endif
-
-       /* Reserve all the builtin and the syscall vector: */
-       for (i = 0; i < FIRST_EXTERNAL_VECTOR; i++)
-               set_bit(i, used_vectors);
-
-#ifdef CONFIG_X86_64
-       set_bit(IA32_SYSCALL_VECTOR, used_vectors);
-#else
        set_bit(SYSCALL_VECTOR, used_vectors);
 #endif
+
        /*
         * Should be a barrier for any external CPU state:
         */
author
#7 lore raw
In reply to Ingo Molnar

it is werid...

original patch is right, it seem git merge mess it up again.

commit bb3f0b59ad005d2d2ecbbe9bd048eab6d1ecbd31

Author: Yinghai Lu <[email protected]>
Date:   Sun Jan 25 02:38:09 2009 -0800

x86: make irqinit_32.c more like irqinit_64.c, v2

Impact: cleanup

1. add smp_intr_init and apic_intr_init for 32bit, the same as 64bit
2. move the apic_intr_init() call before set gate with interrupt[i]
3. for 64bit, if ia32_emulation is not used, will make per_cpu to use 0x80 vector.

[ v2: should use !test_bit() instead of test_bit() with 32bit ]

arch/x86/kernel/irqinit_32.c +33 -23
index c56496f..ddf3eb7 100644
@@ -120,28 +120,8 @@ int vector_used_by_percpu_irq(unsigned int vector)
 	return 0;
 }
 
-/* Overridden in paravirt.c */
-void init_IRQ(void) __attribute__((weak, alias("native_init_IRQ")));
-
-void __init native_init_IRQ(void)
+static void __init smp_intr_init(void)
 {
-	int i;
-
-	/* all the set up before the call gates are initialised */
-	pre_intr_init_hook();
-
-	/*
-	 * Cover the whole vector space, no vector can escape
-	 * us. (some of these will be overridden and become
-	 * 'special' SMP interrupts)
-	 */
-	for (i =  FIRST_EXTERNAL_VECTOR; i < NR_VECTORS; i++) {
-		/* SYSCALL_VECTOR was reserved in trap_init. */
-		if (i != SYSCALL_VECTOR)
-			set_intr_gate(i, interrupt[i-FIRST_EXTERNAL_VECTOR]);
-	}
-
-
 #if defined(CONFIG_X86_LOCAL_APIC) && defined(CONFIG_SMP)
 	/*
 	 * The reschedule interrupt is a CPU-to-CPU reschedule-helper
@@ -170,8 +150,13 @@ void __init native_init_IRQ(void)
 	set_intr_gate(IRQ_MOVE_CLEANUP_VECTOR, irq_move_cleanup_interrupt);
 	set_bit(IRQ_MOVE_CLEANUP_VECTOR, used_vectors);
 #endif
+}
 
+static void __init apic_intr_init(void)
+{
 #ifdef CONFIG_X86_LOCAL_APIC
+	smp_intr_init();
+
 	/* self generated IPI for local APIC timer */
 	alloc_intr_gate(LOCAL_TIMER_VECTOR, apic_timer_interrupt);
 
@@ -181,12 +166,37 @@ void __init native_init_IRQ(void)
 # ifdef CONFIG_PERF_COUNTERS
 	alloc_intr_gate(LOCAL_PERF_VECTOR, perf_counter_interrupt);
 # endif
-#endif
 
-#if defined(CONFIG_X86_LOCAL_APIC) && defined(CONFIG_X86_MCE_P4THERMAL)
+# ifdef CONFIG_X86_MCE_P4THERMAL
 	/* thermal monitor LVT interrupt */
 	alloc_intr_gate(THERMAL_APIC_VECTOR, thermal_interrupt);
+# endif
 #endif
+}
+
+/* Overridden in paravirt.c */
+void init_IRQ(void) __attribute__((weak, alias("native_init_IRQ")));
+
+void __init native_init_IRQ(void)
+{
+	int i;
+
+	/* all the set up before the call gates are initialised */
+	pre_intr_init_hook();
+
+	apic_intr_init();
+
+	/*
+	 * Cover the whole vector space, no vector can escape
+	 * us. (some of these will be overridden and become
+	 * 'special' SMP interrupts)
+	 */
+	for (i = 0; i < (NR_VECTORS - FIRST_EXTERNAL_VECTOR); i++) {
+		int vector = FIRST_EXTERNAL_VECTOR + i;
+		/* SYSCALL_VECTOR was reserved in trap_init. */
+		if (!test_bit(vector, used_vectors))
+			set_intr_gate(vector, interrupt[i]);
+	}
 
 	if (!acpi_ioapic)
 		setup_irq(2, &irq2);
arch/x86/kernel/irqinit_64.c +4 -3
index 6a71bfc..16e1fc6 100644
@@ -162,6 +162,9 @@ void __init native_init_IRQ(void)
 	int i;
 
 	init_ISA_irqs();
+
+	apic_intr_init();
+
 	/*
 	 * Cover the whole vector space, no vector can escape
 	 * us. (some of these will be overridden and become
@@ -169,12 +172,10 @@ void __init native_init_IRQ(void)
 	 */
 	for (i = 0; i < (NR_VECTORS - FIRST_EXTERNAL_VECTOR); i++) {
 		int vector = FIRST_EXTERNAL_VECTOR + i;
-		if (vector != IA32_SYSCALL_VECTOR)
+		if (!test_bit(vector, used_vectors))
 			set_intr_gate(vector, interrupt[i]);
 	}
 
-	apic_intr_init();
-
 	if (!acpi_ioapic)
 		setup_irq(2, &irq2);
 }
arch/x86/kernel/traps.c +6 -9
index ed5aee5..d36a502 100644
@@ -979,8 +979,13 @@ void __init trap_init(void)
 #endif
 	set_intr_gate(19, &simd_coprocessor_error);
 
+	/* Reserve all the builtin and the syscall vector: */
+	for (i = 0; i < FIRST_EXTERNAL_VECTOR; i++)
+		set_bit(i, used_vectors);
+
 #ifdef CONFIG_IA32_EMULATION
 	set_system_intr_gate(IA32_SYSCALL_VECTOR, ia32_syscall);
+	set_bit(IA32_SYSCALL_VECTOR, used_vectors);
 #endif
 
 #ifdef CONFIG_X86_32
@@ -997,17 +1002,9 @@ void __init trap_init(void)
 	}
 
 	set_system_trap_gate(SYSCALL_VECTOR, &system_call);
-#endif
-
-	/* Reserve all the builtin and the syscall vector: */
-	for (i = 0; i < FIRST_EXTERNAL_VECTOR; i++)
-		set_bit(i, used_vectors);
-
-#ifdef CONFIG_X86_64
-	set_bit(IA32_SYSCALL_VECTOR, used_vectors);
-#else
 	set_bit(SYSCALL_VECTOR, used_vectors);
 #endif
+
 	/*
 	 * Should be a barrier for any external CPU state:
 	 */
#8 lore raw
Quoted 3 lines · Yinghai Lu

it is werid...

original patch is right, it seem git merge mess it up again.

Yes - as i said it in this discussion, i had to do a few ugly conflict resolutions - and the bug might have been introduced there.

Ingo

#9 lore raw

[tip:x86/apic] x86: use used_vectors in init_IRQ()

In reply to Yinghai Lu
Commit-ID:  77857dc07247ed5fa700a197c96ef842d8dbebdf
Gitweb:     http://git.kernel.org/tip/77857dc07247ed5fa700a197c96ef842d8dbebdf
Author:     Yinghai Lu <[email protected]>
AuthorDate: Wed, 15 Apr 2009 11:57:01 -0700
Committer:  Ingo Molnar <[email protected]>
CommitDate: Wed, 15 Apr 2009 21:24:01 +0200

x86: use used_vectors in init_IRQ()

Impact: fix crash with many devices

I found this crash:

[  552.616646] general protection fault: 0403 [#1] SMP
[  552.620013] last sysfs file:
/sys/devices/pci0000:00/0000:00:02.0/usb1/1-1/1-1:1.0/host13/target13:0:0/13:0:0:0/block/sr0/size
[  552.620013] CPU 0
[  552.620013] Modules linked in:
[  552.620013] Pid: 0, comm: swapper Not tainted 2.6.30-rc1-tip-01931-g8fcafd8-dirty #28 Sun Fire X4440
[  552.620013] RIP: 0010:[<ffffffff8023bada>]  [<ffffffff8023bada>] default_idle+0x7d/0xda
[  552.620013] RSP: 0018:ffffffff81345e68  EFLAGS: 00010246
[  552.620013] RAX: 0000000000000000 RBX: ffffffff8133d870 RCX: ffffc20000000000
[  552.620013] RDX: 00000000001d0620 RSI: ffffffff8023bad8 RDI: ffffffff802a3169
[  552.620013] RBP: ffffffff81345e98 R08: 0000000000000000 R09: ffffffff812244a0
[  552.620013] R10: ffffffff81345dc8 R11: 7ebe1b6fa0bcac50 R12: 4ec4ec4ec4ec4ec5
[  552.620013] R13: ffffffff813a54d0 R14: ffffffff813a7a40 R15: 0000000000000000
[  552.620013] FS:  00000000006d1880(0000) GS:ffffc20000000000(0000) knlGS:0000000000000000
[  552.620013] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
[  552.620013] CR2: 00007fec9d936a50 CR3: 000000007d1a9000 CR4: 00000000000006e0
[  552.620013] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  552.620013] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[  552.620013] Process swapper (pid: 0, threadinfo ffffffff81344000,task ffffffff812244a0)
[  552.620013] Stack:
[  552.620013]  0000000000000000 ffffc20000000000 00000000001d0620 7ebe1b6fa0bcac50
[  552.620013]  ffffffff8133d870 4ec4ec4ec4ec4ec5 ffffffff81345ec8 ffffffff8023bd84
[  552.620013]  4ec4ec4ec4ec4ec5 ffffffff813a54d0 7ebe1b6fa0bcac50 ffffffff8133d870
[  552.620013] Call Trace:
[  552.620013]  [<ffffffff8023bd84>] c1e_idle+0x109/0x124
[  552.620013]  [<ffffffff8023314b>] cpu_idle+0xb8/0x101
[  552.620013]  [<ffffffff80c16d6a>] rest_init+0x7e/0x94
[  552.620013]  [<ffffffff81357efc>] start_kernel+0x3dc/0x3fd
[  552.620013]  [<ffffffff813572a9>] x86_64_start_reservations+0xb9/0xd4
[  552.620013]  [<ffffffff813573b2>] x86_64_start_kernel+0xee/0x109
[  552.620013] Code: 48 8b 04 25 f8 b4 00 00 83 a0 3c e0 ff ff fb 0f ae f0 65 48 8b 04 25 f8 b4 00 00 f6 80 38 e0 ff ff 08 75 09 e8 71 76 06 00 fb f4 <eb> 06 e8 68 76 06 00 fb 65 48 8b 04 25 f8 b4 00 00 83 88 3c e0
[  552.620013] RIP  [<ffffffff8023bada>] default_idle+0x7d/0xda
[  552.620013]  RSP <ffffffff81345e68>
[  552.828646] ---[ end trace 4cbfc5c01382af7f ]---

Joerg Roedel said
"The 0403 error code means that there was an external interrupt with vector 0x80. Yinghai, my theory is that the kernel on this machine has no 32bit emulation compiled in, right? In this case the selector points to a zero entry which may cause the #gpf right after the hlt.
But I have no idea where the external int 0x80 comes from"

it turns out that we could use 0x80 for external device on 64-bit when 32-bit emulation is disabled.

But we forgot to set the gate for it.

try to set gate for it by checking used_vectors.

Also move apic_intr_init() early to avoid setting
that gate two times.

Cc Andrew Morton <[email protected]>
Cc Joerg Roedel <[email protected]>

LKML-Reference: <[email protected]>

---
 arch/x86/kernel/irqinit.c |    8 ++++----
 1 files changed, 4 insertions(+), 4 deletions(-)
arch/x86/kernel/irqinit.c +4 -4
index b424c32..2e08b10 100644
@@ -240,19 +240,19 @@ void __init native_init_IRQ(void)
 	/* Execute any quirks before the call gates are initialised: */
 	x86_quirk_pre_intr_init();
 
+	apic_intr_init();
+
 	/*
 	 * Cover the whole vector space, no vector can escape
 	 * us. (some of these will be overridden and become
 	 * 'special' SMP interrupts)
 	 */
 	for (i = FIRST_EXTERNAL_VECTOR; i < NR_VECTORS; i++) {
-		/* IA32_SYSCALL_VECTOR was reserved in trap_init. */
-		if (i != IA32_SYSCALL_VECTOR)
+		/* IA32_SYSCALL_VECTOR could be used in trap_init already. */
+		if (!test_bit(i, used_vectors))
 			set_intr_gate(i, interrupt[i-FIRST_EXTERNAL_VECTOR]);
 	}
 
-	apic_intr_init();
-
 	if (!acpi_ioapic)
 		setup_irq(2, &irq2);