Skip to contentSkip to conversation

powerpc: kdump: Fix NULL pointer dereference in irq disable code

PATCH In review Anton Blanchard 2 participants 6 messages May 11, 2010 PowerPC

lore
Share & tools
lore
#0 lore raw

PATCH 1/3 powerpc: kdump: Fix NULL pointer dereference in irq disable code

With sparse irqs we have to check if we have a descriptor before dereferencing it.

---

arch/powerpc/kernel/crash.c +3 -0
index 6f4613d..5182439 100644
@@ -375,6 +375,9 @@ void default_machine_crash_shutdown(struct pt_regs *regs)
 	for_each_irq(i) {
 		struct irq_desc *desc = irq_to_desc(i);
 
+		if (!desc)
+			continue;
+
 		if (desc->status & IRQ_INPROGRESS)
 			desc->chip->eoi(i);
 
#1 lore raw

PATCH 2/3 powerpc: kdump: CPUs assume the context of the oopsing CPU

We wrap the crash_shutdown_handles[] calls with longjmp/setjmp, so if any of them fault we can recover. The problem is we add a hook to the debugger fault handler hook which calls longjmp unconditionally.

This first part of kdump is run before we marshall the other CPUs, so there is a very good chance some CPU on the box is going to page fault. And when it does it hits the longjmp code and assumes the context of the oopsing CPU. The machine gets very confused when it has 10 CPUs all with the same stack, all thinking they have the same CPU id. I get even more confused trying to debug it.

The patch below adds crash_shutdown_cpu and uses it to specify which cpu is in the protected region. Since it can only be -1 or the oopsing CPU, we don't need to use memory barriers since it is only valid on the local CPU - no other CPU will ever see a value that matches it's local CPU id.

Eventually we should switch the order and marshall all CPUs before doing the crash_shutdown_handles[] calls, but that is a bigger fix.

---

arch/powerpc/kernel/crash.c +5 -1
Index: linux-2.6/arch/powerpc/kernel/crash.c
===================================================================
@@ -281,10 +281,12 @@ int crash_shutdown_unregister(crash_shut
 EXPORT_SYMBOL(crash_shutdown_unregister);
 
 static unsigned long crash_shutdown_buf[JMP_BUF_LEN];
+static int crash_shutdown_cpu = -1;
 
 static int handle_fault(struct pt_regs *regs)
 {
-	longjmp(crash_shutdown_buf, 1);
+	if (crash_shutdown_cpu == smp_processor_id())
+		longjmp(crash_shutdown_buf, 1);
 	return 0;
 }
 
@@ -325,6 +327,7 @@ void default_machine_crash_shutdown(stru
 	 */
 	old_handler = __debugger_fault_handler;
 	__debugger_fault_handler = handle_fault;
+	crash_shutdown_cpu = smp_processor_id();
 	for (i = 0; crash_shutdown_handles[i]; i++) {
 		if (setjmp(crash_shutdown_buf) == 0) {
 			/*
@@ -338,6 +341,7 @@ void default_machine_crash_shutdown(stru
 			asm volatile("sync; isync");
 		}
 	}
+	crash_shutdown_cpu = -1;
 	__debugger_fault_handler = old_handler;
 
 	/*
#2 lore raw

PATCH 3/3 powerpc: kdump: Use chip->shutdown to disable IRQs

I saw this in a kdump kernel:

IOMMU table initialized, virtual merging enabled
Interrupt 155954 (real) is invalid, disabling it.
Interrupt 155953 (real) is invalid, disabling it.

ie we took some spurious interrupts. default_machine_crash_shutdown tries to disable all interrupt sources but uses chip->disable which maps to the default action of:

static void default_disable(unsigned int irq)
{
}

If we use chip->shutdown, then we actually mask the IRQ:

static void default_shutdown(unsigned int irq)

{
        struct irq_desc *desc = irq_to_desc(irq);

        desc->chip->mask(irq);
        desc->status |= IRQ_MASKED;
}

Not sure why we don't implement a ->disable action for xics.c, or why default_disable doesn't mask the interrupt.

---

arch/powerpc/kernel/crash.c +1 -1
Index: linux-2.6/arch/powerpc/kernel/crash.c
===================================================================
@@ -315,7 +315,7 @@ void default_machine_crash_shutdown(stru
 			desc->chip->eoi(i);
 
 		if (!(desc->status & IRQ_DISABLED))
-			desc->chip->disable(i);
+			desc->chip->shutdown(i);
 	}
 
 	/*
#3 lore raw

Re: [PATCH 3/3] powerpc: kdump: Use chip->shutdown to disable IRQs

Quoted 7 lines · Anton Blanchard

I saw this in a kdump kernel:

IOMMU table initialized, virtual merging enabled
Interrupt 155954 (real) is invalid, disabling it.
Interrupt 155953 (real) is invalid, disabling it.

ie we took some spurious interrupts.

OK, but it should have still worked OK?

Quoted 18 lines

default_machine_crash_shutdown tries
to disable all interrupt sources but uses chip->disable which maps to the default action of:

static void default_disable(unsigned int irq)
{
}

If we use chip->shutdown, then we actually mask the IRQ:

static void default_shutdown(unsigned int irq)

{
        struct irq_desc *desc = irq_to_desc(irq);

        desc->chip->mask(irq);
        desc->status |= IRQ_MASKED;
}
Quoted 2 lines

Not sure why we don't implement a ->disable action for xics.c, or why default_disable doesn't mask the interrupt.

It used to mask, see 76d21601, I knew that would bite us somewhere.

Not 100% sure about the change to use shutdown, but it's probably sane and you've tested it so cool :)

cheers

#4 lore raw

Re: [PATCH 2/3] powerpc: kdump: CPUs assume the context of the oopsing CPU

In reply to Anton Blanchard
Quoted 10 lines · Anton Blanchard

We wrap the crash_shutdown_handles[] calls with longjmp/setjmp, so if any of them fault we can recover. The problem is we add a hook to the debugger fault handler hook which calls longjmp unconditionally.

This first part of kdump is run before we marshall the other CPUs, so there is a very good chance some CPU on the box is going to page fault. And when it does it hits the longjmp code and assumes the context of the oopsing CPU. The machine gets very confused when it has 10 CPUs all with the same stack, all thinking they have the same CPU id. I get even more confused trying to debug it.

Lol, guess that one didn't get tested that well :)

Fix looks good.

cheers

#5 lore raw
In reply to Anton Blanchard
Quoted 11 lines · Anton Blanchard

With sparse irqs we have to check if we have a descriptor before dereferencing it.

---

arch/powerpc/kernel/crash.c +0 -0
index 6f4613d..5182439 100644

arch/powerpc/kernel/crash.c void default_machine_crash_shutdown(struct pt_regs *regs) · line 383

Quoted 9 lines
arch/powerpc/kernel/crash.c +3 -0
@@ -375,6 +375,9 @@ void default_machine_crash_shutdown(struct pt_regs *regs)
 	for_each_irq(i) {
 		struct irq_desc *desc = irq_to_desc(i);
 
+		if (!desc)
+			continue;
+
 		if (desc->status & IRQ_INPROGRESS)
 			desc->chip->eoi(i);
 

Ouch, my bad.

cheers