Skip to contentSkip to conversation

Kdump: Disabling PCI interrupts in capture kernel

RFC/PATCH Vivek Goyal 3 participants 9 messages Jun 7, 2005 – Jun 8, 2005

lore
Share & tools
lore
#0 lore raw

Quoting Alan Stern <[email protected]>:

Quoted 27 lines · Vivek Goyal
Quoted 3 lines

Hi Alan, I know very little about consoles and their working. I had a question. Even if console is being managed by platform firmware,

in

Quoted 2 lines

initial states of booting, does it require interrupts to be enabled at VGA contorller (at least for the simple text mode). I was quickly

browsing

Quoted 2 lines

through drivers/video/console/vgacon.c and did not look like that this console driver needed interrupts to be enabled at the controller.

This isn't an issue for VGA, as far as I know. It applies to architectures like PPC-64 and perhaps Alpha or PA-Risc. And I don't know the details; ask Grant Grundler.

Quoted 1 line

Anyway, looks like serial consoles will always work. So at least this can

be

Quoted 2 lines

done for kdump case (CONFIG_CRASH_DUMP) and not generic kernel. Or, as I mentioned in previous mail, while pre-loading capture kernel, pass a

command

Quoted 3 lines

line parameter containing pci dev id of console and capture kernel does not

disable interrupts on this console.

I suspect you're right that implementing this only in kdump kernels will work okay.

Quoted 6 lines

For people interesting in reading some old threads on the subject, here are some pointers:

http://marc.theaimsgroup.com/?l=linux-usb-devel&m=111055702309788&w=2

http://marc.theaimsgroup.com/?l=linux-kernel&m=98383052711171&w=2

I browsed through the discussion threads quickly. Previous proposal included disabling the DMA as well from the devices. Currently, for kdump, we are not looking at disabling the DMA from the devices. So far have not run into any problems due to ongoing DMA (Need to look into IOMMU reprogramming aspect though). Following is a snippet from one of the discussion threads.

|List:       linux-usb-devel
|Subject:    [linux-usb-devel] Re: PCI device initialization and USB
early-handoff
|From:       Grant Grundler <grundler () parisc-linux ! org>
|Date:       2005-03-11 18:12:57
|Message-ID: <20050311181257.GB15070 () colo ! lackof ! org>
|[Download message RAW]

.....

|> > Is it feasible to have the PCI device initialization sequence disable DMA
|> > and IRQs from the device?  This could solve the problems we've been seeing
|> > with non-quiescent devices sharing an IRQ line at startup.
|two potential issues here:
|o ISTR VGA devices may not like disabling Bus Master bit in the command reg.
| But I'm blissfully ignorant of all the issues around VGA and someone
| else will have to comment on that.
|
|o platform devices (e.g. bridges) that don't have PCI drivers to re-enable
|  them later. "transperent" Bridges are the only example I can come up with
|  now but expect more to come out of the woodwork as this gets widely
|  tested. Trolling through PCI quirks might flag some of the known ones.
|  I would expect a few more to show up with this change.

|hth,
|grant

o Bus Master bit is not being disabled, only interrupt generation will be disabled and looks like at least VGA and serial consoles are not impacted due to disabling of interrupt. Any other consoles which require interrupt to be enabled for their working????

o As per pci-to-pci bridge architecture specification revision 1.2, interrupt disable bit in PCI-PCI bridge is optional and if implemented, it will disable interrupt generation from bridge but will have no effect on interrupts that the bridge forwards from the PCI devices on the secondary bus.

So even if interrupts are disabled on PCI-PCI bridge, interrupts generated by PCI devices on secondary bus are not blocked and I hope device should be working fine.

The whole idea is that currently this change is kdump specific. Ofcourse there shall be issues which are not known yet and more devices might not work for kdump kernels. But at the same time kdump kernels are not supposed to do a great deal except capture and save the dump. So this change might not be of a big concern even if some devices don't work as long as kdump kernel can boot.

Disabling interrupts at PCI level should increase the reliability of capturing the dump on newer machines with hardware compliant with PCI 2.3 or higher. Booting a kdump kernel with reduced functionality should always be better than not booting at all.

Thanks
Vivek

#1 lore raw
Quoted 3 lines · Vivek Goyal

So even if interrupts are disabled on PCI-PCI bridge, interrupts generated by PCI devices on secondary bus are not blocked and I hope device should be working fine.

How did you plan on disabling interrupts?
Did you see the MSI discussion that going on now in linux-pci mailing list?

Quoted 2 lines

But at the same time kdump kernels are not supposed to do a great deal except capture and save the dump.

I'd think you want to stop DMA for all devices. Just to prevent them from messing more with memory that you want to dump - ie get a consistent snapshot. Leaving VGA devices alone should be safe.

Quoted 2 lines

Disabling interrupts at PCI level should increase the reliability of capturing the dump on newer machines with hardware compliant with PCI 2.3 or higher.

*lots* of PCI devices predate PCI2.3. Possibly even the majority.

hth,
grant

#2 lore raw
Quoted 16 lines · Grant Grundler
Quoted 3 lines · Vivek Goyal

So even if interrupts are disabled on PCI-PCI bridge, interrupts generated by PCI devices on secondary bus are not blocked and I hope device should be working fine.

How did you plan on disabling interrupts?
Did you see the MSI discussion that going on now in linux-pci mailing list?

Quoted 2 lines

But at the same time kdump kernels are not supposed to do a great deal except capture and save the dump.

I'd think you want to stop DMA for all devices. Just to prevent them from messing more with memory that you want to dump - ie get a consistent snapshot. Leaving VGA devices alone should be safe.

Quoted 5 lines
Quoted 3 lines

Disabling interrupts at PCI level should increase the reliability of capturing

the dump on newer machines with hardware compliant with PCI 2.3 or higher.

*lots* of PCI devices predate PCI2.3. Possibly even the majority.

In general generic hardware bits for disabling DMA, disabling interrupts and the like are all advisory. With the current architecture things will work properly even if you don't manage to disable DMA (assuming you don't reassign IOMMU entries at least).

Non-shared interrupts are not a problem as they can fairly safely be disabled at the interrupt controller.

Shared interrupts are an interesting case. The simplest solution I can think of for a crash dump capture kernel is to periodically poll the hardware, as if all interrupts are shared. At that level I think we could get away with ignoring all hardware interrupt sources.

Does anyone know of a anything that would break by always polling the hardware? I guess there could be a problem with drivers that don't understand shared interrupts, are there enough of those to be an issue.

Eric

#3 lore raw
Quoted 6 lines · Eric W. Biederman
Quoted 1 line

*lots* of PCI devices predate PCI2.3. Possibly even the majority.

In general generic hardware bits for disabling DMA, disabling interrupts and the like are all advisory. With the current architecture things will work properly even if you don't manage to disable DMA (assuming you don't reassign IOMMU entries at least).

ISTR, pSeries (IBM), some alpha, some sparc64, and parisc (64-bit) require use of the IOMMU for *any* DMA. ie IOMMU entries need to be programmed. Probably want to make a choice to ignore those arches for now or sort out how to deal with an IOMMU.

Quoted 4 lines

Shared interrupts are an interesting case. The simplest solution I can think of for a crash dump capture kernel is to periodically poll the hardware, as if all interrupts are shared. At that level I think we could get away with ignoring all hardware interrupt sources.

Yes, that's perfectly ok. We are no longer in a multitasking env.

Quoted 4 lines

Does anyone know of a anything that would break by always polling the hardware? I guess there could be a problem with drivers that don't understand shared interrupts, are there enough of those to be an issue.

PCI requires drivers support Shared IRQs.
A few oddballs might be broken but I expect networking/mass storage drivers get this right.

grant

#4 lore raw
Quoted 8 lines · Grant Grundler
Quoted 6 lines · Eric W. Biederman
Quoted 1 line

*lots* of PCI devices predate PCI2.3. Possibly even the majority.

In general generic hardware bits for disabling DMA, disabling interrupts and the like are all advisory. With the current architecture things will work properly even if you don't manage to disable DMA (assuming you don't reassign IOMMU entries at least).

Quoted 4 lines

ISTR, pSeries (IBM), some alpha, some sparc64, and parisc (64-bit) require use of the IOMMU for *any* DMA. ie IOMMU entries need to be programmed. Probably want to make a choice to ignore those arches for now or sort out how to deal with an IOMMU.

The howto deal with an IOMMU has been sorted out but so far no one has actually done it. What has been discussed previously is simply reserving a handful of IOMMU entries, and then only using those in the crash recover kernel. This is essentially what we do with DMA on architectures that don't have an IOMMU and it seems quite safe enough there.

Quoted 6 lines
Quoted 4 lines

Shared interrupts are an interesting case. The simplest solution I can think of for a crash dump capture kernel is to periodically poll the hardware, as if all interrupts are shared. At that level I think we could get away with ignoring all hardware interrupt sources.

Yes, that's perfectly ok. We are no longer in a multitasking env.

Well we are at least capable of multitasking but that is no longer the primary focus. Having polling as at least an option should make debugging easier. Last I looked Andrews kernel hand an irqpoll option to do something very like this.

Quoted 8 lines
Quoted 4 lines

Does anyone know of a anything that would break by always polling the hardware? I guess there could be a problem with drivers that don't understand shared interrupts, are there enough of those to be an issue.

PCI requires drivers support Shared IRQs.
A few oddballs might be broken but I expect networking/mass storage drivers get this right.

Agreed. Which means any drivers we really need for dumping the system should be fine. If the drivers don't work in that mode at least the concept of fixing it won't be controversial.

Eric

#5 lore raw
Quoted 3 lines · Eric W. Biederman

The howto deal with an IOMMU has been sorted out but so far no one has actually done it. What has been discussed previously is simply reserving a handful of IOMMU entries,

How? with dma_alloc_consistent() or some special hook? I'm just curious.

...

Quoted 4 lines

and then only using those
in the crash recover kernel. This is essentially what we do with DMA on architectures that don't have an IOMMU and it seems quite safe enough there.

Yeah, in general that should be feasible.

One might be able to trivially allocate a small, seperate IO PDIR just for KDUMP and switch to that. Key thing is it be physically contiguous in memory. Very little code is involved with IO Pdir setup for both parisc and IA64. I can't speak for Alpha/sparc/ppc/et al.

...

Quoted 4 lines

Well we are at least capable of multitasking but that is no longer the primary focus. Having polling as at least an option should make debugging easier. Last I looked Andrews kernel hand an irqpoll option to do something very like this.

You could run the itimer but I don't see why you should. Kdump is essentially an embedded linux kernel. It really doesn't need to be premptive multitasking either.

Anyway, sounds like you guys are on the right track.

thanks,
grant

#7 lore raw
In reply to Eric W. Biederman
Quoted 27 lines · Eric W. Biederman
Quoted 8 lines · Grant Grundler
Quoted 6 lines · Eric W. Biederman
Quoted 1 line

*lots* of PCI devices predate PCI2.3. Possibly even the majority.

In general generic hardware bits for disabling DMA, disabling interrupts and the like are all advisory. With the current architecture things will work properly even if you don't manage to disable DMA (assuming you don't reassign IOMMU entries at least).

Quoted 4 lines

ISTR, pSeries (IBM), some alpha, some sparc64, and parisc (64-bit) require use of the IOMMU for *any* DMA. ie IOMMU entries need to be programmed. Probably want to make a choice to ignore those arches for now or sort out how to deal with an IOMMU.

The howto deal with an IOMMU has been sorted out but so far no one has actually done it. What has been discussed previously is simply reserving a handful of IOMMU entries, and then only using those in the crash recover kernel. This is essentially what we do with DMA on architectures that don't have an IOMMU and it seems quite safe enough there.

Quoted 5 lines
Quoted 4 lines

Shared interrupts are an interesting case. The simplest solution I can think of for a crash dump capture kernel is to periodically poll the hardware, as if all interrupts are shared. At that level I think we could get away with ignoring all hardware interrupt sources.

Quoted 7 lines
Quoted 1 line

Yes, that's perfectly ok. We are no longer in a multitasking env.

Well we are at least capable of multitasking but that is no longer the primary focus. Having polling as at least an option should make debugging easier. Last I looked Andrews kernel hand an irqpoll option to do something very like this.

If I understand this right, the idea is that let all irqs be masked (except timer one) and invoke all the irq handlers whenever a timer interrupt occurs. This will automatcally be equivalent to drivers polling their devices for any interrupt.

As you mentioned that irqpoll option comes close. If enabled, it invokes all the irq handlers on every timer interrupt (IRQ0). The only difference is that irqs are not masked (until and unless kernel masks these due to excessive unhandled interrupts).

I tried booting kdump kernel with irqpoll option. It seems to be going little bit ahead than previous point of failure (boot without irqpoll) but panics later. Following is the stack trace.

mptbase: Initiating ioc0 bringup

Unable to handle kernel NULL pointer dereference at virtual address 00000608
 printing eip:
c11e1b73
*pde = 00000000
Oops: 0000 [#1]
Modules linked in:
CPU:    0
EIP:    0060:[<c11e1b73>]    Not tainted VLI
EFLAGS: 00010006   (2.6.12-rc6-mm1-16M)

EIP is at mptscsih_io_done+0x23/0x350
eax: c1778400 ebx: 00000000 ecx: 00000600 edx: 00000250
esi: 00000000 edi: 0000000e ebp: 00000001 esp: c15efcbc
ds: 007b es: 007b ss: 0068
Process swapper (pid: 1, threadinfo=c15ee000 task=c147fa00)

Stack: c15f02d1 c1116cfd c1116d60 c15df660 00000000 0000006c 00000250 00000000
       00000000 0000000e 00000001 c11db9ea c1778400 00000600 00000000 00000000
       00000600 c1788aa0 c1410e80 00000009 00000001 c1039e7c 00000009 c1778400
Call Trace:
 [<c1116cfd>] acpi_ev_gpe_detect+0x83/0x10f
 [<c1116d60>] acpi_ev_gpe_detect+0xe6/0x10f
 [<c11db9ea>] mpt_interrupt+0xfa/0x1e0
 [<c1039e7c>] misrouted_irq+0xec/0x100
 [<c103a007>] note_interrupt+0xb7/0xf0
 [<c10399c4>] __do_IRQ+0xe4/0xf0
 [<c1004e09>] do_IRQ+0x19/0x30
 [<c1003246>] common_interrupt+0x1a/0x20
 [<c1018d8f>] release_console_sem+0x3f/0xa0
 [<c1018c27>] vprintk+0x177/0x220
 [<c126ecbd>] pci_read+0x3d/0x50
 [<c1101a0a>] kobject_get+0x1a/0x30
 [<c116b676>] get_device+0x16/0x30
 [<c110b95a>] pci_dev_get+0x1a/0x30
 [<c1018aa7>] printk+0x17/0x20
 [<c11dcbeb>] mpt_do_ioc_recovery+0x4b/0x540
 [<c11dc4ff>] mpt_attach+0x2ef/0x690
 [<c11e6e03>] mptspi_probe+0x23/0x3e0
 [<c110b4f2>] pci_device_probe_static+0x52/0x70
 [<c110b54c>] __pci_device_probe+0x3c/0x50
 [<c110b58f>] pci_device_probe+0x2f/0x50
 [<c116cb68>] driver_probe_device+0x38/0xb0
 [<c116cc60>] __driver_attach+0x0/0x50
 [<c116cca7>] __driver_attach+0x47/0x50
 [<c116c229>] bus_for_each_dev+0x69/0x80
 [<c116ccd6>] driver_attach+0x26/0x30
 [<c116cc60>] __driver_attach+0x0/0x50
 [<c116c6d8>] bus_add_driver+0x88/0xc0
 [<c110b6d0>] pci_device_shutdown+0x0/0x30
 [<c110b857>] pci_register_driver+0x67/0x90
 [<c142a890>] mptspi_init+0xa0/0xb0
 [<c11e37a0>] mptscsih_ioc_reset+0x0/0x170
 [<c141483c>] do_initcalls+0x2c/0xc0
 [<c142d5fa>] sock_init+0x2a/0x40
 [<c1000290>] init+0x0/0x100
 [<c10002b5>] init+0x25/0x100
 [<c10013a0>] kernel_thread_helper+0x0/0x10
 [<c10013a5>] kernel_thread_helper+0x5/0x10
Code: 00 8d bc 27 00 00 00 00 55 57 56 53 83 ec 1c 8b 44 24 30 8b 4c 24 34 8b 7
 <0>Kernel panic - not syncing: Fatal exception in interrupt

Thanks
Vivek

#6 lore raw
In reply to Grant Grundler
Quoted 7 lines · Grant Grundler
Quoted 3 lines · Eric W. Biederman

The howto deal with an IOMMU has been sorted out but so far no one has actually done it. What has been discussed previously is simply reserving a handful of IOMMU entries,

How? with dma_alloc_consistent() or some special hook? I'm just curious.

We didn't get that far but I believe the idea was a special hook.

Quoted 6 lines

...

Quoted 4 lines

and then only using those
in the crash recover kernel. This is essentially what we do with DMA on architectures that don't have an IOMMU and it seems quite safe enough there.

Quoted 6 lines

Yeah, in general that should be feasible.

One might be able to trivially allocate a small, seperate IO PDIR just for KDUMP and switch to that. Key thing is it be physically contiguous in memory. Very little code is involved with IO Pdir setup for both parisc and IA64. I can't speak for Alpha/sparc/ppc/et al.

Cool.

Quoted 9 lines

...

Quoted 4 lines

Well we are at least capable of multitasking but that is no longer the primary focus. Having polling as at least an option should make debugging easier. Last I looked Andrews kernel hand an irqpoll option to do something very like this.

You could run the itimer but I don't see why you should. Kdump is essentially an embedded linux kernel. It really doesn't need to be premptive multitasking either.

It is mostly a matter of minimizing differences from the norm.

Quoted 1 line

Anyway, sounds like you guys are on the right track.

Thanks. It just takes a while for the simple solutions to
get there.

Eric

#8 lore raw
In reply to Vivek Goyal
Quoted 19 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

Shared interrupts are an interesting case. The simplest solution I can think of for a crash dump capture kernel is to periodically poll the hardware, as if all interrupts are shared. At that level I think we could get away with ignoring all hardware interrupt sources.

Quoted 7 lines
Quoted 1 line

Yes, that's perfectly ok. We are no longer in a multitasking env.

Well we are at least capable of multitasking but that is no longer the primary focus. Having polling as at least an option should make debugging easier. Last I looked Andrews kernel hand an irqpoll option to do something very like this.

If I understand this right, the idea is that let all irqs be masked (except timer one) and invoke all the irq handlers whenever a timer interrupt occurs. This will automatcally be equivalent to drivers polling their devices for any interrupt.

Quoted 9 lines

As you mentioned that irqpoll option comes close. If enabled, it invokes all the irq handlers on every timer interrupt (IRQ0). The only difference is that irqs are not masked (until and unless kernel masks these due to excessive unhandled interrupts).

I tried booting kdump kernel with irqpoll option. It seems to be going little bit ahead than previous point of failure (boot without irqpoll) but panics later. Following is the stack trace.

Second kernel booted fine with MPT_DEBUG_IRQ enabled (with irqpoll option). There were few warning messages though spitted by the code under MPT_DEBUG_IRQ.

Looks like drivers need to be hardened on case to case basis to initialize properly even if underlying device is not in a reset state.

Thanks
Vivek