Skip to contentSkip to conversation

What determines which interrupts are shared under Linux?

DISCUSSION Roger Heflin 7 participants 16 messages Aug 15, 2006 – Aug 17, 2006 libata

lore
Share & tools
lore
#0 lore raw

Hello,

On Linux when interrupts are defined similar to below, what defines say
ide2, ide3 to be on the same interrupt? The bios, linux, the driver using
the interrupt? And can that be controlled/overrode at the kernel/driver level?

I have identified that the disks that are shared on ide2, ide3 do funny things when both are being heavily used (dma_expiry), this is an older driver versions but I have experienced it before with a lot newer driver, and a bios adjustment previously fixed a similar issue, so that may be what is needed in this case also, I am not sure how they fixed it, but I suspect that the setup the interrupt
to not be shared. I have a large number of machines and under heavy loads all
seem to duplicate the issue, and it always happens with the disks on ide2/ide3, never on the disk connected to ide4.

CPU0 CPU1 CPU2 CPU3

  0:   56616921    5359998    7002142     938817          XT-PIC  timer
  1:          8         88         96          0    IO-APIC-edge  i8042
  2:          0          0          0          0          XT-PIC  cascade
  4:       2091        100        208       2477    IO-APIC-edge  serial
  8:          0          0          0          0    IO-APIC-edge  rtc
  9:          0          0          0          0   IO-APIC-level  acpi
 20:          0          0          0          0   IO-APIC-level  ehci_hcd
 21:          0        950     401419     414482   IO-APIC-level  ide4, 
ohci_hcd
 22:       1165    1704243     576247       6796   IO-APIC-level  ide2, 
ide3
 47:      65971          0          0          0   IO-APIC-level  eth0
NMI:          1          1          1          1
LOC:   69904264   69877733   69879541   69901903
ERR:          0
MIS:        105

Roger

#1 lore raw

Hello,

Quoted 2 lines · Roger Heflin

On Linux when interrupts are defined similar to below, what defines say ide2, ide3 to be on the same interrupt? The bios, linux, the driver using the interrupt?

In this particular case (I suppoer both belong to the same IDE conrtoller)
the hardware design determines this in part and software in another part.
If the PCI IDE controller was in the leagcy mode (it should be ide0/1 then), the interrupts would've been 14 and 15. But in the native PCI mode, both chanels share the same interrupt level (if they're both in native mode that is).

Quoted 1 line

And can that be controlled/overrode at the kernel/driver level?

The only possibility might be do disable ide0/1 in the BIOS I guess.

Quoted 6 lines

I have identified that the disks that are shared on ide2, ide3 do funny things when both are being heavily used (dma_expiry), this is an older driver versions but I have experienced it before with a lot newer driver, and a bios adjustment previously fixed a similar issue, so that may be what is needed in this case also, I am not sure how they fixed it, but I suspect that the setup the interrupt to not be shared.

I doubt that your suspicions are justified.

Quoted 4 lines

I have a large number of machines and under heavy
loads all
seem to duplicate the issue, and it always happens with the disks on ide2/ide3, never on the disk connected to ide4.

BTW, you never named your particular IDE hardware.

Quoted 17 lines

CPU0 CPU1 CPU2 CPU3

  0:   56616921    5359998    7002142     938817          XT-PIC  timer
  1:          8         88         96          0    IO-APIC-edge  i8042
  2:          0          0          0          0          XT-PIC  cascade
  4:       2091        100        208       2477    IO-APIC-edge  serial
  8:          0          0          0          0    IO-APIC-edge  rtc
  9:          0          0          0          0   IO-APIC-level  acpi
 20:          0          0          0          0   IO-APIC-level  ehci_hcd
 21:          0        950     401419     414482   IO-APIC-level  ide4, 
ohci_hcd
 22:       1165    1704243     576247       6796   IO-APIC-level  ide2, 
ide3
 47:      65971          0          0          0   IO-APIC-level  eth0
NMI:          1          1          1          1
LOC:   69904264   69877733   69879541   69901903
ERR:          0
MIS:        105

MBR, Sergei

#2 lore raw
Quoted 9 lines · Sergei Shtylyov

Hello,

Quoted 6 lines

I have identified that the disks that are shared on ide2, ide3 do funny things when both are being heavily used (dma_expiry), this is an older driver versions but I have experienced it before with a lot newer driver, and a bios adjustment previously fixed a similar issue, so that may be what is needed in this case also, I am not sure how they fixed it, but I suspect that the setup the interrupt to not be shared.

Quoted 7 lines

I doubt that your suspicions are justified.

Quoted 3 lines

I have a large number of machines and under heavy loads all
seem to duplicate the issue, and it always happens with the disks on ide2/ide3, never on the disk connected to ide4.

BTW, you never named your particular IDE hardware.

NVIDIA, CK804 chipset.

And the disks are on the sata part, and are found as e g and i so if they are in legacy mode, it would make me suspect that the disks believe they are on separate channels, and the performance characteristics appear to confirm that as the disks don't seem to interfere with each other.

Quoted 22 lines
Quoted 14 lines

CPU0 CPU1 CPU2 CPU3

  0:   56616921    5359998    7002142     938817          XT-PIC  timer
  1:          8         88         96          0    IO-APIC-edge  i8042
  2:          0          0          0          0          XT-PIC  cascade
  4:       2091        100        208       2477    IO-APIC-edge  serial
  8:          0          0          0          0    IO-APIC-edge  rtc
  9:          0          0          0          0   IO-APIC-level  acpi

20: 0 0 0 0 IO-APIC-level
ehci_hcd
21: 0 950 401419 414482 IO-APIC-level
ide4, ohci_hcd
22: 1165 1704243 576247 6796 IO-APIC-level
ide2, ide3

Quoted 5 lines
 47:      65971          0          0          0   IO-APIC-level  eth0
NMI:          1          1          1          1
LOC:   69904264   69877733   69879541   69901903
ERR:          0
MIS:        105

MBR, Sergei

#3 lore raw
In reply to Roger Heflin

Ar Maw, 2006-08-15 am 09:17 -0500, ysgrifennodd Roger Heflin:

Quoted 4 lines

On Linux when interrupts are defined similar to below, what defines say
ide2, ide3 to be on the same interrupt? The bios, linux, the driver using
the interrupt? And can that be controlled/overrode at the
kernel/driver level?

Only with a soldering iron. They are the way the system is wired. Moving boards between slots may change the IRQ allocation.

Quoted 3 lines

I have identified that the disks that are shared on ide2, ide3 do funny things when both are being heavily used (dma_expiry), this is an older driver versions

That could be occuring just through lack of PCI bus bandwidth.

#11 lore raw
In reply to Roger Heflin
Quoted 1 line · Roger Heflin

22:       1165    1704243     576247       6796   IO-APIC-level  ide2, ide3

The first thing that determines interrupt sharing is where the physical wires from the devices are routed. If these ide controllers are add-on boards, then you could try moving them between PCI slots -- as the slots generally have different primary interrupt lines. When the board runs out of unique lines, they are generally wired to re-use lines on different slots.
The manual for the board will generally tell you how the lines
are routed.

If the devices are using the same physical interrupt line, then it is not possible for software or BIOS to move them to different lines.

If these devices are functions in the same device, it is possible that there is some internal (BIOS or firmware) setting to tell it to use two interrupt wires instead of one.

Once the wire for the device is determined, it is up to the BIOS and the OS to program (or not) an interrupt router to assign the wire to an interrupt input pin. Your dmesg will tell us if this is happening on this board.

-Len

#12 lore raw
In reply to Roger Heflin
Quoted 40 lines · Roger Heflin

Hello,

On Linux when interrupts are defined similar to below, what defines say
ide2, ide3 to be on the same interrupt? The bios, linux, the driver using
the interrupt? And can that be controlled/overrode at the
kernel/driver level?

I have identified that the disks that are shared on ide2, ide3 do funny things when both are being heavily used (dma_expiry), this is an older driver versions
but I have experienced it before with a lot newer driver, and a bios adjustment
previously fixed a similar issue, so that may be what is needed in this case also,
I am not sure how they fixed it, but I suspect that the setup the interrupt
to not be shared. I have a large number of machines and under heavy
loads all
seem to duplicate the issue, and it always happens with the disks on ide2/ide3,
never on the disk connected to ide4.

CPU0 CPU1 CPU2 CPU3

  0:   56616921    5359998    7002142     938817          XT-PIC  timer
  1:          8         88         96          0    IO-APIC-edge  i8042
  2:          0          0          0          0          XT-PIC  cascade
  4:       2091        100        208       2477    IO-APIC-edge  serial
  8:          0          0          0          0    IO-APIC-edge  rtc
  9:          0          0          0          0   IO-APIC-level  acpi
 20:          0          0          0          0   IO-APIC-level  ehci_hcd
 21:          0        950     401419     414482   IO-APIC-level  ide4,
ohci_hcd
 22:       1165    1704243     576247       6796   IO-APIC-level  ide2,
ide3
 47:      65971          0          0          0   IO-APIC-level  eth0
NMI:          1          1          1          1
LOC:   69904264   69877733   69879541   69901903
ERR:          0
MIS:        105
Quoted 1 line

Roger

The hardware defines what traces go to whatever chips on the motherboard and to the feature module sockets. Sometimes, the ultimate IRQ that these traces go to can be configured in the BIOS. Sometimes not. The kernel just uses what was set up by the BIOS, convention, and the PCI devices themselves.

Cheers,
Dick Johnson
Penguin : Linux version 2.6.16.24 on an i686 machine (5592.62 BogoMips). New book: http://www.AbominableFirebug.com/
_


****************************************************************
The information transmitted in this message is confidential and may be privileged. Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited. If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to [email protected] - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.

#4 lore raw
In reply to Alan Cox
Quoted 5 lines · Alan Cox

Ar Maw, 2006-08-15 am 09:17 -0500, ysgrifennodd Roger Heflin:

Quoted 3 lines

On Linux when interrupts are defined similar to below, what defines say
ide2, ide3 to be on the same interrupt? The bios, linux, the driver using
the interrupt? And can that be controlled/overrode at the kernel/driver level?

Quoted 10 lines

Only with a soldering iron. They are the way the system is wired. Moving boards between slots may change the IRQ allocation.

Quoted 2 lines

I have identified that the disks that are shared on ide2, ide3 do funny things when both are being heavily used (dma_expiry), this is an older driver versions

That could be occuring just through lack of PCI bus bandwidth.

Right now, all controllers are part of the NVIDIA chipset which should
mean the are on the same PCI bandwidth. The block diagram does indicate
the 4 sata channels as one unit, though I don't know exactly how things were internally done, performance testing seems to indicate that all 3 I am using are independent hardware as they don't seem to affect each others load.

I am currently retesting under 2.6.17.8 to see if I have similar issues there, under that it show interrupts like below, different interrupt numbers, but similar sharing as ata1/ata2, and ata3/ata4 are on the same interrupt.

ata1: SATA max UDMA/133 cmd 0xEC00 ctl 0xE802 bmdma 0xDC00 irq 16
ata2: SATA max UDMA/133 cmd 0xE400 ctl 0xE002 bmdma 0xDC08 irq 16
ata1: SATA link up 3.0 Gbps (SStatus 123)
ata1: dev 0 cfg 49:2f00 82:706b 83:7e61 84:4023 85:7069 86:3e41 87:4023 88:407f
ata1: dev 0 ATA-7, max UDMA/133, 321672960 sectors: LBA48
ata1: dev 0 configured for UDMA/133
ata2: SATA link up 3.0 Gbps (SStatus 123)
ata2: dev 0 cfg 49:2f00 82:706b 83:7e61 84:4023 85:7069 86:3e41 87:4023 88:407f
ata2: dev 0 ATA-7, max UDMA/133, 321672960 sectors: LBA48
ata2: dev 0 configured for UDMA/133
ata3: SATA max UDMA/133 cmd 0xD800 ctl 0xD402 bmdma 0xC800 irq 17
ata4: SATA max UDMA/133 cmd 0xD000 ctl 0xCC02 bmdma 0xC808 irq 17
ata3: SATA link up 3.0 Gbps (SStatus 123)
ata3: dev 0 cfg 49:2f00 82:706b 83:7e61 84:4023 85:7069 86:3e41 87:4023 88:407f
ata3: dev 0 ATA-7, max UDMA/133, 321672960 sectors: LBA48
ata3: dev 0 configured for UDMA/133
ata4: SATA link down (SStatus 0)

Roger

#5 lore raw

Ar Maw, 2006-08-15 am 10:24 -0500, ysgrifennodd Roger Heflin:

Quoted 4 lines

I am currently retesting under 2.6.17.8 to see if I have similar issues there, under that it show interrupts like below, different interrupt numbers,
but similar sharing as ata1/ata2, and ata3/ata4 are on the same interrupt.

Thats what I would expect to see - two channels per PCI device is the normal layout and they will always share the IRQ.

#13 lore raw
In reply to Roger Heflin

we've seen a lot of problems on ck804 chipsets when multiple devices share level-triggered interrupts. I think some of the earlier sample bioses assumed that interrupts would be configured via ACPI, and when ACPI is not used, the interrupts end up as level-triggered instead of edge-triggered.

you might investigate whether ACPI is failing to configure your system (are you on 2.4? updating to 2.6 might help), using the 'noapic' kernel parameter to force interrupts back to PIC, or the 'irqpoll' kernel parameter to try to work around the level-triggered interrupts.

Thanks,
Terence

Quoted 40 lines · On Tue, Aug 15, 2006 at 09:17:04AM -0500, [email protected] wrote:

Hello,

On Linux when interrupts are defined similar to below, what defines say
ide2, ide3 to be on the same interrupt? The bios, linux, the driver using
the interrupt? And can that be controlled/overrode at the
kernel/driver level?

I have identified that the disks that are shared on ide2, ide3 do funny things when both are being heavily used (dma_expiry), this is an older driver versions
but I have experienced it before with a lot newer driver, and a bios adjustment
previously fixed a similar issue, so that may be what is needed in this case also,
I am not sure how they fixed it, but I suspect that the setup the interrupt
to not be shared. I have a large number of machines and under heavy
loads all
seem to duplicate the issue, and it always happens with the disks on ide2/ide3,
never on the disk connected to ide4.

CPU0 CPU1 CPU2 CPU3

  0:   56616921    5359998    7002142     938817          XT-PIC  timer
  1:          8         88         96          0    IO-APIC-edge  i8042
  2:          0          0          0          0          XT-PIC  cascade
  4:       2091        100        208       2477    IO-APIC-edge  serial
  8:          0          0          0          0    IO-APIC-edge  rtc
  9:          0          0          0          0   IO-APIC-level  acpi
 20:          0          0          0          0   IO-APIC-level  ehci_hcd
 21:          0        950     401419     414482   IO-APIC-level  ide4, 
ohci_hcd
 22:       1165    1704243     576247       6796   IO-APIC-level  ide2, 
ide3
 47:      65971          0          0          0   IO-APIC-level  eth0
NMI:          1          1          1          1
LOC:   69904264   69877733   69879541   69901903
ERR:          0
MIS:        105
Quoted 8 lines

Roger
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------

#14 lore raw

Hello.

Quoted 5 lines · Terence Ripperda

we've seen a lot of problems on ck804 chipsets when multiple devices share level-triggered interrupts. I think some of the earlier sample bioses assumed that interrupts would be configured via ACPI, and when ACPI is not used, the interrupts end up as level-triggered instead of edge-triggered.

Edge-triggered *shared* interrupts?! Now that sounds interesting (I'm not
saying impossible).

WBR, Sergei

#15 lore raw
Quoted 11 lines · On Tue, Aug 15, 2006 at 09:42:49PM +0400, [email protected] wrote:

Hello.

Quoted 5 lines · Terence Ripperda

we've seen a lot of problems on ck804 chipsets when multiple devices share level-triggered interrupts. I think some of the earlier sample bioses assumed that interrupts would be configured via ACPI, and when ACPI is not used, the interrupts end up as level-triggered instead of edge-triggered.

Edge-triggered *shared* interrupts?! Now that sounds interesting (I'm not saying impossible).

ah shoot, you're right. doing too much and got the two backwards. ignore my email, I'm going off to sit in the corner with a dunce cap on..

Quoted 2 lines

WBR, Sergei

-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------

#6 lore raw
In reply to Alan Cox
Quoted 10 lines · Alan Cox

Ar Maw, 2006-08-15 am 10:24 -0500, ysgrifennodd Roger Heflin:

Quoted 3 lines

I am currently retesting under 2.6.17.8 to see if I have similar issues there, under that it show interrupts like below, different interrupt numbers, but similar sharing as ata1/ata2, and ata3/ata4 are on the same interrupt.

Thats what I would expect to see - two channels per PCI device is the normal layout and they will always share the IRQ.

After looking at other information that looks correct.

It looks like the older DMA recovery code never works on this chipset,
once it goes into DMA recovery it never comes out of it. I am looking
at that to see if anything can be done about it.

The problem I am having is that we cannot use a later kernel because it has some other issues (MPT fusion driver is 1/4 the speed of the older one for some unknown reason-this is in 2.6.17, the older MPT fusion works fine on exactly the same machine/cards/bios-I am not sure of the actual underlying casue of this slowdown).

On the specific kernel that I have I appear to have both IDE and sata_nv drivers, is there a way to force things to use sata_nv/libata rather than the older ide driver for the NVIDIA sata controller?

Roger

#7 lore raw

Ar Maw, 2006-08-15 am 12:47 -0500, ysgrifennodd Roger Heflin:

Quoted 3 lines

On the specific kernel that I have I appear to have both IDE and sata_nv drivers, is there a way to force things to use sata_nv/libata rather than the older ide driver for the NVIDIA sata controller?

You can build 2.6.18-mm like that (say N to AMD/NVIDIA PATA driver, y to sata_nv and pata_amd libata drivers). Still have to figure out why your MPT fusion is slow but if its still slow in 2.6.18-mm then it needs figuring out anyway

#8 lore raw
In reply to Roger Heflin
Quoted 4 lines · Roger Heflin

It looks like the older DMA recovery code never works on this chipset,
once it goes into DMA recovery it never comes out of it. I am looking
at that to see if anything can be done about it.

You could try booting with ide2=serialize as a kernel parameter. That should ensure the two channels are never in use simultaneously. Trades off a bit of performance for a bit of reliability.

Cheers

#9 lore raw

Ar Mer, 2006-08-16 am 18:54 -0400, ysgrifennodd Mark Lord:

Quoted 4 lines
Quoted 3 lines · Roger Heflin

It looks like the older DMA recovery code never works on this chipset, once it goes into DMA recovery it never comes out of it.

DMA recovery is fairly broken in drivers/ide especially if it tries to change mode. libata does not have this problem and I have no plans to even try and fix the drivers/ide code for this issue as its a major piece of work.

#10 lore raw
Quoted 11 lines · Alan Cox

Ar Mer, 2006-08-16 am 18:54 -0400, ysgrifennodd Mark Lord:

Quoted 3 lines
Quoted 2 lines · Roger Heflin

It looks like the older DMA recovery code never works on this chipset, once it goes into DMA recovery it never comes out of it.

DMA recovery is fairly broken in drivers/ide especially if it tries to change mode. libata does not have this problem and I have no plans to even try and fix the drivers/ide code for this issue as its a major piece of work.

If already figured that since I was pretty sure all of it was
being rolled to libata. I was able to switch that controller and
disk to the libata sata_nv.

Roger