What determines which interrupts are shared under Linux?
Hello,
On Linux when interrupts are defined similar to below, what defines say
ide2, ide3 to be on the same interrupt? The bios, linux, the driver using
the interrupt? And can that be controlled/overrode at the kernel/driver level?
I have identified that the disks that are shared on ide2, ide3 do funny things when both are being heavily used (dma_expiry), this is an older driver versions but I have experienced it before with a lot newer driver, and a bios adjustment previously fixed a similar issue, so that may be what is needed in this case also, I am not sure how they fixed it, but I suspect that the setup the interrupt
to not be shared. I have a large number of machines and under heavy loads all
seem to duplicate the issue, and it always happens with the disks on ide2/ide3, never on the disk connected to ide4.
CPU0 CPU1 CPU2 CPU3
0: 56616921 5359998 7002142 938817 XT-PIC timer 1: 8 88 96 0 IO-APIC-edge i8042 2: 0 0 0 0 XT-PIC cascade 4: 2091 100 208 2477 IO-APIC-edge serial 8: 0 0 0 0 IO-APIC-edge rtc 9: 0 0 0 0 IO-APIC-level acpi 20: 0 0 0 0 IO-APIC-level ehci_hcd 21: 0 950 401419 414482 IO-APIC-level ide4, ohci_hcd 22: 1165 1704243 576247 6796 IO-APIC-level ide2, ide3 47: 65971 0 0 0 IO-APIC-level eth0 NMI: 1 1 1 1 LOC: 69904264 69877733 69879541 69901903 ERR: 0 MIS: 105
Roger
Hello,
Quoted 2 lines · Roger Heflin
On Linux when interrupts are defined similar to below, what defines say ide2, ide3 to be on the same interrupt? The bios, linux, the driver using the interrupt?
In this particular case (I suppoer both belong to the same IDE conrtoller)
the hardware design determines this in part and software in another part.
If the PCI IDE controller was in the leagcy mode (it should be ide0/1 then), the interrupts would've been 14 and 15. But in the native PCI mode, both chanels share the same interrupt level (if they're both in native mode that is).
Quoted 1 line
And can that be controlled/overrode at the kernel/driver level?
The only possibility might be do disable ide0/1 in the BIOS I guess.
Quoted 6 lines
I have identified that the disks that are shared on ide2, ide3 do funny things when both are being heavily used (
dma_expiry), this is an older driver versions but I have experienced it before with a lot newer driver, and a bios adjustment previously fixed a similar issue, so that may be what is needed in this case also, I am not sure how they fixed it, but I suspect that the setup the interrupt to not be shared.
I doubt that your suspicions are justified.
Quoted 4 lines
I have a large number of machines and under heavy
loads all
seem to duplicate the issue, and it always happens with the disks on ide2/ide3, never on the disk connected to ide4.
BTW, you never named your particular IDE hardware.
Quoted 17 lines
CPU0 CPU1 CPU2 CPU3
0: 56616921 5359998 7002142 938817 XT-PIC timer 1: 8 88 96 0 IO-APIC-edge i8042 2: 0 0 0 0 XT-PIC cascade 4: 2091 100 208 2477 IO-APIC-edge serial 8: 0 0 0 0 IO-APIC-edge rtc 9: 0 0 0 0 IO-APIC-level acpi 20: 0 0 0 0 IO-APIC-level ehci_hcd 21: 0 950 401419 414482 IO-APIC-level ide4, ohci_hcd 22: 1165 1704243 576247 6796 IO-APIC-level ide2, ide3 47: 65971 0 0 0 IO-APIC-level eth0 NMI: 1 1 1 1 LOC: 69904264 69877733 69879541 69901903 ERR: 0 MIS: 105
MBR, Sergei
Quoted 9 lines · Sergei Shtylyov
Hello,
Quoted 6 lines
I have identified that the disks that are shared on ide2, ide3 do funny things when both are being heavily used (
dma_expiry), this is an older driver versions but I have experienced it before with a lot newer driver, and a bios adjustment previously fixed a similar issue, so that may be what is needed in this case also, I am not sure how they fixed it, but I suspect that the setup the interrupt to not be shared.
Quoted 7 lines
I doubt that your suspicions are justified.
Quoted 3 lines
I have a large number of machines and under heavy loads all
seem to duplicate the issue, and it always happens with the disks on ide2/ide3, never on the disk connected to ide4.BTW, you never named your particular IDE hardware.
NVIDIA, CK804 chipset.
And the disks are on the sata part, and are found as e g and i so if they are in legacy mode, it would make me suspect that the disks believe they are on separate channels, and the performance characteristics appear to confirm that as the disks don't seem to interfere with each other.
Quoted 22 lines
Quoted 14 lines
CPU0 CPU1 CPU2 CPU3
0: 56616921 5359998 7002142 938817 XT-PIC timer 1: 8 88 96 0 IO-APIC-edge i8042 2: 0 0 0 0 XT-PIC cascade 4: 2091 100 208 2477 IO-APIC-edge serial 8: 0 0 0 0 IO-APIC-edge rtc 9: 0 0 0 0 IO-APIC-level acpi20: 0 0 0 0 IO-APIC-level
ehci_hcd
21: 0 950 401419 414482 IO-APIC-level
ide4,ohci_hcd
22: 1165 1704243 576247 6796 IO-APIC-level
ide2, ide3Quoted 5 lines
47: 65971 0 0 0 IO-APIC-level eth0 NMI: 1 1 1 1 LOC: 69904264 69877733 69879541 69901903 ERR: 0 MIS: 105MBR, Sergei
Ar Maw, 2006-08-15 am 09:17 -0500, ysgrifennodd Roger Heflin:
Quoted 4 lines
On Linux when interrupts are defined similar to below, what defines say
ide2, ide3 to be on the same interrupt? The bios, linux, the driver using
the interrupt? And can that be controlled/overrode at the
kernel/driver level?
Only with a soldering iron. They are the way the system is wired. Moving boards between slots may change the IRQ allocation.
Quoted 3 lines
I have identified that the disks that are shared on ide2, ide3 do funny things when both are being heavily used (
dma_expiry), this is an older driver versions
That could be occuring just through lack of PCI bus bandwidth.
Quoted 1 line · Roger Heflin
22: 1165 1704243 576247 6796 IO-APIC-level ide2, ide3
The first thing that determines interrupt sharing is where the physical wires from the devices are routed. If these ide controllers are add-on boards, then you could try moving them between PCI slots -- as the slots generally have different primary interrupt lines. When the board runs out of unique lines, they are generally wired to re-use lines on different slots.
The manual for the board will generally tell you how the lines
are routed.
If the devices are using the same physical interrupt line, then it is not possible for software or BIOS to move them to different lines.
If these devices are functions in the same device, it is possible that there is some internal (BIOS or firmware) setting to tell it to use two interrupt wires instead of one.
Once the wire for the device is determined, it is up to the BIOS and the OS to program (or not) an interrupt router to assign the wire to an interrupt input pin. Your dmesg will tell us if this is happening on this board.
-Len
Quoted 40 lines · Roger Heflin
Hello,
On Linux when interrupts are defined similar to below, what defines say
ide2, ide3 to be on the same interrupt? The bios, linux, the driver using
the interrupt? And can that be controlled/overrode at the
kernel/driver level?I have identified that the disks that are shared on ide2, ide3 do funny things when both are being heavily used (
dma_expiry), this is an older driver versions
but I have experienced it before with a lot newer driver, and a bios adjustment
previously fixed a similar issue, so that may be what is needed in this case also,
I am not sure how they fixed it, but I suspect that the setup the interrupt
to not be shared. I have a large number of machines and under heavy
loads all
seem to duplicate the issue, and it always happens with the disks on ide2/ide3,
never on the disk connected to ide4.CPU0 CPU1 CPU2 CPU3
0: 56616921 5359998 7002142 938817 XT-PIC timer 1: 8 88 96 0 IO-APIC-edge i8042 2: 0 0 0 0 XT-PIC cascade 4: 2091 100 208 2477 IO-APIC-edge serial 8: 0 0 0 0 IO-APIC-edge rtc 9: 0 0 0 0 IO-APIC-level acpi 20: 0 0 0 0 IO-APIC-level ehci_hcd 21: 0 950 401419 414482 IO-APIC-level ide4, ohci_hcd 22: 1165 1704243 576247 6796 IO-APIC-level ide2, ide3 47: 65971 0 0 0 IO-APIC-level eth0 NMI: 1 1 1 1 LOC: 69904264 69877733 69879541 69901903 ERR: 0 MIS: 105
Quoted 1 line
Roger
The hardware defines what traces go to whatever chips on the motherboard and to the feature module sockets. Sometimes, the ultimate IRQ that these traces go to can be configured in the BIOS. Sometimes not. The kernel just uses what was set up by the BIOS, convention, and the PCI devices themselves.
Cheers,
Dick Johnson
Penguin : Linux version 2.6.16.24 on an i686 machine (5592.62 BogoMips). New book: http://www.AbominableFirebug.com/
_
****************************************************************
The information transmitted in this message is confidential and may be privileged. Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited. If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to [email protected] - and destroy all copies of this information, including any attachments, without reading or disclosing them.
Thank you.
Quoted 5 lines · Alan Cox
Ar Maw, 2006-08-15 am 09:17 -0500, ysgrifennodd Roger Heflin:
Quoted 3 lines
On Linux when interrupts are defined similar to below, what defines say
ide2, ide3 to be on the same interrupt? The bios, linux, the driver using
the interrupt? And can that be controlled/overrode at the kernel/driver level?
Quoted 10 lines
Only with a soldering iron. They are the way the system is wired. Moving boards between slots may change the IRQ allocation.
Quoted 2 lines
I have identified that the disks that are shared on ide2, ide3 do funny things when both are being heavily used (
dma_expiry), this is an older driver versionsThat could be occuring just through lack of PCI bus bandwidth.
Right now, all controllers are part of the NVIDIA chipset which should
mean the are on the same PCI bandwidth. The block diagram does indicate
the 4 sata channels as one unit, though I don't know exactly how things were internally done, performance testing seems to indicate that all 3 I am using are independent hardware as they don't seem to affect each others load.
I am currently retesting under 2.6.17.8 to see if I have similar issues there, under that it show interrupts like below, different interrupt numbers, but similar sharing as ata1/ata2, and ata3/ata4 are on the same interrupt.
ata1: SATA max UDMA/133 cmd 0xEC00 ctl 0xE802 bmdma 0xDC00 irq 16
ata2: SATA max UDMA/133 cmd 0xE400 ctl 0xE002 bmdma 0xDC08 irq 16
ata1: SATA link up 3.0 Gbps (SStatus 123)
ata1: dev 0 cfg 49:2f00 82:706b 83:7e61 84:4023 85:7069 86:3e41 87:4023 88:407f
ata1: dev 0 ATA-7, max UDMA/133, 321672960 sectors: LBA48
ata1: dev 0 configured for UDMA/133
ata2: SATA link up 3.0 Gbps (SStatus 123)
ata2: dev 0 cfg 49:2f00 82:706b 83:7e61 84:4023 85:7069 86:3e41 87:4023 88:407f
ata2: dev 0 ATA-7, max UDMA/133, 321672960 sectors: LBA48
ata2: dev 0 configured for UDMA/133
ata3: SATA max UDMA/133 cmd 0xD800 ctl 0xD402 bmdma 0xC800 irq 17
ata4: SATA max UDMA/133 cmd 0xD000 ctl 0xCC02 bmdma 0xC808 irq 17
ata3: SATA link up 3.0 Gbps (SStatus 123)
ata3: dev 0 cfg 49:2f00 82:706b 83:7e61 84:4023 85:7069 86:3e41 87:4023 88:407f
ata3: dev 0 ATA-7, max UDMA/133, 321672960 sectors: LBA48
ata3: dev 0 configured for UDMA/133
ata4: SATA link down (SStatus 0)
Roger
Ar Maw, 2006-08-15 am 10:24 -0500, ysgrifennodd Roger Heflin:
Quoted 4 lines
I am currently retesting under 2.6.17.8 to see if I have similar issues there, under that it show interrupts like below, different interrupt numbers,
but similar sharing as ata1/ata2, and ata3/ata4 are on the same interrupt.
Thats what I would expect to see - two channels per PCI device is the normal layout and they will always share the IRQ.
we've seen a lot of problems on ck804 chipsets when multiple devices share level-triggered interrupts. I think some of the earlier sample bioses assumed that interrupts would be configured via ACPI, and when ACPI is not used, the interrupts end up as level-triggered instead of edge-triggered.
you might investigate whether ACPI is failing to configure your system (are you on 2.4? updating to 2.6 might help), using the 'noapic' kernel parameter to force interrupts back to PIC, or the 'irqpoll' kernel parameter to try to work around the level-triggered interrupts.
Thanks,
Terence
Quoted 40 lines · On Tue, Aug 15, 2006 at 09:17:04AM -0500, [email protected] wrote:
Hello,
On Linux when interrupts are defined similar to below, what defines say
ide2, ide3 to be on the same interrupt? The bios, linux, the driver using
the interrupt? And can that be controlled/overrode at the
kernel/driver level?I have identified that the disks that are shared on ide2, ide3 do funny things when both are being heavily used (
dma_expiry), this is an older driver versions
but I have experienced it before with a lot newer driver, and a bios adjustment
previously fixed a similar issue, so that may be what is needed in this case also,
I am not sure how they fixed it, but I suspect that the setup the interrupt
to not be shared. I have a large number of machines and under heavy
loads all
seem to duplicate the issue, and it always happens with the disks on ide2/ide3,
never on the disk connected to ide4.CPU0 CPU1 CPU2 CPU3
0: 56616921 5359998 7002142 938817 XT-PIC timer 1: 8 88 96 0 IO-APIC-edge i8042 2: 0 0 0 0 XT-PIC cascade 4: 2091 100 208 2477 IO-APIC-edge serial 8: 0 0 0 0 IO-APIC-edge rtc 9: 0 0 0 0 IO-APIC-level acpi 20: 0 0 0 0 IO-APIC-level ehci_hcd 21: 0 950 401419 414482 IO-APIC-level ide4, ohci_hcd 22: 1165 1704243 576247 6796 IO-APIC-level ide2, ide3 47: 65971 0 0 0 IO-APIC-level eth0 NMI: 1 1 1 1 LOC: 69904264 69877733 69879541 69901903 ERR: 0 MIS: 105
Quoted 8 lines
Roger
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------
Hello.
Quoted 5 lines · Terence Ripperda
we've seen a lot of problems on ck804 chipsets when multiple devices share level-triggered interrupts. I think some of the earlier sample bioses assumed that interrupts would be configured via ACPI, and when ACPI is not used, the interrupts end up as level-triggered instead of edge-triggered.
Edge-triggered *shared* interrupts?! Now that sounds interesting (I'm not
saying impossible).
WBR, Sergei
Quoted 11 lines · On Tue, Aug 15, 2006 at 09:42:49PM +0400, [email protected] wrote:
Hello.
Quoted 5 lines · Terence Ripperda
we've seen a lot of problems on ck804 chipsets when multiple devices share level-triggered interrupts. I think some of the earlier sample bioses assumed that interrupts would be configured via ACPI, and when ACPI is not used, the interrupts end up as level-triggered instead of edge-triggered.
Edge-triggered *shared* interrupts?! Now that sounds interesting (I'm not saying impossible).
ah shoot, you're right. doing too much and got the two backwards. ignore my email, I'm going off to sit in the corner with a dunce cap on..
Quoted 2 lines
WBR, Sergei
-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------
Quoted 10 lines · Alan Cox
Ar Maw, 2006-08-15 am 10:24 -0500, ysgrifennodd Roger Heflin:
Quoted 3 lines
I am currently retesting under 2.6.17.8 to see if I have similar issues there, under that it show interrupts like below, different interrupt numbers, but similar sharing as ata1/ata2, and ata3/ata4 are on the same interrupt.
Thats what I would expect to see - two channels per PCI device is the normal layout and they will always share the IRQ.
After looking at other information that looks correct.
It looks like the older DMA recovery code never works on this chipset,
once it goes into DMA recovery it never comes out of it. I am looking
at that to see if anything can be done about it.
The problem I am having is that we cannot use a later kernel because it has some other issues (MPT fusion driver is 1/4 the speed of the older one for some unknown reason-this is in 2.6.17, the older MPT fusion works fine on exactly the same machine/cards/bios-I am not sure of the actual underlying casue of this slowdown).
On the specific kernel that I have I appear to have both IDE and sata_nv drivers, is there a way to force things to use sata_nv/libata rather than the older ide driver for the NVIDIA sata controller?
Roger
Ar Maw, 2006-08-15 am 12:47 -0500, ysgrifennodd Roger Heflin:
Quoted 3 lines
On the specific kernel that I have I appear to have both IDE and
sata_nvdrivers, is there a way to force things to use sata_nv/libata rather than the older ide driver for the NVIDIA sata controller?
You can build 2.6.18-mm like that (say N to AMD/NVIDIA PATA driver, y to sata_nv and pata_amd libata drivers). Still have to figure out why your MPT fusion is slow but if its still slow in 2.6.18-mm then it needs figuring out anyway
Quoted 4 lines · Roger Heflin
It looks like the older DMA recovery code never works on this chipset,
once it goes into DMA recovery it never comes out of it. I am looking
at that to see if anything can be done about it.
You could try booting with ide2=serialize as a kernel parameter. That should ensure the two channels are never in use simultaneously. Trades off a bit of performance for a bit of reliability.
Cheers
Ar Mer, 2006-08-16 am 18:54 -0400, ysgrifennodd Mark Lord:
Quoted 4 lines
Quoted 3 lines · Roger Heflin
It looks like the older DMA recovery code never works on this chipset, once it goes into DMA recovery it never comes out of it.
DMA recovery is fairly broken in drivers/ide especially if it tries to change mode. libata does not have this problem and I have no plans to even try and fix the drivers/ide code for this issue as its a major piece of work.
Quoted 11 lines · Alan Cox
Ar Mer, 2006-08-16 am 18:54 -0400, ysgrifennodd Mark Lord:
Quoted 3 lines
Quoted 2 lines · Roger Heflin
It looks like the older DMA recovery code never works on this chipset, once it goes into DMA recovery it never comes out of it.
DMA recovery is fairly broken in
drivers/ideespecially if it tries to change mode. libata does not have this problem and I have no plans to even try and fix thedrivers/idecode for this issue as its a major piece of work.
If already figured that since I was pretty sure all of it was
being rolled to libata. I was able to switch that controller and
disk to the libata sata_nv.
Roger