controlling ACPI IRQ routing
Hello.
I frequently work with certain Asterisk-compatible PCI telephony interfaces that use the "zaptel" module/driver. This module generates 1000 interrupts per second (yes, I know...) and any noticeable latency in handling that interrupt can cause significant problems for modulated data audio (such as fax) passing through those telephony interfaces.
It's become apparent that IRQ priorities seem to have to a lot to do with the performance of the zaptel driver. For example, if the zaptel device sits on interrupt 16, and the network interfaces and hard drive interfaces are on higher-numbered interrupts then things tend to go well. But if the zaptel device sits on interrupt 19, and the network interfaces and hard drive interfaces are on lesser-numbered interrupts (such as 16, 17, and 18) then things tend to go poorly.
The zaptel driver comes with a utility called "zttest" which basically indicates the reliability for the application to retrieve audio samples from the device driver before too much time passes... or in other words, the latency (or lack thereof) in handling device interrupts. A good zttest score on a well-configured system is 100%, although scores as low as 99.98% seem to be tolerable for modulated data audio. However, anything less... 99.97%, 99.96, I've even seen 99.91% and worse... these constitute poor scores and inevitably indicate that audio samples will be lost during usage. (You can see that the difference between "good" and "poor" is rather a narrow margin.) Lost audio samples is tolerable for voice applications, but it is detrimental to data/fax applications.
Usually the trick, then, has been to simply start rearranging the PCI cards in the system and disabling unused devices (such as serial, parallel, and USB ports with the respective drivers) until some combination was reached where zttest scores were good - inevitably meaning that the device had a high-priority (low-numbered) interrupt.
Recently this trick is seemingly more and more difficult to perform... and it's especially difficult in cases where (like on a 1U rack-mount server) there is but one PCI slot and the motherboard BIOS offers no helpful IRQ configuration mechanism. For example, a recent installation of Fedora 8 on such a server put the zaptel device at a low-priority interrupt. After hours of futile BIOS option changes and tests and even after testing a kernel (2.6.23.11) patched with Ingo Molnar's "rt" Real-Time patches (and using chrt for the associated IRQ to increase priority) nothing seemed to help. Then I simply reverted to an old 2.6.20 kernel from Fedora Core 5 that we had used successfully on previous deployments on similar hardware... and all was fine. And, of course, the apparent difference was that the 2.6.23.11 kernel put the device interrupt high-numbered while the 2.6.20 kernel put the device interrupt low-numbered.
Sometimes, however, it's just not ideal ... or even possible ... to downgrade to a "known good" kernel. It's certainly not desirable. So my question is, "How can we deliberately control the IRQ routing assignments made by ACPI so that certain PCI devices get the highest-priority interrupts?"
I've looked into the pirq kernel option ... only to find that it's insanely difficult to understand *how* to use... and simply trying it out with various values doesn't seem to accomplish much as far as I can tell. (Yes, I have read Documentation/i386/IO-APIC.txt many times... not that it reading it makes the pirq usage any easier to understand.)
I've looked into the setpci utility... but besides the man page for it being mostly unhelpful, it's difficult to know whether or not using setpci will actually do anything for us in this respect, as there seems to be a lot of conflicting information in various posts and mail archives on whether or not an IRQ assignment can actually be changed by using setpci. And then there's the fear that by using setpci incorrectly I may actually break the hardware.
I'm fairly certain that this is simply a matter of ACPI tuning... because with one kernel the interrupt assignments are one way while with another kernel the interrupt assignments are a different way... and both kernels use IO-APIC and IRQ routing by ACPI.
I would appreciate any feedback or assistance in this matter. How can we deliberately control the IRQ routing assignments made by ACPI so that certain PCI devices get the highest-priority interrupts?
Thanks,
Lee.
Quoted 87 lines · Lee Howard
Hello.
I frequently work with certain Asterisk-compatible PCI telephony interfaces that use the "zaptel" module/driver. This module generates 1000 interrupts per second (yes, I know...) and any noticeable latency in handling that interrupt can cause significant problems for modulated
data audio (such as fax) passing through those telephony interfaces.It's become apparent that IRQ priorities seem to have to a lot to do with the performance of the zaptel driver. For example, if the zaptel device sits on interrupt 16, and the network interfaces and hard drive interfaces are on higher-numbered interrupts then things tend to go well. But if the zaptel device sits on interrupt 19, and the network interfaces and hard drive interfaces are on lesser-numbered interrupts (such as 16, 17, and 18) then things tend to go poorly.
The zaptel driver comes with a utility called "zttest" which basically indicates the reliability for the application to retrieve audio samples
from the device driver before too much time passes... or in other words,
the latency (or lack thereof) in handling device interrupts. A good zttest score on a well-configured system is 100%, although scores as low
as 99.98% seem to be tolerable for modulated data audio. However, anything less... 99.97%, 99.96, I've even seen 99.91% and worse... these
constitute poor scores and inevitably indicate that audio samples will be lost during usage. (You can see that the difference between "good" and "poor" is rather a narrow margin.) Lost audio samples is tolerable for voice applications, but it is detrimental to data/fax applications.Usually the trick, then, has been to simply start rearranging the PCI cards in the system and disabling unused devices (such as serial, parallel, and USB ports with the respective drivers) until some combination was reached where zttest scores were good - inevitably meaning that the device had a high-priority (low-numbered) interrupt.
Recently this trick is seemingly more and more difficult to perform... and it's especially difficult in cases where (like on a 1U rack-mount server) there is but one PCI slot and the motherboard BIOS offers no helpful IRQ configuration mechanism. For example, a recent installation
of Fedora 8 on such a server put the zaptel device at a low-priority interrupt. After hours of futile BIOS option changes and tests and even
after testing a kernel (2.6.23.11) patched with Ingo Molnar's "rt" Real-Time patches (and using chrt for the associated IRQ to increase priority) nothing seemed to help. Then I simply reverted to an old 2.6.20 kernel from Fedora Core 5 that we had used successfully on previous deployments on similar hardware... and all was fine. And, of course, the apparent difference was that the 2.6.23.11 kernel put the device interrupt high-numbered while the 2.6.20 kernel put the device interrupt low-numbered.Sometimes, however, it's just not ideal ... or even possible ... to downgrade to a "known good" kernel. It's certainly not desirable. So my question is, "How can we deliberately control the IRQ routing assignments made by ACPI so that certain PCI devices get the highest-priority interrupts?"
I've looked into the pirq kernel option ... only to find that it's insanely difficult to understand *how* to use... and simply trying it out with various values doesn't seem to accomplish much as far as I can
tell. (Yes, I have readDocumentation/i386/IO-APIC.txtmany times... not that it reading it makes the pirq usage any easier to understand.)I've looked into the setpci utility... but besides the man page for it being mostly unhelpful, it's difficult to know whether or not using setpci will actually do anything for us in this respect, as there seems
to be a lot of conflicting information in various posts and mail archives on whether or not an IRQ assignment can actually be changed by
using setpci. And then there's the fear that by using setpci incorrectly I may actually break the hardware.I'm fairly certain that this is simply a matter of ACPI tuning... because with one kernel the interrupt assignments are one way while with
another kernel the interrupt assignments are a different way... and both
kernels use IO-APIC and IRQ routing by ACPI.
Quoted 4 lines
I would appreciate any feedback or assistance in this matter. How can we deliberately control the IRQ routing assignments made by ACPI so that
certain PCI devices get the highest-priority interrupts?
In IOAPIC mode, interrupt priority isn't related with the pin (in your case, irq 16 or 19), but the vector of the pin. How vector of a pin is allocated is quite random. Usually driver who calls pci_enable_device earier will get a lower priority vector.
Thanks,
Shaohua
Quoted 4 lines · Shaohua Li
In IOAPIC mode, interrupt priority isn't related with the pin (in your case, irq 16 or 19), but the vector of the pin. How vector of a pin is allocated is quite random. Usually driver who calls
pci_enable_deviceearier will get a lower priority vector.
You used the words "random" and "usually". Could you elaborate on the randomness and when usually doesn't apply?
So the means to prioritize a driver is to see that it gets installed earlier? There is no other mechanism to "reserve" pin vector allocations for a specific device or driver?
Thanks,
Lee.
Quoted 6 lines · Lee Howard
Quoted 4 lines · Shaohua Li
In IOAPIC mode, interrupt priority isn't related with the pin (in your case, irq 16 or 19), but the vector of the pin. How vector of a pin is allocated is quite random. Usually driver who calls
pci_enable_deviceearier will get a lower priority vector.
Quoted 7 lines
You used the words "random" and "usually". Could you elaborate on the
randomness and when usually doesn't apply?So the means to prioritize a driver is to see that it gets installed earlier? There is no other mechanism to "reserve" pin vector allocations for a specific device or driver?
Currently vector is allocated when pci_enable_device is called. So which vector is allocated depends on how many drivers already called the routine. The first vector is 0x31, later higher priority (higher) vector will be allocated. In latest kernel, a vector of a irq could be variable when irq affinity is set, so it's much complex. There is no existing method to reserve a vector for a device.
Thanks,
Shaohua
Len Brown
reviewerQuoted 16 lines · Shaohua Li
Quoted 6 lines · Lee Howard
Quoted 4 lines · Shaohua Li
In IOAPIC mode, interrupt priority isn't related with the pin (in your case, irq 16 or 19), but the vector of the pin. How vector of a pin is allocated is quite random. Usually driver who calls
pci_enable_deviceearier will get a lower priority vector.Quoted 7 lines
You used the words "random" and "usually". Could you elaborate on the
randomness and when usually doesn't apply?So the means to prioritize a driver is to see that it gets installed earlier? There is no other mechanism to "reserve" pin vector allocations for a specific device or driver?
Quoted 6 lines
Currently vector is allocated when
pci_enable_deviceis called. So which vector is allocated depends on how many drivers already called the routine. The first vector is0x31, later higher priority (higher) vector will be allocated. In latest kernel, a vector of a irq could be variable when irq affinity is set, so it's much complex. There is no existing method to reserve a vector for a device.
1000 interrupts/second isn't a lot on modern hardware. Indeed, many linux distros run with 1000 clock ticks/second today.
I don't understand why interrupt priority has anything to do with what you are seeing. To notice such a thing, you'd have to have a lot of competing interrupts firing at the same time and the messages queued up inside the LAPIC and the processor spending a large % of its time in interrupt context. (does top(1) say that you're running a large %sys?)
What is the total interrupt rate on the system when this device is doing 1000/second?
Are there multiple cores on the system? If so,
are the interrupts bound to certain cores or is
irqbalance running?
-Len
Quoted 10 lines · Len Brown
1000 interrupts/second isn't a lot on modern hardware. Indeed, many linux distros run with 1000 clock ticks/second today.
I don't understand why interrupt priority has anything to do with what you are seeing. To notice such a thing, you'd have to have a lot of competing interrupts firing at the same time and the messages queued up inside the LAPIC and the processor spending a large % of its time in interrupt context. (does top(1) say that you're running a large %sys?)
No, top says nothing unusual to me...
top - 13:36:50 up 12 days, 4:00, 1 user, load average: 0.01, 0.05, 0.05
Tasks: 189 total, 1 running, 185 sleeping, 0 stopped, 3 zombie Cpu(s): 0.7% us, 1.3% sy, 0.0% ni, 97.7% id, 0.0% wa, 0.0% hi, 0.3% si, 0.0% st Mem: 2054796k total, 2018560k used, 36236k free, 239084k buffers Swap: 2007992k total, 520k used, 2007472k free, 1266672k cached
Quoted 3 lines
What is the total interrupt rate on the system when this device is doing 1000/second?
I don't understand the question, really... and I'm not sure how to determine the answer, either. Here's what /proc/interrupts says:
# cat /proc/interrupts
CPU0
0: 1051464247 IO-APIC-edge timer
1: 8 IO-APIC-edge i8042
8: 0 IO-APIC-edge rtc
9: 0 IO-APIC-fasteoi acpi
12: 104 IO-APIC-edge i8042
14: 9414304 IO-APIC-edge ide0
16: 1051172722 IO-APIC-fasteoi wct4xxp
19: 1 IO-APIC-fasteoi eth1
21: 158008518 IO-APIC-fasteoi eth0
22: 6974044 IO-APIC-fasteoi libata
23: 7071112 IO-APIC-fasteoi libata
NMI: 0
LOC: 1051371544
ERR: 0Quoted 1 line
Are there multiple cores on the system?
No, otherwise it appears that I could use IRQ affinity to dedicate a processor to handling the wct4xxp (zaptel) interrupt.
Quoted 3 lines
If so,
are the interrupts bound to certain cores or is
irqbalance running?
irqbalance is available and running, but I don't think that it does anything on a single-core system.
Thanks,
Lee.
Quoted 8 lines · Lee Howard
# cat /proc/interrupts
CPU0
0: 1051464247 IO-APIC-edge timer
1: 8 IO-APIC-edge i8042
8: 0 IO-APIC-edge rtc
9: 0 IO-APIC-fasteoi acpi
12: 104 IO-APIC-edge i8042
14: 9414304 IO-APIC-edge ide0Quoted 10 lines
16: 1051172722 IO-APIC-fasteoi wct4xxp 19: 1 IO-APIC-fasteoi eth1 21: 158008518 IO-APIC-fasteoi eth0 22: 6974044 IO-APIC-fasteoi libata 23: 7071112 IO-APIC-fasteoi libata NMI: 0 LOC: 1051371544 ERR: 0
Based on that, you may want to use rtirq. It is a boot script (need schedutils as dependency) with associated config scripts. http://alsa.opensrc.org/Rtirq
In /etc/conf.d/rtirq, you will find:
# IRQ thread service names
# (space separated list, from higher to lower priority). RTIRQ_NAME_LIST="rtc snd usb i8042"
The most important things is that the devices listed here doesn't have any shared IRQ with some other device and that the rtc remain the first listed device (the one with the higer priority), or the system will hang soon or later.
This setup is for audio workstation but is very easy to adapt to any kind of work.
Dominique
-
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Len Brown
reviewerQuoted 32 lines · Dominique Michel
Quoted 8 lines · Lee Howard
# cat /proc/interrupts CPU0 0: 1051464247 IO-APIC-edge timer 1: 8 IO-APIC-edge i8042 8: 0 IO-APIC-edge rtc 9: 0 IO-APIC-fasteoi acpi 12: 104 IO-APIC-edge i8042 14: 9414304 IO-APIC-edge ide0Quoted 10 lines
16: 1051172722 IO-APIC-fasteoi wct4xxp 19: 1 IO-APIC-fasteoi eth1 21: 158008518 IO-APIC-fasteoi eth0 22: 6974044 IO-APIC-fasteoi libata 23: 7071112 IO-APIC-fasteoi libata NMI: 0 LOC: 1051371544 ERR: 0Based on that, you may want to use rtirq. It is a boot script (need schedutils as dependency) with associated config scripts. http://alsa.opensrc.org/Rtirq
In
/etc/conf.d/rtirq, you will find:# IRQ thread service names
# (space separated list, from higher to lower priority). RTIRQ_NAME_LIST="rtc snd usb i8042"
Quoted 7 lines
The most important things is that the devices listed here doesn't have any shared IRQ with some other device and that the rtc remain the first listed device (the one with the higer priority), or the system will hang soon or later.
This setup is for audio workstation but is very easy to adapt to any kind of work.
I've got no experience with the thread priority thing above. However, the better way to handle quality of service contention is to not do the things you don't care about.
Does the system really need HZ=1000? you could slow that down to 100, or run CONFIG_NO_HZ to reduce it even more...
Also, on a UP, the LAPIC timer interrupt is redundant -- you could build your kernel w/o APIC support or boot with nolapic.
This would get rid of the only interrupts which are frequent enough to be competing with the device, clock ticks.
-Len
-
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 48 lines · Len Brown
Quoted 33 lines · Dominique Michel
Quoted 8 lines · Lee Howard
# cat /proc/interrupts CPU0 0: 1051464247 IO-APIC-edge timer 1: 8 IO-APIC-edge i8042 8: 0 IO-APIC-edge rtc 9: 0 IO-APIC-fasteoi acpi 12: 104 IO-APIC-edge i8042 14: 9414304 IO-APIC-edge ide0Quoted 11 lines
16: 1051172722 IO-APIC-fasteoi wct4xxp 19: 1 IO-APIC-fasteoi eth1 21: 158008518 IO-APIC-fasteoi eth0 22: 6974044 IO-APIC-fasteoi libata 23: 7071112 IO-APIC-fasteoi libata NMI: 0 LOC: 1051371544 ERR: 0Based on that, you may want to use rtirq. It is a boot script (need schedutils as dependency) with associated config scripts. http://alsa.opensrc.org/Rtirq
In
/etc/conf.d/rtirq, you will find:# IRQ thread service names
# (space separated list, from higher to lower priority). RTIRQ_NAME_LIST="rtc snd usb i8042"Quoted 8 lines
The most important things is that the devices listed here doesn't have any shared IRQ with some other device and that the rtc remain the first listed device (the one with the higer priority), or the system will hang soon or later.
This setup is for audio workstation but is very easy to adapt to any kind of work.
I've got no experience with the thread priority thing above. However, the better way to handle quality of service contention is to not do the things you don't care about.
Quoted 7 lines
Does the system really need HZ=1000? you could slow that down to 100, or run
CONFIG_NO_HZto reduce it even more...
Also, on a UP, the LAPIC timer interrupt is redundant -- you could build your kernel w/o APIC support or boot with nolapic.This would get rid of the only interrupts which are frequent enough to be competing with the device, clock ticks.
If I boot with nolapic then XT-PIC is used, and the wcfxo device ends up sharing an IRQ with libata which results in worse performance than with it the way that it is.
Thanks,
Lee.
-
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 59 lines · Lee Howard
Quoted 58 lines · Len Brown
Quoted 41 lines · Dominique Michel
Quoted 7 lines · Lee Howard
# cat /proc/interrupts CPU0 0: 1051464247 IO-APIC-edge timer 1: 8 IO-APIC-edge i8042 8: 0 IO-APIC-edge rtc 9: 0 IO-APIC-fasteoi acpi 12: 104 IO-APIC-edge i8042 14: 9414304 IO-APIC-edge ide0Quoted 11 lines
16: 1051172722 IO-APIC-fasteoi wct4xxp 19: 1 IO-APIC-fasteoi eth1 21: 158008518 IO-APIC-fasteoi eth0 22: 6974044 IO-APIC-fasteoi libata 23: 7071112 IO-APIC-fasteoi libata NMI: 0 LOC: 1051371544 ERR: 0Based on that, you may want to use rtirq. It is a boot script (need schedutils
as dependency) with associated config scripts.
http://alsa.opensrc.org/RtirqIn
/etc/conf.d/rtirq, you will find:# IRQ thread service names
# (space separated list, from higher to lower priority). RTIRQ_NAME_LIST="rtc snd usb i8042"The most important things is that the devices listed here doesn't have any
shared IRQ with some other device and that the rtc remain the first listed
device (the one with the higer priority), or the system will hang soon or later.Quoted 5 lines
This setup is for audio workstation but is very easy to adapt to any kind of
work.I've got no experience with the thread priority thing above. However, the better way to handle quality of service contention is to not do the things you don't care about.
Does the system really need HZ=1000? you could slow that down to 100, or run
CONFIG_NO_HZto reduce it even more...
Also, on a UP, the LAPIC timer interrupt is redundant -- you could build your kernel w/o APIC support or boot with nolapic.
Quoted 7 lines
Quoted 2 lines
This would get rid of the only interrupts which are frequent enough to be competing with the device, clock ticks.
If I boot with nolapic then XT-PIC is used, and the wcfxo device ends up sharing an IRQ with libata which results in worse performance than with it the way that it is.
Try nolapic_timer ?
-
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 6 lines · Shaohua Li
Currently vector is allocated when
pci_enable_deviceis called. So which vector is allocated depends on how many drivers already called the routine. The first vector is0x31, later higher priority (higher) vector will be allocated. In latest kernel, a vector of a irq could be variable when irq affinity is set, so it's much complex. There is no existing method to reserve a vector for a device.
The systems I've been using are all single-processor AMD x86 or x86_64 systems. So I doubt that IRQ affinity applies in these cases.
I've made the driver get "preloaded" through mkinitrd/initrd on boot-time, but that doesn't seem to make a difference in device performance.
Is it possible to see in 'dmesg' output which vector was allocated to which IRQ/device?
Thanks,
Lee.
Len Brown
reviewerQuoted 8 lines · Lee Howard
Quoted 6 lines · Shaohua Li
Currently vector is allocated when
pci_enable_deviceis called. So which vector is allocated depends on how many drivers already called the routine. The first vector is0x31, later higher priority (higher) vector will be allocated. In latest kernel, a vector of a irq could be variable when irq affinity is set, so it's much complex. There is no existing method to reserve a vector for a device.
Quoted 8 lines
The systems I've been using are all single-processor AMD x86 or
x86_64systems. So I doubt that IRQ affinity applies in these cases.I've made the driver get "preloaded" through mkinitrd/initrd on boot-time, but that doesn't seem to make a difference in device performance.
Is it possible to see in 'dmesg' output which vector was allocated to which IRQ/device?
no. the vectors are chosesn when the IOAPIC is programmed and today, linux just prints out the GSI (which is the IOAPIC pin #) and the IRQ, which is a corresponding data structure.
Quoted 6 lines
Quoted 2 lines
In IOAPIC mode, interrupt priority isn't related with the pin (in your case, irq 16 or 19), but the vector of the pin.
Just to help my understanding a bit... does the Linux ACPI driver determine which IRQ numbers are given to certain devices? And if so, how does it make those assignments?
Yes. It uses a PCI Routing Table (_PRT) in the DSDT. The _PRT may either hard-code the relationship between PCI interrupt pin (as is typical in IOAPIC mode)
or use a PCI Interrupt Link Device (as is common in PIC mode) that can be programmed to a number of differet input pins.
This is used to minimize interrupt line sharing. But on your system, you have no interrupt line sharing, so that isnt' the problem.
-Len
(I'm still looking into this matter...)
Quoted 2 lines · Shaohua Li
In IOAPIC mode, interrupt priority isn't related with the pin (in your case, irq 16 or 19), but the vector of the pin.
Just to help my understanding a bit... does the Linux ACPI driver determine which IRQ numbers are given to certain devices? And if so, how does it make those assignments?
Thanks,
Lee.