avoid IRQ0 ioapic pin collision
Subsystems (2)
Quoted 3 lines
IRQ is an interrupt request, from a device. Currently they can come in over a pin, or over a packet. IRQs at the source can be shared.
Well stated.
Quoted 4 lines
For an IRQ number it is just an enumeration of interrupt sources. Ideally in a stable manner so that the assigned number does not depend on compile options, or which version of the kernel you are running.
While desirable, the "stable" part is not guaranteed.
On an ACPI system with PCI Interrupt Link devices, request_irq() goes into the pci-enable-device path,
which ends up asking ACPI what interrupt pin this device uses. But the pin is programmable, so we pick one from the list of possible, program the router accordingly, and this is what gets returned to request_irq(). ACPI PCI Interrupt Link Devices, of course, are just a wrapper for what x86 legacy calls PIRQ routers.
In practice, if you boot the same or similar kernel on the same configuration, you get 'stable', but if something changes such as enabling/disabling a device or loading an additional device driver
or otherwise tinkering with probe order, then all bets are off, since the sequence of selecting and
balancing possible device interrupt lines between available IRQs has changed.
Quoted 9 lines
Quoted 3 lines
Re: irq making sense
Depends on the defintion, I suppose. I'm still looking for one that matches how Linux uses the term.The global
irq_descarray is the core of any definition. That is how linux knows about irqs. Beyond that we ideally have stable numbers across reboots, and recompiles, and hardware addition and removal. Stable numbers are not really possible but we can come quite close.
Okay, I'm good with irq_desc, Linux's generic IRQ definition, being the center of the universe. I'm also good with the index into irq_desc[] being the "IRQ number" show in /proc/interrupts. I guess I was thinking from a HW-specific point of view this afternoon...
I don't like vector numbers being exposed in /proc/interrupts on x86. On an ACPI-enabled system, I think that they should be the GSI. This is consistent with the 0-15 legacy definitions being pin numbers (which I think we must keep), and consistent with the 16-n numbers being IOAPIC pin numbers (+offset), (which I think we should have kept).
Of course, to do this simply, we'd have to select the irq_desc[i] with i = gsi, and not i = vector.
Quoted 8 lines
Quoted 3 lines
We should never have had a problem with un-connected interrupt lines consuming vectors, as the vectors are handed out at run-time only when interrupts are requested by drivers.
Incorrect. By being requested by drivers I assume you mean by
request_irq.assign_irq_vectoris called long beforerequest_irqand in fact there is currently not a call fromrequest_irqtoassign_irq_vector. Look whereassign_irq_vectoris called inio_apic.c
You are right. This code is wrong.
It makes absolutely no sense to reserve vectors in advance for every RTE in the IOAPIC when we don't even know if they are going to be used.
This is clearly a holdover from the early IOAPIC/MPS days when we were talking about 4 to 8 non-legacy RTEs.
This is where the big system vector shortage problem should be addressed.
Quoted 3 lines
I think there is a legitimate case for legacy edge triggered interrupts to request a vector sooner, as there are so many races in the enable/disable paths.
I don't follow you on this part.
Quoted 24 lines
Quoted 10 lines
Quoted 9 lines
Problem 2.
Some systems have more than 224 GSIs that are actually connected to devices.
There are three possible ways to handle this case.
- Fail after we run out of vectors.
- Share a vector.- Allow vectors of different cpus to handle different irqs. The is the most elegant and scalable, and Natalie's suggestion.
Quoted 6 lines
So here we allow the same vector to be used by different IOAPICS, or IOAPIC pins, but have them it directed to different CPUs who have per-cpu tables to vector to different devices?
A practical workaround? Certainly.
An elegant solution? No, that would require better hardware;-)Agreed. But we lost elegant at the point of on demand assigning of vectors to irqs.
Quoted 3 lines
The problem with this workaround is going to be choose a policy of where to direct what, and how to move things if interrupts become un-balanced.
Quoted 7 lines
The directing problem already exists. In general an I/O apic can only direct an irq at a specific cpu, and linux already supports cpusets on which an irq may be delivered.
But yes on systems with 8 or fewer cpus the I/O apics can do the directing themselves and it is likely we still want to handle that case.
I don't think it is important to optimize for <= 8 CPUs having hundreds of unique interrupts. Systems with huge numbers of interrutps will have more than 8 CPUs. If they don't, then the should function, but being optimal isn't what the administrator had in mind. With multiple cores being added to systems, this becomes even more true over time.
Quoted 28 lines
Quoted 8 lines
Quoted 8 lines
So what would be a path to get there from here?
- Fix the irqset_affinitycode so that it makes the changes to irqs when the interrupt is actually disabled.
Callingdesc->handler->disable,desc->handler->enabledoes not work immediately so the current code is racy. Which shows up very badly if you attempt to change the irq vector, and may cause rare problems todayQuoted 8 lines
Quoted 2 lines
- Modify the MSI code to allocate irqs and not vectors. So we don't have two paths through the code, for no good reason.
Modify the MSI code to allocate "something to do with a specific interrupt". Hmmm, but it does this already:-)
Seriously, aren't the interrupt vectors, or perhaps the vector/cpu pair, the fundamental resource being allocated here?
So there are two pieces. struct
irq_descthat describes an interrupt source, and the vector that catches the interrupt source and tells the cpu about it.To be able to manage an irq source we need to enumerate and give it a struct
irq_descentry in the globalirq_descarray. One of the operations we perform after that point is to allocate it a vector so we can catch that irq source. But other operations such as reporting, and the assignment of cpu affinity also happen based on the entry in theirq_descarray.
Quoted 5 lines
Except for architecture implementations of irq routing the vectors that we catch irqs with should be completely invisible. The ideal architecture would not even have a separate concept and except for the msi code linux has not exported the concept of vectors outside of the architecture implementations.
Agreed.
Quoted 6 lines
Quoted 3 lines
what about per/irq kstats?
would you keep stats on a per-vector basis
and translate back to irqs for/proc/interrupts?No. What we have on a per irq basis is what makes sense and I would leave that untouched.
Agreed.
Quoted 4 lines
Quoted 2 lines
what about when the same vector is independently
used by multiple CPUS?Accounting by irq makes sense.
Agreed.
Quoted 25 lines
Quoted 3 lines
Quoted 1 line
- Remove the current irq compression.
maybe this can be moved up in the to-do list?
Not much.
- The currentset_affinitycode has a real bug.
- The MSI mismatch is worse and fixing it reduces testing later.
- We need to raise the number of irqs we can support. Either by vastly increasing the number of stubs, whose number looks hard to control at compile time, or by turning this into a data problem.Quoted 8 lines
Quoted 8 lines
- Move irq vector allocation/deallocation into
request_irq, andfree_irq.- Make
vector_irqper_cpu.- Modify
assign_irqto allocate different vectors on different cpus, to fail if we cannot find a free irq vector somewhere in the irqs cpu set, and callassign_irqfromset_affinitysowhen we change
Quoted 4 lines
Quoted 4 lines
cpus we can allocate a different irq.
I have proof of concept level patches for everything thing except making MSI actually allocate irqs, but that should be straight
Quoted 8 lines
Quoted 5 lines
forward.
except that irqs should be allocating vectors,
the fundamental currency here, and MSI should
be allocating the same, no?Not quite. The MSI code should first allocate
anirq_descentry get an IRQ number.
You're right. I agree.
Quoted 11 lines
Then when the driver calls
request_irqthe architecture specific code should arrange for__do_IRQto be called with the appropriate irq number when the interrupt happens. The architecture specific code needs to do this by setting the MSI address and the MSI data to an architecture specific values, that on x86 will involve the a vector.It is reassuring to see that the way I want to make the code work tends to be they way you already think the code works. I can't be too far off in reducing the complexity.
Yeah, thanks for pointing out that the vectors are assigned at IOAPIC init-time. I hadn't noticed that, and it needs to be addressed.
-Len
Quoted 7 lines · Len
You are right. This code is wrong.
It makes absolutely no sense to reserve vectors in advance for every RTE in the IOAPIC when we don't even know if they are going to be used.This is clearly a holdover from the early IOAPIC/MPS days when we were talking about 4 to 8 non-legacy RTEs.
Yes I agree. A lot of the IO-APIC code could probably need some renovation.
Quoted 2 lines
This is where the big system vector shortage problem should be addressed.
If we go to per CPU IDTs it will be much less pressing, but still a good idea.
-Andi
P.S.: There seems to be a lot of confusion about all this. Maybe it would make sense to do a write up defining all the terms and stick it into Documentation/* ?
RFC PATCH 1/1 Document what in IRQ is.
Quoted 3 lines · Andi Kleen
P.S.: There seems to be a lot of confusion about all this. Maybe it would make sense to do a write up defining all the terms and stick it intoDocumentation/*?
How does this look?
I am pretty horrible when it comes to Documentation, but this seems to be the essence of what I was saying earlier.
Eric
Documentation/IRQ.txt new file +22 -0
@@ -0,0 +1,22 @@ +What is an IRQ? + +An IRQ is an interrupt request from a device. +Currently they can come in over a pin, or over a packet. +IRQs at the source can be shared. + +An IRQ number is a kernel identifier used to talk about a hardware +interrupt source. Typically this is an index into the global irq_desc +array, but except for what linux/interrupt.h implements the details +are architecture specific. + +An IRQ number is an enumeration of the possible interrupt sources on a +machine. Typically what is enumerated is the number of input pins on +all of the interrupt controller in the system. In the case of ISA +what is enumerated are the 16 input pins to the pair of i8259 +interrupt controllers. + +Architectures can assign additional meaning to the IRQ numbers, and +are encouraged to in the case where there is any manual configuration +of the hardware involved. The ISA IRQ case on x86 where anyone who +has been around a while can tell you how the first 16 IRQs map to the +input pins on a pair of i8259s is the classic example.
Re: [RFC][PATCH] Document what in IRQ is.
Quoted 20 lines · On Tue, 02 May 2006 07:52:22 -0600 Eric W. Biederman wrote:
Quoted 3 lines · Andi Kleen
P.S.: There seems to be a lot of confusion about all this. Maybe it would make sense to do a write up defining all the terms and stick it intoDocumentation/*?How does this look?
I am pretty horrible when it comes to Documentation, but this seems to be the essence of what I was saying earlier.
Eric
Documentation/IRQ.txt new file +0
-0
Documentation/IRQ.txt line 4
Quoted 4 lines
Documentation/IRQ.txt new file +4
-0@@ -0,0 +1,22 @@ +What is an IRQ? + +An IRQ is an interrupt request from a device. +Currently they can come in over a pin, or over a packet.
No comma. Change packet to message?
Documentation/IRQ.txt line 5
Quoted 1 line
Documentation/IRQ.txt new file +1
-0+IRQs at the source can be shared.
Huh? That simple sentence confuses me. Should "source" really be "sink" or "destination"? Or maybe say "IRQs at an interrupt controller can be shared." Or is that too hardware-specific?
Anyway, what source is meant here? It doesn't mean that IRQs at the producer device can be shared, right? It's more at the consumer device where they can be shared.
Documentation/IRQ.txt line 14
Quoted 8 lines
Documentation/IRQ.txt new file +8
-0+An IRQ number is a kernel identifier used to talk about a hardware +interrupt source. Typically this is an index into the global irq_desc +array, but except for what linux/interrupt.h implements the details +are architecture specific. + +An IRQ number is an enumeration of the possible interrupt sources on a +machine. Typically what is enumerated is the number of input pins on +all of the interrupt controller in the system. In the case of ISA
controllers
Documentation/IRQ.txt line 15
Quoted 1 line
Documentation/IRQ.txt new file +1
-0+what is enumerated are the 16 input pins to the pair of i8259
is
Documentation/IRQ.txt line 21
Quoted 6 lines
Documentation/IRQ.txt new file +6
-0+interrupt controllers. + +Architectures can assign additional meaning to the IRQ numbers, and +are encouraged to in the case where there is any manual configuration +of the hardware involved. The ISA IRQ case on x86 where anyone who +has been around a while can tell you how the first 16 IRQs map to the
awhile
Documentation/IRQ.txt line 22
Quoted 1 line
Documentation/IRQ.txt new file +1
-0+input pins on a pair of i8259s is the classic example.
---
~Randy
Re: [RFC][PATCH] Document what in IRQ is.
Quoted 16 lines · Randy.Dunlap
Quoted 14 lines · On Tue, 02 May 2006 07:52:22 -0600 Eric W. Biederman wrote:
Quoted 3 lines · Andi Kleen
P.S.: There seems to be a lot of confusion about all this. Maybe it would make sense to do a write up defining all the terms and stick it intoDocumentation/*?How does this look?
I am pretty horrible when it comes to Documentation, but this seems to be the essence of what I was saying earlier.
Eric
Quoted 12 lines
Documentation/IRQ.txt line 4
Quoted 10 lines
Documentation/IRQ.txt new file +4
-0@@ -0,0 +1,22 @@ +What is an IRQ? + +An IRQ is an interrupt request from a device. +Currently they can come in over a pin, or over a packet.No comma. Change packet to message?
Sounds good.
Quoted 8 lines
Documentation/IRQ.txt line 5
Quoted 1 line
Documentation/IRQ.txt new file +1
-0+IRQs at the source can be shared.Huh? That simple sentence confuses me. Should "source" really be "sink" or "destination"? Or maybe say "IRQs at an interrupt controller can be shared." Or is that too hardware-specific?
Anyway, what source is meant here? It doesn't mean that IRQs at the producer device can be shared, right? It's more at the consumer device where they can be shared.
By source I was thinking at the irq controller pin.
Interrupts are usually thrown from interrupt controllers to something in the chipset that interrupts the cpu, giving the cpu a token (ie an interrupt vector) that uniquely identifies which interrupt source threw the interrupt.
Linux does not have generic infrastructure to allow two interrupt sources to share the same token passed to the kernel.
In addition there are good reasons on some systems to change the token dynamically, (say to point the IRQ at a different CPU). So no generic code in the code should know about the token the cpu receives. The fact that msi.c actually knows about that token today makes is inflexible and maintenance problem.
I guess I need to figure out how to work this additional information into my documentation then.
Quoted 10 lines
Documentation/IRQ.txt line 14
Quoted 8 lines
Documentation/IRQ.txt new file +8
-0+An IRQ number is a kernel identifier used to talk about a hardware +interrupt source. Typically this is an index into the global irq_desc +array, but except for what linux/interrupt.h implements the details +are architecture specific. + +An IRQ number is an enumeration of the possible interrupt sources on a +machine. Typically what is enumerated is the number of input pins on +all of the interrupt controller in the system. In the case of ISAcontrollers
Documentation/IRQ.txt line 15
Quoted 1 line
Documentation/IRQ.txt new file +1
-0+what is enumerated are the 16 input pins to the pair of i8259
Quoted 9 lines
is
Documentation/IRQ.txt line 21
Quoted 6 lines
Documentation/IRQ.txt new file +6
-0+interrupt controllers. + +Architectures can assign additional meaning to the IRQ numbers, and +are encouraged to in the case where there is any manual configuration +of the hardware involved. The ISA IRQ case on x86 where anyone who +has been around a while can tell you how the first 16 IRQs map to theawhile
Quoted 1 line
Documentation/IRQ.txt new file +1
-0+input pins on a pair of i8259s is the classic example.
Eric
Quoted 20 lines · Len
Quoted 3 lines
IRQ is an interrupt request, from a device. Currently they can come in over a pin, or over a packet. IRQs at the source can be shared.
Well stated.
Quoted 4 lines
For an IRQ number it is just an enumeration of interrupt sources. Ideally in a stable manner so that the assigned number does not depend on compile options, or which version of the kernel you are running.
While desirable, the "stable" part is not guaranteed.
On an ACPI system with PCI Interrupt Link devices,request_irq()goes into the pci-enable-device path,
which ends up asking ACPI what interrupt pin this device uses. But the pin is programmable, so we pick one from the list of possible, program the router accordingly, and this is what gets returned torequest_irq(). ACPI PCI Interrupt Link Devices, of course, are just a wrapper for what x86 legacy calls PIRQ routers.
Quoted 8 lines
In practice, if you boot the same or similar kernel on the same configuration, you get 'stable', but if something changes such as enabling/disabling a device or loading an additional device driver
or otherwise tinkering with probe order, then all bets are off, since the sequence of selecting and
balancing possible device interrupt lines between available IRQs has changed.
Agreed in the general case there is nothing we can do. But with a reasonable degree of stability things will stay put long enough for people to reason about them.
Although I would actually expect us to use the normal twiddle of irq lines on a bus. But if we can have less contention that is certainly better.
Quoted 11 lines
Quoted 5 lines
The global
irq_descarray is the core of any definition. That is how linux knows about irqs. Beyond that we ideally have stable numbers across reboots, and recompiles, and hardware addition and removal. Stable numbers are not really possible but we can come quite close.Okay, I'm good with
irq_desc, Linux's generic IRQ definition, being the center of the universe. I'm also good with the index into irq_desc[] being the "IRQ number" show in/proc/interrupts. I guess I was thinking from a HW-specific point of view this afternoon...
I had to stare at the code for a while to see this as well.
Quoted 6 lines
I don't like vector numbers being exposed in
/proc/interruptson x86. On an ACPI-enabled system, I think that they should be the GSI. This is consistent with the 0-15 legacy definitions being pin numbers (which I think we must keep), and consistent with the 16-n numbers being IOAPIC pin numbers (+offset), (which I think we should have kept).
Agreed.
Quoted 2 lines
Of course, to do this simply, we'd have to select the irq_desc[i] with i = gsi, and not i = vector.
Exactly.
Quoted 14 lines
Quoted 8 lines
Quoted 3 lines
We should never have had a problem with un-connected interrupt lines consuming vectors, as the vectors are handed out at run-time only when interrupts are requested by drivers.
Incorrect. By being requested by drivers I assume you mean by
request_irq.assign_irq_vectoris called long beforerequest_irqand in fact there is currently not a call fromrequest_irqtoassign_irq_vector. Look whereassign_irq_vectoris called inio_apic.cYou are right. This code is wrong.
It makes absolutely no sense to reserve vectors in advance for every RTE in the IOAPIC when we don't even know if they are going to be used.
Quoted 5 lines
This is clearly a holdover from the early IOAPIC/MPS days when we were talking about 4 to 8 non-legacy RTEs.
This is where the big system vector shortage problem should be addressed.
Yes. Natalie mentioned that there are some IBM systems that still run out of processor vectors with our current irq collapsing.
Quoted 5 lines
Quoted 3 lines
I think there is a legitimate case for legacy edge triggered interrupts to request a vector sooner, as there are so many races in the enable/disable paths.
I don't follow you on this part.
For edge triggered legacy irqs coming off the i8259 it probably makes sense to assign them vectors early like we do currently.
I believe although I am not certain that with edge triggered irqs actually disabling them makes it easy to miss edges.
Quoted 13 lines
Quoted 5 lines
Quoted 4 lines
The problem with this workaround is going to be choose a policy of where to direct what, and how to move things if interrupts become un-balanced.
Quoted 7 lines
The directing problem already exists. In general an I/O apic can only direct an irq at a specific cpu, and linux already supports cpusets on which an irq may be delivered.
But yes on systems with 8 or fewer cpus the I/O apics can do the directing themselves and it is likely we still want to handle that case.
Quoted 7 lines
I don't think it is important to optimize for <= 8 CPUs having hundreds of unique interrupts. Systems with huge numbers of interrutps will have more than 8 CPUs. If they don't, then the should function, but being optimal isn't what the administrator had in mind. With multiple cores being added to systems, this becomes even more true over time.
Sorry. To be clear for systems with <= 8 CPUs we need to allocate a common vector for all 8 cpus at once if we want to use flat mode and this makes a version of assign_irq_vector that can work assign per cpu vectors more complicated, since it has to handle both cases. Hmm, should assign_irq_vector become a ioapic method?
I don't think we want to loose the optimization of letting the hardware select select the cpu on small systems.
Quoted 6 lines
Quoted 3 lines
It is reassuring to see that the way I want to make the code work tends to be they way you already think the code works. I can't be too far off in reducing the complexity.
Yeah, thanks for pointing out that the vectors are assigned at IOAPIC init-time. I hadn't noticed that, and it needs to be addressed.
Exactly. Once that is fixed setting gsi == irq should be trivial. Then we only have to worry about big systems that actually use more than 244 IRQs. Which is where the per cpu vector assignment comes in.
Eric