Skip to contentSkip to conversation

avoid IRQ0 ioapic pin collision

PATCH In review Brown, Len 4 participants 6 messages May 2, 2006 – May 4, 2006

Subsystems (2)
lore
Share & tools
lore

Brown, Len

author
#0 lore raw
Quoted 3 lines

IRQ is an interrupt request, from a device. Currently they can come in over a pin, or over a packet. IRQs at the source can be shared.

Well stated.

Quoted 4 lines

For an IRQ number it is just an enumeration of interrupt sources. Ideally in a stable manner so that the assigned number does not depend on compile options, or which version of the kernel you are running.

While desirable, the "stable" part is not guaranteed.
On an ACPI system with PCI Interrupt Link devices, request_irq() goes into the pci-enable-device path,
which ends up asking ACPI what interrupt pin this device uses. But the pin is programmable, so we pick one from the list of possible, program the router accordingly, and this is what gets returned to request_irq(). ACPI PCI Interrupt Link Devices, of course, are just a wrapper for what x86 legacy calls PIRQ routers.

In practice, if you boot the same or similar kernel on the same configuration, you get 'stable', but if something changes such as enabling/disabling a device or loading an additional device driver
or otherwise tinkering with probe order, then all bets are off, since the sequence of selecting and
balancing possible device interrupt lines between available IRQs has changed.

Quoted 9 lines
Quoted 3 lines

Re: irq making sense
Depends on the defintion, I suppose. I'm still looking for one that matches how Linux uses the term.

The global irq_desc array is the core of any definition. That is how linux knows about irqs. Beyond that we ideally have stable numbers across reboots, and recompiles, and hardware addition and removal. Stable numbers are not really possible but we can come quite close.

Okay, I'm good with irq_desc, Linux's generic IRQ definition, being the center of the universe. I'm also good with the index into irq_desc[] being the "IRQ number" show in /proc/interrupts. I guess I was thinking from a HW-specific point of view this afternoon...

I don't like vector numbers being exposed in /proc/interrupts on x86. On an ACPI-enabled system, I think that they should be the GSI. This is consistent with the 0-15 legacy definitions being pin numbers (which I think we must keep), and consistent with the 16-n numbers being IOAPIC pin numbers (+offset), (which I think we should have kept).

Of course, to do this simply, we'd have to select the irq_desc[i] with i = gsi, and not i = vector.

Quoted 8 lines
Quoted 3 lines

We should never have had a problem with un-connected interrupt lines consuming vectors, as the vectors are handed out at run-time only when interrupts are requested by drivers.

Incorrect. By being requested by drivers I assume you mean by request_irq. assign_irq_vector is called long before request_irq and in fact there is currently not a call from request_irq to assign_irq_vector. Look where assign_irq_vector is called in io_apic.c

You are right. This code is wrong.
It makes absolutely no sense to reserve vectors in advance for every RTE in the IOAPIC when we don't even know if they are going to be used.

This is clearly a holdover from the early IOAPIC/MPS days when we were talking about 4 to 8 non-legacy RTEs.

This is where the big system vector shortage problem should be addressed.

Quoted 3 lines

I think there is a legitimate case for legacy edge triggered interrupts to request a vector sooner, as there are so many races in the enable/disable paths.

I don't follow you on this part.

Quoted 24 lines
Quoted 10 lines
Quoted 9 lines

Problem 2.
Some systems have more than 224 GSIs that are actually connected to devices.
There are three possible ways to handle this case.
- Fail after we run out of vectors.
- Share a vector.

- Allow vectors of different cpus to handle different irqs. The is the most elegant and scalable, and Natalie's suggestion.

Quoted 6 lines

So here we allow the same vector to be used by different IOAPICS, or IOAPIC pins, but have them it directed to different CPUs who have per-cpu tables to vector to different devices?

A practical workaround? Certainly.
An elegant solution? No, that would require better hardware;-)

Agreed. But we lost elegant at the point of on demand assigning of vectors to irqs.

Quoted 3 lines

The problem with this workaround is going to be choose a policy of where to direct what, and how to move things if interrupts become un-balanced.

Quoted 7 lines

The directing problem already exists. In general an I/O apic can only direct an irq at a specific cpu, and linux already supports cpusets on which an irq may be delivered.

But yes on systems with 8 or fewer cpus the I/O apics can do the directing themselves and it is likely we still want to handle that case.

I don't think it is important to optimize for <= 8 CPUs having hundreds of unique interrupts. Systems with huge numbers of interrutps will have more than 8 CPUs. If they don't, then the should function, but being optimal isn't what the administrator had in mind. With multiple cores being added to systems, this becomes even more true over time.

Quoted 28 lines
Quoted 8 lines
Quoted 8 lines

So what would be a path to get there from here?
- Fix the irq set_affinity code so that it makes the changes to irqs when the interrupt is actually disabled.
Calling desc->handler->disable, desc->handler->enable does not work immediately so the current code is racy. Which shows up very badly if you attempt to change the irq vector, and may cause rare problems today

Quoted 8 lines
Quoted 2 lines

- Modify the MSI code to allocate irqs and not vectors. So we don't have two paths through the code, for no good reason.

Modify the MSI code to allocate "something to do with a specific interrupt". Hmmm, but it does this already:-)

Seriously, aren't the interrupt vectors, or perhaps the vector/cpu pair, the fundamental resource being allocated here?

So there are two pieces. struct irq_desc that describes an interrupt source, and the vector that catches the interrupt source and tells the cpu about it.

To be able to manage an irq source we need to enumerate and give it a struct irq_desc entry in the global irq_desc array. One of the operations we perform after that point is to allocate it a vector so we can catch that irq source. But other operations such as reporting, and the assignment of cpu affinity also happen based on the entry in the irq_desc array.

Quoted 5 lines

Except for architecture implementations of irq routing the vectors that we catch irqs with should be completely invisible. The ideal architecture would not even have a separate concept and except for the msi code linux has not exported the concept of vectors outside of the architecture implementations.

Agreed.

Quoted 6 lines
Quoted 3 lines

what about per/irq kstats?
would you keep stats on a per-vector basis
and translate back to irqs for /proc/interrupts?

No. What we have on a per irq basis is what makes sense and I would leave that untouched.

Agreed.

Quoted 4 lines
Quoted 2 lines

what about when the same vector is independently
used by multiple CPUS?

Accounting by irq makes sense.

Agreed.

Quoted 25 lines
Quoted 3 lines
Quoted 1 line

- Remove the current irq compression.

maybe this can be moved up in the to-do list?

Not much.
- The current set_affinity code has a real bug.
- The MSI mismatch is worse and fixing it reduces testing later.
- We need to raise the number of irqs we can support. Either by vastly increasing the number of stubs, whose number looks hard to control at compile time, or by turning this into a data problem.

Quoted 8 lines
Quoted 8 lines

- Move irq vector allocation/deallocation into request_irq, and free_irq.

- Make vector_irq per_cpu.

- Modify assign_irq to allocate different vectors on different cpus, to fail if we cannot find a free irq vector somewhere in the irqs cpu set, and call assign_irq from set_affinity so

when we change

Quoted 4 lines
Quoted 4 lines

cpus we can allocate a different irq.

I have proof of concept level patches for everything thing except making MSI actually allocate irqs, but that should be straight

Quoted 8 lines
Quoted 5 lines

forward.

except that irqs should be allocating vectors,
the fundamental currency here, and MSI should
be allocating the same, no?

Not quite. The MSI code should first allocate
an irq_desc entry get an IRQ number.

You're right. I agree.

Quoted 11 lines

Then when the driver calls request_irq the architecture specific code should arrange for __do_IRQ to be called with the appropriate irq number when the interrupt happens. The architecture specific code needs to do this by setting the MSI address and the MSI data to an architecture specific values, that on x86 will involve the a vector.

It is reassuring to see that the way I want to make the code work tends to be they way you already think the code works. I can't be too far off in reducing the complexity.

Yeah, thanks for pointing out that the vectors are assigned at IOAPIC init-time. I hadn't noticed that, and it needs to be addressed.

-Len

#1 lore raw
Quoted 7 lines · Len

You are right. This code is wrong.
It makes absolutely no sense to reserve vectors in advance for every RTE in the IOAPIC when we don't even know if they are going to be used.

This is clearly a holdover from the early IOAPIC/MPS days when we were talking about 4 to 8 non-legacy RTEs.

Yes I agree. A lot of the IO-APIC code could probably need some renovation.

Quoted 2 lines

This is where the big system vector shortage problem should be addressed.

If we go to per CPU IDTs it will be much less pressing, but still a good idea.

-Andi

P.S.: There seems to be a lot of confusion about all this. Maybe it would make sense to do a write up defining all the terms and stick it into Documentation/* ?

#2 lore raw

RFC PATCH 1/1 Document what in IRQ is.

Quoted 3 lines · Andi Kleen

P.S.: There seems to be a lot of confusion about all this. Maybe it would make sense to do a write up defining all the terms and stick it into Documentation/* ?

How does this look?

I am pretty horrible when it comes to Documentation, but this seems to be the essence of what I was saying earlier.

Eric

Documentation/IRQ.txt new file +22 -0
new file mode 100644
index 0000000..5340369
@@ -0,0 +1,22 @@
+What is an IRQ?
+
+An IRQ is an interrupt request from a device.
+Currently they can come in over a pin, or over a packet.
+IRQs at the source can be shared.
+
+An IRQ number is a kernel identifier used to talk about a hardware
+interrupt source.  Typically this is an index into the global irq_desc
+array, but except for what linux/interrupt.h implements the details
+are architecture specific.
+
+An IRQ number is an enumeration of the possible interrupt sources on a
+machine.  Typically what is enumerated is the number of input pins on
+all of the interrupt controller in the system.  In the case of ISA
+what is enumerated are the 16 input pins to the pair of i8259
+interrupt controllers.
+
+Architectures can assign additional meaning to the IRQ numbers, and
+are encouraged to in the case  where there is any manual configuration
+of the hardware involved.  The ISA IRQ case on x86 where anyone who
+has been around a while can tell you how the first 16 IRQs map to the
+input pins on a pair of i8259s is the classic example.
#3 lore raw

Re: [RFC][PATCH] Document what in IRQ is.

Quoted 20 lines · On Tue, 02 May 2006 07:52:22 -0600 Eric W. Biederman wrote:
Quoted 3 lines · Andi Kleen

P.S.: There seems to be a lot of confusion about all this. Maybe it would make sense to do a write up defining all the terms and stick it into Documentation/* ?

How does this look?

I am pretty horrible when it comes to Documentation, but this seems to be the essence of what I was saying earlier.

Eric

Documentation/IRQ.txt new file +0 -0
new file mode 100644
index 0000000..5340369

Documentation/IRQ.txt line 4

Quoted 4 lines
Documentation/IRQ.txt new file +4 -0
@@ -0,0 +1,22 @@
+What is an IRQ?
+
+An IRQ is an interrupt request from a device.
+Currently they can come in over a pin, or over a packet.

No comma. Change packet to message?

Documentation/IRQ.txt line 5

Quoted 1 line
Documentation/IRQ.txt new file +1 -0
+IRQs at the source can be shared.

Huh? That simple sentence confuses me. Should "source" really be "sink" or "destination"? Or maybe say "IRQs at an interrupt controller can be shared." Or is that too hardware-specific?
Anyway, what source is meant here? It doesn't mean that IRQs at the producer device can be shared, right? It's more at the consumer device where they can be shared.

Documentation/IRQ.txt line 14

Quoted 8 lines
Documentation/IRQ.txt new file +8 -0
+An IRQ number is a kernel identifier used to talk about a hardware
+interrupt source.  Typically this is an index into the global irq_desc
+array, but except for what linux/interrupt.h implements the details
+are architecture specific.
+
+An IRQ number is an enumeration of the possible interrupt sources on a
+machine.  Typically what is enumerated is the number of input pins on
+all of the interrupt controller in the system.  In the case of ISA

controllers

Documentation/IRQ.txt line 15

Quoted 1 line
Documentation/IRQ.txt new file +1 -0
+what is enumerated are the 16 input pins to the pair of i8259

is

Documentation/IRQ.txt line 21

Quoted 6 lines
Documentation/IRQ.txt new file +6 -0
+interrupt controllers.
+
+Architectures can assign additional meaning to the IRQ numbers, and
+are encouraged to in the case  where there is any manual configuration
+of the hardware involved.  The ISA IRQ case on x86 where anyone who
+has been around a while can tell you how the first 16 IRQs map to the

awhile

Documentation/IRQ.txt line 22

Quoted 1 line
Documentation/IRQ.txt new file +1 -0
+input pins on a pair of i8259s is the classic example.

---
~Randy

#4 lore raw

Re: [RFC][PATCH] Document what in IRQ is.

Quoted 16 lines · Randy.Dunlap
Quoted 14 lines · On Tue, 02 May 2006 07:52:22 -0600 Eric W. Biederman wrote:
Quoted 3 lines · Andi Kleen

P.S.: There seems to be a lot of confusion about all this. Maybe it would make sense to do a write up defining all the terms and stick it into Documentation/* ?

How does this look?

I am pretty horrible when it comes to Documentation, but this seems to be the essence of what I was saying earlier.

Eric

Quoted 12 lines

Documentation/IRQ.txt line 4

Quoted 10 lines
Documentation/IRQ.txt new file +4 -0
new file mode 100644
index 0000000..5340369
@@ -0,0 +1,22 @@
+What is an IRQ?
+
+An IRQ is an interrupt request from a device.
+Currently they can come in over a pin, or over a packet.

No comma. Change packet to message?

Sounds good.

Quoted 8 lines

Documentation/IRQ.txt line 5

Quoted 1 line
Documentation/IRQ.txt new file +1 -0
+IRQs at the source can be shared.

Huh? That simple sentence confuses me. Should "source" really be "sink" or "destination"? Or maybe say "IRQs at an interrupt controller can be shared." Or is that too hardware-specific?
Anyway, what source is meant here? It doesn't mean that IRQs at the producer device can be shared, right? It's more at the consumer device where they can be shared.

By source I was thinking at the irq controller pin.

Interrupts are usually thrown from interrupt controllers to something in the chipset that interrupts the cpu, giving the cpu a token (ie an interrupt vector) that uniquely identifies which interrupt source threw the interrupt.

Linux does not have generic infrastructure to allow two interrupt sources to share the same token passed to the kernel.

In addition there are good reasons on some systems to change the token dynamically, (say to point the IRQ at a different CPU). So no generic code in the code should know about the token the cpu receives. The fact that msi.c actually knows about that token today makes is inflexible and maintenance problem.

I guess I need to figure out how to work this additional information into my documentation then.

Quoted 10 lines

Documentation/IRQ.txt line 14

Quoted 8 lines
Documentation/IRQ.txt new file +8 -0
+An IRQ number is a kernel identifier used to talk about a hardware
+interrupt source.  Typically this is an index into the global irq_desc
+array, but except for what linux/interrupt.h implements the details
+are architecture specific.
+
+An IRQ number is an enumeration of the possible interrupt sources on a
+machine.  Typically what is enumerated is the number of input pins on
+all of the interrupt controller in the system.  In the case of ISA

controllers

Documentation/IRQ.txt line 15

Quoted 1 line
Documentation/IRQ.txt new file +1 -0
+what is enumerated are the 16 input pins to the pair of i8259
Quoted 9 lines

is

Documentation/IRQ.txt line 21

Quoted 6 lines
Documentation/IRQ.txt new file +6 -0
+interrupt controllers.
+
+Architectures can assign additional meaning to the IRQ numbers, and
+are encouraged to in the case  where there is any manual configuration
+of the hardware involved.  The ISA IRQ case on x86 where anyone who
+has been around a while can tell you how the first 16 IRQs map to the

awhile

Quoted 1 line
Documentation/IRQ.txt new file +1 -0
+input pins on a pair of i8259s is the classic example.

Eric

#5 lore raw
In reply to Brown, Len
Quoted 20 lines · Len
Quoted 3 lines

IRQ is an interrupt request, from a device. Currently they can come in over a pin, or over a packet. IRQs at the source can be shared.

Well stated.

Quoted 4 lines

For an IRQ number it is just an enumeration of interrupt sources. Ideally in a stable manner so that the assigned number does not depend on compile options, or which version of the kernel you are running.

While desirable, the "stable" part is not guaranteed.
On an ACPI system with PCI Interrupt Link devices, request_irq() goes into the pci-enable-device path,
which ends up asking ACPI what interrupt pin this device uses. But the pin is programmable, so we pick one from the list of possible, program the router accordingly, and this is what gets returned to request_irq(). ACPI PCI Interrupt Link Devices, of course, are just a wrapper for what x86 legacy calls PIRQ routers.

Quoted 8 lines

In practice, if you boot the same or similar kernel on the same configuration, you get 'stable', but if something changes such as enabling/disabling a device or loading an additional device driver
or otherwise tinkering with probe order, then all bets are off, since the sequence of selecting and
balancing possible device interrupt lines between available IRQs has changed.

Agreed in the general case there is nothing we can do. But with a reasonable degree of stability things will stay put long enough for people to reason about them.

Although I would actually expect us to use the normal twiddle of irq lines on a bus. But if we can have less contention that is certainly better.

Quoted 11 lines
Quoted 5 lines

The global irq_desc array is the core of any definition. That is how linux knows about irqs. Beyond that we ideally have stable numbers across reboots, and recompiles, and hardware addition and removal. Stable numbers are not really possible but we can come quite close.

Okay, I'm good with irq_desc, Linux's generic IRQ definition, being the center of the universe. I'm also good with the index into irq_desc[] being the "IRQ number" show in /proc/interrupts. I guess I was thinking from a HW-specific point of view this afternoon...

I had to stare at the code for a while to see this as well.

Quoted 6 lines

I don't like vector numbers being exposed in /proc/interrupts on x86. On an ACPI-enabled system, I think that they should be the GSI. This is consistent with the 0-15 legacy definitions being pin numbers (which I think we must keep), and consistent with the 16-n numbers being IOAPIC pin numbers (+offset), (which I think we should have kept).

Agreed.

Quoted 2 lines

Of course, to do this simply, we'd have to select the irq_desc[i] with i = gsi, and not i = vector.

Exactly.

Quoted 14 lines
Quoted 8 lines
Quoted 3 lines

We should never have had a problem with un-connected interrupt lines consuming vectors, as the vectors are handed out at run-time only when interrupts are requested by drivers.

Incorrect. By being requested by drivers I assume you mean by request_irq. assign_irq_vector is called long before request_irq and in fact there is currently not a call from request_irq to assign_irq_vector. Look where assign_irq_vector is called in io_apic.c

You are right. This code is wrong.
It makes absolutely no sense to reserve vectors in advance for every RTE in the IOAPIC when we don't even know if they are going to be used.

Quoted 5 lines

This is clearly a holdover from the early IOAPIC/MPS days when we were talking about 4 to 8 non-legacy RTEs.

This is where the big system vector shortage problem should be addressed.

Yes. Natalie mentioned that there are some IBM systems that still run out of processor vectors with our current irq collapsing.

Quoted 5 lines
Quoted 3 lines

I think there is a legitimate case for legacy edge triggered interrupts to request a vector sooner, as there are so many races in the enable/disable paths.

I don't follow you on this part.

For edge triggered legacy irqs coming off the i8259 it probably makes sense to assign them vectors early like we do currently.

I believe although I am not certain that with edge triggered irqs actually disabling them makes it easy to miss edges.

Quoted 13 lines
Quoted 5 lines
Quoted 4 lines

The problem with this workaround is going to be choose a policy of where to direct what, and how to move things if interrupts become un-balanced.

Quoted 7 lines

The directing problem already exists. In general an I/O apic can only direct an irq at a specific cpu, and linux already supports cpusets on which an irq may be delivered.

But yes on systems with 8 or fewer cpus the I/O apics can do the directing themselves and it is likely we still want to handle that case.

Quoted 7 lines

I don't think it is important to optimize for <= 8 CPUs having hundreds of unique interrupts. Systems with huge numbers of interrutps will have more than 8 CPUs. If they don't, then the should function, but being optimal isn't what the administrator had in mind. With multiple cores being added to systems, this becomes even more true over time.

Sorry. To be clear for systems with <= 8 CPUs we need to allocate a common vector for all 8 cpus at once if we want to use flat mode and this makes a version of assign_irq_vector that can work assign per cpu vectors more complicated, since it has to handle both cases. Hmm, should assign_irq_vector become a ioapic method?

I don't think we want to loose the optimization of letting the hardware select select the cpu on small systems.

Quoted 6 lines
Quoted 3 lines

It is reassuring to see that the way I want to make the code work tends to be they way you already think the code works. I can't be too far off in reducing the complexity.

Yeah, thanks for pointing out that the vectors are assigned at IOAPIC init-time. I hadn't noticed that, and it needs to be addressed.

Exactly. Once that is fixed setting gsi == irq should be trivial. Then we only have to worry about big systems that actually use more than 244 IRQs. Which is where the per cpu vector assignment comes in.

Eric