x86_64: Dynamically allocate arch specific system vectors

Subject: [PATCH] x86_64: Dynamically allocate arch specific system vectors

From: Alan Mayer <[email protected]>

On some systems (e. g., UV) it is necessary to use an interrupt vector as a "system" vector, that is, it is generated by system hardware, not an IO device. This patch dynamically allocates them from the pool of interrupt vectors below the fixed system vectors. This may include stealing some from the device interrupt vector pool, so they are allocated dynamically so that other archs don't have to pay the price. In UV, examples of these hardware and software systems that need dynamically allocated vectors are the GRU, the BAU, and XPM/XPC.

Quoted 4 lines · Alan Mayer

Subject: [PATCH] x86_64: Dynamically allocate arch specific system vectors

From: Alan Mayer <[email protected]>

Quoted 8 lines

On some systems (e. g., UV) it is necessary to use an interrupt vector as a "system" vector, that is, it is generated by system hardware, not an IO device. This patch dynamically allocates them from the pool of interrupt vectors below the fixed system vectors. This may include stealing some from the device interrupt vector pool, so they are allocated dynamically so that other archs don't have to pay the price. In UV, examples of these hardware and software systems that need dynamically allocated vectors are the GRU, the BAU, and XPM/XPC.

Could you please explain words why the vector allocator does not work for you?

This code at first glance looks a lot like duplicating the vector allocator.

Eric

I assume you mean create_irq() and destroy_irq().
They are close to what we need.

However for any given system use:
- we need to request a high priority vector for some irq's, rather than one randomly allocated as per __assign_irq_vector().

- we want the irq/vector to be targeted to all cpu's (as specified in a mask,
and can include currently offline cpu's) rather than a single cpu.

I suppose those abilities could be added to create_irq(), but we didn't want to intrude into that interface.

A smaller consideration is simplicity of use. We want any such user to use
the generic do_IRQ() flow (not alloc_intr_gate()).

But make it easy to set
up the irq/vector, irq_chip and irq_desc without getting intimate with the details.
I suppose some other wrapper for an enhanced create_irq() could be done.

We are going to need such irq/vector pairs for a couple of UV drivers (drivers/misc/sgi-gru/ and sgi-xp/). And would prefer it for the UV TLB shootdown (x86/kernel/tlb.uv.c) rather than using alloc_intr_gate().

Quoted 2 lines · Cliff Wickman

I assume you mean create_irq() and destroy_irq().
They are close to what we need.

Actually that isn't quite what I was thinking.

Quoted 3 lines

However for any given system use:
- we need to request a high priority vector for some irq's, rather than one randomly allocated as per __assign_irq_vector().

Well I was actually thinking mostly about reusing assign_irq_vector.

The high priority vector seems to be the place where we would need to reexamine the vector allocator.

Quoted 3 lines

- we want the irq/vector to be targeted to all cpu's (as specified in a mask,
and can include currently offline cpu's) rather than a single cpu.

__assign_irq_vector should be able to handle that case.

Quoted 2 lines

I suppose those abilities could be added to create_irq(), but we didn't want to intrude into that interface.

Right. That part gets custom and there is no intersection.

Quoted 3 lines

A smaller consideration is simplicity of use. We want any such user to use
the generic do_IRQ() flow (not alloc_intr_gate()).

My primary concern is that you are generalizing an interface that is designed for alloc_intr_gate consumers. In particular for people who wish to allocate vectors that can be triggered by software to do this.

If this is normal irq handling I would really prefer to use the normal data structures that we use for allocating vectors to irqs, because that is what you are doing.

Quoted 8 lines

But make it easy to set
up the irq/vector, irq_chip and irq_desc without getting intimate with the details.
I suppose some other wrapper for an enhanced create_irq() could be done.

We are going to need such irq/vector pairs for a couple of UV drivers (drivers/misc/sgi-gru/ and sgi-xp/). And would prefer it for the UV TLB shootdown (x86/kernel/tlb.uv.c) rather than using alloc_intr_gate().

I also want to ask why do you need an irq that fires on every cpu?

That concept seems to be responsible for one of the nastiest data structures in the kernel. Where we track how many times any irq has occurred on any cpu. Look at the percpu variable kernel_stat. It seems to be one of the nastiest variables I know of. Of size:
NR_CPUS*NR_CPUS*32 Assuming you have a machine with a reasonable number of irqs per cpu.

Eric

Okay, I think we have it now. assign_irq_vector *almost* does what we need. One minor thing is that assign_irq_vector ANDs against cpu_online_map. We would need cpu_possible_map, so we get the vector on offline cpus that may come online. The other thing is that assign_irq_vector doesn't allow the specification of interrupt priorities. It would need to be modified to handle returning either a high priority vector or a low priority vector. Would modifying the api for assign_irq_vector be the proper approach?

The interrupts don't necessarily fire on all cpus, it's just that they *can* fire on any cpu. For example, the GRU triggers an interrupt (it is very IPI'ish) to a particular cpu in the event of a GRU TLB fault. That cpu handles the fault and returns. But the fault can happen on any cpu, so all cpus need to be registered for the same vector and irq. This is probably splitting hairs; it is certainly no different in principal from timer interrupts or processor TLB faults.

As far as kernel_stat is concerned. I see you're point. NR_CPUS on our machines is going to be big (4K? 8K? something like that). NR_IRQS is also going to big because of that. It's unfortunate since the actual number of interrupt sources is going to be an order of magnitude smaller, at least.

Quoted 7 lines · Alan Mayer

Okay, I think we have it now. assign_irq_vector *almost* does what we need. One minor thing is that assign_irq_vector ANDs against cpu_online_map. We would need cpu_possible_map, so we get the vector on offline cpus that may come online. The other thing is that assign_irq_vector doesn't allow the specification of interrupt priorities. It would need to be modified to handle returning either a high priority vector or a low priority vector. Would modifying the api for assign_irq_vector be the proper approach?

I don't know if it makes sense to modify assign_irq_vector or to have a companion function that uses the same data structures.

I think I would work on the companion function and if the code can be made sufficiently similar merge the two functions.

Quoted 7 lines

The interrupts don't necessarily fire on all cpus, it's just that they *can* fire on any cpu. For example, the GRU triggers an interrupt (it is very IPI'ish) to a particular cpu in the event of a GRU TLB fault. That cpu handles the fault and returns. But the fault can happen on any cpu, so all cpus need to be registered for the same vector and irq. This is probably splitting hairs; it is certainly no different in principal from timer interrupts or processor TLB faults.

Reasonable. As long as you don't need to read a status register to figure out what to do that sounds reasonable. This does sound very much like splitting hairs on a very platform specific capability.

If we can generalize the mechanism to things like per cpu timer interrupts and such so that we reduced the total amount of code we have to maintain I would find it a very compelling mechanism.

Quoted 4 lines

As far as kernel_stat is concerned. I see you're point. NR_CPUS on our machines is going to be big (4K? 8K? something like that). NR_IRQS is also going to big because of that. It's unfortunate since the actual number of interrupt sources is going to be an order of magnitude smaller, at least.

The number of interrupts sources is going to be smaller only because SGI machines have or at least appear to have poor I/O compared to most of the rest of machines in existence. NR_CPUS*16 is a fairly reasonable estimate on most machines in existence. In the short term it is going to get worse in the presence of MSI-X. I was talking to a developer at Intel last week about 256 irqs for one card. I keep having dreams about finding a way to just keep stats for a few cpus but alas I don't think that is going to happen. Silly us.

Eric

Quoted 14 lines · Eric W. Biederman

Quoted 11 lines · Alan Mayer

Okay, I think we have it now. assign_irq_vector *almost* does what we need.
One minor thing is that assign_irq_vector ANDs against cpu_online_map. We would
need cpu_possible_map, so we get the vector on offline cpus that may come
online. The other thing is that assign_irq_vector doesn't allow the specification of interrupt priorities. It would need to be modified to handle
returning either a high priority vector or a low priority vector. Would modifying the api for assign_irq_vector be the proper approach?

Quoted 6 lines

I don't know if it makes sense to modify assign_irq_vector or to have a companion function that uses the same data structures.

I think I would work on the companion function and if the code can be made sufficiently similar merge the two functions.

Okay, If I understand you, here's what we can do. We currently have this function that does pretty much what the combination of create_irq() and __assign_irq_vector() do. We can accomplish the same thing that our routine does using create_irq() and __assign_irq_vector() do if we make the following changes:

__assign_irq_vector(int irq, cpumask_t mask) ==>

    __assign_irq_vector(int irq, cpumask_t mask, int priority);

priority has three values: priority_none, priority_low, priority_high priority_none means do everything the way it is done now. priority_low means do everything the way its is done now, except use cpu_possible_map rather than cpu_online_map.
priority_high means search the interrupt vectors from the top down, rather than from the bottom up and use cpu_possible_map rather than cpu_online_map.

create_irq(void) ==> create_irq(int priority, cpumask_t *mask) priority_none, means do everything the way it is done now, passing in TARGET_CPUS as the mask, but also sending the priority arg. into __assign_irq_vector().
priority_low and priority_high means use create_irq()'s mask arg. as the mask passed to __assign_irq_vector).

We would add an additional small routine on top of create_irq() to do any massaging of the irq_desc, etc. that we need for these system vectors.

Is that what you were thinking about?

            --ajm

Quoted 41 lines

Quoted 4 lines

The interrupts don't necessarily fire on all cpus, it's just that they *can*
fire on any cpu. For example, the GRU triggers an interrupt (it is very IPI'ish) to a particular cpu in the event of a GRU TLB fault. That

Quoted 8 lines

cpu handles
the fault and returns. But the fault can happen on any cpu, so all cpus need to
be registered for the same vector and irq. This is probably splitting hairs; it
is certainly no different in principal from timer interrupts or processor TLB
faults.

Reasonable. As long as you don't need to read a status register to figure
out what to do that sounds reasonable. This does sound very much like splitting hairs on a very platform specific capability.

If we can generalize the mechanism to things like per cpu timer interrupts and such so that we reduced the total amount of code we have to maintain I would find it a very compelling mechanism.

Quoted 7 lines

As far as kernel_stat is concerned. I see you're point. NR_CPUS on our machines is going to be big (4K? 8K? something like that). NR_IRQS is also
going to big because of that. It's unfortunate since the actual number of
interrupt sources is going to be an order of magnitude smaller, at least.

The number of interrupts sources is going to be smaller only because SGI machines have or at least appear to have poor I/O compared to most of the rest of machines in existence. NR_CPUS*16 is a fairly reasonable estimate on most machines in existence. In the short term it is going to get worse in the presence of MSI-X. I was talking to a developer at Intel last week about 256 irqs for one card. I keep having dreams about finding a way to just keep stats for a few cpus but alas I don't think that is going to happen. Silly us.

Eric