Very sparse and high interrupt map with SPARSE_IRQ

Hi,

I am working with a Qualcomm SPMI device that supports up to 32768 interrupts. Now, in practice, not nearly all of these interrupts will be populated on a real device. Most likely on the order of 200-300 interrupts will be specified in Device Tree. The problem is the set of active interrupts will change on future devices sharing the same architecture, and there's really no predicting which of this range will be active. Ideally, the supporting code should never have to change.

To support such a device, I am considering using SPARSE_IRQ and allocating the irq_desc at runtime as necessary while walking the Device Tree. To keep the mapping function simple and fast, I was thinking of using discontinuous, high system interrupt numbers that can be computed with a simple O(1) operation. Alternatively, I could use a radix tree or hash to map these to more traditional, lower and contiguous interrupt numbers, but I'm not aware of any significant benefit in doing so.

As far as I can tell, the only potential problems with using such high interrupt numbers (eg. 33102) are:

1. IRQ_BITMAP_BITS must be expanded to support the entire possible range (eg. ~0-33500). IRQ_BITMAP_BITS is defined as NR_IRQS. This will waste ~3 KB on such a range. To me 3 KB can be justified if it speeds up the fast path interrupt handling.
2. NR_IRQS will increase beyond the HARDIRQ_BITS limitation, which governs the number of nested interrupts. But as mentioned, we won't actually have more real interrupts than the maximum setting (10 bit) -- it's just that our NR_IRQ definition will be so high to trip an older ARM check for NR_IRQS going beyond HARDIRQ_BITS.

So basically, I'm asking whether this analysis is correct, and what I'm doing seems reasonable. I'd also like to propose a couple changes as a consequence of what I mentioned above:

1. Add another macro to distinguish between the actual number of interrupts a system supports and the *highest* number it supports. Eg. NR_IRQS seems to imply a quantity - not a maximum value. But it's currently being used to cover both constraints.
2. Remove the check in arch/arm/include/asm/hardirq.h for HARDIRQ_BITS being too low. Actually, if 1) were really implemented, then most likely NR_IRQS would be below 1024, and this check would not be violated. But regardless, per the comment in kernel/irq/internals.h, we're probably bound by interrupt stack for such systems, anyways.

Thanks,
Mike

Employee of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum

Have you looked at irq_domain (kernel/irq/irqdomain.c). This is meant to support complex mappings like this. Although in its current form it needs some work to support this. Is there no sub-grouping of interrupts at all?

The hwirq # is stored in irq_data, so converting from Linux irq to hwirq

# is O(1). Going the other way is implemented per domain and would
depend on the implementation of .to_irq.

Rob

Thanks for the advice. The chip driver houses a number of devices shared over a SPMI bus. The SPMI bus supports up to 16 slave IDs, and each slave can have 256 devices. Each device can have up to 8 interrupts.

So this is pretty easy to support with irq_domains if we treat all 32768 interrupts as one domain and allow the chip driver to own the entire 32768 block of interrupts. Each translation is merely a O(1) calculation as you mentioned. And with SPARSE_IRQ, we only allocate the interrupts we use on the bus, and so there's very little cost associated with the large number of interrupts.

Were you thinking of supporting these sub-groups (eg. group of devices) in separate irq_domains? I'm somewhat curious if there's future work planned around this area. As it currently stands, it seems like this would only complicate things.

There is one issue that needs to be resolved in order to support even one irq_domain with this configuration. The problem is that we allocate irq_descs dynamically based on Device Tree information, and so irq_domain_add() needs to be modified to support this. In fact, there's even a comment in the function regarding this limitation. There's two problems with the implementation of irq_domain_add() for my usage:

1. It assumes irq_descs are already allocated. In my case, I would like to add the irq_domain at init time through of_irq_init(). But the actual descriptors are not allocated until much later.

2. It assumes that every hw_irq is available within the range of interrupts the domain manages, which is not true in my case. In my case, they are truly sparse.

Thus I am proposing we have a more simple irq_domain_add() function that solely adds the irq_domain to the list. Then we can create registration interface that does the extra initialization for the irq_data of each specified hw_irq. What do you think about this? I can submit a patch if others think this is reasonable.

Thanks,
Mike

Employee of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum