Increase the number of IRQ descriptors for SPARSEIRQ
Status from this message: The following commit has been merged into the irq/core branch of tip:
Get series with b4
b4 am [email protected]Versions newest first · linked by author and title
- v5 Latest3 patches 8 repliesApplied
- v43 patches 0 repliesSuperseded
- v33 patches 14 repliesSuperseded
Earlier versions · 2
| Version | Posted | Patches | Replies | Status |
|---|---|---|---|---|
| v2 | 3 patches | 3 replies | Superseded | |
| v1 | 5 patches | 26 replies | Superseded |
Changes in v5 from the cover letter
- Change function name irq_find_next_irq() to irq_find_at_or_after()
- Update comment to reflect the return value of irq_get_next_irq()
Patches in v5 Applied
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/3 | genirq: Use hlist for managing resend handlersinclude/ kernel/ kernel/ +2 more | +38 | 8 replies | |
| 2/3 | genirq: Encapsulate sparse bitmap handlingkernel/ kernel/ | +22 | 0 replies | |
| 3/3 | genirq: Use the maple tree for IRQ descriptors managementkernel/ kernel/ | +33 | 0 replies |
Reviews trailers per patch
No Reviewed-by, Acked-by or Tested-by trailers for v5 yet.
Cover letter [PATCH v5 0/3] Increase the number of IRQ descriptors for SPARSEIRQ
The ARM64 architecture uses SPARSEIRQ with a default value of NR_IRQS, which is set to 64. This means that only 64+8192 IRQ descriptors are allowed, which may not be sufficient for modern ARM64 servers that have a large number of IO devices and GIC hardware that supports direct vSGI and vLPI injection features.
This limitation has caused issues when attempting to launch multiple virtual machines with GICv4.1 features, resulting in the error message 'kvm_err("VPE IRQ allocation failure\n")'. The root cause of this issue is the ~8K IRQ descriptor limit.
To address this issue, an initial proposal was made to define NR_IRQS to 2^19 for ARM64. However, Marc Zyngier suggested implementing a generic solution instead of hard-coded values. Thomas Gleixner advised to use the maple tree data structure and provided most of the necessary functions.
For more information, refer to the discussion thread at
https://lore.kernel.org/linux-arm-kernel/[email protected]/.
This patch series converts the static memory allocation to dynamic using the maple tree, and increases the maximum number of IRQ descriptors to INT_MAX from NR_IRQS+8192. This change has been tested on an ARM64 server with CONFIG_SPARSE_IRQ=y, where 256 virtual machines were launched, creating a total of 128K+ IRQ descriptors, and IRQ injection was verified.
Teted on ARM64 system:
- Normal boot with ~200 cores
- CPU hot-unplug/hot-plug using sysfs
- Booted virtual machines with PCIe device pass-through
- Hot-unplug of CPU where vfio-pci interrupts are bounded
Changes in v5:
- Change function name irq_find_next_irq() to irq_find_at_or_after()
- Update comment to reflect the return value of irq_get_next_irq()
Changes in v4:
- Fix the iterator function using mt_find() instead of nt_next()
- Tested CPU hot-unplug and hot-plug on ARM64 system. This is completely broken before v4.
Changes in v3:
- Edited commit text
- Added a helper function irq_resend_init()
- Rebased to v6.3-rc6
Changes in v2:
- The patches have been updated to v6.3-rc5.
- The patches 2/5 and 4/5 have been removed as they are unnecessary.
- The review comments from Thomas have been addressed.
Shanker Donthineni (3):
genirq: Use hlist for managing resend handlers
genirq: Encapsulate sparse bitmap handling
genirq: Use the maple tree for IRQ descriptors management
include/linux/irqdesc.h | 3 ++ kernel/irq/chip.c | 1 + kernel/irq/internals.h | 6 ++-- kernel/irq/irqdesc.c | 77 +++++++++++++++++++++++++---------------- kernel/irq/resend.c | 47 ++++++++++++++++--------- 5 files changed, 86 insertions(+), 48 deletions(-)
Signature · 1 line
2.25.1