Skip to contentSkip to conversation

IRQ domain logging?

DISCUSSION Bjorn Helgaas 4 participants 4 messages Apr 22, 2025 – Apr 28, 2025 PCI

lore
Share & tools
lore
#0 lore raw

Hi Thomas,

IRQ domains and IRQs are critical infrastructure, but we don't really log anything when we discover controllers or set them up. Do you think there would be any value in exposing some of this structure in dmesg to help people (like me!) understand how these are connected to devices and drivers?

For example, in a simple qemu system:

IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
ACPI: Using IOAPIC for interrupt routing
ACPI: PCI: Interrupt link LNKA configured for IRQ 10
ACPI: PCI: Interrupt link GSIA configured for IRQ 16
hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0
ACPI: \_SB_.GSIA: Enabled at IRQ 16
pcieport 0000:00:1c.0: PME: Signaling with IRQ 24
00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
ata1: SATA max UDMA/133 abar m4096@0xfeadb000 port 0xfeadb100 irq 28 lpm-pol 0

I think these are all wired interrupts, and maybe IRQ==GSI (?), and I think the ACPI link devices are configurable connections between an INTx and the IOAPIC, but it's kind of hard to connect them all together.

This from an arm64 system is even more obscure to me:

NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
GICv3: 256 SPIs implemented
Root IRQ handler: gic_handle_irq
GICv3: GICv3 features: 16 PPIs
kvm [1]: vgic interrupt IRQ18
xhci-hcd xhci-hcd.0.auto: irq 67, io mem 0xfe800000

I have no clue where irq 67 goes.

Maybe there's no useful way to log anything here, I dunno; it just occurred to me when looking at Jiri's series to reduce the number of irqdomain interfaces. PCI controller drivers do a lot of interrupt domain setup, and if that were more visible/concrete in dmesg, I think I might understand it better.

Bjorn

#1 lore raw
Quoted 10 lines · Bjorn Helgaas

This from an arm64 system is even more obscure to me:

NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
GICv3: 256 SPIs implemented
Root IRQ handler: gic_handle_irq
GICv3: GICv3 features: 16 PPIs
kvm [1]: vgic interrupt IRQ18
xhci-hcd xhci-hcd.0.auto: irq 67, io mem 0xfe800000

I have no clue where irq 67 goes.

There's quite a bit of information available in /proc/interrupts, /proc/irq/ and /sys/kernel/irq/, I guess that's what most people use.

Thanks,

Lukas

#2 lore raw
Quoted 10 lines · Lukas Wunner
Quoted 9 lines · Bjorn Helgaas

This from an arm64 system is even more obscure to me:

NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
GICv3: 256 SPIs implemented
Root IRQ handler: gic_handle_irq
GICv3: GICv3 features: 16 PPIs
kvm [1]: vgic interrupt IRQ18
xhci-hcd xhci-hcd.0.auto: irq 67, io mem 0xfe800000

Quoted 4 lines
Quoted 1 line

I have no clue where irq 67 goes.

There's quite a bit of information available in /proc/interrupts, /proc/irq/ and /sys/kernel/irq/, I guess that's what most people use.

/sys/kernel/debug/irq/....

Gives the most comprehensive insight into the domain hierarchy.

# cat /sys/kernel/debug/irq/irqs/204
handler:  handle_edge_irq
device:   0000:01:00.0
status:   0x00000000
istate:   0x00004000
ddepth:   0
wdepth:   0
dstate:   0x1d401200
            IRQD_ACTIVATED
            IRQD_IRQ_STARTED
            IRQD_SINGLE_TARGET
            IRQD_AFFINITY_SET
            IRQD_AFFINITY_ON_ACTIVATE
            IRQD_CAN_RESERVE
            IRQD_HANDLE_ENFORCE_IRQCTX
node:     0
affinity: 50
effectiv: 50
pending:  
domain:  PCI-MSIX-0000:01:00.0-12
 hwirq:   0x65
 chip:    PCI-MSIX-0000:01:00.0
  flags:   0x1430
             IRQCHIP_SKIP_SET_WAKE
             IRQCHIP_ONESHOT_SAFE
             IRQCHIP_MOVE_DEFERRED

address_hi: 0x00000000
address_lo: 0xfee32000
msg_data: 0x00000023
parent:

    domain:  VECTOR
     hwirq:   0xcc
     chip:    APIC
      flags:   0x0
     Vector:    35
     Target:    50
     move_in_progress: 0
     is_managed:       0
     can_reserve:      1
     has_reserved:     0
     cleanup_pending:  0

It shows the complete domain hierarchy for each interrupt an irq/domains/.... gives extra information about the domains.

You need to enable GENERIC_IRQ_DEBUGFS.

Exposing this in dmesg in a comprehensible and condensed form might be possible, but needs some thought.

So far I survived pretty good with the debugfs data.

Thanks,

tglx

#3 lore raw
In reply to Bjorn Helgaas
Quoted 34 lines · Bjorn Helgaas

Hi Thomas,

IRQ domains and IRQs are critical infrastructure, but we don't really log anything when we discover controllers or set them up. Do you think there would be any value in exposing some of this structure in dmesg to help people (like me!) understand how these are connected to devices and drivers?

For example, in a simple qemu system:

IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
ACPI: Using IOAPIC for interrupt routing
ACPI: PCI: Interrupt link LNKA configured for IRQ 10
ACPI: PCI: Interrupt link GSIA configured for IRQ 16
hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0
ACPI: \_SB_.GSIA: Enabled at IRQ 16
pcieport 0000:00:1c.0: PME: Signaling with IRQ 24
00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
ata1: SATA max UDMA/133 abar m4096@0xfeadb000 port 0xfeadb100 irq 28 lpm-pol 0

I think these are all wired interrupts, and maybe IRQ==GSI (?), and I think the ACPI link devices are configurable connections between an INTx and the IOAPIC, but it's kind of hard to connect them all together.

This from an arm64 system is even more obscure to me:

NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
GICv3: 256 SPIs implemented
Root IRQ handler: gic_handle_irq
GICv3: GICv3 features: 16 PPIs
kvm [1]: vgic interrupt IRQ18
xhci-hcd xhci-hcd.0.auto: irq 67, io mem 0xfe800000

Quoted 7 lines

I have no clue where irq 67 goes.

Maybe there's no useful way to log anything here, I dunno; it just occurred to me when looking at Jiri's series to reduce the number of irqdomain interfaces. PCI controller drivers do a lot of interrupt domain setup, and if that were more visible/concrete in dmesg, I think I might understand it better.

The current visibility into interrupt routing in systems is fragmented, making it challenging to observe the routing behavior of specific interrupts or interrupt types. For enthusiasts exploring system internals, having a ​traceroute-like tool to map interrupt handling paths would significantly enhance transparency and debugging capabilities.

e.g. How an MSI is routed and remapped between different domains on x86

MSI pci-dev-->ioapic-->iommu-->apic-->cpu

But so far it seems there is no enough info from KABI(/proc /sysfs syscall, dmesg etc) to compose a complete chain like that ?

Thanks,
Ethan

Quoted 3 lines

Bjorn