Skip to contentSkip to conversation

mlx5_core receives no interrupts with maxcpus=8

5.14-rc1 Thomas Gleixner 2 participants 2 messages Aug 28, 2021 – Aug 29, 2021 1 message not in the archive

Subsystems (3)
Networking Network drivers PCI
lore
Share & tools
lore

Thomas Gleixner

author
#1 lore raw
In reply to a message not in the archive

Dexuan,

Quoted 8 lines · Thomas Gleixner
Quoted 7 lines · Dexuan Cui
Quoted 7 lines

Sorry for the late response! I checked the below sys file, and the output is exactly the same in the good/bad cases -- in both cases, I use maxcpus=8; the only difference in the good case is that I online and then offline CPU 8~31:

for i in `seq 8 31`;  do echo 1 >  /sys/devices/system/cpu/cpu$i/online; done
for i in `seq 8 31`;  do echo 0 >  /sys/devices/system/cpu/cpu$i/online; done
# cat /sys/kernel/debug/irq/irqs/209

Yes, that looks correct.

Quoted 11 lines
Quoted 11 lines

I tried the kernel parameter "intremap=nosid,no_x2apic_optout,nopost" but it didn't help. Only "intremap=off" can work round the no interrupt issue.

When the no interrupt issue happens, irq 209's effective_affinity_list is 5. I modified modify_irte() to print the irte->low, irte->high, and I also printed the irte_index for irq 209, and they were all normal to me, and they were exactly the same in the bad case and the good case -- it looks like, with "intremap=on maxcpus=8", MSI-X on CPU5 can't work for the NIC device (MSI-X on CPU5 works for other devices like a NVMe controller) , and somehow "onlining and then offlining CPU 8~31" can "fix" the issue, which is really weird.

Just for the record: maxcpus=N is a dangerous boot option as it leaves the non brought up CPUs in a state where they can be hit by MCE broadcasting without being able to act on it. Which means you're operating the system out of spec.

According to your debug output the interrupt in question belongs to the INTEL-IR-3 interrupt domain, which means it hangs of IOMMU3, aka DMAR unit 3.

To which DMAR/remap unit are the other unaffected devices connected to?

Thanks,

tglx

#2 lore raw
Quoted 12 lines

From: Thomas Gleixner <[email protected]>
Sent: Saturday, August 28, 2021 1:44 PM

Quoted 1 line
Quoted 1 line

I tried the kernel parameter "intremap=nosid,no_x2apic_optout,nopost"

but

Quoted 8 lines
Quoted 8 lines

it didn't help. Only "intremap=off" can work round the no interrupt issue.

When the no interrupt issue happens, irq 209's effective_affinity_list is 5. I modified modify_irte() to print the irte->low, irte->high, and I also printed the irte_index for irq 209, and they were all normal to me, and they were exactly the same in the bad case and the good case -- it looks like, with "intremap=on maxcpus=8", MSI-X on CPU5 can't work for the NIC device (MSI-X on CPU5 works for other devices like a NVMe controller) , and

Quoted 8 lines

somehow

Quoted 1 line
Quoted 1 line

"onlining and then offlining CPU 8~31" can "fix" the issue, which is really

weird.

Just for the record: maxcpus=N is a dangerous boot option as it leaves the non brought up CPUs in a state where they can be hit by MCE broadcasting without being able to act on it. Which means you're operating the system out of spec.

I didn't know about this. Thanks for the reply!

Quoted 7 lines

According to your debug output the interrupt in question belongs to the INTEL-IR-3 interrupt domain, which means it hangs of IOMMU3, aka DMAR unit 3.

To which DMAR/remap unit are the other unaffected devices connected to?

tglx

With maxcpus=8, on CPU 5, the NIC receives no interrupt, but a NVMe interrupt ("INTEL-IR-6") on the CPU works, and two "IOAT" interrupts ("INTEL-IR-7") also work.

Except the NIC, the only IRQs connected to the faulty IOMMU3 are irq33 and irq34:

root@lsg-gen7-a:~# cat /sys/kernel/debug/irq/irqs/33
handler:  handle_fasteoi_irq
device:   (null)
status:   0x00004100
istate:   0x00000000
ddepth:   1
wdepth:   0
dstate:   0x3503a000
            IRQD_LEVEL
            IRQD_IRQ_DISABLED
            IRQD_IRQ_MASKED
            IRQD_SINGLE_TARGET
            IRQD_MOVE_PCNTXT
            IRQD_AFFINITY_ON_ACTIVATE
            IRQD_CAN_RESERVE
            IRQD_HANDLE_ENFORCE_IRQCTX
node:     1
affinity: 0-103
effectiv: 0
pending:
domain:  IO-APIC-18
 hwirq:   0x0
 chip:    IR-IO-APIC
  flags:   0x10
             IRQCHIP_SKIP_SET_WAKE
 parent:
    domain:  INTEL-IR-3
     hwirq:   0x0
     chip:    INTEL-IR
      flags:   0x0
     parent:
        domain:  VECTOR
         hwirq:   0x21
         chip:    APIC
          flags:   0x0
         Vector:     0
         Target:     0
         move_in_progress: 0
         is_managed:       0
         can_reserve:      1
         has_reserved:     1
         cleanup_pending:  0
root@lsg-gen7-a:~# cat /sys/kernel/debug/irq/irqs/34
handler:  handle_edge_irq
device:   0000:d7:00.0
status:   0x00004000
istate:   0x00000000
ddepth:   0
wdepth:   0
dstate:   0x37408200
            IRQD_ACTIVATED
            IRQD_IRQ_STARTED
            IRQD_SINGLE_TARGET
            IRQD_MOVE_PCNTXT
            IRQD_AFFINITY_ON_ACTIVATE
            IRQD_CAN_RESERVE
            IRQD_DEFAULT_TRIGGER_SET
            IRQD_HANDLE_ENFORCE_IRQCTX
node:     1
affinity: 0-7
effectiv: 1
pending:
domain:  INTEL-IR-MSI-3-3
 hwirq:   0x6b80000
 chip:    IR-PCI-MSI
  flags:   0x30
             IRQCHIP_SKIP_SET_WAKE
             IRQCHIP_ONESHOT_SAFE
 parent:
    domain:  INTEL-IR-3
     hwirq:   0x10000
     chip:    INTEL-IR
      flags:   0x0
     parent:
        domain:  VECTOR
         hwirq:   0x22
         chip:    APIC
          flags:   0x0
         Vector:    34
         Target:     1
         move_in_progress: 0
         is_managed:       0
         can_reserve:      1
         has_reserved:     0
         cleanup_pending:  0
root@lsg-gen7-a:~# lspci |grep d7:00.0
d7:00.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port A (rev 07)

irq 33 doesn't appear in /proc/interupts.
irq 34 in /proc/interupts also receives no interrupts.

So it looks like IOMMU3 is somehow not working at all with maxcpus=8. "onlining and offlining CPU 8~31" can somehow "fix" it. :-) I'm not sure if this is a kernel issue or firmware issue.

Thanks,
Dexuan