Skip to contentSkip to conversation

MSI interrupt for xhci still lost on 5.6-rc6 after cpu hotplug

DISCUSSION Thomas Gleixner 2 participants 5 messages May 8, 2020 – May 11, 2020 1 message not in the archive PCI

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): 20200508005528.GB61703@otc-nc-03

#1 lore raw

Ashok,

Quoted 7 lines · Ashok
Quoted 5 lines

But as this last one is the migration thread, aka stomp machine, I assume this is a hotplug operation. Which means the CPU cannot handle interrupts anymore. In that case we check the old vector on the unplugged CPU in fixup_irqs() and do the retrigger from there. Can you please add tracing to that one as well?

New tracelog attached. It just happened once.

Correct and it worked as expected.

     migration/3-24    [003] d..1   275.665751: msi_set_affinity: quirk[1] new vector allocated, new apic = 4 vector = 36 this apic = 6
     migration/3-24    [003] d..1   275.665776: msi_set_affinity: Redirect to new vector 36 on old apic 6
     migration/3-24    [003] d..1   275.665789: msi_set_affinity: Transition to new target apic 4 vector 36
     migration/3-24    [003] d..1   275.665790: msi_set_affinity: Update Done [IRR 0]: irq 123 Nvec 36 Napic 4
     migration/3-24    [003] d..1   275.666792: fixup_irqs: retrigger vector 33 irq 123

So looking at your trace further down, the problem is not the last one. It dies already before that:

           <...>-14    [001] d..1   284.901587: msi_set_affinity: quirk[1] new vector allocated, new apic = 6 vector = 33 this apic = 2
           <...>-14    [001] d..1   284.901604: msi_set_affinity: Direct Update: irq 123 Ovec=33 Oapic 2 Nvec 33 Napic 6

Here, the interrupts stop coming in and that's just a regular direct update, i.e. same vector, different CPU. The update below is updating a dead device already.

     migration/3-24    [003] d..1   284.924960: msi_set_affinity: quirk[1] new vector allocated, new apic = 4 vector = 36 this apic = 6
     migration/3-24    [003] d..1   284.924987: msi_set_affinity: Redirect to new vector 36 on old apic 6
     migration/3-24    [003] d..1   284.924999: msi_set_affinity: Transition to new target apic 4 vector 36
     migration/3-24    [003] d..1   284.925000: msi_set_affinity: Update Done [IRR 0]: irq 123 Nvec 36 Napic 4

TBH, I can't see anything what's wrong here from the kernel side and as this is new silicon and you're the only ones reporting this it seems that this is something which is specific to that particular hardware. Have you talked to the hardware people about this?

Thanks,

tglx

#2 lore raw

Hi Thomas,

Quoted 22 lines · Thomas Gleixner

Ashok,

Quoted 7 lines · Ashok
Quoted 5 lines

But as this last one is the migration thread, aka stomp machine, I assume this is a hotplug operation. Which means the CPU cannot handle interrupts anymore. In that case we check the old vector on the unplugged CPU in fixup_irqs() and do the retrigger from there. Can you please add tracing to that one as well?

New tracelog attached. It just happened once.

Correct and it worked as expected.

     migration/3-24    [003] d..1   275.665751: msi_set_affinity: quirk[1] new vector allocated, new apic = 4 vector = 36 this apic = 6
     migration/3-24    [003] d..1   275.665776: msi_set_affinity: Redirect to new vector 36 on old apic 6
     migration/3-24    [003] d..1   275.665789: msi_set_affinity: Transition to new target apic 4 vector 36
     migration/3-24    [003] d..1   275.665790: msi_set_affinity: Update Done [IRR 0]: irq 123 Nvec 36 Napic 4
     migration/3-24    [003] d..1   275.666792: fixup_irqs: retrigger vector 33 irq 123

So looking at your trace further down, the problem is not the last one. It dies already before that:

Quoted 6 lines
           <...>-14    [001] d..1   284.901587: msi_set_affinity: quirk[1] new vector allocated, new apic = 6 vector = 33 this apic = 2
           <...>-14    [001] d..1   284.901604: msi_set_affinity: Direct Update: irq 123 Ovec=33 Oapic 2 Nvec 33 Napic 6

Here, the interrupts stop coming in and that's just a regular direct update, i.e. same vector, different CPU. The update below is updating a dead device already.

So we lost the interrupt either before or after we migrated to apic2?

In the direct update when we migrate same vector but different CPU if there was a pending IRR on the current CPU, where is that getting cleaned up?

Quoted 11 lines
     migration/3-24    [003] d..1   284.924960: msi_set_affinity: quirk[1] new vector allocated, new apic = 4 vector = 36 this apic = 6
     migration/3-24    [003] d..1   284.924987: msi_set_affinity: Redirect to new vector 36 on old apic 6
     migration/3-24    [003] d..1   284.924999: msi_set_affinity: Transition to new target apic 4 vector 36
     migration/3-24    [003] d..1   284.925000: msi_set_affinity: Update Done [IRR 0]: irq 123 Nvec 36 Napic 4

TBH, I can't see anything what's wrong here from the kernel side and as this is new silicon and you're the only ones reporting this it seems that this is something which is specific to that particular hardware. Have you talked to the hardware people about this?

After chasing it, I'm also trending to think that way. We had a question about the moderation timer and how its affecting this behavior. Mathias tried to turn off the moderation timer, and we are still able to see this hang. We are reaching out to the HW folks to get a handle on this.

With legacy MSI we can have these races and kernel is trying to do the song and dance, but we see this happening even when IR is turned on. Which is perplexing. I think when we have IR, once we do the change vector and flush the interrupt entry cache, if there was an outstandng one in flight it should be in IRR. Possibly should be clearned up by the send_cleanup_vector() i suppose.

Cheers,
Ashok

#3 lore raw

Ashok,

Quoted 12 lines · Ashok
Quoted 5 lines · Thomas Gleixner

TBH, I can't see anything what's wrong here from the kernel side and as this is new silicon and you're the only ones reporting this it seems that this is something which is specific to that particular hardware. Have you talked to the hardware people about this?

After chasing it, I'm also trending to think that way. We had a question about the moderation timer and how its affecting this behavior. Mathias tried to turn off the moderation timer, and we are still able to see this hang. We are reaching out to the HW folks to get a handle on this.

Quoted 6 lines

With legacy MSI we can have these races and kernel is trying to do the song and dance, but we see this happening even when IR is turned on. Which is perplexing. I think when we have IR, once we do the change vector and flush the interrupt entry cache, if there was an outstandng one in flight it should be in IRR. Possibly should be clearned up by the send_cleanup_vector() i suppose.

Ouch. With IR this really should never happen and yes the old vector will catch one which was raised just before the migration disabled the IR entry. During the change nothing can go wrong because the entry is disabled and only reenabled after it's flushed which will send a pending one to the new vector.

But if you see the issue with IR as well, then there is something wrong with the local APICs in that CPU.

Thanks,

tglx

#4 lore raw

Hi Thomas,

Quoted 10 lines · Thomas Gleixner

Ashok,

Quoted 6 lines · Ashok

With legacy MSI we can have these races and kernel is trying to do the song and dance, but we see this happening even when IR is turned on. Which is perplexing. I think when we have IR, once we do the change vector and flush the interrupt entry cache, if there was an outstandng one in flight it should be in IRR. Possibly should be clearned up by the send_cleanup_vector() i suppose.

Quoted 5 lines

Ouch. With IR this really should never happen and yes the old vector will catch one which was raised just before the migration disabled the IR entry. During the change nothing can go wrong because the entry is disabled and only reenabled after it's flushed which will send a pending one to the new vector.

with IR, I'm not sure if we actually mask the interrupt except when its a Posted Interrupt.

We do an atomic update to IRTE, with cmpxchg_double

	ret = cmpxchg_double(&irte->low, &irte->high,
			     irte->low, irte->high,
			     irte_modified->low, irte_modified->high);

followed by flushing the interrupt entry cache. After which any old ones in flight before the flush should be sittig in IRR on the outgoing cpu.

The send_cleanup_vector() sends IPI to the apic_id->old_cpu which would be the cpu we are running on correct? and this is a self_ipi to IRQ_MOVE_CLEANUP_VECTOR.

smp_irq_move_cleanup_interrupt() seems to check IRR with apicid_prev_vector()

	irr = apic_read(APIC_IRR + (vector / 32 * 0x10));
	if (irr & (1U << (vector % 32))) {
		apic->send_IPI_self(IRQ_MOVE_CLEANUP_VECTOR);
		continue;
	}

And this would allow any pending IRR bits in the outgoing CPU to call the relevant ISR's before draining all vectors on the outgoing CPU.

Does it sound right?

I couldn't quite pin down how the device ISR's are hooked up through this send_cleanup_vector() and what follows.

#5 lore raw

Ashok,

Quoted 9 lines · Ashok
Quoted 8 lines · Thomas Gleixner
Quoted 6 lines · Ashok

With legacy MSI we can have these races and kernel is trying to do the song and dance, but we see this happening even when IR is turned on. Which is perplexing. I think when we have IR, once we do the change vector and flush the interrupt entry cache, if there was an outstandng one in flight it should be in IRR. Possibly should be clearned up by the send_cleanup_vector() i suppose.

Quoted 8 lines
Quoted 5 lines

Ouch. With IR this really should never happen and yes the old vector will catch one which was raised just before the migration disabled the IR entry. During the change nothing can go wrong because the entry is disabled and only reenabled after it's flushed which will send a pending one to the new vector.

with IR, I'm not sure if we actually mask the interrupt except when its a Posted Interrupt.

Indeed. Memory tricked me.

Quoted 5 lines

We do an atomic update to IRTE, with cmpxchg_double

	ret = cmpxchg_double(&irte->low, &irte->high,
			     irte->low, irte->high,
			     irte_modified->low, irte_modified->high);

We only do this if:

        if ((irte->pst == 1) || (irte_modified->pst == 1))

i.e. the irte entry was or is going to be used for posted interrupts.

In the non-posted remapping case we do:

       set_64bit(&irte->low, irte_modified->low);
       set_64bit(&irte->high, irte_modified->high);
Quoted 3 lines

followed by flushing the interrupt entry cache. After which any old ones in flight before the flush should be sittig in IRR on the outgoing cpu.

Correct.

Quoted 3 lines

The send_cleanup_vector() sends IPI to the apic_id->old_cpu which would be the cpu we are running on correct? and this is a self_ipi to IRQ_MOVE_CLEANUP_VECTOR.

Yes, for the IR case the cleanup vector IPI is sent right away because the IR logic allegedly guarantees that after flushing the IRTE cache no interrupt can be sent to the old vector. IOW, after the flush a vector sent to the previous CPU must be already in the IRR of that CPU.

Quoted 9 lines

smp_irq_move_cleanup_interrupt() seems to check IRR with apicid_prev_vector()

	irr = apic_read(APIC_IRR + (vector / 32 * 0x10));
	if (irr & (1U << (vector % 32))) {
		apic->send_IPI_self(IRQ_MOVE_CLEANUP_VECTOR);
		continue;
	}
Quoted 3 lines

And this would allow any pending IRR bits in the outgoing CPU to call the relevant ISR's before draining all vectors on the outgoing CPU.

No. When the CPU goes offline it does not handle anything after moving the interrupts away. The pending bits are handled in fixup_irq() after the forced migration. If the vector is set in the IRR it sends an IPI to the new target CPU.

That IRR check is paranoia for the following case where interrupt migration happens between live CPUs:

Old CPU

set affinity <- device sends to old vector/cpu
...
irte_...()
send_cleanup_vector();

handle_cleanup_vector() <- Would remove the vector
without that IRR check and then result in a spurious interrupt

This should actually never happen because the cleanup vector is the lowest priority vector.

Quoted 2 lines

I couldn't quite pin down how the device ISR's are hooked up through this send_cleanup_vector() and what follows.

Device ISRs don't know anything about the underlying irq machinery.

The non IR migration does:

    Old CPU          New CPU

    set affinity
    ...                 
                                <- device sends to new vector/cpu
                     handle_irq
                       ack()
                        apic_ack_irq()
                          irq_complete_move()
                            send_cleanup_vector()

    handle_cleanup_vector()

In case that an interrupt still hits the old CPU:

    set affinity                <- device sends to old vector/cpu
    ...
    handle_irq
     ack()
      apic_ack_irq()
       irq_complete_move()      (Does not send cleanup vector because wrong CPU)

                                <- device sends to new vector/cpu
                     handle_irq
                       ack()
                        apic_ack_irq()
                          irq_complete_move()
                            send_cleanup_vector()

    handle_cleanup_vector()

Thanks,

tglx