MSI interrupt for xhci still lost on 5.6-rc6 after cpu hotplug
A message that is not in the archive (sent off-list or not yet received): 20200508005528.GB61703@otc-nc-03
Ashok,
Quoted 7 lines · Ashok
Quoted 5 lines
But as this last one is the migration thread, aka stomp machine, I assume this is a hotplug operation. Which means the CPU cannot handle interrupts anymore. In that case we check the old vector on the unplugged CPU in
fixup_irqs()and do the retrigger from there. Can you please add tracing to that one as well?New tracelog attached. It just happened once.
Correct and it worked as expected.
migration/3-24 [003] d..1 275.665751: msi_set_affinity: quirk[1] new vector allocated, new apic = 4 vector = 36 this apic = 6
migration/3-24 [003] d..1 275.665776: msi_set_affinity: Redirect to new vector 36 on old apic 6
migration/3-24 [003] d..1 275.665789: msi_set_affinity: Transition to new target apic 4 vector 36
migration/3-24 [003] d..1 275.665790: msi_set_affinity: Update Done [IRR 0]: irq 123 Nvec 36 Napic 4
migration/3-24 [003] d..1 275.666792: fixup_irqs: retrigger vector 33 irq 123So looking at your trace further down, the problem is not the last one. It dies already before that:
<...>-14 [001] d..1 284.901587: msi_set_affinity: quirk[1] new vector allocated, new apic = 6 vector = 33 this apic = 2
<...>-14 [001] d..1 284.901604: msi_set_affinity: Direct Update: irq 123 Ovec=33 Oapic 2 Nvec 33 Napic 6Here, the interrupts stop coming in and that's just a regular direct update, i.e. same vector, different CPU. The update below is updating a dead device already.
migration/3-24 [003] d..1 284.924960: msi_set_affinity: quirk[1] new vector allocated, new apic = 4 vector = 36 this apic = 6
migration/3-24 [003] d..1 284.924987: msi_set_affinity: Redirect to new vector 36 on old apic 6
migration/3-24 [003] d..1 284.924999: msi_set_affinity: Transition to new target apic 4 vector 36
migration/3-24 [003] d..1 284.925000: msi_set_affinity: Update Done [IRR 0]: irq 123 Nvec 36 Napic 4TBH, I can't see anything what's wrong here from the kernel side and as this is new silicon and you're the only ones reporting this it seems that this is something which is specific to that particular hardware. Have you talked to the hardware people about this?
Thanks,
tglx
Hi Thomas,
Quoted 22 lines · Thomas Gleixner
Ashok,
Quoted 7 lines · Ashok
Quoted 5 lines
But as this last one is the migration thread, aka stomp machine, I assume this is a hotplug operation. Which means the CPU cannot handle interrupts anymore. In that case we check the old vector on the unplugged CPU in
fixup_irqs()and do the retrigger from there. Can you please add tracing to that one as well?New tracelog attached. It just happened once.
Correct and it worked as expected.
migration/3-24 [003] d..1 275.665751: msi_set_affinity: quirk[1] new vector allocated, new apic = 4 vector = 36 this apic = 6 migration/3-24 [003] d..1 275.665776: msi_set_affinity: Redirect to new vector 36 on old apic 6 migration/3-24 [003] d..1 275.665789: msi_set_affinity: Transition to new target apic 4 vector 36 migration/3-24 [003] d..1 275.665790: msi_set_affinity: Update Done [IRR 0]: irq 123 Nvec 36 Napic 4 migration/3-24 [003] d..1 275.666792: fixup_irqs: retrigger vector 33 irq 123So looking at your trace further down, the problem is not the last one. It dies already before that:
Quoted 6 lines
<...>-14 [001] d..1 284.901587: msi_set_affinity: quirk[1] new vector allocated, new apic = 6 vector = 33 this apic = 2 <...>-14 [001] d..1 284.901604: msi_set_affinity: Direct Update: irq 123 Ovec=33 Oapic 2 Nvec 33 Napic 6Here, the interrupts stop coming in and that's just a regular direct update, i.e. same vector, different CPU. The update below is updating a dead device already.
So we lost the interrupt either before or after we migrated to apic2?
In the direct update when we migrate same vector but different CPU if there was a pending IRR on the current CPU, where is that getting cleaned up?
Quoted 11 lines
migration/3-24 [003] d..1 284.924960: msi_set_affinity: quirk[1] new vector allocated, new apic = 4 vector = 36 this apic = 6 migration/3-24 [003] d..1 284.924987: msi_set_affinity: Redirect to new vector 36 on old apic 6 migration/3-24 [003] d..1 284.924999: msi_set_affinity: Transition to new target apic 4 vector 36 migration/3-24 [003] d..1 284.925000: msi_set_affinity: Update Done [IRR 0]: irq 123 Nvec 36 Napic 4TBH, I can't see anything what's wrong here from the kernel side and as this is new silicon and you're the only ones reporting this it seems that this is something which is specific to that particular hardware. Have you talked to the hardware people about this?
After chasing it, I'm also trending to think that way. We had a question about the moderation timer and how its affecting this behavior. Mathias tried to turn off the moderation timer, and we are still able to see this hang. We are reaching out to the HW folks to get a handle on this.
With legacy MSI we can have these races and kernel is trying to do the song and dance, but we see this happening even when IR is turned on. Which is perplexing. I think when we have IR, once we do the change vector and flush the interrupt entry cache, if there was an outstandng one in flight it should be in IRR. Possibly should be clearned up by the send_cleanup_vector() i suppose.
Cheers,
Ashok
Ashok,
Quoted 12 lines · Ashok
Quoted 5 lines · Thomas Gleixner
TBH, I can't see anything what's wrong here from the kernel side and as this is new silicon and you're the only ones reporting this it seems that this is something which is specific to that particular hardware. Have you talked to the hardware people about this?
After chasing it, I'm also trending to think that way. We had a question about the moderation timer and how its affecting this behavior. Mathias tried to turn off the moderation timer, and we are still able to see this hang. We are reaching out to the HW folks to get a handle on this.
Quoted 6 lines
With legacy MSI we can have these races and kernel is trying to do the song and dance, but we see this happening even when IR is turned on. Which is perplexing. I think when we have IR, once we do the change vector and flush the interrupt entry cache, if there was an outstandng one in flight it should be in IRR. Possibly should be clearned up by the
send_cleanup_vector()i suppose.
Ouch. With IR this really should never happen and yes the old vector will catch one which was raised just before the migration disabled the IR entry. During the change nothing can go wrong because the entry is disabled and only reenabled after it's flushed which will send a pending one to the new vector.
But if you see the issue with IR as well, then there is something wrong with the local APICs in that CPU.
Thanks,
tglx
Hi Thomas,
Quoted 10 lines · Thomas Gleixner
Ashok,
Quoted 6 lines · Ashok
With legacy MSI we can have these races and kernel is trying to do the song and dance, but we see this happening even when IR is turned on. Which is perplexing. I think when we have IR, once we do the change vector and flush the interrupt entry cache, if there was an outstandng one in flight it should be in IRR. Possibly should be clearned up by the
send_cleanup_vector()i suppose.
Quoted 5 lines
Ouch. With IR this really should never happen and yes the old vector will catch one which was raised just before the migration disabled the IR entry. During the change nothing can go wrong because the entry is disabled and only reenabled after it's flushed which will send a pending one to the new vector.
with IR, I'm not sure if we actually mask the interrupt except when its a Posted Interrupt.
We do an atomic update to IRTE, with cmpxchg_double
ret = cmpxchg_double(&irte->low, &irte->high, irte->low, irte->high, irte_modified->low, irte_modified->high);
followed by flushing the interrupt entry cache. After which any old ones in flight before the flush should be sittig in IRR on the outgoing cpu.
The send_cleanup_vector() sends IPI to the apic_id->old_cpu which would be the cpu we are running on correct? and this is a self_ipi to IRQ_MOVE_CLEANUP_VECTOR.
smp_irq_move_cleanup_interrupt() seems to check IRR with apicid_prev_vector()
irr = apic_read(APIC_IRR + (vector / 32 * 0x10));
if (irr & (1U << (vector % 32))) {
apic->send_IPI_self(IRQ_MOVE_CLEANUP_VECTOR);
continue;
}And this would allow any pending IRR bits in the outgoing CPU to call the relevant ISR's before draining all vectors on the outgoing CPU.
Does it sound right?
I couldn't quite pin down how the device ISR's are hooked up through this send_cleanup_vector() and what follows.
Ashok,
Quoted 9 lines · Ashok
Quoted 8 lines · Thomas Gleixner
Quoted 6 lines · Ashok
With legacy MSI we can have these races and kernel is trying to do the song and dance, but we see this happening even when IR is turned on. Which is perplexing. I think when we have IR, once we do the change vector and flush the interrupt entry cache, if there was an outstandng one in flight it should be in IRR. Possibly should be clearned up by the
send_cleanup_vector()i suppose.
Quoted 8 lines
Quoted 5 lines
Ouch. With IR this really should never happen and yes the old vector will catch one which was raised just before the migration disabled the IR entry. During the change nothing can go wrong because the entry is disabled and only reenabled after it's flushed which will send a pending one to the new vector.
with IR, I'm not sure if we actually mask the interrupt except when its a Posted Interrupt.
Indeed. Memory tricked me.
Quoted 5 lines
We do an atomic update to IRTE, with
cmpxchg_doubleret = cmpxchg_double(&irte->low, &irte->high, irte->low, irte->high, irte_modified->low, irte_modified->high);
We only do this if:
if ((irte->pst == 1) || (irte_modified->pst == 1))
i.e. the irte entry was or is going to be used for posted interrupts.
In the non-posted remapping case we do:
set_64bit(&irte->low, irte_modified->low);
set_64bit(&irte->high, irte_modified->high);Quoted 3 lines
followed by flushing the interrupt entry cache. After which any old ones in flight before the flush should be sittig in IRR on the outgoing cpu.
Correct.
Quoted 3 lines
The
send_cleanup_vector()sends IPI to theapic_id->old_cpuwhich would be the cpu we are running on correct? and this is aself_ipitoIRQ_MOVE_CLEANUP_VECTOR.
Yes, for the IR case the cleanup vector IPI is sent right away because the IR logic allegedly guarantees that after flushing the IRTE cache no interrupt can be sent to the old vector. IOW, after the flush a vector sent to the previous CPU must be already in the IRR of that CPU.
Quoted 9 lines
smp_irq_move_cleanup_interrupt()seems to check IRR withapicid_prev_vector()irr = apic_read(APIC_IRR + (vector / 32 * 0x10)); if (irr & (1U << (vector % 32))) { apic->send_IPI_self(IRQ_MOVE_CLEANUP_VECTOR); continue; }
Quoted 3 lines
And this would allow any pending IRR bits in the outgoing CPU to call the relevant ISR's before draining all vectors on the outgoing CPU.
No. When the CPU goes offline it does not handle anything after moving the interrupts away. The pending bits are handled in fixup_irq() after the forced migration. If the vector is set in the IRR it sends an IPI to the new target CPU.
That IRR check is paranoia for the following case where interrupt migration happens between live CPUs:
Old CPU
set affinity <- device sends to old vector/cpu
...
irte_...()
send_cleanup_vector();
handle_cleanup_vector() <- Would remove the vector
without that IRR check and then result in a spurious interrupt
This should actually never happen because the cleanup vector is the lowest priority vector.
Quoted 2 lines
I couldn't quite pin down how the device ISR's are hooked up through this
send_cleanup_vector()and what follows.
Device ISRs don't know anything about the underlying irq machinery.
The non IR migration does:
Old CPU New CPU
set affinity
...
<- device sends to new vector/cpu
handle_irq
ack()
apic_ack_irq()
irq_complete_move()
send_cleanup_vector()
handle_cleanup_vector()In case that an interrupt still hits the old CPU:
set affinity <- device sends to old vector/cpu
...
handle_irq
ack()
apic_ack_irq()
irq_complete_move() (Does not send cleanup vector because wrong CPU)
<- device sends to new vector/cpu
handle_irq
ack()
apic_ack_irq()
irq_complete_move()
send_cleanup_vector()
handle_cleanup_vector()Thanks,
tglx