IRQ affinity enforced only after first interrupt.
Hello,
I'm working on an issue where affinity changes to IRQ only have effect after the first interrupt which still happens on the original core. I understand that the decision regarding it takes place in this code:
if (irq_can_move_pcntxt(data)) {
ret = chip->irq_set_affinity(data, mask, false);
switch (ret) {
case IRQ_SET_MASK_OK:
cpumask_copy(data->affinity, mask);
case IRQ_SET_MASK_OK_NOCOPY:
irq_set_thread_affinity(desc);
ret = 0;
}
} else {
irqd_set_move_pending(data);
irq_copy_pending(desc, mask);
}Which means that the "IRQD_MOVE_PCNTXT" flag is not set in irq_data->state_use_accessors. I was able to add this flag using irq_modify_status(), which is probably not the way to go. This option also doesn't exist in older kernels (2.6.32)
So the question is, when irq_desc is created, how is it determined that "IRQD_MOVE_PCNTXT" flag is set?
Thanks,
Yevgeny
Bjorn Helgaas
maintainer[This is not really a PCI question, so +cc Thomas, LKML.]
Quoted 31 lines · Yevgeny Petrilin
Hello,
I'm working on an issue where affinity changes to IRQ only have effect after the first interrupt which still happens on the original core. I understand that the decision regarding it takes place in this code:
if (
irq_can_move_pcntxt(data)) {
ret =chip->irq_set_affinity(data, mask, false); switch (ret) {
caseIRQ_SET_MASK_OK:
cpumask_copy(data->affinity, mask); caseIRQ_SET_MASK_OK_NOCOPY:
irq_set_thread_affinity(desc); ret = 0;
}
} else {
irqd_set_move_pending(data);irq_copy_pending(desc, mask);
}Which means that the "
IRQD_MOVE_PCNTXT" flag is not set inirq_data->state_use_accessors. I was able to add this flag usingirq_modify_status(), which is probably not the way to go. This option also doesn't exist in older kernels (2.6.32)So the question is, when
irq_descis created, how is it determined that "IRQD_MOVE_PCNTXT" flag is set?Thanks,
Yevgeny
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
The architecture specific code will determine whether the IRQ could be migrated in process context. For example, the IRQ_MOVE_PCNTXT flag will be set on x86 systems if interrupt remapping is enabled.
Quoted 39 lines · Bjorn Helgaas
[This is not really a PCI question, so +cc Thomas, LKML.]
Quoted 23 lines · Yevgeny Petrilin
Hello,
I'm working on an issue where affinity changes to IRQ only have effect after the first interrupt which still happens on the original core. I understand that the decision regarding it takes place in this code:
if (irq_can_move_pcntxt(data)) { ret = chip->irq_set_affinity(data, mask, false); switch (ret) { case IRQ_SET_MASK_OK: cpumask_copy(data->affinity, mask); case IRQ_SET_MASK_OK_NOCOPY: irq_set_thread_affinity(desc); ret = 0; } } else { irqd_set_move_pending(data); irq_copy_pending(desc, mask); }Which means that the "
IRQD_MOVE_PCNTXT" flag is not set inirq_data->state_use_accessors. I was able to add this flag usingirq_modify_status(), which is probably not the way to go. This option also doesn't exist in older kernels (2.6.32)Quoted 8 lines
So the question is, when
irq_descis created, how is it determined that "IRQD_MOVE_PCNTXT" flag is set?Thanks,
Yevgeny
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 4 lines
The architecture specific code will determine whether the IRQ could be migrated in process context. For example, the
IRQ_MOVE_PCNTXTflag will be set on x86 systems if interrupt remapping is enabled.
Actually I am encountering this issue with x86, and see different behavior with different HW devices (NICs). On same machine I have one device that responds immediately to affinity changes while the other one changes the affinity only after first interrupt.
Quoted 33 lines · Bjorn Helgaas
Quoted 31 lines
[This is not really a PCI question, so +cc Thomas, LKML.]
Quoted 27 lines · Yevgeny Petrilin
Hello,
I'm working on an issue where affinity changes to IRQ only have effect after the first interrupt which still happens on the original core. I understand that the decision regarding it takes place in this code:
if (irq_can_move_pcntxt(data)) { ret = chip->irq_set_affinity(data, mask, false); switch (ret) { case IRQ_SET_MASK_OK: cpumask_copy(data->affinity, mask); case IRQ_SET_MASK_OK_NOCOPY: irq_set_thread_affinity(desc); ret = 0; } } else { irqd_set_move_pending(data); irq_copy_pending(desc, mask); }Which means that the "
IRQD_MOVE_PCNTXT" flag is not set inirq_data->state_use_accessors. I was able to add this flag usingirq_modify_status(), which is probably not the way to go. This option also doesn't exist in older kernels (2.6.32)So the question is, when
irq_descis created, how is it determined that "IRQD_MOVE_PCNTXT" flag is set?Thanks,
Yevgeny
Quoted 9 lines · Yevgeny Petrilin
Quoted 4 lines
The architecture specific code will determine whether the IRQ could be migrated in process context. For example, the
IRQ_MOVE_PCNTXTflag will be set on x86 systems if interrupt remapping is enabled.Actually I am encountering this issue with x86, and see different behavior with different HW devices (NICs). On same machine I have one device that responds immediately to affinity changes while the other one changes the affinity only after first interrupt.
That simply depends on the underlying hardware. On certain hardware we can change the affinity only in hard interrupt context, that means right when a interrupt of that device is delivered.
On the other devices we can change it right away and the corresponding interrupt chips set IRQ_MOVE_PCNTXT to indicate that.
There is nothing we can do about this. It's dictated by hardware.
Thanks,
tglx
Quoted 10 lines
Quoted 9 lines
Quoted 4 lines
The architecture specific code will determine whether the IRQ could be migrated in process context. For example, the
IRQ_MOVE_PCNTXTflag will be set on x86 systems if interrupt remapping is enabled.Actually I am encountering this issue with x86, and see different behavior with different HW devices (NICs). On same machine I have one device that responds immediately to affinity changes while the other one changes the affinity only after first interrupt.
Quoted 9 lines
That simply depends on the underlying hardware. On certain hardware we can change the affinity only in hard interrupt context, that means right when a interrupt of that device is delivered.
On the other devices we can change it right away and the corresponding interrupt chips set
IRQ_MOVE_PCNTXTto indicate that.There is nothing we can do about this. It's dictated by hardware.
Thanks for the explanation,
Which capabilities of the HW show whether IRQ_MOVE_PCNTXT can be set or not? Is it done by reading configuration from PCI?
Thanks,
Yevgeny
Quoted 17 lines · Yevgeny Petrilin
Quoted 17 lines
Quoted 9 lines
Quoted 4 lines
The architecture specific code will determine whether the IRQ could be migrated in process context. For example, the
IRQ_MOVE_PCNTXTflag will be set on x86 systems if interrupt remapping is enabled.Actually I am encountering this issue with x86, and see different behavior with different HW devices (NICs). On same machine I have one device that responds immediately to affinity changes while the other one changes the affinity only after first interrupt.
That simply depends on the underlying hardware. On certain hardware we can change the affinity only in hard interrupt context, that means right when a interrupt of that device is delivered.
On the other devices we can change it right away and the corresponding interrupt chips set
IRQ_MOVE_PCNTXTto indicate that.
Quoted 6 lines
Quoted 2 lines
There is nothing we can do about this. It's dictated by hardware.
Thanks for the explanation,
Which capabilities of the HW show whetherIRQ_MOVE_PCNTXTcan be set or not? Is it done by reading configuration from PCI?
It's done by reading the specs of the interrupt controllers. This is not at PCI (device) level. It's a property of the interrupt controller (PIC, APIC, IOAPIC) and additional features like interrupt remapping.
The device merily uses an interrupt, but it does not know at all which underlying interrupt controller is handling it.
The only choice a device driver has is between pin based interrupts and Message Signaled Interrupts, when the hardware supports it. This information is retrieved from the PCI config space.
Thanks,
tglx
Quoted 12 lines · Thomas Gleixner
Quoted 9 lines · Yevgeny Petrilin
Quoted 4 lines
The architecture specific code will determine whether the IRQ could be migrated in process context. For example, the
IRQ_MOVE_PCNTXTflag will be set on x86 systems if interrupt remapping is enabled.Actually I am encountering this issue with x86, and see different behavior with different HW devices (NICs). On same machine I have one device that responds immediately to affinity changes while the other one changes the affinity only after first interrupt.
Quoted 6 lines
That simply depends on the underlying hardware. On certain hardware we can change the affinity only in hard interrupt context, that means right when a interrupt of that device is delivered.
On the other devices we can change it right away and the corresponding interrupt chips set
IRQ_MOVE_PCNTXTto indicate that.
Actually, even with IRQ_MOVE_PCNTXT capable chips, a hardware handler still might be called on a core that belongs to old affinity, after the successful write of new affinity. Threaded handlers are also racy with irq affinity updates.
If that is inconsistency, bug or design?
Quoted 1 line
There is nothing we can do about this. It's dictated by hardware.
May be we could wait for desc->pending_mask to be cleared before returning from irq_set_affinity()?
Quoted 8 lines
Thanks,
tglx
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Signature · 3 lines
Regards, Alexander Gordeev [email protected]
Quoted 20 lines · Alexander Gordeev
Quoted 12 lines · Thomas Gleixner
Quoted 9 lines · Yevgeny Petrilin
Quoted 4 lines
The architecture specific code will determine whether the IRQ could be migrated in process context. For example, the
IRQ_MOVE_PCNTXTflag will be set on x86 systems if interrupt remapping is enabled.Actually I am encountering this issue with x86, and see different behavior with different HW devices (NICs). On same machine I have one device that responds immediately to affinity changes while the other one changes the affinity only after first interrupt.
Quoted 6 lines
That simply depends on the underlying hardware. On certain hardware we can change the affinity only in hard interrupt context, that means right when a interrupt of that device is delivered.
On the other devices we can change it right away and the corresponding interrupt chips set
IRQ_MOVE_PCNTXTto indicate that.
Quoted 6 lines
Actually, even with
IRQ_MOVE_PCNTXTcapable chips, a hardware handler still might be called on a core that belongs to old affinity, after the successful write of new affinity. Threaded handlers are also racy with irq affinity updates.If that is inconsistency, bug or design?
Well, irq affinity updates are not designed to be immediate. There is no point in doing so.
Quoted 4 lines
Quoted 1 line
There is nothing we can do about this. It's dictated by hardware.
May be we could wait for
desc->pending_maskto be cleared before returning fromirq_set_affinity()?
If that device does not issue an interrupt for a long time, e.g. because the interface is down, then you are stuck there forever.
What's the point of this? One interrupt on the wrong core is nothing we need to worry about.
Thanks,
tglx