Skip to contentSkip to conversation

IRQ affinity enforced only after first interrupt.

DISCUSSION Yevgeny Petrilin 6 participants 9 messages Mar 26, 2012 – Apr 5, 2012 PCI

lore
Share & tools
lore
#0 lore raw

Hello,

I'm working on an issue where affinity changes to IRQ only have effect after the first interrupt which still happens on the original core. I understand that the decision regarding it takes place in this code:

	if (irq_can_move_pcntxt(data)) {
		ret = chip->irq_set_affinity(data, mask, false);
		switch (ret) {
		case IRQ_SET_MASK_OK:
			cpumask_copy(data->affinity, mask);
		case IRQ_SET_MASK_OK_NOCOPY:
			irq_set_thread_affinity(desc);
			ret = 0;
		}
	} else {
		irqd_set_move_pending(data);
		irq_copy_pending(desc, mask);
	}

Which means that the "IRQD_MOVE_PCNTXT" flag is not set in irq_data->state_use_accessors. I was able to add this flag using irq_modify_status(), which is probably not the way to go. This option also doesn't exist in older kernels (2.6.32)

So the question is, when irq_desc is created, how is it determined that "IRQD_MOVE_PCNTXT" flag is set?

Thanks,
Yevgeny

maintainer
#1 lore raw

[This is not really a PCI question, so +cc Thomas, LKML.]

Quoted 31 lines · Yevgeny Petrilin

Hello,

I'm working on an issue where affinity changes to IRQ only have effect after the first interrupt which still happens on the original core. I understand that the decision regarding it takes place in this code:

       if (irq_can_move_pcntxt(data)) {
               ret = chip->irq_set_affinity(data, mask, false); switch (ret) {
               case IRQ_SET_MASK_OK:
                       cpumask_copy(data->affinity, mask); case IRQ_SET_MASK_OK_NOCOPY:
                       irq_set_thread_affinity(desc); ret = 0;
               }
       } else {
               irqd_set_move_pending(data); irq_copy_pending(desc, mask);
       }

Which means that the "IRQD_MOVE_PCNTXT" flag is not set in irq_data->state_use_accessors. I was able to add this flag using irq_modify_status(), which is probably not the way to go. This option also doesn't exist in older kernels (2.6.32)

So the question is, when irq_desc is created, how is it determined that "IRQD_MOVE_PCNTXT" flag is set?

Thanks,
Yevgeny
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

#2 lore raw

The architecture specific code will determine whether the IRQ could be migrated in process context. For example, the IRQ_MOVE_PCNTXT flag will be set on x86 systems if interrupt remapping is enabled.

Quoted 39 lines · Bjorn Helgaas

[This is not really a PCI question, so +cc Thomas, LKML.]

Quoted 23 lines · Yevgeny Petrilin

Hello,

I'm working on an issue where affinity changes to IRQ only have effect after the first interrupt which still happens on the original core. I understand that the decision regarding it takes place in this code:

       if (irq_can_move_pcntxt(data)) {
               ret = chip->irq_set_affinity(data, mask, false);
               switch (ret) {
               case IRQ_SET_MASK_OK:
                       cpumask_copy(data->affinity, mask);
               case IRQ_SET_MASK_OK_NOCOPY:
                       irq_set_thread_affinity(desc);
                       ret = 0;
               }
       } else {
               irqd_set_move_pending(data);
               irq_copy_pending(desc, mask);
       }

Which means that the "IRQD_MOVE_PCNTXT" flag is not set in irq_data->state_use_accessors. I was able to add this flag using irq_modify_status(), which is probably not the way to go. This option also doesn't exist in older kernels (2.6.32)

Quoted 8 lines

So the question is, when irq_desc is created, how is it determined that "IRQD_MOVE_PCNTXT" flag is set?

Thanks,
Yevgeny
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#3 lore raw
Quoted 4 lines

The architecture specific code will determine whether the IRQ could be migrated in process context. For example, the IRQ_MOVE_PCNTXT flag will be set on x86 systems if interrupt remapping is enabled.

Actually I am encountering this issue with x86, and see different behavior with different HW devices (NICs). On same machine I have one device that responds immediately to affinity changes while the other one changes the affinity only after first interrupt.

Quoted 33 lines · Bjorn Helgaas
Quoted 31 lines

[This is not really a PCI question, so +cc Thomas, LKML.]

Quoted 27 lines · Yevgeny Petrilin

Hello,

I'm working on an issue where affinity changes to IRQ only have effect after the first interrupt which still happens on the original core. I understand that the decision regarding it takes place in this code:

       if (irq_can_move_pcntxt(data)) {
               ret = chip->irq_set_affinity(data, mask, false);
               switch (ret) {
               case IRQ_SET_MASK_OK:
                       cpumask_copy(data->affinity, mask);
               case IRQ_SET_MASK_OK_NOCOPY:
                       irq_set_thread_affinity(desc);
                       ret = 0;
               }
       } else {
               irqd_set_move_pending(data);
               irq_copy_pending(desc, mask);
       }

Which means that the "IRQD_MOVE_PCNTXT" flag is not set in irq_data->state_use_accessors. I was able to add this flag using irq_modify_status(), which is probably not the way to go. This option also doesn't exist in older kernels (2.6.32)

So the question is, when irq_desc is created, how is it determined that "IRQD_MOVE_PCNTXT" flag is set?

Thanks,
Yevgeny

#4 lore raw
Quoted 9 lines · Yevgeny Petrilin
Quoted 4 lines

The architecture specific code will determine whether the IRQ could be migrated in process context. For example, the IRQ_MOVE_PCNTXT flag will be set on x86 systems if interrupt remapping is enabled.

Actually I am encountering this issue with x86, and see different behavior with different HW devices (NICs). On same machine I have one device that responds immediately to affinity changes while the other one changes the affinity only after first interrupt.

That simply depends on the underlying hardware. On certain hardware we can change the affinity only in hard interrupt context, that means right when a interrupt of that device is delivered.

On the other devices we can change it right away and the corresponding interrupt chips set IRQ_MOVE_PCNTXT to indicate that.

There is nothing we can do about this. It's dictated by hardware.

Thanks,

tglx

#5 lore raw
Quoted 10 lines
Quoted 9 lines
Quoted 4 lines

The architecture specific code will determine whether the IRQ could be migrated in process context. For example, the IRQ_MOVE_PCNTXT flag will be set on x86 systems if interrupt remapping is enabled.

Actually I am encountering this issue with x86, and see different behavior with different HW devices (NICs). On same machine I have one device that responds immediately to affinity changes while the other one changes the affinity only after first interrupt.

Quoted 9 lines

That simply depends on the underlying hardware. On certain hardware we can change the affinity only in hard interrupt context, that means right when a interrupt of that device is delivered.

On the other devices we can change it right away and the corresponding interrupt chips set IRQ_MOVE_PCNTXT to indicate that.

There is nothing we can do about this. It's dictated by hardware.

Thanks for the explanation,
Which capabilities of the HW show whether IRQ_MOVE_PCNTXT can be set or not? Is it done by reading configuration from PCI?

Thanks,
Yevgeny

#6 lore raw
Quoted 17 lines · Yevgeny Petrilin
Quoted 17 lines
Quoted 9 lines
Quoted 4 lines

The architecture specific code will determine whether the IRQ could be migrated in process context. For example, the IRQ_MOVE_PCNTXT flag will be set on x86 systems if interrupt remapping is enabled.

Actually I am encountering this issue with x86, and see different behavior with different HW devices (NICs). On same machine I have one device that responds immediately to affinity changes while the other one changes the affinity only after first interrupt.

That simply depends on the underlying hardware. On certain hardware we can change the affinity only in hard interrupt context, that means right when a interrupt of that device is delivered.

On the other devices we can change it right away and the corresponding interrupt chips set IRQ_MOVE_PCNTXT to indicate that.

Quoted 6 lines
Quoted 2 lines

There is nothing we can do about this. It's dictated by hardware.

Thanks for the explanation,
Which capabilities of the HW show whether IRQ_MOVE_PCNTXT can be set or not? Is it done by reading configuration from PCI?

It's done by reading the specs of the interrupt controllers. This is not at PCI (device) level. It's a property of the interrupt controller (PIC, APIC, IOAPIC) and additional features like interrupt remapping.

The device merily uses an interrupt, but it does not know at all which underlying interrupt controller is handling it.

The only choice a device driver has is between pin based interrupts and Message Signaled Interrupts, when the hardware supports it. This information is retrieved from the PCI config space.

Thanks,

tglx

#7 lore raw
In reply to Thomas Gleixner
Quoted 12 lines · Thomas Gleixner
Quoted 9 lines · Yevgeny Petrilin
Quoted 4 lines

The architecture specific code will determine whether the IRQ could be migrated in process context. For example, the IRQ_MOVE_PCNTXT flag will be set on x86 systems if interrupt remapping is enabled.

Actually I am encountering this issue with x86, and see different behavior with different HW devices (NICs). On same machine I have one device that responds immediately to affinity changes while the other one changes the affinity only after first interrupt.

Quoted 6 lines

That simply depends on the underlying hardware. On certain hardware we can change the affinity only in hard interrupt context, that means right when a interrupt of that device is delivered.

On the other devices we can change it right away and the corresponding interrupt chips set IRQ_MOVE_PCNTXT to indicate that.

Actually, even with IRQ_MOVE_PCNTXT capable chips, a hardware handler still might be called on a core that belongs to old affinity, after the successful write of new affinity. Threaded handlers are also racy with irq affinity updates.

If that is inconsistency, bug or design?

Quoted 1 line

There is nothing we can do about this. It's dictated by hardware.

May be we could wait for desc->pending_mask to be cleared before returning from irq_set_affinity()?

Quoted 8 lines

Thanks,

tglx
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Signature · 3 lines
Regards,
Alexander Gordeev
[email protected]
#8 lore raw
Quoted 20 lines · Alexander Gordeev
Quoted 12 lines · Thomas Gleixner
Quoted 9 lines · Yevgeny Petrilin
Quoted 4 lines

The architecture specific code will determine whether the IRQ could be migrated in process context. For example, the IRQ_MOVE_PCNTXT flag will be set on x86 systems if interrupt remapping is enabled.

Actually I am encountering this issue with x86, and see different behavior with different HW devices (NICs). On same machine I have one device that responds immediately to affinity changes while the other one changes the affinity only after first interrupt.

Quoted 6 lines

That simply depends on the underlying hardware. On certain hardware we can change the affinity only in hard interrupt context, that means right when a interrupt of that device is delivered.

On the other devices we can change it right away and the corresponding interrupt chips set IRQ_MOVE_PCNTXT to indicate that.

Quoted 6 lines

Actually, even with IRQ_MOVE_PCNTXT capable chips, a hardware handler still might be called on a core that belongs to old affinity, after the successful write of new affinity. Threaded handlers are also racy with irq affinity updates.

If that is inconsistency, bug or design?

Well, irq affinity updates are not designed to be immediate. There is no point in doing so.

Quoted 4 lines
Quoted 1 line

There is nothing we can do about this. It's dictated by hardware.

May be we could wait for desc->pending_mask to be cleared before returning from irq_set_affinity()?

If that device does not issue an interrupt for a long time, e.g. because the interface is down, then you are stuck there forever.

What's the point of this? One interrupt on the wrong core is nothing we need to worry about.

Thanks,

tglx