Skip to contentSkip to conversation

arm: use irq_set_affinity with force=false when migrating irqs

PATCH New Sudeep Holla 4 participants 7 messages Sep 1, 2014 Arm (32-bit)

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 arm: use irq_set_affinity with force=false when migrating irqs

From: Sudeep Holla <[email protected]>

Commit 01f8fa4f01d8("genirq: Allow forcing cpu affinity of interrupts") enabled the forced irq_set_affinity which previously refused to route an interrupt to an offline cpu.

Commit ffde1de64012("irqchip: Gic: Support forced affinity setting") implements this force logic and disables the cpu online check for GIC interrupt controller.

When __cpu_disable calls migrate_irqs, it disables the current cpu in cpu_online_mask and uses forced irq_set_affinity to migrate the IRQs away from the cpu but passes affinity mask with the cpu being offlined also included in it.

If irq_set_affinity is called with force=true in a cpu hotplug path, the caller must ensure that the cpu being offlined is not present in the affinity mask or it may be selected as the target CPU, leading to the interrupt not being migrated.

This patch fixes the issue by calling irq_set_affinity with force=false so that cpu_online_mask is checked while setting the affinity in the cpu hotplug path.

Tested on TC2 hotpluging CPU0 in and out. Without this patch the system locks up as the IRQs are not migrated away from CPU0.

Cc Russell King <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Mark Rutland <[email protected]>
Cc <[email protected]> # 3.10.x
---
 arch/arm/kernel/irq.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Hi Russell,

If you or tglx has no objections to this patch, I will put it in your patch tracker.

Regards,
Sudeep

arch/arm/kernel/irq.c +1 -1
index 2c4257604513..5c4d38e32a51 100644
@@ -175,7 +175,7 @@ static bool migrate_one_irq(struct irq_desc *desc)
 	c = irq_data_get_irq_chip(d);
 	if (!c->irq_set_affinity)
 		pr_debug("IRQ%u: unable to set affinity\n", d->irq);
-	else if (c->irq_set_affinity(d, affinity, true) == IRQ_SET_MASK_OK && ret)
+	else if (c->irq_set_affinity(d, affinity, false) == IRQ_SET_MASK_OK && ret)
 		cpumask_copy(d->affinity, affinity);
 
 	return ret;
Signature · 1 line
1.8.3.2
#1 lore raw

PATCH 1/1 arm: use irq_set_affinity with force=false when migrating irqs

Quoted 32 lines · Sudeep Holla

From: Sudeep Holla <[email protected]>

Commit 01f8fa4f01d8("genirq: Allow forcing cpu affinity of interrupts") enabled the forced irq_set_affinity which previously refused to route an interrupt to an offline cpu.

Commit ffde1de64012("irqchip: Gic: Support forced affinity setting") implements this force logic and disables the cpu online check for GIC interrupt controller.

When __cpu_disable calls migrate_irqs, it disables the current cpu in cpu_online_mask and uses forced irq_set_affinity to migrate the IRQs away from the cpu but passes affinity mask with the cpu being offlined also included in it.

If irq_set_affinity is called with force=true in a cpu hotplug path, the caller must ensure that the cpu being offlined is not present in the affinity mask or it may be selected as the target CPU, leading to the interrupt not being migrated.

This patch fixes the issue by calling irq_set_affinity with force=false so that cpu_online_mask is checked while setting the affinity in the cpu hotplug path.

Tested on TC2 hotpluging CPU0 in and out. Without this patch the system locks up as the IRQs are not migrated away from CPU0.

Cc Russell King <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Mark Rutland <[email protected]>
Cc <[email protected]> # 3.10.x
Quoted 8 lines
---
 arch/arm/kernel/irq.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Hi Russell,

If you or tglx has no objections to this patch, I will put it in your patch tracker.

Post discussion, I have no objections - except to the above comment. Let me rewrite it in a programming language, and maybe you can spot what's wrong:

	if (russell_has_no_objection(patch) || tglx_has_no_objection(patch))
		submit_patch_to_tracker(patch);

Personally, I'd like to see tglx's ack on this first.

Signature · 2 lines
FTTC broadband for 0.8mile line: currently at 9.5Mbps down 400kbps up
according to speedtest.net.
#2 lore raw

PATCH 1/1 arm: use irq_set_affinity with force=false when migrating irqs

Quoted 42 lines · Russell King - ARM Linux
Quoted 32 lines · Sudeep Holla

From: Sudeep Holla <[email protected]>

Commit 01f8fa4f01d8("genirq: Allow forcing cpu affinity of interrupts") enabled the forced irq_set_affinity which previously refused to route an interrupt to an offline cpu.

Commit ffde1de64012("irqchip: Gic: Support forced affinity setting") implements this force logic and disables the cpu online check for GIC interrupt controller.

When __cpu_disable calls migrate_irqs, it disables the current cpu in cpu_online_mask and uses forced irq_set_affinity to migrate the IRQs away from the cpu but passes affinity mask with the cpu being offlined also included in it.

If irq_set_affinity is called with force=true in a cpu hotplug path, the caller must ensure that the cpu being offlined is not present in the affinity mask or it may be selected as the target CPU, leading to the interrupt not being migrated.

This patch fixes the issue by calling irq_set_affinity with force=false so that cpu_online_mask is checked while setting the affinity in the cpu hotplug path.

Tested on TC2 hotpluging CPU0 in and out. Without this patch the system locks up as the IRQs are not migrated away from CPU0.

Cc Russell King <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Mark Rutland <[email protected]>
Cc <[email protected]> # 3.10.x
Quoted 8 lines
---
  arch/arm/kernel/irq.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)

Hi Russell,

If you or tglx has no objections to this patch, I will put it in your patch tracker.

Quoted 8 lines

Post discussion, I have no objections - except to the above comment. Let me rewrite it in a programming language, and maybe you can spot what's wrong:

	if (russell_has_no_objection(patch) || tglx_has_no_objection(patch))
		submit_patch_to_tracker(patch);

Personally, I'd like to see tglx's ack on this first.

Understood, I will wait for ack from tglx.

Regards,
Sudeep

#3 lore raw

PATCH 1/1 arm: use irq_set_affinity with force=false when migrating irqs

In reply to Russell King - ARM Linux
Quoted 42 lines · Russell King - ARM Linux
Quoted 32 lines · Sudeep Holla

From: Sudeep Holla <[email protected]>

Commit 01f8fa4f01d8("genirq: Allow forcing cpu affinity of interrupts") enabled the forced irq_set_affinity which previously refused to route an interrupt to an offline cpu.

Commit ffde1de64012("irqchip: Gic: Support forced affinity setting") implements this force logic and disables the cpu online check for GIC interrupt controller.

When __cpu_disable calls migrate_irqs, it disables the current cpu in cpu_online_mask and uses forced irq_set_affinity to migrate the IRQs away from the cpu but passes affinity mask with the cpu being offlined also included in it.

If irq_set_affinity is called with force=true in a cpu hotplug path, the caller must ensure that the cpu being offlined is not present in the affinity mask or it may be selected as the target CPU, leading to the interrupt not being migrated.

This patch fixes the issue by calling irq_set_affinity with force=false so that cpu_online_mask is checked while setting the affinity in the cpu hotplug path.

Tested on TC2 hotpluging CPU0 in and out. Without this patch the system locks up as the IRQs are not migrated away from CPU0.

Cc Russell King <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Mark Rutland <[email protected]>
Cc <[email protected]> # 3.10.x
Quoted 8 lines
---
 arch/arm/kernel/irq.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Hi Russell,

If you or tglx has no objections to this patch, I will put it in your patch tracker.

Quoted 8 lines

Post discussion, I have no objections - except to the above comment. Let me rewrite it in a programming language, and maybe you can spot what's wrong:

	if (russell_has_no_objection(patch) || tglx_has_no_objection(patch))
		submit_patch_to_tracker(patch);

Personally, I'd like to see tglx's ack on this first.

I'm just not too happy about the changelog. It's confusing at best:

Call irq_set_affinity() with argument force = false

Reason: IRQ Core and GIC was changed to force set affinity.

If called from hotplug code, then the caller must ensure that the cpu being offlined is not present in the mask

Therefor call it with force = false

That does not make sense. You change something which preceded the forced affinity mechanism by 3+ years and completely miss to explain why it got there in the first place and never should have been.

What about somthing like the following:

Since commit 1dbfa187dad ("ARM: irq migration: force migration off CPU going down") the ARM interrupt migration code on cpu offline calls irqchip.irq_set_affinity() with the argument force=true. At the point of this change the argument had no effect because it was not used by any interrupt chip driver and there was no semantics defined.

This changed with commit 01f8fa4f01d8 ("genirq: Allow forcing cpu affinity of interrupts") which made the force argument useful to route interrupts to not yet online cpus without checking the target cpu against the cpu online mask. The following commit ffde1de64012 ("irqchip: gic: Support forced affinity setting") implemented this for the GIC interrupt chip.

As a consequence the ARM cpu offline irq migration fails if CPU0 is offlined, because CPU0 is still set in the affinity mask and the validataion against cpu online mask is skipped to the force argument being true. The following first_cpu(mask) selection always selects CPU0 as the target.

Solve the issue by calling irq_set_affinity() with force=false from the CPU offline irq migration code so the GIC driver validates the affinity mask against CPU online mask and therefor removes CPU0 from the possible target candidates.

#4 lore raw

PATCH 1/1 arm: use irq_set_affinity with force=false when migrating irqs

Quoted 47 lines · Thomas Gleixner
Quoted 46 lines · Russell King - ARM Linux
Quoted 32 lines · Sudeep Holla

From: Sudeep Holla <[email protected]>

Commit 01f8fa4f01d8("genirq: Allow forcing cpu affinity of interrupts") enabled the forced irq_set_affinity which previously refused to route an interrupt to an offline cpu.

Commit ffde1de64012("irqchip: Gic: Support forced affinity setting") implements this force logic and disables the cpu online check for GIC interrupt controller.

When __cpu_disable calls migrate_irqs, it disables the current cpu in cpu_online_mask and uses forced irq_set_affinity to migrate the IRQs away from the cpu but passes affinity mask with the cpu being offlined also included in it.

If irq_set_affinity is called with force=true in a cpu hotplug path, the caller must ensure that the cpu being offlined is not present in the affinity mask or it may be selected as the target CPU, leading to the interrupt not being migrated.

This patch fixes the issue by calling irq_set_affinity with force=false so that cpu_online_mask is checked while setting the affinity in the cpu hotplug path.

Tested on TC2 hotpluging CPU0 in and out. Without this patch the system locks up as the IRQs are not migrated away from CPU0.

Cc Russell King <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Mark Rutland <[email protected]>
Cc <[email protected]> # 3.10.x
Quoted 8 lines
---
  arch/arm/kernel/irq.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)

Hi Russell,

If you or tglx has no objections to this patch, I will put it in your patch tracker.

Post discussion, I have no objections - except to the above comment. Let me rewrite it in a programming language, and maybe you can spot what's wrong:

Quoted 7 lines
Quoted 4 lines
	if (russell_has_no_objection(patch) || tglx_has_no_objection(patch))
		submit_patch_to_tracker(patch);

Personally, I'd like to see tglx's ack on this first.

Thanks Thomas.

Quoted 7 lines

I'm just not too happy about the changelog. It's confusing at best:

Call irq_set_affinity() with argument force = false

Reason: IRQ Core and GIC was changed to force set affinity.

Quoted 9 lines

If called from hotplug code, then the caller must ensure that the cpu being offlined is not present in the mask

Therefor call it with force = false

That does not make sense. You change something which preceded the forced affinity mechanism by 3+ years and completely miss to explain why it got there in the first place and never should have been.

Agreed, I missed that. I will change the commit log as specified by you below.

Regards,
Sudeep

Quoted 27 lines

What about somthing like the following:

Since commit 1dbfa187dad ("ARM: irq migration: force migration off CPU going down") the ARM interrupt migration code on cpu offline calls irqchip.irq_set_affinity() with the argument force=true. At the point of this change the argument had no effect because it was not used by any interrupt chip driver and there was no semantics defined.

This changed with commit 01f8fa4f01d8 ("genirq: Allow forcing cpu affinity of interrupts") which made the force argument useful to route interrupts to not yet online cpus without checking the target cpu against the cpu online mask. The following commit ffde1de64012 ("irqchip: gic: Support forced affinity setting") implemented this for the GIC interrupt chip.

As a consequence the ARM cpu offline irq migration fails if CPU0 is offlined, because CPU0 is still set in the affinity mask and the validataion against cpu online mask is skipped to the force argument being true. The following first_cpu(mask) selection always selects CPU0 as the target.

Solve the issue by calling irq_set_affinity() with force=false from the CPU offline irq migration code so the GIC driver validates the affinity mask against CPU online mask and therefor removes CPU0 from the possible target candidates.

#5 lore raw

PATCH v2 1/1 arm: use irq_set_affinity with force=false when migrating irqs

In reply to Sudeep Holla

From: Sudeep Holla <[email protected]>

Since commit 1dbfa187dad ("ARM: irq migration: force migration off CPU going down") the ARM interrupt migration code on cpu offline calls irqchip.irq_set_affinity() with the argument force=true. At the point of this change the argument had no effect because it was not used by any interrupt chip driver and there was no semantics defined.

This changed with commit 01f8fa4f01d8 ("genirq: Allow forcing cpu affinity of interrupts") which made the force argument useful to route interrupts to not yet online cpus without checking the target cpu against the cpu online mask. The following commit ffde1de64012 ("irqchip: gic: Support forced affinity setting") implemented this for the GIC interrupt controller.

As a consequence the ARM cpu offline irq migration fails if CPU0 is offlined, because CPU0 is still set in the affinity mask and the validataion against cpu online mask is skipped to the force argument being true. The following first_cpu(mask) selection always selects CPU0 as the target.

Solve the issue by calling irq_set_affinity() with force=false from the CPU offline irq migration code so the GIC driver validates the affinity mask against CPU online mask and therefore removes CPU0 from the possible target candidates.

Tested on TC2 hotpluging CPU0 in and out. Without this patch the system locks up as the IRQs are not migrated away from CPU0.

---
 arch/arm/kernel/irq.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Change v1->v2:
- Updated the changelog to reflect the actual history behind this change as suggested by tglx and added his ack

arch/arm/kernel/irq.c +1 -1
index 2c4257604513..5c4d38e32a51 100644
@@ -175,7 +175,7 @@ static bool migrate_one_irq(struct irq_desc *desc)
 	c = irq_data_get_irq_chip(d);
 	if (!c->irq_set_affinity)
 		pr_debug("IRQ%u: unable to set affinity\n", d->irq);
-	else if (c->irq_set_affinity(d, affinity, true) == IRQ_SET_MASK_OK && ret)
+	else if (c->irq_set_affinity(d, affinity, false) == IRQ_SET_MASK_OK && ret)
 		cpumask_copy(d->affinity, affinity);
 
 	return ret;
Signature · 1 line
1.8.3.2
#6 lore raw

PATCH v2 1/1 arm: use irq_set_affinity with force=false when migrating irqs

Quoted 26 lines · Sudeep Holla

From: Sudeep Holla <[email protected]>

Since commit 1dbfa187dad ("ARM: irq migration: force migration off CPU going down") the ARM interrupt migration code on cpu offline calls irqchip.irq_set_affinity() with the argument force=true. At the point of this change the argument had no effect because it was not used by any interrupt chip driver and there was no semantics defined.

This changed with commit 01f8fa4f01d8 ("genirq: Allow forcing cpu affinity of interrupts") which made the force argument useful to route interrupts to not yet online cpus without checking the target cpu against the cpu online mask. The following commit ffde1de64012 ("irqchip: gic: Support forced affinity setting") implemented this for the GIC interrupt controller.

As a consequence the ARM cpu offline irq migration fails if CPU0 is offlined, because CPU0 is still set in the affinity mask and the validataion against cpu online mask is skipped to the force argument being true. The following first_cpu(mask) selection always selects CPU0 as the target.

Solve the issue by calling irq_set_affinity() with force=false from the CPU offline irq migration code so the GIC driver validates the affinity mask against CPU online mask and therefore removes CPU0 from the possible target candidates.

Quoted 8 lines

Tested on TC2 hotpluging CPU0 in and out. Without this patch the system locks up as the IRQs are not migrated away from CPU0.

It's nice to finally know what we should be doing here. :)

Would you be able to take a look at doing the same for arm64? The current solution (now in -stable) allows for hotplugging CPU0 but IIRC it would break the affinity mask unnecessarily.

Cheers,
Mark.

Quoted 24 lines
---
 arch/arm/kernel/irq.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Change v1->v2:
- Updated the changelog to reflect the actual history behind this change as suggested by tglx and added his ack

arch/arm/kernel/irq.c +1 -1
index 2c4257604513..5c4d38e32a51 100644
@@ -175,7 +175,7 @@ static bool migrate_one_irq(struct irq_desc *desc)
 	c = irq_data_get_irq_chip(d);
 	if (!c->irq_set_affinity)
 		pr_debug("IRQ%u: unable to set affinity\n", d->irq);
-	else if (c->irq_set_affinity(d, affinity, true) == IRQ_SET_MASK_OK && ret)
+	else if (c->irq_set_affinity(d, affinity, false) == IRQ_SET_MASK_OK && ret)
 		cpumask_copy(d->affinity, affinity);
 
 	return ret;
Signature · 1 line
1.8.3.2