arm64: do not force irq affinity setting

PATCH New Prashant Gaikwad 3 participants 7 messages Jun 26, 2014 arm64

lore

Share & tools

lore

Prashant Gaikwad

author

2014-06-26 06:49 UTC #0 lore raw

PATCH 1/1 arm64: do not force irq affinity setting

Unconditional copying cpu_online_mask to affinity may result in migrating affinity to wrong CPU.

For example, IRQ 5 affinity mask contains CPU 4-7, it was affined to CPU4 and CPU 0-7 are online. Now if we hot-unplug CPU4 then with current implementation affinity mask will contain CPU 0-3,5-7 and IRQ 5 will be affined to CPU0.

Instead copy cpu_online_mask to affinity only if no online CPU is present in affinity mask and do not force affinity seeting which would do the CPU online check.

Signed-off-by Prashant Gaikwad <[email protected]>

---
 arch/arm64/kernel/irq.c |   12 ++++--------
 1 files changed, 4 insertions(+), 8 deletions(-)

arch/arm64/kernel/irq.c +4 -8

index 0f08dfd..dfa6e3e 100644
@@ -97,19 +97,15 @@ static bool migrate_one_irq(struct irq_desc *desc)
 	if (irqd_is_per_cpu(d) || !cpumask_test_cpu(smp_processor_id(), affinity))
 		return false;
 
-	if (cpumask_any_and(affinity, cpu_online_mask) >= nr_cpu_ids)
+	if (cpumask_any_and(affinity, cpu_online_mask) >= nr_cpu_ids) {
+		affinity = cpu_online_mask;
 		ret = true;
+	}
 
-	/*
-	 * when using forced irq_set_affinity we must ensure that the cpu
-	 * being offlined is not present in the affinity mask, it may be
-	 * selected as the target CPU otherwise
-	 */
-	affinity = cpu_online_mask;
 	c = irq_data_get_irq_chip(d);
 	if (!c->irq_set_affinity)
 		pr_debug("IRQ%u: unable to set affinity\n", d->irq);
-	else if (c->irq_set_affinity(d, affinity, true) == IRQ_SET_MASK_OK && ret)
+	else if (c->irq_set_affinity(d, affinity, false) == IRQ_SET_MASK_OK && ret)
 		cpumask_copy(d->affinity, affinity);
 
 	return ret;

Signature · 1 line

1.7.4.1

Will Deacon

2014-06-26 10:20 UTC #1 lore raw

PATCH 1/1 arm64: do not force irq affinity setting

Hello,

Quoted 2 lines · Prashant Gaikwad

Unconditional copying cpu_online_mask to affinity may result in migrating affinity to wrong CPU.

We have a bug, but I don't follow your reasoning.

Quoted 1 line

For example, IRQ 5 affinity mask contains CPU 4-7,

Ok, so d->affinity is 0xf0...

Quoted 1 line

it was affined to CPU4 and CPU 0-7 are online.

...and cpu_online_mask is 0xff.

Quoted 3 lines

Now if we hot-unplug CPU4 then with current implementation affinity mask will contain CPU 0-3,5-7 and IRQ 5 will be affined to CPU0.

cpumask_any_and(affinity, cpu_online_mask) will give return < nr_cpu_ids since there is an intersection of 0xf0. That means ret is false.

The bug is that we then do affinity = cpu_online_mask; unconditionally, but we *won't* do the cpumask_copy, since ret is false.

You can fix this by simply bringing the arm64 code into line with the arm code, which begs the question as to why this has to exist in the arch/ backend@all!

Will

Prashant Gaikwad

author

2014-06-26 12:00 UTC #2 lore raw

PATCH 1/1 arm64: do not force irq affinity setting

Quoted 20 lines · Will Deacon

Hello,
Quoted 2 lines · Prashant Gaikwad
Unconditional copying cpu_online_mask to affinity may result in migrating affinity to wrong CPU.
We have a bug, but I don't follow your reasoning.
Quoted 1 line
For example, IRQ 5 affinity mask contains CPU 4-7,
Ok, so d->affinity is 0xf0...
Quoted 1 line
it was affined to CPU4 and CPU 0-7 are online.
...and cpu_online_mask is 0xff.
Quoted 3 lines
Now if we hot-unplug CPU4 then with current implementation affinity mask will contain CPU 0-3,5-7 and IRQ 5 will be affined to CPU0.

Quoted 6 lines

cpumask_any_and(affinity, cpu_online_mask) will give return < nr_cpu_ids since there is an intersection of 0xf0. That means ret is false.
The bug is that we then do affinity = cpu_online_mask; unconditionally, but we *won't* do the cpumask_copy, since ret is false.

We do not copy but the affinity mask passed to irq_set_affinity function is nothing but cpu_online_mask. So in GIC it will set affinity to CPU0.

Quoted 3 lines

You can fix this by simply bringing the arm64 code into line with the arm code, which begs the question as to why this has to exist in the arch/ backend at all!

Where can we move this code?

Quoted 2 lines

Will

Will Deacon

2014-06-26 13:11 UTC #3 lore raw

PATCH 1/1 arm64: do not force irq affinity setting

Quoted 22 lines · Prashant Gaikwad

Quoted 21 lines · Will Deacon
Quoted 2 lines · Prashant Gaikwad
Unconditional copying cpu_online_mask to affinity may result in migrating affinity to wrong CPU.
We have a bug, but I don't follow your reasoning.
Quoted 1 line
For example, IRQ 5 affinity mask contains CPU 4-7,
Ok, so d->affinity is 0xf0...
Quoted 1 line
it was affined to CPU4 and CPU 0-7 are online.
...and cpu_online_mask is 0xff.
Quoted 3 lines
Now if we hot-unplug CPU4 then with current implementation affinity mask will contain CPU 0-3,5-7 and IRQ 5 will be affined to CPU0.
cpumask_any_and(affinity, cpu_online_mask) will give return < nr_cpu_ids since there is an intersection of 0xf0. That means ret is false.

Quoted 6 lines

Quoted 3 lines
The bug is that we then do affinity = cpu_online_mask; unconditionally, but we *won't* do the cpumask_copy, since ret is false.
We do not copy but the affinity mask passed to irq_set_affinity function is nothing but cpu_online_mask. So in GIC it will set affinity to CPU0.

Exactly, but your proposed patch changed more than that.

Quoted 5 lines

Quoted 3 lines
You can fix this by simply bringing the arm64 code into line with the arm code, which begs the question as to why this has to exist in the arch/ backend at all!
Where can we move this code?

kernel/irq/migration.c?

Will

Prashant Gaikwad

author

2014-06-26 13:40 UTC #4 lore raw

PATCH 1/1 arm64: do not force irq affinity setting

Quoted 23 lines · Will Deacon

Quoted 22 lines · Prashant Gaikwad
Quoted 21 lines · Will Deacon
Quoted 2 lines · Prashant Gaikwad
Unconditional copying cpu_online_mask to affinity may result in migrating affinity to wrong CPU.
We have a bug, but I don't follow your reasoning.
Quoted 1 line
For example, IRQ 5 affinity mask contains CPU 4-7,
Ok, so d->affinity is 0xf0...
Quoted 1 line
it was affined to CPU4 and CPU 0-7 are online.
...and cpu_online_mask is 0xff.
Quoted 3 lines
Now if we hot-unplug CPU4 then with current implementation affinity mask will contain CPU 0-3,5-7 and IRQ 5 will be affined to CPU0.
cpumask_any_and(affinity, cpu_online_mask) will give return < nr_cpu_ids since there is an intersection of 0xf0. That means ret is false.

Quoted 9 lines

Quoted 6 lines
Quoted 3 lines
The bug is that we then do affinity = cpu_online_mask; unconditionally, but we *won't* do the cpumask_copy, since ret is false.
We do not copy but the affinity mask passed to irq_set_affinity function is nothing but cpu_online_mask. So in GIC it will set affinity to CPU0.
Exactly, but your proposed patch changed more than that.

I am changing the force flag to false. That is because after I fix this behavior we have another bug where the IRQ affinity is set to offline CPU.

When cpumask_any_and(affinity, cpu_online_mask) return < nr_cpu_ids we pass the affinity mask as it is which contains the offline CPU too and if force flag is true then GIC driver skips online CPU check. If CPU0 is going down then the affinity mask will have CPU0 and GIC driver will keep the affinity to CPU0.

Changing force flag to false ensures that GIC driver checks for online CPU.

Quoted 9 lines

Quoted 5 lines
Quoted 3 lines
You can fix this by simply bringing the arm64 code into line with the arm code, which begs the question as to why this has to exist in the arch/ backend at all!
Where can we move this code?
kernel/irq/migration.c?
Will

Sudeep Holla

2014-06-26 14:04 UTC #5 lore raw

PATCH 1/1 arm64: do not force irq affinity setting

Hi,

Quoted 31 lines · Prashant Gaikwad

Quoted 30 lines · Will Deacon
Quoted 28 lines · Prashant Gaikwad
Quoted 18 lines · Will Deacon
Quoted 2 lines · Prashant Gaikwad
Unconditional copying cpu_online_mask to affinity may result in migrating affinity to wrong CPU.
We have a bug, but I don't follow your reasoning.
Quoted 1 line
For example, IRQ 5 affinity mask contains CPU 4-7,
Ok, so d->affinity is 0xf0...
Quoted 1 line
it was affined to CPU4 and CPU 0-7 are online.
...and cpu_online_mask is 0xff.
Quoted 3 lines
Now if we hot-unplug CPU4 then with current implementation affinity mask will contain CPU 0-3,5-7 and IRQ 5 will be affined to CPU0.
Quoted 6 lines
cpumask_any_and(affinity, cpu_online_mask) will give return < nr_cpu_ids since there is an intersection of 0xf0. That means ret is false.
The bug is that we then do affinity = cpu_online_mask; unconditionally, but we *won't* do the cpumask_copy, since ret is false.
We do not copy but the affinity mask passed to irq_set_affinity function is nothing but cpu_online_mask. So in GIC it will set affinity to CPU0.

Quoted 7 lines

Quoted 2 lines
Exactly, but your proposed patch changed more than that.
I am changing the force flag to false. That is because after I fix this behavior we have another bug where the IRQ affinity is set to offline CPU.

That's correct, it's the original issue I saw and fixed incorrectly which triggered the bug you have now.

The main reason to retain the force flag as true is that the implementation is irqchip specific. GIC implements the way you explained but what if some other irqchip implementation has something different.

I believe that's the reason why Russell wants to get feedback from tglx.

Regards,
Sudeep

Sudeep Holla

2014-06-26 13:45 UTC #6 lore raw

PATCH 1/1 arm64: do not force irq affinity setting

In reply to Will Deacon

Hi Will,

Quoted 23 lines · Will Deacon

Hello,
Quoted 2 lines · Prashant Gaikwad
Unconditional copying cpu_online_mask to affinity may result in migrating affinity to wrong CPU.
We have a bug, but I don't follow your reasoning.
Quoted 1 line
For example, IRQ 5 affinity mask contains CPU 4-7,
Ok, so d->affinity is 0xf0...
Quoted 1 line
it was affined to CPU4 and CPU 0-7 are online.
...and cpu_online_mask is 0xff.
Quoted 3 lines
Now if we hot-unplug CPU4 then with current implementation affinity mask will contain CPU 0-3,5-7 and IRQ 5 will be affined to CPU0.
cpumask_any_and(affinity, cpu_online_mask) will give return < nr_cpu_ids since there is an intersection of 0xf0. That means ret is false.

Quoted 7 lines

The bug is that we then do affinity = cpu_online_mask; unconditionally, but we *won't* do the cpumask_copy, since ret is false.
You can fix this by simply bringing the arm64 code into line with the arm code, which begs the question as to why this has to exist in the arch/ backend at all!

The unconditional assignment was added by me to fix CPU0 hotplug issue explained in commit 601c942176d8 which is wrong and evident from the above usecase. It was added to retain the forced irq_set_affinity. The difference between arm and arm64 is because the arm doesn't have the patch [1]

We can move to irq_set_affinity without force option as this patch does. I had mentioned similar solution[2], but Russell wants to get feedback from tglx[3]

And yes I see similar implementations for many architectures, definitely can be unified.

Regards,
Sudeep

[1] http://lists.infradead.org/pipermail/linux-arm-kernel/2014-May/254838.html
[2] http://lists.infradead.org/pipermail/linux-arm-kernel/2014-May/259255.html
[3] http://www.spinics.net/lists/arm-kernel/msg340279.html