Skip to contentSkip to conversation

x86/x86_64: fix CPU offlining triggered inactive device IRQ interrruption

PATCH New Gary Hade 1 participant 1 message May 14, 2009 x86

lore
Share & tools
lore

Gary Hade

author
#0 lore raw

PATCH v2 1/1 x86/x86_64: fix CPU offlining triggered inactive device IRQ interrruption

Impact: Eliminates a race that can leave the system in an
unusable state

During rapid offlining of multiple CPUs there is a chance that an IRQ affinity move destination CPU will be offlined before the IRQ affinity move initiated during the offlining of a previous CPU completes. This can happen when the device is not very active and thus fails to generate the IRQ that is needed to complete the IRQ affinity move before the move destination CPU is offlined. When this happens there is an -EBUSY return from __assign_irq_vector() during the offlining of the IRQ move destination CPU which prevents initiation of a new IRQ affinity move operation to an online CPU. This leaves the IRQ affinity set to an offlined CPU.

I have been able to reproduce the problem on some of our systems using the following script. When the system is idle the problem often reproduces during the first CPU offlining sequence.

#!/bin/sh

SYS_CPU_DIR=/sys/devices/system/cpu
VICTIM_IRQ=25
IRQ_MASK=f0

iteration=0
while true; do
echo $iteration
echo $IRQ_MASK > /proc/irq/$VICTIM_IRQ/smp_affinity for cpudir in $SYS_CPU_DIR/cpu[1-9] $SYS_CPU_DIR/cpu??; do
echo 0 > $cpudir/online
done
for cpudir in $SYS_CPU_DIR/cpu[1-9] $SYS_CPU_DIR/cpu??; do
echo 1 > $cpudir/online
done
iteration=expr $iteration + 1
done

The proposed fix takes advantage of the fact that when all CPUs in the old domain are offline there is nothing to be done by send_cleanup_vector() during the affinity move completion. So, we simply avoid setting cfg->move_in_progress preventing the above mentioned -EBUSY return from __assign_irq_vector(). This allows initiation of a new IRQ affinity move to a CPU that is not going offline.

Successfully tested with Ingo's linux-2.6-tip (32 and 64-bit builds) on the IBM x460, x3550 M2, x3850, and x3950 M2.

v2: modified to integrate with Yinghai Lu's
"x86/irq: remove leftover code from NUMA_MIGRATE_IRQ_DESC" patch which modified intersecting lines. Only comment changes were affected. The actual change to the code is the same.

---
 arch/x86/kernel/apic/io_apic.c |    5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
arch/x86/kernel/apic/io_apic.c +4 -1
Index: linux-2.6-tip/arch/x86/kernel/apic/io_apic.c
===================================================================
@@ -1218,8 +1218,11 @@ next:
 		current_vector = vector;
 		current_offset = offset;
 		if (old_vector) {
-			cfg->move_in_progress = 1;
 			cpumask_copy(cfg->old_domain, cfg->domain);
+			if (cpumask_intersects(cfg->old_domain,
+					       cpu_online_mask)) {
+				cfg->move_in_progress = 1;
+			}
 		}
 		for_each_cpu_and(new_cpu, tmp_mask, cpu_online_mask)
 			per_cpu(vector_irq, new_cpu)[vector] = irq;