x86/x86_64: fix CPU offlining triggered inactive device IRQ interrruption
PATCH 2/3 x86/x86_64: fix CPU offlining triggered inactive device IRQ interrruption
Impact: Eliminates a race that can leave the system in an
unusable state
During rapid offlining of multiple CPUs there is a chance that an IRQ affinity move destination CPU will be offlined before the IRQ affinity move initiated during the offlining of a previous CPU completes. This can happen when the device is not very active and thus fails to generate the IRQ that is needed to complete the IRQ affinity move before the move destination CPU is offlined. When this happens there is an -EBUSY return from __assign_irq_vector() during the offlining of the IRQ move destination CPU which prevents initiation of a new IRQ affinity move operation to an online CPU. This leaves the IRQ affinity set to an offlined CPU.
I have been able to reproduce the problem on some of our systems using the following script. When the system is idle the problem often reproduces during the first CPU offlining sequence.
#!/bin/sh
SYS_CPU_DIR=/sys/devices/system/cpu
VICTIM_IRQ=25
IRQ_MASK=f0
iteration=0
while true; do
echo $iteration
echo $IRQ_MASK > /proc/irq/$VICTIM_IRQ/smp_affinity for cpudir in $SYS_CPU_DIR/cpu[1-9] $SYS_CPU_DIR/cpu??; do
echo 0 > $cpudir/online
done
for cpudir in $SYS_CPU_DIR/cpu[1-9] $SYS_CPU_DIR/cpu??; do
echo 1 > $cpudir/online
done
iteration=expr $iteration + 1
done
The proposed fix takes advantage of the fact that when all CPUs in the old domain are offline there is nothing to be done by send_cleanup_vector() during the affinity move completion. So, we simply avoid setting cfg->move_in_progress preventing the above mentioned -EBUSY return from __assign_irq_vector(). This allows initiation of a new IRQ affinity move to a CPU that is not going offline.
--- arch/x86/kernel/apic/io_apic.c | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-)
arch/x86/kernel/apic/io_apic.c +8 -3
@@ -363,7 +363,8 @@ set_extra_move_desc(struct irq_desc *desstruct irq_cfg *cfg = desc->chip_data;if (!cfg->move_in_progress) { -/* it means that domain is not changed */ +/* it means that domain has not changed or all CPUs +* in old domain are offline */ if (!cpumask_intersects (desc->affinity, mask)) cfg->move_desc_pending =1 ; }
@@ -1262,8 +1263,11 @@ next: current_vector = vector; current_offset = offset;if (old_vector) { - cfg->move_in_progress =1 ;cpumask_copy (cfg->old_domain, cfg->domain); +if (cpumask_intersects (cfg->old_domain, + cpu_online_mask)) { + cfg->move_in_progress =1 ; + } }for_each_cpu_and (new_cpu, tmp_mask, cpu_online_mask)per_cpu (vector_irq, new_cpu)[vector] = irq;
@@ -2492,7 +2496,8 @@ static void irq_complete_move(struct irqif (likely (!cfg->move_desc_pending))return ; -/* domain has not changed, but affinity did */ +/* domain has not changed or all CPUs in old domain +* are offline, but affinity changed */ me =smp_processor_id ();if (cpumask_test_cpu (me, desc->affinity)) { *descp = desc =move_irq_desc (desc, me);
Quoted 81 lines · Gary Hade
Impact: Eliminates a race that can leave the system in an unusable state
During rapid offlining of multiple CPUs there is a chance that an IRQ affinity move destination CPU will be offlined before the IRQ affinity move initiated during the offlining of a previous CPU completes. This can happen when the device is not very active and thus fails to generate the IRQ that is needed to complete the IRQ affinity move before the move destination CPU is offlined. When this happens there is an
-EBUSYreturn from__assign_irq_vector()during the offlining of the IRQ move destination CPU which prevents initiation of a new IRQ affinity move operation to an online CPU. This leaves the IRQ affinity set to an offlined CPU.I have been able to reproduce the problem on some of our systems using the following script. When the system is idle the problem often reproduces during the first CPU offlining sequence.
#!/bin/sh
SYS_CPU_DIR=/sys/devices/system/cpu
VICTIM_IRQ=25
IRQ_MASK=f0iteration=0
while true; do
echo $iteration
echo $IRQ_MASK >/proc/irq/$VICTIM_IRQ/smp_affinityfor cpudir in $SYS_CPU_DIR/cpu[1-9] $SYS_CPU_DIR/cpu??; do echo 0 > $cpudir/online
done
for cpudir in $SYS_CPU_DIR/cpu[1-9] $SYS_CPU_DIR/cpu??; do echo 1 > $cpudir/online
done
iteration=expr $iteration + 1
doneThe proposed fix takes advantage of the fact that when all CPUs in the old domain are offline there is nothing to be done by
send_cleanup_vector()during the affinity move completion. So, we simply avoid settingcfg->move_in_progresspreventing the above mentioned-EBUSYreturn from__assign_irq_vector(). This allows initiation of a new IRQ affinity move to a CPU that is not going offline.Signed-off-by Gary Hade <[email protected]>---
arch/x86/kernel/apic/io_apic.c | 11 ++++++++---1 file changed, 8 insertions(+), 3 deletions(-)Index:
linux-2.6.30-rc1/arch/x86/kernel/apic/io_apic.c
===================================================================arch/x86/kernel/apic/io_apic.c 2009-04-08 09:23:16.000000000 -0700 renamed from arch/x86/kernel/apic/io_apic.c 2009-04-08 09:23:00.000000000 -0700 +6
-2@@ -363,7 +363,8 @@ set_extra_move_desc(struct irq_desc *des struct irq_cfg *cfg = desc->chip_data; if (!cfg->move_in_progress) { - /* it means that domain is not changed */ + /* it means that domain has not changed or all CPUs + * in old domain are offline */ if (!cpumask_intersects(desc->affinity, mask)) cfg->move_desc_pending = 1; }@@ -1262,8 +1263,11 @@ next: current_vector = vector; current_offset = offset; if (old_vector) { - cfg->move_in_progress = 1; cpumask_copy(cfg->old_domain, cfg->domain); + if (cpumask_intersects(cfg->old_domain, + cpu_online_mask)) { + cfg->move_in_progress = 1; + } } for_each_cpu_and(new_cpu, tmp_mask, cpu_online_mask) per_cpu(vector_irq, new_cpu)[vector] = irq;
arch/x86/kernel/apic/io_apic.c 2009-04-08 09:23:16.000000000 -0700 static void irq_complete_move(struct irq
Quoted 10 lines
arch/x86/kernel/apic/io_apic.c 2009-04-08 09:23:16.000000000 -0700 renamed from arch/x86/kernel/apic/io_apic.c 2009-04-08 09:23:00.000000000 -0700 +2
-2@@ -2492,7 +2496,8 @@ static void irq_complete_move(struct irq if (likely(!cfg->move_desc_pending)) return; - /* domain has not changed, but affinity did */ + /* domain has not changed or all CPUs in old domain + * are offline, but affinity changed */ me = smp_processor_id(); if (cpumask_test_cpu(me, desc->affinity)) { *descp = desc = move_irq_desc(desc, me); --
so you mean during __assign_irq_vector(), cpu_online_mask get updated? with your patch, how about that it just happen right after you check that second time.
it seems we are missing some lock_vector_lock() on the remove cpu from online mask.
YH
Quoted 85 lines · Yinghai Lu
Quoted 84 lines · Gary Hade
Impact: Eliminates a race that can leave the system in an unusable state
During rapid offlining of multiple CPUs there is a chance that an IRQ affinity move destination CPU will be offlined before the IRQ affinity move initiated during the offlining of a previous CPU completes. This can happen when the device is not very active and thus fails to generate the IRQ that is needed to complete the IRQ affinity move before the move destination CPU is offlined. When this happens there is an
-EBUSYreturn from__assign_irq_vector()during the offlining of the IRQ move destination CPU which prevents initiation of a new IRQ affinity move operation to an online CPU. This leaves the IRQ affinity set to an offlined CPU.I have been able to reproduce the problem on some of our systems using the following script. When the system is idle the problem often reproduces during the first CPU offlining sequence.
#!/bin/sh
SYS_CPU_DIR=/sys/devices/system/cpu
VICTIM_IRQ=25
IRQ_MASK=f0iteration=0
while true; do
echo $iteration
echo $IRQ_MASK >/proc/irq/$VICTIM_IRQ/smp_affinityfor cpudir in $SYS_CPU_DIR/cpu[1-9] $SYS_CPU_DIR/cpu??; do echo 0 > $cpudir/online
done
for cpudir in $SYS_CPU_DIR/cpu[1-9] $SYS_CPU_DIR/cpu??; do echo 1 > $cpudir/online
done
iteration=expr $iteration + 1
doneThe proposed fix takes advantage of the fact that when all CPUs in the old domain are offline there is nothing to be done by
send_cleanup_vector()during the affinity move completion. So, we simply avoid settingcfg->move_in_progresspreventing the above mentioned-EBUSYreturn from__assign_irq_vector(). This allows initiation of a new IRQ affinity move to a CPU that is not going offline.Signed-off-by Gary Hade <[email protected]>---
arch/x86/kernel/apic/io_apic.c | 11 ++++++++---1 file changed, 8 insertions(+), 3 deletions(-)Index:
linux-2.6.30-rc1/arch/x86/kernel/apic/io_apic.c
===================================================================arch/x86/kernel/apic/io_apic.c 2009-04-08 09:23:16.000000000 -0700 renamed from arch/x86/kernel/apic/io_apic.c 2009-04-08 09:23:00.000000000 -0700 +6
-2@@ -363,7 +363,8 @@ set_extra_move_desc(struct irq_desc *des struct irq_cfg *cfg = desc->chip_data; if (!cfg->move_in_progress) { - /* it means that domain is not changed */ + /* it means that domain has not changed or all CPUs + * in old domain are offline */ if (!cpumask_intersects(desc->affinity, mask)) cfg->move_desc_pending = 1; }@@ -1262,8 +1263,11 @@ next: current_vector = vector; current_offset = offset; if (old_vector) { - cfg->move_in_progress = 1; cpumask_copy(cfg->old_domain, cfg->domain); + if (cpumask_intersects(cfg->old_domain, + cpu_online_mask)) { + cfg->move_in_progress = 1; + } } for_each_cpu_and(new_cpu, tmp_mask, cpu_online_mask) per_cpu(vector_irq, new_cpu)[vector] = irq;@@ -2492,7 +2496,8 @@ static void irq_complete_move(struct irq if (likely(!cfg->move_desc_pending)) return;
Quoted 9 lines
arch/x86/kernel/apic/io_apic.c 2009-04-08 09:23:16.000000000 -0700 static void irq_complete_move(struct irq
Quoted 7 lines
arch/x86/kernel/apic/io_apic.c 2009-04-08 09:23:16.000000000 -0700 renamed from arch/x86/kernel/apic/io_apic.c 2009-04-08 09:23:00.000000000 -0700 +2
-2- /* domain has not changed, but affinity did */ + /* domain has not changed or all CPUs in old domain + * are offline, but affinity changed */ me = smp_processor_id(); if (cpumask_test_cpu(me, desc->affinity)) { *descp = desc = move_irq_desc(desc, me); --so you mean during
__assign_irq_vector(),cpu_online_maskget updated?
No, the CPU being offlined is removed from cpu_online_mask earlier via a call to remove_cpu_from_maps() from cpu_disable_common(). This happens just before fixup_irqs() is called.
Quoted 5 lines
with your patch, how about that it just happen right after you check that second time.
it seems we are missing some
lock_vector_lock()on the remove cpu from online mask.
The remove_cpu_from_maps() call in cpu_disable_common() is vector lock protected:
void cpu_disable_common(void)
{
< snip >
/* It's now safe to remove this processor from the online map */
lock_vector_lock();
remove_cpu_from_maps(cpu);
unlock_vector_lock();
fixup_irqs();
}Is this what you meant?
Gary
Signature · 6 lines
Gary Hade System x Enablement IBM Linux Technology Center 503-578-4503 IBM T/L: 775-4503 [email protected] http://www.ibm.com/linux/ltc
Quoted 107 lines · Gary Hade
Quoted 85 lines · Yinghai Lu
Quoted 84 lines · Gary Hade
Impact: Eliminates a race that can leave the system in an unusable state
During rapid offlining of multiple CPUs there is a chance that an IRQ affinity move destination CPU will be offlined before the IRQ affinity move initiated during the offlining of a previous CPU completes. This can happen when the device is not very active and thus fails to generate the IRQ that is needed to complete the IRQ affinity move before the move destination CPU is offlined. When this happens there is an
-EBUSYreturn from__assign_irq_vector()during the offlining of the IRQ move destination CPU which prevents initiation of a new IRQ affinity move operation to an online CPU. This leaves the IRQ affinity set to an offlined CPU.I have been able to reproduce the problem on some of our systems using the following script. When the system is idle the problem often reproduces during the first CPU offlining sequence.
#!/bin/sh
SYS_CPU_DIR=/sys/devices/system/cpu
VICTIM_IRQ=25
IRQ_MASK=f0iteration=0
while true; do
echo $iteration
echo $IRQ_MASK >/proc/irq/$VICTIM_IRQ/smp_affinityfor cpudir in $SYS_CPU_DIR/cpu[1-9] $SYS_CPU_DIR/cpu??; do echo 0 > $cpudir/online
done
for cpudir in $SYS_CPU_DIR/cpu[1-9] $SYS_CPU_DIR/cpu??; do echo 1 > $cpudir/online
done
iteration=expr $iteration + 1
doneThe proposed fix takes advantage of the fact that when all CPUs in the old domain are offline there is nothing to be done by
send_cleanup_vector()during the affinity move completion. So, we simply avoid settingcfg->move_in_progresspreventing the above mentioned-EBUSYreturn from__assign_irq_vector(). This allows initiation of a new IRQ affinity move to a CPU that is not going offline.Signed-off-by Gary Hade <[email protected]>---
arch/x86/kernel/apic/io_apic.c | 11 ++++++++---1 file changed, 8 insertions(+), 3 deletions(-)Index:
linux-2.6.30-rc1/arch/x86/kernel/apic/io_apic.c
===================================================================arch/x86/kernel/apic/io_apic.c 2009-04-08 09:23:16.000000000 -0700 renamed from arch/x86/kernel/apic/io_apic.c 2009-04-08 09:23:00.000000000 -0700 +6
-2@@ -363,7 +363,8 @@ set_extra_move_desc(struct irq_desc *des struct irq_cfg *cfg = desc->chip_data; if (!cfg->move_in_progress) { - /* it means that domain is not changed */ + /* it means that domain has not changed or all CPUs + * in old domain are offline */ if (!cpumask_intersects(desc->affinity, mask)) cfg->move_desc_pending = 1; }@@ -1262,8 +1263,11 @@ next: current_vector = vector; current_offset = offset; if (old_vector) { - cfg->move_in_progress = 1; cpumask_copy(cfg->old_domain, cfg->domain); + if (cpumask_intersects(cfg->old_domain, + cpu_online_mask)) { + cfg->move_in_progress = 1; + } } for_each_cpu_and(new_cpu, tmp_mask, cpu_online_mask) per_cpu(vector_irq, new_cpu)[vector] = irq;@@ -2492,7 +2496,8 @@ static void irq_complete_move(struct irq if (likely(!cfg->move_desc_pending)) return;Quoted 9 lines
arch/x86/kernel/apic/io_apic.c 2009-04-08 09:23:16.000000000 -0700 static void irq_complete_move(struct irq
Quoted 7 lines
arch/x86/kernel/apic/io_apic.c 2009-04-08 09:23:16.000000000 -0700 renamed from arch/x86/kernel/apic/io_apic.c 2009-04-08 09:23:00.000000000 -0700 +2
-2- /* domain has not changed, but affinity did */ + /* domain has not changed or all CPUs in old domain + * are offline, but affinity changed */ me = smp_processor_id(); if (cpumask_test_cpu(me, desc->affinity)) { *descp = desc = move_irq_desc(desc, me); --so you mean during
__assign_irq_vector(),cpu_online_maskget updated?No, the CPU being offlined is removed from
cpu_online_maskearlier via a call toremove_cpu_from_maps()fromcpu_disable_common(). This happens just beforefixup_irqs()is called.Quoted 5 lines
with your patch, how about that it just happen right after you check that second time.
it seems we are missing some
lock_vector_lock()on the remove cpu from online mask.
Quoted 11 lines
The
remove_cpu_from_maps()call incpu_disable_common()is vector lock protected:
voidcpu_disable_common(void)
{
< snip >
/* It's now safe to remove this processor from the online map */lock_vector_lock();
remove_cpu_from_maps(cpu);
unlock_vector_lock();
fixup_irqs();
}
__assign_irq_vector always has vector_lock locked...
so cpu_online_mask will not changed during, why do you need to check that again in __assign_irq_vector ?
YH
Quoted 121 lines · Yinghai Lu
Quoted 107 lines · Gary Hade
Quoted 85 lines · Yinghai Lu
Quoted 84 lines · Gary Hade
Impact: Eliminates a race that can leave the system in an unusable state
During rapid offlining of multiple CPUs there is a chance that an IRQ affinity move destination CPU will be offlined before the IRQ affinity move initiated during the offlining of a previous CPU completes. This can happen when the device is not very active and thus fails to generate the IRQ that is needed to complete the IRQ affinity move before the move destination CPU is offlined. When this happens there is an
-EBUSYreturn from__assign_irq_vector()during the offlining of the IRQ move destination CPU which prevents initiation of a new IRQ affinity move operation to an online CPU. This leaves the IRQ affinity set to an offlined CPU.I have been able to reproduce the problem on some of our systems using the following script. When the system is idle the problem often reproduces during the first CPU offlining sequence.
#!/bin/sh
SYS_CPU_DIR=/sys/devices/system/cpu
VICTIM_IRQ=25
IRQ_MASK=f0iteration=0
while true; do
echo $iteration
echo $IRQ_MASK >/proc/irq/$VICTIM_IRQ/smp_affinityfor cpudir in $SYS_CPU_DIR/cpu[1-9] $SYS_CPU_DIR/cpu??; do echo 0 > $cpudir/online
done
for cpudir in $SYS_CPU_DIR/cpu[1-9] $SYS_CPU_DIR/cpu??; do echo 1 > $cpudir/online
done
iteration=expr $iteration + 1
doneThe proposed fix takes advantage of the fact that when all CPUs in the old domain are offline there is nothing to be done by
send_cleanup_vector()during the affinity move completion. So, we simply avoid settingcfg->move_in_progresspreventing the above mentioned-EBUSYreturn from__assign_irq_vector(). This allows initiation of a new IRQ affinity move to a CPU that is not going offline.Signed-off-by Gary Hade <[email protected]>---
arch/x86/kernel/apic/io_apic.c | 11 ++++++++---1 file changed, 8 insertions(+), 3 deletions(-)Index:
linux-2.6.30-rc1/arch/x86/kernel/apic/io_apic.c
===================================================================arch/x86/kernel/apic/io_apic.c 2009-04-08 09:23:16.000000000 -0700 renamed from arch/x86/kernel/apic/io_apic.c 2009-04-08 09:23:00.000000000 -0700 +6
-2@@ -363,7 +363,8 @@ set_extra_move_desc(struct irq_desc *des struct irq_cfg *cfg = desc->chip_data; if (!cfg->move_in_progress) { - /* it means that domain is not changed */ + /* it means that domain has not changed or all CPUs + * in old domain are offline */ if (!cpumask_intersects(desc->affinity, mask)) cfg->move_desc_pending = 1; }@@ -1262,8 +1263,11 @@ next: current_vector = vector; current_offset = offset; if (old_vector) { - cfg->move_in_progress = 1; cpumask_copy(cfg->old_domain, cfg->domain); + if (cpumask_intersects(cfg->old_domain, + cpu_online_mask)) { + cfg->move_in_progress = 1; + } } for_each_cpu_and(new_cpu, tmp_mask, cpu_online_mask) per_cpu(vector_irq, new_cpu)[vector] = irq;@@ -2492,7 +2496,8 @@ static void irq_complete_move(struct irq if (likely(!cfg->move_desc_pending)) return;Quoted 9 lines
arch/x86/kernel/apic/io_apic.c 2009-04-08 09:23:16.000000000 -0700 static void irq_complete_move(struct irq
Quoted 7 lines
arch/x86/kernel/apic/io_apic.c 2009-04-08 09:23:16.000000000 -0700 renamed from arch/x86/kernel/apic/io_apic.c 2009-04-08 09:23:00.000000000 -0700 +2
-2- /* domain has not changed, but affinity did */ + /* domain has not changed or all CPUs in old domain + * are offline, but affinity changed */ me = smp_processor_id(); if (cpumask_test_cpu(me, desc->affinity)) { *descp = desc = move_irq_desc(desc, me); --so you mean during
__assign_irq_vector(),cpu_online_maskget updated?No, the CPU being offlined is removed from
cpu_online_maskearlier via a call toremove_cpu_from_maps()fromcpu_disable_common(). This happens just beforefixup_irqs()is called.Quoted 5 lines
with your patch, how about that it just happen right after you check that second time.
it seems we are missing some
lock_vector_lock()on the remove cpu from online mask.Quoted 11 lines
The
remove_cpu_from_maps()call incpu_disable_common()is vector lock protected:
voidcpu_disable_common(void)
{
< snip >
/* It's now safe to remove this processor from the online map */lock_vector_lock();
remove_cpu_from_maps(cpu);
unlock_vector_lock();
fixup_irqs();
}
Quoted 4 lines
__assign_irq_vectoralways hasvector_locklocked...
socpu_online_maskwill not changed during, why do you need to check that again in__assign_irq_vector?
looks like you need to clear move_in_progress in fixup_irqs()
YH
Quoted 122 lines · Yinghai Lu
Quoted 121 lines · Yinghai Lu
Quoted 118 lines · Gary Hade
Quoted 85 lines · Yinghai Lu
Quoted 84 lines · Gary Hade
Impact: Eliminates a race that can leave the system in an unusable state
During rapid offlining of multiple CPUs there is a chance that an IRQ affinity move destination CPU will be offlined before the IRQ affinity move initiated during the offlining of a previous CPU completes. This can happen when the device is not very active and thus fails to generate the IRQ that is needed to complete the IRQ affinity move before the move destination CPU is offlined. When this happens there is an
-EBUSYreturn from__assign_irq_vector()during the offlining of the IRQ move destination CPU which prevents initiation of a new IRQ affinity move operation to an online CPU. This leaves the IRQ affinity set to an offlined CPU.I have been able to reproduce the problem on some of our systems using the following script. When the system is idle the problem often reproduces during the first CPU offlining sequence.
#!/bin/sh
SYS_CPU_DIR=/sys/devices/system/cpu
VICTIM_IRQ=25
IRQ_MASK=f0iteration=0
while true; do
echo $iteration
echo $IRQ_MASK >/proc/irq/$VICTIM_IRQ/smp_affinityfor cpudir in $SYS_CPU_DIR/cpu[1-9] $SYS_CPU_DIR/cpu??; do echo 0 > $cpudir/online
done
for cpudir in $SYS_CPU_DIR/cpu[1-9] $SYS_CPU_DIR/cpu??; do echo 1 > $cpudir/online
done
iteration=expr $iteration + 1
doneThe proposed fix takes advantage of the fact that when all CPUs in the old domain are offline there is nothing to be done by
send_cleanup_vector()during the affinity move completion. So, we simply avoid settingcfg->move_in_progresspreventing the above mentioned-EBUSYreturn from__assign_irq_vector(). This allows initiation of a new IRQ affinity move to a CPU that is not going offline.Signed-off-by Gary Hade <[email protected]>---
arch/x86/kernel/apic/io_apic.c | 11 ++++++++---1 file changed, 8 insertions(+), 3 deletions(-)Index:
linux-2.6.30-rc1/arch/x86/kernel/apic/io_apic.c
===================================================================arch/x86/kernel/apic/io_apic.c 2009-04-08 09:23:16.000000000 -0700 renamed from arch/x86/kernel/apic/io_apic.c 2009-04-08 09:23:00.000000000 -0700 +6
-2@@ -363,7 +363,8 @@ set_extra_move_desc(struct irq_desc *des struct irq_cfg *cfg = desc->chip_data; if (!cfg->move_in_progress) { - /* it means that domain is not changed */ + /* it means that domain has not changed or all CPUs + * in old domain are offline */ if (!cpumask_intersects(desc->affinity, mask)) cfg->move_desc_pending = 1; }@@ -1262,8 +1263,11 @@ next: current_vector = vector; current_offset = offset; if (old_vector) { - cfg->move_in_progress = 1; cpumask_copy(cfg->old_domain, cfg->domain); + if (cpumask_intersects(cfg->old_domain, + cpu_online_mask)) { + cfg->move_in_progress = 1; + } } for_each_cpu_and(new_cpu, tmp_mask, cpu_online_mask) per_cpu(vector_irq, new_cpu)[vector] = irq;@@ -2492,7 +2496,8 @@ static void irq_complete_move(struct irq if (likely(!cfg->move_desc_pending)) return;Quoted 9 lines
arch/x86/kernel/apic/io_apic.c 2009-04-08 09:23:16.000000000 -0700 static void irq_complete_move(struct irq
Quoted 7 lines
arch/x86/kernel/apic/io_apic.c 2009-04-08 09:23:16.000000000 -0700 renamed from arch/x86/kernel/apic/io_apic.c 2009-04-08 09:23:00.000000000 -0700 +2
-2- /* domain has not changed, but affinity did */ + /* domain has not changed or all CPUs in old domain + * are offline, but affinity changed */ me = smp_processor_id(); if (cpumask_test_cpu(me, desc->affinity)) { *descp = desc = move_irq_desc(desc, me); --so you mean during
__assign_irq_vector(),cpu_online_maskget updated?No, the CPU being offlined is removed from
cpu_online_maskearlier via a call toremove_cpu_from_maps()fromcpu_disable_common(). This happens just beforefixup_irqs()is called.Quoted 5 lines
with your patch, how about that it just happen right after you check that second time.
it seems we are missing some
lock_vector_lock()on the remove cpu from online mask.The
remove_cpu_from_maps()call incpu_disable_common()is vector lock protected:
voidcpu_disable_common(void)
{
< snip >
/* It's now safe to remove this processor from the online map */lock_vector_lock();
remove_cpu_from_maps(cpu);
unlock_vector_lock();
fixup_irqs();
}
Quoted 1 line
Quoted 1 line
__assign_irq_vectoralways hasvector_locklocked...
OK, I see the 'vector_lock' spin_lock_irqsave/spin_unlock_irqrestore surrounding the __assign_irq_vector call in assign_irq_vector.
Quoted 1 line
Quoted 1 line
so
cpu_online_maskwill not changed during,
I understand that this 'vector_lock' acquisition prevents multiple simultaneous executions of __assign_irq_vector but does that really prevent another thread executing outside __assign_irq_vector (or outside other 'vector_lock' serialized code) from modifying cpu_online_mask?
Isn't it really 'cpu_add_remove_lock' (also held when __assign_irq_vector() is called in the context of a CPU add or remove) that is used for this purpose?
Quoted 1 line
Quoted 1 line
why do you need to check that again in
__assign_irq_vector?
Because that is where the cfg->move_in_progress flag was being set.
Is there some reason that the content of cpu_online_mask cannot be trusted at this location?
If all the CPUs in the old domain are offline doesn't that imply that we got to that location in response to a CPU offline request?
Quoted 2 lines
Quoted 1 line
looks like you need to clear
move_in_progressinfixup_irqs()
This would be a difficult since I believe the code is currently partitioned in a manner that prevents access to irq_cfg records from functions defined in arch/x86/kernel/irq_32.c and arch/x86/kernel/irq_64.c. It also doesn't feel right to allow cfg->move_in_progress to be set in __assign_irq_vector and then clear it in fixup_irqs().
Gary
Signature · 6 lines
Gary Hade System x Enablement IBM Linux Technology Center 503-578-4503 IBM T/L: 775-4503 [email protected] http://www.ibm.com/linux/ltc
Quoted 153 lines · Gary Hade
Quoted 122 lines · Yinghai Lu
Quoted 121 lines · Yinghai Lu
Quoted 118 lines · Gary Hade
Quoted 85 lines · Yinghai Lu
Quoted 84 lines · Gary Hade
Impact: Eliminates a race that can leave the system in an unusable state
During rapid offlining of multiple CPUs there is a chance that an IRQ affinity move destination CPU will be offlined before the IRQ affinity move initiated during the offlining of a previous CPU completes. This can happen when the device is not very active and thus fails to generate the IRQ that is needed to complete the IRQ affinity move before the move destination CPU is offlined. When this happens there is an
-EBUSYreturn from__assign_irq_vector()during the offlining of the IRQ move destination CPU which prevents initiation of a new IRQ affinity move operation to an online CPU. This leaves the IRQ affinity set to an offlined CPU.I have been able to reproduce the problem on some of our systems using the following script. When the system is idle the problem often reproduces during the first CPU offlining sequence.
#!/bin/sh
SYS_CPU_DIR=/sys/devices/system/cpu
VICTIM_IRQ=25
IRQ_MASK=f0iteration=0
while true; do
echo $iteration
echo $IRQ_MASK >/proc/irq/$VICTIM_IRQ/smp_affinityfor cpudir in $SYS_CPU_DIR/cpu[1-9] $SYS_CPU_DIR/cpu??; do echo 0 > $cpudir/online
done
for cpudir in $SYS_CPU_DIR/cpu[1-9] $SYS_CPU_DIR/cpu??; do echo 1 > $cpudir/online
done
iteration=expr $iteration + 1
doneThe proposed fix takes advantage of the fact that when all CPUs in the old domain are offline there is nothing to be done by
send_cleanup_vector()during the affinity move completion. So, we simply avoid settingcfg->move_in_progresspreventing the above mentioned-EBUSYreturn from__assign_irq_vector(). This allows initiation of a new IRQ affinity move to a CPU that is not going offline.Signed-off-by Gary Hade <[email protected]>---
arch/x86/kernel/apic/io_apic.c | 11 ++++++++---1 file changed, 8 insertions(+), 3 deletions(-)Index:
linux-2.6.30-rc1/arch/x86/kernel/apic/io_apic.c
===================================================================arch/x86/kernel/apic/io_apic.c 2009-04-08 09:23:16.000000000 -0700 renamed from arch/x86/kernel/apic/io_apic.c 2009-04-08 09:23:00.000000000 -0700 +6
-2@@ -363,7 +363,8 @@ set_extra_move_desc(struct irq_desc *des struct irq_cfg *cfg = desc->chip_data; if (!cfg->move_in_progress) { - /* it means that domain is not changed */ + /* it means that domain has not changed or all CPUs + * in old domain are offline */ if (!cpumask_intersects(desc->affinity, mask)) cfg->move_desc_pending = 1; }@@ -1262,8 +1263,11 @@ next: current_vector = vector; current_offset = offset; if (old_vector) { - cfg->move_in_progress = 1; cpumask_copy(cfg->old_domain, cfg->domain); + if (cpumask_intersects(cfg->old_domain, + cpu_online_mask)) { + cfg->move_in_progress = 1; + } } for_each_cpu_and(new_cpu, tmp_mask, cpu_online_mask) per_cpu(vector_irq, new_cpu)[vector] = irq;@@ -2492,7 +2496,8 @@ static void irq_complete_move(struct irq if (likely(!cfg->move_desc_pending)) return;Quoted 9 lines
arch/x86/kernel/apic/io_apic.c 2009-04-08 09:23:16.000000000 -0700 static void irq_complete_move(struct irq
Quoted 7 lines
arch/x86/kernel/apic/io_apic.c 2009-04-08 09:23:16.000000000 -0700 renamed from arch/x86/kernel/apic/io_apic.c 2009-04-08 09:23:00.000000000 -0700 +2
-2- /* domain has not changed, but affinity did */ + /* domain has not changed or all CPUs in old domain + * are offline, but affinity changed */ me = smp_processor_id(); if (cpumask_test_cpu(me, desc->affinity)) { *descp = desc = move_irq_desc(desc, me); --so you mean during
__assign_irq_vector(),cpu_online_maskget updated?No, the CPU being offlined is removed from
cpu_online_maskearlier via a call toremove_cpu_from_maps()fromcpu_disable_common(). This happens just beforefixup_irqs()is called.Quoted 5 lines
with your patch, how about that it just happen right after you check that second time.
it seems we are missing some
lock_vector_lock()on the remove cpu from online mask.The
remove_cpu_from_maps()call incpu_disable_common()is vector lock protected:
voidcpu_disable_common(void)
{
< snip >
/* It's now safe to remove this processor from the online map */lock_vector_lock();
remove_cpu_from_maps(cpu);
unlock_vector_lock();
fixup_irqs();
}Quoted 1 line
Quoted 1 line
__assign_irq_vectoralways hasvector_locklocked...OK, I see the '
vector_lock' spin_lock_irqsave/spin_unlock_irqrestore surrounding the__assign_irq_vectorcall inassign_irq_vector.Quoted 1 line
Quoted 1 line
so
cpu_online_maskwill not changed during,I understand that this '
vector_lock' acquisition prevents multiple simultaneous executions of__assign_irq_vectorbut does that really prevent another thread executing outside__assign_irq_vector(or outside other 'vector_lock' serialized code) from modifyingcpu_online_mask?Isn't it really '
cpu_add_remove_lock' (also held when__assign_irq_vector()is called in the context of a CPU add or remove) that is used for this purpose?Quoted 1 line
Quoted 1 line
why do you need to check that again in
__assign_irq_vector?Because that is where the
cfg->move_in_progressflag was being set.Is there some reason that the content of
cpu_online_maskcannot be trusted at this location?If all the CPUs in the old domain are offline doesn't that imply that we got to that location in response to a CPU offline request?
Quoted 1 line
Quoted 1 line
Quoted 8 lines
Quoted 1 line
looks like you need to clear
move_in_progressinfixup_irqs()This would be a difficult since I believe the code is currently partitioned in a manner that prevents access to
irq_cfgrecords from functions defined inarch/x86/kernel/irq_32.candarch/x86/kernel/irq_64.c. It also doesn't feel right to allowcfg->move_in_progressto be set in__assign_irq_vectorand then clear it infixup_irqs().
it looks before fixup_irqs() cpu_online_mask get updated, and before irq_complete_move get called.
so we could fixup_irqs to clear move_in_progress and cleanup percpu vector_irq ...
YH
Quoted 156 lines · Yinghai Lu
Quoted 155 lines · Gary Hade
Quoted 122 lines · Yinghai Lu
Quoted 121 lines · Yinghai Lu
Quoted 118 lines · Gary Hade
Quoted 85 lines · Yinghai Lu
Quoted 84 lines · Gary Hade
Impact: Eliminates a race that can leave the system in an unusable state
During rapid offlining of multiple CPUs there is a chance that an IRQ affinity move destination CPU will be offlined before the IRQ affinity move initiated during the offlining of a previous CPU completes. This can happen when the device is not very active and thus fails to generate the IRQ that is needed to complete the IRQ affinity move before the move destination CPU is offlined. When this happens there is an
-EBUSYreturn from__assign_irq_vector()during the offlining of the IRQ move destination CPU which prevents initiation of a new IRQ affinity move operation to an online CPU. This leaves the IRQ affinity set to an offlined CPU.I have been able to reproduce the problem on some of our systems using the following script. When the system is idle the problem often reproduces during the first CPU offlining sequence.
#!/bin/sh
SYS_CPU_DIR=/sys/devices/system/cpu
VICTIM_IRQ=25
IRQ_MASK=f0iteration=0
while true; do
echo $iteration
echo $IRQ_MASK >/proc/irq/$VICTIM_IRQ/smp_affinityfor cpudir in $SYS_CPU_DIR/cpu[1-9] $SYS_CPU_DIR/cpu??; do echo 0 > $cpudir/online
done
for cpudir in $SYS_CPU_DIR/cpu[1-9] $SYS_CPU_DIR/cpu??; do echo 1 > $cpudir/online
done
iteration=expr $iteration + 1
doneThe proposed fix takes advantage of the fact that when all CPUs in the old domain are offline there is nothing to be done by
send_cleanup_vector()during the affinity move completion. So, we simply avoid settingcfg->move_in_progresspreventing the above mentioned-EBUSYreturn from__assign_irq_vector(). This allows initiation of a new IRQ affinity move to a CPU that is not going offline.Signed-off-by Gary Hade <[email protected]>---
arch/x86/kernel/apic/io_apic.c | 11 ++++++++---1 file changed, 8 insertions(+), 3 deletions(-)Index:
linux-2.6.30-rc1/arch/x86/kernel/apic/io_apic.c
===================================================================arch/x86/kernel/apic/io_apic.c 2009-04-08 09:23:16.000000000 -0700 renamed from arch/x86/kernel/apic/io_apic.c 2009-04-08 09:23:00.000000000 -0700 +6
-2@@ -363,7 +363,8 @@ set_extra_move_desc(struct irq_desc *des struct irq_cfg *cfg = desc->chip_data; if (!cfg->move_in_progress) { - /* it means that domain is not changed */ + /* it means that domain has not changed or all CPUs + * in old domain are offline */ if (!cpumask_intersects(desc->affinity, mask)) cfg->move_desc_pending = 1; }@@ -1262,8 +1263,11 @@ next: current_vector = vector; current_offset = offset; if (old_vector) { - cfg->move_in_progress = 1; cpumask_copy(cfg->old_domain, cfg->domain); + if (cpumask_intersects(cfg->old_domain, + cpu_online_mask)) { + cfg->move_in_progress = 1; + } } for_each_cpu_and(new_cpu, tmp_mask, cpu_online_mask) per_cpu(vector_irq, new_cpu)[vector] = irq;@@ -2492,7 +2496,8 @@ static void irq_complete_move(struct irq if (likely(!cfg->move_desc_pending)) return;Quoted 9 lines
arch/x86/kernel/apic/io_apic.c 2009-04-08 09:23:16.000000000 -0700 static void irq_complete_move(struct irq
Quoted 7 lines
arch/x86/kernel/apic/io_apic.c 2009-04-08 09:23:16.000000000 -0700 renamed from arch/x86/kernel/apic/io_apic.c 2009-04-08 09:23:00.000000000 -0700 +2
-2- /* domain has not changed, but affinity did */ + /* domain has not changed or all CPUs in old domain + * are offline, but affinity changed */ me = smp_processor_id(); if (cpumask_test_cpu(me, desc->affinity)) { *descp = desc = move_irq_desc(desc, me); --so you mean during
__assign_irq_vector(),cpu_online_maskget updated?No, the CPU being offlined is removed from
cpu_online_maskearlier via a call toremove_cpu_from_maps()fromcpu_disable_common(). This happens just beforefixup_irqs()is called.Quoted 5 lines
with your patch, how about that it just happen right after you check that second time.
it seems we are missing some
lock_vector_lock()on the remove cpu from online mask.The
remove_cpu_from_maps()call incpu_disable_common()is vector lock protected:
voidcpu_disable_common(void)
{
< snip >
/* It's now safe to remove this processor from the online map */lock_vector_lock();
remove_cpu_from_maps(cpu);
unlock_vector_lock();
fixup_irqs();
}Quoted 1 line
Quoted 1 line
__assign_irq_vectoralways hasvector_locklocked...OK, I see the '
vector_lock' spin_lock_irqsave/spin_unlock_irqrestore surrounding the__assign_irq_vectorcall inassign_irq_vector.Quoted 1 line
Quoted 1 line
so
cpu_online_maskwill not changed during,I understand that this '
vector_lock' acquisition prevents multiple simultaneous executions of__assign_irq_vectorbut does that really prevent another thread executing outside__assign_irq_vector(or outside other 'vector_lock' serialized code) from modifyingcpu_online_mask?Isn't it really '
cpu_add_remove_lock' (also held when__assign_irq_vector()is called in the context of a CPU add or remove) that is used for this purpose?Quoted 1 line
Quoted 1 line
why do you need to check that again in
__assign_irq_vector?Because that is where the
cfg->move_in_progressflag was being set.Is there some reason that the content of
cpu_online_maskcannot be trusted at this location?If all the CPUs in the old domain are offline doesn't that imply that we got to that location in response to a CPU offline request?
Quoted 2 lines
Quoted 1 line
looks like you need to clear
move_in_progressinfixup_irqs()
Quoted 8 lines
Quoted 6 lines
This would be a difficult since I believe the code is currently partitioned in a manner that prevents access to
irq_cfgrecords from functions defined inarch/x86/kernel/irq_32.candarch/x86/kernel/irq_64.c. It also doesn't feel right to allowcfg->move_in_progressto be set in__assign_irq_vectorand then clear it infixup_irqs().it looks before
fixup_irqs()cpu_online_maskget updated,
Correct. It is updated during the remove_cpu_from_maps() call from cpu_disable_common() that preceeds the fixup_irqs() call from cpu_disable_common().
Quoted 1 line
and before
irq_complete_moveget called.
Also correct but it should be noted that irq_complete_move() executes on an IRQ affinity move destination CPU which, depending on how quickly the device generates the next IRQ, may not happen before the IRQ affinity move destination CPU(s) are offlined.
Quoted 3 lines
so we could
fixup_irqsto clearmove_in_progressand cleanup percpuvector_irq...
If you are suggesting that the send_cleanup_vector() work deferred to an IRQ affinity move destination CPU be performed instead by code running on the CPU being removed, I considered that before discovering the justification for the current design.
http://lkml.org/lkml/2007/2/23/92
I also played around with using send_IPI_mask() to send cfg->vector to the IRQ affinity move destination CPU(s) to assure that cfg->move_in_progress would get cleared before those CPU(s) were offlined. This appeared to work, however, it seemed like a hack and I wasn't sure how to deal with the unhandled interrupts.
I then discovered that send_cleanup_vector() wasn't sending IRQ_MOVE_CLEANUP_VECTOR to any CPU when all CPUs in the old domain were offline. This led me to the proposed fix.
Gary
Signature · 6 lines
Gary Hade System x Enablement IBM Linux Technology Center 503-578-4503 IBM T/L: 775-4503 [email protected] http://www.ibm.com/linux/ltc
Quoted 15 lines · Gary Hade
Impact: Eliminates a race that can leave the system in an
unusable stateDuring rapid offlining of multiple CPUs there is a chance that an IRQ affinity move destination CPU will be offlined before the IRQ affinity move initiated during the offlining of a previous CPU completes. This can happen when the device is not very active and thus fails to generate the IRQ that is needed to complete the IRQ affinity move before the move destination CPU is offlined. When this happens there is an
-EBUSYreturn from__assign_irq_vector()during the offlining of the IRQ move destination CPU which prevents initiation of a new IRQ affinity move operation to an online CPU. This leaves the IRQ affinity set to an offlined CPU.
Quoted 4 lines
I have been able to reproduce the problem on some of our systems using the following script. When the system is idle the problem often reproduces during the first CPU offlining sequence.
You appear to be focusing on the IBM x460 and x3835. Can you describe what kind of interrupt setup you are running.
You may be the first person to actually hit the problems with cpu offlining and irq migration that have theoretically been present for a long.
Eric
Quoted 17 lines · Eric W. Biederman
Quoted 15 lines · Gary Hade
Impact: Eliminates a race that can leave the system in an
unusable stateDuring rapid offlining of multiple CPUs there is a chance that an IRQ affinity move destination CPU will be offlined before the IRQ affinity move initiated during the offlining of a previous CPU completes. This can happen when the device is not very active and thus fails to generate the IRQ that is needed to complete the IRQ affinity move before the move destination CPU is offlined. When this happens there is an
-EBUSYreturn from__assign_irq_vector()during the offlining of the IRQ move destination CPU which prevents initiation of a new IRQ affinity move operation to an online CPU. This leaves the IRQ affinity set to an offlined CPU.
Quoted 6 lines
Quoted 4 lines
I have been able to reproduce the problem on some of our systems using the following script. When the system is idle the problem often reproduces during the first CPU offlining sequence.
You appear to be focusing on the IBM x460 and x3835.
True. I have also observed IRQ interruptions on an IBM x3950 M2 which I believe, but am not certain, were due to the other "I/O redirection table register write with Remote IRR bit set" caused problem.
I intend to do more testing on the x3950 M2 and other IBM System x servers but I unfortunately do not currently have access to any Intel based non-IBM MP servers. I was hoping that my testing request might at least get some others interested in running the simple test script on their systems and reporting their results. Have you perhaps tried the test on any of the Intel based MP systems that you have access to?
Quoted 2 lines
Can you describe
what kind of interrupt setup you are running.
Being somewhat of a ioapic neophyte I am not exactly sure what you are asking for here. This is ioapic information logged during boot if that helps at all.
x3850:
ACPI: IOAPIC (id[0x0f] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 15, version 0, address 0xfec00000, GSI 0-35
ACPI: IOAPIC (id[0x0e] address[0xfec01000] gsi_base[36])
IOAPIC[1]: apic_id 14, version 0, address 0xfec01000, GSI 36-71
x460:
ACPI: IOAPIC (id[0x0f] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 15, version 17, address 0xfec00000, GSI 0-35
ACPI: IOAPIC (id[0x0e] address[0xfec01000] gsi_base[36])
IOAPIC[1]: apic_id 14, version 17, address 0xfec01000, GSI 36-71
ACPI: IOAPIC (id[0x0d] address[0xfec02000] gsi_base[72])
IOAPIC[2]: apic_id 13, version 17, address 0xfec02000, GSI 72-107
ACPI: IOAPIC (id[0x0c] address[0xfec03000] gsi_base[108])
IOAPIC[3]: apic_id 12, version 17, address 0xfec03000, GSI 108-143Quoted 3 lines
You may be the first person to actually hit the problems with cpu offlining and irq migration that have theoretically been present for a long.
Your "Safely cleanup an irq after moving it" changes have been present in mainline for quite some time so I have been thinking about this as well.
I can certainly understand why it may not be very likely for users to see the "I/O redirection table register write with Remote IRR bit set" caused problem. It has actually been fairly difficult to reproduce. I very much doubt that there are many users out there that would be continuously offlining and onlining all the offlineable CPUs from a script or program on a heavily loaded system. IMO, this would _not_ be a very common useage scenario. The test script that I provided usually performs many CPU offline/online iterations before the problem is triggered. A much more likely useage scenario, for which there is already code in ack_apic_level() to avoid the problem, would be IRQ affinity adjustments requested from user level (e.g. by the irqbalance daemon) on a very active system.
It is less clear to me why users have not been reporting the idle system race but I suspect that
- script or program driven offlining of multiple CPUs may not be very common
- the actual affinity on an idle system is usually set to cpu0 which is always online
I am glad you are looking at this since I know it involves code that you should be quite familiar with. Thanks!
Gary
Signature · 6 lines
Gary Hade System x Enablement IBM Linux Technology Center 503-578-4503 IBM T/L: 775-4503 [email protected] http://www.ibm.com/linux/ltc
Quoted 46 lines · Gary Hade
Quoted 17 lines · Eric W. Biederman
Quoted 15 lines · Gary Hade
Impact: Eliminates a race that can leave the system in an
unusable stateDuring rapid offlining of multiple CPUs there is a chance that an IRQ affinity move destination CPU will be offlined before the IRQ affinity move initiated during the offlining of a previous CPU completes. This can happen when the device is not very active and thus fails to generate the IRQ that is needed to complete the IRQ affinity move before the move destination CPU is offlined. When this happens there is an
-EBUSYreturn from__assign_irq_vector()during the offlining of the IRQ move destination CPU which prevents initiation of a new IRQ affinity move operation to an online CPU. This leaves the IRQ affinity set to an offlined CPU.Quoted 6 lines
Quoted 4 lines
I have been able to reproduce the problem on some of our systems using the following script. When the system is idle the problem often reproduces during the first CPU offlining sequence.
You appear to be focusing on the IBM x460 and x3835.
True. I have also observed IRQ interruptions on an IBM x3950 M2 which I believe, but am not certain, were due to the other "I/O redirection table register write with Remote IRR bit set" caused problem.
I intend to do more testing on the x3950 M2 and other IBM System x servers but I unfortunately do not currently have access to any Intel based non-IBM MP servers. I was hoping that my testing request might at least get some others interested in running the simple test script on their systems and reporting their results. Have you perhaps tried the test on any of the Intel based MP systems that you have access to?
Quoted 2 lines
Can you describe
what kind of interrupt setup you are running.Being somewhat of a ioapic neophyte I am not exactly sure what you are asking for here. This is ioapic information logged during boot if that helps at all.
x3850:
Quoted 13 lines
ACPI: IOAPIC (id[0x0f] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 15, version 0, address 0xfec00000, GSI 0-35
ACPI: IOAPIC (id[0x0e] address[0xfec01000] gsi_base[36])
IOAPIC[1]: apic_id 14, version 0, address 0xfec01000, GSI 36-71
x460:
ACPI: IOAPIC (id[0x0f] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 15, version 17, address 0xfec00000, GSI 0-35
ACPI: IOAPIC (id[0x0e] address[0xfec01000] gsi_base[36])
IOAPIC[1]: apic_id 14, version 17, address 0xfec01000, GSI 36-71
ACPI: IOAPIC (id[0x0d] address[0xfec02000] gsi_base[72])
IOAPIC[2]: apic_id 13, version 17, address 0xfec02000, GSI 72-107
ACPI: IOAPIC (id[0x0c] address[0xfec03000] gsi_base[108])
IOAPIC[3]: apic_id 12, version 17, address 0xfec03000, GSI 108-143Sorry. My real question is which mode you are running the ioapics in.
Quoted 28 lines
Quoted 2 lines
You may be the first person to actually hit the problems with cpu offlining and irq migration that have theoretically been present for a long.
Your "Safely cleanup an irq after moving it" changes have been present in mainline for quite some time so I have been thinking about this as well.
I can certainly understand why it may not be very likely for users to see the "I/O redirection table register write with Remote IRR bit set" caused problem. It has actually been fairly difficult to reproduce. I very much doubt that there are many users out there that would be continuously offlining and onlining all the offlineable CPUs from a script or program on a heavily loaded system. IMO, this would _not_ be a very common useage scenario. The test script that I provided usually performs many CPU offline/online iterations before the problem is triggered. A much more likely useage scenario, for which there is already code in
ack_apic_level()to avoid the problem, would be IRQ affinity adjustments requested from user level (e.g. by the irqbalance daemon) on a very active system.It is less clear to me why users have not been reporting the idle system race but I suspect that
- script or program driven offlining of multiple CPUs may not be very common
- the actual affinity on an idle system is usually set to cpu0 which is always online
Quoted 2 lines
I am glad you are looking at this since I know it involves code that you should be quite familiar with. Thanks!
The class of irq and the mode in which we run it make a bit difference.
MSIs and machines with iommus that remap irqs we should be able to migrate irqs safely at any time.
Also lowest priority delivery mode when all cpus are setup on the same vector to receive an irq. Changing it's affinity should be safe. This works for up to 8 cpus. And I expect is the common case.
The rest of the case when we change the vector number short of turning off the devices generating the interrupts (and thus stopping them at the source) I do not know of a safe way to disable interrupts.
The only reason I did not make the irq migration at cpu shutdown time depend on CONFIG_BROKEN was that it is by the suspend code on laptops.
Alternatively if you have ioapics that support having their irqs acknowledged with a register I like ia64 has. I think there are some additional options.
So depending on what hardware you have it might be possible to implement this safely.
It looks like some additional bugs have slipped in since last I looked.
set_irq_affinity does this:
ifdef CONFIG_GENERIC_PENDING_IRQ
if (desc->status & IRQ_MOVE_PCNTXT || desc->status & IRQ_DISABLED) {
cpumask_copy(desc->affinity, cpumask);
desc->chip->set_affinity(irq, cpumask);
} else {
desc->status |= IRQ_MOVE_PENDING;
cpumask_copy(desc->pending_mask, cpumask);
}#else
That IRQ_DISABLED case is a software state and as such it has nothing to do with how safe it is to move an irq in process context.
Do any of your device drivers call irq_disable?
Eric
Quoted 62 lines · Eric W. Biederman
Quoted 51 lines · Gary Hade
Quoted 17 lines · Eric W. Biederman
Quoted 15 lines · Gary Hade
Impact: Eliminates a race that can leave the system in an unusable state
During rapid offlining of multiple CPUs there is a chance that an IRQ affinity move destination CPU will be offlined before the IRQ affinity move initiated during the offlining of a previous CPU completes. This can happen when the device is not very active and thus fails to generate the IRQ that is needed to complete the IRQ affinity move before the move destination CPU is offlined. When this happens there is an
-EBUSYreturn from__assign_irq_vector()during the offlining of the IRQ move destination CPU which prevents initiation of a new IRQ affinity move operation to an online CPU. This leaves the IRQ affinity set to an offlined CPU.Quoted 6 lines
Quoted 4 lines
I have been able to reproduce the problem on some of our systems using the following script. When the system is idle the problem often reproduces during the first CPU offlining sequence.
You appear to be focusing on the IBM x460 and x3835.
True. I have also observed IRQ interruptions on an IBM x3950 M2 which I believe, but am not certain, were due to the other "I/O redirection table register write with Remote IRR bit set" caused problem.
I intend to do more testing on the x3950 M2 and other IBM System x servers but I unfortunately do not currently have access to any Intel based non-IBM MP servers. I was hoping that my testing request might at least get some others interested in running the simple test script on their systems and reporting their results. Have you perhaps tried the test on any of the Intel based MP systems that you have access to?
Quoted 2 lines
Can you describe
what kind of interrupt setup you are running.Being somewhat of a ioapic neophyte I am not exactly sure what you are asking for here. This is ioapic information logged during boot if that helps at all.
x3850:
ACPI: IOAPIC (id[0x0f] address[0xfec00000] gsi_base[0]) IOAPIC[0]:apic_id15, version 0, address0xfec00000, GSI 0-35
ACPI: IOAPIC (id[0x0e] address[0xfec01000] gsi_base[36]) IOAPIC[1]:apic_id14, version 0, address0xfec01000, GSI 36-71 x460:Quoted 8 lines
ACPI: IOAPIC (id[0x0f] address[0xfec00000] gsi_base[0]) IOAPIC[0]:
apic_id15, version 17, address0xfec00000, GSI 0-35
ACPI: IOAPIC (id[0x0e] address[0xfec01000] gsi_base[36]) IOAPIC[1]:apic_id14, version 17, address0xfec01000, GSI 36-71
ACPI: IOAPIC (id[0x0d] address[0xfec02000] gsi_base[72]) IOAPIC[2]:apic_id13, version 17, address0xfec02000, GSI 72-107
ACPI: IOAPIC (id[0x0c] address[0xfec03000] gsi_base[108]) IOAPIC[3]:apic_id12, version 17, address0xfec03000, GSI 108-143
Quoted 2 lines
Sorry. My real question is which mode you are running the ioapics in.
looks like ack_level_irq.
YH
Quoted 15 lines · Eric W. Biederman
Gary Hade <[email protected]> writes:
It looks like some additional bugs have slipped in since last I looked.
set_irq_affinitydoes this:
ifdefCONFIG_GENERIC_PENDING_IRQ
if (desc->status&IRQ_MOVE_PCNTXT||desc->status&IRQ_DISABLED) {cpumask_copy(desc->affinity, cpumask);desc->chip->set_affinity(irq, cpumask);
} else {
desc->status |=IRQ_MOVE_PENDING;cpumask_copy(desc->pending_mask, cpumask);
}
#else
Quoted 3 lines
That
IRQ_DISABLEDcase is a software state and as such it has nothing to do with how safe it is to move an irq in process context.
commit 932775a4ab622e3c99bd59f14cc7d96722f79501
Author: [email protected] <[email protected]> Date: Fri Sep 5 18:02:15 2008 -0700
x86: HPET_MSI change IRQ affinity in process context when it is disabled
Change the IRQ affinity in the process context when the IRQ is disabled.
kernel/irq/manage.c +2 -1
@@ -87,10 +87,11 @@ int irq_set_affinity(unsigned int irq, cpumask_t cpumask)return -EINVAL ;#ifdef CONFIG_GENERIC_PENDING_IRQ -if (desc->status &IRQ_MOVE_PCNTXT ) { +if (desc->status &IRQ_MOVE_PCNTXT || desc->status &IRQ_DISABLED ) {unsigned long flags;spin_lock_irqsave (&desc->lock, flags); + desc->affinity = cpumask; desc->chip->set_affinity (irq, cpumask);spin_unlock_irqrestore (&desc->lock, flags); }else
Quoted 19 lines · Yinghai Lu
commit
932775a4ab622e3c99bd59f14cc7d96722f79501Author: [email protected] <[email protected]> Date: Fri Sep 5 18:02:15 2008 -0700x86:
HPET_MSIchange IRQ affinity in process context when it is disabledChange the IRQ affinity in the process context when the IRQ is disabled.
Signed-off-by Venkatesh Pallipadi <[email protected]>Signed-off-by Shaohua Li <[email protected]>Signed-off-by Ingo Molnar <[email protected]>kernel/irq/manage.c +0
-0@@ -87,10 +87,11 @@ int irq_set_affinity(unsigned int irq, cpumask_t cpumask)return -EINVAL ;
kernel/irq/manage.c int irq_set_affinity(unsigned int irq, cpumask_t cpumask) · line 97
Quoted 10 lines
kernel/irq/manage.c +2
-1#ifdef CONFIG_GENERIC_PENDING_IRQ -if (desc->status &IRQ_MOVE_PCNTXT ) { +if (desc->status &IRQ_MOVE_PCNTXT || desc->status &IRQ_DISABLED ) {unsigned long flags;spin_lock_irqsave (&desc->lock, flags); + desc->affinity = cpumask; desc->chip->set_affinity (irq, cpumask);spin_unlock_irqrestore (&desc->lock, flags); }else
If the goal is moving MSIs, we should modify the msi code to be safe in process context and to set IRQ_MOVE_PCNTXT.
The only reason we migrate MSIs in interrupt context today is that there wasn't infrastructure for support migration both in interrupt context and outside of it.
Eric
Quoted 35 lines · Eric W. Biederman
Quoted 19 lines · Yinghai Lu
commit
932775a4ab622e3c99bd59f14cc7d96722f79501Author: [email protected] <[email protected]> Date: Fri Sep 5 18:02:15 2008 -0700x86:
HPET_MSIchange IRQ affinity in process context when it is disabledChange the IRQ affinity in the process context when the IRQ is disabled.
Signed-off-by Venkatesh Pallipadi <[email protected]>Signed-off-by Shaohua Li <[email protected]>Signed-off-by Ingo Molnar <[email protected]>kernel/irq/manage.c +0
-0@@ -87,10 +87,11 @@ int irq_set_affinity(unsigned int irq, cpumask_t cpumask)return -EINVAL ;kernel/irq/manage.c int irq_set_affinity(unsigned int irq, cpumask_t cpumask) · line 97
Quoted 10 lines
kernel/irq/manage.c +2
-1#ifdef CONFIG_GENERIC_PENDING_IRQ -if (desc->status &IRQ_MOVE_PCNTXT ) { +if (desc->status &IRQ_MOVE_PCNTXT || desc->status &IRQ_DISABLED ) {unsigned long flags;spin_lock_irqsave (&desc->lock, flags); + desc->affinity = cpumask; desc->chip->set_affinity (irq, cpumask);spin_unlock_irqrestore (&desc->lock, flags); }else If the goal is moving MSIs, we should modify the msi code to be safe in process context and to set
IRQ_MOVE_PCNTXT.
Quoted 4 lines
The only reason we migrate MSIs in interrupt context today is that there wasn't infrastructure for support migration both in interrupt context and outside of it.
Yes. The idea here was to force the MSI migration to happen in process context. One of the patches in the series did
disable_irq(dev->irq); irq_set_affinity(dev->irq, cpumask_of(dev->cpu)); enable_irq(dev->irq);
with the above patch adding irq/manage code check for interrupt disabled and moving the interrupt in process context.
IIRC, there was no IRQ_MOVE_PCNTXT when we were developing this HPET code and we ended up having this ugly hack. IRQ_MOVE_PCNTXT was there when we eventually submitted the patch upstream. But, looks like I did a blind rebasing instead of using IRQ_MOVE_PCNTXT in hpet MSI code. That was my fault. Will send a patch to fix this ugliness.
Thanks,
Venki
Quoted 48 lines · Venkatesh
Quoted 35 lines · Eric W. Biederman
Quoted 19 lines · Yinghai Lu
commit
932775a4ab622e3c99bd59f14cc7d96722f79501Author: [email protected] <[email protected]> Date: Fri Sep 5 18:02:15 2008 -0700x86:
HPET_MSIchange IRQ affinity in process context when it is disabledChange the IRQ affinity in the process context when the IRQ is disabled.
Signed-off-by Venkatesh Pallipadi <[email protected]>Signed-off-by Shaohua Li <[email protected]>Signed-off-by Ingo Molnar <[email protected]>kernel/irq/manage.c +0
-0@@ -87,10 +87,11 @@ int irq_set_affinity(unsigned int irq, cpumask_t cpumask)return -EINVAL ;kernel/irq/manage.c int irq_set_affinity(unsigned int irq, cpumask_t cpumask) · line 97
Quoted 10 lines
kernel/irq/manage.c +2
-1#ifdef CONFIG_GENERIC_PENDING_IRQ -if (desc->status &IRQ_MOVE_PCNTXT ) { +if (desc->status &IRQ_MOVE_PCNTXT || desc->status &IRQ_DISABLED ) {unsigned long flags;spin_lock_irqsave (&desc->lock, flags); + desc->affinity = cpumask; desc->chip->set_affinity (irq, cpumask);spin_unlock_irqrestore (&desc->lock, flags); }else If the goal is moving MSIs, we should modify the msi code to be safe in process context and to set
IRQ_MOVE_PCNTXT.Quoted 4 lines
The only reason we migrate MSIs in interrupt context today is that there wasn't infrastructure for support migration both in interrupt context and outside of it.
Yes. The idea here was to force the MSI migration to happen in process context. One of the patches in the series did
disable_irq(dev->irq); irq_set_affinity(dev->irq, cpumask_of(dev->cpu)); enable_irq(dev->irq);
Quoted 8 lines
with the above patch adding irq/manage code check for interrupt disabled and moving the interrupt in process context.
IIRC, there was no
IRQ_MOVE_PCNTXTwhen we were developing this HPET code and we ended up having this ugly hack.IRQ_MOVE_PCNTXTwas there when we eventually submitted the patch upstream. But, looks like I did a blind rebasing instead of usingIRQ_MOVE_PCNTXTin hpet MSI code. That was my fault. Will send a patch to fix this ugliness.
Thanks.
Eric
PATCH 1/1 irq, x86: Remove IRQ_DISABLED check in process context IRQ move
As discussed in the thread here
http://marc.info/?l=linux-kernel&m=123964468521142&w=2
Quoted 17 lines · Eric W. Biederman
It looks like some additional bugs have slipped in since last I looked.
set_irq_affinitydoes this:
ifdefCONFIG_GENERIC_PENDING_IRQif (desc->status & IRQ_MOVE_PCNTXT || desc->status & IRQ_DISABLED) { cpumask_copy(desc->affinity, cpumask); desc->chip->set_affinity(irq, cpumask); } else { desc->status |= IRQ_MOVE_PENDING; cpumask_copy(desc->pending_mask, cpumask); }#else
That
IRQ_DISABLEDcase is a software state and as such it has nothing to do with how safe it is to move an irq in process context.
Quoted 16 lines · Venkatesh
Quoted 7 lines · Eric W. Biederman
If the goal is moving MSIs, we should modify the msi code to be safe in process context and to set
IRQ_MOVE_PCNTXT.The only reason we migrate MSIs in interrupt context today is that there wasn't infrastructure for support migration both in interrupt context and outside of it.
Yes. The idea here was to force the MSI migration to happen in process context. One of the patches in the series did
disable_irq(dev->irq); irq_set_affinity(dev->irq, cpumask_of(dev->cpu)); enable_irq(dev->irq);
Quoted 8 lines
with the above patch adding irq/manage code check for interrupt disabled and moving the interrupt in process context.
IIRC, there was no
IRQ_MOVE_PCNTXTwhen we were developing this HPET code and we ended up having this ugly hack.IRQ_MOVE_PCNTXTwas there when we eventually submitted the patch upstream. But, looks like I did a blind rebasing instead of usingIRQ_MOVE_PCNTXTin hpet MSI code. That was my fault. Will send a patch to fix this ugliness.
Below patch fixes this. i.e., revert
commit 932775a4ab622e3c99bd59f14cc7d96722f79501
and add PCNTXT to HPET MSI setup. Also removes copying of desc->affinity in generic code as set_affinity routines are doing it internally.
--- arch/x86/kernel/apic/io_apic.c | 2 ++ kernel/irq/manage.c | 5 ++--- 2 files changed, 4 insertions(+), 3 deletions(-)
arch/x86/kernel/apic/io_apic.c +2 -0
@@ -3667,12 +3667,14 @@ int arch_setup_hpet_msi(unsigned int irq) {int ret;struct msi_msg msg; +struct irq_desc *desc =irq_to_desc (irq); ret =msi_compose_msg (NULL , irq, &msg);if (ret <0 )return ret;hpet_msi_write (irq, &msg); + desc->status |=IRQ_MOVE_PCNTXT ;set_irq_chip_and_handler_name (irq, &hpet_msi_type, handle_edge_irq,"edge" );
kernel/irq/manage.c +2 -3
@@ -109,10 +109,9 @@ int irq_set_affinity(unsigned int irq, const struct cpumask *cpumask)spin_lock_irqsave (&desc->lock, flags);#ifdef CONFIG_GENERIC_PENDING_IRQ -if (desc->status &IRQ_MOVE_PCNTXT || desc->status &IRQ_DISABLED ) { -cpumask_copy (desc->affinity, cpumask); +if (desc->status &IRQ_MOVE_PCNTXT ) desc->chip->set_affinity (irq, cpumask); - }else { +else { desc->status |=IRQ_MOVE_PENDING ;cpumask_copy (desc->pending_mask, cpumask); }
Signature · 1 line
1.6.0.6
Re: [PATCH] irq, x86: Remove IRQ_DISABLED check in process context IRQ move
Quoted 49 lines · Venkatesh
As discussed in the thread here
http://marc.info/?l=linux-kernel&m=123964468521142&w=2Quoted 17 lines · Eric W. Biederman
It looks like some additional bugs have slipped in since last I looked.
set_irq_affinitydoes this:
ifdefCONFIG_GENERIC_PENDING_IRQif (desc->status & IRQ_MOVE_PCNTXT || desc->status & IRQ_DISABLED) { cpumask_copy(desc->affinity, cpumask); desc->chip->set_affinity(irq, cpumask); } else { desc->status |= IRQ_MOVE_PENDING; cpumask_copy(desc->pending_mask, cpumask); }#else
That
IRQ_DISABLEDcase is a software state and as such it has nothing to do with how safe it is to move an irq in process context.Quoted 16 lines · Venkatesh
Quoted 7 lines · Eric W. Biederman
If the goal is moving MSIs, we should modify the msi code to be safe in process context and to set
IRQ_MOVE_PCNTXT.The only reason we migrate MSIs in interrupt context today is that there wasn't infrastructure for support migration both in interrupt context and outside of it.
Yes. The idea here was to force the MSI migration to happen in process context. One of the patches in the series did
disable_irq(dev->irq); irq_set_affinity(dev->irq, cpumask_of(dev->cpu)); enable_irq(dev->irq);Quoted 8 lines
with the above patch adding irq/manage code check for interrupt disabled and moving the interrupt in process context.
IIRC, there was no
IRQ_MOVE_PCNTXTwhen we were developing this HPET code and we ended up having this ugly hack.IRQ_MOVE_PCNTXTwas there when we eventually submitted the patch upstream. But, looks like I did a blind rebasing instead of usingIRQ_MOVE_PCNTXTin hpet MSI code. That was my fault. Will send a patch to fix this ugliness.
Quoted 4 lines
Below patch fixes this. i.e., revert
commit932775a4ab622e3c99bd59f14cc7d96722f79501
and add PCNTXT to HPET MSI setup. Also removes copying ofdesc->affinityin generic code asset_affinityroutines are doing it internally.
This looks good.
Do you think you could take this one step farther, place a read after the hpet_msi_write to flush the write to the interrupt source, and then finish up the work to change the irq reception setup?
Roughly like ir_set_msi_irq_affinity?
That way we really do get everything done in process context.
Quoted 44 lines
Signed-off-by Venkatesh Pallipadi <[email protected]>--- arch/x86/kernel/apic/io_apic.c | 2 ++ kernel/irq/manage.c | 5 ++--- 2 files changed, 4 insertions(+), 3 deletions(-)arch/x86/kernel/apic/io_apic.c +2
-0@@ -3667,12 +3667,14 @@ int arch_setup_hpet_msi(unsigned int irq) {int ret;struct msi_msg msg; +struct irq_desc *desc =irq_to_desc (irq); ret =msi_compose_msg (NULL , irq, &msg);if (ret <0 )return ret;hpet_msi_write (irq, &msg); + desc->status |=IRQ_MOVE_PCNTXT ;set_irq_chip_and_handler_name (irq, &hpet_msi_type, handle_edge_irq,"edge" );kernel/irq/manage.c +2
-3@@ -109,10 +109,9 @@ int irq_set_affinity(unsigned int irq, const struct cpumask *cpumask)spin_lock_irqsave (&desc->lock, flags);#ifdef CONFIG_GENERIC_PENDING_IRQ -if (desc->status &IRQ_MOVE_PCNTXT || desc->status &IRQ_DISABLED ) { -cpumask_copy (desc->affinity, cpumask); +if (desc->status &IRQ_MOVE_PCNTXT ) desc->chip->set_affinity (irq, cpumask); - }else { +else { desc->status |=IRQ_MOVE_PENDING ;cpumask_copy (desc->pending_mask, cpumask); }Signature · 1 line
1.6.0.6
[tip:irq/urgent] x86, irq: Remove IRQ_DISABLED check in process context IRQ move
In reply to Pallipadi, VenkateshCommit-ID: 6ec3cfeca04622e3d80c9270191cd7f5f88214af Gitweb: http://git.kernel.org/tip/6ec3cfeca04622e3d80c9270191cd7f5f88214af Author: Pallipadi, Venkatesh <[email protected]> AuthorDate: Mon, 13 Apr 2009 15:20:58 -0700 Committer: Ingo Molnar <[email protected]> CommitDate: Tue, 14 Apr 2009 15:21:13 +0200
x86, irq: Remove IRQ_DISABLED check in process context IRQ move
As discussed in the thread here:
http://marc.info/?l=linux-kernel&m=123964468521142&w=2
Eric W. Biederman observed:
Quoted 15 lines
It looks like some additional bugs have slipped in since last I looked.
set_irq_affinitydoes this:
ifdefCONFIG_GENERIC_PENDING_IRQif (desc->status & IRQ_MOVE_PCNTXT || desc->status & IRQ_DISABLED) { cpumask_copy(desc->affinity, cpumask); desc->chip->set_affinity(irq, cpumask); } else { desc->status |= IRQ_MOVE_PENDING; cpumask_copy(desc->pending_mask, cpumask); }#else
That
IRQ_DISABLEDcase is a software state and as such it has nothing to do with how safe it is to move an irq in process context.
[...]
Quoted 4 lines
The only reason we migrate MSIs in interrupt context today is that there wasn't infrastructure for support migration both in interrupt context and outside of it.
Yes. The idea here was to force the MSI migration to happen in process context. One of the patches in the series did
disable_irq(dev->irq);
irq_set_affinity(dev->irq, cpumask_of(dev->cpu));
enable_irq(dev->irq);with the above patch adding irq/manage code check for interrupt disabled and moving the interrupt in process context.
IIRC, there was no IRQ_MOVE_PCNTXT when we were developing this HPET code and we ended up having this ugly hack. IRQ_MOVE_PCNTXT was there when we eventually submitted the patch upstream. But, looks like I did a blind rebasing instead of using IRQ_MOVE_PCNTXT in hpet MSI code.
Below patch fixes this. i.e., revert commit 932775a4ab622e3c99bd59f14cc and add PCNTXT to HPET MSI setup. Also removes copying of desc->affinity in generic code as set_affinity routines are doing it internally.
LKML-Reference: <[email protected]>
--- arch/x86/kernel/apic/io_apic.c | 2 ++ kernel/irq/manage.c | 5 ++--- 2 files changed, 4 insertions(+), 3 deletions(-)
arch/x86/kernel/apic/io_apic.c +2 -0
@@ -3670,12 +3670,14 @@ int arch_setup_hpet_msi(unsigned int irq) {int ret;struct msi_msg msg; +struct irq_desc *desc =irq_to_desc (irq); ret =msi_compose_msg (NULL , irq, &msg);if (ret <0 )return ret;hpet_msi_write (irq, &msg); + desc->status |=IRQ_MOVE_PCNTXT ;set_irq_chip_and_handler_name (irq, &hpet_msi_type, handle_edge_irq,"edge" );
kernel/irq/manage.c +2 -3
@@ -109,10 +109,9 @@ int irq_set_affinity(unsigned int irq, const struct cpumask *cpumask)spin_lock_irqsave (&desc->lock, flags);#ifdef CONFIG_GENERIC_PENDING_IRQ -if (desc->status &IRQ_MOVE_PCNTXT || desc->status &IRQ_DISABLED ) { -cpumask_copy (desc->affinity, cpumask); +if (desc->status &IRQ_MOVE_PCNTXT ) desc->chip->set_affinity (irq, cpumask); - }else { +else { desc->status |=IRQ_MOVE_PENDING ;cpumask_copy (desc->pending_mask, cpumask); }
Quoted 15 lines · Gary Hade
Impact: Eliminates a race that can leave the system in an
unusable stateDuring rapid offlining of multiple CPUs there is a chance that an IRQ affinity move destination CPU will be offlined before the IRQ affinity move initiated during the offlining of a previous CPU completes. This can happen when the device is not very active and thus fails to generate the IRQ that is needed to complete the IRQ affinity move before the move destination CPU is offlined. When this happens there is an
-EBUSYreturn from__assign_irq_vector()during the offlining of the IRQ move destination CPU which prevents initiation of a new IRQ affinity move operation to an online CPU. This leaves the IRQ affinity set to an offlined CPU.
Quoted 4 lines
I have been able to reproduce the problem on some of our systems using the following script. When the system is idle the problem often reproduces during the first CPU offlining sequence.
Ok. I have had a chance to think through what you your patches are doing and it is assuming the broken logic in cpu_down is correct and patching over some but not all of the problems.
First the problem is not migrating irqs when IRR is set. The general problem is that the state machines in most ioapics are fragile and can get confused if you reprogram them at any point when an irq can come in. In the middle of an interrupt handler is the one time we know interrupts can not come in.
To really fix this problem we need to do two things.
1) Tack when irqs that can not be migrated from process context are on a cpu, and deny cpu hot-unplug.
2) Modify every interrupt that can be safely migrated in interrupt context to migrate irqs in interrupt context so no one encounters this problem in practice.
We can update MSIs and do a pci read to know when the update has made it to a device. Multi MSI is a disaster but I won't go there.
In lowest priority delivery mode when the irq is not changing domain but just changing the set of possible cpus the interrupt can be delivered to.
And then of course all of the fun iommus that remap irqs.
Eric
Quoted 22 lines · Eric W. Biederman
Quoted 15 lines · Gary Hade
Impact: Eliminates a race that can leave the system in an
unusable stateDuring rapid offlining of multiple CPUs there is a chance that an IRQ affinity move destination CPU will be offlined before the IRQ affinity move initiated during the offlining of a previous CPU completes. This can happen when the device is not very active and thus fails to generate the IRQ that is needed to complete the IRQ affinity move before the move destination CPU is offlined. When this happens there is an
-EBUSYreturn from__assign_irq_vector()during the offlining of the IRQ move destination CPU which prevents initiation of a new IRQ affinity move operation to an online CPU. This leaves the IRQ affinity set to an offlined CPU.Quoted 4 lines
I have been able to reproduce the problem on some of our systems using the following script. When the system is idle the problem often reproduces during the first CPU offlining sequence.
Quoted 5 lines
Ok. I have had a chance to think through what you your patches are doing and it is assuming the broken logic in
cpu_downis correct and patching over some but not all of the problems.First the problem is not migrating irqs when IRR is set.
When the device is very active, a printk in __target_IO_APIC_irq() immediately prior to
io_apic_modify(apic, 0x10 + pin*2, reg);
intermittently displays 'reg' values indicating that the Remote IRR bit is set.
With PATCH 3/3 the same printk displays no 'reg' values indicating that the Remote IRR bit is set _and_ the IRQ interruption problem disappears.
This is what led me to very strongly believe that the problem was caused by writing the I/O redirection table register while the Remote IRR bit was set.
Quoted 4 lines
The general
problem is that the state machines in most ioapics are fragile and can get confused if you reprogram them at any point when an irq can come in.
IRQs are masked [from fixup_irqs() when offlining a CPU, from ack_apic_level() when not offlining a CPU] during the reprogramming. Does this not help avoid the issue? Sorry if this is a nieve question.
Quoted 10 lines
In the middle of an interrupt handler is the one time we
know interrupts can not come in.To really fix this problem we need to do two things.
1) Tack when irqs that can not be migrated from process context are on a cpu, and deny cpu hot-unplug.
2) Modify every interrupt that can be safely migrated in interrupt context to migrate irqs in interrupt context so no one encounters this problem in practice.
Quoted 7 lines
We can update MSIs and do a pci read to know when the update has made it to a device. Multi MSI is a disaster but I won't go there.
In lowest priority delivery mode when the irq is not changing domain but just changing the set of possible cpus the interrupt can be delivered to.
And then of course all of the fun iommus that remap irqs.
Sounds non-trivial.
Gary
Signature · 6 lines
Gary Hade System x Enablement IBM Linux Technology Center 503-578-4503 IBM T/L: 775-4503 [email protected] http://www.ibm.com/linux/ltc