x86/x86_64: fix IRQ migration triggered active device IRQ interrruption
PATCH 3/3 x86/x86_64: fix IRQ migration triggered active device IRQ interrruption
Impact: Eliminates an issue that can leave the system in an
unusable state.
This patch addresses an issue where device generated IRQs are no longer seen by the kernel following IRQ affinity migration while the device is generating IRQs at a high rate. We have seen this problem happen when IRQ affinities are adjusted in response to CPU offlining but I believe it could also happen in during user initiated IRQ affinity changes unrelated to CPU offlining. e.g. while the irqbalance daemon is adjusting IRQ affinities when the system is heavily loaded.
I have been able to consistently reproduce the problem on some of our systems by running the following script (VICTIM_IRQ specifies the IRQ for the aic94xx device) while a single instance of the command
# while true; do find / -exec file {} \;; done
is keeping the filesystem activity and IRQ rate reasonably high.#!/bin/sh
SYS_CPU_DIR=/sys/devices/system/cpu
VICTIM_IRQ=25
IRQ_MASK=f0
iteration=0
while true; do
echo $iteration
echo $IRQ_MASK > /proc/irq/$VICTIM_IRQ/smp_affinity for cpudir in $SYS_CPU_DIR/cpu[1-9] $SYS_CPU_DIR/cpu??; do
echo 0 > $cpudir/online
done
for cpudir in $SYS_CPU_DIR/cpu[1-9] $SYS_CPU_DIR/cpu??; do
echo 1 > $cpudir/online
done
iteration=expr $iteration + 1
done
The root cause is a known issue already addressed for some code paths [e.g. ack_apic_level() and the now obsolete migrate_irq_remapped_level_desc()] where the ioapic can misbehave when the I/O redirection table register is written while the Remote IRR bit is set.
The proposed fix uses the same avoidance method and much of same code that the Interrupt Remapping code previously used to avoid the same problem.
--- arch/x86/kernel/apic/io_apic.c | 72 ++++++++++++++++++++++++++++++- 1 file changed, 71 insertions(+), 1 deletion(-)
arch/x86/kernel/apic/io_apic.c +71 -1
@@ -2331,7 +2331,8 @@ set_desc_affinity(struct irq_desc *desc, }static void -set_ioapic_affinity_irq_desc (struct irq_desc *desc,const struct cpumask *mask) +set_ioapic_irq_affinity_desc (struct irq_desc *desc, +const struct cpumask *mask) {struct irq_cfg *cfg;unsigned long flags;
@@ -2352,6 +2353,75 @@ set_ioapic_affinity_irq_desc(struct irq_ }static void +delayed_irq_move (struct work_struct *work) +{ +unsigned int irq; +struct irq_desc *desc; + +for_each_irq_desc (irq, desc) { +if (desc->status &IRQ_MOVE_PENDING ) { +unsigned long flags; + +spin_lock_irqsave (&desc->lock, flags); +if (!desc->chip->set_affinity || + !(desc->status &IRQ_MOVE_PENDING )) { + desc->status &= ~IRQ_MOVE_PENDING ; +spin_unlock_irqrestore (&desc->lock, flags); +continue ; + } + + desc->chip->set_affinity (irq, desc->pending_mask); +spin_unlock_irqrestore (&desc->lock, flags); + } + } +} + +static DECLARE_DELAYED_WORK(delayed_irq_move_work, delayed_irq_move); + +static void +set_ioapic_irq_affinity_level_desc (struct irq_desc *desc) +{ + +struct irq_cfg *cfg = desc->chip_data; + +mask_IO_APIC_irq_desc (desc); + +if (io_apic_level_ack_pending (cfg)) { +/* +* Interrupt in progress. Migrating irq now will change +* the vector information in the IO-APIC RTE which will +* confuse the EOI broadcast performed by cpu. +* So, we delay the irq migration. +*/ +schedule_delayed_work (&delayed_irq_move_work,1 ); + goto unmask; + } + +/* Interrupt not in progress. we can change the vector +* information in the IO-APIC RTE. */ +set_ioapic_irq_affinity_desc (desc, desc->pending_mask); + + desc->status &= ~IRQ_MOVE_PENDING ; +cpumask_clear (desc->pending_mask); + +unmask: +unmask_IO_APIC_irq_desc (desc); +} + +static void +set_ioapic_affinity_irq_desc (struct irq_desc *desc, +const struct cpumask *mask) +{ +if (desc->status &IRQ_LEVEL ) { + desc->status |=IRQ_MOVE_PENDING ; +cpumask_copy (desc->pending_mask, mask); +set_ioapic_irq_affinity_level_desc (desc); +return ; + } +set_ioapic_irq_affinity_desc (desc, mask); +} + +static void set_ioapic_affinity_irq (unsigned int irq,const struct cpumask *mask) {struct irq_desc *desc;
Quoted 142 lines · Gary Hade
Impact: Eliminates an issue that can leave the system in an unusable state.
This patch addresses an issue where device generated IRQs are no longer seen by the kernel following IRQ affinity migration while the device is generating IRQs at a high rate. We have seen this problem happen when IRQ affinities are adjusted in response to CPU offlining but I believe it could also happen in during user initiated IRQ affinity changes unrelated to CPU offlining. e.g. while the irqbalance daemon is adjusting IRQ affinities when the system is heavily loaded.
I have been able to consistently reproduce the problem on some of our systems by running the following script (
VICTIM_IRQspecifies the IRQ for the aic94xx device) while a single instance of the command# while true; do find / -exec file {} \;; done is keeping the filesystem activity and IRQ rate reasonably high.#!/bin/sh
SYS_CPU_DIR=/sys/devices/system/cpu
VICTIM_IRQ=25
IRQ_MASK=f0iteration=0
while true; do
echo $iteration
echo $IRQ_MASK >/proc/irq/$VICTIM_IRQ/smp_affinityfor cpudir in $SYS_CPU_DIR/cpu[1-9] $SYS_CPU_DIR/cpu??; do echo 0 > $cpudir/online
done
for cpudir in $SYS_CPU_DIR/cpu[1-9] $SYS_CPU_DIR/cpu??; do echo 1 > $cpudir/online
done
iteration=expr $iteration + 1
doneThe root cause is a known issue already addressed for some code paths [e.g.
ack_apic_level()and the now obsoletemigrate_irq_remapped_level_desc()] where the ioapic can misbehave when the I/O redirection table register is written while the Remote IRR bit is set.The proposed fix uses the same avoidance method and much of same code that the Interrupt Remapping code previously used to avoid the same problem.
Signed-off-by Gary Hade <[email protected]>---
arch/x86/kernel/apic/io_apic.c | 72 ++++++++++++++++++++++++++++++-1 file changed, 71 insertions(+), 1 deletion(-)Index:
linux-2.6.30-rc1/arch/x86/kernel/apic/io_apic.c
===================================================================arch/x86/kernel/apic/io_apic.c 2009-04-08 09:24:23.000000000 -0700 renamed from arch/x86/kernel/apic/io_apic.c 2009-04-08 09:24:11.000000000 -0700 +0
-0@@ -2331,7 +2331,8 @@ set_desc_affinity(struct irq_desc *desc, }static void
arch/x86/kernel/apic/io_apic.c 2009-04-08 09:24:23.000000000 -0700 renamed from arch/x86/kernel/apic/io_apic.c 2009-04-08 09:24:11.000000000 -0700 +2
-1-set_ioapic_affinity_irq_desc(struct irq_desc *desc, const struct cpumask *mask) +set_ioapic_irq_affinity_desc(struct irq_desc *desc, + const struct cpumask *mask){
structirq_cfg*cfg;
unsigned long flags;arch/x86/kernel/apic/io_apic.c 2009-04-08 09:24:23.000000000 -0700 renamed from arch/x86/kernel/apic/io_apic.c 2009-04-08 09:24:11.000000000 -0700 +0
-0@@ -2352,6 +2353,75 @@ set_ioapic_affinity_irq_desc(struct irq_ }static void
arch/x86/kernel/apic/io_apic.c 2009-04-08 09:24:23.000000000 -0700 renamed from arch/x86/kernel/apic/io_apic.c 2009-04-08 09:24:11.000000000 -0700 +69
-0+delayed_irq_move(struct work_struct *work) +{ + unsigned int irq; + struct irq_desc *desc; + + for_each_irq_desc(irq, desc) { + if (desc->status & IRQ_MOVE_PENDING) { + unsigned long flags; + + spin_lock_irqsave(&desc->lock, flags); + if (!desc->chip->set_affinity || + !(desc->status & IRQ_MOVE_PENDING)) { + desc->status &= ~IRQ_MOVE_PENDING; + spin_unlock_irqrestore(&desc->lock, flags); + continue; + } + + desc->chip->set_affinity(irq, desc->pending_mask); + spin_unlock_irqrestore(&desc->lock, flags); + } + } +} + +static DECLARE_DELAYED_WORK(delayed_irq_move_work, delayed_irq_move); + +static void +set_ioapic_irq_affinity_level_desc(struct irq_desc *desc) +{ + + struct irq_cfg *cfg = desc->chip_data; + + mask_IO_APIC_irq_desc(desc); + + if (io_apic_level_ack_pending(cfg)) { + /* + * Interrupt in progress. Migrating irq now will change + * the vector information in the IO-APIC RTE which will + * confuse the EOI broadcast performed by cpu. + * So, we delay the irq migration. + */ + schedule_delayed_work(&delayed_irq_move_work, 1); + goto unmask; + } + + /* Interrupt not in progress. we can change the vector + * information in the IO-APIC RTE. */ + set_ioapic_irq_affinity_desc(desc, desc->pending_mask); + + desc->status &= ~IRQ_MOVE_PENDING; + cpumask_clear(desc->pending_mask); + +unmask: + unmask_IO_APIC_irq_desc(desc); +} + +static void +set_ioapic_affinity_irq_desc(struct irq_desc *desc, + const struct cpumask *mask) +{ + if (desc->status & IRQ_LEVEL) { + desc->status |= IRQ_MOVE_PENDING; + cpumask_copy(desc->pending_mask, mask); + set_ioapic_irq_affinity_level_desc(desc); + return; + } + set_ioapic_irq_affinity_desc(desc, mask); +} + +static void
Quoted 4 lines
set_ioapic_affinity_irq(unsigned int irq, const struct cpumask *mask) {
structirq_desc*desc;
--
it seems, ack_apic_level() already checked io_apic_level_ack_pending()
cfg = desc->chip_data;
if (!io_apic_level_ack_pending(cfg))
move_masked_irq(irq);
unmask_IO_APIC_irq_desc(desc);also calling path without irremap - for level triggered.
1. user use /proc/irq/xx/smp_affinity to set affinity
2. kernel/irq/manager.c::irq_set_affinity() is called, and desc->pending_mask is changed
3a. one irq is arrived: arch/x86/kernel/irq.c::do_IRQ() is called, hand_irq==>desc->handle_irq
==>kernel/irq/chip.c::handle_fasteoi_irq() ==> call desc->chip->eoi()
3b. desc->chip->eoi() aka arch/x86/kernel/apic/io_apic.c::ack_apic_level() will call mask_IO_APIC_irq_desc() and kernel/irq/migration.c::move_masked_irq() 3c. move_masked_irq will call desc->set_affinity(), vector and target in io_apic controller and IDT are changed
desc->affinity is changed at last. ( for desc moving,
move_in_progress and extra_move flag is set too.)
3d then ack_apic_level() will unmask the irq.
3e. finished other call in handle_fasteoi_irq()
YH
Quoted 150 lines · Yinghai Lu
Quoted 142 lines · Gary Hade
Impact: Eliminates an issue that can leave the system in an unusable state.
This patch addresses an issue where device generated IRQs are no longer seen by the kernel following IRQ affinity migration while the device is generating IRQs at a high rate. We have seen this problem happen when IRQ affinities are adjusted in response to CPU offlining but I believe it could also happen in during user initiated IRQ affinity changes unrelated to CPU offlining. e.g. while the irqbalance daemon is adjusting IRQ affinities when the system is heavily loaded.
I have been able to consistently reproduce the problem on some of our systems by running the following script (
VICTIM_IRQspecifies the IRQ for the aic94xx device) while a single instance of the command# while true; do find / -exec file {} \;; done is keeping the filesystem activity and IRQ rate reasonably high.#!/bin/sh
SYS_CPU_DIR=/sys/devices/system/cpu
VICTIM_IRQ=25
IRQ_MASK=f0iteration=0
while true; do
echo $iteration
echo $IRQ_MASK >/proc/irq/$VICTIM_IRQ/smp_affinityfor cpudir in $SYS_CPU_DIR/cpu[1-9] $SYS_CPU_DIR/cpu??; do echo 0 > $cpudir/online
done
for cpudir in $SYS_CPU_DIR/cpu[1-9] $SYS_CPU_DIR/cpu??; do echo 1 > $cpudir/online
done
iteration=expr $iteration + 1
doneThe root cause is a known issue already addressed for some code paths [e.g.
ack_apic_level()and the now obsoletemigrate_irq_remapped_level_desc()] where the ioapic can misbehave when the I/O redirection table register is written while the Remote IRR bit is set.The proposed fix uses the same avoidance method and much of same code that the Interrupt Remapping code previously used to avoid the same problem.
Signed-off-by Gary Hade <[email protected]>---
arch/x86/kernel/apic/io_apic.c | 72 ++++++++++++++++++++++++++++++-1 file changed, 71 insertions(+), 1 deletion(-)Index:
linux-2.6.30-rc1/arch/x86/kernel/apic/io_apic.c
===================================================================arch/x86/kernel/apic/io_apic.c 2009-04-08 09:24:23.000000000 -0700 renamed from arch/x86/kernel/apic/io_apic.c 2009-04-08 09:24:11.000000000 -0700 +0
-0@@ -2331,7 +2331,8 @@ set_desc_affinity(struct irq_desc *desc, }static void
arch/x86/kernel/apic/io_apic.c 2009-04-08 09:24:23.000000000 -0700 renamed from arch/x86/kernel/apic/io_apic.c 2009-04-08 09:24:11.000000000 -0700 +2
-1-set_ioapic_affinity_irq_desc(struct irq_desc *desc, const struct cpumask *mask) +set_ioapic_irq_affinity_desc(struct irq_desc *desc, + const struct cpumask *mask){
structirq_cfg*cfg;
unsigned long flags;arch/x86/kernel/apic/io_apic.c 2009-04-08 09:24:23.000000000 -0700 renamed from arch/x86/kernel/apic/io_apic.c 2009-04-08 09:24:11.000000000 -0700 +0
-0@@ -2352,6 +2353,75 @@ set_ioapic_affinity_irq_desc(struct irq_ }static void
arch/x86/kernel/apic/io_apic.c 2009-04-08 09:24:23.000000000 -0700 renamed from arch/x86/kernel/apic/io_apic.c 2009-04-08 09:24:11.000000000 -0700 +69
-0+delayed_irq_move(struct work_struct *work) +{ + unsigned int irq; + struct irq_desc *desc; + + for_each_irq_desc(irq, desc) { + if (desc->status & IRQ_MOVE_PENDING) { + unsigned long flags; + + spin_lock_irqsave(&desc->lock, flags); + if (!desc->chip->set_affinity || + !(desc->status & IRQ_MOVE_PENDING)) { + desc->status &= ~IRQ_MOVE_PENDING; + spin_unlock_irqrestore(&desc->lock, flags); + continue; + } + + desc->chip->set_affinity(irq, desc->pending_mask); + spin_unlock_irqrestore(&desc->lock, flags); + } + } +} + +static DECLARE_DELAYED_WORK(delayed_irq_move_work, delayed_irq_move); + +static void +set_ioapic_irq_affinity_level_desc(struct irq_desc *desc) +{ + + struct irq_cfg *cfg = desc->chip_data; + + mask_IO_APIC_irq_desc(desc); + + if (io_apic_level_ack_pending(cfg)) { + /* + * Interrupt in progress. Migrating irq now will change + * the vector information in the IO-APIC RTE which will + * confuse the EOI broadcast performed by cpu. + * So, we delay the irq migration. + */ + schedule_delayed_work(&delayed_irq_move_work, 1); + goto unmask; + } + + /* Interrupt not in progress. we can change the vector + * information in the IO-APIC RTE. */ + set_ioapic_irq_affinity_desc(desc, desc->pending_mask); + + desc->status &= ~IRQ_MOVE_PENDING; + cpumask_clear(desc->pending_mask); + +unmask: + unmask_IO_APIC_irq_desc(desc); +} + +static void +set_ioapic_affinity_irq_desc(struct irq_desc *desc, + const struct cpumask *mask) +{ + if (desc->status & IRQ_LEVEL) { + desc->status |= IRQ_MOVE_PENDING; + cpumask_copy(desc->pending_mask, mask); + set_ioapic_irq_affinity_level_desc(desc); + return; + } + set_ioapic_irq_affinity_desc(desc, mask); +} + +static voidQuoted 4 lines
set_ioapic_affinity_irq(unsigned int irq, const struct cpumask *mask) {
structirq_desc*desc;
--it seems,
ack_apic_level()already checkedio_apic_level_ack_pending()
Quoted 4 lines
cfg = desc->chip_data;
if (!io_apic_level_ack_pending(cfg))
move_masked_irq(irq);
unmask_IO_APIC_irq_desc(desc);Yes, I have actually observed instances where the command
echo $IRQ_MASK > /proc/irq/$VICTIM_IRQ/smp_affinity
in the above test script failed to modify the affinity due to
a debug printk confirmed non-zero return from io_apic_level_ack_pending() at this location. However, this is definitely not catching all the cases. This was confirmed this with a printk in __target_IO_APIC_irq() that would sometimes shows the Remote IRR bit set before the io_apic_modify() call. e.g.__target_IO_APIC_irq: XXX before io_apic_modify irq=25 vector=0xd9 reg=0x1e0d9
Quoted 13 lines
also calling path without irremap - for level triggered.
1. user use/proc/irq/xx/smp_affinityto set affinity
2. kernel/irq/manager.c::irq_set_affinity() is called, anddesc->pending_maskis changed
3a. one irq is arrived: arch/x86/kernel/irq.c::do_IRQ() is called, hand_irq==>desc->handle_irq==>kernel/irq/chip.c::handle_fasteoi_irq() ==> call desc->chip->eoi()3b.
desc->chip->eoi()aka arch/x86/kernel/apic/io_apic.c::ack_apic_level() will callmask_IO_APIC_irq_desc()and kernel/irq/migration.c::move_masked_irq() 3c.move_masked_irqwill calldesc->set_affinity(), vector and target inio_apiccontroller and IDT are changed
Quoted 4 lines
desc->affinity is changed at last. ( for desc moving,
move_in_progressandextra_moveflag is set too.)
3d thenack_apic_level()will unmask the irq.
3e. finished other call inhandle_fasteoi_irq()
Need to think about this. I'll get back to you.
Thanks!
Gary
Signature · 6 lines
Gary Hade System x Enablement IBM Linux Technology Center 503-578-4503 IBM T/L: 775-4503 [email protected] http://www.ibm.com/linux/ltc
Quoted 167 lines · Yinghai Lu
Quoted 142 lines · Gary Hade
Impact: Eliminates an issue that can leave the system in an unusable state.
This patch addresses an issue where device generated IRQs are no longer seen by the kernel following IRQ affinity migration while the device is generating IRQs at a high rate. We have seen this problem happen when IRQ affinities are adjusted in response to CPU offlining but I believe it could also happen in during user initiated IRQ affinity changes unrelated to CPU offlining. e.g. while the irqbalance daemon is adjusting IRQ affinities when the system is heavily loaded.
I have been able to consistently reproduce the problem on some of our systems by running the following script (
VICTIM_IRQspecifies the IRQ for the aic94xx device) while a single instance of the command# while true; do find / -exec file {} \;; done is keeping the filesystem activity and IRQ rate reasonably high.#!/bin/sh
SYS_CPU_DIR=/sys/devices/system/cpu
VICTIM_IRQ=25
IRQ_MASK=f0iteration=0
while true; do
echo $iteration
echo $IRQ_MASK >/proc/irq/$VICTIM_IRQ/smp_affinityfor cpudir in $SYS_CPU_DIR/cpu[1-9] $SYS_CPU_DIR/cpu??; do echo 0 > $cpudir/online
done
for cpudir in $SYS_CPU_DIR/cpu[1-9] $SYS_CPU_DIR/cpu??; do echo 1 > $cpudir/online
done
iteration=expr $iteration + 1
doneThe root cause is a known issue already addressed for some code paths [e.g.
ack_apic_level()and the now obsoletemigrate_irq_remapped_level_desc()] where the ioapic can misbehave when the I/O redirection table register is written while the Remote IRR bit is set.The proposed fix uses the same avoidance method and much of same code that the Interrupt Remapping code previously used to avoid the same problem.
Signed-off-by Gary Hade <[email protected]>---
arch/x86/kernel/apic/io_apic.c | 72 ++++++++++++++++++++++++++++++-1 file changed, 71 insertions(+), 1 deletion(-)Index:
linux-2.6.30-rc1/arch/x86/kernel/apic/io_apic.c
===================================================================arch/x86/kernel/apic/io_apic.c 2009-04-08 09:24:23.000000000 -0700 renamed from arch/x86/kernel/apic/io_apic.c 2009-04-08 09:24:11.000000000 -0700 +0
-0@@ -2331,7 +2331,8 @@ set_desc_affinity(struct irq_desc *desc, }static void
arch/x86/kernel/apic/io_apic.c 2009-04-08 09:24:23.000000000 -0700 renamed from arch/x86/kernel/apic/io_apic.c 2009-04-08 09:24:11.000000000 -0700 +2
-1-set_ioapic_affinity_irq_desc(struct irq_desc *desc, const struct cpumask *mask) +set_ioapic_irq_affinity_desc(struct irq_desc *desc, + const struct cpumask *mask){
structirq_cfg*cfg;
unsigned long flags;arch/x86/kernel/apic/io_apic.c 2009-04-08 09:24:23.000000000 -0700 renamed from arch/x86/kernel/apic/io_apic.c 2009-04-08 09:24:11.000000000 -0700 +0
-0@@ -2352,6 +2353,75 @@ set_ioapic_affinity_irq_desc(struct irq_ }static void
arch/x86/kernel/apic/io_apic.c 2009-04-08 09:24:23.000000000 -0700 renamed from arch/x86/kernel/apic/io_apic.c 2009-04-08 09:24:11.000000000 -0700 +69
-0+delayed_irq_move(struct work_struct *work) +{ + unsigned int irq; + struct irq_desc *desc; + + for_each_irq_desc(irq, desc) { + if (desc->status & IRQ_MOVE_PENDING) { + unsigned long flags; + + spin_lock_irqsave(&desc->lock, flags); + if (!desc->chip->set_affinity || + !(desc->status & IRQ_MOVE_PENDING)) { + desc->status &= ~IRQ_MOVE_PENDING; + spin_unlock_irqrestore(&desc->lock, flags); + continue; + } + + desc->chip->set_affinity(irq, desc->pending_mask); + spin_unlock_irqrestore(&desc->lock, flags); + } + } +} + +static DECLARE_DELAYED_WORK(delayed_irq_move_work, delayed_irq_move); + +static void +set_ioapic_irq_affinity_level_desc(struct irq_desc *desc) +{ + + struct irq_cfg *cfg = desc->chip_data; + + mask_IO_APIC_irq_desc(desc); + + if (io_apic_level_ack_pending(cfg)) { + /* + * Interrupt in progress. Migrating irq now will change + * the vector information in the IO-APIC RTE which will + * confuse the EOI broadcast performed by cpu. + * So, we delay the irq migration. + */ + schedule_delayed_work(&delayed_irq_move_work, 1); + goto unmask; + } + + /* Interrupt not in progress. we can change the vector + * information in the IO-APIC RTE. */ + set_ioapic_irq_affinity_desc(desc, desc->pending_mask); + + desc->status &= ~IRQ_MOVE_PENDING; + cpumask_clear(desc->pending_mask); + +unmask: + unmask_IO_APIC_irq_desc(desc); +} + +static void +set_ioapic_affinity_irq_desc(struct irq_desc *desc, + const struct cpumask *mask) +{ + if (desc->status & IRQ_LEVEL) { + desc->status |= IRQ_MOVE_PENDING; + cpumask_copy(desc->pending_mask, mask); + set_ioapic_irq_affinity_level_desc(desc); + return; + } + set_ioapic_irq_affinity_desc(desc, mask); +} + +static voidQuoted 4 lines
set_ioapic_affinity_irq(unsigned int irq, const struct cpumask *mask) {
structirq_desc*desc;
--it seems,
ack_apic_level()already checkedio_apic_level_ack_pending()cfg = desc->chip_data; if (!io_apic_level_ack_pending(cfg)) move_masked_irq(irq); unmask_IO_APIC_irq_desc(desc);also calling path without irremap - for level triggered.
1. user use/proc/irq/xx/smp_affinityto set affinity
2. kernel/irq/manager.c::irq_set_affinity() is called, anddesc->pending_maskis changed
3a. one irq is arrived: arch/x86/kernel/irq.c::do_IRQ() is called, hand_irq==>desc->handle_irq==>kernel/irq/chip.c::handle_fasteoi_irq() ==> call desc->chip->eoi()3b.
desc->chip->eoi()aka arch/x86/kernel/apic/io_apic.c::ack_apic_level() will callmask_IO_APIC_irq_desc()and kernel/irq/migration.c::move_masked_irq() 3c.move_masked_irqwill calldesc->set_affinity(), vector and target inio_apiccontroller and IDT are changed
Quoted 4 lines
desc->affinity is changed at last. ( for desc moving,
move_in_progressandextra_moveflag is set too.)
3d thenack_apic_level()will unmask the irq.
3e. finished other call inhandle_fasteoi_irq()
I considered possible conflicts between my proposed changes and this code path before I posted the patch. I had concluded that they should play well together but your comment caused me to take a much more detailed look at this code. The additional review (some detail included below) and my testing have not uncovered anything of concern but please let me know if you (or anyone else that is listening) see a something that I may have overlooked.
move_masked_irq() clears the IRQ_MOVE_PENDING desc->status bit before calling the function pointed to by desc->chip->set_affinity [e.g. set_ioapic_affinity_irq()] AND the content of desc->pending_mask is not used after
cpumask_and(desc->affinity, desc->pending_mask, cpu_online_mask);
which precedes the set_affinity function call. So, I believe this provides the necessary separation between the IRQ_MOVE_PENDING/desc->pending_mask useage in move_masked_irq() and the IRQ_MOVE_PENDING/desc->pending_mask useage in the code that the patch adds.
One interesting observation is that since the IRQ remains masked from before the io_apic_level_ack_pending() check in ack_apic_level() until after the set_affinity function call returns it seems that my speculation in the patch header that the problem could happen when an IRQ affinity change is requested from user level on a heavily loaded system is very likely incorrect.
Thanks,
Gary
Signature · 6 lines
Gary Hade System x Enablement IBM Linux Technology Center 503-578-4503 IBM T/L: 775-4503 [email protected] http://www.ibm.com/linux/ltc
Quoted 156 lines · Gary Hade
Quoted 150 lines · Yinghai Lu
Quoted 142 lines · Gary Hade
Impact: Eliminates an issue that can leave the system in an unusable state.
This patch addresses an issue where device generated IRQs are no longer seen by the kernel following IRQ affinity migration while the device is generating IRQs at a high rate. We have seen this problem happen when IRQ affinities are adjusted in response to CPU offlining but I believe it could also happen in during user initiated IRQ affinity changes unrelated to CPU offlining. e.g. while the irqbalance daemon is adjusting IRQ affinities when the system is heavily loaded.
I have been able to consistently reproduce the problem on some of our systems by running the following script (
VICTIM_IRQspecifies the IRQ for the aic94xx device) while a single instance of the command# while true; do find / -exec file {} \;; done is keeping the filesystem activity and IRQ rate reasonably high.#!/bin/sh
SYS_CPU_DIR=/sys/devices/system/cpu
VICTIM_IRQ=25
IRQ_MASK=f0iteration=0
while true; do
echo $iteration
echo $IRQ_MASK >/proc/irq/$VICTIM_IRQ/smp_affinityfor cpudir in $SYS_CPU_DIR/cpu[1-9] $SYS_CPU_DIR/cpu??; do echo 0 > $cpudir/online
done
for cpudir in $SYS_CPU_DIR/cpu[1-9] $SYS_CPU_DIR/cpu??; do echo 1 > $cpudir/online
done
iteration=expr $iteration + 1
doneThe root cause is a known issue already addressed for some code paths [e.g.
ack_apic_level()and the now obsoletemigrate_irq_remapped_level_desc()] where the ioapic can misbehave when the I/O redirection table register is written while the Remote IRR bit is set.The proposed fix uses the same avoidance method and much of same code that the Interrupt Remapping code previously used to avoid the same problem.
Signed-off-by Gary Hade <[email protected]>---
arch/x86/kernel/apic/io_apic.c | 72 ++++++++++++++++++++++++++++++-1 file changed, 71 insertions(+), 1 deletion(-)Index:
linux-2.6.30-rc1/arch/x86/kernel/apic/io_apic.c
===================================================================arch/x86/kernel/apic/io_apic.c 2009-04-08 09:24:23.000000000 -0700 renamed from arch/x86/kernel/apic/io_apic.c 2009-04-08 09:24:11.000000000 -0700 +0
-0@@ -2331,7 +2331,8 @@ set_desc_affinity(struct irq_desc *desc, }static void
arch/x86/kernel/apic/io_apic.c 2009-04-08 09:24:23.000000000 -0700 renamed from arch/x86/kernel/apic/io_apic.c 2009-04-08 09:24:11.000000000 -0700 +2
-1-set_ioapic_affinity_irq_desc(struct irq_desc *desc, const struct cpumask *mask) +set_ioapic_irq_affinity_desc(struct irq_desc *desc, + const struct cpumask *mask){
structirq_cfg*cfg;
unsigned long flags;arch/x86/kernel/apic/io_apic.c 2009-04-08 09:24:23.000000000 -0700 renamed from arch/x86/kernel/apic/io_apic.c 2009-04-08 09:24:11.000000000 -0700 +0
-0@@ -2352,6 +2353,75 @@ set_ioapic_affinity_irq_desc(struct irq_ }static void
arch/x86/kernel/apic/io_apic.c 2009-04-08 09:24:23.000000000 -0700 renamed from arch/x86/kernel/apic/io_apic.c 2009-04-08 09:24:11.000000000 -0700 +69
-0+delayed_irq_move(struct work_struct *work) +{ + unsigned int irq; + struct irq_desc *desc; + + for_each_irq_desc(irq, desc) { + if (desc->status & IRQ_MOVE_PENDING) { + unsigned long flags; + + spin_lock_irqsave(&desc->lock, flags); + if (!desc->chip->set_affinity || + !(desc->status & IRQ_MOVE_PENDING)) { + desc->status &= ~IRQ_MOVE_PENDING; + spin_unlock_irqrestore(&desc->lock, flags); + continue; + } + + desc->chip->set_affinity(irq, desc->pending_mask); + spin_unlock_irqrestore(&desc->lock, flags); + } + } +} + +static DECLARE_DELAYED_WORK(delayed_irq_move_work, delayed_irq_move); + +static void +set_ioapic_irq_affinity_level_desc(struct irq_desc *desc) +{ + + struct irq_cfg *cfg = desc->chip_data; + + mask_IO_APIC_irq_desc(desc); + + if (io_apic_level_ack_pending(cfg)) { + /* + * Interrupt in progress. Migrating irq now will change + * the vector information in the IO-APIC RTE which will + * confuse the EOI broadcast performed by cpu. + * So, we delay the irq migration. + */ + schedule_delayed_work(&delayed_irq_move_work, 1); + goto unmask; + } + + /* Interrupt not in progress. we can change the vector + * information in the IO-APIC RTE. */ + set_ioapic_irq_affinity_desc(desc, desc->pending_mask); + + desc->status &= ~IRQ_MOVE_PENDING; + cpumask_clear(desc->pending_mask); + +unmask: + unmask_IO_APIC_irq_desc(desc); +} + +static void +set_ioapic_affinity_irq_desc(struct irq_desc *desc, + const struct cpumask *mask) +{ + if (desc->status & IRQ_LEVEL) { + desc->status |= IRQ_MOVE_PENDING; + cpumask_copy(desc->pending_mask, mask); + set_ioapic_irq_affinity_level_desc(desc); + return; + } + set_ioapic_irq_affinity_desc(desc, mask); +} + +static voidQuoted 4 lines
set_ioapic_affinity_irq(unsigned int irq, const struct cpumask *mask) {
structirq_desc*desc;
--it seems,
ack_apic_level()already checkedio_apic_level_ack_pending()Quoted 4 lines
cfg =
desc->chip_data;
if (!io_apic_level_ack_pending(cfg))move_masked_irq(irq);unmask_IO_APIC_irq_desc(desc);
Quoted 9 lines
Yes, I have actually observed instances where the command
echo $IRQ_MASK > /proc/irq/$VICTIM_IRQ/smp_affinity
in the above test script failed to modify the affinity due to
a debug printk confirmed non-zero return fromio_apic_level_ack_pending()at this location. However, this is definitely not catching all the cases. This was confirmed this with a printk in__target_IO_APIC_irq()that would sometimes shows the Remote IRR bit set before theio_apic_modify()call. e.g.__target_IO_APIC_irq: XXX beforeio_apic_modifyirq=25 vector=0xd9 reg=0x1e0d9
that is strange, before that irq is masked,
#ifdef CONFIG_GENERIC_PENDING_IRQ
/* If we are moving the irq we need to mask it */
if (unlikely(desc->status & IRQ_MOVE_PENDING)) {
do_unmask_irq = 1;
mask_IO_APIC_irq_desc(desc);
}#endif
YH
Quoted 34 lines
Quoted 18 lines
also calling path without irremap - for level triggered.
1. user use/proc/irq/xx/smp_affinityto set affinity
2. kernel/irq/manager.c::irq_set_affinity() is called, anddesc->pending_maskis changed
3a. one irq is arrived: arch/x86/kernel/irq.c::do_IRQ() is called, hand_irq==>desc->handle_irq ==>kernel/irq/chip.c::handle_fasteoi_irq() ==> calldesc->chip->eoi()3b.desc->chip->eoi()aka arch/x86/kernel/apic/io_apic.c::ack_apic_level() will callmask_IO_APIC_irq_desc()and kernel/irq/migration.c::move_masked_irq() 3c.move_masked_irqwill calldesc->set_affinity(), vector and target inio_apiccontroller and IDT are changed
desc->affinityis changed at last. ( for desc moving,move_in_progressandextra_moveflag is set too.)
3d thenack_apic_level()will unmask the irq.
3e. finished other call inhandle_fasteoi_irq()Need to think about this. I'll get back to you.
Thanks!
Gary
--
Gary Hade
System x Enablement
IBM Linux Technology Center
503-578-4503 IBM T/L: 775-4503
[email protected]
http://www.ibm.com/linux/ltc
Quoted 11 lines · Gary Hade
move_masked_irq()clears theIRQ_MOVE_PENDINGdesc->statusbit before calling the function pointed to bydesc->chip->set_affinity[e.g.set_ioapic_affinity_irq()] AND the content ofdesc->pending_maskis not used aftercpumask_and(desc->affinity, desc->pending_mask, cpu_online_mask); which precedes theset_affinityfunction call. So, I believe this provides the necessary separation between the IRQ_MOVE_PENDING/desc->pending_mask useage inmove_masked_irq()and the IRQ_MOVE_PENDING/desc->pending_mask useage in the code that the patch adds.
that should be ok, they are proteced by spin_lock(&desc->lock)
YH
Quoted 170 lines · Yinghai Lu
Quoted 156 lines · Gary Hade
Quoted 150 lines · Yinghai Lu
Quoted 142 lines · Gary Hade
Impact: Eliminates an issue that can leave the system in an unusable state.
This patch addresses an issue where device generated IRQs are no longer seen by the kernel following IRQ affinity migration while the device is generating IRQs at a high rate. We have seen this problem happen when IRQ affinities are adjusted in response to CPU offlining but I believe it could also happen in during user initiated IRQ affinity changes unrelated to CPU offlining. e.g. while the irqbalance daemon is adjusting IRQ affinities when the system is heavily loaded.
I have been able to consistently reproduce the problem on some of our systems by running the following script (
VICTIM_IRQspecifies the IRQ for the aic94xx device) while a single instance of the command# while true; do find / -exec file {} \;; done is keeping the filesystem activity and IRQ rate reasonably high.#!/bin/sh
SYS_CPU_DIR=/sys/devices/system/cpu
VICTIM_IRQ=25
IRQ_MASK=f0iteration=0
while true; do
echo $iteration
echo $IRQ_MASK >/proc/irq/$VICTIM_IRQ/smp_affinityfor cpudir in $SYS_CPU_DIR/cpu[1-9] $SYS_CPU_DIR/cpu??; do echo 0 > $cpudir/online
done
for cpudir in $SYS_CPU_DIR/cpu[1-9] $SYS_CPU_DIR/cpu??; do echo 1 > $cpudir/online
done
iteration=expr $iteration + 1
doneThe root cause is a known issue already addressed for some code paths [e.g.
ack_apic_level()and the now obsoletemigrate_irq_remapped_level_desc()] where the ioapic can misbehave when the I/O redirection table register is written while the Remote IRR bit is set.The proposed fix uses the same avoidance method and much of same code that the Interrupt Remapping code previously used to avoid the same problem.
Signed-off-by Gary Hade <[email protected]>---
arch/x86/kernel/apic/io_apic.c | 72 ++++++++++++++++++++++++++++++-1 file changed, 71 insertions(+), 1 deletion(-)Index:
linux-2.6.30-rc1/arch/x86/kernel/apic/io_apic.c
===================================================================arch/x86/kernel/apic/io_apic.c 2009-04-08 09:24:23.000000000 -0700 renamed from arch/x86/kernel/apic/io_apic.c 2009-04-08 09:24:11.000000000 -0700 +0
-0@@ -2331,7 +2331,8 @@ set_desc_affinity(struct irq_desc *desc, }static void
arch/x86/kernel/apic/io_apic.c 2009-04-08 09:24:23.000000000 -0700 renamed from arch/x86/kernel/apic/io_apic.c 2009-04-08 09:24:11.000000000 -0700 +2
-1-set_ioapic_affinity_irq_desc(struct irq_desc *desc, const struct cpumask *mask) +set_ioapic_irq_affinity_desc(struct irq_desc *desc, + const struct cpumask *mask){
structirq_cfg*cfg;
unsigned long flags;arch/x86/kernel/apic/io_apic.c 2009-04-08 09:24:23.000000000 -0700 renamed from arch/x86/kernel/apic/io_apic.c 2009-04-08 09:24:11.000000000 -0700 +0
-0@@ -2352,6 +2353,75 @@ set_ioapic_affinity_irq_desc(struct irq_ }static void
arch/x86/kernel/apic/io_apic.c 2009-04-08 09:24:23.000000000 -0700 renamed from arch/x86/kernel/apic/io_apic.c 2009-04-08 09:24:11.000000000 -0700 +69
-0+delayed_irq_move(struct work_struct *work) +{ + unsigned int irq; + struct irq_desc *desc; + + for_each_irq_desc(irq, desc) { + if (desc->status & IRQ_MOVE_PENDING) { + unsigned long flags; + + spin_lock_irqsave(&desc->lock, flags); + if (!desc->chip->set_affinity || + !(desc->status & IRQ_MOVE_PENDING)) { + desc->status &= ~IRQ_MOVE_PENDING; + spin_unlock_irqrestore(&desc->lock, flags); + continue; + } + + desc->chip->set_affinity(irq, desc->pending_mask); + spin_unlock_irqrestore(&desc->lock, flags); + } + } +} + +static DECLARE_DELAYED_WORK(delayed_irq_move_work, delayed_irq_move); + +static void +set_ioapic_irq_affinity_level_desc(struct irq_desc *desc) +{ + + struct irq_cfg *cfg = desc->chip_data; + + mask_IO_APIC_irq_desc(desc); + + if (io_apic_level_ack_pending(cfg)) { + /* + * Interrupt in progress. Migrating irq now will change + * the vector information in the IO-APIC RTE which will + * confuse the EOI broadcast performed by cpu. + * So, we delay the irq migration. + */ + schedule_delayed_work(&delayed_irq_move_work, 1); + goto unmask; + } + + /* Interrupt not in progress. we can change the vector + * information in the IO-APIC RTE. */ + set_ioapic_irq_affinity_desc(desc, desc->pending_mask); + + desc->status &= ~IRQ_MOVE_PENDING; + cpumask_clear(desc->pending_mask); + +unmask: + unmask_IO_APIC_irq_desc(desc); +} + +static void +set_ioapic_affinity_irq_desc(struct irq_desc *desc, + const struct cpumask *mask) +{ + if (desc->status & IRQ_LEVEL) { + desc->status |= IRQ_MOVE_PENDING; + cpumask_copy(desc->pending_mask, mask); + set_ioapic_irq_affinity_level_desc(desc); + return; + } + set_ioapic_irq_affinity_desc(desc, mask); +} + +static voidQuoted 4 lines
set_ioapic_affinity_irq(unsigned int irq, const struct cpumask *mask) {
structirq_desc*desc;
--it seems,
ack_apic_level()already checkedio_apic_level_ack_pending()Quoted 4 lines
cfg =
desc->chip_data;
if (!io_apic_level_ack_pending(cfg))move_masked_irq(irq);unmask_IO_APIC_irq_desc(desc);Quoted 9 lines
Yes, I have actually observed instances where the command
echo $IRQ_MASK > /proc/irq/$VICTIM_IRQ/smp_affinity
in the above test script failed to modify the affinity due to
a debug printk confirmed non-zero return fromio_apic_level_ack_pending()at this location. However, this is definitely not catching all the cases. This was confirmed this with a printk in__target_IO_APIC_irq()that would sometimes shows the Remote IRR bit set before theio_apic_modify()call. e.g.__target_IO_APIC_irq: XXX beforeio_apic_modifyirq=25 vector=0xd9 reg=0x1e0d9that is strange, before that irq is masked,
Quoted 7 lines
#ifdef CONFIG_GENERIC_PENDING_IRQ /* If we are moving the irq we need to mask it */ if (unlikely(desc->status & IRQ_MOVE_PENDING)) { do_unmask_irq = 1; mask_IO_APIC_irq_desc(desc); }#endif
When the device is very actively generating IRQs I suspect that __target_IO_APIC_irq() is sometimes visited (via CPU offline motivated path) before the IRQ is masked at this location. I believe the vulnerability window is actually a bit larger because it spans this mask_IO_APIC_irq_desc() call.
Gary
Signature · 6 lines
Gary Hade System x Enablement IBM Linux Technology Center 503-578-4503 IBM T/L: 775-4503 [email protected] http://www.ibm.com/linux/ltc
Quoted 179 lines · Gary Hade
Quoted 170 lines · Yinghai Lu
Quoted 156 lines · Gary Hade
Quoted 150 lines · Yinghai Lu
Quoted 142 lines · Gary Hade
Impact: Eliminates an issue that can leave the system in an unusable state.
This patch addresses an issue where device generated IRQs are no longer seen by the kernel following IRQ affinity migration while the device is generating IRQs at a high rate. We have seen this problem happen when IRQ affinities are adjusted in response to CPU offlining but I believe it could also happen in during user initiated IRQ affinity changes unrelated to CPU offlining. e.g. while the irqbalance daemon is adjusting IRQ affinities when the system is heavily loaded.
I have been able to consistently reproduce the problem on some of our systems by running the following script (
VICTIM_IRQspecifies the IRQ for the aic94xx device) while a single instance of the command# while true; do find / -exec file {} \;; done is keeping the filesystem activity and IRQ rate reasonably high.#!/bin/sh
SYS_CPU_DIR=/sys/devices/system/cpu
VICTIM_IRQ=25
IRQ_MASK=f0iteration=0
while true; do
echo $iteration
echo $IRQ_MASK >/proc/irq/$VICTIM_IRQ/smp_affinityfor cpudir in $SYS_CPU_DIR/cpu[1-9] $SYS_CPU_DIR/cpu??; do echo 0 > $cpudir/online
done
for cpudir in $SYS_CPU_DIR/cpu[1-9] $SYS_CPU_DIR/cpu??; do echo 1 > $cpudir/online
done
iteration=expr $iteration + 1
doneThe root cause is a known issue already addressed for some code paths [e.g.
ack_apic_level()and the now obsoletemigrate_irq_remapped_level_desc()] where the ioapic can misbehave when the I/O redirection table register is written while the Remote IRR bit is set.The proposed fix uses the same avoidance method and much of same code that the Interrupt Remapping code previously used to avoid the same problem.
Signed-off-by Gary Hade <[email protected]>---
arch/x86/kernel/apic/io_apic.c | 72 ++++++++++++++++++++++++++++++-1 file changed, 71 insertions(+), 1 deletion(-)Index:
linux-2.6.30-rc1/arch/x86/kernel/apic/io_apic.c
===================================================================arch/x86/kernel/apic/io_apic.c 2009-04-08 09:24:23.000000000 -0700 renamed from arch/x86/kernel/apic/io_apic.c 2009-04-08 09:24:11.000000000 -0700 +0
-0@@ -2331,7 +2331,8 @@ set_desc_affinity(struct irq_desc *desc, }static void
arch/x86/kernel/apic/io_apic.c 2009-04-08 09:24:23.000000000 -0700 renamed from arch/x86/kernel/apic/io_apic.c 2009-04-08 09:24:11.000000000 -0700 +2
-1-set_ioapic_affinity_irq_desc(struct irq_desc *desc, const struct cpumask *mask) +set_ioapic_irq_affinity_desc(struct irq_desc *desc, + const struct cpumask *mask){
structirq_cfg*cfg;
unsigned long flags;arch/x86/kernel/apic/io_apic.c 2009-04-08 09:24:23.000000000 -0700 renamed from arch/x86/kernel/apic/io_apic.c 2009-04-08 09:24:11.000000000 -0700 +0
-0@@ -2352,6 +2353,75 @@ set_ioapic_affinity_irq_desc(struct irq_ }static void
arch/x86/kernel/apic/io_apic.c 2009-04-08 09:24:23.000000000 -0700 renamed from arch/x86/kernel/apic/io_apic.c 2009-04-08 09:24:11.000000000 -0700 +69
-0+delayed_irq_move(struct work_struct *work) +{ + unsigned int irq; + struct irq_desc *desc; + + for_each_irq_desc(irq, desc) { + if (desc->status & IRQ_MOVE_PENDING) { + unsigned long flags; + + spin_lock_irqsave(&desc->lock, flags); + if (!desc->chip->set_affinity || + !(desc->status & IRQ_MOVE_PENDING)) { + desc->status &= ~IRQ_MOVE_PENDING; + spin_unlock_irqrestore(&desc->lock, flags); + continue; + } + + desc->chip->set_affinity(irq, desc->pending_mask); + spin_unlock_irqrestore(&desc->lock, flags); + } + } +} + +static DECLARE_DELAYED_WORK(delayed_irq_move_work, delayed_irq_move); + +static void +set_ioapic_irq_affinity_level_desc(struct irq_desc *desc) +{ + + struct irq_cfg *cfg = desc->chip_data; + + mask_IO_APIC_irq_desc(desc); + + if (io_apic_level_ack_pending(cfg)) { + /* + * Interrupt in progress. Migrating irq now will change + * the vector information in the IO-APIC RTE which will + * confuse the EOI broadcast performed by cpu. + * So, we delay the irq migration. + */ + schedule_delayed_work(&delayed_irq_move_work, 1); + goto unmask; + } + + /* Interrupt not in progress. we can change the vector + * information in the IO-APIC RTE. */ + set_ioapic_irq_affinity_desc(desc, desc->pending_mask); + + desc->status &= ~IRQ_MOVE_PENDING; + cpumask_clear(desc->pending_mask); + +unmask: + unmask_IO_APIC_irq_desc(desc); +} + +static void +set_ioapic_affinity_irq_desc(struct irq_desc *desc, + const struct cpumask *mask) +{ + if (desc->status & IRQ_LEVEL) { + desc->status |= IRQ_MOVE_PENDING; + cpumask_copy(desc->pending_mask, mask); + set_ioapic_irq_affinity_level_desc(desc); + return; + } + set_ioapic_irq_affinity_desc(desc, mask); +} + +static voidQuoted 4 lines
set_ioapic_affinity_irq(unsigned int irq, const struct cpumask *mask) {
structirq_desc*desc;
--it seems,
ack_apic_level()already checkedio_apic_level_ack_pending()Quoted 4 lines
cfg =
desc->chip_data;
if (!io_apic_level_ack_pending(cfg))move_masked_irq(irq);unmask_IO_APIC_irq_desc(desc);Quoted 9 lines
Yes, I have actually observed instances where the command
echo $IRQ_MASK > /proc/irq/$VICTIM_IRQ/smp_affinity
in the above test script failed to modify the affinity due to
a debug printk confirmed non-zero return fromio_apic_level_ack_pending()at this location. However, this is definitely not catching all the cases. This was confirmed this with a printk in__target_IO_APIC_irq()that would sometimes shows the Remote IRR bit set before theio_apic_modify()call. e.g.__target_IO_APIC_irq: XXX beforeio_apic_modifyirq=25 vector=0xd9 reg=0x1e0d9that is strange, before that irq is masked,
Quoted 7 lines
#ifdef CONFIG_GENERIC_PENDING_IRQ /* If we are moving the irq we need to mask it */ if (unlikely(desc->status & IRQ_MOVE_PENDING)) { do_unmask_irq = 1; mask_IO_APIC_irq_desc(desc); }#endif
Quoted 5 lines
When the device is very actively generating IRQs I suspect that
__target_IO_APIC_irq()is sometimes visited (via CPU offline motivated path) before the IRQ is masked at this location. I believe the vulnerability window is actually a bit larger because it spans thismask_IO_APIC_irq_desc()call.
I don't think it makes sense to debate how broken fixup_irqs is. The code is fundamentally wrong, if you don't have code that can migrate an irq in process context.
Not playing games when Remote_IRR is set is the easy part. As
that is required on correctly functioning hardware. Unless
your ioapic has the magic acknowledge register.
The hard part is that ioapic to put it politely are fragile things so you have to handle them just so. It is possible to wedge the state machines into a unrecoverable mess on most common ioapic implementations. Not sending the ack when the ioapic is ready to receive it is easy by comparison.
Eric
Quoted 195 lines · Eric W. Biederman
Quoted 179 lines · Gary Hade
Quoted 170 lines · Yinghai Lu
Quoted 156 lines · Gary Hade
Quoted 150 lines · Yinghai Lu
Quoted 142 lines · Gary Hade
Impact: Eliminates an issue that can leave the system in an unusable state.
This patch addresses an issue where device generated IRQs are no longer seen by the kernel following IRQ affinity migration while the device is generating IRQs at a high rate. We have seen this problem happen when IRQ affinities are adjusted in response to CPU offlining but I believe it could also happen in during user initiated IRQ affinity changes unrelated to CPU offlining. e.g. while the irqbalance daemon is adjusting IRQ affinities when the system is heavily loaded.
I have been able to consistently reproduce the problem on some of our systems by running the following script (
VICTIM_IRQspecifies the IRQ for the aic94xx device) while a single instance of the command# while true; do find / -exec file {} \;; done is keeping the filesystem activity and IRQ rate reasonably high.#!/bin/sh
SYS_CPU_DIR=/sys/devices/system/cpu
VICTIM_IRQ=25
IRQ_MASK=f0iteration=0
while true; do
echo $iteration
echo $IRQ_MASK >/proc/irq/$VICTIM_IRQ/smp_affinityfor cpudir in $SYS_CPU_DIR/cpu[1-9] $SYS_CPU_DIR/cpu??; do echo 0 > $cpudir/online
done
for cpudir in $SYS_CPU_DIR/cpu[1-9] $SYS_CPU_DIR/cpu??; do echo 1 > $cpudir/online
done
iteration=expr $iteration + 1
doneThe root cause is a known issue already addressed for some code paths [e.g.
ack_apic_level()and the now obsoletemigrate_irq_remapped_level_desc()] where the ioapic can misbehave when the I/O redirection table register is written while the Remote IRR bit is set.The proposed fix uses the same avoidance method and much of same code that the Interrupt Remapping code previously used to avoid the same problem.
Signed-off-by Gary Hade <[email protected]>---
arch/x86/kernel/apic/io_apic.c | 72 ++++++++++++++++++++++++++++++-1 file changed, 71 insertions(+), 1 deletion(-)Index:
linux-2.6.30-rc1/arch/x86/kernel/apic/io_apic.c
===================================================================arch/x86/kernel/apic/io_apic.c 2009-04-08 09:24:23.000000000 -0700 renamed from arch/x86/kernel/apic/io_apic.c 2009-04-08 09:24:11.000000000 -0700 +0
-0@@ -2331,7 +2331,8 @@ set_desc_affinity(struct irq_desc *desc, }static void
arch/x86/kernel/apic/io_apic.c 2009-04-08 09:24:23.000000000 -0700 renamed from arch/x86/kernel/apic/io_apic.c 2009-04-08 09:24:11.000000000 -0700 +2
-1-set_ioapic_affinity_irq_desc(struct irq_desc *desc, const struct cpumask *mask) +set_ioapic_irq_affinity_desc(struct irq_desc *desc, + const struct cpumask *mask){
structirq_cfg*cfg;
unsigned long flags;arch/x86/kernel/apic/io_apic.c 2009-04-08 09:24:23.000000000 -0700 renamed from arch/x86/kernel/apic/io_apic.c 2009-04-08 09:24:11.000000000 -0700 +0
-0@@ -2352,6 +2353,75 @@ set_ioapic_affinity_irq_desc(struct irq_ }static void
arch/x86/kernel/apic/io_apic.c 2009-04-08 09:24:23.000000000 -0700 renamed from arch/x86/kernel/apic/io_apic.c 2009-04-08 09:24:11.000000000 -0700 +69
-0+delayed_irq_move(struct work_struct *work) +{ + unsigned int irq; + struct irq_desc *desc; + + for_each_irq_desc(irq, desc) { + if (desc->status & IRQ_MOVE_PENDING) { + unsigned long flags; + + spin_lock_irqsave(&desc->lock, flags); + if (!desc->chip->set_affinity || + !(desc->status & IRQ_MOVE_PENDING)) { + desc->status &= ~IRQ_MOVE_PENDING; + spin_unlock_irqrestore(&desc->lock, flags); + continue; + } + + desc->chip->set_affinity(irq, desc->pending_mask); + spin_unlock_irqrestore(&desc->lock, flags); + } + } +} + +static DECLARE_DELAYED_WORK(delayed_irq_move_work, delayed_irq_move); + +static void +set_ioapic_irq_affinity_level_desc(struct irq_desc *desc) +{ + + struct irq_cfg *cfg = desc->chip_data; + + mask_IO_APIC_irq_desc(desc); + + if (io_apic_level_ack_pending(cfg)) { + /* + * Interrupt in progress. Migrating irq now will change + * the vector information in the IO-APIC RTE which will + * confuse the EOI broadcast performed by cpu. + * So, we delay the irq migration. + */ + schedule_delayed_work(&delayed_irq_move_work, 1); + goto unmask; + } + + /* Interrupt not in progress. we can change the vector + * information in the IO-APIC RTE. */ + set_ioapic_irq_affinity_desc(desc, desc->pending_mask); + + desc->status &= ~IRQ_MOVE_PENDING; + cpumask_clear(desc->pending_mask); + +unmask: + unmask_IO_APIC_irq_desc(desc); +} + +static void +set_ioapic_affinity_irq_desc(struct irq_desc *desc, + const struct cpumask *mask) +{ + if (desc->status & IRQ_LEVEL) { + desc->status |= IRQ_MOVE_PENDING; + cpumask_copy(desc->pending_mask, mask); + set_ioapic_irq_affinity_level_desc(desc); + return; + } + set_ioapic_irq_affinity_desc(desc, mask); +} + +static voidQuoted 4 lines
set_ioapic_affinity_irq(unsigned int irq, const struct cpumask *mask) {
structirq_desc*desc;
--it seems,
ack_apic_level()already checkedio_apic_level_ack_pending()Quoted 4 lines
cfg =
desc->chip_data;
if (!io_apic_level_ack_pending(cfg))move_masked_irq(irq);unmask_IO_APIC_irq_desc(desc);Quoted 9 lines
Yes, I have actually observed instances where the command
echo $IRQ_MASK > /proc/irq/$VICTIM_IRQ/smp_affinity
in the above test script failed to modify the affinity due to
a debug printk confirmed non-zero return fromio_apic_level_ack_pending()at this location. However, this is definitely not catching all the cases. This was confirmed this with a printk in__target_IO_APIC_irq()that would sometimes shows the Remote IRR bit set before theio_apic_modify()call. e.g.__target_IO_APIC_irq: XXX beforeio_apic_modifyirq=25 vector=0xd9 reg=0x1e0d9that is strange, before that irq is masked,
Quoted 7 lines
#ifdef CONFIG_GENERIC_PENDING_IRQ /* If we are moving the irq we need to mask it */ if (unlikely(desc->status & IRQ_MOVE_PENDING)) { do_unmask_irq = 1; mask_IO_APIC_irq_desc(desc); }#endif
Quoted 5 lines
When the device is very actively generating IRQs I suspect that
__target_IO_APIC_irq()is sometimes visited (via CPU offline motivated path) before the IRQ is masked at this location. I believe the vulnerability window is actually a bit larger because it spans thismask_IO_APIC_irq_desc()call.I don't think it makes sense to debate how broken
fixup_irqsis. The code is fundamentally wrong, if you don't have code that can migrate an irq in process context.Not playing games when
Remote_IRRis set is the easy part. As
that is required on correctly functioning hardware. Unless
your ioapic has the magic acknowledge register.
Quoted 5 lines
The hard part is that ioapic to put it politely are fragile things so you have to handle them just so. It is possible to wedge the state machines into a unrecoverable mess on most common ioapic implementations. Not sending the ack when the ioapic is ready to receive it is easy by comparison.
The I/O redirection table register write with the remote IRR set issue has reproduced on every IBM System x server I have tried including the x460, x3850, x3550 M2, and x3950 M2. Nobody has responded to my request to test for the presence of this issue on non-IBM systems but I think it is pretty safe to assume that the problem is not IBM specific.
After incorporating the fix that avoids writes to the the I/O redirection table registers while the remote IRR bit is set, I have _not_ observed any other issues that might be related to the ioapic fragility that you mentioned. This of course does not prove that you are wrong and that there is not a need for the changes you are suggesting. However, until someone has the bandwidth to tackle the difficult changes that you suggest, I propose that we continue to repair problems that are found in the current implementation with fixes such as those that I provided.
Thanks,
Gary
Signature · 6 lines
Gary Hade System x Enablement IBM Linux Technology Center 503-578-4503 IBM T/L: 775-4503 [email protected] http://www.ibm.com/linux/ltc
Quoted 6 lines · Gary Hade
The I/O redirection table register write with the remote IRR set issue has reproduced on every IBM System x server I have tried including the x460, x3850, x3550 M2, and x3950 M2. Nobody has responded to my request to test for the presence of this issue on non-IBM systems but I think it is pretty safe to assume that the problem is not IBM specific.
There is no question. The problem can be verified from a simple code inspection. It results from the brokenness of the current fixup_irqs.
Your suggested change at the very least is likely to result in dropped irqs at runtime. Something some drivers do not tolerate well.
Quoted 10 lines
After incorporating the fix that avoids writes to the the I/O redirection table registers while the remote IRR bit is set, I have _not_ observed any other issues that might be related to the ioapic fragility that you mentioned. This of course does not prove that you are wrong and that there is not a need for the changes you are suggesting. However, until someone has the bandwidth to tackle the difficult changes that you suggest, I propose that we continue to repair problems that are found in the current implementation with fixes such as those that I provided.
The changes I suggest are not difficult.
How is APIC routing setup on your hardware? "Setting APIC routing to flat" Is what my laptop reports.
My apologies for asking it badly last time. For understanding what you are testing that is a critical piece of information.
Below is my draft patch that stops cpu shutdown when irqs can not be migrated off. The code to do that is trivial, and is guaranteed to fix all of your lost irq problems.
Beyond that everything I am proposing is very localized.
You have proposed making interrupts behave like they can be migrated in process context when in fact extensive testing over the years have shown in the general case interrupts can only be migrated from the irq handler, from a location where the irqs are naturally disabled.
I propose detecting the cases that we know are safe to migrate in process context, aka logical deliver with less than 8 cpus aka "flat" routing mode and modifying the code so that those work in process context and simply deny cpu hotplug in all of the rest of the cases.
That gives us: Code that always works. An incremental easily testable path. No need to loose sleep at night worrying about the weird failure modes. Ultimately less complex code.
Eric
arch/x86/include/asm/irq.h +8 -0
@@ -47,4 +47,12 @@ extern unsigned int do_IRQ(struct pt_regs *regs); extern DECLARE_BITMAP(used_vectors, NR_VECTORS);extern int vector_used_by_percpu_irq (unsigned int vector); +#ifdef CONFIG_X86_IO_APIC +extern void cleanup_pending_irqs (unsigned int cpu); +#else +static inline void cleanup_pending_irqs (unsigned int cpu) +{ +} +#endif +#endif /* _ASM_X86_IRQ_H */
arch/x86/kernel/apic/io_apic.c +54 -0
57 lines not on this page · raw message
arch/x86/kernel/irq.c +95 -0
98 lines not on this page · raw message
arch/x86/kernel/irq_32.c +0 -45
48 lines not on this page · raw message
arch/x86/kernel/irq_64.c +0 -59
65 lines not on this page · raw message
Quoted 9 lines · Eric W. Biederman
Quoted 6 lines · Gary Hade
The I/O redirection table register write with the remote IRR set issue has reproduced on every IBM System x server I have tried including the x460, x3850, x3550 M2, and x3950 M2. Nobody has responded to my request to test for the presence of this issue on non-IBM systems but I think it is pretty safe to assume that the problem is not IBM specific.
Quoted 6 lines
There is no question. The problem can be verified from a simple code inspection. It results from the brokenness of the current
fixup_irqs.Your suggested change at the very least is likely to result in dropped irqs at runtime.
Which of the two patches (2/3 or 3/3) could cause the dropped IRQs and exactly how can this happen? If you are possibly referring to the concerns about IRQs being migrated in process context during CPU offlining, that design pre-dates my patches.
Quoted 14 lines
Something some drivers do not
tolerate well.Quoted 10 lines
After incorporating the fix that avoids writes to the the I/O redirection table registers while the remote IRR bit is set, I have _not_ observed any other issues that might be related to the ioapic fragility that you mentioned. This of course does not prove that you are wrong and that there is not a need for the changes you are suggesting. However, until someone has the bandwidth to tackle the difficult changes that you suggest, I propose that we continue to repair problems that are found in the current implementation with fixes such as those that I provided.
Quoted 4 lines
The changes I suggest are not difficult.
How is APIC routing setup on your hardware? "Setting APIC routing to flat" Is what my laptop reports.
Oh, thats easy. On the IBM x3550 M2 where I have confirmed that both bugs exist and where I did the below described testing of your patch, the APIC routing is shown as physical flat:
"Setting APIC routing to physical flat"
Quoted 2 lines
My apologies for asking it badly last time.
No problem! Badly probably depends on the audience and I'm probably not a particularily good one. :)
Quoted 6 lines
For understanding what
you are testing that is a critical piece of information.Below is my draft patch that stops cpu shutdown when irqs can not be migrated off. The code to do that is trivial, and is guaranteed to fix all of your lost irq problems.
This didn't help. Using 2.6.30-rc3 plus your patch both bugs are unfortunately still present.
With respect to the race fixed by patch 2/3 that occurs when the device is not very active I have found that a printk in __assign_irq_vector() showing -EBUSY returns is a reliable indicator of when things go bad and the CPU handling the IRQ is offlined without the affinity being migrated to an online CPU. The following was logged during the failure.
__assign_irq_vector: XXX irq=16 returning -EBUSY __assign_irq_vector: XXX irq=16 returning -EBUSY __assign_irq_vector: XXX irq=16 returning -EBUSY __assign_irq_vector: XXX irq=16 returning -EBUSY __assign_irq_vector: XXX irq=16 returning -EBUSY __assign_irq_vector: XXX irq=16 returning -EBUSY __assign_irq_vector: XXX irq=16 returning -EBUSY __assign_irq_vector: XXX irq=16 returning -EBUSY __assign_irq_vector: XXX irq=16 returning -EBUSY __assign_irq_vector: XXX irq=16 returning -EBUSY __assign_irq_vector: XXX irq=16 returning -EBUSY
I am not sure if I totally understand what your new cleanup_pending_irqs() is doing but I *think* it is only broadcasting an IPI when the same IRQ is being handled on the IRQ move destination CPU. If this understanding is correct, I don't think it covers the case where the device is generating IRQs at such an extremely slow rate that absolutely none are received between the time that the move_in_progress flag was set to 1 and the IRQ move destination CPU is offlined. I actually played around with broadcasting an IPI every time the move_in_progress flag was set to 1 which repaired the problem but did not feel like a good solution especially after I discovered that there was no need to even set move_in_progress to 1 if all CPUs in the old domain were going to be offline when the cleanup code was run i.e. cleanup code does nothing when it eventually runs.
I then incorporated my fix (patch 2/3) for the above issue plus a printk in __target_IO_APIC_irq() to display values written to the I/O redirection table register for the IRQ of interest and tried again. With a low incoming IRQ rate the above problem no longer reproduced but after increasing the incoming IRQ rate I started seeing an increasing number of writes to the I/O redirection table register while the remote IRR bit was set (see below) before IRQs from the device were no longer being seen by the kernel.
Quoted 7 lines
Beyond that everything I am proposing is very localized.
You have proposed making interrupts behave like they can be migrated in process context when in fact extensive testing over the years have shown in the general case interrupts can only be migrated from the irq handler, from a location where the irqs are naturally disabled.
To be clear, IRQs were already being migrated in process context during CPU offlining before I even looked at the code. This was not my doing. I believe it has been this way for quite some time.
Quoted 5 lines
I propose detecting the cases that we know are safe to migrate in process context, aka logical deliver with less than 8 cpus aka "flat" routing mode and modifying the code so that those work in process context and simply deny cpu hotplug in all of the rest of the cases.
Humm, are you suggesting that CPU offlining/onlining would not be possible at all on systems with >8 logical CPUs (i.e. most of our systems) or would this just force users to separately migrate IRQ affinities away from a CPU (e.g. by shutting down the irqbalance daemon and writing to /proc/irq/<irq>/smp_affinity) before attempting to offline it?
Thanks,
Gary
--
Gary Hade
System x Enablement
IBM Linux Technology Center
503-578-4503 IBM T/L: 775-4503
[email protected]
http://www.ibm.com/linux/ltc
ioc0: LSISAS1ities={Initiatorities={Initiator}IC_irq: XXX irqPIC_irq: XXX irq=y with reg=0xa0fy with reg=0xa061:00.0: setting01:00.0: setting 64 melm3b55:~ # m__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a069__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a06a__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a06a__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a06a__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a072__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a07a__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a082__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a08a__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a08a__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a08a__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a08a__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a08a__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a08a__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a08a__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a08a__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a08a__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a08a__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a092__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a092__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a092__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a09a__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0a2__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0aa__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0b2__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0b2__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0b2__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0b2__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0b2__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0b2__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0b2__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0b2__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0b2__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0ba__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0ba__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0ba__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0ba__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0c2__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0ca__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0d2__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0da__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0da__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0da__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0da__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0da__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0da__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0da__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0da__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0da__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0e2__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0e2__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0e2__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0e2__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0ea__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a043__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a04b__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a053__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a053__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a053__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a053__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a053__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a053__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a053__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a053__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a053__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a05b__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e05b__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a05b__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a05b__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a063__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a06b__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a073__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a07b__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a07b__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a07b__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a07b__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a07b__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a07b__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a07b__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a07b__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a07b__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a083__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a083__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a083__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a083__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a08b__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a093__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a09b__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0a3__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0a3__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0a3__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0a3__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0a3__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0a3__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0a3__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0a3__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0a3__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0ab__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0ab__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0ab__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0ab__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0b3__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0bb__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0c3__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0cb__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0cb__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0cb__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0cb__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0cb__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0cb__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0cb__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0cb__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0cb__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0d3__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0d3__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0d3__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0d3__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0db__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0e3__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0eb__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a044__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a044__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a044__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a044__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a044__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a044__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e044__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a044__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a044__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a04c__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a04c__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a04c__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a04c__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a054__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a05c__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a064__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a06c__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e06c__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e06c__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a06c__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e06c__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e06c__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e06c__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a06c__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a06c__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a074__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a074__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a074__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a074__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a07c__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a084__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a08c__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a094__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a094__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a094__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a094__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a094__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a094__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a094__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a094__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a094__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a09c__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e09c__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a09c__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e09c__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0a4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0ac__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0b4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0bc__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0bc__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0bc__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0bc__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0bc__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0bc__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0bc__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0bc__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0bc__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0cc__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0d4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0dc__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0e4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0e4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0e4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0e4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0e4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0e4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0e4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0e4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0e4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0ec__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0ec__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0ec__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0ec__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a045__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a04d__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a055__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a05d__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e05d__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a05d__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a05d__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a05d__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a05d__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a05d__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a05d__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a05d__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a065__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a065__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a065__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e065__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a06d__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a075__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a07d__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a085__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a085__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e085__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a085__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e085__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a085__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a085__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a085__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e085__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a08d__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a08d__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e08d__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a08d__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a095__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a09d__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0a5__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0ad__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0ad__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0ad__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0ad__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0ad__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0ad__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0ad__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0ad__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0ad__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0b5__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0b5__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0b5__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0b5__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0bd__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0c5__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0cd__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0d5__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0d5__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0d5__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0d5__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0d5__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0d5__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0d5__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0d5__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0d5__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0dd__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0dd__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0dd__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0dd__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0e5__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a046__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a04e__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a056__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e056__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a056__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e056__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a056__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a056__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a056__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a056__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a056__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a05e__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a05e__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a05e__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a05e__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a066__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a06e__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a076__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a07e__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e07e__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a07e__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a07e__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a07e__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a07e__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e07e__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a07e__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e07e__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a086__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a086__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a086__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a086__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a08e__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a096__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a09e__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0a6__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0a6__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0a6__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0a6__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0a6__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0a6__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0a6__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0a6__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0a6__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0ae__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0ae__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0ae__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0ae__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0b6__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0be__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0c6__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0ce__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0ce__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0ce__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0ce__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0ce__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0ce__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0ce__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0ce__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0ce__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0d6__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0d6__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0d6__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0d6__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0de__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0e6__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a047__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a04f__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a04f__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a04f__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a04f__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a04f__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a04f__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a04f__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a04f__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a04f__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a057__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a057__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a057__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a057__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a05f__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a067__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a06f__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a077__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a077__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a077__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a077__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a077__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a077__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a077__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a077__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a077__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a07f__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e07f__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a07f__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a07f__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a087__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a08f__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a097__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a09f__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a09f__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a09f__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a09f__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a09f__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a09f__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a09f__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a09f__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a09f__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0a7__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0a7__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0a7__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0a7__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0af__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0b7__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0bf__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0c7__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0c7__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0c7__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0c7__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0c7__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0c7__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0c7__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0c7__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0c7__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0cf__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0cf__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0cf__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0cf__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0d7__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0df__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0e7__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a048__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a048__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a048__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a048__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a048__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a048__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a048__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a048__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a048__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a050__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a050__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a050__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a050__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a058__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a060__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a068__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a070__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a070__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a070__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a070__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a070__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a070__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a070__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a070__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a070__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a078__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a078__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a078__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a078__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a088__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a090__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a098__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0a0__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0a0__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0a0__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0a0__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0a0__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0a0__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0a0__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0a0__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0a0__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0a8__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0a8__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0a8__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0a8__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0b0__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0b8__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0c0__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0c8__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0c8__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0c8__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0c8__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c8__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0c8__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0c8__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0c8__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0c8__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0d0__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0d0__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0d0__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0d0__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0d8__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0e0__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0e8__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a041__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a041__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a041__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a041__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a041__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a041__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a041__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a041__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a041__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a049__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a049__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a049__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a049__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a051__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a059__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a061__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a069__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a069__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a069__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a069__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a069__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a069__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a069__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a069__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a069__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a071__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e071__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a071__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a071__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a079__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a081__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a089__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a06a__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a06a__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a06a__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a06a__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a06a__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a06a__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a06a__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e06a__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a06a__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a072__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a072__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a072__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a072__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a07a__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a082__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a08a__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a092__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a092__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a092__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a092__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a092__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e092__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a092__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a092__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a092__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a09a__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a09a__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a09a__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a09a__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0a2__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0aa__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0b2__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0ba__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0ba__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0ba__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0ba__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0ba__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0ba__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0ba__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0ba__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0ba__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0c2__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0c2__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c2__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0c2__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0ca__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0d2__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0da__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0e2__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0e2__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0e2__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0e2__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0e2__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0e2__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0e2__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0e2__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0e2__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0ea__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0ea__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0ea__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0ea__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a043__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a04b__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a053__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a05b__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a05b__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a05b__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a05b__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a05b__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a05b__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a05b__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a05b__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e05b__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a063__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e063__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e063__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a063__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a06b__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a073__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a07b__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a083__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e083__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a083__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a083__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a083__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a083__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a083__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a083__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a083__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a08b__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a08b__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a08b__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a08b__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a093__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a09b__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0a3__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0ab__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0ab__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0ab__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0ab__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0ab__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0ab__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0ab__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0ab__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0ab__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0b3__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0b3__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0b3__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0b3__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0bb__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0c3__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0cb__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0d3__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0d3__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0d3__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0d3__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0d3__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0d3__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0d3__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0d3__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0d3__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0db__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0db__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0db__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0db__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0e3__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0eb__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a044__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a04c__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a04c__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a04c__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e04c__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a04c__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a04c__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e04c__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a04c__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e04c__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a054__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e054__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a054__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a054__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a05c__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a064__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a06c__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a074__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a074__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a074__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a074__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a074__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a074__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e074__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a074__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a074__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a07c__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a07c__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a07c__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a07c__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a084__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a08c__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a094__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a09c__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a09c__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a09c__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a09c__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a09c__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a09c__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a09c__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a09c__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a09c__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0a4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0a4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1a0a4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0a4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0ac__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0b4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0bc__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4__target_IO_APIC_irq: XXX irq=16 io_apic_modify with reg=0x1e0c4
Quoted 14 lines · Gary Hade
Quoted 13 lines · Eric W. Biederman
Quoted 6 lines · Gary Hade
The I/O redirection table register write with the remote IRR set issue has reproduced on every IBM System x server I have tried including the x460, x3850, x3550 M2, and x3950 M2. Nobody has responded to my request to test for the presence of this issue on non-IBM systems but I think it is pretty safe to assume that the problem is not IBM specific.
There is no question. The problem can be verified from a simple code inspection. It results from the brokenness of the current
fixup_irqs.
Quoted 7 lines
Quoted 2 lines
Your suggested change at the very least is likely to result in dropped irqs at runtime.
Which of the two patches (2/3 or 3/3) could cause the dropped IRQs and exactly how can this happen? If you are possibly referring to the concerns about IRQs being migrated in process context during CPU offlining, that design pre-dates my patches.
I am referring to some of the side effects of a cpu being migrated in process context during CPU offlining. That code has been fundamentally broken since it was merged in 2005. Various changes have changed the odds of how likely it is to fail, and how likely people are to encounter problems.
The practice of masking an irq while it is actively being used and we don't have that irq software pending means we will drop that irq at some point, and that is what the code in irq_64.c:fixup_irqs does today.
Quoted 1 line
From what I can tell your patches simply apply more code dubious
code to the fixup_irqs path.
Quoted 33 lines
Quoted 14 lines
Something some drivers do not
tolerate well.Quoted 10 lines
After incorporating the fix that avoids writes to the the I/O redirection table registers while the remote IRR bit is set, I have _not_ observed any other issues that might be related to the ioapic fragility that you mentioned. This of course does not prove that you are wrong and that there is not a need for the changes you are suggesting. However, until someone has the bandwidth to tackle the difficult changes that you suggest, I propose that we continue to repair problems that are found in the current implementation with fixes such as those that I provided.
Quoted 4 lines
The changes I suggest are not difficult.
How is APIC routing setup on your hardware? "Setting APIC routing to flat" Is what my laptop reports.
Oh, thats easy. On the IBM x3550 M2 where I have confirmed that both bugs exist and where I did the below described testing of your patch, the APIC routing is shown as physical flat:
"Setting APIC routing to physical flat"Quoted 2 lines
My apologies for asking it badly last time.
No problem! Badly probably depends on the audience and I'm probably not a particularily good one. :)
Quoted 3 lines
For understanding what
you are testing that is a critical piece of information.
Quoted 6 lines
Quoted 3 lines
Below is my draft patch that stops cpu shutdown when irqs can not be migrated off. The code to do that is trivial, and is guaranteed to fix all of your lost irq problems.
This didn't help. Using 2.6.30-rc3 plus your patch both bugs are unfortunately still present.
You could offline the cpus? I know when I tested it on my laptop I could not offline the cpus.
If you can offline cpus because your irqs have IRQ_MOVE_PCNT set, then you must have an iommu and the work that needs to be done to get things stable on your system is different.
Quoted 9 lines
With respect to the race fixed by patch 2/3 that occurs when the device is not very active I have found that a printk in
__assign_irq_vector()showing-EBUSYreturns is a reliable indicator of when things go bad and the CPU handling the IRQ is offlined without the affinity being migrated to an online CPU. The following was logged during the failure.
__assign_irq_vector: XXX irq=16 returning-EBUSY__assign_irq_vector: XXX irq=16 returning-EBUSY__assign_irq_vector: XXX irq=16 returning-EBUSY
Quoted 8 lines
__assign_irq_vector: XXX irq=16 returning-EBUSY__assign_irq_vector: XXX irq=16 returning-EBUSY__assign_irq_vector: XXX irq=16 returning-EBUSY__assign_irq_vector: XXX irq=16 returning-EBUSY__assign_irq_vector: XXX irq=16 returning-EBUSY__assign_irq_vector: XXX irq=16 returning-EBUSY__assign_irq_vector: XXX irq=16 returning-EBUSY__assign_irq_vector: XXX irq=16 returning-EBUSY
I have not been looking closely at your reported bugs. So far everything seems to be related to the totally broken moving irqs in process context code in fixup_irqs, and making the races smaller does not interest me when it looks like we can make the races go away.
Quoted 4 lines
I am not sure if I totally understand what your new
cleanup_pending_irqs()is doing but I *think* it is only broadcasting an IPI when the same IRQ is being handled on the IRQ move destination CPU.
My new cleanup_pending_irqs() as written still has a number of bugs, it is more of a placeholder than solid tested code at the moment.
cleanup_pending_irqs() is supposed to happen after a synchronous irq migration (so no irqs should be hitting the cpu) and as such it should be possible to cleanup any irq pending state on the current cpu, and if an irq is pending to simply redirect it to some other cpu that still has the irq active.
cleanup_pending_irqs() currently does not have the code to ack the irqs that are pending in the irr, which could have a correctness consequence with the hardware. Beyond that it is just a matter of reflecting the irq to a different cpu (so we don't miss one).
Quoted 4 lines
If this understanding
is correct, I don't think it covers the case where the device is generating IRQs at such an extremely slow rate that absolutely none are received between the time that the
Quoted 8 lines
move_in_progressflag was set to 1 and the IRQ move destination CPU is offlined. I actually played around with broadcasting an IPI every time themove_in_progressflag was set to 1 which repaired the problem but did not feel like a good solution especially after I discovered that there was no need to even setmove_in_progressto 1 if all CPUs in the old domain were going to be offline when the cleanup code was run i.e. cleanup code does nothing when it eventually runs.
That case should be handled simply by denying cpu down event. And that code was working when I tested it on my laptop.
I am wondering how you managed to get the cpu to go down.
Quoted 18 lines
I then incorporated my fix (patch 2/3) for the above issue plus a printk in
__target_IO_APIC_irq()to display values written to the I/O redirection table register for the IRQ of interest and tried again. With a low incoming IRQ rate the above problem no longer reproduced but after increasing the incoming IRQ rate I started seeing an increasing number of writes to the I/O redirection table register while the remote IRR bit was set (see below) before IRQs from the device were no longer being seen by the kernel.Quoted 7 lines
Beyond that everything I am proposing is very localized.
You have proposed making interrupts behave like they can be migrated in process context when in fact extensive testing over the years have shown in the general case interrupts can only be migrated from the irq handler, from a location where the irqs are naturally disabled.
Quoted 4 lines
To be clear, IRQs were already being migrated in process context during CPU offlining before I even looked at the code. This was not my doing. I believe it has been this way for quite some time.
Yep. It doesn't mean it has ever been correct, and I am trying to development in ways so that we only take down a cpu when it is possible to migrate all of it's irqs in process context.
Quoted 11 lines
Quoted 4 lines
I propose detecting thpe cases that we know are safe to migrate in process context, aka logical deliver with less than 8 cpus aka "flat" routing mode and modifying the code so that those work in process context and simply deny cpu hotplug in all of the rest of the cases.
Humm, are you suggesting that CPU offlining/onlining would not be possible at all on systems with >8 logical CPUs (i.e. most of our systems) or would this just force users to separately migrate IRQ affinities away from a CPU (e.g. by shutting down the irqbalance daemon and writing to /proc/irq/<irq>/smp_affinity) before attempting to offline it?
A separate migration, for those hard to handle irqs.
The newest systems have iommus that irqs go through or are using MSIs for the important irqs, and as such can be migrated in process context. So this is not a restriction for future systems.
Eric
Quoted 24 lines · Eric W. Biederman
Quoted 14 lines · Gary Hade
Quoted 13 lines · Eric W. Biederman
Quoted 6 lines · Gary Hade
The I/O redirection table register write with the remote IRR set issue has reproduced on every IBM System x server I have tried including the x460, x3850, x3550 M2, and x3950 M2. Nobody has responded to my request to test for the presence of this issue on non-IBM systems but I think it is pretty safe to assume that the problem is not IBM specific.
There is no question. The problem can be verified from a simple code inspection. It results from the brokenness of the current
fixup_irqs.Quoted 7 lines
Quoted 2 lines
Your suggested change at the very least is likely to result in dropped irqs at runtime.
Which of the two patches (2/3 or 3/3) could cause the dropped IRQs and exactly how can this happen? If you are possibly referring to the concerns about IRQs being migrated in process context during CPU offlining, that design pre-dates my patches.
Quoted 5 lines
I am referring to some of the side effects of a cpu being migrated in process context during CPU offlining. That code has been fundamentally broken since it was merged in 2005. Various changes have changed the odds of how likely it is to fail, and how likely people are to encounter problems.
IMHO, my proposed fixes clearly reduce the odds of failure.
Quoted 8 lines
The practice of masking an irq while it is actively being used and we don't have that irq software pending means we will drop that irq at some point, and that is what the code in irq_64.c:fixup_irqs does today.
Quoted 1 line
From what I can tell your patches simply apply more code dubious
code to the
fixup_irqspath.
Well, they also fix two insidious problems that can render the system unuseable.
Quoted 38 lines
Quoted 38 lines
Quoted 15 lines
Something some drivers do not
tolerate well.Quoted 10 lines
After incorporating the fix that avoids writes to the the I/O redirection table registers while the remote IRR bit is set, I have _not_ observed any other issues that might be related to the ioapic fragility that you mentioned. This of course does not prove that you are wrong and that there is not a need for the changes you are suggesting. However, until someone has the bandwidth to tackle the difficult changes that you suggest, I propose that we continue to repair problems that are found in the current implementation with fixes such as those that I provided.
Quoted 4 lines
The changes I suggest are not difficult.
How is APIC routing setup on your hardware? "Setting APIC routing to flat" Is what my laptop reports.
Oh, thats easy. On the IBM x3550 M2 where I have confirmed that both bugs exist and where I did the below described testing of your patch, the APIC routing is shown as physical flat:
"Setting APIC routing to physical flat"Quoted 2 lines
My apologies for asking it badly last time.
No problem! Badly probably depends on the audience and I'm probably not a particularily good one. :)
Quoted 6 lines
For understanding what
you are testing that is a critical piece of information.Below is my draft patch that stops cpu shutdown when irqs can not be migrated off. The code to do that is trivial, and is guaranteed to fix all of your lost irq problems.
Quoted 5 lines
Quoted 2 lines
This didn't help. Using 2.6.30-rc3 plus your patch both bugs are unfortunately still present.
You could offline the cpus? I know when I tested it on my laptop I could not offline the cpus.
Eric, I'm sorry! This was due to my stupid mistake. When I went to apply your patch I included --dry-run to test it but apparently got distracted and never actually ran patch(1) without --dry-run. <SIGH>
So, I just rebuilt after _really_ applying the patch and got the following result which probably to be what you intended.
elm3b55:/home/garyh/kernel # cat ./cpus_offline_all.sh #!/bin/sh
##
# Offline all offlineable CPUs
##
SYS_CPU_DIR=/sys/devices/system/cpu
for d in $SYS_CPU_DIR/cpu[1-9] $SYS_CPU_DIR/cpu??; do
cpu="basename $d"
state=cat $d/online
if [ "$state" = 1 ]; then
echo $cpu: offlining
echo 0 > $d/online
else
echo $cpu: already offline
fi
done
elm3b55:/home/garyh/kernel # ./cpus_offline_all.sh
cpu1: offlining
./cpus_offline_all.sh: line 14: echo: write error: Invalid argument
cpu2: offlining
./cpus_offline_all.sh: line 14: echo: write error: Invalid argument
cpu3: offlining
./cpus_offline_all.sh: line 14: echo: write error: Invalid argument
cpu4: offlining
./cpus_offline_all.sh: line 14: echo: write error: Invalid argument
cpu5: offlining
./cpus_offline_all.sh: line 14: echo: write error: Invalid argument
cpu6: offlining
./cpus_offline_all.sh: line 14: echo: write error: Invalid argument
cpu7: offlining
./cpus_offline_all.sh: line 14: echo: write error: Invalid argument
cpu8: offlining
./cpus_offline_all.sh: line 14: echo: write error: Invalid argument
cpu9: offlining
./cpus_offline_all.sh: line 14: echo: write error: Invalid argument
cpu10: offlining
./cpus_offline_all.sh: line 14: echo: write error: Invalid argument
cpu11: offlining
./cpus_offline_all.sh: line 14: echo: write error: Invalid argument
cpu12: offlining
./cpus_offline_all.sh: line 14: echo: write error: Invalid argument
cpu13: offlining
./cpus_offline_all.sh: line 14: echo: write error: Invalid argument
cpu14: offlining
./cpus_offline_all.sh: line 14: echo: write error: Invalid argument
cpu15: offlining
./cpus_offline_all.sh: line 14: echo: write error: Invalid argument
Quoted 105 lines
If you can offline cpus because your irqs have
IRQ_MOVE_PCNTset, then you must have an iommu and the work that needs to be done to get things stable on your system is different.Quoted 9 lines
With respect to the race fixed by patch 2/3 that occurs when the device is not very active I have found that a printk in
__assign_irq_vector()showing-EBUSYreturns is a reliable indicator of when things go bad and the CPU handling the IRQ is offlined without the affinity being migrated to an online CPU. The following was logged during the failure.
__assign_irq_vector: XXX irq=16 returning-EBUSY__assign_irq_vector: XXX irq=16 returning-EBUSY__assign_irq_vector: XXX irq=16 returning-EBUSYQuoted 8 lines
__assign_irq_vector: XXX irq=16 returning-EBUSY__assign_irq_vector: XXX irq=16 returning-EBUSY__assign_irq_vector: XXX irq=16 returning-EBUSY__assign_irq_vector: XXX irq=16 returning-EBUSY__assign_irq_vector: XXX irq=16 returning-EBUSY__assign_irq_vector: XXX irq=16 returning-EBUSY__assign_irq_vector: XXX irq=16 returning-EBUSY__assign_irq_vector: XXX irq=16 returning-EBUSYI have not been looking closely at your reported bugs. So far everything seems to be related to the totally broken moving irqs in process context code in
fixup_irqs, and making the races smaller does not interest me when it looks like we can make the races go away.Quoted 4 lines
I am not sure if I totally understand what your new
cleanup_pending_irqs()is doing but I *think* it is only broadcasting an IPI when the same IRQ is being handled on the IRQ move destination CPU.My new
cleanup_pending_irqs()as written still has a number of bugs, it is more of a placeholder than solid tested code at the moment.
cleanup_pending_irqs()is supposed to happen after a synchronous irq migration (so no irqs should be hitting the cpu) and as such it should be possible to cleanup any irq pending state on the current cpu, and if an irq is pending to simply redirect it to some other cpu that still has the irq active.
cleanup_pending_irqs()currently does not have the code to ack the irqs that are pending in the irr, which could have a correctness consequence with the hardware. Beyond that it is just a matter of reflecting the irq to a different cpu (so we don't miss one).Quoted 4 lines
If this understanding
is correct, I don't think it covers the case where the device is generating IRQs at such an extremely slow rate that absolutely none are received between the time that theQuoted 8 lines
move_in_progressflag was set to 1 and the IRQ move destination CPU is offlined. I actually played around with broadcasting an IPI every time themove_in_progressflag was set to 1 which repaired the problem but did not feel like a good solution especially after I discovered that there was no need to even setmove_in_progressto 1 if all CPUs in the old domain were going to be offline when the cleanup code was run i.e. cleanup code does nothing when it eventually runs.That case should be handled simply by denying cpu down event. And that code was working when I tested it on my laptop.
I am wondering how you managed to get the cpu to go down.
Quoted 18 lines
I then incorporated my fix (patch 2/3) for the above issue plus a printk in
__target_IO_APIC_irq()to display values written to the I/O redirection table register for the IRQ of interest and tried again. With a low incoming IRQ rate the above problem no longer reproduced but after increasing the incoming IRQ rate I started seeing an increasing number of writes to the I/O redirection table register while the remote IRR bit was set (see below) before IRQs from the device were no longer being seen by the kernel.Quoted 7 lines
Beyond that everything I am proposing is very localized.
You have proposed making interrupts behave like they can be migrated in process context when in fact extensive testing over the years have shown in the general case interrupts can only be migrated from the irq handler, from a location where the irqs are naturally disabled.
Quoted 4 lines
To be clear, IRQs were already being migrated in process context during CPU offlining before I even looked at the code. This was not my doing. I believe it has been this way for quite some time.
Yep. It doesn't mean it has ever been correct, and I am trying to development in ways so that we only take down a cpu when it is possible to migrate all of it's irqs in process context.
Quoted 11 lines
Quoted 4 lines
I propose detecting thpe cases that we know are safe to migrate in process context, aka logical deliver with less than 8 cpus aka "flat" routing mode and modifying the code so that those work in process context and simply deny cpu hotplug in all of the rest of the cases.
Humm, are you suggesting that CPU offlining/onlining would not be possible at all on systems with >8 logical CPUs (i.e. most of our systems) or would this just force users to separately migrate IRQ affinities away from a CPU (e.g. by shutting down the irqbalance daemon and writing to /proc/irq/<irq>/smp_affinity) before attempting to offline it?
Quoted 5 lines
A separate migration, for those hard to handle irqs.
The newest systems have iommus that irqs go through or are using MSIs for the important irqs, and as such can be migrated in process context. So this is not a restriction for future systems.
I understand your concerns but we need a solution for the earlier systems that does NOT remove or cripple the existing CPU hotplug functionality. If you can come up with a way to retain CPU hotplug function while doing all IRQ migration in interrupt context I would certainly be willing to try to find some time to help test and debug your changes on our systems.
Thanks,
Gary
Signature · 6 lines
Gary Hade System x Enablement IBM Linux Technology Center 503-578-4503 IBM T/L: 775-4503 [email protected] http://www.ibm.com/linux/ltc
Quoted 6 lines · Gary Hade
Quoted 5 lines
Quoted 2 lines
This didn't help. Using 2.6.30-rc3 plus your patch both bugs are unfortunately still present.
You could offline the cpus? I know when I tested it on my laptop I could not offline the cpus.
Quoted 7 lines
Eric, I'm sorry! This was due to my stupid mistake. When I went to apply your patch I included --dry-run to test it but apparently got distracted and never actually ran patch(1) without --dry-run. <SIGH>
So, I just rebuilt after _really_ applying the patch and got the following result which probably to be what you intended.
Ok. Good to see.
Quoted 18 lines
Quoted 12 lines
Quoted 11 lines
Quoted 4 lines
I propose detecting thpe cases that we know are safe to migrate in process context, aka logical deliver with less than 8 cpus aka "flat" routing mode and modifying the code so that those work in process context and simply deny cpu hotplug in all of the rest of the cases.
Humm, are you suggesting that CPU offlining/onlining would not be possible at all on systems with >8 logical CPUs (i.e. most of our systems) or would this just force users to separately migrate IRQ affinities away from a CPU (e.g. by shutting down the irqbalance daemon and writing to /proc/irq/<irq>/smp_affinity) before attempting to offline it?
Quoted 5 lines
A separate migration, for those hard to handle irqs.
The newest systems have iommus that irqs go through or are using MSIs for the important irqs, and as such can be migrated in process context. So this is not a restriction for future systems.
Quoted 6 lines
I understand your concerns but we need a solution for the earlier systems that does NOT remove or cripple the existing CPU hotplug functionality. If you can come up with a way to retain CPU hotplug function while doing all IRQ migration in interrupt context I would certainly be willing to try to find some time to help test and debug your changes on our systems.
Well that is ultimately what I am looking towards.
How do we move to a system that works by design, instead of one with design goals that are completely conflicting.
Thinking about it, we should be able to preemptively migrate irqs in the hook I am using that denies cpu hotplug.
If they don't migrate after a short while I expect we should still fail but that would relieve some of the pain, and certainly prevent a non-working system.
There are little bits we can tweak like special casing irqs that no-one is using.
My preference here is that I would rather deny cpu hotplug unplug than have the non-working system problems that you have seen.
All of that said I have some questions about your hardware.
- How many sockets and how many cores do you have?
- How many irqs do you have?
- Do you have an iommu that irqs can go through?
If you have <= 8 cores this problem is totally solvable.
Other cases may be but I don't know what the tradeoffs are. For very large systems we don't have enough irqs without limiting running in physical flat mode which makes things even more of a challenge.
It may also be that your ioapics don't have the bugs that intel and amd ioapics have and we could have a way to recognize high quality ioapics.
Eric
Quoted 55 lines · Eric W. Biederman
Quoted 6 lines · Gary Hade
Quoted 5 lines
Quoted 2 lines
This didn't help. Using 2.6.30-rc3 plus your patch both bugs are unfortunately still present.
You could offline the cpus? I know when I tested it on my laptop I could not offline the cpus.
Quoted 7 lines
Eric, I'm sorry! This was due to my stupid mistake. When I went to apply your patch I included --dry-run to test it but apparently got distracted and never actually ran patch(1) without --dry-run. <SIGH>
So, I just rebuilt after _really_ applying the patch and got the following result which probably to be what you intended.
Ok. Good to see.
Quoted 18 lines
Quoted 12 lines
Quoted 11 lines
Quoted 4 lines
I propose detecting thpe cases that we know are safe to migrate in process context, aka logical deliver with less than 8 cpus aka "flat" routing mode and modifying the code so that those work in process context and simply deny cpu hotplug in all of the rest of the cases.
Humm, are you suggesting that CPU offlining/onlining would not be possible at all on systems with >8 logical CPUs (i.e. most of our systems) or would this just force users to separately migrate IRQ affinities away from a CPU (e.g. by shutting down the irqbalance daemon and writing to /proc/irq/<irq>/smp_affinity) before attempting to offline it?
Quoted 5 lines
A separate migration, for those hard to handle irqs.
The newest systems have iommus that irqs go through or are using MSIs for the important irqs, and as such can be migrated in process context. So this is not a restriction for future systems.
Quoted 6 lines
I understand your concerns but we need a solution for the earlier systems that does NOT remove or cripple the existing CPU hotplug functionality. If you can come up with a way to retain CPU hotplug function while doing all IRQ migration in interrupt context I would certainly be willing to try to find some time to help test and debug your changes on our systems.
Well that is ultimately what I am looking towards.
How do we move to a system that works by design, instead of one with design goals that are completely conflicting.
Thinking about it, we should be able to preemptively migrate irqs in the hook I am using that denies cpu hotplug.
If they don't migrate after a short while I expect we should still fail but that would relieve some of the pain, and certainly prevent a non-working system.
Quoted 8 lines
There are little bits we can tweak like special casing irqs that no-one is using.
My preference here is that I would rather deny cpu hotplug unplug than have the non-working system problems that you have seen.
All of that said I have some questions about your hardware.
- How many sockets and how many cores do you have?
The largest is the x3950 M2 with up to 16 sockets and 96 cores in currently supported configurations and I expect that there could be at least double those numbers in the future.
http://www-03.ibm.com/systems/x/hardware/enterprise/x3950m2/index.html
Quoted 1 line
- How many irqs do you have?
On the single node x3950 M2 that I have been using with all of it's 7 PCIe slots vacant I see:
[root@elm3c160 ~]# cat /proc/interrupts | wc -l 21
Up to 4 nodes are currently supported and I expect that there could be at least double that number in the future.
Quoted 1 line
- Do you have an iommu that irqs can go through?
Only a subset of our systems (e.g. x460, x3850, x3950 w/Calgary iommu) have this.
Quoted 2 lines
If you have <= 8 cores this problem is totally solvable.
Dreamer :-)
Quoted 9 lines
Other cases may be but I don't know what the tradeoffs are. For very large systems we don't have enough irqs without limiting running in physical flat mode which makes things even more of a challenge.
It may also be that your ioapics don't have the bugs that intel and amd ioapics have and we could have a way to recognize high quality ioapics.
I believe all our System x boxes have Intel and AMD ioapics.
Gary
Signature · 6 lines
Gary Hade System x Enablement IBM Linux Technology Center 503-578-4503 IBM T/L: 775-4503 [email protected] http://www.ibm.com/linux/ltc
Quoted 97 lines · Gary Hade
Quoted 55 lines · Eric W. Biederman
Quoted 6 lines · Gary Hade
Quoted 5 lines
Quoted 2 lines
This didn't help. Using 2.6.30-rc3 plus your patch both bugs are unfortunately still present.
You could offline the cpus? I know when I tested it on my laptop I could not offline the cpus.
Quoted 7 lines
Eric, I'm sorry! This was due to my stupid mistake. When I went to apply your patch I included --dry-run to test it but apparently got distracted and never actually ran patch(1) without --dry-run. <SIGH>
So, I just rebuilt after _really_ applying the patch and got the following result which probably to be what you intended.
Ok. Good to see.
Quoted 18 lines
Quoted 12 lines
Quoted 11 lines
Quoted 4 lines
I propose detecting thpe cases that we know are safe to migrate in process context, aka logical deliver with less than 8 cpus aka "flat" routing mode and modifying the code so that those work in process context and simply deny cpu hotplug in all of the rest of the cases.
Humm, are you suggesting that CPU offlining/onlining would not be possible at all on systems with >8 logical CPUs (i.e. most of our systems) or would this just force users to separately migrate IRQ affinities away from a CPU (e.g. by shutting down the irqbalance daemon and writing to /proc/irq/<irq>/smp_affinity) before attempting to offline it?
Quoted 5 lines
A separate migration, for those hard to handle irqs.
The newest systems have iommus that irqs go through or are using MSIs for the important irqs, and as such can be migrated in process context. So this is not a restriction for future systems.
Quoted 6 lines
I understand your concerns but we need a solution for the earlier systems that does NOT remove or cripple the existing CPU hotplug functionality. If you can come up with a way to retain CPU hotplug function while doing all IRQ migration in interrupt context I would certainly be willing to try to find some time to help test and debug your changes on our systems.
Well that is ultimately what I am looking towards.
How do we move to a system that works by design, instead of one with design goals that are completely conflicting.
Thinking about it, we should be able to preemptively migrate irqs in the hook I am using that denies cpu hotplug.
If they don't migrate after a short while I expect we should still fail but that would relieve some of the pain, and certainly prevent a non-working system.
Quoted 8 lines
There are little bits we can tweak like special casing irqs that no-one is using.
My preference here is that I would rather deny cpu hotplug unplug than have the non-working system problems that you have seen.
All of that said I have some questions about your hardware.
- How many sockets and how many cores do you have?The largest is the x3950 M2 with up to 16 sockets and 96 cores in currently supported configurations and I expect that there could be at least double those numbers in the future.
http://www-03.ibm.com/systems/x/hardware/enterprise/x3950m2/index.htmlQuoted 1 line
- How many irqs do you have?
On the single node x3950 M2 that I have been using with all of it's 7 PCIe slots vacant I see:
[root@elm3c160 ~]# cat/proc/interrupts| wc -l 21
Up to 4 nodes are currently supported and I expect that there could be at least double that number in the future.Quoted 1 line
- Do you have an iommu that irqs can go through?
Only a subset of our systems (e.g. x460, x3850, x3950 w/Calgary iommu) have this.
Quoted 2 lines
If you have <= 8 cores this problem is totally solvable.
Dreamer :-)
Quoted 6 lines
Other cases may be but I don't know what the tradeoffs are. For very large systems we don't have enough irqs without limiting running in physical flat mode which makes things even more of a challenge.
Quoted 5 lines
Quoted 3 lines
It may also be that your ioapics don't have the bugs that intel and amd ioapics have and we could have a way to recognize high quality ioapics.
I believe all our System x boxes have Intel and AMD ioapics.
Actually, I should have said that many System x boxes have Intel or AMD ioapics. It is my understanding that the ioapic function on some of the high-end systems is integrated into the IBM chipset (e.g. Calgary and CalIOC2). However, since I have also seen the I/O redirection table register write with remote IRR bit set issue on some of those systems it probably doesn't make sense treat them any differently.
Gary
Signature · 6 lines
Gary Hade System x Enablement IBM Linux Technology Center 503-578-4503 IBM T/L: 775-4503 [email protected] http://www.ibm.com/linux/ltc
Quoted 45 lines · Eric W. Biederman
Quoted 3 lines · Gary Hade
So, I just rebuilt after _really_ applying the patch and got the following result which probably to be what you intended.
Ok. Good to see.
Quoted 18 lines
Quoted 12 lines
Quoted 11 lines
Quoted 4 lines
I propose detecting thpe cases that we know are safe to migrate in process context, aka logical deliver with less than 8 cpus aka "flat" routing mode and modifying the code so that those work in process context and simply deny cpu hotplug in all of the rest of the cases.
Humm, are you suggesting that CPU offlining/onlining would not be possible at all on systems with >8 logical CPUs (i.e. most of our systems) or would this just force users to separately migrate IRQ affinities away from a CPU (e.g. by shutting down the irqbalance daemon and writing to /proc/irq/<irq>/smp_affinity) before attempting to offline it?
Quoted 5 lines
A separate migration, for those hard to handle irqs.
The newest systems have iommus that irqs go through or are using MSIs for the important irqs, and as such can be migrated in process context. So this is not a restriction for future systems.
Quoted 6 lines
I understand your concerns but we need a solution for the earlier systems that does NOT remove or cripple the existing CPU hotplug functionality. If you can come up with a way to retain CPU hotplug function while doing all IRQ migration in interrupt context I would certainly be willing to try to find some time to help test and debug your changes on our systems.
Well that is ultimately what I am looking towards.
How do we move to a system that works by design, instead of one with design goals that are completely conflicting.
Thinking about it, we should be able to preemptively migrate irqs in the hook I am using that denies cpu hotplug.
If they don't migrate after a short while I expect we should still fail but that would relieve some of the pain, and certainly prevent a non-working system.
Quoted 5 lines
There are little bits we can tweak like special casing irqs that no-one is using.
My preference here is that I would rather deny cpu hotplug unplug than have the non-working system problems that you have seen.
and use delay work to offline cpu later after irq get moved to other cpu?
YH