make HVM PIC emulation code SMP-safe
PATCH 1/1 make HVM PIC emulation code SMP-safe
The current PIC emulation code for HVM (fully-virtualized) guests is unsafe on SMP hosts. Guests can access the PIC from any VCPU, though they should be (and generally are) controlling access so that at most a single VCPU is accessing the PIC at any time. However, there are two other paths that may access the PIC concurrently with a guest VCPU:
(1) "the APIC kludge" - cpu_get_interrupt() calls pic_update_irq() to bump slave PIC intrs to the master PIC. This is called from all
VCPUS.
(2) hvm_pic_assist() calls do_pic_irqs[_clear]() - from VCPU 0 only
With no PIC concurrency controls, SMP HVM guests[1] are unstable (tend to hang) under high I/O loads (e.g., the LTP aio-stress & dio tests). With the PIC concurrency control introduced in the attached patch, SMP HVM guests are considerably more stable. I have yet to see a hang under heavy I/O loads. I've tested only 64-bit SMP guests (but this code is unchanged for 32 bits), always passing "noapic" as explained below.
Dave
[1] Note we're passing "noapic" to the Linux kernels we're testing on SMP guests. This tells Linux to ignore the I/O APIC. Without this, we lose too many hda interrupts for Linux to boot. The I/O APIC code has many of the same issues as the PIC code. I have a similar fix for the I/O APICs, but since it doesn't fix the "lost interrupt" problem I haven't been able to adequately test it yet.
xen/arch/x86/hvm/hvm.c +3 -0
@@ -240,11 +240,14 @@ int cpu_get_interrupt(struct vcpu *v, in {int intno;struct hvm_virpic *s = &v->domain->arch.hvm_domain.vpic; +unsigned long flags;if ( (intno =cpu_get_apic_interrupt (v, type)) !=-1 ) {/* set irq request if a PIC irq is still pending */ /* XXX: improve that */ +spin_lock_irqsave (&s->lock, flags);pic_update_irq (s); +spin_unlock_irqrestore (&s->lock, flags);return intno; }/* read the irq from the PIC */
xen/arch/x86/hvm/i8259.c +82 -10
312 lines not on this page · raw message
xen/include/asm-x86/hvm/vpic.h +2 -1
@@ -60,6 +60,7 @@ struct hvm_virpic {/* IOAPIC callback support */ void (*alt_irq_func)(void *opaque,int irq_num,int level);void *alt_irq_opaque; +spinlock_t lock; };
@@ -72,7 +73,7 @@ void pic_set_alt_irq_func(struct hvm_virvoid (*alt_irq_func)(void *,int ,int ), void *alt_irq_opaque);int pic_read_irq (struct hvm_virpic *s); -void pic_update_irq (struct hvm_virpic *s); +void pic_update_irq (struct hvm_virpic *s);/* Caller must hold s->lock */ uint32_t pic_intack_read (struct hvm_virpic *s);void register_pic_io_hook (void );int cpu_get_pic_interrupt (struct vcpu *v,int *type);
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel
Resubmit of same with signed-off-by tag ...
Quoted 35 lines · David Lively
The current PIC emulation code for HVM (fully-virtualized) guests is unsafe on SMP hosts. Guests can access the PIC from any VCPU, though they should be (and generally are) controlling access so that at most a single VCPU is accessing the PIC at any time. However, there are two other paths that may access the PIC concurrently with a guest VCPU:
(1) "the APIC kludge" -cpu_get_interrupt()callspic_update_irq()to bump slave PIC intrs to the master PIC. This is called from all
VCPUS.
(2)hvm_pic_assist()calls do_pic_irqs[_clear]() - from VCPU 0 onlyWith no PIC concurrency controls, SMP HVM guests[1] are unstable (tend to hang) under high I/O loads (e.g., the LTP aio-stress & dio tests). With the
PIC concurrency control introduced in the attached patch, SMP HVM guests are considerably more stable. I have yet to see a hang under heavy I/O loads.
I've tested only 64-bit SMP guests (but this code is unchanged for 32 bits), always
passing "noapic" as explained below.Dave
[1] Note we're passing "noapic" to the Linux kernels we're testing on SMP guests. This tells Linux to ignore the I/O APIC. Without this, we lose too
many hda interrupts for Linux to boot. The I/O APIC code has many of the same issues as the PIC code. I have a similar fix for the I/O APICs, but since
it doesn't fix the "lost interrupt" problem I haven't been able to adequately test
it yet.
Quoted 8 lines
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel
# Signed-off-by: David Lively <[email protected]>
xen/arch/x86/hvm/hvm.c +3 -0
@@ -240,11 +240,14 @@ int cpu_get_interrupt(struct vcpu *v, in {int intno;struct hvm_virpic *s = &v->domain->arch.hvm_domain.vpic; +unsigned long flags;if ( (intno =cpu_get_apic_interrupt (v, type)) !=-1 ) {/* set irq request if a PIC irq is still pending */ /* XXX: improve that */ +spin_lock_irqsave (&s->lock, flags);pic_update_irq (s); +spin_unlock_irqrestore (&s->lock, flags);return intno; }/* read the irq from the PIC */
xen/arch/x86/hvm/i8259.c +82 -10
312 lines not on this page · raw message
xen/include/asm-x86/hvm/vpic.h +2 -1
@@ -60,6 +60,7 @@ struct hvm_virpic {/* IOAPIC callback support */ void (*alt_irq_func)(void *opaque,int irq_num,int level);void *alt_irq_opaque; +spinlock_t lock; };
@@ -72,7 +73,7 @@ void pic_set_alt_irq_func(struct hvm_virvoid (*alt_irq_func)(void *,int ,int ), void *alt_irq_opaque);int pic_read_irq (struct hvm_virpic *s); -void pic_update_irq (struct hvm_virpic *s); +void pic_update_irq (struct hvm_virpic *s);/* Caller must hold s->lock */ uint32_t pic_intack_read (struct hvm_virpic *s);void register_pic_io_hook (void );int cpu_get_pic_interrupt (struct vcpu *v,int *type);
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel