Skip to contentSkip to conversation

make HVM PIC emulation code SMP-safe

PATCH In review David Lively 2 participants 2 messages May 15, 2006 – May 16, 2006

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 make HVM PIC emulation code SMP-safe

The current PIC emulation code for HVM (fully-virtualized) guests is unsafe on SMP hosts. Guests can access the PIC from any VCPU, though they should be (and generally are) controlling access so that at most a single VCPU is accessing the PIC at any time. However, there are two other paths that may access the PIC concurrently with a guest VCPU:
(1) "the APIC kludge" - cpu_get_interrupt() calls pic_update_irq() to bump slave PIC intrs to the master PIC. This is called from all
VCPUS.
(2) hvm_pic_assist() calls do_pic_irqs[_clear]() - from VCPU 0 only

With no PIC concurrency controls, SMP HVM guests[1] are unstable (tend to hang) under high I/O loads (e.g., the LTP aio-stress & dio tests). With the PIC concurrency control introduced in the attached patch, SMP HVM guests are considerably more stable. I have yet to see a hang under heavy I/O loads. I've tested only 64-bit SMP guests (but this code is unchanged for 32 bits), always passing "noapic" as explained below.

Dave

[1] Note we're passing "noapic" to the Linux kernels we're testing on SMP guests. This tells Linux to ignore the I/O APIC. Without this, we lose too many hda interrupts for Linux to boot. The I/O APIC code has many of the same issues as the PIC code. I have a similar fix for the I/O APICs, but since it doesn't fix the "lost interrupt" problem I haven't been able to adequately test it yet.

xen/arch/x86/hvm/hvm.c +3 -0
diff -r d056f91cfd95 xen/arch/x86/hvm/hvm.c
@@ -240,11 +240,14 @@ int cpu_get_interrupt(struct vcpu *v, in
 {
     int intno;
     struct hvm_virpic *s = &v->domain->arch.hvm_domain.vpic;
+    unsigned long flags;
 
     if ( (intno = cpu_get_apic_interrupt(v, type)) != -1 ) {
         /* set irq request if a PIC irq is still pending */
         /* XXX: improve that */
+        spin_lock_irqsave(&s->lock, flags);
         pic_update_irq(s);
+        spin_unlock_irqrestore(&s->lock, flags);
         return intno;
     }
     /* read the irq from the PIC */
xen/arch/x86/hvm/i8259.c +82 -10

312 lines not on this page · raw message

xen/include/asm-x86/hvm/vpic.h +2 -1
diff -r d056f91cfd95 xen/include/asm-x86/hvm/vpic.h
@@ -60,6 +60,7 @@ struct hvm_virpic {
     /* IOAPIC callback support */
     void (*alt_irq_func)(void *opaque, int irq_num, int level);
     void *alt_irq_opaque;
+    spinlock_t lock;
 };
 
 
@@ -72,7 +73,7 @@ void pic_set_alt_irq_func(struct hvm_vir
                           void (*alt_irq_func)(void *, int, int),
                           void *alt_irq_opaque);
 int pic_read_irq(struct hvm_virpic *s);
-void pic_update_irq(struct hvm_virpic *s);
+void pic_update_irq(struct hvm_virpic *s);	/* Caller must hold s->lock */
 uint32_t pic_intack_read(struct hvm_virpic *s);
 void register_pic_io_hook (void);
 int cpu_get_pic_interrupt(struct vcpu *v, int *type);

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

#1 lore raw

Resubmit of same with signed-off-by tag ...

Quoted 35 lines · David Lively

The current PIC emulation code for HVM (fully-virtualized) guests is unsafe on SMP hosts. Guests can access the PIC from any VCPU, though they should be (and generally are) controlling access so that at most a single VCPU is accessing the PIC at any time. However, there are two other paths that may access the PIC concurrently with a guest VCPU:
(1) "the APIC kludge" - cpu_get_interrupt() calls pic_update_irq() to bump slave PIC intrs to the master PIC. This is called from all
VCPUS.
(2) hvm_pic_assist() calls do_pic_irqs[_clear]() - from VCPU 0 only

With no PIC concurrency controls, SMP HVM guests[1] are unstable (tend to hang) under high I/O loads (e.g., the LTP aio-stress & dio tests). With the
PIC concurrency control introduced in the attached patch, SMP HVM guests are considerably more stable. I have yet to see a hang under heavy I/O loads.
I've tested only 64-bit SMP guests (but this code is unchanged for 32 bits), always
passing "noapic" as explained below.

Dave

[1] Note we're passing "noapic" to the Linux kernels we're testing on SMP guests. This tells Linux to ignore the I/O APIC. Without this, we lose too
many hda interrupts for Linux to boot. The I/O APIC code has many of the same issues as the PIC code. I have a similar fix for the I/O APICs, but since
it doesn't fix the "lost interrupt" problem I haven't been able to adequately test
it yet.

Quoted 8 lines

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

# Signed-off-by: David Lively <[email protected]>

xen/arch/x86/hvm/hvm.c +3 -0
diff -r d056f91cfd95 xen/arch/x86/hvm/hvm.c
@@ -240,11 +240,14 @@ int cpu_get_interrupt(struct vcpu *v, in
 {
     int intno;
     struct hvm_virpic *s = &v->domain->arch.hvm_domain.vpic;
+    unsigned long flags;
 
     if ( (intno = cpu_get_apic_interrupt(v, type)) != -1 ) {
         /* set irq request if a PIC irq is still pending */
         /* XXX: improve that */
+        spin_lock_irqsave(&s->lock, flags);
         pic_update_irq(s);
+        spin_unlock_irqrestore(&s->lock, flags);
         return intno;
     }
     /* read the irq from the PIC */
xen/arch/x86/hvm/i8259.c +82 -10

312 lines not on this page · raw message

xen/include/asm-x86/hvm/vpic.h +2 -1
diff -r d056f91cfd95 xen/include/asm-x86/hvm/vpic.h
@@ -60,6 +60,7 @@ struct hvm_virpic {
     /* IOAPIC callback support */
     void (*alt_irq_func)(void *opaque, int irq_num, int level);
     void *alt_irq_opaque;
+    spinlock_t lock;
 };
 
 
@@ -72,7 +73,7 @@ void pic_set_alt_irq_func(struct hvm_vir
                           void (*alt_irq_func)(void *, int, int),
                           void *alt_irq_opaque);
 int pic_read_irq(struct hvm_virpic *s);
-void pic_update_irq(struct hvm_virpic *s);
+void pic_update_irq(struct hvm_virpic *s);	/* Caller must hold s->lock */
 uint32_t pic_intack_read(struct hvm_virpic *s);
 void register_pic_io_hook (void);
 int cpu_get_pic_interrupt(struct vcpu *v, int *type);

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel