Skip to contentSkip to conversation

fix SMP migration and loadvm/savevm (V2)

PATCH In review Marcelo Tosatti 2 participants 9 messages Apr 7, 2008 – Apr 11, 2008 KVM

lore
Share & tools
lore

Marcelo Tosatti

author
#0 lore raw

cover fix SMP migration and loadvm/savevm (V2)

Now guarded by KVM_CAP and with increased version_id of cpu_save/load as requested.

Avi, I prefer not to fold mpstate into kvm_save_registers() as a hidden register because the MPSTATE is only used during migration, whereas save_registers() is not (seems safer).

Signature · 5 lines
-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
Register now and save $200. Hurry, offer ends at 11:59 p.m., 
Monday, April 7! Use priority code J8TLD2. 
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
author
#1 lore raw

PATCH 1/4 QEMU/KVM: properly copy the in-kernel apicbase value

The MSR_IA32_APICBASE_ENABLE/MSR_IA32_APICBASE_BSP bits in s->apicbase are not initialized if in-kernel APIC emulation is used, so save the actual value passed by cpu_set_apic_base() caller.

Signed-off-by Marcelo Tosatti <[email protected]>
git/kvm-userspace.io/qemu/hw/apic.c +5 -2
Index: marcelo/git/kvm-userspace.io/qemu/hw/apic.c
===================================================================
@@ -248,8 +248,11 @@ void cpu_set_apic_base(CPUState *env, ui
 #ifdef DEBUG_APIC
     printf("cpu_set_apic_base: %016" PRIx64 "\n", val);
 #endif
-    s->apicbase = (val & 0xfffff000) |
-        (s->apicbase & (MSR_IA32_APICBASE_BSP | MSR_IA32_APICBASE_ENABLE));
+    if (kvm_enabled() && qemu_kvm_irqchip_in_kernel())
+        s->apicbase = val;
+    else
+        s->apicbase = (val & 0xfffff000) |
+            (s->apicbase & (MSR_IA32_APICBASE_BSP | MSR_IA32_APICBASE_ENABLE));
     /* if disabled, cannot be enabled again */
     if (!(val & MSR_IA32_APICBASE_ENABLE)) {
         s->apicbase &= ~MSR_IA32_APICBASE_ENABLE;
git/kvm-userspace.io/qemu/qemu-kvm-x86.c +2 -7
Index: marcelo/git/kvm-userspace.io/qemu/qemu-kvm-x86.c
===================================================================
@@ -248,13 +248,8 @@ void kvm_arch_load_regs(CPUState *env)
     sregs.cr3 = env->cr[3];
     sregs.cr4 = env->cr[4];
 
-    if (kvm_irqchip_in_kernel(kvm_context)) {
-        sregs.cr8 = kvm_get_cr8(kvm_context, env->cpu_index);
-        sregs.apic_base = kvm_get_apic_base(kvm_context, env->cpu_index);
-    } else {
-        sregs.cr8 = cpu_get_apic_tpr(env);
-        sregs.apic_base = cpu_get_apic_base(env);
-    }
+    sregs.cr8 = cpu_get_apic_tpr(env);
+    sregs.apic_base = cpu_get_apic_base(env);
 
     sregs.efer = env->efer;
Signature · 5 lines
-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
Register now and save $200. Hurry, offer ends at 11:59 p.m., 
Monday, April 7! Use priority code J8TLD2. 
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
author
#2 lore raw

PATCH 2/4 QEMU/KVM: get/set mpstate libkvm interface

In reply to Marcelo Tosatti

Add libkvm interface to get/set the mpstate.

Signed-off-by Marcelo Tosatti <[email protected]>
git/kvm-userspace.io/libkvm/libkvm.c +26 -0
Index: marcelo/git/kvm-userspace.io/libkvm/libkvm.c
===================================================================
@@ -776,6 +776,32 @@ int kvm_set_sregs(kvm_context_t kvm, int
     return ioctl(kvm->vcpu_fd[vcpu], KVM_SET_SREGS, sregs);
 }
 
+int kvm_get_mpstate(kvm_context_t kvm, int vcpu, int *mpstate)
+{
+    int r;
+
+#ifdef KVM_CAP_MP_STATE
+    r = ioctl(kvm->fd, KVM_CHECK_EXTENSION, KVM_CAP_MP_STATE);
+    if (r > 0)
+        return ioctl(kvm->vcpu_fd[vcpu], KVM_GET_MP_STATE, mpstate);
+#endif
+    return -1;
+}
+
+int kvm_set_mpstate(kvm_context_t kvm, int vcpu, int *mpstate)
+{
+    int r;
+
+#ifdef KVM_CAP_MP_STATE
+    r = ioctl(kvm->fd, KVM_CHECK_EXTENSION, KVM_CAP_MP_STATE);
+    if (r > 0)
+        return ioctl(kvm->vcpu_fd[vcpu], KVM_SET_MP_STATE, mpstate);
+#endif
+    return -1;
+}
+
+
+
 static int handle_mmio(kvm_context_t kvm, struct kvm_run *kvm_run)
 {
 	unsigned long addr = kvm_run->mmio.phys_addr;
git/kvm-userspace.io/libkvm/libkvm.h +12 -0
Index: marcelo/git/kvm-userspace.io/libkvm/libkvm.h
===================================================================
@@ -301,6 +301,18 @@ int kvm_get_sregs(kvm_context_t kvm, int
 int kvm_set_sregs(kvm_context_t kvm, int vcpu, struct kvm_sregs *regs);
 
 /*!
+ *  * \brief Read VCPU MP state
+ *
+ */
+int kvm_get_mpstate(kvm_context_t kvm, int vcpu, int *mpstate);
+
+/*!
+ *  * \brief Write VCPU MP state
+ *
+ */
+int kvm_set_mpstate(kvm_context_t kvm, int vcpu, int *mpstate);
+
+/*!
  * \brief Simulate an external vectored interrupt
  *
  * This allows you to simulate an external vectored interrupt.
Signature · 5 lines
-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
Register now and save $200. Hurry, offer ends at 11:59 p.m., 
Monday, April 7! Use priority code J8TLD2. 
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
#3 lore raw

Re: [patch 2/4] QEMU/KVM: get/set mpstate libkvm interface

Quoted 14 lines · Marcelo Tosatti

Add libkvm interface to get/set the mpstate.

Signed-off-by Marcelo Tosatti <[email protected]>

Index: marcelo/git/kvm-userspace.io/libkvm/libkvm.c
===================================================================

git/kvm-userspace.io/libkvm/libkvm.c +2 -0
@@ -776,6 +776,32 @@ int kvm_set_sregs(kvm_context_t kvm, int
     return ioctl(kvm->vcpu_fd[vcpu], KVM_SET_SREGS, sregs);
 }
 
+int kvm_get_mpstate(kvm_context_t kvm, int vcpu, int *mpstate)
+{

git/kvm-userspace.io/libkvm/libkvm.c int kvm_set_sregs(kvm_context_t kvm, int · line 790

Quoted 10 lines
git/kvm-userspace.io/libkvm/libkvm.c +9 -0
+    int r;
+
+#ifdef KVM_CAP_MP_STATE
+    r = ioctl(kvm->fd, KVM_CHECK_EXTENSION, KVM_CAP_MP_STATE);
+    if (r > 0)
+        return ioctl(kvm->vcpu_fd[vcpu], KVM_GET_MP_STATE, mpstate);
+#endif
+    return -1;
+}

We usually follow the convention of returning -errno in case of error instead of -1 in libkvm.

Signature · 8 lines
Any sufficiently difficult bug is indistinguishable from a feature.


-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
Don't miss this year's exciting event. There's still time to save $100. 
Use priority code J8TL2D2. 
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
author
#4 lore raw

PATCH 3/4 QEMU/KVM: save and load mp state

In reply to Marcelo Tosatti

Use the new interface to save and restore MP_STATE for all vcpu's.

Increase version_id for cpu_load/cpu_save.

Fixes SMP migration.

Signed-off-by Marcelo Tosatti <[email protected]>
git/kvm-userspace.io/qemu/qemu-kvm-x86.c +18 -0
Index: marcelo/git/kvm-userspace.io/qemu/qemu-kvm-x86.c
===================================================================
@@ -277,6 +277,24 @@ void kvm_arch_load_regs(CPUState *env)
         perror("kvm_set_msrs FAILED");
 }
 
+void kvm_save_mpstate(CPUState *env)
+{
+    int r;
+
+    r = kvm_get_mpstate(kvm_context, env->cpu_index, &env->mp_state);
+    if (r == -1)
+        env->mp_state = r;
+}
+
+void kvm_load_mpstate(CPUState *env)
+{
+    /*
+     * -1 indicates that the host did not support GET_MP_STATE ioctl,
+     *  so don't touch it.
+     */
+    if (env->mp_state != -1)
+        kvm_set_mpstate(kvm_context, env->cpu_index, &env->mp_state);
+}
 
 void kvm_arch_save_regs(CPUState *env)
 {
git/kvm-userspace.io/qemu/qemu-kvm.h +2 -0
Index: marcelo/git/kvm-userspace.io/qemu/qemu-kvm.h
===================================================================
@@ -18,6 +18,8 @@ int kvm_init_ap(void);
 void kvm_qemu_destroy(void);
 void kvm_load_registers(CPUState *env);
 void kvm_save_registers(CPUState *env);
+void kvm_load_mpstate(CPUState *env);
+void kvm_save_mpstate(CPUState *env);
 int kvm_cpu_exec(CPUState *env);
 int kvm_update_debugger(CPUState *env);
 int kvm_qemu_init_env(CPUState *env);
git/kvm-userspace.io/qemu/target-i386/cpu.h +1 -0
Index: marcelo/git/kvm-userspace.io/qemu/target-i386/cpu.h
===================================================================
@@ -599,6 +599,7 @@ typedef struct CPUX86State {
     /* in order to simplify APIC support, we leave this pointer to the
        user */
     struct APICState *apic_state;
+    int mp_state;
 } CPUX86State;
 
 CPUX86State *cpu_x86_init(const char *cpu_model);
git/kvm-userspace.io/qemu/vl.c +9 -2
Index: marcelo/git/kvm-userspace.io/qemu/vl.c
===================================================================
@@ -6655,8 +6655,10 @@ void cpu_save(QEMUFile *f, void *opaque)
     uint32_t hflags;
     int i;
 
-    if (kvm_enabled())
+    if (kvm_enabled()) {
         kvm_save_registers(env);
+        kvm_save_mpstate(env);
+    }
 
     for(i = 0; i < CPU_NB_REGS; i++)
         qemu_put_betls(f, &env->regs[i]);

@@ -6748,6 +6750,7 @@ void cpu_save(QEMUFile *f, void *opaque)
             qemu_put_be32s(f, &env->kvm_interrupt_bitmap[i]);
         }
         qemu_put_be64s(f, &env->tsc);
+        qemu_put_be32s(f, &env->mp_state);
     }
 }
 

@@ -6782,7 +6785,7 @@ int cpu_load(QEMUFile *f, void *opaque, 
     uint32_t hflags;
     uint16_t fpus, fpuc, fptag, fpregs_format;
 
-    if (version_id != 3 && version_id != 4)
+    if (version_id < 3 || version_id > 5)
         return -EINVAL;
     for(i = 0; i < CPU_NB_REGS; i++)
         qemu_get_betls(f, &env->regs[i]);

@@ -6900,6 +6903,10 @@ int cpu_load(QEMUFile *f, void *opaque, 
         }
         qemu_get_be64s(f, &env->tsc);
         kvm_load_registers(env);
+        if (version_id >= 5) {
+            qemu_get_be32s(f, &env->mp_state);
+            kvm_load_mpstate(env);
+        }
     }
     return 0;
 }
git/kvm-userspace.io/qemu/hw/pc.c +1 -1
Index: marcelo/git/kvm-userspace.io/qemu/hw/pc.c
===================================================================
@@ -750,7 +750,7 @@ CPUState *pc_new_cpu(int cpu, const char
             /* XXX: enable it in all cases */
             env->cpuid_features |= CPUID_APIC;
         }
-        register_savevm("cpu", cpu, 4, cpu_save, cpu_load, env);
+        register_savevm("cpu", cpu, 5, cpu_save, cpu_load, env);
         qemu_register_reset(main_cpu_reset, env);
         if (pci_enabled) {
             apic_init(env);
Signature · 5 lines
-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
Register now and save $200. Hurry, offer ends at 11:59 p.m., 
Monday, April 7! Use priority code J8TLD2. 
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
author
#5 lore raw

PATCH 4/4 QEMU/KVM: ignore SIG_IPI signals in userspace

In reply to Marcelo Tosatti

Otherwise a signal can be received in userspace and a vcpu goes back to the kernel while it should stay still.

Signed-off-by Marcelo Tosatti <[email protected]>
git/kvm-userspace.io/qemu/qemu-kvm.c +0 -1
Index: marcelo/git/kvm-userspace.io/qemu/qemu-kvm.c
===================================================================
@@ -350,7 +350,6 @@ static void *ap_main_loop(void *_env)
     vcpu->env = env;
     vcpu->env->thread_id = kvm_get_thread_id();
     sigfillset(&signals);
-    sigdelset(&signals, SIG_IPI);
     sigprocmask(SIG_BLOCK, &signals, NULL);
     kvm_create_vcpu(kvm_context, env->cpu_index);
     kvm_qemu_init_env(env);
Signature · 5 lines
-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
Register now and save $200. Hurry, offer ends at 11:59 p.m., 
Monday, April 7! Use priority code J8TLD2. 
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
#6 lore raw
In reply to Marcelo Tosatti
Quoted 3 lines · Marcelo Tosatti

Avi, I prefer not to fold mpstate into kvm_save_registers() as a hidden register because the MPSTATE is only used during migration, whereas save_registers() is not (seems safer)

But that's the point... what about savevm/loadvm, etc? They deserve to work too.

Signature · 8 lines
Any sufficiently difficult bug is indistinguishable from a feature.


-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
Register now and save $200. Hurry, offer ends at 11:59 p.m., 
Monday, April 7! Use priority code J8TLD2. 
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
author
#7 lore raw
Quoted 7 lines · Avi Kivity
Quoted 3 lines · Marcelo Tosatti

Avi, I prefer not to fold mpstate into kvm_save_registers() as a hidden register because the MPSTATE is only used during migration, whereas save_registers() is not (seems safer)

But that's the point... what about savevm/loadvm, etc? They deserve to work too.

savevm/loadvm will work through cpu_save/cpu_load just as live migration does, so they are covered.

-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2.
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone

#8 lore raw
Quoted 8 lines · Marcelo Tosatti
Quoted 6 lines · Avi Kivity
Quoted 4 lines · Marcelo Tosatti

Avi, I prefer not to fold mpstate into kvm_save_registers() as a hidden register because the MPSTATE is only used during migration, whereas save_registers() is not (seems safer)

Quoted 8 lines
Quoted 3 lines

But that's the point... what about savevm/loadvm, etc? They deserve to work too.

savevm/loadvm will work through cpu_save/cpu_load just as live migration does, so they are covered.

Right.

Signature · 8 lines
Any sufficiently difficult bug is indistinguishable from a feature.


-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
Don't miss this year's exciting event. There's still time to save $100. 
Use priority code J8TL2D2. 
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone