Skip to contentSkip to conversation

make cpu creation happen inside the right thread.

RFC In review Glauber Costa 2 participants 3 messages Nov 3, 2009 KVM

lore
Share & tools
lore
#0 lore raw

RFC 1/1 make cpu creation happen inside the right thread.

Right now, we issue cpu creation from the i/o thread, and then shoot a thread from inside that code. Over the last months, a lot of subtle bugs were reported, usually arising from the very fragile order of that initialization.

I propose we rethink that a little. This is a patch that received basic testing only, and I'd like to hear on the overall direction. The idea is to issue the new thread as early as possible. The first direct benefits I can identify are that we no longer have to rely at on_vcpu-like schemes for issuing vcpu ioctls, since we are already on the right thread. Apic creation has far less spots for race conditions as well.

I am implementing this on qemu-kvm first, since we can show the benefits of it a bit better in there (since we already support smp)

Let me know what you guys think

CC Marcelo Tosatti <[email protected]>
CC Avi Kivity <[email protected]>
CC Jan Kiszka <[email protected]>
CC Anthony Liguori <[email protected]>
---
 cpu-defs.h     |    2 +-
 hw/acpi.c      |    2 +-
 hw/pc.c        |   26 ++++++++++++--------------
 hw/pc.h        |    2 +-
 qemu-kvm-x86.c |    5 +++++
 qemu-kvm.c     |   44 +++++++++++++++++++++++++++++++-------------
 qemu-kvm.h     |    2 ++
 vl.c           |    2 --
 8 files changed, 53 insertions(+), 32 deletions(-)
cpu-defs.h +1 -1
index cf502e9..6d026e0 100644
@@ -139,7 +139,7 @@ typedef struct CPUWatchpoint {
 struct qemu_work_item;
 
 struct KVMCPUState {
-    pthread_t thread;
+    pthread_t *thread;
     int signalled;
     struct qemu_work_item *queued_work_first, *queued_work_last;
     int regs_modified;
hw/acpi.c +1 -1
index 7564abf..cc68188 100644
@@ -781,7 +781,7 @@ void qemu_system_cpu_hot_add(int cpu, int state)
     CPUState *env;
 
     if (state && !qemu_get_cpu(cpu)) {
-        env = pc_new_cpu(model);
+        pc_new_cpu(model, &env);
         if (!env) {
             fprintf(stderr, "cpu %d creation failed\n", cpu);
             return;
hw/pc.c +12 -14
index 83012a9..53e7273 100644
@@ -1013,29 +1013,26 @@ int cpu_is_bsp(CPUState *env)
     return env->cpuid_apic_id == 0;
 }
 
-CPUState *pc_new_cpu(const char *cpu_model)
+void pc_new_cpu(const char *cpu_model, CPUState **env)
 {
-    CPUState *env;
-
-    env = cpu_init(cpu_model);
-    if (!env) {
+    *env = cpu_init(cpu_model);
+    if (!*env) {
         fprintf(stderr, "Unable to find x86 CPU definition\n");
         exit(1);
     }
-    env->kvm_cpu_state.regs_modified = 1;
-    if ((env->cpuid_features & CPUID_APIC) || smp_cpus > 1) {
-        env->cpuid_apic_id = env->cpu_index;
+    (*env)->kvm_cpu_state.regs_modified = 1;
+    if (((*env)->cpuid_features & CPUID_APIC) || smp_cpus > 1) {
+        (*env)->cpuid_apic_id = (*env)->cpu_index;
         /* APIC reset callback resets cpu */
-        apic_init(env);
+        apic_init(*env);
     } else {
-        qemu_register_reset((QEMUResetHandler*)cpu_reset, env);
+        qemu_register_reset((QEMUResetHandler*)cpu_reset, *env);
     }
 
     /* kvm needs this to run after the apic is initialized. Otherwise,
      * it can access invalid state and crash.
      */
-    qemu_init_vcpu(env);
-    return env;
+    qemu_init_vcpu(*env);
 }
 
 /* PC hardware initialisation */
@@ -1055,7 +1052,6 @@ static void pc_init1(ram_addr_t ram_size,
     PCIBus *pci_bus;
     ISADevice *isa_dev;
     int piix3_devfn = -1;
-    CPUState *env;
     qemu_irq *cpu_irq;
     qemu_irq *isa_irq;
     qemu_irq *i8259;
@@ -1086,8 +1082,10 @@ static void pc_init1(ram_addr_t ram_size,
     if (kvm_enabled()) {
         kvm_set_boot_cpu_id(0);
     }
+
     for (i = 0; i < smp_cpus; i++) {
-        env = pc_new_cpu(cpu_model);
+//        pc_new_cpu(cpu_model, &env);
+	  kvm_init_vcpu(NULL);
     }
 
     vmport_init();
hw/pc.h +1 -1
index 93eb34d..f931380 100644
@@ -100,7 +100,7 @@ extern int fd_bootchk;
 
 void ioport_set_a20(int enable);
 int ioport_get_a20(void);
-CPUState *pc_new_cpu(const char *cpu_model);
+void pc_new_cpu(const char *cpu_model, CPUState **env);
 
 /* acpi.c */
 extern int acpi_enabled;
qemu-kvm-x86.c +5 -0
index 0d263ca..4084312 100644
@@ -160,6 +160,11 @@ int kvm_arch_create(kvm_context_t kvm, unsigned long phys_mem_bytes,
 	return 0;
 }
 
+void kvm_arch_create_vcpu(const char *model, CPUState **env)
+{
+    pc_new_cpu("qemu64", env);
+}
+
 #ifdef KVM_EXIT_TPR_ACCESS
 
 static int kvm_handle_tpr_access(CPUState *env)
qemu-kvm.c +31 -13

112 lines not on this page · raw message

qemu-kvm.h +2 -0
index 525d78d..4dd50f8 100644
@@ -87,6 +87,8 @@ int kvm_alloc_userspace_memory(kvm_context_t kvm, unsigned long memory,
 int kvm_arch_create(kvm_context_t kvm, unsigned long phys_mem_bytes,
                     void **vm_mem);
 
+void kvm_arch_create_vcpu(const char *model, CPUState **env);
+
 int kvm_arch_run(CPUState *env);
 
 
vl.c +0 -2
index 5dc7b2b..8dfae43 100644
@@ -3570,8 +3570,6 @@ void qemu_init_vcpu(void *_env)
 {
     CPUState *env = _env;
 
-    if (kvm_enabled())
-        kvm_init_vcpu(env);
     env->nr_cores = smp_cores;
     env->nr_threads = smp_threads;
     return;
Signature · 1 line
1.6.2.5
#1 lore raw
Quoted 11 lines · Glauber Costa

Right now, we issue cpu creation from the i/o thread, and then shoot a thread from inside that code. Over the last months, a lot of subtle bugs were reported, usually arising from the very fragile order of that initialization.

I propose we rethink that a little. This is a patch that received basic testing only, and I'd like to hear on the overall direction. The idea is to issue the new thread as early as possible. The first direct benefits I can identify are that we no longer have to rely at on_vcpu-like schemes for issuing vcpu ioctls, since we are already on the right thread. Apic creation has far less spots for race conditions as well.

Quoted 4 lines

I am implementing this on qemu-kvm first, since we can show the benefits of it a bit better in there (since we already support smp)

Let me know what you guys think

Makes sense to me. You still need on_vcpu for issuing vcpu ioctls though (after initialization).

#2 lore raw
Quoted 12 lines · Marcelo Tosatti
Quoted 11 lines · Glauber Costa

Right now, we issue cpu creation from the i/o thread, and then shoot a thread from inside that code. Over the last months, a lot of subtle bugs were reported, usually arising from the very fragile order of that initialization.

I propose we rethink that a little. This is a patch that received basic testing only, and I'd like to hear on the overall direction. The idea is to issue the new thread as early as possible. The first direct benefits I can identify are that we no longer have to rely at on_vcpu-like schemes for issuing vcpu ioctls, since we are already on the right thread. Apic creation has far less spots for race conditions as well.

Quoted 7 lines
Quoted 4 lines

I am implementing this on qemu-kvm first, since we can show the benefits of it a bit better in there (since we already support smp)

Let me know what you guys think

Makes sense to me. You still need on_vcpu for issuing vcpu ioctls though (after initialization).

Yes, but I believe we can avoid most of them. There is a performance hit of using it, but I am not so concerned with that. The nasty races that arises from it, are more a concern to me.