make cpu creation happen inside the right thread.
RFC 1/1 make cpu creation happen inside the right thread.
Right now, we issue cpu creation from the i/o thread, and then shoot a thread from inside that code. Over the last months, a lot of subtle bugs were reported, usually arising from the very fragile order of that initialization.
I propose we rethink that a little. This is a patch that received basic testing only, and I'd like to hear on the overall direction. The idea is to issue the new thread as early as possible. The first direct benefits I can identify are that we no longer have to rely at on_vcpu-like schemes for issuing vcpu ioctls, since we are already on the right thread. Apic creation has far less spots for race conditions as well.
I am implementing this on qemu-kvm first, since we can show the benefits of it a bit better in there (since we already support smp)
Let me know what you guys think
--- cpu-defs.h | 2 +- hw/acpi.c | 2 +- hw/pc.c | 26 ++++++++++++-------------- hw/pc.h | 2 +- qemu-kvm-x86.c | 5 +++++ qemu-kvm.c | 44 +++++++++++++++++++++++++++++++------------- qemu-kvm.h | 2 ++ vl.c | 2 -- 8 files changed, 53 insertions(+), 32 deletions(-)
cpu-defs.h +1 -1
@@ -139,7 +139,7 @@ typedef struct CPUWatchpoint {struct qemu_work_item ;struct KVMCPUState { -pthread_t thread; +pthread_t *thread;int signalled;struct qemu_work_item *queued_work_first, *queued_work_last;int regs_modified;
hw/acpi.c +1 -1
@@ -781,7 +781,7 @@ void qemu_system_cpu_hot_add(int cpu, int state)CPUState *env;if (state && !qemu_get_cpu (cpu)) { - env =pc_new_cpu (model); +pc_new_cpu (model, &env);if (!env) {fprintf (stderr,"cpu %d creation failed\n" , cpu);return ;
hw/pc.c +12 -14
@@ -1013,29 +1013,26 @@ int cpu_is_bsp(CPUState *env)return env->cpuid_apic_id ==0 ; } -CPUState *pc_new_cpu (const char *cpu_model) +void pc_new_cpu (const char *cpu_model,CPUState **env) { -CPUState *env; - - env =cpu_init (cpu_model); -if (!env) { + *env =cpu_init (cpu_model); +if (!*env) {fprintf (stderr,"Unable to find x86 CPU definition\n" );exit (1 ); } - env->kvm_cpu_state.regs_modified =1 ; -if ((env->cpuid_features &CPUID_APIC ) || smp_cpus >1 ) { - env->cpuid_apic_id = env->cpu_index; + (*env)->kvm_cpu_state.regs_modified =1 ; +if (((*env)->cpuid_features &CPUID_APIC ) || smp_cpus >1 ) { + (*env)->cpuid_apic_id = (*env)->cpu_index;/* APIC reset callback resets cpu */ -apic_init (env); +apic_init (*env); }else { -qemu_register_reset ((QEMUResetHandler *)cpu_reset, env); +qemu_register_reset ((QEMUResetHandler *)cpu_reset, *env); }/* kvm needs this to run after the apic is initialized. Otherwise, * it can access invalid state and crash. */ -qemu_init_vcpu (env); -return env; +qemu_init_vcpu (*env); }/* PC hardware initialisation */
@@ -1055,7 +1052,6 @@ static void pc_init1(ram_addr_t ram_size,PCIBus *pci_bus;ISADevice *isa_dev;int piix3_devfn =-1 ; -CPUState *env;qemu_irq *cpu_irq;qemu_irq *isa_irq;qemu_irq *i8259;
@@ -1086,8 +1082,10 @@ static void pc_init1(ram_addr_t ram_size,if (kvm_enabled ()) {kvm_set_boot_cpu_id (0 ); } +for (i =0 ; i < smp_cpus; i++) { - env =pc_new_cpu (cpu_model); +// pc_new_cpu(cpu_model, &env); +kvm_init_vcpu (NULL ); }vmport_init ();
hw/pc.h +1 -1
@@ -100,7 +100,7 @@ extern int fd_bootchk;void ioport_set_a20 (int enable);int ioport_get_a20 (void ); -CPUState *pc_new_cpu (const char *cpu_model); +void pc_new_cpu (const char *cpu_model,CPUState **env);/* acpi.c */ extern int acpi_enabled;
qemu-kvm-x86.c +5 -0
@@ -160,6 +160,11 @@ int kvm_arch_create(kvm_context_t kvm, unsigned long phys_mem_bytes,return 0 ; } +void kvm_arch_create_vcpu (const char *model,CPUState **env) +{ +pc_new_cpu ("qemu64" , env); +} +#ifdef KVM_EXIT_TPR_ACCESS static int kvm_handle_tpr_access (CPUState *env)
qemu-kvm.c +31 -13
112 lines not on this page · raw message
qemu-kvm.h +2 -0
@@ -87,6 +87,8 @@ int kvm_alloc_userspace_memory(kvm_context_t kvm, unsigned long memory,int kvm_arch_create (kvm_context_t kvm,unsigned long phys_mem_bytes,void **vm_mem); +void kvm_arch_create_vcpu (const char *model,CPUState **env); +int kvm_arch_run (CPUState *env);
vl.c +0 -2
@@ -3570,8 +3570,6 @@ void qemu_init_vcpu(void *_env) {CPUState *env = _env; -if (kvm_enabled ()) -kvm_init_vcpu (env); env->nr_cores = smp_cores; env->nr_threads = smp_threads;return ;
Signature · 1 line
1.6.2.5
Quoted 11 lines · Glauber Costa
Right now, we issue cpu creation from the i/o thread, and then shoot a thread from inside that code. Over the last months, a lot of subtle bugs were reported, usually arising from the very fragile order of that initialization.
I propose we rethink that a little. This is a patch that received basic testing only, and I'd like to hear on the overall direction. The idea is to issue the new thread as early as possible. The first direct benefits I can identify are that we no longer have to rely at on_vcpu-like schemes for issuing vcpu ioctls, since we are already on the right thread. Apic creation has far less spots for race conditions as well.
Quoted 4 lines
I am implementing this on qemu-kvm first, since we can show the benefits of it a bit better in there (since we already support smp)
Let me know what you guys think
Makes sense to me. You still need on_vcpu for issuing vcpu ioctls though (after initialization).
Quoted 12 lines · Marcelo Tosatti
Quoted 11 lines · Glauber Costa
Right now, we issue cpu creation from the i/o thread, and then shoot a thread from inside that code. Over the last months, a lot of subtle bugs were reported, usually arising from the very fragile order of that initialization.
I propose we rethink that a little. This is a patch that received basic testing only, and I'd like to hear on the overall direction. The idea is to issue the new thread as early as possible. The first direct benefits I can identify are that we no longer have to rely at on_vcpu-like schemes for issuing vcpu ioctls, since we are already on the right thread. Apic creation has far less spots for race conditions as well.
Quoted 7 lines
Quoted 4 lines
I am implementing this on qemu-kvm first, since we can show the benefits of it a bit better in there (since we already support smp)
Let me know what you guys think
Makes sense to me. You still need
on_vcpufor issuing vcpu ioctls though (after initialization).
Yes, but I believe we can avoid most of them. There is a performance hit of using it, but I am not so concerned with that. The nasty races that arises from it, are more a concern to me.