return default values for apic probe functions.
PATCH 1/1 return default values for apic probe functions.
As KVM cpus runs on threads, it is possible that
we call kvm_load_registers() from a cpu thread, while the apic has not yet fully initialized. kvm_load_registers() is called from ap_main_loop.
This is not a problem when we're starting the whole machine together, but is a problem for hotplug, since we don't have the protection of the locks that protect machine initialization. Currently, some executions of cpu hotplug on rainy sundays fail with a segfault.
Moving apic initialization to before kvm_init_vpcu proved fruitful, as there are some dependencies involved. (kvm irqchip would fail to initialize).
This patch provides default values to be used for tpr and apic_base, that will be returned when the apic is not yet properly initialized. It is aimed at kvm, where the problem exists, but it could equally be used for qemu too, if there is agreement.
--- qemu/hw/apic.c | 12 ++++++++++-- 1 files changed, 10 insertions(+), 2 deletions(-)
qemu/hw/apic.c +10 -2
@@ -301,7 +301,12 @@ uint64_t cpu_get_apic_base(CPUState *env)#ifdef DEBUG_APIC printf ("cpu_get_apic_base: %016" PRIx64"\n" , (uint64_t )s->apicbase);#endif -return s->apicbase; +if (s) { +return s->apicbase; + } +else { +return 0xfee00000 |MSR_IA32_APICBASE_ENABLE ; + } }void cpu_set_apic_tpr (CPUX86State *env,uint8_t val)
@@ -314,7 +319,10 @@ void cpu_set_apic_tpr(CPUX86State *env, uint8_t val)uint8_t cpu_get_apic_tpr (CPUX86State *env) {APICState *s = env->apic_state; -return s->tpr >>4 ; +if (s) +return s->tpr >>4 ; +else +return 0 ; }/* return -1 if no bit is set */
Signature · 1 line
1.5.6.6
Quoted 30 lines · Glauber Costa
As KVM cpus runs on threads, it is possible that
we callkvm_load_registers()from a cpu thread, while the apic has not yet fully initialized.kvm_load_registers()is called fromap_main_loop.This is not a problem when we're starting the whole machine together, but is a problem for hotplug, since we don't have the protection of the locks that protect machine initialization. Currently, some executions of cpu hotplug on rainy sundays fail with a segfault.
Moving apic initialization to before
kvm_init_vpcuproved fruitful, as there are some dependencies involved. (kvm irqchip would fail to initialize).This patch provides default values to be used for tpr and
apic_base, that will be returned when the apic is not yet properly initialized. It is aimed at kvm, where the problem exists, but it could equally be used for qemu too, if there is agreement.Signed-off-by Glauber Costa <[email protected]>--- qemu/hw/apic.c | 12 ++++++++++-- 1 files changed, 10 insertions(+), 2 deletions(-)qemu/hw/apic.c +0
-0@@ -301,7 +301,12 @@ uint64_t cpu_get_apic_base(CPUState *env)#ifdef DEBUG_APIC
qemu/hw/apic.c uint64_t cpu_get_apic_base(CPUState *env) · line 310
Quoted 10 lines
qemu/hw/apic.c +6
-1printf ("cpu_get_apic_base: %016" PRIx64"\n" , (uint64_t )s->apicbase);#endif -return s->apicbase; +if (s) { +return s->apicbase; + } +else { +return 0xfee00000 |MSR_IA32_APICBASE_ENABLE ; + } }
Even on sunny days, this collides with QEMU commit #7048. :)
Does Intel specify what non-existent MSRs should return, ie. is your version still correct if !s->apicbase means that there is actually no APIC? And does kvm depend on the default base? If so, I would say:
provide a patch against upstream.
Quoted 4 lines
qemu/hw/apic.c +0
-0void cpu_set_apic_tpr (CPUX86State *env,uint8_t val)@@ -314,7 +319,10 @@ void cpu_set_apic_tpr(CPUX86State *env, uint8_t val)uint8_t cpu_get_apic_tpr (CPUX86State *env)
qemu/hw/apic.c void cpu_set_apic_tpr(CPUX86State *env, uint8_t val) · line 328
Quoted 10 lines
qemu/hw/apic.c +4
-1{APICState *s = env->apic_state; -return s->tpr >>4 ; +if (s) +return s->tpr >>4 ; +else +return 0 ; }/* return -1 if no bit is set */
This is already upstream.
Jan
Quoted 6 lines
Even on sunny days, this collides with QEMU commit #7048. :)
Does Intel specify what non-existent MSRs should return, ie. is your version still correct if !s->apicbase means that there is actually no APIC? And does kvm depend on the default base? If so, I would say:
provide a patch against upstream.
hummm, I missed this one going in.
But sadly, your patch also breaks cpu hotplug. Not a segfault anymore, but the VM will freeze instead of shutting down, if we ask too. It does not even respond to ^C anymore.
By leaving your patch as is, and changing the apic base return to
return s ? s->apicbase : (0xfee00000 | MSR_IA32_APICBASE_ENABLE);
fixes the issue.
I'm not sure about what the manual says (will check now), but I believe if we ever try to
read from apic, we should return a meaningful value. Can you verify if this also works for your
test case?
Quoted 14 lines
Quoted 14 lines
void
cpu_set_apic_tpr(CPUX86State *env, uint8_t val)qemu/hw/apic.c +4
-1@@ -314,7 +319,10 @@ void cpu_set_apic_tpr(CPUX86State *env, uint8_t val) uint8_t cpu_get_apic_tpr(CPUX86State *env) { APICState *s = env->apic_state; - return s->tpr >> 4; + if (s) + return s->tpr >> 4; + else + return 0; }
Quoted 3 lines
qemu/hw/apic.c void cpu_set_apic_tpr(CPUX86State *env, uint8_t val) · line 328
Quoted 1 line
qemu/hw/apic.c +0
-0/* return -1 if no bit is set */This is already upstream.
Yeah, and the rest of your patch is totally ok for me.
Signature · 5 lines
Glauber Costa. "Free as in Freedom" http://glommer.net "The less confident you are, the more serious you have to act."
Quoted 12 lines · Glauber Costa
Quoted 6 lines
Even on sunny days, this collides with QEMU commit #7048. :)
Does Intel specify what non-existent MSRs should return, ie. is your version still correct if !s->apicbase means that there is actually no APIC? And does kvm depend on the default base? If so, I would say:
provide a patch against upstream.hummm, I missed this one going in.
But sadly, your patch also breaks cpu hotplug. Not a segfault anymore, but the VM will freeze instead of shutting down, if we ask too. It does not even respond to ^C anymore.
Quoted 6 lines
By leaving your patch as is, and changing the apic base return to
return s ?
s->apicbase: (0xfee00000|MSR_IA32_APICBASE_ENABLE);fixes the issue.
After reading the manual, my understanding is that only the flag must be set. I tried,
and in fact:
return s ? s->apicbase : MSR_IA32_APICBASE_ENABLE;
still fixes it.
If it works for you, I believe this is a good solution, and will send a descriptive patch.
If we ever read the apic as disabled, we will have problems enabling it again. So for
my test case, kvm should never see a disabled apic.
For yours, you'll still see the apic base address as zero.
what do you think?
Signature · 5 lines
Glauber Costa. "Free as in Freedom" http://glommer.net "The less confident you are, the more serious you have to act."
Quoted 31 lines · Glauber Costa
Quoted 11 lines · Glauber Costa
Quoted 6 lines
Even on sunny days, this collides with QEMU commit #7048. :)
Does Intel specify what non-existent MSRs should return, ie. is your version still correct if !s->apicbase means that there is actually no APIC? And does kvm depend on the default base? If so, I would say:
provide a patch against upstream.hummm, I missed this one going in.
But sadly, your patch also breaks cpu hotplug. Not a segfault anymore, but the VM will freeze instead of shutting down, if we ask too. It does not even respond to ^C anymore.Quoted 6 lines
By leaving your patch as is, and changing the apic base return to
return s ? s->apicbase : (0xfee00000 | MSR_IA32_APICBASE_ENABLE);fixes the issue.
After reading the manual, my understanding is that only the flag must be set. I tried,
and in fact:return s ? s->apicbase : MSR_IA32_APICBASE_ENABLE;still fixes it.
If it works for you, I believe this is a good solution, and will send a descriptive patch.
If we ever read the apic as disabled, we will have problems enabling it again. So for
my test case, kvm should never see a disabled apic.
Quoted 4 lines
For yours, you'll still see the apic base address as zero.
what do you think?
My use case (you can try it yourself: -M isapc) will likely already be fine if there is no segv on accidentally reading that msr. I was more concerned about the correct value according to the spec - if that case isn't simply "undefined".
On the other hand, I didn't get the precise race yet, and my feeling about this patch as a fix for something else than an inexistent apic is not that good. But it's just a feeling.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT SE 2 Corporate Competence Center Embedded Linux
Hi Glauber,
Quoted 9 lines · Glauber Costa
As KVM cpus runs on threads, it is possible that
we callkvm_load_registers()from a cpu thread, while the apic has not yet fully initialized.kvm_load_registers()is called fromap_main_loop.This is not a problem when we're starting the whole machine together, but is a problem for hotplug, since we don't have the protection of the locks that protect machine initialization. Currently, some executions of cpu hotplug on rainy sundays fail with a segfault.
/* and wait for machine initialization */
while (!qemu_system_ready)
qemu_cond_wait(&qemu_system_cond);
pthread_mutex_unlock(&qemu_mutex);Shouldnt this cover the cpu hotplug case too? Perhaps have:
/* wait for machine initialization */
while (!qemu_system_ready)
qemu_cond_wait(&qemu_system_cond);
/* wait for vcpu initialization */
while (!env->initialized)
qemu_cond_wait(&qemu_system_cond);
pthread_mutex_unlock(&qemu_mutex);And then set env->initialized when the cpu is good to go.
Because there could be other dependencies other than APIC initialization, for eg in pc_new_cpu
if (cpu != 0)
env->halted = 1;Quoted 51 lines
Moving apic initialization to before
kvm_init_vpcuproved fruitful, as there are some dependencies involved. (kvm irqchip would fail to initialize).This patch provides default values to be used for tpr and
apic_base, that will be returned when the apic is not yet properly initialized. It is aimed at kvm, where the problem exists, but it could equally be used for qemu too, if there is agreement.Signed-off-by Glauber Costa <[email protected]>--- qemu/hw/apic.c | 12 ++++++++++-- 1 files changed, 10 insertions(+), 2 deletions(-)qemu/hw/apic.c +10
-2@@ -301,7 +301,12 @@ uint64_t cpu_get_apic_base(CPUState *env)#ifdef DEBUG_APIC printf ("cpu_get_apic_base: %016" PRIx64"\n" , (uint64_t )s->apicbase);#endif -return s->apicbase; +if (s) { +return s->apicbase; + } +else { +return 0xfee00000 |MSR_IA32_APICBASE_ENABLE ; + } }void cpu_set_apic_tpr (CPUX86State *env,uint8_t val)@@ -314,7 +319,10 @@ void cpu_set_apic_tpr(CPUX86State *env, uint8_t val)uint8_t cpu_get_apic_tpr (CPUX86State *env) {APICState *s = env->apic_state; -return s->tpr >>4 ; +if (s) +return s->tpr >>4 ; +else +return 0 ; }/* return -1 if no bit is set */ Signature · 6 lines
1.5.6.6 -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 28 lines · Marcelo Tosatti
Hi Glauber,
Quoted 9 lines · Glauber Costa
As KVM cpus runs on threads, it is possible that
we callkvm_load_registers()from a cpu thread, while the apic has not yet fully initialized.kvm_load_registers()is called fromap_main_loop.This is not a problem when we're starting the whole machine together, but is a problem for hotplug, since we don't have the protection of the locks that protect machine initialization. Currently, some executions of cpu hotplug on rainy sundays fail with a segfault.
/* and wait for machine initialization */
while (!qemu_system_ready)
qemu_cond_wait(&qemu_system_cond);
pthread_mutex_unlock(&qemu_mutex);Shouldnt this cover the cpu hotplug case too? Perhaps have:
/* wait for machine initialization */
while (!qemu_system_ready)
qemu_cond_wait(&qemu_system_cond);
/* wait for vcpu initialization */
while (!env->initialized)
qemu_cond_wait(&qemu_system_cond);
pthread_mutex_unlock(&qemu_mutex);
Quoted 1 line
And then set
env->initializedwhen the cpu is good to go.
From my understanding, all this is only useful when the whole machine is starting, since they are global locks that wait for a system wide condition.
This is not the case with cpu hotplug, since the box is already on.
Quoted 6 lines
Because there could be other dependencies other than APIC initialization, for eg in
pc_new_cpuif (cpu != 0)
env->halted = 1;
it is okay for the cpu to be halted. Btw, I believe this should be moved inside cpu init.
Quoted 19 lines · Glauber Costa
Quoted 18 lines · Marcelo Tosatti
Hi Glauber,
Quoted 9 lines · Glauber Costa
As KVM cpus runs on threads, it is possible that
we callkvm_load_registers()from a cpu thread, while the apic has not yet fully initialized.kvm_load_registers()is called fromap_main_loop.This is not a problem when we're starting the whole machine together, but is a problem for hotplug, since we don't have the protection of the locks that protect machine initialization. Currently, some executions of cpu hotplug on rainy sundays fail with a segfault.
/* and wait for machine initialization */
while (!qemu_system_ready)
qemu_cond_wait(&qemu_system_cond);
pthread_mutex_unlock(&qemu_mutex);
Quoted 7 lines
Quoted 7 lines
Shouldnt this cover the cpu hotplug case too? Perhaps have:
/* wait for machine initialization */
while (!qemu_system_ready)
qemu_cond_wait(&qemu_system_cond);
/* wait for vcpu initialization */
while (!env->initialized)
^^^^^^^^^^^
Quoted 8 lines
Quoted 5 lines
qemu_cond_wait(&qemu_system_cond);
pthread_mutex_unlock(&qemu_mutex);And then set
env->initializedwhen the cpu is good to go. From my understanding, all this is only useful when the whole machineis starting, since they are global locks that wait for a system wide condition.
This is not the case with cpu hotplug, since the box is already on.
Yes, but the above proposes the addition of vcpu "initialized" state in addition to the system wide state, which would fix the current cpu hotplug breakage with apic, or any other access to uninitialized state from the vcpu thread.
Quoted 8 lines
Quoted 5 lines
Because there could be other dependencies other than APIC initialization, for eg in
pc_new_cpuif (cpu != 0)
env->halted = 1;it is okay for the cpu to be halted. Btw, I believe this should be moved inside cpu init.
Point here is the vcpu might not be halted by the time the vcpu thread starts to run, while it should (ie: its an example of uninitialized state other than the apic).
Quoted 5 lines · Glauber Costa
As KVM cpus runs on threads, it is possible that
we callkvm_load_registers()from a cpu thread, while the apic has not yet fully initialized.kvm_load_registers()is called fromap_main_loop.
Quoted 9 lines
This is not a problem when we're starting the whole machine together, but is a problem for hotplug, since we don't have the protection of the locks that protect machine initialization. Currently, some executions of cpu hotplug on rainy sundays fail with a segfault.
Moving apic initialization to before
kvm_init_vpcuproved fruitful, as there are some dependencies involved. (kvm irqchip would fail to initialize).
I presume you mean unfruitful (or perhaps a nasty kind of fruit).
Quoted 5 lines
This patch provides default values to be used for tpr and
apic_base, that will be returned when the apic is not yet properly initialized. It is aimed at kvm, where the problem exists, but it could equally be used for qemu too, if there is agreement.
Seems like a hack... can you try not to make the vcpu visible until it is completely initialized?
(and what is the problem exactly - someone accessing the registers from a different thread? that shouldn't happen)
Signature · 1 line
Do not meddle in the internals of kernels, for they are subtle and quick to panic.