tcg: handle EXCP_ATOMIC exception properly
PATCH 1/1 tcg: handle EXCP_ATOMIC exception properly
The current method of executing atomic code in a guest uses cpu_exec_step_atomic() from the outermost loop. This causes an abort() when single stepping over atomic code since debug exception longjmp will point to the the setlongjmp in cpu_exec(). Another issue with this mechanism is that the flags which were set in atomic execution will be lost since we do not call cpu_exec_enter().
The following patch moves atomic exception handling to the exception handler where all these issues are taken care of. The change in start_exclusive() is necessary since now the cpu in atomic execution will have its running flag set, but we do not want to count it as pending.
Thanks to Alex for helping me debug the issue.
--- cpu-exec.c | 2 ++ cpus-common.c | 2 +- cpus.c | 4 ---- 3 files changed, 3 insertions(+), 5 deletions(-)
cpu-exec.c +2 -0
@@ -427,6 +427,8 @@ static inline bool cpu_handle_exception(CPUState *cpu, int *ret) *ret = cpu->exception_index;if (*ret ==EXCP_DEBUG ) {cpu_handle_debug_exception (cpu); + }else if (*ret ==EXCP_ATOMIC ) { +cpu_exec_step_atomic (cpu); } cpu->exception_index =-1 ;return true;
cpus-common.c +1 -1
@@ -192,7 +192,7 @@ void start_exclusive(void)smp_mb (); running_cpus =0 ;CPU_FOREACH (other_cpu) { -if (atomic_read (&other_cpu->running)) { +if (atomic_read (&other_cpu->running) && !qemu_cpu_is_self (other_cpu)) { other_cpu->has_waiter = true; running_cpus++;qemu_cpu_kick (other_cpu);
cpus.c +0 -4
@@ -1461,10 +1461,6 @@ static void *qemu_tcg_cpu_thread_fn(void *arg)*/ g_assert (cpu->halted);break ; -case EXCP_ATOMIC : -qemu_mutex_unlock_iothread (); -cpu_exec_step_atomic (cpu); -qemu_mutex_lock_iothread ();default :/* Ignore everything else? */ break ;
Signature · 1 line
2.11.0
Quoted 31 lines · Pranith Kumar
The current method of executing atomic code in a guest uses
cpu_exec_step_atomic()from the outermost loop. This causes anabort()when single stepping over atomic code since debug exception longjmp will point to the the setlongjmp incpu_exec(). Another issue with this mechanism is that the flags which were set in atomic execution will be lost since we do not callcpu_exec_enter().The following patch moves atomic exception handling to the exception handler where all these issues are taken care of. The change in
start_exclusive()is necessary since now the cpu in atomic execution will have its running flag set, but we do not want to count it as pending.Thanks to Alex for helping me debug the issue.
CC Alex Bennée <[email protected]>CC Richard Henderson <[email protected]>CC Paolo Bonzini <[email protected]>Signed-off-by Pranith Kumar <[email protected]>--- cpu-exec.c | 2 ++ cpus-common.c | 2 +- cpus.c | 4 ---- 3 files changed, 3 insertions(+), 5 deletions(-)cpu-exec.c +0
-0
cpu-exec.c static inline bool cpu_handle_exception(CPUState *cpu, int *ret) · line 434
Quoted 8 lines
cpu-exec.c +2
-0@@ -427,6 +427,8 @@ static inline bool cpu_handle_exception(CPUState *cpu, int *ret) *ret = cpu->exception_index;if (*ret ==EXCP_DEBUG ) {cpu_handle_debug_exception (cpu); + }else if (*ret ==EXCP_ATOMIC ) { +cpu_exec_step_atomic (cpu); } cpu->exception_index =-1 ;return true;
Looks like this is going to be a problem since we should not call start_exclusive() from cpu_exec() (doh', I just read the comment for this :-/).
It'll be great if we can make it callable from there. Thoughts?
Thanks,
Signature · 1 line
Pranith
Quoted 30 lines · Pranith Kumar
The current method of executing atomic code in a guest uses
cpu_exec_step_atomic()from the outermost loop. This causes anabort()when single stepping over atomic code since debug exception longjmp will point to the the setlongjmp incpu_exec(). Another issue with this mechanism is that the flags which were set in atomic execution will be lost since we do not callcpu_exec_enter().The following patch moves atomic exception handling to the exception handler where all these issues are taken care of. The change in
start_exclusive()is necessary since now the cpu in atomic execution will have its running flag set, but we do not want to count it as pending.Thanks to Alex for helping me debug the issue.
CC Alex Bennée <[email protected]>CC Richard Henderson <[email protected]>CC Paolo Bonzini <[email protected]>Signed-off-by Pranith Kumar <[email protected]>--- cpu-exec.c | 2 ++ cpus-common.c | 2 +- cpus.c | 4 ---- 3 files changed, 3 insertions(+), 5 deletions(-)cpu-exec.c +0
-0
cpu-exec.c static inline bool cpu_handle_exception(CPUState *cpu, int *ret) · line 431
Quoted 5 lines
cpu-exec.c +2
-0@@ -427,6 +427,8 @@ static inline bool cpu_handle_exception(CPUState *cpu, int *ret) *ret = cpu->exception_index;if (*ret ==EXCP_DEBUG ) {cpu_handle_debug_exception (cpu); + }else if (*ret ==EXCP_ATOMIC ) { +cpu_exec_step_atomic (cpu);
I think you can unlock/lock the iothread here, and also call cpu_exec_end/start to work around the limitation in start_exclusive.
Paolo
Quoted 32 lines
cpu-exec.c +0
-0} cpu->exception_index =-1 ;return true;cpus-common.c +1
-1@@ -192,7 +192,7 @@ void start_exclusive(void)smp_mb (); running_cpus =0 ;CPU_FOREACH (other_cpu) { -if (atomic_read (&other_cpu->running)) { +if (atomic_read (&other_cpu->running) && !qemu_cpu_is_self (other_cpu)) { other_cpu->has_waiter = true; running_cpus++;qemu_cpu_kick (other_cpu);cpus.c +0
-4@@ -1461,10 +1461,6 @@ static void *qemu_tcg_cpu_thread_fn(void *arg)*/ g_assert (cpu->halted);break ; -case EXCP_ATOMIC : -qemu_mutex_unlock_iothread (); -cpu_exec_step_atomic (cpu); -qemu_mutex_lock_iothread ();default :/* Ignore everything else? */ break ;
Quoted 26 lines · Paolo Bonzini
Quoted 25 lines · Pranith Kumar
The current method of executing atomic code in a guest uses
cpu_exec_step_atomic()from the outermost loop. This causes anabort()when single stepping over atomic code since debug exception longjmp will point to the the setlongjmp incpu_exec(). Another issue with this mechanism is that the flags which were set in atomic execution will be lost since we do not callcpu_exec_enter().The following patch moves atomic exception handling to the exception handler where all these issues are taken care of. The change in
start_exclusive()is necessary since now the cpu in atomic execution will have its running flag set, but we do not want to count it as pending.Thanks to Alex for helping me debug the issue.
CC Alex Bennée <[email protected]>CC Richard Henderson <[email protected]>CC Paolo Bonzini <[email protected]>Signed-off-by Pranith Kumar <[email protected]>--- cpu-exec.c | 2 ++ cpus-common.c | 2 +- cpus.c | 4 ---- 3 files changed, 3 insertions(+), 5 deletions(-)
Quoted 12 lines
cpu-exec.c static inline bool cpu_handle_exception(CPUState *cpu, int *ret) · line 431
Quoted 10 lines
cpu-exec.c +2
-0@@ -427,6 +427,8 @@ static inline bool cpu_handle_exception(CPUState *cpu, int *ret) *ret = cpu->exception_index;if (*ret ==EXCP_DEBUG ) {cpu_handle_debug_exception (cpu); + }else if (*ret ==EXCP_ATOMIC ) { +cpu_exec_step_atomic (cpu);I think you can unlock/lock the iothread here, and also call
The iothread is already unlocked by this point (see tcg_cpu_exec).
Quoted 1 line
cpu_exec_end/start to work around the limitation in
start_exclusive.
While that seems right it also seems very messy as it inverts the calls so far. I fear we may end up very confused in special casing. Is there a cleaner way we can unwind this?
Quoted 27 lines
Paolo
Quoted 24 lines
cpu-exec.c +0
-0} cpu->exception_index =-1 ;return true;cpus-common.c +1
-1@@ -192,7 +192,7 @@ void start_exclusive(void)smp_mb (); running_cpus =0 ;CPU_FOREACH (other_cpu) { -if (atomic_read (&other_cpu->running)) { +if (atomic_read (&other_cpu->running) && !qemu_cpu_is_self (other_cpu)) { other_cpu->has_waiter = true; running_cpus++;qemu_cpu_kick (other_cpu);cpus.c +0
-0@@ -1461,10 +1461,6 @@ static void *qemu_tcg_cpu_thread_fn(void *arg)*/ g_assert (cpu->halted);break ;
Quoted 8 lines
cpus.c static void *qemu_tcg_cpu_thread_fn(void *arg) · line 1466
Quoted 8 lines
cpus.c +0
-4-case EXCP_ATOMIC : -qemu_mutex_unlock_iothread (); -cpu_exec_step_atomic (cpu); -qemu_mutex_lock_iothread ();default :/* Ignore everything else? */ break ;
--
Alex Bennée
Quoted 3 lines · Alex Bennée
Quoted 1 line
I think you can unlock/lock the iothread here, and also call
The iothread is already unlocked by this point (see
tcg_cpu_exec).
Is this patch on top of the MTTCG branch? If not, cpu_handle_exception runs with the iothread lock taken doesn't it?
Quoted 5 lines
Quoted 1 line
cpu_exec_end/start to work around the limitation in
start_exclusive.While that seems right it also seems very messy as it inverts the calls so far. I fear we may end up very confused in special casing. Is there a cleaner way we can unwind this?
I'm not sure what is messy... cpu_exec_start/end and start/end_exclusive are fundamentally a rwlock.
Doing
cpu_exec_end start_exclusive ... end_exclusive cpu_exec_start
simply means temporarily upgrading the lock from read to write.
Paolo
Quoted 19 lines · Paolo Bonzini
Quoted 3 lines · Alex Bennée
Quoted 1 line
I think you can unlock/lock the iothread here, and also call
The iothread is already unlocked by this point (see
tcg_cpu_exec).Is this patch on top of the MTTCG branch? If not,
cpu_handle_exceptionruns with the iothread lock taken doesn't it?Quoted 5 lines
Quoted 1 line
cpu_exec_end/start to work around the limitation in
start_exclusive.While that seems right it also seems very messy as it inverts the calls so far. I fear we may end up very confused in special casing. Is there a cleaner way we can unwind this?
I'm not sure what is messy... cpu_exec_start/end and start/end_exclusive are fundamentally a rwlock.
Doing
Quoted 7 lines
cpu_exec_end start_exclusive ... end_exclusive cpu_exec_startsimply means temporarily upgrading the lock from read to write.
I mean messy from the point of reading the code where we reverse a sequence done from higher up in the call chain. Maybe I'm being overly picky because we aren't exactly on a hot path here anyway.
--
Alex Bennée
Quoted 21 lines · Paolo Bonzini
Quoted 3 lines · Alex Bennée
Quoted 1 line
I think you can unlock/lock the iothread here, and also call
The iothread is already unlocked by this point (see
tcg_cpu_exec).Is this patch on top of the MTTCG branch? If not,
cpu_handle_exceptionruns with the iothread lock taken doesn't it?Quoted 5 lines
Quoted 1 line
cpu_exec_end/start to work around the limitation in
start_exclusive.While that seems right it also seems very messy as it inverts the calls so far. I fear we may end up very confused in special casing. Is there a cleaner way we can unwind this?
I'm not sure what is messy... cpu_exec_start/end and start/end_exclusive are fundamentally a rwlock.
Doing
Quoted 8 lines
cpu_exec_end start_exclusive ... end_exclusive cpu_exec_startsimply means temporarily upgrading the lock from read to write.
This seems to be the simplest thing to do for now. I'll send a v2.
Thanks,
Signature · 1 line
Pranith
Quoted 6 lines · Pranith Kumar
The current method of executing atomic code in a guest uses
cpu_exec_step_atomic()from the outermost loop. This causes anabort()when single stepping over atomic code since debug exception longjmp will point to the the setlongjmp incpu_exec(). Another issue with this mechanism is that the flags which were set in atomic execution will be lost since we do not callcpu_exec_enter().
I should not the original patch (which is still in my tree so I guess I should squash it) says:
The patch enables handling atomic code in the guest. This should be preferably done in cpu_handle_exception(), but the current assumptions regarding when we can execute atomic sections cause a deadlock.
Quoted 36 lines
The following patch moves atomic exception handling to the exception handler where all these issues are taken care of. The change in
start_exclusive()is necessary since now the cpu in atomic execution will have its running flag set, but we do not want to count it as pending.Thanks to Alex for helping me debug the issue.
CC Alex Bennée <[email protected]>CC Richard Henderson <[email protected]>CC Paolo Bonzini <[email protected]>Signed-off-by Pranith Kumar <[email protected]>--- cpu-exec.c | 2 ++ cpus-common.c | 2 +- cpus.c | 4 ---- 3 files changed, 3 insertions(+), 5 deletions(-)cpu-exec.c +2
-0@@ -427,6 +427,8 @@ static inline bool cpu_handle_exception(CPUState *cpu, int *ret) *ret = cpu->exception_index;if (*ret ==EXCP_DEBUG ) {cpu_handle_debug_exception (cpu); + }else if (*ret ==EXCP_ATOMIC ) { +cpu_exec_step_atomic (cpu); } cpu->exception_index =-1 ;return true;cpus-common.c +0
-0
cpus-common.c void start_exclusive(void) · line 196
Quoted 6 lines
cpus-common.c +1
-1@@ -192,7 +192,7 @@ void start_exclusive(void)smp_mb (); running_cpus =0 ;CPU_FOREACH (other_cpu) { -if (atomic_read (&other_cpu->running)) { +if (atomic_read (&other_cpu->running) && !qemu_cpu_is_self (other_cpu)) {
The comment above reads:
Must only be called from outside cpu_exec.
So we need to revise this comment. Is this really a limitation or was it originally the design goal?
Quoted 8 lines
cpus-common.c +0
-0other_cpu->has_waiter = true; running_cpus++;qemu_cpu_kick(other_cpu);cpus.c +0
-0
cpus.c static void *qemu_tcg_cpu_thread_fn(void *arg) · line 1466
Quoted 10 lines
cpus.c +0
-4@@ -1461,10 +1461,6 @@ static void *qemu_tcg_cpu_thread_fn(void *arg)*/ g_assert (cpu->halted);break ; -case EXCP_ATOMIC : -qemu_mutex_unlock_iothread (); -cpu_exec_step_atomic (cpu); -qemu_mutex_lock_iothread ();default :/* Ignore everything else? */ break ;
--
Alex Bennée
Quoted 8 lines · Alex Bennée
Quoted 2 lines
+ if (
atomic_read(&other_cpu->running)&&
!qemu_cpu_is_self(other_cpu)) {The comment above reads:
Must only be called from outside
cpu_exec.So we need to revise this comment. Is this really a limitation or was it originally the design goal?
If you want to call it within cpu_exec, you can always use cpu_exec_end/start around it.
I think we should first of all get rid of the iothread lock within cpu_exec, and then look at this patch again.
Paolo
Quoted 7 lines · Paolo Bonzini
Quoted 6 lines · Alex Bennée
Quoted 2 lines
+ if (
atomic_read(&other_cpu->running)&&
!qemu_cpu_is_self(other_cpu)) {The comment above reads:
Must only be called from outside
cpu_exec.
Quoted 8 lines
Quoted 2 lines
So we need to revise this comment. Is this really a limitation or was it originally the design goal?
If you want to call it within
cpu_exec, you can always use cpu_exec_end/start around it.I think we should first of all get rid of the iothread lock within
cpu_exec, and then look at this patch again.
What to do with the MTTCG series in the meantime?
I'm hesitant to hold up the whole series for a potentially messy re-factor of the cpu loop code to push out the BQL which could take some time, although of course I don't want to merge something that makes that harder.
That said for TCG system emulation EXCP_ATOMIC is currently broken with respect to debugging. However for the initial guests/host combination of ARMv7/8 on x86_64 we don't need the fallback with pretty much 99% of deployed hosts. How about the following:
- drop Pranith's patch for the current MTTCG series
- replace with an error/abort on EXCP_ATOMIC
- proceed with merge as plan
- tackle the BQL lock next (along with more MTTCG guest/targets enablement)
Richard/Peter,
Any thoughts/opinions on this?
--
Alex Bennée
Quoted 7 lines · Alex Bennée
That said for TCG system emulation
EXCP_ATOMICis currently broken with respect to debugging. However for the initial guests/host combination of ARMv7/8 onx86_64we don't need the fallback with pretty much 99% of deployed hosts. How about the following:- drop Pranith's patch for the current MTTCG series
- replace with an error/abort onEXCP_ATOMIC
Don't even replace it? :)
Paolo
Quoted 2 lines
- proceed with merge as plan
- tackle the BQL lock next (along with more MTTCG guest/targets enablement)
Quoted 10 lines · Paolo Bonzini
Quoted 7 lines · Alex Bennée
That said for TCG system emulation
EXCP_ATOMICis currently broken with respect to debugging. However for the initial guests/host combination of ARMv7/8 onx86_64we don't need the fallback with pretty much 99% of deployed hosts. How about the following:- drop Pranith's patch for the current MTTCG series
- replace with an error/abort onEXCP_ATOMICDon't even replace it? :)
I guess - the failure mode is only when we single step? Should we not try to emit a warning like we do for the mismatched MOs?
Quoted 5 lines
Paolo
Quoted 2 lines
- proceed with merge as plan
- tackle the BQL lock next (along with more MTTCG guest/targets enablement)
--
Alex Bennée
Quoted 6 lines · Alex Bennée
Quoted 3 lines
Quoted 2 lines
- drop Pranith's patch for the current MTTCG series
- replace with an error/abort onEXCP_ATOMICDon't even replace it? :)
I guess - the failure mode is only when we single step? Should we not try to emit a warning like we do for the mismatched MOs?
If we're confident we can fix it before 2.9, we can afford the small bug for the time being.
Paolo