Implement qemu_thread_yield for posix, use it in mttcg to handle EXCP_YIELD
RFC PATCH 1/1 Implement qemu_thread_yield for posix, use it in mttcg to handle EXCP_YIELD
This is a bit of proof of concept in case mttcg becomes more important yield could be handled like this. You can have by accident or deliberately force vCPUs onto the same physical CPU and cause inversion issues when the lock holder was preempted by the waiter. This is lightly tested but not to the point of measuring performance difference.
I really consider the previous confer/prod patches more important just to provide a more complete guest environment and better test coverage, than performance, but maybe someone wants to persue this.
Thanks,
Nick
--- cpus.c | 6 ++++++ hw/ppc/spapr_hcall.c | 14 +++++++------- include/qemu/thread.h | 1 + util/qemu-thread-posix.c | 5 +++++ util/qemu-thread-win32.c | 4 ++++ 5 files changed, 23 insertions(+), 7 deletions(-)
cpus.c +6 -0
@@ -1760,6 +1760,12 @@ static void *qemu_tcg_cpu_thread_fn(void *arg)qemu_mutex_unlock_iothread ();cpu_exec_step_atomic (cpu);qemu_mutex_lock_iothread (); +break ; +case EXCP_YIELD : +qemu_mutex_unlock_iothread (); +qemu_thread_yield (); +qemu_mutex_lock_iothread (); +break ;default :/* Ignore everything else? */ break ;
hw/ppc/spapr_hcall.c +7 -7
@@ -1162,13 +1162,13 @@ static target_ulong h_confer(PowerPCCPU *cpu, SpaprMachineState *spapr,return H_SUCCESS ; } -/* -* The targeted confer does not do anything special beyond yielding -* the current vCPU, but even this should be better than nothing. -* At least for single-threaded tcg, it gives the target a chance to -* run before we run again. Multi-threaded tcg does not really do -* anything with EXCP_YIELD yet. -*/ +/* +* The targeted confer does not do anything special beyond yielding +* the current vCPU, but even this should be better than nothing. +* For single-threaded tcg, it gives the target a chance to run +* before we run again, multi-threaded tcg will yield the CPU to +* another vCPU. +*/ } cs->exception_index =EXCP_YIELD ;
include/qemu/thread.h +1 -0
@@ -160,6 +160,7 @@ void qemu_thread_get_self(QemuThread *thread);bool qemu_thread_is_self (QemuThread *thread);void qemu_thread_exit (void *retval);void qemu_thread_naming (bool enable); +void qemu_thread_yield (void );struct Notifier ;/**
util/qemu-thread-posix.c +5 -0
@@ -573,3 +573,8 @@ void *qemu_thread_join(QemuThread *thread) }return ret; } + +void qemu_thread_yield (void ) +{ +pthread_yield (); +}
util/qemu-thread-win32.c +4 -0
@@ -442,3 +442,7 @@ bool qemu_thread_is_self(QemuThread *thread) {return GetCurrentThreadId () == thread->tid; } + +void qemu_thread_yield (void ) +{ +}
Signature · 1 line
2.20.1
Quoted 49 lines · Nicholas Piggin
This is a bit of proof of concept in case mttcg becomes more important yield could be handled like this. You can have by accident or deliberately force vCPUs onto the same physical CPU and cause inversion issues when the lock holder was preempted by the waiter. This is lightly tested but not to the point of measuring performance difference.
I really consider the previous confer/prod patches more important just to provide a more complete guest environment and better test coverage, than performance, but maybe someone wants to persue this.
Thanks,
Nick--- cpus.c | 6 ++++++ hw/ppc/spapr_hcall.c | 14 +++++++------- include/qemu/thread.h | 1 + util/qemu-thread-posix.c | 5 +++++ util/qemu-thread-win32.c | 4 ++++ 5 files changed, 23 insertions(+), 7 deletions(-)cpus.c +6
-0@@ -1760,6 +1760,12 @@ static void *qemu_tcg_cpu_thread_fn(void *arg)qemu_mutex_unlock_iothread ();cpu_exec_step_atomic (cpu);qemu_mutex_lock_iothread (); +break ; +case EXCP_YIELD : +qemu_mutex_unlock_iothread (); +qemu_thread_yield (); +qemu_mutex_lock_iothread (); +break ;default :/* Ignore everything else? */ break ;hw/ppc/spapr_hcall.c +0
-4@@ -1162,13 +1162,13 @@ static target_ulong h_confer(PowerPCCPU *cpu, SpaprMachineState *spapr,return H_SUCCESS ; } -/* -* The targeted confer does not do anything special beyond yielding -* the current vCPU, but even this should be better than nothing. -* At least for single-threaded tcg, it gives the target a chance to
hw/ppc/spapr_hcall.c static target_ulong h_confer(PowerPCCPU *cpu, SpaprMachineState *spapr, · line 1171
Quoted 10 lines
hw/ppc/spapr_hcall.c +7
-3-* run before we run again. Multi-threaded tcg does not really do -* anything with EXCP_YIELD yet. -*/ +/* +* The targeted confer does not do anything special beyond yielding +* the current vCPU, but even this should be better than nothing. +* For single-threaded tcg, it gives the target a chance to run +* before we run again, multi-threaded tcg will yield the CPU to +* another vCPU. +*/
Uh.. this looks like a whitespace fixup leaked in from your other patches.
Quoted 40 lines
hw/ppc/spapr_hcall.c +0
-0} cs->exception_index =EXCP_YIELD ;include/qemu/thread.h +1
-0@@ -160,6 +160,7 @@ void qemu_thread_get_self(QemuThread *thread);bool qemu_thread_is_self (QemuThread *thread);void qemu_thread_exit (void *retval);void qemu_thread_naming (bool enable); +void qemu_thread_yield (void );struct Notifier ;/** util/qemu-thread-posix.c +5
-0@@ -573,3 +573,8 @@ void *qemu_thread_join(QemuThread *thread) }return ret; } + +void qemu_thread_yield (void ) +{ +pthread_yield (); +}util/qemu-thread-win32.c +4
-0@@ -442,3 +442,7 @@ bool qemu_thread_is_self(QemuThread *thread) {return GetCurrentThreadId () == thread->tid; } + +void qemu_thread_yield (void ) +{ +}
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
Quoted 5 lines · Nicholas Piggin
This is a bit of proof of concept in case mttcg becomes more important yield could be handled like this. You can have by accident or deliberately force vCPUs onto the same physical CPU and cause inversion issues when the lock holder was preempted by the waiter. This is lightly tested but not to the point of measuring performance difference.
Sorry I'm so late replying.
Really this comes down to what EXCP_YIELD semantics are meant to mean. For ARM it's a hint operation because we also have WFE which should halt until there is some sort of change of state. In those cases exiting the main-loop and sitting in wait_for_io should be the correct response. If a vCPU is suspended waiting on the halt condition doesn't it have the same effect?
Quoted 94 lines
I really consider the previous confer/prod patches more important just to provide a more complete guest environment and better test coverage, than performance, but maybe someone wants to persue this.
Thanks,
Nick--- cpus.c | 6 ++++++ hw/ppc/spapr_hcall.c | 14 +++++++------- include/qemu/thread.h | 1 + util/qemu-thread-posix.c | 5 +++++ util/qemu-thread-win32.c | 4 ++++ 5 files changed, 23 insertions(+), 7 deletions(-)cpus.c +6
-0@@ -1760,6 +1760,12 @@ static void *qemu_tcg_cpu_thread_fn(void *arg)qemu_mutex_unlock_iothread ();cpu_exec_step_atomic (cpu);qemu_mutex_lock_iothread (); +break ; +case EXCP_YIELD : +qemu_mutex_unlock_iothread (); +qemu_thread_yield (); +qemu_mutex_lock_iothread (); +break ;default :/* Ignore everything else? */ break ;hw/ppc/spapr_hcall.c +7
-7@@ -1162,13 +1162,13 @@ static target_ulong h_confer(PowerPCCPU *cpu, SpaprMachineState *spapr,return H_SUCCESS ; } -/* -* The targeted confer does not do anything special beyond yielding -* the current vCPU, but even this should be better than nothing. -* At least for single-threaded tcg, it gives the target a chance to -* run before we run again. Multi-threaded tcg does not really do -* anything with EXCP_YIELD yet. -*/ +/* +* The targeted confer does not do anything special beyond yielding +* the current vCPU, but even this should be better than nothing. +* For single-threaded tcg, it gives the target a chance to run +* before we run again, multi-threaded tcg will yield the CPU to +* another vCPU. +*/ } cs->exception_index =EXCP_YIELD ;include/qemu/thread.h +1
-0@@ -160,6 +160,7 @@ void qemu_thread_get_self(QemuThread *thread);bool qemu_thread_is_self (QemuThread *thread);void qemu_thread_exit (void *retval);void qemu_thread_naming (bool enable); +void qemu_thread_yield (void );struct Notifier ;/** util/qemu-thread-posix.c +5
-0@@ -573,3 +573,8 @@ void *qemu_thread_join(QemuThread *thread) }return ret; } + +void qemu_thread_yield (void ) +{ +pthread_yield (); +}util/qemu-thread-win32.c +4
-0@@ -442,3 +442,7 @@ bool qemu_thread_is_self(QemuThread *thread) {return GetCurrentThreadId () == thread->tid; } + +void qemu_thread_yield (void ) +{ +}
Signature · 1 line
Alex Bennée
Alex Bennée's on December 20, 2019 11:11 pm:
Quoted 10 lines
Quoted 5 lines · Nicholas Piggin
This is a bit of proof of concept in case mttcg becomes more important yield could be handled like this. You can have by accident or deliberately force vCPUs onto the same physical CPU and cause inversion issues when the lock holder was preempted by the waiter. This is lightly tested but not to the point of measuring performance difference.
Sorry I'm so late replying.
That's fine if you'll also forgive me :)
Quoted 6 lines
Really this comes down to what
EXCP_YIELDsemantics are meant to mean. For ARM it's a hint operation because we also have WFE which should halt until there is some sort of change of state. In those cases exiting the main-loop and sitting inwait_for_ioshould be the correct response. If a vCPU is suspended waiting on the halt condition doesn't it have the same effect?
For powerpc H_CONFER, the vCPU does not want to wait for ever, but just give up a some time slice on the physical CPU and allow other vCPUs to run. But it's not necessary that one does run (if they are all sleeping, the hypervisor must prevent deadlock). How would you wait on such a conditon?
Thanks,
Nick
Quoted 21 lines · Nicholas Piggin
Alex Bennée's on December 20, 2019 11:11 pm:
Quoted 10 lines
Quoted 5 lines · Nicholas Piggin
This is a bit of proof of concept in case mttcg becomes more important yield could be handled like this. You can have by accident or deliberately force vCPUs onto the same physical CPU and cause inversion issues when the lock holder was preempted by the waiter. This is lightly tested but not to the point of measuring performance difference.
Sorry I'm so late replying.
That's fine if you'll also forgive me :)
Quoted 6 lines
Really this comes down to what
EXCP_YIELDsemantics are meant to mean. For ARM it's a hint operation because we also have WFE which should halt until there is some sort of change of state. In those cases exiting the main-loop and sitting inwait_for_ioshould be the correct response. If a vCPU is suspended waiting on the halt condition doesn't it have the same effect?
Quoted 5 lines
For powerpc
H_CONFER, the vCPU does not want to wait for ever, but just give up a some time slice on the physical CPU and allow other vCPUs to run. But it's not necessary that one does run (if they are all sleeping, the hypervisor must prevent deadlock). How would you wait on such a conditon?
Isn't H_CONFER a hypercall rather than instruction though? In QEMU's TCG emulation case I would expect it just to exit to the (guest) hypervisor which then schedules the next (guest) vCPU. It shouldn't be something QEMU has to deal with.
If you are running QEMU as a KVM monitor this is still outside of it's scope as all the scheduling shenanigans are dealt with inside the kernel.
From QEMU's TCG point of view we want to concern ourselves with what the real hardware would do - which I think in this case is drop to the hypervisor and let it sort it out.
--
Alex Bennée
Quoted 30 lines · Alex Bennée
Quoted 21 lines · Nicholas Piggin
Alex Bennée's on December 20, 2019 11:11 pm:
Quoted 10 lines
Quoted 5 lines · Nicholas Piggin
This is a bit of proof of concept in case mttcg becomes more important yield could be handled like this. You can have by accident or deliberately force vCPUs onto the same physical CPU and cause inversion issues when the lock holder was preempted by the waiter. This is lightly tested but not to the point of measuring performance difference.
Sorry I'm so late replying.
That's fine if you'll also forgive me :)
Quoted 6 lines
Really this comes down to what
EXCP_YIELDsemantics are meant to mean. For ARM it's a hint operation because we also have WFE which should halt until there is some sort of change of state. In those cases exiting the main-loop and sitting inwait_for_ioshould be the correct response. If a vCPU is suspended waiting on the halt condition doesn't it have the same effect?Quoted 5 lines
For powerpc
H_CONFER, the vCPU does not want to wait for ever, but just give up a some time slice on the physical CPU and allow other vCPUs to run. But it's not necessary that one does run (if they are all sleeping, the hypervisor must prevent deadlock). How would you wait on such a conditon?
Quoted 4 lines
Isn't
H_CONFERa hypercall rather than instruction though? In QEMU's TCG emulation case I would expect it just to exit to the (guest) hypervisor which then schedules the next (guest) vCPU. It shouldn't be something QEMU has to deal with.
That's true if you're emulating a whole system complete with hypervisor under TCG, which is what the "pnv" machine type does.
However, a more common use of qemu is the "pseries" machine type, which emulates only a guest (in the cpu architectural sense) with qemu taking the place of the hypervisor as well as emulating the cpus. In that case the H_CONFER hypercall goes to qemu.
Quoted 7 lines
If you are running QEMU as a KVM monitor this is still outside of it's scope as all the scheduling shenanigans are dealt with inside the kernel.
From QEMU's TCG point of view we want to concern ourselves with what the real hardware would do - which I think in this case is drop to the hypervisor and let it sort it out.
Right, but with the "pseries" machine type qemu *is* the hypervisor.
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
Quoted 9 lines · David Gibson
However, a more common use of qemu is the "pseries" machine type, which emulates only a guest (in the cpu architectural sense) with qemu taking the place of the hypervisor as well as emulating the cpus. In that case the
H_CONFERhypercall goes to qemu.Quoted 4 lines
If you are running QEMU as a KVM monitor this is still outside of it's scope as all the scheduling shenanigans are dealt with inside the kernel.
Quoted 5 lines
Quoted 3 lines
From QEMU's TCG point of view we want to concern ourselves with what the real hardware would do - which I think in this case is drop to the hypervisor and let it sort it out.
Right, but with the "pseries" machine type qemu *is* the hypervisor.
In which case this behaviour doesn't seem implausible.
I will note that "pthread_yield" isn't standardized; "sched_yield" is the one in POSIX. Though that says nothing about how that syscall might affect a hypothetical many-to-one pthread implementation. You could, I suppose, have a configure test for pthread_yield.
Also, the win32 implementation would be SwitchToThread():
It looks like one need do nothing for the single-threaded implementation, qemu_tcg_rr_cpu_thread_fn, as any return to the main loop will select the next round-robin cpu. But a note to say that's been tested would be nice.
r~