RFC: separate thread for IO
cover RFC: separate thread for IO
Introduce a separate thread for handling IO. Not intended for inclusion, but should be something for people to comment on.
PATCH 1/2 qemu: sem/thread helpers
Makefile.target +1 -1
@@ -500,7 +500,7 @@ endif #CONFIG_BSD_USER # System emulator target ifndef CONFIG_USER_ONLY -OBJS=vl.o osdep.o monitor.o pci.o loader.o isa_mmio.o machine.o +OBJS=vl.o qemu-thread.o osdep.o monitor.o pci.o loader.o isa_mmio.o machine.o # virtio has to be here due to weird dependency between PCI and virtio-net. # need to fix this properly OBJS+=virtio.o virtio-blk.o virtio-balloon.o virtio-net.o virtio-console.o
vl.c +9 -0
@@ -36,6 +36,7 @@#include "gdbstub.h" #include "qemu-timer.h" #include "qemu-char.h" +#include "qemu-thread.h" #include "cache-utils.h" #include "block.h" #include "audio/audio.h"
@@ -263,6 +264,8 @@ static QEMUTimer *nographic_timer;uint8_t qemu_uuid[16 ]; +QemuSem qemu_sem; +/***********************************************************/ /* x86 ISA bus support */
@@ -3650,7 +3653,10 @@ void main_loop_wait(int timeout) slirp_select_fill(&nfds, &rfds, &wfds, &xfds); }#endif + +qemu_sem_unlock (&qemu_sem); ret =select (nfds +1 , &rfds, &wfds, &xfds, &tv); +qemu_sem_lock (&qemu_sem);if (ret >0 ) {IOHandlerRecord **pioh;
@@ -3708,6 +3714,9 @@ static int main_loop(void)#endif CPUState *env; +qemu_sem_init (&qemu_sem); +qemu_sem_lock (&qemu_sem); + cur_cpu = first_cpu; next_cpu = cur_cpu->next_cpu ?: first_cpu;for (;;) {
qemu-thread.c new file +78 -0
78 lines not on this page · raw message
qemu-thread.h new file +27 -0
@@ -0,0 +1,27 @@ +#include "semaphore.h" +#include "pthread.h" + +struct QemuSem { +sem_t sema; +}; + +struct QemuThread { +pthread_t thread; +}; + +typedef struct QemuSem QemuSem ; +typedef struct QemuThread QemuThread ; + +int qemu_sem_init (QemuSem *sem); +int qemu_sem_lock (QemuSem *sem); +int qemu_sem_trylock (QemuSem *sem); +int qemu_sem_timedlock (QemuSem *sem,unsigned int msecs); +int qemu_sem_unlock (QemuSem *sem); + +int qemu_thread_create (QemuThread *thread, +void *(*start_routine)(void *), +void *arg); +int qemu_thread_signal (QemuThread *thread,int sig); +void qemu_thread_self (QemuThread *thread); +int qemu_thread_equal (QemuThread *thread1,QemuThread *thread2); +
[Qemu-devel] Re: [patch 1/2] qemu: sem/thread helpers
Quoted 35 lines · Marcelo Tosatti
Index: qemu/Makefile.target
===================================================================Makefile.target +1
-1@@ -500,7 +500,7 @@ endif #CONFIG_BSD_USER # System emulator target ifndef CONFIG_USER_ONLY -OBJS=vl.o osdep.o monitor.o pci.o loader.o isa_mmio.o machine.o +OBJS=vl.o qemu-thread.o osdep.o monitor.o pci.o loader.o isa_mmio.o machine.o # virtio has to be here due to weird dependency between PCI and virtio-net. # need to fix this properly OBJS+=virtio.o virtio-blk.o virtio-balloon.o virtio-net.o virtio-console.oIndex:
qemu/vl.c
===================================================================vl.c +3
-0@@ -36,6 +36,7 @@#include "gdbstub.h" #include "qemu-timer.h" #include "qemu-char.h" +#include "qemu-thread.h" #include "cache-utils.h" #include "block.h" #include "audio/audio.h" @@ -263,6 +264,8 @@ static QEMUTimer *nographic_timer;uint8_t qemu_uuid[16 ]; +QemuSem qemu_sem; +/***********************************************************/ /* x86 ISA bus support */
vl.c void main_loop_wait(int timeout) · line 3660
Quoted 8 lines
vl.c +3
-0@@ -3650,7 +3653,10 @@ void main_loop_wait(int timeout) slirp_select_fill(&nfds, &rfds, &wfds, &xfds); }#endif + +qemu_sem_unlock (&qemu_sem); ret =select (nfds +1 , &rfds, &wfds, &xfds, &tv); +qemu_sem_lock (&qemu_sem);
I've looked at this a number of times, so I feel comfortable with this, but I think it's subtle enough that it deserves a comment.
main_loop_wait() *must* not assume any global state is consistent across select() invocations. We should make that very clear as it's a detail that can be easily missed.
Regards,
Anthony Liguori
Re: [Qemu-devel] [patch 1/2] qemu: sem/thread helpers
In reply to Marcelo Tosattiqemu-thread.h line 16
Quoted 2 lines
qemu-thread.h new file +2
-0+int qemu_sem_init (QemuSem *sem); +int qemu_sem_lock (QemuSem *sem);
Please don't introduce yet annother set of locking routines.
Also, your function names suggest you actually want a mutex, not a semaphore.
Review
Quoted 2 lines
(quoted diff) +2
-0+int qemu_sem_lock(QemuSem *sem) +{
This is bogus. You never check the return value.
Quoted 1 line
+QemuSem
qemu_sem;
It's entirely unclear what is actually protected by the semaphore.
What exactly does the IO thread do? AFAICS device MMIO is still run from within the CPU thread. Device code is not threadsafe (and probably never will be), so you can't run any of the device callbacks in the IO thread either. Doesn't seem like there's a lot left for it to do...
Paul
Re: [Qemu-devel] [patch 1/2] qemu: sem/thread helpers
Quoted 7 lines · Paul Brook
Quoted 2 lines
+QemuSem
qemu_sem;It's entirely unclear what is actually protected by the semaphore.
What exactly does the IO thread do? AFAICS device MMIO is still run from within the CPU thread. Device code is not threadsafe (and probably never will be), so you can't run any of the device callbacks in the IO thread either. Doesn't seem like there's a lot left for it to do...
The goal is to drop qemu_mutex while the VCPU is running to allow the device model to run while the VCPU executes. This is not currently safe with TCG but it is safe when using KVM.
There are a couple reasons to drop qemu_mutex while running the VCPU. When using KVM's in-kernel APIC, hlt emulation occurs within the kernel. This means that the KVM_RUN ioctl blocks indefinitely. We currently don't use in-kernel APIC emulation in upstream QEMU's KVM support as this set of patches is a pre-requisite for that.
This also enables true SMP support in KVM. You can allow multiple VCPUs to run concurrently once you're dropping qemu_mutex during VCPU execution.
The VCPU threads have to acquire qemu_mutex once they drop back to QEMU (to handle MMIO, for instance), but this only happens during IO.
From an infrastructure perspective, this is a feature for KVM but also a step in a better direction for TCG too. If TCG can be made to allow qemu_mutex to be dropped (perhaps for x86->x86 translation, as a start), then the same infrastructure can be used for true SMP support with TCG.
Once we have this, we can start making some of the device model code thread safe too...
Regards,
Anthony Liguori
Regards,
Anthony Liguori
Quoted 2 lines
Paul
Re: [Qemu-devel] [patch 1/2] qemu: sem/thread helpers
In reply to Paul BrookQuoted 3 lines · Paul Brook
Quoted 1 line
+QemuSem
qemu_sem;It's entirely unclear what is actually protected by the semaphore.
Everything at the moment.
Quoted 4 lines
What exactly does the IO thread do? AFAICS device MMIO is still run from within the CPU thread. Device code is not threadsafe (and probably never will be), so you can't run any of the device callbacks in the IO thread either. Doesn't seem like there's a lot left for it to do...
There are things which can run in parallel. VNC dirty update, read()'ing of data into QEMU, for two examples.
Re: [Qemu-devel] [patch 1/2] qemu: sem/thread helpers
In reply to Paul BrookQuoted 4 lines · Paul Brook
qemu-thread.h line 16
Quoted 2 lines
qemu-thread.h new file +2
-0+int qemu_sem_init (QemuSem *sem); +int qemu_sem_lock (QemuSem *sem);Please don't introduce yet annother set of locking routines.
I don't see any mutexes around? Or do you mean the wrappers around pthread mutexes/threads are bad? I assume it might be useful for the Windows port.
Re: [Qemu-devel] [patch 1/2] qemu: sem/thread helpers
In reply to Marcelo Tosattiqemu-thread.h line 6
Quoted 5 lines · Marcelo Tosatti
qemu-thread.h new file +4
-0+ +struct QemuSem { +sem_t sema; +};
Why aren't you using pthread_mutex_t?
Signature · 1 line
error compiling committee.c: too many arguments to function
Re: [Qemu-devel] [patch 1/2] qemu: sem/thread helpers
Quoted 8 lines · Avi Kivity
qemu-thread.h line 6
Quoted 5 lines · Marcelo Tosatti
qemu-thread.h new file +4
-0+ +struct QemuSem { +sem_t sema; +};Why aren't you using
pthread_mutex_t?
Because we want the thread waiting on the lock to be awakened once the holder releases it. This does not happen with pthread mutexes AFAIK.
Re: [Qemu-devel] [patch 1/2] qemu: sem/thread helpers
Quoted 9 lines · Marcelo Tosatti
Quoted 7 lines · Avi Kivity
qemu-thread.h line 6
Quoted 4 lines · Marcelo Tosatti
qemu-thread.h new file +4
-0+ +struct QemuSem { +sem_t sema; +};Why aren't you using
pthread_mutex_t?
Quoted 3 lines
Because we want the thread waiting on the lock to be awakened once the holder releases it. This does not happen with pthread mutexes AFAIK.
Really? I'd be surprised if there's any difference. The whole point of a mutex is that it blocks until the other thread releases it. At best it sounds like you're making fairly sketchy assumptions about the host OS scheduler heuristics.
Paul
Re: [Qemu-devel] [patch 1/2] qemu: sem/thread helpers
Quoted 10 lines · Paul Brook
Quoted 9 lines · Marcelo Tosatti
Quoted 7 lines · Avi Kivity
qemu-thread.h line 6
Quoted 4 lines · Marcelo Tosatti
qemu-thread.h new file +4
-0+ +struct QemuSem { +sem_t sema; +};Why aren't you using
pthread_mutex_t?
Quoted 8 lines
Quoted 3 lines
Because we want the thread waiting on the lock to be awakened once the holder releases it. This does not happen with pthread mutexes AFAIK.
Really? I'd be surprised if there's any difference. The whole point of a mutex is that it blocks until the other thread releases it. At best it sounds like you're making fairly sketchy assumptions about the host OS scheduler heuristics.
Actually scratch that. Of course mutex_unlock wakes up waiters. Will send a better patch soon.
PATCH 2/2 qemu: separate thread for io
In reply to Marcelo Tosattiexec.c +14 -0
@@ -1513,6 +1513,20 @@ void cpu_interrupt(CPUState *env, int ma/* FIXME: This is probably not threadsafe. A different thread could be in the middle of a read-modify-write operation. */ env->interrupt_request |= mask; + +switch (mask) { +case CPU_INTERRUPT_HARD : +case CPU_INTERRUPT_SMI : +case CPU_INTERRUPT_NMI : +case CPU_INTERRUPT_EXIT : +/* +* only unlink the TB's if we're called from cpu thread context, +* otherwise signal cpu thread to do it. +*/ +if (qemu_notify_event (env)) +return ; + } +#if defined(USE_NPTL )/* FIXME: TB unchaining isn't SMP safe. For now just ignore the problem and hope the cpu will stop of its own accord. For userspace
qemu-common.h +2 -0
@@ -190,6 +190,8 @@ int cpu_load(QEMUFile *f, void *opaque,/* Force QEMU to stop what it's doing and service IO */ void qemu_service_io (void ); +void main_loop_break (void ); +int qemu_notify_event (void *env); typedef struct QEMUIOVector { struct iovec *iov;
vl.c +143 -12
209 lines not on this page · raw message