Skip to contentSkip to conversation

RFC: separate thread for IO

PATCH In review Marcelo Tosatti 4 participants 12 messages Mar 11, 2009 – Mar 18, 2009 Kbuild

lore
Share & tools
lore
#0 lore raw

cover RFC: separate thread for IO

Introduce a separate thread for handling IO. Not intended for inclusion, but should be something for people to comment on.

#1 lore raw

PATCH 1/2 qemu: sem/thread helpers

Makefile.target +1 -1
Index: qemu/Makefile.target
===================================================================
@@ -500,7 +500,7 @@ endif #CONFIG_BSD_USER
 # System emulator target
 ifndef CONFIG_USER_ONLY
 
-OBJS=vl.o osdep.o monitor.o pci.o loader.o isa_mmio.o machine.o
+OBJS=vl.o qemu-thread.o osdep.o monitor.o pci.o loader.o isa_mmio.o machine.o
 # virtio has to be here due to weird dependency between PCI and virtio-net.
 # need to fix this properly
 OBJS+=virtio.o virtio-blk.o virtio-balloon.o virtio-net.o virtio-console.o
vl.c +9 -0
Index: qemu/vl.c
===================================================================
@@ -36,6 +36,7 @@
 #include "gdbstub.h"
 #include "qemu-timer.h"
 #include "qemu-char.h"
+#include "qemu-thread.h"
 #include "cache-utils.h"
 #include "block.h"
 #include "audio/audio.h"
@@ -263,6 +264,8 @@ static QEMUTimer *nographic_timer;
 
 uint8_t qemu_uuid[16];
 
+QemuSem qemu_sem;
+
 /***********************************************************/
 /* x86 ISA bus support */
 
@@ -3650,7 +3653,10 @@ void main_loop_wait(int timeout)
         slirp_select_fill(&nfds, &rfds, &wfds, &xfds);
     }
 #endif
+
+    qemu_sem_unlock(&qemu_sem);
     ret = select(nfds + 1, &rfds, &wfds, &xfds, &tv);
+    qemu_sem_lock(&qemu_sem);
     if (ret > 0) {
         IOHandlerRecord **pioh;
 
@@ -3708,6 +3714,9 @@ static int main_loop(void)
 #endif
     CPUState *env;
 
+    qemu_sem_init(&qemu_sem);
+    qemu_sem_lock(&qemu_sem);
+
     cur_cpu = first_cpu;
     next_cpu = cur_cpu->next_cpu ?: first_cpu;
     for(;;) {
qemu-thread.c new file +78 -0

78 lines not on this page · raw message

qemu-thread.h new file +27 -0
Index: qemu/qemu-thread.h
===================================================================
@@ -0,0 +1,27 @@
+#include "semaphore.h"
+#include "pthread.h"
+
+struct QemuSem {
+    sem_t sema;
+};
+
+struct QemuThread {
+    pthread_t thread;
+};
+
+typedef struct QemuSem QemuSem;
+typedef struct QemuThread QemuThread;
+
+int qemu_sem_init(QemuSem *sem);
+int qemu_sem_lock(QemuSem *sem);
+int qemu_sem_trylock(QemuSem *sem);
+int qemu_sem_timedlock(QemuSem *sem, unsigned int msecs);
+int qemu_sem_unlock(QemuSem *sem);
+
+int qemu_thread_create(QemuThread *thread,
+                       void *(*start_routine)(void*),
+                       void *arg);
+int qemu_thread_signal(QemuThread *thread, int sig);
+void qemu_thread_self(QemuThread *thread);
+int qemu_thread_equal(QemuThread *thread1, QemuThread *thread2);
+
#2 lore raw

[Qemu-devel] Re: [patch 1/2] qemu: sem/thread helpers

Quoted 35 lines · Marcelo Tosatti

Index: qemu/Makefile.target
===================================================================

Makefile.target +1 -1
@@ -500,7 +500,7 @@ endif #CONFIG_BSD_USER
 # System emulator target
 ifndef CONFIG_USER_ONLY
 
-OBJS=vl.o osdep.o monitor.o pci.o loader.o isa_mmio.o machine.o
+OBJS=vl.o qemu-thread.o osdep.o monitor.o pci.o loader.o isa_mmio.o machine.o
 # virtio has to be here due to weird dependency between PCI and virtio-net.
 # need to fix this properly
 OBJS+=virtio.o virtio-blk.o virtio-balloon.o virtio-net.o virtio-console.o

Index: qemu/vl.c
===================================================================

vl.c +3 -0
@@ -36,6 +36,7 @@
 #include "gdbstub.h"
 #include "qemu-timer.h"
 #include "qemu-char.h"
+#include "qemu-thread.h"
 #include "cache-utils.h"
 #include "block.h"
 #include "audio/audio.h"
@@ -263,6 +264,8 @@ static QEMUTimer *nographic_timer;
 
 uint8_t qemu_uuid[16];
 
+QemuSem qemu_sem;
+
 /***********************************************************/
 /* x86 ISA bus support */
 

vl.c void main_loop_wait(int timeout) · line 3660

Quoted 8 lines
vl.c +3 -0
@@ -3650,7 +3653,10 @@ void main_loop_wait(int timeout)
         slirp_select_fill(&nfds, &rfds, &wfds, &xfds);
     }
 #endif
+
+    qemu_sem_unlock(&qemu_sem);
     ret = select(nfds + 1, &rfds, &wfds, &xfds, &tv);
+    qemu_sem_lock(&qemu_sem);
 

I've looked at this a number of times, so I feel comfortable with this, but I think it's subtle enough that it deserves a comment.

main_loop_wait() *must* not assume any global state is consistent across select() invocations. We should make that very clear as it's a detail that can be easily missed.

Regards,

Anthony Liguori

#3 lore raw

Re: [Qemu-devel] [patch 1/2] qemu: sem/thread helpers

In reply to Marcelo Tosatti

qemu-thread.h line 16

Quoted 2 lines
qemu-thread.h new file +2 -0
+int qemu_sem_init(QemuSem *sem);
+int qemu_sem_lock(QemuSem *sem);

Please don't introduce yet annother set of locking routines.

Also, your function names suggest you actually want a mutex, not a semaphore.

Review

Quoted 2 lines
(quoted diff) +2 -0
+int qemu_sem_lock(QemuSem *sem)
+{

This is bogus. You never check the return value.

Quoted 1 line

+QemuSem qemu_sem;

It's entirely unclear what is actually protected by the semaphore.

What exactly does the IO thread do? AFAICS device MMIO is still run from within the CPU thread. Device code is not threadsafe (and probably never will be), so you can't run any of the device callbacks in the IO thread either. Doesn't seem like there's a lot left for it to do...

Paul

#4 lore raw

Re: [Qemu-devel] [patch 1/2] qemu: sem/thread helpers

Quoted 7 lines · Paul Brook
Quoted 2 lines

+QemuSem qemu_sem;

It's entirely unclear what is actually protected by the semaphore.

What exactly does the IO thread do? AFAICS device MMIO is still run from within the CPU thread. Device code is not threadsafe (and probably never will be), so you can't run any of the device callbacks in the IO thread either. Doesn't seem like there's a lot left for it to do...

The goal is to drop qemu_mutex while the VCPU is running to allow the device model to run while the VCPU executes. This is not currently safe with TCG but it is safe when using KVM.

There are a couple reasons to drop qemu_mutex while running the VCPU. When using KVM's in-kernel APIC, hlt emulation occurs within the kernel. This means that the KVM_RUN ioctl blocks indefinitely. We currently don't use in-kernel APIC emulation in upstream QEMU's KVM support as this set of patches is a pre-requisite for that.

This also enables true SMP support in KVM. You can allow multiple VCPUs to run concurrently once you're dropping qemu_mutex during VCPU execution.

The VCPU threads have to acquire qemu_mutex once they drop back to QEMU (to handle MMIO, for instance), but this only happens during IO.

From an infrastructure perspective, this is a feature for KVM but also a step in a better direction for TCG too. If TCG can be made to allow qemu_mutex to be dropped (perhaps for x86->x86 translation, as a start), then the same infrastructure can be used for true SMP support with TCG.

Once we have this, we can start making some of the device model code thread safe too...

Regards,

Anthony Liguori

Regards,

Anthony Liguori

Quoted 2 lines

Paul

#5 lore raw

Re: [Qemu-devel] [patch 1/2] qemu: sem/thread helpers

In reply to Paul Brook
Quoted 3 lines · Paul Brook
Quoted 1 line

+QemuSem qemu_sem;

It's entirely unclear what is actually protected by the semaphore.

Everything at the moment.

Quoted 4 lines

What exactly does the IO thread do? AFAICS device MMIO is still run from within the CPU thread. Device code is not threadsafe (and probably never will be), so you can't run any of the device callbacks in the IO thread either. Doesn't seem like there's a lot left for it to do...

There are things which can run in parallel. VNC dirty update, read()'ing of data into QEMU, for two examples.

#6 lore raw

Re: [Qemu-devel] [patch 1/2] qemu: sem/thread helpers

In reply to Paul Brook
Quoted 4 lines · Paul Brook

qemu-thread.h line 16

Quoted 2 lines
qemu-thread.h new file +2 -0
+int qemu_sem_init(QemuSem *sem);
+int qemu_sem_lock(QemuSem *sem);

Please don't introduce yet annother set of locking routines.

I don't see any mutexes around? Or do you mean the wrappers around pthread mutexes/threads are bad? I assume it might be useful for the Windows port.

#7 lore raw

Re: [Qemu-devel] [patch 1/2] qemu: sem/thread helpers

In reply to Marcelo Tosatti

qemu-thread.h line 6

Quoted 5 lines · Marcelo Tosatti
qemu-thread.h new file +4 -0
+
+struct QemuSem {
+    sem_t sema;
+};

Why aren't you using pthread_mutex_t?

Signature · 1 line
error compiling committee.c: too many arguments to function
#8 lore raw

Re: [Qemu-devel] [patch 1/2] qemu: sem/thread helpers

Quoted 8 lines · Avi Kivity

qemu-thread.h line 6

Quoted 5 lines · Marcelo Tosatti
qemu-thread.h new file +4 -0
+
+struct QemuSem {
+    sem_t sema;
+};

Why aren't you using pthread_mutex_t?

Because we want the thread waiting on the lock to be awakened once the holder releases it. This does not happen with pthread mutexes AFAIK.

#9 lore raw

Re: [Qemu-devel] [patch 1/2] qemu: sem/thread helpers

Quoted 9 lines · Marcelo Tosatti
Quoted 7 lines · Avi Kivity

qemu-thread.h line 6

Quoted 4 lines · Marcelo Tosatti
qemu-thread.h new file +4 -0
+
+struct QemuSem {
+    sem_t sema;
+};

Why aren't you using pthread_mutex_t?

Quoted 3 lines

Because we want the thread waiting on the lock to be awakened once the holder releases it. This does not happen with pthread mutexes AFAIK.

Really? I'd be surprised if there's any difference. The whole point of a mutex is that it blocks until the other thread releases it. At best it sounds like you're making fairly sketchy assumptions about the host OS scheduler heuristics.

Paul

#10 lore raw

Re: [Qemu-devel] [patch 1/2] qemu: sem/thread helpers

Quoted 10 lines · Paul Brook
Quoted 9 lines · Marcelo Tosatti
Quoted 7 lines · Avi Kivity

qemu-thread.h line 6

Quoted 4 lines · Marcelo Tosatti
qemu-thread.h new file +4 -0
+
+struct QemuSem {
+    sem_t sema;
+};

Why aren't you using pthread_mutex_t?

Quoted 8 lines
Quoted 3 lines

Because we want the thread waiting on the lock to be awakened once the holder releases it. This does not happen with pthread mutexes AFAIK.

Really? I'd be surprised if there's any difference. The whole point of a mutex is that it blocks until the other thread releases it. At best it sounds like you're making fairly sketchy assumptions about the host OS scheduler heuristics.

Actually scratch that. Of course mutex_unlock wakes up waiters. Will send a better patch soon.

#11 lore raw

PATCH 2/2 qemu: separate thread for io

In reply to Marcelo Tosatti
exec.c +14 -0
Index: qemu/exec.c
===================================================================
@@ -1513,6 +1513,20 @@ void cpu_interrupt(CPUState *env, int ma
     /* FIXME: This is probably not threadsafe.  A different thread could
        be in the middle of a read-modify-write operation.  */
     env->interrupt_request |= mask;
+
+    switch(mask) {
+    case CPU_INTERRUPT_HARD:
+    case CPU_INTERRUPT_SMI:
+    case CPU_INTERRUPT_NMI:
+    case CPU_INTERRUPT_EXIT:
+    /*
+     * only unlink the TB's if we're called from cpu thread context,
+     * otherwise signal cpu thread to do it.
+     */
+        if (qemu_notify_event(env))
+               return;
+    }
+
 #if defined(USE_NPTL)
     /* FIXME: TB unchaining isn't SMP safe.  For now just ignore the
        problem and hope the cpu will stop of its own accord.  For userspace
qemu-common.h +2 -0
Index: qemu/qemu-common.h
===================================================================
@@ -190,6 +190,8 @@ int cpu_load(QEMUFile *f, void *opaque, 
 
 /* Force QEMU to stop what it's doing and service IO */
 void qemu_service_io(void);
+void main_loop_break(void);
+int qemu_notify_event(void *env);
 
 typedef struct QEMUIOVector {
     struct iovec *iov;
vl.c +143 -12

209 lines not on this page · raw message