Skip to contentSkip to conversation

queue_work proposal

RFC Superseded Glauber Costa 3 participants 9 messages Sep 3, 2009

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

RFC 1/1 queue_work proposal

Hi guys

In this patch, I am attaching an early version of a new "on_vcpu" mechanism (after making it generic, I saw no reason to keep its name). It allows us to guarantee that a piece of code will be executed in a certain vcpu, indicated by a CPUState.

I am sorry for the big patch, I just dumped what I had so we can have early directions. When it comes to submission state, I'll split it accordingly.

As we discussed these days at qemu-devel, I am using pthread_set/get_specific for dealing with thread-local variables. Note that they are not used from signal handlers. A first optimization would be to use TLS variables where available.

In vl.c, I am providing a version of queue_work for the IO-thread, and other for normal operation. The "normal" one should fix the problems Jan is having, since it does nothing more than just issuing the function we want to execute.

The io-thread version is tested with both tcg and kvm, and works (to the extent they were working before, which in kvm case, is not much)

---
 cpu-all.h  |    3 ++
 cpu-defs.h |   15 +++++++++++-
 exec.c     |    1 +
 kvm-all.c  |   58 +++++++++++++++++++----------------------------
 kvm.h      |    7 +++++
 vl.c       |   73 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 6 files changed, 122 insertions(+), 35 deletions(-)
cpu-all.h +3 -0
index 1a6a812..120510b 100644
@@ -763,6 +763,9 @@ extern CPUState *cpu_single_env;
 extern int64_t qemu_icount;
 extern int use_icount;
 
+void qemu_queue_work(CPUState *env, void (*func)(void *data), void *data, int wait);
+void qemu_flush_work(CPUState *env);
+
 #define CPU_INTERRUPT_HARD   0x02 /* hardware interrupt pending */
 #define CPU_INTERRUPT_EXITTB 0x04 /* exit the current TB (use for x86 a20 case) */
 #define CPU_INTERRUPT_TIMER  0x08 /* internal timer exception pending */
cpu-defs.h +14 -1
index b6c8b95..6b564cf 100644
@@ -134,6 +134,15 @@ typedef struct CPUWatchpoint {
     TAILQ_ENTRY(CPUWatchpoint) entry;
 } CPUWatchpoint;
 
+typedef struct QemuWorkItem {
+    void (*func)(void *data);
+    void *data;
+    int done;
+    int wait;
+    TAILQ_ENTRY(QemuWorkItem) entry;
+} QemuWorkItem;
+
+
 #define CPU_TEMP_BUF_NLONGS 128
 #define CPU_COMMON                                                      \
     struct TranslationBlock *current_tb; /* currently executing TB  */  \
@@ -175,6 +184,9 @@ typedef struct CPUWatchpoint {
     TAILQ_HEAD(watchpoints_head, CPUWatchpoint) watchpoints;            \
     CPUWatchpoint *watchpoint_hit;                                      \
                                                                         \
+    TAILQ_HEAD(queued_work_head, QemuWorkItem) queued_work;             \
+    uint64_t queued_local, queued_total;                                \
+                                                                        \
     struct GDBRegisterState *gdb_regs;                                  \
                                                                         \
     /* Core interrupt code */                                           \
@@ -194,9 +206,10 @@ typedef struct CPUWatchpoint {
     uint32_t created;                                                   \
     struct QemuThread *thread;                                          \
     struct QemuCond *halt_cond;                                         \
+    struct QemuCond *work_cond;                                         \
     const char *cpu_model_str;                                          \
     struct KVMState *kvm_state;                                         \
     struct kvm_run *kvm_run;                                            \
-    int kvm_fd;
+    int kvm_fd;                                                         \
 
 #endif
exec.c +1 -0
index 21f10b9..2db5e5b 100644
@@ -577,6 +577,7 @@ void cpu_exec_init(CPUState *env)
     env->numa_node = 0;
     TAILQ_INIT(&env->breakpoints);
     TAILQ_INIT(&env->watchpoints);
+    TAILQ_INIT(&env->queued_work);
     *penv = env;
 #if defined(CONFIG_USER_ONLY)
     cpu_list_unlock();
kvm-all.c +24 -34

92 lines not on this page · raw message

kvm.h +7 -0
index dbe825f..7595141 100644
@@ -139,4 +139,11 @@ static inline void cpu_synchronize_state(CPUState *env)
     }
 }
 
+typedef struct KVMIoctl {
+    int fd;
+    int type;
+    int ret;
+    void *data;
+} KVMIoctl;
+
 #endif
vl.c +73 -0

127 lines not on this page · raw message

Signature · 1 line
1.6.2.2
#1 lore raw

vl.c void qemu_cpu_kick(void *_env) · line 3843

Quoted 5 lines
vl.c +4 -0
+    env->queued_total++;
+
+    if (env == qemu_get_current_env()) {
+        env->queued_total++;

Why increment twice? (though queued_total is write only and queued_local is unused, so...)

Quoted 5 lines
vl.c +5 -0
+        func(data);
+        return;
+    }
+
+    wii = qemu_mallocz(sizeof(*wii));

vl.c void qemu_cpu_kick(void *_env) · line 3858

Quoted 10 lines
vl.c +10 -0
+    wii->func = func;
+    wii->data = data;
+    wii->wait = wait;
+    TAILQ_INSERT_TAIL(&env->queued_work, wii, entry);
+
+    qemu_thread_signal(env->thread, SIGUSR1);
+
+    while (wait&&  !wii->done) {
+        qemu_cond_wait(env->work_cond,&qemu_global_mutex);
+    }

You need to lock qemu_global_mutex around this while statement, or to add env->queue_mutex and include the TAILQ_INSERT_TAIL in the mutex.

Paolo

#2 lore raw
Quoted 8 lines · Paolo Bonzini

vl.c void qemu_cpu_kick(void *_env) · line 3843

Quoted 5 lines
vl.c +4 -0
+    env->queued_total++;
+
+    if (env == qemu_get_current_env()) {
+        env->queued_total++;

Why increment twice? (though queued_total is write only and queued_local is unused, so...)

yeah, you got it =p

As I said, I just dumped whatever I had.

Quoted 11 lines
Quoted 11 lines
vl.c +10 -0
+        func(data);
+        return;
+    }
+
+    wii = qemu_mallocz(sizeof(*wii));
+    wii->func = func;
+    wii->data = data;
+    wii->wait = wait;
+    TAILQ_INSERT_TAIL(&env->queued_work, wii, entry);
+
Quoted 8 lines

vl.c void qemu_cpu_kick(void *_env) · line 3858

Quoted 5 lines
vl.c +5 -0
+    qemu_thread_signal(env->thread, SIGUSR1);
+
+    while (wait&&  !wii->done) {
+        qemu_cond_wait(env->work_cond,&qemu_global_mutex);
+    }

You need to lock qemu_global_mutex around this while statement, or to add env->queue_mutex and include the TAILQ_INSERT_TAIL in the mutex.

Thanks for catching. The later is clearly preferred , IMHO, for scalability purposes.

#3 lore raw
In reply to Glauber Costa
Quoted 13 lines · Glauber Costa

Hi guys

In this patch, I am attaching an early version of a new "on_vcpu" mechanism (after making it generic, I saw no reason to keep its name). It allows us to guarantee that a piece of code will be executed in a certain vcpu, indicated by a CPUState.

I am sorry for the big patch, I just dumped what I had so we can have early directions. When it comes to submission state, I'll split it accordingly.

As we discussed these days at qemu-devel, I am using pthread_set/get_specific for dealing with thread-local variables. Note that they are not used from signal handlers. A first optimization would be to use TLS variables where available.

Quoted 7 lines

In vl.c, I am providing a version of queue_work for the IO-thread, and other for normal operation. The "normal" one should fix the problems Jan is having, since it does nothing more than just issuing the function we want to execute.

The io-thread version is tested with both tcg and kvm, and works (to the extent they were working before, which in kvm case, is not much)

on_vcpu() and queue_work() are fundamentally different (yes, I see the wait parameter, and I think there should be two separate functions for such different behaviours).

Why do we need queue_work() in the first place?

Is there a way to limit the queue size to prevent overflow?

Signature · 1 line
error compiling committee.c: too many arguments to function
#4 lore raw
Quoted 18 lines · Avi Kivity
Quoted 17 lines · Glauber Costa

Hi guys

In this patch, I am attaching an early version of a new "on_vcpu" mechanism (after making it generic, I saw no reason to keep its name). It allows us to guarantee that a piece of code will be executed in a certain vcpu, indicated by a CPUState.

I am sorry for the big patch, I just dumped what I had so we can have early directions. When it comes to submission state, I'll split it accordingly.

As we discussed these days at qemu-devel, I am using pthread_set/get_specific for dealing with thread-local variables. Note that they are not used from signal handlers. A first optimization would be to use TLS variables where available.

In vl.c, I am providing a version of queue_work for the IO-thread, and other for normal operation. The "normal" one should fix the problems Jan is having, since it does nothing more than just issuing the function we want to execute.

Quoted 7 lines
Quoted 3 lines

The io-thread version is tested with both tcg and kvm, and works (to the extent they were working before, which in kvm case, is not much)

on_vcpu() and queue_work() are fundamentally different (yes, I see the wait parameter, and I think there should be two separate functions for such different behaviours).

Therefore, the name change. The exact on_vcpu behaviour, however, can be implemented ontop of queue_work(). Instead of doing that, I opted for using it implicitly inside kvm_vcpu_ioctl, to guarantee that vcpu ioctls will always run on the right thread context. Looking at qemu-kvm, it seems that there are a couple of other functions that are not ioctls, and need on_vcpu semantics. Using them becomes a simple matter of doing:

   queue_work(env, func, data, 1);

I really don't see the big difference you point. They are both there to force a specific function to be executed in the right thread context.

Quoted 2 lines

Why do we need queue_work() in the first place?

To force a function to be executed in the correct thread context. Why do we need on_vcpu in the first place?

Quoted 2 lines

Is there a way to limit the queue size to prevent overflow?

It can be, but it gets awkward. What do you do when you want a function needs to execute on another thread, but you can't? Block it? Refuse?

We could pick one, but I see no need. The vast majority of work will never get queued, since we'll be in the right context already.

#5 lore raw
Quoted 7 lines · Glauber Costa
Quoted 5 lines

on_vcpu() and queue_work() are fundamentally different (yes, I see the wait parameter, and I think there should be two separate functions for such different behaviours).

Therefore, the name change. The exact on_vcpu behaviour, however, can be implemented ontop of queue_work().

Will there be any use for asynchronous queue_work()?

It's a dangerous API.

Quoted 3 lines

Instead of doing that, I opted for using it
implicitly inside kvm_vcpu_ioctl, to guarantee that vcpu ioctls will always run on the right thread context.

I think it's reasonable to demand that whoever calls kvm_vcpu_ioctl() know what they are doing (and they'll get surprising results if it switches threads implicitly).

Quoted 9 lines

Looking at qemu-kvm, it seems that there are a couple
of other functions that are not ioctls, and need on_vcpu semantics. Using them becomes a simple matter of doing:

    queue_work(env, func, data, 1);

I really don't see the big difference you point. They are both there to force a specific function to be executed in the right thread context.

One of them is synchronous, meaning the data can live on stack and no special synchronization is needed, while the other is synchronous, meaning explicit memory management and end-of-work synchronization is needed.

Quoted 5 lines
Quoted 2 lines

Why do we need queue_work() in the first place?

To force a function to be executed in the correct thread context. Why do we need on_vcpu in the first place?

on_vcpu() is a subset of queue_work(). I meant, why to we need the extra functionality?

Quoted 5 lines
Quoted 2 lines

Is there a way to limit the queue size to prevent overflow?

It can be, but it gets awkward. What do you do when you want a function needs to execute on another thread, but you can't? Block it? Refuse?

What if the thread is busy? You grow the queue to an unbounded size?

Quoted 3 lines

We could pick one, but I see no need. The vast majority of work will never get queued, since we'll be in the right context already.

A more powerful API comes with increased responsibilities.

Signature · 1 line
error compiling committee.c: too many arguments to function
#6 lore raw
Quoted 6 lines · Avi Kivity
Quoted 5 lines · Glauber Costa
Quoted 5 lines

on_vcpu() and queue_work() are fundamentally different (yes, I see the wait parameter, and I think there should be two separate functions for such different behaviours).

Quoted 6 lines
Quoted 2 lines

Therefore, the name change. The exact on_vcpu behaviour, however, can be implemented ontop of queue_work().

Will there be any use for asynchronous queue_work()?

It's a dangerous API.

Initially, I thought we could use it for batching, if we forced a flush in the end of a sequence of operations. This can makes things faster if we are doing a bunch of calls in a row, from the wrong thread.

Quoted 8 lines
Quoted 4 lines

Instead of doing that, I opted for using it
implicitly inside kvm_vcpu_ioctl, to guarantee that vcpu ioctls will always run on the right thread context.

I think it's reasonable to demand that whoever calls kvm_vcpu_ioctl() know what they are doing (and they'll get surprising results if it switches threads implicitly).

I respectfully disagree. Not that I want people not to know what they are doing, but I believe that, forcing something that can only run in a specific thread to be run there, provides us with a much saner interface, that will make code a lot more readable and maintainable.

Quoted 7 lines
Quoted 7 lines

Looking at qemu-kvm, it seems that there are a couple
of other functions that are not ioctls, and need on_vcpu semantics. Using them becomes a simple matter of doing:

    queue_work(env, func, data, 1);
Quoted 8 lines
Quoted 3 lines

I really don't see the big difference you point. They are both there to force a specific function to be executed in the right thread context.

One of them is synchronous, meaning the data can live on stack and no special synchronization is needed, while the other is synchronous, meaning explicit memory management and end-of-work synchronization is needed.

I will assume you meant "the other is assynchronous". It does not need to be. I though about including the assynchronous version in this RFC to let doors open for performance improvements *if* we needed them. But again: the absolute majority of the calls will be local. So it is not that important.

Quoted 9 lines
Quoted 6 lines
Quoted 3 lines

Why do we need queue_work() in the first place?

To force a function to be executed in the correct thread context. Why do we need on_vcpu in the first place?

on_vcpu() is a subset of queue_work(). I meant, why to we need the extra functionality?

As I said, if you oppose it hardly, we don't really need to.

Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

Is there a way to limit the queue size to prevent overflow?

It can be, but it gets awkward. What do you do when you want a function needs to execute on another thread, but you can't? Block it? Refuse?

Quoted 7 lines

What if the thread is busy? You grow the queue to an unbounded size?

Quoted 3 lines

We could pick one, but I see no need. The vast majority of work will never get queued, since we'll be in the right context already.

A more powerful API comes with increased responsibilities.

You suddenly sounds like spider man.

#7 lore raw
Quoted 13 lines · Glauber Costa
Quoted 11 lines · Avi Kivity
Quoted 9 lines · Glauber Costa
Quoted 6 lines

on_vcpu() and queue_work() are fundamentally different (yes, I see the wait parameter, and I think there should be two separate functions for such different behaviours).

Therefore, the name change. The exact on_vcpu behaviour, however, can be implemented ontop of queue_work().

Quoted 8 lines
Quoted 4 lines

Will there be any use for asynchronous queue_work()?

It's a dangerous API.

Initially, I thought we could use it for batching, if we forced a flush in the end of a sequence of operations. This can makes things faster if we are doing a bunch of calls in a row, from the wrong thread.

It's a lot easier and safer to write a function that does your batch job and on_vcpu() it.

Quoted 9 lines
Quoted 4 lines

I think it's reasonable to demand that whoever calls kvm_vcpu_ioctl() know what they are doing (and they'll get surprising results if it switches threads implicitly).

I respectfully disagree. Not that I want people not to know what they are doing, but I believe that, forcing something that can only run in a specific thread to be run there, provides us with a much saner interface, that will make code a lot more readable and maintainable.

Example:

kvm_vcpu_ioctl(get_regs)
regs->rflags |= some_flag
kvm_vcpu_ioctl(set_regs)

This looks totally sane but is racy if kvm_vcpu_ioctl() does an implicit on_vcpu().

Quoted 10 lines
Quoted 5 lines

One of them is synchronous, meaning the data can live on stack and no special synchronization is needed, while the other is synchronous, meaning explicit memory management and end-of-work synchronization is needed.

I will assume you meant "the other is assynchronous". It does not need to be. I though about including the assynchronous version in this RFC to let doors open for performance improvements *if* we needed them. But again: the absolute majority of the calls will be local. So it is not that important.

Then there's even less reason to include the async version.

Quoted 5 lines
Quoted 3 lines

on_vcpu() is a subset of queue_work(). I meant, why to we need the extra functionality?

As I said, if you oppose it hardly, we don't really need to.

I do oppose it, but the reason for not including it should be the reasons I cited, not that I oppose it.

Quoted 4 lines
Quoted 2 lines

A more powerful API comes with increased responsibilities.

You suddenly sounds like spider man.

I hate it when I get unmasked.

Signature · 1 line
error compiling committee.c: too many arguments to function
#8 lore raw
Quoted 18 lines · Avi Kivity
Quoted 17 lines · Glauber Costa
Quoted 15 lines · Avi Kivity
Quoted 9 lines · Glauber Costa
Quoted 6 lines

on_vcpu() and queue_work() are fundamentally different (yes, I see the wait parameter, and I think there should be two separate functions for such different behaviours).

Therefore, the name change. The exact on_vcpu behaviour, however, can be implemented ontop of queue_work().

Will there be any use for asynchronous queue_work()?

It's a dangerous API.

Quoted 7 lines
Quoted 4 lines

Initially, I thought we could use it for batching, if we forced a flush in the end of a sequence of operations. This can makes things faster if we are doing a bunch of calls in a row, from the wrong thread.

It's a lot easier and safer to write a function that does your batch job and on_vcpu() it.

agree.

Quoted 11 lines
Quoted 10 lines
Quoted 5 lines

I think it's reasonable to demand that whoever calls kvm_vcpu_ioctl() know what they are doing (and they'll get surprising results if it switches threads implicitly).

I respectfully disagree. Not that I want people not to know what they are doing, but I believe that, forcing something that can only run in a specific thread to be run there, provides us with a much saner interface, that will make code a lot more readable and maintainable.

Quoted 8 lines

Example:

kvm_vcpu_ioctl(get_regs)
regs->rflags |= some_flag
kvm_vcpu_ioctl(set_regs)

This looks totally sane but is racy if kvm_vcpu_ioctl() does an implicit on_vcpu().

Not at all. If we do queue_work once, it will execute whatever function you ask for in the correct thread, and for there on, everything will be local. So even if you nest functions that call queue_work themselves, only the first will involve synchronization at all.

Quoted 7 lines
Quoted 5 lines

I will assume you meant "the other is assynchronous". It does not need to be. I though about including the assynchronous version in this RFC to let doors open for performance improvements *if* we needed them. But again: the absolute majority of the calls will be local. So it is not that important.

Then there's even less reason to include the async version.

I am pretty much convinced by now.

Quoted 9 lines
Quoted 6 lines
Quoted 4 lines

on_vcpu() is a subset of queue_work(). I meant, why to we need the extra functionality?

As I said, if you oppose it hardly, we don't really need to.

I do oppose it, but the reason for not including it should be the reasons I cited, not that I oppose it.

For a masked vigilante like you, I thought it was clear enough that "fierce opposition" automatically meant you got strong reasons, and were actually right.