queue_work proposal
RFC 1/1 queue_work proposal
Hi guys
In this patch, I am attaching an early version of a new "on_vcpu" mechanism (after making it generic, I saw no reason to keep its name). It allows us to guarantee that a piece of code will be executed in a certain vcpu, indicated by a CPUState.
I am sorry for the big patch, I just dumped what I had so we can have early directions. When it comes to submission state, I'll split it accordingly.
As we discussed these days at qemu-devel, I am using pthread_set/get_specific for dealing with thread-local variables. Note that they are not used from signal handlers. A first optimization would be to use TLS variables where available.
In vl.c, I am providing a version of queue_work for the IO-thread, and other for normal operation. The "normal" one should fix the problems Jan is having, since it does nothing more than just issuing the function we want to execute.
The io-thread version is tested with both tcg and kvm, and works (to the extent they were working before, which in kvm case, is not much)
--- cpu-all.h | 3 ++ cpu-defs.h | 15 +++++++++++- exec.c | 1 + kvm-all.c | 58 +++++++++++++++++++---------------------------- kvm.h | 7 +++++ vl.c | 73 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 6 files changed, 122 insertions(+), 35 deletions(-)
cpu-all.h +3 -0
@@ -763,6 +763,9 @@ extern CPUState *cpu_single_env;extern int64_t qemu_icount;extern int use_icount; +void qemu_queue_work (CPUState *env,void (*func)(void *data),void *data,int wait); +void qemu_flush_work (CPUState *env); +#define CPU_INTERRUPT_HARD 0x02/* hardware interrupt pending */ #define CPU_INTERRUPT_EXITTB 0x04/* exit the current TB (use for x86 a20 case) */ #define CPU_INTERRUPT_TIMER 0x08/* internal timer exception pending */
cpu-defs.h +14 -1
@@ -134,6 +134,15 @@ typedef struct CPUWatchpoint {TAILQ_ENTRY (CPUWatchpoint) entry; } CPUWatchpoint; +typedef struct QemuWorkItem { +void (*func)(void *data); +void *data; +int done; +int wait; +TAILQ_ENTRY (QemuWorkItem) entry; +}QemuWorkItem ; + +#define CPU_TEMP_BUF_NLONGS 128#define CPU_COMMON \ struct TranslationBlock *current_tb;/* currently executing TB */ \
@@ -175,6 +184,9 @@ typedef struct CPUWatchpoint { TAILQ_HEAD(watchpoints_head, CPUWatchpoint) watchpoints; \CPUWatchpoint *watchpoint_hit; \ \ + TAILQ_HEAD(queued_work_head, QemuWorkItem) queued_work; \ +uint64_t queued_local, queued_total; \ + \struct GDBRegisterState *gdb_regs; \ \/* Core interrupt code */ \
@@ -194,9 +206,10 @@ typedef struct CPUWatchpoint {uint32_t created; \struct QemuThread *thread; \struct QemuCond *halt_cond; \ +struct QemuCond *work_cond; \const char *cpu_model_str; \struct KVMState *kvm_state; \struct kvm_run *kvm_run; \ -int kvm_fd; +int kvm_fd; \#endif
exec.c +1 -0
@@ -577,6 +577,7 @@ void cpu_exec_init(CPUState *env) env->numa_node =0 ;TAILQ_INIT (&env->breakpoints);TAILQ_INIT (&env->watchpoints); +TAILQ_INIT (&env->queued_work); *penv = env;#if defined(CONFIG_USER_ONLY )cpu_list_unlock ();
kvm-all.c +24 -34
92 lines not on this page · raw message
kvm.h +7 -0
@@ -139,4 +139,11 @@ static inline void cpu_synchronize_state(CPUState *env) } } +typedef struct KVMIoctl { +int fd; +int type; +int ret; +void *data; +}KVMIoctl ; +#endif
vl.c +73 -0
127 lines not on this page · raw message
Signature · 1 line
1.6.2.2
vl.c void qemu_cpu_kick(void *_env) · line 3843
Quoted 5 lines
vl.c +4
-0+ env->queued_total++; + +if (env ==qemu_get_current_env ()) { + env->queued_total++;
Why increment twice? (though queued_total is write only and queued_local is unused, so...)
Quoted 5 lines
vl.c +5
-0+func (data); +return ; + } + + wii =qemu_mallocz (sizeof (*wii));
vl.c void qemu_cpu_kick(void *_env) · line 3858
Quoted 10 lines
vl.c +10
-0+ wii->func = func; + wii->data = data; + wii->wait = wait; +TAILQ_INSERT_TAIL (&env->queued_work, wii, entry); + +qemu_thread_signal (env->thread,SIGUSR1 ); + +while (wait&& !wii->done) { +qemu_cond_wait (env->work_cond,&qemu_global_mutex); + }
You need to lock qemu_global_mutex around this while statement, or to add env->queue_mutex and include the TAILQ_INSERT_TAIL in the mutex.
Paolo
Quoted 8 lines · Paolo Bonzini
vl.c void qemu_cpu_kick(void *_env) · line 3843
Quoted 5 lines
vl.c +4
-0+ env->queued_total++; + +if (env ==qemu_get_current_env ()) { + env->queued_total++;Why increment twice? (though
queued_totalis write only andqueued_localis unused, so...)
yeah, you got it =p
As I said, I just dumped whatever I had.
Quoted 11 lines
Quoted 11 lines
vl.c +10
-0+func (data); +return ; + } + + wii =qemu_mallocz (sizeof (*wii)); + wii->func = func; + wii->data = data; + wii->wait = wait; +TAILQ_INSERT_TAIL (&env->queued_work, wii, entry); +
Quoted 8 lines
vl.c void qemu_cpu_kick(void *_env) · line 3858
Quoted 5 lines
vl.c +5
-0+qemu_thread_signal (env->thread,SIGUSR1 ); + +while (wait&& !wii->done) { +qemu_cond_wait (env->work_cond,&qemu_global_mutex); + }You need to lock
qemu_global_mutexaround this while statement, or to addenv->queue_mutexand include theTAILQ_INSERT_TAILin the mutex.
Thanks for catching. The later is clearly preferred , IMHO, for scalability purposes.
Quoted 13 lines · Glauber Costa
Hi guys
In this patch, I am attaching an early version of a new "
on_vcpu" mechanism (after making it generic, I saw no reason to keep its name). It allows us to guarantee that a piece of code will be executed in a certain vcpu, indicated by a CPUState.I am sorry for the big patch, I just dumped what I had so we can have early directions. When it comes to submission state, I'll split it accordingly.
As we discussed these days at qemu-devel, I am using pthread_set/get_specific for dealing with thread-local variables. Note that they are not used from signal handlers. A first optimization would be to use TLS variables where available.
Quoted 7 lines
In
vl.c, I am providing a version ofqueue_workfor the IO-thread, and other for normal operation. The "normal" one should fix the problems Jan is having, since it does nothing more than just issuing the function we want to execute.The io-thread version is tested with both tcg and kvm, and works (to the extent they were working before, which in kvm case, is not much)
on_vcpu() and queue_work() are fundamentally different (yes, I see the wait parameter, and I think there should be two separate functions for such different behaviours).
Why do we need queue_work() in the first place?
Is there a way to limit the queue size to prevent overflow?
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 18 lines · Avi Kivity
Quoted 17 lines · Glauber Costa
Hi guys
In this patch, I am attaching an early version of a new "
on_vcpu" mechanism (after making it generic, I saw no reason to keep its name). It allows us to guarantee that a piece of code will be executed in a certain vcpu, indicated by a CPUState.I am sorry for the big patch, I just dumped what I had so we can have early directions. When it comes to submission state, I'll split it accordingly.
As we discussed these days at qemu-devel, I am using pthread_set/get_specific for dealing with thread-local variables. Note that they are not used from signal handlers. A first optimization would be to use TLS variables where available.
In
vl.c, I am providing a version ofqueue_workfor the IO-thread, and other for normal operation. The "normal" one should fix the problems Jan is having, since it does nothing more than just issuing the function we want to execute.
Quoted 7 lines
Quoted 3 lines
The io-thread version is tested with both tcg and kvm, and works (to the extent they were working before, which in kvm case, is not much)
on_vcpu()andqueue_work()are fundamentally different (yes, I see the wait parameter, and I think there should be two separate functions for such different behaviours).
Therefore, the name change. The exact on_vcpu behaviour, however, can be implemented ontop of queue_work(). Instead of doing that, I opted for using it implicitly inside kvm_vcpu_ioctl, to guarantee that vcpu ioctls will always run on the right thread context. Looking at qemu-kvm, it seems that there are a couple of other functions that are not ioctls, and need on_vcpu semantics. Using them becomes a simple matter of doing:
queue_work(env, func, data, 1);
I really don't see the big difference you point. They are both there to force a specific function to be executed in the right thread context.
Quoted 2 lines
Why do we need
queue_work()in the first place?
To force a function to be executed in the correct thread context. Why do we need on_vcpu in the first place?
Quoted 2 lines
Is there a way to limit the queue size to prevent overflow?
It can be, but it gets awkward. What do you do when you want a function needs to execute on another thread, but you can't? Block it? Refuse?
We could pick one, but I see no need. The vast majority of work will never get queued, since we'll be in the right context already.
Quoted 7 lines · Glauber Costa
Quoted 5 lines
on_vcpu()andqueue_work()are fundamentally different (yes, I see the wait parameter, and I think there should be two separate functions for such different behaviours).Therefore, the name change. The exact
on_vcpubehaviour, however, can be implemented ontop ofqueue_work().
Will there be any use for asynchronous queue_work()?
It's a dangerous API.
Quoted 3 lines
Instead of doing that, I opted for using it
implicitly insidekvm_vcpu_ioctl, to guarantee that vcpu ioctls will always run on the right thread context.
I think it's reasonable to demand that whoever calls kvm_vcpu_ioctl() know what they are doing (and they'll get surprising results if it switches threads implicitly).
Quoted 9 lines
Looking at qemu-kvm, it seems that there are a couple
of other functions that are not ioctls, and needon_vcpusemantics. Using them becomes a simple matter of doing:queue_work(env, func, data, 1);I really don't see the big difference you point. They are both there to force a specific function to be executed in the right thread context.
One of them is synchronous, meaning the data can live on stack and no special synchronization is needed, while the other is synchronous, meaning explicit memory management and end-of-work synchronization is needed.
Quoted 5 lines
Quoted 2 lines
Why do we need
queue_work()in the first place?To force a function to be executed in the correct thread context. Why do we need
on_vcpuin the first place?
on_vcpu() is a subset of queue_work(). I meant, why to we need the extra functionality?
Quoted 5 lines
Quoted 2 lines
Is there a way to limit the queue size to prevent overflow?
It can be, but it gets awkward. What do you do when you want a function needs to execute on another thread, but you can't? Block it? Refuse?
What if the thread is busy? You grow the queue to an unbounded size?
Quoted 3 lines
We could pick one, but I see no need. The vast majority of work will never get queued, since we'll be in the right context already.
A more powerful API comes with increased responsibilities.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 6 lines · Avi Kivity
Quoted 5 lines · Glauber Costa
Quoted 5 lines
on_vcpu()andqueue_work()are fundamentally different (yes, I see the wait parameter, and I think there should be two separate functions for such different behaviours).
Quoted 6 lines
Quoted 2 lines
Therefore, the name change. The exact
on_vcpubehaviour, however, can be implemented ontop ofqueue_work().Will there be any use for asynchronous
queue_work()?It's a dangerous API.
Initially, I thought we could use it for batching, if we forced a flush in the end of a sequence of operations. This can makes things faster if we are doing a bunch of calls in a row, from the wrong thread.
Quoted 8 lines
Quoted 4 lines
Instead of doing that, I opted for using it
implicitly insidekvm_vcpu_ioctl, to guarantee that vcpu ioctls will always run on the right thread context.I think it's reasonable to demand that whoever calls
kvm_vcpu_ioctl()know what they are doing (and they'll get surprising results if it switches threads implicitly).
I respectfully disagree. Not that I want people not to know what they are doing, but I believe that, forcing something that can only run in a specific thread to be run there, provides us with a much saner interface, that will make code a lot more readable and maintainable.
Quoted 7 lines
Quoted 7 lines
Looking at qemu-kvm, it seems that there are a couple
of other functions that are not ioctls, and needon_vcpusemantics. Using them becomes a simple matter of doing:queue_work(env, func, data, 1);
Quoted 8 lines
Quoted 3 lines
I really don't see the big difference you point. They are both there to force a specific function to be executed in the right thread context.
One of them is synchronous, meaning the data can live on stack and no special synchronization is needed, while the other is synchronous, meaning explicit memory management and end-of-work synchronization is needed.
I will assume you meant "the other is assynchronous". It does not need to be. I though about including the assynchronous version in this RFC to let doors open for performance improvements *if* we needed them. But again: the absolute majority of the calls will be local. So it is not that important.
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
Why do we need
queue_work()in the first place?To force a function to be executed in the correct thread context. Why do we need
on_vcpuin the first place?
on_vcpu()is a subset ofqueue_work(). I meant, why to we need the extra functionality?
As I said, if you oppose it hardly, we don't really need to.
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
Is there a way to limit the queue size to prevent overflow?
It can be, but it gets awkward. What do you do when you want a function needs to execute on another thread, but you can't? Block it? Refuse?
Quoted 7 lines
What if the thread is busy? You grow the queue to an unbounded size?
Quoted 3 lines
We could pick one, but I see no need. The vast majority of work will never get queued, since we'll be in the right context already.
A more powerful API comes with increased responsibilities.
You suddenly sounds like spider man.
Quoted 13 lines · Glauber Costa
Quoted 11 lines · Avi Kivity
Quoted 9 lines · Glauber Costa
Quoted 6 lines
on_vcpu()andqueue_work()are fundamentally different (yes, I see the wait parameter, and I think there should be two separate functions for such different behaviours).Therefore, the name change. The exact
on_vcpubehaviour, however, can be implemented ontop ofqueue_work().
Quoted 8 lines
Quoted 4 lines
Will there be any use for asynchronous
queue_work()?It's a dangerous API.
Initially, I thought we could use it for batching, if we forced a flush in the end of a sequence of operations. This can makes things faster if we are doing a bunch of calls in a row, from the wrong thread.
It's a lot easier and safer to write a function that does your batch job and on_vcpu() it.
Quoted 9 lines
Quoted 4 lines
I think it's reasonable to demand that whoever calls
kvm_vcpu_ioctl()know what they are doing (and they'll get surprising results if it switches threads implicitly).I respectfully disagree. Not that I want people not to know what they are doing, but I believe that, forcing something that can only run in a specific thread to be run there, provides us with a much saner interface, that will make code a lot more readable and maintainable.
Example:
kvm_vcpu_ioctl(get_regs)
regs->rflags |= some_flag
kvm_vcpu_ioctl(set_regs)
This looks totally sane but is racy if kvm_vcpu_ioctl() does an implicit on_vcpu().
Quoted 10 lines
Quoted 5 lines
One of them is synchronous, meaning the data can live on stack and no special synchronization is needed, while the other is synchronous, meaning explicit memory management and end-of-work synchronization is needed.
I will assume you meant "the other is assynchronous". It does not need to be. I though about including the assynchronous version in this RFC to let doors open for performance improvements *if* we needed them. But again: the absolute majority of the calls will be local. So it is not that important.
Then there's even less reason to include the async version.
Quoted 5 lines
Quoted 3 lines
on_vcpu()is a subset ofqueue_work(). I meant, why to we need the extra functionality?As I said, if you oppose it hardly, we don't really need to.
I do oppose it, but the reason for not including it should be the reasons I cited, not that I oppose it.
Quoted 4 lines
Quoted 2 lines
A more powerful API comes with increased responsibilities.
You suddenly sounds like spider man.
I hate it when I get unmasked.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 18 lines · Avi Kivity
Quoted 17 lines · Glauber Costa
Quoted 15 lines · Avi Kivity
Quoted 9 lines · Glauber Costa
Quoted 6 lines
on_vcpu()andqueue_work()are fundamentally different (yes, I see the wait parameter, and I think there should be two separate functions for such different behaviours).Therefore, the name change. The exact
on_vcpubehaviour, however, can be implemented ontop ofqueue_work().Will there be any use for asynchronous
queue_work()?It's a dangerous API.
Quoted 7 lines
Quoted 4 lines
Initially, I thought we could use it for batching, if we forced a flush in the end of a sequence of operations. This can makes things faster if we are doing a bunch of calls in a row, from the wrong thread.
It's a lot easier and safer to write a function that does your batch job and
on_vcpu()it.
agree.
Quoted 11 lines
Quoted 10 lines
Quoted 5 lines
I think it's reasonable to demand that whoever calls
kvm_vcpu_ioctl()know what they are doing (and they'll get surprising results if it switches threads implicitly).I respectfully disagree. Not that I want people not to know what they are doing, but I believe that, forcing something that can only run in a specific thread to be run there, provides us with a much saner interface, that will make code a lot more readable and maintainable.
Quoted 8 lines
Example:
kvm_vcpu_ioctl(get_regs)
regs->rflags|=some_flag
kvm_vcpu_ioctl(set_regs)This looks totally sane but is racy if
kvm_vcpu_ioctl()does an impliciton_vcpu().
Not at all. If we do queue_work once, it will execute whatever function you ask for in the correct thread, and for there on, everything will be local. So even if you nest functions that call queue_work themselves, only the first will involve synchronization at all.
Quoted 7 lines
Quoted 5 lines
I will assume you meant "the other is assynchronous". It does not need to be. I though about including the assynchronous version in this RFC to let doors open for performance improvements *if* we needed them. But again: the absolute majority of the calls will be local. So it is not that important.
Then there's even less reason to include the async version.
I am pretty much convinced by now.
Quoted 9 lines
Quoted 6 lines
Quoted 4 lines
on_vcpu()is a subset ofqueue_work(). I meant, why to we need the extra functionality?As I said, if you oppose it hardly, we don't really need to.
I do oppose it, but the reason for not including it should be the reasons I cited, not that I oppose it.
For a masked vigilante like you, I thought it was clear enough that "fierce opposition" automatically meant you got strong reasons, and were actually right.