io-thread optimizations
v2 cover io-thread optimizations
These patches were posted before. They bring down the overhead of the io-thread mode for TCG here, specifically when emulating SMP.
The major change in this version, besides rebasing, is the exclusion of KVM from the main loop polling optimization.
Jan Kiszka (3):
Do not drop global mutex for polled main loop runs Poll main loop after I/O events were received Do not kick vcpus in TCG mode
cpus.c | 2 +- sysemu.h | 2 +- vl.c | 22 ++++++++++++++++------ 3 files changed, 18 insertions(+), 8 deletions(-)
PATCH v2 1/3 Do not drop global mutex for polled main loop runs
From: Jan Kiszka <[email protected]>
If we call select without a timeout, it's more efficient to keep the global mutex locked as we may otherwise just play ping pong with a vcpu thread contending for it. This is particularly important for TCG mode where we run in lock-step with the vcpu thread.
--- vl.c | 10 ++++++++-- 1 files changed, 8 insertions(+), 2 deletions(-)
vl.c +8 -2
@@ -1314,9 +1314,15 @@ void main_loop_wait(int nonblocking)qemu_iohandler_fill (&nfds, &rfds, &wfds, &xfds);slirp_select_fill (&nfds, &rfds, &wfds, &xfds); -qemu_mutex_unlock_iothread (); +if (timeout >0 ) { +qemu_mutex_unlock_iothread (); + } + ret =select (nfds +1 , &rfds, &wfds, &xfds, &tv); -qemu_mutex_lock_iothread (); + +if (timeout >0 ) { +qemu_mutex_lock_iothread (); + }qemu_iohandler_poll (&rfds, &wfds, &xfds, ret);slirp_select_poll (&rfds, &wfds, &xfds, (ret <0 ));
Signature · 1 line
1.7.1
PATCH v2 2/3 Poll main loop after I/O events were received
In reply to Jan KiszkaFrom: Jan Kiszka <[email protected]>
Polling until select returns empty fdsets helps to reduce the switches between iothread and vcpus. The benefit of this patch is best visible when running an SMP guest on an SMP host in emulation mode.
--- sysemu.h | 2 +- vl.c | 12 ++++++++---- 2 files changed, 9 insertions(+), 5 deletions(-)
sysemu.h +1 -1
@@ -87,7 +87,7 @@ void cpu_synchronize_all_post_init(void);void qemu_announce_self (void ); -void main_loop_wait (int nonblocking); +int main_loop_wait (int nonblocking);bool qemu_savevm_state_blocked (Monitor *mon); int qemu_savevm_state_begin(Monitor *mon, QEMUFile *f, int blk_enable,
vl.c +8 -4
@@ -1286,7 +1286,7 @@ void qemu_system_vmstop_request(int reason)qemu_notify_event (); } -void main_loop_wait (int nonblocking) +int main_loop_wait (int nonblocking) {fd_set rfds, wfds, xfds;int ret, nfds;
@@ -1333,6 +1333,7 @@ void main_loop_wait(int nonblocking)them. */ qemu_bh_poll (); +return ret; }#ifndef CONFIG_IOTHREAD
@@ -1350,7 +1351,8 @@ qemu_irq qemu_system_powerdown;static void main_loop (void ) { -bool nonblocking = false; +bool nonblocking; +int last_io =0 ;#ifdef CONFIG_PROFILER int64_t ti;#endif
@@ -1359,7 +1361,9 @@ static void main_loop(void)qemu_main_loop_start ();for (;;) { -#ifndef CONFIG_IOTHREAD +#ifdef CONFIG_IOTHREAD + nonblocking = !kvm_enabled () && last_io >0 ; +#else nonblocking =cpu_exec_all ();if (vm_request_pending ()) { nonblocking = true;
@@ -1368,7 +1372,7 @@ static void main_loop(void)#ifdef CONFIG_PROFILER ti =profile_getclock ();#endif -main_loop_wait (nonblocking); + last_io =main_loop_wait (nonblocking);#ifdef CONFIG_PROFILER dev_time +=profile_getclock () - ti;#endif
Signature · 1 line
1.7.1
PATCH v2 3/3 Do not kick vcpus in TCG mode
In reply to Jan KiszkaFrom: Jan Kiszka <[email protected]>
In TCG mode, iothread and vcpus run in lock-step. So it's pointless to send a signal from qemu_cpu_kick to the vcpu thread - if we got here, the receiver already left the vcpu loop.
--- cpus.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
cpus.c +1 -1
@@ -860,7 +860,7 @@ void qemu_cpu_kick(void *_env)CPUState *env = _env;qemu_cond_broadcast (env->halt_cond); -if (!env->thread_kicked) { +if (kvm_enabled () && !env->thread_kicked) {qemu_cpu_kick_thread (env); env->thread_kicked = true; }
Signature · 1 line
1.7.1
Quoted 13 lines · Jan Kiszka
These patches were posted before. They bring down the overhead of the io-thread mode for TCG here, specifically when emulating SMP.
The major change in this version, besides rebasing, is the exclusion of KVM from the main loop polling optimization.
Jan Kiszka (3):
Do not drop global mutex for polled main loop runs Poll main loop after I/O events were received Do not kick vcpus in TCG mode
Quoted 5 lines
cpus.c | 2 +- sysemu.h | 2 +- vl.c | 22 ++++++++++++++++------ 3 files changed, 18 insertions(+), 8 deletions(-)
Quoted 13 lines · Jan Kiszka
These patches were posted before. They bring down the overhead of the io-thread mode for TCG here, specifically when emulating SMP.
The major change in this version, besides rebasing, is the exclusion of KVM from the main loop polling optimization.
Jan Kiszka (3):
Do not drop global mutex for polled main loop runs Poll main loop after I/O events were received Do not kick vcpus in TCG mode
Quoted 5 lines
cpus.c | 2 +- sysemu.h | 2 +- vl.c | 22 ++++++++++++++++------ 3 files changed, 18 insertions(+), 8 deletions(-)
Thanks for working on improving the io-thread with TCG. Your patches make sense, but they don't seems to fix the slowdown observed when enabling the io-thread. Well maybe they were not supposed to. This is for example the results of netperf between guest and host using virtio:
no io-thread 122 MB/s
io-thread 97 MB/s
io-thread + patches 98 MB/s
Signature · 2 lines
Aurelien Jarno GPG: 1024D/F1BCDB73 [email protected] http://www.aurel32.net
Quoted 20 lines · Aurelien Jarno
Quoted 13 lines · Jan Kiszka
These patches were posted before. They bring down the overhead of the io-thread mode for TCG here, specifically when emulating SMP.
The major change in this version, besides rebasing, is the exclusion of KVM from the main loop polling optimization.
Jan Kiszka (3):
Do not drop global mutex for polled main loop runs Poll main loop after I/O events were received Do not kick vcpus in TCG modeQuoted 5 lines
cpus.c | 2 +- sysemu.h | 2 +- vl.c | 22 ++++++++++++++++------ 3 files changed, 18 insertions(+), 8 deletions(-)
Quoted 9 lines
Thanks for working on improving the io-thread with TCG. Your patches make sense, but they don't seems to fix the slowdown observed when enabling the io-thread. Well maybe they were not supposed to. This is for example the results of netperf between guest and host using virtio:
no io-thread 122 MB/s
io-thread 97 MB/s
io-thread + patches 98 MB/s
Can you capture ftraces of io-thread enabled & disabled runs? They just need to cover a hand full of frames.
Thanks,
Jan
Quoted 26 lines · Jan Kiszka
Quoted 25 lines · Aurelien Jarno
Quoted 13 lines · Jan Kiszka
These patches were posted before. They bring down the overhead of the io-thread mode for TCG here, specifically when emulating SMP.
The major change in this version, besides rebasing, is the exclusion of KVM from the main loop polling optimization.
Jan Kiszka (3):
Do not drop global mutex for polled main loop runs Poll main loop after I/O events were received Do not kick vcpus in TCG modeQuoted 5 lines
cpus.c | 2 +- sysemu.h | 2 +- vl.c | 22 ++++++++++++++++------ 3 files changed, 18 insertions(+), 8 deletions(-)Thanks for working on improving the io-thread with TCG. Your patches make sense, but they don't seems to fix the slowdown observed when enabling the io-thread. Well maybe they were not supposed to. This is for example the results of netperf between guest and host using virtio:
Quoted 8 lines
Quoted 4 lines
no io-thread 122 MB/s
io-thread 97 MB/s
io-thread + patches 98 MB/sCan you capture ftraces of io-thread enabled& disabled runs? They just need to cover a hand full of frames.
Also interesting would be the context switch rates on the host.
If they're large, perhaps using user-space threading instead of native threads would help.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 35 lines · Avi Kivity
Quoted 34 lines · Jan Kiszka
Quoted 2 lines · Aurelien Jarno
Quoted 1 line · Jan Kiszka
These patches were posted before. They bring down the overhead of
the
Quoted 3 lines
Quoted 3 lines
io-thread mode for TCG here, specifically when emulating SMP.
The major change in this version, besides rebasing, is the
exclusion of
Quoted 15 lines
Quoted 9 lines
KVM from the main loop polling optimization.
Jan Kiszka (3):
Do not drop global mutex for polled main loop runs Poll main loop after I/O events were received Do not kick vcpus in TCG modeQuoted 5 lines
cpus.c | 2 +- sysemu.h | 2 +- vl.c | 22 ++++++++++++++++------ 3 files changed, 18 insertions(+), 8 deletions(-)Quoted 4 lines
Thanks for working on improving the io-thread with TCG. Your patches make sense, but they don't seems to fix the slowdown observed when enabling the io-thread. Well maybe they were not supposed to. This is for example the results of netperf between guest and host using
virtio:
Quoted 5 lines
no io-thread 122 MB/s
io-thread 97 MB/s
io-thread + patches 98 MB/s
Quoted 7 lines
Quoted 3 lines
Can you capture ftraces of io-thread enabled& disabled runs? They just need to cover a hand full of frames.
Also interesting would be the context switch rates on the host.
If they're large, perhaps using user-space threading instead of native threads would help.
I still suspect mitigation as the culprit here. Select is going to get to run more often which means more interrupt generation.
I bet if you count the number of packets per interrupt/notify you'll find that less batching is occurring.
Regards,
Anthony Liguori
Quoted 41 lines · Anthony Liguori
Quoted 40 lines · Avi Kivity
Quoted 38 lines · Jan Kiszka
Quoted 2 lines · Aurelien Jarno
Quoted 1 line · Jan Kiszka
These patches were posted before. They bring down the overhead
of the
Quoted 3 lines
Quoted 3 lines
io-thread mode for TCG here, specifically when emulating SMP.
The major change in this version, besides rebasing, is the
exclusion of
Quoted 15 lines
Quoted 9 lines
KVM from the main loop polling optimization.
Jan Kiszka (3):
Do not drop global mutex for polled main loop runs Poll main loop after I/O events were received Do not kick vcpus in TCG modeQuoted 5 lines
cpus.c | 2 +- sysemu.h | 2 +- vl.c | 22 ++++++++++++++++------ 3 files changed, 18 insertions(+), 8 deletions(-)Quoted 3 lines
Thanks for working on improving the io-thread with TCG. Your patches make sense, but they don't seems to fix the slowdown observed when enabling the io-thread. Well maybe they were not supposed to.
This is
Quoted 1 line
for example the results of netperf between guest and host using
virtio:
Quoted 5 lines
no io-thread 122 MB/s
io-thread 97 MB/s
io-thread + patches 98 MB/sCan you capture ftraces of io-thread enabled& disabled runs? They just need to cover a hand full of frames.
Quoted 8 lines
Quoted 3 lines
Also interesting would be the context switch rates on the host.
If they're large, perhaps using user-space threading instead of native threads would help.
I still suspect mitigation as the culprit here. Select is going to get to run more often which means more interrupt generation.
I bet if you count the number of packets per interrupt/notify you'll find that less batching is occurring.
Can you clarify? Which mitigation? virtio-net interrupt mitigation?
virtio-net interrupt mitigation is time-based, no? so why should threading affect it? and why would select() run more often? since we make all fds generate a signal, we ought to run a similar number same number of select()s.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 31 lines · Jan Kiszka
Quoted 20 lines · Aurelien Jarno
Quoted 13 lines · Jan Kiszka
These patches were posted before. They bring down the overhead of the io-thread mode for TCG here, specifically when emulating SMP.
The major change in this version, besides rebasing, is the exclusion of KVM from the main loop polling optimization.
Jan Kiszka (3):
Do not drop global mutex for polled main loop runs Poll main loop after I/O events were received Do not kick vcpus in TCG modeQuoted 5 lines
cpus.c | 2 +- sysemu.h | 2 +- vl.c | 22 ++++++++++++++++------ 3 files changed, 18 insertions(+), 8 deletions(-)Quoted 9 lines
Thanks for working on improving the io-thread with TCG. Your patches make sense, but they don't seems to fix the slowdown observed when enabling the io-thread. Well maybe they were not supposed to. This is for example the results of netperf between guest and host using virtio:
no io-thread 122 MB/s
io-thread 97 MB/s
io-thread + patches 98 MB/s
Quoted 5 lines
Can you capture ftraces of io-thread enabled & disabled runs? They just need to cover a hand full of frames.
From what I have been able to get from the ftraces documentation, it's
possible multiple tracers. Which tracers would you like to use there? The best would be a set of command lines to run.
Signature · 2 lines
Aurelien Jarno GPG: 1024D/F1BCDB73 [email protected] http://www.aurel32.net
Quoted 32 lines · Aurelien Jarno
Quoted 31 lines · Jan Kiszka
Quoted 29 lines · Aurelien Jarno
Quoted 13 lines · Jan Kiszka
These patches were posted before. They bring down the overhead of the io-thread mode for TCG here, specifically when emulating SMP.
The major change in this version, besides rebasing, is the exclusion of KVM from the main loop polling optimization.
Jan Kiszka (3):
Do not drop global mutex for polled main loop runs Poll main loop after I/O events were received Do not kick vcpus in TCG modeQuoted 5 lines
cpus.c | 2 +- sysemu.h | 2 +- vl.c | 22 ++++++++++++++++------ 3 files changed, 18 insertions(+), 8 deletions(-)Thanks for working on improving the io-thread with TCG. Your patches make sense, but they don't seems to fix the slowdown observed when enabling the io-thread. Well maybe they were not supposed to. This is for example the results of netperf between guest and host using virtio:
no io-thread 122 MB/s
io-thread 97 MB/s
io-thread + patches 98 MB/s
Quoted 7 lines
Quoted 3 lines
Can you capture ftraces of io-thread enabled & disabled runs? They just need to cover a hand full of frames.
From what I have been able to get from the ftraces documentation, it's possible multiple tracers. Which tracers would you like to use there? The best would be a set of command lines to run.
Sorry, of course: Just download, build & install trace-cmd [1], then execute "trace-cmd record -b 16000 -e all" while qemu is running. The result is written to trace.dat in the current directory. Visualize it via "trace-cmd report" (or kernelshark if you built that as well).
Jan
[1] git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/trace-cmd.git
Quoted 20 lines · Aurelien Jarno
Quoted 13 lines · Jan Kiszka
These patches were posted before. They bring down the overhead of the io-thread mode for TCG here, specifically when emulating SMP.
The major change in this version, besides rebasing, is the exclusion of KVM from the main loop polling optimization.
Jan Kiszka (3):
Do not drop global mutex for polled main loop runs Poll main loop after I/O events were received Do not kick vcpus in TCG modeQuoted 5 lines
cpus.c | 2 +- sysemu.h | 2 +- vl.c | 22 ++++++++++++++++------ 3 files changed, 18 insertions(+), 8 deletions(-)
Quoted 9 lines
Thanks for working on improving the io-thread with TCG. Your patches make sense, but they don't seems to fix the slowdown observed when enabling the io-thread. Well maybe they were not supposed to. This is for example the results of netperf between guest and host using virtio:
no io-thread 122 MB/s
io-thread 97 MB/s
io-thread + patches 98 MB/s
Given that everyone seems to agree that these patches are a step into the right direction (for the current TCG locking architecture at least), can we please finally apply them? They do have positive impact in some use cases. If rebasing is required (I don't think so), just let me know.
BTW, did you make any progress with tracing the remaining issues?
Thanks,
Jan
Am 25.06.2011 um 10:38 schrieb Jan Kiszka:
Quoted 31 lines · Aurelien Jarno
Quoted 20 lines · Jan Kiszka
Quoted 13 lines
These patches were posted before. They bring down the overhead of the io-thread mode for TCG here, specifically when emulating SMP.
The major change in this version, besides rebasing, is the exclusion of KVM from the main loop polling optimization.
Jan Kiszka (3):
Do not drop global mutex for polled main loop runs Poll main loop after I/O events were received Do not kick vcpus in TCG modeQuoted 5 lines
cpus.c | 2 +- sysemu.h | 2 +- vl.c | 22 ++++++++++++++++------ 3 files changed, 18 insertions(+), 8 deletions(-)Quoted 9 lines
Thanks for working on improving the io-thread with TCG. Your patches make sense, but they don't seems to fix the slowdown observed when enabling the io-thread. Well maybe they were not supposed to. This is for example the results of netperf between guest and host using virtio:
no io-thread 122 MB/s
io-thread 97 MB/s
io-thread + patches 98 MB/s
Quoted 6 lines
Given that everyone seems to agree that these patches are a step into the right direction (for the current TCG locking architecture at least), can we please finally apply them? They do have positive impact in some use cases. If rebasing is required (I don't think so), just let me know.
BTW, did you make any progress with tracing the remaining issues?
I've tested these together with Paolo's fixes, but it still hangs on Darwin. Doesn't appear to make it worse though.
Andreas
Quoted 38 lines · Andreas Färber
Am 25.06.2011 um 10:38 schrieb Jan Kiszka:
Quoted 36 lines · Aurelien Jarno
Quoted 20 lines · Jan Kiszka
Quoted 13 lines
These patches were posted before. They bring down the overhead of the io-thread mode for TCG here, specifically when emulating SMP.
The major change in this version, besides rebasing, is the exclusion of KVM from the main loop polling optimization.
Jan Kiszka (3):
Do not drop global mutex for polled main loop runs Poll main loop after I/O events were received Do not kick vcpus in TCG modeQuoted 5 lines
cpus.c | 2 +- sysemu.h | 2 +- vl.c | 22 ++++++++++++++++------ 3 files changed, 18 insertions(+), 8 deletions(-)Quoted 9 lines
Thanks for working on improving the io-thread with TCG. Your patches make sense, but they don't seems to fix the slowdown observed when enabling the io-thread. Well maybe they were not supposed to. This is for example the results of netperf between guest and host using virtio:
no io-thread 122 MB/s
io-thread 97 MB/s
io-thread + patches 98 MB/sGiven that everyone seems to agree that these patches are a step into the right direction (for the current TCG locking architecture at least), can we please finally apply them? They do have positive impact in some use cases. If rebasing is required (I don't think so), just let me know.
Quoted 4 lines
Quoted 1 line
BTW, did you make any progress with tracing the remaining issues?
I've tested these together with Paolo's fixes, but it still hangs on Darwin. Doesn't appear to make it worse though.
http://thread.gmane.org/gmane.comp.emulators.qemu/106225 is still not merged.
Jan