Some more io-thread optimizations

RFC Superseded Jan Kiszka 3 participants 3 messages Feb 14, 2011 – Feb 16, 2011

lore

Share & tools

lore

Jan Kiszka

author

2011-02-14 09:50 UTC #0 lore raw

RFC 1/1 Some more io-thread optimizations

Hi,

patch below further reduces the io-thread overhead in tcg mode so that specifically emulating smp boxes gets noticeably faster. Its essence:
poll the file descriptors until select returns 0, keeping the global mutex locked. This reduces ping pong with the vcpu threads, most noticeably in tcg mode where we run in lock-step.

Split up in two patches, I'm planning to route those changes via the kvm queue (as they collide with other patches there).

Jan

--------8<---------

sysemu.h +1 -1

index 23ae17e..0a69464 100644
@@ -73,7 +73,7 @@ void cpu_synchronize_all_post_init(void);
 
 void qemu_announce_self(void);
 
-void main_loop_wait(int nonblocking);
+int main_loop_wait(int nonblocking);
 
 bool qemu_savevm_state_blocked(Monitor *mon);
 int qemu_savevm_state_begin(Monitor *mon, QEMUFile *f, int blk_enable,

vl.c +18 -4

index ed2cdfa..66b7c6f 100644
@@ -1311,7 +1311,7 @@ void qemu_system_powerdown_request(void)
     qemu_notify_event();
 }
 
-void main_loop_wait(int nonblocking)
+int main_loop_wait(int nonblocking)
 {
     IOHandlerRecord *ioh;
     fd_set rfds, wfds, xfds;

@@ -1356,9 +1356,16 @@ void main_loop_wait(int nonblocking)
 
     slirp_select_fill(&nfds, &rfds, &wfds, &xfds);
 
-    qemu_mutex_unlock_iothread();
+    if (timeout > 0) {
+        qemu_mutex_unlock_iothread();
+    }
+
     ret = select(nfds + 1, &rfds, &wfds, &xfds, &tv);
-    qemu_mutex_lock_iothread();
+
+    if (timeout > 0) {
+        qemu_mutex_lock_iothread();
+    }
+
     if (ret > 0) {
         IOHandlerRecord *pioh;

@@ -1386,6 +1393,7 @@ void main_loop_wait(int nonblocking)
        them.  */
     qemu_bh_poll();
 
+    return ret;
 }
 
 static int vm_can_run(void)

@@ -1405,6 +1413,7 @@ qemu_irq qemu_system_powerdown;
 
 static void main_loop(void)
 {
+    int last_io = 0;
     int r;
 
     qemu_main_loop_start();

@@ -1421,7 +1430,12 @@ static void main_loop(void)
 #ifdef CONFIG_PROFILER
             ti = profile_getclock();
 #endif
-            main_loop_wait(nonblocking);
+#ifdef CONFIG_IOTHREAD
+            if (last_io > 0) {
+                nonblocking = true;
+            }
+#endif
+            last_io = main_loop_wait(nonblocking);
 #ifdef CONFIG_PROFILER
             dev_time += profile_getclock() - ti;
 #endif

Marcelo Tosatti

2011-02-15 23:34 UTC #1 lore raw

Quoted 8 lines · Jan Kiszka

Hi,
patch below further reduces the io-thread overhead in tcg mode so that specifically emulating smp boxes gets noticeably faster. Its essence:
poll the file descriptors until select returns 0, keeping the global mutex locked. This reduces ping pong with the vcpu threads, most noticeably in tcg mode where we run in lock-step.

Quoted 4 lines

Split up in two patches, I'm planning to route those changes via the kvm queue (as they collide with other patches there).
Jan

Not sure this makes sense for all cases. There could be scenarios where a single pass is more efficient (think latency to acquire mutex from vcpu context in kvm mode, with intensive file IO in progress).

Jan Kiszka

2011-02-16 08:27 UTC #2 lore raw

Quoted 9 lines · Marcelo Tosatti

Quoted 8 lines · Jan Kiszka
Hi,
patch below further reduces the io-thread overhead in tcg mode so that specifically emulating smp boxes gets noticeably faster. Its essence:
poll the file descriptors until select returns 0, keeping the global mutex locked. This reduces ping pong with the vcpu threads, most noticeably in tcg mode where we run in lock-step.

Quoted 8 lines

Quoted 4 lines
Split up in two patches, I'm planning to route those changes via the kvm queue (as they collide with other patches there).
Jan
Not sure this makes sense for all cases. There could be scenarios where a single pass is more efficient (think latency to acquire mutex from vcpu context in kvm mode, with intensive file IO in progress).

Yeah, likely true. Only TCG has these insane long lock-holding times and requires signal-based mutex handover. I will exclude KVM from this.

Jan