Skip to contentSkip to conversation

colo-compare bugfixes

PATCH Superseded Lukas Straub 4 participants 15 messages Apr 8, 2020 – Apr 27, 2020 Networking

v1 of 6 Superseded Latest is v6 →

lore
Share & tools
lore
#0 lore raw

cover colo-compare bugfixes

Hello Everyone,
Here are fixes for bugs that I found in my tests.

Regards,
Lukas Straub

Lukas Straub (3):
net/colo-compare.c: Create event_bh with the right AioContext chardev/char.c: Use qemu_co_sleep_ns if in coroutine net/colo-compare.c: Fix deadlock

 chardev/char.c     |  7 +++-
 net/colo-compare.c | 85 +++++++++++++++++++++++++++++++++++++++-------
 2 files changed, 79 insertions(+), 13 deletions(-)
Signature · 1 line
2.20.1
#1 lore raw

PATCH 1/3 net/colo-compare.c: Create event_bh with the right AioContext

qemu_bh_new will set the bh to be executed in the main loop. This causes problems as colo_compare_handle_event assumes that it has exclusive access the queues, which are also accessed in the iothread. It also assumes that it runs in a different thread than the caller and takes the appropriate locks.

Create the bh with the AioContext of the iothread to fulfill these assumptions.

---
 net/colo-compare.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
net/colo-compare.c +2 -1
index 10c0239f9d..1de4220fe2 100644
@@ -890,6 +890,7 @@ static void colo_compare_handle_event(void *opaque)
 
 static void colo_compare_iothread(CompareState *s)
 {
+    AioContext *ctx = iothread_get_aio_context(s->iothread);
     object_ref(OBJECT(s->iothread));
     s->worker_context = iothread_get_g_main_context(s->iothread);
 
@@ -906,7 +907,7 @@ static void colo_compare_iothread(CompareState *s)
     }
 
     colo_compare_timer_init(s);
-    s->event_bh = qemu_bh_new(colo_compare_handle_event, s);
+    s->event_bh = aio_bh_new(ctx, colo_compare_handle_event, s);
 }
 
 static char *compare_get_pri_indev(Object *obj, Error **errp)
Signature · 1 line
2.20.1
#2 lore raw

RE: [PATCH 1/3] net/colo-compare.c: Create event_bh with the right AioContext

Quoted 13 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Thursday, April 9, 2020 2:34 AM
To: qemu-devel <[email protected]>
Cc: Zhang, Chen <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: [PATCH 1/3] net/colo-compare.c: Create event_bh with the right AioContext

Quoted 9 lines

qemu_bh_new will set the bh to be executed in the main loop. This causes problems as colo_compare_handle_event assumes that it has exclusive access the queues, which are also accessed in the iothread. It also assumes that it runs in a different thread than the caller and takes the appropriate locks.

Create the bh with the AioContext of the iothread to fulfill these assumptions.

Looks good for me, I assume it will increase performance. Do you have related data?

Thanks
Zhang Chen

Quoted 28 lines
---
 net/colo-compare.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
net/colo-compare.c index renamed from net/colo-compare.c +0 -0

10c0239f9d..1de4220fe2 100644

net/colo-compare.c +2 -2
@@ -890,6 +890,7 @@ static void colo_compare_handle_event(void
 *opaque)
 
 static void colo_compare_iothread(CompareState *s)  {
+    AioContext *ctx = iothread_get_aio_context(s->iothread);
     object_ref(OBJECT(s->iothread));
     s->worker_context = iothread_get_g_main_context(s->iothread);
 
@@ -906,7 +907,7 @@ static void colo_compare_iothread(CompareState *s)
     }
 
     colo_compare_timer_init(s);
-    s->event_bh = qemu_bh_new(colo_compare_handle_event, s);
+    s->event_bh = aio_bh_new(ctx, colo_compare_handle_event, s);
 }
 
 static char *compare_get_pri_indev(Object *obj, Error **errp)
--

2.20.1

#3 lore raw

Re: [PATCH 1/3] net/colo-compare.c: Create event_bh with the right AioContext

Quoted 17 lines · Chen
Quoted 17 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Thursday, April 9, 2020 2:34 AM
To: qemu-devel <[email protected]>
Cc: Zhang, Chen <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: [PATCH 1/3] net/colo-compare.c: Create event_bh with the right AioContext

qemu_bh_new will set the bh to be executed in the main loop. This causes problems as colo_compare_handle_event assumes that it has exclusive access the queues, which are also accessed in the iothread. It also assumes that it runs in a different thread than the caller and takes the appropriate locks.

Quoted 5 lines
Quoted 3 lines

Create the bh with the AioContext of the iothread to fulfill these assumptions.

Looks good for me, I assume it will increase performance. Do you have related data?

No, this fixes several crashes because the queues where accessed concurrently from multiple threads. Sorry for my bad wording.

Regards,
Lukas Straub

Quoted 32 lines

Thanks
Zhang Chen

Quoted 28 lines
---
 net/colo-compare.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
net/colo-compare.c index renamed from net/colo-compare.c +0 -0

10c0239f9d..1de4220fe2 100644

net/colo-compare.c +2 -2
@@ -890,6 +890,7 @@ static void colo_compare_handle_event(void
 *opaque)
 
 static void colo_compare_iothread(CompareState *s)  {
+    AioContext *ctx = iothread_get_aio_context(s->iothread);
     object_ref(OBJECT(s->iothread));
     s->worker_context = iothread_get_g_main_context(s->iothread);
 
@@ -906,7 +907,7 @@ static void colo_compare_iothread(CompareState *s)
     }
 
     colo_compare_timer_init(s);
-    s->event_bh = qemu_bh_new(colo_compare_handle_event, s);
+    s->event_bh = aio_bh_new(ctx, colo_compare_handle_event, s);
 }
 
 static char *compare_get_pri_indev(Object *obj, Error **errp)
--

2.20.1

#4 lore raw

RE: [PATCH 1/3] net/colo-compare.c: Create event_bh with the right AioContext

Quoted 37 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Wednesday, April 22, 2020 4:43 PM
To: Zhang, Chen <[email protected]>
Cc: qemu-devel <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: Re: [PATCH 1/3] net/colo-compare.c: Create event_bh with the right AioContext

Quoted 21 lines · Chen
Quoted 20 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Thursday, April 9, 2020 2:34 AM
To: qemu-devel <[email protected]>
Cc: Zhang, Chen <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: [PATCH 1/3] net/colo-compare.c: Create event_bh with the right AioContext

qemu_bh_new will set the bh to be executed in the main loop. This causes problems as colo_compare_handle_event assumes that it has exclusive access the queues, which are also accessed in the iothread. It also assumes that it runs in a different thread than the caller and takes the appropriate locks.

Create the bh with the AioContext of the iothread to fulfill these assumptions.

Quoted 5 lines
Quoted 1 line

Looks good for me, I assume it will increase performance. Do you have

related data?

No, this fixes several crashes because the queues where accessed concurrently from multiple threads. Sorry for my bad wording.

Can you describe some details about the crash? Step by step? Maybe I can re-produce and test it for this patch.

Thanks
Zhang Chen

Quoted 37 lines

Regards,
Lukas Straub

Quoted 15 lines

Thanks
Zhang Chen

Quoted 12 lines
---
 net/colo-compare.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
net/colo-compare.c index renamed from net/colo-compare.c +0 -0

10c0239f9d..1de4220fe2 100644

net/colo-compare.c +0 -0
@@ -890,6 +890,7 @@ static void colo_compare_handle_event(void
 *opaque)
 
Quoted 6 lines

net/colo-compare.c static void colo_compare_iothread(CompareState

Quoted 6 lines
net/colo-compare.c +1 -0
 static void colo_compare_iothread(CompareState *s)  {
+    AioContext *ctx = iothread_get_aio_context(s->iothread);
     object_ref(OBJECT(s->iothread));
     s->worker_context = iothread_get_g_main_context(s->iothread);
 
@@ -906,7 +907,7 @@ static void colo_compare_iothread(CompareState

*s)

Quoted 11 lines
Quoted 10 lines
net/colo-compare.c +1 -2
     }
 
     colo_compare_timer_init(s);
-    s->event_bh = qemu_bh_new(colo_compare_handle_event, s);
+    s->event_bh = aio_bh_new(ctx, colo_compare_handle_event, s);
 }
 
 static char *compare_get_pri_indev(Object *obj, Error **errp)
--

2.20.1

#5 lore raw

Re: [PATCH 1/3] net/colo-compare.c: Create event_bh with the right AioContext

Quoted 35 lines · Chen
Quoted 35 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Wednesday, April 22, 2020 4:43 PM
To: Zhang, Chen <[email protected]>
Cc: qemu-devel <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: Re: [PATCH 1/3] net/colo-compare.c: Create event_bh with the right AioContext

Quoted 21 lines · Chen
Quoted 20 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Thursday, April 9, 2020 2:34 AM
To: qemu-devel <[email protected]>
Cc: Zhang, Chen <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: [PATCH 1/3] net/colo-compare.c: Create event_bh with the right AioContext

qemu_bh_new will set the bh to be executed in the main loop. This causes problems as colo_compare_handle_event assumes that it has exclusive access the queues, which are also accessed in the iothread. It also assumes that it runs in a different thread than the caller and takes the appropriate locks.

Create the bh with the AioContext of the iothread to fulfill these assumptions.

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

Looks good for me, I assume it will increase performance. Do you have

related data?

No, this fixes several crashes because the queues where accessed concurrently from multiple threads. Sorry for my bad wording.

Can you describe some details about the crash? Step by step? Maybe I can re-produce and test it for this patch.

There is no clear test case. For me the crashes happened after 1-20h of runtime with lots of checkpoints (800ms) and some network traffic. The coredump always showed that two threads where doing operations on the queues simultaneously. Unfortunately, I don't have the coredumps anymore.

Regards,
Lukas Straub

Quoted 41 lines

Thanks
Zhang Chen

Quoted 37 lines

Regards,
Lukas Straub

Quoted 15 lines

Thanks
Zhang Chen

Quoted 12 lines
---
 net/colo-compare.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
net/colo-compare.c index renamed from net/colo-compare.c +0 -0

10c0239f9d..1de4220fe2 100644

net/colo-compare.c +0 -0
@@ -890,6 +890,7 @@ static void colo_compare_handle_event(void
 *opaque)
 
Quoted 6 lines

net/colo-compare.c static void colo_compare_iothread(CompareState

Quoted 6 lines
net/colo-compare.c +1 -0
 static void colo_compare_iothread(CompareState *s)  {
+    AioContext *ctx = iothread_get_aio_context(s->iothread);
     object_ref(OBJECT(s->iothread));
     s->worker_context = iothread_get_g_main_context(s->iothread);
 
@@ -906,7 +907,7 @@ static void colo_compare_iothread(CompareState  

*s)

Quoted 11 lines
Quoted 10 lines
net/colo-compare.c +1 -2
     }
 
     colo_compare_timer_init(s);
-    s->event_bh = qemu_bh_new(colo_compare_handle_event, s);
+    s->event_bh = aio_bh_new(ctx, colo_compare_handle_event, s);
 }
 
 static char *compare_get_pri_indev(Object *obj, Error **errp)
--

2.20.1

Reviewed-by
#6 lore raw

RE: [PATCH 1/3] net/colo-compare.c: Create event_bh with the right AioContext

Quoted 58 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Wednesday, April 22, 2020 5:40 PM
To: Zhang, Chen <[email protected]>
Cc: qemu-devel <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: Re: [PATCH 1/3] net/colo-compare.c: Create event_bh with the right AioContext

Quoted 42 lines · Chen
Quoted 41 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Wednesday, April 22, 2020 4:43 PM
To: Zhang, Chen <[email protected]>
Cc: qemu-devel <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: Re: [PATCH 1/3] net/colo-compare.c: Create event_bh with the right AioContext

Quoted 17 lines · Chen
Quoted 17 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Thursday, April 9, 2020 2:34 AM
To: qemu-devel <[email protected]>
Cc: Zhang, Chen <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: [PATCH 1/3] net/colo-compare.c: Create event_bh with the right AioContext

qemu_bh_new will set the bh to be executed in the main loop. This causes problems as colo_compare_handle_event assumes that it has exclusive access the queues, which are also accessed in the iothread. It also assumes that it runs in a different thread than the caller and takes the appropriate locks.

Quoted 6 lines
Quoted 3 lines

Create the bh with the AioContext of the iothread to fulfill these assumptions.

Looks good for me, I assume it will increase performance. Do you have

related data?

No, this fixes several crashes because the queues where accessed concurrently from multiple threads. Sorry for my bad wording.

Quoted 8 lines
Quoted 2 lines

Can you describe some details about the crash? Step by step? Maybe I can re-produce and test it for this patch.

There is no clear test case. For me the crashes happened after 1-20h of runtime with lots of checkpoints (800ms) and some network traffic. The coredump always showed that two threads where doing operations on the queues simultaneously.
Unfortunately, I don't have the coredumps anymore.

OK, Although I have not encountered the problem you described. I have test this patch, looks running fine.

Thanks
Zhang Chen

Quoted 48 lines

Regards,
Lukas Straub

Quoted 44 lines

Thanks
Zhang Chen

Quoted 40 lines

Regards,
Lukas Straub

Quoted 15 lines

Thanks
Zhang Chen

Quoted 12 lines
---
 net/colo-compare.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
net/colo-compare.c index renamed from net/colo-compare.c +0 -0

10c0239f9d..1de4220fe2 100644

net/colo-compare.c +0 -0
@@ -890,6 +890,7 @@ static void colo_compare_handle_event(void
 *opaque)
 
Quoted 8 lines
Quoted 8 lines
net/colo-compare.c +1 -0
 static void colo_compare_iothread(CompareState *s)  {
+    AioContext *ctx = iothread_get_aio_context(s->iothread);
     object_ref(OBJECT(s->iothread));
     s->worker_context =
 iothread_get_g_main_context(s->iothread);
@@ -906,7 +907,7 @@ static void

colo_compare_iothread(CompareState

*s)

Quoted 12 lines
Quoted 11 lines
     }

     colo_compare_timer_init(s);
net/colo-compare.c +2 -2
-    s->event_bh = qemu_bh_new(colo_compare_handle_event, s);
+    s->event_bh = aio_bh_new(ctx, colo_compare_handle_event,
+ s);
 }
 
 static char *compare_get_pri_indev(Object *obj, Error **errp)
--

2.20.1

Reviewed-by Tested-by
#7 lore raw

Re: [PATCH 1/3] net/colo-compare.c: Create event_bh with the right AioContext

Quoted 67 lines · Chen
Quoted 58 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Wednesday, April 22, 2020 5:40 PM
To: Zhang, Chen <[email protected]>
Cc: qemu-devel <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: Re: [PATCH 1/3] net/colo-compare.c: Create event_bh with the right AioContext

Quoted 42 lines · Chen
Quoted 41 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Wednesday, April 22, 2020 4:43 PM
To: Zhang, Chen <[email protected]>
Cc: qemu-devel <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: Re: [PATCH 1/3] net/colo-compare.c: Create event_bh with the right AioContext

Quoted 17 lines · Chen
Quoted 17 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Thursday, April 9, 2020 2:34 AM
To: qemu-devel <[email protected]>
Cc: Zhang, Chen <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: [PATCH 1/3] net/colo-compare.c: Create event_bh with the right AioContext

qemu_bh_new will set the bh to be executed in the main loop. This causes problems as colo_compare_handle_event assumes that it has exclusive access the queues, which are also accessed in the iothread. It also assumes that it runs in a different thread than the caller and takes the appropriate locks.

Quoted 6 lines
Quoted 3 lines

Create the bh with the AioContext of the iothread to fulfill these assumptions.

Looks good for me, I assume it will increase performance. Do you have

related data?

No, this fixes several crashes because the queues where accessed concurrently from multiple threads. Sorry for my bad wording.

Quoted 8 lines
Quoted 2 lines

Can you describe some details about the crash? Step by step? Maybe I can re-produce and test it for this patch.

There is no clear test case. For me the crashes happened after 1-20h of runtime with lots of checkpoints (800ms) and some network traffic. The coredump always showed that two threads where doing operations on the queues simultaneously.
Unfortunately, I don't have the coredumps anymore.

Quoted 7 lines

OK, Although I have not encountered the problem you described. I have test this patch, looks running fine.

Thanks
Zhang Chen

Hi,

I encountered PVM crash caused by the race condition issue in v4.2.0. Here is the coredump for reference.

```
warning: core file may not match specified executable file.
 Core was generated by `qemu-system-x86_64 -name source -enable-kvm 
-cpu core2duo -m 1024 -global kvm-a'.
 Program terminated with signal SIGSEGV, Segmentation fault.
 #0 0x000055cb478bcd25 in qemu_hexdump (buf=0x0, fp=0x7f6e9122b680 
<IO_2_1_stderr>, prefix=0x55cb47a388f5 "colo-compare spkt", size=1514) at util/hexdump.c:34
 34 fprintf(fp, " %02x", (unsigned char)buf[b + i]);
 [Current thread is 1 (Thread 0x7f6da1ade700 (LWP 6119))]
 (gdb) where
 #0 0x000055cb478bcd25 in qemu_hexdump (buf=0x0, fp=0x7f6e9122b680 
<IO_2_1_stderr>, prefix=0x55cb47a388f5 "colo-compare spkt", size=1514) at util/hexdump.c:34
 #1 0x000055cb476fa1b5 in colo_compare_tcp (s=0x55cb496429f0, 
conn=0x7f6e78003e30) at net/colo-compare.c:462
 #2 0x000055cb476fa8c1 in colo_compare_connection 
(opaque=0x7f6e78003e30, user_data=0x55cb496429f0) at net/colo-compare.c:687
 #3 0x000055cb476fb4ab in compare_pri_rs_finalize 
(pri_rs=0x55cb49642b18) at net/colo-compare.c:1001
 #4 0x000055cb476eb46f in net_fill_rstate (rs=0x55cb49642b18, 
buf=0x7f6da1add2c8 "", size=1064) at net/net.c:1764
 #5 0x000055cb476faafa in compare_pri_chr_in (opaque=0x55cb496429f0, 
buf=0x7f6da1adc6f0 "`E˧V\210RT", size=4096) at net/colo-compare.c:776
 #6 0x000055cb47815363 in qemu_chr_be_write_impl (s=0x55cb48c87ec0, 
buf=0x7f6da1adc6f0 "`E˧V\210RT", len=4096) at chardev/char.c:177
 #7 0x000055cb478153c7 in qemu_chr_be_write (s=0x55cb48c87ec0, 
buf=0x7f6da1adc6f0 "`E˧V\210RT", len=4096) at chardev/char.c:189
 #8 0x000055cb4781e002 in tcp_chr_read (chan=0x55cb48ef7690, 
cond=G_IO_IN, opaque=0x55cb48c87ec0) at chardev/char-socket.c:525
 #9 0x000055cb47839655 in qio_channel_fd_source_dispatch 
(source=0x7f6e78002050, callback=0x55cb4781de53 <tcp_chr_read>, user_data=0x55cb48c87ec0) at io/channel-watch.c:84
 #10 0x00007f6e950e1285 in g_main_context_dispatch () at 
/usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
 #11 0x00007f6e950e1650 in () at /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
 #12 0x00007f6e950e1962 in g_main_loop_run () at 
/usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
 #13 0x000055cb474920ae in iothread_run (opaque=0x55cb48c67f10) at 
iothread.c:82
 #14 0x000055cb478a699d in qemu_thread_start (args=0x55cb498035d0) at 
util/qemu-thread-posix.c:519
 #15 0x00007f6e912376db in start_thread (arg=0x7f6da1ade700) at 
pthread_create.c:463
 #16 0x00007f6e90f6088f in clone () at 
../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 (gdb)
```

COLO works well after applying this patch in my tests.

Regards,
Derek

Quoted 50 lines
Quoted 49 lines

Regards,
Lukas Straub

Quoted 44 lines

Thanks
Zhang Chen

Quoted 40 lines

Regards,
Lukas Straub

Quoted 15 lines

Thanks
Zhang Chen

Quoted 12 lines
---
  net/colo-compare.c | 3 ++-
  1 file changed, 2 insertions(+), 1 deletion(-)
net/colo-compare.c index renamed from net/colo-compare.c +0 -0

10c0239f9d..1de4220fe2 100644

net/colo-compare.c +0 -0
@@ -890,6 +890,7 @@ static void colo_compare_handle_event(void
 *opaque)
 
Quoted 8 lines
Quoted 8 lines
net/colo-compare.c +1 -0
  static void colo_compare_iothread(CompareState *s)  {
+    AioContext *ctx = iothread_get_aio_context(s->iothread);
      object_ref(OBJECT(s->iothread));
      s->worker_context =
 iothread_get_g_main_context(s->iothread);
@@ -906,7 +907,7 @@ static void

colo_compare_iothread(CompareState

*s)

Quoted 12 lines
Quoted 11 lines
      }

      colo_compare_timer_init(s);
net/colo-compare.c +2 -2
-    s->event_bh = qemu_bh_new(colo_compare_handle_event, s);
+    s->event_bh = aio_bh_new(ctx, colo_compare_handle_event,
+ s);
  }
 
  static char *compare_get_pri_indev(Object *obj, Error **errp)
--

2.20.1

#8 lore raw

RE: [PATCH 1/3] net/colo-compare.c: Create event_bh with the right AioContext

Quoted 150 lines

-----Original Message-----
From: Derek Su <[email protected]>
Sent: Friday, April 24, 2020 12:37 PM
To: Zhang, Chen <[email protected]>; Lukas Straub <[email protected]>
Cc: qemu-devel <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: Re: [PATCH 1/3] net/colo-compare.c: Create event_bh with the right AioContext

Quoted 16 lines · Chen
Quoted 16 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Wednesday, April 22, 2020 5:40 PM
To: Zhang, Chen <[email protected]>
Cc: qemu-devel <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: Re: [PATCH 1/3] net/colo-compare.c: Create event_bh with the right AioContext

On Wed, 22 Apr 2020 09:03:00 +0000
"Zhang, Chen" <[email protected]> wrote:

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Wednesday, April 22, 2020 4:43 PM
To: Zhang, Chen <[email protected]>
Cc: qemu-devel <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>;

Marc-

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: Re: [PATCH 1/3] net/colo-compare.c: Create event_bh with the right AioContext

On Wed, 22 Apr 2020 08:29:39 +0000
"Zhang, Chen" <[email protected]> wrote:

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Thursday, April 9, 2020 2:34 AM
To: qemu-devel <[email protected]>
Cc: Zhang, Chen <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo

Bonzini

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

<[email protected]>
Subject: [PATCH 1/3] net/colo-compare.c: Create event_bh with the right AioContext

qemu_bh_new will set the bh to be executed in the main loop. This causes problems as colo_compare_handle_event assumes that

it

Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 14 lines
Quoted 10 lines
Quoted 7 lines

has exclusive access the queues, which are also accessed in the iothread. It also assumes that it runs in a different thread than the caller and takes the appropriate locks.

Create the bh with the AioContext of the iothread to fulfill these assumptions.

Looks good for me, I assume it will increase performance. Do you have

related data?

No, this fixes several crashes because the queues where accessed concurrently from multiple threads. Sorry for my bad wording.

Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

Can you describe some details about the crash? Step by step? Maybe I can re-produce and test it for this patch.

There is no clear test case. For me the crashes happened after 1-20h of runtime with lots of checkpoints (800ms) and some network traffic. The coredump always showed that two threads where doing operations

on

Quoted 10 lines
Quoted 2 lines

the queues simultaneously.
Unfortunately, I don't have the coredumps anymore.

OK, Although I have not encountered the problem you described. I have test this patch, looks running fine.

Thanks
Zhang Chen

Hi,

I encountered PVM crash caused by the race condition issue in v4.2.0. Here is the coredump for reference.

```
warning: core file may not match specified executable file.
  Core was generated by `qemu-system-x86_64 -name source -enable-kvm -
cpu core2duo -m 1024 -global kvm-a'.
  Program terminated with signal SIGSEGV, Segmentation fault.
  #0 0x000055cb478bcd25 in qemu_hexdump (buf=0x0, fp=0x7f6e9122b680
<IO_2_1_stderr>, prefix=0x55cb47a388f5 "colo-compare spkt", size=1514) at
util/hexdump.c:34
  34 fprintf(fp, " %02x", (unsigned char)buf[b + i]);
  [Current thread is 1 (Thread 0x7f6da1ade700 (LWP 6119))]
  (gdb) where
  #0 0x000055cb478bcd25 in qemu_hexdump (buf=0x0, fp=0x7f6e9122b680
<IO_2_1_stderr>, prefix=0x55cb47a388f5 "colo-compare spkt", size=1514) at
util/hexdump.c:34
  #1 0x000055cb476fa1b5 in colo_compare_tcp (s=0x55cb496429f0,
conn=0x7f6e78003e30) at net/colo-compare.c:462
  #2 0x000055cb476fa8c1 in colo_compare_connection
(opaque=0x7f6e78003e30, user_data=0x55cb496429f0) at net/colo-
compare.c:687
  #3 0x000055cb476fb4ab in compare_pri_rs_finalize
(pri_rs=0x55cb49642b18) at net/colo-compare.c:1001
  #4 0x000055cb476eb46f in net_fill_rstate (rs=0x55cb49642b18,
buf=0x7f6da1add2c8 "", size=1064) at net/net.c:1764
  #5 0x000055cb476faafa in compare_pri_chr_in (opaque=0x55cb496429f0,
buf=0x7f6da1adc6f0 "`E˧V\210RT", size=4096) at net/colo-compare.c:776
  #6 0x000055cb47815363 in qemu_chr_be_write_impl (s=0x55cb48c87ec0,
buf=0x7f6da1adc6f0 "`E˧V\210RT", len=4096) at chardev/char.c:177
  #7 0x000055cb478153c7 in qemu_chr_be_write (s=0x55cb48c87ec0,
buf=0x7f6da1adc6f0 "`E˧V\210RT", len=4096) at chardev/char.c:189
  #8 0x000055cb4781e002 in tcp_chr_read (chan=0x55cb48ef7690,
cond=G_IO_IN, opaque=0x55cb48c87ec0) at chardev/char-socket.c:525
  #9 0x000055cb47839655 in qio_channel_fd_source_dispatch
(source=0x7f6e78002050, callback=0x55cb4781de53 <tcp_chr_read>,
user_data=0x55cb48c87ec0) at io/channel-watch.c:84
  #10 0x00007f6e950e1285 in g_main_context_dispatch () at
/usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
  #11 0x00007f6e950e1650 in () at /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
  #12 0x00007f6e950e1962 in g_main_loop_run () at
/usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
  #13 0x000055cb474920ae in iothread_run (opaque=0x55cb48c67f10) at
iothread.c:82
  #14 0x000055cb478a699d in qemu_thread_start (args=0x55cb498035d0) at
util/qemu-thread-posix.c:519
  #15 0x00007f6e912376db in start_thread (arg=0x7f6da1ade700) at
pthread_create.c:463
  #16 0x00007f6e90f6088f in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:95
  (gdb)
```
Quoted 5 lines

COLO works well after applying this patch in my tests.

Thanks Derek.

Quoted 55 lines

Regards,
Derek

Quoted 49 lines
Quoted 48 lines

Regards,
Lukas Straub

Quoted 43 lines

Thanks
Zhang Chen

Quoted 39 lines

Regards,
Lukas Straub

Quoted 15 lines

Thanks
Zhang Chen

Quoted 12 lines
---
  net/colo-compare.c | 3 ++-
  1 file changed, 2 insertions(+), 1 deletion(-)
net/colo-compare.c index renamed from net/colo-compare.c +0 -0

10c0239f9d..1de4220fe2 100644

net/colo-compare.c +0 -0
@@ -890,6 +890,7 @@ static void colo_compare_handle_event(void
 *opaque)
 
Quoted 8 lines
Quoted 8 lines
net/colo-compare.c +1 -0
  static void colo_compare_iothread(CompareState *s)  {
+    AioContext *ctx = iothread_get_aio_context(s->iothread);
      object_ref(OBJECT(s->iothread));
      s->worker_context =
 iothread_get_g_main_context(s->iothread);
@@ -906,7 +907,7 @@ static void

colo_compare_iothread(CompareState

*s)

Quoted 11 lines
Quoted 10 lines
      }

      colo_compare_timer_init(s);
net/colo-compare.c +1 -2
-    s->event_bh = qemu_bh_new(colo_compare_handle_event, s);
+    s->event_bh = aio_bh_new(ctx, colo_compare_handle_event, s);
  }
 
  static char *compare_get_pri_indev(Object *obj, Error **errp)
--

2.20.1

#9 lore raw

PATCH 2/3 chardev/char.c: Use qemu_co_sleep_ns if in coroutine

In reply to Lukas Straub

This will be needed in the next patch.

---
 chardev/char.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
chardev/char.c +6 -1
index 04075389bf..51ad0dc6b3 100644
@@ -38,6 +38,7 @@
 #include "qemu/module.h"
 #include "qemu/option.h"
 #include "qemu/id.h"
+#include "qemu/coroutine.h"
 
 #include "chardev/char-mux.h"
 
@@ -119,7 +120,11 @@ static int qemu_chr_write_buffer(Chardev *s,
     retry:
         res = cc->chr_write(s, buf + *offset, len - *offset);
         if (res < 0 && errno == EAGAIN && write_all) {
-            g_usleep(100);
+            if (qemu_in_coroutine()) {
+                qemu_co_sleep_ns(QEMU_CLOCK_REALTIME, 100000);
+            } else {
+                g_usleep(100);
+            }
             goto retry;
         }
 
Signature · 1 line
2.20.1
Reviewed-by on v1 2/3
#10 lore raw

Re: [PATCH 2/3] chardev/char.c: Use qemu_co_sleep_ns if in coroutine

Hi

Quoted 21 lines · Lukas Straub

This will be needed in the next patch.

---
 chardev/char.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
chardev/char.c +1 -0
index 04075389bf..51ad0dc6b3 100644
@@ -38,6 +38,7 @@
 #include "qemu/module.h"
 #include "qemu/option.h"
 #include "qemu/id.h"
+#include "qemu/coroutine.h"
 
 #include "chardev/char-mux.h"
 

chardev/char.c static int qemu_chr_write_buffer(Chardev *s, · line 128

Quoted 10 lines
chardev/char.c +5 -1
@@ -119,7 +120,11 @@ static int qemu_chr_write_buffer(Chardev *s,
     retry:
         res = cc->chr_write(s, buf + *offset, len - *offset);
         if (res < 0 && errno == EAGAIN && write_all) {
-            g_usleep(100);
+            if (qemu_in_coroutine()) {
+                qemu_co_sleep_ns(QEMU_CLOCK_REALTIME, 100000);
+            } else {
+                g_usleep(100);
+            }
             goto retry;

Although we don't have any coroutine code in chardev/, it dos this kind of coroutine handling indirectly through qio. Thus it should be fine too here.

Quoted 5 lines
chardev/char.c +0 -1
         }
 
--

2.20.1

Signature · 1 line
Marc-André Lureau
Reviewed-by on v1 2/3
#11 lore raw

RE: [PATCH 2/3] chardev/char.c: Use qemu_co_sleep_ns if in coroutine

In reply to Lukas Straub
Quoted 12 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Thursday, April 9, 2020 2:34 AM
To: qemu-devel <[email protected]>
Cc: Zhang, Chen <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: [PATCH 2/3] chardev/char.c: Use qemu_co_sleep_ns if in coroutine

Quoted 3 lines

This will be needed in the next patch.

Quoted 31 lines
---
 chardev/char.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
chardev/char.c index 04075389bf..51ad0dc6b3 renamed from chardev/char.c +0 -0

100644

chardev/char.c +6 -2
@@ -38,6 +38,7 @@
 #include "qemu/module.h"
 #include "qemu/option.h"
 #include "qemu/id.h"
+#include "qemu/coroutine.h"
 
 #include "chardev/char-mux.h"
 
@@ -119,7 +120,11 @@ static int qemu_chr_write_buffer(Chardev *s,
     retry:
         res = cc->chr_write(s, buf + *offset, len - *offset);
         if (res < 0 && errno == EAGAIN && write_all) {
-            g_usleep(100);
+            if (qemu_in_coroutine()) {
+                qemu_co_sleep_ns(QEMU_CLOCK_REALTIME, 100000);
+            } else {
+                g_usleep(100);
+            }
             goto retry;
         }
 
--

2.20.1

#12 lore raw

PATCH 3/3 net/colo-compare.c: Fix deadlock

In reply to Lukas Straub

The chr_out chardev is connected to a filter-redirector running in the main loop. qemu_chr_fe_write_all might block here in compare_chr_send if the (socket-)buffer is full. If another filter-redirector in the main loop want's to send data to chr_pri_in it might also block if the buffer is full. This leads to a deadlock because both event loops get blocked.

Fix this by converting compare_chr_send to a coroutine and return error if it is in use.

---
 net/colo-compare.c | 82 +++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 71 insertions(+), 11 deletions(-)
net/colo-compare.c +71 -11
index 1de4220fe2..82787d3055 100644
@@ -32,6 +32,9 @@
 #include "migration/migration.h"
 #include "util.h"
 
+#include "block/aio-wait.h"
+#include "qemu/coroutine.h"
+
 #define TYPE_COLO_COMPARE "colo-compare"
 #define COLO_COMPARE(obj) \
     OBJECT_CHECK(CompareState, (obj), TYPE_COLO_COMPARE)
@@ -77,6 +80,17 @@ static int event_unhandled_count;
  *                    |packet  |  |packet  +    |packet  | |packet  +
  *                    +--------+  +--------+    +--------+ +--------+
  */
+
+typedef struct SendCo {
+    Coroutine *co;
+    uint8_t *buf;
+    uint32_t size;
+    uint32_t vnet_hdr_len;
+    bool notify_remote_frame;
+    bool done;
+    int ret;
+} SendCo;
+
 typedef struct CompareState {
     Object parent;
 
@@ -91,6 +105,7 @@ typedef struct CompareState {
     SocketReadState pri_rs;
     SocketReadState sec_rs;
     SocketReadState notify_rs;
+    SendCo sendco;
     bool vnet_hdr;
     uint32_t compare_timeout;
     uint32_t expired_scan_cycle;
@@ -699,19 +714,17 @@ static void colo_compare_connection(void *opaque, void *user_data)
     }
 }
 
-static int compare_chr_send(CompareState *s,
-                            const uint8_t *buf,
-                            uint32_t size,
-                            uint32_t vnet_hdr_len,
-                            bool notify_remote_frame)
+static void coroutine_fn _compare_chr_send(void *opaque)
 {
+    CompareState *s = opaque;
+    SendCo *sendco = &s->sendco;
+    const uint8_t *buf = sendco->buf;
+    uint32_t size = sendco->size;
+    uint32_t vnet_hdr_len = sendco->vnet_hdr_len;
+    bool notify_remote_frame = sendco->notify_remote_frame;
     int ret = 0;
     uint32_t len = htonl(size);
 
-    if (!size) {
-        return 0;
-    }
-
     if (notify_remote_frame) {
         ret = qemu_chr_fe_write_all(&s->chr_notify_dev,
                                     (uint8_t *)&len,
@@ -754,10 +767,50 @@ static int compare_chr_send(CompareState *s,
         goto err;
     }
 
-    return 0;
+    sendco->ret = 0;
+    goto out;
 
 err:
-    return ret < 0 ? ret : -EIO;
+    sendco->ret = ret < 0 ? ret : -EIO;
+out:
+    sendco->co = NULL;
+    g_free(sendco->buf);
+    sendco->buf = NULL;
+    sendco->done = true;
+    aio_wait_kick();
+}
+
+static int compare_chr_send(CompareState *s,
+                            const uint8_t *buf,
+                            uint32_t size,
+                            uint32_t vnet_hdr_len,
+                            bool notify_remote_frame)
+{
+    SendCo *sendco = &s->sendco;
+
+    if (!size) {
+        return 0;
+    }
+
+    if (sendco->done) {
+        sendco->co = qemu_coroutine_create(_compare_chr_send, s);
+        sendco->buf = g_malloc(size);
+        sendco->size = size;
+        sendco->vnet_hdr_len = vnet_hdr_len;
+        sendco->notify_remote_frame = notify_remote_frame;
+        sendco->done = false;
+        memcpy(sendco->buf, buf, size);
+        qemu_coroutine_enter(sendco->co);
+        if (sendco->done) {
+            /* report early errors */
+            return sendco->ret;
+        } else {
+            /* else assume success */
+            return 0;
+        }
+    }
+
+    return -ENOBUFS;
 }
 
 static int compare_chr_can_read(void *opaque)
@@ -1146,6 +1199,8 @@ static void colo_compare_complete(UserCreatable *uc, Error **errp)
     CompareState *s = COLO_COMPARE(uc);
     Chardev *chr;
 
+    s->sendco.done = true;
+
     if (!s->pri_indev || !s->sec_indev || !s->outdev || !s->iothread) {
         error_setg(errp, "colo compare needs 'primary_in' ,"
                    "'secondary_in','outdev','iothread' property set");
@@ -1281,6 +1336,11 @@ static void colo_compare_finalize(Object *obj)
     CompareState *s = COLO_COMPARE(obj);
     CompareState *tmp = NULL;
 
+    AioContext *ctx = iothread_get_aio_context(s->iothread);
+    aio_context_acquire(ctx);
+    AIO_WAIT_WHILE(ctx, !s->sendco.done);
+    aio_context_release(ctx);
+
     qemu_chr_fe_deinit(&s->chr_pri_in, false);
     qemu_chr_fe_deinit(&s->chr_sec_in, false);
     qemu_chr_fe_deinit(&s->chr_out, false);
Signature · 1 line
2.20.1
#13 lore raw

RE: [PATCH 3/3] net/colo-compare.c: Fix deadlock

Quoted 19 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Thursday, April 9, 2020 2:34 AM
To: qemu-devel <[email protected]>
Cc: Zhang, Chen <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: [PATCH 3/3] net/colo-compare.c: Fix deadlock

The chr_out chardev is connected to a filter-redirector running in the main loop. qemu_chr_fe_write_all might block here in compare_chr_send if the (socket-)buffer is full.
If another filter-redirector in the main loop want's to send data to chr_pri_in it might also block if the buffer is full. This leads to a deadlock because both event loops get blocked.

Quoted 3 lines

Fix this by converting compare_chr_send to a coroutine and return error if it is in use.

I have tested this series, running fine currently. Can you share performance data after this patch?

Thanks
Zhang Chen

Quoted 149 lines
---
 net/colo-compare.c | 82
+++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 71 insertions(+), 11 deletions(-)
net/colo-compare.c index renamed from net/colo-compare.c +0 -0

1de4220fe2..82787d3055 100644

net/colo-compare.c +23 -9
@@ -32,6 +32,9 @@
 #include "migration/migration.h"
 #include "util.h"
 
+#include "block/aio-wait.h"
+#include "qemu/coroutine.h"
+
 #define TYPE_COLO_COMPARE "colo-compare"
 #define COLO_COMPARE(obj) \
     OBJECT_CHECK(CompareState, (obj), TYPE_COLO_COMPARE) @@ -77,6
+80,17 @@ static int event_unhandled_count;
  *                    |packet  |  |packet  +    |packet  | |packet  +
  *                    +--------+  +--------+    +--------+ +--------+
  */
+
+typedef struct SendCo {
+    Coroutine *co;
+    uint8_t *buf;
+    uint32_t size;
+    uint32_t vnet_hdr_len;
+    bool notify_remote_frame;
+    bool done;
+    int ret;
+} SendCo;
+
 typedef struct CompareState {
     Object parent;
@@ -91,6 +105,7 @@ typedef struct CompareState {
     SocketReadState pri_rs;
     SocketReadState sec_rs;
     SocketReadState notify_rs;
+    SendCo sendco;
     bool vnet_hdr;
     uint32_t compare_timeout;
     uint32_t expired_scan_cycle;
@@ -699,19 +714,17 @@ static void colo_compare_connection(void
 *opaque, void *user_data)
     }
 }
 
-static int compare_chr_send(CompareState *s,
-                            const uint8_t *buf,
-                            uint32_t size,
-                            uint32_t vnet_hdr_len,
-                            bool notify_remote_frame)
+static void coroutine_fn _compare_chr_send(void *opaque)
 {
+    CompareState *s = opaque;
+    SendCo *sendco = &s->sendco;
+    const uint8_t *buf = sendco->buf;
+    uint32_t size = sendco->size;
+    uint32_t vnet_hdr_len = sendco->vnet_hdr_len;
+    bool notify_remote_frame = sendco->notify_remote_frame;
     int ret = 0;
     uint32_t len = htonl(size);
 
-    if (!size) {
-        return 0;
-    }
-
     if (notify_remote_frame) {
         ret = qemu_chr_fe_write_all(&s->chr_notify_dev,
                                     (uint8_t *)&len, @@ -754,10 +767,50 @@ static int
compare_chr_send(CompareState *s,
         goto err;
     }
net/colo-compare.c +41 -2
-    return 0;
+    sendco->ret = 0;
+    goto out;
 
 err:
-    return ret < 0 ? ret : -EIO;
+    sendco->ret = ret < 0 ? ret : -EIO;
+out:
+    sendco->co = NULL;
+    g_free(sendco->buf);
+    sendco->buf = NULL;
+    sendco->done = true;
+    aio_wait_kick();
+}
+
+static int compare_chr_send(CompareState *s,
+                            const uint8_t *buf,
+                            uint32_t size,
+                            uint32_t vnet_hdr_len,
+                            bool notify_remote_frame) {
+    SendCo *sendco = &s->sendco;
+
+    if (!size) {
+        return 0;
+    }
+
+    if (sendco->done) {
+        sendco->co = qemu_coroutine_create(_compare_chr_send, s);
+        sendco->buf = g_malloc(size);
+        sendco->size = size;
+        sendco->vnet_hdr_len = vnet_hdr_len;
+        sendco->notify_remote_frame = notify_remote_frame;
+        sendco->done = false;
+        memcpy(sendco->buf, buf, size);
+        qemu_coroutine_enter(sendco->co);
+        if (sendco->done) {
+            /* report early errors */
+            return sendco->ret;
+        } else {
+            /* else assume success */
+            return 0;
+        }
+    }
+
+    return -ENOBUFS;
 }
 
 static int compare_chr_can_read(void *opaque) @@ -1146,6 +1199,8 @@

static void colo_compare_complete(UserCreatable *uc, Error **errp)

     CompareState *s = COLO_COMPARE(uc);
     Chardev *chr;
net/colo-compare.c +8 -1
+    s->sendco.done = true;
+
     if (!s->pri_indev || !s->sec_indev || !s->outdev || !s->iothread) {
         error_setg(errp, "colo compare needs 'primary_in' ,"
                    "'secondary_in','outdev','iothread' property set"); @@ -1281,6
+1336,11 @@ static void colo_compare_finalize(Object *obj)
     CompareState *s = COLO_COMPARE(obj);
     CompareState *tmp = NULL;
 
+    AioContext *ctx = iothread_get_aio_context(s->iothread);
+    aio_context_acquire(ctx);
+    AIO_WAIT_WHILE(ctx, !s->sendco.done);
+    aio_context_release(ctx);
+
     qemu_chr_fe_deinit(&s->chr_pri_in, false);
     qemu_chr_fe_deinit(&s->chr_sec_in, false);
     qemu_chr_fe_deinit(&s->chr_out, false);
--

2.20.1

#14 lore raw

Re: [PATCH 3/3] net/colo-compare.c: Fix deadlock

Quoted 21 lines · Chen
Quoted 17 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Thursday, April 9, 2020 2:34 AM
To: qemu-devel <[email protected]>
Cc: Zhang, Chen <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: [PATCH 3/3] net/colo-compare.c: Fix deadlock

The chr_out chardev is connected to a filter-redirector running in the main loop. qemu_chr_fe_write_all might block here in compare_chr_send if the (socket-)buffer is full.
If another filter-redirector in the main loop want's to send data to chr_pri_in it might also block if the buffer is full. This leads to a deadlock because both event loops get blocked.

Quoted 3 lines

Fix this by converting compare_chr_send to a coroutine and return error if it is in use.

Quoted 5 lines

I have tested this series, running fine currently. Can you share performance data after this patch?

Thanks
Zhang Chen

Hello,
Here are the results (using iperf3):
Client-to-server tcp:
without patch: ~64.2 Mbit/s
with patch: ~28.9 Mbit/s
Server-to-client tcp:
without patch: 360 Kbit/s (when it doesn't deadlock :) with patch: 220 Kbit/s

Yeah, it hurts performance somewhat, but the deadlock happens often with lots of server-to-client traffic. (It deadlocked in 2 of 4 runs)

Do you have a better idea to solve this issue?

Regards,
Lukas Straub

Quoted 149 lines
Quoted 149 lines
---
 net/colo-compare.c | 82
+++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 71 insertions(+), 11 deletions(-)
net/colo-compare.c index renamed from net/colo-compare.c +0 -0

1de4220fe2..82787d3055 100644

net/colo-compare.c +23 -9
@@ -32,6 +32,9 @@
 #include "migration/migration.h"
 #include "util.h"
 
+#include "block/aio-wait.h"
+#include "qemu/coroutine.h"
+
 #define TYPE_COLO_COMPARE "colo-compare"
 #define COLO_COMPARE(obj) \
     OBJECT_CHECK(CompareState, (obj), TYPE_COLO_COMPARE) @@ -77,6
+80,17 @@ static int event_unhandled_count;
  *                    |packet  |  |packet  +    |packet  | |packet  +
  *                    +--------+  +--------+    +--------+ +--------+
  */
+
+typedef struct SendCo {
+    Coroutine *co;
+    uint8_t *buf;
+    uint32_t size;
+    uint32_t vnet_hdr_len;
+    bool notify_remote_frame;
+    bool done;
+    int ret;
+} SendCo;
+
 typedef struct CompareState {
     Object parent;
@@ -91,6 +105,7 @@ typedef struct CompareState {
     SocketReadState pri_rs;
     SocketReadState sec_rs;
     SocketReadState notify_rs;
+    SendCo sendco;
     bool vnet_hdr;
     uint32_t compare_timeout;
     uint32_t expired_scan_cycle;
@@ -699,19 +714,17 @@ static void colo_compare_connection(void
 *opaque, void *user_data)
     }
 }
 
-static int compare_chr_send(CompareState *s,
-                            const uint8_t *buf,
-                            uint32_t size,
-                            uint32_t vnet_hdr_len,
-                            bool notify_remote_frame)
+static void coroutine_fn _compare_chr_send(void *opaque)
 {
+    CompareState *s = opaque;
+    SendCo *sendco = &s->sendco;
+    const uint8_t *buf = sendco->buf;
+    uint32_t size = sendco->size;
+    uint32_t vnet_hdr_len = sendco->vnet_hdr_len;
+    bool notify_remote_frame = sendco->notify_remote_frame;
     int ret = 0;
     uint32_t len = htonl(size);
 
-    if (!size) {
-        return 0;
-    }
-
     if (notify_remote_frame) {
         ret = qemu_chr_fe_write_all(&s->chr_notify_dev,
                                     (uint8_t *)&len, @@ -754,10 +767,50 @@ static int
compare_chr_send(CompareState *s,
         goto err;
     }
net/colo-compare.c +41 -2
-    return 0;
+    sendco->ret = 0;
+    goto out;
 
 err:
-    return ret < 0 ? ret : -EIO;
+    sendco->ret = ret < 0 ? ret : -EIO;
+out:
+    sendco->co = NULL;
+    g_free(sendco->buf);
+    sendco->buf = NULL;
+    sendco->done = true;
+    aio_wait_kick();
+}
+
+static int compare_chr_send(CompareState *s,
+                            const uint8_t *buf,
+                            uint32_t size,
+                            uint32_t vnet_hdr_len,
+                            bool notify_remote_frame) {
+    SendCo *sendco = &s->sendco;
+
+    if (!size) {
+        return 0;
+    }
+
+    if (sendco->done) {
+        sendco->co = qemu_coroutine_create(_compare_chr_send, s);
+        sendco->buf = g_malloc(size);
+        sendco->size = size;
+        sendco->vnet_hdr_len = vnet_hdr_len;
+        sendco->notify_remote_frame = notify_remote_frame;
+        sendco->done = false;
+        memcpy(sendco->buf, buf, size);
+        qemu_coroutine_enter(sendco->co);
+        if (sendco->done) {
+            /* report early errors */
+            return sendco->ret;
+        } else {
+            /* else assume success */
+            return 0;
+        }
+    }
+
+    return -ENOBUFS;
 }
 
 static int compare_chr_can_read(void *opaque) @@ -1146,6 +1199,8 @@

static void colo_compare_complete(UserCreatable *uc, Error **errp)

     CompareState *s = COLO_COMPARE(uc);
     Chardev *chr;
net/colo-compare.c +8 -1
+    s->sendco.done = true;
+
     if (!s->pri_indev || !s->sec_indev || !s->outdev || !s->iothread) {
         error_setg(errp, "colo compare needs 'primary_in' ,"
                    "'secondary_in','outdev','iothread' property set"); @@ -1281,6
+1336,11 @@ static void colo_compare_finalize(Object *obj)
     CompareState *s = COLO_COMPARE(obj);
     CompareState *tmp = NULL;
 
+    AioContext *ctx = iothread_get_aio_context(s->iothread);
+    aio_context_acquire(ctx);
+    AIO_WAIT_WHILE(ctx, !s->sendco.done);
+    aio_context_release(ctx);
+
     qemu_chr_fe_deinit(&s->chr_pri_in, false);
     qemu_chr_fe_deinit(&s->chr_sec_in, false);
     qemu_chr_fe_deinit(&s->chr_out, false);
--

2.20.1