Skip to contentSkip to conversation

colo-compare bugfixes

PATCH Superseded Lukas Straub 5 participants 30 messages May 4, 2020 – May 15, 2020 Networking

v4 of 6 Superseded Latest is v6 →

lore
Share & tools
lore
#1 lore raw

PATCH v4 1/6 net/colo-compare.c: Create event_bh with the right AioContext

In reply to Lukas Straub

qemu_bh_new will set the bh to be executed in the main loop. This causes crashes as colo_compare_handle_event assumes that it has exclusive access the queues, which are also concurrently accessed in the iothread.

Create the bh with the AioContext of the iothread to fulfill these assumptions and fix the crashes. This is safe, because the bh already takes the appropriate locks.

---
 net/colo-compare.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
net/colo-compare.c +2 -1
index 10c0239f9d..1de4220fe2 100644
@@ -890,6 +890,7 @@ static void colo_compare_handle_event(void *opaque)
 
 static void colo_compare_iothread(CompareState *s)
 {
+    AioContext *ctx = iothread_get_aio_context(s->iothread);
     object_ref(OBJECT(s->iothread));
     s->worker_context = iothread_get_g_main_context(s->iothread);
 
@@ -906,7 +907,7 @@ static void colo_compare_iothread(CompareState *s)
     }
 
     colo_compare_timer_init(s);
-    s->event_bh = qemu_bh_new(colo_compare_handle_event, s);
+    s->event_bh = aio_bh_new(ctx, colo_compare_handle_event, s);
 }
 
 static char *compare_get_pri_indev(Object *obj, Error **errp)
Signature · 1 line
2.20.1
#0 lore raw

v4 cover colo-compare bugfixes

Hello Everyone,
Here are fixes for bugs that I found in my tests.

Benchmark results:
Client-to-server tcp:
without patch: ~63 Mbit/s
with patch: ~66 Mbit/s
Server-to-client tcp:
without patch: ~771 Kbit/s
with patch: ~702 Kbit/s

Regards,
Lukas Straub

Version changes:
v4:
-fix potential deadlock with notify_remote_frame
-avoid malloc and memcpy in many cases

v3:
-fix checkpatch.pl error

v2:
-better wording
-fix performance-regression in patch 3 "net/colo-compare.c: Fix deadlock in compare_chr_send" -add more bugfixes

Lukas Straub (6):
net/colo-compare.c: Create event_bh with the right AioContext chardev/char.c: Use qemu_co_sleep_ns if in coroutine net/colo-compare.c: Fix deadlock in compare_chr_send net/colo-compare.c: Only hexdump packets if tracing is enabled net/colo-compare.c, softmmu/vl.c: Check that colo-compare is active net/colo-compare.c: Correct ordering in complete and finalize

 chardev/char.c     |   7 +-
 net/colo-compare.c | 250 +++++++++++++++++++++++++++++++++------------
 net/colo-compare.h |   1 +
 net/colo.c         |   7 ++
 net/colo.h         |   1 +
 softmmu/vl.c       |   2 +
 6 files changed, 204 insertions(+), 64 deletions(-)
Signature · 1 line
2.20.1
#24 lore raw

PATCH v4 6/6 net/colo-compare.c: Correct ordering in complete and finalize

In colo_compare_complete, insert CompareState into net_compares only after everything has been initialized.
In colo_compare_finalize, remove CompareState from net_compares before anything is deinitialized.

---
 net/colo-compare.c | 45 +++++++++++++++++++++++----------------------
 1 file changed, 23 insertions(+), 22 deletions(-)
net/colo-compare.c +23 -22
index c7572d75e9..6f80bcece6 100644
@@ -1283,15 +1283,6 @@ static void colo_compare_complete(UserCreatable *uc, Error **errp)
                            s->vnet_hdr);
     }
 
-    qemu_mutex_lock(&colo_compare_mutex);
-    if (!colo_compare_active) {
-        qemu_mutex_init(&event_mtx);
-        qemu_cond_init(&event_complete_cond);
-        colo_compare_active = true;
-    }
-    QTAILQ_INSERT_TAIL(&net_compares, s, next);
-    qemu_mutex_unlock(&colo_compare_mutex);
-
     s->out_sendco.s = s;
     s->out_sendco.chr = &s->chr_out;
     s->out_sendco.notify_remote_frame = false;
@@ -1312,6 +1303,16 @@ static void colo_compare_complete(UserCreatable *uc, Error **errp)
                                                       connection_destroy);
 
     colo_compare_iothread(s);
+
+    qemu_mutex_lock(&colo_compare_mutex);
+    if (!colo_compare_active) {
+        qemu_mutex_init(&event_mtx);
+        qemu_cond_init(&event_complete_cond);
+        colo_compare_active = true;
+    }
+    QTAILQ_INSERT_TAIL(&net_compares, s, next);
+    qemu_mutex_unlock(&colo_compare_mutex);
+
     return;
 }
 
@@ -1384,19 +1385,6 @@ static void colo_compare_finalize(Object *obj)
     CompareState *s = COLO_COMPARE(obj);
     CompareState *tmp = NULL;
 
-    qemu_chr_fe_deinit(&s->chr_pri_in, false);
-    qemu_chr_fe_deinit(&s->chr_sec_in, false);
-    qemu_chr_fe_deinit(&s->chr_out, false);
-    if (s->notify_dev) {
-        qemu_chr_fe_deinit(&s->chr_notify_dev, false);
-    }
-
-    if (s->iothread) {
-        colo_compare_timer_del(s);
-    }
-
-    qemu_bh_delete(s->event_bh);
-
     qemu_mutex_lock(&colo_compare_mutex);
     QTAILQ_FOREACH(tmp, &net_compares, next) {
         if (tmp == s) {
@@ -1411,6 +1399,19 @@ static void colo_compare_finalize(Object *obj)
     }
     qemu_mutex_unlock(&colo_compare_mutex);
 
+    qemu_chr_fe_deinit(&s->chr_pri_in, false);
+    qemu_chr_fe_deinit(&s->chr_sec_in, false);
+    qemu_chr_fe_deinit(&s->chr_out, false);
+    if (s->notify_dev) {
+        qemu_chr_fe_deinit(&s->chr_notify_dev, false);
+    }
+
+    if (s->iothread) {
+        colo_compare_timer_del(s);
+    }
+
+    qemu_bh_delete(s->event_bh);
+
     AioContext *ctx = iothread_get_aio_context(s->iothread);
     aio_context_acquire(ctx);
     AIO_WAIT_WHILE(ctx, !s->out_sendco.done);
Signature · 1 line
2.20.1
#2 lore raw

PATCH v4 2/6 chardev/char.c: Use qemu_co_sleep_ns if in coroutine

In reply to Lukas Straub

This will be needed in the next patch.

---
 chardev/char.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
chardev/char.c +6 -1
index e77564060d..5c8014199f 100644
@@ -38,6 +38,7 @@
 #include "qemu/module.h"
 #include "qemu/option.h"
 #include "qemu/id.h"
+#include "qemu/coroutine.h"
 
 #include "chardev/char-mux.h"
 
@@ -119,7 +120,11 @@ static int qemu_chr_write_buffer(Chardev *s,
     retry:
         res = cc->chr_write(s, buf + *offset, len - *offset);
         if (res < 0 && errno == EAGAIN && write_all) {
-            g_usleep(100);
+            if (qemu_in_coroutine()) {
+                qemu_co_sleep_ns(QEMU_CLOCK_REALTIME, 100000);
+            } else {
+                g_usleep(100);
+            }
             goto retry;
         }
 
Signature · 1 line
2.20.1
#3 lore raw

PATCH v4 3/6 net/colo-compare.c: Fix deadlock in compare_chr_send

In reply to Lukas Straub

The chr_out chardev is connected to a filter-redirector running in the main loop. qemu_chr_fe_write_all might block here in compare_chr_send if the (socket-)buffer is full. If another filter-redirector in the main loop want's to send data to chr_pri_in it might also block if the buffer is full. This leads to a deadlock because both event loops get blocked.

Fix this by converting compare_chr_send to a coroutine and putting the packets in a send queue.

---
 net/colo-compare.c | 187 ++++++++++++++++++++++++++++++++++-----------
 net/colo.c         |   7 ++
 net/colo.h         |   1 +
 3 files changed, 150 insertions(+), 45 deletions(-)
net/colo-compare.c +142 -45
index 1de4220fe2..2a4e7f7c4e 100644
@@ -32,6 +32,9 @@
 #include "migration/migration.h"
 #include "util.h"
 
+#include "block/aio-wait.h"
+#include "qemu/coroutine.h"
+
 #define TYPE_COLO_COMPARE "colo-compare"
 #define COLO_COMPARE(obj) \
     OBJECT_CHECK(CompareState, (obj), TYPE_COLO_COMPARE)
@@ -77,6 +80,23 @@ static int event_unhandled_count;
  *                    |packet  |  |packet  +    |packet  | |packet  +
  *                    +--------+  +--------+    +--------+ +--------+
  */
+
+typedef struct SendCo {
+    Coroutine *co;
+    struct CompareState *s;
+    CharBackend *chr;
+    GQueue send_list;
+    bool notify_remote_frame;
+    bool done;
+    int ret;
+} SendCo;
+
+typedef struct SendEntry {
+    uint32_t size;
+    uint32_t vnet_hdr_len;
+    uint8_t *buf;
+} SendEntry;
+
 typedef struct CompareState {
     Object parent;
 
@@ -91,6 +111,8 @@ typedef struct CompareState {
     SocketReadState pri_rs;
     SocketReadState sec_rs;
     SocketReadState notify_rs;
+    SendCo out_sendco;
+    SendCo notify_sendco;
     bool vnet_hdr;
     uint32_t compare_timeout;
     uint32_t expired_scan_cycle;
@@ -124,10 +146,11 @@ enum {
 
 
 static int compare_chr_send(CompareState *s,
-                            const uint8_t *buf,
+                            uint8_t *buf,
                             uint32_t size,
                             uint32_t vnet_hdr_len,
-                            bool notify_remote_frame);
+                            bool notify_remote_frame,
+                            bool zero_copy);
 
 static bool packet_matches_str(const char *str,
                                const uint8_t *buf,
@@ -145,7 +168,7 @@ static void notify_remote_frame(CompareState *s)
     char msg[] = "DO_CHECKPOINT";
     int ret = 0;
 
-    ret = compare_chr_send(s, (uint8_t *)msg, strlen(msg), 0, true);
+    ret = compare_chr_send(s, (uint8_t *)msg, strlen(msg), 0, true, false);
     if (ret < 0) {
         error_report("Notify Xen COLO-frame failed");
     }
@@ -272,12 +295,13 @@ static void colo_release_primary_pkt(CompareState *s, Packet *pkt)
                            pkt->data,
                            pkt->size,
                            pkt->vnet_hdr_len,
-                           false);
+                           false,
+                           true);
     if (ret < 0) {
         error_report("colo send primary packet failed");
     }
     trace_colo_compare_main("packet same and release packet");
-    packet_destroy(pkt, NULL);
+    packet_destroy_partial(pkt, NULL);
 }
 
 /*
@@ -699,65 +723,115 @@ static void colo_compare_connection(void *opaque, void *user_data)
     }
 }
 
-static int compare_chr_send(CompareState *s,
-                            const uint8_t *buf,
-                            uint32_t size,
-                            uint32_t vnet_hdr_len,
-                            bool notify_remote_frame)
+static void coroutine_fn _compare_chr_send(void *opaque)
 {
+    SendCo *sendco = opaque;
+    CompareState *s = sendco->s;
     int ret = 0;
-    uint32_t len = htonl(size);
 
-    if (!size) {
-        return 0;
-    }
+    while (!g_queue_is_empty(&sendco->send_list)) {
+        SendEntry *entry = g_queue_pop_tail(&sendco->send_list);
+        uint32_t len = htonl(entry->size);
 
-    if (notify_remote_frame) {
-        ret = qemu_chr_fe_write_all(&s->chr_notify_dev,
-                                    (uint8_t *)&len,
-                                    sizeof(len));
-    } else {
-        ret = qemu_chr_fe_write_all(&s->chr_out, (uint8_t *)&len, sizeof(len));
-    }
+        ret = qemu_chr_fe_write_all(sendco->chr, (uint8_t *)&len, sizeof(len));
 
-    if (ret != sizeof(len)) {
-        goto err;
-    }
+        if (ret != sizeof(len)) {
+            g_free(entry->buf);
+            g_slice_free(SendEntry, entry);
+            goto err;
+        }
 
-    if (s->vnet_hdr) {
-        /*
-         * We send vnet header len make other module(like filter-redirector)
-         * know how to parse net packet correctly.
-         */
-        len = htonl(vnet_hdr_len);
+        if (!sendco->notify_remote_frame && s->vnet_hdr) {
+            /*
+             * We send vnet header len make other module(like filter-redirector)
+             * know how to parse net packet correctly.
+             */
+            len = htonl(entry->vnet_hdr_len);
 
-        if (!notify_remote_frame) {
-            ret = qemu_chr_fe_write_all(&s->chr_out,
+            ret = qemu_chr_fe_write_all(sendco->chr,
                                         (uint8_t *)&len,
                                         sizeof(len));
+
+            if (ret != sizeof(len)) {
+                g_free(entry->buf);
+                g_slice_free(SendEntry, entry);
+                goto err;
+            }
         }
 
-        if (ret != sizeof(len)) {
+        ret = qemu_chr_fe_write_all(sendco->chr,
+                                    (uint8_t *)entry->buf,
+                                    entry->size);
+
+        if (ret != entry->size) {
+            g_free(entry->buf);
+            g_slice_free(SendEntry, entry);
             goto err;
         }
+
+        g_free(entry->buf);
+        g_slice_free(SendEntry, entry);
     }
 
+    sendco->ret = 0;
+    goto out;
+
+err:
+    while (!g_queue_is_empty(&sendco->send_list)) {
+        SendEntry *entry = g_queue_pop_tail(&sendco->send_list);
+        g_free(entry->buf);
+        g_slice_free(SendEntry, entry);
+    }
+    sendco->ret = ret < 0 ? ret : -EIO;
+out:
+    sendco->co = NULL;
+    sendco->done = true;
+    aio_wait_kick();
+}
+
+static int compare_chr_send(CompareState *s,
+                            uint8_t *buf,
+                            uint32_t size,
+                            uint32_t vnet_hdr_len,
+                            bool notify_remote_frame,
+                            bool zero_copy)
+{
+    SendCo *sendco;
+    SendEntry *entry;
+
     if (notify_remote_frame) {
-        ret = qemu_chr_fe_write_all(&s->chr_notify_dev,
-                                    (uint8_t *)buf,
-                                    size);
+        sendco = &s->notify_sendco;
     } else {
-        ret = qemu_chr_fe_write_all(&s->chr_out, (uint8_t *)buf, size);
+        sendco = &s->out_sendco;
     }
 
-    if (ret != size) {
-        goto err;
+    if (!size) {
+        return 0;
     }
 
-    return 0;
+    entry = g_slice_new(SendEntry);
+    entry->size = size;
+    entry->vnet_hdr_len = vnet_hdr_len;
+    if (zero_copy) {
+        entry->buf = buf;
+    } else {
+        entry->buf = g_malloc(size);
+        memcpy(entry->buf, buf, size);
+    }
+    g_queue_push_head(&sendco->send_list, entry);
+
+    if (sendco->done) {
+        sendco->co = qemu_coroutine_create(_compare_chr_send, sendco);
+        sendco->done = false;
+        qemu_coroutine_enter(sendco->co);
+        if (sendco->done) {
+            /* report early errors */
+            return sendco->ret;
+        }
+    }
 
-err:
-    return ret < 0 ? ret : -EIO;
+    /* assume success */
+    return 0;
 }
 
 static int compare_chr_can_read(void *opaque)
@@ -1063,6 +1137,7 @@ static void compare_pri_rs_finalize(SocketReadState *pri_rs)
                          pri_rs->buf,
                          pri_rs->packet_len,
                          pri_rs->vnet_hdr_len,
+                         false,
                          false);
     } else {
         /* compare packet in the specified connection */
@@ -1093,7 +1168,7 @@ static void compare_notify_rs_finalize(SocketReadState *notify_rs)
     if (packet_matches_str("COLO_USERSPACE_PROXY_INIT",
                            notify_rs->buf,
                            notify_rs->packet_len)) {
-        ret = compare_chr_send(s, (uint8_t *)msg, strlen(msg), 0, true);
+        ret = compare_chr_send(s, (uint8_t *)msg, strlen(msg), 0, true, false);
         if (ret < 0) {
             error_report("Notify Xen COLO-frame INIT failed");
         }
@@ -1199,6 +1274,18 @@ static void colo_compare_complete(UserCreatable *uc, Error **errp)
 
     QTAILQ_INSERT_TAIL(&net_compares, s, next);
 
+    s->out_sendco.s = s;
+    s->out_sendco.chr = &s->chr_out;
+    s->out_sendco.notify_remote_frame = false;
+    s->out_sendco.done = true;
+    g_queue_init(&s->out_sendco.send_list);
+
+    s->notify_sendco.s = s;
+    s->notify_sendco.chr = &s->chr_notify_dev;
+    s->notify_sendco.notify_remote_frame = true;
+    s->notify_sendco.done = true;
+    g_queue_init(&s->notify_sendco.send_list);
+
     g_queue_init(&s->conn_list);
 
     qemu_mutex_init(&event_mtx);
@@ -1225,8 +1312,9 @@ static void colo_flush_packets(void *opaque, void *user_data)
                          pkt->data,
                          pkt->size,
                          pkt->vnet_hdr_len,
-                         false);
-        packet_destroy(pkt, NULL);
+                         false,
+                         true);
+        packet_destroy_partial(pkt, NULL);
     }
     while (!g_queue_is_empty(&conn->secondary_list)) {
         pkt = g_queue_pop_head(&conn->secondary_list);
@@ -1301,10 +1389,19 @@ static void colo_compare_finalize(Object *obj)
         }
     }
 
+    AioContext *ctx = iothread_get_aio_context(s->iothread);
+    aio_context_acquire(ctx);
+    AIO_WAIT_WHILE(ctx, !s->out_sendco.done);
+    AIO_WAIT_WHILE(ctx, !s->notify_sendco.done);
+    aio_context_release(ctx);
+
     /* Release all unhandled packets after compare thead exited */
     g_queue_foreach(&s->conn_list, colo_flush_packets, s);
+    AIO_WAIT_WHILE(NULL, !s->out_sendco.done);
 
     g_queue_clear(&s->conn_list);
+    g_queue_clear(&s->out_sendco.send_list);
+    g_queue_clear(&s->notify_sendco.send_list);
 
     if (s->connection_track_table) {
         g_hash_table_destroy(s->connection_track_table);
net/colo.c +7 -0
index 8196b35837..a6c66d829a 100644
@@ -185,6 +185,13 @@ void packet_destroy(void *opaque, void *user_data)
     g_slice_free(Packet, pkt);
 }
 
+void packet_destroy_partial(void *opaque, void *user_data)
+{
+    Packet *pkt = opaque;
+
+    g_slice_free(Packet, pkt);
+}
+
 /*
  * Clear hashtable, stop this hash growing really huge
  */
net/colo.h +1 -0
index 679314b1ca..573ab91785 100644
@@ -102,5 +102,6 @@ bool connection_has_tracked(GHashTable *connection_track_table,
 void connection_hashtable_reset(GHashTable *connection_track_table);
 Packet *packet_new(const void *data, int size, int vnet_hdr_len);
 void packet_destroy(void *opaque, void *user_data);
+void packet_destroy_partial(void *opaque, void *user_data);
 
 #endif /* NET_COLO_H */
Signature · 1 line
2.20.1
#14 lore raw

PATCH v4 5/6 net/colo-compare.c, softmmu/vl.c: Check that colo-compare is active

In reply to Lukas Straub

If the colo-compare object is removed before failover and a checkpoint happens, qemu crashes because it tries to lock the destroyed event_mtx in colo_notify_compares_event.

Fix this by checking if everything is initialized by introducing a new variable colo_compare_active which is protected by a new mutex colo_compare_mutex. The new mutex also protects against concurrent access of the net_compares list and makes sure that colo_notify_compares_event isn't active while we destroy event_mtx and event_complete_cond.

With this it also is again possible to use colo without colo-compare (periodic mode) and to use multiple colo-compare for multiple network interfaces.

---
 net/colo-compare.c | 35 +++++++++++++++++++++++++++++------
 net/colo-compare.h |  1 +
 softmmu/vl.c       |  2 ++
 3 files changed, 32 insertions(+), 6 deletions(-)
net/colo-compare.c +29 -6
index 56db3d3bfc..c7572d75e9 100644
@@ -54,6 +54,8 @@ static NotifierList colo_compare_notifiers =
 #define REGULAR_PACKET_CHECK_MS 3000
 #define DEFAULT_TIME_OUT_MS 3000
 
+static QemuMutex colo_compare_mutex;
+static bool colo_compare_active;
 static QemuMutex event_mtx;
 static QemuCond event_complete_cond;
 static int event_unhandled_count;
@@ -906,6 +908,12 @@ static void check_old_packet_regular(void *opaque)
 void colo_notify_compares_event(void *opaque, int event, Error **errp)
 {
     CompareState *s;
+    qemu_mutex_lock(&colo_compare_mutex);
+
+    if (!colo_compare_active) {
+        qemu_mutex_unlock(&colo_compare_mutex);
+        return;
+    }
 
     qemu_mutex_lock(&event_mtx);
     QTAILQ_FOREACH(s, &net_compares, next) {
@@ -919,6 +927,7 @@ void colo_notify_compares_event(void *opaque, int event, Error **errp)
     }
 
     qemu_mutex_unlock(&event_mtx);
+    qemu_mutex_unlock(&colo_compare_mutex);
 }
 
 static void colo_compare_timer_init(CompareState *s)
@@ -1274,7 +1283,14 @@ static void colo_compare_complete(UserCreatable *uc, Error **errp)
                            s->vnet_hdr);
     }
 
+    qemu_mutex_lock(&colo_compare_mutex);
+    if (!colo_compare_active) {
+        qemu_mutex_init(&event_mtx);
+        qemu_cond_init(&event_complete_cond);
+        colo_compare_active = true;
+    }
     QTAILQ_INSERT_TAIL(&net_compares, s, next);
+    qemu_mutex_unlock(&colo_compare_mutex);
 
     s->out_sendco.s = s;
     s->out_sendco.chr = &s->chr_out;
@@ -1290,9 +1306,6 @@ static void colo_compare_complete(UserCreatable *uc, Error **errp)
 
     g_queue_init(&s->conn_list);
 
-    qemu_mutex_init(&event_mtx);
-    qemu_cond_init(&event_complete_cond);
-
     s->connection_track_table = g_hash_table_new_full(connection_key_hash,
                                                       connection_key_equal,
                                                       g_free,
@@ -1384,12 +1397,19 @@ static void colo_compare_finalize(Object *obj)
 
     qemu_bh_delete(s->event_bh);
 
+    qemu_mutex_lock(&colo_compare_mutex);
     QTAILQ_FOREACH(tmp, &net_compares, next) {
         if (tmp == s) {
             QTAILQ_REMOVE(&net_compares, s, next);
             break;
         }
     }
+    if (QTAILQ_EMPTY(&net_compares)) {
+        colo_compare_active = false;
+        qemu_mutex_destroy(&event_mtx);
+        qemu_cond_destroy(&event_complete_cond);
+    }
+    qemu_mutex_unlock(&colo_compare_mutex);
 
     AioContext *ctx = iothread_get_aio_context(s->iothread);
     aio_context_acquire(ctx);
@@ -1413,15 +1433,18 @@ static void colo_compare_finalize(Object *obj)
         object_unref(OBJECT(s->iothread));
     }
 
-    qemu_mutex_destroy(&event_mtx);
-    qemu_cond_destroy(&event_complete_cond);
-
     g_free(s->pri_indev);
     g_free(s->sec_indev);
     g_free(s->outdev);
     g_free(s->notify_dev);
 }
 
+void colo_compare_init_globals(void)
+{
+    colo_compare_active = false;
+    qemu_mutex_init(&colo_compare_mutex);
+}
+
 static const TypeInfo colo_compare_info = {
     .name = TYPE_COLO_COMPARE,
     .parent = TYPE_OBJECT,
net/colo-compare.h +1 -0
index 22ddd512e2..eb483ac586 100644
@@ -17,6 +17,7 @@
 #ifndef QEMU_COLO_COMPARE_H
 #define QEMU_COLO_COMPARE_H
 
+void colo_compare_init_globals(void);
 void colo_notify_compares_event(void *opaque, int event, Error **errp);
 void colo_compare_register_notifier(Notifier *notify);
 void colo_compare_unregister_notifier(Notifier *notify);
softmmu/vl.c +2 -0
index 32c0047889..a913ed5469 100644
@@ -112,6 +112,7 @@
 #include "qapi/qmp/qerror.h"
 #include "sysemu/iothread.h"
 #include "qemu/guest-random.h"
+#include "net/colo-compare.h"
 
 #define MAX_VIRTIO_CONSOLES 1
 
@@ -2906,6 +2907,7 @@ void qemu_init(int argc, char **argv, char **envp)
     precopy_infrastructure_init();
     postcopy_infrastructure_init();
     monitor_init_globals();
+    colo_compare_init_globals();
 
     if (qcrypto_init(&err) < 0) {
         error_reportf_err(err, "cannot initialize crypto: ");
Signature · 1 line
2.20.1
#11 lore raw

PATCH v4 4/6 net/colo-compare.c: Only hexdump packets if tracing is enabled

In reply to Lukas Straub

Else the log will be flooded if there is a lot of network traffic.

---
 net/colo-compare.c | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)
net/colo-compare.c +6 -4
index 2a4e7f7c4e..56db3d3bfc 100644
@@ -483,10 +483,12 @@ sec:
         g_queue_push_head(&conn->primary_list, ppkt);
         g_queue_push_head(&conn->secondary_list, spkt);
 
-        qemu_hexdump((char *)ppkt->data, stderr,
-                     "colo-compare ppkt", ppkt->size);
-        qemu_hexdump((char *)spkt->data, stderr,
-                     "colo-compare spkt", spkt->size);
+        if (trace_event_get_state_backends(TRACE_COLO_COMPARE_MISCOMPARE)) {
+            qemu_hexdump((char *)ppkt->data, stderr,
+                        "colo-compare ppkt", ppkt->size);
+            qemu_hexdump((char *)spkt->data, stderr,
+                        "colo-compare spkt", spkt->size);
+        }
 
         colo_compare_inconsistency_notify(s);
     }
Signature · 1 line
2.20.1
Reviewed-by on v4 4/6
#12 lore raw

Re: [PATCH v4 4/6] net/colo-compare.c: Only hexdump packets if tracing is enabled

Quoted 14 lines · Lukas Straub

Else the log will be flooded if there is a lot of network traffic.

---
  net/colo-compare.c | 10 ++++++----
  1 file changed, 6 insertions(+), 4 deletions(-)
net/colo-compare.c +0 -0
index 2a4e7f7c4e..56db3d3bfc 100644

net/colo-compare.c sec: · line 486

Quoted 8 lines
net/colo-compare.c +1 -4
@@ -483,10 +483,12 @@ sec:
          g_queue_push_head(&conn->primary_list, ppkt);
          g_queue_push_head(&conn->secondary_list, spkt);
 
-        qemu_hexdump((char *)ppkt->data, stderr,
-                     "colo-compare ppkt", ppkt->size);
-        qemu_hexdump((char *)spkt->data, stderr,
-                     "colo-compare spkt", spkt->size);
+        if (trace_event_get_state_backends(TRACE_COLO_COMPARE_MISCOMPARE)) {
Quoted 9 lines
net/colo-compare.c +5 -0
+            qemu_hexdump((char *)ppkt->data, stderr,
+                        "colo-compare ppkt", ppkt->size);
+            qemu_hexdump((char *)spkt->data, stderr,
+                        "colo-compare spkt", spkt->size);
+        }
 
          colo_compare_inconsistency_notify(s);
      }
#13 lore raw

Re: [PATCH v4 4/6] net/colo-compare.c: Only hexdump packets if tracing is enabled

Quoted 10 lines · Philippe Mathieu-Daudé
Quoted 9 lines · Lukas Straub

Else the log will be flooded if there is a lot of network traffic.

---
  net/colo-compare.c | 10 ++++++----

  1 file changed, 6 insertions(+), 4 deletions(-)
Quoted 15 lines

net/colo-compare.c sec: · line 486

Quoted 13 lines
net/colo-compare.c +1 -4
index 2a4e7f7c4e..56db3d3bfc 100644
@@ -483,10 +483,12 @@ sec:
           g_queue_push_head(&conn->primary_list, ppkt);
           g_queue_push_head(&conn->secondary_list, spkt);
-        qemu_hexdump((char *)ppkt->data, stderr,
-                     "colo-compare ppkt", ppkt->size);
-        qemu_hexdump((char *)spkt->data, stderr,
-                     "colo-compare spkt", spkt->size);
+        if 
 (trace_event_get_state_backends(TRACE_COLO_COMPARE_MISCOMPARE)) {
Quoted 9 lines
Quoted 9 lines
net/colo-compare.c +5 -0
+            qemu_hexdump((char *)ppkt->data, stderr,
+                        "colo-compare ppkt", ppkt->size);
+            qemu_hexdump((char *)spkt->data, stderr,
+                        "colo-compare spkt", spkt->size);
+        }
           colo_compare_inconsistency_notify(s);
       }
 
Tested-by on v4 3/6
#4 lore raw

RE: [PATCH v4 3/6] net/colo-compare.c: Fix deadlock in compare_chr_send

In reply to Lukas Straub
Quoted 299 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Monday, May 4, 2020 6:28 PM
To: qemu-devel <[email protected]>
Cc: Zhang, Chen <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: [PATCH v4 3/6] net/colo-compare.c: Fix deadlock in compare_chr_send

The chr_out chardev is connected to a filter-redirector running in the main loop. qemu_chr_fe_write_all might block here in compare_chr_send if the (socket-)buffer is full.
If another filter-redirector in the main loop want's to send data to chr_pri_in it might also block if the buffer is full. This leads to a deadlock because both event loops get blocked.

Fix this by converting compare_chr_send to a coroutine and putting the packets in a send queue.

---
 net/colo-compare.c | 187 ++++++++++++++++++++++++++++++++++-------
----
 net/colo.c         |   7 ++
 net/colo.h         |   1 +
 3 files changed, 150 insertions(+), 45 deletions(-)
net/colo-compare.c index renamed from net/colo-compare.c +0 -0

1de4220fe2..2a4e7f7c4e 100644

net/colo-compare.c +26 -2
@@ -32,6 +32,9 @@
 #include "migration/migration.h"
 #include "util.h"
 
+#include "block/aio-wait.h"
+#include "qemu/coroutine.h"
+
 #define TYPE_COLO_COMPARE "colo-compare"
 #define COLO_COMPARE(obj) \
     OBJECT_CHECK(CompareState, (obj), TYPE_COLO_COMPARE) @@ -77,6
+80,23 @@ static int event_unhandled_count;
  *                    |packet  |  |packet  +    |packet  | |packet  +
  *                    +--------+  +--------+    +--------+ +--------+
  */
+
+typedef struct SendCo {
+    Coroutine *co;
+    struct CompareState *s;
+    CharBackend *chr;
+    GQueue send_list;
+    bool notify_remote_frame;
+    bool done;
+    int ret;
+} SendCo;
+
+typedef struct SendEntry {
+    uint32_t size;
+    uint32_t vnet_hdr_len;
+    uint8_t *buf;
+} SendEntry;
+
 typedef struct CompareState {
     Object parent;
@@ -91,6 +111,8 @@ typedef struct CompareState {
     SocketReadState pri_rs;
     SocketReadState sec_rs;
     SocketReadState notify_rs;
+    SendCo out_sendco;
+    SendCo notify_sendco;
     bool vnet_hdr;
     uint32_t compare_timeout;
     uint32_t expired_scan_cycle;
@@ -124,10 +146,11 @@ enum {
 
 
 static int compare_chr_send(CompareState *s,
-                            const uint8_t *buf,
+                            uint8_t *buf,
                             uint32_t size,
                             uint32_t vnet_hdr_len,
-                            bool notify_remote_frame);
+                            bool notify_remote_frame,
+                            bool zero_copy);
 
 static bool packet_matches_str(const char *str,
                                const uint8_t *buf, @@ -145,7 +168,7 

@@ static void notify_remote_frame(CompareState *s)

     char msg[] = "DO_CHECKPOINT";
     int ret = 0;
net/colo-compare.c +5 -3
-    ret = compare_chr_send(s, (uint8_t *)msg, strlen(msg), 0, true);
+    ret = compare_chr_send(s, (uint8_t *)msg, strlen(msg), 0, true, 
+ false);
     if (ret < 0) {
         error_report("Notify Xen COLO-frame failed");
     }
@@ -272,12 +295,13 @@ static void
 colo_release_primary_pkt(CompareState *s, Packet *pkt)
                            pkt->data,
                            pkt->size,
                            pkt->vnet_hdr_len,
-                           false);
+                           false,
+                           true);
     if (ret < 0) {
         error_report("colo send primary packet failed");
     }
     trace_colo_compare_main("packet same and release packet");
-    packet_destroy(pkt, NULL);
+    packet_destroy_partial(pkt, NULL);
 }
 

/*

net/colo-compare.c +87 -37
@@ -699,65 +723,115 @@ static void colo_compare_connection(void 
 *opaque, void *user_data)
     }
 }
 
-static int compare_chr_send(CompareState *s,
-                            const uint8_t *buf,
-                            uint32_t size,
-                            uint32_t vnet_hdr_len,
-                            bool notify_remote_frame)
+static void coroutine_fn _compare_chr_send(void *opaque)
 {
+    SendCo *sendco = opaque;
+    CompareState *s = sendco->s;
     int ret = 0;
-    uint32_t len = htonl(size);
 
-    if (!size) {
-        return 0;
-    }
+    while (!g_queue_is_empty(&sendco->send_list)) {
+        SendEntry *entry = g_queue_pop_tail(&sendco->send_list);
+        uint32_t len = htonl(entry->size);
 
-    if (notify_remote_frame) {
-        ret = qemu_chr_fe_write_all(&s->chr_notify_dev,
-                                    (uint8_t *)&len,
-                                    sizeof(len));
-    } else {
-        ret = qemu_chr_fe_write_all(&s->chr_out, (uint8_t *)&len, sizeof(len));
-    }
+        ret = qemu_chr_fe_write_all(sendco->chr, (uint8_t *)&len, 
+ sizeof(len));
 
-    if (ret != sizeof(len)) {
-        goto err;
-    }
+        if (ret != sizeof(len)) {
+            g_free(entry->buf);
+            g_slice_free(SendEntry, entry);
+            goto err;
+        }
 
-    if (s->vnet_hdr) {
-        /*
-         * We send vnet header len make other module(like filter-redirector)
-         * know how to parse net packet correctly.
-         */
-        len = htonl(vnet_hdr_len);
+        if (!sendco->notify_remote_frame && s->vnet_hdr) {
+            /*
+             * We send vnet header len make other module(like filter-redirector)
+             * know how to parse net packet correctly.
+             */
+            len = htonl(entry->vnet_hdr_len);
 
-        if (!notify_remote_frame) {
-            ret = qemu_chr_fe_write_all(&s->chr_out,
+            ret = qemu_chr_fe_write_all(sendco->chr,
                                         (uint8_t *)&len,
                                         sizeof(len));
+
+            if (ret != sizeof(len)) {
+                g_free(entry->buf);
+                g_slice_free(SendEntry, entry);
+                goto err;
+            }
         }
 
-        if (ret != sizeof(len)) {
+        ret = qemu_chr_fe_write_all(sendco->chr,
+                                    (uint8_t *)entry->buf,
+                                    entry->size);
+
+        if (ret != entry->size) {
+            g_free(entry->buf);
+            g_slice_free(SendEntry, entry);
             goto err;
         }
+
+        g_free(entry->buf);
+        g_slice_free(SendEntry, entry);
     }
 
+    sendco->ret = 0;
+    goto out;
+
+err:
+    while (!g_queue_is_empty(&sendco->send_list)) {
+        SendEntry *entry = g_queue_pop_tail(&sendco->send_list);
+        g_free(entry->buf);
+        g_slice_free(SendEntry, entry);
+    }
+    sendco->ret = ret < 0 ? ret : -EIO;
+out:
+    sendco->co = NULL;
+    sendco->done = true;
+    aio_wait_kick();
+}
+
+static int compare_chr_send(CompareState *s,
+                            uint8_t *buf,
+                            uint32_t size,
+                            uint32_t vnet_hdr_len,
+                            bool notify_remote_frame,
+                            bool zero_copy) {
+    SendCo *sendco;
+    SendEntry *entry;
+
     if (notify_remote_frame) {
-        ret = qemu_chr_fe_write_all(&s->chr_notify_dev,
-                                    (uint8_t *)buf,
-                                    size);
+        sendco = &s->notify_sendco;
     } else {
-        ret = qemu_chr_fe_write_all(&s->chr_out, (uint8_t *)buf, size);
+        sendco = &s->out_sendco;
     }
 
-    if (ret != size) {
-        goto err;
+    if (!size) {
+        return 0;
     }
 
-    return 0;
+    entry = g_slice_new(SendEntry);
+    entry->size = size;
+    entry->vnet_hdr_len = vnet_hdr_len;
+    if (zero_copy) {
+        entry->buf = buf;
+    } else {
+        entry->buf = g_malloc(size);
+        memcpy(entry->buf, buf, size);
+    }
+    g_queue_push_head(&sendco->send_list, entry);
+
+    if (sendco->done) {
+        sendco->co = qemu_coroutine_create(_compare_chr_send, sendco);
+        sendco->done = false;
+        qemu_coroutine_enter(sendco->co);
+        if (sendco->done) {
+            /* report early errors */
+            return sendco->ret;
+        }
+    }
 
-err:
-    return ret < 0 ? ret : -EIO;
+    /* assume success */
+    return 0;
 }
 

static int compare_chr_can_read(void *opaque) @@ -1063,6 +1137,7 @@
static void compare_pri_rs_finalize(SocketReadState *pri_rs)

                          pri_rs->buf,
                          pri_rs->packet_len,
                          pri_rs->vnet_hdr_len,
net/colo-compare.c +7 -1
+                         false,
                          false);
     } else {
         /* compare packet in the specified connection */ @@ -1093,7
+1168,7 @@ static void compare_notify_rs_finalize(SocketReadState
+*notify_rs)
     if (packet_matches_str("COLO_USERSPACE_PROXY_INIT",
                            notify_rs->buf,
                            notify_rs->packet_len)) {
-        ret = compare_chr_send(s, (uint8_t *)msg, strlen(msg), 0, true);
+        ret = compare_chr_send(s, (uint8_t *)msg, strlen(msg), 0, 
+ true, false);
         if (ret < 0) {
             error_report("Notify Xen COLO-frame INIT failed");
         }
@@ -1199,6 +1274,18 @@ static void
 colo_compare_complete(UserCreatable *uc, Error **errp)
 
     QTAILQ_INSERT_TAIL(&net_compares, s, next);
 
+    s->out_sendco.s = s;
+    s->out_sendco.chr = &s->chr_out;

net/colo-compare.c static void · line 1289

Quoted 10 lines
net/colo-compare.c +10 -0
+    s->out_sendco.notify_remote_frame = false;
+    s->out_sendco.done = true;
+    g_queue_init(&s->out_sendco.send_list);
+
+    s->notify_sendco.s = s;
+    s->notify_sendco.chr = &s->chr_notify_dev;
+    s->notify_sendco.notify_remote_frame = true;
+    s->notify_sendco.done = true;
+    g_queue_init(&s->notify_sendco.send_list);
+

No need to init the notify_sendco each time, because the notify dev just an optional parameter. You can use the if (s->notify_dev) here. Just Xen use the chr_notify_dev.

Overall, make the chr_send job to coroutine is a good idea. It looks good for me.
And your patch inspired me, it looks we can re-use the compare_chr_send code on filter mirror/redirector too.

Quoted 18 lines
net/colo-compare.c +0 -0
     g_queue_init(&s->conn_list);
 
     qemu_mutex_init(&event_mtx);
net/colo-compare.c +0 -0
@@ -1225,8 +1312,9 @@ static void colo_flush_packets(void *opaque, 

void
*user_data)

                          pkt->data,
                          pkt->size,
                          pkt->vnet_hdr_len,
net/colo-compare.c +3 -2
-                         false);
-        packet_destroy(pkt, NULL);
+                         false,
+                         true);
+        packet_destroy_partial(pkt, NULL);
     }
     while (!g_queue_is_empty(&conn->secondary_list)) {
         pkt = g_queue_pop_head(&conn->secondary_list);

net/colo-compare.c static void colo_compare_finalize(Object *obj) · line 1395

Quoted 7 lines
net/colo-compare.c +4 -0
@@ -1301,10 +1389,19 @@ static void colo_compare_finalize(Object *obj)
         }
     }
 
+    AioContext *ctx = iothread_get_aio_context(s->iothread);
+    aio_context_acquire(ctx);
+    AIO_WAIT_WHILE(ctx, !s->out_sendco.done);
+    AIO_WAIT_WHILE(ctx, !s->notify_sendco.done);

Same as above.

net/colo-compare.c static void colo_compare_finalize(Object *obj) · line 1404

Quoted 9 lines
net/colo-compare.c +5 -0
+    aio_context_release(ctx);
+
     /* Release all unhandled packets after compare thead exited */
     g_queue_foreach(&s->conn_list, colo_flush_packets, s);
+    AIO_WAIT_WHILE(NULL, !s->out_sendco.done);
 
     g_queue_clear(&s->conn_list);
+    g_queue_clear(&s->out_sendco.send_list);
+    g_queue_clear(&s->notify_sendco.send_list);

Same as above.

Quoted 34 lines
net/colo-compare.c +0 -0
     if (s->connection_track_table) {
         g_hash_table_destroy(s->connection_track_table);
net/colo.c +6 -0
index 8196b35837..a6c66d829a 100644
@@ -185,6 +185,13 @@ void packet_destroy(void *opaque, void *user_data)
     g_slice_free(Packet, pkt);
 }
 
+void packet_destroy_partial(void *opaque, void *user_data) {
+    Packet *pkt = opaque;
+
+    g_slice_free(Packet, pkt);
+}
+
 /*
  * Clear hashtable, stop this hash growing really huge
  */
net/colo.h +0 -0
index 679314b1ca..573ab91785 100644
@@ -102,5 +102,6 @@ bool connection_has_tracked(GHashTable 
 *connection_track_table,  void connection_hashtable_reset(GHashTable
 *connection_track_table);  Packet *packet_new(const void *data, int 

size, int vnet_hdr_len); void packet_destroy(void *opaque, void *user_data);
+void packet_destroy_partial(void *opaque, void *user_data);

#endif /* NET_COLO_H */
--
2.20.1

#15 lore raw

RE: [PATCH v4 5/6] net/colo-compare.c, softmmu/vl.c: Check that colo-compare is active

In reply to Lukas Straub
Quoted 24 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Monday, May 4, 2020 6:28 PM
To: qemu-devel <[email protected]>
Cc: Zhang, Chen <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: [PATCH v4 5/6] net/colo-compare.c, softmmu/vl.c: Check that colo- compare is active

If the colo-compare object is removed before failover and a checkpoint happens, qemu crashes because it tries to lock the destroyed event_mtx in colo_notify_compares_event.

Fix this by checking if everything is initialized by introducing a new variable colo_compare_active which is protected by a new mutex colo_compare_mutex. The new mutex also protects against concurrent access of the net_compares list and makes sure that colo_notify_compares_event isn't active while we destroy event_mtx and event_complete_cond.

Quoted 3 lines

With this it also is again possible to use colo without colo-compare (periodic mode) and to use multiple colo-compare for multiple network interfaces.

Hi Lukas,

For this case I think we don't need to touch vl.c code, we can solve this issue from another perspective:
How to remove colo-compare?
User will use qemu-monitor or QMP command to disable an object, so we just need return operation failed When user try to remove colo-compare object while COLO is running.

Thanks
Zhang Chen

Quoted 141 lines
---
 net/colo-compare.c | 35 +++++++++++++++++++++++++++++------
 net/colo-compare.h |  1 +
 softmmu/vl.c       |  2 ++
 3 files changed, 32 insertions(+), 6 deletions(-)
net/colo-compare.c index renamed from net/colo-compare.c +0 -0

56db3d3bfc..c7572d75e9 100644

net/colo-compare.c +0 -0
@@ -54,6 +54,8 @@ static NotifierList colo_compare_notifiers =  #define

REGULAR_PACKET_CHECK_MS 3000 #define DEFAULT_TIME_OUT_MS 3000

net/colo-compare.c +8 -0
+static QemuMutex colo_compare_mutex;
+static bool colo_compare_active;
 static QemuMutex event_mtx;
 static QemuCond event_complete_cond;
 static int event_unhandled_count;
@@ -906,6 +908,12 @@ static void check_old_packet_regular(void *opaque)
 void colo_notify_compares_event(void *opaque, int event, Error **errp)  {
     CompareState *s;
+    qemu_mutex_lock(&colo_compare_mutex);
+
+    if (!colo_compare_active) {
+        qemu_mutex_unlock(&colo_compare_mutex);
+        return;
+    }
 
     qemu_mutex_lock(&event_mtx);
     QTAILQ_FOREACH(s, &net_compares, next) { @@ -919,6 +927,7 @@ void
 colo_notify_compares_event(void *opaque, int event, Error **errp)
     }

     qemu_mutex_unlock(&event_mtx);

+ qemu_mutex_unlock(&colo_compare_mutex);
}

static void colo_compare_timer_init(CompareState *s) @@ -1274,7 +1283,14
@@ static void colo_compare_complete(UserCreatable *uc, Error **errp)

                            s->vnet_hdr);
     }
net/colo-compare.c +7 -3
+    qemu_mutex_lock(&colo_compare_mutex);
+    if (!colo_compare_active) {
+        qemu_mutex_init(&event_mtx);
+        qemu_cond_init(&event_complete_cond);
+        colo_compare_active = true;
+    }
     QTAILQ_INSERT_TAIL(&net_compares, s, next);
+    qemu_mutex_unlock(&colo_compare_mutex);
 
     s->out_sendco.s = s;
     s->out_sendco.chr = &s->chr_out;
@@ -1290,9 +1306,6 @@ static void colo_compare_complete(UserCreatable
 *uc, Error **errp)
 
     g_queue_init(&s->conn_list);
 
-    qemu_mutex_init(&event_mtx);
-    qemu_cond_init(&event_complete_cond);
-
     s->connection_track_table =
 g_hash_table_new_full(connection_key_hash,
                                                       connection_key_equal,
                                                       g_free, @@ -1384,12 +1397,19 @@ static void

colo_compare_finalize(Object *obj)

     qemu_bh_delete(s->event_bh);
net/colo-compare.c +13 -3
+    qemu_mutex_lock(&colo_compare_mutex);
     QTAILQ_FOREACH(tmp, &net_compares, next) {
         if (tmp == s) {
             QTAILQ_REMOVE(&net_compares, s, next);
             break;
         }
     }
+    if (QTAILQ_EMPTY(&net_compares)) {
+        colo_compare_active = false;
+        qemu_mutex_destroy(&event_mtx);
+        qemu_cond_destroy(&event_complete_cond);
+    }
+    qemu_mutex_unlock(&colo_compare_mutex);
 
     AioContext *ctx = iothread_get_aio_context(s->iothread);
     aio_context_acquire(ctx);
@@ -1413,15 +1433,18 @@ static void colo_compare_finalize(Object *obj)
         object_unref(OBJECT(s->iothread));
     }
 
-    qemu_mutex_destroy(&event_mtx);
-    qemu_cond_destroy(&event_complete_cond);
-
     g_free(s->pri_indev);
     g_free(s->sec_indev);
     g_free(s->outdev);
     g_free(s->notify_dev);
 }
 
+void colo_compare_init_globals(void)
+{
+    colo_compare_active = false;
+    qemu_mutex_init(&colo_compare_mutex);
+}
+
 static const TypeInfo colo_compare_info = {
     .name = TYPE_COLO_COMPARE,
     .parent = TYPE_OBJECT,
net/colo-compare.h index renamed from net/colo-compare.h +0 -0

22ddd512e2..eb483ac586 100644

net/colo-compare.h +1 -0
@@ -17,6 +17,7 @@
 #ifndef QEMU_COLO_COMPARE_H
 #define QEMU_COLO_COMPARE_H
 
+void colo_compare_init_globals(void);
 void colo_notify_compares_event(void *opaque, int event, Error **errp);

void colo_compare_register_notifier(Notifier *notify); void colo_compare_unregister_notifier(Notifier *notify); diff --git a/softmmu/vl.c b/softmmu/vl.c index 32c0047889..a913ed5469 100644

softmmu/vl.c +2 -1
@@ -112,6 +112,7 @@
 #include "qapi/qmp/qerror.h"
 #include "sysemu/iothread.h"
 #include "qemu/guest-random.h"
+#include "net/colo-compare.h"
 
 #define MAX_VIRTIO_CONSOLES 1
 
@@ -2906,6 +2907,7 @@ void qemu_init(int argc, char **argv, char **envp)
     precopy_infrastructure_init();
     postcopy_infrastructure_init();
     monitor_init_globals();
+    colo_compare_init_globals();
 
     if (qcrypto_init(&err) < 0) {
         error_reportf_err(err, "cannot initialize crypto: ");
--

2.20.1

#25 lore raw

RE: [PATCH v4 6/6] net/colo-compare.c: Correct ordering in complete and finalize

In reply to Lukas Straub
Quoted 13 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Monday, May 4, 2020 6:28 PM
To: qemu-devel <[email protected]>
Cc: Zhang, Chen <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: [PATCH v4 6/6] net/colo-compare.c: Correct ordering in complete and finalize

Quoted 4 lines

In colo_compare_complete, insert CompareState into net_compares only after everything has been initialized.
In colo_compare_finalize, remove CompareState from net_compares before anything is deinitialized.

S/deinitialized/finalized

It looks no dependences on each step on initialization and finalization. Do you means we just need add/remove each colo-compare module at last in logic? Or current code have some issue?

Thanks
Zhang Chen

Quoted 86 lines
---
 net/colo-compare.c | 45 +++++++++++++++++++++++----------------------
 1 file changed, 23 insertions(+), 22 deletions(-)
net/colo-compare.c index renamed from net/colo-compare.c +0 -0

c7572d75e9..6f80bcece6 100644

net/colo-compare.c +0 -9
@@ -1283,15 +1283,6 @@ static void
 colo_compare_complete(UserCreatable *uc, Error **errp)
                            s->vnet_hdr);
     }
 
-    qemu_mutex_lock(&colo_compare_mutex);
-    if (!colo_compare_active) {
-        qemu_mutex_init(&event_mtx);
-        qemu_cond_init(&event_complete_cond);
-        colo_compare_active = true;
-    }
-    QTAILQ_INSERT_TAIL(&net_compares, s, next);
-    qemu_mutex_unlock(&colo_compare_mutex);
-
     s->out_sendco.s = s;
     s->out_sendco.chr = &s->chr_out;
     s->out_sendco.notify_remote_frame = false; @@ -1312,6 +1303,16 @@
static void colo_compare_complete(UserCreatable *uc, Error **errp)
                                                       connection_destroy);

     colo_compare_iothread(s);
net/colo-compare.c +23 -14
+
+    qemu_mutex_lock(&colo_compare_mutex);
+    if (!colo_compare_active) {
+        qemu_mutex_init(&event_mtx);
+        qemu_cond_init(&event_complete_cond);
+        colo_compare_active = true;
+    }
+    QTAILQ_INSERT_TAIL(&net_compares, s, next);
+    qemu_mutex_unlock(&colo_compare_mutex);
+
     return;
 }
@@ -1384,19 +1385,6 @@ static void colo_compare_finalize(Object *obj)
     CompareState *s = COLO_COMPARE(obj);
     CompareState *tmp = NULL;
 
-    qemu_chr_fe_deinit(&s->chr_pri_in, false);
-    qemu_chr_fe_deinit(&s->chr_sec_in, false);
-    qemu_chr_fe_deinit(&s->chr_out, false);
-    if (s->notify_dev) {
-        qemu_chr_fe_deinit(&s->chr_notify_dev, false);
-    }
-
-    if (s->iothread) {
-        colo_compare_timer_del(s);
-    }
-
-    qemu_bh_delete(s->event_bh);
-
     qemu_mutex_lock(&colo_compare_mutex);
     QTAILQ_FOREACH(tmp, &net_compares, next) {
         if (tmp == s) {
@@ -1411,6 +1399,19 @@ static void colo_compare_finalize(Object *obj)
     }
     qemu_mutex_unlock(&colo_compare_mutex);
 
+    qemu_chr_fe_deinit(&s->chr_pri_in, false);
+    qemu_chr_fe_deinit(&s->chr_sec_in, false);
+    qemu_chr_fe_deinit(&s->chr_out, false);
+    if (s->notify_dev) {
+        qemu_chr_fe_deinit(&s->chr_notify_dev, false);
+    }
+
+    if (s->iothread) {
+        colo_compare_timer_del(s);
+    }
+
+    qemu_bh_delete(s->event_bh);
+
     AioContext *ctx = iothread_get_aio_context(s->iothread);
     aio_context_acquire(ctx);
     AIO_WAIT_WHILE(ctx, !s->out_sendco.done);
--

2.20.1

#5 lore raw

Re: [PATCH v4 3/6] net/colo-compare.c: Fix deadlock in compare_chr_send

In reply to Zhang, Chen
Quoted 295 lines · Chen
Quoted 295 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Monday, May 4, 2020 6:28 PM
To: qemu-devel <[email protected]>
Cc: Zhang, Chen <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: [PATCH v4 3/6] net/colo-compare.c: Fix deadlock in compare_chr_send

The chr_out chardev is connected to a filter-redirector running in the main loop. qemu_chr_fe_write_all might block here in compare_chr_send if the (socket-)buffer is full.
If another filter-redirector in the main loop want's to send data to chr_pri_in it might also block if the buffer is full. This leads to a deadlock because both event loops get blocked.

Fix this by converting compare_chr_send to a coroutine and putting the packets in a send queue.

---
 net/colo-compare.c | 187 ++++++++++++++++++++++++++++++++++-------
----
 net/colo.c         |   7 ++
 net/colo.h         |   1 +
 3 files changed, 150 insertions(+), 45 deletions(-)
net/colo-compare.c index renamed from net/colo-compare.c +0 -0

1de4220fe2..2a4e7f7c4e 100644

net/colo-compare.c +26 -2
@@ -32,6 +32,9 @@
 #include "migration/migration.h"
 #include "util.h"
 
+#include "block/aio-wait.h"
+#include "qemu/coroutine.h"
+
 #define TYPE_COLO_COMPARE "colo-compare"
 #define COLO_COMPARE(obj) \
     OBJECT_CHECK(CompareState, (obj), TYPE_COLO_COMPARE) @@ -77,6
+80,23 @@ static int event_unhandled_count;
  *                    |packet  |  |packet  +    |packet  | |packet  +
  *                    +--------+  +--------+    +--------+ +--------+
  */
+
+typedef struct SendCo {
+    Coroutine *co;
+    struct CompareState *s;
+    CharBackend *chr;
+    GQueue send_list;
+    bool notify_remote_frame;
+    bool done;
+    int ret;
+} SendCo;
+
+typedef struct SendEntry {
+    uint32_t size;
+    uint32_t vnet_hdr_len;
+    uint8_t *buf;
+} SendEntry;
+
 typedef struct CompareState {
     Object parent;
@@ -91,6 +111,8 @@ typedef struct CompareState {
     SocketReadState pri_rs;
     SocketReadState sec_rs;
     SocketReadState notify_rs;
+    SendCo out_sendco;
+    SendCo notify_sendco;
     bool vnet_hdr;
     uint32_t compare_timeout;
     uint32_t expired_scan_cycle;
@@ -124,10 +146,11 @@ enum {
 
 
 static int compare_chr_send(CompareState *s,
-                            const uint8_t *buf,
+                            uint8_t *buf,
                             uint32_t size,
                             uint32_t vnet_hdr_len,
-                            bool notify_remote_frame);
+                            bool notify_remote_frame,
+                            bool zero_copy);
 
 static bool packet_matches_str(const char *str,
                                const uint8_t *buf, @@ -145,7 +168,7 

@@ static void notify_remote_frame(CompareState *s)

     char msg[] = "DO_CHECKPOINT";
     int ret = 0;
net/colo-compare.c +5 -3
-    ret = compare_chr_send(s, (uint8_t *)msg, strlen(msg), 0, true);
+    ret = compare_chr_send(s, (uint8_t *)msg, strlen(msg), 0, true, 
+ false);
     if (ret < 0) {
         error_report("Notify Xen COLO-frame failed");
     }
@@ -272,12 +295,13 @@ static void
 colo_release_primary_pkt(CompareState *s, Packet *pkt)
                            pkt->data,
                            pkt->size,
                            pkt->vnet_hdr_len,
-                           false);
+                           false,
+                           true);
     if (ret < 0) {
         error_report("colo send primary packet failed");
     }
     trace_colo_compare_main("packet same and release packet");
-    packet_destroy(pkt, NULL);
+    packet_destroy_partial(pkt, NULL);
 }
 

/*

net/colo-compare.c +87 -37
@@ -699,65 +723,115 @@ static void colo_compare_connection(void 
 *opaque, void *user_data)
     }
 }
 
-static int compare_chr_send(CompareState *s,
-                            const uint8_t *buf,
-                            uint32_t size,
-                            uint32_t vnet_hdr_len,
-                            bool notify_remote_frame)
+static void coroutine_fn _compare_chr_send(void *opaque)
 {
+    SendCo *sendco = opaque;
+    CompareState *s = sendco->s;
     int ret = 0;
-    uint32_t len = htonl(size);
 
-    if (!size) {
-        return 0;
-    }
+    while (!g_queue_is_empty(&sendco->send_list)) {
+        SendEntry *entry = g_queue_pop_tail(&sendco->send_list);
+        uint32_t len = htonl(entry->size);
 
-    if (notify_remote_frame) {
-        ret = qemu_chr_fe_write_all(&s->chr_notify_dev,
-                                    (uint8_t *)&len,
-                                    sizeof(len));
-    } else {
-        ret = qemu_chr_fe_write_all(&s->chr_out, (uint8_t *)&len, sizeof(len));
-    }
+        ret = qemu_chr_fe_write_all(sendco->chr, (uint8_t *)&len, 
+ sizeof(len));
 
-    if (ret != sizeof(len)) {
-        goto err;
-    }
+        if (ret != sizeof(len)) {
+            g_free(entry->buf);
+            g_slice_free(SendEntry, entry);
+            goto err;
+        }
 
-    if (s->vnet_hdr) {
-        /*
-         * We send vnet header len make other module(like filter-redirector)
-         * know how to parse net packet correctly.
-         */
-        len = htonl(vnet_hdr_len);
+        if (!sendco->notify_remote_frame && s->vnet_hdr) {
+            /*
+             * We send vnet header len make other module(like filter-redirector)
+             * know how to parse net packet correctly.
+             */
+            len = htonl(entry->vnet_hdr_len);
 
-        if (!notify_remote_frame) {
-            ret = qemu_chr_fe_write_all(&s->chr_out,
+            ret = qemu_chr_fe_write_all(sendco->chr,
                                         (uint8_t *)&len,
                                         sizeof(len));
+
+            if (ret != sizeof(len)) {
+                g_free(entry->buf);
+                g_slice_free(SendEntry, entry);
+                goto err;
+            }
         }
 
-        if (ret != sizeof(len)) {
+        ret = qemu_chr_fe_write_all(sendco->chr,
+                                    (uint8_t *)entry->buf,
+                                    entry->size);
+
+        if (ret != entry->size) {
+            g_free(entry->buf);
+            g_slice_free(SendEntry, entry);
             goto err;
         }
+
+        g_free(entry->buf);
+        g_slice_free(SendEntry, entry);
     }
 
+    sendco->ret = 0;
+    goto out;
+
+err:
+    while (!g_queue_is_empty(&sendco->send_list)) {
+        SendEntry *entry = g_queue_pop_tail(&sendco->send_list);
+        g_free(entry->buf);
+        g_slice_free(SendEntry, entry);
+    }
+    sendco->ret = ret < 0 ? ret : -EIO;
+out:
+    sendco->co = NULL;
+    sendco->done = true;
+    aio_wait_kick();
+}
+
+static int compare_chr_send(CompareState *s,
+                            uint8_t *buf,
+                            uint32_t size,
+                            uint32_t vnet_hdr_len,
+                            bool notify_remote_frame,
+                            bool zero_copy) {
+    SendCo *sendco;
+    SendEntry *entry;
+
     if (notify_remote_frame) {
-        ret = qemu_chr_fe_write_all(&s->chr_notify_dev,
-                                    (uint8_t *)buf,
-                                    size);
+        sendco = &s->notify_sendco;
     } else {
-        ret = qemu_chr_fe_write_all(&s->chr_out, (uint8_t *)buf, size);
+        sendco = &s->out_sendco;
     }
 
-    if (ret != size) {
-        goto err;
+    if (!size) {
+        return 0;
     }
 
-    return 0;
+    entry = g_slice_new(SendEntry);
+    entry->size = size;
+    entry->vnet_hdr_len = vnet_hdr_len;
+    if (zero_copy) {
+        entry->buf = buf;
+    } else {
+        entry->buf = g_malloc(size);
+        memcpy(entry->buf, buf, size);
+    }
+    g_queue_push_head(&sendco->send_list, entry);
+
+    if (sendco->done) {
+        sendco->co = qemu_coroutine_create(_compare_chr_send, sendco);
+        sendco->done = false;
+        qemu_coroutine_enter(sendco->co);
+        if (sendco->done) {
+            /* report early errors */
+            return sendco->ret;
+        }
+    }
 
-err:
-    return ret < 0 ? ret : -EIO;
+    /* assume success */
+    return 0;
 }
 

static int compare_chr_can_read(void *opaque) @@ -1063,6 +1137,7 @@
static void compare_pri_rs_finalize(SocketReadState *pri_rs)

                          pri_rs->buf,
                          pri_rs->packet_len,
                          pri_rs->vnet_hdr_len,
net/colo-compare.c +5 -1
+                         false,
                          false);
     } else {
         /* compare packet in the specified connection */ @@ -1093,7
+1168,7 @@ static void compare_notify_rs_finalize(SocketReadState
+*notify_rs)
     if (packet_matches_str("COLO_USERSPACE_PROXY_INIT",
                            notify_rs->buf,
                            notify_rs->packet_len)) {
-        ret = compare_chr_send(s, (uint8_t *)msg, strlen(msg), 0, true);
+        ret = compare_chr_send(s, (uint8_t *)msg, strlen(msg), 0, 
+ true, false);
         if (ret < 0) {
             error_report("Notify Xen COLO-frame INIT failed");
         }
@@ -1199,6 +1274,18 @@ static void
 colo_compare_complete(UserCreatable *uc, Error **errp)
 
     QTAILQ_INSERT_TAIL(&net_compares, s, next);
 
Quoted 15 lines

net/colo-compare.c static void colo_compare_complete(UserCreatable *uc, Error **errp) · line 1288

Quoted 12 lines
net/colo-compare.c +12 -0
+    s->out_sendco.s = s;
+    s->out_sendco.chr = &s->chr_out;
+    s->out_sendco.notify_remote_frame = false;
+    s->out_sendco.done = true;
+    g_queue_init(&s->out_sendco.send_list);
+
+    s->notify_sendco.s = s;
+    s->notify_sendco.chr = &s->chr_notify_dev;
+    s->notify_sendco.notify_remote_frame = true;
+    s->notify_sendco.done = true;
+    g_queue_init(&s->notify_sendco.send_list);
+  

No need to init the notify_sendco each time, because the notify dev just an optional parameter. You can use the if (s->notify_dev) here. Just Xen use the chr_notify_dev.

Ok, I will change that and the code below in the next version.

Quoted 2 lines

Overall, make the chr_send job to coroutine is a good idea. It looks good for me.
And your patch inspired me, it looks we can re-use the compare_chr_send code on filter mirror/redirector too.

I already have patch for that, but I don't think it is a good idea, because the guest then can send packets faster than colo-compare can process. This leads bufferbloat and the performance drops in my tests:
Client-to-server tcp:
without patch: ~66 Mbit/s
with patch: ~59 Mbit/s
Server-to-client tcp:
without patch: ~702 Kbit/s
with patch: ~328 Kbit/s

Regards,
Lukas Straub

Quoted 78 lines
Quoted 18 lines
net/colo-compare.c +0 -0
     g_queue_init(&s->conn_list);
 
     qemu_mutex_init(&event_mtx);
net/colo-compare.c +0 -0
@@ -1225,8 +1312,9 @@ static void colo_flush_packets(void *opaque, 

void
*user_data)

                          pkt->data,
                          pkt->size,
                          pkt->vnet_hdr_len,
net/colo-compare.c +3 -2
-                         false);
-        packet_destroy(pkt, NULL);
+                         false,
+                         true);
+        packet_destroy_partial(pkt, NULL);
     }
     while (!g_queue_is_empty(&conn->secondary_list)) {
         pkt = g_queue_pop_head(&conn->secondary_list);

net/colo-compare.c static void colo_compare_finalize(Object *obj) · line 1395

Quoted 7 lines
net/colo-compare.c +4 -0
@@ -1301,10 +1389,19 @@ static void colo_compare_finalize(Object *obj)
         }
     }
 
+    AioContext *ctx = iothread_get_aio_context(s->iothread);
+    aio_context_acquire(ctx);
+    AIO_WAIT_WHILE(ctx, !s->out_sendco.done);
+    AIO_WAIT_WHILE(ctx, !s->notify_sendco.done);  

Same as above.

net/colo-compare.c static void colo_compare_finalize(Object *obj) · line 1404

Quoted 9 lines
net/colo-compare.c +5 -0
+    aio_context_release(ctx);
+
     /* Release all unhandled packets after compare thead exited */
     g_queue_foreach(&s->conn_list, colo_flush_packets, s);
+    AIO_WAIT_WHILE(NULL, !s->out_sendco.done);
 
     g_queue_clear(&s->conn_list);
+    g_queue_clear(&s->out_sendco.send_list);
+    g_queue_clear(&s->notify_sendco.send_list);  

Same as above.

Quoted 34 lines
net/colo-compare.c +0 -0
     if (s->connection_track_table) {
         g_hash_table_destroy(s->connection_track_table);
net/colo.c +6 -0
index 8196b35837..a6c66d829a 100644
@@ -185,6 +185,13 @@ void packet_destroy(void *opaque, void *user_data)
     g_slice_free(Packet, pkt);
 }
 
+void packet_destroy_partial(void *opaque, void *user_data) {
+    Packet *pkt = opaque;
+
+    g_slice_free(Packet, pkt);
+}
+
 /*
  * Clear hashtable, stop this hash growing really huge
  */
net/colo.h +0 -0
index 679314b1ca..573ab91785 100644
@@ -102,5 +102,6 @@ bool connection_has_tracked(GHashTable 
 *connection_track_table,  void connection_hashtable_reset(GHashTable
 *connection_track_table);  Packet *packet_new(const void *data, int 

size, int vnet_hdr_len); void packet_destroy(void *opaque, void *user_data);
+void packet_destroy_partial(void *opaque, void *user_data);

#endif /* NET_COLO_H */
--
2.20.1

#16 lore raw

Re: [PATCH v4 5/6] net/colo-compare.c, softmmu/vl.c: Check that colo-compare is active

In reply to Zhang, Chen
Quoted 26 lines · Chen
Quoted 22 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Monday, May 4, 2020 6:28 PM
To: qemu-devel <[email protected]>
Cc: Zhang, Chen <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: [PATCH v4 5/6] net/colo-compare.c, softmmu/vl.c: Check that colo- compare is active

If the colo-compare object is removed before failover and a checkpoint happens, qemu crashes because it tries to lock the destroyed event_mtx in colo_notify_compares_event.

Fix this by checking if everything is initialized by introducing a new variable colo_compare_active which is protected by a new mutex colo_compare_mutex. The new mutex also protects against concurrent access of the net_compares list and makes sure that colo_notify_compares_event isn't active while we destroy event_mtx and event_complete_cond.

Quoted 3 lines

With this it also is again possible to use colo without colo-compare (periodic mode) and to use multiple colo-compare for multiple network interfaces.

Quoted 6 lines

Hi Lukas,

For this case I think we don't need to touch vl.c code, we can solve this issue from another perspective:
How to remove colo-compare?
User will use qemu-monitor or QMP command to disable an object, so we just need return operation failed When user try to remove colo-compare object while COLO is running.

Yeah, but that still leaves the other problem that colo can't be used without colo-compare (qemu crashes then).

Regards,
Lukas Straub

Quoted 145 lines

Thanks
Zhang Chen

Quoted 141 lines
---
 net/colo-compare.c | 35 +++++++++++++++++++++++++++++------
 net/colo-compare.h |  1 +
 softmmu/vl.c       |  2 ++
 3 files changed, 32 insertions(+), 6 deletions(-)
net/colo-compare.c index renamed from net/colo-compare.c +0 -0

56db3d3bfc..c7572d75e9 100644

net/colo-compare.c +0 -0
@@ -54,6 +54,8 @@ static NotifierList colo_compare_notifiers =  #define

REGULAR_PACKET_CHECK_MS 3000 #define DEFAULT_TIME_OUT_MS 3000

net/colo-compare.c +8 -0
+static QemuMutex colo_compare_mutex;
+static bool colo_compare_active;
 static QemuMutex event_mtx;
 static QemuCond event_complete_cond;
 static int event_unhandled_count;
@@ -906,6 +908,12 @@ static void check_old_packet_regular(void *opaque)
 void colo_notify_compares_event(void *opaque, int event, Error **errp)  {
     CompareState *s;
+    qemu_mutex_lock(&colo_compare_mutex);
+
+    if (!colo_compare_active) {
+        qemu_mutex_unlock(&colo_compare_mutex);
+        return;
+    }
 
     qemu_mutex_lock(&event_mtx);
     QTAILQ_FOREACH(s, &net_compares, next) { @@ -919,6 +927,7 @@ void
 colo_notify_compares_event(void *opaque, int event, Error **errp)
     }

     qemu_mutex_unlock(&event_mtx);

+ qemu_mutex_unlock(&colo_compare_mutex);
}

static void colo_compare_timer_init(CompareState *s) @@ -1274,7 +1283,14
@@ static void colo_compare_complete(UserCreatable *uc, Error **errp)

                            s->vnet_hdr);
     }
net/colo-compare.c +7 -3
+    qemu_mutex_lock(&colo_compare_mutex);
+    if (!colo_compare_active) {
+        qemu_mutex_init(&event_mtx);
+        qemu_cond_init(&event_complete_cond);
+        colo_compare_active = true;
+    }
     QTAILQ_INSERT_TAIL(&net_compares, s, next);
+    qemu_mutex_unlock(&colo_compare_mutex);
 
     s->out_sendco.s = s;
     s->out_sendco.chr = &s->chr_out;
@@ -1290,9 +1306,6 @@ static void colo_compare_complete(UserCreatable
 *uc, Error **errp)
 
     g_queue_init(&s->conn_list);
 
-    qemu_mutex_init(&event_mtx);
-    qemu_cond_init(&event_complete_cond);
-
     s->connection_track_table =
 g_hash_table_new_full(connection_key_hash,
                                                       connection_key_equal,
                                                       g_free, @@ -1384,12 +1397,19 @@ static void

colo_compare_finalize(Object *obj)

     qemu_bh_delete(s->event_bh);
net/colo-compare.c +13 -3
+    qemu_mutex_lock(&colo_compare_mutex);
     QTAILQ_FOREACH(tmp, &net_compares, next) {
         if (tmp == s) {
             QTAILQ_REMOVE(&net_compares, s, next);
             break;
         }
     }
+    if (QTAILQ_EMPTY(&net_compares)) {
+        colo_compare_active = false;
+        qemu_mutex_destroy(&event_mtx);
+        qemu_cond_destroy(&event_complete_cond);
+    }
+    qemu_mutex_unlock(&colo_compare_mutex);
 
     AioContext *ctx = iothread_get_aio_context(s->iothread);
     aio_context_acquire(ctx);
@@ -1413,15 +1433,18 @@ static void colo_compare_finalize(Object *obj)
         object_unref(OBJECT(s->iothread));
     }
 
-    qemu_mutex_destroy(&event_mtx);
-    qemu_cond_destroy(&event_complete_cond);
-
     g_free(s->pri_indev);
     g_free(s->sec_indev);
     g_free(s->outdev);
     g_free(s->notify_dev);
 }
 
+void colo_compare_init_globals(void)
+{
+    colo_compare_active = false;
+    qemu_mutex_init(&colo_compare_mutex);
+}
+
 static const TypeInfo colo_compare_info = {
     .name = TYPE_COLO_COMPARE,
     .parent = TYPE_OBJECT,
net/colo-compare.h index renamed from net/colo-compare.h +0 -0

22ddd512e2..eb483ac586 100644

net/colo-compare.h +1 -0
@@ -17,6 +17,7 @@
 #ifndef QEMU_COLO_COMPARE_H
 #define QEMU_COLO_COMPARE_H
 
+void colo_compare_init_globals(void);
 void colo_notify_compares_event(void *opaque, int event, Error **errp);

void colo_compare_register_notifier(Notifier *notify); void colo_compare_unregister_notifier(Notifier *notify); diff --git a/softmmu/vl.c b/softmmu/vl.c index 32c0047889..a913ed5469 100644

softmmu/vl.c +2 -1
@@ -112,6 +112,7 @@
 #include "qapi/qmp/qerror.h"
 #include "sysemu/iothread.h"
 #include "qemu/guest-random.h"
+#include "net/colo-compare.h"
 
 #define MAX_VIRTIO_CONSOLES 1
 
@@ -2906,6 +2907,7 @@ void qemu_init(int argc, char **argv, char **envp)
     precopy_infrastructure_init();
     postcopy_infrastructure_init();
     monitor_init_globals();
+    colo_compare_init_globals();
 
     if (qcrypto_init(&err) < 0) {
         error_reportf_err(err, "cannot initialize crypto: ");
--

2.20.1

#26 lore raw

Re: [PATCH v4 6/6] net/colo-compare.c: Correct ordering in complete and finalize

In reply to Zhang, Chen
Quoted 16 lines · Chen
Quoted 11 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Monday, May 4, 2020 6:28 PM
To: qemu-devel <[email protected]>
Cc: Zhang, Chen <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: [PATCH v4 6/6] net/colo-compare.c: Correct ordering in complete and finalize

Quoted 4 lines

In colo_compare_complete, insert CompareState into net_compares only after everything has been initialized.
In colo_compare_finalize, remove CompareState from net_compares before anything is deinitialized.

Quoted 4 lines

S/deinitialized/finalized

It looks no dependences on each step on initialization and finalization. Do you means we just need add/remove each colo-compare module at last in logic?

Yes. While I didn't see any crashes here, there is the possibility that if colo-compare is removed during checkpoint, the destroyed event_bh is called from colo_notify_compares_event. Same with colo_compare_complete (very unlikely) if colo-compare is created while colo is running, colo_notify_compares_event may call the uninitialized event_bh.

Regards,
Lukas Straub

Quoted 91 lines

Or current code have some issue?

Thanks
Zhang Chen

Quoted 86 lines
---
 net/colo-compare.c | 45 +++++++++++++++++++++++----------------------
 1 file changed, 23 insertions(+), 22 deletions(-)
net/colo-compare.c index renamed from net/colo-compare.c +0 -0

c7572d75e9..6f80bcece6 100644

net/colo-compare.c +0 -9
@@ -1283,15 +1283,6 @@ static void
 colo_compare_complete(UserCreatable *uc, Error **errp)
                            s->vnet_hdr);
     }
 
-    qemu_mutex_lock(&colo_compare_mutex);
-    if (!colo_compare_active) {
-        qemu_mutex_init(&event_mtx);
-        qemu_cond_init(&event_complete_cond);
-        colo_compare_active = true;
-    }
-    QTAILQ_INSERT_TAIL(&net_compares, s, next);
-    qemu_mutex_unlock(&colo_compare_mutex);
-
     s->out_sendco.s = s;
     s->out_sendco.chr = &s->chr_out;
     s->out_sendco.notify_remote_frame = false; @@ -1312,6 +1303,16 @@
static void colo_compare_complete(UserCreatable *uc, Error **errp)
                                                       connection_destroy);

     colo_compare_iothread(s);
net/colo-compare.c +23 -14
+
+    qemu_mutex_lock(&colo_compare_mutex);
+    if (!colo_compare_active) {
+        qemu_mutex_init(&event_mtx);
+        qemu_cond_init(&event_complete_cond);
+        colo_compare_active = true;
+    }
+    QTAILQ_INSERT_TAIL(&net_compares, s, next);
+    qemu_mutex_unlock(&colo_compare_mutex);
+
     return;
 }
@@ -1384,19 +1385,6 @@ static void colo_compare_finalize(Object *obj)
     CompareState *s = COLO_COMPARE(obj);
     CompareState *tmp = NULL;
 
-    qemu_chr_fe_deinit(&s->chr_pri_in, false);
-    qemu_chr_fe_deinit(&s->chr_sec_in, false);
-    qemu_chr_fe_deinit(&s->chr_out, false);
-    if (s->notify_dev) {
-        qemu_chr_fe_deinit(&s->chr_notify_dev, false);
-    }
-
-    if (s->iothread) {
-        colo_compare_timer_del(s);
-    }
-
-    qemu_bh_delete(s->event_bh);
-
     qemu_mutex_lock(&colo_compare_mutex);
     QTAILQ_FOREACH(tmp, &net_compares, next) {
         if (tmp == s) {
@@ -1411,6 +1399,19 @@ static void colo_compare_finalize(Object *obj)
     }
     qemu_mutex_unlock(&colo_compare_mutex);
 
+    qemu_chr_fe_deinit(&s->chr_pri_in, false);
+    qemu_chr_fe_deinit(&s->chr_sec_in, false);
+    qemu_chr_fe_deinit(&s->chr_out, false);
+    if (s->notify_dev) {
+        qemu_chr_fe_deinit(&s->chr_notify_dev, false);
+    }
+
+    if (s->iothread) {
+        colo_compare_timer_del(s);
+    }
+
+    qemu_bh_delete(s->event_bh);
+
     AioContext *ctx = iothread_get_aio_context(s->iothread);
     aio_context_acquire(ctx);
     AIO_WAIT_WHILE(ctx, !s->out_sendco.done);
--

2.20.1

#6 lore raw

RE: [PATCH v4 3/6] net/colo-compare.c: Fix deadlock in compare_chr_send

In reply to Lukas Straub
Quoted 341 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Thursday, May 7, 2020 11:51 PM
To: Zhang, Chen <[email protected]>
Cc: qemu-devel <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: Re: [PATCH v4 3/6] net/colo-compare.c: Fix deadlock in compare_chr_send

Quoted 18 lines · Chen
Quoted 18 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Monday, May 4, 2020 6:28 PM
To: qemu-devel <[email protected]>
Cc: Zhang, Chen <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: [PATCH v4 3/6] net/colo-compare.c: Fix deadlock in compare_chr_send

The chr_out chardev is connected to a filter-redirector running in the main loop. qemu_chr_fe_write_all might block here in compare_chr_send if the (socket-)buffer is full.
If another filter-redirector in the main loop want's to send data to chr_pri_in it might also block if the buffer is full. This leads to a deadlock because both event loops get blocked.

Quoted 6 lines
Quoted 6 lines

Fix this by converting compare_chr_send to a coroutine and putting the packets in a send queue.

---
 net/colo-compare.c | 187 ++++++++++++++++++++++++++++++++++---

----

Quoted 13 lines
Quoted 13 lines

----

 net/colo.c         |   7 ++
 net/colo.h         |   1 +
 3 files changed, 150 insertions(+), 45 deletions(-)
net/colo-compare.c index renamed from net/colo-compare.c +0 -0

1de4220fe2..2a4e7f7c4e 100644

net/colo-compare.c +0 -0
@@ -32,6 +32,9 @@
 #include "migration/migration.h"
 #include "util.h"
 
Quoted 6 lines

net/colo-compare.c

Quoted 6 lines
net/colo-compare.c +3 -0
+#include "block/aio-wait.h"
+#include "qemu/coroutine.h"
+
 #define TYPE_COLO_COMPARE "colo-compare"
 #define COLO_COMPARE(obj) \
     OBJECT_CHECK(CompareState, (obj), TYPE_COLO_COMPARE) @@ -

77,6

Quoted 98 lines
Quoted 98 lines

+80,23 @@ static int event_unhandled_count;

  *                    |packet  |  |packet  +    |packet  | |packet  +
  *                    +--------+  +--------+    +--------+ +--------+

*/

net/colo-compare.c +22 -2
+
+typedef struct SendCo {
+    Coroutine *co;
+    struct CompareState *s;
+    CharBackend *chr;
+    GQueue send_list;
+    bool notify_remote_frame;
+    bool done;
+    int ret;
+} SendCo;
+
+typedef struct SendEntry {
+    uint32_t size;
+    uint32_t vnet_hdr_len;
+    uint8_t *buf;
+} SendEntry;
+
 typedef struct CompareState {
     Object parent;
@@ -91,6 +111,8 @@ typedef struct CompareState {
     SocketReadState pri_rs;
     SocketReadState sec_rs;
     SocketReadState notify_rs;
+    SendCo out_sendco;
+    SendCo notify_sendco;
     bool vnet_hdr;
     uint32_t compare_timeout;
     uint32_t expired_scan_cycle;
@@ -124,10 +146,11 @@ enum {
 
 
 static int compare_chr_send(CompareState *s,
-                            const uint8_t *buf,
+                            uint8_t *buf,
                             uint32_t size,
                             uint32_t vnet_hdr_len,
-                            bool notify_remote_frame);
+                            bool notify_remote_frame,
+                            bool zero_copy);
 
 static bool packet_matches_str(const char *str,
                                const uint8_t *buf, @@ -145,7 +168,7

@@ static void notify_remote_frame(CompareState *s)

     char msg[] = "DO_CHECKPOINT";
     int ret = 0;
net/colo-compare.c +5 -3
-    ret = compare_chr_send(s, (uint8_t *)msg, strlen(msg), 0, true);
+    ret = compare_chr_send(s, (uint8_t *)msg, strlen(msg), 0, true,
+ false);
     if (ret < 0) {
         error_report("Notify Xen COLO-frame failed");
     }
@@ -272,12 +295,13 @@ static void
 colo_release_primary_pkt(CompareState *s, Packet *pkt)
                            pkt->data,
                            pkt->size,
                            pkt->vnet_hdr_len,
-                           false);
+                           false,
+                           true);
     if (ret < 0) {
         error_report("colo send primary packet failed");
     }
     trace_colo_compare_main("packet same and release packet");
-    packet_destroy(pkt, NULL);
+    packet_destroy_partial(pkt, NULL);
 }
 

/*

net/colo-compare.c +6 -9
@@ -699,65 +723,115 @@ static void colo_compare_connection(void
 *opaque, void *user_data)
     }
 }
 
-static int compare_chr_send(CompareState *s,
-                            const uint8_t *buf,
-                            uint32_t size,
-                            uint32_t vnet_hdr_len,
-                            bool notify_remote_frame)
+static void coroutine_fn _compare_chr_send(void *opaque)
 {
+    SendCo *sendco = opaque;
+    CompareState *s = sendco->s;
     int ret = 0;
-    uint32_t len = htonl(size);
 
-    if (!size) {
-        return 0;
-    }
+    while (!g_queue_is_empty(&sendco->send_list)) {
+        SendEntry *entry = g_queue_pop_tail(&sendco->send_list);
+        uint32_t len = htonl(entry->size);
 
Quoted 6 lines

net/colo-compare.c static void colo_compare_connection(void *opaque, void *user_data)

Quoted 6 lines
net/colo-compare.c +0 -6
-    if (notify_remote_frame) {
-        ret = qemu_chr_fe_write_all(&s->chr_notify_dev,
-                                    (uint8_t *)&len,
-                                    sizeof(len));
-    } else {
-        ret = qemu_chr_fe_write_all(&s->chr_out, (uint8_t *)&len,

sizeof(len));

Quoted 13 lines
Quoted 13 lines
net/colo-compare.c +7 -4
-    }
+        ret = qemu_chr_fe_write_all(sendco->chr, (uint8_t *)&len,
+ sizeof(len));
 
-    if (ret != sizeof(len)) {
-        goto err;
-    }
+        if (ret != sizeof(len)) {
+            g_free(entry->buf);
+            g_slice_free(SendEntry, entry);
+            goto err;
+        }
 
Quoted 9 lines

net/colo-compare.c static void colo_compare_connection(void *opaque, void *user_data)

Quoted 9 lines
net/colo-compare.c +3 -6
-    if (s->vnet_hdr) {
-        /*
-         * We send vnet header len make other module(like filter-redirector)
-         * know how to parse net packet correctly.
-         */
-        len = htonl(vnet_hdr_len);
+        if (!sendco->notify_remote_frame && s->vnet_hdr) {
+            /*
+             * We send vnet header len make other module(like filter-

redirector)

Quoted 73 lines
Quoted 73 lines
net/colo-compare.c +49 -9
+             * know how to parse net packet correctly.
+             */
+            len = htonl(entry->vnet_hdr_len);
 
-        if (!notify_remote_frame) {
-            ret = qemu_chr_fe_write_all(&s->chr_out,
+            ret = qemu_chr_fe_write_all(sendco->chr,
                                         (uint8_t *)&len,
                                         sizeof(len));
+
+            if (ret != sizeof(len)) {
+                g_free(entry->buf);
+                g_slice_free(SendEntry, entry);
+                goto err;
+            }
         }
 
-        if (ret != sizeof(len)) {
+        ret = qemu_chr_fe_write_all(sendco->chr,
+                                    (uint8_t *)entry->buf,
+                                    entry->size);
+
+        if (ret != entry->size) {
+            g_free(entry->buf);
+            g_slice_free(SendEntry, entry);
             goto err;
         }
+
+        g_free(entry->buf);
+        g_slice_free(SendEntry, entry);
     }
 
+    sendco->ret = 0;
+    goto out;
+
+err:
+    while (!g_queue_is_empty(&sendco->send_list)) {
+        SendEntry *entry = g_queue_pop_tail(&sendco->send_list);
+        g_free(entry->buf);
+        g_slice_free(SendEntry, entry);
+    }
+    sendco->ret = ret < 0 ? ret : -EIO;
+out:
+    sendco->co = NULL;
+    sendco->done = true;
+    aio_wait_kick();
+}
+
+static int compare_chr_send(CompareState *s,
+                            uint8_t *buf,
+                            uint32_t size,
+                            uint32_t vnet_hdr_len,
+                            bool notify_remote_frame,
+                            bool zero_copy) {
+    SendCo *sendco;
+    SendEntry *entry;
+
     if (notify_remote_frame) {
-        ret = qemu_chr_fe_write_all(&s->chr_notify_dev,
-                                    (uint8_t *)buf,
-                                    size);
+        sendco = &s->notify_sendco;
     } else {
-        ret = qemu_chr_fe_write_all(&s->chr_out, (uint8_t *)buf, size);
+        sendco = &s->out_sendco;
     }
 
-    if (ret != size) {
-        goto err;
+    if (!size) {
+        return 0;
     }
 
Quoted 14 lines
Quoted 4 lines
net/colo-compare.c +3 -1
-    return 0;
+    entry = g_slice_new(SendEntry);
+    entry->size = size;
+    entry->vnet_hdr_len = vnet_hdr_len;

net/colo-compare.c static void colo_compare_connection(void *opaque, void *user_data)

Quoted 10 lines
net/colo-compare.c +10 -0
+    if (zero_copy) {
+        entry->buf = buf;
+    } else {
+        entry->buf = g_malloc(size);
+        memcpy(entry->buf, buf, size);
+    }
+    g_queue_push_head(&sendco->send_list, entry);
+
+    if (sendco->done) {
+        sendco->co = qemu_coroutine_create(_compare_chr_send,

sendco);

Quoted 8 lines
Quoted 8 lines
net/colo-compare.c +7 -0
+        sendco->done = false;
+        qemu_coroutine_enter(sendco->co);
+        if (sendco->done) {
+            /* report early errors */
+            return sendco->ret;
+        }
+    }
 
Quoted 7 lines
Quoted 7 lines
net/colo-compare.c +2 -2
-err:
-    return ret < 0 ? ret : -EIO;
+    /* assume success */
+    return 0;
 }
 

static int compare_chr_can_read(void *opaque) @@ -1063,6 +1137,7

@@

Quoted 24 lines
Quoted 24 lines

static void compare_pri_rs_finalize(SocketReadState *pri_rs)

                          pri_rs->buf,
                          pri_rs->packet_len,
                          pri_rs->vnet_hdr_len,
net/colo-compare.c +5 -1
+                         false,
                          false);
     } else {
         /* compare packet in the specified connection */ @@ -1093,7
+1168,7 @@ static void compare_notify_rs_finalize(SocketReadState
+*notify_rs)
     if (packet_matches_str("COLO_USERSPACE_PROXY_INIT",
                            notify_rs->buf,
                            notify_rs->packet_len)) {
-        ret = compare_chr_send(s, (uint8_t *)msg, strlen(msg), 0, true);
+        ret = compare_chr_send(s, (uint8_t *)msg, strlen(msg), 0,
+ true, false);
         if (ret < 0) {
             error_report("Notify Xen COLO-frame INIT failed");
         }
@@ -1199,6 +1274,18 @@ static void
 colo_compare_complete(UserCreatable *uc, Error **errp)
 
     QTAILQ_INSERT_TAIL(&net_compares, s, next);
 
Quoted 14 lines

net/colo-compare.c static void colo_compare_complete(UserCreatable *uc, Error **errp) · line 1288

Quoted 12 lines
net/colo-compare.c +12 -0
+    s->out_sendco.s = s;
+    s->out_sendco.chr = &s->chr_out;
+    s->out_sendco.notify_remote_frame = false;
+    s->out_sendco.done = true;
+    g_queue_init(&s->out_sendco.send_list);
+
+    s->notify_sendco.s = s;
+    s->notify_sendco.chr = &s->chr_notify_dev;
+    s->notify_sendco.notify_remote_frame = true;
+    s->notify_sendco.done = true;
+    g_queue_init(&s->notify_sendco.send_list);
+

No need to init the notify_sendco each time, because the notify dev just

an optional parameter.

Quoted 1 line

You can use the if (s->notify_dev) here. Just Xen use the chr_notify_dev.

Ok, I will change that and the code below in the next version.

Quoted 1 line

Overall, make the chr_send job to coroutine is a good idea. It looks good

for me.

Quoted 1 line

And your patch inspired me, it looks we can re-use the compare_chr_send

code on filter mirror/redirector too.

Quoted 9 lines

I already have patch for that, but I don't think it is a good idea, because the guest then can send packets faster than colo-compare can process. This leads bufferbloat and the performance drops in my tests:
Client-to-server tcp:
without patch: ~66 Mbit/s
with patch: ~59 Mbit/s
Server-to-client tcp:
without patch: ~702 Kbit/s
with patch: ~328 Kbit/s

Oh, a big performance drop, is that caused by memcpy/zero_copy parts ?

Thanks
Zhang Chen

Quoted 83 lines

Regards,
Lukas Straub

Quoted 44 lines
Quoted 18 lines
net/colo-compare.c +0 -0
     g_queue_init(&s->conn_list);
 
     qemu_mutex_init(&event_mtx);
net/colo-compare.c +0 -0
@@ -1225,8 +1312,9 @@ static void colo_flush_packets(void *opaque,

void
*user_data)

                          pkt->data,
                          pkt->size,
                          pkt->vnet_hdr_len,
net/colo-compare.c +3 -2
-                         false);
-        packet_destroy(pkt, NULL);
+                         false,
+                         true);
+        packet_destroy_partial(pkt, NULL);
     }
     while (!g_queue_is_empty(&conn->secondary_list)) {
         pkt = g_queue_pop_head(&conn->secondary_list);

net/colo-compare.c static void colo_compare_finalize(Object *obj) · line 1395

Quoted 7 lines
net/colo-compare.c +4 -0
@@ -1301,10 +1389,19 @@ static void colo_compare_finalize(Object *obj)
         }
     }
 
+    AioContext *ctx = iothread_get_aio_context(s->iothread);
+    aio_context_acquire(ctx);
+    AIO_WAIT_WHILE(ctx, !s->out_sendco.done);
+    AIO_WAIT_WHILE(ctx, !s->notify_sendco.done);

Same as above.

net/colo-compare.c static void colo_compare_finalize(Object *obj) · line 1404

Quoted 9 lines
net/colo-compare.c +5 -0
+    aio_context_release(ctx);
+
     /* Release all unhandled packets after compare thead exited */
     g_queue_foreach(&s->conn_list, colo_flush_packets, s);
+    AIO_WAIT_WHILE(NULL, !s->out_sendco.done);
 
     g_queue_clear(&s->conn_list);
+    g_queue_clear(&s->out_sendco.send_list);
+    g_queue_clear(&s->notify_sendco.send_list);

Same as above.

Quoted 1 line
Quoted 7 lines

net/colo.c void packet_destroy(void *opaque, void

Quoted 7 lines
     if (s->connection_track_table) {
         g_hash_table_destroy(s->connection_track_table);
net/colo.c index 8196b35837..a6c66d829a renamed from net/colo.c +0 -0

100644

net/colo.c +0 -0
@@ -185,6 +185,13 @@ void packet_destroy(void *opaque, void

*user_data)

Quoted 27 lines
Quoted 26 lines
net/colo.c +6 -0
     g_slice_free(Packet, pkt);
 }
 
+void packet_destroy_partial(void *opaque, void *user_data) {
+    Packet *pkt = opaque;
+
+    g_slice_free(Packet, pkt);
+}
+
 /*
  * Clear hashtable, stop this hash growing really huge
  */
net/colo.h index 679314b1ca..573ab91785 renamed from net/colo.h +0 -0

100644

net/colo.h +0 -0
@@ -102,5 +102,6 @@ bool connection_has_tracked(GHashTable
 *connection_track_table,  void connection_hashtable_reset(GHashTable
 *connection_track_table);  Packet *packet_new(const void *data, int

size, int vnet_hdr_len); void packet_destroy(void *opaque, void *user_data);
+void packet_destroy_partial(void *opaque, void *user_data);

#endif /* NET_COLO_H */
--
2.20.1

#17 lore raw

RE: [PATCH v4 5/6] net/colo-compare.c, softmmu/vl.c: Check that colo-compare is active

In reply to Lukas Straub
Quoted 46 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Thursday, May 7, 2020 11:54 PM
To: Zhang, Chen <[email protected]>
Cc: qemu-devel <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: Re: [PATCH v4 5/6] net/colo-compare.c, softmmu/vl.c: Check that colo-compare is active

Quoted 22 lines · Chen
Quoted 22 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Monday, May 4, 2020 6:28 PM
To: qemu-devel <[email protected]>
Cc: Zhang, Chen <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: [PATCH v4 5/6] net/colo-compare.c, softmmu/vl.c: Check that colo- compare is active

If the colo-compare object is removed before failover and a checkpoint happens, qemu crashes because it tries to lock the destroyed event_mtx in colo_notify_compares_event.

Fix this by checking if everything is initialized by introducing a new variable colo_compare_active which is protected by a new mutex colo_compare_mutex. The new mutex also protects against concurrent access of the net_compares list and makes sure that colo_notify_compares_event isn't active while we destroy event_mtx and event_complete_cond.

Quoted 8 lines
Quoted 4 lines

With this it also is again possible to use colo without colo-compare (periodic
mode) and to use multiple colo-compare for multiple network interfaces.

Hi Lukas,

For this case I think we don't need to touch vl.c code, we can solve this

Quoted 8 lines

issue from another perspective:

Quoted 3 lines

How to remove colo-compare?
User will use qemu-monitor or QMP command to disable an object, so we just need return operation failed When user try to remove colo-compare

object while COLO is running.

Yeah, but that still leaves the other problem that colo can't be used without colo-compare (qemu crashes then).

Yes, the COLO-compare is necessary module in COLO original design.
At most cases, user need it do dynamic sync.
For rare cases, maybe we can add a new colo-compare parameter to bypass all the network workload.

Thanks
Zhang Chen

Quoted 152 lines

Regards,
Lukas Straub

Quoted 10 lines

Thanks
Zhang Chen

Quoted 7 lines
---
 net/colo-compare.c | 35 +++++++++++++++++++++++++++++------
 net/colo-compare.h |  1 +
 softmmu/vl.c       |  2 ++
 3 files changed, 32 insertions(+), 6 deletions(-)
Quoted 6 lines

net/colo-compare.c static NotifierList colo_compare_notifiers = · line 54

Quoted 6 lines
net/colo-compare.c index renamed from net/colo-compare.c +0 -0

56db3d3bfc..c7572d75e9 100644

net/colo-compare.c +0 -0
@@ -54,6 +54,8 @@ static NotifierList colo_compare_notifiers =
 #define REGULAR_PACKET_CHECK_MS 3000  #define

DEFAULT_TIME_OUT_MS

Quoted 22 lines
Quoted 22 lines

3000

net/colo-compare.c +2 -0
+static QemuMutex colo_compare_mutex; static bool
+colo_compare_active;
 static QemuMutex event_mtx;
 static QemuCond event_complete_cond;  static int

event_unhandled_count; @@ -906,6 +908,12 @@ static void check_old_packet_regular(void *opaque) void

colo_notify_compares_event(void *opaque, int event, Error **errp)  {
     CompareState *s;
net/colo-compare.c +6 -0
+    qemu_mutex_lock(&colo_compare_mutex);
+
+    if (!colo_compare_active) {
+        qemu_mutex_unlock(&colo_compare_mutex);
+        return;
+    }
 
     qemu_mutex_lock(&event_mtx);
     QTAILQ_FOREACH(s, &net_compares, next) { @@ -919,6 +927,7 @@

void colo_notify_compares_event(void *opaque, int event, Error **errp)

     }
Quoted 6 lines
Quoted 6 lines
     qemu_mutex_unlock(&event_mtx);

+ qemu_mutex_unlock(&colo_compare_mutex);
}

static void colo_compare_timer_init(CompareState *s) @@ -1274,7
+1283,14 @@ static void colo_compare_complete(UserCreatable *uc,

Error **errp)

Quoted 88 lines
Quoted 88 lines
                            s->vnet_hdr);
     }
net/colo-compare.c +7 -0
+    qemu_mutex_lock(&colo_compare_mutex);
+    if (!colo_compare_active) {
+        qemu_mutex_init(&event_mtx);
+        qemu_cond_init(&event_complete_cond);
+        colo_compare_active = true;
+    }
     QTAILQ_INSERT_TAIL(&net_compares, s, next);
+    qemu_mutex_unlock(&colo_compare_mutex);
 
     s->out_sendco.s = s;
     s->out_sendco.chr = &s->chr_out; @@ -1290,9 +1306,6 @@ static

void colo_compare_complete(UserCreatable
*uc, Error **errp)

     g_queue_init(&s->conn_list);
net/colo-compare.c +0 -3
-    qemu_mutex_init(&event_mtx);
-    qemu_cond_init(&event_complete_cond);
-
     s->connection_track_table =

g_hash_table_new_full(connection_key_hash,

                                                       connection_key_equal,
                                                       g_free, @@

-1384,12 +1397,19 @@ static void colo_compare_finalize(Object *obj)

     qemu_bh_delete(s->event_bh);
net/colo-compare.c +12 -3
+    qemu_mutex_lock(&colo_compare_mutex);
     QTAILQ_FOREACH(tmp, &net_compares, next) {
         if (tmp == s) {
             QTAILQ_REMOVE(&net_compares, s, next);
             break;
         }
     }
+    if (QTAILQ_EMPTY(&net_compares)) {
+        colo_compare_active = false;
+        qemu_mutex_destroy(&event_mtx);
+        qemu_cond_destroy(&event_complete_cond);
+    }
+    qemu_mutex_unlock(&colo_compare_mutex);
 
     AioContext *ctx = iothread_get_aio_context(s->iothread);
     aio_context_acquire(ctx);
@@ -1413,15 +1433,18 @@ static void colo_compare_finalize(Object *obj)
         object_unref(OBJECT(s->iothread));
     }
 
-    qemu_mutex_destroy(&event_mtx);
-    qemu_cond_destroy(&event_complete_cond);
-
     g_free(s->pri_indev);
     g_free(s->sec_indev);
     g_free(s->outdev);
     g_free(s->notify_dev);
 }
 
+void colo_compare_init_globals(void) {
+    colo_compare_active = false;
+    qemu_mutex_init(&colo_compare_mutex);
+}
+
 static const TypeInfo colo_compare_info = {
     .name = TYPE_COLO_COMPARE,
     .parent = TYPE_OBJECT,
net/colo-compare.h index renamed from net/colo-compare.h +0 -0

22ddd512e2..eb483ac586 100644

net/colo-compare.h +1 -0
@@ -17,6 +17,7 @@
 #ifndef QEMU_COLO_COMPARE_H
 #define QEMU_COLO_COMPARE_H
 
+void colo_compare_init_globals(void);
 void colo_notify_compares_event(void *opaque, int event, Error
 **errp); void colo_compare_register_notifier(Notifier *notify);

void colo_compare_unregister_notifier(Notifier *notify); diff --git a/softmmu/vl.c b/softmmu/vl.c index 32c0047889..a913ed5469 100644

softmmu/vl.c +1 -0
@@ -112,6 +112,7 @@
 #include "qapi/qmp/qerror.h"
 #include "sysemu/iothread.h"
 #include "qemu/guest-random.h"
+#include "net/colo-compare.h"
 
Quoted 3 lines

softmmu/vl.c void qemu_init(int argc, char **argv, char

Quoted 3 lines
softmmu/vl.c +0 -0
 #define MAX_VIRTIO_CONSOLES 1
 
@@ -2906,6 +2907,7 @@ void qemu_init(int argc, char **argv, char

**envp)

Quoted 10 lines
Quoted 9 lines
softmmu/vl.c +1 -1
     precopy_infrastructure_init();
     postcopy_infrastructure_init();
     monitor_init_globals();
+    colo_compare_init_globals();
 
     if (qcrypto_init(&err) < 0) {
         error_reportf_err(err, "cannot initialize crypto: ");
--

2.20.1

#27 lore raw

RE: [PATCH v4 6/6] net/colo-compare.c: Correct ordering in complete and finalize

In reply to Lukas Straub
Quoted 37 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Friday, May 8, 2020 12:09 AM
To: Zhang, Chen <[email protected]>
Cc: qemu-devel <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: Re: [PATCH v4 6/6] net/colo-compare.c: Correct ordering in complete and finalize

Quoted 11 lines · Chen
Quoted 11 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Monday, May 4, 2020 6:28 PM
To: qemu-devel <[email protected]>
Cc: Zhang, Chen <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: [PATCH v4 6/6] net/colo-compare.c: Correct ordering in complete and finalize

Quoted 1 line
Quoted 1 line

In colo_compare_complete, insert CompareState into net_compares

only

Quoted 8 lines
Quoted 3 lines

after everything has been initialized.
In colo_compare_finalize, remove CompareState from net_compares before anything is deinitialized.

S/deinitialized/finalized

It looks no dependences on each step on initialization and finalization. Do you means we just need add/remove each colo-compare module at last

Quoted 7 lines

in logic?

Yes. While I didn't see any crashes here, there is the possibility that if colo- compare is removed during checkpoint, the destroyed event_bh is called from colo_notify_compares_event. Same with colo_compare_complete (very unlikely) if colo-compare is created while colo is running, colo_notify_compares_event may call the uninitialized event_bh.

As we discussed on 5/6, remove colo-compare during checkpoint is an illegal operation.

Thanks
Zhang Chen

Quoted 97 lines

Regards,
Lukas Straub

Quoted 93 lines

Or current code have some issue?

Thanks
Zhang Chen

Quoted 88 lines
---
 net/colo-compare.c | 45
+++++++++++++++++++++++----------------------
 1 file changed, 23 insertions(+), 22 deletions(-)
net/colo-compare.c index renamed from net/colo-compare.c +0 -0

c7572d75e9..6f80bcece6 100644

net/colo-compare.c +0 -9
@@ -1283,15 +1283,6 @@ static void
 colo_compare_complete(UserCreatable *uc, Error **errp)
                            s->vnet_hdr);
     }
 
-    qemu_mutex_lock(&colo_compare_mutex);
-    if (!colo_compare_active) {
-        qemu_mutex_init(&event_mtx);
-        qemu_cond_init(&event_complete_cond);
-        colo_compare_active = true;
-    }
-    QTAILQ_INSERT_TAIL(&net_compares, s, next);
-    qemu_mutex_unlock(&colo_compare_mutex);
-
     s->out_sendco.s = s;
     s->out_sendco.chr = &s->chr_out;
     s->out_sendco.notify_remote_frame = false; @@ -1312,6 +1303,16

@@ static void colo_compare_complete(UserCreatable *uc, Error **errp)

connection_destroy);

     colo_compare_iothread(s);
net/colo-compare.c +23 -14
+
+    qemu_mutex_lock(&colo_compare_mutex);
+    if (!colo_compare_active) {
+        qemu_mutex_init(&event_mtx);
+        qemu_cond_init(&event_complete_cond);
+        colo_compare_active = true;
+    }
+    QTAILQ_INSERT_TAIL(&net_compares, s, next);
+    qemu_mutex_unlock(&colo_compare_mutex);
+
     return;
 }
@@ -1384,19 +1385,6 @@ static void colo_compare_finalize(Object *obj)
     CompareState *s = COLO_COMPARE(obj);
     CompareState *tmp = NULL;
 
-    qemu_chr_fe_deinit(&s->chr_pri_in, false);
-    qemu_chr_fe_deinit(&s->chr_sec_in, false);
-    qemu_chr_fe_deinit(&s->chr_out, false);
-    if (s->notify_dev) {
-        qemu_chr_fe_deinit(&s->chr_notify_dev, false);
-    }
-
-    if (s->iothread) {
-        colo_compare_timer_del(s);
-    }
-
-    qemu_bh_delete(s->event_bh);
-
     qemu_mutex_lock(&colo_compare_mutex);
     QTAILQ_FOREACH(tmp, &net_compares, next) {
         if (tmp == s) {
@@ -1411,6 +1399,19 @@ static void colo_compare_finalize(Object *obj)
     }
     qemu_mutex_unlock(&colo_compare_mutex);
 
+    qemu_chr_fe_deinit(&s->chr_pri_in, false);
+    qemu_chr_fe_deinit(&s->chr_sec_in, false);
+    qemu_chr_fe_deinit(&s->chr_out, false);
+    if (s->notify_dev) {
+        qemu_chr_fe_deinit(&s->chr_notify_dev, false);
+    }
+
+    if (s->iothread) {
+        colo_compare_timer_del(s);
+    }
+
+    qemu_bh_delete(s->event_bh);
+
     AioContext *ctx = iothread_get_aio_context(s->iothread);
     aio_context_acquire(ctx);
     AIO_WAIT_WHILE(ctx, !s->out_sendco.done);
--

2.20.1

#18 lore raw

Re: [PATCH v4 5/6] net/colo-compare.c, softmmu/vl.c: Check that colo-compare is active

In reply to Zhang, Chen
Quoted 50 lines · Chen
Quoted 50 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Thursday, May 7, 2020 11:54 PM
To: Zhang, Chen <[email protected]>
Cc: qemu-devel <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: Re: [PATCH v4 5/6] net/colo-compare.c, softmmu/vl.c: Check that colo-compare is active

Quoted 22 lines · Chen
Quoted 22 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Monday, May 4, 2020 6:28 PM
To: qemu-devel <[email protected]>
Cc: Zhang, Chen <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: [PATCH v4 5/6] net/colo-compare.c, softmmu/vl.c: Check that colo- compare is active

If the colo-compare object is removed before failover and a checkpoint happens, qemu crashes because it tries to lock the destroyed event_mtx in colo_notify_compares_event.

Fix this by checking if everything is initialized by introducing a new variable colo_compare_active which is protected by a new mutex colo_compare_mutex. The new mutex also protects against concurrent access of the net_compares list and makes sure that colo_notify_compares_event isn't active while we destroy event_mtx and event_complete_cond.

Quoted 8 lines
Quoted 4 lines

With this it also is again possible to use colo without colo-compare (periodic
mode) and to use multiple colo-compare for multiple network interfaces.

Hi Lukas,

For this case I think we don't need to touch vl.c code, we can solve this

issue from another perspective:

Quoted 3 lines

How to remove colo-compare?
User will use qemu-monitor or QMP command to disable an object, so we just need return operation failed When user try to remove colo-compare

Quoted 8 lines
Quoted 4 lines

object while COLO is running.

Yeah, but that still leaves the other problem that colo can't be used without colo-compare (qemu crashes then).

Yes, the COLO-compare is necessary module in COLO original design.
At most cases, user need it do dynamic sync.
For rare cases, maybe we can add a new colo-compare parameter to bypass all the network workload.

Hi, Chen

In our application, we only need "periodical mode" because of the performance issue, and have internal patch now. Is it OK to send the internal patch for review?

Thanks.

Regards,
Derek

Quoted 157 lines

Thanks
Zhang Chen

Quoted 152 lines

Regards,
Lukas Straub

Quoted 10 lines

Thanks
Zhang Chen

Quoted 7 lines
---
  net/colo-compare.c | 35 +++++++++++++++++++++++++++++------
  net/colo-compare.h |  1 +
  softmmu/vl.c       |  2 ++
  3 files changed, 32 insertions(+), 6 deletions(-)
Quoted 6 lines

net/colo-compare.c static NotifierList colo_compare_notifiers = · line 54

Quoted 6 lines
net/colo-compare.c index renamed from net/colo-compare.c +0 -0

56db3d3bfc..c7572d75e9 100644

net/colo-compare.c +0 -0
@@ -54,6 +54,8 @@ static NotifierList colo_compare_notifiers =
 #define REGULAR_PACKET_CHECK_MS 3000  #define

DEFAULT_TIME_OUT_MS

Quoted 22 lines
Quoted 22 lines

3000

net/colo-compare.c +2 -0
+static QemuMutex colo_compare_mutex; static bool
+colo_compare_active;
  static QemuMutex event_mtx;
  static QemuCond event_complete_cond;  static int

event_unhandled_count; @@ -906,6 +908,12 @@ static void check_old_packet_regular(void *opaque) void

colo_notify_compares_event(void *opaque, int event, Error **errp)  {
      CompareState *s;
net/colo-compare.c +6 -0
+    qemu_mutex_lock(&colo_compare_mutex);
+
+    if (!colo_compare_active) {
+        qemu_mutex_unlock(&colo_compare_mutex);
+        return;
+    }
 
      qemu_mutex_lock(&event_mtx);
      QTAILQ_FOREACH(s, &net_compares, next) { @@ -919,6 +927,7 @@

void colo_notify_compares_event(void *opaque, int event, Error **errp)

      }
Quoted 6 lines
Quoted 6 lines
      qemu_mutex_unlock(&event_mtx);

+ qemu_mutex_unlock(&colo_compare_mutex);
}

static void colo_compare_timer_init(CompareState *s) @@ -1274,7
+1283,14 @@ static void colo_compare_complete(UserCreatable *uc,

Error **errp)

Quoted 88 lines
Quoted 88 lines
                             s->vnet_hdr);
      }
net/colo-compare.c +7 -0
+    qemu_mutex_lock(&colo_compare_mutex);
+    if (!colo_compare_active) {
+        qemu_mutex_init(&event_mtx);
+        qemu_cond_init(&event_complete_cond);
+        colo_compare_active = true;
+    }
      QTAILQ_INSERT_TAIL(&net_compares, s, next);
+    qemu_mutex_unlock(&colo_compare_mutex);
 
      s->out_sendco.s = s;
      s->out_sendco.chr = &s->chr_out; @@ -1290,9 +1306,6 @@ static

void colo_compare_complete(UserCreatable
*uc, Error **errp)

      g_queue_init(&s->conn_list);
net/colo-compare.c +0 -3
-    qemu_mutex_init(&event_mtx);
-    qemu_cond_init(&event_complete_cond);
-
      s->connection_track_table =

g_hash_table_new_full(connection_key_hash,

                                                        connection_key_equal,
                                                        g_free, @@

-1384,12 +1397,19 @@ static void colo_compare_finalize(Object *obj)

      qemu_bh_delete(s->event_bh);
net/colo-compare.c +12 -3
+    qemu_mutex_lock(&colo_compare_mutex);
      QTAILQ_FOREACH(tmp, &net_compares, next) {
          if (tmp == s) {
              QTAILQ_REMOVE(&net_compares, s, next);
              break;
          }
      }
+    if (QTAILQ_EMPTY(&net_compares)) {
+        colo_compare_active = false;
+        qemu_mutex_destroy(&event_mtx);
+        qemu_cond_destroy(&event_complete_cond);
+    }
+    qemu_mutex_unlock(&colo_compare_mutex);
 
      AioContext *ctx = iothread_get_aio_context(s->iothread);
      aio_context_acquire(ctx);
@@ -1413,15 +1433,18 @@ static void colo_compare_finalize(Object *obj)
          object_unref(OBJECT(s->iothread));
      }
 
-    qemu_mutex_destroy(&event_mtx);
-    qemu_cond_destroy(&event_complete_cond);
-
      g_free(s->pri_indev);
      g_free(s->sec_indev);
      g_free(s->outdev);
      g_free(s->notify_dev);
  }
 
+void colo_compare_init_globals(void) {
+    colo_compare_active = false;
+    qemu_mutex_init(&colo_compare_mutex);
+}
+
  static const TypeInfo colo_compare_info = {
      .name = TYPE_COLO_COMPARE,
      .parent = TYPE_OBJECT,
net/colo-compare.h index renamed from net/colo-compare.h +0 -0

22ddd512e2..eb483ac586 100644

net/colo-compare.h +1 -0
@@ -17,6 +17,7 @@
  #ifndef QEMU_COLO_COMPARE_H
  #define QEMU_COLO_COMPARE_H
 
+void colo_compare_init_globals(void);
  void colo_notify_compares_event(void *opaque, int event, Error
 **errp); void colo_compare_register_notifier(Notifier *notify);

void colo_compare_unregister_notifier(Notifier *notify); diff --git a/softmmu/vl.c b/softmmu/vl.c index 32c0047889..a913ed5469 100644

softmmu/vl.c +1 -0
@@ -112,6 +112,7 @@
  #include "qapi/qmp/qerror.h"
  #include "sysemu/iothread.h"
  #include "qemu/guest-random.h"
+#include "net/colo-compare.h"
 
Quoted 3 lines

softmmu/vl.c void qemu_init(int argc, char **argv, char

Quoted 3 lines
softmmu/vl.c +0 -0
  #define MAX_VIRTIO_CONSOLES 1
 
@@ -2906,6 +2907,7 @@ void qemu_init(int argc, char **argv, char

**envp)

Quoted 10 lines
Quoted 9 lines
softmmu/vl.c +1 -1
      precopy_infrastructure_init();
      postcopy_infrastructure_init();
      monitor_init_globals();
+    colo_compare_init_globals();
 
      if (qcrypto_init(&err) < 0) {
          error_reportf_err(err, "cannot initialize crypto: ");
--

2.20.1

#7 lore raw

Re: [PATCH v4 3/6] net/colo-compare.c: Fix deadlock in compare_chr_send

In reply to Zhang, Chen
Quoted 21 lines · Chen
Quoted 11 lines
Quoted 1 line

No need to init the notify_sendco each time, because the notify dev just

an optional parameter.

Quoted 1 line

You can use the if (s->notify_dev) here. Just Xen use the chr_notify_dev.

Ok, I will change that and the code below in the next version.

Quoted 1 line

Overall, make the chr_send job to coroutine is a good idea. It looks good

for me.

Quoted 1 line

And your patch inspired me, it looks we can re-use the compare_chr_send

code on filter mirror/redirector too.

Quoted 9 lines

I already have patch for that, but I don't think it is a good idea, because the guest then can send packets faster than colo-compare can process. This leads bufferbloat and the performance drops in my tests:
Client-to-server tcp:
without patch: ~66 Mbit/s
with patch: ~59 Mbit/s
Server-to-client tcp:
without patch: ~702 Kbit/s
with patch: ~328 Kbit/s

Quoted 4 lines

Oh, a big performance drop, is that caused by memcpy/zero_copy parts ?

Thanks
Zhang Chen

No, there is no memcpy overhead with this patch, see below.

Regards,
Lukas Straub

---
 net/filter-mirror.c | 142 +++++++++++++++++++++++++++++++++-----------
 1 file changed, 106 insertions(+), 36 deletions(-)
net/filter-mirror.c +106 -36
index d83e815545..6bcd317502 100644
@@ -20,6 +20,8 @@
 #include "chardev/char-fe.h"
 #include "qemu/iov.h"
 #include "qemu/sockets.h"
+#include "block/aio-wait.h"
+#include "qemu/coroutine.h"
 
 #define FILTER_MIRROR(obj) \
     OBJECT_CHECK(MirrorState, (obj), TYPE_FILTER_MIRROR)
@@ -31,6 +33,18 @@
 #define TYPE_FILTER_REDIRECTOR "filter-redirector"
 #define REDIRECTOR_MAX_LEN NET_BUFSIZE
 
+typedef struct SendCo {
+    Coroutine *co;
+    GQueue send_list;
+    bool done;
+    int ret;
+} SendCo;
+
+typedef struct SendEntry {
+    ssize_t size;
+    uint8_t buf[];
+} SendEntry;
+
 typedef struct MirrorState {
     NetFilterState parent_obj;
     char *indev;
@@ -38,59 +52,101 @@ typedef struct MirrorState {
     CharBackend chr_in;
     CharBackend chr_out;
     SocketReadState rs;
+    SendCo sendco;
     bool vnet_hdr;
 } MirrorState;
 
-static int filter_send(MirrorState *s,
-                       const struct iovec *iov,
-                       int iovcnt)
+static void coroutine_fn _filter_send(void *opaque)
 {
+    MirrorState *s = opaque;
+    SendCo *sendco = &s->sendco;
     NetFilterState *nf = NETFILTER(s);
     int ret = 0;
-    ssize_t size = 0;
-    uint32_t len = 0;
-    char *buf;
-
-    size = iov_size(iov, iovcnt);
-    if (!size) {
-        return 0;
-    }
 
-    len = htonl(size);
-    ret = qemu_chr_fe_write_all(&s->chr_out, (uint8_t *)&len, sizeof(len));
-    if (ret != sizeof(len)) {
-        goto err;
-    }
+    while (!g_queue_is_empty(&sendco->send_list)) {
+        SendEntry *entry = g_queue_pop_tail(&sendco->send_list);
+        uint32_t len = htonl(entry->size);
 
-    if (s->vnet_hdr) {
-        /*
-         * If vnet_hdr = on, we send vnet header len to make other
-         * module(like colo-compare) know how to parse net
-         * packet correctly.
-         */
-        ssize_t vnet_hdr_len;
+        ret = qemu_chr_fe_write_all(&s->chr_out,
+                                    (uint8_t *)&len,
+                                    sizeof(len));
+        if (ret != sizeof(len)) {
+            g_free(entry);
+            goto err;
+        }
 
-        vnet_hdr_len = nf->netdev->vnet_hdr_len;
+        if (s->vnet_hdr) {
+            /*
+             * If vnet_hdr = on, we send vnet header len to make other
+             * module(like colo-compare) know how to parse net
+             * packet correctly.
+             */
+
+            len = htonl(nf->netdev->vnet_hdr_len);
+            ret = qemu_chr_fe_write_all(&s->chr_out,
+                                        (uint8_t *)&len,
+                                        sizeof(len));
+            if (ret != sizeof(len)) {
+                g_free(entry);
+                goto err;
+            }
+        }
 
-        len = htonl(vnet_hdr_len);
-        ret = qemu_chr_fe_write_all(&s->chr_out, (uint8_t *)&len, sizeof(len));
-        if (ret != sizeof(len)) {
+        ret = qemu_chr_fe_write_all(&s->chr_out,
+                                    (uint8_t *)entry->buf,
+                                    entry->size);
+        if (ret != entry->size) {
+            g_free(entry);
             goto err;
         }
-    }
 
-    buf = g_malloc(size);
-    iov_to_buf(iov, iovcnt, 0, buf, size);
-    ret = qemu_chr_fe_write_all(&s->chr_out, (uint8_t *)buf, size);
-    g_free(buf);
-    if (ret != size) {
-        goto err;
+        g_free(entry);
     }
 
-    return 0;
+    sendco->ret = 0;
+    goto out;
 
 err:
-    return ret < 0 ? ret : -EIO;
+    while (!g_queue_is_empty(&sendco->send_list)) {
+        SendEntry *entry = g_queue_pop_tail(&sendco->send_list);
+        g_free(entry);
+    }
+    sendco->ret = ret < 0 ? ret : -EIO;
+out:
+    sendco->co = NULL;
+    sendco->done = true;
+    aio_wait_kick();
+}
+
+static int filter_send(MirrorState *s,
+                       const struct iovec *iov,
+                       int iovcnt)
+{
+    SendCo *sendco = &s->sendco;
+    SendEntry *entry;
+
+    ssize_t size = iov_size(iov, iovcnt);
+    if (!size) {
+        return 0;
+    }
+
+    entry = g_malloc(sizeof(SendEntry) + size);
+    entry->size = size;
+    iov_to_buf(iov, iovcnt, 0, entry->buf, size);
+    g_queue_push_head(&sendco->send_list, entry);
+
+    if (sendco->done) {
+        sendco->co = qemu_coroutine_create(_filter_send, s);
+        sendco->done = false;
+        qemu_coroutine_enter(sendco->co);
+        if (sendco->done) {
+            /* report early errors */
+            return sendco->ret;
+        }
+    }
+
+    /* assume success */
+    return 0;
 }
 
 static void redirector_to_filter(NetFilterState *nf,
@@ -194,6 +250,10 @@ static void filter_mirror_cleanup(NetFilterState *nf)
 {
     MirrorState *s = FILTER_MIRROR(nf);
 
+    AIO_WAIT_WHILE(NULL, !s->sendco.done);
+
+    g_queue_clear(&s->sendco.send_list);
+
     qemu_chr_fe_deinit(&s->chr_out, false);
 }
 
@@ -201,6 +261,10 @@ static void filter_redirector_cleanup(NetFilterState *nf)
 {
     MirrorState *s = FILTER_REDIRECTOR(nf);
 
+    AIO_WAIT_WHILE(NULL, !s->sendco.done);
+
+    g_queue_clear(&s->sendco.send_list);
+
     qemu_chr_fe_deinit(&s->chr_in, false);
     qemu_chr_fe_deinit(&s->chr_out, false);
 }
@@ -224,6 +288,9 @@ static void filter_mirror_setup(NetFilterState *nf, Error **errp)
     }
 
     qemu_chr_fe_init(&s->chr_out, chr, errp);
+
+    s->sendco.done = true;
+    g_queue_init(&s->sendco.send_list);
 }
 
 static void redirector_rs_finalize(SocketReadState *rs)
@@ -281,6 +348,9 @@ static void filter_redirector_setup(NetFilterState *nf, Error **errp)
             return;
         }
     }
+
+    s->sendco.done = true;
+    g_queue_init(&s->sendco.send_list);
 }
 
 static void filter_mirror_class_init(ObjectClass *oc, void *data)
Signature · 1 line
2.20.1
#19 lore raw

Re: [PATCH v4 5/6] net/colo-compare.c, softmmu/vl.c: Check that colo-compare is active

In reply to Zhang, Chen
Quoted 48 lines · Chen
Quoted 48 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Thursday, May 7, 2020 11:54 PM
To: Zhang, Chen <[email protected]>
Cc: qemu-devel <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: Re: [PATCH v4 5/6] net/colo-compare.c, softmmu/vl.c: Check that colo-compare is active

Quoted 22 lines · Chen
Quoted 22 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Monday, May 4, 2020 6:28 PM
To: qemu-devel <[email protected]>
Cc: Zhang, Chen <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: [PATCH v4 5/6] net/colo-compare.c, softmmu/vl.c: Check that colo- compare is active

If the colo-compare object is removed before failover and a checkpoint happens, qemu crashes because it tries to lock the destroyed event_mtx in colo_notify_compares_event.

Fix this by checking if everything is initialized by introducing a new variable colo_compare_active which is protected by a new mutex colo_compare_mutex. The new mutex also protects against concurrent access of the net_compares list and makes sure that colo_notify_compares_event isn't active while we destroy event_mtx and event_complete_cond.

Quoted 8 lines
Quoted 4 lines

With this it also is again possible to use colo without colo-compare (periodic
mode) and to use multiple colo-compare for multiple network interfaces.

Hi Lukas,

For this case I think we don't need to touch vl.c code, we can solve this

issue from another perspective:

Quoted 3 lines

How to remove colo-compare?
User will use qemu-monitor or QMP command to disable an object, so we just need return operation failed When user try to remove colo-compare

Quoted 8 lines
Quoted 4 lines

object while COLO is running.

Yeah, but that still leaves the other problem that colo can't be used without colo-compare (qemu crashes then).

Yes, the COLO-compare is necessary module in COLO original design.
At most cases, user need it do dynamic sync.
For rare cases, maybe we can add a new colo-compare parameter to bypass all the network workload.

I think such an parameter would only be a workaround instead of a real solution like this patch.

Regards,
Lukas Straub

Quoted 156 lines

Thanks
Zhang Chen

Quoted 152 lines

Regards,
Lukas Straub

Quoted 10 lines

Thanks
Zhang Chen

Quoted 7 lines
---
 net/colo-compare.c | 35 +++++++++++++++++++++++++++++------
 net/colo-compare.h |  1 +
 softmmu/vl.c       |  2 ++
 3 files changed, 32 insertions(+), 6 deletions(-)
Quoted 6 lines

net/colo-compare.c static NotifierList colo_compare_notifiers = · line 54

Quoted 6 lines
net/colo-compare.c index renamed from net/colo-compare.c +0 -0

56db3d3bfc..c7572d75e9 100644

net/colo-compare.c +0 -0
@@ -54,6 +54,8 @@ static NotifierList colo_compare_notifiers =
 #define REGULAR_PACKET_CHECK_MS 3000  #define  

DEFAULT_TIME_OUT_MS

Quoted 22 lines
Quoted 22 lines

3000

net/colo-compare.c +2 -0
+static QemuMutex colo_compare_mutex; static bool
+colo_compare_active;
 static QemuMutex event_mtx;
 static QemuCond event_complete_cond;  static int

event_unhandled_count; @@ -906,6 +908,12 @@ static void check_old_packet_regular(void *opaque) void

colo_notify_compares_event(void *opaque, int event, Error **errp)  {
     CompareState *s;
net/colo-compare.c +6 -0
+    qemu_mutex_lock(&colo_compare_mutex);
+
+    if (!colo_compare_active) {
+        qemu_mutex_unlock(&colo_compare_mutex);
+        return;
+    }
 
     qemu_mutex_lock(&event_mtx);
     QTAILQ_FOREACH(s, &net_compares, next) { @@ -919,6 +927,7 @@

void colo_notify_compares_event(void *opaque, int event, Error **errp)

     }
Quoted 6 lines
Quoted 6 lines
     qemu_mutex_unlock(&event_mtx);

+ qemu_mutex_unlock(&colo_compare_mutex);
}

static void colo_compare_timer_init(CompareState *s) @@ -1274,7
+1283,14 @@ static void colo_compare_complete(UserCreatable *uc,

Error **errp)

Quoted 88 lines
Quoted 88 lines
                            s->vnet_hdr);
     }
net/colo-compare.c +7 -0
+    qemu_mutex_lock(&colo_compare_mutex);
+    if (!colo_compare_active) {
+        qemu_mutex_init(&event_mtx);
+        qemu_cond_init(&event_complete_cond);
+        colo_compare_active = true;
+    }
     QTAILQ_INSERT_TAIL(&net_compares, s, next);
+    qemu_mutex_unlock(&colo_compare_mutex);
 
     s->out_sendco.s = s;
     s->out_sendco.chr = &s->chr_out; @@ -1290,9 +1306,6 @@ static

void colo_compare_complete(UserCreatable
*uc, Error **errp)

     g_queue_init(&s->conn_list);
net/colo-compare.c +0 -3
-    qemu_mutex_init(&event_mtx);
-    qemu_cond_init(&event_complete_cond);
-
     s->connection_track_table =

g_hash_table_new_full(connection_key_hash,

                                                       connection_key_equal,
                                                       g_free, @@

-1384,12 +1397,19 @@ static void colo_compare_finalize(Object *obj)

     qemu_bh_delete(s->event_bh);
net/colo-compare.c +12 -3
+    qemu_mutex_lock(&colo_compare_mutex);
     QTAILQ_FOREACH(tmp, &net_compares, next) {
         if (tmp == s) {
             QTAILQ_REMOVE(&net_compares, s, next);
             break;
         }
     }
+    if (QTAILQ_EMPTY(&net_compares)) {
+        colo_compare_active = false;
+        qemu_mutex_destroy(&event_mtx);
+        qemu_cond_destroy(&event_complete_cond);
+    }
+    qemu_mutex_unlock(&colo_compare_mutex);
 
     AioContext *ctx = iothread_get_aio_context(s->iothread);
     aio_context_acquire(ctx);
@@ -1413,15 +1433,18 @@ static void colo_compare_finalize(Object *obj)
         object_unref(OBJECT(s->iothread));
     }
 
-    qemu_mutex_destroy(&event_mtx);
-    qemu_cond_destroy(&event_complete_cond);
-
     g_free(s->pri_indev);
     g_free(s->sec_indev);
     g_free(s->outdev);
     g_free(s->notify_dev);
 }
 
+void colo_compare_init_globals(void) {
+    colo_compare_active = false;
+    qemu_mutex_init(&colo_compare_mutex);
+}
+
 static const TypeInfo colo_compare_info = {
     .name = TYPE_COLO_COMPARE,
     .parent = TYPE_OBJECT,
net/colo-compare.h index renamed from net/colo-compare.h +0 -0

22ddd512e2..eb483ac586 100644

net/colo-compare.h +1 -0
@@ -17,6 +17,7 @@
 #ifndef QEMU_COLO_COMPARE_H
 #define QEMU_COLO_COMPARE_H
 
+void colo_compare_init_globals(void);
 void colo_notify_compares_event(void *opaque, int event, Error
 **errp); void colo_compare_register_notifier(Notifier *notify);

void colo_compare_unregister_notifier(Notifier *notify); diff --git a/softmmu/vl.c b/softmmu/vl.c index 32c0047889..a913ed5469 100644

softmmu/vl.c +1 -0
@@ -112,6 +112,7 @@
 #include "qapi/qmp/qerror.h"
 #include "sysemu/iothread.h"
 #include "qemu/guest-random.h"
+#include "net/colo-compare.h"
 
Quoted 3 lines

softmmu/vl.c void qemu_init(int argc, char **argv, char

Quoted 3 lines
softmmu/vl.c +0 -0
 #define MAX_VIRTIO_CONSOLES 1
 
@@ -2906,6 +2907,7 @@ void qemu_init(int argc, char **argv, char  

**envp)

Quoted 10 lines
Quoted 9 lines
softmmu/vl.c +1 -1
     precopy_infrastructure_init();
     postcopy_infrastructure_init();
     monitor_init_globals();
+    colo_compare_init_globals();
 
     if (qcrypto_init(&err) < 0) {
         error_reportf_err(err, "cannot initialize crypto: ");
--

2.20.1

#8 lore raw

RE: [PATCH v4 3/6] net/colo-compare.c: Fix deadlock in compare_chr_send

In reply to Lukas Straub
Quoted 40 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Friday, May 8, 2020 2:08 PM
To: Zhang, Chen <[email protected]>
Cc: qemu-devel <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: Re: [PATCH v4 3/6] net/colo-compare.c: Fix deadlock in compare_chr_send

Quoted 4 lines · Chen
Quoted 4 lines
Quoted 2 lines

No need to init the notify_sendco each time, because the notify dev just

an optional parameter.

Quoted 1 line

You can use the if (s->notify_dev) here. Just Xen use the

chr_notify_dev.

Quoted 20 lines
Quoted 19 lines

Ok, I will change that and the code below in the next version.

Quoted 2 lines

Overall, make the chr_send job to coroutine is a good idea. It looks good

for me.

Quoted 2 lines

And your patch inspired me, it looks we can re-use the compare_chr_send

code on filter mirror/redirector too.

I already have patch for that, but I don't think it is a good idea, because the guest then can send packets faster than colo-compare can process. This leads bufferbloat and the performance drops in my tests:
Client-to-server tcp:
without patch: ~66 Mbit/s
with patch: ~59 Mbit/s
Server-to-client tcp:
without patch: ~702 Kbit/s
with patch: ~328 Kbit/s

Quoted 6 lines
Quoted 4 lines

Oh, a big performance drop, is that caused by memcpy/zero_copy parts ?

Thanks
Zhang Chen

No, there is no memcpy overhead with this patch, see below.

I means for the filter mirror/redirector parts why coroutine will lead huge performance drop?

Thanks
Zhang Chen

Quoted 218 lines

Regards,
Lukas Straub

---
 net/filter-mirror.c | 142 +++++++++++++++++++++++++++++++++-----------
 1 file changed, 106 insertions(+), 36 deletions(-)
net/filter-mirror.c index renamed from net/filter-mirror.c +0 -0

d83e815545..6bcd317502 100644

net/filter-mirror.c +93 -36
@@ -20,6 +20,8 @@
 #include "chardev/char-fe.h"
 #include "qemu/iov.h"
 #include "qemu/sockets.h"
+#include "block/aio-wait.h"
+#include "qemu/coroutine.h"
 
 #define FILTER_MIRROR(obj) \
     OBJECT_CHECK(MirrorState, (obj), TYPE_FILTER_MIRROR) @@ -31,6
+33,18 @@  #define TYPE_FILTER_REDIRECTOR "filter-redirector"
 #define REDIRECTOR_MAX_LEN NET_BUFSIZE
 
+typedef struct SendCo {
+    Coroutine *co;
+    GQueue send_list;
+    bool done;
+    int ret;
+} SendCo;
+
+typedef struct SendEntry {
+    ssize_t size;
+    uint8_t buf[];
+} SendEntry;
+
 typedef struct MirrorState {
     NetFilterState parent_obj;
     char *indev;
@@ -38,59 +52,101 @@ typedef struct MirrorState {
     CharBackend chr_in;
     CharBackend chr_out;
     SocketReadState rs;
+    SendCo sendco;
     bool vnet_hdr;
 } MirrorState;
 
-static int filter_send(MirrorState *s,
-                       const struct iovec *iov,
-                       int iovcnt)
+static void coroutine_fn _filter_send(void *opaque)
 {
+    MirrorState *s = opaque;
+    SendCo *sendco = &s->sendco;
     NetFilterState *nf = NETFILTER(s);
     int ret = 0;
-    ssize_t size = 0;
-    uint32_t len = 0;
-    char *buf;
-
-    size = iov_size(iov, iovcnt);
-    if (!size) {
-        return 0;
-    }
 
-    len = htonl(size);
-    ret = qemu_chr_fe_write_all(&s->chr_out, (uint8_t *)&len, sizeof(len));
-    if (ret != sizeof(len)) {
-        goto err;
-    }
+    while (!g_queue_is_empty(&sendco->send_list)) {
+        SendEntry *entry = g_queue_pop_tail(&sendco->send_list);
+        uint32_t len = htonl(entry->size);
 
-    if (s->vnet_hdr) {
-        /*
-         * If vnet_hdr = on, we send vnet header len to make other
-         * module(like colo-compare) know how to parse net
-         * packet correctly.
-         */
-        ssize_t vnet_hdr_len;
+        ret = qemu_chr_fe_write_all(&s->chr_out,
+                                    (uint8_t *)&len,
+                                    sizeof(len));
+        if (ret != sizeof(len)) {
+            g_free(entry);
+            goto err;
+        }
 
-        vnet_hdr_len = nf->netdev->vnet_hdr_len;
+        if (s->vnet_hdr) {
+            /*
+             * If vnet_hdr = on, we send vnet header len to make other
+             * module(like colo-compare) know how to parse net
+             * packet correctly.
+             */
+
+            len = htonl(nf->netdev->vnet_hdr_len);
+            ret = qemu_chr_fe_write_all(&s->chr_out,
+                                        (uint8_t *)&len,
+                                        sizeof(len));
+            if (ret != sizeof(len)) {
+                g_free(entry);
+                goto err;
+            }
+        }
 
-        len = htonl(vnet_hdr_len);
-        ret = qemu_chr_fe_write_all(&s->chr_out, (uint8_t *)&len, sizeof(len));
-        if (ret != sizeof(len)) {
+        ret = qemu_chr_fe_write_all(&s->chr_out,
+                                    (uint8_t *)entry->buf,
+                                    entry->size);
+        if (ret != entry->size) {
+            g_free(entry);
             goto err;
         }
-    }
 
-    buf = g_malloc(size);
-    iov_to_buf(iov, iovcnt, 0, buf, size);
-    ret = qemu_chr_fe_write_all(&s->chr_out, (uint8_t *)buf, size);
-    g_free(buf);
-    if (ret != size) {
-        goto err;
+        g_free(entry);
     }
 
-    return 0;
+    sendco->ret = 0;
+    goto out;
 
 err:
-    return ret < 0 ? ret : -EIO;
+    while (!g_queue_is_empty(&sendco->send_list)) {
+        SendEntry *entry = g_queue_pop_tail(&sendco->send_list);
+        g_free(entry);
+    }
+    sendco->ret = ret < 0 ? ret : -EIO;
+out:
+    sendco->co = NULL;
+    sendco->done = true;
+    aio_wait_kick();
+}
+
+static int filter_send(MirrorState *s,
+                       const struct iovec *iov,
+                       int iovcnt)
+{
+    SendCo *sendco = &s->sendco;
+    SendEntry *entry;
+
+    ssize_t size = iov_size(iov, iovcnt);
+    if (!size) {
+        return 0;
+    }
+
+    entry = g_malloc(sizeof(SendEntry) + size);
+    entry->size = size;
+    iov_to_buf(iov, iovcnt, 0, entry->buf, size);
+    g_queue_push_head(&sendco->send_list, entry);
+
+    if (sendco->done) {
+        sendco->co = qemu_coroutine_create(_filter_send, s);
+        sendco->done = false;
+        qemu_coroutine_enter(sendco->co);
+        if (sendco->done) {
+            /* report early errors */
+            return sendco->ret;
+        }
+    }
+
+    /* assume success */
+    return 0;
 }
 
 static void redirector_to_filter(NetFilterState *nf, @@ -194,6 +250,10 @@
static void filter_mirror_cleanup(NetFilterState *nf)  {
     MirrorState *s = FILTER_MIRROR(nf);
net/filter-mirror.c +8 -0
+    AIO_WAIT_WHILE(NULL, !s->sendco.done);
+
+    g_queue_clear(&s->sendco.send_list);
+
     qemu_chr_fe_deinit(&s->chr_out, false);  }
@@ -201,6 +261,10 @@ static void filter_redirector_cleanup(NetFilterState
 *nf)  {
     MirrorState *s = FILTER_REDIRECTOR(nf);
 
+    AIO_WAIT_WHILE(NULL, !s->sendco.done);
+
+    g_queue_clear(&s->sendco.send_list);
+
     qemu_chr_fe_deinit(&s->chr_in, false);
     qemu_chr_fe_deinit(&s->chr_out, false);  } @@ -224,6 +288,9 @@ static
 void filter_mirror_setup(NetFilterState *nf, Error **errp)
     }

     qemu_chr_fe_init(&s->chr_out, chr, errp);
net/filter-mirror.c +3 -0
+
+    s->sendco.done = true;
+    g_queue_init(&s->sendco.send_list);
 }
 
 static void redirector_rs_finalize(SocketReadState *rs) @@ -281,6 +348,9

@@ static void filter_redirector_setup(NetFilterState *nf, Error **errp)

             return;
         }
     }
net/filter-mirror.c +3 -1
+
+    s->sendco.done = true;
+    g_queue_init(&s->sendco.send_list);
 }
 
 static void filter_mirror_class_init(ObjectClass *oc, void *data)
--

2.20.1

#20 lore raw

RE: [PATCH v4 5/6] net/colo-compare.c, softmmu/vl.c: Check that colo-compare is active

In reply to Lukas Straub
Quoted 73 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Friday, May 8, 2020 2:11 PM
To: Zhang, Chen <[email protected]>
Cc: qemu-devel <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: Re: [PATCH v4 5/6] net/colo-compare.c, softmmu/vl.c: Check that colo-compare is active

Quoted 37 lines · Chen
Quoted 37 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Thursday, May 7, 2020 11:54 PM
To: Zhang, Chen <[email protected]>
Cc: qemu-devel <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: Re: [PATCH v4 5/6] net/colo-compare.c, softmmu/vl.c: Check that colo-compare is active

Quoted 23 lines · Chen
Quoted 23 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Monday, May 4, 2020 6:28 PM
To: qemu-devel <[email protected]>
Cc: Zhang, Chen <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: [PATCH v4 5/6] net/colo-compare.c, softmmu/vl.c: Check that
colo- compare is active

If the colo-compare object is removed before failover and a checkpoint happens, qemu crashes because it tries to lock the destroyed event_mtx in colo_notify_compares_event.

Fix this by checking if everything is initialized by introducing a new variable colo_compare_active which is protected by a new mutex colo_compare_mutex. The new mutex also protects against concurrent access of the net_compares list and makes sure that colo_notify_compares_event isn't active while we destroy event_mtx and event_complete_cond.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

With this it also is again possible to use colo without colo-compare (periodic
mode) and to use multiple colo-compare for multiple network

interfaces.

Quoted 16 lines
Quoted 15 lines
Quoted 6 lines
Quoted 1 line

Hi Lukas,

For this case I think we don't need to touch vl.c code, we can solve this

issue from another perspective:

Quoted 4 lines

How to remove colo-compare?
User will use qemu-monitor or QMP command to disable an object, so we just need return operation failed When user try to remove colo-compare

object while COLO is running.

Yeah, but that still leaves the other problem that colo can't be used without colo-compare (qemu crashes then).

Quoted 7 lines
Quoted 3 lines

Yes, the COLO-compare is necessary module in COLO original design. At most cases, user need it do dynamic sync.
For rare cases, maybe we can add a new colo-compare parameter to

bypass all the network workload.

I think such an parameter would only be a workaround instead of a real solution like this patch.

The root problem is why COLO-compare is necessary.
Yes, maybe someone want to use pure periodic synchronization mode,
But it means it will lost all guest network support(without colo-compare/filter-mirror/filter-redirector/filter-rewriter). The secondary guest just a solid backup for the primary guest, when occur failover the new build stateful connection (like TCP) will crashed, need userspace to handle this status. It lost the original meaning for COLO FT/HA solution, no need use do HA in application layer. it looks like normal/remote periodic VM snapshot here.
Dave or Jason have any comments here?

Thanks
Zhang Chen

Quoted 166 lines

Regards,
Lukas Straub

Quoted 26 lines

Thanks
Zhang Chen

Quoted 23 lines

Regards,
Lukas Straub

Quoted 10 lines

Thanks
Zhang Chen

Quoted 7 lines
---
 net/colo-compare.c | 35 +++++++++++++++++++++++++++++------
 net/colo-compare.h |  1 +
 softmmu/vl.c       |  2 ++
 3 files changed, 32 insertions(+), 6 deletions(-)
Quoted 6 lines

net/colo-compare.c static NotifierList colo_compare_notifiers = · line 54

Quoted 6 lines
net/colo-compare.c index renamed from net/colo-compare.c +0 -0

56db3d3bfc..c7572d75e9 100644

net/colo-compare.c +0 -0
@@ -54,6 +54,8 @@ static NotifierList colo_compare_notifiers =
 #define REGULAR_PACKET_CHECK_MS 3000  #define

DEFAULT_TIME_OUT_MS

Quoted 2 lines
Quoted 2 lines

3000

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
net/colo-compare.c +2 -0
+static QemuMutex colo_compare_mutex; static bool
+colo_compare_active;
 static QemuMutex event_mtx;
 static QemuCond event_complete_cond;  static int

event_unhandled_count; @@ -906,6 +908,12 @@ static void check_old_packet_regular(void *opaque) void colo_notify_compares_event(void *opaque, int event, Error **errp)

{

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
     CompareState *s;
net/colo-compare.c +6 -0
+    qemu_mutex_lock(&colo_compare_mutex);
+
+    if (!colo_compare_active) {
+        qemu_mutex_unlock(&colo_compare_mutex);
+        return;
+    }
 
     qemu_mutex_lock(&event_mtx);
     QTAILQ_FOREACH(s, &net_compares, next) { @@ -919,6 +927,7

@@ void colo_notify_compares_event(void *opaque, int event,

Error **errp)

Quoted 55 lines
Quoted 55 lines
Quoted 8 lines
Quoted 8 lines
     }

     qemu_mutex_unlock(&event_mtx);

+ qemu_mutex_unlock(&colo_compare_mutex);
}

static void colo_compare_timer_init(CompareState *s) @@ -1274,7
+1283,14 @@ static void colo_compare_complete(UserCreatable *uc,

Error **errp)

Quoted 46 lines
Quoted 46 lines
                            s->vnet_hdr);
     }
net/colo-compare.c +7 -0
+    qemu_mutex_lock(&colo_compare_mutex);
+    if (!colo_compare_active) {
+        qemu_mutex_init(&event_mtx);
+        qemu_cond_init(&event_complete_cond);
+        colo_compare_active = true;
+    }
     QTAILQ_INSERT_TAIL(&net_compares, s, next);
+    qemu_mutex_unlock(&colo_compare_mutex);
 
     s->out_sendco.s = s;
     s->out_sendco.chr = &s->chr_out; @@ -1290,9 +1306,6 @@

static void colo_compare_complete(UserCreatable
*uc, Error **errp)

     g_queue_init(&s->conn_list);
net/colo-compare.c +0 -3
-    qemu_mutex_init(&event_mtx);
-    qemu_cond_init(&event_complete_cond);
-
     s->connection_track_table =

g_hash_table_new_full(connection_key_hash,

                                                       connection_key_equal,
                                                       g_free,

@@
-1384,12 +1397,19 @@ static void colo_compare_finalize(Object *obj)

     qemu_bh_delete(s->event_bh);
net/colo-compare.c +7 -0
+    qemu_mutex_lock(&colo_compare_mutex);
     QTAILQ_FOREACH(tmp, &net_compares, next) {
         if (tmp == s) {
             QTAILQ_REMOVE(&net_compares, s, next);
             break;
         }
     }
+    if (QTAILQ_EMPTY(&net_compares)) {
+        colo_compare_active = false;
+        qemu_mutex_destroy(&event_mtx);
+        qemu_cond_destroy(&event_complete_cond);
+    }
+    qemu_mutex_unlock(&colo_compare_mutex);
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

net/colo-compare.c static void colo_compare_finalize(Object

Quoted 3 lines
     AioContext *ctx = iothread_get_aio_context(s->iothread);
     aio_context_acquire(ctx);
net/colo-compare.c +0 -0
@@ -1413,15 +1433,18 @@ static void colo_compare_finalize(Object

*obj)

Quoted 57 lines
Quoted 56 lines
Quoted 42 lines
Quoted 42 lines
net/colo-compare.c +5 -3
         object_unref(OBJECT(s->iothread));
     }
 
-    qemu_mutex_destroy(&event_mtx);
-    qemu_cond_destroy(&event_complete_cond);
-
     g_free(s->pri_indev);
     g_free(s->sec_indev);
     g_free(s->outdev);
     g_free(s->notify_dev);
 }
 
+void colo_compare_init_globals(void) {
+    colo_compare_active = false;
+    qemu_mutex_init(&colo_compare_mutex);
+}
+
 static const TypeInfo colo_compare_info = {
     .name = TYPE_COLO_COMPARE,
     .parent = TYPE_OBJECT,
net/colo-compare.h index renamed from net/colo-compare.h +0 -0

22ddd512e2..eb483ac586 100644

net/colo-compare.h +1 -0
@@ -17,6 +17,7 @@
 #ifndef QEMU_COLO_COMPARE_H
 #define QEMU_COLO_COMPARE_H
 
+void colo_compare_init_globals(void);
 void colo_notify_compares_event(void *opaque, int event, Error
 **errp); void colo_compare_register_notifier(Notifier *notify);

void colo_compare_unregister_notifier(Notifier *notify); diff --git a/softmmu/vl.c b/softmmu/vl.c index 32c0047889..a913ed5469 100644

softmmu/vl.c +1 -0
@@ -112,6 +112,7 @@
 #include "qapi/qmp/qerror.h"
 #include "sysemu/iothread.h"
 #include "qemu/guest-random.h"
+#include "net/colo-compare.h"
 
Quoted 3 lines

softmmu/vl.c void qemu_init(int argc, char **argv, char

Quoted 3 lines
softmmu/vl.c +0 -0
 #define MAX_VIRTIO_CONSOLES 1
 
@@ -2906,6 +2907,7 @@ void qemu_init(int argc, char **argv, char

**envp)

Quoted 10 lines
Quoted 9 lines
softmmu/vl.c +1 -1
     precopy_infrastructure_init();
     postcopy_infrastructure_init();
     monitor_init_globals();
+    colo_compare_init_globals();
 
     if (qcrypto_init(&err) < 0) {
         error_reportf_err(err, "cannot initialize crypto: ");
--

2.20.1

#9 lore raw

Re: [PATCH v4 3/6] net/colo-compare.c: Fix deadlock in compare_chr_send

In reply to Zhang, Chen
Quoted 38 lines · Chen
Quoted 38 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Friday, May 8, 2020 2:08 PM
To: Zhang, Chen <[email protected]>
Cc: qemu-devel <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: Re: [PATCH v4 3/6] net/colo-compare.c: Fix deadlock in compare_chr_send

Quoted 4 lines · Chen
Quoted 4 lines
Quoted 2 lines

No need to init the notify_sendco each time, because the notify dev just

an optional parameter.

Quoted 1 line

You can use the if (s->notify_dev) here. Just Xen use the

chr_notify_dev.

Quoted 20 lines
Quoted 19 lines

Ok, I will change that and the code below in the next version.

Quoted 2 lines

Overall, make the chr_send job to coroutine is a good idea. It looks good

for me.

Quoted 2 lines

And your patch inspired me, it looks we can re-use the compare_chr_send

code on filter mirror/redirector too.

I already have patch for that, but I don't think it is a good idea, because the guest then can send packets faster than colo-compare can process. This leads bufferbloat and the performance drops in my tests:
Client-to-server tcp:
without patch: ~66 Mbit/s
with patch: ~59 Mbit/s
Server-to-client tcp:
without patch: ~702 Kbit/s
with patch: ~328 Kbit/s

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

Oh, a big performance drop, is that caused by memcpy/zero_copy parts ?

Thanks
Zhang Chen

No, there is no memcpy overhead with this patch, see below.

I means for the filter mirror/redirector parts why coroutine will lead huge performance drop?

It's because having a additional buffer before the network bottleneck (colo-compare in our case) confuses TCP's congestion-control:
TCP will speed up the data transfer until packets start to drop (or the network interface is blocked). This feedback has to be quick so TCP can select a suitable transfer speed. But with the patch, the guest will fill the buffer as fast as it can (it does not "see" the slow bandwidth of colo-compare behind the buffer) until it it hits against the TCP congestion window. At this point TCP drastically reduces its transfer speed and it stays low because the full buffer delays the packets so it doesn't receive ACK's so it can't speed up the transfer again. Until the buffer is empty again (can take up to a second in my tests). Then this cycle repeats.

Regards,
Lukas Straub

Quoted 222 lines

Thanks
Zhang Chen

Quoted 218 lines

Regards,
Lukas Straub

---
 net/filter-mirror.c | 142 +++++++++++++++++++++++++++++++++-----------
 1 file changed, 106 insertions(+), 36 deletions(-)
net/filter-mirror.c index renamed from net/filter-mirror.c +0 -0

d83e815545..6bcd317502 100644

net/filter-mirror.c +93 -36
@@ -20,6 +20,8 @@
 #include "chardev/char-fe.h"
 #include "qemu/iov.h"
 #include "qemu/sockets.h"
+#include "block/aio-wait.h"
+#include "qemu/coroutine.h"
 
 #define FILTER_MIRROR(obj) \
     OBJECT_CHECK(MirrorState, (obj), TYPE_FILTER_MIRROR) @@ -31,6
+33,18 @@  #define TYPE_FILTER_REDIRECTOR "filter-redirector"
 #define REDIRECTOR_MAX_LEN NET_BUFSIZE
 
+typedef struct SendCo {
+    Coroutine *co;
+    GQueue send_list;
+    bool done;
+    int ret;
+} SendCo;
+
+typedef struct SendEntry {
+    ssize_t size;
+    uint8_t buf[];
+} SendEntry;
+
 typedef struct MirrorState {
     NetFilterState parent_obj;
     char *indev;
@@ -38,59 +52,101 @@ typedef struct MirrorState {
     CharBackend chr_in;
     CharBackend chr_out;
     SocketReadState rs;
+    SendCo sendco;
     bool vnet_hdr;
 } MirrorState;
 
-static int filter_send(MirrorState *s,
-                       const struct iovec *iov,
-                       int iovcnt)
+static void coroutine_fn _filter_send(void *opaque)
 {
+    MirrorState *s = opaque;
+    SendCo *sendco = &s->sendco;
     NetFilterState *nf = NETFILTER(s);
     int ret = 0;
-    ssize_t size = 0;
-    uint32_t len = 0;
-    char *buf;
-
-    size = iov_size(iov, iovcnt);
-    if (!size) {
-        return 0;
-    }
 
-    len = htonl(size);
-    ret = qemu_chr_fe_write_all(&s->chr_out, (uint8_t *)&len, sizeof(len));
-    if (ret != sizeof(len)) {
-        goto err;
-    }
+    while (!g_queue_is_empty(&sendco->send_list)) {
+        SendEntry *entry = g_queue_pop_tail(&sendco->send_list);
+        uint32_t len = htonl(entry->size);
 
-    if (s->vnet_hdr) {
-        /*
-         * If vnet_hdr = on, we send vnet header len to make other
-         * module(like colo-compare) know how to parse net
-         * packet correctly.
-         */
-        ssize_t vnet_hdr_len;
+        ret = qemu_chr_fe_write_all(&s->chr_out,
+                                    (uint8_t *)&len,
+                                    sizeof(len));
+        if (ret != sizeof(len)) {
+            g_free(entry);
+            goto err;
+        }
 
-        vnet_hdr_len = nf->netdev->vnet_hdr_len;
+        if (s->vnet_hdr) {
+            /*
+             * If vnet_hdr = on, we send vnet header len to make other
+             * module(like colo-compare) know how to parse net
+             * packet correctly.
+             */
+
+            len = htonl(nf->netdev->vnet_hdr_len);
+            ret = qemu_chr_fe_write_all(&s->chr_out,
+                                        (uint8_t *)&len,
+                                        sizeof(len));
+            if (ret != sizeof(len)) {
+                g_free(entry);
+                goto err;
+            }
+        }
 
-        len = htonl(vnet_hdr_len);
-        ret = qemu_chr_fe_write_all(&s->chr_out, (uint8_t *)&len, sizeof(len));
-        if (ret != sizeof(len)) {
+        ret = qemu_chr_fe_write_all(&s->chr_out,
+                                    (uint8_t *)entry->buf,
+                                    entry->size);
+        if (ret != entry->size) {
+            g_free(entry);
             goto err;
         }
-    }
 
-    buf = g_malloc(size);
-    iov_to_buf(iov, iovcnt, 0, buf, size);
-    ret = qemu_chr_fe_write_all(&s->chr_out, (uint8_t *)buf, size);
-    g_free(buf);
-    if (ret != size) {
-        goto err;
+        g_free(entry);
     }
 
-    return 0;
+    sendco->ret = 0;
+    goto out;
 
 err:
-    return ret < 0 ? ret : -EIO;
+    while (!g_queue_is_empty(&sendco->send_list)) {
+        SendEntry *entry = g_queue_pop_tail(&sendco->send_list);
+        g_free(entry);
+    }
+    sendco->ret = ret < 0 ? ret : -EIO;
+out:
+    sendco->co = NULL;
+    sendco->done = true;
+    aio_wait_kick();
+}
+
+static int filter_send(MirrorState *s,
+                       const struct iovec *iov,
+                       int iovcnt)
+{
+    SendCo *sendco = &s->sendco;
+    SendEntry *entry;
+
+    ssize_t size = iov_size(iov, iovcnt);
+    if (!size) {
+        return 0;
+    }
+
+    entry = g_malloc(sizeof(SendEntry) + size);
+    entry->size = size;
+    iov_to_buf(iov, iovcnt, 0, entry->buf, size);
+    g_queue_push_head(&sendco->send_list, entry);
+
+    if (sendco->done) {
+        sendco->co = qemu_coroutine_create(_filter_send, s);
+        sendco->done = false;
+        qemu_coroutine_enter(sendco->co);
+        if (sendco->done) {
+            /* report early errors */
+            return sendco->ret;
+        }
+    }
+
+    /* assume success */
+    return 0;
 }
 
 static void redirector_to_filter(NetFilterState *nf, @@ -194,6 +250,10 @@
static void filter_mirror_cleanup(NetFilterState *nf)  {
     MirrorState *s = FILTER_MIRROR(nf);
net/filter-mirror.c +8 -0
+    AIO_WAIT_WHILE(NULL, !s->sendco.done);
+
+    g_queue_clear(&s->sendco.send_list);
+
     qemu_chr_fe_deinit(&s->chr_out, false);  }
@@ -201,6 +261,10 @@ static void filter_redirector_cleanup(NetFilterState
 *nf)  {
     MirrorState *s = FILTER_REDIRECTOR(nf);
 
+    AIO_WAIT_WHILE(NULL, !s->sendco.done);
+
+    g_queue_clear(&s->sendco.send_list);
+
     qemu_chr_fe_deinit(&s->chr_in, false);
     qemu_chr_fe_deinit(&s->chr_out, false);  } @@ -224,6 +288,9 @@ static
 void filter_mirror_setup(NetFilterState *nf, Error **errp)
     }

     qemu_chr_fe_init(&s->chr_out, chr, errp);
net/filter-mirror.c +3 -0
+
+    s->sendco.done = true;
+    g_queue_init(&s->sendco.send_list);
 }
 
 static void redirector_rs_finalize(SocketReadState *rs) @@ -281,6 +348,9

@@ static void filter_redirector_setup(NetFilterState *nf, Error **errp)

             return;
         }
     }
net/filter-mirror.c +3 -1
+
+    s->sendco.done = true;
+    g_queue_init(&s->sendco.send_list);
 }
 
 static void filter_mirror_class_init(ObjectClass *oc, void *data)
--

2.20.1

#21 lore raw

Re: [PATCH v4 5/6] net/colo-compare.c, softmmu/vl.c: Check that colo-compare is active

In reply to Zhang, Chen
Quoted 79 lines · Chen
Quoted 71 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Friday, May 8, 2020 2:11 PM
To: Zhang, Chen <[email protected]>
Cc: qemu-devel <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: Re: [PATCH v4 5/6] net/colo-compare.c, softmmu/vl.c: Check that colo-compare is active

Quoted 37 lines · Chen
Quoted 37 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Thursday, May 7, 2020 11:54 PM
To: Zhang, Chen <[email protected]>
Cc: qemu-devel <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: Re: [PATCH v4 5/6] net/colo-compare.c, softmmu/vl.c: Check that colo-compare is active

Quoted 23 lines · Chen
Quoted 23 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Monday, May 4, 2020 6:28 PM
To: qemu-devel <[email protected]>
Cc: Zhang, Chen <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: [PATCH v4 5/6] net/colo-compare.c, softmmu/vl.c: Check that
colo- compare is active

If the colo-compare object is removed before failover and a checkpoint happens, qemu crashes because it tries to lock the destroyed event_mtx in colo_notify_compares_event.

Fix this by checking if everything is initialized by introducing a new variable colo_compare_active which is protected by a new mutex colo_compare_mutex. The new mutex also protects against concurrent access of the net_compares list and makes sure that colo_notify_compares_event isn't active while we destroy event_mtx and event_complete_cond.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

With this it also is again possible to use colo without colo-compare (periodic
mode) and to use multiple colo-compare for multiple network

interfaces.

Quoted 16 lines
Quoted 15 lines
Quoted 6 lines
Quoted 1 line

Hi Lukas,

For this case I think we don't need to touch vl.c code, we can solve this

issue from another perspective:

Quoted 4 lines

How to remove colo-compare?
User will use qemu-monitor or QMP command to disable an object, so we just need return operation failed When user try to remove colo-compare

object while COLO is running.

Yeah, but that still leaves the other problem that colo can't be used without colo-compare (qemu crashes then).

Quoted 7 lines
Quoted 3 lines

Yes, the COLO-compare is necessary module in COLO original design. At most cases, user need it do dynamic sync.
For rare cases, maybe we can add a new colo-compare parameter to

bypass all the network workload.

I think such an parameter would only be a workaround instead of a real solution like this patch.

Quoted 6 lines

The root problem is why COLO-compare is necessary.
Yes, maybe someone want to use pure periodic synchronization mode,
But it means it will lost all guest network support(without colo-compare/filter-mirror/filter-redirector/filter-rewriter). The secondary guest just a solid backup for the primary guest, when occur failover the new build stateful connection (like TCP) will crashed, need userspace to handle this status. It lost the original meaning for COLO FT/HA solution, no need use do HA in application layer. it looks like normal/remote periodic VM snapshot here.

Sure, but maybe the user doesn't need (reliable) network on failover. Also proper network support with periodic mode can easily be implemented by modifying filter-buffer to buffer packets until checkpoint. Being able to use colo without colo-compare gives more flexibility to the user.

Regards,
Lukas Straub

Quoted 172 lines

Dave or Jason have any comments here?

Thanks
Zhang Chen

Quoted 166 lines

Regards,
Lukas Straub

Quoted 26 lines

Thanks
Zhang Chen

Quoted 23 lines

Regards,
Lukas Straub

Quoted 10 lines

Thanks
Zhang Chen

Quoted 7 lines
---
 net/colo-compare.c | 35 +++++++++++++++++++++++++++++------
 net/colo-compare.h |  1 +
 softmmu/vl.c       |  2 ++
 3 files changed, 32 insertions(+), 6 deletions(-)
Quoted 6 lines

net/colo-compare.c static NotifierList colo_compare_notifiers = · line 54

Quoted 6 lines
net/colo-compare.c index renamed from net/colo-compare.c +0 -0

56db3d3bfc..c7572d75e9 100644

net/colo-compare.c +0 -0
@@ -54,6 +54,8 @@ static NotifierList colo_compare_notifiers =
 #define REGULAR_PACKET_CHECK_MS 3000  #define  

DEFAULT_TIME_OUT_MS

Quoted 2 lines
Quoted 2 lines

3000

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
net/colo-compare.c +2 -0
+static QemuMutex colo_compare_mutex; static bool
+colo_compare_active;
 static QemuMutex event_mtx;
 static QemuCond event_complete_cond;  static int

event_unhandled_count; @@ -906,6 +908,12 @@ static void check_old_packet_regular(void *opaque) void colo_notify_compares_event(void *opaque, int event, Error **errp)

{

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
     CompareState *s;
net/colo-compare.c +6 -0
+    qemu_mutex_lock(&colo_compare_mutex);
+
+    if (!colo_compare_active) {
+        qemu_mutex_unlock(&colo_compare_mutex);
+        return;
+    }
 
     qemu_mutex_lock(&event_mtx);
     QTAILQ_FOREACH(s, &net_compares, next) { @@ -919,6 +927,7

@@ void colo_notify_compares_event(void *opaque, int event,

Error **errp)

Quoted 55 lines
Quoted 55 lines
Quoted 8 lines
Quoted 8 lines
     }

     qemu_mutex_unlock(&event_mtx);

+ qemu_mutex_unlock(&colo_compare_mutex);
}

static void colo_compare_timer_init(CompareState *s) @@ -1274,7
+1283,14 @@ static void colo_compare_complete(UserCreatable *uc,

Error **errp)

Quoted 46 lines
Quoted 46 lines
                            s->vnet_hdr);
     }
net/colo-compare.c +7 -0
+    qemu_mutex_lock(&colo_compare_mutex);
+    if (!colo_compare_active) {
+        qemu_mutex_init(&event_mtx);
+        qemu_cond_init(&event_complete_cond);
+        colo_compare_active = true;
+    }
     QTAILQ_INSERT_TAIL(&net_compares, s, next);
+    qemu_mutex_unlock(&colo_compare_mutex);
 
     s->out_sendco.s = s;
     s->out_sendco.chr = &s->chr_out; @@ -1290,9 +1306,6 @@

static void colo_compare_complete(UserCreatable
*uc, Error **errp)

     g_queue_init(&s->conn_list);
net/colo-compare.c +0 -3
-    qemu_mutex_init(&event_mtx);
-    qemu_cond_init(&event_complete_cond);
-
     s->connection_track_table =

g_hash_table_new_full(connection_key_hash,

                                                       connection_key_equal,
                                                       g_free,

@@
-1384,12 +1397,19 @@ static void colo_compare_finalize(Object *obj)

     qemu_bh_delete(s->event_bh);
net/colo-compare.c +7 -0
+    qemu_mutex_lock(&colo_compare_mutex);
     QTAILQ_FOREACH(tmp, &net_compares, next) {
         if (tmp == s) {
             QTAILQ_REMOVE(&net_compares, s, next);
             break;
         }
     }
+    if (QTAILQ_EMPTY(&net_compares)) {
+        colo_compare_active = false;
+        qemu_mutex_destroy(&event_mtx);
+        qemu_cond_destroy(&event_complete_cond);
+    }
+    qemu_mutex_unlock(&colo_compare_mutex);
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

net/colo-compare.c static void colo_compare_finalize(Object

Quoted 3 lines
     AioContext *ctx = iothread_get_aio_context(s->iothread);
     aio_context_acquire(ctx);
net/colo-compare.c +0 -0
@@ -1413,15 +1433,18 @@ static void colo_compare_finalize(Object  

*obj)

Quoted 57 lines
Quoted 56 lines
Quoted 42 lines
Quoted 42 lines
net/colo-compare.c +5 -3
         object_unref(OBJECT(s->iothread));
     }
 
-    qemu_mutex_destroy(&event_mtx);
-    qemu_cond_destroy(&event_complete_cond);
-
     g_free(s->pri_indev);
     g_free(s->sec_indev);
     g_free(s->outdev);
     g_free(s->notify_dev);
 }
 
+void colo_compare_init_globals(void) {
+    colo_compare_active = false;
+    qemu_mutex_init(&colo_compare_mutex);
+}
+
 static const TypeInfo colo_compare_info = {
     .name = TYPE_COLO_COMPARE,
     .parent = TYPE_OBJECT,
net/colo-compare.h index renamed from net/colo-compare.h +0 -0

22ddd512e2..eb483ac586 100644

net/colo-compare.h +1 -0
@@ -17,6 +17,7 @@
 #ifndef QEMU_COLO_COMPARE_H
 #define QEMU_COLO_COMPARE_H
 
+void colo_compare_init_globals(void);
 void colo_notify_compares_event(void *opaque, int event, Error
 **errp); void colo_compare_register_notifier(Notifier *notify);

void colo_compare_unregister_notifier(Notifier *notify); diff --git a/softmmu/vl.c b/softmmu/vl.c index 32c0047889..a913ed5469 100644

softmmu/vl.c +1 -0
@@ -112,6 +112,7 @@
 #include "qapi/qmp/qerror.h"
 #include "sysemu/iothread.h"
 #include "qemu/guest-random.h"
+#include "net/colo-compare.h"
 
Quoted 3 lines

softmmu/vl.c void qemu_init(int argc, char **argv, char

Quoted 3 lines
softmmu/vl.c +0 -0
 #define MAX_VIRTIO_CONSOLES 1
 
@@ -2906,6 +2907,7 @@ void qemu_init(int argc, char **argv, char  

**envp)

Quoted 10 lines
Quoted 9 lines
softmmu/vl.c +1 -1
     precopy_infrastructure_init();
     postcopy_infrastructure_init();
     monitor_init_globals();
+    colo_compare_init_globals();
 
     if (qcrypto_init(&err) < 0) {
         error_reportf_err(err, "cannot initialize crypto: ");
--

2.20.1

Reviewed-by
#10 lore raw

RE: [PATCH v4 3/6] net/colo-compare.c: Fix deadlock in compare_chr_send

In reply to Lukas Straub
Quoted 66 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Friday, May 8, 2020 3:56 PM
To: Zhang, Chen <[email protected]>
Cc: qemu-devel <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: Re: [PATCH v4 3/6] net/colo-compare.c: Fix deadlock in compare_chr_send

Quoted 23 lines · Chen
Quoted 23 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Friday, May 8, 2020 2:08 PM
To: Zhang, Chen <[email protected]>
Cc: qemu-devel <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: Re: [PATCH v4 3/6] net/colo-compare.c: Fix deadlock in compare_chr_send

Quoted 4 lines · Chen
Quoted 4 lines
Quoted 2 lines

No need to init the notify_sendco each time, because the notify dev just

an optional parameter.

Quoted 1 line

You can use the if (s->notify_dev) here. Just Xen use the

chr_notify_dev.

Quoted 5 lines
Quoted 5 lines

Ok, I will change that and the code below in the next version.

Quoted 2 lines

Overall, make the chr_send job to coroutine is a good idea. It looks good

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

for me.

Quoted 2 lines

And your patch inspired me, it looks we can re-use the compare_chr_send

code on filter mirror/redirector too.

I already have patch for that, but I don't think it is a good idea, because the guest then can send packets faster than colo-compare can process. This leads bufferbloat and the

performance drops in my tests:

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines

Client-to-server tcp:
without patch: ~66 Mbit/s
with patch: ~59 Mbit/s
Server-to-client tcp:
without patch: ~702 Kbit/s
with patch: ~328 Kbit/s

Oh, a big performance drop, is that caused by memcpy/zero_copy

parts ?

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

Thanks
Zhang Chen

No, there is no memcpy overhead with this patch, see below.

I means for the filter mirror/redirector parts why coroutine will lead huge

performance drop?

Quoted 11 lines

It's because having a additional buffer before the network bottleneck (colo- compare in our case) confuses TCP's congestion-control:
TCP will speed up the data transfer until packets start to drop (or the network interface is blocked). This feedback has to be quick so TCP can select a suitable transfer speed. But with the patch, the guest will fill the buffer as fast as it can (it does not "see" the slow bandwidth of colo-compare behind the buffer) until it it hits against the TCP congestion window. At this point TCP drastically reduces its transfer speed and it stays low because the full buffer delays the packets so it doesn't receive ACK's so it can't speed up the transfer again. Until the buffer is empty again (can take up to a second in my tests). Then this cycle repeats.

Make sense!
After fix above issue:

Thanks
Zhang Chen

Quoted 231 lines

Regards,
Lukas Straub

Quoted 69 lines

Thanks
Zhang Chen

Quoted 66 lines

Regards,
Lukas Straub

---
 net/filter-mirror.c | 142
+++++++++++++++++++++++++++++++++-----------
 1 file changed, 106 insertions(+), 36 deletions(-)
net/filter-mirror.c index renamed from net/filter-mirror.c +0 -0

d83e815545..6bcd317502 100644

net/filter-mirror.c +19 -11
@@ -20,6 +20,8 @@
 #include "chardev/char-fe.h"
 #include "qemu/iov.h"
 #include "qemu/sockets.h"
+#include "block/aio-wait.h"
+#include "qemu/coroutine.h"
 
 #define FILTER_MIRROR(obj) \
     OBJECT_CHECK(MirrorState, (obj), TYPE_FILTER_MIRROR) @@ -31,6
+33,18 @@  #define TYPE_FILTER_REDIRECTOR "filter-redirector"
 #define REDIRECTOR_MAX_LEN NET_BUFSIZE
 
+typedef struct SendCo {
+    Coroutine *co;
+    GQueue send_list;
+    bool done;
+    int ret;
+} SendCo;
+
+typedef struct SendEntry {
+    ssize_t size;
+    uint8_t buf[];
+} SendEntry;
+
 typedef struct MirrorState {
     NetFilterState parent_obj;
     char *indev;
@@ -38,59 +52,101 @@ typedef struct MirrorState {
     CharBackend chr_in;
     CharBackend chr_out;
     SocketReadState rs;
+    SendCo sendco;
     bool vnet_hdr;
 } MirrorState;
 
-static int filter_send(MirrorState *s,
-                       const struct iovec *iov,
-                       int iovcnt)
+static void coroutine_fn _filter_send(void *opaque)
 {
+    MirrorState *s = opaque;
+    SendCo *sendco = &s->sendco;
     NetFilterState *nf = NETFILTER(s);
     int ret = 0;
-    ssize_t size = 0;
-    uint32_t len = 0;
-    char *buf;
-
-    size = iov_size(iov, iovcnt);
-    if (!size) {
-        return 0;
-    }
 
Quoted 2 lines

net/filter-mirror.c typedef struct MirrorState { · line 60

Quoted 2 lines
net/filter-mirror.c +0 -2
-    len = htonl(size);
-    ret = qemu_chr_fe_write_all(&s->chr_out, (uint8_t *)&len,

sizeof(len));

Quoted 40 lines
Quoted 40 lines
net/filter-mirror.c +26 -11
-    if (ret != sizeof(len)) {
-        goto err;
-    }
+    while (!g_queue_is_empty(&sendco->send_list)) {
+        SendEntry *entry = g_queue_pop_tail(&sendco->send_list);
+        uint32_t len = htonl(entry->size);
 
-    if (s->vnet_hdr) {
-        /*
-         * If vnet_hdr = on, we send vnet header len to make other
-         * module(like colo-compare) know how to parse net
-         * packet correctly.
-         */
-        ssize_t vnet_hdr_len;
+        ret = qemu_chr_fe_write_all(&s->chr_out,
+                                    (uint8_t *)&len,
+                                    sizeof(len));
+        if (ret != sizeof(len)) {
+            g_free(entry);
+            goto err;
+        }
 
-        vnet_hdr_len = nf->netdev->vnet_hdr_len;
+        if (s->vnet_hdr) {
+            /*
+             * If vnet_hdr = on, we send vnet header len to make other
+             * module(like colo-compare) know how to parse net
+             * packet correctly.
+             */
+
+            len = htonl(nf->netdev->vnet_hdr_len);
+            ret = qemu_chr_fe_write_all(&s->chr_out,
+                                        (uint8_t *)&len,
+                                        sizeof(len));
+            if (ret != sizeof(len)) {
+                g_free(entry);
+                goto err;
+            }
+        }
 
Quoted 2 lines

net/filter-mirror.c typedef struct MirrorState { · line 76

Quoted 2 lines
net/filter-mirror.c +0 -2
-        len = htonl(vnet_hdr_len);
-        ret = qemu_chr_fe_write_all(&s->chr_out, (uint8_t *)&len,

sizeof(len));

Quoted 91 lines
Quoted 91 lines
net/filter-mirror.c +53 -10
-        if (ret != sizeof(len)) {
+        ret = qemu_chr_fe_write_all(&s->chr_out,
+                                    (uint8_t *)entry->buf,
+                                    entry->size);
+        if (ret != entry->size) {
+            g_free(entry);
             goto err;
         }
-    }
 
-    buf = g_malloc(size);
-    iov_to_buf(iov, iovcnt, 0, buf, size);
-    ret = qemu_chr_fe_write_all(&s->chr_out, (uint8_t *)buf, size);
-    g_free(buf);
-    if (ret != size) {
-        goto err;
+        g_free(entry);
     }
 
-    return 0;
+    sendco->ret = 0;
+    goto out;
 
 err:
-    return ret < 0 ? ret : -EIO;
+    while (!g_queue_is_empty(&sendco->send_list)) {
+        SendEntry *entry = g_queue_pop_tail(&sendco->send_list);
+        g_free(entry);
+    }
+    sendco->ret = ret < 0 ? ret : -EIO;
+out:
+    sendco->co = NULL;
+    sendco->done = true;
+    aio_wait_kick();
+}
+
+static int filter_send(MirrorState *s,
+                       const struct iovec *iov,
+                       int iovcnt)
+{
+    SendCo *sendco = &s->sendco;
+    SendEntry *entry;
+
+    ssize_t size = iov_size(iov, iovcnt);
+    if (!size) {
+        return 0;
+    }
+
+    entry = g_malloc(sizeof(SendEntry) + size);
+    entry->size = size;
+    iov_to_buf(iov, iovcnt, 0, entry->buf, size);
+    g_queue_push_head(&sendco->send_list, entry);
+
+    if (sendco->done) {
+        sendco->co = qemu_coroutine_create(_filter_send, s);
+        sendco->done = false;
+        qemu_coroutine_enter(sendco->co);
+        if (sendco->done) {
+            /* report early errors */
+            return sendco->ret;
+        }
+    }
+
+    /* assume success */
+    return 0;
 }
 
 static void redirector_to_filter(NetFilterState *nf, @@ -194,6
+250,10 @@ static void filter_mirror_cleanup(NetFilterState *nf)  {
     MirrorState *s = FILTER_MIRROR(nf);
 
+    AIO_WAIT_WHILE(NULL, !s->sendco.done);
+
+    g_queue_clear(&s->sendco.send_list);
+
     qemu_chr_fe_deinit(&s->chr_out, false);  }
@@ -201,6 +261,10 @@ static void

filter_redirector_cleanup(NetFilterState

*nf)  {
     MirrorState *s = FILTER_REDIRECTOR(nf);
net/filter-mirror.c +4 -0
+    AIO_WAIT_WHILE(NULL, !s->sendco.done);
+
+    g_queue_clear(&s->sendco.send_list);
+
     qemu_chr_fe_deinit(&s->chr_in, false);
     qemu_chr_fe_deinit(&s->chr_out, false);  } @@ -224,6 +288,9 @@

static void filter_mirror_setup(NetFilterState *nf, Error **errp)

     }
Quoted 8 lines

net/filter-mirror.c static void filter_mirror_setup(NetFilterState *nf, Error **errp)

Quoted 8 lines
     qemu_chr_fe_init(&s->chr_out, chr, errp);
net/filter-mirror.c +4 -0
+
+    s->sendco.done = true;
+    g_queue_init(&s->sendco.send_list);
 }
 
 static void redirector_rs_finalize(SocketReadState *rs) @@ -281,6
+348,9 @@ static void filter_redirector_setup(NetFilterState *nf, Error

**errp)

Quoted 12 lines
Quoted 11 lines
             return;
         }
     }
net/filter-mirror.c +3 -1
+
+    s->sendco.done = true;
+    g_queue_init(&s->sendco.send_list);
 }
 
 static void filter_mirror_class_init(ObjectClass *oc, void *data)
--

2.20.1

Reviewed-by
#28 lore raw

RE: [PATCH v4 6/6] net/colo-compare.c: Correct ordering in complete and finalize

In reply to Lukas Straub
Quoted 37 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Friday, May 8, 2020 12:09 AM
To: Zhang, Chen <[email protected]>
Cc: qemu-devel <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: Re: [PATCH v4 6/6] net/colo-compare.c: Correct ordering in complete and finalize

Quoted 11 lines · Chen
Quoted 11 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Monday, May 4, 2020 6:28 PM
To: qemu-devel <[email protected]>
Cc: Zhang, Chen <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: [PATCH v4 6/6] net/colo-compare.c: Correct ordering in complete and finalize

Quoted 1 line
Quoted 1 line

In colo_compare_complete, insert CompareState into net_compares

only

Quoted 8 lines
Quoted 3 lines

after everything has been initialized.
In colo_compare_finalize, remove CompareState from net_compares before anything is deinitialized.

S/deinitialized/finalized

It looks no dependences on each step on initialization and finalization. Do you means we just need add/remove each colo-compare module at last

Quoted 8 lines

in logic?

Yes. While I didn't see any crashes here, there is the possibility that if colo- compare is removed during checkpoint, the destroyed event_bh is called from colo_notify_compares_event. Same with colo_compare_complete (very unlikely) if colo-compare is created while colo is running, colo_notify_compares_event may call the uninitialized event_bh.

Thanks
Zhang Chen

Quoted 96 lines

Regards,
Lukas Straub

Quoted 93 lines

Or current code have some issue?

Thanks
Zhang Chen

Quoted 88 lines
---
 net/colo-compare.c | 45
+++++++++++++++++++++++----------------------
 1 file changed, 23 insertions(+), 22 deletions(-)
net/colo-compare.c index renamed from net/colo-compare.c +0 -0

c7572d75e9..6f80bcece6 100644

net/colo-compare.c +0 -9
@@ -1283,15 +1283,6 @@ static void
 colo_compare_complete(UserCreatable *uc, Error **errp)
                            s->vnet_hdr);
     }
 
-    qemu_mutex_lock(&colo_compare_mutex);
-    if (!colo_compare_active) {
-        qemu_mutex_init(&event_mtx);
-        qemu_cond_init(&event_complete_cond);
-        colo_compare_active = true;
-    }
-    QTAILQ_INSERT_TAIL(&net_compares, s, next);
-    qemu_mutex_unlock(&colo_compare_mutex);
-
     s->out_sendco.s = s;
     s->out_sendco.chr = &s->chr_out;
     s->out_sendco.notify_remote_frame = false; @@ -1312,6 +1303,16

@@ static void colo_compare_complete(UserCreatable *uc, Error **errp)

connection_destroy);

     colo_compare_iothread(s);
net/colo-compare.c +23 -14
+
+    qemu_mutex_lock(&colo_compare_mutex);
+    if (!colo_compare_active) {
+        qemu_mutex_init(&event_mtx);
+        qemu_cond_init(&event_complete_cond);
+        colo_compare_active = true;
+    }
+    QTAILQ_INSERT_TAIL(&net_compares, s, next);
+    qemu_mutex_unlock(&colo_compare_mutex);
+
     return;
 }
@@ -1384,19 +1385,6 @@ static void colo_compare_finalize(Object *obj)
     CompareState *s = COLO_COMPARE(obj);
     CompareState *tmp = NULL;
 
-    qemu_chr_fe_deinit(&s->chr_pri_in, false);
-    qemu_chr_fe_deinit(&s->chr_sec_in, false);
-    qemu_chr_fe_deinit(&s->chr_out, false);
-    if (s->notify_dev) {
-        qemu_chr_fe_deinit(&s->chr_notify_dev, false);
-    }
-
-    if (s->iothread) {
-        colo_compare_timer_del(s);
-    }
-
-    qemu_bh_delete(s->event_bh);
-
     qemu_mutex_lock(&colo_compare_mutex);
     QTAILQ_FOREACH(tmp, &net_compares, next) {
         if (tmp == s) {
@@ -1411,6 +1399,19 @@ static void colo_compare_finalize(Object *obj)
     }
     qemu_mutex_unlock(&colo_compare_mutex);
 
+    qemu_chr_fe_deinit(&s->chr_pri_in, false);
+    qemu_chr_fe_deinit(&s->chr_sec_in, false);
+    qemu_chr_fe_deinit(&s->chr_out, false);
+    if (s->notify_dev) {
+        qemu_chr_fe_deinit(&s->chr_notify_dev, false);
+    }
+
+    if (s->iothread) {
+        colo_compare_timer_del(s);
+    }
+
+    qemu_bh_delete(s->event_bh);
+
     AioContext *ctx = iothread_get_aio_context(s->iothread);
     aio_context_acquire(ctx);
     AIO_WAIT_WHILE(ctx, !s->out_sendco.done);
--

2.20.1

#22 lore raw

RE: [PATCH v4 5/6] net/colo-compare.c, softmmu/vl.c: Check that colo-compare is active

In reply to Lukas Straub
Quoted 102 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Saturday, May 9, 2020 8:21 PM
To: Zhang, Chen <[email protected]>
Cc: qemu-devel <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>; Dr . David Alan Gilbert <[email protected]>
Subject: Re: [PATCH v4 5/6] net/colo-compare.c, softmmu/vl.c: Check that colo-compare is active

Quoted 75 lines · Chen
Quoted 75 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Friday, May 8, 2020 2:11 PM
To: Zhang, Chen <[email protected]>
Cc: qemu-devel <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: Re: [PATCH v4 5/6] net/colo-compare.c, softmmu/vl.c: Check that colo-compare is active

Quoted 38 lines · Chen
Quoted 38 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Thursday, May 7, 2020 11:54 PM
To: Zhang, Chen <[email protected]>
Cc: qemu-devel <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: Re: [PATCH v4 5/6] net/colo-compare.c, softmmu/vl.c:
Check that colo-compare is active

Quoted 24 lines · Chen
Quoted 24 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Monday, May 4, 2020 6:28 PM
To: qemu-devel <[email protected]>
Cc: Zhang, Chen <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>;
Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: [PATCH v4 5/6] net/colo-compare.c, softmmu/vl.c:
Check that
colo- compare is active

If the colo-compare object is removed before failover and a checkpoint happens, qemu crashes because it tries to lock the destroyed event_mtx in colo_notify_compares_event.

Fix this by checking if everything is initialized by introducing a new variable colo_compare_active which is protected by a new mutex colo_compare_mutex. The new mutex also protects against concurrent access of the net_compares list and makes sure that colo_notify_compares_event isn't active while we destroy event_mtx and event_complete_cond.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

With this it also is again possible to use colo without colo-compare (periodic
mode) and to use multiple colo-compare for multiple network

interfaces.

Quoted 19 lines
Quoted 7 lines
Quoted 6 lines
Quoted 1 line

Hi Lukas,

For this case I think we don't need to touch vl.c code, we can solve this

issue from another perspective:

Quoted 8 lines
Quoted 4 lines

How to remove colo-compare?
User will use qemu-monitor or QMP command to disable an object, so we just need return operation failed When user try to remove colo-compare

object while COLO is running.

Yeah, but that still leaves the other problem that colo can't be used without colo-compare (qemu crashes then).

Yes, the COLO-compare is necessary module in COLO original design. At most cases, user need it do dynamic sync.
For rare cases, maybe we can add a new colo-compare parameter to

Quoted 8 lines
Quoted 4 lines

bypass all the network workload.

I think such an parameter would only be a workaround instead of a real solution like this patch.

The root problem is why COLO-compare is necessary.
Yes, maybe someone want to use pure periodic synchronization mode, But it means it will lost all guest network support(without colo-compare/filter-

mirror/filter-redirector/filter-rewriter).

Quoted 2 lines

The secondary guest just a solid backup for the primary guest, when occur failover the new build stateful connection (like TCP) will crashed,

Quoted 8 lines

need userspace to handle this status. It lost the original meaning for COLO FT/HA solution, no need use do HA in application layer.

Quoted 1 line

it looks like normal/remote periodic VM snapshot here.

Sure, but maybe the user doesn't need (reliable) network on failover. Also proper network support with periodic mode can easily be implemented by modifying filter-buffer to buffer packets until checkpoint. Being able to use colo without colo-compare gives more flexibility to the user.

OK, use the filter-buffer instead of colo-compare is a new use case here.
If other maintainers have no comments about touch the vl.c for COLO , I think it's OK here.

Thanks
Zhang Chen

Quoted 184 lines

Regards,
Lukas Straub

Quoted 22 lines

Dave or Jason have any comments here?

Thanks
Zhang Chen

Quoted 17 lines

Regards,
Lukas Straub

Quoted 13 lines

Thanks
Zhang Chen

Quoted 10 lines

Regards,
Lukas Straub

Quoted 6 lines

Thanks
Zhang Chen

Quoted 3 lines
---
 net/colo-compare.c | 35 +++++++++++++++++++++++++++++---

---

Quoted 108 lines
Quoted 108 lines
Quoted 13 lines
Quoted 13 lines
Quoted 10 lines
Quoted 10 lines
 net/colo-compare.h |  1 +
 softmmu/vl.c       |  2 ++
 3 files changed, 32 insertions(+), 6 deletions(-)
net/colo-compare.c index renamed from net/colo-compare.c +0 -0

56db3d3bfc..c7572d75e9 100644

net/colo-compare.c +0 -0
@@ -54,6 +54,8 @@ static NotifierList colo_compare_notifiers

= #define REGULAR_PACKET_CHECK_MS 3000 #define

DEFAULT_TIME_OUT_MS

Quoted 2 lines
Quoted 2 lines

3000

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
net/colo-compare.c +2 -0
+static QemuMutex colo_compare_mutex; static bool
+colo_compare_active;
 static QemuMutex event_mtx;  static QemuCond

event_complete_cond; static int event_unhandled_count; @@ -906,6 +908,12 @@ static void check_old_packet_regular(void *opaque) void colo_notify_compares_event(void *opaque, int event, Error **errp)

{

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
     CompareState *s;
net/colo-compare.c +6 -0
+    qemu_mutex_lock(&colo_compare_mutex);
+
+    if (!colo_compare_active) {
+        qemu_mutex_unlock(&colo_compare_mutex);
+        return;
+    }
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
     qemu_mutex_lock(&event_mtx);
     QTAILQ_FOREACH(s, &net_compares, next) { @@ -919,6

+927,7 @@ void colo_notify_compares_event(void *opaque, int event,

Error **errp)

Quoted 58 lines
Quoted 58 lines
Quoted 10 lines

net/colo-compare.c

Quoted 10 lines
     }

     qemu_mutex_unlock(&event_mtx);
net/colo-compare.c +3 -1
+    qemu_mutex_unlock(&colo_compare_mutex);
 }
 
 static void colo_compare_timer_init(CompareState *s) @@
-1274,7
+1283,14 @@ static void colo_compare_complete(UserCreatable
+*uc,

Error **errp)

Quoted 47 lines
Quoted 47 lines
                            s->vnet_hdr);
     }
net/colo-compare.c +7 -0
+    qemu_mutex_lock(&colo_compare_mutex);
+    if (!colo_compare_active) {
+        qemu_mutex_init(&event_mtx);
+        qemu_cond_init(&event_complete_cond);
+        colo_compare_active = true;
+    }
     QTAILQ_INSERT_TAIL(&net_compares, s, next);
+    qemu_mutex_unlock(&colo_compare_mutex);
 
     s->out_sendco.s = s;
     s->out_sendco.chr = &s->chr_out; @@ -1290,9 +1306,6 @@

static void colo_compare_complete(UserCreatable *uc, Error **errp)

     g_queue_init(&s->conn_list);
net/colo-compare.c +0 -3
-    qemu_mutex_init(&event_mtx);
-    qemu_cond_init(&event_complete_cond);
-
     s->connection_track_table =

g_hash_table_new_full(connection_key_hash,
connection_key_equal,

g_free, @@
-1384,12 +1397,19 @@ static void
colo_compare_finalize(Object
*obj)

     qemu_bh_delete(s->event_bh);
net/colo-compare.c +7 -0
+    qemu_mutex_lock(&colo_compare_mutex);
     QTAILQ_FOREACH(tmp, &net_compares, next) {
         if (tmp == s) {
             QTAILQ_REMOVE(&net_compares, s, next);
             break;
         }
     }
+    if (QTAILQ_EMPTY(&net_compares)) {
+        colo_compare_active = false;
+        qemu_mutex_destroy(&event_mtx);
+        qemu_cond_destroy(&event_complete_cond);
+    }
+    qemu_mutex_unlock(&colo_compare_mutex);
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
     AioContext *ctx = iothread_get_aio_context(s->iothread);
     aio_context_acquire(ctx); @@ -1413,15 +1433,18 @@

static void colo_compare_finalize(Object

*obj)

Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
         object_unref(OBJECT(s->iothread));
     }
net/colo-compare.c +0 -3
-    qemu_mutex_destroy(&event_mtx);
-    qemu_cond_destroy(&event_complete_cond);
-
     g_free(s->pri_indev);
     g_free(s->sec_indev);
     g_free(s->outdev);
     g_free(s->notify_dev);
 }
Quoted 14 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
net/colo-compare.c +5 -0
+void colo_compare_init_globals(void) {
+    colo_compare_active = false;
+    qemu_mutex_init(&colo_compare_mutex);
+}
+
 static const TypeInfo colo_compare_info = {
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

net/colo-compare.h line 17

Quoted 8 lines

.name = TYPE_COLO_COMPARE,
.parent = TYPE_OBJECT,

net/colo-compare.h index renamed from net/colo-compare.h +0 -0

22ddd512e2..eb483ac586 100644

net/colo-compare.h +0 -0
@@ -17,6 +17,7 @@
 #ifndef QEMU_COLO_COMPARE_H  #define

QEMU_COLO_COMPARE_H

Quoted 34 lines
Quoted 33 lines
Quoted 33 lines
Quoted 32 lines
Quoted 16 lines
Quoted 16 lines
net/colo-compare.h +1 -0
+void colo_compare_init_globals(void);
 void colo_notify_compares_event(void *opaque, int event,

Error **errp); void colo_compare_register_notifier(Notifier *notify); void colo_compare_unregister_notifier(Notifier *notify); diff --git a/softmmu/vl.c b/softmmu/vl.c index 32c0047889..a913ed5469
100644

softmmu/vl.c +1 -0
@@ -112,6 +112,7 @@
 #include "qapi/qmp/qerror.h"
 #include "sysemu/iothread.h"
 #include "qemu/guest-random.h"
+#include "net/colo-compare.h"
 
Quoted 4 lines
Quoted 4 lines
softmmu/vl.c +0 -0
 #define MAX_VIRTIO_CONSOLES 1
 
@@ -2906,6 +2907,7 @@ void qemu_init(int argc, char **argv,

char

**envp)

Quoted 11 lines
Quoted 10 lines
     precopy_infrastructure_init();
     postcopy_infrastructure_init();
     monitor_init_globals();

+ colo_compare_init_globals();

if (qcrypto_init(&err) < 0) {
error_reportf_err(err, "cannot initialize crypto:
");
--
2.20.1

#23 lore raw

Re: [PATCH v4 5/6] net/colo-compare.c, softmmu/vl.c: Check that colo-compare is active

In reply to Zhang, Chen
Quoted 81 lines · * Zhang, Chen ([email protected]) wrote:
Quoted 73 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Friday, May 8, 2020 2:11 PM
To: Zhang, Chen <[email protected]>
Cc: qemu-devel <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: Re: [PATCH v4 5/6] net/colo-compare.c, softmmu/vl.c: Check that colo-compare is active

Quoted 37 lines · Chen
Quoted 37 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Thursday, May 7, 2020 11:54 PM
To: Zhang, Chen <[email protected]>
Cc: qemu-devel <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: Re: [PATCH v4 5/6] net/colo-compare.c, softmmu/vl.c: Check that colo-compare is active

Quoted 23 lines · Chen
Quoted 23 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Monday, May 4, 2020 6:28 PM
To: qemu-devel <[email protected]>
Cc: Zhang, Chen <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: [PATCH v4 5/6] net/colo-compare.c, softmmu/vl.c: Check that
colo- compare is active

If the colo-compare object is removed before failover and a checkpoint happens, qemu crashes because it tries to lock the destroyed event_mtx in colo_notify_compares_event.

Fix this by checking if everything is initialized by introducing a new variable colo_compare_active which is protected by a new mutex colo_compare_mutex. The new mutex also protects against concurrent access of the net_compares list and makes sure that colo_notify_compares_event isn't active while we destroy event_mtx and event_complete_cond.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

With this it also is again possible to use colo without colo-compare (periodic
mode) and to use multiple colo-compare for multiple network

interfaces.

Quoted 16 lines
Quoted 15 lines
Quoted 6 lines
Quoted 1 line

Hi Lukas,

For this case I think we don't need to touch vl.c code, we can solve this

issue from another perspective:

Quoted 4 lines

How to remove colo-compare?
User will use qemu-monitor or QMP command to disable an object, so we just need return operation failed When user try to remove colo-compare

object while COLO is running.

Yeah, but that still leaves the other problem that colo can't be used without colo-compare (qemu crashes then).

Quoted 7 lines
Quoted 3 lines

Yes, the COLO-compare is necessary module in COLO original design. At most cases, user need it do dynamic sync.
For rare cases, maybe we can add a new colo-compare parameter to

bypass all the network workload.

I think such an parameter would only be a workaround instead of a real solution like this patch.

Quoted 7 lines

The root problem is why COLO-compare is necessary.
Yes, maybe someone want to use pure periodic synchronization mode,
But it means it will lost all guest network support(without colo-compare/filter-mirror/filter-redirector/filter-rewriter). The secondary guest just a solid backup for the primary guest, when occur failover the new build stateful connection (like TCP) will crashed, need userspace to handle this status. It lost the original meaning for COLO FT/HA solution, no need use do HA in application layer. it looks like normal/remote periodic VM snapshot here.
Dave or Jason have any comments here?

People have done fixed-rate VM synchronisation in the past; and there are workloads where the packet stream is particularly random so you almost always get comparison miscompares.

But in those setups, the rest of the configuration is very different - since the destination isn't running at the same time as the source, the block mirroring also gets simpler in those scenarious.

One thing we played with in the past was dynamically turning comparison on and off; switching back to simpler synchronisation if the workload turns out to suit it.

Dave

Quoted 170 lines

Thanks
Zhang Chen

Quoted 166 lines

Regards,
Lukas Straub

Quoted 26 lines

Thanks
Zhang Chen

Quoted 23 lines

Regards,
Lukas Straub

Quoted 10 lines

Thanks
Zhang Chen

Quoted 7 lines
---
 net/colo-compare.c | 35 +++++++++++++++++++++++++++++------
 net/colo-compare.h |  1 +
 softmmu/vl.c       |  2 ++
 3 files changed, 32 insertions(+), 6 deletions(-)
Quoted 6 lines

net/colo-compare.c static NotifierList colo_compare_notifiers = · line 54

Quoted 6 lines
net/colo-compare.c index renamed from net/colo-compare.c +0 -0

56db3d3bfc..c7572d75e9 100644

net/colo-compare.c +0 -0
@@ -54,6 +54,8 @@ static NotifierList colo_compare_notifiers =
 #define REGULAR_PACKET_CHECK_MS 3000  #define

DEFAULT_TIME_OUT_MS

Quoted 2 lines
Quoted 2 lines

3000

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
net/colo-compare.c +2 -0
+static QemuMutex colo_compare_mutex; static bool
+colo_compare_active;
 static QemuMutex event_mtx;
 static QemuCond event_complete_cond;  static int

event_unhandled_count; @@ -906,6 +908,12 @@ static void check_old_packet_regular(void *opaque) void colo_notify_compares_event(void *opaque, int event, Error **errp)

{

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
     CompareState *s;
net/colo-compare.c +6 -0
+    qemu_mutex_lock(&colo_compare_mutex);
+
+    if (!colo_compare_active) {
+        qemu_mutex_unlock(&colo_compare_mutex);
+        return;
+    }
 
     qemu_mutex_lock(&event_mtx);
     QTAILQ_FOREACH(s, &net_compares, next) { @@ -919,6 +927,7

@@ void colo_notify_compares_event(void *opaque, int event,

Error **errp)

Quoted 55 lines
Quoted 55 lines
Quoted 8 lines
Quoted 8 lines
     }

     qemu_mutex_unlock(&event_mtx);

+ qemu_mutex_unlock(&colo_compare_mutex);
}

static void colo_compare_timer_init(CompareState *s) @@ -1274,7
+1283,14 @@ static void colo_compare_complete(UserCreatable *uc,

Error **errp)

Quoted 46 lines
Quoted 46 lines
                            s->vnet_hdr);
     }
net/colo-compare.c +7 -0
+    qemu_mutex_lock(&colo_compare_mutex);
+    if (!colo_compare_active) {
+        qemu_mutex_init(&event_mtx);
+        qemu_cond_init(&event_complete_cond);
+        colo_compare_active = true;
+    }
     QTAILQ_INSERT_TAIL(&net_compares, s, next);
+    qemu_mutex_unlock(&colo_compare_mutex);
 
     s->out_sendco.s = s;
     s->out_sendco.chr = &s->chr_out; @@ -1290,9 +1306,6 @@

static void colo_compare_complete(UserCreatable
*uc, Error **errp)

     g_queue_init(&s->conn_list);
net/colo-compare.c +0 -3
-    qemu_mutex_init(&event_mtx);
-    qemu_cond_init(&event_complete_cond);
-
     s->connection_track_table =

g_hash_table_new_full(connection_key_hash,

                                                       connection_key_equal,
                                                       g_free,

@@
-1384,12 +1397,19 @@ static void colo_compare_finalize(Object *obj)

     qemu_bh_delete(s->event_bh);
net/colo-compare.c +7 -0
+    qemu_mutex_lock(&colo_compare_mutex);
     QTAILQ_FOREACH(tmp, &net_compares, next) {
         if (tmp == s) {
             QTAILQ_REMOVE(&net_compares, s, next);
             break;
         }
     }
+    if (QTAILQ_EMPTY(&net_compares)) {
+        colo_compare_active = false;
+        qemu_mutex_destroy(&event_mtx);
+        qemu_cond_destroy(&event_complete_cond);
+    }
+    qemu_mutex_unlock(&colo_compare_mutex);
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

net/colo-compare.c static void colo_compare_finalize(Object

Quoted 3 lines
     AioContext *ctx = iothread_get_aio_context(s->iothread);
     aio_context_acquire(ctx);
net/colo-compare.c +0 -0
@@ -1413,15 +1433,18 @@ static void colo_compare_finalize(Object

*obj)

Quoted 57 lines
Quoted 56 lines
Quoted 42 lines
Quoted 42 lines
net/colo-compare.c +5 -3
         object_unref(OBJECT(s->iothread));
     }
 
-    qemu_mutex_destroy(&event_mtx);
-    qemu_cond_destroy(&event_complete_cond);
-
     g_free(s->pri_indev);
     g_free(s->sec_indev);
     g_free(s->outdev);
     g_free(s->notify_dev);
 }
 
+void colo_compare_init_globals(void) {
+    colo_compare_active = false;
+    qemu_mutex_init(&colo_compare_mutex);
+}
+
 static const TypeInfo colo_compare_info = {
     .name = TYPE_COLO_COMPARE,
     .parent = TYPE_OBJECT,
net/colo-compare.h index renamed from net/colo-compare.h +0 -0

22ddd512e2..eb483ac586 100644

net/colo-compare.h +1 -0
@@ -17,6 +17,7 @@
 #ifndef QEMU_COLO_COMPARE_H
 #define QEMU_COLO_COMPARE_H
 
+void colo_compare_init_globals(void);
 void colo_notify_compares_event(void *opaque, int event, Error
 **errp); void colo_compare_register_notifier(Notifier *notify);

void colo_compare_unregister_notifier(Notifier *notify); diff --git a/softmmu/vl.c b/softmmu/vl.c index 32c0047889..a913ed5469 100644

softmmu/vl.c +1 -0
@@ -112,6 +112,7 @@
 #include "qapi/qmp/qerror.h"
 #include "sysemu/iothread.h"
 #include "qemu/guest-random.h"
+#include "net/colo-compare.h"
 
Quoted 3 lines

softmmu/vl.c void qemu_init(int argc, char **argv, char

Quoted 3 lines
softmmu/vl.c +0 -0
 #define MAX_VIRTIO_CONSOLES 1
 
@@ -2906,6 +2907,7 @@ void qemu_init(int argc, char **argv, char

**envp)

Quoted 10 lines
Quoted 9 lines
softmmu/vl.c +1 -1
     precopy_infrastructure_init();
     postcopy_infrastructure_init();
     monitor_init_globals();
+    colo_compare_init_globals();
 
     if (qcrypto_init(&err) < 0) {
         error_reportf_err(err, "cannot initialize crypto: ");
--

2.20.1

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#29 lore raw
In reply to Lukas Straub

Please update this series, I will queue it to COLO branch.

Thanks
Zhang Chen

Quoted 58 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Monday, May 4, 2020 6:28 PM
To: qemu-devel <[email protected]>
Cc: Zhang, Chen <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: [PATCH v4 0/6] colo-compare bugfixes

Hello Everyone,
Here are fixes for bugs that I found in my tests.

Benchmark results:
Client-to-server tcp:
without patch: ~63 Mbit/s
with patch: ~66 Mbit/s
Server-to-client tcp:
without patch: ~771 Kbit/s
with patch: ~702 Kbit/s

Regards,
Lukas Straub

Version changes:
v4:
-fix potential deadlock with notify_remote_frame -avoid malloc and
memcpy in many cases

v3:
-fix checkpatch.pl error

v2:
-better wording
-fix performance-regression in patch 3 "net/colo-compare.c: Fix deadlock in
compare_chr_send"
-add more bugfixes

Lukas Straub (6):
net/colo-compare.c: Create event_bh with the right AioContext chardev/char.c: Use qemu_co_sleep_ns if in coroutine net/colo-compare.c: Fix deadlock in compare_chr_send net/colo-compare.c: Only hexdump packets if tracing is enabled net/colo-compare.c, softmmu/vl.c: Check that colo-compare is active net/colo-compare.c: Correct ordering in complete and finalize

 chardev/char.c     |   7 +-
 net/colo-compare.c | 250 +++++++++++++++++++++++++++++++++---------
---
 net/colo-compare.h |   1 +
 net/colo.c         |   7 ++
 net/colo.h         |   1 +
 softmmu/vl.c       |   2 +
 6 files changed, 204 insertions(+), 64 deletions(-)

--
2.20.1