colo-compare bugfixes
PATCH v4 1/6 net/colo-compare.c: Create event_bh with the right AioContext
In reply to Lukas Straubqemu_bh_new will set the bh to be executed in the main loop. This causes crashes as colo_compare_handle_event assumes that it has exclusive access the queues, which are also concurrently accessed in the iothread.
Create the bh with the AioContext of the iothread to fulfill these assumptions and fix the crashes. This is safe, because the bh already takes the appropriate locks.
--- net/colo-compare.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
net/colo-compare.c +2 -1
@@ -890,6 +890,7 @@ static void colo_compare_handle_event(void *opaque)static void colo_compare_iothread (CompareState *s) { +AioContext *ctx =iothread_get_aio_context (s->iothread);object_ref (OBJECT (s->iothread)); s->worker_context =iothread_get_g_main_context (s->iothread);
@@ -906,7 +907,7 @@ static void colo_compare_iothread(CompareState *s) }colo_compare_timer_init (s); - s->event_bh =qemu_bh_new (colo_compare_handle_event, s); + s->event_bh =aio_bh_new (ctx, colo_compare_handle_event, s); }static char *compare_get_pri_indev (Object *obj,Error **errp)
Signature · 1 line
2.20.1
v4 cover colo-compare bugfixes
Hello Everyone,
Here are fixes for bugs that I found in my tests.
Benchmark results:
Client-to-server tcp:
without patch: ~63 Mbit/s
with patch: ~66 Mbit/s
Server-to-client tcp:
without patch: ~771 Kbit/s
with patch: ~702 Kbit/s
Regards,
Lukas Straub
Version changes:
v4:
-fix potential deadlock with notify_remote_frame
-avoid malloc and memcpy in many cases
v3:
-fix checkpatch.pl error
v2:
-better wording
-fix performance-regression in patch 3 "net/colo-compare.c: Fix deadlock in compare_chr_send" -add more bugfixes
Lukas Straub (6):
net/colo-compare.c: Create event_bh with the right AioContext chardev/char.c: Use qemu_co_sleep_ns if in coroutine net/colo-compare.c: Fix deadlock in compare_chr_send net/colo-compare.c: Only hexdump packets if tracing is enabled net/colo-compare.c, softmmu/vl.c: Check that colo-compare is active net/colo-compare.c: Correct ordering in complete and finalize
chardev/char.c | 7 +- net/colo-compare.c | 250 +++++++++++++++++++++++++++++++++------------ net/colo-compare.h | 1 + net/colo.c | 7 ++ net/colo.h | 1 + softmmu/vl.c | 2 + 6 files changed, 204 insertions(+), 64 deletions(-)
Signature · 1 line
2.20.1
PATCH v4 6/6 net/colo-compare.c: Correct ordering in complete and finalize
In colo_compare_complete, insert CompareState into net_compares only after everything has been initialized.
In colo_compare_finalize, remove CompareState from net_compares before anything is deinitialized.
--- net/colo-compare.c | 45 +++++++++++++++++++++++---------------------- 1 file changed, 23 insertions(+), 22 deletions(-)
net/colo-compare.c +23 -22
@@ -1283,15 +1283,6 @@ static void colo_compare_complete(UserCreatable *uc, Error **errp) s->vnet_hdr); } -qemu_mutex_lock (&colo_compare_mutex); -if (!colo_compare_active) { -qemu_mutex_init (&event_mtx); -qemu_cond_init (&event_complete_cond); - colo_compare_active = true; - } -QTAILQ_INSERT_TAIL (&net_compares, s, next); -qemu_mutex_unlock (&colo_compare_mutex); - s->out_sendco.s = s; s->out_sendco.chr = &s->chr_out; s->out_sendco.notify_remote_frame = false;
@@ -1312,6 +1303,16 @@ static void colo_compare_complete(UserCreatable *uc, Error **errp) connection_destroy);colo_compare_iothread (s); + +qemu_mutex_lock (&colo_compare_mutex); +if (!colo_compare_active) { +qemu_mutex_init (&event_mtx); +qemu_cond_init (&event_complete_cond); + colo_compare_active = true; + } +QTAILQ_INSERT_TAIL (&net_compares, s, next); +qemu_mutex_unlock (&colo_compare_mutex); +return ; }
@@ -1384,19 +1385,6 @@ static void colo_compare_finalize(Object *obj)CompareState *s =COLO_COMPARE (obj);CompareState *tmp =NULL ; -qemu_chr_fe_deinit (&s->chr_pri_in, false); -qemu_chr_fe_deinit (&s->chr_sec_in, false); -qemu_chr_fe_deinit (&s->chr_out, false); -if (s->notify_dev) { -qemu_chr_fe_deinit (&s->chr_notify_dev, false); - } - -if (s->iothread) { -colo_compare_timer_del (s); - } - -qemu_bh_delete (s->event_bh); -qemu_mutex_lock (&colo_compare_mutex); QTAILQ_FOREACH(tmp, &net_compares, next) {if (tmp == s) {
@@ -1411,6 +1399,19 @@ static void colo_compare_finalize(Object *obj) }qemu_mutex_unlock (&colo_compare_mutex); +qemu_chr_fe_deinit (&s->chr_pri_in, false); +qemu_chr_fe_deinit (&s->chr_sec_in, false); +qemu_chr_fe_deinit (&s->chr_out, false); +if (s->notify_dev) { +qemu_chr_fe_deinit (&s->chr_notify_dev, false); + } + +if (s->iothread) { +colo_compare_timer_del (s); + } + +qemu_bh_delete (s->event_bh); +AioContext *ctx =iothread_get_aio_context (s->iothread);aio_context_acquire (ctx);AIO_WAIT_WHILE (ctx, !s->out_sendco.done);
Signature · 1 line
2.20.1
PATCH v4 2/6 chardev/char.c: Use qemu_co_sleep_ns if in coroutine
In reply to Lukas StraubThis will be needed in the next patch.
--- chardev/char.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
chardev/char.c +6 -1
@@ -38,6 +38,7 @@#include "qemu/module.h" #include "qemu/option.h" #include "qemu/id.h" +#include "qemu/coroutine.h" #include "chardev/char-mux.h"
@@ -119,7 +120,11 @@ static int qemu_chr_write_buffer(Chardev *s, retry: res = cc->chr_write (s, buf + *offset, len - *offset);if (res <0 && errno ==EAGAIN && write_all) { -g_usleep (100 ); +if (qemu_in_coroutine ()) { +qemu_co_sleep_ns (QEMU_CLOCK_REALTIME ,100000 ); + }else { +g_usleep (100 ); + } goto retry; }
Signature · 1 line
2.20.1
PATCH v4 3/6 net/colo-compare.c: Fix deadlock in compare_chr_send
In reply to Lukas StraubThe chr_out chardev is connected to a filter-redirector running in the main loop. qemu_chr_fe_write_all might block here in compare_chr_send if the (socket-)buffer is full. If another filter-redirector in the main loop want's to send data to chr_pri_in it might also block if the buffer is full. This leads to a deadlock because both event loops get blocked.
Fix this by converting compare_chr_send to a coroutine and putting the packets in a send queue.
--- net/colo-compare.c | 187 ++++++++++++++++++++++++++++++++++----------- net/colo.c | 7 ++ net/colo.h | 1 + 3 files changed, 150 insertions(+), 45 deletions(-)
net/colo-compare.c +142 -45
@@ -32,6 +32,9 @@#include "migration/migration.h" #include "util.h" +#include "block/aio-wait.h" +#include "qemu/coroutine.h" +#define TYPE_COLO_COMPARE "colo-compare"#define COLO_COMPARE (obj) \ OBJECT_CHECK(CompareState, (obj), TYPE_COLO_COMPARE)
@@ -77,6 +80,23 @@ static int event_unhandled_count;* |packet | |packet + |packet | |packet + * +--------+ +--------+ +--------+ +--------+ */ + +typedef struct SendCo { +Coroutine *co; +struct CompareState *s; +CharBackend *chr; +GQueue send_list; +bool notify_remote_frame; +bool done; +int ret; +}SendCo ; + +typedef struct SendEntry { +uint32_t size; +uint32_t vnet_hdr_len; +uint8_t *buf; +}SendEntry ; + typedef struct CompareState { Object parent;
@@ -91,6 +111,8 @@ typedef struct CompareState {SocketReadState pri_rs;SocketReadState sec_rs;SocketReadState notify_rs; +SendCo out_sendco; +SendCo notify_sendco;bool vnet_hdr;uint32_t compare_timeout;uint32_t expired_scan_cycle;
@@ -124,10 +146,11 @@ enum {static int compare_chr_send (CompareState *s, -const uint8_t *buf, +uint8_t *buf,uint32_t size,uint32_t vnet_hdr_len, -bool notify_remote_frame); +bool notify_remote_frame, +bool zero_copy); static bool packet_matches_str(const char *str, const uint8_t *buf,
@@ -145,7 +168,7 @@ static void notify_remote_frame(CompareState *s)char msg[] ="DO_CHECKPOINT" ;int ret =0 ; - ret =compare_chr_send (s, (uint8_t *)msg,strlen (msg),0 , true); + ret =compare_chr_send (s, (uint8_t *)msg,strlen (msg),0 , true, false);if (ret <0 ) {error_report ("Notify Xen COLO-frame failed" ); }
@@ -272,12 +295,13 @@ static void colo_release_primary_pkt(CompareState *s, Packet *pkt) pkt->data, pkt->size, pkt->vnet_hdr_len, - false); + false, + true);if (ret <0 ) {error_report ("colo send primary packet failed" ); }trace_colo_compare_main ("packet same and release packet" ); -packet_destroy (pkt,NULL ); +packet_destroy_partial (pkt,NULL ); }/*
@@ -699,65 +723,115 @@ static void colo_compare_connection(void *opaque, void *user_data) } } -static int compare_chr_send (CompareState *s, -const uint8_t *buf, -uint32_t size, -uint32_t vnet_hdr_len, -bool notify_remote_frame) +static void coroutine_fn _compare_chr_send(void *opaque) { +SendCo *sendco = opaque; +CompareState *s = sendco->s;int ret =0 ; -uint32_t len =htonl (size); -if (!size) { -return 0 ; - } +while (!g_queue_is_empty (&sendco->send_list)) { +SendEntry *entry =g_queue_pop_tail (&sendco->send_list); +uint32_t len =htonl (entry->size); -if (notify_remote_frame) { - ret =qemu_chr_fe_write_all (&s->chr_notify_dev, - (uint8_t *)&len, -sizeof (len)); - }else { - ret =qemu_chr_fe_write_all (&s->chr_out, (uint8_t *)&len,sizeof (len)); - } + ret =qemu_chr_fe_write_all (sendco->chr, (uint8_t *)&len,sizeof (len)); -if (ret !=sizeof (len)) { - goto err; - } +if (ret !=sizeof (len)) { +g_free (entry->buf); +g_slice_free (SendEntry, entry); + goto err; + } -if (s->vnet_hdr) { -/* -* We send vnet header len make other module(like filter-redirector) -* know how to parse net packet correctly. -*/ - len =htonl (vnet_hdr_len); +if (!sendco->notify_remote_frame && s->vnet_hdr) { +/* +* We send vnet header len make other module(like filter-redirector) +* know how to parse net packet correctly. +*/ + len =htonl (entry->vnet_hdr_len); -if (!notify_remote_frame) { - ret =qemu_chr_fe_write_all (&s->chr_out, + ret =qemu_chr_fe_write_all (sendco->chr, (uint8_t *)&len,sizeof (len)); + +if (ret !=sizeof (len)) { +g_free (entry->buf); +g_slice_free (SendEntry, entry); + goto err; + } } -if (ret !=sizeof (len)) { + ret =qemu_chr_fe_write_all (sendco->chr, + (uint8_t *)entry->buf, + entry->size); + +if (ret != entry->size) { +g_free (entry->buf); +g_slice_free (SendEntry, entry); goto err; } + +g_free (entry->buf); +g_slice_free (SendEntry, entry); } + sendco->ret =0 ; + goto out; + +err: +while (!g_queue_is_empty (&sendco->send_list)) { +SendEntry *entry =g_queue_pop_tail (&sendco->send_list); +g_free (entry->buf); +g_slice_free (SendEntry, entry); + } + sendco->ret = ret <0 ? ret : -EIO ; +out: + sendco->co =NULL ; + sendco->done = true; +aio_wait_kick (); +} + +static int compare_chr_send (CompareState *s, +uint8_t *buf, +uint32_t size, +uint32_t vnet_hdr_len, +bool notify_remote_frame, +bool zero_copy) +{ +SendCo *sendco; +SendEntry *entry; +if (notify_remote_frame) { - ret =qemu_chr_fe_write_all (&s->chr_notify_dev, - (uint8_t *)buf, - size); + sendco = &s->notify_sendco; }else { - ret =qemu_chr_fe_write_all (&s->chr_out, (uint8_t *)buf, size); + sendco = &s->out_sendco; } -if (ret != size) { - goto err; +if (!size) { +return 0 ; } -return 0 ; + entry =g_slice_new (SendEntry); + entry->size = size; + entry->vnet_hdr_len = vnet_hdr_len; +if (zero_copy) { + entry->buf = buf; + }else { + entry->buf =g_malloc (size); +memcpy (entry->buf, buf, size); + } +g_queue_push_head (&sendco->send_list, entry); + +if (sendco->done) { + sendco->co =qemu_coroutine_create (_compare_chr_send, sendco); + sendco->done = false; +qemu_coroutine_enter (sendco->co); +if (sendco->done) { +/* report early errors */ +return sendco->ret; + } + } -err: -return ret <0 ? ret : -EIO ; +/* assume success */ +return 0 ; }static int compare_chr_can_read (void *opaque)
@@ -1063,6 +1137,7 @@ static void compare_pri_rs_finalize(SocketReadState *pri_rs) pri_rs->buf, pri_rs->packet_len, pri_rs->vnet_hdr_len, + false, false); }else {/* compare packet in the specified connection */
@@ -1093,7 +1168,7 @@ static void compare_notify_rs_finalize(SocketReadState *notify_rs)if (packet_matches_str ("COLO_USERSPACE_PROXY_INIT" , notify_rs->buf, notify_rs->packet_len)) { - ret =compare_chr_send (s, (uint8_t *)msg,strlen (msg),0 , true); + ret =compare_chr_send (s, (uint8_t *)msg,strlen (msg),0 , true, false);if (ret <0 ) {error_report ("Notify Xen COLO-frame INIT failed" ); }
@@ -1199,6 +1274,18 @@ static void colo_compare_complete(UserCreatable *uc, Error **errp)QTAILQ_INSERT_TAIL (&net_compares, s, next); + s->out_sendco.s = s; + s->out_sendco.chr = &s->chr_out; + s->out_sendco.notify_remote_frame = false; + s->out_sendco.done = true; +g_queue_init (&s->out_sendco.send_list); + + s->notify_sendco.s = s; + s->notify_sendco.chr = &s->chr_notify_dev; + s->notify_sendco.notify_remote_frame = true; + s->notify_sendco.done = true; +g_queue_init (&s->notify_sendco.send_list); +g_queue_init (&s->conn_list);qemu_mutex_init (&event_mtx);
@@ -1225,8 +1312,9 @@ static void colo_flush_packets(void *opaque, void *user_data) pkt->data, pkt->size, pkt->vnet_hdr_len, - false); -packet_destroy (pkt,NULL ); + false, + true); +packet_destroy_partial (pkt,NULL ); }while (!g_queue_is_empty (&conn->secondary_list)) { pkt =g_queue_pop_head (&conn->secondary_list);
@@ -1301,10 +1389,19 @@ static void colo_compare_finalize(Object *obj) } } +AioContext *ctx =iothread_get_aio_context (s->iothread); +aio_context_acquire (ctx); +AIO_WAIT_WHILE (ctx, !s->out_sendco.done); +AIO_WAIT_WHILE (ctx, !s->notify_sendco.done); +aio_context_release (ctx); +/* Release all unhandled packets after compare thead exited */ g_queue_foreach (&s->conn_list, colo_flush_packets, s); +AIO_WAIT_WHILE (NULL , !s->out_sendco.done);g_queue_clear (&s->conn_list); +g_queue_clear (&s->out_sendco.send_list); +g_queue_clear (&s->notify_sendco.send_list);if (s->connection_track_table) {g_hash_table_destroy (s->connection_track_table);
net/colo.c +7 -0
@@ -185,6 +185,13 @@ void packet_destroy(void *opaque, void *user_data)g_slice_free (Packet, pkt); } +void packet_destroy_partial (void *opaque,void *user_data) +{ +Packet *pkt = opaque; + +g_slice_free (Packet, pkt); +} +/* * Clear hashtable, stop this hash growing really huge */
net/colo.h +1 -0
@@ -102,5 +102,6 @@ bool connection_has_tracked(GHashTable *connection_track_table,void connection_hashtable_reset (GHashTable *connection_track_table);Packet *packet_new (const void *data,int size,int vnet_hdr_len);void packet_destroy (void *opaque,void *user_data); +void packet_destroy_partial (void *opaque,void *user_data);#endif /* NET_COLO_H */
Signature · 1 line
2.20.1
PATCH v4 5/6 net/colo-compare.c, softmmu/vl.c: Check that colo-compare is active
In reply to Lukas StraubIf the colo-compare object is removed before failover and a checkpoint happens, qemu crashes because it tries to lock the destroyed event_mtx in colo_notify_compares_event.
Fix this by checking if everything is initialized by introducing a new variable colo_compare_active which is protected by a new mutex colo_compare_mutex. The new mutex also protects against concurrent access of the net_compares list and makes sure that colo_notify_compares_event isn't active while we destroy event_mtx and event_complete_cond.
With this it also is again possible to use colo without colo-compare (periodic mode) and to use multiple colo-compare for multiple network interfaces.
--- net/colo-compare.c | 35 +++++++++++++++++++++++++++++------ net/colo-compare.h | 1 + softmmu/vl.c | 2 ++ 3 files changed, 32 insertions(+), 6 deletions(-)
net/colo-compare.c +29 -6
@@ -54,6 +54,8 @@ static NotifierList colo_compare_notifiers =#define REGULAR_PACKET_CHECK_MS 3000#define DEFAULT_TIME_OUT_MS 3000 +static QemuMutex colo_compare_mutex; +static bool colo_compare_active;static QemuMutex event_mtx;static QemuCond event_complete_cond;static int event_unhandled_count;
@@ -906,6 +908,12 @@ static void check_old_packet_regular(void *opaque)void colo_notify_compares_event (void *opaque,int event,Error **errp) {CompareState *s; +qemu_mutex_lock (&colo_compare_mutex); + +if (!colo_compare_active) { +qemu_mutex_unlock (&colo_compare_mutex); +return ; + }qemu_mutex_lock (&event_mtx); QTAILQ_FOREACH(s, &net_compares, next) {
@@ -919,6 +927,7 @@ void colo_notify_compares_event(void *opaque, int event, Error **errp) }qemu_mutex_unlock (&event_mtx); +qemu_mutex_unlock (&colo_compare_mutex); }static void colo_compare_timer_init (CompareState *s)
@@ -1274,7 +1283,14 @@ static void colo_compare_complete(UserCreatable *uc, Error **errp) s->vnet_hdr); } +qemu_mutex_lock (&colo_compare_mutex); +if (!colo_compare_active) { +qemu_mutex_init (&event_mtx); +qemu_cond_init (&event_complete_cond); + colo_compare_active = true; + }QTAILQ_INSERT_TAIL (&net_compares, s, next); +qemu_mutex_unlock (&colo_compare_mutex); s->out_sendco.s = s; s->out_sendco.chr = &s->chr_out;
@@ -1290,9 +1306,6 @@ static void colo_compare_complete(UserCreatable *uc, Error **errp)g_queue_init (&s->conn_list); -qemu_mutex_init (&event_mtx); -qemu_cond_init (&event_complete_cond); - s->connection_track_table = g_hash_table_new_full(connection_key_hash, connection_key_equal, g_free,
@@ -1384,12 +1397,19 @@ static void colo_compare_finalize(Object *obj)qemu_bh_delete (s->event_bh); +qemu_mutex_lock (&colo_compare_mutex); QTAILQ_FOREACH(tmp, &net_compares, next) {if (tmp == s) {QTAILQ_REMOVE (&net_compares, s, next);break ; } } +if (QTAILQ_EMPTY (&net_compares)) { + colo_compare_active = false; +qemu_mutex_destroy (&event_mtx); +qemu_cond_destroy (&event_complete_cond); + } +qemu_mutex_unlock (&colo_compare_mutex);AioContext *ctx =iothread_get_aio_context (s->iothread);aio_context_acquire (ctx);
@@ -1413,15 +1433,18 @@ static void colo_compare_finalize(Object *obj)object_unref (OBJECT (s->iothread)); } -qemu_mutex_destroy (&event_mtx); -qemu_cond_destroy (&event_complete_cond); -g_free (s->pri_indev);g_free (s->sec_indev);g_free (s->outdev);g_free (s->notify_dev); } +void colo_compare_init_globals (void ) +{ + colo_compare_active = false; +qemu_mutex_init (&colo_compare_mutex); +} +static const TypeInfo colo_compare_info = { .name =TYPE_COLO_COMPARE , .parent =TYPE_OBJECT ,
net/colo-compare.h +1 -0
@@ -17,6 +17,7 @@#ifndef QEMU_COLO_COMPARE_H #define QEMU_COLO_COMPARE_H +void colo_compare_init_globals (void );void colo_notify_compares_event (void *opaque,int event,Error **errp);void colo_compare_register_notifier (Notifier *notify);void colo_compare_unregister_notifier (Notifier *notify);
softmmu/vl.c +2 -0
@@ -112,6 +112,7 @@#include "qapi/qmp/qerror.h" #include "sysemu/iothread.h" #include "qemu/guest-random.h" +#include "net/colo-compare.h" #define MAX_VIRTIO_CONSOLES 1
@@ -2906,6 +2907,7 @@ void qemu_init(int argc, char **argv, char **envp)precopy_infrastructure_init ();postcopy_infrastructure_init ();monitor_init_globals (); +colo_compare_init_globals ();if (qcrypto_init (&err) <0 ) {error_reportf_err (err,"cannot initialize crypto: " );
Signature · 1 line
2.20.1
PATCH v4 4/6 net/colo-compare.c: Only hexdump packets if tracing is enabled
In reply to Lukas StraubElse the log will be flooded if there is a lot of network traffic.
--- net/colo-compare.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-)
net/colo-compare.c +6 -4
@@ -483,10 +483,12 @@ sec:g_queue_push_head (&conn->primary_list, ppkt);g_queue_push_head (&conn->secondary_list, spkt); -qemu_hexdump ((char *)ppkt->data, stderr, -"colo-compare ppkt" , ppkt->size); -qemu_hexdump ((char *)spkt->data, stderr, -"colo-compare spkt" , spkt->size); +if (trace_event_get_state_backends (TRACE_COLO_COMPARE_MISCOMPARE )) { +qemu_hexdump ((char *)ppkt->data, stderr, +"colo-compare ppkt" , ppkt->size); +qemu_hexdump ((char *)spkt->data, stderr, +"colo-compare spkt" , spkt->size); + }colo_compare_inconsistency_notify (s); }
Signature · 1 line
2.20.1
Re: [PATCH v4 4/6] net/colo-compare.c: Only hexdump packets if tracing is enabled
Quoted 14 lines · Lukas Straub
Else the log will be flooded if there is a lot of network traffic.
Signed-off-by Lukas Straub <[email protected]>Reviewed-by Zhang Chen <[email protected]>--- net/colo-compare.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-)net/colo-compare.c +0
-0
net/colo-compare.c sec: · line 486
Quoted 8 lines
net/colo-compare.c +1
-4@@ -483,10 +483,12 @@ sec:g_queue_push_head (&conn->primary_list, ppkt);g_queue_push_head (&conn->secondary_list, spkt); -qemu_hexdump ((char *)ppkt->data, stderr, -"colo-compare ppkt" , ppkt->size); -qemu_hexdump ((char *)spkt->data, stderr, -"colo-compare spkt" , spkt->size); +if (trace_event_get_state_backends (TRACE_COLO_COMPARE_MISCOMPARE )) {
Quoted 9 lines
net/colo-compare.c +5
-0+qemu_hexdump ((char *)ppkt->data, stderr, +"colo-compare ppkt" , ppkt->size); +qemu_hexdump ((char *)spkt->data, stderr, +"colo-compare spkt" , spkt->size); + }colo_compare_inconsistency_notify (s); }
Philippe Mathieu-Daudé
Tested-byRe: [PATCH v4 4/6] net/colo-compare.c: Only hexdump packets if tracing is enabled
Quoted 10 lines · Philippe Mathieu-Daudé
Quoted 9 lines · Lukas Straub
Else the log will be flooded if there is a lot of network traffic.
Signed-off-by Lukas Straub <[email protected]>Reviewed-by Zhang Chen <[email protected]>---
net/colo-compare.c| 10 ++++++----1 file changed, 6 insertions(+), 4 deletions(-)
Quoted 15 lines
net/colo-compare.c sec: · line 486
Quoted 13 lines
net/colo-compare.c +1
-4@@ -483,10 +483,12 @@ sec: g_queue_push_head(&conn->primary_list, ppkt); g_queue_push_head(&conn->secondary_list, spkt); - qemu_hexdump((char *)ppkt->data, stderr, - "colo-compare ppkt", ppkt->size); - qemu_hexdump((char *)spkt->data, stderr, - "colo-compare spkt", spkt->size); + if (trace_event_get_state_backends (TRACE_COLO_COMPARE_MISCOMPARE )) {Reviewed-by Philippe Mathieu-Daudé <[email protected]>
Quoted 9 lines
Quoted 9 lines
net/colo-compare.c +5
-0+ qemu_hexdump((char *)ppkt->data, stderr, + "colo-compare ppkt", ppkt->size); + qemu_hexdump((char *)spkt->data, stderr, + "colo-compare spkt", spkt->size); + } colo_compare_inconsistency_notify(s); }
RE: [PATCH v4 3/6] net/colo-compare.c: Fix deadlock in compare_chr_send
In reply to Lukas StraubQuoted 299 lines
-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Monday, May 4, 2020 6:28 PM
To: qemu-devel <[email protected]>
Cc: Zhang, Chen <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: [PATCH v4 3/6]net/colo-compare.c: Fix deadlock incompare_chr_sendThe
chr_outchardev is connected to a filter-redirector running in the main loop.qemu_chr_fe_write_allmight block here incompare_chr_sendif the (socket-)buffer is full.
If another filter-redirector in the main loop want's to send data tochr_pri_init might also block if the buffer is full. This leads to a deadlock because both event loops get blocked.Fix this by converting
compare_chr_sendto a coroutine and putting the packets in a send queue.Signed-off-by Lukas Straub <[email protected]>--- net/colo-compare.c | 187 ++++++++++++++++++++++++++++++++++------- ---- net/colo.c | 7 ++ net/colo.h | 1 + 3 files changed, 150 insertions(+), 45 deletions(-)net/colo-compare.c index renamed from net/colo-compare.c +0
-0
1de4220fe2..2a4e7f7c4e100644net/colo-compare.c +26
-2@@ -32,6 +32,9 @@#include "migration/migration.h" #include "util.h" +#include "block/aio-wait.h" +#include "qemu/coroutine.h" +#define TYPE_COLO_COMPARE "colo-compare" #define COLO_COMPARE(obj) \ OBJECT_CHECK(CompareState, (obj), TYPE_COLO_COMPARE) @@ -77,6 +80,23 @@ static int event_unhandled_count; * |packet | |packet + |packet | |packet + * +--------+ +--------+ +--------+ +--------+ */ + +typedef struct SendCo { +Coroutine *co; +struct CompareState *s; +CharBackend *chr; +GQueue send_list; +bool notify_remote_frame; +bool done; +int ret; +}SendCo ; + +typedef struct SendEntry { +uint32_t size; +uint32_t vnet_hdr_len; +uint8_t *buf; +}SendEntry ; + typedef struct CompareState { Object parent;@@ -91,6 +111,8 @@ typedef struct CompareState {SocketReadState pri_rs;SocketReadState sec_rs;SocketReadState notify_rs; +SendCo out_sendco; +SendCo notify_sendco;bool vnet_hdr;uint32_t compare_timeout;uint32_t expired_scan_cycle;@@ -124,10 +146,11 @@ enum {static int compare_chr_send (CompareState *s, -const uint8_t *buf, +uint8_t *buf,uint32_t size,uint32_t vnet_hdr_len, -bool notify_remote_frame); +bool notify_remote_frame, +bool zero_copy); static bool packet_matches_str(const char *str, const uint8_t *buf, @@ -145,7 +168,7@@ static void
notify_remote_frame(CompareState *s)char msg[] = "DO_CHECKPOINT"; int ret = 0;net/colo-compare.c +5
-3- ret =compare_chr_send (s, (uint8_t *)msg,strlen (msg),0 , true); + ret =compare_chr_send (s, (uint8_t *)msg,strlen (msg),0 , true, + false);if (ret <0 ) {error_report ("Notify Xen COLO-frame failed" ); }@@ -272,12 +295,13 @@ static voidcolo_release_primary_pkt (CompareState *s, Packet *pkt) pkt->data, pkt->size, pkt->vnet_hdr_len, - false); + false, + true);if (ret <0 ) {error_report ("colo send primary packet failed" ); }trace_colo_compare_main ("packet same and release packet" ); -packet_destroy (pkt,NULL ); +packet_destroy_partial (pkt,NULL ); }/*
net/colo-compare.c +87
-37@@ -699,65 +723,115 @@ static void colo_compare_connection(void *opaque, void *user_data) } } -static int compare_chr_send (CompareState *s, -const uint8_t *buf, -uint32_t size, -uint32_t vnet_hdr_len, -bool notify_remote_frame) +static void coroutine_fn _compare_chr_send(void *opaque) { +SendCo *sendco = opaque; +CompareState *s = sendco->s;int ret =0 ; -uint32_t len =htonl (size); -if (!size) { -return 0 ; - } +while (!g_queue_is_empty (&sendco->send_list)) { +SendEntry *entry =g_queue_pop_tail (&sendco->send_list); +uint32_t len =htonl (entry->size); -if (notify_remote_frame) { - ret =qemu_chr_fe_write_all (&s->chr_notify_dev, - (uint8_t *)&len, -sizeof (len)); - }else { - ret =qemu_chr_fe_write_all (&s->chr_out, (uint8_t *)&len,sizeof (len)); - } + ret =qemu_chr_fe_write_all (sendco->chr, (uint8_t *)&len, +sizeof (len)); -if (ret !=sizeof (len)) { - goto err; - } +if (ret !=sizeof (len)) { +g_free (entry->buf); +g_slice_free (SendEntry, entry); + goto err; + } -if (s->vnet_hdr) { -/* -* We send vnet header len make other module(like filter-redirector) -* know how to parse net packet correctly. -*/ - len =htonl (vnet_hdr_len); +if (!sendco->notify_remote_frame && s->vnet_hdr) { +/* +* We send vnet header len make other module(like filter-redirector) +* know how to parse net packet correctly. +*/ + len =htonl (entry->vnet_hdr_len); -if (!notify_remote_frame) { - ret =qemu_chr_fe_write_all (&s->chr_out, + ret =qemu_chr_fe_write_all (sendco->chr, (uint8_t *)&len,sizeof (len)); + +if (ret !=sizeof (len)) { +g_free (entry->buf); +g_slice_free (SendEntry, entry); + goto err; + } } -if (ret !=sizeof (len)) { + ret =qemu_chr_fe_write_all (sendco->chr, + (uint8_t *)entry->buf, + entry->size); + +if (ret != entry->size) { +g_free (entry->buf); +g_slice_free (SendEntry, entry); goto err; } + +g_free (entry->buf); +g_slice_free (SendEntry, entry); } + sendco->ret =0 ; + goto out; + +err: +while (!g_queue_is_empty (&sendco->send_list)) { +SendEntry *entry =g_queue_pop_tail (&sendco->send_list); +g_free (entry->buf); +g_slice_free (SendEntry, entry); + } + sendco->ret = ret <0 ? ret : -EIO ; +out: + sendco->co =NULL ; + sendco->done = true; +aio_wait_kick (); +} + +static int compare_chr_send (CompareState *s, +uint8_t *buf, +uint32_t size, +uint32_t vnet_hdr_len, +bool notify_remote_frame, +bool zero_copy) { +SendCo *sendco; +SendEntry *entry; +if (notify_remote_frame) { - ret =qemu_chr_fe_write_all (&s->chr_notify_dev, - (uint8_t *)buf, - size); + sendco = &s->notify_sendco; }else { - ret =qemu_chr_fe_write_all (&s->chr_out, (uint8_t *)buf, size); + sendco = &s->out_sendco; } -if (ret != size) { - goto err; +if (!size) { +return 0 ; } -return 0 ; + entry =g_slice_new (SendEntry); + entry->size = size; + entry->vnet_hdr_len = vnet_hdr_len; +if (zero_copy) { + entry->buf = buf; + }else { + entry->buf =g_malloc (size); +memcpy (entry->buf, buf, size); + } +g_queue_push_head (&sendco->send_list, entry); + +if (sendco->done) { + sendco->co =qemu_coroutine_create (_compare_chr_send, sendco); + sendco->done = false; +qemu_coroutine_enter (sendco->co); +if (sendco->done) { +/* report early errors */ +return sendco->ret; + } + } -err: -return ret <0 ? ret : -EIO ; +/* assume success */ +return 0 ; }static int
compare_chr_can_read(void *opaque)@@ -1063,6 +1137,7 @@
static voidcompare_pri_rs_finalize(SocketReadState *pri_rs)pri_rs->buf, pri_rs->packet_len, pri_rs->vnet_hdr_len,net/colo-compare.c +7
-1+ false, false); }else { /* compare packet in the specified connection */ @@ -1093,7 +1168,7 @@ static void compare_notify_rs_finalize(SocketReadState +*notify_rs) if (packet_matches_str("COLO_USERSPACE_PROXY_INIT", notify_rs->buf, notify_rs->packet_len)) { - ret =compare_chr_send (s, (uint8_t *)msg,strlen (msg),0 , true); + ret =compare_chr_send (s, (uint8_t *)msg,strlen (msg),0 , + true, false);if (ret <0 ) {error_report ("Notify Xen COLO-frame INIT failed" ); }@@ -1199,6 +1274,18 @@ static void colo_compare_complete(UserCreatable *uc, Error **errp)QTAILQ_INSERT_TAIL (&net_compares, s, next); + s->out_sendco.s = s; + s->out_sendco.chr = &s->chr_out;
net/colo-compare.c static void · line 1289
Quoted 10 lines
net/colo-compare.c +10
-0+ s->out_sendco.notify_remote_frame = false; + s->out_sendco.done = true; +g_queue_init (&s->out_sendco.send_list); + + s->notify_sendco.s = s; + s->notify_sendco.chr = &s->chr_notify_dev; + s->notify_sendco.notify_remote_frame = true; + s->notify_sendco.done = true; +g_queue_init (&s->notify_sendco.send_list); +
No need to init the notify_sendco each time, because the notify dev just an optional parameter. You can use the if (s->notify_dev) here. Just Xen use the chr_notify_dev.
Overall, make the chr_send job to coroutine is a good idea. It looks good for me.
And your patch inspired me, it looks we can re-use the compare_chr_send code on filter mirror/redirector too.
Quoted 18 lines
net/colo-compare.c +0
-0g_queue_init (&s->conn_list);qemu_mutex_init(&event_mtx);net/colo-compare.c +0
-0@@ -1225,8 +1312,9 @@ static void colo_flush_packets(void *opaque,void
*user_data)pkt->data, pkt->size, pkt->vnet_hdr_len,net/colo-compare.c +3
-2- false); -packet_destroy (pkt,NULL ); + false, + true); +packet_destroy_partial (pkt,NULL ); }while (!g_queue_is_empty (&conn->secondary_list)) { pkt =g_queue_pop_head (&conn->secondary_list);
net/colo-compare.c static void colo_compare_finalize(Object *obj) · line 1395
Quoted 7 lines
net/colo-compare.c +4
-0@@ -1301,10 +1389,19 @@ static void colo_compare_finalize(Object *obj) } } +AioContext *ctx =iothread_get_aio_context (s->iothread); +aio_context_acquire (ctx); +AIO_WAIT_WHILE (ctx, !s->out_sendco.done); +AIO_WAIT_WHILE (ctx, !s->notify_sendco.done);
Same as above.
net/colo-compare.c static void colo_compare_finalize(Object *obj) · line 1404
Quoted 9 lines
net/colo-compare.c +5
-0+aio_context_release (ctx); +/* Release all unhandled packets after compare thead exited */ g_queue_foreach (&s->conn_list, colo_flush_packets, s); +AIO_WAIT_WHILE (NULL , !s->out_sendco.done);g_queue_clear (&s->conn_list); +g_queue_clear (&s->out_sendco.send_list); +g_queue_clear (&s->notify_sendco.send_list);
Same as above.
Quoted 34 lines
net/colo-compare.c +0
-0if (s->connection_track_table) {g_hash_table_destroy (s->connection_track_table);net/colo.c +6
-0@@ -185,6 +185,13 @@ void packet_destroy(void *opaque, void *user_data)g_slice_free (Packet, pkt); } +void packet_destroy_partial (void *opaque,void *user_data) { +Packet *pkt = opaque; + +g_slice_free (Packet, pkt); +} +/* * Clear hashtable, stop this hash growing really huge */ net/colo.h +0
-0@@ -102,5 +102,6 @@ bool connection_has_tracked(GHashTable *connection_track_table, void connection_hashtable_reset(GHashTable *connection_track_table); Packet *packet_new(const void *data, intsize, int
vnet_hdr_len); voidpacket_destroy(void *opaque, void *user_data);
+voidpacket_destroy_partial(void *opaque, void *user_data);#endif /*
NET_COLO_H*/
--
2.20.1
RE: [PATCH v4 5/6] net/colo-compare.c, softmmu/vl.c: Check that colo-compare is active
In reply to Lukas StraubQuoted 24 lines
-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Monday, May 4, 2020 6:28 PM
To: qemu-devel <[email protected]>
Cc: Zhang, Chen <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: [PATCH v4 5/6]net/colo-compare.c,softmmu/vl.c: Check that colo- compare is activeIf the colo-compare object is removed before failover and a checkpoint happens, qemu crashes because it tries to lock the destroyed
event_mtxincolo_notify_compares_event.Fix this by checking if everything is initialized by introducing a new variable
colo_compare_activewhich is protected by a new mutexcolo_compare_mutex. The new mutex also protects against concurrent access of thenet_compareslist and makes sure thatcolo_notify_compares_eventisn't active while we destroyevent_mtxandevent_complete_cond.
Quoted 3 lines
With this it also is again possible to use colo without colo-compare (periodic mode) and to use multiple colo-compare for multiple network interfaces.
Hi Lukas,
For this case I think we don't need to touch vl.c code, we can solve this issue from another perspective:
How to remove colo-compare?
User will use qemu-monitor or QMP command to disable an object, so we just need return operation failed When user try to remove colo-compare object while COLO is running.
Thanks
Zhang Chen
Quoted 141 lines
Signed-off-by Lukas Straub <[email protected]>--- net/colo-compare.c | 35 +++++++++++++++++++++++++++++------ net/colo-compare.h | 1 + softmmu/vl.c | 2 ++ 3 files changed, 32 insertions(+), 6 deletions(-)net/colo-compare.c index renamed from net/colo-compare.c +0
-0
56db3d3bfc..c7572d75e9100644net/colo-compare.c +0
-0@@ -54,6 +54,8 @@ static NotifierList colo_compare_notifiers = #define
REGULAR_PACKET_CHECK_MS3000 #defineDEFAULT_TIME_OUT_MS3000net/colo-compare.c +8
-0+static QemuMutex colo_compare_mutex; +static bool colo_compare_active;static QemuMutex event_mtx;static QemuCond event_complete_cond;static int event_unhandled_count;@@ -906,6 +908,12 @@ static void check_old_packet_regular(void *opaque)void colo_notify_compares_event (void *opaque,int event,Error **errp) {CompareState *s; +qemu_mutex_lock (&colo_compare_mutex); + +if (!colo_compare_active) { +qemu_mutex_unlock (&colo_compare_mutex); +return ; + }qemu_mutex_lock (&event_mtx); QTAILQ_FOREACH(s, &net_compares, next) { @@ -919,6 +927,7 @@ voidcolo_notify_compares_event (void *opaque,int event,Error **errp)} qemu_mutex_unlock(&event_mtx);+
qemu_mutex_unlock(&colo_compare_mutex);
}static void
colo_compare_timer_init(CompareState *s)@@ -1274,7 +1283,14
@@ static voidcolo_compare_complete(UserCreatable *uc, Error **errp)s->vnet_hdr); }net/colo-compare.c +7
-3+qemu_mutex_lock (&colo_compare_mutex); +if (!colo_compare_active) { +qemu_mutex_init (&event_mtx); +qemu_cond_init (&event_complete_cond); + colo_compare_active = true; + }QTAILQ_INSERT_TAIL (&net_compares, s, next); +qemu_mutex_unlock (&colo_compare_mutex); s->out_sendco.s = s; s->out_sendco.chr = &s->chr_out;@@ -1290,9 +1306,6 @@ static void colo_compare_complete(UserCreatable *uc, Error **errp)g_queue_init (&s->conn_list); -qemu_mutex_init (&event_mtx); -qemu_cond_init (&event_complete_cond); - s->connection_track_table = g_hash_table_new_full(connection_key_hash,connection_key_equal, g_free, @@ -1384,12 +1397,19 @@ static void
colo_compare_finalize(Object *obj)qemu_bh_delete(s->event_bh);net/colo-compare.c +13
-3+qemu_mutex_lock (&colo_compare_mutex); QTAILQ_FOREACH(tmp, &net_compares, next) {if (tmp == s) {QTAILQ_REMOVE (&net_compares, s, next);break ; } } +if (QTAILQ_EMPTY (&net_compares)) { + colo_compare_active = false; +qemu_mutex_destroy (&event_mtx); +qemu_cond_destroy (&event_complete_cond); + } +qemu_mutex_unlock (&colo_compare_mutex);AioContext *ctx =iothread_get_aio_context (s->iothread);aio_context_acquire (ctx);@@ -1413,15 +1433,18 @@ static void colo_compare_finalize(Object *obj)object_unref (OBJECT (s->iothread)); } -qemu_mutex_destroy (&event_mtx); -qemu_cond_destroy (&event_complete_cond); -g_free (s->pri_indev);g_free (s->sec_indev);g_free (s->outdev);g_free (s->notify_dev); } +void colo_compare_init_globals (void ) +{ + colo_compare_active = false; +qemu_mutex_init (&colo_compare_mutex); +} +static const TypeInfo colo_compare_info = { .name =TYPE_COLO_COMPARE , .parent =TYPE_OBJECT ,net/colo-compare.h index renamed from net/colo-compare.h +0
-0
22ddd512e2..eb483ac586100644net/colo-compare.h +1
-0@@ -17,6 +17,7 @@#ifndef QEMU_COLO_COMPARE_H #define QEMU_COLO_COMPARE_H +void colo_compare_init_globals (void );void colo_notify_compares_event (void *opaque,int event,Error **errp);void
colo_compare_register_notifier(Notifier *notify); voidcolo_compare_unregister_notifier(Notifier *notify); diff --gita/softmmu/vl.cb/softmmu/vl.cindex32c0047889..a913ed5469100644softmmu/vl.c +2
-1@@ -112,6 +112,7 @@#include "qapi/qmp/qerror.h" #include "sysemu/iothread.h" #include "qemu/guest-random.h" +#include "net/colo-compare.h" #define MAX_VIRTIO_CONSOLES 1@@ -2906,6 +2907,7 @@ void qemu_init(int argc, char **argv, char **envp)precopy_infrastructure_init ();postcopy_infrastructure_init ();monitor_init_globals (); +colo_compare_init_globals ();if (qcrypto_init (&err) <0 ) {error_reportf_err (err,"cannot initialize crypto: " ); --2.20.1
RE: [PATCH v4 6/6] net/colo-compare.c: Correct ordering in complete and finalize
In reply to Lukas StraubQuoted 13 lines
-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Monday, May 4, 2020 6:28 PM
To: qemu-devel <[email protected]>
Cc: Zhang, Chen <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: [PATCH v4 6/6]net/colo-compare.c: Correct ordering in complete and finalize
Quoted 4 lines
In
colo_compare_complete, insert CompareState intonet_comparesonly after everything has been initialized.
Incolo_compare_finalize, remove CompareState fromnet_comparesbefore anything is deinitialized.
S/deinitialized/finalized
It looks no dependences on each step on initialization and finalization. Do you means we just need add/remove each colo-compare module at last in logic? Or current code have some issue?
Thanks
Zhang Chen
Quoted 86 lines
Signed-off-by Lukas Straub <[email protected]>--- net/colo-compare.c | 45 +++++++++++++++++++++++---------------------- 1 file changed, 23 insertions(+), 22 deletions(-)net/colo-compare.c index renamed from net/colo-compare.c +0
-0
c7572d75e9..6f80bcece6100644net/colo-compare.c +0
-9@@ -1283,15 +1283,6 @@ static voidcolo_compare_complete (UserCreatable *uc, Error **errp) s->vnet_hdr); } -qemu_mutex_lock (&colo_compare_mutex); -if (!colo_compare_active) { -qemu_mutex_init (&event_mtx); -qemu_cond_init (&event_complete_cond); - colo_compare_active = true; - } -QTAILQ_INSERT_TAIL (&net_compares, s, next); -qemu_mutex_unlock (&colo_compare_mutex); - s->out_sendco.s = s; s->out_sendco.chr = &s->chr_out;s->out_sendco.notify_remote_frame = false; @@ -1312,6 +1303,16 @@ static void colo_compare_complete(UserCreatable *uc, Error **errp) connection_destroy); colo_compare_iothread(s);net/colo-compare.c +23
-14+ +qemu_mutex_lock (&colo_compare_mutex); +if (!colo_compare_active) { +qemu_mutex_init (&event_mtx); +qemu_cond_init (&event_complete_cond); + colo_compare_active = true; + } +QTAILQ_INSERT_TAIL (&net_compares, s, next); +qemu_mutex_unlock (&colo_compare_mutex); +return ; }@@ -1384,19 +1385,6 @@ static void colo_compare_finalize(Object *obj)CompareState *s =COLO_COMPARE (obj);CompareState *tmp =NULL ; -qemu_chr_fe_deinit (&s->chr_pri_in, false); -qemu_chr_fe_deinit (&s->chr_sec_in, false); -qemu_chr_fe_deinit (&s->chr_out, false); -if (s->notify_dev) { -qemu_chr_fe_deinit (&s->chr_notify_dev, false); - } - -if (s->iothread) { -colo_compare_timer_del (s); - } - -qemu_bh_delete (s->event_bh); -qemu_mutex_lock (&colo_compare_mutex); QTAILQ_FOREACH(tmp, &net_compares, next) {if (tmp == s) {@@ -1411,6 +1399,19 @@ static void colo_compare_finalize(Object *obj) }qemu_mutex_unlock (&colo_compare_mutex); +qemu_chr_fe_deinit (&s->chr_pri_in, false); +qemu_chr_fe_deinit (&s->chr_sec_in, false); +qemu_chr_fe_deinit (&s->chr_out, false); +if (s->notify_dev) { +qemu_chr_fe_deinit (&s->chr_notify_dev, false); + } + +if (s->iothread) { +colo_compare_timer_del (s); + } + +qemu_bh_delete (s->event_bh); +AioContext *ctx =iothread_get_aio_context (s->iothread);aio_context_acquire (ctx);AIO_WAIT_WHILE (ctx, !s->out_sendco.done); --2.20.1
Re: [PATCH v4 3/6] net/colo-compare.c: Fix deadlock in compare_chr_send
In reply to Zhang, ChenQuoted 295 lines · Chen
Quoted 295 lines
-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Monday, May 4, 2020 6:28 PM
To: qemu-devel <[email protected]>
Cc: Zhang, Chen <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: [PATCH v4 3/6]net/colo-compare.c: Fix deadlock incompare_chr_sendThe
chr_outchardev is connected to a filter-redirector running in the main loop.qemu_chr_fe_write_allmight block here incompare_chr_sendif the (socket-)buffer is full.
If another filter-redirector in the main loop want's to send data tochr_pri_init might also block if the buffer is full. This leads to a deadlock because both event loops get blocked.Fix this by converting
compare_chr_sendto a coroutine and putting the packets in a send queue.Signed-off-by Lukas Straub <[email protected]>--- net/colo-compare.c | 187 ++++++++++++++++++++++++++++++++++------- ---- net/colo.c | 7 ++ net/colo.h | 1 + 3 files changed, 150 insertions(+), 45 deletions(-)net/colo-compare.c index renamed from net/colo-compare.c +0
-0
1de4220fe2..2a4e7f7c4e100644net/colo-compare.c +26
-2@@ -32,6 +32,9 @@#include "migration/migration.h" #include "util.h" +#include "block/aio-wait.h" +#include "qemu/coroutine.h" +#define TYPE_COLO_COMPARE "colo-compare" #define COLO_COMPARE(obj) \ OBJECT_CHECK(CompareState, (obj), TYPE_COLO_COMPARE) @@ -77,6 +80,23 @@ static int event_unhandled_count; * |packet | |packet + |packet | |packet + * +--------+ +--------+ +--------+ +--------+ */ + +typedef struct SendCo { +Coroutine *co; +struct CompareState *s; +CharBackend *chr; +GQueue send_list; +bool notify_remote_frame; +bool done; +int ret; +}SendCo ; + +typedef struct SendEntry { +uint32_t size; +uint32_t vnet_hdr_len; +uint8_t *buf; +}SendEntry ; + typedef struct CompareState { Object parent;@@ -91,6 +111,8 @@ typedef struct CompareState {SocketReadState pri_rs;SocketReadState sec_rs;SocketReadState notify_rs; +SendCo out_sendco; +SendCo notify_sendco;bool vnet_hdr;uint32_t compare_timeout;uint32_t expired_scan_cycle;@@ -124,10 +146,11 @@ enum {static int compare_chr_send (CompareState *s, -const uint8_t *buf, +uint8_t *buf,uint32_t size,uint32_t vnet_hdr_len, -bool notify_remote_frame); +bool notify_remote_frame, +bool zero_copy); static bool packet_matches_str(const char *str, const uint8_t *buf, @@ -145,7 +168,7@@ static void
notify_remote_frame(CompareState *s)char msg[] = "DO_CHECKPOINT"; int ret = 0;net/colo-compare.c +5
-3- ret =compare_chr_send (s, (uint8_t *)msg,strlen (msg),0 , true); + ret =compare_chr_send (s, (uint8_t *)msg,strlen (msg),0 , true, + false);if (ret <0 ) {error_report ("Notify Xen COLO-frame failed" ); }@@ -272,12 +295,13 @@ static voidcolo_release_primary_pkt (CompareState *s, Packet *pkt) pkt->data, pkt->size, pkt->vnet_hdr_len, - false); + false, + true);if (ret <0 ) {error_report ("colo send primary packet failed" ); }trace_colo_compare_main ("packet same and release packet" ); -packet_destroy (pkt,NULL ); +packet_destroy_partial (pkt,NULL ); }/*
net/colo-compare.c +87
-37@@ -699,65 +723,115 @@ static void colo_compare_connection(void *opaque, void *user_data) } } -static int compare_chr_send (CompareState *s, -const uint8_t *buf, -uint32_t size, -uint32_t vnet_hdr_len, -bool notify_remote_frame) +static void coroutine_fn _compare_chr_send(void *opaque) { +SendCo *sendco = opaque; +CompareState *s = sendco->s;int ret =0 ; -uint32_t len =htonl (size); -if (!size) { -return 0 ; - } +while (!g_queue_is_empty (&sendco->send_list)) { +SendEntry *entry =g_queue_pop_tail (&sendco->send_list); +uint32_t len =htonl (entry->size); -if (notify_remote_frame) { - ret =qemu_chr_fe_write_all (&s->chr_notify_dev, - (uint8_t *)&len, -sizeof (len)); - }else { - ret =qemu_chr_fe_write_all (&s->chr_out, (uint8_t *)&len,sizeof (len)); - } + ret =qemu_chr_fe_write_all (sendco->chr, (uint8_t *)&len, +sizeof (len)); -if (ret !=sizeof (len)) { - goto err; - } +if (ret !=sizeof (len)) { +g_free (entry->buf); +g_slice_free (SendEntry, entry); + goto err; + } -if (s->vnet_hdr) { -/* -* We send vnet header len make other module(like filter-redirector) -* know how to parse net packet correctly. -*/ - len =htonl (vnet_hdr_len); +if (!sendco->notify_remote_frame && s->vnet_hdr) { +/* +* We send vnet header len make other module(like filter-redirector) +* know how to parse net packet correctly. +*/ + len =htonl (entry->vnet_hdr_len); -if (!notify_remote_frame) { - ret =qemu_chr_fe_write_all (&s->chr_out, + ret =qemu_chr_fe_write_all (sendco->chr, (uint8_t *)&len,sizeof (len)); + +if (ret !=sizeof (len)) { +g_free (entry->buf); +g_slice_free (SendEntry, entry); + goto err; + } } -if (ret !=sizeof (len)) { + ret =qemu_chr_fe_write_all (sendco->chr, + (uint8_t *)entry->buf, + entry->size); + +if (ret != entry->size) { +g_free (entry->buf); +g_slice_free (SendEntry, entry); goto err; } + +g_free (entry->buf); +g_slice_free (SendEntry, entry); } + sendco->ret =0 ; + goto out; + +err: +while (!g_queue_is_empty (&sendco->send_list)) { +SendEntry *entry =g_queue_pop_tail (&sendco->send_list); +g_free (entry->buf); +g_slice_free (SendEntry, entry); + } + sendco->ret = ret <0 ? ret : -EIO ; +out: + sendco->co =NULL ; + sendco->done = true; +aio_wait_kick (); +} + +static int compare_chr_send (CompareState *s, +uint8_t *buf, +uint32_t size, +uint32_t vnet_hdr_len, +bool notify_remote_frame, +bool zero_copy) { +SendCo *sendco; +SendEntry *entry; +if (notify_remote_frame) { - ret =qemu_chr_fe_write_all (&s->chr_notify_dev, - (uint8_t *)buf, - size); + sendco = &s->notify_sendco; }else { - ret =qemu_chr_fe_write_all (&s->chr_out, (uint8_t *)buf, size); + sendco = &s->out_sendco; } -if (ret != size) { - goto err; +if (!size) { +return 0 ; } -return 0 ; + entry =g_slice_new (SendEntry); + entry->size = size; + entry->vnet_hdr_len = vnet_hdr_len; +if (zero_copy) { + entry->buf = buf; + }else { + entry->buf =g_malloc (size); +memcpy (entry->buf, buf, size); + } +g_queue_push_head (&sendco->send_list, entry); + +if (sendco->done) { + sendco->co =qemu_coroutine_create (_compare_chr_send, sendco); + sendco->done = false; +qemu_coroutine_enter (sendco->co); +if (sendco->done) { +/* report early errors */ +return sendco->ret; + } + } -err: -return ret <0 ? ret : -EIO ; +/* assume success */ +return 0 ; }static int
compare_chr_can_read(void *opaque)@@ -1063,6 +1137,7 @@
static voidcompare_pri_rs_finalize(SocketReadState *pri_rs)pri_rs->buf, pri_rs->packet_len, pri_rs->vnet_hdr_len,net/colo-compare.c +5
-1+ false, false); }else { /* compare packet in the specified connection */ @@ -1093,7 +1168,7 @@ static void compare_notify_rs_finalize(SocketReadState +*notify_rs) if (packet_matches_str("COLO_USERSPACE_PROXY_INIT", notify_rs->buf, notify_rs->packet_len)) { - ret =compare_chr_send (s, (uint8_t *)msg,strlen (msg),0 , true); + ret =compare_chr_send (s, (uint8_t *)msg,strlen (msg),0 , + true, false);if (ret <0 ) {error_report ("Notify Xen COLO-frame INIT failed" ); }@@ -1199,6 +1274,18 @@ static void colo_compare_complete(UserCreatable *uc, Error **errp)QTAILQ_INSERT_TAIL (&net_compares, s, next);
Quoted 15 lines
net/colo-compare.c static void colo_compare_complete(UserCreatable *uc, Error **errp) · line 1288
Quoted 12 lines
net/colo-compare.c +12
-0+ s->out_sendco.s = s; + s->out_sendco.chr = &s->chr_out; + s->out_sendco.notify_remote_frame = false; + s->out_sendco.done = true; +g_queue_init (&s->out_sendco.send_list); + + s->notify_sendco.s = s; + s->notify_sendco.chr = &s->chr_notify_dev; + s->notify_sendco.notify_remote_frame = true; + s->notify_sendco.done = true; +g_queue_init (&s->notify_sendco.send_list); +No need to init the
notify_sendcoeach time, because the notify dev just an optional parameter. You can use the if (s->notify_dev) here. Just Xen use thechr_notify_dev.
Ok, I will change that and the code below in the next version.
Quoted 2 lines
Overall, make the
chr_sendjob to coroutine is a good idea. It looks good for me.
And your patch inspired me, it looks we can re-use thecompare_chr_sendcode on filter mirror/redirector too.
I already have patch for that, but I don't think it is a good idea, because the guest then can send packets faster than colo-compare can process. This leads bufferbloat and the performance drops in my tests:
Client-to-server tcp:
without patch: ~66 Mbit/s
with patch: ~59 Mbit/s
Server-to-client tcp:
without patch: ~702 Kbit/s
with patch: ~328 Kbit/s
Regards,
Lukas Straub
Quoted 78 lines
Tested-by Zhang Chen <[email protected]>Quoted 18 lines
net/colo-compare.c +0
-0g_queue_init (&s->conn_list);qemu_mutex_init(&event_mtx);net/colo-compare.c +0
-0@@ -1225,8 +1312,9 @@ static void colo_flush_packets(void *opaque,void
*user_data)pkt->data, pkt->size, pkt->vnet_hdr_len,net/colo-compare.c +3
-2- false); -packet_destroy (pkt,NULL ); + false, + true); +packet_destroy_partial (pkt,NULL ); }while (!g_queue_is_empty (&conn->secondary_list)) { pkt =g_queue_pop_head (&conn->secondary_list);net/colo-compare.c static void colo_compare_finalize(Object *obj) · line 1395
Quoted 7 lines
net/colo-compare.c +4
-0@@ -1301,10 +1389,19 @@ static void colo_compare_finalize(Object *obj) } } +AioContext *ctx =iothread_get_aio_context (s->iothread); +aio_context_acquire (ctx); +AIO_WAIT_WHILE (ctx, !s->out_sendco.done); +AIO_WAIT_WHILE (ctx, !s->notify_sendco.done);Same as above.
net/colo-compare.c static void colo_compare_finalize(Object *obj) · line 1404
Quoted 9 lines
net/colo-compare.c +5
-0+aio_context_release (ctx); +/* Release all unhandled packets after compare thead exited */ g_queue_foreach (&s->conn_list, colo_flush_packets, s); +AIO_WAIT_WHILE (NULL , !s->out_sendco.done);g_queue_clear (&s->conn_list); +g_queue_clear (&s->out_sendco.send_list); +g_queue_clear (&s->notify_sendco.send_list);Same as above.
Quoted 34 lines
net/colo-compare.c +0
-0if (s->connection_track_table) {g_hash_table_destroy (s->connection_track_table);net/colo.c +6
-0@@ -185,6 +185,13 @@ void packet_destroy(void *opaque, void *user_data)g_slice_free (Packet, pkt); } +void packet_destroy_partial (void *opaque,void *user_data) { +Packet *pkt = opaque; + +g_slice_free (Packet, pkt); +} +/* * Clear hashtable, stop this hash growing really huge */ net/colo.h +0
-0@@ -102,5 +102,6 @@ bool connection_has_tracked(GHashTable *connection_track_table, void connection_hashtable_reset(GHashTable *connection_track_table); Packet *packet_new(const void *data, intsize, int
vnet_hdr_len); voidpacket_destroy(void *opaque, void *user_data);
+voidpacket_destroy_partial(void *opaque, void *user_data);#endif /*
NET_COLO_H*/
--
2.20.1
Re: [PATCH v4 5/6] net/colo-compare.c, softmmu/vl.c: Check that colo-compare is active
In reply to Zhang, ChenQuoted 26 lines · Chen
Quoted 22 lines
-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Monday, May 4, 2020 6:28 PM
To: qemu-devel <[email protected]>
Cc: Zhang, Chen <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: [PATCH v4 5/6]net/colo-compare.c,softmmu/vl.c: Check that colo- compare is activeIf the colo-compare object is removed before failover and a checkpoint happens, qemu crashes because it tries to lock the destroyed
event_mtxincolo_notify_compares_event.Fix this by checking if everything is initialized by introducing a new variable
colo_compare_activewhich is protected by a new mutexcolo_compare_mutex. The new mutex also protects against concurrent access of thenet_compareslist and makes sure thatcolo_notify_compares_eventisn't active while we destroyevent_mtxandevent_complete_cond.Quoted 3 lines
With this it also is again possible to use colo without colo-compare (periodic mode) and to use multiple colo-compare for multiple network interfaces.
Quoted 6 lines
Hi Lukas,
For this case I think we don't need to touch
vl.ccode, we can solve this issue from another perspective:
How to remove colo-compare?
User will use qemu-monitor or QMP command to disable an object, so we just need return operation failed When user try to remove colo-compare object while COLO is running.
Yeah, but that still leaves the other problem that colo can't be used without colo-compare (qemu crashes then).
Regards,
Lukas Straub
Quoted 145 lines
Thanks
Zhang ChenQuoted 141 lines
Signed-off-by Lukas Straub <[email protected]>--- net/colo-compare.c | 35 +++++++++++++++++++++++++++++------ net/colo-compare.h | 1 + softmmu/vl.c | 2 ++ 3 files changed, 32 insertions(+), 6 deletions(-)net/colo-compare.c index renamed from net/colo-compare.c +0
-0
56db3d3bfc..c7572d75e9100644net/colo-compare.c +0
-0@@ -54,6 +54,8 @@ static NotifierList colo_compare_notifiers = #define
REGULAR_PACKET_CHECK_MS3000 #defineDEFAULT_TIME_OUT_MS3000net/colo-compare.c +8
-0+static QemuMutex colo_compare_mutex; +static bool colo_compare_active;static QemuMutex event_mtx;static QemuCond event_complete_cond;static int event_unhandled_count;@@ -906,6 +908,12 @@ static void check_old_packet_regular(void *opaque)void colo_notify_compares_event (void *opaque,int event,Error **errp) {CompareState *s; +qemu_mutex_lock (&colo_compare_mutex); + +if (!colo_compare_active) { +qemu_mutex_unlock (&colo_compare_mutex); +return ; + }qemu_mutex_lock (&event_mtx); QTAILQ_FOREACH(s, &net_compares, next) { @@ -919,6 +927,7 @@ voidcolo_notify_compares_event (void *opaque,int event,Error **errp)} qemu_mutex_unlock(&event_mtx);+
qemu_mutex_unlock(&colo_compare_mutex);
}static void
colo_compare_timer_init(CompareState *s)@@ -1274,7 +1283,14
@@ static voidcolo_compare_complete(UserCreatable *uc, Error **errp)s->vnet_hdr); }net/colo-compare.c +7
-3+qemu_mutex_lock (&colo_compare_mutex); +if (!colo_compare_active) { +qemu_mutex_init (&event_mtx); +qemu_cond_init (&event_complete_cond); + colo_compare_active = true; + }QTAILQ_INSERT_TAIL (&net_compares, s, next); +qemu_mutex_unlock (&colo_compare_mutex); s->out_sendco.s = s; s->out_sendco.chr = &s->chr_out;@@ -1290,9 +1306,6 @@ static void colo_compare_complete(UserCreatable *uc, Error **errp)g_queue_init (&s->conn_list); -qemu_mutex_init (&event_mtx); -qemu_cond_init (&event_complete_cond); - s->connection_track_table = g_hash_table_new_full(connection_key_hash,connection_key_equal, g_free, @@ -1384,12 +1397,19 @@ static void
colo_compare_finalize(Object *obj)qemu_bh_delete(s->event_bh);net/colo-compare.c +13
-3+qemu_mutex_lock (&colo_compare_mutex); QTAILQ_FOREACH(tmp, &net_compares, next) {if (tmp == s) {QTAILQ_REMOVE (&net_compares, s, next);break ; } } +if (QTAILQ_EMPTY (&net_compares)) { + colo_compare_active = false; +qemu_mutex_destroy (&event_mtx); +qemu_cond_destroy (&event_complete_cond); + } +qemu_mutex_unlock (&colo_compare_mutex);AioContext *ctx =iothread_get_aio_context (s->iothread);aio_context_acquire (ctx);@@ -1413,15 +1433,18 @@ static void colo_compare_finalize(Object *obj)object_unref (OBJECT (s->iothread)); } -qemu_mutex_destroy (&event_mtx); -qemu_cond_destroy (&event_complete_cond); -g_free (s->pri_indev);g_free (s->sec_indev);g_free (s->outdev);g_free (s->notify_dev); } +void colo_compare_init_globals (void ) +{ + colo_compare_active = false; +qemu_mutex_init (&colo_compare_mutex); +} +static const TypeInfo colo_compare_info = { .name =TYPE_COLO_COMPARE , .parent =TYPE_OBJECT ,net/colo-compare.h index renamed from net/colo-compare.h +0
-0
22ddd512e2..eb483ac586100644net/colo-compare.h +1
-0@@ -17,6 +17,7 @@#ifndef QEMU_COLO_COMPARE_H #define QEMU_COLO_COMPARE_H +void colo_compare_init_globals (void );void colo_notify_compares_event (void *opaque,int event,Error **errp);void
colo_compare_register_notifier(Notifier *notify); voidcolo_compare_unregister_notifier(Notifier *notify); diff --gita/softmmu/vl.cb/softmmu/vl.cindex32c0047889..a913ed5469100644softmmu/vl.c +2
-1@@ -112,6 +112,7 @@#include "qapi/qmp/qerror.h" #include "sysemu/iothread.h" #include "qemu/guest-random.h" +#include "net/colo-compare.h" #define MAX_VIRTIO_CONSOLES 1@@ -2906,6 +2907,7 @@ void qemu_init(int argc, char **argv, char **envp)precopy_infrastructure_init ();postcopy_infrastructure_init ();monitor_init_globals (); +colo_compare_init_globals ();if (qcrypto_init (&err) <0 ) {error_reportf_err (err,"cannot initialize crypto: " ); --2.20.1
Re: [PATCH v4 6/6] net/colo-compare.c: Correct ordering in complete and finalize
In reply to Zhang, ChenQuoted 16 lines · Chen
Quoted 11 lines
-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Monday, May 4, 2020 6:28 PM
To: qemu-devel <[email protected]>
Cc: Zhang, Chen <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: [PATCH v4 6/6]net/colo-compare.c: Correct ordering in complete and finalizeQuoted 4 lines
In
colo_compare_complete, insert CompareState intonet_comparesonly after everything has been initialized.
Incolo_compare_finalize, remove CompareState fromnet_comparesbefore anything is deinitialized.
Quoted 4 lines
S/deinitialized/finalized
It looks no dependences on each step on initialization and finalization. Do you means we just need add/remove each colo-compare module at last in logic?
Yes. While I didn't see any crashes here, there is the possibility that if colo-compare is removed during checkpoint, the destroyed event_bh is called from colo_notify_compares_event. Same with colo_compare_complete (very unlikely) if colo-compare is created while colo is running, colo_notify_compares_event may call the uninitialized event_bh.
Regards,
Lukas Straub
Quoted 91 lines
Or current code have some issue?
Thanks
Zhang ChenQuoted 86 lines
Signed-off-by Lukas Straub <[email protected]>--- net/colo-compare.c | 45 +++++++++++++++++++++++---------------------- 1 file changed, 23 insertions(+), 22 deletions(-)net/colo-compare.c index renamed from net/colo-compare.c +0
-0
c7572d75e9..6f80bcece6100644net/colo-compare.c +0
-9@@ -1283,15 +1283,6 @@ static voidcolo_compare_complete (UserCreatable *uc, Error **errp) s->vnet_hdr); } -qemu_mutex_lock (&colo_compare_mutex); -if (!colo_compare_active) { -qemu_mutex_init (&event_mtx); -qemu_cond_init (&event_complete_cond); - colo_compare_active = true; - } -QTAILQ_INSERT_TAIL (&net_compares, s, next); -qemu_mutex_unlock (&colo_compare_mutex); - s->out_sendco.s = s; s->out_sendco.chr = &s->chr_out;s->out_sendco.notify_remote_frame = false; @@ -1312,6 +1303,16 @@ static void colo_compare_complete(UserCreatable *uc, Error **errp) connection_destroy); colo_compare_iothread(s);net/colo-compare.c +23
-14+ +qemu_mutex_lock (&colo_compare_mutex); +if (!colo_compare_active) { +qemu_mutex_init (&event_mtx); +qemu_cond_init (&event_complete_cond); + colo_compare_active = true; + } +QTAILQ_INSERT_TAIL (&net_compares, s, next); +qemu_mutex_unlock (&colo_compare_mutex); +return ; }@@ -1384,19 +1385,6 @@ static void colo_compare_finalize(Object *obj)CompareState *s =COLO_COMPARE (obj);CompareState *tmp =NULL ; -qemu_chr_fe_deinit (&s->chr_pri_in, false); -qemu_chr_fe_deinit (&s->chr_sec_in, false); -qemu_chr_fe_deinit (&s->chr_out, false); -if (s->notify_dev) { -qemu_chr_fe_deinit (&s->chr_notify_dev, false); - } - -if (s->iothread) { -colo_compare_timer_del (s); - } - -qemu_bh_delete (s->event_bh); -qemu_mutex_lock (&colo_compare_mutex); QTAILQ_FOREACH(tmp, &net_compares, next) {if (tmp == s) {@@ -1411,6 +1399,19 @@ static void colo_compare_finalize(Object *obj) }qemu_mutex_unlock (&colo_compare_mutex); +qemu_chr_fe_deinit (&s->chr_pri_in, false); +qemu_chr_fe_deinit (&s->chr_sec_in, false); +qemu_chr_fe_deinit (&s->chr_out, false); +if (s->notify_dev) { +qemu_chr_fe_deinit (&s->chr_notify_dev, false); + } + +if (s->iothread) { +colo_compare_timer_del (s); + } + +qemu_bh_delete (s->event_bh); +AioContext *ctx =iothread_get_aio_context (s->iothread);aio_context_acquire (ctx);AIO_WAIT_WHILE (ctx, !s->out_sendco.done); --2.20.1
RE: [PATCH v4 3/6] net/colo-compare.c: Fix deadlock in compare_chr_send
In reply to Lukas StraubQuoted 341 lines
-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Thursday, May 7, 2020 11:51 PM
To: Zhang, Chen <[email protected]>
Cc: qemu-devel <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: Re: [PATCH v4 3/6]net/colo-compare.c: Fix deadlock incompare_chr_sendQuoted 18 lines · Chen
Quoted 18 lines
-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Monday, May 4, 2020 6:28 PM
To: qemu-devel <[email protected]>
Cc: Zhang, Chen <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: [PATCH v4 3/6]net/colo-compare.c: Fix deadlock incompare_chr_sendThe
chr_outchardev is connected to a filter-redirector running in the main loop.qemu_chr_fe_write_allmight block here incompare_chr_sendif the (socket-)buffer is full.
If another filter-redirector in the main loop want's to send data tochr_pri_init might also block if the buffer is full. This leads to a deadlock because both event loops get blocked.Quoted 6 lines
Quoted 6 lines
Fix this by converting
compare_chr_sendto a coroutine and putting the packets in a send queue.Signed-off-by Lukas Straub <[email protected]>--- net/colo-compare.c | 187 ++++++++++++++++++++++++++++++++++-------
Quoted 13 lines
Quoted 13 lines
----
net/colo.c | 7 ++ net/colo.h | 1 + 3 files changed, 150 insertions(+), 45 deletions(-)net/colo-compare.c index renamed from net/colo-compare.c +0
-0
1de4220fe2..2a4e7f7c4e100644net/colo-compare.c +0
-0@@ -32,6 +32,9 @@#include "migration/migration.h" #include "util.h" Quoted 6 lines
net/colo-compare.c
Quoted 6 lines
net/colo-compare.c +3
-0+#include "block/aio-wait.h" +#include "qemu/coroutine.h" +#define TYPE_COLO_COMPARE "colo-compare"#define COLO_COMPARE (obj) \ OBJECT_CHECK(CompareState, (obj), TYPE_COLO_COMPARE) @@ -77,6
Quoted 98 lines
Quoted 98 lines
+80,23 @@ static int
event_unhandled_count;* |packet | |packet + |packet | |packet + * +--------+ +--------+ +--------+ +--------+*/
net/colo-compare.c +22
-2+ +typedef struct SendCo { +Coroutine *co; +struct CompareState *s; +CharBackend *chr; +GQueue send_list; +bool notify_remote_frame; +bool done; +int ret; +}SendCo ; + +typedef struct SendEntry { +uint32_t size; +uint32_t vnet_hdr_len; +uint8_t *buf; +}SendEntry ; + typedef struct CompareState { Object parent;@@ -91,6 +111,8 @@ typedef struct CompareState {SocketReadState pri_rs;SocketReadState sec_rs;SocketReadState notify_rs; +SendCo out_sendco; +SendCo notify_sendco;bool vnet_hdr;uint32_t compare_timeout;uint32_t expired_scan_cycle;@@ -124,10 +146,11 @@ enum {static int compare_chr_send (CompareState *s, -const uint8_t *buf, +uint8_t *buf,uint32_t size,uint32_t vnet_hdr_len, -bool notify_remote_frame); +bool notify_remote_frame, +bool zero_copy); static bool packet_matches_str(const char *str, const uint8_t *buf, @@ -145,7 +168,7@@ static void
notify_remote_frame(CompareState *s)char msg[] = "DO_CHECKPOINT"; int ret = 0;net/colo-compare.c +5
-3- ret =compare_chr_send (s, (uint8_t *)msg,strlen (msg),0 , true); + ret =compare_chr_send (s, (uint8_t *)msg,strlen (msg),0 , true, + false);if (ret <0 ) {error_report ("Notify Xen COLO-frame failed" ); }@@ -272,12 +295,13 @@ static voidcolo_release_primary_pkt (CompareState *s, Packet *pkt) pkt->data, pkt->size, pkt->vnet_hdr_len, - false); + false, + true);if (ret <0 ) {error_report ("colo send primary packet failed" ); }trace_colo_compare_main ("packet same and release packet" ); -packet_destroy (pkt,NULL ); +packet_destroy_partial (pkt,NULL ); }/*
net/colo-compare.c +6
-9@@ -699,65 +723,115 @@ static void colo_compare_connection(void *opaque, void *user_data) } } -static int compare_chr_send (CompareState *s, -const uint8_t *buf, -uint32_t size, -uint32_t vnet_hdr_len, -bool notify_remote_frame) +static void coroutine_fn _compare_chr_send(void *opaque) { +SendCo *sendco = opaque; +CompareState *s = sendco->s;int ret =0 ; -uint32_t len =htonl (size); -if (!size) { -return 0 ; - } +while (!g_queue_is_empty (&sendco->send_list)) { +SendEntry *entry =g_queue_pop_tail (&sendco->send_list); +uint32_t len =htonl (entry->size);Quoted 6 lines
net/colo-compare.c static void colo_compare_connection(void *opaque, void *user_data)
Quoted 6 lines
net/colo-compare.c +0
-6-if (notify_remote_frame) { - ret =qemu_chr_fe_write_all (&s->chr_notify_dev, - (uint8_t *)&len, -sizeof (len)); - }else { - ret = qemu_chr_fe_write_all(&s->chr_out, (uint8_t *)&len,sizeof(len));
Quoted 13 lines
Quoted 13 lines
net/colo-compare.c +7
-4- } + ret =qemu_chr_fe_write_all (sendco->chr, (uint8_t *)&len, +sizeof (len)); -if (ret !=sizeof (len)) { - goto err; - } +if (ret !=sizeof (len)) { +g_free (entry->buf); +g_slice_free (SendEntry, entry); + goto err; + }Quoted 9 lines
net/colo-compare.c static void colo_compare_connection(void *opaque, void *user_data)
Quoted 9 lines
net/colo-compare.c +3
-6-if (s->vnet_hdr) { -/* -* We send vnet header len make other module(like filter-redirector) -* know how to parse net packet correctly. -*/ - len =htonl (vnet_hdr_len); +if (!sendco->notify_remote_frame && s->vnet_hdr) { +/* +* We send vnet header len make other module(like filter- redirector)
Quoted 73 lines
Quoted 73 lines
net/colo-compare.c +49
-9+* know how to parse net packet correctly. +*/ + len =htonl (entry->vnet_hdr_len); -if (!notify_remote_frame) { - ret =qemu_chr_fe_write_all (&s->chr_out, + ret =qemu_chr_fe_write_all (sendco->chr, (uint8_t *)&len,sizeof (len)); + +if (ret !=sizeof (len)) { +g_free (entry->buf); +g_slice_free (SendEntry, entry); + goto err; + } } -if (ret !=sizeof (len)) { + ret =qemu_chr_fe_write_all (sendco->chr, + (uint8_t *)entry->buf, + entry->size); + +if (ret != entry->size) { +g_free (entry->buf); +g_slice_free (SendEntry, entry); goto err; } + +g_free (entry->buf); +g_slice_free (SendEntry, entry); } + sendco->ret =0 ; + goto out; + +err: +while (!g_queue_is_empty (&sendco->send_list)) { +SendEntry *entry =g_queue_pop_tail (&sendco->send_list); +g_free (entry->buf); +g_slice_free (SendEntry, entry); + } + sendco->ret = ret <0 ? ret : -EIO ; +out: + sendco->co =NULL ; + sendco->done = true; +aio_wait_kick (); +} + +static int compare_chr_send (CompareState *s, +uint8_t *buf, +uint32_t size, +uint32_t vnet_hdr_len, +bool notify_remote_frame, +bool zero_copy) { +SendCo *sendco; +SendEntry *entry; +if (notify_remote_frame) { - ret =qemu_chr_fe_write_all (&s->chr_notify_dev, - (uint8_t *)buf, - size); + sendco = &s->notify_sendco; }else { - ret =qemu_chr_fe_write_all (&s->chr_out, (uint8_t *)buf, size); + sendco = &s->out_sendco; } -if (ret != size) { - goto err; +if (!size) { +return 0 ; }Quoted 14 lines
Quoted 4 lines
net/colo-compare.c +3
-1-return 0 ; + entry =g_slice_new (SendEntry); + entry->size = size; + entry->vnet_hdr_len = vnet_hdr_len;net/colo-compare.c static void colo_compare_connection(void *opaque, void *user_data)
Quoted 10 lines
net/colo-compare.c +10
-0+if (zero_copy) { + entry->buf = buf; + }else { + entry->buf =g_malloc (size); +memcpy (entry->buf, buf, size); + } +g_queue_push_head (&sendco->send_list, entry); + +if (sendco->done) { + sendco->co = qemu_coroutine_create(_compare_chr_send,sendco);
Quoted 8 lines
Quoted 8 lines
net/colo-compare.c +7
-0+ sendco->done = false; +qemu_coroutine_enter (sendco->co); +if (sendco->done) { +/* report early errors */ +return sendco->ret; + } + }Quoted 7 lines
Quoted 7 lines
net/colo-compare.c +2
-2-err: -return ret <0 ? ret : -EIO ; +/* assume success */ +return 0 ; }static int
compare_chr_can_read(void *opaque)@@ -1063,6 +1137,7@@
Quoted 24 lines
Quoted 24 lines
static void
compare_pri_rs_finalize(SocketReadState *pri_rs)pri_rs->buf, pri_rs->packet_len, pri_rs->vnet_hdr_len,net/colo-compare.c +5
-1+ false, false); }else { /* compare packet in the specified connection */ @@ -1093,7 +1168,7 @@ static void compare_notify_rs_finalize(SocketReadState +*notify_rs) if (packet_matches_str("COLO_USERSPACE_PROXY_INIT", notify_rs->buf, notify_rs->packet_len)) { - ret =compare_chr_send (s, (uint8_t *)msg,strlen (msg),0 , true); + ret =compare_chr_send (s, (uint8_t *)msg,strlen (msg),0 , + true, false);if (ret <0 ) {error_report ("Notify Xen COLO-frame INIT failed" ); }@@ -1199,6 +1274,18 @@ static void colo_compare_complete(UserCreatable *uc, Error **errp)QTAILQ_INSERT_TAIL (&net_compares, s, next);Quoted 14 lines
net/colo-compare.c static void colo_compare_complete(UserCreatable *uc, Error **errp) · line 1288
Quoted 12 lines
net/colo-compare.c +12
-0+ s->out_sendco.s = s; + s->out_sendco.chr = &s->chr_out; + s->out_sendco.notify_remote_frame = false; + s->out_sendco.done = true; +g_queue_init (&s->out_sendco.send_list); + + s->notify_sendco.s = s; + s->notify_sendco.chr = &s->chr_notify_dev; + s->notify_sendco.notify_remote_frame = true; + s->notify_sendco.done = true; +g_queue_init (&s->notify_sendco.send_list); +No need to init the
notify_sendcoeach time, because the notify dev justan optional parameter.
Quoted 1 line
You can use the if (
s->notify_dev) here. Just Xen use thechr_notify_dev.Ok, I will change that and the code below in the next version.
Quoted 1 line
Overall, make the
chr_sendjob to coroutine is a good idea. It looks goodfor me.
Quoted 1 line
And your patch inspired me, it looks we can re-use the
compare_chr_sendcode on filter mirror/redirector too.
Quoted 9 lines
I already have patch for that, but I don't think it is a good idea, because the guest then can send packets faster than colo-compare can process. This leads bufferbloat and the performance drops in my tests:
Client-to-server tcp:
without patch: ~66 Mbit/s
with patch: ~59 Mbit/s
Server-to-client tcp:
without patch: ~702 Kbit/s
with patch: ~328 Kbit/s
Oh, a big performance drop, is that caused by memcpy/zero_copy parts ?
Thanks
Zhang Chen
Quoted 83 lines
Regards,
Lukas StraubQuoted 44 lines
Tested-by Zhang Chen <[email protected]>Quoted 18 lines
net/colo-compare.c +0
-0g_queue_init (&s->conn_list);qemu_mutex_init(&event_mtx);net/colo-compare.c +0
-0@@ -1225,8 +1312,9 @@ static void colo_flush_packets(void *opaque,void
*user_data)pkt->data, pkt->size, pkt->vnet_hdr_len,net/colo-compare.c +3
-2- false); -packet_destroy (pkt,NULL ); + false, + true); +packet_destroy_partial (pkt,NULL ); }while (!g_queue_is_empty (&conn->secondary_list)) { pkt =g_queue_pop_head (&conn->secondary_list);net/colo-compare.c static void colo_compare_finalize(Object *obj) · line 1395
Quoted 7 lines
net/colo-compare.c +4
-0@@ -1301,10 +1389,19 @@ static void colo_compare_finalize(Object *obj) } } +AioContext *ctx =iothread_get_aio_context (s->iothread); +aio_context_acquire (ctx); +AIO_WAIT_WHILE (ctx, !s->out_sendco.done); +AIO_WAIT_WHILE (ctx, !s->notify_sendco.done);Same as above.
net/colo-compare.c static void colo_compare_finalize(Object *obj) · line 1404
Quoted 9 lines
net/colo-compare.c +5
-0+aio_context_release (ctx); +/* Release all unhandled packets after compare thead exited */ g_queue_foreach (&s->conn_list, colo_flush_packets, s); +AIO_WAIT_WHILE (NULL , !s->out_sendco.done);g_queue_clear (&s->conn_list); +g_queue_clear (&s->out_sendco.send_list); +g_queue_clear (&s->notify_sendco.send_list);Same as above.
Quoted 1 line
Quoted 7 lines
net/colo.c void packet_destroy(void *opaque, void
Quoted 7 lines
if (s->connection_track_table) { g_hash_table_destroy(s->connection_track_table);net/colo.c index 8196b35837..a6c66d829a renamed from net/colo.c +0
-0100644
net/colo.c +0
-0@@ -185,6 +185,13 @@ void packet_destroy(void *opaque, void
*user_data)Quoted 27 lines
Quoted 26 lines
net/colo.c +6
-0g_slice_free (Packet, pkt); } +void packet_destroy_partial (void *opaque,void *user_data) { +Packet *pkt = opaque; + +g_slice_free (Packet, pkt); +} +/* * Clear hashtable, stop this hash growing really huge */ net/colo.h index 679314b1ca..573ab91785 renamed from net/colo.h +0
-0100644
net/colo.h +0
-0@@ -102,5 +102,6 @@ bool connection_has_tracked(GHashTable *connection_track_table, void connection_hashtable_reset(GHashTable *connection_track_table); Packet *packet_new(const void *data, intsize, int
vnet_hdr_len); voidpacket_destroy(void *opaque, void *user_data);
+voidpacket_destroy_partial(void *opaque, void *user_data);#endif /*
NET_COLO_H*/
--
2.20.1
RE: [PATCH v4 5/6] net/colo-compare.c, softmmu/vl.c: Check that colo-compare is active
In reply to Lukas StraubQuoted 46 lines
-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Thursday, May 7, 2020 11:54 PM
To: Zhang, Chen <[email protected]>
Cc: qemu-devel <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: Re: [PATCH v4 5/6]net/colo-compare.c,softmmu/vl.c: Check that colo-compare is activeQuoted 22 lines · Chen
Quoted 22 lines
-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Monday, May 4, 2020 6:28 PM
To: qemu-devel <[email protected]>
Cc: Zhang, Chen <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: [PATCH v4 5/6]net/colo-compare.c,softmmu/vl.c: Check that colo- compare is activeIf the colo-compare object is removed before failover and a checkpoint happens, qemu crashes because it tries to lock the destroyed
event_mtxincolo_notify_compares_event.Fix this by checking if everything is initialized by introducing a new variable
colo_compare_activewhich is protected by a new mutexcolo_compare_mutex. The new mutex also protects against concurrent access of thenet_compareslist and makes sure thatcolo_notify_compares_eventisn't active while we destroyevent_mtxandevent_complete_cond.Quoted 8 lines
Quoted 4 lines
With this it also is again possible to use colo without colo-compare (periodic
mode) and to use multiple colo-compare for multiple network interfaces.Hi Lukas,
For this case I think we don't need to touch
vl.ccode, we can solve this
Quoted 8 lines
issue from another perspective:
Quoted 3 lines
How to remove colo-compare?
User will use qemu-monitor or QMP command to disable an object, so we just need return operation failed When user try to remove colo-compareobject while COLO is running.
Yeah, but that still leaves the other problem that colo can't be used without colo-compare (qemu crashes then).
Yes, the COLO-compare is necessary module in COLO original design.
At most cases, user need it do dynamic sync.
For rare cases, maybe we can add a new colo-compare parameter to bypass all the network workload.
Thanks
Zhang Chen
Quoted 152 lines
Regards,
Lukas StraubQuoted 10 lines
Thanks
Zhang ChenQuoted 7 lines
Signed-off-by Lukas Straub <[email protected]>--- net/colo-compare.c | 35 +++++++++++++++++++++++++++++------ net/colo-compare.h | 1 + softmmu/vl.c | 2 ++ 3 files changed, 32 insertions(+), 6 deletions(-)Quoted 6 lines
net/colo-compare.c static NotifierList colo_compare_notifiers = · line 54
Quoted 6 lines
net/colo-compare.c index renamed from net/colo-compare.c +0
-0
56db3d3bfc..c7572d75e9100644net/colo-compare.c +0
-0@@ -54,6 +54,8 @@ static NotifierList colo_compare_notifiers =#define REGULAR_PACKET_CHECK_MS 3000 #define
DEFAULT_TIME_OUT_MSQuoted 22 lines
Quoted 22 lines
3000
net/colo-compare.c +2
-0+static QemuMutex colo_compare_mutex;static bool +colo_compare_active;static QemuMutex event_mtx; static QemuCond event_complete_cond; static int
event_unhandled_count; @@ -906,6 +908,12 @@ static voidcheck_old_packet_regular(void *opaque)voidcolo_notify_compares_event(void *opaque, int event, Error **errp) { CompareState *s;net/colo-compare.c +6
-0+qemu_mutex_lock (&colo_compare_mutex); + +if (!colo_compare_active) { +qemu_mutex_unlock (&colo_compare_mutex); +return ; + }qemu_mutex_lock (&event_mtx); QTAILQ_FOREACH(s, &net_compares, next) { @@ -919,6 +927,7 @@void
colo_notify_compares_event(void *opaque, int event, Error **errp)}Quoted 6 lines
Quoted 6 lines
qemu_mutex_unlock(&event_mtx);+
qemu_mutex_unlock(&colo_compare_mutex);
}static void
colo_compare_timer_init(CompareState *s)@@ -1274,7
+1283,14 @@ static void colo_compare_complete(UserCreatable *uc,Error **errp)
Quoted 88 lines
Quoted 88 lines
s->vnet_hdr); }net/colo-compare.c +7
-0+qemu_mutex_lock (&colo_compare_mutex); +if (!colo_compare_active) { +qemu_mutex_init (&event_mtx); +qemu_cond_init (&event_complete_cond); + colo_compare_active = true; + }QTAILQ_INSERT_TAIL (&net_compares, s, next); +qemu_mutex_unlock (&colo_compare_mutex); s->out_sendco.s = s; s->out_sendco.chr = &s->chr_out; @@ -1290,9 +1306,6 @@ staticvoid colo_compare_complete(UserCreatable
*uc, Error **errp)g_queue_init(&s->conn_list);net/colo-compare.c +0
-3-qemu_mutex_init (&event_mtx); -qemu_cond_init (&event_complete_cond); - s->connection_track_table =g_hash_table_new_full(connection_key_hash,
connection_key_equal, g_free, @@-1384,12 +1397,19 @@ static void
colo_compare_finalize(Object *obj)qemu_bh_delete(s->event_bh);net/colo-compare.c +12
-3+qemu_mutex_lock (&colo_compare_mutex); QTAILQ_FOREACH(tmp, &net_compares, next) {if (tmp == s) {QTAILQ_REMOVE (&net_compares, s, next);break ; } } +if (QTAILQ_EMPTY (&net_compares)) { + colo_compare_active = false; +qemu_mutex_destroy (&event_mtx); +qemu_cond_destroy (&event_complete_cond); + } +qemu_mutex_unlock (&colo_compare_mutex);AioContext *ctx =iothread_get_aio_context (s->iothread);aio_context_acquire (ctx);@@ -1413,15 +1433,18 @@ static void colo_compare_finalize(Object *obj)object_unref (OBJECT (s->iothread)); } -qemu_mutex_destroy (&event_mtx); -qemu_cond_destroy (&event_complete_cond); -g_free (s->pri_indev);g_free (s->sec_indev);g_free (s->outdev);g_free (s->notify_dev); } +void colo_compare_init_globals (void ) { + colo_compare_active = false; +qemu_mutex_init (&colo_compare_mutex); +} +static const TypeInfo colo_compare_info = { .name =TYPE_COLO_COMPARE , .parent =TYPE_OBJECT ,net/colo-compare.h index renamed from net/colo-compare.h +0
-0
22ddd512e2..eb483ac586100644net/colo-compare.h +1
-0@@ -17,6 +17,7 @@#ifndef QEMU_COLO_COMPARE_H #define QEMU_COLO_COMPARE_H +void colo_compare_init_globals (void );void colo_notify_compares_event (void *opaque,int event,Error **errp);void colo_compare_register_notifier (Notifier *notify);void
colo_compare_unregister_notifier(Notifier *notify); diff --gita/softmmu/vl.cb/softmmu/vl.cindex32c0047889..a913ed5469100644softmmu/vl.c +1
-0@@ -112,6 +112,7 @@#include "qapi/qmp/qerror.h" #include "sysemu/iothread.h" #include "qemu/guest-random.h" +#include "net/colo-compare.h" Quoted 3 lines
softmmu/vl.c void qemu_init(int argc, char **argv, char
Quoted 3 lines
softmmu/vl.c +0
-0#define MAX_VIRTIO_CONSOLES 1@@ -2906,6 +2907,7 @@ void qemu_init(int argc, char **argv, char**envp)
Quoted 10 lines
Quoted 9 lines
softmmu/vl.c +1
-1precopy_infrastructure_init ();postcopy_infrastructure_init ();monitor_init_globals (); +colo_compare_init_globals ();if (qcrypto_init (&err) <0 ) {error_reportf_err (err,"cannot initialize crypto: " ); --2.20.1
RE: [PATCH v4 6/6] net/colo-compare.c: Correct ordering in complete and finalize
In reply to Lukas StraubQuoted 37 lines
-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Friday, May 8, 2020 12:09 AM
To: Zhang, Chen <[email protected]>
Cc: qemu-devel <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: Re: [PATCH v4 6/6]net/colo-compare.c: Correct ordering in complete and finalizeQuoted 11 lines · Chen
Quoted 11 lines
-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Monday, May 4, 2020 6:28 PM
To: qemu-devel <[email protected]>
Cc: Zhang, Chen <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: [PATCH v4 6/6]net/colo-compare.c: Correct ordering in complete and finalizeQuoted 1 line
Quoted 1 line
In
colo_compare_complete, insert CompareState intonet_comparesonly
Quoted 8 lines
Quoted 3 lines
after everything has been initialized.
Incolo_compare_finalize, remove CompareState fromnet_comparesbefore anything is deinitialized.S/deinitialized/finalized
It looks no dependences on each step on initialization and finalization. Do you means we just need add/remove each colo-compare module at last
Quoted 7 lines
in logic?
Yes. While I didn't see any crashes here, there is the possibility that if colo- compare is removed during checkpoint, the destroyed
event_bhis called fromcolo_notify_compares_event. Same withcolo_compare_complete(very unlikely) if colo-compare is created while colo is running,colo_notify_compares_eventmay call the uninitializedevent_bh.
As we discussed on 5/6, remove colo-compare during checkpoint is an illegal operation.
Thanks
Zhang Chen
Quoted 97 lines
Regards,
Lukas StraubQuoted 93 lines
Or current code have some issue?
Thanks
Zhang ChenQuoted 88 lines
Signed-off-by Lukas Straub <[email protected]>--- net/colo-compare.c | 45 +++++++++++++++++++++++---------------------- 1 file changed, 23 insertions(+), 22 deletions(-)net/colo-compare.c index renamed from net/colo-compare.c +0
-0
c7572d75e9..6f80bcece6100644net/colo-compare.c +0
-9@@ -1283,15 +1283,6 @@ static voidcolo_compare_complete (UserCreatable *uc, Error **errp) s->vnet_hdr); } -qemu_mutex_lock (&colo_compare_mutex); -if (!colo_compare_active) { -qemu_mutex_init (&event_mtx); -qemu_cond_init (&event_complete_cond); - colo_compare_active = true; - } -QTAILQ_INSERT_TAIL (&net_compares, s, next); -qemu_mutex_unlock (&colo_compare_mutex); - s->out_sendco.s = s; s->out_sendco.chr = &s->chr_out;s->out_sendco.notify_remote_frame = false; @@ -1312,6 +1303,16@@ static void
colo_compare_complete(UserCreatable *uc, Error **errp)
connection_destroy);colo_compare_iothread(s);net/colo-compare.c +23
-14+ +qemu_mutex_lock (&colo_compare_mutex); +if (!colo_compare_active) { +qemu_mutex_init (&event_mtx); +qemu_cond_init (&event_complete_cond); + colo_compare_active = true; + } +QTAILQ_INSERT_TAIL (&net_compares, s, next); +qemu_mutex_unlock (&colo_compare_mutex); +return ; }@@ -1384,19 +1385,6 @@ static void colo_compare_finalize(Object *obj)CompareState *s =COLO_COMPARE (obj);CompareState *tmp =NULL ; -qemu_chr_fe_deinit (&s->chr_pri_in, false); -qemu_chr_fe_deinit (&s->chr_sec_in, false); -qemu_chr_fe_deinit (&s->chr_out, false); -if (s->notify_dev) { -qemu_chr_fe_deinit (&s->chr_notify_dev, false); - } - -if (s->iothread) { -colo_compare_timer_del (s); - } - -qemu_bh_delete (s->event_bh); -qemu_mutex_lock (&colo_compare_mutex); QTAILQ_FOREACH(tmp, &net_compares, next) {if (tmp == s) {@@ -1411,6 +1399,19 @@ static void colo_compare_finalize(Object *obj) }qemu_mutex_unlock (&colo_compare_mutex); +qemu_chr_fe_deinit (&s->chr_pri_in, false); +qemu_chr_fe_deinit (&s->chr_sec_in, false); +qemu_chr_fe_deinit (&s->chr_out, false); +if (s->notify_dev) { +qemu_chr_fe_deinit (&s->chr_notify_dev, false); + } + +if (s->iothread) { +colo_compare_timer_del (s); + } + +qemu_bh_delete (s->event_bh); +AioContext *ctx =iothread_get_aio_context (s->iothread);aio_context_acquire (ctx);AIO_WAIT_WHILE (ctx, !s->out_sendco.done); --2.20.1
Re: [PATCH v4 5/6] net/colo-compare.c, softmmu/vl.c: Check that colo-compare is active
In reply to Zhang, ChenQuoted 50 lines · Chen
Quoted 50 lines
-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Thursday, May 7, 2020 11:54 PM
To: Zhang, Chen <[email protected]>
Cc: qemu-devel <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: Re: [PATCH v4 5/6]net/colo-compare.c,softmmu/vl.c: Check that colo-compare is activeQuoted 22 lines · Chen
Quoted 22 lines
-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Monday, May 4, 2020 6:28 PM
To: qemu-devel <[email protected]>
Cc: Zhang, Chen <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: [PATCH v4 5/6]net/colo-compare.c,softmmu/vl.c: Check that colo- compare is activeIf the colo-compare object is removed before failover and a checkpoint happens, qemu crashes because it tries to lock the destroyed
event_mtxincolo_notify_compares_event.Fix this by checking if everything is initialized by introducing a new variable
colo_compare_activewhich is protected by a new mutexcolo_compare_mutex. The new mutex also protects against concurrent access of thenet_compareslist and makes sure thatcolo_notify_compares_eventisn't active while we destroyevent_mtxandevent_complete_cond.Quoted 8 lines
Quoted 4 lines
With this it also is again possible to use colo without colo-compare (periodic
mode) and to use multiple colo-compare for multiple network interfaces.Hi Lukas,
For this case I think we don't need to touch
vl.ccode, we can solve thisissue from another perspective:
Quoted 3 lines
How to remove colo-compare?
User will use qemu-monitor or QMP command to disable an object, so we just need return operation failed When user try to remove colo-compare
Quoted 8 lines
Quoted 4 lines
object while COLO is running.
Yeah, but that still leaves the other problem that colo can't be used without colo-compare (qemu crashes then).
Yes, the COLO-compare is necessary module in COLO original design.
At most cases, user need it do dynamic sync.
For rare cases, maybe we can add a new colo-compare parameter to bypass all the network workload.
Hi, Chen
In our application, we only need "periodical mode" because of the performance issue, and have internal patch now. Is it OK to send the internal patch for review?
Thanks.
Regards,
Derek
Quoted 157 lines
Thanks
Zhang ChenQuoted 152 lines
Regards,
Lukas StraubQuoted 10 lines
Thanks
Zhang ChenQuoted 7 lines
Signed-off-by Lukas Straub <[email protected]>--- net/colo-compare.c | 35 +++++++++++++++++++++++++++++------ net/colo-compare.h | 1 + softmmu/vl.c | 2 ++ 3 files changed, 32 insertions(+), 6 deletions(-)Quoted 6 lines
net/colo-compare.c static NotifierList colo_compare_notifiers = · line 54
Quoted 6 lines
net/colo-compare.c index renamed from net/colo-compare.c +0
-0
56db3d3bfc..c7572d75e9100644net/colo-compare.c +0
-0@@ -54,6 +54,8 @@ static NotifierList colo_compare_notifiers =#define REGULAR_PACKET_CHECK_MS 3000 #define
DEFAULT_TIME_OUT_MSQuoted 22 lines
Quoted 22 lines
3000
net/colo-compare.c +2
-0+static QemuMutex colo_compare_mutex;static bool +colo_compare_active;static QemuMutex event_mtx; static QemuCond event_complete_cond; static int
event_unhandled_count; @@ -906,6 +908,12 @@ static voidcheck_old_packet_regular(void *opaque)voidcolo_notify_compares_event(void *opaque, int event, Error **errp) { CompareState *s;net/colo-compare.c +6
-0+qemu_mutex_lock (&colo_compare_mutex); + +if (!colo_compare_active) { +qemu_mutex_unlock (&colo_compare_mutex); +return ; + }qemu_mutex_lock (&event_mtx); QTAILQ_FOREACH(s, &net_compares, next) { @@ -919,6 +927,7 @@void
colo_notify_compares_event(void *opaque, int event, Error **errp)}Quoted 6 lines
Quoted 6 lines
qemu_mutex_unlock(&event_mtx);+
qemu_mutex_unlock(&colo_compare_mutex);
}static void
colo_compare_timer_init(CompareState *s)@@ -1274,7
+1283,14 @@ static void colo_compare_complete(UserCreatable *uc,Error **errp)
Quoted 88 lines
Quoted 88 lines
s->vnet_hdr); }net/colo-compare.c +7
-0+qemu_mutex_lock (&colo_compare_mutex); +if (!colo_compare_active) { +qemu_mutex_init (&event_mtx); +qemu_cond_init (&event_complete_cond); + colo_compare_active = true; + }QTAILQ_INSERT_TAIL (&net_compares, s, next); +qemu_mutex_unlock (&colo_compare_mutex); s->out_sendco.s = s; s->out_sendco.chr = &s->chr_out; @@ -1290,9 +1306,6 @@ staticvoid colo_compare_complete(UserCreatable
*uc, Error **errp)g_queue_init(&s->conn_list);net/colo-compare.c +0
-3-qemu_mutex_init (&event_mtx); -qemu_cond_init (&event_complete_cond); - s->connection_track_table =g_hash_table_new_full(connection_key_hash,
connection_key_equal, g_free, @@-1384,12 +1397,19 @@ static void
colo_compare_finalize(Object *obj)qemu_bh_delete(s->event_bh);net/colo-compare.c +12
-3+qemu_mutex_lock (&colo_compare_mutex); QTAILQ_FOREACH(tmp, &net_compares, next) {if (tmp == s) {QTAILQ_REMOVE (&net_compares, s, next);break ; } } +if (QTAILQ_EMPTY (&net_compares)) { + colo_compare_active = false; +qemu_mutex_destroy (&event_mtx); +qemu_cond_destroy (&event_complete_cond); + } +qemu_mutex_unlock (&colo_compare_mutex);AioContext *ctx =iothread_get_aio_context (s->iothread);aio_context_acquire (ctx);@@ -1413,15 +1433,18 @@ static void colo_compare_finalize(Object *obj)object_unref (OBJECT (s->iothread)); } -qemu_mutex_destroy (&event_mtx); -qemu_cond_destroy (&event_complete_cond); -g_free (s->pri_indev);g_free (s->sec_indev);g_free (s->outdev);g_free (s->notify_dev); } +void colo_compare_init_globals (void ) { + colo_compare_active = false; +qemu_mutex_init (&colo_compare_mutex); +} +static const TypeInfo colo_compare_info = { .name =TYPE_COLO_COMPARE , .parent =TYPE_OBJECT ,net/colo-compare.h index renamed from net/colo-compare.h +0
-0
22ddd512e2..eb483ac586100644net/colo-compare.h +1
-0@@ -17,6 +17,7 @@#ifndef QEMU_COLO_COMPARE_H #define QEMU_COLO_COMPARE_H +void colo_compare_init_globals (void );void colo_notify_compares_event (void *opaque,int event,Error **errp);void colo_compare_register_notifier (Notifier *notify);void
colo_compare_unregister_notifier(Notifier *notify); diff --gita/softmmu/vl.cb/softmmu/vl.cindex32c0047889..a913ed5469100644softmmu/vl.c +1
-0@@ -112,6 +112,7 @@#include "qapi/qmp/qerror.h" #include "sysemu/iothread.h" #include "qemu/guest-random.h" +#include "net/colo-compare.h" Quoted 3 lines
softmmu/vl.c void qemu_init(int argc, char **argv, char
Quoted 3 lines
softmmu/vl.c +0
-0#define MAX_VIRTIO_CONSOLES 1@@ -2906,6 +2907,7 @@ void qemu_init(int argc, char **argv, char**envp)
Quoted 10 lines
Quoted 9 lines
softmmu/vl.c +1
-1precopy_infrastructure_init ();postcopy_infrastructure_init ();monitor_init_globals (); +colo_compare_init_globals ();if (qcrypto_init (&err) <0 ) {error_reportf_err (err,"cannot initialize crypto: " ); --2.20.1
Re: [PATCH v4 3/6] net/colo-compare.c: Fix deadlock in compare_chr_send
In reply to Zhang, ChenQuoted 21 lines · Chen
Quoted 11 lines
Quoted 1 line
No need to init the
notify_sendcoeach time, because the notify dev justan optional parameter.
Quoted 1 line
You can use the if (
s->notify_dev) here. Just Xen use thechr_notify_dev.Ok, I will change that and the code below in the next version.
Quoted 1 line
Overall, make the
chr_sendjob to coroutine is a good idea. It looks goodfor me.
Quoted 1 line
And your patch inspired me, it looks we can re-use the
compare_chr_sendcode on filter mirror/redirector too.
Quoted 9 lines
I already have patch for that, but I don't think it is a good idea, because the guest then can send packets faster than colo-compare can process. This leads bufferbloat and the performance drops in my tests:
Client-to-server tcp:
without patch: ~66 Mbit/s
with patch: ~59 Mbit/s
Server-to-client tcp:
without patch: ~702 Kbit/s
with patch: ~328 Kbit/s
Quoted 4 lines
Oh, a big performance drop, is that caused by memcpy/zero_copy parts ?
Thanks
Zhang Chen
No, there is no memcpy overhead with this patch, see below.
Regards,
Lukas Straub
--- net/filter-mirror.c | 142 +++++++++++++++++++++++++++++++++----------- 1 file changed, 106 insertions(+), 36 deletions(-)
net/filter-mirror.c +106 -36
@@ -20,6 +20,8 @@#include "chardev/char-fe.h" #include "qemu/iov.h" #include "qemu/sockets.h" +#include "block/aio-wait.h" +#include "qemu/coroutine.h" #define FILTER_MIRROR (obj) \ OBJECT_CHECK(MirrorState, (obj), TYPE_FILTER_MIRROR)
@@ -31,6 +33,18 @@#define TYPE_FILTER_REDIRECTOR "filter-redirector"#define REDIRECTOR_MAX_LEN NET_BUFSIZE +typedef struct SendCo { +Coroutine *co; +GQueue send_list; +bool done; +int ret; +}SendCo ; + +typedef struct SendEntry { +ssize_t size; +uint8_t buf[]; +}SendEntry ; + typedef struct MirrorState { NetFilterState parent_obj; char *indev;
@@ -38,59 +52,101 @@ typedef struct MirrorState {CharBackend chr_in;CharBackend chr_out;SocketReadState rs; +SendCo sendco;bool vnet_hdr; } MirrorState; -static int filter_send (MirrorState *s, -const struct iovec *iov, -int iovcnt) +static void coroutine_fn _filter_send(void *opaque) { +MirrorState *s = opaque; +SendCo *sendco = &s->sendco;NetFilterState *nf =NETFILTER (s);int ret =0 ; -ssize_t size =0 ; -uint32_t len =0 ; -char *buf; - - size =iov_size (iov, iovcnt); -if (!size) { -return 0 ; - } - len =htonl (size); - ret =qemu_chr_fe_write_all (&s->chr_out, (uint8_t *)&len,sizeof (len)); -if (ret !=sizeof (len)) { - goto err; - } +while (!g_queue_is_empty (&sendco->send_list)) { +SendEntry *entry =g_queue_pop_tail (&sendco->send_list); +uint32_t len =htonl (entry->size); -if (s->vnet_hdr) { -/* -* If vnet_hdr = on, we send vnet header len to make other -* module(like colo-compare) know how to parse net -* packet correctly. -*/ -ssize_t vnet_hdr_len; + ret =qemu_chr_fe_write_all (&s->chr_out, + (uint8_t *)&len, +sizeof (len)); +if (ret !=sizeof (len)) { +g_free (entry); + goto err; + } - vnet_hdr_len = nf->netdev->vnet_hdr_len; +if (s->vnet_hdr) { +/* +* If vnet_hdr = on, we send vnet header len to make other +* module(like colo-compare) know how to parse net +* packet correctly. +*/ + + len =htonl (nf->netdev->vnet_hdr_len); + ret =qemu_chr_fe_write_all (&s->chr_out, + (uint8_t *)&len, +sizeof (len)); +if (ret !=sizeof (len)) { +g_free (entry); + goto err; + } + } - len =htonl (vnet_hdr_len); - ret =qemu_chr_fe_write_all (&s->chr_out, (uint8_t *)&len,sizeof (len)); -if (ret !=sizeof (len)) { + ret =qemu_chr_fe_write_all (&s->chr_out, + (uint8_t *)entry->buf, + entry->size); +if (ret != entry->size) { +g_free (entry); goto err; } - } - buf =g_malloc (size); -iov_to_buf (iov, iovcnt,0 , buf, size); - ret =qemu_chr_fe_write_all (&s->chr_out, (uint8_t *)buf, size); -g_free (buf); -if (ret != size) { - goto err; +g_free (entry); } -return 0 ; + sendco->ret =0 ; + goto out; err: -return ret <0 ? ret : -EIO ; +while (!g_queue_is_empty (&sendco->send_list)) { +SendEntry *entry =g_queue_pop_tail (&sendco->send_list); +g_free (entry); + } + sendco->ret = ret <0 ? ret : -EIO ; +out: + sendco->co =NULL ; + sendco->done = true; +aio_wait_kick (); +} + +static int filter_send (MirrorState *s, +const struct iovec *iov, +int iovcnt) +{ +SendCo *sendco = &s->sendco; +SendEntry *entry; + +ssize_t size =iov_size (iov, iovcnt); +if (!size) { +return 0 ; + } + + entry =g_malloc (sizeof (SendEntry) + size); + entry->size = size; +iov_to_buf (iov, iovcnt,0 , entry->buf, size); +g_queue_push_head (&sendco->send_list, entry); + +if (sendco->done) { + sendco->co =qemu_coroutine_create (_filter_send, s); + sendco->done = false; +qemu_coroutine_enter (sendco->co); +if (sendco->done) { +/* report early errors */ +return sendco->ret; + } + } + +/* assume success */ +return 0 ; } static void redirector_to_filter(NetFilterState *nf,
@@ -194,6 +250,10 @@ static void filter_mirror_cleanup(NetFilterState *nf) {MirrorState *s =FILTER_MIRROR (nf); +AIO_WAIT_WHILE (NULL , !s->sendco.done); + +g_queue_clear (&s->sendco.send_list); +qemu_chr_fe_deinit (&s->chr_out, false); }
@@ -201,6 +261,10 @@ static void filter_redirector_cleanup(NetFilterState *nf) {MirrorState *s =FILTER_REDIRECTOR (nf); +AIO_WAIT_WHILE (NULL , !s->sendco.done); + +g_queue_clear (&s->sendco.send_list); +qemu_chr_fe_deinit (&s->chr_in, false);qemu_chr_fe_deinit (&s->chr_out, false); }
@@ -224,6 +288,9 @@ static void filter_mirror_setup(NetFilterState *nf, Error **errp) }qemu_chr_fe_init (&s->chr_out, chr, errp); + + s->sendco.done = true; +g_queue_init (&s->sendco.send_list); }static void redirector_rs_finalize (SocketReadState *rs)
@@ -281,6 +348,9 @@ static void filter_redirector_setup(NetFilterState *nf, Error **errp)return ; } } + + s->sendco.done = true; +g_queue_init (&s->sendco.send_list); }static void filter_mirror_class_init (ObjectClass *oc,void *data)
Signature · 1 line
2.20.1
Re: [PATCH v4 5/6] net/colo-compare.c, softmmu/vl.c: Check that colo-compare is active
In reply to Zhang, ChenQuoted 48 lines · Chen
Quoted 48 lines
-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Thursday, May 7, 2020 11:54 PM
To: Zhang, Chen <[email protected]>
Cc: qemu-devel <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: Re: [PATCH v4 5/6]net/colo-compare.c,softmmu/vl.c: Check that colo-compare is activeQuoted 22 lines · Chen
Quoted 22 lines
-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Monday, May 4, 2020 6:28 PM
To: qemu-devel <[email protected]>
Cc: Zhang, Chen <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: [PATCH v4 5/6]net/colo-compare.c,softmmu/vl.c: Check that colo- compare is activeIf the colo-compare object is removed before failover and a checkpoint happens, qemu crashes because it tries to lock the destroyed
event_mtxincolo_notify_compares_event.Fix this by checking if everything is initialized by introducing a new variable
colo_compare_activewhich is protected by a new mutexcolo_compare_mutex. The new mutex also protects against concurrent access of thenet_compareslist and makes sure thatcolo_notify_compares_eventisn't active while we destroyevent_mtxandevent_complete_cond.Quoted 8 lines
Quoted 4 lines
With this it also is again possible to use colo without colo-compare (periodic
mode) and to use multiple colo-compare for multiple network interfaces.Hi Lukas,
For this case I think we don't need to touch
vl.ccode, we can solve thisissue from another perspective:
Quoted 3 lines
How to remove colo-compare?
User will use qemu-monitor or QMP command to disable an object, so we just need return operation failed When user try to remove colo-compare
Quoted 8 lines
Quoted 4 lines
object while COLO is running.
Yeah, but that still leaves the other problem that colo can't be used without colo-compare (qemu crashes then).
Yes, the COLO-compare is necessary module in COLO original design.
At most cases, user need it do dynamic sync.
For rare cases, maybe we can add a new colo-compare parameter to bypass all the network workload.
I think such an parameter would only be a workaround instead of a real solution like this patch.
Regards,
Lukas Straub
Quoted 156 lines
Thanks
Zhang ChenQuoted 152 lines
Regards,
Lukas StraubQuoted 10 lines
Thanks
Zhang ChenQuoted 7 lines
Signed-off-by Lukas Straub <[email protected]>--- net/colo-compare.c | 35 +++++++++++++++++++++++++++++------ net/colo-compare.h | 1 + softmmu/vl.c | 2 ++ 3 files changed, 32 insertions(+), 6 deletions(-)Quoted 6 lines
net/colo-compare.c static NotifierList colo_compare_notifiers = · line 54
Quoted 6 lines
net/colo-compare.c index renamed from net/colo-compare.c +0
-0
56db3d3bfc..c7572d75e9100644net/colo-compare.c +0
-0@@ -54,6 +54,8 @@ static NotifierList colo_compare_notifiers =#define REGULAR_PACKET_CHECK_MS 3000 #define
DEFAULT_TIME_OUT_MSQuoted 22 lines
Quoted 22 lines
3000
net/colo-compare.c +2
-0+static QemuMutex colo_compare_mutex;static bool +colo_compare_active;static QemuMutex event_mtx; static QemuCond event_complete_cond; static int
event_unhandled_count; @@ -906,6 +908,12 @@ static voidcheck_old_packet_regular(void *opaque)voidcolo_notify_compares_event(void *opaque, int event, Error **errp) { CompareState *s;net/colo-compare.c +6
-0+qemu_mutex_lock (&colo_compare_mutex); + +if (!colo_compare_active) { +qemu_mutex_unlock (&colo_compare_mutex); +return ; + }qemu_mutex_lock (&event_mtx); QTAILQ_FOREACH(s, &net_compares, next) { @@ -919,6 +927,7 @@void
colo_notify_compares_event(void *opaque, int event, Error **errp)}Quoted 6 lines
Quoted 6 lines
qemu_mutex_unlock(&event_mtx);+
qemu_mutex_unlock(&colo_compare_mutex);
}static void
colo_compare_timer_init(CompareState *s)@@ -1274,7
+1283,14 @@ static void colo_compare_complete(UserCreatable *uc,Error **errp)
Quoted 88 lines
Quoted 88 lines
s->vnet_hdr); }net/colo-compare.c +7
-0+qemu_mutex_lock (&colo_compare_mutex); +if (!colo_compare_active) { +qemu_mutex_init (&event_mtx); +qemu_cond_init (&event_complete_cond); + colo_compare_active = true; + }QTAILQ_INSERT_TAIL (&net_compares, s, next); +qemu_mutex_unlock (&colo_compare_mutex); s->out_sendco.s = s; s->out_sendco.chr = &s->chr_out; @@ -1290,9 +1306,6 @@ staticvoid colo_compare_complete(UserCreatable
*uc, Error **errp)g_queue_init(&s->conn_list);net/colo-compare.c +0
-3-qemu_mutex_init (&event_mtx); -qemu_cond_init (&event_complete_cond); - s->connection_track_table =g_hash_table_new_full(connection_key_hash,
connection_key_equal, g_free, @@-1384,12 +1397,19 @@ static void
colo_compare_finalize(Object *obj)qemu_bh_delete(s->event_bh);net/colo-compare.c +12
-3+qemu_mutex_lock (&colo_compare_mutex); QTAILQ_FOREACH(tmp, &net_compares, next) {if (tmp == s) {QTAILQ_REMOVE (&net_compares, s, next);break ; } } +if (QTAILQ_EMPTY (&net_compares)) { + colo_compare_active = false; +qemu_mutex_destroy (&event_mtx); +qemu_cond_destroy (&event_complete_cond); + } +qemu_mutex_unlock (&colo_compare_mutex);AioContext *ctx =iothread_get_aio_context (s->iothread);aio_context_acquire (ctx);@@ -1413,15 +1433,18 @@ static void colo_compare_finalize(Object *obj)object_unref (OBJECT (s->iothread)); } -qemu_mutex_destroy (&event_mtx); -qemu_cond_destroy (&event_complete_cond); -g_free (s->pri_indev);g_free (s->sec_indev);g_free (s->outdev);g_free (s->notify_dev); } +void colo_compare_init_globals (void ) { + colo_compare_active = false; +qemu_mutex_init (&colo_compare_mutex); +} +static const TypeInfo colo_compare_info = { .name =TYPE_COLO_COMPARE , .parent =TYPE_OBJECT ,net/colo-compare.h index renamed from net/colo-compare.h +0
-0
22ddd512e2..eb483ac586100644net/colo-compare.h +1
-0@@ -17,6 +17,7 @@#ifndef QEMU_COLO_COMPARE_H #define QEMU_COLO_COMPARE_H +void colo_compare_init_globals (void );void colo_notify_compares_event (void *opaque,int event,Error **errp);void colo_compare_register_notifier (Notifier *notify);void
colo_compare_unregister_notifier(Notifier *notify); diff --gita/softmmu/vl.cb/softmmu/vl.cindex32c0047889..a913ed5469100644softmmu/vl.c +1
-0@@ -112,6 +112,7 @@#include "qapi/qmp/qerror.h" #include "sysemu/iothread.h" #include "qemu/guest-random.h" +#include "net/colo-compare.h" Quoted 3 lines
softmmu/vl.c void qemu_init(int argc, char **argv, char
Quoted 3 lines
softmmu/vl.c +0
-0#define MAX_VIRTIO_CONSOLES 1@@ -2906,6 +2907,7 @@ void qemu_init(int argc, char **argv, char**envp)
Quoted 10 lines
Quoted 9 lines
softmmu/vl.c +1
-1precopy_infrastructure_init ();postcopy_infrastructure_init ();monitor_init_globals (); +colo_compare_init_globals ();if (qcrypto_init (&err) <0 ) {error_reportf_err (err,"cannot initialize crypto: " ); --2.20.1
RE: [PATCH v4 3/6] net/colo-compare.c: Fix deadlock in compare_chr_send
In reply to Lukas StraubQuoted 40 lines
-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Friday, May 8, 2020 2:08 PM
To: Zhang, Chen <[email protected]>
Cc: qemu-devel <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: Re: [PATCH v4 3/6]net/colo-compare.c: Fix deadlock incompare_chr_sendQuoted 4 lines · Chen
Quoted 4 lines
Quoted 2 lines
No need to init the
notify_sendcoeach time, because the notify dev justan optional parameter.
Quoted 1 line
You can use the if (
s->notify_dev) here. Just Xen use the
chr_notify_dev.Quoted 20 lines
Quoted 19 lines
Ok, I will change that and the code below in the next version.
Quoted 2 lines
Overall, make the
chr_sendjob to coroutine is a good idea. It looks goodfor me.
Quoted 2 lines
And your patch inspired me, it looks we can re-use the
compare_chr_sendcode on filter mirror/redirector too.
I already have patch for that, but I don't think it is a good idea, because the guest then can send packets faster than colo-compare can process. This leads bufferbloat and the performance drops in my tests:
Client-to-server tcp:
without patch: ~66 Mbit/s
with patch: ~59 Mbit/s
Server-to-client tcp:
without patch: ~702 Kbit/s
with patch: ~328 Kbit/s
Quoted 6 lines
Quoted 4 lines
Oh, a big performance drop, is that caused by memcpy/zero_copy parts ?
Thanks
Zhang ChenNo, there is no memcpy overhead with this patch, see below.
I means for the filter mirror/redirector parts why coroutine will lead huge performance drop?
Thanks
Zhang Chen
Quoted 218 lines
Regards,
Lukas Straub--- net/filter-mirror.c | 142 +++++++++++++++++++++++++++++++++----------- 1 file changed, 106 insertions(+), 36 deletions(-)net/filter-mirror.c index renamed from net/filter-mirror.c +0
-0
d83e815545..6bcd317502100644net/filter-mirror.c +93
-36@@ -20,6 +20,8 @@#include "chardev/char-fe.h" #include "qemu/iov.h" #include "qemu/sockets.h" +#include "block/aio-wait.h" +#include "qemu/coroutine.h" #define FILTER_MIRROR (obj) \ OBJECT_CHECK(MirrorState, (obj), TYPE_FILTER_MIRROR) @@ -31,6 +33,18 @@ #define TYPE_FILTER_REDIRECTOR "filter-redirector"#define REDIRECTOR_MAX_LEN NET_BUFSIZE +typedef struct SendCo { +Coroutine *co; +GQueue send_list; +bool done; +int ret; +}SendCo ; + +typedef struct SendEntry { +ssize_t size; +uint8_t buf[]; +}SendEntry ; + typedef struct MirrorState { NetFilterState parent_obj; char *indev;@@ -38,59 +52,101 @@ typedef struct MirrorState {CharBackend chr_in;CharBackend chr_out;SocketReadState rs; +SendCo sendco;bool vnet_hdr; } MirrorState; -static int filter_send (MirrorState *s, -const struct iovec *iov, -int iovcnt) +static void coroutine_fn _filter_send(void *opaque) { +MirrorState *s = opaque; +SendCo *sendco = &s->sendco;NetFilterState *nf =NETFILTER (s);int ret =0 ; -ssize_t size =0 ; -uint32_t len =0 ; -char *buf; - - size =iov_size (iov, iovcnt); -if (!size) { -return 0 ; - } - len =htonl (size); - ret =qemu_chr_fe_write_all (&s->chr_out, (uint8_t *)&len,sizeof (len)); -if (ret !=sizeof (len)) { - goto err; - } +while (!g_queue_is_empty (&sendco->send_list)) { +SendEntry *entry =g_queue_pop_tail (&sendco->send_list); +uint32_t len =htonl (entry->size); -if (s->vnet_hdr) { -/* -* If vnet_hdr = on, we send vnet header len to make other -* module(like colo-compare) know how to parse net -* packet correctly. -*/ -ssize_t vnet_hdr_len; + ret =qemu_chr_fe_write_all (&s->chr_out, + (uint8_t *)&len, +sizeof (len)); +if (ret !=sizeof (len)) { +g_free (entry); + goto err; + } - vnet_hdr_len = nf->netdev->vnet_hdr_len; +if (s->vnet_hdr) { +/* +* If vnet_hdr = on, we send vnet header len to make other +* module(like colo-compare) know how to parse net +* packet correctly. +*/ + + len =htonl (nf->netdev->vnet_hdr_len); + ret =qemu_chr_fe_write_all (&s->chr_out, + (uint8_t *)&len, +sizeof (len)); +if (ret !=sizeof (len)) { +g_free (entry); + goto err; + } + } - len =htonl (vnet_hdr_len); - ret =qemu_chr_fe_write_all (&s->chr_out, (uint8_t *)&len,sizeof (len)); -if (ret !=sizeof (len)) { + ret =qemu_chr_fe_write_all (&s->chr_out, + (uint8_t *)entry->buf, + entry->size); +if (ret != entry->size) { +g_free (entry); goto err; } - } - buf =g_malloc (size); -iov_to_buf (iov, iovcnt,0 , buf, size); - ret =qemu_chr_fe_write_all (&s->chr_out, (uint8_t *)buf, size); -g_free (buf); -if (ret != size) { - goto err; +g_free (entry); } -return 0 ; + sendco->ret =0 ; + goto out; err: -return ret <0 ? ret : -EIO ; +while (!g_queue_is_empty (&sendco->send_list)) { +SendEntry *entry =g_queue_pop_tail (&sendco->send_list); +g_free (entry); + } + sendco->ret = ret <0 ? ret : -EIO ; +out: + sendco->co =NULL ; + sendco->done = true; +aio_wait_kick (); +} + +static int filter_send (MirrorState *s, +const struct iovec *iov, +int iovcnt) +{ +SendCo *sendco = &s->sendco; +SendEntry *entry; + +ssize_t size =iov_size (iov, iovcnt); +if (!size) { +return 0 ; + } + + entry =g_malloc (sizeof (SendEntry) + size); + entry->size = size; +iov_to_buf (iov, iovcnt,0 , entry->buf, size); +g_queue_push_head (&sendco->send_list, entry); + +if (sendco->done) { + sendco->co =qemu_coroutine_create (_filter_send, s); + sendco->done = false; +qemu_coroutine_enter (sendco->co); +if (sendco->done) { +/* report early errors */ +return sendco->ret; + } + } + +/* assume success */ +return 0 ; } static void redirector_to_filter(NetFilterState *nf, @@ -194,6 +250,10 @@static void filter_mirror_cleanup(NetFilterState *nf) { MirrorState *s = FILTER_MIRROR(nf);net/filter-mirror.c +8
-0+AIO_WAIT_WHILE (NULL , !s->sendco.done); + +g_queue_clear (&s->sendco.send_list); +qemu_chr_fe_deinit (&s->chr_out, false); }@@ -201,6 +261,10 @@ static void filter_redirector_cleanup(NetFilterState *nf) {MirrorState *s =FILTER_REDIRECTOR (nf); +AIO_WAIT_WHILE (NULL , !s->sendco.done); + +g_queue_clear (&s->sendco.send_list); +qemu_chr_fe_deinit (&s->chr_in, false); qemu_chr_fe_deinit(&s->chr_out, false); } @@ -224,6 +288,9 @@ staticvoid filter_mirror_setup (NetFilterState *nf,Error **errp)} qemu_chr_fe_init(&s->chr_out, chr, errp);net/filter-mirror.c +3
-0+ + s->sendco.done = true; +g_queue_init (&s->sendco.send_list); } static void redirector_rs_finalize(SocketReadState *rs) @@ -281,6 +348,9@@ static void
filter_redirector_setup(NetFilterState *nf, Error **errp)return; } }net/filter-mirror.c +3
-1+ + s->sendco.done = true; +g_queue_init (&s->sendco.send_list); }static void filter_mirror_class_init (ObjectClass *oc,void *data) --2.20.1
RE: [PATCH v4 5/6] net/colo-compare.c, softmmu/vl.c: Check that colo-compare is active
In reply to Lukas StraubQuoted 73 lines
-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Friday, May 8, 2020 2:11 PM
To: Zhang, Chen <[email protected]>
Cc: qemu-devel <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: Re: [PATCH v4 5/6]net/colo-compare.c,softmmu/vl.c: Check that colo-compare is activeQuoted 37 lines · Chen
Quoted 37 lines
-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Thursday, May 7, 2020 11:54 PM
To: Zhang, Chen <[email protected]>
Cc: qemu-devel <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: Re: [PATCH v4 5/6]net/colo-compare.c,softmmu/vl.c: Check that colo-compare is activeQuoted 23 lines · Chen
Quoted 23 lines
-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Monday, May 4, 2020 6:28 PM
To: qemu-devel <[email protected]>
Cc: Zhang, Chen <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: [PATCH v4 5/6]net/colo-compare.c,softmmu/vl.c: Check that
colo- compare is activeIf the colo-compare object is removed before failover and a checkpoint happens, qemu crashes because it tries to lock the destroyed
event_mtxincolo_notify_compares_event.Fix this by checking if everything is initialized by introducing a new variable
colo_compare_activewhich is protected by a new mutexcolo_compare_mutex. The new mutex also protects against concurrent access of thenet_compareslist and makes sure thatcolo_notify_compares_eventisn't active while we destroyevent_mtxandevent_complete_cond.Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
With this it also is again possible to use colo without colo-compare (periodic
mode) and to use multiple colo-compare for multiple networkinterfaces.
Quoted 16 lines
Quoted 15 lines
Quoted 6 lines
Quoted 1 line
Hi Lukas,
For this case I think we don't need to touch
vl.ccode, we can solve thisissue from another perspective:
Quoted 4 lines
How to remove colo-compare?
User will use qemu-monitor or QMP command to disable an object, so we just need return operation failed When user try to remove colo-compareobject while COLO is running.
Yeah, but that still leaves the other problem that colo can't be used without colo-compare (qemu crashes then).
Quoted 7 lines
Quoted 3 lines
Yes, the COLO-compare is necessary module in COLO original design. At most cases, user need it do dynamic sync.
For rare cases, maybe we can add a new colo-compare parameter tobypass all the network workload.
I think such an parameter would only be a workaround instead of a real solution like this patch.
The root problem is why COLO-compare is necessary.
Yes, maybe someone want to use pure periodic synchronization mode,
But it means it will lost all guest network support(without colo-compare/filter-mirror/filter-redirector/filter-rewriter). The secondary guest just a solid backup for the primary guest, when occur failover the new build stateful connection (like TCP) will crashed, need userspace to handle this status. It lost the original meaning for COLO FT/HA solution, no need use do HA in application layer. it looks like normal/remote periodic VM snapshot here.
Dave or Jason have any comments here?
Thanks
Zhang Chen
Quoted 166 lines
Regards,
Lukas StraubQuoted 26 lines
Thanks
Zhang ChenQuoted 23 lines
Regards,
Lukas StraubQuoted 10 lines
Thanks
Zhang ChenQuoted 7 lines
Signed-off-by Lukas Straub <[email protected]>--- net/colo-compare.c | 35 +++++++++++++++++++++++++++++------ net/colo-compare.h | 1 + softmmu/vl.c | 2 ++ 3 files changed, 32 insertions(+), 6 deletions(-)Quoted 6 lines
net/colo-compare.c static NotifierList colo_compare_notifiers = · line 54
Quoted 6 lines
net/colo-compare.c index renamed from net/colo-compare.c +0
-0
56db3d3bfc..c7572d75e9100644net/colo-compare.c +0
-0@@ -54,6 +54,8 @@ static NotifierList colo_compare_notifiers =#define REGULAR_PACKET_CHECK_MS 3000 #define
DEFAULT_TIME_OUT_MSQuoted 2 lines
Quoted 2 lines
3000
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
net/colo-compare.c +2
-0+static QemuMutex colo_compare_mutex;static bool +colo_compare_active;static QemuMutex event_mtx; static QemuCond event_complete_cond; static int
event_unhandled_count; @@ -906,6 +908,12 @@ static voidcheck_old_packet_regular(void *opaque)voidcolo_notify_compares_event(void *opaque, int event, Error **errp){
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
CompareState *s;net/colo-compare.c +6
-0+qemu_mutex_lock (&colo_compare_mutex); + +if (!colo_compare_active) { +qemu_mutex_unlock (&colo_compare_mutex); +return ; + }qemu_mutex_lock (&event_mtx); QTAILQ_FOREACH(s, &net_compares, next) { @@ -919,6 +927,7@@ void colo_notify_compares_event(void *opaque, int event,
Error **errp)
Quoted 55 lines
Quoted 55 lines
Quoted 8 lines
Quoted 8 lines
} qemu_mutex_unlock(&event_mtx);+
qemu_mutex_unlock(&colo_compare_mutex);
}static void
colo_compare_timer_init(CompareState *s)@@ -1274,7
+1283,14 @@ static void colo_compare_complete(UserCreatable *uc,Error **errp)
Quoted 46 lines
Quoted 46 lines
s->vnet_hdr); }net/colo-compare.c +7
-0+qemu_mutex_lock (&colo_compare_mutex); +if (!colo_compare_active) { +qemu_mutex_init (&event_mtx); +qemu_cond_init (&event_complete_cond); + colo_compare_active = true; + }QTAILQ_INSERT_TAIL (&net_compares, s, next); +qemu_mutex_unlock (&colo_compare_mutex); s->out_sendco.s = s; s->out_sendco.chr = &s->chr_out; @@ -1290,9 +1306,6 @@static void colo_compare_complete(UserCreatable
*uc, Error **errp)g_queue_init(&s->conn_list);net/colo-compare.c +0
-3-qemu_mutex_init (&event_mtx); -qemu_cond_init (&event_complete_cond); - s->connection_track_table =g_hash_table_new_full(connection_key_hash,
connection_key_equal, g_free,@@
-1384,12 +1397,19 @@ static voidcolo_compare_finalize(Object *obj)qemu_bh_delete(s->event_bh);net/colo-compare.c +7
-0+qemu_mutex_lock (&colo_compare_mutex); QTAILQ_FOREACH(tmp, &net_compares, next) {if (tmp == s) {QTAILQ_REMOVE (&net_compares, s, next);break ; } } +if (QTAILQ_EMPTY (&net_compares)) { + colo_compare_active = false; +qemu_mutex_destroy (&event_mtx); +qemu_cond_destroy (&event_complete_cond); + } +qemu_mutex_unlock (&colo_compare_mutex);Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
net/colo-compare.c static void colo_compare_finalize(Object
Quoted 3 lines
AioContext *ctx = iothread_get_aio_context(s->iothread); aio_context_acquire(ctx);net/colo-compare.c +0
-0@@ -1413,15 +1433,18 @@ static void colo_compare_finalize(Object*obj)
Quoted 57 lines
Quoted 56 lines
Quoted 42 lines
Quoted 42 lines
net/colo-compare.c +5
-3object_unref (OBJECT (s->iothread)); } -qemu_mutex_destroy (&event_mtx); -qemu_cond_destroy (&event_complete_cond); -g_free (s->pri_indev);g_free (s->sec_indev);g_free (s->outdev);g_free (s->notify_dev); } +void colo_compare_init_globals (void ) { + colo_compare_active = false; +qemu_mutex_init (&colo_compare_mutex); +} +static const TypeInfo colo_compare_info = { .name =TYPE_COLO_COMPARE , .parent =TYPE_OBJECT ,net/colo-compare.h index renamed from net/colo-compare.h +0
-0
22ddd512e2..eb483ac586100644net/colo-compare.h +1
-0@@ -17,6 +17,7 @@#ifndef QEMU_COLO_COMPARE_H #define QEMU_COLO_COMPARE_H +void colo_compare_init_globals (void );void colo_notify_compares_event (void *opaque,int event,Error **errp);void colo_compare_register_notifier (Notifier *notify);void
colo_compare_unregister_notifier(Notifier *notify); diff --gita/softmmu/vl.cb/softmmu/vl.cindex32c0047889..a913ed5469100644softmmu/vl.c +1
-0@@ -112,6 +112,7 @@#include "qapi/qmp/qerror.h" #include "sysemu/iothread.h" #include "qemu/guest-random.h" +#include "net/colo-compare.h" Quoted 3 lines
softmmu/vl.c void qemu_init(int argc, char **argv, char
Quoted 3 lines
softmmu/vl.c +0
-0#define MAX_VIRTIO_CONSOLES 1@@ -2906,6 +2907,7 @@ void qemu_init(int argc, char **argv, char**envp)
Quoted 10 lines
Quoted 9 lines
softmmu/vl.c +1
-1precopy_infrastructure_init ();postcopy_infrastructure_init ();monitor_init_globals (); +colo_compare_init_globals ();if (qcrypto_init (&err) <0 ) {error_reportf_err (err,"cannot initialize crypto: " ); --2.20.1
Re: [PATCH v4 3/6] net/colo-compare.c: Fix deadlock in compare_chr_send
In reply to Zhang, ChenQuoted 38 lines · Chen
Quoted 38 lines
-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Friday, May 8, 2020 2:08 PM
To: Zhang, Chen <[email protected]>
Cc: qemu-devel <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: Re: [PATCH v4 3/6]net/colo-compare.c: Fix deadlock incompare_chr_sendQuoted 4 lines · Chen
Quoted 4 lines
Quoted 2 lines
No need to init the
notify_sendcoeach time, because the notify dev justan optional parameter.
Quoted 1 line
You can use the if (
s->notify_dev) here. Just Xen use the
chr_notify_dev.Quoted 20 lines
Quoted 19 lines
Ok, I will change that and the code below in the next version.
Quoted 2 lines
Overall, make the
chr_sendjob to coroutine is a good idea. It looks goodfor me.
Quoted 2 lines
And your patch inspired me, it looks we can re-use the
compare_chr_sendcode on filter mirror/redirector too.
I already have patch for that, but I don't think it is a good idea, because the guest then can send packets faster than colo-compare can process. This leads bufferbloat and the performance drops in my tests:
Client-to-server tcp:
without patch: ~66 Mbit/s
with patch: ~59 Mbit/s
Server-to-client tcp:
without patch: ~702 Kbit/s
with patch: ~328 Kbit/s
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Oh, a big performance drop, is that caused by memcpy/zero_copy parts ?
Thanks
Zhang ChenNo, there is no memcpy overhead with this patch, see below.
I means for the filter mirror/redirector parts why coroutine will lead huge performance drop?
It's because having a additional buffer before the network bottleneck (colo-compare in our case) confuses TCP's congestion-control:
TCP will speed up the data transfer until packets start to drop (or the network interface is blocked). This feedback has to be quick so TCP can select a suitable transfer speed. But with the patch, the guest will fill the buffer as fast as it can (it does not "see" the slow bandwidth of colo-compare behind the buffer) until it it hits against the TCP congestion window. At this point TCP drastically reduces its transfer speed and it stays low because the full buffer delays the packets so it doesn't receive ACK's so it can't speed up the transfer again. Until the buffer is empty again (can take up to a second in my tests). Then this cycle repeats.
Regards,
Lukas Straub
Quoted 222 lines
Thanks
Zhang ChenQuoted 218 lines
Regards,
Lukas Straub--- net/filter-mirror.c | 142 +++++++++++++++++++++++++++++++++----------- 1 file changed, 106 insertions(+), 36 deletions(-)net/filter-mirror.c index renamed from net/filter-mirror.c +0
-0
d83e815545..6bcd317502100644net/filter-mirror.c +93
-36@@ -20,6 +20,8 @@#include "chardev/char-fe.h" #include "qemu/iov.h" #include "qemu/sockets.h" +#include "block/aio-wait.h" +#include "qemu/coroutine.h" #define FILTER_MIRROR (obj) \ OBJECT_CHECK(MirrorState, (obj), TYPE_FILTER_MIRROR) @@ -31,6 +33,18 @@ #define TYPE_FILTER_REDIRECTOR "filter-redirector"#define REDIRECTOR_MAX_LEN NET_BUFSIZE +typedef struct SendCo { +Coroutine *co; +GQueue send_list; +bool done; +int ret; +}SendCo ; + +typedef struct SendEntry { +ssize_t size; +uint8_t buf[]; +}SendEntry ; + typedef struct MirrorState { NetFilterState parent_obj; char *indev;@@ -38,59 +52,101 @@ typedef struct MirrorState {CharBackend chr_in;CharBackend chr_out;SocketReadState rs; +SendCo sendco;bool vnet_hdr; } MirrorState; -static int filter_send (MirrorState *s, -const struct iovec *iov, -int iovcnt) +static void coroutine_fn _filter_send(void *opaque) { +MirrorState *s = opaque; +SendCo *sendco = &s->sendco;NetFilterState *nf =NETFILTER (s);int ret =0 ; -ssize_t size =0 ; -uint32_t len =0 ; -char *buf; - - size =iov_size (iov, iovcnt); -if (!size) { -return 0 ; - } - len =htonl (size); - ret =qemu_chr_fe_write_all (&s->chr_out, (uint8_t *)&len,sizeof (len)); -if (ret !=sizeof (len)) { - goto err; - } +while (!g_queue_is_empty (&sendco->send_list)) { +SendEntry *entry =g_queue_pop_tail (&sendco->send_list); +uint32_t len =htonl (entry->size); -if (s->vnet_hdr) { -/* -* If vnet_hdr = on, we send vnet header len to make other -* module(like colo-compare) know how to parse net -* packet correctly. -*/ -ssize_t vnet_hdr_len; + ret =qemu_chr_fe_write_all (&s->chr_out, + (uint8_t *)&len, +sizeof (len)); +if (ret !=sizeof (len)) { +g_free (entry); + goto err; + } - vnet_hdr_len = nf->netdev->vnet_hdr_len; +if (s->vnet_hdr) { +/* +* If vnet_hdr = on, we send vnet header len to make other +* module(like colo-compare) know how to parse net +* packet correctly. +*/ + + len =htonl (nf->netdev->vnet_hdr_len); + ret =qemu_chr_fe_write_all (&s->chr_out, + (uint8_t *)&len, +sizeof (len)); +if (ret !=sizeof (len)) { +g_free (entry); + goto err; + } + } - len =htonl (vnet_hdr_len); - ret =qemu_chr_fe_write_all (&s->chr_out, (uint8_t *)&len,sizeof (len)); -if (ret !=sizeof (len)) { + ret =qemu_chr_fe_write_all (&s->chr_out, + (uint8_t *)entry->buf, + entry->size); +if (ret != entry->size) { +g_free (entry); goto err; } - } - buf =g_malloc (size); -iov_to_buf (iov, iovcnt,0 , buf, size); - ret =qemu_chr_fe_write_all (&s->chr_out, (uint8_t *)buf, size); -g_free (buf); -if (ret != size) { - goto err; +g_free (entry); } -return 0 ; + sendco->ret =0 ; + goto out; err: -return ret <0 ? ret : -EIO ; +while (!g_queue_is_empty (&sendco->send_list)) { +SendEntry *entry =g_queue_pop_tail (&sendco->send_list); +g_free (entry); + } + sendco->ret = ret <0 ? ret : -EIO ; +out: + sendco->co =NULL ; + sendco->done = true; +aio_wait_kick (); +} + +static int filter_send (MirrorState *s, +const struct iovec *iov, +int iovcnt) +{ +SendCo *sendco = &s->sendco; +SendEntry *entry; + +ssize_t size =iov_size (iov, iovcnt); +if (!size) { +return 0 ; + } + + entry =g_malloc (sizeof (SendEntry) + size); + entry->size = size; +iov_to_buf (iov, iovcnt,0 , entry->buf, size); +g_queue_push_head (&sendco->send_list, entry); + +if (sendco->done) { + sendco->co =qemu_coroutine_create (_filter_send, s); + sendco->done = false; +qemu_coroutine_enter (sendco->co); +if (sendco->done) { +/* report early errors */ +return sendco->ret; + } + } + +/* assume success */ +return 0 ; } static void redirector_to_filter(NetFilterState *nf, @@ -194,6 +250,10 @@static void filter_mirror_cleanup(NetFilterState *nf) { MirrorState *s = FILTER_MIRROR(nf);net/filter-mirror.c +8
-0+AIO_WAIT_WHILE (NULL , !s->sendco.done); + +g_queue_clear (&s->sendco.send_list); +qemu_chr_fe_deinit (&s->chr_out, false); }@@ -201,6 +261,10 @@ static void filter_redirector_cleanup(NetFilterState *nf) {MirrorState *s =FILTER_REDIRECTOR (nf); +AIO_WAIT_WHILE (NULL , !s->sendco.done); + +g_queue_clear (&s->sendco.send_list); +qemu_chr_fe_deinit (&s->chr_in, false); qemu_chr_fe_deinit(&s->chr_out, false); } @@ -224,6 +288,9 @@ staticvoid filter_mirror_setup (NetFilterState *nf,Error **errp)} qemu_chr_fe_init(&s->chr_out, chr, errp);net/filter-mirror.c +3
-0+ + s->sendco.done = true; +g_queue_init (&s->sendco.send_list); } static void redirector_rs_finalize(SocketReadState *rs) @@ -281,6 +348,9@@ static void
filter_redirector_setup(NetFilterState *nf, Error **errp)return; } }net/filter-mirror.c +3
-1+ + s->sendco.done = true; +g_queue_init (&s->sendco.send_list); }static void filter_mirror_class_init (ObjectClass *oc,void *data) --2.20.1
Re: [PATCH v4 5/6] net/colo-compare.c, softmmu/vl.c: Check that colo-compare is active
In reply to Zhang, ChenQuoted 79 lines · Chen
Quoted 71 lines
-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Friday, May 8, 2020 2:11 PM
To: Zhang, Chen <[email protected]>
Cc: qemu-devel <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: Re: [PATCH v4 5/6]net/colo-compare.c,softmmu/vl.c: Check that colo-compare is activeQuoted 37 lines · Chen
Quoted 37 lines
-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Thursday, May 7, 2020 11:54 PM
To: Zhang, Chen <[email protected]>
Cc: qemu-devel <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: Re: [PATCH v4 5/6]net/colo-compare.c,softmmu/vl.c: Check that colo-compare is activeQuoted 23 lines · Chen
Quoted 23 lines
-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Monday, May 4, 2020 6:28 PM
To: qemu-devel <[email protected]>
Cc: Zhang, Chen <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: [PATCH v4 5/6]net/colo-compare.c,softmmu/vl.c: Check that
colo- compare is activeIf the colo-compare object is removed before failover and a checkpoint happens, qemu crashes because it tries to lock the destroyed
event_mtxincolo_notify_compares_event.Fix this by checking if everything is initialized by introducing a new variable
colo_compare_activewhich is protected by a new mutexcolo_compare_mutex. The new mutex also protects against concurrent access of thenet_compareslist and makes sure thatcolo_notify_compares_eventisn't active while we destroyevent_mtxandevent_complete_cond.Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
With this it also is again possible to use colo without colo-compare (periodic
mode) and to use multiple colo-compare for multiple networkinterfaces.
Quoted 16 lines
Quoted 15 lines
Quoted 6 lines
Quoted 1 line
Hi Lukas,
For this case I think we don't need to touch
vl.ccode, we can solve thisissue from another perspective:
Quoted 4 lines
How to remove colo-compare?
User will use qemu-monitor or QMP command to disable an object, so we just need return operation failed When user try to remove colo-compareobject while COLO is running.
Yeah, but that still leaves the other problem that colo can't be used without colo-compare (qemu crashes then).
Quoted 7 lines
Quoted 3 lines
Yes, the COLO-compare is necessary module in COLO original design. At most cases, user need it do dynamic sync.
For rare cases, maybe we can add a new colo-compare parameter tobypass all the network workload.
I think such an parameter would only be a workaround instead of a real solution like this patch.
Quoted 6 lines
The root problem is why COLO-compare is necessary.
Yes, maybe someone want to use pure periodic synchronization mode,
But it means it will lost all guest network support(without colo-compare/filter-mirror/filter-redirector/filter-rewriter). The secondary guest just a solid backup for the primary guest, when occur failover the new build stateful connection (like TCP) will crashed, need userspace to handle this status. It lost the original meaning for COLO FT/HA solution, no need use do HA in application layer. it looks like normal/remote periodic VM snapshot here.
Sure, but maybe the user doesn't need (reliable) network on failover. Also proper network support with periodic mode can easily be implemented by modifying filter-buffer to buffer packets until checkpoint. Being able to use colo without colo-compare gives more flexibility to the user.
Regards,
Lukas Straub
Quoted 172 lines
Dave or Jason have any comments here?
Thanks
Zhang ChenQuoted 166 lines
Regards,
Lukas StraubQuoted 26 lines
Thanks
Zhang ChenQuoted 23 lines
Regards,
Lukas StraubQuoted 10 lines
Thanks
Zhang ChenQuoted 7 lines
Signed-off-by Lukas Straub <[email protected]>--- net/colo-compare.c | 35 +++++++++++++++++++++++++++++------ net/colo-compare.h | 1 + softmmu/vl.c | 2 ++ 3 files changed, 32 insertions(+), 6 deletions(-)Quoted 6 lines
net/colo-compare.c static NotifierList colo_compare_notifiers = · line 54
Quoted 6 lines
net/colo-compare.c index renamed from net/colo-compare.c +0
-0
56db3d3bfc..c7572d75e9100644net/colo-compare.c +0
-0@@ -54,6 +54,8 @@ static NotifierList colo_compare_notifiers =#define REGULAR_PACKET_CHECK_MS 3000 #define
DEFAULT_TIME_OUT_MSQuoted 2 lines
Quoted 2 lines
3000
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
net/colo-compare.c +2
-0+static QemuMutex colo_compare_mutex;static bool +colo_compare_active;static QemuMutex event_mtx; static QemuCond event_complete_cond; static int
event_unhandled_count; @@ -906,6 +908,12 @@ static voidcheck_old_packet_regular(void *opaque)voidcolo_notify_compares_event(void *opaque, int event, Error **errp){
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
CompareState *s;net/colo-compare.c +6
-0+qemu_mutex_lock (&colo_compare_mutex); + +if (!colo_compare_active) { +qemu_mutex_unlock (&colo_compare_mutex); +return ; + }qemu_mutex_lock (&event_mtx); QTAILQ_FOREACH(s, &net_compares, next) { @@ -919,6 +927,7@@ void colo_notify_compares_event(void *opaque, int event,
Error **errp)
Quoted 55 lines
Quoted 55 lines
Quoted 8 lines
Quoted 8 lines
} qemu_mutex_unlock(&event_mtx);+
qemu_mutex_unlock(&colo_compare_mutex);
}static void
colo_compare_timer_init(CompareState *s)@@ -1274,7
+1283,14 @@ static void colo_compare_complete(UserCreatable *uc,Error **errp)
Quoted 46 lines
Quoted 46 lines
s->vnet_hdr); }net/colo-compare.c +7
-0+qemu_mutex_lock (&colo_compare_mutex); +if (!colo_compare_active) { +qemu_mutex_init (&event_mtx); +qemu_cond_init (&event_complete_cond); + colo_compare_active = true; + }QTAILQ_INSERT_TAIL (&net_compares, s, next); +qemu_mutex_unlock (&colo_compare_mutex); s->out_sendco.s = s; s->out_sendco.chr = &s->chr_out; @@ -1290,9 +1306,6 @@static void colo_compare_complete(UserCreatable
*uc, Error **errp)g_queue_init(&s->conn_list);net/colo-compare.c +0
-3-qemu_mutex_init (&event_mtx); -qemu_cond_init (&event_complete_cond); - s->connection_track_table =g_hash_table_new_full(connection_key_hash,
connection_key_equal, g_free,@@
-1384,12 +1397,19 @@ static voidcolo_compare_finalize(Object *obj)qemu_bh_delete(s->event_bh);net/colo-compare.c +7
-0+qemu_mutex_lock (&colo_compare_mutex); QTAILQ_FOREACH(tmp, &net_compares, next) {if (tmp == s) {QTAILQ_REMOVE (&net_compares, s, next);break ; } } +if (QTAILQ_EMPTY (&net_compares)) { + colo_compare_active = false; +qemu_mutex_destroy (&event_mtx); +qemu_cond_destroy (&event_complete_cond); + } +qemu_mutex_unlock (&colo_compare_mutex);Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
net/colo-compare.c static void colo_compare_finalize(Object
Quoted 3 lines
AioContext *ctx = iothread_get_aio_context(s->iothread); aio_context_acquire(ctx);net/colo-compare.c +0
-0@@ -1413,15 +1433,18 @@ static void colo_compare_finalize(Object*obj)
Quoted 57 lines
Quoted 56 lines
Quoted 42 lines
Quoted 42 lines
net/colo-compare.c +5
-3object_unref (OBJECT (s->iothread)); } -qemu_mutex_destroy (&event_mtx); -qemu_cond_destroy (&event_complete_cond); -g_free (s->pri_indev);g_free (s->sec_indev);g_free (s->outdev);g_free (s->notify_dev); } +void colo_compare_init_globals (void ) { + colo_compare_active = false; +qemu_mutex_init (&colo_compare_mutex); +} +static const TypeInfo colo_compare_info = { .name =TYPE_COLO_COMPARE , .parent =TYPE_OBJECT ,net/colo-compare.h index renamed from net/colo-compare.h +0
-0
22ddd512e2..eb483ac586100644net/colo-compare.h +1
-0@@ -17,6 +17,7 @@#ifndef QEMU_COLO_COMPARE_H #define QEMU_COLO_COMPARE_H +void colo_compare_init_globals (void );void colo_notify_compares_event (void *opaque,int event,Error **errp);void colo_compare_register_notifier (Notifier *notify);void
colo_compare_unregister_notifier(Notifier *notify); diff --gita/softmmu/vl.cb/softmmu/vl.cindex32c0047889..a913ed5469100644softmmu/vl.c +1
-0@@ -112,6 +112,7 @@#include "qapi/qmp/qerror.h" #include "sysemu/iothread.h" #include "qemu/guest-random.h" +#include "net/colo-compare.h" Quoted 3 lines
softmmu/vl.c void qemu_init(int argc, char **argv, char
Quoted 3 lines
softmmu/vl.c +0
-0#define MAX_VIRTIO_CONSOLES 1@@ -2906,6 +2907,7 @@ void qemu_init(int argc, char **argv, char**envp)
Quoted 10 lines
Quoted 9 lines
softmmu/vl.c +1
-1precopy_infrastructure_init ();postcopy_infrastructure_init ();monitor_init_globals (); +colo_compare_init_globals ();if (qcrypto_init (&err) <0 ) {error_reportf_err (err,"cannot initialize crypto: " ); --2.20.1
Zhang, Chen
Reviewed-byRE: [PATCH v4 3/6] net/colo-compare.c: Fix deadlock in compare_chr_send
In reply to Lukas StraubQuoted 66 lines
-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Friday, May 8, 2020 3:56 PM
To: Zhang, Chen <[email protected]>
Cc: qemu-devel <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: Re: [PATCH v4 3/6]net/colo-compare.c: Fix deadlock incompare_chr_sendQuoted 23 lines · Chen
Quoted 23 lines
-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Friday, May 8, 2020 2:08 PM
To: Zhang, Chen <[email protected]>
Cc: qemu-devel <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: Re: [PATCH v4 3/6]net/colo-compare.c: Fix deadlock incompare_chr_sendQuoted 4 lines · Chen
Quoted 4 lines
Quoted 2 lines
No need to init the
notify_sendcoeach time, because the notify dev justan optional parameter.
Quoted 1 line
You can use the if (
s->notify_dev) here. Just Xen use the
chr_notify_dev.Quoted 5 lines
Quoted 5 lines
Ok, I will change that and the code below in the next version.
Quoted 2 lines
Overall, make the
chr_sendjob to coroutine is a good idea. It looks goodQuoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
for me.
Quoted 2 lines
And your patch inspired me, it looks we can re-use the
compare_chr_sendcode on filter mirror/redirector too.
I already have patch for that, but I don't think it is a good idea, because the guest then can send packets faster than colo-compare can process. This leads bufferbloat and the
performance drops in my tests:
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
Client-to-server tcp:
without patch: ~66 Mbit/s
with patch: ~59 Mbit/s
Server-to-client tcp:
without patch: ~702 Kbit/s
with patch: ~328 Kbit/sOh, a big performance drop, is that caused by memcpy/zero_copy
parts ?
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Thanks
Zhang ChenNo, there is no memcpy overhead with this patch, see below.
I means for the filter mirror/redirector parts why coroutine will lead huge
performance drop?
Quoted 11 lines
It's because having a additional buffer before the network bottleneck (colo- compare in our case) confuses TCP's congestion-control:
TCP will speed up the data transfer until packets start to drop (or the network interface is blocked). This feedback has to be quick so TCP can select a suitable transfer speed. But with the patch, the guest will fill the buffer as fast as it can (it does not "see" the slow bandwidth of colo-compare behind the buffer) until it it hits against the TCP congestion window. At this point TCP drastically reduces its transfer speed and it stays low because the full buffer delays the packets so it doesn't receive ACK's so it can't speed up the transfer again. Until the buffer is empty again (can take up to a second in my tests). Then this cycle repeats.
Make sense!
After fix above issue:
Thanks
Zhang Chen
Quoted 231 lines
Regards,
Lukas StraubQuoted 69 lines
Thanks
Zhang ChenQuoted 66 lines
Regards,
Lukas Straub--- net/filter-mirror.c | 142 +++++++++++++++++++++++++++++++++----------- 1 file changed, 106 insertions(+), 36 deletions(-)net/filter-mirror.c index renamed from net/filter-mirror.c +0
-0
d83e815545..6bcd317502100644net/filter-mirror.c +19
-11@@ -20,6 +20,8 @@#include "chardev/char-fe.h" #include "qemu/iov.h" #include "qemu/sockets.h" +#include "block/aio-wait.h" +#include "qemu/coroutine.h" #define FILTER_MIRROR (obj) \ OBJECT_CHECK(MirrorState, (obj), TYPE_FILTER_MIRROR) @@ -31,6 +33,18 @@ #define TYPE_FILTER_REDIRECTOR "filter-redirector"#define REDIRECTOR_MAX_LEN NET_BUFSIZE +typedef struct SendCo { +Coroutine *co; +GQueue send_list; +bool done; +int ret; +}SendCo ; + +typedef struct SendEntry { +ssize_t size; +uint8_t buf[]; +}SendEntry ; + typedef struct MirrorState { NetFilterState parent_obj; char *indev;@@ -38,59 +52,101 @@ typedef struct MirrorState {CharBackend chr_in;CharBackend chr_out;SocketReadState rs; +SendCo sendco;bool vnet_hdr; } MirrorState; -static int filter_send (MirrorState *s, -const struct iovec *iov, -int iovcnt) +static void coroutine_fn _filter_send(void *opaque) { +MirrorState *s = opaque; +SendCo *sendco = &s->sendco;NetFilterState *nf =NETFILTER (s);int ret =0 ; -ssize_t size =0 ; -uint32_t len =0 ; -char *buf; - - size =iov_size (iov, iovcnt); -if (!size) { -return 0 ; - }Quoted 2 lines
net/filter-mirror.c typedef struct MirrorState { · line 60
Quoted 2 lines
net/filter-mirror.c +0
-2- len =htonl (size); - ret = qemu_chr_fe_write_all(&s->chr_out, (uint8_t *)&len,sizeof(len));
Quoted 40 lines
Quoted 40 lines
net/filter-mirror.c +26
-11-if (ret !=sizeof (len)) { - goto err; - } +while (!g_queue_is_empty (&sendco->send_list)) { +SendEntry *entry =g_queue_pop_tail (&sendco->send_list); +uint32_t len =htonl (entry->size); -if (s->vnet_hdr) { -/* -* If vnet_hdr = on, we send vnet header len to make other -* module(like colo-compare) know how to parse net -* packet correctly. -*/ -ssize_t vnet_hdr_len; + ret =qemu_chr_fe_write_all (&s->chr_out, + (uint8_t *)&len, +sizeof (len)); +if (ret !=sizeof (len)) { +g_free (entry); + goto err; + } - vnet_hdr_len = nf->netdev->vnet_hdr_len; +if (s->vnet_hdr) { +/* +* If vnet_hdr = on, we send vnet header len to make other +* module(like colo-compare) know how to parse net +* packet correctly. +*/ + + len =htonl (nf->netdev->vnet_hdr_len); + ret =qemu_chr_fe_write_all (&s->chr_out, + (uint8_t *)&len, +sizeof (len)); +if (ret !=sizeof (len)) { +g_free (entry); + goto err; + } + }Quoted 2 lines
net/filter-mirror.c typedef struct MirrorState { · line 76
Quoted 2 lines
net/filter-mirror.c +0
-2- len =htonl (vnet_hdr_len); - ret = qemu_chr_fe_write_all(&s->chr_out, (uint8_t *)&len,sizeof(len));
Quoted 91 lines
Quoted 91 lines
net/filter-mirror.c +53
-10-if (ret !=sizeof (len)) { + ret =qemu_chr_fe_write_all (&s->chr_out, + (uint8_t *)entry->buf, + entry->size); +if (ret != entry->size) { +g_free (entry); goto err; } - } - buf =g_malloc (size); -iov_to_buf (iov, iovcnt,0 , buf, size); - ret =qemu_chr_fe_write_all (&s->chr_out, (uint8_t *)buf, size); -g_free (buf); -if (ret != size) { - goto err; +g_free (entry); } -return 0 ; + sendco->ret =0 ; + goto out; err: -return ret <0 ? ret : -EIO ; +while (!g_queue_is_empty (&sendco->send_list)) { +SendEntry *entry =g_queue_pop_tail (&sendco->send_list); +g_free (entry); + } + sendco->ret = ret <0 ? ret : -EIO ; +out: + sendco->co =NULL ; + sendco->done = true; +aio_wait_kick (); +} + +static int filter_send (MirrorState *s, +const struct iovec *iov, +int iovcnt) +{ +SendCo *sendco = &s->sendco; +SendEntry *entry; + +ssize_t size =iov_size (iov, iovcnt); +if (!size) { +return 0 ; + } + + entry =g_malloc (sizeof (SendEntry) + size); + entry->size = size; +iov_to_buf (iov, iovcnt,0 , entry->buf, size); +g_queue_push_head (&sendco->send_list, entry); + +if (sendco->done) { + sendco->co =qemu_coroutine_create (_filter_send, s); + sendco->done = false; +qemu_coroutine_enter (sendco->co); +if (sendco->done) { +/* report early errors */ +return sendco->ret; + } + } + +/* assume success */ +return 0 ; } static void redirector_to_filter(NetFilterState *nf, @@ -194,6 +250,10 @@ static void filter_mirror_cleanup(NetFilterState *nf) {MirrorState *s =FILTER_MIRROR (nf); +AIO_WAIT_WHILE (NULL , !s->sendco.done); + +g_queue_clear (&s->sendco.send_list); +qemu_chr_fe_deinit (&s->chr_out, false); }@@ -201,6 +261,10 @@ static voidfilter_redirector_cleanup(NetFilterState
*nf) { MirrorState *s = FILTER_REDIRECTOR(nf);net/filter-mirror.c +4
-0+AIO_WAIT_WHILE (NULL , !s->sendco.done); + +g_queue_clear (&s->sendco.send_list); +qemu_chr_fe_deinit (&s->chr_in, false); qemu_chr_fe_deinit(&s->chr_out, false); } @@ -224,6 +288,9 @@static void
filter_mirror_setup(NetFilterState *nf, Error **errp)}Quoted 8 lines
net/filter-mirror.c static void filter_mirror_setup(NetFilterState *nf, Error **errp)
Quoted 8 lines
qemu_chr_fe_init(&s->chr_out, chr, errp);net/filter-mirror.c +4
-0+ + s->sendco.done = true; +g_queue_init (&s->sendco.send_list); } static void redirector_rs_finalize(SocketReadState *rs) @@ -281,6 +348,9 @@ static void filter_redirector_setup(NetFilterState *nf, Error**errp)
Quoted 12 lines
Quoted 11 lines
return; } }net/filter-mirror.c +3
-1+ + s->sendco.done = true; +g_queue_init (&s->sendco.send_list); }static void filter_mirror_class_init (ObjectClass *oc,void *data) --2.20.1
Zhang, Chen
Reviewed-byRE: [PATCH v4 6/6] net/colo-compare.c: Correct ordering in complete and finalize
In reply to Lukas StraubQuoted 37 lines
-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Friday, May 8, 2020 12:09 AM
To: Zhang, Chen <[email protected]>
Cc: qemu-devel <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: Re: [PATCH v4 6/6]net/colo-compare.c: Correct ordering in complete and finalizeQuoted 11 lines · Chen
Quoted 11 lines
-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Monday, May 4, 2020 6:28 PM
To: qemu-devel <[email protected]>
Cc: Zhang, Chen <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: [PATCH v4 6/6]net/colo-compare.c: Correct ordering in complete and finalizeQuoted 1 line
Quoted 1 line
In
colo_compare_complete, insert CompareState intonet_comparesonly
Quoted 8 lines
Quoted 3 lines
after everything has been initialized.
Incolo_compare_finalize, remove CompareState fromnet_comparesbefore anything is deinitialized.S/deinitialized/finalized
It looks no dependences on each step on initialization and finalization. Do you means we just need add/remove each colo-compare module at last
Quoted 8 lines
in logic?
Yes. While I didn't see any crashes here, there is the possibility that if colo- compare is removed during checkpoint, the destroyed
event_bhis called fromcolo_notify_compares_event. Same withcolo_compare_complete(very unlikely) if colo-compare is created while colo is running,colo_notify_compares_eventmay call the uninitializedevent_bh.
Thanks
Zhang Chen
Quoted 96 lines
Regards,
Lukas StraubQuoted 93 lines
Or current code have some issue?
Thanks
Zhang ChenQuoted 88 lines
Signed-off-by Lukas Straub <[email protected]>--- net/colo-compare.c | 45 +++++++++++++++++++++++---------------------- 1 file changed, 23 insertions(+), 22 deletions(-)net/colo-compare.c index renamed from net/colo-compare.c +0
-0
c7572d75e9..6f80bcece6100644net/colo-compare.c +0
-9@@ -1283,15 +1283,6 @@ static voidcolo_compare_complete (UserCreatable *uc, Error **errp) s->vnet_hdr); } -qemu_mutex_lock (&colo_compare_mutex); -if (!colo_compare_active) { -qemu_mutex_init (&event_mtx); -qemu_cond_init (&event_complete_cond); - colo_compare_active = true; - } -QTAILQ_INSERT_TAIL (&net_compares, s, next); -qemu_mutex_unlock (&colo_compare_mutex); - s->out_sendco.s = s; s->out_sendco.chr = &s->chr_out;s->out_sendco.notify_remote_frame = false; @@ -1312,6 +1303,16@@ static void
colo_compare_complete(UserCreatable *uc, Error **errp)
connection_destroy);colo_compare_iothread(s);net/colo-compare.c +23
-14+ +qemu_mutex_lock (&colo_compare_mutex); +if (!colo_compare_active) { +qemu_mutex_init (&event_mtx); +qemu_cond_init (&event_complete_cond); + colo_compare_active = true; + } +QTAILQ_INSERT_TAIL (&net_compares, s, next); +qemu_mutex_unlock (&colo_compare_mutex); +return ; }@@ -1384,19 +1385,6 @@ static void colo_compare_finalize(Object *obj)CompareState *s =COLO_COMPARE (obj);CompareState *tmp =NULL ; -qemu_chr_fe_deinit (&s->chr_pri_in, false); -qemu_chr_fe_deinit (&s->chr_sec_in, false); -qemu_chr_fe_deinit (&s->chr_out, false); -if (s->notify_dev) { -qemu_chr_fe_deinit (&s->chr_notify_dev, false); - } - -if (s->iothread) { -colo_compare_timer_del (s); - } - -qemu_bh_delete (s->event_bh); -qemu_mutex_lock (&colo_compare_mutex); QTAILQ_FOREACH(tmp, &net_compares, next) {if (tmp == s) {@@ -1411,6 +1399,19 @@ static void colo_compare_finalize(Object *obj) }qemu_mutex_unlock (&colo_compare_mutex); +qemu_chr_fe_deinit (&s->chr_pri_in, false); +qemu_chr_fe_deinit (&s->chr_sec_in, false); +qemu_chr_fe_deinit (&s->chr_out, false); +if (s->notify_dev) { +qemu_chr_fe_deinit (&s->chr_notify_dev, false); + } + +if (s->iothread) { +colo_compare_timer_del (s); + } + +qemu_bh_delete (s->event_bh); +AioContext *ctx =iothread_get_aio_context (s->iothread);aio_context_acquire (ctx);AIO_WAIT_WHILE (ctx, !s->out_sendco.done); --2.20.1
RE: [PATCH v4 5/6] net/colo-compare.c, softmmu/vl.c: Check that colo-compare is active
In reply to Lukas StraubQuoted 102 lines
-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Saturday, May 9, 2020 8:21 PM
To: Zhang, Chen <[email protected]>
Cc: qemu-devel <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>; Dr . David Alan Gilbert <[email protected]>
Subject: Re: [PATCH v4 5/6]net/colo-compare.c,softmmu/vl.c: Check that colo-compare is activeQuoted 75 lines · Chen
Quoted 75 lines
-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Friday, May 8, 2020 2:11 PM
To: Zhang, Chen <[email protected]>
Cc: qemu-devel <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: Re: [PATCH v4 5/6]net/colo-compare.c,softmmu/vl.c: Check that colo-compare is activeQuoted 38 lines · Chen
Quoted 38 lines
-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Thursday, May 7, 2020 11:54 PM
To: Zhang, Chen <[email protected]>
Cc: qemu-devel <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: Re: [PATCH v4 5/6]net/colo-compare.c,softmmu/vl.c:
Check that colo-compare is activeQuoted 24 lines · Chen
Quoted 24 lines
-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Monday, May 4, 2020 6:28 PM
To: qemu-devel <[email protected]>
Cc: Zhang, Chen <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>;
Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: [PATCH v4 5/6]net/colo-compare.c,softmmu/vl.c:
Check that
colo- compare is activeIf the colo-compare object is removed before failover and a checkpoint happens, qemu crashes because it tries to lock the destroyed
event_mtxincolo_notify_compares_event.Fix this by checking if everything is initialized by introducing a new variable
colo_compare_activewhich is protected by a new mutexcolo_compare_mutex. The new mutex also protects against concurrent access of thenet_compareslist and makes sure thatcolo_notify_compares_eventisn't active while we destroyevent_mtxandevent_complete_cond.Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
With this it also is again possible to use colo without colo-compare (periodic
mode) and to use multiple colo-compare for multiple networkinterfaces.
Quoted 19 lines
Quoted 7 lines
Quoted 6 lines
Quoted 1 line
Hi Lukas,
For this case I think we don't need to touch
vl.ccode, we can solve thisissue from another perspective:
Quoted 8 lines
Quoted 4 lines
How to remove colo-compare?
User will use qemu-monitor or QMP command to disable an object, so we just need return operation failed When user try to remove colo-compareobject while COLO is running.
Yeah, but that still leaves the other problem that colo can't be used without colo-compare (qemu crashes then).
Yes, the COLO-compare is necessary module in COLO original design. At most cases, user need it do dynamic sync.
For rare cases, maybe we can add a new colo-compare parameter toQuoted 8 lines
Quoted 4 lines
bypass all the network workload.
I think such an parameter would only be a workaround instead of a real solution like this patch.
The root problem is why COLO-compare is necessary.
Yes, maybe someone want to use pure periodic synchronization mode, But it means it will lost all guest network support(without colo-compare/filter-mirror/filter-redirector/filter-rewriter).
Quoted 2 lines
The secondary guest just a solid backup for the primary guest, when occur failover the new build stateful connection (like TCP) will crashed,
Quoted 8 lines
need userspace to handle this status. It lost the original meaning for COLO FT/HA solution, no need use do HA in application layer.
Quoted 1 line
it looks like normal/remote periodic VM snapshot here.
Sure, but maybe the user doesn't need (reliable) network on failover. Also proper network support with periodic mode can easily be implemented by modifying filter-buffer to buffer packets until checkpoint. Being able to use colo without colo-compare gives more flexibility to the user.
OK, use the filter-buffer instead of colo-compare is a new use case here.
If other maintainers have no comments about touch the vl.c for COLO , I think it's OK here.
Thanks
Zhang Chen
Quoted 184 lines
Regards,
Lukas StraubQuoted 22 lines
Dave or Jason have any comments here?
Thanks
Zhang ChenQuoted 17 lines
Regards,
Lukas StraubQuoted 13 lines
Thanks
Zhang ChenQuoted 10 lines
Regards,
Lukas StraubQuoted 6 lines
Thanks
Zhang ChenQuoted 3 lines
Signed-off-by Lukas Straub <[email protected]>--- net/colo-compare.c | 35 +++++++++++++++++++++++++++++------
Quoted 108 lines
Quoted 108 lines
Quoted 13 lines
Quoted 13 lines
Quoted 10 lines
Quoted 10 lines
net/colo-compare.h | 1 + softmmu/vl.c | 2 ++ 3 files changed, 32 insertions(+), 6 deletions(-)net/colo-compare.c index renamed from net/colo-compare.c +0
-0
56db3d3bfc..c7572d75e9100644net/colo-compare.c +0
-0@@ -54,6 +54,8 @@ static NotifierList colo_compare_notifiers= #define
REGULAR_PACKET_CHECK_MS3000 #define
DEFAULT_TIME_OUT_MSQuoted 2 lines
Quoted 2 lines
3000
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
net/colo-compare.c +2
-0+static QemuMutex colo_compare_mutex;static bool +colo_compare_active; static QemuMutex event_mtx; static QemuCond
event_complete_cond; static intevent_unhandled_count; @@ -906,6 +908,12 @@ static voidcheck_old_packet_regular(void *opaque)voidcolo_notify_compares_event(void *opaque, int event, Error **errp){
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
CompareState *s;net/colo-compare.c +6
-0+qemu_mutex_lock (&colo_compare_mutex); + +if (!colo_compare_active) { +qemu_mutex_unlock (&colo_compare_mutex); +return ; + }Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
qemu_mutex_lock(&event_mtx); QTAILQ_FOREACH(s, &net_compares, next) { @@ -919,6+927,7 @@ void colo_notify_compares_event(void *opaque, int event,
Error **errp)
Quoted 58 lines
Quoted 58 lines
Quoted 10 lines
net/colo-compare.c
Quoted 10 lines
} qemu_mutex_unlock(&event_mtx);net/colo-compare.c +3
-1+qemu_mutex_unlock (&colo_compare_mutex); } static void colo_compare_timer_init(CompareState *s) @@ -1274 ,7 +1283,14 @@ static void colo_compare_complete(UserCreatable +*uc,Error **errp)
Quoted 47 lines
Quoted 47 lines
s->vnet_hdr); }net/colo-compare.c +7
-0+qemu_mutex_lock (&colo_compare_mutex); +if (!colo_compare_active) { +qemu_mutex_init (&event_mtx); +qemu_cond_init (&event_complete_cond); + colo_compare_active = true; + }QTAILQ_INSERT_TAIL (&net_compares, s, next); +qemu_mutex_unlock (&colo_compare_mutex); s->out_sendco.s = s; s->out_sendco.chr = &s->chr_out; @@ -1290,9 +1306,6 @@static void
colo_compare_complete(UserCreatable *uc, Error **errp)g_queue_init(&s->conn_list);net/colo-compare.c +0
-3-qemu_mutex_init (&event_mtx); -qemu_cond_init (&event_complete_cond); - s->connection_track_table =g_hash_table_new_full(connection_key_hash,
connection_key_equal,
g_free, @@
-1384,12 +1397,19 @@ static void
colo_compare_finalize(Object
*obj)qemu_bh_delete(s->event_bh);net/colo-compare.c +7
-0+qemu_mutex_lock (&colo_compare_mutex); QTAILQ_FOREACH(tmp, &net_compares, next) {if (tmp == s) {QTAILQ_REMOVE (&net_compares, s, next);break ; } } +if (QTAILQ_EMPTY (&net_compares)) { + colo_compare_active = false; +qemu_mutex_destroy (&event_mtx); +qemu_cond_destroy (&event_complete_cond); + } +qemu_mutex_unlock (&colo_compare_mutex);Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
AioContext *ctx = iothread_get_aio_context(s->iothread); aio_context_acquire(ctx); @@ -1413,15 +1433,18 @@static void colo_compare_finalize(Object
*obj)
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
object_unref(OBJECT(s->iothread)); }net/colo-compare.c +0
-3-qemu_mutex_destroy (&event_mtx); -qemu_cond_destroy (&event_complete_cond); -g_free (s->pri_indev);g_free (s->sec_indev);g_free (s->outdev);g_free (s->notify_dev); }Quoted 14 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
net/colo-compare.c +5
-0+void colo_compare_init_globals (void ) { + colo_compare_active = false; +qemu_mutex_init (&colo_compare_mutex); +} +static const TypeInfo colo_compare_info = {Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
net/colo-compare.h line 17
Quoted 8 lines
.name =
TYPE_COLO_COMPARE,
.parent =TYPE_OBJECT,net/colo-compare.h index renamed from net/colo-compare.h +0
-0
22ddd512e2..eb483ac586100644net/colo-compare.h +0
-0@@ -17,6 +17,7 @@ #ifndef QEMU_COLO_COMPARE_H #define
QEMU_COLO_COMPARE_HQuoted 34 lines
Quoted 33 lines
Quoted 33 lines
Quoted 32 lines
Quoted 16 lines
Quoted 16 lines
net/colo-compare.h +1
-0+void colo_compare_init_globals (void ); void colo_notify_compares_event(void *opaque, int event,Error **errp); void
colo_compare_register_notifier(Notifier *notify); voidcolo_compare_unregister_notifier(Notifier *notify); diff --gita/softmmu/vl.cb/softmmu/vl.cindex32c0047889..a913ed5469
100644softmmu/vl.c +1
-0@@ -112,6 +112,7 @@#include "qapi/qmp/qerror.h" #include "sysemu/iothread.h" #include "qemu/guest-random.h" +#include "net/colo-compare.h" Quoted 4 lines
Quoted 4 lines
softmmu/vl.c +0
-0#define MAX_VIRTIO_CONSOLES 1@@ -2906,6 +2907,7 @@ void qemu_init(int argc, char **argv,char
**envp)
Quoted 11 lines
Quoted 10 lines
precopy_infrastructure_init(); postcopy_infrastructure_init(); monitor_init_globals();+
colo_compare_init_globals();if (
qcrypto_init(&err)< 0) {
error_reportf_err(err, "cannot initialize crypto:
");
--
2.20.1
Re: [PATCH v4 5/6] net/colo-compare.c, softmmu/vl.c: Check that colo-compare is active
In reply to Zhang, ChenQuoted 81 lines · * Zhang, Chen ([email protected]) wrote:
Quoted 73 lines
-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Friday, May 8, 2020 2:11 PM
To: Zhang, Chen <[email protected]>
Cc: qemu-devel <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: Re: [PATCH v4 5/6]net/colo-compare.c,softmmu/vl.c: Check that colo-compare is activeQuoted 37 lines · Chen
Quoted 37 lines
-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Thursday, May 7, 2020 11:54 PM
To: Zhang, Chen <[email protected]>
Cc: qemu-devel <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: Re: [PATCH v4 5/6]net/colo-compare.c,softmmu/vl.c: Check that colo-compare is activeQuoted 23 lines · Chen
Quoted 23 lines
-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Monday, May 4, 2020 6:28 PM
To: qemu-devel <[email protected]>
Cc: Zhang, Chen <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: [PATCH v4 5/6]net/colo-compare.c,softmmu/vl.c: Check that
colo- compare is activeIf the colo-compare object is removed before failover and a checkpoint happens, qemu crashes because it tries to lock the destroyed
event_mtxincolo_notify_compares_event.Fix this by checking if everything is initialized by introducing a new variable
colo_compare_activewhich is protected by a new mutexcolo_compare_mutex. The new mutex also protects against concurrent access of thenet_compareslist and makes sure thatcolo_notify_compares_eventisn't active while we destroyevent_mtxandevent_complete_cond.Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
With this it also is again possible to use colo without colo-compare (periodic
mode) and to use multiple colo-compare for multiple networkinterfaces.
Quoted 16 lines
Quoted 15 lines
Quoted 6 lines
Quoted 1 line
Hi Lukas,
For this case I think we don't need to touch
vl.ccode, we can solve thisissue from another perspective:
Quoted 4 lines
How to remove colo-compare?
User will use qemu-monitor or QMP command to disable an object, so we just need return operation failed When user try to remove colo-compareobject while COLO is running.
Yeah, but that still leaves the other problem that colo can't be used without colo-compare (qemu crashes then).
Quoted 7 lines
Quoted 3 lines
Yes, the COLO-compare is necessary module in COLO original design. At most cases, user need it do dynamic sync.
For rare cases, maybe we can add a new colo-compare parameter tobypass all the network workload.
I think such an parameter would only be a workaround instead of a real solution like this patch.
Quoted 7 lines
The root problem is why COLO-compare is necessary.
Yes, maybe someone want to use pure periodic synchronization mode,
But it means it will lost all guest network support(without colo-compare/filter-mirror/filter-redirector/filter-rewriter). The secondary guest just a solid backup for the primary guest, when occur failover the new build stateful connection (like TCP) will crashed, need userspace to handle this status. It lost the original meaning for COLO FT/HA solution, no need use do HA in application layer. it looks like normal/remote periodic VM snapshot here.
Dave or Jason have any comments here?
People have done fixed-rate VM synchronisation in the past; and there are workloads where the packet stream is particularly random so you almost always get comparison miscompares.
But in those setups, the rest of the configuration is very different - since the destination isn't running at the same time as the source, the block mirroring also gets simpler in those scenarious.
One thing we played with in the past was dynamically turning comparison on and off; switching back to simpler synchronisation if the workload turns out to suit it.
Dave
Quoted 170 lines
Thanks
Zhang ChenQuoted 166 lines
Regards,
Lukas StraubQuoted 26 lines
Thanks
Zhang ChenQuoted 23 lines
Regards,
Lukas StraubQuoted 10 lines
Thanks
Zhang ChenQuoted 7 lines
Signed-off-by Lukas Straub <[email protected]>--- net/colo-compare.c | 35 +++++++++++++++++++++++++++++------ net/colo-compare.h | 1 + softmmu/vl.c | 2 ++ 3 files changed, 32 insertions(+), 6 deletions(-)Quoted 6 lines
net/colo-compare.c static NotifierList colo_compare_notifiers = · line 54
Quoted 6 lines
net/colo-compare.c index renamed from net/colo-compare.c +0
-0
56db3d3bfc..c7572d75e9100644net/colo-compare.c +0
-0@@ -54,6 +54,8 @@ static NotifierList colo_compare_notifiers =#define REGULAR_PACKET_CHECK_MS 3000 #define
DEFAULT_TIME_OUT_MSQuoted 2 lines
Quoted 2 lines
3000
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
net/colo-compare.c +2
-0+static QemuMutex colo_compare_mutex;static bool +colo_compare_active;static QemuMutex event_mtx; static QemuCond event_complete_cond; static int
event_unhandled_count; @@ -906,6 +908,12 @@ static voidcheck_old_packet_regular(void *opaque)voidcolo_notify_compares_event(void *opaque, int event, Error **errp){
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
CompareState *s;net/colo-compare.c +6
-0+qemu_mutex_lock (&colo_compare_mutex); + +if (!colo_compare_active) { +qemu_mutex_unlock (&colo_compare_mutex); +return ; + }qemu_mutex_lock (&event_mtx); QTAILQ_FOREACH(s, &net_compares, next) { @@ -919,6 +927,7@@ void colo_notify_compares_event(void *opaque, int event,
Error **errp)
Quoted 55 lines
Quoted 55 lines
Quoted 8 lines
Quoted 8 lines
} qemu_mutex_unlock(&event_mtx);+
qemu_mutex_unlock(&colo_compare_mutex);
}static void
colo_compare_timer_init(CompareState *s)@@ -1274,7
+1283,14 @@ static void colo_compare_complete(UserCreatable *uc,Error **errp)
Quoted 46 lines
Quoted 46 lines
s->vnet_hdr); }net/colo-compare.c +7
-0+qemu_mutex_lock (&colo_compare_mutex); +if (!colo_compare_active) { +qemu_mutex_init (&event_mtx); +qemu_cond_init (&event_complete_cond); + colo_compare_active = true; + }QTAILQ_INSERT_TAIL (&net_compares, s, next); +qemu_mutex_unlock (&colo_compare_mutex); s->out_sendco.s = s; s->out_sendco.chr = &s->chr_out; @@ -1290,9 +1306,6 @@static void colo_compare_complete(UserCreatable
*uc, Error **errp)g_queue_init(&s->conn_list);net/colo-compare.c +0
-3-qemu_mutex_init (&event_mtx); -qemu_cond_init (&event_complete_cond); - s->connection_track_table =g_hash_table_new_full(connection_key_hash,
connection_key_equal, g_free,@@
-1384,12 +1397,19 @@ static voidcolo_compare_finalize(Object *obj)qemu_bh_delete(s->event_bh);net/colo-compare.c +7
-0+qemu_mutex_lock (&colo_compare_mutex); QTAILQ_FOREACH(tmp, &net_compares, next) {if (tmp == s) {QTAILQ_REMOVE (&net_compares, s, next);break ; } } +if (QTAILQ_EMPTY (&net_compares)) { + colo_compare_active = false; +qemu_mutex_destroy (&event_mtx); +qemu_cond_destroy (&event_complete_cond); + } +qemu_mutex_unlock (&colo_compare_mutex);Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
net/colo-compare.c static void colo_compare_finalize(Object
Quoted 3 lines
AioContext *ctx = iothread_get_aio_context(s->iothread); aio_context_acquire(ctx);net/colo-compare.c +0
-0@@ -1413,15 +1433,18 @@ static void colo_compare_finalize(Object*obj)
Quoted 57 lines
Quoted 56 lines
Quoted 42 lines
Quoted 42 lines
net/colo-compare.c +5
-3object_unref (OBJECT (s->iothread)); } -qemu_mutex_destroy (&event_mtx); -qemu_cond_destroy (&event_complete_cond); -g_free (s->pri_indev);g_free (s->sec_indev);g_free (s->outdev);g_free (s->notify_dev); } +void colo_compare_init_globals (void ) { + colo_compare_active = false; +qemu_mutex_init (&colo_compare_mutex); +} +static const TypeInfo colo_compare_info = { .name =TYPE_COLO_COMPARE , .parent =TYPE_OBJECT ,net/colo-compare.h index renamed from net/colo-compare.h +0
-0
22ddd512e2..eb483ac586100644net/colo-compare.h +1
-0@@ -17,6 +17,7 @@#ifndef QEMU_COLO_COMPARE_H #define QEMU_COLO_COMPARE_H +void colo_compare_init_globals (void );void colo_notify_compares_event (void *opaque,int event,Error **errp);void colo_compare_register_notifier (Notifier *notify);void
colo_compare_unregister_notifier(Notifier *notify); diff --gita/softmmu/vl.cb/softmmu/vl.cindex32c0047889..a913ed5469100644softmmu/vl.c +1
-0@@ -112,6 +112,7 @@#include "qapi/qmp/qerror.h" #include "sysemu/iothread.h" #include "qemu/guest-random.h" +#include "net/colo-compare.h" Quoted 3 lines
softmmu/vl.c void qemu_init(int argc, char **argv, char
Quoted 3 lines
softmmu/vl.c +0
-0#define MAX_VIRTIO_CONSOLES 1@@ -2906,6 +2907,7 @@ void qemu_init(int argc, char **argv, char**envp)
Quoted 10 lines
Quoted 9 lines
softmmu/vl.c +1
-1precopy_infrastructure_init ();postcopy_infrastructure_init ();monitor_init_globals (); +colo_compare_init_globals ();if (qcrypto_init (&err) <0 ) {error_reportf_err (err,"cannot initialize crypto: " ); --2.20.1
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Please update this series, I will queue it to COLO branch.
Thanks
Zhang Chen
Quoted 58 lines
-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Monday, May 4, 2020 6:28 PM
To: qemu-devel <[email protected]>
Cc: Zhang, Chen <[email protected]>; Li Zhijian <[email protected]>; Jason Wang <[email protected]>; Marc- André Lureau <[email protected]>; Paolo Bonzini <[email protected]>
Subject: [PATCH v4 0/6] colo-compare bugfixesHello Everyone,
Here are fixes for bugs that I found in my tests.Benchmark results:
Client-to-server tcp:
without patch: ~63 Mbit/s
with patch: ~66 Mbit/s
Server-to-client tcp:
without patch: ~771 Kbit/s
with patch: ~702 Kbit/sRegards,
Lukas StraubVersion changes:
v4:
-fix potential deadlock withnotify_remote_frame-avoid malloc and
memcpy in many casesv3:
-fix checkpatch.pl errorv2:
-better wording
-fix performance-regression in patch 3 "net/colo-compare.c: Fix deadlock incompare_chr_send"
-add more bugfixesLukas Straub (6):
net/colo-compare.c: Createevent_bhwith the right AioContextchardev/char.c: Useqemu_co_sleep_nsif in coroutinenet/colo-compare.c: Fix deadlock incompare_chr_sendnet/colo-compare.c: Only hexdump packets if tracing is enablednet/colo-compare.c,softmmu/vl.c: Check that colo-compare is activenet/colo-compare.c: Correct ordering in complete and finalizechardev/char.c | 7 +- net/colo-compare.c | 250 +++++++++++++++++++++++++++++++++--------- --- net/colo-compare.h | 1 + net/colo.c | 7 ++ net/colo.h | 1 + softmmu/vl.c | 2 + 6 files changed, 204 insertions(+), 64 deletions(-)--
2.20.1