Skip to contentSkip to conversation

colo: Add support for continuous replication

PATCH Superseded Lukas Straub 2 participants 17 messages Sep 15, 2019 – Sep 30, 2019

Subsystems (2)

v5 of 7 Superseded Latest is v7 →

lore
Share & tools
lore
#0 lore raw

v5 cover colo: Add support for continuous replication

Hello Everyone,
These Patches add support for continuous replication to colo. This means that after the Primary fails and the Secondary did a failover, the Secondary can then become Primary and resume replication to a new Secondary.

On a side note, I wrote a Pacemaker Resource Agent for colo which I will post when its ready.
I have to say it's quite fun to randomly kill a Node, wait for resync and repeat and see how the VM stays alive all the time. :)

Regards,
Lukas Straub

v5:
- change syntax for the position= parameter
- fix spelling mistake

v4:
- fix checkpatch.pl warnings

v3:
- add test for replication changes
- check if the filter to be inserted before/behind belongs to the same interface
- fix the error message for the position= parameter
- rename term "after" -> "behind" and variable "insert_before" -> "insert_before_flag"
- document the quorum node on the secondary side
- simplify quorum parameters in documentation
- remove trailing spaces in documentation
- clarify the testing procedure in documentation

v2:
- fix email formating
- fix checkpatch.pl warnings
- fix patchew error
- clearer commit messages

Lukas Straub (4):
block/replication.c: Ignore requests after failover tests/test-replication.c: Add test for ignoring requests after
failover
net/filter.c: Add Options to insert filters anywhere in the filter
list
colo: Update Documentation for continuous replication

 block/replication.c        |  38 ++++++-
 docs/COLO-FT.txt           | 212 +++++++++++++++++++++++++++----------
 docs/block-replication.txt |  28 +++--
 include/net/filter.h       |   2 +
 net/filter.c               |  92 +++++++++++++++-
 qemu-options.hx            |  10 +-
 tests/test-replication.c   |  52 +++++++++
 7 files changed, 358 insertions(+), 76 deletions(-)

--
2.20.1

#1 lore raw

PATCH v5 1/4 block/replication.c: Ignore requests after failover

After failover the Secondary side of replication shouldn't change state, because it now functions as our primary disk.

In replication_start, replication_do_checkpoint, replication_stop, ignore the request if current state is BLOCK_REPLICATION_DONE (sucessful failover) or BLOCK_REPLICATION_FAILOVER (failover in progres i.e. currently merging active and hidden images into the base image).

---
 block/replication.c | 38 +++++++++++++++++++++++++++++++++++---
 1 file changed, 35 insertions(+), 3 deletions(-)
block/replication.c +35 -3
index 3d4dedddfc..97cc65c0cf 100644
@@ -454,6 +454,17 @@ static void replication_start(ReplicationState *rs, ReplicationMode mode,
     aio_context_acquire(aio_context);
     s = bs->opaque;
 
+    if (s->stage == BLOCK_REPLICATION_DONE ||
+        s->stage == BLOCK_REPLICATION_FAILOVER) {
+        /*
+         * This case happens when a secondary is promoted to primary.
+         * Ignore the request because the secondary side of replication
+         * doesn't have to do anything anymore.
+         */
+        aio_context_release(aio_context);
+        return;
+    }
+
     if (s->stage != BLOCK_REPLICATION_NONE) {
         error_setg(errp, "Block replication is running or done");
         aio_context_release(aio_context);
@@ -529,8 +540,7 @@ static void replication_start(ReplicationState *rs, ReplicationMode mode,
                    "Block device is in use by internal backup job");
 
         top_bs = bdrv_lookup_bs(s->top_id, s->top_id, NULL);
-        if (!top_bs || !bdrv_is_root_node(top_bs) ||
-            !check_top_bs(top_bs, bs)) {
+        if (!top_bs || !check_top_bs(top_bs, bs)) {
             error_setg(errp, "No top_bs or it is invalid");
             reopen_backing_file(bs, false, NULL);
             aio_context_release(aio_context);
@@ -577,6 +587,17 @@ static void replication_do_checkpoint(ReplicationState *rs, Error **errp)
     aio_context_acquire(aio_context);
     s = bs->opaque;
 
+    if (s->stage == BLOCK_REPLICATION_DONE ||
+        s->stage == BLOCK_REPLICATION_FAILOVER) {
+        /*
+         * This case happens when a secondary was promoted to primary.
+         * Ignore the request because the secondary side of replication
+         * doesn't have to do anything anymore.
+         */
+        aio_context_release(aio_context);
+        return;
+    }
+
     if (s->mode == REPLICATION_MODE_SECONDARY) {
         secondary_do_checkpoint(s, errp);
     }
@@ -593,7 +614,7 @@ static void replication_get_error(ReplicationState *rs, Error **errp)
     aio_context_acquire(aio_context);
     s = bs->opaque;
 
-    if (s->stage != BLOCK_REPLICATION_RUNNING) {
+    if (s->stage == BLOCK_REPLICATION_NONE) {
         error_setg(errp, "Block replication is not running");
         aio_context_release(aio_context);
         return;
@@ -635,6 +656,17 @@ static void replication_stop(ReplicationState *rs, bool failover, Error **errp)
     aio_context_acquire(aio_context);
     s = bs->opaque;
 
+    if (s->stage == BLOCK_REPLICATION_DONE ||
+        s->stage == BLOCK_REPLICATION_FAILOVER) {
+        /*
+         * This case happens when a secondary was promoted to primary.
+         * Ignore the request because the secondary side of replication
+         * doesn't have to do anything anymore.
+         */
+        aio_context_release(aio_context);
+        return;
+    }
+
     if (s->stage != BLOCK_REPLICATION_RUNNING) {
         error_setg(errp, "Block replication is not running");
         aio_context_release(aio_context);

--
2.20.1

Reviewed-by on v5 1/4
#2 lore raw

RE: [PATCH v5 1/4] block/replication.c: Ignore requests after failover

Quoted 11 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Monday, September 16, 2019 3:20 AM
To: qemu-devel <[email protected]>
Cc: Zhang, Chen <[email protected]>; Jason Wang <[email protected]>; Wen Congyang <[email protected]>; Xie Changlong <[email protected]>; [email protected]; [email protected]
Subject: [PATCH v5 1/4] block/replication.c: Ignore requests after failover

Quoted 8 lines

After failover the Secondary side of replication shouldn't change state, because it now functions as our primary disk.

In replication_start, replication_do_checkpoint, replication_stop, ignore the request if current state is BLOCK_REPLICATION_DONE (sucessful failover) or BLOCK_REPLICATION_FAILOVER (failover in progres i.e. currently merging active and hidden images into the base image).

It looks good for me, and this patch works well in COLO continuous backup status in my tests.

But I think this patch still need reviewed by Xie Changlong or Kevin Wolf.

Thanks
Zhang Chen

Quoted 87 lines
---
 block/replication.c | 38 +++++++++++++++++++++++++++++++++++---
 1 file changed, 35 insertions(+), 3 deletions(-)
block/replication.c index renamed from block/replication.c +0 -0

3d4dedddfc..97cc65c0cf 100644

block/replication.c +11 -0
@@ -454,6 +454,17 @@ static void replication_start(ReplicationState *rs,
 ReplicationMode mode,
     aio_context_acquire(aio_context);
     s = bs->opaque;
 
+    if (s->stage == BLOCK_REPLICATION_DONE ||
+        s->stage == BLOCK_REPLICATION_FAILOVER) {
+        /*
+         * This case happens when a secondary is promoted to primary.
+         * Ignore the request because the secondary side of replication
+         * doesn't have to do anything anymore.
+         */
+        aio_context_release(aio_context);
+        return;
+    }
+
     if (s->stage != BLOCK_REPLICATION_NONE) {
         error_setg(errp, "Block replication is running or done");
         aio_context_release(aio_context); @@ -529,8 +540,7 @@ static void
replication_start(ReplicationState *rs, ReplicationMode mode,
                    "Block device is in use by internal backup job");

         top_bs = bdrv_lookup_bs(s->top_id, s->top_id, NULL);
block/replication.c +1 -2
-        if (!top_bs || !bdrv_is_root_node(top_bs) ||
-            !check_top_bs(top_bs, bs)) {
+        if (!top_bs || !check_top_bs(top_bs, bs)) {
             error_setg(errp, "No top_bs or it is invalid");
             reopen_backing_file(bs, false, NULL);
             aio_context_release(aio_context); @@ -577,6 +587,17 @@ static void

replication_do_checkpoint(ReplicationState *rs, Error **errp)

     aio_context_acquire(aio_context);
     s = bs->opaque;
block/replication.c +12 -1
+    if (s->stage == BLOCK_REPLICATION_DONE ||
+        s->stage == BLOCK_REPLICATION_FAILOVER) {
+        /*
+         * This case happens when a secondary was promoted to primary.
+         * Ignore the request because the secondary side of replication
+         * doesn't have to do anything anymore.
+         */
+        aio_context_release(aio_context);
+        return;
+    }
+
     if (s->mode == REPLICATION_MODE_SECONDARY) {
         secondary_do_checkpoint(s, errp);
     }
@@ -593,7 +614,7 @@ static void replication_get_error(ReplicationState *rs,
 Error **errp)
     aio_context_acquire(aio_context);
     s = bs->opaque;
 
-    if (s->stage != BLOCK_REPLICATION_RUNNING) {
+    if (s->stage == BLOCK_REPLICATION_NONE) {
         error_setg(errp, "Block replication is not running");
         aio_context_release(aio_context);
         return;
block/replication.c +11 -0
@@ -635,6 +656,17 @@ static void replication_stop(ReplicationState *rs,
 bool failover, Error **errp)
     aio_context_acquire(aio_context);
     s = bs->opaque;
 
+    if (s->stage == BLOCK_REPLICATION_DONE ||
+        s->stage == BLOCK_REPLICATION_FAILOVER) {
+        /*
+         * This case happens when a secondary was promoted to primary.
+         * Ignore the request because the secondary side of replication
+         * doesn't have to do anything anymore.
+         */
+        aio_context_release(aio_context);
+        return;
+    }
+
     if (s->stage != BLOCK_REPLICATION_RUNNING) {
         error_setg(errp, "Block replication is not running");
         aio_context_release(aio_context);

--
2.20.1

#3 lore raw

PATCH v5 2/4 tests/test-replication.c: Add test for ignoring requests after failover

In reply to Lukas Straub

This simulates the case that happens when we resume COLO after failover.

---
 tests/test-replication.c | 52 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 52 insertions(+)
tests/test-replication.c +52 -0
index f085d1993a..5addfc2227 100644
@@ -489,6 +489,56 @@ static void test_secondary_stop(void)
     teardown_secondary();
 }
 
+static void test_secondary_failover_then_ignore_requests(void)
+{
+    BlockBackend *top_blk, *local_blk;
+    Error *local_err = NULL;
+
+    top_blk = start_secondary();
+    replication_start_all(REPLICATION_MODE_SECONDARY, &local_err);
+    g_assert(!local_err);
+
+    /* write 0x22 to s_local_disk (IMG_SIZE / 2, IMG_SIZE) */
+    local_blk = blk_by_name(S_LOCAL_DISK_ID);
+    test_blk_write(local_blk, 0x22, IMG_SIZE / 2, IMG_SIZE / 2, false);
+
+    /* replication will backup s_local_disk to s_hidden_disk */
+    test_blk_read(top_blk, 0x11, IMG_SIZE / 2,
+                  IMG_SIZE / 2, 0, IMG_SIZE, false);
+
+    /* write 0x33 to s_active_disk (0, IMG_SIZE / 2) */
+    test_blk_write(top_blk, 0x33, 0, IMG_SIZE / 2, false);
+
+    /* do failover (active commit) */
+    replication_stop_all(true, &local_err);
+    g_assert(!local_err);
+
+    /* it should ignore all requests from now on */
+
+    /* start after failover */
+    replication_start_all(REPLICATION_MODE_PRIMARY, &local_err);
+    g_assert(!local_err);
+
+    /* checkpoint */
+    replication_do_checkpoint_all(&local_err);
+    g_assert(!local_err);
+
+    /* stop */
+    replication_stop_all(true, &local_err);
+    g_assert(!local_err);
+
+    /* read from s_local_disk (0, IMG_SIZE / 2) */
+    test_blk_read(top_blk, 0x33, 0, IMG_SIZE / 2,
+                  0, IMG_SIZE / 2, false);
+
+
+    /* read from s_local_disk (IMG_SIZE / 2, IMG_SIZE) */
+    test_blk_read(top_blk, 0x22, IMG_SIZE / 2,
+                  IMG_SIZE / 2, 0, IMG_SIZE, false);
+
+    teardown_secondary();
+}
+
 static void test_secondary_do_checkpoint(void)
 {
     BlockBackend *top_blk, *local_blk;
@@ -584,6 +634,8 @@ int main(int argc, char **argv)
     g_test_add_func("/replication/secondary/write", test_secondary_write);
     g_test_add_func("/replication/secondary/start", test_secondary_start);
     g_test_add_func("/replication/secondary/stop",  test_secondary_stop);
+    g_test_add_func("/replication/secondary/failover_then_ignore_requests",
+                    test_secondary_failover_then_ignore_requests);
     g_test_add_func("/replication/secondary/do_checkpoint",
                     test_secondary_do_checkpoint);
     g_test_add_func("/replication/secondary/get_error_all",

--
2.20.1

#4 lore raw

RE: [PATCH v5 2/4] tests/test-replication.c: Add test for ignoring requests after failover

Quoted 12 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Monday, September 16, 2019 3:20 AM
To: qemu-devel <[email protected]>
Cc: Zhang, Chen <[email protected]>; Jason Wang <[email protected]>; Wen Congyang <[email protected]>; Xie Changlong <[email protected]>; [email protected]; [email protected]
Subject: [PATCH v5 2/4] tests/test-replication.c: Add test for ignoring requests after failover

Quoted 2 lines

This simulates the case that happens when we resume COLO after failover.

It looks change the title to "Add test for secondary node continuous backup" is better.

Quoted 11 lines
---
 tests/test-replication.c | 52
++++++++++++++++++++++++++++++++++++++++
 1 file changed, 52 insertions(+)
tests/test-replication.c index renamed from tests/test-replication.c +0 -0

f085d1993a..5addfc2227 100644

tests/test-replication.c static void test_secondary_stop(void) · line 492

Quoted 4 lines
tests/test-replication.c +1 -0
@@ -489,6 +489,56 @@ static void test_secondary_stop(void)
     teardown_secondary();
 }
 
+static void test_secondary_failover_then_ignore_requests(void)

Same as title, I think change to "test_secondary_continuous_backup" is more clear.

Thanks
Zhang Chen

Quoted 64 lines
tests/test-replication.c +51 -0
+{
+    BlockBackend *top_blk, *local_blk;
+    Error *local_err = NULL;
+
+    top_blk = start_secondary();
+    replication_start_all(REPLICATION_MODE_SECONDARY, &local_err);
+    g_assert(!local_err);
+
+    /* write 0x22 to s_local_disk (IMG_SIZE / 2, IMG_SIZE) */
+    local_blk = blk_by_name(S_LOCAL_DISK_ID);
+    test_blk_write(local_blk, 0x22, IMG_SIZE / 2, IMG_SIZE / 2, false);
+
+    /* replication will backup s_local_disk to s_hidden_disk */
+    test_blk_read(top_blk, 0x11, IMG_SIZE / 2,
+                  IMG_SIZE / 2, 0, IMG_SIZE, false);
+
+    /* write 0x33 to s_active_disk (0, IMG_SIZE / 2) */
+    test_blk_write(top_blk, 0x33, 0, IMG_SIZE / 2, false);
+
+    /* do failover (active commit) */
+    replication_stop_all(true, &local_err);
+    g_assert(!local_err);
+
+    /* it should ignore all requests from now on */
+
+    /* start after failover */
+    replication_start_all(REPLICATION_MODE_PRIMARY, &local_err);
+    g_assert(!local_err);
+
+    /* checkpoint */
+    replication_do_checkpoint_all(&local_err);
+    g_assert(!local_err);
+
+    /* stop */
+    replication_stop_all(true, &local_err);
+    g_assert(!local_err);
+
+    /* read from s_local_disk (0, IMG_SIZE / 2) */
+    test_blk_read(top_blk, 0x33, 0, IMG_SIZE / 2,
+                  0, IMG_SIZE / 2, false);
+
+
+    /* read from s_local_disk (IMG_SIZE / 2, IMG_SIZE) */
+    test_blk_read(top_blk, 0x22, IMG_SIZE / 2,
+                  IMG_SIZE / 2, 0, IMG_SIZE, false);
+
+    teardown_secondary();
+}
+
 static void test_secondary_do_checkpoint(void)
 {
     BlockBackend *top_blk, *local_blk;
@@ -584,6 +634,8 @@ int main(int argc, char **argv)
     g_test_add_func("/replication/secondary/write", test_secondary_write);
     g_test_add_func("/replication/secondary/start", test_secondary_start);
     g_test_add_func("/replication/secondary/stop",  test_secondary_stop);
+
 g_test_add_func("/replication/secondary/failover_then_ignore_requests",
+                    test_secondary_failover_then_ignore_requests);
     g_test_add_func("/replication/secondary/do_checkpoint",
                     test_secondary_do_checkpoint);

g_test_add_func("/replication/secondary/get_error_all",
--
2.20.1

#5 lore raw

Re: [PATCH v5 2/4] tests/test-replication.c: Add test for ignoring requests after failover

Quoted 11 lines · Chen
Quoted 11 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Monday, September 16, 2019 3:20 AM
To: qemu-devel <[email protected]>
Cc: Zhang, Chen <[email protected]>; Jason Wang <[email protected]>; Wen Congyang <[email protected]>; Xie Changlong <[email protected]>; [email protected]; [email protected]
Subject: [PATCH v5 2/4] tests/test-replication.c: Add test for ignoring requests after failover

Quoted 4 lines
Quoted 2 lines

This simulates the case that happens when we resume COLO after failover.

It looks change the title to "Add test for secondary node continuous backup" is better.

Did you mean "continuous replication"? Would "Add test for secondary node continuing replication" be Ok?

Quoted 86 lines
Quoted 11 lines
---
 tests/test-replication.c | 52
++++++++++++++++++++++++++++++++++++++++
 1 file changed, 52 insertions(+)
tests/test-replication.c index renamed from tests/test-replication.c +0 -0

f085d1993a..5addfc2227 100644

tests/test-replication.c static void test_secondary_stop(void) · line 492

Quoted 4 lines
tests/test-replication.c +1 -0
@@ -489,6 +489,56 @@ static void test_secondary_stop(void)
     teardown_secondary();
 }
 
+static void test_secondary_failover_then_ignore_requests(void)

Same as title, I think change to "test_secondary_continuous_backup" is more clear.

Thanks
Zhang Chen

Quoted 64 lines
tests/test-replication.c +51 -0
+{
+    BlockBackend *top_blk, *local_blk;
+    Error *local_err = NULL;
+
+    top_blk = start_secondary();
+    replication_start_all(REPLICATION_MODE_SECONDARY, &local_err);
+    g_assert(!local_err);
+
+    /* write 0x22 to s_local_disk (IMG_SIZE / 2, IMG_SIZE) */
+    local_blk = blk_by_name(S_LOCAL_DISK_ID);
+    test_blk_write(local_blk, 0x22, IMG_SIZE / 2, IMG_SIZE / 2, false);
+
+    /* replication will backup s_local_disk to s_hidden_disk */
+    test_blk_read(top_blk, 0x11, IMG_SIZE / 2,
+                  IMG_SIZE / 2, 0, IMG_SIZE, false);
+
+    /* write 0x33 to s_active_disk (0, IMG_SIZE / 2) */
+    test_blk_write(top_blk, 0x33, 0, IMG_SIZE / 2, false);
+
+    /* do failover (active commit) */
+    replication_stop_all(true, &local_err);
+    g_assert(!local_err);
+
+    /* it should ignore all requests from now on */
+
+    /* start after failover */
+    replication_start_all(REPLICATION_MODE_PRIMARY, &local_err);
+    g_assert(!local_err);
+
+    /* checkpoint */
+    replication_do_checkpoint_all(&local_err);
+    g_assert(!local_err);
+
+    /* stop */
+    replication_stop_all(true, &local_err);
+    g_assert(!local_err);
+
+    /* read from s_local_disk (0, IMG_SIZE / 2) */
+    test_blk_read(top_blk, 0x33, 0, IMG_SIZE / 2,
+                  0, IMG_SIZE / 2, false);
+
+
+    /* read from s_local_disk (IMG_SIZE / 2, IMG_SIZE) */
+    test_blk_read(top_blk, 0x22, IMG_SIZE / 2,
+                  IMG_SIZE / 2, 0, IMG_SIZE, false);
+
+    teardown_secondary();
+}
+
 static void test_secondary_do_checkpoint(void)
 {
     BlockBackend *top_blk, *local_blk;
@@ -584,6 +634,8 @@ int main(int argc, char **argv)
     g_test_add_func("/replication/secondary/write", test_secondary_write);
     g_test_add_func("/replication/secondary/start", test_secondary_start);
     g_test_add_func("/replication/secondary/stop",  test_secondary_stop);
+
 g_test_add_func("/replication/secondary/failover_then_ignore_requests",
+                    test_secondary_failover_then_ignore_requests);
     g_test_add_func("/replication/secondary/do_checkpoint",
                     test_secondary_do_checkpoint);

g_test_add_func("/replication/secondary/get_error_all",
--
2.20.1

#6 lore raw

RE: [PATCH v5 2/4] tests/test-replication.c: Add test for ignoring requests after failover

Quoted 28 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Saturday, September 28, 2019 7:07 PM
To: Zhang, Chen <[email protected]>
Cc: qemu-devel <[email protected]>; Jason Wang <[email protected]>; Wen Congyang <[email protected]>; Xie Changlong <[email protected]>; [email protected]; [email protected]
Subject: Re: [PATCH v5 2/4] tests/test-replication.c: Add test for ignoring requests after failover

Quoted 6 lines · Chen
Quoted 6 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Monday, September 16, 2019 3:20 AM
To: qemu-devel <[email protected]>
Cc: Zhang, Chen <[email protected]>; Jason Wang <[email protected]>; Wen Congyang

<[email protected]>; Xie

Quoted 6 lines
Quoted 6 lines

Changlong <[email protected]>; [email protected]; [email protected]
Subject: [PATCH v5 2/4] tests/test-replication.c: Add test for ignoring requests after failover

This simulates the case that happens when we resume COLO after

Quoted 8 lines

failover.

Quoted 3 lines
Quoted 1 line

It looks change the title to "Add test for secondary node continuous

backup" is better.

Did you mean "continuous replication"? Would "Add test for secondary node continuing replication" be Ok?

OK for me.

Thanks
Zhang Chen

Quoted 94 lines
Quoted 16 lines
Quoted 16 lines
---
 tests/test-replication.c | 52
++++++++++++++++++++++++++++++++++++++++
 1 file changed, 52 insertions(+)
tests/test-replication.c +0 -0

index
f085d1993a..5addfc2227 100644

tests/test-replication.c +0 -0
@@ -489,6 +489,56 @@ static void test_secondary_stop(void)
     teardown_secondary();
 }
 
Quoted 3 lines

tests/test-replication.c static void test_secondary_stop(void) · line 492

Quoted 1 line
tests/test-replication.c +1 -0
+static void test_secondary_failover_then_ignore_requests(void)

Same as title, I think change to "test_secondary_continuous_backup" is

more clear.

Quoted 51 lines

Thanks
Zhang Chen

Quoted 47 lines
tests/test-replication.c +47 -0
+{
+    BlockBackend *top_blk, *local_blk;
+    Error *local_err = NULL;
+
+    top_blk = start_secondary();
+    replication_start_all(REPLICATION_MODE_SECONDARY, &local_err);
+    g_assert(!local_err);
+
+    /* write 0x22 to s_local_disk (IMG_SIZE / 2, IMG_SIZE) */
+    local_blk = blk_by_name(S_LOCAL_DISK_ID);
+    test_blk_write(local_blk, 0x22, IMG_SIZE / 2, IMG_SIZE / 2,
+ false);
+
+    /* replication will backup s_local_disk to s_hidden_disk */
+    test_blk_read(top_blk, 0x11, IMG_SIZE / 2,
+                  IMG_SIZE / 2, 0, IMG_SIZE, false);
+
+    /* write 0x33 to s_active_disk (0, IMG_SIZE / 2) */
+    test_blk_write(top_blk, 0x33, 0, IMG_SIZE / 2, false);
+
+    /* do failover (active commit) */
+    replication_stop_all(true, &local_err);
+    g_assert(!local_err);
+
+    /* it should ignore all requests from now on */
+
+    /* start after failover */
+    replication_start_all(REPLICATION_MODE_PRIMARY, &local_err);
+    g_assert(!local_err);
+
+    /* checkpoint */
+    replication_do_checkpoint_all(&local_err);
+    g_assert(!local_err);
+
+    /* stop */
+    replication_stop_all(true, &local_err);
+    g_assert(!local_err);
+
+    /* read from s_local_disk (0, IMG_SIZE / 2) */
+    test_blk_read(top_blk, 0x33, 0, IMG_SIZE / 2,
+                  0, IMG_SIZE / 2, false);
+
+
+    /* read from s_local_disk (IMG_SIZE / 2, IMG_SIZE) */
+    test_blk_read(top_blk, 0x22, IMG_SIZE / 2,
+                  IMG_SIZE / 2, 0, IMG_SIZE, false);
+
Quoted 8 lines
Quoted 8 lines
tests/test-replication.c +3 -0
+    teardown_secondary();
+}
+
 static void test_secondary_do_checkpoint(void)
 {
     BlockBackend *top_blk, *local_blk; @@ -584,6 +634,8 @@ int

main(int argc, char **argv)
g_test_add_func("/replication/secondary/write",

test_secondary_write);

Quoted 1 line
Quoted 1 line

g_test_add_func("/replication/secondary/start",

test_secondary_start);

Quoted 12 lines
Quoted 11 lines

g_test_add_func("/replication/secondary/stop",
test_secondary_stop);
+ g_test_add_func("/replication/secondary/failover_then_ignore_request s",

+                    test_secondary_failover_then_ignore_requests);
     g_test_add_func("/replication/secondary/do_checkpoint",
                     test_secondary_do_checkpoint);
     g_test_add_func("/replication/secondary/get_error_all",

--
2.20.1

#7 lore raw

PATCH v5 3/4 net/filter.c: Add Options to insert filters anywhere in the filter list

In reply to Lukas Straub

To switch the Secondary to Primary, we need to insert new filters before the filter-rewriter.

Add the options insert= and position= to be able to insert filters anywhere in the filter list.

position should be "head" or "tail" to insert at the head or tail of the filter list or it should be "id=<id>" to specify the id of another filter.
insert should be either "before" or "behind" to specify where to insert the new filter relative to the one specified with position.

---
 include/net/filter.h |  2 +
 net/filter.c         | 92 +++++++++++++++++++++++++++++++++++++++++++-
 qemu-options.hx      | 10 ++---
 3 files changed, 98 insertions(+), 6 deletions(-)
include/net/filter.h +2 -0
index 49da666ac0..22a723305b 100644
@@ -62,6 +62,8 @@ struct NetFilterState {
     NetClientState *netdev;
     NetFilterDirection direction;
     bool on;
+    char *position;
+    bool insert_before_flag;
     QTAILQ_ENTRY(NetFilterState) next;
 };
 
net/filter.c +91 -1

132 lines not on this page · raw message

qemu-options.hx +5 -5
index 08749a3391..23fa5a344e 100644
@@ -4368,7 +4368,7 @@ applications, they can do this through this parameter. Its format is
 a gnutls priority string as described at
 @url{https://gnutls.org/manual/html_node/Priority-Strings.html}.
 
-@item -object filter-buffer,id=@var{id},netdev=@var{netdevid},interval=@var{t}[,queue=@var{all|rx|tx}][,status=@var{on|off}]
+@item -object filter-buffer,id=@var{id},netdev=@var{netdevid},interval=@var{t}[,queue=@var{all|rx|tx}][,status=@var{on|off}][,position=@var{head|tail|id=<id>}][,insert=@var{behind|before}]
 
 Interval @var{t} can't be 0, this filter batches the packet delivery: all
 packets arriving in a given interval on netdev @var{netdevid} are delayed
@@ -4387,11 +4387,11 @@ queue @var{all|rx|tx} is an option that can be applied to any netfilter.
 @option{tx}: the filter is attached to the transmit queue of the netdev,
              where it will receive packets sent by the netdev.
 
-@item -object filter-mirror,id=@var{id},netdev=@var{netdevid},outdev=@var{chardevid},queue=@var{all|rx|tx}[,vnet_hdr_support]
+@item -object filter-mirror,id=@var{id},netdev=@var{netdevid},outdev=@var{chardevid},queue=@var{all|rx|tx}[,vnet_hdr_support][,position=@var{head|tail|id=<id>}][,insert=@var{behind|before}]
 
 filter-mirror on netdev @var{netdevid},mirror net packet to chardev@var{chardevid}, if it has the vnet_hdr_support flag, filter-mirror will mirror packet with vnet_hdr_len.
 
-@item -object filter-redirector,id=@var{id},netdev=@var{netdevid},indev=@var{chardevid},outdev=@var{chardevid},queue=@var{all|rx|tx}[,vnet_hdr_support]
+@item -object filter-redirector,id=@var{id},netdev=@var{netdevid},indev=@var{chardevid},outdev=@var{chardevid},queue=@var{all|rx|tx}[,vnet_hdr_support][,position=@var{head|tail|id=<id>}][,insert=@var{behind|before}]
 
 filter-redirector on netdev @var{netdevid},redirect filter's net packet to chardev
 @var{chardevid},and redirect indev's packet to filter.if it has the vnet_hdr_support flag,
@@ -4400,7 +4400,7 @@ Create a filter-redirector we need to differ outdev id from indev id, id can not
 be the same. we can just use indev or outdev, but at least one of indev or outdev
 need to be specified.
 
-@item -object filter-rewriter,id=@var{id},netdev=@var{netdevid},queue=@var{all|rx|tx},[vnet_hdr_support]
+@item -object filter-rewriter,id=@var{id},netdev=@var{netdevid},queue=@var{all|rx|tx},[vnet_hdr_support][,position=@var{head|tail|id=<id>}][,insert=@var{behind|before}]
 
 Filter-rewriter is a part of COLO project.It will rewrite tcp packet to
 secondary from primary to keep secondary tcp connection,and rewrite
@@ -4413,7 +4413,7 @@ colo secondary:
 -object filter-redirector,id=f2,netdev=hn0,queue=rx,outdev=red1
 -object filter-rewriter,id=rew0,netdev=hn0,queue=all
 
-@item -object filter-dump,id=@var{id},netdev=@var{dev}[,file=@var{filename}][,maxlen=@var{len}]
+@item -object filter-dump,id=@var{id},netdev=@var{dev}[,file=@var{filename}][,maxlen=@var{len}][,position=@var{head|tail|id=<id>}][,insert=@var{behind|before}]
 
 Dump the network traffic on netdev @var{dev} to the file specified by
 @var{filename}. At most @var{len} bytes (64k by default) per packet are stored.

--
2.20.1

Reviewed-by on v5 3/4
#8 lore raw

RE: [PATCH v5 3/4] net/filter.c: Add Options to insert filters anywhere in the filter list

Quoted 215 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Monday, September 16, 2019 3:20 AM
To: qemu-devel <[email protected]>
Cc: Zhang, Chen <[email protected]>; Jason Wang <[email protected]>; Wen Congyang <[email protected]>;
Xie Changlong <[email protected]>; [email protected]; [email protected]
Subject: [PATCH v5 3/4] net/filter.c: Add Options to insert filters anywhere in the filter list

To switch the Secondary to Primary, we need to insert new filters before the filter-rewriter.

Add the options insert= and position= to be able to insert filters anywhere in the filter list.

position should be "head" or "tail" to insert at the head or tail of the filter list or it should be "id=<id>" to specify the id of another filter.
insert should be either "before" or "behind" to specify where to insert the new filter relative to the one specified with position.

---
 include/net/filter.h |  2 +
 net/filter.c         | 92
+++++++++++++++++++++++++++++++++++++++++++-
 qemu-options.hx      | 10 ++---
 3 files changed, 98 insertions(+), 6 deletions(-)
include/net/filter.h index renamed from include/net/filter.h +0 -0

49da666ac0..22a723305b 100644

include/net/filter.h +2 -0
@@ -62,6 +62,8 @@ struct NetFilterState {
     NetClientState *netdev;
     NetFilterDirection direction;
     bool on;
+    char *position;
+    bool insert_before_flag;
     QTAILQ_ENTRY(NetFilterState) next;
 };
 
net/filter.c +33 -0
@@ -171,11 +171,47 @@ static void netfilter_set_status(Object *obj, const
 char *str, Error **errp)
     }
 }
 
+static char *netfilter_get_position(Object *obj, Error **errp) {
+    NetFilterState *nf = NETFILTER(obj);
+
+    return g_strdup(nf->position);
+}
+
+static void netfilter_set_position(Object *obj, const char *str, Error
+**errp) {
+    NetFilterState *nf = NETFILTER(obj);
+
+    nf->position = g_strdup(str);
+}
+
+static char *netfilter_get_insert(Object *obj, Error **errp) {
+    NetFilterState *nf = NETFILTER(obj);
+
+    return nf->insert_before_flag ? g_strdup("before") :
+g_strdup("behind"); }
+
+static void netfilter_set_insert(Object *obj, const char *str, Error
+**errp) {
+    NetFilterState *nf = NETFILTER(obj);
+
+    if (strcmp(str, "before") && strcmp(str, "behind")) {
+        error_setg(errp, "Invalid value for netfilter insert, "
+                         "should be 'before' or 'behind'");
+        return;
+    }
+
+    nf->insert_before_flag = !strcmp(str, "before"); }
+
 static void netfilter_init(Object *obj)  {
     NetFilterState *nf = NETFILTER(obj);
 
     nf->on = true;
+    nf->insert_before_flag = false;
+    nf->position = g_strdup("tail");
 
     object_property_add_str(obj, "netdev",
                             netfilter_get_netdev_id, netfilter_set_netdev_id, @@ -

187,11 +223,18 @@ static void netfilter_init(Object *obj)

     object_property_add_str(obj, "status",
                             netfilter_get_status, netfilter_set_status,
                             NULL);
net/filter.c +55 -1
+    object_property_add_str(obj, "position",
+                            netfilter_get_position, netfilter_set_position,
+                            NULL);
+    object_property_add_str(obj, "insert",
+                            netfilter_get_insert, netfilter_set_insert,
+                            NULL);
 }
 
 static void netfilter_complete(UserCreatable *uc, Error **errp)  {
     NetFilterState *nf = NETFILTER(uc);
+    NetFilterState *position = NULL;
     NetClientState *ncs[MAX_QUEUE_NUM];
     NetFilterClass *nfc = NETFILTER_GET_CLASS(uc);
     int queues;
@@ -219,6 +262,41 @@ static void netfilter_complete(UserCreatable *uc,
 Error **errp)
         return;
     }
 
+    if (strcmp(nf->position, "head") && strcmp(nf->position, "tail")) {
+        Object *container;
+        Object *obj;
+        char *position_id;
+
+        if (!g_str_has_prefix(nf->position, "id=")) {
+            error_setg(errp, QERR_INVALID_PARAMETER_VALUE, "position",
+                       "'head', 'tail' or 'id=<id>'");
+            return;
+        }
+
+        /* get the id from the string */
+        position_id = g_strndup(nf->position + 3, strlen(nf->position)
+ - 3);
+
+        /* Search for the position to insert before/behind */
+        container = object_get_objects_root();
+        obj = object_resolve_path_component(container, position_id);
+        if (!obj) {
+            error_setg(errp, "filter '%s' not found", position_id);
+            g_free(position_id);
+            return;
+        }
+
+        position = NETFILTER(obj);
+
+        if (position->netdev != ncs[0]) {
+            error_setg(errp, "filter '%s' belongs to a different netdev",
+                        position_id);
+            g_free(position_id);
+            return;
+        }
+
+        g_free(position_id);
+    }
+
     nf->netdev = ncs[0];
 
     if (nfc->setup) {
@@ -228,7 +306,18 @@ static void netfilter_complete(UserCreatable *uc,
 Error **errp)
             return;
         }
     }
-    QTAILQ_INSERT_TAIL(&nf->netdev->filters, nf, next);
+
+    if (position) {
+        if (nf->insert_before_flag) {
+            QTAILQ_INSERT_BEFORE(position, nf, next);
+        } else {
+            QTAILQ_INSERT_AFTER(&nf->netdev->filters, position, nf, next);
+        }
+    } else if (!strcmp(nf->position, "head")) {
+        QTAILQ_INSERT_HEAD(&nf->netdev->filters, nf, next);
+    } else if (!strcmp(nf->position, "tail")) {
+        QTAILQ_INSERT_TAIL(&nf->netdev->filters, nf, next);
+    }
 }
 

static void netfilter_finalize(Object *obj) @@ -245,6 +334,7 @@ static void
netfilter_finalize(Object *obj)

         QTAILQ_REMOVE(&nf->netdev->filters, nf, next);
     }
     g_free(nf->netdev_id);
net/filter.c +1 -0
+    g_free(nf->position);
 }
 
 static void default_handle_event(NetFilterState *nf, int event, Error **errp)
qemu-options.hx index renamed from qemu-options.hx +0 -0

08749a3391..23fa5a344e 100644

qemu-options.hx +0 -0
@@ -4368,7 +4368,7 @@ applications, they can do this through this

parameter. Its format is a gnutls priority string as described at @url{https://gnutls.org/manual/html_node/Priority-Strings.html}.

-@item -object filter- buffer,id=@var{id},netdev=@var{netdevid},interval=@var{t}[,queue=@var{ all|rx|tx}][,status=@var{on|off}]
+@item -object +filter-buffer,id=@var{id},netdev=@var{netdevid},interval=@var{t}[,queue

+=@var{all|rx|tx}][,status=@var{on|off}][,position=@var{head|tail|id=<id
+>}][,insert=@var{behind|before}]

Interval @var{t} can't be 0, this filter batches the packet delivery: all packets
arriving in a given interval on netdev @var{netdevid} are delayed @@ - 4387,11 +4387,11 @@ queue @var{all|rx|tx} is an option that can be applied to any netfilter.
@option{tx}: the filter is attached to the transmit queue of the netdev,
where it will receive packets sent by the netdev.

-@item -object filter- mirror,id=@var{id},netdev=@var{netdevid},outdev=@var{chardevid},queue =@var{all|rx|tx}[,vnet_hdr_support]
+@item -object

qemu-options.hx +2 -0
+filter-mirror,id=@var{id},netdev=@var{netdevid},outdev=@var{chardevid},
+queue=@var{all|rx|tx}[,vnet_hdr_support][,position=@var{head|tail|id=<

i
+d>}][,insert=@var{behind|before}]

Quoted 4 lines

filter-mirror on netdev @var{netdevid},mirror net packet to
chardev@var{chardevid}, if it has the vnet_hdr_support flag, filter-mirror will mirror packet with vnet_hdr_len.

Please add description for the newly added parameter in each filter. After that:

Thanks
Zhang Chen

Quoted 44 lines

-@item -object filter- redirector,id=@var{id},netdev=@var{netdevid},indev=@var{chardevid},out dev=@var{chardevid},queue=@var{all|rx|tx}[,vnet_hdr_support] +@item -object

qemu-options.hx +2 -0
+filter-redirector,id=@var{id},netdev=@var{netdevid},indev=@var{chardevi
+d},outdev=@var{chardevid},queue=@var{all|rx|tx}[,vnet_hdr_support][,p

os +ition=@var{head|tail|id=<id>}][,insert=@var{behind|before}]

filter-redirector on netdev @var{netdevid},redirect filter's net packet to
chardev @var{chardevid},and redirect indev's packet to filter.if it has the vnet_hdr_support flag, @@ -4400,7 +4400,7 @@ Create a filter-redirector we need to differ outdev id from indev id, id can not be the same. we can just use indev or outdev, but at least one of indev or outdev need to be specified.

-@item -object filter- rewriter,id=@var{id},netdev=@var{netdevid},queue=@var{all|rx|tx},[vnet_ hdr_support]
+@item -object

qemu-options.hx +3 -0
+filter-rewriter,id=@var{id},netdev=@var{netdevid},queue=@var{all|rx|tx}
+,[vnet_hdr_support][,position=@var{head|tail|id=<id>}][,insert=@var{beh
+ind|before}]
 
 Filter-rewriter is a part of COLO project.It will rewrite tcp packet to

secondary from primary to keep secondary tcp connection,and rewrite @@ - 4413,7 +4413,7 @@ colo secondary:
-object filter-redirector,id=f2,netdev=hn0,queue=rx,outdev=red1 -object filter-rewriter,id=rew0,netdev=hn0,queue=all

-@item -object filter- dump,id=@var{id},netdev=@var{dev}[,file=@var{filename}][,maxlen=@var{ len}]
+@item -object
+filter- dump,id=@var{id},netdev=@var{dev}[,file=@var{filename}][,maxlen= +@var{len}][,position=@var{head|tail|id=<id>}][,insert=@var{behind|befor +e}]

Dump the network traffic on netdev @var{dev} to the file specified by
@var{filename}. At most @var{len} bytes (64k by default) per packet are stored.
--
2.20.1

#9 lore raw

Re: [PATCH v5 3/4] net/filter.c: Add Options to insert filters anywhere in the filter list

Quoted 39 lines · Chen
Quoted 32 lines
qemu-options.hx index renamed from qemu-options.hx +0 -0

08749a3391..23fa5a344e 100644

qemu-options.hx +0 -0
@@ -4368,7 +4368,7 @@ applications, they can do this through this

parameter. Its format is a gnutls priority string as described at @url{https://gnutls.org/manual/html_node/Priority-Strings.html}.

-@item -object filter- buffer,id=@var{id},netdev=@var{netdevid},interval=@var{t}[,queue=@var{ all|rx|tx}][,status=@var{on|off}]
+@item -object +filter-buffer,id=@var{id},netdev=@var{netdevid},interval=@var{t}[,queue

+=@var{all|rx|tx}][,status=@var{on|off}][,position=@var{head|tail|id=<id
+>}][,insert=@var{behind|before}]

Interval @var{t} can't be 0, this filter batches the packet delivery: all packets
arriving in a given interval on netdev @var{netdevid} are delayed @@ - 4387,11 +4387,11 @@ queue @var{all|rx|tx} is an option that can be applied to any netfilter.
@option{tx}: the filter is attached to the transmit queue of the netdev,
where it will receive packets sent by the netdev.

-@item -object filter- mirror,id=@var{id},netdev=@var{netdevid},outdev=@var{chardevid},queue =@var{all|rx|tx}[,vnet_hdr_support]
+@item -object

qemu-options.hx +2 -0
+filter-mirror,id=@var{id},netdev=@var{netdevid},outdev=@var{chardevid},
+queue=@var{all|rx|tx}[,vnet_hdr_support][,position=@var{head|tail|id=<

i
+d>}][,insert=@var{behind|before}]

Quoted 4 lines

filter-mirror on netdev @var{netdevid},mirror net packet to
chardev@var{chardevid}, if it has the vnet_hdr_support flag, filter-mirror will mirror packet with vnet_hdr_len.

Please add description for the newly added parameter in each filter. After that:

Quoted 4 lines

Thanks
Zhang Chen

Hi,
I will add a single description like its currently done with the "queue" option, noting that it applies to any netfilter. Is that Ok?

Regards,
Lukas Straub

Quoted 46 lines
Quoted 45 lines

-@item -object filter- redirector,id=@var{id},netdev=@var{netdevid},indev=@var{chardevid},out dev=@var{chardevid},queue=@var{all|rx|tx}[,vnet_hdr_support] +@item -object

qemu-options.hx +2 -0
+filter-redirector,id=@var{id},netdev=@var{netdevid},indev=@var{chardevi
+d},outdev=@var{chardevid},queue=@var{all|rx|tx}[,vnet_hdr_support][,p

os +ition=@var{head|tail|id=<id>}][,insert=@var{behind|before}]

filter-redirector on netdev @var{netdevid},redirect filter's net packet to
chardev @var{chardevid},and redirect indev's packet to filter.if it has the vnet_hdr_support flag, @@ -4400,7 +4400,7 @@ Create a filter-redirector we need to differ outdev id from indev id, id can not be the same. we can just use indev or outdev, but at least one of indev or outdev need to be specified.

-@item -object filter- rewriter,id=@var{id},netdev=@var{netdevid},queue=@var{all|rx|tx},[vnet_ hdr_support]
+@item -object

qemu-options.hx +3 -0
+filter-rewriter,id=@var{id},netdev=@var{netdevid},queue=@var{all|rx|tx}
+,[vnet_hdr_support][,position=@var{head|tail|id=<id>}][,insert=@var{beh
+ind|before}]
 
 Filter-rewriter is a part of COLO project.It will rewrite tcp packet to

secondary from primary to keep secondary tcp connection,and rewrite @@ - 4413,7 +4413,7 @@ colo secondary:
-object filter-redirector,id=f2,netdev=hn0,queue=rx,outdev=red1 -object filter-rewriter,id=rew0,netdev=hn0,queue=all

-@item -object filter- dump,id=@var{id},netdev=@var{dev}[,file=@var{filename}][,maxlen=@var{ len}]
+@item -object
+filter- dump,id=@var{id},netdev=@var{dev}[,file=@var{filename}][,maxlen= +@var{len}][,position=@var{head|tail|id=<id>}][,insert=@var{behind|befor +e}]

Dump the network traffic on netdev @var{dev} to the file specified by
@var{filename}. At most @var{len} bytes (64k by default) per packet are stored.
--
2.20.1

#10 lore raw

RE: [PATCH v5 3/4] net/filter.c: Add Options to insert filters anywhere in the filter list

Quoted 64 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Saturday, September 28, 2019 6:45 PM
To: Zhang, Chen <[email protected]>
Cc: qemu-devel <[email protected]>; Jason Wang <[email protected]>; Wen Congyang <[email protected]>; Xie Changlong <[email protected]>; [email protected]; [email protected]
Subject: Re: [PATCH v5 3/4] net/filter.c: Add Options to insert filters anywhere in the filter list

Quoted 10 lines · Chen
Quoted 10 lines
qemu-options.hx index renamed from qemu-options.hx +0 -0

08749a3391..23fa5a344e 100644

qemu-options.hx +0 -0
@@ -4368,7 +4368,7 @@ applications, they can do this through this

parameter. Its format is a gnutls priority string as described at @url{https://gnutls.org/manual/html_node/Priority-Strings.html}.

-@item -object filter-

buffer,id=@var{id},netdev=@var{netdevid},interval=@var{t}[,queue=@va

Quoted 17 lines
Quoted 17 lines

r{
all|rx|tx}][,status=@var{on|off}]
+@item -object

qemu-options.hx +5 -0
+filter-buffer,id=@var{id},netdev=@var{netdevid},interval=@var{t}[,q
+ueue
+=@var{all|rx|tx}][,status=@var{on|off}][,position=@var{head|tail|id
+=<id
+>}][,insert=@var{behind|before}]
 
 Interval @var{t} can't be 0, this filter batches the packet

delivery: all packets arriving in a given interval on netdev @var{netdevid} are delayed @@ -
4387,11 +4387,11 @@ queue @var{all|rx|tx} is an option that can be applied to any netfilter.
@option{tx}: the filter is attached to the transmit queue of the netdev,
where it will receive packets sent by the netdev.

Quoted 2 lines
Quoted 2 lines

-@item -object filter-

mirror,id=@var{id},netdev=@var{netdevid},outdev=@var{chardevid},queu

Quoted 4 lines
Quoted 4 lines

e
=@var{all|rx|tx}[,vnet_hdr_support]
+@item -object
+filter-

mirror,id=@var{id},netdev=@var{netdevid},outdev=@var{chardev

Quoted 15 lines
Quoted 10 lines
qemu-options.hx +3 -0
+id},
+queue=@var{all|rx|tx}[,vnet_hdr_support][,position=@var{head|tail|i
+d=<

i
+d>}][,insert=@var{behind|before}]

filter-mirror on netdev @var{netdevid},mirror net packet to
chardev@var{chardevid}, if it has the vnet_hdr_support flag, filter-mirror will mirror packet with vnet_hdr_len.

Please add description for the newly added parameter in each filter. After that:

Quoted 6 lines
Quoted 2 lines

Thanks
Zhang Chen

Hi,
I will add a single description like its currently done with the "queue" option, noting that it applies to any netfilter. Is that Ok?

It is enough for me.

Thanks
Zhang Chen

Quoted 61 lines

Regards,
Lukas Straub

Quoted 3 lines
Quoted 3 lines

-@item -object filter-

redirector,id=@var{id},netdev=@var{netdevid},indev=@var{chardevid},o

Quoted 5 lines

qemu-options.hx

Quoted 5 lines

ut dev=@var{chardevid},queue=@var{all|rx|tx}[,vnet_hdr_support] +@item -object

qemu-options.hx +2 -0
+filter-redirector,id=@var{id},netdev=@var{netdevid},indev=@var{char
+devi

+d},outdev=@var{chardevid},queue=@var{all|rx|tx}[,vnet_hdr_support][

Quoted 29 lines
Quoted 29 lines

+,p
os +ition=@var{head|tail|id=<id>}][,insert=@var{behind|before}]

filter-redirector on netdev @var{netdevid},redirect filter's net
packet to chardev @var{chardevid},and redirect indev's packet to filter.if it has the vnet_hdr_support flag, @@ -4400,7 +4400,7 @@ Create a filter-redirector we need to differ outdev id from indev id, id can not be the same. we can just use indev or outdev, but at least one of indev or outdev need to be specified.

-@item -object filter- rewriter,id=@var{id},netdev=@var{netdevid},queue=@var{all|rx|tx},[vn et_
hdr_support]
+@item -object

qemu-options.hx +5 -0
+filter-rewriter,id=@var{id},netdev=@var{netdevid},queue=@var{all|rx
+|tx}
+,[vnet_hdr_support][,position=@var{head|tail|id=<id>}][,insert=@var
+{beh
+ind|before}]
 
 Filter-rewriter is a part of COLO project.It will rewrite tcp

packet to secondary from primary to keep secondary tcp connection,and rewrite @@ -
4413,7 +4413,7 @@ colo secondary:
-object filter-redirector,id=f2,netdev=hn0,queue=rx,outdev=red1 -object filter-rewriter,id=rew0,netdev=hn0,queue=all

Quoted 2 lines
Quoted 2 lines

-@item -object filter-

dump,id=@var{id},netdev=@var{dev}[,file=@var{filename}][,maxlen=@var

Quoted 15 lines
Quoted 14 lines

{
len}]
+@item -object
+filter- dump,id=@var{id},netdev=@var{dev}[,file=@var{filename}][,maxlen= +@var{len}][,position=@var{head|tail|id=<id>}][,insert=@var{behind|b

qemu-options.hx +2 -0
+efor
+e}]
 
 Dump the network traffic on netdev @var{dev} to the file specified

by @var{filename}. At most @var{len} bytes (64k by default) per packet are stored.
--
2.20.1

#11 lore raw

PATCH v5 4/4 colo: Update Documentation for continuous replication

In reply to Lukas Straub

Document the qemu command-line and qmp commands for continuous replication

---
 docs/COLO-FT.txt           | 212 +++++++++++++++++++++++++++----------
 docs/block-replication.txt |  28 +++--
 2 files changed, 173 insertions(+), 67 deletions(-)
docs/COLO-FT.txt +155 -57
index ad24680d13..e98c95ca2b 100644
@@ -145,35 +145,65 @@ The diagram just shows the main qmp command, you can get the detail
 in test procedure.
 
 == Test procedure ==
-1. Startup qemu
-Primary:
-# qemu-system-x86_64 -accel kvm -m 2048 -smp 2 -qmp stdio -name primary \
-  -device piix3-usb-uhci -vnc :7 \
-  -device usb-tablet -netdev tap,id=hn0,vhost=off \
-  -device virtio-net-pci,id=net-pci0,netdev=hn0 \
-  -drive if=virtio,id=primary-disk0,driver=quorum,read-pattern=fifo,vote-threshold=1,\
-         children.0.file.filename=1.raw,\
-         children.0.driver=raw -S
-Secondary:
-# qemu-system-x86_64 -accel kvm -m 2048 -smp 2 -qmp stdio -name secondary \
-  -device piix3-usb-uhci -vnc :7 \
-  -device usb-tablet -netdev tap,id=hn0,vhost=off \
-  -device virtio-net-pci,id=net-pci0,netdev=hn0 \
-  -drive if=none,id=secondary-disk0,file.filename=1.raw,driver=raw,node-name=node0 \
-  -drive if=virtio,id=active-disk0,driver=replication,mode=secondary,\
-         file.driver=qcow2,top-id=active-disk0,\
-         file.file.filename=/mnt/ramfs/active_disk.img,\
-         file.backing.driver=qcow2,\
-         file.backing.file.filename=/mnt/ramfs/hidden_disk.img,\
-         file.backing.backing=secondary-disk0 \
-  -incoming tcp:0:8888
-
-2. On Secondary VM's QEMU monitor, issue command
+Note: Here we are running both instances on the same Machine for testing,
+change the IP Addresses if you want to run it on two Hosts
+
+== Startup qemu ==
+1. Primary:
+# imagefolder="/mnt/vms/colo-test"
+
+The disks for the primary and secondary need to be the same before starting colo
+# cp --reflink=auto $imagefolder/primary.qcow2 $imagefolder/primary-copy.qcow2
+
+# qemu-system-x86_64 -enable-kvm -cpu qemu64,+kvmclock -m 512 -smp 1 -qmp stdio \
+   -vnc :0 -device piix3-usb-uhci -device usb-tablet -name primary \
+   -netdev tap,id=hn0,vhost=off,helper=/usr/lib/qemu/qemu-bridge-helper \
+   -device rtl8139,id=e0,netdev=hn0 \
+   -chardev socket,id=mirror0,host=127.0.0.1,port=9003,server,nowait \
+   -chardev socket,id=compare1,host=127.0.0.1,port=9004,server,wait \
+   -chardev socket,id=compare0,host=127.0.0.1,port=9001,server,nowait \
+   -chardev socket,id=compare0-0,host=127.0.0.1,port=9001 \
+   -chardev socket,id=compare_out,host=127.0.0.1,port=9005,server,nowait \
+   -chardev socket,id=compare_out0,host=127.0.0.1,port=9005 \
+   -object filter-mirror,id=m0,netdev=hn0,queue=tx,outdev=mirror0 \
+   -object filter-redirector,netdev=hn0,id=redire0,queue=rx,indev=compare_out \
+   -object filter-redirector,netdev=hn0,id=redire1,queue=rx,outdev=compare0 \
+   -object iothread,id=iothread1 \
+   -object colo-compare,id=comp0,primary_in=compare0-0,secondary_in=compare1,\
+outdev=compare_out0,iothread=iothread1 \
+   -drive if=ide,id=colo-disk0,driver=quorum,read-pattern=fifo,vote-threshold=1,\
+children.0.file.filename=$imagefolder/primary.qcow2,children.0.driver=qcow2 -S
+
+2. Secondary:
+# imagefolder="/mnt/vms/colo-test"
+
+# qemu-img create -f qcow2 $imagefolder/secondary-active.qcow2 10G
+
+# qemu-img create -f qcow2 $imagefolder/secondary-hidden.qcow2 10G
+
+# qemu-system-x86_64 -enable-kvm -cpu qemu64,+kvmclock -m 512 -smp 1 -qmp stdio \
+   -vnc :1 -device piix3-usb-uhci -device usb-tablet -name secondary \
+   -netdev tap,id=hn0,vhost=off,helper=/usr/lib/qemu/qemu-bridge-helper \
+   -device rtl8139,id=e0,netdev=hn0 \
+   -chardev socket,id=red0,host=127.0.0.1,port=9003,reconnect=1 \
+   -chardev socket,id=red1,host=127.0.0.1,port=9004,reconnect=1 \
+   -object filter-redirector,id=f1,netdev=hn0,queue=tx,indev=red0 \
+   -object filter-redirector,id=f2,netdev=hn0,queue=rx,outdev=red1 \
+   -object filter-rewriter,id=rew0,netdev=hn0,queue=all \
+   -drive if=none,id=parent0,file.filename=$imagefolder/primary-copy.qcow2,driver=qcow2 \
+   -drive if=none,id=childs0,driver=replication,mode=secondary,file.driver=qcow2,\
+top-id=childs0,file.file.filename=$imagefolder/secondary-active.qcow2,\
+file.backing.driver=qcow2,file.backing.file.filename=$imagefolder/secondary-hidden.qcow2,\
+file.backing.backing=parent0 \
+   -drive if=ide,id=colo-disk0,driver=quorum,read-pattern=fifo,vote-threshold=1,\
+children.0=childs0 \
+   -incoming tcp:0:9998
+
+
+3. On Secondary VM's QEMU monitor, issue command
 {'execute':'qmp_capabilities'}
-{ 'execute': 'nbd-server-start',
-  'arguments': {'addr': {'type': 'inet', 'data': {'host': 'xx.xx.xx.xx', 'port': '8889'} } }
-}
-{'execute': 'nbd-server-add', 'arguments': {'device': 'secondary-disk0', 'writable': true } }
+{'execute': 'nbd-server-start', 'arguments': {'addr': {'type': 'inet', 'data': {'host': '127.0.0.1', 'port': '9999'} } } }
+{'execute': 'nbd-server-add', 'arguments': {'device': 'parent0', 'writable': true } }
 
 Note:
   a. The qmp command nbd-server-start and nbd-server-add must be run
@@ -182,44 +212,112 @@ Note:
      same.
   c. It is better to put active disk and hidden disk in ramdisk.
 
-3. On Primary VM's QEMU monitor, issue command:
+4. On Primary VM's QEMU monitor, issue command:
 {'execute':'qmp_capabilities'}
-{ 'execute': 'human-monitor-command',
-  'arguments': {'command-line': 'drive_add -n buddy driver=replication,mode=primary,file.driver=nbd,file.host=xx.xx.xx.xx,file.port=8889,file.export=secondary-disk0,node-name=nbd_client0'}}
-{ 'execute':'x-blockdev-change', 'arguments':{'parent': 'primary-disk0', 'node': 'nbd_client0' } }
-{ 'execute': 'migrate-set-capabilities',
-      'arguments': {'capabilities': [ {'capability': 'x-colo', 'state': true } ] } }
-{ 'execute': 'migrate', 'arguments': {'uri': 'tcp:xx.xx.xx.xx:8888' } }
+{'execute': 'human-monitor-command', 'arguments': {'command-line': 'drive_add -n buddy driver=replication,mode=primary,file.driver=nbd,file.host=127.0.0.1,file.port=9999,file.export=parent0,node-name=replication0'}}
+{'execute': 'x-blockdev-change', 'arguments':{'parent': 'colo-disk0', 'node': 'replication0' } }
+{'execute': 'migrate-set-capabilities', 'arguments': {'capabilities': [ {'capability': 'x-colo', 'state': true } ] } }
+{'execute': 'migrate', 'arguments': {'uri': 'tcp:127.0.0.1:9998' } }
 
   Note:
   a. There should be only one NBD Client for each primary disk.
-  b. xx.xx.xx.xx is the secondary physical machine's hostname or IP
-  c. The qmp command line must be run after running qmp command line in
+  b. The qmp command line must be run after running qmp command line in
      secondary qemu.
 
-4. After the above steps, you will see, whenever you make changes to PVM, SVM will be synced.
+5. After the above steps, you will see, whenever you make changes to PVM, SVM will be synced.
 You can issue command '{ "execute": "migrate-set-parameters" , "arguments":{ "x-checkpoint-delay": 2000 } }'
-to change the checkpoint period time
+to change the idle checkpoint period time
+
+6. Failover test
+You can kill one of the VMs and Failover on the surviving VM:
+
+If you killed the Secondary, then follow "Primary Failover". After that,
+if you want to resume the replication, follow "Primary resume replication"
+
+If you killed the Primary, then follow "Secondary Failover". After that,
+if you want to resume the replication, follow "Secondary resume replication"
+
+== Primary Failover ==
+The Secondary died, resume on the Primary
+
+{'execute': 'x-blockdev-change', 'arguments':{ 'parent': 'colo-disk0', 'child': 'children.1'} }
+{'execute': 'human-monitor-command', 'arguments':{ 'command-line': 'drive_del replication0' } }
+{'execute': 'object-del', 'arguments':{ 'id': 'comp0' } }
+{'execute': 'object-del', 'arguments':{ 'id': 'iothread1' } }
+{'execute': 'object-del', 'arguments':{ 'id': 'm0' } }
+{'execute': 'object-del', 'arguments':{ 'id': 'redire0' } }
+{'execute': 'object-del', 'arguments':{ 'id': 'redire1' } }
+{'execute': 'x-colo-lost-heartbeat' }
+
+== Secondary Failover ==
+The Primary died, resume on the Secondary and prepare to become the new Primary
+
+{'execute': 'nbd-server-stop'}
+{'execute': 'x-colo-lost-heartbeat'}
+
+{'execute': 'object-del', 'arguments':{ 'id': 'f2' } }
+{'execute': 'object-del', 'arguments':{ 'id': 'f1' } }
+{'execute': 'chardev-remove', 'arguments':{ 'id': 'red1' } }
+{'execute': 'chardev-remove', 'arguments':{ 'id': 'red0' } }
+
+{'execute': 'chardev-add', 'arguments':{ 'id': 'mirror0', 'backend': {'type': 'socket', 'data': {'addr': { 'type': 'inet', 'data': { 'host': '127.0.0.1', 'port': '9003' } }, 'server': true } } } }
+{'execute': 'chardev-add', 'arguments':{ 'id': 'compare1', 'backend': {'type': 'socket', 'data': {'addr': { 'type': 'inet', 'data': { 'host': '127.0.0.1', 'port': '9004' } }, 'server': true } } } }
+{'execute': 'chardev-add', 'arguments':{ 'id': 'compare0', 'backend': {'type': 'socket', 'data': {'addr': { 'type': 'inet', 'data': { 'host': '127.0.0.1', 'port': '9001' } }, 'server': true } } } }
+{'execute': 'chardev-add', 'arguments':{ 'id': 'compare0-0', 'backend': {'type': 'socket', 'data': {'addr': { 'type': 'inet', 'data': { 'host': '127.0.0.1', 'port': '9001' } }, 'server': false } } } }
+{'execute': 'chardev-add', 'arguments':{ 'id': 'compare_out', 'backend': {'type': 'socket', 'data': {'addr': { 'type': 'inet', 'data': { 'host': '127.0.0.1', 'port': '9005' } }, 'server': true } } } }
+{'execute': 'chardev-add', 'arguments':{ 'id': 'compare_out0', 'backend': {'type': 'socket', 'data': {'addr': { 'type': 'inet', 'data': { 'host': '127.0.0.1', 'port': '9005' } }, 'server': false } } } }
+
+== Primary resume replication ==
+Resume replication after new Secondary is up.
+
+Start the new Secondary (Steps 2 and 3 above), then:
+{'execute': 'drive-mirror', 'arguments':{ 'device': 'colo-disk0', 'job-id': 'resync', 'target': 'nbd://127.0.0.1:9999/parent0', 'mode': 'existing', 'format': 'raw', 'sync': 'full'} }
+
+Wait until disk is synced, then:
+{'execute': 'stop'}
+{'execute': 'block-job-cancel', 'arguments':{ 'device': 'resync'} }
+
+{'execute': 'human-monitor-command', 'arguments':{ 'command-line': 'drive_add -n buddy driver=replication,mode=primary,file.driver=nbd,file.host=127.0.0.1,file.port=9999,file.export=parent0,node-name=replication0'}}
+{'execute': 'x-blockdev-change', 'arguments':{ 'parent': 'colo-disk0', 'node': 'replication0' } }
+
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'filter-mirror', 'id': 'm0', 'props': { 'netdev': 'hn0', 'queue': 'tx', 'outdev': 'mirror0' } } }
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'filter-redirector', 'id': 'redire0', 'props': { 'netdev': 'hn0', 'queue': 'rx', 'indev': 'compare_out' } } }
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'filter-redirector', 'id': 'redire1', 'props': { 'netdev': 'hn0', 'queue': 'rx', 'outdev': 'compare0' } } }
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'iothread', 'id': 'iothread1' } }
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'colo-compare', 'id': 'comp0', 'props': { 'primary_in': 'compare0-0', 'secondary_in': 'compare1', 'outdev': 'compare_out0', 'iothread': 'iothread1' } } }
+
+{'execute': 'migrate-set-capabilities', 'arguments':{ 'capabilities': [ {'capability': 'x-colo', 'state': true } ] } }
+{'execute': 'migrate', 'arguments':{ 'uri': 'tcp:127.0.0.1:9998' } }
+
+Note:
+If this Primary previously was a Secondary, then we need to insert the
+filters before the filter-rewriter by using the
+"'insert': 'before', 'position': 'rew0'" Options. See below.
+
+== Secondary resume replication ==
+Become Primary and resume replication after new Secondary is up.
+
+Start the new Secondary (Steps 2 and 3 above, but with different image than
+primary-copy.qcow2 and different VNC port), then:
+{'execute': 'drive-mirror', 'arguments':{ 'device': 'colo-disk0', 'job-id': 'resync', 'target': 'nbd://127.0.0.1:9999/parent0', 'mode': 'existing', 'format': 'raw', 'sync': 'full'} }
+
+Wait until disk is synced, then:
+{'execute': 'stop'}
+{'execute': 'block-job-cancel', 'arguments':{ 'device': 'resync' } }
 
-5. Failover test
-You can kill Primary VM and run 'x_colo_lost_heartbeat' in Secondary VM's
-monitor at the same time, then SVM will failover and client will not detect this
-change.
+{'execute': 'human-monitor-command', 'arguments':{ 'command-line': 'drive_add -n buddy driver=replication,mode=primary,file.driver=nbd,file.host=127.0.0.1,file.port=9999,file.export=parent0,node-name=replication0'}}
+{'execute': 'x-blockdev-change', 'arguments':{ 'parent': 'colo-disk0', 'node': 'replication0' } }
 
-Before issuing '{ "execute": "x-colo-lost-heartbeat" }' command, we have to
-issue block related command to stop block replication.
-Primary:
-  Remove the nbd child from the quorum:
-  { 'execute': 'x-blockdev-change', 'arguments': {'parent': 'colo-disk0', 'child': 'children.1'}}
-  { 'execute': 'human-monitor-command','arguments': {'command-line': 'drive_del blk-buddy0'}}
-  Note: there is no qmp command to remove the blockdev now
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'filter-mirror', 'id': 'm0', 'props': { 'insert': 'before', 'position': 'id=rew0', 'netdev': 'hn0', 'queue': 'tx', 'outdev': 'mirror0' } } }
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'filter-redirector', 'id': 'redire0', 'props': { 'insert': 'before', 'position': 'id=rew0', 'netdev': 'hn0', 'queue': 'rx', 'indev': 'compare_out' } } }
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'filter-redirector', 'id': 'redire1', 'props': { 'insert': 'before', 'position': 'id=rew0', 'netdev': 'hn0', 'queue': 'rx', 'outdev': 'compare0' } } }
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'iothread', 'id': 'iothread1' } }
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'colo-compare', 'id': 'comp0', 'props': { 'primary_in': 'compare0-0', 'secondary_in': 'compare1', 'outdev': 'compare_out0', 'iothread': 'iothread1' } } }
 
-Secondary:
-  The primary host is down, so we should do the following thing:
-  { 'execute': 'nbd-server-stop' }
+{'execute': 'migrate-set-capabilities', 'arguments':{ 'capabilities': [ {'capability': 'x-colo', 'state': true } ] } }
+{'execute': 'migrate', 'arguments':{ 'uri': 'tcp:127.0.0.1:9998' } }
 
 == TODO ==
-1. Support continuous VM replication.
-2. Support shared storage.
-3. Develop the heartbeat part.
-4. Reduce checkpoint VM’s downtime while doing checkpoint.
+1. Support shared storage.
+2. Develop the heartbeat part.
+3. Reduce checkpoint VM’s downtime while doing checkpoint.
docs/block-replication.txt +18 -10
index 6bde6737fb..108e9166a8 100644
@@ -65,12 +65,12 @@ blocks that are already in QEMU.
              ^            ||                            .----------
              |            ||                            | Secondary
         1 Quorum          ||                            '----------
-         /      \         ||
-        /        \        ||
-   Primary    2 filter
-     disk         ^                                                             virtio-blk
-                  |                                                                  ^
-                3 NBD  ------->  3 NBD                                               |
+         /      \         ||                                                           virtio-blk
+        /        \        ||                                                               ^
+   Primary    2 filter                                                                     |
+     disk         ^                                                                   7 Quorum
+                  |                                                                    /
+                3 NBD  ------->  3 NBD                                                /
                 client    ||     server                                          2 filter
                           ||        ^                                                ^
 --------.                 ||        |                                                |
@@ -106,6 +106,10 @@ any state that would otherwise be lost by the speculative write-through
 of the NBD server into the secondary disk. So before block replication,
 the primary disk and secondary disk should contain the same data.
 
+7) The secondary also has a quorum node, so after secondary failover it
+can become the new primary and continue replication.
+
+
 == Failure Handling ==
 There are 7 internal errors when block replication is running:
 1. I/O error on primary disk
@@ -171,16 +175,18 @@ Primary:
      leading whitespace.
   5. The qmp command line must be run after running qmp command line in
      secondary qemu.
-  6. After failover we need remove children.1 (replication driver).
+  6. After primary failover we need remove children.1 (replication driver).
 
 Secondary:
   -drive if=none,driver=raw,file.filename=1.raw,id=colo1 \
-  -drive if=xxx,id=topxxx,driver=replication,mode=secondary,top-id=topxxx\
+  -drive if=none,id=childs1,driver=replication,mode=secondary,top-id=childs1
          file.file.filename=active_disk.qcow2,\
          file.driver=qcow2,\
          file.backing.file.filename=hidden_disk.qcow2,\
          file.backing.driver=qcow2,\
          file.backing.backing=colo1
+  -drive if=xxx,driver=quorum,read-pattern=fifo,id=top-disk1,\
+         vote-threshold=1,children.0=childs1
 
   Then run qmp command in secondary qemu:
     { 'execute': 'nbd-server-start',
@@ -234,6 +240,8 @@ Secondary:
   The primary host is down, so we should do the following thing:
   { 'execute': 'nbd-server-stop' }
 
+Promote Secondary to Primary:
+  see COLO-FT.txt
+
 TODO:
-1. Continuous block replication
-2. Shared disk
+1. Shared disk
Signature · 1 line
2.20.1
#12 lore raw

RE: [PATCH v5 4/4] colo: Update Documentation for continuous replication

Quoted 118 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Monday, September 16, 2019 3:20 AM
To: qemu-devel <[email protected]>
Cc: Zhang, Chen <[email protected]>; Jason Wang <[email protected]>; Wen Congyang <[email protected]>; Xie Changlong <[email protected]>; [email protected]; [email protected]
Subject: [PATCH v5 4/4] colo: Update Documentation for continuous replication

Document the qemu command-line and qmp commands for continuous replication

---
 docs/COLO-FT.txt           | 212 +++++++++++++++++++++++++++----------
 docs/block-replication.txt |  28 +++--
 2 files changed, 173 insertions(+), 67 deletions(-)
docs/COLO-FT.txt index renamed from docs/COLO-FT.txt +0 -0

ad24680d13..e98c95ca2b 100644

docs/COLO-FT.txt +0 -0
@@ -145,35 +145,65 @@ The diagram just shows the main qmp command,

you can get the detail in test procedure.

== Test procedure ==
-1. Startup qemu

docs/COLO-FT.txt +0 -2
-Primary:
-# qemu-system-x86_64 -accel kvm -m 2048 -smp 2 -qmp stdio -name

primary \

docs/COLO-FT.txt +0 -4
-  -device piix3-usb-uhci -vnc :7 \
-  -device usb-tablet -netdev tap,id=hn0,vhost=off \
-  -device virtio-net-pci,id=net-pci0,netdev=hn0 \
-  -drive if=virtio,id=primary-disk0,driver=quorum,read-pattern=fifo,vote-

threshold=1,\

docs/COLO-FT.txt +0 -4
-         children.0.file.filename=1.raw,\
-         children.0.driver=raw -S
-Secondary:
-# qemu-system-x86_64 -accel kvm -m 2048 -smp 2 -qmp stdio -name

secondary \

docs/COLO-FT.txt +0 -4
-  -device piix3-usb-uhci -vnc :7 \
-  -device usb-tablet -netdev tap,id=hn0,vhost=off \
-  -device virtio-net-pci,id=net-pci0,netdev=hn0 \
-  -drive if=none,id=secondary-disk0,file.filename=1.raw,driver=raw,node-

name=node0 \

docs/COLO-FT.txt +12 -9
-  -drive if=virtio,id=active-disk0,driver=replication,mode=secondary,\
-         file.driver=qcow2,top-id=active-disk0,\
-         file.file.filename=/mnt/ramfs/active_disk.img,\
-         file.backing.driver=qcow2,\
-         file.backing.file.filename=/mnt/ramfs/hidden_disk.img,\
-         file.backing.backing=secondary-disk0 \
-  -incoming tcp:0:8888
-
-2. On Secondary VM's QEMU monitor, issue command
+Note: Here we are running both instances on the same Machine for
+testing, change the IP Addresses if you want to run it on two Hosts
+
+== Startup qemu ==
+1. Primary:
+# imagefolder="/mnt/vms/colo-test"
+
+The disks for the primary and secondary need to be the same before
+starting colo # cp --reflink=auto $imagefolder/primary.qcow2
+$imagefolder/primary-copy.qcow2
+
+# qemu-system-x86_64 -enable-kvm -cpu qemu64,+kvmclock -m 512 -smp

1 -qmp stdio \

docs/COLO-FT.txt +2 -0
+   -vnc :0 -device piix3-usb-uhci -device usb-tablet -name primary \
+   -netdev tap,id=hn0,vhost=off,helper=/usr/lib/qemu/qemu-bridge-helper

\

docs/COLO-FT.txt +6 -0
+   -device rtl8139,id=e0,netdev=hn0 \
+   -chardev socket,id=mirror0,host=127.0.0.1,port=9003,server,nowait \
+   -chardev socket,id=compare1,host=127.0.0.1,port=9004,server,wait \
+   -chardev socket,id=compare0,host=127.0.0.1,port=9001,server,nowait \
+   -chardev socket,id=compare0-0,host=127.0.0.1,port=9001 \
+   -chardev socket,id=compare_out,host=127.0.0.1,port=9005,server,nowait

\

docs/COLO-FT.txt +3 -0
+   -chardev socket,id=compare_out0,host=127.0.0.1,port=9005 \
+   -object filter-mirror,id=m0,netdev=hn0,queue=tx,outdev=mirror0 \
+   -object filter-

redirector,netdev=hn0,id=redire0,queue=rx,indev=compare_out \
+ -object filter-
redirector,netdev=hn0,id=redire1,queue=rx,outdev=compare0 \

docs/COLO-FT.txt +3 -0
+   -object iothread,id=iothread1 \
+   -object
+colo-compare,id=comp0,primary_in=compare0-

0,secondary_in=compare1,\

docs/COLO-FT.txt +14 -0
+outdev=compare_out0,iothread=iothread1 \
+   -drive
+if=ide,id=colo-disk0,driver=quorum,read-pattern=fifo,vote-threshold=1,\
+children.0.file.filename=$imagefolder/primary.qcow2,children.0.driver=q
+cow2 -S
+
+2. Secondary:
+# imagefolder="/mnt/vms/colo-test"
+
+# qemu-img create -f qcow2 $imagefolder/secondary-active.qcow2 10G
+
+# qemu-img create -f qcow2 $imagefolder/secondary-hidden.qcow2 10G
+
+# qemu-system-x86_64 -enable-kvm -cpu qemu64,+kvmclock -m 512 -smp

1 -qmp stdio \

docs/COLO-FT.txt +2 -0
+   -vnc :1 -device piix3-usb-uhci -device usb-tablet -name secondary \
+   -netdev tap,id=hn0,vhost=off,helper=/usr/lib/qemu/qemu-bridge-helper

\

docs/COLO-FT.txt +7 -0
+   -device rtl8139,id=e0,netdev=hn0 \
+   -chardev socket,id=red0,host=127.0.0.1,port=9003,reconnect=1 \
+   -chardev socket,id=red1,host=127.0.0.1,port=9004,reconnect=1 \
+   -object filter-redirector,id=f1,netdev=hn0,queue=tx,indev=red0 \
+   -object filter-redirector,id=f2,netdev=hn0,queue=rx,outdev=red1 \
+   -object filter-rewriter,id=rew0,netdev=hn0,queue=all \
+   -drive if=none,id=parent0,file.filename=$imagefolder/primary-

copy.qcow2,driver=qcow2 \

docs/COLO-FT.txt +2 -0
+   -drive
+if=none,id=childs0,driver=replication,mode=secondary,file.driver=qcow2,

docs/COLO-FT.txt

Quoted 10 lines
docs/COLO-FT.txt +10 -0
+\
+top-id=childs0,file.file.filename=$imagefolder/secondary-active.qcow2,\
+file.backing.driver=qcow2,file.backing.file.filename=$imagefolder/secon
+dary-hidden.qcow2,\
+file.backing.backing=parent0 \
+   -drive
+if=ide,id=colo-disk0,driver=quorum,read-pattern=fifo,vote-threshold=1,\
+children.0=childs0 \
+   -incoming tcp:0:9998
+

I know in your COLO test environment, all the node running in same physical machine, but current COLO start parameter demo easy to misunderstand users.
I think we can introduce primary node as 192.168.1.1, secondary node as 192.168.1.2. Same issue for the VM image, we should avoid the "primary-copy.qow2". Make two node independent is more clear.

Quoted 109 lines
docs/COLO-FT.txt +2 -2
+
+3. On Secondary VM's QEMU monitor, issue command
 {'execute':'qmp_capabilities'}
-{ 'execute': 'nbd-server-start',
-  'arguments': {'addr': {'type': 'inet', 'data': {'host': 'xx.xx.xx.xx', 'port':

'8889'} } } -}
-{'execute': 'nbd-server-add', 'arguments': {'device': 'secondary-disk0', 'writable': true } }

docs/COLO-FT.txt +5 -3
+{'execute': 'nbd-server-start', 'arguments': {'addr': {'type': 'inet',
+'data': {'host': '127.0.0.1', 'port': '9999'} } } }
+{'execute': 'nbd-server-add', 'arguments': {'device': 'parent0',
+'writable': true } }
 
 Note:
   a. The qmp command nbd-server-start and nbd-server-add must be run
@@ -182,44 +212,112 @@ Note:
      same.
   c. It is better to put active disk and hidden disk in ramdisk.
 
-3. On Primary VM's QEMU monitor, issue command:
+4. On Primary VM's QEMU monitor, issue command:
 {'execute':'qmp_capabilities'}
-{ 'execute': 'human-monitor-command',
-  'arguments': {'command-line': 'drive_add -n buddy

driver=replication,mode=primary,file.driver=nbd,file.host=xx.xx.xx.xx,file.p ort=8889,file.export=secondary-disk0,node-name=nbd_client0'}}
-{ 'execute':'x-blockdev-change', 'arguments':{'parent': 'primary-disk0', 'node': 'nbd_client0' } } -{ 'execute': 'migrate-set-capabilities',

docs/COLO-FT.txt +10 -5
-      'arguments': {'capabilities': [ {'capability': 'x-colo', 'state': true } ] } }
-{ 'execute': 'migrate', 'arguments': {'uri': 'tcp:xx.xx.xx.xx:8888' } }
+{'execute': 'human-monitor-command', 'arguments': {'command-line':
+'drive_add -n buddy
+driver=replication,mode=primary,file.driver=nbd,file.host=127.0.0.1,fil
+e.port=9999,file.export=parent0,node-name=replication0'}}
+{'execute': 'x-blockdev-change', 'arguments':{'parent': 'colo-disk0',
+'node': 'replication0' } }
+{'execute': 'migrate-set-capabilities', 'arguments': {'capabilities': [
+{'capability': 'x-colo', 'state': true } ] } }
+{'execute': 'migrate', 'arguments': {'uri': 'tcp:127.0.0.1:9998' } }
 
   Note:
   a. There should be only one NBD Client for each primary disk.
-  b. xx.xx.xx.xx is the secondary physical machine's hostname or IP
-  c. The qmp command line must be run after running qmp command line in
+  b. The qmp command line must be run after running qmp command line in
      secondary qemu.
 
-4. After the above steps, you will see, whenever you make changes to PVM,

SVM will be synced.
+5. After the above steps, you will see, whenever you make changes to PVM, SVM will be synced.
You can issue command '{ "execute": "migrate-set-parameters" ,
"arguments":{ "x-checkpoint-delay": 2000 } }'

docs/COLO-FT.txt +7 -1
-to change the checkpoint period time
+to change the idle checkpoint period time
+
+6. Failover test
+You can kill one of the VMs and Failover on the surviving VM:
+
+If you killed the Secondary, then follow "Primary Failover". After
+that, if you want to resume the replication, follow "Primary resume

replication"

docs/COLO-FT.txt +3 -0
+
+If you killed the Primary, then follow "Secondary Failover". After
+that, if you want to resume the replication, follow "Secondary resume

replication"

docs/COLO-FT.txt +17 -0
+
+== Primary Failover ==
+The Secondary died, resume on the Primary
+
+{'execute': 'x-blockdev-change', 'arguments':{ 'parent': 'colo-disk0',
+'child': 'children.1'} }
+{'execute': 'human-monitor-command', 'arguments':{ 'command-line':
+'drive_del replication0' } }
+{'execute': 'object-del', 'arguments':{ 'id': 'comp0' } }
+{'execute': 'object-del', 'arguments':{ 'id': 'iothread1' } }
+{'execute': 'object-del', 'arguments':{ 'id': 'm0' } }
+{'execute': 'object-del', 'arguments':{ 'id': 'redire0' } }
+{'execute': 'object-del', 'arguments':{ 'id': 'redire1' } }
+{'execute': 'x-colo-lost-heartbeat' }
+
+== Secondary Failover ==
+The Primary died, resume on the Secondary and prepare to become the

new

docs/COLO-FT.txt +25 -0
+Primary
+
+{'execute': 'nbd-server-stop'}
+{'execute': 'x-colo-lost-heartbeat'}
+
+{'execute': 'object-del', 'arguments':{ 'id': 'f2' } }
+{'execute': 'object-del', 'arguments':{ 'id': 'f1' } }
+{'execute': 'chardev-remove', 'arguments':{ 'id': 'red1' } }
+{'execute': 'chardev-remove', 'arguments':{ 'id': 'red0' } }
+
+{'execute': 'chardev-add', 'arguments':{ 'id': 'mirror0', 'backend':
+{'type': 'socket', 'data': {'addr': { 'type': 'inet', 'data': { 'host':
+'127.0.0.1', 'port': '9003' } }, 'server': true } } } }
+{'execute': 'chardev-add', 'arguments':{ 'id': 'compare1', 'backend':
+{'type': 'socket', 'data': {'addr': { 'type': 'inet', 'data': { 'host':
+'127.0.0.1', 'port': '9004' } }, 'server': true } } } }
+{'execute': 'chardev-add', 'arguments':{ 'id': 'compare0', 'backend':
+{'type': 'socket', 'data': {'addr': { 'type': 'inet', 'data': { 'host':
+'127.0.0.1', 'port': '9001' } }, 'server': true } } } }
+{'execute': 'chardev-add', 'arguments':{ 'id': 'compare0-0', 'backend':
+{'type': 'socket', 'data': {'addr': { 'type': 'inet', 'data': { 'host':
+'127.0.0.1', 'port': '9001' } }, 'server': false } } } }
+{'execute': 'chardev-add', 'arguments':{ 'id': 'compare_out',
+'backend': {'type': 'socket', 'data': {'addr': { 'type': 'inet',
+'data': { 'host': '127.0.0.1', 'port': '9005' } }, 'server': true } } }

docs/COLO-FT.txt

Quoted 10 lines
docs/COLO-FT.txt +10 -0
+}
+{'execute': 'chardev-add', 'arguments':{ 'id': 'compare_out0',
+'backend': {'type': 'socket', 'data': {'addr': { 'type': 'inet',
+'data': { 'host': '127.0.0.1', 'port': '9005' } }, 'server': false } }
+} }
+
+== Primary resume replication ==
+Resume replication after new Secondary is up.
+
+Start the new Secondary (Steps 2 and 3 above), then:

Please note here the "then" is the original secondary node. It is easy to misunderstand here.

docs/COLO-FT.txt

Quoted 4 lines
docs/COLO-FT.txt +4 -0
+{'execute': 'drive-mirror', 'arguments':{ 'device': 'colo-disk0',
+'job-id': 'resync', 'target': 'nbd://127.0.0.1:9999/parent0', 'mode':
+'existing', 'format': 'raw', 'sync': 'full'} }
+

As I mentioned before, the 'nbd://127.0.0.1:9999/parent0' is very confusing, user don't know which node will be connected.

Quoted 20 lines
docs/COLO-FT.txt +20 -0
+Wait until disk is synced, then:
+{'execute': 'stop'}
+{'execute': 'block-job-cancel', 'arguments':{ 'device': 'resync'} }
+
+{'execute': 'human-monitor-command', 'arguments':{ 'command-line':
+'drive_add -n buddy
+driver=replication,mode=primary,file.driver=nbd,file.host=127.0.0.1,fil
+e.port=9999,file.export=parent0,node-name=replication0'}}
+{'execute': 'x-blockdev-change', 'arguments':{ 'parent': 'colo-disk0',
+'node': 'replication0' } }
+
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'filter-mirror',
+'id': 'm0', 'props': { 'netdev': 'hn0', 'queue': 'tx', 'outdev':
+'mirror0' } } }
+{'execute': 'object-add', 'arguments':{ 'qom-type':
+'filter-redirector', 'id': 'redire0', 'props': { 'netdev': 'hn0',
+'queue': 'rx', 'indev': 'compare_out' } } }
+{'execute': 'object-add', 'arguments':{ 'qom-type':
+'filter-redirector', 'id': 'redire1', 'props': { 'netdev': 'hn0',
+'queue': 'rx', 'outdev': 'compare0' } } }

docs/COLO-FT.txt

Quoted 10 lines
docs/COLO-FT.txt +10 -0
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'iothread', 'id':
+'iothread1' } }
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'colo-compare',
+'id': 'comp0', 'props': { 'primary_in': 'compare0-0', 'secondary_in':
+'compare1', 'outdev': 'compare_out0', 'iothread': 'iothread1' } } }
+
+{'execute': 'migrate-set-capabilities', 'arguments':{ 'capabilities': [
+{'capability': 'x-colo', 'state': true } ] } }
+{'execute': 'migrate', 'arguments':{ 'uri': 'tcp:127.0.0.1:9998' } }
+

The guest network have issues here in my test, I can't ping the guest from the host. Is it works in your test environment?

By the way, after finish this doc we can sync it to Qemu COLO WIKI page.

Thanks
Zhang Chen

Quoted 142 lines
docs/COLO-FT.txt +18 -2
+Note:
+If this Primary previously was a Secondary, then we need to insert the
+filters before the filter-rewriter by using the
+"'insert': 'before', 'position': 'rew0'" Options. See below.
+
+== Secondary resume replication ==
+Become Primary and resume replication after new Secondary is up.
+
+Start the new Secondary (Steps 2 and 3 above, but with different image
+than
+primary-copy.qcow2 and different VNC port), then:
+{'execute': 'drive-mirror', 'arguments':{ 'device': 'colo-disk0',
+'job-id': 'resync', 'target': 'nbd://127.0.0.1:9999/parent0', 'mode':
+'existing', 'format': 'raw', 'sync': 'full'} }
+
+Wait until disk is synced, then:
+{'execute': 'stop'}
+{'execute': 'block-job-cancel', 'arguments':{ 'device': 'resync' } }
 
-5. Failover test
-You can kill Primary VM and run 'x_colo_lost_heartbeat' in Secondary VM's -

monitor at the same time, then SVM will failover and client will not detect this -change.

docs/COLO-FT.txt +6 -5
+{'execute': 'human-monitor-command', 'arguments':{ 'command-line':
+'drive_add -n buddy
+driver=replication,mode=primary,file.driver=nbd,file.host=127.0.0.1,fil
+e.port=9999,file.export=parent0,node-name=replication0'}}
+{'execute': 'x-blockdev-change', 'arguments':{ 'parent': 'colo-disk0',
+'node': 'replication0' } }
 
-Before issuing '{ "execute": "x-colo-lost-heartbeat" }' command, we have to
-issue block related command to stop block replication.
-Primary:
-  Remove the nbd child from the quorum:
-  { 'execute': 'x-blockdev-change', 'arguments': {'parent': 'colo-disk0', 'child':

'children.1'}}
- { 'execute': 'human-monitor-command','arguments': {'command-line':
'drive_del blk-buddy0'}}

docs/COLO-FT.txt +22 -8
-  Note: there is no qmp command to remove the blockdev now
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'filter-mirror',
+'id': 'm0', 'props': { 'insert': 'before', 'position': 'id=rew0',
+'netdev': 'hn0', 'queue': 'tx', 'outdev': 'mirror0' } } }
+{'execute': 'object-add', 'arguments':{ 'qom-type':
+'filter-redirector', 'id': 'redire0', 'props': { 'insert': 'before',
+'position': 'id=rew0', 'netdev': 'hn0', 'queue': 'rx', 'indev':
+'compare_out' } } }
+{'execute': 'object-add', 'arguments':{ 'qom-type':
+'filter-redirector', 'id': 'redire1', 'props': { 'insert': 'before',
+'position': 'id=rew0', 'netdev': 'hn0', 'queue': 'rx', 'outdev':
+'compare0' } } }
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'iothread', 'id':
+'iothread1' } }
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'colo-compare',
+'id': 'comp0', 'props': { 'primary_in': 'compare0-0', 'secondary_in':
+'compare1', 'outdev': 'compare_out0', 'iothread': 'iothread1' } } }
 
-Secondary:
-  The primary host is down, so we should do the following thing:
-  { 'execute': 'nbd-server-stop' }
+{'execute': 'migrate-set-capabilities', 'arguments':{ 'capabilities': [
+{'capability': 'x-colo', 'state': true } ] } }
+{'execute': 'migrate', 'arguments':{ 'uri': 'tcp:127.0.0.1:9998' } }
 
 == TODO ==
-1. Support continuous VM replication.
-2. Support shared storage.
-3. Develop the heartbeat part.
-4. Reduce checkpoint VM’s downtime while doing checkpoint.
+1. Support shared storage.
+2. Develop the heartbeat part.
+3. Reduce checkpoint VM’s downtime while doing checkpoint.
docs/block-replication.txt index renamed from docs/block-replication.txt +0 -0

6bde6737fb..108e9166a8 100644

docs/block-replication.txt +6 -6
@@ -65,12 +65,12 @@ blocks that are already in QEMU.
              ^            ||                            .----------
              |            ||                            | Secondary
         1 Quorum          ||                            '----------
-         /      \         ||
-        /        \        ||
-   Primary    2 filter
-     disk         ^                                                             virtio-blk
-                  |                                                                  ^
-                3 NBD  ------->  3 NBD                                               |
+         /      \         ||                                                           virtio-blk
+        /        \        ||                                                               ^
+   Primary    2 filter                                                                     |
+     disk         ^                                                                   7 Quorum
+                  |                                                                    /
+                3 NBD  ------->  3 NBD                                                /
                 client    ||     server                                          2 filter
                           ||        ^                                                ^
 --------.                 ||        |                                                |
@@ -106,6 +106,10 @@ any state that would otherwise be lost by the

speculative write-through of the NBD server into the secondary disk. So before block replication, the primary disk and secondary disk should contain the same data.

+7) The secondary also has a quorum node, so after secondary failover it

docs/block-replication.txt +12 -5
+can become the new primary and continue replication.
+
+
 == Failure Handling ==
 There are 7 internal errors when block replication is running:
 1. I/O error on primary disk
@@ -171,16 +175,18 @@ Primary:
      leading whitespace.
   5. The qmp command line must be run after running qmp command line in
      secondary qemu.
-  6. After failover we need remove children.1 (replication driver).
+  6. After primary failover we need remove children.1 (replication driver).
 
 Secondary:
   -drive if=none,driver=raw,file.filename=1.raw,id=colo1 \
-  -drive if=xxx,id=topxxx,driver=replication,mode=secondary,top-
 id=topxxx\
+  -drive
+ if=none,id=childs1,driver=replication,mode=secondary,top-id=childs1
          file.file.filename=active_disk.qcow2,\
          file.driver=qcow2,\
          file.backing.file.filename=hidden_disk.qcow2,\
          file.backing.driver=qcow2,\
          file.backing.backing=colo1
+  -drive if=xxx,driver=quorum,read-pattern=fifo,id=top-disk1,\
+         vote-threshold=1,children.0=childs1
 
   Then run qmp command in secondary qemu:
     { 'execute': 'nbd-server-start',
@@ -234,6 +240,8 @@ Secondary:
   The primary host is down, so we should do the following thing:
   { 'execute': 'nbd-server-stop' }
 
+Promote Secondary to Primary:
+  see COLO-FT.txt
+
 TODO:
-1. Continuous block replication
-2. Shared disk
+1. Shared disk
--

2.20.1

#13 lore raw

Re: [PATCH v5 4/4] colo: Update Documentation for continuous replication

Quoted 125 lines · Chen
Quoted 125 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Monday, September 16, 2019 3:20 AM
To: qemu-devel <[email protected]>
Cc: Zhang, Chen <[email protected]>; Jason Wang <[email protected]>; Wen Congyang <[email protected]>; Xie Changlong <[email protected]>; [email protected]; [email protected]
Subject: [PATCH v5 4/4] colo: Update Documentation for continuous replication

Document the qemu command-line and qmp commands for continuous replication

---
 docs/COLO-FT.txt           | 212 +++++++++++++++++++++++++++----------
 docs/block-replication.txt |  28 +++--
 2 files changed, 173 insertions(+), 67 deletions(-)
docs/COLO-FT.txt index renamed from docs/COLO-FT.txt +0 -0

ad24680d13..e98c95ca2b 100644

docs/COLO-FT.txt +0 -0
@@ -145,35 +145,65 @@ The diagram just shows the main qmp command,

you can get the detail in test procedure.

== Test procedure ==
-1. Startup qemu

docs/COLO-FT.txt +0 -2
-Primary:
-# qemu-system-x86_64 -accel kvm -m 2048 -smp 2 -qmp stdio -name

primary \

docs/COLO-FT.txt +0 -4
-  -device piix3-usb-uhci -vnc :7 \
-  -device usb-tablet -netdev tap,id=hn0,vhost=off \
-  -device virtio-net-pci,id=net-pci0,netdev=hn0 \
-  -drive if=virtio,id=primary-disk0,driver=quorum,read-pattern=fifo,vote-

threshold=1,\

docs/COLO-FT.txt +0 -4
-         children.0.file.filename=1.raw,\
-         children.0.driver=raw -S
-Secondary:
-# qemu-system-x86_64 -accel kvm -m 2048 -smp 2 -qmp stdio -name

secondary \

docs/COLO-FT.txt +0 -4
-  -device piix3-usb-uhci -vnc :7 \
-  -device usb-tablet -netdev tap,id=hn0,vhost=off \
-  -device virtio-net-pci,id=net-pci0,netdev=hn0 \
-  -drive if=none,id=secondary-disk0,file.filename=1.raw,driver=raw,node-

name=node0 \

docs/COLO-FT.txt +12 -9
-  -drive if=virtio,id=active-disk0,driver=replication,mode=secondary,\
-         file.driver=qcow2,top-id=active-disk0,\
-         file.file.filename=/mnt/ramfs/active_disk.img,\
-         file.backing.driver=qcow2,\
-         file.backing.file.filename=/mnt/ramfs/hidden_disk.img,\
-         file.backing.backing=secondary-disk0 \
-  -incoming tcp:0:8888
-
-2. On Secondary VM's QEMU monitor, issue command
+Note: Here we are running both instances on the same Machine for
+testing, change the IP Addresses if you want to run it on two Hosts
+
+== Startup qemu ==
+1. Primary:
+# imagefolder="/mnt/vms/colo-test"
+
+The disks for the primary and secondary need to be the same before
+starting colo # cp --reflink=auto $imagefolder/primary.qcow2
+$imagefolder/primary-copy.qcow2
+
+# qemu-system-x86_64 -enable-kvm -cpu qemu64,+kvmclock -m 512 -smp

1 -qmp stdio \

docs/COLO-FT.txt +2 -0
+   -vnc :0 -device piix3-usb-uhci -device usb-tablet -name primary \
+   -netdev tap,id=hn0,vhost=off,helper=/usr/lib/qemu/qemu-bridge-helper

\

docs/COLO-FT.txt +6 -0
+   -device rtl8139,id=e0,netdev=hn0 \
+   -chardev socket,id=mirror0,host=127.0.0.1,port=9003,server,nowait \
+   -chardev socket,id=compare1,host=127.0.0.1,port=9004,server,wait \
+   -chardev socket,id=compare0,host=127.0.0.1,port=9001,server,nowait \
+   -chardev socket,id=compare0-0,host=127.0.0.1,port=9001 \
+   -chardev socket,id=compare_out,host=127.0.0.1,port=9005,server,nowait

\

docs/COLO-FT.txt +3 -0
+   -chardev socket,id=compare_out0,host=127.0.0.1,port=9005 \
+   -object filter-mirror,id=m0,netdev=hn0,queue=tx,outdev=mirror0 \
+   -object filter-

redirector,netdev=hn0,id=redire0,queue=rx,indev=compare_out \
+ -object filter-
redirector,netdev=hn0,id=redire1,queue=rx,outdev=compare0 \

docs/COLO-FT.txt +3 -0
+   -object iothread,id=iothread1 \
+   -object
+colo-compare,id=comp0,primary_in=compare0-

0,secondary_in=compare1,\

docs/COLO-FT.txt +14 -0
+outdev=compare_out0,iothread=iothread1 \
+   -drive
+if=ide,id=colo-disk0,driver=quorum,read-pattern=fifo,vote-threshold=1,\
+children.0.file.filename=$imagefolder/primary.qcow2,children.0.driver=q
+cow2 -S
+
+2. Secondary:
+# imagefolder="/mnt/vms/colo-test"
+
+# qemu-img create -f qcow2 $imagefolder/secondary-active.qcow2 10G
+
+# qemu-img create -f qcow2 $imagefolder/secondary-hidden.qcow2 10G
+
+# qemu-system-x86_64 -enable-kvm -cpu qemu64,+kvmclock -m 512 -smp

1 -qmp stdio \

docs/COLO-FT.txt +2 -0
+   -vnc :1 -device piix3-usb-uhci -device usb-tablet -name secondary \
+   -netdev tap,id=hn0,vhost=off,helper=/usr/lib/qemu/qemu-bridge-helper

\

docs/COLO-FT.txt +7 -0
+   -device rtl8139,id=e0,netdev=hn0 \
+   -chardev socket,id=red0,host=127.0.0.1,port=9003,reconnect=1 \
+   -chardev socket,id=red1,host=127.0.0.1,port=9004,reconnect=1 \
+   -object filter-redirector,id=f1,netdev=hn0,queue=tx,indev=red0 \
+   -object filter-redirector,id=f2,netdev=hn0,queue=rx,outdev=red1 \
+   -object filter-rewriter,id=rew0,netdev=hn0,queue=all \
+   -drive if=none,id=parent0,file.filename=$imagefolder/primary-

copy.qcow2,driver=qcow2 \

docs/COLO-FT.txt +10 -0
+   -drive
+if=none,id=childs0,driver=replication,mode=secondary,file.driver=qcow2,
+\
+top-id=childs0,file.file.filename=$imagefolder/secondary-active.qcow2,\
+file.backing.driver=qcow2,file.backing.file.filename=$imagefolder/secon
+dary-hidden.qcow2,\
+file.backing.backing=parent0 \
+   -drive
+if=ide,id=colo-disk0,driver=quorum,read-pattern=fifo,vote-threshold=1,\
+children.0=childs0 \
Quoted 9 lines

docs/COLO-FT.txt

Quoted 2 lines
docs/COLO-FT.txt +2 -0
+   -incoming tcp:0:9998
+  

I know in your COLO test environment, all the node running in same physical machine, but current COLO start parameter demo easy to misunderstand users.
I think we can introduce primary node as 192.168.1.1, secondary node as 192.168.1.2. Same issue for the VM image, we should avoid the "primary-copy.qow2". Make two node independent is more clear.

Ok, I will remove the primary-copy stuff. But I have a better idea for the IP addresses:
Using 127.0.0.1 for Primary and 127.0.0.2 for Secondary. Both point to localhost, making quick testing by simple copy-paste possible when using different values for the imagefolder Variable. At the same time Users see which IP to substitute for the remote node when using multiple Hosts and we avoid the confusion with primary-copy.
What do you think?

Quoted 115 lines
Quoted 115 lines
docs/COLO-FT.txt +2 -2
+
+3. On Secondary VM's QEMU monitor, issue command
 {'execute':'qmp_capabilities'}
-{ 'execute': 'nbd-server-start',
-  'arguments': {'addr': {'type': 'inet', 'data': {'host': 'xx.xx.xx.xx', 'port':

'8889'} } } -}
-{'execute': 'nbd-server-add', 'arguments': {'device': 'secondary-disk0', 'writable': true } }

docs/COLO-FT.txt +5 -3
+{'execute': 'nbd-server-start', 'arguments': {'addr': {'type': 'inet',
+'data': {'host': '127.0.0.1', 'port': '9999'} } } }
+{'execute': 'nbd-server-add', 'arguments': {'device': 'parent0',
+'writable': true } }
 
 Note:
   a. The qmp command nbd-server-start and nbd-server-add must be run
@@ -182,44 +212,112 @@ Note:
      same.
   c. It is better to put active disk and hidden disk in ramdisk.
 
-3. On Primary VM's QEMU monitor, issue command:
+4. On Primary VM's QEMU monitor, issue command:
 {'execute':'qmp_capabilities'}
-{ 'execute': 'human-monitor-command',
-  'arguments': {'command-line': 'drive_add -n buddy

driver=replication,mode=primary,file.driver=nbd,file.host=xx.xx.xx.xx,file.p ort=8889,file.export=secondary-disk0,node-name=nbd_client0'}}
-{ 'execute':'x-blockdev-change', 'arguments':{'parent': 'primary-disk0', 'node': 'nbd_client0' } } -{ 'execute': 'migrate-set-capabilities',

docs/COLO-FT.txt +10 -5
-      'arguments': {'capabilities': [ {'capability': 'x-colo', 'state': true } ] } }
-{ 'execute': 'migrate', 'arguments': {'uri': 'tcp:xx.xx.xx.xx:8888' } }
+{'execute': 'human-monitor-command', 'arguments': {'command-line':
+'drive_add -n buddy
+driver=replication,mode=primary,file.driver=nbd,file.host=127.0.0.1,fil
+e.port=9999,file.export=parent0,node-name=replication0'}}
+{'execute': 'x-blockdev-change', 'arguments':{'parent': 'colo-disk0',
+'node': 'replication0' } }
+{'execute': 'migrate-set-capabilities', 'arguments': {'capabilities': [
+{'capability': 'x-colo', 'state': true } ] } }
+{'execute': 'migrate', 'arguments': {'uri': 'tcp:127.0.0.1:9998' } }
 
   Note:
   a. There should be only one NBD Client for each primary disk.
-  b. xx.xx.xx.xx is the secondary physical machine's hostname or IP
-  c. The qmp command line must be run after running qmp command line in
+  b. The qmp command line must be run after running qmp command line in
      secondary qemu.
 
-4. After the above steps, you will see, whenever you make changes to PVM,

SVM will be synced.
+5. After the above steps, you will see, whenever you make changes to PVM, SVM will be synced.
You can issue command '{ "execute": "migrate-set-parameters" ,
"arguments":{ "x-checkpoint-delay": 2000 } }'

docs/COLO-FT.txt +7 -1
-to change the checkpoint period time
+to change the idle checkpoint period time
+
+6. Failover test
+You can kill one of the VMs and Failover on the surviving VM:
+
+If you killed the Secondary, then follow "Primary Failover". After
+that, if you want to resume the replication, follow "Primary resume

replication"

docs/COLO-FT.txt +3 -0
+
+If you killed the Primary, then follow "Secondary Failover". After
+that, if you want to resume the replication, follow "Secondary resume

replication"

docs/COLO-FT.txt +17 -0
+
+== Primary Failover ==
+The Secondary died, resume on the Primary
+
+{'execute': 'x-blockdev-change', 'arguments':{ 'parent': 'colo-disk0',
+'child': 'children.1'} }
+{'execute': 'human-monitor-command', 'arguments':{ 'command-line':
+'drive_del replication0' } }
+{'execute': 'object-del', 'arguments':{ 'id': 'comp0' } }
+{'execute': 'object-del', 'arguments':{ 'id': 'iothread1' } }
+{'execute': 'object-del', 'arguments':{ 'id': 'm0' } }
+{'execute': 'object-del', 'arguments':{ 'id': 'redire0' } }
+{'execute': 'object-del', 'arguments':{ 'id': 'redire1' } }
+{'execute': 'x-colo-lost-heartbeat' }
+
+== Secondary Failover ==
+The Primary died, resume on the Secondary and prepare to become the

new

docs/COLO-FT.txt +30 -0
+Primary
+
+{'execute': 'nbd-server-stop'}
+{'execute': 'x-colo-lost-heartbeat'}
+
+{'execute': 'object-del', 'arguments':{ 'id': 'f2' } }
+{'execute': 'object-del', 'arguments':{ 'id': 'f1' } }
+{'execute': 'chardev-remove', 'arguments':{ 'id': 'red1' } }
+{'execute': 'chardev-remove', 'arguments':{ 'id': 'red0' } }
+
+{'execute': 'chardev-add', 'arguments':{ 'id': 'mirror0', 'backend':
+{'type': 'socket', 'data': {'addr': { 'type': 'inet', 'data': { 'host':
+'127.0.0.1', 'port': '9003' } }, 'server': true } } } }
+{'execute': 'chardev-add', 'arguments':{ 'id': 'compare1', 'backend':
+{'type': 'socket', 'data': {'addr': { 'type': 'inet', 'data': { 'host':
+'127.0.0.1', 'port': '9004' } }, 'server': true } } } }
+{'execute': 'chardev-add', 'arguments':{ 'id': 'compare0', 'backend':
+{'type': 'socket', 'data': {'addr': { 'type': 'inet', 'data': { 'host':
+'127.0.0.1', 'port': '9001' } }, 'server': true } } } }
+{'execute': 'chardev-add', 'arguments':{ 'id': 'compare0-0', 'backend':
+{'type': 'socket', 'data': {'addr': { 'type': 'inet', 'data': { 'host':
+'127.0.0.1', 'port': '9001' } }, 'server': false } } } }
+{'execute': 'chardev-add', 'arguments':{ 'id': 'compare_out',
+'backend': {'type': 'socket', 'data': {'addr': { 'type': 'inet',
+'data': { 'host': '127.0.0.1', 'port': '9005' } }, 'server': true } } }
+}
+{'execute': 'chardev-add', 'arguments':{ 'id': 'compare_out0',
+'backend': {'type': 'socket', 'data': {'addr': { 'type': 'inet',
+'data': { 'host': '127.0.0.1', 'port': '9005' } }, 'server': false } }
+} }
Quoted 8 lines

docs/COLO-FT.txt Note: · line 273

Quoted 5 lines
docs/COLO-FT.txt +5 -0
+
+== Primary resume replication ==
+Resume replication after new Secondary is up.
+
+Start the new Secondary (Steps 2 and 3 above), then:  

Please note here the "then" is the original secondary node. It is easy to misunderstand here.

Ok, I will make this more clear.

Quoted 32 lines

docs/COLO-FT.txt

Quoted 4 lines
docs/COLO-FT.txt +4 -0
+{'execute': 'drive-mirror', 'arguments':{ 'device': 'colo-disk0',
+'job-id': 'resync', 'target': 'nbd://127.0.0.1:9999/parent0', 'mode':
+'existing', 'format': 'raw', 'sync': 'full'} }
+  

As I mentioned before, the 'nbd://127.0.0.1:9999/parent0' is very confusing, user don't know which node will be connected.

Quoted 25 lines
docs/COLO-FT.txt +25 -0
+Wait until disk is synced, then:
+{'execute': 'stop'}
+{'execute': 'block-job-cancel', 'arguments':{ 'device': 'resync'} }
+
+{'execute': 'human-monitor-command', 'arguments':{ 'command-line':
+'drive_add -n buddy
+driver=replication,mode=primary,file.driver=nbd,file.host=127.0.0.1,fil
+e.port=9999,file.export=parent0,node-name=replication0'}}
+{'execute': 'x-blockdev-change', 'arguments':{ 'parent': 'colo-disk0',
+'node': 'replication0' } }
+
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'filter-mirror',
+'id': 'm0', 'props': { 'netdev': 'hn0', 'queue': 'tx', 'outdev':
+'mirror0' } } }
+{'execute': 'object-add', 'arguments':{ 'qom-type':
+'filter-redirector', 'id': 'redire0', 'props': { 'netdev': 'hn0',
+'queue': 'rx', 'indev': 'compare_out' } } }
+{'execute': 'object-add', 'arguments':{ 'qom-type':
+'filter-redirector', 'id': 'redire1', 'props': { 'netdev': 'hn0',
+'queue': 'rx', 'outdev': 'compare0' } } }
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'iothread', 'id':
+'iothread1' } }
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'colo-compare',
+'id': 'comp0', 'props': { 'primary_in': 'compare0-0', 'secondary_in':
+'compare1', 'outdev': 'compare_out0', 'iothread': 'iothread1' } } }
Quoted 8 lines

docs/COLO-FT.txt Note: · line 291

Quoted 5 lines
docs/COLO-FT.txt +5 -0
+
+{'execute': 'migrate-set-capabilities', 'arguments':{ 'capabilities': [
+{'capability': 'x-colo', 'state': true } ] } }
+{'execute': 'migrate', 'arguments':{ 'uri': 'tcp:127.0.0.1:9998' } }
+  

The guest network have issues here in my test, I can't ping the guest from the host. Is it works in your test environment?

Yes, it works for me (based on v4.1.0). I simply copy-paste everything here 1:1 for testing.

Regards,
Lukas Straub

Quoted 148 lines

By the way, after finish this doc we can sync it to Qemu COLO WIKI page.

Thanks
Zhang Chen

Quoted 142 lines
docs/COLO-FT.txt +18 -2
+Note:
+If this Primary previously was a Secondary, then we need to insert the
+filters before the filter-rewriter by using the
+"'insert': 'before', 'position': 'rew0'" Options. See below.
+
+== Secondary resume replication ==
+Become Primary and resume replication after new Secondary is up.
+
+Start the new Secondary (Steps 2 and 3 above, but with different image
+than
+primary-copy.qcow2 and different VNC port), then:
+{'execute': 'drive-mirror', 'arguments':{ 'device': 'colo-disk0',
+'job-id': 'resync', 'target': 'nbd://127.0.0.1:9999/parent0', 'mode':
+'existing', 'format': 'raw', 'sync': 'full'} }
+
+Wait until disk is synced, then:
+{'execute': 'stop'}
+{'execute': 'block-job-cancel', 'arguments':{ 'device': 'resync' } }
 
-5. Failover test
-You can kill Primary VM and run 'x_colo_lost_heartbeat' in Secondary VM's -

monitor at the same time, then SVM will failover and client will not detect this -change.

docs/COLO-FT.txt +6 -5
+{'execute': 'human-monitor-command', 'arguments':{ 'command-line':
+'drive_add -n buddy
+driver=replication,mode=primary,file.driver=nbd,file.host=127.0.0.1,fil
+e.port=9999,file.export=parent0,node-name=replication0'}}
+{'execute': 'x-blockdev-change', 'arguments':{ 'parent': 'colo-disk0',
+'node': 'replication0' } }
 
-Before issuing '{ "execute": "x-colo-lost-heartbeat" }' command, we have to
-issue block related command to stop block replication.
-Primary:
-  Remove the nbd child from the quorum:
-  { 'execute': 'x-blockdev-change', 'arguments': {'parent': 'colo-disk0', 'child':

'children.1'}}
- { 'execute': 'human-monitor-command','arguments': {'command-line':
'drive_del blk-buddy0'}}

docs/COLO-FT.txt +22 -8
-  Note: there is no qmp command to remove the blockdev now
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'filter-mirror',
+'id': 'm0', 'props': { 'insert': 'before', 'position': 'id=rew0',
+'netdev': 'hn0', 'queue': 'tx', 'outdev': 'mirror0' } } }
+{'execute': 'object-add', 'arguments':{ 'qom-type':
+'filter-redirector', 'id': 'redire0', 'props': { 'insert': 'before',
+'position': 'id=rew0', 'netdev': 'hn0', 'queue': 'rx', 'indev':
+'compare_out' } } }
+{'execute': 'object-add', 'arguments':{ 'qom-type':
+'filter-redirector', 'id': 'redire1', 'props': { 'insert': 'before',
+'position': 'id=rew0', 'netdev': 'hn0', 'queue': 'rx', 'outdev':
+'compare0' } } }
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'iothread', 'id':
+'iothread1' } }
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'colo-compare',
+'id': 'comp0', 'props': { 'primary_in': 'compare0-0', 'secondary_in':
+'compare1', 'outdev': 'compare_out0', 'iothread': 'iothread1' } } }
 
-Secondary:
-  The primary host is down, so we should do the following thing:
-  { 'execute': 'nbd-server-stop' }
+{'execute': 'migrate-set-capabilities', 'arguments':{ 'capabilities': [
+{'capability': 'x-colo', 'state': true } ] } }
+{'execute': 'migrate', 'arguments':{ 'uri': 'tcp:127.0.0.1:9998' } }
 
 == TODO ==
-1. Support continuous VM replication.
-2. Support shared storage.
-3. Develop the heartbeat part.
-4. Reduce checkpoint VM’s downtime while doing checkpoint.
+1. Support shared storage.
+2. Develop the heartbeat part.
+3. Reduce checkpoint VM’s downtime while doing checkpoint.
docs/block-replication.txt index renamed from docs/block-replication.txt +0 -0

6bde6737fb..108e9166a8 100644

docs/block-replication.txt +6 -6
@@ -65,12 +65,12 @@ blocks that are already in QEMU.
              ^            ||                            .----------
              |            ||                            | Secondary
         1 Quorum          ||                            '----------
-         /      \         ||
-        /        \        ||
-   Primary    2 filter
-     disk         ^                                                             virtio-blk
-                  |                                                                  ^
-                3 NBD  ------->  3 NBD                                               |
+         /      \         ||                                                           virtio-blk
+        /        \        ||                                                               ^
+   Primary    2 filter                                                                     |
+     disk         ^                                                                   7 Quorum
+                  |                                                                    /
+                3 NBD  ------->  3 NBD                                                /
                 client    ||     server                                          2 filter
                           ||        ^                                                ^
 --------.                 ||        |                                                |
@@ -106,6 +106,10 @@ any state that would otherwise be lost by the

speculative write-through of the NBD server into the secondary disk. So before block replication, the primary disk and secondary disk should contain the same data.

+7) The secondary also has a quorum node, so after secondary failover it

docs/block-replication.txt +12 -5
+can become the new primary and continue replication.
+
+
 == Failure Handling ==
 There are 7 internal errors when block replication is running:
 1. I/O error on primary disk
@@ -171,16 +175,18 @@ Primary:
      leading whitespace.
   5. The qmp command line must be run after running qmp command line in
      secondary qemu.
-  6. After failover we need remove children.1 (replication driver).
+  6. After primary failover we need remove children.1 (replication driver).
 
 Secondary:
   -drive if=none,driver=raw,file.filename=1.raw,id=colo1 \
-  -drive if=xxx,id=topxxx,driver=replication,mode=secondary,top-
 id=topxxx\
+  -drive
+ if=none,id=childs1,driver=replication,mode=secondary,top-id=childs1
          file.file.filename=active_disk.qcow2,\
          file.driver=qcow2,\
          file.backing.file.filename=hidden_disk.qcow2,\
          file.backing.driver=qcow2,\
          file.backing.backing=colo1
+  -drive if=xxx,driver=quorum,read-pattern=fifo,id=top-disk1,\
+         vote-threshold=1,children.0=childs1
 
   Then run qmp command in secondary qemu:
     { 'execute': 'nbd-server-start',
@@ -234,6 +240,8 @@ Secondary:
   The primary host is down, so we should do the following thing:
   { 'execute': 'nbd-server-stop' }
 
+Promote Secondary to Primary:
+  see COLO-FT.txt
+
 TODO:
-1. Continuous block replication
-2. Shared disk
+1. Shared disk
--

2.20.1

#14 lore raw

RE: [PATCH v5 4/4] colo: Update Documentation for continuous replication

Quoted 169 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Saturday, September 28, 2019 9:25 PM
To: Zhang, Chen <[email protected]>
Cc: qemu-devel <[email protected]>; Jason Wang <[email protected]>; Wen Congyang <[email protected]>; Xie Changlong <[email protected]>; [email protected]; [email protected]
Subject: Re: [PATCH v5 4/4] colo: Update Documentation for continuous replication

Quoted 6 lines · Chen
Quoted 6 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Monday, September 16, 2019 3:20 AM
To: qemu-devel <[email protected]>
Cc: Zhang, Chen <[email protected]>; Jason Wang <[email protected]>; Wen Congyang

<[email protected]>; Xie

Quoted 14 lines
Quoted 14 lines

Changlong <[email protected]>; [email protected]; [email protected]
Subject: [PATCH v5 4/4] colo: Update Documentation for continuous replication

Document the qemu command-line and qmp commands for continuous replication

---
 docs/COLO-FT.txt           | 212 +++++++++++++++++++++++++++----------
 docs/block-replication.txt |  28 +++--
 2 files changed, 173 insertions(+), 67 deletions(-)
Quoted 5 lines

docs/COLO-FT.txt The diagram just shows the main qmp

Quoted 5 lines
docs/COLO-FT.txt index renamed from docs/COLO-FT.txt +0 -0

ad24680d13..e98c95ca2b 100644

docs/COLO-FT.txt +0 -0
@@ -145,35 +145,65 @@ The diagram just shows the main qmp

command,

Quoted 37 lines
Quoted 37 lines

you can get the detail in test procedure.

== Test procedure ==
-1. Startup qemu

docs/COLO-FT.txt +0 -2
-Primary:
-# qemu-system-x86_64 -accel kvm -m 2048 -smp 2 -qmp stdio -name

primary \

docs/COLO-FT.txt +0 -4
-  -device piix3-usb-uhci -vnc :7 \
-  -device usb-tablet -netdev tap,id=hn0,vhost=off \
-  -device virtio-net-pci,id=net-pci0,netdev=hn0 \
-  -drive

if=virtio,id=primary-disk0,driver=quorum,read-pattern=fifo,vote- threshold=1,\

docs/COLO-FT.txt +0 -4
-         children.0.file.filename=1.raw,\
-         children.0.driver=raw -S
-Secondary:
-# qemu-system-x86_64 -accel kvm -m 2048 -smp 2 -qmp stdio -name

secondary \

docs/COLO-FT.txt +0 -4
-  -device piix3-usb-uhci -vnc :7 \
-  -device usb-tablet -netdev tap,id=hn0,vhost=off \
-  -device virtio-net-pci,id=net-pci0,netdev=hn0 \
-  -drive

if=none,id=secondary-disk0,file.filename=1.raw,driver=raw,node- name=node0 \

docs/COLO-FT.txt +4 -9
-  -drive if=virtio,id=active-disk0,driver=replication,mode=secondary,\
-         file.driver=qcow2,top-id=active-disk0,\
-         file.file.filename=/mnt/ramfs/active_disk.img,\
-         file.backing.driver=qcow2,\
-         file.backing.file.filename=/mnt/ramfs/hidden_disk.img,\
-         file.backing.backing=secondary-disk0 \
-  -incoming tcp:0:8888
-
-2. On Secondary VM's QEMU monitor, issue command
+Note: Here we are running both instances on the same Machine for
+testing, change the IP Addresses if you want to run it on two Hosts
+
+== Startup qemu ==
Quoted 8 lines

docs/COLO-FT.txt

Quoted 8 lines

+1. Primary:

docs/COLO-FT.txt +7 -0
+# imagefolder="/mnt/vms/colo-test"
+
+The disks for the primary and secondary need to be the same before
+starting colo # cp --reflink=auto $imagefolder/primary.qcow2
+$imagefolder/primary-copy.qcow2
+
+# qemu-system-x86_64 -enable-kvm -cpu qemu64,+kvmclock -m 512 -

smp

Quoted 9 lines

docs/COLO-FT.txt The diagram just shows the main qmp command, you can get the detail

Quoted 9 lines

1 -qmp stdio \

docs/COLO-FT.txt +3 -0
+   -vnc :0 -device piix3-usb-uhci -device usb-tablet -name primary \
+   -netdev
+ tap,id=hn0,vhost=off,helper=/usr/lib/qemu/qemu-bridge-helper

\

docs/COLO-FT.txt +4 -0
+   -device rtl8139,id=e0,netdev=hn0 \
+   -chardev socket,id=mirror0,host=127.0.0.1,port=9003,server,nowait \
+   -chardev socket,id=compare1,host=127.0.0.1,port=9004,server,wait \
+   -chardev socket,id=compare0,host=127.0.0.1,port=9001,server,nowait

\

Quoted 20 lines
Quoted 20 lines
docs/COLO-FT.txt +3 -0
+   -chardev socket,id=compare0-0,host=127.0.0.1,port=9001 \
+   -chardev
+ socket,id=compare_out,host=127.0.0.1,port=9005,server,nowait

\

docs/COLO-FT.txt +3 -0
+   -chardev socket,id=compare_out0,host=127.0.0.1,port=9005 \
+   -object filter-mirror,id=m0,netdev=hn0,queue=tx,outdev=mirror0 \
+   -object filter-

redirector,netdev=hn0,id=redire0,queue=rx,indev=compare_out \
+ -object filter-
redirector,netdev=hn0,id=redire1,queue=rx,outdev=compare0 \

docs/COLO-FT.txt +3 -0
+   -object iothread,id=iothread1 \
+   -object
+colo-compare,id=comp0,primary_in=compare0-

0,secondary_in=compare1,\

docs/COLO-FT.txt +6 -0
+outdev=compare_out0,iothread=iothread1 \
+   -drive
+if=ide,id=colo-disk0,driver=quorum,read-pattern=fifo,vote-threshold
+=1,\
+children.0.file.filename=$imagefolder/primary.qcow2,children.0.driv
+er=q
Quoted 8 lines

docs/COLO-FT.txt The diagram just shows the main qmp command, you can get the detail

Quoted 8 lines
docs/COLO-FT.txt +8 -0
+cow2 -S
+
+2. Secondary:
+# imagefolder="/mnt/vms/colo-test"
+
+# qemu-img create -f qcow2 $imagefolder/secondary-active.qcow2 10G
+
+# qemu-img create -f qcow2 $imagefolder/secondary-hidden.qcow2

10G

Quoted 2 lines

docs/COLO-FT.txt

Quoted 2 lines
docs/COLO-FT.txt +2 -0
+
+# qemu-system-x86_64 -enable-kvm -cpu qemu64,+kvmclock -m 512 -

smp

Quoted 24 lines
Quoted 24 lines

1 -qmp stdio \

docs/COLO-FT.txt +3 -0
+   -vnc :1 -device piix3-usb-uhci -device usb-tablet -name secondary \
+   -netdev
+ tap,id=hn0,vhost=off,helper=/usr/lib/qemu/qemu-bridge-helper

\

docs/COLO-FT.txt +7 -0
+   -device rtl8139,id=e0,netdev=hn0 \
+   -chardev socket,id=red0,host=127.0.0.1,port=9003,reconnect=1 \
+   -chardev socket,id=red1,host=127.0.0.1,port=9004,reconnect=1 \
+   -object filter-redirector,id=f1,netdev=hn0,queue=tx,indev=red0 \
+   -object filter-redirector,id=f2,netdev=hn0,queue=rx,outdev=red1 \
+   -object filter-rewriter,id=rew0,netdev=hn0,queue=all \
+   -drive if=none,id=parent0,file.filename=$imagefolder/primary-

copy.qcow2,driver=qcow2 \

docs/COLO-FT.txt +11 -0
+   -drive
+if=none,id=childs0,driver=replication,mode=secondary,file.driver=qc
+ow2,
+\
+top-id=childs0,file.file.filename=$imagefolder/secondary-active.qco
+w2,\
+file.backing.driver=qcow2,file.backing.file.filename=$imagefolder/s
+econ
+dary-hidden.qcow2,\
+file.backing.backing=parent0 \
+   -drive
Quoted 8 lines

docs/COLO-FT.txt The diagram just shows the main qmp command, you can get the detail · line 201

Quoted 5 lines
docs/COLO-FT.txt +5 -0
+if=ide,id=colo-disk0,driver=quorum,read-pattern=fifo,vote-threshold
+=1,\
+children.0=childs0 \
+   -incoming tcp:0:9998
+

I know in your COLO test environment, all the node running in same physical machine, but current COLO start parameter demo easy to

misunderstand users.

Quoted 1 line

I think we can introduce primary node as 192.168.1.1, secondary node as

192.168.1.2.

Quoted 3 lines

Same issue for the VM image, we should avoid the "primary-copy.qow2". Make two node independent is more clear.

Quoted 9 lines

Ok, I will remove the primary-copy stuff. But I have a better idea for the IP addresses:
Using 127.0.0.1 for Primary and 127.0.0.2 for Secondary. Both point to localhost, making quick testing by simple copy-paste possible when using different values for the imagefolder Variable. At the same time Users see which IP to substitute for the remote node when using multiple Hosts and we avoid the confusion with primary-copy.
What do you think?

I think it is a good idea~

Thanks
Zhang Chen

Quoted 350 lines
Quoted 128 lines
Quoted 128 lines
docs/COLO-FT.txt +2 -2
+
+3. On Secondary VM's QEMU monitor, issue command
 {'execute':'qmp_capabilities'}
-{ 'execute': 'nbd-server-start',
-  'arguments': {'addr': {'type': 'inet', 'data': {'host': 'xx.xx.xx.xx', 'port':

'8889'} } } -}
-{'execute': 'nbd-server-add', 'arguments': {'device':
'secondary-disk0',
'writable': true } }

docs/COLO-FT.txt +5 -0
+{'execute': 'nbd-server-start', 'arguments': {'addr': {'type':
+'inet',
+'data': {'host': '127.0.0.1', 'port': '9999'} } } }
+{'execute': 'nbd-server-add', 'arguments': {'device': 'parent0',
+'writable': true } }
 
 Note:
   a. The qmp command nbd-server-start and nbd-server-add must be

run @@ -182,44 +212,112 @@ Note:
same.
c. It is better to put active disk and hidden disk in ramdisk.

-3. On Primary VM's QEMU monitor, issue command:
+4. On Primary VM's QEMU monitor, issue command:
{'execute':'qmp_capabilities'}

docs/COLO-FT.txt +0 -2
-{ 'execute': 'human-monitor-command',
-  'arguments': {'command-line': 'drive_add -n buddy

driver=replication,mode=primary,file.driver=nbd,file.host=xx.xx.xx.x x,file.p ort=8889,file.export=secondary-disk0,node-name=nbd_client0'}} -{ 'execute':'x-blockdev-change', 'arguments':{'parent':
'primary-disk0',
'node': 'nbd_client0' } } -{ 'execute': 'migrate-set-capabilities',

docs/COLO-FT.txt +0 -2
-      'arguments': {'capabilities': [ {'capability': 'x-colo', 'state': true } ] } }
-{ 'execute': 'migrate', 'arguments': {'uri': 'tcp:xx.xx.xx.xx:8888'

} }

docs/COLO-FT.txt +12 -2
+{'execute': 'human-monitor-command', 'arguments': {'command-line':
+'drive_add -n buddy
+driver=replication,mode=primary,file.driver=nbd,file.host=127.0.0.1
+,fil e.port=9999,file.export=parent0,node-name=replication0'}}
+{'execute': 'x-blockdev-change', 'arguments':{'parent':
+'colo-disk0',
+'node': 'replication0' } }
+{'execute': 'migrate-set-capabilities', 'arguments':
+{'capabilities': [
+{'capability': 'x-colo', 'state': true } ] } }
+{'execute': 'migrate', 'arguments': {'uri': 'tcp:127.0.0.1:9998' }
+}
 
   Note:
   a. There should be only one NBD Client for each primary disk.
-  b. xx.xx.xx.xx is the secondary physical machine's hostname or IP
-  c. The qmp command line must be run after running qmp command

line in
+ b. The qmp command line must be run after running qmp command
+ line in
secondary qemu.

-4. After the above steps, you will see, whenever you make changes to PVM, SVM will be synced.
+5. After the above steps, you will see, whenever you make changes +to PVM,
SVM will be synced.
You can issue command '{ "execute": "migrate-set-parameters" ,
"arguments":{ "x-checkpoint-delay": 2000 } }'

docs/COLO-FT.txt +7 -1
-to change the checkpoint period time
+to change the idle checkpoint period time
+
+6. Failover test
+You can kill one of the VMs and Failover on the surviving VM:
+
+If you killed the Secondary, then follow "Primary Failover". After
+that, if you want to resume the replication, follow "Primary resume

replication"

docs/COLO-FT.txt +4 -0
+
+If you killed the Primary, then follow "Secondary Failover". After
+that, if you want to resume the replication, follow "Secondary
+resume

replication"

docs/COLO-FT.txt +18 -0
+
+== Primary Failover ==
+The Secondary died, resume on the Primary
+
+{'execute': 'x-blockdev-change', 'arguments':{ 'parent':
+'colo-disk0',
+'child': 'children.1'} }
+{'execute': 'human-monitor-command', 'arguments':{ 'command-line':
+'drive_del replication0' } }
+{'execute': 'object-del', 'arguments':{ 'id': 'comp0' } }
+{'execute': 'object-del', 'arguments':{ 'id': 'iothread1' } }
+{'execute': 'object-del', 'arguments':{ 'id': 'm0' } }
+{'execute': 'object-del', 'arguments':{ 'id': 'redire0' } }
+{'execute': 'object-del', 'arguments':{ 'id': 'redire1' } }
+{'execute': 'x-colo-lost-heartbeat' }
+
+== Secondary Failover ==
+The Primary died, resume on the Secondary and prepare to become the

new

docs/COLO-FT.txt +30 -0
+Primary
+
+{'execute': 'nbd-server-stop'}
+{'execute': 'x-colo-lost-heartbeat'}
+
+{'execute': 'object-del', 'arguments':{ 'id': 'f2' } }
+{'execute': 'object-del', 'arguments':{ 'id': 'f1' } }
+{'execute': 'chardev-remove', 'arguments':{ 'id': 'red1' } }
+{'execute': 'chardev-remove', 'arguments':{ 'id': 'red0' } }
+
+{'execute': 'chardev-add', 'arguments':{ 'id': 'mirror0', 'backend':
+{'type': 'socket', 'data': {'addr': { 'type': 'inet', 'data': { 'host':
+'127.0.0.1', 'port': '9003' } }, 'server': true } } } }
+{'execute': 'chardev-add', 'arguments':{ 'id': 'compare1', 'backend':
+{'type': 'socket', 'data': {'addr': { 'type': 'inet', 'data': { 'host':
+'127.0.0.1', 'port': '9004' } }, 'server': true } } } }
+{'execute': 'chardev-add', 'arguments':{ 'id': 'compare0', 'backend':
+{'type': 'socket', 'data': {'addr': { 'type': 'inet', 'data': { 'host':
+'127.0.0.1', 'port': '9001' } }, 'server': true } } } }
+{'execute': 'chardev-add', 'arguments':{ 'id': 'compare0-0', 'backend':
+{'type': 'socket', 'data': {'addr': { 'type': 'inet', 'data': { 'host':
+'127.0.0.1', 'port': '9001' } }, 'server': false } } } }
+{'execute': 'chardev-add', 'arguments':{ 'id': 'compare_out',
+'backend': {'type': 'socket', 'data': {'addr': { 'type': 'inet',
+'data': { 'host': '127.0.0.1', 'port': '9005' } }, 'server': true }
+} } }
+{'execute': 'chardev-add', 'arguments':{ 'id': 'compare_out0',
+'backend': {'type': 'socket', 'data': {'addr': { 'type': 'inet',
+'data': { 'host': '127.0.0.1', 'port': '9005' } }, 'server': false
+} } } }
Quoted 8 lines

docs/COLO-FT.txt Note: · line 273

Quoted 5 lines
docs/COLO-FT.txt +5 -0
+
+== Primary resume replication ==
+Resume replication after new Secondary is up.
+
+Start the new Secondary (Steps 2 and 3 above), then:

Please note here the "then" is the original secondary node. It is easy to misunderstand here.

Ok, I will make this more clear.

Quoted 6 lines

docs/COLO-FT.txt

Quoted 4 lines
docs/COLO-FT.txt +4 -0
+{'execute': 'drive-mirror', 'arguments':{ 'device': 'colo-disk0',
+'job-id': 'resync', 'target': 'nbd://127.0.0.1:9999/parent0', 'mode':
+'existing', 'format': 'raw', 'sync': 'full'} }
+

As I mentioned before, the 'nbd://127.0.0.1:9999/parent0' is very confusing,

user don't know which node will be connected.

Quoted 29 lines
Quoted 29 lines
docs/COLO-FT.txt +28 -0
+Wait until disk is synced, then:
+{'execute': 'stop'}
+{'execute': 'block-job-cancel', 'arguments':{ 'device': 'resync'} }
+
+{'execute': 'human-monitor-command', 'arguments':{ 'command-line':
+'drive_add -n buddy
+driver=replication,mode=primary,file.driver=nbd,file.host=127.0.0.1
+,fil e.port=9999,file.export=parent0,node-name=replication0'}}
+{'execute': 'x-blockdev-change', 'arguments':{ 'parent':
+'colo-disk0',
+'node': 'replication0' } }
+
+{'execute': 'object-add', 'arguments':{ 'qom-type':
+'filter-mirror',
+'id': 'm0', 'props': { 'netdev': 'hn0', 'queue': 'tx', 'outdev':
+'mirror0' } } }
+{'execute': 'object-add', 'arguments':{ 'qom-type':
+'filter-redirector', 'id': 'redire0', 'props': { 'netdev': 'hn0',
+'queue': 'rx', 'indev': 'compare_out' } } }
+{'execute': 'object-add', 'arguments':{ 'qom-type':
+'filter-redirector', 'id': 'redire1', 'props': { 'netdev': 'hn0',
+'queue': 'rx', 'outdev': 'compare0' } } }
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'iothread', 'id':
+'iothread1' } }
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'colo-compare',
+'id': 'comp0', 'props': { 'primary_in': 'compare0-0', 'secondary_in':
+'compare1', 'outdev': 'compare_out0', 'iothread': 'iothread1' } } }
+
Quoted 8 lines

docs/COLO-FT.txt

Quoted 6 lines
docs/COLO-FT.txt +6 -0
+{'execute': 'migrate-set-capabilities', 'arguments':{
+'capabilities': [
+{'capability': 'x-colo', 'state': true } ] } }
+{'execute': 'migrate', 'arguments':{ 'uri': 'tcp:127.0.0.1:9998' }
+}
+

The guest network have issues here in my test, I can't ping the guest from

the host.

Quoted 1 line

Is it works in your test environment?

Yes, it works for me (based on v4.1.0). I simply copy-paste everything here 1:1 for testing.

Regards,
Lukas Straub

Quoted 38 lines

By the way, after finish this doc we can sync it to Qemu COLO WIKI page.

Thanks
Zhang Chen

Quoted 32 lines
docs/COLO-FT.txt +19 -2
+Note:
+If this Primary previously was a Secondary, then we need to insert
+the filters before the filter-rewriter by using the
+"'insert': 'before', 'position': 'rew0'" Options. See below.
+
+== Secondary resume replication ==
+Become Primary and resume replication after new Secondary is up.
+
+Start the new Secondary (Steps 2 and 3 above, but with different
+image than
+primary-copy.qcow2 and different VNC port), then:
+{'execute': 'drive-mirror', 'arguments':{ 'device': 'colo-disk0',
+'job-id': 'resync', 'target': 'nbd://127.0.0.1:9999/parent0', 'mode':
+'existing', 'format': 'raw', 'sync': 'full'} }
+
+Wait until disk is synced, then:
+{'execute': 'stop'}
+{'execute': 'block-job-cancel', 'arguments':{ 'device': 'resync' }
+}
 
-5. Failover test
-You can kill Primary VM and run 'x_colo_lost_heartbeat' in

Secondary VM's - monitor at the same time, then SVM will failover and client will not detect this -change.

docs/COLO-FT.txt +7 -0
+{'execute': 'human-monitor-command', 'arguments':{ 'command-line':
+'drive_add -n buddy
+driver=replication,mode=primary,file.driver=nbd,file.host=127.0.0.1
+,fil e.port=9999,file.export=parent0,node-name=replication0'}}
+{'execute': 'x-blockdev-change', 'arguments':{ 'parent':
+'colo-disk0',
+'node': 'replication0' } }
Quoted 5 lines

docs/COLO-FT.txt Note:

Quoted 5 lines

-Before issuing '{ "execute": "x-colo-lost-heartbeat" }' command, we have to -issue block related command to stop block replication.

docs/COLO-FT.txt +0 -3
-Primary:
-  Remove the nbd child from the quorum:
-  { 'execute': 'x-blockdev-change', 'arguments': {'parent': 'colo-disk0',

'child':

Quoted 70 lines
Quoted 70 lines

'children.1'}}
- { 'execute': 'human-monitor-command','arguments': {'command-line':
'drive_del blk-buddy0'}}

docs/COLO-FT.txt +27 -8
-  Note: there is no qmp command to remove the blockdev now
+{'execute': 'object-add', 'arguments':{ 'qom-type':
+'filter-mirror',
+'id': 'm0', 'props': { 'insert': 'before', 'position': 'id=rew0',
+'netdev': 'hn0', 'queue': 'tx', 'outdev': 'mirror0' } } }
+{'execute': 'object-add', 'arguments':{ 'qom-type':
+'filter-redirector', 'id': 'redire0', 'props': { 'insert':
+'before',
+'position': 'id=rew0', 'netdev': 'hn0', 'queue': 'rx', 'indev':
+'compare_out' } } }
+{'execute': 'object-add', 'arguments':{ 'qom-type':
+'filter-redirector', 'id': 'redire1', 'props': { 'insert':
+'before',
+'position': 'id=rew0', 'netdev': 'hn0', 'queue': 'rx', 'outdev':
+'compare0' } } }
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'iothread', 'id':
+'iothread1' } }
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'colo-compare',
+'id': 'comp0', 'props': { 'primary_in': 'compare0-0', 'secondary_in':
+'compare1', 'outdev': 'compare_out0', 'iothread': 'iothread1' } } }
 
-Secondary:
-  The primary host is down, so we should do the following thing:
-  { 'execute': 'nbd-server-stop' }
+{'execute': 'migrate-set-capabilities', 'arguments':{
+'capabilities': [
+{'capability': 'x-colo', 'state': true } ] } }
+{'execute': 'migrate', 'arguments':{ 'uri': 'tcp:127.0.0.1:9998' }
+}
 
 == TODO ==
-1. Support continuous VM replication.
-2. Support shared storage.
-3. Develop the heartbeat part.
-4. Reduce checkpoint VM’s downtime while doing checkpoint.
+1. Support shared storage.
+2. Develop the heartbeat part.
+3. Reduce checkpoint VM’s downtime while doing checkpoint.
docs/block-replication.txt +0 -0

index
6bde6737fb..108e9166a8 100644

docs/block-replication.txt +6 -6
@@ -65,12 +65,12 @@ blocks that are already in QEMU.
              ^            ||                            .----------
              |            ||                            | Secondary
         1 Quorum          ||                            '----------
-         /      \         ||
-        /        \        ||
-   Primary    2 filter
-     disk         ^                                                             virtio-blk
-                  |                                                                  ^
-                3 NBD  ------->  3 NBD                                               |
+         /      \         ||                                                           virtio-blk
+        /        \        ||                                                               ^
+   Primary    2 filter                                                                     |
+     disk         ^                                                                   7 Quorum
+                  |                                                                    /
+                3 NBD  ------->  3 NBD                                                /
                 client    ||     server                                          2 filter
                           ||        ^                                                ^
 --------.                 ||        |                                                |
@@ -106,6 +106,10 @@ any state that would otherwise be lost by the

speculative write-through of the NBD server into the secondary disk. So before block replication, the primary disk and secondary disk should contain the same data.

Quoted 10 lines

docs/block-replication.txt Primary: · line 176

Quoted 10 lines

+7) The secondary also has a quorum node, so after secondary

docs/block-replication.txt +3 -0
+failover it can become the new primary and continue replication.
+
+
 == Failure Handling ==
 There are 7 internal errors when block replication is running:
 1. I/O error on primary disk
@@ -171,16 +175,18 @@ Primary:
      leading whitespace.
   5. The qmp command line must be run after running qmp command line

in

Quoted 33 lines
Quoted 33 lines
docs/block-replication.txt +10 -5
      secondary qemu.
-  6. After failover we need remove children.1 (replication driver).
+  6. After primary failover we need remove children.1 (replication driver).
 
 Secondary:
   -drive if=none,driver=raw,file.filename=1.raw,id=colo1 \
-  -drive if=xxx,id=topxxx,driver=replication,mode=secondary,top-
 id=topxxx\
+  -drive
+ if=none,id=childs1,driver=replication,mode=secondary,top-id=childs
+ 1
          file.file.filename=active_disk.qcow2,\
          file.driver=qcow2,\
          file.backing.file.filename=hidden_disk.qcow2,\
          file.backing.driver=qcow2,\
          file.backing.backing=colo1
+  -drive if=xxx,driver=quorum,read-pattern=fifo,id=top-disk1,\
+         vote-threshold=1,children.0=childs1
 
   Then run qmp command in secondary qemu:
     { 'execute': 'nbd-server-start', @@ -234,6 +240,8 @@ Secondary:
   The primary host is down, so we should do the following thing:
   { 'execute': 'nbd-server-stop' }
 
+Promote Secondary to Primary:
+  see COLO-FT.txt
+
 TODO:
-1. Continuous block replication
-2. Shared disk
+1. Shared disk
--

2.20.1

#15 lore raw
In reply to Lukas Straub
Quoted 45 lines · On Sun, 15 Sep 2019 21:19:50 +0200 Lukas Straub <[email protected]> wrote:

Hello Everyone,
These Patches add support for continuous replication to colo. This means that after the Primary fails and the Secondary did a failover, the Secondary can then become Primary and resume replication to a new Secondary.

On a side note, I wrote a Pacemaker Resource Agent for colo which I will post when its ready.
I have to say it's quite fun to randomly kill a Node, wait for resync and repeat and see how the VM stays alive all the time. :)

Regards,
Lukas Straub

v5:
- change syntax for the position= parameter
- fix spelling mistake

v4:
- fix checkpatch.pl warnings

v3:
- add test for replication changes
- check if the filter to be inserted before/behind belongs to the same interface
- fix the error message for the position= parameter
- rename term "after" -> "behind" and variable "insert_before" -> "insert_before_flag"
- document the quorum node on the secondary side
- simplify quorum parameters in documentation
- remove trailing spaces in documentation
- clarify the testing procedure in documentation

v2:
- fix email formating
- fix checkpatch.pl warnings
- fix patchew error
- clearer commit messages

Lukas Straub (4):
block/replication.c: Ignore requests after failover tests/test-replication.c: Add test for ignoring requests after
failover
net/filter.c: Add Options to insert filters anywhere in the filter
list
colo: Update Documentation for continuous replication

Quoted 9 lines
 block/replication.c        |  38 ++++++-
 docs/COLO-FT.txt           | 212 +++++++++++++++++++++++++++----------
 docs/block-replication.txt |  28 +++--
 include/net/filter.h       |   2 +
 net/filter.c               |  92 +++++++++++++++-
 qemu-options.hx            |  10 +-
 tests/test-replication.c   |  52 +++++++++
 7 files changed, 358 insertions(+), 76 deletions(-)

Hello Everyone,
Ping.

Regards,
Lukas Straub

#16 lore raw
Quoted 72 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Friday, September 27, 2019 12:10 AM
To: qemu-devel <[email protected]>
Cc: Zhang, Chen <[email protected]>; Jason Wang <[email protected]>; Wen Congyang <[email protected]>; Xie Changlong <[email protected]>; [email protected]; [email protected]
Subject: Re: [PATCH v5 0/4] colo: Add support for continuous replication

Quoted 4 lines · On Sun, 15 Sep 2019 21:19:50 +0200 Lukas Straub <[email protected]> wrote:

Hello Everyone,
These Patches add support for continuous replication to colo. This means that after the Primary fails and the Secondary did a failover, the Secondary can then become Primary and resume replication to a new

Secondary.

Quoted 16 lines

On a side note, I wrote a Pacemaker Resource Agent for colo which I will post when its ready.
I have to say it's quite fun to randomly kill a Node, wait for resync and repeat and see how the VM stays alive all the time. :)

Regards,
Lukas Straub

v5:
- change syntax for the position= parameter
- fix spelling mistake

v4:
- fix checkpatch.pl warnings

Quoted 6 lines

v3:
- add test for replication changes
- check if the filter to be inserted before/behind belongs to the
same interface
- fix the error message for the position= parameter
- rename term "after" -> "behind" and variable "insert_before" ->

"insert_before_flag"

Quoted 20 lines

- document the quorum node on the secondary side
- simplify quorum parameters in documentation
- remove trailing spaces in documentation
- clarify the testing procedure in documentation

v2:
- fix email formating
- fix checkpatch.pl warnings
- fix patchew error
- clearer commit messages

Lukas Straub (4):
block/replication.c: Ignore requests after failover tests/test-replication.c: Add test for ignoring requests after
failover
net/filter.c: Add Options to insert filters anywhere in the filter
list
colo: Update Documentation for continuous replication

Quoted 9 lines
 block/replication.c        |  38 ++++++-
 docs/COLO-FT.txt           | 212 +++++++++++++++++++++++++++----------
 docs/block-replication.txt |  28 +++--
 include/net/filter.h       |   2 +
 net/filter.c               |  92 +++++++++++++++-
 qemu-options.hx            |  10 +-
 tests/test-replication.c   |  52 +++++++++
 7 files changed, 358 insertions(+), 76 deletions(-)
Quoted 2 lines

Hello Everyone,
Ping.

Sorry for slow reply, I'm reviewing~

Thanks
Zhang Chen

Quoted 3 lines

Regards,
Lukas Straub