Skip to contentSkip to conversation

colo: Add support for continuous replication

PATCH In review Lukas Straub 4 participants 13 messages Oct 24, 2019 – Feb 20, 2020

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

v7 cover colo: Add support for continuous replication

Hello Everyone,
These Patches add support for continuous replication to colo. This means that after the Primary fails and the Secondary did a failover, the Secondary can then become Primary and resume replication to a new Secondary.

Regards,
Lukas Straub

v7:
- clarify meaning of ip's in documentation and note that active and hidden images just need to be created once
- reverted removal of bdrv_is_root_node(top_bs) in replication and adjusted the top-id= parameter in documentation acordingly

v6:
- documented the position= and insert= options
- renamed replication test
- clarified documentation by using different ip's for primary and secondary
- added Reviewed-by tags

v5:
- change syntax for the position= parameter
- fix spelling mistake

v4:
- fix checkpatch.pl warnings

v3:
- add test for replication changes
- check if the filter to be inserted before/behind belongs to the same interface
- fix the error message for the position= parameter
- rename term "after" -> "behind" and variable "insert_before" -> "insert_before_flag"
- document the quorum node on the secondary side
- simplify quorum parameters in documentation
- remove trailing spaces in documentation
- clarify the testing procedure in documentation

v2:
- fix email formating
- fix checkpatch.pl warnings
- fix patchew error
- clearer commit messages

Lukas Straub (4):
block/replication.c: Ignore requests after failover tests/test-replication.c: Add test for for secondary node continuing
replication
net/filter.c: Add Options to insert filters anywhere in the filter
list
colo: Update Documentation for continuous replication

 block/replication.c        |  35 +++++-
 docs/COLO-FT.txt           | 224 +++++++++++++++++++++++++++----------
 docs/block-replication.txt |  28 +++--
 include/net/filter.h       |   2 +
 net/filter.c               |  92 ++++++++++++++-
 qemu-options.hx            |  31 ++++-
 tests/test-replication.c   |  52 +++++++++
 7 files changed, 389 insertions(+), 75 deletions(-)

--
2.20.1

#1 lore raw

PATCH v7 1/4 block/replication.c: Ignore requests after failover

After failover the Secondary side of replication shouldn't change state, because it now functions as our primary disk.

In replication_start, replication_do_checkpoint, replication_stop, ignore the request if current state is BLOCK_REPLICATION_DONE (sucessful failover) or BLOCK_REPLICATION_FAILOVER (failover in progres i.e. currently merging active and hidden images into the base image).

---
 block/replication.c | 35 ++++++++++++++++++++++++++++++++++-
 1 file changed, 34 insertions(+), 1 deletion(-)
block/replication.c +34 -1
index 3d4dedddfc..78234d1a04 100644
@@ -454,6 +454,17 @@ static void replication_start(ReplicationState *rs, ReplicationMode mode,
     aio_context_acquire(aio_context);
     s = bs->opaque;
 
+    if (s->stage == BLOCK_REPLICATION_DONE ||
+        s->stage == BLOCK_REPLICATION_FAILOVER) {
+        /*
+         * This case happens when a secondary is promoted to primary.
+         * Ignore the request because the secondary side of replication
+         * doesn't have to do anything anymore.
+         */
+        aio_context_release(aio_context);
+        return;
+    }
+
     if (s->stage != BLOCK_REPLICATION_NONE) {
         error_setg(errp, "Block replication is running or done");
         aio_context_release(aio_context);
@@ -577,6 +588,17 @@ static void replication_do_checkpoint(ReplicationState *rs, Error **errp)
     aio_context_acquire(aio_context);
     s = bs->opaque;
 
+    if (s->stage == BLOCK_REPLICATION_DONE ||
+        s->stage == BLOCK_REPLICATION_FAILOVER) {
+        /*
+         * This case happens when a secondary was promoted to primary.
+         * Ignore the request because the secondary side of replication
+         * doesn't have to do anything anymore.
+         */
+        aio_context_release(aio_context);
+        return;
+    }
+
     if (s->mode == REPLICATION_MODE_SECONDARY) {
         secondary_do_checkpoint(s, errp);
     }
@@ -593,7 +615,7 @@ static void replication_get_error(ReplicationState *rs, Error **errp)
     aio_context_acquire(aio_context);
     s = bs->opaque;
 
-    if (s->stage != BLOCK_REPLICATION_RUNNING) {
+    if (s->stage == BLOCK_REPLICATION_NONE) {
         error_setg(errp, "Block replication is not running");
         aio_context_release(aio_context);
         return;
@@ -635,6 +657,17 @@ static void replication_stop(ReplicationState *rs, bool failover, Error **errp)
     aio_context_acquire(aio_context);
     s = bs->opaque;
 
+    if (s->stage == BLOCK_REPLICATION_DONE ||
+        s->stage == BLOCK_REPLICATION_FAILOVER) {
+        /*
+         * This case happens when a secondary was promoted to primary.
+         * Ignore the request because the secondary side of replication
+         * doesn't have to do anything anymore.
+         */
+        aio_context_release(aio_context);
+        return;
+    }
+
     if (s->stage != BLOCK_REPLICATION_RUNNING) {
         error_setg(errp, "Block replication is not running");
         aio_context_release(aio_context);

--
2.20.1

Acked-by on v7 1/4
#2 lore raw

Re: [PATCH v7 1/4] block/replication.c: Ignore requests after failover

Quoted 8 lines · Lukas Straub

After failover the Secondary side of replication shouldn't change state, because it now functions as our primary disk.

In replication_start, replication_do_checkpoint, replication_stop, ignore the request if current state is BLOCK_REPLICATION_DONE (sucessful failover) or BLOCK_REPLICATION_FAILOVER (failover in progres i.e. currently merging active and hidden images into the base image).

Quoted 5 lines
---
 block/replication.c | 35 ++++++++++++++++++++++++++++++++++-
 1 file changed, 34 insertions(+), 1 deletion(-)
#3 lore raw

PATCH v7 2/4 tests/test-replication.c: Add test for for secondary node continuing replication

In reply to Lukas Straub

This simulates the case that happens when we resume COLO after failover.

---
 tests/test-replication.c | 52 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 52 insertions(+)
tests/test-replication.c +52 -0
index f085d1993a..8e18ecd998 100644
@@ -489,6 +489,56 @@ static void test_secondary_stop(void)
     teardown_secondary();
 }
 
+static void test_secondary_continuous_replication(void)
+{
+    BlockBackend *top_blk, *local_blk;
+    Error *local_err = NULL;
+
+    top_blk = start_secondary();
+    replication_start_all(REPLICATION_MODE_SECONDARY, &local_err);
+    g_assert(!local_err);
+
+    /* write 0x22 to s_local_disk (IMG_SIZE / 2, IMG_SIZE) */
+    local_blk = blk_by_name(S_LOCAL_DISK_ID);
+    test_blk_write(local_blk, 0x22, IMG_SIZE / 2, IMG_SIZE / 2, false);
+
+    /* replication will backup s_local_disk to s_hidden_disk */
+    test_blk_read(top_blk, 0x11, IMG_SIZE / 2,
+                  IMG_SIZE / 2, 0, IMG_SIZE, false);
+
+    /* write 0x33 to s_active_disk (0, IMG_SIZE / 2) */
+    test_blk_write(top_blk, 0x33, 0, IMG_SIZE / 2, false);
+
+    /* do failover (active commit) */
+    replication_stop_all(true, &local_err);
+    g_assert(!local_err);
+
+    /* it should ignore all requests from now on */
+
+    /* start after failover */
+    replication_start_all(REPLICATION_MODE_PRIMARY, &local_err);
+    g_assert(!local_err);
+
+    /* checkpoint */
+    replication_do_checkpoint_all(&local_err);
+    g_assert(!local_err);
+
+    /* stop */
+    replication_stop_all(true, &local_err);
+    g_assert(!local_err);
+
+    /* read from s_local_disk (0, IMG_SIZE / 2) */
+    test_blk_read(top_blk, 0x33, 0, IMG_SIZE / 2,
+                  0, IMG_SIZE / 2, false);
+
+
+    /* read from s_local_disk (IMG_SIZE / 2, IMG_SIZE) */
+    test_blk_read(top_blk, 0x22, IMG_SIZE / 2,
+                  IMG_SIZE / 2, 0, IMG_SIZE, false);
+
+    teardown_secondary();
+}
+
 static void test_secondary_do_checkpoint(void)
 {
     BlockBackend *top_blk, *local_blk;
@@ -584,6 +634,8 @@ int main(int argc, char **argv)
     g_test_add_func("/replication/secondary/write", test_secondary_write);
     g_test_add_func("/replication/secondary/start", test_secondary_start);
     g_test_add_func("/replication/secondary/stop",  test_secondary_stop);
+    g_test_add_func("/replication/secondary/continuous_replication",
+                    test_secondary_continuous_replication);
     g_test_add_func("/replication/secondary/do_checkpoint",
                     test_secondary_do_checkpoint);
     g_test_add_func("/replication/secondary/get_error_all",

--
2.20.1

#4 lore raw

PATCH v7 3/4 net/filter.c: Add Options to insert filters anywhere in the filter list

In reply to Lukas Straub

To switch the Secondary to Primary, we need to insert new filters before the filter-rewriter.

Add the options insert= and position= to be able to insert filters anywhere in the filter list.

position should be "head" or "tail" to insert at the head or tail of the filter list or it should be "id=<id>" to specify the id of another filter.
insert should be either "before" or "behind" to specify where to insert the new filter relative to the one specified with position.

---
 include/net/filter.h |  2 +
 net/filter.c         | 92 +++++++++++++++++++++++++++++++++++++++++++-
 qemu-options.hx      | 31 ++++++++++++---
 3 files changed, 119 insertions(+), 6 deletions(-)
include/net/filter.h +2 -0
index 49da666ac0..22a723305b 100644
@@ -62,6 +62,8 @@ struct NetFilterState {
     NetClientState *netdev;
     NetFilterDirection direction;
     bool on;
+    char *position;
+    bool insert_before_flag;
     QTAILQ_ENTRY(NetFilterState) next;
 };
 
net/filter.c +91 -1

132 lines not on this page · raw message

qemu-options.hx +26 -5
index 08749a3391..d2a6cb7da1 100644
@@ -4368,7 +4368,7 @@ applications, they can do this through this parameter. Its format is
 a gnutls priority string as described at
 @url{https://gnutls.org/manual/html_node/Priority-Strings.html}.
 
-@item -object filter-buffer,id=@var{id},netdev=@var{netdevid},interval=@var{t}[,queue=@var{all|rx|tx}][,status=@var{on|off}]
+@item -object filter-buffer,id=@var{id},netdev=@var{netdevid},interval=@var{t}[,queue=@var{all|rx|tx}][,status=@var{on|off}][,position=@var{head|tail|id=<id>}][,insert=@var{behind|before}]
 
 Interval @var{t} can't be 0, this filter batches the packet delivery: all
 packets arriving in a given interval on netdev @var{netdevid} are delayed
@@ -4387,11 +4387,32 @@ queue @var{all|rx|tx} is an option that can be applied to any netfilter.
 @option{tx}: the filter is attached to the transmit queue of the netdev,
              where it will receive packets sent by the netdev.
 
-@item -object filter-mirror,id=@var{id},netdev=@var{netdevid},outdev=@var{chardevid},queue=@var{all|rx|tx}[,vnet_hdr_support]
+position @var{head|tail|id=<id>} is an option to specify where the
+filter should be inserted in the filter list. It can be applied to any
+netfilter.
+
+@option{head}: the filter is inserted at the head of the filter
+               list, before any existing filters.
+
+@option{tail}: the filter is inserted at the tail of the filter
+               list, behind any existing filters (default).
+
+@option{id=<id>}: the filter is inserted before or behind the filter
+                  specified by <id>, see the insert option below.
+
+insert @var{behind|before} is an option to specify where to insert the
+new filter relative to the one specified with position=id=<id>. It can
+be applied to any netfilter.
+
+@option{before}: insert before the specified filter.
+
+@option{behind}: insert behind the specified filter (default).
+
+@item -object filter-mirror,id=@var{id},netdev=@var{netdevid},outdev=@var{chardevid},queue=@var{all|rx|tx}[,vnet_hdr_support][,position=@var{head|tail|id=<id>}][,insert=@var{behind|before}]
 
 filter-mirror on netdev @var{netdevid},mirror net packet to chardev@var{chardevid}, if it has the vnet_hdr_support flag, filter-mirror will mirror packet with vnet_hdr_len.
 
-@item -object filter-redirector,id=@var{id},netdev=@var{netdevid},indev=@var{chardevid},outdev=@var{chardevid},queue=@var{all|rx|tx}[,vnet_hdr_support]
+@item -object filter-redirector,id=@var{id},netdev=@var{netdevid},indev=@var{chardevid},outdev=@var{chardevid},queue=@var{all|rx|tx}[,vnet_hdr_support][,position=@var{head|tail|id=<id>}][,insert=@var{behind|before}]
 
 filter-redirector on netdev @var{netdevid},redirect filter's net packet to chardev
 @var{chardevid},and redirect indev's packet to filter.if it has the vnet_hdr_support flag,
@@ -4400,7 +4421,7 @@ Create a filter-redirector we need to differ outdev id from indev id, id can not
 be the same. we can just use indev or outdev, but at least one of indev or outdev
 need to be specified.
 
-@item -object filter-rewriter,id=@var{id},netdev=@var{netdevid},queue=@var{all|rx|tx},[vnet_hdr_support]
+@item -object filter-rewriter,id=@var{id},netdev=@var{netdevid},queue=@var{all|rx|tx},[vnet_hdr_support][,position=@var{head|tail|id=<id>}][,insert=@var{behind|before}]
 
 Filter-rewriter is a part of COLO project.It will rewrite tcp packet to
 secondary from primary to keep secondary tcp connection,and rewrite
@@ -4413,7 +4434,7 @@ colo secondary:
 -object filter-redirector,id=f2,netdev=hn0,queue=rx,outdev=red1
 -object filter-rewriter,id=rew0,netdev=hn0,queue=all
 
-@item -object filter-dump,id=@var{id},netdev=@var{dev}[,file=@var{filename}][,maxlen=@var{len}]
+@item -object filter-dump,id=@var{id},netdev=@var{dev}[,file=@var{filename}][,maxlen=@var{len}][,position=@var{head|tail|id=<id>}][,insert=@var{behind|before}]
 
 Dump the network traffic on netdev @var{dev} to the file specified by
 @var{filename}. At most @var{len} bytes (64k by default) per packet are stored.

--
2.20.1

#5 lore raw

PATCH v7 4/4 colo: Update Documentation for continuous replication

In reply to Lukas Straub

Document the qemu command-line and qmp commands for continuous replication

---
 docs/COLO-FT.txt           | 224 +++++++++++++++++++++++++++----------
 docs/block-replication.txt |  28 +++--
 2 files changed, 184 insertions(+), 68 deletions(-)
docs/COLO-FT.txt +166 -58
index ad24680d13..c8e1740935 100644
@@ -145,81 +145,189 @@ The diagram just shows the main qmp command, you can get the detail
 in test procedure.
 
 == Test procedure ==
-1. Startup qemu
-Primary:
-# qemu-system-x86_64 -accel kvm -m 2048 -smp 2 -qmp stdio -name primary \
-  -device piix3-usb-uhci -vnc :7 \
-  -device usb-tablet -netdev tap,id=hn0,vhost=off \
-  -device virtio-net-pci,id=net-pci0,netdev=hn0 \
-  -drive if=virtio,id=primary-disk0,driver=quorum,read-pattern=fifo,vote-threshold=1,\
-         children.0.file.filename=1.raw,\
-         children.0.driver=raw -S
-Secondary:
-# qemu-system-x86_64 -accel kvm -m 2048 -smp 2 -qmp stdio -name secondary \
-  -device piix3-usb-uhci -vnc :7 \
-  -device usb-tablet -netdev tap,id=hn0,vhost=off \
-  -device virtio-net-pci,id=net-pci0,netdev=hn0 \
-  -drive if=none,id=secondary-disk0,file.filename=1.raw,driver=raw,node-name=node0 \
-  -drive if=virtio,id=active-disk0,driver=replication,mode=secondary,\
-         file.driver=qcow2,top-id=active-disk0,\
-         file.file.filename=/mnt/ramfs/active_disk.img,\
-         file.backing.driver=qcow2,\
-         file.backing.file.filename=/mnt/ramfs/hidden_disk.img,\
-         file.backing.backing=secondary-disk0 \
-  -incoming tcp:0:8888
-
-2. On Secondary VM's QEMU monitor, issue command
+Note: Here we are running both instances on the same host for testing,
+change the IP Addresses if you want to run it on two hosts. Initally
+127.0.0.1 is the Primary Host and 127.0.0.2 is the Secondary Host.
+
+== Startup qemu ==
+1. Primary:
+Note: Initally, $imagefolder/primary.qcow2 needs to be copied to all hosts.
+You don't need to change any IP's here, because 0.0.0.0 listens on any
+interface. The chardev's with 127.0.0.1 IP's loopback to the local qemu
+instance.
+
+# imagefolder="/mnt/vms/colo-test-primary"
+
+# qemu-system-x86_64 -enable-kvm -cpu qemu64,+kvmclock -m 512 -smp 1 -qmp stdio \
+   -device piix3-usb-uhci -device usb-tablet -name primary \
+   -netdev tap,id=hn0,vhost=off,helper=/usr/lib/qemu/qemu-bridge-helper \
+   -device rtl8139,id=e0,netdev=hn0 \
+   -chardev socket,id=mirror0,host=0.0.0.0,port=9003,server,nowait \
+   -chardev socket,id=compare1,host=0.0.0.0,port=9004,server,wait \
+   -chardev socket,id=compare0,host=127.0.0.1,port=9001,server,nowait \
+   -chardev socket,id=compare0-0,host=127.0.0.1,port=9001 \
+   -chardev socket,id=compare_out,host=127.0.0.1,port=9005,server,nowait \
+   -chardev socket,id=compare_out0,host=127.0.0.1,port=9005 \
+   -object filter-mirror,id=m0,netdev=hn0,queue=tx,outdev=mirror0 \
+   -object filter-redirector,netdev=hn0,id=redire0,queue=rx,indev=compare_out \
+   -object filter-redirector,netdev=hn0,id=redire1,queue=rx,outdev=compare0 \
+   -object iothread,id=iothread1 \
+   -object colo-compare,id=comp0,primary_in=compare0-0,secondary_in=compare1,\
+outdev=compare_out0,iothread=iothread1 \
+   -drive if=ide,id=colo-disk0,driver=quorum,read-pattern=fifo,vote-threshold=1,\
+children.0.file.filename=$imagefolder/primary.qcow2,children.0.driver=qcow2 -S
+
+2. Secondary:
+Note: Active and hidden images need to be created only once and the
+size should be the same as primary.qcow2. Again, you don't need to change
+any IP's here, except for the $primary_ip variable.
+
+# imagefolder="/mnt/vms/colo-test-secondary"
+# primary_ip=127.0.0.1
+
+# qemu-img create -f qcow2 $imagefolder/secondary-active.qcow2 10G
+
+# qemu-img create -f qcow2 $imagefolder/secondary-hidden.qcow2 10G
+
+# qemu-system-x86_64 -enable-kvm -cpu qemu64,+kvmclock -m 512 -smp 1 -qmp stdio \
+   -device piix3-usb-uhci -device usb-tablet -name secondary \
+   -netdev tap,id=hn0,vhost=off,helper=/usr/lib/qemu/qemu-bridge-helper \
+   -device rtl8139,id=e0,netdev=hn0 \
+   -chardev socket,id=red0,host=$primary_ip,port=9003,reconnect=1 \
+   -chardev socket,id=red1,host=$primary_ip,port=9004,reconnect=1 \
+   -object filter-redirector,id=f1,netdev=hn0,queue=tx,indev=red0 \
+   -object filter-redirector,id=f2,netdev=hn0,queue=rx,outdev=red1 \
+   -object filter-rewriter,id=rew0,netdev=hn0,queue=all \
+   -drive if=none,id=parent0,file.filename=$imagefolder/primary.qcow2,driver=qcow2 \
+   -drive if=none,id=childs0,driver=replication,mode=secondary,file.driver=qcow2,\
+top-id=colo-disk0,file.file.filename=$imagefolder/secondary-active.qcow2,\
+file.backing.driver=qcow2,file.backing.file.filename=$imagefolder/secondary-hidden.qcow2,\
+file.backing.backing=parent0 \
+   -drive if=ide,id=colo-disk0,driver=quorum,read-pattern=fifo,vote-threshold=1,\
+children.0=childs0 \
+   -incoming tcp:0.0.0.0:9998
+
+
+3. On Secondary VM's QEMU monitor, issue command
 {'execute':'qmp_capabilities'}
-{ 'execute': 'nbd-server-start',
-  'arguments': {'addr': {'type': 'inet', 'data': {'host': 'xx.xx.xx.xx', 'port': '8889'} } }
-}
-{'execute': 'nbd-server-add', 'arguments': {'device': 'secondary-disk0', 'writable': true } }
+{'execute': 'nbd-server-start', 'arguments': {'addr': {'type': 'inet', 'data': {'host': '0.0.0.0', 'port': '9999'} } } }
+{'execute': 'nbd-server-add', 'arguments': {'device': 'parent0', 'writable': true } }
 
 Note:
   a. The qmp command nbd-server-start and nbd-server-add must be run
      before running the qmp command migrate on primary QEMU
   b. Active disk, hidden disk and nbd target's length should be the
      same.
-  c. It is better to put active disk and hidden disk in ramdisk.
+  c. It is better to put active disk and hidden disk in ramdisk. They
+     will be merged into the parent disk on failover.
 
-3. On Primary VM's QEMU monitor, issue command:
+4. On Primary VM's QEMU monitor, issue command:
 {'execute':'qmp_capabilities'}
-{ 'execute': 'human-monitor-command',
-  'arguments': {'command-line': 'drive_add -n buddy driver=replication,mode=primary,file.driver=nbd,file.host=xx.xx.xx.xx,file.port=8889,file.export=secondary-disk0,node-name=nbd_client0'}}
-{ 'execute':'x-blockdev-change', 'arguments':{'parent': 'primary-disk0', 'node': 'nbd_client0' } }
-{ 'execute': 'migrate-set-capabilities',
-      'arguments': {'capabilities': [ {'capability': 'x-colo', 'state': true } ] } }
-{ 'execute': 'migrate', 'arguments': {'uri': 'tcp:xx.xx.xx.xx:8888' } }
+{'execute': 'human-monitor-command', 'arguments': {'command-line': 'drive_add -n buddy driver=replication,mode=primary,file.driver=nbd,file.host=127.0.0.2,file.port=9999,file.export=parent0,node-name=replication0'}}
+{'execute': 'x-blockdev-change', 'arguments':{'parent': 'colo-disk0', 'node': 'replication0' } }
+{'execute': 'migrate-set-capabilities', 'arguments': {'capabilities': [ {'capability': 'x-colo', 'state': true } ] } }
+{'execute': 'migrate', 'arguments': {'uri': 'tcp:127.0.0.2:9998' } }
 
   Note:
   a. There should be only one NBD Client for each primary disk.
-  b. xx.xx.xx.xx is the secondary physical machine's hostname or IP
-  c. The qmp command line must be run after running qmp command line in
+  b. The qmp command line must be run after running qmp command line in
      secondary qemu.
 
-4. After the above steps, you will see, whenever you make changes to PVM, SVM will be synced.
+5. After the above steps, you will see, whenever you make changes to PVM, SVM will be synced.
 You can issue command '{ "execute": "migrate-set-parameters" , "arguments":{ "x-checkpoint-delay": 2000 } }'
-to change the checkpoint period time
+to change the idle checkpoint period time
+
+6. Failover test
+You can kill one of the VMs and Failover on the surviving VM:
+
+If you killed the Secondary, then follow "Primary Failover". After that,
+if you want to resume the replication, follow "Primary resume replication"
+
+If you killed the Primary, then follow "Secondary Failover". After that,
+if you want to resume the replication, follow "Secondary resume replication"
+
+== Primary Failover ==
+The Secondary died, resume on the Primary
+
+{'execute': 'x-blockdev-change', 'arguments':{ 'parent': 'colo-disk0', 'child': 'children.1'} }
+{'execute': 'human-monitor-command', 'arguments':{ 'command-line': 'drive_del replication0' } }
+{'execute': 'object-del', 'arguments':{ 'id': 'comp0' } }
+{'execute': 'object-del', 'arguments':{ 'id': 'iothread1' } }
+{'execute': 'object-del', 'arguments':{ 'id': 'm0' } }
+{'execute': 'object-del', 'arguments':{ 'id': 'redire0' } }
+{'execute': 'object-del', 'arguments':{ 'id': 'redire1' } }
+{'execute': 'x-colo-lost-heartbeat' }
+
+== Secondary Failover ==
+The Primary died, resume on the Secondary and prepare to become the new Primary
+
+{'execute': 'nbd-server-stop'}
+{'execute': 'x-colo-lost-heartbeat'}
+
+{'execute': 'object-del', 'arguments':{ 'id': 'f2' } }
+{'execute': 'object-del', 'arguments':{ 'id': 'f1' } }
+{'execute': 'chardev-remove', 'arguments':{ 'id': 'red1' } }
+{'execute': 'chardev-remove', 'arguments':{ 'id': 'red0' } }
+
+{'execute': 'chardev-add', 'arguments':{ 'id': 'mirror0', 'backend': {'type': 'socket', 'data': {'addr': { 'type': 'inet', 'data': { 'host': '0.0.0.0', 'port': '9003' } }, 'server': true } } } }
+{'execute': 'chardev-add', 'arguments':{ 'id': 'compare1', 'backend': {'type': 'socket', 'data': {'addr': { 'type': 'inet', 'data': { 'host': '0.0.0.0', 'port': '9004' } }, 'server': true } } } }
+{'execute': 'chardev-add', 'arguments':{ 'id': 'compare0', 'backend': {'type': 'socket', 'data': {'addr': { 'type': 'inet', 'data': { 'host': '127.0.0.1', 'port': '9001' } }, 'server': true } } } }
+{'execute': 'chardev-add', 'arguments':{ 'id': 'compare0-0', 'backend': {'type': 'socket', 'data': {'addr': { 'type': 'inet', 'data': { 'host': '127.0.0.1', 'port': '9001' } }, 'server': false } } } }
+{'execute': 'chardev-add', 'arguments':{ 'id': 'compare_out', 'backend': {'type': 'socket', 'data': {'addr': { 'type': 'inet', 'data': { 'host': '127.0.0.1', 'port': '9005' } }, 'server': true } } } }
+{'execute': 'chardev-add', 'arguments':{ 'id': 'compare_out0', 'backend': {'type': 'socket', 'data': {'addr': { 'type': 'inet', 'data': { 'host': '127.0.0.1', 'port': '9005' } }, 'server': false } } } }
+
+== Primary resume replication ==
+Resume replication after new Secondary is up.
+
+Start the new Secondary (Steps 2 and 3 above), then on the Primary:
+{'execute': 'drive-mirror', 'arguments':{ 'device': 'colo-disk0', 'job-id': 'resync', 'target': 'nbd://127.0.0.2:9999/parent0', 'mode': 'existing', 'format': 'raw', 'sync': 'full'} }
+
+Wait until disk is synced, then:
+{'execute': 'stop'}
+{'execute': 'block-job-cancel', 'arguments':{ 'device': 'resync'} }
+
+{'execute': 'human-monitor-command', 'arguments':{ 'command-line': 'drive_add -n buddy driver=replication,mode=primary,file.driver=nbd,file.host=127.0.0.2,file.port=9999,file.export=parent0,node-name=replication0'}}
+{'execute': 'x-blockdev-change', 'arguments':{ 'parent': 'colo-disk0', 'node': 'replication0' } }
+
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'filter-mirror', 'id': 'm0', 'props': { 'netdev': 'hn0', 'queue': 'tx', 'outdev': 'mirror0' } } }
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'filter-redirector', 'id': 'redire0', 'props': { 'netdev': 'hn0', 'queue': 'rx', 'indev': 'compare_out' } } }
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'filter-redirector', 'id': 'redire1', 'props': { 'netdev': 'hn0', 'queue': 'rx', 'outdev': 'compare0' } } }
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'iothread', 'id': 'iothread1' } }
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'colo-compare', 'id': 'comp0', 'props': { 'primary_in': 'compare0-0', 'secondary_in': 'compare1', 'outdev': 'compare_out0', 'iothread': 'iothread1' } } }
+
+{'execute': 'migrate-set-capabilities', 'arguments':{ 'capabilities': [ {'capability': 'x-colo', 'state': true } ] } }
+{'execute': 'migrate', 'arguments':{ 'uri': 'tcp:127.0.0.2:9998' } }
+
+Note:
+If this Primary previously was a Secondary, then we need to insert the
+filters before the filter-rewriter by using the
+"'insert': 'before', 'position': 'id=rew0'" Options. See below.
+
+== Secondary resume replication ==
+Become Primary and resume replication after new Secondary is up. Note
+that now 127.0.0.1 is the Secondary and 127.0.0.2 is the Primary.
+
+Start the new Secondary (Steps 2 and 3 above, but with primary_ip=127.0.0.2),
+then on the old Secondary:
+{'execute': 'drive-mirror', 'arguments':{ 'device': 'colo-disk0', 'job-id': 'resync', 'target': 'nbd://127.0.0.1:9999/parent0', 'mode': 'existing', 'format': 'raw', 'sync': 'full'} }
+
+Wait until disk is synced, then:
+{'execute': 'stop'}
+{'execute': 'block-job-cancel', 'arguments':{ 'device': 'resync' } }
 
-5. Failover test
-You can kill Primary VM and run 'x_colo_lost_heartbeat' in Secondary VM's
-monitor at the same time, then SVM will failover and client will not detect this
-change.
+{'execute': 'human-monitor-command', 'arguments':{ 'command-line': 'drive_add -n buddy driver=replication,mode=primary,file.driver=nbd,file.host=127.0.0.1,file.port=9999,file.export=parent0,node-name=replication0'}}
+{'execute': 'x-blockdev-change', 'arguments':{ 'parent': 'colo-disk0', 'node': 'replication0' } }
 
-Before issuing '{ "execute": "x-colo-lost-heartbeat" }' command, we have to
-issue block related command to stop block replication.
-Primary:
-  Remove the nbd child from the quorum:
-  { 'execute': 'x-blockdev-change', 'arguments': {'parent': 'colo-disk0', 'child': 'children.1'}}
-  { 'execute': 'human-monitor-command','arguments': {'command-line': 'drive_del blk-buddy0'}}
-  Note: there is no qmp command to remove the blockdev now
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'filter-mirror', 'id': 'm0', 'props': { 'insert': 'before', 'position': 'id=rew0', 'netdev': 'hn0', 'queue': 'tx', 'outdev': 'mirror0' } } }
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'filter-redirector', 'id': 'redire0', 'props': { 'insert': 'before', 'position': 'id=rew0', 'netdev': 'hn0', 'queue': 'rx', 'indev': 'compare_out' } } }
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'filter-redirector', 'id': 'redire1', 'props': { 'insert': 'before', 'position': 'id=rew0', 'netdev': 'hn0', 'queue': 'rx', 'outdev': 'compare0' } } }
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'iothread', 'id': 'iothread1' } }
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'colo-compare', 'id': 'comp0', 'props': { 'primary_in': 'compare0-0', 'secondary_in': 'compare1', 'outdev': 'compare_out0', 'iothread': 'iothread1' } } }
 
-Secondary:
-  The primary host is down, so we should do the following thing:
-  { 'execute': 'nbd-server-stop' }
+{'execute': 'migrate-set-capabilities', 'arguments':{ 'capabilities': [ {'capability': 'x-colo', 'state': true } ] } }
+{'execute': 'migrate', 'arguments':{ 'uri': 'tcp:127.0.0.1:9998' } }
 
 == TODO ==
-1. Support continuous VM replication.
-2. Support shared storage.
-3. Develop the heartbeat part.
-4. Reduce checkpoint VM’s downtime while doing checkpoint.
+1. Support shared storage.
+2. Develop the heartbeat part.
+3. Reduce checkpoint VM’s downtime while doing checkpoint.
docs/block-replication.txt +18 -10
index 6bde6737fb..108e9166a8 100644
@@ -65,12 +65,12 @@ blocks that are already in QEMU.
              ^            ||                            .----------
              |            ||                            | Secondary
         1 Quorum          ||                            '----------
-         /      \         ||
-        /        \        ||
-   Primary    2 filter
-     disk         ^                                                             virtio-blk
-                  |                                                                  ^
-                3 NBD  ------->  3 NBD                                               |
+         /      \         ||                                                           virtio-blk
+        /        \        ||                                                               ^
+   Primary    2 filter                                                                     |
+     disk         ^                                                                   7 Quorum
+                  |                                                                    /
+                3 NBD  ------->  3 NBD                                                /
                 client    ||     server                                          2 filter
                           ||        ^                                                ^
 --------.                 ||        |                                                |
@@ -106,6 +106,10 @@ any state that would otherwise be lost by the speculative write-through
 of the NBD server into the secondary disk. So before block replication,
 the primary disk and secondary disk should contain the same data.
 
+7) The secondary also has a quorum node, so after secondary failover it
+can become the new primary and continue replication.
+
+
 == Failure Handling ==
 There are 7 internal errors when block replication is running:
 1. I/O error on primary disk
@@ -171,16 +175,18 @@ Primary:
      leading whitespace.
   5. The qmp command line must be run after running qmp command line in
      secondary qemu.
-  6. After failover we need remove children.1 (replication driver).
+  6. After primary failover we need remove children.1 (replication driver).
 
 Secondary:
   -drive if=none,driver=raw,file.filename=1.raw,id=colo1 \
-  -drive if=xxx,id=topxxx,driver=replication,mode=secondary,top-id=topxxx\
+  -drive if=none,id=childs1,driver=replication,mode=secondary,top-id=childs1
          file.file.filename=active_disk.qcow2,\
          file.driver=qcow2,\
          file.backing.file.filename=hidden_disk.qcow2,\
          file.backing.driver=qcow2,\
          file.backing.backing=colo1
+  -drive if=xxx,driver=quorum,read-pattern=fifo,id=top-disk1,\
+         vote-threshold=1,children.0=childs1
 
   Then run qmp command in secondary qemu:
     { 'execute': 'nbd-server-start',
@@ -234,6 +240,8 @@ Secondary:
   The primary host is down, so we should do the following thing:
   { 'execute': 'nbd-server-stop' }
 
+Promote Secondary to Primary:
+  see COLO-FT.txt
+
 TODO:
-1. Continuous block replication
-2. Shared disk
+1. Shared disk
Signature · 1 line
2.20.1
Reviewed-by on v7 4/4
#6 lore raw

RE: [PATCH v7 4/4] colo: Update Documentation for continuous replication

Quoted 13 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Thursday, October 24, 2019 10:26 PM
To: qemu-devel <[email protected]>
Cc: Zhang, Chen <[email protected]>; Jason Wang <[email protected]>; Wen Congyang <[email protected]>; Wen Congyang <[email protected]>; Xie Changlong <[email protected]>; Kevin Wolf <[email protected]>; Max Reitz <[email protected]>; qemu-block <[email protected]>
Subject: [PATCH v7 4/4] colo: Update Documentation for continuous replication

Quoted 3 lines

Document the qemu command-line and qmp commands for continuous replication

Looks good for me, thanks for your hard working.

Thanks
Zhang Chen

Quoted 420 lines
---
 docs/COLO-FT.txt           | 224 +++++++++++++++++++++++++++----------
 docs/block-replication.txt |  28 +++--
 2 files changed, 184 insertions(+), 68 deletions(-)
docs/COLO-FT.txt index renamed from docs/COLO-FT.txt +0 -0

ad24680d13..c8e1740935 100644

docs/COLO-FT.txt +0 -0
@@ -145,81 +145,189 @@ The diagram just shows the main qmp command,

you can get the detail in test procedure.

== Test procedure ==
-1. Startup qemu

docs/COLO-FT.txt +0 -2
-Primary:
-# qemu-system-x86_64 -accel kvm -m 2048 -smp 2 -qmp stdio -name

primary \

docs/COLO-FT.txt +0 -4
-  -device piix3-usb-uhci -vnc :7 \
-  -device usb-tablet -netdev tap,id=hn0,vhost=off \
-  -device virtio-net-pci,id=net-pci0,netdev=hn0 \
-  -drive if=virtio,id=primary-disk0,driver=quorum,read-pattern=fifo,vote-

threshold=1,\

docs/COLO-FT.txt +0 -4
-         children.0.file.filename=1.raw,\
-         children.0.driver=raw -S
-Secondary:
-# qemu-system-x86_64 -accel kvm -m 2048 -smp 2 -qmp stdio -name

secondary \

docs/COLO-FT.txt +0 -4
-  -device piix3-usb-uhci -vnc :7 \
-  -device usb-tablet -netdev tap,id=hn0,vhost=off \
-  -device virtio-net-pci,id=net-pci0,netdev=hn0 \
-  -drive if=none,id=secondary-disk0,file.filename=1.raw,driver=raw,node-

name=node0 \

docs/COLO-FT.txt +14 -9
-  -drive if=virtio,id=active-disk0,driver=replication,mode=secondary,\
-         file.driver=qcow2,top-id=active-disk0,\
-         file.file.filename=/mnt/ramfs/active_disk.img,\
-         file.backing.driver=qcow2,\
-         file.backing.file.filename=/mnt/ramfs/hidden_disk.img,\
-         file.backing.backing=secondary-disk0 \
-  -incoming tcp:0:8888
-
-2. On Secondary VM's QEMU monitor, issue command
+Note: Here we are running both instances on the same host for testing,
+change the IP Addresses if you want to run it on two hosts. Initally
+127.0.0.1 is the Primary Host and 127.0.0.2 is the Secondary Host.
+
+== Startup qemu ==
+1. Primary:
+Note: Initally, $imagefolder/primary.qcow2 needs to be copied to all hosts.
+You don't need to change any IP's here, because 0.0.0.0 listens on any
+interface. The chardev's with 127.0.0.1 IP's loopback to the local qemu
+instance.
+
+# imagefolder="/mnt/vms/colo-test-primary"
+
+# qemu-system-x86_64 -enable-kvm -cpu qemu64,+kvmclock -m 512 -smp

1 -qmp stdio \

docs/COLO-FT.txt +2 -0
+   -device piix3-usb-uhci -device usb-tablet -name primary \
+   -netdev tap,id=hn0,vhost=off,helper=/usr/lib/qemu/qemu-bridge-helper

\

docs/COLO-FT.txt +6 -0
+   -device rtl8139,id=e0,netdev=hn0 \
+   -chardev socket,id=mirror0,host=0.0.0.0,port=9003,server,nowait \
+   -chardev socket,id=compare1,host=0.0.0.0,port=9004,server,wait \
+   -chardev socket,id=compare0,host=127.0.0.1,port=9001,server,nowait \
+   -chardev socket,id=compare0-0,host=127.0.0.1,port=9001 \
+   -chardev socket,id=compare_out,host=127.0.0.1,port=9005,server,nowait

\

docs/COLO-FT.txt +3 -0
+   -chardev socket,id=compare_out0,host=127.0.0.1,port=9005 \
+   -object filter-mirror,id=m0,netdev=hn0,queue=tx,outdev=mirror0 \
+   -object filter-

redirector,netdev=hn0,id=redire0,queue=rx,indev=compare_out \
+ -object filter-
redirector,netdev=hn0,id=redire1,queue=rx,outdev=compare0 \

docs/COLO-FT.txt +3 -0
+   -object iothread,id=iothread1 \
+   -object
+colo-compare,id=comp0,primary_in=compare0-

0,secondary_in=compare1,\

docs/COLO-FT.txt +19 -0
+outdev=compare_out0,iothread=iothread1 \
+   -drive
+if=ide,id=colo-disk0,driver=quorum,read-pattern=fifo,vote-threshold=1,\
+children.0.file.filename=$imagefolder/primary.qcow2,children.0.driver=q
+cow2 -S
+
+2. Secondary:
+Note: Active and hidden images need to be created only once and the
+size should be the same as primary.qcow2. Again, you don't need to
+change any IP's here, except for the $primary_ip variable.
+
+# imagefolder="/mnt/vms/colo-test-secondary"
+# primary_ip=127.0.0.1
+
+# qemu-img create -f qcow2 $imagefolder/secondary-active.qcow2 10G
+
+# qemu-img create -f qcow2 $imagefolder/secondary-hidden.qcow2 10G
+
+# qemu-system-x86_64 -enable-kvm -cpu qemu64,+kvmclock -m 512 -smp

1 -qmp stdio \

docs/COLO-FT.txt +2 -0
+   -device piix3-usb-uhci -device usb-tablet -name secondary \
+   -netdev tap,id=hn0,vhost=off,helper=/usr/lib/qemu/qemu-bridge-helper

\

docs/COLO-FT.txt +7 -0
+   -device rtl8139,id=e0,netdev=hn0 \
+   -chardev socket,id=red0,host=$primary_ip,port=9003,reconnect=1 \
+   -chardev socket,id=red1,host=$primary_ip,port=9004,reconnect=1 \
+   -object filter-redirector,id=f1,netdev=hn0,queue=tx,indev=red0 \
+   -object filter-redirector,id=f2,netdev=hn0,queue=rx,outdev=red1 \
+   -object filter-rewriter,id=rew0,netdev=hn0,queue=all \
+   -drive

if=none,id=parent0,file.filename=$imagefolder/primary.qcow2,driver=qcow 2 \

docs/COLO-FT.txt +15 -2
+   -drive
+if=none,id=childs0,driver=replication,mode=secondary,file.driver=qcow2,
+\
+top-id=colo-disk0,file.file.filename=$imagefolder/secondary-active.qcow
+2,\
+file.backing.driver=qcow2,file.backing.file.filename=$imagefolder/secon
+dary-hidden.qcow2,\
+file.backing.backing=parent0 \
+   -drive
+if=ide,id=colo-disk0,driver=quorum,read-pattern=fifo,vote-threshold=1,\
+children.0=childs0 \
+   -incoming tcp:0.0.0.0:9998
+
+
+3. On Secondary VM's QEMU monitor, issue command
 {'execute':'qmp_capabilities'}
-{ 'execute': 'nbd-server-start',
-  'arguments': {'addr': {'type': 'inet', 'data': {'host': 'xx.xx.xx.xx', 'port':

'8889'} } } -}
-{'execute': 'nbd-server-add', 'arguments': {'device': 'secondary-disk0', 'writable': true } }

docs/COLO-FT.txt +7 -4
+{'execute': 'nbd-server-start', 'arguments': {'addr': {'type': 'inet',
+'data': {'host': '0.0.0.0', 'port': '9999'} } } }
+{'execute': 'nbd-server-add', 'arguments': {'device': 'parent0',
+'writable': true } }
 
 Note:
   a. The qmp command nbd-server-start and nbd-server-add must be run
      before running the qmp command migrate on primary QEMU
   b. Active disk, hidden disk and nbd target's length should be the
      same.
-  c. It is better to put active disk and hidden disk in ramdisk.
+  c. It is better to put active disk and hidden disk in ramdisk. They
+     will be merged into the parent disk on failover.
 
-3. On Primary VM's QEMU monitor, issue command:
+4. On Primary VM's QEMU monitor, issue command:
 {'execute':'qmp_capabilities'}
-{ 'execute': 'human-monitor-command',
-  'arguments': {'command-line': 'drive_add -n buddy

driver=replication,mode=primary,file.driver=nbd,file.host=xx.xx.xx.xx,file.p ort=8889,file.export=secondary-disk0,node-name=nbd_client0'}}
-{ 'execute':'x-blockdev-change', 'arguments':{'parent': 'primary-disk0', 'node': 'nbd_client0' } } -{ 'execute': 'migrate-set-capabilities',

docs/COLO-FT.txt +10 -5
-      'arguments': {'capabilities': [ {'capability': 'x-colo', 'state': true } ] } }
-{ 'execute': 'migrate', 'arguments': {'uri': 'tcp:xx.xx.xx.xx:8888' } }
+{'execute': 'human-monitor-command', 'arguments': {'command-line':
+'drive_add -n buddy
+driver=replication,mode=primary,file.driver=nbd,file.host=127.0.0.2,fil
+e.port=9999,file.export=parent0,node-name=replication0'}}
+{'execute': 'x-blockdev-change', 'arguments':{'parent': 'colo-disk0',
+'node': 'replication0' } }
+{'execute': 'migrate-set-capabilities', 'arguments': {'capabilities': [
+{'capability': 'x-colo', 'state': true } ] } }
+{'execute': 'migrate', 'arguments': {'uri': 'tcp:127.0.0.2:9998' } }
 
   Note:
   a. There should be only one NBD Client for each primary disk.
-  b. xx.xx.xx.xx is the secondary physical machine's hostname or IP
-  c. The qmp command line must be run after running qmp command line in
+  b. The qmp command line must be run after running qmp command line in
      secondary qemu.
 
-4. After the above steps, you will see, whenever you make changes to PVM,

SVM will be synced.
+5. After the above steps, you will see, whenever you make changes to PVM, SVM will be synced.
You can issue command '{ "execute": "migrate-set-parameters" ,
"arguments":{ "x-checkpoint-delay": 2000 } }'

docs/COLO-FT.txt +7 -1
-to change the checkpoint period time
+to change the idle checkpoint period time
+
+6. Failover test
+You can kill one of the VMs and Failover on the surviving VM:
+
+If you killed the Secondary, then follow "Primary Failover". After
+that, if you want to resume the replication, follow "Primary resume

replication"

docs/COLO-FT.txt +3 -0
+
+If you killed the Primary, then follow "Secondary Failover". After
+that, if you want to resume the replication, follow "Secondary resume

replication"

docs/COLO-FT.txt +17 -0
+
+== Primary Failover ==
+The Secondary died, resume on the Primary
+
+{'execute': 'x-blockdev-change', 'arguments':{ 'parent': 'colo-disk0',
+'child': 'children.1'} }
+{'execute': 'human-monitor-command', 'arguments':{ 'command-line':
+'drive_del replication0' } }
+{'execute': 'object-del', 'arguments':{ 'id': 'comp0' } }
+{'execute': 'object-del', 'arguments':{ 'id': 'iothread1' } }
+{'execute': 'object-del', 'arguments':{ 'id': 'm0' } }
+{'execute': 'object-del', 'arguments':{ 'id': 'redire0' } }
+{'execute': 'object-del', 'arguments':{ 'id': 'redire1' } }
+{'execute': 'x-colo-lost-heartbeat' }
+
+== Secondary Failover ==
+The Primary died, resume on the Secondary and prepare to become the

new

docs/COLO-FT.txt +87 -2
+Primary
+
+{'execute': 'nbd-server-stop'}
+{'execute': 'x-colo-lost-heartbeat'}
+
+{'execute': 'object-del', 'arguments':{ 'id': 'f2' } }
+{'execute': 'object-del', 'arguments':{ 'id': 'f1' } }
+{'execute': 'chardev-remove', 'arguments':{ 'id': 'red1' } }
+{'execute': 'chardev-remove', 'arguments':{ 'id': 'red0' } }
+
+{'execute': 'chardev-add', 'arguments':{ 'id': 'mirror0', 'backend':
+{'type': 'socket', 'data': {'addr': { 'type': 'inet', 'data': { 'host':
+'0.0.0.0', 'port': '9003' } }, 'server': true } } } }
+{'execute': 'chardev-add', 'arguments':{ 'id': 'compare1', 'backend':
+{'type': 'socket', 'data': {'addr': { 'type': 'inet', 'data': { 'host':
+'0.0.0.0', 'port': '9004' } }, 'server': true } } } }
+{'execute': 'chardev-add', 'arguments':{ 'id': 'compare0', 'backend':
+{'type': 'socket', 'data': {'addr': { 'type': 'inet', 'data': { 'host':
+'127.0.0.1', 'port': '9001' } }, 'server': true } } } }
+{'execute': 'chardev-add', 'arguments':{ 'id': 'compare0-0', 'backend':
+{'type': 'socket', 'data': {'addr': { 'type': 'inet', 'data': { 'host':
+'127.0.0.1', 'port': '9001' } }, 'server': false } } } }
+{'execute': 'chardev-add', 'arguments':{ 'id': 'compare_out',
+'backend': {'type': 'socket', 'data': {'addr': { 'type': 'inet',
+'data': { 'host': '127.0.0.1', 'port': '9005' } }, 'server': true } } }
+}
+{'execute': 'chardev-add', 'arguments':{ 'id': 'compare_out0',
+'backend': {'type': 'socket', 'data': {'addr': { 'type': 'inet',
+'data': { 'host': '127.0.0.1', 'port': '9005' } }, 'server': false } }
+} }
+
+== Primary resume replication ==
+Resume replication after new Secondary is up.
+
+Start the new Secondary (Steps 2 and 3 above), then on the Primary:
+{'execute': 'drive-mirror', 'arguments':{ 'device': 'colo-disk0',
+'job-id': 'resync', 'target': 'nbd://127.0.0.2:9999/parent0', 'mode':
+'existing', 'format': 'raw', 'sync': 'full'} }
+
+Wait until disk is synced, then:
+{'execute': 'stop'}
+{'execute': 'block-job-cancel', 'arguments':{ 'device': 'resync'} }
+
+{'execute': 'human-monitor-command', 'arguments':{ 'command-line':
+'drive_add -n buddy
+driver=replication,mode=primary,file.driver=nbd,file.host=127.0.0.2,fil
+e.port=9999,file.export=parent0,node-name=replication0'}}
+{'execute': 'x-blockdev-change', 'arguments':{ 'parent': 'colo-disk0',
+'node': 'replication0' } }
+
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'filter-mirror',
+'id': 'm0', 'props': { 'netdev': 'hn0', 'queue': 'tx', 'outdev':
+'mirror0' } } }
+{'execute': 'object-add', 'arguments':{ 'qom-type':
+'filter-redirector', 'id': 'redire0', 'props': { 'netdev': 'hn0',
+'queue': 'rx', 'indev': 'compare_out' } } }
+{'execute': 'object-add', 'arguments':{ 'qom-type':
+'filter-redirector', 'id': 'redire1', 'props': { 'netdev': 'hn0',
+'queue': 'rx', 'outdev': 'compare0' } } }
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'iothread', 'id':
+'iothread1' } }
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'colo-compare',
+'id': 'comp0', 'props': { 'primary_in': 'compare0-0', 'secondary_in':
+'compare1', 'outdev': 'compare_out0', 'iothread': 'iothread1' } } }
+
+{'execute': 'migrate-set-capabilities', 'arguments':{ 'capabilities': [
+{'capability': 'x-colo', 'state': true } ] } }
+{'execute': 'migrate', 'arguments':{ 'uri': 'tcp:127.0.0.2:9998' } }
+
+Note:
+If this Primary previously was a Secondary, then we need to insert the
+filters before the filter-rewriter by using the
+"'insert': 'before', 'position': 'id=rew0'" Options. See below.
+
+== Secondary resume replication ==
+Become Primary and resume replication after new Secondary is up. Note
+that now 127.0.0.1 is the Secondary and 127.0.0.2 is the Primary.
+
+Start the new Secondary (Steps 2 and 3 above, but with
+primary_ip=127.0.0.2), then on the old Secondary:
+{'execute': 'drive-mirror', 'arguments':{ 'device': 'colo-disk0',
+'job-id': 'resync', 'target': 'nbd://127.0.0.1:9999/parent0', 'mode':
+'existing', 'format': 'raw', 'sync': 'full'} }
+
+Wait until disk is synced, then:
+{'execute': 'stop'}
+{'execute': 'block-job-cancel', 'arguments':{ 'device': 'resync' } }
 
-5. Failover test
-You can kill Primary VM and run 'x_colo_lost_heartbeat' in Secondary VM's -

monitor at the same time, then SVM will failover and client will not detect this -change.

docs/COLO-FT.txt +6 -5
+{'execute': 'human-monitor-command', 'arguments':{ 'command-line':
+'drive_add -n buddy
+driver=replication,mode=primary,file.driver=nbd,file.host=127.0.0.1,fil
+e.port=9999,file.export=parent0,node-name=replication0'}}
+{'execute': 'x-blockdev-change', 'arguments':{ 'parent': 'colo-disk0',
+'node': 'replication0' } }
 
-Before issuing '{ "execute": "x-colo-lost-heartbeat" }' command, we have to
-issue block related command to stop block replication.
-Primary:
-  Remove the nbd child from the quorum:
-  { 'execute': 'x-blockdev-change', 'arguments': {'parent': 'colo-disk0', 'child':

'children.1'}}
- { 'execute': 'human-monitor-command','arguments': {'command-line':
'drive_del blk-buddy0'}}

docs/COLO-FT.txt +22 -8
-  Note: there is no qmp command to remove the blockdev now
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'filter-mirror',
+'id': 'm0', 'props': { 'insert': 'before', 'position': 'id=rew0',
+'netdev': 'hn0', 'queue': 'tx', 'outdev': 'mirror0' } } }
+{'execute': 'object-add', 'arguments':{ 'qom-type':
+'filter-redirector', 'id': 'redire0', 'props': { 'insert': 'before',
+'position': 'id=rew0', 'netdev': 'hn0', 'queue': 'rx', 'indev':
+'compare_out' } } }
+{'execute': 'object-add', 'arguments':{ 'qom-type':
+'filter-redirector', 'id': 'redire1', 'props': { 'insert': 'before',
+'position': 'id=rew0', 'netdev': 'hn0', 'queue': 'rx', 'outdev':
+'compare0' } } }
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'iothread', 'id':
+'iothread1' } }
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'colo-compare',
+'id': 'comp0', 'props': { 'primary_in': 'compare0-0', 'secondary_in':
+'compare1', 'outdev': 'compare_out0', 'iothread': 'iothread1' } } }
 
-Secondary:
-  The primary host is down, so we should do the following thing:
-  { 'execute': 'nbd-server-stop' }
+{'execute': 'migrate-set-capabilities', 'arguments':{ 'capabilities': [
+{'capability': 'x-colo', 'state': true } ] } }
+{'execute': 'migrate', 'arguments':{ 'uri': 'tcp:127.0.0.1:9998' } }
 
 == TODO ==
-1. Support continuous VM replication.
-2. Support shared storage.
-3. Develop the heartbeat part.
-4. Reduce checkpoint VM’s downtime while doing checkpoint.
+1. Support shared storage.
+2. Develop the heartbeat part.
+3. Reduce checkpoint VM’s downtime while doing checkpoint.
docs/block-replication.txt index renamed from docs/block-replication.txt +0 -0

6bde6737fb..108e9166a8 100644

docs/block-replication.txt +6 -6
@@ -65,12 +65,12 @@ blocks that are already in QEMU.
              ^            ||                            .----------
              |            ||                            | Secondary
         1 Quorum          ||                            '----------
-         /      \         ||
-        /        \        ||
-   Primary    2 filter
-     disk         ^                                                             virtio-blk
-                  |                                                                  ^
-                3 NBD  ------->  3 NBD                                               |
+         /      \         ||                                                           virtio-blk
+        /        \        ||                                                               ^
+   Primary    2 filter                                                                     |
+     disk         ^                                                                   7 Quorum
+                  |                                                                    /
+                3 NBD  ------->  3 NBD                                                /
                 client    ||     server                                          2 filter
                           ||        ^                                                ^
 --------.                 ||        |                                                |
@@ -106,6 +106,10 @@ any state that would otherwise be lost by the

speculative write-through of the NBD server into the secondary disk. So before block replication, the primary disk and secondary disk should contain the same data.

+7) The secondary also has a quorum node, so after secondary failover it

docs/block-replication.txt +12 -5
+can become the new primary and continue replication.
+
+
 == Failure Handling ==
 There are 7 internal errors when block replication is running:
 1. I/O error on primary disk
@@ -171,16 +175,18 @@ Primary:
      leading whitespace.
   5. The qmp command line must be run after running qmp command line in
      secondary qemu.
-  6. After failover we need remove children.1 (replication driver).
+  6. After primary failover we need remove children.1 (replication driver).
 
 Secondary:
   -drive if=none,driver=raw,file.filename=1.raw,id=colo1 \
-  -drive if=xxx,id=topxxx,driver=replication,mode=secondary,top-
 id=topxxx\
+  -drive
+ if=none,id=childs1,driver=replication,mode=secondary,top-id=childs1
          file.file.filename=active_disk.qcow2,\
          file.driver=qcow2,\
          file.backing.file.filename=hidden_disk.qcow2,\
          file.backing.driver=qcow2,\
          file.backing.backing=colo1
+  -drive if=xxx,driver=quorum,read-pattern=fifo,id=top-disk1,\
+         vote-threshold=1,children.0=childs1
 
   Then run qmp command in secondary qemu:
     { 'execute': 'nbd-server-start',
@@ -234,6 +240,8 @@ Secondary:
   The primary host is down, so we should do the following thing:
   { 'execute': 'nbd-server-stop' }
 
+Promote Secondary to Primary:
+  see COLO-FT.txt
+
 TODO:
-1. Continuous block replication
-2. Shared disk
+1. Shared disk
--

2.20.1

#7 lore raw
In reply to Lukas Straub
Quoted 52 lines · On Fri, 25 Oct 2019 19:06:31 +0200 Lukas Straub <[email protected]> wrote:

Hello Everyone,
These Patches add support for continuous replication to colo. This means that after the Primary fails and the Secondary did a failover, the Secondary can then become Primary and resume replication to a new Secondary.

Regards,
Lukas Straub

v7:
- clarify meaning of ip's in documentation and note that active and hidden images just need to be created once
- reverted removal of bdrv_is_root_node(top_bs) in replication and adjusted the top-id= parameter in documentation acordingly

v6:
- documented the position= and insert= options
- renamed replication test
- clarified documentation by using different ip's for primary and secondary
- added Reviewed-by tags

v5:
- change syntax for the position= parameter
- fix spelling mistake

v4:
- fix checkpatch.pl warnings

v3:
- add test for replication changes
- check if the filter to be inserted before/behind belongs to the same interface
- fix the error message for the position= parameter
- rename term "after" -> "behind" and variable "insert_before" -> "insert_before_flag"
- document the quorum node on the secondary side
- simplify quorum parameters in documentation
- remove trailing spaces in documentation
- clarify the testing procedure in documentation

v2:
- fix email formating
- fix checkpatch.pl warnings
- fix patchew error
- clearer commit messages

Lukas Straub (4):
block/replication.c: Ignore requests after failover tests/test-replication.c: Add test for for secondary node continuing
replication
net/filter.c: Add Options to insert filters anywhere in the filter
list
colo: Update Documentation for continuous replication

Quoted 9 lines
 block/replication.c        |  35 +++++-
 docs/COLO-FT.txt           | 224 +++++++++++++++++++++++++++----------
 docs/block-replication.txt |  28 +++--
 include/net/filter.h       |   2 +
 net/filter.c               |  92 ++++++++++++++-
 qemu-options.hx            |  31 ++++-
 tests/test-replication.c   |  52 +++++++++
 7 files changed, 389 insertions(+), 75 deletions(-)

Hello Everyone,
So I guess this is ready for merging or will that have to wait until the 4.2 release is done?

Regards,
Lukas Straub

#8 lore raw
Quoted 83 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Thursday, November 14, 2019 12:36 AM
To: qemu-devel <[email protected]>
Cc: Zhang, Chen <[email protected]>; Jason Wang <[email protected]>; Wen Congyang <[email protected]>; Xie Changlong <[email protected]>; Kevin Wolf <[email protected]>; Max Reitz <[email protected]>; qemu-block <[email protected]>
Subject: Re: [PATCH v7 0/4] colo: Add support for continuous replication

Quoted 4 lines · On Fri, 25 Oct 2019 19:06:31 +0200 Lukas Straub <[email protected]> wrote:

Hello Everyone,
These Patches add support for continuous replication to colo. This means that after the Primary fails and the Secondary did a failover, the Secondary can then become Primary and resume replication to a new

Secondary.

Quoted 8 lines

Regards,
Lukas Straub

v7:
- clarify meaning of ip's in documentation and note that active and hidden images just need to be created once
- reverted removal of bdrv_is_root_node(top_bs) in replication and

adjusted

Quoted 16 lines

the top-id= parameter in documentation acordingly

v6:
- documented the position= and insert= options
- renamed replication test
- clarified documentation by using different ip's for primary and
secondary
- added Reviewed-by tags

v5:
- change syntax for the position= parameter
- fix spelling mistake

v4:
- fix checkpatch.pl warnings

Quoted 6 lines

v3:
- add test for replication changes
- check if the filter to be inserted before/behind belongs to the
same interface
- fix the error message for the position= parameter
- rename term "after" -> "behind" and variable "insert_before" ->

"insert_before_flag"

Quoted 20 lines

- document the quorum node on the secondary side
- simplify quorum parameters in documentation
- remove trailing spaces in documentation
- clarify the testing procedure in documentation

v2:
- fix email formating
- fix checkpatch.pl warnings
- fix patchew error
- clearer commit messages

Lukas Straub (4):
block/replication.c: Ignore requests after failover tests/test-replication.c: Add test for for secondary node continuing
replication
net/filter.c: Add Options to insert filters anywhere in the filter
list
colo: Update Documentation for continuous replication

Quoted 9 lines
 block/replication.c        |  35 +++++-
 docs/COLO-FT.txt           | 224 +++++++++++++++++++++++++++----------
 docs/block-replication.txt |  28 +++--
 include/net/filter.h       |   2 +
 net/filter.c               |  92 ++++++++++++++-
 qemu-options.hx            |  31 ++++-
 tests/test-replication.c   |  52 +++++++++
 7 files changed, 389 insertions(+), 75 deletions(-)
Quoted 3 lines

Hello Everyone,
So I guess this is ready for merging or will that have to wait until the 4.2 release is done?

Due to Qemu 4.2 release schedule, after soft feature freeze(Oct29) the master branch does not accept feature changes. But I don't know if sub-maintainer(block or net) can queue this series first then merge it after 4.2 release?

Thanks
Zhang Chen

Quoted 3 lines

Regards,
Lukas Straub

#9 lore raw
Quoted 80 lines · Chen
Quoted 80 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Thursday, November 14, 2019 12:36 AM
To: qemu-devel <[email protected]>
Cc: Zhang, Chen <[email protected]>; Jason Wang <[email protected]>; Wen Congyang <[email protected]>; Xie Changlong <[email protected]>; Kevin Wolf <[email protected]>; Max Reitz <[email protected]>; qemu-block <[email protected]>
Subject: Re: [PATCH v7 0/4] colo: Add support for continuous replication

Quoted 4 lines · On Fri, 25 Oct 2019 19:06:31 +0200 Lukas Straub <[email protected]> wrote:

Hello Everyone,
These Patches add support for continuous replication to colo. This means that after the Primary fails and the Secondary did a failover, the Secondary can then become Primary and resume replication to a new

Secondary.

Quoted 7 lines

Regards,
Lukas Straub

v7:
- clarify meaning of ip's in documentation and note that active and hidden images just need to be created once
- reverted removal of bdrv_is_root_node(top_bs) in replication and

adjusted

Quoted 16 lines

the top-id= parameter in documentation acordingly

v6:
- documented the position= and insert= options
- renamed replication test
- clarified documentation by using different ip's for primary and
secondary
- added Reviewed-by tags

v5:
- change syntax for the position= parameter
- fix spelling mistake

v4:
- fix checkpatch.pl warnings

Quoted 6 lines

v3:
- add test for replication changes
- check if the filter to be inserted before/behind belongs to the
same interface
- fix the error message for the position= parameter
- rename term "after" -> "behind" and variable "insert_before" ->

"insert_before_flag"

Quoted 29 lines

- document the quorum node on the secondary side
- simplify quorum parameters in documentation
- remove trailing spaces in documentation
- clarify the testing procedure in documentation

v2:
- fix email formating
- fix checkpatch.pl warnings
- fix patchew error
- clearer commit messages

Lukas Straub (4):
block/replication.c: Ignore requests after failover tests/test-replication.c: Add test for for secondary node continuing
replication
net/filter.c: Add Options to insert filters anywhere in the filter
list
colo: Update Documentation for continuous replication

  block/replication.c        |  35 +++++-
  docs/COLO-FT.txt           | 224 +++++++++++++++++++++++++++----------
  docs/block-replication.txt |  28 +++--
  include/net/filter.h       |   2 +
  net/filter.c               |  92 ++++++++++++++-
  qemu-options.hx            |  31 ++++-
  tests/test-replication.c   |  52 +++++++++
  7 files changed, 389 insertions(+), 75 deletions(-)
Quoted 8 lines
Quoted 3 lines

Hello Everyone,
So I guess this is ready for merging or will that have to wait until the 4.2 release is done?

Due to Qemu 4.2 release schedule, after soft feature freeze(Oct29) the master branch does not accept feature changes. But I don't know if sub-maintainer(block or net) can queue this series first then merge it after 4.2 release?

Thanks
Zhang Chen

Will try to queue this series.

Thanks

Quoted 3 lines
Quoted 3 lines

Regards,
Lukas Straub

#10 lore raw
Quoted 104 lines

-----Original Message-----
From: Jason Wang <[email protected]>
Sent: Tuesday, November 19, 2019 11:03 PM
To: Zhang, Chen <[email protected]>; Lukas Straub <[email protected]>; qemu-devel <[email protected]>
Cc: Kevin Wolf <[email protected]>; qemu-block <qemu- [email protected]>; Wen Congyang <[email protected]>; Max Reitz <[email protected]>; Xie Changlong <[email protected]>
Subject: Re: [PATCH v7 0/4] colo: Add support for continuous replication

Quoted 7 lines · Chen
Quoted 7 lines

-----Original Message-----
From: Lukas Straub <[email protected]>
Sent: Thursday, November 14, 2019 12:36 AM
To: qemu-devel <[email protected]>
Cc: Zhang, Chen <[email protected]>; Jason Wang <[email protected]>; Wen Congyang

<[email protected]>; Xie

Quoted 1 line
Quoted 1 line

Changlong <[email protected]>; Kevin Wolf

<[email protected]>;

Quoted 12 lines
Quoted 12 lines

Max Reitz <[email protected]>; qemu-block <[email protected]>
Subject: Re: [PATCH v7 0/4] colo: Add support for continuous replication

Quoted 5 lines · On Fri, 25 Oct 2019 19:06:31 +0200 Lukas Straub <[email protected]> wrote:

Hello Everyone,
These Patches add support for continuous replication to colo. This means that after the Primary fails and the Secondary did a failover, the Secondary can then become Primary and resume replication to a new

Quoted 6 lines
Quoted 6 lines

Secondary.

Quoted 5 lines

Regards,
Lukas Straub

v7:
- clarify meaning of ip's in documentation and note that active and

hidden

Quoted 46 lines
Quoted 46 lines
Quoted 2 lines

images just need to be created once
- reverted removal of bdrv_is_root_node(top_bs) in replication and

adjusted

Quoted 16 lines

the top-id= parameter in documentation acordingly

v6:
- documented the position= and insert= options
- renamed replication test
- clarified documentation by using different ip's for primary and
secondary
- added Reviewed-by tags

v5:
- change syntax for the position= parameter
- fix spelling mistake

v4:
- fix checkpatch.pl warnings

Quoted 6 lines

v3:
- add test for replication changes
- check if the filter to be inserted before/behind belongs to the
same interface
- fix the error message for the position= parameter
- rename term "after" -> "behind" and variable "insert_before" ->

"insert_before_flag"

Quoted 20 lines

- document the quorum node on the secondary side
- simplify quorum parameters in documentation
- remove trailing spaces in documentation
- clarify the testing procedure in documentation

v2:
- fix email formating
- fix checkpatch.pl warnings
- fix patchew error
- clearer commit messages

Lukas Straub (4):
block/replication.c: Ignore requests after failover tests/test-replication.c: Add test for for secondary node continuing
replication
net/filter.c: Add Options to insert filters anywhere in the filter
list
colo: Update Documentation for continuous replication

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
  block/replication.c        |  35 +++++-
  docs/COLO-FT.txt           | 224 +++++++++++++++++++++++++++---------

-

Quoted 11 lines
Quoted 10 lines
Quoted 7 lines
  docs/block-replication.txt |  28 +++--
  include/net/filter.h       |   2 +
  net/filter.c               |  92 ++++++++++++++-
  qemu-options.hx            |  31 ++++-
  tests/test-replication.c   |  52 +++++++++
  7 files changed, 389 insertions(+), 75 deletions(-)

Hello Everyone,
So I guess this is ready for merging or will that have to wait until the 4.2 release is done?

Due to Qemu 4.2 release schedule, after soft feature freeze(Oct29) the

master branch does not accept feature changes.

Quoted 8 lines
Quoted 1 line

But I don't know if sub-maintainer(block or net) can queue this series first

then merge it after 4.2 release?

Quoted 3 lines

Thanks
Zhang Chen

Will try to queue this series.

Thank you Jason~

Thanks
Zhang Chen

Quoted 7 lines

Thanks

Quoted 3 lines
Quoted 3 lines

Regards,
Lukas Straub

#11 lore raw

Hi Jason,

I noticed this series can't be merged or queued, do you met some problem about it?

Thanks

Zhang Chen

Quoted 96 lines
Quoted 78 lines
Quoted 12 lines
Quoted 12 lines

Max Reitz <[email protected]>; qemu-block <[email protected]>
Subject: Re: [PATCH v7 0/4] colo: Add support for continuous replication

Quoted 5 lines · On Fri, 25 Oct 2019 19:06:31 +0200 Lukas Straub <[email protected]> wrote:

Hello Everyone,
These Patches add support for continuous replication to colo. This means that after the Primary fails and the Secondary did a failover, the Secondary can then become Primary and resume replication to a new

Quoted 6 lines
Quoted 6 lines

Secondary.

Quoted 5 lines

Regards,
Lukas Straub

v7:
- clarify meaning of ip's in documentation and note that active and

hidden

Quoted 46 lines
Quoted 46 lines
Quoted 2 lines

images just need to be created once
- reverted removal of bdrv_is_root_node(top_bs) in replication and

adjusted

Quoted 16 lines

the top-id= parameter in documentation acordingly

v6:
- documented the position= and insert= options
- renamed replication test
- clarified documentation by using different ip's for primary and
secondary
- added Reviewed-by tags

v5:
- change syntax for the position= parameter
- fix spelling mistake

v4:
- fix checkpatch.pl warnings

Quoted 6 lines

v3:
- add test for replication changes
- check if the filter to be inserted before/behind belongs to the
same interface
- fix the error message for the position= parameter
- rename term "after" -> "behind" and variable "insert_before" ->

"insert_before_flag"

Quoted 20 lines

- document the quorum node on the secondary side
- simplify quorum parameters in documentation
- remove trailing spaces in documentation
- clarify the testing procedure in documentation

v2:
- fix email formating
- fix checkpatch.pl warnings
- fix patchew error
- clearer commit messages

Lukas Straub (4):
block/replication.c: Ignore requests after failover tests/test-replication.c: Add test for for secondary node continuing
replication
net/filter.c: Add Options to insert filters anywhere in the filter
list
colo: Update Documentation for continuous replication

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
   block/replication.c        |  35 +++++-
   docs/COLO-FT.txt           | 224 +++++++++++++++++++++++++++---------

-

Quoted 10 lines
Quoted 10 lines
Quoted 7 lines
   docs/block-replication.txt |  28 +++--
   include/net/filter.h       |   2 +
   net/filter.c               |  92 ++++++++++++++-
   qemu-options.hx            |  31 ++++-
   tests/test-replication.c   |  52 +++++++++
   7 files changed, 389 insertions(+), 75 deletions(-)

Hello Everyone,
So I guess this is ready for merging or will that have to wait until the 4.2 release is done?

Quoted 8 lines
Quoted 1 line

Due to Qemu 4.2 release schedule, after soft feature freeze(Oct29) the

master branch does not accept feature changes.

Quoted 1 line

But I don't know if sub-maintainer(block or net) can queue this series first

then merge it after 4.2 release?

Quoted 2 lines

Thanks
Zhang Chen

Will try to queue this series.

Thank you Jason~

Thanks
Zhang Chen

Quoted 5 lines

Thanks

Quoted 2 lines
Quoted 2 lines

Regards,
Lukas Straub

#12 lore raw
Quoted 8 lines · Chen

Hi Jason,

I noticed this series can't be merged or queued, do you met some problem about it?

Thanks

Zhang Chen

Not, I've queued this.

Thanks