Skip to contentSkip to conversation

colo: Introduce resource agent and high-level test

PATCH In review Lukas Straub 4 participants 13 messages Nov 21, 2019 – Dec 18, 2019 Block layer

lore
Share & tools
lore
#0 lore raw

cover colo: Introduce resource agent and high-level test

Hello Everyone,
These patches introduce a resource agent for use with the Pacemaker CRM and a high-level test utilizing it for testing qemu COLO.

The resource agent manages qemu COLO including continuous replication.

Currently the second test case (where the peer qemu is frozen) fails on primary failover, because qemu hangs while removing the replication related block nodes. Note that this also happens in real world test when cutting power to the peer host, so this needs to be fixed.

Based-on: <[email protected]> ([PATCH v7 0/4] colo: Add support for continuous replication)

Lukas Straub (4):
block/quorum.c: stable children names
colo: Introduce resource agent
colo: Introduce high-level test
MAINTAINERS: Add myself as maintainer for COLO resource agent

 MAINTAINERS                            |    6 +
 block/quorum.c                         |    6 +
 scripts/colo-resource-agent/colo       | 1026 ++++++++++++++++++++++++
 scripts/colo-resource-agent/crm_master |   44 +
 tests/acceptance/colo.py               |  444 ++++++++++
 5 files changed, 1526 insertions(+)
 create mode 100755 scripts/colo-resource-agent/colo
 create mode 100755 scripts/colo-resource-agent/crm_master
 create mode 100644 tests/acceptance/colo.py

--
2.20.1

#1 lore raw

PATCH 1/4 block/quorum.c: stable children names

If we remove the child with the highest index from the quorum, decrement s->next_child_index. This way we get stable children names as long as we only remove the last child.

---
 block/quorum.c | 6 ++++++
 1 file changed, 6 insertions(+)
block/quorum.c +6 -0
index df68adcfaa..6100d4108a 100644
@@ -1054,6 +1054,12 @@ static void quorum_del_child(BlockDriverState *bs, BdrvChild *child,
     /* We know now that num_children > threshold, so blkverify must be false */
     assert(!s->is_blkverify);
 
+    unsigned child_id;
+    sscanf(child->name, "children.%u", &child_id);
+    if (child_id == s->next_child_index - 1) {
+        s->next_child_index--;
+    }
+
     bdrv_drained_begin(bs);
 
     /* We can safely remove this child now */

--
2.20.1

#2 lore raw

Re: [PATCH 1/4] block/quorum.c: stable children names

Quoted 14 lines · Lukas Straub

If we remove the child with the highest index from the quorum, decrement s->next_child_index. This way we get stable children names as long as we only remove the last child.

---
  block/quorum.c | 6 ++++++
  1 file changed, 6 insertions(+)
block/quorum.c +0 -0
index df68adcfaa..6100d4108a 100644

block/quorum.c static void quorum_del_child(BlockDriverState *bs, BdrvChild *child, · line 1058

Quoted 5 lines
block/quorum.c +2 -0
@@ -1054,6 +1054,12 @@ static void quorum_del_child(BlockDriverState *bs, BdrvChild *child,
      /* We know now that num_children > threshold, so blkverify must be false */
      assert(!s->is_blkverify);
 
+    unsigned child_id;
+    sscanf(child->name, "children.%u", &child_id);

sscanf() cannot detect overflow. Do we trust our input enough to ignore this shortfall in the interface, or should we be using saner interfaces like qemu_strtoul()? For that matter, why do we have to reparse something; is it not already available somewhere in numerical form?

Quoted 11 lines
block/quorum.c +4 -1
+    if (child_id == s->next_child_index - 1) {
+        s->next_child_index--;
+    }
+
      bdrv_drained_begin(bs);
 
      /* We can safely remove this child now */
--

2.20.1

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#3 lore raw

Re: [PATCH 1/4] block/quorum.c: stable children names

Quoted 18 lines · On Thu, 21 Nov 2019 12:04:58 -0600 Eric Blake <[email protected]> wrote:
Quoted 17 lines · Lukas Straub

If we remove the child with the highest index from the quorum, decrement s->next_child_index. This way we get stable children names as long as we only remove the last child.

---
  block/quorum.c | 6 ++++++
  1 file changed, 6 insertions(+)
block/quorum.c +0 -0
index df68adcfaa..6100d4108a 100644
@@ -1054,6 +1054,12 @@ static void quorum_del_child(BlockDriverState *bs, BdrvChild *child,
      /* We know now that num_children > threshold, so blkverify must be false */
      assert(!s->is_blkverify);
 
Quoted 7 lines

block/quorum.c static void quorum_del_child(BlockDriverState *bs, BdrvChild *child, · line 1058

Quoted 2 lines
block/quorum.c +2 -0
+    unsigned child_id;
+    sscanf(child->name, "children.%u", &child_id);

sscanf() cannot detect overflow. Do we trust our input enough to ignore this shortfall in the interface, or should we be using saner interfaces like qemu_strtoul()? For that matter, why do we have to reparse something; is it not already available somewhere in numerical form?

Hi,
Yes, I wondered about that too, but found no other way. But the input is trusted, AFAIK the only way to add child nodes is trough quorum_add_child above and quorum_open and there already are adequate checks there.

Quoted 12 lines
Quoted 11 lines
block/quorum.c +4 -1
+    if (child_id == s->next_child_index - 1) {
+        s->next_child_index--;
+    }
+
      bdrv_drained_begin(bs);
 
      /* We can safely remove this child now */
--

2.20.1

#4 lore raw

Re: [PATCH 1/4] block/quorum.c: stable children names

Quoted 17 lines · Lukas Straub
Quoted 8 lines
Quoted 8 lines
block/quorum.c +0 -0
index df68adcfaa..6100d4108a 100644
@@ -1054,6 +1054,12 @@ static void quorum_del_child(BlockDriverState *bs, BdrvChild *child,
      /* We know now that num_children > threshold, so blkverify must be false */
      assert(!s->is_blkverify);
 
Quoted 8 lines

block/quorum.c static void quorum_del_child(BlockDriverState *bs, BdrvChild *child, · line 1058

Quoted 2 lines
block/quorum.c +2 -0
+    unsigned child_id;
+    sscanf(child->name, "children.%u", &child_id);

sscanf() cannot detect overflow. Do we trust our input enough to ignore this shortfall in the interface, or should we be using saner interfaces like qemu_strtoul()? For that matter, why do we have to reparse something; is it not already available somewhere in numerical form?

Quoted 4 lines

Yes, I wondered about that too, but found no other way. But the input is trusted, AFAIK the only way to add child nodes is trough quorum_add_child above and quorum_open and there already are adequate checks there.

I also don't see any other way to get that value, unless we change BDRVQuorumState to store that information (e.g. instead of children being a list of pointers BdrvChild ** it could be a list of {pointer, index}, or something like that).

There's another (more convoluted) alternative if we don't want to parse child->name. Since we only want to know if the child number equals s->next_child_index - 1, we can do it the other way around:

   snprintf(str, 32, "children.%u", s->next_child_index - 1);

and then compare str and child->name.

Berto

#5 lore raw

Re: [PATCH 1/4] block/quorum.c: stable children names

Quoted 32 lines · On Tue, 26 Nov 2019 15:21:37 +0100 Alberto Garcia <[email protected]> wrote:
Quoted 17 lines · Lukas Straub
Quoted 8 lines
Quoted 8 lines
block/quorum.c +0 -0
index df68adcfaa..6100d4108a 100644
@@ -1054,6 +1054,12 @@ static void quorum_del_child(BlockDriverState *bs, BdrvChild *child,
      /* We know now that num_children > threshold, so blkverify must be false */
      assert(!s->is_blkverify);
 
Quoted 8 lines

block/quorum.c static void quorum_del_child(BlockDriverState *bs, BdrvChild *child, · line 1058

Quoted 2 lines
block/quorum.c +2 -0
+    unsigned child_id;
+    sscanf(child->name, "children.%u", &child_id);

sscanf() cannot detect overflow. Do we trust our input enough to ignore this shortfall in the interface, or should we be using saner interfaces like qemu_strtoul()? For that matter, why do we have to reparse something; is it not already available somewhere in numerical form?

Quoted 4 lines

Yes, I wondered about that too, but found no other way. But the input is trusted, AFAIK the only way to add child nodes is trough quorum_add_child above and quorum_open and there already are adequate checks there.

I also don't see any other way to get that value, unless we change BDRVQuorumState to store that information (e.g. instead of children being a list of pointers BdrvChild ** it could be a list of {pointer, index}, or something like that).

There's another (more convoluted) alternative if we don't want to parse child->name. Since we only want to know if the child number equals s->next_child_index - 1, we can do it the other way around:

Quoted 5 lines
   snprintf(str, 32, "children.%u", s->next_child_index - 1);

and then compare str and child->name.

Berto

Hi,
I will do it your way, then it's also more consistent with the name creation in quorum_add and quorum_open.

Regards,
Lukas Straub

#6 lore raw

PATCH 2/4 colo: Introduce resource agent

In reply to Lukas Straub

Introduce a resource agent which can be used in a
Pacemaker cluster to manage qemu COLO.

---
 scripts/colo-resource-agent/colo | 1026 ++++++++++++++++++++++++++++++
 1 file changed, 1026 insertions(+)
 create mode 100755 scripts/colo-resource-agent/colo
scripts/colo-resource-agent/colo new file +1026 -0
new file mode 100755
index 0000000000..5fd9cfc0b5
@@ -0,0 +1,1026 @@
+#!/usr/bin/env python
+
+# Resource agent for qemu COLO for use with Pacemaker CRM
+#
+# Copyright (c) Lukas Straub <[email protected]>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+
+from __future__ import print_function
+import subprocess
+import sys
+import os
+import os.path
+import signal
+import socket
+import json
+import re
+import time
+import logging
+import logging.handlers
+
+# Constants
+OCF_SUCCESS = 0
+OCF_ERR_GENERIC = 1
+OCF_ERR_ARGS = 2
+OCF_ERR_UNIMPLEMENTED = 3
+OCF_ERR_PERM = 4
+OCF_ERR_INSTALLED = 5
+OCF_ERR_CONFIGURED = 6
+OCF_NOT_RUNNING = 7
+OCF_RUNNING_MASTER = 8
+OCF_FAILED_MASTER = 9
+
+# Get environment variables
+OCF_RESKEY_CRM_meta_notify_type \
+    = os.getenv("OCF_RESKEY_CRM_meta_notify_type")
+OCF_RESKEY_CRM_meta_notify_operation \
+    = os.getenv("OCF_RESKEY_CRM_meta_notify_operation")
+OCF_RESKEY_CRM_meta_notify_start_uname \
+    = os.getenv("OCF_RESKEY_CRM_meta_notify_start_uname", "")
+OCF_RESKEY_CRM_meta_notify_stop_uname \
+    = os.getenv("OCF_RESKEY_CRM_meta_notify_stop_uname", "")
+OCF_RESKEY_CRM_meta_notify_active_uname \
+    = os.getenv("OCF_RESKEY_CRM_meta_notify_active_uname", "")
+OCF_RESKEY_CRM_meta_notify_promote_uname \
+    = os.getenv("OCF_RESKEY_CRM_meta_notify_promote_uname", "")
+OCF_RESKEY_CRM_meta_notify_demote_uname \
+    = os.getenv("OCF_RESKEY_CRM_meta_notify_demote_uname", "")
+OCF_RESKEY_CRM_meta_notify_master_uname \
+    = os.getenv("OCF_RESKEY_CRM_meta_notify_master_uname", "")
+OCF_RESKEY_CRM_meta_notify_slave_uname \
+    = os.getenv("OCF_RESKEY_CRM_meta_notify_slave_uname", "")
+
+HA_RSCTMP = os.getenv("HA_RSCTMP", "/run/resource-agents")
+HA_LOGFACILITY = os.getenv("HA_LOGFACILITY")
+HA_LOGFILE = os.getenv("HA_LOGFILE")
+HA_DEBUGLOG = os.getenv("HA_DEBUGLOG")
+OCF_RESOURCE_INSTANCE = os.getenv("OCF_RESOURCE_INSTANCE", "default-instance")
+OCF_RESKEY_CRM_meta_timeout \
+    = int(os.getenv("OCF_RESKEY_CRM_meta_timeout", "60000"))
+OCF_RESKEY_CRM_meta_interval \
+    = int(os.getenv("OCF_RESKEY_CRM_meta_interval", "1"))
+OCF_RESKEY_CRM_meta_clone_max \
+    = int(os.getenv("OCF_RESKEY_CRM_meta_clone_max", "1"))
+OCF_RESKEY_CRM_meta_clone_node_max \
+    = int(os.getenv("OCF_RESKEY_CRM_meta_clone_node_max", "1"))
+OCF_RESKEY_CRM_meta_master_max \
+    = int(os.getenv("OCF_RESKEY_CRM_meta_master_max", "1"))
+OCF_RESKEY_CRM_meta_master_node_max \
+    = int(os.getenv("OCF_RESKEY_CRM_meta_master_node_max", "1"))
+OCF_RESKEY_CRM_meta_notify \
+    = os.getenv("OCF_RESKEY_CRM_meta_notify")
+OCF_RESKEY_CRM_meta_globally_unique \
+    = os.getenv("OCF_RESKEY_CRM_meta_globally_unique")
+
+HOSTNAME = os.getenv("OCF_RESKEY_CRM_meta_on_node", socket.gethostname())
+
+OCF_ACTION = os.getenv("__OCF_ACTION")
+if not OCF_ACTION and len(sys.argv) == 2:
+    OCF_ACTION = sys.argv[1]
+
+# Resource parameters
+OCF_RESKEY_binary_default = "qemu-system-x86_64"
+OCF_RESKEY_log_dir_default = HA_RSCTMP
+OCF_RESKEY_options_default = ""
+OCF_RESKEY_disk_size_default = ""
+OCF_RESKEY_active_hidden_dir_default = ""
+OCF_RESKEY_listen_address_default = "0.0.0.0"
+OCF_RESKEY_base_port_default = "9000"
+OCF_RESKEY_checkpoint_interval_default = "20000"
+OCF_RESKEY_debug_default = "false"
+
+OCF_RESKEY_binary = os.getenv("OCF_RESKEY_binary", OCF_RESKEY_binary_default)
+OCF_RESKEY_log_dir = os.getenv("OCF_RESKEY_log_dir", OCF_RESKEY_log_dir_default)
+OCF_RESKEY_options = os.getenv("OCF_RESKEY_options", OCF_RESKEY_options_default)
+OCF_RESKEY_disk_size \
+    = os.getenv("OCF_RESKEY_disk_size", OCF_RESKEY_disk_size_default)
+OCF_RESKEY_active_hidden_dir \
+    = os.getenv("OCF_RESKEY_active_hidden_dir", \
+                OCF_RESKEY_active_hidden_dir_default)
+OCF_RESKEY_listen_address \
+    = os.getenv("OCF_RESKEY_listen_address", OCF_RESKEY_listen_address_default)
+OCF_RESKEY_base_port \
+    = os.getenv("OCF_RESKEY_base_port", OCF_RESKEY_base_port_default)
+OCF_RESKEY_checkpoint_interval \
+    = os.getenv("OCF_RESKEY_checkpoint_interval", \
+                OCF_RESKEY_checkpoint_interval_default)
+OCF_RESKEY_debug = os.getenv("OCF_RESKEY_debug", OCF_RESKEY_debug_default)
+
+ACTIVE_IMAGE = os.path.join(OCF_RESKEY_active_hidden_dir, \
+                            OCF_RESOURCE_INSTANCE + "-active.qcow2")
+HIDDEN_IMAGE = os.path.join(OCF_RESKEY_active_hidden_dir, \
+                            OCF_RESOURCE_INSTANCE + "-hidden.qcow2")
+
+QMP_SOCK = os.path.join(HA_RSCTMP, OCF_RESOURCE_INSTANCE + "-qmp.sock")
+COMP_SOCK = os.path.join(HA_RSCTMP, OCF_RESOURCE_INSTANCE + "-compare.sock")
+COMP_OUT_SOCK = os.path.join(HA_RSCTMP, OCF_RESOURCE_INSTANCE \
+                                        + "-comp_out.sock")
+
+PID_FILE = os.path.join(HA_RSCTMP, OCF_RESOURCE_INSTANCE + "-qemu.pid")
+
+QMP_LOG = os.path.join(OCF_RESKEY_log_dir, OCF_RESOURCE_INSTANCE + "-qmp.log")
+QEMU_LOG = os.path.join(OCF_RESKEY_log_dir, OCF_RESOURCE_INSTANCE + "-qemu.log")
+
+# Exception only raised by ourself
+class Error(Exception):
+    pass
+
+def setup_constants():
+    # This function is called after the parameters where validated
+    global MIGRATE_PORT, MIROR_PORT, COMPARE_IN_PORT, NBD_PORT
+    MIGRATE_PORT = int(OCF_RESKEY_base_port)
+    MIROR_PORT = int(OCF_RESKEY_base_port) + 1
+    COMPARE_IN_PORT = int(OCF_RESKEY_base_port) + 2
+    NBD_PORT = int(OCF_RESKEY_base_port) + 3
+
+    global QEMU_PRIMARY_CMDLINE
+    QEMU_PRIMARY_CMDLINE = ("'%(OCF_RESKEY_binary)s' %(OCF_RESKEY_options)s"
+        " -chardev socket,id=comp0,path='%(COMP_SOCK)s',server,nowait"
+        " -chardev socket,id=comp0-0,path='%(COMP_SOCK)s'"
+        " -chardev socket,id=comp_out,path='%(COMP_OUT_SOCK)s',server,nowait"
+        " -chardev socket,id=comp_out0,path='%(COMP_OUT_SOCK)s'"
+        " -drive if=none,node-name=colo-disk0,driver=quorum,read-pattern=fifo,"
+        "vote-threshold=1,children.0=parent0"
+        " -qmp unix:'%(QMP_SOCK)s',server,nowait"
+        " -daemonize -D '%(QEMU_LOG)s' -pidfile '%(PID_FILE)s'") % globals()
+
+    global QEMU_SECONDARY_CMDLINE
+    QEMU_SECONDARY_CMDLINE = ("'%(OCF_RESKEY_binary)s' %(OCF_RESKEY_options)s"
+        " -chardev socket,id=red0,host='%(OCF_RESKEY_listen_address)s',"
+        "port=%(MIROR_PORT)s,server,nowait"
+        " -chardev socket,id=red1,host='%(OCF_RESKEY_listen_address)s',"
+        "port=%(COMPARE_IN_PORT)s,server,nowait"
+        " -object filter-redirector,id=f1,netdev=hn0,queue=tx,indev=red0"
+        " -object filter-redirector,id=f2,netdev=hn0,queue=rx,outdev=red1"
+        " -object filter-rewriter,id=rew0,netdev=hn0,queue=all"
+        " -drive if=none,node-name=childs0,top-id=colo-disk0,"
+        "driver=replication,mode=secondary,file.driver=qcow2,"
+        "file.file.filename='%(ACTIVE_IMAGE)s',file.backing.driver=qcow2,"
+        "file.backing.file.filename='%(HIDDEN_IMAGE)s',"
+        "file.backing.backing=parent0"
+        " -drive if=none,node-name=colo-disk0,driver=quorum,read-pattern=fifo,"
+        "vote-threshold=1,children.0=childs0"
+        " -incoming tcp:'%(OCF_RESKEY_listen_address)s':%(MIGRATE_PORT)s"
+        " -qmp unix:'%(QMP_SOCK)s',server,nowait"
+        " -daemonize -D '%(QEMU_LOG)s' -pidfile '%(PID_FILE)s'") % globals()
+
+def qemu_colo_meta_data():
+    print("""\
+<?xml version="1.0"?>
+<!DOCTYPE resource-agent SYSTEM "ra-api-1.dtd">
+<resource-agent name="colo">
+
+    <version>0.1</version>
+    <longdesc lang="en">
+Resource agent for qemu COLO. (https://wiki.qemu.org/Features/COLO)
+
+After defining the master/slave instance, the master score has to be
+manually set to show which node has up-to-date data. So you copy your
+image to one host (and create empty images the other host(s)) and then
+run "crm_master -r name_of_your_primitive -v 10" on that host.
+
+Also, you have to set 'notify=true' in the metadata attributes when
+defining the master/slave instance.
+    </longdesc>
+    <shortdesc lang="en">Qemu COLO</shortdesc>
+
+    <parameters>
+
+    <parameter name="binary" unique="0" required="0">
+        <longdesc lang="en">Qemu binary to use</longdesc>
+        <shortdesc lang="en">Qemu binary</shortdesc>
+        <content type="string" default=\"""" \
+            + OCF_RESKEY_binary_default + """\"/>
+    </parameter>
+
+    <parameter name="log_dir" unique="0" required="0">
+        <longdesc lang="en">Directory to place logs in</longdesc>
+        <shortdesc lang="en">Log directory</shortdesc>
+        <content type="string" default=\"""" \
+            + OCF_RESKEY_log_dir_default + """\"/>
+    </parameter>
+
+    <parameter name="options" unique="0" required="1">
+        <longdesc lang="en">
+Options to pass to qemu. These will be passed alongside COLO specific
+options, so you need to follow these conventions: The netdev should have
+id=hn0 and the disk controller drive=colo-disk0. The image node should
+have id/node-name=parent0, but should not be connected to the guest.
+Example:
+-vnc :0 -enable-kvm -cpu qemu64,+kvmclock -m 512 -netdev bridge,id=hn0
+-device virtio-net,netdev=hn0 -device virtio-blk,drive=colo-disk0
+-drive if=none,id=parent0,format=qcow2,file=/mnt/vms/vm01.qcow2
+        </longdesc>
+        <shortdesc lang="en">Options to pass to qemu.</shortdesc>
+    </parameter>
+
+    <parameter name="disk_size" unique="0" required="1">
+        <longdesc lang="en">Disk size of the image</longdesc>
+        <shortdesc lang="en">Disk size of the image</shortdesc>
+        <content type="string" default=\"""" \
+            + OCF_RESKEY_disk_size_default + """\"/>
+    </parameter>
+
+    <parameter name="active_hidden_dir" unique="0" required="1">
+        <longdesc lang="en">
+Directory where the active and hidden images will be stored. It is
+recommended to put this on a ramdisk.
+        </longdesc>
+        <shortdesc lang="en">Path to active and hidden images</shortdesc>
+        <content type="string" default=\"""" \
+            + OCF_RESKEY_active_hidden_dir_default + """\"/>
+    </parameter>
+
+    <parameter name="listen_address" unique="0" required="0">
+        <longdesc lang="en">Address to listen on.</longdesc>
+        <shortdesc lang="en">Listen address</shortdesc>
+        <content type="string" default=\"""" \
+            + OCF_RESKEY_listen_address_default + """\"/>
+    </parameter>
+
+    <parameter name="base_port" unique="1" required="0">
+        <longdesc lang="en">
+4 tcp ports that are unique for each instance. (base_port to base_port + 3)
+        </longdesc>
+        <shortdesc lang="en">Ports to use</shortdesc>
+        <content type="integer" default=\"""" \
+            + OCF_RESKEY_base_port_default + """\"/>
+    </parameter>
+
+    <parameter name="checkpoint_interval" unique="0" required="0">
+        <longdesc lang="en">
+Interval for regular checkpoints in milliseconds.
+        </longdesc>
+        <shortdesc lang="en">Interval for regular checkpoints</shortdesc>
+        <content type="integer" default=\"""" \
+            + OCF_RESKEY_checkpoint_interval_default + """\"/>
+    </parameter>
+
+    <parameter name="debug" unique="0" required="0">
+        <longdesc lang="en">Enable debuging</longdesc>
+        <shortdesc lang="en">Enable debuging</shortdesc>
+        <content type="string" default=\"""" \
+            + OCF_RESKEY_debug_default + """\"/>
+    </parameter>
+
+    </parameters>
+
+    <actions>
+        <action name="start"        timeout="30s" />
+        <action name="stop"         timeout="20s" />
+        <action name="monitor"      timeout="40s" \
+            interval="12s" depth="0" />
+        <action name="monitor"      timeout="40s" \
+            interval="10s" depth="0" role="Slave" />
+        <action name="monitor"      timeout="40s" \
+            interval="11s" depth="0" role="Master" />
+        <action name="notify"       timeout="120s" />
+        <action name="promote"      timeout="120s" />
+        <action name="demote"       timeout="120s" />
+        <action name="meta-data"    timeout="5s" />
+        <action name="validate-all" timeout="20s" />
+    </actions>
+
+</resource-agent>
+""")
+
+def logs_open():
+    global log
+    log = logging.getLogger(OCF_RESOURCE_INSTANCE)
+    log.setLevel(logging.DEBUG)
+    formater = logging.Formatter("(%(name)s) %(levelname)s: %(message)s")
+
+    if sys.stdout.isatty():
+        handler = logging.StreamHandler(stream=sys.stderr)
+        handler.setFormatter(formater)
+        log.addHandler(handler)
+
+    if HA_LOGFACILITY:
+        handler = logging.handlers.SysLogHandler("/dev/log")
+        handler.setFormatter(formater)
+        log.addHandler(handler)
+
+    if HA_LOGFILE:
+        handler = logging.FileHandler(HA_LOGFILE)
+        handler.setFormatter(formater)
+        log.addHandler(handler)
+
+    if HA_DEBUGLOG and HA_DEBUGLOG != HA_LOGFILE:
+        handler = logging.FileHandler(HA_DEBUGLOG)
+        handler.setFormatter(formater)
+        log.addHandler(handler)
+
+    global qmp_log
+    qmp_log = logging.getLogger("qmp_log")
+    qmp_log.setLevel(logging.DEBUG)
+    formater = logging.Formatter("%(message)s")
+
+    if is_true(OCF_RESKEY_debug):
+        handler = logging.handlers.WatchedFileHandler(QMP_LOG)
+        handler.setFormatter(formater)
+        qmp_log.addHandler(handler)
+    else:
+        handler = logging.NullHandler()
+        qmp_log.addHandler(handler)
+
+def rotate_logfile(logfile, numlogs):
+    numlogs -= 1
+    for n in range(numlogs, -1, -1):
+        file = logfile
+        if n != 0:
+            file = "%s.%s" % (file, n)
+        if os.path.exists(file):
+            if n == numlogs:
+                os.remove(file)
+            else:
+                newname = "%s.%s" % (logfile, n + 1)
+                os.rename(file, newname)
+
+def is_writable(file):
+    return os.access(file, os.W_OK)
+
+def is_executable_file(file):
+    return os.path.isfile(file) and os.access(file, os.X_OK)
+
+def is_true(var):
+    return re.match("yes|true|1|YES|TRUE|True|ja|on|ON", str(var)) != None
+
+# Check if the binary exists and is executable
+def check_binary(binary):
+    if is_executable_file(binary):
+        return True
+    PATH = os.getenv("PATH", os.defpath)
+    for dir in PATH.split(os.pathsep):
+        if is_executable_file(os.path.join(dir, binary)):
+            return True
+    log.error("binary \"%s\" doesn't exist or not executable" % binary)
+    return False
+
+def run_command(commandline):
+    proc = subprocess.Popen(commandline, shell=True, stdout=subprocess.PIPE,\
+                          stderr=subprocess.STDOUT, universal_newlines=True)
+    stdout, stderr = proc.communicate()
+    if proc.returncode != 0:
+        log.error("command \"%s\" failed with code %s:\n%s" \
+                    % (commandline, proc.returncode, stdout))
+        raise Error()
+
+# Functions for setting and getting the master score to tell Pacemaker which
+# host has the most recent data
+def set_master_score(score):
+    if score <= 0:
+        os.system("crm_master -q -l forever -D")
+    else:
+        os.system("crm_master -q -l forever -v %s" % score)
+
+def set_remote_master_score(remote, score):
+    if score <= 0:
+        os.system("crm_master -q -l forever -N '%s' -D" % remote)
+    else:
+        os.system("crm_master -q -l forever -N '%s' -v %s" % (remote, score))
+
+def get_master_score():
+    proc = subprocess.Popen("crm_master -q -G", shell=True, \
+                            stdout=subprocess.PIPE, universal_newlines=True)
+    stdout, stderr = proc.communicate()
+    if proc.returncode != 0:
+        return 0
+    else:
+        return int(str.strip(stdout))
+
+def get_remote_master_score(remote):
+    proc = subprocess.Popen("crm_master -q -N '%s' -G" % remote, shell=True, \
+                            stdout=subprocess.PIPE, universal_newlines=True)
+    stdout, stderr = proc.communicate()
+    if proc.returncode != 0:
+        return 0
+    else:
+        return int(str.strip(stdout))
+
+def recv_line(fd):
+    line = ""
+    while True:
+        tmp = fd.recv(1)
+        if tmp == "\n":
+            break
+        line += tmp
+    return line
+
+# Filter out events
+def read_answer(fd):
+    while True:
+        line = recv_line(fd)
+        qmp_log.debug(line)
+
+        answer = json.loads(line)
+        # Ignore everything else
+        if "return" in answer or "error" in answer:
+            break
+    return answer
+
+# Execute one or more qmp commands
+def qmp_execute(fd, commands, ignore_error = False):
+    for command in commands:
+        if not command:
+            continue
+
+        try:
+            to_send = json.dumps(command)
+            fd.sendall(to_send + "\n")
+            qmp_log.debug(to_send)
+
+            answer = read_answer(fd)
+        except Exception as e:
+            log.error("while executing qmp command: %s\n%s" \
+                        % (json.dumps(command), e))
+            raise Error()
+
+        if not ignore_error and ("error" in answer):
+            log.error("qmp command returned error:\n%s\n%s" \
+                        % (json.dumps(command), json.dumps(answer)))
+            raise Error()
+
+    return answer
+
+# Open qemu qmp connection
+def qmp_open(fail_fast = False):
+    # Timeout for commands = our timeout minus 10s
+    timeout = max(1, (OCF_RESKEY_CRM_meta_timeout/1000)-10)
+
+    try:
+        fd = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
+        if fail_fast:
+            fd.settimeout(min(10, (OCF_RESKEY_CRM_meta_timeout/1000)))
+        else:
+            fd.settimeout(timeout)
+        fd.connect(QMP_SOCK)
+
+        answer = qmp_execute(fd, [{"execute": "qmp_capabilities"}], True)
+    except Exception as e:
+        log.error("while connecting to qmp socket: %s" % e)
+        raise Error()
+
+    if "error" in answer:
+        log.error("while connecting to qmp socket: %s" % json.dumps(answer))
+        raise Error()
+
+    fd.settimeout(timeout)
+    return fd
+
+# Check qemu health and colo role
+def qmp_check_state(fd):
+    answer = qmp_execute(fd, [{"execute": "query-status"}])
+    vm_status = answer["return"]
+
+    answer = qmp_execute(fd, [{"execute": "query-colo-status"}])
+    colo_status = answer["return"]
+
+    if vm_status["status"] == "inmigrate":
+        role = OCF_SUCCESS
+        replication = OCF_NOT_RUNNING
+
+    elif (vm_status["status"] == "running" \
+          or vm_status["status"] == "colo" \
+          or vm_status["status"] == "finish-migrate") \
+         and colo_status["mode"] == "none" \
+         and (colo_status["reason"] == "request" \
+              or colo_status["reason"] == "none"):
+        role = OCF_RUNNING_MASTER
+        replication = OCF_NOT_RUNNING
+
+    elif (vm_status["status"] == "running" \
+          or vm_status["status"] == "colo" \
+          or vm_status["status"] == "finish-migrate") \
+         and colo_status["mode"] == "secondary":
+        role = OCF_SUCCESS
+        replication = OCF_SUCCESS
+
+    elif (vm_status["status"] == "running" \
+          or vm_status["status"] == "colo" \
+          or vm_status["status"] == "finish-migrate") \
+         and colo_status["mode"] == "primary":
+        role = OCF_RUNNING_MASTER
+        replication = OCF_SUCCESS
+
+    else:
+        log.error("unknown qemu status: vm status: %s colo mode: %s" \
+                    % (vm_status["status"], colo_status["mode"]))
+        role = OCF_ERR_GENERIC
+        replication = OCF_ERR_GENERIC
+
+    return role, replication
+
+# Get the host of the nbd node
+def qmp_get_nbd_remote(fd):
+    block_nodes = qmp_execute(fd, [{"execute": "query-named-block-nodes"}])
+    for node in block_nodes["return"]:
+        if node["node-name"] == "nbd0":
+            url = str(node["image"]["filename"])
+            return str.split(url, "//")[1].split("/")[0].split(":")[0]
+    return None
+
+# Check if we are currently resyncing
+def qmp_check_resync(fd):
+    answer = qmp_execute(fd, [{"execute": "query-block-jobs"}])
+    for job in answer["return"]:
+        if job["device"] == "resync":
+            return job
+    return None
+
+def qmp_start_resync(fd, remote):
+    qmp_execute(fd, [
+        {"execute": "blockdev-add", "arguments": {"driver": "nbd", "node-name": "nbd0", "server": {"type": "inet", "host": str(remote), "port": str(NBD_PORT)}, "export": "parent0"}},
+        {"execute": "blockdev-mirror", "arguments": {"device": "colo-disk0", "job-id": "resync", "target": "nbd0", "sync": "full", "auto-dismiss": False}}
+        ])
+
+def qmp_cancel_resync(fd):
+    if qmp_check_resync(fd)["status"] != "concluded":
+        qmp_execute(fd, [{"execute": "block-job-cancel", "arguments": {"device": "resync", "force": True}}], True)
+        # Wait for the block-job to finish
+        while qmp_check_resync(fd)["status"] != "concluded":
+            time.sleep(1)
+
+    qmp_execute(fd, [
+        {"execute": "block-job-dismiss", "arguments": {"id": "resync"}},
+        {"execute": "blockdev-del", "arguments": {"node-name": "nbd0"}}
+        ])
+
+def qmp_start_colo(fd, remote):
+    # Check if we have a filter-rewriter
+    answer = qmp_execute(fd, [{"execute": "qom-list", "arguments": {"path": "/objects/rew0"}}], True)
+    if "error" in answer:
+        if answer["error"]["class"] == "DeviceNotFound":
+            have_filter_rewriter = False
+        else:
+            log.error("while checking for filter-rewriter:\n%s" \
+                        % json.dumps(answer))
+            raise Error()
+    else:
+        have_filter_rewriter = True
+
+    # Pause VM and cancel resync
+    qmp_execute(fd, [
+        {"execute": "stop"},
+        {"execute": "block-job-cancel", "arguments": {"device": "resync"}}
+        ])
+
+    # Wait for the block-job to finish
+    while qmp_check_resync(fd)["status"] != "concluded":
+        time.sleep(1)
+
+    # Add the replication node
+    qmp_execute(fd, [
+        {"execute": "block-job-dismiss", "arguments": {"id": "resync"}},
+        {"execute": "blockdev-add", "arguments": {"driver": "replication", "node-name": "replication0", "mode": "primary", "file": "nbd0"}},
+        {"execute": "x-blockdev-change", "arguments": {"parent": "colo-disk0", "node": "replication0"}}
+        ])
+
+    # Connect mirror and compare_in to secondary
+    qmp_execute(fd, [
+        {"execute": "chardev-add", "arguments": {"id": "mirror0", "backend": {"type": "socket", "data": {"addr": {"type": "inet", "data": {"host": str(remote), "port": str(MIROR_PORT)}}, "server": False, "reconnect": 1}}}},
+        {"execute": "chardev-add", "arguments": {"id": "compare1", "backend": {"type": "socket", "data": {"addr": {"type": "inet", "data": {"host": str(remote), "port": str(COMPARE_IN_PORT)}}, "server": False, "reconnect": 1}}}}
+        ])
+
+    # Add the COLO filters
+    if have_filter_rewriter:
+        qmp_execute(fd, [
+            {"execute": "object-add", "arguments": {"qom-type": "filter-mirror", "id": "m0", "props": {"insert": "before", "position": "id=rew0", "netdev": "hn0", "queue": "tx", "outdev": "mirror0"}}},
+            {"execute": "object-add", "arguments": {"qom-type": "filter-redirector", "id": "redire0", "props": {"insert": "before", "position": "id=rew0", "netdev": "hn0", "queue": "rx", "indev": "comp_out"}}},
+            {"execute": "object-add", "arguments": {"qom-type": "filter-redirector", "id": "redire1", "props": {"insert": "before", "position": "id=rew0", "netdev": "hn0", "queue": "rx", "outdev": "comp0"}}},
+            {"execute": "object-add", "arguments": {"qom-type": "iothread", "id": "iothread1"}},
+            {"execute": "object-add", "arguments": {"qom-type": "colo-compare", "id": "comp0", "props": {"primary_in": "comp0-0", "secondary_in": "compare1", "outdev": "comp_out0", "iothread": "iothread1"}}}
+            ])
+    else:
+        qmp_execute(fd, [
+            {"execute": "object-add", "arguments": {"qom-type": "filter-mirror", "id": "m0", "props": {"netdev": "hn0", "queue": "tx", "outdev": "mirror0"}}},
+            {"execute": "object-add", "arguments": {"qom-type": "filter-redirector", "id": "redire0", "props": {"netdev": "hn0", "queue": "rx", "indev": "comp_out"}}},
+            {"execute": "object-add", "arguments": {"qom-type": "filter-redirector", "id": "redire1", "props": {"netdev": "hn0", "queue": "rx", "outdev": "comp0"}}},
+            {"execute": "object-add", "arguments": {"qom-type": "iothread", "id": "iothread1"}},
+            {"execute": "object-add", "arguments": {"qom-type": "colo-compare", "id": "comp0", "props": {"primary_in": "comp0-0", "secondary_in": "compare1", "outdev": "comp_out0", "iothread": "iothread1"}}}
+            ])
+
+    # Start COLO
+    qmp_execute(fd, [
+        {"execute": "migrate-set-capabilities", "arguments": {"capabilities": [{"capability": "x-colo", "state": True }] }},
+        {"execute": "migrate-set-parameters" , "arguments": {"x-checkpoint-delay": int(OCF_RESKEY_checkpoint_interval)}},
+        {"execute": "migrate", "arguments": {"uri": "tcp:%s:%s" % (remote, MIGRATE_PORT)}}
+        ])
+
+    # Wait for COLO to start
+    while qmp_execute(fd, [{"execute": "query-status"}])["return"]["status"] \
+            == "paused":
+        time.sleep(1)
+
+def qmp_primary_failover(fd):
+    qmp_execute(fd, [
+        {"execute": "x-blockdev-change", "arguments": {"parent": "colo-disk0", "child": "children.1"}},
+        {"execute": "blockdev-del", "arguments": {"node-name": "replication0"}},
+        {"execute": "blockdev-del", "arguments": {"node-name": "nbd0"}},
+        {"execute": "object-del", "arguments": {"id": "comp0"}},
+        {"execute": "object-del", "arguments": {"id": "iothread1"}},
+        {"execute": "object-del", "arguments": {"id": "m0"}},
+        {"execute": "object-del", "arguments": {"id": "redire0"}},
+        {"execute": "object-del", "arguments": {"id": "redire1"}},
+        {"execute": "chardev-remove", "arguments": {"id": "mirror0"}},
+        {"execute": "chardev-remove", "arguments": {"id": "compare1"}},
+        {"execute": "x-colo-lost-heartbeat"}
+        ])
+
+def qmp_secondary_failover(fd):
+    # Stop the NBD server, and resume
+    qmp_execute(fd, [
+        {"execute": "nbd-server-stop"},
+        {"execute": "x-colo-lost-heartbeat"}
+        ])
+
+    # Prepare for continuing replication when we have a new secondary
+    qmp_execute(fd, [
+        {"execute": "object-del", "arguments": {"id": "f2"}},
+        {"execute": "object-del", "arguments": {"id": "f1"}},
+        {"execute": "chardev-remove", "arguments": {"id": "red1"}},
+        {"execute": "chardev-remove", "arguments": {"id": "red0"}},
+        {"execute": "chardev-add", "arguments": {"id": "comp0", "backend": {"type": "socket", "data": {"addr": {"type": "unix", "data": {"path": str(COMP_SOCK)}}, "server": True}}}},
+        {"execute": "chardev-add", "arguments": {"id": "comp0-0", "backend": {"type": "socket", "data": {"addr": {"type": "unix", "data": {"path": str(COMP_SOCK)}}, "server": False}}}},
+        {"execute": "chardev-add", "arguments": {"id": "comp_out", "backend": {"type": "socket", "data": {"addr": {"type": "unix", "data": {"path": str(COMP_OUT_SOCK)}}, "server": True}}}},
+        {"execute": "chardev-add", "arguments": {"id": "comp_out0", "backend": {"type": "socket", "data": {"addr": {"type": "unix", "data": {"path": str(COMP_OUT_SOCK)}}, "server": False}}}}
+        ])
+
+# Sanity checks: check parameters, files, binaries, etc.
+def qemu_colo_validate_all():
+    # Check resource parameters
+    if not str.isdigit(OCF_RESKEY_base_port):
+        log.error("base_port needs to be a number")
+        return OCF_ERR_CONFIGURED
+
+    if not str.isdigit(OCF_RESKEY_checkpoint_interval):
+        log.error("checkpoint_interval needs to be a number")
+        return OCF_ERR_CONFIGURED
+
+    if not OCF_RESKEY_active_hidden_dir:
+        log.error("active_hidden_dir needs to be specified")
+        return OCF_ERR_CONFIGURED
+
+    if not OCF_RESKEY_disk_size:
+        log.error("disk_size needs to be specified")
+        return OCF_ERR_CONFIGURED
+
+    # Check resource meta configuration
+    if OCF_ACTION != "stop":
+        if OCF_RESKEY_CRM_meta_master_max != 1:
+            log.error("only one master allowed")
+            return OCF_ERR_CONFIGURED
+
+        if OCF_RESKEY_CRM_meta_clone_max > 2:
+            log.error("maximum 2 clones allowed")
+            return OCF_ERR_CONFIGURED
+
+        if OCF_RESKEY_CRM_meta_master_node_max != 1:
+            log.error("only one master per node allowed")
+            return OCF_ERR_CONFIGURED
+
+        if OCF_RESKEY_CRM_meta_clone_node_max != 1:
+            log.error("only one clone per node allowed")
+            return OCF_ERR_CONFIGURED
+
+    # Check if notify is enabled
+    if OCF_ACTION != "stop" and OCF_ACTION != "monitor":
+        if not is_true(OCF_RESKEY_CRM_meta_notify) \
+           and not OCF_RESKEY_CRM_meta_notify_start_uname:
+            log.error("notify needs to be enabled")
+            return OCF_ERR_CONFIGURED
+
+    # Check that globally-unique is disabled
+    if is_true(OCF_RESKEY_CRM_meta_globally_unique):
+        log.error("globally-unique needs to be disabled")
+        return OCF_ERR_CONFIGURED
+
+    # Check binaries
+    if not check_binary(OCF_RESKEY_binary):
+        return OCF_ERR_INSTALLED
+
+    if not check_binary("qemu-img"):
+        return OCF_ERR_INSTALLED
+
+    # Check paths and files
+    if not is_writable(OCF_RESKEY_active_hidden_dir) \
+        or not os.path.isdir(OCF_RESKEY_active_hidden_dir):
+        log.error("active and hidden image directory missing or not writable")
+        return OCF_ERR_PERM
+
+    return OCF_SUCCESS
+
+# Check if qemu is running
+def check_pid():
+    if not os.path.exists(PID_FILE):
+        return OCF_NOT_RUNNING, None
+
+    fd = open(PID_FILE, "r")
+    pid = int(str.strip(fd.readline()))
+    fd.close()
+    try:
+        os.kill(pid, 0)
+    except OSError:
+        log.info("qemu is not running")
+        return OCF_NOT_RUNNING, pid
+    else:
+        return OCF_SUCCESS, pid
+
+def qemu_colo_monitor(fail_fast = False):
+    status, pid = check_pid()
+    if status != OCF_SUCCESS:
+        return status, OCF_ERR_GENERIC
+
+    fd = qmp_open(fail_fast)
+
+    role, replication = qmp_check_state(fd)
+    if role != OCF_SUCCESS and role != OCF_RUNNING_MASTER:
+        return role, replication
+
+    if not fail_fast and OCF_RESKEY_CRM_meta_interval != 0:
+        # This isn't a probe monitor
+        block_job = qmp_check_resync(fd)
+        if block_job != None:
+            if "error" in block_job:
+                log.error("resync error: %s" % block_job["error"])
+                qmp_cancel_resync(fd)
+                # TODO: notify pacemaker about peer failure
+            elif block_job["ready"] == True:
+                log.info("resync done, starting colo")
+                peer = qmp_get_nbd_remote(fd)
+                qmp_start_colo(fd, peer)
+                # COLO started, our secondary now can be promoted if the
+                # primary fails
+                set_remote_master_score(peer, 100)
+            else:
+                pct_done = (float(block_job["offset"]) \
+                            / float(block_job["len"])) * 100
+                log.info("resync %.2f%% done" % pct_done)
+
+    fd.close()
+
+    return role, replication
+
+def qemu_colo_start():
+    if check_pid()[0] == OCF_SUCCESS:
+        log.info("qemu is already running")
+        return OCF_SUCCESS
+
+    run_command("qemu-img create -q -f qcow2 %s %s" \
+                % (ACTIVE_IMAGE, OCF_RESKEY_disk_size))
+    run_command("qemu-img create -q -f qcow2 %s %s" \
+                % (HIDDEN_IMAGE, OCF_RESKEY_disk_size))
+
+    rotate_logfile(QMP_LOG, 4)
+    rotate_logfile(QEMU_LOG, 4)
+    run_command(QEMU_SECONDARY_CMDLINE)
+
+    fd = qmp_open()
+    qmp_execute(fd, [
+        {"execute": "nbd-server-start", "arguments": {"addr": {"type": "inet", "data": {"host": str(OCF_RESKEY_listen_address), "port": str(NBD_PORT)}}}},
+        {"execute": "nbd-server-add", "arguments": {"device": "parent0", "writable": True}}
+        ])
+    fd.close()
+
+    return OCF_SUCCESS
+
+def env_do_shutdown_guest():
+    return OCF_RESKEY_CRM_meta_notify_active_uname \
+           and OCF_RESKEY_CRM_meta_notify_stop_uname \
+           and str.strip(OCF_RESKEY_CRM_meta_notify_active_uname) \
+               == str.strip(OCF_RESKEY_CRM_meta_notify_stop_uname)
+
+def env_find_secondary():
+    # slave(s) =
+    # OCF_RESKEY_CRM_meta_notify_slave_uname
+    # - OCF_RESKEY_CRM_meta_notify_stop_uname
+    # + OCF_RESKEY_CRM_meta_notify_start_uname
+    # Filter out hosts that are stopping and ourselves
+    for host in str.split(OCF_RESKEY_CRM_meta_notify_slave_uname, " "):
+        if host:
+            for stopping_host \
+                in str.split(OCF_RESKEY_CRM_meta_notify_stop_uname, " "):
+                if host == stopping_host:
+                    break
+            else:
+                if host != HOSTNAME:
+                    # we found a valid secondary
+                    return host
+
+    for host in str.split(OCF_RESKEY_CRM_meta_notify_start_uname, " "):
+        if host != HOSTNAME:
+            # we found a valid secondary
+            return host
+
+    # we found no secondary
+    return None
+
+def _qemu_colo_stop(monstatus, shutdown_guest):
+    # stop action must do everything possible to stop the resource
+    try:
+        now = time.time()
+        timeout = now + (OCF_RESKEY_CRM_meta_timeout/1000)-10
+        force_stop = False
+
+        if monstatus == OCF_NOT_RUNNING:
+            log.info("resource is already stopped")
+            return OCF_SUCCESS
+        elif monstatus == OCF_RUNNING_MASTER or monstatus == OCF_SUCCESS:
+            force_stop = False
+        else:
+            force_stop = True
+
+        if not force_stop:
+            fd = qmp_open(True)
+            if shutdown_guest:
+                if monstatus == OCF_RUNNING_MASTER:
+                    qmp_execute(fd, [{"execute": "system_powerdown"}])
+            else:
+                qmp_execute(fd, [{"execute": "quit"}])
+            fd.close()
+
+            # wait for qemu to stop
+            while time.time() < timeout:
+                status, pid = check_pid()
+                if status == OCF_NOT_RUNNING:
+                    # qemu stopped
+                    return OCF_SUCCESS
+                elif status == OCF_SUCCESS:
+                    # wait
+                    time.sleep(1)
+                else:
+                    # something went wrong, force stop instead
+                    break
+
+            log.warning("clean stop timeout reached")
+    except Exception as e:
+        log.warning("error while stopping: \"%s\"" % e.message)
+
+    log.info("force stopping qemu")
+
+    status, pid = check_pid()
+    if status == OCF_NOT_RUNNING:
+        return OCF_SUCCESS
+    try:
+        os.kill(pid, signal.SIGTERM)
+        time.sleep(2)
+        os.kill(pid, signal.SIGKILL)
+    except Exception:
+        pass
+
+    while check_pid()[0] != OCF_NOT_RUNNING:
+        time.sleep(1)
+
+    return OCF_SUCCESS
+
+def qemu_colo_stop():
+    shutdown_guest = env_do_shutdown_guest()
+    try:
+        role, replication = qemu_colo_monitor(True)
+    except Exception:
+        role, replication = OCF_ERR_GENERIC, OCF_ERR_GENERIC
+
+    status = _qemu_colo_stop(role, shutdown_guest)
+
+    if HOSTNAME == str.strip(OCF_RESKEY_CRM_meta_notify_master_uname):
+        peer = env_find_secondary()
+        if peer and (get_remote_master_score(peer) > 10) \
+           and not shutdown_guest:
+            # We where a healthy primary and had a healthy secondary. We where
+            # stopped so outdate ourselves, as the secondary will take over.
+            set_master_score(0)
+        else:
+            if role == OCF_RUNNING_MASTER:
+                # We where a healthy primary but had no healty secondary or it
+                # was stopped as well. So we have up-to-date data.
+                set_master_score(10)
+            else:
+                # We where a unhealthy primary but also had no healty secondary.
+                # So we still should have up-to-date data.
+                set_master_score(5)
+    else:
+        if get_master_score() > 10:
+            if role == OCF_SUCCESS:
+                if shutdown_guest:
+                    # We where a healthy secondary and (probably) had a healthy
+                    # primary and both where stopped. So we have up-to-date data
+                    # too.
+                    set_master_score(10)
+                else:
+                    # We where a healthy secondary and (probably) had a healthy
+                    # primary still running. So we are now out of date.
+                    set_master_score(0)
+            else:
+                # We where a unhealthy secondary. So we are now out of date.
+                set_master_score(0)
+
+    return status
+
+def qemu_colo_notify():
+    action = "%s-%s" % (OCF_RESKEY_CRM_meta_notify_type, \
+                        OCF_RESKEY_CRM_meta_notify_operation)
+
+    if action == "post-start":
+        if HOSTNAME == str.strip(OCF_RESKEY_CRM_meta_notify_master_uname):
+            peer = str.strip(OCF_RESKEY_CRM_meta_notify_start_uname)
+            fd = qmp_open()
+            qmp_start_resync(fd, peer)
+            fd.close()
+            # The secondary has inconsistent data until resync is finished
+            set_remote_master_score(peer, 0)
+
+    elif action == "pre-stop":
+        if not env_do_shutdown_guest() \
+           and HOSTNAME == str.strip(OCF_RESKEY_CRM_meta_notify_master_uname):
+            fd = qmp_open()
+            peer = qmp_get_nbd_remote(fd)
+            if peer == str.strip(OCF_RESKEY_CRM_meta_notify_stop_uname):
+                if qmp_check_resync(fd) != None:
+                    qmp_cancel_resync(fd)
+                elif peer and get_remote_master_score(peer) > 10:
+                    qmp_primary_failover(fd)
+            fd.close()
+
+    return OCF_SUCCESS
+
+def qemu_colo_promote():
+    role, replication = qemu_colo_monitor()
+
+    if role == OCF_SUCCESS and replication == OCF_NOT_RUNNING:
+        status = _qemu_colo_stop(role, False)
+        if status != OCF_SUCCESS:
+            return status
+
+        rotate_logfile(QMP_LOG, 4)
+        rotate_logfile(QEMU_LOG, 4)
+        run_command(QEMU_PRIMARY_CMDLINE)
+        set_master_score(101)
+
+        peer = env_find_secondary()
+        if peer:
+            fd = qmp_open()
+            qmp_start_resync(fd, peer)
+            # The secondary has inconsistent data until resync is finished
+            set_remote_master_score(peer, 0)
+            fd.close()
+        return OCF_SUCCESS
+    elif role == OCF_SUCCESS and replication == OCF_SUCCESS:
+        fd = qmp_open()
+        qmp_secondary_failover(fd)
+        set_master_score(101)
+
+        peer = env_find_secondary()
+        if peer:
+            qmp_start_resync(fd, peer)
+            # The secondary has inconsistent data until resync is finished
+            set_remote_master_score(peer, 0)
+        fd.close()
+        return OCF_SUCCESS
+    else:
+        return OCF_ERR_GENERIC
+
+def qemu_colo_demote():
+    status = qemu_colo_stop()
+    if status != OCF_SUCCESS:
+        return status
+    return qemu_colo_start()
+
+
+if OCF_ACTION == "meta-data":
+    qemu_colo_meta_data()
+    exit(OCF_SUCCESS)
+
+logs_open()
+
+status = qemu_colo_validate_all()
+# Exit here if our sanity checks fail, but try to continue if we need to stop
+if status != OCF_SUCCESS and OCF_ACTION != "stop":
+    exit(status)
+
+setup_constants()
+
+os.system("echo 'Action: %s' >> /tmp/ra-env.log" % OCF_ACTION)
+os.system("env >> /tmp/ra-env.log")
+
+try:
+    if OCF_ACTION == "start":
+        status = qemu_colo_start()
+    elif OCF_ACTION == "stop":
+        status = qemu_colo_stop()
+    elif OCF_ACTION == "monitor":
+        status = qemu_colo_monitor()[0]
+    elif OCF_ACTION == "notify":
+        status = qemu_colo_notify()
+    elif OCF_ACTION == "promote":
+        status = qemu_colo_promote()
+    elif OCF_ACTION == "demote":
+        status = qemu_colo_demote()
+    elif OCF_ACTION == "validate-all":
+        status = qemu_colo_validate_all()
+    else:
+        status = OCF_ERR_UNIMPLEMENTED
+except Error:
+    exit(OCF_ERR_GENERIC)
+else:
+    exit(status)

--
2.20.1

#7 lore raw

PATCH 3/4 colo: Introduce high-level test

In reply to Lukas Straub

Add high-level test relying on the colo resource-agent to test all failover cases while checking guest network connectivity

---
 scripts/colo-resource-agent/crm_master |  44 +++
 tests/acceptance/colo.py               | 444 +++++++++++++++++++++++++
 2 files changed, 488 insertions(+)
 create mode 100755 scripts/colo-resource-agent/crm_master
 create mode 100644 tests/acceptance/colo.py
scripts/colo-resource-agent/crm_master new file +44 -0
new file mode 100755
index 0000000000..00c386b949
@@ -0,0 +1,44 @@
+#!/bin/bash
+
+# Fake crm_master for COLO testing
+#
+# Copyright (c) Lukas Straub <[email protected]>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+
+TMPDIR="$HA_RSCTMP"
+score=0
+query=0
+
+OPTIND=1
+while getopts 'Qql:Dv:N:G' opt; do
+    case "$opt" in
+        Q|q)
+            # Noop
+        ;;
+        "l")
+            # Noop
+        ;;
+        "D")
+            score=0
+        ;;
+        "v")
+            score=$OPTARG
+        ;;
+        "N")
+            TMPDIR="$COLO_SMOKE_REMOTE_TMP"
+        ;;
+        "G")
+            query=1
+        ;;
+    esac
+done
+
+if (( query )); then
+    cat "${TMPDIR}/master_score" || exit 1
+else
+    echo $score > "${TMPDIR}/master_score" || exit 1
+fi
+
+exit 0
tests/acceptance/colo.py new file +444 -0

444 lines not on this page · raw message

--
2.20.1

#8 lore raw

PATCH 4/4 MAINTAINERS: Add myself as maintainer for COLO resource agent

In reply to Lukas Straub

While I'm not going to have much time for this, I'll still try to test and review patches.

---
 MAINTAINERS | 6 ++++++
 1 file changed, 6 insertions(+)
MAINTAINERS +6 -0
index d6de200453..aad8356149 100644
@@ -2239,6 +2239,12 @@ F: net/colo*
 F: net/filter-rewriter.c
 F: net/filter-mirror.c
 
+COLO resource agent and testing
+M: Lukas Straub <[email protected]>
+S: Odd fixes
+F: scripts/colo-resource-agent/*
+F: tests/acceptance/colo.py
+
 Record/replay
 M: Pavel Dovgalyuk <[email protected]>
 R: Paolo Bonzini <[email protected]>

--
2.20.1

#9 lore raw
In reply to Lukas Straub
Quoted 10 lines · * Lukas Straub ([email protected]) wrote:

Hello Everyone,
These patches introduce a resource agent for use with the Pacemaker CRM and a high-level test utilizing it for testing qemu COLO.

The resource agent manages qemu COLO including continuous replication.

Currently the second test case (where the peer qemu is frozen) fails on primary failover, because qemu hangs while removing the replication related block nodes. Note that this also happens in real world test when cutting power to the peer host, so this needs to be fixed.

Do you understand why that happens? Is this it's trying to finish a read/write to the dead partner?

Dave

Quoted 19 lines

Based-on: <[email protected]> ([PATCH v7 0/4] colo: Add support for continuous replication)

Lukas Straub (4):
block/quorum.c: stable children names
colo: Introduce resource agent
colo: Introduce high-level test
MAINTAINERS: Add myself as maintainer for COLO resource agent

 MAINTAINERS                            |    6 +
 block/quorum.c                         |    6 +
 scripts/colo-resource-agent/colo       | 1026 ++++++++++++++++++++++++
 scripts/colo-resource-agent/crm_master |   44 +
 tests/acceptance/colo.py               |  444 ++++++++++
 5 files changed, 1526 insertions(+)
 create mode 100755 scripts/colo-resource-agent/colo
 create mode 100755 scripts/colo-resource-agent/crm_master
 create mode 100644 tests/acceptance/colo.py
Quoted 3 lines

--
2.20.1

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#10 lore raw
Quoted 12 lines · On Fri, 22 Nov 2019 09:46:46 +0000 "Dr. David Alan Gilbert"…
Quoted 10 lines · * Lukas Straub ([email protected]) wrote:

Hello Everyone,
These patches introduce a resource agent for use with the Pacemaker CRM and a high-level test utilizing it for testing qemu COLO.

The resource agent manages qemu COLO including continuous replication.

Currently the second test case (where the peer qemu is frozen) fails on primary failover, because qemu hangs while removing the replication related block nodes. Note that this also happens in real world test when cutting power to the peer host, so this needs to be fixed.

Quoted 4 lines

Do you understand why that happens? Is this it's trying to finish a read/write to the dead partner?

Dave

I haven't looked into it too closely yet, but it's often hanging in bdrv_flush() while removing the replication blockdev and of course thats probably because the nbd client waits for a reply. So I tried with the workaround below, which will actively kill the TCP connection and with it the test passes, though I haven't tested it in real world yet.

A proper solution to this would probably be a "force" parameter for blockdev-del, which skips all flushing and aborts all inflight io. Or we could add a timeout to the nbd client.

Regards,
Lukas Straub

scripts/colo-resource-agent/colo +1 -0
index 5fd9cfc0b5..62210af2a1 100755
@@ -935,6 +935,7 @@ def qemu_colo_notify():
            and HOSTNAME == str.strip(OCF_RESKEY_CRM_meta_notify_master_uname):
             fd = qmp_open()
             peer = qmp_get_nbd_remote(fd)
+            os.system("sudo ss -K dst %s dport = %s" % (peer, NBD_PORT))
             if peer == str.strip(OCF_RESKEY_CRM_meta_notify_stop_uname):
                 if qmp_check_resync(fd) != None:
                     qmp_cancel_resync(fd)
#11 lore raw
Quoted 18 lines · On Wed, 27 Nov 2019 22:11:34 +0100 Lukas Straub <[email protected]> wrote:
Quoted 15 lines · On Fri, 22 Nov 2019 09:46:46 +0000 "Dr. David Alan Gilbert"…
Quoted 10 lines · * Lukas Straub ([email protected]) wrote:

Hello Everyone,
These patches introduce a resource agent for use with the Pacemaker CRM and a high-level test utilizing it for testing qemu COLO.

The resource agent manages qemu COLO including continuous replication.

Currently the second test case (where the peer qemu is frozen) fails on primary failover, because qemu hangs while removing the replication related block nodes. Note that this also happens in real world test when cutting power to the peer host, so this needs to be fixed.

Do you understand why that happens? Is this it's trying to finish a read/write to the dead partner?

Quoted 8 lines
Quoted 1 line

Dave

I haven't looked into it too closely yet, but it's often hanging in bdrv_flush() while removing the replication blockdev and of course thats probably because the nbd client waits for a reply. So I tried with the workaround below, which will actively kill the TCP connection and with it the test passes, though I haven't tested it in real world yet.

In the real cluster, sometimes qemu even hangs while connecting to qmp (after remote poweroff). But I currently don't have the time to look into it.

Still a failing test is better than no test. Could we mark this test as known-bad and fix this issue later? How should I mark it as known-bad? By tag? Or warn in the log?

Regards,
Lukas Straub

#12 lore raw
Quoted 30 lines · * Lukas Straub ([email protected]) wrote:
Quoted 18 lines · On Wed, 27 Nov 2019 22:11:34 +0100 Lukas Straub <[email protected]> wrote:
Quoted 15 lines · On Fri, 22 Nov 2019 09:46:46 +0000 "Dr. David Alan Gilbert"…
Quoted 10 lines · * Lukas Straub ([email protected]) wrote:

Hello Everyone,
These patches introduce a resource agent for use with the Pacemaker CRM and a high-level test utilizing it for testing qemu COLO.

The resource agent manages qemu COLO including continuous replication.

Currently the second test case (where the peer qemu is frozen) fails on primary failover, because qemu hangs while removing the replication related block nodes. Note that this also happens in real world test when cutting power to the peer host, so this needs to be fixed.

Do you understand why that happens? Is this it's trying to finish a read/write to the dead partner?

Quoted 8 lines
Quoted 1 line

Dave

I haven't looked into it too closely yet, but it's often hanging in bdrv_flush() while removing the replication blockdev and of course thats probably because the nbd client waits for a reply. So I tried with the workaround below, which will actively kill the TCP connection and with it the test passes, though I haven't tested it in real world yet.

Quoted 2 lines

In the real cluster, sometimes qemu even hangs while connecting to qmp (after remote poweroff). But I currently don't have the time to look into it.

That doesn't surprise me too much; QMP is mostly handled in the main thread, as are a lot of other things; hanging in COLO has been my assumption for a while because of that. However, there's a way to fix it.

A while ago, Peter Xu added a feature called 'out of band' to QMP; you can open a QMP connection, set the OOB feature, and then commands that are marked as OOB are executed off the main thread on that connection.

At the moment we've just got the one real OOB command, 'migrate-recover' which is used for recovering postcopy from a similar failure to the COLO case.

To fix this you'd have to convert colo-lost-heartbeat to be an OOB command; note it's not that trivial, because you have to make sure the code that's run as part of the OOB command doesn't take any locks that could block on something in the main thread; so it can set flags, start new threads, perhaps call shutdown() on a socket; but it takes some thinking about.

Quoted 2 lines

Still a failing test is better than no test. Could we mark this test as known-bad and fix this issue later? How should I mark it as known-bad? By tag? Or warn in the log?

Not sure of that; cc'ing Maybe thuth knows?

Dave

Quoted 3 lines

Regards,
Lukas Straub

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK