Skip to contentSkip to conversation

Remus/Libxl: Network buffering support

PATCH Superseded Shriram Rajagopalan 3 participants 37 messages Oct 21, 2013 – Nov 14, 2013

v3 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

PATCH v3 1/1 Remus/Libxl: Network buffering support

This patch series adds support for network buffering in the Remus codebase in libxl.

Changes in V3:
[1/5] Fix redundant checks in configure scripts
(based on Ian Campbell's suggestions)

[2/5] Introduce locking in the script, during IFB setup.
Add xenstore paths used by netbuf scripts to xenstore-paths.markdown

[3/5] Hotplug scripts setup/teardown invocations are now asynchronous
following IanJ's feedback. However, the invocations are still sequential.

[5/5] Allow per-domain specification of netbuffer scripts in xl remus
commmand.

And minor nits throughout the series based on feedback from the last version

Changes in V2:
[1/5] Configure script will automatically enable/disable network
buffer support depending on the availability of the appropriate libnl3 version. [If libnl3 is unavailable, a warning message will be printed to let the user know that the feature has been disabled.]

use macros from pkg.m4 instead of pkg-config commands removed redundant checks for libnl3 libraries.

[3,4/5] - Minor nits.

Version 1:

[1/5] Changes to autoconf scripts to check for libnl3. Add linker flags
to libxl Makefile.

[2/5] External script to setup/teardown network buffering using libnl3's
CLI. This script will be invoked by libxl before starting Remus. The script's main job is to bring up an IFB device with plug qdisc attached to it. It then re-routes egress traffic from the guest's vif to the IFB device.

[3/5] Libxl code to invoke the external setup script, followed by netlink
related setup to obtain a handle on the output buffers attached to each vif.

[4/5] Libxl interaction with network buffer module in the kernel via
libnl3 API.

[5/5] xl cmdline switch to explicitly enable network buffering when
starting remus.

Few things to note:

a) Based on previous email discussions, the setup/teardown task has been moved to a hotplug style shell script which can be customized as desired, instead of implementing it as C code inside libxl.

b) Libnl3 is not available on NetBSD. Nor is it available on CentOS
(Linux). So I have made network buffering support an optional feature so that it can be disabled if desired.

c) NetBSD does not have libnl3. So I have put the setup script under tools/hotplug/Linux folder.

thanks
shriram

#1 lore raw

PATCH v3 1/1 remus: add libnl3 dependency to autoconf scripts

# HG changeset patch
# User Shriram Rajagopalan <[email protected]> # Date 1381540581 25200
# Node ID 357655a1598cba8f8460758714885563dd2b4a9c # Parent 808aba98084edee7b6dc78c7333da71d2be70386
remus: add libnl3 dependency to autoconf scripts

Libnl3 is required for controlling Remus network buffering. This patch adds dependency on libnl3 (>= 3.2.8) to autoconf scripts. Also provide ability to configure tools without libnl3 support, that is without network buffering support.

config/Tools.mk.in +3 -0
diff -r 808aba98084e -r 357655a1598c config/Tools.mk.in
@@ -37,6 +37,8 @@ PTHREAD_LIBS        := @PTHREAD_LIBS@
 
 PTYFUNCS_LIBS       := @PTYFUNCS_LIBS@
 
+LIBNL3_LIBS         := @LIBNL3_LIBS@
+LIBNL3_CFLAGS       := @LIBNL3_CFLAGS@
 # Download GIT repositories via HTTP or GIT's own protocol?
 # GIT's protocol is faster and more robust, when it works at all (firewalls
 # may block it). We make it the default, but if your GIT repository downloads
@@ -55,6 +57,7 @@ CONFIG_QEMU_TRAD    := @qemu_traditional
 CONFIG_QEMU_XEN     := @qemu_xen@
 CONFIG_XEND         := @xend@
 CONFIG_BLKTAP1      := @blktap1@
+CONFIG_REMUS_NETBUF := @remus_netbuf@
 
 #System options
 ZLIB                := @zlib@
tools/configure.ac +18 -0
diff -r 808aba98084e -r 357655a1598c tools/configure.ac
@@ -219,6 +219,24 @@ AC_SUBST(libiconv)
 # Checks for header files.
 AC_CHECK_HEADERS([yajl/yajl_version.h sys/eventfd.h])
 
+# Check for libnl3 >=3.2.8. If present enable remus network buffering.
+PKG_CHECK_MODULES(LIBNL3, [libnl-3.0 >= 3.2.8 libnl-route-3.0 >= 3.2.8],
+		[libnl3_lib="y"], [libnl3_lib="n"])
+
+# Check for libnl3 command line utilities
+PKG_CHECK_EXISTS([libnl-cli-3.0 >= 3.2.8], [libnl3_cli="y"], [libnl3_cli="n"])
+
+AS_IF([test "x$libnl3_lib" = "xn" || test "x$libnl3_cli" = "xn"], [
+	    AC_MSG_WARN([Disabling support for Remus network buffering.
+	    Please install libnl3 libraries, command line tools and devel
+	    headers - version 3.2.8 or higher])
+	    AC_SUBST(remus_netbuf, [n])
+	    ],[
+	    AC_SUBST(LIBNL3_LIBS)
+	    AC_SUBST(LIBNL3_CFLAGS)
+	    AC_SUBST(remus_netbuf, [y])
+])
+
 AC_OUTPUT()
 
 AS_IF([test "x$xend" = "xy" ], [
tools/libxl/Makefile +2 -0
diff -r 808aba98084e -r 357655a1598c tools/libxl/Makefile
@@ -21,11 +21,13 @@ endif
 
 LIBXL_LIBS =
 LIBXL_LIBS = $(LDLIBS_libxenctrl) $(LDLIBS_libxenguest) $(LDLIBS_libxenstore) $(LDLIBS_libblktapctl) $(PTYFUNCS_LIBS) $(LIBUUID_LIBS)
+LIBXL_LIBS += $(LIBNL3_LIBS)
 
 CFLAGS_LIBXL += $(CFLAGS_libxenctrl)
 CFLAGS_LIBXL += $(CFLAGS_libxenguest)
 CFLAGS_LIBXL += $(CFLAGS_libxenstore)
 CFLAGS_LIBXL += $(CFLAGS_libblktapctl) 
+CFLAGS_LIBXL += $(LIBNL3_CFLAGS)
 CFLAGS_LIBXL += -Wshadow
 
 CFLAGS += $(PTHREAD_CFLAGS)
tools/remus/README +6 -0
diff -r 808aba98084e -r 357655a1598c tools/remus/README
@@ -2,3 +2,9 @@ Remus provides fault tolerance for virtu
 checkpoints to a backup, which will activate if the target VM fails.
 
 See the website at http://nss.cs.ubc.ca/remus/ for details.
+
+Using Remus with libxl on Xen 4.4 and higher:
+ To enable network buffering, you need libnl 3.2.8
+ or higher along with the development headers and command line utilities.
+ If your distro does not have the appropriate libnl3 version, you can find
+ the latest source tarball of libnl3 at http://www.carisma.slowglass.com/~tgr/libnl/
#3 lore raw

PATCH v3 1/1 tools/hotplug: Remus network buffering setup scripts

In reply to Shriram Rajagopalan

# HG changeset patch
# User Shriram Rajagopalan <[email protected]> # Date 1382274772 25200
# Node ID 0001a8222a785865b753acf8bcdf97c10c9e3819 # Parent 357655a1598cba8f8460758714885563dd2b4a9c tools/hotplug: Remus network buffering setup scripts

This patch introduces remus-netbuf-setup hotplug script responsible for setting up and tearing down the necessary infrastructure required for network output buffering in Remus. This script is intended to be invoked by libxl for each guest interface, when starting or stopping Remus.

Apart from returning success/failure indication via the usual hotplug entries in xenstore, this script also writes to xenstore, the name of the IFB device to be used to control the vif's network output.

The script relies on libnl3 command line utilities to perform various setup/teardown functions. The script is confined to Linux platforms only since NetBSD does not seem to have libnl3.

docs/misc/xenstore-paths.markdown +4 -0
diff -r 357655a1598c -r 0001a8222a78 docs/misc/xenstore-paths.markdown
@@ -356,6 +356,10 @@ The guest's virtual time offset from UTC
 
 The device model version for a domain.
 
+#### /libxl/$DOMID/remus/netbuf/$DEVID/ifb = STRING [n,INTERNAL]
+
+IFB device used by Remus to buffer network output from the associated vif.
+
 [BLKIF]: http://xenbits.xen.org/docs/unstable/hypercall/include,public,io,blkif.h.html
 [FBIF]: http://xenbits.xen.org/docs/unstable/hypercall/include,public,io,fbif.h.html
 [HVMPARAMS]: http://xenbits.xen.org/docs/unstable/hypercall/include,public,hvm,params.h.html
tools/hotplug/Linux/Makefile +1 -0
diff -r 357655a1598c -r 0001a8222a78 tools/hotplug/Linux/Makefile
@@ -16,6 +16,7 @@ XEN_SCRIPTS += network-nat vif-nat
 XEN_SCRIPTS += vif-openvswitch
 XEN_SCRIPTS += vif2
 XEN_SCRIPTS += vif-setup
+XEN_SCRIPTS-$(CONFIG_REMUS_NETBUF) += remus-netbuf-setup
 XEN_SCRIPTS += block
 XEN_SCRIPTS += block-enbd block-nbd
 XEN_SCRIPTS-$(CONFIG_BLKTAP1) += blktap
tools/hotplug/Linux/remus-netbuf-setup new file +187 -0

187 lines not on this page · raw message

#8 lore raw

PATCH v3 1/1 tools/libxl: setup/teardown Remus network buffering

In reply to Shriram Rajagopalan

# HG changeset patch
# User Shriram Rajagopalan <[email protected]> # Date 1382295266 25200
# Node ID d3f088236c550213fc04ed982df47b4771b28d2f # Parent 0001a8222a785865b753acf8bcdf97c10c9e3819 tools/libxl: setup/teardown Remus network buffering

Setup/teardown remus network buffering for a given guest, when libxl_domain_remus_start API is invoked.

This patch does the following during setup:
a) call the hotplug script for each vif to setup its network buffer

b) establish a dedicated remus context containing libnl related state (netlink sockets, qdisc caches, etc.,)

c) Obtain handles to plug qdiscs installed on the IFB devices chosen by the hotplug scripts.

During teardown, the netlink resources are released, followed by invocation of hotplug scripts to remove the IFB devices.

tools/libxl/Makefile +7 -0
diff -r 0001a8222a78 -r d3f088236c55 tools/libxl/Makefile
@@ -42,6 +42,13 @@ LIBXL_OBJS-y += libxl_blktap2.o
 else
 LIBXL_OBJS-y += libxl_noblktap2.o
 endif
+
+ifeq ($(CONFIG_REMUS_NETBUF),y)
+LIBXL_OBJS-y += libxl_netbuffer.o
+else
+LIBXL_OBJS-y += libxl_nonetbuffer.o
+endif
+
 LIBXL_OBJS-$(CONFIG_X86) += libxl_cpuid.o libxl_x86.o
 LIBXL_OBJS-$(CONFIG_IA64) += libxl_nocpuid.o libxl_noarch.o
 LIBXL_OBJS-$(CONFIG_ARM) += libxl_nocpuid.o libxl_noarch.o
tools/libxl/libxl.c +35 -17

83 lines not on this page · raw message

tools/libxl/libxl_dom.c +59 -8

103 lines not on this page · raw message

tools/libxl/libxl_internal.h +45 -2

65 lines not on this page · raw message

tools/libxl/libxl_netbuffer.c new file +516 -0

516 lines not on this page · raw message

tools/libxl/libxl_nonetbuffer.c new file +56 -0

56 lines not on this page · raw message

tools/libxl/libxl_types.idl +2 -0
diff -r 0001a8222a78 -r d3f088236c55 tools/libxl/libxl_types.idl
@@ -539,6 +539,8 @@ libxl_domain_remus_info = Struct("domain
     ("interval",     integer),
     ("blackhole",    bool),
     ("compression",  bool),
+    ("netbuf",       bool),
+    ("netbufscript", string),
     ])
 
 libxl_event_type = Enumeration("event_type", [
#10 lore raw

PATCH v3 1/1 tools/libxl: Control network buffering in remus callbacks

In reply to Shriram Rajagopalan

# HG changeset patch
# User Shriram Rajagopalan <[email protected]> # Date 1382295546 25200
# Node ID a8deb9499e9dcce9869025fa1c02cf2e0d58612a # Parent d3f088236c550213fc04ed982df47b4771b28d2f tools/libxl: Control network buffering in remus callbacks

This patch constitutes the core network buffering logic. and does the following:
a) create a new network buffer when the domain is suspended (remus_domain_suspend_callback)
b) release the previous network buffer pertaining to the committed checkpoint (remus_domain_checkpoint_dm_saved)

tools/libxl/libxl_dom.c +55 -8
diff -r d3f088236c55 -r a8deb9499e9d tools/libxl/libxl_dom.c
@@ -1259,8 +1259,24 @@ void libxl__remus_teardown_done(libxl__e
 
 static int libxl__remus_domain_suspend_callback(void *data)
 {
-    /* REMUS TODO: Issue disk and network checkpoint reqs. */
-    return libxl__domain_suspend_common_callback(data);
+    /* REMUS TODO: Issue disk checkpoint reqs. */
+    libxl__save_helper_state *shs = data;
+    libxl__domain_suspend_state *dss = CONTAINER_OF(shs, *dss, shs);
+    libxl__remus_ctx *remus_ctx = dss->remus_ctx;
+    bool is_suspended;
+    STATE_AO_GC(dss->ao);
+
+    is_suspended = !!libxl__domain_suspend_common_callback(data);
+
+    if (!remus_ctx->netbuf_ctx) return is_suspended;
+
+    if (is_suspended) {
+        if (libxl__remus_netbuf_start_new_epoch(gc, dss->domid,
+                                                remus_ctx))
+            return !is_suspended;
+    }
+
+    return is_suspended;
 }
 
 static int libxl__remus_domain_resume_callback(void *data)
@@ -1273,7 +1289,7 @@ static int libxl__remus_domain_resume_ca
     if (libxl__domain_resume(gc, dss->domid, /* Fast Suspend */1))
         return 0;
 
-    /* REMUS TODO: Deal with disk. Start a new network output buffer */
+    /* REMUS TODO: Deal with disk. */
     return 1;
 }
 
@@ -1300,11 +1316,42 @@ static void libxl__remus_domain_checkpoi
 static void remus_checkpoint_dm_saved(libxl__egc *egc,
                                       libxl__domain_suspend_state *dss, int rc)
 {
-    /* REMUS TODO: Wait for disk and memory ack, release network buffer */
-    /* REMUS TODO: make this asynchronous */
-    assert(!rc); /* REMUS TODO handle this error properly */
-    usleep(dss->remus_ctx->interval * 1000);
-    libxl__xc_domain_saverestore_async_callback_done(egc, &dss->shs, 1);
+    /*
+     * REMUS TODO: Wait for disk and explicit memory ack (through restore
+     * callback from remote) before releasing network buffer.
+     */
+    libxl__remus_ctx *remus_ctx = dss->remus_ctx;
+    struct timespec epoch;
+    int do_next_iter = 0;
+    STATE_AO_GC(dss->ao);
+
+    if (rc) {
+        LOG(ERROR, "Failed to save device model. Terminating Remus..");
+        goto out;
+    }
+
+    if (remus_ctx->netbuf_ctx) {
+        rc = libxl__remus_netbuf_release_prev_epoch(gc, dss->domid,
+                                                     remus_ctx);
+        if (rc) {
+            LOG(ERROR, "Failed to release network buffer."
+                " Terminating Remus..");
+            goto out;
+        }
+    }
+
+    epoch.tv_sec = remus_ctx->interval / 1000; /* interval is in ms */
+    epoch.tv_nsec = remus_ctx->interval * 1000L * 1000L;
+    nanosleep(&epoch, 0);
+    /*
+     * Set return value to 1, so that the infinite checkpoint cycle
+     * continues. See xc_domain_save.c: xc_domain_save()
+     */
+    do_next_iter = 1;
+
+ out:
+    libxl__xc_domain_saverestore_async_callback_done(egc, &dss->shs,
+                                                     do_next_iter);
 }
 
 /*----- main code for suspending, in order of execution -----*/
#32 lore raw

PATCH v3 1/1 tools/xl: Remus - Network buffering cmdline switch

In reply to Shriram Rajagopalan

# HG changeset patch
# User Shriram Rajagopalan <[email protected]> # Date 1382295556 25200
# Node ID f4eea1e0ac3ed92ed17bbd74b0705743d20b302d # Parent a8deb9499e9dcce9869025fa1c02cf2e0d58612a tools/xl: Remus - Network buffering cmdline switch

Command line switch to 'xl remus' command, to enable network buffering. Pass on this flag to libxl so that it can act accordingly. Also update man pages to reflect the addition of a new option to 'xl remus' command.

docs/man/xl.conf.pod.5 +6 -0
diff -r a8deb9499e9d -r f4eea1e0ac3e docs/man/xl.conf.pod.5
@@ -105,6 +105,12 @@ Configures the default gateway device to
 
 Default: C<None>
 
+=item B<remus.default.netbufscript="PATH">
+
+Configures the default script used by Remus to setup network buffering.
+
+Default: C</etc/xen/scripts/remus-netbuf-setup>
+
 =item B<output_format="json|sxp">
 
 Configures the default output format used by xl when printing "machine
docs/man/xl.pod.1 +8 -2
diff -r a8deb9499e9d -r f4eea1e0ac3e docs/man/xl.pod.1
@@ -398,8 +398,7 @@ Print huge (!) amount of debug during th
 Enable Remus HA for domain. By default B<xl> relies on ssh as a transport
 mechanism between the two hosts.
 
-N.B: Remus support in xl is still in experimental (proof-of-concept) phase.
-     There is no support for network or disk buffering at the moment.
+N.B: There is no support for disk buffering at the moment.
 
 B<OPTIONS>
 
@@ -418,6 +417,13 @@ Generally useful for debugging.
 
 Disable memory checkpoint compression.
 
+=item B<-n>
+
+Enable network output buffering.  The default script used to configure
+network buffering is /etc/xen/scripts/remus-netbuf-setup. If you wish to
+use a custom script, set the global variable "remus.default.netbufscript"
+in /etc/xen/xl.conf to point to your script.
+
 =item B<-s> I<sshcommand>
 
 Use <sshcommand> instead of ssh.  String will be passed to sh.
tools/libxl/xl.c +4 -0
diff -r a8deb9499e9d -r f4eea1e0ac3e tools/libxl/xl.c
@@ -46,6 +46,7 @@ char *default_vifscript = NULL;
 char *default_bridge = NULL;
 char *default_gatewaydev = NULL;
 char *default_vifbackend = NULL;
+char *default_remus_netbufscript = NULL;
 enum output_format default_output_format = OUTPUT_FORMAT_JSON;
 int claim_mode = 1;
 
@@ -177,6 +178,9 @@ static void parse_global_config(const ch
     if (!xlu_cfg_get_long (config, "claim_mode", &l, 0))
         claim_mode = l;
 
+    xlu_cfg_replace_string (config, "remus.default.netbufscript",
+                            &default_remus_netbufscript, 0);
+
     xlu_cfg_destroy(config);
 }
 
tools/libxl/xl.h +1 -0
diff -r a8deb9499e9d -r f4eea1e0ac3e tools/libxl/xl.h
@@ -152,6 +152,7 @@ extern char *default_vifscript;
 extern char *default_bridge;
 extern char *default_gatewaydev;
 extern char *default_vifbackend;
+extern char *default_remus_netbufscript;
 extern char *blkdev_start;
 
 enum output_format {
tools/libxl/xl_cmdimpl.c +16 -7
diff -r a8deb9499e9d -r f4eea1e0ac3e tools/libxl/xl_cmdimpl.c
@@ -7111,8 +7111,9 @@ int main_remus(int argc, char **argv)
     r_info.interval = 200;
     r_info.blackhole = 0;
     r_info.compression = 1;
-
-    SWITCH_FOREACH_OPT(opt, "bui:s:e", NULL, "remus", 2) {
+    r_info.netbuf = 0;
+
+    SWITCH_FOREACH_OPT(opt, "buni:s:N:e", NULL, "remus", 2) {
     case 'i':
         r_info.interval = atoi(optarg);
         break;
@@ -7122,6 +7123,12 @@ int main_remus(int argc, char **argv)
     case 'u':
         r_info.compression = 0;
         break;
+    case 'n':
+        r_info.netbuf = 1;
+        break;
+    case 'N':
+        r_info.netbufscript = optarg;
+        break;
     case 's':
         ssh_command = optarg;
         break;
@@ -7133,6 +7140,11 @@ int main_remus(int argc, char **argv)
     domid = find_domain(argv[optind]);
     host = argv[optind + 1];
 
+    if (r_info.netbuf && !r_info.netbufscript) {
+        if (default_remus_netbufscript)
+            r_info.netbufscript = default_remus_netbufscript;
+    }
+
     if (r_info.blackhole) {
         send_fd = open("/dev/null", O_RDWR, 0644);
         if (send_fd < 0) {
@@ -7170,13 +7182,10 @@ int main_remus(int argc, char **argv)
     /* Point of no return */
     rc = libxl_domain_remus_start(ctx, &r_info, domid, send_fd, recv_fd, 0);
 
-    /* If we are here, it means backup has failed/domain suspend failed.
-     * Try to resume the domain and exit gracefully.
+    /* If we are here, it means remus setup/domain suspend/backup has
+     * failed. Try to resume the domain and exit gracefully.
      * TODO: Split-Brain check.
      */
-    fprintf(stderr, "remus sender: libxl_domain_suspend failed"
-            " (rc=%d)\n", rc);
-
     if (rc == ERROR_GUEST_TIMEDOUT)
         fprintf(stderr, "Failed to suspend domain at primary.\n");
     else {
tools/libxl/xl_cmdtable.c +3 -0
diff -r a8deb9499e9d -r f4eea1e0ac3e tools/libxl/xl_cmdtable.c
@@ -481,6 +481,9 @@ struct cmd_spec cmd_table[] = {
       "-i MS                   Checkpoint domain memory every MS milliseconds (def. 200ms).\n"
       "-b                      Replicate memory checkpoints to /dev/null (blackhole)\n"
       "-u                      Disable memory checkpoint compression.\n"
+      "-n                      Enable network output buffering.\n"
+      "-N <netbufscript>       Use netbufscript to setup network buffering instead of the\n"
+      "                        instead of the default (/etc/xen/scripts/remus-netbuf-setup).\n"
       "-s <sshcommand>         Use <sshcommand> instead of ssh.  String will be passed\n"
       "                        to sh. If empty, run <host> instead of \n"
       "                        ssh <host> xl migrate-receive -r [-e]\n"
#36 lore raw
In reply to Shriram Rajagopalan
Quoted 67 lines · Shriram Rajagopalan

This patch series adds support for network buffering in the Remus codebase in libxl.

Changes in V3:
[1/5] Fix redundant checks in configure scripts
(based on Ian Campbell's suggestions)

[2/5] Introduce locking in the script, during IFB setup.
Add xenstore paths used by netbuf scripts to xenstore-paths.markdown

[3/5] Hotplug scripts setup/teardown invocations are now asynchronous
following IanJ's feedback. However, the invocations are still sequential.

[5/5] Allow per-domain specification of netbuffer scripts in xl remus
commmand.

And minor nits throughout the series based on feedback from the last version

Changes in V2:
[1/5] Configure script will automatically enable/disable network
buffer support depending on the availability of the appropriate libnl3 version. [If libnl3 is unavailable, a warning message will be printed to let the user know that the feature has been disabled.]

use macros from pkg.m4 instead of pkg-config commands removed redundant checks for libnl3 libraries.

[3,4/5] - Minor nits.

Version 1:

[1/5] Changes to autoconf scripts to check for libnl3. Add linker flags
to libxl Makefile.

[2/5] External script to setup/teardown network buffering using libnl3's
CLI. This script will be invoked by libxl before starting Remus. The script's main job is to bring up an IFB device with plug qdisc attached to it. It then re-routes egress traffic from the guest's vif to the IFB device.

[3/5] Libxl code to invoke the external setup script, followed by netlink
related setup to obtain a handle on the output buffers attached to each vif.

[4/5] Libxl interaction with network buffer module in the kernel via
libnl3 API.

[5/5] xl cmdline switch to explicitly enable network buffering when
starting remus.

Few things to note:

a) Based on previous email discussions, the setup/teardown task has been moved to a hotplug style shell script which can be customized as desired, instead of implementing it as C code inside libxl.

b) Libnl3 is not available on NetBSD. Nor is it available on CentOS
(Linux). So I have made network buffering support an optional feature so that it can be disabled if desired.

c) NetBSD does not have libnl3. So I have put the setup script under tools/hotplug/Linux folder.

Quoted 8 lines

thanks
shriram

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

ping!

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

#2 lore raw

Re: [PATCH 1 of 5 V3] remus: add libnl3 dependency to autoconf scripts

In reply to Shriram Rajagopalan

tools/configure.ac AC_SUBST(libiconv) · line 226

Quoted 12 lines · Shriram Rajagopalan

diff -r 808aba98084e -r 357655a1598c tools/configure.ac

tools/configure.ac +5 -0
@@ -219,6 +219,24 @@ AC_SUBST(libiconv)
 # Checks for header files.
 AC_CHECK_HEADERS([yajl/yajl_version.h sys/eventfd.h])
 
+# Check for libnl3 >=3.2.8. If present enable remus network buffering.
+PKG_CHECK_MODULES(LIBNL3, [libnl-3.0 >= 3.2.8 libnl-route-3.0 >= 3.2.8],
+		[libnl3_lib="y"], [libnl3_lib="n"])
+
+# Check for libnl3 command line utilities

These cli utils are needed on the runtime host but not the build host, right? (for the benefit of the hotplug scripts I suppose?)

This check only runs on the build host, so it will fail needlessly if the tools aren't there while at the same time not catching the case where they are missing from the runtime host.

Best bet is to omit this check and simply list them in the README, in the list of optional tools (and reference tools/remus/README for more info if you like).

Ian.

#4 lore raw

Re: [PATCH 2 of 5 V3] tools/hotplug: Remus network buffering setup scripts

In reply to Shriram Rajagopalan

tools/hotplug/Linux/remus-netbuf-setup line 76

Quoted 11 lines · Shriram Rajagopalan
tools/hotplug/Linux/remus-netbuf-setup new file +11 -0
+check_libnl_tools() {
+    if ! command -v nl-qdisc-list > /dev/null 2>&1; then
+        fatal "Unable to find nl-qdisc-list tool"
+    fi
+    if ! command -v nl-qdisc-add > /dev/null 2>&1; then
+        fatal "Unable to find nl-qdisc-add tool"
+    fi
+    if ! command -v nl-qdisc-delete > /dev/null 2>&1; then
+        fatal "Unable to find nl-qdisc-delete tool"
+    fi
+}

This probably suffices in place of the configure check I commented on earlier.

tools/hotplug/Linux/remus-netbuf-setup line 136

Quoted 12 lines
tools/hotplug/Linux/remus-netbuf-setup new file +12 -0
+add_plug_qdisc() {
+    local vif=$1
+    local ifb=$2
+
+    nl-qdisc-add --dev="$ifb" --parent root plug >/dev/null 2>&1
+    if [ $? -ne 0 ]
+    then
+	do_without_error tc qdisc del dev "$vif" ingress
+	fatal "Failed to add plug qdisc to $ifb"
+    fi
+
+    #set ifb buffering limit in bytes. Its okay if this command

... "fails" ?

Quoted 4 lines
tools/hotplug/Linux/remus-netbuf-setup new file +4 -0
+
+case "$command" in
+    setup)
+	check_libnl_tools

tools/hotplug/Linux/remus-netbuf-setup line 173

Quoted 10 lines
tools/hotplug/Linux/remus-netbuf-setup new file +10 -0
+	check_modules
+
+	claim_lock "pickifb"
+	setup_ifb
+	redirect_vif_traffic "$vifname" "$IFB"
+	add_plug_qdisc "$vifname" "$IFB"
+	release_lock "pickifb"
+
+	#not using xenstore_write that automatically exits on error
+	# because we need to cleanup

whitespace inconsistency.

tools/hotplug/Linux/remus-netbuf-setup line 177

Quoted 4 lines
tools/hotplug/Linux/remus-netbuf-setup new file +4 -0
+	_xenstore_write "$XENBUS_PATH/ifb" "$IFB" || xs_write_failed "$vifname" "$IFB"
+	;;
+    teardown)
+	: ${IFB?}

Do you mean log debug or something here?

tools/hotplug/Linux/remus-netbuf-setup line 187

Quoted 10 lines
tools/hotplug/Linux/remus-netbuf-setup new file +10 -0
+	teardown_netbuf "$vifname" "$IFB"
+	;;
+esac
+
+log debug "Successful remus-netbuf-setup $command for $vifname, ifb $IFB."
+
+if [ "$command" = "setup" ]
+then
+  success
+fi

Why not put this in the case for setup?

Ian.

#9 lore raw

Re: [PATCH 3 of 5 V3] tools/libxl: setup/teardown Remus network buffering

In reply to Shriram Rajagopalan
Quoted 16 lines · Shriram Rajagopalan

# HG changeset patch
# User Shriram Rajagopalan <[email protected]> # Date 1382295266 25200
# Node ID d3f088236c550213fc04ed982df47b4771b28d2f # Parent 0001a8222a785865b753acf8bcdf97c10c9e3819 tools/libxl: setup/teardown Remus network buffering

Setup/teardown remus network buffering for a given guest, when libxl_domain_remus_start API is invoked.

This patch does the following during setup:
a) call the hotplug script for each vif to setup its network buffer

b) establish a dedicated remus context containing libnl related state (netlink sockets, qdisc caches, etc.,)

Quoted 7 lines

c) Obtain handles to plug qdiscs installed on the IFB devices chosen by the hotplug scripts.

During teardown, the netlink resources are released, followed by invocation of hotplug scripts to remove the IFB devices.

All this async stuff needs Ian J's eye on it really, not mine. It looks vaguely plausible to my untrained eye.

tools/libxl/Makefile LIBXL_OBJS-y += libxl_blktap2.o · line 47

Quoted 10 lines

diff -r 0001a8222a78 -r d3f088236c55 tools/libxl/Makefile

tools/libxl/Makefile +3 -0
@@ -42,6 +42,13 @@ LIBXL_OBJS-y += libxl_blktap2.o
 else
 LIBXL_OBJS-y += libxl_noblktap2.o
 endif
+
+ifeq ($(CONFIG_REMUS_NETBUF),y)
+LIBXL_OBJS-y += libxl_netbuffer.o

Should the addition of the libnl stuff to LIBS and CFLAGS have been under here too?

tools/libxl/libxl_types.idl libxl_domain_remus_info = Struct("domain · line 542

Quoted 8 lines

diff -r 0001a8222a78 -r d3f088236c55 tools/libxl/libxl_types.idl

tools/libxl/libxl_types.idl +1 -0
@@ -539,6 +539,8 @@ libxl_domain_remus_info = Struct("domain
     ("interval",     integer),
     ("blackhole",    bool),
     ("compression",  bool),
+    ("netbuf",       bool),

I guess libxl_defbool doesn't make sense here.

tools/libxl/libxl_types.idl libxl_domain_remus_info = Struct("domain · line 543

Quoted 1 line
tools/libxl/libxl_types.idl +1 -0
+    ("netbufscript", string),

This requires a LIBXL_HAVE #define in libxl.h so users know it is available. One for the overall netbuf feature should do I think.

Ian.

#11 lore raw

Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks

In reply to Shriram Rajagopalan
Quoted 7 lines · Shriram Rajagopalan

# HG changeset patch
# User Shriram Rajagopalan <[email protected]> # Date 1382295546 25200
# Node ID a8deb9499e9dcce9869025fa1c02cf2e0d58612a # Parent d3f088236c550213fc04ed982df47b4771b28d2f tools/libxl: Control network buffering in remus callbacks

Quoted 8 lines

This patch constitutes the core network buffering logic. and does the following:
a) create a new network buffer when the domain is suspended (remus_domain_suspend_callback)
b) release the previous network buffer pertaining to the committed checkpoint (remus_domain_checkpoint_dm_saved)

Another one where Ian J's opinion matters more than mine, but it looks ok to me.

Ian.

#33 lore raw

Re: [PATCH 5 of 5 V3] tools/xl: Remus - Network buffering cmdline switch

In reply to Shriram Rajagopalan
Quoted 14 lines · Shriram Rajagopalan

diff -r a8deb9499e9d -r f4eea1e0ac3e docs/man/xl.pod.1

docs/man/xl.pod.1 +1 -2
@@ -398,8 +398,7 @@ Print huge (!) amount of debug during th
 Enable Remus HA for domain. By default B<xl> relies on ssh as a transport
 mechanism between the two hosts.
 
-N.B: Remus support in xl is still in experimental (proof-of-concept) phase.
-     There is no support for network or disk buffering at the moment.
+N.B: There is no support for disk buffering at the moment.
 
 B<OPTIONS>
 

docs/man/xl.pod.1 Generally useful for debugging. · line 425

Quoted 9 lines
docs/man/xl.pod.1 +6 -0
@@ -418,6 +417,13 @@ Generally useful for debugging.
 
 Disable memory checkpoint compression.
 
+=item B<-n>
+
+Enable network output buffering.  The default script used to configure
+network buffering is /etc/xen/scripts/remus-netbuf-setup. If you wish to
+use a custom script, set the global variable "remus.default.netbufscript"
+in /etc/xen/xl.conf to point to your script.

No docs for -N ?

docs/man/xl.pod.1 int main_remus(int argc, char **argv) · line 7143

Quoted 5 lines
docs/man/xl.pod.1 +1 -0
@@ -7133,6 +7140,11 @@ int main_remus(int argc, char **argv)
     domid = find_domain(argv[optind]);
     host = argv[optind + 1];
 
+    if (r_info.netbuf && !r_info.netbufscript) {

Will the code do anything if netbuf==False but netbufscript!=NULL?

docs/man/xl.pod.1 int main_remus(int argc, char **argv) · line 7144

Quoted 1 line
docs/man/xl.pod.1 +1 -0
+        if (default_remus_netbufscript)

This if is redundant isn't it, it just avoids assigning NULL to a variable which is already NULL.

Given the above two comments this whole block could be
if (!r_info.netbufscript)
r_info.netbufscript = default...
[...]

docs/man/xl.pod.1 int main_remus(int argc, char **argv) · line 7137

Quoted 2 lines
docs/man/xl.pod.1 +0 -2
-    fprintf(stderr, "remus sender: libxl_domain_suspend failed"
-            " (rc=%d)\n", rc);

Was this removed on purpose? It seemed useful looking and wasn't mentioned in the commit message.

Ian.

#5 lore raw

Re: [PATCH 2 of 5 V3] tools/hotplug: Remus network buffering setup scripts

In reply to Ian Campbell
Quoted 8 lines · Ian Campbell
Quoted 8 lines
tools/hotplug/Linux/remus-netbuf-setup new file +8 -0
+
+case "$command" in
+    setup)
+     check_libnl_tools
+     check_modules
+
+     claim_lock "pickifb"
+     setup_ifb
Quoted 10 lines

tools/hotplug/Linux/remus-netbuf-setup line 173

Quoted 6 lines
tools/hotplug/Linux/remus-netbuf-setup new file +6 -0
+     redirect_vif_traffic "$vifname" "$IFB"
+     add_plug_qdisc "$vifname" "$IFB"
+     release_lock "pickifb"
+
+     #not using xenstore_write that automatically exits on error
+     # because we need to cleanup

whitespace inconsistency.

I seem to get this wrong again and again..
Did you mean the space trailing the second # ? or should the code block inside setup/teardown
be indented with two tabs instead of one ?

Quoted 9 lines
Quoted 1 line

+ _xenstore_write "$XENBUS_PATH/ifb" "$IFB" || xs_write_failed

"$vifname" "$IFB"

tools/hotplug/Linux/remus-netbuf-setup line 177

Quoted 3 lines
tools/hotplug/Linux/remus-netbuf-setup new file +3 -0
+     ;;
+    teardown)
+     : ${IFB?}

Do you mean log debug or something here?

This was to just make sure that the IFB variable was supplied as part of the environment..
Just like the two checks on top of this script..
"
: ${vifname?}
: ${XENBUS_PATH?}
"

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

#34 lore raw

Re: [PATCH 5 of 5 V3] tools/xl: Remus - Network buffering cmdline switch

In reply to Ian Campbell
Quoted 47 lines · Ian Campbell

docs/man/xl.pod.1 Sun Oct 20 11:59:16 2013 -0700 Print huge (!) amount of debug during th · line 398

Quoted 5 lines · Shriram Rajagopalan

diff -r a8deb9499e9d -r f4eea1e0ac3e docs/man/xl.pod.1

docs/man/xl.pod.1 Sun Oct 20 11:59:16 2013 -0700 renamed from docs/man/xl.pod.1 Sun Oct 20 11:59:06 2013 -0700 +0 -0
@@ -398,8 +398,7 @@ Print huge (!) amount of debug during th
 Enable Remus HA for domain. By default B<xl> relies on ssh as a

transport

docs/man/xl.pod.1 Sun Oct 20 11:59:16 2013 -0700 Print huge (!) amount of debug during th · line 401

Quoted 3 lines
docs/man/xl.pod.1 Sun Oct 20 11:59:16 2013 -0700 renamed from docs/man/xl.pod.1 Sun Oct 20 11:59:06 2013 -0700 +0 -1
 mechanism between the two hosts.
 
-N.B: Remus support in xl is still in experimental (proof-of-concept)

phase.

Quoted 6 lines
docs/man/xl.pod.1 Sun Oct 20 11:59:16 2013 -0700 renamed from docs/man/xl.pod.1 Sun Oct 20 11:59:06 2013 -0700 +1 -1
-     There is no support for network or disk buffering at the moment.
+N.B: There is no support for disk buffering at the moment.
 
 B<OPTIONS>
 

docs/man/xl.pod.1 Sun Oct 20 11:59:16 2013 -0700 Generally useful for debugging. · line 424

Quoted 8 lines
docs/man/xl.pod.1 Sun Oct 20 11:59:16 2013 -0700 renamed from docs/man/xl.pod.1 Sun Oct 20 11:59:06 2013 -0700 +5 -0
@@ -418,6 +417,13 @@ Generally useful for debugging.
 
 Disable memory checkpoint compression.
 
+=item B<-n>
+
+Enable network output buffering.  The default script used to configure
+network buffering is /etc/xen/scripts/remus-netbuf-setup. If you wish to
+use a custom script, set the global variable

"remus.default.netbufscript"

docs/man/xl.pod.1 Sun Oct 20 11:59:16 2013 -0700 Generally useful for debugging. · line 425

Quoted 1 line
docs/man/xl.pod.1 Sun Oct 20 11:59:16 2013 -0700 renamed from docs/man/xl.pod.1 Sun Oct 20 11:59:06 2013 -0700 +1 -0
+in /etc/xen/xl.conf to point to your script.

No docs for -N ?

docs/man/xl.pod.1 Sun Oct 20 11:59:16 2013 -0700 int main_remus(int argc, char **argv) · line 7143

Quoted 5 lines
docs/man/xl.pod.1 Sun Oct 20 11:59:16 2013 -0700 renamed from docs/man/xl.pod.1 Sun Oct 20 11:59:06 2013 -0700 +1 -0
@@ -7133,6 +7140,11 @@ int main_remus(int argc, char **argv)
     domid = find_domain(argv[optind]);
     host = argv[optind + 1];
 
+    if (r_info.netbuf && !r_info.netbufscript) {

Will the code do anything if netbuf==False but netbufscript!=NULL?

docs/man/xl.pod.1 Sun Oct 20 11:59:16 2013 -0700 int main_remus(int argc, char **argv) · line 7144

Quoted 1 line
docs/man/xl.pod.1 Sun Oct 20 11:59:16 2013 -0700 renamed from docs/man/xl.pod.1 Sun Oct 20 11:59:06 2013 -0700 +1 -0
+        if (default_remus_netbufscript)

This if is redundant isn't it, it just avoids assigning NULL to a variable which is already NULL.

Given the above two comments this whole block could be
if (!r_info.netbufscript)
r_info.netbufscript = default...
[...]

Quoted 7 lines

docs/man/xl.pod.1 Sun Oct 20 11:59:16 2013 -0700 int main_remus(int argc, char **argv) · line 7137

Quoted 2 lines
docs/man/xl.pod.1 Sun Oct 20 11:59:16 2013 -0700 renamed from docs/man/xl.pod.1 Sun Oct 20 11:59:06 2013 -0700 +0 -2
-    fprintf(stderr, "remus sender: libxl_domain_suspend failed"
-            " (rc=%d)\n", rc);

Was this removed on purpose? It seemed useful looking and wasn't mentioned in the commit message.

Yes, because the error message was not representative. That code block currently looks like this (after removing the fprintf), which is bit more informative.

/* If we are here, it means remus setup/domain suspend/backup has

     * failed. Try to resume the domain and exit gracefully.


     * TODO: Split-Brain check.


     */
    if (rc == ERROR_GUEST_TIMEDOUT)
        fprintf(stderr, "Failed to suspend domain at primary.\n");
    else {
        fprintf(stderr, "Remus: Backup failed? resuming domain at
primary.\n");
        libxl_domain_resume(ctx, domid, 1, 0);
    }
Quoted 5 lines

Ian.

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

#6 lore raw

Re: [PATCH 2 of 5 V3] tools/hotplug: Remus network buffering setup scripts

In reply to Shriram Rajagopalan
Quoted 25 lines · Shriram Rajagopalan

On Thu, Oct 31, 2013 at 1:21 PM, Ian Campbell
<[email protected]> wrote:
> +

> +case "$command" in
> + setup)
> + check_libnl_tools
> + check_modules
> +
> + claim_lock "pickifb"
> + setup_ifb
> + redirect_vif_traffic "$vifname" "$IFB"
> + add_plug_qdisc "$vifname" "$IFB"
> + release_lock "pickifb"
> +
> + #not using xenstore_write that automatically exits on
error
> + # because we need to cleanup

whitespace inconsistency.

Quoted 4 lines

I seem to get this wrong again and again..
Did you mean the space trailing the second # ? or should the code block inside setup/teardown
be indented with two tabs instead of one ?

I meant "#foo" (1st line) vs "# foo" (2nd line) which just caught my eye. They should be the same. More normal would be a space after the # I think.

I don't think we have a particularly strict coding style for shell scripts, but code being consistent with itself is a good start.

Quoted 15 lines

> + _xenstore_write "$XENBUS_PATH/ifb" "$IFB" ||
xs_write_failed "$vifname" "$IFB"
> + ;;
> + teardown)
> + : ${IFB?}

Do you mean log debug or something here?

Quoted 7 lines

This was to just make sure that the IFB variable was supplied as part of the environment..
Just like the two checks on top of this script..
"
: ${vifname?}
: ${XENBUS_PATH?}
"

Do these result in useful error reporting to the end user? Bearing in mind that the end user might be using libxl but not xl and therefore not see stdout/err (which might be going into some daemon's log file).

Also, either I'm missing it or the bash manpage only documents ${parameter:?} and not ${parameter?}. Are both actually valid?

Ian.

#35 lore raw

Re: [PATCH 5 of 5 V3] tools/xl: Remus - Network buffering cmdline switch

In reply to Shriram Rajagopalan
Quoted 4 lines · Shriram Rajagopalan

Yes, because the error message was not representative. That code block currently looks like this (after removing the fprintf), which is bit more informative.

Makes sense, thanks.

Quoted 31 lines

/* If we are here, it means remus setup/domain suspend/backup has

     * failed. Try to resume the domain and exit gracefully.


           
     * TODO: Split-Brain check.


          
     */
    if (rc == ERROR_GUEST_TIMEDOUT)
        fprintf(stderr, "Failed to suspend domain at primary.\n");
    else {
        fprintf(stderr, "Remus: Backup failed? resuming domain at
primary.\n");
        libxl_domain_resume(ctx, domid, 1, 0);
    }


 
        Ian.
#12 lore raw

Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks

In reply to Shriram Rajagopalan

Shriram Rajagopalan writes ("[PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks"):

Quoted 1 line

tools/libxl: Control network buffering in remus callbacks

...

tools/libxl/libxl_dom.c void libxl__remus_teardown_done(libxl__e · line 1269

Quoted 12 lines

static int libxl__remus_domain_suspend_callback(void *data) {

tools/libxl/libxl_dom.c +8 -2
-    /* REMUS TODO: Issue disk and network checkpoint reqs. */
-    return libxl__domain_suspend_common_callback(data);
+    /* REMUS TODO: Issue disk checkpoint reqs. */
+    libxl__save_helper_state *shs = data;
+    libxl__domain_suspend_state *dss = CONTAINER_OF(shs, *dss, shs);
+    libxl__remus_ctx *remus_ctx = dss->remus_ctx;
+    bool is_suspended;
+    STATE_AO_GC(dss->ao);
+
+    is_suspended = !!libxl__domain_suspend_common_callback(data);

I think this function would be less confusing if the return value variable was called "ok" or something similar. It's really just an error flag.

Also AFAICT the !! has no function here. The return value is essentially a boolean.

tools/libxl/libxl_dom.c void libxl__remus_teardown_done(libxl__e · line 1276

Quoted 6 lines
tools/libxl/libxl_dom.c +6 -0
+    if (!remus_ctx->netbuf_ctx) return is_suspended;
+
+    if (is_suspended) {
+        if (libxl__remus_netbuf_start_new_epoch(gc, dss->domid,
+                                                remus_ctx))
+            return !is_suspended;

This construction is rather odd. Why return !is_suspended when you know that !!is_suspended ?

Quoted 6 lines
tools/libxl/libxl_dom.c +0 -2
@@ -1300,11 +1316,42 @@ static void libxl__remus_domain_checkpoi
 static void remus_checkpoint_dm_saved(libxl__egc *egc,
                                       libxl__domain_suspend_state *dss, int rc)
 {
-    /* REMUS TODO: Wait for disk and memory ack, release network buffer */
-    /* REMUS TODO: make this asynchronous */

tools/libxl/libxl_dom.c static void libxl__remus_domain_checkpoi · line 1325

Quoted 10 lines
tools/libxl/libxl_dom.c +7 -3
-    assert(!rc); /* REMUS TODO handle this error properly */
-    usleep(dss->remus_ctx->interval * 1000);
-    libxl__xc_domain_saverestore_async_callback_done(egc, &dss->shs, 1);
+    /*
+     * REMUS TODO: Wait for disk and explicit memory ack (through restore
+     * callback from remote) before releasing network buffer.
+     */
+    libxl__remus_ctx *remus_ctx = dss->remus_ctx;
+    struct timespec epoch;
+    int do_next_iter = 0;

Again, this variable is probably best called "ok".

tools/libxl/libxl_dom.c static void libxl__remus_domain_checkpoi · line 1345

Quoted 3 lines
tools/libxl/libxl_dom.c +3 -0
+    epoch.tv_sec = remus_ctx->interval / 1000; /* interval is in ms */
+    epoch.tv_nsec = remus_ctx->interval * 1000L * 1000L;
+    nanosleep(&epoch, 0);

This is no good, I'm afraid. You should be using a libxl event loop timer. (Also why are you changing your existing usleep to this new nanosleep which seems functionally very similar?)

Ian.

#13 lore raw

Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks

Quoted 60 lines · Ian Jackson

Shriram Rajagopalan writes ("[PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks"):

Quoted 1 line

tools/libxl: Control network buffering in remus callbacks

...

tools/libxl/libxl_dom.c void libxl__remus_teardown_done(libxl__e · line 1269

Quoted 12 lines

static int libxl__remus_domain_suspend_callback(void *data) {

tools/libxl/libxl_dom.c +8 -2
-    /* REMUS TODO: Issue disk and network checkpoint reqs. */
-    return libxl__domain_suspend_common_callback(data);
+    /* REMUS TODO: Issue disk checkpoint reqs. */
+    libxl__save_helper_state *shs = data;
+    libxl__domain_suspend_state *dss = CONTAINER_OF(shs, *dss, shs);
+    libxl__remus_ctx *remus_ctx = dss->remus_ctx;
+    bool is_suspended;
+    STATE_AO_GC(dss->ao);
+
+    is_suspended = !!libxl__domain_suspend_common_callback(data);

I think this function would be less confusing if the return value variable was called "ok" or something similar. It's really just an error flag.

Also AFAICT the !! has no function here. The return value is essentially a boolean.

tools/libxl/libxl_dom.c void libxl__remus_teardown_done(libxl__e · line 1276

Quoted 6 lines
tools/libxl/libxl_dom.c +6 -0
+    if (!remus_ctx->netbuf_ctx) return is_suspended;
+
+    if (is_suspended) {
+        if (libxl__remus_netbuf_start_new_epoch(gc, dss->domid,
+                                                remus_ctx))
+            return !is_suspended;

This construction is rather odd. Why return !is_suspended when you know that !!is_suspended ?

Review static void libxl__remus_domain_checkpoi · line 1317

Quoted 3 lines
(quoted diff) +0 -0
@@ -1300,11 +1316,42 @@ static void libxl__remus_domain_checkpoi
 static void remus_checkpoint_dm_saved(libxl__egc *egc,
                                       libxl__domain_suspend_state *dss,

int rc)

Review static void libxl__remus_domain_checkpoi · line 1303

Quoted 2 lines
(quoted diff) +0 -1
 {
-    /* REMUS TODO: Wait for disk and memory ack, release network buffer

*/

tools/libxl/libxl_dom.c static void libxl__remus_domain_checkpoi

Quoted 6 lines
tools/libxl/libxl_dom.c +2 -4
-    /* REMUS TODO: make this asynchronous */
-    assert(!rc); /* REMUS TODO handle this error properly */
-    usleep(dss->remus_ctx->interval * 1000);
-    libxl__xc_domain_saverestore_async_callback_done(egc, &dss->shs, 1);
+    /*
+     * REMUS TODO: Wait for disk and explicit memory ack (through

restore

tools/libxl/libxl_dom.c static void libxl__remus_domain_checkpoi · line 1325

Quoted 5 lines
tools/libxl/libxl_dom.c +5 -0
+     * callback from remote) before releasing network buffer.
+     */
+    libxl__remus_ctx *remus_ctx = dss->remus_ctx;
+    struct timespec epoch;
+    int do_next_iter = 0;

Again, this variable is probably best called "ok".

tools/libxl/libxl_dom.c static void libxl__remus_domain_checkpoi · line 1345

Quoted 3 lines
tools/libxl/libxl_dom.c +3 -0
+    epoch.tv_sec = remus_ctx->interval / 1000; /* interval is in ms */
+    epoch.tv_nsec = remus_ctx->interval * 1000L * 1000L;
+    nanosleep(&epoch, 0);
Quoted 7 lines

This is no good, I'm afraid. You should be using a libxl event loop timer. (Also why are you changing your existing usleep to this new nanosleep which seems functionally very similar?)

Ian.

Nanosleep was more of my personal preference, just as the '!!' above was Andrew's suggestion, due to the fact that libxl__domain_suspend_common_callback()
does not strictly return boolean. But I agree that renaming that variable makes more sense.

As far as using the event loop timers, I couldn't find anything suitable in libxl_event.h/c that
I could use instead of usleep/nanosleep. Specifically, they were all asynchronous, while the
caller of these functions is libxc's xc_domain_save infinite loop (synchronous).

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

#31 lore raw

Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks

In reply to Ian Jackson
Quoted 31 lines · Ian Jackson

Shriram Rajagopalan writes ("[PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks"):

Quoted 1 line

tools/libxl: Control network buffering in remus callbacks

...

tools/libxl/libxl_dom.c void libxl__remus_teardown_done(libxl__e · line 1269

Quoted 12 lines

static int libxl__remus_domain_suspend_callback(void *data) {

tools/libxl/libxl_dom.c +8 -2
-    /* REMUS TODO: Issue disk and network checkpoint reqs. */
-    return libxl__domain_suspend_common_callback(data);
+    /* REMUS TODO: Issue disk checkpoint reqs. */
+    libxl__save_helper_state *shs = data;
+    libxl__domain_suspend_state *dss = CONTAINER_OF(shs, *dss, shs);
+    libxl__remus_ctx *remus_ctx = dss->remus_ctx;
+    bool is_suspended;
+    STATE_AO_GC(dss->ao);
+
+    is_suspended = !!libxl__domain_suspend_common_callback(data);

I think this function would be less confusing if the return value variable was called "ok" or something similar. It's really just an error flag.

Also AFAICT the !! has no function here. The return value is essentially a boolean.

tools/libxl/libxl_dom.c void libxl__remus_teardown_done(libxl__e · line 1276

Quoted 6 lines
tools/libxl/libxl_dom.c +6 -0
+    if (!remus_ctx->netbuf_ctx) return is_suspended;
+
+    if (is_suspended) {
+        if (libxl__remus_netbuf_start_new_epoch(gc, dss->domid,
+                                                remus_ctx))
+            return !is_suspended;
Quoted 4 lines

This construction is rather odd. Why return !is_suspended when you know that !!is_suspended ?

If a new network buffer cannot be created (for the next epoch), then return an error. This ends up terminating the checkpointing process.

Quoted 26 lines

Review static void libxl__remus_domain_checkpoi · line 1317

Quoted 3 lines
(quoted diff) +0 -0
@@ -1300,11 +1316,42 @@ static void libxl__remus_domain_checkpoi
 static void remus_checkpoint_dm_saved(libxl__egc *egc,
                                       libxl__domain_suspend_state *dss,

int rc)

Review static void libxl__remus_domain_checkpoi · line 1303

Quoted 2 lines
(quoted diff) +0 -1
 {
-    /* REMUS TODO: Wait for disk and memory ack, release network buffer

*/

tools/libxl/libxl_dom.c static void libxl__remus_domain_checkpoi

Quoted 6 lines
tools/libxl/libxl_dom.c +2 -4
-    /* REMUS TODO: make this asynchronous */
-    assert(!rc); /* REMUS TODO handle this error properly */
-    usleep(dss->remus_ctx->interval * 1000);
-    libxl__xc_domain_saverestore_async_callback_done(egc, &dss->shs, 1);
+    /*
+     * REMUS TODO: Wait for disk and explicit memory ack (through

restore

tools/libxl/libxl_dom.c static void libxl__remus_domain_checkpoi · line 1325

Quoted 5 lines
tools/libxl/libxl_dom.c +5 -0
+     * callback from remote) before releasing network buffer.
+     */
+    libxl__remus_ctx *remus_ctx = dss->remus_ctx;
+    struct timespec epoch;
+    int do_next_iter = 0;

Again, this variable is probably best called "ok".

tools/libxl/libxl_dom.c static void libxl__remus_domain_checkpoi · line 1345

Quoted 3 lines
tools/libxl/libxl_dom.c +3 -0
+    epoch.tv_sec = remus_ctx->interval / 1000; /* interval is in ms */
+    epoch.tv_nsec = remus_ctx->interval * 1000L * 1000L;
+    nanosleep(&epoch, 0);
Quoted 7 lines

This is no good, I'm afraid. You should be using a libxl event loop timer. (Also why are you changing your existing usleep to this new nanosleep which seems functionally very similar?)

Ian.

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

#14 lore raw

Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks [and 1 more messages]

In reply to Shriram Rajagopalan

Shriram Rajagopalan writes ("Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks"):

Quoted 1 line

Nanosleep was more of my personal preference,

I don't think that's a good enough reason for the churn, but as I say this really ought to be replaced with use of a timeout event.

Quoted 3 lines

just as the '!!' above was Andrew's suggestion, due to the fact that libxl__domain_suspend_common_callback() does not strictly return boolean.

Are you looking at a different tree to me ? I have checked the return statements in libxl__domain_suspend_common_callback and they are all "return 0" or "return 1".

Quoted 1 line

But I agree that renaming that variable makes more sense.

Good :-). Thanks.

Quoted 5 lines

As far as using the event loop timers, I couldn't find anything suitable in libxl_event.h/c that I could use instead of usleep/nanosleep. Specifically, they were all asynchronous, while the caller of these functions is libxc's xc_domain_save infinite loop (synchronous).

No, you need to make it an asynchronous call.

The libxl save helper machinery arranges to run xc_domain_save in a subprocess so that the main program can continue asynchronously.

It's true that the suspend checkpoint is currently synchronous. It needs to be made synchronous by telling the stub generator (libxl_save_msgs_gen.pl) to generate an asynchronous binding, like it does for switch_qemu_logdirty.

Perhaps it would be helpful if I provided a pre-patch to make that change for you.

Quoted 9 lines

tools/libxl/libxl_dom.c void libxl__remus_teardown_done(libxl__e · line 1276

Quoted 6 lines
tools/libxl/libxl_dom.c +6 -0
+    if (!remus_ctx->netbuf_ctx) return is_suspended;
+
+    if (is_suspended) {
+        if (libxl__remus_netbuf_start_new_epoch(gc, dss->domid,
+                                                remus_ctx))
+            return !is_suspended;

This construction is rather odd. Why return !is_suspended when you know that !!is_suspended ?

...

Quoted 2 lines

If a new network buffer cannot be created (for the next epoch), then return an error. This ends up terminating the checkpointing process.

Yes. But why not "return 0" ?

Thanks,
Ian.

#15 lore raw

Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks [and 1 more messages]

Quoted 8 lines · Ian Jackson

Shriram Rajagopalan writes ("Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks"):

Quoted 1 line

Nanosleep was more of my personal preference,

I don't think that's a good enough reason for the churn, but as I say this really ought to be replaced with use of a timeout event.

Fair enough. My question is what is the overhead of setting up, firing and tearing down
a timeout event using the event gen framework, if I wish to checkpoint the VM, say every 20ms ?
If you happen to have the numbers off the top of your head, it would help. Or if you are sure that
the system can easily handle this rate of events with very little overhead (<0.2ms per event)

Quoted 11 lines
Quoted 5 lines

As far as using the event loop timers, I couldn't find anything suitable in libxl_event.h/c that I could use instead of usleep/nanosleep. Specifically, they were all asynchronous, while the caller of these functions is libxc's xc_domain_save infinite loop (synchronous).

No, you need to make it an asynchronous call.

The libxl save helper machinery arranges to run xc_domain_save in a subprocess so that the main program can continue asynchronously.

Quoted 9 lines

It's true that the suspend checkpoint is currently synchronous. It needs to be made synchronous by telling the stub generator (libxl_save_msgs_gen.pl) to generate an asynchronous binding, like it does for switch_qemu_logdirty.

Perhaps it would be helpful if I provided a pre-patch to make that change for you.

Yes, that would be pretty helpful. Thanks!

shriram

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

#16 lore raw

Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks [and 1 more messages]

Quoted 14 lines · Shriram Rajagopalan

On Mon, Nov 4, 2013 at 6:12 AM, Ian Jackson <[email protected]> wrote:
Shriram Rajagopalan writes ("Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks"):
> Nanosleep was more of my personal preference,

I don't think that's a good enough reason for the churn, but as I say
this really ought to be replaced with use of a timeout event.

Quoted 8 lines

Fair enough. My question is what is the overhead of setting up, firing and tearing down
a timeout event using the event gen framework, if I wish to checkpoint the VM, say every 20ms ?
If you happen to have the numbers off the top of your head, it would help. Or if you are sure that
the system can easily handle this rate of events with very little overhead (<0.2ms per event)

Regardless of the answer here, would it make sense to do some/all of the checkpoint processing in the helper subprocess anyway and only signal the eventual failover up to the libxl process?

This async op is potentially quite long running I think compared to a normal one i.e. if the guest doesn't die it is expected that the ao lives "forever". Since the associated gc's persist until the ao ends this might end up accumulating lots of allocations? Ian had a similar concern about Roger's hotplug daemon series and suggested creating a per iteration gc or something.

Ian.

#17 lore raw

Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks [and 1 more messages]

Ian Campbell writes ("Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks [and 1 more messages]"):

Quoted 3 lines

Regardless of the answer here, would it make sense to do some/all of the checkpoint processing in the helper subprocess anyway and only signal the eventual failover up to the libxl process?

It might do. Mind you, the code in libxc is tangled enough as it is and is due for a rewrite. Perhaps this could be done in the helper executable, although there isn't currently any way to easily intersperse code in there.

Quoted 6 lines

This async op is potentially quite long running I think compared to a normal one i.e. if the guest doesn't die it is expected that the ao lives "forever". Since the associated gc's persist until the ao ends this might end up accumulating lots of allocations? Ian had a similar concern about Roger's hotplug daemon series and suggested creating a per iteration gc or something.

Yes, this is indeed a problem. Well spotted.

Which of the xc_domain_save (and _restore) callbacks are called each remus iteration ?

I think introducing a per-iteration gc here is going to involve taking some care, since we need to be sure not to put per-iteraton-gc-allocated objects into data structures which are used by subsequent iterations.

Quoted 3 lines · Shriram

Fair enough. My question is what is the overhead of setting up, firing and tearing down a timeout event using the event gen framework, if I wish to checkpoint the VM, say every 20ms ?

The ultimate cost of going back into the event loop to wait for a timeout will depend on what else the process is doing. If the process is doing nothing else, it's about two calls to gettimeofday and one to poll. Plus a bit of in-process computation, but that's going to be swamped by system call overhead.

Having said that, libxl is not performance-optimised. Indeed the callback mechanism involves context switching, and IPC, between the save/restore helper and libxl proper. Probably not too much to be doing every 20ms for a single domain, but if you have a lot of these it's going to end up taking a lot of dom0 cpu etc.

I assume you're not doing this for HVM domains, which involve saving the qemu state each time too.

Ian.

#23 lore raw

Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks [and 1 more messages] [and 1 more messages]

In reply to Shriram Rajagopalan

Shriram Rajagopalan writes ("Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks [and 1 more messages]"):

Quoted 4 lines

On Mon, Nov 4, 2013 at 6:12 AM, Ian Jackson <[email protected]> wrote> Perhaps it would be helpful if I provided a pre-patch to make that
change for you.

Yes, that would be pretty helpful. Thanks!

See below. I have compiled this but not tested it. It should be safe but I can't rule out having perpetrated some howler of a bug. Please let me know if it doesn't work.

Ian Jackson writes ("Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks [and 1 more messages]"):

Quoted 11 lines
Quoted 3 lines · Shriram

Fair enough. My question is what is the overhead of setting up, firing and tearing down a timeout event using the event gen framework, if I wish to checkpoint the VM, say every 20ms ?

The ultimate cost of going back into the event loop to wait for a timeout will depend on what else the process is doing. If the process is doing nothing else, it's about two calls to gettimeofday and one to poll. Plus a bit of in-process computation, but that's going to be swamped by system call overhead.

Quoted 8 lines

Having said that, libxl is not performance-optimised. Indeed the callback mechanism involves context switching, and IPC, between the save/restore helper and libxl proper. Probably not too much to be doing every 20ms for a single domain, but if you have a lot of these it's going to end up taking a lot of dom0 cpu etc.

I assume you're not doing this for HVM domains, which involve saving the qemu state each time too.

I guess another way to look at this is that changing this one timeout from a synchronous to asynchronous version is not going to make any noticeable difference to the performance of the whole thing. You're already using all of the asynchronous save/restore helper machinery and the libxl event loop.

So if the performance of your V3 patches is acceptable, this will be fine too.

Ian.

Quoted 1 line

From 46b08918302a8c1d2e470b7af7045557f73afde9 Mon Sep 17 00:00:00 2001

From: Ian Jackson <[email protected]>
Date: Mon, 4 Nov 2013 16:27:53 +0000
Subject: [PATCH] libxl: make libxl__domain_suspend_callback be asynchronous

Mark the suspend callback as asynchronous in the helper stub generator (libxl_save_msgs_gen.pl). Remus is going to want to provide an asynchronous version of this function.

libxl__domain_suspend_common_callback, the common synchronous core, which used to be provided directly as the callback function for the helper machinery, becomes libxl__domain_suspend_callback_common. It can now take a typesafe parameter.

For now, provide two very similar asynchronous wrappers for it. Each is a simple function which contains only boilerplate, calls the common synchronous core, and returns the asynchronous response.

Essentially, we have just moved (in the case of suspend callbacks) the call site of libxl__srm_callout_callback_complete. It was in the switch statement in the autogenerated _libxl_save_msgs_callout.c, and is now in the handwritten libxl_dom.c.

No functional change.

Cc Shriram Rajagopalan <[email protected]>
Cc Ian Campbell <[email protected]>
---
 tools/libxl/libxl_dom.c            |   25 +++++++++++++++++++------
 tools/libxl/libxl_internal.h       |    2 +-
 tools/libxl/libxl_save_msgs_gen.pl |    2 +-
 3 files changed, 21 insertions(+), 8 deletions(-)
tools/libxl/libxl_dom.c +19 -6
index 1812bdc..b5cde42 100644
@@ -1003,10 +1003,8 @@ int libxl__domain_resume_device_model(libxl__gc *gc, uint32_t domid)
     return 0;
 }
 
-int libxl__domain_suspend_common_callback(void *user)
+int libxl__domain_suspend_callback_common(libxl__domain_suspend_state *dss)
 {
-    libxl__save_helper_state *shs = user;
-    libxl__domain_suspend_state *dss = CONTAINER_OF(shs, *dss, shs);
     STATE_AO_GC(dss->ao);
     unsigned long hvm_s_state = 0, hvm_pvdrv = 0;
     int ret;
@@ -1225,12 +1223,27 @@ int libxl__toolstack_save(uint32_t domid, uint8_t **buf,
     return 0;
 }
 
+static void libxl__domain_suspend_callback(void *data)
+{
+    libxl__save_helper_state *shs = data;
+    libxl__egc *egc = shs->egc;
+    libxl__domain_suspend_state *dss = CONTAINER_OF(shs, *dss, shs);
+
+    int ok = libxl__domain_suspend_callback_common(dss);
+    libxl__xc_domain_saverestore_async_callback_done(egc, shs, ok);
+}
+
 /*----- remus callbacks -----*/
 
-static int libxl__remus_domain_suspend_callback(void *data)
+static void libxl__remus_domain_suspend_callback(void *data)
 {
+    libxl__save_helper_state *shs = data;
+    libxl__egc *egc = shs->egc;
+    libxl__domain_suspend_state *dss = CONTAINER_OF(shs, *dss, shs);
+
     /* REMUS TODO: Issue disk and network checkpoint reqs. */
-    return libxl__domain_suspend_common_callback(data);
+    int ok = libxl__domain_suspend_callback_common(dss);
+    libxl__xc_domain_saverestore_async_callback_done(egc, shs, ok);
 }
 
 static int libxl__remus_domain_resume_callback(void *data)
@@ -1354,7 +1367,7 @@ void libxl__domain_suspend(libxl__egc *egc, libxl__domain_suspend_state *dss)
         callbacks->postcopy = libxl__remus_domain_resume_callback;
         callbacks->checkpoint = libxl__remus_domain_checkpoint_callback;
     } else
-        callbacks->suspend = libxl__domain_suspend_common_callback;
+        callbacks->suspend = libxl__domain_suspend_callback;
 
     callbacks->switch_qemu_logdirty = libxl__domain_suspend_common_switch_qemu_logdirty;
     dss->shs.callbacks.save.toolstack_save = libxl__toolstack_save;
tools/libxl/libxl_internal.h +1 -1
index 4f92522..79eb8f8 100644
@@ -2551,7 +2551,7 @@ _hidden void libxl__xc_domain_save_done(libxl__egc*, void *dss_void,
 void libxl__xc_domain_saverestore_async_callback_done(libxl__egc *egc,
                            libxl__save_helper_state *shs, int return_value);
 
-_hidden int libxl__domain_suspend_common_callback(void *data);
+_hidden int libxl__domain_suspend_callback_common(libxl__domain_suspend_state*);
 _hidden void libxl__domain_suspend_common_switch_qemu_logdirty
                                (int domid, unsigned int enable, void *data);
 _hidden int libxl__toolstack_save(uint32_t domid, uint8_t **buf,
tools/libxl/libxl_save_msgs_gen.pl +1 -1
index ee126c7..3c6bd57 100755
@@ -23,7 +23,7 @@ our @msgs = (
                                                  STRING doing_what),
                                                 'unsigned long', 'done',
                                                 'unsigned long', 'total'] ],
-    [  3, 'scxW',   "suspend", [] ],         
+    [  3, 'scxA',   "suspend", [] ],         
     [  4, 'scxW',   "postcopy", [] ],        
     [  5, 'scxA',   "checkpoint", [] ],      
     [  6, 'scxA',   "switch_qemu_logdirty",  [qw(int domid
Signature · 1 line
1.7.10.4
#18 lore raw

Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks [and 1 more messages]

In reply to Ian Jackson
Quoted 8 lines · Ian Jackson

Ian Campbell writes ("Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks [and 1 more messages]"):

Quoted 3 lines

Regardless of the answer here, would it make sense to do some/all of the checkpoint processing in the helper subprocess anyway and only signal the eventual failover up to the libxl process?

It might do. Mind you, the code in libxc is tangled enough as it is and is due for a rewrite. Perhaps this could be done in the helper executable,

That's where I meant.

Quoted 2 lines

although there isn't currently any way to easily
intersperse code in there.

A simple matter of programming surely ;-)

Ian.

#19 lore raw

Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks [and 1 more messages]

In reply to Ian Jackson
Quoted 6 lines · Ian Jackson

Ian Campbell writes ("Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks [and 1 more messages]"):

Quoted 3 lines

Regardless of the answer here, would it make sense to do some/all of the checkpoint processing in the helper subprocess anyway and only signal the eventual failover up to the libxl process?

Quoted 13 lines

It might do. Mind you, the code in libxc is tangled enough as it is and is due for a rewrite. Perhaps this could be done in the helper executable, although there isn't currently any way to easily intersperse code in there.

This async op is potentially quite long running I think compared to a

Quoted 5 lines

normal one i.e. if the guest doesn't die it is expected that the ao lives "forever". Since the associated gc's persist until the ao ends this might end up accumulating lots of allocations? Ian had a similar concern about Roger's hotplug daemon series and suggested creating a per iteration gc or something.

Yes, this is indeed a problem. Well spotted.

Quoted 1 line

Which of the xc_domain_save (and _restore) callbacks are called each

Quoted 3 lines

remus iteration ?

Almost all of them on the xc_domain_save side. (suspend, resume, save_qemu state, checkpoint).
xc_domain_restore doesn't have any callbacks AFAIK. And remus as of now does not have a component on the restore side. It piggybacks on live migration's
restore framework.

Quoted 6 lines

I think introducing a per-iteration gc here is going to involve taking some care, since we need to be sure not to put per-iteraton-gc-allocated objects into data structures which are used by subsequent iterations.

FWIW, the remus related code that executes per iteration does not allocate anything.
All allocations happen only during setup and I was under the impression that no other
allocations are taking place everytime xc_domain_save calls back into libxl.

However, it may be possible that other parts of the AO machinery (and there are a lot of them) are allocating stuff per iteration. And if that is the case, it could
easily lead to OOMs since Remus technically runs as long as the domain lives.

Quoted 10 lines · Shriram
Quoted 3 lines

Fair enough. My question is what is the overhead of setting up, firing and tearing down a timeout event using the event gen framework, if I wish to checkpoint the VM, say every 20ms ?

The ultimate cost of going back into the event loop to wait for a timeout will depend on what else the process is doing. If the process is doing nothing else, it's about two calls to gettimeofday and one to poll. Plus a bit of in-process computation, but that's going to be swamped by system call overhead.

Quoted 7 lines

Having said that, libxl is not performance-optimised. Indeed the callback mechanism involves context switching, and IPC, between the save/restore helper and libxl proper. Probably not too much to be doing every 20ms for a single domain, but if you have a lot of these it's going to end up taking a lot of dom0 cpu etc.

Yes and that is a problem. Xend+Remus avoided this by linking the libcheckpoint library that interfaced with both the python & libxc code.

Quoted 4 lines

I assume you're not doing this for HVM domains, which involve saving the qemu state each time too.

It includes HVM domains too. Although in that case, xenstore based suspend takes about 5ms. So the checkpoint interval is typically 50ms or so.

If there is a latency sensitive task running inside the VM, lower checkpoint interval leads to better performance.

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

#20 lore raw

Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks [and 1 more messages]

Shriram Rajagopalan writes ("Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks [and 1 more messages]"):

Quoted 6 lines · Ian Jackson

Which of the xc_domain_save (and _restore) callbacks are called each remus iteration ?

Almost all of them on the xc_domain_save side. (suspend, resume, save_qemu state, checkpoint).

Right.

Quoted 4 lines

xc_domain_restore doesn't have any
callbacks AFAIK. And remus as of now does not have a component on the restore side. It piggybacks on live migration's restore framework.

But the libxl memory management in the restore code is currently written to assume a finite lifetime for the ao. So I think this needs to be improved.

Perhaps all the suspend/restore callbacks should each get one of the nested ao type things that Roger needs for his driver domain daemon.

Quoted 4 lines

FWIW, the remus related code that executes per iteration does not allocate anything. All allocations happen only during setup and I was under the impression that no other allocations are taking place everytime xc_domain_save calls back into libxl.

If this is true, then good, because we don't need to do anything, but there is a lot of code there and I would want to check.

Quoted 4 lines

However, it may be possible that other parts of the AO machinery (and there are a lot of them) are allocating stuff per iteration. And if that is the case, it could easily lead to OOMs since Remus technically runs as long as the domain lives.

The ao and event machinery doesn't do much allocation itself.

Quoted 8 lines

Having said that, libxl is not performance-optimised. Indeed the callback mechanism involves context switching, and IPC, between the save/restore helper and libxl proper. Probably not too much to be doing every 20ms for a single domain, but if you have a lot of these it's going to end up taking a lot of dom0 cpu etc.

Yes and that is a problem. Xend+Remus avoided this by linking the libcheckpoint library that interfaced with both the python & libxc code.

Have you observed whether the performance is acceptable with your V3 patches ?

Quoted 2 lines

I assume you're not doing this for HVM domains, which involve saving the qemu state each time too.

...

Quoted 2 lines

It includes HVM domains too. Although in that case, xenstore based suspend takes about 5ms. So the checkpoint interval is typically 50ms or so.

Right.

Quoted 2 lines

If there is a latency sensitive task running inside the VM, lower checkpoint interval leads to better performance.

Yes.

Ian.

#21 lore raw

Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks [and 1 more messages]

Quoted 7 lines · Ian Jackson
Quoted 1 line

Having said that, libxl is not performance-optimised. Indeed the

callback mechanism involves context switching, and IPC, between the

Quoted 4 lines

save/restore helper and libxl proper. Probably not too much to be doing every 20ms for a single domain, but if you have a lot of these it's going to end up taking a lot of dom0 cpu etc.

Quoted 8 lines
Quoted 2 lines

Yes and that is a problem. Xend+Remus avoided this by linking the libcheckpoint library that interfaced with both the python & libxc

code.

Have you observed whether the performance is acceptable with your V3 patches ?

Frankly I don't know. At the moment, I am trying to make sure that the libxl-remus
implementation is correct before I attempt to make it fast.

With/without these patches, there is a huge overhead per checkpoint. I cannot get to
a 20ms checkpoint interval. Time to suspend/resume a domain is on the order of tens of
milliseconds even if it was a PV domain with fast suspend support -- where it used to
take under 1ms to suspend, checkpoint and resume an idle PV domain on Xend+Remus.

This was one of the issues I raised a long time back. And this is currently the second
element in my queue, the first being to get the patches mainline, then slowly tighten up
relevant code for performance.

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

#22 lore raw

Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks [and 1 more messages]

Shriram Rajagopalan writes ("Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks [and 1 more messages]"):

Quoted 10 lines

Frankly I don't know. At the moment, I am trying to make sure that the libxl-remus implementation is correct before I attempt to make it fast.

With/without these patches, there is a huge overhead per checkpoint. I cannot get to a 20ms checkpoint interval. Time to suspend/resume a domain is on the order of tens of milliseconds even if it was a PV domain with fast suspend support -- where it used to take under 1ms to suspend, checkpoint and resume an idle PV domain on Xend+Remus.

Oh dear.

Quoted 4 lines

This was one of the issues I raised a long time back. And this is currently the second element in my queue, the first being to get the patches mainline, then slowly tighten up relevant code for performance.

I think this is probably a bad approach. Under the circumstances, it seems like you really need to move the relevant code lower in the call stack. In particular, you want everything to be handled synchronously in the same process as is doing the save/restore, at least for PV domains.

That's likely to involve substantially rewriting it.

Ian.

#24 lore raw

Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks [and 1 more messages] [and 1 more messages]

In reply to Ian Jackson
Quoted 173 lines · Ian Jackson

Shriram Rajagopalan writes ("Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks [and 1 more messages]"):

Quoted 1 line

On Mon, Nov 4, 2013 at 6:12 AM, Ian Jackson <[email protected]>

wrote> Perhaps it would be helpful if I provided a pre-patch to make
that

Quoted 3 lines

change for you.

Yes, that would be pretty helpful. Thanks!

See below. I have compiled this but not tested it. It should be safe but I can't rule out having perpetrated some howler of a bug. Please let me know if it doesn't work.

Ian Jackson writes ("Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks [and 1 more messages]"):

Quoted 11 lines
Quoted 3 lines · Shriram

Fair enough. My question is what is the overhead of setting up, firing and tearing down a timeout event using the event gen framework, if I wish to checkpoint the VM, say every 20ms ?

The ultimate cost of going back into the event loop to wait for a timeout will depend on what else the process is doing. If the process is doing nothing else, it's about two calls to gettimeofday and one to poll. Plus a bit of in-process computation, but that's going to be swamped by system call overhead.

Quoted 8 lines

Having said that, libxl is not performance-optimised. Indeed the callback mechanism involves context switching, and IPC, between the save/restore helper and libxl proper. Probably not too much to be doing every 20ms for a single domain, but if you have a lot of these it's going to end up taking a lot of dom0 cpu etc.

I assume you're not doing this for HVM domains, which involve saving the qemu state each time too.

I guess another way to look at this is that changing this one timeout from a synchronous to asynchronous version is not going to make any noticeable difference to the performance of the whole thing. You're already using all of the asynchronous save/restore helper machinery and the libxl event loop.

So if the performance of your V3 patches is acceptable, this will be fine too.

Ian.

Quoted 1 line

From 46b08918302a8c1d2e470b7af7045557f73afde9 Mon Sep 17 00:00:00 2001

From: Ian Jackson <[email protected]>
Date: Mon, 4 Nov 2013 16:27:53 +0000
Subject: [PATCH] libxl: make libxl__domain_suspend_callback be asynchronous

Mark the suspend callback as asynchronous in the helper stub generator (libxl_save_msgs_gen.pl). Remus is going to want to provide an asynchronous version of this function.

libxl__domain_suspend_common_callback, the common synchronous core, which used to be provided directly as the callback function for the helper machinery, becomes libxl__domain_suspend_callback_common. It can now take a typesafe parameter.

For now, provide two very similar asynchronous wrappers for it. Each is a simple function which contains only boilerplate, calls the common synchronous core, and returns the asynchronous response.

Essentially, we have just moved (in the case of suspend callbacks) the call site of libxl__srm_callout_callback_complete. It was in the switch statement in the autogenerated _libxl_save_msgs_callout.c, and is now in the handwritten libxl_dom.c.

No functional change.

Cc Shriram Rajagopalan <[email protected]>
Cc Ian Campbell <[email protected]>
---
 tools/libxl/libxl_dom.c            |   25 +++++++++++++++++++------
 tools/libxl/libxl_internal.h       |    2 +-
 tools/libxl/libxl_save_msgs_gen.pl |    2 +-
 3 files changed, 21 insertions(+), 8 deletions(-)
tools/libxl/libxl_dom.c +1 -3
index 1812bdc..b5cde42 100644
@@ -1003,10 +1003,8 @@ int libxl__domain_resume_device_model(libxl__gc
 *gc, uint32_t domid)
     return 0;
 }
 
-int libxl__domain_suspend_common_callback(void *user)
+int libxl__domain_suspend_callback_common(libxl__domain_suspend_state
 *dss)
 {
-    libxl__save_helper_state *shs = user;
-    libxl__domain_suspend_state *dss = CONTAINER_OF(shs, *dss, shs);
     STATE_AO_GC(dss->ao);
     unsigned long hvm_s_state = 0, hvm_pvdrv = 0;
     int ret;
tools/libxl/libxl_dom.c +17 -2
@@ -1225,12 +1223,27 @@ int libxl__toolstack_save(uint32_t domid, uint8_t
 **buf,
     return 0;
 }
 
+static void libxl__domain_suspend_callback(void *data)
+{
+    libxl__save_helper_state *shs = data;
+    libxl__egc *egc = shs->egc;
+    libxl__domain_suspend_state *dss = CONTAINER_OF(shs, *dss, shs);
+
+    int ok = libxl__domain_suspend_callback_common(dss);
+    libxl__xc_domain_saverestore_async_callback_done(egc, shs, ok);
+}
+
 /*----- remus callbacks -----*/
 
-static int libxl__remus_domain_suspend_callback(void *data)
+static void libxl__remus_domain_suspend_callback(void *data)
 {
+    libxl__save_helper_state *shs = data;
+    libxl__egc *egc = shs->egc;
+    libxl__domain_suspend_state *dss = CONTAINER_OF(shs, *dss, shs);
+
     /* REMUS TODO: Issue disk and network checkpoint reqs. */
-    return libxl__domain_suspend_common_callback(data);
+    int ok = libxl__domain_suspend_callback_common(dss);
+    libxl__xc_domain_saverestore_async_callback_done(egc, shs, ok);
 }
 

static int libxl__remus_domain_resume_callback(void *data)

tools/libxl/libxl_dom.c +1 -1
@@ -1354,7 +1367,7 @@ void libxl__domain_suspend(libxl__egc *egc,
 libxl__domain_suspend_state *dss)
         callbacks->postcopy = libxl__remus_domain_resume_callback;
         callbacks->checkpoint = libxl__remus_domain_checkpoint_callback;
     } else
-        callbacks->suspend = libxl__domain_suspend_common_callback;
+        callbacks->suspend = libxl__domain_suspend_callback;
 
     callbacks->switch_qemu_logdirty =
libxl__domain_suspend_common_switch_qemu_logdirty;
     dss->shs.callbacks.save.toolstack_save = libxl__toolstack_save;
tools/libxl/libxl_internal.h +1 -1
index 4f92522..79eb8f8 100644
@@ -2551,7 +2551,7 @@ _hidden void libxl__xc_domain_save_done(libxl__egc*,
 void *dss_void,
 void libxl__xc_domain_saverestore_async_callback_done(libxl__egc *egc,
                            libxl__save_helper_state *shs, int
 return_value);
 
-_hidden int libxl__domain_suspend_common_callback(void *data);
+_hidden int
 libxl__domain_suspend_callback_common(libxl__domain_suspend_state*);

_hidden void libxl__domain_suspend_common_switch_qemu_logdirty
(int domid, unsigned int enable, void
*data);
_hidden int libxl__toolstack_save(uint32_t domid, uint8_t **buf,

tools/libxl/ renamed from tools/libxl/libxl_save_msgs_gen.pl +0 -0

libxl_save_msgs_gen.pl
index ee126c7..3c6bd57 100755

tools/libxl/libxl_save_msgs_gen.pl +1 -1
@@ -23,7 +23,7 @@ our @msgs = (
                                                  STRING doing_what),
                                                 'unsigned long', 'done',
                                                 'unsigned long', 'total']
 ],
-    [  3, 'scxW',   "suspend", [] ],
+    [  3, 'scxA',   "suspend", [] ],
     [  4, 'scxW',   "postcopy", [] ],
     [  5, 'scxA',   "checkpoint", [] ],
Quoted 5 lines

[ 6, 'scxA', "switch_qemu_logdirty", [qw(int domid
--
1.7.10.4

The patch seems harmless enough. How do you want to go about this? Do you want to post/commit this patch ? Because I have to modify my patches accordingly. Or should I post this along with my patch series, avoiding the need
to wait on you before I post mine ?

shriram

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

#25 lore raw

Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks [and 1 more messages] [and 1 more messages]

Quoted 4 lines · Shriram Rajagopalan

The patch seems harmless enough. How do you want to go about this? Do you want to post/commit this patch ? Because I have to modify my patches accordingly. Or should I post this along with my patch series, avoiding the need to wait on you before I post mine ?

I recommend doing both, i.e. include it at the head of your series but also be prepared for it to go in independently, that way nobody is blocked.

Ian.

#26 lore raw

Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks [and 1 more messages] [and 1 more messages]

In reply to Shriram Rajagopalan

Shriram Rajagopalan writes ("Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks [and 1 more messages] [and 1 more messages]"):

Quoted 1 line · Ian Jackson

...

Quoted 1 line

The patch seems harmless enough. How do you want to go about this?

If you think this is the right approach, you should start on it right away based on my patch. The reason I posted it quickly was because I wanted to unblock you.

Quoted 3 lines

Do you want to post/commit this patch ? Because I have to modify my patches accordingly. Or should I post this along with my patch series, avoiding the need to wait on you before I post mine ?

You can certainly take it into the start of your patch series, yes. Like what Roger did with the nested ao patch.

Currently there isn't any other reason to make the change in this patch, so I don't think it should be committed right away. But if for some reason it does get committed to staging, you or we can just drop it from the start of your series.

Regards,
Ian.

#27 lore raw

Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks [and 1 more messages] [and 1 more messages]

Quoted 13 lines · Ian Jackson

Shriram Rajagopalan writes ("Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks [and 1 more messages] [and 1 more messages]"):

Quoted 1 line

On Mon, Nov 4, 2013 at 10:40 AM, Ian Jackson <[email protected]>

wrote:
...

Quoted 1 line

The patch seems harmless enough. How do you want to go about this?

If you think this is the right approach, you should start on it right away based on my patch. The reason I posted it quickly was because I wanted to unblock you.

The nested-ao patch makes sense for Remus, even without fixing this timeout issue.
I can modify my stuff accordingly. Probably create a nested-ao per iteration and drop
it at the start of the next iteration.

However, the timeout part is not convincing enough. For example, libxl__domain_suspend_common_callback [the version before your patch] has two 6 second wait loops in the worst case..

LOG(DEBUG, "issuing %s suspend request via XenBus control node",

          dss->hvm ? "PVHVM" : "PV");
  libxl__domain_pvcontrol_write(gc, XBT_NULL, domid, "suspend");

LOG(DEBUG, "wait for the guest to acknowledge suspend request");

        watchdog = 60;
        while (!strcmp(state, "suspend") && watchdog > 0) {
            usleep(100000);

            state = libxl__domain_pvcontrol_read(gc, XBT_NULL, domid);
            if (!state) state = "";

            watchdog--;
        }

and then once again

LOG(DEBUG, "wait for the guest to suspend");

    watchdog = 60;
    while (watchdog > 0) {
        xc_domaininfo_t info;

        usleep(100000);
        ret = xc_domain_getinfolist(CTX->xch, domid, 1, &info);

Now I know where the 200ms overhead per checkpoint comes from.

Shouldn't this also be made into an event loop? Irrespective of whether it is invoked in
Remus' context or normal suspend/resume/save/restore/migrate context.

And if this remains, the Remus checkpoint interval is much lower compared to this.
Typically between 25-100ms.

Quoted 4 lines
Quoted 3 lines

Do you want to post/commit this patch ? Because I have to modify my patches accordingly. Or should I post this along with my patch series, avoiding the need to wait on you before I post mine ?

Quoted 9 lines

You can certainly take it into the start of your patch series, yes. Like what Roger did with the nested ao patch.

Currently there isn't any other reason to make the change in this patch, so I don't think it should be committed right away. But if for some reason it does get committed to staging, you or we can just drop it from the start of your series.

The only reason it might get committed to staging without other remus patches
would be to fix the issue I cited above.

cheers
shriram

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

#28 lore raw

Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks [and 1 more messages] [and 1 more messages]

Shriram Rajagopalan writes ("Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks [and 1 more messages] [and 1 more messages]"):

Quoted 4 lines

The nested-ao patch makes sense for Remus, even without fixing this timeout issue. I can modify my stuff accordingly. Probably create a nested-ao per iteration and drop it at the start of the next iteration.

Right. Great.

Quoted 3 lines

However, the timeout part is not convincing enough. For example, libxl__domain_suspend_common_callback [the version before your patch] has two 6 second wait loops in the worst case..

...

Quoted 4 lines

LOG(DEBUG, "wait for the guest to acknowledge suspend request");

        watchdog = 60;
        while (!strcmp(state, "suspend") && watchdog > 0) {
            usleep(100000);

...

Quoted 1 line

and then once again

...

Quoted 1 line
        usleep(100000);

Oh dear. That is very poor.

Quoted 5 lines

Now I know where the 200ms overhead per checkpoint comes from.

Shouldn't this also be made into an event loop? Irrespective of whether it is invoked in Remus' context or normal suspend/resume/save/restore/migrate context.

Yes, you are entitrely correct.

Both of these loops should be replaced with timeout/event/callback approaches.

Do you want to attempt this or would you like me to do it ?

Quoted 4 lines

Currently there isn't any other reason to make the change in this patch, so I don't think it should be committed right away. But if for some reason it does get committed to staging, you or we can just drop it from the start of your series.

...

Quoted 2 lines

The only reason it might get committed to staging without other remus patches would be to fix the issue I cited above.

Yes.

Ian.

#29 lore raw

Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks [and 1 more messages] [and 1 more messages]

Quoted 31 lines · Ian Jackson

Shriram Rajagopalan writes ("Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks [and 1 more messages] [and 1 more messages]"):

Quoted 4 lines

The nested-ao patch makes sense for Remus, even without fixing this timeout issue. I can modify my stuff accordingly. Probably create a nested-ao per iteration and drop it at the start of the next iteration.

Right. Great.

Quoted 3 lines

However, the timeout part is not convincing enough. For example, libxl__domain_suspend_common_callback [the version before your patch] has two 6 second wait loops in the worst case..

...

Quoted 4 lines

LOG(DEBUG, "wait for the guest to acknowledge suspend request");

        watchdog = 60;
        while (!strcmp(state, "suspend") && watchdog > 0) {
            usleep(100000);

...

Quoted 1 line

and then once again

...

Quoted 1 line
        usleep(100000);

Oh dear. That is very poor.

Quoted 5 lines

Now I know where the 200ms overhead per checkpoint comes from.

Shouldn't this also be made into an event loop? Irrespective of whether it is invoked in Remus' context or normal suspend/resume/save/restore/migrate context.

Quoted 7 lines

Yes, you are entitrely correct.

Both of these loops should be replaced with timeout/event/callback approaches.

Do you want to attempt this or would you like me to do it ?

I can take a crack at it.

Thanks
Shriram

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

#30 lore raw

Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks [and 1 more messages] [and 1 more messages]

Shriram Rajagopalan writes ("Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks [and 1 more messages] [and 1 more messages]"):

Quoted 5 lines

On Tue, Nov 12, 2013 at 10:38 AM, Ian Jackson <[email protected]> wrote:
Shriram Rajagopalan writes ("Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks [and 1 more messages] [and 1 more messages]"):

...

Quoted 6 lines

Both of these loops should be replaced with timeout/event/callback approaches.

Do you want to attempt this or would you like me to do it ?

I can take a crack at it.

OK. Good luck. If you get stuck please do shout for help. Also note that feature freeze is quite soon!

Ian.

#7 lore raw

Re: [PATCH 2 of 5 V3] tools/hotplug: Remus network buffering setup scripts

In reply to Ian Campbell
Quoted 9 lines · Ian Campbell
Quoted 7 lines

This was to just make sure that the IFB variable was supplied as part of the environment..
Just like the two checks on top of this script..
"
: ${vifname?}
: ${XENBUS_PATH?}
"

Do these result in useful error reporting to the end user?

It displays a message "<script name>:lineno: vifname: parameter not set or null"
on stderr.

Quoted 5 lines

Bearing in
mind that the end user might be using libxl but not xl and therefore not see stdout/err (which might be going into some daemon's log file).

These parameters are supposed to be set by libxl before fork+exec-ing the script.
And their values are not obtained from the user or the command line. They are
generated inside libxl, and checked for non-null status. So I dont think we need
to worry about libxl not being able to see the error.

Quoted 4 lines

Also, either I'm missing it or the bash manpage only documents ${parameter:?} and not ${parameter?}. Are both actually valid?

You are right. The man page documents only the ${parameter:?} format, although the "${parameter?}" also seems to work for me. I will stick to the documented format.

Quoted 5 lines

Ian.

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel