Skip to contentSkip to conversation

Remus/Libxl: Network buffering support

PATCH Superseded [email protected] 3 participants 14 messages Aug 25, 2013 – Aug 30, 2013

v1 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Remus/Libxl: Network buffering support

This patch series adds support for network buffering in the Remus codebase in libxl.

The series is organized as follows:
1/5 - changes to autoconf scripts to check for libnl3. Add linker flags
to libxl Makefile.

2/5 - External script to setup/teardown network buffering using libnl3's CLI.
This script will be invoked by libxl before starting Remus. The script's main job is to bring up an IFB device with plug qdisc attached to it. It then re-routes egress traffic from the guest's vif to the IFB device.

3/5 - Libxl code to invoke the external setup script, followed by
netlink related setup to obtain a handle on the output buffers attached to each vif.

4/5 - Libxl interaction with network buffer module in the kernel via libnl3 API.

5/5 - xl cmdline switch to explicitly enable network buffering when starting remus.

Few things to note:
a) Based on previous email discussions, the setup/teardown task has been moved to a hotplug style shell script which can be customized as desired, instead of implementing it as C code inside libxl.

b) Libnl3 is not available on NetBSD. Nor is it available on CentOS (Linux). So I have made network buffering support an optional feature so that it can be disabled if desired.

c) NetBSD does not have libnl3. So I have put the setup script under tools/hotplug/Linux folder.

thanks
shriram

#1 lore raw

PATCH 1/1 remus: add libnl3 dependency to autoconf scripts

# HG changeset patch
# User Shriram Rajagopalan <[email protected]> # Date 1377473602 25200
# Node ID 0ae8b121b67b3cb3a6b00f2cc97e98fec5758e20 # Parent 40b079bd57dea24c3b000f233ed123763483a4d5
remus: add libnl3 dependency to autoconf scripts

Libnl3 is required for controlling Remus network buffering. This patch adds dependency on libnl3 (>= 3.2.8) to autoconf scripts. Also provide ability to configure tools without libnl3 support, that is without network buffering support.

config/Tools.mk.in +2 -0
diff -r 40b079bd57de -r 0ae8b121b67b config/Tools.mk.in
@@ -35,6 +35,7 @@ PTHREAD_LDFLAGS     := @PTHREAD_LDFLAGS@
 PTHREAD_LIBS        := @PTHREAD_LIBS@
 
 PTYFUNCS_LIBS       := @PTYFUNCS_LIBS@
+LIBNL3_LIBS         := @LIBNL3_LIBS@
 
 # Download GIT repositories via HTTP or GIT's own protocol?
 # GIT's protocol is faster and more robust, when it works at all (firewalls
@@ -54,6 +55,7 @@ CONFIG_QEMU_TRAD    := @qemu_traditional
 CONFIG_QEMU_XEN     := @qemu_xen@
 CONFIG_XEND         := @xend@
 CONFIG_BLKTAP1      := @blktap1@
+CONFIG_REMUS_NETBUF := @remusnetbuf@
 
 #System options
 ZLIB                := @zlib@
tools/configure.ac +27 -0
diff -r 40b079bd57de -r 0ae8b121b67b tools/configure.ac
@@ -56,6 +56,8 @@ AX_ARG_DEFAULT_ENABLE([xsmpolicy], [Disa
 AX_ARG_DEFAULT_DISABLE([ovmf], [Enable OVMF])
 AX_ARG_DEFAULT_ENABLE([rombios], [Disable ROM BIOS])
 AX_ARG_DEFAULT_ENABLE([seabios], [Disable SeaBIOS])
+AX_ARG_DEFAULT_ENABLE([remusnetbuf], [ Enable/Disable Remus 
+				     network buffering support (requires libnl3)])
 AX_ARG_DEFAULT_ENABLE([debug], [Disable debug build of tools])
 AX_ARG_DEFAULT_ENABLE([xend], [Disable xend toolstack])
 AX_ARG_DEFAULT_DISABLE([blktap1], [Disable blktap1 tools])
@@ -208,4 +210,29 @@ AC_SUBST(libiconv)
 # Checks for header files.
 AC_CHECK_HEADERS([yajl/yajl_version.h sys/eventfd.h])
 
+AS_IF([test "x$remusnetbuf" = "xy"], [
+
+dnl Need libnl version 3.2.8 or higher for plug qdisc support
+  AS_IF(
+       [ pkg-config --exists 'libnl-3.0 >= 3.2.8 libnl-route-3.0 >= 3.2.8 libnl-cli-3.0 >= 3.2.8' ],
+       [], [
+           AC_MSG_ERROR([Need libnl3, libnl-route-3 and libnl-cli-3 (>= 3.2.8) for Remus network buffering])
+       ])
+
+  AC_CHECK_HEADER([netlink/route/qdisc/plug.h], [
+	   AC_CHECK_LIB(nl-3, nl_socket_alloc, [
+		AC_CHECK_LIB(nl-route-3, rtnl_qdisc_alloc, [
+			LIBNL3_LIBS="-lnl-3 -lnl-route-3"
+			AC_SUBST(LIBNL3_LIBS)
+		], [
+		   AC_MSG_ERROR([Could not find libnl-route-3])
+		])
+	   ], [
+	      AC_MSG_ERROR([Could not find libnl3])
+	   ])
+	], [
+	   AC_MSG_ERROR([Could not find libnl3 dev headers (esp. plug.h)])
+	])
+], [])
+
 AC_OUTPUT()
tools/libxl/Makefile +1 -1
diff -r 40b079bd57de -r 0ae8b121b67b tools/libxl/Makefile
@@ -20,7 +20,7 @@ LIBUUID_LIBS += -luuid
 endif
 
 LIBXL_LIBS =
-LIBXL_LIBS = $(LDLIBS_libxenctrl) $(LDLIBS_libxenguest) $(LDLIBS_libxenstore) $(LDLIBS_libblktapctl) $(PTYFUNCS_LIBS) $(LIBUUID_LIBS)
+LIBXL_LIBS = $(LDLIBS_libxenctrl) $(LDLIBS_libxenguest) $(LDLIBS_libxenstore) $(LDLIBS_libblktapctl) $(PTYFUNCS_LIBS) $(LIBUUID_LIBS) $(LIBNL3_LIBS)
 
 CFLAGS_LIBXL += $(CFLAGS_libxenctrl)
 CFLAGS_LIBXL += $(CFLAGS_libxenguest)
#6 lore raw

PATCH 1/1 tools/hotplug: Remus network buffering setup scripts

In reply to [email protected]

# HG changeset patch
# User Shriram Rajagopalan <[email protected]> # Date 1377473607 25200
# Node ID 4d24d36d30c37e4fb5985952db92c1e789c90894 # Parent 0ae8b121b67b3cb3a6b00f2cc97e98fec5758e20 tools/hotplug: Remus network buffering setup scripts

This patch introduces remus-netbuf-setup hotplug script responsible for setting up and tearing down the necessary infrastructure required for network output buffering in Remus. This script is intended to be invoked by libxl for each guest interface, when starting or stopping Remus.

Apart from returning success/failure indication via the usual hotplug entires in xenstore, this script also writes to xenstore, the name of the IFB device to be used to control the vif's network output.

The script relies on libnl3 command line utilities to perform various setup/teardown functions. The script is confined to Linux platforms only since NetBSD does not seem to have libnl3.

tools/hotplug/Linux/Makefile +1 -0
diff -r 0ae8b121b67b -r 4d24d36d30c3 tools/hotplug/Linux/Makefile
@@ -16,6 +16,7 @@ XEN_SCRIPTS += network-nat vif-nat
 XEN_SCRIPTS += vif-openvswitch
 XEN_SCRIPTS += vif2
 XEN_SCRIPTS += vif-setup
+XEN_SCRIPTS-$(CONFIG_REMUS_NETBUF) += remus-netbuf-setup
 XEN_SCRIPTS += block
 XEN_SCRIPTS += block-enbd block-nbd
 XEN_SCRIPTS-$(CONFIG_BLKTAP1) += blktap
tools/hotplug/Linux/remus-netbuf-setup new file +186 -0

186 lines not on this page · raw message

#7 lore raw

PATCH 1/1 tools/libxl: setup/teardown Remus network buffering

In reply to [email protected]

# HG changeset patch
# User Shriram Rajagopalan <[email protected]> # Date 1377473609 25200
# Node ID 4b23104828c09218aa7f9fbde1578bb6706e13d6 # Parent 4d24d36d30c37e4fb5985952db92c1e789c90894 tools/libxl: setup/teardown Remus network buffering

Setup/teardown remus network buffering for a given guest, when libxl_domain_remus_start API is invoked.

This patch does the following during setup:
a) call the hotplug script for each vif to setup its network buffer

b) establish a dedicated remus context containing libnl related state (netlink sockets, qdisc caches, etc.,)

c) Obtain handles to plug qdiscs installed on the IFB devices chosen by the hotplug scripts.

During teardown, the hotplug scripts are called to remove the IFB devices. This is followed by releasing netlink resources.

tools/libxl/Makefile +7 -0
diff -r 4d24d36d30c3 -r 4b23104828c0 tools/libxl/Makefile
@@ -40,6 +40,13 @@ LIBXL_OBJS-y += libxl_blktap2.o
 else
 LIBXL_OBJS-y += libxl_noblktap2.o
 endif
+
+ifeq ($(CONFIG_REMUS_NETBUF),y)
+LIBXL_OBJS-y += libxl_netbuffer.o
+else
+LIBXL_OBJS-y += libxl_nonetbuffer.o
+endif
+
 LIBXL_OBJS-$(CONFIG_X86) += libxl_cpuid.o libxl_x86.o
 LIBXL_OBJS-$(CONFIG_IA64) += libxl_nocpuid.o libxl_noarch.o
 LIBXL_OBJS-$(CONFIG_ARM) += libxl_nocpuid.o libxl_noarch.o
tools/libxl/libxl.c +39 -10

82 lines not on this page · raw message

tools/libxl/libxl_dom.c +4 -6
diff -r 4d24d36d30c3 -r 4b23104828c0 tools/libxl/libxl_dom.c
@@ -1259,7 +1259,7 @@ static void remus_checkpoint_dm_saved(li
     /* REMUS TODO: Wait for disk and memory ack, release network buffer */
     /* REMUS TODO: make this asynchronous */
     assert(!rc); /* REMUS TODO handle this error properly */
-    usleep(dss->interval * 1000);
+    usleep(dss->remus_ctx->interval * 1000);
     libxl__xc_domain_saverestore_async_callback_done(egc, &dss->shs, 1);
 }
 
@@ -1277,7 +1277,6 @@ void libxl__domain_suspend(libxl__egc *e
     const libxl_domain_type type = dss->type;
     const int live = dss->live;
     const int debug = dss->debug;
-    const libxl_domain_remus_info *const r_info = dss->remus;
     libxl__srm_save_autogen_callbacks *const callbacks =
         &dss->shs.callbacks.save.a;
 
@@ -1312,9 +1311,8 @@ void libxl__domain_suspend(libxl__egc *e
     dss->guest_responded = 0;
     dss->dm_savefile = libxl__device_model_savefile(gc, domid);
 
-    if (r_info != NULL) {
-        dss->interval = r_info->interval;
-        if (r_info->compression)
+    if (dss->remus_ctx != NULL) {
+        if (dss->remus_ctx->compression)
             dss->xcflags |= XCFLAGS_CHECKPOINT_COMPRESS;
     }
 
@@ -1335,7 +1333,7 @@ void libxl__domain_suspend(libxl__egc *e
     }
 
     memset(callbacks, 0, sizeof(*callbacks));
-    if (r_info != NULL) {
+    if (dss->remus_ctx != NULL) {
         callbacks->suspend = libxl__remus_domain_suspend_callback;
         callbacks->postcopy = libxl__remus_domain_resume_callback;
         callbacks->checkpoint = libxl__remus_domain_checkpoint_callback;
tools/libxl/libxl_internal.h +24 -2
diff -r 4d24d36d30c3 -r 4b23104828c0 tools/libxl/libxl_internal.h
@@ -2242,6 +2242,29 @@ typedef struct libxl__logdirty_switch {
     libxl__ev_time timeout;
 } libxl__logdirty_switch;
 
+typedef struct libxl__remus_ctx {
+    /* checkpoint interval */
+    int interval;
+    int blackhole;
+    int compression;
+    /* Script to setup/teardown network buffers */
+    const char *netbufscript;
+    /* Opaque context containing network buffer related stuff */
+    void *netbuf_ctx;
+} libxl__remus_ctx;
+
+_hidden int libxl__remus_netbuf_setup(libxl__gc *gc, uint32_t domid,
+                                     libxl__remus_ctx *remus_ctx);
+
+_hidden int libxl__remus_netbuf_teardown(libxl__gc *gc, uint32_t domid,
+                                        libxl__remus_ctx *remus_ctx);
+
+_hidden int libxl__remus_netbuf_start_new_epoch(libxl__gc *gc, uint32_t domid,
+                                               libxl__remus_ctx *remus_ctx);
+
+_hidden int libxl__remus_netbuf_release_prev_epoch(libxl__gc *gc, uint32_t domid,
+                                                  libxl__remus_ctx *remus_ctx);
+
 struct libxl__domain_suspend_state {
     /* set by caller of libxl__domain_suspend */
     libxl__ao *ao;
@@ -2252,7 +2275,7 @@ struct libxl__domain_suspend_state {
     libxl_domain_type type;
     int live;
     int debug;
-    const libxl_domain_remus_info *remus;
+    libxl__remus_ctx *remus_ctx;
     /* private */
     xc_evtchn *xce; /* event channel handle */
     int suspend_eventchn;
@@ -2260,7 +2283,6 @@ struct libxl__domain_suspend_state {
     int xcflags;
     int guest_responded;
     const char *dm_savefile;
-    int interval; /* checkpoint interval (for Remus) */
     libxl__save_helper_state shs;
     libxl__logdirty_switch logdirty;
     /* private for libxl__domain_save_device_model */
tools/libxl/libxl_netbuffer.c new file +434 -0

434 lines not on this page · raw message

tools/libxl/libxl_nonetbuffer.c new file +54 -0

54 lines not on this page · raw message

tools/libxl/libxl_types.idl +2 -0
diff -r 4d24d36d30c3 -r 4b23104828c0 tools/libxl/libxl_types.idl
@@ -526,6 +526,8 @@ libxl_domain_remus_info = Struct("domain
     ("interval",     integer),
     ("blackhole",    bool),
     ("compression",  bool),
+    ("netbuf",  bool),
+    ("netbufscript", string),
     ])
 
 libxl_event_type = Enumeration("event_type", [
#9 lore raw

PATCH 1/1 tools/libxl: Control network buffering in remus callbacks

In reply to [email protected]

# HG changeset patch
# User Shriram Rajagopalan <[email protected]> # Date 1377473611 25200
# Node ID c6804ccfe660cb9c373c5f53a8996d0443316684 # Parent 4b23104828c09218aa7f9fbde1578bb6706e13d6 tools/libxl: Control network buffering in remus callbacks

This patch constitutes the core network buffering logic. and does the following:
a) create a new network buffer when the domain is suspended (remus_domain_suspend_callback)
b) release the previous network buffer pertaining to the committed checkpoint (remus_domain_checkpoint_dm_saved)

tools/libxl/libxl_dom.c +46 -7
diff -r 4b23104828c0 -r c6804ccfe660 tools/libxl/libxl_dom.c
@@ -1213,12 +1213,28 @@ int libxl__toolstack_save(uint32_t domid
 
 /*----- remus callbacks -----*/
 
+/* REMUS TODO: Issue disk checkpoint reqs. */
 static int libxl__remus_domain_suspend_callback(void *data)
 {
-    /* REMUS TODO: Issue disk and network checkpoint reqs. */
-    return libxl__domain_suspend_common_callback(data);
+    libxl__save_helper_state *shs = data;
+    libxl__domain_suspend_state *dss = CONTAINER_OF(shs, *dss, shs);
+    libxl__remus_ctx *remus_ctx = dss->remus_ctx;
+    int is_suspended = 0;
+    STATE_AO_GC(dss->ao);
+
+    is_suspended = libxl__domain_suspend_common_callback(data);
+    if (!remus_ctx->netbuf_ctx) return is_suspended;
+
+    if (is_suspended) {
+        if (libxl__remus_netbuf_start_new_epoch(gc, dss->domid,
+                                                remus_ctx))
+            return !is_suspended;
+    }
+
+    return is_suspended;
 }
 
+/* REMUS TODO: Deal with disk. */
 static int libxl__remus_domain_resume_callback(void *data)
 {
     libxl__save_helper_state *shs = data;
@@ -1229,7 +1245,6 @@ static int libxl__remus_domain_resume_ca
     if (libxl__domain_resume(gc, dss->domid, /* Fast Suspend */1))
         return 0;
 
-    /* REMUS TODO: Deal with disk. Start a new network output buffer */
     return 1;
 }
 
@@ -1256,10 +1271,34 @@ static void libxl__remus_domain_checkpoi
 static void remus_checkpoint_dm_saved(libxl__egc *egc,
                                       libxl__domain_suspend_state *dss, int rc)
 {
-    /* REMUS TODO: Wait for disk and memory ack, release network buffer */
-    /* REMUS TODO: make this asynchronous */
-    assert(!rc); /* REMUS TODO handle this error properly */
-    usleep(dss->remus_ctx->interval * 1000);
+    /* REMUS TODO: Wait for disk and explicit memory ack (through restore
+       callback from remote) before releasing network buffer. */
+    libxl__remus_ctx *remus_ctx = dss->remus_ctx;
+    struct timespec epoch;
+    int ret;
+    STATE_AO_GC(dss->ao);
+
+    if (rc) {
+        LOG(ERROR, "Failed to save device model. Terminating Remus..");
+        libxl__xc_domain_saverestore_async_callback_done(egc,
+                                                         &dss->shs, rc);
+        return;
+    }
+
+    if (remus_ctx->netbuf_ctx) {
+        ret = libxl__remus_netbuf_release_prev_epoch(gc, dss->domid,
+                                                     remus_ctx);
+        if (ret) {
+            libxl__xc_domain_saverestore_async_callback_done(egc,
+                                                             &dss->shs,
+                                                             ret);
+            return;
+        }
+    }
+
+    epoch.tv_sec = remus_ctx->interval / 1000; /* interval is in ms */
+    epoch.tv_nsec = remus_ctx->interval * 1000L * 1000L;
+    nanosleep(&epoch, 0);
     libxl__xc_domain_saverestore_async_callback_done(egc, &dss->shs, 1);
 }
#13 lore raw

PATCH 1/1 tools/xl: Remus - Network buffering cmdline switch

In reply to [email protected]

# HG changeset patch
# User Shriram Rajagopalan <[email protected]> # Date 1377473613 25200
# Node ID 496b22ccf0e2b7e48388b5a76298e76a911ba193 # Parent c6804ccfe660cb9c373c5f53a8996d0443316684 tools/xl: Remus - Network buffering cmdline switch

Command line switch to 'xl remus' command, to enable network buffering. Pass on this flag to libxl so that it can act accordingly. Also update man pages to reflect the addition of a new option to 'xl remus' command.

docs/man/xl.pod.1 +8 -2
diff -r c6804ccfe660 -r 496b22ccf0e2 docs/man/xl.pod.1
@@ -398,8 +398,7 @@ Print huge (!) amount of debug during th
 Enable Remus HA for domain. By default B<xl> relies on ssh as a transport
 mechanism between the two hosts.
 
-N.B: Remus support in xl is still in experimental (proof-of-concept) phase.
-     There is no support for network or disk buffering at the moment.
+N.B: There is no support for disk buffering at the moment.
 
 B<OPTIONS>
 
@@ -418,6 +417,13 @@ Generally useful for debugging.
 
 Disable memory checkpoint compression.
 
+=item B<-n>
+
+Enable network output buffering.  The default script used to configure
+network buffering is /etc/xen/scripts/remus-netbuf-setup. If you wish to
+use a custom script, set the global variable "remus.default.netbufscript"
+in /etc/xen/xl.conf to point to your script.
+
 =item B<-s> I<sshcommand>
 
 Use <sshcommand> instead of ssh.  String will be passed to sh.
tools/libxl/xl.c +6 -0
diff -r c6804ccfe660 -r 496b22ccf0e2 tools/libxl/xl.c
@@ -46,6 +46,7 @@ char *default_vifscript = NULL;
 char *default_bridge = NULL;
 char *default_gatewaydev = NULL;
 char *default_vifbackend = NULL;
+char *default_remus_netbufscript = NULL;
 enum output_format default_output_format = OUTPUT_FORMAT_JSON;
 int claim_mode = 1;
 
@@ -177,6 +178,11 @@ static void parse_global_config(const ch
     if (!xlu_cfg_get_long (config, "claim_mode", &l, 0))
         claim_mode = l;
 
+    if (!xlu_cfg_get_string (config, "remus.default.netbufscript", &buf, 0)) {
+        free(default_remus_netbufscript);
+        default_remus_netbufscript = strdup(buf);
+    }
+
     xlu_cfg_destroy(config);
 }
 
tools/libxl/xl.h +1 -0
diff -r c6804ccfe660 -r 496b22ccf0e2 tools/libxl/xl.h
@@ -152,6 +152,7 @@ extern char *default_vifscript;
 extern char *default_bridge;
 extern char *default_gatewaydev;
 extern char *default_vifbackend;
+extern char *default_remus_netbufscript;
 extern char *blkdev_start;
 
 enum output_format {
tools/libxl/xl_cmdimpl.c +13 -7
diff -r c6804ccfe660 -r 496b22ccf0e2 tools/libxl/xl_cmdimpl.c
@@ -7070,8 +7070,9 @@ int main_remus(int argc, char **argv)
     r_info.interval = 200;
     r_info.blackhole = 0;
     r_info.compression = 1;
-
-    SWITCH_FOREACH_OPT(opt, "bui:s:e", NULL, "remus", 2) {
+    r_info.netbuf = 0;
+
+    SWITCH_FOREACH_OPT(opt, "buni:s:e", NULL, "remus", 2) {
     case 'i':
         r_info.interval = atoi(optarg);
         break;
@@ -7081,6 +7082,9 @@ int main_remus(int argc, char **argv)
     case 'u':
         r_info.compression = 0;
         break;
+    case 'n':
+        r_info.netbuf = 1;
+        break;
     case 's':
         ssh_command = optarg;
         break;
@@ -7092,6 +7096,11 @@ int main_remus(int argc, char **argv)
     domid = find_domain(argv[optind]);
     host = argv[optind + 1];
 
+    if (r_info.netbuf) {
+        if (default_remus_netbufscript)
+            r_info.netbufscript = strdup(default_remus_netbufscript);
+    }
+
     if (r_info.blackhole) {
         send_fd = open("/dev/null", O_RDWR, 0644);
         if (send_fd < 0) {
@@ -7126,13 +7135,10 @@ int main_remus(int argc, char **argv)
     /* Point of no return */
     rc = libxl_domain_remus_start(ctx, &r_info, domid, send_fd, recv_fd, 0);
 
-    /* If we are here, it means backup has failed/domain suspend failed.
-     * Try to resume the domain and exit gracefully.
+    /* If we are here, it means remus setup/domain susppend/backup has
+     * failed. Try to resume the domain and exit gracefully.
      * TODO: Split-Brain check.
      */
-    fprintf(stderr, "remus sender: libxl_domain_suspend failed"
-            " (rc=%d)\n", rc);
-
     if (rc == ERROR_GUEST_TIMEDOUT)
         fprintf(stderr, "Failed to suspend domain at primary.\n");
     else {
tools/libxl/xl_cmdtable.c +1 -0
diff -r c6804ccfe660 -r 496b22ccf0e2 tools/libxl/xl_cmdtable.c
@@ -481,6 +481,7 @@ struct cmd_spec cmd_table[] = {
       "-i MS                   Checkpoint domain memory every MS milliseconds (def. 200ms).\n"
       "-b                      Replicate memory checkpoints to /dev/null (blackhole)\n"
       "-u                      Disable memory checkpoint compression.\n"
+      "-n                      Enable network output buffering.\n"
       "-s <sshcommand>         Use <sshcommand> instead of ssh.  String will be passed\n"
       "                        to sh. If empty, run <host> instead of \n"
       "                        ssh <host> xl migrate-receive -r [-e]\n"
#2 lore raw

Re: [PATCH 1 of 5] remus: add libnl3 dependency to autoconf scripts

In reply to [email protected]
Quoted 7 lines · On 26/08/2013 00:45, [email protected] wrote:

# HG changeset patch
# User Shriram Rajagopalan <[email protected]> # Date 1377473602 25200
# Node ID 0ae8b121b67b3cb3a6b00f2cc97e98fec5758e20 # Parent 40b079bd57dea24c3b000f233ed123763483a4d5
remus: add libnl3 dependency to autoconf scripts

Quoted 6 lines

Libnl3 is required for controlling Remus network buffering. This patch adds dependency on libnl3 (>= 3.2.8) to autoconf scripts. Also provide ability to configure tools without libnl3 support, that is without network buffering support.

As identified in the patch 0, libnl3 is not present for CentOS (so by implication RHEL and derivatives) and various BSDs, so enabling it by default and erroring out of autoconf if it cant be found is not really acceptable imo.

It would be better to detect the presence/absence of an appropriate libnl3, and use that as the default. Then, the logic would be: "By default build in libnl3 support if it is available, but don't if it is not. If the user explicitlicy configures --with-remusnetbuf and libnl3 is not present, present an error" which is far more appropriate for an option which affects a large swathe of distributions.

~Andrew

Quoted 79 lines

diff -r 40b079bd57de -r 0ae8b121b67b config/Tools.mk.in

config/Tools.mk.in +2 -0
@@ -35,6 +35,7 @@ PTHREAD_LDFLAGS     := @PTHREAD_LDFLAGS@
 PTHREAD_LIBS        := @PTHREAD_LIBS@
 
 PTYFUNCS_LIBS       := @PTYFUNCS_LIBS@
+LIBNL3_LIBS         := @LIBNL3_LIBS@
 
 # Download GIT repositories via HTTP or GIT's own protocol?
 # GIT's protocol is faster and more robust, when it works at all (firewalls
@@ -54,6 +55,7 @@ CONFIG_QEMU_TRAD    := @qemu_traditional
 CONFIG_QEMU_XEN     := @qemu_xen@
 CONFIG_XEND         := @xend@
 CONFIG_BLKTAP1      := @blktap1@
+CONFIG_REMUS_NETBUF := @remusnetbuf@
 
 #System options
 ZLIB                := @zlib@

diff -r 40b079bd57de -r 0ae8b121b67b tools/configure.ac

tools/configure.ac +27 -0
@@ -56,6 +56,8 @@ AX_ARG_DEFAULT_ENABLE([xsmpolicy], [Disa
 AX_ARG_DEFAULT_DISABLE([ovmf], [Enable OVMF])
 AX_ARG_DEFAULT_ENABLE([rombios], [Disable ROM BIOS])
 AX_ARG_DEFAULT_ENABLE([seabios], [Disable SeaBIOS])
+AX_ARG_DEFAULT_ENABLE([remusnetbuf], [ Enable/Disable Remus 
+				     network buffering support (requires libnl3)])
 AX_ARG_DEFAULT_ENABLE([debug], [Disable debug build of tools])
 AX_ARG_DEFAULT_ENABLE([xend], [Disable xend toolstack])
 AX_ARG_DEFAULT_DISABLE([blktap1], [Disable blktap1 tools])
@@ -208,4 +210,29 @@ AC_SUBST(libiconv)
 # Checks for header files.
 AC_CHECK_HEADERS([yajl/yajl_version.h sys/eventfd.h])
 
+AS_IF([test "x$remusnetbuf" = "xy"], [
+
+dnl Need libnl version 3.2.8 or higher for plug qdisc support
+  AS_IF(
+       [ pkg-config --exists 'libnl-3.0 >= 3.2.8 libnl-route-3.0 >= 3.2.8 libnl-cli-3.0 >= 3.2.8' ],
+       [], [
+           AC_MSG_ERROR([Need libnl3, libnl-route-3 and libnl-cli-3 (>= 3.2.8) for Remus network buffering])
+       ])
+
+  AC_CHECK_HEADER([netlink/route/qdisc/plug.h], [
+	   AC_CHECK_LIB(nl-3, nl_socket_alloc, [
+		AC_CHECK_LIB(nl-route-3, rtnl_qdisc_alloc, [
+			LIBNL3_LIBS="-lnl-3 -lnl-route-3"
+			AC_SUBST(LIBNL3_LIBS)
+		], [
+		   AC_MSG_ERROR([Could not find libnl-route-3])
+		])
+	   ], [
+	      AC_MSG_ERROR([Could not find libnl3])
+	   ])
+	], [
+	   AC_MSG_ERROR([Could not find libnl3 dev headers (esp. plug.h)])
+	])
+], [])
+
 AC_OUTPUT()

diff -r 40b079bd57de -r 0ae8b121b67b tools/libxl/Makefile

tools/libxl/Makefile +1 -1
@@ -20,7 +20,7 @@ LIBUUID_LIBS += -luuid
 endif
 
 LIBXL_LIBS =
-LIBXL_LIBS = $(LDLIBS_libxenctrl) $(LDLIBS_libxenguest) $(LDLIBS_libxenstore) $(LDLIBS_libblktapctl) $(PTYFUNCS_LIBS) $(LIBUUID_LIBS)
+LIBXL_LIBS = $(LDLIBS_libxenctrl) $(LDLIBS_libxenguest) $(LDLIBS_libxenstore) $(LDLIBS_libblktapctl) $(PTYFUNCS_LIBS) $(LIBUUID_LIBS) $(LIBNL3_LIBS)
 
 CFLAGS_LIBXL += $(CFLAGS_libxenctrl)
 CFLAGS_LIBXL += $(CFLAGS_libxenguest)

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

#8 lore raw

Re: [PATCH 3 of 5] tools/libxl: setup/teardown Remus network buffering

In reply to [email protected]
Quoted 151 lines · On 26/08/2013 00:45, [email protected] wrote:

# HG changeset patch
# User Shriram Rajagopalan <[email protected]> # Date 1377473609 25200
# Node ID 4b23104828c09218aa7f9fbde1578bb6706e13d6 # Parent 4d24d36d30c37e4fb5985952db92c1e789c90894 tools/libxl: setup/teardown Remus network buffering

Setup/teardown remus network buffering for a given guest, when libxl_domain_remus_start API is invoked.

This patch does the following during setup:
a) call the hotplug script for each vif to setup its network buffer

b) establish a dedicated remus context containing libnl related state (netlink sockets, qdisc caches, etc.,)

c) Obtain handles to plug qdiscs installed on the IFB devices chosen by the hotplug scripts.

During teardown, the hotplug scripts are called to remove the IFB devices. This is followed by releasing netlink resources.

diff -r 4d24d36d30c3 -r 4b23104828c0 tools/libxl/Makefile

tools/libxl/Makefile +7 -0
@@ -40,6 +40,13 @@ LIBXL_OBJS-y += libxl_blktap2.o
 else
 LIBXL_OBJS-y += libxl_noblktap2.o
 endif
+
+ifeq ($(CONFIG_REMUS_NETBUF),y)
+LIBXL_OBJS-y += libxl_netbuffer.o
+else
+LIBXL_OBJS-y += libxl_nonetbuffer.o
+endif
+
 LIBXL_OBJS-$(CONFIG_X86) += libxl_cpuid.o libxl_x86.o
 LIBXL_OBJS-$(CONFIG_IA64) += libxl_nocpuid.o libxl_noarch.o
 LIBXL_OBJS-$(CONFIG_ARM) += libxl_nocpuid.o libxl_noarch.o

diff -r 4d24d36d30c3 -r 4b23104828c0 tools/libxl/libxl.c

tools/libxl/libxl.c +39 -10
@@ -692,8 +692,9 @@ libxl_vminfo * libxl_list_vm(libxl_ctx *
     return ptr;
 }
 
-static void remus_failover_cb(libxl__egc *egc,
-                              libxl__domain_suspend_state *dss, int rc);
+static void remus_replication_failure_cb(libxl__egc *egc,
+                                         libxl__domain_suspend_state *dss,
+                                         int rc);
 
 /* TODO: Explicit Checkpoint acknowledgements via recv_fd. */
 int libxl_domain_remus_start(libxl_ctx *ctx, libxl_domain_remus_info *info,
@@ -712,18 +713,44 @@ int libxl_domain_remus_start(libxl_ctx *
 
     GCNEW(dss);
     dss->ao = ao;
-    dss->callback = remus_failover_cb;
+    dss->callback = remus_replication_failure_cb;
     dss->domid = domid;
     dss->fd = send_fd;
     /* TODO do something with recv_fd */
     dss->type = type;
     dss->live = 1;
     dss->debug = 0;
-    dss->remus = info;
 
     assert(info);
 
-    /* TBD: Remus setup - i.e. attach qdisc, enable disk buffering, etc */
+    GCNEW(dss->remus_ctx);
+
+    /* convenience shorthand */
+    libxl__remus_ctx *remus_ctx = dss->remus_ctx;
+    remus_ctx->blackhole = info->blackhole;
+    remus_ctx->interval = info->interval;
+    remus_ctx->compression = info->compression;
+
+    /* Setup network buffering before invoking domain_suspend */
+    if (info->netbuf) {
+        if (info->netbufscript) {
+            remus_ctx->netbufscript =
+                libxl__strdup(gc, info->netbufscript);
+        } else {
+            remus_ctx->netbufscript =
+                libxl__sprintf(gc, "%s/remus-netbuf-setup",
+                              libxl__xen_script_dir_path());
+        }
+
+        if (libxl__remus_netbuf_setup(gc, domid, remus_ctx)) {
+            LOG(ERROR, "Remus: failed to setup network buffering"
+                " for guest with domid %u", domid);
+            rc = ERROR_FAIL;
+            goto out;
+        }
+    }
+
+    /* TBD: enable disk buffering */
 
     /* Point of no return */
     libxl__domain_suspend(egc, dss);
@@ -733,8 +760,9 @@ int libxl_domain_remus_start(libxl_ctx *
     return AO_ABORT(rc);
 }
 
-static void remus_failover_cb(libxl__egc *egc,
-                              libxl__domain_suspend_state *dss, int rc)
+static void remus_replication_failure_cb(libxl__egc *egc,
+                                         libxl__domain_suspend_state *dss,
+                                         int rc)
 {
     STATE_AO_GC(dss->ao);
     /*
@@ -743,9 +771,10 @@ static void remus_failover_cb(libxl__egc
      * from sending checkpoints.
      */
 
-    /* TBD: Remus cleanup - i.e. detach qdisc, release other
-     * resources.
-     */
+    /* Teardown the network buffers and release netlink resources. */
+    if (dss->remus_ctx && dss->remus_ctx->netbuf_ctx)
+        libxl__remus_netbuf_teardown(gc, dss->domid, dss->remus_ctx);
+
     libxl__ao_complete(egc, ao, rc);
 }
 

diff -r 4d24d36d30c3 -r 4b23104828c0 tools/libxl/libxl_dom.c

tools/libxl/libxl_dom.c +1 -2
@@ -1259,7 +1259,7 @@ static void remus_checkpoint_dm_saved(li
     /* REMUS TODO: Wait for disk and memory ack, release network buffer */
     /* REMUS TODO: make this asynchronous */
     assert(!rc); /* REMUS TODO handle this error properly */
-    usleep(dss->interval * 1000);
+    usleep(dss->remus_ctx->interval * 1000);
     libxl__xc_domain_saverestore_async_callback_done(egc, &dss->shs, 1);
 }
 
@@ -1277,7 +1277,6 @@ void libxl__domain_suspend(libxl__egc *e
     const libxl_domain_type type = dss->type;
     const int live = dss->live;
     const int debug = dss->debug;
-    const libxl_domain_remus_info *const r_info = dss->remus;
     libxl__srm_save_autogen_callbacks *const callbacks =
         &dss->shs.callbacks.save.a;
 

tools/libxl/libxl_dom.c void libxl__domain_suspend(libxl__egc *e · line 1315

Quoted 8 lines
tools/libxl/libxl_dom.c +2 -3
@@ -1312,9 +1311,8 @@ void libxl__domain_suspend(libxl__egc *e
     dss->guest_responded = 0;
     dss->dm_savefile = libxl__device_model_savefile(gc, domid);
 
-    if (r_info != NULL) {
-        dss->interval = r_info->interval;
-        if (r_info->compression)
+    if (dss->remus_ctx != NULL) {
+        if (dss->remus_ctx->compression)

This can be reduced to:

if (dss->remus_ctx && dss->remus_ctx->compression)

~Andrew

Quoted 575 lines
tools/libxl/libxl_dom.c +1 -1
             dss->xcflags |= XCFLAGS_CHECKPOINT_COMPRESS;
     }
 
@@ -1335,7 +1333,7 @@ void libxl__domain_suspend(libxl__egc *e
     }
 
     memset(callbacks, 0, sizeof(*callbacks));
-    if (r_info != NULL) {
+    if (dss->remus_ctx != NULL) {
         callbacks->suspend = libxl__remus_domain_suspend_callback;
         callbacks->postcopy = libxl__remus_domain_resume_callback;
         callbacks->checkpoint = libxl__remus_domain_checkpoint_callback;

diff -r 4d24d36d30c3 -r 4b23104828c0 tools/libxl/libxl_internal.h

tools/libxl/libxl_internal.h +24 -2
@@ -2242,6 +2242,29 @@ typedef struct libxl__logdirty_switch {
     libxl__ev_time timeout;
 } libxl__logdirty_switch;
 
+typedef struct libxl__remus_ctx {
+    /* checkpoint interval */
+    int interval;
+    int blackhole;
+    int compression;
+    /* Script to setup/teardown network buffers */
+    const char *netbufscript;
+    /* Opaque context containing network buffer related stuff */
+    void *netbuf_ctx;
+} libxl__remus_ctx;
+
+_hidden int libxl__remus_netbuf_setup(libxl__gc *gc, uint32_t domid,
+                                     libxl__remus_ctx *remus_ctx);
+
+_hidden int libxl__remus_netbuf_teardown(libxl__gc *gc, uint32_t domid,
+                                        libxl__remus_ctx *remus_ctx);
+
+_hidden int libxl__remus_netbuf_start_new_epoch(libxl__gc *gc, uint32_t domid,
+                                               libxl__remus_ctx *remus_ctx);
+
+_hidden int libxl__remus_netbuf_release_prev_epoch(libxl__gc *gc, uint32_t domid,
+                                                  libxl__remus_ctx *remus_ctx);
+
 struct libxl__domain_suspend_state {
     /* set by caller of libxl__domain_suspend */
     libxl__ao *ao;
@@ -2252,7 +2275,7 @@ struct libxl__domain_suspend_state {
     libxl_domain_type type;
     int live;
     int debug;
-    const libxl_domain_remus_info *remus;
+    libxl__remus_ctx *remus_ctx;
     /* private */
     xc_evtchn *xce; /* event channel handle */
     int suspend_eventchn;
@@ -2260,7 +2283,6 @@ struct libxl__domain_suspend_state {
     int xcflags;
     int guest_responded;
     const char *dm_savefile;
-    int interval; /* checkpoint interval (for Remus) */
     libxl__save_helper_state shs;
     libxl__logdirty_switch logdirty;
     /* private for libxl__domain_save_device_model */

diff -r 4d24d36d30c3 -r 4b23104828c0 tools/libxl/libxl_netbuffer.c

tools/libxl/libxl_netbuffer.c new file +434 -0
@@ -0,0 +1,434 @@
+/*
+ * Copyright (C) 2013
+ * Author Shriram Rajagopalan <[email protected]>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser General Public License as published
+ * by the Free Software Foundation; version 2.1 only. with the special
+ * exception on linking described in file LICENSE.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Lesser General Public License for more details.
+ */
+
+#include "libxl_osdeps.h" /* must come before any other headers */
+
+#include "libxl_internal.h"
+
+#include <netlink/cache.h>
+#include <netlink/socket.h>
+#include <netlink/attr.h>
+#include <netlink/route/link.h>
+#include <netlink/route/route.h>
+#include <netlink/route/qdisc.h>
+#include <netlink/route/qdisc/plug.h>
+
+typedef struct libxl__remus_netbuf_ctx {
+    struct rtnl_qdisc **netbuf_qdisc_list;
+    struct nl_sock *nlsock;
+    struct nl_cache *qdisc_cache;
+    char **vif_list;
+    char **ifb_list;
+    uint32_t num_netbufs;
+    uint32_t unused;
+} libxl__remus_netbuf_ctx;
+
+/* If the device has a vifname, then use that instead of
+ * the vifX.Y format.
+ */
+static char *get_vifname(libxl__gc *gc, uint32_t domid,
+                         libxl_device_nic *nic)
+{
+    char *vifname = NULL;
+    vifname = libxl__xs_read(gc, XBT_NULL,
+                             libxl__sprintf(gc,
+                                            "%s/backend/vif/%d/%d/vifname",
+                                            libxl__xs_get_dompath(gc, 0),
+                                            domid, nic->devid));
+    if (!vifname) {
+        vifname = (char *)libxl__device_nic_devname(gc, domid,
+                                                    nic->devid,
+                                                    nic->nictype);
+    }
+
+    return vifname;
+}
+
+static char **get_guest_vif_list(libxl__gc *gc, uint32_t domid,
+                                 int *num_vifs)
+{
+    libxl_device_nic *nics;
+    int nb, i = 0;
+    char **vif_list = NULL;
+
+    nics = libxl_device_nic_list(CTX, domid, &nb);
+    if (!nics) {
+        *num_vifs = 0;
+        return NULL;
+    }
+
+    vif_list = libxl__calloc(gc, nb, sizeof(char *));
+    for (i = 0; i < nb; ++i) {
+        vif_list[i] = get_vifname(gc, domid, &nics[i]);
+        libxl_device_nic_dispose(&nics[i]);
+    }
+    free(nics);
+
+    *num_vifs = nb;
+    return vif_list;
+}
+
+static void netbuf_script_child_death_cb(libxl__egc *egc,
+                                         libxl__ev_child *child,
+                                         pid_t pid, int status)
+{
+    /* No-op. hotplug-error will be read in setup/teardown_ifb */
+    return;
+}
+
+/* the script needs the following env & args
+ * $vifname
+ * $XENBUS_PATH (/libxl/<domid>/remus/netbuf/<devid>/)
+ * setup/teardown as command line arg.
+ * In return, the script writes the name of IFB device to be used for
+ * output buffering into XENBUS_PATH/ifb
+ *
+ * The setup and teardown ops are synchronous, as they are executed during
+ * the context of an asynchronous API call (libxl_domain_remus_start).
+ */
+static int libxl__exec_netbuf_script(libxl__gc *gc, uint32_t domid,
+                                     int devid, char *vif, char *script,
+                                     char *op)
+{
+    int arraysize, nr;
+    char **env = NULL, **args = NULL;
+    pid_t pid;
+    libxl__ev_child childw_out;
+
+    arraysize = 5; nr = 0;
+    GCNEW_ARRAY(env, arraysize);
+    env[nr++] = "vifname";
+    env[nr++] = vif;
+    env[nr++] = "XENBUS_PATH";
+    env[nr++] = GCSPRINTF("%s/remus/netbuf/%d",
+                          libxl__xs_libxl_path(gc, domid), devid);
+    env[nr++] = NULL;
+    assert(nr <= arraysize);
+
+    arraysize = 3; nr = 0;
+    GCNEW_ARRAY(args, arraysize);
+    args[nr++] = script;
+    args[nr++] = op;
+    args[nr++] = NULL;
+    assert(nr == arraysize);
+
+    libxl__ev_child_init(&childw_out);
+    LOG(DEBUG, "Calling netbuf script: %s %s", script, op);
+
+    /* Fork and exec netbuf script */
+    pid = libxl__ev_child_fork(gc, &childw_out, netbuf_script_child_death_cb);
+    if (pid == -1) {
+        LOG(ERROR, "unable to fork netbuf script %s", script);
+        return ERROR_FAIL;
+    }
+
+    if (!pid) {
+        /* child: Launch netbuf script */
+        libxl__exec(gc, -1, -1, -1, args[0], args, env);
+        /* notreached */
+        abort();
+    }
+
+    assert(libxl__ev_child_inuse(&childw_out));
+
+    return 0;
+}
+
+static int script_done_cb(libxl__gc *gc, uint32_t domid,
+                          const char *state, void *userdata)
+{
+    return 0; /* No-op callback */
+}
+
+static int libxl__netbuf_script_setup(libxl__gc *gc, uint32_t domid,
+                                      int devid, char *vif, char *script,
+                                      char **ifb)
+{
+    int rc;
+    char *out_path_base, *hotplug_error;
+
+    rc = libxl__exec_netbuf_script(gc, domid, devid, vif,
+                                   script, "setup");
+    if (rc) return rc;
+
+    out_path_base = GCSPRINTF("%s/remus/netbuf/%d",
+                              libxl__xs_libxl_path(gc, domid), devid);
+
+    /* Now wait for the result (XENBUS_PATH/hotplug-status).
+     * It doesnt matter what the result is. If the hotplug-status path
+     * appears, then we are good to go.
+     */
+    rc = libxl__wait_for_offspring(gc, domid, LIBXL_HOTPLUG_TIMEOUT,
+                                   script,
+                                   /* path */
+                                   GCSPRINTF("%s/hotplug-status",
+                                             out_path_base),
+                                   NULL /* state */,
+                                   NULL, script_done_cb, NULL);
+    if (rc) {
+        LOG(ERROR, "unable to wait for %s setup to complete", script);
+        return ERROR_FAIL;
+    }
+
+    hotplug_error = libxl__xs_read(gc, XBT_NULL,
+                                   GCSPRINTF("%s/hotplug-error",
+                                             out_path_base));
+    if (hotplug_error) {
+        LOG(ERROR, "netbuf script %s setup failed: %s",
+            script, hotplug_error);
+        return ERROR_FAIL;
+    }
+
+    *ifb = libxl__xs_read(gc, XBT_NULL,
+                          GCSPRINTF("%s/remus/netbuf/%d/ifb",
+                                    libxl__xs_libxl_path(gc, domid),
+                                    devid));
+    if (!(*ifb)) {
+        LOG(ERROR, "Cannot get ifb dev name for domain %u dev %s",
+            domid, vif);
+        return ERROR_FAIL;
+    }
+
+    return 0;
+}
+
+static int libxl__netbuf_script_teardown(libxl__gc *gc, uint32_t domid,
+                                         int devid, char *vif,
+                                         char *script)
+{
+    /* Nothing to wait for during tear down */
+    return libxl__exec_netbuf_script(gc, domid, devid, vif,
+                                     script, "teardown");
+}
+
+/* Scan through the list of vifs belonging to domid and
+ * invoke the netbufscript to setup the IFB device & plug qdisc
+ * for each vif. Then scan through the list of IFB devices to obtain
+ * a handle on the plug qdisc installed on these IFB devices.
+ * Network output buffering is controlled via these qdiscs.
+ */
+int libxl__remus_netbuf_setup(libxl__gc *gc, uint32_t domid,
+			      libxl__remus_ctx *remus_ctx)
+{
+    int i, ret, ifindex, num_netbufs = 0;
+    struct rtnl_link *ifb = NULL;
+    struct rtnl_qdisc *qdisc = NULL;
+    libxl__remus_netbuf_ctx *netbuf_ctx = NULL;
+
+    /* If the domain backend is a stubdom, do nothing. We dont
+     * support stubdom network buffering yet.
+     */
+    if (libxl_get_stubdom_id(CTX, domid)) {
+        LOG(ERROR, "Network buffering is not supported with stubdoms");
+        return ERROR_FAIL;
+    }
+
+    netbuf_ctx = libxl__zalloc(gc, sizeof(libxl__remus_netbuf_ctx));
+    netbuf_ctx->vif_list = get_guest_vif_list(gc, domid, &num_netbufs);
+    if (!num_netbufs) return 0;
+
+    netbuf_ctx->ifb_list = libxl__calloc(gc, num_netbufs,
+                                         sizeof(char *));
+
+    /* convenience vars */
+    char **vif_list = netbuf_ctx->vif_list;
+    char **ifb_list = netbuf_ctx->ifb_list;
+
+    for (i = 0; i < num_netbufs; ++i) {
+
+        /* The setup script does the following
+         * find a free IFB to act as buffer for the vif.
+         * set up the plug qdisc on the IFB.
+         */
+        ret = libxl__netbuf_script_setup(gc, domid, i, vif_list[i],
+                                         (char *) remus_ctx->netbufscript,
+                                         &ifb_list[i]);
+        if (ret) {
+            LOG(ERROR, "Failed to setup ifb device for dev %s",
+                vif_list[i]);
+            return ERROR_FAIL;
+        }
+
+        LOG(DEBUG, "dev %s will be buffered at ifb %s", vif_list[i],
+            ifb_list[i]);
+    }
+
+    /* Now that we have brought up IFB devices with plug qdisc for
+     * each vif, lets get a netlink handle on the plug qdisc for use
+     * during checkpointing.
+     */
+    netbuf_ctx->nlsock = nl_socket_alloc();
+    if (!netbuf_ctx->nlsock) {
+        LOG(ERROR, "cannot allocate nl socket");
+        return ERROR_FAIL;
+    }
+
+    ret = nl_connect(netbuf_ctx->nlsock, NETLINK_ROUTE);
+    if (ret) {
+        LOG(ERROR, "failed to open netlink socket: %s",
+            nl_geterror(ret));
+        return ERROR_FAIL;
+    }
+
+    /* get list of all qdiscs installed on network devs. */
+    ret = rtnl_qdisc_alloc_cache(netbuf_ctx->nlsock,
+                                 &netbuf_ctx->qdisc_cache);
+    if (ret) {
+        LOG(ERROR, "failed to allocate qdisc cache: %s",
+            nl_geterror(ret));
+        goto end;
+    }
+
+    /* list of handles to plug qdiscs */
+    netbuf_ctx->netbuf_qdisc_list =
+        libxl__calloc(gc, num_netbufs,
+                      sizeof(struct rtnl_qdisc *));
+
+    ifb = rtnl_link_alloc();
+
+    for (i = 0; i < num_netbufs; ++i) {
+
+        /* get a handle to the IFB interface */
+        ret = rtnl_link_get_kernel(netbuf_ctx->nlsock, 0,
+                                   ifb_list[i], &ifb);
+        if (ret) {
+            LOG(ERROR, "cannot obtain handle for %s: %s", ifb_list[i],
+                nl_geterror(ret));
+            goto end;
+        }
+
+        ifindex = rtnl_link_get_ifindex(ifb);
+        if (!ifindex) {
+            LOG(ERROR, "interface %s has no index", ifb_list[i]);
+            goto end;
+        }
+
+        /* Get a reference to the root qdisc installed on the IFB, by
+         * querying the qdisc list we obtained earlier. The netbufscript
+         * sets up the plug qdisc as the root qdisc, so we dont have to
+         * search the entire qdisc tree on the IFB dev.
+
+         * There is no need to explicitly free this qdisc as its just a
+         * reference from the qdisc cache we allocated earlier.
+         */
+        qdisc = rtnl_qdisc_get_by_parent(netbuf_ctx->qdisc_cache, ifindex,
+                                         TC_H_ROOT);
+
+        if (qdisc) {
+            const char *tc_kind = rtnl_tc_get_kind(TC_CAST(qdisc));
+            /* Sanity check: Ensure that the root qdisc is a plug qdisc. */
+            if (!tc_kind || strcmp(tc_kind, "plug")) {
+                LOG(ERROR, "plug qdisc is not installed on %s", ifb_list[i]);
+                goto end;
+            }
+            netbuf_ctx->netbuf_qdisc_list[i] = qdisc;
+        } else {
+            LOG(ERROR, "Cannot get qdisc handle from ifb %s", ifb_list[i]);
+            goto end;
+        }
+    }
+
+    rtnl_link_put(ifb);
+
+    netbuf_ctx->num_netbufs = num_netbufs;
+
+    /* Update remus_ctx to point to the newly setup netbuffer context */
+    remus_ctx->netbuf_ctx = netbuf_ctx;
+    return 0;
+
+ end:
+    if (ifb) rtnl_link_put(ifb);
+    if (netbuf_ctx->qdisc_cache) nl_cache_free(netbuf_ctx->qdisc_cache);
+    if (netbuf_ctx->nlsock) nl_close(netbuf_ctx->nlsock);
+    return ERROR_FAIL;
+}
+
+/* Note: This function will be called in the same gc context as
+ * libxl__remus_netbuf_setup, created during the libxl_domain_remus_start
+ * API call.
+ */
+int libxl__remus_netbuf_teardown(libxl__gc *gc, uint32_t domid,
+				 libxl__remus_ctx *remus_ctx)
+{
+    libxl__remus_netbuf_ctx *netbuf_ctx = remus_ctx->netbuf_ctx;
+    char **vif_list = NULL, **ifb_list = NULL;
+    int i;
+
+    if (!(remus_ctx && netbuf_ctx && netbuf_ctx->num_netbufs))
+        return 0;
+
+    nl_cache_free(netbuf_ctx->qdisc_cache);
+    nl_close(netbuf_ctx->nlsock);
+
+    vif_list = netbuf_ctx->vif_list;
+    ifb_list = netbuf_ctx->ifb_list;
+
+    for (i = 0; i < netbuf_ctx->num_netbufs; ++i)
+        libxl__netbuf_script_teardown(gc, domid, i, vif_list[i],
+                                      (char *) remus_ctx->netbufscript);
+
+    return 0;
+}
+
+#define TC_BUFFER_START 1
+#define TC_BUFFER_RELEASE 2
+static int remus_netbuf_op(libxl__gc *gc, uint32_t domid,
+			   libxl__remus_ctx *remus_ctx,
+			   int buffer_op)
+{
+    int i, ret;
+    libxl__remus_netbuf_ctx *netbuf_ctx = remus_ctx->netbuf_ctx;
+
+    for (i = 0; i < netbuf_ctx->num_netbufs; ++i) {
+        if (buffer_op == TC_BUFFER_START)
+            ret = rtnl_qdisc_plug_buffer(netbuf_ctx->netbuf_qdisc_list[i]);
+        else
+            ret = rtnl_qdisc_plug_release_one(netbuf_ctx->netbuf_qdisc_list[i]);
+
+        if (!ret)
+            ret = rtnl_qdisc_add(netbuf_ctx->nlsock,
+                                 netbuf_ctx->netbuf_qdisc_list[i],
+                                 NLM_F_REQUEST);
+        if (ret) {
+            LOG(ERROR, "Remus: cannot do netbuf op %s on %s:%s",
+                ((buffer_op == TC_BUFFER_START) ?
+                 "start_new_epoch" : "release_prev_epoch"),
+                netbuf_ctx->ifb_list[i], nl_geterror(ret));
+            return ERROR_FAIL;
+        }
+    }
+
+    return 0;
+}
+
+int libxl__remus_netbuf_start_new_epoch(libxl__gc *gc, uint32_t domid,
+					libxl__remus_ctx *remus_ctx)
+{
+    return remus_netbuf_op(gc, domid, remus_ctx, TC_BUFFER_START);
+}
+
+int libxl__remus_netbuf_release_prev_epoch(libxl__gc *gc, uint32_t domid,
+					   libxl__remus_ctx *remus_ctx)
+{
+    return remus_netbuf_op(gc, domid, remus_ctx, TC_BUFFER_RELEASE);
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */

diff -r 4d24d36d30c3 -r 4b23104828c0 tools/libxl/libxl_nonetbuffer.c

tools/libxl/libxl_nonetbuffer.c new file +54 -0
@@ -0,0 +1,54 @@
+/*
+ * Copyright (C) 2013
+ * Author Shriram Rajagopalan <[email protected]>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser General Public License as published
+ * by the Free Software Foundation; version 2.1 only. with the special
+ * exception on linking described in file LICENSE.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Lesser General Public License for more details.
+ */
+
+#include "libxl_osdeps.h" /* must come before any other headers */
+
+#include "libxl_internal.h"
+
+/* Remus network buffer related stubs */
+int libxl__remus_netbuf_setup(libxl__gc *gc, uint32_t domid,
+			      libxl__remus_ctx *remus_ctx)
+{
+    LOG(ERROR, "Remus: No support for network buffering");
+    return ERROR_FAIL;
+}
+
+int libxl__remus_netbuf_teardown(libxl__gc *gc, uint32_t domid,
+				 libxl__remus_ctx *remus_ctx)
+{
+    return 0; /* No point complaining in a teardown call. */
+}
+
+int libxl__remus_netbuf_start_new_epoch(libxl__gc *gc, uint32_t domid,
+					libxl__remus_ctx *remus_ctx)
+{
+    LOG(ERROR, "Remus: No support for network buffering");
+    return ERROR_FAIL;
+}
+
+int libxl__remus_netbuf_release_prev_epoch(libxl__gc *gc, uint32_t domid,
+					   libxl__remus_ctx *remus_ctx)
+{
+    LOG(ERROR, "Remus: No support for network buffering");
+    return ERROR_FAIL;
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */

diff -r 4d24d36d30c3 -r 4b23104828c0 tools/libxl/libxl_types.idl

tools/libxl/libxl_types.idl +2 -0
@@ -526,6 +526,8 @@ libxl_domain_remus_info = Struct("domain
     ("interval",     integer),
     ("blackhole",    bool),
     ("compression",  bool),
+    ("netbuf",  bool),
+    ("netbufscript", string),
     ])
 
 libxl_event_type = Enumeration("event_type", [

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

#10 lore raw

Re: [PATCH 4 of 5] tools/libxl: Control network buffering in remus callbacks

In reply to [email protected]
Quoted 20 lines · On 26/08/2013 00:45, [email protected] wrote:

# HG changeset patch
# User Shriram Rajagopalan <[email protected]> # Date 1377473611 25200
# Node ID c6804ccfe660cb9c373c5f53a8996d0443316684 # Parent 4b23104828c09218aa7f9fbde1578bb6706e13d6 tools/libxl: Control network buffering in remus callbacks

This patch constitutes the core network buffering logic. and does the following:
a) create a new network buffer when the domain is suspended (remus_domain_suspend_callback)
b) release the previous network buffer pertaining to the committed checkpoint (remus_domain_checkpoint_dm_saved)

diff -r 4b23104828c0 -r c6804ccfe660 tools/libxl/libxl_dom.c

tools/libxl/libxl_dom.c int libxl__toolstack_save(uint32_t domid · line 1216

Quoted 4 lines
tools/libxl/libxl_dom.c +1 -0
@@ -1213,12 +1213,28 @@ int libxl__toolstack_save(uint32_t domid
 
 /*----- remus callbacks -----*/
 
+/* REMUS TODO: Issue disk checkpoint reqs. */

Why does this comment need to move?

tools/libxl/libxl_dom.c int libxl__toolstack_save(uint32_t domid

Quoted 12 lines
tools/libxl/libxl_dom.c +8 -2
 static int libxl__remus_domain_suspend_callback(void *data)
 {
-    /* REMUS TODO: Issue disk and network checkpoint reqs. */
-    return libxl__domain_suspend_common_callback(data);
+    libxl__save_helper_state *shs = data;
+    libxl__domain_suspend_state *dss = CONTAINER_OF(shs, *dss, shs);
+    libxl__remus_ctx *remus_ctx = dss->remus_ctx;
+    int is_suspended = 0;
+    STATE_AO_GC(dss->ao);
+
+    is_suspended = libxl__domain_suspend_common_callback(data);
+    if (!remus_ctx->netbuf_ctx) 

split to new line please.

tools/libxl/libxl_dom.c int libxl__toolstack_save(uint32_t domid · line 1234

Quoted 9 lines
tools/libxl/libxl_dom.c +8 -0
 return is_suspended;
+
+    if (is_suspended) {
+        if (libxl__remus_netbuf_start_new_epoch(gc, dss->domid,
+                                                remus_ctx))
+            return !is_suspended;
+    }
+
+    return is_suspended;

is_suspended is logically a boolean, so should be bool_t. Unhelpfully, libxl__domain_suspend_common_callback() returns an int, although its implementation strictly returns 0 on failure and 1 on success.

IMO, It would probably be best to have "bool_t is_suspended" set to "!!libxl__domain_suspend_common_callback()", at which the subsequent return !is_suspended doesn't look as suspect, although that is just a matter of style.

~Andrew

Quoted 58 lines
tools/libxl/libxl_dom.c +1 -0
 }
 
+/* REMUS TODO: Deal with disk. */
 static int libxl__remus_domain_resume_callback(void *data)
 {
     libxl__save_helper_state *shs = data;
tools/libxl/libxl_dom.c +28 -5
@@ -1229,7 +1245,6 @@ static int libxl__remus_domain_resume_ca
     if (libxl__domain_resume(gc, dss->domid, /* Fast Suspend */1))
         return 0;
 
-    /* REMUS TODO: Deal with disk. Start a new network output buffer */
     return 1;
 }
 
@@ -1256,10 +1271,34 @@ static void libxl__remus_domain_checkpoi
 static void remus_checkpoint_dm_saved(libxl__egc *egc,
                                       libxl__domain_suspend_state *dss, int rc)
 {
-    /* REMUS TODO: Wait for disk and memory ack, release network buffer */
-    /* REMUS TODO: make this asynchronous */
-    assert(!rc); /* REMUS TODO handle this error properly */
-    usleep(dss->remus_ctx->interval * 1000);
+    /* REMUS TODO: Wait for disk and explicit memory ack (through restore
+       callback from remote) before releasing network buffer. */
+    libxl__remus_ctx *remus_ctx = dss->remus_ctx;
+    struct timespec epoch;
+    int ret;
+    STATE_AO_GC(dss->ao);
+
+    if (rc) {
+        LOG(ERROR, "Failed to save device model. Terminating Remus..");
+        libxl__xc_domain_saverestore_async_callback_done(egc,
+                                                         &dss->shs, rc);
+        return;
+    }
+
+    if (remus_ctx->netbuf_ctx) {
+        ret = libxl__remus_netbuf_release_prev_epoch(gc, dss->domid,
+                                                     remus_ctx);
+        if (ret) {
+            libxl__xc_domain_saverestore_async_callback_done(egc,
+                                                             &dss->shs,
+                                                             ret);
+            return;
+        }
+    }
+
+    epoch.tv_sec = remus_ctx->interval / 1000; /* interval is in ms */
+    epoch.tv_nsec = remus_ctx->interval * 1000L * 1000L;
+    nanosleep(&epoch, 0);
     libxl__xc_domain_saverestore_async_callback_done(egc, &dss->shs, 1);
 }
 

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

#3 lore raw

Re: [PATCH 1 of 5] remus: add libnl3 dependency to autoconf scripts

In reply to Andrew Cooper
Quoted 20 lines · Andrew Cooper
Quoted 7 lines · On 26/08/2013 00:45, [email protected] wrote:

# HG changeset patch
# User Shriram Rajagopalan <[email protected]> # Date 1377473602 25200
# Node ID 0ae8b121b67b3cb3a6b00f2cc97e98fec5758e20 # Parent 40b079bd57dea24c3b000f233ed123763483a4d5
remus: add libnl3 dependency to autoconf scripts

Quoted 6 lines

Libnl3 is required for controlling Remus network buffering. This patch adds dependency on libnl3 (>= 3.2.8) to autoconf scripts. Also provide ability to configure tools without libnl3 support, that is without network buffering support.

As identified in the patch 0, libnl3 is not present for CentOS (so by implication RHEL and derivatives) and various BSDs, so enabling it by default and erroring out of autoconf if it cant be found is not really acceptable imo.

Quoted 4 lines

It would be better to detect the presence/absence of an appropriate libnl3, and use that as the default. Then, the logic would be: "By default build in libnl3 support if it is available, but don't if it is not.

This is the minimum we should require IMHO

Quoted 2 lines

If the user explicitlicy configures --with-remusnetbuf and libnl3
is not present, present an error"

The presence of an option to force enable it is a stretch goal as far as I'm concerned.

Ian.

#4 lore raw

Re: [PATCH 1 of 5] remus: add libnl3 dependency to autoconf scripts

In reply to [email protected]

tools/configure.ac AC_SUBST(libiconv) · line 220

Quoted 8 lines · On Sun, 2013-08-25 at 16:45 -0700, [email protected] wrote:
tools/configure.ac +8 -0
+AS_IF([test "x$remusnetbuf" = "xy"], [
+
+dnl Need libnl version 3.2.8 or higher for plug qdisc support
+  AS_IF(
+       [ pkg-config --exists 'libnl-3.0 >= 3.2.8 libnl-route-3.0 >= 3.2.8 libnl-cli-3.0 >= 3.2.8' ],
+       [], [
+           AC_MSG_ERROR([Need libnl3, libnl-route-3 and libnl-cli-3 (>= 3.2.8) for Remus network buffering])
+       ])

At least on Debian pkg-config ships pkg.m4 which contains handy looking autoconf macros for doing pkg-config stuff. Better to use them, gets you free cross compile support and general consistency with other projects.

Quoted 4 lines
tools/configure.ac +4 -0
+
+  AC_CHECK_HEADER([netlink/route/qdisc/plug.h], [
+	   AC_CHECK_LIB(nl-3, nl_socket_alloc, [
+		AC_CHECK_LIB(nl-route-3, rtnl_qdisc_alloc, [

tools/configure.ac AC_SUBST(libiconv) · line 234

Quoted 10 lines
tools/configure.ac +10 -0
+			LIBNL3_LIBS="-lnl-3 -lnl-route-3"
+			AC_SUBST(LIBNL3_LIBS)
+		], [
+		   AC_MSG_ERROR([Could not find libnl-route-3])
+		])
+	   ], [
+	      AC_MSG_ERROR([Could not find libnl3])
+	   ])
+	], [
+	   AC_MSG_ERROR([Could not find libnl3 dev headers (esp. plug.h)])

Is there any chance of this failing if pkg-config says it exists? IOW isn't this test redundant?

Ian.

#5 lore raw

Re: [PATCH 1 of 5] remus: add libnl3 dependency to autoconf scripts

Quoted 31 lines · Ian Campbell

tools/configure.ac AC_SUBST(libiconv)

Quoted 5 lines · On Sun, 2013-08-25 at 16:45 -0700, [email protected] wrote:
tools/configure.ac +5 -0
+AS_IF([test "x$remusnetbuf" = "xy"], [
+
+dnl Need libnl version 3.2.8 or higher for plug qdisc support
+  AS_IF(
+       [ pkg-config --exists 'libnl-3.0 >= 3.2.8 libnl-route-3.0 >=

3.2.8 libnl-cli-3.0 >= 3.2.8' ],

Review

Quoted 2 lines
(quoted diff) +2 -0
+       [], [
+           AC_MSG_ERROR([Need libnl3, libnl-route-3 and libnl-cli-3 (>=

3.2.8) for Remus network buffering])

Quoted 1 line

+ ])

At least on Debian pkg-config ships pkg.m4 which contains handy looking autoconf macros for doing pkg-config stuff. Better to use them, gets you free cross compile support and general consistency with other projects.

Quoted 4 lines
tools/configure.ac +4 -0
+
+  AC_CHECK_HEADER([netlink/route/qdisc/plug.h], [
+        AC_CHECK_LIB(nl-3, nl_socket_alloc, [
+             AC_CHECK_LIB(nl-route-3, rtnl_qdisc_alloc, [

tools/configure.ac AC_SUBST(libiconv) · line 234

Quoted 10 lines
tools/configure.ac +10 -0
+                     LIBNL3_LIBS="-lnl-3 -lnl-route-3"
+                     AC_SUBST(LIBNL3_LIBS)
+             ], [
+                AC_MSG_ERROR([Could not find libnl-route-3])
+             ])
+        ], [
+           AC_MSG_ERROR([Could not find libnl3])
+        ])
+     ], [
+        AC_MSG_ERROR([Could not find libnl3 dev headers (esp. plug.h)])
Quoted 6 lines

Is there any chance of this failing if pkg-config says it exists? IOW isn't this test redundant?

Ian.

Yeah, that is redundant. [my autoconf naiveté ]. I was under the impression that the ac_check_lib
would automatically add the linker flags when compiling code, only to realize later that it didnt work
for libxenlight.so. So I ended up explicitly using LIBNL3_LIBS (in the libxl makefile)

will fix this and the other suggestions: I was thinking of getting rid of that config option altogether.
Compile in libnl3 if available (and enable libxl_netbuffer.c). If not, issue a warning and compile libxl_nonetbuffer.c.

shriram

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

#11 lore raw

Re: [PATCH 4 of 5] tools/libxl: Control network buffering in remus callbacks

In reply to Andrew Cooper
Quoted 57 lines · Andrew Cooper
Quoted 20 lines · On 26/08/2013 00:45, [email protected] wrote:

# HG changeset patch
# User Shriram Rajagopalan <[email protected]> # Date 1377473611 25200
# Node ID c6804ccfe660cb9c373c5f53a8996d0443316684 # Parent 4b23104828c09218aa7f9fbde1578bb6706e13d6 tools/libxl: Control network buffering in remus callbacks

This patch constitutes the core network buffering logic. and does the following:
a) create a new network buffer when the domain is suspended (remus_domain_suspend_callback)
b) release the previous network buffer pertaining to the committed checkpoint (remus_domain_checkpoint_dm_saved)

diff -r 4b23104828c0 -r c6804ccfe660 tools/libxl/libxl_dom.c

tools/libxl/libxl_dom.c Sun Aug 25 16:33:31 2013 -0700 int libxl__toolstack_save(uint32_t domid · line 1216

Quoted 4 lines
tools/libxl/libxl_dom.c Sun Aug 25 16:33:31 2013 -0700 renamed from tools/libxl/libxl_dom.c Sun Aug 25 16:33:29 2013 -0700 +1 -0
@@ -1213,12 +1213,28 @@ int libxl__toolstack_save(uint32_t domid
 
 /*----- remus callbacks -----*/
 
+/* REMUS TODO: Issue disk checkpoint reqs. */

Why does this comment need to move?

tools/libxl/libxl_dom.c Sun Aug 25 16:33:31 2013 -0700 int libxl__toolstack_save(uint32_t domid · line 1226

Quoted 12 lines
tools/libxl/libxl_dom.c Sun Aug 25 16:33:31 2013 -0700 renamed from tools/libxl/libxl_dom.c Sun Aug 25 16:33:29 2013 -0700 +8 -2
 static int libxl__remus_domain_suspend_callback(void *data)
 {
-    /* REMUS TODO: Issue disk and network checkpoint reqs. */
-    return libxl__domain_suspend_common_callback(data);
+    libxl__save_helper_state *shs = data;
+    libxl__domain_suspend_state *dss = CONTAINER_OF(shs, *dss, shs);
+    libxl__remus_ctx *remus_ctx = dss->remus_ctx;
+    int is_suspended = 0;
+    STATE_AO_GC(dss->ao);
+
+    is_suspended = libxl__domain_suspend_common_callback(data);
+    if (!remus_ctx->netbuf_ctx)

split to new line please.

tools/libxl/libxl_dom.c Sun Aug 25 16:33:31 2013 -0700 int libxl__toolstack_save(uint32_t domid · line 1235

Quoted 9 lines
tools/libxl/libxl_dom.c Sun Aug 25 16:33:31 2013 -0700 renamed from tools/libxl/libxl_dom.c Sun Aug 25 16:33:29 2013 -0700 +8 -0
 return is_suspended;
+
+    if (is_suspended) {
+        if (libxl__remus_netbuf_start_new_epoch(gc, dss->domid,
+                                                remus_ctx))
+            return !is_suspended;
+    }
+
+    return is_suspended;

is_suspended is logically a boolean, so should be bool_t. Unhelpfully, libxl__domain_suspend_common_callback() returns an int, although its implementation strictly returns 0 on failure and 1 on success.

Quoted 6 lines

IMO, It would probably be best to have "bool_t is_suspended" set to "!!libxl__domain_suspend_common_callback()", at which the subsequent return !is_suspended doesn't look as suspect, although that is just a matter of style.

Did you mean bool (not bool_t) ? Because I didnt see a typedef in libxl folder.

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

#12 lore raw

Re: [PATCH 4 of 5] tools/libxl: Control network buffering in remus callbacks

Quoted 71 lines · Shriram Rajagopalan

On Mon, Aug 26, 2013 at 6:28 PM, Andrew Cooper <[email protected] <mailto:[email protected]>> wrote:

    On 26/08/2013 00:45, [email protected]
    <mailto:[email protected]> wrote:
    > # HG changeset patch
    > # User Shriram Rajagopalan <[email protected]
    <mailto:[email protected]>>
    > # Date 1377473611 25200
    > # Node ID c6804ccfe660cb9c373c5f53a8996d0443316684
    > # Parent  4b23104828c09218aa7f9fbde1578bb6706e13d6
    > tools/libxl: Control network buffering in remus callbacks
    >
    > This patch constitutes the core network buffering logic.
    > and does the following:
    >  a) create a new network buffer when the domain is suspended
    >     (remus_domain_suspend_callback)
    >  b) release the previous network buffer pertaining to the
    >     committed checkpoint (remus_domain_checkpoint_dm_saved)
    >
    > Signed-off-by: Shriram Rajagopalan <[email protected]
    <mailto:[email protected]>>
    >
    > diff -r 4b23104828c0 -r c6804ccfe660 tools/libxl/libxl_dom.c
    > --- a/tools/libxl/libxl_dom.c Sun Aug 25 16:33:29 2013 -0700
    > +++ b/tools/libxl/libxl_dom.c Sun Aug 25 16:33:31 2013 -0700
    > @@ -1213,12 +1213,28 @@ int libxl__toolstack_save(uint32_t domid
    >
    >  /*----- remus callbacks -----*/
    >
    > +/* REMUS TODO: Issue disk checkpoint reqs. */

    Why does this comment need to move?

    >  static int libxl__remus_domain_suspend_callback(void *data)
    >  {
    > -    /* REMUS TODO: Issue disk and network checkpoint reqs. */
    > -    return libxl__domain_suspend_common_callback(data);
    > +    libxl__save_helper_state *shs = data;
    > +    libxl__domain_suspend_state *dss = CONTAINER_OF(shs, *dss,
    shs);
    > +    libxl__remus_ctx *remus_ctx = dss->remus_ctx;
    > +    int is_suspended = 0;
    > +    STATE_AO_GC(dss->ao);
    > +
    > +    is_suspended = libxl__domain_suspend_common_callback(data);
    > +    if (!remus_ctx->netbuf_ctx)

    split to new line please.

    > return is_suspended;
    > +
    > +    if (is_suspended) {
    > +        if (libxl__remus_netbuf_start_new_epoch(gc, dss->domid,
    > +                                                remus_ctx))
    > +            return !is_suspended;
    > +    }
    > +
    > +    return is_suspended;

    is_suspended is logically a boolean, so should be bool_t.
     Unhelpfully,
    libxl__domain_suspend_common_callback() returns an int, although its
    implementation strictly returns 0 on failure and 1 on success.

    IMO, It would probably be best to have "bool_t is_suspended" set to
    "!!libxl__domain_suspend_common_callback()", at which the subsequent
    return !is_suspended doesn't look as suspect, although that is just a
    matter of style.
Quoted 3 lines

Did you mean bool (not bool_t) ? Because I didnt see a typedef in libxl folder.

Indeed. Sorry. (I had Xen on the brain when reviewing, which does use bool_t rather than bool).

~Andrew

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel