Skip to contentSkip to conversation

Remus/Libxl: Network buffering support

PATCH In review Shriram Rajagopalan 4 participants 22 messages Nov 18, 2013 – Dec 19, 2013

lore
Share & tools
lore
#0 lore raw

PATCH v5 1/1 Remus/Libxl: Network buffering support

This patch series adds support for network buffering in the Remus codebase in libxl. NB: This series does not contain the bug fix related to usleep calls in libxl__domain_suspend. I will send it out later as a separate series.

Changes in V5:

Merge hotplug script patch (2/5) and hotplug script setup/teardown patch (3/5) into a single patch.

Changes in V4:

[1/5] Remove check for libnl command line utils in autoconf checks

[2/5] minor nits

[3/5] define LIBXL_HAVE_REMUS_NETBUF in libxl.h

[4/5] clean ups. Make the usleep in checkpoint callback asynchronous

[5/5] minor nits

Changes in V3:
[1/5] Fix redundant checks in configure scripts
(based on Ian Campbell's suggestions)

[2/5] Introduce locking in the script, during IFB setup.
Add xenstore paths used by netbuf scripts to xenstore-paths.markdown

[3/5] Hotplug scripts setup/teardown invocations are now asynchronous
following IanJ's feedback. However, the invocations are still sequential.

[5/5] Allow per-domain specification of netbuffer scripts in xl remus
commmand.

And minor nits throughout the series based on feedback from the last version

Changes in V2:
[1/5] Configure script will automatically enable/disable network
buffer support depending on the availability of the appropriate libnl3 version. [If libnl3 is unavailable, a warning message will be printed to let the user know that the feature has been disabled.]

use macros from pkg.m4 instead of pkg-config commands removed redundant checks for libnl3 libraries.

[3,4/5] - Minor nits.

Version 1:

[1/5] Changes to autoconf scripts to check for libnl3. Add linker flags
to libxl Makefile.

[2/5] External script to setup/teardown network buffering using libnl3's
CLI. This script will be invoked by libxl before starting Remus. The script's main job is to bring up an IFB device with plug qdisc attached to it. It then re-routes egress traffic from the guest's vif to the IFB device.

[3/5] Libxl code to invoke the external setup script, followed by netlink
related setup to obtain a handle on the output buffers attached to each vif.

[4/5] Libxl interaction with network buffer module in the kernel via
libnl3 API.

[5/5] xl cmdline switch to explicitly enable network buffering when
starting remus.

Few things to note:

a) Based on previous email discussions, the setup/teardown task has been moved to a hotplug style shell script which can be customized as desired, instead of implementing it as C code inside libxl.

b) Libnl3 is not available on NetBSD. Nor is it available on CentOS
(Linux). So I have made network buffering support an optional feature so that it can be disabled if desired.

c) NetBSD does not have libnl3. So I have put the setup script under tools/hotplug/Linux folder.

thanks
shriram

#1 lore raw

PATCH v5 1/1 remus: add libnl3 dependency to autoconf scripts

# HG changeset patch
# User Shriram Rajagopalan <[email protected]> # Date 1384798655 28800
# Node ID 4b471809a5bb016afb2d24521b5f8f548057b29f # Parent 974c8559f8234d46c0e5cd06a5caca072548f4f8
remus: add libnl3 dependency to autoconf scripts

Libnl3 is required for controlling Remus network buffering. This patch adds dependency on libnl3 (>= 3.2.8) to autoconf scripts. Also provide ability to configure tools without libnl3 support, that is without network buffering support.

README +4 -0
diff -r 974c8559f823 -r 4b471809a5bb README
@@ -71,6 +71,10 @@ disabled at compile time:
       includes the alternative ocaml xenstored.
     * cmake (if building vtpm stub domains)
     * markdown
+    * Development install of libnl3 (e.g., libnl-3-200,
+      libnl-3-dev, etc).  Required if network buffering is desired
+      when using Remus with libxl.  See tools/remus/README for detailed
+      information.
 
 Second, you need to acquire a suitable kernel for use in domain 0. If
 possible you should use a kernel provided by your OS distributor. If
config/Tools.mk.in +3 -0
diff -r 974c8559f823 -r 4b471809a5bb config/Tools.mk.in
@@ -38,6 +38,8 @@ PTHREAD_LIBS        := @PTHREAD_LIBS@
 
 PTYFUNCS_LIBS       := @PTYFUNCS_LIBS@
 
+LIBNL3_LIBS         := @LIBNL3_LIBS@
+LIBNL3_CFLAGS       := @LIBNL3_CFLAGS@
 # Download GIT repositories via HTTP or GIT's own protocol?
 # GIT's protocol is faster and more robust, when it works at all (firewalls
 # may block it). We make it the default, but if your GIT repository downloads
@@ -56,6 +58,7 @@ CONFIG_QEMU_TRAD    := @qemu_traditional
 CONFIG_QEMU_XEN     := @qemu_xen@
 CONFIG_XEND         := @xend@
 CONFIG_BLKTAP1      := @blktap1@
+CONFIG_REMUS_NETBUF := @remus_netbuf@
 
 #System options
 ZLIB                := @zlib@
tools/configure.ac +15 -0
diff -r 974c8559f823 -r 4b471809a5bb tools/configure.ac
@@ -230,6 +230,21 @@ AC_SUBST(libiconv)
 # Checks for header files.
 AC_CHECK_HEADERS([yajl/yajl_version.h sys/eventfd.h])
 
+# Check for libnl3 >=3.2.8. If present enable remus network buffering.
+PKG_CHECK_MODULES(LIBNL3, [libnl-3.0 >= 3.2.8 libnl-route-3.0 >= 3.2.8],
+		[libnl3_lib="y"], [libnl3_lib="n"])
+
+AS_IF([test "x$libnl3_lib" = "xn" ], [
+	    AC_MSG_WARN([Disabling support for Remus network buffering.
+	    Please install libnl3 libraries, command line tools and devel
+	    headers - version 3.2.8 or higher])
+	    AC_SUBST(remus_netbuf, [n])
+	    ],[
+	    AC_SUBST(LIBNL3_LIBS)
+	    AC_SUBST(LIBNL3_CFLAGS)
+	    AC_SUBST(remus_netbuf, [y])
+])
+
 AC_OUTPUT()
 
 AS_IF([test "x$xend" = "xy" ], [
tools/libxl/Makefile +2 -0
diff -r 974c8559f823 -r 4b471809a5bb tools/libxl/Makefile
@@ -21,11 +21,13 @@ endif
 
 LIBXL_LIBS =
 LIBXL_LIBS = $(LDLIBS_libxenctrl) $(LDLIBS_libxenguest) $(LDLIBS_libxenstore) $(LDLIBS_libblktapctl) $(PTYFUNCS_LIBS) $(LIBUUID_LIBS)
+LIBXL_LIBS += $(LIBNL3_LIBS)
 
 CFLAGS_LIBXL += $(CFLAGS_libxenctrl)
 CFLAGS_LIBXL += $(CFLAGS_libxenguest)
 CFLAGS_LIBXL += $(CFLAGS_libxenstore)
 CFLAGS_LIBXL += $(CFLAGS_libblktapctl) 
+CFLAGS_LIBXL += $(LIBNL3_CFLAGS)
 CFLAGS_LIBXL += -Wshadow
 
 CFLAGS += $(PTHREAD_CFLAGS)
tools/remus/README +6 -0
diff -r 974c8559f823 -r 4b471809a5bb tools/remus/README
@@ -2,3 +2,9 @@ Remus provides fault tolerance for virtu
 checkpoints to a backup, which will activate if the target VM fails.
 
 See the website at http://nss.cs.ubc.ca/remus/ for details.
+
+Using Remus with libxl on Xen 4.4 and higher:
+ To enable network buffering, you need libnl 3.2.8
+ or higher along with the development headers and command line utilities.
+ If your distro does not have the appropriate libnl3 version, you can find
+ the latest source tarball of libnl3 at http://www.carisma.slowglass.com/~tgr/libnl/
#2 lore raw

PATCH v5 1/1 tools/libxl: Remus network buffering - hotplug scripts and setup code

In reply to Shriram Rajagopalan

# HG changeset patch
# User Shriram Rajagopalan <[email protected]>
# Date 1384801774 28800
# Node ID 5b4e029bb79b8db4210a182911a4ff7ec7957c4f
# Parent 4b471809a5bb016afb2d24521b5f8f548057b29f tools/libxl: Remus network buffering - hotplug scripts and setup code

This patch introduces remus-netbuf-setup hotplug script responsible for setting up and tearing down the necessary infrastructure required for network output buffering in Remus. This script is intended to be invoked by libxl for each guest interface, when starting or stopping Remus.

Apart from returning success/failure indication via the usual hotplug entries in xenstore, this script also writes to xenstore, the name of the IFB device to be used to control the vif's network output.

The script relies on libnl3 command line utilities to perform various setup/teardown functions. The script is confined to Linux platforms only since NetBSD does not seem to have libnl3.

Additionally, this patch introduces code to setup/teardown remus network buffering for a given guest, when libxl_domain_remus_start API is invoked.

The following steps are taken during setup:
a) call the hotplug script for each vif to setup its network buffer

b) establish a dedicated remus context containing libnl related state (netlink sockets, qdisc caches, etc.,)

c) Obtain handles to plug qdiscs installed on the IFB devices chosen by the hotplug scripts.

During teardown, the netlink resources are released, followed by invocation of hotplug scripts to remove the IFB devices.

docs/misc/xenstore-paths.markdown +4 -0
diff -r 4b471809a5bb -r 5b4e029bb79b docs/misc/xenstore-paths.markdown
@@ -356,6 +356,10 @@ The guest's virtual time offset from UTC
 
 The device model version for a domain.
 
+#### /libxl/$DOMID/remus/netbuf/$DEVID/ifb = STRING [n,INTERNAL]
+
+IFB device used by Remus to buffer network output from the associated vif.
+
 [BLKIF]: http://xenbits.xen.org/docs/unstable/hypercall/include,public,io,blkif.h.html
 [FBIF]: http://xenbits.xen.org/docs/unstable/hypercall/include,public,io,fbif.h.html
 [HVMPARAMS]: http://xenbits.xen.org/docs/unstable/hypercall/include,public,hvm,params.h.html
tools/hotplug/Linux/Makefile +1 -0
diff -r 4b471809a5bb -r 5b4e029bb79b tools/hotplug/Linux/Makefile
@@ -16,6 +16,7 @@ XEN_SCRIPTS += network-nat vif-nat
 XEN_SCRIPTS += vif-openvswitch
 XEN_SCRIPTS += vif2
 XEN_SCRIPTS += vif-setup
+XEN_SCRIPTS-$(CONFIG_REMUS_NETBUF) += remus-netbuf-setup
 XEN_SCRIPTS += block
 XEN_SCRIPTS += block-enbd block-nbd
 XEN_SCRIPTS-$(CONFIG_BLKTAP1) += blktap
tools/hotplug/Linux/remus-netbuf-setup new file +183 -0

183 lines not on this page · raw message

tools/libxl/Makefile +7 -0
diff -r 4b471809a5bb -r 5b4e029bb79b tools/libxl/Makefile
@@ -42,6 +42,13 @@ LIBXL_OBJS-y += libxl_blktap2.o
 else
 LIBXL_OBJS-y += libxl_noblktap2.o
 endif
+
+ifeq ($(CONFIG_REMUS_NETBUF),y)
+LIBXL_OBJS-y += libxl_netbuffer.o
+else
+LIBXL_OBJS-y += libxl_nonetbuffer.o
+endif
+
 LIBXL_OBJS-$(CONFIG_X86) += libxl_cpuid.o libxl_x86.o
 LIBXL_OBJS-$(CONFIG_IA64) += libxl_nocpuid.o libxl_noarch.o
 LIBXL_OBJS-$(CONFIG_ARM) += libxl_nocpuid.o libxl_noarch.o
tools/libxl/libxl.c +35 -17

83 lines not on this page · raw message

tools/libxl/libxl.h +13 -0
diff -r 4b471809a5bb -r 5b4e029bb79b tools/libxl/libxl.h
@@ -375,6 +375,19 @@
  */
 #define LIBXL_HAVE_CREATEINFO_PVH 1
 
+/*
+ * LIBXL_HAVE_REMUS_NETBUF 1
+ *
+ * If this is defined, then the libxl_domain_remus_info structure will
+ * have a boolean field (netbuf) and a string field (netbufscript).
+ *
+ * netbuf, if true, indicates that network buffering should be enabled.
+ *
+ * netbufscript, if set, indicates the path to the hotplug script to
+ * setup or teardown network buffers.
+ */
+#define LIBXL_HAVE_REMUS_NETBUF 1
+
 /* Functions annotated with LIBXL_EXTERNAL_CALLERS_ONLY may not be
  * called from within libxl itself. Callers outside libxl, who
  * do not #include libxl_internal.h, are fine. */
tools/libxl/libxl_dom.c +60 -8

104 lines not on this page · raw message

tools/libxl/libxl_internal.h +45 -2

65 lines not on this page · raw message

tools/libxl/libxl_netbuffer.c new file +516 -0

516 lines not on this page · raw message

tools/libxl/libxl_nonetbuffer.c new file +56 -0

56 lines not on this page · raw message

tools/libxl/libxl_types.idl +2 -0
diff -r 4b471809a5bb -r 5b4e029bb79b tools/libxl/libxl_types.idl
@@ -558,6 +558,8 @@ libxl_domain_remus_info = Struct("domain
     ("interval",     integer),
     ("blackhole",    bool),
     ("compression",  bool),
+    ("netbuf",       bool),
+    ("netbufscript", string),
     ])
 
 libxl_event_type = Enumeration("event_type", [
#3 lore raw

Re: [PATCH 2 of 4 V5] tools/libxl: Remus network buffering - hotplug scripts and setup code

Shriram Rajagopalan writes ("[PATCH 2 of 4 V5] tools/libxl: Remus network buffering - hotplug scripts and setup code"):

Quoted 1 line

tools/libxl: Remus network buffering - hotplug scripts and setup code

Thanks. This is going in the right direction, but I think there's still some way to go.

One thing that makes reviewing it particularly difficult is the physical ordering of the new code in your patch.

The callback programming model makes it easy to write spaghetti logic; to counter this we try to have a firm structure to each set of event handling code:

* Firstly, function forward declarations, types, structs, etc.

* Then helper functions which have a simple "call this and it does something and returns" control flow.

* Then the main body of the event callbacks, in the order in which they will normally be called. Obviously there is some judgement needed because the call flow won't always be entirely linear. But keeping the layout as linear as possible means that one is never hunting for the next function.

* Call flow of a single operation does not bounce back and forth between files. If this is necessary a formal sub-operation is defined (see how it's done with the bootloader, for example).

I think you would be able to structure the code this way. Would you do that please ?

I think you may need to move several of the entrypoints into a new "remus" file. The "netbuf" file may need to become a more formally separated part of the suspend/resume function.

We also try to break out sub-operations, giving them their own types and callbacks.

Quoted 9 lines

diff -r 4b471809a5bb -r 5b4e029bb79b tools/libxl/Makefile

tools/libxl/Makefile +3 -2
@@ -42,6 +42,13 @@ LIBXL_OBJS-y += libxl_blktap2.o
-static void remus_failover_cb(libxl__egc *egc,
-                              libxl__domain_suspend_state *dss, int rc);
+static void remus_replication_failure_cb(libxl__egc *egc,
+                                         libxl__domain_suspend_state *dss,
+                                         int rc);

...

Quoted 1 line

- dss->remus = info;

This change from dss->remus to dss->remus_ctx needs a mention in the commit message. To be honest, I'm not sure why you don't just change the type of the remus member.

Quoted 12 lines

diff -r 4b471809a5bb -r 5b4e029bb79b tools/libxl/libxl_internal.h

tools/libxl/libxl_internal.h +5 -0
@@ -2291,6 +2291,50 @@ typedef struct libxl__logdirty_switch {
     libxl__ev_time timeout;
 } libxl__logdirty_switch;
 
+typedef struct libxl__remus_ctx {
+    /* checkpoint interval */
+    int interval;
+    int blackhole;
+    int compression;

tools/libxl/libxl_internal.h typedef struct libxl__logdirty_switch { · line 2308

Quoted 10 lines
tools/libxl/libxl_internal.h +10 -0
+    int saved_rc;
+    /* Script to setup/teardown network buffers */
+    const char *netbufscript;
+    /* Opaque context containing network buffer related stuff */
+    void *netbuf_ctx;
+    libxl__domain_suspend_state *dss;
+    libxl__ev_time timeout;
+    libxl__ev_child child;
+    int num_exec;
+} libxl__remus_ctx;

I think you should probably call this a "libxl__remus_state", to correspond to all of the other "libxl__foo_state" structs.

You also need to specify in comments who fills in what parts of it, as is done in the other libxl__foo_state's.

Quoted 4 lines

diff -r 4b471809a5bb -r 5b4e029bb79b tools/libxl/libxl_netbuffer.c

tools/libxl/libxl_netbuffer.c new file +0 -0
@@ -0,0 +1,516 @@

...

tools/libxl/libxl_netbuffer.c line 36

Quoted 9 lines
tools/libxl/libxl_netbuffer.c new file +9 -0
+typedef struct libxl__remus_netbuf_ctx {
+    struct rtnl_qdisc **netbuf_qdisc_list;
+    struct nl_sock *nlsock;
+    struct nl_cache *qdisc_cache;
+    char **vif_list;
+    char **ifb_list;
+    uint32_t num_netbufs;
+    uint32_t unused;
+} libxl__remus_netbuf_ctx;

Presumably this wants to be a libxl__remus_netbuf_state, then.

The callbacks from the remus and netbuf code to the suspend code should not be hardwired. Instead, function pointers should be passed, the way the domain creation stuff does with a bootloader, and the way the bootloader does with openpty.

This may seem like extra complication if there is only one caller and it will always pass the same function, but it makes the logical separation much clearer. It's analogous to the separation between an ordinary function and its caller, even if there is in fact only one call site.

That is, functions like this

+_hidden void libxl__remus_setup_done(libxl__egc *egc,
+ libxl__domain_suspend_state *dss,
+ int rc);

should be private functions in the remus file which make the appropriate callback.

tools/libxl/libxl_netbuffer.c line 49

Quoted 12 lines
tools/libxl/libxl_netbuffer.c new file +12 -0
+/* If the device has a vifname, then use that instead of
+ * the vifX.Y format.
+ */
+static char *get_vifname(libxl__gc *gc, uint32_t domid,
+                               libxl_device_nic *nic)
+{
+    const char *vifname = NULL;
+    vifname = libxl__xs_read(gc, XBT_NULL,
+                             libxl__sprintf(gc,
+                                            "%s/backend/vif/%d/%d/vifname",
+                                            libxl__xs_get_dompath(gc, 0),
+                                            domid, nic->devid));

Why not libxl__xs_read_checked ?

Quoted 1 line

+ if (!vifname) {

Surely this error handling is wrong. NULL here might mean "failed" or ENOENT. If it means failed then get_vifname needs to return the error to its caller, by returning NULL.

tools/libxl/libxl_netbuffer.c line 56

Quoted 6 lines
tools/libxl/libxl_netbuffer.c new file +6 -0
+        vifname = libxl__device_nic_devname(gc, domid,
+                                            nic->devid,
+                                            nic->nictype);
+    }
+
+    return libxl__strdup(gc, vifname);

Why this strdup ?

Quoted 6 lines
tools/libxl/libxl_netbuffer.c new file +6 -0
+static char **get_guest_vif_list(libxl__gc *gc, uint32_t domid,
+                                 int *num_vifs)
+{
+    libxl_device_nic *nics = NULL;
+    int nb, i = 0;
+    char **vif_list = NULL;

tools/libxl/libxl_netbuffer.c line 74

Quoted 10 lines
tools/libxl/libxl_netbuffer.c new file +10 -0
+
+    *num_vifs = 0;
+    nics = libxl_device_nic_list(CTX, domid, &nb);
+    if (!nics) return NULL;
+
+    /* Ensure that none of the vifs are backed by driver domains */
+    for (i = 0; i < nb; i++) {
+        if (nics[i].backend_domid != LIBXL_TOOLSTACK_DOMID) {
+            LOG(ERROR, "vif %s has driver domain (%u) as its backend.\n"
+                "Network buffering is not supported with driver domains",

The LOG macros should not normally be passed messages with newlines in.

tools/libxl/libxl_netbuffer.c line 76

Quoted 2 lines
tools/libxl/libxl_netbuffer.c new file +2 -0
+                get_vifname(gc, domid, &nics[i]), nics[i].backend_domid);
+            *num_vifs = -1;

This calling convention is strange. I think get_guest_vif_list should return a libxl rc value.

tools/libxl/libxl_netbuffer.c line 83

Quoted 3 lines
tools/libxl/libxl_netbuffer.c new file +3 -0
+    vif_list = libxl__calloc(gc, nb, sizeof(char *));
+    for (i = 0; i < nb; ++i)
+        vif_list[i] = get_vifname(gc, domid, &nics[i]);

What if get_vifname fails ?

...

Quoted 7 lines
tools/libxl/libxl_netbuffer.c new file +7 -0
+
+static int init_qdiscs(libxl__gc *gc,
+                       libxl__remus_ctx *remus_ctx)
+{
+    int i, ret, ifindex;
+    struct rtnl_link *ifb = NULL;
+    struct rtnl_qdisc *qdisc = NULL;

...

Quoted 1 line

+ /* convenience vars */

We use the term "Convenience aliases". It's best to use the exact same terminology as elsewhere.

tools/libxl/libxl_netbuffer.c line 121

Quoted 3 lines
tools/libxl/libxl_netbuffer.c new file +3 -0
+    libxl__remus_netbuf_ctx *netbuf_ctx = remus_ctx->netbuf_ctx;
+    int num_netbufs = netbuf_ctx->num_netbufs;
+    char **ifb_list = netbuf_ctx->ifb_list;

These should all be declared const.

tools/libxl/libxl_netbuffer.c line 152

Quoted 4 lines
tools/libxl/libxl_netbuffer.c new file +4 -0
+    /* list of handles to plug qdiscs */
+    netbuf_ctx->netbuf_qdisc_list =
+        libxl__calloc(gc, num_netbufs,
+                      sizeof(struct rtnl_qdisc *));

Please use GCNEW_ARRAY.

tools/libxl/libxl_netbuffer.c line 188

Quoted 5 lines
tools/libxl/libxl_netbuffer.c new file +5 -0
+        if (qdisc) {
+            const char *tc_kind = rtnl_tc_get_kind(TC_CAST(qdisc));
+            /* Sanity check: Ensure that the root qdisc is a plug qdisc. */
+            if (!tc_kind || strcmp(tc_kind, "plug")) {
+                LOG(ERROR, "plug qdisc is not installed on %s", ifb_list[i]);

You seem to assume that all the failures from rtnl_tc_get_kind are the due to the expected cause. You need to check errno, I think.

tools/libxl/libxl_netbuffer.c line 201

Quoted 4 lines
tools/libxl/libxl_netbuffer.c new file +4 -0
+    rtnl_link_put(ifb);
+    return 0;
+
+ end:

We call these "out", not "end". This should be changed everywhere.

tools/libxl/libxl_netbuffer.c

Quoted 2 lines
tools/libxl/libxl_netbuffer.c new file +2 -0
+    if (ifb) rtnl_link_put(ifb);
+    return ERROR_FAIL;

Can't this leak netbuf_ctx->nlsock and netbuf_ctx->qdisc_cache ?

tools/libxl/libxl_netbuffer.c line 213

Quoted 10 lines
tools/libxl/libxl_netbuffer.c new file +10 -0
+}
+
+/* the script needs the following env & args
+ * $vifname
+ * $XENBUS_PATH (/libxl/<domid>/remus/netbuf/<devid>/)
+ * $IFB (for teardown)
+ * setup/teardown as command line arg.
+ * In return, the script writes the name of IFB device (during setup) to be
+ * used for output buffering into XENBUS_PATH/ifb
+ */

This information should be somewhere else, not buried in the libxl source code. For now, could it be in the one script which you actually supply ?

tools/libxl/libxl_netbuffer.c line 219

Quoted 6 lines
tools/libxl/libxl_netbuffer.c new file +6 -0
+static int exec_netbuf_script(libxl__gc *gc, libxl__remus_ctx *remus_ctx,
+                              char *op, libxl__ev_child_callback *death)
+{
+    int arraysize, nr = 0;
+    char **env = NULL, **args = NULL;
+    pid_t pid;

This code has many important similarities to the device_hotplug functions in libxl_device.c.

For example your timeout callback is pretty much identical to device_hotplug_timeout_cb.

I think you should consider whether you can use libxl__ao_device, and perhaps share other parts of that code.

Quoted 1 line

+ libxl__ev_child_init(child);

This should be done where the remus state is first populated, not here.

Quoted 1 line

+ char *script = libxl__strdup(gc, remus_ctx->netbufscript);

These next lot are all convenience aliases AFIACT. This should be mentioned, and they should all be const:

tools/libxl/libxl_netbuffer.c line 227

Quoted 7 lines
tools/libxl/libxl_netbuffer.c new file +7 -0
+    libxl__ev_child *child = &remus_ctx->child;
+    libxl__ev_time *timeout = &remus_ctx->timeout;
+    uint32_t domid = remus_ctx->dss->domid;
+    int devid = remus_ctx->num_exec;
+    libxl__remus_netbuf_ctx *netbuf_ctx = remus_ctx->netbuf_ctx;
+    char *vif = netbuf_ctx->vif_list[devid];
+    char *ifb = netbuf_ctx->ifb_list[devid];

So instead, something like:

+ libxl__ev_child *const child = &remus_ctx->child;
...
+ const uint32_t domid = remus_ctx->dss->domid;
...
+ char *const ifb = netbuf_ctx->ifb_list[devid];

There are other places in the code with a similar pattern which also need to be changed.

tools/libxl/libxl_netbuffer.c line 232

Quoted 4 lines
tools/libxl/libxl_netbuffer.c new file +4 -0
+    if (!strcmp(op, "setup"))
+        arraysize = 5;
+    else
+        arraysize = 7;

Perhaps we could do away with this if() and just always allocate 7 ?

tools/libxl/libxl_netbuffer.c line 240

Quoted 7 lines
tools/libxl/libxl_netbuffer.c new file +7 -0
+    GCNEW_ARRAY(env, arraysize);
+    env[nr++] = "vifname";
+    env[nr++] = vif;
+    env[nr++] = "XENBUS_PATH";
+    env[nr++] = GCSPRINTF("%s/remus/netbuf/%d",
+                          libxl__xs_libxl_path(gc, domid), devid);
+    if (!strcmp(op, "teardown")) {

I think you should pass op as an enum, and convert it to a string right at the end. That would do away with these strcmps.

Quoted 1 line

+ remus_ctx->num_exec++;

num_exec is a misleading variable name here. In libxl_linux.c and libxl_device.c it refers to the number of times we have run the hotplug script _for each device_, not the iteration over the number of devices.

Quoted 4 lines
tools/libxl/libxl_netbuffer.c new file +4 -0
+static void netbuf_setup_script_cb(libxl__egc *egc,
+                                   libxl__ev_child *child,
+                                   pid_t pid, int status)
+{

...

tools/libxl/libxl_netbuffer.c line 334

Quoted 3 lines
tools/libxl/libxl_netbuffer.c new file +3 -0
+    hotplug_error = libxl__xs_read(gc, XBT_NULL,
+                                   GCSPRINTF("%s/hotplug-error",
+                                             out_path_base));

Again, libxl__xs_read_checked (several times) ?

tools/libxl/libxl_netbuffer.c line 376

Quoted 4 lines
tools/libxl/libxl_netbuffer.c new file +4 -0
+static void netbuf_setup_timeout_cb(libxl__egc *egc,
+                                    libxl__ev_time *ev,
+                                    const struct timeval *requested_abs)
+{

This seems to be yet another copy of device_hotplug_timeout_cb.

Quoted 8 lines
tools/libxl/libxl_netbuffer.c new file +8 -0
+void libxl__remus_netbuf_setup(libxl__egc *egc,
+                               libxl__domain_suspend_state *dss)
+{
+    libxl__remus_netbuf_ctx *netbuf_ctx = NULL;
+    uint32_t domid = dss->domid;
+    libxl__remus_ctx *remus_ctx = dss->remus_ctx;
+    int num_netbufs = 0;
+    int rc = ERROR_FAIL;

...

Quoted 1 line

+ netbuf_ctx = libxl__zalloc(gc, sizeof(libxl__remus_netbuf_ctx));

Please use GCNEW, not libxl__zalloc, when you can.

Quoted 7 lines
tools/libxl/libxl_netbuffer.c new file +7 -0
+/* Note: This function will be called in the same gc context as
+ * libxl__remus_netbuf_setup, created during the libxl_domain_remus_start
+ * API call.
+ */
+void libxl__remus_netbuf_teardown(libxl__egc *egc,
+                                  libxl__domain_suspend_state *dss)
+{

...

tools/libxl/libxl_netbuffer.c line 464

Quoted 4 lines
tools/libxl/libxl_netbuffer.c new file +4 -0
+    remus_ctx->num_exec = 0; //start at devid == 0
+    if (exec_netbuf_script(gc, remus_ctx, "teardown",
+                           netbuf_teardown_script_cb))
+        libxl__remus_teardown_done(egc, dss);

I think the return value from exec_netbuf_script should be put into an rc variable. And we don't correctly propagate the rc here AFAICT.

tools/libxl/libxl_netbuffer.c line 468

Quoted 2 lines
tools/libxl/libxl_netbuffer.c new file +2 -0
+#define TC_BUFFER_START 1
+#define TC_BUFFER_RELEASE 2

Are these #defines not supposed to come from some Linux header file ? I don't think having them as constants here can be right.

Thanks,
Ian.

#4 lore raw

Re: [PATCH 2 of 4 V5] tools/libxl: Remus network buffering - hotplug scripts and setup code

Quoted 183 lines · Ian Jackson

Shriram Rajagopalan writes ("[PATCH 2 of 4 V5] tools/libxl: Remus network buffering - hotplug scripts and setup code"):

Quoted 1 line

tools/libxl: Remus network buffering - hotplug scripts and setup code

Thanks. This is going in the right direction, but I think there's still some way to go.

One thing that makes reviewing it particularly difficult is the physical ordering of the new code in your patch.

The callback programming model makes it easy to write spaghetti logic; to counter this we try to have a firm structure to each set of event handling code:

* Firstly, function forward declarations, types, structs, etc.

* Then helper functions which have a simple "call this and it does something and returns" control flow.

* Then the main body of the event callbacks, in the order in which they will normally be called. Obviously there is some judgement needed because the call flow won't always be entirely linear. But keeping the layout as linear as possible means that one is never hunting for the next function.

* Call flow of a single operation does not bounce back and forth between files. If this is necessary a formal sub-operation is defined (see how it's done with the bootloader, for example).

I think you would be able to structure the code this way. Would you do that please ?

I think you may need to move several of the entrypoints into a new "remus" file. The "netbuf" file may need to become a more formally separated part of the suspend/resume function.

We also try to break out sub-operations, giving them their own types and callbacks.

tools/libxl/Makefile Mon Nov 18 11:09:34 2013 -0800 LIBXL_OBJS-y += libxl_blktap2.o · line 43

Quoted 8 lines

diff -r 4b471809a5bb -r 5b4e029bb79b tools/libxl/Makefile

tools/libxl/Makefile Mon Nov 18 11:09:34 2013 -0800 renamed from tools/libxl/Makefile Mon Nov 18 10:17:35 2013 -0800 +2 -2
@@ -42,6 +42,13 @@ LIBXL_OBJS-y += libxl_blktap2.o
-static void remus_failover_cb(libxl__egc *egc,
-                              libxl__domain_suspend_state *dss, int rc);
+static void remus_replication_failure_cb(libxl__egc *egc,
+                                         libxl__domain_suspend_state

*dss,

Quoted 1 line
tools/libxl/Makefile Mon Nov 18 11:09:34 2013 -0800 renamed from tools/libxl/Makefile Mon Nov 18 10:17:35 2013 -0800 +1 -0
+                                         int rc);

...

Quoted 1 line

- dss->remus = info;

This change from dss->remus to dss->remus_ctx needs a mention in the commit message. To be honest, I'm not sure why you don't just change the type of the remus member.

Quoted 12 lines

diff -r 4b471809a5bb -r 5b4e029bb79b tools/libxl/libxl_internal.h

tools/libxl/libxl_internal.h Mon Nov 18 11:09:34 2013 -0800 renamed from tools/libxl/libxl_internal.h Mon Nov 18 10:17:35 2013 -0800 +5 -0
@@ -2291,6 +2291,50 @@ typedef struct libxl__logdirty_switch {
     libxl__ev_time timeout;
 } libxl__logdirty_switch;
 
+typedef struct libxl__remus_ctx {
+    /* checkpoint interval */
+    int interval;
+    int blackhole;
+    int compression;

tools/libxl/libxl_internal.h Mon Nov 18 11:09:34 2013 -0800 typedef struct libxl__logdirty_switch { · line 2308

Quoted 10 lines
tools/libxl/libxl_internal.h Mon Nov 18 11:09:34 2013 -0800 renamed from tools/libxl/libxl_internal.h Mon Nov 18 10:17:35 2013 -0800 +10 -0
+    int saved_rc;
+    /* Script to setup/teardown network buffers */
+    const char *netbufscript;
+    /* Opaque context containing network buffer related stuff */
+    void *netbuf_ctx;
+    libxl__domain_suspend_state *dss;
+    libxl__ev_time timeout;
+    libxl__ev_child child;
+    int num_exec;
+} libxl__remus_ctx;

I think you should probably call this a "libxl__remus_state", to correspond to all of the other "libxl__foo_state" structs.

You also need to specify in comments who fills in what parts of it, as is done in the other libxl__foo_state's.

Quoted 4 lines

diff -r 4b471809a5bb -r 5b4e029bb79b tools/libxl/libxl_netbuffer.c

tools/libxl/libxl_netbuffer.c Mon Nov 18 11:09:34 2013 -0800 renamed from dev/null Thu Jan 01 00:00:00 1970 +0000 +0 -0
@@ -0,0 +1,516 @@

...

tools/libxl/libxl_netbuffer.c Mon Nov 18 11:09:34 2013 -0800

Quoted 9 lines
tools/libxl/libxl_netbuffer.c Mon Nov 18 11:09:34 2013 -0800 renamed from dev/null Thu Jan 01 00:00:00 1970 +0000 +9 -0
+typedef struct libxl__remus_netbuf_ctx {
+    struct rtnl_qdisc **netbuf_qdisc_list;
+    struct nl_sock *nlsock;
+    struct nl_cache *qdisc_cache;
+    char **vif_list;
+    char **ifb_list;
+    uint32_t num_netbufs;
+    uint32_t unused;
+} libxl__remus_netbuf_ctx;

Presumably this wants to be a libxl__remus_netbuf_state, then.

The callbacks from the remus and netbuf code to the suspend code should not be hardwired. Instead, function pointers should be passed, the way the domain creation stuff does with a bootloader, and the way the bootloader does with openpty.

This may seem like extra complication if there is only one caller and it will always pass the same function, but it makes the logical separation much clearer. It's analogous to the separation between an ordinary function and its caller, even if there is in fact only one call site.

That is, functions like this

tools/libxl/libxl_internal.h +3 -0
+_hidden void libxl__remus_setup_done(libxl__egc *egc,
+                                     libxl__domain_suspend_state *dss,
+                                     int rc);

should be private functions in the remus file which make the appropriate callback.

tools/libxl/libxl_netbuffer.c Mon Nov 18 11:09:34 2013 -0800

Quoted 10 lines
tools/libxl/libxl_netbuffer.c Mon Nov 18 11:09:34 2013 -0800 renamed from dev/null Thu Jan 01 00:00:00 1970 +0000 +10 -0
+/* If the device has a vifname, then use that instead of
+ * the vifX.Y format.
+ */
+static char *get_vifname(libxl__gc *gc, uint32_t domid,
+                               libxl_device_nic *nic)
+{
+    const char *vifname = NULL;
+    vifname = libxl__xs_read(gc, XBT_NULL,
+                             libxl__sprintf(gc,
+

"%s/backend/vif/%d/%d/vifname",

Quoted 1 line

+ libxl__xs_get_dompath(gc,

0),

Quoted 1 line

+ domid, nic->devid));

Why not libxl__xs_read_checked ?

Quoted 1 line

+ if (!vifname) {

Surely this error handling is wrong. NULL here might mean "failed" or ENOENT. If it means failed then get_vifname needs to return the error to its caller, by returning NULL.

tools/libxl/libxl_netbuffer.c Mon Nov 18 11:09:34 2013 -0800

Quoted 6 lines
tools/libxl/libxl_netbuffer.c Mon Nov 18 11:09:34 2013 -0800 renamed from dev/null Thu Jan 01 00:00:00 1970 +0000 +6 -0
+        vifname = libxl__device_nic_devname(gc, domid,
+                                            nic->devid,
+                                            nic->nictype);
+    }
+
+    return libxl__strdup(gc, vifname);

Why this strdup ?

Quoted 6 lines
tools/libxl/libxl_netbuffer.c Mon Nov 18 11:09:34 2013 -0800 renamed from dev/null Thu Jan 01 00:00:00 1970 +0000 +6 -0
+static char **get_guest_vif_list(libxl__gc *gc, uint32_t domid,
+                                 int *num_vifs)
+{
+    libxl_device_nic *nics = NULL;
+    int nb, i = 0;
+    char **vif_list = NULL;

tools/libxl/libxl_netbuffer.c Mon Nov 18 11:09:34 2013 -0800

Quoted 10 lines
tools/libxl/libxl_netbuffer.c Mon Nov 18 11:09:34 2013 -0800 renamed from dev/null Thu Jan 01 00:00:00 1970 +0000 +10 -0
+
+    *num_vifs = 0;
+    nics = libxl_device_nic_list(CTX, domid, &nb);
+    if (!nics) return NULL;
+
+    /* Ensure that none of the vifs are backed by driver domains */
+    for (i = 0; i < nb; i++) {
+        if (nics[i].backend_domid != LIBXL_TOOLSTACK_DOMID) {
+            LOG(ERROR, "vif %s has driver domain (%u) as its backend.\n"
+                "Network buffering is not supported with driver

domains",

The LOG macros should not normally be passed messages with newlines in.

Quoted 1 line

+ get_vifname(gc, domid, &nics[i]),

nics[i].backend_domid);

Quoted 1 line

+ *num_vifs = -1;

Quoted 10 lines

This calling convention is strange. I think get_guest_vif_list should return a libxl rc value.

tools/libxl/libxl_netbuffer.c Mon Nov 18 11:09:34 2013 -0800

Quoted 3 lines
tools/libxl/libxl_netbuffer.c Mon Nov 18 11:09:34 2013 -0800 renamed from dev/null Thu Jan 01 00:00:00 1970 +0000 +3 -0
+    vif_list = libxl__calloc(gc, nb, sizeof(char *));
+    for (i = 0; i < nb; ++i)
+        vif_list[i] = get_vifname(gc, domid, &nics[i]);

What if get_vifname fails ?

The idea here is that if the interface has a name, then we use it. Otherwise,
we simply revert to the name that Xen assigns it - which is in vifX.Y format.
But I agree with you that the vifname reading process from xenstore may not be successful always.

Quoted 31 lines

...

Quoted 7 lines
tools/libxl/libxl_netbuffer.c Mon Nov 18 11:09:34 2013 -0800 renamed from dev/null Thu Jan 01 00:00:00 1970 +0000 +7 -0
+
+static int init_qdiscs(libxl__gc *gc,
+                       libxl__remus_ctx *remus_ctx)
+{
+    int i, ret, ifindex;
+    struct rtnl_link *ifb = NULL;
+    struct rtnl_qdisc *qdisc = NULL;

...

Quoted 1 line

+ /* convenience vars */

We use the term "Convenience aliases". It's best to use the exact same terminology as elsewhere.

tools/libxl/libxl_netbuffer.c Mon Nov 18 11:09:34 2013 -0800

Quoted 3 lines
tools/libxl/libxl_netbuffer.c Mon Nov 18 11:09:34 2013 -0800 renamed from dev/null Thu Jan 01 00:00:00 1970 +0000 +3 -0
+    libxl__remus_netbuf_ctx *netbuf_ctx = remus_ctx->netbuf_ctx;
+    int num_netbufs = netbuf_ctx->num_netbufs;
+    char **ifb_list = netbuf_ctx->ifb_list;

These should all be declared const.

tools/libxl/libxl_netbuffer.c Mon Nov 18 11:09:34 2013 -0800

Quoted 4 lines
tools/libxl/libxl_netbuffer.c Mon Nov 18 11:09:34 2013 -0800 renamed from dev/null Thu Jan 01 00:00:00 1970 +0000 +4 -0
+    /* list of handles to plug qdiscs */
+    netbuf_ctx->netbuf_qdisc_list =
+        libxl__calloc(gc, num_netbufs,
+                      sizeof(struct rtnl_qdisc *));

Please use GCNEW_ARRAY.

tools/libxl/libxl_netbuffer.c Mon Nov 18 11:09:34 2013 -0800

Quoted 3 lines
tools/libxl/libxl_netbuffer.c Mon Nov 18 11:09:34 2013 -0800 renamed from dev/null Thu Jan 01 00:00:00 1970 +0000 +3 -0
+        if (qdisc) {
+            const char *tc_kind = rtnl_tc_get_kind(TC_CAST(qdisc));
+            /* Sanity check: Ensure that the root qdisc is a plug
Quoted 8 lines

qdisc. */

tools/libxl/libxl_netbuffer.c Mon Nov 18 11:09:34 2013 -0800

Quoted 2 lines
tools/libxl/libxl_netbuffer.c Mon Nov 18 11:09:34 2013 -0800 renamed from dev/null Thu Jan 01 00:00:00 1970 +0000 +2 -0
+            if (!tc_kind || strcmp(tc_kind, "plug")) {
+                LOG(ERROR, "plug qdisc is not installed on %s",

ifb_list[i]);

You seem to assume that all the failures from rtnl_tc_get_kind are the due to the expected cause. You need to check errno, I think.

Not according to the documentation. Its return value is either the name of the
qdisc or NULL if its not set. There is no mention of errno.
I can only go by whats documented :).

Quoted 5 lines

tools/libxl/libxl_netbuffer.c Mon Nov 18 11:09:34 2013 -0800

Quoted 4 lines
tools/libxl/libxl_netbuffer.c Mon Nov 18 11:09:34 2013 -0800 renamed from dev/null Thu Jan 01 00:00:00 1970 +0000 +4 -0
+    rtnl_link_put(ifb);
+    return 0;
+
+ end:
Quoted 8 lines

We call these "out", not "end". This should be changed everywhere.

tools/libxl/libxl_netbuffer.c Mon Nov 18 11:09:34 2013 -0800

Quoted 2 lines
tools/libxl/libxl_netbuffer.c Mon Nov 18 11:09:34 2013 -0800 renamed from dev/null Thu Jan 01 00:00:00 1970 +0000 +2 -0
+    if (ifb) rtnl_link_put(ifb);
+    return ERROR_FAIL;

Can't this leak netbuf_ctx->nlsock and netbuf_ctx->qdisc_cache ?

You asked this question before (in a previous version). There is a cleanup code that gets called that releases these resources (the teardown function).

Quoted 10 lines

tools/libxl/libxl_netbuffer.c Mon Nov 18 11:09:34 2013 -0800

Quoted 9 lines
tools/libxl/libxl_netbuffer.c Mon Nov 18 11:09:34 2013 -0800 renamed from dev/null Thu Jan 01 00:00:00 1970 +0000 +8 -0
+}
+
+/* the script needs the following env & args
+ * $vifname
+ * $XENBUS_PATH (/libxl/<domid>/remus/netbuf/<devid>/)
+ * $IFB (for teardown)
+ * setup/teardown as command line arg.
+ * In return, the script writes the name of IFB device (during setup)

to be

Quoted 8 lines
Quoted 2 lines

+ * used for output buffering into XENBUS_PATH/ifb
+ */

This information should be somewhere else, not buried in the libxl source code. For now, could it be in the one script which you actually supply ?

It is also documented in the script.

Quoted 9 lines
Quoted 2 lines

+static int exec_netbuf_script(libxl__gc *gc, libxl__remus_ctx

*remus_ctx,

tools/libxl/libxl_netbuffer.c Mon Nov 18 11:09:34 2013 -0800

Quoted 5 lines
tools/libxl/libxl_netbuffer.c Mon Nov 18 11:09:34 2013 -0800 renamed from dev/null Thu Jan 01 00:00:00 1970 +0000 +5 -0
+                              char *op, libxl__ev_child_callback *death)
+{
+    int arraysize, nr = 0;
+    char **env = NULL, **args = NULL;
+    pid_t pid;
Quoted 10 lines

This code has many important similarities to the device_hotplug functions in libxl_device.c.

For example your timeout callback is pretty much identical to device_hotplug_timeout_cb.

I think you should consider whether you can use libxl__ao_device, and perhaps share other parts of that code.

I looked at them before deciding to roll out my own. Please correct me if I am
wrong about this:
The ao_device interface while sharing some similarities, does not really fit in here.
It attempts bring up/down a device (that has been created already, for e.g., a vif),
add it to the guest and represent it internally using libxl__device or something.

The netbuf code invokes the script, which in turn sets up an IFB device that is not
part of the guest's configuration. The information about the device returned by the
script is collected in an internal data structure and new handles (plug qdisc handle)
are created, and added to remus' state.

The control flows are different. The only commonality is invoke external script, wait
until it finishes and invoke callback.

On alternative is to hack into the device_hotplug code, and add special cases. For e.g.,
if device_ptr is null, then this is a remus op, so do Y instead of X.

This kind of special casing would percolate throughout the device hotplug code,
polluting a what was otherwise cleanly defined interface to handle generic device hotplug.

If you have any other alternatives in mind, I would be happy to check it out.

Quoted 66 lines
Quoted 2 lines

+ libxl__ev_child_init(child);

This should be done where the remus state is first populated, not here.

Quoted 1 line

+ char *script = libxl__strdup(gc, remus_ctx->netbufscript);

These next lot are all convenience aliases AFIACT. This should be mentioned, and they should all be const:

tools/libxl/libxl_netbuffer.c Mon Nov 18 11:09:34 2013 -0800

Quoted 7 lines
tools/libxl/libxl_netbuffer.c Mon Nov 18 11:09:34 2013 -0800 renamed from dev/null Thu Jan 01 00:00:00 1970 +0000 +7 -0
+    libxl__ev_child *child = &remus_ctx->child;
+    libxl__ev_time *timeout = &remus_ctx->timeout;
+    uint32_t domid = remus_ctx->dss->domid;
+    int devid = remus_ctx->num_exec;
+    libxl__remus_netbuf_ctx *netbuf_ctx = remus_ctx->netbuf_ctx;
+    char *vif = netbuf_ctx->vif_list[devid];
+    char *ifb = netbuf_ctx->ifb_list[devid];

So instead, something like:

+ libxl__ev_child *const child = &remus_ctx->child;
...
+ const uint32_t domid = remus_ctx->dss->domid;
...
+ char *const ifb = netbuf_ctx->ifb_list[devid];

There are other places in the code with a similar pattern which also need to be changed.

tools/libxl/libxl_netbuffer.c Mon Nov 18 11:09:34 2013 -0800

Quoted 4 lines
tools/libxl/libxl_netbuffer.c Mon Nov 18 11:09:34 2013 -0800 renamed from dev/null Thu Jan 01 00:00:00 1970 +0000 +4 -0
+    if (!strcmp(op, "setup"))
+        arraysize = 5;
+    else
+        arraysize = 7;

Perhaps we could do away with this if() and just always allocate 7 ?

tools/libxl/libxl_netbuffer.c Mon Nov 18 11:09:34 2013 -0800

Quoted 7 lines
tools/libxl/libxl_netbuffer.c Mon Nov 18 11:09:34 2013 -0800 renamed from dev/null Thu Jan 01 00:00:00 1970 +0000 +7 -0
+    GCNEW_ARRAY(env, arraysize);
+    env[nr++] = "vifname";
+    env[nr++] = vif;
+    env[nr++] = "XENBUS_PATH";
+    env[nr++] = GCSPRINTF("%s/remus/netbuf/%d",
+                          libxl__xs_libxl_path(gc, domid), devid);
+    if (!strcmp(op, "teardown")) {

I think you should pass op as an enum, and convert it to a string right at the end. That would do away with these strcmps.

Quoted 1 line

+ remus_ctx->num_exec++;

num_exec is a misleading variable name here. In libxl_linux.c and libxl_device.c it refers to the number of times we have run the hotplug script _for each device_, not the iteration over the number of devices.

Quoted 4 lines
tools/libxl/libxl_netbuffer.c Mon Nov 18 11:09:34 2013 -0800 renamed from dev/null Thu Jan 01 00:00:00 1970 +0000 +4 -0
+static void netbuf_setup_script_cb(libxl__egc *egc,
+                                   libxl__ev_child *child,
+                                   pid_t pid, int status)
+{

...

tools/libxl/libxl_netbuffer.c Mon Nov 18 11:09:34 2013 -0800

Quoted 3 lines
tools/libxl/libxl_netbuffer.c Mon Nov 18 11:09:34 2013 -0800 renamed from dev/null Thu Jan 01 00:00:00 1970 +0000 +3 -0
+    hotplug_error = libxl__xs_read(gc, XBT_NULL,
+                                   GCSPRINTF("%s/hotplug-error",
+                                             out_path_base));
Quoted 10 lines

Again, libxl__xs_read_checked (several times) ?

tools/libxl/libxl_netbuffer.c Mon Nov 18 11:09:34 2013 -0800

Quoted 4 lines
tools/libxl/libxl_netbuffer.c Mon Nov 18 11:09:34 2013 -0800 renamed from dev/null Thu Jan 01 00:00:00 1970 +0000 +4 -0
+static void netbuf_setup_timeout_cb(libxl__egc *egc,
+                                    libxl__ev_time *ev,
+                                    const struct timeval *requested_abs)
+{

This seems to be yet another copy of device_hotplug_timeout_cb.

Yep. See my answer above.

Quoted 27 lines

+void libxl__remus_netbuf_setup(libxl__egc *egc,

Quoted 7 lines
tools/libxl/libxl_netbuffer.c Mon Nov 18 11:09:34 2013 -0800 renamed from dev/null Thu Jan 01 00:00:00 1970 +0000 +7 -0
+                               libxl__domain_suspend_state *dss)
+{
+    libxl__remus_netbuf_ctx *netbuf_ctx = NULL;
+    uint32_t domid = dss->domid;
+    libxl__remus_ctx *remus_ctx = dss->remus_ctx;
+    int num_netbufs = 0;
+    int rc = ERROR_FAIL;

...

Quoted 1 line

+ netbuf_ctx = libxl__zalloc(gc, sizeof(libxl__remus_netbuf_ctx));

Please use GCNEW, not libxl__zalloc, when you can.

Quoted 2 lines

+/* Note: This function will be called in the same gc context as
+ * libxl__remus_netbuf_setup, created during the

libxl_domain_remus_start

Quoted 5 lines

+ * API call.
+ */

tools/libxl/libxl_netbuffer.c Mon Nov 18 11:09:34 2013 -0800 renamed from dev/null Thu Jan 01 00:00:00 1970 +0000 +3 -0
+void libxl__remus_netbuf_teardown(libxl__egc *egc,
+                                  libxl__domain_suspend_state *dss)
+{

...

tools/libxl/libxl_netbuffer.c Mon Nov 18 11:09:34 2013 -0800

Quoted 4 lines
tools/libxl/libxl_netbuffer.c Mon Nov 18 11:09:34 2013 -0800 renamed from dev/null Thu Jan 01 00:00:00 1970 +0000 +4 -0
+    remus_ctx->num_exec = 0; //start at devid == 0
+    if (exec_netbuf_script(gc, remus_ctx, "teardown",
+                           netbuf_teardown_script_cb))
+        libxl__remus_teardown_done(egc, dss);
Quoted 10 lines

I think the return value from exec_netbuf_script should be put into an rc variable. And we don't correctly propagate the rc here AFAICT.

tools/libxl/libxl_netbuffer.c Mon Nov 18 11:09:34 2013 -0800

Quoted 2 lines
tools/libxl/libxl_netbuffer.c Mon Nov 18 11:09:34 2013 -0800 renamed from dev/null Thu Jan 01 00:00:00 1970 +0000 +2 -0
+#define TC_BUFFER_START 1
+#define TC_BUFFER_RELEASE 2

Are these #defines not supposed to come from some Linux header file ? I don't think having them as constants here can be right.

They are not constants taken from any library and stuck in here. They are merely
for readability. The related code is remus_netbuffer_op, where you would find
the buffer/release code blocks controlled through the value of op variable (which is either BUFFER_START or BUFFER_RELEASE)

thanks
shriram

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

#5 lore raw

PATCH v5 1/1 tools/libxl: Control network buffering in remus callbacks

In reply to Shriram Rajagopalan

# HG changeset patch
# User Shriram Rajagopalan <[email protected]> # Date 1384801802 28800
# Node ID 94eea030e0095ebd62bd6b40e02a819fdb778c82 # Parent 5b4e029bb79b8db4210a182911a4ff7ec7957c4f tools/libxl: Control network buffering in remus callbacks

This patch constitutes the core network buffering logic. and does the following:
a) create a new network buffer when the domain is suspended (remus_domain_suspend_callback)
b) release the previous network buffer pertaining to the committed checkpoint (remus_domain_checkpoint_dm_saved)

tools/libxl/libxl_dom.c +70 -7
diff -r 5b4e029bb79b -r 94eea030e009 tools/libxl/libxl_dom.c
@@ -1277,8 +1277,27 @@ void libxl__remus_teardown_done(libxl__e
 
 static int libxl__remus_domain_suspend_callback(void *data)
 {
-    /* REMUS TODO: Issue disk and network checkpoint reqs. */
-    return libxl__domain_suspend_common_callback(data);
+    libxl__save_helper_state *shs = data;
+    libxl__domain_suspend_state *dss = CONTAINER_OF(shs, *dss, shs);
+    libxl__remus_ctx *remus_ctx = dss->remus_ctx;
+    STATE_AO_GC(dss->ao);
+
+    /* REMUS TODO: Issue disk checkpoint reqs. */
+    int ok = libxl__domain_suspend_common_callback(data);
+
+    if (!remus_ctx->netbuf_ctx || !ok) goto out;
+
+    /* The domain was suspended successfully. Start a new network
+     * buffer for the next epoch. If this operation fails, then act
+     * as though domain suspend failed -- libxc exits its infinite
+     * loop and ultimately, the replication stops.
+     */
+    if (libxl__remus_netbuf_start_new_epoch(gc, dss->domid,
+                                            remus_ctx))
+        ok = 0;
+
+ out:
+    return ok;
 }
 
 static int libxl__remus_domain_resume_callback(void *data)
@@ -1291,7 +1310,7 @@ static int libxl__remus_domain_resume_ca
     if (libxl__domain_resume(gc, dss->domid, /* Fast Suspend */1))
         return 0;
 
-    /* REMUS TODO: Deal with disk. Start a new network output buffer */
+    /* REMUS TODO: Deal with disk. */
     return 1;
 }
 
@@ -1300,6 +1319,9 @@ static int libxl__remus_domain_resume_ca
 static void remus_checkpoint_dm_saved(libxl__egc *egc,
                                       libxl__domain_suspend_state *dss, int rc);
 
+static void remus_next_checkpoint(libxl__egc *egc, libxl__ev_time *ev,
+                                  const struct timeval *requested_abs);
+
 static void libxl__remus_domain_checkpoint_callback(void *data)
 {
     libxl__save_helper_state *shs = data;
@@ -1318,10 +1340,51 @@ static void libxl__remus_domain_checkpoi
 static void remus_checkpoint_dm_saved(libxl__egc *egc,
                                       libxl__domain_suspend_state *dss, int rc)
 {
-    /* REMUS TODO: Wait for disk and memory ack, release network buffer */
-    /* REMUS TODO: make this asynchronous */
-    assert(!rc); /* REMUS TODO handle this error properly */
-    usleep(dss->remus_ctx->interval * 1000);
+    /*
+     * REMUS TODO: Wait for disk and explicit memory ack (through restore
+     * callback from remote) before releasing network buffer.
+     */
+    libxl__remus_ctx *remus_ctx = dss->remus_ctx;
+    STATE_AO_GC(dss->ao);
+
+    if (rc) {
+        LOG(ERROR, "Failed to save device model. Terminating Remus..");
+        goto out;
+    }
+
+    if (remus_ctx->netbuf_ctx) {
+        rc = libxl__remus_netbuf_release_prev_epoch(gc, dss->domid,
+                                                     remus_ctx);
+        if (rc) {
+            LOG(ERROR, "Failed to release network buffer."
+                " Terminating Remus..");
+            goto out;
+        }
+    }
+
+    /* Set checkpoint interval timeout */
+    rc = libxl__ev_time_register_rel(gc, &remus_ctx->timeout,
+                                     remus_next_checkpoint,
+                                     dss->remus_ctx->interval);
+    if (rc) {
+        LOG(ERROR, "unable to register timeout for next epoch."
+            " Terminating Remus..");
+        goto out;
+    }
+    return;
+
+ out:
+    libxl__xc_domain_saverestore_async_callback_done(egc, &dss->shs, 0);
+}
+
+static void remus_next_checkpoint(libxl__egc *egc, libxl__ev_time *ev,
+                                  const struct timeval *requested_abs)
+{
+    libxl__remus_ctx *remus_ctx = CONTAINER_OF(ev, *remus_ctx, timeout);
+    libxl__domain_suspend_state *dss = remus_ctx->dss;
+    STATE_AO_GC(dss->ao);
+
+    libxl__ev_time_deregister(gc, &remus_ctx->timeout);
     libxl__xc_domain_saverestore_async_callback_done(egc, &dss->shs, 1);
 }
#6 lore raw

Re: [PATCH 3 of 4 V5] tools/libxl: Control network buffering in remus callbacks

Shriram Rajagopalan writes ("[PATCH 3 of 4 V5] tools/libxl: Control network buffering in remus callbacks"):

Quoted 1 line

tools/libxl: Control network buffering in remus callbacks

I had a quick look at this and some of my style/structure comments on 2/4 seem to apply to this too.

Thanks,
Ian.

#7 lore raw

PATCH v5 1/1 tools/xl: Remus - Network buffering cmdline switch

In reply to Shriram Rajagopalan

# HG changeset patch
# User Shriram Rajagopalan <[email protected]> # Date 1384804015 28800
# Node ID 3c11efb5e8fef9f070272edefed47f31b7c619ef # Parent 94eea030e0095ebd62bd6b40e02a819fdb778c82 tools/xl: Remus - Network buffering cmdline switch

Command line switch to 'xl remus' command, to enable network buffering. Pass on this flag to libxl so that it can act accordingly. Also update man pages to reflect the addition of a new option to 'xl remus' command.

docs/man/xl.conf.pod.5 +6 -0
diff -r 94eea030e009 -r 3c11efb5e8fe docs/man/xl.conf.pod.5
@@ -105,6 +105,12 @@ Configures the default gateway device to
 
 Default: C<None>
 
+=item B<remus.default.netbufscript="PATH">
+
+Configures the default script used by Remus to setup network buffering.
+
+Default: C</etc/xen/scripts/remus-netbuf-setup>
+
 =item B<output_format="json|sxp">
 
 Configures the default output format used by xl when printing "machine
docs/man/xl.pod.1 +13 -2
diff -r 94eea030e009 -r 3c11efb5e8fe docs/man/xl.pod.1
@@ -398,8 +398,7 @@ Print huge (!) amount of debug during th
 Enable Remus HA for domain. By default B<xl> relies on ssh as a transport
 mechanism between the two hosts.
 
-N.B: Remus support in xl is still in experimental (proof-of-concept) phase.
-     There is no support for network or disk buffering at the moment.
+N.B: There is no support for disk buffering at the moment.
 
 B<OPTIONS>
 
@@ -418,6 +417,18 @@ Generally useful for debugging.
 
 Disable memory checkpoint compression.
 
+=item B<-n>
+
+Enable network output buffering.  The default script used to configure
+network buffering is /etc/xen/scripts/remus-netbuf-setup. If you wish to
+use a custom script, use the I<-N> option or set the global variable
+I<remus.default.netbufscript> in /etc/xen/xl.conf to point to your script.
+
+=item B<-N> I<netbufscript>
+
+Use <netbufscript> to setup network buffering instead of the instead of
+the default (/etc/xen/scripts/remus-netbuf-setup).
+
 =item B<-s> I<sshcommand>
 
 Use <sshcommand> instead of ssh.  String will be passed to sh.
tools/libxl/xl.c +4 -0
diff -r 94eea030e009 -r 3c11efb5e8fe tools/libxl/xl.c
@@ -46,6 +46,7 @@ char *default_vifscript = NULL;
 char *default_bridge = NULL;
 char *default_gatewaydev = NULL;
 char *default_vifbackend = NULL;
+char *default_remus_netbufscript = NULL;
 enum output_format default_output_format = OUTPUT_FORMAT_JSON;
 int claim_mode = 1;
 
@@ -177,6 +178,9 @@ static void parse_global_config(const ch
     if (!xlu_cfg_get_long (config, "claim_mode", &l, 0))
         claim_mode = l;
 
+    xlu_cfg_replace_string (config, "remus.default.netbufscript",
+                            &default_remus_netbufscript, 0);
+
     xlu_cfg_destroy(config);
 }
 
tools/libxl/xl.h +1 -0
diff -r 94eea030e009 -r 3c11efb5e8fe tools/libxl/xl.h
@@ -152,6 +152,7 @@ extern char *default_vifscript;
 extern char *default_bridge;
 extern char *default_gatewaydev;
 extern char *default_vifbackend;
+extern char *default_remus_netbufscript;
 extern char *blkdev_start;
 
 enum output_format {
tools/libxl/xl_cmdimpl.c +23 -7
diff -r 94eea030e009 -r 3c11efb5e8fe tools/libxl/xl_cmdimpl.c
@@ -7116,8 +7116,9 @@ int main_remus(int argc, char **argv)
     r_info.interval = 200;
     r_info.blackhole = 0;
     r_info.compression = 1;
-
-    SWITCH_FOREACH_OPT(opt, "bui:s:e", NULL, "remus", 2) {
+    r_info.netbuf = 0;
+
+    SWITCH_FOREACH_OPT(opt, "buni:s:N:e", NULL, "remus", 2) {
     case 'i':
         r_info.interval = atoi(optarg);
         break;
@@ -7127,6 +7128,12 @@ int main_remus(int argc, char **argv)
     case 'u':
         r_info.compression = 0;
         break;
+    case 'n':
+        r_info.netbuf = 1;
+        break;
+    case 'N':
+        r_info.netbufscript = optarg;
+        break;
     case 's':
         ssh_command = optarg;
         break;
@@ -7138,6 +7145,9 @@ int main_remus(int argc, char **argv)
     domid = find_domain(argv[optind]);
     host = argv[optind + 1];
 
+    if(!r_info.netbufscript)
+        r_info.netbufscript = default_remus_netbufscript;
+
     if (r_info.blackhole) {
         send_fd = open("/dev/null", O_RDWR, 0644);
         if (send_fd < 0) {
@@ -7175,13 +7185,19 @@ int main_remus(int argc, char **argv)
     /* Point of no return */
     rc = libxl_domain_remus_start(ctx, &r_info, domid, send_fd, recv_fd, 0);
 
-    /* If we are here, it means backup has failed/domain suspend failed.
-     * Try to resume the domain and exit gracefully.
+    /* check if the domain exists. User may have xl destroyed the
+     * domain to force failover
+     */
+    if (libxl_domain_info(ctx, 0, domid)) {
+        fprintf(stderr, "Remus: Primary domain has been destroyed.\n");
+        close(send_fd);
+        return 0;
+    }
+
+    /* If we are here, it means remus setup/domain suspend/backup has
+     * failed. Try to resume the domain and exit gracefully.
      * TODO: Split-Brain check.
      */
-    fprintf(stderr, "remus sender: libxl_domain_suspend failed"
-            " (rc=%d)\n", rc);
-
     if (rc == ERROR_GUEST_TIMEDOUT)
         fprintf(stderr, "Failed to suspend domain at primary.\n");
     else {
tools/libxl/xl_cmdtable.c +3 -0
diff -r 94eea030e009 -r 3c11efb5e8fe tools/libxl/xl_cmdtable.c
@@ -481,6 +481,9 @@ struct cmd_spec cmd_table[] = {
       "-i MS                   Checkpoint domain memory every MS milliseconds (def. 200ms).\n"
       "-b                      Replicate memory checkpoints to /dev/null (blackhole)\n"
       "-u                      Disable memory checkpoint compression.\n"
+      "-n                      Enable network output buffering.\n"
+      "-N <netbufscript>       Use netbufscript to setup network buffering instead of the\n"
+      "                        instead of the default (/etc/xen/scripts/remus-netbuf-setup).\n"
       "-s <sshcommand>         Use <sshcommand> instead of ssh.  String will be passed\n"
       "                        to sh. If empty, run <host> instead of \n"
       "                        ssh <host> xl migrate-receive -r [-e]\n"
#8 lore raw

Re: [PATCH 4 of 4 V5] tools/xl: Remus - Network buffering cmdline switch

Shriram Rajagopalan writes ("[PATCH 4 of 4 V5] tools/xl: Remus - Network buffering cmdline switch"):

Quoted 6 lines

tools/xl: Remus - Network buffering cmdline switch

Command line switch to 'xl remus' command, to enable network buffering. Pass on this flag to libxl so that it can act accordingly. Also update man pages to reflect the addition of a new option to 'xl remus' command.

Shouldn't enabling network buffering be the default ?

Is it really useful to have a command-line option to change the script ?

docs/man/xl.pod.1 Print huge (!) amount of debug during th · line 401

Quoted 10 lines

diff -r 94eea030e009 -r 3c11efb5e8fe docs/man/xl.pod.1

docs/man/xl.pod.1 +1 -2
@@ -398,8 +398,7 @@ Print huge (!) amount of debug during th
 Enable Remus HA for domain. By default B<xl> relies on ssh as a transport
 mechanism between the two hosts.
 
-N.B: Remus support in xl is still in experimental (proof-of-concept) phase.
-     There is no support for network or disk buffering at the moment.
+N.B: There is no support for disk buffering at the moment.

I think you need to keep the "experimental (proof-of-concept)" note. Without disk buffering, surely any VM which uses remus might corrupt its disk ?

docs/man/xl.pod.1 Generally useful for debugging. · line 425

Quoted 6 lines
docs/man/xl.pod.1 +6 -0
+=item B<-n>
+
+Enable network output buffering.  The default script used to configure
+network buffering is /etc/xen/scripts/remus-netbuf-setup. If you wish to
+use a custom script, use the I<-N> option or set the global variable
+I<remus.default.netbufscript> in /etc/xen/xl.conf to point to your script.

There is no need to mention the default script again in this paragraph.

I seem to have missed something, but should it be possible to specify the script individually per domain ?

Ian.

#9 lore raw

Re: [PATCH 4 of 4 V5] tools/xl: Remus - Network buffering cmdline switch

Quoted 12 lines · Ian Jackson

Shriram Rajagopalan writes ("[PATCH 4 of 4 V5] tools/xl: Remus - Network buffering cmdline switch"):

Quoted 6 lines

tools/xl: Remus - Network buffering cmdline switch

Command line switch to 'xl remus' command, to enable network buffering. Pass on this flag to libxl so that it can act accordingly. Also update man pages to reflect the addition of a new option to 'xl remus' command.

Shouldn't enabling network buffering be the default ?

Given that network buffering support is conditionally compiled into libxl, it makes sense to let the user explicitly enable network buffering. If network buffering was enabled by default, but libxl was missing the feature,
then running xl remus (without any option) would immediately fail saying network
buffering support is missing. Might as well make people enable it explicitly and
then discover the lack of support for it inside libxl.

Quoted 3 lines

Is it really useful to have a command-line option to change the script ?

That was added on Ian C's suggestion. It might be useful if people have openvswitch based setups and want to tweak the network buffering setup.

Quoted 8 lines

docs/man/xl.pod.1 Mon Nov 18 11:46:55 2013 -0800 Print huge (!) amount of debug during th · line 398

Quoted 5 lines

diff -r 94eea030e009 -r 3c11efb5e8fe docs/man/xl.pod.1

docs/man/xl.pod.1 Mon Nov 18 11:46:55 2013 -0800 renamed from docs/man/xl.pod.1 Mon Nov 18 11:10:02 2013 -0800 +0 -0
@@ -398,8 +398,7 @@ Print huge (!) amount of debug during th
 Enable Remus HA for domain. By default B<xl> relies on ssh as a

transport

Quoted 2 lines
docs/man/xl.pod.1 Mon Nov 18 11:46:55 2013 -0800 renamed from docs/man/xl.pod.1 Mon Nov 18 11:10:02 2013 -0800 +0 -0
 mechanism between the two hosts.
 
Quoted 10 lines

docs/man/xl.pod.1 Mon Nov 18 11:46:55 2013 -0800 Print huge (!) amount of debug during th · line 401

Quoted 1 line
docs/man/xl.pod.1 Mon Nov 18 11:46:55 2013 -0800 renamed from docs/man/xl.pod.1 Mon Nov 18 11:10:02 2013 -0800 +0 -1
-N.B: Remus support in xl is still in experimental (proof-of-concept)

phase.

docs/man/xl.pod.1 Mon Nov 18 11:46:55 2013 -0800 Print huge (!) amount of debug during th · line 401

Quoted 2 lines
docs/man/xl.pod.1 Mon Nov 18 11:46:55 2013 -0800 renamed from docs/man/xl.pod.1 Mon Nov 18 11:10:02 2013 -0800 +1 -1
-     There is no support for network or disk buffering at the moment.
+N.B: There is no support for disk buffering at the moment.

I think you need to keep the "experimental (proof-of-concept)" note. Without disk buffering, surely any VM which uses remus might corrupt its disk ?

agreed.

Quoted 8 lines

docs/man/xl.pod.1 Mon Nov 18 11:46:55 2013 -0800 Print huge (!) amount of debug during th

Quoted 6 lines
docs/man/xl.pod.1 Mon Nov 18 11:46:55 2013 -0800 renamed from docs/man/xl.pod.1 Mon Nov 18 11:10:02 2013 -0800 +6 -0
+=item B<-n>
+
+Enable network output buffering.  The default script used to configure
+network buffering is /etc/xen/scripts/remus-netbuf-setup. If you wish to
+use a custom script, use the I<-N> option or set the global variable
+I<remus.default.netbufscript> in /etc/xen/xl.conf to point to your

script.

Quoted 7 lines

There is no need to mention the default script again in this paragraph.

I seem to have missed something, but should it be possible to specify the script individually per domain ?

using the -N option.

Quoted 3 lines

Ian.

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

#10 lore raw

Re: [PATCH 4 of 4 V5] tools/xl: Remus - Network buffering cmdline switch

Quoted 9 lines · Shriram Rajagopalan

On Mon, Nov 25, 2013 at 7:37 AM, Ian Jackson <[email protected]> wrote:
Shriram Rajagopalan writes ("[PATCH 4 of 4 V5] tools/xl: Remus - Network buffering cmdline switch"):
> tools/xl: Remus - Network buffering cmdline switch
>
> Command line switch to 'xl remus' command, to enable network buffering.
> Pass on this flag to libxl so that it can act accordingly.
> Also update man pages to reflect the addition of a new option to
> 'xl remus' command.

Quoted 8 lines

Shouldn't enabling network buffering be the default ?

Given that network buffering support is conditionally compiled into libxl, it makes sense to let the user explicitly enable network buffering. If network buffering was enabled by default, but libxl was missing the feature, then running xl remus (without any option) would immediately fail saying network buffering support is missing. Might as well make people enable it explicitly and then discover the lack of support for it inside libxl.

The default should be set depending on whether support was enabled in libxl or not. libxl_defbool is the mechanism which allows this sort of thing to be decided at the library level.

Ian.

#11 lore raw

Re: [PATCH 4 of 4 V5] tools/xl: Remus - Network buffering cmdline switch

Ian Campbell writes ("Re: [PATCH 4 of 4 V5] tools/xl: Remus - Network buffering cmdline switch"):

Quoted 3 lines

The default should be set depending on whether support was enabled in libxl or not. libxl_defbool is the mechanism which allows this sort of thing to be decided at the library level.

I don't think I agree with this.

As I understand it, operating without network buffering could lead to strange behaviours by the domain when it fails over - violations of network protocols by "the" VM.

The default should be to operate correctly, and if correct operation is not supported the default should be to fail. (Cf security.)

Ian.

#12 lore raw

Re: [PATCH 4 of 4 V5] tools/xl: Remus - Network buffering cmdline switch

Quoted 5 lines · Ian Jackson

Ian Campbell writes ("Re: [PATCH 4 of 4 V5] tools/xl: Remus - Network buffering cmdline switch"):

Quoted 3 lines

The default should be set depending on whether support was enabled in libxl or not. libxl_defbool is the mechanism which allows this sort of thing to be decided at the library level.

Quoted 8 lines

I don't think I agree with this.

As I understand it, operating without network buffering could lead to strange behaviours by the domain when it fails over - violations of network protocols by "the" VM.

The default should be to operate correctly, and if correct operation is not supported the default should be to fail. (Cf security.)

Er, yes, that's absolutely correct, not sure what I was thinking.

Ian.

#13 lore raw

Re: [PATCH 4 of 4 V5] tools/xl: Remus - Network buffering cmdline switch

In reply to Shriram Rajagopalan

Shriram Rajagopalan writes ("Re: [PATCH 4 of 4 V5] tools/xl: Remus - Network buffering cmdline switch"):

Quoted 4 lines · Ian Jackson

Shriram Rajagopalan writes ("[PATCH 4 of 4 V5] tools/xl: Remus - Network buffering cmdline switch"):
> tools/xl: Remus - Network buffering cmdline switch

...

Quoted 10 lines

Shouldn't enabling network buffering be the default ?

Given that network buffering support is conditionally compiled into libxl, it makes sense to let the user explicitly enable network buffering. If network buffering was enabled by default, but libxl was missing the feature, then running xl remus (without any option) would immediately fail saying network buffering support is missing. Might as well make people enable it explicitly and then discover the lack of support for it inside libxl.

Is Remus actually useable without network buffering ? I don't think we would want to change the default later, and clearly the default should be to behave correctly.

Quoted 4 lines

Is it really useful to have a command-line option to change the script ?

That was added on Ian C's suggestion. It might be useful if people have openvswitch based setups and want to tweak the network buffering setup.

I think this should be a per-vif configuration option, with a default taken from the domain and/or global config. Not a command-line option.

Thanks,
Ian.

#14 lore raw

Re: [PATCH 4 of 4 V5] tools/xl: Remus - Network buffering cmdline switch

Quoted 20 lines · Ian Jackson

Shriram Rajagopalan writes ("Re: [PATCH 4 of 4 V5] tools/xl: Remus - Network buffering cmdline switch"):

Quoted 1 line

On Mon, Nov 25, 2013 at 7:37 AM, Ian Jackson <[email protected]>

Quoted 1 line · wrote:

Shriram Rajagopalan writes ("[PATCH 4 of 4 V5] tools/xl: Remus -

Network

Quoted 2 lines

buffering cmdline switch"):
> tools/xl: Remus - Network buffering cmdline switch

...

Quoted 10 lines

Shouldn't enabling network buffering be the default ?

Given that network buffering support is conditionally compiled into libxl, it makes sense to let the user explicitly enable network buffering. If network buffering was enabled by default, but libxl was missing the feature, then running xl remus (without any option) would immediately fail saying network buffering support is missing. Might as well make people enable it explicitly and then discover the lack of support for it inside libxl.

Quoted 4 lines

Is Remus actually useable without network buffering ? I don't think we would want to change the default later, and clearly the default should be to behave correctly.

It is usable without network buffering. Memory state is preserved on failover.
On failover, without network buffering, exising network connectivity (tcp connections) "may" be lost, under some circumstances.

You are right about the fact that for Remus to behave 'correctly', one needs both network and disk buffering -- the latter being absent is the reason
Remus in libxl is still experimental.

So, might as well enable network buffering by default, punt if the support is
missing and suggest that the user invoke Remus without network buffering (while providing sufficient warnings that the failover may be incomplete)

Quoted 11 lines
Quoted 1 line

Is it really useful to have a command-line option to change the

script ?

Quoted 3 lines

That was added on Ian C's suggestion. It might be useful if people have openvswitch based setups and want to tweak the network buffering setup.

I think this should be a per-vif configuration option, with a default taken from the domain and/or global config. Not a command-line option.

This is getting needlessly complicated. These are "nice to have" features, which I am least concerned about at this moment.
I am more than happy to fall back to simply using the default script specified in xl.conf,
for the initial version. If users want to provide a per-vif configuration script, the support
can be integrated later.

Quoted 4 lines

Thanks,
Ian.

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

#15 lore raw

Re: [PATCH 4 of 4 V5] tools/xl: Remus - Network buffering cmdline switch

Shriram Rajagopalan writes ("Re: [PATCH 4 of 4 V5] tools/xl: Remus - Network buffering cmdline switch"):

Quoted 3 lines

So, might as well enable network buffering by default, punt if the support is missing and suggest that the user invoke Remus without network buffering (while providing sufficient warnings that the failover may be incomplete)

Right.

Quoted 3 lines

I think this should be a per-vif configuration option, with a default taken from the domain and/or global config. Not a command-line option.

...

Quoted 6 lines

This is getting needlessly complicated. These are "nice to have" features, which I am least concerned about at this moment.
I am more than happy to fall back to simply using the default script specified in xl.conf, for the initial version. If users want to provide a per-vif configuration script, the support can be integrated later.

Fair enough.

Thanks,
Ian.

#16 lore raw

Re: [PATCH 4 of 4 V5] tools/xl: Remus - Network buffering cmdline switch

In reply to Shriram Rajagopalan
Quoted 27 lines · Shriram Rajagopalan
Quoted 20 lines · Ian Jackson

Shriram Rajagopalan writes ("Re: [PATCH 4 of 4 V5] tools/xl: Remus - Network buffering cmdline switch"):

Quoted 1 line

On Mon, Nov 25, 2013 at 7:37 AM, Ian Jackson <[email protected]>

Quoted 1 line · wrote:

Shriram Rajagopalan writes ("[PATCH 4 of 4 V5] tools/xl: Remus -

Network

Quoted 2 lines

buffering cmdline switch"):
> tools/xl: Remus - Network buffering cmdline switch

...

Quoted 10 lines

Shouldn't enabling network buffering be the default ?

Given that network buffering support is conditionally compiled into libxl, it makes sense to let the user explicitly enable network buffering. If network buffering was enabled by default, but libxl was missing the feature, then running xl remus (without any option) would immediately fail saying network buffering support is missing. Might as well make people enable it explicitly and then discover the lack of support for it inside libxl.

Quoted 4 lines

Is Remus actually useable without network buffering ? I don't think we would want to change the default later, and clearly the default should be to behave correctly.

Quoted 4 lines

It is usable without network buffering. Memory state is preserved on failover.
On failover, without network buffering, exising network connectivity (tcp connections) "may" be lost, under some circumstances.

uh, wouldn't it be possible to acknowledge client requests whose backing state hasn't been replicated? So that on failover the backup might have no memory of a promised it had made? it's more than just tcp resets, isn't it?

Quoted 40 lines

You are right about the fact that for Remus to behave 'correctly', one needs both network and disk buffering -- the latter being absent is the reason
Remus in libxl is still experimental.

So, might as well enable network buffering by default, punt if the support is
missing and suggest that the user invoke Remus without network buffering (while providing sufficient warnings that the failover may be incomplete)

Quoted 11 lines
Quoted 1 line

Is it really useful to have a command-line option to change the

script ?

Quoted 3 lines

That was added on Ian C's suggestion. It might be useful if people have openvswitch based setups and want to tweak the network buffering setup.

I think this should be a per-vif configuration option, with a default taken from the domain and/or global config. Not a command-line option.

This is getting needlessly complicated. These are "nice to have" features, which I am least concerned about at this moment.
I am more than happy to fall back to simply using the default script specified in xl.conf,
for the initial version. If users want to provide a per-vif configuration script, the support
can be integrated later.

Quoted 4 lines

Thanks,
Ian.

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

#17 lore raw

Re: [PATCH 4 of 4 V5] tools/xl: Remus - Network buffering cmdline switch

Quoted 13 lines
Quoted 8 lines
Quoted 7 lines

Is Remus actually useable without network buffering ? I don't think we would want to change the default later, and clearly the default should be to behave correctly.

Quoted 4 lines

It is usable without network buffering. Memory state is preserved on failover.
On failover, without network buffering, exising network connectivity (tcp connections) "may" be lost, under some circumstances.

Quoted 6 lines

uh, wouldn't it be possible to acknowledge client requests whose backing state hasn't been replicated? So that on failover the backup might have no memory of a promised it had made? it's more than just tcp resets, isn't it?

true.. which is why I stated ..

Quoted 12 lines

You are right about the fact that for Remus to behave 'correctly', one

Quoted 5 lines

needs both network and disk buffering -- the latter being absent is the reason
Remus in libxl is still experimental.

So, might as well enable network buffering by default, punt if the

support

Quoted 4 lines

is
missing and suggest that the user invoke Remus without network buffering (while providing sufficient warnings that the failover may be incomplete)

:)

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

#18 lore raw

Re: [PATCH 4 of 4 V5] tools/xl: Remus - Network buffering cmdline switch

Quoted 13 lines · Shriram Rajagopalan
Quoted 13 lines
Quoted 12 lines
Quoted 7 lines

Is Remus actually useable without network buffering ? I don't think we would want to change the default later, and clearly the default should be to behave correctly.

It is usable without network buffering. Memory state is preserved on failover.
On failover, without network buffering, exising network connectivity (tcp connections) "may" be lost, under some circumstances.

Quoted 12 lines
Quoted 6 lines

uh, wouldn't it be possible to acknowledge client requests whose backing state hasn't been replicated? So that on failover the backup might have no memory of a promised it had made? it's more than just tcp resets, isn't it?

true.. which is why I stated ..

Quoted 4 lines

You are right about the fact that for Remus to behave 'correctly', one

Quoted 3 lines

needs both network and disk buffering -- the latter being absent is the reason
Remus in libxl is still experimental.

ah, maybe you meant to leave the quotes off of 'correctly'.

Quoted 10 lines
Quoted 8 lines
Quoted 2 lines

So, might as well enable network buffering by default, punt if the

support

Quoted 4 lines

is
missing and suggest that the user invoke Remus without network buffering (while providing sufficient warnings that the failover may be incomplete)

:)

#19 lore raw
In reply to Shriram Rajagopalan
Quoted 9 lines · Shriram Rajagopalan

This patch series adds support for network buffering in the Remus codebase in libxl. NB: This series does not contain the bug fix related to usleep calls in libxl__domain_suspend. I will send it out later as a separate series.

Changes in V5:

Merge hotplug script patch (2/5) and hotplug script setup/teardown patch (3/5) into a single patch.

This has dropped to 4 patches vs 7 in v4. One of the delta is this merging, what happened to the other two? Especially given that Ian J asked for #7 to be split last time around.

Quoted 1 line

From an initial glance this looks like code this time around is that

right? I know some stuff is to come later but IIRC that wasn't in the previous series either.

Quoted 1 line

From a release point of view (George CCd) it looks like it only touches

Remus code other than a few little changes in the generic save/restore code which are guarded by (in effect) if (remus). Is that correct?

If the only risk is to Remus users and the plus side is a far more useful Remus then I'd say we should make a freeze exception, subject to the code being reviewed of course.

Ian.

#20 lore raw
Quoted 6 lines · Ian Campbell
Quoted 5 lines · Shriram Rajagopalan

This patch series adds support for network buffering in the Remus codebase in libxl. NB: This series does not contain the bug fix related to usleep calls in libxl__domain_suspend. I will send it out later as a separate series.

Quoted 10 lines
Quoted 4 lines

Changes in V5:

Merge hotplug script patch (2/5) and hotplug script setup/teardown patch (3/5) into a single patch.

This has dropped to 4 patches vs 7 in v4. One of the delta is this merging, what happened to the other two? Especially given that Ian J asked for #7 to be split last time around.

Two patches in v4 are related solely to the bug fix. The first was by Ian while the other was by me, which Ian J asked me to split.

Quoted 5 lines

From an initial glance this looks like code this time around is that right? I know some stuff is to come later but IIRC that wasn't in the previous series either.

This is basically same as v3, with all the feedback.

Quoted 3 lines
Quoted 1 line

From a release point of view (George CCd) it looks like it only touches

Remus code other than a few little changes in the generic save/restore code which are guarded by (in effect) if (remus). Is that correct?

Yep.

Quoted 4 lines

If the only risk is to Remus users and the plus side is a far more useful Remus then I'd say we should make a freeze exception, subject to the code being reviewed of course.

Yes, these patches are confined totally to Remus' functionality in libxl.

The other bug fix series otoh may need more testing as it touches domain suspend -- something core to both live migration and Remus. But that is a bug fix, not a feature addition.

Quoted 3 lines

Ian.

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

#21 lore raw
Quoted 43 lines · Shriram Rajagopalan
Quoted 7 lines · Ian Campbell
Quoted 5 lines · Shriram Rajagopalan

This patch series adds support for network buffering in the Remus codebase in libxl. NB: This series does not contain the bug fix related to usleep calls in libxl__domain_suspend. I will send it out later as a separate series.

Quoted 9 lines
Quoted 4 lines

Changes in V5:

Merge hotplug script patch (2/5) and hotplug script setup/teardown patch (3/5) into a single patch.

This has dropped to 4 patches vs 7 in v4. One of the delta is this merging, what happened to the other two? Especially given that Ian J asked for #7 to be split last time around.

Two patches in v4 are related solely to the bug fix. The first was by Ian while the other was by me, which Ian J asked me to split.

Quoted 4 lines
Quoted 1 line

From an initial glance this looks like code this time around is that

right? I know some stuff is to come later but IIRC that wasn't in the previous series either.

This is basically same as v3, with all the feedback.

Quoted 4 lines
Quoted 2 lines

From a release point of view (George CCd) it looks like it only touches

Remus code other than a few little changes in the generic save/restore code which are guarded by (in effect) if (remus). Is that correct?

Yep.

Quoted 5 lines

If the only risk is to Remus users and the plus side is a far more useful Remus then I'd say we should make a freeze exception, subject to the code being reviewed of course.

Quoted 10 lines

Yes, these patches are confined totally to Remus' functionality in libxl.

The other bug fix series otoh may need more testing as it touches domain suspend -- something core to both live migration and Remus. But that is a bug fix, not a feature addition.

Quoted 3 lines

Ian.

ping!