Remus/Libxl: Network buffering support
PATCH v3 1/1 Remus/Libxl: Network buffering support
This patch series adds support for network buffering in the Remus codebase in libxl.
Changes in V3:
[1/5] Fix redundant checks in configure scripts
(based on Ian Campbell's suggestions)
[2/5] Introduce locking in the script, during IFB setup.
Add xenstore paths used by netbuf scripts to xenstore-paths.markdown
[3/5] Hotplug scripts setup/teardown invocations are now asynchronous
following IanJ's feedback. However, the invocations are still sequential.
[5/5] Allow per-domain specification of netbuffer scripts in xl remus
commmand.
And minor nits throughout the series based on feedback from the last version
Changes in V2:
[1/5] Configure script will automatically enable/disable network
buffer support depending on the availability of the appropriate libnl3 version. [If libnl3 is unavailable, a warning message will be printed to let the user know that the feature has been disabled.]
use macros from pkg.m4 instead of pkg-config commands removed redundant checks for libnl3 libraries.
[3,4/5] - Minor nits.
Version 1:
[1/5] Changes to autoconf scripts to check for libnl3. Add linker flags
to libxl Makefile.
[2/5] External script to setup/teardown network buffering using libnl3's
CLI. This script will be invoked by libxl before starting Remus. The script's main job is to bring up an IFB device with plug qdisc attached to it. It then re-routes egress traffic from the guest's vif to the IFB device.
[3/5] Libxl code to invoke the external setup script, followed by netlink
related setup to obtain a handle on the output buffers attached to each vif.
[4/5] Libxl interaction with network buffer module in the kernel via
libnl3 API.
[5/5] xl cmdline switch to explicitly enable network buffering when
starting remus.
Few things to note:
a) Based on previous email discussions, the setup/teardown task has been moved to a hotplug style shell script which can be customized as desired, instead of implementing it as C code inside libxl.
b) Libnl3 is not available on NetBSD. Nor is it available on CentOS
(Linux). So I have made network buffering support an optional feature so that it can be disabled if desired.
c) NetBSD does not have libnl3. So I have put the setup script under tools/hotplug/Linux folder.
thanks
shriram
PATCH v3 1/1 remus: add libnl3 dependency to autoconf scripts
# HG changeset patch
# User Shriram Rajagopalan <[email protected]> # Date 1381540581 25200
# Node ID 357655a1598cba8f8460758714885563dd2b4a9c # Parent 808aba98084edee7b6dc78c7333da71d2be70386
remus: add libnl3 dependency to autoconf scripts
Libnl3 is required for controlling Remus network buffering. This patch adds dependency on libnl3 (>= 3.2.8) to autoconf scripts. Also provide ability to configure tools without libnl3 support, that is without network buffering support.
config/Tools.mk.in +3 -0
@@ -37,6 +37,8 @@ PTHREAD_LIBS := @PTHREAD_LIBS@ PTYFUNCS_LIBS := @PTYFUNCS_LIBS@ +LIBNL3_LIBS := @LIBNL3_LIBS@ +LIBNL3_CFLAGS := @LIBNL3_CFLAGS@ # Download GIT repositories via HTTP or GIT's own protocol? # GIT's protocol is faster and more robust, when it works at all (firewalls # may block it). We make it the default, but if your GIT repository downloads
@@ -55,6 +57,7 @@ CONFIG_QEMU_TRAD := @qemu_traditional CONFIG_QEMU_XEN := @qemu_xen@ CONFIG_XEND := @xend@ CONFIG_BLKTAP1 := @blktap1@ +CONFIG_REMUS_NETBUF := @remus_netbuf@ #System options ZLIB := @zlib@
tools/configure.ac +18 -0
@@ -219,6 +219,24 @@ AC_SUBST(libiconv) # Checks for header files. AC_CHECK_HEADERS([yajl/yajl_version.h sys/eventfd.h]) +# Check for libnl3 >=3.2.8. If present enable remus network buffering. +PKG_CHECK_MODULES(LIBNL3, [libnl-3.0 >= 3.2.8 libnl-route-3.0 >= 3.2.8], + [libnl3_lib="y"], [libnl3_lib="n"]) + +# Check for libnl3 command line utilities +PKG_CHECK_EXISTS([libnl-cli-3.0 >= 3.2.8], [libnl3_cli="y"], [libnl3_cli="n"]) + +AS_IF([test "x$libnl3_lib" = "xn" || test "x$libnl3_cli" = "xn"], [ + AC_MSG_WARN([Disabling support for Remus network buffering. + Please install libnl3 libraries, command line tools and devel + headers - version 3.2.8 or higher]) + AC_SUBST(remus_netbuf, [n]) + ],[ + AC_SUBST(LIBNL3_LIBS) + AC_SUBST(LIBNL3_CFLAGS) + AC_SUBST(remus_netbuf, [y]) +]) + AC_OUTPUT() AS_IF([test "x$xend" = "xy" ], [
tools/libxl/Makefile +2 -0
@@ -21,11 +21,13 @@ endif LIBXL_LIBS = LIBXL_LIBS = $(LDLIBS_libxenctrl) $(LDLIBS_libxenguest) $(LDLIBS_libxenstore) $(LDLIBS_libblktapctl) $(PTYFUNCS_LIBS) $(LIBUUID_LIBS) +LIBXL_LIBS += $(LIBNL3_LIBS) CFLAGS_LIBXL += $(CFLAGS_libxenctrl) CFLAGS_LIBXL += $(CFLAGS_libxenguest) CFLAGS_LIBXL += $(CFLAGS_libxenstore) CFLAGS_LIBXL += $(CFLAGS_libblktapctl) +CFLAGS_LIBXL += $(LIBNL3_CFLAGS) CFLAGS_LIBXL += -Wshadow CFLAGS += $(PTHREAD_CFLAGS)
tools/remus/README +6 -0
@@ -2,3 +2,9 @@ Remus provides fault tolerance for virtu checkpoints to a backup, which will activate if the target VM fails. See the website at http://nss.cs.ubc.ca/remus/ for details. + +Using Remus with libxl on Xen 4.4 and higher: + To enable network buffering, you need libnl 3.2.8 + or higher along with the development headers and command line utilities. + If your distro does not have the appropriate libnl3 version, you can find + the latest source tarball of libnl3 at http://www.carisma.slowglass.com/~tgr/libnl/
Re: [PATCH 1 of 5 V3] remus: add libnl3 dependency to autoconf scripts
tools/configure.ac AC_SUBST(libiconv) · line 226
Quoted 12 lines · Shriram Rajagopalan
diff -r
808aba98084e-r357655a1598ctools/configure.actools/configure.ac +5
-0@@ -219,6 +219,24 @@ AC_SUBST(libiconv) # Checks for header files. AC_CHECK_HEADERS([yajl/yajl_version.h sys/eventfd.h]) +# Check for libnl3 >=3.2.8. If present enable remus network buffering. +PKG_CHECK_MODULES(LIBNL3, [libnl-3.0 >= 3.2.8 libnl-route-3.0 >= 3.2.8], + [libnl3_lib="y"], [libnl3_lib="n"]) + +# Check for libnl3 command line utilities
These cli utils are needed on the runtime host but not the build host, right? (for the benefit of the hotplug scripts I suppose?)
This check only runs on the build host, so it will fail needlessly if the tools aren't there while at the same time not catching the case where they are missing from the runtime host.
Best bet is to omit this check and simply list them in the README, in the list of optional tools (and reference tools/remus/README for more info if you like).
Ian.
PATCH v3 1/1 tools/hotplug: Remus network buffering setup scripts
In reply to Shriram Rajagopalan# HG changeset patch
# User Shriram Rajagopalan <[email protected]> # Date 1382274772 25200
# Node ID 0001a8222a785865b753acf8bcdf97c10c9e3819 # Parent 357655a1598cba8f8460758714885563dd2b4a9c tools/hotplug: Remus network buffering setup scripts
This patch introduces remus-netbuf-setup hotplug script responsible for setting up and tearing down the necessary infrastructure required for network output buffering in Remus. This script is intended to be invoked by libxl for each guest interface, when starting or stopping Remus.
Apart from returning success/failure indication via the usual hotplug entries in xenstore, this script also writes to xenstore, the name of the IFB device to be used to control the vif's network output.
The script relies on libnl3 command line utilities to perform various setup/teardown functions. The script is confined to Linux platforms only since NetBSD does not seem to have libnl3.
docs/misc/xenstore-paths.markdown +4 -0
@@ -356,6 +356,10 @@ The guest's virtual time offset from UTC The device model version for a domain. +#### /libxl/$DOMID/remus/netbuf/$DEVID/ifb = STRING [n,INTERNAL] + +IFB device used by Remus to buffer network output from the associated vif. + [BLKIF]: http://xenbits.xen.org/docs/unstable/hypercall/include,public,io,blkif.h.html [FBIF]: http://xenbits.xen.org/docs/unstable/hypercall/include,public,io,fbif.h.html [HVMPARAMS]: http://xenbits.xen.org/docs/unstable/hypercall/include,public,hvm,params.h.html
tools/hotplug/Linux/Makefile +1 -0
@@ -16,6 +16,7 @@ XEN_SCRIPTS += network-nat vif-nat XEN_SCRIPTS += vif-openvswitch XEN_SCRIPTS += vif2 XEN_SCRIPTS += vif-setup +XEN_SCRIPTS-$(CONFIG_REMUS_NETBUF) += remus-netbuf-setup XEN_SCRIPTS += block XEN_SCRIPTS += block-enbd block-nbd XEN_SCRIPTS-$(CONFIG_BLKTAP1) += blktap
tools/hotplug/Linux/remus-netbuf-setup new file +187 -0
187 lines not on this page · raw message
Re: [PATCH 2 of 5 V3] tools/hotplug: Remus network buffering setup scripts
tools/hotplug/Linux/remus-netbuf-setup line 76
Quoted 11 lines · Shriram Rajagopalan
tools/hotplug/Linux/remus-netbuf-setup new file +11
-0+check_libnl_tools() { + if ! command -v nl-qdisc-list > /dev/null 2>&1; then + fatal "Unable to find nl-qdisc-list tool" + fi + if ! command -v nl-qdisc-add > /dev/null 2>&1; then + fatal "Unable to find nl-qdisc-add tool" + fi + if ! command -v nl-qdisc-delete > /dev/null 2>&1; then + fatal "Unable to find nl-qdisc-delete tool" + fi +}
This probably suffices in place of the configure check I commented on earlier.
tools/hotplug/Linux/remus-netbuf-setup line 136
Quoted 12 lines
tools/hotplug/Linux/remus-netbuf-setup new file +12
-0+add_plug_qdisc() { + local vif=$1 + local ifb=$2 + + nl-qdisc-add --dev="$ifb" --parent root plug >/dev/null 2>&1 + if [ $? -ne 0 ] + then + do_without_error tc qdisc del dev "$vif" ingress + fatal "Failed to add plug qdisc to $ifb" + fi + + #set ifb buffering limit in bytes. Its okay if this command
... "fails" ?
Quoted 4 lines
tools/hotplug/Linux/remus-netbuf-setup new file +4
-0+ +case "$command" in + setup) + check_libnl_tools
tools/hotplug/Linux/remus-netbuf-setup line 173
Quoted 10 lines
tools/hotplug/Linux/remus-netbuf-setup new file +10
-0+ check_modules + + claim_lock "pickifb" + setup_ifb + redirect_vif_traffic "$vifname" "$IFB" + add_plug_qdisc "$vifname" "$IFB" + release_lock "pickifb" + + #not using xenstore_write that automatically exits on error + # because we need to cleanup
whitespace inconsistency.
tools/hotplug/Linux/remus-netbuf-setup line 177
Quoted 4 lines
tools/hotplug/Linux/remus-netbuf-setup new file +4
-0+ _xenstore_write "$XENBUS_PATH/ifb" "$IFB" || xs_write_failed "$vifname" "$IFB" + ;; + teardown) + : ${IFB?}
Do you mean log debug or something here?
tools/hotplug/Linux/remus-netbuf-setup line 187
Quoted 10 lines
tools/hotplug/Linux/remus-netbuf-setup new file +10
-0+ teardown_netbuf "$vifname" "$IFB" + ;; +esac + +log debug "Successful remus-netbuf-setup $command for $vifname, ifb $IFB." + +if [ "$command" = "setup" ] +then + success +fi
Why not put this in the case for setup?
Ian.
Re: [PATCH 2 of 5 V3] tools/hotplug: Remus network buffering setup scripts
Quoted 8 lines · Ian Campbell
Quoted 8 lines
tools/hotplug/Linux/remus-netbuf-setup new file +8
-0+ +case "$command" in + setup) + check_libnl_tools + check_modules + + claim_lock "pickifb" + setup_ifb
Quoted 10 lines
tools/hotplug/Linux/remus-netbuf-setup line 173
Quoted 6 lines
tools/hotplug/Linux/remus-netbuf-setup new file +6
-0+ redirect_vif_traffic "$vifname" "$IFB" + add_plug_qdisc "$vifname" "$IFB" + release_lock "pickifb" + + #not using xenstore_write that automatically exits on error + # because we need to cleanupwhitespace inconsistency.
I seem to get this wrong again and again..
Did you mean the space trailing the second # ? or should the code block inside setup/teardown
be indented with two tabs instead of one ?
Quoted 9 lines
Quoted 1 line
+
_xenstore_write"$XENBUS_PATH/ifb" "$IFB" ||xs_write_failed"$vifname" "$IFB"
tools/hotplug/Linux/remus-netbuf-setup line 177
Quoted 3 lines
tools/hotplug/Linux/remus-netbuf-setup new file +3
-0+ ;; + teardown) + : ${IFB?}Do you mean log debug or something here?
This was to just make sure that the IFB variable was supplied as part of the environment..
Just like the two checks on top of this script..
"
: ${vifname?}
: ${XENBUS_PATH?}
"
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel
Re: [PATCH 2 of 5 V3] tools/hotplug: Remus network buffering setup scripts
Quoted 25 lines · Shriram Rajagopalan
On Thu, Oct 31, 2013 at 1:21 PM, Ian Campbell
<[email protected]> wrote:
> +> +case "$command" in
> + setup)
> +check_libnl_tools
> +check_modules
> +
> +claim_lock"pickifb"
> +setup_ifb
> +redirect_vif_traffic"$vifname" "$IFB"
> +add_plug_qdisc"$vifname" "$IFB"
> +release_lock"pickifb"
> +
> + #not usingxenstore_writethat automatically exits on
error
> + # because we need to cleanupwhitespace inconsistency.
Quoted 4 lines
I seem to get this wrong again and again..
Did you mean the space trailing the second # ? or should the code block inside setup/teardown
be indented with two tabs instead of one ?
I meant "#foo" (1st line) vs "# foo" (2nd line) which just caught my eye. They should be the same. More normal would be a space after the # I think.
I don't think we have a particularly strict coding style for shell scripts, but code being consistent with itself is a good start.
Quoted 15 lines
> +
_xenstore_write"$XENBUS_PATH/ifb" "$IFB" ||
xs_write_failed"$vifname" "$IFB"
> + ;;
> + teardown)
> + : ${IFB?}Do you mean log debug or something here?
Quoted 7 lines
This was to just make sure that the IFB variable was supplied as part of the environment..
Just like the two checks on top of this script..
"
: ${vifname?}
: ${XENBUS_PATH?}
"
Do these result in useful error reporting to the end user? Bearing in mind that the end user might be using libxl but not xl and therefore not see stdout/err (which might be going into some daemon's log file).
Also, either I'm missing it or the bash manpage only documents ${parameter:?} and not ${parameter?}. Are both actually valid?
Ian.
Re: [PATCH 2 of 5 V3] tools/hotplug: Remus network buffering setup scripts
Quoted 9 lines · Ian Campbell
Quoted 7 lines
This was to just make sure that the IFB variable was supplied as part of the environment..
Just like the two checks on top of this script..
"
: ${vifname?}
: ${XENBUS_PATH?}
"Do these result in useful error reporting to the end user?
It displays a message "<script name>:lineno: vifname: parameter not set or null"
on stderr.
Quoted 5 lines
Bearing in
mind that the end user might be using libxl but not xl and therefore not see stdout/err (which might be going into some daemon's log file).
These parameters are supposed to be set by libxl before fork+exec-ing the script.
And their values are not obtained from the user or the command line. They are
generated inside libxl, and checked for non-null status. So I dont think we need
to worry about libxl not being able to see the error.
Quoted 4 lines
Also, either I'm missing it or the bash manpage only documents ${parameter:?} and not ${parameter?}. Are both actually valid?
You are right. The man page documents only the ${parameter:?} format, although the "${parameter?}" also seems to work for me. I will stick to the documented format.
Quoted 5 lines
Ian.
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel
PATCH v3 1/1 tools/libxl: setup/teardown Remus network buffering
In reply to Shriram Rajagopalan# HG changeset patch
# User Shriram Rajagopalan <[email protected]> # Date 1382295266 25200
# Node ID d3f088236c550213fc04ed982df47b4771b28d2f # Parent 0001a8222a785865b753acf8bcdf97c10c9e3819 tools/libxl: setup/teardown Remus network buffering
Setup/teardown remus network buffering for a given guest, when libxl_domain_remus_start API is invoked.
This patch does the following during setup:
a) call the hotplug script for each vif to setup its network buffer
b) establish a dedicated remus context containing libnl related state (netlink sockets, qdisc caches, etc.,)
c) Obtain handles to plug qdiscs installed on the IFB devices chosen by the hotplug scripts.
During teardown, the netlink resources are released, followed by invocation of hotplug scripts to remove the IFB devices.
tools/libxl/Makefile +7 -0
@@ -42,6 +42,13 @@ LIBXL_OBJS-y += libxl_blktap2.o else LIBXL_OBJS-y += libxl_noblktap2.o endif + +ifeq ($(CONFIG_REMUS_NETBUF),y) +LIBXL_OBJS-y += libxl_netbuffer.o +else +LIBXL_OBJS-y += libxl_nonetbuffer.o +endif + LIBXL_OBJS-$(CONFIG_X86) += libxl_cpuid.o libxl_x86.o LIBXL_OBJS-$(CONFIG_IA64) += libxl_nocpuid.o libxl_noarch.o LIBXL_OBJS-$(CONFIG_ARM) += libxl_nocpuid.o libxl_noarch.o
tools/libxl/libxl.c +35 -17
83 lines not on this page · raw message
tools/libxl/libxl_dom.c +59 -8
103 lines not on this page · raw message
tools/libxl/libxl_internal.h +45 -2
65 lines not on this page · raw message
tools/libxl/libxl_netbuffer.c new file +516 -0
516 lines not on this page · raw message
tools/libxl/libxl_nonetbuffer.c new file +56 -0
56 lines not on this page · raw message
tools/libxl/libxl_types.idl +2 -0
@@ -539,6 +539,8 @@ libxl_domain_remus_info = Struct("domain ("interval", integer), ("blackhole", bool), ("compression", bool), + ("netbuf", bool), + ("netbufscript", string), ]) libxl_event_type = Enumeration("event_type", [
Re: [PATCH 3 of 5 V3] tools/libxl: setup/teardown Remus network buffering
Quoted 16 lines · Shriram Rajagopalan
# HG changeset patch
# User Shriram Rajagopalan <[email protected]> # Date 1382295266 25200
# Node IDd3f088236c550213fc04ed982df47b4771b28d2f# Parent0001a8222a785865b753acf8bcdf97c10c9e3819tools/libxl: setup/teardown Remus network bufferingSetup/teardown remus network buffering for a given guest, when
libxl_domain_remus_startAPI is invoked.This patch does the following during setup:
a) call the hotplug script for each vif to setup its network bufferb) establish a dedicated remus context containing libnl related state (netlink sockets, qdisc caches, etc.,)
Quoted 7 lines
c) Obtain handles to plug qdiscs installed on the IFB devices chosen by the hotplug scripts.
During teardown, the netlink resources are released, followed by invocation of hotplug scripts to remove the IFB devices.
Signed-off-by Shriram Rajagopalan <[email protected]>
All this async stuff needs Ian J's eye on it really, not mine. It looks vaguely plausible to my untrained eye.
tools/libxl/Makefile LIBXL_OBJS-y += libxl_blktap2.o · line 47
Quoted 10 lines
diff -r
0001a8222a78-rd3f088236c55tools/libxl/Makefiletools/libxl/Makefile +3
-0@@ -42,6 +42,13 @@ LIBXL_OBJS-y += libxl_blktap2.o else LIBXL_OBJS-y += libxl_noblktap2.o endif + +ifeq ($(CONFIG_REMUS_NETBUF),y) +LIBXL_OBJS-y += libxl_netbuffer.o
Should the addition of the libnl stuff to LIBS and CFLAGS have been under here too?
tools/libxl/libxl_types.idl libxl_domain_remus_info = Struct("domain · line 542
Quoted 8 lines
diff -r
0001a8222a78-rd3f088236c55tools/libxl/libxl_types.idltools/libxl/libxl_types.idl +1
-0@@ -539,6 +539,8 @@ libxl_domain_remus_info = Struct("domain ("interval", integer), ("blackhole", bool), ("compression", bool), + ("netbuf", bool),
I guess libxl_defbool doesn't make sense here.
tools/libxl/libxl_types.idl libxl_domain_remus_info = Struct("domain · line 543
Quoted 1 line
tools/libxl/libxl_types.idl +1
-0+ ("netbufscript", string),
This requires a LIBXL_HAVE #define in libxl.h so users know it is available. One for the overall netbuf feature should do I think.
Ian.
PATCH v3 1/1 tools/libxl: Control network buffering in remus callbacks
In reply to Shriram Rajagopalan# HG changeset patch
# User Shriram Rajagopalan <[email protected]> # Date 1382295546 25200
# Node ID a8deb9499e9dcce9869025fa1c02cf2e0d58612a # Parent d3f088236c550213fc04ed982df47b4771b28d2f tools/libxl: Control network buffering in remus callbacks
This patch constitutes the core network buffering logic. and does the following:
a) create a new network buffer when the domain is suspended (remus_domain_suspend_callback)
b) release the previous network buffer pertaining to the committed checkpoint (remus_domain_checkpoint_dm_saved)
tools/libxl/libxl_dom.c +55 -8
@@ -1259,8 +1259,24 @@ void libxl__remus_teardown_done(libxl__estatic int libxl__remus_domain_suspend_callback (void *data) { -/* REMUS TODO: Issue disk and network checkpoint reqs. */ -return libxl__domain_suspend_common_callback (data); +/* REMUS TODO: Issue disk checkpoint reqs. */ +libxl__save_helper_state *shs = data; +libxl__domain_suspend_state *dss =CONTAINER_OF (shs, *dss, shs); +libxl__remus_ctx *remus_ctx = dss->remus_ctx; +bool is_suspended; +STATE_AO_GC (dss->ao); + + is_suspended = !!libxl__domain_suspend_common_callback (data); + +if (!remus_ctx->netbuf_ctx)return is_suspended; + +if (is_suspended) { +if (libxl__remus_netbuf_start_new_epoch (gc, dss->domid, + remus_ctx)) +return !is_suspended; + } + +return is_suspended; }static int libxl__remus_domain_resume_callback (void *data)
@@ -1273,7 +1289,7 @@ static int libxl__remus_domain_resume_caif (libxl__domain_resume (gc, dss->domid,/* Fast Suspend */ 1 ))return 0 ; -/* REMUS TODO: Deal with disk. Start a new network output buffer */ +/* REMUS TODO: Deal with disk. */ return 1 ; }
@@ -1300,11 +1316,42 @@ static void libxl__remus_domain_checkpoistatic void remus_checkpoint_dm_saved (libxl__egc *egc,libxl__domain_suspend_state *dss,int rc) { -/* REMUS TODO: Wait for disk and memory ack, release network buffer */ -/* REMUS TODO: make this asynchronous */ -assert (!rc);/* REMUS TODO handle this error properly */ -usleep (dss->remus_ctx->interval *1000 ); -libxl__xc_domain_saverestore_async_callback_done (egc, &dss->shs,1 ); +/* +* REMUS TODO: Wait for disk and explicit memory ack (through restore +* callback from remote) before releasing network buffer. +*/ +libxl__remus_ctx *remus_ctx = dss->remus_ctx; +struct timespec epoch; +int do_next_iter =0 ; +STATE_AO_GC (dss->ao); + +if (rc) { +LOG (ERROR ,"Failed to save device model. Terminating Remus.." ); + goto out; + } + +if (remus_ctx->netbuf_ctx) { + rc =libxl__remus_netbuf_release_prev_epoch (gc, dss->domid, + remus_ctx); +if (rc) { +LOG (ERROR ,"Failed to release network buffer." +" Terminating Remus.." ); + goto out; + } + } + + epoch.tv_sec = remus_ctx->interval /1000 ;/* interval is in ms */ + epoch.tv_nsec = remus_ctx->interval *1000L *1000L ; +nanosleep (&epoch,0 ); +/* +* Set return value to 1, so that the infinite checkpoint cycle +* continues. See xc_domain_save.c: xc_domain_save() +*/ + do_next_iter =1 ; + + out: +libxl__xc_domain_saverestore_async_callback_done (egc, &dss->shs, + do_next_iter); }/*----- main code for suspending, in order of execution -----*/
Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks
Quoted 7 lines · Shriram Rajagopalan
# HG changeset patch
# User Shriram Rajagopalan <[email protected]> # Date 1382295546 25200
# Node IDa8deb9499e9dcce9869025fa1c02cf2e0d58612a# Parentd3f088236c550213fc04ed982df47b4771b28d2ftools/libxl: Control network buffering in remus callbacks
Quoted 8 lines
This patch constitutes the core network buffering logic. and does the following:
a) create a new network buffer when the domain is suspended (remus_domain_suspend_callback)
b) release the previous network buffer pertaining to the committed checkpoint (remus_domain_checkpoint_dm_saved)Signed-off-by Shriram Rajagopalan <[email protected]>
Another one where Ian J's opinion matters more than mine, but it looks ok to me.
Ian.
Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks
In reply to Shriram RajagopalanShriram Rajagopalan writes ("[PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks"):
Quoted 1 line
tools/libxl: Control network buffering in remus callbacks
...
tools/libxl/libxl_dom.c void libxl__remus_teardown_done(libxl__e · line 1269
Quoted 12 lines
static int
libxl__remus_domain_suspend_callback(void *data){tools/libxl/libxl_dom.c +8
-2-/* REMUS TODO: Issue disk and network checkpoint reqs. */ -return libxl__domain_suspend_common_callback (data); +/* REMUS TODO: Issue disk checkpoint reqs. */ +libxl__save_helper_state *shs = data; +libxl__domain_suspend_state *dss =CONTAINER_OF (shs, *dss, shs); +libxl__remus_ctx *remus_ctx = dss->remus_ctx; +bool is_suspended; +STATE_AO_GC (dss->ao); + + is_suspended = !!libxl__domain_suspend_common_callback (data);
I think this function would be less confusing if the return value variable was called "ok" or something similar. It's really just an error flag.
Also AFAICT the !! has no function here. The return value is essentially a boolean.
tools/libxl/libxl_dom.c void libxl__remus_teardown_done(libxl__e · line 1276
Quoted 6 lines
tools/libxl/libxl_dom.c +6
-0+if (!remus_ctx->netbuf_ctx)return is_suspended; + +if (is_suspended) { +if (libxl__remus_netbuf_start_new_epoch (gc, dss->domid, + remus_ctx)) +return !is_suspended;
This construction is rather odd. Why return !is_suspended when you know that !!is_suspended ?
Quoted 6 lines
tools/libxl/libxl_dom.c +0
-2@@ -1300,11 +1316,42 @@ static void libxl__remus_domain_checkpoistatic void remus_checkpoint_dm_saved (libxl__egc *egc,libxl__domain_suspend_state *dss,int rc) { -/* REMUS TODO: Wait for disk and memory ack, release network buffer */ -/* REMUS TODO: make this asynchronous */
tools/libxl/libxl_dom.c static void libxl__remus_domain_checkpoi · line 1325
Quoted 10 lines
tools/libxl/libxl_dom.c +7
-3-assert (!rc);/* REMUS TODO handle this error properly */ -usleep (dss->remus_ctx->interval *1000 ); -libxl__xc_domain_saverestore_async_callback_done (egc, &dss->shs,1 ); +/* +* REMUS TODO: Wait for disk and explicit memory ack (through restore +* callback from remote) before releasing network buffer. +*/ +libxl__remus_ctx *remus_ctx = dss->remus_ctx; +struct timespec epoch; +int do_next_iter =0 ;
Again, this variable is probably best called "ok".
tools/libxl/libxl_dom.c static void libxl__remus_domain_checkpoi · line 1345
Quoted 3 lines
tools/libxl/libxl_dom.c +3
-0+ epoch.tv_sec = remus_ctx->interval /1000 ;/* interval is in ms */ + epoch.tv_nsec = remus_ctx->interval *1000L *1000L ; +nanosleep (&epoch,0 );
This is no good, I'm afraid. You should be using a libxl event loop timer. (Also why are you changing your existing usleep to this new nanosleep which seems functionally very similar?)
Ian.
Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks
Quoted 60 lines · Ian Jackson
Shriram Rajagopalan writes ("[PATCH 4 of 5 V3]
tools/libxl: Control network buffering in remus callbacks"):Quoted 1 line
tools/libxl: Control network buffering in remus callbacks...
tools/libxl/libxl_dom.c void libxl__remus_teardown_done(libxl__e · line 1269
Quoted 12 lines
static int
libxl__remus_domain_suspend_callback(void *data){tools/libxl/libxl_dom.c +8
-2-/* REMUS TODO: Issue disk and network checkpoint reqs. */ -return libxl__domain_suspend_common_callback (data); +/* REMUS TODO: Issue disk checkpoint reqs. */ +libxl__save_helper_state *shs = data; +libxl__domain_suspend_state *dss =CONTAINER_OF (shs, *dss, shs); +libxl__remus_ctx *remus_ctx = dss->remus_ctx; +bool is_suspended; +STATE_AO_GC (dss->ao); + + is_suspended = !!libxl__domain_suspend_common_callback (data);I think this function would be less confusing if the return value variable was called "ok" or something similar. It's really just an error flag.
Also AFAICT the !! has no function here. The return value is essentially a boolean.
tools/libxl/libxl_dom.c void libxl__remus_teardown_done(libxl__e · line 1276
Quoted 6 lines
tools/libxl/libxl_dom.c +6
-0+if (!remus_ctx->netbuf_ctx)return is_suspended; + +if (is_suspended) { +if (libxl__remus_netbuf_start_new_epoch (gc, dss->domid, + remus_ctx)) +return !is_suspended;This construction is rather odd. Why return !is_suspended when you know that !!is_suspended ?
Review static void libxl__remus_domain_checkpoi · line 1317
Quoted 3 lines
(quoted diff) +0
-0@@ -1300,11 +1316,42 @@ static void libxl__remus_domain_checkpoi static void remus_checkpoint_dm_saved(libxl__egc *egc, libxl__domain_suspend_state *dss,int rc)
Review static void libxl__remus_domain_checkpoi · line 1303
Quoted 2 lines
(quoted diff) +0
-1{ - /* REMUS TODO: Wait for disk and memory ack, release network buffer*/
tools/libxl/libxl_dom.c static void libxl__remus_domain_checkpoi
Quoted 6 lines
tools/libxl/libxl_dom.c +2
-4-/* REMUS TODO: make this asynchronous */ -assert (!rc);/* REMUS TODO handle this error properly */ -usleep (dss->remus_ctx->interval *1000 ); -libxl__xc_domain_saverestore_async_callback_done (egc, &dss->shs,1 ); +/* +* REMUS TODO: Wait for disk and explicit memory ack (through restore
tools/libxl/libxl_dom.c static void libxl__remus_domain_checkpoi · line 1325
Quoted 5 lines
tools/libxl/libxl_dom.c +5
-0+* callback from remote) before releasing network buffer. +*/ +libxl__remus_ctx *remus_ctx = dss->remus_ctx; +struct timespec epoch; +int do_next_iter =0 ;Again, this variable is probably best called "ok".
tools/libxl/libxl_dom.c static void libxl__remus_domain_checkpoi · line 1345
Quoted 3 lines
tools/libxl/libxl_dom.c +3
-0+ epoch.tv_sec = remus_ctx->interval /1000 ;/* interval is in ms */ + epoch.tv_nsec = remus_ctx->interval *1000L *1000L ; +nanosleep (&epoch,0 );
Quoted 7 lines
This is no good, I'm afraid. You should be using a libxl event loop timer. (Also why are you changing your existing usleep to this new nanosleep which seems functionally very similar?)
Ian.
Nanosleep was more of my personal preference, just as the '!!' above was Andrew's suggestion, due to the fact that libxl__domain_suspend_common_callback()
does not strictly return boolean. But I agree that renaming that variable makes more sense.
As far as using the event loop timers, I couldn't find anything suitable in libxl_event.h/c that
I could use instead of usleep/nanosleep. Specifically, they were all asynchronous, while the
caller of these functions is libxc's xc_domain_save infinite loop (synchronous).
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel
Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks [and 1 more messages]
Shriram Rajagopalan writes ("Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks"):
Quoted 1 line
Nanosleep was more of my personal preference,
I don't think that's a good enough reason for the churn, but as I say this really ought to be replaced with use of a timeout event.
Quoted 3 lines
just as the '!!' above was Andrew's suggestion, due to the fact that
libxl__domain_suspend_common_callback()does not strictly return boolean.
Are you looking at a different tree to me ? I have checked the return statements in libxl__domain_suspend_common_callback and they are all "return 0" or "return 1".
Quoted 1 line
But I agree that renaming that variable makes more sense.
Good :-). Thanks.
Quoted 5 lines
As far as using the event loop timers, I couldn't find anything suitable in libxl_event.h/c that I could use instead of usleep/nanosleep. Specifically, they were all asynchronous, while the caller of these functions is libxc's
xc_domain_saveinfinite loop (synchronous).
No, you need to make it an asynchronous call.
The libxl save helper machinery arranges to run xc_domain_save in a subprocess so that the main program can continue asynchronously.
It's true that the suspend checkpoint is currently synchronous. It needs to be made synchronous by telling the stub generator (libxl_save_msgs_gen.pl) to generate an asynchronous binding, like it does for switch_qemu_logdirty.
Perhaps it would be helpful if I provided a pre-patch to make that change for you.
Quoted 9 lines
tools/libxl/libxl_dom.c void libxl__remus_teardown_done(libxl__e · line 1276
Quoted 6 lines
tools/libxl/libxl_dom.c +6
-0+if (!remus_ctx->netbuf_ctx)return is_suspended; + +if (is_suspended) { +if (libxl__remus_netbuf_start_new_epoch (gc, dss->domid, + remus_ctx)) +return !is_suspended;This construction is rather odd. Why return !is_suspended when you know that !!is_suspended ?
...
Quoted 2 lines
If a new network buffer cannot be created (for the next epoch), then return an error. This ends up terminating the checkpointing process.
Yes. But why not "return 0" ?
Thanks,
Ian.
Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks [and 1 more messages]
Quoted 8 lines · Ian Jackson
Shriram Rajagopalan writes ("Re: [PATCH 4 of 5 V3]
tools/libxl: Control network buffering in remus callbacks"):Quoted 1 line
Nanosleep was more of my personal preference,
I don't think that's a good enough reason for the churn, but as I say this really ought to be replaced with use of a timeout event.
Fair enough. My question is what is the overhead of setting up, firing and tearing down
a timeout event using the event gen framework, if I wish to checkpoint the VM, say every 20ms ?
If you happen to have the numbers off the top of your head, it would help. Or if you are sure that
the system can easily handle this rate of events with very little overhead (<0.2ms per event)
Quoted 11 lines
Quoted 5 lines
As far as using the event loop timers, I couldn't find anything suitable in libxl_event.h/c that I could use instead of usleep/nanosleep. Specifically, they were all asynchronous, while the caller of these functions is libxc's
xc_domain_saveinfinite loop (synchronous).No, you need to make it an asynchronous call.
The libxl save helper machinery arranges to run
xc_domain_savein a subprocess so that the main program can continue asynchronously.
Quoted 9 lines
It's true that the suspend checkpoint is currently synchronous. It needs to be made synchronous by telling the stub generator (libxl_save_msgs_gen.pl) to generate an asynchronous binding, like it does for
switch_qemu_logdirty.Perhaps it would be helpful if I provided a pre-patch to make that change for you.
Yes, that would be pretty helpful. Thanks!
shriram
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel
Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks [and 1 more messages]
Quoted 14 lines · Shriram Rajagopalan
On Mon, Nov 4, 2013 at 6:12 AM, Ian Jackson <[email protected]> wrote:
Shriram Rajagopalan writes ("Re: [PATCH 4 of 5 V3]tools/libxl: Control network buffering in remus callbacks"):
> Nanosleep was more of my personal preference,I don't think that's a good enough reason for the churn, but as I say
this really ought to be replaced with use of a timeout event.
Quoted 8 lines
Fair enough. My question is what is the overhead of setting up, firing and tearing down
a timeout event using the event gen framework, if I wish to checkpoint the VM, say every 20ms ?
If you happen to have the numbers off the top of your head, it would help. Or if you are sure that
the system can easily handle this rate of events with very little overhead (<0.2ms per event)
Regardless of the answer here, would it make sense to do some/all of the checkpoint processing in the helper subprocess anyway and only signal the eventual failover up to the libxl process?
This async op is potentially quite long running I think compared to a normal one i.e. if the guest doesn't die it is expected that the ao lives "forever". Since the associated gc's persist until the ao ends this might end up accumulating lots of allocations? Ian had a similar concern about Roger's hotplug daemon series and suggested creating a per iteration gc or something.
Ian.
Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks [and 1 more messages]
Ian Campbell writes ("Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks [and 1 more messages]"):
Quoted 3 lines
Regardless of the answer here, would it make sense to do some/all of the checkpoint processing in the helper subprocess anyway and only signal the eventual failover up to the libxl process?
It might do. Mind you, the code in libxc is tangled enough as it is and is due for a rewrite. Perhaps this could be done in the helper executable, although there isn't currently any way to easily intersperse code in there.
Quoted 6 lines
This async op is potentially quite long running I think compared to a normal one i.e. if the guest doesn't die it is expected that the ao lives "forever". Since the associated gc's persist until the ao ends this might end up accumulating lots of allocations? Ian had a similar concern about Roger's hotplug daemon series and suggested creating a per iteration gc or something.
Yes, this is indeed a problem. Well spotted.
Which of the xc_domain_save (and _restore) callbacks are called each remus iteration ?
I think introducing a per-iteration gc here is going to involve taking some care, since we need to be sure not to put per-iteraton-gc-allocated objects into data structures which are used by subsequent iterations.
Quoted 3 lines · Shriram
Fair enough. My question is what is the overhead of setting up, firing and tearing down a timeout event using the event gen framework, if I wish to checkpoint the VM, say every 20ms ?
The ultimate cost of going back into the event loop to wait for a timeout will depend on what else the process is doing. If the process is doing nothing else, it's about two calls to gettimeofday and one to poll. Plus a bit of in-process computation, but that's going to be swamped by system call overhead.
Having said that, libxl is not performance-optimised. Indeed the callback mechanism involves context switching, and IPC, between the save/restore helper and libxl proper. Probably not too much to be doing every 20ms for a single domain, but if you have a lot of these it's going to end up taking a lot of dom0 cpu etc.
I assume you're not doing this for HVM domains, which involve saving the qemu state each time too.
Ian.
Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks [and 1 more messages]
Quoted 8 lines · Ian Jackson
Ian Campbell writes ("Re: [PATCH 4 of 5 V3]
tools/libxl: Control network buffering in remus callbacks [and 1 more messages]"):Quoted 3 lines
Regardless of the answer here, would it make sense to do some/all of the checkpoint processing in the helper subprocess anyway and only signal the eventual failover up to the libxl process?
It might do. Mind you, the code in libxc is tangled enough as it is and is due for a rewrite. Perhaps this could be done in the helper executable,
That's where I meant.
Quoted 2 lines
although there isn't currently any way to easily
intersperse code in there.
A simple matter of programming surely ;-)
Ian.
Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks [and 1 more messages]
In reply to Ian JacksonQuoted 6 lines · Ian Jackson
Ian Campbell writes ("Re: [PATCH 4 of 5 V3]
tools/libxl: Control network buffering in remus callbacks [and 1 more messages]"):Quoted 3 lines
Regardless of the answer here, would it make sense to do some/all of the checkpoint processing in the helper subprocess anyway and only signal the eventual failover up to the libxl process?
Quoted 13 lines
It might do. Mind you, the code in libxc is tangled enough as it is and is due for a rewrite. Perhaps this could be done in the helper executable, although there isn't currently any way to easily intersperse code in there.
This async op is potentially quite long running I think compared to a
Quoted 5 lines
normal one i.e. if the guest doesn't die it is expected that the ao lives "forever". Since the associated gc's persist until the ao ends this might end up accumulating lots of allocations? Ian had a similar concern about Roger's hotplug daemon series and suggested creating a per iteration gc or something.
Yes, this is indeed a problem. Well spotted.
Quoted 1 line
Which of the xc_domain_save (and _restore) callbacks are called each
Quoted 3 lines
remus iteration ?
Almost all of them on the xc_domain_save side. (suspend, resume, save_qemu state, checkpoint).xc_domain_restore doesn't have any callbacks AFAIK. And remus as of now does not have a component on the restore side. It piggybacks on live migration's
restore framework.
Quoted 6 lines
I think introducing a per-iteration gc here is going to involve taking some care, since we need to be sure not to put per-iteraton-gc-allocated objects into data structures which are used by subsequent iterations.
FWIW, the remus related code that executes per iteration does not allocate anything.
All allocations happen only during setup and I was under the impression that no other
allocations are taking place everytime xc_domain_save calls back into libxl.
However, it may be possible that other parts of the AO machinery (and there are a lot of them) are allocating stuff per iteration. And if that is the case, it could
easily lead to OOMs since Remus technically runs as long as the domain lives.
Quoted 10 lines · Shriram
Quoted 3 lines
Fair enough. My question is what is the overhead of setting up, firing and tearing down a timeout event using the event gen framework, if I wish to checkpoint the VM, say every 20ms ?
The ultimate cost of going back into the event loop to wait for a timeout will depend on what else the process is doing. If the process is doing nothing else, it's about two calls to gettimeofday and one to poll. Plus a bit of in-process computation, but that's going to be swamped by system call overhead.
Quoted 7 lines
Having said that, libxl is not performance-optimised. Indeed the callback mechanism involves context switching, and IPC, between the save/restore helper and libxl proper. Probably not too much to be doing every 20ms for a single domain, but if you have a lot of these it's going to end up taking a lot of dom0 cpu etc.
Yes and that is a problem. Xend+Remus avoided this by linking the libcheckpoint library that interfaced with both the python & libxc code.
Quoted 4 lines
I assume you're not doing this for HVM domains, which involve saving the qemu state each time too.
It includes HVM domains too. Although in that case, xenstore based suspend takes about 5ms. So the checkpoint interval is typically 50ms or so.
If there is a latency sensitive task running inside the VM, lower checkpoint interval leads to better performance.
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel
Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks [and 1 more messages]
Shriram Rajagopalan writes ("Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks [and 1 more messages]"):
Quoted 6 lines · Ian Jackson
Which of the
xc_domain_save(and_restore) callbacks are called each remus iteration ?Almost all of them on the
xc_domain_saveside. (suspend, resume,save_qemustate, checkpoint).
Right.
Quoted 4 lines
xc_domain_restoredoesn't have any
callbacks AFAIK. And remus as of now does not have a component on the restore side. It piggybacks on live migration's restore framework.
But the libxl memory management in the restore code is currently written to assume a finite lifetime for the ao. So I think this needs to be improved.
Perhaps all the suspend/restore callbacks should each get one of the nested ao type things that Roger needs for his driver domain daemon.
Quoted 4 lines
FWIW, the remus related code that executes per iteration does not allocate anything. All allocations happen only during setup and I was under the impression that no other allocations are taking place everytime
xc_domain_savecalls back into libxl.
If this is true, then good, because we don't need to do anything, but there is a lot of code there and I would want to check.
Quoted 4 lines
However, it may be possible that other parts of the AO machinery (and there are a lot of them) are allocating stuff per iteration. And if that is the case, it could easily lead to OOMs since Remus technically runs as long as the domain lives.
The ao and event machinery doesn't do much allocation itself.
Quoted 8 lines
Having said that, libxl is not performance-optimised. Indeed the callback mechanism involves context switching, and IPC, between the save/restore helper and libxl proper. Probably not too much to be doing every 20ms for a single domain, but if you have a lot of these it's going to end up taking a lot of dom0 cpu etc.
Yes and that is a problem. Xend+Remus avoided this by linking the libcheckpoint library that interfaced with both the python & libxc code.
Have you observed whether the performance is acceptable with your V3 patches ?
Quoted 2 lines
I assume you're not doing this for HVM domains, which involve saving the qemu state each time too.
...
Quoted 2 lines
It includes HVM domains too. Although in that case, xenstore based suspend takes about 5ms. So the checkpoint interval is typically 50ms or so.
Right.
Quoted 2 lines
If there is a latency sensitive task running inside the VM, lower checkpoint interval leads to better performance.
Yes.
Ian.
Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks [and 1 more messages]
Quoted 7 lines · Ian Jackson
Quoted 1 line
Having said that, libxl is not performance-optimised. Indeed the
callback mechanism involves context switching, and IPC, between the
Quoted 4 lines
save/restore helper and libxl proper. Probably not too much to be doing every 20ms for a single domain, but if you have a lot of these it's going to end up taking a lot of dom0 cpu etc.
Quoted 8 lines
Quoted 2 lines
Yes and that is a problem. Xend+Remus avoided this by linking the libcheckpoint library that interfaced with both the python & libxc
code.
Have you observed whether the performance is acceptable with your V3 patches ?
Frankly I don't know. At the moment, I am trying to make sure that the libxl-remus
implementation is correct before I attempt to make it fast.
With/without these patches, there is a huge overhead per checkpoint. I cannot get to
a 20ms checkpoint interval. Time to suspend/resume a domain is on the order of tens of
milliseconds even if it was a PV domain with fast suspend support -- where it used to
take under 1ms to suspend, checkpoint and resume an idle PV domain on Xend+Remus.
This was one of the issues I raised a long time back. And this is currently the second
element in my queue, the first being to get the patches mainline, then slowly tighten up
relevant code for performance.
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel
Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks [and 1 more messages]
Shriram Rajagopalan writes ("Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks [and 1 more messages]"):
Quoted 10 lines
Frankly I don't know. At the moment, I am trying to make sure that the libxl-remus implementation is correct before I attempt to make it fast.
With/without these patches, there is a huge overhead per checkpoint. I cannot get to a 20ms checkpoint interval. Time to suspend/resume a domain is on the order of tens of milliseconds even if it was a PV domain with fast suspend support -- where it used to take under 1ms to suspend, checkpoint and resume an idle PV domain on Xend+Remus.
Oh dear.
Quoted 4 lines
This was one of the issues I raised a long time back. And this is currently the second element in my queue, the first being to get the patches mainline, then slowly tighten up relevant code for performance.
I think this is probably a bad approach. Under the circumstances, it seems like you really need to move the relevant code lower in the call stack. In particular, you want everything to be handled synchronously in the same process as is doing the save/restore, at least for PV domains.
That's likely to involve substantially rewriting it.
Ian.
Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks [and 1 more messages] [and 1 more messages]
In reply to Shriram RajagopalanShriram Rajagopalan writes ("Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks [and 1 more messages]"):
Quoted 4 lines
On Mon, Nov 4, 2013 at 6:12 AM, Ian Jackson <[email protected]> wrote> Perhaps it would be helpful if I provided a pre-patch to make that
change for you.Yes, that would be pretty helpful. Thanks!
See below. I have compiled this but not tested it. It should be safe but I can't rule out having perpetrated some howler of a bug. Please let me know if it doesn't work.
Ian Jackson writes ("Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks [and 1 more messages]"):
Quoted 11 lines
Quoted 3 lines · Shriram
Fair enough. My question is what is the overhead of setting up, firing and tearing down a timeout event using the event gen framework, if I wish to checkpoint the VM, say every 20ms ?
The ultimate cost of going back into the event loop to wait for a timeout will depend on what else the process is doing. If the process is doing nothing else, it's about two calls to gettimeofday and one to poll. Plus a bit of in-process computation, but that's going to be swamped by system call overhead.
Quoted 8 lines
Having said that, libxl is not performance-optimised. Indeed the callback mechanism involves context switching, and IPC, between the save/restore helper and libxl proper. Probably not too much to be doing every 20ms for a single domain, but if you have a lot of these it's going to end up taking a lot of dom0 cpu etc.
I assume you're not doing this for HVM domains, which involve saving the qemu state each time too.
I guess another way to look at this is that changing this one timeout from a synchronous to asynchronous version is not going to make any noticeable difference to the performance of the whole thing. You're already using all of the asynchronous save/restore helper machinery and the libxl event loop.
So if the performance of your V3 patches is acceptable, this will be fine too.
Ian.
Quoted 1 line
From
46b08918302a8c1d2e470b7af7045557f73afde9Mon Sep 17 00:00:00 2001
From: Ian Jackson <[email protected]>
Date: Mon, 4 Nov 2013 16:27:53 +0000
Subject: [PATCH] libxl: make libxl__domain_suspend_callback be asynchronous
Mark the suspend callback as asynchronous in the helper stub generator (libxl_save_msgs_gen.pl). Remus is going to want to provide an asynchronous version of this function.
libxl__domain_suspend_common_callback, the common synchronous core, which used to be provided directly as the callback function for the helper machinery, becomes libxl__domain_suspend_callback_common. It can now take a typesafe parameter.
For now, provide two very similar asynchronous wrappers for it. Each is a simple function which contains only boilerplate, calls the common synchronous core, and returns the asynchronous response.
Essentially, we have just moved (in the case of suspend callbacks) the call site of libxl__srm_callout_callback_complete. It was in the switch statement in the autogenerated _libxl_save_msgs_callout.c, and is now in the handwritten libxl_dom.c.
No functional change.
--- tools/libxl/libxl_dom.c | 25 +++++++++++++++++++------ tools/libxl/libxl_internal.h | 2 +- tools/libxl/libxl_save_msgs_gen.pl | 2 +- 3 files changed, 21 insertions(+), 8 deletions(-)
tools/libxl/libxl_dom.c +19 -6
@@ -1003,10 +1003,8 @@ int libxl__domain_resume_device_model(libxl__gc *gc, uint32_t domid)return 0 ; } -int libxl__domain_suspend_common_callback (void *user) +int libxl__domain_suspend_callback_common (libxl__domain_suspend_state *dss) { -libxl__save_helper_state *shs = user; -libxl__domain_suspend_state *dss =CONTAINER_OF (shs, *dss, shs);STATE_AO_GC (dss->ao);unsigned long hvm_s_state =0 , hvm_pvdrv =0 ;int ret;
@@ -1225,12 +1223,27 @@ int libxl__toolstack_save(uint32_t domid, uint8_t **buf,return 0 ; } +static void libxl__domain_suspend_callback (void *data) +{ +libxl__save_helper_state *shs = data; +libxl__egc *egc = shs->egc; +libxl__domain_suspend_state *dss =CONTAINER_OF (shs, *dss, shs); + +int ok =libxl__domain_suspend_callback_common (dss); +libxl__xc_domain_saverestore_async_callback_done (egc, shs, ok); +} +/*----- remus callbacks -----*/ -static int libxl__remus_domain_suspend_callback (void *data) +static void libxl__remus_domain_suspend_callback (void *data) { +libxl__save_helper_state *shs = data; +libxl__egc *egc = shs->egc; +libxl__domain_suspend_state *dss =CONTAINER_OF (shs, *dss, shs); +/* REMUS TODO: Issue disk and network checkpoint reqs. */ -return libxl__domain_suspend_common_callback (data); +int ok =libxl__domain_suspend_callback_common (dss); +libxl__xc_domain_saverestore_async_callback_done (egc, shs, ok); }static int libxl__remus_domain_resume_callback (void *data)
@@ -1354,7 +1367,7 @@ void libxl__domain_suspend(libxl__egc *egc, libxl__domain_suspend_state *dss) callbacks->postcopy = libxl__remus_domain_resume_callback; callbacks->checkpoint = libxl__remus_domain_checkpoint_callback; }else - callbacks->suspend = libxl__domain_suspend_common_callback; + callbacks->suspend = libxl__domain_suspend_callback; callbacks->switch_qemu_logdirty = libxl__domain_suspend_common_switch_qemu_logdirty; dss->shs.callbacks.save.toolstack_save = libxl__toolstack_save;
tools/libxl/libxl_internal.h +1 -1
@@ -2551,7 +2551,7 @@ _hidden void libxl__xc_domain_save_done(libxl__egc*, void *dss_void,void libxl__xc_domain_saverestore_async_callback_done (libxl__egc *egc,libxl__save_helper_state *shs,int return_value); -_hidden int libxl__domain_suspend_common_callback(void *data); +_hidden int libxl__domain_suspend_callback_common(libxl__domain_suspend_state*); _hidden void libxl__domain_suspend_common_switch_qemu_logdirty (int domid,unsigned int enable,void *data); _hidden int libxl__toolstack_save(uint32_t domid, uint8_t **buf,
tools/libxl/libxl_save_msgs_gen.pl +1 -1
@@ -23,7 +23,7 @@ our @msgs = ( STRING doing_what), 'unsigned long', 'done', 'unsigned long', 'total'] ], - [ 3, 'scxW', "suspend", [] ], + [ 3, 'scxA', "suspend", [] ], [ 4, 'scxW', "postcopy", [] ], [ 5, 'scxA', "checkpoint", [] ], [ 6, 'scxA', "switch_qemu_logdirty", [qw(int domid
Signature · 1 line
1.7.10.4
Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks [and 1 more messages] [and 1 more messages]
Quoted 173 lines · Ian Jackson
Shriram Rajagopalan writes ("Re: [PATCH 4 of 5 V3]
tools/libxl: Control network buffering in remus callbacks [and 1 more messages]"):Quoted 1 line
On Mon, Nov 4, 2013 at 6:12 AM, Ian Jackson <[email protected]>
wrote> Perhaps it would be helpful if I provided a pre-patch to make
thatQuoted 3 lines
change for you.
Yes, that would be pretty helpful. Thanks!
See below. I have compiled this but not tested it. It should be safe but I can't rule out having perpetrated some howler of a bug. Please let me know if it doesn't work.
Ian Jackson writes ("Re: [PATCH 4 of 5 V3]
tools/libxl: Control network buffering in remus callbacks [and 1 more messages]"):Quoted 11 lines
Quoted 3 lines · Shriram
Fair enough. My question is what is the overhead of setting up, firing and tearing down a timeout event using the event gen framework, if I wish to checkpoint the VM, say every 20ms ?
The ultimate cost of going back into the event loop to wait for a timeout will depend on what else the process is doing. If the process is doing nothing else, it's about two calls to gettimeofday and one to poll. Plus a bit of in-process computation, but that's going to be swamped by system call overhead.
Quoted 8 lines
Having said that, libxl is not performance-optimised. Indeed the callback mechanism involves context switching, and IPC, between the save/restore helper and libxl proper. Probably not too much to be doing every 20ms for a single domain, but if you have a lot of these it's going to end up taking a lot of dom0 cpu etc.
I assume you're not doing this for HVM domains, which involve saving the qemu state each time too.
I guess another way to look at this is that changing this one timeout from a synchronous to asynchronous version is not going to make any noticeable difference to the performance of the whole thing. You're already using all of the asynchronous save/restore helper machinery and the libxl event loop.
So if the performance of your V3 patches is acceptable, this will be fine too.
Ian.
Quoted 1 line
From
46b08918302a8c1d2e470b7af7045557f73afde9Mon Sep 17 00:00:00 2001From: Ian Jackson <[email protected]>
Date: Mon, 4 Nov 2013 16:27:53 +0000
Subject: [PATCH] libxl: makelibxl__domain_suspend_callbackbe asynchronousMark the suspend callback as asynchronous in the helper stub generator (libxl_save_msgs_gen.pl). Remus is going to want to provide an asynchronous version of this function.
libxl__domain_suspend_common_callback, the common synchronous core, which used to be provided directly as the callback function for the helper machinery, becomeslibxl__domain_suspend_callback_common. It can now take a typesafe parameter.For now, provide two very similar asynchronous wrappers for it. Each is a simple function which contains only boilerplate, calls the common synchronous core, and returns the asynchronous response.
Essentially, we have just moved (in the case of suspend callbacks) the call site of
libxl__srm_callout_callback_complete. It was in the switch statement in the autogenerated_libxl_save_msgs_callout.c, and is now in the handwrittenlibxl_dom.c.No functional change.
Signed-off-by Ian Jackson <[email protected]>Cc Shriram Rajagopalan <[email protected]>Cc Ian Campbell <[email protected]>--- tools/libxl/libxl_dom.c | 25 +++++++++++++++++++------ tools/libxl/libxl_internal.h | 2 +- tools/libxl/libxl_save_msgs_gen.pl | 2 +- 3 files changed, 21 insertions(+), 8 deletions(-)tools/libxl/libxl_dom.c +1
-3@@ -1003,10 +1003,8 @@ int libxl__domain_resume_device_model(libxl__gc *gc, uint32_t domid)return 0 ; } -int libxl__domain_suspend_common_callback (void *user) +int libxl__domain_suspend_callback_common (libxl__domain_suspend_state *dss) { -libxl__save_helper_state *shs = user; -libxl__domain_suspend_state *dss =CONTAINER_OF (shs, *dss, shs);STATE_AO_GC (dss->ao);unsigned long hvm_s_state = 0, hvm_pvdrv = 0; int ret;tools/libxl/libxl_dom.c +17
-2@@ -1225,12 +1223,27 @@ int libxl__toolstack_save(uint32_t domid, uint8_t **buf, return 0; } +static void libxl__domain_suspend_callback (void *data) +{ +libxl__save_helper_state *shs = data; +libxl__egc *egc = shs->egc; +libxl__domain_suspend_state *dss =CONTAINER_OF (shs, *dss, shs); + +int ok =libxl__domain_suspend_callback_common (dss); +libxl__xc_domain_saverestore_async_callback_done (egc, shs, ok); +} +/*----- remus callbacks -----*/ -static int libxl__remus_domain_suspend_callback (void *data) +static void libxl__remus_domain_suspend_callback (void *data) { +libxl__save_helper_state *shs = data; +libxl__egc *egc = shs->egc; +libxl__domain_suspend_state *dss =CONTAINER_OF (shs, *dss, shs); +/* REMUS TODO: Issue disk and network checkpoint reqs. */ -return libxl__domain_suspend_common_callback (data); +int ok =libxl__domain_suspend_callback_common (dss); +libxl__xc_domain_saverestore_async_callback_done (egc, shs, ok); }static int
libxl__remus_domain_resume_callback(void *data)tools/libxl/libxl_dom.c +1
-1@@ -1354,7 +1367,7 @@ void libxl__domain_suspend(libxl__egc *egc, libxl__domain_suspend_state *dss) callbacks->postcopy = libxl__remus_domain_resume_callback; callbacks->checkpoint = libxl__remus_domain_checkpoint_callback; }else - callbacks->suspend = libxl__domain_suspend_common_callback; + callbacks->suspend = libxl__domain_suspend_callback; callbacks->switch_qemu_logdirty =libxl__domain_suspend_common_switch_qemu_logdirty; dss->shs.callbacks.save.toolstack_save = libxl__toolstack_save;tools/libxl/libxl_internal.h +1
-1@@ -2551,7 +2551,7 @@ _hidden void libxl__xc_domain_save_done(libxl__egc*,void *dss_void, void libxl__xc_domain_saverestore_async_callback_done(libxl__egc *egc,libxl__save_helper_state *shs,int return_value); -_hidden int libxl__domain_suspend_common_callback(void *data); +_hidden intlibxl__domain_suspend_callback_common (libxl__domain_suspend_state *);
_hiddenvoidlibxl__domain_suspend_common_switch_qemu_logdirty
(int domid, unsigned int enable, void
*data);
_hiddenint libxl__toolstack_save(uint32_t domid,uint8_t**buf,tools/libxl/ renamed from tools/libxl/libxl_save_msgs_gen.pl +0
-0libxl_save_msgs_gen.pl
index ee126c7..3c6bd57 100755tools/libxl/libxl_save_msgs_gen.pl +1
-1@@ -23,7 +23,7 @@ our @msgs = ( STRING doing_what), 'unsigned long', 'done', 'unsigned long', 'total'] ], - [ 3, 'scxW', "suspend", [] ], + [ 3, 'scxA', "suspend", [] ], [ 4, 'scxW', "postcopy", [] ], [ 5, 'scxA', "checkpoint", [] ],
Quoted 5 lines
[ 6, 'scxA', "
switch_qemu_logdirty", [qw(int domid
--
1.7.10.4
The patch seems harmless enough. How do you want to go about this? Do you want to post/commit this patch ? Because I have to modify my patches accordingly. Or should I post this along with my patch series, avoiding the need
to wait on you before I post mine ?
shriram
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel
Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks [and 1 more messages] [and 1 more messages]
Quoted 4 lines · Shriram Rajagopalan
The patch seems harmless enough. How do you want to go about this? Do you want to post/commit this patch ? Because I have to modify my patches accordingly. Or should I post this along with my patch series, avoiding the need to wait on you before I post mine ?
I recommend doing both, i.e. include it at the head of your series but also be prepared for it to go in independently, that way nobody is blocked.
Ian.
Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks [and 1 more messages] [and 1 more messages]
In reply to Shriram RajagopalanShriram Rajagopalan writes ("Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks [and 1 more messages] [and 1 more messages]"):
Quoted 1 line · Ian Jackson
...
Quoted 1 line
The patch seems harmless enough. How do you want to go about this?
If you think this is the right approach, you should start on it right away based on my patch. The reason I posted it quickly was because I wanted to unblock you.
Quoted 3 lines
Do you want to post/commit this patch ? Because I have to modify my patches accordingly. Or should I post this along with my patch series, avoiding the need to wait on you before I post mine ?
You can certainly take it into the start of your patch series, yes. Like what Roger did with the nested ao patch.
Currently there isn't any other reason to make the change in this patch, so I don't think it should be committed right away. But if for some reason it does get committed to staging, you or we can just drop it from the start of your series.
Regards,
Ian.
Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks [and 1 more messages] [and 1 more messages]
Quoted 13 lines · Ian Jackson
Shriram Rajagopalan writes ("Re: [PATCH 4 of 5 V3]
tools/libxl: Control network buffering in remus callbacks [and 1 more messages] [and 1 more messages]"):Quoted 1 line
On Mon, Nov 4, 2013 at 10:40 AM, Ian Jackson <[email protected]>
wrote:
...Quoted 1 line
The patch seems harmless enough. How do you want to go about this?
If you think this is the right approach, you should start on it right away based on my patch. The reason I posted it quickly was because I wanted to unblock you.
The nested-ao patch makes sense for Remus, even without fixing this timeout issue.
I can modify my stuff accordingly. Probably create a nested-ao per iteration and drop
it at the start of the next iteration.
However, the timeout part is not convincing enough. For example, libxl__domain_suspend_common_callback [the version before your patch] has two 6 second wait loops in the worst case..
LOG(DEBUG, "issuing %s suspend request via XenBus control node",
dss->hvm ? "PVHVM" : "PV");
libxl__domain_pvcontrol_write(gc, XBT_NULL, domid, "suspend");
LOG(DEBUG, "wait for the guest to acknowledge suspend request");
watchdog = 60;
while (!strcmp(state, "suspend") && watchdog > 0) {
usleep(100000);
state = libxl__domain_pvcontrol_read(gc, XBT_NULL, domid);
if (!state) state = "";
watchdog--;
}and then once again
LOG(DEBUG, "wait for the guest to suspend");
watchdog = 60;
while (watchdog > 0) {
xc_domaininfo_t info;
usleep(100000);
ret = xc_domain_getinfolist(CTX->xch, domid, 1, &info);Now I know where the 200ms overhead per checkpoint comes from.
Shouldn't this also be made into an event loop? Irrespective of whether it is invoked in
Remus' context or normal suspend/resume/save/restore/migrate context.
And if this remains, the Remus checkpoint interval is much lower compared to this.
Typically between 25-100ms.
Quoted 4 lines
Quoted 3 lines
Do you want to post/commit this patch ? Because I have to modify my patches accordingly. Or should I post this along with my patch series, avoiding the need to wait on you before I post mine ?
Quoted 9 lines
You can certainly take it into the start of your patch series, yes. Like what Roger did with the nested ao patch.
Currently there isn't any other reason to make the change in this patch, so I don't think it should be committed right away. But if for some reason it does get committed to staging, you or we can just drop it from the start of your series.
The only reason it might get committed to staging without other remus patches
would be to fix the issue I cited above.
cheers
shriram
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel
Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks [and 1 more messages] [and 1 more messages]
Shriram Rajagopalan writes ("Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks [and 1 more messages] [and 1 more messages]"):
Quoted 4 lines
The nested-ao patch makes sense for Remus, even without fixing this timeout issue. I can modify my stuff accordingly. Probably create a nested-ao per iteration and drop it at the start of the next iteration.
Right. Great.
Quoted 3 lines
However, the timeout part is not convincing enough. For example,
libxl__domain_suspend_common_callback[the version before your patch] has two 6 second wait loops in the worst case..
...
Quoted 4 lines
LOG(DEBUG, "wait for the guest to acknowledge suspend request");watchdog = 60; while (!strcmp(state, "suspend") && watchdog > 0) { usleep(100000);
...
Quoted 1 line
and then once again
...
Quoted 1 line
usleep(100000);
Oh dear. That is very poor.
Quoted 5 lines
Now I know where the 200ms overhead per checkpoint comes from.
Shouldn't this also be made into an event loop? Irrespective of whether it is invoked in Remus' context or normal suspend/resume/save/restore/migrate context.
Yes, you are entitrely correct.
Both of these loops should be replaced with timeout/event/callback approaches.
Do you want to attempt this or would you like me to do it ?
Quoted 4 lines
Currently there isn't any other reason to make the change in this patch, so I don't think it should be committed right away. But if for some reason it does get committed to staging, you or we can just drop it from the start of your series.
...
Quoted 2 lines
The only reason it might get committed to staging without other remus patches would be to fix the issue I cited above.
Yes.
Ian.
Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks [and 1 more messages] [and 1 more messages]
Quoted 31 lines · Ian Jackson
Shriram Rajagopalan writes ("Re: [PATCH 4 of 5 V3]
tools/libxl: Control network buffering in remus callbacks [and 1 more messages] [and 1 more messages]"):Quoted 4 lines
The nested-ao patch makes sense for Remus, even without fixing this timeout issue. I can modify my stuff accordingly. Probably create a nested-ao per iteration and drop it at the start of the next iteration.
Right. Great.
Quoted 3 lines
However, the timeout part is not convincing enough. For example,
libxl__domain_suspend_common_callback[the version before your patch] has two 6 second wait loops in the worst case.....
Quoted 4 lines
LOG(DEBUG, "wait for the guest to acknowledge suspend request");watchdog = 60; while (!strcmp(state, "suspend") && watchdog > 0) { usleep(100000);...
Quoted 1 line
and then once again
...
Quoted 1 line
usleep(100000);Oh dear. That is very poor.
Quoted 5 lines
Now I know where the 200ms overhead per checkpoint comes from.
Shouldn't this also be made into an event loop? Irrespective of whether it is invoked in Remus' context or normal suspend/resume/save/restore/migrate context.
Quoted 7 lines
Yes, you are entitrely correct.
Both of these loops should be replaced with timeout/event/callback approaches.
Do you want to attempt this or would you like me to do it ?
I can take a crack at it.
Thanks
Shriram
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel
Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks [and 1 more messages] [and 1 more messages]
Shriram Rajagopalan writes ("Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks [and 1 more messages] [and 1 more messages]"):
Quoted 5 lines
On Tue, Nov 12, 2013 at 10:38 AM, Ian Jackson <[email protected]> wrote:
Shriram Rajagopalan writes ("Re: [PATCH 4 of 5 V3]tools/libxl: Control network buffering in remus callbacks [and 1 more messages] [and 1 more messages]"):
...
Quoted 6 lines
Both of these loops should be replaced with timeout/event/callback approaches.
Do you want to attempt this or would you like me to do it ?
I can take a crack at it.
OK. Good luck. If you get stuck please do shout for help. Also note that feature freeze is quite soon!
Ian.
Re: [PATCH 4 of 5 V3] tools/libxl: Control network buffering in remus callbacks
In reply to Ian JacksonQuoted 31 lines · Ian Jackson
Shriram Rajagopalan writes ("[PATCH 4 of 5 V3]
tools/libxl: Control network buffering in remus callbacks"):Quoted 1 line
tools/libxl: Control network buffering in remus callbacks...
tools/libxl/libxl_dom.c void libxl__remus_teardown_done(libxl__e · line 1269
Quoted 12 lines
static int
libxl__remus_domain_suspend_callback(void *data){tools/libxl/libxl_dom.c +8
-2-/* REMUS TODO: Issue disk and network checkpoint reqs. */ -return libxl__domain_suspend_common_callback (data); +/* REMUS TODO: Issue disk checkpoint reqs. */ +libxl__save_helper_state *shs = data; +libxl__domain_suspend_state *dss =CONTAINER_OF (shs, *dss, shs); +libxl__remus_ctx *remus_ctx = dss->remus_ctx; +bool is_suspended; +STATE_AO_GC (dss->ao); + + is_suspended = !!libxl__domain_suspend_common_callback (data);I think this function would be less confusing if the return value variable was called "ok" or something similar. It's really just an error flag.
Also AFAICT the !! has no function here. The return value is essentially a boolean.
tools/libxl/libxl_dom.c void libxl__remus_teardown_done(libxl__e · line 1276
Quoted 6 lines
tools/libxl/libxl_dom.c +6
-0+if (!remus_ctx->netbuf_ctx)return is_suspended; + +if (is_suspended) { +if (libxl__remus_netbuf_start_new_epoch (gc, dss->domid, + remus_ctx)) +return !is_suspended;
Quoted 4 lines
This construction is rather odd. Why return !is_suspended when you know that !!is_suspended ?
If a new network buffer cannot be created (for the next epoch), then return an error. This ends up terminating the checkpointing process.
Quoted 26 lines
Review static void libxl__remus_domain_checkpoi · line 1317
Quoted 3 lines
(quoted diff) +0
-0@@ -1300,11 +1316,42 @@ static void libxl__remus_domain_checkpoi static void remus_checkpoint_dm_saved(libxl__egc *egc, libxl__domain_suspend_state *dss,int rc)
Review static void libxl__remus_domain_checkpoi · line 1303
Quoted 2 lines
(quoted diff) +0
-1{ - /* REMUS TODO: Wait for disk and memory ack, release network buffer*/
tools/libxl/libxl_dom.c static void libxl__remus_domain_checkpoi
Quoted 6 lines
tools/libxl/libxl_dom.c +2
-4-/* REMUS TODO: make this asynchronous */ -assert (!rc);/* REMUS TODO handle this error properly */ -usleep (dss->remus_ctx->interval *1000 ); -libxl__xc_domain_saverestore_async_callback_done (egc, &dss->shs,1 ); +/* +* REMUS TODO: Wait for disk and explicit memory ack (through restore
tools/libxl/libxl_dom.c static void libxl__remus_domain_checkpoi · line 1325
Quoted 5 lines
tools/libxl/libxl_dom.c +5
-0+* callback from remote) before releasing network buffer. +*/ +libxl__remus_ctx *remus_ctx = dss->remus_ctx; +struct timespec epoch; +int do_next_iter =0 ;Again, this variable is probably best called "ok".
tools/libxl/libxl_dom.c static void libxl__remus_domain_checkpoi · line 1345
Quoted 3 lines
tools/libxl/libxl_dom.c +3
-0+ epoch.tv_sec = remus_ctx->interval /1000 ;/* interval is in ms */ + epoch.tv_nsec = remus_ctx->interval *1000L *1000L ; +nanosleep (&epoch,0 );
Quoted 7 lines
This is no good, I'm afraid. You should be using a libxl event loop timer. (Also why are you changing your existing usleep to this new nanosleep which seems functionally very similar?)
Ian.
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel
PATCH v3 1/1 tools/xl: Remus - Network buffering cmdline switch
In reply to Shriram Rajagopalan# HG changeset patch
# User Shriram Rajagopalan <[email protected]> # Date 1382295556 25200
# Node ID f4eea1e0ac3ed92ed17bbd74b0705743d20b302d # Parent a8deb9499e9dcce9869025fa1c02cf2e0d58612a tools/xl: Remus - Network buffering cmdline switch
Command line switch to 'xl remus' command, to enable network buffering. Pass on this flag to libxl so that it can act accordingly. Also update man pages to reflect the addition of a new option to 'xl remus' command.
docs/man/xl.conf.pod.5 +6 -0
@@ -105,6 +105,12 @@ Configures the default gateway device to Default: C<None> +=item B<remus.default.netbufscript="PATH"> + +Configures the default script used by Remus to setup network buffering. + +Default: C</etc/xen/scripts/remus-netbuf-setup> + =item B<output_format="json|sxp"> Configures the default output format used by xl when printing "machine
docs/man/xl.pod.1 +8 -2
@@ -398,8 +398,7 @@ Print huge (!) amount of debug during th Enable Remus HA for domain. By default B<xl> relies on ssh as a transport mechanism between the two hosts. -N.B: Remus support in xl is still in experimental (proof-of-concept) phase. - There is no support for network or disk buffering at the moment. +N.B: There is no support for disk buffering at the moment. B<OPTIONS>
@@ -418,6 +417,13 @@ Generally useful for debugging. Disable memory checkpoint compression. +=item B<-n> + +Enable network output buffering. The default script used to configure +network buffering is /etc/xen/scripts/remus-netbuf-setup. If you wish to +use a custom script, set the global variable "remus.default.netbufscript" +in /etc/xen/xl.conf to point to your script. + =item B<-s> I<sshcommand> Use <sshcommand> instead of ssh. String will be passed to sh.
tools/libxl/xl.c +4 -0
@@ -46,6 +46,7 @@ char *default_vifscript = NULL;char *default_bridge =NULL ;char *default_gatewaydev =NULL ;char *default_vifbackend =NULL ; +char *default_remus_netbufscript =NULL ;enum output_format default_output_format =OUTPUT_FORMAT_JSON ;int claim_mode =1 ;
@@ -177,6 +178,9 @@ static void parse_global_config(const chif (!xlu_cfg_get_long (config,"claim_mode" , &l,0 )) claim_mode = l; +xlu_cfg_replace_string (config,"remus.default.netbufscript" , + &default_remus_netbufscript,0 ); +xlu_cfg_destroy (config); }
tools/libxl/xl.h +1 -0
@@ -152,6 +152,7 @@ extern char *default_vifscript;extern char *default_bridge;extern char *default_gatewaydev;extern char *default_vifbackend; +extern char *default_remus_netbufscript;extern char *blkdev_start;enum output_format {
tools/libxl/xl_cmdimpl.c +16 -7
@@ -7111,8 +7111,9 @@ int main_remus(int argc, char **argv) r_info.interval =200 ; r_info.blackhole =0 ; r_info.compression =1 ; - - SWITCH_FOREACH_OPT(opt, "bui:s:e", NULL, "remus", 2) { + r_info.netbuf =0 ; + + SWITCH_FOREACH_OPT(opt, "buni:s:N:e", NULL, "remus", 2) {case 'i' : r_info.interval =atoi (optarg);break ;
@@ -7122,6 +7123,12 @@ int main_remus(int argc, char **argv)case 'u' : r_info.compression =0 ;break ; +case 'n' : + r_info.netbuf =1 ; +break ; +case 'N' : + r_info.netbufscript = optarg; +break ;case 's' : ssh_command = optarg;break ;
@@ -7133,6 +7140,11 @@ int main_remus(int argc, char **argv) domid =find_domain (argv[optind]); host = argv[optind +1 ]; +if (r_info.netbuf && !r_info.netbufscript) { +if (default_remus_netbufscript) + r_info.netbufscript = default_remus_netbufscript; + } +if (r_info.blackhole) { send_fd =open ("/dev/null" ,O_RDWR ,0644 );if (send_fd <0 ) {
@@ -7170,13 +7182,10 @@ int main_remus(int argc, char **argv)/* Point of no return */ rc =libxl_domain_remus_start (ctx, &r_info, domid, send_fd, recv_fd,0 ); -/* If we are here, it means backup has failed/domain suspend failed. -* Try to resume the domain and exit gracefully. +/* If we are here, it means remus setup/domain suspend/backup has +* failed. Try to resume the domain and exit gracefully. * TODO: Split-Brain check. */ -fprintf (stderr,"remus sender: libxl_domain_suspend failed" -" (rc=%d)\n" , rc); -if (rc ==ERROR_GUEST_TIMEDOUT )fprintf (stderr,"Failed to suspend domain at primary.\n" );else {
tools/libxl/xl_cmdtable.c +3 -0
@@ -481,6 +481,9 @@ struct cmd_spec cmd_table[] = {"-i MS Checkpoint domain memory every MS milliseconds (def. 200ms).\n" "-b Replicate memory checkpoints to /dev/null (blackhole)\n" "-u Disable memory checkpoint compression.\n" +"-n Enable network output buffering.\n" +"-N <netbufscript> Use netbufscript to setup network buffering instead of the\n" +" instead of the default (/etc/xen/scripts/remus-netbuf-setup).\n" "-s <sshcommand> Use <sshcommand> instead of ssh. String will be passed\n" " to sh. If empty, run <host> instead of \n" " ssh <host> xl migrate-receive -r [-e]\n"
Re: [PATCH 5 of 5 V3] tools/xl: Remus - Network buffering cmdline switch
Quoted 14 lines · Shriram Rajagopalan
diff -r
a8deb9499e9d-rf4eea1e0ac3edocs/man/xl.pod.1docs/man/xl.pod.1 +1
-2@@ -398,8 +398,7 @@ Print huge (!) amount of debug during th Enable Remus HA for domain. By default B<xl> relies on ssh as a transport mechanism between the two hosts. -N.B: Remus support in xl is still in experimental (proof-of-concept) phase. - There is no support for network or disk buffering at the moment. +N.B: There is no support for disk buffering at the moment. B<OPTIONS>
docs/man/xl.pod.1 Generally useful for debugging. · line 425
Quoted 9 lines
docs/man/xl.pod.1 +6
-0@@ -418,6 +417,13 @@ Generally useful for debugging. Disable memory checkpoint compression. +=item B<-n> + +Enable network output buffering. The default script used to configure +network buffering is /etc/xen/scripts/remus-netbuf-setup. If you wish to +use a custom script, set the global variable "remus.default.netbufscript" +in /etc/xen/xl.conf to point to your script.
No docs for -N ?
docs/man/xl.pod.1 int main_remus(int argc, char **argv) · line 7143
Quoted 5 lines
docs/man/xl.pod.1 +1
-0@@ -7133,6 +7140,11 @@ int main_remus(int argc, char **argv) domid = find_domain(argv[optind]); host = argv[optind + 1]; + if (r_info.netbuf && !r_info.netbufscript) {
Will the code do anything if netbuf==False but netbufscript!=NULL?
docs/man/xl.pod.1 int main_remus(int argc, char **argv) · line 7144
Quoted 1 line
docs/man/xl.pod.1 +1
-0+ if (default_remus_netbufscript)
This if is redundant isn't it, it just avoids assigning NULL to a variable which is already NULL.
Given the above two comments this whole block could be
if (!r_info.netbufscript)
r_info.netbufscript = default...
[...]
docs/man/xl.pod.1 int main_remus(int argc, char **argv) · line 7137
Quoted 2 lines
docs/man/xl.pod.1 +0
-2- fprintf(stderr, "remus sender: libxl_domain_suspend failed" - " (rc=%d)\n", rc);
Was this removed on purpose? It seemed useful looking and wasn't mentioned in the commit message.
Ian.
Re: [PATCH 5 of 5 V3] tools/xl: Remus - Network buffering cmdline switch
Quoted 47 lines · Ian Campbell
docs/man/xl.pod.1 Sun Oct 20 11:59:16 2013 -0700 Print huge (!) amount of debug during th · line 398
Quoted 5 lines · Shriram Rajagopalan
diff -r
a8deb9499e9d-rf4eea1e0ac3edocs/man/xl.pod.1docs/man/xl.pod.1 Sun Oct 20 11:59:16 2013 -0700 renamed from docs/man/xl.pod.1 Sun Oct 20 11:59:06 2013 -0700 +0
-0@@ -398,8 +398,7 @@ Print huge (!) amount of debug during th Enable Remus HA for domain. By default B<xl> relies on ssh as atransport
docs/man/xl.pod.1 Sun Oct 20 11:59:16 2013 -0700 Print huge (!) amount of debug during th · line 401
Quoted 3 lines
docs/man/xl.pod.1 Sun Oct 20 11:59:16 2013 -0700 renamed from docs/man/xl.pod.1 Sun Oct 20 11:59:06 2013 -0700 +0
-1mechanism between the two hosts. -N.B: Remus support in xl is still in experimental (proof-of-concept)phase.
Quoted 6 lines
docs/man/xl.pod.1 Sun Oct 20 11:59:16 2013 -0700 renamed from docs/man/xl.pod.1 Sun Oct 20 11:59:06 2013 -0700 +1
-1- There is no support for network or disk buffering at the moment. +N.B: There is no support for disk buffering at the moment. B<OPTIONS>docs/man/xl.pod.1 Sun Oct 20 11:59:16 2013 -0700 Generally useful for debugging. · line 424
Quoted 8 lines
docs/man/xl.pod.1 Sun Oct 20 11:59:16 2013 -0700 renamed from docs/man/xl.pod.1 Sun Oct 20 11:59:06 2013 -0700 +5
-0@@ -418,6 +417,13 @@ Generally useful for debugging. Disable memory checkpoint compression. +=item B<-n> + +Enable network output buffering. The default script used to configure +network buffering is /etc/xen/scripts/remus-netbuf-setup. If you wish to +use a custom script, set the global variable"remus.default.netbufscript"
docs/man/xl.pod.1 Sun Oct 20 11:59:16 2013 -0700 Generally useful for debugging. · line 425
Quoted 1 line
docs/man/xl.pod.1 Sun Oct 20 11:59:16 2013 -0700 renamed from docs/man/xl.pod.1 Sun Oct 20 11:59:06 2013 -0700 +1
-0+in /etc/xen/xl.conf to point to your script.No docs for -N ?
docs/man/xl.pod.1 Sun Oct 20 11:59:16 2013 -0700 int main_remus(int argc, char **argv) · line 7143
Quoted 5 lines
docs/man/xl.pod.1 Sun Oct 20 11:59:16 2013 -0700 renamed from docs/man/xl.pod.1 Sun Oct 20 11:59:06 2013 -0700 +1
-0@@ -7133,6 +7140,11 @@ int main_remus(int argc, char **argv) domid = find_domain(argv[optind]); host = argv[optind + 1]; + if (r_info.netbuf && !r_info.netbufscript) {Will the code do anything if netbuf==False but netbufscript!=NULL?
docs/man/xl.pod.1 Sun Oct 20 11:59:16 2013 -0700 int main_remus(int argc, char **argv) · line 7144
Quoted 1 line
docs/man/xl.pod.1 Sun Oct 20 11:59:16 2013 -0700 renamed from docs/man/xl.pod.1 Sun Oct 20 11:59:06 2013 -0700 +1
-0+ if (default_remus_netbufscript)This if is redundant isn't it, it just avoids assigning NULL to a variable which is already NULL.
Given the above two comments this whole block could be
if (!r_info.netbufscript)
r_info.netbufscript = default...
[...]
Quoted 7 lines
docs/man/xl.pod.1 Sun Oct 20 11:59:16 2013 -0700 int main_remus(int argc, char **argv) · line 7137
Quoted 2 lines
docs/man/xl.pod.1 Sun Oct 20 11:59:16 2013 -0700 renamed from docs/man/xl.pod.1 Sun Oct 20 11:59:06 2013 -0700 +0
-2- fprintf(stderr, "remus sender: libxl_domain_suspend failed" - " (rc=%d)\n", rc);Was this removed on purpose? It seemed useful looking and wasn't mentioned in the commit message.
Yes, because the error message was not representative. That code block currently looks like this (after removing the fprintf), which is bit more informative.
/* If we are here, it means remus setup/domain suspend/backup has
* failed. Try to resume the domain and exit gracefully.
* TODO: Split-Brain check.
*/
if (rc == ERROR_GUEST_TIMEDOUT)
fprintf(stderr, "Failed to suspend domain at primary.\n");
else {
fprintf(stderr, "Remus: Backup failed? resuming domain at
primary.\n");
libxl_domain_resume(ctx, domid, 1, 0);
}Quoted 5 lines
Ian.
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel
Re: [PATCH 5 of 5 V3] tools/xl: Remus - Network buffering cmdline switch
Quoted 4 lines · Shriram Rajagopalan
Yes, because the error message was not representative. That code block currently looks like this (after removing the fprintf), which is bit more informative.
Makes sense, thanks.
Quoted 31 lines
/* If we are here, it means remus setup/domain suspend/backup has
* failed. Try to resume the domain and exit gracefully. * TODO: Split-Brain check. */ if (rc == ERROR_GUEST_TIMEDOUT) fprintf(stderr, "Failed to suspend domain at primary.\n"); else { fprintf(stderr, "Remus: Backup failed? resuming domain at primary.\n"); libxl_domain_resume(ctx, domid, 1, 0); } Ian.
Quoted 67 lines · Shriram Rajagopalan
This patch series adds support for network buffering in the Remus codebase in libxl.
Changes in V3:
[1/5] Fix redundant checks in configure scripts
(based on Ian Campbell's suggestions)[2/5] Introduce locking in the script, during IFB setup.
Add xenstore paths used by netbuf scripts to xenstore-paths.markdown[3/5] Hotplug scripts setup/teardown invocations are now asynchronous
following IanJ's feedback. However, the invocations are still sequential.[5/5] Allow per-domain specification of netbuffer scripts in xl remus
commmand.And minor nits throughout the series based on feedback from the last version
Changes in V2:
[1/5] Configure script will automatically enable/disable network
buffer support depending on the availability of the appropriate libnl3 version. [If libnl3 is unavailable, a warning message will be printed to let the user know that the feature has been disabled.]use macros from pkg.m4 instead of pkg-config commands removed redundant checks for libnl3 libraries.
[3,4/5] - Minor nits.
Version 1:
[1/5] Changes to autoconf scripts to check for libnl3. Add linker flags
to libxlMakefile.[2/5] External script to setup/teardown network buffering using libnl3's
CLI. This script will be invoked by libxl before starting Remus. The script's main job is to bring up an IFB device with plug qdisc attached to it. It then re-routes egress traffic from the guest's vif to the IFB device.[3/5] Libxl code to invoke the external setup script, followed by netlink
related setup to obtain a handle on the output buffers attached to each vif.[4/5] Libxl interaction with network buffer module in the kernel via
libnl3 API.[5/5] xl cmdline switch to explicitly enable network buffering when
starting remus.Few things to note:
a) Based on previous email discussions, the setup/teardown task has been moved to a hotplug style shell script which can be customized as desired, instead of implementing it as C code inside libxl.
b) Libnl3 is not available on NetBSD. Nor is it available on CentOS
(Linux). So I have made network buffering support an optional feature so that it can be disabled if desired.c) NetBSD does not have libnl3. So I have put the setup script under
tools/hotplug/Linuxfolder.
Quoted 8 lines
thanks
shriram_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel
ping!
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel