Remus: control tool
PATCH 1/1 Remus: control tool
The following patch series integrates the Remus control layer into Xen. It provides a single user-visible script ("remus") to activate Remus on a guest virtual machine, and the libraries required by that script.
Network buffering requires the linux IMQ (http://linuximq.net) patch to be applied to dom0. I'll mail the upstream version that applies to the linux-2.6.18-xen.hg tree separately.
PATCH 1/1 Remus: add python control extensions
# HG changeset patch
# User Brendan Cully <[email protected]>
# Date 1258073720 28800
# Node ID 213fb814acf431d2a382e8f9c09b4cea106c0958 # Parent accded2f185f4178f875b170a5c01544648a68d2
Remus: add python control extensions
tools/python/setup.py +22 -3
@@ -67,10 +67,28 @@ libraries = libraries, sources = [ "ptsname/ptsname.c" ]) +checkpoint = Extension("checkpoint", + extra_compile_args = extra_compile_args, + include_dirs = include_dirs, + library_dirs = library_dirs, + libraries = libraries + [ "rt" ], + sources = [ "xen/lowlevel/checkpoint/checkpoint.c", + "xen/lowlevel/checkpoint/libcheckpoint.c"]) + +netlink = Extension("netlink", + extra_compile_args = extra_compile_args, + include_dirs = include_dirs, + library_dirs = library_dirs, + libraries = libraries, + sources = [ "xen/lowlevel/netlink/netlink.c", + "xen/lowlevel/netlink/libnetlink.c"]) + modules = [ xc, xs, ptsname, acm, flask ] -if os.uname()[0] == 'SunOS': - modules.append(scf) - modules.append(process) +plat = os.uname()[0] +if plat == 'SunOS': + modules.extend([ scf, process ]) +if plat == 'Linux': + modules.extend([ checkpoint, netlink ]) setup(name = 'xen', version = '3.0',
@@ -89,6 +107,7 @@ 'xen.web', 'xen.sv', 'xen.xsview', + 'xen.remus', 'xen.xend.tests', 'xen.xend.server.tests',
tools/python/xen/lowlevel/checkpoint/checkpoint.c new file +363 -0
363 lines not on this page · raw message
tools/python/xen/lowlevel/checkpoint/checkpoint.h new file +59 -0
59 lines not on this page · raw message
tools/python/xen/lowlevel/checkpoint/libcheckpoint.c new file +782 -0
782 lines not on this page · raw message
tools/python/xen/lowlevel/netlink/libnetlink.c new file +585 -0
585 lines not on this page · raw message
tools/python/xen/lowlevel/netlink/libnetlink.h new file +58 -0
58 lines not on this page · raw message
tools/python/xen/lowlevel/netlink/netlink.c new file +211 -0
211 lines not on this page · raw message
tools/python/xen/remus/__init__.py new file +0 -0
tools/python/xen/remus/blkdev.py new file +31 -0
@@ -0,0 +1,31 @@ +handlers = [] + +class BlkDevException(Exception): pass + +class BlkDev(object): + "Object representing a VM block device" + def __init__(self, **props): + self.uname = '' + if 'dev' not in props: + raise BlkDevException('no device') + #if 'uname' not in props: + #raise BlkDevException('no uname') + if 'mode' not in props: + raise BlkDevException('no mode') + self.__dict__.update(props) + self.dev = props['dev'].rstrip(':disk') + + def __str__(self): + return '%s,%s,%s' % (self.uname, self.dev, self.mode) + +def register(handler): + "register a block device class with parser" + if handler not in handlers: + handlers.insert(0, handler) + +def parse(props): + "turn a vm device dictionary into a blkdev object" + for handler in handlers: + if handler.handles(**props): + return handler(**props) + return BlkDev(**props)
tools/python/xen/remus/image.py new file +227 -0
227 lines not on this page · raw message
tools/python/xen/remus/netlink.py new file +314 -0
314 lines not on this page · raw message
tools/python/xen/remus/profile.py new file +56 -0
56 lines not on this page · raw message
tools/python/xen/remus/qdisc.py new file +178 -0
178 lines not on this page · raw message
tools/python/xen/remus/save.py new file +172 -0
98 lines not on this page · raw message
tools/python/xen/remus/tapdisk.py new file +4 -0
Lines of this file are not shown. View the raw message.
tools/python/xen/remus/util.py new file +31 -0
Lines of this file are not shown. View the raw message.
tools/python/xen/remus/vdi.py new file +121 -0
121 lines not on this page · raw message
tools/python/xen/remus/vif.py new file +14 -0
Lines of this file are not shown. View the raw message.
tools/python/xen/remus/vm.py new file +156 -0
156 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Part of the diff is not shown here. View the raw message.
PATCH 1/1 Remus: add control script to activate remus on a VM
In reply to Brendan Cully# HG changeset patch
# User Brendan Cully <[email protected]>
# Date 1258074147 28800
# Node ID 4e36da19dc8f433910be8adabd8a3e4e5cead5d6 # Parent 213fb814acf431d2a382e8f9c09b4cea106c0958
Remus: add control script to activate remus on a VM
tools/Makefile +1 -0
@@ -33,6 +33,7 @@ SUBDIRS-$(CONFIG_IOEMU) += ioemu-dir SUBDIRS-y += xenpmd SUBDIRS-y += libxl +SUBDIRS-y += remus # These don't cross-compile ifeq ($(XEN_COMPILE_ARCH),$(XEN_TARGET_ARCH))
tools/remus/Makefile new file +20 -0
@@ -0,0 +1,20 @@ +XEN_ROOT=../.. +include $(XEN_ROOT)/tools/Rules.mk + +SCRIPTS = remus + +.PHONY: all +all: build + +.PHONY: build +build: + echo "Nothing to do" + +.PHONY: install +install: + $(INSTALL_DIR) $(DESTDIR)$(BINDIR) + $(INSTALL_PYTHON_PROG) $(SCRIPTS) $(DESTDIR)$(BINDIR) + +.PHONY: clean +clean: + echo "Nothing to do"
tools/remus/README new file +4 -0
@@ -0,0 +1,4 @@ +Remus provides fault tolerance for virtual machines by sending continuous +checkpoints to a backup, which will activate if the target VM fails. + +See the website at http://nss.cs.ubc.ca/remus/ for details.
tools/remus/remus new file +362 -0
362 lines not on this page · raw message
PATCH 1/1 Remus: support for network buffering
In reply to Brendan Cully# HG changeset patch
# User Brendan Cully <[email protected]>
# Date 1258074170 28800
# Node ID 6caed9eff54375d6fb561dab0ea1213e37e98339 # Parent 4e36da19dc8f433910be8adabd8a3e4e5cead5d6
Remus: support for network buffering
This currently relies on the third-party IMQ patch (linuximq.net) being present in dom0. The plan is to replace this with a direct hook into netback eventually.
This patch includes a pared-down and patched copy of ebtables to install IMQ on a VIF.
.hgignore +2 -0
@@ -205,6 +205,8 @@ ^tools/pygrub/build/.*$ ^tools/python/build/.*$ ^tools/python/xen/util/path\.py$ +^tools/remus/imqebt/imqebt$ +^tools/remus/kmod/.*(\.cmd|\.mod|\.ko|\.mod\.c|\.symvers|\.xen)$ ^tools/security/secpol_tool$ ^tools/security/xen/.*$ ^tools/security/xensec_tool$
tools/remus/Makefile +5 -8
@@ -1,20 +1,17 @@ XEN_ROOT=../.. include $(XEN_ROOT)/tools/Rules.mk +SUBDIRS-y := imqebt kmod + SCRIPTS = remus .PHONY: all -all: build - -.PHONY: build -build: - echo "Nothing to do" +all: subdirs-all .PHONY: install -install: +install: subdirs-install $(INSTALL_DIR) $(DESTDIR)$(BINDIR) $(INSTALL_PYTHON_PROG) $(SCRIPTS) $(DESTDIR)$(BINDIR) .PHONY: clean -clean: - echo "Nothing to do" +clean: subdirs-clean
tools/remus/imqebt/Makefile new file +97 -0
97 lines not on this page · raw message
tools/remus/imqebt/README new file +2 -0
@@ -0,0 +1,2 @@ +This is a fork of ebtables for installing IMQ on a bridged device. +Like the original code, it is released under the GPL.
tools/remus/imqebt/communication.c new file +762 -0
762 lines not on this page · raw message
tools/remus/imqebt/ebtables-standalone.c new file +14 -0
@@ -0,0 +1,14 @@ +#include <string.h> +#include "include/ebtables_u.h" + +static struct ebt_u_replace replace; +void ebt_early_init_once (); + +int main (int argc,char *argv[]) +{ + ebt_silent =0 ; +ebt_early_init_once (); +strcpy (replace.name,"filter" ); +do_command (argc, argv,EXEC_STYLE_PRG , &replace); +return 0 ; +}
tools/remus/imqebt/ebtables.c new file +1233 -0
1,233 lines not on this page · raw message
tools/remus/imqebt/extensions/Makefile new file +29 -0
@@ -0,0 +1,29 @@ +#! /usr/bin/make + +EXT_FUNC+=standard imq +EXT_TABLES+=filter +EXT_OBJS+=$(foreach T,$(EXT_FUNC), extensions/ebt_$(T).o) +EXT_OBJS+=$(foreach T,$(EXT_TABLES), extensions/ebtable_$(T).o) +EXT_LIBS+=$(foreach T,$(EXT_FUNC), extensions/libebt_$(T).so) +EXT_LIBS+=$(foreach T,$(EXT_TABLES), extensions/libebtable_$(T).so) +EXT_LIBSI+=$(foreach T,$(EXT_FUNC), -lebt_$(T)) +EXT_LIBSI+=$(foreach T,$(EXT_TABLES), -lebtable_$(T)) + +extensions/ebt_%.so: extensions/ebt_%.o + $(CC) -shared -o $@ -lc $< -nostartfiles + +extensions/libebt_%.so: extensions/ebt_%.so + mv $< $@ + +extensions/ebtable_%.so: extensions/ebtable_%.o + $(CC) -shared -o $@ -lc $< -nostartfiles + +extensions/libebtable_%.so: extensions/ebtable_%.so + mv $< $@ + +extensions/ebt_%.o: extensions/ebt_%.c include/ebtables_u.h + $(CC) $(CFLAGS) $(CFLAGS_SH_LIB) $(PROGSPECS) -c -o $@ $< -I$(KERNEL_INCLUDES) + +extensions/ebtable_%.o: extensions/ebtable_%.c + $(CC) $(CFLAGS) $(CFLAGS_SH_LIB) $(PROGSPECS) -c -o $@ $< -I$(KERNEL_INCLUDES) +
tools/remus/imqebt/extensions/ebt_imq.c new file +84 -0
84 lines not on this page · raw message
tools/remus/imqebt/extensions/ebt_standard.c new file +90 -0
90 lines not on this page · raw message
tools/remus/imqebt/extensions/ebtable_filter.c new file +35 -0
@@ -0,0 +1,35 @@ +/* ebtable_filter +* +* Authors: +* Bart De Schuymer <[email protected]> +* +* April, 2002 +*/ + +#include <stdio.h> +#include "../include/ebtables_u.h" + +#define FILTER_VALID_HOOKS ((1 << NF_BR_LOCAL_IN) | (1 << NF_BR_FORWARD) | \ + (1 << NF_BR_LOCAL_OUT)) + +static void print_help (const char **hn) +{ +int i; + +printf ("Supported chains for the filter table:\n" ); +for (i =0 ; i <NF_BR_NUMHOOKS ; i++) +if (FILTER_VALID_HOOKS & (1 << i)) +printf ("%s " , hn[i]); +printf ("\n" ); +} + +static struct ebt_u_table table = +{ + .name ="filter" , + .help = print_help, +}; + +void _init (void ) +{ +ebt_register_table (&table); +}
tools/remus/imqebt/getethertype.c new file +162 -0
162 lines not on this page · raw message
tools/remus/imqebt/include/ebtables_u.h new file +379 -0
379 lines not on this page · raw message
tools/remus/imqebt/include/ethernetdb.h new file +58 -0
58 lines not on this page · raw message
tools/remus/imqebt/include/linux/if_ether.h new file +146 -0
21 lines not on this page · raw message
tools/remus/imqebt/include/linux/netfilter_bridge.h new file +91 -0
91 lines not on this page · raw message
tools/remus/imqebt/include/linux/netfilter_bridge/ebt_imq.h new file +8 -0
Lines of this file are not shown. View the raw message.
tools/remus/imqebt/include/linux/netfilter_bridge/ebtables.h new file +276 -0
276 lines not on this page · raw message
tools/remus/imqebt/include/linux/types.h new file +209 -0
209 lines not on this page · raw message
tools/remus/imqebt/libebtc.c new file +1280 -0
1,280 lines not on this page · raw message
tools/remus/imqebt/useful_functions.c new file +413 -0
413 lines not on this page · raw message
tools/remus/kmod/ebt_imq.c new file +45 -0
45 lines not on this page · raw message
tools/remus/kmod/sch_queue.c new file +208 -0
208 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Part of the diff is not shown here. View the raw message.
Quoted 4 lines · Brendan Cully
The following patch series integrates the Remus control layer into Xen. It provides a single user-visible script ("remus") to activate Remus on a guest virtual machine, and the libraries required by that script.
Cool. Looking forward to read them.
Quoted 4 lines
Network buffering requires the linux IMQ (http://linuximq.net) patch to be applied to dom0. I'll mail the upstream version that applies to the linux-2.6.18-xen.hg tree separately.
Is there a 2.6.31.x variant of the code? Are there plans to submit the patches to LKML?
Quoted 6 lines · Konrad Rzeszutek Wilk
Quoted 4 lines · Brendan Cully
The following patch series integrates the Remus control layer into Xen. It provides a single user-visible script ("remus") to activate Remus on a guest virtual machine, and the libraries required by that script.
Quoted 8 lines
Cool. Looking forward to read them.
Quoted 4 lines
Network buffering requires the linux IMQ (http://linuximq.net) patch to be applied to dom0. I'll mail the upstream version that applies to the linux-2.6.18-xen.hg tree separately.
Is there a 2.6.31.x variant of the code? Are there plans to submit the patches to LKML?
I haven't tried to get this working on pvops yet, but it's simple in theory:
domU ought to work now, albeit somewhat slowly since the pvops domU doesn't support suspend requests over a dedicated event channel. Cooking up a patch for this is probably not too hard.
dom0 requires nothing but the upstream IMQ patch, and blktap2 support, which I am told it currently lacks, but I'm sure it will get in the not-too-distant future.
Quoted 16 lines · Brendan Cully
Quoted 6 lines · Konrad Rzeszutek Wilk
Quoted 4 lines · Brendan Cully
The following patch series integrates the Remus control layer into Xen. It provides a single user-visible script ("remus") to activate Remus on a guest virtual machine, and the libraries required by that script.
Quoted 8 lines
Cool. Looking forward to read them.
Quoted 4 lines
Network buffering requires the linux IMQ (http://linuximq.net) patch to be applied to dom0. I'll mail the upstream version that applies to the linux-2.6.18-xen.hg tree separately.
Is there a 2.6.31.x variant of the code? Are there plans to submit the patches to LKML?
Quoted 7 lines
I haven't tried to get this working on pvops yet, but it's simple in theory:
domU ought to work now, albeit somewhat slowly since the pvops domU doesn't support suspend requests over a dedicated event channel. Cooking up a patch for this is probably not too hard.
Do you know if anyone is working on the pvops support for suspend requests over dedicated even channel?
-- Pasi
Quoted 20 lines · Pasi Kärkkäinen
Quoted 19 lines · Brendan Cully
Quoted 6 lines · Konrad Rzeszutek Wilk
Quoted 4 lines · Brendan Cully
The following patch series integrates the Remus control layer into Xen. It provides a single user-visible script ("remus") to activate Remus on a guest virtual machine, and the libraries required by that script.
Quoted 8 lines
Cool. Looking forward to read them.
Quoted 4 lines
Network buffering requires the linux IMQ (http://linuximq.net) patch to be applied to dom0. I'll mail the upstream version that applies to the linux-2.6.18-xen.hg tree separately.
Is there a 2.6.31.x variant of the code? Are there plans to submit the patches to LKML?
I haven't tried to get this working on pvops yet, but it's simple in theory:
Quoted 7 lines
Quoted 4 lines
domU ought to work now, albeit somewhat slowly since the pvops domU doesn't support suspend requests over a dedicated event channel. Cooking up a patch for this is probably not too hard.
Do you know if anyone is working on the pvops support for suspend requests over dedicated even channel?
I'll probably take a crack at this soon. From a quick look at the pvops suspend code, it seems like it may suffer from a race when multiple suspends are issued that Keir fixed in the 2.6.18 tree some time ago -- it'd be better to get that fixed before porting the event channel patch.
In the meantime, it looks like I didn't include fallback support for checkpointing with xenstore, so Remus doesn't support pvops domu at all. I should have a patch to support the slow mode out today.
Quoted 6 lines · Brendan Cully
I'll probably take a crack at this soon. From a quick look at the pvops suspend code, it seems like it may suffer from a race when multiple suspends are issued that Keir fixed in the 2.6.18 tree some time ago -- it'd be better to get that fixed before porting the event channel patch.
What's the bug? I don't see a reentrancy issue there because the suspend happens synchronously in the xenwatch thread under xenwatch_mutex. Or race for that matter.
Am I missing something?
Of course, if you start triggering suspends via event channels, we'll need to work out something else.
J
Quoted 8 lines · Jeremy Fitzhardinge
Quoted 6 lines · Brendan Cully
I'll probably take a crack at this soon. From a quick look at the pvops suspend code, it seems like it may suffer from a race when multiple suspends are issued that Keir fixed in the 2.6.18 tree some time ago -- it'd be better to get that fixed before porting the event channel patch.
Quoted 5 lines
What's the bug? I don't see a reentrancy issue there because the suspend happens synchronously in the xenwatch thread under
xenwatch_mutex. Or race for that matter.Am I missing something?
I'll need to take a closer look, but the race I'm remembering was fixed in the 2.6.18 tree here:
http://xenbits.xen.org/linux-2.6.18-xen.hg/rev/49ffe9ef67d4
Quoted 2 lines
Of course, if you start triggering suspends via event channels, we'll need to work out something else.
yep.
Quoted 9 lines · Jeremy Fitzhardinge
What's the bug? I don't see a reentrancy issue there because the suspend happens synchronously in the xenwatch thread under
xenwatch_mutex. Or race for that matter.Am I missing something?
Of course, if you start triggering suspends via event channels, we'll need to work out something else.
The issue in 2.6.18 was that, if doing back-to-back save/restores, the next event-channel notification could come in before domU was finished with previous s/r cycle, and then the notification got dropped. There are a number of ways of dealing with that of course: I implemented a little state machine; or you could probably do it with some kind of ticket-based scheme; or perhaps have the evtchn irq handler spawn a kthread which blocks on the mutex (I liked that one least as it needs to allocate resources).
-- Keir
Quoted 8 lines · Keir Fraser
The issue in 2.6.18 was that, if doing back-to-back save/restores, the next event-channel notification could come in before domU was finished with previous s/r cycle, and then the notification got dropped. There are a number of ways of dealing with that of course: I implemented a little state machine; or you could probably do it with some kind of ticket-based scheme; or perhaps have the evtchn irq handler spawn a kthread which blocks on the mutex (I liked that one least as it needs to allocate resources).
Hm, my first thought is "why does that matter?". But I guess the host/guest save protocol is fairly brittle, and if the guest doesn't respond to a particular save it will get wedged. But then, should the control stack be sending back to back save requests? Shouldn't it wait until the previous save has finished?
J
Quoted 10 lines · Jeremy Fitzhardinge
Quoted 8 lines · Keir Fraser
The issue in 2.6.18 was that, if doing back-to-back save/restores, the next event-channel notification could come in before domU was finished with previous s/r cycle, and then the notification got dropped. There are a number of ways of dealing with that of course: I implemented a little state machine; or you could probably do it with some kind of ticket-based scheme; or perhaps have the evtchn irq handler spawn a kthread which blocks on the mutex (I liked that one least as it needs to allocate resources).
Quoted 5 lines
Hm, my first thought is "why does that matter?". But I guess the host/guest save protocol is fairly brittle, and if the guest doesn't respond to a particular save it will get wedged. But then, should the control stack be sending back to back save requests? Shouldn't it wait until the previous save has finished?
What signal do you have in mind for telling the control stack that the guest has completed its resume procedure and is running normally again?
Quoted 9 lines · Jeremy Fitzhardinge
Quoted 7 lines · Keir Fraser
The issue in 2.6.18 was that, if doing back-to-back save/restores, the next event-channel notification could come in before domU was finished with previous s/r cycle, and then the notification got dropped. There are a number of ways of dealing with that of course: I implemented a little state machine; or you could probably do it with some kind of ticket-based scheme; or perhaps have the evtchn irq handler spawn a kthread which blocks on the mutex (I liked that one least as it needs to allocate resources).
Quoted 7 lines
Hm, my first thought is "why does that matter?". But I guess the host/guest save protocol is fairly brittle, and if the guest doesn't respond to a particular save it will get wedged. But then, should the control stack be sending back to back save requests? Shouldn't it wait until the previous save has finished?
From the tools p.o.v. the restore has finished when it kicks off execution
of the guest. It's not that hard to handle this in the guest; you just need to do it. ;-)
-- Keir
Quoted 4 lines · Brendan Cully
What signal do you have in mind for telling the control stack that the guest has completed its resume procedure and is running normally again?
Hm, point. I was thinking of waiting for the "I'm suspended" hypercall, but that doesn't help.
It has always worried me that the suspend protocol is very brittle. For example, there's no way for a guest to reject a suspend attempt, either because it doesn't support suspending, or it isn't convenient right now, or it tried but failed. A backchannel xenstore entry would allow the guest to indicate what stage its up to to the control stack, including holding off suspend attempts until it is ready to accept new ones. But that doesn't help much if you want to eliminate the xenstore overhead from the process...
J
Quoted 3 lines · Keir Fraser
From the tools p.o.v. the restore has finished when it kicks off execution of the guest. It's not that hard to handle this in the guest; you just need to do it. ;-)
Well, I think that just happens at the moment; the suspend is happening from the xenstore watch thread, so there's no way it will even notice a subsequent attempt until the suspend/resume cycle is done.
When we start triggering suspends from event channels, I'd suggest something along the lines of wrapping the whole thing in a big fat suspend mutex, and running the suspend from a workqueue triggered from the interrupt or watch handler. And then deal with all the edge cases.
J