External device migration support
PATCH 1/1 External device migration support
This patch enables external devices, such as for example a mounted hard drive image or a TPM, to be migrated to a remote machine. The patch hooks into the checkpointing (XendCheckpoint.py) code and performs migration in 4 different steps:
In a 1st step (step = 0 in the code) migration of all devices of a domain is 'tested', that means their driver implementations (blkif.py, netif.py, tpmif.py, usbif.py, pciif.py) are queried whether migration is possible at all. Currently all device representations respond with a 'yes' (=0), although probably a VM mounting a hard drive partition should respond with a 'no' (-1) already. This first step is a quick check to see whether devices can be migrated.
The 2nd step is to do whatever can be done before the domain is suspended. At this point migration of the device could be initiated, if at all possible.
The 3rd step is to migrate a device after the domain has been suspended, meaning that it is not scheduled anymore and the VM is 'settled'. All devices are called again and a good implementation would initiate the migration in a background process to achieve as much concurrency as possible.
The 4th step is to synchronize with the 3rd step. At this point the implementor has to make sure that anything that was initiated in step 3 has completed. Once all steps 4 have been processed, the VM will resume on the remove machine.
I have implemented hooks for migration of a TPM. These hooks for now call the os.system() call with a configurable tool but a fixed parameter set and refuse to migrate a VM attached to a virtual TPM if no tool has been provided for migration.
All other devices do not currently overload the 'migrate' method defined in the DevController.py and therefor will just let migration happen.
Please let me know what you think about this idea.
xen-unstable.hg/tools/python/xen/xend/XendCheckpoint.py +5 -1
@@ -53,7 +53,7 @@ def read_exact(fd, size, errmsg): -def save(fd, dominfo, live): +def save(fd, dominfo, live, dst): write_exact(fd, SIGNATURE, "could not write guest state file: signature") config = sxp.to_string(dominfo.sxpr())
@@ -64,6 +64,8 @@ def save(fd, dominfo, live): # thing is useful for debugging. dominfo.setName('migrating-' + domain_name) + dominfo.migrateDevices(live, dst, 1, domain_name) + try: write_exact(fd, pack("!i", len(config)), "could not write guest state file: config len")
@@ -85,7 +87,9 @@ def save(fd, dominfo, live): log.debug("Suspending %d ...", dominfo.getDomid()) dominfo.shutdown('suspend') dominfo.waitForShutdown() + dominfo.migrateDevices(live, dst, 2, domain_name) log.info("Domain %d suspended.", dominfo.getDomid()) + dominfo.migrateDevices(live, dst, 3, domain_name) tochild.write("done\n") tochild.flush() log.debug('Written done')
xen-unstable.hg/tools/python/xen/xend/XendDomainInfo.py +20 -0
@@ -1395,6 +1395,26 @@ class XendDomainInfo: if self.image: self.image.createDeviceModel() + ## public: + + def testMigrateDevices(self, live, dst): + """ Notify all device about intention of migration + @raise: XendError for a device that cannot be migrated + """ + for (n, c) in self.info['device']: + rc = self.migrateDevice(n, live, dst, 0) + if rc != 0: + raise XendError("Device of type '%s' refuses migration." % n) + + def migrateDevices(self, live, dst, step, domName=''): + """Notify the devices about migration + """ + + for (n, c) in self.info['device']: + self.migrateDevice(n, live, dst, step, domName) + + def migrateDevice(self, deviceClass, live, dst, step, domName=''): + return self.getDeviceController(deviceClass).migrate(live, dst, step, domName) def waitForDevices(self): """Wait for this domain's configured devices to connect.
xen-unstable.hg/tools/python/xen/xend/server/DevController.py +28 -0
@@ -256,6 +256,34 @@ class DevController: raise NotImplementedError() + def migrate(self, live, dst, step, domName): + """ Migration of a device. The 'live' parameter indicates + whether the device is live-migrated (live=1). 'dst' then gives + the hostname of the machine to migrate to. + This function is called for 4 steps: + If step == 0: Check whether the device is ready to be migrated + or can at all be migrated; return a '-1' if + the device is NOT ready, a '0' otherwise. If it is + not ready ( = not possible to migrate this device), + migration will not take place. + step == 1: Called immediately after step 0; migration + of the kernel has started; + step == 2: Called after the suspend has been issued + to the domain and the domain is not scheduled anymore. + Synchronize with what was started in step 1, if necessary. + Now the device should initiate its transfer to the + given target. Since there might be more than just + one device initiating a migration, this step should + put the process performing the transfer into the + background and return immediately to achieve as much + concurrency as possible. + step == 3: Synchronize with the migration of the device that + was initiated in step 2. + Make sure that the migration has finished and only + then return from the call. + """ + return 0 + def getDomid(self): """Stub to {@link XendDomainInfo.getDomid}, for use by our
xen-unstable.hg/tools/python/xen/xend/server/tpmif.py +27 -0
@@ -23,9 +23,15 @@ from xen.xend import sxp from xen.xend.XendLogging import log +from xen.xend import XendRoot from xen.xend.server.DevController import DevController +import os + + +xroot = XendRoot.instance() + class TPMifController(DevController): """TPM interface controller. Handles all TPM devices for a domain.
@@ -61,3 +67,24 @@ class TPMifController(DevController): result.append(['instance', instance]) return result + + def migrate(self, live, dst, step, domName): + """@see DevContoller.migrate""" + if live: + tool = xroot.get_vtpm_migration_tool() + if tool != '': + log.info("Request to live-migrate device to %s. step=%d.", dst, step) + + if step == 0: + """Assuming for now that everything is ok and migration + with the given tool can proceed. + """ + return 0 + else: + os.system("%s -step %d -host %s -domname %s" % (tool, step, dst, domName)) + return 0 + else: + log.debug("Virtual TPM migration tool not in configuration.") + return -1 + return 0 +
xen-unstable.hg/tools/python/xen/xend/XendDomain.py +6 -3
@@ -398,6 +398,9 @@ class XendDomain: if dominfo.getDomid() == PRIV_DOMAIN: raise XendError("Cannot migrate privileged domain %i" % domid) + """ The following call may raise a XendError exception """ + dominfo.testMigrateDevices(live, dst) + if port == 0: port = xroot.get_xend_relocation_port() try:
@@ -407,8 +410,8 @@ class XendDomain: raise XendError("can't connect: %s" % err[1]) sock.send("receive\n") - sock.recv(80) - XendCheckpoint.save(sock.fileno(), dominfo, live) + sock.recv(80) + XendCheckpoint.save(sock.fileno(), dominfo, live, dst) def domain_save(self, domid, dst):
@@ -426,7 +429,7 @@ class XendDomain: fd = os.open(dst, os.O_WRONLY | os.O_CREAT | os.O_TRUNC) try: # For now we don't support 'live checkpoint' - return XendCheckpoint.save(fd, dominfo, False) + return XendCheckpoint.save(fd, dominfo, False, dst) finally: os.close(fd) except OSError, ex:
xen-unstable.hg/tools/python/xen/xend/XendRoot.py +6 -0
@@ -86,6 +86,9 @@ class XendRoot: server (deprecated).""" xend_unix_server_default = 'no' + """Default vtpm migration tool """ + vtpm_migration_tool_default = '' + """Default path the unix-domain server listens at.""" xend_unix_path_default = '/var/lib/xend/xend-socket'
@@ -250,6 +253,9 @@ class XendRoot: else: return None + def get_vtpm_migration_tool(self): + """@return the name of the tool to handle virtual TPM migration.""" + return self.get_config_value('vtpm-migration-tool', self.vtpm_migration_tool_default) def get_enable_dump(self): return self.get_config_bool('enable-dump', 'no')
xen-unstable.hg/tools/examples/xend-config.sxp +3 -0
@@ -127,3 +127,6 @@ # Whether to enable core-dumps when domains crash. #(enable-dump no) + +# The tool used for initiating virtual TPM migration +#(vtpm-migration-tool '')
xen-unstable.hg/tools/examples/vtpm-common.sh +0 -6
@@ -261,12 +261,6 @@ function vtpm_create_instance () { if [ "$REASON" == "create" ]; then vtpm_reset $instance - elif [ "$REASON" == "resume" ]; then - vtpm_setup $instance - else - #default case for 'now' - #vtpm_reset $instance - true fi xenstore_write $XENBUS_PATH/instance $instance }
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel
Quoted 4 lines · Stefan Berger
This patch enables external devices, such as for example a mounted hard drive image or a TPM, to be migrated to a remote machine. The patch hooks into the checkpointing (
XendCheckpoint.py) code and performs migration in 4 different steps:
...
Quoted 1 line
Please let me know what you think about this idea.
I really like the idea because it seems to work for any external device that can be migrated but does not allow concurrent access from both migration source and destination.
I have not looked at the patch in detail, but I saw the specific inclusion of +# The tool used for initiating virtual TPM migration +#(vtpm-migration-tool '')
Could this migration tool specification be more generalized so that any device migration can be configured, not just vtpm devices?
Best Regards,
Michael Paesold
"Michael Paesold" <[email protected]> wrote on 03/27/2006 08:31:08 AM:
Quoted 3 lines
Quoted 1 line · Stefan Berger
This patch enables external devices, such as for example a mounted
hard
Quoted 7 lines
Quoted 3 lines
drive image or a TPM, to be migrated to a remote machine. The patch hooks into the checkpointing (
XendCheckpoint.py) code and performs migration in 4 different steps:...
Quoted 1 line
Please let me know what you think about this idea.
I really like the idea because it seems to work for any external device
that
Quoted 4 lines
can be migrated but does not allow concurrent access from both migration
source and destination.
You mean that for example concurrent access to a disk image file is not possible by both systems?
Quoted 1 line
I have not looked at the patch in detail, but I saw the specific
inclusion
Quoted 6 lines
of
xen-unstable.hg/tools/examples/xend-config.sxp +2
-0+# The tool used for initiating virtual TPM migration +#(vtpm-migration-tool '')Could this migration tool specification be more generalized so that any device migration can be configured, not just vtpm devices?
I am going to change this and make this one single 'migration-tool'. I will have that tool know by command line parameter (s.th. like '-type vtpm') what device the call for migration is about. If the external tool is a script you can switch over the '-type' parameter and call another external program that handles that particular device type, otherwise that external tool needs to know how to handle migration of all kinds of devices. I will change this and repost.
Stefan
Quoted 5 lines
Best Regards,
Michael Paesold
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel
Quoted 10 lines · Stefan Berger
"Michael Paesold" <[email protected]> wrote on 03/27/2006 08:31:08 AM:
Quoted 7 lines
Quoted 4 lines · Stefan Berger
This patch enables external devices, such as for example a mounted hard drive image or a TPM, to be migrated to a remote machine. The patch hooks into the checkpointing (
XendCheckpoint.py) code and performs migration in 4 different steps:...
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
Please let me know what you think about this idea.
I really like the idea because it seems to work for any external
device that
Quoted 3 lines
can be migrated but does not allow concurrent access from both migration source and destination.
You mean that for example concurrent access to a disk image file is not possible by both systems?
I was thinking of a DRBD-backed (see http://www.drbd.org) partition that is replicated between two hosts (RAID-over-network). In the current version, this is a failover-only soluation, so only one node of a two-node cluster can activate the partition in r/w mode.
So if you want to migrate from host A to host B, you first have to disable access on host A, and only then activate access on host B. As I understood, this should be possible with your proposed solution. (I hope I am right)
Another use case would be a root-partition with a cow device, where you would have to copy the cow to the other host. If the cow is small, this seem reasonable, but again, it must happen after the domain is paused at host A and before the domain is unpaused on host B.
Quoted 10 lines
Quoted 1 line
I have not looked at the patch in detail, but I saw the specific
inclusion
Quoted 6 lines
of
xen-unstable.hg/tools/examples/xend-config.sxp +2
-0+# The tool used for initiating virtual TPM migration +#(vtpm-migration-tool '')Could this migration tool specification be more generalized so that any device migration can be configured, not just vtpm devices?
I am going to change this and make this one single 'migration-tool'. I will have that tool know by command line parameter (s.th. like '-type vtpm') what device the call for migration is about. If the external tool is a script you can switch over the '-type' parameter and call another external program that handles that particular device type, otherwise that external tool needs to know how to handle migration of all kinds of devices. I will change this and repost.
Sounds good to me.
Best Regards,
Michael Paesold