Kemari v1.0 released

Hi,

I would like to announce the first release of Kemari, an open-source virtual machine synchronization mechanism for fault tolerance. It offers a feasible approach to fault tolerance that does not require the use of specific hardware or modification of applications/OS. Kemari aims to keep VMs transparently running in times of hardware failures. It transfers the state of the primary VM to the secondary VM when the primary VM is about to send an event to devices such as storage and networks. The source code and further information are available at the following site.

http://www.osrg.net/kemari/

Older slides and video are also available at,

http://www.xen.org/community/projects.html

In addition to Linux with PV Drivers, we tested Windows with James Harpers's GPL PV. The video
(http://www.osrg.net/kemari/download/kemari_windows.mov) demonstrates that Windows continued transparently when power failure occurred to the primary server.

Kemari is now listed on Xen Product Roadmap,
(http://www.xen.org/download/roadmap.html)
and we would like to ask for your reviews and comments. Briefly, Kemari in VMM taps event channel, pauses the guest (not suspend), prepares for transfer, and Kemari in userland transfers the guest. On failover, Kemari on the secondary restores the guest, and the backend drivers in dom0 set up the backend rings from the state of the shared rings in the guest.

Could you please review the code? Any comments are appreciated.

We're also planning to merge with Remus from UBC to propose a better solution for Xen 3.4 together.

Thanks,

Yoshi

TAMURA, Yoshiaki

NTT Cyber Space Labs
OSS Computing Project
Kernel Group
E-mail: [email protected]
TEL: +81-46-859-2771
FAX: +81-46-855-1152
Address: 1-1 Hikarinooka, Yokosuka
	 Kanagawa 239-0847 JAPAN

Hi,

We're currently preparing to submit patches of Kemari.
Since I didn't give presentation at the summit last week, I would like to share the design document of Kemari, which is attached to this message. Any comments are greatly appreciated. We'll submit patches based on this design this week.

You can also get the undivided patches from the following links.
http://www.osrg.net/kemari/download/kemari-v1-xen-20081120.patch
http://www.osrg.net/kemari/download/kemari-v1-linux-20081120.patch

If you need further information on the design or how it works, please visit
http://www.osrg.net/kemari/

Thanks,

Yoshi

TAMURA, Yoshiaki

NTT Cyber Space Labs
OSS Computing Project
Kernel Group
E-mail: [email protected]
TEL: +81-46-859-2771
FAX: +81-46-855-1152
Address: 1-1 Hikarinooka, Yokosuka
	 Kanagawa 239-0847 JAPAN

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

The two obvious things I note are the duplicated save/restore functions, which will simply bitrot, and the additions to evtchn.c. In particular the hack to relax a validity check in evtchn_bind_interdomain() looks pretty suspicious. Apart from that there are various spurious white-space and debug-tracing changes which I assume you would clean up anyway. Those are just the things that jump out from a quick read through.

-- Keir

Keir,

Thanks for your quick review.
I'm glad my messages have finally reached to the list.

Although the issues you pointed out here aren't fixed yet, let me post the divided patches which are updated a little. We'll keep up cleaning and updating the code.

Yoshi

I wonder how Kemari differs from Remus?

regards
john

I believe Kemari and Remus shares a lot in common.
Stephen and Ian have described both Kemari and Remus on the roadmap document, and that might help for quick review.

http://blog.xen.org/index.php/2008/12/08/xen-roadmap-document-2/

In short, Kemari transfers the guest VM on event-driven, while Remus transfers on specific interval.

Thanks,

Yoshi

Thanks. Do you have performance numbers? It sounds like your approach is intended to give lower I/O response latencies, but Remus will give better throughput with high I/O load as it doesn't sync on every single outgoing I/O.

If both projects are indeed headed for the tree, people are going to need some ideas of which to use...

regards
john

Thanks for your interest.
Although the numbers are old, you can get an extended abstract and slides which I used at Xen Summit from http://www.osrg.net/kemari

The patchset I sent today seems to perform a lot better, but I don't have papers yet.

Regarding the throughput, possible idea is to buffer some events to reduce the frequency, which would introduce latencies as a trade off. Although we haven't implemented this feature yet, it may be interesting how the numbers change.

Well, I sent the following message when I open-sourced Kemari in November.

http://lists.xensource.com/archives/html/xen-devel/2008-11/msg00682.html

Thanks,

Yoshi

Thanks, that has some more info. Seems like there's a good opportunity for someone to do some more detailed comparisons.

"We're also planning to merge with Remus from UBC to propose a better solution for Xen 3.4 together."

That sounds very interesting.

regards
john

Yes, Yoshi and I had a brief discussion a few months ago about this. Kemari has been released, and I plan to release Remus as soon as possible. Unfortunately, I've had a very busy start to 2009 (related to the Tralfamadore project Andy presented at the last Xen summit) and haven't had the cycles to forward port the code to unstable and test it. I intend to do that ASAP, and no later than the end of the month.

The good news is that I believe all of the Xen and Linux support necessary for Remus has already been merged, so the code I'll be releasing is just tools patches (I should double check whether that also holds for HVM Remus, but I believe it does). Once the code is released we can start looking at how to share code between Kemari and Remus. I think there is a lot of common ground between the two approaches -- the heart of Remus is fine-grained checkpoints, and I think that's true for Kemari as well, though I haven't yet looked closely at it.

By the way, there's another project going on here at UBC that complements both Kemari and Remus very well. Patrick Colp is working on copy-on-write VM checkpoints, which could be a huge win for checkpoint-based availability. He also presented some of his work at the last Xen summit:

http://www.getxen.org/xensummit/xensummit_winter_2009.html

Hi Keir,

Since I have implemented the save/restore functions, will you let me ask some?

I could not understand the meaning of "the duplicated save/restore functions". Do you mean there are some duplicated code inside xc_dom_kemari_save.c and xc_dom_kemari_restore.c? Or, do you mean there are some duplicated code, e.g. test_bit and clear_bit and so on, between xc_dom_kemari_save and xc_domain_save, and between xc_dom_kemari_restore and xc_domain_restore?

I will clean up those spurious white-space.

Thank you in advance,

Yoshisato YANAGISAWA, Ph.D.
Open Source Software Computing Project
NTT Cyber Space Laboratories
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
1-1 Hikarinooka, Yokosuka-Shi,
Kanagawa 239-0847, Japan

xc_domain_{save,restore} and xc_dom_kemari_{save,restore} look to have quite a bit in common.

It will be interesting to see Remus posted and compare patch sizes and approaches.

-- Keir

Yes they do. We kept the original xc_domain_{save,restore} to make kemari specific part less intrusive. If you suggest to integrate them, we would certainly do so. Would that be appropriate?

Yoshi

Yes, we prefer a merged save/restore codebase.

-- Keir

Since Kemari requires transaction-like feature, xc_dom_kemari_save should wait acknowledgment from xc_dom_kemari_restore. Xc_dom_kemari_restore should also buffer data until complete snapshot received from xc_dom_kemari_save. Because of this reason, there might be large if-else statements if those code are merged.

Isn't it make xc_domain_save and xc_domain_restore too complexed?

Yoshisato YANAGISAWA, Ph.D.
Open Source Software Computing Project
NTT Cyber Space Laboratories
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
1-1 Hikarinooka, Yokosuka-Shi,
Kanagawa 239-0847, Japan

Well, you can leave it for now pending further investigation. I also want to see what the Remus approach is for this.

-- Keir

The main change to xc_domain_save is to split out the code that does a round into its own function so that it can be called externally. I'll have the patches together soon.

Hi,

We're currently preparing to submit patches of Kemari.
Since I didn't give presentation at the summit last week, I would like to share the design document of Kemari, which is attached to this message. Any comments are greatly appreciated. We'll submit patches based on this design this week.

You can also get the undivided patches from the following links.
http://www.osrg.net/kemari/download/kemari-v1-xen-20081120.patch
http://www.osrg.net/kemari/download/kemari-v1-linux-20081120.patch

If you need further information on the design or how it works, please visit
http://www.osrg.net/kemari/

Thanks,

Yoshi

TAMURA, Yoshiaki

NTT Cyber Space Labs
OSS Computing Project
Kernel Group
E-mail: [email protected]
TEL: +81-46-859-2771
FAX: +81-46-855-1152
Address: 1-1 Hikarinooka, Yokosuka
	 Kanagawa 239-0847 JAPAN

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel