Sheepdog: Distributed Storage System for KVM

Hello,

I am getting the following error trying to compile sheepdog on Ubuntu 9.10 ( 2.6.31-14 x64 ) :

cd shepherd; make
make[1]: Entering directory `/home/shiny/Packages/sheepdog-2009102101/shepherd'
cc -c -g -O2 -Wall -Wstrict-prototypes -I../include -D_GNU_SOURCE shepherd.c -o shepherd.o shepherd.c: In function ‘main’:
shepherd.c:300: warning: dereferencing pointer ‘hdr.55’ does break strict-aliasing rules shepherd.c:300: note: initialized from here
cc -c -g -O2 -Wall -Wstrict-prototypes -I../include -D_GNU_SOURCE treeview.c -o treeview.o cc -c -g -O2 -Wall -Wstrict-prototypes -I../include -D_GNU_SOURCE ../lib/event.c -o ../lib/event.o cc -c -g -O2 -Wall -Wstrict-prototypes -I../include -D_GNU_SOURCE ../lib/net.c -o ../lib/net.o ../lib/net.c: In function ‘write_object’:
../lib/net.c:358: warning: ‘vosts’ may be used uninitialized in this function
cc -c -g -O2 -Wall -Wstrict-prototypes -I../include -D_GNU_SOURCE ../lib/logger.c -o ../lib/logger.o cc shepherd.o treeview.o ../lib/event.o ../lib/net.o ../lib/logger.o -o shepherd -lncurses -lcrypto make[1]: Leaving directory `/home/shiny/Packages/sheepdog-2009102101/shepherd'
cd sheep; make
make[1]: Entering directory `/home/shiny/Packages/sheepdog-2009102101/sheep'
cc -c -g -O2 -Wall -Wstrict-prototypes -I../include -D_GNU_SOURCE sheep.c -o sheep.o cc -c -g -O2 -Wall -Wstrict-prototypes -I../include -D_GNU_SOURCE store.c -o store.o cc -c -g -O2 -Wall -Wstrict-prototypes -I../include -D_GNU_SOURCE net.c -o net.o
cc -c -g -O2 -Wall -Wstrict-prototypes -I../include -D_GNU_SOURCE work.c -o work.o In file included from /usr/include/asm/fcntl.h:1,
                from /usr/include/linux/fcntl.h:4,
                from /usr/include/linux/signalfd.h:13,
                from work.c:31:
/usr/include/asm-generic/fcntl.h:117: error: redefinition of ‘struct flock’ /usr/include/asm-generic/fcntl.h:140: error: redefinition of ‘struct flock64’ make[1]: *** [work.o] Error 1
make[1]: Leaving directory `/home/shiny/Packages/sheepdog-2009102101/sheep'
make: *** [all] Error 2

I have all the required libs installed. Patching and compiling qemu-kvm went flawless.

----- Original Message -----
From: "MORITA Kazutaka" <[email protected]>
To: [email protected], [email protected], [email protected]
Sent: Wednesday, October 21, 2009 8:13:47 AM
Subject: [ANNOUNCE] Sheepdog: Distributed Storage System for KVM

Hi everyone,

Sheepdog is a distributed storage system for KVM/QEMU. It provides highly available block level storage volumes to VMs like Amazon EBS. Sheepdog supports advanced volume management features such as snapshot, cloning, and thin provisioning. Sheepdog runs on several tens or hundreds of nodes, and the architecture is fully symmetric; there is no central node such as a meta-data server.

The following list describes the features of Sheepdog.

    * Linear scalability in performance and capacity
    * No single point of failure
    * Redundant architecture (data is written to multiple nodes)
    - Tolerance against network failure
    * Zero configuration (newly added machines will join the cluster automatically)
    - Autonomous load balancing
    * Snapshot
    - Online snapshot from qemu-monitor
    * Clone from a snapshot volume
    * Thin provisioning
    - Amazon EBS API support (to use from a Eucalyptus instance)

(* = current features, - = on our todo list)

More details and download links are here:

http://www.osrg.net/sheepdog/

Note that the code is still in an early stage.
There are some critical TODO items:

    - VM image deletion support
    - Support architectures other than X86_64
    - Data recoverys
    - Free space management
    - Guarantee reliability and availability under heavy load
    - Performance improvement
    - Reclaim unused blocks
    - More documentation

We hope finding people interested in working together. Enjoy!

Here are examples:

- create images

$ kvm-img create -f sheepdog "Alice's Disk" 256G
$ kvm-img create -f sheepdog "Bob's Disk" 256G

- list images

$ shepherd info -t vdi
   40000 : Alice's Disk  256 GB (allocated: 0 MB, shared: 0 MB), 2009-10-15
16:17:18, tag:        0, current
   80000 : Bob's Disk    256 GB (allocated: 0 MB, shared: 0 MB), 2009-10-15
16:29:20, tag:        0, current

- start up a virtual machine

$ kvm --drive format=sheepdog,file="Alice's Disk"

- create a snapshot

$ kvm-img snapshot -c name sheepdog:"Alice's Disk"

- clone from a snapshot

$ kvm-img create -b sheepdog:"Alice's Disk":0 -f sheepdog "Charlie's Disk"

Thanks.

--
MORITA, Kazutaka

NTT Cyber Space Labs
OSS Computing Project
Kernel Group
E-mail: [email protected]

--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of MORITA Kazutaka
Sent: Mittwoch, 21. Oktober 2009 07:14
To: [email protected]; [email protected]; linux- [email protected]
Subject: [ANNOUNCE] Sheepdog: Distributed Storage System for KVM

Hi everyone,

Sheepdog is a distributed storage system for KVM/QEMU. It provides highly available block level storage volumes to VMs like Amazon EBS. Sheepdog supports advanced volume management features such as snapshot, cloning, and thin provisioning. Sheepdog runs on several tens or hundreds
of nodes, and the architecture is fully symmetric; there is no central node such as a meta-data server.

Quite interesting. But would it be possible to use corosync for the cluster communication? The point is that we need corosync anyways for pacemaker, it is written in C (high performance) and seem to implement the feature you need?

Quoted 10 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of MORITA Kazutaka
Sent: Mittwoch, 21. Oktober 2009 07:14
To: [email protected]; [email protected]; linux- [email protected]
Subject: [ANNOUNCE] Sheepdog: Distributed Storage System for KVM

Hi everyone,

Quoted 7 lines

Sheepdog is a distributed storage system for KVM/QEMU. It provides highly available block level storage volumes to VMs like Amazon EBS. Sheepdog supports advanced volume management features such as snapshot, cloning, and thin provisioning. Sheepdog runs on several tens or hundreds
of nodes, and the architecture is fully symmetric; there is no central node such as a meta-data server.

--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

We use JGroups (Java library) for reliable multicast communication in our cluster manager daemon. We don't worry about the performance much since the cluster manager daemon is not involved in the I/O path. We might think about moving to corosync if it is more stable than JGroups.

Quoted 5 lines · MORITA Kazutaka

We use JGroups (Java library) for reliable multicast communication in our cluster manager daemon. We don't worry about the performance much since the cluster manager daemon is not involved in the I/O path. We might think about moving to corosync if it is more stable than JGroups.

I'd love to see this running on top of corosync too. Corosync is a well tested, stable cluster manager, and doesn't have the JVM dependency of jgroups so feels more suitable for building 'thin virtualisation fabrics'.

We use JGroups (Java library) for reliable multicast communication in our cluster manager daemon.

We don't worry about the performance much
since the cluster manager daemon is not involved in the I/O path. We might think about moving to corosync if it is more stable than JGroups.

Anyways, I do not know JGroups - maybe that 'reliable multicast' solves all network problems somehow - Is there any documentation about how they do it?

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of MORITA Kazutaka
Sent: Mittwoch, 21. Oktober 2009 07:14
To: [email protected]; [email protected]; linux- [email protected]
Subject: [ANNOUNCE] Sheepdog: Distributed Storage System for KVM

Hi everyone,

Sheepdog is a distributed storage system for KVM/QEMU. It provides highly available block level storage volumes to VMs like Amazon EBS. Sheepdog supports advanced volume management features such as snapshot, cloning, and thin provisioning. Sheepdog runs on several tens or hundreds
of nodes, and the architecture is fully symmetric; there is no central node such as a meta-data server.

Hi everyone,

Sheepdog is a distributed storage system for KVM/QEMU. It provides highly available block level storage volumes to VMs like Amazon EBS. Sheepdog supports advanced volume management features such as snapshot, cloning, and thin provisioning. Sheepdog runs on several tens or hundreds of nodes, and the architecture is fully symmetric; there is no central node such as a meta-data server.

Quoted 8 lines · MORITA Kazutaka

Hi everyone,

Sheepdog is a distributed storage system for KVM/QEMU. It provides highly available block level storage volumes to VMs like Amazon EBS. Sheepdog supports advanced volume management features such as snapshot, cloning, and thin provisioning. Sheepdog runs on several tens or hundreds of nodes, and the architecture is fully symmetric; there is no central node such as a meta-data server.

Very interesting! From a very brief look at the code, it looks like the sheepdog block format driver is a network client that is able to access highly available images, yes?

If so, is it reasonable to compare this to a cluster file system setup (like GFS) with images as files on this filesystem? The difference would be that clustering is implemented in userspace in sheepdog, but in the kernel for a clustering filesystem.

Quoted 10 lines · Avi Kivity

Quoted 8 lines · MORITA Kazutaka

Hi everyone,

Sheepdog is a distributed storage system for KVM/QEMU. It provides highly available block level storage volumes to VMs like Amazon EBS. Sheepdog supports advanced volume management features such as snapshot, cloning, and thin provisioning. Sheepdog runs on several tens or hundreds of nodes, and the architecture is fully symmetric; there is no central node such as a meta-data server.

Quoted 3 lines

Very interesting! From a very brief look at the code, it looks like the sheepdog block format driver is a network client that is able to access highly available images, yes?

If so, is it reasonable to compare this to a cluster file system setup (like GFS) with images as files on this filesystem? The difference would be that clustering is implemented in userspace in sheepdog, but in the kernel for a clustering filesystem.

I'm still in the process of reading the code, but that's the impression I got too. It made me think that the protocol for qemu to communicate with sheepdog could be a filesystem protocol (like 9p)

Quoted 9 lines · MORITA Kazutaka

Hi everyone,

Sheepdog is a distributed storage system for KVM/QEMU. It provides highly available block level storage volumes to VMs like Amazon EBS. Sheepdog supports advanced volume management features such as snapshot, cloning, and thin provisioning. Sheepdog runs on several tens or hundreds of nodes, and the architecture is fully symmetric; there is no central node such as a meta-data server.

Very interesting!  From a very brief look at the code, it looks like the sheepdog block format driver is a network client that is able to access highly available images, yes?

If so, is it reasonable to compare this to a cluster file system setup (like GFS) with images as files on this filesystem?  The difference would be that clustering is implemented in userspace in sheepdog, but in the kernel for a clustering filesystem.

How is load balancing implemented?  Can you move an image transparently while a guest is running?  Will an image be moved closer to its guest?

Can you stripe an image across nodes?

Do you support multiple guests accessing the same image?

What about fault tolerance - storing an image redundantly on multiple nodes?

Quoted 2 lines · Avi Kivity

How is load balancing implemented? Can you move an image transparently while a guest is running? Will an image be moved closer to its guest?

Sheepdog uses consistent hashing to decide where objects store; I/O load is balanced across the nodes. When a new node is added or the existing node is removed, the hash table changes and the data automatically and transparently are moved over nodes.

We plan to implement a mechanism to distribute the data not randomly but intelligently; we could use machine load, the locations of VMs, etc.

Quoted 1 line

Do you support multiple guests accessing the same image?

A VM image can be attached to any VMs but one VM at a time; multiple running VMs cannot access to the same VM image.

On 23.10.2009, at 12:41, MORITA Kazutaka wrote:

On Fri, Oct 23, 2009 at 12:30 AM, Avi Kivity <[email protected]> wrote:

How is load balancing implemented?  Can you move an image transparently

while a guest is running?  Will an image be moved closer to its guest?

Sheepdog uses consistent hashing to decide where objects store; I/O load is balanced across the nodes. When a new node is added or the existing node is removed, the hash table changes and the data automatically and transparently are moved over nodes.

We plan to implement a mechanism to distribute the data not randomly but intelligently; we could use machine load, the locations of VMs, etc.

What exactly does balanced mean? Can it cope with individual nodes having more disk space than others?

Do you support multiple guests accessing the same image?

A VM image can be attached to any VMs but one VM at a time; multiple running VMs cannot access to the same VM image.

What about read-only access? Imagine you'd have 5 kvm instances each accessing it using -snapshot.

Quoted 4 lines · Avi Kivity

If so, is it reasonable to compare this to a cluster file system setup (like GFS) with images as files on this filesystem?  The difference would be that clustering is implemented in userspace in sheepdog, but in the kernel for a clustering filesystem.

I think that the major difference between sheepdog and cluster file systems such as Google File system, pNFS, etc is the interface between clients and a storage system.

Quoted 2 lines

How is load balancing implemented?  Can you move an image transparently while a guest is running?  Will an image be moved closer to its guest?

Sheepdog uses consistent hashing to decide where objects store; I/O load is balanced across the nodes. When a new node is added or the existing node is removed, the hash table changes and the data automatically and transparently are moved over nodes.

We plan to implement a mechanism to distribute the data not randomly but intelligently; we could use machine load, the locations of VMs, etc.

Quoted 1 line

Do you support multiple guests accessing the same image?

A VM image can be attached to any VMs but one VM at a time; multiple running VMs cannot access to the same VM image.

i'd just want to add my '+1 votes' on both getting rid of JVM dependency and using block devices (usually LVM) instead of ext3/btrfs

If the chunks into which the virtual drives are split are quite small (say the 64MB used by Hadoop), LVM may be a less appropriate choice. It doesn't support very large numbers of very small logical volumes very well.

Quoted 2 lines · Javier Guerra

i'd just want to add my '+1 votes' on both getting rid of JVM dependency and using block devices (usually LVM) instead of ext3/btrfs

If the chunks into which the virtual drives are split are quite small (say the 64MB used by Hadoop), LVM may be a less appropriate choice. It doesn't support very large numbers of very small logical volumes very well.

Also, on _loaded_ systems, I noticed creating/removing logical volumes can take really long (several minutes); where allocating a file of a given size would just take a fraction of that.

Quoted 3 lines

Also, on _loaded_ systems, I noticed creating/removing logical volumes can take really long (several minutes); where allocating a file of a given size would just take a fraction of that.

Allocating a file takes much longer, unless you use a 'sparse' file.

Quoted 3 lines

I think that the major difference between sheepdog and cluster file systems such as Google File system, pNFS, etc is the interface between clients and a storage system.

note that GFS is "Global File System" (written by Sistina (the same folks from LVM) and bought by RedHat). Google Filesystem is a different thing, and ironically the client/storage interface is a little more like sheepdog and unlike a regular cluster filesystem.

Quoted 7 lines

Sheepdog uses consistent hashing to decide where objects store; I/O load is balanced across the nodes. When a new node is added or the existing node is removed, the hash table changes and the data automatically and transparently are moved over nodes.

We plan to implement a mechanism to distribute the data not randomly but intelligently; we could use machine load, the locations of VMs, etc.

i don't have much hands-on experience on consistent hashing; but it sounds reasonable to make each node's ring segment proportional to its storage capacity.

dynamic load balancing seems a tougher nut to
crack, especially while keeping all clients mapping consistent.

i'd just want to add my '+1 votes' on both getting rid of JVM dependency and using block devices (usually LVM) instead of ext3/btrfs

Quoted 4 lines · On Fri, 23 Oct 2009 09:14:29 -0500 Javier Guerra<[email protected]> wrote:

Quoted 4 lines

I think that the major difference between sheepdog and cluster file systems such as Google File system, pNFS, etc is the interface between clients and a storage system.

Quoted 5 lines

note that GFS is "Global File System" (written by Sistina (the same folks from LVM) and bought by RedHat). Google Filesystem is a different thing, and ironically the client/storage interface is a little more like sheepdog and unlike a regular cluster filesystem.

Hmm, Avi referred to Global File System? I wasn't sure. 'GFS' is ambiguous. Anyway, Global File System is a SAN file system. It's a completely different architecture from Sheepdog.

Quoted 4 lines

Quoted 1 line

Do you support multiple guests accessing the same image?

A VM image can be attached to any VMs but one VM at a time; multiple running VMs cannot access to the same VM image.

Quoted 3 lines

Quoted 3 lines

Quoted 1 line

Do you support multiple guests accessing the same image?

A VM image can be attached to any VMs but one VM at a time; multiple running VMs cannot access to the same VM image.

I guess this is a problem when you want to do live migrations?

This looks very interesting - how does this compare with Exanodes/Seanodes?

Thanks,
Avishay
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Hi everyone,

Sheepdog is a distributed storage system for KVM/QEMU. It provides highly available block level storage volumes to VMs like Amazon EBS. Sheepdog supports advanced volume management features such as snapshot, cloning, and thin provisioning. Sheepdog runs on several tens or hundreds of nodes, and the architecture is fully symmetric; there is no central node such as a meta-data server.

The following list describes the features of Sheepdog.

   * Linear scalability in performance and capacity
   * No single point of failure
   * Redundant architecture (data is written to multiple nodes)
   - Tolerance against network failure
   * Zero configuration (newly added machines will join the cluster automatically)
   - Autonomous load balancing
   * Snapshot
   - Online snapshot from qemu-monitor
   * Clone from a snapshot volume
   * Thin provisioning
   - Amazon EBS API support (to use from a Eucalyptus instance)

(* = current features, - = on our todo list)

More details and download links are here:

http://www.osrg.net/sheepdog/

Note that the code is still in an early stage.
There are some critical TODO items:

   - VM image deletion support
   - Support architectures other than X86_64
   - Data recoverys
   - Free space management
   - Guarantee reliability and availability under heavy load
   - Performance improvement
   - Reclaim unused blocks
   - More documentation

We hope finding people interested in working together. Enjoy!

Here are examples:

- create images

$ kvm-img create -f sheepdog "Alice's Disk" 256G
$ kvm-img create -f sheepdog "Bob's Disk" 256G

- list images

$ shepherd info -t vdi
  40000 : Alice's Disk  256 GB (allocated: 0 MB, shared: 0 MB), 2009-10-15
16:17:18, tag:        0, current
  80000 : Bob's Disk    256 GB (allocated: 0 MB, shared: 0 MB), 2009-10-15
16:29:20, tag:        0, current

- start up a virtual machine

$ kvm --drive format=sheepdog,file="Alice's Disk"

- create a snapshot

$ kvm-img snapshot -c name sheepdog:"Alice's Disk"

- clone from a snapshot

$ kvm-img create -b sheepdog:"Alice's Disk":0 -f sheepdog "Charlie's Disk"

Thanks.

--
MORITA, Kazutaka

NTT Cyber Space Labs
OSS Computing Project
Kernel Group
E-mail: [email protected]

Thanks for many comments.

Sheepdog git trees are created.

Quoted 3 lines · MORITA Kazutaka

Thanks for many comments.

Sheepdog git trees are created.

great!

is there any client (no matter how crude) besides the patched KVM/Qemu?  it would make it far easier to hack around...

Hi everyone,

Sheepdog is a distributed storage system for KVM/QEMU. It provides highly available block level storage volumes to VMs like Amazon EBS. Sheepdog supports advanced volume management features such as snapshot, cloning, and thin provisioning. Sheepdog runs on several tens or hundreds of nodes, and the architecture is fully symmetric; there is no central node such as a meta-data server.