Skip to contentSkip to conversation

ceph/rbd block driver for qemu-kvm (v8)

PATCH In review Christian Brunner 5 participants 10 messages Nov 17, 2010 – Dec 6, 2010

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 ceph/rbd block driver for qemu-kvm (v8)

Here is another update for the ceph storage driver. It includes changes for the annotations Stefan made last week and a bit more things Sage discovered while looking over the driver again.

I really hope that this time we are not only close, but have reached a quality that everyone is satisfied with. - Of course suggestions for further improvements are always welcome.

Regards,
Christian

RBD is an block driver for the distributed file system Ceph
(http://ceph.newdream.net/). This driver uses librados (which is part of the Ceph server) for direct access to the Ceph object store and is running entirely in userspace (Yehuda also wrote a driver for the linux kernel, that can be used to access rbd volumes as a block device).

---
 Makefile.objs     |    1 +
 block/rbd.c       | 1059 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 block/rbd_types.h |   71 ++++
 configure         |   31 ++
 4 files changed, 1162 insertions(+), 0 deletions(-)
 create mode 100644 block/rbd.c
 create mode 100644 block/rbd_types.h
Makefile.objs +1 -0
index 6ee077c..56a13c1 100644
@@ -19,6 +19,7 @@ block-nested-y += parallels.o nbd.o blkdebug.o
 block-nested-$(CONFIG_WIN32) += raw-win32.o
 block-nested-$(CONFIG_POSIX) += raw-posix.o
 block-nested-$(CONFIG_CURL) += curl.o
+block-nested-$(CONFIG_RBD) += rbd.o
 
 block-obj-y +=  $(addprefix block/, $(block-nested-y))
 
block/rbd.c new file +1059 -0

1,059 lines not on this page · raw message

block/rbd_types.h new file +71 -0

71 lines not on this page · raw message

configure +31 -0
index af50607..5d8f620 100755
@@ -325,6 +325,7 @@ cpu_emulation="yes"
 check_utests="no"
 user_pie="no"
 zero_malloc=""
+rbd=""
 
 # OS specific
 if check_define __linux__ ; then
@@ -724,6 +725,10 @@ for opt do
   ;;
   --*dir)
   ;;
+  --disable-rbd) rbd="no"
+  ;;
+  --enable-rbd) rbd="yes"
+  ;;
   *) echo "ERROR: unknown option $opt"; show_help="yes"
   ;;
   esac
@@ -909,6 +914,7 @@ echo "  --enable-docs            enable documentation build"
 echo "  --disable-docs           disable documentation build"
 echo "  --disable-vhost-net      disable vhost-net acceleration support"
 echo "  --enable-vhost-net       enable vhost-net acceleration support"
+echo "  --enable-rbd             enable building the rados block device (rbd)"
 echo ""
 echo "NOTE: The object files are built at the place where configure is launched"
 exit 1
@@ -1755,6 +1761,27 @@ if test "$mingw32" != yes -a "$pthread" = no; then
 fi
 
 ##########################################
+# rbd probe
+if test "$rbd" != "no" ; then
+  cat > $TMPC <<EOF
+#include <stdio.h>
+#include <rados/librados.h>
+int main(void) { rados_initialize(0, NULL); return 0; }
+EOF
+  rbd_libs="-lrados -lcrypto"
+  if compile_prog "" "$rbd_libs" ; then
+    rbd=yes
+    libs_tools="$rbd_libs $libs_tools"
+    libs_softmmu="$rbd_libs $libs_softmmu"
+  else
+    if test "$rbd" = "yes" ; then
+      feature_not_found "rados block device"
+    fi
+    rbd=no
+  fi
+fi
+
+##########################################
 # linux-aio probe
 
 if test "$linux_aio" != "no" ; then
@@ -2256,6 +2283,7 @@ echo "preadv support    $preadv"
 echo "fdatasync         $fdatasync"
 echo "uuid support      $uuid"
 echo "vhost-net support $vhost_net"
+echo "rbd support       $rbd"
 
 if test $sdl_too_old = "yes"; then
 echo "-> Your SDL version is too old - please upgrade to have SDL support"
@@ -2498,6 +2526,9 @@ echo "CONFIG_UNAME_RELEASE=\"$uname_release\"" >> $config_host_mak
 if test "$zero_malloc" = "yes" ; then
   echo "CONFIG_ZERO_MALLOC=y" >> $config_host_mak
 fi
+if test "$rbd" = "yes" ; then
+  echo "CONFIG_RBD=y" >> $config_host_mak
+fi
 
 # USB host support
 case "$usb" in
Signature · 1 line
1.7.1
#3 lore raw
Quoted 1 line · Christian Brunner

Thanks for the review. What am I supposed to do now?

Kevin is the block maintainer. His review is the next step, I have CCed him. After that rbd would be ready to merge.

Stefan

#4 lore raw

Am 27.11.2010 08:12, schrieb Stefan Hajnoczi:

Quoted 6 lines
Quoted 1 line · Christian Brunner

Thanks for the review. What am I supposed to do now?

Kevin is the block maintainer. His review is the next step, I have CCed him. After that rbd would be ready to merge.

If I don't find anything really obvious and it doesn't break the build, I'll merge it based on your review.

Kevin

#5 lore raw

Am 29.11.2010 09:59, schrieb Kevin Wolf:

Quoted 10 lines

Am 27.11.2010 08:12, schrieb Stefan Hajnoczi:

Quoted 6 lines
Quoted 1 line · Christian Brunner

Thanks for the review. What am I supposed to do now?

Kevin is the block maintainer. His review is the next step, I have CCed him. After that rbd would be ready to merge.

If I don't find anything really obvious and it doesn't break the build, I'll merge it based on your review.

Which librados version is this supposed to require? My F12 one seems to be too old, however configure still automatically enables it (so the build fails in the default configuration for me). I think you need to add some check there.

$ rpm -q ceph-devel
ceph-devel-0.20.2-1.fc12.x86_64
$ LANG=C make
  CC    block/rbd.o
block/rbd.c: In function 'rbd_register_image':
block/rbd.c:191: error: 'CEPH_OSD_TMAP_SET' undeclared (first use in
this function)
block/rbd.c:191: error: (Each undeclared identifier is reported only once
block/rbd.c:191: error: for each function it appears in.)
cc1: warnings being treated as errors
block/rbd.c: In function 'rbd_set_snapc':
block/rbd.c:468: error: implicit declaration of function
'rados_set_snap_context'
block/rbd.c:468: error: nested extern declaration of
'rados_set_snap_context'
block/rbd.c: In function 'rbd_snap_create':
block/rbd.c:844: error: implicit declaration of function
'rados_selfmanaged_snap_create'
block/rbd.c:844: error: nested extern declaration of
'rados_selfmanaged_snap_create'
make: *** [block/rbd.o] Error 1

Kevin

#6 lore raw
Quoted 31 lines · Kevin Wolf

Am 29.11.2010 09:59, schrieb Kevin Wolf:

Quoted 10 lines

Am 27.11.2010 08:12, schrieb Stefan Hajnoczi:

Quoted 6 lines
Quoted 1 line · Christian Brunner

Thanks for the review. What am I supposed to do now?

Kevin is the block maintainer. His review is the next step, I have CCed him. After that rbd would be ready to merge.

If I don't find anything really obvious and it doesn't break the build, I'll merge it based on your review.

Which librados version is this supposed to require? My F12 one seems to be too old, however configure still automatically enables it (so the build fails in the default configuration for me). I think you need to add some check there.

$ rpm -q ceph-devel
ceph-devel-0.20.2-1.fc12.x86_64
$ LANG=C make
  CC    block/rbd.o
block/rbd.c: In function 'rbd_register_image':
block/rbd.c:191: error: 'CEPH_OSD_TMAP_SET' undeclared (first use in
this function)
block/rbd.c:191: error: (Each undeclared identifier is reported only once
block/rbd.c:191: error: for each function it appears in.)
cc1: warnings being treated as errors
block/rbd.c: In function 'rbd_set_snapc':
block/rbd.c:468: error: implicit declaration of function
'rados_set_snap_context'
Quoted 9 lines
block/rbd.c:468: error: nested extern declaration of
'rados_set_snap_context'
block/rbd.c: In function 'rbd_snap_create':
block/rbd.c:844: error: implicit declaration of function
'rados_selfmanaged_snap_create'
block/rbd.c:844: error: nested extern declaration of
'rados_selfmanaged_snap_create'
make: *** [block/rbd.o] Error 1

Right. The CEPH_OSD_TMAP_SET can be fixed and in theory we can get it compiled without snapshots, but we're not sure that it is really a good idea at this point. The following patch disables rbd when librados is too old.

Thanks,
Yehuda

--
From: Yehuda Sadeh <[email protected]>
Date: Mon, 29 Nov 2010 10:38:41 -0800
Subject: [PATCH 1/1] rbd: disable rbd in configure if librados is too old

This checks for the existence of the certain function and other definition.

---
 configure |   27 ++++++++++++++++++++++++---
 1 files changed, 24 insertions(+), 3 deletions(-)
configure +24 -3
index 5d8f620..18ae07c 100755
@@ -1770,15 +1770,36 @@ int main(void) { rados_initialize(0, NULL); return 0; }
 EOF
   rbd_libs="-lrados -lcrypto"
   if compile_prog "" "$rbd_libs" ; then
-    rbd=yes
-    libs_tools="$rbd_libs $libs_tools"
-    libs_softmmu="$rbd_libs $libs_softmmu"
+    librados_too_old=no
+    cat > $TMPC <<EOF
+#include <stdio.h>
+#include <rados/librados.h>
+#ifndef CEPH_OSD_TMAP_SET
+#error missing CEPH_OSD_TMAP_SET
+#endif
+int main(void) {
+    int (*func)(const rados_pool_t pool, uint64_t *snapid) = rados_selfmanaged_snap_create;
+    rados_initialize(0, NULL);
+    return 0;
+}
+EOF
+    if compile_prog "" "$rbd_libs" ; then
+      rbd=yes
+      libs_tools="$rbd_libs $libs_tools"
+      libs_softmmu="$rbd_libs $libs_softmmu"
+    else
+      rbd=no
+      librados_too_old=yes
+    fi
   else
     if test "$rbd" = "yes" ; then
       feature_not_found "rados block device"
     fi
     rbd=no
   fi
+  if test "$librados_too_old" = "yes" ; then
+    echo "-> Your librados version is too old - upgrade needed to have rbd support"
+  fi
 fi
 
 ##########################################
Signature · 1 line
1.5.6.5
#7 lore raw
In reply to Christian Brunner

Am 17.11.2010 22:42, schrieb Christian Brunner:

Quoted 18 lines

Here is another update for the ceph storage driver. It includes changes for the annotations Stefan made last week and a bit more things Sage discovered while looking over the driver again.

I really hope that this time we are not only close, but have reached a quality that everyone is satisfied with. - Of course suggestions for further improvements are always welcome.

Regards,
Christian

RBD is an block driver for the distributed file system Ceph
(http://ceph.newdream.net/). This driver uses librados (which is part of the Ceph server) for direct access to the Ceph object store and is running entirely in userspace (Yehuda also wrote a driver for the linux kernel, that can be used to access rbd volumes as a block device).

Quoted 8 lines
---
 Makefile.objs     |    1 +
 block/rbd.c       | 1059 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 block/rbd_types.h |   71 ++++
 configure         |   31 ++
 4 files changed, 1162 insertions(+), 0 deletions(-)
 create mode 100644 block/rbd.c
 create mode 100644 block/rbd_types.h

This lacks a Signed-off-by. Please merge Yehuda's fix for configure when you resend the patch.

What's the easiest way to try it out? I tried to use vstart.sh and copy the generated ceph.conf to /etc/ceph/ceph.conf so that qemu-img etc. find the monitor address. However, that leads to a hang when I try "rbd list" or "./qemu-img create -f rbd rbd:data/test.img 4G", so I seem to be missing something.

The only thing I have achieved until now with my attempts of trying it out (and trying wrong things, of course) is that I stumbled over the the following segfault in librados:

Program received signal SIGSEGV, Segmentation fault. Objecter::shutdown (this=0x0) at osdc/Objecter.cc:59
59 assert(client_lock.is_locked()); // otherwise event
cancellation is unsafe
(gdb) bt
#0 Objecter::shutdown (this=0x0) at osdc/Objecter.cc:59 #1 0x00007ffff7ca5ce4 in RadosClient::shutdown (this=0xa58a90) at librados.cc:392
#2 0x00007ffff7ca8ccc in rados_deinitialize () at librados.cc:1770 #3 0x000000000043150c in rbd_create (filename=<value optimized out>, options=<value optimized out>) at block/rbd.c:304
#4 0x0000000000405f10 in img_create (argc=5, argv=0x7fffffffde80) at qemu-img.c:409
#5 0x0000003c9f01eb1d in __libc_start_main () from /lib64/libc.so.6 #6 0x0000000000403999 in _start ()

Kevin

#8 lore raw
Quoted 7 lines · Kevin Wolf

What's the easiest way to try it out? I tried to use vstart.sh and copy the generated ceph.conf to /etc/ceph/ceph.conf so that qemu-img etc. find the monitor address. However, that leads to a hang when I try "rbd list" or "./qemu-img create -f rbd rbd:data/test.img 4G", so I seem to be missing something.

What ceph version are you running? Is your system up? What's the 'ceph -s' output?

Quoted 5 lines

The only thing I have achieved until now with my attempts of trying it out (and trying wrong things, of course) is that I stumbled over the the following segfault in librados:

Quoted 16 lines

Program received signal SIGSEGV, Segmentation fault. Objecter::shutdown (this=0x0) at osdc/Objecter.cc:59
59        assert(client_lock.is_locked());  // otherwise event cancellation is unsafe
(gdb) bt
#0  Objecter::shutdown (this=0x0) at osdc/Objecter.cc:59 #1  0x00007ffff7ca5ce4 in RadosClient::shutdown (this=0xa58a90) at librados.cc:392
#2  0x00007ffff7ca8ccc in rados_deinitialize () at librados.cc:1770 #3  0x000000000043150c in rbd_create (filename=<value optimized out>, options=<value optimized out>) at block/rbd.c:304
#4  0x0000000000405f10 in img_create (argc=5, argv=0x7fffffffde80) at qemu-img.c:409
#5  0x0000003c9f01eb1d in __libc_start_main () from /lib64/libc.so.6 #6  0x0000000000403999 in _start ()

This was a bug in the librados C interface. Basically it ignored errors when doing initialization and later on when trying to clean up stuff after failing to do some operation (since it failed to init) it crashed. I pushed a fix for that to the ceph rc branch (and also to the unstable branch).

The question is still why it failed to initialize in the first place. Were there any other messages printed? It could be that it still couldn't find the monitors, or that it failed to authenticate for some reason (if cephx was being used). you can try turning on several ceph modules logs by adding the following to your ceph.conf in the global section:

debug ms = 1
debug rados = 20
debug monc = 10
debug objecter = 10

If everything seems ok and it still doesn't work you can try to run the rbd utility:

  $ ./rbd create test.img --size=4096 -p data

and you can add '--debug-ms=1 --debug-rados=20 --debug-...' to the comman line too.

Let us know if you still have any problems.

Thanks,
Yehuda

Thanks,
Yehuda

#9 lore raw
In reply to Kevin Wolf

2010/12/6 Kevin Wolf <[email protected]>:

Hi Kevin,

Quoted 2 lines

This lacks a Signed-off-by. Please merge Yehuda's fix for configure when you resend the patch.

I've sent an updated patch.

Quoted 5 lines

What's the easiest way to try it out? I tried to use vstart.sh and copy the generated ceph.conf to /etc/ceph/ceph.conf so that qemu-img etc. find the monitor address. However, that leads to a hang when I try "rbd list" or "./qemu-img create -f rbd rbd:data/test.img 4G", so I seem to be missing something.

The most simple ceph.conf I can think about, is the following:

[global]
auth supported = none

[mon]
mon data = /ceph/mon$id

[mon0]

        host = {hostname}
        mon addr = 127.0.0.1:6789

[osd]
osd data = /ceph/osd\$id

[osd0]

        host = {hostname}
        btrfs devs = {devicename}

Replace {hostname} with your hostname -s and {devicename} with the name of an empty volume. Create a directory for the monitor and a mountpoint for the osd volume:

# mkdir -p /ceph/mon0
# mkdir -p /ceph/osd0

After you have created the ceph.conf file, you can create your ceph-filesystem with the following command (attention - this will format the configured volume):

# mkcephfs -c /etc/ceph/ceph.conf --mkbtrfs -a

Now you should be able to start ceph (assuming you are using the redhat rpm):

# service ceph start

Check if ceph is running with ceph -w or rados df. qemu-img create -f rbd rbd:data/test.img 4G should work now, too.

Regards
Christian