ceph/rbd block driver for qemu-kvm (v8)
Subsystems (3)
PATCH 1/1 ceph/rbd block driver for qemu-kvm (v8)
Here is another update for the ceph storage driver. It includes changes for the annotations Stefan made last week and a bit more things Sage discovered while looking over the driver again.
I really hope that this time we are not only close, but have reached a quality that everyone is satisfied with. - Of course suggestions for further improvements are always welcome.
Regards,
Christian
RBD is an block driver for the distributed file system Ceph
(http://ceph.newdream.net/). This driver uses librados (which is part of the Ceph server) for direct access to the Ceph object store and is running entirely in userspace (Yehuda also wrote a driver for the linux kernel, that can be used to access rbd volumes as a block device).
--- Makefile.objs | 1 + block/rbd.c | 1059 +++++++++++++++++++++++++++++++++++++++++++++++++++++ block/rbd_types.h | 71 ++++ configure | 31 ++ 4 files changed, 1162 insertions(+), 0 deletions(-) create mode 100644 block/rbd.c create mode 100644 block/rbd_types.h
Makefile.objs +1 -0
@@ -19,6 +19,7 @@ block-nested-y += parallels.o nbd.o blkdebug.o block-nested-$(CONFIG_WIN32) += raw-win32.o block-nested-$(CONFIG_POSIX) += raw-posix.o block-nested-$(CONFIG_CURL) += curl.o +block-nested-$(CONFIG_RBD) += rbd.o block-obj-y += $(addprefix block/, $(block-nested-y))
block/rbd.c new file +1059 -0
1,059 lines not on this page · raw message
block/rbd_types.h new file +71 -0
71 lines not on this page · raw message
configure +31 -0
@@ -325,6 +325,7 @@ cpu_emulation="yes" check_utests="no" user_pie="no" zero_malloc="" +rbd="" # OS specific if check_define __linux__ ; then
@@ -724,6 +725,10 @@ for opt do ;; --*dir) ;; + --disable-rbd) rbd="no" + ;; + --enable-rbd) rbd="yes" + ;; *) echo "ERROR: unknown option $opt"; show_help="yes" ;; esac
@@ -909,6 +914,7 @@ echo " --enable-docs enable documentation build" echo " --disable-docs disable documentation build" echo " --disable-vhost-net disable vhost-net acceleration support" echo " --enable-vhost-net enable vhost-net acceleration support" +echo " --enable-rbd enable building the rados block device (rbd)" echo "" echo "NOTE: The object files are built at the place where configure is launched" exit 1
@@ -1755,6 +1761,27 @@ if test "$mingw32" != yes -a "$pthread" = no; then fi ########################################## +# rbd probe +if test "$rbd" != "no" ; then + cat > $TMPC <<EOF +#include <stdio.h> +#include <rados/librados.h> +int main(void) { rados_initialize(0, NULL); return 0; } +EOF + rbd_libs="-lrados -lcrypto" + if compile_prog "" "$rbd_libs" ; then + rbd=yes + libs_tools="$rbd_libs $libs_tools" + libs_softmmu="$rbd_libs $libs_softmmu" + else + if test "$rbd" = "yes" ; then + feature_not_found "rados block device" + fi + rbd=no + fi +fi + +########################################## # linux-aio probe if test "$linux_aio" != "no" ; then
@@ -2256,6 +2283,7 @@ echo "preadv support $preadv" echo "fdatasync $fdatasync" echo "uuid support $uuid" echo "vhost-net support $vhost_net" +echo "rbd support $rbd" if test $sdl_too_old = "yes"; then echo "-> Your SDL version is too old - please upgrade to have SDL support"
@@ -2498,6 +2526,9 @@ echo "CONFIG_UNAME_RELEASE=\"$uname_release\"" >> $config_host_mak if test "$zero_malloc" = "yes" ; then echo "CONFIG_ZERO_MALLOC=y" >> $config_host_mak fi +if test "$rbd" = "yes" ; then + echo "CONFIG_RBD=y" >> $config_host_mak +fi # USB host support case "$usb" in
Signature · 1 line
1.7.1
Thanks for the review. What am I supposed to do now?
Christian
2010/11/18 Stefan Hajnoczi <[email protected]>:
Quoted 6 lines
Reviewed-by Stefan Hajnoczi <[email protected]>--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 1 line · Christian Brunner
Thanks for the review. What am I supposed to do now?
Kevin is the block maintainer. His review is the next step, I have CCed him. After that rbd would be ready to merge.
Stefan
Am 27.11.2010 08:12, schrieb Stefan Hajnoczi:
Quoted 6 lines
Quoted 1 line · Christian Brunner
Thanks for the review. What am I supposed to do now?
Kevin is the block maintainer. His review is the next step, I have CCed him. After that rbd would be ready to merge.
If I don't find anything really obvious and it doesn't break the build, I'll merge it based on your review.
Kevin
Am 29.11.2010 09:59, schrieb Kevin Wolf:
Quoted 10 lines
Am 27.11.2010 08:12, schrieb Stefan Hajnoczi:
Quoted 6 lines
Quoted 1 line · Christian Brunner
Thanks for the review. What am I supposed to do now?
Kevin is the block maintainer. His review is the next step, I have CCed him. After that rbd would be ready to merge.
If I don't find anything really obvious and it doesn't break the build, I'll merge it based on your review.
Which librados version is this supposed to require? My F12 one seems to be too old, however configure still automatically enables it (so the build fails in the default configuration for me). I think you need to add some check there.
$ rpm -q ceph-devel ceph-devel-0.20.2-1.fc12.x86_64
$ LANG=C make CC block/rbd.o block/rbd.c: In function 'rbd_register_image': block/rbd.c:191: error: 'CEPH_OSD_TMAP_SET' undeclared (first use in this function) block/rbd.c:191: error: (Each undeclared identifier is reported only once block/rbd.c:191: error: for each function it appears in.) cc1: warnings being treated as errors block/rbd.c: In function 'rbd_set_snapc': block/rbd.c:468: error: implicit declaration of function 'rados_set_snap_context' block/rbd.c:468: error: nested extern declaration of 'rados_set_snap_context' block/rbd.c: In function 'rbd_snap_create': block/rbd.c:844: error: implicit declaration of function 'rados_selfmanaged_snap_create' block/rbd.c:844: error: nested extern declaration of 'rados_selfmanaged_snap_create' make: *** [block/rbd.o] Error 1
Kevin
Quoted 31 lines · Kevin Wolf
Am 29.11.2010 09:59, schrieb Kevin Wolf:
Quoted 10 lines
Am 27.11.2010 08:12, schrieb Stefan Hajnoczi:
Quoted 6 lines
Quoted 1 line · Christian Brunner
Thanks for the review. What am I supposed to do now?
Kevin is the block maintainer. His review is the next step, I have CCed him. After that rbd would be ready to merge.
If I don't find anything really obvious and it doesn't break the build, I'll merge it based on your review.
Which librados version is this supposed to require? My F12 one seems to be too old, however configure still automatically enables it (so the build fails in the default configuration for me). I think you need to add some check there.
$ rpm -q ceph-devel ceph-devel-0.20.2-1.fc12.x86_64$ LANG=C make CC block/rbd.o block/rbd.c: In function 'rbd_register_image': block/rbd.c:191: error: 'CEPH_OSD_TMAP_SET' undeclared (first use in this function) block/rbd.c:191: error: (Each undeclared identifier is reported only once block/rbd.c:191: error: for each function it appears in.) cc1: warnings being treated as errors block/rbd.c: In function 'rbd_set_snapc': block/rbd.c:468: error: implicit declaration of function 'rados_set_snap_context'
Quoted 9 lines
block/rbd.c:468: error: nested extern declaration of 'rados_set_snap_context' block/rbd.c: In function 'rbd_snap_create': block/rbd.c:844: error: implicit declaration of function 'rados_selfmanaged_snap_create' block/rbd.c:844: error: nested extern declaration of 'rados_selfmanaged_snap_create' make: *** [block/rbd.o] Error 1
Right. The CEPH_OSD_TMAP_SET can be fixed and in theory we can get it compiled without snapshots, but we're not sure that it is really a good idea at this point. The following patch disables rbd when librados is too old.
Thanks,
Yehuda
--
From: Yehuda Sadeh <[email protected]>
Date: Mon, 29 Nov 2010 10:38:41 -0800
Subject: [PATCH 1/1] rbd: disable rbd in configure if librados is too old
This checks for the existence of the certain function and other definition.
--- configure | 27 ++++++++++++++++++++++++--- 1 files changed, 24 insertions(+), 3 deletions(-)
configure +24 -3
@@ -1770,15 +1770,36 @@ int main(void) { rados_initialize(0, NULL); return 0; } EOF rbd_libs="-lrados -lcrypto" if compile_prog "" "$rbd_libs" ; then - rbd=yes - libs_tools="$rbd_libs $libs_tools" - libs_softmmu="$rbd_libs $libs_softmmu" + librados_too_old=no + cat > $TMPC <<EOF +#include <stdio.h> +#include <rados/librados.h> +#ifndef CEPH_OSD_TMAP_SET +#error missing CEPH_OSD_TMAP_SET +#endif +int main(void) { + int (*func)(const rados_pool_t pool, uint64_t *snapid) = rados_selfmanaged_snap_create; + rados_initialize(0, NULL); + return 0; +} +EOF + if compile_prog "" "$rbd_libs" ; then + rbd=yes + libs_tools="$rbd_libs $libs_tools" + libs_softmmu="$rbd_libs $libs_softmmu" + else + rbd=no + librados_too_old=yes + fi else if test "$rbd" = "yes" ; then feature_not_found "rados block device" fi rbd=no fi + if test "$librados_too_old" = "yes" ; then + echo "-> Your librados version is too old - upgrade needed to have rbd support" + fi fi ##########################################
Signature · 1 line
1.5.6.5
Am 17.11.2010 22:42, schrieb Christian Brunner:
Quoted 18 lines
Here is another update for the ceph storage driver. It includes changes for the annotations Stefan made last week and a bit more things Sage discovered while looking over the driver again.
I really hope that this time we are not only close, but have reached a quality that everyone is satisfied with. - Of course suggestions for further improvements are always welcome.
Regards,
ChristianRBD is an block driver for the distributed file system Ceph
(http://ceph.newdream.net/). This driver uses librados (which is part of the Ceph server) for direct access to the Ceph object store and is running entirely in userspace (Yehuda also wrote a driver for the linux kernel, that can be used to access rbd volumes as a block device).
Quoted 8 lines
--- Makefile.objs | 1 + block/rbd.c | 1059 +++++++++++++++++++++++++++++++++++++++++++++++++++++ block/rbd_types.h | 71 ++++ configure | 31 ++ 4 files changed, 1162 insertions(+), 0 deletions(-) create mode 100644 block/rbd.c create mode 100644 block/rbd_types.h
This lacks a Signed-off-by. Please merge Yehuda's fix for configure when you resend the patch.
What's the easiest way to try it out? I tried to use vstart.sh and copy the generated ceph.conf to /etc/ceph/ceph.conf so that qemu-img etc. find the monitor address. However, that leads to a hang when I try "rbd list" or "./qemu-img create -f rbd rbd:data/test.img 4G", so I seem to be missing something.
The only thing I have achieved until now with my attempts of trying it out (and trying wrong things, of course) is that I stumbled over the the following segfault in librados:
Program received signal SIGSEGV, Segmentation fault. Objecter::shutdown (this=0x0) at osdc/Objecter.cc:59
59 assert(client_lock.is_locked()); // otherwise event
cancellation is unsafe
(gdb) bt
#0 Objecter::shutdown (this=0x0) at osdc/Objecter.cc:59 #1 0x00007ffff7ca5ce4 in RadosClient::shutdown (this=0xa58a90) at librados.cc:392
#2 0x00007ffff7ca8ccc in rados_deinitialize () at librados.cc:1770 #3 0x000000000043150c in rbd_create (filename=<value optimized out>, options=<value optimized out>) at block/rbd.c:304
#4 0x0000000000405f10 in img_create (argc=5, argv=0x7fffffffde80) at qemu-img.c:409
#5 0x0000003c9f01eb1d in __libc_start_main () from /lib64/libc.so.6 #6 0x0000000000403999 in _start ()
Kevin
Quoted 7 lines · Kevin Wolf
What's the easiest way to try it out? I tried to use
vstart.shand copy the generated ceph.conf to/etc/ceph/ceph.confso that qemu-img etc. find the monitor address. However, that leads to a hang when I try "rbd list" or "./qemu-img create -f rbd rbd:data/test.img 4G", so I seem to be missing something.
What ceph version are you running? Is your system up? What's the 'ceph -s' output?
Quoted 5 lines
The only thing I have achieved until now with my attempts of trying it out (and trying wrong things, of course) is that I stumbled over the the following segfault in librados:
Quoted 16 lines
Program received signal SIGSEGV, Segmentation fault.
Objecter::shutdown(this=0x0) at osdc/Objecter.cc:59
59 assert(client_lock.is_locked()); // otherwise event cancellation is unsafe
(gdb) bt
#0 Objecter::shutdown (this=0x0) at osdc/Objecter.cc:59 #1 0x00007ffff7ca5ce4 inRadosClient::shutdown(this=0xa58a90) at librados.cc:392
#2 0x00007ffff7ca8ccc inrados_deinitialize() at librados.cc:1770 #3 0x000000000043150c inrbd_create(filename=<value optimized out>, options=<value optimized out>) atblock/rbd.c:304
#4 0x0000000000405f10 inimg_create(argc=5, argv=0x7fffffffde80) at qemu-img.c:409
#5 0x0000003c9f01eb1d in__libc_start_main() from/lib64/libc.so.6#6 0x0000000000403999 in_start()
This was a bug in the librados C interface. Basically it ignored errors when doing initialization and later on when trying to clean up stuff after failing to do some operation (since it failed to init) it crashed. I pushed a fix for that to the ceph rc branch (and also to the unstable branch).
The question is still why it failed to initialize in the first place. Were there any other messages printed? It could be that it still couldn't find the monitors, or that it failed to authenticate for some reason (if cephx was being used). you can try turning on several ceph modules logs by adding the following to your ceph.conf in the global section:
debug ms = 1
debug rados = 20
debug monc = 10
debug objecter = 10
If everything seems ok and it still doesn't work you can try to run the rbd utility:
$ ./rbd create test.img --size=4096 -p data
and you can add '--debug-ms=1 --debug-rados=20 --debug-...' to the comman line too.
Let us know if you still have any problems.
Thanks,
Yehuda
Thanks,
Yehuda
2010/12/6 Kevin Wolf <[email protected]>:
Hi Kevin,
Quoted 2 lines
This lacks a Signed-off-by. Please merge Yehuda's fix for configure when you resend the patch.
I've sent an updated patch.
Quoted 5 lines
What's the easiest way to try it out? I tried to use
vstart.shand copy the generated ceph.conf to/etc/ceph/ceph.confso that qemu-img etc. find the monitor address. However, that leads to a hang when I try "rbd list" or "./qemu-img create -f rbd rbd:data/test.img 4G", so I seem to be missing something.
The most simple ceph.conf I can think about, is the following:
[global]
auth supported = none
[mon]
mon data = /ceph/mon$id
[mon0]
host = {hostname}
mon addr = 127.0.0.1:6789[osd]
osd data = /ceph/osd\$id
[osd0]
host = {hostname}
btrfs devs = {devicename}Replace {hostname} with your hostname -s and {devicename} with the name of an empty volume. Create a directory for the monitor and a mountpoint for the osd volume:
# mkdir -p /ceph/mon0 # mkdir -p /ceph/osd0
After you have created the ceph.conf file, you can create your ceph-filesystem with the following command (attention - this will format the configured volume):
# mkcephfs -c /etc/ceph/ceph.conf --mkbtrfs -a
Now you should be able to start ceph (assuming you are using the redhat rpm):
# service ceph start
Check if ceph is running with ceph -w or rados df. qemu-img create -f rbd rbd:data/test.img 4G should work now, too.
Regards
Christian