ceph/rbd block driver for qemu-kvm
Subsystems (3)
RFC PATCH 1/1 ceph/rbd block driver for qemu-kvm
The attached patch is a block driver for the distributed file system Ceph (http://ceph.newdream.net/). This driver uses librados (which is part of the Ceph server) for direct access to the Ceph object store and is running entirely in userspace. Therefore it is called "rbd" - rados block device.
To compile the driver a recent version of ceph (>= 0.20.1) is needed and you have to "--enable-rbd" when running configure.
Additional information is available on the Ceph-Wiki:
http://ceph.newdream.net/wiki/Kvm-rbd
--- Makefile | 3 + Makefile.objs | 1 + block/rados.h | 376 ++++++++++++++++++++++++++++++++++ block/rbd.c | 585 +++++++++++++++++++++++++++++++++++++++++++++++++++++ block/rbd_types.h | 48 +++++ configure | 27 +++ 6 files changed, 1040 insertions(+), 0 deletions(-) create mode 100644 block/rados.h create mode 100644 block/rbd.c create mode 100644 block/rbd_types.h
Makefile +3 -0
@@ -27,6 +27,9 @@ configure: ; $(call set-vpath, $(SRC_PATH):$(SRC_PATH)/hw) LIBS+=-lz $(LIBS_TOOLS) +ifdef CONFIG_RBD +LIBS+=-lrados +endif ifdef BUILD_DOCS DOCS=qemu-doc.html qemu-tech.html qemu.1 qemu-img.1 qemu-nbd.8
Makefile.objs +1 -0
@@ -18,6 +18,7 @@ block-nested-y += parallels.o nbd.o blkdebug.o block-nested-$(CONFIG_WIN32) += raw-win32.o block-nested-$(CONFIG_POSIX) += raw-posix.o block-nested-$(CONFIG_CURL) += curl.o +block-nested-$(CONFIG_RBD) += rbd.o block-obj-y += $(addprefix block/, $(block-nested-y))
block/rados.h new file +376 -0
376 lines not on this page · raw message
block/rbd.c new file +585 -0
585 lines not on this page · raw message
block/rbd_types.h new file +48 -0
48 lines not on this page · raw message
configure +27 -0
@@ -299,6 +299,7 @@ pkgversion="" check_utests="no" user_pie="no" zero_malloc="" +rbd="no" # OS specific if check_define __linux__ ; then
@@ -660,6 +661,8 @@ for opt do ;; --enable-vhost-net) vhost_net="yes" ;; + --enable-rbd) rbd="yes" + ;; *) echo "ERROR: unknown option $opt"; show_help="yes" ;; esac
@@ -826,6 +829,7 @@ echo " --enable-docs enable documentation build" echo " --disable-docs disable documentation build" echo " --disable-vhost-net disable vhost-net acceleration support" echo " --enable-vhost-net enable vhost-net acceleration support" +echo " --enable-rbd enable building the rados block device (rbd)" echo "" echo "NOTE: The object files are built at the place where configure is launched" exit 1
@@ -1569,6 +1573,25 @@ if test "$mingw32" != yes -a "$pthread" = no; then fi ########################################## +# rbd probe +if test "$rbd" != "no" ; then + cat > $TMPC <<EOF +#include <stdio.h> +#include <rados/librados.h> +int main(void) { rados_initialize(0, NULL); return 0; } +EOF + if compile_prog "" "-lrados -lcrypto" ; then + rbd=yes + LIBS="$LIBS -lrados -lcrypto" + else + if test "$rbd" = "yes" ; then + feature_not_found "rados block device" + fi + rbd=no + fi +fi + +########################################## # linux-aio probe if test "$linux_aio" != "no" ; then
@@ -2031,6 +2054,7 @@ echo "preadv support $preadv" echo "fdatasync $fdatasync" echo "uuid support $uuid" echo "vhost-net support $vhost_net" +echo "rbd support $rbd" if test $sdl_too_old = "yes"; then echo "-> Your SDL version is too old - please upgrade to have SDL support"
@@ -2260,6 +2284,9 @@ echo "CONFIG_UNAME_RELEASE=\"$uname_release\"" >> $config_host_mak if test "$zero_malloc" = "yes" ; then echo "CONFIG_ZERO_MALLOC=y" >> $config_host_mak fi +if test "$rbd" = "yes" ; then + echo "CONFIG_RBD=y" >> $config_host_mak +fi # USB host support case "$usb" in
Signature · 1 line
1.7.0.4
Quoted 6 lines · Christian Brunner
The attached patch is a block driver for the distributed file system Ceph (http://ceph.newdream.net/). This driver uses librados (which is part of the Ceph server) for direct access to the Ceph object store and is running entirely in userspace. Therefore it is called "rbd" - rados block device.
Quoted 6 lines
To compile the driver a recent version of ceph (>= 0.20.1) is needed and you have to "--enable-rbd" when running configure.
Additional information is available on the Ceph-Wiki:
I have no idea whether it makes sense to add Ceph (no objection either). I have some minor comments below.
Quoted 45 lines
---
Makefile | 3 +
Makefile.objs | 1 +
block/rados.h | 376 ++++++++++++++++++++++++++++++++++block/rbd.c| 585 +++++++++++++++++++++++++++++++++++++++++++++++++++++block/rbd_types.h| 48 +++++
configure | 27 +++6 files changed, 1040 insertions(+), 0 deletions(-)create mode 100644
block/rados.h
create mode 100644block/rbd.c
create mode 100644block/rbd_types.hMakefile +3
-0@@ -27,6 +27,9 @@ configure: ; $(call set-vpath, $(SRC_PATH):$(SRC_PATH)/hw) LIBS+=-lz $(LIBS_TOOLS) +ifdef CONFIG_RBD +LIBS+=-lrados +endififdef
BUILD_DOCS
DOCS=qemu-doc.html qemu-tech.html qemu.1 qemu-img.1 qemu-nbd.8Makefile.objs +0
-0@@ -18,6 +18,7 @@ block-nested-y += parallels.o nbd.o blkdebug.oblock-nested-$(CONFIG_WIN32) += raw-win32.o
block-nested-$(CONFIG_POSIX) += raw-posix.o
block-nested-$(CONFIG_CURL) += curl.o
+block-nested-$(CONFIG_RBD) += rbd.oblock-obj-y += $(addprefix
block/, $(block-nested-y))block/rados.h new file +0
-0
block/rados.h line 2
Quoted 2 lines
block/rados.h new file +2
-0@@ -0,0 +1,376 @@ +#ifndef __RADOS_H +#define __RADOS_H
IIRC underscores here may conflict with system header use. Please use something like QEMU_BLOCK_RADOS_H.
Quoted 11 lines
block/rados.h new file +11
-0+ +/* +* Data types for the Ceph distributed object storage layer RADOS +* (Reliable Autonomic Distributed Object Store). +*/ + + + +/* +* osdmap encoding versions +*/
block/rados.h line 23
Quoted 10 lines
block/rados.h new file +10
-0+#define CEPH_OSDMAP_INC_VERSION 5 +#define CEPH_OSDMAP_INC_VERSION_EXT 5 +#define CEPH_OSDMAP_VERSION 5 +#define CEPH_OSDMAP_VERSION_EXT 5 + +/* +* fs id +*/ +struct ceph_fsid { + unsigned char fsid[16];
Too large indent, please check also elsewhere.
block/rados.h line 35
Quoted 12 lines
block/rados.h new file +12
-0+}; + +static inline int ceph_fsid_compare (const struct ceph_fsid *a, + const struct ceph_fsid *b) +{ + return memcmp(a, b, sizeof(*a)); +} + +/* +* ino, object, etc. +*/ +typedef __le64 ceph_snapid_t ;
Please use uint64_t and le_to_cpu()/cpu_to_le().
block/rados.h line 36
Quoted 1 line
block/rados.h new file +1
-0+#define CEPH_SNAPDIR ((__u64)(-1))/* reserved for hidden .snap dir */
Likewise, uint64_t is the standard type. Also other places.
Quoted 53 lines
block/rados.h new file +53
-0+#define CEPH_NOSNAP ((__u64)(-2))/* "head", "live" revision */ +#define CEPH_MAXSNAP ((__u64)(-3))/* largest valid snapid */ + +struct ceph_timespec { + __le32 tv_sec; + __le32 tv_nsec; +} __attribute__ ((packed)); + + +/* +* object layout - how objects are mapped into PGs +*/ +#define CEPH_OBJECT_LAYOUT_HASH 1 +#define CEPH_OBJECT_LAYOUT_LINEAR 2 +#define CEPH_OBJECT_LAYOUT_HASHINO 3 + +/* +* pg layout -- how PGs are mapped onto (sets of) OSDs +*/ +#define CEPH_PG_LAYOUT_CRUSH 0 +#define CEPH_PG_LAYOUT_HASH 1 +#define CEPH_PG_LAYOUT_LINEAR 2 +#define CEPH_PG_LAYOUT_HYBRID 3 + + +/* +* placement group. +* we encode this into one __le64. +*/ +struct ceph_pg { + __le16 preferred; /* preferred primary osd */ + __le16 ps; /* placement seed */ + __le32 pool; /* object pool */ +} __attribute__ ((packed)); + +/* +* pg_pool is a set of pgs storing a pool of objects +* +* pg_num -- base number of pseudorandomly placed pgs +* +* pgp_num -- effective number when calculating pg placement. this +* is used for pg_num increases. new pgs result in data being "split" +* into new pgs. for this to proceed smoothly, new pgs are intiially +* colocated with their parents; that is, pgp_num doesn't increase +* until the new pgs have successfully split. only _then_ are the new +* pgs placed independently. +* +* lpg_num -- localized pg count (per device). replicas are randomly +* selected. +* +* lpgp_num -- as above. +*/ +#define CEPH_PG_TYPE_REP 1
block/rados.h line 99
Quoted 10 lines
block/rados.h new file +10
-0+#define CEPH_PG_TYPE_RAID4 2 +#define CEPH_PG_POOL_VERSION 2 +struct ceph_pg_pool { + __u8 type; /* CEPH_PG_TYPE_* */ + __u8 size; /* number of osds in each pg */ + __u8 crush_ruleset; /* crush placement rule */ + __u8 object_hash; /* hash mapping object name to ps */ + __le32 pg_num, pgp_num; /* number of pg's */ + __le32 lpg_num, lpgp_num; /* number of localized pg's */ + __le32 last_change; /* most recent epoch changed */
Is the intent here that one uint32_t is implicitly added for padding or is the structure really unaligned? I'd make the padding explicit to be sure.
block/rados.h line 103
Quoted 4 lines
block/rados.h new file +4
-0+ __le64 snap_seq; /* seq for per-pool snapshot */ + __le32 snap_epoch; /* epoch of last snap */ + __le32 num_snaps; + __le32 num_removed_snap_intervals; /* if non-empty, NO per-pool snaps */
Unaligned?
Quoted 8 lines
block/rados.h new file +8
-0+ __le64 auid; /* who owns the pg */ +} __attribute__ ((packed)); + +/* +* stable_mod func is used to control number of placement groups. +* similar to straight-up modulo, but produces a stable mapping as b +* increases over time. b is the number of bins, and bmask is the +* containing power of 2 minus 1.
block/rados.h line 121
Quoted 10 lines
block/rados.h new file +10
-0+* +* b <= bmask and bmask=(2**n)-1 +* e.g., b=12 -> bmask=15, b=123 -> bmask=127 +*/ +static inline int ceph_stable_mod (int x,int b,int bmask) +{ + if ((x & bmask) < b) + return x & bmask; + else + return x & (bmask >> 1);
Please check CODING_STYLE for brace use.
Quoted 5 lines
block/rados.h new file +5
-0+} + +/* +* object layout - how a given object should be stored. +*/
block/rados.h line 136
Quoted 10 lines
block/rados.h new file +10
-0+struct ceph_object_layout { + struct ceph_pg ol_pgid; /* raw pg, with _full_ ps precision. */ + __le32 ol_stripe_unit; /* for per-object parity, if any */ +} __attribute__ ((packed)); + +/* +* compound epoch+version, used by storage layer to serialize mutations +*/ +struct ceph_eversion { + __le32 epoch;
Unaligned?
Quoted 159 lines
block/rados.h new file +159
-0+ __le64 version; +} __attribute__ ((packed)); + +/* +* osd map bits +*/ + +/* status bits */ +#define CEPH_OSD_EXISTS 1 +#define CEPH_OSD_UP 2 + +/* osd weights. fixed point value: 0x10000 == 1.0 ("in"), 0 == "out" */ +#define CEPH_OSD_IN 0x10000 +#define CEPH_OSD_OUT 0 + + +/* +* osd map flag bits +*/ +#define CEPH_OSDMAP_NEARFULL (1<<0)/* sync writes (near ENOSPC) */ +#define CEPH_OSDMAP_FULL (1<<1)/* no data writes (ENOSPC) */ +#define CEPH_OSDMAP_PAUSERD (1<<2)/* pause all reads */ +#define CEPH_OSDMAP_PAUSEWR (1<<3)/* pause all writes */ +#define CEPH_OSDMAP_PAUSEREC (1<<4)/* pause recovery */ + +/* +* osd ops +*/ +#define CEPH_OSD_OP_MODE 0xf000 +#define CEPH_OSD_OP_MODE_RD 0x1000 +#define CEPH_OSD_OP_MODE_WR 0x2000 +#define CEPH_OSD_OP_MODE_RMW 0x3000 +#define CEPH_OSD_OP_MODE_SUB 0x4000 + +#define CEPH_OSD_OP_TYPE 0x0f00 +#define CEPH_OSD_OP_TYPE_LOCK 0x0100 +#define CEPH_OSD_OP_TYPE_DATA 0x0200 +#define CEPH_OSD_OP_TYPE_ATTR 0x0300 +#define CEPH_OSD_OP_TYPE_EXEC 0x0400 +#define CEPH_OSD_OP_TYPE_PG 0x0500 + +enum { + /** data **/ + /* read */ + CEPH_OSD_OP_READ = CEPH_OSD_OP_MODE_RD | CEPH_OSD_OP_TYPE_DATA | 1, + CEPH_OSD_OP_STAT = CEPH_OSD_OP_MODE_RD | CEPH_OSD_OP_TYPE_DATA | 2, + + /* fancy read */ + CEPH_OSD_OP_MASKTRUNC = CEPH_OSD_OP_MODE_RD | CEPH_OSD_OP_TYPE_DATA | 4, + + /* write */ + CEPH_OSD_OP_WRITE = CEPH_OSD_OP_MODE_WR | CEPH_OSD_OP_TYPE_DATA | 1, + CEPH_OSD_OP_WRITEFULL = CEPH_OSD_OP_MODE_WR | CEPH_OSD_OP_TYPE_DATA | 2, + CEPH_OSD_OP_TRUNCATE = CEPH_OSD_OP_MODE_WR | CEPH_OSD_OP_TYPE_DATA | 3, + CEPH_OSD_OP_ZERO = CEPH_OSD_OP_MODE_WR | CEPH_OSD_OP_TYPE_DATA | 4, + CEPH_OSD_OP_DELETE = CEPH_OSD_OP_MODE_WR | CEPH_OSD_OP_TYPE_DATA | 5, + + /* fancy write */ + CEPH_OSD_OP_APPEND = CEPH_OSD_OP_MODE_WR | CEPH_OSD_OP_TYPE_DATA | 6, + CEPH_OSD_OP_STARTSYNC = CEPH_OSD_OP_MODE_WR | CEPH_OSD_OP_TYPE_DATA | 7, + CEPH_OSD_OP_SETTRUNC = CEPH_OSD_OP_MODE_WR | CEPH_OSD_OP_TYPE_DATA | 8, + CEPH_OSD_OP_TRIMTRUNC = CEPH_OSD_OP_MODE_WR | CEPH_OSD_OP_TYPE_DATA | 9, + + CEPH_OSD_OP_TMAPUP = CEPH_OSD_OP_MODE_RMW | CEPH_OSD_OP_TYPE_DATA | 10, + CEPH_OSD_OP_TMAPPUT = CEPH_OSD_OP_MODE_WR | CEPH_OSD_OP_TYPE_DATA | 11, + CEPH_OSD_OP_TMAPGET = CEPH_OSD_OP_MODE_RD | CEPH_OSD_OP_TYPE_DATA | 12, + + CEPH_OSD_OP_CREATE = CEPH_OSD_OP_MODE_WR | CEPH_OSD_OP_TYPE_DATA | 13, + + /** attrs **/ + /* read */ + CEPH_OSD_OP_GETXATTR = CEPH_OSD_OP_MODE_RD | CEPH_OSD_OP_TYPE_ATTR | 1, + CEPH_OSD_OP_GETXATTRS = CEPH_OSD_OP_MODE_RD | CEPH_OSD_OP_TYPE_ATTR | 2, + + /* write */ + CEPH_OSD_OP_SETXATTR = CEPH_OSD_OP_MODE_WR | CEPH_OSD_OP_TYPE_ATTR | 1, + CEPH_OSD_OP_SETXATTRS = CEPH_OSD_OP_MODE_WR | CEPH_OSD_OP_TYPE_ATTR | 2, + CEPH_OSD_OP_RESETXATTRS = CEPH_OSD_OP_MODE_WR|CEPH_OSD_OP_TYPE_ATTR | 3, + CEPH_OSD_OP_RMXATTR = CEPH_OSD_OP_MODE_WR | CEPH_OSD_OP_TYPE_ATTR | 4, + + /** subop **/ + CEPH_OSD_OP_PULL = CEPH_OSD_OP_MODE_SUB | 1, + CEPH_OSD_OP_PUSH = CEPH_OSD_OP_MODE_SUB | 2, + CEPH_OSD_OP_BALANCEREADS = CEPH_OSD_OP_MODE_SUB | 3, + CEPH_OSD_OP_UNBALANCEREADS = CEPH_OSD_OP_MODE_SUB | 4, + CEPH_OSD_OP_SCRUB = CEPH_OSD_OP_MODE_SUB | 5, + + /** lock **/ + CEPH_OSD_OP_WRLOCK = CEPH_OSD_OP_MODE_WR | CEPH_OSD_OP_TYPE_LOCK | 1, + CEPH_OSD_OP_WRUNLOCK = CEPH_OSD_OP_MODE_WR | CEPH_OSD_OP_TYPE_LOCK | 2, + CEPH_OSD_OP_RDLOCK = CEPH_OSD_OP_MODE_WR | CEPH_OSD_OP_TYPE_LOCK | 3, + CEPH_OSD_OP_RDUNLOCK = CEPH_OSD_OP_MODE_WR | CEPH_OSD_OP_TYPE_LOCK | 4, + CEPH_OSD_OP_UPLOCK = CEPH_OSD_OP_MODE_WR | CEPH_OSD_OP_TYPE_LOCK | 5, + CEPH_OSD_OP_DNLOCK = CEPH_OSD_OP_MODE_WR | CEPH_OSD_OP_TYPE_LOCK | 6, + + /** exec **/ + CEPH_OSD_OP_CALL = CEPH_OSD_OP_MODE_RD | CEPH_OSD_OP_TYPE_EXEC | 1, + + /** pg **/ + CEPH_OSD_OP_PGLS = CEPH_OSD_OP_MODE_RD | CEPH_OSD_OP_TYPE_PG | 1, +}; + +static inline int ceph_osd_op_type_lock (int op) +{ + return (op & CEPH_OSD_OP_TYPE) == CEPH_OSD_OP_TYPE_LOCK; +} +static inline int ceph_osd_op_type_data (int op) +{ + return (op & CEPH_OSD_OP_TYPE) == CEPH_OSD_OP_TYPE_DATA; +} +static inline int ceph_osd_op_type_attr (int op) +{ + return (op & CEPH_OSD_OP_TYPE) == CEPH_OSD_OP_TYPE_ATTR; +} +static inline int ceph_osd_op_type_exec (int op) +{ + return (op & CEPH_OSD_OP_TYPE) == CEPH_OSD_OP_TYPE_EXEC; +} +static inline int ceph_osd_op_type_pg (int op) +{ + return (op & CEPH_OSD_OP_TYPE) == CEPH_OSD_OP_TYPE_PG; +} + +static inline int ceph_osd_op_mode_subop (int op) +{ + return (op & CEPH_OSD_OP_MODE) == CEPH_OSD_OP_MODE_SUB; +} +static inline int ceph_osd_op_mode_read (int op) +{ + return (op & CEPH_OSD_OP_MODE) == CEPH_OSD_OP_MODE_RD; +} +static inline int ceph_osd_op_mode_modify (int op) +{ + return (op & CEPH_OSD_OP_MODE) == CEPH_OSD_OP_MODE_WR; +} + +#define CEPH_OSD_TMAP_HDR 'h' +#define CEPH_OSD_TMAP_SET 's' +#define CEPH_OSD_TMAP_RM 'r' + +extern const char *ceph_osd_op_name (int op); + + +/* +* osd op flags +* +* An op may be READ, WRITE, or READ|WRITE. +*/ +enum { + CEPH_OSD_FLAG_ACK = 1, /* want (or is) "ack" ack */ + CEPH_OSD_FLAG_ONNVRAM = 2, /* want (or is) "onnvram" ack */ + CEPH_OSD_FLAG_ONDISK = 4, /* want (or is) "ondisk" ack */ + CEPH_OSD_FLAG_RETRY = 8, /* resend attempt */ + CEPH_OSD_FLAG_READ = 16, /* op may read */ + CEPH_OSD_FLAG_WRITE = 32, /* op may write */ + CEPH_OSD_FLAG_ORDERSNAP = 64, /* EOLDSNAP if snapc is out of order */ + CEPH_OSD_FLAG_PEERSTAT = 128, /* msg includes osd_peer_stat */ + CEPH_OSD_FLAG_BALANCE_READS = 256, + CEPH_OSD_FLAG_PARALLELEXEC = 512, /* execute op in parallel */
block/rados.h line 305
Quoted 10 lines
block/rados.h new file +10
-0+ CEPH_OSD_FLAG_PGOP = 1024, /* pg op, no object */ + CEPH_OSD_FLAG_EXEC = 2048, /* op may exec */ +}; + +enum { + CEPH_OSD_OP_FLAG_EXCL = 1, /* EXCL object create */ +}; + +#define EOLDSNAPC ERESTART/* ORDERSNAP flag set; writer has old snapc*/ +#define EBLACKLISTED ESHUTDOWN/* blacklisted */
Are these used somewhere? Maybe these could clash with system errnos.
block/rados.h line 312
Quoted 7 lines
block/rados.h new file +7
-0+ +/* +* an individual object operation. each may be accompanied by some data +* payload +*/ +struct ceph_osd_op { + __le16 op; /* CEPH_OSD_OP_* */
Unaligned?
Quoted 5 lines
block/rados.h new file +5
-0+ __le32 flags; /* CEPH_OSD_FLAG_* */ + union { + struct { + __le64 offset, length; + __le64 truncate_size;
block/rados.h line 327
Quoted 10 lines
block/rados.h new file +10
-0+ __le32 truncate_seq; + } __attribute__ ((packed)) extent; + struct { + __le32 name_len; + __le32 value_len; + } __attribute__ ((packed)) xattr; + struct { + __u8 class_len; + __u8 method_len; + __u8 argc;
Unaligned?
Quoted 6 lines
block/rados.h new file +6
-0+ __le32 indata_len; + } __attribute__ ((packed)) cls; + struct { + __le64 cookie, count; + } __attribute__ ((packed)) pgls; + };
block/rados.h line 343
Quoted 10 lines
block/rados.h new file +10
-0+ __le32 payload_len; +} __attribute__ ((packed)); + +/* +* osd request message header. each request may include multiple +* ceph_osd_op object operations. +*/ +struct ceph_osd_request_head { + __le32 client_inc; /* client incarnation */ + struct ceph_object_layout layout; /* pgid */
Unaligned on 64 bit hosts?
block/rados.h line 353
Quoted 10 lines
block/rados.h new file +10
-0+ __le32 osdmap_epoch; /* client's osdmap epoch */ + + __le32 flags; + + struct ceph_timespec mtime; /* for mutations only */ + struct ceph_eversion reassert_version; /* if we are replaying op */ + + __le32 object_len; /* length of object name */ + + __le64 snapid; /* snapid to read */
Unaligned?
block/rados.h line 358
Quoted 5 lines
block/rados.h new file +5
-0+ __le64 snap_seq; /* writer's snap context */ + __le32 num_snaps; + + __le16 num_ops; + struct ceph_osd_op ops[]; /* followed by ops[], obj, ticket, snaps */
Unaligned?
block/rados.h line 365
Quoted 7 lines
block/rados.h new file +7
-0+} __attribute__ ((packed)); + +struct ceph_osd_reply_head { + __le32 client_inc; /* client incarnation */ + __le32 flags; + struct ceph_object_layout layout; + __le32 osdmap_epoch;
Unaligned on 64 bit hosts?
block/rados.h line 372
Quoted 7 lines
block/rados.h new file +7
-0+ struct ceph_eversion reassert_version; /* for replaying uncommitted */ + + __le32 result; /* result code */ + + __le32 object_len; /* length of object name */ + __le32 num_ops; + struct ceph_osd_op ops[0]; /* ops[], object */
Unaligned on 64 bit hosts?
Quoted 109 lines
block/rados.h new file +4
-0+} __attribute__ ((packed)); + + +#endif block/rbd.c new file +99
-0@@ -0,0 +1,585 @@ +/* +* QEMU Block driver for RADOS (Ceph) +* +* Copyright (C) 2010 Christian Brunner <[email protected]> +* +* This work is licensed under the terms of the GNU GPL, version 2. See +* the COPYING file in the top-level directory. +* +*/ + +#include "qemu-common.h" +#include <sys/types.h> +#include <stdbool.h> + +#include <qemu-common.h> + +#include "rbd_types.h" +#include "rados.h" +#include "module.h" +#include "block_int.h" + +#include <stdio.h> +#include <stdlib.h> +#include <rados/librados.h> + +#include <signal.h> + +/* +* When specifying the image filename use: +* +* rbd:poolname/devicename +* +* poolname must be the name of an existing rados pool +* +* devicename is the basename for all objects used to +* emulate the raw device. +* +* Metadata information (image size, ...) is stored in an +* object with the name "devicename.rbd". +* +* The raw device is split into 4MB sized objects by default. +* The sequencenumber is encoded in a 12 byte long hex-string, +* and is attached to the devicename, separated by a dot. +* e.g. "devicename.1234567890ab" +* +*/ + +#define OBJ_MAX_SIZE (1UL << OBJ_DEFAULT_OBJ_ORDER) + +typedef struct RBDAIOCB { + BlockDriverAIOCB common; + QEMUBH *bh; + int ret; + QEMUIOVector *qiov; + char *bounce; + int write; + int64_t sector_num; + int aiocnt; + int error; +}RBDAIOCB ; + +typedef struct RADOSCB { + int rcbid; + RBDAIOCB *acb; + int done; + int64_t segsize; + char *buf; +}RADOSCB ; + +typedef struct RBDRVRBDState { + rados_pool_t pool; + char name[RBD_MAX_OBJ_NAME_SIZE]; + int name_len; + uint64_t size; + uint64_t objsize; +}RBDRVRBDState ; + +typedef struct rbd_obj_header_ondisk RbdHeader1 ; + +static int rbd_parsename (const char *filename,char *pool,char *name) +{ + const char *rbdname; + char *p, *n; + int l; + + if (!strstart(filename, "rbd:", &rbdname)) { + return -EINVAL; + } + + pstrcpy(pool, 2 * RBD_MAX_SEG_NAME_SIZE, rbdname); + p = strchr(pool, '/'); + if (p == NULL) { + return -EINVAL; + } + + *p = '\0'; + n = ++p; + + l = strlen(n);
block/rbd.c line 109
Quoted 10 lines
block/rbd.c new file +10
-0+ + if (l > RBD_MAX_OBJ_NAME_SIZE) { + fprintf(stderr, "object name to long\n"); + return -EINVAL; + } else if (l <= 0) { + fprintf(stderr, "object name to short\n"); + return -EINVAL; + } + + strcpy(name, n);
pstrcpy(name, l, n);
Quoted 4 lines
block/rbd.c new file +4
-0+ + return l; +} +
block/rbd.c line 123
Quoted 10 lines
block/rbd.c new file +10
-0+static int create_tmap_op (uint8_t op,const char *name,char **tmap_desc) +{ + uint32_t len = strlen(name); + uint32_t total_len = 1 + (sizeof(uint32_t) + len) + sizeof(uint32_t); /* encoding op + name + empty buffer */ + char *desc; + + desc = qemu_malloc(total_len); + if (!desc) { + return -ENOMEM; + }
qemu_malloc won't return NULL, the check is not useful.
Quoted 195 lines
block/rbd.c new file +195
-0+ + *tmap_desc = desc; + + *desc = op; + desc++; + memcpy(desc, &len, sizeof(len)); + desc += sizeof(len); + memcpy(desc, name, len); + desc += len; + len = 0; + memcpy(desc, &len, sizeof(len)); + desc += sizeof(len); + + return desc - *tmap_desc; +} + +static void free_tmap_op (char *tmap_desc) +{ + qemu_free(tmap_desc); +} + +static int rbd_register_image (rados_pool_t pool,const char *name) +{ + char *tmap_desc; + const char *dir = RBD_DIRECTORY; + int ret; + + ret = create_tmap_op(CEPH_OSD_TMAP_SET, name, &tmap_desc); + if (ret < 0) { + return ret; + } + + ret = rados_tmap_update(pool, dir, tmap_desc, ret); + free_tmap_op(tmap_desc); + + return ret; +} + +static int rbd_create (const char *filename,QEMUOptionParameter *options) +{ + int64_t bytes = 0; + int64_t objsize; + uint64_t size; + time_t mtime; + uint8_t obj_order = RBD_DEFAULT_OBJ_ORDER; + char pool[RBD_MAX_SEG_NAME_SIZE]; + char n[RBD_MAX_SEG_NAME_SIZE]; + char name[RBD_MAX_SEG_NAME_SIZE]; + RbdHeader1 header; + rados_pool_t p; + int name_len; + int ret; + + if ((name_len = rbd_parsename(filename, pool, name)) < 0) { + return -EINVAL; + } + + snprintf(n, RBD_MAX_SEG_NAME_SIZE, "%s%s", name, RBD_SUFFIX); + + /* Read out options */ + while (options && options->name) { + if (!strcmp(options->name, BLOCK_OPT_SIZE)) { + bytes = options->value.n; + } else if (!strcmp(options->name, BLOCK_OPT_CLUSTER_SIZE)) { + if (options->value.n) { + objsize = options->value.n; + if (!objsize || ((objsize - 1) & objsize)) { /* not a power of 2? */ + fprintf(stderr, "obj size needs to be power of 2\n"); + return -EINVAL; + } + if (objsize < 4096) { + fprintf(stderr, "obj size too small\n"); + return -EINVAL; + } + + for (obj_order = 0; obj_order < 64; obj_order++) { + if (objsize == 1) + break; + objsize >>= 1; + } + } + } + options++; + } + + memset(&header, 0, sizeof(header)); + pstrcpy(header.text, sizeof(header.text), rbd_text); + pstrcpy(header.signature, sizeof(header.signature), rbd_signature); + pstrcpy(header.version, sizeof(header.version), rbd_version); + header.image_size = bytes; + cpu_to_le64s((uint64_t *) & header.image_size); + header.obj_order = obj_order; + header.crypt_type = RBD_CRYPT_NONE; + header.comp_type = RBD_COMP_NONE; + header.snap_seq = 0; + header.snap_count = 0; + cpu_to_le32s(&header.snap_count); + + if (rados_initialize(0, NULL) < 0) { + fprintf(stderr, "error initializing\n"); + return -EIO; + } + + if (rados_open_pool(pool, &p)) { + fprintf(stderr, "error opening pool %s\n", pool); + return -EIO; + } + + /* check for existing rbd header file */ + ret = rados_stat(p, n, &size, &mtime); + if (ret == 0) { + ret=-EEXIST; + goto done; + } + + /* create header file */ + ret = rados_write(p, n, 0, (const char *)&header, sizeof(header)); + if (ret < 0) { + goto done; + } + + ret = rbd_register_image(p, name); +done: + rados_close_pool(p); + rados_deinitialize(); + + return ret; +} + +static int rbd_open (BlockDriverState *bs,const char *filename,int flags) +{ + RBDRVRBDState *s = bs->opaque; + char pool[RBD_MAX_SEG_NAME_SIZE]; + char n[RBD_MAX_SEG_NAME_SIZE]; + char hbuf[4096]; + + if ((s->name_len = rbd_parsename(filename, pool, s->name)) < 0) { + return -EINVAL; + } + snprintf(n, RBD_MAX_SEG_NAME_SIZE, "%s%s", s->name, RBD_SUFFIX); + + if (rados_initialize(0, NULL) < 0) { + fprintf(stderr, "error initializing\n"); + return -EIO; + } + + if (rados_open_pool(pool, &s->pool)) { + fprintf(stderr, "error opening pool %s\n", pool); + return -EIO; + } + + if (rados_read(s->pool, n, 0, hbuf, 4096) < 0) { + fprintf(stderr, "error reading header from %s\n", s->name); + return -EIO; + } + if (!strncmp(hbuf + 64, rbd_signature, 4)) { + if (!strncmp(hbuf + 68, rbd_version, 8)) { + RbdHeader1 *header; + + header = (RbdHeader1 *) hbuf; + le64_to_cpus((uint64_t *) & header->image_size); + s->size = header->image_size; + s->objsize = 1 << header->obj_order; + } else { + fprintf(stderr, "Unknown image version %s\n", hbuf + 68); + return -EIO; + } + } else { + fprintf(stderr, "Invalid header signature %s\n", hbuf + 64); + return -EIO; + } + + return 0; +} + +static void rbd_close (BlockDriverState *bs) +{ + RBDRVRBDState *s = bs->opaque; + + rados_close_pool(s->pool); + rados_deinitialize(); +} + +static int rbd_rw (BlockDriverState *bs,int64_t sector_num, + uint8_t *buf, int nb_sectors, int write) +{ + RBDRVRBDState *s = bs->opaque; + char n[RBD_MAX_SEG_NAME_SIZE]; + + int64_t segnr, segoffs, segsize, r; + int64_t off, size; + + off = sector_num * 512; + size = nb_sectors * 512; + segnr = (int64_t) (off / s->objsize);
block/rbd.c line 328
Quoted 10 lines
block/rbd.c new file +10
-0+ segoffs = (int64_t) (off % s->objsize); + segsize = (int64_t) (s->objsize - segoffs); + + while (size > 0) { + if (size < segsize) { + segsize = size; + } + + snprintf(n, RBD_MAX_SEG_NAME_SIZE, "%s.%012llx", s->name, + (long long unsigned int)segnr);
Please use PRIx64 instead of llx, the cast won't be needed then. Also elsewhere.
Quoted 263 lines
block/rbd.c new file +257
-0+ + if (write) { + if ((r = rados_write(s->pool, n, segoffs, (const char *)buf, + segsize)) < 0) { + return r; + } + } else { + r = rados_read(s->pool, n, segoffs, (char *)buf, segsize); + if (r == -ENOENT) { + memset(buf, 0, segsize); + } else if (r < 0) { + return(r); + } else if (r < segsize) { + memset(buf + r, 0, segsize - r); + } + r = segsize; + } + + buf += segsize; + size -= segsize; + segoffs = 0; + segsize = s->objsize; + segnr++; + } + + return (0); +} + +static int rbd_read (BlockDriverState *bs,int64_t sector_num, + uint8_t *buf, int nb_sectors) +{ + return rbd_rw(bs, sector_num, buf, nb_sectors, 0); +} + +static int rbd_write (BlockDriverState *bs,int64_t sector_num, + const uint8_t *buf, int nb_sectors) +{ + return rbd_rw(bs, sector_num, (uint8_t *) buf, nb_sectors, 1); +} + +static void rbd_aio_cancel (BlockDriverAIOCB *blockacb) +{ + RBDAIOCB *acb = (RBDAIOCB *) blockacb; + qemu_bh_delete(acb->bh); + acb->bh = NULL; + qemu_aio_release(acb); +} + +static AIOPool rbd_aio_pool = { + .aiocb_size = sizeof(RBDAIOCB), + .cancel = rbd_aio_cancel, +}; + +/* This is the callback function for rados_aio_read and _write */ +static void rbd_finish_aiocb (rados_completion_t c,RADOSCB *rcb) +{ + RBDAIOCB *acb = rcb->acb; + int64_t r; + int i; + + acb->aiocnt--; + r = rados_aio_get_return_value(c); + rados_aio_release(c); + if (acb->write) { + if (r < 0) { + acb->ret = r; + acb->error = 1; + } else if (!acb->error) { + acb->ret += rcb->segsize; + } + } else { + if (r == -ENOENT) { + memset(rcb->buf, 0, rcb->segsize); + if (!acb->error) { + acb->ret += rcb->segsize; + } + } else if (r < 0) { + acb->ret = r; + acb->error = 1; + } else if (r < rcb->segsize) { + memset(rcb->buf + r, 0, rcb->segsize - r); + if (!acb->error) { + acb->ret += rcb->segsize; + } + } else if (!acb->error) { + acb->ret += r; + } + } + qemu_free(rcb); + i = 0; + if (!acb->aiocnt && acb->bh) { + qemu_bh_schedule(acb->bh); + } +} + +/* Callback when all queued rados_aio requests are complete */ +static void rbd_aio_bh_cb (void *opaque) +{ + RBDAIOCB *acb = opaque; + + if (!acb->write) { + qemu_iovec_from_buffer(acb->qiov, acb->bounce, acb->qiov->size); + } + qemu_vfree(acb->bounce); + acb->common.cb(acb->common.opaque, (acb->ret > 0 ? 0 : acb->ret)); + qemu_bh_delete(acb->bh); + acb->bh = NULL; + qemu_aio_release(acb); +} + +static BlockDriverAIOCB *rbd_aio_rw_vector (BlockDriverState *bs, + int64_t sector_num, + QEMUIOVector *qiov, + int nb_sectors, + BlockDriverCompletionFunc *cb, + void *opaque, int write) +{ + RBDAIOCB *acb; + RADOSCB *rcb; + rados_completion_t c; + char n[RBD_MAX_SEG_NAME_SIZE]; + int64_t segnr, segoffs, segsize, last_segnr; + int64_t off, size; + char *buf; + + RBDRVRBDState *s = bs->opaque; + + acb = qemu_aio_get(&rbd_aio_pool, bs, cb, opaque); + acb->write = write; + acb->qiov = qiov; + acb->bounce = qemu_blockalign(bs, qiov->size); + acb->aiocnt = 0; + acb->ret = 0; + acb->error = 0; + + if (!acb->bh) { + acb->bh = qemu_bh_new(rbd_aio_bh_cb, acb); + } + + if (write) { + qemu_iovec_to_buffer(acb->qiov, acb->bounce); + } + + buf = acb->bounce; + + off = sector_num * 512; + size = nb_sectors * 512; + segnr = (int64_t) (off / s->objsize); + segoffs = (int64_t) (off % s->objsize); + segsize = (int64_t) (s->objsize - segoffs); + + last_segnr = ((off + size - 1) / s->objsize); + acb->aiocnt = (last_segnr - segnr) + 1; + + while (size > 0) { + if (size < segsize) { + segsize = size; + } + + snprintf(n, RBD_MAX_SEG_NAME_SIZE, "%s.%012llx", s->name, + (long long unsigned int)segnr); + + rcb = qemu_malloc(sizeof(RADOSCB)); + rcb->done = 0; + rcb->acb = acb; + rcb->segsize = segsize; + rcb->buf = buf; + + if (write) { + rados_aio_create_completion(rcb, NULL, + (rados_callback_t) rbd_finish_aiocb, &c); + rados_aio_write(s->pool, n, segoffs, buf, segsize, c); + } else { + rados_aio_create_completion(rcb, (rados_callback_t) rbd_finish_aiocb, + NULL, &c); + rados_aio_read(s->pool, n, segoffs, buf, segsize, c); + } + + buf += segsize; + size -= segsize; + segoffs = 0; + segsize = s->objsize; + segnr++; + } + + return &acb->common; +} + +static BlockDriverAIOCB *rbd_aio_readv (BlockDriverState *bs, + int64_t sector_num, QEMUIOVector *qiov, + int nb_sectors, + BlockDriverCompletionFunc *cb, + void *opaque) +{ + return rbd_aio_rw_vector(bs, sector_num, qiov, nb_sectors, cb, opaque, 0); +} + +static BlockDriverAIOCB *rbd_aio_writev (BlockDriverState *bs, + int64_t sector_num, QEMUIOVector *qiov, + int nb_sectors, + BlockDriverCompletionFunc *cb, + void *opaque) +{ + return rbd_aio_rw_vector(bs, sector_num, qiov, nb_sectors, cb, opaque, 1); +} + +static int rbd_getinfo (BlockDriverState *bs,BlockDriverInfo *bdi) +{ + RBDRVRBDState *s = bs->opaque; + bdi->cluster_size = s->objsize; + return 0; +} + +static int64_t rbd_getlength (BlockDriverState *bs) +{ + RBDRVRBDState *s = bs->opaque; + + return s->size; +} + +static QEMUOptionParameter rbd_create_options[] = { + { + .name = BLOCK_OPT_SIZE, + .type = OPT_SIZE, + .help = "Virtual disk size" + }, + { + .name = BLOCK_OPT_CLUSTER_SIZE, + .type = OPT_SIZE, + .help = "RBD object size" + }, + {NULL} +}; + +static BlockDriver bdrv_rbd = { + .format_name = "rbd", + .instance_size = sizeof(RBDRVRBDState), + .bdrv_open = rbd_open, + .bdrv_read = rbd_read, + .bdrv_write = rbd_write, + .bdrv_close = rbd_close, + .bdrv_create = rbd_create, + .bdrv_get_info = rbd_getinfo, + .create_options = rbd_create_options, + .bdrv_getlength = rbd_getlength, + .protocol_name = "rbd", + + .bdrv_aio_readv = rbd_aio_readv, + .bdrv_aio_writev = rbd_aio_writev, +}; + +static void bdrv_rbd_init (void ) +{ + bdrv_register(&bdrv_rbd); +} + +block_init (bdrv_rbd_init);block/rbd_types.h new file +0
-0
block/rbd_types.h line 2
Quoted 2 lines
block/rbd_types.h new file +2
-0@@ -0,0 +1,48 @@ +#ifndef _FS_CEPH_RBD +#define _FS_CEPH_RBD
QEMU_BLOCK_RBD?
block/rbd_types.h line 4
Quoted 2 lines
block/rbd_types.h new file +2
-0+ +#include <linux/types.h>
Can you use standard includes, like <sys/types.h> or <inttypes.h>? Are Ceph libraries used in other systems than Linux?
Quoted 24 lines
block/rbd_types.h new file +24
-0+ +/* +* rbd image 'foo' consists of objects +* foo.rbd - image metadata +* foo.00000000 +* foo.00000001 +* ... - data +*/ + +#define RBD_SUFFIX ".rbd" +#define RBD_DIRECTORY "rbd_directory" + +#define RBD_DEFAULT_OBJ_ORDER 22/* 4MB */ + +#define RBD_MAX_OBJ_NAME_SIZE 96 +#define RBD_MAX_SEG_NAME_SIZE 128 + +#define RBD_COMP_NONE 0 +#define RBD_CRYPT_NONE 0 + +static const char rbd_text[] ="<<< Rados Block Device Image >>>\n" ; +static const char rbd_signature[] ="RBD" ; +static const char rbd_version[] ="001.001" ; +
block/rbd_types.h line 38
Quoted 10 lines
block/rbd_types.h new file +10
-0+struct rbd_obj_snap_ondisk { + __le64 id; + __le64 image_size; +} __attribute__((packed)); + +struct rbd_obj_header_ondisk { + char text[64]; + char signature[4]; + char version[8]; + __le64 image_size;
Unaligned? Is the disk format fixed?
block/rbd_types.h line 41
Quoted 3 lines
block/rbd_types.h new file +3
-0+ __u8 obj_order; + __u8 crypt_type; + __u8 comp_type;
Unaligned?
block/rbd_types.h line 44
Quoted 3 lines
block/rbd_types.h new file +3
-0+ __le32 snap_seq; + __le32 snap_count; + __le64 snap_names_len;
Unaligned?
Quoted 82 lines
block/rbd_types.h new file +4
-0+ struct rbd_obj_snap_ondisk snaps[0]; +} __attribute__((packed)); + +#endif configure +0
-0@@ -299,6 +299,7 @@ pkgversion=""check_utests="no"
user_pie="no"
zero_malloc=""
+rbd="no"# OS specific
ifcheck_define__linux__; thenconfigure +0
-0@@ -660,6 +661,8 @@ for opt do ;;--enable-vhost-net) vhost_net="yes"
;;
+ --enable-rbd) rbd="yes"
+ ;;
*) echo "ERROR: unknown option $opt"; show_help="yes" ;;
esacconfigure +0
-0@@ -826,6 +829,7 @@ echo " --enable-docs enable documentation build"echo " --disable-docs disable documentation build"
echo " --disable-vhost-net disable vhost-net acceleration support" echo " --enable-vhost-net enable vhost-net acceleration support" +echo " --enable-rbd enable building the rados block device (rbd)" echo ""
echo "NOTE: The object files are built at the place where configure is launched" exit 1configure +0
-0@@ -1569,6 +1573,25 @@ if test "$mingw32" != yes -a "$pthread" = no; thenfi
##########################################
configure +19
-0+# rbd probe +if test "$rbd" != "no" ; then + cat > $TMPC <<EOF +#include <stdio.h> +#include <rados/librados.h> +int main(void) { rados_initialize(0, NULL); return 0; } +EOF + if compile_prog "" "-lrados -lcrypto" ; then + rbd=yes + LIBS="$LIBS -lrados -lcrypto" + else + if test "$rbd" = "yes" ; then + feature_not_found "rados block device" + fi + rbd=no + fi +fi + +########################################### linux-aio probeif test "$linux_aio" != "no" ; then
configure +0
-0@@ -2031,6 +2054,7 @@ echo "preadv support $preadv"echo "fdatasync $fdatasync"
echo "uuid support $uuid"
echo "vhost-net support $vhost_net"
+echo "rbd support $rbd"if test $sdl_too_old = "yes"; then
echo "-> Your SDL version is too old - please upgrade to have SDL support"configure +0
-0@@ -2260,6 +2284,9 @@ echo "CONFIG_UNAME_RELEASE=\"$uname_release\"" >> $config_host_makif test "$zero_malloc" = "yes" ; then
echo "CONFIG_ZERO_MALLOC=y" >> $config_host_mak
ficonfigure +3
-0+if test "$rbd" = "yes" ; then + echo "CONFIG_RBD=y" >> $config_host_mak +fi# USB host support
case "$usb" in
--
1.7.0.4
2010/5/20 Blue Swirl <[email protected]>:
Quoted 10 lines · Christian Brunner
Quoted 9 lines
The attached patch is a block driver for the distributed file system Ceph (http://ceph.newdream.net/). This driver uses librados (which is part of the Ceph server) for direct access to the Ceph object store and is running entirely in userspace. Therefore it is called "rbd" - rados block device.
To compile the driver a recent version of ceph (>= 0.20.1) is needed and you have to "--enable-rbd" when running configure.
Quoted 7 lines
Quoted 3 lines
Additional information is available on the Ceph-Wiki:
I have no idea whether it makes sense to add Ceph (no objection either). I have some minor comments below.
Thanks for your comments. I'll send an updated patch in a few days.
Having a central storage system is quite essential in larger hosting environments, it enables you to move your guest systems from one node to another easily (live-migration or dynamic restart). Traditionally this has been done using SAN, iSCSI or NFS. However most of these systems don't scale very well and and the costs for high-availability are quite high.
With new approaches like Sheepdog or Ceph, things are getting a lot cheaper and you can scale your system without disrupting your service. The concepts are quite similar to what Amazon is doing in their EC2 environment, but they certainly won't publish it as OpenSource anytime soon.
Both projects have advantages and disadvantages. Ceph is a bit more universal as it implements a whole filesystem. Sheepdog is more feature complete in regards of managing images (e.g. snapshots). Both projects require some additional work to become stable, but they are on a good way.
I would really like to see both drivers in the qemu tree, as they are the key to a design shift in how storage in the datacenter is being built.
Christian
Quoted 21 lines · Christian Brunner
Thanks for your comments. I'll send an updated patch in a few days.
Having a central storage system is quite essential in larger hosting environments, it enables you to move your guest systems from one node to another easily (live-migration or dynamic restart). Traditionally this has been done using SAN, iSCSI or NFS. However most of these systems don't scale very well and and the costs for high-availability are quite high.
With new approaches like Sheepdog or Ceph, things are getting a lot cheaper and you can scale your system without disrupting your service. The concepts are quite similar to what Amazon is doing in their EC2 environment, but they certainly won't publish it as OpenSource anytime soon.
Both projects have advantages and disadvantages. Ceph is a bit more universal as it implements a whole filesystem. Sheepdog is more feature complete in regards of managing images (e.g. snapshots). Both projects require some additional work to become stable, but they are on a good way.
Quoted 4 lines
I would really like to see both drivers in the qemu tree, as they are the key to a design shift in how storage in the datacenter is being built.
I'd be more interested in enabling people to build these types of storage systems without touching qemu.
Both sheepdog and ceph ultimately transmit I/O over a socket to a central daemon, right? So could we not standardize a protocol for this that both sheepdog and ceph could implement?
Regards,
Anthony Liguori
Quoted 6 lines
Christian
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
2010/5/20 Anthony Liguori <[email protected]>:
Quoted 12 lines
Quoted 12 lines
With new approaches like Sheepdog or Ceph, things are getting a lot cheaper and you can scale your system without disrupting your service. The concepts are quite similar to what Amazon is doing in their EC2 environment, but they certainly won't publish it as OpenSource anytime soon.
Both projects have advantages and disadvantages. Ceph is a bit more universal as it implements a whole filesystem. Sheepdog is more feature complete in regards of managing images (e.g. snapshots). Both projects require some additional work to become stable, but they are on a good way.
Quoted 7 lines
Quoted 4 lines
I would really like to see both drivers in the qemu tree, as they are the key to a design shift in how storage in the datacenter is being built.
I'd be more interested in enabling people to build these types of storage systems without touching qemu.
You could do this by using Yehuda's rbd kernel driver, but I think that it would be better to avoid this additional layer.
Quoted 3 lines
Both sheepdog and ceph ultimately transmit I/O over a socket to a central daemon, right? So could we not standardize a protocol for this that both sheepdog and ceph could implement?
There is no central daemon. The concept is that they talk to many storage nodes at the same time. Data is distributed and replicated over many nodes in the network. The mechanism to do this is quite complex. I don't know about sheepdog, but in Ceph this is called RADOS (reliable autonomic distributed object store). Sheepdog and Ceph may look similar, but this is where they act different. I don't think that it would be possible to implement a common protocol.
Regards,
Christian
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 5 lines · Christian Brunner
2010/5/20 Anthony Liguori <[email protected]>:
Quoted 3 lines
Both sheepdog and ceph ultimately transmit I/O over a socket to a central daemon, right? So could we not standardize a protocol for this that both sheepdog and ceph could implement?
Quoted 7 lines
There is no central daemon. The concept is that they talk to many storage nodes at the same time. Data is distributed and replicated over many nodes in the network. The mechanism to do this is quite complex. I don't know about sheepdog, but in Ceph this is called RADOS (reliable autonomic distributed object store). Sheepdog and Ceph may look similar, but this is where they act different. I don't think that it would be possible to implement a common protocol.
I believe Sheepdog has a local daemon on each node. The QEMU storage backend talks to the daemon on the same node, which then does the real network communication with the rest of the distributed storage system.
So I think we're not talking about a network protocol here, we're
talking about a common interface that can be used by QEMU and other programs to take advantage of Ceph, Sheepdog, etc services available on the local node.
Haven't looked into your patch enough yet, but does librados talk directly over the network or does it connect to a local daemon/driver?
Stefan
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
At Fri, 21 May 2010 06:28:42 +0100,
Quoted 15 lines · Stefan Hajnoczi
Quoted 5 lines · Christian Brunner
2010/5/20 Anthony Liguori <[email protected]>:
Quoted 3 lines
Both sheepdog and ceph ultimately transmit I/O over a socket to a central daemon, right? So could we not standardize a protocol for this that both sheepdog and ceph could implement?
Quoted 7 lines
There is no central daemon. The concept is that they talk to many storage nodes at the same time. Data is distributed and replicated over many nodes in the network. The mechanism to do this is quite complex. I don't know about sheepdog, but in Ceph this is called RADOS (reliable autonomic distributed object store). Sheepdog and Ceph may look similar, but this is where they act different. I don't think that it would be possible to implement a common protocol.
Quoted 3 lines
I believe Sheepdog has a local daemon on each node. The QEMU storage backend talks to the daemon on the same node, which then does the real network communication with the rest of the distributed storage system.
Yes. It is because Sheepdog doesn't have a configuration about cluster membership as I mentioned in another mail, so the drvier doesn't know which node to access other than localhost.
Quoted 8 lines
So I think we're not talking about a network protocol here, we're
talking about a common interface that can be used by QEMU and other programs to take advantage of Ceph, Sheepdog, etc services available on the local node.Haven't looked into your patch enough yet, but does librados talk directly over the network or does it connect to a local daemon/driver?
AFAIK, librados access directly over the network, so I think it is difficult to define a common interface.
Thanks,
Kazutaka
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
At Fri, 21 May 2010 00:16:46 +0200,
Quoted 11 lines · Christian Brunner
2010/5/20 Anthony Liguori <[email protected]>:
Quoted 9 lines
Quoted 9 lines
With new approaches like Sheepdog or Ceph, things are getting a lot cheaper and you can scale your system without disrupting your service. The concepts are quite similar to what Amazon is doing in their EC2 environment, but they certainly won't publish it as OpenSource anytime soon.
Both projects have advantages and disadvantages. Ceph is a bit more universal as it implements a whole filesystem. Sheepdog is more feature complete in regards of managing images (e.g. snapshots). Both
I think a major difference is that Sheepdog servers act fully autonomously. Any Sheepdog server has no fixed role such as a monitor server, and Sheepdog doesn't require any configuration about a list of nodes in the cluster.
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
projects require some additional work to become stable, but they are on a good way.
I would really like to see both drivers in the qemu tree, as they are the key to a design shift in how storage in the datacenter is being built.
Quoted 6 lines
Quoted 2 lines
I'd be more interested in enabling people to build these types of storage systems without touching qemu.
You could do this by using Yehuda's rbd kernel driver, but I think that it would be better to avoid this additional layer.
I agree. In addition, if a storage client is a qemu driver, the storage system can support some features specific to qemu such as live snapshot from qemu monitor.
Regards,
Kazutaka
Quoted 4 lines · Anthony Liguori
I'd be more interested in enabling people to build these types of storage systems without touching qemu.
Both sheepdog and ceph ultimately transmit I/O over a socket to a central daemon, right?
That incurs an extra copy.
Quoted 1 line
So could we not standardize a protocol for this that both sheepdog and ceph could implement?
The protocol already exists, nbd. It doesn't support snapshotting etc. but we could extend it.
But IMO what's needed is a plugin API for the block layer.
Signature · 1 line
error compiling committee.c: too many arguments to function
At Sun, 23 May 2010 15:01:59 +0300,
Quoted 9 lines · Avi Kivity
Quoted 6 lines · Anthony Liguori
I'd be more interested in enabling people to build these types of storage systems without touching qemu.
Both sheepdog and ceph ultimately transmit I/O over a socket to a central daemon, right?
Quoted 8 lines
That incurs an extra copy.
Quoted 2 lines
So could we not standardize a protocol for this that both sheepdog and ceph could implement?
The protocol already exists, nbd. It doesn't support snapshotting etc. but we could extend it.
I have no objection to use another protocol for Sheepdog support, but I think nbd protocol is unsuitable for the large storage pool with many VM images. It is because nbd protocol doesn't support specifing a file name to open. If we use nbd with such a storage system, the server needs to listen ports as many as the number of VM images. As far as I see the protocol, It looks difficult to extend it without breaking backward compatibility.
Regards,
Kazutaka
Quoted 2 lines
But IMO what's needed is a plugin API for the block layer.
Quoted 21 lines · MORITA Kazutaka
At Sun, 23 May 2010 15:01:59 +0300,
Quoted 8 lines · Avi Kivity
Quoted 6 lines · Anthony Liguori
I'd be more interested in enabling people to build these types of storage systems without touching qemu.
Both sheepdog and ceph ultimately transmit I/O over a socket to a central daemon, right?
Quoted 10 lines
That incurs an extra copy.
Quoted 3 lines
So could we not standardize a protocol for this that both sheepdog and ceph could implement?
The protocol already exists, nbd. It doesn't support snapshotting etc. but we could extend it.
Quoted 8 lines
I have no objection to use another protocol for Sheepdog support, but I think nbd protocol is unsuitable for the large storage pool with many VM images. It is because nbd protocol doesn't support specifing a file name to open. If we use nbd with such a storage system, the server needs to listen ports as many as the number of VM images. As far as I see the protocol, It looks difficult to extend it without breaking backward compatibility.
The server would be local and talk over a unix domain socket, perhaps anonymous.
nbd has other issues though, such as requiring a copy and no support for metadata operations such as snapshot and file size extension.
Signature · 1 line
Do not meddle in the internals of kernels, for they are subtle and quick to panic.
At Mon, 24 May 2010 14:05:58 +0300,
Quoted 32 lines · Avi Kivity
Quoted 21 lines · MORITA Kazutaka
At Sun, 23 May 2010 15:01:59 +0300,
Quoted 8 lines · Avi Kivity
Quoted 6 lines · Anthony Liguori
I'd be more interested in enabling people to build these types of storage systems without touching qemu.
Both sheepdog and ceph ultimately transmit I/O over a socket to a central daemon, right?
Quoted 10 lines
That incurs an extra copy.
Quoted 3 lines
So could we not standardize a protocol for this that both sheepdog and ceph could implement?
The protocol already exists, nbd. It doesn't support snapshotting etc. but we could extend it.
Quoted 8 lines
I have no objection to use another protocol for Sheepdog support, but I think nbd protocol is unsuitable for the large storage pool with many VM images. It is because nbd protocol doesn't support specifing a file name to open. If we use nbd with such a storage system, the server needs to listen ports as many as the number of VM images. As far as I see the protocol, It looks difficult to extend it without breaking backward compatibility.
Quoted 6 lines
The server would be local and talk over a unix domain socket, perhaps anonymous.
nbd has other issues though, such as requiring a copy and no support for metadata operations such as snapshot and file size extension.
Sorry, my explanation was unclear. I'm not sure how running servers on localhost can solve the problem.
What I wanted to say was that we cannot specify the image of VM. With nbd protocol, command line arguments are as follows:
$ qemu nbd:hostname:port
As this syntax shows, with nbd protocol the client cannot pass the VM image name to the server.
Regards,
Kazutaka
Quoted 11 lines · MORITA Kazutaka
Quoted 8 lines
The server would be local and talk over a unix domain socket, perhaps anonymous.
nbd has other issues though, such as requiring a copy and no support for metadata operations such as snapshot and file size extension.
Sorry, my explanation was unclear. I'm not sure how running servers on localhost can solve the problem.
The local server can convert from the local (nbd) protocol to the remote (sheepdog, ceph) protocol.
Quoted 8 lines
What I wanted to say was that we cannot specify the image of VM. With nbd protocol, command line arguments are as follows:
$ qemu nbd:hostname:portAs this syntax shows, with nbd protocol the client cannot pass the VM image name to the server.
We would extend it to allow it to connect to a unix domain socket:
qemu nbd:unix:/path/to/socket
The server at the other end would associate the socket with a filename and forward it to the server using the remote protocol.
However, I don't think nbd would be a good protocol. My preference would be for a plugin API, or for a new local protocol that uses splice() to avoid copies.
Signature · 1 line
Do not meddle in the internals of kernels, for they are subtle and quick to panic.
Quoted 9 lines · On Mon, 24 May 2010 14:56:29 +0300 Avi Kivity <[email protected]> wrote:
Quoted 8 lines · MORITA Kazutaka
Quoted 8 lines
The server would be local and talk over a unix domain socket, perhaps anonymous.
nbd has other issues though, such as requiring a copy and no support for metadata operations such as snapshot and file size extension.
Quoted 7 lines
Quoted 3 lines
Sorry, my explanation was unclear. I'm not sure how running servers on localhost can solve the problem.
The local server can convert from the local (nbd) protocol to the remote (sheepdog, ceph) protocol.
Please note that this shouldn't be relevant to the block driver based
on ceph, as it does not use a local daemon -- it connects to the Object Storage Devices directly over the network.
Best regards
Cláudio
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
At Mon, 24 May 2010 14:56:29 +0300,
Quoted 26 lines · Avi Kivity
Quoted 11 lines · MORITA Kazutaka
Quoted 8 lines
The server would be local and talk over a unix domain socket, perhaps anonymous.
nbd has other issues though, such as requiring a copy and no support for metadata operations such as snapshot and file size extension.
Sorry, my explanation was unclear. I'm not sure how running servers on localhost can solve the problem.
The local server can convert from the local (nbd) protocol to the remote (sheepdog, ceph) protocol.
Quoted 8 lines
What I wanted to say was that we cannot specify the image of VM. With nbd protocol, command line arguments are as follows:
$ qemu nbd:hostname:portAs this syntax shows, with nbd protocol the client cannot pass the VM image name to the server.
Quoted 7 lines
We would extend it to allow it to connect to a unix domain socket:
qemu nbd:unix:/path/to/socket
The server at the other end would associate the socket with a filename and forward it to the server using the remote protocol.
Thank you for the explanation. Sheepdog could achieve desired behavior by creating socket files for all the VM images when the daemon starts up.
Quoted 4 lines
However, I don't think nbd would be a good protocol. My preference would be for a plugin API, or for a new local protocol that uses
splice()to avoid copies.
Both would be okay for Sheepdog. I want to take a suitable approach for qemu.
Thanks,
Kazutaka
2010/5/24 MORITA Kazutaka <[email protected]>:
Quoted 7 lines
Quoted 4 lines
However, I don't think nbd would be a good protocol. My preference would be for a plugin API, or for a new local protocol that uses
splice()to avoid copies.Both would be okay for Sheepdog. I want to take a suitable approach for qemu.
I think both should be possible:
- Using splice() we would need a daemon that is listening on a control socket for
requests from qemu-processes or admin commands. When a qemu-process wants to open an image it could call open_image("protocol:imagename") on the controll socket and the daemon has to create a pipe to which the
image is mapped.
(What I'm unsure about, are the security implications. Do we need some kind of authentication for the sockets? What about sVirt?
- Building a plugin API seems a bit simpler to me, although I'm to sure if I'd get the
idea correctly:
The block layer has already some kind of api (.bdrv_file_open, .bdrv_read). We could simply compile the block-drivers as shared objects and create a method for loading the necessary modules at runtime.
Are you planing to use this for all block drivers?
Regards,
Christian
Quoted 13 lines · Christian Brunner
2010/5/24 MORITA Kazutaka<[email protected]>:
Quoted 8 lines
Quoted 5 lines
However, I don't think nbd would be a good protocol. My preference would be for a plugin API, or for a new local protocol that uses
splice()to avoid copies.Both would be okay for Sheepdog. I want to take a suitable approach for qemu.
I think both should be possible:
Quoted 9 lines
- Using
splice()we would need a daemon that is listening on a control socket for
requests from qemu-processes or admin commands. When a qemu-process wants to open an image it could callopen_image("protocol:imagename")on the controll socket and the daemon has to create a pipe to which the
image is mapped.
(What I'm unsure about, are the security implications. Do we need some kind of authentication for the sockets? What about sVirt?
This is a fairly old patch that I dug out of a backup. It uses the 9p protocol and does proper support for AIO.
At one point in time, I actually implemented splice() support but it didn't result in a significant improvement in benchmarks.
Quoted 7 lines
- Building a plugin API seems a bit simpler to me, although I'm to sure if I'd get the
idea correctly:
The block layer has already some kind of api (.bdrv_file_open, .bdrv_read). We could simply compile the block-drivers as shared objects and create a method for loading the necessary modules at runtime.
That approach would be a recipe for disaster. We would have to introduce a new, reduced functionality block API that was supported for plugins. Otherwise, the only way a plugin could keep up with our API changes would be if it was in tree which defeats the purpose of having plugins.
Regards,
Anthony Liguori
Quoted 5 lines
Are you planing to use this for all block drivers?
Regards,
Christian
Makefile +1 -1
@@ -53,7 +53,7 @@ BLOCK_OBJS=cutils.o qemu-malloc.o BLOCK_OBJS+=block-cow.o block-qcow.o aes.o block-vmdk.o block-cloop.o BLOCK_OBJS+=block-dmg.o block-bochs.o block-vpc.o block-vvfat.o BLOCK_OBJS+=block-qcow2.o block-parallels.o block-nbd.o -BLOCK_OBJS+=nbd.o block.o aio.o +BLOCK_OBJS+=nbd.o block.o aio.o block-9p.o p9.o p9c.o ifdef CONFIG_WIN32 BLOCK_OBJS += block-raw-win32.o
block-9p.c new file +573 -0
573 lines not on this page · raw message
block.c +1 -0
@@ -1535,6 +1535,7 @@ void bdrv_init(void)bdrv_register (&bdrv_qcow2);bdrv_register (&bdrv_parallels);bdrv_register (&bdrv_nbd); +bdrv_register (&bdrv_9p); } void *qemu_aio_get(BlockDriverState *bs, BlockDriverCompletionFunc *cb,
block.h +1 -0
@@ -20,6 +20,7 @@ extern BlockDriver bdrv_vvfat;extern BlockDriver bdrv_qcow2;extern BlockDriver bdrv_parallels;extern BlockDriver bdrv_nbd; +extern BlockDriver bdrv_9p; typedef struct BlockDriverInfo { /* in bytes, 0 if irrelevant */
p9.c new file +637 -0
637 lines not on this page · raw message
p9.h new file +163 -0
163 lines not on this page · raw message
p9c.c new file +437 -0
437 lines not on this page · raw message
p9c.h new file +81 -0
81 lines not on this page · raw message
Quoted 11 lines · Anthony Liguori
Quoted 9 lines
- Building a plugin API seems a bit simpler to me, although I'm to sure if I'd get the
idea correctly:
The block layer has already some kind of api (.bdrv_file_open,
.bdrv_read). We
could simply compile the block-drivers as shared objects and
create a method
for loading the necessary modules at runtime.That approach would be a recipe for disaster. We would have to introduce a new, reduced functionality block API that was supported for plugins. Otherwise, the only way a plugin could keep up with our API changes would be if it was in tree which defeats the purpose of having plugins.
We could guarantee API/ABI stability in a stable branch but not across releases.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 11 lines · Avi Kivity
Quoted 10 lines · Anthony Liguori
Quoted 9 lines
- Building a plugin API seems a bit simpler to me, although I'm to sure if I'd get the
idea correctly:
The block layer has already some kind of api (.bdrv_file_open,
.bdrv_read). We
could simply compile the block-drivers as shared objects and
create a method
for loading the necessary modules at runtime.
Quoted 3 lines
Quoted 1 line
That approach would be a recipe for disaster. We would have to introduce a new, reduced functionality block API that was supported for plugins. Otherwise, the only way a plugin could keep up with our API changes would be if it was in tree which defeats the purpose of having plugins.
We could guarantee API/ABI stability in a stable branch but not across releases.
We have releases every six months. There would be tons of block plugins that didn't work for random sets of releases. That creates a lot of user confusion and unhappiness.
Regards,
Anthony Liguori
Quoted 12 lines · Anthony Liguori
Quoted 11 lines · Avi Kivity
Quoted 10 lines · Anthony Liguori
Quoted 9 lines
- Building a plugin API seems a bit simpler to me, although I'm to sure if I'd get the
idea correctly:
The block layer has already some kind of api (.bdrv_file_open,
.bdrv_read). We
could simply compile the block-drivers as shared objects and
create a method
for loading the necessary modules at runtime.
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
That approach would be a recipe for disaster. We would have to introduce a new, reduced functionality block API that was supported for plugins. Otherwise, the only way a plugin could keep up with our API changes would be if it was in tree which defeats the purpose of having plugins.
We could guarantee API/ABI stability in a stable branch but not across releases.
We have releases every six months. There would be tons of block plugins that didn't work for random sets of releases. That creates a lot of user confusion and unhappiness.
The current situation is that those block format drivers only exist in qemu.git or as patches. Surely that's even more unhappiness.
Confusion could be mitigated:
$ qemu -module my-fancy-block-format-driver.so my-fancy-block-format-driver.so does not support this version of qemu (0.19.2). Please contact [email protected].
The question is how many such block format drivers we expect. We now have two in the pipeline (ceph, sheepdog), it's reasonable to assume we'll want an lvm2 driver and btrfs driver. This is an area with a lot of activity and a relatively simply interface.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 21 lines · Avi Kivity
Quoted 12 lines · Anthony Liguori
Quoted 11 lines · Avi Kivity
Quoted 10 lines · Anthony Liguori
Quoted 9 lines
- Building a plugin API seems a bit simpler to me, although I'm to sure if I'd get the
idea correctly:
The block layer has already some kind of api (.bdrv_file_open,
.bdrv_read). We
could simply compile the block-drivers as shared objects and
create a method
for loading the necessary modules at runtime.Quoted 5 lines
Quoted 3 lines
Quoted 1 line
That approach would be a recipe for disaster. We would have to introduce a new, reduced functionality block API that was supported for plugins. Otherwise, the only way a plugin could keep up with our API changes would be if it was in tree which defeats the purpose of having plugins.
We could guarantee API/ABI stability in a stable branch but not across releases.
We have releases every six months. There would be tons of block plugins that didn't work for random sets of releases. That creates a lot of user confusion and unhappiness.
The current situation is that those block format drivers only exist in qemu.git or as patches. Surely that's even more unhappiness.
Quoted 7 lines
Confusion could be mitigated:
$ qemu -module my-fancy-block-format-driver.so my-fancy-block-format-driver.so does not support this version of qemu (0.19.2). Please contact [email protected].The question is how many such block format drivers we expect. We now have two in the pipeline (ceph, sheepdog), it's reasonable to assume we'll want an lvm2 driver and btrfs driver. This is an area with a lot of activity and a relatively simply interface.
If we expose a simple interface, I'm all for it. But BlockDriver is not simple and things like the snapshoting API need love.
Of course, there's certainly a question of why we're solving this in qemu at all. Wouldn't it be more appropriate to either (1) implement a kernel module for ceph/sheepdog if performance matters or (2) implement BUSE to complement FUSE and CUSE to enable proper userspace block devices.
If you want to use a block device within qemu, you almost certainly want to be able to manipulate it on the host using standard tools (like mount and parted) so it stands to reason that addressing this in the kernel makes more sense.
Regards,
Anthony Liguori
Quoted 8 lines · Anthony Liguori
Quoted 8 lines
The current situation is that those block format drivers only exist in qemu.git or as patches. Surely that's even more unhappiness.
Confusion could be mitigated:
$ qemu -module my-fancy-block-format-driver.so my-fancy-block-format-driver.so does not support this version of qemu (0.19.2). Please contact [email protected].
Quoted 6 lines
Quoted 1 line
The question is how many such block format drivers we expect. We now have two in the pipeline (ceph, sheepdog), it's reasonable to assume we'll want an lvm2 driver and btrfs driver. This is an area with a lot of activity and a relatively simply interface.
If we expose a simple interface, I'm all for it. But BlockDriver is not simple and things like the snapshoting API need love.
Of course, there's certainly a question of why we're solving this in qemu at all. Wouldn't it be more appropriate to either (1) implement a kernel module for ceph/sheepdog if performance matters
We'd need a kernel-level generic snapshot API for this eventually.
Quoted 1 line
or (2) implement BUSE to complement FUSE and CUSE to enable proper userspace block devices.
Likely slow due do lots of copying. Also needs a snapshot API.
(ABUSE was proposed a while ago by Zach).
Quoted 1 line
If you want to use a block device within qemu, you almost certainly want to be able to manipulate it on the host using standard tools (like mount and parted) so it stands to reason that addressing this in the kernel makes more sense.
qemu-nbd also allows this.
This reasoning also applies to qcow2, btw.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 6 lines · Avi Kivity
We'd need a kernel-level generic snapshot API for this eventually.
Quoted 1 line
or (2) implement BUSE to complement FUSE and CUSE to enable proper userspace block devices.
Likely slow due do lots of copying. Also needs a snapshot API.
The kernel could use splice.
Quoted 7 lines
(ABUSE was proposed a while ago by Zach).
Quoted 1 line
If you want to use a block device within qemu, you almost certainly want to be able to manipulate it on the host using standard tools (like mount and parted) so it stands to reason that addressing this in the kernel makes more sense.
qemu-nbd also allows this.
This reasoning also applies to qcow2, btw.
I know.
Regards,
Anthony Liguori
Quoted 9 lines · Anthony Liguori
Quoted 6 lines · Avi Kivity
We'd need a kernel-level generic snapshot API for this eventually.
Quoted 1 line
or (2) implement BUSE to complement FUSE and CUSE to enable proper userspace block devices.
Likely slow due do lots of copying. Also needs a snapshot API.
The kernel could use splice.
Still can't make guest memory appear in (A)BUSE process memory without either mmu tricks (vmsplice in reverse) or a copy. May be workable for an (A)BUSE driver that talks over a network, and thus can splice() its way out.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 5 lines · Avi Kivity
Quoted 4 lines · Anthony Liguori
Quoted 3 lines · Avi Kivity
We'd need a kernel-level generic snapshot API for this eventually.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
or (2) implement BUSE to complement FUSE and CUSE to enable proper userspace block devices.
Likely slow due do lots of copying. Also needs a snapshot API.
The kernel could use splice.
Still can't make guest memory appear in (A)BUSE process memory without either mmu tricks (vmsplice in reverse) or a copy. May be workable for an (A)BUSE driver that talks over a network, and thus can
splice()its way out.
splice() actually takes offset parameter so it may be possible to treat that offset parameter as a file offset. That would essentially allow you to implement a splice() based thread pool where splice() replaces preadv/pwritev.
It's not quite linux-aio, but it should take you pretty far. I think the main point is that the problem of allowing block plugins to qemu is the same as block plugins for the kernel. The kernel doesn't provide a stable interface (and we probably can't for the same reasons) and it's generally discourage from a code quality perspective.
That said, making an external program work well as a block backend is identical to making userspace block devices fast.
Regards,
Anthony Liguori
Quoted 8 lines · Anthony Liguori
Quoted 7 lines · Avi Kivity
Quoted 6 lines · Anthony Liguori
Quoted 5 lines · Avi Kivity
We'd need a kernel-level generic snapshot API for this eventually.
Quoted 1 line
or (2) implement BUSE to complement FUSE and CUSE to enable proper userspace block devices.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
Likely slow due do lots of copying. Also needs a snapshot API.
The kernel could use splice.
Still can't make guest memory appear in (A)BUSE process memory without either mmu tricks (vmsplice in reverse) or a copy. May be workable for an (A)BUSE driver that talks over a network, and thus can
splice()its way out.
splice()actually takes offset parameter so it may be possible to treat that offset parameter as a file offset. That would essentially allow you to implement asplice()based thread pool wheresplice()replaces preadv/pwritev.
Right.
(note: need splicev() here)
Quoted 2 lines
It's not quite linux-aio, but it should take you pretty far. I think the main point is that the problem of allowing block plugins to qemu is the same as block plugins for the kernel. The kernel doesn't provide a stable interface (and we probably can't for the same reasons) and it's generally discourage from a code quality perspective.
The kernel does provide a stable interface for FUSE, and it could provide a stable interface for ABUSE. Why can the kernel support these and qemu can't support essentially the same thing?
Quoted 1 line
That said, making an external program work well as a block backend is identical to making userspace block devices fast.
More or less, yes.
Signature · 1 line
error compiling committee.c: too many arguments to function
Am 25.05.2010 15:25, schrieb Avi Kivity:
Quoted 22 lines · Anthony Liguori
Quoted 21 lines · Avi Kivity
Quoted 19 lines · Anthony Liguori
Quoted 10 lines
Quoted 9 lines
- Building a plugin API seems a bit simpler to me, although I'm to sure if I'd get the
idea correctly:
The block layer has already some kind of api (.bdrv_file_open,
.bdrv_read). We
could simply compile the block-drivers as shared objects and
create a method
for loading the necessary modules at runtime.Quoted 5 lines
That approach would be a recipe for disaster. We would have to
introduce a new, reduced functionality block API that was supported for plugins. Otherwise, the only way a plugin could keep up with our API changes would be if it was in tree which defeats the purpose of having plugins.We could guarantee API/ABI stability in a stable branch but not across releases.
Quoted 6 lines
Quoted 3 lines
We have releases every six months. There would be tons of block plugins that didn't work for random sets of releases. That creates a lot of user confusion and unhappiness.
The current situation is that those block format drivers only exist in qemu.git or as patches. Surely that's even more unhappiness.
The difference is that in the current situation these drivers will be part of the next qemu release, so the patch may be obsolete, but you don't even need it any more.
If you start keeping block drivers outside qemu and not even try integrating them, they'll stay external.
Quoted 10 lines
Confusion could be mitigated:
$ qemu -module my-fancy-block-format-driver.so my-fancy-block-format-driver.so does not support this version of qemu (0.19.2). Please contact [email protected].The question is how many such block format drivers we expect. We now have two in the pipeline (ceph, sheepdog), it's reasonable to assume we'll want an lvm2 driver and btrfs driver. This is an area with a lot of activity and a relatively simply interface.
What's the reason for not having these drivers upstream? Do we gain anything by hiding them from our users and requiring them to install the drivers separately from somewhere else?
Kevin
Quoted 8 lines · Kevin Wolf
Quoted 4 lines
The current situation is that those block format drivers only exist in qemu.git or as patches. Surely that's even more unhappiness.
The difference is that in the current situation these drivers will be part of the next qemu release, so the patch may be obsolete, but you don't even need it any more.
The next qemu release may be six months in the future. So if you're not happy with running qemu.git master or with patching a stable release, you have to wait.
Quoted 3 lines
If you start keeping block drivers outside qemu and not even try integrating them, they'll stay external.
Which may or may not be a problem.
Quoted 6 lines
Quoted 6 lines
Confusion could be mitigated:
$ qemu -module my-fancy-block-format-driver.so my-fancy-block-format-driver.so does not support this version of qemu (0.19.2). Please contact [email protected].
Quoted 9 lines
Quoted 5 lines
The question is how many such block format drivers we expect. We now have two in the pipeline (ceph, sheepdog), it's reasonable to assume we'll want an lvm2 driver and btrfs driver. This is an area with a lot of activity and a relatively simply interface.
What's the reason for not having these drivers upstream? Do we gain anything by hiding them from our users and requiring them to install the drivers separately from somewhere else?
Six months.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 6 lines · Avi Kivity
Quoted 4 lines
What's the reason for not having these drivers upstream? Do we gain anything by hiding them from our users and requiring them to install the drivers separately from somewhere else?
Six months.
FWIW, we (Ceph) aren't complaining about the 6 month lag time (and I don't think the Sheepdog guys are either).
From our perspective, the current BlockDriver abstraction is ideal, as it represents the reality of qemu's interaction with storage. Any 'external' interface will be inferior to that in one way or another. But either way, we are perfectly willing to work with you to all to keep in sync with any future BlockDriver API improvements. It is worth our time investment even if the API is less stable.
The ability to dynamically load a shared object using the existing api would make development a bit easier, but I'm not convinced it's better for for users. I think having ceph and sheepdog upstream with qemu will serve end users best, and we at least are willing to spend the time to help maintain that code in qemu.git.
sage
At Tue, 25 May 2010 10:12:53 -0700 (PDT),
Quoted 7 lines · Sage Weil
Quoted 5 lines · Avi Kivity
Quoted 4 lines
What's the reason for not having these drivers upstream? Do we gain anything by hiding them from our users and requiring them to install the drivers separately from somewhere else?
Quoted 5 lines
Quoted 1 line
Six months.
FWIW, we (Ceph) aren't complaining about the 6 month lag time (and I don't think the Sheepdog guys are either).
I agree. We aren't complaining about it.
Quoted 7 lines
From our perspective, the current BlockDriver abstraction is ideal, as it represents the reality of qemu's interaction with storage. Any 'external' interface will be inferior to that in one way or another. But either way, we are perfectly willing to work with you to all to keep in sync with any future BlockDriver API improvements. It is worth our time investment even if the API is less stable.
I agree.
Quoted 6 lines
The ability to dynamically load a shared object using the existing api would make development a bit easier, but I'm not convinced it's better for for users. I think having ceph and sheepdog upstream with qemu will serve end users best, and we at least are willing to spend the time to help maintain that code in qemu.git.
I agree.
Regards,
Kazutaka
Quoted 7 lines · Sage Weil
Quoted 5 lines · Avi Kivity
Quoted 5 lines
What's the reason for not having these drivers upstream? Do we gain anything by hiding them from our users and requiring them to install the drivers separately from somewhere else?
Quoted 6 lines
Quoted 2 lines
Six months.
FWIW, we (Ceph) aren't complaining about the 6 month lag time (and I don't think the Sheepdog guys are either).
In that case (and if there are no other potential users), then there's no need for a plugin API.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 16 lines · Avi Kivity
Quoted 9 lines · MORITA Kazutaka
Quoted 7 lines
The server would be local and talk over a unix domain socket, perhaps anonymous.
nbd has other issues though, such as requiring a copy and no support for metadata operations such as snapshot and file size extension.
Sorry, my explanation was unclear. I'm not sure how running servers on localhost can solve the problem.
The local server can convert from the local (nbd) protocol to the remote (sheepdog, ceph) protocol.
Quoted 3 lines
What I wanted to say was that we cannot specify the image of VM. With nbd protocol, command line arguments are as follows:
Quoted 8 lines
Quoted 4 lines
$ qemu nbd:hostname:portAs this syntax shows, with nbd protocol the client cannot pass the VM image name to the server.
We would extend it to allow it to connect to a unix domain socket:
qemu nbd:unix:/path/to/socket
nbd is a no-go because it only supports a single, synchronous I/O operation at a time and has no mechanism for extensibility.
If we go this route, I think two options are worth considering. The first would be a purely socket based approach where we just accepted the extra copy.
The other potential approach would be shared memory based. We export all guest ram as shared memory along with a small bounce buffer pool. We would then use a ring queue (potentially even using virtio-blk) and an eventfd for notification.
Quoted 3 lines
The server at the other end would associate the socket with a filename and forward it to the server using the remote protocol.
However, I don't think nbd would be a good protocol. My preference would be for a plugin API, or for a new local protocol that uses
splice()to avoid copies.
I think a good shared memory implementation would be preferable to plugins. I think it's worth attempting to do a plugin interface for the block layer but I strongly suspect it would not be sufficient.
I would not want to see plugins that interacted with BlockDriverState directly, for instance. We change it far too often. Our main loop functions are also not terribly stable so I'm not sure how we would handle that (unless we forced all block plugins to be in a separate thread).
Regards,
Anthony Liguori
Quoted 24 lines · Anthony Liguori
Quoted 23 lines · Avi Kivity
Quoted 9 lines · MORITA Kazutaka
Quoted 7 lines
The server would be local and talk over a unix domain socket, perhaps anonymous.
nbd has other issues though, such as requiring a copy and no support for metadata operations such as snapshot and file size extension.
Sorry, my explanation was unclear. I'm not sure how running servers on localhost can solve the problem.
The local server can convert from the local (nbd) protocol to the remote (sheepdog, ceph) protocol.
Quoted 7 lines
What I wanted to say was that we cannot specify the image of VM. With nbd protocol, command line arguments are as follows:
$ qemu nbd:hostname:portAs this syntax shows, with nbd protocol the client cannot pass the VM image name to the server.
We would extend it to allow it to connect to a unix domain socket:
Quoted 7 lines
Quoted 1 line
qemu nbd:unix:/path/to/socket
nbd is a no-go because it only supports a single, synchronous I/O operation at a time and has no mechanism for extensibility.
If we go this route, I think two options are worth considering. The first would be a purely socket based approach where we just accepted the extra copy.
The other potential approach would be shared memory based. We export all guest ram as shared memory along with a small bounce buffer pool. We would then use a ring queue (potentially even using virtio-blk) and an eventfd for notification.
We can't actually export guest memory unless we allocate it as a shared memory object, which has many disadvantages. The only way to export anonymous memory now is vmsplice(), which is fairly limited.
Quoted 8 lines
Quoted 4 lines
The server at the other end would associate the socket with a filename and forward it to the server using the remote protocol.
However, I don't think nbd would be a good protocol. My preference would be for a plugin API, or for a new local protocol that uses
splice()to avoid copies.I think a good shared memory implementation would be preferable to plugins. I think it's worth attempting to do a plugin interface for the block layer but I strongly suspect it would not be sufficient.
I would not want to see plugins that interacted with BlockDriverState directly, for instance. We change it far too often. Our main loop functions are also not terribly stable so I'm not sure how we would handle that (unless we forced all block plugins to be in a separate thread).
If we manage to make a good long-term stable plugin API, it would be a good candidate for the block layer itself.
Some OSes manage to have a stable block driver ABI, so it should be possible, if difficult.
Signature · 1 line
error compiling committee.c: too many arguments to function
At Mon, 24 May 2010 14:16:32 -0500,
Quoted 32 lines · Anthony Liguori
Quoted 18 lines · Avi Kivity
Quoted 10 lines · MORITA Kazutaka
Quoted 8 lines
The server would be local and talk over a unix domain socket, perhaps anonymous.
nbd has other issues though, such as requiring a copy and no support for
metadata operations such as snapshot and file size extension.Sorry, my explanation was unclear. I'm not sure how running servers on localhost can solve the problem.
The local server can convert from the local (nbd) protocol to the remote (sheepdog, ceph) protocol.
Quoted 3 lines
What I wanted to say was that we cannot specify the image of VM. With nbd protocol, command line arguments are as follows:
Quoted 8 lines
Quoted 4 lines
$ qemu nbd:hostname:portAs this syntax shows, with nbd protocol the client cannot pass the VM image name to the server.
We would extend it to allow it to connect to a unix domain socket:
qemu nbd:unix:/path/to/socket
nbd is a no-go because it only supports a single, synchronous I/O operation at a time and has no mechanism for extensibility.
Quoted 9 lines
If we go this route, I think two options are worth considering. The first would be a purely socket based approach where we just accepted the extra copy.
The other potential approach would be shared memory based. We export all guest ram as shared memory along with a small bounce buffer pool. We would then use a ring queue (potentially even using virtio-blk) and an eventfd for notification.
The shared memory approach assumes that there is a local server who can talk with the storage system. But Ceph doesn't require the local server, and Sheepdog would be extended to support VMs running outside the storage system. We could run a local daemon who can only work as proxy, but I don't think it looks a clean approach. So I think a socket based approach is the right way to go.
BTW, is it required to design a common interface? The way Sheepdog replicates data is different from Ceph, so I think it is not possible to define a common protocol as Christian says.
Regards,
Kazutaka
Quoted 16 lines
Quoted 6 lines
The server at the other end would associate the socket with a filename and forward it to the server using the remote protocol.
However, I don't think nbd would be a good protocol. My preference would be for a plugin API, or for a new local protocol that uses
splice()to avoid copies.I think a good shared memory implementation would be preferable to plugins. I think it's worth attempting to do a plugin interface for the block layer but I strongly suspect it would not be sufficient.
I would not want to see plugins that interacted with BlockDriverState directly, for instance. We change it far too often. Our main loop functions are also not terribly stable so I'm not sure how we would handle that (unless we forced all block plugins to be in a separate thread).
Quoted 9 lines · Avi Kivity
Quoted 6 lines · Anthony Liguori
I'd be more interested in enabling people to build these types of storage systems without touching qemu.
Both sheepdog and ceph ultimately transmit I/O over a socket to a central daemon, right?
That incurs an extra copy.
Besides a shared memory approach, I wonder if the splice() family of syscalls could be used to send/receive data through a storage daemon without the daemon looking at or copying the data?
Stefan
Quoted 7 lines · Stefan Hajnoczi
Quoted 5 lines · Avi Kivity
Quoted 3 lines · Anthony Liguori
I'd be more interested in enabling people to build these types of storage systems without touching qemu.
Quoted 9 lines
Quoted 5 lines
Quoted 3 lines
Both sheepdog and ceph ultimately transmit I/O over a socket to a central daemon, right?
That incurs an extra copy.
Besides a shared memory approach, I wonder if the
splice()family of syscalls could be used to send/receive data through a storage daemon without the daemon looking at or copying the data?
Excellent idea.
Signature · 1 line
Do not meddle in the internals of kernels, for they are subtle and quick to panic.
Quoted 6 lines · Avi Kivity
Quoted 5 lines · Stefan Hajnoczi
Quoted 4 lines · Avi Kivity
Quoted 3 lines · Anthony Liguori
I'd be more interested in enabling people to build these types of storage systems without touching qemu.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Quoted 2 lines
Both sheepdog and ceph ultimately transmit I/O over a socket to a central daemon, right?
That incurs an extra copy.
Besides a shared memory approach, I wonder if the
splice()family of syscalls could be used to send/receive data through a storage daemon without the daemon looking at or copying the data?Excellent idea.
splice() eventually requires a copy. You cannot splice() to linux-aio so you'd have to splice() to a temporary buffer and then call into linux-aio. With shared memory, you can avoid ever bringing the data into memory via O_DIRECT and linux-aio.
Regards,
Anthony Liguori
Quoted 9 lines · Anthony Liguori
Quoted 8 lines · Avi Kivity
Quoted 7 lines · Stefan Hajnoczi
Quoted 6 lines · Avi Kivity
Quoted 5 lines · Anthony Liguori
I'd be more interested in enabling people to build these types of storage systems without touching qemu.
Both sheepdog and ceph ultimately transmit I/O over a socket to a central daemon, right?
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
That incurs an extra copy.
Besides a shared memory approach, I wonder if the
splice()family of syscalls could be used to send/receive data through a storage daemon without the daemon looking at or copying the data?Excellent idea.
splice()eventually requires a copy. You cannotsplice()to linux-aio so you'd have tosplice()to a temporary buffer and then call into linux-aio. With shared memory, you can avoid ever bringing the data into memory viaO_DIRECTand linux-aio.
If the final destination is a socket, then you end up queuing guest memory as an skbuff. In theory we could do an aio splice to block devices but I don't think that's realistic given our experience with aio changes.
Signature · 1 line
error compiling committee.c: too many arguments to function
Am 23.05.2010 14:01, schrieb Avi Kivity:
Quoted 10 lines · Anthony Liguori
Quoted 6 lines
I'd be more interested in enabling people to build these types of storage systems without touching qemu.
Both sheepdog and ceph ultimately transmit I/O over a socket to a central daemon, right?
That incurs an extra copy.
Quoted 7 lines
Quoted 2 lines
So could we not standardize a protocol for this that both sheepdog and ceph could implement?
The protocol already exists, nbd. It doesn't support snapshotting etc. but we could extend it.
But IMO what's needed is a plugin API for the block layer.
What would it buy us, apart from more downstreams and having to maintain a stable API and ABI? Hiding block drivers somewhere else doesn't make them stop existing, they just might not be properly integrated, but rather hacked in to fit that limited stable API.
Kevin
Quoted 5 lines · Kevin Wolf
Quoted 5 lines
Quoted 5 lines
So could we not standardize a protocol for this that both sheepdog and ceph could implement?
Quoted 7 lines
Quoted 5 lines
The protocol already exists, nbd. It doesn't support snapshotting etc. but we could extend it.
But IMO what's needed is a plugin API for the block layer.
What would it buy us, apart from more downstreams and having to maintain a stable API and ABI?
Currently if someone wants to add a new block format, they have to upstream it and wait for a new qemu to be released. With a plugin API, they can add a new block format to an existing, supported qemu.
Quoted 4 lines
Hiding block drivers somewhere else doesn't make
them stop existing, they just might not be properly integrated, but rather hacked in to fit that limited stable API.
They would hack it to fit the current API, and hack the API in qemu.git to fit their requirements for the next release.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 3 lines · Avi Kivity
Currently if someone wants to add a new block format, they have to upstream it and wait for a new qemu to be released. With a plugin API, they can add a new block format to an existing, supported qemu.
So? Unless we want a stable driver ABI which I fundamentally oppose as it would make block driver development hell they'd have to wait for a new release of the block layer. It's really just going to be a lot of pain for no major gain. qemu releases are frequent enough, and if users care enough they can also easily patch qemu.
Quoted 8 lines · Christoph Hellwig
Quoted 4 lines · Avi Kivity
Currently if someone wants to add a new block format, they have to upstream it and wait for a new qemu to be released. With a plugin API, they can add a new block format to an existing, supported qemu.
So? Unless we want a stable driver ABI which I fundamentally oppose as it would make block driver development hell
We'd only freeze it for a major release.
Quoted 5 lines
they'd have to wait for
a new release of the block layer. It's really just going to be a lot of pain for no major gain. qemu releases are frequent enough, and if users care enough they can also easily patch qemu.
May not be so easy for them, they lose binary updates from their distro and have to keep repatching.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 5 lines · Avi Kivity
Quoted 4 lines · Kevin Wolf
Quoted 4 lines
Quoted 4 lines
So could we not standardize a protocol for this that both sheepdog and ceph could implement?
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
The protocol already exists, nbd. It doesn't support snapshotting etc. but we could extend it.
But IMO what's needed is a plugin API for the block layer.
What would it buy us, apart from more downstreams and having to maintain a stable API and ABI?
Currently if someone wants to add a new block format, they have to upstream it and wait for a new qemu to be released. With a plugin API, they can add a new block format to an existing, supported qemu.
Whether we have a plugin or protocol based mechanism to implement block formats really ends up being just an implementation detail.
In order to implement either, we need to take a subset of block functionality that we feel we can support long term and expose that. Right now, that's basically just querying characteristics (like size and geometry) and asynchronous reads and writes.
A protocol based mechanism has the advantage of being more robust in the face of poorly written block backends so if it's possible to make it perform as well as a plugin, it's a preferable approach.
Plugins that just expose chunks of QEMU internal state directly (like BlockDriver) are a really bad idea IMHO.
Regards,
Anthony Liguori
Quoted 4 lines · Anthony Liguori
Quoted 1 line
Currently if someone wants to add a new block format, they have to upstream it and wait for a new qemu to be released. With a plugin API, they can add a new block format to an existing, supported qemu.
Whether we have a plugin or protocol based mechanism to implement block formats really ends up being just an implementation detail.
True.
Quoted 1 line
In order to implement either, we need to take a subset of block functionality that we feel we can support long term and expose that. Right now, that's basically just querying characteristics (like size and geometry) and asynchronous reads and writes.
Unfortunately, you're right.
Quoted 1 line
A protocol based mechanism has the advantage of being more robust in the face of poorly written block backends so if it's possible to make it perform as well as a plugin, it's a preferable approach.
May be hard due to difficulty of exposing guest memory.
Quoted 2 lines
Plugins that just expose chunks of QEMU internal state directly (like BlockDriver) are a really bad idea IMHO.
Also, we don't want to expose all of the qemu API. We should default the visibility attribute to "hidden" and expose only select functions, perhaps under their own interface. And no inlines.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 3 lines · Avi Kivity
Quoted 1 line
A protocol based mechanism has the advantage of being more robust in the face of poorly written block backends so if it's possible to make it perform as well as a plugin, it's a preferable approach.
May be hard due to difficulty of exposing guest memory.
If someone did a series to add plugins, I would expect a very strong argument as to why a shared memory mechanism was not possible or at least plausible.
I'm not sure I understand why shared memory is such a bad thing wrt KVM. Can you elaborate? Is it simply a matter of fork()?
Quoted 4 lines
Quoted 2 lines
Plugins that just expose chunks of QEMU internal state directly (like BlockDriver) are a really bad idea IMHO.
Also, we don't want to expose all of the qemu API. We should default the visibility attribute to "hidden" and expose only select functions, perhaps under their own interface. And no inlines.
Yeah, if we did plugins, this would be a key requirement.
Regards,
Anthony Liguori
Quoted 8 lines · Anthony Liguori
Quoted 3 lines · Avi Kivity
Quoted 1 line
A protocol based mechanism has the advantage of being more robust in the face of poorly written block backends so if it's possible to make it perform as well as a plugin, it's a preferable approach.
May be hard due to difficulty of exposing guest memory.
If someone did a series to add plugins, I would expect a very strong argument as to why a shared memory mechanism was not possible or at least plausible.
I'm not sure I understand why shared memory is such a bad thing wrt KVM. Can you elaborate? Is it simply a matter of
fork()?
fork() doesn't work in the with of memory hotplug. What else is there?
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 4 lines · Avi Kivity
Quoted 3 lines · Anthony Liguori
Quoted 2 lines · Avi Kivity
Quoted 1 line
A protocol based mechanism has the advantage of being more robust in the face of poorly written block backends so if it's possible to make it perform as well as a plugin, it's a preferable approach.
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
May be hard due to difficulty of exposing guest memory.
If someone did a series to add plugins, I would expect a very strong argument as to why a shared memory mechanism was not possible or at least plausible.
I'm not sure I understand why shared memory is such a bad thing wrt KVM. Can you elaborate? Is it simply a matter of
fork()?
fork()doesn't work in the with of memory hotplug. What else is there?
Is it that fork() doesn't work or is it that fork() is very expensive?
Regards,
Anthony Liguori
Quoted 7 lines · Anthony Liguori
Quoted 6 lines · Avi Kivity
Quoted 5 lines · Anthony Liguori
Quoted 3 lines · Avi Kivity
Quoted 1 line
A protocol based mechanism has the advantage of being more robust in the face of poorly written block backends so if it's possible to make it perform as well as a plugin, it's a preferable approach.
May be hard due to difficulty of exposing guest memory.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
If someone did a series to add plugins, I would expect a very strong argument as to why a shared memory mechanism was not possible or at least plausible.
I'm not sure I understand why shared memory is such a bad thing wrt KVM. Can you elaborate? Is it simply a matter of
fork()?
fork()doesn't work in the with of memory hotplug. What else is there?Is it that
fork()doesn't work or is it thatfork()is very expensive?
It doesn't work, fork() is done at block device creation time, which freezes the child memory map, while guest memory is allocated at hotplug time.
fork() actually isn't very expensive since we use MADV_DONTFORK (probably fast enough for everything except realtime).
It may be possible to do a processfd() which can be mmap()ed by another process to export anonymous memory using mmu notifiers, not sure how easy or mergeable that is.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 10 lines · Avi Kivity
Quoted 9 lines · Anthony Liguori
Quoted 8 lines · Avi Kivity
Quoted 7 lines · Anthony Liguori
Quoted 3 lines · Avi Kivity
Quoted 1 line
A protocol based mechanism has the advantage of being more robust in the face of poorly written block backends so if it's possible to make it perform as well as a plugin, it's a preferable approach.
May be hard due to difficulty of exposing guest memory.
If someone did a series to add plugins, I would expect a very strong argument as to why a shared memory mechanism was not possible or at least plausible.
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
I'm not sure I understand why shared memory is such a bad thing wrt KVM. Can you elaborate? Is it simply a matter of
fork()?
fork()doesn't work in the with of memory hotplug. What else is there?Is it that
fork()doesn't work or is it thatfork()is very expensive?It doesn't work,
fork()is done at block device creation time, which freezes the child memory map, while guest memory is allocated at hotplug time.
Now I'm confused. I thought you were saying shared memory somehow affects fork(). If you're talking about shared memory inheritance via fork(), that's less important. You can also pass /dev/shm fd's via SCM_RIGHTs to establish shared memory segments dynamically.
Regards,
Anthony Liguori
Quoted 4 lines
fork()actually isn't very expensive since we useMADV_DONTFORK(probably fast enough for everything except realtime).It may be possible to do a
processfd()which can be mmap()ed by another process to export anonymous memory using mmu notifiers, not sure how easy or mergeable that is.
Quoted 13 lines · Anthony Liguori
Quoted 12 lines · Avi Kivity
Quoted 11 lines · Anthony Liguori
Quoted 10 lines · Avi Kivity
Quoted 8 lines · Anthony Liguori
Quoted 3 lines · Avi Kivity
Quoted 1 line
A protocol based mechanism has the advantage of being more robust in the face of poorly written block backends so if it's possible to make it perform as well as a plugin, it's a preferable approach.
May be hard due to difficulty of exposing guest memory.
If someone did a series to add plugins, I would expect a very strong argument as to why a shared memory mechanism was not possible or at least plausible.
I'm not sure I understand why shared memory is such a bad thing wrt KVM. Can you elaborate? Is it simply a matter of
fork()?
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
fork()doesn't work in the with of memory hotplug. What else is there?Is it that
fork()doesn't work or is it thatfork()is very expensive?It doesn't work,
fork()is done at block device creation time, which freezes the child memory map, while guest memory is allocated at hotplug time.Now I'm confused. I thought you were saying shared memory somehow affects
fork(). If you're talking about shared memory inheritance viafork(), that's less important.
The latter. Why is it less important? If you don't inherit the memory, you can't access it.
Quoted 1 line
You can also pass
/dev/shmfd's viaSCM_RIGHTsto establish shared memory segments dynamically.
Doesn't work for anonymous memory.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 5 lines · Avi Kivity
The latter. Why is it less important? If you don't inherit the memory, you can't access it.
Quoted 1 line
You can also pass
/dev/shmfd's viaSCM_RIGHTsto establish shared memory segments dynamically.Doesn't work for anonymous memory.
What's wrong with /dev/shm memory?
Regards,
Anthony Liguori
Quoted 8 lines · Anthony Liguori
Quoted 5 lines · Avi Kivity
The latter. Why is it less important? If you don't inherit the memory, you can't access it.
Quoted 1 line
You can also pass
/dev/shmfd's viaSCM_RIGHTsto establish shared memory segments dynamically.Doesn't work for anonymous memory.
What's wrong with
/dev/shmmemory?
The kernel treats anonymous and nonymous memory differently for swapping (see /proc/sys/vm/swappiness); transparent hugepages won't work for /dev/shm (though it may be argued that that's a problem with thp); setup (/dev/shm defaults to half memory IIRC, we want mem+swap); different cgroup handling; somewhat clunky (a minor concern to be sure).
Nothing is a killer, but we should prefer anonymous memory.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 11 lines · Avi Kivity
Quoted 8 lines · Anthony Liguori
Quoted 5 lines · Avi Kivity
The latter. Why is it less important? If you don't inherit the memory, you can't access it.
Quoted 1 line
You can also pass
/dev/shmfd's viaSCM_RIGHTsto establish shared memory segments dynamically.Doesn't work for anonymous memory.
What's wrong with
/dev/shmmemory?The kernel treats anonymous and nonymous memory differently for swapping (see
/proc/sys/vm/swappiness); transparent hugepages won't work for/dev/shm(though it may be argued that that's a problem with thp); setup (/dev/shmdefaults to half memory IIRC, we want mem+swap); different cgroup handling; somewhat clunky (a minor concern to be sure).
Surely, with mmu notifiers, it wouldn't be that hard to share anonymous memory via an fd though, no?
Regards,
Anthony Liguori
Quoted 3 lines
Nothing is a killer, but we should prefer anonymous memory.
Quoted 7 lines · Anthony Liguori
Quoted 6 lines · Avi Kivity
Quoted 5 lines · Anthony Liguori
Quoted 4 lines · Avi Kivity
The latter. Why is it less important? If you don't inherit the memory, you can't access it.
Quoted 1 line
You can also pass
/dev/shmfd's viaSCM_RIGHTsto establish shared memory segments dynamically.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
Doesn't work for anonymous memory.
What's wrong with
/dev/shmmemory?The kernel treats anonymous and nonymous memory differently for swapping (see
/proc/sys/vm/swappiness); transparent hugepages won't work for/dev/shm(though it may be argued that that's a problem with thp); setup (/dev/shmdefaults to half memory IIRC, we want mem+swap); different cgroup handling; somewhat clunky (a minor concern to be sure).Surely, with mmu notifiers, it wouldn't be that hard to share anonymous memory via an fd though, no?
That's what I suggested with processfd(). I wouldn't call it easy but it's likely doable. Whether it's mergable is a different issue.
Signature · 1 line
error compiling committee.c: too many arguments to function
Am 25.05.2010 15:25, schrieb Anthony Liguori:
Quoted 25 lines · Avi Kivity
Quoted 8 lines · Kevin Wolf
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
So could we not standardize a protocol for this that both sheepdog and ceph could implement?
The protocol already exists, nbd. It doesn't support snapshotting etc. but we could extend it.
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
But IMO what's needed is a plugin API for the block layer.
What would it buy us, apart from more downstreams and having to maintain a stable API and ABI?
Currently if someone wants to add a new block format, they have to upstream it and wait for a new qemu to be released. With a plugin API, they can add a new block format to an existing, supported qemu.
Whether we have a plugin or protocol based mechanism to implement block formats really ends up being just an implementation detail.
In order to implement either, we need to take a subset of block functionality that we feel we can support long term and expose that. Right now, that's basically just querying characteristics (like size and geometry) and asynchronous reads and writes.
Quoted 6 lines
A protocol based mechanism has the advantage of being more robust in the face of poorly written block backends so if it's possible to make it perform as well as a plugin, it's a preferable approach.
Plugins that just expose chunks of QEMU internal state directly (like BlockDriver) are a really bad idea IMHO.
I'm still not convinced that we need either. I share Christoph's concern that we would make our life harder for almost no gain. It's probably a very small group of users (if it exists at all) that wants to add new block drivers themselves, but at the same time can't run upstream qemu.
But if we were to decide that there's no way around it, I agree with you that directly exposing the internal API isn't going to work.
Kevin
Quoted 7 lines · Kevin Wolf
I'm still not convinced that we need either. I share Christoph's concern that we would make our life harder for almost no gain. It's probably a very small group of users (if it exists at all) that wants to add new block drivers themselves, but at the same time can't run upstream qemu.
The first part of your argument may be true, but the second isn't. No user can run upstream qemu.git. It's not tested or supported, and has no backwards compatibility guarantees.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 9 lines · Avi Kivity
Quoted 6 lines · Kevin Wolf
I'm still not convinced that we need either. I share Christoph's concern that we would make our life harder for almost no gain. It's probably a very small group of users (if it exists at all) that wants to add new block drivers themselves, but at the same time can't run upstream qemu.
The first part of your argument may be true, but the second isn't. No user can run upstream qemu.git. It's not tested or supported, and has no backwards compatibility guarantees.
Yes, it does have backwards compatibility guarantees.
Regards,
Anthony Liguori
Quoted 9 lines · Anthony Liguori
Quoted 8 lines · Avi Kivity
Quoted 6 lines · Kevin Wolf
I'm still not convinced that we need either. I share Christoph's concern that we would make our life harder for almost no gain. It's probably a very small group of users (if it exists at all) that wants to add new block drivers themselves, but at the same time can't run upstream qemu.
Quoted 3 lines
Quoted 1 line
The first part of your argument may be true, but the second isn't. No user can run upstream qemu.git. It's not tested or supported, and has no backwards compatibility guarantees.
Yes, it does have backwards compatibility guarantees.
I meant a random untagged qemu.git snapshot. Do we guarantee anything about it, except that it's likely to be broken?
Signature · 1 line
error compiling committee.c: too many arguments to function
Am 25.05.2010 15:55, schrieb Avi Kivity:
Quoted 9 lines · Kevin Wolf
Quoted 7 lines
I'm still not convinced that we need either. I share Christoph's concern that we would make our life harder for almost no gain. It's probably a very small group of users (if it exists at all) that wants to add new block drivers themselves, but at the same time can't run upstream qemu.
Quoted 3 lines
The first part of your argument may be true, but the second isn't. No user can run upstream qemu.git. It's not tested or supported, and has no backwards compatibility guarantees.
The second part was basically meant to say "developers don't count here".
Kevin
Quoted 7 lines · Kevin Wolf
Quoted 5 lines
The first part of your argument may be true, but the second isn't. No user can run upstream qemu.git. It's not tested or supported, and has no backwards compatibility guarantees.
The second part was basically meant to say "developers don't count here".
Agreed.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 6 lines · Blue Swirl
Quoted 5 lines · Christian Brunner
The attached patch is a block driver for the distributed file system Ceph (http://ceph.newdream.net/). This driver uses librados (which is part of the Ceph server) for direct access to the Ceph object store and is running entirely in userspace. Therefore it is called "rbd" - rados block device.
...
Quoted 3 lines
IIRC underscores here may conflict with system header use. Please use something like
QEMU_BLOCK_RADOS_H.
This header is shared between the linux kernel client and the ceph userspace servers and client. We can actually get rid of it, as we only need it to define CEPH_OSD_TMAP_SET. We can move this definition to librados.h.
Quoted 10 lines
block/rbd_types.h line 2
Quoted 8 lines
block/rbd_types.h new file +2
-0@@ -0,0 +1,48 @@ +#ifndef _FS_CEPH_RBD +#define _FS_CEPH_RBD
QEMU_BLOCK_RBD?
This header is shared between the ceph kernel client, between the qemu rbd module (and between other ceph utilities). It'd be much easier maintaining it without having to have a different implementation for each. The same goes to the use of __le32/64 and __u32/64 within these headers.
Quoted 6 lines
block/rbd_types.h line 4
Quoted 3 lines
block/rbd_types.h new file +2
-0+ +#include <linux/types.h> Can you use standard includes, like <
sys/types.h> or <inttypes.h>? Are Ceph libraries used in other systems than Linux?
Not at the moment. I guess that we can take this include out.
Quoted 29 lines
Quoted 29 lines
block/rbd_types.h new file +28
-0+ +/* +* rbd image 'foo' consists of objects +* foo.rbd - image metadata +* foo.00000000 +* foo.00000001 +* ... - data +*/ + +#define RBD_SUFFIX ".rbd" +#define RBD_DIRECTORY "rbd_directory" + +#define RBD_DEFAULT_OBJ_ORDER 22/* 4MB */ + +#define RBD_MAX_OBJ_NAME_SIZE 96 +#define RBD_MAX_SEG_NAME_SIZE 128 + +#define RBD_COMP_NONE 0 +#define RBD_CRYPT_NONE 0 + +static const char rbd_text[] ="<<< Rados Block Device Image >>>\n" ; +static const char rbd_signature[] ="RBD" ; +static const char rbd_version[] ="001.001" ; + +struct rbd_obj_snap_ondisk { + __le64 id; + __le64 image_size; +} __attribute__((packed));
Quoted 8 lines
block/rbd_types.h line 38
Quoted 6 lines
block/rbd_types.h new file +6
-0+ +struct rbd_obj_header_ondisk { + char text[64]; + char signature[4]; + char version[8]; + __le64 image_size;Unaligned? Is the disk format fixed?
This is a packed structure that represents the on disk format. Operations on it are being done only to read from the disk header or to write to the disk header.
Yehuda
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 28 lines · Yehuda Sadeh Weinraub
Quoted 6 lines · Blue Swirl
Quoted 5 lines · Christian Brunner
The attached patch is a block driver for the distributed file system Ceph (http://ceph.newdream.net/). This driver uses librados (which is part of the Ceph server) for direct access to the Ceph object store and is running entirely in userspace. Therefore it is called "rbd" - rados block device.
...
Quoted 3 lines
IIRC underscores here may conflict with system header use. Please use something like
QEMU_BLOCK_RADOS_H.This header is shared between the linux kernel client and the ceph userspace servers and client. We can actually get rid of it, as we only need it to define
CEPH_OSD_TMAP_SET. We can move this definition tolibrados.h.Quoted 10 lines
block/rbd_types.h line 2
Quoted 8 lines
block/rbd_types.h new file +2
-0@@ -0,0 +1,48 @@ +#ifndef _FS_CEPH_RBD +#define _FS_CEPH_RBD
QEMU_BLOCK_RBD?
Quoted 5 lines
This header is shared between the ceph kernel client, between the qemu rbd module (and between other ceph utilities). It'd be much easier maintaining it without having to have a different implementation for each. The same goes to the use of __le32/64 and __u32/64 within these headers.
This is user space, so identifiers must conform to C standards. The identifiers beginning with underscores are reserved.
Doesn't __le32/64 also depend on some GCC extension? Or sparse magic?
Quoted 48 lines
Quoted 7 lines
block/rbd_types.h line 4
Quoted 4 lines
block/rbd_types.h new file +2
-0+ +#include <linux/types.h> Can you use standard includes, like <
sys/types.h> or <inttypes.h>? Are Ceph libraries used in other systems than Linux?Not at the moment. I guess that we can take this include out.
Quoted 29 lines
Quoted 29 lines
block/rbd_types.h new file +28
-0+ +/* +* rbd image 'foo' consists of objects +* foo.rbd - image metadata +* foo.00000000 +* foo.00000001 +* ... - data +*/ + +#define RBD_SUFFIX ".rbd" +#define RBD_DIRECTORY "rbd_directory" + +#define RBD_DEFAULT_OBJ_ORDER 22/* 4MB */ + +#define RBD_MAX_OBJ_NAME_SIZE 96 +#define RBD_MAX_SEG_NAME_SIZE 128 + +#define RBD_COMP_NONE 0 +#define RBD_CRYPT_NONE 0 + +static const char rbd_text[] ="<<< Rados Block Device Image >>>\n" ; +static const char rbd_signature[] ="RBD" ; +static const char rbd_version[] ="001.001" ; + +struct rbd_obj_snap_ondisk { + __le64 id; + __le64 image_size; +} __attribute__((packed));Quoted 8 lines
block/rbd_types.h line 38
Quoted 6 lines
block/rbd_types.h new file +6
-0+ +struct rbd_obj_header_ondisk { + char text[64]; + char signature[4]; + char version[8]; + __le64 image_size;Unaligned? Is the disk format fixed?
Quoted 3 lines
This is a packed structure that represents the on disk format. Operations on it are being done only to read from the disk header or to write to the disk header.
That's clear. But what exactly is the alignment of field 'image_size'? Could there be implicit padding to mod 8 between 'version' and 'image_size' with some compilers?
If there were no other constraints, I'd either make the padding explicit, or rearrange/resize fields so that the field alignment is natural. Thus my question, can you change the disk format or are there already some deployments?
Otherwise, I'd just add some warning comment so people don't try to use clever pointer tricks which will crash on machines with enforced alignment.
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 36 lines · Blue Swirl
Quoted 28 lines · Yehuda Sadeh Weinraub
Quoted 6 lines · Blue Swirl
Quoted 5 lines · Christian Brunner
The attached patch is a block driver for the distributed file system Ceph (http://ceph.newdream.net/). This driver uses librados (which is part of the Ceph server) for direct access to the Ceph object store and is running entirely in userspace. Therefore it is called "rbd" - rados block device.
...
Quoted 3 lines
IIRC underscores here may conflict with system header use. Please use something like
QEMU_BLOCK_RADOS_H.This header is shared between the linux kernel client and the ceph userspace servers and client. We can actually get rid of it, as we only need it to define
CEPH_OSD_TMAP_SET. We can move this definition tolibrados.h.Quoted 10 lines
block/rbd_types.h line 2
Quoted 8 lines
block/rbd_types.h new file +2
-0@@ -0,0 +1,48 @@ +#ifndef _FS_CEPH_RBD +#define _FS_CEPH_RBD
QEMU_BLOCK_RBD?Quoted 5 lines
This header is shared between the ceph kernel client, between the qemu rbd module (and between other ceph utilities). It'd be much easier maintaining it without having to have a different implementation for each. The same goes to the use of __le32/64 and __u32/64 within these headers.
Quoted 4 lines
This is user space, so identifiers must conform to C standards. The identifiers beginning with underscores are reserved.
Doesn't __le32/64 also depend on some GCC extension? Or sparse magic?
It depends on gcc extension. If needed we can probably have a separate header for the qemu block device that uses alternative types. Though looking at the qemu code I see use of other gcc extensions so I'm not sure this is a real issue.
Quoted 49 lines
Quoted 49 lines
Quoted 8 lines
block/rbd_types.h line 4
Quoted 5 lines
block/rbd_types.h new file +2
-0+ +#include <linux/types.h> Can you use standard includes, like <
sys/types.h> or <inttypes.h>? Are Ceph libraries used in other systems than Linux?Not at the moment. I guess that we can take this include out.
Quoted 37 lines
Quoted 25 lines
block/rbd_types.h new file +24
-0+ +/* +* rbd image 'foo' consists of objects +* foo.rbd - image metadata +* foo.00000000 +* foo.00000001 +* ... - data +*/ + +#define RBD_SUFFIX ".rbd" +#define RBD_DIRECTORY "rbd_directory" + +#define RBD_DEFAULT_OBJ_ORDER 22/* 4MB */ + +#define RBD_MAX_OBJ_NAME_SIZE 96 +#define RBD_MAX_SEG_NAME_SIZE 128 + +#define RBD_COMP_NONE 0 +#define RBD_CRYPT_NONE 0 + +static const char rbd_text[] ="<<< Rados Block Device Image >>>\n" ; +static const char rbd_signature[] ="RBD" ; +static const char rbd_version[] ="001.001" ; +block/rbd_types.h line 38
Quoted 10 lines
block/rbd_types.h new file +10
-0+struct rbd_obj_snap_ondisk { + __le64 id; + __le64 image_size; +} __attribute__((packed)); + +struct rbd_obj_header_ondisk { + char text[64]; + char signature[4]; + char version[8]; + __le64 image_size;Unaligned? Is the disk format fixed?
Quoted 7 lines
Quoted 3 lines
This is a packed structure that represents the on disk format. Operations on it are being done only to read from the disk header or to write to the disk header.
That's clear. But what exactly is the alignment of field '
image_size'? Could there be implicit padding to mod 8 between 'version' and 'image_size' with some compilers?
Obviously it's not 64 bit aligned. As it's an on-disk header, I don't see alignment a real issue. As was said before, any operation on these fields have to go through endianity conversion anyway, and this structure should not be used directly. For such datastructures I'd rather have the fields ordered in some logical order than maintaining the alignment by ourselves. That's why we have that __attribute__ packed in the end to let the compiler deal with those issues. Other compilers though have their own syntax for packed structures (but I do see other uses of this packed syntax in the qemu code).
Quoted 5 lines
If there were no other constraints, I'd either make the padding explicit, or rearrange/resize fields so that the field alignment is natural. Thus my question, can you change the disk format or are there already some deployments?
We can certainly make changes to the disk format at this point. I'm not very happy with those 3 __u8 in the middle, and they can probably be changed to a 32 bit flags field. We can get it 64 bit aligned too.
Quoted 5 lines
Otherwise, I'd just add some warning comment so people don't try to use clever pointer tricks which will crash on machines with enforced alignment.
Any clever pointer tricks that'll work on one architecture will probably be wrong on another (different word size/alignment/endianity), so maybe crashing machines is a good indicator to bad implementation. We shouldn't try to hide the problems.
Thanks,
Yehuda
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 37 lines · Yehuda Sadeh Weinraub
Quoted 36 lines · Blue Swirl
Quoted 33 lines · Yehuda Sadeh Weinraub
Quoted 6 lines · Blue Swirl
Quoted 5 lines · Christian Brunner
The attached patch is a block driver for the distributed file system Ceph (http://ceph.newdream.net/). This driver uses librados (which is part of the Ceph server) for direct access to the Ceph object store and is running entirely in userspace. Therefore it is called "rbd" - rados block device.
...
Quoted 3 lines
IIRC underscores here may conflict with system header use. Please use something like
QEMU_BLOCK_RADOS_H.This header is shared between the linux kernel client and the ceph userspace servers and client. We can actually get rid of it, as we only need it to define
CEPH_OSD_TMAP_SET. We can move this definition tolibrados.h.Quoted 10 lines
block/rbd_types.h line 2
Quoted 8 lines
block/rbd_types.h new file +2
-0@@ -0,0 +1,48 @@ +#ifndef _FS_CEPH_RBD +#define _FS_CEPH_RBD
QEMU_BLOCK_RBD?This header is shared between the ceph kernel client, between the qemu rbd module (and between other ceph utilities). It'd be much easier maintaining it without having to have a different implementation for each. The same goes to the use of __le32/64 and __u32/64 within these headers.
Quoted 8 lines
Quoted 4 lines
This is user space, so identifiers must conform to C standards. The identifiers beginning with underscores are reserved.
Doesn't __le32/64 also depend on some GCC extension? Or sparse magic?
It depends on gcc extension. If needed we can probably have a separate header for the qemu block device that uses alternative types. Though looking at the qemu code I see use of other gcc extensions so I'm not sure this is a real issue.
We use some (contained with for example macros if possible), but in earlier discussions, __le32 etc. were considered problematic. IIRC it's hard to provide alternate versions for other compilers (or older versions of gcc).
Quoted 58 lines
Quoted 50 lines
Quoted 50 lines
Quoted 9 lines
block/rbd_types.h line 4
Quoted 6 lines
block/rbd_types.h new file +2
-0+ +#include <linux/types.h> Can you use standard includes, like <
sys/types.h> or <inttypes.h>? Are Ceph libraries used in other systems than Linux?Not at the moment. I guess that we can take this include out.
Quoted 37 lines
Quoted 25 lines
block/rbd_types.h new file +24
-0+ +/* +* rbd image 'foo' consists of objects +* foo.rbd - image metadata +* foo.00000000 +* foo.00000001 +* ... - data +*/ + +#define RBD_SUFFIX ".rbd" +#define RBD_DIRECTORY "rbd_directory" + +#define RBD_DEFAULT_OBJ_ORDER 22/* 4MB */ + +#define RBD_MAX_OBJ_NAME_SIZE 96 +#define RBD_MAX_SEG_NAME_SIZE 128 + +#define RBD_COMP_NONE 0 +#define RBD_CRYPT_NONE 0 + +static const char rbd_text[] ="<<< Rados Block Device Image >>>\n" ; +static const char rbd_signature[] ="RBD" ; +static const char rbd_version[] ="001.001" ; +block/rbd_types.h line 38
Quoted 10 lines
block/rbd_types.h new file +10
-0+struct rbd_obj_snap_ondisk { + __le64 id; + __le64 image_size; +} __attribute__((packed)); + +struct rbd_obj_header_ondisk { + char text[64]; + char signature[4]; + char version[8]; + __le64 image_size;Unaligned? Is the disk format fixed?
Quoted 7 lines
Quoted 3 lines
This is a packed structure that represents the on disk format. Operations on it are being done only to read from the disk header or to write to the disk header.
That's clear. But what exactly is the alignment of field '
image_size'? Could there be implicit padding to mod 8 between 'version' and 'image_size' with some compilers?
Quoted 9 lines
Obviously it's not 64 bit aligned. As it's an on-disk header, I don't see alignment a real issue. As was said before, any operation on these fields have to go through endianity conversion anyway, and this structure should not be used directly. For such datastructures I'd rather have the fields ordered in some logical order than maintaining the alignment by ourselves. That's why we have that
__attribute__packed in the end to let the compiler deal with those issues. Other compilers though have their own syntax for packed structures (but I do see other uses of this packed syntax in the qemu code).
Packed structures are OK, but the padding should be explicit to avoid compiler problems.
Eventually the disk format is read into memory buffer and then aligned fields should be also faster on all architectures, even on x86.
Quoted 9 lines
Quoted 5 lines
If there were no other constraints, I'd either make the padding explicit, or rearrange/resize fields so that the field alignment is natural. Thus my question, can you change the disk format or are there already some deployments?
We can certainly make changes to the disk format at this point. I'm not very happy with those 3
__u8in the middle, and they can probably be changed to a 32 bit flags field. We can get it 64 bit aligned too.
I hope my comments helped you to avoid possible problems in the future. From purely QEMU code base point of view, any architecture goes. Some architectures are faster to emulate, others are slower.
Quoted 14 lines
Quoted 5 lines
Otherwise, I'd just add some warning comment so people don't try to use clever pointer tricks which will crash on machines with enforced alignment.
Any clever pointer tricks that'll work on one architecture will probably be wrong on another (different word size/alignment/endianity), so maybe crashing machines is a good indicator to bad implementation. We shouldn't try to hide the problems.
Thanks,
Yehuda