Skip to contentSkip to conversation

exofs: Multi-device mirror support

PATCH In review Boaz Harrosh 1 participant 7 messages Nov 10, 2009 – Nov 16, 2009 VFS

lore
Share & tools
lore
#0 lore raw

cover exofs: Multi-device mirror support

Submitted for review an exofs support for multiple osd targets in a mirror configuration.

Depending on pending scsi/osd patches I would like to push these in the next exofs push-out. I've been playing with some version of these patches for a while now, testing both as stand alone, as well as, exported to pnfs-objects clients. They are now quite stable, though a bit conservative in regard to what more can be done even in simple mirror configuration.

The user-mode mkfs.exofs supports almost the full range of pnfs-objects fixtures, but the Kernel driver will only mount a mirror arrangement. Once these go through we are planning support for raids 0, 4, 5, and 6 over mirrors over device groups. See second patch's commit log for some more explanations.

The new user-mode utilities as well as these patches can be found at:
http://git.open-osd.org/gitweb.cgi?p=open-osd.git;a=summary

I will push these patches to linux-next once all comments subside.

List of patches:
[PATCH 1/3] exofs: move osd.c to ios.c
[PATCH 2/3] exofs: Move all operations to an io_engine [PATCH 3/3] exofs: Multi-device mirror support

Thanks for review
Boaz

#1 lore raw

PATCH 1/3 exofs: move osd.c to ios.c

If I do a "git mv" together with a massive code change and commit in one patch, git looses the rename and records a delete/new instead. This is bad because I want a rename recorded so later rebased/cherry-picked patches to the old name will work. Also the --follow is lost.

---
 fs/exofs/Kbuild           |    2 +-
 fs/exofs/{osd.c => ios.c} |    0
 2 files changed, 1 insertions(+), 1 deletions(-)
 rename fs/exofs/{osd.c => ios.c} (100%)
fs/exofs/Kbuild +1 -1
index cc2d22d..2d0f757 100644
@@ -12,5 +12,5 @@
 # Kbuild - Gets included from the Kernels Makefile and build system
 #
 
-exofs-y := osd.o inode.o file.o symlink.o namei.o dir.o super.o
+exofs-y := ios.o inode.o file.o symlink.o namei.o dir.o super.o
 obj-$(CONFIG_EXOFS_FS) += exofs.o
fs/exofs/ios.c renamed from fs/exofs/osd.c +0 -0
similarity index 100%
rename from fs/exofs/osd.c
rename to fs/exofs/ios.c
Signature · 1 line
1.6.5.1
#2 lore raw

PATCH 2/3 exofs: Move all operations to an io_engine

In reply to Boaz Harrosh

In anticipation for multi-device operations, we separate osd operations into an abstract I/O API. Currently only one device is used but later when adding more devices, we will drive all devices in parallel according to a "data_map" that describes how data is arranged on multiple devices. The file system level operates, like before, as if there is one object (inode-number) and an i_size. The io engine will split this to the same object-number but on multiple device.

At first we introduce Mirror (raid 1) layout. But at the final outcome we intend to fully implement the pNFS-Objects data-map, including raid 0,4,5,6 over mirrored devices, over multiple device-groups. And more. See: http://tools.ietf.org/html/draft-ietf-nfsv4-pnfs-obj-12

* Define an io_state based API for accessing osd storage devices in an abstract way.
Usage:

	First a caller allocates an io state with:
		exofs_get_io_state(struct exofs_sb_info *sbi,
				   struct exofs_io_state** ios);

	Then calles one of:
		exofs_sbi_create(struct exofs_io_state *ios);
		exofs_sbi_remove(struct exofs_io_state *ios);
		exofs_sbi_write(struct exofs_io_state *ios);
		exofs_sbi_read(struct exofs_io_state *ios);
		exofs_oi_truncate(struct exofs_i_info *oi, u64 new_len);

	And when done
		exofs_put_io_state(struct exofs_io_state *ios);

* Convert all source files to use this new API
* Convert from bio_alloc to bio_kmalloc
* In io engine we make use of the now fixed osd_req_decode_sense

There are no functional changes or on disk additions after this patch.

---
 fs/exofs/common.h |   18 ---
 fs/exofs/exofs.h  |   87 ++++++++++-
 fs/exofs/inode.c  |  369 +++++++++++++++++++++------------------------
 fs/exofs/ios.c    |  433 ++++++++++++++++++++++++++++++++++++++++++++++-------
 fs/exofs/super.c  |  120 ++++++----------
 5 files changed, 682 insertions(+), 345 deletions(-)
fs/exofs/common.h +0 -18
index c6718e4..ce1c716 100644
@@ -155,22 +155,4 @@ enum {
 	(((name_len) + offsetof(struct exofs_dir_entry, name)  + \
 	  EXOFS_DIR_ROUND) & ~EXOFS_DIR_ROUND)
 
-/*************************
- * function declarations *
- *************************/
-/* osd.c                 */
-void exofs_make_credential(u8 cred_a[OSD_CAP_LEN],
-			   const struct osd_obj_id *obj);
-
-int exofs_check_ok_resid(struct osd_request *or, u64 *in_resid, u64 *out_resid);
-static inline int exofs_check_ok(struct osd_request *or)
-{
-	return exofs_check_ok_resid(or, NULL, NULL);
-}
-int exofs_sync_op(struct osd_request *or, int timeout, u8 *cred);
-int exofs_async_op(struct osd_request *or,
-	osd_req_done_fn *async_done, void *caller_context, u8 *cred);
-
-int extract_attr_from_req(struct osd_request *or, struct osd_attr *attr);
-
 #endif /*ifndef __EXOFS_COM_H__*/
fs/exofs/exofs.h +84 -3

122 lines not on this page · raw message

fs/exofs/inode.c +174 -195

754 lines not on this page · raw message

fs/exofs/ios.c +381 -52

479 lines not on this page · raw message

fs/exofs/super.c +43 -77

220 lines not on this page · raw message

Signature · 1 line
1.6.5.1
#5 lore raw

PATCH 3/3 exofs: Multi-device mirror support

In reply to Boaz Harrosh

This patch changes on-disk format, it is accompanied with a parallel patch to mkfs.exofs that enables multi-device capabilities.

After this patch, old exofs will refuse to mount a new formatted FS and new exofs will refuse an old format. This is done by moving the magic field offset inside the FSCB. A new FSCB *version* field was added. In the future, exofs will refuse to mount unmatched FSCB version. To up-grade or down-grade an exofs one must use mkfs.exofs --upgrade option before mounting.

Introduced, a new object that contains a *device-table*. This object contains the default *data-map* and a linear array of devices information, which identifies the devices used in the filesystem. This object is only written to offline by mkfs.exofs. This is why it is kept separate from the FSCB, since the later is written to while mounted.

Same partition number, same object number is used on all devices only the device varies.

* define the new format, then load the device table on mount time make sure every thing is supported.

* Change I/O engine to now support Mirror IO, .i.e write same data to multiple devices, read from a random device to spread the read-load from multiple clients (TODO: stripe read)

Implementation notes:
A few points introduced in previous patch should be mentioned here:

* Special care was made so absolutely all operation that have any chance of failing are done before any osd-request is executed. This is to minimize the need for a data consistency recovery, to only real IO errors.

* Each IO state has a kref. It starts at 1, any osd-request executed will increment the kref, finally when all are executed the first ref is dropped. At IO-done, each request completion decrements the kref, the last one to return executes the internal _last_io() routine. _last_io() will call the registered io_state_done. On sync mode a caller does not supply a done method, indicating a synchronous request, the caller is put to sleep and a special io_state_done is registered that will awaken the caller. Though also in sync mode all operations are executed in parallel.

---
 fs/exofs/common.h |   63 ++++++++++++++--
 fs/exofs/exofs.h  |   12 +++-
 fs/exofs/inode.c  |    5 +-
 fs/exofs/ios.c    |   35 ++++++---
 fs/exofs/pnfs.h   |   51 ++++++++++++
 fs/exofs/super.c  |  220 +++++++++++++++++++++++++++++++++++++++++++++++++++--
 6 files changed, 358 insertions(+), 28 deletions(-)
 create mode 100644 fs/exofs/pnfs.h
fs/exofs/common.h +57 -6
index ce1c716..b1b178e 100644
@@ -49,6 +49,7 @@
 #define EXOFS_MIN_PID   0x10000	/* Smallest partition ID */
 #define EXOFS_OBJ_OFF	0x10000	/* offset for objects */
 #define EXOFS_SUPER_ID	0x10000	/* object ID for on-disk superblock */
+#define EXOFS_DEVTABLE_ID 0x10001 /* object ID for on-disk device table */
 #define EXOFS_ROOT_ID	0x10002	/* object ID for root directory */
 
 /* exofs Application specific page/attribute */
@@ -78,17 +79,67 @@ enum {
 #define EXOFS_SUPER_MAGIC	0x5DF5
 
 /*
- * The file system control block - stored in an object's data (mainly, the one
- * with ID EXOFS_SUPER_ID).  This is where the in-memory superblock is stored
- * on disk.  Right now it just has a magic value, which is basically a sanity
- * check on our ability to communicate with the object store.
+ * The file system control block - stored in object EXOFS_SUPER_ID's data.
+ * This is where the in-memory superblock is stored on disk.
  */
+enum {EXOFS_FSCB_VER = 1, EXOFS_DT_VER = 1};
 struct exofs_fscb {
 	__le64  s_nextid;	/* Highest object ID used */
-	__le32  s_numfiles;	/* Number of files on fs */
+	__le64  s_numfiles;	/* Number of files on fs */
+	__le32	s_version;	/* == EXOFS_FSCB_VER */
 	__le16  s_magic;	/* Magic signature */
 	__le16  s_newfs;	/* Non-zero if this is a new fs */
-};
+
+	/* From here on it's a static part, only written by mkexofs */
+	__le64	s_dev_table_oid;   /* Resurved, not used */
+	__le64	s_dev_table_count; /* == 0 means no dev_table */
+} __packed;
+
+/*
+ * Describes the raid used in the FS. It is part of the device table.
+ * This here is taken from the pNFS-objects definition. In exofs we
+ * use one raid policy through-out the filesystem. (NOTE: the funny
+ * alignment at beginning. We take care of it at exofs_device_table.
+ */
+struct exofs_dt_data_map {
+	__le32	cb_num_comps;
+	__le64	cb_stripe_unit;
+	__le32	cb_group_width;
+	__le32	cb_group_depth;
+	__le32	cb_mirror_cnt;
+	__le32	cb_raid_algorithm;
+} __packed;
+
+/*
+ * This is an osd device information descriptor. It is a single entry in
+ * the exofs device table. It describes an osd target lun which
+ * contains data belonging to this FS. (Same partition_id on all devices)
+ */
+struct exofs_dt_device_info {
+	__le32	systemid_len;
+	u8	systemid[OSD_SYSTEMID_LEN];
+	__le64	long_name_offset;	/* If !0 then offset-in-file */
+	__le32	osdname_len;		/* */
+	u8	osdname[44];		/* Embbeded, Ususally an asci uuid */
+} __packed;
+
+/*
+ * The EXOFS device table - stored in object EXOFS_DEVTABLE_ID's data.
+ * It contains the raid used for this multy-device FS and an array of
+ * participating devices.
+ */
+struct exofs_device_table {
+	__le32				dt_version;	/* == EXOFS_DT_VER */
+	struct exofs_dt_data_map	dt_data_map;	/* Raid policy to use */
+
+	/* Reserved space For future use. Total including this:
+	 * (8 * sizeof(le64))
+	 */
+	__le64				__Reserved[4];
+
+	__le64				dt_num_devices;	/* Array size */
+	struct exofs_dt_device_info	dt_dev_table[];	/* Array of devices */
+} __packed;
 
 /****************************************************************************
  * inode-related things
fs/exofs/exofs.h +10 -2
index 2e08859..c35fd46 100644
@@ -37,6 +37,11 @@
 #include <linux/time.h>
 #include "common.h"
 
+/* FIXME: Remove once pnfs hits mainline
+ * #include <linux/exportfs/pnfs_osd_xdr.h>
+ */
+#include "pnfs.h"
+
 #define EXOFS_ERR(fmt, a...) printk(KERN_ERR "exofs: " fmt, ##a)
 
 #ifdef CONFIG_EXOFS_DEBUG
@@ -54,7 +59,6 @@
  * our extension to the in-memory superblock
  */
 struct exofs_sb_info {
-	struct osd_dev	*s_dev;			/* returned by get_osd_dev    */
 	struct exofs_fscb s_fscb;		/* Written often, pre-allocate*/
 	osd_id		s_pid;			/* partition ID of file system*/
 	int		s_timeout;		/* timeout for OSD operations */
@@ -63,7 +67,11 @@ struct exofs_sb_info {
 	spinlock_t	s_next_gen_lock;	/* spinlock for gen # update  */
 	u32		s_next_generation;	/* next gen # to use          */
 	atomic_t	s_curr_pending;		/* number of pending commands */
-	uint8_t		s_cred[OSD_CAP_LEN];	/* all-powerful credential    */
+	uint8_t		s_cred[OSD_CAP_LEN];	/* credential for the fscb    */
+
+	struct pnfs_osd_data_map data_map;	/* Default raid to use        */
+	unsigned	s_numdevs;		/* Num of devices in array    */
+	struct osd_dev	*s_ods[1];		/* Variable length, minimum 1 */
 };
 
 /*
fs/exofs/inode.c +4 -1
index 8206fcd..f3f287a 100644
@@ -62,7 +62,10 @@ static void _pcol_init(struct page_collect *pcol, unsigned expected_pages,
 	struct exofs_sb_info *sbi = inode->i_sb->s_fs_info;
 
 	pcol->sbi = sbi;
-	pcol->req_q = osd_request_queue(sbi->s_dev);
+	/* Create master bios on first Q, later on cloning, each clone will be
+	 * allocated on it's destination Q
+	 */
+	pcol->req_q = osd_request_queue(sbi->s_ods[0]);
 	pcol->inode = inode;
 	pcol->expected_pages = expected_pages;
 
fs/exofs/ios.c +24 -11
index 4ae1832..369c364 100644
@@ -71,7 +71,7 @@ int exofs_get_io_state(struct exofs_sb_info *sbi, struct exofs_io_state** pios)
 	/*TODO: Maybe use kmem_cach per sbi of size
 	 * exofs_io_state_size(sbi->s_numdevs)
 	 */
-	ios = kzalloc(exofs_io_state_size(1), GFP_KERNEL);
+	ios = kzalloc(exofs_io_state_size(sbi->s_numdevs), GFP_KERNEL);
 	if (unlikely(!ios)) {
 		*pios = NULL;
 		return -ENOMEM;
@@ -203,10 +203,10 @@ int exofs_sbi_create(struct exofs_io_state *ios)
 
 	kref_init(&ios->kref);
 
-	for (i = 0; i < 1; i++) {
+	for (i = 0; i < ios->sbi->s_numdevs; i++) {
 		struct osd_request *or;
 
-		or = osd_start_request(ios->sbi->s_dev, GFP_KERNEL);
+		or = osd_start_request(ios->sbi->s_ods[i], GFP_KERNEL);
 		if (unlikely(!or)) {
 			EXOFS_ERR("%s: osd_start_request failed\n", __func__);
 			ret = -ENOMEM;
@@ -241,10 +241,10 @@ int exofs_sbi_remove(struct exofs_io_state *ios)
 
 	kref_init(&ios->kref);
 
-	for (i = 0; i < 1; i++) {
+	for (i = 0; i < ios->sbi->s_numdevs; i++) {
 		struct osd_request *or;
 
-		or = osd_start_request(ios->sbi->s_dev, GFP_KERNEL);
+		or = osd_start_request(ios->sbi->s_ods[i], GFP_KERNEL);
 		if (unlikely(!or)) {
 			EXOFS_ERR("%s: osd_start_request failed\n", __func__);
 			ret = -ENOMEM;
@@ -278,10 +278,10 @@ int exofs_sbi_write(struct exofs_io_state *ios)
 
 	kref_init(&ios->kref);
 
-	for (i = 0; i < 1; i++) {
+	for (i = 0; i < ios->sbi->s_numdevs; i++) {
 		struct osd_request *or;
 
-		or = osd_start_request(ios->sbi->s_dev, GFP_KERNEL);
+		or = osd_start_request(ios->sbi->s_ods[i], GFP_KERNEL);
 		if (unlikely(!or)) {
 			EXOFS_ERR("%s: osd_start_request failed\n", __func__);
 			ret = -ENOMEM;
@@ -293,7 +293,19 @@ int exofs_sbi_write(struct exofs_io_state *ios)
 		if (ios->bio) {
 			struct bio *bio;
 
-			bio = ios->bio;
+			if (i != 0) {
+				bio = bio_kmalloc(GFP_KERNEL,
+						  ios->bio->bi_max_vecs);
+				if (!bio)
+					goto out;
+
+				__bio_clone(bio, ios->bio);
+				bio->bi_bdev = NULL;
+				bio->bi_next = NULL;
+				ios->per_dev[i].bio =  bio;
+			} else {
+				bio = ios->bio;
+			}
 
 			osd_req_write(or, &ios->obj, ios->offset, bio,
 				      ios->length);
@@ -340,8 +352,9 @@ int exofs_sbi_read(struct exofs_io_state *ios)
 
 	for (i = 0; i < 1; i++) {
 		struct osd_request *or;
+		unsigned first_dev = ios->obj.id % ios->sbi->s_numdevs;
 
-		or = osd_start_request(ios->sbi->s_dev, GFP_KERNEL);
+		or = osd_start_request(ios->sbi->s_ods[first_dev], GFP_KERNEL);
 		if (unlikely(!or)) {
 			EXOFS_ERR("%s: osd_start_request failed\n", __func__);
 			ret = -ENOMEM;
@@ -426,10 +439,10 @@ int exofs_oi_truncate(struct exofs_i_info *oi, u64 size)
 
 	kref_init(&ios->kref);
 
-	for (i = 0; i < 1; i++) {
+	for (i = 0; i < sbi->s_numdevs; i++) {
 		struct osd_request *or;
 
-		or = osd_start_request(sbi->s_dev, GFP_KERNEL);
+		or = osd_start_request(sbi->s_ods[i], GFP_KERNEL);
 		if (unlikely(!or)) {
 			EXOFS_ERR("%s: osd_start_request failed\n", __func__);
 			ret = -ENOMEM;
fs/exofs/pnfs.h new file +51 -0

51 lines not on this page · raw message

fs/exofs/super.c +212 -8

280 lines not on this page · raw message

Signature · 1 line
1.6.5.1
#3 lore raw

PATCH 2/3 exofs: Move all operations to an io_engine

In reply to Boaz Harrosh

In anticipation for multi-device operations, we separate osd operations into an abstract I/O API. Currently only one device is used but later when adding more devices, we will drive all devices in parallel according to a "data_map" that describes how data is arranged on multiple devices. The file system level operates, like before, as if there is one object (inode-number) and an i_size. The io engine will split this to the same object-number but on multiple device.

At first we introduce Mirror (raid 1) layout. But at the final outcome we intend to fully implement the pNFS-Objects data-map, including raid 0,4,5,6 over mirrored devices, over multiple device-groups. And more. See: http://tools.ietf.org/html/draft-ietf-nfsv4-pnfs-obj-12

* Define an io_state based API for accessing osd storage devices in an abstract way.
Usage:

	First a caller allocates an io state with:
		exofs_get_io_state(struct exofs_sb_info *sbi,
				   struct exofs_io_state** ios);

	Then calles one of:
		exofs_sbi_create(struct exofs_io_state *ios);
		exofs_sbi_remove(struct exofs_io_state *ios);
		exofs_sbi_write(struct exofs_io_state *ios);
		exofs_sbi_read(struct exofs_io_state *ios);
		exofs_oi_truncate(struct exofs_i_info *oi, u64 new_len);

	And when done
		exofs_put_io_state(struct exofs_io_state *ios);

* Convert all source files to use this new API
* Convert from bio_alloc to bio_kmalloc
* In io engine we make use of the now fixed osd_req_decode_sense

There are no functional changes or on disk additions after this patch.

---
 fs/exofs/common.h |   18 ---
 fs/exofs/exofs.h  |   87 ++++++++++++-
 fs/exofs/inode.c  |  383 +++++++++++++++++++++++++---------------------------
 fs/exofs/ios.c    |  386 +++++++++++++++++++++++++++++++++++++++++++++--------
 fs/exofs/super.c  |  120 ++++++-----------
 5 files changed, 644 insertions(+), 350 deletions(-)
fs/exofs/common.h +0 -18
index c6718e4..ce1c716 100644
@@ -155,22 +155,4 @@ enum {
 	(((name_len) + offsetof(struct exofs_dir_entry, name)  + \
 	  EXOFS_DIR_ROUND) & ~EXOFS_DIR_ROUND)
 
-/*************************
- * function declarations *
- *************************/
-/* osd.c                 */
-void exofs_make_credential(u8 cred_a[OSD_CAP_LEN],
-			   const struct osd_obj_id *obj);
-
-int exofs_check_ok_resid(struct osd_request *or, u64 *in_resid, u64 *out_resid);
-static inline int exofs_check_ok(struct osd_request *or)
-{
-	return exofs_check_ok_resid(or, NULL, NULL);
-}
-int exofs_sync_op(struct osd_request *or, int timeout, u8 *cred);
-int exofs_async_op(struct osd_request *or,
-	osd_req_done_fn *async_done, void *caller_context, u8 *cred);
-
-int extract_attr_from_req(struct osd_request *or, struct osd_attr *attr);
-
 #endif /*ifndef __EXOFS_COM_H__*/
fs/exofs/exofs.h +84 -3

122 lines not on this page · raw message

fs/exofs/inode.c +184 -199

768 lines not on this page · raw message

fs/exofs/ios.c +333 -53

431 lines not on this page · raw message

fs/exofs/super.c +43 -77

220 lines not on this page · raw message

Signature · 1 line
1.6.5.2
#6 lore raw

PATCH 3/3 exofs: Multi-device mirror support

In reply to Boaz Harrosh

This patch changes on-disk format, it is accompanied with a parallel patch to mkfs.exofs that enables multi-device capabilities.

After this patch, old exofs will refuse to mount a new formatted FS and new exofs will refuse an old format. This is done by moving the magic field offset inside the FSCB. A new FSCB *version* field was added. In the future, exofs will refuse to mount unmatched FSCB version. To up-grade or down-grade an exofs one must use mkfs.exofs --upgrade option before mounting.

Introduced, a new object that contains a *device-table*. This object contains the default *data-map* and a linear array of devices information, which identifies the devices used in the filesystem. This object is only written to offline by mkfs.exofs. This is why it is kept separate from the FSCB, since the later is written to while mounted.

Same partition number, same object number is used on all devices only the device varies.

* define the new format, then load the device table on mount time make sure every thing is supported.

* Change I/O engine to now support Mirror IO, .i.e write same data to multiple devices, read from a random device to spread the read-load from multiple clients (TODO: stripe read)

Implementation notes:
A few points introduced in previous patch should be mentioned here:

* Special care was made so absolutlly all operation that have any chance of failing are done before any osd-request is executed. This is to minimize the need for a data consistency recovery, to only real IO errors.

* Each IO state has a kref. It starts at 1, any osd-request executed will increment the kref, finally when all are executed the first ref is dropped. At IO-done, each request completion decrements the kref, the last one to return executes the internal _last_io() routine. _last_io() will call the registered io_state_done. On sync mode a caller does not supply a done method, indicating a synchronous request, the caller is put to sleep and a special io_state_done is registered that will awaken the caller. Though also in sync mode all operations are executed in parallel.

---
 fs/exofs/common.h |   63 ++++++++++++++--
 fs/exofs/exofs.h  |   12 +++-
 fs/exofs/inode.c  |    5 +-
 fs/exofs/ios.c    |   37 ++++++---
 fs/exofs/pnfs.h   |   51 ++++++++++++
 fs/exofs/super.c  |  220 +++++++++++++++++++++++++++++++++++++++++++++++++++--
 6 files changed, 360 insertions(+), 28 deletions(-)
 create mode 100644 fs/exofs/pnfs.h
fs/exofs/common.h +57 -6
index ce1c716..b1b178e 100644
@@ -49,6 +49,7 @@
 #define EXOFS_MIN_PID   0x10000	/* Smallest partition ID */
 #define EXOFS_OBJ_OFF	0x10000	/* offset for objects */
 #define EXOFS_SUPER_ID	0x10000	/* object ID for on-disk superblock */
+#define EXOFS_DEVTABLE_ID 0x10001 /* object ID for on-disk device table */
 #define EXOFS_ROOT_ID	0x10002	/* object ID for root directory */
 
 /* exofs Application specific page/attribute */
@@ -78,17 +79,67 @@ enum {
 #define EXOFS_SUPER_MAGIC	0x5DF5
 
 /*
- * The file system control block - stored in an object's data (mainly, the one
- * with ID EXOFS_SUPER_ID).  This is where the in-memory superblock is stored
- * on disk.  Right now it just has a magic value, which is basically a sanity
- * check on our ability to communicate with the object store.
+ * The file system control block - stored in object EXOFS_SUPER_ID's data.
+ * This is where the in-memory superblock is stored on disk.
  */
+enum {EXOFS_FSCB_VER = 1, EXOFS_DT_VER = 1};
 struct exofs_fscb {
 	__le64  s_nextid;	/* Highest object ID used */
-	__le32  s_numfiles;	/* Number of files on fs */
+	__le64  s_numfiles;	/* Number of files on fs */
+	__le32	s_version;	/* == EXOFS_FSCB_VER */
 	__le16  s_magic;	/* Magic signature */
 	__le16  s_newfs;	/* Non-zero if this is a new fs */
-};
+
+	/* From here on it's a static part, only written by mkexofs */
+	__le64	s_dev_table_oid;   /* Resurved, not used */
+	__le64	s_dev_table_count; /* == 0 means no dev_table */
+} __packed;
+
+/*
+ * Describes the raid used in the FS. It is part of the device table.
+ * This here is taken from the pNFS-objects definition. In exofs we
+ * use one raid policy through-out the filesystem. (NOTE: the funny
+ * alignment at begining. We take care of it at exofs_device_table.
+ */
+struct exofs_dt_data_map {
+	__le32	cb_num_comps;
+	__le64	cb_stripe_unit;
+	__le32	cb_group_width;
+	__le32	cb_group_depth;
+	__le32	cb_mirror_cnt;
+	__le32	cb_raid_algorithm;
+} __packed;
+
+/*
+ * This is an osd device information descriptor. It is a single entry in
+ * the exofs device table. It describes an osd target lun which
+ * contains data belonging to this FS. (Same partition_id on all devices)
+ */
+struct exofs_dt_device_info {
+	__le32	systemid_len;
+	u8	systemid[OSD_SYSTEMID_LEN];
+	__le64	long_name_offset;	/* If !0 then offset-in-file */
+	__le32	osdname_len;		/* */
+	u8	osdname[44];		/* Embbeded, Ususally an asci uuid */
+} __packed;
+
+/*
+ * The EXOFS device table - stored in object EXOFS_DEVTABLE_ID's data.
+ * It contains the raid used for this multy-device FS and an array of
+ * participating devices.
+ */
+struct exofs_device_table {
+	__le32				dt_version;	/* == EXOFS_DT_VER */
+	struct exofs_dt_data_map	dt_data_map;	/* Raid policy to use */
+
+	/* Resurved space For future use. Total includeing this:
+	 * (8 * sizeof(le64))
+	 */
+	__le64				__Resurved[4];
+
+	__le64				dt_num_devices;	/* Array size */
+	struct exofs_dt_device_info	dt_dev_table[];	/* Array of devices */
+} __packed;
 
 /****************************************************************************
  * inode-related things
fs/exofs/exofs.h +10 -2
index 2e08859..c35fd46 100644
@@ -37,6 +37,11 @@
 #include <linux/time.h>
 #include "common.h"
 
+/* FIXME: Remove once pnfs hits mainline
+ * #include <linux/exportfs/pnfs_osd_xdr.h>
+ */
+#include "pnfs.h"
+
 #define EXOFS_ERR(fmt, a...) printk(KERN_ERR "exofs: " fmt, ##a)
 
 #ifdef CONFIG_EXOFS_DEBUG
@@ -54,7 +59,6 @@
  * our extension to the in-memory superblock
  */
 struct exofs_sb_info {
-	struct osd_dev	*s_dev;			/* returned by get_osd_dev    */
 	struct exofs_fscb s_fscb;		/* Written often, pre-allocate*/
 	osd_id		s_pid;			/* partition ID of file system*/
 	int		s_timeout;		/* timeout for OSD operations */
@@ -63,7 +67,11 @@ struct exofs_sb_info {
 	spinlock_t	s_next_gen_lock;	/* spinlock for gen # update  */
 	u32		s_next_generation;	/* next gen # to use          */
 	atomic_t	s_curr_pending;		/* number of pending commands */
-	uint8_t		s_cred[OSD_CAP_LEN];	/* all-powerful credential    */
+	uint8_t		s_cred[OSD_CAP_LEN];	/* credential for the fscb    */
+
+	struct pnfs_osd_data_map data_map;	/* Default raid to use        */
+	unsigned	s_numdevs;		/* Num of devices in array    */
+	struct osd_dev	*s_ods[1];		/* Variable length, minimum 1 */
 };
 
 /*
fs/exofs/inode.c +4 -1
index 7578950..698a863 100644
@@ -62,7 +62,10 @@ static void _pcol_init(struct page_collect *pcol, unsigned expected_pages,
 	struct exofs_sb_info *sbi = inode->i_sb->s_fs_info;
 
 	pcol->sbi = sbi;
-	pcol->req_q = osd_request_queue(sbi->s_dev);
+	/* Create master bios on first Q, later on cloning, each clone will be
+	 * allocated on it's destination Q
+	 */
+	pcol->req_q = osd_request_queue(sbi->s_ods[0]);
 	pcol->inode = inode;
 	pcol->expected_pages = expected_pages;
 
fs/exofs/ios.c +26 -11
index bb2f9d3..14b2600 100644
@@ -71,7 +71,7 @@ int exofs_get_io_state(struct exofs_sb_info *sbi, struct exofs_io_state** pios)
 	/*TODO: Maybe use kmem_cach per sbi of size
 	 * exofs_io_state_size(sbi->s_numdevs)
 	 */
-	ios = kzalloc(exofs_io_state_size(1), GFP_KERNEL);
+	ios = kzalloc(exofs_io_state_size(sbi->s_numdevs), GFP_KERNEL);
 	if (unlikely(!ios)) {
 		*pios = NULL;
 		return -ENOMEM;
@@ -209,10 +209,10 @@ int exofs_sbi_create(struct exofs_io_state *ios)
 {
 	int i, ret;
 
-	for (i = 0; i < 1; i++) {
+	for (i = 0; i < ios->sbi->s_numdevs; i++) {
 		struct osd_request *or;
 
-		or = osd_start_request(ios->sbi->s_dev, GFP_KERNEL);
+		or = osd_start_request(ios->sbi->s_ods[i], GFP_KERNEL);
 		if (unlikely(!or)) {
 			EXOFS_ERR("%s: osd_start_request failed\n", __func__);
 			ret = -ENOMEM;
@@ -233,10 +233,10 @@ int exofs_sbi_remove(struct exofs_io_state *ios)
 {
 	int i, ret;
 
-	for (i = 0; i < 1; i++) {
+	for (i = 0; i < ios->sbi->s_numdevs; i++) {
 		struct osd_request *or;
 
-		or = osd_start_request(ios->sbi->s_dev, GFP_KERNEL);
+		or = osd_start_request(ios->sbi->s_ods[i], GFP_KERNEL);
 		if (unlikely(!or)) {
 			EXOFS_ERR("%s: osd_start_request failed\n", __func__);
 			ret = -ENOMEM;
@@ -257,10 +257,10 @@ int exofs_sbi_write(struct exofs_io_state *ios)
 {
 	int i, ret;
 
-	for (i = 0; i < 1; i++) {
+	for (i = 0; i < ios->sbi->s_numdevs; i++) {
 		struct osd_request *or;
 
-		or = osd_start_request(ios->sbi->s_dev, GFP_KERNEL);
+		or = osd_start_request(ios->sbi->s_ods[i], GFP_KERNEL);
 		if (unlikely(!or)) {
 			EXOFS_ERR("%s: osd_start_request failed\n", __func__);
 			ret = -ENOMEM;
@@ -272,7 +272,21 @@ int exofs_sbi_write(struct exofs_io_state *ios)
 		if (ios->bio) {
 			struct bio *bio;
 
-			bio = ios->bio;
+			if (i != 0) {
+				bio = bio_kmalloc(GFP_KERNEL,
+						  ios->bio->bi_max_vecs);
+				if (unlikely(!bio)) {
+					ret = -ENOMEM;
+					goto out;
+				}
+
+				__bio_clone(bio, ios->bio);
+				bio->bi_bdev = NULL;
+				bio->bi_next = NULL;
+				ios->per_dev[i].bio =  bio;
+			} else {
+				bio = ios->bio;
+			}
 
 			osd_req_write(or, &ios->obj, ios->offset, bio,
 				      ios->length);
@@ -306,8 +320,9 @@ int exofs_sbi_read(struct exofs_io_state *ios)
 
 	for (i = 0; i < 1; i++) {
 		struct osd_request *or;
+		unsigned first_dev = ios->obj.id % ios->sbi->s_numdevs;
 
-		or = osd_start_request(ios->sbi->s_dev, GFP_KERNEL);
+		or = osd_start_request(ios->sbi->s_ods[first_dev], GFP_KERNEL);
 		if (unlikely(!or)) {
 			EXOFS_ERR("%s: osd_start_request failed\n", __func__);
 			ret = -ENOMEM;
@@ -382,10 +397,10 @@ int exofs_oi_truncate(struct exofs_i_info *oi, u64 size)
 	attr = g_attr_logical_length;
 	attr.val_ptr = &newsize;
 
-	for (i = 0; i < 1; i++) {
+	for (i = 0; i < sbi->s_numdevs; i++) {
 		struct osd_request *or;
 
-		or = osd_start_request(sbi->s_dev, GFP_KERNEL);
+		or = osd_start_request(sbi->s_ods[i], GFP_KERNEL);
 		if (unlikely(!or)) {
 			EXOFS_ERR("%s: osd_start_request failed\n", __func__);
 			ret = -ENOMEM;
fs/exofs/pnfs.h new file +51 -0

51 lines not on this page · raw message

fs/exofs/super.c +212 -8

280 lines not on this page · raw message

Signature · 1 line
1.6.5.2
#4 lore raw

Re: [osd-dev] [PATCH 2/3] exofs: Move all operations to an io_engine

In reply to Boaz Harrosh
Quoted 31 lines · Boaz Harrosh

In anticipation for multi-device operations, we separate osd operations into an abstract I/O API. Currently only one device is used but later when adding more devices, we will drive all devices in parallel according to a "data_map" that describes how data is arranged on multiple devices. The file system level operates, like before, as if there is one object (inode-number) and an i_size. The io engine will split this to the same object-number but on multiple device.

At first we introduce Mirror (raid 1) layout. But at the final outcome we intend to fully implement the pNFS-Objects data-map, including raid 0,4,5,6 over mirrored devices, over multiple device-groups. And more. See: http://tools.ietf.org/html/draft-ietf-nfsv4-pnfs-obj-12

* Define an io_state based API for accessing osd storage devices in an abstract way.
Usage:

	First a caller allocates an io state with:
		exofs_get_io_state(struct exofs_sb_info *sbi,
				   struct exofs_io_state** ios);

	Then calles one of:
		exofs_sbi_create(struct exofs_io_state *ios);
		exofs_sbi_remove(struct exofs_io_state *ios);
		exofs_sbi_write(struct exofs_io_state *ios);
		exofs_sbi_read(struct exofs_io_state *ios);
		exofs_oi_truncate(struct exofs_i_info *oi, u64 new_len);

	And when done
		exofs_put_io_state(struct exofs_io_state *ios);
Quoted 7 lines

* Convert all source files to use this new API
* Convert from bio_alloc to bio_kmalloc
* In io engine we make use of the now fixed osd_req_decode_sense

There are no functional changes or on disk additions after this patch.

Putting my mailing-list-reader hat produced some "What was that good for?"

Below is the diff of the new posts from the old ones. (New as replay to originals)

---
git diff --stat -p -M -R afb8f9c089c07711104bb26e224440415ba3d324 -- fs/exofs/

 fs/exofs/inode.c |   14 ++++++--
 fs/exofs/ios.c   |   93 +++++++++++++----------------------------------------
 2 files changed, 33 insertions(+), 74 deletions(-)
fs/exofs/inode.c +10 -4
index f3f287a..698a863 100644
@@ -444,7 +444,7 @@ static int exofs_readpage(struct file *file, struct page *page)
 	return _readpage(page, false);
 }
 
-/* Callback for osd_write. All writes are asynchronouse */
+/* Callback for osd_write. All writes are asynchronous */
 static void writepages_done(struct exofs_io_state *ios, void *p)
 {
 	struct page_collect *pcol = p;
@@ -1029,9 +1029,15 @@ static void create_done(struct exofs_io_state *ios, void *p)
 	if (unlikely(ret)) {
 		EXOFS_ERR("object=0x%llx creation faild in pid=0x%llx",
 			  _LLU(exofs_oi_objno(oi)), _LLU(sbi->s_pid));
-		make_bad_inode(inode);
-	} else
-		set_obj_created(oi);
+		/*TODO: When FS is corrupted creation can fail, object already
+		 * exist. Get rid of this asynchronous creation, if exist
+		 * increment the obj counter and try the next object. Until we
+		 * succeed. All these dangling objects will be made into lost
+		 * files by chkfs.exofs
+		 */
+	}
+
+	set_obj_created(oi);
 
 	atomic_dec(&inode->i_count);
 	wake_up(&oi->i_wq);
fs/exofs/ios.c +23 -70

187 lines not on this page · raw message