exofs: Multi-device mirror support
cover exofs: Multi-device mirror support
Submitted for review an exofs support for multiple osd targets in a mirror configuration.
Depending on pending scsi/osd patches I would like to push these in the next exofs push-out. I've been playing with some version of these patches for a while now, testing both as stand alone, as well as, exported to pnfs-objects clients. They are now quite stable, though a bit conservative in regard to what more can be done even in simple mirror configuration.
The user-mode mkfs.exofs supports almost the full range of pnfs-objects fixtures, but the Kernel driver will only mount a mirror arrangement. Once these go through we are planning support for raids 0, 4, 5, and 6 over mirrors over device groups. See second patch's commit log for some more explanations.
The new user-mode utilities as well as these patches can be found at:
http://git.open-osd.org/gitweb.cgi?p=open-osd.git;a=summary
I will push these patches to linux-next once all comments subside.
List of patches:
[PATCH 1/3] exofs: move osd.c to ios.c
[PATCH 2/3] exofs: Move all operations to an io_engine [PATCH 3/3] exofs: Multi-device mirror support
Thanks for review
Boaz
PATCH 1/3 exofs: move osd.c to ios.c
If I do a "git mv" together with a massive code change and commit in one patch, git looses the rename and records a delete/new instead. This is bad because I want a rename recorded so later rebased/cherry-picked patches to the old name will work. Also the --follow is lost.
---
fs/exofs/Kbuild | 2 +-
fs/exofs/{osd.c => ios.c} | 0
2 files changed, 1 insertions(+), 1 deletions(-)
rename fs/exofs/{osd.c => ios.c} (100%)fs/exofs/Kbuild +1 -1
@@ -12,5 +12,5 @@ # Kbuild - Gets included from the Kernels Makefile and build system # -exofs-y := osd.o inode.o file.o symlink.o namei.o dir.o super.o +exofs-y := ios.o inode.o file.o symlink.o namei.o dir.o super.o obj-$(CONFIG_EXOFS_FS) += exofs.o
fs/exofs/ios.c renamed from fs/exofs/osd.c +0 -0
Signature · 1 line
1.6.5.1
PATCH 2/3 exofs: Move all operations to an io_engine
In reply to Boaz HarroshIn anticipation for multi-device operations, we separate osd operations into an abstract I/O API. Currently only one device is used but later when adding more devices, we will drive all devices in parallel according to a "data_map" that describes how data is arranged on multiple devices. The file system level operates, like before, as if there is one object (inode-number) and an i_size. The io engine will split this to the same object-number but on multiple device.
At first we introduce Mirror (raid 1) layout. But at the final outcome we intend to fully implement the pNFS-Objects data-map, including raid 0,4,5,6 over mirrored devices, over multiple device-groups. And more. See: http://tools.ietf.org/html/draft-ietf-nfsv4-pnfs-obj-12
* Define an io_state based API for accessing osd storage devices in an abstract way.
Usage:
First a caller allocates an io state with: exofs_get_io_state(struct exofs_sb_info *sbi, struct exofs_io_state** ios); Then calles one of: exofs_sbi_create(struct exofs_io_state *ios); exofs_sbi_remove(struct exofs_io_state *ios); exofs_sbi_write(struct exofs_io_state *ios); exofs_sbi_read(struct exofs_io_state *ios); exofs_oi_truncate(struct exofs_i_info *oi, u64 new_len); And when done exofs_put_io_state(struct exofs_io_state *ios);
* Convert all source files to use this new API
* Convert from bio_alloc to bio_kmalloc
* In io engine we make use of the now fixed osd_req_decode_sense
There are no functional changes or on disk additions after this patch.
--- fs/exofs/common.h | 18 --- fs/exofs/exofs.h | 87 ++++++++++- fs/exofs/inode.c | 369 +++++++++++++++++++++------------------------ fs/exofs/ios.c | 433 ++++++++++++++++++++++++++++++++++++++++++++++------- fs/exofs/super.c | 120 ++++++---------- 5 files changed, 682 insertions(+), 345 deletions(-)
fs/exofs/common.h +0 -18
@@ -155,22 +155,4 @@ enum { (((name_len) + offsetof(struct exofs_dir_entry , name) + \ EXOFS_DIR_ROUND) & ~EXOFS_DIR_ROUND) -/************************* -* function declarations * -*************************/ -/* osd.c */ -void exofs_make_credential (u8 cred_a[OSD_CAP_LEN ], -const struct osd_obj_id *obj); - -int exofs_check_ok_resid (struct osd_request *or,u64 *in_resid,u64 *out_resid); -static inline int exofs_check_ok (struct osd_request *or) -{ -return exofs_check_ok_resid (or,NULL ,NULL ); -} -int exofs_sync_op (struct osd_request *or,int timeout,u8 *cred); -int exofs_async_op (struct osd_request *or, -osd_req_done_fn *async_done,void *caller_context,u8 *cred); - -int extract_attr_from_req (struct osd_request *or,struct osd_attr *attr); -#endif /*ifndef __EXOFS_COM_H__*/
fs/exofs/exofs.h +84 -3
122 lines not on this page · raw message
fs/exofs/inode.c +174 -195
754 lines not on this page · raw message
fs/exofs/ios.c +381 -52
479 lines not on this page · raw message
fs/exofs/super.c +43 -77
220 lines not on this page · raw message
Signature · 1 line
1.6.5.1
PATCH 3/3 exofs: Multi-device mirror support
In reply to Boaz HarroshThis patch changes on-disk format, it is accompanied with a parallel patch to mkfs.exofs that enables multi-device capabilities.
After this patch, old exofs will refuse to mount a new formatted FS and new exofs will refuse an old format. This is done by moving the magic field offset inside the FSCB. A new FSCB *version* field was added. In the future, exofs will refuse to mount unmatched FSCB version. To up-grade or down-grade an exofs one must use mkfs.exofs --upgrade option before mounting.
Introduced, a new object that contains a *device-table*. This object contains the default *data-map* and a linear array of devices information, which identifies the devices used in the filesystem. This object is only written to offline by mkfs.exofs. This is why it is kept separate from the FSCB, since the later is written to while mounted.
Same partition number, same object number is used on all devices only the device varies.
* define the new format, then load the device table on mount time make sure every thing is supported.
* Change I/O engine to now support Mirror IO, .i.e write same data to multiple devices, read from a random device to spread the read-load from multiple clients (TODO: stripe read)
Implementation notes:
A few points introduced in previous patch should be mentioned here:
* Special care was made so absolutely all operation that have any chance of failing are done before any osd-request is executed. This is to minimize the need for a data consistency recovery, to only real IO errors.
* Each IO state has a kref. It starts at 1, any osd-request executed will increment the kref, finally when all are executed the first ref is dropped. At IO-done, each request completion decrements the kref, the last one to return executes the internal _last_io() routine. _last_io() will call the registered io_state_done. On sync mode a caller does not supply a done method, indicating a synchronous request, the caller is put to sleep and a special io_state_done is registered that will awaken the caller. Though also in sync mode all operations are executed in parallel.
--- fs/exofs/common.h | 63 ++++++++++++++-- fs/exofs/exofs.h | 12 +++- fs/exofs/inode.c | 5 +- fs/exofs/ios.c | 35 ++++++--- fs/exofs/pnfs.h | 51 ++++++++++++ fs/exofs/super.c | 220 +++++++++++++++++++++++++++++++++++++++++++++++++++-- 6 files changed, 358 insertions(+), 28 deletions(-) create mode 100644 fs/exofs/pnfs.h
fs/exofs/common.h +57 -6
@@ -49,6 +49,7 @@#define EXOFS_MIN_PID 0x10000/* Smallest partition ID */ #define EXOFS_OBJ_OFF 0x10000/* offset for objects */ #define EXOFS_SUPER_ID 0x10000/* object ID for on-disk superblock */ +#define EXOFS_DEVTABLE_ID 0x10001/* object ID for on-disk device table */ #define EXOFS_ROOT_ID 0x10002/* object ID for root directory */ /* exofs Application specific page/attribute */
@@ -78,17 +79,67 @@ enum {#define EXOFS_SUPER_MAGIC 0x5DF5/* -* The file system control block - stored in an object's data (mainly, the one -* with ID EXOFS_SUPER_ID). This is where the in-memory superblock is stored -* on disk. Right now it just has a magic value, which is basically a sanity -* check on our ability to communicate with the object store. +* The file system control block - stored in object EXOFS_SUPER_ID's data. +* This is where the in-memory superblock is stored on disk. */ +enum {EXOFS_FSCB_VER =1 ,EXOFS_DT_VER =1 };struct exofs_fscb {__le64 s_nextid;/* Highest object ID used */ -__le32 s_numfiles;/* Number of files on fs */ +__le64 s_numfiles;/* Number of files on fs */ +__le32 s_version;/* == EXOFS_FSCB_VER */ __le16 s_magic;/* Magic signature */ __le16 s_newfs;/* Non-zero if this is a new fs */ -}; + +/* From here on it's a static part, only written by mkexofs */ +__le64 s_dev_table_oid;/* Resurved, not used */ +__le64 s_dev_table_count;/* == 0 means no dev_table */ +} __packed; + +/* +* Describes the raid used in the FS. It is part of the device table. +* This here is taken from the pNFS-objects definition. In exofs we +* use one raid policy through-out the filesystem. (NOTE: the funny +* alignment at beginning. We take care of it at exofs_device_table. +*/ +struct exofs_dt_data_map { +__le32 cb_num_comps; +__le64 cb_stripe_unit; +__le32 cb_group_width; +__le32 cb_group_depth; +__le32 cb_mirror_cnt; +__le32 cb_raid_algorithm; +} __packed; + +/* +* This is an osd device information descriptor. It is a single entry in +* the exofs device table. It describes an osd target lun which +* contains data belonging to this FS. (Same partition_id on all devices) +*/ +struct exofs_dt_device_info { +__le32 systemid_len; +u8 systemid[OSD_SYSTEMID_LEN ]; +__le64 long_name_offset;/* If !0 then offset-in-file */ +__le32 osdname_len;/* */ +u8 osdname[44 ];/* Embbeded, Ususally an asci uuid */ +} __packed; + +/* +* The EXOFS device table - stored in object EXOFS_DEVTABLE_ID's data. +* It contains the raid used for this multy-device FS and an array of +* participating devices. +*/ +struct exofs_device_table { +__le32 dt_version;/* == EXOFS_DT_VER */ +struct exofs_dt_data_map dt_data_map;/* Raid policy to use */ + +/* Reserved space For future use. Total including this: +* (8 * sizeof(le64)) +*/ +__le64 __Reserved[4 ]; + +__le64 dt_num_devices;/* Array size */ +struct exofs_dt_device_info dt_dev_table[];/* Array of devices */ +} __packed;/**************************************************************************** * inode-related things
fs/exofs/exofs.h +10 -2
@@ -37,6 +37,11 @@#include <linux/time.h> #include "common.h" +/* FIXME: Remove once pnfs hits mainline +* #include <linux/exportfs/pnfs_osd_xdr.h> +*/ +#include "pnfs.h" + #define EXOFS_ERR(fmt, a...) printk(KERN_ERR "exofs: " fmt, ##a)#ifdef CONFIG_EXOFS_DEBUG
@@ -54,7 +59,6 @@* our extension to the in-memory superblock */ struct exofs_sb_info { -struct osd_dev *s_dev;/* returned by get_osd_dev */ struct exofs_fscb s_fscb;/* Written often, pre-allocate*/ osd_id s_pid;/* partition ID of file system*/ int s_timeout;/* timeout for OSD operations */
@@ -63,7 +67,11 @@ struct exofs_sb_info {spinlock_t s_next_gen_lock;/* spinlock for gen # update */ u32 s_next_generation;/* next gen # to use */ atomic_t s_curr_pending;/* number of pending commands */ -uint8_t s_cred[OSD_CAP_LEN ];/* all-powerful credential */ +uint8_t s_cred[OSD_CAP_LEN ];/* credential for the fscb */ + +struct pnfs_osd_data_map data_map;/* Default raid to use */ +unsigned s_numdevs;/* Num of devices in array */ +struct osd_dev *s_ods[1 ];/* Variable length, minimum 1 */ };/*
fs/exofs/inode.c +4 -1
@@ -62,7 +62,10 @@ static void _pcol_init(struct page_collect *pcol, unsigned expected_pages,struct exofs_sb_info *sbi = inode->i_sb->s_fs_info; pcol->sbi = sbi; - pcol->req_q =osd_request_queue (sbi->s_dev); +/* Create master bios on first Q, later on cloning, each clone will be +* allocated on it's destination Q +*/ + pcol->req_q =osd_request_queue (sbi->s_ods[0 ]); pcol->inode = inode; pcol->expected_pages = expected_pages;
fs/exofs/ios.c +24 -11
@@ -71,7 +71,7 @@ int exofs_get_io_state(struct exofs_sb_info *sbi, struct exofs_io_state** pios)/*TODO: Maybe use kmem_cach per sbi of size * exofs_io_state_size(sbi->s_numdevs) */ - ios =kzalloc (exofs_io_state_size (1 ),GFP_KERNEL ); + ios =kzalloc (exofs_io_state_size (sbi->s_numdevs),GFP_KERNEL );if (unlikely (!ios)) { *pios =NULL ;return -ENOMEM ;
@@ -203,10 +203,10 @@ int exofs_sbi_create(struct exofs_io_state *ios)kref_init (&ios->kref); -for (i =0 ; i <1 ; i++) { +for (i =0 ; i < ios->sbi->s_numdevs; i++) {struct osd_request *or; - or =osd_start_request (ios->sbi->s_dev,GFP_KERNEL ); + or =osd_start_request (ios->sbi->s_ods[i],GFP_KERNEL );if (unlikely (!or)) {EXOFS_ERR ("%s: osd_start_request failed\n" , __func__); ret = -ENOMEM ;
@@ -241,10 +241,10 @@ int exofs_sbi_remove(struct exofs_io_state *ios)kref_init (&ios->kref); -for (i =0 ; i <1 ; i++) { +for (i =0 ; i < ios->sbi->s_numdevs; i++) {struct osd_request *or; - or =osd_start_request (ios->sbi->s_dev,GFP_KERNEL ); + or =osd_start_request (ios->sbi->s_ods[i],GFP_KERNEL );if (unlikely (!or)) {EXOFS_ERR ("%s: osd_start_request failed\n" , __func__); ret = -ENOMEM ;
@@ -278,10 +278,10 @@ int exofs_sbi_write(struct exofs_io_state *ios)kref_init (&ios->kref); -for (i =0 ; i <1 ; i++) { +for (i =0 ; i < ios->sbi->s_numdevs; i++) {struct osd_request *or; - or =osd_start_request (ios->sbi->s_dev,GFP_KERNEL ); + or =osd_start_request (ios->sbi->s_ods[i],GFP_KERNEL );if (unlikely (!or)) {EXOFS_ERR ("%s: osd_start_request failed\n" , __func__); ret = -ENOMEM ;
@@ -293,7 +293,19 @@ int exofs_sbi_write(struct exofs_io_state *ios)if (ios->bio) {struct bio *bio; - bio = ios->bio; +if (i !=0 ) { + bio =bio_kmalloc (GFP_KERNEL , + ios->bio->bi_max_vecs); +if (!bio) + goto out; + +__bio_clone (bio, ios->bio); + bio->bi_bdev =NULL ; + bio->bi_next =NULL ; + ios->per_dev[i].bio = bio; + }else { + bio = ios->bio; + }osd_req_write (or, &ios->obj, ios->offset, bio, ios->length);
@@ -340,8 +352,9 @@ int exofs_sbi_read(struct exofs_io_state *ios)for (i =0 ; i <1 ; i++) {struct osd_request *or; +unsigned first_dev = ios->obj.id % ios->sbi->s_numdevs; - or =osd_start_request (ios->sbi->s_dev,GFP_KERNEL ); + or =osd_start_request (ios->sbi->s_ods[first_dev],GFP_KERNEL );if (unlikely (!or)) {EXOFS_ERR ("%s: osd_start_request failed\n" , __func__); ret = -ENOMEM ;
@@ -426,10 +439,10 @@ int exofs_oi_truncate(struct exofs_i_info *oi, u64 size)kref_init (&ios->kref); -for (i =0 ; i <1 ; i++) { +for (i =0 ; i < sbi->s_numdevs; i++) {struct osd_request *or; - or =osd_start_request (sbi->s_dev,GFP_KERNEL ); + or =osd_start_request (sbi->s_ods[i],GFP_KERNEL );if (unlikely (!or)) {EXOFS_ERR ("%s: osd_start_request failed\n" , __func__); ret = -ENOMEM ;
fs/exofs/pnfs.h new file +51 -0
51 lines not on this page · raw message
fs/exofs/super.c +212 -8
280 lines not on this page · raw message
Signature · 1 line
1.6.5.1
PATCH 2/3 exofs: Move all operations to an io_engine
In reply to Boaz HarroshIn anticipation for multi-device operations, we separate osd operations into an abstract I/O API. Currently only one device is used but later when adding more devices, we will drive all devices in parallel according to a "data_map" that describes how data is arranged on multiple devices. The file system level operates, like before, as if there is one object (inode-number) and an i_size. The io engine will split this to the same object-number but on multiple device.
At first we introduce Mirror (raid 1) layout. But at the final outcome we intend to fully implement the pNFS-Objects data-map, including raid 0,4,5,6 over mirrored devices, over multiple device-groups. And more. See: http://tools.ietf.org/html/draft-ietf-nfsv4-pnfs-obj-12
* Define an io_state based API for accessing osd storage devices in an abstract way.
Usage:
First a caller allocates an io state with: exofs_get_io_state(struct exofs_sb_info *sbi, struct exofs_io_state** ios); Then calles one of: exofs_sbi_create(struct exofs_io_state *ios); exofs_sbi_remove(struct exofs_io_state *ios); exofs_sbi_write(struct exofs_io_state *ios); exofs_sbi_read(struct exofs_io_state *ios); exofs_oi_truncate(struct exofs_i_info *oi, u64 new_len); And when done exofs_put_io_state(struct exofs_io_state *ios);
* Convert all source files to use this new API
* Convert from bio_alloc to bio_kmalloc
* In io engine we make use of the now fixed osd_req_decode_sense
There are no functional changes or on disk additions after this patch.
--- fs/exofs/common.h | 18 --- fs/exofs/exofs.h | 87 ++++++++++++- fs/exofs/inode.c | 383 +++++++++++++++++++++++++--------------------------- fs/exofs/ios.c | 386 +++++++++++++++++++++++++++++++++++++++++++++-------- fs/exofs/super.c | 120 ++++++----------- 5 files changed, 644 insertions(+), 350 deletions(-)
fs/exofs/common.h +0 -18
@@ -155,22 +155,4 @@ enum { (((name_len) + offsetof(struct exofs_dir_entry , name) + \ EXOFS_DIR_ROUND) & ~EXOFS_DIR_ROUND) -/************************* -* function declarations * -*************************/ -/* osd.c */ -void exofs_make_credential (u8 cred_a[OSD_CAP_LEN ], -const struct osd_obj_id *obj); - -int exofs_check_ok_resid (struct osd_request *or,u64 *in_resid,u64 *out_resid); -static inline int exofs_check_ok (struct osd_request *or) -{ -return exofs_check_ok_resid (or,NULL ,NULL ); -} -int exofs_sync_op (struct osd_request *or,int timeout,u8 *cred); -int exofs_async_op (struct osd_request *or, -osd_req_done_fn *async_done,void *caller_context,u8 *cred); - -int extract_attr_from_req (struct osd_request *or,struct osd_attr *attr); -#endif /*ifndef __EXOFS_COM_H__*/
fs/exofs/exofs.h +84 -3
122 lines not on this page · raw message
fs/exofs/inode.c +184 -199
768 lines not on this page · raw message
fs/exofs/ios.c +333 -53
431 lines not on this page · raw message
fs/exofs/super.c +43 -77
220 lines not on this page · raw message
Signature · 1 line
1.6.5.2
PATCH 3/3 exofs: Multi-device mirror support
In reply to Boaz HarroshThis patch changes on-disk format, it is accompanied with a parallel patch to mkfs.exofs that enables multi-device capabilities.
After this patch, old exofs will refuse to mount a new formatted FS and new exofs will refuse an old format. This is done by moving the magic field offset inside the FSCB. A new FSCB *version* field was added. In the future, exofs will refuse to mount unmatched FSCB version. To up-grade or down-grade an exofs one must use mkfs.exofs --upgrade option before mounting.
Introduced, a new object that contains a *device-table*. This object contains the default *data-map* and a linear array of devices information, which identifies the devices used in the filesystem. This object is only written to offline by mkfs.exofs. This is why it is kept separate from the FSCB, since the later is written to while mounted.
Same partition number, same object number is used on all devices only the device varies.
* define the new format, then load the device table on mount time make sure every thing is supported.
* Change I/O engine to now support Mirror IO, .i.e write same data to multiple devices, read from a random device to spread the read-load from multiple clients (TODO: stripe read)
Implementation notes:
A few points introduced in previous patch should be mentioned here:
* Special care was made so absolutlly all operation that have any chance of failing are done before any osd-request is executed. This is to minimize the need for a data consistency recovery, to only real IO errors.
* Each IO state has a kref. It starts at 1, any osd-request executed will increment the kref, finally when all are executed the first ref is dropped. At IO-done, each request completion decrements the kref, the last one to return executes the internal _last_io() routine. _last_io() will call the registered io_state_done. On sync mode a caller does not supply a done method, indicating a synchronous request, the caller is put to sleep and a special io_state_done is registered that will awaken the caller. Though also in sync mode all operations are executed in parallel.
--- fs/exofs/common.h | 63 ++++++++++++++-- fs/exofs/exofs.h | 12 +++- fs/exofs/inode.c | 5 +- fs/exofs/ios.c | 37 ++++++--- fs/exofs/pnfs.h | 51 ++++++++++++ fs/exofs/super.c | 220 +++++++++++++++++++++++++++++++++++++++++++++++++++-- 6 files changed, 360 insertions(+), 28 deletions(-) create mode 100644 fs/exofs/pnfs.h
fs/exofs/common.h +57 -6
@@ -49,6 +49,7 @@#define EXOFS_MIN_PID 0x10000/* Smallest partition ID */ #define EXOFS_OBJ_OFF 0x10000/* offset for objects */ #define EXOFS_SUPER_ID 0x10000/* object ID for on-disk superblock */ +#define EXOFS_DEVTABLE_ID 0x10001/* object ID for on-disk device table */ #define EXOFS_ROOT_ID 0x10002/* object ID for root directory */ /* exofs Application specific page/attribute */
@@ -78,17 +79,67 @@ enum {#define EXOFS_SUPER_MAGIC 0x5DF5/* -* The file system control block - stored in an object's data (mainly, the one -* with ID EXOFS_SUPER_ID). This is where the in-memory superblock is stored -* on disk. Right now it just has a magic value, which is basically a sanity -* check on our ability to communicate with the object store. +* The file system control block - stored in object EXOFS_SUPER_ID's data. +* This is where the in-memory superblock is stored on disk. */ +enum {EXOFS_FSCB_VER =1 ,EXOFS_DT_VER =1 };struct exofs_fscb {__le64 s_nextid;/* Highest object ID used */ -__le32 s_numfiles;/* Number of files on fs */ +__le64 s_numfiles;/* Number of files on fs */ +__le32 s_version;/* == EXOFS_FSCB_VER */ __le16 s_magic;/* Magic signature */ __le16 s_newfs;/* Non-zero if this is a new fs */ -}; + +/* From here on it's a static part, only written by mkexofs */ +__le64 s_dev_table_oid;/* Resurved, not used */ +__le64 s_dev_table_count;/* == 0 means no dev_table */ +} __packed; + +/* +* Describes the raid used in the FS. It is part of the device table. +* This here is taken from the pNFS-objects definition. In exofs we +* use one raid policy through-out the filesystem. (NOTE: the funny +* alignment at begining. We take care of it at exofs_device_table. +*/ +struct exofs_dt_data_map { +__le32 cb_num_comps; +__le64 cb_stripe_unit; +__le32 cb_group_width; +__le32 cb_group_depth; +__le32 cb_mirror_cnt; +__le32 cb_raid_algorithm; +} __packed; + +/* +* This is an osd device information descriptor. It is a single entry in +* the exofs device table. It describes an osd target lun which +* contains data belonging to this FS. (Same partition_id on all devices) +*/ +struct exofs_dt_device_info { +__le32 systemid_len; +u8 systemid[OSD_SYSTEMID_LEN ]; +__le64 long_name_offset;/* If !0 then offset-in-file */ +__le32 osdname_len;/* */ +u8 osdname[44 ];/* Embbeded, Ususally an asci uuid */ +} __packed; + +/* +* The EXOFS device table - stored in object EXOFS_DEVTABLE_ID's data. +* It contains the raid used for this multy-device FS and an array of +* participating devices. +*/ +struct exofs_device_table { +__le32 dt_version;/* == EXOFS_DT_VER */ +struct exofs_dt_data_map dt_data_map;/* Raid policy to use */ + +/* Resurved space For future use. Total includeing this: +* (8 * sizeof(le64)) +*/ +__le64 __Resurved[4 ]; + +__le64 dt_num_devices;/* Array size */ +struct exofs_dt_device_info dt_dev_table[];/* Array of devices */ +} __packed;/**************************************************************************** * inode-related things
fs/exofs/exofs.h +10 -2
@@ -37,6 +37,11 @@#include <linux/time.h> #include "common.h" +/* FIXME: Remove once pnfs hits mainline +* #include <linux/exportfs/pnfs_osd_xdr.h> +*/ +#include "pnfs.h" + #define EXOFS_ERR(fmt, a...) printk(KERN_ERR "exofs: " fmt, ##a)#ifdef CONFIG_EXOFS_DEBUG
@@ -54,7 +59,6 @@* our extension to the in-memory superblock */ struct exofs_sb_info { -struct osd_dev *s_dev;/* returned by get_osd_dev */ struct exofs_fscb s_fscb;/* Written often, pre-allocate*/ osd_id s_pid;/* partition ID of file system*/ int s_timeout;/* timeout for OSD operations */
@@ -63,7 +67,11 @@ struct exofs_sb_info {spinlock_t s_next_gen_lock;/* spinlock for gen # update */ u32 s_next_generation;/* next gen # to use */ atomic_t s_curr_pending;/* number of pending commands */ -uint8_t s_cred[OSD_CAP_LEN ];/* all-powerful credential */ +uint8_t s_cred[OSD_CAP_LEN ];/* credential for the fscb */ + +struct pnfs_osd_data_map data_map;/* Default raid to use */ +unsigned s_numdevs;/* Num of devices in array */ +struct osd_dev *s_ods[1 ];/* Variable length, minimum 1 */ };/*
fs/exofs/inode.c +4 -1
@@ -62,7 +62,10 @@ static void _pcol_init(struct page_collect *pcol, unsigned expected_pages,struct exofs_sb_info *sbi = inode->i_sb->s_fs_info; pcol->sbi = sbi; - pcol->req_q =osd_request_queue (sbi->s_dev); +/* Create master bios on first Q, later on cloning, each clone will be +* allocated on it's destination Q +*/ + pcol->req_q =osd_request_queue (sbi->s_ods[0 ]); pcol->inode = inode; pcol->expected_pages = expected_pages;
fs/exofs/ios.c +26 -11
@@ -71,7 +71,7 @@ int exofs_get_io_state(struct exofs_sb_info *sbi, struct exofs_io_state** pios)/*TODO: Maybe use kmem_cach per sbi of size * exofs_io_state_size(sbi->s_numdevs) */ - ios =kzalloc (exofs_io_state_size (1 ),GFP_KERNEL ); + ios =kzalloc (exofs_io_state_size (sbi->s_numdevs),GFP_KERNEL );if (unlikely (!ios)) { *pios =NULL ;return -ENOMEM ;
@@ -209,10 +209,10 @@ int exofs_sbi_create(struct exofs_io_state *ios) {int i, ret; -for (i =0 ; i <1 ; i++) { +for (i =0 ; i < ios->sbi->s_numdevs; i++) {struct osd_request *or; - or =osd_start_request (ios->sbi->s_dev,GFP_KERNEL ); + or =osd_start_request (ios->sbi->s_ods[i],GFP_KERNEL );if (unlikely (!or)) {EXOFS_ERR ("%s: osd_start_request failed\n" , __func__); ret = -ENOMEM ;
@@ -233,10 +233,10 @@ int exofs_sbi_remove(struct exofs_io_state *ios) {int i, ret; -for (i =0 ; i <1 ; i++) { +for (i =0 ; i < ios->sbi->s_numdevs; i++) {struct osd_request *or; - or =osd_start_request (ios->sbi->s_dev,GFP_KERNEL ); + or =osd_start_request (ios->sbi->s_ods[i],GFP_KERNEL );if (unlikely (!or)) {EXOFS_ERR ("%s: osd_start_request failed\n" , __func__); ret = -ENOMEM ;
@@ -257,10 +257,10 @@ int exofs_sbi_write(struct exofs_io_state *ios) {int i, ret; -for (i =0 ; i <1 ; i++) { +for (i =0 ; i < ios->sbi->s_numdevs; i++) {struct osd_request *or; - or =osd_start_request (ios->sbi->s_dev,GFP_KERNEL ); + or =osd_start_request (ios->sbi->s_ods[i],GFP_KERNEL );if (unlikely (!or)) {EXOFS_ERR ("%s: osd_start_request failed\n" , __func__); ret = -ENOMEM ;
@@ -272,7 +272,21 @@ int exofs_sbi_write(struct exofs_io_state *ios)if (ios->bio) {struct bio *bio; - bio = ios->bio; +if (i !=0 ) { + bio =bio_kmalloc (GFP_KERNEL , + ios->bio->bi_max_vecs); +if (unlikely (!bio)) { + ret = -ENOMEM ; + goto out; + } + +__bio_clone (bio, ios->bio); + bio->bi_bdev =NULL ; + bio->bi_next =NULL ; + ios->per_dev[i].bio = bio; + }else { + bio = ios->bio; + }osd_req_write (or, &ios->obj, ios->offset, bio, ios->length);
@@ -306,8 +320,9 @@ int exofs_sbi_read(struct exofs_io_state *ios)for (i =0 ; i <1 ; i++) {struct osd_request *or; +unsigned first_dev = ios->obj.id % ios->sbi->s_numdevs; - or =osd_start_request (ios->sbi->s_dev,GFP_KERNEL ); + or =osd_start_request (ios->sbi->s_ods[first_dev],GFP_KERNEL );if (unlikely (!or)) {EXOFS_ERR ("%s: osd_start_request failed\n" , __func__); ret = -ENOMEM ;
@@ -382,10 +397,10 @@ int exofs_oi_truncate(struct exofs_i_info *oi, u64 size) attr = g_attr_logical_length; attr.val_ptr = &newsize; -for (i =0 ; i <1 ; i++) { +for (i =0 ; i < sbi->s_numdevs; i++) {struct osd_request *or; - or =osd_start_request (sbi->s_dev,GFP_KERNEL ); + or =osd_start_request (sbi->s_ods[i],GFP_KERNEL );if (unlikely (!or)) {EXOFS_ERR ("%s: osd_start_request failed\n" , __func__); ret = -ENOMEM ;
fs/exofs/pnfs.h new file +51 -0
51 lines not on this page · raw message
fs/exofs/super.c +212 -8
280 lines not on this page · raw message
Signature · 1 line
1.6.5.2
Re: [osd-dev] [PATCH 2/3] exofs: Move all operations to an io_engine
In reply to Boaz HarroshQuoted 31 lines · Boaz Harrosh
In anticipation for multi-device operations, we separate osd operations into an abstract I/O API. Currently only one device is used but later when adding more devices, we will drive all devices in parallel according to a "
data_map" that describes how data is arranged on multiple devices. The file system level operates, like before, as if there is one object (inode-number) and ani_size. The io engine will split this to the same object-number but on multiple device.At first we introduce Mirror (raid 1) layout. But at the final outcome we intend to fully implement the pNFS-Objects data-map, including raid 0,4,5,6 over mirrored devices, over multiple device-groups. And more. See: http://tools.ietf.org/html/draft-ietf-nfsv4-pnfs-obj-12
* Define an
io_statebased API for accessing osd storage devices in an abstract way.
Usage:First a caller allocates an io state with: exofs_get_io_state(struct exofs_sb_info *sbi, struct exofs_io_state** ios); Then calles one of: exofs_sbi_create(struct exofs_io_state *ios); exofs_sbi_remove(struct exofs_io_state *ios); exofs_sbi_write(struct exofs_io_state *ios); exofs_sbi_read(struct exofs_io_state *ios); exofs_oi_truncate(struct exofs_i_info *oi, u64 new_len); And when done exofs_put_io_state(struct exofs_io_state *ios);
Quoted 7 lines
* Convert all source files to use this new API
* Convert frombio_alloctobio_kmalloc
* In io engine we make use of the now fixedosd_req_decode_senseThere are no functional changes or on disk additions after this patch.
Signed-off-by Boaz Harrosh <[email protected]>
Putting my mailing-list-reader hat produced some "What was that good for?"
Below is the diff of the new posts from the old ones. (New as replay to originals)
---
git diff --stat -p -M -R afb8f9c089c07711104bb26e224440415ba3d324 -- fs/exofs/
fs/exofs/inode.c | 14 ++++++-- fs/exofs/ios.c | 93 +++++++++++++---------------------------------------- 2 files changed, 33 insertions(+), 74 deletions(-)
fs/exofs/inode.c +10 -4
@@ -444,7 +444,7 @@ static int exofs_readpage(struct file *file, struct page *page)return _readpage (page, false); } -/* Callback for osd_write. All writes are asynchronouse */ +/* Callback for osd_write. All writes are asynchronous */ static void writepages_done (struct exofs_io_state *ios,void *p) {struct page_collect *pcol = p;
@@ -1029,9 +1029,15 @@ static void create_done(struct exofs_io_state *ios, void *p)if (unlikely (ret)) {EXOFS_ERR ("object=0x%llx creation faild in pid=0x%llx" ,_LLU (exofs_oi_objno (oi)),_LLU (sbi->s_pid)); -make_bad_inode (inode); - }else -set_obj_created (oi); +/*TODO: When FS is corrupted creation can fail, object already +* exist. Get rid of this asynchronous creation, if exist +* increment the obj counter and try the next object. Until we +* succeed. All these dangling objects will be made into lost +* files by chkfs.exofs +*/ + } + +set_obj_created (oi);atomic_dec (&inode->i_count);wake_up (&oi->i_wq);
fs/exofs/ios.c +23 -70
187 lines not on this page · raw message