osdblk: a Linux block device for OSD objects

PATCH In review Boaz Harrosh 2 participants 9 messages May 21, 2009 – Jun 10, 2009

Subsystems (2)

SCSI Block layer

lore

Share & tools

lore

Boaz Harrosh

author

2009-05-21 14:06 UTC #0 lore raw

cover osdblk: a Linux block device for OSD objects

I'm posting for review a new version of the osdblk driver. What's new?

* Once block/for-2.6.31 and all pending osd patches hit mainline. this new version is ready for submission.
- The relevant osd patches have been posted on the mailing list, but I'll send an orderly set for scsi-misc and scsi-post-merge on Sunday.
- All the prerequisite block patches are already in Jens's tree.

* Below is the diff from Jeff's last version of the patch. these things have changed:
{SQUASHME: osdblk} Block and OSD Api fixups and bug fixes

- Block API changes from Tejuns revamps
- OSD Api changes for supporting bio-chaining
- do_flush requests do not need bio clonning
(And might not have any so prevent such a crash)
- osdblk_make_credential is here to stay
- Use bio_kmalloc and avoid the bio_alloc dead/live locks.
TODO: Split request into smaller chunks if allocations fail.
- Only use __GFP_WAIT on first bio allocation. (Not relevant since __GFP_WAIT is not used)

* Added an extra patch:
- [PATCH 2/2] osdblk: Adjust queue limits to lower device's limits

This is ontop of the post-merge tree. Jeff? will you push this driver through your tree?

What is left is to bang some serious testing on this driver. I'll do that next.

Thanks
Boaz

---
 drivers/block/osdblk.c |   48 +++++++++++++++++++++---------------------------
 1 files changed, 21 insertions(+), 27 deletions(-)

---

drivers/block/osdblk.c +21 ~~-27~~

index c7a1bb7..531d234 100644
@@ -118,13 +118,13 @@ static struct block_device_operations osdblk_bd_ops = {
 static const struct osd_attr g_attr_logical_length = ATTR_DEF(
 	OSD_APAGE_OBJECT_INFORMATION, OSD_ATTR_OI_LOGICAL_LENGTH, 8);
 
-/* copied from exofs; move to libosd? */
-static void osd_make_credential(u8 cred_a[OSD_CAP_LEN],
-				const struct osd_obj_id *obj)
+static void osdblk_make_credential(u8 cred_a[OSD_CAP_LEN],
+				   const struct osd_obj_id *obj)
 {
 	osd_sec_init_nosec_doall_caps(cred_a, obj, false, true);
 }
 
+/* copied from exofs; move to libosd? */
 /*
  * Perform a synchronous OSD operation.  copied from exofs; move to libosd?
  */

@@ -216,17 +216,6 @@ out:
 
 }
 
-static void osdblk_end_request(struct osdblk_device *osdev,
-			       struct osdblk_request *orq,
-			       int error)
-{
-	struct request *rq = orq->rq;
-	int rc;
-
-	/* complete request, at block layer */
-	rc = __blk_end_request(rq, error, blk_rq_bytes(rq));
-}
-
 static void osdblk_osd_complete(struct osd_request *or, void *private)
 {
 	struct osdblk_request *orq = private;

@@ -240,7 +229,7 @@ static void osdblk_osd_complete(struct osd_request *or, void *private)
 	osd_end_request(or);
 
 	/* complete request passed to osdblk by block layer */
-	osdblk_end_request(orq->osdev, orq, ret);
+	__blk_end_request_all(orq->rq, ret);
 }
 
 static void bio_chain_put(struct bio *chain)

@@ -260,10 +249,12 @@ static struct bio *bio_chain_clone(struct bio *old_chain, gfp_t gfpmask)
 	struct bio *tmp, *new_chain = NULL, *tail = NULL;
 
 	while (old_chain) {
-		tmp = bio_clone(old_chain, gfpmask);
+		tmp = bio_kmalloc(gfpmask, old_chain->bi_vcnt);
 		if (!tmp)
 			goto err_out;
 
+		__bio_clone(tmp, old_chain);
+		gfpmask &= ~__GFP_WAIT;
 		tmp->bi_next = NULL;
 		if (!new_chain)
 			new_chain = tail = tmp;

@@ -293,13 +284,13 @@ static void osdblk_rq_fn(struct request_queue *q)
 
 	while (1) {
 		/* peek at request from block layer */
-		rq = elv_next_request(q);
+		rq = blk_fetch_request(q);
 		if (!rq)
 			break;
 
 		/* filter out block requests we don't understand */
 		if (!blk_fs_request(rq) && !blk_barrier_rq(rq)) {
-			end_request(rq, 0);
+			blk_end_request_all(rq, 0);
 			continue;
 		}

@@ -313,10 +304,13 @@ static void osdblk_rq_fn(struct request_queue *q)
 		do_flush = (rq->special == (void *) 0xdeadbeefUL);
 		do_write = (rq_data_dir(rq) == WRITE);
 
-		/* a bio clone to be passed down to OSD request */
-		bio = bio_chain_clone(rq->bio, GFP_ATOMIC);
-		if (!bio)
-			break;
+		if (!do_flush) { /* osd_flush does not use a bio */
+			/* a bio clone to be passed down to OSD request */
+			bio = bio_chain_clone(rq->bio, GFP_ATOMIC);
+			if (!bio)
+				break;
+		} else
+			bio = NULL;
 
 		/* alloc internal OSD request, for OSD command execution */
 		or = osd_start_request(osdev->osd, GFP_ATOMIC);

@@ -335,11 +329,11 @@ static void osdblk_rq_fn(struct request_queue *q)
 			osd_req_flush_object(or, &osdev->obj,
 					     OSD_CDB_FLUSH_ALL, 0, 0);
 		else if (do_write)
-			osd_req_write(or, &osdev->obj, bio,
-				      rq->sector * 512ULL);
+			osd_req_write(or, &osdev->obj, blk_rq_pos(rq) * 512ULL,
+				      bio, blk_rq_bytes(rq));
 		else
-			osd_req_read(or, &osdev->obj, bio,
-				     rq->sector * 512ULL);
+			osd_req_read(or, &osdev->obj, blk_rq_pos(rq) * 512ULL,
+				     bio, blk_rq_bytes(rq));
 
 		/* begin OSD command execution */
 		if (osd_async_op(or, osdblk_osd_complete, orq,

@@ -527,7 +521,7 @@ static ssize_t class_osdblk_add(struct class *c, const char *buf, size_t count)
 	}
 
 	/* build OSD credential */
-	osd_make_credential(osdev->obj_cred, &osdev->obj);
+	osdblk_make_credential(osdev->obj_cred, &osdev->obj);
 
 	/* register our block device */
 	irc = register_blkdev(0, osdev->name);

Boaz Harrosh

author

2009-05-21 14:09 UTC #1 lore raw

PATCH 1/2 osdblk: a Linux block device for OSD objects

From: Jeff Garzik <[email protected]>

TODO: Commit log

NOT-Signed-off-by Jeff Garzik <[email protected]>

WILL-Signed-off-by Boaz Harrosh <[email protected]>

---
 drivers/block/Kconfig  |   16 ++
 drivers/block/Makefile |    1 +
 drivers/block/osdblk.c |  663 ++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 680 insertions(+), 0 deletions(-)
 create mode 100644 drivers/block/osdblk.c

drivers/block/Kconfig +16 -0

index f42fa50..15d1b94 100644
@@ -298,6 +298,22 @@ config BLK_DEV_NBD
 
 	  If unsure, say N.
 
+config BLK_DEV_OSD
+	tristate "OSD object-as-blkdev support"
+	depends on SCSI_OSD_INITIATOR
+	---help---
+	  Saying Y or M here will allow the exporting of a single SCSI
+	  OSD (object-based storage) object as a Linux block device.
+
+	  For example, if you create a 2G object on an OSD device,
+	  you can then use this module to present that 2G object as
+	  a Linux block device.
+
+	  To compile this driver as a module, choose M here: the
+	  module will be called osdblk.
+
+	  If unsure, say N.
+
 config BLK_DEV_SX8
 	tristate "Promise SATA SX8 support"
 	depends on PCI

drivers/block/Makefile +1 -0

index 7755a5e..cdaa3f8 100644
@@ -23,6 +23,7 @@ obj-$(CONFIG_XILINX_SYSACE)	+= xsysace.o
 obj-$(CONFIG_CDROM_PKTCDVD)	+= pktcdvd.o
 obj-$(CONFIG_MG_DISK)		+= mg_disk.o
 obj-$(CONFIG_SUNVDC)		+= sunvdc.o
+obj-$(CONFIG_BLK_DEV_OSD)	+= osdblk.o
 
 obj-$(CONFIG_BLK_DEV_UMEM)	+= umem.o
 obj-$(CONFIG_BLK_DEV_NBD)	+= nbd.o

drivers/block/osdblk.c new file +663 -0

663 lines not on this page · raw message

Signature · 1 line

1.6.2.1

Boaz Harrosh

author

2009-05-21 14:11 UTC #2 lore raw

PATCH 2/2 osdblk: Adjust queue limits to lower device's limits

In reply to Boaz Harrosh

call blk_queue_stack_limits() to copy queue limits from the underline osd scsi_device. This is absolutely needed because osdblk cannot sleep when allocating a lower-request and therefore cannot be bouncing.

TODO: Dynamic changes of limits to the lower device queue will not reflect in the upper driver

Signed-off-by Boaz Harrosh <[email protected]>

---
 drivers/block/osdblk.c |    7 +++++++
 1 files changed, 7 insertions(+), 0 deletions(-)

drivers/block/osdblk.c +7 -0

index 531d234..0da8ce0 100644
@@ -66,6 +66,7 @@
 #include <scsi/osd_initiator.h>
 #include <scsi/osd_attributes.h>
 #include <scsi/osd_sec.h>
+#include <scsi/scsi_device.h>
 
 #define DRV_NAME "osdblk"
 #define PFX DRV_NAME ": "

@@ -410,6 +411,12 @@ static int osdblk_init_disk(struct osdblk_device *osdev)
 		return rc;
 	}
 
+	/* Set our limits to the lower device limits, because osdblk cannot
+	 * sleep when allocating a lower-request and therefore cannot be
+	 * bouncing.
+	 */
+	blk_queue_stack_limits(q, osd_request_queue(osdev->osd));
+
 	blk_queue_prep_rq(q, blk_queue_start_tag);
 	blk_queue_ordered(q, QUEUE_ORDERED_DRAIN_FLUSH, osdblk_prepare_flush);

Signature · 1 line

1.6.2.1

Jeff Garzik

2009-05-21 22:41 UTC #3 lore raw

In reply to Boaz Harrosh

Quoted 21 lines · Boaz Harrosh

I'm posting for review a new version of the osdblk driver. What's new?
* Once block/for-2.6.31 and all pending osd patches hit mainline. this new version is ready for submission.
- The relevant osd patches have been posted on the mailing list, but I'll send an orderly set for scsi-misc and scsi-post-merge on Sunday.
- All the prerequisite block patches are already in Jens's tree.
* Below is the diff from Jeff's last version of the patch. these things have changed:
{SQUASHME: osdblk} Block and OSD Api fixups and bug fixes
- Block API changes from Tejuns revamps
- OSD Api changes for supporting bio-chaining
- do_flush requests do not need bio clonning
(And might not have any so prevent such a crash)
- osdblk_make_credential is here to stay
- Use bio_kmalloc and avoid the bio_alloc dead/live locks.
TODO: Split request into smaller chunks if allocations fail.
- Only use __GFP_WAIT on first bio allocation. (Not relevant since __GFP_WAIT is not used)

Quoted 8 lines

* Added an extra patch:
- [PATCH 2/2] osdblk: Adjust queue limits to lower device's limits
This is ontop of the post-merge tree. Jeff? will you push this driver through your tree?
What is left is to bang some serious testing on this driver. I'll do that next.

The changes look reasonable to me... if you wanted to get it into your tree and push it with other OSD stuff, that would be fine to me.

I think you are in a better position to deal with all the pre-req's, and in a better position to test osdblk more completely.

Have you messed around with the user tools yet? osdblk needs a tool that creates an OSD object of a specified size, etc.

Jeff

Boaz Harrosh

author

2009-05-26 07:43 UTC #4 lore raw

Quoted 34 lines · Jeff Garzik

Quoted 21 lines · Boaz Harrosh
I'm posting for review a new version of the osdblk driver. What's new?
* Once block/for-2.6.31 and all pending osd patches hit mainline. this new version is ready for submission.
- The relevant osd patches have been posted on the mailing list, but I'll send an orderly set for scsi-misc and scsi-post-merge on Sunday.
- All the prerequisite block patches are already in Jens's tree.
* Below is the diff from Jeff's last version of the patch. these things have changed:
{SQUASHME: osdblk} Block and OSD Api fixups and bug fixes
- Block API changes from Tejuns revamps
- OSD Api changes for supporting bio-chaining
- do_flush requests do not need bio clonning
(And might not have any so prevent such a crash)
- osdblk_make_credential is here to stay
- Use bio_kmalloc and avoid the bio_alloc dead/live locks.
TODO: Split request into smaller chunks if allocations fail.
- Only use __GFP_WAIT on first bio allocation. (Not relevant since __GFP_WAIT is not used)
Quoted 8 lines
* Added an extra patch:
- [PATCH 2/2] osdblk: Adjust queue limits to lower device's limits
This is ontop of the post-merge tree. Jeff? will you push this driver through your tree?
What is left is to bang some serious testing on this driver. I'll do that next.
The changes look reasonable to me... if you wanted to get it into your
tree and push it with other OSD stuff, that would be fine to me.

Quoted 8 lines

I think you are in a better position to deal with all the pre-req's, and in a better position to test osdblk more completely.
Have you messed around with the user tools yet? osdblk needs a tool that creates an OSD object of a specified size, etc.
Jeff

Thanks Jeff.

So is this a:

Signed-off-by Jeff Garzik <[email protected]>

But please send me a commit log, or should I scribble one?

About the user-mode tool: Sorry I'm so very busy with the pNFS layout driver and export that I do not have time for it right now.

For testing I just use exofs, create a file and dd to some offset to make it of some size. Very very stupid I know, but easy. (The obj-id I can guess as I know the code)

Tell me if you absolutely need a fast hack that just takes the ids and size and creates one object, for now.

Boaz

Jeff Garzik

2009-05-28 22:31 UTC #5 lore raw

Quoted 37 lines · Boaz Harrosh

Quoted 36 lines · Jeff Garzik
Quoted 21 lines · Boaz Harrosh
I'm posting for review a new version of the osdblk driver. What's new?
* Once block/for-2.6.31 and all pending osd patches hit mainline. this new version is ready for submission.
- The relevant osd patches have been posted on the mailing list, but I'll send an orderly set for scsi-misc and scsi-post-merge on Sunday.
- All the prerequisite block patches are already in Jens's tree.
* Below is the diff from Jeff's last version of the patch. these things have changed:
{SQUASHME: osdblk} Block and OSD Api fixups and bug fixes
- Block API changes from Tejuns revamps
- OSD Api changes for supporting bio-chaining
- do_flush requests do not need bio clonning
(And might not have any so prevent such a crash)
- osdblk_make_credential is here to stay
- Use bio_kmalloc and avoid the bio_alloc dead/live locks.
TODO: Split request into smaller chunks if allocations fail.
- Only use __GFP_WAIT on first bio allocation. (Not relevant since __GFP_WAIT is not used)
Quoted 8 lines
* Added an extra patch:
- [PATCH 2/2] osdblk: Adjust queue limits to lower device's limits
This is ontop of the post-merge tree. Jeff? will you push this driver through your tree?
What is left is to bang some serious testing on this driver. I'll do that next.
The changes look reasonable to me... if you wanted to get it into your tree and push it with other OSD stuff, that would be fine to me.
I think you are in a better position to deal with all the pre-req's, and in a better position to test osdblk more completely.
Have you messed around with the user tools yet? osdblk needs a tool that creates an OSD object of a specified size, etc.

Quoted 7 lines

Quoted 2 lines
Jeff
Thanks Jeff.
So is this a:
Signed-off-by Jeff Garzik <[email protected]>

Yes, if it works in your testing. Please make sure the commit is "From: Jeff Garzik" as well, thanks.

Quoted 1 line

But please send me a commit log, or should I scribble one?

Just a one-line "add osdblk, block driver for OSD objects" should be fine.

Quoted 9 lines

About the user-mode tool: Sorry I'm so very busy with the pNFS layout driver and export that I do not have time for it right now.
For testing I just use exofs, create a file and dd to some offset to make it of some size. Very very stupid I know, but easy. (The obj-id I can guess as I know the code)
Tell me if you absolutely need a fast hack that just takes the ids and size and creates one object, for now.

Some simple tool like that is needed; a user shouldn't have to know exofs just to be able to use osdblk :)

If nothing else appears, I'll whip something up before 2.6.31, but I was of course hoping to talk you into it, since you could do it faster and better than me :)

Jeff

Boaz Harrosh

author

2009-05-31 09:50 UTC #6 lore raw

Quoted 67 lines · Jeff Garzik

Quoted 40 lines · Boaz Harrosh
Quoted 39 lines · Jeff Garzik
Quoted 21 lines · Boaz Harrosh
I'm posting for review a new version of the osdblk driver. What's new?
* Once block/for-2.6.31 and all pending osd patches hit mainline. this new version is ready for submission.
- The relevant osd patches have been posted on the mailing list, but I'll send an orderly set for scsi-misc and scsi-post-merge on Sunday.
- All the prerequisite block patches are already in Jens's tree.
* Below is the diff from Jeff's last version of the patch. these things have changed:
{SQUASHME: osdblk} Block and OSD Api fixups and bug fixes
- Block API changes from Tejuns revamps
- OSD Api changes for supporting bio-chaining
- do_flush requests do not need bio clonning
(And might not have any so prevent such a crash)
- osdblk_make_credential is here to stay
- Use bio_kmalloc and avoid the bio_alloc dead/live locks.
TODO: Split request into smaller chunks if allocations fail.
- Only use __GFP_WAIT on first bio allocation. (Not relevant since __GFP_WAIT is not used)
Quoted 8 lines
* Added an extra patch:
- [PATCH 2/2] osdblk: Adjust queue limits to lower device's limits
This is ontop of the post-merge tree. Jeff? will you push this driver through your tree?
What is left is to bang some serious testing on this driver. I'll do that next.
The changes look reasonable to me... if you wanted to get it into your
tree and push it with other OSD stuff, that would be fine to me.
I think you are in a better position to deal with all the pre-req's, and in a better position to test osdblk more completely.
Have you messed around with the user tools yet? osdblk needs a tool that creates an OSD object of a specified size, etc.
Quoted 6 lines
Quoted 2 lines
Jeff
Thanks Jeff.
So is this a:
Signed-off-by Jeff Garzik <[email protected]>
Yes, if it works in your testing. Please make sure the commit is "From:
Jeff Garzik" as well, thanks.
Quoted 1 line
But please send me a commit log, or should I scribble one?
Just a one-line "add osdblk, block driver for OSD objects" should be fine.
Quoted 9 lines
About the user-mode tool: Sorry I'm so very busy with the pNFS layout driver and export that I do not have time for it right now.
For testing I just use exofs, create a file and dd to some offset to make it of some size. Very very stupid I know, but easy. (The obj-id I can guess as I know the code)
Tell me if you absolutely need a fast hack that just takes the ids and size and creates one object, for now.

Quoted 9 lines

Some simple tool like that is needed; a user shouldn't have to know exofs just to be able to use osdblk :)
If nothing else appears, I'll whip something up before 2.6.31, but I was of course hoping to talk you into it, since you could do it faster and better than me :)
Jeff

OK I will release something Quick-And-Dirty end of this week. It should do those three tasks osdblk needs:
Create Resize Delete object@partition@device

Boaz

Jeff Garzik

2009-06-10 12:52 UTC #7 lore raw

Quoted 3 lines · Boaz Harrosh

OK I will release something Quick-And-Dirty end of this week. It should do those three tasks osdblk needs:
Create Resize Delete object@partition@device

That works, thanks! Honestly, only create and delete are really _needed_ IMO.

Jeff

Boaz Harrosh

author

2009-06-10 13:33 UTC #8 lore raw

Quoted 11 lines · Jeff Garzik

Quoted 3 lines · Boaz Harrosh
OK I will release something Quick-And-Dirty end of this week. It should do those three tasks osdblk needs:
Create Resize Delete object@partition@device
That works, thanks! Honestly, only create and delete are really _needed_ IMO.
Jeff

I'll be on this ASAP. I have a working draft just needs a cleanup and submission, which is queued behind all these other pressing things.

Boaz