Skip to contentSkip to conversation

NBD Cleanup patch and bugfix in ll_rw_blk.c

PATCH Steve Whitehouse 5 participants 14 messages Feb 25, 2001 – Mar 1, 2001

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi,

Here is a new version of the patch I recently sent to the list with some NBD cleanups and a bug fix in ll_rw_blk.c. The changes to NBD have Pavel Machek's approval as I've left out the two changes as he suggested.

The bug fix in ll_rw_blk.c prevents hangs when using block devices which don't have plugging functions,

Steve.

------------------------------------------------------------------------------

drivers/block/ll_rw_blk.c +3 -0
diff -u -r linux-2.4.2/drivers/block/ll_rw_blk.c linux/drivers/block/ll_rw_blk.c
@@ -588,6 +588,9 @@
 	 * inserted at elevator_merge time
 	 */
 	list_add(&req->queue, insert_here);
+
+	if (!q->plugged && insert_here == &q->queue_head)
+		q->request_fn(q);
 }
 
 void inline blk_refill_freelist(request_queue_t *q, int rw)
drivers/block/nbd.c +12 -11
diff -u -r linux-2.4.2/drivers/block/nbd.c linux/drivers/block/nbd.c
@@ -29,7 +29,7 @@
 #include <linux/major.h>
 
 #include <linux/module.h>
-
+#include <linux/init.h>
 #include <linux/sched.h>
 #include <linux/fs.h>
 #include <linux/stat.h>
@@ -149,12 +149,13 @@
 {
 	int result;
 	struct nbd_request request;
+	unsigned long size = req->current_nr_sectors << 9;
 
 	DEBUG("NBD: sending control, ");
 	request.magic = htonl(NBD_REQUEST_MAGIC);
 	request.type = htonl(req->cmd);
 	request.from = cpu_to_be64( (u64) req->sector << 9);
-	request.len = htonl(req->current_nr_sectors << 9);
+	request.len = htonl(size);
 	memcpy(request.handle, &req, sizeof(req));
 
 	result = nbd_xmit(1, sock, (char *) &request, sizeof(request));
@@ -163,7 +164,7 @@
 
 	if (req->cmd == WRITE) {
 		DEBUG("data, ");
-		result = nbd_xmit(1, sock, req->buffer, req->current_nr_sectors << 9);
+		result = nbd_xmit(1, sock, req->buffer, size);
 		if (result <= 0)
 			FAIL("Send data failed.");
 	}
@@ -475,11 +476,7 @@
  *  (Just smiley confuses emacs :-)
  */
 
-#ifdef MODULE
-#define nbd_init init_module
-#endif
-
-int nbd_init(void)
+int __init nbd_init(void)
 {
 	int i;
 
@@ -526,8 +523,7 @@
 	return 0;
 }
 
-#ifdef MODULE
-void cleanup_module(void)
+void __exit nbd_cleanup(void)
 {
 	devfs_unregister (devfs_handle);
 	blk_cleanup_queue(BLK_DEFAULT_QUEUE(MAJOR_NR));
@@ -537,4 +533,9 @@
 	else
 		printk("nbd: module cleaned up.\n");
 }
-#endif
+
+module_init(nbd_init);
+module_exit(nbd_cleanup);
+
+MODULE_DESCRIPTION("Network Block Device");
+
#1 lore raw
Quoted 2 lines · Steve Whitehouse

The bug fix in ll_rw_blk.c prevents hangs when using block devices which don't have plugging functions,

It looks the right fix (better than 2.4.0 that didn't had such bug but that was recalling the request_fn at every inserction in the queue).

Thanks,
Andrea

#2 lore raw
In reply to Steve Whitehouse
Quoted 6 lines · Andrea Arcangeli
Quoted 2 lines · Steve Whitehouse

The bug fix in ll_rw_blk.c prevents hangs when using block devices which don't have plugging functions,

It looks the right fix (better than 2.4.0 that didn't had such bug but that was recalling the request_fn at every inserction in the queue).

Indeed, the removal was too quick and forgot private plugs.

Signature · 1 line
Jens Axboe
#3 lore raw
In reply to Steve Whitehouse

Review

Quoted 9 lines · Steve Whitehouse
(quoted diff) +5 -2
-int nbd_init(void)
+int __init nbd_init(void)
 
-void cleanup_module(void)
+void __exit nbd_cleanup(void)
 
+
+module_init(nbd_init);
+module_exit(nbd_cleanup);

If you're using module_init/module_exit, shouldn't nbd_init/nbd_cleanup be declared statically?

--
Russell King ([email protected]) The developer of ARM Linux
http://www.arm.linux.org.uk/personal/aboutme.html

#4 lore raw
In reply to Steve Whitehouse
Quoted 7 lines · Russell King
Quoted 5 lines · Steve Whitehouse
drivers/block/nbd.c +2 -2
-int nbd_init(void)
+int __init nbd_init(void)
 
-void cleanup_module(void)
+void __exit nbd_cleanup(void)
Quoted 6 lines

drivers/block/nbd.c line 538

Quoted 3 lines
drivers/block/nbd.c +3 -0
+
+module_init(nbd_init);
+module_exit(nbd_cleanup);

If you're using module_init/module_exit, shouldn't nbd_init/nbd_cleanup be declared statically?

And more importantly, the init calls from ll_rw_blk.c:blk_dev_init() should be removed too.

Signature · 1 line
Jens Axboe
#5 lore raw

Hi,

Oops, sorry. Updated patch below. Jens & Russell: Thanks for pointing this out,

Steve.

Quoted 16 lines · Russell King
Quoted 7 lines · Steve Whitehouse
Quoted 5 lines
(quoted diff) +2 -2
-int nbd_init(void)
+int __init nbd_init(void)
 
-void cleanup_module(void)
+void __exit nbd_cleanup(void)
Quoted 6 lines

Review

Quoted 3 lines
(quoted diff) +3 -0
+
+module_init(nbd_init);
+module_exit(nbd_cleanup);

If you're using module_init/module_exit, shouldn't nbd_init/nbd_cleanup be declared statically?

Quoted 6 lines

And more importantly, the init calls from ll_rw_blk.c:blk_dev_init() should be removed too.

Signature · 1 line
Jens Axboe

-----------------------------------------------------------------------------

drivers/block/ll_rw_blk.c +3 -3
diff -u -r linux-2.4.2/drivers/block/ll_rw_blk.c linux/drivers/block/ll_rw_blk.c
@@ -588,6 +588,9 @@
 	 * inserted at elevator_merge time
 	 */
 	list_add(&req->queue, insert_here);
+
+	if (!q->plugged && insert_here == &q->queue_head)
+		q->request_fn(q);
 }
 
 void inline blk_refill_freelist(request_queue_t *q, int rw)
@@ -1320,9 +1323,6 @@
 #endif
 #ifdef CONFIG_DDV
 	ddv_init();
-#endif
-#ifdef CONFIG_BLK_DEV_NBD
-	nbd_init();
 #endif
 #ifdef CONFIG_MDISK
 	mdisk_init();
drivers/block/nbd.c +12 -11
diff -u -r linux-2.4.2/drivers/block/nbd.c linux/drivers/block/nbd.c
@@ -29,7 +29,7 @@
 #include <linux/major.h>
 
 #include <linux/module.h>
-
+#include <linux/init.h>
 #include <linux/sched.h>
 #include <linux/fs.h>
 #include <linux/stat.h>
@@ -149,12 +149,13 @@
 {
 	int result;
 	struct nbd_request request;
+	unsigned long size = req->current_nr_sectors << 9;
 
 	DEBUG("NBD: sending control, ");
 	request.magic = htonl(NBD_REQUEST_MAGIC);
 	request.type = htonl(req->cmd);
 	request.from = cpu_to_be64( (u64) req->sector << 9);
-	request.len = htonl(req->current_nr_sectors << 9);
+	request.len = htonl(size);
 	memcpy(request.handle, &req, sizeof(req));
 
 	result = nbd_xmit(1, sock, (char *) &request, sizeof(request));
@@ -163,7 +164,7 @@
 
 	if (req->cmd == WRITE) {
 		DEBUG("data, ");
-		result = nbd_xmit(1, sock, req->buffer, req->current_nr_sectors << 9);
+		result = nbd_xmit(1, sock, req->buffer, size);
 		if (result <= 0)
 			FAIL("Send data failed.");
 	}
@@ -475,11 +476,7 @@
  *  (Just smiley confuses emacs :-)
  */
 
-#ifdef MODULE
-#define nbd_init init_module
-#endif
-
-int nbd_init(void)
+static int __init nbd_init(void)
 {
 	int i;
 
@@ -526,8 +523,7 @@
 	return 0;
 }
 
-#ifdef MODULE
-void cleanup_module(void)
+static void __exit nbd_cleanup(void)
 {
 	devfs_unregister (devfs_handle);
 	blk_cleanup_queue(BLK_DEFAULT_QUEUE(MAJOR_NR));
@@ -537,4 +533,9 @@
 	else
 		printk("nbd: module cleaned up.\n");
 }
-#endif
+
+module_init(nbd_init);
+module_exit(nbd_cleanup);
+
+MODULE_DESCRIPTION("Network Block Device");
+
#6 lore raw
In reply to Steve Whitehouse
Quoted 7 lines · Steve Whitehouse

Here is a new version of the patch I recently sent to the list with some NBD cleanups and a bug fix in ll_rw_blk.c. The changes to NBD have Pavel Machek's approval as I've left out the two changes as he suggested.

The bug fix in ll_rw_blk.c prevents hangs when using block devices which don't have plugging functions,

I'm convinced that the right fix is to just make everybody have plugging functions.

Right now, who doesn't? The fix is unbelievably ugly, AND can break real drivers that _do_ have plugging functions (where they get surprised by having their request function called several times per plug just because somebody unplugged them and new requests came in).

Just fix ndb instead.

Linus

#7 lore raw

Hi,

Quoted 9 lines · Steve Whitehouse
Quoted 5 lines

Here is a new version of the patch I recently sent to the list with some NBD cleanups and a bug fix in ll_rw_blk.c. The changes to NBD have Pavel Machek's approval as I've left out the two changes as he suggested.

Quoted 6 lines
Quoted 2 lines

The bug fix in ll_rw_blk.c prevents hangs when using block devices which don't have plugging functions,

I'm convinced that the right fix is to just make everybody have plugging functions.

I'm working on that. Once I've discovered why enabling plugging causes nbd to hang, then I'll send a patch assuming nobody beats me to it :-)

Quoted 9 lines

Right now, who doesn't? The fix is unbelievably ugly, AND can break real drivers that _do_ have plugging functions (where they get surprised by having their request function called several times per plug just because somebody unplugged them and new requests came in).

Just fix ndb instead.

Linus

I tested the patch with a printk() which printed whenever the new call to the request function was triggered. It didn't happen once in normal fs use with ext2 on a scsi disk. From the code I think its not even possible for this to be called at all for a device which has plugging. For a plugged device when I/O comes in, there appear to be only two cases:

- Device queue empty. Device gets plugged. New request_fn call not called
- Device queue not empty. New I/O added to back of queue. New request_fn not called (it only gets called when the I/O is added to the front of the queue).

I think nbd is the only device which doesn't use plugging at the moment (from a quick grep of the kernel source),

Steve.

#8 lore raw
Quoted 4 lines · Steve Whitehouse

I tested the patch with a printk() which printed whenever the new call to the request function was triggered. It didn't happen once in normal fs use with ext2 on a scsi disk.

As far as I can tell, the patch will trigger only for a not-empty request list, where the elevator decides to put the new request at the head of the queue.

Which is probably unlikely (and with the current elevator it might even be impossible). But it does not look impossible in theory. And I'd really prefer _not_ to have something that
- looks completely bogus from a design standpoint
- might be buggy under some rather unlikely circumstances

Reading them together they become "might cause disk corruption in some really hard-to-trigger circumstances". No, thank you.

Note that I suspect that all current drivers (or at least the common ones) have protection against being called multiple times, simply because 2.2.x used to do it. Which again means that you'd probably never see problems in practice. But that doesn't make it _right_.

Linus

#9 lore raw
In reply to Steve Whitehouse
Quoted 10 lines · Linus Torvalds

As far as I can tell, the patch will trigger only for a not-empty request list, where the elevator decides to put the new request at the head of the queue.

Which is probably unlikely (and with the current elevator it might even be impossible). But it does not look impossible in theory. And I'd really prefer _not_ to have something that
- looks completely bogus from a design standpoint
- might be buggy under some rather unlikely circumstances

Quoted 7 lines

Reading them together they become "might cause disk corruption in some really hard-to-trigger circumstances". No, thank you.

Note that I suspect that all current drivers (or at least the common ones) have protection against being called multiple times, simply because 2.2.x used to do it. Which again means that you'd probably never see problems in practice. But that doesn't make it _right_.

I think most of the "we want to disable plugging" behaviour stems from the way task queues behave. Once somebody starts a tq_disk run, the list is fried and walked one by one. Both old loop and nbd drop the io_request_lock and block, possibly waiting for I/O to be done (at least the loop case, don't know about ndb). But this I/O won't be done just because the target plug every now and then just happens to be queued behind the nbd/loop one and a new tq_disk run won't start it.

Signature · 1 line
Jens Axboe
#10 lore raw
In reply to Steve Whitehouse
Quoted 9 lines · Jens Axboe

I think most of the "we want to disable plugging" behaviour stems from the way task queues behave. Once somebody starts a tq_disk run, the list is fried and walked one by one. Both old loop and nbd drop the io_request_lock and block, possibly waiting for I/O to be done (at least the loop case, don't know about ndb). But this I/O won't be done just because the target plug every now and then just happens to be queued behind the nbd/loop one and a new tq_disk run won't start it.

Ugh.

How about loop/ndb intercepting the damn requests at the "elevator" layer - that way you see every one of them, and the actual request function might as well just be a no-op?

Linus

#11 lore raw
In reply to Steve Whitehouse
Quoted 9 lines · Linus Torvalds
Quoted 8 lines

I think most of the "we want to disable plugging" behaviour stems from the way task queues behave. Once somebody starts a tq_disk run, the list is fried and walked one by one. Both old loop and nbd drop the io_request_lock and block, possibly waiting for I/O to be done (at least the loop case, don't know about ndb). But this I/O won't be done just because the target plug every now and then just happens to be queued behind the nbd/loop one and a new tq_disk run won't start it.

Quoted 5 lines

Ugh.

How about loop/ndb intercepting the damn requests at the "elevator" layer - that way you see every one of them, and the actual request function might as well just be a no-op?

I've suggested that before, turn ndb into ndb_make_request style driver and all of this will disappear. I'll give it a shot.

Signature · 1 line
Jens Axboe
#12 lore raw
In reply to Steve Whitehouse
Quoted 2 lines · Jens Axboe

I've suggested that before, turn ndb into ndb_make_request style driver and all of this will disappear. I'll give it a shot.

Here's a first shot, compile tested but that's it. Steve, does this work for you? Against 2.4.2-ac7

Signature · 1 line
Jens Axboe
drivers/block/nbd.c renamed from opt/kernel/linux-2.4.2-ac7/drivers/block/nbd.c +96 -93
diff -ur --exclude-from /home/axboe/cdrom/exclude /opt/kernel/linux-2.4.2-ac7/drivers/block/nbd.c linux/drivers/block/nbd.c
@@ -24,7 +24,6 @@
  *          structure with userland
  */
 
-#undef	NBD_PLUGGABLE
 #define PARANOIA
 #include <linux/major.h>
 
@@ -44,6 +43,8 @@
 #include <asm/uaccess.h>
 #include <asm/types.h>
 
+static kmem_cache_t *nbd_cachep;
+
 #define MAJOR_NR NBD_MAJOR
 #include <linux/nbd.h>
 
@@ -66,8 +67,6 @@
 static int requests_out;
 #endif
 
-static void nbd_plug_device(request_queue_t *q, kdev_t dev) { }
-
 static int nbd_open(struct inode *inode, struct file *file)
 {
 	int dev;
@@ -145,52 +144,52 @@
 
 #define FAIL( s ) { printk( KERN_ERR "NBD: " s "(result %d)\n", result ); goto error_out; }
 
-void nbd_send_req(struct socket *sock, struct request *req)
+void nbd_send_req(struct socket *sock, struct nbd_bh *nbh)
 {
 	int result;
 	struct nbd_request request;
-	unsigned long size = req->current_nr_sectors << 9;
+	struct buffer_head *bh = nbh->bh;
 
 	DEBUG("NBD: sending control, ");
 	request.magic = htonl(NBD_REQUEST_MAGIC);
-	request.type = htonl(req->cmd);
-	request.from = cpu_to_be64( (u64) req->sector << 9);
-	request.len = htonl(size);
-	memcpy(request.handle, &req, sizeof(req));
+	request.type = htonl(nbh->cmd);
+	request.from = cpu_to_be64( (u64) bh->b_rsector << 9);
+	request.len = htonl(bh->b_size);
+	memcpy(request.handle, &nbh, sizeof(nbh));
 
 	result = nbd_xmit(1, sock, (char *) &request, sizeof(request));
 	if (result <= 0)
 		FAIL("Sendmsg failed for control.");
 
-	if (req->cmd == WRITE) {
+	if (nbh->cmd == WRITE) {
 		DEBUG("data, ");
-		result = nbd_xmit(1, sock, req->buffer, size);
+		result = nbd_xmit(1, sock, bh->b_data, bh->b_size);
 		if (result <= 0)
 			FAIL("Send data failed.");
 	}
 	return;
 
       error_out:
-	req->errors++;
+	nbh->errors++;
 }
 
 #define HARDFAIL( s ) { printk( KERN_ERR "NBD: " s "(result %d)\n", result ); lo->harderror = result; return NULL; }
-struct request *nbd_read_stat(struct nbd_device *lo)
+struct nbd_bh *nbd_read_stat(struct nbd_device *lo)
 		/* NULL returned = something went wrong, inform userspace       */ 
 {
 	int result;
 	struct nbd_reply reply;
-	struct request *xreq, *req;
+	struct nbd_bh *xnbh, *nbh;
 
 	DEBUG("reading control, ");
 	reply.magic = 0;
 	result = nbd_xmit(0, lo->sock, (char *) &reply, sizeof(reply));
 	if (result <= 0)
 		HARDFAIL("Recv control failed.");
-	memcpy(&xreq, reply.handle, sizeof(xreq));
-	req = blkdev_entry_prev_request(&lo->queue_head);
+	memcpy(&xnbh, reply.handle, sizeof(xnbh));
+	nbh = list_entry(lo->queue_head.prev, struct nbd_bh, queue);
 
-	if (xreq != req)
+	if (xnbh != nbh)
 		FAIL("Unexpected handle received.\n");
 
 	DEBUG("ok, ");
@@ -198,41 +197,37 @@
 		HARDFAIL("Not enough magic.");
 	if (ntohl(reply.error))
 		FAIL("Other side returned error.");
-	if (req->cmd == READ) {
+	if (nbh->cmd == READ) {
 		DEBUG("data, ");
-		result = nbd_xmit(0, lo->sock, req->buffer, req->current_nr_sectors << 9);
+		result = nbd_xmit(0, lo->sock, nbh->bh->b_data,nbh->bh->b_size);
 		if (result <= 0)
 			HARDFAIL("Recv data failed.");
 	}
 	DEBUG("done.\n");
-	return req;
+	return nbh;
 
 /* Can we get here? Yes, if other side returns error */
       error_out:
-	req->errors++;
-	return req;
+	nbh->errors++;
+	return nbh;
 }
 
 void nbd_do_it(struct nbd_device *lo)
 {
-	struct request *req;
-	int dequeued;
+	struct nbd_bh *nbh;
 
 	down (&lo->queue_lock);
 	while (1) {
 		up (&lo->queue_lock);
-		req = nbd_read_stat(lo);
+		nbh = nbd_read_stat(lo);
 		down (&lo->queue_lock);
 
-		if (!req) {
+		if (!nbh) {
 			printk(KERN_ALERT "req should never be null\n" );
 			goto out;
 		}
 #ifdef PARANOIA
-		if (req != blkdev_entry_prev_request(&lo->queue_head)) {
-			printk(KERN_ALERT "NBD: I have problem...\n");
-		}
-		if (lo != &nbd_dev[MINOR(req->rq_dev)]) {
+		if (lo != &nbd_dev[MINOR(nbh->bh->b_rdev)]) {
 			printk(KERN_ALERT "NBD: request corrupted!\n");
 			continue;
 		}
@@ -241,14 +236,12 @@
 			goto out;
 		}
 #endif
-		list_del(&req->queue);
+		list_del(&nbh->queue);
 		up (&lo->queue_lock);
 		
-		dequeued = nbd_end_request(req);
+		nbd_end_request(nbh);
 
 		down (&lo->queue_lock);
-		if (!dequeued)
-			list_add(&req->queue, &lo->queue_head);
 	}
  out:
 	up (&lo->queue_lock);
@@ -256,8 +249,7 @@
 
 void nbd_clear_que(struct nbd_device *lo)
 {
-	struct request *req;
-	int dequeued;
+	struct nbd_bh *nbh;
 
 #ifdef PARANOIA
 	if (lo->magic != LO_MAGIC) {
@@ -267,29 +259,43 @@
 #endif
 
 	while (!list_empty(&lo->queue_head)) {
-		req = blkdev_entry_prev_request(&lo->queue_head);
+		nbh = list_entry(lo->queue_head.prev, struct nbd_bh, queue);
 #ifdef PARANOIA
-		if (!req) {
-			printk( KERN_ALERT "NBD: panic, panic, panic\n" );
-			break;
-		}
-		if (lo != &nbd_dev[MINOR(req->rq_dev)]) {
+		if (lo != &nbd_dev[MINOR(nbh->bh->b_rdev)]) {
 			printk(KERN_ALERT "NBD: request corrupted when clearing!\n");
 			continue;
 		}
 #endif
-		req->errors++;
-		list_del(&req->queue);
+		nbh->errors++;
+		list_del(&nbh->queue);
 		up(&lo->queue_lock);
 
-		dequeued = nbd_end_request(req);
+		nbd_end_request(nbh);
 
 		down(&lo->queue_lock);
-		if (!dequeued)
-			list_add(&req->queue, &lo->queue_head);
 	}
 }
 
+static struct nbd_bh *nbd_alloc_bh(struct buffer_head *bh, int rw)
+{
+	struct nbd_bh *nbh;
+
+	do {
+		nbh = kmem_cache_alloc(nbd_cachep, SLAB_BUFFER);
+		if (nbh)
+			break;
+
+		run_task_queue(&tq_disk);
+		schedule_timeout(HZ);
+	} while (1);
+
+	nbh->bh = bh;
+	nbh->cmd = rw;
+	nbh->errors = 0;
+	INIT_LIST_HEAD(&nbh->queue);
+	return nbh;
+}
+
 /*
  * We always wait for result of write, for now. It would be nice to make it optional
  * in future
@@ -300,53 +306,45 @@
 #undef FAIL
 #define FAIL( s ) { printk( KERN_ERR "NBD, minor %d: " s "\n", dev ); goto error_out; }
 
-static void do_nbd_request(request_queue_t * q)
+static int nbd_make_request(request_queue_t *q, int rw, struct buffer_head *bh)
 {
-	struct request *req;
 	int dev = 0;
 	struct nbd_device *lo;
+	struct nbd_bh *nbh;
 
-	while (!QUEUE_EMPTY) {
-		req = CURRENT;
+	dev = MINOR(bh->b_rdev);
 #ifdef PARANOIA
-		if (!req)
-			FAIL("que not empty but no request?");
-#endif
-		dev = MINOR(req->rq_dev);
-#ifdef PARANOIA
-		if (dev >= MAX_NBD)
-			FAIL("Minor too big.");		/* Probably can not happen */
+	if (dev >= MAX_NBD)
+		FAIL("Minor too big.");		/* Probably can not happen */
 #endif
-		lo = &nbd_dev[dev];
-		if (!lo->file)
-			FAIL("Request when not-ready.");
-		if ((req->cmd == WRITE) && (lo->flags & NBD_READ_ONLY))
-			FAIL("Write on read-only");
+	lo = &nbd_dev[dev];
+	if (!lo->file)
+		FAIL("Request when not-ready.");
+	if ((rw == WRITE) && (lo->flags & NBD_READ_ONLY))
+		FAIL("Write on read-only");
 #ifdef PARANOIA
-		if (lo->magic != LO_MAGIC)
-			FAIL("nbd[] is not magical!");
-		requests_in++;
+	if (lo->magic != LO_MAGIC)
+		FAIL("nbd[] is not magical!");
+	requests_in++;
 #endif
-		req->errors = 0;
-		blkdev_dequeue_request(req);
-		spin_unlock_irq(&io_request_lock);
 
-		down (&lo->queue_lock);
-		list_add(&req->queue, &lo->queue_head);
-		nbd_send_req(lo->sock, req);	/* Why does this block?         */
-		up (&lo->queue_lock);
+	/*
+	 * not an error, but we don't want to do read-aheads
+	 */
+	if (rw == READA)
+		goto error_out;
 
-		spin_lock_irq(&io_request_lock);
-		continue;
+	nbh = nbd_alloc_bh(bh, rw);
 
-	      error_out:
-		req->errors++;
-		blkdev_dequeue_request(req);
-		spin_unlock(&io_request_lock);
-		nbd_end_request(req);
-		spin_lock(&io_request_lock);
-	}
-	return;
+	down (&lo->queue_lock);
+	list_add(&nbh->queue, &lo->queue_head);
+	nbd_send_req(lo->sock, nbh);	/* Why does this block?         */
+	up (&lo->queue_lock);
+	return 0;
+
+error_out:
+	bh->b_end_io(bh, test_bit(BH_Uptodate, &bh->b_state));
+	return 0;
 }
 
 static int nbd_ioctl(struct inode *inode, struct file *file,
@@ -354,7 +352,7 @@
 {
 	struct nbd_device *lo;
 	int dev, error, temp;
-	struct request sreq ;
+	struct nbd_bh snbh;
 
 	/* Anyone capable of this syscall can do *real bad* things */
 
@@ -370,9 +368,9 @@
 	switch (cmd) {
 	case NBD_DISCONNECT:
 	        printk("NBD_DISCONNECT\n") ;
-                sreq.cmd=2 ; /* shutdown command */
+                snbh.cmd=2 ; /* shutdown command */
                 if (!lo->sock) return -EINVAL ;
-                nbd_send_req(lo->sock,&sreq) ;
+                nbd_send_req(lo->sock, &snbh);
                 return 0 ;
  
 	case NBD_CLEAR_SOCK:
@@ -476,16 +474,25 @@
  *  (Just smiley confuses emacs :-)
  */
 
-static int __init nbd_init(void)
+int __init nbd_init(void)
 {
 	int i;
 
+	nbd_cachep = kmem_cache_create("nbd_requests", sizeof(struct nbd_bh),
+					0, SLAB_HWCACHE_ALIGN, NULL, NULL);
+	if (nbd_cachep == NULL) {
+		printk("nbd: unable to setup slab cache\n");
+		return -ENOMEM;
+	}
+
 	if (sizeof(struct nbd_request) != 28) {
+		kmem_cache_destroy(nbd_cachep);
 		printk(KERN_CRIT "Sizeof nbd_request needs to be 28 in order to work!\n" );
 		return -EIO;
 	}
 
 	if (register_blkdev(MAJOR_NR, "nbd", &nbd_fops)) {
+		kmem_cache_destroy(nbd_cachep);
 		printk("Unable to get major number %d for NBD\n",
 		       MAJOR_NR);
 		return -EIO;
@@ -495,11 +502,7 @@
 #endif
 	blksize_size[MAJOR_NR] = nbd_blksizes;
 	blk_size[MAJOR_NR] = nbd_sizes;
-	blk_init_queue(BLK_DEFAULT_QUEUE(MAJOR_NR), do_nbd_request);
-#ifndef NBD_PLUGGABLE
-	blk_queue_pluggable(BLK_DEFAULT_QUEUE(MAJOR_NR), nbd_plug_device);
-#endif
-	blk_queue_headactive(BLK_DEFAULT_QUEUE(MAJOR_NR), 0);
+	blk_queue_make_request(BLK_DEFAULT_QUEUE(MAJOR_NR), nbd_make_request);
 	for (i = 0; i < MAX_NBD; i++) {
 		nbd_dev[i].refcnt = 0;
 		nbd_dev[i].file = NULL;
@@ -526,7 +529,7 @@
 static void __exit nbd_cleanup(void)
 {
 	devfs_unregister (devfs_handle);
-	blk_cleanup_queue(BLK_DEFAULT_QUEUE(MAJOR_NR));
+	kmem_cache_destroy(nbd_cachep);
 
 	if (unregister_blkdev(MAJOR_NR, "nbd") != 0)
 		printk("nbd: cleanup_module failed\n");
include/linux/blk.h renamed from opt/kernel/linux-2.4.2-ac7/include/linux/blk.h +0 -6
diff -ur --exclude-from /home/axboe/cdrom/exclude /opt/kernel/linux-2.4.2-ac7/include/linux/blk.h linux/include/linux/blk.h
@@ -291,12 +291,6 @@
 #define DEVICE_REQUEST do_mfm_request
 #define DEVICE_NR(device) (MINOR(device) >> 6)
 
-#elif (MAJOR_NR == NBD_MAJOR)
-
-#define DEVICE_NAME "nbd"
-#define DEVICE_REQUEST do_nbd_request
-#define DEVICE_NR(device) (MINOR(device))
-
 #elif (MAJOR_NR == MDISK_MAJOR)
 
 #define DEVICE_NAME "mdisk"
include/linux/nbd.h renamed from opt/kernel/linux-2.4.2-ac7/include/linux/nbd.h +19 -25

62 lines not on this page · raw message

#13 lore raw
In reply to Linus Torvalds
Quoted 8 lines · Steve Whitehouse
Quoted 6 lines
Quoted 2 lines

The bug fix in ll_rw_blk.c prevents hangs when using block devices which don't have plugging functions,

I'm convinced that the right fix is to just make everybody have plugging functions.

I'm working on that. Once I've discovered why enabling plugging causes nbd to hang, then I'll send a patch assuming nobody beats me to it :-)

Does anyone actually know why? Pavel didn't seem to, so maybe it was just disabled on a hunch?

Quoted 10 lines

I tested the patch with a printk() which printed whenever the new call to the request function was triggered. It didn't happen once in normal fs use with ext2 on a scsi disk. From the code I think its not even possible for this to be called at all for a device which has plugging. For a plugged device when I/O comes in, there appear to be only two cases:

- Device queue empty. Device gets plugged. New request_fn call not called
- Device queue not empty. New I/O added to back of queue. New request_fn not called (it only gets called when the I/O is added to the front of the queue).

Exactly right.

Quoted 2 lines

I think nbd is the only device which doesn't use plugging at the moment (from a quick grep of the kernel source),

Probably right -- loop used to disable plugging to disallow merging and tq_disk deadlocks, but now (-ac) it's a make_request style driver instead.

Signature · 1 line
Jens Axboe