Skip to contentSkip to conversation

block: use a separate plug list for blk-mq requests

RFC PATCH In review Christoph Hellwig 2 participants 6 messages Oct 6, 2013 – Oct 22, 2013 Block layer

lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 block: use a separate plug list for blk-mq requests

blk_flush_plug_list became a bit of a mess with the introduction of blk-mq, so I started looking into separating the blk-mq handling from it. Turns out that by doing this we can streamline the blk-mq submission path a lot.

If we branch out to a blk-mq specific code path early we can do the list sort based on the hw ctx instead of the queue and thus avoid the later improvised loop to sort again. In addition we can also remove the hw irq disabling in the submission path entirely and collapse a couple of functions in blk-mq.c, all at the cost of an additional list_head in struct blk_plug which can go away again as soon as we remove old-school request_fn based drivers.

---
 block/blk-core.c       |   29 ++++-------------
 block/blk-mq.c         |   81 ++++++++++++++++++++++++++++---------------------
 include/linux/blk-mq.h |    4 +-
 include/linux/blkdev.h |    6 +++
 4 files changed, 62 insertions(+), 58 deletions(-)
block/blk-core.c +8 -21
index 6d7fd79..7bedff6 100644
@@ -2887,6 +2887,7 @@ void blk_start_plug(struct blk_plug *plug)
 
 	plug->magic = PLUG_MAGIC;
 	INIT_LIST_HEAD(&plug->list);
+	INIT_LIST_HEAD(&plug->mq_list);
 	INIT_LIST_HEAD(&plug->cb_list);
 
 	/*
@@ -2973,28 +2974,21 @@ struct blk_plug_cb *blk_check_plugged(blk_plug_cb_fn unplug, void *data,
 }
 EXPORT_SYMBOL(blk_check_plugged);
 
-static void do_queue_unplug(struct request_queue *q, bool from_schedule,
-			    unsigned int depth, struct list_head *list)
-{
-	if (q->mq_ops) {
-		trace_block_unplug(q, depth, !from_schedule);
-		blk_mq_insert_requests(q, list, 1, from_schedule);
-	} else
-		queue_unplugged(q, depth, from_schedule);
-}
-
 void blk_flush_plug_list(struct blk_plug *plug, bool from_schedule)
 {
 	struct request_queue *q;
 	unsigned long flags;
 	struct request *rq;
 	LIST_HEAD(list);
-	LIST_HEAD(q_list);
 	unsigned int depth;
 
 	BUG_ON(plug->magic != PLUG_MAGIC);
 
 	flush_plug_callbacks(plug, from_schedule);
+
+	if (!list_empty(&plug->mq_list))
+		blk_mq_flush_plug_list(plug, from_schedule);
+
 	if (list_empty(&plug->list))
 		return;
 
@@ -3019,17 +3013,10 @@ void blk_flush_plug_list(struct blk_plug *plug, bool from_schedule)
 			 * This drops the queue lock
 			 */
 			if (q)
-				do_queue_unplug(q, from_schedule, depth, &q_list);
+				queue_unplugged(q, depth, from_schedule);
 			q = rq->q;
 			depth = 0;
-			if (!q->mq_ops)
-				spin_lock(q->queue_lock);
-		}
-
-		if (q->mq_ops) {
-			depth++;
-			list_add_tail(&rq->queuelist, &q_list);
-			continue;
+			spin_lock(q->queue_lock);
 		}
 
 		/*
@@ -3055,7 +3042,7 @@ void blk_flush_plug_list(struct blk_plug *plug, bool from_schedule)
 	 * This drops the queue lock
 	 */
 	if (q)
-		do_queue_unplug(q, from_schedule, depth, &q_list);
+		queue_unplugged(q, depth, from_schedule);
 
 	local_irq_restore(flags);
 }
block/blk-mq.c +47 -34

130 lines not on this page · raw message

include/linux/blk-mq.h +2 -2
index 4fddab2..746042ff 100644
@@ -118,9 +118,9 @@ int blk_mq_register_disk(struct gendisk *);
 void blk_mq_unregister_disk(struct gendisk *);
 void blk_mq_init_commands(struct request_queue *, void (*init)(void *data, struct blk_mq_hw_ctx *, struct request *, unsigned int), void *data);
 
-void blk_mq_flush_plug(struct request_queue *, bool);
+void blk_mq_flush_plug_list(struct blk_plug *plug, bool from_schedule);
+
 void blk_mq_insert_request(struct request_queue *, struct request *, bool);
-void blk_mq_insert_requests(struct request_queue *, struct list_head *, bool, bool);
 void blk_mq_run_queues(struct request_queue *q, bool async);
 void blk_mq_free_request(struct request *rq);
 bool blk_mq_can_queue(struct blk_mq_hw_ctx *);
include/linux/blkdev.h +5 -1
index 8771c0b..1b79592 100644
@@ -1035,6 +1035,7 @@ static inline void blk_post_runtime_resume(struct request_queue *q, int err) {}
 struct blk_plug {
 	unsigned long magic; /* detect uninitialized use-cases */
 	struct list_head list; /* requests */
+	struct list_head mq_list; /* blk-mq requests */
 	struct list_head cb_list; /* md requires an unplug callback */
 };
 #define BLK_MAX_REQUEST_COUNT 16
@@ -1072,7 +1073,10 @@ static inline bool blk_needs_flush_plug(struct task_struct *tsk)
 {
 	struct blk_plug *plug = tsk->plug;
 
-	return plug && (!list_empty(&plug->list) || !list_empty(&plug->cb_list));
+	return plug &&
+		(!list_empty(&plug->list) ||
+		 !list_empty(&plug->mq_list) ||
+		 !list_empty(&plug->cb_list));
 }
 
 /*
maintainer
#1 lore raw
Quoted 10 lines · Christoph Hellwig

blk_flush_plug_list became a bit of a mess with the introduction of blk-mq, so I started looking into separating the blk-mq handling from it. Turns out that by doing this we can streamline the blk-mq submission path a lot.

If we branch out to a blk-mq specific code path early we can do the list sort based on the hw ctx instead of the queue and thus avoid the later improvised loop to sort again. In addition we can also remove the hw irq disabling in the submission path entirely and collapse a couple of functions in blk-mq.c, all at the cost of an additional list_head in struct blk_plug which can go away again as soon as we remove old-school request_fn based drivers.

Thanks, I'll take a look at this. The plugging was done mostly hacky when implementing it, it was meant to be revisited. So definitely room for improvement there.

Signature · 1 line
Jens Axboe
#2 lore raw
Quoted 3 lines · Jens Axboe

Thanks, I'll take a look at this. The plugging was done mostly hacky when implementing it, it was meant to be revisited. So definitely room for improvement there.

Did you manage to take a look?

maintainer
#3 lore raw
Quoted 6 lines · Christoph Hellwig
Quoted 3 lines · Jens Axboe

Thanks, I'll take a look at this. The plugging was done mostly hacky when implementing it, it was meant to be revisited. So definitely room for improvement there.

Did you manage to take a look?

Yep, just did on the flight. I think it looks pretty good and I want to queue it up. Did you test it?

Signature · 1 line
Jens Axboe
#4 lore raw
Quoted 10 lines · Jens Axboe
Quoted 6 lines · Christoph Hellwig
Quoted 3 lines · Jens Axboe

Thanks, I'll take a look at this. The plugging was done mostly hacky when implementing it, it was meant to be revisited. So definitely room for improvement there.

Did you manage to take a look?

Yep, just did on the flight. I think it looks pretty good and I want to queue it up. Did you test it?

Yes, with my usual virtio-blk setup.

maintainer
#5 lore raw
Quoted 7 lines · Christoph Hellwig
Quoted 6 lines · Jens Axboe
Quoted 5 lines · Christoph Hellwig
Quoted 3 lines · Jens Axboe

Thanks, I'll take a look at this. The plugging was done mostly hacky when implementing it, it was meant to be revisited. So definitely room for improvement there.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

Did you manage to take a look?

Yep, just did on the flight. I think it looks pretty good and I want to queue it up. Did you test it?

Yes, with my usual virtio-blk setup.

Excellent, just wanted to be sure before I pushed anything out, since I can't test personally on the road...

Signature · 1 line
Jens Axboe