Skip to contentSkip to conversation

dm: add full blk-mq support to request-based DM

PATCH Applied Mike Snitzer 4 participants 19 messages Mar 12, 2015 – Mar 13, 2015

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

cover dm: add full blk-mq support to request-based DM

This patchset updates request-based DM (used exclussively by DM multipath) to support blk-mq I/O path or the old ->request_fn I/O path -- the default is still the old ->request_fn I/O path.

Like was done for SCSI, I've exposed a dm-mod.ko 'use_blk_mq' module parameter and a CONFIG_DM_MQ_DEFAULT. I've queued these patches in linux-next via linux-dm.git's 'for-next' branch -- my goal is for these changes to land upstream in 4.1.

I've tested all 4 supported permutations of rq-based queue stacking:
1) blk-mq mpath stacked on blk-mq device(s)
2) blk-mq mpath stacked on ->request_fn device(s)
3) old ->request_fn mpath stacked on blk-mq device(s)
4) old ->request_fn mpath stacked on old ->request_fn device(s)

There is definitely much more room for optimizing the blk-mq mpath queue (via dm_mq_ops setup) to have more awareness about the characteristics of the underlying blk-mq device(s) -- Jens offered some suggestions that are worth pursuing. Goal being to leverage the ability to have multiple software submission queues that map to the underlying paths' HW queue. I expect Keith to be the most likely person to pursue these follow-on optimizations given his immediate access to NVMe devices, etc.

blk-mq allows for the removal of 2 of the 3 mempools and associated allocations that were traditionally performed using old ->request_fn I/O path. The md->bs bioset still remains for cloning a request's bios.

I'll be pursuing the possibility of removing the bio cloning that happens when cloning a request (it was put in place for partial completion of a request's bios but the prospect of avoiding those bio clones warrants further review of how useful those partial completions are, see: http://www.redhat.com/archives/dm-devel/2014-June/msg00030.html)

Jens and Keith, special thanks for your help answering more blk-mq questions than I'd have hoped would be needed while at LSF/MM.

Keith Busch (1):
blk-mq: don't wait in blk_mq_queue_enter() if __GFP_WAIT isn't set

Mike Snitzer (6):
blk-mq: fix use of incorrect goto label in blk_mq_init_queue error path
blk-mq: add blk_mq_init_allocated_queue and export blk_mq_register_disk
blk-mq: export blk_mq_run_hw_queues
dm: add full blk-mq support to request-based DM
dm: optimize dm_mq_queue_rq to _not_ use kthread if using pure blk-mq
dm: add 'use_blk_mq' module param and expose in per-device ro sysfs attr

 block/blk-mq-sysfs.c   |   1 +
 block/blk-mq.c         |  47 ++++--
 drivers/md/Kconfig     |  11 ++
 drivers/md/dm-mpath.c  |   4 +-
 drivers/md/dm-sysfs.c  |   9 ++
 drivers/md/dm-table.c  |  17 +-
 drivers/md/dm.c        | 414 +++++++++++++++++++++++++++++++++++++++----------
 drivers/md/dm.h        |   5 +-
 include/linux/blk-mq.h |   3 +
 9 files changed, 401 insertions(+), 110 deletions(-)
Signature · 1 line
1.9.5 (Apple Git-50.3)
#1 lore raw

PATCH 1/7 blk-mq: fix use of incorrect goto label in blk_mq_init_queue error path

If percpu_ref_init() fails the 'err_hctxs' label should be used instead of 'err_map'.

---
 block/blk-mq.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
block/blk-mq.c +1 -1
index 4f4bea2..459840c 100644
@@ -1938,7 +1938,7 @@ struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *set)
 	 */
 	if (percpu_ref_init(&q->mq_usage_counter, blk_mq_usage_counter_release,
 			    PERCPU_REF_INIT_ATOMIC, GFP_KERNEL))
-		goto err_map;
+		goto err_hctxs;
 
 	setup_timer(&q->timeout, blk_mq_rq_timer, (unsigned long) q);
 	blk_queue_rq_timeout(q, 30000);
Signature · 1 line
1.9.5 (Apple Git-50.3)
#2 lore raw

Re: [PATCH 1/7] blk-mq: fix use of incorrect goto label in blk_mq_init_queue error path

Quoted 13 lines · Mike Snitzer

If percpu_ref_init() fails the 'err_hctxs' label should be used instead of 'err_map'.

---
 block/blk-mq.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
block/blk-mq.c +0 -0
index 4f4bea2..459840c 100644

block/blk-mq.c struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *set) · line 1941

Quoted 5 lines
block/blk-mq.c +1 -1
@@ -1938,7 +1938,7 @@ struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *set)
         */
        if (percpu_ref_init(&q->mq_usage_counter, blk_mq_usage_counter_release,
                            PERCPU_REF_INIT_ATOMIC, GFP_KERNEL))
-               goto err_map;
+               goto err_hctxs;

If it is changed to 'goto err_hctxs', percpu_ref_init() need to move before blk_alloc_queue_node(), otherwise just 'goto err_hw' is enough, but the former is better.

Thanks,

#3 lore raw

Re: [PATCH 1/7] blk-mq: fix use of incorrect goto label in blk_mq_init_queue error path

Quoted 9 lines · Ming Lei
Quoted 8 lines · Mike Snitzer

If percpu_ref_init() fails the 'err_hctxs' label should be used instead of 'err_map'.

---
 block/blk-mq.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 14 lines

block/blk-mq.c struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *set) · line 1941

Quoted 10 lines
block/blk-mq.c +1 -1
index 4f4bea2..459840c 100644
@@ -1938,7 +1938,7 @@ struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *set)
         */
        if (percpu_ref_init(&q->mq_usage_counter, blk_mq_usage_counter_release,
                            PERCPU_REF_INIT_ATOMIC, GFP_KERNEL))
-               goto err_map;
+               goto err_hctxs;

If it is changed to 'goto err_hctxs', percpu_ref_init() need to move before blk_alloc_queue_node(), otherwise just 'goto err_hw' is enough, but the former is better.

Yes, you're correct. Patch 2 happened to eliminate this problem. But I'll get it fixed up and post v2.

The reason this thinko happened is I noticed the problem after I moved the blk_alloc_queue_node() out in patch 2 and when I rebased to reorder the fix before that patch 2 context got lost in the shuffle.

Thanks for your review!

#4 lore raw

Re: [PATCH 1/7] blk-mq: fix use of incorrect goto label in blk_mq_init_queue error path

Quoted 27 lines · Mike Snitzer
Quoted 9 lines · Ming Lei
Quoted 8 lines · Mike Snitzer

If percpu_ref_init() fails the 'err_hctxs' label should be used instead of 'err_map'.

---
 block/blk-mq.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 14 lines

block/blk-mq.c struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *set) · line 1941

Quoted 10 lines
block/blk-mq.c +1 -1
index 4f4bea2..459840c 100644
@@ -1938,7 +1938,7 @@ struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *set)
         */
        if (percpu_ref_init(&q->mq_usage_counter, blk_mq_usage_counter_release,
                            PERCPU_REF_INIT_ATOMIC, GFP_KERNEL))
-               goto err_map;
+               goto err_hctxs;

If it is changed to 'goto err_hctxs', percpu_ref_init() need to move before blk_alloc_queue_node(), otherwise just 'goto err_hw' is enough, but the former is better.

Quoted 2 lines

Yes, you're correct. Patch 2 happened to eliminate this problem. But I'll get it fixed up and post v2.

Actually, percpu_ref_init clearly cannot go before blk_alloc_queue_node since percpu_ref_init deferences q. (buildbot just emailed complaining about crashes with my v2 patch). I didn't test this 1/7 patch in isolation but should have -- when 2/7 is applied the problem goes away.

v3 of these first 2 patches will be inbound shortly.

#5 lore raw

PATCH v2 1/7 blk-mq: fix use of incorrect goto label in blk_mq_init_queue error path

In reply to Mike Snitzer

If percpu_ref_init() fails the 'err_hctxs' label should be used instead of 'err_map'.

Rather than reuse 'err_hw', which was introduced if the later blk_mq_init_hw_queues() call fails, move percpu_ref_init() before blk_alloc_queue_node().

---
 block/blk-mq.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)
block/blk-mq.c +5 -5
index 4f4bea2..0a877d2 100644
@@ -1928,17 +1928,17 @@ struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *set)
 		hctxs[i]->queue_num = i;
 	}
 
-	q = blk_alloc_queue_node(GFP_KERNEL, set->numa_node);
-	if (!q)
-		goto err_hctxs;
-
 	/*
 	 * Init percpu_ref in atomic mode so that it's faster to shutdown.
 	 * See blk_register_queue() for details.
 	 */
 	if (percpu_ref_init(&q->mq_usage_counter, blk_mq_usage_counter_release,
 			    PERCPU_REF_INIT_ATOMIC, GFP_KERNEL))
-		goto err_map;
+		goto err_hctxs;
+
+	q = blk_alloc_queue_node(GFP_KERNEL, set->numa_node);
+	if (!q)
+		goto err_hctxs;
 
 	setup_timer(&q->timeout, blk_mq_rq_timer, (unsigned long) q);
 	blk_queue_rq_timeout(q, 30000);
Signature · 1 line
1.9.5 (Apple Git-50.3)
Reviewed-by on v2 1/7
#6 lore raw

Re: [PATCH 1/7 v2] blk-mq: fix use of incorrect goto label in blk_mq_init_queue error path

Quoted 9 lines · Mike Snitzer

If percpu_ref_init() fails the 'err_hctxs' label should be used instead of 'err_map'.

Rather than reuse 'err_hw', which was introduced if the later blk_mq_init_hw_queues() call fails, move percpu_ref_init() before blk_alloc_queue_node().

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke		      zSeries & Storage
[email protected]			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)
#7 lore raw

PATCH v3 1/7 blk-mq: fix use of incorrect goto label in blk_mq_init_queue error path

In reply to Mike Snitzer

If percpu_ref_init() fails the allocated q and hctxs must get cleaned up; using 'err_map' doesn't allow that to happen.

---
 block/blk-mq.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
block/blk-mq.c +3 -3
index 4f4bea2..b7b8933 100644
@@ -1938,7 +1938,7 @@ struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *set)
 	 */
 	if (percpu_ref_init(&q->mq_usage_counter, blk_mq_usage_counter_release,
 			    PERCPU_REF_INIT_ATOMIC, GFP_KERNEL))
-		goto err_map;
+		goto err_mq_usage;
 
 	setup_timer(&q->timeout, blk_mq_rq_timer, (unsigned long) q);
 	blk_queue_rq_timeout(q, 30000);
@@ -1981,7 +1981,7 @@ struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *set)
 	blk_mq_init_cpu_queues(q, set->nr_hw_queues);
 
 	if (blk_mq_init_hw_queues(q, set))
-		goto err_hw;
+		goto err_mq_usage;
 
 	mutex_lock(&all_q_mutex);
 	list_add_tail(&q->all_q_node, &all_q_list);
@@ -1993,7 +1993,7 @@ struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *set)
 
 	return q;
 
-err_hw:
+err_mq_usage:
 	blk_cleanup_queue(q);
 err_hctxs:
 	kfree(map);
Signature · 1 line
1.9.5 (Apple Git-50.3)
#8 lore raw

PATCH 2/7 blk-mq: add blk_mq_init_allocated_queue and export blk_mq_register_disk

In reply to Mike Snitzer

Add a variant of blk_mq_init_queue that allows a previously allocated queue to be initialized. blk_mq_init_allocated_queue models blk_init_allocated_queue -- which was also created for DM's use.

DM's approach to device creation requires a placeholder request_queue be allocated for use with alloc_dev() but the decision about what type of request_queue will be ultimately created is deferred until all component devices referenced in the DM table are processed to determine the table type (request-based, blk-mq request-based, or bio-based).

---
 block/blk-mq-sysfs.c   |  1 +
 block/blk-mq.c         | 28 +++++++++++++++++++---------
 include/linux/blk-mq.h |  2 ++
 3 files changed, 22 insertions(+), 9 deletions(-)
block/blk-mq-sysfs.c +1 -0
index 1630a20..b79685e 100644
@@ -436,6 +436,7 @@ int blk_mq_register_disk(struct gendisk *disk)
 
 	return 0;
 }
+EXPORT_SYMBOL_GPL(blk_mq_register_disk);
 
 void blk_mq_sysfs_unregister(struct request_queue *q)
 {
block/blk-mq.c +19 -9
index 459840c..d576a78 100644
@@ -1891,9 +1891,25 @@ void blk_mq_release(struct request_queue *q)
 
 struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *set)
 {
+	struct request_queue *uninit_q, *q;
+
+	uninit_q = blk_alloc_queue_node(GFP_KERNEL, set->numa_node);
+	if (!uninit_q)
+		return ERR_PTR(-ENOMEM);
+
+	q = blk_mq_init_allocated_queue(set, uninit_q);
+	if (!q)
+		blk_cleanup_queue(uninit_q);
+
+	return q;
+}
+EXPORT_SYMBOL(blk_mq_init_queue);
+
+struct request_queue *blk_mq_init_allocated_queue(struct blk_mq_tag_set *set,
+						  struct request_queue *q)
+{
 	struct blk_mq_hw_ctx **hctxs;
 	struct blk_mq_ctx __percpu *ctx;
-	struct request_queue *q;
 	unsigned int *map;
 	int i;
 
@@ -1928,10 +1944,6 @@ struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *set)
 		hctxs[i]->queue_num = i;
 	}
 
-	q = blk_alloc_queue_node(GFP_KERNEL, set->numa_node);
-	if (!q)
-		goto err_hctxs;
-
 	/*
 	 * Init percpu_ref in atomic mode so that it's faster to shutdown.
 	 * See blk_register_queue() for details.
@@ -1981,7 +1993,7 @@ struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *set)
 	blk_mq_init_cpu_queues(q, set->nr_hw_queues);
 
 	if (blk_mq_init_hw_queues(q, set))
-		goto err_hw;
+		goto err_hctxs;
 
 	mutex_lock(&all_q_mutex);
 	list_add_tail(&q->all_q_node, &all_q_list);
@@ -1993,8 +2005,6 @@ struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *set)
 
 	return q;
 
-err_hw:
-	blk_cleanup_queue(q);
 err_hctxs:
 	kfree(map);
 	for (i = 0; i < set->nr_hw_queues; i++) {
@@ -2009,7 +2019,7 @@ err_percpu:
 	free_percpu(ctx);
 	return ERR_PTR(-ENOMEM);
 }
-EXPORT_SYMBOL(blk_mq_init_queue);
+EXPORT_SYMBOL(blk_mq_init_allocated_queue);
 
 void blk_mq_free_queue(struct request_queue *q)
 {
include/linux/blk-mq.h +2 -0
index 7aec861..9a75c88 100644
@@ -164,6 +164,8 @@ enum {
 		<< BLK_MQ_F_ALLOC_POLICY_START_BIT)
 
 struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *);
+struct request_queue *blk_mq_init_allocated_queue(struct blk_mq_tag_set *set,
+						  struct request_queue *q);
 void blk_mq_finish_init(struct request_queue *q);
 int blk_mq_register_disk(struct gendisk *);
 void blk_mq_unregister_disk(struct gendisk *);
Signature · 1 line
1.9.5 (Apple Git-50.3)
#9 lore raw

Re: [PATCH 2/7] blk-mq: add blk_mq_init_allocated_queue and export blk_mq_register_disk

Quoted 38 lines · Mike Snitzer

Add a variant of blk_mq_init_queue that allows a previously allocated queue to be initialized. blk_mq_init_allocated_queue models blk_init_allocated_queue -- which was also created for DM's use.

DM's approach to device creation requires a placeholder request_queue be allocated for use with alloc_dev() but the decision about what type of request_queue will be ultimately created is deferred until all component devices referenced in the DM table are processed to determine the table type (request-based, blk-mq request-based, or bio-based).

---
 block/blk-mq-sysfs.c   |  1 +
 block/blk-mq.c         | 28 +++++++++++++++++++---------
 include/linux/blk-mq.h |  2 ++
 3 files changed, 22 insertions(+), 9 deletions(-)
block/blk-mq-sysfs.c +1 -0
index 1630a20..b79685e 100644
@@ -436,6 +436,7 @@ int blk_mq_register_disk(struct gendisk *disk)
 
        return 0;
 }
+EXPORT_SYMBOL_GPL(blk_mq_register_disk);
 
 void blk_mq_sysfs_unregister(struct request_queue *q)
 {
block/blk-mq.c +1 -0
index 459840c..d576a78 100644
@@ -1891,9 +1891,25 @@ void blk_mq_release(struct request_queue *q)
 
 struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *set)
 {
+       struct request_queue *uninit_q, *q;

block/blk-mq.c void blk_mq_release(struct request_queue *q) · line 1904

Quoted 10 lines
block/blk-mq.c +10 -0
+
+       uninit_q = blk_alloc_queue_node(GFP_KERNEL, set->numa_node);
+       if (!uninit_q)
+               return ERR_PTR(-ENOMEM);
+
+       q = blk_mq_init_allocated_queue(set, uninit_q);
+       if (!q)
+               blk_cleanup_queue(uninit_q);
+
+       return q;

Current drivers suppose that queue won't be returned as null, and looks you need to change the check of !q as IS_ERR(q) and return ERR_PTR() if it is true.

Quoted 70 lines
block/blk-mq.c +8 -9
+}
+EXPORT_SYMBOL(blk_mq_init_queue);
+
+struct request_queue *blk_mq_init_allocated_queue(struct blk_mq_tag_set *set,
+                                                 struct request_queue *q)
+{
        struct blk_mq_hw_ctx **hctxs;
        struct blk_mq_ctx __percpu *ctx;
-       struct request_queue *q;
        unsigned int *map;
        int i;
 
@@ -1928,10 +1944,6 @@ struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *set)
                hctxs[i]->queue_num = i;
        }
 
-       q = blk_alloc_queue_node(GFP_KERNEL, set->numa_node);
-       if (!q)
-               goto err_hctxs;
-
        /*
         * Init percpu_ref in atomic mode so that it's faster to shutdown.
         * See blk_register_queue() for details.
@@ -1981,7 +1993,7 @@ struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *set)
        blk_mq_init_cpu_queues(q, set->nr_hw_queues);
 
        if (blk_mq_init_hw_queues(q, set))
-               goto err_hw;
+               goto err_hctxs;
 
        mutex_lock(&all_q_mutex);
        list_add_tail(&q->all_q_node, &all_q_list);
@@ -1993,8 +2005,6 @@ struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *set)
 
        return q;
 
-err_hw:
-       blk_cleanup_queue(q);
 err_hctxs:
        kfree(map);
        for (i = 0; i < set->nr_hw_queues; i++) {
@@ -2009,7 +2019,7 @@ err_percpu:
        free_percpu(ctx);
        return ERR_PTR(-ENOMEM);
 }
-EXPORT_SYMBOL(blk_mq_init_queue);
+EXPORT_SYMBOL(blk_mq_init_allocated_queue);
 
 void blk_mq_free_queue(struct request_queue *q)
 {
include/linux/blk-mq.h +2 -1
index 7aec861..9a75c88 100644
@@ -164,6 +164,8 @@ enum {
                << BLK_MQ_F_ALLOC_POLICY_START_BIT)
 
 struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *);
+struct request_queue *blk_mq_init_allocated_queue(struct blk_mq_tag_set *set,
+                                                 struct request_queue *q);
 void blk_mq_finish_init(struct request_queue *q);
 int blk_mq_register_disk(struct gendisk *);
 void blk_mq_unregister_disk(struct gendisk *);
--

1.9.5 (Apple Git-50.3)

--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 1 line
Ming Lei
#10 lore raw

PATCH v2 2/7 blk-mq: add blk_mq_init_allocated_queue and export blk_mq_register_disk

In reply to Mike Snitzer

Add a variant of blk_mq_init_queue that allows a previously allocated queue to be initialized. blk_mq_init_allocated_queue models blk_init_allocated_queue -- which was also created for DM's use.

DM's approach to device creation requires a placeholder request_queue be allocated for use with alloc_dev() but the decision about what type of request_queue will be ultimately created is deferred until all component devices referenced in the DM table are processed to determine the table type (request-based, blk-mq request-based, or bio-based).

Also, because of DM's late finalization of the request_queue type the call to blk_mq_register_disk() doesn't happen during alloc_dev(). Must export blk_mq_register_disk() so that DM can backfill the 'mq' dir once the blk-mq queue is fully allocated.

---
 block/blk-mq-sysfs.c   |  1 +
 block/blk-mq.c         | 24 +++++++++++++++++++-----
 include/linux/blk-mq.h |  2 ++
 3 files changed, 22 insertions(+), 5 deletions(-)
block/blk-mq-sysfs.c +1 -0
index 1630a20..b79685e 100644
@@ -436,6 +436,7 @@ int blk_mq_register_disk(struct gendisk *disk)
 
 	return 0;
 }
+EXPORT_SYMBOL_GPL(blk_mq_register_disk);
 
 void blk_mq_sysfs_unregister(struct request_queue *q)
 {
block/blk-mq.c +19 -5
index 0a877d2..1cab7831 100644
@@ -1891,9 +1891,25 @@ void blk_mq_release(struct request_queue *q)
 
 struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *set)
 {
+	struct request_queue *uninit_q, *q;
+
+	uninit_q = blk_alloc_queue_node(GFP_KERNEL, set->numa_node);
+	if (!uninit_q)
+		return ERR_PTR(-ENOMEM);
+
+	q = blk_mq_init_allocated_queue(set, uninit_q);
+	if (IS_ERR(q))
+		blk_cleanup_queue(uninit_q);
+
+	return q;
+}
+EXPORT_SYMBOL(blk_mq_init_queue);
+
+struct request_queue *blk_mq_init_allocated_queue(struct blk_mq_tag_set *set,
+						  struct request_queue *q)
+{
 	struct blk_mq_hw_ctx **hctxs;
 	struct blk_mq_ctx __percpu *ctx;
-	struct request_queue *q;
 	unsigned int *map;
 	int i;
 
@@ -1981,7 +1997,7 @@ struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *set)
 	blk_mq_init_cpu_queues(q, set->nr_hw_queues);
 
 	if (blk_mq_init_hw_queues(q, set))
-		goto err_hw;
+		goto err_hctxs;
 
 	mutex_lock(&all_q_mutex);
 	list_add_tail(&q->all_q_node, &all_q_list);
@@ -1993,8 +2009,6 @@ struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *set)
 
 	return q;
 
-err_hw:
-	blk_cleanup_queue(q);
 err_hctxs:
 	kfree(map);
 	for (i = 0; i < set->nr_hw_queues; i++) {
@@ -2009,7 +2023,7 @@ err_percpu:
 	free_percpu(ctx);
 	return ERR_PTR(-ENOMEM);
 }
-EXPORT_SYMBOL(blk_mq_init_queue);
+EXPORT_SYMBOL(blk_mq_init_allocated_queue);
 
 void blk_mq_free_queue(struct request_queue *q)
 {
include/linux/blk-mq.h +2 -0
index 7aec861..9a75c88 100644
@@ -164,6 +164,8 @@ enum {
 		<< BLK_MQ_F_ALLOC_POLICY_START_BIT)
 
 struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *);
+struct request_queue *blk_mq_init_allocated_queue(struct blk_mq_tag_set *set,
+						  struct request_queue *q);
 void blk_mq_finish_init(struct request_queue *q);
 int blk_mq_register_disk(struct gendisk *);
 void blk_mq_unregister_disk(struct gendisk *);
Signature · 1 line
1.9.5 (Apple Git-50.3)
Reviewed-by on v2 2/7
#11 lore raw

Re: [PATCH 2/7 v2] blk-mq: add blk_mq_init_allocated_queue and export blk_mq_register_disk

Quoted 10 lines · Mike Snitzer

Add a variant of blk_mq_init_queue that allows a previously allocated queue to be initialized. blk_mq_init_allocated_queue models blk_init_allocated_queue -- which was also created for DM's use.

DM's approach to device creation requires a placeholder request_queue be allocated for use with alloc_dev() but the decision about what type of request_queue will be ultimately created is deferred until all component devices referenced in the DM table are processed to determine the table type (request-based, blk-mq request-based, or bio-based).

Quoted 7 lines

Also, because of DM's late finalization of the request_queue type the call to blk_mq_register_disk() doesn't happen during alloc_dev(). Must export blk_mq_register_disk() so that DM can backfill the 'mq' dir once the blk-mq queue is fully allocated.

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke		      zSeries & Storage
[email protected]			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)
#12 lore raw

PATCH v3 2/7 blk-mq: add blk_mq_init_allocated_queue and export blk_mq_register_disk

In reply to Mike Snitzer

Add a variant of blk_mq_init_queue that allows a previously allocated queue to be initialized. blk_mq_init_allocated_queue models blk_init_allocated_queue -- which was also created for DM's use.

DM's approach to device creation requires a placeholder request_queue be allocated for use with alloc_dev() but the decision about what type of request_queue will be ultimately created is deferred until all component devices referenced in the DM table are processed to determine the table type (request-based, blk-mq request-based, or bio-based).

Also, because of DM's late finalization of the request_queue type the call to blk_mq_register_disk() doesn't happen during alloc_dev(). Must export blk_mq_register_disk() so that DM can backfill the 'mq' dir once the blk-mq queue is fully allocated.

---
 block/blk-mq-sysfs.c   |  1 +
 block/blk-mq.c         | 30 ++++++++++++++++++++----------
 include/linux/blk-mq.h |  2 ++
 3 files changed, 23 insertions(+), 10 deletions(-)
block/blk-mq-sysfs.c +1 -0
index 1630a20..b79685e 100644
@@ -436,6 +436,7 @@ int blk_mq_register_disk(struct gendisk *disk)
 
 	return 0;
 }
+EXPORT_SYMBOL_GPL(blk_mq_register_disk);
 
 void blk_mq_sysfs_unregister(struct request_queue *q)
 {
block/blk-mq.c +20 -10
index b7b8933..3000121 100644
@@ -1891,9 +1891,25 @@ void blk_mq_release(struct request_queue *q)
 
 struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *set)
 {
+	struct request_queue *uninit_q, *q;
+
+	uninit_q = blk_alloc_queue_node(GFP_KERNEL, set->numa_node);
+	if (!uninit_q)
+		return ERR_PTR(-ENOMEM);
+
+	q = blk_mq_init_allocated_queue(set, uninit_q);
+	if (IS_ERR(q))
+		blk_cleanup_queue(uninit_q);
+
+	return q;
+}
+EXPORT_SYMBOL(blk_mq_init_queue);
+
+struct request_queue *blk_mq_init_allocated_queue(struct blk_mq_tag_set *set,
+						  struct request_queue *q)
+{
 	struct blk_mq_hw_ctx **hctxs;
 	struct blk_mq_ctx __percpu *ctx;
-	struct request_queue *q;
 	unsigned int *map;
 	int i;
 
@@ -1928,17 +1944,13 @@ struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *set)
 		hctxs[i]->queue_num = i;
 	}
 
-	q = blk_alloc_queue_node(GFP_KERNEL, set->numa_node);
-	if (!q)
-		goto err_hctxs;
-
 	/*
 	 * Init percpu_ref in atomic mode so that it's faster to shutdown.
 	 * See blk_register_queue() for details.
 	 */
 	if (percpu_ref_init(&q->mq_usage_counter, blk_mq_usage_counter_release,
 			    PERCPU_REF_INIT_ATOMIC, GFP_KERNEL))
-		goto err_mq_usage;
+		goto err_hctxs;
 
 	setup_timer(&q->timeout, blk_mq_rq_timer, (unsigned long) q);
 	blk_queue_rq_timeout(q, 30000);
@@ -1981,7 +1993,7 @@ struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *set)
 	blk_mq_init_cpu_queues(q, set->nr_hw_queues);
 
 	if (blk_mq_init_hw_queues(q, set))
-		goto err_mq_usage;
+		goto err_hctxs;
 
 	mutex_lock(&all_q_mutex);
 	list_add_tail(&q->all_q_node, &all_q_list);
@@ -1993,8 +2005,6 @@ struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *set)
 
 	return q;
 
-err_mq_usage:
-	blk_cleanup_queue(q);
 err_hctxs:
 	kfree(map);
 	for (i = 0; i < set->nr_hw_queues; i++) {
@@ -2009,7 +2019,7 @@ err_percpu:
 	free_percpu(ctx);
 	return ERR_PTR(-ENOMEM);
 }
-EXPORT_SYMBOL(blk_mq_init_queue);
+EXPORT_SYMBOL(blk_mq_init_allocated_queue);
 
 void blk_mq_free_queue(struct request_queue *q)
 {
include/linux/blk-mq.h +2 -0
index 7aec861..9a75c88 100644
@@ -164,6 +164,8 @@ enum {
 		<< BLK_MQ_F_ALLOC_POLICY_START_BIT)
 
 struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *);
+struct request_queue *blk_mq_init_allocated_queue(struct blk_mq_tag_set *set,
+						  struct request_queue *q);
 void blk_mq_finish_init(struct request_queue *q);
 int blk_mq_register_disk(struct gendisk *);
 void blk_mq_unregister_disk(struct gendisk *);
Signature · 1 line
1.9.5 (Apple Git-50.3)
#13 lore raw

PATCH 3/7 blk-mq: export blk_mq_run_hw_queues

In reply to Mike Snitzer

Rename blk_mq_run_queues to blk_mq_run_hw_queues, add async argument, and export it.

DM's suspend support must be able to run the queue without starting stopped hw queues.

---
 block/blk-mq.c         | 8 ++++----
 include/linux/blk-mq.h | 1 +
 2 files changed, 5 insertions(+), 4 deletions(-)
block/blk-mq.c +4 -4
index d576a78..5706dd9 100644
@@ -33,7 +33,6 @@ static DEFINE_MUTEX(all_q_mutex);
 static LIST_HEAD(all_q_list);
 
 static void __blk_mq_run_hw_queue(struct blk_mq_hw_ctx *hctx);
-static void blk_mq_run_queues(struct request_queue *q);
 
 /*
  * Check if any of the ctx's have pending work in this hardware queue
@@ -118,7 +117,7 @@ void blk_mq_freeze_queue_start(struct request_queue *q)
 
 	if (freeze) {
 		percpu_ref_kill(&q->mq_usage_counter);
-		blk_mq_run_queues(q);
+		blk_mq_run_hw_queues(q, false);
 	}
 }
 EXPORT_SYMBOL_GPL(blk_mq_freeze_queue_start);
@@ -904,7 +903,7 @@ void blk_mq_run_hw_queue(struct blk_mq_hw_ctx *hctx, bool async)
 			&hctx->run_work, 0);
 }
 
-static void blk_mq_run_queues(struct request_queue *q)
+void blk_mq_run_hw_queues(struct request_queue *q, bool async)
 {
 	struct blk_mq_hw_ctx *hctx;
 	int i;
@@ -915,9 +914,10 @@ static void blk_mq_run_queues(struct request_queue *q)
 		    test_bit(BLK_MQ_S_STOPPED, &hctx->state))
 			continue;
 
-		blk_mq_run_hw_queue(hctx, false);
+		blk_mq_run_hw_queue(hctx, async);
 	}
 }
+EXPORT_SYMBOL(blk_mq_run_hw_queues);
 
 void blk_mq_stop_hw_queue(struct blk_mq_hw_ctx *hctx)
 {
include/linux/blk-mq.h +1 -0
index 9a75c88..ebfe707 100644
@@ -220,6 +220,7 @@ void blk_mq_start_hw_queue(struct blk_mq_hw_ctx *hctx);
 void blk_mq_stop_hw_queues(struct request_queue *q);
 void blk_mq_start_hw_queues(struct request_queue *q);
 void blk_mq_start_stopped_hw_queues(struct request_queue *q, bool async);
+void blk_mq_run_hw_queues(struct request_queue *q, bool async);
 void blk_mq_delay_queue(struct blk_mq_hw_ctx *hctx, unsigned long msecs);
 void blk_mq_tag_busy_iter(struct blk_mq_hw_ctx *hctx, busy_iter_fn *fn,
 		void *priv);
Signature · 1 line
1.9.5 (Apple Git-50.3)
#14 lore raw

PATCH 4/7 blk-mq: don't wait in blk_mq_queue_enter() if __GFP_WAIT isn't set

In reply to Mike Snitzer

From: Keith Busch <[email protected]>

Return -EBUSY if we're unable to enter a queue immediately when allocating a blk-mq request without __GFP_WAIT.

---
 block/blk-mq.c | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)
block/blk-mq.c +6 -3
index 5706dd9..01b2912 100644
@@ -77,7 +77,7 @@ static void blk_mq_hctx_clear_pending(struct blk_mq_hw_ctx *hctx,
 	clear_bit(CTX_TO_BIT(hctx, ctx), &bm->word);
 }
 
-static int blk_mq_queue_enter(struct request_queue *q)
+static int blk_mq_queue_enter(struct request_queue *q, gfp_t gfp)
 {
 	while (true) {
 		int ret;
@@ -85,6 +85,9 @@ static int blk_mq_queue_enter(struct request_queue *q)
 		if (percpu_ref_tryget_live(&q->mq_usage_counter))
 			return 0;
 
+		if (!(gfp & __GFP_WAIT))
+			return -EBUSY;
+
 		ret = wait_event_interruptible(q->mq_freeze_wq,
 				!q->mq_freeze_depth || blk_queue_dying(q));
 		if (blk_queue_dying(q))
@@ -256,7 +259,7 @@ struct request *blk_mq_alloc_request(struct request_queue *q, int rw, gfp_t gfp,
 	struct blk_mq_alloc_data alloc_data;
 	int ret;
 
-	ret = blk_mq_queue_enter(q);
+	ret = blk_mq_queue_enter(q, gfp);
 	if (ret)
 		return ERR_PTR(ret);
 
@@ -1186,7 +1189,7 @@ static struct request *blk_mq_map_request(struct request_queue *q,
 	int rw = bio_data_dir(bio);
 	struct blk_mq_alloc_data alloc_data;
 
-	if (unlikely(blk_mq_queue_enter(q))) {
+	if (unlikely(blk_mq_queue_enter(q, GFP_KERNEL))) {
 		bio_endio(bio, -EIO);
 		return NULL;
 	}
Signature · 1 line
1.9.5 (Apple Git-50.3)
#15 lore raw

PATCH 5/7 dm: add full blk-mq support to request-based DM

In reply to Mike Snitzer

Commit e5863d9ad ("dm: allocate requests in target when stacking on blk-mq devices") served as the first step toward fully utilizing blk-mq in request-based DM -- it enabled stacking a old-style (request_fn) request_queue ontop of the underlying blk-mq device(s). This first step didn't improve performance of DM multipath ontop of fast blk-mq devices (e.g. NVMe) because the top-level old-style request_queue was severely limited by the queue_lock.

The second step offered here enables stacking a blk-mq request_queue ontop of the underlying blk-mq device(s). This unlocks significant performance gains on fast blk-mq devices, Keith Busch tested on his NVMe testbed and offered this really positive news:

"Just providing a performance update. All my fio tests are getting
roughly equal performance whether accessed through the raw block device or the multipath device mapper (~470k IOPS). I could only push ~20% of the raw iops through dm before this conversion, so this latest tree is looking really solid from a performance standpoint."

---
 drivers/md/dm-mpath.c |   4 +-
 drivers/md/dm-table.c |  11 +-
 drivers/md/dm.c       | 326 +++++++++++++++++++++++++++++++++++++++-----------
 3 files changed, 269 insertions(+), 72 deletions(-)
drivers/md/dm-mpath.c +2 -2
index add6391..2bb6599 100644
@@ -428,9 +428,9 @@ static int __multipath_map(struct dm_target *ti, struct request *clone,
 	} else {
 		/* blk-mq request-based interface */
 		*__clone = blk_get_request(bdev_get_queue(bdev),
-					   rq_data_dir(rq), GFP_KERNEL);
+					   rq_data_dir(rq), GFP_ATOMIC);
 		if (IS_ERR(*__clone))
-			/* ENOMEM, requeue */
+			/* ENOMEM or EBUSY, requeue */
 			return r;
 		(*__clone)->bio = (*__clone)->biotail = NULL;
 		(*__clone)->rq_disk = bdev->bd_disk;
drivers/md/dm-table.c +8 -3
index 0573120..66600ca 100644
@@ -18,6 +18,7 @@
 #include <linux/mutex.h>
 #include <linux/delay.h>
 #include <linux/atomic.h>
+#include <linux/blk-mq.h>
 
 #define DM_MSG_PREFIX "table"
 
@@ -1695,9 +1696,13 @@ void dm_table_run_md_queue_async(struct dm_table *t)
 	md = dm_table_get_md(t);
 	queue = dm_get_md_queue(md);
 	if (queue) {
-		spin_lock_irqsave(queue->queue_lock, flags);
-		blk_run_queue_async(queue);
-		spin_unlock_irqrestore(queue->queue_lock, flags);
+		if (queue->mq_ops)
+			blk_mq_run_hw_queues(queue, true);
+		else {
+			spin_lock_irqsave(queue->queue_lock, flags);
+			blk_run_queue_async(queue);
+			spin_unlock_irqrestore(queue->queue_lock, flags);
+		}
 	}
 }
 EXPORT_SYMBOL(dm_table_run_md_queue_async);
drivers/md/dm.c +259 -67

549 lines not on this page · raw message

Signature · 1 line
1.9.5 (Apple Git-50.3)
#16 lore raw

PATCH 6/7 dm: optimize dm_mq_queue_rq to _not_ use kthread if using pure blk-mq

In reply to Mike Snitzer

dm_mq_queue_rq() is in atomic context so care must be taken to not sleep -- as such GFP_ATOMIC is used for the md->bs bioset allocations and dm-mpath's call to blk_get_request(). In the future the bioset allocations will hopefully go away (by removing support for partial completions of a request).

But the kthread will still be used to queue work if blk-mq is used ontop of old-style request_fn device(s). Also prepare for supporting DM blk-mq ontop of old-style request_fn device(s) if a new dm-mod 'use_blk_mq' parameter is set.

---
 drivers/md/dm.c | 65 +++++++++++++++++++++++++++++++++++++++------------------
 1 file changed, 45 insertions(+), 20 deletions(-)
drivers/md/dm.c +45 -20
index b5409ac..b0c965a 100644
@@ -1074,9 +1074,10 @@ static void free_rq_clone(struct request *clone)
 
 	blk_rq_unprep_clone(clone);
 
-	if (clone->q && clone->q->mq_ops)
+	if (clone->q->mq_ops)
 		tio->ti->type->release_clone_rq(clone);
-	else
+	else if (!md->queue->mq_ops)
+		/* request_fn queue stacked on request_fn queue(s) */
 		free_clone_request(md, clone);
 
 	if (!md->queue->mq_ops)
@@ -1835,15 +1836,25 @@ static int setup_clone(struct request *clone, struct request *rq,
 static struct request *clone_rq(struct request *rq, struct mapped_device *md,
 				struct dm_rq_target_io *tio, gfp_t gfp_mask)
 {
-	struct request *clone = alloc_clone_request(md, gfp_mask);
+	/*
+	 * Do not allocate a clone if tio->clone was already set
+	 * (see: dm_mq_queue_rq).
+	 */
+	bool alloc_clone = !tio->clone;
+	struct request *clone;
 
-	if (!clone)
-		return NULL;
+	if (alloc_clone) {
+		clone = alloc_clone_request(md, gfp_mask);
+		if (!clone)
+			return NULL;
+	} else
+		clone = tio->clone;
 
 	blk_rq_init(NULL, clone);
 	if (setup_clone(clone, rq, tio, gfp_mask)) {
 		/* -ENOMEM */
-		free_clone_request(md, clone);
+		if (alloc_clone)
+			free_clone_request(md, clone);
 		return NULL;
 	}
 
@@ -1861,7 +1872,8 @@ static void init_tio(struct dm_rq_target_io *tio, struct request *rq,
 	tio->orig = rq;
 	tio->error = 0;
 	memset(&tio->info, 0, sizeof(tio->info));
-	init_kthread_work(&tio->work, map_tio_request);
+	if (md->kworker_task)
+		init_kthread_work(&tio->work, map_tio_request);
 }
 
 static struct dm_rq_target_io *prep_tio(struct request *rq,
@@ -1938,7 +1950,7 @@ static int map_request(struct dm_rq_target_io *tio, struct request *rq,
 		}
 		if (IS_ERR(clone))
 			return DM_MAPIO_REQUEUE;
-		if (setup_clone(clone, rq, tio, GFP_NOIO)) {
+		if (setup_clone(clone, rq, tio, GFP_ATOMIC)) {
 			/* -ENOMEM */
 			ti->type->release_clone_rq(clone);
 			return DM_MAPIO_REQUEUE;
@@ -2403,7 +2415,7 @@ static void __bind_mempools(struct mapped_device *md, struct dm_table *t)
 	p->bs = NULL;
 
 out:
-	/* mempool bind completed, now no need any mempools in the table */
+	/* mempool bind completed, no longer need any mempools in the table */
 	dm_table_free_md_mempools(t);
 }
 
@@ -2708,17 +2720,25 @@ static int dm_mq_queue_rq(struct blk_mq_hw_ctx *hctx,
 	/* Init tio using md established in .init_request */
 	init_tio(tio, rq, md);
 
+	/*
+	 * Establish tio->ti before queuing work (map_tio_request)
+	 * or making direct call to map_request().
+	 */
+	tio->ti = ti;
+
 	/* Clone the request if underlying devices aren't blk-mq */
 	if (dm_table_get_type(map) == DM_TYPE_REQUEST_BASED) {
-		// FIXME: make the memory for clone part of the pdu
+		/* clone request is allocated at the end of the pdu */
+		tio->clone = (void *)blk_mq_rq_to_pdu(rq) + sizeof(struct dm_rq_target_io);
 		if (!clone_rq(rq, md, tio, GFP_ATOMIC))
 			return BLK_MQ_RQ_QUEUE_BUSY;
+		queue_kthread_work(&md->kworker, &tio->work);
+	} else {
+		/* Direct call is fine since .queue_rq allows allocations */
+		if (map_request(tio, rq, md) == DM_MAPIO_REQUEUE)
+			dm_requeue_unmapped_original_request(md, rq);
 	}
 
-	/* Establish tio->ti before queuing work (map_tio_request) */
-	tio->ti = ti;
-	queue_kthread_work(&md->kworker, &tio->work);
-
 	return BLK_MQ_RQ_QUEUE_OK;
 }
 
@@ -2731,6 +2751,7 @@ static struct blk_mq_ops dm_mq_ops = {
 
 static int dm_init_request_based_blk_mq_queue(struct mapped_device *md)
 {
+	unsigned md_type = dm_get_md_type(md);
 	struct request_queue *q;
 	int err;
 
@@ -2740,9 +2761,11 @@ static int dm_init_request_based_blk_mq_queue(struct mapped_device *md)
 	md->tag_set.numa_node = NUMA_NO_NODE;
 	md->tag_set.flags = BLK_MQ_F_SHOULD_MERGE | BLK_MQ_F_SG_MERGE;
 	md->tag_set.nr_hw_queues = 1;
-	// FIXME: make the memory for non-blk-mq clone part of the pdu
-	// would need to be done only if new 'use_blk_mq' is set in DM sysfs
-	md->tag_set.cmd_size = sizeof(struct dm_rq_target_io);
+	if (md_type == DM_TYPE_REQUEST_BASED) {
+		/* make the memory for non-blk-mq clone part of the pdu */
+		md->tag_set.cmd_size = sizeof(struct dm_rq_target_io) + sizeof(struct request);
+	} else
+		md->tag_set.cmd_size = sizeof(struct dm_rq_target_io);
 	md->tag_set.driver_data = md;
 
 	err = blk_mq_alloc_tag_set(&md->tag_set);
@@ -2760,7 +2783,8 @@ static int dm_init_request_based_blk_mq_queue(struct mapped_device *md)
 	/* backfill 'mq' sysfs registration normally done in blk_register_queue */
 	blk_mq_register_disk(md->disk);
 
-	init_rq_based_worker_thread(md);
+	if (md_type == DM_TYPE_REQUEST_BASED)
+		init_rq_based_worker_thread(md);
 
 	return 0;
 
@@ -2879,7 +2903,7 @@ static void __dm_destroy(struct mapped_device *md, bool wait)
 	set_bit(DMF_FREEING, &md->flags);
 	spin_unlock(&_minor_lock);
 
-	if (dm_request_based(md))
+	if (dm_request_based(md) && md->kworker_task)
 		flush_kthread_worker(&md->kworker);
 
 	/*
@@ -3133,7 +3157,8 @@ static int __dm_suspend(struct mapped_device *md, struct dm_table *map,
 	 */
 	if (dm_request_based(md)) {
 		stop_queue(md->queue);
-		flush_kthread_worker(&md->kworker);
+		if (md->kworker_task)
+			flush_kthread_worker(&md->kworker);
 	}
 
 	flush_workqueue(md->wq);
Signature · 1 line
1.9.5 (Apple Git-50.3)
#17 lore raw

PATCH 7/7 dm: add 'use_blk_mq' module param and expose in per-device ro sysfs attr

In reply to Mike Snitzer

Request-based DM's blk-mq support defaults to off; but a user can easily change the default using the dm_mod.use_blk_mq module/boot option.

Also, you can check what mode a given request-based DM device is using
with: cat /sys/block/dm-X/dm/use_blk_mq

This change enabled further cleanup and reduced work (e.g. the md->io_pool and md->rq_pool isn't created if using blk-mq).

---
 drivers/md/Kconfig    | 11 +++++++++++
 drivers/md/dm-sysfs.c |  9 +++++++++
 drivers/md/dm-table.c |  6 +++---
 drivers/md/dm.c       | 51 ++++++++++++++++++++++++++++++++++++++++-----------
 drivers/md/dm.h       |  5 ++++-
 5 files changed, 67 insertions(+), 15 deletions(-)
drivers/md/Kconfig +11 -0
index 63e05e3..109f9dc 100644
@@ -196,6 +196,17 @@ config BLK_DEV_DM
 
 	  If unsure, say N.
 
+config DM_MQ_DEFAULT
+	bool "request-based DM: use blk-mq I/O path by default"
+	depends on BLK_DEV_DM
+	---help---
+	  This option enables the blk-mq based I/O path for request-based
+	  DM devices by default.  With the option the dm_mod.use_blk_mq
+	  module/boot option defaults to Y, without it to N, but it can
+	  still be overriden either way.
+
+	  If unsure say N.
+
 config DM_DEBUG
 	bool "Device mapper debugging support"
 	depends on BLK_DEV_DM
drivers/md/dm-sysfs.c +9 -0
index f5bb394..7e818f5 100644
@@ -89,15 +89,24 @@ static ssize_t dm_attr_suspended_show(struct mapped_device *md, char *buf)
 	return strlen(buf);
 }
 
+static ssize_t dm_attr_use_blk_mq_show(struct mapped_device *md, char *buf)
+{
+	sprintf(buf, "%d\n", dm_use_blk_mq(md));
+
+	return strlen(buf);
+}
+
 static DM_ATTR_RO(name);
 static DM_ATTR_RO(uuid);
 static DM_ATTR_RO(suspended);
+static DM_ATTR_RO(use_blk_mq);
 static DM_ATTR_RW(rq_based_seq_io_merge_deadline);
 
 static struct attribute *dm_attrs[] = {
 	&dm_attr_name.attr,
 	&dm_attr_uuid.attr,
 	&dm_attr_suspended.attr,
+	&dm_attr_use_blk_mq.attr,
 	&dm_attr_rq_based_seq_io_merge_deadline.attr,
 	NULL,
 };
drivers/md/dm-table.c +3 -3
index 66600ca..8d025f3 100644
@@ -940,7 +940,7 @@ bool dm_table_mq_request_based(struct dm_table *t)
 	return dm_table_get_type(t) == DM_TYPE_MQ_REQUEST_BASED;
 }
 
-static int dm_table_alloc_md_mempools(struct dm_table *t)
+static int dm_table_alloc_md_mempools(struct dm_table *t, struct mapped_device *md)
 {
 	unsigned type = dm_table_get_type(t);
 	unsigned per_bio_data_size = 0;
@@ -958,7 +958,7 @@ static int dm_table_alloc_md_mempools(struct dm_table *t)
 			per_bio_data_size = max(per_bio_data_size, tgt->per_bio_data_size);
 		}
 
-	t->mempools = dm_alloc_md_mempools(type, t->integrity_supported, per_bio_data_size);
+	t->mempools = dm_alloc_md_mempools(md, type, t->integrity_supported, per_bio_data_size);
 	if (!t->mempools)
 		return -ENOMEM;
 
@@ -1128,7 +1128,7 @@ int dm_table_complete(struct dm_table *t)
 		return r;
 	}
 
-	r = dm_table_alloc_md_mempools(t);
+	r = dm_table_alloc_md_mempools(t, t->md);
 	if (r)
 		DMERR("unable to allocate mempools");
 
drivers/md/dm.c +40 -11

141 lines not on this page · raw message

drivers/md/dm.h +4 -1
index 5522422..6123c2b 100644
@@ -211,6 +211,8 @@ int dm_kobject_uevent(struct mapped_device *md, enum kobject_action action,
 void dm_internal_suspend(struct mapped_device *md);
 void dm_internal_resume(struct mapped_device *md);
 
+bool dm_use_blk_mq(struct mapped_device *md);
+
 int dm_io_init(void);
 void dm_io_exit(void);
 
@@ -220,7 +222,8 @@ void dm_kcopyd_exit(void);
 /*
  * Mempool operations
  */
-struct dm_md_mempools *dm_alloc_md_mempools(unsigned type, unsigned integrity, unsigned per_bio_data_size);
+struct dm_md_mempools *dm_alloc_md_mempools(struct mapped_device *md, unsigned type,
+					    unsigned integrity, unsigned per_bio_data_size);
 void dm_free_md_mempools(struct dm_md_mempools *pools);
 
 /*
Signature · 1 line
1.9.5 (Apple Git-50.3)
maintainer Applied
#18 lore raw
In reply to Mike Snitzer
Quoted 41 lines · Mike Snitzer

This patchset updates request-based DM (used exclussively by DM multipath) to support blk-mq I/O path or the old ->request_fn I/O path -- the default is still the old ->request_fn I/O path.

Like was done for SCSI, I've exposed a dm-mod.ko 'use_blk_mq' module parameter and a CONFIG_DM_MQ_DEFAULT. I've queued these patches in linux-next via linux-dm.git's 'for-next' branch -- my goal is for these changes to land upstream in 4.1.

I've tested all 4 supported permutations of rq-based queue stacking:
1) blk-mq mpath stacked on blk-mq device(s)
2) blk-mq mpath stacked on ->request_fn device(s)
3) old ->request_fn mpath stacked on blk-mq device(s)
4) old ->request_fn mpath stacked on old ->request_fn device(s)

There is definitely much more room for optimizing the blk-mq mpath queue (via dm_mq_ops setup) to have more awareness about the characteristics of the underlying blk-mq device(s) -- Jens offered some suggestions that are worth pursuing. Goal being to leverage the ability to have multiple software submission queues that map to the underlying paths' HW queue. I expect Keith to be the most likely person to pursue these follow-on optimizations given his immediate access to NVMe devices, etc.

blk-mq allows for the removal of 2 of the 3 mempools and associated allocations that were traditionally performed using old ->request_fn I/O path. The md->bs bioset still remains for cloning a request's bios.

I'll be pursuing the possibility of removing the bio cloning that happens when cloning a request (it was put in place for partial completion of a request's bios but the prospect of avoiding those bio clones warrants further review of how useful those partial completions are, see: http://www.redhat.com/archives/dm-devel/2014-June/msg00030.html)

Jens and Keith, special thanks for your help answering more blk-mq questions than I'd have hoped would be needed while at LSF/MM.

Keith Busch (1):
blk-mq: don't wait in blk_mq_queue_enter() if __GFP_WAIT isn't set

Quoted 7 lines

Mike Snitzer (6):
blk-mq: fix use of incorrect goto label in blk_mq_init_queue error path
blk-mq: add blk_mq_init_allocated_queue and export blk_mq_register_disk
blk-mq: export blk_mq_run_hw_queues
dm: add full blk-mq support to request-based DM
dm: optimize dm_mq_queue_rq to _not_ use kthread if using pure blk-mq
dm: add 'use_blk_mq' module param and expose in per-device ro sysfs attr

Applied patch 1 to current tree, marked for stable. 2-4 are applied for 4.1, in for-4.1/core.

Signature · 1 line
Jens Axboe