dm: add full blk-mq support to request-based DM
Subsystems (4)
cover dm: add full blk-mq support to request-based DM
This patchset updates request-based DM (used exclussively by DM multipath) to support blk-mq I/O path or the old ->request_fn I/O path -- the default is still the old ->request_fn I/O path.
Like was done for SCSI, I've exposed a dm-mod.ko 'use_blk_mq' module parameter and a CONFIG_DM_MQ_DEFAULT. I've queued these patches in linux-next via linux-dm.git's 'for-next' branch -- my goal is for these changes to land upstream in 4.1.
I've tested all 4 supported permutations of rq-based queue stacking:
1) blk-mq mpath stacked on blk-mq device(s)
2) blk-mq mpath stacked on ->request_fn device(s)
3) old ->request_fn mpath stacked on blk-mq device(s)
4) old ->request_fn mpath stacked on old ->request_fn device(s)
There is definitely much more room for optimizing the blk-mq mpath queue (via dm_mq_ops setup) to have more awareness about the characteristics of the underlying blk-mq device(s) -- Jens offered some suggestions that are worth pursuing. Goal being to leverage the ability to have multiple software submission queues that map to the underlying paths' HW queue. I expect Keith to be the most likely person to pursue these follow-on optimizations given his immediate access to NVMe devices, etc.
blk-mq allows for the removal of 2 of the 3 mempools and associated allocations that were traditionally performed using old ->request_fn I/O path. The md->bs bioset still remains for cloning a request's bios.
I'll be pursuing the possibility of removing the bio cloning that happens when cloning a request (it was put in place for partial completion of a request's bios but the prospect of avoiding those bio clones warrants further review of how useful those partial completions are, see: http://www.redhat.com/archives/dm-devel/2014-June/msg00030.html)
Jens and Keith, special thanks for your help answering more blk-mq questions than I'd have hoped would be needed while at LSF/MM.
Keith Busch (1):
blk-mq: don't wait in blk_mq_queue_enter() if __GFP_WAIT isn't set
Mike Snitzer (6):
blk-mq: fix use of incorrect goto label in blk_mq_init_queue error path
blk-mq: add blk_mq_init_allocated_queue and export blk_mq_register_disk
blk-mq: export blk_mq_run_hw_queues
dm: add full blk-mq support to request-based DM
dm: optimize dm_mq_queue_rq to _not_ use kthread if using pure blk-mq
dm: add 'use_blk_mq' module param and expose in per-device ro sysfs attr
block/blk-mq-sysfs.c | 1 + block/blk-mq.c | 47 ++++-- drivers/md/Kconfig | 11 ++ drivers/md/dm-mpath.c | 4 +- drivers/md/dm-sysfs.c | 9 ++ drivers/md/dm-table.c | 17 +- drivers/md/dm.c | 414 +++++++++++++++++++++++++++++++++++++++---------- drivers/md/dm.h | 5 +- include/linux/blk-mq.h | 3 + 9 files changed, 401 insertions(+), 110 deletions(-)
Signature · 1 line
1.9.5 (Apple Git-50.3)
PATCH 1/7 blk-mq: fix use of incorrect goto label in blk_mq_init_queue error path
If percpu_ref_init() fails the 'err_hctxs' label should be used instead of 'err_map'.
--- block/blk-mq.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
block/blk-mq.c +1 -1
@@ -1938,7 +1938,7 @@ struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *set)*/ if (percpu_ref_init (&q->mq_usage_counter, blk_mq_usage_counter_release,PERCPU_REF_INIT_ATOMIC ,GFP_KERNEL )) - goto err_map; + goto err_hctxs;setup_timer (&q->timeout, blk_mq_rq_timer, (unsigned long ) q);blk_queue_rq_timeout (q,30000 );
Signature · 1 line
1.9.5 (Apple Git-50.3)
Re: [PATCH 1/7] blk-mq: fix use of incorrect goto label in blk_mq_init_queue error path
Quoted 13 lines · Mike Snitzer
If
percpu_ref_init()fails the 'err_hctxs' label should be used instead of 'err_map'.Signed-off-by Mike Snitzer <[email protected]>--- block/blk-mq.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)block/blk-mq.c +0
-0
block/blk-mq.c struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *set) · line 1941
Quoted 5 lines
block/blk-mq.c +1
-1@@ -1938,7 +1938,7 @@ struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *set)*/ if (percpu_ref_init (&q->mq_usage_counter, blk_mq_usage_counter_release,PERCPU_REF_INIT_ATOMIC ,GFP_KERNEL )) - goto err_map; + goto err_hctxs;
If it is changed to 'goto err_hctxs', percpu_ref_init() need to move before blk_alloc_queue_node(), otherwise just 'goto err_hw' is enough, but the former is better.
Thanks,
Re: [PATCH 1/7] blk-mq: fix use of incorrect goto label in blk_mq_init_queue error path
Quoted 9 lines · Ming Lei
Quoted 8 lines · Mike Snitzer
If
percpu_ref_init()fails the 'err_hctxs' label should be used instead of 'err_map'.Signed-off-by Mike Snitzer <[email protected]>--- block/blk-mq.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 14 lines
block/blk-mq.c struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *set) · line 1941
Quoted 10 lines
block/blk-mq.c +1
-1@@ -1938,7 +1938,7 @@ struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *set)*/ if (percpu_ref_init (&q->mq_usage_counter, blk_mq_usage_counter_release,PERCPU_REF_INIT_ATOMIC ,GFP_KERNEL )) - goto err_map; + goto err_hctxs;If it is changed to 'goto
err_hctxs',percpu_ref_init()need to move beforeblk_alloc_queue_node(), otherwise just 'gotoerr_hw' is enough, but the former is better.
Yes, you're correct. Patch 2 happened to eliminate this problem. But I'll get it fixed up and post v2.
The reason this thinko happened is I noticed the problem after I moved the blk_alloc_queue_node() out in patch 2 and when I rebased to reorder the fix before that patch 2 context got lost in the shuffle.
Thanks for your review!
Re: [PATCH 1/7] blk-mq: fix use of incorrect goto label in blk_mq_init_queue error path
Quoted 27 lines · Mike Snitzer
Quoted 9 lines · Ming Lei
Quoted 8 lines · Mike Snitzer
If
percpu_ref_init()fails the 'err_hctxs' label should be used instead of 'err_map'.Signed-off-by Mike Snitzer <[email protected]>--- block/blk-mq.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)Quoted 14 lines
block/blk-mq.c struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *set) · line 1941
Quoted 10 lines
block/blk-mq.c +1
-1@@ -1938,7 +1938,7 @@ struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *set)*/ if (percpu_ref_init (&q->mq_usage_counter, blk_mq_usage_counter_release,PERCPU_REF_INIT_ATOMIC ,GFP_KERNEL )) - goto err_map; + goto err_hctxs;If it is changed to 'goto
err_hctxs',percpu_ref_init()need to move beforeblk_alloc_queue_node(), otherwise just 'gotoerr_hw' is enough, but the former is better.
Quoted 2 lines
Yes, you're correct. Patch 2 happened to eliminate this problem. But I'll get it fixed up and post v2.
Actually, percpu_ref_init clearly cannot go before blk_alloc_queue_node since percpu_ref_init deferences q. (buildbot just emailed complaining about crashes with my v2 patch). I didn't test this 1/7 patch in isolation but should have -- when 2/7 is applied the problem goes away.
v3 of these first 2 patches will be inbound shortly.
PATCH v2 1/7 blk-mq: fix use of incorrect goto label in blk_mq_init_queue error path
In reply to Mike SnitzerIf percpu_ref_init() fails the 'err_hctxs' label should be used instead of 'err_map'.
Rather than reuse 'err_hw', which was introduced if the later blk_mq_init_hw_queues() call fails, move percpu_ref_init() before blk_alloc_queue_node().
--- block/blk-mq.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
block/blk-mq.c +5 -5
@@ -1928,17 +1928,17 @@ struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *set) hctxs[i]->queue_num = i; } - q =blk_alloc_queue_node (GFP_KERNEL , set->numa_node); -if (!q) - goto err_hctxs; -/* * Init percpu_ref in atomic mode so that it's faster to shutdown. * See blk_register_queue() for details. */ if (percpu_ref_init (&q->mq_usage_counter, blk_mq_usage_counter_release,PERCPU_REF_INIT_ATOMIC ,GFP_KERNEL )) - goto err_map; + goto err_hctxs; + + q =blk_alloc_queue_node (GFP_KERNEL , set->numa_node); +if (!q) + goto err_hctxs;setup_timer (&q->timeout, blk_mq_rq_timer, (unsigned long ) q);blk_queue_rq_timeout (q,30000 );
Signature · 1 line
1.9.5 (Apple Git-50.3)
Re: [PATCH 1/7 v2] blk-mq: fix use of incorrect goto label in blk_mq_init_queue error path
Quoted 9 lines · Mike Snitzer
If
percpu_ref_init()fails the 'err_hctxs' label should be used instead of 'err_map'.Rather than reuse '
err_hw', which was introduced if the laterblk_mq_init_hw_queues()call fails, movepercpu_ref_init()beforeblk_alloc_queue_node().Signed-off-by Mike Snitzer <[email protected]>Reviewed-by Ming Lei <[email protected]>
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke zSeries & Storage [email protected] +49 911 74053 688 SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)
PATCH v3 1/7 blk-mq: fix use of incorrect goto label in blk_mq_init_queue error path
In reply to Mike SnitzerIf percpu_ref_init() fails the allocated q and hctxs must get cleaned up; using 'err_map' doesn't allow that to happen.
--- block/blk-mq.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
block/blk-mq.c +3 -3
@@ -1938,7 +1938,7 @@ struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *set)*/ if (percpu_ref_init (&q->mq_usage_counter, blk_mq_usage_counter_release,PERCPU_REF_INIT_ATOMIC ,GFP_KERNEL )) - goto err_map; + goto err_mq_usage;setup_timer (&q->timeout, blk_mq_rq_timer, (unsigned long ) q);blk_queue_rq_timeout (q,30000 );
@@ -1981,7 +1981,7 @@ struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *set)blk_mq_init_cpu_queues (q, set->nr_hw_queues);if (blk_mq_init_hw_queues (q, set)) - goto err_hw; + goto err_mq_usage;mutex_lock (&all_q_mutex);list_add_tail (&q->all_q_node, &all_q_list);
@@ -1993,7 +1993,7 @@ struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *set)return q; -err_hw: +err_mq_usage:blk_cleanup_queue (q); err_hctxs:kfree (map);
Signature · 1 line
1.9.5 (Apple Git-50.3)
PATCH 2/7 blk-mq: add blk_mq_init_allocated_queue and export blk_mq_register_disk
In reply to Mike SnitzerAdd a variant of blk_mq_init_queue that allows a previously allocated queue to be initialized. blk_mq_init_allocated_queue models blk_init_allocated_queue -- which was also created for DM's use.
DM's approach to device creation requires a placeholder request_queue be allocated for use with alloc_dev() but the decision about what type of request_queue will be ultimately created is deferred until all component devices referenced in the DM table are processed to determine the table type (request-based, blk-mq request-based, or bio-based).
--- block/blk-mq-sysfs.c | 1 + block/blk-mq.c | 28 +++++++++++++++++++--------- include/linux/blk-mq.h | 2 ++ 3 files changed, 22 insertions(+), 9 deletions(-)
block/blk-mq-sysfs.c +1 -0
@@ -436,6 +436,7 @@ int blk_mq_register_disk(struct gendisk *disk)return 0 ; } +EXPORT_SYMBOL_GPL (blk_mq_register_disk);void blk_mq_sysfs_unregister (struct request_queue *q) {
block/blk-mq.c +19 -9
@@ -1891,9 +1891,25 @@ void blk_mq_release(struct request_queue *q)struct request_queue *blk_mq_init_queue (struct blk_mq_tag_set *set) { +struct request_queue *uninit_q, *q; + + uninit_q =blk_alloc_queue_node (GFP_KERNEL , set->numa_node); +if (!uninit_q) +return ERR_PTR (-ENOMEM ); + + q =blk_mq_init_allocated_queue (set, uninit_q); +if (!q) +blk_cleanup_queue (uninit_q); + +return q; +} +EXPORT_SYMBOL (blk_mq_init_queue); + +struct request_queue *blk_mq_init_allocated_queue (struct blk_mq_tag_set *set, +struct request_queue *q) +{struct blk_mq_hw_ctx **hctxs;struct blk_mq_ctx __percpu *ctx; -struct request_queue *q;unsigned int *map;int i;
@@ -1928,10 +1944,6 @@ struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *set) hctxs[i]->queue_num = i; } - q =blk_alloc_queue_node (GFP_KERNEL , set->numa_node); -if (!q) - goto err_hctxs; -/* * Init percpu_ref in atomic mode so that it's faster to shutdown. * See blk_register_queue() for details.
@@ -1981,7 +1993,7 @@ struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *set)blk_mq_init_cpu_queues (q, set->nr_hw_queues);if (blk_mq_init_hw_queues (q, set)) - goto err_hw; + goto err_hctxs;mutex_lock (&all_q_mutex);list_add_tail (&q->all_q_node, &all_q_list);
@@ -1993,8 +2005,6 @@ struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *set)return q; -err_hw: -blk_cleanup_queue (q); err_hctxs:kfree (map);for (i =0 ; i < set->nr_hw_queues; i++) {
@@ -2009,7 +2019,7 @@ err_percpu:free_percpu (ctx);return ERR_PTR (-ENOMEM ); } -EXPORT_SYMBOL (blk_mq_init_queue); +EXPORT_SYMBOL (blk_mq_init_allocated_queue);void blk_mq_free_queue (struct request_queue *q) {
include/linux/blk-mq.h +2 -0
@@ -164,6 +164,8 @@ enum { << BLK_MQ_F_ALLOC_POLICY_START_BIT)struct request_queue *blk_mq_init_queue (struct blk_mq_tag_set *); +struct request_queue *blk_mq_init_allocated_queue (struct blk_mq_tag_set *set, +struct request_queue *q);void blk_mq_finish_init (struct request_queue *q);int blk_mq_register_disk (struct gendisk *);void blk_mq_unregister_disk (struct gendisk *);
Signature · 1 line
1.9.5 (Apple Git-50.3)
Re: [PATCH 2/7] blk-mq: add blk_mq_init_allocated_queue and export blk_mq_register_disk
Quoted 38 lines · Mike Snitzer
Add a variant of
blk_mq_init_queuethat allows a previously allocated queue to be initialized.blk_mq_init_allocated_queuemodelsblk_init_allocated_queue-- which was also created for DM's use.DM's approach to device creation requires a placeholder
request_queuebe allocated for use withalloc_dev()but the decision about what type ofrequest_queuewill be ultimately created is deferred until all component devices referenced in the DM table are processed to determine the table type (request-based, blk-mq request-based, or bio-based).Signed-off-by Mike Snitzer <[email protected]>--- block/blk-mq-sysfs.c | 1 + block/blk-mq.c | 28 +++++++++++++++++++--------- include/linux/blk-mq.h | 2 ++ 3 files changed, 22 insertions(+), 9 deletions(-)block/blk-mq-sysfs.c +1
-0@@ -436,6 +436,7 @@ int blk_mq_register_disk(struct gendisk *disk)return 0 ; } +EXPORT_SYMBOL_GPL (blk_mq_register_disk);void blk_mq_sysfs_unregister (struct request_queue *q) {block/blk-mq.c +1
-0@@ -1891,9 +1891,25 @@ void blk_mq_release(struct request_queue *q)struct request_queue *blk_mq_init_queue (struct blk_mq_tag_set *set) { +struct request_queue *uninit_q, *q;
block/blk-mq.c void blk_mq_release(struct request_queue *q) · line 1904
Quoted 10 lines
block/blk-mq.c +10
-0+ + uninit_q =blk_alloc_queue_node (GFP_KERNEL , set->numa_node); +if (!uninit_q) +return ERR_PTR (-ENOMEM ); + + q =blk_mq_init_allocated_queue (set, uninit_q); +if (!q) +blk_cleanup_queue (uninit_q); + +return q;
Current drivers suppose that queue won't be returned as null, and looks you need to change the check of !q as IS_ERR(q) and return ERR_PTR() if it is true.
Quoted 70 lines
block/blk-mq.c +8
-9+} +EXPORT_SYMBOL (blk_mq_init_queue); + +struct request_queue *blk_mq_init_allocated_queue (struct blk_mq_tag_set *set, +struct request_queue *q) +{struct blk_mq_hw_ctx **hctxs;struct blk_mq_ctx __percpu *ctx; -struct request_queue *q;unsigned int *map;int i;@@ -1928,10 +1944,6 @@ struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *set) hctxs[i]->queue_num = i; } - q =blk_alloc_queue_node (GFP_KERNEL , set->numa_node); -if (!q) - goto err_hctxs; -/* * Init percpu_ref in atomic mode so that it's faster to shutdown. * See blk_register_queue() for details. @@ -1981,7 +1993,7 @@ struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *set)blk_mq_init_cpu_queues (q, set->nr_hw_queues);if (blk_mq_init_hw_queues (q, set)) - goto err_hw; + goto err_hctxs;mutex_lock (&all_q_mutex);list_add_tail (&q->all_q_node, &all_q_list);@@ -1993,8 +2005,6 @@ struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *set)return q; -err_hw: -blk_cleanup_queue (q); err_hctxs:kfree (map);for (i =0 ; i < set->nr_hw_queues; i++) {@@ -2009,7 +2019,7 @@ err_percpu:free_percpu (ctx);return ERR_PTR (-ENOMEM ); } -EXPORT_SYMBOL (blk_mq_init_queue); +EXPORT_SYMBOL (blk_mq_init_allocated_queue);void blk_mq_free_queue (struct request_queue *q) {include/linux/blk-mq.h +2
-1@@ -164,6 +164,8 @@ enum { << BLK_MQ_F_ALLOC_POLICY_START_BIT)struct request_queue *blk_mq_init_queue (struct blk_mq_tag_set *); +struct request_queue *blk_mq_init_allocated_queue (struct blk_mq_tag_set *set, +struct request_queue *q);void blk_mq_finish_init (struct request_queue *q);int blk_mq_register_disk (struct gendisk *);void blk_mq_unregister_disk (struct gendisk *); --1.9.5 (Apple Git-50.3)
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 1 line
Ming Lei
PATCH v2 2/7 blk-mq: add blk_mq_init_allocated_queue and export blk_mq_register_disk
In reply to Mike SnitzerAdd a variant of blk_mq_init_queue that allows a previously allocated queue to be initialized. blk_mq_init_allocated_queue models blk_init_allocated_queue -- which was also created for DM's use.
DM's approach to device creation requires a placeholder request_queue be allocated for use with alloc_dev() but the decision about what type of request_queue will be ultimately created is deferred until all component devices referenced in the DM table are processed to determine the table type (request-based, blk-mq request-based, or bio-based).
Also, because of DM's late finalization of the request_queue type the call to blk_mq_register_disk() doesn't happen during alloc_dev(). Must export blk_mq_register_disk() so that DM can backfill the 'mq' dir once the blk-mq queue is fully allocated.
--- block/blk-mq-sysfs.c | 1 + block/blk-mq.c | 24 +++++++++++++++++++----- include/linux/blk-mq.h | 2 ++ 3 files changed, 22 insertions(+), 5 deletions(-)
block/blk-mq-sysfs.c +1 -0
@@ -436,6 +436,7 @@ int blk_mq_register_disk(struct gendisk *disk)return 0 ; } +EXPORT_SYMBOL_GPL (blk_mq_register_disk);void blk_mq_sysfs_unregister (struct request_queue *q) {
block/blk-mq.c +19 -5
@@ -1891,9 +1891,25 @@ void blk_mq_release(struct request_queue *q)struct request_queue *blk_mq_init_queue (struct blk_mq_tag_set *set) { +struct request_queue *uninit_q, *q; + + uninit_q =blk_alloc_queue_node (GFP_KERNEL , set->numa_node); +if (!uninit_q) +return ERR_PTR (-ENOMEM ); + + q =blk_mq_init_allocated_queue (set, uninit_q); +if (IS_ERR (q)) +blk_cleanup_queue (uninit_q); + +return q; +} +EXPORT_SYMBOL (blk_mq_init_queue); + +struct request_queue *blk_mq_init_allocated_queue (struct blk_mq_tag_set *set, +struct request_queue *q) +{struct blk_mq_hw_ctx **hctxs;struct blk_mq_ctx __percpu *ctx; -struct request_queue *q;unsigned int *map;int i;
@@ -1981,7 +1997,7 @@ struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *set)blk_mq_init_cpu_queues (q, set->nr_hw_queues);if (blk_mq_init_hw_queues (q, set)) - goto err_hw; + goto err_hctxs;mutex_lock (&all_q_mutex);list_add_tail (&q->all_q_node, &all_q_list);
@@ -1993,8 +2009,6 @@ struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *set)return q; -err_hw: -blk_cleanup_queue (q); err_hctxs:kfree (map);for (i =0 ; i < set->nr_hw_queues; i++) {
@@ -2009,7 +2023,7 @@ err_percpu:free_percpu (ctx);return ERR_PTR (-ENOMEM ); } -EXPORT_SYMBOL (blk_mq_init_queue); +EXPORT_SYMBOL (blk_mq_init_allocated_queue);void blk_mq_free_queue (struct request_queue *q) {
include/linux/blk-mq.h +2 -0
@@ -164,6 +164,8 @@ enum { << BLK_MQ_F_ALLOC_POLICY_START_BIT)struct request_queue *blk_mq_init_queue (struct blk_mq_tag_set *); +struct request_queue *blk_mq_init_allocated_queue (struct blk_mq_tag_set *set, +struct request_queue *q);void blk_mq_finish_init (struct request_queue *q);int blk_mq_register_disk (struct gendisk *);void blk_mq_unregister_disk (struct gendisk *);
Signature · 1 line
1.9.5 (Apple Git-50.3)
Re: [PATCH 2/7 v2] blk-mq: add blk_mq_init_allocated_queue and export blk_mq_register_disk
Quoted 10 lines · Mike Snitzer
Add a variant of
blk_mq_init_queuethat allows a previously allocated queue to be initialized.blk_mq_init_allocated_queuemodelsblk_init_allocated_queue-- which was also created for DM's use.DM's approach to device creation requires a placeholder
request_queuebe allocated for use withalloc_dev()but the decision about what type ofrequest_queuewill be ultimately created is deferred until all component devices referenced in the DM table are processed to determine the table type (request-based, blk-mq request-based, or bio-based).
Quoted 7 lines
Also, because of DM's late finalization of the
request_queuetype the call toblk_mq_register_disk()doesn't happen duringalloc_dev(). Must exportblk_mq_register_disk()so that DM can backfill the 'mq' dir once the blk-mq queue is fully allocated.Signed-off-by Mike Snitzer <[email protected]>Reviewed-by Ming Lei <[email protected]>
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke zSeries & Storage [email protected] +49 911 74053 688 SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)
PATCH v3 2/7 blk-mq: add blk_mq_init_allocated_queue and export blk_mq_register_disk
In reply to Mike SnitzerAdd a variant of blk_mq_init_queue that allows a previously allocated queue to be initialized. blk_mq_init_allocated_queue models blk_init_allocated_queue -- which was also created for DM's use.
DM's approach to device creation requires a placeholder request_queue be allocated for use with alloc_dev() but the decision about what type of request_queue will be ultimately created is deferred until all component devices referenced in the DM table are processed to determine the table type (request-based, blk-mq request-based, or bio-based).
Also, because of DM's late finalization of the request_queue type the call to blk_mq_register_disk() doesn't happen during alloc_dev(). Must export blk_mq_register_disk() so that DM can backfill the 'mq' dir once the blk-mq queue is fully allocated.
--- block/blk-mq-sysfs.c | 1 + block/blk-mq.c | 30 ++++++++++++++++++++---------- include/linux/blk-mq.h | 2 ++ 3 files changed, 23 insertions(+), 10 deletions(-)
block/blk-mq-sysfs.c +1 -0
@@ -436,6 +436,7 @@ int blk_mq_register_disk(struct gendisk *disk)return 0 ; } +EXPORT_SYMBOL_GPL (blk_mq_register_disk);void blk_mq_sysfs_unregister (struct request_queue *q) {
block/blk-mq.c +20 -10
@@ -1891,9 +1891,25 @@ void blk_mq_release(struct request_queue *q)struct request_queue *blk_mq_init_queue (struct blk_mq_tag_set *set) { +struct request_queue *uninit_q, *q; + + uninit_q =blk_alloc_queue_node (GFP_KERNEL , set->numa_node); +if (!uninit_q) +return ERR_PTR (-ENOMEM ); + + q =blk_mq_init_allocated_queue (set, uninit_q); +if (IS_ERR (q)) +blk_cleanup_queue (uninit_q); + +return q; +} +EXPORT_SYMBOL (blk_mq_init_queue); + +struct request_queue *blk_mq_init_allocated_queue (struct blk_mq_tag_set *set, +struct request_queue *q) +{struct blk_mq_hw_ctx **hctxs;struct blk_mq_ctx __percpu *ctx; -struct request_queue *q;unsigned int *map;int i;
@@ -1928,17 +1944,13 @@ struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *set) hctxs[i]->queue_num = i; } - q =blk_alloc_queue_node (GFP_KERNEL , set->numa_node); -if (!q) - goto err_hctxs; -/* * Init percpu_ref in atomic mode so that it's faster to shutdown. * See blk_register_queue() for details. */ if (percpu_ref_init (&q->mq_usage_counter, blk_mq_usage_counter_release,PERCPU_REF_INIT_ATOMIC ,GFP_KERNEL )) - goto err_mq_usage; + goto err_hctxs;setup_timer (&q->timeout, blk_mq_rq_timer, (unsigned long ) q);blk_queue_rq_timeout (q,30000 );
@@ -1981,7 +1993,7 @@ struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *set)blk_mq_init_cpu_queues (q, set->nr_hw_queues);if (blk_mq_init_hw_queues (q, set)) - goto err_mq_usage; + goto err_hctxs;mutex_lock (&all_q_mutex);list_add_tail (&q->all_q_node, &all_q_list);
@@ -1993,8 +2005,6 @@ struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *set)return q; -err_mq_usage: -blk_cleanup_queue (q); err_hctxs:kfree (map);for (i =0 ; i < set->nr_hw_queues; i++) {
@@ -2009,7 +2019,7 @@ err_percpu:free_percpu (ctx);return ERR_PTR (-ENOMEM ); } -EXPORT_SYMBOL (blk_mq_init_queue); +EXPORT_SYMBOL (blk_mq_init_allocated_queue);void blk_mq_free_queue (struct request_queue *q) {
include/linux/blk-mq.h +2 -0
@@ -164,6 +164,8 @@ enum { << BLK_MQ_F_ALLOC_POLICY_START_BIT)struct request_queue *blk_mq_init_queue (struct blk_mq_tag_set *); +struct request_queue *blk_mq_init_allocated_queue (struct blk_mq_tag_set *set, +struct request_queue *q);void blk_mq_finish_init (struct request_queue *q);int blk_mq_register_disk (struct gendisk *);void blk_mq_unregister_disk (struct gendisk *);
Signature · 1 line
1.9.5 (Apple Git-50.3)
PATCH 3/7 blk-mq: export blk_mq_run_hw_queues
In reply to Mike SnitzerRename blk_mq_run_queues to blk_mq_run_hw_queues, add async argument, and export it.
DM's suspend support must be able to run the queue without starting stopped hw queues.
--- block/blk-mq.c | 8 ++++---- include/linux/blk-mq.h | 1 + 2 files changed, 5 insertions(+), 4 deletions(-)
block/blk-mq.c +4 -4
@@ -33,7 +33,6 @@ static DEFINE_MUTEX(all_q_mutex);static LIST_HEAD (all_q_list);static void __blk_mq_run_hw_queue (struct blk_mq_hw_ctx *hctx); -static void blk_mq_run_queues (struct request_queue *q);/* * Check if any of the ctx's have pending work in this hardware queue
@@ -118,7 +117,7 @@ void blk_mq_freeze_queue_start(struct request_queue *q)if (freeze) {percpu_ref_kill (&q->mq_usage_counter); -blk_mq_run_queues (q); +blk_mq_run_hw_queues (q, false); } }EXPORT_SYMBOL_GPL (blk_mq_freeze_queue_start);
@@ -904,7 +903,7 @@ void blk_mq_run_hw_queue(struct blk_mq_hw_ctx *hctx, bool async) &hctx->run_work, 0); } -static void blk_mq_run_queues (struct request_queue *q) +void blk_mq_run_hw_queues (struct request_queue *q,bool async) {struct blk_mq_hw_ctx *hctx;int i;
@@ -915,9 +914,10 @@ static void blk_mq_run_queues(struct request_queue *q) test_bit(BLK_MQ_S_STOPPED, &hctx->state))continue ; -blk_mq_run_hw_queue (hctx, false); +blk_mq_run_hw_queue (hctx, async); } } +EXPORT_SYMBOL (blk_mq_run_hw_queues);void blk_mq_stop_hw_queue (struct blk_mq_hw_ctx *hctx) {
include/linux/blk-mq.h +1 -0
@@ -220,6 +220,7 @@ void blk_mq_start_hw_queue(struct blk_mq_hw_ctx *hctx);void blk_mq_stop_hw_queues (struct request_queue *q);void blk_mq_start_hw_queues (struct request_queue *q);void blk_mq_start_stopped_hw_queues (struct request_queue *q,bool async); +void blk_mq_run_hw_queues (struct request_queue *q,bool async);void blk_mq_delay_queue (struct blk_mq_hw_ctx *hctx,unsigned long msecs);void blk_mq_tag_busy_iter (struct blk_mq_hw_ctx *hctx,busy_iter_fn *fn,void *priv);
Signature · 1 line
1.9.5 (Apple Git-50.3)
PATCH 4/7 blk-mq: don't wait in blk_mq_queue_enter() if __GFP_WAIT isn't set
In reply to Mike SnitzerFrom: Keith Busch <[email protected]>
Return -EBUSY if we're unable to enter a queue immediately when allocating a blk-mq request without __GFP_WAIT.
--- block/blk-mq.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)
block/blk-mq.c +6 -3
@@ -77,7 +77,7 @@ static void blk_mq_hctx_clear_pending(struct blk_mq_hw_ctx *hctx,clear_bit (CTX_TO_BIT (hctx, ctx), &bm->word); } -static int blk_mq_queue_enter (struct request_queue *q) +static int blk_mq_queue_enter (struct request_queue *q,gfp_t gfp) {while (true) {int ret;
@@ -85,6 +85,9 @@ static int blk_mq_queue_enter(struct request_queue *q)if (percpu_ref_tryget_live (&q->mq_usage_counter))return 0 ; +if (!(gfp & __GFP_WAIT)) +return -EBUSY ; + ret =wait_event_interruptible (q->mq_freeze_wq, !q->mq_freeze_depth ||blk_queue_dying (q));if (blk_queue_dying (q))
@@ -256,7 +259,7 @@ struct request *blk_mq_alloc_request(struct request_queue *q, int rw, gfp_t gfp,struct blk_mq_alloc_data alloc_data;int ret; - ret =blk_mq_queue_enter (q); + ret =blk_mq_queue_enter (q, gfp);if (ret)return ERR_PTR (ret);
@@ -1186,7 +1189,7 @@ static struct request *blk_mq_map_request(struct request_queue *q,int rw =bio_data_dir (bio);struct blk_mq_alloc_data alloc_data; -if (unlikely (blk_mq_queue_enter (q))) { +if (unlikely (blk_mq_queue_enter (q,GFP_KERNEL ))) {bio_endio (bio, -EIO );return NULL ; }
Signature · 1 line
1.9.5 (Apple Git-50.3)
PATCH 5/7 dm: add full blk-mq support to request-based DM
In reply to Mike SnitzerCommit e5863d9ad ("dm: allocate requests in target when stacking on blk-mq devices") served as the first step toward fully utilizing blk-mq in request-based DM -- it enabled stacking a old-style (request_fn) request_queue ontop of the underlying blk-mq device(s). This first step didn't improve performance of DM multipath ontop of fast blk-mq devices (e.g. NVMe) because the top-level old-style request_queue was severely limited by the queue_lock.
The second step offered here enables stacking a blk-mq request_queue ontop of the underlying blk-mq device(s). This unlocks significant performance gains on fast blk-mq devices, Keith Busch tested on his NVMe testbed and offered this really positive news:
"Just providing a performance update. All my fio tests are getting
roughly equal performance whether accessed through the raw block device or the multipath device mapper (~470k IOPS). I could only push ~20% of the raw iops through dm before this conversion, so this latest tree is looking really solid from a performance standpoint."
--- drivers/md/dm-mpath.c | 4 +- drivers/md/dm-table.c | 11 +- drivers/md/dm.c | 326 +++++++++++++++++++++++++++++++++++++++----------- 3 files changed, 269 insertions(+), 72 deletions(-)
drivers/md/dm-mpath.c +2 -2
@@ -428,9 +428,9 @@ static int __multipath_map(struct dm_target *ti, struct request *clone, }else {/* blk-mq request-based interface */ *__clone =blk_get_request (bdev_get_queue (bdev), -rq_data_dir (rq),GFP_KERNEL ); +rq_data_dir (rq),GFP_ATOMIC );if (IS_ERR (*__clone)) -/* ENOMEM, requeue */ +/* ENOMEM or EBUSY, requeue */ return r; (*__clone)->bio = (*__clone)->biotail =NULL ; (*__clone)->rq_disk = bdev->bd_disk;
drivers/md/dm-table.c +8 -3
@@ -18,6 +18,7 @@#include <linux/mutex.h> #include <linux/delay.h> #include <linux/atomic.h> +#include <linux/blk-mq.h> #define DM_MSG_PREFIX "table"
@@ -1695,9 +1696,13 @@ void dm_table_run_md_queue_async(struct dm_table *t) md =dm_table_get_md (t); queue =dm_get_md_queue (md);if (queue) { -spin_lock_irqsave (queue->queue_lock, flags); -blk_run_queue_async (queue); -spin_unlock_irqrestore (queue->queue_lock, flags); +if (queue->mq_ops) +blk_mq_run_hw_queues (queue, true); +else { +spin_lock_irqsave (queue->queue_lock, flags); +blk_run_queue_async (queue); +spin_unlock_irqrestore (queue->queue_lock, flags); + } } }EXPORT_SYMBOL (dm_table_run_md_queue_async);
drivers/md/dm.c +259 -67
549 lines not on this page · raw message
Signature · 1 line
1.9.5 (Apple Git-50.3)
PATCH 6/7 dm: optimize dm_mq_queue_rq to _not_ use kthread if using pure blk-mq
In reply to Mike Snitzerdm_mq_queue_rq() is in atomic context so care must be taken to not sleep -- as such GFP_ATOMIC is used for the md->bs bioset allocations and dm-mpath's call to blk_get_request(). In the future the bioset allocations will hopefully go away (by removing support for partial completions of a request).
But the kthread will still be used to queue work if blk-mq is used ontop of old-style request_fn device(s). Also prepare for supporting DM blk-mq ontop of old-style request_fn device(s) if a new dm-mod 'use_blk_mq' parameter is set.
--- drivers/md/dm.c | 65 +++++++++++++++++++++++++++++++++++++++------------------ 1 file changed, 45 insertions(+), 20 deletions(-)
drivers/md/dm.c +45 -20
@@ -1074,9 +1074,10 @@ static void free_rq_clone(struct request *clone)blk_rq_unprep_clone (clone); -if (clone->q && clone->q->mq_ops) +if (clone->q->mq_ops) tio->ti->type->release_clone_rq (clone); -else +else if (!md->queue->mq_ops) +/* request_fn queue stacked on request_fn queue(s) */ free_clone_request (md, clone);if (!md->queue->mq_ops)
@@ -1835,15 +1836,25 @@ static int setup_clone(struct request *clone, struct request *rq,static struct request *clone_rq (struct request *rq,struct mapped_device *md,struct dm_rq_target_io *tio,gfp_t gfp_mask) { -struct request *clone =alloc_clone_request (md, gfp_mask); +/* +* Do not allocate a clone if tio->clone was already set +* (see: dm_mq_queue_rq). +*/ +bool alloc_clone = !tio->clone; +struct request *clone; -if (!clone) -return NULL ; +if (alloc_clone) { + clone =alloc_clone_request (md, gfp_mask); +if (!clone) +return NULL ; + }else + clone = tio->clone;blk_rq_init (NULL , clone);if (setup_clone (clone, rq, tio, gfp_mask)) {/* -ENOMEM */ -free_clone_request (md, clone); +if (alloc_clone) +free_clone_request (md, clone);return NULL ; }
@@ -1861,7 +1872,8 @@ static void init_tio(struct dm_rq_target_io *tio, struct request *rq, tio->orig = rq; tio->error =0 ;memset (&tio->info,0 ,sizeof (tio->info)); -init_kthread_work (&tio->work, map_tio_request); +if (md->kworker_task) +init_kthread_work (&tio->work, map_tio_request); } static struct dm_rq_target_io *prep_tio(struct request *rq,
@@ -1938,7 +1950,7 @@ static int map_request(struct dm_rq_target_io *tio, struct request *rq, }if (IS_ERR (clone))return DM_MAPIO_REQUEUE ; -if (setup_clone (clone, rq, tio,GFP_NOIO )) { +if (setup_clone (clone, rq, tio,GFP_ATOMIC )) {/* -ENOMEM */ ti->type->release_clone_rq (clone);return DM_MAPIO_REQUEUE ;
@@ -2403,7 +2415,7 @@ static void __bind_mempools(struct mapped_device *md, struct dm_table *t) p->bs =NULL ; out: -/* mempool bind completed, now no need any mempools in the table */ +/* mempool bind completed, no longer need any mempools in the table */ dm_table_free_md_mempools (t); }
@@ -2708,17 +2720,25 @@ static int dm_mq_queue_rq(struct blk_mq_hw_ctx *hctx,/* Init tio using md established in .init_request */ init_tio (tio, rq, md); +/* +* Establish tio->ti before queuing work (map_tio_request) +* or making direct call to map_request(). +*/ + tio->ti = ti; +/* Clone the request if underlying devices aren't blk-mq */ if (dm_table_get_type (map) ==DM_TYPE_REQUEST_BASED ) { -// FIXME: make the memory for clone part of the pdu +/* clone request is allocated at the end of the pdu */ + tio->clone = (void *)blk_mq_rq_to_pdu (rq) +sizeof (struct dm_rq_target_io );if (!clone_rq (rq, md, tio,GFP_ATOMIC ))return BLK_MQ_RQ_QUEUE_BUSY ; +queue_kthread_work (&md->kworker, &tio->work); + }else { +/* Direct call is fine since .queue_rq allows allocations */ +if (map_request (tio, rq, md) ==DM_MAPIO_REQUEUE ) +dm_requeue_unmapped_original_request (md, rq); } -/* Establish tio->ti before queuing work (map_tio_request) */ - tio->ti = ti; -queue_kthread_work (&md->kworker, &tio->work); -return BLK_MQ_RQ_QUEUE_OK ; }
@@ -2731,6 +2751,7 @@ static struct blk_mq_ops dm_mq_ops = {static int dm_init_request_based_blk_mq_queue (struct mapped_device *md) { +unsigned md_type =dm_get_md_type (md);struct request_queue *q;int err;
@@ -2740,9 +2761,11 @@ static int dm_init_request_based_blk_mq_queue(struct mapped_device *md) md->tag_set.numa_node =NUMA_NO_NODE ; md->tag_set.flags =BLK_MQ_F_SHOULD_MERGE |BLK_MQ_F_SG_MERGE ; md->tag_set.nr_hw_queues =1 ; -// FIXME: make the memory for non-blk-mq clone part of the pdu -// would need to be done only if new 'use_blk_mq' is set in DM sysfs - md->tag_set.cmd_size =sizeof (struct dm_rq_target_io ); +if (md_type ==DM_TYPE_REQUEST_BASED ) { +/* make the memory for non-blk-mq clone part of the pdu */ + md->tag_set.cmd_size =sizeof (struct dm_rq_target_io ) +sizeof (struct request ); + }else + md->tag_set.cmd_size =sizeof (struct dm_rq_target_io ); md->tag_set.driver_data = md; err =blk_mq_alloc_tag_set (&md->tag_set);
@@ -2760,7 +2783,8 @@ static int dm_init_request_based_blk_mq_queue(struct mapped_device *md)/* backfill 'mq' sysfs registration normally done in blk_register_queue */ blk_mq_register_disk (md->disk); -init_rq_based_worker_thread (md); +if (md_type ==DM_TYPE_REQUEST_BASED ) +init_rq_based_worker_thread (md);return 0 ;
@@ -2879,7 +2903,7 @@ static void __dm_destroy(struct mapped_device *md, bool wait)set_bit (DMF_FREEING , &md->flags);spin_unlock (&_minor_lock); -if (dm_request_based (md)) +if (dm_request_based (md) && md->kworker_task)flush_kthread_worker (&md->kworker);/*
@@ -3133,7 +3157,8 @@ static int __dm_suspend(struct mapped_device *md, struct dm_table *map,*/ if (dm_request_based (md)) {stop_queue (md->queue); -flush_kthread_worker (&md->kworker); +if (md->kworker_task) +flush_kthread_worker (&md->kworker); }flush_workqueue (md->wq);
Signature · 1 line
1.9.5 (Apple Git-50.3)
PATCH 7/7 dm: add 'use_blk_mq' module param and expose in per-device ro sysfs attr
In reply to Mike SnitzerRequest-based DM's blk-mq support defaults to off; but a user can easily change the default using the dm_mod.use_blk_mq module/boot option.
Also, you can check what mode a given request-based DM device is using
with: cat /sys/block/dm-X/dm/use_blk_mq
This change enabled further cleanup and reduced work (e.g. the md->io_pool and md->rq_pool isn't created if using blk-mq).
--- drivers/md/Kconfig | 11 +++++++++++ drivers/md/dm-sysfs.c | 9 +++++++++ drivers/md/dm-table.c | 6 +++--- drivers/md/dm.c | 51 ++++++++++++++++++++++++++++++++++++++++----------- drivers/md/dm.h | 5 ++++- 5 files changed, 67 insertions(+), 15 deletions(-)
drivers/md/Kconfig +11 -0
@@ -196,6 +196,17 @@ config BLK_DEV_DM If unsure, say N. +config DM_MQ_DEFAULT + bool "request-based DM: use blk-mq I/O path by default" + depends on BLK_DEV_DM + ---help--- + This option enables the blk-mq based I/O path for request-based + DM devices by default. With the option the dm_mod.use_blk_mq + module/boot option defaults to Y, without it to N, but it can + still be overriden either way. + + If unsure say N. + config DM_DEBUG bool "Device mapper debugging support" depends on BLK_DEV_DM
drivers/md/dm-sysfs.c +9 -0
@@ -89,15 +89,24 @@ static ssize_t dm_attr_suspended_show(struct mapped_device *md, char *buf)return strlen (buf); } +static ssize_t dm_attr_use_blk_mq_show (struct mapped_device *md,char *buf) +{ +sprintf (buf,"%d\n" ,dm_use_blk_mq (md)); + +return strlen (buf); +} +static DM_ATTR_RO (name);static DM_ATTR_RO (uuid);static DM_ATTR_RO (suspended); +static DM_ATTR_RO (use_blk_mq);static DM_ATTR_RW (rq_based_seq_io_merge_deadline);static struct attribute *dm_attrs[] = { &dm_attr_name.attr, &dm_attr_uuid.attr, &dm_attr_suspended.attr, + &dm_attr_use_blk_mq.attr, &dm_attr_rq_based_seq_io_merge_deadline.attr,NULL , };
drivers/md/dm-table.c +3 -3
@@ -940,7 +940,7 @@ bool dm_table_mq_request_based(struct dm_table *t)return dm_table_get_type (t) ==DM_TYPE_MQ_REQUEST_BASED ; } -static int dm_table_alloc_md_mempools (struct dm_table *t) +static int dm_table_alloc_md_mempools (struct dm_table *t,struct mapped_device *md) {unsigned type =dm_table_get_type (t);unsigned per_bio_data_size =0 ;
@@ -958,7 +958,7 @@ static int dm_table_alloc_md_mempools(struct dm_table *t) per_bio_data_size =max (per_bio_data_size, tgt->per_bio_data_size); } - t->mempools =dm_alloc_md_mempools (type, t->integrity_supported, per_bio_data_size); + t->mempools =dm_alloc_md_mempools (md, type, t->integrity_supported, per_bio_data_size);if (!t->mempools)return -ENOMEM ;
@@ -1128,7 +1128,7 @@ int dm_table_complete(struct dm_table *t)return r; } - r =dm_table_alloc_md_mempools (t); + r =dm_table_alloc_md_mempools (t, t->md);if (r)DMERR ("unable to allocate mempools" );
drivers/md/dm.c +40 -11
141 lines not on this page · raw message
drivers/md/dm.h +4 -1
@@ -211,6 +211,8 @@ int dm_kobject_uevent(struct mapped_device *md, enum kobject_action action,void dm_internal_suspend (struct mapped_device *md);void dm_internal_resume (struct mapped_device *md); +bool dm_use_blk_mq (struct mapped_device *md); +int dm_io_init (void );void dm_io_exit (void );
@@ -220,7 +222,8 @@ void dm_kcopyd_exit(void);/* * Mempool operations */ -struct dm_md_mempools *dm_alloc_md_mempools (unsigned type,unsigned integrity,unsigned per_bio_data_size); +struct dm_md_mempools *dm_alloc_md_mempools (struct mapped_device *md,unsigned type, +unsigned integrity,unsigned per_bio_data_size);void dm_free_md_mempools (struct dm_md_mempools *pools);/*
Signature · 1 line
1.9.5 (Apple Git-50.3)
Quoted 41 lines · Mike Snitzer
This patchset updates request-based DM (used exclussively by DM multipath) to support blk-mq I/O path or the old ->request_fn I/O path -- the default is still the old ->request_fn I/O path.
Like was done for SCSI, I've exposed a dm-mod.ko '
use_blk_mq' module parameter and aCONFIG_DM_MQ_DEFAULT. I've queued these patches in linux-next via linux-dm.git's 'for-next' branch -- my goal is for these changes to land upstream in 4.1.I've tested all 4 supported permutations of rq-based queue stacking:
1) blk-mq mpath stacked on blk-mq device(s)
2) blk-mq mpath stacked on ->request_fn device(s)
3) old ->request_fn mpath stacked on blk-mq device(s)
4) old ->request_fn mpath stacked on old ->request_fn device(s)There is definitely much more room for optimizing the blk-mq mpath queue (via
dm_mq_opssetup) to have more awareness about the characteristics of the underlying blk-mq device(s) -- Jens offered some suggestions that are worth pursuing. Goal being to leverage the ability to have multiple software submission queues that map to the underlying paths' HW queue. I expect Keith to be the most likely person to pursue these follow-on optimizations given his immediate access to NVMe devices, etc.blk-mq allows for the removal of 2 of the 3 mempools and associated allocations that were traditionally performed using old ->request_fn I/O path. The
md->bsbioset still remains for cloning a request's bios.I'll be pursuing the possibility of removing the bio cloning that happens when cloning a request (it was put in place for partial completion of a request's bios but the prospect of avoiding those bio clones warrants further review of how useful those partial completions are, see: http://www.redhat.com/archives/dm-devel/2014-June/msg00030.html)
Jens and Keith, special thanks for your help answering more blk-mq questions than I'd have hoped would be needed while at LSF/MM.
Keith Busch (1):
blk-mq: don't wait inblk_mq_queue_enter()if__GFP_WAITisn't set
Quoted 7 lines
Mike Snitzer (6):
blk-mq: fix use of incorrect goto label inblk_mq_init_queueerror path
blk-mq: addblk_mq_init_allocated_queueand exportblk_mq_register_disk
blk-mq: exportblk_mq_run_hw_queues
dm: add full blk-mq support to request-based DM
dm: optimizedm_mq_queue_rqto _not_ use kthread if using pure blk-mq
dm: add 'use_blk_mq' module param and expose in per-device ro sysfs attr
Applied patch 1 to current tree, marked for stable. 2-4 are applied for 4.1, in for-4.1/core.
Signature · 1 line
Jens Axboe