blk-mq/nvme-loop: use nvme-loop's lock class for addressing lockdep false positive warning
Subsystems (3)
cover blk-mq/nvme-loop: use nvme-loop's lock class for addressing lockdep false positive warning
Hi,
Qian reported there is hang during booting when shared host tagset is introduced on megaraid sas. Sumit reported the whole SCSI probe takes about ~45min in his test.
Turns out it is caused by nr_hw_queues increased, especially commit b3c6a5997541("block: Fix a lockdep complaint triggered by request queue flushing") adds synchronize_rcu() for each hctx's release handler.
Address the original lockdep false positive warning by simpler way, then long scsi probe can be avoided with lockdep enabled.
Ming Lei (3):
blk-mq: add new API of blk_mq_hctx_set_fq_lock_class
nvme-loop: use blk_mq_hctx_set_fq_lock_class to set loop's lock class Revert "block: Fix a lockdep complaint triggered by request queue
flushing"
block/blk-flush.c | 30 +++++++++++++++++++++++++----- block/blk.h | 1 - drivers/nvme/target/loop.c | 10 ++++++++++ include/linux/blk-mq.h | 3 +++ 4 files changed, 38 insertions(+), 6 deletions(-)
Signature · 1 line
2.25.4
PATCH 1/3 blk-mq: add new API of blk_mq_hctx_set_fq_lock_class
flush_end_io() may be called recursively from some driver, such as nvme-loop, so lockdep may complain 'possible recursive locking'. Commit b3c6a5997541("block: Fix a lockdep complaint triggered by request queue flushing") tried to address this issue by assigning dynamically allocated per-flush-queue lock class. This solution adds synchronize_rcu() for each hctx's release handler, and causes horrible SCSI MQ probe delay(more than half an hour on megaraid sas).
Add new API of blk_mq_hctx_set_fq_lock_class() for these drivers, so we just need to use driver specific lock class for avoiding the lockdep warning of 'possible recursive locking'.
--- block/blk-flush.c | 25 +++++++++++++++++++++++++ include/linux/blk-mq.h | 3 +++ 2 files changed, 28 insertions(+)
block/blk-flush.c +25 -0
@@ -490,3 +490,28 @@ void blk_free_flush_queue(struct blk_flush_queue *fq)kfree (fq->flush_rq);kfree (fq); } + +/* +* Allow driver to set its own lock class to fq->mq_flush_lock for +* avoiding lockdep complaint. +* +* flush_end_io() may be called recursively from some driver, such as +* nvme-loop, so lockdep may complain 'possible recursive locking' because +* all 'struct blk_flush_queue' instance share same mq_flush_lock lock class +* key. We need to assign different lock class for these driver's +* fq->mq_flush_lock for avoiding the lockdep warning. +* +* Use dynamically allocated lock class key for each 'blk_flush_queue' +* instance is over-kill, and more worse it introduces horrible boot delay +* issue because synchronize_rcu() is implied in lockdep_unregister_key which +* is called for each hctx release. SCSI probing may synchronously create and +* destroy lots of MQ request_queues for non-existent devices, and some robot +* test kernel always enable lockdep option. It is observed that more than half +* an hour is taken during SCSI MQ probe with per-fq lock class. +*/ +void blk_mq_hctx_set_fq_lock_class (struct blk_mq_hw_ctx *hctx, +struct lock_class_key *key) +{ +lockdep_set_class (&hctx->fq->mq_flush_lock, key); +} +EXPORT_SYMBOL_GPL (blk_mq_hctx_set_fq_lock_class);
include/linux/blk-mq.h +3 -0
@@ -5,6 +5,7 @@#include <linux/blkdev.h> #include <linux/sbitmap.h> #include <linux/srcu.h> +#include <linux/lockdep.h> struct blk_mq_tags ;struct blk_flush_queue ;
@@ -594,5 +595,7 @@ static inline void blk_mq_cleanup_rq(struct request *rq) }blk_qc_t blk_mq_submit_bio (struct bio *bio); +void blk_mq_hctx_set_fq_lock_class (struct blk_mq_hw_ctx *hctx, +struct lock_class_key *key);#endif
Signature · 1 line
2.25.4
Christoph Hellwig
maintainerRe: [PATCH 1/3] blk-mq: add new API of blk_mq_hctx_set_fq_lock_class
Quoted 11 lines · Ming Lei
flush_end_io()may be called recursively from some driver, such as nvme-loop, so lockdep may complain 'possible recursive locking'. Commitb3c6a5997541("block: Fix a lockdep complaint triggered by request queue flushing")tried to address this issue by assigning dynamically allocated per-flush-queue lock class. This solution addssynchronize_rcu()for each hctx's release handler, and causes horrible SCSI MQ probe delay(more than half an hour on megaraid sas).Add new API of
blk_mq_hctx_set_fq_lock_class()for these drivers, so we just need to use driver specific lock class for avoiding the lockdep warning of 'possible recursive locking'.
I'd turn this into an inline function to avoid the (although very minimal) cost when LOCKDEP is not enabled.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 1/3] blk-mq: add new API of blk_mq_hctx_set_fq_lock_class
Quoted 9 lines · Christoph Hellwig
Quoted 8 lines · Ming Lei
flush_end_io()may be called recursively from some driver, such as nvme-loop, so lockdep may complain 'possible recursive locking'. Commitb3c6a5997541("block: Fix a lockdep complaint triggered by request queue flushing")tried to address this issue by assigning dynamically allocated per-flush-queue lock class. This solution addssynchronize_rcu()for each hctx's release handler, and causes horrible SCSI MQ probe delay(more than half an hour on megaraid sas).
Quoted 6 lines
Quoted 3 lines
Add new API of
blk_mq_hctx_set_fq_lock_class()for these drivers, so we just need to use driver specific lock class for avoiding the lockdep warning of 'possible recursive locking'.I'd turn this into an inline function to avoid the (although very minimal) cost when LOCKDEP is not enabled.
blk_mq_hctx_set_fq_lock_class() is just one-shot thing, do you really care the cost?
thanks,
Ming
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 1/3] blk-mq: add new API of blk_mq_hctx_set_fq_lock_class
Quoted 14 lines · Ming Lei
Quoted 13 lines · Christoph Hellwig
Quoted 11 lines · Ming Lei
flush_end_io()may be called recursively from some driver, such as nvme-loop, so lockdep may complain 'possible recursive locking'. Commitb3c6a5997541("block: Fix a lockdep complaint triggered by request queue flushing")tried to address this issue by assigning dynamically allocated per-flush-queue lock class. This solution addssynchronize_rcu()for each hctx's release handler, and causes horrible SCSI MQ probe delay(more than half an hour on megaraid sas).Add new API of
blk_mq_hctx_set_fq_lock_class()for these drivers, so we just need to use driver specific lock class for avoiding the lockdep warning of 'possible recursive locking'.
Quoted 5 lines
Quoted 2 lines
I'd turn this into an inline function to avoid the (although very minimal) cost when LOCKDEP is not enabled.
blk_mq_hctx_set_fq_lock_class()is just one-shot thing, do you really care the cost?
Forget to mention, 'blk_flush_queue' is one private structure inside block layer, so we can't define as inline.
thanks,
Ming
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Kashyap Desai
Tested-byRE: [PATCH 1/3] blk-mq: add new API of blk_mq_hctx_set_fq_lock_class
Quoted 10 lines · Ming Lei
Quoted 9 lines · Christoph Hellwig
Quoted 8 lines · Ming Lei
Quoted 7 lines
flush_end_io()may be called recursively from some driver, such as nvme-loop, so lockdep may complain 'possible recursive locking'. Commitb3c6a5997541("block: Fix a lockdep complaint triggered by request queue flushing")tried to address this issue by assigning dynamically allocated per-flush-queue lock class. This solution addssynchronize_rcu()for each hctx's release handler, and causes horrible SCSI MQ probe delay(more than half an hour on megaraid
sas).
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Add new API of
blk_mq_hctx_set_fq_lock_class()for these drivers, so we just need to use driver specific lock class for avoiding the lockdep warning of 'possible recursive locking'.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
I'd turn this into an inline function to avoid the (although very minimal) cost when LOCKDEP is not enabled.
blk_mq_hctx_set_fq_lock_class()is just one-shot thing, do you really care the cost?Forget to mention, '
blk_flush_queue' is one private structure inside
block
Quoted 1 line
layer, so we can't define as inline.
Original issue reported by Qian is fixed by this patch series. Tested on MegaRaid h/w.
Quoted 3 lines
thanks,
Ming
PATCH 2/3 nvme-loop: use blk_mq_hctx_set_fq_lock_class to set loop's lock class
In reply to Ming LeiSet nvme-loop's lock class via blk_mq_hctx_set_fq_lock_class for avoiding lockdep possible recursive locking, then we can remove the dynamically allocated lock class for each flush queue, finally we can avoid horrible SCSI probe delay.
This way may not address situation in which one nvme-loop is backed on another nvme-loop. However, in reality, people seldom uses this way for test. Even though someone played in this way, it is just one recursive locking false positive, no real deadlock issue.
--- drivers/nvme/target/loop.c | 10 ++++++++++ 1 file changed, 10 insertions(+)
drivers/nvme/target/loop.c +10 -0
@@ -211,6 +211,8 @@ static int nvme_loop_init_request(struct blk_mq_tag_set *set, (set == &ctrl->tag_set) ? hctx_idx + 1 : 0); } +static struct lock_class_key loop_hctx_fq_lock_key; +static int nvme_loop_init_hctx (struct blk_mq_hw_ctx *hctx,void *data,unsigned int hctx_idx) {
@@ -219,6 +221,14 @@ static int nvme_loop_init_hctx(struct blk_mq_hw_ctx *hctx, void *data,BUG_ON (hctx_idx >= ctrl->ctrl.queue_count); +/* +* flush_end_io() can be called recursively for us, so use our own +* lock class key for avoiding lockdep possible recursive locking, +* then we can remove the dynamically allocated lock class for each +* flush queue, that way may cause horrible boot delay. +*/ +blk_mq_hctx_set_fq_lock_class (hctx, &loop_hctx_fq_lock_key); + hctx->driver_data = queue;return 0 ; }
Signature · 1 line
2.25.4
Re: [PATCH 2/3] nvme-loop: use blk_mq_hctx_set_fq_lock_class to set loop's lock class
Looks good,
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 3/3 Revert "block: Fix a lockdep complaint triggered by request queue flushing"
In reply to Ming LeiThis reverts commit b3c6a59975415bde29cfd76ff1ab008edbf614a9.
Now we can avoid nvme-loop lockdep warning of 'lockdep possible recursive locking' by nvme-loop's lock class, no need to apply dynamically allocated lock class key, so revert commit b3c6a5997541("block: Fix a lockdep complaint triggered by request queue flushing").
This way fixes horrible SCSI probe delay issue on megaraid_sas, and it is reported the whole probe may take more than half an hour.
--- block/blk-flush.c | 5 ----- block/blk.h | 1 - 2 files changed, 6 deletions(-)
block/blk-flush.c +0 -5
@@ -69,7 +69,6 @@#include <linux/blkdev.h> #include <linux/gfp.h> #include <linux/blk-mq.h> -#include <linux/lockdep.h> #include "blk.h" #include "blk-mq.h"
@@ -469,9 +468,6 @@ struct blk_flush_queue *blk_alloc_flush_queue(int node, int cmd_size,INIT_LIST_HEAD (&fq->flush_queue[1 ]);INIT_LIST_HEAD (&fq->flush_data_in_flight); -lockdep_register_key (&fq->key); -lockdep_set_class (&fq->mq_flush_lock, &fq->key); -return fq; fail_rq:
@@ -486,7 +482,6 @@ void blk_free_flush_queue(struct blk_flush_queue *fq)if (!fq)return ; -lockdep_unregister_key (&fq->key);kfree (fq->flush_rq);kfree (fq); }
block/blk.h +0 -1
@@ -25,7 +25,6 @@ struct blk_flush_queue {struct list_head flush_data_in_flight;struct request *flush_rq; -struct lock_class_key key;spinlock_t mq_flush_lock; };
Signature · 1 line
2.25.4
Re: [PATCH 3/3] Revert "block: Fix a lockdep complaint triggered by request queue flushing"
Quoted 9 lines · Ming Lei
This reverts commit
b3c6a59975415bde29cfd76ff1ab008edbf614a9.Now we can avoid nvme-loop lockdep warning of 'lockdep possible recursive locking' by nvme-loop's lock class, no need to apply dynamically allocated lock class key, so revert commit
b3c6a5997541("block: Fix a lockdep complaint triggered by request queue flushing").This way fixes horrible SCSI probe delay issue on
megaraid_sas, and it is reported the whole probe may take more than half an hour.
The code touched by this patch is compiled out with locked disabled so it is not clear to me how this patch could affect a probe delay for megaraid_sas? Has the megaraid_sas probe issue perhaps not been root caused correctly?
Thanks,
Bart.
Re: [PATCH 3/3] Revert "block: Fix a lockdep complaint triggered by request queue flushing"
Quoted 8 lines · Bart Van Assche
Quoted 7 lines · Ming Lei
This reverts commit
b3c6a59975415bde29cfd76ff1ab008edbf614a9.Now we can avoid nvme-loop lockdep warning of 'lockdep possible recursive locking' by nvme-loop's lock class, no need to apply dynamically allocated lock class key, so revert commit
b3c6a5997541("block: Fix a lockdep complaint triggered by request queue flushing").
Quoted 7 lines
Quoted 2 lines
This way fixes horrible SCSI probe delay issue on
megaraid_sas, and it is reported the whole probe may take more than half an hour.The code touched by this patch is compiled out with locked disabled so it is not clear to me how this patch could affect a probe delay for
megaraid_sas? Has themegaraid_sasprobe issue perhaps not been root caused correctly?
(replying to my own email)
I found the answer in the descriptions of the previous patches of this series. I think this patch would benefit from a more complete patch description.
Thanks,
Bart.
Re: [PATCH 3/3] Revert "block: Fix a lockdep complaint triggered by request queue flushing"
Quoted 17 lines · Bart Van Assche
Quoted 8 lines · Bart Van Assche
Quoted 7 lines · Ming Lei
This reverts commit
b3c6a59975415bde29cfd76ff1ab008edbf614a9.Now we can avoid nvme-loop lockdep warning of 'lockdep possible recursive locking' by nvme-loop's lock class, no need to apply dynamically allocated lock class key, so revert commit
b3c6a5997541("block: Fix a lockdep complaint triggered by request queue flushing").Quoted 7 lines
Quoted 2 lines
This way fixes horrible SCSI probe delay issue on
megaraid_sas, and it is reported the whole probe may take more than half an hour.The code touched by this patch is compiled out with locked disabled so it is not clear to me how this patch could affect a probe delay for
megaraid_sas? Has themegaraid_sasprobe issue perhaps not been root caused correctly?
Quoted 5 lines
(replying to my own email)
I found the answer in the descriptions of the previous patches of this series. I think this patch would benefit from a more complete patch description.
Hello Bart,
OK, I can add more words to this commit log.
Thanks,
Ming
PATCH v2 3/3 Revert "block: Fix a lockdep complaint triggered by request queue flushing"
In reply to Ming LeiFrom a519f421957a1205918e9bcc15087d15234e4e9f Mon Sep 17 00:00:00 2001
From: Ming Lei <[email protected]>
Date: Thu, 12 Nov 2020 09:56:02 +0800
Subject: [PATCH V2 3/3] Revert "block: Fix a lockdep complaint triggered by
request queue flushing"
This reverts commit b3c6a59975415bde29cfd76ff1ab008edbf614a9.
Now we can avoid nvme-loop lockdep warning of 'lockdep possible recursive locking' by nvme-loop's lock class, no need to apply dynamically allocated lock class key, so revert commit b3c6a5997541("block: Fix a lockdep complaint triggered by request queue flushing").
This way fixes horrible SCSI probe delay issue on megaraid_sas(host_tagset is 1), and it is reported the whole probe may take more than half an hour. The reason is that synchronize_rcu() is implied in lockdep_unregister_key() which is called from each hctx's release handler, and some SCSI hosts can support too many hw queues, meantime generic SCSI probe may synchronously create and destroy lots of MQ request_queues for non-existent devices.
Another benefit is that lockdep doesn't maintain so many runtime lock class for fq->mq_flush_lock which is per-hctx, then lock validation can be improved much.
---
V2:
- add more commit log
block/blk-flush.c | 5 ----- block/blk.h | 1 - 2 files changed, 6 deletions(-)
block/blk-flush.c +0 -5
@@ -69,7 +69,6 @@#include <linux/blkdev.h> #include <linux/gfp.h> #include <linux/blk-mq.h> -#include <linux/lockdep.h> #include "blk.h" #include "blk-mq.h"
@@ -469,9 +468,6 @@ struct blk_flush_queue *blk_alloc_flush_queue(int node, int cmd_size,INIT_LIST_HEAD (&fq->flush_queue[1 ]);INIT_LIST_HEAD (&fq->flush_data_in_flight); -lockdep_register_key (&fq->key); -lockdep_set_class (&fq->mq_flush_lock, &fq->key); -return fq; fail_rq:
@@ -486,7 +482,6 @@ void blk_free_flush_queue(struct blk_flush_queue *fq)if (!fq)return ; -lockdep_unregister_key (&fq->key);kfree (fq->flush_rq);kfree (fq); }
block/blk.h +0 -1
@@ -25,7 +25,6 @@ struct blk_flush_queue {struct list_head flush_data_in_flight;struct request *flush_rq; -struct lock_class_key key;spinlock_t mq_flush_lock; };
Signature · 1 line
2.25.4
Re: [PATCH V2 3/3] Revert "block: Fix a lockdep complaint triggered by request queue flushing"
Quoted 8 lines · Ming Lei
Quoted 1 line
From
a519f421957a1205918e9bcc15087d15234e4e9fMon Sep 17 00:00:00 2001From: Ming Lei <[email protected]>
Date: Thu, 12 Nov 2020 09:56:02 +0800
Subject: [PATCH V2 3/3] Revert "block: Fix a lockdep complaint triggered by
request queue flushing"This reverts commit
b3c6a59975415bde29cfd76ff1ab008edbf614a9.
Quoted 7 lines
Now we can avoid nvme-loop lockdep warning of 'lockdep possible recursive locking' by nvme-loop's lock class, no need to apply dynamically allocated lock class key, so revert commit
b3c6a5997541("block: Fix a lockdep complaint triggered by request queue flushing").This way fixes horrible SCSI probe delay issue on
megaraid_sas(host_tagset is 1), and it is reported the whole probe may take more than half an hour. The reason
There are a bunch of overly long lines in the commit log.
Otherwise this looks good:
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 25 lines · Ming Lei
Hi,
Qian reported there is hang during booting when shared host tagset is introduced on megaraid sas. Sumit reported the whole SCSI probe takes about ~45min in his test.
Turns out it is caused by
nr_hw_queuesincreased, especially commitb3c6a5997541("block: Fix a lockdep complaint triggered by request queue flushing")addssynchronize_rcu()for each hctx's release handler.Address the original lockdep false positive warning by simpler way, then long scsi probe can be avoided with lockdep enabled.
Ming Lei (3):
blk-mq: add new API ofblk_mq_hctx_set_fq_lock_class
nvme-loop: useblk_mq_hctx_set_fq_lock_classto set loop's lock class Revert "block: Fix a lockdep complaint triggered by request queue
flushing"block/blk-flush.c | 30 +++++++++++++++++++++++++----- block/blk.h | 1 - drivers/nvme/target/loop.c | 10 ++++++++++ include/linux/blk-mq.h | 3 +++ 4 files changed, 38 insertions(+), 6 deletions(-)
Quoted 6 lines
Cc Qian Cai <[email protected]>Cc Sumit Saxena <[email protected]>Cc John Garry <[email protected]>Cc Kashyap Desai <[email protected]>Cc Bart Van Assche <[email protected]>Cc Hannes Reinecke <[email protected]>
Hello Jens,
Ping...
thanks,
Ming
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 25 lines · Ming Lei
Hi,
Qian reported there is hang during booting when shared host tagset is introduced on megaraid sas. Sumit reported the whole SCSI probe takes about ~45min in his test.
Turns out it is caused by
nr_hw_queuesincreased, especially commitb3c6a5997541("block: Fix a lockdep complaint triggered by request queue flushing")addssynchronize_rcu()for each hctx's release handler.Address the original lockdep false positive warning by simpler way, then long scsi probe can be avoided with lockdep enabled.
Ming Lei (3):
blk-mq: add new API ofblk_mq_hctx_set_fq_lock_class
nvme-loop: useblk_mq_hctx_set_fq_lock_classto set loop's lock class Revert "block: Fix a lockdep complaint triggered by request queue
flushing"block/blk-flush.c | 30 +++++++++++++++++++++++++----- block/blk.h | 1 - drivers/nvme/target/loop.c | 10 ++++++++++ include/linux/blk-mq.h | 3 +++ 4 files changed, 38 insertions(+), 6 deletions(-)
Quoted 6 lines
Cc Qian Cai <[email protected]>Cc Sumit Saxena <[email protected]>Cc John Garry <[email protected]>Cc Kashyap Desai <[email protected]>Cc Bart Van Assche <[email protected]>Cc Hannes Reinecke <[email protected]>
Hello Jens,
Any chance to take a look? And this issue has been reported several times in RH internal test.
Thanks,
Ming
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 2 lines · Ming Lei
Any chance to take a look? And this issue has been reported several times in RH internal test.
I suppose that you will need to rebase as it does not apply cleanly on today's linux-next.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 7 lines · Qian Cai
Quoted 2 lines · Ming Lei
Any chance to take a look? And this issue has been reported several times in RH internal test.
I suppose that you will need to rebase as it does not apply cleanly on today's linux-next.
I just apply the three patches against for-5.11/block, and they can be done cleanly.
Thanks,
Ming
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Qian Cai
Tested-byQuoted 5 lines · Ming Lei
Quoted 4 lines · Qian Cai
Quoted 2 lines · Ming Lei
Any chance to take a look? And this issue has been reported several times in RH internal test.
Quoted 7 lines
Quoted 4 lines
I suppose that you will need to rebase as it does not apply cleanly on today's
linux-next.I just apply the three patches against for-5.11/block, and they can be done cleanly.
Ah, sorry. It was a bit whitespace damages on my end. This did really fix the megasas_raid boot hang here. Since 103fbf8e4020 ("scsi: megaraid_sas: Added support for shared host tagset for cpuhotplug") is now in the mainline, it probably makes sense to merge this patchset for v5.10 as well to avoid breakage. Anyway,
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme