nvme-multipath: fix lockdep warning on shutdown
Subsystems (3)
PATCH 1/1 nvme-multipath: fix lockdep warning on shutdown
From: Hannes Reinecke <[email protected]>
During shutdown of multipath devices lockdep complained about a potential circular locking:
WARNING: possible circular locking dependency detected (udev-worker)/2792 is trying to acquire lock:ffff8881012a4348 ((wq_completion)kblockd){+.+.}-{0:0}, at: touch_wq_lockdep_map+0 x26/0x90
but task is already holding lock:ffff88811e4b7cc8 (&disk->open_mutex){+.+.}-{4:4}, at: bdev_release+0x61/0x1a0 which lock already depends on the new lock.
the existing dependency chain (in reverse order) is:
-> #2 (&disk->open_mutex){+.+.}-{4:4}:
__mutex_lock+0xa5/0xe00
nvme_partition_scan_work+0x31/0x60
process_scheduled_works+0x37c/0x6f0
-> #1 ((work_completion)(&head->partition_scan_work)){+.+.}-{0:0}:
process_scheduled_works+0x348/0x6f0
worker_thread+0x127/0x2a0
-> #0 ((wq_completion)kblockd){+.+.}-{0:0}:
__lock_acquire+0x11f9/0x1790
lock_acquire+0x245/0x2d0
touch_wq_lockdep_map+0x3b/0x90
__flush_work+0x240/0x4b0
nvme_mpath_remove_disk+0x2b/0x50
nvme_free_ns_head+0x19/0x90So the problem is that nvme_mpath_remove_disk() is called with the disk->open_mutex held, hence calling flush_work on partition_scan_work (which also will try to lock disk->open_mutex) will deadlock. Fix this by checking for NVME_NSHEAD_DISK_LIVE before trying to lock disk->open_mutex.
--- block/blk-ioprio.c | 6 ++++- drivers/nvme/host/multipath.c | 2 ++ drivers/nvme/target/core.c | 42 +++++++++++++++---------------- drivers/nvme/target/io-cmd-bdev.c | 9 +++++++ 4 files changed, 37 insertions(+), 22 deletions(-)
block/blk-ioprio.c +5 -1
@@ -141,9 +141,13 @@ static struct blkcg_policy ioprio_policy = {void blkcg_set_ioprio (struct bio *bio) { -struct ioprio_blkcg *blkcg =blkcg_to_ioprio_blkcg (bio->bi_blkg->blkcg); +struct ioprio_blkcg *blkcg;u16 prio; +if (WARN_ON (!bio->bi_blkg || ! bio->bi_blkg->blkcg)) +return ; + + blkcg =blkcg_to_ioprio_blkcg (bio->bi_blkg->blkcg);if (!blkcg || blkcg->prio_policy ==POLICY_NO_CHANGE )return ;
drivers/nvme/host/multipath.c +2 -0
@@ -593,6 +593,8 @@ static void nvme_partition_scan_work(struct work_struct *work)if (WARN_ON_ONCE (!test_and_clear_bit (GD_SUPPRESS_PART_SCAN , &head->disk->state)))return ; +if (!test_bit (NVME_NSHEAD_DISK_LIVE , &head->flags)) +return ;mutex_lock (&head->disk->open_mutex);bdev_disk_changed (head->disk, false);
drivers/nvme/target/core.c +21 -21
75 lines not on this page · raw message
drivers/nvme/target/io-cmd-bdev.c +9 -0
@@ -285,8 +285,16 @@ static void nvmet_bdev_execute_rw(struct nvmet_req *req)bio_init (bio, req->ns->bdev, req->inline_bvec,ARRAY_SIZE (req->inline_bvec), opf); }else { +if (!req->ns->enabled) { +nvmet_req_complete (req,NVME_SC_INTERNAL_PATH_ERROR ); +return ; + } bio =bio_alloc (req->ns->bdev,bio_max_segs (sg_cnt), opf,GFP_KERNEL ); +if (!bio) { +nvmet_req_complete (req,NVME_SC_INTERNAL ); +return ; + } } bio->bi_iter.bi_sector = sector; bio->bi_private = req;
@@ -313,6 +321,7 @@ static void nvmet_bdev_execute_rw(struct nvmet_req *req) bio =bio_alloc (req->ns->bdev,bio_max_segs (sg_cnt), opf,GFP_KERNEL ); +WARN_ON (!bio); bio->bi_iter.bi_sector = sector;bio_chain (bio, prev);
Signature · 1 line
2.35.3
Quoted 38 lines · On 1/24/25 08:14, [email protected] wrote:
From: Hannes Reinecke <[email protected]>
During shutdown of multipath devices lockdep complained about a potential circular locking:
WARNING: possible circular locking dependency detected (udev-worker)/2792 is trying to acquire lock:
ffff8881012a4348((wq_completion)kblockd){+.+.}-{0:0}, at: touch_wq_lockdep_map+0 x26/0x90but task is already holding lock:
ffff88811e4b7cc8(&disk->open_mutex){+.+.}-{4:4}, at:bdev_release+0x61/0x1a0which lock already depends on the new lock.the existing dependency chain (in reverse order) is:
-> #2 (&disk->open_mutex){+.+.}-{4:4}:__mutex_lock+0xa5/0xe00 nvme_partition_scan_work+0x31/0x60 process_scheduled_works+0x37c/0x6f0 -> #1 ((work_completion)(&head->partition_scan_work)){+.+.}-{0:0}: process_scheduled_works+0x348/0x6f0 worker_thread+0x127/0x2a0 -> #0 ((wq_completion)kblockd){+.+.}-{0:0}: __lock_acquire+0x11f9/0x1790 lock_acquire+0x245/0x2d0 touch_wq_lockdep_map+0x3b/0x90 __flush_work+0x240/0x4b0 nvme_mpath_remove_disk+0x2b/0x50 nvme_free_ns_head+0x19/0x90So the problem is that
nvme_mpath_remove_disk()is called with thedisk->open_mutexheld, hence callingflush_workonpartition_scan_work(which also will try to lockdisk->open_mutex) will deadlock. Fix this by checking forNVME_NSHEAD_DISK_LIVEbefore trying to lockdisk->open_mutex.Fixes 1f021341eef4 ("nvme-multipath: defer partition scanning")
Quoted 8 lines
Signed-off-by Hannes Reinecke <[email protected]>--- block/blk-ioprio.c | 6 ++++- drivers/nvme/host/multipath.c | 2 ++ drivers/nvme/target/core.c | 42 +++++++++++++++---------------- drivers/nvme/target/io-cmd-bdev.c | 9 +++++++ 4 files changed, 37 insertions(+), 22 deletions(-)
Nice analysis, but wrong patch.
Please ignore.
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich