Skip to contentSkip to conversation

nvme-multipath: fix lockdep warning on shutdown

PATCH In review [email protected] 2 participants 2 messages Jan 24, 2025

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 nvme-multipath: fix lockdep warning on shutdown

From: Hannes Reinecke <[email protected]>

During shutdown of multipath devices lockdep complained about a potential circular locking:

WARNING: possible circular locking dependency detected (udev-worker)/2792 is trying to acquire lock:
ffff8881012a4348 ((wq_completion)kblockd){+.+.}-{0:0}, at: touch_wq_lockdep_map+0 x26/0x90

but task is already holding lock:
ffff88811e4b7cc8 (&disk->open_mutex){+.+.}-{4:4}, at: bdev_release+0x61/0x1a0 which lock already depends on the new lock.

the existing dependency chain (in reverse order) is:
-> #2 (&disk->open_mutex){+.+.}-{4:4}:

        __mutex_lock+0xa5/0xe00
       nvme_partition_scan_work+0x31/0x60
        process_scheduled_works+0x37c/0x6f0
-> #1 ((work_completion)(&head->partition_scan_work)){+.+.}-{0:0}:
        process_scheduled_works+0x348/0x6f0
        worker_thread+0x127/0x2a0
-> #0 ((wq_completion)kblockd){+.+.}-{0:0}:
        __lock_acquire+0x11f9/0x1790
        lock_acquire+0x245/0x2d0
        touch_wq_lockdep_map+0x3b/0x90
        __flush_work+0x240/0x4b0
        nvme_mpath_remove_disk+0x2b/0x50
        nvme_free_ns_head+0x19/0x90

So the problem is that nvme_mpath_remove_disk() is called with the disk->open_mutex held, hence calling flush_work on partition_scan_work (which also will try to lock disk->open_mutex) will deadlock. Fix this by checking for NVME_NSHEAD_DISK_LIVE before trying to lock disk->open_mutex.

Fixes 1f021341eef4 ("nvme-multipath: defer partition scanning")
---
 block/blk-ioprio.c                |  6 ++++-
 drivers/nvme/host/multipath.c     |  2 ++
 drivers/nvme/target/core.c        | 42 +++++++++++++++----------------
 drivers/nvme/target/io-cmd-bdev.c |  9 +++++++
 4 files changed, 37 insertions(+), 22 deletions(-)
block/blk-ioprio.c +5 -1
index 8fff7ccc0ac7..9f1b2069a3c9 100644
@@ -141,9 +141,13 @@ static struct blkcg_policy ioprio_policy = {
 
 void blkcg_set_ioprio(struct bio *bio)
 {
-	struct ioprio_blkcg *blkcg = blkcg_to_ioprio_blkcg(bio->bi_blkg->blkcg);
+	struct ioprio_blkcg *blkcg;
 	u16 prio;
 
+	if (WARN_ON(!bio->bi_blkg || ! bio->bi_blkg->blkcg))
+		return;
+
+	blkcg = blkcg_to_ioprio_blkcg(bio->bi_blkg->blkcg);
 	if (!blkcg || blkcg->prio_policy == POLICY_NO_CHANGE)
 		return;
 
drivers/nvme/host/multipath.c +2 -0
index a85d190942bd..af763ac4d657 100644
@@ -593,6 +593,8 @@ static void nvme_partition_scan_work(struct work_struct *work)
 	if (WARN_ON_ONCE(!test_and_clear_bit(GD_SUPPRESS_PART_SCAN,
 					     &head->disk->state)))
 		return;
+	if (!test_bit(NVME_NSHEAD_DISK_LIVE, &head->flags))
+		return;
 
 	mutex_lock(&head->disk->open_mutex);
 	bdev_disk_changed(head->disk, false);
drivers/nvme/target/core.c +21 -21

75 lines not on this page · raw message

drivers/nvme/target/io-cmd-bdev.c +9 -0
index 2b09b2c69857..4533e9997c7e 100644
@@ -285,8 +285,16 @@ static void nvmet_bdev_execute_rw(struct nvmet_req *req)
 		bio_init(bio, req->ns->bdev, req->inline_bvec,
 			 ARRAY_SIZE(req->inline_bvec), opf);
 	} else {
+		if (!req->ns->enabled) {
+			nvmet_req_complete(req, NVME_SC_INTERNAL_PATH_ERROR);
+			return;
+		}
 		bio = bio_alloc(req->ns->bdev, bio_max_segs(sg_cnt), opf,
 				GFP_KERNEL);
+		if (!bio) {
+			nvmet_req_complete(req, NVME_SC_INTERNAL);
+			return;
+		}
 	}
 	bio->bi_iter.bi_sector = sector;
 	bio->bi_private = req;
@@ -313,6 +321,7 @@ static void nvmet_bdev_execute_rw(struct nvmet_req *req)
 
 			bio = bio_alloc(req->ns->bdev, bio_max_segs(sg_cnt),
 					opf, GFP_KERNEL);
+			WARN_ON(!bio);
 			bio->bi_iter.bi_sector = sector;
 
 			bio_chain(bio, prev);
Signature · 1 line
2.35.3
#1 lore raw
Quoted 38 lines · On 1/24/25 08:14, [email protected] wrote:

From: Hannes Reinecke <[email protected]>

During shutdown of multipath devices lockdep complained about a potential circular locking:

WARNING: possible circular locking dependency detected (udev-worker)/2792 is trying to acquire lock:
ffff8881012a4348 ((wq_completion)kblockd){+.+.}-{0:0}, at: touch_wq_lockdep_map+0 x26/0x90

but task is already holding lock:
ffff88811e4b7cc8 (&disk->open_mutex){+.+.}-{4:4}, at: bdev_release+0x61/0x1a0 which lock already depends on the new lock.

the existing dependency chain (in reverse order) is:
-> #2 (&disk->open_mutex){+.+.}-{4:4}:

         __mutex_lock+0xa5/0xe00
        nvme_partition_scan_work+0x31/0x60
         process_scheduled_works+0x37c/0x6f0
-> #1 ((work_completion)(&head->partition_scan_work)){+.+.}-{0:0}:
         process_scheduled_works+0x348/0x6f0
         worker_thread+0x127/0x2a0
-> #0 ((wq_completion)kblockd){+.+.}-{0:0}:
         __lock_acquire+0x11f9/0x1790
         lock_acquire+0x245/0x2d0
         touch_wq_lockdep_map+0x3b/0x90
         __flush_work+0x240/0x4b0
         nvme_mpath_remove_disk+0x2b/0x50
         nvme_free_ns_head+0x19/0x90

So the problem is that nvme_mpath_remove_disk() is called with the disk->open_mutex held, hence calling flush_work on partition_scan_work (which also will try to lock disk->open_mutex) will deadlock. Fix this by checking for NVME_NSHEAD_DISK_LIVE before trying to lock disk->open_mutex.

Fixes 1f021341eef4 ("nvme-multipath: defer partition scanning")
Quoted 8 lines
---
  block/blk-ioprio.c                |  6 ++++-
  drivers/nvme/host/multipath.c     |  2 ++
  drivers/nvme/target/core.c        | 42 +++++++++++++++----------------
  drivers/nvme/target/io-cmd-bdev.c |  9 +++++++
  4 files changed, 37 insertions(+), 22 deletions(-)

Nice analysis, but wrong patch.

Please ignore.

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich