Skip to contentSkip to conversation

nvme-core: Fix extra device_put() call on error path

PATCH In review Logan Gunthorpe 2 participants 11 messages Jul 18, 2019 – Jul 25, 2019 NVMe

lore
Share & tools
lore
#0 lore raw

PATCH 1/2 nvme-core: Fix extra device_put() call on error path

In the error path for nvme_init_subsystem(), nvme_put_subsystem() will call device_put(), but it will get called again after the mutex_unlock().

The device_put() only needs to be called if device_add() fails.

This bug caused a KASAN use-after-free error when adding and removing subsytems in a loop:

  BUG: KASAN: use-after-free in device_del+0x8d9/0x9a0
  Read of size 8 at addr ffff8883cdaf7120 by task multipathd/329

  CPU: 0 PID: 329 Comm: multipathd Not tainted 5.2.0-rc6-vmlocalyes-00019-g70a2b39005fd-dirty #314
  Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1 04/01/2014
  Call Trace:
   dump_stack+0x7b/0xb5
   print_address_description+0x6f/0x280
   ? device_del+0x8d9/0x9a0
   __kasan_report+0x148/0x199
   ? device_del+0x8d9/0x9a0
   ? class_release+0x100/0x130
   ? device_del+0x8d9/0x9a0
   kasan_report+0x12/0x20
   __asan_report_load8_noabort+0x14/0x20
   device_del+0x8d9/0x9a0
   ? device_platform_notify+0x70/0x70
   nvme_destroy_subsystem+0xf9/0x150
   nvme_free_ctrl+0x280/0x3a0
   device_release+0x72/0x1d0
   kobject_put+0x144/0x410
   put_device+0x13/0x20
   nvme_free_ns+0xc4/0x100
   nvme_release+0xb3/0xe0
   __blkdev_put+0x549/0x6e0
   ? kasan_check_write+0x14/0x20
   ? bd_set_size+0xb0/0xb0
   ? kasan_check_write+0x14/0x20
   ? mutex_lock+0x8f/0xe0
   ? __mutex_lock_slowpath+0x20/0x20
   ? locks_remove_file+0x239/0x370
   blkdev_put+0x72/0x2c0
   blkdev_close+0x8d/0xd0
   __fput+0x256/0x770
   ? _raw_read_lock_irq+0x40/0x40
   ____fput+0xe/0x10
   task_work_run+0x10c/0x180
   ? filp_close+0xf7/0x140
   exit_to_usermode_loop+0x151/0x170
   do_syscall_64+0x240/0x2e0
   ? prepare_exit_to_usermode+0xd5/0x190
   entry_SYSCALL_64_after_hwframe+0x44/0xa9
  RIP: 0033:0x7f5a79af05d7
  Code: 00 00 0f 05 48 3d 00 f0 ff ff 77 3f c3 66 0f 1f 44 00 00 53 89 fb 48 83 ec 10 e8 c4 fb ff ff 89 df 89 c2 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 2b 89 d7 89 44 24 0c e8 06 fc ff ff 8b 44 24
  RSP: 002b:00007f5a7799c810 EFLAGS: 00000293 ORIG_RAX: 0000000000000003
  RAX: 0000000000000000 RBX: 0000000000000008 RCX: 00007f5a79af05d7
  RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000008
  RBP: 00007f5a58000f98 R08: 0000000000000002 R09: 00007f5a7935ee80
  R10: 0000000000000000 R11: 0000000000000293 R12: 000055e432447240
  R13: 0000000000000000 R14: 0000000000000001 R15: 000055e4324a9cf0

  Allocated by task 1236:
   save_stack+0x21/0x80
   __kasan_kmalloc.constprop.6+0xab/0xe0
   kasan_kmalloc+0x9/0x10
   kmem_cache_alloc_trace+0x102/0x210
   nvme_init_identify+0x13c3/0x3820
   nvme_loop_configure_admin_queue+0x4fa/0x5e0
   nvme_loop_create_ctrl+0x469/0xf40
   nvmf_dev_write+0x19a3/0x21ab
   __vfs_write+0x66/0x120
   vfs_write+0x154/0x490
   ksys_write+0x104/0x240
   __x64_sys_write+0x73/0xb0
   do_syscall_64+0xa5/0x2e0
   entry_SYSCALL_64_after_hwframe+0x44/0xa9

  Freed by task 329:
   save_stack+0x21/0x80
   __kasan_slab_free+0x129/0x190
   kasan_slab_free+0xe/0x10
   kfree+0xa7/0x200
   nvme_release_subsystem+0x49/0x60
   device_release+0x72/0x1d0
   kobject_put+0x144/0x410
   put_device+0x13/0x20
   klist_class_dev_put+0x31/0x40
   klist_put+0x8f/0xf0
   klist_del+0xe/0x10
   device_del+0x3a7/0x9a0
   nvme_destroy_subsystem+0xf9/0x150
   nvme_free_ctrl+0x280/0x3a0
   device_release+0x72/0x1d0
   kobject_put+0x144/0x410
   put_device+0x13/0x20
   nvme_free_ns+0xc4/0x100
   nvme_release+0xb3/0xe0
   __blkdev_put+0x549/0x6e0
   blkdev_put+0x72/0x2c0
   blkdev_close+0x8d/0xd0
   __fput+0x256/0x770
   ____fput+0xe/0x10
   task_work_run+0x10c/0x180
   exit_to_usermode_loop+0x151/0x170
   do_syscall_64+0x240/0x2e0
   entry_SYSCALL_64_after_hwframe+0x44/0xa9
Fixes 32fd90c40768 ("nvme: change locking for the per-subsystem controller list")
Signed-off-by Logan Gunthorpe <logang at deltatee.com>
---
 drivers/nvme/host/core.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/nvme/host/core.c +1 -1
index cc09b81fc7f4..3ca33a2714e5 100644
@@ -2489,6 +2489,7 @@ static int nvme_init_subsystem(struct nvme_ctrl *ctrl, struct nvme_id_ctrl *id)
 		if (ret) {
 			dev_err(ctrl->device,
 				"failed to register subsystem device.\n");
+			put_device(&subsys->dev);
 			goto out_unlock;
 		}
 		ida_init(&subsys->ns_ida);
@@ -2511,7 +2512,6 @@ static int nvme_init_subsystem(struct nvme_ctrl *ctrl, struct nvme_id_ctrl *id)
 	nvme_put_subsystem(subsys);
 out_unlock:
 	mutex_unlock(&nvme_subsystems_lock);
-	put_device(&subsys->dev);
 	return ret;
 }
 
Signature · 1 line
2.20.1
#1 lore raw

PATCH 2/2 nvme-core: Fix deadlock when deleting the ctrl while scanning

With multipath enabled, nvme_scan_work() can read from the device (through nvme_mpath_add_disk()). However, with fabrics, once ctrl->state is set to NVME_CTRL_DELETING, the reads will hang (see nvmf_check_ready()).

After setting the state to deleting, nvme_remove_namespaces() will hang waiting for scan_work to flush and these tasks will hang.

To fix this, ensure we take scan_lock before changing the ctrl-state. Also, ensure the state is checked while the lock is held in nvme_scan_lock_work().

  INFO: task kworker/u4:3:166 blocked for more than 120 seconds.
        Not tainted 5.2.0-rc6-vmlocalyes-00005-g808c8c2dc0cf #316
  "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  kworker/u4:3    D    0   166      2 0x80004000
  Workqueue: nvme-wq nvme_scan_work
  Call Trace:
   __schedule+0x851/0x1400
   schedule+0x99/0x210
   io_schedule+0x21/0x70
   do_read_cache_page+0xa57/0x1330
   read_cache_page+0x4a/0x70
   read_dev_sector+0xbf/0x380
   amiga_partition+0xc4/0x1230
   check_partition+0x30f/0x630
   rescan_partitions+0x19a/0x980
   __blkdev_get+0x85a/0x12f0
   blkdev_get+0x2a5/0x790
   __device_add_disk+0xe25/0x1250
   device_add_disk+0x13/0x20
   nvme_mpath_set_live+0x172/0x2b0
   nvme_update_ns_ana_state+0x130/0x180
   nvme_set_ns_ana_state+0x9a/0xb0
   nvme_parse_ana_log+0x1c3/0x4a0
   nvme_mpath_add_disk+0x157/0x290
   nvme_validate_ns+0x1017/0x1bd0
   nvme_scan_work+0x44d/0x6a0
   process_one_work+0x7d7/0x1240
   worker_thread+0x8e/0xff0
   kthread+0x2c3/0x3b0
   ret_from_fork+0x35/0x40

   INFO: task kworker/u4:1:1034 blocked for more than 120 seconds.
        Not tainted 5.2.0-rc6-vmlocalyes-00005-g808c8c2dc0cf #316
  "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  kworker/u4:1    D    0  1034      2 0x80004000
  Workqueue: nvme-delete-wq nvme_delete_ctrl_work
  Call Trace:
   __schedule+0x851/0x1400
   schedule+0x99/0x210
   schedule_timeout+0x390/0x830
   wait_for_completion+0x1a7/0x310
   __flush_work+0x241/0x5d0
   flush_work+0x10/0x20
   nvme_remove_namespaces+0x85/0x3d0
   nvme_do_delete_ctrl+0xb4/0x1e0
   nvme_delete_ctrl_work+0x15/0x20
   process_one_work+0x7d7/0x1240
   worker_thread+0x8e/0xff0
   kthread+0x2c3/0x3b0
   ret_from_fork+0x35/0x40
Signed-off-by Logan Gunthorpe <logang at deltatee.com>
---
 drivers/nvme/host/core.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
drivers/nvme/host/core.c +7 -1
index 3ca33a2714e5..0a7b46066fe3 100644
@@ -301,6 +301,9 @@ bool nvme_change_ctrl_state(struct nvme_ctrl *ctrl,
 	unsigned long flags;
 	bool changed = false;
 
+	/* Ensure a scan isn't in progress when we change the state */
+	mutex_lock(&ctrl->scan_lock);
+
 	spin_lock_irqsave(&ctrl->lock, flags);
 
 	old_state = ctrl->state;
@@ -375,6 +378,8 @@ bool nvme_change_ctrl_state(struct nvme_ctrl *ctrl,
 		ctrl->state = new_state;
 
 	spin_unlock_irqrestore(&ctrl->lock, flags);
+	mutex_unlock(&ctrl->scan_lock);
+
 	if (changed && ctrl->state == NVME_CTRL_LIVE)
 		nvme_kick_requeue_lists(ctrl);
 	return changed;
@@ -3534,6 +3539,8 @@ static void nvme_scan_work(struct work_struct *work)
 	struct nvme_id_ctrl *id;
 	unsigned nn;
 
+	mutex_lock(&ctrl->scan_lock);
+
 	if (ctrl->state != NVME_CTRL_LIVE)
 		return;
 
@@ -3547,7 +3554,6 @@ static void nvme_scan_work(struct work_struct *work)
 	if (nvme_identify_ctrl(ctrl, &id))
 		return;
 
-	mutex_lock(&ctrl->scan_lock);
 	nn = le32_to_cpu(id->nn);
 	if (ctrl->vs >= NVME_VS(1, 1, 0) &&
 	    !(ctrl->quirks & NVME_QUIRK_IDENTIFY_CNS)) {
Signature · 1 line
2.20.1
maintainer
#2 lore raw

Re: [PATCH 2/2] nvme-core: Fix deadlock when deleting the ctrl while scanning

Quoted 6 lines

With multipath enabled, nvme_scan_work() can read from the device (through nvme_mpath_add_disk()). However, with fabrics, once ctrl->state is set to NVME_CTRL_DELETING, the reads will hang (see nvmf_check_ready()).

Quoted 6 lines

After setting the state to deleting, nvme_remove_namespaces() will hang waiting for scan_work to flush and these tasks will hang.

To fix this, ensure we take scan_lock before changing the ctrl-state. Also, ensure the state is checked while the lock is held in nvme_scan_lock_work().

That's a big hammer...

But this is I/O that we cannot have queued until we have a path..

I would rather have nvme_remove_namespaces() requeue all I/Os for namespaces that serve as the current_path and have the make_request routine to fail I/O if all controllers are deleting as well.

Would something like [1] (untested) make sense instead?

Review

Quoted 4 lines
(quoted diff) +2 -0
+	mutex_lock(&ctrl->scan_lock);
+
  	if (ctrl->state != NVME_CTRL_LIVE)
  		return;

unlock

Review static void nvme_scan_work(struct work_struct *work) · line 3555

Quoted 4 lines
(quoted diff) +0 -0
@@ -3547,7 +3554,6 @@ static void nvme_scan_work(struct work_struct *work)
  	if (nvme_identify_ctrl(ctrl, &id))
  		return;

unlock

[1]:
--

drivers/nvme/host/core.c +5 -0
index 76cd3dd8736a..627f5871858d 100644
@@ -3576,6 +3576,11 @@ void nvme_remove_namespaces(struct nvme_ctrl *ctrl)
         struct nvme_ns *ns, *next;
         LIST_HEAD(ns_list);
 
+       mutex_lock(&ctrl->scan_lock);
+       list_for_each_entry(ns, &ctrl->namespaces, list)
+               nvme_mpath_clear_current_path(ns);
+       mutex_lock(&ctrl->scan_lock);
+
         /* prevent racing with ns scanning */
         flush_work(&ctrl->scan_work);
 
drivers/nvme/host/multipath.c +0 -0
index a9a927677970..da1731266788 100644
@@ -231,6 +231,24 @@ inline struct nvme_ns *nvme_find_path(struct 

nvme_ns_head *head)

         return ns;

}

+static bool nvme_available_path(struct nvme_ns_head *head) +{
+ struct nvme_ns *ns;
+
+ list_for_each_entry_rcu(ns, &head->list, siblings) {

+               switch (ns->ctrl->state) {
+               case NVME_CTRL_LIVE:
+               case NVME_CTRL_RESETTING:
+               case NVME_CTRL_CONNECTING:

+ /* fallthru */
+ return true;
+ default:
+ break;
+ }
+ }
+ return false;
+}
+
static blk_qc_t nvme_ns_head_make_request(struct request_queue *q,
struct bio *bio)
{
@@ -257,14 +275,14 @@ static blk_qc_t nvme_ns_head_make_request(struct request_queue *q,

                                       disk_devt(ns->head->disk),
                                       bio->bi_iter.bi_sector);
                 ret = direct_make_request(bio);

- } else if (!list_empty_careful(&head->list)) {
- dev_warn_ratelimited(dev, "no path available - requeuing
I/O\n");
+ } else if (nvme_available_path(head)) {
+ dev_warn_ratelimited(dev, "no usable path - requeuing
I/O\n");

                 spin_lock_irq(&head->requeue_lock);
                 bio_list_add(&head->requeue_list, bio);
                 spin_unlock_irq(&head->requeue_lock);
         } else {

- dev_warn_ratelimited(dev, "no path - failing I/O\n");
+ dev_warn_ratelimited(dev, "no available path - failing
I/O\n");

                 bio->bi_status = BLK_STS_IOERR;
                 bio_endio(bio);

--

maintainer
#3 lore raw

Re: [PATCH 2/2] nvme-core: Fix deadlock when deleting the ctrl while scanning

Quoted 1 line

[1]:

Or actually:
--

drivers/nvme/host/core.c +6 -0
index 76cd3dd8736a..a0e2072fe73e 100644
@@ -3576,6 +3576,12 @@ void nvme_remove_namespaces(struct nvme_ctrl *ctrl)
         struct nvme_ns *ns, *next;
         LIST_HEAD(ns_list);
 
+       mutex_lock(&ctrl->scan_lock);
+       list_for_each_entry(ns, &ctrl->namespaces, list)
+               if (nvme_mpath_clear_current_path(ns))
+                       kblockd_schedule_work(&ns->head->requeue_work);
+       mutex_lock(&ctrl->scan_lock);
+
         /* prevent racing with ns scanning */
         flush_work(&ctrl->scan_work);
 
drivers/nvme/host/multipath.c +7 -3
index a9a927677970..0e7e41381388 100644
@@ -109,18 +109,22 @@ static const char *nvme_ana_state_names[] = {
         [NVME_ANA_CHANGE]               = "change",
  };
 
-void nvme_mpath_clear_current_path(struct nvme_ns *ns)
+bool nvme_mpath_clear_current_path(struct nvme_ns *ns)
  {
         struct nvme_ns_head *head = ns->head;
+       bool changed = false;
         int node;
 
         if (!head)
-               return;
+               return changed;
 
         for_each_node(node) {
-               if (ns == rcu_access_pointer(head->current_path[node]))
+               if (ns == rcu_access_pointer(head->current_path[node])) {
                         rcu_assign_pointer(head->current_path[node], NULL);
+                       changed = true;
+               }
         }
+       return changed;
  }
 
  static bool nvme_path_is_disabled(struct nvme_ns *ns)
@@ -231,6 +235,24 @@ inline struct nvme_ns *nvme_find_path(struct 

nvme_ns_head *head)

         return ns;

}

+static bool nvme_available_path(struct nvme_ns_head *head) +{
+ struct nvme_ns *ns;
+
+ list_for_each_entry_rcu(ns, &head->list, siblings) {

+               switch (ns->ctrl->state) {
+               case NVME_CTRL_LIVE:
+               case NVME_CTRL_RESETTING:
+               case NVME_CTRL_CONNECTING:

+ /* fallthru */
+ return true;
+ default:
+ break;
+ }
+ }
+ return false;
+}
+
static blk_qc_t nvme_ns_head_make_request(struct request_queue *q,
struct bio *bio)
{
@@ -257,14 +279,14 @@ static blk_qc_t nvme_ns_head_make_request(struct request_queue *q,

                                       disk_devt(ns->head->disk),
                                       bio->bi_iter.bi_sector);
                 ret = direct_make_request(bio);

- } else if (!list_empty_careful(&head->list)) {
- dev_warn_ratelimited(dev, "no path available - requeuing
I/O\n");
+ } else if (nvme_available_path(head)) {
+ dev_warn_ratelimited(dev, "no usable path - requeuing
I/O\n");

                 spin_lock_irq(&head->requeue_lock);
                 bio_list_add(&head->requeue_list, bio);
                 spin_unlock_irq(&head->requeue_lock);
         } else {

- dev_warn_ratelimited(dev, "no path - failing I/O\n");
+ dev_warn_ratelimited(dev, "no available path - failing
I/O\n");

                 bio->bi_status = BLK_STS_IOERR;
                 bio_endio(bio);
drivers/nvme/host/nvme.h +1 -1
index 716a876119c8..915179368d30 100644
@@ -496,7 +496,7 @@ void nvme_mpath_remove_disk(struct nvme_ns_head *head);
  int nvme_mpath_init(struct nvme_ctrl *ctrl, struct nvme_id_ctrl *id);
  void nvme_mpath_uninit(struct nvme_ctrl *ctrl);
  void nvme_mpath_stop(struct nvme_ctrl *ctrl);
-void nvme_mpath_clear_current_path(struct nvme_ns *ns);
+bool nvme_mpath_clear_current_path(struct nvme_ns *ns);
  struct nvme_ns *nvme_find_path(struct nvme_ns_head *head);
 
  static inline void nvme_mpath_check_last_path(struct nvme_ns *ns)

--

maintainer
#4 lore raw

Re: [PATCH 2/2] nvme-core: Fix deadlock when deleting the ctrl while scanning

Quoted 9 lines
Quoted 2 lines

[1]:

Or actually:

drivers/nvme/host/core.c +0 -0
index 76cd3dd8736a..a0e2072fe73e 100644

drivers/nvme/host/core.c void nvme_remove_namespaces(struct nvme_ctrl *ctrl) · line 3583

Quoted 8 lines
drivers/nvme/host/core.c +5 -0
@@ -3576,6 +3576,12 @@ void nvme_remove_namespaces(struct nvme_ctrl *ctrl)
 ??????? struct nvme_ns *ns, *next;
 ??????? LIST_HEAD(ns_list);
 
+?????? mutex_lock(&ctrl->scan_lock);
+?????? list_for_each_entry(ns, &ctrl->namespaces, list)
+?????????????? if (nvme_mpath_clear_current_path(ns))
+?????????????????????? kblockd_schedule_work(&ns->head->requeue_work);
+?????? mutex_lock(&ctrl->scan_lock);

This should be mutex_unlock of course...

#5 lore raw

Re: [PATCH 2/2] nvme-core: Fix deadlock when deleting the ctrl while scanning

In reply to Sagi Grimberg
Quoted 6 lines · Sagi Grimberg
Quoted 6 lines

With multipath enabled, nvme_scan_work() can read from the device (through nvme_mpath_add_disk()). However, with fabrics, once ctrl->state is set to NVME_CTRL_DELETING, the reads will hang (see nvmf_check_ready()).

Quoted 8 lines
Quoted 6 lines

After setting the state to deleting, nvme_remove_namespaces() will hang waiting for scan_work to flush and these tasks will hang.

To fix this, ensure we take scan_lock before changing the ctrl-state. Also, ensure the state is checked while the lock is held in nvme_scan_lock_work().

That's a big hammer...

I didn't think the scan_lock was that contested or that nvme_change_ctrl_state() was really called that often...

Quoted 7 lines

But this is I/O that we cannot have queued until we have a path..

I would rather have nvme_remove_namespaces() requeue all I/Os for namespaces that serve as the current_path and have the make_request routine to fail I/O if all controllers are deleting as well.

Would something like [1] (untested) make sense instead?

I'll have to give this a try next week and I'll let you know then. It kind of makes sense to me but a number of things I tried to fix this that I thought made sense did not work.

Quoted 7 lines
Quoted 5 lines

+??? mutex_lock(&ctrl->scan_lock);
+
????? if (ctrl->state != NVME_CTRL_LIVE)
????????? return;

unlock

If we unlock here and relock below, we'd have to recheck the ctrl->state to avoid any races. If you don't want to call nvme_identify_ctrl with the lock held, then it would probably be better to move the state check below it.

Quoted 79 lines
Quoted 4 lines

? @@ -3547,7 +3554,6 @@ static void nvme_scan_work(struct work_struct *work)
????? if (nvme_identify_ctrl(ctrl, &id))
????????? return;

unlock

[1]:

drivers/nvme/host/core.c +5 -0
index 76cd3dd8736a..627f5871858d 100644
@@ -3576,6 +3576,11 @@ void nvme_remove_namespaces(struct nvme_ctrl *ctrl)
 ??????? struct nvme_ns *ns, *next;
 ??????? LIST_HEAD(ns_list);
 
+?????? mutex_lock(&ctrl->scan_lock);
+?????? list_for_each_entry(ns, &ctrl->namespaces, list)
+?????????????? nvme_mpath_clear_current_path(ns);
+?????? mutex_lock(&ctrl->scan_lock);
+

??????? /* prevent racing with ns scanning */
??????? flush_work(&ctrl->scan_work);

drivers/nvme/host/multipath.c +18 -0
index a9a927677970..da1731266788 100644
@@ -231,6 +231,24 @@ inline struct nvme_ns *nvme_find_path(struct
 nvme_ns_head *head)
 ??????? return ns;
 ?}
 
+static bool nvme_available_path(struct nvme_ns_head *head)
+{
+?????? struct nvme_ns *ns;
+
+?????? list_for_each_entry_rcu(ns, &head->list, siblings) {
+?????????????? switch (ns->ctrl->state) {
+?????????????? case NVME_CTRL_LIVE:
+?????????????? case NVME_CTRL_RESETTING:
+?????????????? case NVME_CTRL_CONNECTING:
+?????????????????????? /* fallthru */
+?????????????????????? return true;
+?????????????? default:
+?????????????????????? break;
+?????????????? }
+?????? }
+?????? return false;
+}
+
 ?static blk_qc_t nvme_ns_head_make_request(struct request_queue *q,
 ??????????????? struct bio *bio)

?{

drivers/nvme/host/multipath.c +3 -3
@@ -257,14 +275,14 @@ static blk_qc_t nvme_ns_head_make_request(struct
 request_queue *q,
 ????????????????????????????????????? disk_devt(ns->head->disk),
 ????????????????????????????????????? bio->bi_iter.bi_sector);
 ??????????????? ret = direct_make_request(bio);
-?????? } else if (!list_empty_careful(&head->list)) {
-?????????????? dev_warn_ratelimited(dev, "no path available - requeuing
 I/O\n");
+?????? } else if (nvme_available_path(head)) {
+?????????????? dev_warn_ratelimited(dev, "no usable path - requeuing
 I/O\n");
 
 ??????????????? spin_lock_irq(&head->requeue_lock);
 ??????????????? bio_list_add(&head->requeue_list, bio);
 ??????????????? spin_unlock_irq(&head->requeue_lock);
 ??????? } else {
-?????????????? dev_warn_ratelimited(dev, "no path - failing I/O\n");
+?????????????? dev_warn_ratelimited(dev, "no available path - failing

I/O\n");

??????????????? bio->bi_status = BLK_STS_IOERR;
??????????????? bio_endio(bio);

maintainer
#6 lore raw

Re: [PATCH 2/2] nvme-core: Fix deadlock when deleting the ctrl while scanning

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

With multipath enabled, nvme_scan_work() can read from the device (through nvme_mpath_add_disk()). However, with fabrics, once ctrl->state is set to NVME_CTRL_DELETING, the reads will hang (see nvmf_check_ready()).

After setting the state to deleting, nvme_remove_namespaces() will hang waiting for scan_work to flush and these tasks will hang.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

To fix this, ensure we take scan_lock before changing the ctrl-state. Also, ensure the state is checked while the lock is held in nvme_scan_lock_work().

That's a big hammer...

I didn't think the scan_lock was that contested or that nvme_change_ctrl_state() was really called that often...

it shouldn't be, but I think it makes the flow more convoluted as we serialize by flushing the scan_work right after...

The design principal is met as we do get the I/O failing, but its just that with mpath we simply queue the I/O again because the head->list happens to not be empty. Perhaps taking care of that check is cleaner.

Quoted 11 lines
Quoted 7 lines

But this is I/O that we cannot have queued until we have a path..

I would rather have nvme_remove_namespaces() requeue all I/Os for namespaces that serve as the current_path and have the make_request routine to fail I/O if all controllers are deleting as well.

Would something like [1] (untested) make sense instead?

I'll have to give this a try next week and I'll let you know then. It kind of makes sense to me but a number of things I tried to fix this that I thought made sense did not work.

Thanks. Do you have a firm reproducer for it?

Quoted 11 lines
Quoted 6 lines
Quoted 4 lines

+??? mutex_lock(&ctrl->scan_lock);
+
????? if (ctrl->state != NVME_CTRL_LIVE)
????????? return;

unlock

If we unlock here and relock below, we'd have to recheck the ctrl->state to avoid any races. If you don't want to call nvme_identify_ctrl with the lock held, then it would probably be better to move the state check below it.

Meant before the return statement.

Quoted 7 lines
Quoted 7 lines
Quoted 5 lines

? @@ -3547,7 +3554,6 @@ static void nvme_scan_work(struct work_struct
*work)
????? if (nvme_identify_ctrl(ctrl, &id))
????????? return;

unlock

Same here.

#7 lore raw

Re: [PATCH 2/2] nvme-core: Fix deadlock when deleting the ctrl while scanning

Hey,

Sorry for the delay.

I tested your patch and it does work. Do you want me to send your change as a full patch? Can I add your signed-off-by?

Quoted 5 lines · Sagi Grimberg
Quoted 2 lines

I didn't think the scan_lock was that contested or that nvme_change_ctrl_state() was really called that often...

it shouldn't be, but I think it makes the flow more convoluted as we serialize by flushing the scan_work right after...

I would argue that the check for state in nvme_scan_work() without a lock is racy and confusing. There's nothing to prevent the state from changing immediately after the check.

Quoted 4 lines

The design principal is met as we do get the I/O failing, but its just that with mpath we simply queue the I/O again because the head->list happens to not be empty. Perhaps taking care of that check is cleaner.

Yes, I feel your patch is a good solution on it's own merits.

Quoted 1 line

Thanks. Do you have a firm reproducer for it?

Yes. If you connect to and then immediately disconnect from a target (at least with nvme-loop) you will reliably trigger this bug -- or one of the others I've sent patches for.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

+??? mutex_lock(&ctrl->scan_lock);
+
?????? if (ctrl->state != NVME_CTRL_LIVE)
?????????? return;

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

unlock

If we unlock here and relock below, we'd have to recheck the ctrl->state to avoid any races. If you don't want to call nvme_identify_ctrl with the lock held, then it would probably be better to move the state check below it.

Meant before the return statement.

Ah, right, my mistake.

Logan

maintainer
#8 lore raw

Re: [PATCH 2/2] nvme-core: Fix deadlock when deleting the ctrl while scanning

Quoted 7 lines

Hey,

Sorry for the delay.

I tested your patch and it does work. Do you want me to send your change as a full patch? Can I add your signed-off-by?

I have a patch ready with a proper comment in place. I can send it out... Can I get your tested-by and reported-by?

#9 lore raw

Re: [PATCH 2/2] nvme-core: Fix deadlock when deleting the ctrl while scanning

Quoted 10 lines · Sagi Grimberg
Quoted 7 lines

Hey,

Sorry for the delay.

I tested your patch and it does work. Do you want me to send your change as a full patch? Can I add your signed-off-by?

I have a patch ready with a proper comment in place. I can send it out... Can I get your tested-by and reported-by?

Yup. I can retest once you send the patch and I'll send a test-by then. For now you can add my reported-by:

Reported-by Logan Gunthorpe <logang at deltatee.com>

Note: I had to modify your original patch a fair bit so it compiled with multipath disabled.

Logan