Skip to contentSkip to conversation

nvme: swap synchronization ordering in nvme_remove_head()

PATCH Applied John Garry 5 participants 5 messages Jul 7 – Jul 14 NVMe

lore
Share & tools
lore

John Garry

author
#0 lore raw

PATCH 1/1 nvme: swap synchronization ordering in nvme_remove_head()

From: John Garry <[email protected]>

sashiko bot reported a potential issue in the requeue handling in [0] - the code there is same as the NVMe driver.

The issue is that when we schedule the requeue work, if a bio is added to the requeue list afterwards in nvme_ns_head_submit_bio(), it is missed by the requeue worker.

This issue can be recreated by hacking a large delay in the bio submission requeue path:

        } else if (nvme_available_path(head)) {
                dev_warn_ratelimited(dev, "no usable path - requeuing I/O\n");
+               msleep(30000);
                spin_lock_irq(&head->requeue_lock);
                bio_list_add(&head->requeue_list, bio);
                spin_unlock_irq(&head->requeue_lock);

Then if we issue a write after removing all paths, a hang can be seen:

# echo 20 > /sys/devices/virtual/nvme-subsystem/nvme-subsys1/nvme1n1/delayed_removal_secs
#
# ./ini_nvme_teardown.sh
[   25.877224] nvme nvme1: Removing ctrl: NQN "nvme-test-target"
[   25.939569] nvme nvme2: Removing ctrl: NQN "nvme-test-target"
#
# xfs_io -d -C "pwrite -b 64k -V 1 -D 0 64k" /dev/nvme1n1p1
[   29.883653] block nvme1n1: no usable path - requeuing I/O

Fix by re-ordering the SRCU synchronization and scheduling the requeue work.

[0] https://lore.kernel.org/linux-scsi/[email protected]/T/#m72af1f29deb0ebfb2973464207f201f1be1f660c

---
I am not sure if we still require the synchronize_srcu() after nvme_cdev_del().

drivers/nvme/host/multipath.c +4 -3
index 016b6b0128c7..0b017eeb82b2 100644
@@ -690,14 +690,15 @@ static void nvme_remove_head(struct nvme_ns_head *head)
 {
 	if (test_and_clear_bit(NVME_NSHEAD_DISK_LIVE, &head->flags)) {
 		/*
-		 * requeue I/O after NVME_NSHEAD_DISK_LIVE has been cleared
-		 * to allow multipath to fail all I/O.
+		 * Requeue I/O after NVME_NSHEAD_DISK_LIVE has been cleared
+		 * to allow multipath to fail all I/O. First synchronize to
+		 * add any bios to the requeue list.
 		 */
+		synchronize_srcu(&head->srcu);
 		kblockd_schedule_work(&head->requeue_work);
 
 		if (test_and_clear_bit(NVME_NSHEAD_CDEV_LIVE, &head->flags))
 			nvme_cdev_del(&head->cdev, &head->cdev_device);
-		synchronize_srcu(&head->srcu);
 		del_gendisk(head->disk);
 	}
 	nvme_put_ns_head(head);
Signature · 1 line
2.43.0
Reviewed-by on v1 1/1
#2 lore raw
In reply to John Garry

Hi John,

Quoted 54 lines · John Garry

From: John Garry <[email protected]>

sashiko bot reported a potential issue in the requeue handling in [0] - the code there is same as the NVMe driver.

The issue is that when we schedule the requeue work, if a bio is added to the requeue list afterwards in nvme_ns_head_submit_bio(), it is missed by the requeue worker.

This issue can be recreated by hacking a large delay in the bio submission requeue path:

         } else if (nvme_available_path(head)) {
                 dev_warn_ratelimited(dev, "no usable path - requeuing I/O\n");
+               msleep(30000);
                 spin_lock_irq(&head->requeue_lock);
                 bio_list_add(&head->requeue_list, bio);
                 spin_unlock_irq(&head->requeue_lock);

Then if we issue a write after removing all paths, a hang can be seen:

# echo 20 > /sys/devices/virtual/nvme-subsystem/nvme-subsys1/nvme1n1/delayed_removal_secs
#
# ./ini_nvme_teardown.sh
[   25.877224] nvme nvme1: Removing ctrl: NQN "nvme-test-target"
[   25.939569] nvme nvme2: Removing ctrl: NQN "nvme-test-target"
#
# xfs_io -d -C "pwrite -b 64k -V 1 -D 0 64k" /dev/nvme1n1p1
[   29.883653] block nvme1n1: no usable path - requeuing I/O

Fix by re-ordering the SRCU synchronization and scheduling the requeue work.

[0] https://lore.kernel.org/linux-scsi/[email protected]/T/#m72af1f29deb0ebfb2973464207f201f1be1f660c

---
I am not sure if we still require the synchronize_srcu() after nvme_cdev_del().

drivers/nvme/host/multipath.c +3 -2
index 016b6b0128c7..0b017eeb82b2 100644
@@ -690,14 +690,15 @@ static void nvme_remove_head(struct nvme_ns_head *head)
  {
  	if (test_and_clear_bit(NVME_NSHEAD_DISK_LIVE, &head->flags)) {
  		/*
-		 * requeue I/O after NVME_NSHEAD_DISK_LIVE has been cleared
-		 * to allow multipath to fail all I/O.
+		 * Requeue I/O after NVME_NSHEAD_DISK_LIVE has been cleared
+		 * to allow multipath to fail all I/O. First synchronize to
+		 * add any bios to the requeue list.

drivers/nvme/host/multipath.c static void nvme_remove_head(struct nvme_ns_head *head) · line 704

Quoted 10 lines
drivers/nvme/host/multipath.c +1 -1
  		 */
+		synchronize_srcu(&head->srcu);
  		kblockd_schedule_work(&head->requeue_work);
 
  		if (test_and_clear_bit(NVME_NSHEAD_CDEV_LIVE, &head->flags))
  			nvme_cdev_del(&head->cdev, &head->cdev_device);
-		synchronize_srcu(&head->srcu);
  		del_gendisk(head->disk);
  	}
  	nvme_put_ns_head(head);

With the above change now I wonder do we really need to schedule ->requeue_work from nvme_mpath_put_disk()? Maybe we still want to keep it as a defensive "safety net". Though having it doesn't harm but looks redundant.

Otherwise, this change look good to me,

maintainer Applied
#3 lore raw
In reply to John Garry
Quoted 8 lines · John Garry

From: John Garry <[email protected]>

sashiko bot reported a potential issue in the requeue handling in [0] - the code there is same as the NVMe driver.

The issue is that when we schedule the requeue work, if a bio is added to the requeue list afterwards in nvme_ns_head_submit_bio(), it is missed by the requeue worker.

Thanks, applied to nvme-7.3.

Reviewed-by on v1 1/1
#4 lore raw
In reply to John Garry
Quoted 32 lines · John Garry

From: John Garry <[email protected]>

sashiko bot reported a potential issue in the requeue handling in [0] - the code there is same as the NVMe driver.

The issue is that when we schedule the requeue work, if a bio is added to the requeue list afterwards in nvme_ns_head_submit_bio(), it is missed by the requeue worker.

This issue can be recreated by hacking a large delay in the bio submission requeue path:

         } else if (nvme_available_path(head)) {
                 dev_warn_ratelimited(dev, "no usable path - requeuing I/O\n");
+               msleep(30000);
                 spin_lock_irq(&head->requeue_lock);
                 bio_list_add(&head->requeue_list, bio);
                 spin_unlock_irq(&head->requeue_lock);

Then if we issue a write after removing all paths, a hang can be seen:

# echo 20 > /sys/devices/virtual/nvme-subsystem/nvme-subsys1/nvme1n1/delayed_removal_secs
#
# ./ini_nvme_teardown.sh
[   25.877224] nvme nvme1: Removing ctrl: NQN "nvme-test-target"
[   25.939569] nvme nvme2: Removing ctrl: NQN "nvme-test-target"
#
# xfs_io -d -C "pwrite -b 64k -V 1 -D 0 64k" /dev/nvme1n1p1
[   29.883653] block nvme1n1: no usable path - requeuing I/O
Quoted 9 lines

Fix by re-ordering the SRCU synchronization and scheduling the requeue work.

[0] https://lore.kernel.org/linux-scsi/[email protected]/T/#m72af1f29deb0ebfb2973464207f201f1be1f660c

---
I am not sure if we still require the synchronize_srcu() after nvme_cdev_del().

Which one? I only see one after 'list_del_rcu()', and there we definitely need it ...

Otherwise:

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich