nvme: swap synchronization ordering in nvme_remove_head()
PATCH 1/1 nvme: swap synchronization ordering in nvme_remove_head()
From: John Garry <[email protected]>
sashiko bot reported a potential issue in the requeue handling in [0] - the code there is same as the NVMe driver.
The issue is that when we schedule the requeue work, if a bio is added to the requeue list afterwards in nvme_ns_head_submit_bio(), it is missed by the requeue worker.
This issue can be recreated by hacking a large delay in the bio submission requeue path:
} else if (nvme_available_path(head)) {
dev_warn_ratelimited(dev, "no usable path - requeuing I/O\n");+ msleep(30000);
spin_lock_irq(&head->requeue_lock);
bio_list_add(&head->requeue_list, bio);
spin_unlock_irq(&head->requeue_lock);Then if we issue a write after removing all paths, a hang can be seen:
# echo 20 > /sys/devices/virtual/nvme-subsystem/nvme-subsys1/nvme1n1/delayed_removal_secs # # ./ini_nvme_teardown.sh [ 25.877224] nvme nvme1: Removing ctrl: NQN "nvme-test-target" [ 25.939569] nvme nvme2: Removing ctrl: NQN "nvme-test-target" # # xfs_io -d -C "pwrite -b 64k -V 1 -D 0 64k" /dev/nvme1n1p1 [ 29.883653] block nvme1n1: no usable path - requeuing I/O
Fix by re-ordering the SRCU synchronization and scheduling the requeue work.
[0] https://lore.kernel.org/linux-scsi/[email protected]/T/#m72af1f29deb0ebfb2973464207f201f1be1f660c
---
I am not sure if we still require the synchronize_srcu() after nvme_cdev_del().
drivers/nvme/host/multipath.c +4 -3
@@ -690,14 +690,15 @@ static void nvme_remove_head(struct nvme_ns_head *head) {if (test_and_clear_bit (NVME_NSHEAD_DISK_LIVE , &head->flags)) {/* -* requeue I/O after NVME_NSHEAD_DISK_LIVE has been cleared -* to allow multipath to fail all I/O. +* Requeue I/O after NVME_NSHEAD_DISK_LIVE has been cleared +* to allow multipath to fail all I/O. First synchronize to +* add any bios to the requeue list. */ +synchronize_srcu (&head->srcu);kblockd_schedule_work (&head->requeue_work);if (test_and_clear_bit (NVME_NSHEAD_CDEV_LIVE , &head->flags))nvme_cdev_del (&head->cdev, &head->cdev_device); -synchronize_srcu (&head->srcu);del_gendisk (head->disk); }nvme_put_ns_head (head);
Signature · 1 line
2.43.0
Looks good:
Hi John,
Quoted 54 lines · John Garry
From: John Garry <[email protected]>
sashiko bot reported a potential issue in the requeue handling in [0] - the code there is same as the NVMe driver.
The issue is that when we schedule the requeue work, if a bio is added to the requeue list afterwards in
nvme_ns_head_submit_bio(), it is missed by the requeue worker.This issue can be recreated by hacking a large delay in the bio submission requeue path:
} else if (nvme_available_path(head)) { dev_warn_ratelimited(dev, "no usable path - requeuing I/O\n");+ msleep(30000); spin_lock_irq(&head->requeue_lock); bio_list_add(&head->requeue_list, bio); spin_unlock_irq(&head->requeue_lock);Then if we issue a write after removing all paths, a hang can be seen:
# echo 20 > /sys/devices/virtual/nvme-subsystem/nvme-subsys1/nvme1n1/delayed_removal_secs # # ./ini_nvme_teardown.sh [ 25.877224] nvme nvme1: Removing ctrl: NQN "nvme-test-target" [ 25.939569] nvme nvme2: Removing ctrl: NQN "nvme-test-target" # # xfs_io -d -C "pwrite -b 64k -V 1 -D 0 64k" /dev/nvme1n1p1 [ 29.883653] block nvme1n1: no usable path - requeuing I/OFix by re-ordering the SRCU synchronization and scheduling the requeue work.
[0] https://lore.kernel.org/linux-scsi/[email protected]/T/#m72af1f29deb0ebfb2973464207f201f1be1f660c
Signed-off-by John Garry <[email protected]>---
I am not sure if we still require thesynchronize_srcu()afternvme_cdev_del().drivers/nvme/host/multipath.c +3
-2@@ -690,14 +690,15 @@ static void nvme_remove_head(struct nvme_ns_head *head) {if (test_and_clear_bit (NVME_NSHEAD_DISK_LIVE , &head->flags)) {/* -* requeue I/O after NVME_NSHEAD_DISK_LIVE has been cleared -* to allow multipath to fail all I/O. +* Requeue I/O after NVME_NSHEAD_DISK_LIVE has been cleared +* to allow multipath to fail all I/O. First synchronize to +* add any bios to the requeue list.
drivers/nvme/host/multipath.c static void nvme_remove_head(struct nvme_ns_head *head) · line 704
Quoted 10 lines
drivers/nvme/host/multipath.c +1
-1*/ +synchronize_srcu (&head->srcu);kblockd_schedule_work (&head->requeue_work);if (test_and_clear_bit (NVME_NSHEAD_CDEV_LIVE , &head->flags))nvme_cdev_del (&head->cdev, &head->cdev_device); -synchronize_srcu (&head->srcu);del_gendisk (head->disk); }nvme_put_ns_head (head);
With the above change now I wonder do we really need to schedule ->requeue_work from nvme_mpath_put_disk()? Maybe we still want to keep it as a defensive "safety net". Though having it doesn't harm but looks redundant.
Otherwise, this change look good to me,
Quoted 8 lines · John Garry
From: John Garry <[email protected]>
sashiko bot reported a potential issue in the requeue handling in [0] - the code there is same as the NVMe driver.
The issue is that when we schedule the requeue work, if a bio is added to the requeue list afterwards in
nvme_ns_head_submit_bio(), it is missed by the requeue worker.
Thanks, applied to nvme-7.3.
Quoted 32 lines · John Garry
From: John Garry <[email protected]>
sashiko bot reported a potential issue in the requeue handling in [0] - the code there is same as the NVMe driver.
The issue is that when we schedule the requeue work, if a bio is added to the requeue list afterwards in
nvme_ns_head_submit_bio(), it is missed by the requeue worker.This issue can be recreated by hacking a large delay in the bio submission requeue path:
} else if (nvme_available_path(head)) { dev_warn_ratelimited(dev, "no usable path - requeuing I/O\n");+ msleep(30000); spin_lock_irq(&head->requeue_lock); bio_list_add(&head->requeue_list, bio); spin_unlock_irq(&head->requeue_lock);Then if we issue a write after removing all paths, a hang can be seen:
# echo 20 > /sys/devices/virtual/nvme-subsystem/nvme-subsys1/nvme1n1/delayed_removal_secs # # ./ini_nvme_teardown.sh [ 25.877224] nvme nvme1: Removing ctrl: NQN "nvme-test-target" [ 25.939569] nvme nvme2: Removing ctrl: NQN "nvme-test-target" # # xfs_io -d -C "pwrite -b 64k -V 1 -D 0 64k" /dev/nvme1n1p1 [ 29.883653] block nvme1n1: no usable path - requeuing I/O
Quoted 9 lines
Fix by re-ordering the SRCU synchronization and scheduling the requeue work.
[0] https://lore.kernel.org/linux-scsi/[email protected]/T/#m72af1f29deb0ebfb2973464207f201f1be1f660c
Signed-off-by John Garry <[email protected]>---
I am not sure if we still require thesynchronize_srcu()afternvme_cdev_del().
Which one? I only see one after 'list_del_rcu()', and there we definitely need it ...
Otherwise:
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich