Skip to contentSkip to conversation

nvme-multipath: requeue work fixes

PATCH Superseded Qiliang Yuan 4 participants 7 messages Nov 19 – Nov 20 NVMe

v5 of 6 Superseded Latest is v6 →

lore
Share & tools
lore
#0 lore raw

v5 cover nvme-multipath: requeue work fixes

Patch 1 applies a new delayed_removal_secs to a head removal that is already pending, so that clearing it fails the I/O queued without a path right away instead of when the old delay expires.

Patch 2 stops nvme_mpath_set_live() from waiting for an SRCU grace period per namespace, which makes an ANA change with many namespaces take tens to hundreds of milliseconds.

---
V4 -> V5:
- Cancel the pending head removal under subsys->lock in nvme_init_ns_head() and drop the list_empty() check and module_put() from delayed_removal_secs_store() (Nilay Shroff)
- Rebase onto v7.4-rc3-230-g4405c18c95d2 and update the numbers of patch 1 from a new run

V3 -> V4:
- Don't requeue the head removal in patch 1 if a path came back before nvme_init_ns_head() cancelled it, drop the module reference it held instead (race reported by Sashiko)
- Rebase onto v7.4-rc3-178-g20eef29b7c84 and update the numbers of patch 1 from a new run

V2 -> V3:
- Requeue a pending head removal with the new delayed_removal_secs instead of only kicking the requeue work when it is cleared
- Rebase onto v7.4-rc2-470-g7e1392a42ea2 and update the numbers from a new run
- Don't carry Hannes's Reviewed-by on the v2 patch 1, which is rewritten

V1 -> V2:
- Add patch 1
- Take the SRCU cookie under head->lock in patch 2 (Sashiko)
- Rerun the tests

v4: https://lore.kernel.org/r/20261118-bug-nvme-mpath-set-live-srcu-v4-0-21f368376c8c@linux.dev
v3: https://lore.kernel.org/r/20261114-bug-nvme-mpath-set-live-srcu-v3-0-d84651de6334@linux.dev
v2: https://lore.kernel.org/r/20261111-bug-nvme-mpath-set-live-srcu-v2-0-fc772914d293@linux.dev
v1: https://lore.kernel.org/r/20261110-bug-nvme-mpath-set-live-srcu-v1-1-1f58b405b0f8@linux.dev

---
Qiliang Yuan (2):
nvme-multipath: apply delayed_removal_secs changes to a pending removal
nvme-multipath: don't wait for a grace period in nvme_mpath_set_live()

 drivers/nvme/host/core.c      |  7 +++++--
 drivers/nvme/host/multipath.c | 21 ++++++++++++++++++++-
 drivers/nvme/host/nvme.h      |  1 +
 3 files changed, 26 insertions(+), 3 deletions(-)

---
base-commit: 4405c18c95d2d0f99f6c581ef0ee8591a1aa6ccd
change-id: 20261110-bug-nvme-mpath-set-live-srcu-cbd8965f1457

Best regards,

Signature · 1 line
Qiliang Yuan <[email protected]>
#1 lore raw

PATCH v5 1/2 nvme-multipath: apply delayed_removal_secs changes to a pending removal

With delayed_removal_secs set, removing the last path of a namespace head doesn't delete the head right away. nvme_mpath_remove_disk() queues remove_work with the configured delay, and bios submitted in the meantime wait on head->requeue_list in case a path comes back.

A write to delayed_removal_secs only takes effect for removals that start later. A removal that is already pending keeps the old delay, so clearing the attribute leaves the queued bios waiting until that delay expires instead of failing them, and lowering or raising it doesn't move the pending removal either.

Requeue a pending remove_work with the new delay when the attribute is written, cancelling it first so that the module reference taken for it carries over. Have nvme_init_ns_head() cancel the pending removal before it drops subsys->lock rather than after, so that a write to the attribute cannot requeue the removal once a path is back.

With delayed_removal_secs set and no path left, the attribute is changed two seconds later and the head is removed after (mean of two runs, min-max in parentheses):

  delay (s)  before (ms)            after (ms)             change
  30 -> 0    26655 (26537-26772)        8 (8-8)            -99.97%
  30 -> 5    28298 (28094-28501)     5514 (5510-5517)      -81%
  10 -> 30    8112 (8106-8117)      30376 (30106-30646)    +274%

A direct read queued before the 30 -> 0 write fails after:

  before (ms)            after (ms)   change
  26607 (26450-26764)    4 (4-4)      -99.98%
Fixes 62188639ec160 ("nvme-multipath: introduce delayed removal of the multipath head node")
---
 drivers/nvme/host/core.c      | 7 +++++--
 drivers/nvme/host/multipath.c | 6 ++++++
 2 files changed, 11 insertions(+), 2 deletions(-)
drivers/nvme/host/core.c +5 -2
index f4aa02869b790..48d432c5d7aeb 100644
@@ -4231,12 +4231,15 @@ static int nvme_init_ns_head(struct nvme_ns *ns, struct nvme_ns_info *info)
 
 	list_add_tail_rcu(&ns->siblings, &head->list);
 	ns->head = head;
-	mutex_unlock(&ctrl->subsys->lock);
-
 #ifdef CONFIG_NVME_MULTIPATH
+	/*
+	 * Cancel a pending head removal under subsys->lock, so that
+	 * delayed_removal_secs_store() never requeues it once a path is back.
+	 */
 	if (cancel_delayed_work(&head->remove_work))
 		module_put(THIS_MODULE);
 #endif
+	mutex_unlock(&ctrl->subsys->lock);
 	return 0;
 
 out_put_ns_head:
drivers/nvme/host/multipath.c +6 -0
index ae8059366b57d..51c59cc3ea246 100644
@@ -1242,6 +1242,12 @@ static ssize_t delayed_removal_secs_store(struct device *dev,
 	mutex_lock(&head->subsys->lock);
 	head->delayed_removal_secs = sec;
 	assign_bit(NVME_NSHEAD_QUEUE_IF_NO_PATH, &head->flags, sec);
+	/*
+	 * Apply the new delay to a removal that is still pending. The module
+	 * reference taken for it carries over to the requeued work.
+	 */
+	if (cancel_delayed_work(&head->remove_work))
+		queue_delayed_work(nvme_wq, &head->remove_work, sec * HZ);
 	mutex_unlock(&head->subsys->lock);
 	/*
 	 * Ensure that update to NVME_NSHEAD_QUEUE_IF_NO_PATH is seen
Signature · 1 line
2.43.0
#2 lore raw

Re: [PATCH v5 1/2] nvme-multipath: apply delayed_removal_secs changes to a pending removal

Thank you for your contribution! Sashiko AI review found 2 potential issues to consider: - [Medium] nvme-multipath: clearing delayed_removal_secs fails to kick requeue_work - [Low] nvme-multipath…

Thank you for your contribution! Sashiko AI review found 2 potential issues to consider:
- [Medium] nvme-multipath: clearing delayed_removal_secs fails to kick requeue_work
- [Low] nvme-multipath: 32-bit overflow in sec * HZ in delayed_removal_secs_store()
--

commit 6d5b135aa1c8a348dae91223319bfb27c4526bcb
Author: Qiliang Yuan <[email protected]>

nvme-multipath: apply delayed_removal_secs changes to a pending removal

This commit updates delayed_removal_secs_store() to cancel and requeue any pending head->remove_work with the new delay, and moves the remove_work cancellation in nvme_init_ns_head() under subsys->lock.

Quoted 5 lines
drivers/nvme/host/multipath.c +0 -0
index ae8059366b57d..51c59cc3ea246 100644

drivers/nvme/host/multipath.c static ssize_t delayed_removal_secs_store(struct device *dev, · line 1250

Quoted 9 lines
drivers/nvme/host/multipath.c +6 -0
@@ -1242,6 +1242,12 @@ static ssize_t delayed_removal_secs_store(struct device *dev,
 	mutex_lock(&head->subsys->lock);
 	head->delayed_removal_secs = sec;
 	assign_bit(NVME_NSHEAD_QUEUE_IF_NO_PATH, &head->flags, sec);
+	/*
+	 * Apply the new delay to a removal that is still pending. The module
+	 * reference taken for it carries over to the requeued work.
+	 */
+	if (cancel_delayed_work(&head->remove_work))
+		queue_delayed_work(nvme_wq, &head->remove_work, sec * HZ);

[Severity: Low]
Can sec * HZ overflow 32-bit unsigned int arithmetic before being widened to unsigned long? sec is parsed with kstrtouint() without an upper bound (and nvme_mpath_remove_disk() does the same head->delayed_removal_secs * HZ multiplication), so a value above UINT_MAX / HZ wraps the delay and can remove the head right away even though NVME_NSHEAD_QUEUE_IF_NO_PATH is set. Using secs_to_jiffies() or clamping sec would avoid the wrap.

drivers/nvme/host/multipath.c static ssize_t delayed_removal_secs_store(struct device *dev, · line 1253

Quoted 3 lines
drivers/nvme/host/multipath.c +0 -0
 	mutex_unlock(&head->subsys->lock);
 	/*
 	 * Ensure that update to NVME_NSHEAD_QUEUE_IF_NO_PATH is seen

[Severity: Medium]
When sec is set to 0 while head->remove_work is not pending, does this leave bios that were already queued on head->requeue_list stuck?

If paths are still present on head->list in NVME_ANA_INACCESSIBLE or NVME_ANA_PERSISTENT_LOSS states, remove_work is not queued, yet nvme_available_path() returns nvme_mpath_queue_if_no_path(head) and nvme_ns_head_submit_bio() parks incoming bios on head->requeue_list while NVME_NSHEAD_QUEUE_IF_NO_PATH is set. Clearing delayed_removal_secs to 0 clears NVME_NSHEAD_QUEUE_IF_NO_PATH, but without scheduling head->requeue_work after synchronize_srcu(&head->srcu) (as fail_on_ctrl_loss_store() does), nothing drains and fails the bios already on head->requeue_list.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/20261119-bug-nvme-mpath-set-live-srcu-v5-0-e1483e2e447f@linux.dev?part=1
Reviewed-by on v5 1/2
#3 lore raw

Re: [PATCH v5 1/2] nvme-multipath: apply delayed_removal_secs changes to a pending removal

In reply to Qiliang Yuan
Quoted 26 lines · Qiliang Yuan

With delayed_removal_secs set, removing the last path of a namespace head doesn't delete the head right away. nvme_mpath_remove_disk() queues remove_work with the configured delay, and bios submitted in the meantime wait on head->requeue_list in case a path comes back.

A write to delayed_removal_secs only takes effect for removals that start later. A removal that is already pending keeps the old delay, so clearing the attribute leaves the queued bios waiting until that delay expires instead of failing them, and lowering or raising it doesn't move the pending removal either.

Requeue a pending remove_work with the new delay when the attribute is written, cancelling it first so that the module reference taken for it carries over. Have nvme_init_ns_head() cancel the pending removal before it drops subsys->lock rather than after, so that a write to the attribute cannot requeue the removal once a path is back.

With delayed_removal_secs set and no path left, the attribute is changed two seconds later and the head is removed after (mean of two runs, min-max in parentheses):

   delay (s)  before (ms)            after (ms)             change
   30 -> 0    26655 (26537-26772)        8 (8-8)            -99.97%
   30 -> 5    28298 (28094-28501)     5514 (5510-5517)      -81%
   10 -> 30    8112 (8106-8117)      30376 (30106-30646)    +274%
Quoted 8 lines

A direct read queued before the 30 -> 0 write fails after:

   before (ms)            after (ms)   change
   26607 (26450-26764)    4 (4-4)      -99.98%
Fixes 62188639ec160 ("nvme-multipath: introduce delayed removal of the multipath head node")

Looks good to me:

#5 lore raw

PATCH v5 2/2 nvme-multipath: don't wait for a grace period in nvme_mpath_set_live()

In reply to Qiliang Yuan

When a path goes live, nvme_mpath_set_live() waits for an SRCU grace period of the namespace head before kicking the requeue work, so that bios which found no usable path are on the requeue list by the time it runs.

The ANA work calls nvme_mpath_set_live() for every namespace that becomes optimized or non-optimized, one after another, so an ANA change waits for one grace period per namespace.

Start the grace period with start_poll_synchronize_srcu() instead and have nvme_requeue_work() wait for it only if it has not completed yet. Take the cookie under head->lock, so that concurrent callers store their cookies in order and the requeue work never sees an older one. The grace periods of all namespaces then run in parallel, and requeue work kicked for other reasons finds the cookie completed and does not wait.

Flipping an ANA group from inaccessible to optimized ten times on each of two boots over nvme-tcp loopback, nvme_read_ana_log() takes (median, min-max in parentheses):

  namespaces  before (ms)          after (ms)        change
         256  52.0 (35.4-74.7)     8.7 (2.5-19.6)      -83%
        1024  154.3 (123.9-193.8)  43.1 (18.2-58.3)    -72%
---
 drivers/nvme/host/multipath.c | 15 ++++++++++++++-
 drivers/nvme/host/nvme.h      |  1 +
 2 files changed, 15 insertions(+), 1 deletion(-)
drivers/nvme/host/multipath.c +14 -1
index 51c59cc3ea246..ed930dd2af24d 100644
@@ -712,6 +712,11 @@ static void nvme_requeue_work(struct work_struct *work)
 		container_of(work, struct nvme_ns_head, requeue_work);
 	struct bio *bio, *next;
 
+	/* Let bios that found no usable path reach the requeue list first. */
+	if (!poll_state_synchronize_srcu(&head->srcu,
+					 READ_ONCE(head->requeue_srcu_cookie)))
+		synchronize_srcu(&head->srcu);
+
 	spin_lock_irq(&head->requeue_lock);
 	next = bio_list_get(&head->requeue_list);
 	spin_unlock_irq(&head->requeue_lock);
@@ -767,6 +772,7 @@ int nvme_mpath_alloc_disk(struct nvme_ctrl *ctrl, struct nvme_ns_head *head)
 	mutex_init(&head->lock);
 	spin_lock_init(&head->requeue_lock);
 	INIT_WORK(&head->requeue_work, nvme_requeue_work);
+	head->requeue_srcu_cookie = get_completed_synchronize_srcu();
 	INIT_WORK(&head->partition_scan_work, nvme_partition_scan_work);
 	INIT_DELAYED_WORK(&head->remove_work, nvme_remove_head_work);
 
@@ -849,9 +855,16 @@ static void nvme_mpath_set_live(struct nvme_ns *ns)
 			__nvme_find_path(head, node);
 		srcu_read_unlock(&head->srcu, srcu_idx);
 	}
+	/*
+	 * Have nvme_requeue_work() wait for the grace period instead of
+	 * blocking here, so that namespaces going live together, e.g. on
+	 * an ANA change, do not wait for one grace period each. Start it
+	 * under head->lock so that the newest cookie is stored last.
+	 */
+	WRITE_ONCE(head->requeue_srcu_cookie,
+		   start_poll_synchronize_srcu(&head->srcu));
 	mutex_unlock(&head->lock);
 
-	synchronize_srcu(&head->srcu);
 	nvme_mpath_revalidate_zones(head);
 	kblockd_schedule_work(&head->requeue_work);
 }
drivers/nvme/host/nvme.h +1 -0
index 8833a463f1c84..3fff9b394b1cd 100644
@@ -587,6 +587,7 @@ struct nvme_ns_head {
 		__guarded_by(&requeue_lock);
 	spinlock_t		requeue_lock;
 	struct work_struct	requeue_work;
+	unsigned long		requeue_srcu_cookie;
 	struct work_struct	partition_scan_work;
 	struct mutex		lock;
 	unsigned long		flags;
Signature · 1 line
2.43.0
maintainer Reviewed-by on v5 2/2
#6 lore raw

Re: [PATCH v5 2/2] nvme-multipath: don't wait for a grace period in nvme_mpath_set_live()

drivers/nvme/host/multipath.c static void nvme_requeue_work(struct work_struct *work) · line 717

Quoted 9 lines
drivers/nvme/host/multipath.c +3 -0
@@ -712,6 +712,11 @@ static void nvme_requeue_work(struct work_struct *work)
 		container_of(work, struct nvme_ns_head, requeue_work);
 	struct bio *bio, *next;
 
+	/* Let bios that found no usable path reach the requeue list first. */
+	if (!poll_state_synchronize_srcu(&head->srcu,
+					 READ_ONCE(head->requeue_srcu_cookie)))

Let's use a two-tab indent here to make this a bit more readable..

Otherwise looks good: