nvme-multipath: requeue work fixes
v5 cover nvme-multipath: requeue work fixes
Patch 1 applies a new delayed_removal_secs to a head removal that is already pending, so that clearing it fails the I/O queued without a path right away instead of when the old delay expires.
Patch 2 stops nvme_mpath_set_live() from waiting for an SRCU grace period per namespace, which makes an ANA change with many namespaces take tens to hundreds of milliseconds.
---
V4 -> V5:
- Cancel the pending head removal under subsys->lock in nvme_init_ns_head() and drop the list_empty() check and module_put() from delayed_removal_secs_store() (Nilay Shroff)
- Rebase onto v7.4-rc3-230-g4405c18c95d2 and update the numbers of patch 1 from a new run
V3 -> V4:
- Don't requeue the head removal in patch 1 if a path came back before nvme_init_ns_head() cancelled it, drop the module reference it held instead (race reported by Sashiko)
- Rebase onto v7.4-rc3-178-g20eef29b7c84 and update the numbers of patch 1 from a new run
V2 -> V3:
- Requeue a pending head removal with the new delayed_removal_secs instead of only kicking the requeue work when it is cleared
- Rebase onto v7.4-rc2-470-g7e1392a42ea2 and update the numbers from a new run
- Don't carry Hannes's Reviewed-by on the v2 patch 1, which is rewritten
V1 -> V2:
- Add patch 1
- Take the SRCU cookie under head->lock in patch 2 (Sashiko)
- Rerun the tests
v4: https://lore.kernel.org/r/20261118-bug-nvme-mpath-set-live-srcu-v4-0-21f368376c8c@linux.dev
v3: https://lore.kernel.org/r/20261114-bug-nvme-mpath-set-live-srcu-v3-0-d84651de6334@linux.dev
v2: https://lore.kernel.org/r/20261111-bug-nvme-mpath-set-live-srcu-v2-0-fc772914d293@linux.dev
v1: https://lore.kernel.org/r/20261110-bug-nvme-mpath-set-live-srcu-v1-1-1f58b405b0f8@linux.dev
---
Qiliang Yuan (2):
nvme-multipath: apply delayed_removal_secs changes to a pending removal
nvme-multipath: don't wait for a grace period in nvme_mpath_set_live()
drivers/nvme/host/core.c | 7 +++++-- drivers/nvme/host/multipath.c | 21 ++++++++++++++++++++- drivers/nvme/host/nvme.h | 1 + 3 files changed, 26 insertions(+), 3 deletions(-)
---
base-commit: 4405c18c95d2d0f99f6c581ef0ee8591a1aa6ccd
change-id: 20261110-bug-nvme-mpath-set-live-srcu-cbd8965f1457
Best regards,
Signature · 1 line
Qiliang Yuan <[email protected]>
PATCH v5 1/2 nvme-multipath: apply delayed_removal_secs changes to a pending removal
With delayed_removal_secs set, removing the last path of a namespace head doesn't delete the head right away. nvme_mpath_remove_disk() queues remove_work with the configured delay, and bios submitted in the meantime wait on head->requeue_list in case a path comes back.
A write to delayed_removal_secs only takes effect for removals that start later. A removal that is already pending keeps the old delay, so clearing the attribute leaves the queued bios waiting until that delay expires instead of failing them, and lowering or raising it doesn't move the pending removal either.
Requeue a pending remove_work with the new delay when the attribute is written, cancelling it first so that the module reference taken for it carries over. Have nvme_init_ns_head() cancel the pending removal before it drops subsys->lock rather than after, so that a write to the attribute cannot requeue the removal once a path is back.
With delayed_removal_secs set and no path left, the attribute is changed two seconds later and the head is removed after (mean of two runs, min-max in parentheses):
delay (s) before (ms) after (ms) change 30 -> 0 26655 (26537-26772) 8 (8-8) -99.97% 30 -> 5 28298 (28094-28501) 5514 (5510-5517) -81% 10 -> 30 8112 (8106-8117) 30376 (30106-30646) +274%
A direct read queued before the 30 -> 0 write fails after:
before (ms) after (ms) change 26607 (26450-26764) 4 (4-4) -99.98%
--- drivers/nvme/host/core.c | 7 +++++-- drivers/nvme/host/multipath.c | 6 ++++++ 2 files changed, 11 insertions(+), 2 deletions(-)
drivers/nvme/host/core.c +5 -2
@@ -4231,12 +4231,15 @@ static int nvme_init_ns_head(struct nvme_ns *ns, struct nvme_ns_info *info)list_add_tail_rcu (&ns->siblings, &head->list); ns->head = head; -mutex_unlock (&ctrl->subsys->lock); -#ifdef CONFIG_NVME_MULTIPATH +/* +* Cancel a pending head removal under subsys->lock, so that +* delayed_removal_secs_store() never requeues it once a path is back. +*/ if (cancel_delayed_work (&head->remove_work))module_put (THIS_MODULE );#endif +mutex_unlock (&ctrl->subsys->lock);return 0 ; out_put_ns_head:
drivers/nvme/host/multipath.c +6 -0
@@ -1242,6 +1242,12 @@ static ssize_t delayed_removal_secs_store(struct device *dev,mutex_lock (&head->subsys->lock); head->delayed_removal_secs = sec;assign_bit (NVME_NSHEAD_QUEUE_IF_NO_PATH , &head->flags, sec); +/* +* Apply the new delay to a removal that is still pending. The module +* reference taken for it carries over to the requeued work. +*/ +if (cancel_delayed_work (&head->remove_work)) +queue_delayed_work (nvme_wq, &head->remove_work, sec *HZ );mutex_unlock (&head->subsys->lock);/* * Ensure that update to NVME_NSHEAD_QUEUE_IF_NO_PATH is seen
Signature · 1 line
2.43.0
[email protected]
AI reviewRe: [PATCH v5 1/2] nvme-multipath: apply delayed_removal_secs changes to a pending removal
Thank you for your contribution! Sashiko AI review found 2 potential issues to consider: - [Medium] nvme-multipath: clearing delayed_removal_secs fails to kick requeue_work - [Low] nvme-multipath…
Thank you for your contribution! Sashiko AI review found 2 potential issues to consider:
- [Medium] nvme-multipath: clearing delayed_removal_secs fails to kick requeue_work
- [Low] nvme-multipath: 32-bit overflow in sec * HZ in delayed_removal_secs_store()
--
commit 6d5b135aa1c8a348dae91223319bfb27c4526bcb
Author: Qiliang Yuan <[email protected]>
nvme-multipath: apply delayed_removal_secs changes to a pending removal
This commit updates delayed_removal_secs_store() to cancel and requeue any pending head->remove_work with the new delay, and moves the remove_work cancellation in nvme_init_ns_head() under subsys->lock.
Quoted 5 lines
drivers/nvme/host/multipath.c +0
-0
drivers/nvme/host/multipath.c static ssize_t delayed_removal_secs_store(struct device *dev, · line 1250
Quoted 9 lines
drivers/nvme/host/multipath.c +6
-0@@ -1242,6 +1242,12 @@ static ssize_t delayed_removal_secs_store(struct device *dev,mutex_lock (&head->subsys->lock); head->delayed_removal_secs = sec;assign_bit (NVME_NSHEAD_QUEUE_IF_NO_PATH , &head->flags, sec); +/* +* Apply the new delay to a removal that is still pending. The module +* reference taken for it carries over to the requeued work. +*/ +if (cancel_delayed_work (&head->remove_work)) +queue_delayed_work (nvme_wq, &head->remove_work, sec *HZ );
[Severity: Low]
Can sec * HZ overflow 32-bit unsigned int arithmetic before being widened to unsigned long? sec is parsed with kstrtouint() without an upper bound (and nvme_mpath_remove_disk() does the same head->delayed_removal_secs * HZ multiplication), so a value above UINT_MAX / HZ wraps the delay and can remove the head right away even though NVME_NSHEAD_QUEUE_IF_NO_PATH is set. Using secs_to_jiffies() or clamping sec would avoid the wrap.
drivers/nvme/host/multipath.c static ssize_t delayed_removal_secs_store(struct device *dev, · line 1253
Quoted 3 lines
drivers/nvme/host/multipath.c +0
-0mutex_unlock (&head->subsys->lock);/* * Ensure that update to NVME_NSHEAD_QUEUE_IF_NO_PATH is seen
[Severity: Medium]
When sec is set to 0 while head->remove_work is not pending, does this leave bios that were already queued on head->requeue_list stuck?
If paths are still present on head->list in NVME_ANA_INACCESSIBLE or NVME_ANA_PERSISTENT_LOSS states, remove_work is not queued, yet nvme_available_path() returns nvme_mpath_queue_if_no_path(head) and nvme_ns_head_submit_bio() parks incoming bios on head->requeue_list while NVME_NSHEAD_QUEUE_IF_NO_PATH is set. Clearing delayed_removal_secs to 0 clears NVME_NSHEAD_QUEUE_IF_NO_PATH, but without scheduling head->requeue_work after synchronize_srcu(&head->srcu) (as fail_on_ctrl_loss_store() does), nothing drains and fails the bios already on head->requeue_list.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/20261119-bug-nvme-mpath-set-live-srcu-v5-0-e1483e2e447f@linux.dev?part=1
Re: [PATCH v5 1/2] nvme-multipath: apply delayed_removal_secs changes to a pending removal
In reply to Qiliang YuanQuoted 26 lines · Qiliang Yuan
With
delayed_removal_secsset, removing the last path of a namespace head doesn't delete the head right away.nvme_mpath_remove_disk()queuesremove_workwith the configured delay, and bios submitted in the meantime wait onhead->requeue_listin case a path comes back.A write to
delayed_removal_secsonly takes effect for removals that start later. A removal that is already pending keeps the old delay, so clearing the attribute leaves the queued bios waiting until that delay expires instead of failing them, and lowering or raising it doesn't move the pending removal either.Requeue a pending
remove_workwith the new delay when the attribute is written, cancelling it first so that the module reference taken for it carries over. Havenvme_init_ns_head()cancel the pending removal before it dropssubsys->lockrather than after, so that a write to the attribute cannot requeue the removal once a path is back.With
delayed_removal_secsset and no path left, the attribute is changed two seconds later and the head is removed after (mean of two runs, min-max in parentheses):delay (s) before (ms) after (ms) change 30 -> 0 26655 (26537-26772) 8 (8-8) -99.97% 30 -> 5 28298 (28094-28501) 5514 (5510-5517) -81% 10 -> 30 8112 (8106-8117) 30376 (30106-30646) +274%
Quoted 8 lines
A direct read queued before the 30 -> 0 write fails after:
before (ms) after (ms) change 26607 (26450-26764) 4 (4-4) -99.98%Fixes 62188639ec160 ("nvme-multipath: introduce delayed removal of the multipath head node")Suggested-by Nilay Shroff<[email protected]>Signed-off-by Qiliang Yuan<[email protected]>
Looks good to me:
Re: [PATCH v5 1/2] nvme-multipath: apply delayed_removal_secs changes to a pending removal
In reply to Qiliang YuanLooks good:
PATCH v5 2/2 nvme-multipath: don't wait for a grace period in nvme_mpath_set_live()
In reply to Qiliang YuanWhen a path goes live, nvme_mpath_set_live() waits for an SRCU grace period of the namespace head before kicking the requeue work, so that bios which found no usable path are on the requeue list by the time it runs.
The ANA work calls nvme_mpath_set_live() for every namespace that becomes optimized or non-optimized, one after another, so an ANA change waits for one grace period per namespace.
Start the grace period with start_poll_synchronize_srcu() instead and have nvme_requeue_work() wait for it only if it has not completed yet. Take the cookie under head->lock, so that concurrent callers store their cookies in order and the requeue work never sees an older one. The grace periods of all namespaces then run in parallel, and requeue work kicked for other reasons finds the cookie completed and does not wait.
Flipping an ANA group from inaccessible to optimized ten times on each of two boots over nvme-tcp loopback, nvme_read_ana_log() takes (median, min-max in parentheses):
namespaces before (ms) after (ms) change
256 52.0 (35.4-74.7) 8.7 (2.5-19.6) -83%
1024 154.3 (123.9-193.8) 43.1 (18.2-58.3) -72%--- drivers/nvme/host/multipath.c | 15 ++++++++++++++- drivers/nvme/host/nvme.h | 1 + 2 files changed, 15 insertions(+), 1 deletion(-)
drivers/nvme/host/multipath.c +14 -1
@@ -712,6 +712,11 @@ static void nvme_requeue_work(struct work_struct *work) container_of(work, struct nvme_ns_head, requeue_work);struct bio *bio, *next; +/* Let bios that found no usable path reach the requeue list first. */ +if (!poll_state_synchronize_srcu (&head->srcu, +READ_ONCE (head->requeue_srcu_cookie))) +synchronize_srcu (&head->srcu); +spin_lock_irq (&head->requeue_lock); next =bio_list_get (&head->requeue_list);spin_unlock_irq (&head->requeue_lock);
@@ -767,6 +772,7 @@ int nvme_mpath_alloc_disk(struct nvme_ctrl *ctrl, struct nvme_ns_head *head)mutex_init (&head->lock);spin_lock_init (&head->requeue_lock);INIT_WORK (&head->requeue_work, nvme_requeue_work); + head->requeue_srcu_cookie =get_completed_synchronize_srcu ();INIT_WORK (&head->partition_scan_work, nvme_partition_scan_work);INIT_DELAYED_WORK (&head->remove_work, nvme_remove_head_work);
@@ -849,9 +855,16 @@ static void nvme_mpath_set_live(struct nvme_ns *ns)__nvme_find_path (head, node);srcu_read_unlock (&head->srcu, srcu_idx); } +/* +* Have nvme_requeue_work() wait for the grace period instead of +* blocking here, so that namespaces going live together, e.g. on +* an ANA change, do not wait for one grace period each. Start it +* under head->lock so that the newest cookie is stored last. +*/ +WRITE_ONCE (head->requeue_srcu_cookie, +start_poll_synchronize_srcu (&head->srcu));mutex_unlock (&head->lock); -synchronize_srcu (&head->srcu);nvme_mpath_revalidate_zones (head);kblockd_schedule_work (&head->requeue_work); }
drivers/nvme/host/nvme.h +1 -0
@@ -587,6 +587,7 @@ struct nvme_ns_head {__guarded_by (&requeue_lock);spinlock_t requeue_lock;struct work_struct requeue_work; +unsigned long requeue_srcu_cookie;struct work_struct partition_scan_work;struct mutex lock;unsigned long flags;
Signature · 1 line
2.43.0
Re: [PATCH v5 2/2] nvme-multipath: don't wait for a grace period in nvme_mpath_set_live()
drivers/nvme/host/multipath.c static void nvme_requeue_work(struct work_struct *work) · line 717
Quoted 9 lines
drivers/nvme/host/multipath.c +3
-0@@ -712,6 +712,11 @@ static void nvme_requeue_work(struct work_struct *work) container_of(work, struct nvme_ns_head, requeue_work);struct bio *bio, *next; +/* Let bios that found no usable path reach the requeue list first. */ +if (!poll_state_synchronize_srcu (&head->srcu, +READ_ONCE (head->requeue_srcu_cookie)))
Let's use a two-tab indent here to make this a bit more readable..
Otherwise looks good: