Skip to contentSkip to conversation

improve NVMe multipath handling

RFC PATCH Superseded Nilay Shroff 3 participants 13 messages May 4, 2025 – May 8, 2025 NVMe

v3 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

v3 cover improve NVMe multipath handling

Hi,

This patch series introduces improvements to NVMe multipath handling by refining the removal behavior of the multipath head node and simplifying configuration options. The idea/POC for this change was originally proposed by Christoph[1] and Keith[2]. I worked upon their original idea/POC and implemented this series.

The first patch in the series addresses an issue where the multipath head node of a PCIe NVMe disk is removed immediately when all disk paths are lost. This can cause problems in scenarios such as:
- Hot removal and re-addition of a disk.
- Transient PCIe link failures that trigger re-enumeration, briefly removing and restoring the disk.

In such cases, premature removal of the head node may result in a device node name change, requiring applications to reopen device handles if they were performing I/O during the failure. To mitigate this, we introduce a delayed removal mechanism. Instead of removing the head node immediately, the system waits for a configurable timeout, allowing the disk to recover. If the disk comes back online within this window, the head node remains unchanged, ensuring uninterrupted workloads.

A new sysfs attribute, delayed_removal_secs, allows users to configure this timeout. By default, it is set to 0 seconds, preserving the existing behavior unless explicitly changed.

The second patch in the series introduced multipath_always_on module param. When this option is set, it forces creating multipath head disk node even for single ported NVMe disks or private namespaces and thus allows delayed head node removal. This would help handle transient PCIe link failures transparently even in case of single ported NVMe disk or a private namespace.

The third patch in the series doesn't make any functional changes but just renames few of the function name which improves code readability and it better aligns function names with their actual roles.

These changes should help improve NVMe multipath reliability and simplify configuration. Feedback and testing are welcome!

[1] https://lore.kernel.org/linux-nvme/[email protected]/
[2] https://lore.kernel.org/linux-nvme/[email protected]/

Changes from v2:
- Rename multipath_head_always to multipath_always_on (Hannes Reinecke)
- Map delayed_removal_secs to queue_if_no_path internally; if delayed_ removal_secs is non-zero then queue_if_no_path is set otherwise its unset (Hannes Reinecke)
- Few minor code readability improvements in the second patch while handling multipath_param_set and multipath_always_on_set (hch)
- Avoid the race in shutdown namespace removal by deleting head->entry during the first critical section of the nvme_ns_remove for the case head delayed_removal is not configured (hch)
- Use ctrl->ops->flags & NVME_F_FABRICS to determine whether the ctrl uses fabric setup (Sagi)

Link to v2: https://lore.kernel.org/all/[email protected]/

Changes from v1:
- Renamed delayed_shutdown_sec to delayed_removal_secs as "shutdown" has a special meaning when used with NVMe device (Martin Petersen)
- Instead of adding mpath head disk node always by default, added new module option nvme_core.multipath_head_always which when set creates mpath head disk node (even for a private namespace or a namespace backed by single ported nvme disk). This way we can preserve the default old behavior.(hch)
- Renamed nvme_mpath_shutdown_disk function as shutdown as in the NVMe context, the term "shutdown" has a specific technical meaning. (hch)
- Undo changes which removed multipath module param as this param is still useful and used for many different things.

Link to v1: https://lore.kernel.org/all/[email protected]/

Nilay Shroff (3):
nvme-multipath: introduce delayed removal of the multipath head node
nvme: introduce multipath_always_on module param
nvme: rename nvme_mpath_shutdown_disk to nvme_mpath_remove_disk

 drivers/nvme/host/core.c      |  14 ++-
 drivers/nvme/host/multipath.c | 213 +++++++++++++++++++++++++++++++---
 drivers/nvme/host/nvme.h      |  33 +++++-
 drivers/nvme/host/sysfs.c     |  13 +++
 4 files changed, 244 insertions(+), 29 deletions(-)
Signature · 1 line
2.49.0
#1 lore raw

RFC PATCH v3 1/3 nvme-multipath: introduce delayed removal of the multipath head node

Currently, the multipath head node of a PCIe NVMe disk is removed immediately as soon as all paths of the disk are removed. However, this can cause issues in scenarios where:

- The disk hot-removal followed by re-addition.
- Transient PCIe link failures that trigger re-enumeration, temporarily removing and then restoring the disk.

In these cases, removing the head node prematurely may lead to a head disk node name change upon re-addition, requiring applications to reopen their handles if they were performing I/O during the failure.

To address this, introduce a delayed removal mechanism of head disk node. During transient failure, instead of immediate removal of head disk node, the system waits for a configurable timeout, allowing the disk to recover.

During transient disk failure, if application sends any IO then we queue it instead of failing such IO immediately. If the disk comes back online within the timeout, the queued IOs are resubmitted to the disk ensuring seamless operation. In case disk couldn't recover from the failure then queued IOs are failed to its completion and application receives the error.

So this way, if disk comes back online within the configured period, the head node remains unchanged, ensuring uninterrupted workloads without requiring applications to reopen device handles.

A new sysfs attribute, named "delayed_removal_secs" is added under head disk blkdev for user who wish to configure time for the delayed removal of head disk node. The default value of this attribute is set to zero second ensuring no behavior change unless explicitly configured.

---
 drivers/nvme/host/core.c      |  10 ++-
 drivers/nvme/host/multipath.c | 137 +++++++++++++++++++++++++++++++---
 drivers/nvme/host/nvme.h      |  25 ++++++-
 drivers/nvme/host/sysfs.c     |  13 ++++
 4 files changed, 171 insertions(+), 14 deletions(-)
drivers/nvme/host/core.c +7 -3
index eb6ea8acb3cc..0f6b01f10c70 100644
@@ -3560,7 +3560,7 @@ static struct nvme_ns_head *nvme_find_ns_head(struct nvme_ctrl *ctrl,
 		 */
 		if (h->ns_id != nsid || !nvme_is_unique_nsid(ctrl, h))
 			continue;
-		if (!list_empty(&h->list) && nvme_tryget_ns_head(h))
+		if (nvme_tryget_ns_head(h))
 			return h;
 	}
 
@@ -3688,6 +3688,8 @@ static struct nvme_ns_head *nvme_alloc_ns_head(struct nvme_ctrl *ctrl,
 	ratelimit_state_init(&head->rs_nuse, 5 * HZ, 1);
 	ratelimit_set_flags(&head->rs_nuse, RATELIMIT_MSG_ON_RELEASE);
 	kref_init(&head->ref);
+	if (ctrl->ops->flags & NVME_F_FABRICS)
+		nvme_mpath_set_fabrics(head);
 
 	if (head->ids.csi) {
 		ret = nvme_get_effects_log(ctrl, head->ids.csi, &head->effects);
@@ -3804,7 +3806,8 @@ static int nvme_init_ns_head(struct nvme_ns *ns, struct nvme_ns_info *info)
 		}
 	} else {
 		ret = -EINVAL;
-		if (!info->is_shared || !head->shared) {
+		if ((!info->is_shared || !head->shared) &&
+		    !list_empty(&head->list)) {
 			dev_err(ctrl->device,
 				"Duplicate unshared namespace %d\n",
 				info->nsid);
@@ -4008,7 +4011,8 @@ static void nvme_ns_remove(struct nvme_ns *ns)
 	mutex_lock(&ns->ctrl->subsys->lock);
 	list_del_rcu(&ns->siblings);
 	if (list_empty(&ns->head->list)) {
-		list_del_init(&ns->head->entry);
+		if (!nvme_mpath_queue_if_no_path(ns->head))
+			list_del_init(&ns->head->entry);
 		last_path = true;
 	}
 	mutex_unlock(&ns->ctrl->subsys->lock);
drivers/nvme/host/multipath.c +127 -10

176 lines not on this page · raw message

drivers/nvme/host/nvme.h +24 -1
index 51e078642127..b2d02425dc3c 100644
@@ -503,7 +503,11 @@ struct nvme_ns_head {
 	struct work_struct	partition_scan_work;
 	struct mutex		lock;
 	unsigned long		flags;
-#define NVME_NSHEAD_DISK_LIVE	0
+	struct delayed_work	remove_work;
+	unsigned int		delayed_removal_secs;
+#define NVME_NSHEAD_DISK_LIVE		0
+#define NVME_NSHEAD_FABRICS		1
+#define NVME_NSHEAD_QUEUE_IF_NO_PATH	2
 	struct nvme_ns __rcu	*current_path[];
 #endif
 };
@@ -986,12 +990,23 @@ extern struct device_attribute dev_attr_ana_grpid;
 extern struct device_attribute dev_attr_ana_state;
 extern struct device_attribute dev_attr_queue_depth;
 extern struct device_attribute dev_attr_numa_nodes;
+extern struct device_attribute dev_attr_delayed_removal_secs;
 extern struct device_attribute subsys_attr_iopolicy;
 
 static inline bool nvme_disk_is_ns_head(struct gendisk *disk)
 {
 	return disk->fops == &nvme_ns_head_ops;
 }
+static inline void nvme_mpath_set_fabrics(struct nvme_ns_head *head)
+{
+	set_bit(NVME_NSHEAD_FABRICS, &head->flags);
+}
+static inline bool nvme_mpath_queue_if_no_path(struct nvme_ns_head *head)
+{
+	if (test_bit(NVME_NSHEAD_QUEUE_IF_NO_PATH, &head->flags))
+		return true;
+	return false;
+}
 #else
 #define multipath false
 static inline bool nvme_ctrl_use_ana(struct nvme_ctrl *ctrl)
@@ -1079,6 +1094,14 @@ static inline bool nvme_disk_is_ns_head(struct gendisk *disk)
 {
 	return false;
 }
+static inline void nvme_mpath_set_fabrics(struct nvme_ns_head *head)
+{
+
+}
+static inline bool nvme_mpath_queue_if_no_path(struct nvme_ns_head *head)
+{
+	return false;
+}
 #endif /* CONFIG_NVME_MULTIPATH */
 
 int nvme_ns_get_unique_id(struct nvme_ns *ns, u8 id[16],
drivers/nvme/host/sysfs.c +13 -0
index 6d31226f7a4f..51633670d177 100644
@@ -260,6 +260,7 @@ static struct attribute *nvme_ns_attrs[] = {
 	&dev_attr_ana_state.attr,
 	&dev_attr_queue_depth.attr,
 	&dev_attr_numa_nodes.attr,
+	&dev_attr_delayed_removal_secs.attr,
 #endif
 	&dev_attr_io_passthru_err_log_enabled.attr,
 	NULL,
@@ -296,6 +297,18 @@ static umode_t nvme_ns_attrs_are_visible(struct kobject *kobj,
 		if (nvme_disk_is_ns_head(dev_to_disk(dev)))
 			return 0;
 	}
+	if (a == &dev_attr_delayed_removal_secs.attr) {
+		struct nvme_ns_head *head = dev_to_ns_head(dev);
+		struct gendisk *disk = dev_to_disk(dev);
+
+		/*
+		 * This attribute is only valid for head node and non-fabric
+		 * setup.
+		 */
+		if (!nvme_disk_is_ns_head(disk) ||
+				test_bit(NVME_NSHEAD_FABRICS, &head->flags))
+			return 0;
+	}
 #endif
 	return a->mode;
 }
Signature · 1 line
2.49.0
#2 lore raw

Re: [RFC PATCHv3 1/3] nvme-multipath: introduce delayed removal of the multipath head node

Quoted 61 lines · Nilay Shroff

Currently, the multipath head node of a PCIe NVMe disk is removed immediately as soon as all paths of the disk are removed. However, this can cause issues in scenarios where:

- The disk hot-removal followed by re-addition.
- Transient PCIe link failures that trigger re-enumeration, temporarily removing and then restoring the disk.

In these cases, removing the head node prematurely may lead to a head disk node name change upon re-addition, requiring applications to reopen their handles if they were performing I/O during the failure.

To address this, introduce a delayed removal mechanism of head disk node. During transient failure, instead of immediate removal of head disk node, the system waits for a configurable timeout, allowing the disk to recover.

During transient disk failure, if application sends any IO then we queue it instead of failing such IO immediately. If the disk comes back online within the timeout, the queued IOs are resubmitted to the disk ensuring seamless operation. In case disk couldn't recover from the failure then queued IOs are failed to its completion and application receives the error.

So this way, if disk comes back online within the configured period, the head node remains unchanged, ensuring uninterrupted workloads without requiring applications to reopen device handles.

A new sysfs attribute, named "delayed_removal_secs" is added under head disk blkdev for user who wish to configure time for the delayed removal of head disk node. The default value of this attribute is set to zero second ensuring no behavior change unless explicitly configured.

---
  drivers/nvme/host/core.c      |  10 ++-
  drivers/nvme/host/multipath.c | 137 +++++++++++++++++++++++++++++++---
  drivers/nvme/host/nvme.h      |  25 ++++++-
  drivers/nvme/host/sysfs.c     |  13 ++++
  4 files changed, 171 insertions(+), 14 deletions(-)
drivers/nvme/host/core.c +1 -1
index eb6ea8acb3cc..0f6b01f10c70 100644
@@ -3560,7 +3560,7 @@ static struct nvme_ns_head *nvme_find_ns_head(struct nvme_ctrl *ctrl,
  		 */
  		if (h->ns_id != nsid || !nvme_is_unique_nsid(ctrl, h))
  			continue;
-		if (!list_empty(&h->list) && nvme_tryget_ns_head(h))
+		if (nvme_tryget_ns_head(h))
  			return h;
  	}
 

drivers/nvme/host/core.c static struct nvme_ns_head *nvme_alloc_ns_head(struct nvme_ctrl *ctrl, · line 3693

Quoted 6 lines
drivers/nvme/host/core.c +2 -0
@@ -3688,6 +3688,8 @@ static struct nvme_ns_head *nvme_alloc_ns_head(struct nvme_ctrl *ctrl,
  	ratelimit_state_init(&head->rs_nuse, 5 * HZ, 1);
  	ratelimit_set_flags(&head->rs_nuse, RATELIMIT_MSG_ON_RELEASE);
  	kref_init(&head->ref);
+	if (ctrl->ops->flags & NVME_F_FABRICS)
+		nvme_mpath_set_fabrics(head);
 

Please make this a separate patch.

Quoted 68 lines
drivers/nvme/host/core.c +4 -2
  	if (head->ids.csi) {
  		ret = nvme_get_effects_log(ctrl, head->ids.csi, &head->effects);
@@ -3804,7 +3806,8 @@ static int nvme_init_ns_head(struct nvme_ns *ns, struct nvme_ns_info *info)
  		}
  	} else {
  		ret = -EINVAL;
-		if (!info->is_shared || !head->shared) {
+		if ((!info->is_shared || !head->shared) &&
+		    !list_empty(&head->list)) {
  			dev_err(ctrl->device,
  				"Duplicate unshared namespace %d\n",
  				info->nsid);
@@ -4008,7 +4011,8 @@ static void nvme_ns_remove(struct nvme_ns *ns)
  	mutex_lock(&ns->ctrl->subsys->lock);
  	list_del_rcu(&ns->siblings);
  	if (list_empty(&ns->head->list)) {
-		list_del_init(&ns->head->entry);
+		if (!nvme_mpath_queue_if_no_path(ns->head))
+			list_del_init(&ns->head->entry);
  		last_path = true;
  	}
  	mutex_unlock(&ns->ctrl->subsys->lock);
drivers/nvme/host/multipath.c +30 -1
index 250f3da67cc9..f3f981c9f3ec 100644
@@ -442,7 +442,24 @@ static bool nvme_available_path(struct nvme_ns_head *head)
  			break;
  		}
  	}
-	return false;
+
+	/*
+	 * If "head->delayed_removal_secs" is configured (i.e., non-zero), do
+	 * not immediately fail I/O. Instead, requeue the I/O for the configured
+	 * duration, anticipating that if there's a transient link failure then
+	 * it may recover within this time window. This parameter is exported to
+	 * userspace via sysfs, and its default value is zero. It is internally
+	 * mapped to NVME_NSHEAD_QUEUE_IF_NO_PATH. When delayed_removal_secs is
+	 * non-zero, this flag is set to true. When zero, the flag is cleared.
+	 *
+	 * Note: This option is not exposed to the users for NVMeoF connections.
+	 * In fabric-based setups, fabric link failure is managed through other
+	 * parameters such as "reconnect_delay" and "max_reconnects" which is
+	 * handled at respective fabric driver layer. Therefor, head->delayed_
+	 * removal_secs" is intended exclusively for non-fabric (e.g., PCIe)
+	 * multipath configurations.
+	 */
+	return nvme_mpath_queue_if_no_path(head);
  }
 
  static void nvme_ns_head_submit_bio(struct bio *bio)
@@ -617,6 +634,40 @@ static void nvme_requeue_work(struct work_struct *work)
  	}
  }
 
+static void nvme_remove_head(struct nvme_ns_head *head)
+{
+	if (test_and_clear_bit(NVME_NSHEAD_DISK_LIVE, &head->flags)) {
+		/*
+		 * requeue I/O after NVME_NSHEAD_DISK_LIVE has been cleared
+		 * to allow multipath to fail all I/O.
+		 */
+		kblockd_schedule_work(&head->requeue_work);
+
+		nvme_cdev_del(&head->cdev, &head->cdev_device);
+		synchronize_srcu(&head->srcu);
+		del_gendisk(head->disk);

drivers/nvme/host/multipath.c static void nvme_requeue_work(struct work_struct *work) · line 658

Quoted 10 lines
drivers/nvme/host/multipath.c +10 -0
+		nvme_put_ns_head(head);
+	}
+}
+
+static void nvme_remove_head_work(struct work_struct *work)
+{
+	struct nvme_ns_head *head = container_of(to_delayed_work(work),
+			struct nvme_ns_head, remove_work);
+	bool shutdown = false;
+

Don't you need to check here if the 'QUEUE_IF_NO_PATH' bit is still cleared?

Quoted 25 lines
drivers/nvme/host/multipath.c +14 -0
+	mutex_lock(&head->subsys->lock);
+	if (list_empty(&head->list)) {
+		list_del_init(&head->entry);
+		shutdown = true;
+	}
+	mutex_unlock(&head->subsys->lock);
+	if (shutdown)
+		nvme_remove_head(head);
+
+	module_put(THIS_MODULE);
+}
+
  int nvme_mpath_alloc_disk(struct nvme_ctrl *ctrl, struct nvme_ns_head *head)
  {
  	struct queue_limits lim;
@@ -626,6 +677,8 @@ int nvme_mpath_alloc_disk(struct nvme_ctrl *ctrl, struct nvme_ns_head *head)
  	spin_lock_init(&head->requeue_lock);
  	INIT_WORK(&head->requeue_work, nvme_requeue_work);
  	INIT_WORK(&head->partition_scan_work, nvme_partition_scan_work);
+	INIT_DELAYED_WORK(&head->remove_work, nvme_remove_head_work);
+	head->delayed_removal_secs = 0;
 
  	/*
  	 * Add a multipath node if the subsystems supports multiple controllers.

drivers/nvme/host/multipath.c int nvme_mpath_alloc_disk(struct nvme_ctrl *ctrl, struct nvme_ns_head *head) · line 715

Quoted 4 lines
drivers/nvme/host/multipath.c +1 -0
@@ -659,6 +712,7 @@ int nvme_mpath_alloc_disk(struct nvme_ctrl *ctrl, struct nvme_ns_head *head)
  	set_bit(GD_SUPPRESS_PART_SCAN, &head->disk->state);
  	sprintf(head->disk->disk_name, "nvme%dn%d",
  			ctrl->subsys->instance, head->instance);
+	nvme_tryget_ns_head(head);

What is that doing here?
Why do you tak an additional reference?

Quoted 178 lines
drivers/nvme/host/multipath.c +72 -9
  	return 0;
  }
 
@@ -1015,6 +1069,49 @@ static ssize_t numa_nodes_show(struct device *dev, struct device_attribute *attr
  }
  DEVICE_ATTR_RO(numa_nodes);
 
+static ssize_t delayed_removal_secs_show(struct device *dev,
+		struct device_attribute *attr, char *buf)
+{
+	struct gendisk *disk = dev_to_disk(dev);
+	struct nvme_ns_head *head = disk->private_data;
+	int ret;
+
+	mutex_lock(&head->subsys->lock);
+	ret = sysfs_emit(buf, "%u\n", head->delayed_removal_secs);
+	mutex_unlock(&head->subsys->lock);
+	return ret;
+}
+
+static ssize_t delayed_removal_secs_store(struct device *dev,
+		struct device_attribute *attr, const char *buf, size_t count)
+{
+	struct gendisk *disk = dev_to_disk(dev);
+	struct nvme_ns_head *head = disk->private_data;
+	unsigned int sec;
+	int ret;
+
+	ret = kstrtouint(buf, 0, &sec);
+	if (ret < 0)
+		return ret;
+
+	mutex_lock(&head->subsys->lock);
+	head->delayed_removal_secs = sec;
+	if (sec)
+		set_bit(NVME_NSHEAD_QUEUE_IF_NO_PATH, &head->flags);
+	else
+		clear_bit(NVME_NSHEAD_QUEUE_IF_NO_PATH, &head->flags);
+	mutex_unlock(&head->subsys->lock);
+	/*
+	 * Ensure that update to NVME_NSHEAD_QUEUE_IF_NO_PATH is seen
+	 * by its reader.
+	 */
+	synchronize_srcu(&head->srcu);
+
+	return count;
+}
+
+DEVICE_ATTR_RW(delayed_removal_secs);
+
  static int nvme_lookup_ana_group_desc(struct nvme_ctrl *ctrl,
  		struct nvme_ana_group_desc *desc, void *data)
  {
@@ -1138,18 +1235,38 @@ void nvme_mpath_add_disk(struct nvme_ns *ns, __le32 anagrpid)
 
  void nvme_mpath_shutdown_disk(struct nvme_ns_head *head)
  {
-	if (!head->disk)
-		return;
-	if (test_and_clear_bit(NVME_NSHEAD_DISK_LIVE, &head->flags)) {
-		nvme_cdev_del(&head->cdev, &head->cdev_device);
+	bool shutdown = false;
+
+	mutex_lock(&head->subsys->lock);
+	/*
+	 * We are called when all paths have been removed, and at that point
+	 * head->list is expected to be empty. However, nvme_remove_ns() and
+	 * nvme_init_ns_head() can run concurrently and so if head->delayed_
+	 * removal_secs is configured, it is possible that by the time we reach
+	 * this point, head->list may no longer be empty. Therefore, we recheck
+	 * head->list here. If it is no longer empty then we skip enqueuing the
+	 * delayed head removal work.
+	 */
+	if (!list_empty(&head->list))
+		goto out;
+
+	if (head->delayed_removal_secs) {
  		/*
-		 * requeue I/O after NVME_NSHEAD_DISK_LIVE has been cleared
-		 * to allow multipath to fail all I/O.
+		 * Ensure that no one could remove this module while the head
+		 * remove work is pending.
  		 */
-		synchronize_srcu(&head->srcu);
-		kblockd_schedule_work(&head->requeue_work);
-		del_gendisk(head->disk);
+		if (!try_module_get(THIS_MODULE))
+			goto out;
+		queue_delayed_work(nvme_wq, &head->remove_work,
+				head->delayed_removal_secs * HZ);
+	} else {
+		list_del_init(&head->entry);
+		shutdown = true;
  	}
+out:
+	mutex_unlock(&head->subsys->lock);
+	if (shutdown)
+		nvme_remove_head(head);
  }
 
  void nvme_mpath_remove_disk(struct nvme_ns_head *head)
drivers/nvme/host/nvme.h +24 -1
index 51e078642127..b2d02425dc3c 100644
@@ -503,7 +503,11 @@ struct nvme_ns_head {
  	struct work_struct	partition_scan_work;
  	struct mutex		lock;
  	unsigned long		flags;
-#define NVME_NSHEAD_DISK_LIVE	0
+	struct delayed_work	remove_work;
+	unsigned int		delayed_removal_secs;
+#define NVME_NSHEAD_DISK_LIVE		0
+#define NVME_NSHEAD_FABRICS		1
+#define NVME_NSHEAD_QUEUE_IF_NO_PATH	2
  	struct nvme_ns __rcu	*current_path[];
  #endif
  };
@@ -986,12 +990,23 @@ extern struct device_attribute dev_attr_ana_grpid;
  extern struct device_attribute dev_attr_ana_state;
  extern struct device_attribute dev_attr_queue_depth;
  extern struct device_attribute dev_attr_numa_nodes;
+extern struct device_attribute dev_attr_delayed_removal_secs;
  extern struct device_attribute subsys_attr_iopolicy;
 
  static inline bool nvme_disk_is_ns_head(struct gendisk *disk)
  {
  	return disk->fops == &nvme_ns_head_ops;
  }
+static inline void nvme_mpath_set_fabrics(struct nvme_ns_head *head)
+{
+	set_bit(NVME_NSHEAD_FABRICS, &head->flags);
+}
+static inline bool nvme_mpath_queue_if_no_path(struct nvme_ns_head *head)
+{
+	if (test_bit(NVME_NSHEAD_QUEUE_IF_NO_PATH, &head->flags))
+		return true;
+	return false;
+}
  #else
  #define multipath false
  static inline bool nvme_ctrl_use_ana(struct nvme_ctrl *ctrl)
@@ -1079,6 +1094,14 @@ static inline bool nvme_disk_is_ns_head(struct gendisk *disk)
  {
  	return false;
  }
+static inline void nvme_mpath_set_fabrics(struct nvme_ns_head *head)
+{
+
+}
+static inline bool nvme_mpath_queue_if_no_path(struct nvme_ns_head *head)
+{
+	return false;
+}
  #endif /* CONFIG_NVME_MULTIPATH */
 
  int nvme_ns_get_unique_id(struct nvme_ns *ns, u8 id[16],
drivers/nvme/host/sysfs.c +6 -0
index 6d31226f7a4f..51633670d177 100644
@@ -260,6 +260,7 @@ static struct attribute *nvme_ns_attrs[] = {
  	&dev_attr_ana_state.attr,
  	&dev_attr_queue_depth.attr,
  	&dev_attr_numa_nodes.attr,
+	&dev_attr_delayed_removal_secs.attr,
  #endif
  	&dev_attr_io_passthru_err_log_enabled.attr,
  	NULL,
@@ -296,6 +297,18 @@ static umode_t nvme_ns_attrs_are_visible(struct kobject *kobj,
  		if (nvme_disk_is_ns_head(dev_to_disk(dev)))
  			return 0;
  	}
+	if (a == &dev_attr_delayed_removal_secs.attr) {
+		struct nvme_ns_head *head = dev_to_ns_head(dev);
+		struct gendisk *disk = dev_to_disk(dev);
+
+		/*

drivers/nvme/host/sysfs.c static umode_t nvme_ns_attrs_are_visible(struct kobject *kobj, · line 314

Quoted 10 lines
drivers/nvme/host/sysfs.c +7 -0
+		 * This attribute is only valid for head node and non-fabric
+		 * setup.
+		 */
+		if (!nvme_disk_is_ns_head(disk) ||
+				test_bit(NVME_NSHEAD_FABRICS, &head->flags))
+			return 0;
+	}
  #endif
  	return a->mode;
  }

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
#3 lore raw

Re: [RFC PATCHv3 1/3] nvme-multipath: introduce delayed removal of the multipath head node

Quoted 15 lines · Hannes Reinecke
Quoted 13 lines · Nilay Shroff
drivers/nvme/host/core.c +1 -1
index eb6ea8acb3cc..0f6b01f10c70 100644
@@ -3560,7 +3560,7 @@ static struct nvme_ns_head *nvme_find_ns_head(struct nvme_ctrl *ctrl,
            */
           if (h->ns_id != nsid || !nvme_is_unique_nsid(ctrl, h))
               continue;
-        if (!list_empty(&h->list) && nvme_tryget_ns_head(h))
+        if (nvme_tryget_ns_head(h))
               return h;
       }
   @@ -3688,6 +3688,8 @@ static struct nvme_ns_head *nvme_alloc_ns_head(struct nvme_ctrl *ctrl,
Quoted 8 lines
Quoted 6 lines

      ratelimit_state_init(&head->rs_nuse, 5 * HZ, 1); ratelimit_set_flags(&head->rs_nuse, RATELIMIT_MSG_ON_RELEASE); kref_init(&head->ref);
+    if (ctrl->ops->flags & NVME_F_FABRICS)
+        nvme_mpath_set_fabrics(head);

Please make this a separate patch.

Yes sure, will do it while I spin the next patchset.

Quoted 72 lines
Quoted 72 lines

      if (head->ids.csi) {
          ret = nvme_get_effects_log(ctrl, head->ids.csi, &head->effects);

drivers/nvme/host/core.c +4 -2
@@ -3804,7 +3806,8 @@ static int nvme_init_ns_head(struct nvme_ns *ns, struct nvme_ns_info *info)
           }
       } else {
           ret = -EINVAL;
-        if (!info->is_shared || !head->shared) {
+        if ((!info->is_shared || !head->shared) &&
+            !list_empty(&head->list)) {
               dev_err(ctrl->device,
                   "Duplicate unshared namespace %d\n",
                   info->nsid);
@@ -4008,7 +4011,8 @@ static void nvme_ns_remove(struct nvme_ns *ns)
       mutex_lock(&ns->ctrl->subsys->lock);
       list_del_rcu(&ns->siblings);
       if (list_empty(&ns->head->list)) {
-        list_del_init(&ns->head->entry);
+        if (!nvme_mpath_queue_if_no_path(ns->head))
+            list_del_init(&ns->head->entry);
           last_path = true;
       }
       mutex_unlock(&ns->ctrl->subsys->lock);
drivers/nvme/host/multipath.c +34 -1
index 250f3da67cc9..f3f981c9f3ec 100644
@@ -442,7 +442,24 @@ static bool nvme_available_path(struct nvme_ns_head *head)
               break;
           }
       }
-    return false;
+
+    /*
+     * If "head->delayed_removal_secs" is configured (i.e., non-zero), do
+     * not immediately fail I/O. Instead, requeue the I/O for the configured
+     * duration, anticipating that if there's a transient link failure then
+     * it may recover within this time window. This parameter is exported to
+     * userspace via sysfs, and its default value is zero. It is internally
+     * mapped to NVME_NSHEAD_QUEUE_IF_NO_PATH. When delayed_removal_secs is
+     * non-zero, this flag is set to true. When zero, the flag is cleared.
+     *
+     * Note: This option is not exposed to the users for NVMeoF connections.
+     * In fabric-based setups, fabric link failure is managed through other
+     * parameters such as "reconnect_delay" and "max_reconnects" which is
+     * handled at respective fabric driver layer. Therefor, head->delayed_
+     * removal_secs" is intended exclusively for non-fabric (e.g., PCIe)
+     * multipath configurations.
+     */
+    return nvme_mpath_queue_if_no_path(head);
   }
     static void nvme_ns_head_submit_bio(struct bio *bio)
@@ -617,6 +634,40 @@ static void nvme_requeue_work(struct work_struct *work)
       }
   }
   +static void nvme_remove_head(struct nvme_ns_head *head)
+{
+    if (test_and_clear_bit(NVME_NSHEAD_DISK_LIVE, &head->flags)) {
+        /*
+         * requeue I/O after NVME_NSHEAD_DISK_LIVE has been cleared
+         * to allow multipath to fail all I/O.
+         */
+        kblockd_schedule_work(&head->requeue_work);
+
+        nvme_cdev_del(&head->cdev, &head->cdev_device);
+        synchronize_srcu(&head->srcu);
+        del_gendisk(head->disk);
+        nvme_put_ns_head(head);
+    }
+}
+
+static void nvme_remove_head_work(struct work_struct *work)
Quoted 8 lines

drivers/nvme/host/multipath.c static void nvme_requeue_work(struct work_struct *work) · line 658

Quoted 5 lines
drivers/nvme/host/multipath.c +5 -0
+{
+    struct nvme_ns_head *head = container_of(to_delayed_work(work),
+            struct nvme_ns_head, remove_work);
+    bool shutdown = false;
+

Don't you need to check here if the 'QUEUE_IF_NO_PATH' bit is still cleared?

No, we don't need to do that, because the QUEUE_IF_NO_PATH flag is tied to the head->delayed_removal_secs setting. Specifically, the QUEUE_IF_NO_PATH flag is set when head->delayed_ removal_secs is configured to a non-zero value. It is cleared only when head->delayed_removal_secs is reset to zero.

For example, if the user sets delayed_removal_secs to a non-zero value for a head node, the QUEUE_IF_NO_PATH flag is set. Even if all paths to a shared namespace are lost, the flag remains set until delayed_removal_secs is reset to zero.

Hence, instead of relying solely on this flag, here we check whether head->list is empty after the delayed removal timeout has elapsed. If head->list is not empty, it indicates that a path to the namespace has been recovered (e.g., the link failure was resolved in time). Therefore, we should not remove the head node.

In short in this function, checking whether head->list is empty is sufficient to determine whether the head node should be removed or retained.

Quoted 24 lines
Quoted 24 lines
drivers/nvme/host/multipath.c +14 -0
+    mutex_lock(&head->subsys->lock);
+    if (list_empty(&head->list)) {
+        list_del_init(&head->entry);
+        shutdown = true;
+    }
+    mutex_unlock(&head->subsys->lock);
+    if (shutdown)
+        nvme_remove_head(head);
+
+    module_put(THIS_MODULE);
+}
+
   int nvme_mpath_alloc_disk(struct nvme_ctrl *ctrl, struct nvme_ns_head *head)
   {
       struct queue_limits lim;
@@ -626,6 +677,8 @@ int nvme_mpath_alloc_disk(struct nvme_ctrl *ctrl, struct nvme_ns_head *head)
       spin_lock_init(&head->requeue_lock);
       INIT_WORK(&head->requeue_work, nvme_requeue_work);
       INIT_WORK(&head->partition_scan_work, nvme_partition_scan_work);
+    INIT_DELAYED_WORK(&head->remove_work, nvme_remove_head_work);
+    head->delayed_removal_secs = 0;
         /*
        * Add a multipath node if the subsystems supports multiple controllers.
Quoted 8 lines

drivers/nvme/host/multipath.c int nvme_mpath_alloc_disk(struct nvme_ctrl *ctrl, struct nvme_ns_head *head) · line 715

Quoted 5 lines
drivers/nvme/host/multipath.c +1 -0
@@ -659,6 +712,7 @@ int nvme_mpath_alloc_disk(struct nvme_ctrl *ctrl, struct nvme_ns_head *head)
       set_bit(GD_SUPPRESS_PART_SCAN, &head->disk->state);
       sprintf(head->disk->disk_name, "nvme%dn%d",
               ctrl->subsys->instance, head->instance);
+    nvme_tryget_ns_head(head);

What is that doing here?
Why do you tak an additional reference?

We take an additional reference to the head node here to ensure that it isn't freed immediately when all paths to the shared namespace are removed. The final reference to the head node is only released when:
- All paths to the namespace are lost, and
- No new path to the namespace has appeared even after the configured delayed_removal_secs (if set) has elapsed.
This way it's ensured that the head node remains valid during the grace period, allowing recovery if a path becomes available again.

Thanks,
--Nilay

#4 lore raw

Re: [RFC PATCHv3 1/3] nvme-multipath: introduce delayed removal of the multipath head node

In reply to Hannes Reinecke
Quoted 5 lines · Hannes Reinecke

drivers/nvme/host/core.c static struct nvme_ns_head *nvme_alloc_ns_head(struct nvme_ctrl *ctrl, · line 3692

Quoted 3 lines
drivers/nvme/host/core.c +2 -0
+	if (ctrl->ops->flags & NVME_F_FABRICS)
+		nvme_mpath_set_fabrics(head);

Please make this a separate patch.

Absolutelty not if we keep this, as there is no other user.

But I still don't like the fabrics special casing at all.

I'd much rather just disable it if opts is present and max_reconnects is set to non-0.

#5 lore raw

Re: [RFC PATCHv3 1/3] nvme-multipath: introduce delayed removal of the multipath head node

Quoted 7 lines · Christoph Hellwig
Quoted 5 lines · Hannes Reinecke

drivers/nvme/host/core.c static struct nvme_ns_head *nvme_alloc_ns_head(struct nvme_ctrl *ctrl, · line 3692

Quoted 3 lines
drivers/nvme/host/core.c +2 -0
+	if (ctrl->ops->flags & NVME_F_FABRICS)
+		nvme_mpath_set_fabrics(head);

Please make this a separate patch.

Quoted 7 lines

Absolutelty not if we keep this, as there is no other user.

But I still don't like the fabrics special casing at all.

I'd much rather just disable it if opts is present and max_reconnects is set to non-0.

That's a possible approach; however, the challenge arises when the head node points to multiple namespace paths, each associated with different fabric controllers.
As we know, max_reconnects is a per-controller attribute. So, if only one of the fabric controllers has max_reconnects set to a non-zero value, but others do not, how do we decide whether to implicitly disable head->delayed_removal_secs?

In such case, should we assume that if any one of the controllers has max_reconnects set to a non-zero value, then the configured head->delayed_removal_secs should be ignored?

Thanks,
--Nilay

#6 lore raw

Re: [RFC PATCHv3 1/3] nvme-multipath: introduce delayed removal of the multipath head node

Quoted 3 lines · Nilay Shroff

That's a possible approach; however, the challenge arises when the head node points to multiple namespace paths, each associated with different fabric controllers.

That's generally a good point. Especially PCIe vs fabrics will be fun to test. We really need a PCI loop device for nvmet to be able to test it.

Quoted 3 lines

As we know, max_reconnects is a per-controller attribute. So, if only one of the fabric controllers has max_reconnects set to a non-zero value, but others do not, how do we decide whether to implicitly disable head->delayed_removal_secs?

Well, as soon as something in the subsystems can't reconnect we want to delay the removal, right?

#7 lore raw

Re: [RFC PATCHv3 1/3] nvme-multipath: introduce delayed removal of the multipath head node

Quoted 9 lines · Christoph Hellwig
Quoted 3 lines · Nilay Shroff

That's a possible approach; however, the challenge arises when the head node points to multiple namespace paths, each associated with different fabric controllers.

That's generally a good point. Especially PCIe vs fabrics will be fun to test. We really need a PCI loop device for nvmet to be able to test it.

Quoted 7 lines
Quoted 3 lines

As we know, max_reconnects is a per-controller attribute. So, if only one of the fabric controllers has max_reconnects set to a non-zero value, but others do not, how do we decide whether to implicitly disable head->delayed_removal_secs?

Well, as soon as something in the subsystems can't reconnect we want to delay the removal, right?

Hmm, okay I agree (as you earlier suggested) lets not make any difference between pcie vs fabric setup and unify both. If PCIe/fabric link fails to reconnect and we lost all paths to a namespace then we'd start the head node's delayed_removal_secs timer. This way we'd not further complicate the logic and we also don't need to learn whether head node points to a fabric setup. I will spin a new patchset with this change and send upstream.

Thanks,
--Nilay

#8 lore raw

RFC PATCH v3 2/3 nvme: introduce multipath_always_on module param

In reply to Nilay Shroff

Currently, a multipath head disk node is not created for single-ported NVMe adapters or private namespaces. However, creating a head node in these cases can help transparently handle transient PCIe link failures. Without a head node, features like delayed removal cannot be leveraged, making it difficult to tolerate such link failures. To address this, this commit introduces nvme_core module parameter multipath_always_on.

When this param is set to true, it forces the creation of a multipath head node regardless NVMe disk or namespace type. So this option allows the use of delayed removal of head node functionality even for single- ported NVMe disks and private namespaces and thus helps transparently handling transient PCIe link failures.

By default multipath_always_on is set to false, thus preserving the existing behavior. Setting it to true enables improved fault tolerance in PCIe setups. Moreover, please note that enabling this option would also implicitly enable nvme_core.multipath.

---
 drivers/nvme/host/multipath.c | 72 +++++++++++++++++++++++++++++++----
 1 file changed, 65 insertions(+), 7 deletions(-)
drivers/nvme/host/multipath.c +65 -7
index f3f981c9f3ec..bc115b79eb13 100644
@@ -10,10 +10,61 @@
 #include "nvme.h"
 
 bool multipath = true;
-module_param(multipath, bool, 0444);
+bool multipath_always_on;	/* default is flase */
+
+static int multipath_param_set(const char *val, const struct kernel_param *kp)
+{
+	int ret;
+	bool *arg = kp->arg;
+
+	ret = param_set_bool(val, kp);
+	if (ret)
+		return ret;
+
+	if (multipath_always_on && !*arg) {
+		pr_err("Can't disable multipath when multipath_always_on is configured.\n");
+		*arg = true;
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static const struct kernel_param_ops multipath_param_ops = {
+	.set = multipath_param_set,
+	.get = param_get_bool,
+};
+
+module_param_cb(multipath, &multipath_param_ops, &multipath, 0444);
 MODULE_PARM_DESC(multipath,
 	"turn on native support for multiple controllers per subsystem");
 
+static int multipath_always_on_set(const char *val,
+		const struct kernel_param *kp)
+{
+	int ret;
+	bool *arg = kp->arg;
+
+	ret = param_set_bool(val, kp);
+	if (ret < 0)
+		return ret;
+
+	if (*arg)
+		multipath = true;
+
+	return 0;
+}
+
+static const struct kernel_param_ops multipath_always_on_ops = {
+	.set = multipath_always_on_set,
+	.get = param_get_bool,
+};
+
+module_param_cb(multipath_always_on, &multipath_always_on_ops,
+		&multipath_always_on, 0444);
+MODULE_PARM_DESC(multipath_always_on,
+	"create multipath node always; note that this also implicitly enables native multipath support");
+
 static const char *nvme_iopolicy_names[] = {
 	[NVME_IOPOLICY_NUMA]	= "numa",
 	[NVME_IOPOLICY_RR]	= "round-robin",
@@ -681,13 +732,20 @@ int nvme_mpath_alloc_disk(struct nvme_ctrl *ctrl, struct nvme_ns_head *head)
 	head->delayed_removal_secs = 0;
 
 	/*
-	 * Add a multipath node if the subsystems supports multiple controllers.
-	 * We also do this for private namespaces as the namespace sharing flag
-	 * could change after a rescan.
+	 * If multipath_always_on is configured then we add a multipath head
+	 * disk node irrespective of disk is single/multi ported or namespace
+	 * is shared/private.
 	 */
-	if (!(ctrl->subsys->cmic & NVME_CTRL_CMIC_MULTI_CTRL) ||
-	    !nvme_is_unique_nsid(ctrl, head) || !multipath)
-		return 0;
+	if (!multipath_always_on) {
+		/*
+		 * Add a multipath node if the subsystems supports multiple
+		 * controllers. We also do this for private namespaces as the
+		 * namespace sharing flag could change after a rescan.
+		 */
+		if (!(ctrl->subsys->cmic & NVME_CTRL_CMIC_MULTI_CTRL) ||
+		    !nvme_is_unique_nsid(ctrl, head) || !multipath)
+			return 0;
+	}
 
 	blk_set_stacking_limits(&lim);
 	lim.dma_alignment = 3;
Signature · 1 line
2.49.0
Reviewed-by on v3 2/3
#9 lore raw

Re: [RFC PATCHv3 2/3] nvme: introduce multipath_always_on module param

Quoted 18 lines · Nilay Shroff

Currently, a multipath head disk node is not created for single-ported NVMe adapters or private namespaces. However, creating a head node in these cases can help transparently handle transient PCIe link failures. Without a head node, features like delayed removal cannot be leveraged, making it difficult to tolerate such link failures. To address this, this commit introduces nvme_core module parameter multipath_always_on.

When this param is set to true, it forces the creation of a multipath head node regardless NVMe disk or namespace type. So this option allows the use of delayed removal of head node functionality even for single- ported NVMe disks and private namespaces and thus helps transparently handling transient PCIe link failures.

By default multipath_always_on is set to false, thus preserving the existing behavior. Setting it to true enables improved fault tolerance in PCIe setups. Moreover, please note that enabling this option would also implicitly enable nvme_core.multipath.

Quoted 5 lines
---
  drivers/nvme/host/multipath.c | 72 +++++++++++++++++++++++++++++++----
  1 file changed, 65 insertions(+), 7 deletions(-)

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
#11 lore raw

RFC PATCH v3 3/3 nvme: rename nvme_mpath_shutdown_disk to nvme_mpath_remove_disk

In reply to Nilay Shroff

In the NVMe context, the term "shutdown" has a specific technical meaning. To avoid confusion, this commit renames the nvme_mpath_ shutdown_disk function to nvme_mpath_remove_disk to better reflect its purpose (i.e. removing the disk from the system). However, nvme_mpath_remove_disk was already in use, and its functionality is related to releasing or putting the head node disk. To resolve this naming conflict and improve clarity, the existing nvme_mpath_ remove_disk function is also renamed to nvme_mpath_put_disk.

This renaming improves code readability and better aligns function names with their actual roles.

---
 drivers/nvme/host/core.c      |  4 ++--
 drivers/nvme/host/multipath.c | 16 ++++++++--------
 drivers/nvme/host/nvme.h      |  8 ++++----
 3 files changed, 14 insertions(+), 14 deletions(-)
drivers/nvme/host/core.c +2 -2
index 0f6b01f10c70..254e6a7249a8 100644
@@ -664,7 +664,7 @@ static void nvme_free_ns_head(struct kref *ref)
 	struct nvme_ns_head *head =
 		container_of(ref, struct nvme_ns_head, ref);
 
-	nvme_mpath_remove_disk(head);
+	nvme_mpath_put_disk(head);
 	ida_free(&head->subsys->ns_ida, head->instance);
 	cleanup_srcu_struct(&head->srcu);
 	nvme_put_subsystem(head->subsys);
@@ -4033,7 +4033,7 @@ static void nvme_ns_remove(struct nvme_ns *ns)
 	synchronize_srcu(&ns->ctrl->srcu);
 
 	if (last_path)
-		nvme_mpath_shutdown_disk(ns->head);
+		nvme_mpath_remove_disk(ns->head);
 	nvme_put_ns(ns);
 }
 
drivers/nvme/host/multipath.c +8 -8
index bc115b79eb13..ca6562d5c17a 100644
@@ -705,15 +705,15 @@ static void nvme_remove_head_work(struct work_struct *work)
 {
 	struct nvme_ns_head *head = container_of(to_delayed_work(work),
 			struct nvme_ns_head, remove_work);
-	bool shutdown = false;
+	bool remove = false;
 
 	mutex_lock(&head->subsys->lock);
 	if (list_empty(&head->list)) {
 		list_del_init(&head->entry);
-		shutdown = true;
+		remove = true;
 	}
 	mutex_unlock(&head->subsys->lock);
-	if (shutdown)
+	if (remove)
 		nvme_remove_head(head);
 
 	module_put(THIS_MODULE);
@@ -1291,9 +1291,9 @@ void nvme_mpath_add_disk(struct nvme_ns *ns, __le32 anagrpid)
 #endif
 }
 
-void nvme_mpath_shutdown_disk(struct nvme_ns_head *head)
+void nvme_mpath_remove_disk(struct nvme_ns_head *head)
 {
-	bool shutdown = false;
+	bool remove = false;
 
 	mutex_lock(&head->subsys->lock);
 	/*
@@ -1319,15 +1319,15 @@ void nvme_mpath_shutdown_disk(struct nvme_ns_head *head)
 				head->delayed_removal_secs * HZ);
 	} else {
 		list_del_init(&head->entry);
-		shutdown = true;
+		remove = true;
 	}
 out:
 	mutex_unlock(&head->subsys->lock);
-	if (shutdown)
+	if (remove)
 		nvme_remove_head(head);
 }
 
-void nvme_mpath_remove_disk(struct nvme_ns_head *head)
+void nvme_mpath_put_disk(struct nvme_ns_head *head)
 {
 	if (!head->disk)
 		return;
drivers/nvme/host/nvme.h +4 -4
index b2d02425dc3c..490fdee867a9 100644
@@ -964,7 +964,7 @@ int nvme_mpath_alloc_disk(struct nvme_ctrl *ctrl,struct nvme_ns_head *head);
 void nvme_mpath_add_sysfs_link(struct nvme_ns_head *ns);
 void nvme_mpath_remove_sysfs_link(struct nvme_ns *ns);
 void nvme_mpath_add_disk(struct nvme_ns *ns, __le32 anagrpid);
-void nvme_mpath_remove_disk(struct nvme_ns_head *head);
+void nvme_mpath_put_disk(struct nvme_ns_head *head);
 int nvme_mpath_init_identify(struct nvme_ctrl *ctrl, struct nvme_id_ctrl *id);
 void nvme_mpath_init_ctrl(struct nvme_ctrl *ctrl);
 void nvme_mpath_update(struct nvme_ctrl *ctrl);
@@ -973,7 +973,7 @@ void nvme_mpath_stop(struct nvme_ctrl *ctrl);
 bool nvme_mpath_clear_current_path(struct nvme_ns *ns);
 void nvme_mpath_revalidate_paths(struct nvme_ns *ns);
 void nvme_mpath_clear_ctrl_paths(struct nvme_ctrl *ctrl);
-void nvme_mpath_shutdown_disk(struct nvme_ns_head *head);
+void nvme_mpath_remove_disk(struct nvme_ns_head *head);
 void nvme_mpath_start_request(struct request *rq);
 void nvme_mpath_end_request(struct request *rq);
 
@@ -1027,7 +1027,7 @@ static inline int nvme_mpath_alloc_disk(struct nvme_ctrl *ctrl,
 static inline void nvme_mpath_add_disk(struct nvme_ns *ns, __le32 anagrpid)
 {
 }
-static inline void nvme_mpath_remove_disk(struct nvme_ns_head *head)
+static inline void nvme_mpath_put_disk(struct nvme_ns_head *head)
 {
 }
 static inline void nvme_mpath_add_sysfs_link(struct nvme_ns *ns)
@@ -1046,7 +1046,7 @@ static inline void nvme_mpath_revalidate_paths(struct nvme_ns *ns)
 static inline void nvme_mpath_clear_ctrl_paths(struct nvme_ctrl *ctrl)
 {
 }
-static inline void nvme_mpath_shutdown_disk(struct nvme_ns_head *head)
+static inline void nvme_mpath_remove_disk(struct nvme_ns_head *head)
 {
 }
 static inline void nvme_trace_bio_complete(struct request *req)
Signature · 1 line
2.49.0
Reviewed-by on v3 3/3
#12 lore raw

Re: [RFC PATCHv3 3/3] nvme: rename nvme_mpath_shutdown_disk to nvme_mpath_remove_disk

Quoted 12 lines · Nilay Shroff

In the NVMe context, the term "shutdown" has a specific technical meaning. To avoid confusion, this commit renames the nvme_mpath_ shutdown_disk function to nvme_mpath_remove_disk to better reflect its purpose (i.e. removing the disk from the system). However, nvme_mpath_remove_disk was already in use, and its functionality is related to releasing or putting the head node disk. To resolve this naming conflict and improve clarity, the existing nvme_mpath_ remove_disk function is also renamed to nvme_mpath_put_disk.

This renaming improves code readability and better aligns function names with their actual roles.

Quoted 9 lines
---
  drivers/nvme/host/core.c      |  4 ++--
  drivers/nvme/host/multipath.c | 16 ++++++++--------
  drivers/nvme/host/nvme.h      |  8 ++++----
  3 files changed, 14 insertions(+), 14 deletions(-)

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich