Skip to contentSkip to conversation

libnvme: add support for discovering multipath of a shared ns

PATCH In review Nilay Shroff 2 participants 7 messages Apr 25, 2025 – Jun 4, 2025 NVMe

lore
Share & tools
lore
#0 lore raw

v3 cover libnvme: add support for discovering multipath of a shared ns

Hi,

Recently released Linux kernel v6.15-rc1 added support for easily discovering multiple paths to a shared NVMe namespace. This multipath information is exposed to userspace via a new sysfs group attribute named "multipath", located under /sys/block/<ns-blkdev>/. More details on this functionality can be found here [1].

This patch series leverages that new functionality to discover multiple paths to a shared namespace and exposes that information in libnvme so that it can later be used by nvme-cli.

There are four patches in this series:
The first patch adds support for discovering NVMe paths using the sysfs "multipath" group attribute.
The second patch adds a new "queue_depth" attribute under the NVMe path object.
The third patch adds a new "numa_nodes" attribute under the NVMe path object. The fourth patch extends the sysfs tree dump test to validate multipath link support

[1]: https://lore.kernel.org/all/[email protected]/

Changes from v2:
- Modified queue_depth get attribure function to always return the latest value insteda of cached value (Hannes Reinecke)

Changes from v1:
- Fixed kernel version typo in the cover letter and commit
- Updated the commit in first patch to explain the change in some detail (Daniel Wagner)
- Added fourth patch to validate changes (Daniel Wagner)

Nilay Shroff (4):
tree: add support for discovering nvme paths using sysfs multipath
link
tree: add queue-depth attribute for nvme path object
tree: add attribute numa_nodes for NVMe path object
test: extend sysfs tree dump test

 src/libnvme.map                  |   2 +
 src/nvme/filters.c               |   6 +
 src/nvme/filters.h               |   9 ++
 src/nvme/json.c                  |  90 +++++++++++++--
 src/nvme/private.h               |  11 +-
 src/nvme/tree.c                  | 191 ++++++++++++++++++++++---------
 src/nvme/tree.h                  |  25 ++++
 test/sysfs/data/tree-pcie.out    |  53 +++++++--
 test/sysfs/data/tree-pcie.tar.xz | Bin 19712 -> 12656 bytes
 9 files changed, 315 insertions(+), 72 deletions(-)
Signature · 1 line
2.49.0
#1 lore raw

PATCH v3 1/4 tree: add support for discovering nvme paths using sysfs multipath link

With the upcoming Linux kernel v6.15, NVMe native multipath now provides a simplified mechanism for discovering all paths to a shared namespace through sysfs.

A new "multipath" directory is created under each NVMe head namespace device in "/sys/block/<head>/multipath/". This directory contains symlinks to all namespace path devices that access the same shared namespace.

For example, consider a shared namespace accessible via two paths under nvme-subsys1:

nvme-subsys1 - NQN=nqn.1994-11.com.samsung:nvme:PM1735a:2.5-inch:S6RTNE0R900057

    hostnqn=nqn.2014-08.org.nvmexpress:uuid:41528538-e8ad-4eaf-84a7-9c552917d988
\
 +- ns 1
 \
  +- nvme0 pcie 052e:78:00.0 live optimized
  +- nvme1 pcie 058e:78:00.0 live optimized

The head device /dev/nvme1n1 will now have the following structure:

/sys/block/nvme1n1/multipath/

├── nvme1c0n1 -> ../../../../../pci052e:78/052e:78:00.0/nvme/nvme0/nvme1c0n1
└── nvme1c1n1 -> ../../../../../pci058e:78/058e:78:00.0/nvme/nvme1/nvme1c1n1

This clearly shows that namespace 1 is accessible through both nvme1c0n1 and nvme1c1n1. This new sysfs structure significantly simplifies multipath discovery and management, making it easier for tools and scripts to enumerate and manage NVMe multipath configurations. So leverage this functionality to update the path links for a shared NVMe namespace, simplifying path discovery and management.

This change adds a new struct nvme_ns_head to represent the head of a shared namespace. It contains a list head linking together struct nvme_path objects, where each path corresponds to a shared namespace instance. Additionally, struct nvme_ns has been updated to reference its associated nvme_ns_head, enabling straightforward traversal of all paths to a shared NVMe namespace.

---
 src/nvme/filters.c |   6 ++
 src/nvme/filters.h |   9 +++
 src/nvme/private.h |   9 ++-
 src/nvme/tree.c    | 162 +++++++++++++++++++++++++++++++--------------
 src/nvme/tree.h    |   9 +++
 5 files changed, 143 insertions(+), 52 deletions(-)
src/nvme/filters.c +6 -0
index ceaba68f..4a8829db 100644
@@ -105,3 +105,9 @@ int nvme_scan_ctrl_namespaces(nvme_ctrl_t c, struct dirent ***ns)
 	return scandir(nvme_ctrl_get_sysfs_dir(c), ns,
 		       nvme_namespace_filter, alphasort);
 }
+
+int nvme_scan_ns_head_paths(nvme_ns_head_t head, struct dirent ***paths)
+{
+	return scandir(nvme_ns_head_get_sysfs_dir(head), paths,
+		       nvme_paths_filter, alphasort);
+}
src/nvme/filters.h +9 -0
index 4ceeffd5..9e9dbb95 100644
@@ -94,4 +94,13 @@ int nvme_scan_ctrl_namespace_paths(nvme_ctrl_t c, struct dirent ***paths);
  */
 int nvme_scan_ctrl_namespaces(nvme_ctrl_t c, struct dirent ***ns);
 
+/**
+ * nvme_scan_ns_head_paths() - Scan for namespace paths
+ * @head: Namespace head node to scan
+ * @paths : Pointer to array of dirents
+ *
+ * Return: number of entries in @ents
+ */
+int nvme_scan_ns_head_paths(nvme_ns_head_t head, struct dirent ***paths);
+
 #endif /* _LIBNVME_FILTERS_H */
src/nvme/private.h +8 -1
index 33cdd555..f45c5823 100644
@@ -36,12 +36,19 @@ struct nvme_path {
 	int grpid;
 };
 
+struct nvme_ns_head {
+	struct list_head paths;
+	struct nvme_ns *n;
+
+	char *sysfs_dir;
+};
+
 struct nvme_ns {
 	struct list_node entry;
-	struct list_head paths;
 
 	struct nvme_subsystem *s;
 	struct nvme_ctrl *c;
+	struct nvme_ns_head *head;
 
 	int fd;
 	__u32 nsid;
src/nvme/tree.c +111 -51

249 lines not on this page · raw message

src/nvme/tree.h +9 -0
index 25d4b31b..9f382e9c 100644
@@ -27,6 +27,7 @@
  */
 
 typedef struct nvme_ns *nvme_ns_t;
+typedef struct nvme_ns_head *nvme_ns_head_t;
 typedef struct nvme_path *nvme_path_t;
 typedef struct nvme_ctrl *nvme_ctrl_t;
 typedef struct nvme_subsystem *nvme_subsystem_t;
@@ -1091,6 +1092,14 @@ void nvme_ctrl_set_dhchap_host_key(nvme_ctrl_t c, const char *key);
  */
 const char *nvme_ctrl_get_dhchap_key(nvme_ctrl_t c);
 
+/**
+ * nvme_ns_head_get_sysfs_dir() - sysfs dir of namespave head
+ * @head: namespace head instance
+ *
+ * Returns: sysfs directory name of @head
+ */
+const char *nvme_ns_head_get_sysfs_dir(nvme_ns_head_t head);
+
 /**
  * nvme_ctrl_set_dhchap_key() - Set controller key
  * @c:		Controller for which the key should be set
Signature · 1 line
2.49.0
#2 lore raw

PATCH v3 2/4 tree: add queue-depth attribute for nvme path object

In reply to Nilay Shroff

Introduce a new sysfs attribute, queue_depth, under the NVMe path object. This attribute is used by the queue-depth I/O policy introduced in kernel v6.11, but the sysfs interface for this attribute was only added later in kernel v6.15.

The queue_depth value is useful for observing which paths are selected for I/O forwarding based on their current queue depths. To make this information available to user space tools such as nvme-cli, the attribute is now exported in libnvme.map.

As queue_depth value could change frequently, nvme_path_get_queue_depth() is implemented to always fetch the latest queue_depth value, rather than relying solely on a cached version. If fetching the latest value fails, the function gracefully falls back to the cached value.

---
 src/libnvme.map    |  1 +
 src/nvme/private.h |  1 +
 src/nvme/tree.c    | 19 ++++++++++++++++++-
 src/nvme/tree.h    |  8 ++++++++
 4 files changed, 28 insertions(+), 1 deletion(-)
src/libnvme.map +1 -0
index 4314705f..e53fad6b 100644
@@ -317,6 +317,7 @@ LIBNVME_1_0 {
 		nvme_path_get_ctrl;
 		nvme_path_get_name;
 		nvme_path_get_ns;
+		nvme_path_get_queue_depth;
 		nvme_path_get_sysfs_dir;
 		nvme_paths_filter;
 		nvme_read_config;
src/nvme/private.h +1 -0
index f45c5823..f94276e2 100644
@@ -34,6 +34,7 @@ struct nvme_path {
 	char *sysfs_dir;
 	char *ana_state;
 	int grpid;
+	int queue_depth;
 };
 
 struct nvme_ns_head {
src/nvme/tree.c +18 -1
index 2f82fd77..20d4c2e2 100644
@@ -903,6 +903,18 @@ const char *nvme_path_get_name(nvme_path_t p)
 	return p->name;
 }
 
+int nvme_path_get_queue_depth(nvme_path_t p)
+{
+	_cleanup_free_ char *queue_depth = NULL;
+
+	queue_depth = nvme_get_path_attr(p, "queue_depth");
+	if (queue_depth) {
+		sscanf(queue_depth, "%d", &p->queue_depth);
+	}
+
+	return p->queue_depth;
+}
+
 const char *nvme_path_get_ana_state(nvme_path_t p)
 {
 	return p->ana_state;
@@ -921,7 +933,7 @@ void nvme_free_path(struct nvme_path *p)
 static int nvme_ctrl_scan_path(nvme_root_t r, struct nvme_ctrl *c, char *name)
 {
 	struct nvme_path *p;
-	_cleanup_free_ char *path = NULL, *grpid = NULL;
+	_cleanup_free_ char *path = NULL, *grpid = NULL, *queue_depth = NULL;
 	int ret;
 
 	nvme_msg(r, LOG_DEBUG, "scan controller %s path %s\n",
@@ -955,6 +967,11 @@ static int nvme_ctrl_scan_path(nvme_root_t r, struct nvme_ctrl *c, char *name)
 		sscanf(grpid, "%d", &p->grpid);
 	}
 
+	queue_depth = nvme_get_path_attr(p, "queue_depth");
+	if (queue_depth) {
+		sscanf(queue_depth, "%d", &p->queue_depth);
+	}
+
 	list_node_init(&p->nentry);
 	list_node_init(&p->entry);
 	list_add_tail(&c->paths, &p->entry);
src/nvme/tree.h +8 -0
index 9f382e9c..a9082f8e 100644
@@ -867,6 +867,14 @@ const char *nvme_path_get_sysfs_dir(nvme_path_t p);
  */
 const char *nvme_path_get_ana_state(nvme_path_t p);
 
+/**
+ * nvme_path_get_queue_depth() - Queue depth of an nvme_path_t object
+ * @p: &nvme_path_t object
+ *
+ * Return: Queue depth of @p
+ */
+int nvme_path_get_queue_depth(nvme_path_t p);
+
 /**
  * nvme_path_get_ctrl() - Parent controller of an nvme_path_t object
  * @p:	&nvme_path_t object
Signature · 1 line
2.49.0
#3 lore raw

PATCH v3 3/4 tree: add attribute numa_nodes for NVMe path object

In reply to Nilay Shroff

Add a new attribute named "numa_nodes" under the NVMe path object. This attribute is used by the iopolicy "numa". The numa_nodes value is stored for each NVMe path and represents the NUMA node(s) associated with it. When the iopolicy is set to "numa", I/O traffic originating from a given NUMA node will be forwarded through the corresponding NVMe path.

The numa_nodes attribute is useful for observing which NVMe path the kernel would choose for I/O forwarding based on NUMA affinity. To support this, export the attribute in libnvme.map so it can be accessed via nvme-cli.

---
 src/libnvme.map    |  1 +
 src/nvme/private.h |  1 +
 src/nvme/tree.c    | 10 ++++++++++
 src/nvme/tree.h    |  8 ++++++++
 4 files changed, 20 insertions(+)
src/libnvme.map +1 -0
index e53fad6b..ffadc647 100644
@@ -317,6 +317,7 @@ LIBNVME_1_0 {
 		nvme_path_get_ctrl;
 		nvme_path_get_name;
 		nvme_path_get_ns;
+		nvme_path_get_numa_nodes;
 		nvme_path_get_queue_depth;
 		nvme_path_get_sysfs_dir;
 		nvme_paths_filter;
src/nvme/private.h +1 -0
index f94276e2..5bfa027b 100644
@@ -33,6 +33,7 @@ struct nvme_path {
 	char *name;
 	char *sysfs_dir;
 	char *ana_state;
+	char *numa_nodes;
 	int grpid;
 	int queue_depth;
 };
src/nvme/tree.c +10 -0
index 20d4c2e2..80dccdc4 100644
@@ -920,6 +920,11 @@ const char *nvme_path_get_ana_state(nvme_path_t p)
 	return p->ana_state;
 }
 
+const char *nvme_path_get_numa_nodes(nvme_path_t p)
+{
+	return p->numa_nodes;
+}
+
 void nvme_free_path(struct nvme_path *p)
 {
 	list_del_init(&p->entry);
@@ -927,6 +932,7 @@ void nvme_free_path(struct nvme_path *p)
 	free(p->name);
 	free(p->sysfs_dir);
 	free(p->ana_state);
+	free(p->numa_nodes);
 	free(p);
 }
 
@@ -962,6 +968,10 @@ static int nvme_ctrl_scan_path(nvme_root_t r, struct nvme_ctrl *c, char *name)
 	if (!p->ana_state)
 		p->ana_state = strdup("optimized");
 
+	p->numa_nodes = nvme_get_path_attr(p, "numa_nodes");
+	if (!p->numa_nodes)
+		p->numa_nodes = strdup("-1");
+
 	grpid = nvme_get_path_attr(p, "ana_grpid");
 	if (grpid) {
 		sscanf(grpid, "%d", &p->grpid);
src/nvme/tree.h +8 -0
index a9082f8e..f6116c2b 100644
@@ -867,6 +867,14 @@ const char *nvme_path_get_sysfs_dir(nvme_path_t p);
  */
 const char *nvme_path_get_ana_state(nvme_path_t p);
 
+/**
+ * nvme_path_get_numa_nodes() - NUMA nodes of an nvme_path_t object
+ * @p : &nvme_path_t object
+ *
+ * Return: NUMA nodes associated to @p
+ */
+const char *nvme_path_get_numa_nodes(nvme_path_t p);
+
 /**
  * nvme_path_get_queue_depth() - Queue depth of an nvme_path_t object
  * @p: &nvme_path_t object
Signature · 1 line
2.49.0
#4 lore raw

PATCH v3 4/4 test: extend sysfs tree dump test

In reply to Nilay Shroff

In order to validate multipath link support extend the existing sysfs tree dump test.

The updated tree-pcie.tar.xz archive includes a sysfs dump with two NVMe subsystems. The nvme-subsys0 simulates a single-port NVMe disk with a private namespace that does not support multipath. And another nvme-subsys1 simulates a dual-port NVMe disk with a shared namespace attached to multiple controllers.

The json tree dump is also updated to include NVMe namespace path information. Since the archive content has changed, the expected output file tree-pcie.out is also updated to reflect the new structure.

---
 src/nvme/json.c                  |  90 +++++++++++++++++++++++++++----
 test/sysfs/data/tree-pcie.out    |  53 ++++++++++++++----
 test/sysfs/data/tree-pcie.tar.xz | Bin 19712 -> 12656 bytes
 3 files changed, 124 insertions(+), 19 deletions(-)
src/nvme/json.c +81 -9
index af28bd48..a2fe8173 100644
@@ -533,25 +533,97 @@ static void json_dump_ctrl(struct json_object *ctrl_array, nvme_ctrl_t c)
 	json_object_array_add(ctrl_array, ctrl_obj);
 }
 
+static unsigned int json_dump_subsys_multipath(nvme_subsystem_t s,
+				struct json_object *ns_array)
+{
+	nvme_ns_t n;
+	nvme_path_t p;
+	unsigned int i = 0;
+
+	nvme_subsystem_for_each_ns(s, n) {
+		struct json_object *ns_obj;
+		struct json_object *path_array;
+
+		ns_obj = json_object_new_object();
+		json_object_object_add(ns_obj, "nsid",
+				json_object_new_int(nvme_ns_get_nsid(n)));
+		json_object_object_add(ns_obj, "name",
+				json_object_new_string(nvme_ns_get_name(n)));
+
+		path_array = json_object_new_array();
+		nvme_namespace_for_each_path(n, p) {
+			struct json_object *path_obj;
+			struct json_object *ctrl_array;
+			nvme_ctrl_t c;
+
+			path_obj = json_object_new_object();
+			json_object_object_add(path_obj, "path",
+				json_object_new_string(nvme_path_get_name(p)));
+			json_object_object_add(path_obj, "ANAState",
+				json_object_new_string(nvme_path_get_ana_state(p)));
+			json_object_object_add(path_obj, "NUMANodes",
+				json_object_new_string(nvme_path_get_numa_nodes(p)));
+			json_object_object_add(path_obj, "qdepth",
+				json_object_new_int(nvme_path_get_queue_depth(p)));
+
+			c = nvme_path_get_ctrl(p);
+			ctrl_array = json_object_new_array();
+			json_dump_ctrl(ctrl_array, c);
+			json_object_object_add(path_obj, "controller", ctrl_array);
+			json_object_array_add(path_array, path_obj);
+		}
+		json_object_object_add(ns_obj, "paths", path_array);
+		json_object_array_add(ns_array, ns_obj);
+		i++;
+	}
+	return i;
+}
+
+static void json_dump_subsys_non_multipath(nvme_subsystem_t s,
+		struct json_object *ns_array)
+{
+	nvme_ctrl_t c;
+	nvme_ns_t n;
+
+	nvme_subsystem_for_each_ctrl(s, c) {
+		nvme_ctrl_for_each_ns(c, n) {
+			struct json_object *ctrl_array;
+			struct json_object *ns_obj;
+
+			ns_obj = json_object_new_object();
+			json_object_object_add(ns_obj, "nsid",
+				json_object_new_int(nvme_ns_get_nsid(n)));
+			json_object_object_add(ns_obj, "name",
+				json_object_new_string(nvme_ns_get_name(n)));
+
+			ctrl_array = json_object_new_array();
+			json_dump_ctrl(ctrl_array, c);
+			json_object_object_add(ns_obj, "controller", ctrl_array);
+
+			json_object_array_add(ns_array, ns_obj);
+		}
+	}
+}
+
 static void json_dump_subsys(struct json_object *subsys_array,
 			       nvme_subsystem_t s)
 {
-	nvme_ctrl_t c;
 	struct json_object *subsys_obj = json_object_new_object();
-	struct json_object *ctrl_array;
+	struct json_object *ns_array;
 
 	json_object_object_add(subsys_obj, "name",
 			       json_object_new_string(nvme_subsystem_get_name(s)));
 	json_object_object_add(subsys_obj, "nqn",
 			       json_object_new_string(nvme_subsystem_get_nqn(s)));
-	ctrl_array = json_object_new_array();
-	nvme_subsystem_for_each_ctrl(s, c) {
-		json_dump_ctrl(ctrl_array, c);
-	}
-	if (json_object_array_length(ctrl_array))
-		json_object_object_add(subsys_obj, "controllers", ctrl_array);
+
+	ns_array = json_object_new_array();
+	if (!json_dump_subsys_multipath(s, ns_array))
+		json_dump_subsys_non_multipath(s, ns_array);
+
+	if (json_object_array_length(ns_array))
+		json_object_object_add(subsys_obj, "namespaces", ns_array);
 	else
-		json_object_put(ctrl_array);
+		json_object_put(ns_array);
 	json_object_array_add(subsys_array, subsys_obj);
 }
 
test/sysfs/data/tree-pcie.out +43 -10

66 lines not on this page · raw message

test/sysfs/data/tree-pcie.tar.xz binary +0 -0
index ee11fdeb0108589b954eb03543665e3a10df0e95..d99d2b943248cc2d7586c084ac639b613524cd2f 100644
GIT binary patch
Signature · 1 line
2.49.0
#5 lore raw
In reply to Nilay Shroff

Hi Daniel, Hannes,

A gentle ping... Do you have any further comment for this patchset?

Thanks,
--Nilay

Quoted 52 lines · Nilay Shroff

Hi,

Recently released Linux kernel v6.15-rc1 added support for easily discovering multiple paths to a shared NVMe namespace. This multipath information is exposed to userspace via a new sysfs group attribute named "multipath", located under /sys/block/<ns-blkdev>/. More details on this functionality can be found here [1].

This patch series leverages that new functionality to discover multiple paths to a shared namespace and exposes that information in libnvme so that it can later be used by nvme-cli.

There are four patches in this series:
The first patch adds support for discovering NVMe paths using the sysfs "multipath" group attribute.
The second patch adds a new "queue_depth" attribute under the NVMe path object.
The third patch adds a new "numa_nodes" attribute under the NVMe path object. The fourth patch extends the sysfs tree dump test to validate multipath link support

[1]: https://lore.kernel.org/all/[email protected]/

Changes from v2:
- Modified queue_depth get attribure function to always return the latest value insteda of cached value (Hannes Reinecke)

Changes from v1:
- Fixed kernel version typo in the cover letter and commit
- Updated the commit in first patch to explain the change in some detail (Daniel Wagner)
- Added fourth patch to validate changes (Daniel Wagner)

Nilay Shroff (4):
tree: add support for discovering nvme paths using sysfs multipath
link
tree: add queue-depth attribute for nvme path object
tree: add attribute numa_nodes for NVMe path object
test: extend sysfs tree dump test

 src/libnvme.map                  |   2 +
 src/nvme/filters.c               |   6 +
 src/nvme/filters.h               |   9 ++
 src/nvme/json.c                  |  90 +++++++++++++--
 src/nvme/private.h               |  11 +-
 src/nvme/tree.c                  | 191 ++++++++++++++++++++++---------
 src/nvme/tree.h                  |  25 ++++
 test/sysfs/data/tree-pcie.out    |  53 +++++++--
 test/sysfs/data/tree-pcie.tar.xz | Bin 19712 -> 12656 bytes
 9 files changed, 315 insertions(+), 72 deletions(-)
#6 lore raw
In reply to Nilay Shroff

Hi Nilay

Quoted 12 lines · Nilay Shroff

Hi,

Recently released Linux kernel v6.15-rc1 added support for easily discovering multiple paths to a shared NVMe namespace. This multipath information is exposed to userspace via a new sysfs group attribute named "multipath", located under /sys/block/<ns-blkdev>/. More details on this functionality can be found here [1].

This patch series leverages that new functionality to discover multiple paths to a shared namespace and exposes that information in libnvme so that it can later be used by nvme-cli.

Quoted 8 lines

There are four patches in this series:
The first patch adds support for discovering NVMe paths using the sysfs "multipath" group attribute.
The second patch adds a new "queue_depth" attribute under the NVMe path object.
The third patch adds a new "numa_nodes" attribute under the NVMe path object. The fourth patch extends the sysfs tree dump test to validate multipath link support

LGTM. I've applied the series.

Thanks,
Daniel