Skip to contentSkip to conversation

libnvme: add support for discovering multipath of a shared ns

PATCH Superseded Nilay Shroff 3 participants 14 messages Apr 17, 2025 – Apr 23, 2025 NVMe

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

v2 cover libnvme: add support for discovering multipath of a shared ns

Hi,

Recently released Linux kernel v6.15-rc1 added support for easily discovering multiple paths to a shared NVMe namespace. This multipath information is exposed to userspace via a new sysfs group attribute named "multipath", located under /sys/block/<ns-blkdev>/. More details on this functionality can be found here [1].

This patch series leverages that new functionality to discover multiple paths to a shared namespace and exposes that information in libnvme so that it can later be used by nvme-cli.

There are four patches in this series:
The first patch adds support for discovering NVMe paths using the sysfs "multipath" group attribute.
The second patch adds a new "queue_depth" attribute under the NVMe path object.
The third patch adds a new "numa_nodes" attribute under the NVMe path object. The fourth patch extends the sysfs tree dump test to validate multipath link support

[1]: https://lore.kernel.org/all/[email protected]/

Changes from v1:
- Fixed kernel version typo in the cover letter and commit
- Updated the commit in first patch to explain the change in some detail (Daniel Wagner)
- Added fourth patch to validate changes (Daniel Wagner)

Nilay Shroff (4):
tree: add support for discovering nvme paths using sysfs multipath
link
tree: add queue-depth attribute for nvme path object
tree: add attribute numa_nodes for NVMe path object
test: extend sysfs tree dump test

 src/libnvme.map                  |   2 +
 src/nvme/filters.c               |   6 +
 src/nvme/filters.h               |   9 ++
 src/nvme/json.c                  |  90 +++++++++++++--
 src/nvme/private.h               |  11 +-
 src/nvme/tree.c                  | 184 ++++++++++++++++++++++---------
 src/nvme/tree.h                  |  25 +++++
 test/sysfs/data/tree-pcie.out    |  53 +++++++--
 test/sysfs/data/tree-pcie.tar.xz | Bin 19712 -> 12656 bytes
 9 files changed, 308 insertions(+), 72 deletions(-)
Signature · 1 line
2.49.0
#4 lore raw

PATCH v2 2/4 tree: add queue-depth attribute for nvme path object

Add a new attribute named "queue_depth" under the NVMe path object. This attribute is used by the iopolicy "queue-depth", which was introduced in kernel v6.11. However, the corresponding sysfs attribute for queue depth was only added in kernel v6.14.

The queue_depth value can be useful for observing which paths are selected for I/O forwarding, based on the depth of each path. To support this, export the attribute in libnvme.map so it can be accessed via nvme-cli.

---
 src/libnvme.map    |  1 +
 src/nvme/private.h |  1 +
 src/nvme/tree.c    | 12 +++++++++++-
 src/nvme/tree.h    |  8 ++++++++
 4 files changed, 21 insertions(+), 1 deletion(-)
src/libnvme.map +1 -0
index 4314705f..e53fad6b 100644
@@ -317,6 +317,7 @@ LIBNVME_1_0 {
 		nvme_path_get_ctrl;
 		nvme_path_get_name;
 		nvme_path_get_ns;
+		nvme_path_get_queue_depth;
 		nvme_path_get_sysfs_dir;
 		nvme_paths_filter;
 		nvme_read_config;
src/nvme/private.h +1 -0
index f45c5823..f94276e2 100644
@@ -34,6 +34,7 @@ struct nvme_path {
 	char *sysfs_dir;
 	char *ana_state;
 	int grpid;
+	int queue_depth;
 };
 
 struct nvme_ns_head {
src/nvme/tree.c +11 -1
index bd7fb53e..b7a38a07 100644
@@ -903,6 +903,11 @@ const char *nvme_path_get_name(nvme_path_t p)
 	return p->name;
 }
 
+int nvme_path_get_queue_depth(nvme_path_t p)
+{
+	return p->queue_depth;
+}
+
 const char *nvme_path_get_ana_state(nvme_path_t p)
 {
 	return p->ana_state;
@@ -921,7 +926,7 @@ void nvme_free_path(struct nvme_path *p)
 static int nvme_ctrl_scan_path(nvme_root_t r, struct nvme_ctrl *c, char *name)
 {
 	struct nvme_path *p;
-	_cleanup_free_ char *path = NULL, *grpid = NULL;
+	_cleanup_free_ char *path = NULL, *grpid = NULL, *queue_depth = NULL;
 	int ret;
 
 	nvme_msg(r, LOG_DEBUG, "scan controller %s path %s\n",
@@ -955,6 +960,11 @@ static int nvme_ctrl_scan_path(nvme_root_t r, struct nvme_ctrl *c, char *name)
 		sscanf(grpid, "%d", &p->grpid);
 	}
 
+	queue_depth = nvme_get_path_attr(p, "queue_depth");
+	if (queue_depth) {
+		sscanf(queue_depth, "%d", &p->queue_depth);
+	}
+
 	list_node_init(&p->nentry);
 	list_node_init(&p->entry);
 	list_add_tail(&c->paths, &p->entry);
src/nvme/tree.h +8 -0
index 9f382e9c..a9082f8e 100644
@@ -867,6 +867,14 @@ const char *nvme_path_get_sysfs_dir(nvme_path_t p);
  */
 const char *nvme_path_get_ana_state(nvme_path_t p);
 
+/**
+ * nvme_path_get_queue_depth() - Queue depth of an nvme_path_t object
+ * @p: &nvme_path_t object
+ *
+ * Return: Queue depth of @p
+ */
+int nvme_path_get_queue_depth(nvme_path_t p);
+
 /**
  * nvme_path_get_ctrl() - Parent controller of an nvme_path_t object
  * @p:	&nvme_path_t object
Signature · 1 line
2.49.0
#1 lore raw

PATCH v2 1/4 tree: add support for discovering nvme paths using sysfs multipath link

In reply to Nilay Shroff

With the upcoming Linux kernel v6.15, NVMe native multipath now provides a simplified mechanism for discovering all paths to a shared namespace through sysfs.

A new "multipath" directory is created under each NVMe head namespace device in "/sys/block/<head>/multipath/". This directory contains symlinks to all namespace path devices that access the same shared namespace.

For example, consider a shared namespace accessible via two paths under nvme-subsys1:

nvme-subsys1 - NQN=nqn.1994-11.com.samsung:nvme:PM1735a:2.5-inch:S6RTNE0R900057

    hostnqn=nqn.2014-08.org.nvmexpress:uuid:41528538-e8ad-4eaf-84a7-9c552917d988
\
 +- ns 1
 \
  +- nvme0 pcie 052e:78:00.0 live optimized
  +- nvme1 pcie 058e:78:00.0 live optimized

The head device /dev/nvme1n1 will now have the following structure:

/sys/block/nvme1n1/multipath/

├── nvme1c0n1 -> ../../../../../pci052e:78/052e:78:00.0/nvme/nvme0/nvme1c0n1
└── nvme1c1n1 -> ../../../../../pci058e:78/058e:78:00.0/nvme/nvme1/nvme1c1n1

This clearly shows that namespace 1 is accessible through both nvme1c0n1 and nvme1c1n1. This new sysfs structure significantly simplifies multipath discovery and management, making it easier for tools and scripts to enumerate and manage NVMe multipath configurations. So leverage this functionality to update the path links for a shared NVMe namespace, simplifying path discovery and management.

This change adds a new struct nvme_ns_head to represent the head of a shared namespace. It contains a list head linking together struct nvme_path objects, where each path corresponds to a shared namespace instance. Additionally, struct nvme_ns has been updated to reference its associated nvme_ns_head, enabling straightforward traversal of all paths to a shared NVMe namespace.

---
 src/nvme/filters.c |   6 ++
 src/nvme/filters.h |   9 +++
 src/nvme/private.h |   9 ++-
 src/nvme/tree.c    | 162 +++++++++++++++++++++++++++++++--------------
 src/nvme/tree.h    |   9 +++
 5 files changed, 143 insertions(+), 52 deletions(-)
src/nvme/filters.c +6 -0
index ceaba68f..4a8829db 100644
@@ -105,3 +105,9 @@ int nvme_scan_ctrl_namespaces(nvme_ctrl_t c, struct dirent ***ns)
 	return scandir(nvme_ctrl_get_sysfs_dir(c), ns,
 		       nvme_namespace_filter, alphasort);
 }
+
+int nvme_scan_ns_head_paths(nvme_ns_head_t head, struct dirent ***paths)
+{
+	return scandir(nvme_ns_head_get_sysfs_dir(head), paths,
+		       nvme_paths_filter, alphasort);
+}
src/nvme/filters.h +9 -0
index 4ceeffd5..9e9dbb95 100644
@@ -94,4 +94,13 @@ int nvme_scan_ctrl_namespace_paths(nvme_ctrl_t c, struct dirent ***paths);
  */
 int nvme_scan_ctrl_namespaces(nvme_ctrl_t c, struct dirent ***ns);
 
+/**
+ * nvme_scan_ns_head_paths() - Scan for namespace paths
+ * @head: Namespace head node to scan
+ * @paths : Pointer to array of dirents
+ *
+ * Return: number of entries in @ents
+ */
+int nvme_scan_ns_head_paths(nvme_ns_head_t head, struct dirent ***paths);
+
 #endif /* _LIBNVME_FILTERS_H */
src/nvme/private.h +8 -1
index 33cdd555..f45c5823 100644
@@ -36,12 +36,19 @@ struct nvme_path {
 	int grpid;
 };
 
+struct nvme_ns_head {
+	struct list_head paths;
+	struct nvme_ns *n;
+
+	char *sysfs_dir;
+};
+
 struct nvme_ns {
 	struct list_node entry;
-	struct list_head paths;
 
 	struct nvme_subsystem *s;
 	struct nvme_ctrl *c;
+	struct nvme_ns_head *head;
 
 	int fd;
 	__u32 nsid;
src/nvme/tree.c +111 -51

249 lines not on this page · raw message

src/nvme/tree.h +9 -0
index 25d4b31b..9f382e9c 100644
@@ -27,6 +27,7 @@
  */
 
 typedef struct nvme_ns *nvme_ns_t;
+typedef struct nvme_ns_head *nvme_ns_head_t;
 typedef struct nvme_path *nvme_path_t;
 typedef struct nvme_ctrl *nvme_ctrl_t;
 typedef struct nvme_subsystem *nvme_subsystem_t;
@@ -1091,6 +1092,14 @@ void nvme_ctrl_set_dhchap_host_key(nvme_ctrl_t c, const char *key);
  */
 const char *nvme_ctrl_get_dhchap_key(nvme_ctrl_t c);
 
+/**
+ * nvme_ns_head_get_sysfs_dir() - sysfs dir of namespave head
+ * @head: namespace head instance
+ *
+ * Returns: sysfs directory name of @head
+ */
+const char *nvme_ns_head_get_sysfs_dir(nvme_ns_head_t head);
+
 /**
  * nvme_ctrl_set_dhchap_key() - Set controller key
  * @c:		Controller for which the key should be set
Signature · 1 line
2.49.0
#9 lore raw

PATCH v2 3/4 tree: add attribute numa_nodes for NVMe path object

In reply to Nilay Shroff

Add a new attribute named "numa_nodes" under the NVMe path object. This attribute is used by the iopolicy "numa". The numa_nodes value is stored for each NVMe path and represents the NUMA node(s) associated with it. When the iopolicy is set to "numa", I/O traffic originating from a given NUMA node will be forwarded through the corresponding NVMe path.

The numa_nodes attribute is useful for observing which NVMe path the kernel would choose for I/O forwarding based on NUMA affinity. To support this, export the attribute in libnvme.map so it can be accessed via nvme-cli.

---
 src/libnvme.map    |  1 +
 src/nvme/private.h |  1 +
 src/nvme/tree.c    | 10 ++++++++++
 src/nvme/tree.h    |  8 ++++++++
 4 files changed, 20 insertions(+)
src/libnvme.map +1 -0
index e53fad6b..ffadc647 100644
@@ -317,6 +317,7 @@ LIBNVME_1_0 {
 		nvme_path_get_ctrl;
 		nvme_path_get_name;
 		nvme_path_get_ns;
+		nvme_path_get_numa_nodes;
 		nvme_path_get_queue_depth;
 		nvme_path_get_sysfs_dir;
 		nvme_paths_filter;
src/nvme/private.h +1 -0
index f94276e2..5bfa027b 100644
@@ -33,6 +33,7 @@ struct nvme_path {
 	char *name;
 	char *sysfs_dir;
 	char *ana_state;
+	char *numa_nodes;
 	int grpid;
 	int queue_depth;
 };
src/nvme/tree.c +10 -0
index b7a38a07..ddb6fd70 100644
@@ -913,6 +913,11 @@ const char *nvme_path_get_ana_state(nvme_path_t p)
 	return p->ana_state;
 }
 
+const char *nvme_path_get_numa_nodes(nvme_path_t p)
+{
+	return p->numa_nodes;
+}
+
 void nvme_free_path(struct nvme_path *p)
 {
 	list_del_init(&p->entry);
@@ -920,6 +925,7 @@ void nvme_free_path(struct nvme_path *p)
 	free(p->name);
 	free(p->sysfs_dir);
 	free(p->ana_state);
+	free(p->numa_nodes);
 	free(p);
 }
 
@@ -955,6 +961,10 @@ static int nvme_ctrl_scan_path(nvme_root_t r, struct nvme_ctrl *c, char *name)
 	if (!p->ana_state)
 		p->ana_state = strdup("optimized");
 
+	p->numa_nodes = nvme_get_path_attr(p, "numa_nodes");
+	if (!p->numa_nodes)
+		p->numa_nodes = strdup("-1");
+
 	grpid = nvme_get_path_attr(p, "ana_grpid");
 	if (grpid) {
 		sscanf(grpid, "%d", &p->grpid);
src/nvme/tree.h +8 -0
index a9082f8e..f6116c2b 100644
@@ -867,6 +867,14 @@ const char *nvme_path_get_sysfs_dir(nvme_path_t p);
  */
 const char *nvme_path_get_ana_state(nvme_path_t p);
 
+/**
+ * nvme_path_get_numa_nodes() - NUMA nodes of an nvme_path_t object
+ * @p : &nvme_path_t object
+ *
+ * Return: NUMA nodes associated to @p
+ */
+const char *nvme_path_get_numa_nodes(nvme_path_t p);
+
 /**
  * nvme_path_get_queue_depth() - Queue depth of an nvme_path_t object
  * @p: &nvme_path_t object
Signature · 1 line
2.49.0
#11 lore raw

PATCH v2 4/4 test: extend sysfs tree dump test

In reply to Nilay Shroff

In order to validate multipath link support extend the existing sysfs tree dump test.

The updated tree-pcie.tar.xz archive includes a sysfs dump with two NVMe subsystems. The nvme-subsys0 simulates a single-port NVMe disk with a private namespace that does not support multipath. And another nvme-subsys1 simulates a dual-port NVMe disk with a shared namespace attached to multiple controllers.

The json tree dump is also updated to include NVMe namespace path information. Since the archive content has changed, the expected output file tree-pcie.out is also updated to reflect the new structure.

---
 src/nvme/json.c                  |  90 +++++++++++++++++++++++++++----
 test/sysfs/data/tree-pcie.out    |  53 ++++++++++++++----
 test/sysfs/data/tree-pcie.tar.xz | Bin 19712 -> 12656 bytes
 3 files changed, 124 insertions(+), 19 deletions(-)
src/nvme/json.c +81 -9
index af28bd48..a2fe8173 100644
@@ -533,25 +533,97 @@ static void json_dump_ctrl(struct json_object *ctrl_array, nvme_ctrl_t c)
 	json_object_array_add(ctrl_array, ctrl_obj);
 }
 
+static unsigned int json_dump_subsys_multipath(nvme_subsystem_t s,
+				struct json_object *ns_array)
+{
+	nvme_ns_t n;
+	nvme_path_t p;
+	unsigned int i = 0;
+
+	nvme_subsystem_for_each_ns(s, n) {
+		struct json_object *ns_obj;
+		struct json_object *path_array;
+
+		ns_obj = json_object_new_object();
+		json_object_object_add(ns_obj, "nsid",
+				json_object_new_int(nvme_ns_get_nsid(n)));
+		json_object_object_add(ns_obj, "name",
+				json_object_new_string(nvme_ns_get_name(n)));
+
+		path_array = json_object_new_array();
+		nvme_namespace_for_each_path(n, p) {
+			struct json_object *path_obj;
+			struct json_object *ctrl_array;
+			nvme_ctrl_t c;
+
+			path_obj = json_object_new_object();
+			json_object_object_add(path_obj, "path",
+				json_object_new_string(nvme_path_get_name(p)));
+			json_object_object_add(path_obj, "ANAState",
+				json_object_new_string(nvme_path_get_ana_state(p)));
+			json_object_object_add(path_obj, "NUMANodes",
+				json_object_new_string(nvme_path_get_numa_nodes(p)));
+			json_object_object_add(path_obj, "qdepth",
+				json_object_new_int(nvme_path_get_queue_depth(p)));
+
+			c = nvme_path_get_ctrl(p);
+			ctrl_array = json_object_new_array();
+			json_dump_ctrl(ctrl_array, c);
+			json_object_object_add(path_obj, "controller", ctrl_array);
+			json_object_array_add(path_array, path_obj);
+		}
+		json_object_object_add(ns_obj, "paths", path_array);
+		json_object_array_add(ns_array, ns_obj);
+		i++;
+	}
+	return i;
+}
+
+static void json_dump_subsys_non_multipath(nvme_subsystem_t s,
+		struct json_object *ns_array)
+{
+	nvme_ctrl_t c;
+	nvme_ns_t n;
+
+	nvme_subsystem_for_each_ctrl(s, c) {
+		nvme_ctrl_for_each_ns(c, n) {
+			struct json_object *ctrl_array;
+			struct json_object *ns_obj;
+
+			ns_obj = json_object_new_object();
+			json_object_object_add(ns_obj, "nsid",
+				json_object_new_int(nvme_ns_get_nsid(n)));
+			json_object_object_add(ns_obj, "name",
+				json_object_new_string(nvme_ns_get_name(n)));
+
+			ctrl_array = json_object_new_array();
+			json_dump_ctrl(ctrl_array, c);
+			json_object_object_add(ns_obj, "controller", ctrl_array);
+
+			json_object_array_add(ns_array, ns_obj);
+		}
+	}
+}
+
 static void json_dump_subsys(struct json_object *subsys_array,
 			       nvme_subsystem_t s)
 {
-	nvme_ctrl_t c;
 	struct json_object *subsys_obj = json_object_new_object();
-	struct json_object *ctrl_array;
+	struct json_object *ns_array;
 
 	json_object_object_add(subsys_obj, "name",
 			       json_object_new_string(nvme_subsystem_get_name(s)));
 	json_object_object_add(subsys_obj, "nqn",
 			       json_object_new_string(nvme_subsystem_get_nqn(s)));
-	ctrl_array = json_object_new_array();
-	nvme_subsystem_for_each_ctrl(s, c) {
-		json_dump_ctrl(ctrl_array, c);
-	}
-	if (json_object_array_length(ctrl_array))
-		json_object_object_add(subsys_obj, "controllers", ctrl_array);
+
+	ns_array = json_object_new_array();
+	if (!json_dump_subsys_multipath(s, ns_array))
+		json_dump_subsys_non_multipath(s, ns_array);
+
+	if (json_object_array_length(ns_array))
+		json_object_object_add(subsys_obj, "namespaces", ns_array);
 	else
-		json_object_put(ctrl_array);
+		json_object_put(ns_array);
 	json_object_array_add(subsys_array, subsys_obj);
 }
 
test/sysfs/data/tree-pcie.out +43 -10

66 lines not on this page · raw message

test/sysfs/data/tree-pcie.tar.xz binary +0 -0
index ee11fdeb0108589b954eb03543665e3a10df0e95..d99d2b943248cc2d7586c084ac639b613524cd2f 100644
GIT binary patch
Signature · 1 line
2.49.0
#2 lore raw

Re: [PATCHv2 1/4] tree: add support for discovering nvme paths using sysfs multipath link

In reply to Nilay Shroff
Quoted 94 lines · Nilay Shroff

With the upcoming Linux kernel v6.15, NVMe native multipath now provides a simplified mechanism for discovering all paths to a shared namespace through sysfs.

A new "multipath" directory is created under each NVMe head namespace device in "/sys/block/<head>/multipath/". This directory contains symlinks to all namespace path devices that access the same shared namespace.

For example, consider a shared namespace accessible via two paths under nvme-subsys1:

nvme-subsys1 - NQN=nqn.1994-11.com.samsung:nvme:PM1735a:2.5-inch:S6RTNE0R900057

     hostnqn=nqn.2014-08.org.nvmexpress:uuid:41528538-e8ad-4eaf-84a7-9c552917d988
\
  +- ns 1
  \
   +- nvme0 pcie 052e:78:00.0 live optimized
   +- nvme1 pcie 058e:78:00.0 live optimized

The head device /dev/nvme1n1 will now have the following structure:

/sys/block/nvme1n1/multipath/

├── nvme1c0n1 -> ../../../../../pci052e:78/052e:78:00.0/nvme/nvme0/nvme1c0n1
└── nvme1c1n1 -> ../../../../../pci058e:78/058e:78:00.0/nvme/nvme1/nvme1c1n1

This clearly shows that namespace 1 is accessible through both nvme1c0n1 and nvme1c1n1. This new sysfs structure significantly simplifies multipath discovery and management, making it easier for tools and scripts to enumerate and manage NVMe multipath configurations. So leverage this functionality to update the path links for a shared NVMe namespace, simplifying path discovery and management.

This change adds a new struct nvme_ns_head to represent the head of a shared namespace. It contains a list head linking together struct nvme_path objects, where each path corresponds to a shared namespace instance. Additionally, struct nvme_ns has been updated to reference its associated nvme_ns_head, enabling straightforward traversal of all paths to a shared NVMe namespace.

---
  src/nvme/filters.c |   6 ++
  src/nvme/filters.h |   9 +++
  src/nvme/private.h |   9 ++-
  src/nvme/tree.c    | 162 +++++++++++++++++++++++++++++++--------------
  src/nvme/tree.h    |   9 +++
  5 files changed, 143 insertions(+), 52 deletions(-)
src/nvme/filters.c +6 -0
index ceaba68f..4a8829db 100644
@@ -105,3 +105,9 @@ int nvme_scan_ctrl_namespaces(nvme_ctrl_t c, struct dirent ***ns)
  	return scandir(nvme_ctrl_get_sysfs_dir(c), ns,
  		       nvme_namespace_filter, alphasort);
  }
+
+int nvme_scan_ns_head_paths(nvme_ns_head_t head, struct dirent ***paths)
+{
+	return scandir(nvme_ns_head_get_sysfs_dir(head), paths,
+		       nvme_paths_filter, alphasort);
+}
src/nvme/filters.h +9 -0
index 4ceeffd5..9e9dbb95 100644
@@ -94,4 +94,13 @@ int nvme_scan_ctrl_namespace_paths(nvme_ctrl_t c, struct dirent ***paths);
   */
  int nvme_scan_ctrl_namespaces(nvme_ctrl_t c, struct dirent ***ns);
 
+/**
+ * nvme_scan_ns_head_paths() - Scan for namespace paths
+ * @head: Namespace head node to scan
+ * @paths : Pointer to array of dirents
+ *
+ * Return: number of entries in @ents
+ */
+int nvme_scan_ns_head_paths(nvme_ns_head_t head, struct dirent ***paths);
+
  #endif /* _LIBNVME_FILTERS_H */
src/nvme/private.h +7 -0
index 33cdd555..f45c5823 100644
@@ -36,12 +36,19 @@ struct nvme_path {
  	int grpid;
  };
 
+struct nvme_ns_head {
+	struct list_head paths;
+	struct nvme_ns *n;
+
+	char *sysfs_dir;
+};
+

src/nvme/private.h struct nvme_path { · line 54

Quoted 10 lines
src/nvme/private.h +1 -1
  struct nvme_ns {
  	struct list_node entry;
-	struct list_head paths;
 
  	struct nvme_subsystem *s;
  	struct nvme_ctrl *c;
+	struct nvme_ns_head *head;
 
  	int fd;
  	__u32 nsid;

Why isn't 'nvme_subsystem' moved to 'nvme_ns_head'? That certainly is how the kernel structures are laid out, and it would make sense to follow it here...

Quoted 205 lines
src/nvme/tree.c +96 -24
index b0a4696f..bd7fb53e 100644
@@ -564,12 +564,12 @@ nvme_ns_t nvme_subsystem_next_ns(nvme_subsystem_t s, nvme_ns_t n)
 
  nvme_path_t nvme_namespace_first_path(nvme_ns_t ns)
  {
-	return list_top(&ns->paths, struct nvme_path, nentry);
+	return list_top(&ns->head->paths, struct nvme_path, nentry);
  }
 
  nvme_path_t nvme_namespace_next_path(nvme_ns_t ns, nvme_path_t p)
  {
-	return p ? list_next(&ns->paths, p, nentry) : NULL;
+	return p ? list_next(&ns->head->paths, p, nentry) : NULL;
  }
 
  static void __nvme_free_ns(struct nvme_ns *n)
@@ -579,6 +579,8 @@ static void __nvme_free_ns(struct nvme_ns *n)
  	free(n->generic_name);
  	free(n->name);
  	free(n->sysfs_dir);
+	free(n->head->sysfs_dir);
+	free(n->head);
  	free(n);
  }
 
@@ -916,25 +918,6 @@ void nvme_free_path(struct nvme_path *p)
  	free(p);
  }
 
-static void nvme_subsystem_set_path_ns(nvme_subsystem_t s, nvme_path_t p)
-{
-	char n_name[32] = { };
-	int i, c, nsid, ret;
-	nvme_ns_t n;
-
-	ret = sscanf(nvme_path_get_name(p), "nvme%dc%dn%d", &i, &c, &nsid);
-	if (ret != 3)
-		return;
-
-	sprintf(n_name, "nvme%dn%d", i, nsid);
-	nvme_subsystem_for_each_ns(s, n) {
-		if (!strcmp(n_name, nvme_ns_get_name(n))) {
-			list_add_tail(&n->paths, &p->nentry);
-			p->n = n;
-		}
-	}
-}
-
  static int nvme_ctrl_scan_path(nvme_root_t r, struct nvme_ctrl *c, char *name)
  {
  	struct nvme_path *p;
@@ -973,7 +956,6 @@ static int nvme_ctrl_scan_path(nvme_root_t r, struct nvme_ctrl *c, char *name)
  	}
 
  	list_node_init(&p->nentry);
-	nvme_subsystem_set_path_ns(c->s, p);
  	list_node_init(&p->entry);
  	list_add_tail(&c->paths, &p->entry);
  	return 0;
@@ -2250,8 +2232,8 @@ nvme_ctrl_t nvme_scan_ctrl(nvme_root_t r, const char *name)
  		return NULL;
 
  	path = NULL;
-	nvme_ctrl_scan_namespaces(r, c);
  	nvme_ctrl_scan_paths(r, c);
+	nvme_ctrl_scan_namespaces(r, c);
  	return c;
  }
 
@@ -2323,6 +2305,11 @@ const char *nvme_ns_get_sysfs_dir(nvme_ns_t n)
  	return n->sysfs_dir;
  }
 
+const char *nvme_ns_head_get_sysfs_dir(nvme_ns_head_t head)
+{
+	return head->sysfs_dir;
+}
+
  const char *nvme_ns_get_name(nvme_ns_t n)
  {
  	return n->name;
@@ -2749,7 +2736,11 @@ static void nvme_ns_set_generic_name(struct nvme_ns *n, const char *name)
 
  static nvme_ns_t nvme_ns_open(const char *sys_path, const char *name)
  {
+	int ret;
  	struct nvme_ns *n;
+	struct nvme_ns_head *head;
+	struct stat arg;
+	_cleanup_free_ char *path = NULL;
 
  	n = calloc(1, sizeof(*n));
  	if (!n) {
@@ -2757,6 +2748,32 @@ static nvme_ns_t nvme_ns_open(const char *sys_path, const char *name)
  		return NULL;
  	}
 
+	head = calloc(1, sizeof(*head));
+	if (!head) {
+		errno = ENOMEM;
+		free(n);
+		return NULL;
+	}
+
+	head->n = n;
+	list_head_init(&head->paths);
+	ret = asprintf(&path, "%s/%s", sys_path, "multipath");
+	if (ret < 0) {
+		errno = ENOMEM;
+		goto free_ns_head;
+	}
+	/*
+	 * The sysfs-dir "multipath" is available only when nvme multipath
+	 * is configured and we're running kernel version >= 6.14.
+	 */
+	ret = stat(path, &arg);
+	if (ret == 0) {
+		head->sysfs_dir = path;
+		path = NULL;
+	} else
+		head->sysfs_dir = NULL;
+
+	n->head = head;
  	n->fd = -1;
  	n->name = strdup(name);
 
@@ -2765,15 +2782,17 @@ static nvme_ns_t nvme_ns_open(const char *sys_path, const char *name)
  	if (nvme_ns_init(sys_path, n) != 0)
  		goto free_ns;
 
-	list_head_init(&n->paths);
  	list_node_init(&n->entry);
 
  	nvme_ns_release_fd(n); /* Do not leak fds */
+
  	return n;
 
  free_ns:
  	free(n->generic_name);
  	free(n->name);
+free_ns_head:
+	free(head);
  	free(n);
  	return NULL;
  }
@@ -2836,6 +2855,71 @@ nvme_ns_t nvme_scan_namespace(const char *name)
  	return __nvme_scan_namespace(nvme_ns_sysfs_dir(), name);
  }
 
+
+static void nvme_ns_head_scan_path(nvme_subsystem_t s, nvme_ns_t n, char *name)
+{
+	nvme_ctrl_t c;
+	nvme_path_t p;
+
+	nvme_subsystem_for_each_ctrl(s, c) {
+		nvme_ctrl_for_each_path(c, p) {
+			if (!strcmp(nvme_path_get_name(p), name)) {
+				list_add_tail(&n->head->paths, &p->nentry);
+				p->n = n;
+				return;
+			}
+		}
+	}
+}
+
+static void nvme_subsystem_set_ns_path(nvme_subsystem_t s, nvme_ns_t n)
+{
+	struct nvme_ns_head *head = n->head;
+
+	if (nvme_ns_head_get_sysfs_dir(head)) {
+		struct dirents paths = {};
+		int i;
+
+		/*
+		 * When multipath is configured on kernel version >= 6.14,
+		 * we use multipath sysfs link to get each path of a namespace.
+		 */
+		paths.num = nvme_scan_ns_head_paths(head, &paths.ents);
+
+		for (i = 0; i < paths.num; i++)
+			nvme_ns_head_scan_path(s, n, paths.ents[i]->d_name);
+	} else {
+		nvme_ctrl_t c;
+		nvme_path_t p;
+		int ns_ctrl, ns_nsid, ret;
+
+		/*
+		 * If multipath is not configured or we're running on kernel
+		 * version < 6.14, fallback to the old way.
+		 */
+		ret = sscanf(nvme_ns_get_name(n), "nvme%dn%d",
+				&ns_ctrl, &ns_nsid);
+		if (ret != 2)
+			return;
+
+		nvme_subsystem_for_each_ctrl(s, c) {
+			nvme_ctrl_for_each_path(c, p) {
+				int p_subsys, p_ctrl, p_nsid;
+
+				ret = sscanf(nvme_path_get_name(p),
+					     "nvme%dc%dn%d",

src/nvme/tree.c nvme_ns_t nvme_scan_namespace(const char *name) · line 2920

Quoted 10 lines
src/nvme/tree.c +10 -0
+					     &p_subsys, &p_ctrl, &p_nsid);
+				if (ret != 3)
+					continue;
+				if (ns_ctrl == p_subsys && ns_nsid == p_nsid) {
+					list_add_tail(&head->paths, &p->nentry);
+					p->n = n;
+				}
+			}
+		}
+	}

Can't you just use the existing nvme_subsystem_set_path_ns() function here instead of deleting and open-code it?

Quoted 31 lines
src/nvme/tree.c +3 -20
+}
+
  static int nvme_ctrl_scan_namespace(nvme_root_t r, struct nvme_ctrl *c,
  				    char *name)
  {
@@ -2861,33 +2945,9 @@ static int nvme_ctrl_scan_namespace(nvme_root_t r, struct nvme_ctrl *c,
  	n->s = c->s;
  	n->c = c;
  	list_add_tail(&c->namespaces, &n->entry);
-	return 0;
-}
-
-static void nvme_subsystem_set_ns_path(nvme_subsystem_t s, nvme_ns_t n)
-{
-	nvme_ctrl_t c;
-	nvme_path_t p;
-	int ns_ctrl, ns_nsid, ret;
-
-	ret = sscanf(nvme_ns_get_name(n), "nvme%dn%d", &ns_ctrl, &ns_nsid);
-	if (ret != 2)
-		return;
+	nvme_subsystem_set_ns_path(c->s, n);
 
-	nvme_subsystem_for_each_ctrl(s, c) {
-		nvme_ctrl_for_each_path(c, p) {
-			int p_subsys, p_ctrl, p_nsid;
-
-			ret = sscanf(nvme_path_get_name(p), "nvme%dc%dn%d",
-				     &p_subsys, &p_ctrl, &p_nsid);
-			if (ret != 3)
-				continue;

src/nvme/tree.c static int nvme_ctrl_scan_namespace(nvme_root_t r, struct nvme_ctrl *c, · line 2953

Quoted 10 lines
src/nvme/tree.c +1 -6
-			if (ns_ctrl == p_subsys && ns_nsid == p_nsid) {
-				list_add_tail(&n->paths, &p->nentry);
-				p->n = n;
-			}
-		}
-	}
+	return 0;
  }
 
  static int nvme_subsystem_scan_namespace(nvme_root_t r, nvme_subsystem_t s,

Similar here; better to use the existing functions.

Quoted 26 lines
src/nvme/tree.c +1 -1
@@ -2917,7 +2977,7 @@ static int nvme_subsystem_scan_namespace(nvme_root_t r, nvme_subsystem_t s,
  			list_del_init(&p->nentry);
  			p->n = NULL;
  		}
-		list_head_init(&_n->paths);
+		list_head_init(&_n->head->paths);
  		__nvme_free_ns(_n);
  	}
  	n->s = s;
src/nvme/tree.h +2 -0
index 25d4b31b..9f382e9c 100644
@@ -27,6 +27,7 @@
   */
 
  typedef struct nvme_ns *nvme_ns_t;
+typedef struct nvme_ns_head *nvme_ns_head_t;
  typedef struct nvme_path *nvme_path_t;
  typedef struct nvme_ctrl *nvme_ctrl_t;
  typedef struct nvme_subsystem *nvme_subsystem_t;
@@ -1091,6 +1092,14 @@ void nvme_ctrl_set_dhchap_host_key(nvme_ctrl_t c, const char *key);
   */
  const char *nvme_ctrl_get_dhchap_key(nvme_ctrl_t c);
 
+/**

src/nvme/tree.h void nvme_ctrl_set_dhchap_host_key(nvme_ctrl_t c, const char *key); · line 1105

Quoted 10 lines
src/nvme/tree.h +7 -0
+ * nvme_ns_head_get_sysfs_dir() - sysfs dir of namespave head
+ * @head: namespace head instance
+ *
+ * Returns: sysfs directory name of @head
+ */
+const char *nvme_ns_head_get_sysfs_dir(nvme_ns_head_t head);
+
  /**
   * nvme_ctrl_set_dhchap_key() - Set controller key
   * @c:		Controller for which the key should be set

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
#5 lore raw

Re: [PATCHv2 2/4] tree: add queue-depth attribute for nvme path object

In reply to Nilay Shroff
Quoted 87 lines · Nilay Shroff

Add a new attribute named "queue_depth" under the NVMe path object. This attribute is used by the iopolicy "queue-depth", which was introduced in kernel v6.11. However, the corresponding sysfs attribute for queue depth was only added in kernel v6.14.

The queue_depth value can be useful for observing which paths are selected for I/O forwarding, based on the depth of each path. To support this, export the attribute in libnvme.map so it can be accessed via nvme-cli.

---
  src/libnvme.map    |  1 +
  src/nvme/private.h |  1 +
  src/nvme/tree.c    | 12 +++++++++++-
  src/nvme/tree.h    |  8 ++++++++
  4 files changed, 21 insertions(+), 1 deletion(-)
src/libnvme.map +1 -0
index 4314705f..e53fad6b 100644
@@ -317,6 +317,7 @@ LIBNVME_1_0 {
  		nvme_path_get_ctrl;
  		nvme_path_get_name;
  		nvme_path_get_ns;
+		nvme_path_get_queue_depth;
  		nvme_path_get_sysfs_dir;
  		nvme_paths_filter;
  		nvme_read_config;
src/nvme/private.h +1 -0
index f45c5823..f94276e2 100644
@@ -34,6 +34,7 @@ struct nvme_path {
  	char *sysfs_dir;
  	char *ana_state;
  	int grpid;
+	int queue_depth;
  };
 
  struct nvme_ns_head {
src/nvme/tree.c +11 -1
index bd7fb53e..b7a38a07 100644
@@ -903,6 +903,11 @@ const char *nvme_path_get_name(nvme_path_t p)
  	return p->name;
  }
 
+int nvme_path_get_queue_depth(nvme_path_t p)
+{
+	return p->queue_depth;
+}
+
  const char *nvme_path_get_ana_state(nvme_path_t p)
  {
  	return p->ana_state;
@@ -921,7 +926,7 @@ void nvme_free_path(struct nvme_path *p)
  static int nvme_ctrl_scan_path(nvme_root_t r, struct nvme_ctrl *c, char *name)
  {
  	struct nvme_path *p;
-	_cleanup_free_ char *path = NULL, *grpid = NULL;
+	_cleanup_free_ char *path = NULL, *grpid = NULL, *queue_depth = NULL;
  	int ret;
 
  	nvme_msg(r, LOG_DEBUG, "scan controller %s path %s\n",
@@ -955,6 +960,11 @@ static int nvme_ctrl_scan_path(nvme_root_t r, struct nvme_ctrl *c, char *name)
  		sscanf(grpid, "%d", &p->grpid);
  	}
 
+	queue_depth = nvme_get_path_attr(p, "queue_depth");
+	if (queue_depth) {
+		sscanf(queue_depth, "%d", &p->queue_depth);
+	}
+
  	list_node_init(&p->nentry);
  	list_node_init(&p->entry);
  	list_add_tail(&c->paths, &p->entry);
src/nvme/tree.h +1 -0
index 9f382e9c..a9082f8e 100644
@@ -867,6 +867,14 @@ const char *nvme_path_get_sysfs_dir(nvme_path_t p);
   */
  const char *nvme_path_get_ana_state(nvme_path_t p);
 
+/**

src/nvme/tree.h const char *nvme_path_get_sysfs_dir(nvme_path_t p); · line 880

Quoted 10 lines
src/nvme/tree.h +7 -0
+ * nvme_path_get_queue_depth() - Queue depth of an nvme_path_t object
+ * @p: &nvme_path_t object
+ *
+ * Return: Queue depth of @p
+ */
+int nvme_path_get_queue_depth(nvme_path_t p);
+
  /**
   * nvme_path_get_ctrl() - Parent controller of an nvme_path_t object
   * @p:	&nvme_path_t object

As discussed: saving the 'queue_depth' attribute is a bad idea, as it changes frequently.
I'd rather read the attribute value from sysfs whenever 'nvme_path_get_queue_depth' is called, and use the tree structure only to hold the pathname.

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Reviewed-by on v2 3/4
#10 lore raw

Re: [PATCHv2 3/4] tree: add attribute numa_nodes for NVMe path object

In reply to Nilay Shroff
Quoted 11 lines · Nilay Shroff

Add a new attribute named "numa_nodes" under the NVMe path object. This attribute is used by the iopolicy "numa". The numa_nodes value is stored for each NVMe path and represents the NUMA node(s) associated with it. When the iopolicy is set to "numa", I/O traffic originating from a given NUMA node will be forwarded through the corresponding NVMe path.

The numa_nodes attribute is useful for observing which NVMe path the kernel would choose for I/O forwarding based on NUMA affinity. To support this, export the attribute in libnvme.map so it can be accessed via nvme-cli.

Quoted 8 lines
---
  src/libnvme.map    |  1 +
  src/nvme/private.h |  1 +
  src/nvme/tree.c    | 10 ++++++++++
  src/nvme/tree.h    |  8 ++++++++
  4 files changed, 20 insertions(+)

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Reviewed-by on v2 4/4
#12 lore raw

Re: [PATCHv2 4/4] test: extend sysfs tree dump test

In reply to Nilay Shroff
Quoted 13 lines · Nilay Shroff

In order to validate multipath link support extend the existing sysfs tree dump test.

The updated tree-pcie.tar.xz archive includes a sysfs dump with two NVMe subsystems. The nvme-subsys0 simulates a single-port NVMe disk with a private namespace that does not support multipath. And another nvme-subsys1 simulates a dual-port NVMe disk with a shared namespace attached to multiple controllers.

The json tree dump is also updated to include NVMe namespace path information. Since the archive content has changed, the expected output file tree-pcie.out is also updated to reflect the new structure.

Quoted 7 lines
---
  src/nvme/json.c                  |  90 +++++++++++++++++++++++++++----
  test/sysfs/data/tree-pcie.out    |  53 ++++++++++++++----
  test/sysfs/data/tree-pcie.tar.xz | Bin 19712 -> 12656 bytes
  3 files changed, 124 insertions(+), 19 deletions(-)

Not a big fan of having tar.gz files in git. Can't we create it on the fly and only store text files?

Otherwise:

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
#13 lore raw

Re: [PATCHv2 4/4] test: extend sysfs tree dump test

Quoted 2 lines · Hannes Reinecke

Not a big fan of having tar.gz files in git. Can't we create it on the fly and only store text files?

Creating is not really an option, we could just store test data in the tree directly. I don't expect that the size of the repo is increasing a lot as the objects in git are also compressed. My main concern is that we have a lot of new files:

$ tar tf config-pcie.tar.xz
sys/class/nvme/
sys/class/nvme/nvme1
sys/class/nvme/nvme0
sys/devices/pci0000:00/0000:00:06.0/0000:0f:00.0/nvme/nvme0/
sys/devices/pci0000:00/0000:00:06.0/0000:0f:00.0/nvme/nvme0/uevent
sys/devices/pci0000:00/0000:00:06.0/0000:0f:00.0/nvme/nvme0/cntlid
sys/devices/pci0000:00/0000:00:06.0/0000:0f:00.0/nvme/nvme0/address
sys/devices/pci0000:00/0000:00:06.0/0000:0f:00.0/nvme/nvme0/nvme0c0n1/
sys/devices/pci0000:00/0000:00:06.0/0000:0f:00.0/nvme/nvme0/nvme0c0n1/uevent
sys/devices/pci0000:00/0000:00:06.0/0000:0f:00.0/nvme/nvme0/nvme0c0n1/ext_range
sys/devices/pci0000:00/0000:00:06.0/0000:0f:00.0/nvme/nvme0/nvme0c0n1/metadata_bytes
sys/devices/pci0000:00/0000:00:06.0/0000:0f:00.0/nvme/nvme0/nvme0c0n1/range
sys/devices/pci0000:00/0000:00:06.0/0000:0f:00.0/nvme/nvme0/nvme0c0n1/alignment_offset
sys/devices/pci0000:00/0000:00:06.0/0000:0f:00.0/nvme/nvme0/nvme0c0n1/diskseq
sys/devices/pci0000:00/0000:00:06.0/0000:0f:00.0/nvme/nvme0/nvme0c0n1/power/
[...]

Currently there are three tar.xz files. If the general mood is that there shouldn't be any tar.xz files around, I am fine to get rid of them.

#3 lore raw

Re: [PATCHv2 1/4] tree: add support for discovering nvme paths using sysfs multipath link

In reply to Hannes Reinecke
Quoted 97 lines · Hannes Reinecke
Quoted 96 lines · Nilay Shroff

With the upcoming Linux kernel v6.15, NVMe native multipath now provides a simplified mechanism for discovering all paths to a shared namespace through sysfs.

A new "multipath" directory is created under each NVMe head namespace device in "/sys/block/<head>/multipath/". This directory contains symlinks to all namespace path devices that access the same shared namespace.

For example, consider a shared namespace accessible via two paths under nvme-subsys1:

nvme-subsys1 - NQN=nqn.1994-11.com.samsung:nvme:PM1735a:2.5-inch:S6RTNE0R900057 hostnqn=nqn.2014-08.org.nvmexpress:uuid:41528538-e8ad-4eaf-84a7-9c552917d988 \
  +- ns 1
  \

   +- nvme0 pcie 052e:78:00.0 live optimized
   +- nvme1 pcie 058e:78:00.0 live optimized

The head device /dev/nvme1n1 will now have the following structure:

/sys/block/nvme1n1/multipath/

├── nvme1c0n1 -> ../../../../../pci052e:78/052e:78:00.0/nvme/nvme0/nvme1c0n1
└── nvme1c1n1 -> ../../../../../pci058e:78/058e:78:00.0/nvme/nvme1/nvme1c1n1

This clearly shows that namespace 1 is accessible through both nvme1c0n1 and nvme1c1n1. This new sysfs structure significantly simplifies multipath discovery and management, making it easier for tools and scripts to enumerate and manage NVMe multipath configurations. So leverage this functionality to update the path links for a shared NVMe namespace, simplifying path discovery and management.

This change adds a new struct nvme_ns_head to represent the head of a shared namespace. It contains a list head linking together struct nvme_path objects, where each path corresponds to a shared namespace instance. Additionally, struct nvme_ns has been updated to reference its associated nvme_ns_head, enabling straightforward traversal of all paths to a shared NVMe namespace.

---

  src/nvme/filters.c |   6 ++
  src/nvme/filters.h |   9 +++
  src/nvme/private.h |   9 ++-
  src/nvme/tree.c    | 162 +++++++++++++++++++++++++++++++--------------
  src/nvme/tree.h    |   9 +++
  5 files changed, 143 insertions(+), 52 deletions(-)
src/nvme/filters.c +6 -0
index ceaba68f..4a8829db 100644
@@ -105,3 +105,9 @@ int nvme_scan_ctrl_namespaces(nvme_ctrl_t c, struct dirent ***ns)
       return scandir(nvme_ctrl_get_sysfs_dir(c), ns,
                  nvme_namespace_filter, alphasort);
   }
+
+int nvme_scan_ns_head_paths(nvme_ns_head_t head, struct dirent ***paths)
+{
+    return scandir(nvme_ns_head_get_sysfs_dir(head), paths,
+               nvme_paths_filter, alphasort);
+}
src/nvme/filters.h +0 -0
index 4ceeffd5..9e9dbb95 100644
@@ -94,4 +94,13 @@ int nvme_scan_ctrl_namespace_paths(nvme_ctrl_t c, struct dirent ***paths);
    */
   int nvme_scan_ctrl_namespaces(nvme_ctrl_t c, struct dirent ***ns);

  +/**
+ * nvme_scan_ns_head_paths() - Scan for namespace paths
+ * @head: Namespace head node to scan
+ * @paths : Pointer to array of dirents
+ *
+ * Return: number of entries in @ents
+ */

src/nvme/filters.h +2 -0
+int nvme_scan_ns_head_paths(nvme_ns_head_t head, struct dirent ***paths);
+

  #endif /* _LIBNVME_FILTERS_H */

src/nvme/private.h +6 -1
index 33cdd555..f45c5823 100644
@@ -36,12 +36,19 @@ struct nvme_path {
       int grpid;
   };
   +struct nvme_ns_head {
+    struct list_head paths;
+    struct nvme_ns *n;
+
+    char *sysfs_dir;
+};
+
   struct nvme_ns {
       struct list_node entry;
-    struct list_head paths;
         struct nvme_subsystem *s;
Quoted 9 lines

src/nvme/private.h struct nvme_path { · line 54

Quoted 4 lines
src/nvme/private.h +1 -0
       struct nvme_ctrl *c;
+    struct nvme_ns_head *head;
         int fd;
       __u32 nsid;

Why isn't 'nvme_subsystem' moved to 'nvme_ns_head'? That certainly is how the kernel structures are laid out, and it would make sense to follow it here...

Ideally yes it could be. However for libnvme, struct nvme_ns actually represents namespace head node when multipath is configured. And struct nvme_subsystem could be directly accessed from struct nvme_ns as we maintain a pointer to the nvme_subsystem from nvme_ns.

For instance, take an example of a shared namespace attached to two controllers in a subsystem and so we've got following topology for this subsystem.

# nvme show-topology 
nvme-subsys2 - NQN=nqn.1994-11.com.samsung:nvme:PM1735a:2.5-inch:S6RTNE0R900057
               hostnqn=nqn.2014-08.org.nvmexpress:uuid:41528538-e8ad-4eaf-84a7-9c552917d988
\
 +- ns 1
 \
  +- nvme0 pcie 052e:78:00.0 live optimized
  +- nvme2 pcie 058e:78:00.0 live optimized

And multipath sysfs link to reach the shared namespace from head node (nvme2n1):

# tree /sys/class/nvme-subsystem/nvme-subsys2/nvme2n1/multipath/
/sys/class/nvme-subsystem/nvme-subsys2/nvme2n1/multipath/
├── nvme2c0n1 -> ../../../../../pci052e:78/052e:78:00.0/nvme/nvme0/nvme2c0n1
└── nvme2c2n1 -> ../../../../../pci058e:78/058e:78:00.0/nvme/nvme2/nvme2c2n1

If we parse above topology in libnvme then we get,

struct nvme_subsystem::name => nvme-subsys2
struct nvme_subsystem::namespaces => <list-of-struct nvme_ns-under-this-subsys>

//first path
struct nvme_ns_path::name => nvme2c0n1
struct nvme_ns_path::sysfs_dir => /sys/class/nvme/nvme0/nvme2c0n1/ //second path
struct nvme_ns_path::name => nvme2c2n1
struct nvme_ns_path::sysfs_dir => /sys/class/nvme/nvme2/nvme2c2n1/

struct nvme_ns_head::paths => struct nvme_ns_path (first path), struct nvme_ns_path (second path) struct nvme_ns_head::sysfs_dir =>/sys/class/nvme-subsystem/nvme-subsys2/nvme2n1/multipath

struct nvme_ns::name => nvme2n1
struct nvme_ns::sysfs_dir => /sys/class/nvme-subsystem/nvme-subsys2/nvme2n1/ struct nvme_ns::head => struct nvme_ns_head
struct nvme_ns::s => struct nvme_subsystem

So as we could see above, struct nvme_ns::s points to nvme_susbsystem and struct nvme_ns actually represents the head node.

Quoted 193 lines
Quoted 193 lines
src/nvme/tree.c +4 -2
index b0a4696f..bd7fb53e 100644
@@ -564,12 +564,12 @@ nvme_ns_t nvme_subsystem_next_ns(nvme_subsystem_t s, nvme_ns_t n)
     nvme_path_t nvme_namespace_first_path(nvme_ns_t ns)
   {
-    return list_top(&ns->paths, struct nvme_path, nentry);
+    return list_top(&ns->head->paths, struct nvme_path, nentry);
   }
     nvme_path_t nvme_namespace_next_path(nvme_ns_t ns, nvme_path_t p)
   {
-    return p ? list_next(&ns->paths, p, nentry) : NULL;
+    return p ? list_next(&ns->head->paths, p, nentry) : NULL;
   }
     static void __nvme_free_ns(struct nvme_ns *n)
@@ -579,6 +579,8 @@ static void __nvme_free_ns(struct nvme_ns *n)
       free(n->generic_name);
       free(n->name);
       free(n->sysfs_dir);
+    free(n->head->sysfs_dir);
+    free(n->head);
       free(n);
   }
   @@ -916,25 +918,6 @@ void nvme_free_path(struct nvme_path *p)

      free(p);
  }
  -static void nvme_subsystem_set_path_ns(nvme_subsystem_t s, nvme_path_t p)

src/nvme/tree.c +0 -18
-{
-    char n_name[32] = { };
-    int i, c, nsid, ret;
-    nvme_ns_t n;
-
-    ret = sscanf(nvme_path_get_name(p), "nvme%dc%dn%d", &i, &c, &nsid);
-    if (ret != 3)
-        return;
-
-    sprintf(n_name, "nvme%dn%d", i, nsid);
-    nvme_subsystem_for_each_ns(s, n) {
-        if (!strcmp(n_name, nvme_ns_get_name(n))) {
-            list_add_tail(&n->paths, &p->nentry);
-            p->n = n;
-        }
-    }
-}
-

  static int nvme_ctrl_scan_path(nvme_root_t r, struct nvme_ctrl *c, char *name) {
      struct nvme_path *p;

src/nvme/tree.c +1 -2
@@ -973,7 +956,6 @@ static int nvme_ctrl_scan_path(nvme_root_t r, struct nvme_ctrl *c, char *name)
       }
         list_node_init(&p->nentry);
-    nvme_subsystem_set_path_ns(c->s, p);
       list_node_init(&p->entry);
       list_add_tail(&c->paths, &p->entry);
       return 0;
@@ -2250,8 +2232,8 @@ nvme_ctrl_t nvme_scan_ctrl(nvme_root_t r, const char *name)
           return NULL;
         path = NULL;
-    nvme_ctrl_scan_namespaces(r, c);
       nvme_ctrl_scan_paths(r, c);
+    nvme_ctrl_scan_namespaces(r, c);
       return c;
   }
   @@ -2323,6 +2305,11 @@ const char *nvme_ns_get_sysfs_dir(nvme_ns_t n)
       return n->sysfs_dir;

  }
  +const char *nvme_ns_head_get_sysfs_dir(nvme_ns_head_t head)

src/nvme/tree.c +4 -0
+{
+    return head->sysfs_dir;
+}
+

  const char *nvme_ns_get_name(nvme_ns_t n)
  {
      return n->name;

src/nvme/tree.c +29 -0
@@ -2749,7 +2736,11 @@ static void nvme_ns_set_generic_name(struct nvme_ns *n, const char *name)
     static nvme_ns_t nvme_ns_open(const char *sys_path, const char *name)
   {
+    int ret;
       struct nvme_ns *n;
+    struct nvme_ns_head *head;
+    struct stat arg;
+    _cleanup_free_ char *path = NULL;
         n = calloc(1, sizeof(*n));
       if (!n) {
@@ -2757,6 +2748,32 @@ static nvme_ns_t nvme_ns_open(const char *sys_path, const char *name)
           return NULL;
       }
   +    head = calloc(1, sizeof(*head));
+    if (!head) {
+        errno = ENOMEM;
+        free(n);
+        return NULL;
+    }
+
+    head->n = n;
+    list_head_init(&head->paths);
+    ret = asprintf(&path, "%s/%s", sys_path, "multipath");
+    if (ret < 0) {
+        errno = ENOMEM;
+        goto free_ns_head;
+    }
+    /*
+     * The sysfs-dir "multipath" is available only when nvme multipath
+     * is configured and we're running kernel version >= 6.14.
+     */
+    ret = stat(path, &arg);
+    if (ret == 0) {
+        head->sysfs_dir = path;
+        path = NULL;
+    } else
+        head->sysfs_dir = NULL;
+
+    n->head = head;
       n->fd = -1;
       n->name = strdup(name);
   @@ -2765,15 +2782,17 @@ static nvme_ns_t nvme_ns_open(const char *sys_path, const char *name)

      if (nvme_ns_init(sys_path, n) != 0) goto free_ns;
  -    list_head_init(&n->paths); list_node_init(&n->entry); nvme_ns_release_fd(n); /* Do not leak fds */ +
      return n;
    free_ns:
      free(n->generic_name);
      free(n->name);
+free_ns_head:
+    free(head);
      free(n);
      return NULL;
  }

src/nvme/tree.c +0 -0
@@ -2836,6 +2855,71 @@ nvme_ns_t nvme_scan_namespace(const char *name)
       return __nvme_scan_namespace(nvme_ns_sysfs_dir(), name);
   }

  +

src/nvme/tree.c +57 -0
+static void nvme_ns_head_scan_path(nvme_subsystem_t s, nvme_ns_t n, char *name)
+{
+    nvme_ctrl_t c;
+    nvme_path_t p;
+
+    nvme_subsystem_for_each_ctrl(s, c) {
+        nvme_ctrl_for_each_path(c, p) {
+            if (!strcmp(nvme_path_get_name(p), name)) {
+                list_add_tail(&n->head->paths, &p->nentry);
+                p->n = n;
+                return;
+            }
+        }
+    }
+}
+
+static void nvme_subsystem_set_ns_path(nvme_subsystem_t s, nvme_ns_t n)
+{
+    struct nvme_ns_head *head = n->head;
+
+    if (nvme_ns_head_get_sysfs_dir(head)) {
+        struct dirents paths = {};
+        int i;
+
+        /*
+         * When multipath is configured on kernel version >= 6.14,
+         * we use multipath sysfs link to get each path of a namespace.
+         */
+        paths.num = nvme_scan_ns_head_paths(head, &paths.ents);
+
+        for (i = 0; i < paths.num; i++)
+            nvme_ns_head_scan_path(s, n, paths.ents[i]->d_name);
+    } else {
+        nvme_ctrl_t c;
+        nvme_path_t p;
+        int ns_ctrl, ns_nsid, ret;
+
+        /*
+         * If multipath is not configured or we're running on kernel
+         * version < 6.14, fallback to the old way.
+         */
+        ret = sscanf(nvme_ns_get_name(n), "nvme%dn%d",
+                &ns_ctrl, &ns_nsid);
+        if (ret != 2)
+            return;
+
+        nvme_subsystem_for_each_ctrl(s, c) {
+            nvme_ctrl_for_each_path(c, p) {
+                int p_subsys, p_ctrl, p_nsid;
+
+                ret = sscanf(nvme_path_get_name(p),
+                         "nvme%dc%dn%d",
+                         &p_subsys, &p_ctrl, &p_nsid);
+                if (ret != 3)
+                    continue;
+                if (ns_ctrl == p_subsys && ns_nsid == p_nsid) {
+                    list_add_tail(&head->paths, &p->nentry);
Quoted 9 lines
Quoted 5 lines

+                    p->n = n;
+                }
+            }
+        }
+    }

Can't you just use the existing nvme_subsystem_set_path_ns() function here instead of deleting and open-code it?

We can but then we don't levearge the newly added (efficient and probably more elegant) way of discovering the multipath information from the head node. So nvme_subsystem_set_ns_path is implemented such that if libnvme is running on the newer kernel supporting multipath sysfs link then we leverage that to discover multipath information. However if libnvme is running on older kernel which doesn't support multipath sysfs link then nvme_subsystem_set_ns_path falls back to the legacy way of discoveing multipath information.

Quoted 33 lines
Quoted 33 lines
src/nvme/tree.c +2 -0
+}
+

  static int nvme_ctrl_scan_namespace(nvme_root_t r, struct nvme_ctrl *c, char *name)
  {

src/nvme/tree.c +1 -22
@@ -2861,33 +2945,9 @@ static int nvme_ctrl_scan_namespace(nvme_root_t r, struct nvme_ctrl *c,
       n->s = c->s;
       n->c = c;
       list_add_tail(&c->namespaces, &n->entry);
-    return 0;
-}
-
-static void nvme_subsystem_set_ns_path(nvme_subsystem_t s, nvme_ns_t n)
-{
-    nvme_ctrl_t c;
-    nvme_path_t p;
-    int ns_ctrl, ns_nsid, ret;
-
-    ret = sscanf(nvme_ns_get_name(n), "nvme%dn%d", &ns_ctrl, &ns_nsid);
-    if (ret != 2)
-        return;
+    nvme_subsystem_set_ns_path(c->s, n);
   -    nvme_subsystem_for_each_ctrl(s, c) {
-        nvme_ctrl_for_each_path(c, p) {
-            int p_subsys, p_ctrl, p_nsid;
-
-            ret = sscanf(nvme_path_get_name(p), "nvme%dc%dn%d",
-                     &p_subsys, &p_ctrl, &p_nsid);
-            if (ret != 3)
-                continue;
-            if (ns_ctrl == p_subsys && ns_nsid == p_nsid) {
-                list_add_tail(&n->paths, &p->nentry);
-                p->n = n;
Quoted 8 lines

src/nvme/tree.c static int nvme_ctrl_scan_namespace(nvme_root_t r, struct nvme_ctrl *c, · line 2953

Quoted 6 lines
src/nvme/tree.c +1 -3
-            }
-        }
-    }
+    return 0;
   }
     static int nvme_subsystem_scan_namespace(nvme_root_t r, nvme_subsystem_t s,

Similar here; better to use the existing functions.

In this patch I re-wrote nvme_subsystem_set_ns_path which is now called either while scanning controler or subsystem. The idea is to use a common function which helps discover the multipath information. And as I stated before, on newer kernel this function uses the multiapth sysfs link to discover multiple paths however on older kernels it falls back to legacy way to discover mulipath infromation.

Thanks,
--Nilay

#6 lore raw

Re: [PATCHv2 2/4] tree: add queue-depth attribute for nvme path object

In reply to Hannes Reinecke
Quoted 91 lines · Hannes Reinecke
Quoted 90 lines · Nilay Shroff

Add a new attribute named "queue_depth" under the NVMe path object. This attribute is used by the iopolicy "queue-depth", which was introduced in kernel v6.11. However, the corresponding sysfs attribute for queue depth was only added in kernel v6.14.

The queue_depth value can be useful for observing which paths are selected for I/O forwarding, based on the depth of each path. To support this, export the attribute in libnvme.map so it can be accessed via nvme-cli.

---
  src/libnvme.map    |  1 +
  src/nvme/private.h |  1 +
  src/nvme/tree.c    | 12 +++++++++++-
  src/nvme/tree.h    |  8 ++++++++

  4 files changed, 21 insertions(+), 1 deletion(-)
src/libnvme.map +1 -0
index 4314705f..e53fad6b 100644
@@ -317,6 +317,7 @@ LIBNVME_1_0 {
           nvme_path_get_ctrl;
           nvme_path_get_name;
           nvme_path_get_ns;
+        nvme_path_get_queue_depth;
           nvme_path_get_sysfs_dir;
           nvme_paths_filter;
           nvme_read_config;
src/nvme/private.h +1 -0
index f45c5823..f94276e2 100644
@@ -34,6 +34,7 @@ struct nvme_path {
       char *sysfs_dir;
       char *ana_state;
       int grpid;
+    int queue_depth;
   };
     struct nvme_ns_head {
src/nvme/tree.c +9 -1
index bd7fb53e..b7a38a07 100644
@@ -903,6 +903,11 @@ const char *nvme_path_get_name(nvme_path_t p)
       return p->name;
   }
   +int nvme_path_get_queue_depth(nvme_path_t p)
+{
+    return p->queue_depth;
+}
+
   const char *nvme_path_get_ana_state(nvme_path_t p)
   {
       return p->ana_state;
@@ -921,7 +926,7 @@ void nvme_free_path(struct nvme_path *p)
   static int nvme_ctrl_scan_path(nvme_root_t r, struct nvme_ctrl *c, char *name)
   {
       struct nvme_path *p;
-    _cleanup_free_ char *path = NULL, *grpid = NULL;
+    _cleanup_free_ char *path = NULL, *grpid = NULL, *queue_depth = NULL;
       int ret;
         nvme_msg(r, LOG_DEBUG, "scan controller %s path %s\n",
@@ -955,6 +960,11 @@ static int nvme_ctrl_scan_path(nvme_root_t r, struct nvme_ctrl *c, char *name)
           sscanf(grpid, "%d", &p->grpid);
       }
   +    queue_depth = nvme_get_path_attr(p, "queue_depth");
+    if (queue_depth) {
+        sscanf(queue_depth, "%d", &p->queue_depth);
+    }
+
       list_node_init(&p->nentry);
       list_node_init(&p->entry);
       list_add_tail(&c->paths, &p->entry);
src/nvme/tree.h +0 -0
index 9f382e9c..a9082f8e 100644
@@ -867,6 +867,14 @@ const char *nvme_path_get_sysfs_dir(nvme_path_t p);
    */
   const char *nvme_path_get_ana_state(nvme_path_t p);

  +/**
+ * nvme_path_get_queue_depth() - Queue depth of an nvme_path_t object
+ * @p: &nvme_path_t object
+ *
+ * Return: Queue depth of @p
+ */

src/nvme/tree.h +2 -0
+int nvme_path_get_queue_depth(nvme_path_t p);
+

  /**

Quoted 9 lines
Quoted 2 lines

   * nvme_path_get_ctrl() - Parent controller of an nvme_path_t object
   * @p:    &nvme_path_t object

As discussed: saving the 'queue_depth' attribute is a bad idea, as it changes frequently.
I'd rather read the attribute value from sysfs whenever 'nvme_path_get_queue_depth' is called, and use the tree structure only to hold the pathname.

Yes correct, but I thought we discussed about implementing non-cached version of API in libnvme 2.x. So I implemented cached version of queue_depth API. Daniel can you confirm?

Thanks,
--Nilay

#7 lore raw

Re: [PATCHv2 2/4] tree: add queue-depth attribute for nvme path object

Quoted 9 lines · Nilay Shroff
Quoted 6 lines

As discussed: saving the 'queue_depth' attribute is a bad idea, as it changes frequently.
I'd rather read the attribute value from sysfs whenever 'nvme_path_get_queue_depth' is called, and use the tree structure only to hold the pathname.

Yes correct, but I thought we discussed about implementing non-cached version of API in libnvme 2.x. So I implemented cached version of queue_depth API. Daniel can you confirm?

I was under the impression that we agreed on the default being non-cached. As it doesn't really make sense to have a cached anyway.

#8 lore raw

Re: [PATCHv2 2/4] tree: add queue-depth attribute for nvme path object

Quoted 7 lines · Daniel Wagner
Quoted 6 lines · Nilay Shroff
Quoted 6 lines

As discussed: saving the 'queue_depth' attribute is a bad idea, as it changes frequently.
I'd rather read the attribute value from sysfs whenever 'nvme_path_get_queue_depth' is called, and use the tree structure only to hold the pathname.

Quoted 6 lines
Quoted 3 lines

Yes correct, but I thought we discussed about implementing non-cached version of API in libnvme 2.x. So I implemented cached version of queue_depth API. Daniel can you confirm?

I was under the impression that we agreed on the default being non-cached. As it doesn't really make sense to have a cached anyway.

Okay, my bad.. probably some misunderstanding on my part. I will fix this in the next version of this patch.

Thanks,
--Nilay