libnvme: add support for discovering multipath of a shared ns
v3 cover libnvme: add support for discovering multipath of a shared ns
Hi,
Recently released Linux kernel v6.15-rc1 added support for easily discovering multiple paths to a shared NVMe namespace. This multipath information is exposed to userspace via a new sysfs group attribute named "multipath", located under /sys/block/<ns-blkdev>/. More details on this functionality can be found here [1].
This patch series leverages that new functionality to discover multiple paths to a shared namespace and exposes that information in libnvme so that it can later be used by nvme-cli.
There are four patches in this series:
The first patch adds support for discovering NVMe paths using the sysfs "multipath" group attribute.
The second patch adds a new "queue_depth" attribute under the NVMe path object.
The third patch adds a new "numa_nodes" attribute under the NVMe path object. The fourth patch extends the sysfs tree dump test to validate multipath link support
[1]: https://lore.kernel.org/all/[email protected]/
Changes from v2:
- Modified queue_depth get attribure function to always return the latest value insteda of cached value (Hannes Reinecke)
Changes from v1:
- Fixed kernel version typo in the cover letter and commit
- Updated the commit in first patch to explain the change in some detail (Daniel Wagner)
- Added fourth patch to validate changes (Daniel Wagner)
Nilay Shroff (4):
tree: add support for discovering nvme paths using sysfs multipath
link
tree: add queue-depth attribute for nvme path object
tree: add attribute numa_nodes for NVMe path object
test: extend sysfs tree dump test
src/libnvme.map | 2 + src/nvme/filters.c | 6 + src/nvme/filters.h | 9 ++ src/nvme/json.c | 90 +++++++++++++-- src/nvme/private.h | 11 +- src/nvme/tree.c | 191 ++++++++++++++++++++++--------- src/nvme/tree.h | 25 ++++ test/sysfs/data/tree-pcie.out | 53 +++++++-- test/sysfs/data/tree-pcie.tar.xz | Bin 19712 -> 12656 bytes 9 files changed, 315 insertions(+), 72 deletions(-)
Signature · 1 line
2.49.0
PATCH v3 1/4 tree: add support for discovering nvme paths using sysfs multipath link
With the upcoming Linux kernel v6.15, NVMe native multipath now provides a simplified mechanism for discovering all paths to a shared namespace through sysfs.
A new "multipath" directory is created under each NVMe head namespace device in "/sys/block/<head>/multipath/". This directory contains symlinks to all namespace path devices that access the same shared namespace.
For example, consider a shared namespace accessible via two paths under nvme-subsys1:
nvme-subsys1 - NQN=nqn.1994-11.com.samsung:nvme:PM1735a:2.5-inch:S6RTNE0R900057
hostnqn=nqn.2014-08.org.nvmexpress:uuid:41528538-e8ad-4eaf-84a7-9c552917d988 \ +- ns 1 \ +- nvme0 pcie 052e:78:00.0 live optimized +- nvme1 pcie 058e:78:00.0 live optimized
The head device /dev/nvme1n1 will now have the following structure:
/sys/block/nvme1n1/multipath/
├── nvme1c0n1 -> ../../../../../pci052e:78/052e:78:00.0/nvme/nvme0/nvme1c0n1 └── nvme1c1n1 -> ../../../../../pci058e:78/058e:78:00.0/nvme/nvme1/nvme1c1n1
This clearly shows that namespace 1 is accessible through both nvme1c0n1 and nvme1c1n1. This new sysfs structure significantly simplifies multipath discovery and management, making it easier for tools and scripts to enumerate and manage NVMe multipath configurations. So leverage this functionality to update the path links for a shared NVMe namespace, simplifying path discovery and management.
This change adds a new struct nvme_ns_head to represent the head of a shared namespace. It contains a list head linking together struct nvme_path objects, where each path corresponds to a shared namespace instance. Additionally, struct nvme_ns has been updated to reference its associated nvme_ns_head, enabling straightforward traversal of all paths to a shared NVMe namespace.
--- src/nvme/filters.c | 6 ++ src/nvme/filters.h | 9 +++ src/nvme/private.h | 9 ++- src/nvme/tree.c | 162 +++++++++++++++++++++++++++++++-------------- src/nvme/tree.h | 9 +++ 5 files changed, 143 insertions(+), 52 deletions(-)
src/nvme/filters.c +6 -0
@@ -105,3 +105,9 @@ int nvme_scan_ctrl_namespaces(nvme_ctrl_t c, struct dirent ***ns)return scandir (nvme_ctrl_get_sysfs_dir (c), ns, nvme_namespace_filter, alphasort); } + +int nvme_scan_ns_head_paths (nvme_ns_head_t head,struct dirent ***paths) +{ +return scandir (nvme_ns_head_get_sysfs_dir (head), paths, + nvme_paths_filter, alphasort); +}
src/nvme/filters.h +9 -0
@@ -94,4 +94,13 @@ int nvme_scan_ctrl_namespace_paths(nvme_ctrl_t c, struct dirent ***paths);*/ int nvme_scan_ctrl_namespaces (nvme_ctrl_t c,struct dirent ***ns); +/** +* nvme_scan_ns_head_paths() - Scan for namespace paths +* @head: Namespace head node to scan +* @paths : Pointer to array of dirents +* +* Return: number of entries in @ents +*/ +int nvme_scan_ns_head_paths (nvme_ns_head_t head,struct dirent ***paths); +#endif /* _LIBNVME_FILTERS_H */
src/nvme/private.h +8 -1
@@ -36,12 +36,19 @@ struct nvme_path {int grpid; }; +struct nvme_ns_head { +struct list_head paths; +struct nvme_ns *n; + +char *sysfs_dir; +}; +struct nvme_ns {struct list_node entry; -struct list_head paths;struct nvme_subsystem *s;struct nvme_ctrl *c; +struct nvme_ns_head *head;int fd;__u32 nsid;
src/nvme/tree.c +111 -51
249 lines not on this page · raw message
src/nvme/tree.h +9 -0
@@ -27,6 +27,7 @@*/ typedef struct nvme_ns *nvme_ns_t ; +typedef struct nvme_ns_head *nvme_ns_head_t ;typedef struct nvme_path *nvme_path_t ;typedef struct nvme_ctrl *nvme_ctrl_t ;typedef struct nvme_subsystem *nvme_subsystem_t ;
@@ -1091,6 +1092,14 @@ void nvme_ctrl_set_dhchap_host_key(nvme_ctrl_t c, const char *key);*/ const char *nvme_ctrl_get_dhchap_key (nvme_ctrl_t c); +/** +* nvme_ns_head_get_sysfs_dir() - sysfs dir of namespave head +* @head: namespace head instance +* +* Returns: sysfs directory name of @head +*/ +const char *nvme_ns_head_get_sysfs_dir (nvme_ns_head_t head); +/** * nvme_ctrl_set_dhchap_key() - Set controller key * @c: Controller for which the key should be set
Signature · 1 line
2.49.0
PATCH v3 2/4 tree: add queue-depth attribute for nvme path object
In reply to Nilay ShroffIntroduce a new sysfs attribute, queue_depth, under the NVMe path object. This attribute is used by the queue-depth I/O policy introduced in kernel v6.11, but the sysfs interface for this attribute was only added later in kernel v6.15.
The queue_depth value is useful for observing which paths are selected for I/O forwarding based on their current queue depths. To make this information available to user space tools such as nvme-cli, the attribute is now exported in libnvme.map.
As queue_depth value could change frequently, nvme_path_get_queue_depth() is implemented to always fetch the latest queue_depth value, rather than relying solely on a cached version. If fetching the latest value fails, the function gracefully falls back to the cached value.
--- src/libnvme.map | 1 + src/nvme/private.h | 1 + src/nvme/tree.c | 19 ++++++++++++++++++- src/nvme/tree.h | 8 ++++++++ 4 files changed, 28 insertions(+), 1 deletion(-)
src/libnvme.map +1 -0
@@ -317,6 +317,7 @@ LIBNVME_1_0 { nvme_path_get_ctrl; nvme_path_get_name; nvme_path_get_ns; + nvme_path_get_queue_depth; nvme_path_get_sysfs_dir; nvme_paths_filter; nvme_read_config;
src/nvme/private.h +1 -0
@@ -34,6 +34,7 @@ struct nvme_path {char *sysfs_dir;char *ana_state;int grpid; +int queue_depth; };struct nvme_ns_head {
src/nvme/tree.c +18 -1
@@ -903,6 +903,18 @@ const char *nvme_path_get_name(nvme_path_t p)return p->name; } +int nvme_path_get_queue_depth (nvme_path_t p) +{ + _cleanup_free_ char *queue_depth = NULL; + + queue_depth =nvme_get_path_attr (p,"queue_depth" ); +if (queue_depth) { +sscanf (queue_depth,"%d" , &p->queue_depth); + } + +return p->queue_depth; +} +const char *nvme_path_get_ana_state (nvme_path_t p) {return p->ana_state;
@@ -921,7 +933,7 @@ void nvme_free_path(struct nvme_path *p)static int nvme_ctrl_scan_path (nvme_root_t r,struct nvme_ctrl *c,char *name) {struct nvme_path *p; - _cleanup_free_ char *path = NULL, *grpid = NULL; + _cleanup_free_ char *path = NULL, *grpid = NULL, *queue_depth = NULL;int ret; nvme_msg(r, LOG_DEBUG, "scan controller %s path %s\n",
@@ -955,6 +967,11 @@ static int nvme_ctrl_scan_path(nvme_root_t r, struct nvme_ctrl *c, char *name)sscanf (grpid,"%d" , &p->grpid); } + queue_depth =nvme_get_path_attr (p,"queue_depth" ); +if (queue_depth) { +sscanf (queue_depth,"%d" , &p->queue_depth); + } +list_node_init (&p->nentry);list_node_init (&p->entry);list_add_tail (&c->paths, &p->entry);
src/nvme/tree.h +8 -0
@@ -867,6 +867,14 @@ const char *nvme_path_get_sysfs_dir(nvme_path_t p);*/ const char *nvme_path_get_ana_state (nvme_path_t p); +/** +* nvme_path_get_queue_depth() - Queue depth of an nvme_path_t object +* @p: &nvme_path_t object +* +* Return: Queue depth of @p +*/ +int nvme_path_get_queue_depth (nvme_path_t p); +/** * nvme_path_get_ctrl() - Parent controller of an nvme_path_t object * @p: &nvme_path_t object
Signature · 1 line
2.49.0
PATCH v3 3/4 tree: add attribute numa_nodes for NVMe path object
In reply to Nilay ShroffAdd a new attribute named "numa_nodes" under the NVMe path object. This attribute is used by the iopolicy "numa". The numa_nodes value is stored for each NVMe path and represents the NUMA node(s) associated with it. When the iopolicy is set to "numa", I/O traffic originating from a given NUMA node will be forwarded through the corresponding NVMe path.
The numa_nodes attribute is useful for observing which NVMe path the kernel would choose for I/O forwarding based on NUMA affinity. To support this, export the attribute in libnvme.map so it can be accessed via nvme-cli.
--- src/libnvme.map | 1 + src/nvme/private.h | 1 + src/nvme/tree.c | 10 ++++++++++ src/nvme/tree.h | 8 ++++++++ 4 files changed, 20 insertions(+)
src/libnvme.map +1 -0
@@ -317,6 +317,7 @@ LIBNVME_1_0 { nvme_path_get_ctrl; nvme_path_get_name; nvme_path_get_ns; + nvme_path_get_numa_nodes; nvme_path_get_queue_depth; nvme_path_get_sysfs_dir; nvme_paths_filter;
src/nvme/private.h +1 -0
@@ -33,6 +33,7 @@ struct nvme_path {char *name;char *sysfs_dir;char *ana_state; +char *numa_nodes;int grpid;int queue_depth; };
src/nvme/tree.c +10 -0
@@ -920,6 +920,11 @@ const char *nvme_path_get_ana_state(nvme_path_t p)return p->ana_state; } +const char *nvme_path_get_numa_nodes (nvme_path_t p) +{ +return p->numa_nodes; +} +void nvme_free_path (struct nvme_path *p) {list_del_init (&p->entry);
@@ -927,6 +932,7 @@ void nvme_free_path(struct nvme_path *p)free (p->name);free (p->sysfs_dir);free (p->ana_state); +free (p->numa_nodes);free (p); }
@@ -962,6 +968,10 @@ static int nvme_ctrl_scan_path(nvme_root_t r, struct nvme_ctrl *c, char *name)if (!p->ana_state) p->ana_state =strdup ("optimized" ); + p->numa_nodes =nvme_get_path_attr (p,"numa_nodes" ); +if (!p->numa_nodes) + p->numa_nodes =strdup ("-1" ); + grpid =nvme_get_path_attr (p,"ana_grpid" );if (grpid) {sscanf (grpid,"%d" , &p->grpid);
src/nvme/tree.h +8 -0
@@ -867,6 +867,14 @@ const char *nvme_path_get_sysfs_dir(nvme_path_t p);*/ const char *nvme_path_get_ana_state (nvme_path_t p); +/** +* nvme_path_get_numa_nodes() - NUMA nodes of an nvme_path_t object +* @p : &nvme_path_t object +* +* Return: NUMA nodes associated to @p +*/ +const char *nvme_path_get_numa_nodes (nvme_path_t p); +/** * nvme_path_get_queue_depth() - Queue depth of an nvme_path_t object * @p: &nvme_path_t object
Signature · 1 line
2.49.0
PATCH v3 4/4 test: extend sysfs tree dump test
In reply to Nilay ShroffIn order to validate multipath link support extend the existing sysfs tree dump test.
The updated tree-pcie.tar.xz archive includes a sysfs dump with two NVMe subsystems. The nvme-subsys0 simulates a single-port NVMe disk with a private namespace that does not support multipath. And another nvme-subsys1 simulates a dual-port NVMe disk with a shared namespace attached to multiple controllers.
The json tree dump is also updated to include NVMe namespace path information. Since the archive content has changed, the expected output file tree-pcie.out is also updated to reflect the new structure.
--- src/nvme/json.c | 90 +++++++++++++++++++++++++++---- test/sysfs/data/tree-pcie.out | 53 ++++++++++++++---- test/sysfs/data/tree-pcie.tar.xz | Bin 19712 -> 12656 bytes 3 files changed, 124 insertions(+), 19 deletions(-)
src/nvme/json.c +81 -9
@@ -533,25 +533,97 @@ static void json_dump_ctrl(struct json_object *ctrl_array, nvme_ctrl_t c)json_object_array_add (ctrl_array, ctrl_obj); } +static unsigned int json_dump_subsys_multipath (nvme_subsystem_t s, +struct json_object *ns_array) +{ +nvme_ns_t n; +nvme_path_t p; +unsigned int i =0 ; + +nvme_subsystem_for_each_ns (s, n) { +struct json_object *ns_obj; +struct json_object *path_array; + + ns_obj =json_object_new_object (); +json_object_object_add (ns_obj,"nsid" , +json_object_new_int (nvme_ns_get_nsid (n))); +json_object_object_add (ns_obj,"name" , +json_object_new_string (nvme_ns_get_name (n))); + + path_array =json_object_new_array (); +nvme_namespace_for_each_path (n, p) { +struct json_object *path_obj; +struct json_object *ctrl_array; +nvme_ctrl_t c; + + path_obj =json_object_new_object (); +json_object_object_add (path_obj,"path" , +json_object_new_string (nvme_path_get_name (p))); +json_object_object_add (path_obj,"ANAState" , +json_object_new_string (nvme_path_get_ana_state (p))); +json_object_object_add (path_obj,"NUMANodes" , +json_object_new_string (nvme_path_get_numa_nodes (p))); +json_object_object_add (path_obj,"qdepth" , +json_object_new_int (nvme_path_get_queue_depth (p))); + + c =nvme_path_get_ctrl (p); + ctrl_array =json_object_new_array (); +json_dump_ctrl (ctrl_array, c); +json_object_object_add (path_obj,"controller" , ctrl_array); +json_object_array_add (path_array, path_obj); + } +json_object_object_add (ns_obj,"paths" , path_array); +json_object_array_add (ns_array, ns_obj); + i++; + } +return i; +} + +static void json_dump_subsys_non_multipath (nvme_subsystem_t s, +struct json_object *ns_array) +{ +nvme_ctrl_t c; +nvme_ns_t n; + +nvme_subsystem_for_each_ctrl (s, c) { +nvme_ctrl_for_each_ns (c, n) { +struct json_object *ctrl_array; +struct json_object *ns_obj; + + ns_obj =json_object_new_object (); +json_object_object_add (ns_obj,"nsid" , +json_object_new_int (nvme_ns_get_nsid (n))); +json_object_object_add (ns_obj,"name" , +json_object_new_string (nvme_ns_get_name (n))); + + ctrl_array =json_object_new_array (); +json_dump_ctrl (ctrl_array, c); +json_object_object_add (ns_obj,"controller" , ctrl_array); + +json_object_array_add (ns_array, ns_obj); + } + } +} +static void json_dump_subsys (struct json_object *subsys_array,nvme_subsystem_t s) { -nvme_ctrl_t c;struct json_object *subsys_obj =json_object_new_object (); -struct json_object *ctrl_array; +struct json_object *ns_array;json_object_object_add (subsys_obj,"name" ,json_object_new_string (nvme_subsystem_get_name (s)));json_object_object_add (subsys_obj,"nqn" ,json_object_new_string (nvme_subsystem_get_nqn (s))); - ctrl_array =json_object_new_array (); -nvme_subsystem_for_each_ctrl (s, c) { -json_dump_ctrl (ctrl_array, c); - } -if (json_object_array_length (ctrl_array)) -json_object_object_add (subsys_obj,"controllers" , ctrl_array); + + ns_array =json_object_new_array (); +if (!json_dump_subsys_multipath (s, ns_array)) +json_dump_subsys_non_multipath (s, ns_array); + +if (json_object_array_length (ns_array)) +json_object_object_add (subsys_obj,"namespaces" , ns_array);else -json_object_put (ctrl_array); +json_object_put (ns_array);json_object_array_add (subsys_array, subsys_obj); }
test/sysfs/data/tree-pcie.out +43 -10
66 lines not on this page · raw message
test/sysfs/data/tree-pcie.tar.xz binary +0 -0
Signature · 1 line
2.49.0
Hi Daniel, Hannes,
A gentle ping... Do you have any further comment for this patchset?
Thanks,
--Nilay
Quoted 52 lines · Nilay Shroff
Hi,
Recently released Linux kernel v6.15-rc1 added support for easily discovering multiple paths to a shared NVMe namespace. This multipath information is exposed to userspace via a new sysfs group attribute named "multipath", located under /sys/block/<ns-blkdev>/. More details on this functionality can be found here [1].
This patch series leverages that new functionality to discover multiple paths to a shared namespace and exposes that information in libnvme so that it can later be used by nvme-cli.
There are four patches in this series:
The first patch adds support for discovering NVMe paths using the sysfs "multipath" group attribute.
The second patch adds a new "queue_depth" attribute under the NVMe path object.
The third patch adds a new "numa_nodes" attribute under the NVMe path object. The fourth patch extends the sysfs tree dump test to validate multipath link support[1]: https://lore.kernel.org/all/[email protected]/
Changes from v2:
- Modifiedqueue_depthget attribure function to always return the latest value insteda of cached value (Hannes Reinecke)Changes from v1:
- Fixed kernel version typo in the cover letter and commit
- Updated the commit in first patch to explain the change in some detail (Daniel Wagner)
- Added fourth patch to validate changes (Daniel Wagner)Nilay Shroff (4):
tree: add support for discovering nvme paths using sysfs multipath
link
tree: add queue-depth attribute for nvme path object
tree: add attributenuma_nodesfor NVMe path object
test: extend sysfs tree dump testsrc/libnvme.map | 2 + src/nvme/filters.c | 6 + src/nvme/filters.h | 9 ++ src/nvme/json.c | 90 +++++++++++++-- src/nvme/private.h | 11 +- src/nvme/tree.c | 191 ++++++++++++++++++++++--------- src/nvme/tree.h | 25 ++++ test/sysfs/data/tree-pcie.out | 53 +++++++-- test/sysfs/data/tree-pcie.tar.xz | Bin 19712 -> 12656 bytes 9 files changed, 315 insertions(+), 72 deletions(-)
Hi Nilay
Quoted 12 lines · Nilay Shroff
Hi,
Recently released Linux kernel v6.15-rc1 added support for easily discovering multiple paths to a shared NVMe namespace. This multipath information is exposed to userspace via a new sysfs group attribute named "multipath", located under /sys/block/<ns-blkdev>/. More details on this functionality can be found here [1].
This patch series leverages that new functionality to discover multiple paths to a shared namespace and exposes that information in libnvme so that it can later be used by nvme-cli.
Quoted 8 lines
There are four patches in this series:
The first patch adds support for discovering NVMe paths using the sysfs "multipath" group attribute.
The second patch adds a new "queue_depth" attribute under the NVMe path object.
The third patch adds a new "numa_nodes" attribute under the NVMe path object. The fourth patch extends the sysfs tree dump test to validate multipath link support
LGTM. I've applied the series.
Thanks,
Daniel