libnvme: add support for discovering multipath of a shared ns
v2 cover libnvme: add support for discovering multipath of a shared ns
Hi,
Recently released Linux kernel v6.15-rc1 added support for easily discovering multiple paths to a shared NVMe namespace. This multipath information is exposed to userspace via a new sysfs group attribute named "multipath", located under /sys/block/<ns-blkdev>/. More details on this functionality can be found here [1].
This patch series leverages that new functionality to discover multiple paths to a shared namespace and exposes that information in libnvme so that it can later be used by nvme-cli.
There are four patches in this series:
The first patch adds support for discovering NVMe paths using the sysfs "multipath" group attribute.
The second patch adds a new "queue_depth" attribute under the NVMe path object.
The third patch adds a new "numa_nodes" attribute under the NVMe path object. The fourth patch extends the sysfs tree dump test to validate multipath link support
[1]: https://lore.kernel.org/all/[email protected]/
Changes from v1:
- Fixed kernel version typo in the cover letter and commit
- Updated the commit in first patch to explain the change in some detail (Daniel Wagner)
- Added fourth patch to validate changes (Daniel Wagner)
Nilay Shroff (4):
tree: add support for discovering nvme paths using sysfs multipath
link
tree: add queue-depth attribute for nvme path object
tree: add attribute numa_nodes for NVMe path object
test: extend sysfs tree dump test
src/libnvme.map | 2 + src/nvme/filters.c | 6 + src/nvme/filters.h | 9 ++ src/nvme/json.c | 90 +++++++++++++-- src/nvme/private.h | 11 +- src/nvme/tree.c | 184 ++++++++++++++++++++++--------- src/nvme/tree.h | 25 +++++ test/sysfs/data/tree-pcie.out | 53 +++++++-- test/sysfs/data/tree-pcie.tar.xz | Bin 19712 -> 12656 bytes 9 files changed, 308 insertions(+), 72 deletions(-)
Signature · 1 line
2.49.0
PATCH v2 1/4 tree: add support for discovering nvme paths using sysfs multipath link
With the upcoming Linux kernel v6.15, NVMe native multipath now provides a simplified mechanism for discovering all paths to a shared namespace through sysfs.
A new "multipath" directory is created under each NVMe head namespace device in "/sys/block/<head>/multipath/". This directory contains symlinks to all namespace path devices that access the same shared namespace.
For example, consider a shared namespace accessible via two paths under nvme-subsys1:
nvme-subsys1 - NQN=nqn.1994-11.com.samsung:nvme:PM1735a:2.5-inch:S6RTNE0R900057
hostnqn=nqn.2014-08.org.nvmexpress:uuid:41528538-e8ad-4eaf-84a7-9c552917d988 \ +- ns 1 \ +- nvme0 pcie 052e:78:00.0 live optimized +- nvme1 pcie 058e:78:00.0 live optimized
The head device /dev/nvme1n1 will now have the following structure:
/sys/block/nvme1n1/multipath/
├── nvme1c0n1 -> ../../../../../pci052e:78/052e:78:00.0/nvme/nvme0/nvme1c0n1 └── nvme1c1n1 -> ../../../../../pci058e:78/058e:78:00.0/nvme/nvme1/nvme1c1n1
This clearly shows that namespace 1 is accessible through both nvme1c0n1 and nvme1c1n1. This new sysfs structure significantly simplifies multipath discovery and management, making it easier for tools and scripts to enumerate and manage NVMe multipath configurations. So leverage this functionality to update the path links for a shared NVMe namespace, simplifying path discovery and management.
This change adds a new struct nvme_ns_head to represent the head of a shared namespace. It contains a list head linking together struct nvme_path objects, where each path corresponds to a shared namespace instance. Additionally, struct nvme_ns has been updated to reference its associated nvme_ns_head, enabling straightforward traversal of all paths to a shared NVMe namespace.
--- src/nvme/filters.c | 6 ++ src/nvme/filters.h | 9 +++ src/nvme/private.h | 9 ++- src/nvme/tree.c | 162 +++++++++++++++++++++++++++++++-------------- src/nvme/tree.h | 9 +++ 5 files changed, 143 insertions(+), 52 deletions(-)
src/nvme/filters.c +6 -0
@@ -105,3 +105,9 @@ int nvme_scan_ctrl_namespaces(nvme_ctrl_t c, struct dirent ***ns)return scandir (nvme_ctrl_get_sysfs_dir (c), ns, nvme_namespace_filter, alphasort); } + +int nvme_scan_ns_head_paths (nvme_ns_head_t head,struct dirent ***paths) +{ +return scandir (nvme_ns_head_get_sysfs_dir (head), paths, + nvme_paths_filter, alphasort); +}
src/nvme/filters.h +9 -0
@@ -94,4 +94,13 @@ int nvme_scan_ctrl_namespace_paths(nvme_ctrl_t c, struct dirent ***paths);*/ int nvme_scan_ctrl_namespaces (nvme_ctrl_t c,struct dirent ***ns); +/** +* nvme_scan_ns_head_paths() - Scan for namespace paths +* @head: Namespace head node to scan +* @paths : Pointer to array of dirents +* +* Return: number of entries in @ents +*/ +int nvme_scan_ns_head_paths (nvme_ns_head_t head,struct dirent ***paths); +#endif /* _LIBNVME_FILTERS_H */
src/nvme/private.h +8 -1
@@ -36,12 +36,19 @@ struct nvme_path {int grpid; }; +struct nvme_ns_head { +struct list_head paths; +struct nvme_ns *n; + +char *sysfs_dir; +}; +struct nvme_ns {struct list_node entry; -struct list_head paths;struct nvme_subsystem *s;struct nvme_ctrl *c; +struct nvme_ns_head *head;int fd;__u32 nsid;
src/nvme/tree.c +111 -51
249 lines not on this page · raw message
src/nvme/tree.h +9 -0
@@ -27,6 +27,7 @@*/ typedef struct nvme_ns *nvme_ns_t ; +typedef struct nvme_ns_head *nvme_ns_head_t ;typedef struct nvme_path *nvme_path_t ;typedef struct nvme_ctrl *nvme_ctrl_t ;typedef struct nvme_subsystem *nvme_subsystem_t ;
@@ -1091,6 +1092,14 @@ void nvme_ctrl_set_dhchap_host_key(nvme_ctrl_t c, const char *key);*/ const char *nvme_ctrl_get_dhchap_key (nvme_ctrl_t c); +/** +* nvme_ns_head_get_sysfs_dir() - sysfs dir of namespave head +* @head: namespace head instance +* +* Returns: sysfs directory name of @head +*/ +const char *nvme_ns_head_get_sysfs_dir (nvme_ns_head_t head); +/** * nvme_ctrl_set_dhchap_key() - Set controller key * @c: Controller for which the key should be set
Signature · 1 line
2.49.0
Re: [PATCHv2 1/4] tree: add support for discovering nvme paths using sysfs multipath link
Quoted 94 lines · Nilay Shroff
With the upcoming Linux kernel v6.15, NVMe native multipath now provides a simplified mechanism for discovering all paths to a shared namespace through sysfs.
A new "multipath" directory is created under each NVMe head namespace device in "/sys/block/<head>/multipath/". This directory contains symlinks to all namespace path devices that access the same shared namespace.
For example, consider a shared namespace accessible via two paths under nvme-subsys1:
nvme-subsys1 - NQN=nqn.1994-11.com.samsung:nvme:PM1735a:2.5-inch:S6RTNE0R900057
hostnqn=nqn.2014-08.org.nvmexpress:uuid:41528538-e8ad-4eaf-84a7-9c552917d988 \ +- ns 1 \ +- nvme0 pcie 052e:78:00.0 live optimized +- nvme1 pcie 058e:78:00.0 live optimizedThe head device
/dev/nvme1n1will now have the following structure:
/sys/block/nvme1n1/multipath/├── nvme1c0n1 -> ../../../../../pci052e:78/052e:78:00.0/nvme/nvme0/nvme1c0n1 └── nvme1c1n1 -> ../../../../../pci058e:78/058e:78:00.0/nvme/nvme1/nvme1c1n1This clearly shows that namespace 1 is accessible through both nvme1c0n1 and nvme1c1n1. This new sysfs structure significantly simplifies multipath discovery and management, making it easier for tools and scripts to enumerate and manage NVMe multipath configurations. So leverage this functionality to update the path links for a shared NVMe namespace, simplifying path discovery and management.
This change adds a new struct
nvme_ns_headto represent the head of a shared namespace. It contains a list head linking together structnvme_pathobjects, where each path corresponds to a shared namespace instance. Additionally, structnvme_nshas been updated to reference its associatednvme_ns_head, enabling straightforward traversal of all paths to a shared NVMe namespace.Signed-off-by Nilay Shroff <[email protected]>--- src/nvme/filters.c | 6 ++ src/nvme/filters.h | 9 +++ src/nvme/private.h | 9 ++- src/nvme/tree.c | 162 +++++++++++++++++++++++++++++++-------------- src/nvme/tree.h | 9 +++ 5 files changed, 143 insertions(+), 52 deletions(-)src/nvme/filters.c +6
-0@@ -105,3 +105,9 @@ int nvme_scan_ctrl_namespaces(nvme_ctrl_t c, struct dirent ***ns)return scandir (nvme_ctrl_get_sysfs_dir (c), ns, nvme_namespace_filter, alphasort); } + +int nvme_scan_ns_head_paths (nvme_ns_head_t head,struct dirent ***paths) +{ +return scandir (nvme_ns_head_get_sysfs_dir (head), paths, + nvme_paths_filter, alphasort); +}src/nvme/filters.h +9
-0@@ -94,4 +94,13 @@ int nvme_scan_ctrl_namespace_paths(nvme_ctrl_t c, struct dirent ***paths);*/ int nvme_scan_ctrl_namespaces (nvme_ctrl_t c,struct dirent ***ns); +/** +* nvme_scan_ns_head_paths() - Scan for namespace paths +* @head: Namespace head node to scan +* @paths : Pointer to array of dirents +* +* Return: number of entries in @ents +*/ +int nvme_scan_ns_head_paths (nvme_ns_head_t head,struct dirent ***paths); +#endif /* _LIBNVME_FILTERS_H */ src/nvme/private.h +7
-0@@ -36,12 +36,19 @@ struct nvme_path {int grpid; }; +struct nvme_ns_head { +struct list_head paths; +struct nvme_ns *n; + +char *sysfs_dir; +}; +
src/nvme/private.h struct nvme_path { · line 54
Quoted 10 lines
src/nvme/private.h +1
-1struct nvme_ns {struct list_node entry; -struct list_head paths;struct nvme_subsystem *s;struct nvme_ctrl *c; +struct nvme_ns_head *head;int fd;__u32 nsid;
Why isn't 'nvme_subsystem' moved to 'nvme_ns_head'? That certainly is how the kernel structures are laid out, and it would make sense to follow it here...
Quoted 205 lines
src/nvme/tree.c +96
-24@@ -564,12 +564,12 @@ nvme_ns_t nvme_subsystem_next_ns(nvme_subsystem_t s, nvme_ns_t n)nvme_path_t nvme_namespace_first_path (nvme_ns_t ns) { - return list_top(&ns->paths, struct nvme_path, nentry); + return list_top(&ns->head->paths, struct nvme_path, nentry); }nvme_path_t nvme_namespace_next_path (nvme_ns_t ns,nvme_path_t p) { -return p ?list_next (&ns->paths, p, nentry) :NULL ; +return p ?list_next (&ns->head->paths, p, nentry) :NULL ; }static void __nvme_free_ns (struct nvme_ns *n)@@ -579,6 +579,8 @@ static void __nvme_free_ns(struct nvme_ns *n)free (n->generic_name);free (n->name);free (n->sysfs_dir); +free (n->head->sysfs_dir); +free (n->head);free (n); }@@ -916,25 +918,6 @@ void nvme_free_path(struct nvme_path *p)free (p); } -static void nvme_subsystem_set_path_ns (nvme_subsystem_t s,nvme_path_t p) -{ -char n_name[32 ] = { }; -int i, c, nsid, ret; -nvme_ns_t n; - - ret =sscanf (nvme_path_get_name (p),"nvme%dc%dn%d" , &i, &c, &nsid); -if (ret !=3 ) -return ; - -sprintf (n_name,"nvme%dn%d" , i, nsid); -nvme_subsystem_for_each_ns (s, n) { -if (!strcmp (n_name,nvme_ns_get_name (n))) { -list_add_tail (&n->paths, &p->nentry); - p->n = n; - } - } -} -static int nvme_ctrl_scan_path (nvme_root_t r,struct nvme_ctrl *c,char *name) {struct nvme_path *p;@@ -973,7 +956,6 @@ static int nvme_ctrl_scan_path(nvme_root_t r, struct nvme_ctrl *c, char *name) }list_node_init (&p->nentry); -nvme_subsystem_set_path_ns (c->s, p);list_node_init (&p->entry);list_add_tail (&c->paths, &p->entry);return 0 ;@@ -2250,8 +2232,8 @@ nvme_ctrl_t nvme_scan_ctrl(nvme_root_t r, const char *name)return NULL ; path =NULL ; -nvme_ctrl_scan_namespaces (r, c);nvme_ctrl_scan_paths (r, c); +nvme_ctrl_scan_namespaces (r, c);return c; }@@ -2323,6 +2305,11 @@ const char *nvme_ns_get_sysfs_dir(nvme_ns_t n)return n->sysfs_dir; } +const char *nvme_ns_head_get_sysfs_dir (nvme_ns_head_t head) +{ +return head->sysfs_dir; +} +const char *nvme_ns_get_name (nvme_ns_t n) {return n->name;@@ -2749,7 +2736,11 @@ static void nvme_ns_set_generic_name(struct nvme_ns *n, const char *name)static nvme_ns_t nvme_ns_open (const char *sys_path,const char *name) { +int ret;struct nvme_ns *n; +struct nvme_ns_head *head; +struct stat arg; + _cleanup_free_ char *path = NULL; n =calloc (1 ,sizeof (*n));if (!n) {@@ -2757,6 +2748,32 @@ static nvme_ns_t nvme_ns_open(const char *sys_path, const char *name)return NULL ; } + head =calloc (1 ,sizeof (*head)); +if (!head) { + errno =ENOMEM ; +free (n); +return NULL ; + } + + head->n = n; +list_head_init (&head->paths); + ret =asprintf (&path,"%s/%s" , sys_path,"multipath" ); +if (ret <0 ) { + errno =ENOMEM ; + goto free_ns_head; + } +/* +* The sysfs-dir "multipath" is available only when nvme multipath +* is configured and we're running kernel version >= 6.14. +*/ + ret =stat (path, &arg); +if (ret ==0 ) { + head->sysfs_dir = path; + path =NULL ; + }else + head->sysfs_dir =NULL ; + + n->head = head; n->fd =-1 ; n->name =strdup (name);@@ -2765,15 +2782,17 @@ static nvme_ns_t nvme_ns_open(const char *sys_path, const char *name)if (nvme_ns_init (sys_path, n) !=0 ) goto free_ns; -list_head_init (&n->paths);list_node_init (&n->entry);nvme_ns_release_fd (n);/* Do not leak fds */ +return n; free_ns:free (n->generic_name);free (n->name); +free_ns_head: +free (head);free (n);return NULL ; }@@ -2836,6 +2855,71 @@ nvme_ns_t nvme_scan_namespace(const char *name)return __nvme_scan_namespace (nvme_ns_sysfs_dir (), name); } + +static void nvme_ns_head_scan_path (nvme_subsystem_t s,nvme_ns_t n,char *name) +{ +nvme_ctrl_t c; +nvme_path_t p; + +nvme_subsystem_for_each_ctrl (s, c) { +nvme_ctrl_for_each_path (c, p) { +if (!strcmp (nvme_path_get_name (p), name)) { +list_add_tail (&n->head->paths, &p->nentry); + p->n = n; +return ; + } + } + } +} + +static void nvme_subsystem_set_ns_path (nvme_subsystem_t s,nvme_ns_t n) +{ +struct nvme_ns_head *head = n->head; + +if (nvme_ns_head_get_sysfs_dir (head)) { +struct dirents paths = {}; +int i; + +/* +* When multipath is configured on kernel version >= 6.14, +* we use multipath sysfs link to get each path of a namespace. +*/ + paths.num =nvme_scan_ns_head_paths (head, &paths.ents); + +for (i =0 ; i < paths.num; i++) +nvme_ns_head_scan_path (s, n, paths.ents[i]->d_name); + }else { +nvme_ctrl_t c; +nvme_path_t p; +int ns_ctrl, ns_nsid, ret; + +/* +* If multipath is not configured or we're running on kernel +* version < 6.14, fallback to the old way. +*/ + ret =sscanf (nvme_ns_get_name (n),"nvme%dn%d" , + &ns_ctrl, &ns_nsid); +if (ret !=2 ) +return ; + +nvme_subsystem_for_each_ctrl (s, c) { +nvme_ctrl_for_each_path (c, p) { +int p_subsys, p_ctrl, p_nsid; + + ret = sscanf(nvme_path_get_name(p), + "nvme%dc%dn%d",
src/nvme/tree.c nvme_ns_t nvme_scan_namespace(const char *name) · line 2920
Quoted 10 lines
src/nvme/tree.c +10
-0+ &p_subsys, &p_ctrl, &p_nsid); +if (ret !=3 ) +continue ; +if (ns_ctrl == p_subsys && ns_nsid == p_nsid) { +list_add_tail (&head->paths, &p->nentry); + p->n = n; + } + } + } + }
Can't you just use the existing nvme_subsystem_set_path_ns() function here instead of deleting and open-code it?
Quoted 31 lines
src/nvme/tree.c +3
-20+} +static int nvme_ctrl_scan_namespace (nvme_root_t r,struct nvme_ctrl *c,char *name) {@@ -2861,33 +2945,9 @@ static int nvme_ctrl_scan_namespace(nvme_root_t r, struct nvme_ctrl *c, n->s = c->s; n->c = c;list_add_tail (&c->namespaces, &n->entry); -return 0 ; -} - -static void nvme_subsystem_set_ns_path (nvme_subsystem_t s,nvme_ns_t n) -{ -nvme_ctrl_t c; -nvme_path_t p; -int ns_ctrl, ns_nsid, ret; - - ret =sscanf (nvme_ns_get_name (n),"nvme%dn%d" , &ns_ctrl, &ns_nsid); -if (ret !=2 ) -return ; +nvme_subsystem_set_ns_path (c->s, n); -nvme_subsystem_for_each_ctrl (s, c) { -nvme_ctrl_for_each_path (c, p) { -int p_subsys, p_ctrl, p_nsid; - - ret =sscanf (nvme_path_get_name (p),"nvme%dc%dn%d" , - &p_subsys, &p_ctrl, &p_nsid); -if (ret !=3 ) -continue ;
src/nvme/tree.c static int nvme_ctrl_scan_namespace(nvme_root_t r, struct nvme_ctrl *c, · line 2953
Quoted 10 lines
src/nvme/tree.c +1
-6-if (ns_ctrl == p_subsys && ns_nsid == p_nsid) { -list_add_tail (&n->paths, &p->nentry); - p->n = n; - } - } - } +return 0 ; } static int nvme_subsystem_scan_namespace(nvme_root_t r, nvme_subsystem_t s,
Similar here; better to use the existing functions.
Quoted 26 lines
src/nvme/tree.c +1
-1@@ -2917,7 +2977,7 @@ static int nvme_subsystem_scan_namespace(nvme_root_t r, nvme_subsystem_t s,list_del_init (&p->nentry); p->n =NULL ; } -list_head_init (&_n->paths); +list_head_init (&_n->head->paths);__nvme_free_ns (_n); } n->s = s;src/nvme/tree.h +2
-0@@ -27,6 +27,7 @@*/ typedef struct nvme_ns *nvme_ns_t ; +typedef struct nvme_ns_head *nvme_ns_head_t ;typedef struct nvme_path *nvme_path_t ;typedef struct nvme_ctrl *nvme_ctrl_t ;typedef struct nvme_subsystem *nvme_subsystem_t ;@@ -1091,6 +1092,14 @@ void nvme_ctrl_set_dhchap_host_key(nvme_ctrl_t c, const char *key);*/ const char *nvme_ctrl_get_dhchap_key (nvme_ctrl_t c); +/**
src/nvme/tree.h void nvme_ctrl_set_dhchap_host_key(nvme_ctrl_t c, const char *key); · line 1105
Quoted 10 lines
src/nvme/tree.h +7
-0+* nvme_ns_head_get_sysfs_dir() - sysfs dir of namespave head +* @head: namespace head instance +* +* Returns: sysfs directory name of @head +*/ +const char *nvme_ns_head_get_sysfs_dir (nvme_ns_head_t head); +/** * nvme_ctrl_set_dhchap_key() - Set controller key * @c: Controller for which the key should be set
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Re: [PATCHv2 1/4] tree: add support for discovering nvme paths using sysfs multipath link
Quoted 97 lines · Hannes Reinecke
Quoted 96 lines · Nilay Shroff
With the upcoming Linux kernel v6.15, NVMe native multipath now provides a simplified mechanism for discovering all paths to a shared namespace through sysfs.
A new "multipath" directory is created under each NVMe head namespace device in "/sys/block/<head>/multipath/". This directory contains symlinks to all namespace path devices that access the same shared namespace.
For example, consider a shared namespace accessible via two paths under nvme-subsys1:
nvme-subsys1 - NQN=nqn.1994-11.com.samsung:nvme:PM1735a:2.5-inch:S6RTNE0R900057 hostnqn=nqn.2014-08.org.nvmexpress:uuid:41528538-e8ad-4eaf-84a7-9c552917d988 \
+- ns 1
\+- nvme0 pcie 052e:78:00.0 live optimized +- nvme1 pcie 058e:78:00.0 live optimizedThe head device
/dev/nvme1n1will now have the following structure:
/sys/block/nvme1n1/multipath/├── nvme1c0n1 -> ../../../../../pci052e:78/052e:78:00.0/nvme/nvme0/nvme1c0n1 └── nvme1c1n1 -> ../../../../../pci058e:78/058e:78:00.0/nvme/nvme1/nvme1c1n1This clearly shows that namespace 1 is accessible through both nvme1c0n1 and nvme1c1n1. This new sysfs structure significantly simplifies multipath discovery and management, making it easier for tools and scripts to enumerate and manage NVMe multipath configurations. So leverage this functionality to update the path links for a shared NVMe namespace, simplifying path discovery and management.
This change adds a new struct
nvme_ns_headto represent the head of a shared namespace. It contains a list head linking together structnvme_pathobjects, where each path corresponds to a shared namespace instance. Additionally, structnvme_nshas been updated to reference its associatednvme_ns_head, enabling straightforward traversal of all paths to a shared NVMe namespace.Signed-off-by Nilay Shroff <[email protected]>---
src/nvme/filters.c | 6 ++ src/nvme/filters.h | 9 +++ src/nvme/private.h | 9 ++- src/nvme/tree.c | 162 +++++++++++++++++++++++++++++++-------------- src/nvme/tree.h | 9 +++ 5 files changed, 143 insertions(+), 52 deletions(-)src/nvme/filters.c +6
-0@@ -105,3 +105,9 @@ int nvme_scan_ctrl_namespaces(nvme_ctrl_t c, struct dirent ***ns) return scandir(nvme_ctrl_get_sysfs_dir(c), ns, nvme_namespace_filter, alphasort); } + +int nvme_scan_ns_head_paths (nvme_ns_head_t head,struct dirent ***paths) +{ + return scandir(nvme_ns_head_get_sysfs_dir(head), paths, + nvme_paths_filter, alphasort); +}src/nvme/filters.h +0
-0@@ -94,4 +94,13 @@ int nvme_scan_ctrl_namespace_paths(nvme_ctrl_t c, struct dirent ***paths);*/ int nvme_scan_ctrl_namespaces(nvme_ctrl_t c, struct dirent ***ns);+/**
+ *nvme_scan_ns_head_paths()- Scan for namespace paths
+ * @head: Namespace head node to scan
+ * @paths : Pointer to array of dirents
+ *
+ * Return: number of entries in @ents
+ */src/nvme/filters.h +2
-0+int nvme_scan_ns_head_paths (nvme_ns_head_t head,struct dirent ***paths); +#endif /*
_LIBNVME_FILTERS_H*/src/nvme/private.h +6
-1@@ -36,12 +36,19 @@ struct nvme_path { int grpid; }; +struct nvme_ns_head { + struct list_head paths; + struct nvme_ns *n; + + char *sysfs_dir; +}; + struct nvme_ns { struct list_node entry; - struct list_head paths; struct nvme_subsystem *s;
Quoted 9 lines
src/nvme/private.h struct nvme_path { · line 54
Quoted 4 lines
src/nvme/private.h +1
-0struct nvme_ctrl *c; + struct nvme_ns_head *head; int fd; __u32 nsid;Why isn't '
nvme_subsystem' moved to 'nvme_ns_head'? That certainly is how the kernel structures are laid out, and it would make sense to follow it here...
Ideally yes it could be. However for libnvme, struct nvme_ns actually represents namespace head node when multipath is configured. And struct nvme_subsystem could be directly accessed from struct nvme_ns as we maintain a pointer to the nvme_subsystem from nvme_ns.
For instance, take an example of a shared namespace attached to two controllers in a subsystem and so we've got following topology for this subsystem.
# nvme show-topology
nvme-subsys2 - NQN=nqn.1994-11.com.samsung:nvme:PM1735a:2.5-inch:S6RTNE0R900057
hostnqn=nqn.2014-08.org.nvmexpress:uuid:41528538-e8ad-4eaf-84a7-9c552917d988
\
+- ns 1
\
+- nvme0 pcie 052e:78:00.0 live optimized
+- nvme2 pcie 058e:78:00.0 live optimizedAnd multipath sysfs link to reach the shared namespace from head node (nvme2n1):
# tree /sys/class/nvme-subsystem/nvme-subsys2/nvme2n1/multipath/ /sys/class/nvme-subsystem/nvme-subsys2/nvme2n1/multipath/ ├── nvme2c0n1 -> ../../../../../pci052e:78/052e:78:00.0/nvme/nvme0/nvme2c0n1 └── nvme2c2n1 -> ../../../../../pci058e:78/058e:78:00.0/nvme/nvme2/nvme2c2n1
If we parse above topology in libnvme then we get,
struct nvme_subsystem::name => nvme-subsys2
struct nvme_subsystem::namespaces => <list-of-struct nvme_ns-under-this-subsys>
//first path
struct nvme_ns_path::name => nvme2c0n1
struct nvme_ns_path::sysfs_dir => /sys/class/nvme/nvme0/nvme2c0n1/ //second path
struct nvme_ns_path::name => nvme2c2n1
struct nvme_ns_path::sysfs_dir => /sys/class/nvme/nvme2/nvme2c2n1/
struct nvme_ns_head::paths => struct nvme_ns_path (first path), struct nvme_ns_path (second path) struct nvme_ns_head::sysfs_dir =>/sys/class/nvme-subsystem/nvme-subsys2/nvme2n1/multipath
struct nvme_ns::name => nvme2n1
struct nvme_ns::sysfs_dir => /sys/class/nvme-subsystem/nvme-subsys2/nvme2n1/ struct nvme_ns::head => struct nvme_ns_head
struct nvme_ns::s => struct nvme_subsystem
So as we could see above, struct nvme_ns::s points to nvme_susbsystem and struct nvme_ns actually represents the head node.
Quoted 193 lines
Quoted 193 lines
src/nvme/tree.c +4
-2@@ -564,12 +564,12 @@ nvme_ns_t nvme_subsystem_next_ns(nvme_subsystem_t s, nvme_ns_t n) nvme_path_t nvme_namespace_first_path(nvme_ns_t ns) { - return list_top(&ns->paths, struct nvme_path, nentry); + return list_top(&ns->head->paths, struct nvme_path, nentry); } nvme_path_t nvme_namespace_next_path(nvme_ns_t ns, nvme_path_t p) { - return p ? list_next(&ns->paths, p, nentry) : NULL; + return p ? list_next(&ns->head->paths, p, nentry) : NULL; } static void __nvme_free_ns(struct nvme_ns *n)@@ -579,6 +579,8 @@ static void __nvme_free_ns(struct nvme_ns *n) free(n->generic_name); free(n->name); free(n->sysfs_dir); + free(n->head->sysfs_dir); + free(n->head); free(n); } @@ -916,25 +918,6 @@ void nvme_free_path(struct nvme_path *p)free(p);
}
-static voidnvme_subsystem_set_path_ns(nvme_subsystem_t s, nvme_path_t p)src/nvme/tree.c +0
-18-{ - char n_name[32] = { }; - int i, c, nsid, ret; - nvme_ns_t n; - - ret = sscanf(nvme_path_get_name(p), "nvme%dc%dn%d", &i, &c, &nsid); - if (ret != 3) - return; - - sprintf(n_name, "nvme%dn%d", i, nsid); - nvme_subsystem_for_each_ns(s, n) { - if (!strcmp(n_name, nvme_ns_get_name(n))) { - list_add_tail(&n->paths, &p->nentry); - p->n = n; - } - } -} -static int
nvme_ctrl_scan_path(nvme_root_t r, struct nvme_ctrl *c, char *name){
structnvme_path*p;src/nvme/tree.c +1
-2@@ -973,7 +956,6 @@ static int nvme_ctrl_scan_path(nvme_root_t r, struct nvme_ctrl *c, char *name) } list_node_init(&p->nentry); - nvme_subsystem_set_path_ns(c->s, p); list_node_init(&p->entry); list_add_tail(&c->paths, &p->entry); return 0;@@ -2250,8 +2232,8 @@ nvme_ctrl_t nvme_scan_ctrl(nvme_root_t r, const char *name) return NULL; path = NULL; - nvme_ctrl_scan_namespaces(r, c); nvme_ctrl_scan_paths(r, c); + nvme_ctrl_scan_namespaces(r, c); return c; } @@ -2323,6 +2305,11 @@ const char *nvme_ns_get_sysfs_dir(nvme_ns_t n) return n->sysfs_dir;}
+const char *nvme_ns_head_get_sysfs_dir(nvme_ns_head_t head)src/nvme/tree.c +4
-0+{ + return head->sysfs_dir; +} +const char *nvme_ns_get_name(nvme_ns_t n)
{
returnn->name;src/nvme/tree.c +29
-0@@ -2749,7 +2736,11 @@ static void nvme_ns_set_generic_name(struct nvme_ns *n, const char *name) static nvme_ns_t nvme_ns_open(const char *sys_path, const char *name) { + int ret; struct nvme_ns *n; + struct nvme_ns_head *head; + struct stat arg; + _cleanup_free_ char *path = NULL; n = calloc(1, sizeof(*n)); if (!n) {@@ -2757,6 +2748,32 @@ static nvme_ns_t nvme_ns_open(const char *sys_path, const char *name) return NULL; } + head = calloc(1, sizeof(*head)); + if (!head) { + errno = ENOMEM; + free(n); + return NULL; + } + + head->n = n; + list_head_init(&head->paths); + ret = asprintf(&path, "%s/%s", sys_path, "multipath"); + if (ret < 0) { + errno = ENOMEM; + goto free_ns_head; + } + /* + * The sysfs-dir "multipath" is available only when nvme multipath + * is configured and we're running kernel version >= 6.14. + */ + ret = stat(path, &arg); + if (ret == 0) { + head->sysfs_dir = path; + path = NULL; + } else + head->sysfs_dir = NULL; + + n->head = head; n->fd = -1; n->name = strdup(name); @@ -2765,15 +2782,17 @@ static nvme_ns_t nvme_ns_open(const char *sys_path, const char *name)if (
nvme_ns_init(sys_path, n)!= 0) gotofree_ns;
-list_head_init(&n->paths);list_node_init(&n->entry);nvme_ns_release_fd(n); /* Do not leak fds */ +
return n;
free_ns:
free(n->generic_name);
free(n->name);
+free_ns_head:
+ free(head);
free(n);
return NULL;
}src/nvme/tree.c +0
-0@@ -2836,6 +2855,71 @@ nvme_ns_t nvme_scan_namespace(const char *name) return __nvme_scan_namespace(nvme_ns_sysfs_dir(), name); }+
src/nvme/tree.c +57
-0+static void nvme_ns_head_scan_path (nvme_subsystem_t s,nvme_ns_t n,char *name) +{ + nvme_ctrl_t c; + nvme_path_t p; + + nvme_subsystem_for_each_ctrl(s, c) { + nvme_ctrl_for_each_path(c, p) { + if (!strcmp(nvme_path_get_name(p), name)) { + list_add_tail(&n->head->paths, &p->nentry); + p->n = n; + return; + } + } + } +} + +static void nvme_subsystem_set_ns_path (nvme_subsystem_t s,nvme_ns_t n) +{ + struct nvme_ns_head *head = n->head; + + if (nvme_ns_head_get_sysfs_dir(head)) { + struct dirents paths = {}; + int i; + + /* + * When multipath is configured on kernel version >= 6.14, + * we use multipath sysfs link to get each path of a namespace. + */ + paths.num = nvme_scan_ns_head_paths(head, &paths.ents); + + for (i = 0; i < paths.num; i++) + nvme_ns_head_scan_path(s, n, paths.ents[i]->d_name); + } else { + nvme_ctrl_t c; + nvme_path_t p; + int ns_ctrl, ns_nsid, ret; + + /* + * If multipath is not configured or we're running on kernel + * version < 6.14, fallback to the old way. + */ + ret = sscanf(nvme_ns_get_name(n), "nvme%dn%d", + &ns_ctrl, &ns_nsid); + if (ret != 2) + return; + + nvme_subsystem_for_each_ctrl(s, c) { + nvme_ctrl_for_each_path(c, p) { + int p_subsys, p_ctrl, p_nsid; + + ret = sscanf(nvme_path_get_name(p), + "nvme%dc%dn%d", + &p_subsys, &p_ctrl, &p_nsid); + if (ret != 3) + continue; + if (ns_ctrl == p_subsys && ns_nsid == p_nsid) { + list_add_tail(&head->paths, &p->nentry);
Quoted 9 lines
Quoted 5 lines
+
p->n= n;
+ }
+ }
+ }
+ }Can't you just use the existing
nvme_subsystem_set_path_ns()function here instead of deleting and open-code it?
We can but then we don't levearge the newly added (efficient and probably more elegant) way of discovering the multipath information from the head node. So nvme_subsystem_set_ns_path is implemented such that if libnvme is running on the newer kernel supporting multipath sysfs link then we leverage that to discover multipath information. However if libnvme is running on older kernel which doesn't support multipath sysfs link then nvme_subsystem_set_ns_path falls back to the legacy way of discoveing multipath information.
Quoted 33 lines
Quoted 33 lines
src/nvme/tree.c +2
-0+} +static int
nvme_ctrl_scan_namespace(nvme_root_t r, struct nvme_ctrl *c, char *name)
{src/nvme/tree.c +1
-22@@ -2861,33 +2945,9 @@ static int nvme_ctrl_scan_namespace(nvme_root_t r, struct nvme_ctrl *c, n->s = c->s; n->c = c; list_add_tail(&c->namespaces, &n->entry); - return 0; -} - -static void nvme_subsystem_set_ns_path (nvme_subsystem_t s,nvme_ns_t n) -{ - nvme_ctrl_t c; - nvme_path_t p; - int ns_ctrl, ns_nsid, ret; - - ret = sscanf(nvme_ns_get_name(n), "nvme%dn%d", &ns_ctrl, &ns_nsid); - if (ret != 2) - return; + nvme_subsystem_set_ns_path(c->s, n); - nvme_subsystem_for_each_ctrl(s, c) { - nvme_ctrl_for_each_path(c, p) { - int p_subsys, p_ctrl, p_nsid; - - ret = sscanf(nvme_path_get_name(p), "nvme%dc%dn%d", - &p_subsys, &p_ctrl, &p_nsid); - if (ret != 3) - continue; - if (ns_ctrl == p_subsys && ns_nsid == p_nsid) { - list_add_tail(&n->paths, &p->nentry); - p->n = n;
Quoted 8 lines
src/nvme/tree.c static int nvme_ctrl_scan_namespace(nvme_root_t r, struct nvme_ctrl *c, · line 2953
Quoted 6 lines
src/nvme/tree.c +1
-3- } - } - } + return 0; } static int nvme_subsystem_scan_namespace(nvme_root_t r, nvme_subsystem_t s,Similar here; better to use the existing functions.
In this patch I re-wrote nvme_subsystem_set_ns_path which is now called either while scanning controler or subsystem. The idea is to use a common function which helps discover the multipath information. And as I stated before, on newer kernel this function uses the multiapth sysfs link to discover multiple paths however on older kernels it falls back to legacy way to discover mulipath infromation.
Thanks,
--Nilay
PATCH v2 2/4 tree: add queue-depth attribute for nvme path object
In reply to Nilay ShroffAdd a new attribute named "queue_depth" under the NVMe path object. This attribute is used by the iopolicy "queue-depth", which was introduced in kernel v6.11. However, the corresponding sysfs attribute for queue depth was only added in kernel v6.14.
The queue_depth value can be useful for observing which paths are selected for I/O forwarding, based on the depth of each path. To support this, export the attribute in libnvme.map so it can be accessed via nvme-cli.
--- src/libnvme.map | 1 + src/nvme/private.h | 1 + src/nvme/tree.c | 12 +++++++++++- src/nvme/tree.h | 8 ++++++++ 4 files changed, 21 insertions(+), 1 deletion(-)
src/libnvme.map +1 -0
@@ -317,6 +317,7 @@ LIBNVME_1_0 { nvme_path_get_ctrl; nvme_path_get_name; nvme_path_get_ns; + nvme_path_get_queue_depth; nvme_path_get_sysfs_dir; nvme_paths_filter; nvme_read_config;
src/nvme/private.h +1 -0
@@ -34,6 +34,7 @@ struct nvme_path {char *sysfs_dir;char *ana_state;int grpid; +int queue_depth; };struct nvme_ns_head {
src/nvme/tree.c +11 -1
@@ -903,6 +903,11 @@ const char *nvme_path_get_name(nvme_path_t p)return p->name; } +int nvme_path_get_queue_depth (nvme_path_t p) +{ +return p->queue_depth; +} +const char *nvme_path_get_ana_state (nvme_path_t p) {return p->ana_state;
@@ -921,7 +926,7 @@ void nvme_free_path(struct nvme_path *p)static int nvme_ctrl_scan_path (nvme_root_t r,struct nvme_ctrl *c,char *name) {struct nvme_path *p; - _cleanup_free_ char *path = NULL, *grpid = NULL; + _cleanup_free_ char *path = NULL, *grpid = NULL, *queue_depth = NULL;int ret; nvme_msg(r, LOG_DEBUG, "scan controller %s path %s\n",
@@ -955,6 +960,11 @@ static int nvme_ctrl_scan_path(nvme_root_t r, struct nvme_ctrl *c, char *name)sscanf (grpid,"%d" , &p->grpid); } + queue_depth =nvme_get_path_attr (p,"queue_depth" ); +if (queue_depth) { +sscanf (queue_depth,"%d" , &p->queue_depth); + } +list_node_init (&p->nentry);list_node_init (&p->entry);list_add_tail (&c->paths, &p->entry);
src/nvme/tree.h +8 -0
@@ -867,6 +867,14 @@ const char *nvme_path_get_sysfs_dir(nvme_path_t p);*/ const char *nvme_path_get_ana_state (nvme_path_t p); +/** +* nvme_path_get_queue_depth() - Queue depth of an nvme_path_t object +* @p: &nvme_path_t object +* +* Return: Queue depth of @p +*/ +int nvme_path_get_queue_depth (nvme_path_t p); +/** * nvme_path_get_ctrl() - Parent controller of an nvme_path_t object * @p: &nvme_path_t object
Signature · 1 line
2.49.0
Re: [PATCHv2 2/4] tree: add queue-depth attribute for nvme path object
Quoted 87 lines · Nilay Shroff
Add a new attribute named "
queue_depth" under the NVMe path object. This attribute is used by the iopolicy "queue-depth", which was introduced in kernel v6.11. However, the corresponding sysfs attribute for queue depth was only added in kernel v6.14.The
queue_depthvalue can be useful for observing which paths are selected for I/O forwarding, based on the depth of each path. To support this, export the attribute in libnvme.map so it can be accessed via nvme-cli.Signed-off-by Nilay Shroff <[email protected]>--- src/libnvme.map | 1 + src/nvme/private.h | 1 + src/nvme/tree.c | 12 +++++++++++- src/nvme/tree.h | 8 ++++++++ 4 files changed, 21 insertions(+), 1 deletion(-)src/libnvme.map +1
-0@@ -317,6 +317,7 @@ LIBNVME_1_0 { nvme_path_get_ctrl; nvme_path_get_name; nvme_path_get_ns; + nvme_path_get_queue_depth; nvme_path_get_sysfs_dir; nvme_paths_filter; nvme_read_config;src/nvme/private.h +1
-0@@ -34,6 +34,7 @@ struct nvme_path {char *sysfs_dir;char *ana_state;int grpid; +int queue_depth; };struct nvme_ns_head {src/nvme/tree.c +11
-1@@ -903,6 +903,11 @@ const char *nvme_path_get_name(nvme_path_t p)return p->name; } +int nvme_path_get_queue_depth (nvme_path_t p) +{ +return p->queue_depth; +} +const char *nvme_path_get_ana_state (nvme_path_t p) {return p->ana_state;@@ -921,7 +926,7 @@ void nvme_free_path(struct nvme_path *p)static int nvme_ctrl_scan_path (nvme_root_t r,struct nvme_ctrl *c,char *name) {struct nvme_path *p; - _cleanup_free_ char *path = NULL, *grpid = NULL; + _cleanup_free_ char *path = NULL, *grpid = NULL, *queue_depth = NULL;int ret; nvme_msg(r, LOG_DEBUG, "scan controller %s path %s\n",@@ -955,6 +960,11 @@ static int nvme_ctrl_scan_path(nvme_root_t r, struct nvme_ctrl *c, char *name)sscanf (grpid,"%d" , &p->grpid); } + queue_depth =nvme_get_path_attr (p,"queue_depth" ); +if (queue_depth) { +sscanf (queue_depth,"%d" , &p->queue_depth); + } +list_node_init (&p->nentry);list_node_init (&p->entry);list_add_tail (&c->paths, &p->entry);src/nvme/tree.h +1
-0@@ -867,6 +867,14 @@ const char *nvme_path_get_sysfs_dir(nvme_path_t p);*/ const char *nvme_path_get_ana_state (nvme_path_t p); +/**
src/nvme/tree.h const char *nvme_path_get_sysfs_dir(nvme_path_t p); · line 880
Quoted 10 lines
src/nvme/tree.h +7
-0+* nvme_path_get_queue_depth() - Queue depth of an nvme_path_t object +* @p: &nvme_path_t object +* +* Return: Queue depth of @p +*/ +int nvme_path_get_queue_depth (nvme_path_t p); +/** * nvme_path_get_ctrl() - Parent controller of an nvme_path_t object * @p: &nvme_path_t object
As discussed: saving the 'queue_depth' attribute is a bad idea, as it changes frequently.
I'd rather read the attribute value from sysfs whenever 'nvme_path_get_queue_depth' is called, and use the tree structure only to hold the pathname.
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Re: [PATCHv2 2/4] tree: add queue-depth attribute for nvme path object
Quoted 91 lines · Hannes Reinecke
Quoted 90 lines · Nilay Shroff
Add a new attribute named "
queue_depth" under the NVMe path object. This attribute is used by the iopolicy "queue-depth", which was introduced in kernel v6.11. However, the corresponding sysfs attribute for queue depth was only added in kernel v6.14.The
queue_depthvalue can be useful for observing which paths are selected for I/O forwarding, based on the depth of each path. To support this, export the attribute in libnvme.map so it can be accessed via nvme-cli.Signed-off-by Nilay Shroff <[email protected]>---
src/libnvme.map | 1 +
src/nvme/private.h| 1 +
src/nvme/tree.c | 12 +++++++++++-
src/nvme/tree.h | 8 ++++++++4 files changed, 21 insertions(+), 1 deletion(-)src/libnvme.map +1
-0@@ -317,6 +317,7 @@ LIBNVME_1_0 { nvme_path_get_ctrl; nvme_path_get_name; nvme_path_get_ns; + nvme_path_get_queue_depth; nvme_path_get_sysfs_dir; nvme_paths_filter; nvme_read_config;src/nvme/private.h +1
-0@@ -34,6 +34,7 @@ struct nvme_path { char *sysfs_dir; char *ana_state; int grpid; + int queue_depth; }; struct nvme_ns_head {src/nvme/tree.c +9
-1@@ -903,6 +903,11 @@ const char *nvme_path_get_name(nvme_path_t p) return p->name; } +int nvme_path_get_queue_depth(nvme_path_t p) +{ + return p->queue_depth; +} + const char *nvme_path_get_ana_state(nvme_path_t p) { return p->ana_state;@@ -921,7 +926,7 @@ void nvme_free_path(struct nvme_path *p) static int nvme_ctrl_scan_path(nvme_root_t r, struct nvme_ctrl *c, char *name) { struct nvme_path *p; - _cleanup_free_ char *path = NULL, *grpid = NULL; + _cleanup_free_ char *path = NULL, *grpid = NULL, *queue_depth = NULL; int ret; nvme_msg(r, LOG_DEBUG, "scan controller %s path %s\n",@@ -955,6 +960,11 @@ static int nvme_ctrl_scan_path(nvme_root_t r, struct nvme_ctrl *c, char *name) sscanf(grpid, "%d", &p->grpid); } + queue_depth = nvme_get_path_attr(p, "queue_depth"); + if (queue_depth) { + sscanf(queue_depth, "%d", &p->queue_depth); + } + list_node_init(&p->nentry); list_node_init(&p->entry); list_add_tail(&c->paths, &p->entry);src/nvme/tree.h +0
-0@@ -867,6 +867,14 @@ const char *nvme_path_get_sysfs_dir(nvme_path_t p);*/ const char *nvme_path_get_ana_state(nvme_path_t p);+/**
+ *nvme_path_get_queue_depth()- Queue depth of annvme_path_tobject
+ * @p:&nvme_path_tobject
+ *
+ * Return: Queue depth of @p
+ */src/nvme/tree.h +2
-0+int nvme_path_get_queue_depth (nvme_path_t p); +/**
Quoted 9 lines
Quoted 2 lines
*
nvme_path_get_ctrl()- Parent controller of annvme_path_tobject
* @p:&nvme_path_tobjectAs discussed: saving the '
queue_depth' attribute is a bad idea, as it changes frequently.
I'd rather read the attribute value from sysfs whenever 'nvme_path_get_queue_depth' is called, and use the tree structure only to hold the pathname.
Yes correct, but I thought we discussed about implementing non-cached version of API in libnvme 2.x. So I implemented cached version of queue_depth API. Daniel can you confirm?
Thanks,
--Nilay
Re: [PATCHv2 2/4] tree: add queue-depth attribute for nvme path object
Quoted 9 lines · Nilay Shroff
Quoted 6 lines
As discussed: saving the '
queue_depth' attribute is a bad idea, as it changes frequently.
I'd rather read the attribute value from sysfs whenever 'nvme_path_get_queue_depth' is called, and use the tree structure only to hold the pathname.Yes correct, but I thought we discussed about implementing non-cached version of API in libnvme 2.x. So I implemented cached version of
queue_depthAPI. Daniel can you confirm?
I was under the impression that we agreed on the default being non-cached. As it doesn't really make sense to have a cached anyway.
Re: [PATCHv2 2/4] tree: add queue-depth attribute for nvme path object
Quoted 7 lines · Daniel Wagner
Quoted 6 lines · Nilay Shroff
Quoted 6 lines
As discussed: saving the '
queue_depth' attribute is a bad idea, as it changes frequently.
I'd rather read the attribute value from sysfs whenever 'nvme_path_get_queue_depth' is called, and use the tree structure only to hold the pathname.
Quoted 6 lines
Quoted 3 lines
Yes correct, but I thought we discussed about implementing non-cached version of API in libnvme 2.x. So I implemented cached version of
queue_depthAPI. Daniel can you confirm?I was under the impression that we agreed on the default being non-cached. As it doesn't really make sense to have a cached anyway.
Okay, my bad.. probably some misunderstanding on my part. I will fix this in the next version of this patch.
Thanks,
--Nilay
PATCH v2 3/4 tree: add attribute numa_nodes for NVMe path object
In reply to Nilay ShroffAdd a new attribute named "numa_nodes" under the NVMe path object. This attribute is used by the iopolicy "numa". The numa_nodes value is stored for each NVMe path and represents the NUMA node(s) associated with it. When the iopolicy is set to "numa", I/O traffic originating from a given NUMA node will be forwarded through the corresponding NVMe path.
The numa_nodes attribute is useful for observing which NVMe path the kernel would choose for I/O forwarding based on NUMA affinity. To support this, export the attribute in libnvme.map so it can be accessed via nvme-cli.
--- src/libnvme.map | 1 + src/nvme/private.h | 1 + src/nvme/tree.c | 10 ++++++++++ src/nvme/tree.h | 8 ++++++++ 4 files changed, 20 insertions(+)
src/libnvme.map +1 -0
@@ -317,6 +317,7 @@ LIBNVME_1_0 { nvme_path_get_ctrl; nvme_path_get_name; nvme_path_get_ns; + nvme_path_get_numa_nodes; nvme_path_get_queue_depth; nvme_path_get_sysfs_dir; nvme_paths_filter;
src/nvme/private.h +1 -0
@@ -33,6 +33,7 @@ struct nvme_path {char *name;char *sysfs_dir;char *ana_state; +char *numa_nodes;int grpid;int queue_depth; };
src/nvme/tree.c +10 -0
@@ -913,6 +913,11 @@ const char *nvme_path_get_ana_state(nvme_path_t p)return p->ana_state; } +const char *nvme_path_get_numa_nodes (nvme_path_t p) +{ +return p->numa_nodes; +} +void nvme_free_path (struct nvme_path *p) {list_del_init (&p->entry);
@@ -920,6 +925,7 @@ void nvme_free_path(struct nvme_path *p)free (p->name);free (p->sysfs_dir);free (p->ana_state); +free (p->numa_nodes);free (p); }
@@ -955,6 +961,10 @@ static int nvme_ctrl_scan_path(nvme_root_t r, struct nvme_ctrl *c, char *name)if (!p->ana_state) p->ana_state =strdup ("optimized" ); + p->numa_nodes =nvme_get_path_attr (p,"numa_nodes" ); +if (!p->numa_nodes) + p->numa_nodes =strdup ("-1" ); + grpid =nvme_get_path_attr (p,"ana_grpid" );if (grpid) {sscanf (grpid,"%d" , &p->grpid);
src/nvme/tree.h +8 -0
@@ -867,6 +867,14 @@ const char *nvme_path_get_sysfs_dir(nvme_path_t p);*/ const char *nvme_path_get_ana_state (nvme_path_t p); +/** +* nvme_path_get_numa_nodes() - NUMA nodes of an nvme_path_t object +* @p : &nvme_path_t object +* +* Return: NUMA nodes associated to @p +*/ +const char *nvme_path_get_numa_nodes (nvme_path_t p); +/** * nvme_path_get_queue_depth() - Queue depth of an nvme_path_t object * @p: &nvme_path_t object
Signature · 1 line
2.49.0
Re: [PATCHv2 3/4] tree: add attribute numa_nodes for NVMe path object
Quoted 11 lines · Nilay Shroff
Add a new attribute named "
numa_nodes" under the NVMe path object. This attribute is used by the iopolicy "numa". Thenuma_nodesvalue is stored for each NVMe path and represents the NUMA node(s) associated with it. When the iopolicy is set to "numa", I/O traffic originating from a given NUMA node will be forwarded through the corresponding NVMe path.The
numa_nodesattribute is useful for observing which NVMe path the kernel would choose for I/O forwarding based on NUMA affinity. To support this, export the attribute in libnvme.map so it can be accessed via nvme-cli.
Quoted 8 lines
Signed-off-by Nilay Shroff <[email protected]>--- src/libnvme.map | 1 + src/nvme/private.h | 1 + src/nvme/tree.c | 10 ++++++++++ src/nvme/tree.h | 8 ++++++++ 4 files changed, 20 insertions(+)
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
PATCH v2 4/4 test: extend sysfs tree dump test
In reply to Nilay ShroffIn order to validate multipath link support extend the existing sysfs tree dump test.
The updated tree-pcie.tar.xz archive includes a sysfs dump with two NVMe subsystems. The nvme-subsys0 simulates a single-port NVMe disk with a private namespace that does not support multipath. And another nvme-subsys1 simulates a dual-port NVMe disk with a shared namespace attached to multiple controllers.
The json tree dump is also updated to include NVMe namespace path information. Since the archive content has changed, the expected output file tree-pcie.out is also updated to reflect the new structure.
--- src/nvme/json.c | 90 +++++++++++++++++++++++++++---- test/sysfs/data/tree-pcie.out | 53 ++++++++++++++---- test/sysfs/data/tree-pcie.tar.xz | Bin 19712 -> 12656 bytes 3 files changed, 124 insertions(+), 19 deletions(-)
src/nvme/json.c +81 -9
@@ -533,25 +533,97 @@ static void json_dump_ctrl(struct json_object *ctrl_array, nvme_ctrl_t c)json_object_array_add (ctrl_array, ctrl_obj); } +static unsigned int json_dump_subsys_multipath (nvme_subsystem_t s, +struct json_object *ns_array) +{ +nvme_ns_t n; +nvme_path_t p; +unsigned int i =0 ; + +nvme_subsystem_for_each_ns (s, n) { +struct json_object *ns_obj; +struct json_object *path_array; + + ns_obj =json_object_new_object (); +json_object_object_add (ns_obj,"nsid" , +json_object_new_int (nvme_ns_get_nsid (n))); +json_object_object_add (ns_obj,"name" , +json_object_new_string (nvme_ns_get_name (n))); + + path_array =json_object_new_array (); +nvme_namespace_for_each_path (n, p) { +struct json_object *path_obj; +struct json_object *ctrl_array; +nvme_ctrl_t c; + + path_obj =json_object_new_object (); +json_object_object_add (path_obj,"path" , +json_object_new_string (nvme_path_get_name (p))); +json_object_object_add (path_obj,"ANAState" , +json_object_new_string (nvme_path_get_ana_state (p))); +json_object_object_add (path_obj,"NUMANodes" , +json_object_new_string (nvme_path_get_numa_nodes (p))); +json_object_object_add (path_obj,"qdepth" , +json_object_new_int (nvme_path_get_queue_depth (p))); + + c =nvme_path_get_ctrl (p); + ctrl_array =json_object_new_array (); +json_dump_ctrl (ctrl_array, c); +json_object_object_add (path_obj,"controller" , ctrl_array); +json_object_array_add (path_array, path_obj); + } +json_object_object_add (ns_obj,"paths" , path_array); +json_object_array_add (ns_array, ns_obj); + i++; + } +return i; +} + +static void json_dump_subsys_non_multipath (nvme_subsystem_t s, +struct json_object *ns_array) +{ +nvme_ctrl_t c; +nvme_ns_t n; + +nvme_subsystem_for_each_ctrl (s, c) { +nvme_ctrl_for_each_ns (c, n) { +struct json_object *ctrl_array; +struct json_object *ns_obj; + + ns_obj =json_object_new_object (); +json_object_object_add (ns_obj,"nsid" , +json_object_new_int (nvme_ns_get_nsid (n))); +json_object_object_add (ns_obj,"name" , +json_object_new_string (nvme_ns_get_name (n))); + + ctrl_array =json_object_new_array (); +json_dump_ctrl (ctrl_array, c); +json_object_object_add (ns_obj,"controller" , ctrl_array); + +json_object_array_add (ns_array, ns_obj); + } + } +} +static void json_dump_subsys (struct json_object *subsys_array,nvme_subsystem_t s) { -nvme_ctrl_t c;struct json_object *subsys_obj =json_object_new_object (); -struct json_object *ctrl_array; +struct json_object *ns_array;json_object_object_add (subsys_obj,"name" ,json_object_new_string (nvme_subsystem_get_name (s)));json_object_object_add (subsys_obj,"nqn" ,json_object_new_string (nvme_subsystem_get_nqn (s))); - ctrl_array =json_object_new_array (); -nvme_subsystem_for_each_ctrl (s, c) { -json_dump_ctrl (ctrl_array, c); - } -if (json_object_array_length (ctrl_array)) -json_object_object_add (subsys_obj,"controllers" , ctrl_array); + + ns_array =json_object_new_array (); +if (!json_dump_subsys_multipath (s, ns_array)) +json_dump_subsys_non_multipath (s, ns_array); + +if (json_object_array_length (ns_array)) +json_object_object_add (subsys_obj,"namespaces" , ns_array);else -json_object_put (ctrl_array); +json_object_put (ns_array);json_object_array_add (subsys_array, subsys_obj); }
test/sysfs/data/tree-pcie.out +43 -10
66 lines not on this page · raw message
test/sysfs/data/tree-pcie.tar.xz binary +0 -0
Signature · 1 line
2.49.0
Re: [PATCHv2 4/4] test: extend sysfs tree dump test
Quoted 13 lines · Nilay Shroff
In order to validate multipath link support extend the existing sysfs tree dump test.
The updated tree-pcie.tar.xz archive includes a sysfs dump with two NVMe subsystems. The nvme-subsys0 simulates a single-port NVMe disk with a private namespace that does not support multipath. And another nvme-subsys1 simulates a dual-port NVMe disk with a shared namespace attached to multiple controllers.
The json tree dump is also updated to include NVMe namespace path information. Since the archive content has changed, the expected output file tree-pcie.out is also updated to reflect the new structure.
Quoted 7 lines
Signed-off-by Nilay Shroff <[email protected]>--- src/nvme/json.c | 90 +++++++++++++++++++++++++++---- test/sysfs/data/tree-pcie.out | 53 ++++++++++++++---- test/sysfs/data/tree-pcie.tar.xz | Bin 19712 -> 12656 bytes 3 files changed, 124 insertions(+), 19 deletions(-)
Not a big fan of having tar.gz files in git. Can't we create it on the fly and only store text files?
Otherwise:
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Re: [PATCHv2 4/4] test: extend sysfs tree dump test
Quoted 2 lines · Hannes Reinecke
Not a big fan of having tar.gz files in git. Can't we create it on the fly and only store text files?
Creating is not really an option, we could just store test data in the tree directly. I don't expect that the size of the repo is increasing a lot as the objects in git are also compressed. My main concern is that we have a lot of new files:
$ tar tf config-pcie.tar.xz sys/class/nvme/ sys/class/nvme/nvme1 sys/class/nvme/nvme0 sys/devices/pci0000:00/0000:00:06.0/0000:0f:00.0/nvme/nvme0/ sys/devices/pci0000:00/0000:00:06.0/0000:0f:00.0/nvme/nvme0/uevent sys/devices/pci0000:00/0000:00:06.0/0000:0f:00.0/nvme/nvme0/cntlid sys/devices/pci0000:00/0000:00:06.0/0000:0f:00.0/nvme/nvme0/address sys/devices/pci0000:00/0000:00:06.0/0000:0f:00.0/nvme/nvme0/nvme0c0n1/ sys/devices/pci0000:00/0000:00:06.0/0000:0f:00.0/nvme/nvme0/nvme0c0n1/uevent sys/devices/pci0000:00/0000:00:06.0/0000:0f:00.0/nvme/nvme0/nvme0c0n1/ext_range sys/devices/pci0000:00/0000:00:06.0/0000:0f:00.0/nvme/nvme0/nvme0c0n1/metadata_bytes sys/devices/pci0000:00/0000:00:06.0/0000:0f:00.0/nvme/nvme0/nvme0c0n1/range sys/devices/pci0000:00/0000:00:06.0/0000:0f:00.0/nvme/nvme0/nvme0c0n1/alignment_offset sys/devices/pci0000:00/0000:00:06.0/0000:0f:00.0/nvme/nvme0/nvme0c0n1/diskseq sys/devices/pci0000:00/0000:00:06.0/0000:0f:00.0/nvme/nvme0/nvme0c0n1/power/ [...]
Currently there are three tar.xz files. If the general mood is that there shouldn't be any tar.xz files around, I am fine to get rid of them.