libnvme: add support for discovering multipath of a shared ns
cover libnvme: add support for discovering multipath of a shared ns
Hi,
Recently released Linux kernel v6.14 added support for easily discovering multiple paths to a shared NVMe namespace. This multipath information is exposed to userspace via a new sysfs group attribute named "multipath", located under /sys/block/<ns-blkdev>/. More details on this functionality can be found here [1].
This patch series leverages that new functionality to discover multiple paths to a shared namespace and exposes that information in libnvme so that it can later be used by nvme-cli.
There are three patches in this series:
The first patch adds support for discovering NVMe paths using the sysfs "multipath" group attribute.
The second patch adds a new "queue_depth" attribute under the NVMe path object. The third patch adds a new "numa_nodes" attribute under the NVMe path object.
[1]: https://lore.kernel.org/all/[email protected]/
Nilay Shroff (3):
tree: add support for discovering nvme paths using sysfs multipath
link
tree: add queue-depth attribute for nvme path object
tree: add attribute numa_nodes for NVMe path object
src/libnvme.map | 2 + src/nvme/filters.c | 6 ++ src/nvme/filters.h | 9 +++ src/nvme/private.h | 11 ++- src/nvme/tree.c | 184 ++++++++++++++++++++++++++++++++------------- src/nvme/tree.h | 25 ++++++ 6 files changed, 184 insertions(+), 53 deletions(-)
Signature · 1 line
2.49.0
PATCH 1/3 tree: add support for discovering nvme paths using sysfs multipath link
With kernel v6.14, when NVMe native multipath is enabled, it is possible to discover all paths to a shared namespace using the sysfs multipath link. Leverage this functionality to update the path links for each shared NVMe namespace, simplifying path discovery and management. Please note that this change is fully backward compatible.
--- src/nvme/filters.c | 6 ++ src/nvme/filters.h | 9 +++ src/nvme/private.h | 9 ++- src/nvme/tree.c | 162 +++++++++++++++++++++++++++++++-------------- src/nvme/tree.h | 9 +++ 5 files changed, 143 insertions(+), 52 deletions(-)
src/nvme/filters.c +6 -0
@@ -105,3 +105,9 @@ int nvme_scan_ctrl_namespaces(nvme_ctrl_t c, struct dirent ***ns)return scandir (nvme_ctrl_get_sysfs_dir (c), ns, nvme_namespace_filter, alphasort); } + +int nvme_scan_ns_head_paths (nvme_ns_head_t head,struct dirent ***paths) +{ +return scandir (nvme_ns_head_get_sysfs_dir (head), paths, + nvme_paths_filter, alphasort); +}
src/nvme/filters.h +9 -0
@@ -94,4 +94,13 @@ int nvme_scan_ctrl_namespace_paths(nvme_ctrl_t c, struct dirent ***paths);*/ int nvme_scan_ctrl_namespaces (nvme_ctrl_t c,struct dirent ***ns); +/** +* nvme_scan_ns_head_paths() - Scan for namespace paths +* @head: Namespace head node to scan +* @paths : Pointer to array of dirents +* +* Return: number of entries in @ents +*/ +int nvme_scan_ns_head_paths (nvme_ns_head_t head,struct dirent ***paths); +#endif /* _LIBNVME_FILTERS_H */
src/nvme/private.h +8 -1
@@ -36,12 +36,19 @@ struct nvme_path {int grpid; }; +struct nvme_ns_head { +struct list_head paths; +struct nvme_ns *n; + +char *sysfs_dir; +}; +struct nvme_ns {struct list_node entry; -struct list_head paths;struct nvme_subsystem *s;struct nvme_ctrl *c; +struct nvme_ns_head *head;int fd;__u32 nsid;
src/nvme/tree.c +111 -51
249 lines not on this page · raw message
src/nvme/tree.h +9 -0
@@ -27,6 +27,7 @@*/ typedef struct nvme_ns *nvme_ns_t ; +typedef struct nvme_ns_head *nvme_ns_head_t ;typedef struct nvme_path *nvme_path_t ;typedef struct nvme_ctrl *nvme_ctrl_t ;typedef struct nvme_subsystem *nvme_subsystem_t ;
@@ -1091,6 +1092,14 @@ void nvme_ctrl_set_dhchap_host_key(nvme_ctrl_t c, const char *key);*/ const char *nvme_ctrl_get_dhchap_key (nvme_ctrl_t c); +/** +* nvme_ns_head_get_sysfs_dir() - sysfs dir of namespave head +* @head: namespace head instance +* +* Returns: sysfs directory name of @head +*/ +const char *nvme_ns_head_get_sysfs_dir (nvme_ns_head_t head); +/** * nvme_ctrl_set_dhchap_key() - Set controller key * @c: Controller for which the key should be set
Signature · 1 line
2.49.0
Re: [PATCH 1/3] tree: add support for discovering nvme paths using sysfs multipath link
Hi Nilay,
Quoted 4 lines · Nilay Shroff
With kernel v6.14, when NVMe native multipath is enabled, it is possible to discover all paths to a shared namespace using the sysfs multipath link. Leverage this functionality to update the path links for each shared NVMe namespace, simplifying path discovery and management.
Since this is not a trivial change, could you extend this section a bit how it works? I understand from reading the patch, a new head namespace object is introduced which links to the different paths.
Quoted 2 lines
Please note that
this change is fully backward compatible.
Could you elaborate what that means? libnvme should still work on older kernels.
Also could you add a tests for this? The is already a test for the current sysfs tree parser, see test/sysfs.
Thanks,
Daniel
Re: [PATCH 1/3] tree: add support for discovering nvme paths using sysfs multipath link
Quoted 11 lines · Daniel Wagner
Hi Nilay,
Quoted 4 lines · Nilay Shroff
With kernel v6.14, when NVMe native multipath is enabled, it is possible to discover all paths to a shared namespace using the sysfs multipath link. Leverage this functionality to update the path links for each shared NVMe namespace, simplifying path discovery and management.
Since this is not a trivial change, could you extend this section a bit how it works? I understand from reading the patch, a new head namespace object is introduced which links to the different paths.
Sure, I will update the commit and add more details.
Quoted 6 lines
Quoted 3 lines
Please note that
this change is fully backward compatible.Could you elaborate what that means? libnvme should still work on older kernels.
Yeah I think as it's given that libnvme should work with older kernels, I probably don't need to spell that out explicitly. The intention is just to mention that this change would not break running libnvme on older kernels. But I think I could remove this sentence.
Quoted 4 lines
Also could you add a tests for this? The is already a test for the current sysfs tree parser, see test/sysfs.
Yes I'd add a testcase for this change. How about extending the existing test/sysfs/tree-dump.c? Or do you want me to add a new test case for this change?
Thanks,
--Nilay
Re: [PATCH 1/3] tree: add support for discovering nvme paths using sysfs multipath link
Quoted 9 lines · Nilay Shroff
Quoted 5 lines
Quoted 2 lines
Please note that
this change is fully backward compatible.Could you elaborate what that means? libnvme should still work on older kernels.
Yeah I think as it's given that libnvme should work with older kernels, I probably don't need to spell that out explicitly. The intention is just to mention that this change would not break running libnvme on older kernels. But I think I could remove this sentence.
For some reason I read 'this change is NOT fully backward compatible'. My bad, sorry.
Quoted 6 lines
Quoted 3 lines
Also could you add a tests for this? The is already a test for the current sysfs tree parser, see test/sysfs.
Yes I'd add a testcase for this change. How about extending the existing
test/sysfs/tree-dump.c? Or do you want me to add a new test case for this change?
Ideally you just need to add new sysfs dump (scripts/collect-sysfs.sh) and have the corresponding output placed in the data directory. If not, yes please extend tree-dump.c.
Re: [PATCH 1/3] tree: add support for discovering nvme paths using sysfs multipath link
Quoted 17 lines · Daniel Wagner
Quoted 9 lines · Nilay Shroff
Quoted 5 lines
Quoted 2 lines
Please note that
this change is fully backward compatible.Could you elaborate what that means? libnvme should still work on older kernels.
Yeah I think as it's given that libnvme should work with older kernels, I probably don't need to spell that out explicitly. The intention is just to mention that this change would not break running libnvme on older kernels. But I think I could remove this sentence.
For some reason I read 'this change is NOT fully backward compatible'. My bad, sorry.
Quoted 3 lines
Quoted 3 lines
Also could you add a tests for this? The is already a test for the current sysfs tree parser, see test/sysfs.
Quoted 7 lines
Quoted 3 lines
Yes I'd add a testcase for this change. How about extending the existing
test/sysfs/tree-dump.c? Or do you want me to add a new test case for this change?Ideally you just need to add new sysfs dump (
scripts/collect-sysfs.sh) and have the corresponding output placed in the data directory. If not, yes please extendtree-dump.c.
I see that the current tree-dump.c test program neither dumps namespace nor dumps namespace path. So I'd extend this program. I'd also replace the existing sysfs dump with another dump which shall have two nvme subsystems. The one of the subsystems shall represent a multi port disk with a shared namespace reachable through multipath, and another subsystem shall represent a single-port disk with a private namespace. So this way, we'd be able to cover both shared/ multipath namespace as well as private namespace use cases.
Thanks,
--Nilay
PATCH 2/3 tree: add queue-depth attribute for nvme path object
In reply to Nilay ShroffAdd a new attribute named "queue_depth" under the NVMe path object. This attribute is used by the iopolicy "queue-depth", which was introduced in kernel v6.11. However, the corresponding sysfs attribute for queue depth was only added in kernel v6.14.
The queue_depth value can be useful for observing which paths are selected for I/O forwarding, based on the depth of each path. To support this, export the attribute in libnvme.map so it can be accessed via nvme-cli.
--- src/libnvme.map | 1 + src/nvme/private.h | 1 + src/nvme/tree.c | 12 +++++++++++- src/nvme/tree.h | 8 ++++++++ 4 files changed, 21 insertions(+), 1 deletion(-)
src/libnvme.map +1 -0
@@ -317,6 +317,7 @@ LIBNVME_1_0 { nvme_path_get_ctrl; nvme_path_get_name; nvme_path_get_ns; + nvme_path_get_queue_depth; nvme_path_get_sysfs_dir; nvme_paths_filter; nvme_read_config;
src/nvme/private.h +1 -0
@@ -34,6 +34,7 @@ struct nvme_path {char *sysfs_dir;char *ana_state;int grpid; +int queue_depth; };struct nvme_ns_head {
src/nvme/tree.c +11 -1
@@ -903,6 +903,11 @@ const char *nvme_path_get_name(nvme_path_t p)return p->name; } +int nvme_path_get_queue_depth (nvme_path_t p) +{ +return p->queue_depth; +} +const char *nvme_path_get_ana_state (nvme_path_t p) {return p->ana_state;
@@ -921,7 +926,7 @@ void nvme_free_path(struct nvme_path *p)static int nvme_ctrl_scan_path (nvme_root_t r,struct nvme_ctrl *c,char *name) {struct nvme_path *p; - _cleanup_free_ char *path = NULL, *grpid = NULL; + _cleanup_free_ char *path = NULL, *grpid = NULL, *queue_depth = NULL;int ret; nvme_msg(r, LOG_DEBUG, "scan controller %s path %s\n",
@@ -955,6 +960,11 @@ static int nvme_ctrl_scan_path(nvme_root_t r, struct nvme_ctrl *c, char *name)sscanf (grpid,"%d" , &p->grpid); } + queue_depth =nvme_get_path_attr (p,"queue_depth" ); +if (queue_depth) { +sscanf (queue_depth,"%d" , &p->queue_depth); + } +list_node_init (&p->nentry);list_node_init (&p->entry);list_add_tail (&c->paths, &p->entry);
src/nvme/tree.h +8 -0
@@ -867,6 +867,14 @@ const char *nvme_path_get_sysfs_dir(nvme_path_t p);*/ const char *nvme_path_get_ana_state (nvme_path_t p); +/** +* nvme_path_get_queue_depth() - Queue depth of an nvme_path_t object +* @p: &nvme_path_t object +* +* Return: Queue depth of @p +*/ +int nvme_path_get_queue_depth (nvme_path_t p); +/** * nvme_path_get_ctrl() - Parent controller of an nvme_path_t object * @p: &nvme_path_t object
Signature · 1 line
2.49.0
Re: [PATCH 2/3] tree: add queue-depth attribute for nvme path object
Hi Nilay,
Quoted 9 lines · Nilay Shroff
Add a new attribute named "
queue_depth" under the NVMe path object. This attribute is used by the iopolicy "queue-depth", which was introduced in kernel v6.11. However, the corresponding sysfs attribute for queue depth was only added in kernel v6.14.The
queue_depthvalue can be useful for observing which paths are selected for I/O forwarding, based on the depth of each path. To support this, export the attribute in libnvme.map so it can be accessed via nvme-cli.
I understand the idea here is to get live update of the queue depth? This here will only read it once and report always the same value. This will work for nvme-cli but users which run the library for longer.
Thanks,
Daniel
PATCH 3/3 tree: add attribute numa_nodes for NVMe path object
In reply to Nilay ShroffAdd a new attribute named "numa_nodes" under the NVMe path object. This attribute is used by the iopolicy "numa". The numa_nodes value is stored for each NVMe path and represents the NUMA node(s) associated with it. When the iopolicy is set to "numa", I/O traffic originating from a given NUMA node will be forwarded through the corresponding NVMe path.
The numa_nodes attribute is useful for observing which NVMe path the kernel would choose for I/O forwarding based on NUMA affinity. To support this, export the attribute in libnvme.map so it can be accessed via nvme-cli.
--- src/libnvme.map | 1 + src/nvme/private.h | 1 + src/nvme/tree.c | 10 ++++++++++ src/nvme/tree.h | 8 ++++++++ 4 files changed, 20 insertions(+)
src/libnvme.map +1 -0
@@ -317,6 +317,7 @@ LIBNVME_1_0 { nvme_path_get_ctrl; nvme_path_get_name; nvme_path_get_ns; + nvme_path_get_numa_nodes; nvme_path_get_queue_depth; nvme_path_get_sysfs_dir; nvme_paths_filter;
src/nvme/private.h +1 -0
@@ -33,6 +33,7 @@ struct nvme_path {char *name;char *sysfs_dir;char *ana_state; +char *numa_nodes;int grpid;int queue_depth; };
src/nvme/tree.c +10 -0
@@ -913,6 +913,11 @@ const char *nvme_path_get_ana_state(nvme_path_t p)return p->ana_state; } +const char *nvme_path_get_numa_nodes (nvme_path_t p) +{ +return p->numa_nodes; +} +void nvme_free_path (struct nvme_path *p) {list_del_init (&p->entry);
@@ -920,6 +925,7 @@ void nvme_free_path(struct nvme_path *p)free (p->name);free (p->sysfs_dir);free (p->ana_state); +free (p->numa_nodes);free (p); }
@@ -955,6 +961,10 @@ static int nvme_ctrl_scan_path(nvme_root_t r, struct nvme_ctrl *c, char *name)if (!p->ana_state) p->ana_state =strdup ("optimized" ); + p->numa_nodes =nvme_get_path_attr (p,"numa_nodes" ); +if (!p->numa_nodes) + p->numa_nodes =strdup ("-1" ); + grpid =nvme_get_path_attr (p,"ana_grpid" );if (grpid) {sscanf (grpid,"%d" , &p->grpid);
src/nvme/tree.h +8 -0
@@ -867,6 +867,14 @@ const char *nvme_path_get_sysfs_dir(nvme_path_t p);*/ const char *nvme_path_get_ana_state (nvme_path_t p); +/** +* nvme_path_get_numa_nodes() - NUMA nodes of an nvme_path_t object +* @p : &nvme_path_t object +* +* Return: NUMA nodes associated to @p +*/ +const char *nvme_path_get_numa_nodes (nvme_path_t p); +/** * nvme_path_get_queue_depth() - Queue depth of an nvme_path_t object * @p: &nvme_path_t object
Signature · 1 line
2.49.0
Re: [PATCH 3/3] tree: add attribute numa_nodes for NVMe path object
Quoted 10 lines · Nilay Shroff
Add a new attribute named "
numa_nodes" under the NVMe path object. This attribute is used by the iopolicy "numa". Thenuma_nodesvalue is stored for each NVMe path and represents the NUMA node(s) associated with it. When the iopolicy is set to "numa", I/O traffic originating from a given NUMA node will be forwarded through the corresponding NVMe path.The
numa_nodesattribute is useful for observing which NVMe path the kernel would choose for I/O forwarding based on NUMA affinity. To support this, export the attribute in libnvme.map so it can be accessed via nvme-cli.
This one has the same limitation as the previous one. Given that libnvme currently caches everything, we could just accept this limitation for the time being. Any thoughts on this?
Re: [PATCH 3/3] tree: add attribute numa_nodes for NVMe path object
Quoted 7 lines · Daniel Wagner
Quoted 6 lines · Nilay Shroff
Add a new attribute named "
numa_nodes" under the NVMe path object. This attribute is used by the iopolicy "numa". Thenuma_nodesvalue is stored for each NVMe path and represents the NUMA node(s) associated with it. When the iopolicy is set to "numa", I/O traffic originating from a given NUMA node will be forwarded through the corresponding NVMe path.
Quoted 8 lines
Quoted 4 lines
The
numa_nodesattribute is useful for observing which NVMe path the kernel would choose for I/O forwarding based on NUMA affinity. To support this, export the attribute in libnvme.map so it can be accessed via nvme-cli.This one has the same limitation as the previous one. Given that libnvme currently caches everything, we could just accept this limitation for the time being. Any thoughts on this?
Yes agreed. So how about adding a new API, for instance, nvme_path_get_numa_nodes__no_cached which would not return the cached value but instead re-read the latest value from sysfs attribute and return the latest value? We may similarly extend other APIs where we don't want to retrieve cached value.
Thanks,
--Nilay
Re: [PATCH 3/3] tree: add attribute numa_nodes for NVMe path object
Quoted 17 lines · Nilay Shroff
Quoted 7 lines · Daniel Wagner
Quoted 6 lines · Nilay Shroff
Add a new attribute named "
numa_nodes" under the NVMe path object. This attribute is used by the iopolicy "numa". Thenuma_nodesvalue is stored for each NVMe path and represents the NUMA node(s) associated with it. When the iopolicy is set to "numa", I/O traffic originating from a given NUMA node will be forwarded through the corresponding NVMe path.Quoted 8 lines
Quoted 4 lines
The
numa_nodesattribute is useful for observing which NVMe path the kernel would choose for I/O forwarding based on NUMA affinity. To support this, export the attribute in libnvme.map so it can be accessed via nvme-cli.This one has the same limitation as the previous one. Given that libnvme currently caches everything, we could just accept this limitation for the time being. Any thoughts on this?
Quoted 6 lines
Yes agreed. So how about adding a new API, for instance,
nvme_path_get_numa_nodes__no_cachedwhich would not return the cached value but instead re-read the latest value from sysfs attribute and return the latest value? We may similarly extend other APIs where we don't want to retrieve cached value.
Adding _no_cached function is certainty an option for libnvme 1.x.
One of the API changes for the next major version of libnvme (aka 2.x) is to add an handle to all functions. Currently, we only have it for the fabrics API. If such handle is available, we could add no-cache flag instead duplicating all functions.
Maybe adding an explicit flags argument for 1.x is an option or should we just keep going with the cached only approach?
Re: [PATCH 3/3] tree: add attribute numa_nodes for NVMe path object
Quoted 27 lines · Daniel Wagner
Quoted 17 lines · Nilay Shroff
Quoted 7 lines · Daniel Wagner
Quoted 6 lines · Nilay Shroff
Add a new attribute named "
numa_nodes" under the NVMe path object. This attribute is used by the iopolicy "numa". Thenuma_nodesvalue is stored for each NVMe path and represents the NUMA node(s) associated with it. When the iopolicy is set to "numa", I/O traffic originating from a given NUMA node will be forwarded through the corresponding NVMe path.Quoted 8 lines
Quoted 4 lines
The
numa_nodesattribute is useful for observing which NVMe path the kernel would choose for I/O forwarding based on NUMA affinity. To support this, export the attribute in libnvme.map so it can be accessed via nvme-cli.This one has the same limitation as the previous one. Given that libnvme currently caches everything, we could just accept this limitation for the time being. Any thoughts on this?
Quoted 6 lines
Yes agreed. So how about adding a new API, for instance,
nvme_path_get_numa_nodes__no_cachedwhich would not return the cached value but instead re-read the latest value from sysfs attribute and return the latest value? We may similarly extend other APIs where we don't want to retrieve cached value.Adding
_no_cachedfunction is certainty an option for libnvme 1.x.
Quoted 7 lines
One of the API changes for the next major version of libnvme (aka 2.x) is to add an handle to all functions. Currently, we only have it for the fabrics API. If such handle is available, we could add no-cache flag instead duplicating all functions.
Maybe adding an explicit flags argument for 1.x is an option or should we just keep going with the cached only approach?
Both approaches — either adding a no-cache flag or introducing dedicated __no_cached APIs — would work. However, in my opinion, we should aim to use a consistent method across both libnvme 1.x and 2.x versions.
As we discussed during LSFMM, if we plan to implement an "nvme top" command, we would need non-cached versions of these APIs even for nvme-cli. So, using the same mechanism for both versions makes sense. Otherwise, we’d also have to maintain different logic in nvme top depending on the libnvme version, which adds unnecessary complexity.
Thanks,
--Nilay
Re: [PATCH 3/3] tree: add attribute numa_nodes for NVMe path object
Quoted 3 lines · Nilay Shroff
Both approaches — either adding a no-cache flag or introducing dedicated
__no_cachedAPIs — would work. However, in my opinion, we should aim to use a consistent method across both libnvme 1.x and 2.x versions.
There will be a lot of API changes for 2.x to address all the silly mistakes. That's way it will be a major version change.
Quoted 2 lines
As we discussed during LSFMM, if we plan to implement an "nvme top" command, we would need non-cached versions of these APIs even for nvme-cli.
Yes, that is why I brought up this discussion. For such a command we would need non cached versions. And I realize that we would need this feature also for other commands. So it's not just the queue depth which changes. Maybe we should just not try to add this feature to 1.x and rather start the 2.x project.
Quoted 3 lines
So, using the same mechanism for both versions makes sense. Otherwise, we’d also have to maintain different logic in nvme top depending on the libnvme version, which adds unnecessary complexity.
I don't see a big problem here. Either use the 1.x or 2.x.
Re: [PATCH 3/3] tree: add attribute numa_nodes for NVMe path object
Quoted 17 lines · Daniel Wagner
Quoted 3 lines · Nilay Shroff
Both approaches — either adding a no-cache flag or introducing dedicated
__no_cachedAPIs — would work. However, in my opinion, we should aim to use a consistent method across both libnvme 1.x and 2.x versions.There will be a lot of API changes for 2.x to address all the silly mistakes. That's way it will be a major version change.
Quoted 2 lines
As we discussed during LSFMM, if we plan to implement an "nvme top" command, we would need non-cached versions of these APIs even for nvme-cli.
Yes, that is why I brought up this discussion. For such a command we would need non cached versions. And I realize that we would need this feature also for other commands. So it's not just the queue depth which changes. Maybe we should just not try to add this feature to 1.x and rather start the 2.x project.
Quoted 5 lines
Quoted 3 lines
So, using the same mechanism for both versions makes sense. Otherwise, we’d also have to maintain different logic in nvme top depending on the libnvme version, which adds unnecessary complexity.
I don't see a big problem here. Either use the 1.x or 2.x.
Why don't we make this option as non-cached by default? In the end, there is limited usability in having it cached.
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Re: [PATCH 3/3] tree: add attribute numa_nodes for NVMe path object
Quoted 18 lines · Hannes Reinecke
Quoted 17 lines · Daniel Wagner
Quoted 3 lines · Nilay Shroff
Both approaches — either adding a no-cache flag or introducing dedicated
__no_cachedAPIs — would work. However, in my opinion, we should aim to use a consistent method across both libnvme 1.x and 2.x versions.There will be a lot of API changes for 2.x to address all the silly mistakes. That's way it will be a major version change.
Quoted 2 lines
As we discussed during LSFMM, if we plan to implement an "nvme top" command, we would need non-cached versions of these APIs even for nvme-cli.
Yes, that is why I brought up this discussion. For such a command we would need non cached versions. And I realize that we would need this feature also for other commands. So it's not just the queue depth which changes. Maybe we should just not try to add this feature to 1.x and rather start the 2.x project.
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
So, using the same mechanism for both versions makes sense. Otherwise, we’d also have to maintain different logic in nvme top depending on the libnvme version, which adds unnecessary complexity.
I don't see a big problem here. Either use the 1.x or 2.x.
Why don't we make this option as non-cached by default? In the end, there is limited usability in having it cached.
Sure, this is also fine. I suppose there will be some more conversion from cached to non-cached necessary for the 'nvme top' feature.
Re: [PATCH 3/3] tree: add attribute numa_nodes for NVMe path object
Quoted 23 lines · Daniel Wagner
Quoted 22 lines · Hannes Reinecke
Quoted 21 lines · Daniel Wagner
Quoted 3 lines · Nilay Shroff
Both approaches — either adding a no-cache flag or introducing dedicated
__no_cachedAPIs — would work. However, in my opinion, we should aim to use a consistent method across both libnvme 1.x and 2.x versions.There will be a lot of API changes for 2.x to address all the silly mistakes. That's way it will be a major version change.
Quoted 2 lines
As we discussed during LSFMM, if we plan to implement an "nvme top" command, we would need non-cached versions of these APIs even for nvme-cli.
Yes, that is why I brought up this discussion. For such a command we would need non cached versions. And I realize that we would need this feature also for other commands. So it's not just the queue depth which changes. Maybe we should just not try to add this feature to 1.x and rather start the 2.x project.
Quoted 3 lines
So, using the same mechanism for both versions makes sense. Otherwise, we’d also have to maintain different logic in nvme top depending on the libnvme version, which adds unnecessary complexity.
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
I don't see a big problem here. Either use the 1.x or 2.x.
Why don't we make this option as non-cached by default? In the end, there is limited usability in having it cached.
Sure, this is also fine. I suppose there will be some more conversion from cached to non-cached necessary for the 'nvme top' feature.
And I dimly remember I did some of this work already during the python binding work (where some attributes changed during the lifetime of the tree). Not sure what happened to that, though.
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Re: [PATCH 3/3] tree: add attribute numa_nodes for NVMe path object
In reply to Daniel WagnerQuoted 23 lines · Daniel Wagner
Quoted 22 lines · Hannes Reinecke
Quoted 21 lines · Daniel Wagner
Quoted 3 lines · Nilay Shroff
Both approaches — either adding a no-cache flag or introducing dedicated
__no_cachedAPIs — would work. However, in my opinion, we should aim to use a consistent method across both libnvme 1.x and 2.x versions.There will be a lot of API changes for 2.x to address all the silly mistakes. That's way it will be a major version change.
Quoted 2 lines
As we discussed during LSFMM, if we plan to implement an "nvme top" command, we would need non-cached versions of these APIs even for nvme-cli.
Yes, that is why I brought up this discussion. For such a command we would need non cached versions. And I realize that we would need this feature also for other commands. So it's not just the queue depth which changes. Maybe we should just not try to add this feature to 1.x and rather start the 2.x project.
Quoted 3 lines
So, using the same mechanism for both versions makes sense. Otherwise, we’d also have to maintain different logic in nvme top depending on the libnvme version, which adds unnecessary complexity.
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
I don't see a big problem here. Either use the 1.x or 2.x.
Why don't we make this option as non-cached by default? In the end, there is limited usability in having it cached.
Sure, this is also fine. I suppose there will be some more conversion from cached to non-cached necessary for the 'nvme top' feature.
Makes sense. For the this patchset, I will spin a new series and send after incorporating comments. We shall address cached vs non-cached version of APIs in another series (probably when we implement nvme top).
However, I like the idea of Hannes to only have non-cached APIs. May be, we should be able to convert existing cached APIs to non-cached APIs (or limit to only those APIs required for nvme top) without any side effect.
Thanks,
--Nilay
Re: [PATCH 3/3] tree: add attribute numa_nodes for NVMe path object
Quoted 4 lines · Nilay Shroff
Makes sense. For the this patchset, I will spin a new series and send after incorporating comments. We shall address cached vs non-cached version of APIs in another series (probably when we implement nvme top).
Okay.
Quoted 4 lines
However, I like the idea of Hannes to only have non-cached APIs. May be, we should be able to convert existing cached APIs to non-cached APIs (or limit to only those APIs required for nvme top) without any side effect.
I expect some more changes due to the 'nvme top' command, but as you suggest, let's first get this series right.
Quoted 8 lines · Nilay Shroff
Hi,
Recently released Linux kernel v6.14 added support for easily discovering multiple paths to a shared NVMe namespace. This multipath information is exposed to userspace via a new sysfs group attribute named "multipath", located under /sys/block/<ns-blkdev>/. More details on this functionality can be found here [1].
Sorry there's a typo here. The kernel version should be upcoming 6.15 and not 6.14. I will correct it in the next patch.
Thanks,
--Nilay