nvme-cli: enhance the visibility of multipath using show-topology command
cover nvme-cli: enhance the visibility of multipath using show-topology command
Hi,
There has been recent work to improve the visibility of NVMe multipath configurations. The native NVMe multipath kernel driver supports three I/O path selection policies: numa, round-robin, and queue-depth. However, until now, users have had no insight into which path is selected by the multipath logic for forwarding I/O.
To address this, an RFC was proposed [1] and has since been merged into the Linux kernel as of v6.15. Following that, libnvme was extended to discover multiple paths to a shared NVMe namespace [2]. This patchset builds on those efforts to improve userspace observability of multipath configurations via nvme-cli.
This patchset contains two patches:
The first patch in the series fixes a <device> filter option using which we could be able to filter the output of nvme show-topology command.
The second patch in the series enhances the nvme show-topology command to display multipath configuration, including all discovered paths to a namespace, along with details such as ANA state, NUMA nodes, and queue depth. The show-topology "--ranking" option is extended with a new sub-
option: multipath.
As usual, any feedback is most welcome!
Thanks!
[1] https://lore.kernel.org/all/[email protected]/#t
[2] https://lore.kernel.org/all/[email protected]/
Nilay Shroff (2):
nvme: support <device> option in show-topology command
nvme: extend show-topology command to add support for multipath
nvme-print-binary.c | 1 + nvme-print-json.c | 25 ++++++++++++++----- nvme-print-stdout.c | 61 ++++++++++++++++++++++++++++++++++++++++++--- nvme-print.c | 4 ++- nvme-print.h | 1 + nvme.c | 21 ++++++++++++++-- nvme.h | 1 + 7 files changed, 101 insertions(+), 13 deletions(-)
Signature · 1 line
2.50.0
PATCH 1/2 nvme: support <device> option in show-topology command
Although the help text for the nvme show-topology command indicates support for a <device> option, this option has no effect in practice — specifying an NVMe device name does not filter the output.
This commit adds proper support for the <device> option, enabling users to filter the topology output based on the specified NVMe device.
--- nvme-print-stdout.c | 9 +++++++++ nvme.c | 17 ++++++++++++++++- 2 files changed, 25 insertions(+), 1 deletion(-)
nvme-print-stdout.c +9 -0
@@ -5679,6 +5679,15 @@ static void stdout_simple_topology(nvme_root_t r,nvme_for_each_host (r, h) {nvme_for_each_subsystem (h, s) { +bool no_ctrl = true; +nvme_ctrl_t c; + +nvme_subsystem_for_each_ctrl (s, c) + no_ctrl = false; + +if (no_ctrl) +continue ; +if (!first)printf ("\n" ); first = false;
nvme.c +16 -1
@@ -10059,6 +10059,8 @@ static int show_topology_cmd(int argc, char **argv, struct command *command, strconst char *ranking ="Ranking order: namespace|ctrl" ;nvme_print_flags_t flags; _cleanup_nvme_root_ nvme_root_t r = NULL; +char *devname =NULL ; +nvme_scan_filter_t filter =NULL ;enum nvme_cli_topo_ranking rank;int err;
@@ -10101,7 +10103,20 @@ static int show_topology_cmd(int argc, char **argv, struct command *command, strreturn -errno; } - err =nvme_scan_topology (r,NULL ,NULL ); +if (optind < argc) + devname =basename (argv[optind++]); + +if (devname) { +int subsys_id, nsid; + +if (sscanf (devname,"nvme%dn%d" , &subsys_id, &nsid) !=2 ) { +nvme_show_error ("Invalid device name %s\n" , devname); +return -EINVAL ; + } + filter = nvme_match_device_filter; + } + + err =nvme_scan_topology (r, filter, (void *)devname);if (err <0 ) {nvme_show_error ("Failed to scan topology: %s" ,nvme_strerror (errno));return err;
Signature · 1 line
2.50.0
PATCH 2/2 nvme: extend show-topology command to add support for multipath
In reply to Nilay ShroffThis commit enhances the show-topology command by adding support for NVMe multipath. With this change, users can now list all paths to a namespace from its corresponding head node device. Each NVMe path entry then also includes additional details such as ANA state, NUMA node, and queue depth, improving visibility into multipath configs. This information can be particularly helpful for debugging and analyzing NVMe multipath setups.
To support this functionality, the "--ranking" option of the nvme show-topology command has been extended with a new sub-option:
"multipath".
Since this enhancement is specific to NVMe multipath, the iopolicy configured under each subsystem is now always displayed. Previously, iopolicy was shown only with nvme show-topology verbose output, but it is now included by default to improve usability and provide better context when reviewing multipath configurations via show-topology.
With this update, users can view the multipath topology of a multi controller/port NVMe disk using:
$ nvme show-topology -r multipath
nvme-subsys2 - NQN=nvmet_subsystem
hostnqn=nqn.2014-08.org.nvmexpress:uuid:12b49f6e-0276-4746-b10c-56815b7e6dc2 iopolicy=numa
_ _ _<head-node>
/ _ _ _ <ana-state>
/ / _ _ _ <numa-node-list>
/ / / _ _ _<queue-depth>
| / / /
+- nvme2n1 (ns 1) / / / \ | | | +- nvme2c2n1 optimized 1,2 0 nvme2 tcp traddr=127.0.0.2,trsvcid=4460,src_addr=127.0.0.1 live +- nvme2c3n1 optimized 3,4 0 nvme3 tcp traddr=127.0.0.3,trsvcid=4460,src_addr=127.0.0.1 live
Please note that the headers shown above (e.g., <numa-node-list>, <ana-state>, <hed-node>, and <queue-depth>) are included for clarity only and are not part of the actual output.
--- nvme-print-binary.c | 1 + nvme-print-json.c | 25 ++++++++++++++++------ nvme-print-stdout.c | 52 +++++++++++++++++++++++++++++++++++++++++---- nvme-print.c | 4 +++- nvme-print.h | 1 + nvme.c | 4 +++- nvme.h | 1 + 7 files changed, 76 insertions(+), 12 deletions(-)
nvme-print-binary.c +1 -0
@@ -428,6 +428,7 @@ static struct print_ops binary_print_ops = { .print_nvme_subsystem_list = NULL, .topology_ctrl = NULL, .topology_namespace = NULL, + .topology_multipath = NULL,/* status and error messages */ .connect_msg = NULL,
nvme-print-json.c +19 -6
@@ -4538,19 +4538,30 @@ static unsigned int json_subsystem_topology_multipath(nvme_subsystem_t s, ns_attrs =json_create_object ();obj_add_int (ns_attrs,"NSID" ,nvme_ns_get_nsid (n)); +obj_add_str (ns_attrs,"Name" ,nvme_ns_get_name (n)); paths =json_create_array ();nvme_namespace_for_each_path (n, p) {struct json_object *path_attrs; - -nvme_ctrl_t c =nvme_path_get_ctrl (p); +struct json_object *ctrls, *ctrl_attrs; +nvme_ctrl_t c; path_attrs =json_create_object (); -obj_add_str (path_attrs,"Name" ,nvme_ctrl_get_name (c)); -obj_add_str (path_attrs,"Transport" ,nvme_ctrl_get_transport (c)); -obj_add_str (path_attrs,"Address" ,nvme_ctrl_get_address (c)); -obj_add_str (path_attrs,"State" ,nvme_ctrl_get_state (c)); +obj_add_str (path_attrs,"Path" ,nvme_path_get_name (p));obj_add_str (path_attrs,"ANAState" ,nvme_path_get_ana_state (p)); +obj_add_str (path_attrs,"NUMANodes" ,nvme_path_get_numa_nodes (p)); +obj_add_int (path_attrs,"Qdepth" ,nvme_path_get_queue_depth (p)); + + c =nvme_path_get_ctrl (p); + ctrls =json_create_array (); + ctrl_attrs =json_create_object (); +obj_add_str (ctrl_attrs,"Name" ,nvme_ctrl_get_name (c)); +obj_add_str (ctrl_attrs,"Transport" ,nvme_ctrl_get_transport (c)); +obj_add_str (ctrl_attrs,"Address" ,nvme_ctrl_get_address (c)); +obj_add_str (ctrl_attrs,"State" ,nvme_ctrl_get_state (c)); +array_add_obj (ctrls, ctrl_attrs); +obj_add_array (path_attrs,"Controller" , ctrls); +array_add_obj (paths, path_attrs); }obj_add_array (ns_attrs,"Paths" , paths);
@@ -4575,6 +4586,7 @@ static void json_print_nvme_subsystem_topology(nvme_subsystem_t s, ns_attrs =json_create_object ();obj_add_int (ns_attrs,"NSID" ,nvme_ns_get_nsid (n)); +obj_add_str (ns_attrs,"Name" ,nvme_ns_get_name (n)); ctrl =json_create_array (); ctrl_attrs =json_create_object ();
@@ -5334,6 +5346,7 @@ static struct print_ops json_print_ops = { .print_nvme_subsystem_list = json_print_nvme_subsystem_list, .topology_ctrl = json_simple_topology, .topology_namespace = json_simple_topology, + .topology_multipath = json_simple_topology,/* status and error messages */ .connect_msg = json_connect_msg,
nvme-print-stdout.c +48 -4
100 lines not on this page · raw message
nvme-print.c +3 -1
@@ -1551,8 +1551,10 @@ void nvme_show_topology(nvme_root_t r, {if (ranking ==NVME_CLI_TOPO_NAMESPACE )nvme_print (topology_namespace, flags, r); -else +else if (ranking ==NVME_CLI_TOPO_CTRL )nvme_print (topology_ctrl, flags, r); +else +nvme_print (topology_multipath, flags, r); }void nvme_show_message (bool error,const char *msg, ...)
nvme-print.h +1 -0
@@ -105,6 +105,7 @@ struct print_ops {void (*print_nvme_subsystem_list)(nvme_root_t r,bool show_ana);void (*topology_ctrl)(nvme_root_t r);void (*topology_namespace)(nvme_root_t r); +void (*topology_multipath)(nvme_root_t r);/* status and error messages */ void (*connect_msg)(nvme_ctrl_t c);
nvme.c +3 -1
@@ -10056,7 +10056,7 @@ static int tls_key(int argc, char **argv, struct command *command, struct pluginstatic int show_topology_cmd (int argc,char **argv,struct command *command,struct plugin *plugin) {const char *desc ="Show the topology\n" ; -const char *ranking ="Ranking order: namespace|ctrl" ; +const char *ranking ="Ranking order: namespace|ctrl|multipath" ;nvme_print_flags_t flags; _cleanup_nvme_root_ nvme_root_t r = NULL;char *devname =NULL ;
@@ -10092,6 +10092,8 @@ static int show_topology_cmd(int argc, char **argv, struct command *command, str rank =NVME_CLI_TOPO_NAMESPACE ; }else if (!strcmp (cfg.ranking,"ctrl" )) { rank =NVME_CLI_TOPO_CTRL ; + }else if (!strcmp (cfg.ranking,"multipath" )) { + rank =NVME_CLI_TOPO_MULTIPATH ; }else {nvme_show_error ("Invalid ranking argument: %s" , cfg.ranking);return -EINVAL ;
nvme.h +1 -0
@@ -46,6 +46,7 @@ typedef uint32_t nvme_print_flags_t;enum nvme_cli_topo_ranking {NVME_CLI_TOPO_NAMESPACE ,NVME_CLI_TOPO_CTRL , +NVME_CLI_TOPO_MULTIPATH , };#define SYS_NVME "/sys/class/nvme"
Signature · 1 line
2.50.0
Re: [PATCH 2/2] nvme: extend show-topology command to add support for multipath
Hi Nilay,
Quoted 11 lines · Nilay Shroff
$ nvme show-topology -r multipathnvme-subsys2 - NQN=nvmet_subsystem
hostnqn=nqn.2014-08.org.nvmexpress:uuid:12b49f6e-0276-4746-b10c-56815b7e6dc2 iopolicy=numa_ _ _<head-node>
/ _ _ _ <ana-state>
/ / _ _ _ <numa-node-list>
/ / / _ _ _<queue-depth>
| / / /
Quoted 8 lines
+- nvme2n1 (ns 1) / / / \ | | | +- nvme2c2n1 optimized 1,2 0 nvme2 tcp traddr=127.0.0.2,trsvcid=4460,src_addr=127.0.0.1 live +- nvme2c3n1 optimized 3,4 0 nvme3 tcp traddr=127.0.0.3,trsvcid=4460,src_addr=127.0.0.1 livePlease note that the headers shown above (e.g., <numa-node-list>, <ana-state>, <hed-node>, and <queue-depth>) are included for clarity only and are not part of the actual output.
I do like the annotation what the numbers mean. Could you also add those annotation to the output?
Rest looks good from a quick look.
Thanks,
Daniel
Re: [PATCH 2/2] nvme: extend show-topology command to add support for multipath
Quoted 19 lines · Daniel Wagner
Hi Nilay,
Quoted 16 lines · Nilay Shroff
$ nvme show-topology -r multipathnvme-subsys2 - NQN=nvmet_subsystem
hostnqn=nqn.2014-08.org.nvmexpress:uuid:12b49f6e-0276-4746-b10c-56815b7e6dc2 iopolicy=numa_ _ _<head-node>
/ _ _ _ <ana-state>
/ / _ _ _ <numa-node-list>
/ / / _ _ _<queue-depth>
| / / /+- nvme2n1 (ns 1) / / / \ | | | +- nvme2c2n1 optimized 1,2 0 nvme2 tcp traddr=127.0.0.2,trsvcid=4460,src_addr=127.0.0.1 live +- nvme2c3n1 optimized 3,4 0 nvme3 tcp traddr=127.0.0.3,trsvcid=4460,src_addr=127.0.0.1 live
Quoted 7 lines
Quoted 3 lines
Please note that the headers shown above (e.g., <numa-node-list>, <ana-state>, <hed-node>, and <queue-depth>) are included for clarity only and are not part of the actual output.
I do like the annotation what the numbers mean. Could you also add those annotation to the output?
Yes, I agree that the annotations explaining the numbers are helpful. However, if there are multiple subsystems on a host, including such annotations inline might make the output look cluttered.
Instead, I propose using the following tabular format, which is more readable and scalable. We can display this enhanced format only when the user specifies a --verbose option. Without the --verbose flag, we can continue using the current tree-style format.
Example verbose output:
$ nvme show-topology --verbose -r multipath
nvme-subsys2 - NQN=nvmet_subsystem
hostnqn=nqn.2014-08.org.nvmexpress:uuid:12b49f6e-0276-4746-b10c-56815b7e6dc2
iopolicy=numa
model=Linux
serial=d5dc1170f62100d47204
firmware=6.16.0-r
type=nvm<head-node> <namespace> <path> <ana-state> <numa-list> <queue-depth> <ctrl> <state> <transport> <address>
| | | | | | | | | | nvme2n1 1 nvme2c2n1 optimized 1,2 0 nvme2 live tcp traddr=127.0.0.2,trsvcid=4460,src_addr=127.0.0.1 nvme2n1 1 nvme2c3n1 optimized 3,4 0 nvme3 live tcp traddr=127.0.0.3,trsvcid=4460,src_addr=127.0.0.1
What do you think?
Thanks,
--Nilay
Re: [PATCH 2/2] nvme: extend show-topology command to add support for multipath
Hi Daniel,
A gentle ping about this patchset... Do you have any further recommendation as I suggested below?
Thanks,
--Nilay
Quoted 61 lines · Nilay Shroff
Quoted 19 lines · Daniel Wagner
Hi Nilay,
Quoted 16 lines · Nilay Shroff
$ nvme show-topology -r multipathnvme-subsys2 - NQN=nvmet_subsystem
hostnqn=nqn.2014-08.org.nvmexpress:uuid:12b49f6e-0276-4746-b10c-56815b7e6dc2 iopolicy=numa_ _ _<head-node>
/ _ _ _ <ana-state>
/ / _ _ _ <numa-node-list>
/ / / _ _ _<queue-depth>
| / / /+- nvme2n1 (ns 1) / / / \ | | | +- nvme2c2n1 optimized 1,2 0 nvme2 tcp traddr=127.0.0.2,trsvcid=4460,src_addr=127.0.0.1 live +- nvme2c3n1 optimized 3,4 0 nvme3 tcp traddr=127.0.0.3,trsvcid=4460,src_addr=127.0.0.1 liveQuoted 7 lines
Quoted 3 lines
Please note that the headers shown above (e.g., <numa-node-list>, <ana-state>, <hed-node>, and <queue-depth>) are included for clarity only and are not part of the actual output.
I do like the annotation what the numbers mean. Could you also add those annotation to the output?
Yes, I agree that the annotations explaining the numbers are helpful. However, if there are multiple subsystems on a host, including such annotations inline might make the output look cluttered.
Instead, I propose using the following tabular format, which is more readable and scalable. We can display this enhanced format only when the user specifies a --verbose option. Without the --verbose flag, we can continue using the current tree-style format.
Example verbose output:
$ nvme show-topology --verbose -r multipathnvme-subsys2 - NQN=nvmet_subsystem
hostnqn=nqn.2014-08.org.nvmexpress:uuid:12b49f6e-0276-4746-b10c-56815b7e6dc2 iopolicy=numa model=Linux serial=d5dc1170f62100d47204 firmware=6.16.0-r type=nvm<head-node> <namespace> <path> <ana-state> <numa-list> <queue-depth> <ctrl> <state> <transport> <address>
| | | | | | | | | | nvme2n1 1 nvme2c2n1 optimized 1,2 0 nvme2 live tcp traddr=127.0.0.2,trsvcid=4460,src_addr=127.0.0.1 nvme2n1 1 nvme2c3n1 optimized 3,4 0 nvme3 live tcp traddr=127.0.0.3,trsvcid=4460,src_addr=127.0.0.1What do you think?
Thanks,
--Nilay
Re: [PATCH 2/2] nvme: extend show-topology command to add support for multipath
Hi Nilay,
Sorry, I forgot about to reply.
Quoted 12 lines · Nilay Shroff
Quoted 12 lines
Example verbose output:
$ nvme show-topology --verbose -r multipathnvme-subsys2 - NQN=nvmet_subsystem
hostnqn=nqn.2014-08.org.nvmexpress:uuid:12b49f6e-0276-4746-b10c-56815b7e6dc2 iopolicy=numa model=Linux serial=d5dc1170f62100d47204 firmware=6.16.0-r type=nvm
Quoted 6 lines
Quoted 6 lines
<head-node> <namespace> <path> <ana-state> <numa-list> <queue-depth> <ctrl> <state> <transport> <address>
| | | | | | | | | | nvme2n1 1 nvme2c2n1 optimized 1,2 0 nvme2 live tcp traddr=127.0.0.2,trsvcid=4460,src_addr=127.0.0.1 nvme2n1 1 nvme2c3n1 optimized 3,4 0 nvme3 live tcp traddr=127.0.0.3,trsvcid=4460,src_addr=127.0.0.1What do you think?
Personally, I did like the previous annotation layout, probably because I am looking way to often the ftrace output. Though, I'd say it would be better to stick with the existing table layout, instead adding something completely new (nvme list, nvme list -v, etc).
Maybe introducing a bit of common code for printing these tables could make it also a bit more streamlined, just as an idea.
Thanks,
Daniel