Skip to contentSkip to conversation

nvme-cli: enhance the visibility of multipath using show-topology command

PATCH Superseded Nilay Shroff 2 participants 7 messages Jul 4, 2025 – Aug 5, 2025 NVMe

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

cover nvme-cli: enhance the visibility of multipath using show-topology command

Hi,

There has been recent work to improve the visibility of NVMe multipath configurations. The native NVMe multipath kernel driver supports three I/O path selection policies: numa, round-robin, and queue-depth. However, until now, users have had no insight into which path is selected by the multipath logic for forwarding I/O.

To address this, an RFC was proposed [1] and has since been merged into the Linux kernel as of v6.15. Following that, libnvme was extended to discover multiple paths to a shared NVMe namespace [2]. This patchset builds on those efforts to improve userspace observability of multipath configurations via nvme-cli.

This patchset contains two patches:
The first patch in the series fixes a <device> filter option using which we could be able to filter the output of nvme show-topology command.

The second patch in the series enhances the nvme show-topology command to display multipath configuration, including all discovered paths to a namespace, along with details such as ANA state, NUMA nodes, and queue depth. The show-topology "--ranking" option is extended with a new sub-
option: multipath.

As usual, any feedback is most welcome!

Thanks!

[1] https://lore.kernel.org/all/[email protected]/#t
[2] https://lore.kernel.org/all/[email protected]/

Nilay Shroff (2):
nvme: support <device> option in show-topology command
nvme: extend show-topology command to add support for multipath

 nvme-print-binary.c |  1 +
 nvme-print-json.c   | 25 ++++++++++++++-----
 nvme-print-stdout.c | 61 ++++++++++++++++++++++++++++++++++++++++++---
 nvme-print.c        |  4 ++-
 nvme-print.h        |  1 +
 nvme.c              | 21 ++++++++++++++--
 nvme.h              |  1 +
 7 files changed, 101 insertions(+), 13 deletions(-)
Signature · 1 line
2.50.0
#1 lore raw

PATCH 1/2 nvme: support <device> option in show-topology command

Although the help text for the nvme show-topology command indicates support for a <device> option, this option has no effect in practice — specifying an NVMe device name does not filter the output.

This commit adds proper support for the <device> option, enabling users to filter the topology output based on the specified NVMe device.

---
 nvme-print-stdout.c |  9 +++++++++
 nvme.c              | 17 ++++++++++++++++-
 2 files changed, 25 insertions(+), 1 deletion(-)
nvme-print-stdout.c +9 -0
index 7d7d4b27..2c97abd7 100644
@@ -5679,6 +5679,15 @@ static void stdout_simple_topology(nvme_root_t r,
 
 	nvme_for_each_host(r, h) {
 		nvme_for_each_subsystem(h, s) {
+			bool no_ctrl = true;
+			nvme_ctrl_t c;
+
+			nvme_subsystem_for_each_ctrl(s, c)
+				no_ctrl = false;
+
+			if (no_ctrl)
+				continue;
+
 			if (!first)
 				printf("\n");
 			first = false;
nvme.c +16 -1
index be0c5f23..772ab000 100644
@@ -10059,6 +10059,8 @@ static int show_topology_cmd(int argc, char **argv, struct command *command, str
 	const char *ranking = "Ranking order: namespace|ctrl";
 	nvme_print_flags_t flags;
 	_cleanup_nvme_root_ nvme_root_t r = NULL;
+	char *devname = NULL;
+	nvme_scan_filter_t filter = NULL;
 	enum nvme_cli_topo_ranking rank;
 	int err;
 
@@ -10101,7 +10103,20 @@ static int show_topology_cmd(int argc, char **argv, struct command *command, str
 		return -errno;
 	}
 
-	err = nvme_scan_topology(r, NULL, NULL);
+	if (optind < argc)
+		devname = basename(argv[optind++]);
+
+	if (devname) {
+		int subsys_id, nsid;
+
+		if (sscanf(devname, "nvme%dn%d", &subsys_id, &nsid) != 2) {
+			nvme_show_error("Invalid device name %s\n", devname);
+			return -EINVAL;
+		}
+		filter = nvme_match_device_filter;
+	}
+
+	err = nvme_scan_topology(r, filter, (void *)devname);
 	if (err < 0) {
 		nvme_show_error("Failed to scan topology: %s", nvme_strerror(errno));
 		return err;
Signature · 1 line
2.50.0
#2 lore raw

PATCH 2/2 nvme: extend show-topology command to add support for multipath

In reply to Nilay Shroff

This commit enhances the show-topology command by adding support for NVMe multipath. With this change, users can now list all paths to a namespace from its corresponding head node device. Each NVMe path entry then also includes additional details such as ANA state, NUMA node, and queue depth, improving visibility into multipath configs. This information can be particularly helpful for debugging and analyzing NVMe multipath setups.

To support this functionality, the "--ranking" option of the nvme show-topology command has been extended with a new sub-option:
"multipath".

Since this enhancement is specific to NVMe multipath, the iopolicy configured under each subsystem is now always displayed. Previously, iopolicy was shown only with nvme show-topology verbose output, but it is now included by default to improve usability and provide better context when reviewing multipath configurations via show-topology.

With this update, users can view the multipath topology of a multi controller/port NVMe disk using:

$ nvme show-topology -r multipath

nvme-subsys2 - NQN=nvmet_subsystem
hostnqn=nqn.2014-08.org.nvmexpress:uuid:12b49f6e-0276-4746-b10c-56815b7e6dc2 iopolicy=numa

_ _ _<head-node>
/ _ _ _ <ana-state>
/ / _ _ _ <numa-node-list>
/ / / _ _ _<queue-depth>
| / / /

 +- nvme2n1 (ns 1)  /      /  /
 \                 |      |  |
  +- nvme2c2n1 optimized 1,2 0 nvme2 tcp traddr=127.0.0.2,trsvcid=4460,src_addr=127.0.0.1 live
  +- nvme2c3n1 optimized 3,4 0 nvme3 tcp traddr=127.0.0.3,trsvcid=4460,src_addr=127.0.0.1 live

Please note that the headers shown above (e.g., <numa-node-list>, <ana-state>, <hed-node>, and <queue-depth>) are included for clarity only and are not part of the actual output.

---
 nvme-print-binary.c |  1 +
 nvme-print-json.c   | 25 ++++++++++++++++------
 nvme-print-stdout.c | 52 +++++++++++++++++++++++++++++++++++++++++----
 nvme-print.c        |  4 +++-
 nvme-print.h        |  1 +
 nvme.c              |  4 +++-
 nvme.h              |  1 +
 7 files changed, 76 insertions(+), 12 deletions(-)
nvme-print-binary.c +1 -0
index ec86e386..3ecdda01 100644
@@ -428,6 +428,7 @@ static struct print_ops binary_print_ops = {
 	.print_nvme_subsystem_list	= NULL,
 	.topology_ctrl			= NULL,
 	.topology_namespace		= NULL,
+	.topology_multipath		= NULL,
 
 	/* status and error messages */
 	.connect_msg			= NULL,
nvme-print-json.c +19 -6
index bbb948f5..8967aefa 100644
@@ -4538,19 +4538,30 @@ static unsigned int json_subsystem_topology_multipath(nvme_subsystem_t s,
 
 		ns_attrs = json_create_object();
 		obj_add_int(ns_attrs, "NSID", nvme_ns_get_nsid(n));
+		obj_add_str(ns_attrs, "Name", nvme_ns_get_name(n));
 
 		paths = json_create_array();
 		nvme_namespace_for_each_path(n, p) {
 			struct json_object *path_attrs;
-
-			nvme_ctrl_t c = nvme_path_get_ctrl(p);
+			struct json_object *ctrls, *ctrl_attrs;
+			nvme_ctrl_t c;
 
 			path_attrs = json_create_object();
-			obj_add_str(path_attrs, "Name", nvme_ctrl_get_name(c));
-			obj_add_str(path_attrs, "Transport", nvme_ctrl_get_transport(c));
-			obj_add_str(path_attrs, "Address", nvme_ctrl_get_address(c));
-			obj_add_str(path_attrs, "State", nvme_ctrl_get_state(c));
+			obj_add_str(path_attrs, "Path", nvme_path_get_name(p));
 			obj_add_str(path_attrs, "ANAState", nvme_path_get_ana_state(p));
+			obj_add_str(path_attrs, "NUMANodes", nvme_path_get_numa_nodes(p));
+			obj_add_int(path_attrs, "Qdepth", nvme_path_get_queue_depth(p));
+
+			c = nvme_path_get_ctrl(p);
+			ctrls = json_create_array();
+			ctrl_attrs = json_create_object();
+			obj_add_str(ctrl_attrs, "Name", nvme_ctrl_get_name(c));
+			obj_add_str(ctrl_attrs, "Transport", nvme_ctrl_get_transport(c));
+			obj_add_str(ctrl_attrs, "Address", nvme_ctrl_get_address(c));
+			obj_add_str(ctrl_attrs, "State", nvme_ctrl_get_state(c));
+			array_add_obj(ctrls, ctrl_attrs);
+			obj_add_array(path_attrs, "Controller", ctrls);
+
 			array_add_obj(paths, path_attrs);
 		}
 		obj_add_array(ns_attrs, "Paths", paths);
@@ -4575,6 +4586,7 @@ static void json_print_nvme_subsystem_topology(nvme_subsystem_t s,
 
 			ns_attrs = json_create_object();
 			obj_add_int(ns_attrs, "NSID", nvme_ns_get_nsid(n));
+			obj_add_str(ns_attrs, "Name", nvme_ns_get_name(n));
 
 			ctrl = json_create_array();
 			ctrl_attrs = json_create_object();
@@ -5334,6 +5346,7 @@ static struct print_ops json_print_ops = {
 	.print_nvme_subsystem_list	= json_print_nvme_subsystem_list,
 	.topology_ctrl			= json_simple_topology,
 	.topology_namespace		= json_simple_topology,
+	.topology_multipath		= json_simple_topology,
 
 	/* status and error messages */
 	.connect_msg			= json_connect_msg,
nvme-print-stdout.c +48 -4

100 lines not on this page · raw message

nvme-print.c +3 -1
index 742999fd..f243dee9 100644
@@ -1551,8 +1551,10 @@ void nvme_show_topology(nvme_root_t r,
 {
 	if (ranking == NVME_CLI_TOPO_NAMESPACE)
 		nvme_print(topology_namespace, flags, r);
-	else
+	else if (ranking == NVME_CLI_TOPO_CTRL)
 		nvme_print(topology_ctrl, flags, r);
+	else
+		nvme_print(topology_multipath, flags, r);
 }
 
 void nvme_show_message(bool error, const char *msg, ...)
nvme-print.h +1 -0
index 7329f0ae..4035c80a 100644
@@ -105,6 +105,7 @@ struct print_ops {
 	void (*print_nvme_subsystem_list)(nvme_root_t r, bool show_ana);
 	void (*topology_ctrl)(nvme_root_t r);
 	void (*topology_namespace)(nvme_root_t r);
+	void (*topology_multipath)(nvme_root_t r);
 
 	/* status and error messages */
 	void (*connect_msg)(nvme_ctrl_t c);
nvme.c +3 -1
index 772ab000..53a29d22 100644
@@ -10056,7 +10056,7 @@ static int tls_key(int argc, char **argv, struct command *command, struct plugin
 static int show_topology_cmd(int argc, char **argv, struct command *command, struct plugin *plugin)
 {
 	const char *desc = "Show the topology\n";
-	const char *ranking = "Ranking order: namespace|ctrl";
+	const char *ranking = "Ranking order: namespace|ctrl|multipath";
 	nvme_print_flags_t flags;
 	_cleanup_nvme_root_ nvme_root_t r = NULL;
 	char *devname = NULL;
@@ -10092,6 +10092,8 @@ static int show_topology_cmd(int argc, char **argv, struct command *command, str
 		rank = NVME_CLI_TOPO_NAMESPACE;
 	} else if (!strcmp(cfg.ranking, "ctrl")) {
 		rank = NVME_CLI_TOPO_CTRL;
+	} else if (!strcmp(cfg.ranking, "multipath")) {
+		rank = NVME_CLI_TOPO_MULTIPATH;
 	} else {
 		nvme_show_error("Invalid ranking argument: %s", cfg.ranking);
 		return -EINVAL;
nvme.h +1 -0
index 1a15fc18..c61aea6c 100644
@@ -46,6 +46,7 @@ typedef uint32_t nvme_print_flags_t;
 enum nvme_cli_topo_ranking {
 	NVME_CLI_TOPO_NAMESPACE,
 	NVME_CLI_TOPO_CTRL,
+	NVME_CLI_TOPO_MULTIPATH,
 };
 
 #define SYS_NVME "/sys/class/nvme"
Signature · 1 line
2.50.0
#3 lore raw

Re: [PATCH 2/2] nvme: extend show-topology command to add support for multipath

Hi Nilay,

Quoted 11 lines · Nilay Shroff
$ nvme show-topology -r multipath

nvme-subsys2 - NQN=nvmet_subsystem
hostnqn=nqn.2014-08.org.nvmexpress:uuid:12b49f6e-0276-4746-b10c-56815b7e6dc2 iopolicy=numa

_ _ _<head-node>
/ _ _ _ <ana-state>
/ / _ _ _ <numa-node-list>
/ / / _ _ _<queue-depth>
| / / /

Quoted 8 lines
 +- nvme2n1 (ns 1)  /      /  /
 \                 |      |  |
  +- nvme2c2n1 optimized 1,2 0 nvme2 tcp traddr=127.0.0.2,trsvcid=4460,src_addr=127.0.0.1 live
  +- nvme2c3n1 optimized 3,4 0 nvme3 tcp traddr=127.0.0.3,trsvcid=4460,src_addr=127.0.0.1 live

Please note that the headers shown above (e.g., <numa-node-list>, <ana-state>, <hed-node>, and <queue-depth>) are included for clarity only and are not part of the actual output.

I do like the annotation what the numbers mean. Could you also add those annotation to the output?

Rest looks good from a quick look.

Thanks,
Daniel

#4 lore raw

Re: [PATCH 2/2] nvme: extend show-topology command to add support for multipath

Quoted 19 lines · Daniel Wagner

Hi Nilay,

Quoted 16 lines · Nilay Shroff
$ nvme show-topology -r multipath

nvme-subsys2 - NQN=nvmet_subsystem
hostnqn=nqn.2014-08.org.nvmexpress:uuid:12b49f6e-0276-4746-b10c-56815b7e6dc2 iopolicy=numa

_ _ _<head-node>
/ _ _ _ <ana-state>
/ / _ _ _ <numa-node-list>
/ / / _ _ _<queue-depth>
| / / /

 +- nvme2n1 (ns 1)  /      /  /
 \                 |      |  |
  +- nvme2c2n1 optimized 1,2 0 nvme2 tcp traddr=127.0.0.2,trsvcid=4460,src_addr=127.0.0.1 live
  +- nvme2c3n1 optimized 3,4 0 nvme3 tcp traddr=127.0.0.3,trsvcid=4460,src_addr=127.0.0.1 live
Quoted 7 lines
Quoted 3 lines

Please note that the headers shown above (e.g., <numa-node-list>, <ana-state>, <hed-node>, and <queue-depth>) are included for clarity only and are not part of the actual output.

I do like the annotation what the numbers mean. Could you also add those annotation to the output?

Yes, I agree that the annotations explaining the numbers are helpful. However, if there are multiple subsystems on a host, including such annotations inline might make the output look cluttered.

Instead, I propose using the following tabular format, which is more readable and scalable. We can display this enhanced format only when the user specifies a --verbose option. Without the --verbose flag, we can continue using the current tree-style format.

Example verbose output:

$ nvme show-topology --verbose -r multipath

nvme-subsys2 - NQN=nvmet_subsystem

               hostnqn=nqn.2014-08.org.nvmexpress:uuid:12b49f6e-0276-4746-b10c-56815b7e6dc2
               iopolicy=numa
               model=Linux
               serial=d5dc1170f62100d47204
               firmware=6.16.0-r
               type=nvm

<head-node> <namespace> <path> <ana-state> <numa-list> <queue-depth> <ctrl> <state> <transport> <address>

    |          |             |           |           |            |              |       |          |             |
nvme2n1        1          nvme2c2n1   optimized     1,2           0            nvme2   live        tcp        traddr=127.0.0.2,trsvcid=4460,src_addr=127.0.0.1 
nvme2n1        1          nvme2c3n1   optimized     3,4           0            nvme3   live        tcp        traddr=127.0.0.3,trsvcid=4460,src_addr=127.0.0.1

What do you think?

Thanks,
--Nilay

#5 lore raw

Re: [PATCH 2/2] nvme: extend show-topology command to add support for multipath

Hi Daniel,

A gentle ping about this patchset... Do you have any further recommendation as I suggested below?

Thanks,
--Nilay

Quoted 61 lines · Nilay Shroff
Quoted 19 lines · Daniel Wagner

Hi Nilay,

Quoted 16 lines · Nilay Shroff
$ nvme show-topology -r multipath

nvme-subsys2 - NQN=nvmet_subsystem
hostnqn=nqn.2014-08.org.nvmexpress:uuid:12b49f6e-0276-4746-b10c-56815b7e6dc2 iopolicy=numa

_ _ _<head-node>
/ _ _ _ <ana-state>
/ / _ _ _ <numa-node-list>
/ / / _ _ _<queue-depth>
| / / /

 +- nvme2n1 (ns 1)  /      /  /
 \                 |      |  |
  +- nvme2c2n1 optimized 1,2 0 nvme2 tcp traddr=127.0.0.2,trsvcid=4460,src_addr=127.0.0.1 live
  +- nvme2c3n1 optimized 3,4 0 nvme3 tcp traddr=127.0.0.3,trsvcid=4460,src_addr=127.0.0.1 live
Quoted 7 lines
Quoted 3 lines

Please note that the headers shown above (e.g., <numa-node-list>, <ana-state>, <hed-node>, and <queue-depth>) are included for clarity only and are not part of the actual output.

I do like the annotation what the numbers mean. Could you also add those annotation to the output?

Yes, I agree that the annotations explaining the numbers are helpful. However, if there are multiple subsystems on a host, including such annotations inline might make the output look cluttered.

Instead, I propose using the following tabular format, which is more readable and scalable. We can display this enhanced format only when the user specifies a --verbose option. Without the --verbose flag, we can continue using the current tree-style format.

Example verbose output:

$ nvme show-topology --verbose -r multipath

nvme-subsys2 - NQN=nvmet_subsystem

               hostnqn=nqn.2014-08.org.nvmexpress:uuid:12b49f6e-0276-4746-b10c-56815b7e6dc2
               iopolicy=numa
               model=Linux
               serial=d5dc1170f62100d47204
               firmware=6.16.0-r
               type=nvm

<head-node> <namespace> <path> <ana-state> <numa-list> <queue-depth> <ctrl> <state> <transport> <address>

    |          |             |           |           |            |              |       |          |             |
nvme2n1        1          nvme2c2n1   optimized     1,2           0            nvme2   live        tcp        traddr=127.0.0.2,trsvcid=4460,src_addr=127.0.0.1 
nvme2n1        1          nvme2c3n1   optimized     3,4           0            nvme3   live        tcp        traddr=127.0.0.3,trsvcid=4460,src_addr=127.0.0.1

What do you think?

Thanks,
--Nilay

#6 lore raw

Re: [PATCH 2/2] nvme: extend show-topology command to add support for multipath

Hi Nilay,

Sorry, I forgot about to reply.

Quoted 12 lines · Nilay Shroff
Quoted 12 lines

Example verbose output:

$ nvme show-topology --verbose -r multipath

nvme-subsys2 - NQN=nvmet_subsystem

               hostnqn=nqn.2014-08.org.nvmexpress:uuid:12b49f6e-0276-4746-b10c-56815b7e6dc2
               iopolicy=numa
               model=Linux
               serial=d5dc1170f62100d47204
               firmware=6.16.0-r
               type=nvm
Quoted 6 lines
Quoted 6 lines

<head-node> <namespace> <path> <ana-state> <numa-list> <queue-depth> <ctrl> <state> <transport> <address>

    |          |             |           |           |            |              |       |          |             |
nvme2n1        1          nvme2c2n1   optimized     1,2           0            nvme2   live        tcp        traddr=127.0.0.2,trsvcid=4460,src_addr=127.0.0.1 
nvme2n1        1          nvme2c3n1   optimized     3,4           0            nvme3   live        tcp        traddr=127.0.0.3,trsvcid=4460,src_addr=127.0.0.1

What do you think?

Personally, I did like the previous annotation layout, probably because I am looking way to often the ftrace output. Though, I'd say it would be better to stick with the existing table layout, instead adding something completely new (nvme list, nvme list -v, etc).

Maybe introducing a bit of common code for printing these tables could make it also a bit more streamlined, just as an idea.

Thanks,
Daniel