nvme-cli: enhance the visibility of multipath using show-topology command
Subsystems (2)
v3 cover nvme-cli: enhance the visibility of multipath using show-topology command
Hi,
There has been recent work to improve the visibility of NVMe multipath configurations. The native NVMe multipath kernel driver supports three I/O path selection policies: numa, round-robin, and queue-depth. However, until now, users have had no insight into which path is selected by the multipath logic for forwarding I/O.
To address this, an RFC was proposed [1] and has since been merged into the Linux kernel as of v6.15. Following that, libnvme was extended to discover multiple paths to a shared NVMe namespace [2]. This patchset builds on those efforts to improve userspace observability of multipath configurations via nvme-cli.
This patchset contains four patches:
The first patch in the series fixes a <device> filter option using which we could be able to filter the output of nvme show-topology command.
The second patch in the series enhances the nvme show-topology command to display multipath configuration, including all discovered paths to a namespace, along with details such as ANA state, NUMA nodes, and queue depth. The show-topology "--ranking" option is extended with a new sub-
option: multipath.
The third patch in the series adds common table APIs for printing nvme cli command output in tabular format. With these APIs, developers no longer need to pre-calculate column or row widths. The output is consistently aligned and easy to read.
The fourth patch adds support for printing show-topology in tabular form leveraging the introduced table APIs to produce well-aligned, easy-to-read output.
As usual, any feedback/sugegstion is most welcome!
Thanks!
Changes from v2:
- For iopolicy=numa, print NUMA nodes and exclude queue-depth. Similarly, for iopolicy=queue-depth, print queue-depth but exclude NUMA nodes. And for iopolicy=round-robin, exclude printing
both NUMA nodes and queue-depth. (Hannes Reinecke, Daniel
Wagner)
Link to v2: https://lore.kernel.org/all/[email protected]/
Changes from v1:
- Added the third patch in the series that implements the common table APIs for printing nvme cli command output in tabular format (Daniel Wagner)
- Added the fourth patch in the series which adds the support for printing show-topology in tabular form (Daniel Wagner)
Link to v1: https://lore.kernel.org/all/[email protected]/
Nilay Shroff (4):
nvme: support <device> option in show-topology command
nvme: extend show-topology command to add support for multipath
nvme: add common APIs for printing tabular format output
nvme: add support for printing show-topology in tabular form
nvme-print-binary.c | 1 + nvme-print-json.c | 36 ++++- nvme-print-stdout.c | 312 +++++++++++++++++++++++++++++++++++++++++- nvme-print.c | 9 +- nvme-print.h | 3 + nvme.c | 29 +++- nvme.h | 2 + util/meson.build | 3 +- util/table.c | 320 ++++++++++++++++++++++++++++++++++++++++++++ util/table.h | 149 +++++++++++++++++++++ 10 files changed, 849 insertions(+), 15 deletions(-) create mode 100644 util/table.c create mode 100644 util/table.h
Signature · 1 line
2.51.0
PATCH v3 1/4 nvme: support <device> option in show-topology command
Although the help text for the nvme show-topology command indicates support for a <device> option, this option has no effect in practice — specifying an NVMe device name does not filter the output.
This commit adds proper support for the <device> option, enabling users to filter the topology output based on the specified NVMe device.
--- nvme-print-stdout.c | 9 +++++++++ nvme.c | 17 ++++++++++++++++- 2 files changed, 25 insertions(+), 1 deletion(-)
nvme-print-stdout.c +9 -0
@@ -5683,6 +5683,15 @@ static void stdout_simple_topology(nvme_root_t r,nvme_for_each_host (r, h) {nvme_for_each_subsystem (h, s) { +bool no_ctrl = true; +nvme_ctrl_t c; + +nvme_subsystem_for_each_ctrl (s, c) + no_ctrl = false; + +if (no_ctrl) +continue ; +if (!first)printf ("\n" ); first = false;
nvme.c +16 -1
@@ -10182,6 +10182,8 @@ static int show_topology_cmd(int argc, char **argv, struct command *command, strconst char *ranking ="Ranking order: namespace|ctrl" ;nvme_print_flags_t flags; _cleanup_nvme_root_ nvme_root_t r = NULL; +char *devname =NULL ; +nvme_scan_filter_t filter =NULL ;enum nvme_cli_topo_ranking rank;int err;
@@ -10224,7 +10226,20 @@ static int show_topology_cmd(int argc, char **argv, struct command *command, strreturn -errno; } - err =nvme_scan_topology (r,NULL ,NULL ); +if (optind < argc) + devname =basename (argv[optind++]); + +if (devname) { +int subsys_id, nsid; + +if (sscanf (devname,"nvme%dn%d" , &subsys_id, &nsid) !=2 ) { +nvme_show_error ("Invalid device name %s\n" , devname); +return -EINVAL ; + } + filter = nvme_match_device_filter; + } + + err =nvme_scan_topology (r, filter, (void *)devname);if (err <0 ) {nvme_show_error ("Failed to scan topology: %s" ,nvme_strerror (errno));return err;
Signature · 1 line
2.51.0
PATCH v3 2/4 nvme: extend show-topology command to add support for multipath
In reply to Nilay ShroffThis commit enhances the show-topology command by adding support for NVMe multipath. With this change, users can now list all paths to a namespace from its corresponding head node device. Each NVMe path entry then also includes additional details such as ANA state, NUMA node, and queue depth, improving visibility into multipath configs. This information can be particularly helpful for debugging and analyzing NVMe multipath setups.
To support this functionality, the "--ranking" option of the nvme show-topology command has been extended with a new sub-option:
"multipath".
Since this enhancement is specific to NVMe multipath, the iopolicy configured under each subsystem is now always displayed. Previously, iopolicy was shown only with nvme show-topology verbose output, but it is now included by default to improve usability and provide better context when reviewing multipath configurations via show-topology.
With this update, users can view the multipath topology of a multi controller/port NVMe disk.
Examples:
$ nvme show-topology -r multipath
nvme-subsys2 - NQN=nvmet_subsystem
hostnqn=nqn.2014-08.org.nvmexpress:uuid:12b49f6e-0276-4746-b10c-56815b7e6dc2 iopolicy=numa
_ _ _<head-node>
/ _ _ _ <ana-state>
/ / _ _ _ <numa-node-list>
/ / /
| / /
+- nvme2n1 (ns 1) / / \ | | +- nvme2c2n1 optimized 1,2 nvme2 tcp traddr=127.0.0.2,trsvcid=4460,src_addr=127.0.0.1 live +- nvme2c3n1 optimized 3,4 nvme3 tcp traddr=127.0.0.3,trsvcid=4460,src_addr=127.0.0.1 live
For iopolicy=numa, only NUMA node list is shown (queue depth is hidden).
$ nvme show-topology -r multipath
nvme-subsys2 - NQN=nvmet_subsystem
hostnqn=nqn.2014-08.org.nvmexpress:uuid:12b49f6e-0276-4746-b10c-56815b7e6dc2 iopolicy=queue-depth
_ _ _<head-node>
/ _ _ _ <ana-state>
/ /
/ / _ _ _<queue-depth>
| / /
+- nvme2n1 (ns 1) / / \ | | +- nvme2c2n1 optimized 0 nvme2 tcp traddr=127.0.0.2,trsvcid=4460,src_addr=127.0.0.1 live +- nvme2c3n1 optimized 0 nvme3 tcp traddr=127.0.0.3,trsvcid=4460,src_addr=127.0.0.1 live
For iopolicy=queue-depth, queue depth is shown (NUMA node list is hidden).
$ nvme show-topology -r multipath
nvme-subsys2 - NQN=nvmet_subsystem
hostnqn=nqn.2014-08.org.nvmexpress:uuid:12b49f6e-0276-4746-b10c-56815b7e6dc2
iopolicy=round-robin
_ _ _<head-node>
/ _ _ _ <ana-state>
/ /
/ /
| /
+- nvme2n1 (ns 1) /
\ |
+- nvme2c2n1 optimized nvme2 tcp traddr=127.0.0.2,trsvcid=4460,src_addr=127.0.0.1 live
+- nvme2c3n1 optimized nvme3 tcp traddr=127.0.0.3,trsvcid=4460,src_addr=127.0.0.1 liveFor iopolicy=round-robin, both NUMA node list and queue depth are hidden.
Note:
The annotations above (e.g., <numa-node-list>, <ana-state>, <head-node>, <queue-depth>) are for illustration only and are not part of the actual command output. A more human-friendly tabular format will be introduced in a follow-up patches.
--- nvme-print-binary.c | 1 + nvme-print-json.c | 36 +++++++++++++++---- nvme-print-stdout.c | 85 ++++++++++++++++++++++++++++++++++++++++++--- nvme-print.c | 4 ++- nvme-print.h | 1 + nvme.c | 4 ++- nvme.h | 1 + 7 files changed, 120 insertions(+), 12 deletions(-)
nvme-print-binary.c +1 -0
@@ -428,6 +428,7 @@ static struct print_ops binary_print_ops = { .print_nvme_subsystem_list = NULL, .topology_ctrl = NULL, .topology_namespace = NULL, + .topology_multipath = NULL,/* status and error messages */ .connect_msg = NULL,
nvme-print-json.c +30 -6
@@ -4743,6 +4743,7 @@ static unsigned int json_subsystem_topology_multipath(nvme_subsystem_t s,nvme_ns_t n;nvme_path_t p;unsigned int i =0 ; +const char *iopolicy =nvme_subsystem_get_iopolicy (s);nvme_subsystem_for_each_ns (s, n) {struct json_object *ns_attrs;
@@ -4750,19 +4751,40 @@ static unsigned int json_subsystem_topology_multipath(nvme_subsystem_t s, ns_attrs =json_create_object ();obj_add_int (ns_attrs,"NSID" ,nvme_ns_get_nsid (n)); +obj_add_str (ns_attrs,"Name" ,nvme_ns_get_name (n)); paths =json_create_array ();nvme_namespace_for_each_path (n, p) {struct json_object *path_attrs; - -nvme_ctrl_t c =nvme_path_get_ctrl (p); +struct json_object *ctrls, *ctrl_attrs; +nvme_ctrl_t c; path_attrs =json_create_object (); -obj_add_str (path_attrs,"Name" ,nvme_ctrl_get_name (c)); -obj_add_str (path_attrs,"Transport" ,nvme_ctrl_get_transport (c)); -obj_add_str (path_attrs,"Address" ,nvme_ctrl_get_address (c)); -obj_add_str (path_attrs,"State" ,nvme_ctrl_get_state (c)); +obj_add_str (path_attrs,"Path" ,nvme_path_get_name (p));obj_add_str (path_attrs,"ANAState" ,nvme_path_get_ana_state (p)); + +/* +* For iopolicy numa exclude "Qdepth", for iopolicy +* queue-depth exclude "NUMANodes" and for iopolicy +* round-robin exclude both "Qdepth" and "NUMANodes". +*/ +if (!strcmp (iopolicy,"numa" )) +obj_add_str (path_attrs,"NUMANodes" , +nvme_path_get_numa_nodes (p)); +else if (!strcmp (iopolicy,"queue-depth" )) +obj_add_int (path_attrs,"Qdepth" , +nvme_path_get_queue_depth (p)); + + c =nvme_path_get_ctrl (p); + ctrls =json_create_array (); + ctrl_attrs =json_create_object (); +obj_add_str (ctrl_attrs,"Name" ,nvme_ctrl_get_name (c)); +obj_add_str (ctrl_attrs,"Transport" ,nvme_ctrl_get_transport (c)); +obj_add_str (ctrl_attrs,"Address" ,nvme_ctrl_get_address (c)); +obj_add_str (ctrl_attrs,"State" ,nvme_ctrl_get_state (c)); +array_add_obj (ctrls, ctrl_attrs); +obj_add_array (path_attrs,"Controller" , ctrls); +array_add_obj (paths, path_attrs); }obj_add_array (ns_attrs,"Paths" , paths);
@@ -4787,6 +4809,7 @@ static void json_print_nvme_subsystem_topology(nvme_subsystem_t s, ns_attrs =json_create_object ();obj_add_int (ns_attrs,"NSID" ,nvme_ns_get_nsid (n)); +obj_add_str (ns_attrs,"Name" ,nvme_ns_get_name (n)); ctrl =json_create_array (); ctrl_attrs =json_create_object ();
@@ -5546,6 +5569,7 @@ static struct print_ops json_print_ops = { .print_nvme_subsystem_list = json_print_nvme_subsystem_list, .topology_ctrl = json_simple_topology, .topology_namespace = json_simple_topology, + .topology_multipath = json_simple_topology,/* status and error messages */ .connect_msg = json_connect_msg,
nvme-print-stdout.c +81 -4
139 lines not on this page · raw message
nvme-print.c +3 -1
@@ -1551,8 +1551,10 @@ void nvme_show_topology(nvme_root_t r, {if (ranking ==NVME_CLI_TOPO_NAMESPACE )nvme_print (topology_namespace, flags, r); -else +else if (ranking ==NVME_CLI_TOPO_CTRL )nvme_print (topology_ctrl, flags, r); +else +nvme_print (topology_multipath, flags, r); }void nvme_show_message (bool error,const char *msg, ...)
nvme-print.h +1 -0
@@ -106,6 +106,7 @@ struct print_ops {void (*print_nvme_subsystem_list)(nvme_root_t r,bool show_ana);void (*topology_ctrl)(nvme_root_t r);void (*topology_namespace)(nvme_root_t r); +void (*topology_multipath)(nvme_root_t r);/* status and error messages */ void (*connect_msg)(nvme_ctrl_t c);
nvme.c +3 -1
@@ -10179,7 +10179,7 @@ static int tls_key(int argc, char **argv, struct command *command, struct pluginstatic int show_topology_cmd (int argc,char **argv,struct command *command,struct plugin *plugin) {const char *desc ="Show the topology\n" ; -const char *ranking ="Ranking order: namespace|ctrl" ; +const char *ranking ="Ranking order: namespace|ctrl|multipath" ;nvme_print_flags_t flags; _cleanup_nvme_root_ nvme_root_t r = NULL;char *devname =NULL ;
@@ -10215,6 +10215,8 @@ static int show_topology_cmd(int argc, char **argv, struct command *command, str rank =NVME_CLI_TOPO_NAMESPACE ; }else if (!strcmp (cfg.ranking,"ctrl" )) { rank =NVME_CLI_TOPO_CTRL ; + }else if (!strcmp (cfg.ranking,"multipath" )) { + rank =NVME_CLI_TOPO_MULTIPATH ; }else {nvme_show_error ("Invalid ranking argument: %s" , cfg.ranking);return -EINVAL ;
nvme.h +1 -0
@@ -46,6 +46,7 @@ typedef uint32_t nvme_print_flags_t;enum nvme_cli_topo_ranking {NVME_CLI_TOPO_NAMESPACE ,NVME_CLI_TOPO_CTRL , +NVME_CLI_TOPO_MULTIPATH , };#define SYS_NVME "/sys/class/nvme"
Signature · 1 line
2.51.0
Re: [PATCHv3 2/4] nvme: extend show-topology command to add support for multipath
Quoted 84 lines · Nilay Shroff
This commit enhances the show-topology command by adding support for NVMe multipath. With this change, users can now list all paths to a namespace from its corresponding head node device. Each NVMe path entry then also includes additional details such as ANA state, NUMA node, and queue depth, improving visibility into multipath configs. This information can be particularly helpful for debugging and analyzing NVMe multipath setups.
To support this functionality, the "--ranking" option of the nvme show-topology command has been extended with a new sub-option:
"multipath".Since this enhancement is specific to NVMe multipath, the iopolicy configured under each subsystem is now always displayed. Previously, iopolicy was shown only with nvme show-topology verbose output, but it is now included by default to improve usability and provide better context when reviewing multipath configurations via show-topology.
With this update, users can view the multipath topology of a multi controller/port NVMe disk.
Examples:
$ nvme show-topology -r multipathnvme-subsys2 - NQN=nvmet_subsystem
hostnqn=nqn.2014-08.org.nvmexpress:uuid:12b49f6e-0276-4746-b10c-56815b7e6dc2 iopolicy=numa_ _ _<head-node>
/ _ _ _ <ana-state>
/ / _ _ _ <numa-node-list>
/ / /
| / /+- nvme2n1 (ns 1) / / \ | | +- nvme2c2n1 optimized 1,2 nvme2 tcp traddr=127.0.0.2,trsvcid=4460,src_addr=127.0.0.1 live +- nvme2c3n1 optimized 3,4 nvme3 tcp traddr=127.0.0.3,trsvcid=4460,src_addr=127.0.0.1 liveFor iopolicy=numa, only NUMA node list is shown (queue depth is hidden).
$ nvme show-topology -r multipathnvme-subsys2 - NQN=nvmet_subsystem
hostnqn=nqn.2014-08.org.nvmexpress:uuid:12b49f6e-0276-4746-b10c-56815b7e6dc2 iopolicy=queue-depth_ _ _<head-node>
/ _ _ _ <ana-state>
/ /
/ / _ _ _<queue-depth>
| / /+- nvme2n1 (ns 1) / / \ | | +- nvme2c2n1 optimized 0 nvme2 tcp traddr=127.0.0.2,trsvcid=4460,src_addr=127.0.0.1 live +- nvme2c3n1 optimized 0 nvme3 tcp traddr=127.0.0.3,trsvcid=4460,src_addr=127.0.0.1 liveFor iopolicy=queue-depth, queue depth is shown (NUMA node list is hidden).
$ nvme show-topology -r multipathnvme-subsys2 - NQN=nvmet_subsystem
hostnqn=nqn.2014-08.org.nvmexpress:uuid:12b49f6e-0276-4746-b10c-56815b7e6dc2 iopolicy=round-robin _ _ _<head-node> / _ _ _ <ana-state> / / / / | / +- nvme2n1 (ns 1) / \ | +- nvme2c2n1 optimized nvme2 tcp traddr=127.0.0.2,trsvcid=4460,src_addr=127.0.0.1 live +- nvme2c3n1 optimized nvme3 tcp traddr=127.0.0.3,trsvcid=4460,src_addr=127.0.0.1 liveFor iopolicy=round-robin, both NUMA node list and queue depth are hidden.
Note:
The annotations above (e.g., <numa-node-list>, <ana-state>, <head-node>, <queue-depth>) are for illustration only and are not part of the actual command output. A more human-friendly tabular format will be introduced in a follow-up patches.Signed-off-by Nilay Shroff <[email protected]>
Quoted 10 lines
--- nvme-print-binary.c | 1 + nvme-print-json.c | 36 +++++++++++++++---- nvme-print-stdout.c | 85 ++++++++++++++++++++++++++++++++++++++++++--- nvme-print.c | 4 ++- nvme-print.h | 1 + nvme.c | 4 ++- nvme.h | 1 + 7 files changed, 120 insertions(+), 12 deletions(-)
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
PATCH v3 3/4 nvme: add common APIs for printing tabular format output
In reply to Nilay ShroffSome nvme-cli commands, such as nvme list and nvme list -v, support output in tabular format. Currently, the table output is not well aligned because column widths are fixed at print time, regardless of the length of the data in each column. This often results in uneven and hard-to-read output.For any new CLI command that requires tabular output, developers must manually specify the column width and row value width, which is both error-prone and inconsistent.
This patch introduces a set of common table APIs that:
- Automatically calculate column widths based on the content
- Maintain proper alignment regardless of value length
- Simplify adding tabular output support to new and existing commands
The new APIs are:
1. table_init() — Allocate a table instance.
2. table_add_columns() — Add column definitions (struct table_column), including name and alignment (LEFT, RIGHT, CENTERED).
3. table_add_columns_filter() - Same as table_add_columns() but also provide a filter function callback which could be then used by the caller for filtering any column.
3. table_get_row_id() — Reserve a row index for inserting values.
4. table_add_row() — Add a row to the table.
5. table_print() — Print the table with auto-calculated widths.
6. table_free() — Free resources allocated for the table.
For adding values, the following setter APIs are provided, each supporting alignment types (LEFT, RIGHT, or CENTERED):
- table_set_value_str()
- table_set_value_int()
- table_set_value_unsigned()
- table_set_value_long()
- table_set_value_unsigned_long()
Usage steps:
1. Call table_init() to create a table handle.
2. Define an array of struct table_column specifying column names and alignment, then call table_add_columns() or if you want to filter column then table_add_columns_filter().
3. Obtain a row ID using table_get_row_id() and set values using the appropriate setter table APIs : table_set_value_*() function.
4. Add the completed row using table_add_row().
5. Repeat steps 3–4 for each additional row.
5. Call table_print() to display the table.
6. Call table_free() to release table resources.
With these APIs, developers no longer need to pre-calculate column or row widths. The output is consistently aligned and easy to read.
--- util/meson.build | 3 +- util/table.c | 320 +++++++++++++++++++++++++++++++++++++++++++++++ util/table.h | 149 ++++++++++++++++++++++ 3 files changed, 471 insertions(+), 1 deletion(-) create mode 100644 util/table.c create mode 100644 util/table.h
util/meson.build +2 -1
@@ -8,7 +8,8 @@ sources += [ 'util/sighdl.c', 'util/suffix.c', 'util/types.c', - 'util/utils.c' + 'util/utils.c', + 'util/table.c' ] if json_c_dep.found()
util/table.c new file +320 -0
320 lines not on this page · raw message
util/table.h new file +149 -0
149 lines not on this page · raw message
Signature · 1 line
2.51.0
PATCH v3 4/4 nvme: add support for printing show-topology in tabular form
In reply to Nilay ShroffThe nvme CLI command show-topology currently prints the topology output in a tree format. However, in some cases it is more convenient and easier to read or interpret the output when displayed in a tabular form.
This patch adds support for printing the show-topology output in a tabular format. To achieve this, the --output-format option, which previously supported only normal and json formats, is extended to include a new tabular format.
With this change, the user can now choose to print the topology in any of the following formats: normal, json, or tabular. The new tabular output leverages the recently introduced table APIs to produce well- aligned, easy-to-read output.
--- nvme-print-stdout.c | 218 ++++++++++++++++++++++++++++++++++++++++++++ nvme-print.c | 5 + nvme-print.h | 2 + nvme.c | 8 +- nvme.h | 1 + 5 files changed, 233 insertions(+), 1 deletion(-)
nvme-print-stdout.c +218 -0
@@ -20,6 +20,7 @@#include "nvme-models.h" #include "util/suffix.h" #include "util/types.h" +#include "util/table.h" #include "logging.h" #include "common.h"
@@ -5589,6 +5590,127 @@ static void stdout_list_items(nvme_root_t r)stdout_simple_list (r); } +static bool subsystem_iopolicy_filter (const char *name,void *arg) +{ +nvme_subsystem_t s = arg; +const char *iopolicy =nvme_subsystem_get_iopolicy (s); + +if (!strcmp (iopolicy,"queue-depth" )) { +/* exclude "Nodes" for iopolicy queue-depth */ +if (!strcmp (name,"Nodes" )) +return false; + }else if (!strcmp (iopolicy,"numa" )) { +/* exclude "Qdepth" for iopolicy numa */ +if (!strcmp (name,"Qdepth" )) +return false; + }else {/* round-robin */ +/* exclude "Nodes" and "Qdepth" for iopolicy round-robin */ +if (!strcmp (name,"Nodes" ) || !strcmp (name,"Qdepth" )) +return false; + } + +return true; +} + +static void stdout_tabular_subsystem_topology_multipath (nvme_subsystem_t s) +{ +nvme_ns_t n; +nvme_path_t p; +nvme_ctrl_t c; +int row, col; +bool first; +struct table *t; +const char *iopolicy =nvme_subsystem_get_iopolicy (s); +struct table_column columns[] = { + {"NSHead" ,LEFT ,0 }, + {"NSID" ,LEFT ,0 }, + {"NSPath" ,LEFT ,0 }, + {"ANAState" ,LEFT ,0 }, + {"Nodes" ,LEFT ,0 }, + {"Qdepth" ,LEFT ,0 }, + {"Controller" ,LEFT ,0 }, + {"TrType" ,LEFT ,0 }, + {"Address" ,LEFT ,0 }, + {"State" ,LEFT ,0 }, + }; + + t =table_init (); +if (!t) { +printf ("Failed to init table\n" ); +return ; + } + +if (table_add_columns_filter (t, columns,ARRAY_SIZE (columns), + subsystem_iopolicy_filter, (void *)s) <0 ) { +printf ("Failed to add columns\n" ); + goto free_tbl; + } + +nvme_subsystem_for_each_ns (s, n) { + first = true; +nvme_namespace_for_each_path (n, p) { + c =nvme_path_get_ctrl (p); + + row =table_get_row_id (t); +if (row <0 ) { +printf ("Failed to add row\n" ); + goto free_tbl; + } +/* For the first row we print actual NSHead name, +* however, for the subsequent rows we print "arrow" +* ("-->") symbol for NSHead. This "arrow" style makes +* it visually obvious that susequenet entries (if +* present) are a path under the first NSHead. +*/ + col =-1 ; +/* col 0: NSHead */ +if (first) { +table_set_value_str (t, ++col, row, +nvme_ns_get_name (n),LEFT ); + first = false; + }else +table_set_value_str (t, ++col, row, +"-->" ,CENTERED ); +/* col 1: NSID */ +table_set_value_int (t, ++col, row, +nvme_ns_get_nsid (n),CENTERED ); +/* col 2: NSPath */ +table_set_value_str (t, ++col, row, +nvme_path_get_name (p),LEFT ); +/* col 3: ANAState */ +table_set_value_str (t, ++col, row, +nvme_path_get_ana_state (p),LEFT ); + +if (!strcmp (iopolicy,"numa" )) +/* col 4: Nodes */ +table_set_value_str (t, ++col, row, +nvme_path_get_numa_nodes (p),CENTERED ); +else if (!strcmp (iopolicy,"queue-depth" )) +/* col 4 : Qdepth */ +table_set_value_int (t, ++col, row, +nvme_path_get_queue_depth (p),CENTERED ); + +/* col 5: Controller */ +table_set_value_str (t, ++col, row, +nvme_ctrl_get_name (c),LEFT ); +/* col 6: TrType */ +table_set_value_str (t, ++col, row, +nvme_ctrl_get_transport (c),LEFT ); +/* col 7: Address */ +table_set_value_str (t, ++col, row, +nvme_ctrl_get_address (c),LEFT ); +/* col 8: State */ +table_set_value_str (t, ++col, row, +nvme_ctrl_get_state (c),LEFT ); + +table_add_row (t, row); + } + } +table_print (t); +free_tbl: +table_free (t); +} +static void stdout_subsystem_topology_multipath (nvme_subsystem_t s,enum nvme_cli_topo_ranking ranking) {
@@ -5693,6 +5815,69 @@ static void stdout_subsystem_topology_multipath(nvme_subsystem_t s, } } +static void stdout_tabular_subsystem_topology (nvme_subsystem_t s) +{ +nvme_ctrl_t c; +nvme_ns_t n; +int row; +struct table *t; +struct table_column columns[] = { + {"Namespace" ,LEFT ,0 }, + {"NSID" ,LEFT ,0 }, + {"Controller" ,LEFT ,0 }, + {"Trtype" ,LEFT ,0 }, + {"Address" ,LEFT ,0 }, + {"State" ,LEFT ,0 }, + }; + + t =table_init (); +if (!t) { +printf ("Failed to init table\n" ); +return ; + } + +if (table_add_columns (t, columns,ARRAY_SIZE (columns)) <0 ) { +printf ("Failed to add columns\n" ); + goto free_tbl; + } + +nvme_subsystem_for_each_ctrl (s, c) { +nvme_ctrl_for_each_ns (c, n) { + c =nvme_ns_get_ctrl (n); + + row =table_get_row_id (t); +if (row <0 ) { +printf ("Failed to add row\n" ); + goto free_tbl; + } + +/* col 0: Namespace */ +table_set_value_str (t,0 , row, +nvme_ns_get_name (n),LEFT ); +/* col 1: NSID */ +table_set_value_int (t,1 , row, +nvme_ns_get_nsid (n),CENTERED ); +/* col 2: Controller */ +table_set_value_str (t,2 , row, +nvme_ctrl_get_name (c),LEFT ); +/* col 3: Trtype */ +table_set_value_str (t,3 , row, +nvme_ctrl_get_transport (c),LEFT ); +/* col 4: Address */ +table_set_value_str (t,4 , row, +nvme_ctrl_get_address (c),LEFT ); +/* col 5: State */ +table_set_value_str (t,5 , row, +nvme_ctrl_get_state (c),LEFT ); + +table_add_row (t, row); + } + } +table_print (t); +free_tbl: +table_free (t); +} +static void stdout_subsystem_topology (nvme_subsystem_t s,enum nvme_cli_topo_ranking ranking) {
@@ -5745,6 +5930,38 @@ static void stdout_subsystem_topology(nvme_subsystem_t s, } } +static void stdout_topology_tabular (nvme_root_t r) +{ +nvme_host_t h; +nvme_subsystem_t s; +bool first = true; + +nvme_for_each_host (r, h) { +nvme_for_each_subsystem (h, s) { +bool no_ctrl = true; +nvme_ctrl_t c; + +nvme_subsystem_for_each_ctrl (s, c) + no_ctrl = false; + +if (no_ctrl) +continue ; + +if (!first) +printf ("\n" ); + first = false; + +stdout_subsys_config (s); +printf ("\n" ); + +if (nvme_is_multipath (s)) +stdout_tabular_subsystem_topology_multipath (s); +else +stdout_tabular_subsystem_topology (s); + } + } +} +static void stdout_simple_topology (nvme_root_t r,enum nvme_cli_topo_ranking ranking) {
@@ -6363,6 +6580,7 @@ static struct print_ops stdout_print_ops = { .topology_ctrl = stdout_topology_ctrl, .topology_namespace = stdout_topology_namespace, .topology_multipath = stdout_topology_multipath, + .topology_tabular = stdout_topology_tabular,/* status and error messages */ .connect_msg = stdout_connect_msg,
nvme-print.c +5 -0
@@ -1557,6 +1557,11 @@ void nvme_show_topology(nvme_root_t r,nvme_print (topology_multipath, flags, r); } +void nvme_show_topology_tabular (nvme_root_t r,nvme_print_flags_t flags) +{ +nvme_print (topology_tabular, flags, r); +} +void nvme_show_message (bool error,const char *msg, ...) {struct print_ops *ops =nvme_print_ops (NORMAL );
nvme-print.h +2 -0
@@ -107,6 +107,7 @@ struct print_ops {void (*topology_ctrl)(nvme_root_t r);void (*topology_namespace)(nvme_root_t r);void (*topology_multipath)(nvme_root_t r); +void (*topology_tabular)(nvme_root_t r);/* status and error messages */ void (*connect_msg)(nvme_ctrl_t c);
@@ -251,6 +252,7 @@ void nvme_show_list_ns(struct nvme_ns_list *ns_list,void nvme_show_topology (nvme_root_t t,enum nvme_cli_topo_ranking ranking,nvme_print_flags_t flags); +void nvme_show_topology_tabular (nvme_root_t t,nvme_print_flags_t flags);void nvme_feature_show (enum nvme_features_id fid,int sel,unsigned int result);void nvme_feature_show_fields (enum nvme_features_id fid,unsigned int result,unsigned char *buf);
nvme.c +7 -1
@@ -545,6 +545,8 @@ int validate_output_format(const char *format, nvme_print_flags_t *flags)#endif /* CONFIG_JSONC */ else if (!strcmp(format, "binary")) f =BINARY ; + else if (!strcmp(format, "tabular")) + f =TABULAR ; else return -EINVAL;
@@ -10179,6 +10181,7 @@ static int tls_key(int argc, char **argv, struct command *command, struct pluginstatic int show_topology_cmd (int argc,char **argv,struct command *command,struct plugin *plugin) {const char *desc ="Show the topology\n" ; +const char *output_format ="Output format: normal|json|binary|tabular" ;const char *ranking ="Ranking order: namespace|ctrl|multipath" ;nvme_print_flags_t flags; _cleanup_nvme_root_ nvme_root_t r = NULL;
@@ -10247,7 +10250,10 @@ static int show_topology_cmd(int argc, char **argv, struct command *command, strreturn err; } -nvme_show_topology (r, rank, flags); +if (flags &TABULAR ) +nvme_show_topology_tabular (r, flags); +else +nvme_show_topology (r, rank, flags);return err; }
nvme.h +1 -0
@@ -39,6 +39,7 @@ enum nvme_print_flags {JSON =1 <<1 ,/* display in json format */ VS =1 <<2 ,/* hex dump vendor specific data areas */ BINARY =1 <<3 ,/* binary dump raw bytes */ + TABULAR = 1 << 4, /* prints aligned columns for easy reading */ };typedef uint32_t nvme_print_flags_t ;
Signature · 1 line
2.51.0
Hi Daniel,
A gentle ping on this patchset. Please let me know, in case, if you've any further query/comment/suggestion for this patchset.
Thanks,
--Nilay
Quoted 73 lines · Nilay Shroff
Hi,
There has been recent work to improve the visibility of NVMe multipath configurations. The native NVMe multipath kernel driver supports three I/O path selection policies: numa, round-robin, and queue-depth. However, until now, users have had no insight into which path is selected by the multipath logic for forwarding I/O.
To address this, an RFC was proposed [1] and has since been merged into the Linux kernel as of v6.15. Following that, libnvme was extended to discover multiple paths to a shared NVMe namespace [2]. This patchset builds on those efforts to improve userspace observability of multipath configurations via nvme-cli.
This patchset contains four patches:
The first patch in the series fixes a <device> filter option using which we could be able to filter the output of nvme show-topology command.The second patch in the series enhances the nvme show-topology command to display multipath configuration, including all discovered paths to a namespace, along with details such as ANA state, NUMA nodes, and queue depth. The show-topology "--ranking" option is extended with a new sub-
option: multipath.The third patch in the series adds common table APIs for printing nvme cli command output in tabular format. With these APIs, developers no longer need to pre-calculate column or row widths. The output is consistently aligned and easy to read.
The fourth patch adds support for printing show-topology in tabular form leveraging the introduced table APIs to produce well-aligned, easy-to-read output.
As usual, any feedback/sugegstion is most welcome!
Thanks!
Changes from v2:
- For iopolicy=numa, print NUMA nodes and exclude queue-depth. Similarly, for iopolicy=queue-depth, print queue-depth but exclude NUMA nodes. And for iopolicy=round-robin, exclude printing
both NUMA nodes and queue-depth. (Hannes Reinecke, Daniel
Wagner)
Link to v2: https://lore.kernel.org/all/[email protected]/Changes from v1:
- Added the third patch in the series that implements the common table APIs for printing nvme cli command output in tabular format (Daniel Wagner)
- Added the fourth patch in the series which adds the support for printing show-topology in tabular form (Daniel Wagner)
Link to v1: https://lore.kernel.org/all/[email protected]/Nilay Shroff (4):
nvme: support <device> option in show-topology command
nvme: extend show-topology command to add support for multipath
nvme: add common APIs for printing tabular format output
nvme: add support for printing show-topology in tabular formnvme-print-binary.c | 1 + nvme-print-json.c | 36 ++++- nvme-print-stdout.c | 312 +++++++++++++++++++++++++++++++++++++++++- nvme-print.c | 9 +- nvme-print.h | 3 + nvme.c | 29 +++- nvme.h | 2 + util/meson.build | 3 +- util/table.c | 320 ++++++++++++++++++++++++++++++++++++++++++++ util/table.h | 149 +++++++++++++++++++++ 10 files changed, 849 insertions(+), 15 deletions(-) create mode 100644 util/table.c create mode 100644 util/table.h
Hi Nilay,
Quoted 2 lines · Nilay Shroff
A gentle ping on this patchset. Please let me know, in case, if you've any further query/comment/suggestion for this patchset.
All good, I've applied the series. FTR, the tabular output looks nice:
nvme-subsys29 - NQN=nqn.1992-08.com.netapp:sn.44ede5aaa52511e8abb200a09890f5bf:subsystem.nvme-svm-dolin-qla-subsystem
hostnqn=nqn.2014-08.org.nvmexpress:uuid:31333937-3136-584d-5135-323430365637
iopolicy=queue-depthNSHead NSID NSPath ANAState Qdepth Controller TrType Address State
-------- ---- ----------- ------------- ------ ---------- ------ ---------------------------------------------------------------------------------------------------------- -----
nvme29n1 2 nvme29c11n1 optimized 0 nvme11 fc traddr=nn-0x200200a09890f5bf:pn-0x203600a09890f5bf,host_traddr=nn-0x20000024ff7fa448:pn-0x21000024ff7fa448 live --> 2 nvme29c29n1 optimized 0 nvme29 fc traddr=nn-0x200200a09890f5bf:pn-0x203800a09890f5bf,host_traddr=nn-0x20000024ff7fa448:pn-0x21000024ff7fa448 live --> 2 nvme29c30n1 non-optimized 0 nvme30 fc traddr=nn-0x200200a09890f5bf:pn-0x203700a09890f5bf,host_traddr=nn-0x20000024ff7fa448:pn-0x21000024ff7fa448 live --> 2 nvme29c31n1 non-optimized 0 nvme31 fc traddr=nn-0x200200a09890f5bf:pn-0x203900a09890f5bf,host_traddr=nn-0x20000024ff7fa448:pn-0x21000024ff7fa448 live --> 2 nvme29c32n1 non-optimized 0 nvme32 fc traddr=nn-0x200200a09890f5bf:pn-0x203700a09890f5bf,host_traddr=nn-0x20000024ff7fa449:pn-0x21000024ff7fa449 live --> 2 nvme29c33n1 optimized 0 nvme33 fc traddr=nn-0x200200a09890f5bf:pn-0x203800a09890f5bf,host_traddr=nn-0x20000024ff7fa449:pn-0x21000024ff7fa449 live --> 2 nvme29c34n1 non-optimized 0 nvme34 fc traddr=nn-0x200200a09890f5bf:pn-0x203900a09890f5bf,host_traddr=nn-0x20000024ff7fa449:pn-0x21000024ff7fa449 live --> 2 nvme29c35n1 optimized 0 nvme35 fc traddr=nn-0x200200a09890f5bf:pn-0x203600a09890f5bf,host_traddr=nn-0x20000024ff7fa449:pn-0x21000024ff7fa449 live nvme29n2 1 nvme29c11n2 optimized 0 nvme11 fc traddr=nn-0x200200a09890f5bf:pn-0x203600a09890f5bf,host_traddr=nn-0x20000024ff7fa448:pn-0x21000024ff7fa448 live --> 1 nvme29c29n2 optimized 0 nvme29 fc traddr=nn-0x200200a09890f5bf:pn-0x203800a09890f5bf,host_traddr=nn-0x20000024ff7fa448:pn-0x21000024ff7fa448 live --> 1 nvme29c30n2 non-optimized 0 nvme30 fc traddr=nn-0x200200a09890f5bf:pn-0x203700a09890f5bf,host_traddr=nn-0x20000024ff7fa448:pn-0x21000024ff7fa448 live --> 1 nvme29c31n2 non-optimized 0 nvme31 fc traddr=nn-0x200200a09890f5bf:pn-0x203900a09890f5bf,host_traddr=nn-0x20000024ff7fa448:pn-0x21000024ff7fa448 live --> 1 nvme29c32n2 non-optimized 0 nvme32 fc traddr=nn-0x200200a09890f5bf:pn-0x203700a09890f5bf,host_traddr=nn-0x20000024ff7fa449:pn-0x21000024ff7fa449 live --> 1 nvme29c33n2 optimized 0 nvme33 fc traddr=nn-0x200200a09890f5bf:pn-0x203800a09890f5bf,host_traddr=nn-0x20000024ff7fa449:pn-0x21000024ff7fa449 live --> 1 nvme29c34n2 non-optimized 0 nvme34 fc traddr=nn-0x200200a09890f5bf:pn-0x203900a09890f5bf,host_traddr=nn-0x20000024ff7fa449:pn-0x21000024ff7fa449 live --> 1 nvme29c35n2 optimized 0 nvme35 fc traddr=nn-0x200200a09890f5bf:pn-0x203600a09890f5bf,host_traddr=nn-0x20000024ff7fa449:pn-0x21000024ff7fa449 live