libnvme: add support for retrieving additional NVMe stat
v3 cover libnvme: add support for retrieving additional NVMe stat
Hi,
Certain NVMe attributes and statistics can change dynamically at runtime. However, the current libnvme implementation caches attribute values while scanning and building the topology. Subsequent libnvme API calls return these cached values.
While this approach works well for one-shot nvme-cli commands, it is not suitable for use cases that require upto date information. In particular, real-time monitoring tools such as nvme-top require access to the latest (non-cached) values of such attributes.
To address this, this patchset introduces support for retrieving non-cached values for selected attributes whose values may change dynamically. The first three patches disable auto-generated accessors for ana_state, numa_nodes, and iopolicy using "!accessors:none", and provide custom implementations that always return the latest values.
In addition, libnvme lacks support for retrieving gendisk statistics, which provide useful insight into disk activity (e.g., number of I/Os processed, time spent servicing I/O, and in-flight I/O counts). This patchset adds support for retrieving gendisk I/O statistics for both per-path and namespace gendisks.
Finally, this series adds support for retrieving diagnostic counters at different levels, including per-path, namespace, nshead, and controller. These counters should improve visibility into NVMe native multipath behavior.
All of the above statistics and counters can be leveraged by tools such as nvme-top to provide real-time monitoring and analysis.
As usual, feedback and suggestions are welcome!
Changes from v2:
- Rebase the code against the latest nvme-cli (Daniel Wagner)
(base-commit: 6f3cfb15a515ddc62aef94c1d8b4f3be5ad33ad0)
Link to v2: https://lore.kernel.org/all/[email protected]/
Changes from v1:
- Avoid exposing internal state of libnvme to its users
while exporting the nvme stat; add a switch while updating
nvme stat which determines whether getters shall return the
nvme diffstat or absolute stat (Daniel Wagner)
- If libnvme API fails then return appropriate error code
instead of -1 (Daniel Wagner)
Link to v1: https://lore.kernel.org/all/[email protected]/
Nilay Shroff (9):
libnvme: annotate libnvme_path::ana_state with !accessors:none
libnvme: annotate libnvme_path::numa_nodes with !accessors:none
libnvme: annotate libnvme_subsystem::iopolicy with !accessors:none
libnvme: add support for retrieving per-path gendisk I/O statistics
libnvme: add support for retrieving namespace gendisk I/O statistics
libnvme: add support for per-path diagnostic counters
libnvme: add support for namespace diagnostic counters
libnvme: add support for nshead diagnostic counters
libnvme: add support for ctrl diagnostic counters
libnvme/src/accessors.ld | 6 - libnvme/src/libnvme.ld | 35 +++ libnvme/src/nvme/accessors.c | 40 --- libnvme/src/nvme/accessors.h | 49 --- libnvme/src/nvme/private.h | 54 +++- libnvme/src/nvme/tree.c | 568 +++++++++++++++++++++++++++++++++++ libnvme/src/nvme/tree.h | 296 ++++++++++++++++++ 7 files changed, 949 insertions(+), 99 deletions(-)
Signature · 1 line
2.53.0
PATCH v3 1/9 libnvme: annotate libnvme_path::ana_state with !accessors:none
The default accessor generation creates getters that return cached attribute values. However, for libnvme_path::ana_state, a real-time (non-cached) value is required.
This is particularly useful for tools such as nvme-top, which rely on up-to-date information for displaying a real-time dashboard.
Annotate libnvme_path::ana_state with "!accessors:none" to disable the auto-generated accessor, and provide a custom implementation of libnvme_path_get_ana_state() that returns the current value.
--- libnvme/src/accessors.ld | 2 -- libnvme/src/libnvme.ld | 1 + libnvme/src/nvme/accessors.c | 13 ------------- libnvme/src/nvme/accessors.h | 15 --------------- libnvme/src/nvme/private.h | 2 +- libnvme/src/nvme/tree.c | 15 +++++++++++++++ libnvme/src/nvme/tree.h | 8 ++++++++ 7 files changed, 25 insertions(+), 31 deletions(-)
libnvme/src/accessors.ld +0 -2
@@ -166,12 +166,10 @@ LIBNVME_ACCESSORS_3 { libnvme_ns_set_name; libnvme_ns_set_nsid; libnvme_ns_set_sysfs_dir; - libnvme_path_get_ana_state; libnvme_path_get_grpid; libnvme_path_get_name; libnvme_path_get_numa_nodes; libnvme_path_get_sysfs_dir; - libnvme_path_set_ana_state; libnvme_path_set_grpid; libnvme_path_set_name; libnvme_path_set_numa_nodes;
libnvme/src/libnvme.ld +1 -0
@@ -139,6 +139,7 @@ LIBNVME_3 { libnvme_path_get_ctrl; libnvme_path_get_ns; libnvme_path_get_queue_depth; + libnvme_path_get_ana_state; libnvme_random_uuid; libnvme_read_config; libnvme_read_hostid;
libnvme/src/nvme/accessors.c +0 -13
@@ -291,19 +291,6 @@ __public const char *libnvme_path_get_sysfs_dir(const struct libnvme_path *p)return p->sysfs_dir; } -__public void libnvme_path_set_ana_state( -struct libnvme_path *p, -const char *ana_state) -{ -free (p->ana_state); - p->ana_state = ana_state ?strdup (ana_state) :NULL ; -} - -__public const char *libnvme_path_get_ana_state(const struct libnvme_path *p) -{ -return p->ana_state; -} - __public void libnvme_path_set_numa_nodes(struct libnvme_path *p,const char *numa_nodes)
libnvme/src/nvme/accessors.h +0 -15
@@ -391,21 +391,6 @@ void libnvme_path_set_sysfs_dir(struct libnvme_path *p, const char *sysfs_dir);*/ const char *libnvme_path_get_sysfs_dir (const struct libnvme_path *p); -/** -* libnvme_path_set_ana_state() - Set ana_state. -* @p: The &struct libnvme_path instance to update. -* @ana_state: New string; a copy is stored. Pass NULL to clear. -*/ -void libnvme_path_set_ana_state (struct libnvme_path *p,const char *ana_state); - -/** -* libnvme_path_get_ana_state() - Get ana_state. -* @p: The &struct libnvme_path instance to query. -* -* Return: The value of the ana_state field, or NULL if not set. -*/ -const char *libnvme_path_get_ana_state (const struct libnvme_path *p); -/** * libnvme_path_set_numa_nodes() - Set numa_nodes. * @p: The &struct libnvme_path instance to update.
libnvme/src/nvme/private.h +1 -1
@@ -176,8 +176,8 @@ struct libnvme_path { // !generate-accessorschar *name;char *sysfs_dir; -char *ana_state;char *numa_nodes; +char *ana_state;//!accessors:none int grpid;int queue_depth;// !accessors:none };
libnvme/src/nvme/tree.c +15 -0
@@ -839,6 +839,21 @@ __public int libnvme_path_get_queue_depth(libnvme_path_t p)return p->queue_depth; } +__public char *libnvme_path_get_ana_state(libnvme_path_t p) +{ + __cleanup_free char *ana_state = NULL; + + ana_state =libnvme_get_path_attr (p,"ana_state" ); +if (ana_state) { +if (!p->ana_state ||strcmp (ana_state, p->ana_state)) { +free (p->ana_state); + p->ana_state =strdup (ana_state); + } + } + +return p->ana_state; +} +void nvme_free_path (struct libnvme_path *p) {if (!p)
libnvme/src/nvme/tree.h +8 -0
@@ -660,6 +660,14 @@ int libnvme_ns_identify_descs(libnvme_ns_t n, struct nvme_ns_id_desc *descs);*/ int libnvme_path_get_queue_depth (libnvme_path_t p); +/** +* libnvme_path_get_ana_state() - ANA state of an nvme_path_t object +* @p: &libnvme_path_t object +* +* Return: ANA state of @p +*/ +char *libnvme_path_get_ana_state (libnvme_path_t p); +/** * libnvme_path_get_ctrl() - Parent controller of an libnvme_path_t object * @p: &libnvme_path_t object
Signature · 1 line
2.53.0
PATCH v3 2/9 libnvme: annotate libnvme_path::numa_nodes with !accessors:none
In reply to Nilay ShroffThe default accessor generation creates getters that return cached attribute values. However, for libnvme_path::numa_nodes, a real-time (non-cached) value is required.
This is particularly useful for tools such as nvme-top, which rely on up-to-date information for displaying a real-time dashboard.
Annotate libnvme_path::numa_nodes with "!accessors:none" to disable the auto-generated accessor, and provide a custom implementation of libnvme_path_get_numa_nodes() that returns the current value.
--- libnvme/src/accessors.ld | 2 -- libnvme/src/libnvme.ld | 1 + libnvme/src/nvme/accessors.c | 13 ------------- libnvme/src/nvme/accessors.h | 17 ----------------- libnvme/src/nvme/private.h | 2 +- libnvme/src/nvme/tree.c | 15 +++++++++++++++ libnvme/src/nvme/tree.h | 8 ++++++++ 7 files changed, 25 insertions(+), 33 deletions(-)
libnvme/src/accessors.ld +0 -2
@@ -168,11 +168,9 @@ LIBNVME_ACCESSORS_3 { libnvme_ns_set_sysfs_dir; libnvme_path_get_grpid; libnvme_path_get_name; - libnvme_path_get_numa_nodes; libnvme_path_get_sysfs_dir; libnvme_path_set_grpid; libnvme_path_set_name; - libnvme_path_set_numa_nodes; libnvme_path_set_sysfs_dir; libnvme_subsystem_get_application; libnvme_subsystem_get_firmware;
libnvme/src/libnvme.ld +1 -0
@@ -140,6 +140,7 @@ LIBNVME_3 { libnvme_path_get_ns; libnvme_path_get_queue_depth; libnvme_path_get_ana_state; + libnvme_path_get_numa_nodes; libnvme_random_uuid; libnvme_read_config; libnvme_read_hostid;
libnvme/src/nvme/accessors.c +0 -13
@@ -291,19 +291,6 @@ __public const char *libnvme_path_get_sysfs_dir(const struct libnvme_path *p)return p->sysfs_dir; } -__public void libnvme_path_set_numa_nodes( -struct libnvme_path *p, -const char *numa_nodes) -{ -free (p->numa_nodes); - p->numa_nodes = numa_nodes ?strdup (numa_nodes) :NULL ; -} - -__public const char *libnvme_path_get_numa_nodes(const struct libnvme_path *p) -{ -return p->numa_nodes; -} - __public void libnvme_path_set_grpid(struct libnvme_path *p, int grpid) { p->grpid = grpid;
libnvme/src/nvme/accessors.h +0 -17
@@ -391,23 +391,6 @@ void libnvme_path_set_sysfs_dir(struct libnvme_path *p, const char *sysfs_dir);*/ const char *libnvme_path_get_sysfs_dir (const struct libnvme_path *p); -/** -* libnvme_path_set_numa_nodes() - Set numa_nodes. -* @p: The &struct libnvme_path instance to update. -* @numa_nodes: New string; a copy is stored. Pass NULL to clear. -*/ -void libnvme_path_set_numa_nodes ( -struct libnvme_path *p, -const char *numa_nodes); - -/** -* libnvme_path_get_numa_nodes() - Get numa_nodes. -* @p: The &struct libnvme_path instance to query. -* -* Return: The value of the numa_nodes field, or NULL if not set. -*/ -const char *libnvme_path_get_numa_nodes (const struct libnvme_path *p); -/** * libnvme_path_set_grpid() - Set grpid. * @p: The &struct libnvme_path instance to update.
libnvme/src/nvme/private.h +1 -1
@@ -176,8 +176,8 @@ struct libnvme_path { // !generate-accessorschar *name;char *sysfs_dir; -char *numa_nodes;char *ana_state;//!accessors:none +char *numa_nodes;//!accessors:none int grpid;int queue_depth;// !accessors:none };
libnvme/src/nvme/tree.c +15 -0
@@ -854,6 +854,21 @@ __public char *libnvme_path_get_ana_state(libnvme_path_t p)return p->ana_state; } +__public char *libnvme_path_get_numa_nodes(libnvme_path_t p) +{ + __cleanup_free char *numa_nodes = NULL; + + numa_nodes =libnvme_get_path_attr (p,"numa_nodes" ); +if (numa_nodes) { +if (!p->numa_nodes ||strcmp (numa_nodes, p->numa_nodes)) { +free (p->numa_nodes); + p->numa_nodes =strdup (numa_nodes); + } + } + +return p->numa_nodes; +} +void nvme_free_path (struct libnvme_path *p) {if (!p)
libnvme/src/nvme/tree.h +8 -0
@@ -668,6 +668,14 @@ int libnvme_path_get_queue_depth(libnvme_path_t p);*/ char *libnvme_path_get_ana_state (libnvme_path_t p); +/** +* libnvme_path_get_numa_nodes() - Numa nodes of an nvme_path_t object +* @p: &libnvme_path_t object +* +* Return: Numa nodes of @p +*/ +char *libnvme_path_get_numa_nodes (libnvme_path_t p); +/** * libnvme_path_get_ctrl() - Parent controller of an libnvme_path_t object * @p: &libnvme_path_t object
Signature · 1 line
2.53.0
PATCH v3 3/9 libnvme: annotate libnvme_subsystem::iopolicy with !accessors:none
In reply to Nilay ShroffThe default accessor generation creates getters that return cached attribute values. However, for libnvme_subsystem::iopolicy, a real-time (non-cached) value is required.
This is particularly useful for tools such as nvme-top, which rely on up-to-date information for displaying a real-time dashboard.
Annotate libnvme_subsystem::iopolicy with "!accessors:none" to disable the auto-generated accessor, and provide a custom implementation of libnvme_subsystem_get_iopolicy() that returns the current value.
--- libnvme/src/accessors.ld | 2 -- libnvme/src/libnvme.ld | 1 + libnvme/src/nvme/accessors.c | 14 -------------- libnvme/src/nvme/accessors.h | 17 ----------------- libnvme/src/nvme/private.h | 2 +- libnvme/src/nvme/tree.c | 15 +++++++++++++++ libnvme/src/nvme/tree.h | 8 ++++++++ 7 files changed, 25 insertions(+), 34 deletions(-)
libnvme/src/accessors.ld +0 -2
@@ -174,7 +174,6 @@ LIBNVME_ACCESSORS_3 { libnvme_path_set_sysfs_dir; libnvme_subsystem_get_application; libnvme_subsystem_get_firmware; - libnvme_subsystem_get_iopolicy; libnvme_subsystem_get_model; libnvme_subsystem_get_name; libnvme_subsystem_get_serial;
@@ -182,5 +181,4 @@ LIBNVME_ACCESSORS_3 { libnvme_subsystem_get_subsystype; libnvme_subsystem_get_sysfs_dir; libnvme_subsystem_set_application; - libnvme_subsystem_set_iopolicy; };
libnvme/src/libnvme.ld +1 -0
@@ -177,6 +177,7 @@ LIBNVME_3 { libnvme_subsystem_first_ctrl; libnvme_subsystem_first_ns; libnvme_subsystem_get_host; + libnvme_subsystem_get_iopolicy; libnvme_subsystem_lookup_namespace; libnvme_subsystem_next_ctrl; libnvme_subsystem_next_ns;
libnvme/src/nvme/accessors.c +0 -14
@@ -661,20 +661,6 @@ __public const char *libnvme_subsystem_get_application(return p->application; } -__public void libnvme_subsystem_set_iopolicy( -struct libnvme_subsystem *p, -const char *iopolicy) -{ -free (p->iopolicy); - p->iopolicy = iopolicy ?strdup (iopolicy) :NULL ; -} - -__public const char *libnvme_subsystem_get_iopolicy( -const struct libnvme_subsystem *p) -{ -return p->iopolicy; -} -/**************************************************************************** * Accessors for: struct libnvme_host ****************************************************************************/
libnvme/src/nvme/accessors.h +0 -17
@@ -901,23 +901,6 @@ void libnvme_subsystem_set_application(const char *libnvme_subsystem_get_application (const struct libnvme_subsystem *p); -/** -* libnvme_subsystem_set_iopolicy() - Set iopolicy. -* @p: The &struct libnvme_subsystem instance to update. -* @iopolicy: New string; a copy is stored. Pass NULL to clear. -*/ -void libnvme_subsystem_set_iopolicy ( -struct libnvme_subsystem *p, -const char *iopolicy); - -/** -* libnvme_subsystem_get_iopolicy() - Get iopolicy. -* @p: The &struct libnvme_subsystem instance to query. -* -* Return: The value of the iopolicy field, or NULL if not set. -*/ -const char *libnvme_subsystem_get_iopolicy (const struct libnvme_subsystem *p); -/**************************************************************************** * Accessors for: struct libnvme_host ****************************************************************************/
libnvme/src/nvme/private.h +1 -1
@@ -269,7 +269,7 @@ struct libnvme_subsystem { // !generate-accessorschar *firmware;// !accessors:readonly char *subsystype;// !accessors:readonly char *application; -char *iopolicy; +char *iopolicy;//!accessors:none };struct libnvme_host {// !generate-accessors
libnvme/src/nvme/tree.c +15 -0
@@ -464,6 +464,21 @@ __public libnvme_host_t libnvme_subsystem_get_host(libnvme_subsystem_t s)return s->h; } +__public char *libnvme_subsystem_get_iopolicy(libnvme_subsystem_t s) +{ + __cleanup_free char *iopolicy = NULL; + + iopolicy =libnvme_get_subsys_attr (s,"iopolicy" ); +if (iopolicy) { +if (!s->iopolicy ||strcmp (iopolicy, s->iopolicy)) { +free (s->iopolicy); + s->iopolicy =strdup (iopolicy); + } + } + +return s->iopolicy; +} + __public libnvme_ns_t libnvme_subsystem_first_ns(libnvme_subsystem_t s) { return list_top(&s->namespaces, struct libnvme_ns, entry);
libnvme/src/nvme/tree.h +8 -0
@@ -214,6 +214,14 @@ void libnvme_free_subsystem(struct libnvme_subsystem *s);*/ libnvme_host_t libnvme_subsystem_get_host (libnvme_subsystem_t s); +/** +* libnvme_subsystem_get_iopolicy() - Get subsystem iopolicy name +* @s: subsystem +* +* Return: The iopolicy configured in subsystem @s +*/ +char *libnvme_subsystem_get_iopolicy (libnvme_subsystem_t s); +/** * libnvme_ctrl_first_ns() - Start namespace iterator * @c: Controller instance
Signature · 1 line
2.53.0
PATCH v3 4/9 libnvme: add support for retrieving per-path gendisk I/O statistics
In reply to Nilay ShroffGendisk I/O statistics provide useful insight into disk activity, including read/write/discard/flush operations, as well as information about in-flight I/Os and I/O timing.
Parsing these statistics allows users to determine the number of I/Os processed, time spent servicing I/O, number of sectors accessed, and the count of in-flight requests.
Add support for retrieving per-path gendisk I/O statistics. Also add support for computing deltas of these statistics between samples, such as I/O ticks, number of sectors, and number of serviced I/Os.
These metrics can be used by tools such as nvme-top to display real-time disk activity. Per-path metrics are particularly useful when NVMe native multipath is enabled.
--- libnvme/src/libnvme.ld | 11 ++ libnvme/src/nvme/private.h | 30 ++++ libnvme/src/nvme/tree.c | 271 +++++++++++++++++++++++++++++++++++++ libnvme/src/nvme/tree.h | 93 +++++++++++++ 4 files changed, 405 insertions(+)
libnvme/src/libnvme.ld +11 -0
@@ -141,6 +141,17 @@ LIBNVME_3 { libnvme_path_get_queue_depth; libnvme_path_get_ana_state; libnvme_path_get_numa_nodes; + libnvme_path_get_read_ios; + libnvme_path_get_write_ios; + libnvme_path_get_inflights; + libnvme_path_get_stat_interval; + libnvme_path_get_io_ticks; + libnvme_path_get_read_ticks; + libnvme_path_get_write_ticks; + libnvme_path_get_read_sectors; + libnvme_path_get_write_sectors; + libnvme_path_reset_stat; + libnvme_path_update_stat; libnvme_random_uuid; libnvme_read_config; libnvme_read_hostid;
libnvme/src/nvme/private.h +30 -0
@@ -15,6 +15,8 @@#include "nvme/nvme-types.h" #include "nvme/lib-types.h" +#include <nvme/tree.h> +const char *libnvme_subsys_sysfs_dir (void );const char *libnvme_ctrl_sysfs_dir (void );const char *libnvme_ns_sysfs_dir (void );
@@ -167,10 +169,38 @@ struct libnvme_transport_handle {struct libnvme_log *log; }; +enum libnvme_stat_group { +READ =0 , +WRITE , +DISCARD , +FLUSH , + +NR_STAT_GROUPS +}; + +struct libnvme_stat { +struct { +unsigned long ios; +unsigned long merges; +unsigned long long sectors; +unsigned int ticks;/* in milliseconds */ + } group[NR_STAT_GROUPS ]; + +unsigned int inflights; +unsigned int io_ticks;/* in milliseconds */ +unsigned int tot_ticks;/* in milliseconds */ + +double ts_ms;/* timestamp when the stat is updated */ +}; +struct libnvme_path {// !generate-accessors struct list_node entry;struct list_node nentry; +struct libnvme_stat stat[2 ];/* gendisk I/O stat */ +unsigned int curr_idx;/* current index into the stat[] */ +bool diffstat;//!accessors:none +struct libnvme_ctrl *c;struct libnvme_ns *n;
libnvme/src/nvme/tree.c +271 -0
283 lines not on this page · raw message
libnvme/src/nvme/tree.h +93 -0
105 lines not on this page · raw message
Signature · 1 line
2.53.0
PATCH v3 5/9 libnvme: add support for retrieving namespace gendisk I/O statistics
In reply to Nilay ShroffGendisk I/O statistics provide useful insight into disk activity, including read/write/discard/flush operations, as well as information about in-flight I/Os and I/O timing.
Parsing these statistics allows users to determine the number of I/Os processed, time spent servicing I/O, number of sectors accessed, and the count of in-flight requests.
Add support for retrieving namespace gendisk I/O statistics. Also add support for computing deltas of these statistics between samples, such as I/O ticks, number of sectors, and number of serviced I/Os.
These metrics can be used by tools such as nvme-top to display real-time disk activity.
--- libnvme/src/libnvme.ld | 11 +++ libnvme/src/nvme/private.h | 5 ++ libnvme/src/nvme/tree.c | 142 +++++++++++++++++++++++++++++++++++++ libnvme/src/nvme/tree.h | 93 ++++++++++++++++++++++++ 4 files changed, 251 insertions(+)
libnvme/src/libnvme.ld +11 -0
@@ -129,6 +129,17 @@ LIBNVME_3 { libnvme_ns_get_serial; libnvme_ns_get_subsystem; libnvme_ns_get_uuid; + libnvme_ns_reset_stat; + libnvme_ns_update_stat; + libnvme_ns_get_stat_interval; + libnvme_ns_get_read_ios; + libnvme_ns_get_write_ios; + libnvme_ns_get_read_ticks; + libnvme_ns_get_write_ticks; + libnvme_ns_get_read_sectors; + libnvme_ns_get_write_sectors; + libnvme_ns_get_inflights; + libnvme_ns_get_io_ticks; libnvme_ns_identify; libnvme_ns_read; libnvme_ns_verify;
libnvme/src/nvme/private.h +5 -0
@@ -227,6 +227,11 @@ struct libnvme_ns { // !generate-accessorsstruct libnvme_ns_head *head;struct libnvme_global_ctx *ctx; + +struct libnvme_stat stat[2 ];/* gendisk I/O stat */ +unsigned int curr_idx;/* current index into the stat[] */ +bool diffstat;//!accessors:none +struct libnvme_transport_handle *hdl;__u32 nsid;char *name;
libnvme/src/nvme/tree.c +142 -0
190 lines not on this page · raw message
libnvme/src/nvme/tree.h +93 -0
99 lines not on this page · raw message
Signature · 1 line
2.53.0
PATCH v3 6/9 libnvme: add support for per-path diagnostic counters
In reply to Nilay ShroffAdd support for retrieving per-path diagnostic counters such as command_retry_count, command_error_count, and multipath_failover_count. These counters improve visibility into NVMe native multipath behavior and can be useful for tools such as nvme-top to display real-time statistics.
Unlike other sysfs attributes, these counters can change dynamically. Annotate them with "!accessors:none" and provide custom implementations to always retrieve the latest (non-cached) values.
--- libnvme/src/libnvme.ld | 3 +++ libnvme/src/nvme/private.h | 5 ++++- libnvme/src/nvme/tree.c | 33 +++++++++++++++++++++++++++++++++ libnvme/src/nvme/tree.h | 27 +++++++++++++++++++++++++++ 4 files changed, 67 insertions(+), 1 deletion(-)
libnvme/src/libnvme.ld +3 -0
@@ -163,6 +163,9 @@ LIBNVME_3 { libnvme_path_get_write_sectors; libnvme_path_reset_stat; libnvme_path_update_stat; + libnvme_path_get_command_retry_count; + libnvme_path_get_command_error_count; + libnvme_path_get_multipath_failover_count; libnvme_random_uuid; libnvme_read_config; libnvme_read_hostid;
libnvme/src/nvme/private.h +4 -1
@@ -209,7 +209,10 @@ struct libnvme_path { // !generate-accessorschar *ana_state;//!accessors:none char *numa_nodes;//!accessors:none int grpid; -int queue_depth;// !accessors:none +int queue_depth;//!accessors:none +long multipath_failover_count;//!accessors:none +long command_retry_count;//!accessors:none +long command_error_count;//!accessors:none };struct libnvme_ns_head {
libnvme/src/nvme/tree.c +33 -0
@@ -885,6 +885,39 @@ __public char *libnvme_path_get_numa_nodes(libnvme_path_t p)return p->numa_nodes; } +__public long libnvme_path_get_multipath_failover_count (libnvme_path_t p) +{ + __cleanup_free char *failover_count = NULL; + + failover_count =libnvme_get_path_attr (p,"multipath_failover_count" ); +if (failover_count) +sscanf (failover_count,"%ld" , &p->multipath_failover_count); + +return p->multipath_failover_count; +} + +__public long libnvme_path_get_command_retry_count (libnvme_path_t p) +{ + __cleanup_free char *retry_count = NULL; + + retry_count =libnvme_get_path_attr (p,"command_retry_count" ); +if (retry_count) +sscanf (retry_count,"%ld" , &p->command_retry_count); + +return p->command_retry_count; +} + +__public long libnvme_path_get_command_error_count (libnvme_path_t p) +{ + __cleanup_free char *error_count = NULL; + + error_count =libnvme_get_path_attr (p,"command_error_count" ); +if (error_count) +sscanf (error_count,"%ld" , &p->command_error_count); + +return p->command_error_count; +} +static libnvme_stat_t libnvme_path_get_stat (libnvme_path_t p,unsigned int idx) {if (idx >1 )
libnvme/src/nvme/tree.h +27 -0
@@ -685,6 +685,33 @@ char *libnvme_path_get_ana_state(libnvme_path_t p);*/ char *libnvme_path_get_numa_nodes (libnvme_path_t p); +/** +* libnvme_path_get_multipath_failover_count() - Get multipath failover count +* @p: &libnvme_path_t object +* +* Return: Number of times I/Os have to be failed over to another active path +* from path @p maybe due to any transient error observed on path @p +*/ +long libnvme_path_get_multipath_failover_count (libnvme_path_t p); + +/** +* libnvme_path_get_command_retry_count() - Get command retry count +* @p: &libnvme_path_t object +* +* Return: Number of times any command issued to the namespace represented by +* path @p has to be retried +*/ +long libnvme_path_get_command_retry_count (libnvme_path_t p); + +/** +* libnvme_path_get_command_error_count() - Get command error count +* @p: &libnvme_path_t object +* +* Return: Number of times command issued to the namespace represented by path +* @p returns non-zero status or error +*/ +long libnvme_path_get_command_error_count (libnvme_path_t p); +/** * libnvme_path_get_ctrl() - Parent controller of an libnvme_path_t object * @p: &libnvme_path_t object
Signature · 1 line
2.53.0
PATCH v3 7/9 libnvme: add support for namespace diagnostic counters
In reply to Nilay ShroffAdd support for retrieving namespace diagnostic counters such as command_retry_count and command_error_count. These counters improve visibility and can be useful for tools such as nvme-top to display real-time statistics.
Unlike other sysfs attributes, these counters can change dynamically. Annotate them with "!accessors:none" and provide custom implementations to always retrieve the latest (non-cached) values.
--- libnvme/src/libnvme.ld | 2 ++ libnvme/src/nvme/private.h | 3 +++ libnvme/src/nvme/tree.c | 22 ++++++++++++++++++++++ libnvme/src/nvme/tree.h | 17 +++++++++++++++++ 4 files changed, 44 insertions(+)
libnvme/src/libnvme.ld +2 -0
@@ -140,6 +140,8 @@ LIBNVME_3 { libnvme_ns_get_write_sectors; libnvme_ns_get_inflights; libnvme_ns_get_io_ticks; + libnvme_ns_get_command_retry_count; + libnvme_ns_get_command_error_count; libnvme_ns_identify; libnvme_ns_read; libnvme_ns_verify;
libnvme/src/nvme/private.h +3 -0
@@ -251,6 +251,9 @@ struct libnvme_ns { // !generate-accessorsuint8_t nguid[16 ];unsigned char uuid[NVME_UUID_LEN ];enum nvme_csi csi; + +long command_retry_count;//!accessors:none +long command_error_count;//!accessors:none };struct libnvme_ctrl {// !generate-accessors
libnvme/src/nvme/tree.c +22 -0
@@ -2611,6 +2611,28 @@ __public void libnvme_ns_get_uuid(libnvme_ns_t n,memcpy (out, n->uuid,NVME_UUID_LEN ); } +__public long libnvme_ns_get_command_retry_count (libnvme_ns_t n) +{ + __cleanup_free char *retry_count = NULL; + + retry_count =libnvme_get_ns_attr (n,"command_retry_count" ); +if (retry_count) +sscanf (retry_count,"%ld" , &n->command_retry_count); + +return n->command_retry_count; +} + +__public long libnvme_ns_get_command_error_count (libnvme_ns_t n) +{ + __cleanup_free char *error_count = NULL; + + error_count =libnvme_get_ns_attr (n,"command_error_count" ); +if (error_count) +sscanf (error_count,"%ld" , &n->command_error_count); + +return n->command_error_count; +} + __public int libnvme_ns_identify(libnvme_ns_t n, struct nvme_id_ns *ns) {struct libnvme_transport_handle *hdl;
libnvme/src/nvme/tree.h +17 -0
@@ -510,6 +510,23 @@ const uint8_t *libnvme_ns_get_nguid(libnvme_ns_t n);*/ void libnvme_ns_get_uuid (libnvme_ns_t n,unsigned char out[NVME_UUID_LEN ]); +/** +* libnvme_ns_get_command_retry_count() - Get command retry count +* @n: &libnvme_ns_t object +* +* Return: Number of times any command issued to namespace @n has to be retried +*/ +long libnvme_ns_get_command_retry_count (libnvme_ns_t n); + +/** +* libnvme_ns_get_command_error_count() - Get command error count +* @n: &libnvme_ns_t object +* +* Return: Number of times command issued to namespace @n returns non-zero +* status or error +*/ +long libnvme_ns_get_command_error_count (libnvme_ns_t n); +/** * libnvme_ns_get_generic_name() - Returns name of generic namespace chardev. * @n: Namespace instance
Signature · 1 line
2.53.0
PATCH v3 8/9 libnvme: add support for nshead diagnostic counters
In reply to Nilay ShroffAdd support for retrieving nshead diagnostic counters such as requeue_ no_usable_path_count and fail_no_available_path_count.
The requeue_no_usable_path_count counter represents the number of I/Os requeued to the namespace head when no usable path is available (for example, due to transient link issues). The fail_no_available_path_count counter represents the number of I/Os failed when no I/O path is available (for example, when all paths are removed).
These counters improve visibility into NVMe native multipath behavior and can be useful for tools such as nvme-top to display real-time statistics.
Unlike other sysfs attributes, these counters can change dynamically. Annotate them with "!accessors:none" and provide custom implementations to always retrieve the latest (non-cached) values.
--- libnvme/src/libnvme.ld | 2 ++ libnvme/src/nvme/private.h | 2 ++ libnvme/src/nvme/tree.c | 22 ++++++++++++++++++++++ libnvme/src/nvme/tree.h | 17 +++++++++++++++++ 4 files changed, 43 insertions(+)
libnvme/src/libnvme.ld +2 -0
@@ -142,6 +142,8 @@ LIBNVME_3 { libnvme_ns_get_io_ticks; libnvme_ns_get_command_retry_count; libnvme_ns_get_command_error_count; + libnvme_ns_get_requeue_no_usable_path_count; + libnvme_ns_get_fail_no_available_path_count; libnvme_ns_identify; libnvme_ns_read; libnvme_ns_verify;
libnvme/src/nvme/private.h +2 -0
@@ -254,6 +254,8 @@ struct libnvme_ns { // !generate-accessorslong command_retry_count;//!accessors:none long command_error_count;//!accessors:none +long requeue_no_usable_path_count;//!accessors:none +long fail_no_available_path_count;//!accessors:none };struct libnvme_ctrl {// !generate-accessors
libnvme/src/nvme/tree.c +22 -0
@@ -2633,6 +2633,28 @@ __public long libnvme_ns_get_command_error_count(libnvme_ns_t n)return n->command_error_count; } +__public long libnvme_ns_get_requeue_no_usable_path_count (libnvme_ns_t n) +{ + __cleanup_free char *requeue_count = NULL; + + requeue_count =libnvme_get_ns_attr (n,"requeue_no_usable_path_count" ); +if (requeue_count) +sscanf (requeue_count,"%ld" , &n->requeue_no_usable_path_count); + +return n->requeue_no_usable_path_count; +} + +__public long libnvme_ns_get_fail_no_available_path_count (libnvme_ns_t n) +{ + __cleanup_free char *fail_count = NULL; + + fail_count =libnvme_get_ns_attr (n,"fail_no_available_path_count" ); +if (fail_count) +sscanf (fail_count,"%ld" , &n->fail_no_available_path_count); + +return n->fail_no_available_path_count; +} + __public int libnvme_ns_identify(libnvme_ns_t n, struct nvme_id_ns *ns) {struct libnvme_transport_handle *hdl;
libnvme/src/nvme/tree.h +17 -0
@@ -527,6 +527,23 @@ long libnvme_ns_get_command_retry_count(libnvme_ns_t n);*/ long libnvme_ns_get_command_error_count (libnvme_ns_t n); +/** +* libnvme_ns_get_requeue_no_usable_path_count() - Get num of I/O requeue count +* @n: &libnvme_ns_t object +* +* Return: Number of I/Os which are re-queued due to the unavalibility of +* any usable path (maybe path is currently experiencing transinet link failure) +*/ +long libnvme_ns_get_requeue_no_usable_path_count (libnvme_ns_t n); + +/** +* libnvme_ns_get_fail_no_available_path_count() - Get num of I/Os forced to fail +* @n: &libnvme_ns_t object +* +* Return: Number of I/Os which are forced to fail due to no path available +*/ +long libnvme_ns_get_fail_no_available_path_count (libnvme_ns_t n); +/** * libnvme_ns_get_generic_name() - Returns name of generic namespace chardev. * @n: Namespace instance
Signature · 1 line
2.53.0
PATCH v3 9/9 libnvme: add support for ctrl diagnostic counters
In reply to Nilay ShroffAdd support for retrieving ctrl diagnostic counters such as command_ error_count, reset_count and reconnect_count. These counters improve visibility and can be useful for tools such as nvme-top to display real-time statistics.
Unlike other sysfs attributes, these counters can change dynamically. Annotate them with "!accessors:none" and provide custom implementations to always retrieve the latest (non-cached) values.
--- libnvme/src/libnvme.ld | 3 +++ libnvme/src/nvme/private.h | 3 +++ libnvme/src/nvme/tree.c | 33 +++++++++++++++++++++++++++++++++ libnvme/src/nvme/tree.h | 25 +++++++++++++++++++++++++ 4 files changed, 64 insertions(+)
libnvme/src/libnvme.ld +3 -0
@@ -14,6 +14,9 @@ LIBNVME_3 { libnvme_ctrl_get_subsysnqn; libnvme_ctrl_get_subsystem; libnvme_ctrl_get_transport_handle; + libnvme_ctrl_get_command_error_count; + libnvme_ctrl_get_reset_count; + libnvme_ctrl_get_reconnect_count; libnvme_ctrl_identify; libnvme_ctrl_match_config; libnvme_ctrl_next_ns;
libnvme/src/nvme/private.h +3 -0
@@ -295,6 +295,9 @@ struct libnvme_ctrl { // !generate-accessorsbool unique_discovery_ctrl;bool discovered;bool persistent; +long command_error_count;//!accessors:none +long reset_count;//!accessors:none +long reconnect_count;//!accessors:none struct libnvme_fabrics_config cfg; };
libnvme/src/nvme/tree.c +33 -0
@@ -1458,6 +1458,39 @@ __public const char *libnvme_ctrl_get_state(libnvme_ctrl_t c)return c->state; } +__public long libnvme_ctrl_get_command_error_count (libnvme_ctrl_t c) +{ + __cleanup_free char *error_count = NULL; + + error_count =libnvme_get_ctrl_attr (c,"command_error_count" ); +if (error_count) +sscanf (error_count,"%ld" , &c->command_error_count); + +return c->command_error_count; +} + +__public long libnvme_ctrl_get_reset_count (libnvme_ctrl_t c) +{ + __cleanup_free char *reset_count = NULL; + + reset_count =libnvme_get_ctrl_attr (c,"reset_count" ); +if (reset_count) +sscanf (reset_count,"%ld" , &c->reset_count); + +return c->reset_count; +} + +__public long libnvme_ctrl_get_reconnect_count (libnvme_ctrl_t c) +{ + __cleanup_free char *reconnect_count = NULL; + + reconnect_count =libnvme_get_ctrl_attr (c,"reconnect_count" ); +if (reconnect_count) +sscanf (reconnect_count,"%ld" , &c->reconnect_count); + +return c->reconnect_count; +} + __public int libnvme_ctrl_identify(libnvme_ctrl_t c, struct nvme_id_ctrl *id) { struct libnvme_transport_handle *hdl =
libnvme/src/nvme/tree.h +25 -0
@@ -1007,6 +1007,31 @@ unsigned long long libnvme_ns_get_read_sectors(libnvme_ns_t n);*/ unsigned long long libnvme_ns_get_write_sectors (libnvme_ns_t n); +/** +* libnvme_ctrl_get_command_error_count() - Get admin command error count +* @c: Controller instance +* +* Return: Number of times admin command issued to controller @c failed or +* returned error status +*/ +long libnvme_ctrl_get_command_error_count (libnvme_ctrl_t c); + +/** +* libnvme_ctrl_get_reset_count() - Get controller reset count +* @c: Controller instance +* +* Return: Number of timer controller @c is reset +*/ +long libnvme_ctrl_get_reset_count (libnvme_ctrl_t c); + +/** +* libnvme_ctrl_get_reconnect_count() - Get controller reconnect count +* @c: Controller instance +* +* Return: Number of times controller has to reconnect to the target +*/ +long libnvme_ctrl_get_reconnect_count (libnvme_ctrl_t c); +/** * libnvme_ctrl_identify() - Issues an 'identify controller' command * @c: Controller instance
Signature · 1 line
2.53.0
Quoted 22 lines · Nilay Shroff
Certain NVMe attributes and statistics can change dynamically at runtime. However, the current libnvme implementation caches attribute values while scanning and building the topology. Subsequent libnvme API calls return these cached values.
While this approach works well for one-shot nvme-cli commands, it is not suitable for use cases that require upto date information. In particular, real-time monitoring tools such as nvme-top require access to the latest (non-cached) values of such attributes.
To address this, this patchset introduces support for retrieving non-cached values for selected attributes whose values may change dynamically. The first three patches disable auto-generated accessors for
ana_state,numa_nodes, and iopolicy using "!accessors:none", and provide custom implementations that always return the latest values.In addition, libnvme lacks support for retrieving gendisk statistics, which provide useful insight into disk activity (e.g., number of I/Os processed, time spent servicing I/O, and in-flight I/O counts). This patchset adds support for retrieving gendisk I/O statistics for both per-path and namespace gendisks.
Quoted 8 lines
Finally, this series adds support for retrieving diagnostic counters at different levels, including per-path, namespace, nshead, and controller. These counters should improve visibility into NVMe native multipath behavior.
All of the above statistics and counters can be leveraged by tools such as nvme-top to provide real-time monitoring and analysis.
As usual, feedback and suggestions are welcome!
I went ahead and applied them. I did some cosmetic formatting changes for the accessors generator comment and added the manually written getters to the python binding define table.
This should unblock your work on 'nvme monitor'
Thanks,
Daniel
Quoted 30 lines · Daniel Wagner
Quoted 29 lines · Nilay Shroff
Certain NVMe attributes and statistics can change dynamically at runtime. However, the current libnvme implementation caches attribute values while scanning and building the topology. Subsequent libnvme API calls return these cached values.
While this approach works well for one-shot nvme-cli commands, it is not suitable for use cases that require upto date information. In particular, real-time monitoring tools such as nvme-top require access to the latest (non-cached) values of such attributes.
To address this, this patchset introduces support for retrieving non-cached values for selected attributes whose values may change dynamically. The first three patches disable auto-generated accessors for
ana_state,numa_nodes, and iopolicy using "!accessors:none", and provide custom implementations that always return the latest values.In addition, libnvme lacks support for retrieving gendisk statistics, which provide useful insight into disk activity (e.g., number of I/Os processed, time spent servicing I/O, and in-flight I/O counts). This patchset adds support for retrieving gendisk I/O statistics for both per-path and namespace gendisks.
Finally, this series adds support for retrieving diagnostic counters at different levels, including per-path, namespace, nshead, and controller. These counters should improve visibility into NVMe native multipath behavior.
All of the above statistics and counters can be leveraged by tools such as nvme-top to provide real-time monitoring and analysis.
Quoted 8 lines
Quoted 1 line
As usual, feedback and suggestions are welcome!
I went ahead and applied them. I did some cosmetic formatting changes for the accessors generator comment and added the manually written getters to the python binding define table.
This should unblock your work on 'nvme monitor'
Thanks Daniel! I'll now work on nvme monitor dashboard.
Thanks,
--Nilay