nvme: export additional diagnostic counters via sysfs
Subsystems (2)
v4 cover nvme: export additional diagnostic counters via sysfs
Hi,
The NVMe driver encounters various events and conditions during normal operation that are either not tracked today or not exposed to userspace via sysfs. Lack of visibility into these events can make it difficult to diagnose subtle issues related to controller behavior, multipath stability, and I/O reliability.
This patchset adds several diagnostic counters that provide improved observability into NVMe behavior. These counters are intended to help users understand events such as transient path unavailability, controller retries/reconnect/reset, failovers, and I/O failures. They can also be consumed by monitoring tools such as nvme-top.
Specifically, this series proposes to export the following counters via sysfs:
- Command retry count
- Multipath failover count
- Command error count
- I/O requeue count
- I/O failure count
- Controller reset event counts
- Controller reconnect counts
The first patch in the series adds a new diag attribute group under per-path, ns-head and ctrl sysfs directories so that all diagnostics counters could be grouped together under diag sub-directory. The subsequent patches in the series adds diagnostics counters listed above.
Please note that this patchset doesn't make any functional change but rather export relevant counters to user space via sysfs.
As usual, feedback/comments/suggestions are welcome!
Changes from v3:
- To be consistent in naming, all counters are suffixed with _count (Keith Busch)
- The first patch in the series creates new attribute group named diag and all counters are now grouped under this new sysfs attribute group (Keith Busch)
- Counters are defined as atomic_long_t instead of size_t (Keith Busch)
- Removed RB and TB tags due to above changes
Link to v3: https://lore.kernel.org/all/[email protected]/
Changes from v2:
- Allow user to write to sysfs attributes so that user could
reset stat counters, if needed (Sagi)
- The controller reconnect counter nr_reconnects could reset
to zero once connection is re-established, so instead of
exposing nr_reconnects counter via sysfs introduce a new
counter which accumulates the reconnect attempts and export
this accumulated counter via sysfs (Sagi)
Link to v2: https://lore.kernel.org/all/[email protected]/
Changes from v1:
- Remove export of stats for admin command rerty count (Keith)
- Use size_add() to ensure stat counters don't overflow (Keith)
Link to v1: https://lore.kernel.org/all/[email protected]/
Nilay Shroff (8):
nvme: add diag attribute group under sysfs
nvme: export command retry count via sysfs
nvme: export multipath failover count via sysfs
nvme: export command error counters via sysfs
nvme: export I/O requeue count when no path is available via sysfs
nvme: export I/O failure count when no path is available via sysfs
nvme: export controller reset event count via sysfs
nvme: export controller reconnect event count via sysfs
drivers/nvme/host/core.c | 15 ++- drivers/nvme/host/fc.c | 3 + drivers/nvme/host/multipath.c | 87 ++++++++++++++ drivers/nvme/host/nvme.h | 13 +++ drivers/nvme/host/pci.c | 1 + drivers/nvme/host/rdma.c | 2 + drivers/nvme/host/sysfs.c | 214 ++++++++++++++++++++++++++++++++++ drivers/nvme/host/tcp.c | 2 + 8 files changed, 336 insertions(+), 1 deletion(-)
Signature · 1 line
2.53.0
PATCH v4 1/8 nvme: add diag attribute group under sysfs
Add a new diag attribute group under:
/sys/class/nvme/<ctrl>/
/sys/block/<nvme-path-dev>/
/sys/block/<ns-head-dev>/
This new sysfs attribute group will be used to organize NVMe diagnostic and telemetry-related counters under it.
--- drivers/nvme/host/nvme.h | 1 + drivers/nvme/host/pci.c | 1 + drivers/nvme/host/sysfs.c | 35 +++++++++++++++++++++++++++++++++++ 3 files changed, 37 insertions(+)
drivers/nvme/host/nvme.h +1 -0
@@ -1012,6 +1012,7 @@ extern const struct attribute_group nvme_ns_mpath_attr_group;extern const struct pr_ops nvme_pr_ops;extern const struct block_device_operations nvme_ns_head_ops;extern const struct attribute_group nvme_dev_attrs_group; +extern const struct attribute_group nvme_dev_diag_attrs_group;extern const struct attribute_group *nvme_subsys_attrs_groups[];extern const struct attribute_group *nvme_dev_attr_groups[];extern const struct block_device_operations nvme_bdev_ops;
drivers/nvme/host/pci.c +1 -0
@@ -2810,6 +2810,7 @@ static const struct attribute_group nvme_pci_dev_attrs_group = {static const struct attribute_group *nvme_pci_dev_attr_groups[] = { &nvme_dev_attrs_group, &nvme_pci_dev_attrs_group, + &nvme_dev_diag_attrs_group,NULL , };
drivers/nvme/host/sysfs.c +35 -0
@@ -351,11 +351,28 @@ const struct attribute_group nvme_ns_mpath_attr_group = { };#endif +static struct attribute *nvme_ns_diag_attrs[] = { +NULL , +}; + +static umode_t nvme_ns_diag_attrs_are_visible (struct kobject *kobj, +struct attribute *a,int n) +{ +return a->mode; +} + +const struct attribute_group nvme_ns_diag_attr_group = { + .name ="diag" , + .attrs = nvme_ns_diag_attrs, + .is_visible = nvme_ns_diag_attrs_are_visible, +}; +const struct attribute_group *nvme_ns_attr_groups[] = { &nvme_ns_attr_group, #ifdef CONFIG_NVME_MULTIPATH &nvme_ns_mpath_attr_group, #endif + &nvme_ns_diag_attr_group,NULL , };
@@ -937,11 +954,29 @@ static const struct attribute_group nvme_tls_attrs_group = { };#endif +static struct attribute *nvme_dev_diag_attrs[] = { +NULL , +}; + +static umode_t nvme_dev_diag_attrs_are_visible (struct kobject *kobj, +struct attribute *a,int n) +{ +return a->mode; +} + +const struct attribute_group nvme_dev_diag_attrs_group = { + .name ="diag" , + .attrs = nvme_dev_diag_attrs, + .is_visible = nvme_dev_diag_attrs_are_visible, +}; +EXPORT_SYMBOL_GPL (nvme_dev_diag_attrs_group); +const struct attribute_group *nvme_dev_attr_groups[] = { &nvme_dev_attrs_group, #ifdef CONFIG_NVME_TCP_TLS &nvme_tls_attrs_group, #endif + &nvme_dev_diag_attrs_group,NULL , };
Signature · 1 line
2.53.0
PATCH v4 2/8 nvme: export command retry count via sysfs
In reply to Nilay ShroffWhen Advanced Command Retry Enable (ACRE) is configured, a controller may interrupt command execution and return a completion status indicating command interrupted with the DNR bit cleared. In this case, the driver retries the command based on the Command Retry Delay (CRD) value provided in the completion status.
Currently, these command retries are handled entirely within the NVMe driver and are not visible to userspace. As a result, there is no observability into retry behavior, which can be a useful diagnostic signal.
Expose a per-namespace sysfs attribute command_retries_count, under diag attribute group to provide visibility into retry activity. This information can help identify controller-side congestion under load and enables comparison across paths in multipath setups (for example, detecting cases where one path experiences significantly more retries than another under identical workloads).
This exported metric is intended for diagnostics and monitoring tools such as nvme-top, and does not change command retry behavior. A new sysfs attribute named "command_retries_count" is added for this purpose. This attribute is both readable as well as writable. So user could reset this counter if needed.
--- drivers/nvme/host/core.c | 4 ++++ drivers/nvme/host/nvme.h | 1 + drivers/nvme/host/sysfs.c | 33 +++++++++++++++++++++++++++++++++ 3 files changed, 38 insertions(+)
drivers/nvme/host/core.c +4 -0
@@ -323,6 +323,7 @@ static void nvme_retry_req(struct request *req) {unsigned long delay =0 ;u16 crd; +struct nvme_ns *ns = req->q->queuedata;/* The mask and shift result must be <= 3 */ crd = (nvme_req (req)->status &NVME_STATUS_CRD ) >>11 ;
@@ -330,6 +331,9 @@ static void nvme_retry_req(struct request *req) delay =nvme_req (req)->ctrl->crdt[crd -1 ] *100 ;nvme_req (req)->retries++; +if (ns) +atomic_long_inc (&ns->retries); +blk_mq_requeue_request (req, false);blk_mq_delay_kick_requeue_list (req->q, delay); }
drivers/nvme/host/nvme.h +1 -0
@@ -590,6 +590,7 @@ struct nvme_ns {enum nvme_ana_state ana_state;u32 ana_grpid;#endif +atomic_long_t retries;struct list_head siblings;struct kref kref;struct nvme_ns_head *head;
drivers/nvme/host/sysfs.c +33 -0
@@ -351,13 +351,46 @@ const struct attribute_group nvme_ns_mpath_attr_group = { };#endif +static ssize_t command_retries_count_show (struct device *dev, +struct device_attribute *attr,char *buf) +{ +struct nvme_ns *ns =nvme_get_ns_from_dev (dev); + +return sysfs_emit (buf,"%lu\n" ,atomic_long_read (&ns->retries)); +} + +static ssize_t command_retries_count_store (struct device *dev, +struct device_attribute *attr,const char *buf,size_t count) +{ +unsigned long retries; +int err; +struct nvme_ns *ns =nvme_get_ns_from_dev (dev); + + err =kstrtoul (buf,0 , &retries); +if (err) +return -EINVAL ; + +atomic_long_set (&ns->retries, retries); + +return count; +} +static DEVICE_ATTR_RW (command_retries_count); +static struct attribute *nvme_ns_diag_attrs[] = { + &dev_attr_command_retries_count.attr,NULL , };static umode_t nvme_ns_diag_attrs_are_visible (struct kobject *kobj,struct attribute *a,int n) { + struct device *dev = container_of(kobj, struct device, kobj); + +if (a == &dev_attr_command_retries_count.attr) { +if (nvme_disk_is_ns_head (dev_to_disk (dev))) +return 0 ; + } +return a->mode; }
Signature · 1 line
2.53.0
PATCH v4 3/8 nvme: export multipath failover count via sysfs
In reply to Nilay ShroffWhen an NVMe command completes with a path-specific error, the NVMe driver may retry the command on an alternate controller or path if one is available. These failover events indicate that I/O was redirected away from the original path.
Currently, the number of times requests are failed over to another available path is not visible to userspace. Exposing this information can be useful for diagnosing path health and stability.
Export per-path sysfs attribute "multipath_failover_count" under diag attribute group. This attribute is both readable and writable and thus allowing user to reset the counter. This counter can be consumed by monitoring tools such as nvme-top to help identify paths that consistently trigger failovers under load.
--- drivers/nvme/host/multipath.c | 27 +++++++++++++++++++++++++++ drivers/nvme/host/nvme.h | 2 ++ drivers/nvme/host/sysfs.c | 10 +++++++++- 3 files changed, 38 insertions(+), 1 deletion(-)
drivers/nvme/host/multipath.c +27 -0
@@ -142,6 +142,7 @@ void nvme_failover_req(struct request *req)struct bio *bio;nvme_mpath_clear_current_path (ns); +atomic_long_inc (&ns->failover);/* * If we got back an ANA error, we know the controller is alive but not
@@ -1151,6 +1152,32 @@ static ssize_t delayed_removal_secs_store(struct device *dev,DEVICE_ATTR_RW (delayed_removal_secs); +static ssize_t multipath_failover_count_show (struct device *dev, +struct device_attribute *attr,char *buf) +{ +struct nvme_ns *ns =nvme_get_ns_from_dev (dev); + +return sysfs_emit (buf,"%lu\n" ,atomic_long_read (&ns->failover)); +} + +static ssize_t multipath_failover_count_store (struct device *dev, +struct device_attribute *attr,const char *buf,size_t count) +{ +unsigned long failover; +int ret; +struct nvme_ns *ns =nvme_get_ns_from_dev (dev); + + ret =kstrtoul (buf,0 , &failover); +if (ret) +return -EINVAL ; + +atomic_long_set (&ns->failover, failover); + +return count; +} + +DEVICE_ATTR_RW (multipath_failover_count); +static int nvme_lookup_ana_group_desc (struct nvme_ctrl *ctrl,struct nvme_ana_group_desc *desc,void *data) {
drivers/nvme/host/nvme.h +2 -0
@@ -589,6 +589,7 @@ struct nvme_ns {#ifdef CONFIG_NVME_MULTIPATH enum nvme_ana_state ana_state;u32 ana_grpid; +atomic_long_t failover;#endif atomic_long_t retries;struct list_head siblings;
@@ -1063,6 +1064,7 @@ extern struct device_attribute dev_attr_ana_state;extern struct device_attribute dev_attr_queue_depth;extern struct device_attribute dev_attr_numa_nodes;extern struct device_attribute dev_attr_delayed_removal_secs; +extern struct device_attribute dev_attr_multipath_failover_count;extern struct device_attribute subsys_attr_iopolicy;static inline bool nvme_disk_is_ns_head (struct gendisk *disk)
drivers/nvme/host/sysfs.c +9 -1
@@ -378,6 +378,9 @@ static DEVICE_ATTR_RW(command_retries_count);static struct attribute *nvme_ns_diag_attrs[] = { &dev_attr_command_retries_count.attr, +#ifdef CONFIG_NVME_MULTIPATH + &dev_attr_multipath_failover_count.attr, +#endifNULL , };
@@ -390,7 +393,12 @@ static umode_t nvme_ns_diag_attrs_are_visible(struct kobject *kobj,if (nvme_disk_is_ns_head (dev_to_disk (dev)))return 0 ; } - +#ifdef CONFIG_NVME_MULTIPATH +if (a == &dev_attr_multipath_failover_count.attr) { +if (nvme_disk_is_ns_head (dev_to_disk (dev))) +return 0 ; + } +#endif return a->mode; }
Signature · 1 line
2.53.0
PATCH v4 4/8 nvme: export command error counters via sysfs
In reply to Nilay ShroffWhen an NVMe command completes with an error status, the driver logs the error to the kernel log. However, these messages may be lost or overwritten over time since dmesg is a circular buffer.
Expose per-path and ctrl sysfs attribute command_error_count, under diag attribute group to provide persistent visibility into error occurrences. This allows users to observe the total number of commands that have failed on a given path over time, which can be useful for diagnosing path health and stability.
This attribute is both readable and writable thus allowing user to reset these counters. These counters can also be consumed by observability tools such as nvme-top to provide additional insight into NVMe error behavior.
--- drivers/nvme/host/core.c | 10 +++++- drivers/nvme/host/nvme.h | 2 ++ drivers/nvme/host/sysfs.c | 66 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 77 insertions(+), 1 deletion(-)
drivers/nvme/host/core.c +9 -1
@@ -438,11 +438,19 @@ static inline void nvme_end_req_zoned(struct request *req)static inline void __nvme_end_req (struct request *req) { -if (unlikely (nvme_req (req)->status && !(req->rq_flags &RQF_QUIET ))) { +struct nvme_ns *ns = req->q->queuedata; +struct nvme_request *nr =nvme_req (req); + +if (unlikely (nr->status && !(req->rq_flags &RQF_QUIET ))) {if (blk_rq_is_passthrough (req))nvme_log_err_passthru (req);else nvme_log_error (req); + +if (ns) +atomic_long_inc (&ns->errors); +else +atomic_long_inc (&nr->ctrl->errors); }nvme_end_req_zoned (req);nvme_trace_bio_complete (req);
drivers/nvme/host/nvme.h +2 -0
@@ -413,6 +413,7 @@ struct nvme_ctrl {unsigned long ka_last_check_time;struct work_struct fw_act_work;unsigned long events; +atomic_long_t errors;#ifdef CONFIG_NVME_MULTIPATH /* asymmetric namespace access: */
@@ -592,6 +593,7 @@ struct nvme_ns {atomic_long_t failover;#endif atomic_long_t retries; +atomic_long_t errors;struct list_head siblings;struct kref kref;struct nvme_ns_head *head;
drivers/nvme/host/sysfs.c +66 -0
93 lines not on this page · raw message
Signature · 1 line
2.53.0
PATCH v4 5/8 nvme: export I/O requeue count when no path is usable via sysfs
In reply to Nilay ShroffWhen the NVMe namespace head determines that there is no currently available path to handle I/O (for example, while a controller is resetting/connecting or due to a transient link failure), incoming I/Os are added to the requeue list.
Currently, there is no visibility into how many I/Os have been requeued in this situation. Add a new ns-head sysfs counter io_requeue_no_usable_path_count, under diag attribute group to expose the number of I/Os that were requeued due to the absence of an available path. This counter is also writable thus allowing user to reset it, if needed.
This statistic can help users understand I/O slowdowns or stalls caused by temporary path unavailability, and can be consumed by monitoring tools such as nvme-top for real-time observability.
--- drivers/nvme/host/multipath.c | 30 ++++++++++++++++++++++++++++++ drivers/nvme/host/nvme.h | 2 ++ drivers/nvme/host/sysfs.c | 5 +++++ 3 files changed, 37 insertions(+)
drivers/nvme/host/multipath.c +30 -0
@@ -522,6 +522,7 @@ static void nvme_ns_head_submit_bio(struct bio *bio)spin_lock_irq (&head->requeue_lock);bio_list_add (&head->requeue_list, bio);spin_unlock_irq (&head->requeue_lock); +atomic_long_inc (&head->io_requeue_no_usable_path_count); }else {dev_warn_ratelimited (dev,"no available path - failing I/O\n" );
@@ -1178,6 +1179,35 @@ static ssize_t multipath_failover_count_store(struct device *dev,DEVICE_ATTR_RW (multipath_failover_count); +static ssize_t io_requeue_no_usable_path_count_show (struct device *dev, +struct device_attribute *attr,char *buf) +{ +struct gendisk *disk =dev_to_disk (dev); +struct nvme_ns_head *head = disk->private_data; + +return sysfs_emit (buf,"%lu\n" , +atomic_long_read (&head->io_requeue_no_usable_path_count)); +} + +static ssize_t io_requeue_no_usable_path_count_store (struct device *dev, +struct device_attribute *attr,const char *buf,size_t count) +{ +int err; +unsigned long requeue_cnt; +struct gendisk *disk =dev_to_disk (dev); +struct nvme_ns_head *head = disk->private_data; + + err =kstrtoul (buf,0 , &requeue_cnt); +if (err) +return -EINVAL ; + +atomic_long_set (&head->io_requeue_no_usable_path_count, requeue_cnt); + +return count; +} + +DEVICE_ATTR_RW (io_requeue_no_usable_path_count); +static int nvme_lookup_ana_group_desc (struct nvme_ctrl *ctrl,struct nvme_ana_group_desc *desc,void *data) {
drivers/nvme/host/nvme.h +2 -0
@@ -564,6 +564,7 @@ struct nvme_ns_head {unsigned long flags;struct delayed_work remove_work;unsigned int delayed_removal_secs; +atomic_long_t io_requeue_no_usable_path_count;#define NVME_NSHEAD_DISK_LIVE 0#define NVME_NSHEAD_QUEUE_IF_NO_PATH 1struct nvme_ns __rcu *current_path[];
@@ -1067,6 +1068,7 @@ extern struct device_attribute dev_attr_queue_depth;extern struct device_attribute dev_attr_numa_nodes;extern struct device_attribute dev_attr_delayed_removal_secs;extern struct device_attribute dev_attr_multipath_failover_count; +extern struct device_attribute dev_attr_io_requeue_no_usable_path_count;extern struct device_attribute subsys_attr_iopolicy;static inline bool nvme_disk_is_ns_head (struct gendisk *disk)
drivers/nvme/host/sysfs.c +5 -0
@@ -410,6 +410,7 @@ static struct attribute *nvme_ns_diag_attrs[] = { &dev_attr_io_errors.attr, #ifdef CONFIG_NVME_MULTIPATH &dev_attr_multipath_failover_count.attr, + &dev_attr_io_requeue_no_usable_path_count.attr, #endif NULL, };
@@ -434,6 +435,10 @@ static umode_t nvme_ns_diag_attrs_are_visible(struct kobject *kobj,if (nvme_disk_is_ns_head (dev_to_disk (dev))) return 0; } +if (a == &dev_attr_io_requeue_no_usable_path_count.attr) { +if (!nvme_disk_is_ns_head (dev_to_disk (dev))) +return 0 ; + }#endif return a->mode; }
Signature · 1 line
2.53.0
PATCH v4 6/8 nvme: export I/O failure count when no path is available via sysfs
In reply to Nilay ShroffWhen I/O is submitted to the NVMe namespace head and no available path can handle the request, the driver fails the I/O immediately. Currently, such failures are only reported via kernel log messages, which may be lost over time since dmesg is a circular buffer.
Add a new ns-head sysfs counter io_fail_no_available_path_count, under diag attribute group to expose the number of I/Os that failed due to the absence of an available path. This provides persistent visibility into path-related I/O failures and can help users diagnose the cause of I/O errors. This counter is also writable and so user may reset its value, if needed.
This counter can also be consumed by monitoring tools such as nvme-top.
--- drivers/nvme/host/multipath.c | 30 ++++++++++++++++++++++++++++++ drivers/nvme/host/nvme.h | 2 ++ drivers/nvme/host/sysfs.c | 5 +++++ 3 files changed, 37 insertions(+)
drivers/nvme/host/multipath.c +30 -0
@@ -527,6 +527,7 @@ static void nvme_ns_head_submit_bio(struct bio *bio)dev_warn_ratelimited (dev,"no available path - failing I/O\n" );bio_io_error (bio); +atomic_long_inc (&head->io_fail_no_available_path_count); }srcu_read_unlock (&head->srcu, srcu_idx);
@@ -1208,6 +1209,35 @@ static ssize_t io_requeue_no_usable_path_count_store(struct device *dev,DEVICE_ATTR_RW (io_requeue_no_usable_path_count); +static ssize_t io_fail_no_available_path_count_show (struct device *dev, +struct device_attribute *attr,char *buf) +{ +struct gendisk *disk =dev_to_disk (dev); +struct nvme_ns_head *head = disk->private_data; + +return sysfs_emit (buf,"%lu\n" , +atomic_long_read (&head->io_fail_no_available_path_count)); +} + +static ssize_t io_fail_no_available_path_count_store (struct device *dev, +struct device_attribute *attr,const char *buf,size_t count) +{ +int err; +unsigned long fail_cnt; +struct gendisk *disk =dev_to_disk (dev); +struct nvme_ns_head *head = disk->private_data; + + err =kstrtoul (buf,0 , &fail_cnt); +if (err) +return -EINVAL ; + +atomic_long_set (&head->io_fail_no_available_path_count, fail_cnt); + +return count; +} + +DEVICE_ATTR_RW (io_fail_no_available_path_count); +static int nvme_lookup_ana_group_desc (struct nvme_ctrl *ctrl,struct nvme_ana_group_desc *desc,void *data) {
drivers/nvme/host/nvme.h +2 -0
@@ -565,6 +565,7 @@ struct nvme_ns_head {struct delayed_work remove_work;unsigned int delayed_removal_secs;atomic_long_t io_requeue_no_usable_path_count; +atomic_long_t io_fail_no_available_path_count;#define NVME_NSHEAD_DISK_LIVE 0#define NVME_NSHEAD_QUEUE_IF_NO_PATH 1struct nvme_ns __rcu *current_path[];
@@ -1069,6 +1070,7 @@ extern struct device_attribute dev_attr_numa_nodes;extern struct device_attribute dev_attr_delayed_removal_secs;extern struct device_attribute dev_attr_multipath_failover_count;extern struct device_attribute dev_attr_io_requeue_no_usable_path_count; +extern struct device_attribute dev_attr_io_fail_no_available_path_count;extern struct device_attribute subsys_attr_iopolicy;static inline bool nvme_disk_is_ns_head (struct gendisk *disk)
drivers/nvme/host/sysfs.c +5 -0
@@ -411,6 +411,7 @@ static struct attribute *nvme_ns_diag_attrs[] = {#ifdef CONFIG_NVME_MULTIPATH &dev_attr_multipath_failover_count.attr, &dev_attr_io_requeue_no_usable_path_count.attr, + &dev_attr_io_fail_no_available_path_count.attr, #endif NULL, };
@@ -439,6 +440,10 @@ static umode_t nvme_ns_diag_attrs_are_visible(struct kobject *kobj,if (!nvme_disk_is_ns_head (dev_to_disk (dev))) return 0; } +if (a == &dev_attr_io_fail_no_available_path_count.attr) { +if (!nvme_disk_is_ns_head (dev_to_disk (dev))) +return 0 ; + }#endif return a->mode; }
Signature · 1 line
2.53.0
Re: [PATCHv4 6/8] nvme: export I/O failure count when no path is available via sysfs
Quoted 1 line · Nilay Shroff
+
I'm just looking at this now for the scsi multipath/libmultipath work.
drivers/nvme/host/multipath.c static ssize_t io_requeue_no_usable_path_count_store(struct device *dev, · line 1230
Quoted 9 lines
drivers/nvme/host/multipath.c +9
-0+static ssize_t io_fail_no_available_path_count_store (struct device *dev, +struct device_attribute *attr,const char *buf,size_t count) +{ +int err; +unsigned long fail_cnt; +struct gendisk *disk =dev_to_disk (dev); +struct nvme_ns_head *head = disk->private_data; + + err =kstrtoul (buf,0 , &fail_cnt);
is fail_cnt ever going to be non-zero? I assume that we allow the value to be stored to just reset the counter (to zero).
drivers/nvme/host/multipath.c static ssize_t io_requeue_no_usable_path_count_store(struct device *dev, · line 1239
Quoted 9 lines
drivers/nvme/host/multipath.c +9
-0+if (err) +return -EINVAL ; + +atomic_long_set (&head->io_fail_no_available_path_count, fail_cnt); + +return count; +} + +DEVICE_ATTR_RW (io_fail_no_available_path_count);
Thanks,
John
Re: [PATCHv4 6/8] nvme: export I/O failure count when no path is available via sysfs
Quoted 5 lines · John Garry
Quoted 1 line · Nilay Shroff
+
I'm just looking at this now for the scsi multipath/libmultipath work.
Quoted 11 lines
drivers/nvme/host/multipath.c static ssize_t io_requeue_no_usable_path_count_store(struct device *dev, · line 1230
Quoted 9 lines
drivers/nvme/host/multipath.c +9
-0+static ssize_t io_fail_no_available_path_count_store (struct device *dev, +struct device_attribute *attr,const char *buf,size_t count) +{ +int err; +unsigned long fail_cnt; +struct gendisk *disk =dev_to_disk (dev); +struct nvme_ns_head *head = disk->private_data; + + err =kstrtoul (buf,0 , &fail_cnt);is
fail_cntever going to be non-zero? I assume that we allow the value to be stored to just reset the counter (to zero).
Yes, this is possible when the namespace head has no available path to forward I/O. In that case, the counter can become non-zero for a transient period before the namespace head is deleted. The namespace head is removed once its reference count drops to zero.
The primary purpose of this attribute is to aid debugging. Under normal operation, the counter is expected to be zero most of the time because the namespace head is short-lived after all paths are removed. However, if there is a subtle bug in the namespace head reference counting logic that prevents it from being deleted, a non-zero value would remain visible under:
/sys/block/<nshead-dev>/diag/io_fail_no_available_path_count
And yes this attribute is writable (as all other diag attributes), only to allow users to reset the counter to zero and then observe subsequent events.
Thanks,
--Nilay
Re: [PATCHv4 6/8] nvme: export I/O failure count when no path is available via sysfs
Quoted 8 lines · Nilay Shroff
The primary purpose of this attribute is to aid debugging. Under normal operation, the counter is expected to be zero most of the time because the namespace head is short-lived after all paths are removed. However, if there is a subtle bug in the namespace head reference counting logic that prevents it from being deleted, a non-zero value would remain visible under:
Quoted 5 lines
/sys/block/<nshead-dev>/diag/io_fail_no_available_path_count
And yes this attribute is writable (as all other diag attributes), only to allow users to reset the counter to zero and then observe subsequent events.
It's something minor, but it makes sense to me that we only allow users to reset the value to zero, like:
@@ -1249,7 +1249,7 @@ static ssize_t io_fail_no_available_path_count_store(struct device *dev,
struct nvme_ns_head *head = disk->private_data;
err = kstrtoul(buf, 0, &fail_cnt);
- if (err)
+ if (err || fail_cnt)
return -EINVAL;
atomic_long_set(&head->io_fail_no_available_path_count, fail_cnt);Re: [PATCHv4 6/8] nvme: export I/O failure count when no path is available via sysfs
Quoted 26 lines · John Garry
Quoted 8 lines · Nilay Shroff
The primary purpose of this attribute is to aid debugging. Under normal operation, the counter is expected to be zero most of the time because the namespace head is short-lived after all paths are removed. However, if there is a subtle bug in the namespace head reference counting logic that prevents it from being deleted, a non-zero value would remain visible under:
Quoted 5 lines
/sys/block/<nshead-dev>/diag/io_fail_no_available_path_count
And yes this attribute is writable (as all other diag attributes), only to allow users to reset the counter to zero and then observe subsequent events.
It's something minor, but it makes sense to me that we only allow users to reset the value to zero, like:
(quoted diff) +1
-1@@ -1249,7 +1249,7 @@ static ssize_t io_fail_no_available_path_count_store(struct device *dev, struct nvme_ns_head *head = disk->private_data; err = kstrtoul(buf, 0, &fail_cnt); - if (err) + if (err || fail_cnt) return -EINVAL;
Quoted 2 lines
atomic_long_set(&head->io_fail_no_available_path_count, fail_cnt);
Yes practically, I think no one would want to set these counters value to anything but zero. So I'm okay with the above change.
Thanks,
--Nilay
PATCH v4 7/8 nvme: export controller reset event count via sysfs
In reply to Nilay ShroffThe NVMe controller transitions into the RESETTING state during error recovery, link instability, firmware activation, or when a reset is explicitly triggered by the user.
Expose a per-ctrl sysfs attribute reset_count, under diag attribute group to provide visibility into these RESETTING state transitions. Observing the frequency of reset events can help users identify issues such as PCIe errors or unstable fabric links. This counter is also writable thus allowing user to reset its value, if needed.
This counter can also be consumed by monitoring tools such as nvme-top to improve controller-level observability.
--- drivers/nvme/host/core.c | 1 + drivers/nvme/host/nvme.h | 1 + drivers/nvme/host/sysfs.c | 27 +++++++++++++++++++++++++++ 3 files changed, 29 insertions(+)
drivers/nvme/host/core.c +1 -0
@@ -596,6 +596,7 @@ bool nvme_change_ctrl_state(struct nvme_ctrl *ctrl,case NVME_CTRL_NEW :case NVME_CTRL_LIVE : changed = true; +atomic_long_inc (&ctrl->nr_reset); fallthrough;default :break ;
drivers/nvme/host/nvme.h +1 -0
@@ -414,6 +414,7 @@ struct nvme_ctrl {struct work_struct fw_act_work;unsigned long events;atomic_long_t errors; +atomic_long_t nr_reset;#ifdef CONFIG_NVME_MULTIPATH /* asymmetric namespace access: */
drivers/nvme/host/sysfs.c +27 -0
@@ -1070,8 +1070,35 @@ struct device_attribute dev_attr_adm_errors =__ATTR (command_error_count,0644 , nvme_adm_errors_show, nvme_adm_errors_store); +static ssize_t reset_count_show (struct device *dev, +struct device_attribute *attr,char *buf) +{ +struct nvme_ctrl *ctrl =dev_get_drvdata (dev); + +return sysfs_emit (buf,"%lu\n" ,atomic_long_read (&ctrl->nr_reset)); +} + +static ssize_t reset_count_store (struct device *dev, +struct device_attribute *attr,const char *buf,size_t count) +{ +int err; +unsigned long reset_cnt; +struct nvme_ctrl *ctrl =dev_get_drvdata (dev); + + err =kstrtoul (buf,0 , &reset_cnt); +if (err) +return -EINVAL ; + +atomic_long_set (&ctrl->nr_reset, reset_cnt); + +return count; +} + +static DEVICE_ATTR_RW (reset_count); +static struct attribute *nvme_dev_diag_attrs[] = { &dev_attr_adm_errors.attr, + &dev_attr_reset_count.attr,NULL , };
Signature · 1 line
2.53.0
PATCH v4 8/8 nvme: export controller reconnect event count via sysfs
In reply to Nilay ShroffWhen an NVMe-oF link goes down, the driver attempts to recover the connection by repeatedly reconnecting to the remote controller at configured intervals. A maximum number of reconnect attempts is also configured, after which recovery stops and the controller is removed if the connection cannot be re-established.
The driver maintains a counter, nr_reconnects, which is incremented on each reconnect attempt. However if in case the reconnect is successful then this counter reset to zero. Moreover, currently, this counter is only reported via kernel log messages and is not exposed to userspace. Since dmesg is a circular buffer, this information may be lost over time.
So introduce a new accumulator which accumulates nr_reconnect attempts and also expose this accumulator per-fabric ctrl via a new sysfs attribute reconnect_count, under diag attribute grroup to provide persistent visibility into the number of reconnect attempts made by the host. This information can help users diagnose unstable links or connectivity issues. Furthermore, this sysfs attribute is also writable so user may reset it to zero, if needed.
The reconnect_count can also be consumed by monitoring tools such as nvme-top to improve controller-level observability.
--- drivers/nvme/host/fc.c | 3 +++ drivers/nvme/host/nvme.h | 2 ++ drivers/nvme/host/rdma.c | 2 ++ drivers/nvme/host/sysfs.c | 35 +++++++++++++++++++++++++++++++++++ drivers/nvme/host/tcp.c | 2 ++ 5 files changed, 44 insertions(+)
drivers/nvme/host/fc.c +3 -0
@@ -3148,6 +3148,8 @@ nvme_fc_create_association(struct nvme_fc_ctrl *ctrl) goto out_term_aen_ops; } +/* accumulate reconnect attempts before resetting it to zero */ +atomic_long_add (ctrl->ctrl.nr_reconnects, &ctrl->ctrl.acc_reconnects); ctrl->ctrl.nr_reconnects =0 ;nvme_start_ctrl (&ctrl->ctrl);
@@ -3470,6 +3472,7 @@ nvme_fc_alloc_ctrl(struct device *dev, struct nvmf_ctrl_options *opts, ctrl->ctrl.opts = opts; ctrl->ctrl.nr_reconnects =0 ; +atomic_long_set (&ctrl->ctrl.acc_reconnects,0 );INIT_LIST_HEAD (&ctrl->ctrl_list); ctrl->lport = lport; ctrl->rport = rport;
drivers/nvme/host/nvme.h +2 -0
@@ -456,6 +456,8 @@ struct nvme_ctrl {u16 icdoff;u16 maxcmd;int nr_reconnects; +/* accumulate reconenct attempts, as nr_reconnects can reset to zero */ +atomic_long_t acc_reconnects;unsigned long flags;struct nvmf_ctrl_options *opts;
drivers/nvme/host/rdma.c +2 -0
@@ -1110,6 +1110,8 @@ static void nvme_rdma_reconnect_ctrl_work(struct work_struct *work)dev_info (ctrl->ctrl.device,"Successfully reconnected (%d attempts)\n" , ctrl->ctrl.nr_reconnects); +/* accumulate reconnect attempts before resetting it to zero */ +atomic_long_add (ctrl->ctrl.nr_reconnects, &ctrl->ctrl.acc_reconnects); ctrl->ctrl.nr_reconnects =0 ;return ;
drivers/nvme/host/sysfs.c +35 -0
@@ -1094,17 +1094,52 @@ static ssize_t reset_count_store(struct device *dev,return count; } +static ssize_t reconnect_count_show (struct device *dev, +struct device_attribute *attr,char *buf) +{ +struct nvme_ctrl *ctrl =dev_get_drvdata (dev); + +return sysfs_emit (buf,"%lu\n" , +atomic_long_read (&ctrl->acc_reconnects) + + ctrl->nr_reconnects); +} + +static ssize_t reconnect_count_store (struct device *dev, +struct device_attribute *attr,const char *buf,size_t count) +{ +int err; +unsigned long reconnect_cnt; +struct nvme_ctrl *ctrl =dev_get_drvdata (dev); + + err =kstrtoul (buf,0 , &reconnect_cnt); +if (err) +return -EINVAL ; + +atomic_long_set (&ctrl->acc_reconnects, reconnect_cnt); + +return count; +} + +static DEVICE_ATTR_RW (reconnect_count); +static DEVICE_ATTR_RW (reset_count);static struct attribute *nvme_dev_diag_attrs[] = { &dev_attr_adm_errors.attr, &dev_attr_reset_count.attr, + &dev_attr_reconnect_count.attr,NULL , };static umode_t nvme_dev_diag_attrs_are_visible (struct kobject *kobj,struct attribute *a,int n) { + struct device *dev = container_of(kobj, struct device, kobj); +struct nvme_ctrl *ctrl =dev_get_drvdata (dev); + +if (a == &dev_attr_reconnect_count.attr && !ctrl->opts) +return 0 ; +return a->mode; }
drivers/nvme/host/tcp.c +2 -0
@@ -2475,6 +2475,8 @@ static void nvme_tcp_reconnect_ctrl_work(struct work_struct *work)dev_info (ctrl->device,"Successfully reconnected (attempt %d/%d)\n" , ctrl->nr_reconnects, ctrl->opts->max_reconnects); +/* accumulate reconnect attempts before resetting it to zero */ +atomic_long_add (ctrl->nr_reconnects, &ctrl->acc_reconnects); ctrl->nr_reconnects =0 ;return ;
Signature · 1 line
2.53.0
Hi Keith,
Quoted 34 lines · Nilay Shroff
Hi,
The NVMe driver encounters various events and conditions during normal operation that are either not tracked today or not exposed to userspace via sysfs. Lack of visibility into these events can make it difficult to diagnose subtle issues related to controller behavior, multipath stability, and I/O reliability.
This patchset adds several diagnostic counters that provide improved observability into NVMe behavior. These counters are intended to help users understand events such as transient path unavailability, controller retries/reconnect/reset, failovers, and I/O failures. They can also be consumed by monitoring tools such as nvme-top.
Specifically, this series proposes to export the following counters via sysfs:
- Command retry count
- Multipath failover count
- Command error count
- I/O requeue count
- I/O failure count
- Controller reset event counts
- Controller reconnect countsThe first patch in the series adds a new diag attribute group under per-path, ns-head and ctrl sysfs directories so that all diagnostics counters could be grouped together under diag sub-directory. The subsequent patches in the series adds diagnostics counters listed above.
Please note that this patchset doesn't make any functional change but rather export relevant counters to user space via sysfs.
As usual, feedback/comments/suggestions are welcome!
Quoted 9 lines
Changes from v3:
- To be consistent in naming, all counters are suffixed with_count(Keith Busch)
- The first patch in the series creates new attribute group named diag and all counters are now grouped under this new sysfs attribute group (Keith Busch)
- Counters are defined asatomic_long_tinstead ofsize_t(Keith Busch)
- Removed RB and TB tags due to above changes
Link to v3: https://lore.kernel.org/all/[email protected]/
As discussed during LSFMM, I have incorporated all of your feedback listed in the changelog above in current patchset, except for integrating command_error_count into the generic gendisk statistics.
While working through the implementation, I realized that although some namespace/ path-level error statistics could conceptually fit within generic block-layer accounting, controller-wide error statistics do not naturally map onto a gendisk object. This is because NVMe controllers themselves do not have an associated block device representation.
As a result, controller-scoped telemetry cannot be cleanly integrated into generic gendisk statistics. Therefore, this patchset exports these counters through NVMe-specific sysfs attributes instead. Let me know if this approach looks reasonable.
Thanks,
--Nilay
Quoted 69 lines · Nilay Shroff
Hi,
The NVMe driver encounters various events and conditions during normal operation that are either not tracked today or not exposed to userspace via sysfs. Lack of visibility into these events can make it difficult to diagnose subtle issues related to controller behavior, multipath stability, and I/O reliability.
This patchset adds several diagnostic counters that provide improved observability into NVMe behavior. These counters are intended to help users understand events such as transient path unavailability, controller retries/reconnect/reset, failovers, and I/O failures. They can also be consumed by monitoring tools such as nvme-top.
Specifically, this series proposes to export the following counters via sysfs:
- Command retry count
- Multipath failover count
- Command error count
- I/O requeue count
- I/O failure count
- Controller reset event counts
- Controller reconnect countsThe first patch in the series adds a new diag attribute group under per-path, ns-head and ctrl sysfs directories so that all diagnostics counters could be grouped together under diag sub-directory. The subsequent patches in the series adds diagnostics counters listed above.
Please note that this patchset doesn't make any functional change but rather export relevant counters to user space via sysfs.
As usual, feedback/comments/suggestions are welcome!
Changes from v3:
- To be consistent in naming, all counters are suffixed with_count(Keith Busch)
- The first patch in the series creates new attribute group named diag and all counters are now grouped under this new sysfs attribute group (Keith Busch)
- Counters are defined asatomic_long_tinstead ofsize_t(Keith Busch)
- Removed RB and TB tags due to above changes
Link to v3: https://lore.kernel.org/all/[email protected]/Changes from v2:
- Allow user to write to sysfs attributes so that user could
reset stat counters, if needed (Sagi)
- The controller reconnect counternr_reconnectscould reset
to zero once connection is re-established, so instead of
exposingnr_reconnectscounter via sysfs introduce a new
counter which accumulates the reconnect attempts and export
this accumulated counter via sysfs (Sagi)
Link to v2: https://lore.kernel.org/all/[email protected]/Changes from v1:
- Remove export of stats for admin command rerty count (Keith)
- Usesize_add()to ensure stat counters don't overflow (Keith)
Link to v1: https://lore.kernel.org/all/[email protected]/Nilay Shroff (8):
nvme: add diag attribute group under sysfs
nvme: export command retry count via sysfs
nvme: export multipath failover count via sysfs
nvme: export command error counters via sysfs
nvme: export I/O requeue count when no path is available via sysfs
nvme: export I/O failure count when no path is available via sysfs
nvme: export controller reset event count via sysfs
nvme: export controller reconnect event count via sysfs
Quoted 10 lines
drivers/nvme/host/core.c | 15 ++- drivers/nvme/host/fc.c | 3 + drivers/nvme/host/multipath.c | 87 ++++++++++++++ drivers/nvme/host/nvme.h | 13 +++ drivers/nvme/host/pci.c | 1 + drivers/nvme/host/rdma.c | 2 + drivers/nvme/host/sysfs.c | 214 ++++++++++++++++++++++++++++++++++ drivers/nvme/host/tcp.c | 2 + 8 files changed, 336 insertions(+), 1 deletion(-)
Hello Nilay,
Applied this patch series on top of v7.1-rc5 and boot-tested on ppc64le.
Verified the new NVMe diag sysfs hierarchy and counters exposed by this series.
Validation steps executed:
Read all exported NVMe diag counters: for f in $(find /sys -path '*nvme*diag/*_count' 2>/dev/null); do echo "$f: $(cat "$f")"; done
Reset all writable counters to zero: for f in $(find /sys -path '*nvme*diag/*_count' 2>/dev/null); do echo 0 > "$f" && echo "reset ok $f"; done
Negative test with invalid input: echo abc > /sys/devices/pci0525:48/0525:48:00.0/nvme/nvme0/diag/command_error_count
Observed results:
diag directories were present under:
controller paths, e.g. /sys/devices/.../nvme/nvmeX/diag/
per-path namespace paths, e.g. /sys/devices/.../nvme/nvmeX/nvmeYcZnW/diag/
namespace-head paths, e.g. /sys/devices/virtual/nvme-subsystem/nvme-subsysX/nvmeYnZ/diag/
Controller counters observed:reset_countcommand_error_countreconnect_count on fabrics controllers
# ll /sys/devices/virtual/nvme-fabrics/ctl/nvme7/diag total 0 -rw-r--r--. 1 root root 65536 May 25 03:58 command_error_count -rw-r--r--. 1 root root 65536 May 25 03:58 reconnect_count -rw-r--r--. 1 root root 65536 May 25 03:58 reset_count
# ll /sys/devices/pci052a:58/052a:58:00.0/nvme/nvme2/diag total 0 -rw-r--r--. 1 root root 65536 May 25 03:58 command_error_count -rw-r--r--. 1 root root 65536 May 25 03:58 reset_count
Per-path counters observed:multipath_failover_countcommand_error_countcommand_retries_count
# ll /sys/devices/pci052a:58/052a:58:00.0/nvme/nvme2/nvme2c2n1/diag total 0 -rw-r--r--. 1 root root 65536 May 25 03:58 command_error_count -rw-r--r--. 1 root root 65536 May 25 03:58 command_retries_count -rw-r--r--. 1 root root 65536 May 25 03:58 multipath_failover_count
Namespace-head counters observed:io_fail_no_available_path_countio_requeue_no_usable_path_count
# ll /sys/devices/virtual/nvme-subsystem/nvme-subsys1/nvme1n2/diag total 0 -rw-r--r--. 1 root root 65536 May 25 03:58 io_fail_no_available_path_count -rw-r--r--. 1 root root 65536 May 25 03:58 io_requeue_no_usable_path_count
All reads returned numeric values
All reset writes to 0 succeeded
Invalid text write failed as expected: -bash: echo: write error: Invalid argument.
If it all looks good, please add below tag.
Regards,
Venkat.
Regards,
Venkat.
Keith Busch
maintainerQuoted 8 lines · Nilay Shroff
Hi,
The NVMe driver encounters various events and conditions during normal operation that are either not tracked today or not exposed to userspace via sysfs. Lack of visibility into these events can make it difficult to diagnose subtle issues related to controller behavior, multipath stability, and I/O reliability.
Quoted 5 lines
This patchset adds several diagnostic counters that provide improved observability into NVMe behavior. These counters are intended to help users understand events such as transient path unavailability, controller retries/reconnect/reset, failovers, and I/O failures. They can also be consumed by monitoring tools such as nvme-top.
Thanks, Nilay. This looks good to me. I'll queue it up if I don't hear any objections.
Quoted 14 lines · Nilay Shroff
Hi,
The NVMe driver encounters various events and conditions during normal operation that are either not tracked today or not exposed to userspace via sysfs. Lack of visibility into these events can make it difficult to diagnose subtle issues related to controller behavior, multipath stability, and I/O reliability.
This patchset adds several diagnostic counters that provide improved observability into NVMe behavior. These counters are intended to help users understand events such as transient path unavailability, controller retries/reconnect/reset, failovers, and I/O failures. They can also be consumed by monitoring tools such as nvme-top.
Quoted 9 lines
Specifically, this series proposes to export the following counters via sysfs:
- Command retry count
- Multipath failover count
- Command error count
- I/O requeue count
- I/O failure count
- Controller reset event counts
- Controller reconnect counts
Thanks, applied to nvme-7.2.