nvme-multipath: expose path_state via sysfs
PATCH 1/1 nvme-multipath: expose path_state via sysfs
Add a read-only "path_state" sysfs attribute to each NVMe path namespace device (/sys/class/nvme/nvmeX/nvmeXcYnZ/path_state) that exposes whether the path is currently enabled or disabled.
The attribute reflects the result of nvme_path_is_disabled(), showing "disabled" when the controller is not live/deleting, the namespace has an ANA state change pending, or the namespace is not ready. Otherwise it shows "enabled".
This gives userspace visibility into the multipath path selection state without requiring users to piece together controller state and namespace flags manually.
--- drivers/nvme/host/multipath.c | 10 ++++++++++ drivers/nvme/host/nvme.h | 1 + drivers/nvme/host/sysfs.c | 4 +++- 3 files changed, 14 insertions(+), 1 deletion(-)
drivers/nvme/host/multipath.c +10 -0
@@ -1210,6 +1210,16 @@ static ssize_t in_flight_bytes_show(struct device *dev, }DEVICE_ATTR_RO (in_flight_bytes); +static ssize_t path_state_show (struct device *dev, +struct device_attribute *attr,char *buf) +{ +struct nvme_ns *ns =nvme_get_ns_from_dev (dev); + +return sysfs_emit (buf,"%s\n" , +nvme_path_is_disabled (ns) ?"disabled" :"enabled" ); +} +DEVICE_ATTR_RO (path_state); +static ssize_t relative_throughput_show (struct device *dev,struct device_attribute *attr,char *buf) {
drivers/nvme/host/nvme.h +1 -0
@@ -1066,6 +1066,7 @@ extern struct device_attribute dev_attr_queue_depth;extern struct device_attribute dev_attr_in_flight_bytes;extern struct device_attribute dev_attr_relative_throughput;extern struct device_attribute dev_attr_numa_nodes; +extern struct device_attribute dev_attr_path_state;extern struct device_attribute dev_attr_delayed_removal_secs;extern struct device_attribute subsys_attr_iopolicy;
drivers/nvme/host/sysfs.c +3 -1
@@ -262,6 +262,7 @@ static struct attribute *nvme_ns_attrs[] = { &dev_attr_in_flight_bytes.attr, &dev_attr_relative_throughput.attr, &dev_attr_numa_nodes.attr, + &dev_attr_path_state.attr, &dev_attr_delayed_removal_secs.attr, #endif &dev_attr_io_passthru_err_log_enabled.attr,
@@ -296,7 +297,8 @@ static umode_t nvme_ns_attrs_are_visible(struct kobject *kobj,return 0 ; }if (a == &dev_attr_queue_depth.attr || a == &dev_attr_in_flight_bytes.attr || - a == &dev_attr_relative_throughput.attr || a == &dev_attr_numa_nodes.attr) { + a == &dev_attr_relative_throughput.attr || a == &dev_attr_numa_nodes.attr || + a == &dev_attr_path_state.attr) {if (nvme_disk_is_ns_head (dev_to_disk (dev)))return 0 ; }
Signature · 1 line
2.43.7
Quoted 4 lines · Guixin Liu
Add a read-only "
path_state" sysfs attribute to each NVMe path namespace device (/sys/class/nvme/nvmeX/nvmeXcYnZ/path_state) that exposes whether the path is currently enabled or disabled.
Quoted 8 lines
The attribute reflects the result of
nvme_path_is_disabled(), showing "disabled" when the controller is not live/deleting, the namespace has an ANA state change pending, or the namespace is not ready. Otherwise it shows "enabled".This gives userspace visibility into the multipath path selection state without requiring users to piece together controller state and namespace flags manually.
The ctrl state is already exposed to the sysfs. What's the use case for this sysfs entry?
Quoted 10 lines · Daniel Wagner
Quoted 9 lines · Guixin Liu
Add a read-only "
path_state" sysfs attribute to each NVMe path namespace device (/sys/class/nvme/nvmeX/nvmeXcYnZ/path_state) that exposes whether the path is currently enabled or disabled.The attribute reflects the result of
nvme_path_is_disabled(), showing "disabled" when the controller is not live/deleting, the namespace has an ANA state change pending, or the namespace is not ready. Otherwise it shows "enabled".
Quoted 5 lines
Quoted 3 lines
This gives userspace visibility into the multipath path selection state without requiring users to piece together controller state and namespace flags manually.
The ctrl state is already exposed to the sysfs. What's the use case for this sysfs entry?
1. A controller being live only indicates that the transport connection is healthy. However, due to ANA, a path through that controller may be inaccessible or non-optimized. The ctrl state simply cannot express this.
2. Different namespaces on the same controller can belong to different ANA groups with different access states. For example, ns1 may be optimized while ns2 is inaccessible on the same controller. A single ctrl state value cannot represent this granularity.
3. The multipath iopolicy (numa, round-robin, queue-depth, service-time) makes routing decisions based on path state. Without visibility into path state, users cannot understand or predict why I/O is routed to a particular path, making performance debugging a black box.
4. For monitoring and diagnosis, When I/O latency spikes in production, operators need
to determine within seconds: Did an ANA failover occur? Which path became inaccessible? Is a non-optimized path now carrying traffic? The ctrl state just shows live and cannot answer any of these questions.
Best Regards,
Guixin Liu
Quoted 7 lines · Guixin Liu
Quoted 2 lines
The ctrl state is already exposed to the sysfs. What's the use case for this sysfs entry?
1. A controller being live only indicates that the transport connection is healthy.
However, due to ANA, a path through that controller may be inaccessible or non-optimized.
The ctrl state simply cannot express this.
Ah, I thought this info is already in the sysfs but it's just the ns head one. Though I think it would be better to expose the complete state not the filtered one via nvme_path_is_disabled().
Quoted 3 lines · Daniel Wagner
Ah, I thought this info is already in the sysfs but it's just the ns head one. Though I think it would be better to expose the complete state not the filtered one via
nvme_path_is_disabled().
Yes, Nilay suggested that we can expose the disable reason too.
I will change this in v2.
Best Regards,
Guixin Liu
Quoted 28 lines · Guixin Liu
Add a read-only "
path_state" sysfs attribute to each NVMe path namespace device (/sys/class/nvme/nvmeX/nvmeXcYnZ/path_state) that exposes whether the path is currently enabled or disabled.The attribute reflects the result of
nvme_path_is_disabled(), showing "disabled" when the controller is not live/deleting, the namespace has an ANA state change pending, or the namespace is not ready. Otherwise it shows "enabled".This gives userspace visibility into the multipath path selection state without requiring users to piece together controller state and namespace flags manually.
Signed-off-by Guixin Liu<[email protected]>--- drivers/nvme/host/multipath.c | 10 ++++++++++ drivers/nvme/host/nvme.h | 1 + drivers/nvme/host/sysfs.c | 4 +++- 3 files changed, 14 insertions(+), 1 deletion(-)drivers/nvme/host/multipath.c +0
-0@@ -1210,6 +1210,16 @@ static ssize_t in_flight_bytes_show(struct device *dev, }DEVICE_ATTR_RO (in_flight_bytes);
drivers/nvme/host/multipath.c static ssize_t in_flight_bytes_show(struct device *dev, · line 1222
Quoted 10 lines
drivers/nvme/host/multipath.c +10
-0+static ssize_t path_state_show (struct device *dev, +struct device_attribute *attr,char *buf) +{ +struct nvme_ns *ns =nvme_get_ns_from_dev (dev); + +return sysfs_emit (buf,"%s\n" , +nvme_path_is_disabled (ns) ?"disabled" :"enabled" ); +} +DEVICE_ATTR_RO (path_state); +
Instead of only exposing disabled/enabled state, can we also expose the reason why the path was not usable?
For instance, path_state attribute shall expose "enabled" when it's available for I/O (this is straightforward):
- enabled
But in case path is not usable/available then expose "disabled" along with the reason:
- disabled (ctrl_down)
- disabled (ana_pending)
- disabled (ns_not_ready)
Or any other combination depending on the ns->flags and ctrl->state.
That way, the userspace would know the exact reason when path is not available/usable for I/O.
Thanks,
--Nilay
Quoted 14 lines
Instead of only exposing disabled/enabled state, can we also expose the reason why the path was not usable?
For instance,
path_stateattribute shall expose "enabled" when it's available for I/O (this is straightforward):
- enabledBut in case path is not usable/available then expose "disabled" along with the reason:
- disabled (ctrl_down)
- disabled (ana_pending)
- disabled (ns_not_ready)
Quoted 7 lines
Or any other combination depending on the
ns->flagsandctrl->state.That way, the userspace would know the exact reason when path is not available/usable for I/O.
Thanks,
--Nilay
Good idea, I will add this in v2, thanks.
Best Regards,
Guixin Liu