Skip to contentSkip to conversation

nvme-multipath: expose path_state via sysfs

PATCH Superseded Guixin Liu 3 participants 7 messages Jun 17 – Jun 22 NVMe

v1 of 5 Superseded Latest is v5 →

lore
Share & tools
lore

Guixin Liu

author
#0 lore raw

PATCH 1/1 nvme-multipath: expose path_state via sysfs

Add a read-only "path_state" sysfs attribute to each NVMe path namespace device (/sys/class/nvme/nvmeX/nvmeXcYnZ/path_state) that exposes whether the path is currently enabled or disabled.

The attribute reflects the result of nvme_path_is_disabled(), showing "disabled" when the controller is not live/deleting, the namespace has an ANA state change pending, or the namespace is not ready. Otherwise it shows "enabled".

This gives userspace visibility into the multipath path selection state without requiring users to piece together controller state and namespace flags manually.

---
 drivers/nvme/host/multipath.c | 10 ++++++++++
 drivers/nvme/host/nvme.h      |  1 +
 drivers/nvme/host/sysfs.c     |  4 +++-
 3 files changed, 14 insertions(+), 1 deletion(-)
drivers/nvme/host/multipath.c +10 -0
index 81fff2f20d23..65bd3dd188ee 100644
@@ -1210,6 +1210,16 @@ static ssize_t in_flight_bytes_show(struct device *dev,
 }
 DEVICE_ATTR_RO(in_flight_bytes);
 
+static ssize_t path_state_show(struct device *dev,
+		struct device_attribute *attr, char *buf)
+{
+	struct nvme_ns *ns = nvme_get_ns_from_dev(dev);
+
+	return sysfs_emit(buf, "%s\n",
+			  nvme_path_is_disabled(ns) ? "disabled" : "enabled");
+}
+DEVICE_ATTR_RO(path_state);
+
 static ssize_t relative_throughput_show(struct device *dev,
 		struct device_attribute *attr, char *buf)
 {
drivers/nvme/host/nvme.h +1 -0
index 2b2627e0d3ce..bb9b8cdf973a 100644
@@ -1066,6 +1066,7 @@ extern struct device_attribute dev_attr_queue_depth;
 extern struct device_attribute dev_attr_in_flight_bytes;
 extern struct device_attribute dev_attr_relative_throughput;
 extern struct device_attribute dev_attr_numa_nodes;
+extern struct device_attribute dev_attr_path_state;
 extern struct device_attribute dev_attr_delayed_removal_secs;
 extern struct device_attribute subsys_attr_iopolicy;
 
drivers/nvme/host/sysfs.c +3 -1
index 6309af224c93..5110a3bf5279 100644
@@ -262,6 +262,7 @@ static struct attribute *nvme_ns_attrs[] = {
 	&dev_attr_in_flight_bytes.attr,
 	&dev_attr_relative_throughput.attr,
 	&dev_attr_numa_nodes.attr,
+	&dev_attr_path_state.attr,
 	&dev_attr_delayed_removal_secs.attr,
 #endif
 	&dev_attr_io_passthru_err_log_enabled.attr,
@@ -296,7 +297,8 @@ static umode_t nvme_ns_attrs_are_visible(struct kobject *kobj,
 			return 0;
 	}
 	if (a == &dev_attr_queue_depth.attr || a == &dev_attr_in_flight_bytes.attr ||
-	    a == &dev_attr_relative_throughput.attr || a == &dev_attr_numa_nodes.attr) {
+	    a == &dev_attr_relative_throughput.attr || a == &dev_attr_numa_nodes.attr ||
+	    a == &dev_attr_path_state.attr) {
 		if (nvme_disk_is_ns_head(dev_to_disk(dev)))
 			return 0;
 	}
Signature · 1 line
2.43.7
#1 lore raw
Quoted 4 lines · Guixin Liu

Add a read-only "path_state" sysfs attribute to each NVMe path namespace device (/sys/class/nvme/nvmeX/nvmeXcYnZ/path_state) that exposes whether the path is currently enabled or disabled.

Quoted 8 lines

The attribute reflects the result of nvme_path_is_disabled(), showing "disabled" when the controller is not live/deleting, the namespace has an ANA state change pending, or the namespace is not ready. Otherwise it shows "enabled".

This gives userspace visibility into the multipath path selection state without requiring users to piece together controller state and namespace flags manually.

The ctrl state is already exposed to the sysfs. What's the use case for this sysfs entry?

author
#2 lore raw
Quoted 10 lines · Daniel Wagner
Quoted 9 lines · Guixin Liu

Add a read-only "path_state" sysfs attribute to each NVMe path namespace device (/sys/class/nvme/nvmeX/nvmeXcYnZ/path_state) that exposes whether the path is currently enabled or disabled.

The attribute reflects the result of nvme_path_is_disabled(), showing "disabled" when the controller is not live/deleting, the namespace has an ANA state change pending, or the namespace is not ready. Otherwise it shows "enabled".

Quoted 5 lines
Quoted 3 lines

This gives userspace visibility into the multipath path selection state without requiring users to piece together controller state and namespace flags manually.

The ctrl state is already exposed to the sysfs. What's the use case for this sysfs entry?

1. A controller being live only indicates that the transport connection is healthy. However, due to ANA, a path through that controller may be inaccessible or non-optimized. The ctrl state simply cannot express this.

2. Different namespaces on the same controller can belong to different ANA groups with different access states. For example, ns1 may be optimized while ns2 is inaccessible on the same controller. A single ctrl state value cannot represent this granularity.

3. The multipath iopolicy (numa, round-robin, queue-depth, service-time) makes routing decisions based on path state. Without visibility into path state, users cannot understand or predict why I/O is routed to a particular path, making performance debugging a black box.

4. For monitoring and diagnosis, When I/O latency spikes in production, operators need
to determine within seconds: Did an ANA failover occur? Which path became inaccessible? Is a non-optimized path now carrying traffic? The ctrl state just shows live and cannot answer any of these questions.

Best Regards,
Guixin Liu

#3 lore raw
Quoted 7 lines · Guixin Liu
Quoted 2 lines

The ctrl state is already exposed to the sysfs. What's the use case for this sysfs entry?

1. A controller being live only indicates that the transport connection is healthy.
However, due to ANA, a path through that controller may be inaccessible or non-optimized.
The ctrl state simply cannot express this.

Ah, I thought this info is already in the sysfs but it's just the ns head one. Though I think it would be better to expose the complete state not the filtered one via nvme_path_is_disabled().

author
#4 lore raw
Quoted 3 lines · Daniel Wagner

Ah, I thought this info is already in the sysfs but it's just the ns head one. Though I think it would be better to expose the complete state not the filtered one via nvme_path_is_disabled().

Yes, Nilay suggested that we can expose the disable reason too.

I will change this in v2.

Best Regards,
Guixin Liu

#5 lore raw
In reply to Guixin Liu
Quoted 28 lines · Guixin Liu

Add a read-only "path_state" sysfs attribute to each NVMe path namespace device (/sys/class/nvme/nvmeX/nvmeXcYnZ/path_state) that exposes whether the path is currently enabled or disabled.

The attribute reflects the result of nvme_path_is_disabled(), showing "disabled" when the controller is not live/deleting, the namespace has an ANA state change pending, or the namespace is not ready. Otherwise it shows "enabled".

This gives userspace visibility into the multipath path selection state without requiring users to piece together controller state and namespace flags manually.

---
  drivers/nvme/host/multipath.c | 10 ++++++++++
  drivers/nvme/host/nvme.h      |  1 +
  drivers/nvme/host/sysfs.c     |  4 +++-
  3 files changed, 14 insertions(+), 1 deletion(-)
drivers/nvme/host/multipath.c +0 -0
index 81fff2f20d23..65bd3dd188ee 100644
@@ -1210,6 +1210,16 @@ static ssize_t in_flight_bytes_show(struct device *dev,
  }
  DEVICE_ATTR_RO(in_flight_bytes);
 

drivers/nvme/host/multipath.c static ssize_t in_flight_bytes_show(struct device *dev, · line 1222

Quoted 10 lines
drivers/nvme/host/multipath.c +10 -0
+static ssize_t path_state_show(struct device *dev,
+		struct device_attribute *attr, char *buf)
+{
+	struct nvme_ns *ns = nvme_get_ns_from_dev(dev);
+
+	return sysfs_emit(buf, "%s\n",
+			  nvme_path_is_disabled(ns) ? "disabled" : "enabled");
+}
+DEVICE_ATTR_RO(path_state);
+

Instead of only exposing disabled/enabled state, can we also expose the reason why the path was not usable?

For instance, path_state attribute shall expose "enabled" when it's available for I/O (this is straightforward):
- enabled

But in case path is not usable/available then expose "disabled" along with the reason:
- disabled (ctrl_down)
- disabled (ana_pending)
- disabled (ns_not_ready)

Or any other combination depending on the ns->flags and ctrl->state.

That way, the userspace would know the exact reason when path is not available/usable for I/O.

Thanks,
--Nilay

author
#6 lore raw
Quoted 14 lines

Instead of only exposing disabled/enabled state, can we also expose the reason why the path was not usable?

For instance, path_state attribute shall expose "enabled" when it's available for I/O (this is straightforward):
- enabled

But in case path is not usable/available then expose "disabled" along with the reason:
- disabled (ctrl_down)
- disabled (ana_pending)
- disabled (ns_not_ready)

Quoted 7 lines

Or any other combination depending on the ns->flags and ctrl->state.

That way, the userspace would know the exact reason when path is not available/usable for I/O.

Thanks,
--Nilay

Good idea, I will add this in v2, thanks.

Best Regards,
Guixin Liu