Skip to contentSkip to conversation

Add visibility for native NVMe multipath using sysfs

RFC PATCH Superseded Nilay Shroff 2 participants 5 messages Sep 3, 2024 – Sep 10, 2024 NVMe

v3 of 7 Superseded Latest is v7 →

lore
Share & tools
lore
#0 lore raw

v3 cover Add visibility for native NVMe multipath using sysfs

Hi,

This patch propose adding new sysfs attributes for adding visibility of native multipath I/O.

The first version of this RFC[1] proposed using debugfs for visibility however the general feedback was to instead export the multipath I/O information using sysfs attributes and then latter parse and format those sysfs attributes using libnvme/nvme-cli.

The second version of this RFC[2] uses sysfs however the sysfs attribute file contains multiple lines of output and the feedback was to instead follow the principal of one value per one attribute.

So now third version of this RFC follows the one value per one attrbiute principal.

As we know, NVMe native multipath supports three different io policies (numa, round-robin and queue-depth) for selecting I/O path, however, we don't have any visibility about which path is being selected by multipath code for forwarding I/O. This RFC helps add that visibility by adding new sysfs attribute files named "numa_nodes" and "queue_depth" under each namespace block device path /sys/block/nvmeXcYnZ/. We also create a "multipath" sysfs directory under head disk node and then from this directory add a link to each namespace path device this head disk node points to.

Please find below output generated with this proposed RFC patch applied on a system with two multi-controller PCIe NVMe disks attached to it. This system is also an NVMf-TCP host which is connected to an NVMf-TCP target over two NIC cards. This system has four numa nodes online when the below output was captured:

# cat /sys/devices/system/node/online 
0-3
# lscpu
<snip>
NUMA:
  NUMA node(s):           4
  NUMA node0 CPU(s):
  NUMA node1 CPU(s):      0-7
  NUMA node2 CPU(s):      8-31
  NUMA node3 CPU(s):      32-63
<snip>

Please note that numa node 0 though online, doesn't have any CPU currently assigned to it.

# nvme list -v 
Subsystem        Subsystem-NQN                                                                                    Controllers
---------------- ------------------------------------------------------------------------------------------------ ----------------
nvme-subsys1     nqn.1994-11.com.samsung:nvme:PM1735a:2.5-inch:S6RTNE0R900057                                     nvme0, nvme1
nvme-subsys3     nqn.2019-10.com.kioxia:KCM7DRUG1T92:3D60A04906N1                                                 nvme2, nvme3
nvme-subsys4     nvmet_subsystem                                                                                  nvme4, nvme5

Device Cntlid SN MN FR TxPort Address Slot Subsystem Namespaces
---------------- ------ -------------------- ---------------------------------------- -------- ------ -------------- ------ ------------ ----------------

nvme0    66     S6RTNE0R900057       3.2TB NVMe Gen4 U.2 SSD III              REV.SN66 pcie   052e:78:00.0   U50EE.001.WZS000E-P3-C14-R1 nvme-subsys1 nvme1n1
nvme1    65     S6RTNE0R900057       3.2TB NVMe Gen4 U.2 SSD III              REV.SN66 pcie   058e:78:00.0   U50EE.001.WZS000E-P3-C14-R2 nvme-subsys1 nvme1n1
nvme2    2      3D60A04906N1         1.6TB NVMe Gen4 U.2 SSD IV               REV.CAS2 pcie   0524:28:00.0   U50EE.001.WZS000E-P3-C4-R1 nvme-subsys3 nvme3n1
nvme3    1      3D60A04906N1         1.6TB NVMe Gen4 U.2 SSD IV               REV.CAS2 pcie   0584:28:00.0   U50EE.001.WZS000E-P3-C4-R2 nvme-subsys3 nvme3n1
nvme4    1      a224673364d1dcb6fab9 Linux                                    6.9.0-rc tcp    traddr=20.0.0.200,trsvcid=4420,src_addr=20.0.0.100        nvme-subsys4 nvme4n1
nvme5    2      a224673364d1dcb6fab9 Linux                                    6.9.0-rc tcp    traddr=10.0.0.200,trsvcid=4420,src_addr=10.0.0.100        nvme-subsys4 nvme4n1

Device Generic NSID Usage Format Controllers
----------------- ----------------- ---------- -------------------------- ---------------- ----------------

/dev/nvme1n1 /dev/ng1n1   0x1          5.75  GB /   5.75  GB      4 KiB +  0 B   nvme0, nvme1
/dev/nvme3n1 /dev/ng3n1   0x2          0.00   B /   5.75  GB      4 KiB +  0 B   nvme2, nvme3
/dev/nvme4n1 /dev/ng4n1   0x1          5.75  GB /   5.75  GB      4 KiB +  0 B   nvme4, nvme5
# nvme show-topology
nvme-subsys1 - NQN=nqn.1994-11.com.samsung:nvme:PM1735a:2.5-inch:S6RTNE0R900057
               hostnqn=nqn.2014-08.org.nvmexpress:uuid:41528538-e8ad-4eaf-84a7-9c552917d988
               iopolicy=numa
\
 +- ns 1
 \
  +- nvme0 pcie 052e:78:00.0 live optimized
  +- nvme1 pcie 058e:78:00.0 live optimized

nvme-subsys3 - NQN=nqn.2019-10.com.kioxia:KCM7DRUG1T92:3D60A04906N1

               hostnqn=nqn.2014-08.org.nvmexpress:uuid:41528538-e8ad-4eaf-84a7-9c552917d988
               iopolicy=round-robin
\
 +- ns 2
 \
  +- nvme2 pcie 0524:28:00.0 live optimized
  +- nvme3 pcie 0584:28:00.0 live optimized

nvme-subsys4 - NQN=nvmet_subsystem

               hostnqn=nqn.2014-08.org.nvmexpress:uuid:41528538-e8ad-4eaf-84a7-9c552917d988
               iopolicy=queue-depth
\
 +- ns 1
 \
  +- nvme4 tcp traddr=20.0.0.200,trsvcid=4420,src_addr=20.0.0.100 live optimized
  +- nvme5 tcp traddr=10.0.0.200,trsvcid=4420,src_addr=10.0.0.100 live optimized

As we could see above, we've three shared namespaces created. In terms of iopolicy, we have "numa" configured for nvme-subsys1, "round-robin" configured for nvme-subsys3 and "queue-depth" configured for nvme-subsys4.

Now, under each namespace "head disk node", we create a sysfs group attribute named "multipath". The "multipath" group then points to the each path this head disk node points to:

# tree /sys/block/nvme1n1/multipath/
/sys/block/nvme1n1/multipath/
├── nvme1c0n1 -> ../../../../../pci052e:78/052e:78:00.0/nvme/nvme0/nvme1c0n1
└── nvme1c1n1 -> ../../../../../pci058e:78/058e:78:00.0/nvme/nvme1/nvme1c1n1
# tree /sys/block/nvme3n1/multipath/
/sys/block/nvme3n1/multipath/
├── nvme3c2n1 -> ../../../../../pci0524:28/0524:28:00.0/nvme/nvme2/nvme3c2n1
└── nvme3c3n1 -> ../../../../../pci0584:28/0584:28:00.0/nvme/nvme3/nvme3c3n1
# tree /sys/block/nvme4n1/multipath/
/sys/block/nvme4n1/multipath/
├── nvme4c4n1 -> ../../../../nvme-fabrics/ctl/nvme4/nvme4c4n1
└── nvme4c5n1 -> ../../../../nvme-fabrics/ctl/nvme5/nvme4c5n1

One can easily infer from the above output that for the "round-robin" I/O policy, configured under nvme-subsys3, the I/O workload targeted at nvme3n1 would toggle across nvme3c2n1 and nvme3c3n1 assuming the ana state of each path is optimized (as can be seen in the output of show-topology).

For numa I/O policy, configured under nvme-subsys1, the "numa_nodes" attribute file shows the numa nodes being preferred by the respective namespace path. The numa nodes value is comma delimited list of nodes or A-B range of nodes.

# cat  /sys/block/nvme1n1/multipath/nvme1c0n1/numa_nodes 
0-1
# cat  /sys/block/nvme1n1/multipath/nvme1c1n1/numa_nodes
2-3

From the above output, one can easily infer that I/O workload targeted at nvme1n1 and running on numa nodes 0 and 1 would use path nvme1c0n1. Similarly, I/O workload running on numa nodes 2 and 3 would use path nvme1c1n1.

For queue-depth I/O policy, configured under nvme-subsys4, the "queue_depth" attribute file shows the number of active/in-flight I/O requests currently queued for each path.

# cat  /sys/block/nvme4n1/multipath/nvme4c4n1/queue_depth 
518
# cat  /sys/block/nvme4n1/multipath/nvme4c5n1/queue_depth
504

From the above output, one can easily infer that I/O workload targeted at nvme4n1 uses two paths nvme4c4n1 and nvme4c5n1 and the current queue depth of each path is 518 and 504 respectively.

[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/

Changes since v2:
- Use one value per one sysfs attribute (Keith Busch)

Changes since v1:
- Use sysfs to export multipath I/O information instead of debugfs

Nilay Shroff (1):
nvme-multipath: Add sysfs attributes for showing multipath info

 drivers/nvme/host/core.c      |  3 ++
 drivers/nvme/host/multipath.c | 69 +++++++++++++++++++++++++++++++++++
 drivers/nvme/host/nvme.h      | 20 ++++++++--
 drivers/nvme/host/sysfs.c     | 20 ++++++++++
 4 files changed, 108 insertions(+), 4 deletions(-)
Signature · 1 line
2.45.2
#1 lore raw

RFC PATCH v3 1/1 nvme-multipath: Add sysfs attributes for showing multipath info

NVMe native multipath supports different IO policies for selecting I/O path, however we don't have any visibility about which path is being selected by multipath code for forwarding I/O.
This patch helps add that visibility by adding new sysfs attribute files named "numa_nodes" and "queue_depth" under each namespace block device path /sys/block/nvmeXcYnZ/. We also create a "multipath" sysfs directory under head disk node and then from this directory add a link to each namespace path device this head disk node points to.

For instance, /sys/block/nvmeXnY/multipath/ would create a soft link to each path the head disk node <nvmeXnY> points to:

$ ls -1 /sys/block/nvme1n1/
nvme1c1n1 -> ../../../../../pci052e:78/052e:78:00.0/nvme/nvme1/nvme1c1n1
nvme1c3n1 -> ../../../../../pci058e:78/058e:78:00.0/nvme/nvme3/nvme1c3n1

For round-robin I/O policy, we could easily infer from the above output that I/O workload targeted to nvme3n1 would toggle across paths nvme1c1n1 and nvme1c3n1.

For numa I/O policy, the "numa_nodes" attribute file shows the numa nodes being preferred by the respective block device path. The numa nodes value is comma delimited list of nodes or A-B range of nodes.

For queue-depth I/O policy, the "queue_depth" attribute file shows the number of active/in-flight I/O requests currently queued for each path.

---
 drivers/nvme/host/core.c      |  3 ++
 drivers/nvme/host/multipath.c | 69 +++++++++++++++++++++++++++++++++++
 drivers/nvme/host/nvme.h      | 20 ++++++++--
 drivers/nvme/host/sysfs.c     | 20 ++++++++++
 4 files changed, 108 insertions(+), 4 deletions(-)
drivers/nvme/host/core.c +3 -0
index 0dc8bcc664f2..e48a317d70b6 100644
@@ -3951,6 +3951,9 @@ static void nvme_ns_remove(struct nvme_ns *ns)
 
 	if (!nvme_ns_head_multipath(ns->head))
 		nvme_cdev_del(&ns->cdev, &ns->cdev_device);
+
+	nvme_mpath_remove_sysfs_link(ns);
+
 	del_gendisk(ns->disk);
 
 	mutex_lock(&ns->ctrl->namespaces_lock);
drivers/nvme/host/multipath.c +69 -0

87 lines not on this page · raw message

drivers/nvme/host/nvme.h +16 -4
index da57947130cc..dfd3b23bcc6f 100644
@@ -522,10 +522,11 @@ struct nvme_ns {
 	struct nvme_ns_head *head;
 
 	unsigned long flags;
-#define NVME_NS_REMOVING	0
-#define NVME_NS_ANA_PENDING	2
-#define NVME_NS_FORCE_RO	3
-#define NVME_NS_READY		4
+#define NVME_NS_REMOVING		0
+#define NVME_NS_ANA_PENDING		2
+#define NVME_NS_FORCE_RO		3
+#define NVME_NS_READY			4
+#define NVME_NS_SYSFS_ATTR_LINK	5
 
 	struct cdev		cdev;
 	struct device		cdev_device;
@@ -921,6 +922,7 @@ int nvme_getgeo(struct block_device *bdev, struct hd_geometry *geo);
 int nvme_dev_uring_cmd(struct io_uring_cmd *ioucmd, unsigned int issue_flags);
 
 extern const struct attribute_group *nvme_ns_attr_groups[];
+extern const struct attribute_group nvme_ns_mpath_attr_group;
 extern const struct pr_ops nvme_pr_ops;
 extern const struct block_device_operations nvme_ns_head_ops;
 extern const struct attribute_group nvme_dev_attrs_group;
@@ -943,6 +945,8 @@ void nvme_mpath_default_iopolicy(struct nvme_subsystem *subsys);
 void nvme_failover_req(struct request *req);
 void nvme_kick_requeue_lists(struct nvme_ctrl *ctrl);
 int nvme_mpath_alloc_disk(struct nvme_ctrl *ctrl,struct nvme_ns_head *head);
+void nvme_mpath_add_sysfs_link(struct nvme_ns *ns);
+void nvme_mpath_remove_sysfs_link(struct nvme_ns *ns);
 void nvme_mpath_add_disk(struct nvme_ns *ns, __le32 anagrpid);
 void nvme_mpath_remove_disk(struct nvme_ns_head *head);
 int nvme_mpath_init_identify(struct nvme_ctrl *ctrl, struct nvme_id_ctrl *id);
@@ -968,6 +972,8 @@ static inline void nvme_trace_bio_complete(struct request *req)
 extern bool multipath;
 extern struct device_attribute dev_attr_ana_grpid;
 extern struct device_attribute dev_attr_ana_state;
+extern struct device_attribute dev_attr_queue_depth;
+extern struct device_attribute dev_attr_numa_nodes;
 extern struct device_attribute subsys_attr_iopolicy;
 
 static inline bool nvme_disk_is_ns_head(struct gendisk *disk)
@@ -997,6 +1003,12 @@ static inline void nvme_mpath_add_disk(struct nvme_ns *ns, __le32 anagrpid)
 static inline void nvme_mpath_remove_disk(struct nvme_ns_head *head)
 {
 }
+static inline void nvme_mpath_add_sysfs_link(struct nvme_ns *ns)
+{
+}
+static inline void nvme_mpath_remove_sysfs_link(struct nvme_ns *ns)
+{
+}
 static inline bool nvme_mpath_clear_current_path(struct nvme_ns *ns)
 {
 	return false;
drivers/nvme/host/sysfs.c +20 -0
index ba05faaac562..92bcdd4f6072 100644
@@ -258,6 +258,8 @@ static struct attribute *nvme_ns_attrs[] = {
 #ifdef CONFIG_NVME_MULTIPATH
 	&dev_attr_ana_grpid.attr,
 	&dev_attr_ana_state.attr,
+	&dev_attr_queue_depth.attr,
+	&dev_attr_numa_nodes.attr,
 #endif
 	&dev_attr_io_passthru_err_log_enabled.attr,
 	NULL,
@@ -290,6 +292,10 @@ static umode_t nvme_ns_attrs_are_visible(struct kobject *kobj,
 		if (!nvme_ctrl_use_ana(nvme_get_ns_from_dev(dev)->ctrl))
 			return 0;
 	}
+	if (a == &dev_attr_queue_depth.attr || a == &dev_attr_numa_nodes.attr) {
+		if (nvme_disk_is_ns_head(dev_to_disk(dev)))
+			return 0;
+	}
 #endif
 	return a->mode;
 }
@@ -299,8 +305,22 @@ static const struct attribute_group nvme_ns_attr_group = {
 	.is_visible	= nvme_ns_attrs_are_visible,
 };
 
+#ifdef CONFIG_NVME_MULTIPATH
+static struct attribute *nvme_ns_mpath_attrs[] = {
+	NULL,
+};
+
+const struct attribute_group nvme_ns_mpath_attr_group = {
+	.name           = "multipath",
+	.attrs		= nvme_ns_mpath_attrs,
+};
+#endif
+
 const struct attribute_group *nvme_ns_attr_groups[] = {
 	&nvme_ns_attr_group,
+#ifdef CONFIG_NVME_MULTIPATH
+	&nvme_ns_mpath_attr_group,
+#endif
 	NULL,
 };
 
Signature · 1 line
2.45.2
#2 lore raw

Re: [PATCHv3 RFC 1/1] nvme-multipath: Add sysfs attributes for showing multipath info

Quoted 20 lines · Nilay Shroff

NVMe native multipath supports different IO policies for selecting I/O path, however we don't have any visibility about which path is being selected by multipath code for forwarding I/O.
This patch helps add that visibility by adding new sysfs attribute files named "numa_nodes" and "queue_depth" under each namespace block device path /sys/block/nvmeXcYnZ/. We also create a "multipath" sysfs directory under head disk node and then from this directory add a link to each namespace path device this head disk node points to.

For instance, /sys/block/nvmeXnY/multipath/ would create a soft link to each path the head disk node <nvmeXnY> points to:

$ ls -1 /sys/block/nvme1n1/
nvme1c1n1 -> ../../../../../pci052e:78/052e:78:00.0/nvme/nvme1/nvme1c1n1
nvme1c3n1 -> ../../../../../pci058e:78/058e:78:00.0/nvme/nvme3/nvme1c3n1

For round-robin I/O policy, we could easily infer from the above output that I/O workload targeted to nvme3n1 would toggle across paths nvme1c1n1 and nvme1c3n1.

Quoted 6 lines

For numa I/O policy, the "numa_nodes" attribute file shows the numa nodes being preferred by the respective block device path. The numa nodes value is comma delimited list of nodes or A-B range of nodes.

For queue-depth I/O policy, the "queue_depth" attribute file shows the number of active/in-flight I/O requests currently queued for each path.

As far I can tell, this looks good to me.

Quoted 4 lines
drivers/nvme/host/multipath.c +4 -0
+static ssize_t numa_nodes_show(struct device *dev, struct device_attribute *attr,
+		char *buf)
+{
+	int node;

drivers/nvme/host/multipath.c static ssize_t ana_state_show(struct device *dev, struct device_attribute *attr, · line 947

Quoted 10 lines
drivers/nvme/host/multipath.c +10 -0
+	nodemask_t numa_nodes;
+	struct nvme_ns *current_ns;
+	struct nvme_ns *ns = nvme_get_ns_from_dev(dev);
+	struct nvme_ns_head *head = ns->head;
+
+	nodes_clear(numa_nodes);
+
+	for_each_node(node) {
+		current_ns = srcu_dereference(head->current_path[node],
+				&head->srcu);

Don't you need to use srcu_read_lock() first?

drivers/nvme/host/multipath.c static ssize_t ana_state_show(struct device *dev, struct device_attribute *attr, · line 949

Quoted 2 lines
drivers/nvme/host/multipath.c +2 -0
+		if (ns == current_ns)
+			node_set(node, numa_nodes);

And if ns matches current_ns can't you break the loop?

Thanks,
Daniel

#3 lore raw

Re: [PATCHv3 RFC 1/1] nvme-multipath: Add sysfs attributes for showing multipath info

Quoted 30 lines · Daniel Wagner
Quoted 20 lines · Nilay Shroff

NVMe native multipath supports different IO policies for selecting I/O path, however we don't have any visibility about which path is being selected by multipath code for forwarding I/O.
This patch helps add that visibility by adding new sysfs attribute files named "numa_nodes" and "queue_depth" under each namespace block device path /sys/block/nvmeXcYnZ/. We also create a "multipath" sysfs directory under head disk node and then from this directory add a link to each namespace path device this head disk node points to.

For instance, /sys/block/nvmeXnY/multipath/ would create a soft link to each path the head disk node <nvmeXnY> points to:

$ ls -1 /sys/block/nvme1n1/
nvme1c1n1 -> ../../../../../pci052e:78/052e:78:00.0/nvme/nvme1/nvme1c1n1
nvme1c3n1 -> ../../../../../pci058e:78/058e:78:00.0/nvme/nvme3/nvme1c3n1

For round-robin I/O policy, we could easily infer from the above output that I/O workload targeted to nvme3n1 would toggle across paths nvme1c1n1 and nvme1c3n1.

Quoted 6 lines

For numa I/O policy, the "numa_nodes" attribute file shows the numa nodes being preferred by the respective block device path. The numa nodes value is comma delimited list of nodes or A-B range of nodes.

For queue-depth I/O policy, the "queue_depth" attribute file shows the number of active/in-flight I/O requests currently queued for each path.

As far I can tell, this looks good to me.

Quoted 16 lines
Quoted 4 lines
drivers/nvme/host/multipath.c +4 -0
+static ssize_t numa_nodes_show(struct device *dev, struct device_attribute *attr,
+		char *buf)
+{
+	int node;

drivers/nvme/host/multipath.c static ssize_t ana_state_show(struct device *dev, struct device_attribute *attr, · line 947

Quoted 10 lines
drivers/nvme/host/multipath.c +10 -0
+	nodemask_t numa_nodes;
+	struct nvme_ns *current_ns;
+	struct nvme_ns *ns = nvme_get_ns_from_dev(dev);
+	struct nvme_ns_head *head = ns->head;
+
+	nodes_clear(numa_nodes);
+
+	for_each_node(node) {
+		current_ns = srcu_dereference(head->current_path[node],
+				&head->srcu);

Don't you need to use srcu_read_lock() first?

Yeah I overlooked it. You're right, I need to use srcu_read_lock(). I will update it in the next revision of the patch.

Quoted 5 lines

drivers/nvme/host/multipath.c static ssize_t ana_state_show(struct device *dev, struct device_attribute *attr, · line 949

Quoted 3 lines
drivers/nvme/host/multipath.c +2 -0
+		if (ns == current_ns)
+			node_set(node, numa_nodes);

And if ns matches current_ns can't you break the loop?

I think no. Here we map each numa node preferred for a namespace path. So for instance, if the current_ns (nvmeXcYnZ) path is preferred for I/O workload running on numa nodes i and j then we accordingly set the respective node values (i and j) in the nodemask (numa_nodes) which is a bitmap.

So in practice, when we cat this file,

# cat  /sys/block/nvme1n1/multipath/nvme1c0n1/numa_nodes 
0-1

we get the output node 0 and 1. That implies that I/O workload targeted to /dev/nvme1n1 and running on numa node 0 or 1 would prefer using the path nvme1c0n1.

And similarly, when we cat this file,

# cat  /sys/block/nvme1n1/multipath/nvme1c1n1/numa_nodes
2-3

we get the output node 2 and 3. That implies that I/O workload targeted to /dev/nvme1n1 and running on numa node 2 or 3 would prefer using the path nvme1c1n1.

Hope that clarifies. Let me know if you still think this doesn't make sense.

Quoted 3 lines

Thanks,
Daniel

Thanks,
--Nilay

#4 lore raw

Re: [PATCHv3 RFC 1/1] nvme-multipath: Add sysfs attributes for showing multipath info

Quoted 9 lines · Nilay Shroff
Quoted 4 lines

drivers/nvme/host/multipath.c static ssize_t ana_state_show(struct device *dev, struct device_attribute *attr, · line 949

Quoted 2 lines
drivers/nvme/host/multipath.c +2 -0
+		if (ns == current_ns)
+			node_set(node, numa_nodes);

And if ns matches current_ns can't you break the loop?

I think no. Here we map each numa node preferred for a namespace path. So for instance, if the current_ns (nvmeXcYnZ) path is preferred for I/O workload running on numa nodes i and j then we accordingly set the respective node values (i and j) in the nodemask (numa_nodes) which is a bitmap.

Okay, got it.

Thanks,
Daniel