Skip to contentSkip to conversation

Add visibility for native NVMe miltipath using debugfs

RFC PATCH Superseded Nilay Shroff 5 participants 11 messages Jul 22, 2024 – Jul 29, 2024 NVMe

v1 of 7 Superseded Latest is v7 →

lore
Share & tools
lore
#0 lore raw

cover Add visibility for native NVMe miltipath using debugfs

Hi,

This patch propose adding a new debugfs file entry for NVMe native multipath. As we know NVMe native multipath today supports three different io-policies (numa, round-robin and queue-depth) for selecting optimal I/O path and forwarding data. However we don't have yet any visibility to find the I/O path being selected by NVMe native multipath code.

IMO, it'd be nice to have this visibility information available under debugfs which could help a user to validate the I/O path being chosen is optimal for a given io policy. This patch propose adding a debugfs file for each head disk node on the system. The proposal is to create a file named "multipath" under "/sys/kernel/debug/nvmeXnY/".

Please find below output generated with this patch applied on a system with a multi-controller PCIe NVMe disk attached to it. This system is also an NVMf-TCP host which is connected to NVMf-TCP target over two NIC cards. This system has two numa nodes online when the below output was captured:

# cat /sys/devices/system/node/online
2-3
# nvme list -v
Subsystem        Subsystem-NQN                                                                                    Controllers
---------------- ------------------------------------------------------------------------------------------------ ----------------
nvme-subsys1     nvmet_subsystem                                                                                  nvme1, nvme3
nvme-subsys2     nqn.2019-10.com.kioxia:KCM7DRUG1T92:3D60A04906N1                                                 nvme0, nvme2

Device Cntlid SN MN FR TxPort Address Slot Subsystem Namespaces
---------------- ------ -------------------- ---------------------------------------- -------- ------ -------------- ------ ------------ ----------------

nvme0    2      3D60A04906N1         1.6TB NVMe Gen4 U.2 SSD IV               REV.CAS2 pcie   0524:28:00.0   U50EE.001.WZS000E-P3-C4-R1 nvme-subsys2 nvme2n2
nvme2    1      3D60A04906N1         1.6TB NVMe Gen4 U.2 SSD IV               REV.CAS2 pcie   0584:28:00.0   U50EE.001.WZS000E-P3-C4-R2 nvme-subsys2 nvme2n2
nvme1    1      a224673364d1dcb6fab9 Linux                                    6.9.0    tcp    traddr=10.0.0.200,trsvcid=4420,src_addr=10.0.0.100        nvme-subsys1 nvme1n1
nvme3    2      a224673364d1dcb6fab9 Linux                                    6.9.0    tcp    traddr=20.0.0.200,trsvcid=4420,src_addr=20.0.0.100        nvme-subsys1 nvme1n1

Device Generic NSID Usage Format Controllers
----------------- ----------------- ---------- -------------------------- ---------------- ----------------

/dev/nvme1n1 /dev/ng1n1   0x1          5.75  GB /   5.75  GB      4 KiB +  0 B   nvme1, nvme3
/dev/nvme2n2 /dev/ng2n2   0x2          0.00   B /   5.75  GB      4 KiB +  0 B   nvme0, nvme2
# cat /sys/class/nvme-subsystem/nvme-subsys2/iopolicy
numa
# cat /sys/kernel/debug/block/nvme2n2/multipath
io-policy: numa
io-path:
--------
node  current-path  ctrl    ana-state
2     nvme2c2n2     nvme2   optimized
3     nvme2c0n2     nvme0   optimized

The above output shows that current selected iopolicy is numa. And when we have workload running I/O on numa node 2, accessing namespace "nvme2n2", it uses path nvme2c2n2 and controller nvme2 for forwarding data. Moreover the current ana-state for this path is optimized. Similarly, for I/O workload running on numa node 3 would use path nvme2c0n2 and controller nvme0.

Now changing the iopolicy to round-robin,

# echo "round-robin" > /sys/class/nvme-subsystem/nvme-subsys2/iopolicy
# cat /sys/kernel/debug/block/nvme2n2/multipath
io-policy: round-robin
io-path:
--------
node  rr-path       ctrl    ana-state
2     nvme2c2n2     nvme2   optimized
2     nvme2c0n2     nvme0   optimized
3     nvme2c2n2     nvme2   optimized
3     nvme2c0n2     nvme0   optimized

The above output shows that current selected iopolicy is round-robin, and when we have I/O workload running on numa node 2, accessing namespace "nvme2n2", the I/O path would toggle between nvme2c2n2/nvme2 and nvme2c0n2/nvme0. And the same is true for I/O workload running on node 3. Both I/O paths are currently optimized.

The namespace "nvme1n1" is accessible over fabric(NVMf-TCP).

# cat /sys/kernel/debug/block/nvme1n1/multipath
io-policy: queue-depth
io-path:
--------
node  path          ctrl    qdepth      ana-state
2     nvme1c1n1     nvme1   1328        optimized
2     nvme1c3n1     nvme3   1324        optimized
3     nvme1c1n1     nvme1   1328        optimized
3     nvme1c3n1     nvme3   1324        optimized

The above output was captured while I/O was running and accessing namespace nvme1n1. From the above output, we see that iopolicy is set to "queue-depth". When we have I/O workload running on numa node 2, accessing namespace "nvme1n1", the I/O path nvme1c1n1/nvme1 has queue depth of 1328 and another I/O path nvme1c3n1/nvme3 has queue depth of 1324. Both paths are optimized and seems that both paths are equally utilized for forwarding I/O. The same could be said for workload running on numa node 3.

Nilay Shroff (1):
nvme-multipath: Add debugfs entry for showing multipath info

 drivers/nvme/host/multipath.c | 92 +++++++++++++++++++++++++++++++++++
 drivers/nvme/host/nvme.h      |  1 +
 2 files changed, 93 insertions(+)
Signature · 1 line
2.45.2
#1 lore raw

RFC PATCH 1/1 nvme-multipath: Add debugfs entry for showing multipath info

NVMe native multipath supports different io policies for selecting I/O path, however, we don't have any visibility about which path is being selected by multipath code for forwarding I/O. This patch helps add that visibility by adding a debugfs file for each head disk node on the system. It creates a file named "multipath" under "/sys/kernel/debug/block/nvmeXnY/". This file shows the information about current selected "io-policy" as well as it prints a "table" showing information about each online node and it's respective I/O path, controller name, ana-state and optionally queue depth of each path (if selected io-policy is queue-depth).

---
 drivers/nvme/host/multipath.c | 90 +++++++++++++++++++++++++++++++++++
 drivers/nvme/host/nvme.h      |  1 +
 2 files changed, 91 insertions(+)
drivers/nvme/host/multipath.c +90 -0
index 91d9eb3c22ef..143d4b279b43 100644
@@ -6,6 +6,7 @@
 #include <linux/backing-dev.h>
 #include <linux/moduleparam.h>
 #include <linux/vmalloc.h>
+#include <linux/debugfs.h>
 #include <trace/events/block.h>
 #include "nvme.h"
 
@@ -628,6 +629,91 @@ int nvme_mpath_alloc_disk(struct nvme_ctrl *ctrl, struct nvme_ns_head *head)
 			ctrl->subsys->instance, head->instance);
 	return 0;
 }
+static void nvme_mpath_numa_show(struct seq_file *m, struct nvme_ns_head *head)
+{
+	int node;
+	struct nvme_ns *ns;
+
+	seq_printf(m, "%-4s  %-12s  %-6s  %s\n",
+			"node", "current-path", "ctrl", "ana-state");
+
+	for_each_online_node(node) {
+		ns = srcu_dereference(head->current_path[node], &head->srcu);
+		if (ns)
+			seq_printf(m, "%-4d  %-12s  %-6s  %s\n",
+					node, ns->disk->disk_name,
+					dev_name(ns->ctrl->device),
+					nvme_ana_state_names[ns->ana_state]);
+	}
+}
+
+static void nvme_mpath_rr_show(struct seq_file *m, struct nvme_ns_head *head)
+{
+	int node;
+	struct nvme_ns *ns;
+
+	seq_printf(m, "%-4s  %-12s  %-6s  %s\n",
+			"node", "rr-path", "ctrl", "ana-state");
+
+	for_each_online_node(node) {
+		list_for_each_entry_rcu(ns, &head->list, siblings) {
+			seq_printf(m, "%-4d  %-12s  %-6s  %s\n",
+					node, ns->disk->disk_name,
+					dev_name(ns->ctrl->device),
+					nvme_ana_state_names[ns->ana_state]);
+		}
+	}
+}
+
+static void nvme_mpath_qd_show(struct seq_file *m, struct nvme_ns_head *head)
+{
+	int node;
+	struct nvme_ns *ns;
+
+	seq_printf(m, "%-4s  %-12s  %-6s  %-10s  %s\n",
+			"node", "path", "ctrl", "qdepth", "ana-state");
+
+	for_each_online_node(node) {
+		list_for_each_entry_rcu(ns, &head->list, siblings) {
+			seq_printf(m, "%-4d  %-12s  %-6s  %-10d  %s\n",
+					node, ns->disk->disk_name,
+					dev_name(ns->ctrl->device),
+					atomic_read(&ns->ctrl->nr_active),
+					nvme_ana_state_names[ns->ana_state]);
+
+		}
+	}
+}
+
+static int nvme_mpath_show(struct seq_file *m, void *p)
+{
+	struct nvme_ns_head *head = m->private;
+	int iopolicy = READ_ONCE(head->subsys->iopolicy);
+
+	seq_printf(m, "io-policy: %s\n", nvme_iopolicy_names[iopolicy]);
+
+	seq_puts(m, "io-path:\n");
+	seq_puts(m, "--------\n");
+
+	if (iopolicy == NVME_IOPOLICY_NUMA)
+		nvme_mpath_numa_show(m, head);
+	else if (iopolicy == NVME_IOPOLICY_RR)
+		nvme_mpath_rr_show(m, head);
+	else if (iopolicy == NVME_IOPOLICY_QD)
+		nvme_mpath_qd_show(m, head);
+
+	return 0;
+}
+
+static int nvme_mpath_open(struct inode *inode, struct file *file)
+{
+	return single_open(file, nvme_mpath_show, inode->i_private);
+}
+static const struct file_operations nvme_mpath_fops = {
+	.open = nvme_mpath_open,
+	.read = seq_read,
+	.release = single_release
+};
 
 static void nvme_mpath_set_live(struct nvme_ns *ns)
 {
@@ -650,6 +736,9 @@ static void nvme_mpath_set_live(struct nvme_ns *ns)
 			return;
 		}
 		nvme_add_ns_head_cdev(head);
+		head->debugfs = debugfs_create_file("multipath", 0400,
+					head->disk->queue->debugfs_dir, head,
+					&nvme_mpath_fops);
 	}
 
 	mutex_lock(&head->lock);
@@ -969,6 +1058,7 @@ void nvme_mpath_shutdown_disk(struct nvme_ns_head *head)
 		return;
 	kblockd_schedule_work(&head->requeue_work);
 	if (test_bit(NVME_NSHEAD_DISK_LIVE, &head->flags)) {
+		debugfs_remove(head->debugfs);
 		nvme_cdev_del(&head->cdev, &head->cdev_device);
 		del_gendisk(head->disk);
 	}
drivers/nvme/host/nvme.h +1 -0
index f900e44243ae..5b4c0b70cedf 100644
@@ -493,6 +493,7 @@ struct nvme_ns_head {
 	struct work_struct	requeue_work;
 	struct mutex		lock;
 	unsigned long		flags;
+	struct dentry		*debugfs;
 #define NVME_NSHEAD_DISK_LIVE	0
 	struct nvme_ns __rcu	*current_path[];
 #endif
Signature · 1 line
2.45.2
#2 lore raw
In reply to Nilay Shroff
Quoted 12 lines · Nilay Shroff

This patch propose adding a new debugfs file entry for NVMe native multipath. As we know NVMe native multipath today supports three different io-policies (numa, round-robin and queue-depth) for selecting optimal I/O path and forwarding data. However we don't have yet any visibility to find the I/O path being selected by NVMe native multipath code.

IMO, it'd be nice to have this visibility information available under debugfs which could help a user to validate the I/O path being chosen is optimal for a given io policy. This patch propose adding a debugfs file for each head disk node on the system. The proposal is to create a file named "multipath" under "/sys/kernel/debug/nvmeXnY/".

Quoted 5 lines

Please find below output generated with this patch applied on a system with a multi-controller PCIe NVMe disk attached to it. This system is also an NVMf-TCP host which is connected to NVMf-TCP target over two NIC cards. This system has two numa nodes online when the below output was captured:

Wouldn't it make sense to extend nvme-cli instead adding additional debugfs entries to the kernel, e.g. extending show-topology?

#3 lore raw
Quoted 13 lines · Daniel Wagner
Quoted 12 lines · Nilay Shroff

This patch propose adding a new debugfs file entry for NVMe native multipath. As we know NVMe native multipath today supports three different io-policies (numa, round-robin and queue-depth) for selecting optimal I/O path and forwarding data. However we don't have yet any visibility to find the I/O path being selected by NVMe native multipath code.

IMO, it'd be nice to have this visibility information available under debugfs which could help a user to validate the I/O path being chosen is optimal for a given io policy. This patch propose adding a debugfs file for each head disk node on the system. The proposal is to create a file named "multipath" under "/sys/kernel/debug/nvmeXnY/".

Quoted 9 lines
Quoted 5 lines

Please find below output generated with this patch applied on a system with a multi-controller PCIe NVMe disk attached to it. This system is also an NVMf-TCP host which is connected to NVMf-TCP target over two NIC cards. This system has two numa nodes online when the below output was captured:

Wouldn't it make sense to extend nvme-cli instead adding additional debugfs entries to the kernel, e.g. extending show-topology?

Yeah we may extend nvme-cli to print this(multipathing) information however from where would nvme-cli retrieve that information? AFAIK, today this multipath information is not exported by NVMe driver. So we have to first make this information available from driver either through sysfs or ioctl and then nvme-cli could parse it and show it to the user. If everyone thinks that it's worth extending nvme-cli so that it could display this information then yes we can certainly implement it. Please suggest.

Thanks,
--Nilay

#4 lore raw
Quoted 23 lines · Nilay Shroff
Quoted 13 lines · Daniel Wagner
Quoted 12 lines · Nilay Shroff

This patch propose adding a new debugfs file entry for NVMe native multipath. As we know NVMe native multipath today supports three different io-policies (numa, round-robin and queue-depth) for selecting optimal I/O path and forwarding data. However we don't have yet any visibility to find the I/O path being selected by NVMe native multipath code.

IMO, it'd be nice to have this visibility information available under debugfs which could help a user to validate the I/O path being chosen is optimal for a given io policy. This patch propose adding a debugfs file for each head disk node on the system. The proposal is to create a file named "multipath" under "/sys/kernel/debug/nvmeXnY/".

Quoted 9 lines
Quoted 5 lines

Please find below output generated with this patch applied on a system with a multi-controller PCIe NVMe disk attached to it. This system is also an NVMf-TCP host which is connected to NVMf-TCP target over two NIC cards. This system has two numa nodes online when the below output was captured:

Wouldn't it make sense to extend nvme-cli instead adding additional debugfs entries to the kernel, e.g. extending show-topology?

Quoted 6 lines

Yeah we may extend nvme-cli to print this(multipathing) information however from where would nvme-cli retrieve that information? AFAIK, today this multipath information is not exported by NVMe driver. So we have to first make this information available from driver either through sysfs or ioctl and then nvme-cli could parse it and show it to the user. If everyone thinks that it's worth extending nvme-cli so that it could display this information then yes we can certainly implement it. Please suggest.

debugfs might always be available. IIRC when lockdown is enabled, debugfs is not available or parts of it.

I'd suggest going the full way and add expose the relevant information via sysfs and extend libnvme and nvme-cli. But this just my take on this.

#5 lore raw
Quoted 5 lines · Daniel Wagner

debugfs might always be available. IIRC when lockdown is enabled, debugfs is not available or parts of it.

I'd suggest going the full way and add expose the relevant information via sysfs and extend libnvme and nvme-cli. But this just my take on this.

Yes, if we want to do this properly sysfs is the place to go, not debugfs.

#6 lore raw
Quoted 10 lines · Christoph Hellwig
Quoted 5 lines · Daniel Wagner

debugfs might always be available. IIRC when lockdown is enabled, debugfs is not available or parts of it.

I'd suggest going the full way and add expose the relevant information via sysfs and extend libnvme and nvme-cli. But this just my take on this.

Yes, if we want to do this properly sysfs is the place to go, not debugfs.

Alright, I will send out another patch with relevant changes for sysfs and then latter in libnvme/nvme-cli.

Thanks,
--Nilay

maintainer
#7 lore raw
In reply to Nilay Shroff
Quoted 10 lines · Nilay Shroff
# cat /sys/kernel/debug/block/nvme1n1/multipath
io-policy: queue-depth
io-path:
--------
node  path          ctrl    qdepth      ana-state
2     nvme1c1n1     nvme1   1328        optimized
2     nvme1c3n1     nvme3   1324        optimized
3     nvme1c1n1     nvme1   1328        optimized
3     nvme1c3n1     nvme3   1324        optimized
Quoted 7 lines

The above output was captured while I/O was running and accessing namespace nvme1n1. From the above output, we see that iopolicy is set to "queue-depth". When we have I/O workload running on numa node 2, accessing namespace "nvme1n1", the I/O path nvme1c1n1/nvme1 has queue depth of 1328 and another I/O path nvme1c3n1/nvme3 has queue depth of 1324. Both paths are optimized and seems that both paths are equally utilized for forwarding I/O.

You can get the outstanding queue-depth from iostats too, and that doesn't rely on queue-depth io policy. It does, however, require stats are enabled, but that's probably a more reasonable given than an io policy.

Quoted 2 lines

The same could be said for workload running on numa node 3.

The output for all numa nodes will be the same regardless of which node a workload is running on (the accounting isn't per-node), so I'm not sure outputting qdepth again for each node is useful.

#8 lore raw
Quoted 19 lines · Keith Busch
Quoted 10 lines · Nilay Shroff
# cat /sys/kernel/debug/block/nvme1n1/multipath
io-policy: queue-depth
io-path:
--------
node  path          ctrl    qdepth      ana-state
2     nvme1c1n1     nvme1   1328        optimized
2     nvme1c3n1     nvme3   1324        optimized
3     nvme1c1n1     nvme1   1328        optimized
3     nvme1c3n1     nvme3   1324        optimized
Quoted 7 lines

The above output was captured while I/O was running and accessing namespace nvme1n1. From the above output, we see that iopolicy is set to "queue-depth". When we have I/O workload running on numa node 2, accessing namespace "nvme1n1", the I/O path nvme1c1n1/nvme1 has queue depth of 1328 and another I/O path nvme1c3n1/nvme3 has queue depth of 1324. Both paths are optimized and seems that both paths are equally utilized for forwarding I/O.

Quoted 5 lines

You can get the outstanding queue-depth from iostats too, and that doesn't rely on queue-depth io policy. It does, however, require stats are enabled, but that's probably a more reasonable given than an io policy.

Yes correct, user could use iostat to find the queue-depth in real-time when I/O workload is running.

Quoted 6 lines
Quoted 2 lines

The same could be said for workload running on numa node 3.

The output for all numa nodes will be the same regardless of which node a workload is running on (the accounting isn't per-node), so I'm not sure outputting qdepth again for each node is useful.

Agreed, so in that case we may only show the available I/O paths for head disk node when I/O policy is set to "queue-depth". In this case, we don't need to show paths per numa node as you suggested. And then for each I/O path we can show the "qdepth" once.

IMO, though it's possible to find the queue-depth monitoring the iostat output, it'd be convenient to have it readily available under one place where we would add further visibility of multipathing.

Thanks,
--Nilay

maintainer
#9 lore raw
In reply to Nilay Shroff
Quoted 20 lines
# cat /sys/kernel/debug/block/nvme2n2/multipath
io-policy: numa
io-path:
--------
node  current-path  ctrl    ana-state
2     nvme2c2n2     nvme2   optimized
3     nvme2c0n2     nvme0   optimized

The above output shows that current selected iopolicy is numa. And when we have workload running I/O on numa node 2, accessing namespace "nvme2n2", it uses path nvme2c2n2 and controller nvme2 for forwarding data. Moreover the current ana-state for this path is optimized. Similarly, for I/O workload running on numa node 3 would use path nvme2c0n2 and controller nvme0.

Now changing the iopolicy to round-robin,

# echo "round-robin" > /sys/class/nvme-subsystem/nvme-subsys2/iopolicy
Quoted 9 lines
# cat /sys/kernel/debug/block/nvme2n2/multipath
io-policy: round-robin
io-path:
--------
node  rr-path       ctrl    ana-state
2     nvme2c2n2     nvme2   optimized
2     nvme2c0n2     nvme0   optimized
3     nvme2c2n2     nvme2   optimized
3     nvme2c0n2     nvme0   optimized

Can we avoid a formatted output in sysfs? I'd much rather prefer that nvme-cli/libnvme to format this (maybe this may be wanted as json in the future for example)...

Can we simply expose the individual components and have userpace format the output?

#10 lore raw
Quoted 30 lines · Sagi Grimberg
Quoted 20 lines
# cat /sys/kernel/debug/block/nvme2n2/multipath
io-policy: numa
io-path:
--------
node  current-path  ctrl    ana-state
2     nvme2c2n2     nvme2   optimized
3     nvme2c0n2     nvme0   optimized

The above output shows that current selected iopolicy is numa. And when we have workload running I/O on numa node 2, accessing namespace "nvme2n2", it uses path nvme2c2n2 and controller nvme2 for forwarding data. Moreover the current ana-state for this path is optimized. Similarly, for I/O workload running on numa node 3 would use path nvme2c0n2 and controller nvme0.

Now changing the iopolicy to round-robin,

# echo "round-robin" > /sys/class/nvme-subsystem/nvme-subsys2/iopolicy
Quoted 9 lines
# cat /sys/kernel/debug/block/nvme2n2/multipath
io-policy: round-robin
io-path:
--------
node  rr-path       ctrl    ana-state
2     nvme2c2n2     nvme2   optimized
2     nvme2c0n2     nvme0   optimized
3     nvme2c2n2     nvme2   optimized
3     nvme2c0n2     nvme0   optimized
Quoted 4 lines

Can we avoid a formatted output in sysfs? I'd much rather prefer that nvme-cli/libnvme to format this (maybe this may be wanted as json in the future for example)...

Can we simply expose the individual components and have userpace format the output?

Yes that's what I am planning to implement. The sysfs would only expose the relevant information from NVMe driver and then libnvme/nvme-cli format the sysfs output as needed.

Thanks,
--Nilay