Skip to contentSkip to conversation

Add visibility for native NVMe multipath using sysfs

RFC PATCH Superseded Nilay Shroff 2 participants 6 messages Aug 9, 2024 – Aug 28, 2024 NVMe

v2 of 7 Superseded Latest is v7 →

lore
Share & tools
lore
#0 lore raw

v2 cover Add visibility for native NVMe multipath using sysfs

Hi,

This patch propose adding new sysfs attributes for adding visibility of native multipath I/O. The previous version of this RFC[1] proposed using debugfs for visibility however the general feedback was to instead export the multipath I/O information using sysfs attributes and then latter parse and format those sysfs attributes using libnvme/nvme-cli.

As we know, NVMe native multipath supports three different io policies (numa, round-robin and queue-depth) for selecting I/O path, however, we don't have any visibility about which path is being selected by multipath code for forwarding I/O. This RFC propose adding three new sysfs attribute files named "numa", "round_robin", and "queue_depth" under the namespace head block path /sys/block/<nvmeXnY>/multipath/. As the name suggests, attribute file named "numa" shall show the information about multipath I/O for numa poilcy, "round_robin" shall show the information about multipath I/O for round-robin policy and "queue_depth" shall show the information about multipath I/O for queue-depth policy.

Please find below output generated with this proposed RFC patch applied on a system with two multi-controller PCIe NVMe disks attached to it. This system is also an NVMf-TCP host which is connected to an NVMf-TCP target over two NIC cards. This system has four numa nodes online when the below output was captured:

# cat /sys/devices/system/node/online 
0-3
# lscpu
<snip>
NUMA:
  NUMA node(s):           4
  NUMA node0 CPU(s):
  NUMA node1 CPU(s):      0-7
  NUMA node2 CPU(s):      8-31
  NUMA node3 CPU(s):      32-63
<snip>

Please note that numa node 0 though online, doesn't have any CPU assigned to it.

# nvme list -v 
Subsystem        Subsystem-NQN                                                                                    Controllers
---------------- ------------------------------------------------------------------------------------------------ ----------------
nvme-subsys0     nvmet_subsystem                                                                                  nvme0, nvme3
nvme-subsys2     nqn.2019-10.com.kioxia:KCM7DRUG1T92:3D60A04906N1                                                 nvme1, nvme2
nvme-subsys4     nqn.1994-11.com.samsung:nvme:PM1735a:2.5-inch:S6RTNE0R900057                                     nvme4, nvme5

Device Cntlid SN MN FR TxPort Address Slot Subsystem Namespaces
---------------- ------ -------------------- ---------------------------------------- -------- ------ -------------- ------ ------------ ----------------

nvme0    1      a224673364d1dcb6fab9 Linux                                    6.9.0-rc tcp    traddr=20.0.0.200,trsvcid=4420,src_addr=20.0.0.100        nvme-subsys0 nvme0n1
nvme1    2      3D60A04906N1         1.6TB NVMe Gen4 U.2 SSD IV               REV.CAS2 pcie   0524:28:00.0   U50EE.001.WZS000E-P3-C4-R1 nvme-subsys2 nvme2n2
nvme2    1      3D60A04906N1         1.6TB NVMe Gen4 U.2 SSD IV               REV.CAS2 pcie   0584:28:00.0   U50EE.001.WZS000E-P3-C4-R2 nvme-subsys2 nvme2n2
nvme3    2      a224673364d1dcb6fab9 Linux                                    6.9.0-rc tcp    traddr=10.0.0.200,trsvcid=4420,src_addr=10.0.0.100        nvme-subsys0 nvme0n1
nvme4    65     S6RTNE0R900057       3.2TB NVMe Gen4 U.2 SSD III              REV.SN66 pcie   058e:78:00.0   U50EE.001.WZS000E-P3-C14-R2 nvme-subsys4 nvme4n1
nvme5    66     S6RTNE0R900057       3.2TB NVMe Gen4 U.2 SSD III              REV.SN66 pcie   052e:78:00.0   U50EE.001.WZS000E-P3-C14-R1 nvme-subsys4 nvme4n1

Device Generic NSID Usage Format Controllers
----------------- ----------------- ---------- -------------------------- ---------------- ----------------

/dev/nvme0n1 /dev/ng0n1   0x1          5.75  GB /   5.75  GB      4 KiB +  0 B   nvme0, nvme3
/dev/nvme2n2 /dev/ng2n2   0x2          0.00   B /   5.75  GB      4 KiB +  0 B   nvme1, nvme2
/dev/nvme4n1 /dev/ng4n1   0x1          5.75  GB /   5.75  GB      4 KiB +  0 B   nvme4, nvme5

As we could see above, we've three shared namespaces created and for each shared namespace there's a head disk node created. In terms of iopolicy, we have "queue-depth" configured for nvme-subsys0, "numa" configured for nvme-subsys2 and "round-robin" configured for nvme-subsys4.

# cat /sys/class/nvme-subsystem/nvme-subsys0/iopolicy
queue-depth
# cat /sys/class/nvme-subsystem/nvme-subsys2/iopolicy
numa
# cat /sys/class/nvme-subsystem/nvme-subsys4/iopolicy
round-robin

Now, under each "namespace head disk node", we create sysfs attributes "numa", "round-robin" and "queue-depth" as shown below.

# tree /sys/block/nvme0n1/multipath/
/sys/block/nvme0n1/multipath/
├── numa
├── queue_depth
└── round_robin

0 directories, 3 files

# tree /sys/block/nvme2n2/multipath/
/sys/block/nvme2n2/multipath/
├── numa
├── queue_depth
└── round_robin

0 directories, 3 files

# tree /sys/block/nvme4n1/multipath/
/sys/block/nvme4n1/multipath/
├── numa
├── queue_depth
└── round_robin

0 directories, 3 files

The mutlipath I/O information for each of the above head disk node and it's respective iopolicy is then respresented as below.

# cat /sys/block/nvme0n1/multipath/queue_depth 
nvme0c0n1 423
nvme0c3n1 425

In the above output, we print the multipath I/O information about the queue_depth iopolicy as the head disk node "nvme0n1" is defined under nvme-subsys0 which has queue-depth iopolicy configured. For queue-depth iopolicy, it doesn't matter on which numa node IO workload is running. So we print each I/O path once and its respective current queue depth.

# cat /sys/block/nvme2n2/multipath/numa 
node1: nvme2c1n2
node2: nvme2c2n2
node3: nvme2c2n2

In the above output, we print the multipath I/O information about the numa iopolicy as the head disk node "nvme2n2" is defined under nvme-subsys2 which has numa iopolicy configured. For numa iopolicy, we print the preferred I/O path for each numa node.

# cat /sys/block/nvme4n1/multipath/round_robin 
node1: nvme4c4n1 nvme4c5n1 
node2: nvme4c4n1 nvme4c5n1 
node3: nvme4c5n1 nvme4c4n1 

In the above output, we print the multipath I/O information about the round-robin iopolicy as the head disk node "nvme4n1" is defined under nvme-subsys4 which has round-robin iopolicy configured. For round-robin iopolicy, we print the per node I/O paths in round-robin fashion.

Please note that we don't print ana_state information in the above output because that could be retrieved from another sysfs attribute per I/O path. Also we don't print I/O controller in the output because that can be also easily made up from the respective I/O path. For instance, I/O path nvme4c5n1 is accessible from controller nvme5.

IMO, once we have above information exported through sysfs, we shall be able to easily parse it in libnvme/nvme-cli and use it to further extend the output of nvme show-topology command or add a new nvme-cli command.

[1] https://lore.kernel.org/all/[email protected]/

Changes since v1:
- Use sysfs to export multipath I/O information instead of debugfs

Nilay Shroff (1):
nvme-multipath: Add sysfs attributes for showing multipath info

 drivers/nvme/host/multipath.c |   2 +-
 drivers/nvme/host/nvme.h      |   1 +
 drivers/nvme/host/sysfs.c     | 108 ++++++++++++++++++++++++++++++++++
 3 files changed, 110 insertions(+), 1 deletion(-)
Signature · 1 line
2.45.2
#1 lore raw

RFC PATCH v2 1/1 nvme-multipath: Add sysfs attributes for showing multipath info

NVMe native multipath supports different IO policies for selecting I/O path, however we don't have any visibility about which path is being selected by multipath code for forwarding I/O. This patch helps add that visibility by adding three new sysfs attribute files named "numa", "round_robin", and "queue_depth" under each namespace head block path /sys/block/<nvmeXnY>/multipath. As the name suggests, "numa" attribute file shows the information about multipath I/O for numa policy, "round_robin" attribute file shows the information about multipath I/O for round-robin policy and "queue_depth" attribute file shows the information about multipath I/O for queue-depth IO policy.

---
 drivers/nvme/host/multipath.c |   2 +-
 drivers/nvme/host/nvme.h      |   1 +
 drivers/nvme/host/sysfs.c     | 108 ++++++++++++++++++++++++++++++++++
 3 files changed, 110 insertions(+), 1 deletion(-)
drivers/nvme/host/multipath.c +1 -1
index 91d9eb3c22ef..1a5ca3e7cd5c 100644
@@ -292,7 +292,7 @@ static struct nvme_ns *__nvme_find_path(struct nvme_ns_head *head, int node)
 	return found;
 }
 
-static struct nvme_ns *nvme_next_ns(struct nvme_ns_head *head,
+struct nvme_ns *nvme_next_ns(struct nvme_ns_head *head,
 		struct nvme_ns *ns)
 {
 	ns = list_next_or_null_rcu(&head->list, &ns->siblings, struct nvme_ns,
drivers/nvme/host/nvme.h +1 -0
index f900e44243ae..3fe5fac5067f 100644
@@ -955,6 +955,7 @@ void nvme_mpath_stop(struct nvme_ctrl *ctrl);
 bool nvme_mpath_clear_current_path(struct nvme_ns *ns);
 void nvme_mpath_revalidate_paths(struct nvme_ns *ns);
 void nvme_mpath_clear_ctrl_paths(struct nvme_ctrl *ctrl);
+struct nvme_ns *nvme_next_ns(struct nvme_ns_head *head, struct nvme_ns *ns);
 void nvme_mpath_shutdown_disk(struct nvme_ns_head *head);
 void nvme_mpath_start_request(struct request *rq);
 void nvme_mpath_end_request(struct request *rq);
drivers/nvme/host/sysfs.c +108 -0

120 lines not on this page · raw message

Signature · 1 line
2.45.2
maintainer
#2 lore raw
In reply to Nilay Shroff
Quoted 3 lines · Nilay Shroff
# cat /sys/block/nvme0n1/multipath/queue_depth 
nvme0c0n1 423
nvme0c3n1 425

I am not sure this matches up to the desire for 1 value per file. What I thought we wanted was a directory for each path, so it'd look something like:

/sys/block/nvme0n1/multipatch/nvme0c0n1/
/sys/block/nvme0n1/multipatch/nvme0c3n1/

And each directory has their attributes so they print exactly one value instead of the multi-line output. You'd know which path the output corresponds to from the file's directory.

#3 lore raw
Quoted 9 lines · Keith Busch
Quoted 3 lines · Nilay Shroff
# cat /sys/block/nvme0n1/multipath/queue_depth 
nvme0c0n1 423
nvme0c3n1 425

I am not sure this matches up to the desire for 1 value per file. What I thought we wanted was a directory for each path, so it'd look something like:

Quoted 7 lines

/sys/block/nvme0n1/multipatch/nvme0c0n1/
/sys/block/nvme0n1/multipatch/nvme0c3n1/

And each directory has their attributes so they print exactly one value instead of the multi-line output. You'd know which path the output corresponds to from the file's directory.

Yes you were right, we need 1 value per file but I thought keeping multipath details concisely in a single file may be then easier for libnvme/nvme-cli to format and parse.

I also read in the sysfs documentation, "attribute file should preferably contain only one value per file. However, it is noted that it may not be efficient to contain only one value per file, so it is socially acceptable to express an array of values of the same type."

Anyways, I believe, most of us prefer 1 value per file rule for the sysfs attributes, so how about exporting multipath details as shown below?

Let's assume namespace head node nvmeXnY points to two different paths:
nvmeXc1nY and nvmeXc3nY.

First we create the "multipath" directory under /sys/block/nvmeXnY

The multipath directory would then contain two sub-directories (which are the two paths pointed by namespace head node) named nvmeXc1nY and nvmeXc3nY. In fact, these two sub-directories are actually soft link to the respective namespace block device under /sys/block

For instance, we have a namespace head node nvme1n1 which points to two different paths: nvme1c1n1 and nvme1c3n1.

# ls -l /sys/block/nvme1n1/multipath
lrwxrwxrwx. 1 root root     0 Aug 11 12:30 nvme1c1n1 -> ../../../../../pci052e:78/052e:78:00.0/nvme/nvme1/nvme1c1n1
lrwxrwxrwx. 1 root root     0 Aug 11 12:30 nvme1c3n1 -> ../../../../../pci058e:78/058e:78:00.0/nvme/nvme3/nvme1c3n1
# ls -l /sys/block/
lrwxrwxrwx. 1 root root 0 Aug 11 12:30 nvme1c1n1 -> ../devices/pci052e:78/052e:78:00.0/nvme/nvme1/nvme1c1n1
lrwxrwxrwx. 1 root root 0 Aug 11 12:30 nvme1c3n1 -> ../devices/pci058e:78/058e:78:00.0/nvme/nvme3/nvme1c3n1

As we see above, /sys/block/nvme1n1/multipath/nvme1c1n1 is soft link to /sys/block/nvme1c1n1 and similarly /sys/block/nvme1n1/multipath/nvme1c3n1 is soft link to /sys/block/nvme1c3n1.

For round-robin I/O policy, we could easily infer from the above output that I/O workload targeted to nvme1n1 would toggle across nvme1c1n1 and nvme1c3n1.

We also create two new sysfs attribute files named "numa_nodes" and "queue_depth" under /sys/block/nvmeXcYnZ.

# cat  /sys/block/nvme1n1/multipath/nvme1c1n1/numa_nodes 
0-1
# cat  /sys/block/nvme1n1/multipath/nvme1c3n1/numa_nodes 
2-3

For I/O policy numa, above output signifies that I/O workload targeted to nvme1n1 and running on node 0 and 1 would prefer using path nvm1c1n1. Similarly, I/O workload targeted to nvme1n1 and running on node 2 and 3 would prefer using path nvme1c3n1.

# cat /sys/block/nvme1n1/multipath/nvme1c1n1/queue_depth
423
# cat /sys/block/nvme1n1/multipath/nvme1c3n1/queue_depth
425

For I/O policy queue-depth, above output signifies that for I/O workload targeted to nvme1n1, we have got two paths nvme1c1n1 and nvme1c3n1 and the queue_depth for each path is 423 and 425 respectively.

Thanks,
--Nilay

#4 lore raw

Hi Keith,

Quoted 60 lines
Quoted 8 lines

/sys/block/nvme0n1/multipatch/nvme0c0n1/
/sys/block/nvme0n1/multipatch/nvme0c3n1/

And each directory has their attributes so they print exactly one value instead of the multi-line output. You'd know which path the output corresponds to from the file's directory.

Yes you were right, we need 1 value per file but I thought keeping multipath details concisely in a single file may be then easier for libnvme/nvme-cli to format and parse.

I also read in the sysfs documentation, "attribute file should preferably contain only one value per file. However, it is noted that it may not be efficient to contain only one value per file, so it is socially acceptable to express an array of values of the same type."

Anyways, I believe, most of us prefer 1 value per file rule for the sysfs attributes, so how about exporting multipath details as shown below?

Let's assume namespace head node nvmeXnY points to two different paths:
nvmeXc1nY and nvmeXc3nY.

First we create the "multipath" directory under /sys/block/nvmeXnY

The multipath directory would then contain two sub-directories (which are the two paths pointed by namespace head node) named nvmeXc1nY and nvmeXc3nY. In fact, these two sub-directories are actually soft link to the respective namespace block device under /sys/block

For instance, we have a namespace head node nvme1n1 which points to two different paths: nvme1c1n1 and nvme1c3n1.

# ls -l /sys/block/nvme1n1/multipath
lrwxrwxrwx. 1 root root     0 Aug 11 12:30 nvme1c1n1 -> ../../../../../pci052e:78/052e:78:00.0/nvme/nvme1/nvme1c1n1
lrwxrwxrwx. 1 root root     0 Aug 11 12:30 nvme1c3n1 -> ../../../../../pci058e:78/058e:78:00.0/nvme/nvme3/nvme1c3n1
# ls -l /sys/block/
lrwxrwxrwx. 1 root root 0 Aug 11 12:30 nvme1c1n1 -> ../devices/pci052e:78/052e:78:00.0/nvme/nvme1/nvme1c1n1
lrwxrwxrwx. 1 root root 0 Aug 11 12:30 nvme1c3n1 -> ../devices/pci058e:78/058e:78:00.0/nvme/nvme3/nvme1c3n1

As we see above, /sys/block/nvme1n1/multipath/nvme1c1n1 is soft link to /sys/block/nvme1c1n1 and similarly /sys/block/nvme1n1/multipath/nvme1c3n1 is soft link to /sys/block/nvme1c3n1.

For round-robin I/O policy, we could easily infer from the above output that I/O workload targeted to nvme1n1 would toggle across nvme1c1n1 and nvme1c3n1.

We also create two new sysfs attribute files named "numa_nodes" and "queue_depth" under /sys/block/nvmeXcYnZ.

# cat  /sys/block/nvme1n1/multipath/nvme1c1n1/numa_nodes 
0-1
# cat  /sys/block/nvme1n1/multipath/nvme1c3n1/numa_nodes 
2-3

For I/O policy numa, above output signifies that I/O workload targeted to nvme1n1 and running on node 0 and 1 would prefer using path nvm1c1n1. Similarly, I/O workload targeted to nvme1n1 and running on node 2 and 3 would prefer using path nvme1c3n1.

Quoted 9 lines
# cat /sys/block/nvme1n1/multipath/nvme1c1n1/queue_depth
423
# cat /sys/block/nvme1n1/multipath/nvme1c3n1/queue_depth
425

For I/O policy queue-depth, above output signifies that for I/O workload targeted to nvme1n1, we have got two paths nvme1c1n1 and nvme1c3n1 and the queue_depth for each path is 423 and 425 respectively.

A gentle ping about the above proposed solution.

As you proposed about 1 file 1 attribute value, I reworked the proposal and would like to know if you have any further feedback/comment. If this looks good then shall I resend the RFC v3 with the above changes incorporated?

Thanks,
--Nilay

#5 lore raw

Hi Keith,

A gentle ping for this RFC. It would be really helpful if you would share your feedback/comment.

Thanks,
--Nilay

Quoted 80 lines · Nilay Shroff

Hi Keith,

Quoted 60 lines
Quoted 8 lines

/sys/block/nvme0n1/multipatch/nvme0c0n1/
/sys/block/nvme0n1/multipatch/nvme0c3n1/

And each directory has their attributes so they print exactly one value instead of the multi-line output. You'd know which path the output corresponds to from the file's directory.

Yes you were right, we need 1 value per file but I thought keeping multipath details concisely in a single file may be then easier for libnvme/nvme-cli to format and parse.

I also read in the sysfs documentation, "attribute file should preferably contain only one value per file. However, it is noted that it may not be efficient to contain only one value per file, so it is socially acceptable to express an array of values of the same type."

Anyways, I believe, most of us prefer 1 value per file rule for the sysfs attributes, so how about exporting multipath details as shown below?

Let's assume namespace head node nvmeXnY points to two different paths:
nvmeXc1nY and nvmeXc3nY.

First we create the "multipath" directory under /sys/block/nvmeXnY

The multipath directory would then contain two sub-directories (which are the two paths pointed by namespace head node) named nvmeXc1nY and nvmeXc3nY. In fact, these two sub-directories are actually soft link to the respective namespace block device under /sys/block

For instance, we have a namespace head node nvme1n1 which points to two different paths: nvme1c1n1 and nvme1c3n1.

# ls -l /sys/block/nvme1n1/multipath
lrwxrwxrwx. 1 root root     0 Aug 11 12:30 nvme1c1n1 -> ../../../../../pci052e:78/052e:78:00.0/nvme/nvme1/nvme1c1n1
lrwxrwxrwx. 1 root root     0 Aug 11 12:30 nvme1c3n1 -> ../../../../../pci058e:78/058e:78:00.0/nvme/nvme3/nvme1c3n1
# ls -l /sys/block/
lrwxrwxrwx. 1 root root 0 Aug 11 12:30 nvme1c1n1 -> ../devices/pci052e:78/052e:78:00.0/nvme/nvme1/nvme1c1n1
lrwxrwxrwx. 1 root root 0 Aug 11 12:30 nvme1c3n1 -> ../devices/pci058e:78/058e:78:00.0/nvme/nvme3/nvme1c3n1

As we see above, /sys/block/nvme1n1/multipath/nvme1c1n1 is soft link to /sys/block/nvme1c1n1 and similarly /sys/block/nvme1n1/multipath/nvme1c3n1 is soft link to /sys/block/nvme1c3n1.

For round-robin I/O policy, we could easily infer from the above output that I/O workload targeted to nvme1n1 would toggle across nvme1c1n1 and nvme1c3n1.

We also create two new sysfs attribute files named "numa_nodes" and "queue_depth" under /sys/block/nvmeXcYnZ.

# cat  /sys/block/nvme1n1/multipath/nvme1c1n1/numa_nodes 
0-1
# cat  /sys/block/nvme1n1/multipath/nvme1c3n1/numa_nodes 
2-3

For I/O policy numa, above output signifies that I/O workload targeted to nvme1n1 and running on node 0 and 1 would prefer using path nvm1c1n1. Similarly, I/O workload targeted to nvme1n1 and running on node 2 and 3 would prefer using path nvme1c3n1.

Quoted 9 lines
# cat /sys/block/nvme1n1/multipath/nvme1c1n1/queue_depth
423
# cat /sys/block/nvme1n1/multipath/nvme1c3n1/queue_depth
425

For I/O policy queue-depth, above output signifies that for I/O workload targeted to nvme1n1, we have got two paths nvme1c1n1 and nvme1c3n1 and the queue_depth for each path is 423 and 425 respectively.

A gentle ping about the above proposed solution.

As you proposed about 1 file 1 attribute value, I reworked the proposal and would like to know if you have any further feedback/comment. If this looks good then shall I resend the RFC v3 with the above changes incorporated?

Thanks,
--Nilay