Skip to contentSkip to conversation

incorrect sysfs reported by realpath for nvme disk when nvme-core:multipath enabled

DISCUSSION Yi Zhang 3 participants 5 messages Sep 13, 2019 – Sep 16, 2019 1 message not in the archive NVMe

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

Yi Zhang

author
#1 lore raw

Hello

When I use 'realpath' to get the nvme sysfs on latest 5.3.0-rc8 for NVMe SSD, seems it reported incorrectly sysfs, could anyone help check it?

[1] nvme-core multipath=Y

# lspci | grep -i non
86:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller 172Xa/172Xb (rev 01)
af:00.0 Non-Volatile memory controller: Intel Corporation NVMe DC SSD [3DNAND, Beta Rock Controller]
# nvme list
Node             SN                   Model                                    Namespace Usage                      Format           FW Rev  
---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme0n1     BTLE8190003X2P0IGN   Dell Express Flash NVMe P4600 2.0TB AIC  1           2.00  TB /   2.00  TB      4 KiB +  0 B   QDV1DP15
/dev/nvme1n1           S39WNA0K201139 Dell Express Flash PM1725a 1.6TB AIC     1           1.60  TB /   1.60  TB    512   B +  0 B   1.1.1 
# cat /sys/module/nvme_core/parameters/multipath 
Y
# lsblk 
NAME                         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                            8:0    0 222.5G  0 disk 
├─sda1                         8:1    0   600M  0 part /boot/efi
├─sda2                         8:2    0     1G  0 part /boot
└─sda3                         8:3    0 220.9G  0 part 
  ├─rhel_rdma--perf--07-root 253:0    0    50G  0 lvm  /
  ├─rhel_rdma--perf--07-swap 253:1    0     4G  0 lvm  [SWAP]
  └─rhel_rdma--perf--07-home 253:2    0 166.9G  0 lvm  /home
nvme0n1                      259:0    0   1.8T  0 disk 
nvme1n1                      259:2    0   1.5T  0 disk 
# realpath  /sys/dev/block/259:0
/sys/devices/pci0000:ae/0000:ae:00.0/0000:af:00.0/nvme/nvme1/nvme0n1   --------> should it be /sys/devices/pci0000:ae/0000:ae:00.0/0000:af:00.0/nvme/nvme0/nvme0n1 ?
# realpath  /sys/dev/block/259:2
/sys/devices/virtual/nvme-subsystem/nvme-subsys1/nvme1n1               --------> No pci address returned, bug?

[2] nvme-core multipath=N

# rmmod nvme
# rmmod nvme-core
# nvme list
# modprobe nvme-core multipath=N
# cat /sys/module/nvme_core/parameters/multipath
N
[root@rdma-perf-07 ~]$ modprobe nvme
NAME                         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                            8:0    0 222.5G  0 disk 
├─sda1                         8:1    0   600M  0 part /boot/efi
├─sda2                         8:2    0     1G  0 part /boot
└─sda3                         8:3    0 220.9G  0 part 
  ├─rhel_rdma--perf--07-root 253:0    0    50G  0 lvm  /
  ├─rhel_rdma--perf--07-swap 253:1    0     4G  0 lvm  [SWAP]
  └─rhel_rdma--perf--07-home 253:2    0 166.9G  0 lvm  /home
nvme1n1                      259:0    0   1.8T  0 disk 
nvme0n1                      259:2    0   1.5T  0 disk 

[root@rdma-perf-07 ~]$ realpath /sys/dev/block/259:0 /sys/devices/pci0000:ae/0000:ae:00.0/0000:af:00.0/nvme/nvme1/nvme1n1

[root@rdma-perf-07 ~]$ realpath /sys/dev/block/259:2 /sys/devices/pci0000:85/0000:85:02.0/0000:86:00.0/nvme/nvme0/nvme0n1

Best Regards,
Yi Zhang

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

maintainer
#2 lore raw
Quoted 2 lines · Yi Zhang
# realpath  /sys/dev/block/259:0
/sys/devices/pci0000:ae/0000:ae:00.0/0000:af:00.0/nvme/nvme1/nvme0n1   --------> should it be /sys/devices/pci0000:ae/0000:ae:00.0/0000:af:00.0/nvme/nvme0/nvme0n1 ?

The sysfs links are correct. The controller-namespacenaming is just not related, which is what you're observing.

This was confusing to enough people that we've since changed the naming to something people expect to see, but that's staged in 5.4. Given the number of people who find this confusing, this might be worth making a stable patch.

Quoted 2 lines
# realpath  /sys/dev/block/259:2
/sys/devices/virtual/nvme-subsystem/nvme-subsys1/nvme1n1               --------> No pci address returned, bug?

For mulitpath capable controllers, the parent of a namespace is the subsystem rather than a controller. The subsystem is a virtual device that doesn't have a physical pci address.

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

maintainer
#3 lore raw
Quoted 11 lines
Quoted 3 lines
# realpath  /sys/dev/block/259:0
/sys/devices/pci0000:ae/0000:ae:00.0/0000:af:00.0/nvme/nvme1/nvme0n1   --------> should it be /sys/devices/pci0000:ae/0000:ae:00.0/0000:af:00.0/nvme/nvme0/nvme0n1 ?

The sysfs links are correct. The controller-namespacenaming is just not related, which is what you're observing.

This was confusing to enough people that we've since changed the naming to something people expect to see, but that's staged in 5.4. Given the number of people who find this confusing, this might be worth making a stable patch.

I agree here.

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

maintainer
#4 lore raw
Quoted 6 lines · Sagi Grimberg
Quoted 4 lines

This was confusing to enough people that we've since changed the naming to something people expect to see, but that's staged in 5.4. Given the number of people who find this confusing, this might be worth making a stable patch.

I agree here.

Okay, I'll forward the commit to stable once merged to mainline. I don't think we can count on the stable bot to pick this one up. We only need this back to 4.19 as all the previous stables with this naming scheme have be EOL'ed.

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

author
#5 lore raw
Quoted 6 lines · Keith Busch
Quoted 5 lines · Sagi Grimberg
Quoted 4 lines

This was confusing to enough people that we've since changed the naming to something people expect to see, but that's staged in 5.4. Given the number of people who find this confusing, this might be worth making a stable patch.

I agree here.

Hi Keith

Thanks for the info, I just test the nvme-5.4, and it works as expect.

[root@rdma-perf-07 ~]$ lsblk
NAME                         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT sda                            8:0    0 222.5G  0 disk

├─sda1                         8:1    0   600M  0 part /boot/efi
├─sda2                         8:2    0     1G  0 part /boot
└─sda3                         8:3    0 220.9G  0 part
  ├─rhel_rdma--perf--07-root 253:0    0    50G  0 lvm  /
  ├─rhel_rdma--perf--07-swap 253:1    0     4G  0 lvm  [SWAP]
  └─rhel_rdma--perf--07-home 253:2    0 166.9G  0 lvm  /home

nvme1n1                      259:0    0   1.8T  0 disk └─nvme1n1p1                  259:1    0   1.8T  0 part nvme0n1                      259:3    0   1.5T  0 disk [root@rdma-perf-07 ~]$ realpath /sys/dev/block/259:0 /sys/devices/pci0000:ae/0000:ae:00.0/0000:af:00.0/nvme/nvme1/nvme1n1 [root@rdma-perf-07 ~]$ realpath /sys/dev/block/259:3 /sys/devices/virtual/nvme-subsystem/nvme-subsys0/nvme0n1

Thanks

Yi

Quoted 9 lines

Okay, I'll forward the commit to stable once merged to mainline. I don't think we can count on the stable bot to pick this one up. We only need this back to 4.19 as all the previous stables with this naming scheme have be EOL'ed.

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme