nvme: subsystem handle not released on hot removal under raid1 configuration and multipath enabled
Hi all,
I'm trying to run a few tests on multipath capable nvme devices under raid-1 configuration.
When I try to hot remove a device via sysfs, only the block device is removed while the subsystem handle is still in-tact.
Before hot removal:
===================
# nvme list
Node SN Model Namespace Usage Format FW Rev ---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- /dev/nvme0n1 PHKD9325000C375AGN INTEL SSDPD21K375GA 1 375.08 GB / 375.08 GB 512 B + 0 B E2010453 /dev/nvme1n1 PHKD9325000N1P5CGN INTEL SSDPD21K015TA 1 1.50 TB / 1.50 TB 512 B + 0 B E2010453
# ll /sys/block
md127 -> ../devices/virtual/block/md127
nvme0c0n1 -> ../devices/pci0000:64/0000:64:00.0/0000:65:00.0/0000:66:01.0/0000:67:00.0/nvme/nvme0/nvme0c0n1 nvme0n1 -> ../devices/virtual/nvme-subsystem/nvme-subsys0/nvme0n1
nvme1c1n1 -> ../devices/pci0000:64/0000:64:00.0/0000:65:00.0/0000:66:02.0/0000:68:00.0/nvme/nvme1/nvme1c1n1 nvme1n1 -> ../devices/virtual/nvme-subsystem/nvme-subsys1/nvme1n1
# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT nvme0n1 259:1 0 349.3G 0 disk └─md127 9:127 0 349.2G 0 raid1 nvme1n1 259:3 0 1.4T 0 disk └─md127 9:127 0 349.2G 0 raid1
Hot removal: echo 1 > /sys/bus/pci/devices/0000\:67\:00.0/remove
============
After hot removal:
===================
# nvme list
Node SN Model Namespace Usage Format FW Rev ---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- /dev/nvme0n1 -1 0.00 B / 0.00 B 1 B + 0 B /dev/nvme1n1 PHKD9325000N1P5CGN INTEL SSDPD21K015TA 1 1.50 TB / 1.50 TB 512 B + 0 B E2010453
# ll /sys/block
md127 -> ../devices/virtual/block/md127
nvme0n1 -> ../devices/virtual/nvme-subsystem/nvme-subsys0/nvme0n1
nvme1c1n1 -> ../devices/pci0000:64/0000:64:00.0/0000:65:00.0/0000:66:02.0/0000:68:00.0/nvme/nvme1/nvme1c1n1 nvme1n1 -> ../devices/virtual/nvme-subsystem/nvme-subsys1/nvme1n1
# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT nvme0n1 259:1 0 349.3G 0 disk └─md127 9:127 0 349.2G 0 raid1 nvme1n1 259:3 0 1.4T 0 disk └─md127 9:127 0 349.2G 0 raid1
I guess this is because the md-raid1 driver is still holding a reference to the nvme driver (kref count is 2 in the nvme removal path). Is this a bug in nvme multipath? Because it works fine with multipath off.
Please let me know if more information is required :)
Thank you,
Revanth
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Keith Busch
maintainerQuoted 3 lines · Revanth
I guess this is because the md-raid1 driver is still holding a reference to the nvme driver (kref count is 2 in the nvme removal path). Is this a bug in nvme multipath? Because it works fine with multipath off.
Please let me know if more information is required :)
Correct, the md raid is holding a reference. It's not really a bug, but others have commented that this behavior isn't necessarily desirable.
I posted an RFC about this issue here:
http://lists.infradead.org/pipermail/linux-nvme/2020-September/019873.html
The discussion evolved a bit since then, but hasn't had much activity recently. I'm hoping to get back on that conversation soon.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Hi Keith,
Thank you so much for getting back on this :)
Quoted 10 lines · Keith Busch
Quoted 3 lines · Revanth
I guess this is because the md-raid1 driver is still holding a reference to the nvme driver (kref count is 2 in the nvme removal path). Is this a bug in nvme multipath? Because it works fine with multipath off.
Please let me know if more information is required :)Correct, the md raid is holding a reference. It's not really a bug, but others have commented that this behavior isn't necessarily desirable.
I posted an RFC about this issue here:
http://lists.infradead.org/pipermail/linux-nvme/2020-September/019873.html
The patch suggested in the thread seems to fix the issue.
Quoted 3 lines
The discussion evolved a bit since then, but hasn't had much activity recently. I'm hoping to get back on that conversation soon.
It's interesting that the thread ended abruptly. Are there any plans to move forward with this RFC...?
Thanks!
Revanth
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme