Skip to contentSkip to conversation

nvme: subsystem handle not released on hot removal under raid1 configuration and multipath enabled

DISCUSSION Rajashekar, Revanth 2 participants 3 messages Nov 17, 2020 NVMe

lore
Share & tools
lore
#0 lore raw

Hi all,
I'm trying to run a few tests on multipath capable nvme devices under raid-1 configuration.
When I try to hot remove a device via sysfs, only the block device is removed while the subsystem handle is still in-tact.

Before hot removal:
===================
# nvme list
Node             SN                   Model                                    Namespace Usage                      Format           FW Rev ---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- /dev/nvme0n1     PHKD9325000C375AGN   INTEL SSDPD21K375GA                      1         375.08  GB / 375.08  GB    512   B +  0 B   E2010453 /dev/nvme1n1     PHKD9325000N1P5CGN   INTEL SSDPD21K015TA                      1           1.50  TB /   1.50  TB    512   B +  0 B   E2010453

# ll /sys/block
md127 -> ../devices/virtual/block/md127
nvme0c0n1 -> ../devices/pci0000:64/0000:64:00.0/0000:65:00.0/0000:66:01.0/0000:67:00.0/nvme/nvme0/nvme0c0n1 nvme0n1 -> ../devices/virtual/nvme-subsystem/nvme-subsys0/nvme0n1
nvme1c1n1 -> ../devices/pci0000:64/0000:64:00.0/0000:65:00.0/0000:66:02.0/0000:68:00.0/nvme/nvme1/nvme1c1n1 nvme1n1 -> ../devices/virtual/nvme-subsystem/nvme-subsys1/nvme1n1

# lsblk
NAME          MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT nvme0n1       259:1    0 349.3G  0 disk └─md127         9:127  0 349.2G  0 raid1 nvme1n1       259:3    0   1.4T  0 disk └─md127         9:127  0 349.2G  0 raid1

Hot removal: echo 1 > /sys/bus/pci/devices/0000\:67\:00.0/remove
============

After hot removal:
===================
# nvme list
Node             SN                   Model                                    Namespace Usage                      Format           FW Rev ---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- /dev/nvme0n1                                                                   -1          0.00   B /   0.00   B      1   B +  0 B /dev/nvme1n1     PHKD9325000N1P5CGN   INTEL SSDPD21K015TA                      1           1.50  TB /   1.50  TB    512   B +  0 B   E2010453

# ll /sys/block
md127 -> ../devices/virtual/block/md127
nvme0n1 -> ../devices/virtual/nvme-subsystem/nvme-subsys0/nvme0n1
nvme1c1n1 -> ../devices/pci0000:64/0000:64:00.0/0000:65:00.0/0000:66:02.0/0000:68:00.0/nvme/nvme1/nvme1c1n1 nvme1n1 -> ../devices/virtual/nvme-subsystem/nvme-subsys1/nvme1n1

# lsblk
NAME          MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT nvme0n1       259:1    0 349.3G  0 disk └─md127         9:127  0 349.2G  0 raid1 nvme1n1       259:3    0   1.4T  0 disk └─md127         9:127  0 349.2G  0 raid1

I guess this is because the md-raid1 driver is still holding a reference to the nvme driver (kref count is 2 in the nvme removal path). Is this a bug in nvme multipath? Because it works fine with multipath off.
Please let me know if more information is required :)

Thank you,
Revanth

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

maintainer
#1 lore raw
Quoted 3 lines · Revanth

I guess this is because the md-raid1 driver is still holding a reference to the nvme driver (kref count is 2 in the nvme removal path). Is this a bug in nvme multipath? Because it works fine with multipath off.
Please let me know if more information is required :)

Correct, the md raid is holding a reference. It's not really a bug, but others have commented that this behavior isn't necessarily desirable.

I posted an RFC about this issue here:

http://lists.infradead.org/pipermail/linux-nvme/2020-September/019873.html

The discussion evolved a bit since then, but hasn't had much activity recently. I'm hoping to get back on that conversation soon.

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#2 lore raw

Hi Keith,
Thank you so much for getting back on this :)

Quoted 10 lines · Keith Busch
Quoted 3 lines · Revanth

I guess this is because the md-raid1 driver is still holding a reference to the nvme driver (kref count is 2 in the nvme removal path). Is this a bug in nvme multipath? Because it works fine with multipath off.
Please let me know if more information is required :)

Correct, the md raid is holding a reference. It's not really a bug, but others have commented that this behavior isn't necessarily desirable.

I posted an RFC about this issue here:

http://lists.infradead.org/pipermail/linux-nvme/2020-September/019873.html

The patch suggested in the thread seems to fix the issue.

Quoted 3 lines

The discussion evolved a bit since then, but hasn't had much activity recently. I'm hoping to get back on that conversation soon.

It's interesting that the thread ended abruptly.  Are there any plans to move forward with this RFC...?

Thanks!
Revanth

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme