Skip to contentSkip to conversation

nvme: Add module reference counting for multipath devices

PATCH In review [email protected] 5 participants 12 messages Aug 12 – Aug 30 NVMe

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 nvme: Add module reference counting for multipath devices

From: Wen Xiong <[email protected]>

Add proper module reference counting to prevent premature unloading of NVMe transport modules while multipath namespaces are still active.

When a namespace is added to a multipath device via nvme_mpath_add_disk(), the underlying transport module (PCIe, FC, RDMA, TCP, etc.) must remain loaded as long as the multipath device references that namespace. Without proper reference counting, the transport module could be unloaded while the multipath device is still using resources from that module, leading to the potential system crashes.

This ensures the transport module remains loaded for the entire lifetime of the multipath namespace association.

---
 drivers/nvme/host/multipath.c | 5 +++++
 1 file changed, 5 insertions(+)
drivers/nvme/host/multipath.c +5 -0
index 9b9a657fa330..707b8f95727d 100644
@@ -1348,6 +1348,8 @@ void nvme_mpath_remove_sysfs_link(struct nvme_ns *ns)
 	sysfs_remove_link_from_group(kobj, nvme_ns_mpath_attr_group.name,
 			dev_name(target));
 	clear_bit(NVME_NS_SYSFS_ATTR_LINK, &ns->flags);
+
+	module_put(ns->ctrl->ops->module);
 }
 
 void nvme_mpath_add_disk(struct nvme_ns *ns, __le32 anagrpid)
@@ -1379,6 +1381,9 @@ void nvme_mpath_add_disk(struct nvme_ns *ns, __le32 anagrpid)
 	if (blk_queue_is_zoned(ns->queue) && ns->head->disk)
 		ns->head->disk->nr_zones = ns->disk->nr_zones;
 #endif
+	if (!try_module_get(ns->ctrl->ops->module))
+		dev_err(disk_to_dev(ns->disk),
+				"Failed to get module reference\n");
 }
 
 void nvme_mpath_remove_disk(struct nvme_ns_head *head)
Signature · 1 line
2.52.0
Tested-by on v2 1/1
#1 lore raw
Quoted 12 lines · On 13/08/26 4:02 am, [email protected] wrote:

From: Wen Xiong <[email protected]>

Add proper module reference counting to prevent premature unloading of NVMe transport modules while multipath namespaces are still active.

When a namespace is added to a multipath device via nvme_mpath_add_disk(), the underlying transport module (PCIe, FC, RDMA, TCP, etc.) must remain loaded as long as the multipath device references that namespace. Without proper reference counting, the transport module could be unloaded while the multipath device is still using resources from that module, leading to the potential system crashes.

Quoted 5 lines

This ensures the transport module remains loaded for the entire lifetime of the multipath namespace association.

---

Hi Wen,

I tested this patch and it is working as expected.

Patch Validation Results
--------------------------------

# lsmod | grep -i nvme
nvme_tcp              262144  0
nvme                  262144  9
nvme_fabrics          262144  1 nvme_tcp
nvme_core             458752  8 nvme_tcp,nvme,nvme_fabrics
nvme_keyring          262144  3 nvme_tcp,nvme_core,nvme_fabrics
nvme_auth             262144  1 nvme_core
# rmmod nvme
rmmod: ERROR: Module nvme is in use.
#
# rmmod nvme_tcp
#
# lsmod | grep -i nvme
nvme                  262144  9
nvme_fabrics          262144  0
nvme_core             458752  7 nvme,nvme_fabrics
nvme_keyring          262144  2 nvme_core,nvme_fabrics
nvme_auth             262144  1 nvme_core
# rmmod nvme_core
rmmod: ERROR: Module nvme_core is in use by: nvme nvme_fabrics
#
# rmmod nvme_fabrics
#
# rmmod nvme_core
rmmod: ERROR: Module nvme_core is in use by: nvme
#
# lsmod | grep -i nvme
nvme                  262144  9
nvme_core             458752  6 nvme
nvme_keyring          262144  1 nvme_core
nvme_auth             262144  1 nvme_core
#

Please add below tag :

Thanks and Regards,

Naresh.

Quoted 26 lines
  drivers/nvme/host/multipath.c | 5 +++++
  1 file changed, 5 insertions(+)
drivers/nvme/host/multipath.c +5 -0
index 9b9a657fa330..707b8f95727d 100644
@@ -1348,6 +1348,8 @@ void nvme_mpath_remove_sysfs_link(struct nvme_ns *ns)
  	sysfs_remove_link_from_group(kobj, nvme_ns_mpath_attr_group.name,
  			dev_name(target));
  	clear_bit(NVME_NS_SYSFS_ATTR_LINK, &ns->flags);
+
+	module_put(ns->ctrl->ops->module);
  }
 
  void nvme_mpath_add_disk(struct nvme_ns *ns, __le32 anagrpid)
@@ -1379,6 +1381,9 @@ void nvme_mpath_add_disk(struct nvme_ns *ns, __le32 anagrpid)
  	if (blk_queue_is_zoned(ns->queue) && ns->head->disk)
  		ns->head->disk->nr_zones = ns->disk->nr_zones;
  #endif
+	if (!try_module_get(ns->ctrl->ops->module))
+		dev_err(disk_to_dev(ns->disk),
+				"Failed to get module reference\n");
  }
 
  void nvme_mpath_remove_disk(struct nvme_ns_head *head)
maintainer
#2 lore raw
In reply to [email protected]
Quoted 12 lines · On 13/08/2026 1:32, [email protected] wrote:

From: Wen Xiong <[email protected]>

Add proper module reference counting to prevent premature unloading of NVMe transport modules while multipath namespaces are still active.

When a namespace is added to a multipath device via nvme_mpath_add_disk(), the underlying transport module (PCIe, FC, RDMA, TCP, etc.) must remain loaded as long as the multipath device references that namespace. Without proper reference counting, the transport module could be unloaded while the multipath device is still using resources from that module, leading to the potential system crashes.

Quoted 2 lines

This ensures the transport module remains loaded for the entire lifetime of the multipath namespace association.

Can you provide details about the issue that this patch is supposed to address?
(e.g. potential system crashes). The driver unload should be able to disconnect all of the connected controllers when tearing down...

maintainer
#3 lore raw
Quoted 15 lines · Sagi Grimberg
Quoted 12 lines · On 13/08/2026 1:32, [email protected] wrote:

From: Wen Xiong <[email protected]>

Add proper module reference counting to prevent premature unloading of NVMe transport modules while multipath namespaces are still active.

When a namespace is added to a multipath device via nvme_mpath_add_disk(), the underlying transport module (PCIe, FC, RDMA, TCP, etc.) must remain loaded as long as the multipath device references that namespace. Without proper reference counting, the transport module could be unloaded while the multipath device is still using resources from that module, leading to the potential system crashes.

Quoted 5 lines
Quoted 2 lines

This ensures the transport module remains loaded for the entire lifetime of the multipath namespace association.

Can you provide details about the issue that this patch is supposed to address?
(e.g. potential system crashes). The driver unload should be able to disconnect all of the connected controllers when tearing down...

OK, I now read the original report regarding nvme root device. please disregard.

author
#4 lore raw
Quoted 1 line · Sagi Grimberg

OK, I now read the original report regarding nvme root device. please disregard.

Hi Sagi,

Thanks for taking the time to review this!

If multipath NVMe is the root device and there is no single NVMe device in the system, the tester can still rmmod nvme because the NVMe module's reference count is 0 (PCIe).

For example,
System has 2 multipath nvme devices(nvme0 has 1 NS and nvme1 has 2 NS).

# nvme list
Node                  Generic               SN                   Model   
                                 Namespace  Usage                      
Format           FW Rev
--------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- --------
/dev/nvme0n1          /dev/ng0n1            S6RUNE0R900042       1.6TB NVMe Gen4 U.2 SSD III              0x1        400.08  GB / 400.08  GB    
  4 KiB +  0 B   REV.SN66
/dev/nvme1n1          /dev/ng1n1            S6KZNE0RA00056       800GB NVMe Gen4 U.2 SSD                  0x1        400.08  GB / 400.08  GB    
  4 KiB +  0 B   REV.SN4B
/dev/nvme1n2          /dev/ng1n2            S6KZNE0RA00056       800GB NVMe Gen4 U.2 SSD

Without patch, reference count of nvme module is 0, "rmmod nvme" causes system crashing. #lsmod|grep nvme
nvme_tcp 262144 0

nvme                  262144  0
nvme_fabrics          262144  1 nvme_tcp
nvme_core             458752  7 nvme_tcp,nvme,nvme_fabrics
nvme_keyring          262144  3 nvme_tcp,nvme_core,nvme_fabrics
nvme_auth             262144  1 nvme_core

static int nvme_ns_open(struct nvme_ns *ns)
{

        /* should never be called due to GENHD_FL_HIDDEN */
        if (WARN_ON_ONCE(nvme_ns_head_multipath(ns->head))) ---------> 
skip multipath devs
                goto fail;
        if (!nvme_get_ns(ns))
                goto fail;
        if (!try_module_get(ns->ctrl->ops->module))
                goto fail_put_ns;

}

#rmmod nvme

[  434.859171] block nvme2n1: no available path - failing I/O
[  434.859204] XFS (nvme2n1p2): log I/O error -5
[  434.859211] XFS (nvme2n1p2): Filesystem has been shut down due to log error (0x2).
[  434.859215] XFS (nvme2n1p2): Please unmount the filesystem and rectify the problem(s).
[root@ltcrain119-lp4 nvme_upstream]# [  439.349490] XFS (dm-0): metadata I/O error in "xfs_imap_to_bp+0x74/0x108 [xfs]" at daddr 0x73d4ca0 len 32 error 5
[  439.349651] XFS (dm-0): metadata I/O error in "xfs_imap_to_bp+0x74/0x108 [xfs]" at daddr 0x73d4ca0 len 32 error 5
[  439.353244] coredump: 26799(fwupd): |/usr/lib/systemd/systemd-coredump pipe failed
[  457.438961] XFS (dm-0): log I/O error -5
[  457.438998] XFS (dm-0): Filesystem has been shut down due to log error (0x2).
[  457.439005] XFS (dm-0): Please unmount the filesystem and rectify the problem(s).

With patch, reference count of nvme is 6 after system boots up. 2 paths for nvme0 and 4 paths for nvme1:
Can't do "rmmod nvme" command now.

#lsmod|grep nvme
# lsmod|grep nvme

nvme_tcp              262144  0
nvme                  262144  6
nvme_fabrics          262144  1 nvme_tcp
nvme_core             458752  7 nvme_tcp,nvme,nvme_fabrics
nvme_keyring          262144  3 nvme_tcp,nvme_core,nvme_fabrics
nvme_auth             262144  1 nvme_core
# rmmod nvme
rmmod: ERROR: Module nvme is in use

If I remove 1 NS nvme device, reference count decreases 6 – 2 = 4 paths in total. # lsmod|grep nvme

nvme_tcp              262144  0
nvme                  262144  4
nvme_fabrics          262144  1 nvme_tcp
nvme_core             458752  7 nvme_tcp,nvme,nvme_fabrics
nvme_keyring          262144  3 nvme_tcp,nvme_core,nvme_fabrics
nvme_auth             262144  1 nvme_core

If you need any additional information or have any other considerations, please let me know. Could you please help review the patch and share any feedback?

Thanks,
Wen

author
#5 lore raw
In reply to Sagi Grimberg
Quoted 1 line · Sagi Grimberg

OK, I now read the original report regarding nvme root device. please disregard.

Hi Keith and Sagi,

For single port nvme device, I found out:
If NS does not have any filesystems configured and the root filesystem is located on another SCSI disk, the NVMe device's reference count is 0. When a filesystem is created on NS, the NVMe device's reference count increases to 1.

With my V2 patch for multipath devices,
Nvme's reference count always increases if NS doesn't have any filesystem configured. If nvme1 has 1 NS without any filesystem, reference count is 2 after system boots up.

Should Nvme's reference count increases if no filesystem configured on a NS?

Thanks,
Wendy

maintainer
#6 lore raw
Quoted 10 lines · Wen Xiong
Quoted 1 line · Sagi Grimberg

OK, I now read the original report regarding nvme root device. please disregard.

Hi Keith and Sagi,

For single port nvme device, I found out:
If NS does not have any filesystems configured and the root filesystem is located on another SCSI disk, the NVMe device's reference count is 0. When a filesystem is created on NS, the NVMe device's reference count increases to 1.

Quoted 5 lines

With my V2 patch for multipath devices,
Nvme's reference count always increases if NS doesn't have any filesystem configured. If nvme1 has 1 NS without any filesystem, reference count is 2 after system boots up.

Should Nvme's reference count increases if no filesystem configured on a NS?

No I dont think it should

#7 lore raw
In reply to [email protected]
Quoted 34 lines · On 8/13/26 4:02 AM, [email protected] wrote:

From: Wen Xiong <[email protected]>

Add proper module reference counting to prevent premature unloading of NVMe transport modules while multipath namespaces are still active.

When a namespace is added to a multipath device via nvme_mpath_add_disk(), the underlying transport module (PCIe, FC, RDMA, TCP, etc.) must remain loaded as long as the multipath device references that namespace. Without proper reference counting, the transport module could be unloaded while the multipath device is still using resources from that module, leading to the potential system crashes.

This ensures the transport module remains loaded for the entire lifetime of the multipath namespace association.

---
  drivers/nvme/host/multipath.c | 5 +++++
  1 file changed, 5 insertions(+)
drivers/nvme/host/multipath.c +2 -0
index 9b9a657fa330..707b8f95727d 100644
@@ -1348,6 +1348,8 @@ void nvme_mpath_remove_sysfs_link(struct nvme_ns *ns)
  	sysfs_remove_link_from_group(kobj, nvme_ns_mpath_attr_group.name,
  			dev_name(target));
  	clear_bit(NVME_NS_SYSFS_ATTR_LINK, &ns->flags);
+
+	module_put(ns->ctrl->ops->module);
  }
 
  void nvme_mpath_add_disk(struct nvme_ns *ns, __le32 anagrpid)

drivers/nvme/host/multipath.c void nvme_mpath_add_disk(struct nvme_ns *ns, __le32 anagrpid) · line 1389

Quoted 9 lines
drivers/nvme/host/multipath.c +3 -0
@@ -1379,6 +1381,9 @@ void nvme_mpath_add_disk(struct nvme_ns *ns, __le32 anagrpid)
  	if (blk_queue_is_zoned(ns->queue) && ns->head->disk)
  		ns->head->disk->nr_zones = ns->disk->nr_zones;
  #endif
+	if (!try_module_get(ns->ctrl->ops->module))
+		dev_err(disk_to_dev(ns->disk),
+				"Failed to get module reference\n");
  }
 
  void nvme_mpath_remove_disk(struct nvme_ns_head *head)

I know I am late to the party here, but I don't think adding the module refcounting in nvme_mpath_{add|remove}_sysfs_link() is the right approach. That would increment the transport module reference for every namespace/path, which means the transport module could not be unloaded until all namespaces using it are deleted, even when none of those namespaces are actually in use. So for instance, with this change now, I can't rmmod nvme.ko until I delete all namespaces/paths created under PCIe nvme subsystem, even though no one is using the namespace/path under that subsystem.

Instead, I think the module reference should track the lifetime of an active user of the multipath namespace. When the namespace is in use, the corresponding transport module(s) need to remain loaded and once the namespace is no longer in use, those references should be released.

Looking at the current code, nvme_ns_head_{open|release}() seems to be a more appropriate place for this. When a multipath namespace is opened, nvme_ns_head_open() could take the required transport module references, and nvme_ns_head_release() could drop them when the last user releases the namespace.

Also, since a multipath head can have paths through different transports , we should not take a reference only to the transport of one namespace/path found through nvme_find_path(), as was done in v1. Instead, when opening the head, take a module reference iterating through each controller reachable from the corresponding NVMe subsystem. This ensures that every transport that can service I/O for the active multipath namespace remains loaded for the duration of its use.

Thanks,
--Nilay

maintainer
#8 lore raw
Quoted 4 lines · Nilay Shroff

when none of those namespaces are actually in use. So for instance, with this change now, I can't rmmod nvme.ko until I delete all namespaces/paths created under PCIe nvme subsystem, even though no one is using the namespace/path under that subsystem.

I'd call that a feature. :)

Quoted 4 lines

Instead, I think the module reference should track the lifetime of an active user of the multipath namespace. When the namespace is in use, the corresponding transport module(s) need to remain loaded and once the namespace is no longer in use, those references should be released.

That's what v1 was trying to do, but it doesn't work. The transports used to reach a namespace may not be the same across the open and close.

author
#9 lore raw
In reply to Nilay Shroff
Quoted 8 lines · Nilay Shroff

Also, since a multipath head can have paths through different transports , we should not take a reference only to the transport of one namespace/path found through nvme_find_path(), as was done in v1. Instead, when opening the head, take a module reference iterating through each controller reachable from the corresponding NVMe subsystem. This ensures that every transport that can service I/O for the active multipath namespace remains loaded for the duration of its use.

I will look into iterating though each controller/each namespace from nvme subsystem.

Thanks,
Wendy

maintainer
#10 lore raw
Quoted 14 lines · Wen Xiong
Quoted 12 lines · Nilay Shroff

Also, since a multipath head can have paths through different transports , we should not take a reference only to the transport of one namespace/path
found through nvme_find_path(), as was done in v1. Instead, when opening the
head, take a module reference iterating through each controller reachable
from the corresponding NVMe subsystem. This ensures that every transport that can service I/O for the active multipath namespace remains loaded for
the duration of its use.

Quoted 2 lines

I will look into iterating though each controller/each namespace from nvme subsystem.

This is not viable. You can add and remove paths to a namespace at any time such that the transports counted on open are not the namespace's transports on close.

#11 lore raw
Quoted 15 lines · Keith Busch
Quoted 14 lines · Wen Xiong
Quoted 12 lines · Nilay Shroff

Also, since a multipath head can have paths through different transports , we should not take a reference only to the transport of one namespace/path
found through nvme_find_path(), as was done in v1. Instead, when opening the
head, take a module reference iterating through each controller reachable
from the corresponding NVMe subsystem. This ensures that every transport that can service I/O for the active multipath namespace remains loaded for
the duration of its use.

Quoted 6 lines
Quoted 2 lines

I will look into iterating though each controller/each namespace from nvme subsystem.

This is not viable. You can add and remove paths to a namespace at any time such that the transports counted on open are not the namespace's transports on close.

Yes correct, and I think we need some additional change in the code to handle this gracefully. I though about it have some initial idea to address this:

1. Add nr_openers to struct nvme_ns_head.

2. When the ns head is opened, iterate through each namespace associated with the head and increment the reference count of its underlying transport module. Then increment nr_openers.

3. If a new ns/path is added while the head is open, check nr_openers and, if it is non-zero, increment the reference count of the corresponding transport module nr_openers times.

4. If an existing ns/path is removed while the head is open, check nr_openers and decrement the reference count of the corresponding transport module nr_openers times.

5. When the ns head is closed, iterate through the namespaces associated with the head and decrement the reference count of each underlying transport module. Then decrement nr_openers.

The above operations are serialized by subsys->lock, so nr_openers serves as the number of users that have actually opened the head node.

For example, suppose we have a shared namespace reachable through TCP and RDMA paths:

1. User opens the head node:
head->nr_openers = 1; tcp_module_ref_count = 1; rdma_module_ref_count = 1
2. The RDMA path is removed:
head->nr_openers = 1; tcp_module_ref_count = 1; rdma_module_ref_count = 0
3. User closes the head node:
head->nr_openers = 0; tcp_module_ref_count = 0; rdma_module_ref_count = 0

Another example with multiple openers:

1. User A opens the head node:
head->nr_openers = 1; tcp_module_ref_count = 1; rdma_module_ref_count = 1
2. A new TCP path is added and linked to the head:
head->nr_openers = 1; tcp_module_ref_count = 2; rdma_module_ref_count = 1
3. User B opens the head node:
head->nr_openers = 2; tcp_module_ref_count = 4; rdma_module_ref_count = 2
4. User A closes the head node:
head->nr_openers = 1; tcp_module_ref_count = 2; rdma_module_ref_count = 1
5. The RDMA path is removed:
head->nr_openers = 1; tcp_module_ref_count = 2; rdma_module_ref_count = 0
6. User B closes the head node:
head->nr_openers = 0; tcp_module_ref_count = 0; rdma_module_ref_count = 0

This way, the transport module references track the actual number of openers and the set of paths associated with the head, even when paths are dynamically added or removed while the head remains open.

Thanks,
--Nilay