Skip to contentSkip to conversation

Ensure ordered namespace registration during async scan

PATCH In review Maurizio Lombardi 7 participants 20 messages Feb 25 – Jun 25

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

v3 cover Ensure ordered namespace registration during async scan

The NVMe fully asynchronous namespace scanning introduced in commit 4e893ca81170 ("nvme-core: scan namespaces asynchronously") significantly improved discovery times. However, it also introduced non-deterministic ordering for namespace registration.

While kernel device names (/dev/nvmeXnY) are not guaranteed to be stable across reboots, this unpredictable ordering has caused considerable user confusion and has been perceived as a regression, leading to multiple bug reports.

This series introduces a solution to enforce strict sequential registration based on NSID order, entirely preserving the performance benefits of the asynchronous scan approach.

Instead of adding an NVMe-specific hack, this series abstracts the serialization mechanism currently open-coded in the SCSI subsystem (drivers/scsi/scsi_scan.c) into a generic library helper called the completion chain (compl_chain).

By enforcing a strict First-In, First-Out (FIFO) completion order for asynchronous tasks, we can ensure that namespaces are allocated and registered sequentially without blocking the underlying parallel discovery processes.

PATCH 3 Refactors the existing SCSI asynchronous scanning implementation to use the new compl_chain helper, stripping out the custom, open-coded task list and reducing code duplication.

Original code:

$ nvme list
Node                  Generic               Namespace
--------------------- --------------------- ----------
/dev/nvme0n1          /dev/ng0n1            0x2
/dev/nvme0n2          /dev/ng0n2            0x1
/dev/nvme0n3          /dev/ng0n3            0x5
/dev/nvme0n4          /dev/ng0n4            0x3
/dev/nvme0n5          /dev/ng0n5            0x4
[...]
/dev/nvme0n10         /dev/ng0n10           0xa
/dev/nvme0n11         /dev/ng0n11           0x8
/dev/nvme0n12         /dev/ng0n12           0x12
/dev/nvme0n13         /dev/ng0n13           0x17
/dev/nvme0n14         /dev/ng0n14           0xc
/dev/nvme0n15         /dev/ng0n15           0x11
/dev/nvme0n16         /dev/ng0n16           0x14
/dev/nvme0n17         /dev/ng0n17           0x13
/dev/nvme0n18         /dev/ng0n18           0xe
/dev/nvme0n19         /dev/ng0n19           0xf

With this patch:

$ nvme list
Node                  Generic               Namespace
--------------------- --------------------- ----------
/dev/nvme0n1          /dev/ng0n1            0x1
/dev/nvme0n2          /dev/ng0n2            0x2
/dev/nvme0n3          /dev/ng0n3            0x3
/dev/nvme0n4          /dev/ng0n4            0x4
/dev/nvme0n5          /dev/ng0n5            0x5
/dev/nvme0n6          /dev/ng0n6            0x6
[...]
/dev/nvme0n10         /dev/ng0n10           0xa
/dev/nvme0n11         /dev/ng0n11           0xb
/dev/nvme0n12         /dev/ng0n12           0xc
/dev/nvme0n13         /dev/ng0n13           0xd
/dev/nvme0n14         /dev/ng0n14           0xe
/dev/nvme0n15         /dev/ng0n15           0xf
/dev/nvme0n16         /dev/ng0n16           0x10
/dev/nvme0n17         /dev/ng0n17           0x11
/dev/nvme0n18         /dev/ng0n18           0x12
/dev/nvme0n19         /dev/ng0n19           0x13

V3: fixed some comments
PATCH 3: declare scanning_hosts as static remove "extern" keyword from scsi_complete_async_scans() prototype declaration

V2: create the compl_chain helper that both SCSI and NVMe can share

Maurizio Lombardi (3):
lib: Introduce completion chain helper
nvme-core: register namespaces in order during async scan
scsi: Convert async scanning to use the completion chain helper

 drivers/nvme/host/core.c    |  94 +++++++++++++++++-----------
 drivers/nvme/host/nvme.h    |   2 +
 drivers/scsi/scsi_priv.h    |   2 +-
 drivers/scsi/scsi_scan.c    |  68 +++------------------
 include/linux/compl_chain.h |  35 +++++++++++
 lib/Makefile                |   2 +-
 lib/compl_chain.c           | 118 ++++++++++++++++++++++++++++++++++++
 7 files changed, 225 insertions(+), 96 deletions(-)
 create mode 100644 include/linux/compl_chain.h
 create mode 100644 lib/compl_chain.c
Signature · 1 line
2.53.0
#1 lore raw

PATCH v3 1/3 lib: Introduce completion chain helper

Introduce a new helper library, the completion chain, designed to serialize asynchronous operations that must complete in a strict First-In, First-Out (FIFO) order.

Certain workflows, particularly in storage drivers, require operations to complete in the same sequence they were submitted.
This helper provides a generic mechanism to enforce this ordering.

compl_chain: The main structure representing the queue of operations
compl_chain_entry: An entry embedded in a per-operation structure

The typical usage pattern is:

* An operation is enqueued by calling compl_chain_add().

* The worker thread for the operation calls compl_chain_wait(), which blocks until the previously enqueued operation has finished.

* After the work is done, the thread calls compl_chain_complete(). This signals the next operation in the chain that it can now proceed and removes the current entry from the list.

---
 include/linux/compl_chain.h |  35 +++++++++++
 lib/Makefile                |   2 +-
 lib/compl_chain.c           | 118 ++++++++++++++++++++++++++++++++++++
 3 files changed, 154 insertions(+), 1 deletion(-)
 create mode 100644 include/linux/compl_chain.h
 create mode 100644 lib/compl_chain.c
include/linux/compl_chain.h new file +35 -0
new file mode 100644
index 000000000000..a2bf271144e0
@@ -0,0 +1,35 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_COMPLETION_CHAIN_H
+#define _LINUX_COMPLETION_CHAIN_H
+
+#include <linux/list.h>
+#include <linux/completion.h>
+#include <linux/spinlock.h>
+
+struct compl_chain {
+	spinlock_t lock;
+	struct list_head list;
+};
+
+#define COMPL_CHAIN_INIT(name) \
+	{ .lock = __SPIN_LOCK_UNLOCKED((name).lock), \
+	  .list = LIST_HEAD_INIT((name).list) }
+
+#define DEFINE_COMPL_CHAIN(name) \
+	struct compl_chain name = COMPL_CHAIN_INIT(name)
+
+struct compl_chain_entry {
+	struct compl_chain *chain;
+	struct list_head list;
+	struct completion prev_finished;
+};
+
+void compl_chain_init(struct compl_chain *chain);
+void compl_chain_add(struct compl_chain *chain,
+			struct compl_chain_entry *entry);
+void compl_chain_wait(struct compl_chain_entry *entry);
+void compl_chain_complete(struct compl_chain_entry *entry);
+bool compl_chain_pending(struct compl_chain_entry *entry);
+void compl_chain_flush(struct compl_chain *chain);
+
+#endif /* _LINUX_COMPLETION_CHAIN_H */
lib/Makefile +1 -1
index 1b9ee167517f..c3ccd82bb190 100644
@@ -58,7 +58,7 @@ obj-y += bcd.o sort.o parser.o debug_locks.o random32.o \
 	 bsearch.o find_bit.o llist.o lwq.o memweight.o kfifo.o \
 	 percpu-refcount.o rhashtable.o base64.o \
 	 once.o refcount.o rcuref.o usercopy.o errseq.o bucket_locks.o \
-	 generic-radix-tree.o bitmap-str.o
+	 generic-radix-tree.o bitmap-str.o compl_chain.o
 obj-y += string_helpers.o
 obj-y += hexdump.o
 obj-$(CONFIG_TEST_HEXDUMP) += test_hexdump.o
lib/compl_chain.c new file +118 -0

118 lines not on this page · raw message

Signature · 1 line
2.53.0
#2 lore raw

PATCH v3 2/3 nvme-core: register namespaces in order during async scan

In reply to Maurizio Lombardi

The fully asynchronous namespace scanning, while fast, can result in namespaces being allocated and registered out of order. This leads to unpredictable device naming across reboots which can be confusing for users.

To solve this, introduce a serialization mechanism for the asynchronous namespace scan. This is achieved by using the generic compl_chain helper, which ensures that the initialization of one namespace (nvme_alloc_ns) completes before the next one begins.

This approach preserves the performance benefits of asynchronous identification while guaranteeing that the final device registration occurs in the correct order.

Performance testing shows that this change has no noticeable impact on scan times compared to the fully asynchronous method.

High latency NVMe/TCP, ~150ms ping, 100 namespaces

Synchronous namespace scan (RHEL-10.1): 32375ms
Fully async namespace scan (7.0-rc1): 2543ms
Async namespace scan with dependency chain (7.0-rc1): 2431ms

Low latency NVMe/TCP, ~0.2ms ping, 100 namespaces

Synchronous namespace scan (RHEL-10.1): 352ms Fully async namespace scan (7.0-rc1): 248ms
Async namespace scan with dependency chain (7.0-rc1): 191ms

---
 drivers/nvme/host/core.c | 94 +++++++++++++++++++++++++---------------
 drivers/nvme/host/nvme.h |  2 +
 2 files changed, 62 insertions(+), 34 deletions(-)
drivers/nvme/host/core.c +60 -34
index f5ebcaa2f859..d186c0082cc8 100644
@@ -4105,13 +4105,27 @@ static void nvme_ns_add_to_ctrl_list(struct nvme_ns *ns)
 	list_add_rcu(&ns->list, &ns->ctrl->namespaces);
 }
 
-static void nvme_alloc_ns(struct nvme_ctrl *ctrl, struct nvme_ns_info *info)
+/**
+ * struct async_scan_task - keeps track of controller & NSID to scan
+ * @entry:	link to the completion chain list
+ * @ctrl:	Controller on which namespaces are being scanned
+ * @nsid:	The NSID to scan
+ */
+struct async_scan_task {
+	struct compl_chain_entry chain_entry;
+	struct nvme_ctrl *ctrl;
+	u32 nsid;
+};
+
+static void nvme_alloc_ns(struct nvme_ctrl *ctrl, struct nvme_ns_info *info,
+				struct compl_chain_entry *cc_entry)
 {
 	struct queue_limits lim = { };
 	struct nvme_ns *ns;
 	struct gendisk *disk;
 	int node = ctrl->numa_node;
 	bool last_path = false;
+	int r;
 
 	ns = kzalloc_node(sizeof(*ns), GFP_KERNEL, node);
 	if (!ns)
@@ -4134,7 +4148,19 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, struct nvme_ns_info *info)
 	ns->ctrl = ctrl;
 	kref_init(&ns->kref);
 
-	if (nvme_init_ns_head(ns, info))
+	/*
+	 * Wait for the previous async task to finish before
+	 * allocating the namespace.
+	 */
+	if (cc_entry)
+		compl_chain_wait(cc_entry);
+
+	r = nvme_init_ns_head(ns, info);
+
+	if (cc_entry)
+		compl_chain_complete(cc_entry);
+
+	if (r)
 		goto out_cleanup_disk;
 
 	/*
@@ -4309,7 +4335,8 @@ static void nvme_validate_ns(struct nvme_ns *ns, struct nvme_ns_info *info)
 		nvme_ns_remove(ns);
 }
 
-static void nvme_scan_ns(struct nvme_ctrl *ctrl, unsigned nsid)
+static void nvme_scan_ns(struct nvme_ctrl *ctrl, unsigned int nsid,
+				struct compl_chain_entry *cc_entry)
 {
 	struct nvme_ns_info info = { .nsid = nsid };
 	struct nvme_ns *ns;
@@ -4348,40 +4375,30 @@ static void nvme_scan_ns(struct nvme_ctrl *ctrl, unsigned nsid)
 
 	ns = nvme_find_get_ns(ctrl, nsid);
 	if (ns) {
+		/* Release the chain early so the next task can proceed */
+		if (cc_entry)
+			compl_chain_complete(cc_entry);
 		nvme_validate_ns(ns, &info);
 		nvme_put_ns(ns);
 	} else {
-		nvme_alloc_ns(ctrl, &info);
+		nvme_alloc_ns(ctrl, &info, cc_entry);
 	}
 }
 
-/**
- * struct async_scan_info - keeps track of controller & NSIDs to scan
- * @ctrl:	Controller on which namespaces are being scanned
- * @next_nsid:	Index of next NSID to scan in ns_list
- * @ns_list:	Pointer to list of NSIDs to scan
- *
- * Note: There is a single async_scan_info structure shared by all instances
- * of nvme_scan_ns_async() scanning a given controller, so the atomic
- * operations on next_nsid are critical to ensure each instance scans a unique
- * NSID.
- */
-struct async_scan_info {
-	struct nvme_ctrl *ctrl;
-	atomic_t next_nsid;
-	__le32 *ns_list;
-};
-
 static void nvme_scan_ns_async(void *data, async_cookie_t cookie)
 {
-	struct async_scan_info *scan_info = data;
-	int idx;
-	u32 nsid;
+	struct async_scan_task *task = data;
 
-	idx = (u32)atomic_fetch_inc(&scan_info->next_nsid);
-	nsid = le32_to_cpu(scan_info->ns_list[idx]);
+	nvme_scan_ns(task->ctrl, task->nsid, &task->chain_entry);
 
-	nvme_scan_ns(scan_info->ctrl, nsid);
+	/*
+	 * If the task failed early and returned without completing the
+	 * chain entry, ensure the chain progresses safely.
+	 */
+	if (compl_chain_pending(&task->chain_entry))
+		compl_chain_complete(&task->chain_entry);
+
+	kfree(task);
 }
 
 static void nvme_remove_invalid_namespaces(struct nvme_ctrl *ctrl,
@@ -4411,14 +4428,12 @@ static int nvme_scan_ns_list(struct nvme_ctrl *ctrl)
 	u32 prev = 0;
 	int ret = 0, i;
 	ASYNC_DOMAIN(domain);
-	struct async_scan_info scan_info;
+	struct async_scan_task *task;
 
 	ns_list = kzalloc(NVME_IDENTIFY_DATA_SIZE, GFP_KERNEL);
 	if (!ns_list)
 		return -ENOMEM;
 
-	scan_info.ctrl = ctrl;
-	scan_info.ns_list = ns_list;
 	for (;;) {
 		struct nvme_command cmd = {
 			.identify.opcode	= nvme_admin_identify,
@@ -4434,20 +4449,30 @@ static int nvme_scan_ns_list(struct nvme_ctrl *ctrl)
 			goto free;
 		}
 
-		atomic_set(&scan_info.next_nsid, 0);
 		for (i = 0; i < nr_entries; i++) {
 			u32 nsid = le32_to_cpu(ns_list[i]);
 
 			if (!nsid)	/* end of the list? */
 				goto out;
-			async_schedule_domain(nvme_scan_ns_async, &scan_info,
+
+			task = kmalloc_obj(*task);
+			if (!task) {
+				ret = -ENOMEM;
+				goto out;
+			}
+
+			task->nsid = nsid;
+			task->ctrl = ctrl;
+			compl_chain_add(&ctrl->scan_chain, &task->chain_entry);
+
+			async_schedule_domain(nvme_scan_ns_async, task,
 						&domain);
 			while (++prev < nsid)
 				nvme_ns_remove_by_nsid(ctrl, prev);
 		}
-		async_synchronize_full_domain(&domain);
 	}
  out:
+	async_synchronize_full_domain(&domain);
 	nvme_remove_invalid_namespaces(ctrl, prev);
  free:
 	async_synchronize_full_domain(&domain);
@@ -4466,7 +4491,7 @@ static void nvme_scan_ns_sequential(struct nvme_ctrl *ctrl)
 	kfree(id);
 
 	for (i = 1; i <= nn; i++)
-		nvme_scan_ns(ctrl, i);
+		nvme_scan_ns(ctrl, i, NULL);
 
 	nvme_remove_invalid_namespaces(ctrl, nn);
 }
@@ -5094,6 +5119,7 @@ int nvme_init_ctrl(struct nvme_ctrl *ctrl, struct device *dev,
 
 	mutex_init(&ctrl->scan_lock);
 	INIT_LIST_HEAD(&ctrl->namespaces);
+	compl_chain_init(&ctrl->scan_chain);
 	xa_init(&ctrl->cels);
 	ctrl->dev = dev;
 	ctrl->ops = ops;
drivers/nvme/host/nvme.h +2 -0
index 9a5f28c5103c..95f8c40ec86b 100644
@@ -17,6 +17,7 @@
 #include <linux/wait.h>
 #include <linux/t10-pi.h>
 #include <linux/ratelimit_types.h>
+#include <linux/compl_chain.h>
 
 #include <trace/events/block.h>
 
@@ -294,6 +295,7 @@ struct nvme_ctrl {
 	struct blk_mq_tag_set *tagset;
 	struct blk_mq_tag_set *admin_tagset;
 	struct list_head namespaces;
+	struct compl_chain scan_chain;
 	struct mutex namespaces_lock;
 	struct srcu_struct srcu;
 	struct device ctrl_device;
Signature · 1 line
2.53.0
#3 lore raw

Re: [PATCH V3 2/3] nvme-core: register namespaces in order during async scan

kernel test robot noticed the following build warnings:

Hi Maurizio,

kernel test robot noticed the following build warnings:

[auto build test WARNING on jejb-scsi/for-next]
[also build test WARNING on mkp-scsi/for-next linus/master v7.0-rc1 next-20260225] [cannot apply to linux-nvme/for-next]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Maurizio-Lombardi/lib-Introduce-completion-chain-helper/20260226-001842
base:   https://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi.git for-next
patch link:    https://lore.kernel.org/r/20260225161203.76168-3-mlombard%40redhat.com
patch subject: [PATCH V3 2/3] nvme-core: register namespaces in order during async scan
config: x86_64-randconfig-161-20260226 (https://download.01.org/0day-ci/archive/20260226/[email protected]/config)
compiler: gcc-14 (Debian 14.2.0-19) 14.2.0
smatch version: v0.5.0-8994-gd50c5a4c
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260226/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All warnings (new ones prefixed by >>):

>> Warning: drivers/nvme/host/core.c:4117 struct member 'chain_entry' not described in 'async_scan_task'
>> Warning: drivers/nvme/host/core.c:4117 struct member 'chain_entry' not described in 'async_scan_task'
Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
#4 lore raw

PATCH v3 3/3 scsi: Convert async scanning to use the completion chain helper

In reply to Maurizio Lombardi

The asynchronous host scanning logic in scsi_scan.c uses a custom, open-coded implementation to serialize scans. This involves a manually managed list of tasks, each with its own completion, to ensure that hosts are scanned and added to the system in a deterministic order.

Refactors the SCSI async scanning implementation to use the new compl_chain helper. This simplifies the scsi_scan.c code and makes the serialization logic more readable.

---
 drivers/scsi/scsi_priv.h |  2 +-
 drivers/scsi/scsi_scan.c | 68 +++++-----------------------------------
 2 files changed, 9 insertions(+), 61 deletions(-)
drivers/scsi/scsi_priv.h +1 -1
index 7a193cc04e5b..274fdd7edac4 100644
@@ -132,7 +132,7 @@ extern void scsi_exit_procfs(void);
 
 /* scsi_scan.c */
 void scsi_enable_async_suspend(struct device *dev);
-extern int scsi_complete_async_scans(void);
+void scsi_complete_async_scans(void);
 extern int scsi_scan_host_selected(struct Scsi_Host *, unsigned int,
 				   unsigned int, u64, enum scsi_scan_mode);
 extern void scsi_forget_host(struct Scsi_Host *);
drivers/scsi/scsi_scan.c +8 -60

115 lines not on this page · raw message

Signature · 1 line
2.53.0
maintainer
#5 lore raw
In reply to Maurizio Lombardi
Quoted 9 lines · Maurizio Lombardi

The NVMe fully asynchronous namespace scanning introduced in commit 4e893ca81170 ("nvme-core: scan namespaces asynchronously") significantly improved discovery times. However, it also introduced non-deterministic ordering for namespace registration.

While kernel device names (/dev/nvmeXnY) are not guaranteed to be stable across reboots, this unpredictable ordering has caused considerable user confusion and has been perceived as a regression, leading to multiple bug reports.

The nvme-pci driver also probes the controllers asynchronously, which can also create non-determinisitic names. Is that part not a problem?

Just on the suffix part of the namespace's block handle, I have a potential alternate suggestion here. The instance names pulled from the ida guarantee we'll always have unique names for the lifetime of the backing kobject. I introduced that a while ago, but I'm testing this out now and it seems kobject_del is sufficient to reuse that name. The driver already did that to all the objects when deleting the namespace, so there doesn't appear to be a reason to wait for the final kobject_put.

What I'm saying is I may have been mistaken about the naming collision issues and we can just use the head's ns_id to get a consistent and meaningful name based off the backing namespaces. There's some unlikely races with multipath at the moment if we did use ns_id, but I think they're all fixable.

#6 lore raw
Quoted 6 lines · Keith Busch
Quoted 5 lines · Maurizio Lombardi

The NVMe fully asynchronous namespace scanning introduced in commit 4e893ca81170 ("nvme-core: scan namespaces asynchronously") significantly improved discovery times. However, it also introduced non-deterministic ordering for namespace registration.

Quoted 7 lines
Quoted 4 lines

While kernel device names (/dev/nvmeXnY) are not guaranteed to be stable across reboots, this unpredictable ordering has caused considerable user confusion and has been perceived as a regression, leading to multiple bug reports.

The nvme-pci driver also probes the controllers asynchronously, which can also create non-determinisitic names. Is that part not a problem?

Potentially, it is. The difference is that so far no one ever complained about it, while with namespace async scanning we immediately received regression reports, to the point we had to revert the changes and restore the sequential namespaces scan in RHEL.

Quoted 10 lines

Just on the suffix part of the namespace's block handle, I have a potential alternate suggestion here. The instance names pulled from the ida guarantee we'll always have unique names for the lifetime of the backing kobject. I introduced that a while ago, but I'm testing this out now and it seems kobject_del is sufficient to reuse that name. The driver already did that to all the objects when deleting the namespace, so there doesn't appear to be a reason to wait for the final kobject_put.

Quoted 5 lines

What I'm saying is I may have been mistaken about the naming collision issues and we can just use the head's ns_id to get a consistent and meaningful name based off the backing namespaces. There's some unlikely races with multipath at the moment if we did use ns_id, but I think they're all fixable.

Ok, so you'd like to use the namespace's NSID as the suffix.
I also considered this approach, the reason I didn't implemented
it is that I wished to have the async namespace scan performance improvements while preserving the same enumeration we had for years with the sequential scan:

Before the introduction of the async scan, /dev/nvme0n1 always pointed to the first entry of the NSID list, /dev/nvme0n2 to the second entry and so on.

With your proposal, if a user has sparse NSIDs (1, 10, 333)
then he will get /dev/nvme0n1, /dev/nvme0n10, /dev/nvme0n333.
On one hand, yes, they are "more stable" and more meaningful too, on the other hand this breaks the assumption of contiguous naming. This might not be a problem for the mainline kernel, but I suspect we will have people complaining again that the /dev/nvmeXnY enumeration changed

Maurizio

maintainer
#7 lore raw
Quoted 6 lines · Maurizio Lombardi

With your proposal, if a user has sparse NSIDs (1, 10, 333)
then he will get /dev/nvme0n1, /dev/nvme0n10, /dev/nvme0n333.
On one hand, yes, they are "more stable" and more meaningful too, on the other hand this breaks the assumption of contiguous naming. This might not be a problem for the mainline kernel, but I suspect we will have people complaining again that the /dev/nvmeXnY enumeration changed

The bonus of using the nsid is that it will always enumerate with the same name even after you alter the other attached namespaces.

#8 lore raw
In reply to Maurizio Lombardi
Quoted 12 lines · Maurizio Lombardi
Quoted 11 lines · Keith Busch
Quoted 9 lines · Maurizio Lombardi

The NVMe fully asynchronous namespace scanning introduced in commit 4e893ca81170 ("nvme-core: scan namespaces asynchronously") significantly improved discovery times. However, it also introduced non-deterministic ordering for namespace registration.

While kernel device names (/dev/nvmeXnY) are not guaranteed to be stable across reboots, this unpredictable ordering has caused considerable user confusion and has been perceived as a regression, leading to multiple bug reports.

Quoted 7 lines
Quoted 2 lines

The nvme-pci driver also probes the controllers asynchronously, which can also create non-determinisitic names. Is that part not a problem?

Potentially, it is. The difference is that so far no one ever complained about it, while with namespace async scanning we immediately received regression reports, to the point we had to revert the changes and restore the sequential namespaces scan in RHEL.

It's worse than this. Yes, in RHEL we carry out of tree patches to tun off the async scanning with SCSI, and we reverted this async namespace scanning patch in NVMe.

We had to do this because, as soon as we turned these async scanning mechanisms on, we immediately received customer escalations. Customer were not able to upgrade their systems. We have customer issues and complaints open about this and we see this async namespace scanning as a barrier to adoption with NVMEe - especially with NVME-OF which tends to have many more Namespaces than PCIe.

We've talked about this at LSF/MM - more than once - and several solutions have been proposed in the past, but nothing ever happened.

And yes, the PCIe async discovery stuff does cause some problems. The difference is: the PCIe bus configuration does not change nearly as often as, e.g., the nvme namespace configuration in a fabric, so customers don't notice the changing pci ids. Unless some one is going lots of hot unplugging and plugging with their PCI bus, the PCI ids typically don't change at all.

So from boot to boot, pci id don't usually change. This async namespace scanning causes the namespace ids to change with every reboot, especially on a system with 100's of nvme-of namespaces.

So, we really need this change, or something like this, to be accepted upstream.

/John

maintainer
#9 lore raw
Quoted 7 lines · John Meneghini

It's worse than this. Yes, in RHEL we carry out of tree patches to tun off the async scanning with SCSI, and we reverted this async namespace scanning patch in NVMe.

We had to do this because, as soon as we turned these async scanning mechanisms on, we immediately received customer escalations. Customer were not able to upgrade their systems. We have customer issues and complaints open about this and we see this async namespace scanning as a barrier to adoption with NVMEe - especially with NVME-OF which tends to have many more Namespaces than PCIe.

Sounds like some people just don't know how to use labels or persistent names. Relying on /dev/nvmeXnY or /dev/sdX to always be a handle to the same device is a fragile solution.

Quoted 3 lines

And yes, the PCIe async discovery stuff does cause some problems. The difference is: the PCIe bus configuration does not change nearly as often as, e.g., the nvme namespace configuration in a fabric, so customers don't notice the changing pci ids. Unless some one is going lots of hot unplugging and plugging with their PCI bus, the PCI ids typically don't change at all.

It's not about the PCI topology changing. The async probe makes it non-deterministic as to which PCI device is going to claim which instance out of the nvme ida since they all try to run concurrently.

#10 lore raw
Quoted 4 lines · Keith Busch
Quoted 3 lines · John Meneghini

It's worse than this. Yes, in RHEL we carry out of tree patches to tun off the async scanning with SCSI, and we reverted this async namespace scanning patch in NVMe.

Quoted 9 lines
Quoted 4 lines

We had to do this because, as soon as we turned these async scanning mechanisms on, we immediately received customer escalations. Customer were not able to upgrade their systems. We have customer issues and complaints open about this and we see this async namespace scanning as a barrier to adoption with NVMEe - especially with NVME-OF which tends to have many more Namespaces than PCIe.

Sounds like some people just don't know how to use labels or persistent names. Relying on /dev/nvmeXnY or /dev/sdX to always be a handle to the same device is a fragile solution.

Yeah. We have undergone this (admittedly, rather painful) process quite some time back for SLES (with the switch from SLES12 to SLES15 if memory serves correctly). Since then our customer seem to be happy with using
persistent device links.

Quoted 7 lines
Quoted 3 lines

And yes, the PCIe async discovery stuff does cause some problems. The difference is: the PCIe bus configuration does not change nearly as often as, e.g., the nvme namespace configuration in a fabric, so customers don't notice the changing pci ids. Unless some one is going lots of hot unplugging and plugging with their PCI bus, the PCI ids typically don't change at all.

It's not about the PCI topology changing. The async probe makes it non-deterministic as to which PCI device is going to claim which instance out of the nvme ida since they all try to run concurrently.

I really would like to go with the nsid based solution from Keith. That would avoid quite some cumbersome code here.

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
maintainer
#11 lore raw
Quoted 2 lines · Hannes Reinecke

I really would like to go with the nsid based solution from Keith. That would avoid quite some cumbersome code here.

I've seen various documentation that assumes the current naming indicates the nsid, so the scheme follows at least some people's expectations. I don't know if we can make everyone happy here, though. :(

I've fixed up most of the multipath races that get us closer to allowing nsid suffix, but there's one left: nvme_remove_head is called outside the subsys lock after detaching the head from the subsystem list. That could cause a subsequent add event to call nvme_alloc_ns() before the mpath side has completed del_gendisk() for the old nsid.

#12 lore raw

Hello Keith,

Quoted 9 lines · Keith Busch
Quoted 2 lines · Hannes Reinecke

I really would like to go with the nsid based solution from Keith. That would avoid quite some cumbersome code here.

I've seen various documentation that assumes the current naming indicates the nsid, so the scheme follows at least some people's expectations. I don't know if we can make everyone happy here, though. :(

Quoted 5 lines

I've fixed up most of the multipath races that get us closer to allowing nsid suffix, but there's one left: nvme_remove_head is called outside the subsys lock after detaching the head from the subsystem list. That could cause a subsequent add event to call nvme_alloc_ns() before the mpath side has completed del_gendisk() for the old nsid.

Did you manage to find a solution for this race?
I just wanted to check whether you had any patches ready for testing since this discussion.

Thanks,
Maurizio

maintainer
#13 lore raw
Quoted 3 lines · Maurizio Lombardi

Did you manage to find a solution for this race?
I just wanted to check whether you had any patches ready for testing since this discussion.

At last month's LSFMM, I heard concerns that this would break something. I don't remember the specifics as I wasn't trying to push the issue. I think this feedback was from the more fabrics focused folks, maybe Randy Jennings, Nilay Shroff or John Meneghini remembers?

#14 lore raw
Quoted 9 lines · Keith Busch
Quoted 3 lines · Maurizio Lombardi

Did you manage to find a solution for this race?
I just wanted to check whether you had any patches ready for testing since this discussion.

At last month's LSFMM, I heard concerns that this would break something. I don't remember the specifics as I wasn't trying to push the issue. I think this feedback was from the more fabrics focused folks, maybe Randy Jennings, Nilay Shroff or John Meneghini remembers?

The problem here is namespace lifetime. The ns_ida is only ever released
at the very last step, so the 'number' of the namespace will only be freed once all references to the namespace are dropped. So if you were trying to keep the namespace number ordered you would
have to delay the creation of the namespace until that point, and you
would induce a serialization between deletion and creation.
Which will drastically prolong the rescan process.
And of course you need to hope that no-one triggers another rescan
process while the old one isn't finished, as that would need to wait
for the previous one to finish, too.
Or you need to introduce a mechanism to terminate an already running
rescan process.
So really, not a good idea.

I fail to see why one cannot use persistent names here.

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
maintainer
#15 lore raw
Quoted 6 lines · Hannes Reinecke

The problem here is namespace lifetime. The ns_ida is only ever released at the very last step, so the 'number' of the namespace will only be freed once all references to the namespace are dropped.
So if you were trying to keep the namespace number ordered you would have to delay the creation of the namespace until that point, and you would induce a serialization between deletion and creation.

Under the proposed scheme, there is no ns_ida. You just use the NSID of the namespace, and that's it. You have to ensure that del_gendisk completed on all heads and paths that was using it prior to bringing up the next one, but that's not really a problem.

The problem I recall has something to do with the nsid not being a consistent value when migrating a namespace to another array or something like that. Not that we currently have proper support for such a thing...

#16 lore raw
Quoted 8 lines · Keith Busch
Quoted 6 lines · Hannes Reinecke

The problem here is namespace lifetime. The ns_ida is only ever released at the very last step, so the 'number' of the namespace will only be freed once all references to the namespace are dropped.
So if you were trying to keep the namespace number ordered you would have to delay the creation of the namespace until that point, and you would induce a serialization between deletion and creation.

Quoted 5 lines

Under the proposed scheme, there is no ns_ida. You just use the NSID of the namespace, and that's it. You have to ensure that del_gendisk completed on all heads and paths that was using it prior to bringing up the next one, but that's not really a problem.

But then you'll have to delay the (re-)scan until the very last reference is gone, otherwise the nsid the scan is about to create will be blocked by the nsid still pending to be deleted.

And we do have blktest nvme/058 as a really nice testcase for executing rapid namespace remapping; that regularly manages to get the 'nsid' and 'ns_ida' numbers getting out of sync.

In general I fail to see the issue here.
Any modern distro should be using persistent device links to access devices, so the actual device name is pretty much irrelevant. We on our side haven't had any issues here since ages.

And scanning has been one of the most complex operations we are doing on nvme, and so I'd really think twice before changing that.

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
#17 lore raw
Quoted 5 lines · Hannes Reinecke

In general I fail to see the issue here.
Any modern distro should be using persistent device links to access devices, so the actual device name is pretty much irrelevant. We on our side haven't had any issues here since ages.

In principle, I agree with you that persistent links are best practice. That said, many users still rely on /dev/nvmeXnY links, for example for some nvme-cli commands. Because kernel 6.11 made these names totally random across reboots, it's causing some confusion for them.

But yes, I totally understand the reason why you don't perceive this as an issue.

Maurizio

maintainer
#18 lore raw
In reply to Hannes Reinecke
Quoted 3 lines · Hannes Reinecke

But then you'll have to delay the (re-)scan until the very last reference is gone, otherwise the nsid the scan is about to create
will be blocked by the nsid still pending to be deleted.

It's not about the last reference. Either something changed or there was some previous misunderstanding when that kobj name uniqueness was introduced to this driver. We just need to wait for del_gendisk to complete, which is usually already serialized in the same scan_work. It doesn't appear to matter if a reference is held on a kobj waiting to be deleted.

Quoted 4 lines

In general I fail to see the issue here.
Any modern distro should be using persistent device links to access devices, so the actual device name is pretty much irrelevant. We on our side haven't had any issues here since ages.

I agree there's not a real issue here. The suggestion is purely a quality-of-life improvement to provide a visual clue that aligns with people's expectations, reducing any surprises. There are people and documentation that still think the "n1" in the nvme0n1 means it's NSID
1. If we can easily align to that, then why not? But I'm not exactly needing this feature either, so if you think there are some "gotcha's" here that may destablize the current scanning, then I have no problem shelving this one.

#19 lore raw
Quoted 6 lines · Keith Busch
Quoted 5 lines

In general I fail to see the issue here.
Any modern distro should be using persistent device links to access devices, so the actual device name is pretty much irrelevant. We on our side haven't had any issues here since ages.

Quoted 8 lines

I agree there's not a real issue here. The suggestion is purely a quality-of-life improvement to provide a visual clue that aligns with people's expectations, reducing any surprises. There are people and documentation that still think the "n1" in the nvme0n1 means it's NSID
1. If we can easily align to that, then why not? But I'm not exactly needing this feature either, so if you think there are some "gotcha's" here that may destablize the current scanning, then I have no problem shelving this one.

There is the issue where the NSID of a specific namespace can change when there are no hosts connected that it is attached to. It does not happen often, but it can need to happen (even without namespace migration). This is not speculative. Over the course of years, our array has had to do it for specific scenarios a couple of times.

However, the more fundamental problem is this:

Quoted 3 lines

That said, many users still rely on /dev/nvmeXnY links, for example for some nvme-cli commands. Because kernel 6.11 made these names totally random across reboots, it's causing some confusion for them.

Relying on NSID to fix references to a specific namespace is not a safe way to operate. And making the device name predictable leads to people taking these shortcuts. To find a specific namespace, the NGUID should be used. That guarantees you are on the same storage. Enabling shortcuts that are not reliable is not a good practice.

Sincerely,
Randy Jennings