Skip to contentSkip to conversation

cxl: Fix nvdimm_bus race for nvdimm devices

PATCH Superseded Dave Jiang 3 participants 9 messages Feb 11 – Feb 12 CXL

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Dave Jiang

author maintainer
#0 lore raw

v2 cover cxl: Fix nvdimm_bus race for nvdimm devices

v2:
- Try a different approach

A small series to address a race for nvdimm_bus object when using the nvdimm_bridge object.

Dave Jiang (3):
cxl: Move devm_cxl_add_nvdimm_bridge() to cxl_pmem.ko
cxl: Fix race of nvdimm_bus object when creating nvdimm objects
cxl: Clean up cxl_nvdimm devices when removing nvdimm_bus

 drivers/cxl/core/core.h |   1 -
 drivers/cxl/core/pmem.c | 105 ++++++++--------------------------------
 drivers/cxl/cxl.h       |   2 +
 drivers/cxl/pmem.c      | 102 ++++++++++++++++++++++++++++++++++++++
 4 files changed, 125 insertions(+), 85 deletions(-)

base-commit: 63fbf275fa9f18f7020fb8acf54fa107e51d0f23

Signature · 1 line
2.52.0
author maintainer
#1 lore raw

PATCH v2 1/3 cxl: Move devm_cxl_add_nvdimm_bridge() to cxl_pmem.ko

Moving devm_cxl_add_nvdimm_bridge() and related functions to drivers/cxl/cxl_pmem.c, so that cxl_pmem can export a symbol that gives cxl_acpi a depedency on cxl_pmem kernel module. This is a prepatory patch to resolve the issue of a race for nvdimm_bus object that is created during cxl_acpi_probe().

No functional changes besides moving code.

---
 drivers/cxl/core/core.h |  1 -
 drivers/cxl/core/pmem.c | 86 ++---------------------------------------
 drivers/cxl/cxl.h       |  1 +
 drivers/cxl/pmem.c      | 84 ++++++++++++++++++++++++++++++++++++++++
 4 files changed, 88 insertions(+), 84 deletions(-)
drivers/cxl/core/core.h +0 -1
index 007b8aff0238..8ee0c3bb2065 100644
@@ -7,7 +7,6 @@
 #include <cxl/mailbox.h>
 #include <linux/rwsem.h>
 
-extern const struct device_type cxl_nvdimm_bridge_type;
 extern const struct device_type cxl_nvdimm_type;
 extern const struct device_type cxl_pmu_type;
 
drivers/cxl/core/pmem.c +3 -83

104 lines not on this page · raw message

drivers/cxl/cxl.h +1 -0
index 04c673e7cdb0..3b79d5c4765a 100644
@@ -877,6 +877,7 @@ int cxl_dvsec_rr_decode(struct cxl_dev_state *cxlds,
 bool is_cxl_region(struct device *dev);
 
 extern const struct bus_type cxl_bus_type;
+extern const struct device_type cxl_nvdimm_bridge_type;
 
 /*
  * Note, add_dport() is expressly for the cxl_port driver. TODO: investigate a
drivers/cxl/pmem.c +84 -0

90 lines not on this page · raw message

Signature · 1 line
2.52.0
#2 lore raw

Re: [PATCH v2 1/3] cxl: Move devm_cxl_add_nvdimm_bridge() to cxl_pmem.ko

Quoted 44 lines · Dave Jiang

Moving devm_cxl_add_nvdimm_bridge() and related functions to drivers/cxl/cxl_pmem.c, so that cxl_pmem can export a symbol that gives cxl_acpi a depedency on cxl_pmem kernel module. This is a prepatory patch to resolve the issue of a race for nvdimm_bus object that is created during cxl_acpi_probe().

No functional changes besides moving code.

---
 drivers/cxl/core/core.h |  1 -
 drivers/cxl/core/pmem.c | 86 ++---------------------------------------
 drivers/cxl/cxl.h       |  1 +
 drivers/cxl/pmem.c      | 84 ++++++++++++++++++++++++++++++++++++++++
 4 files changed, 88 insertions(+), 84 deletions(-)
drivers/cxl/core/core.h +0 -1
index 007b8aff0238..8ee0c3bb2065 100644
@@ -7,7 +7,6 @@
 #include <cxl/mailbox.h>
 #include <linux/rwsem.h>
 
-extern const struct device_type cxl_nvdimm_bridge_type;
 extern const struct device_type cxl_nvdimm_type;
 extern const struct device_type cxl_pmu_type;
 
drivers/cxl/core/pmem.c +2 -1
index e7b1e6fa0ea0..50557d71a45f 100644
@@ -21,7 +21,8 @@
  * operations, for example, namespace label access commands.
  */
 
-static DEFINE_IDA(cxl_nvdimm_bridge_ida);
+DEFINE_IDA(cxl_nvdimm_bridge_ida);
+EXPORT_SYMBOL_NS_GPL(cxl_nvdimm_bridge_ida, "CXL");
 
 static void cxl_nvdimm_bridge_release(struct device *dev)
 {

drivers/cxl/core/pmem.c const struct device_type cxl_nvdimm_bridge_type = { · line 45

Quoted 4 lines
drivers/cxl/core/pmem.c +1 -0
@@ -41,6 +42,7 @@ const struct device_type cxl_nvdimm_bridge_type = {
 	.release = cxl_nvdimm_bridge_release,
 	.groups = cxl_nvdimm_bridge_attribute_groups,
 };
+EXPORT_SYMBOL_NS_GPL(cxl_nvdimm_bridge_type, "CXL");

-ETOOMANYEXPORTS

What about just copying the approach of:

1f1cb7f0c255 cxl/mem: Arrange for always-synchronous memdev attach

...that did:

struct cxl_memdev *devm_cxl_add_memdev(struct cxl_dev_state *cxlds,
const struct cxl_memdev_attach *attach)

{
        return __devm_cxl_add_memdev(cxlds, attach);
}

EXPORT_SYMBOL_NS_GPL(devm_cxl_add_memdev, "CXL");

In other words keep all the core infra in the core and only have a private export from the core to the driver module.

author maintainer
#3 lore raw

PATCH v2 2/3 cxl: Fix race of nvdimm_bus object when creating nvdimm objects

In reply to Dave Jiang

Found issue during running of cxl-translate.sh unit test. Adding a 3s sleep right before the test seems to make the issue reproduce fairly consistently. The cxl_translate module has dependency on cxl_acpi and causes orphaned nvdimm objects to reprobe after cxl_acpi is removed. The nvdimm_bus object is registered by the cxl_nvb object when cxl_acpi_probe() is called. With the nvdimm_bus object missing, __nd_device_register() will trigger NULL pointer dereference when accessing the dev->parent that points to &nvdimm_bus->dev.

[  192.884510] BUG: kernel NULL pointer dereference, address: 000000000000006c
[  192.895383] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS edk2-20250812-19.fc42 08/12/2025
[  192.897721] Workqueue: cxl_port cxl_bus_rescan_queue [cxl_core]
[  192.899459] RIP: 0010:kobject_get+0xc/0x90
[  192.924871] Call Trace:
[  192.925959]  <TASK>
[  192.926976]  ? pm_runtime_init+0xb9/0xe0
[  192.929712]  __nd_device_register.part.0+0x4d/0xc0 [libnvdimm]
[  192.933314]  __nvdimm_create+0x206/0x290 [libnvdimm]
[  192.936662]  cxl_nvdimm_probe+0x119/0x1d0 [cxl_pmem]
[  192.940245]  cxl_bus_probe+0x1a/0x60 [cxl_core]
[  192.943349]  really_probe+0xde/0x380
[  192.945614]  ? _raw_spin_unlock_irq+0x18/0x40
[  192.948402]  ? __pfx___device_attach_driver+0x10/0x10
[  192.951407]  __driver_probe_device+0xc0/0x150
[  192.953997]  driver_probe_device+0x1f/0xa0
[  192.956456]  __device_attach_driver+0x85/0x130
[  192.959231]  ? _raw_spin_unlock+0x12/0x30
[  192.961615]  bus_for_each_drv+0x6c/0xb0
[  192.963935]  __device_attach+0xad/0x1c0
[  192.966213]  ? __pfx_cxl_rescan_attach+0x10/0x10 [cxl_core]
[  192.969350]  cxl_rescan_attach+0xa/0x20 [cxl_core]
[  192.972063]  bus_for_each_dev+0x63/0xa0
[  192.974367]  process_one_work+0x166/0x340
[  192.976758]  worker_thread+0x258/0x3a0
[  192.979011]  ? __pfx_worker_thread+0x10/0x10
[  192.981487]  kthread+0x108/0x220
[  192.983503]  ? __pfx_kthread+0x10/0x10
[  192.985335]  ? __pfx_kthread+0x10/0x10
[  192.986148]  ret_from_fork+0x246/0x280
[  192.987018]  ? __pfx_kthread+0x10/0x10
[  192.987891]  ret_from_fork_asm+0x1a/0x30
[  192.988768]  </TASK>
[  192.989359] Modules linked in: cxl_pmem(O) cxl_acpi(O-) kmem device_dax dax_cxl dax_pmem nd_pmem nd_btt cxl_mock_mem(O) dax_hmem cxl_pci nd_e820 nfit cxl_mem(O) cxl_port(O) cxl_mock(O) libnvdimm cxl_core(O) fwctl [last unloaded: cxl_translate(O)]
[  192.993533] CR2: 000000000000006c
[  192.994417] ---[ end trace 0000000000000000 ]---

1. Set probe_type of cxl_nvb to PROBE_FORCE_SYNCHRONOUS to ensure the driver is probed synchronously when add_device() is called.
2. Add a check after add_device() for cxl_nvb to ensure the driver is bound, which also means the nvdimm_bus object is registered.
3. Add a check in __nd_device_register() to avoid NULL pointer dereference when nvdimm_bus is not registered. Hold the uport_dev lock of the cxl rootport to sychronize between cxl_acpi_probe() and cxl_devm_cxl_add_nvdimm(). Also hold the cxl_nvb->dev lock to ensure a consistent state of if the driver is bound (or not).
4. Add a check for nvdimm_bus in cxl_nvdimm_probe() to avoid accessing without nvdimm_bus for unbound nvdimm object reprobes.

This fix also requies the commit that moves devm_cxl_add_nvdimm_bridge() from core/pmem.c to pmem.c.

Fixes 21083f51521f ("cxl/pmem: Register 'pmem' / cxl_nvdimm devices")
---
 drivers/cxl/core/pmem.c | 13 +++++++++++++
 drivers/cxl/pmem.c      | 13 +++++++++++++
 2 files changed, 26 insertions(+)
drivers/cxl/core/pmem.c +13 -0
index 50557d71a45f..f7f784047450 100644
@@ -175,6 +175,19 @@ int devm_cxl_add_nvdimm(struct device *host, struct cxl_port *port,
 	if (!cxl_nvb)
 		return -ENODEV;
 
+	/*
+	 * Take the uport_dev lock to guard against race of nvdimm_bus object.
+	 * cxl_acpi_probe() registers the nvdimm_bus and is done under the
+	 * root port uport_dev lock.
+	 *
+	 * Take the cxl_nvb device lock to ensure that cxl_nvb driver is in a
+	 * consistent state. And the driver registers nvdimm_bus.
+	 */
+	guard(device)(cxl_nvb->port->uport_dev);
+	guard(device)(&cxl_nvb->dev);
+	if (!cxl_nvb->nvdimm_bus)
+		return -ENODEV;
+
 	cxl_nvd = cxl_nvdimm_alloc(cxl_nvb, cxlmd);
 	if (IS_ERR(cxl_nvd)) {
 		rc = PTR_ERR(cxl_nvd);
drivers/cxl/pmem.c +13 -0
index 6f010b3cce44..0b200be1fd36 100644
@@ -85,6 +85,15 @@ struct cxl_nvdimm_bridge *devm_cxl_add_nvdimm_bridge(struct device *host,
 	if (rc)
 		goto err;
 
+	/* Ensure that cxl_nvb driver has been bound since it is synchronous. */
+	device_lock(dev);
+	if (!dev->driver) {
+		device_unlock(dev);
+		unregister_nvb(cxl_nvb);
+		return ERR_PTR(-ENODEV);
+	}
+	device_unlock(dev);
+
 	rc = devm_add_action_or_reset(host, unregister_nvb, cxl_nvb);
 	if (rc)
 		return ERR_PTR(rc);
@@ -213,6 +222,9 @@ static int cxl_nvdimm_probe(struct device *dev)
 	struct nvdimm *nvdimm;
 	int rc;
 
+	if (!cxl_nvb->nvdimm_bus)
+		return -ENODEV;
+
 	set_exclusive_cxl_commands(mds, exclusive_cmds);
 	rc = devm_add_action_or_reset(dev, clear_exclusive, mds);
 	if (rc)
@@ -437,6 +449,7 @@ static struct cxl_driver cxl_nvdimm_bridge_driver = {
 	.probe = cxl_nvdimm_bridge_probe,
 	.id = CXL_DEVICE_NVDIMM_BRIDGE,
 	.drv = {
+		.probe_type = PROBE_FORCE_SYNCHRONOUS,
 		.suppress_bind_attrs = true,
 	},
 };
Signature · 1 line
2.52.0
#4 lore raw

Re: [PATCH v2 2/3] cxl: Fix race of nvdimm_bus object when creating nvdimm objects

Quoted 9 lines · Dave Jiang

Found issue during running of cxl-translate.sh unit test. Adding a 3s sleep right before the test seems to make the issue reproduce fairly consistently. The cxl_translate module has dependency on cxl_acpi and causes orphaned nvdimm objects to reprobe after cxl_acpi is removed. The nvdimm_bus object is registered by the cxl_nvb object when cxl_acpi_probe() is called. With the nvdimm_bus object missing, __nd_device_register() will trigger NULL pointer dereference when accessing the dev->parent that points to &nvdimm_bus->dev.

Please trim backtraces of useless info, something like:

 BUG: kernel NULL pointer dereference, address: 000000000000006c
 Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS edk2-20250812-19.fc42 08/12/2025
 Workqueue: cxl_port cxl_bus_rescan_queue [cxl_core]
 RIP: 0010:kobject_get+0xc/0x90
 Call Trace:
  <TASK>
  ? pm_runtime_init+0xb9/0xe0
  __nd_device_register.part.0+0x4d/0xc0 [libnvdimm]
  __nvdimm_create+0x206/0x290 [libnvdimm]
  cxl_nvdimm_probe+0x119/0x1d0 [cxl_pmem]
  cxl_bus_probe+0x1a/0x60 [cxl_core]
  really_probe+0xde/0x380
Quoted 7 lines

1. Set probe_type of cxl_nvb to PROBE_FORCE_SYNCHRONOUS to ensure the driver is probed synchronously when add_device() is called.
2. Add a check after add_device() for cxl_nvb to ensure the driver is bound, which also means the nvdimm_bus object is registered.
3. Add a check in __nd_device_register() to avoid NULL pointer dereference when nvdimm_bus is not registered.

This is not done in this patch and not needed.

Quoted 6 lines

Hold the uport_dev
lock of the cxl rootport to sychronize between cxl_acpi_probe() and cxl_devm_cxl_add_nvdimm(). Also hold the cxl_nvb->dev lock to ensure a consistent state of if the driver is bound (or not).
4. Add a check for nvdimm_bus in cxl_nvdimm_probe() to avoid accessing without nvdimm_bus for unbound nvdimm object reprobes.

See below, I would have expected that deleting stale nvdimm devices would also be part of this patch.

Quoted 41 lines

This fix also requies the commit that moves devm_cxl_add_nvdimm_bridge() from core/pmem.c to pmem.c.

Fixes 21083f51521f ("cxl/pmem: Register 'pmem' / cxl_nvdimm devices")
---
 drivers/cxl/core/pmem.c | 13 +++++++++++++
 drivers/cxl/pmem.c      | 13 +++++++++++++
 2 files changed, 26 insertions(+)
drivers/cxl/core/pmem.c +13 -0
index 50557d71a45f..f7f784047450 100644
@@ -175,6 +175,19 @@ int devm_cxl_add_nvdimm(struct device *host, struct cxl_port *port,
 	if (!cxl_nvb)
 		return -ENODEV;
 
+	/*
+	 * Take the uport_dev lock to guard against race of nvdimm_bus object.
+	 * cxl_acpi_probe() registers the nvdimm_bus and is done under the
+	 * root port uport_dev lock.
+	 *
+	 * Take the cxl_nvb device lock to ensure that cxl_nvb driver is in a
+	 * consistent state. And the driver registers nvdimm_bus.
+	 */
+	guard(device)(cxl_nvb->port->uport_dev);
+	guard(device)(&cxl_nvb->dev);
+	if (!cxl_nvb->nvdimm_bus)
+		return -ENODEV;
+
 	cxl_nvd = cxl_nvdimm_alloc(cxl_nvb, cxlmd);
 	if (IS_ERR(cxl_nvd)) {
 		rc = PTR_ERR(cxl_nvd);
drivers/cxl/pmem.c +0 -0
index 6f010b3cce44..0b200be1fd36 100644
@@ -85,6 +85,15 @@ struct cxl_nvdimm_bridge *devm_cxl_add_nvdimm_bridge(struct device *host,
 	if (rc)

drivers/cxl/pmem.c struct cxl_nvdimm_bridge *devm_cxl_add_nvdimm_bridge(struct device *host, · line 95

Quoted 10 lines
drivers/cxl/pmem.c +8 -0
 		goto err;
 
+	/* Ensure that cxl_nvb driver has been bound since it is synchronous. */
+	device_lock(dev);
+	if (!dev->driver) {
+		device_unlock(dev);
+		unregister_nvb(cxl_nvb);
+		return ERR_PTR(-ENODEV);
+	}
+	device_unlock(dev);

This can be a cxl_nvdimm_bridge_failed_attach() helper what uses guard().

drivers/cxl/pmem.c static int cxl_nvdimm_probe(struct device *dev) · line 226

Quoted 10 lines
drivers/cxl/pmem.c +3 -0
+
 	rc = devm_add_action_or_reset(host, unregister_nvb, cxl_nvb);
 	if (rc)
 		return ERR_PTR(rc);
@@ -213,6 +222,9 @@ static int cxl_nvdimm_probe(struct device *dev)
 	struct nvdimm *nvdimm;
 	int rc;
 
+	if (!cxl_nvb->nvdimm_bus)
+		return -ENODEV;

If I understand the failure scenario correctly, this is racy. What keeps nvdimm_bus valid for the duration of cxl_nvdimm probe?

This is why I think the cxl_nvdimm_bridge disable path needs to evacuate all prior cxl_nvdimm objects that saw nvdimm_bus live at their registration time.

author maintainer
#5 lore raw

Re: [PATCH v2 2/3] cxl: Fix race of nvdimm_bus object when creating nvdimm objects

Quoted 108 lines · On 2/11/26 4:55 PM, [email protected] wrote:
Quoted 9 lines · Dave Jiang

Found issue during running of cxl-translate.sh unit test. Adding a 3s sleep right before the test seems to make the issue reproduce fairly consistently. The cxl_translate module has dependency on cxl_acpi and causes orphaned nvdimm objects to reprobe after cxl_acpi is removed. The nvdimm_bus object is registered by the cxl_nvb object when cxl_acpi_probe() is called. With the nvdimm_bus object missing, __nd_device_register() will trigger NULL pointer dereference when accessing the dev->parent that points to &nvdimm_bus->dev.

Please trim backtraces of useless info, something like:

 BUG: kernel NULL pointer dereference, address: 000000000000006c
 Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS edk2-20250812-19.fc42 08/12/2025
 Workqueue: cxl_port cxl_bus_rescan_queue [cxl_core]
 RIP: 0010:kobject_get+0xc/0x90
 Call Trace:
  <TASK>
  ? pm_runtime_init+0xb9/0xe0
  __nd_device_register.part.0+0x4d/0xc0 [libnvdimm]
  __nvdimm_create+0x206/0x290 [libnvdimm]
  cxl_nvdimm_probe+0x119/0x1d0 [cxl_pmem]
  cxl_bus_probe+0x1a/0x60 [cxl_core]
  really_probe+0xde/0x380
Quoted 7 lines

1. Set probe_type of cxl_nvb to PROBE_FORCE_SYNCHRONOUS to ensure the driver is probed synchronously when add_device() is called.
2. Add a check after add_device() for cxl_nvb to ensure the driver is bound, which also means the nvdimm_bus object is registered.
3. Add a check in __nd_device_register() to avoid NULL pointer dereference when nvdimm_bus is not registered.

This is not done in this patch and not needed.

Quoted 6 lines

Hold the uport_dev
lock of the cxl rootport to sychronize between cxl_acpi_probe() and cxl_devm_cxl_add_nvdimm(). Also hold the cxl_nvb->dev lock to ensure a consistent state of if the driver is bound (or not).
4. Add a check for nvdimm_bus in cxl_nvdimm_probe() to avoid accessing without nvdimm_bus for unbound nvdimm object reprobes.

See below, I would have expected that deleting stale nvdimm devices would also be part of this patch.

Quoted 41 lines

This fix also requies the commit that moves devm_cxl_add_nvdimm_bridge() from core/pmem.c to pmem.c.

Fixes 21083f51521f ("cxl/pmem: Register 'pmem' / cxl_nvdimm devices")
---
 drivers/cxl/core/pmem.c | 13 +++++++++++++
 drivers/cxl/pmem.c      | 13 +++++++++++++
 2 files changed, 26 insertions(+)
drivers/cxl/core/pmem.c +13 -0
index 50557d71a45f..f7f784047450 100644
@@ -175,6 +175,19 @@ int devm_cxl_add_nvdimm(struct device *host, struct cxl_port *port,
 	if (!cxl_nvb)
 		return -ENODEV;
 
+	/*
+	 * Take the uport_dev lock to guard against race of nvdimm_bus object.
+	 * cxl_acpi_probe() registers the nvdimm_bus and is done under the
+	 * root port uport_dev lock.
+	 *
+	 * Take the cxl_nvb device lock to ensure that cxl_nvb driver is in a
+	 * consistent state. And the driver registers nvdimm_bus.
+	 */
+	guard(device)(cxl_nvb->port->uport_dev);
+	guard(device)(&cxl_nvb->dev);
+	if (!cxl_nvb->nvdimm_bus)
+		return -ENODEV;
+
 	cxl_nvd = cxl_nvdimm_alloc(cxl_nvb, cxlmd);
 	if (IS_ERR(cxl_nvd)) {
 		rc = PTR_ERR(cxl_nvd);
drivers/cxl/pmem.c +0 -0
index 6f010b3cce44..0b200be1fd36 100644
@@ -85,6 +85,15 @@ struct cxl_nvdimm_bridge *devm_cxl_add_nvdimm_bridge(struct device *host,
 	if (rc)

drivers/cxl/pmem.c struct cxl_nvdimm_bridge *devm_cxl_add_nvdimm_bridge(struct device *host, · line 95

Quoted 10 lines
drivers/cxl/pmem.c +8 -0
 		goto err;
 
+	/* Ensure that cxl_nvb driver has been bound since it is synchronous. */
+	device_lock(dev);
+	if (!dev->driver) {
+		device_unlock(dev);
+		unregister_nvb(cxl_nvb);
+		return ERR_PTR(-ENODEV);
+	}
+	device_unlock(dev);

This can be a cxl_nvdimm_bridge_failed_attach() helper what uses guard().

Quoted 8 lines
drivers/cxl/pmem.c +1 -0
+
 	rc = devm_add_action_or_reset(host, unregister_nvb, cxl_nvb);
 	if (rc)
 		return ERR_PTR(rc);
@@ -213,6 +222,9 @@ static int cxl_nvdimm_probe(struct device *dev)
 	struct nvdimm *nvdimm;
 	int rc;
 
Quoted 5 lines

drivers/cxl/pmem.c static int cxl_nvdimm_probe(struct device *dev) · line 226

Quoted 2 lines
drivers/cxl/pmem.c +2 -0
+	if (!cxl_nvb->nvdimm_bus)
+		return -ENODEV;

If I understand the failure scenario correctly, this is racy. What keeps nvdimm_bus valid for the duration of cxl_nvdimm probe?

With patch 3 merged into this one, we can drop this check as there wouldn't be any devices to probe.

Quoted 4 lines

This is why I think the cxl_nvdimm_bridge disable path needs to evacuate all prior cxl_nvdimm objects that saw nvdimm_bus live at their registration time.

Right I split that out because I was worried about patch size for review. I'll squash the two.

author maintainer
#6 lore raw

PATCH v2 3/3 cxl: Clean up cxl_nvdimm devices when removing nvdimm_bus

In reply to Dave Jiang

In the current implementation, when the nvdimm_bus is removed, the cxl_nvdimm devices only have the driver unbound which leaves orphaned cxl_nvdimm devices around. Add removal of cxl_nvdimm devices when the nvdimm_bus object is removed on cxl_nvb driver removal.

---
 drivers/cxl/core/pmem.c | 6 +++++-
 drivers/cxl/cxl.h       | 1 +
 drivers/cxl/pmem.c      | 5 +++++
 3 files changed, 11 insertions(+), 1 deletion(-)
drivers/cxl/core/pmem.c +5 -1
index f7f784047450..4edc87ceb993 100644
@@ -142,18 +142,22 @@ static struct cxl_nvdimm *cxl_nvdimm_alloc(struct cxl_nvdimm_bridge *cxl_nvb,
 	return cxl_nvd;
 }
 
-static void cxlmd_release_nvdimm(void *_cxlmd)
+void cxlmd_release_nvdimm(void *_cxlmd)
 {
 	struct cxl_memdev *cxlmd = _cxlmd;
 	struct cxl_nvdimm *cxl_nvd = cxlmd->cxl_nvd;
 	struct cxl_nvdimm_bridge *cxl_nvb = cxlmd->cxl_nvb;
 
+	dev_info(&cxlmd->dev, "XXX release nvdimm %s\n",
+		 cxl_nvd ? dev_name(&cxl_nvd->dev) : "none");
+
 	cxl_nvd->cxlmd = NULL;
 	cxlmd->cxl_nvd = NULL;
 	cxlmd->cxl_nvb = NULL;
 	device_unregister(&cxl_nvd->dev);
 	put_device(&cxl_nvb->dev);
 }
+EXPORT_SYMBOL_NS_GPL(cxlmd_release_nvdimm, "CXL");
 
 /**
  * devm_cxl_add_nvdimm() - add a bridge between a cxl_memdev and an nvdimm
drivers/cxl/cxl.h +1 -0
index 3b79d5c4765a..3f0d927e5772 100644
@@ -926,6 +926,7 @@ bool is_cxl_nvdimm(struct device *dev);
 int devm_cxl_add_nvdimm(struct device *host, struct cxl_port *port,
 			struct cxl_memdev *cxlmd);
 struct cxl_nvdimm_bridge *cxl_find_nvdimm_bridge(struct cxl_port *port);
+void cxlmd_release_nvdimm(void *cxlmd);
 
 #ifdef CONFIG_CXL_REGION
 bool is_cxl_pmem_region(struct device *dev);
drivers/cxl/pmem.c +5 -0
index 0b200be1fd36..edce5269a682 100644
@@ -397,6 +397,7 @@ static int cxl_pmem_ctl(struct nvdimm_bus_descriptor *nd_desc,
 static int detach_nvdimm(struct device *dev, void *data)
 {
 	struct cxl_nvdimm *cxl_nvd;
+	struct cxl_memdev *cxlmd;
 	bool release = false;
 
 	if (!is_cxl_nvdimm(dev))
@@ -411,6 +412,10 @@ static int detach_nvdimm(struct device *dev, void *data)
 	}
 	if (release)
 		device_release_driver(dev);
+
+	cxlmd = cxl_nvd->cxlmd;
+	devm_release_action(&cxlmd->dev, cxlmd_release_nvdimm, cxlmd);
+
 	return 0;
 }
 
Signature · 1 line
2.52.0
#7 lore raw

Re: [PATCH v2 3/3] cxl: Clean up cxl_nvdimm devices when removing nvdimm_bus

Quoted 20 lines · Dave Jiang

In the current implementation, when the nvdimm_bus is removed, the cxl_nvdimm devices only have the driver unbound which leaves orphaned cxl_nvdimm devices around. Add removal of cxl_nvdimm devices when the nvdimm_bus object is removed on cxl_nvb driver removal.

---
 drivers/cxl/core/pmem.c | 6 +++++-
 drivers/cxl/cxl.h       | 1 +
 drivers/cxl/pmem.c      | 5 +++++
 3 files changed, 11 insertions(+), 1 deletion(-)
drivers/cxl/core/pmem.c +0 -0
index f7f784047450..4edc87ceb993 100644
@@ -142,18 +142,22 @@ static struct cxl_nvdimm *cxl_nvdimm_alloc(struct cxl_nvdimm_bridge *cxl_nvb,
 	return cxl_nvd;
 }
 

drivers/cxl/core/pmem.c static struct cxl_nvdimm *cxl_nvdimm_alloc(struct cxl_nvdimm_bridge *cxl_nvb, · line 153

Quoted 10 lines
drivers/cxl/core/pmem.c +4 -1
-static void cxlmd_release_nvdimm(void *_cxlmd)
+void cxlmd_release_nvdimm(void *_cxlmd)
 {
 	struct cxl_memdev *cxlmd = _cxlmd;
 	struct cxl_nvdimm *cxl_nvd = cxlmd->cxl_nvd;
 	struct cxl_nvdimm_bridge *cxl_nvb = cxlmd->cxl_nvb;
 
+	dev_info(&cxlmd->dev, "XXX release nvdimm %s\n",
+		 cxl_nvd ? dev_name(&cxl_nvd->dev) : "none");
+

Stale debug?

Quoted 35 lines
drivers/cxl/core/pmem.c +1 -0
 	cxl_nvd->cxlmd = NULL;
 	cxlmd->cxl_nvd = NULL;
 	cxlmd->cxl_nvb = NULL;
 	device_unregister(&cxl_nvd->dev);
 	put_device(&cxl_nvb->dev);
 }
+EXPORT_SYMBOL_NS_GPL(cxlmd_release_nvdimm, "CXL");
 
 /**
  * devm_cxl_add_nvdimm() - add a bridge between a cxl_memdev and an nvdimm
drivers/cxl/cxl.h +1 -0
index 3b79d5c4765a..3f0d927e5772 100644
@@ -926,6 +926,7 @@ bool is_cxl_nvdimm(struct device *dev);
 int devm_cxl_add_nvdimm(struct device *host, struct cxl_port *port,
 			struct cxl_memdev *cxlmd);
 struct cxl_nvdimm_bridge *cxl_find_nvdimm_bridge(struct cxl_port *port);
+void cxlmd_release_nvdimm(void *cxlmd);
 
 #ifdef CONFIG_CXL_REGION
 bool is_cxl_pmem_region(struct device *dev);
drivers/cxl/pmem.c +1 -0
index 0b200be1fd36..edce5269a682 100644
@@ -397,6 +397,7 @@ static int cxl_pmem_ctl(struct nvdimm_bus_descriptor *nd_desc,
 static int detach_nvdimm(struct device *dev, void *data)
 {
 	struct cxl_nvdimm *cxl_nvd;
+	struct cxl_memdev *cxlmd;
 	bool release = false;
 
 	if (!is_cxl_nvdimm(dev))

drivers/cxl/pmem.c static int detach_nvdimm(struct device *dev, void *data) · line 417

Quoted 6 lines
drivers/cxl/pmem.c +3 -0
@@ -411,6 +412,10 @@ static int detach_nvdimm(struct device *dev, void *data)
 	}
 	if (release)
 		device_release_driver(dev);
+
+	cxlmd = cxl_nvd->cxlmd;
+	devm_release_action(&cxlmd->dev, cxlmd_release_nvdimm, cxlmd);

You need to hold the cxlmd device lock and check against dev.driver before invoking its release actions. I worry that lockdep will not like that, but worth a shot.

It might be sufficient to just set an INVALIDATED flag in the cxl_nvdimm object that cxl_nvdimm_probe() can check to give up early on trying to re-enable the device.

This would be similar to this check in cxl_mem_probe():

        if (work_pending(&cxlmd->detach_work))
                return -EBUSY;

...which prevents reattach while detach cleanup is pending.

#8 lore raw

Re: [PATCH v2 3/3] cxl: Clean up cxl_nvdimm devices when removing nvdimm_bus

In reply to Dave Jiang
kernel test robot noticed the following build warnings:

Hi Dave,

kernel test robot noticed the following build warnings:

[auto build test WARNING on 63fbf275fa9f18f7020fb8acf54fa107e51d0f23]

url:    https://github.com/intel-lab-lkp/linux/commits/Dave-Jiang/cxl-Move-devm_cxl_add_nvdimm_bridge-to-cxl_pmem-ko/20260212-073008
base:   63fbf275fa9f18f7020fb8acf54fa107e51d0f23
patch link:    https://lore.kernel.org/r/20260211232630.480509-4-dave.jiang%40intel.com
patch subject: [PATCH v2 3/3] cxl: Clean up cxl_nvdimm devices when removing nvdimm_bus
config: s390-allmodconfig (https://download.01.org/0day-ci/archive/20260212/[email protected]/config)
compiler: clang version 18.1.8 (https://github.com/llvm/llvm-project 3b5b5c1ec4a3095ab096dd780e84d7ab81f3d7ff)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260212/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All warnings (new ones prefixed by >>):

>> drivers/cxl/pmem.c:407:7: warning: variable 'cxl_nvd' is used uninitialized whenever 'if' condition is false [-Wsometimes-uninitialized]
     407 |                 if (dev->driver) {
         |                     ^~~~~~~~~~~
   drivers/cxl/pmem.c:416:10: note: uninitialized use occurs here
     416 |         cxlmd = cxl_nvd->cxlmd;
         |                 ^~~~~~~
   drivers/cxl/pmem.c:407:3: note: remove the 'if' if its condition is always true
     407 |                 if (dev->driver) {
         |                 ^~~~~~~~~~~~~~~~
   drivers/cxl/pmem.c:399:28: note: initialize the variable 'cxl_nvd' to silence this warning
     399 |         struct cxl_nvdimm *cxl_nvd;
         |                                   ^
         |                                    = NULL

1 warning generated.

vim +407 drivers/cxl/pmem.c

8fdcb1704f61a8 Dan Williams 2021-06-15  396  
19398821b25a9c Dan Williams 2023-01-20  397  static int detach_nvdimm(struct device *dev, void *data)
19398821b25a9c Dan Williams 2023-01-20  398  {
19398821b25a9c Dan Williams 2023-01-20  399  	struct cxl_nvdimm *cxl_nvd;
5f761998c93e30 Dave Jiang   2026-02-11  400  	struct cxl_memdev *cxlmd;
19398821b25a9c Dan Williams 2023-01-20  401  	bool release = false;
19398821b25a9c Dan Williams 2023-01-20  402  
19398821b25a9c Dan Williams 2023-01-20  403  	if (!is_cxl_nvdimm(dev))
19398821b25a9c Dan Williams 2023-01-20  404  		return 0;
19398821b25a9c Dan Williams 2023-01-20  405  
7f569e917b7866 Li Ming      2024-08-30  406  	scoped_guard(device, dev) {
7f569e917b7866 Li Ming      2024-08-30 @407  		if (dev->driver) {
19398821b25a9c Dan Williams 2023-01-20  408  			cxl_nvd = to_cxl_nvdimm(dev);
19398821b25a9c Dan Williams 2023-01-20  409  			if (cxl_nvd->cxlmd && cxl_nvd->cxlmd->cxl_nvb == data)
19398821b25a9c Dan Williams 2023-01-20  410  				release = true;
7f569e917b7866 Li Ming      2024-08-30  411  		}
7f569e917b7866 Li Ming      2024-08-30  412  	}
19398821b25a9c Dan Williams 2023-01-20  413  	if (release)
19398821b25a9c Dan Williams 2023-01-20  414  		device_release_driver(dev);
5f761998c93e30 Dave Jiang   2026-02-11  415  
5f761998c93e30 Dave Jiang   2026-02-11  416  	cxlmd = cxl_nvd->cxlmd;
5f761998c93e30 Dave Jiang   2026-02-11  417  	devm_release_action(&cxlmd->dev, cxlmd_release_nvdimm, cxlmd);
5f761998c93e30 Dave Jiang   2026-02-11  418  
19398821b25a9c Dan Williams 2023-01-20  419  	return 0;
19398821b25a9c Dan Williams 2023-01-20  420  }
19398821b25a9c Dan Williams 2023-01-20  421  
Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki