Skip to contentSkip to conversation

cxl/region: Unregister the pmem region bridge on setup failure

PATCH In review Guixin Liu 4 participants 7 messages Aug 12 – Aug 31 CXL

lore
Share & tools
lore

Guixin Liu

author
#0 lore raw

PATCH v2 1/1 cxl/region: Unregister the pmem region bridge on setup failure

devm_cxl_add_pmem_region() publishes the cxl_pmem_region with device_add() and only afterwards, under the nvdimm bridge's device lock, arranges for its removal - and only if the bridge has a driver bound. If it does not, the function sets -ENXIO and leaves through err_bridge, which drops the reference this function took on the bridge and returns. The device that was just added has no owner at that point: no device_del(), no put_device(), and no devm action to do either later. The sibling failure, a devm_add_action_or_reset() that cannot allocate, is already covered, because devm_add_action_or_reset() runs cxlr_pmem_unregister() itself on that path.

An unbound bridge is a normal state, not an error state. The bridge is unbound whenever cxl_pmem is unloaded or its device is detached through sysfs, and a region can be probed in that window.

The added device then stays in sysfs, along with the reference it holds on the region, until the module is unloaded. cxlr->cxlr_pmem still points at it, and worse, the name is still taken: a later probe of the same region allocates a second cxl_pmem_region and fails in device_add() on the duplicate "pmem_region%d", so once this has happened the region can no longer be brought up at all.

Call cxlr_pmem_unregister() on that branch. It is invoked from inside the scoped_guard() that holds the bridge's device lock, which is what its device_lock_assert() requires, and it performs the same teardown the devm action would have performed, including clearing cxlr->cxlr_pmem, so err_bridge is left with only the bridge reference to drop.

Fixes f17b558d6663 ("cxl/pmem: Refactor nvdimm device registration, delete the workqueue")

---
This was patch 8/8 of the "cxl: Assorted fixes" series [1]. Per review feedback that series is not being reworked as a whole; the fixes are resent individually instead. Patches 1, 2 and 7 of the series are dropped, as those issues are already fixed in cxl/next.

v1->v2:
- rebase onto cxl/next
- rewrite the commit message to describe the behaviour rather than narrate the code change (Alison Schofield)

[1] https://lore.kernel.org/linux-cxl/[email protected]/

 drivers/cxl/core/region_pmem.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
drivers/cxl/core/region_pmem.c +4 -2
index 23d97e3d78b6..7ab1373a95e0 100644
@@ -168,12 +168,14 @@ int devm_cxl_add_pmem_region(struct cxl_region *cxlr)
 		dev_name(dev));
 
 	scoped_guard(device, &cxl_nvb->dev) {
-		if (cxl_nvb->dev.driver)
+		if (cxl_nvb->dev.driver) {
 			rc = devm_add_action_or_reset(&cxl_nvb->dev,
 						      cxlr_pmem_unregister,
 						      cxlr_pmem);
-		else
+		} else {
 			rc = -ENXIO;
+			cxlr_pmem_unregister(cxlr_pmem);
+		}
 	}
 
 	if (rc)

base-commit: 7098e9cd98a05c0c5de2fae0c2465f9d966fdd07

Signature · 1 line
2.43.7
Reviewed-by on v2 1/1
#1 lore raw
Quoted 68 lines · Guixin Liu

devm_cxl_add_pmem_region() publishes the cxl_pmem_region with device_add() and only afterwards, under the nvdimm bridge's device lock, arranges for its removal - and only if the bridge has a driver bound. If it does not, the function sets -ENXIO and leaves through err_bridge, which drops the reference this function took on the bridge and returns. The device that was just added has no owner at that point: no device_del(), no put_device(), and no devm action to do either later. The sibling failure, a devm_add_action_or_reset() that cannot allocate, is already covered, because devm_add_action_or_reset() runs cxlr_pmem_unregister() itself on that path.

An unbound bridge is a normal state, not an error state. The bridge is unbound whenever cxl_pmem is unloaded or its device is detached through sysfs, and a region can be probed in that window.

The added device then stays in sysfs, along with the reference it holds on the region, until the module is unloaded. cxlr->cxlr_pmem still points at it, and worse, the name is still taken: a later probe of the same region allocates a second cxl_pmem_region and fails in device_add() on the duplicate "pmem_region%d", so once this has happened the region can no longer be brought up at all.

Call cxlr_pmem_unregister() on that branch. It is invoked from inside the scoped_guard() that holds the bridge's device lock, which is what its device_lock_assert() requires, and it performs the same teardown the devm action would have performed, including clearing cxlr->cxlr_pmem, so err_bridge is left with only the bridge reference to drop.

Fixes f17b558d6663 ("cxl/pmem: Refactor nvdimm device registration, delete the workqueue")

---
This was patch 8/8 of the "cxl: Assorted fixes" series [1]. Per review feedback that series is not being reworked as a whole; the fixes are resent individually instead. Patches 1, 2 and 7 of the series are dropped, as those issues are already fixed in cxl/next.

v1->v2:
- rebase onto cxl/next
- rewrite the commit message to describe the behaviour rather than narrate the code change (Alison Schofield)

[1] https://lore.kernel.org/linux-cxl/[email protected]/

 drivers/cxl/core/region_pmem.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
drivers/cxl/core/region_pmem.c +4 -2
index 23d97e3d78b6..7ab1373a95e0 100644
@@ -168,12 +168,14 @@ int devm_cxl_add_pmem_region(struct cxl_region *cxlr)
 		dev_name(dev));
 
 	scoped_guard(device, &cxl_nvb->dev) {
-		if (cxl_nvb->dev.driver)
+		if (cxl_nvb->dev.driver) {
 			rc = devm_add_action_or_reset(&cxl_nvb->dev,
 						      cxlr_pmem_unregister,
 						      cxlr_pmem);
-		else
+		} else {
 			rc = -ENXIO;
+			cxlr_pmem_unregister(cxlr_pmem);
+		}
 	}
 
 	if (rc)
Quoted 5 lines

base-commit: 7098e9cd98a05c0c5de2fae0c2465f9d966fdd07

Signature · 1 line
2.43.7

Best regards,
Richard Cheng

reviewer Reviewed-by on v2 1/1
#2 lore raw
In reply to Guixin Liu
Quoted 22 lines · Guixin Liu

devm_cxl_add_pmem_region() publishes the cxl_pmem_region with device_add() and only afterwards, under the nvdimm bridge's device lock, arranges for its removal - and only if the bridge has a driver bound. If it does not, the function sets -ENXIO and leaves through err_bridge, which drops the reference this function took on the bridge and returns. The device that was just added has no owner at that point: no device_del(), no put_device(), and no devm action to do either later. The sibling failure, a devm_add_action_or_reset() that cannot allocate, is already covered, because devm_add_action_or_reset() runs cxlr_pmem_unregister() itself on that path.

An unbound bridge is a normal state, not an error state. The bridge is unbound whenever cxl_pmem is unloaded or its device is detached through sysfs, and a region can be probed in that window.

The added device then stays in sysfs, along with the reference it holds on the region, until the module is unloaded. cxlr->cxlr_pmem still points at it, and worse, the name is still taken: a later probe of the same region allocates a second cxl_pmem_region and fails in device_add() on the duplicate "pmem_region%d", so once this has happened the region can no longer be brought up at all.

Quoted 8 lines

Call cxlr_pmem_unregister() on that branch. It is invoked from inside the scoped_guard() that holds the bridge's device lock, which is what its device_lock_assert() requires, and it performs the same teardown the devm action would have performed, including clearing cxlr->cxlr_pmem, so err_bridge is left with only the bridge reference to drop.

Fixes f17b558d6663 ("cxl/pmem: Refactor nvdimm device registration, delete the workqueue")
Quoted 39 lines

---
This was patch 8/8 of the "cxl: Assorted fixes" series [1]. Per review feedback that series is not being reworked as a whole; the fixes are resent individually instead. Patches 1, 2 and 7 of the series are dropped, as those issues are already fixed in cxl/next.

v1->v2:
- rebase onto cxl/next
- rewrite the commit message to describe the behaviour rather than narrate the code change (Alison Schofield)

[1] https://lore.kernel.org/linux-cxl/[email protected]/

  drivers/cxl/core/region_pmem.c | 6 ++++--
  1 file changed, 4 insertions(+), 2 deletions(-)
drivers/cxl/core/region_pmem.c +4 -2
index 23d97e3d78b6..7ab1373a95e0 100644
@@ -168,12 +168,14 @@ int devm_cxl_add_pmem_region(struct cxl_region *cxlr)
  		dev_name(dev));
 
  	scoped_guard(device, &cxl_nvb->dev) {
-		if (cxl_nvb->dev.driver)
+		if (cxl_nvb->dev.driver) {
  			rc = devm_add_action_or_reset(&cxl_nvb->dev,
  						      cxlr_pmem_unregister,
  						      cxlr_pmem);
-		else
+		} else {
  			rc = -ENXIO;
+			cxlr_pmem_unregister(cxlr_pmem);
+		}
  	}
 
  	if (rc)

base-commit: 7098e9cd98a05c0c5de2fae0c2465f9d966fdd07

maintainer
#3 lore raw
In reply to Guixin Liu

On Wed, Aug 12, 2026 at 02:10:43PM +0800, Guixin Liu wrote:

About the subject line:

Do we unregister the bridge or do we unregister the pmem region when the bridge is unboound?

Quoted 15 lines

devm_cxl_add_pmem_region() publishes the cxl_pmem_region with device_add() and only afterwards, under the nvdimm bridge's device lock, arranges for its removal - and only if the bridge has a driver bound. If it does not, the function sets -ENXIO and leaves through err_bridge, which drops the reference this function took on the bridge and returns. The device that was just added has no owner at that point: no device_del(), no put_device(), and no devm action to do either later. The sibling failure, a devm_add_action_or_reset() that cannot allocate, is already covered, because devm_add_action_or_reset() runs cxlr_pmem_unregister() itself on that path.

An unbound bridge is a normal state, not an error state. The bridge is unbound whenever cxl_pmem is unloaded or its device is detached through sysfs, and a region can be probed in that window.

Quoted 6 lines

The added device then stays in sysfs, along with the reference it holds on the region, until the module is unloaded. cxlr->cxlr_pmem still points at it, and worse, the name is still taken: a later probe of the same region allocates a second cxl_pmem_region and fails in device_add() on the duplicate "pmem_region%d", so once this has happened the region can no longer be brought up at all.

Is above an observed behavior? ie can you reproduce this, apply the fix, see it go away? That would be far more valuable than this excessive narration as proof of repair.

More below...

Quoted 15 lines

Call cxlr_pmem_unregister() on that branch. It is invoked from inside the scoped_guard() that holds the bridge's device lock, which is what its device_lock_assert() requires, and it performs the same teardown the devm action would have performed, including clearing cxlr->cxlr_pmem, so err_bridge is left with only the bridge reference to drop.

Fixes f17b558d6663 ("cxl/pmem: Refactor nvdimm device registration, delete the workqueue")

---
This was patch 8/8 of the "cxl: Assorted fixes" series [1]. Per review feedback that series is not being reworked as a whole; the fixes are resent individually instead. Patches 1, 2 and 7 of the series are dropped, as those issues are already fixed in cxl/next.

Quoted 4 lines

v1->v2:
- rebase onto cxl/next
- rewrite the commit message to describe the behaviour rather than narrate the code change (Alison Schofield)

This v2 misses the spirit of my v1 feedback. Spirit, meaning I guess I was not clear enough.

It is more behavior-oriented than v1, but it is still far too much investigation in the permanent commit log. The first paragraph walks the reader through control flow almost statement by statement:
device_add(), lock acquisition, driver check, -ENXIO, err_bridge, reference drop, missing device_del(), missing put_device(), and then the sibling devm_add_action_or_reset() case. That is essentially code narration with more behavioral vocabulary around it.

I was hoping the outcome would simply state the failing condition, the consequence, and the fix, maybe like:

Quoted 8 lines
Quoted 8 lines

cxl/region: Unregister pmem region when bridge is unbound

Probing a pmem region while the nvdimm bridge is unbound leaves the newly registered cxl_pmem_region in sysfs. A later probe attempts to register another device with the same name and fails with -EEXIST, preventing the region from being brought up.

Unregister the cxl_pmem_region when the bridge is unbound.

Please consider that in future patches to CXL.
You can v3 if you wish or not.

Quoted 33 lines

[1] https://lore.kernel.org/linux-cxl/[email protected]/

 drivers/cxl/core/region_pmem.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
drivers/cxl/core/region_pmem.c +4 -2
index 23d97e3d78b6..7ab1373a95e0 100644
@@ -168,12 +168,14 @@ int devm_cxl_add_pmem_region(struct cxl_region *cxlr)
 		dev_name(dev));
 
 	scoped_guard(device, &cxl_nvb->dev) {
-		if (cxl_nvb->dev.driver)
+		if (cxl_nvb->dev.driver) {
 			rc = devm_add_action_or_reset(&cxl_nvb->dev,
 						      cxlr_pmem_unregister,
 						      cxlr_pmem);
-		else
+		} else {
 			rc = -ENXIO;
+			cxlr_pmem_unregister(cxlr_pmem);
+		}
 	}
 
 	if (rc)

base-commit: 7098e9cd98a05c0c5de2fae0c2465f9d966fdd07

Signature · 1 line
2.43.7
author
#4 lore raw
Quoted 7 lines · Alison Schofield

On Wed, Aug 12, 2026 at 02:10:43PM +0800, Guixin Liu wrote:

About the subject line:

Do we unregister the bridge or do we unregister the pmem region when the bridge is unboound?

Unregister the pmem region, sorry for the confusing description.

Quoted 21 lines
Quoted 15 lines

devm_cxl_add_pmem_region() publishes the cxl_pmem_region with device_add() and only afterwards, under the nvdimm bridge's device lock, arranges for its removal - and only if the bridge has a driver bound. If it does not, the function sets -ENXIO and leaves through err_bridge, which drops the reference this function took on the bridge and returns. The device that was just added has no owner at that point: no device_del(), no put_device(), and no devm action to do either later. The sibling failure, a devm_add_action_or_reset() that cannot allocate, is already covered, because devm_add_action_or_reset() runs cxlr_pmem_unregister() itself on that path.

An unbound bridge is a normal state, not an error state. The bridge is unbound whenever cxl_pmem is unloaded or its device is detached through sysfs, and a region can be probed in that window.

Quoted 6 lines

The added device then stays in sysfs, along with the reference it holds on the region, until the module is unloaded. cxlr->cxlr_pmem still points at it, and worse, the name is still taken: a later probe of the same region allocates a second cxl_pmem_region and fails in device_add() on the duplicate "pmem_region%d", so once this has happened the region can no longer be brought up at all.

Quoted 5 lines

Is above an observed behavior? ie can you reproduce this, apply the fix, see it go away? That would be far more valuable than this excessive narration as proof of repair.

More below...

I'm so sorry, I thought that we should add cleanup in
failure branch, but now I realize that this branch is never actually reached.

Please ignore this patch, sorry again.

Best Regards,
Guixin Liu

maintainer
#5 lore raw
Quoted 32 lines · Guixin Liu
Quoted 7 lines · Alison Schofield

On Wed, Aug 12, 2026 at 02:10:43PM +0800, Guixin Liu wrote:

About the subject line:

Do we unregister the bridge or do we unregister the pmem region when the bridge is unboound?

Unregister the pmem region, sorry for the confusing description.

Quoted 21 lines
Quoted 21 lines

devm_cxl_add_pmem_region() publishes the cxl_pmem_region with device_add() and only afterwards, under the nvdimm bridge's device lock, arranges for its removal - and only if the bridge has a driver bound. If it does not, the function sets -ENXIO and leaves through err_bridge, which drops the reference this function took on the bridge and returns. The device that was just added has no owner at that point: no device_del(), no put_device(), and no devm action to do either later. The sibling failure, a devm_add_action_or_reset() that cannot allocate, is already covered, because devm_add_action_or_reset() runs cxlr_pmem_unregister() itself on that path.

An unbound bridge is a normal state, not an error state. The bridge is unbound whenever cxl_pmem is unloaded or its device is detached through sysfs, and a region can be probed in that window.

The added device then stays in sysfs, along with the reference it holds on the region, until the module is unloaded. cxlr->cxlr_pmem still points at it, and worse, the name is still taken: a later probe of the same region allocates a second cxl_pmem_region and fails in device_add() on the duplicate "pmem_region%d", so once this has happened the region can no longer be brought up at all.

Quoted 9 lines
Quoted 5 lines

Is above an observed behavior? ie can you reproduce this, apply the fix, see it go away? That would be far more valuable than this excessive narration as proof of repair.

More below...

I'm so sorry, I thought that we should add cleanup in failure branch, but now I realize that this branch is never actually reached.

I think the patch may still apply as-is, so before dropping it, please take a look at this from the teardown side rather than the "cxl_pmem is not loaded" side.

cxl_acpi removal unbinds and deletes the bridge before the regions it serves. So, a region probe that already found the bridge can reach that check after the bridge is gone, set -ENXIO, and return leaving the just-added device with no owner.

If that is true (I'm not sure it is) then the fix stands and the changelog needs to tell that story.

-- Alison

Quoted 5 lines

Please ignore this patch, sorry again.

Best Regards,
Guixin Liu

author
#6 lore raw
Quoted 41 lines · Alison Schofield
Quoted 31 lines · Guixin Liu
Quoted 7 lines · Alison Schofield

On Wed, Aug 12, 2026 at 02:10:43PM +0800, Guixin Liu wrote:

About the subject line:

Do we unregister the bridge or do we unregister the pmem region when the bridge is unboound?

Unregister the pmem region, sorry for the confusing description.

Quoted 21 lines
Quoted 21 lines

devm_cxl_add_pmem_region() publishes the cxl_pmem_region with device_add() and only afterwards, under the nvdimm bridge's device lock, arranges for its removal - and only if the bridge has a driver bound. If it does not, the function sets -ENXIO and leaves through err_bridge, which drops the reference this function took on the bridge and returns. The device that was just added has no owner at that point: no device_del(), no put_device(), and no devm action to do either later. The sibling failure, a devm_add_action_or_reset() that cannot allocate, is already covered, because devm_add_action_or_reset() runs cxlr_pmem_unregister() itself on that path.

An unbound bridge is a normal state, not an error state. The bridge is unbound whenever cxl_pmem is unloaded or its device is detached through sysfs, and a region can be probed in that window.

The added device then stays in sysfs, along with the reference it holds on the region, until the module is unloaded. cxlr->cxlr_pmem still points at it, and worse, the name is still taken: a later probe of the same region allocates a second cxl_pmem_region and fails in device_add() on the duplicate "pmem_region%d", so once this has happened the region can no longer be brought up at all.

Quoted 9 lines
Quoted 5 lines

Is above an observed behavior? ie can you reproduce this, apply the fix, see it go away? That would be far more valuable than this excessive narration as proof of repair.

More below...

I'm so sorry, I thought that we should add cleanup in failure branch, but now I realize that this branch is never actually reached.

Quoted 6 lines

I think the patch may still apply as-is, so before dropping it, please take a look at this from the teardown side rather than the "cxl_pmem is not loaded" side.

cxl_acpi removal unbinds and deletes the bridge before the regions it serves. So, a region probe that already found the bridge can reach that check after the bridge is gone, set -ENXIO, and return leaving the just-added device with no owner.

Well, you are right, I test this on vm, it reached the failure branch.

I sent the v3 patch for shorter commit body and testing description.

Best Regards,
Guixin Liu

Quoted 13 lines

If that is true (I'm not sure it is) then the fix stands and the changelog needs to tell that story.

-- Alison

Quoted 5 lines

Please ignore this patch, sorry again.

Best Regards,
Guixin Liu