Skip to contentSkip to conversation

cxl/region: Improve Soft Reserved resource handling

PATCH Superseded [email protected] 3 participants 9 messages Aug 4, 2023 – Aug 15, 2023 CXL

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

[email protected]

author maintainer
#0 lore raw

v2 cover cxl/region: Improve Soft Reserved resource handling

From: Alison Schofield <[email protected]>

Changes in v2:
- Rebase to 6.5-rc1

v1: https://lore.kernel.org/linux-cxl/[email protected]/

Make the cxl/region driver handle these 2 observed scenarios:

1) Soft reserved resources were observed as sometimes being the parent and sometimes being the child of a region resource. Patch 1 clears up that inconsistency.

2) Soft reserved resources were also observed as stranded after region teardown, making the address space the region released unavailable for reallocation. Patch 2 implements soft reserved resource removal.

Alison Schofield (2):
cxl/region: Try to add a region resource to a soft reserved parent cxl/region: Remove a soft reserved resource at region teardown

 drivers/cxl/core/region.c | 164 +++++++++++++++++++++++++++++++++++---
 1 file changed, 153 insertions(+), 11 deletions(-)

base-commit: 06c2afb862f9da8dc5efa4b6076a0e48c3fbaaa5

Signature · 1 line
2.37.3
author maintainer
#1 lore raw

PATCH v2 1/2 cxl/region: Try to add a region resource to a soft reserved parent

From: Alison Schofield <[email protected]>

During region autodiscovery, the region driver always inserts the region resource as a child of the root decoder, a CXL WINDOW. It has the effect of making a soft reserved resource, with an exactly matching address range, a child of the region resource.

It looks like this in /proc/iomem:

2080000000-29dbfffffff : CXL Window 0
2080000000-247fffffff : region0

    2080000000-247fffffff : Soft Reserved

Search for soft reserved resources that include the region resource, and add the new region resource as a child of that found resource. If a soft reserved resource is not found, insert to the root decoder as usual.

With this change, it looks like this:

2080000000-29dbfffffff : CXL Window 0
2080000000-247fffffff : Soft Reserved

    2080000000-247fffffff : region0

This odd parenting only occurs when the resources are an exact match. When the region resource only uses part of a soft reserved resource, the parenting appears more logical like this:

2080000000-29dbfffffff : CXL Window 0
2080000000-287fffffff : Soft Reserved

    2080000000-247fffffff : region0

Aside from the more logical appearance, this change is in preparation for further cleanup in region teardown. A follow-on patch intends to find and free soft reserved resources upon region teardown.

---
 drivers/cxl/core/region.c | 32 ++++++++++++++++++++++++++++++--
 1 file changed, 30 insertions(+), 2 deletions(-)
drivers/cxl/core/region.c +30 -2
index e115ba382e04..36c0c0dd5697 100644
@@ -2709,6 +2709,28 @@ static int match_region_by_range(struct device *dev, void *data)
 	return rc;
 }
 
+static int insert_resource_soft_reserved(struct resource *soft_res, void *arg)
+{
+	struct resource *parent, *new, *res = arg;
+	bool found = false;
+
+	parent = soft_res->parent;
+	if (!parent)
+		return 0;
+
+	/* Caller provides a copy of soft_res. Find the actual resource. */
+	for (new = parent->child; new; new = new->sibling) {
+		if (resource_contains(new, soft_res)) {
+			found = true;
+			break;
+		}
+	}
+	if (found)
+		return insert_resource(new, res) == 0;
+
+	return 0;
+}
+
 /* Establish an empty region covering the given HPA range */
 static struct cxl_region *construct_region(struct cxl_root_decoder *cxlrd,
 					   struct cxl_endpoint_decoder *cxled)
@@ -2719,7 +2741,7 @@ static struct cxl_region *construct_region(struct cxl_root_decoder *cxlrd,
 	struct cxl_region_params *p;
 	struct cxl_region *cxlr;
 	struct resource *res;
-	int rc;
+	int rc = 0;
 
 	do {
 		cxlr = __create_region(cxlrd, cxled->mode,
@@ -2755,7 +2777,13 @@ static struct cxl_region *construct_region(struct cxl_root_decoder *cxlrd,
 
 	*res = DEFINE_RES_MEM_NAMED(hpa->start, range_len(hpa),
 				    dev_name(&cxlr->dev));
-	rc = insert_resource(cxlrd->res, res);
+
+	/* Try inserting to a Soft Reserved parent, fallback to root decoder */
+	if (walk_iomem_res_desc(IORES_DESC_SOFT_RESERVED, 0,
+				res->start, res->end, res,
+				insert_resource_soft_reserved) != 1)
+		rc = insert_resource(cxlrd->res, res);
+
 	if (rc) {
 		/*
 		 * Platform-firmware may not have split resources like "System
Signature · 1 line
2.37.3
#2 lore raw

Re: [PATCH v2 1/2] cxl/region: Try to add a region resource to a soft reserved parent

Quoted 30 lines · On Fri, 4 Aug 2023 12:31:39 -0700 [email protected] wrote:

From: Alison Schofield <[email protected]>

During region autodiscovery, the region driver always inserts the region resource as a child of the root decoder, a CXL WINDOW. It has the effect of making a soft reserved resource, with an exactly matching address range, a child of the region resource.

It looks like this in /proc/iomem:

2080000000-29dbfffffff : CXL Window 0
2080000000-247fffffff : region0

    2080000000-247fffffff : Soft Reserved

Search for soft reserved resources that include the region resource, and add the new region resource as a child of that found resource. If a soft reserved resource is not found, insert to the root decoder as usual.

With this change, it looks like this:

2080000000-29dbfffffff : CXL Window 0
2080000000-247fffffff : Soft Reserved

    2080000000-247fffffff : region0

This odd parenting only occurs when the resources are an exact match. When the region resource only uses part of a soft reserved resource, the parenting appears more logical like this:

2080000000-29dbfffffff : CXL Window 0
2080000000-287fffffff : Soft Reserved

Quoted 7 lines
    2080000000-247fffffff : region0

Aside from the more logical appearance, this change is in preparation for further cleanup in region teardown. A follow-on patch intends to find and free soft reserved resources upon region teardown.

Hi Alison

A comment and a question inline.

Thanks,

Jonathan

Quoted 20 lines
---
 drivers/cxl/core/region.c | 32 ++++++++++++++++++++++++++++++--
 1 file changed, 30 insertions(+), 2 deletions(-)
drivers/cxl/core/region.c +8 -0
index e115ba382e04..36c0c0dd5697 100644
@@ -2709,6 +2709,28 @@ static int match_region_by_range(struct device *dev, void *data)
 	return rc;
 }
 
+static int insert_resource_soft_reserved(struct resource *soft_res, void *arg)
+{
+	struct resource *parent, *new, *res = arg;
+	bool found = false;
+
+	parent = soft_res->parent;
+	if (!parent)
+		return 0;

drivers/cxl/core/region.c static int match_region_by_range(struct device *dev, void *data) · line 2729

Quoted 10 lines
drivers/cxl/core/region.c +10 -0
+
+	/* Caller provides a copy of soft_res. Find the actual resource. */
+	for (new = parent->child; new; new = new->sibling) {
+		if (resource_contains(new, soft_res)) {
+			found = true;
+			break;
+		}
+	}
+	if (found)
+		return insert_resource(new, res) == 0;

Why not do this in the the if above then directly return there? Also, should we get an error here, does it make sense to carry on? (I don't think we can get an error, but if that's the case we shouldn't
be checking for one) If it's a corner we aren't sure about, good
to add a comment on what the error case might be and what we are doing as a result.

Quoted 31 lines
drivers/cxl/core/region.c +12 -2
+
+	return 0;
+}
+
 /* Establish an empty region covering the given HPA range */
 static struct cxl_region *construct_region(struct cxl_root_decoder *cxlrd,
 					   struct cxl_endpoint_decoder *cxled)
@@ -2719,7 +2741,7 @@ static struct cxl_region *construct_region(struct cxl_root_decoder *cxlrd,
 	struct cxl_region_params *p;
 	struct cxl_region *cxlr;
 	struct resource *res;
-	int rc;
+	int rc = 0;
 
 	do {
 		cxlr = __create_region(cxlrd, cxled->mode,
@@ -2755,7 +2777,13 @@ static struct cxl_region *construct_region(struct cxl_root_decoder *cxlrd,
 
 	*res = DEFINE_RES_MEM_NAMED(hpa->start, range_len(hpa),
 				    dev_name(&cxlr->dev));
-	rc = insert_resource(cxlrd->res, res);
+
+	/* Try inserting to a Soft Reserved parent, fallback to root decoder */
+	if (walk_iomem_res_desc(IORES_DESC_SOFT_RESERVED, 0,
+				res->start, res->end, res,
+				insert_resource_soft_reserved) != 1)
+		rc = insert_resource(cxlrd->res, res);
+
 	if (rc) {
 		/*
 		 * Platform-firmware may not have split resources like "System
author maintainer
#3 lore raw

Re: [PATCH v2 1/2] cxl/region: Try to add a region resource to a soft reserved parent

Quoted 71 lines · Jonathan Cameron
Quoted 30 lines · On Fri, 4 Aug 2023 12:31:39 -0700 [email protected] wrote:

From: Alison Schofield <[email protected]>

During region autodiscovery, the region driver always inserts the region resource as a child of the root decoder, a CXL WINDOW. It has the effect of making a soft reserved resource, with an exactly matching address range, a child of the region resource.

It looks like this in /proc/iomem:

2080000000-29dbfffffff : CXL Window 0
2080000000-247fffffff : region0

    2080000000-247fffffff : Soft Reserved

Search for soft reserved resources that include the region resource, and add the new region resource as a child of that found resource. If a soft reserved resource is not found, insert to the root decoder as usual.

With this change, it looks like this:

2080000000-29dbfffffff : CXL Window 0
2080000000-247fffffff : Soft Reserved

    2080000000-247fffffff : region0

This odd parenting only occurs when the resources are an exact match. When the region resource only uses part of a soft reserved resource, the parenting appears more logical like this:

2080000000-29dbfffffff : CXL Window 0
2080000000-287fffffff : Soft Reserved

Quoted 7 lines
    2080000000-247fffffff : region0

Aside from the more logical appearance, this change is in preparation for further cleanup in region teardown. A follow-on patch intends to find and free soft reserved resources upon region teardown.

Hi Alison

A comment and a question inline.

Thanks,

Jonathan

Quoted 22 lines
---
 drivers/cxl/core/region.c | 32 ++++++++++++++++++++++++++++++--
 1 file changed, 30 insertions(+), 2 deletions(-)
drivers/cxl/core/region.c +10 -0
index e115ba382e04..36c0c0dd5697 100644
@@ -2709,6 +2709,28 @@ static int match_region_by_range(struct device *dev, void *data)
 	return rc;
 }
 
+static int insert_resource_soft_reserved(struct resource *soft_res, void *arg)
+{
+	struct resource *parent, *new, *res = arg;
+	bool found = false;
+
+	parent = soft_res->parent;
+	if (!parent)
+		return 0;
+
+	/* Caller provides a copy of soft_res. Find the actual resource. */
Quoted 9 lines

drivers/cxl/core/region.c static int match_region_by_range(struct device *dev, void *data) · line 2729

Quoted 8 lines
drivers/cxl/core/region.c +8 -0
+	for (new = parent->child; new; new = new->sibling) {
+		if (resource_contains(new, soft_res)) {
+			found = true;
+			break;
+		}
+	}
+	if (found)
+		return insert_resource(new, res) == 0;

Thanks for the review Jonathan,

Quoted 1 line

Why not do this in the the if above then directly return there?

That would be better. Thanks.

Quoted 5 lines

Also, should we get an error here, does it make sense to carry on? (I don't think we can get an error, but if that's the case we shouldn't
be checking for one) If it's a corner we aren't sure about, good
to add a comment on what the error case might be and what we are doing as a result.'

I see now this is hiding 2 cases in one error return
1) no soft reserved found
2) soft reserved found but failed to insert.

I think it can get an error if firmware set up soft reserved on bad boundaries.

...scroll down...

Quoted 19 lines
Quoted 19 lines
drivers/cxl/core/region.c +5 -1
+
+	return 0;
+}
+
 /* Establish an empty region covering the given HPA range */
 static struct cxl_region *construct_region(struct cxl_root_decoder *cxlrd,
 					   struct cxl_endpoint_decoder *cxled)
@@ -2719,7 +2741,7 @@ static struct cxl_region *construct_region(struct cxl_root_decoder *cxlrd,
 	struct cxl_region_params *p;
 	struct cxl_region *cxlr;
 	struct resource *res;
-	int rc;
+	int rc = 0;
 
 	do {
 		cxlr = __create_region(cxlrd, cxled->mode,
@@ -2755,7 +2777,13 @@ static struct cxl_region *construct_region(struct cxl_root_decoder *cxlrd,
 
Quoted 10 lines

drivers/cxl/core/region.c static struct cxl_region *construct_region(struct cxl_root_decoder *cxlrd, · line 2786

Quoted 10 lines
drivers/cxl/core/region.c +7 -1
 	*res = DEFINE_RES_MEM_NAMED(hpa->start, range_len(hpa),
 				    dev_name(&cxlr->dev));
-	rc = insert_resource(cxlrd->res, res);
+
+	/* Try inserting to a Soft Reserved parent, fallback to root decoder */
+	if (walk_iomem_res_desc(IORES_DESC_SOFT_RESERVED, 0,
+				res->start, res->end, res,
+				insert_resource_soft_reserved) != 1)
+		rc = insert_resource(cxlrd->res, res);
+

I'll rework this error handling to not try to insert to root decoder if insert to soft reserved fails. Only try to insert to root decoder if soft reserved does not exist.

Quoted 4 lines

drivers/cxl/core/region.c static struct cxl_region *construct_region(struct cxl_root_decoder *cxlrd, · line 2789

Quoted 3 lines
drivers/cxl/core/region.c +0 -0
 	if (rc) {
 		/*
 		 * Platform-firmware may not have split resources like "System

I think the insert to soft reserved can fail as describe here:
(Let me show this complete error case above)

	if (rc) {
                /*
                 * Platform-firmware may not have split resources like "System
                 * RAM" on CXL window boundaries see cxl_region_iomem_release()
                 */
                dev_warn(cxlmd->dev.parent,
                         "%s:%s: %s %s cannot insert resource\n",
                         dev_name(&cxlmd->dev), dev_name(&cxled->cxld.dev),
                         __func__, dev_name(&cxlr->dev));
        }
maintainer
#4 lore raw

Re: [PATCH v2 1/2] cxl/region: Try to add a region resource to a soft reserved parent

In reply to [email protected]
Quoted 7 lines · On 8/4/23 12:31, [email protected] wrote:

From: Alison Schofield <[email protected]>

During region autodiscovery, the region driver always inserts the region resource as a child of the root decoder, a CXL WINDOW. It has the effect of making a soft reserved resource, with an exactly matching address range, a child of the region resource.

Quoted 8 lines

It looks like this in /proc/iomem:

2080000000-29dbfffffff : CXL Window 0
2080000000-247fffffff : region0

     2080000000-247fffffff : Soft Reserved

Search for soft reserved resources that include the region resource, and add the new region resource as a child of that found resource.

Don't think the comma is needed here.

Quoted 83 lines

If a soft reserved resource is not found, insert to the root decoder as usual.

With this change, it looks like this:

2080000000-29dbfffffff : CXL Window 0
2080000000-247fffffff : Soft Reserved

     2080000000-247fffffff : region0

This odd parenting only occurs when the resources are an exact match. When the region resource only uses part of a soft reserved resource, the parenting appears more logical like this:

2080000000-29dbfffffff : CXL Window 0
2080000000-287fffffff : Soft Reserved

     2080000000-247fffffff : region0

Aside from the more logical appearance, this change is in preparation for further cleanup in region teardown. A follow-on patch intends to find and free soft reserved resources upon region teardown.

---
  drivers/cxl/core/region.c | 32 ++++++++++++++++++++++++++++++--
  1 file changed, 30 insertions(+), 2 deletions(-)
drivers/cxl/core/region.c +30 -2
index e115ba382e04..36c0c0dd5697 100644
@@ -2709,6 +2709,28 @@ static int match_region_by_range(struct device *dev, void *data)
  	return rc;
  }
 
+static int insert_resource_soft_reserved(struct resource *soft_res, void *arg)
+{
+	struct resource *parent, *new, *res = arg;
+	bool found = false;
+
+	parent = soft_res->parent;
+	if (!parent)
+		return 0;
+
+	/* Caller provides a copy of soft_res. Find the actual resource. */
+	for (new = parent->child; new; new = new->sibling) {
+		if (resource_contains(new, soft_res)) {
+			found = true;
+			break;
+		}
+	}
+	if (found)
+		return insert_resource(new, res) == 0;
+
+	return 0;
+}
+
  /* Establish an empty region covering the given HPA range */
  static struct cxl_region *construct_region(struct cxl_root_decoder *cxlrd,
  					   struct cxl_endpoint_decoder *cxled)
@@ -2719,7 +2741,7 @@ static struct cxl_region *construct_region(struct cxl_root_decoder *cxlrd,
  	struct cxl_region_params *p;
  	struct cxl_region *cxlr;
  	struct resource *res;
-	int rc;
+	int rc = 0;
 
  	do {
  		cxlr = __create_region(cxlrd, cxled->mode,
@@ -2755,7 +2777,13 @@ static struct cxl_region *construct_region(struct cxl_root_decoder *cxlrd,
 
  	*res = DEFINE_RES_MEM_NAMED(hpa->start, range_len(hpa),
  				    dev_name(&cxlr->dev));
-	rc = insert_resource(cxlrd->res, res);
+
+	/* Try inserting to a Soft Reserved parent, fallback to root decoder */
+	if (walk_iomem_res_desc(IORES_DESC_SOFT_RESERVED, 0,
+				res->start, res->end, res,
+				insert_resource_soft_reserved) != 1)
+		rc = insert_resource(cxlrd->res, res);
+
  	if (rc) {
  		/*
  		 * Platform-firmware may not have split resources like "System
author maintainer
#5 lore raw

Re: [PATCH v2 1/2] cxl/region: Try to add a region resource to a soft reserved parent

Quoted 12 lines · Dave Jiang
Quoted 9 lines · On 8/4/23 12:31, [email protected] wrote:

From: Alison Schofield <[email protected]>

During region autodiscovery, the region driver always inserts the region resource as a child of the root decoder, a CXL WINDOW. It has the effect of making a soft reserved resource, with an exactly matching address range, a child of the region resource.

It looks like this in /proc/iomem:

Quoted 8 lines
Quoted 6 lines

2080000000-29dbfffffff : CXL Window 0
2080000000-247fffffff : region0

     2080000000-247fffffff : Soft Reserved

Search for soft reserved resources that include the region resource, and add the new region resource as a child of that found resource.

Don't think the comma is needed here.

Got it. Thanks!

Quoted 81 lines
Quoted 81 lines

If a soft reserved resource is not found, insert to the root decoder as usual.

With this change, it looks like this:

2080000000-29dbfffffff : CXL Window 0
2080000000-247fffffff : Soft Reserved

     2080000000-247fffffff : region0

This odd parenting only occurs when the resources are an exact match. When the region resource only uses part of a soft reserved resource, the parenting appears more logical like this:

2080000000-29dbfffffff : CXL Window 0
2080000000-287fffffff : Soft Reserved

     2080000000-247fffffff : region0

Aside from the more logical appearance, this change is in preparation for further cleanup in region teardown. A follow-on patch intends to find and free soft reserved resources upon region teardown.

---
  drivers/cxl/core/region.c | 32 ++++++++++++++++++++++++++++++--
  1 file changed, 30 insertions(+), 2 deletions(-)
drivers/cxl/core/region.c +30 -2
index e115ba382e04..36c0c0dd5697 100644
@@ -2709,6 +2709,28 @@ static int match_region_by_range(struct device *dev, void *data)
  	return rc;
  }
+static int insert_resource_soft_reserved(struct resource *soft_res, void *arg)
+{
+	struct resource *parent, *new, *res = arg;
+	bool found = false;
+
+	parent = soft_res->parent;
+	if (!parent)
+		return 0;
+
+	/* Caller provides a copy of soft_res. Find the actual resource. */
+	for (new = parent->child; new; new = new->sibling) {
+		if (resource_contains(new, soft_res)) {
+			found = true;
+			break;
+		}
+	}
+	if (found)
+		return insert_resource(new, res) == 0;
+
+	return 0;
+}
+
  /* Establish an empty region covering the given HPA range */
  static struct cxl_region *construct_region(struct cxl_root_decoder *cxlrd,
  					   struct cxl_endpoint_decoder *cxled)
@@ -2719,7 +2741,7 @@ static struct cxl_region *construct_region(struct cxl_root_decoder *cxlrd,
  	struct cxl_region_params *p;
  	struct cxl_region *cxlr;
  	struct resource *res;
-	int rc;
+	int rc = 0;
  	do {
  		cxlr = __create_region(cxlrd, cxled->mode,
@@ -2755,7 +2777,13 @@ static struct cxl_region *construct_region(struct cxl_root_decoder *cxlrd,
  	*res = DEFINE_RES_MEM_NAMED(hpa->start, range_len(hpa),
  				    dev_name(&cxlr->dev));
-	rc = insert_resource(cxlrd->res, res);
+
+	/* Try inserting to a Soft Reserved parent, fallback to root decoder */
+	if (walk_iomem_res_desc(IORES_DESC_SOFT_RESERVED, 0,
+				res->start, res->end, res,
+				insert_resource_soft_reserved) != 1)
+		rc = insert_resource(cxlrd->res, res);
+
  	if (rc) {
  		/*
  		 * Platform-firmware may not have split resources like "System
author maintainer
#6 lore raw

PATCH v2 2/2 cxl/region: Remove a soft reserved resource at region teardown

In reply to [email protected]

From: Alison Schofield <[email protected]>

When CXL regions are created through autodiscovery their resource may be a child of a soft reserved resource. Today, when such a region is destroyed, its soft reserved resource remains in place. That means that the HPA range that the region could release, is actually unavailable for reuse.

Free the soft reserved resource on region teardown by examining the alignment of the resources, and handling accordingly. The two resources, will be exactly aligned, partially aligned, or not aligned at all.

|----------- "Soft Reserved" -----------|
|-------------- "Region #" -------------|

Exactly aligned. Any dangling children move up to a parent on removal. The removal of this soft reserved seems guaranteed, however, the availability of the address range for use depends on complete cleanup of the region child resources also.

|----------- "Soft Reserved" -----------|
|-------- "Region #" -------|
or
|----------- "Soft Reserved" -----------|
            |-------- "Region #" -------|

Either start or end aligns. Unlike removing a resource, which simply moves child resources up the resource tree, adjustments fail if any child resources map the range being truncated. So, this one will fail for dangling child resources of the region.

|---------- "Soft Reserved" ----------|
        |---- "Region #" ----|

No alignment. Freeing the resource of a region in the middle is possible if no child resources map the leading or trailing address space.

---
 drivers/cxl/core/region.c | 132 +++++++++++++++++++++++++++++++++++---
 1 file changed, 123 insertions(+), 9 deletions(-)
drivers/cxl/core/region.c +123 -9
index 36c0c0dd5697..b8a02afa3514 100644
@@ -569,22 +569,136 @@ static int alloc_hpa(struct cxl_region *cxlr, resource_size_t size)
 	return 0;
 }
 
+static int add_soft_reserved(struct resource *parent, resource_size_t start,
+			     resource_size_t len, unsigned long flags)
+{
+	struct resource *res;
+	int rc;
+
+	res = kmalloc(sizeof(*res), GFP_KERNEL);
+	if (!res)
+		return -ENOMEM;
+
+	*res = DEFINE_RES_MEM_NAMED(start, len, "Soft Reserved");
+
+	res->desc = IORES_DESC_SOFT_RESERVED;
+	res->flags = res->flags | flags;
+	rc = insert_resource(parent, res);
+	if (rc)
+		kfree(res);
+
+	return rc;
+}
+
+static void remove_soft_reserved(struct cxl_region *cxlr, struct resource *soft,
+				 resource_size_t start, resource_size_t end)
+{
+	struct cxl_root_decoder *cxlrd = to_cxl_root_decoder(cxlr->dev.parent);
+	resource_size_t new_start, new_end;
+	struct resource *parent;
+	unsigned long flags;
+	int rc;
+
+	/* Prevent new usage while removing or adjusting the resource */
+	mutex_lock(&cxlrd->range_lock);
+
+	/* Aligns at both resource start and end */
+	if (soft->start == start && soft->end == end) {
+		rc = remove_resource(soft);
+		if (rc)
+			dev_dbg(&cxlr->dev,
+				"cannot remove soft reserved resource %pr\n",
+				soft);
+		else
+			kfree(soft);
+
+		goto out;
+	}
+
+	/* Aligns at either resource start or end */
+	if (soft->start == start || soft->end == end) {
+		if (soft->start == start) {
+			new_start = end + 1;
+			new_end = soft->end;
+		} else {
+			new_start = soft->start;
+			new_end = start + 1;
+		}
+		rc =  adjust_resource(soft, new_start, new_end - new_start + 1);
+		if (rc)
+			dev_dbg(&cxlr->dev,
+				"cannot adjust soft reserved resource %pr\n",
+				soft);
+		goto out;
+	}
+
+	/*
+	 * No alignment. Attempt a 3-way split that removes the part of
+	 * the resource the region occupied, and then creates new soft
+	 * reserved resources for the leading and trailing addr space.
+	 * adjust_resource() will stop the attempt if there are any
+	 * child resources.
+	 */
+
+	/* Save the original soft reserved resource params before adjusting */
+	new_start = soft->start;
+	new_end = soft->end;
+	parent = soft->parent;
+	flags = soft->flags;
+
+	rc = adjust_resource(soft, start, end - start);
+	if (rc) {
+		dev_dbg(&cxlr->dev,
+			"cannot adjust soft reserved resource %pr\n", soft);
+		goto out;
+	}
+	rc = remove_resource(soft);
+	if (rc)
+		dev_warn(&cxlr->dev,
+			 "cannot remove soft reserved resource %pr\n", soft);
+
+	rc = add_soft_reserved(parent, new_start, start - new_start, flags);
+	if (rc)
+		dev_warn(&cxlr->dev,
+			 "cannot add new soft reserved resource at %pa\n",
+			 &new_start);
+
+	rc = add_soft_reserved(parent, end + 1, new_end - end, flags);
+	if (rc)
+		dev_warn(&cxlr->dev,
+			 "cannot add new soft reserved resource at %pa + 1\n",
+			 &end);
+out:
+	mutex_unlock(&cxlrd->range_lock);
+}
+
 static void cxl_region_iomem_release(struct cxl_region *cxlr)
 {
 	struct cxl_region_params *p = &cxlr->params;
+	struct resource *parent, *res = p->res;
+	resource_size_t start, end;
 
 	if (device_is_registered(&cxlr->dev))
 		lockdep_assert_held_write(&cxl_region_rwsem);
-	if (p->res) {
-		/*
-		 * Autodiscovered regions may not have been able to insert their
-		 * resource.
-		 */
-		if (p->res->parent)
-			remove_resource(p->res);
-		kfree(p->res);
-		p->res = NULL;
+	if (!res)
+		return;
+	/*
+	 * Autodiscovered regions may not have been able to insert their
+	 * resource. If a Soft Reserved parent resource exists, try to
+	 * remove that also.
+	 */
+	if (p->res->parent) {
+		parent = p->res->parent;
+		start = p->res->start;
+		end = p->res->end;
+		remove_resource(p->res);
+		if (test_bit(CXL_REGION_F_AUTO, &cxlr->flags) &&
+		    parent->desc == IORES_DESC_SOFT_RESERVED)
+			remove_soft_reserved(cxlr, parent, start, end);
 	}
+
+	kfree(p->res);
+	p->res = NULL;
 }
 
 static int free_hpa(struct cxl_region *cxlr)
Signature · 1 line
2.37.3
maintainer
#7 lore raw

Re: [PATCH v2 2/2] cxl/region: Remove a soft reserved resource at region teardown

Quoted 7 lines · On 8/4/23 12:31, [email protected] wrote:

From: Alison Schofield <[email protected]>

When CXL regions are created through autodiscovery their resource may be a child of a soft reserved resource. Today, when such a region is destroyed, its soft reserved resource remains in place. That means that the HPA range that the region could release, is actually unavailable for reuse.

I think it reads better without the comma here.

Quoted 183 lines

Free the soft reserved resource on region teardown by examining the alignment of the resources, and handling accordingly. The two resources, will be exactly aligned, partially aligned, or not aligned at all.

|----------- "Soft Reserved" -----------|
|-------------- "Region #" -------------|

Exactly aligned. Any dangling children move up to a parent on removal. The removal of this soft reserved seems guaranteed, however, the availability of the address range for use depends on complete cleanup of the region child resources also.

|----------- "Soft Reserved" -----------|
|-------- "Region #" -------|
or
|----------- "Soft Reserved" -----------|
             |-------- "Region #" -------|

Either start or end aligns. Unlike removing a resource, which simply moves child resources up the resource tree, adjustments fail if any child resources map the range being truncated. So, this one will fail for dangling child resources of the region.

|---------- "Soft Reserved" ----------|
         |---- "Region #" ----|

No alignment. Freeing the resource of a region in the middle is possible if no child resources map the leading or trailing address space.

---
  drivers/cxl/core/region.c | 132 +++++++++++++++++++++++++++++++++++---
  1 file changed, 123 insertions(+), 9 deletions(-)
drivers/cxl/core/region.c +123 -9
index 36c0c0dd5697..b8a02afa3514 100644
@@ -569,22 +569,136 @@ static int alloc_hpa(struct cxl_region *cxlr, resource_size_t size)
  	return 0;
  }
 
+static int add_soft_reserved(struct resource *parent, resource_size_t start,
+			     resource_size_t len, unsigned long flags)
+{
+	struct resource *res;
+	int rc;
+
+	res = kmalloc(sizeof(*res), GFP_KERNEL);
+	if (!res)
+		return -ENOMEM;
+
+	*res = DEFINE_RES_MEM_NAMED(start, len, "Soft Reserved");
+
+	res->desc = IORES_DESC_SOFT_RESERVED;
+	res->flags = res->flags | flags;
+	rc = insert_resource(parent, res);
+	if (rc)
+		kfree(res);
+
+	return rc;
+}
+
+static void remove_soft_reserved(struct cxl_region *cxlr, struct resource *soft,
+				 resource_size_t start, resource_size_t end)
+{
+	struct cxl_root_decoder *cxlrd = to_cxl_root_decoder(cxlr->dev.parent);
+	resource_size_t new_start, new_end;
+	struct resource *parent;
+	unsigned long flags;
+	int rc;
+
+	/* Prevent new usage while removing or adjusting the resource */
+	mutex_lock(&cxlrd->range_lock);
+
+	/* Aligns at both resource start and end */
+	if (soft->start == start && soft->end == end) {
+		rc = remove_resource(soft);
+		if (rc)
+			dev_dbg(&cxlr->dev,
+				"cannot remove soft reserved resource %pr\n",
+				soft);
+		else
+			kfree(soft);
+
+		goto out;
+	}
+
+	/* Aligns at either resource start or end */
+	if (soft->start == start || soft->end == end) {
+		if (soft->start == start) {
+			new_start = end + 1;
+			new_end = soft->end;
+		} else {
+			new_start = soft->start;
+			new_end = start + 1;
+		}
+		rc =  adjust_resource(soft, new_start, new_end - new_start + 1);
+		if (rc)
+			dev_dbg(&cxlr->dev,
+				"cannot adjust soft reserved resource %pr\n",
+				soft);
+		goto out;
+	}
+
+	/*
+	 * No alignment. Attempt a 3-way split that removes the part of
+	 * the resource the region occupied, and then creates new soft
+	 * reserved resources for the leading and trailing addr space.
+	 * adjust_resource() will stop the attempt if there are any
+	 * child resources.
+	 */
+
+	/* Save the original soft reserved resource params before adjusting */
+	new_start = soft->start;
+	new_end = soft->end;
+	parent = soft->parent;
+	flags = soft->flags;
+
+	rc = adjust_resource(soft, start, end - start);
+	if (rc) {
+		dev_dbg(&cxlr->dev,
+			"cannot adjust soft reserved resource %pr\n", soft);
+		goto out;
+	}
+	rc = remove_resource(soft);
+	if (rc)
+		dev_warn(&cxlr->dev,
+			 "cannot remove soft reserved resource %pr\n", soft);
+
+	rc = add_soft_reserved(parent, new_start, start - new_start, flags);
+	if (rc)
+		dev_warn(&cxlr->dev,
+			 "cannot add new soft reserved resource at %pa\n",
+			 &new_start);
+
+	rc = add_soft_reserved(parent, end + 1, new_end - end, flags);
+	if (rc)
+		dev_warn(&cxlr->dev,
+			 "cannot add new soft reserved resource at %pa + 1\n",
+			 &end);
+out:
+	mutex_unlock(&cxlrd->range_lock);
+}
+
  static void cxl_region_iomem_release(struct cxl_region *cxlr)
  {
  	struct cxl_region_params *p = &cxlr->params;
+	struct resource *parent, *res = p->res;
+	resource_size_t start, end;
 
  	if (device_is_registered(&cxlr->dev))
  		lockdep_assert_held_write(&cxl_region_rwsem);
-	if (p->res) {
-		/*
-		 * Autodiscovered regions may not have been able to insert their
-		 * resource.
-		 */
-		if (p->res->parent)
-			remove_resource(p->res);
-		kfree(p->res);
-		p->res = NULL;
+	if (!res)
+		return;
+	/*
+	 * Autodiscovered regions may not have been able to insert their
+	 * resource. If a Soft Reserved parent resource exists, try to
+	 * remove that also.
+	 */
+	if (p->res->parent) {
+		parent = p->res->parent;
+		start = p->res->start;
+		end = p->res->end;
+		remove_resource(p->res);
+		if (test_bit(CXL_REGION_F_AUTO, &cxlr->flags) &&
+		    parent->desc == IORES_DESC_SOFT_RESERVED)
+			remove_soft_reserved(cxlr, parent, start, end);
  	}
+
+	kfree(p->res);
+	p->res = NULL;
  }
 
  static int free_hpa(struct cxl_region *cxlr)
author maintainer
#8 lore raw

Re: [PATCH v2 2/2] cxl/region: Remove a soft reserved resource at region teardown

Quoted 5 lines · Dave Jiang
Quoted 2 lines · On 8/4/23 12:31, [email protected] wrote:

From: Alison Schofield <[email protected]>

Quoted 7 lines
Quoted 5 lines

When CXL regions are created through autodiscovery their resource may be a child of a soft reserved resource. Today, when such a region is destroyed, its soft reserved resource remains in place. That means that the HPA range that the region could release, is actually unavailable for reuse.

I think it reads better without the comma here.

Agree. Thanks!
Alison

Quoted 181 lines
Quoted 181 lines

Free the soft reserved resource on region teardown by examining the alignment of the resources, and handling accordingly. The two resources, will be exactly aligned, partially aligned, or not aligned at all.

|----------- "Soft Reserved" -----------|
|-------------- "Region #" -------------|

Exactly aligned. Any dangling children move up to a parent on removal. The removal of this soft reserved seems guaranteed, however, the availability of the address range for use depends on complete cleanup of the region child resources also.

|----------- "Soft Reserved" -----------|
|-------- "Region #" -------|
or
|----------- "Soft Reserved" -----------|
             |-------- "Region #" -------|

Either start or end aligns. Unlike removing a resource, which simply moves child resources up the resource tree, adjustments fail if any child resources map the range being truncated. So, this one will fail for dangling child resources of the region.

|---------- "Soft Reserved" ----------|
         |---- "Region #" ----|

No alignment. Freeing the resource of a region in the middle is possible if no child resources map the leading or trailing address space.

---
  drivers/cxl/core/region.c | 132 +++++++++++++++++++++++++++++++++++---
  1 file changed, 123 insertions(+), 9 deletions(-)
drivers/cxl/core/region.c +123 -9
index 36c0c0dd5697..b8a02afa3514 100644
@@ -569,22 +569,136 @@ static int alloc_hpa(struct cxl_region *cxlr, resource_size_t size)
  	return 0;
  }
+static int add_soft_reserved(struct resource *parent, resource_size_t start,
+			     resource_size_t len, unsigned long flags)
+{
+	struct resource *res;
+	int rc;
+
+	res = kmalloc(sizeof(*res), GFP_KERNEL);
+	if (!res)
+		return -ENOMEM;
+
+	*res = DEFINE_RES_MEM_NAMED(start, len, "Soft Reserved");
+
+	res->desc = IORES_DESC_SOFT_RESERVED;
+	res->flags = res->flags | flags;
+	rc = insert_resource(parent, res);
+	if (rc)
+		kfree(res);
+
+	return rc;
+}
+
+static void remove_soft_reserved(struct cxl_region *cxlr, struct resource *soft,
+				 resource_size_t start, resource_size_t end)
+{
+	struct cxl_root_decoder *cxlrd = to_cxl_root_decoder(cxlr->dev.parent);
+	resource_size_t new_start, new_end;
+	struct resource *parent;
+	unsigned long flags;
+	int rc;
+
+	/* Prevent new usage while removing or adjusting the resource */
+	mutex_lock(&cxlrd->range_lock);
+
+	/* Aligns at both resource start and end */
+	if (soft->start == start && soft->end == end) {
+		rc = remove_resource(soft);
+		if (rc)
+			dev_dbg(&cxlr->dev,
+				"cannot remove soft reserved resource %pr\n",
+				soft);
+		else
+			kfree(soft);
+
+		goto out;
+	}
+
+	/* Aligns at either resource start or end */
+	if (soft->start == start || soft->end == end) {
+		if (soft->start == start) {
+			new_start = end + 1;
+			new_end = soft->end;
+		} else {
+			new_start = soft->start;
+			new_end = start + 1;
+		}
+		rc =  adjust_resource(soft, new_start, new_end - new_start + 1);
+		if (rc)
+			dev_dbg(&cxlr->dev,
+				"cannot adjust soft reserved resource %pr\n",
+				soft);
+		goto out;
+	}
+
+	/*
+	 * No alignment. Attempt a 3-way split that removes the part of
+	 * the resource the region occupied, and then creates new soft
+	 * reserved resources for the leading and trailing addr space.
+	 * adjust_resource() will stop the attempt if there are any
+	 * child resources.
+	 */
+
+	/* Save the original soft reserved resource params before adjusting */
+	new_start = soft->start;
+	new_end = soft->end;
+	parent = soft->parent;
+	flags = soft->flags;
+
+	rc = adjust_resource(soft, start, end - start);
+	if (rc) {
+		dev_dbg(&cxlr->dev,
+			"cannot adjust soft reserved resource %pr\n", soft);
+		goto out;
+	}
+	rc = remove_resource(soft);
+	if (rc)
+		dev_warn(&cxlr->dev,
+			 "cannot remove soft reserved resource %pr\n", soft);
+
+	rc = add_soft_reserved(parent, new_start, start - new_start, flags);
+	if (rc)
+		dev_warn(&cxlr->dev,
+			 "cannot add new soft reserved resource at %pa\n",
+			 &new_start);
+
+	rc = add_soft_reserved(parent, end + 1, new_end - end, flags);
+	if (rc)
+		dev_warn(&cxlr->dev,
+			 "cannot add new soft reserved resource at %pa + 1\n",
+			 &end);
+out:
+	mutex_unlock(&cxlrd->range_lock);
+}
+
  static void cxl_region_iomem_release(struct cxl_region *cxlr)
  {
  	struct cxl_region_params *p = &cxlr->params;
+	struct resource *parent, *res = p->res;
+	resource_size_t start, end;
  	if (device_is_registered(&cxlr->dev))
  		lockdep_assert_held_write(&cxl_region_rwsem);
-	if (p->res) {
-		/*
-		 * Autodiscovered regions may not have been able to insert their
-		 * resource.
-		 */
-		if (p->res->parent)
-			remove_resource(p->res);
-		kfree(p->res);
-		p->res = NULL;
+	if (!res)
+		return;
+	/*
+	 * Autodiscovered regions may not have been able to insert their
+	 * resource. If a Soft Reserved parent resource exists, try to
+	 * remove that also.
+	 */
+	if (p->res->parent) {
+		parent = p->res->parent;
+		start = p->res->start;
+		end = p->res->end;
+		remove_resource(p->res);
+		if (test_bit(CXL_REGION_F_AUTO, &cxlr->flags) &&
+		    parent->desc == IORES_DESC_SOFT_RESERVED)
+			remove_soft_reserved(cxlr, parent, start, end);
  	}
+
+	kfree(p->res);
+	p->res = NULL;
  }
  static int free_hpa(struct cxl_region *cxlr)