Skip to contentSkip to conversation

Add managed SOFT RESERVE resource handling

PATCH In review Terry Bowman 5 participants 21 messages Apr 3, 2025 – Apr 14, 2025

Subsystems (8)
lore
Share & tools
lore
#0 lore raw

v3 cover Add managed SOFT RESERVE resource handling

Add the ability to manage SOFT RESERVE iomem resources prior to them being added to the iomem resource tree. This allows drivers, such as CXL, to remove any pieces of the SOFT RESERVE resource that intersect with created CXL regions.

The current approach of leaving the SOFT RESERVE resources as is can cause failures during hotplug of devices, such as CXL, because the resource is not available for reuse after teardown of the device.

The approach is to add SOFT RESERVE resources to a separate tree during boot. This allows any drivers to update the SOFT RESERVE resources before they are merged into the iomem resource tree. In addition a notifier chain is added so that drivers can be notified when these SOFT RESERVE resources are added to the ioeme resource tree.

The CXL driver is modified to use a worker thread that waits for the CXL PCI and CXL mem drivers to be loaded and for their probe routine to complete. Then the driver walks through any created CXL regions to trim any intersections with SOFT RESERVE resources in the iomem tree.

The dax driver uses the new soft reserve notifier chain so it can consume any remaining SOFT RESERVES once they're added to the iomem tree.

V3 updates:
- Remove srmem resource tree from kernel/resource.c, this is no longer needed in the current implementation. All SOFT RESERVE resources now put on the iomem resource tree.
- Remove the no longer needed SOFT_RESERVED_MANAGED kernel config option.
- Add the 'nid' parameter back to hmem_register_resource();
- Remove the no longer used soft reserve notification chain (introduced in v2). The dax driver is now notified of SOFT RESERVED resources by the CXL driver.

v2 updates:
- Add config option SOFT_RESERVE_MANAGED to control use of the separate srmem resource tree at boot.
- Only add SOFT RESERVE resources to the soft reserve tree during boot, they go to the iomem resource tree after boot.
- Remove the resource trimming code in the previous patch to re-use the existing code in kernel/resource.c
- Add functionality for the cxl acpi driver to wait for the cxl PCI and me drivers to load.

Nathan Fontenot (4):
kernel/resource: Provide mem region release for SOFT RESERVES
cxl: Update Soft Reserved resources upon region creation dax/mum: Save the dax mum platform device pointer cxl/dax: Delay consumption of SOFT RESERVE resources

 drivers/cxl/Kconfig        |  4 ---
 drivers/cxl/acpi.c         | 28 +++++++++++++++++++
 drivers/cxl/core/Makefile  |  2 +-
 drivers/cxl/core/region.c  | 34 ++++++++++++++++++++++-
 drivers/cxl/core/suspend.c | 41 ++++++++++++++++++++++++++++
 drivers/cxl/cxl.h          |  3 +++
 drivers/cxl/cxlmem.h       |  9 -------
 drivers/cxl/cxlpci.h       |  1 +
 drivers/cxl/pci.c          |  2 ++
 drivers/dax/hmem/device.c  | 47 ++++++++++++++++----------------
 drivers/dax/hmem/hmem.c    | 10 ++++---
 include/linux/dax.h        | 11 +++++---
 include/linux/ioport.h     |  3 +++
 include/linux/pm.h         |  7 -----
 kernel/resource.c          | 55 +++++++++++++++++++++++++++++++++++---
 15 files changed, 202 insertions(+), 55 deletions(-)

base-commit: aae0594a7053c60b82621136257c8b648c67b512

Signature · 1 line
2.34.1
#1 lore raw

PATCH v3 1/4 kernel/resource: Provide mem region release for SOFT RESERVES

From: Nathan Fontenot <[email protected]>

Add a release_Sam_region_adjustable() interface to allow for removing SOFT RESERVE memory resources. This extracts out the code to remove a mem region into a common __release_mem_region_adjustable() routine, this routine takes additional parameters of an IORES descriptor type to add checks for IORES_DESC_* and a flag to check for IORESOURCE_BUSY to control it's behavior.

The existing release_mem_region_adjustable() is a front end to the common code and a new release_srmem_region_adjustable() is added to release SOFT RESERVE resources.

---
 include/linux/ioport.h |  3 +++
 kernel/resource.c      | 55 +++++++++++++++++++++++++++++++++++++++---
 2 files changed, 54 insertions(+), 4 deletions(-)
include/linux/ioport.h +3 -0
index 5385349f0b8a..718360c9c724 100644
@@ -357,6 +357,9 @@ extern void __release_region(struct resource *, resource_size_t,
 #ifdef CONFIG_MEMORY_HOTREMOVE
 extern void release_mem_region_adjustable(resource_size_t, resource_size_t);
 #endif
+#ifdef CONFIG_CXL_REGION
+extern void release_srmem_region_adjustable(resource_size_t, resource_size_t);
+#endif
 #ifdef CONFIG_MEMORY_HOTPLUG
 extern void merge_system_ram_resource(struct resource *res);
 #endif
kernel/resource.c +51 -4

79 lines not on this page · raw message

Signature · 1 line
2.34.1
#2 lore raw

Re: [PATCH v3 1/4] kernel/resource: Provide mem region release for SOFT RESERVES

Quoted 12 lines · Terry Bowman

From: Nathan Fontenot <[email protected]>

Add a release_Sam_region_adjustable() interface to allow for removing SOFT RESERVE memory resources. This extracts out the code to remove a mem region into a common __release_mem_region_adjustable() routine, this routine takes additional parameters of an IORES descriptor type to add checks for IORES_DESC_* and a flag to check for IORESOURCE_BUSY to control it's behavior.

The existing release_mem_region_adjustable() is a front end to the common code and a new release_srmem_region_adjustable() is added to release SOFT RESERVE resources.

...

kernel/resource.c void release_mem_region_adjustable(resource_size_t start, resource_size_t size) · line 1531

Quoted 3 lines
kernel/resource.c +3 -0
+void release_mem_region_adjustable(resource_size_t start, resource_size_t size)
+{
+	return __release_mem_region_adjustable(start, size,

You have still room on the previous line for the parameters.

kernel/resource.c void release_mem_region_adjustable(resource_size_t start, resource_size_t size) · line 1532

Quoted 1 line
kernel/resource.c +1 -0
+					       true, IORES_DESC_NONE);

Return on void?! Interesting... What do you want to do here?

kernel/resource.c void release_mem_region_adjustable(resource_size_t start, resource_size_t size) · line 1539

Quoted 7 lines
kernel/resource.c +7 -0
+}
+EXPORT_SYMBOL(release_mem_region_adjustable);
+#endif
+
+#ifdef CONFIG_CXL_REGION
+void release_srmem_region_adjustable(resource_size_t start,
+				     resource_size_t size)

This can be put on a single line.

kernel/resource.c void release_mem_region_adjustable(resource_size_t start, resource_size_t size) · line 1542

Quoted 3 lines
kernel/resource.c +3 -0
+{
+	return __release_mem_region_adjustable(start, size,
+					       false, IORES_DESC_SOFT_RESERVED);

Same comments as per above function.

Quoted 3 lines
kernel/resource.c +3 -0
+}
+EXPORT_SYMBOL(release_srmem_region_adjustable);
+#endif
Signature · 2 lines
With Best Regards,
Andy Shevchenko
#3 lore raw

Re: [PATCH v3 1/4] kernel/resource: Provide mem region release for SOFT RESERVES

Hi Andy,

Thanks for reviewing this. I'll make updates for all you pointed out below.

Regards,
Terry

Quoted 40 lines · Andy Shevchenko
Quoted 12 lines · Terry Bowman

From: Nathan Fontenot <[email protected]>

Add a release_Sam_region_adjustable() interface to allow for removing SOFT RESERVE memory resources. This extracts out the code to remove a mem region into a common __release_mem_region_adjustable() routine, this routine takes additional parameters of an IORES descriptor type to add checks for IORES_DESC_* and a flag to check for IORESOURCE_BUSY to control it's behavior.

The existing release_mem_region_adjustable() is a front end to the common code and a new release_srmem_region_adjustable() is added to release SOFT RESERVE resources.

...

kernel/resource.c void release_mem_region_adjustable(resource_size_t start, resource_size_t size) · line 1531

Quoted 3 lines
kernel/resource.c +3 -0
+void release_mem_region_adjustable(resource_size_t start, resource_size_t size)
+{
+	return __release_mem_region_adjustable(start, size,

You have still room on the previous line for the parameters.

kernel/resource.c void release_mem_region_adjustable(resource_size_t start, resource_size_t size) · line 1532

Quoted 1 line
kernel/resource.c +1 -0
+					       true, IORES_DESC_NONE);

Return on void?! Interesting... What do you want to do here?

kernel/resource.c void release_mem_region_adjustable(resource_size_t start, resource_size_t size) · line 1539

Quoted 7 lines
kernel/resource.c +7 -0
+}
+EXPORT_SYMBOL(release_mem_region_adjustable);
+#endif
+
+#ifdef CONFIG_CXL_REGION
+void release_srmem_region_adjustable(resource_size_t start,
+				     resource_size_t size)

This can be put on a single line.

kernel/resource.c void release_mem_region_adjustable(resource_size_t start, resource_size_t size) · line 1542

Quoted 3 lines
kernel/resource.c +3 -0
+{
+	return __release_mem_region_adjustable(start, size,
+					       false, IORES_DESC_SOFT_RESERVED);

Same comments as per above function.

Quoted 3 lines
kernel/resource.c +3 -0
+}
+EXPORT_SYMBOL(release_srmem_region_adjustable);
+#endif
#4 lore raw

Re: [PATCH v3 1/4] kernel/resource: Provide mem region release for SOFT RESERVES

In reply to Terry Bowman
kernel test robot noticed the following build warnings:

Hi Terry,

kernel test robot noticed the following build warnings:

[auto build test WARNING on aae0594a7053c60b82621136257c8b648c67b512]

url:    https://github.com/intel-lab-lkp/linux/commits/Terry-Bowman/kernel-resource-Provide-mem-region-release-for-SOFT-RESERVES/20250404-023601
base:   aae0594a7053c60b82621136257c8b648c67b512
patch link:    https://lore.kernel.org/r/20250403183315.286710-2-terry.bowman%40amd.com
patch subject: [PATCH v3 1/4] kernel/resource: Provide mem region release for SOFT RESERVES
config: i386-buildonly-randconfig-003-20250404 (https://download.01.org/0day-ci/archive/20250404/[email protected]/config)
compiler: gcc-12 (Debian 12.2.0-14) 12.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250404/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All warnings (new ones prefixed by >>):

>> kernel/resource.c:1414: warning: Function parameter or struct member 'busy_check' not described in '__release_mem_region_adjustable'
>> kernel/resource.c:1414: warning: Function parameter or struct member 'res_desc' not described in '__release_mem_region_adjustable'
>> kernel/resource.c:1414: warning: expecting prototype for release_mem_region_adjustable(). Prototype was for __release_mem_region_adjustable() instead

vim +1414 kernel/resource.c

^1da177e4c3f41 Linus Torvalds    2005-04-16  1389  
e4ebc182a59bbb Nathan Fontenot   2025-04-03  1390  #if defined(CONFIG_MEMORY_HOTREMOVE) || defined(CONFIG_CXL_REGION)
825f787bb49676 Toshi Kani        2013-04-29  1391  /**
825f787bb49676 Toshi Kani        2013-04-29  1392   * release_mem_region_adjustable - release a previously reserved memory region
825f787bb49676 Toshi Kani        2013-04-29  1393   * @start: resource start address
825f787bb49676 Toshi Kani        2013-04-29  1394   * @size: resource region size
825f787bb49676 Toshi Kani        2013-04-29  1395   *
825f787bb49676 Toshi Kani        2013-04-29  1396   * This interface is intended for memory hot-delete.  The requested region
825f787bb49676 Toshi Kani        2013-04-29  1397   * is released from a currently busy memory resource.  The requested region
825f787bb49676 Toshi Kani        2013-04-29  1398   * must either match exactly or fit into a single busy resource entry.  In
825f787bb49676 Toshi Kani        2013-04-29  1399   * the latter case, the remaining resource is adjusted accordingly.
825f787bb49676 Toshi Kani        2013-04-29  1400   * Existing children of the busy memory resource must be immutable in the
825f787bb49676 Toshi Kani        2013-04-29  1401   * request.
825f787bb49676 Toshi Kani        2013-04-29  1402   *
825f787bb49676 Toshi Kani        2013-04-29  1403   * Note:
825f787bb49676 Toshi Kani        2013-04-29  1404   * - Additional release conditions, such as overlapping region, can be
825f787bb49676 Toshi Kani        2013-04-29  1405   *   supported after they are confirmed as valid cases.
825f787bb49676 Toshi Kani        2013-04-29  1406   * - When a busy memory resource gets split into two entries, the code
825f787bb49676 Toshi Kani        2013-04-29  1407   *   assumes that all children remain in the lower address entry for
825f787bb49676 Toshi Kani        2013-04-29  1408   *   simplicity.  Enhance this logic when necessary.
825f787bb49676 Toshi Kani        2013-04-29  1409   */
e4ebc182a59bbb Nathan Fontenot   2025-04-03  1410  static void __release_mem_region_adjustable(resource_size_t start,
e4ebc182a59bbb Nathan Fontenot   2025-04-03  1411  					    resource_size_t size,
e4ebc182a59bbb Nathan Fontenot   2025-04-03  1412  					    bool busy_check,
e4ebc182a59bbb Nathan Fontenot   2025-04-03  1413  					    int res_desc)
825f787bb49676 Toshi Kani        2013-04-29 @1414  {
cb8e3c8b4f45e4 David Hildenbrand 2020-10-15  1415  	struct resource *parent = &iomem_resource;
ec62d04e3fdc4b David Hildenbrand 2020-10-15  1416  	struct resource *new_res = NULL;
ec62d04e3fdc4b David Hildenbrand 2020-10-15  1417  	bool alloc_nofail = false;
825f787bb49676 Toshi Kani        2013-04-29  1418  	struct resource **p;
825f787bb49676 Toshi Kani        2013-04-29  1419  	struct resource *res;
825f787bb49676 Toshi Kani        2013-04-29  1420  	resource_size_t end;
825f787bb49676 Toshi Kani        2013-04-29  1421  
825f787bb49676 Toshi Kani        2013-04-29  1422  	end = start + size - 1;
ec62d04e3fdc4b David Hildenbrand 2020-10-15  1423  	if (WARN_ON_ONCE((start < parent->start) || (end > parent->end)))
ec62d04e3fdc4b David Hildenbrand 2020-10-15  1424  		return;
825f787bb49676 Toshi Kani        2013-04-29  1425  
ec62d04e3fdc4b David Hildenbrand 2020-10-15  1426  	/*
ec62d04e3fdc4b David Hildenbrand 2020-10-15  1427  	 * We free up quite a lot of memory on memory hotunplug (esp., memap),
ec62d04e3fdc4b David Hildenbrand 2020-10-15  1428  	 * just before releasing the region. This is highly unlikely to
ec62d04e3fdc4b David Hildenbrand 2020-10-15  1429  	 * fail - let's play save and make it never fail as the caller cannot
ec62d04e3fdc4b David Hildenbrand 2020-10-15  1430  	 * perform any error handling (e.g., trying to re-add memory will fail
ec62d04e3fdc4b David Hildenbrand 2020-10-15  1431  	 * similarly).
ec62d04e3fdc4b David Hildenbrand 2020-10-15  1432  	 */
ec62d04e3fdc4b David Hildenbrand 2020-10-15  1433  retry:
ec62d04e3fdc4b David Hildenbrand 2020-10-15  1434  	new_res = alloc_resource(GFP_KERNEL | (alloc_nofail ? __GFP_NOFAIL : 0));
825f787bb49676 Toshi Kani        2013-04-29  1435  
825f787bb49676 Toshi Kani        2013-04-29  1436  	p = &parent->child;
825f787bb49676 Toshi Kani        2013-04-29  1437  	write_lock(&resource_lock);
825f787bb49676 Toshi Kani        2013-04-29  1438  
825f787bb49676 Toshi Kani        2013-04-29  1439  	while ((res = *p)) {
825f787bb49676 Toshi Kani        2013-04-29  1440  		if (res->start >= end)
825f787bb49676 Toshi Kani        2013-04-29  1441  			break;
825f787bb49676 Toshi Kani        2013-04-29  1442  
825f787bb49676 Toshi Kani        2013-04-29  1443  		/* look for the next resource if it does not fit into */
825f787bb49676 Toshi Kani        2013-04-29  1444  		if (res->start > start || res->end < end) {
825f787bb49676 Toshi Kani        2013-04-29  1445  			p = &res->sibling;
825f787bb49676 Toshi Kani        2013-04-29  1446  			continue;
825f787bb49676 Toshi Kani        2013-04-29  1447  		}
825f787bb49676 Toshi Kani        2013-04-29  1448  
825f787bb49676 Toshi Kani        2013-04-29  1449  		if (!(res->flags & IORESOURCE_MEM))
825f787bb49676 Toshi Kani        2013-04-29  1450  			break;
825f787bb49676 Toshi Kani        2013-04-29  1451  
e4ebc182a59bbb Nathan Fontenot   2025-04-03  1452  		if (busy_check && !(res->flags & IORESOURCE_BUSY)) {
e4ebc182a59bbb Nathan Fontenot   2025-04-03  1453  			p = &res->child;
e4ebc182a59bbb Nathan Fontenot   2025-04-03  1454  			continue;
e4ebc182a59bbb Nathan Fontenot   2025-04-03  1455  		}
e4ebc182a59bbb Nathan Fontenot   2025-04-03  1456  
e4ebc182a59bbb Nathan Fontenot   2025-04-03  1457  		if (res_desc != IORES_DESC_NONE && res->desc != res_desc) {
825f787bb49676 Toshi Kani        2013-04-29  1458  			p = &res->child;
825f787bb49676 Toshi Kani        2013-04-29  1459  			continue;
825f787bb49676 Toshi Kani        2013-04-29  1460  		}
825f787bb49676 Toshi Kani        2013-04-29  1461  
825f787bb49676 Toshi Kani        2013-04-29  1462  		/* found the target resource; let's adjust accordingly */
825f787bb49676 Toshi Kani        2013-04-29  1463  		if (res->start == start && res->end == end) {
825f787bb49676 Toshi Kani        2013-04-29  1464  			/* free the whole entry */
825f787bb49676 Toshi Kani        2013-04-29  1465  			*p = res->sibling;
ebff7d8f270d04 Yasuaki Ishimatsu 2013-04-29  1466  			free_resource(res);
825f787bb49676 Toshi Kani        2013-04-29  1467  		} else if (res->start == start && res->end != end) {
825f787bb49676 Toshi Kani        2013-04-29  1468  			/* adjust the start */
ec62d04e3fdc4b David Hildenbrand 2020-10-15  1469  			WARN_ON_ONCE(__adjust_resource(res, end + 1,
ec62d04e3fdc4b David Hildenbrand 2020-10-15  1470  						       res->end - end));
825f787bb49676 Toshi Kani        2013-04-29  1471  		} else if (res->start != start && res->end == end) {
825f787bb49676 Toshi Kani        2013-04-29  1472  			/* adjust the end */
ec62d04e3fdc4b David Hildenbrand 2020-10-15  1473  			WARN_ON_ONCE(__adjust_resource(res, res->start,
ec62d04e3fdc4b David Hildenbrand 2020-10-15  1474  						       start - res->start));
825f787bb49676 Toshi Kani        2013-04-29  1475  		} else {
ec62d04e3fdc4b David Hildenbrand 2020-10-15  1476  			/* split into two entries - we need a new resource */
825f787bb49676 Toshi Kani        2013-04-29  1477  			if (!new_res) {
ec62d04e3fdc4b David Hildenbrand 2020-10-15  1478  				new_res = alloc_resource(GFP_ATOMIC);
ec62d04e3fdc4b David Hildenbrand 2020-10-15  1479  				if (!new_res) {
ec62d04e3fdc4b David Hildenbrand 2020-10-15  1480  					alloc_nofail = true;
ec62d04e3fdc4b David Hildenbrand 2020-10-15  1481  					write_unlock(&resource_lock);
ec62d04e3fdc4b David Hildenbrand 2020-10-15  1482  					goto retry;
ec62d04e3fdc4b David Hildenbrand 2020-10-15  1483  				}
825f787bb49676 Toshi Kani        2013-04-29  1484  			}
825f787bb49676 Toshi Kani        2013-04-29  1485  			new_res->name = res->name;
825f787bb49676 Toshi Kani        2013-04-29  1486  			new_res->start = end + 1;
825f787bb49676 Toshi Kani        2013-04-29  1487  			new_res->end = res->end;
825f787bb49676 Toshi Kani        2013-04-29  1488  			new_res->flags = res->flags;
43ee493bde78da Toshi Kani        2016-01-26  1489  			new_res->desc = res->desc;
825f787bb49676 Toshi Kani        2013-04-29  1490  			new_res->parent = res->parent;
825f787bb49676 Toshi Kani        2013-04-29  1491  			new_res->sibling = res->sibling;
825f787bb49676 Toshi Kani        2013-04-29  1492  			new_res->child = NULL;
825f787bb49676 Toshi Kani        2013-04-29  1493  
ec62d04e3fdc4b David Hildenbrand 2020-10-15  1494  			if (WARN_ON_ONCE(__adjust_resource(res, res->start,
ec62d04e3fdc4b David Hildenbrand 2020-10-15  1495  							   start - res->start)))
825f787bb49676 Toshi Kani        2013-04-29  1496  				break;
825f787bb49676 Toshi Kani        2013-04-29  1497  			res->sibling = new_res;
825f787bb49676 Toshi Kani        2013-04-29  1498  			new_res = NULL;
825f787bb49676 Toshi Kani        2013-04-29  1499  		}
825f787bb49676 Toshi Kani        2013-04-29  1500  
825f787bb49676 Toshi Kani        2013-04-29  1501  		break;
825f787bb49676 Toshi Kani        2013-04-29  1502  	}
825f787bb49676 Toshi Kani        2013-04-29  1503  
825f787bb49676 Toshi Kani        2013-04-29  1504  	write_unlock(&resource_lock);
ebff7d8f270d04 Yasuaki Ishimatsu 2013-04-29  1505  	free_resource(new_res);
825f787bb49676 Toshi Kani        2013-04-29  1506  }
e4ebc182a59bbb Nathan Fontenot   2025-04-03  1507  #endif
e4ebc182a59bbb Nathan Fontenot   2025-04-03  1508  
Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
#5 lore raw

Re: [PATCH v3 1/4] kernel/resource: Provide mem region release for SOFT RESERVES

In reply to Terry Bowman
Quoted 3 lines · On Thu, 3 Apr 2025 13:33:12 -0500 Terry Bowman <[email protected]> wrote:

From: Nathan Fontenot <[email protected]>

Add a release_Sam_region_adjustable() interface to allow for

Who is Sam? (typo)

Quoted 22 lines

removing SOFT RESERVE memory resources. This extracts out the code to remove a mem region into a common __release_mem_region_adjustable() routine, this routine takes additional parameters of an IORES descriptor type to add checks for IORES_DESC_* and a flag to check for IORESOURCE_BUSY to control it's behavior.

The existing release_mem_region_adjustable() is a front end to the common code and a new release_srmem_region_adjustable() is added to release SOFT RESERVE resources.

---
 include/linux/ioport.h |  3 +++
 kernel/resource.c      | 55 +++++++++++++++++++++++++++++++++++++++---
 2 files changed, 54 insertions(+), 4 deletions(-)
include/linux/ioport.h +0 -0
index 5385349f0b8a..718360c9c724 100644

include/linux/ioport.h extern void __release_region(struct resource *, resource_size_t, · line 361

Quoted 5 lines
include/linux/ioport.h +2 -0
@@ -357,6 +357,9 @@ extern void __release_region(struct resource *, resource_size_t,
 #ifdef CONFIG_MEMORY_HOTREMOVE
 extern void release_mem_region_adjustable(resource_size_t, resource_size_t);
 #endif
+#ifdef CONFIG_CXL_REGION
+extern void release_srmem_region_adjustable(resource_size_t, resource_size_t);

I'm not sure the srmem is obvious enough. Maybe it's worth the long name to spell it out some more.. e.g. something like

extern void release_softresv_mem_region_adjustable() ?

Quoted 9 lines
include/linux/ioport.h +1 -0
+#endif
 #ifdef CONFIG_MEMORY_HOTPLUG
 extern void merge_system_ram_resource(struct resource *res);
 #endif
kernel/resource.c +0 -0
index 12004452d999..0195b31064b0 100644

kernel/resource.c void __release_region(struct resource *parent, resource_size_t start, · line 1392

Quoted 7 lines
kernel/resource.c +1 -1
@@ -1387,7 +1387,7 @@ void __release_region(struct resource *parent, resource_size_t start,
 }
 EXPORT_SYMBOL(__release_region);
 
-#ifdef CONFIG_MEMORY_HOTREMOVE
+#if defined(CONFIG_MEMORY_HOTREMOVE) || defined(CONFIG_CXL_REGION)
 /**
  * release_mem_region_adjustable - release a previously reserved memory region

Looks like you left the old docs which I'm guessing is not the intent.

Quoted 28 lines
kernel/resource.c +10 -2
  * @start: resource start address
@@ -1407,7 +1407,10 @@ EXPORT_SYMBOL(__release_region);
  *   assumes that all children remain in the lower address entry for
  *   simplicity.  Enhance this logic when necessary.
  */
-void release_mem_region_adjustable(resource_size_t start, resource_size_t size)
+static void __release_mem_region_adjustable(resource_size_t start,
+					    resource_size_t size,
+					    bool busy_check,
+					    int res_desc)
 {
 	struct resource *parent = &iomem_resource;
 	struct resource *new_res = NULL;
@@ -1446,7 +1449,12 @@ void release_mem_region_adjustable(resource_size_t start, resource_size_t size)
 		if (!(res->flags & IORESOURCE_MEM))
 			break;
 
-		if (!(res->flags & IORESOURCE_BUSY)) {
+		if (busy_check && !(res->flags & IORESOURCE_BUSY)) {
+			p = &res->child;
+			continue;
+		}
+
+		if (res_desc != IORES_DESC_NONE && res->desc != res_desc) {
 			p = &res->child;
 			continue;
 		}

kernel/resource.c void release_mem_region_adjustable(resource_size_t start, resource_size_t size) · line 1511

Quoted 9 lines
kernel/resource.c +5 -1
@@ -1496,7 +1504,46 @@ void release_mem_region_adjustable(resource_size_t start, resource_size_t size)
 	write_unlock(&resource_lock);
 	free_resource(new_res);
 }
-#endif	/* CONFIG_MEMORY_HOTREMOVE */
+#endif
+
+#ifdef CONFIG_MEMORY_HOTREMOVE
+/**
+ * release_mem_region_adjustable - release a previously reserved memory region

As above. I was surprised to see new docs in here for an existing function. I think you forgot to delete the now wrongly placed ones above.

Jonathan

#6 lore raw

Re: [PATCH v3 1/4] kernel/resource: Provide mem region release for SOFT RESERVES

Quoted 5 lines · Jonathan Cameron
Quoted 1 line · On Thu, 3 Apr 2025 13:33:12 -0500 Terry Bowman <[email protected]> wrote:

Add a release_Sam_region_adjustable() interface to allow for

Who is Sam? (typo)

Somebody's uncle?

...

Quoted 7 lines

include/linux/ioport.h extern void __release_region(struct resource *, resource_size_t, · line 361

Quoted 5 lines

#ifdef CONFIG_MEMORY_HOTREMOVE
extern void release_mem_region_adjustable(resource_size_t, resource_size_t); #endif

include/linux/ioport.h +2 -0
+#ifdef CONFIG_CXL_REGION
+extern void release_srmem_region_adjustable(resource_size_t, resource_size_t);

I'm not sure the srmem is obvious enough. Maybe it's worth the long name to spell it out some more.. e.g. something like

And perhaps drop 'extern' as it's not needed.

Quoted 5 lines

extern void release_softresv_mem_region_adjustable() ?

Quoted 3 lines

#ifdef CONFIG_MEMORY_HOTPLUG
extern void merge_system_ram_resource(struct resource *res); #endif

Signature · 2 lines
With Best Regards,
Andy Shevchenko
#7 lore raw

Re: [PATCH v3 1/4] kernel/resource: Provide mem region release for SOFT RESERVES

Quoted 19 lines · Andy Shevchenko
Quoted 5 lines · Jonathan Cameron
Quoted 1 line · On Thu, 3 Apr 2025 13:33:12 -0500 Terry Bowman <[email protected]> wrote:

Add a release_Sam_region_adjustable() interface to allow for

Who is Sam? (typo)

Somebody's uncle?

...

Quoted 7 lines

include/linux/ioport.h extern void __release_region(struct resource *, resource_size_t, · line 361

Quoted 5 lines

#ifdef CONFIG_MEMORY_HOTREMOVE
extern void release_mem_region_adjustable(resource_size_t, resource_size_t); #endif

include/linux/ioport.h +2 -0
+#ifdef CONFIG_CXL_REGION
+extern void release_srmem_region_adjustable(resource_size_t, resource_size_t);

I'm not sure the srmem is obvious enough. Maybe it's worth the long name to spell it out some more.. e.g. something like

Quoted 2 lines

And perhaps drop 'extern' as it's not needed.

Got it.

-Terry

Quoted 6 lines
Quoted 5 lines

extern void release_softresv_mem_region_adjustable() ?

Quoted 3 lines

#ifdef CONFIG_MEMORY_HOTPLUG
extern void merge_system_ram_resource(struct resource *res); #endif

#8 lore raw

Re: [PATCH v3 1/4] kernel/resource: Provide mem region release for SOFT RESERVES

In reply to Jonathan Cameron
Quoted 9 lines · Jonathan Cameron
Quoted 3 lines · On Thu, 3 Apr 2025 13:33:12 -0500 Terry Bowman <[email protected]> wrote:

From: Nathan Fontenot <[email protected]>

Add a release_Sam_region_adjustable() interface to allow for

Who is Sam? (typo)

Hi Jonathan,

It is a typo. I will fix.

Quoted 17 lines
Quoted 17 lines

removing SOFT RESERVE memory resources. This extracts out the code to remove a mem region into a common __release_mem_region_adjustable() routine, this routine takes additional parameters of an IORES descriptor type to add checks for IORES_DESC_* and a flag to check for IORESOURCE_BUSY to control it's behavior.

The existing release_mem_region_adjustable() is a front end to the common code and a new release_srmem_region_adjustable() is added to release SOFT RESERVE resources.

---
 include/linux/ioport.h |  3 +++
 kernel/resource.c      | 55 +++++++++++++++++++++++++++++++++++++++---
 2 files changed, 54 insertions(+), 4 deletions(-)
Quoted 13 lines

include/linux/ioport.h extern void __release_region(struct resource *, resource_size_t, · line 361

Quoted 10 lines
include/linux/ioport.h +2 -0
index 5385349f0b8a..718360c9c724 100644
@@ -357,6 +357,9 @@ extern void __release_region(struct resource *, resource_size_t,
 #ifdef CONFIG_MEMORY_HOTREMOVE
 extern void release_mem_region_adjustable(resource_size_t, resource_size_t);
 #endif
+#ifdef CONFIG_CXL_REGION
+extern void release_srmem_region_adjustable(resource_size_t, resource_size_t);

I'm not sure the srmem is obvious enough. Maybe it's worth the long name to spell it out some more.. e.g. something like

Yes, I'll update with a re name.

Quoted 13 lines

extern void release_softresv_mem_region_adjustable() ?

Quoted 12 lines
include/linux/ioport.h +1 -0
+#endif
 #ifdef CONFIG_MEMORY_HOTPLUG
 extern void merge_system_ram_resource(struct resource *res);
 #endif
kernel/resource.c +0 -0
index 12004452d999..0195b31064b0 100644
@@ -1387,7 +1387,7 @@ void __release_region(struct resource *parent, resource_size_t start,
 }
 EXPORT_SYMBOL(__release_region);
 
Quoted 7 lines

kernel/resource.c void __release_region(struct resource *parent, resource_size_t start, · line 1392

Quoted 4 lines
kernel/resource.c +1 -1
-#ifdef CONFIG_MEMORY_HOTREMOVE
+#if defined(CONFIG_MEMORY_HOTREMOVE) || defined(CONFIG_CXL_REGION)
 /**
  * release_mem_region_adjustable - release a previously reserved memory region

Looks like you left the old docs which I'm guessing is not the intent.

Correct, this is not intended. Ill remove.

Quoted 40 lines
Quoted 28 lines
kernel/resource.c +10 -2
  * @start: resource start address
@@ -1407,7 +1407,10 @@ EXPORT_SYMBOL(__release_region);
  *   assumes that all children remain in the lower address entry for
  *   simplicity.  Enhance this logic when necessary.
  */
-void release_mem_region_adjustable(resource_size_t start, resource_size_t size)
+static void __release_mem_region_adjustable(resource_size_t start,
+					    resource_size_t size,
+					    bool busy_check,
+					    int res_desc)
 {
 	struct resource *parent = &iomem_resource;
 	struct resource *new_res = NULL;
@@ -1446,7 +1449,12 @@ void release_mem_region_adjustable(resource_size_t start, resource_size_t size)
 		if (!(res->flags & IORESOURCE_MEM))
 			break;
 
-		if (!(res->flags & IORESOURCE_BUSY)) {
+		if (busy_check && !(res->flags & IORESOURCE_BUSY)) {
+			p = &res->child;
+			continue;
+		}
+
+		if (res_desc != IORES_DESC_NONE && res->desc != res_desc) {
 			p = &res->child;
 			continue;
 		}

kernel/resource.c void release_mem_region_adjustable(resource_size_t start, resource_size_t size) · line 1511

Quoted 9 lines
kernel/resource.c +5 -1
@@ -1496,7 +1504,46 @@ void release_mem_region_adjustable(resource_size_t start, resource_size_t size)
 	write_unlock(&resource_lock);
 	free_resource(new_res);
 }
-#endif	/* CONFIG_MEMORY_HOTREMOVE */
+#endif
+
+#ifdef CONFIG_MEMORY_HOTREMOVE
+/**
+ * release_mem_region_adjustable - release a previously reserved memory region

As above. I was surprised to see new docs in here for an existing function. I think you forgot to delete the now wrongly placed ones above.

Quoted 2 lines

Jonathan

Right, I'll remove the now old function comment.

-Terry

#9 lore raw

PATCH v3 2/4 cxl: Update Soft Reserved resources upon region creation

In reply to Terry Bowman

From: Nathan Fontenot <[email protected]>

Update handling of SOFT RESERVE iomem resources that intersect with CXL region resources to remove intersections from the SOFT RESERVE resources. The current approach of leaving SOFT RESERVE resources as is can cause failures during hotplug replace of CXL devices because the resource is not available for reuse after teardown of the CXL device.

To accomplish this the cxl acpi driver creates a worker thread at the end of cxl_acpi_probe(). This worker thread first waits for the CXL PCI CXL mem drivers have loaded. The cxl core/suspend.c code is updated to add a pci_loaded variable, in addition to the mem_active variable, that is updated when the pci driver loads. Remove CONFIG_CXL_SUSPEND Kconfig as it is no longer needed. A new cxl_wait_for_pci_mem() routine uses a waitqueue for both these driver to be loaded. The need to add this additional waitqueue is ensure the CXL PCI and CXL mem drivers have loaded before we wait for their probe, without it the cxl acpi probe worker thread calls wait_for_device_probe() before these drivers are loaded.

After the CXL PCI and CXL mem drivers load the cxl acpi worker thread uses wait_for_device_probe() to ensure device probe routines have completed.

Once probe completes and regions have been created, find all cxl regions that have been created and trim any SOFT RESERVE resources that intersect with the region.

Update cxl_acpi_exit() to cancel pending waitqueue work.

---
 drivers/cxl/Kconfig        |  4 ----
 drivers/cxl/acpi.c         | 28 ++++++++++++++++++++++++++
 drivers/cxl/core/Makefile  |  2 +-
 drivers/cxl/core/region.c  | 24 +++++++++++++++++++++-
 drivers/cxl/core/suspend.c | 41 ++++++++++++++++++++++++++++++++++++++
 drivers/cxl/cxl.h          |  3 +++
 drivers/cxl/cxlmem.h       |  9 ---------
 drivers/cxl/cxlpci.h       |  1 +
 drivers/cxl/pci.c          |  2 ++
 include/linux/pm.h         |  7 -------
 10 files changed, 99 insertions(+), 22 deletions(-)
drivers/cxl/Kconfig +0 -4
index 205547e5543a..c7377956c1d5 100644
@@ -117,10 +117,6 @@ config CXL_PORT
 	default CXL_BUS
 	tristate
 
-config CXL_SUSPEND
-	def_bool y
-	depends on SUSPEND && CXL_MEM
-
 config CXL_REGION
 	bool "CXL: Region Support"
 	default CXL_BUS
drivers/cxl/acpi.c +28 -0
index cb14829bb9be..94f2d649bb30 100644
@@ -7,6 +7,8 @@
 #include <linux/acpi.h>
 #include <linux/pci.h>
 #include <linux/node.h>
+#include <linux/pm.h>
+#include <linux/workqueue.h>
 #include <asm/div64.h>
 #include "cxlpci.h"
 #include "cxl.h"
@@ -813,6 +815,27 @@ static int pair_cxl_resource(struct device *dev, void *data)
 	return 0;
 }
 
+static void cxl_srmem_work_fn(struct work_struct *work)
+{
+	/* Wait for CXL PCI and mem drivers to load */
+	cxl_wait_for_pci_mem();
+
+	/*
+	 * Once the CXL PCI and mem drivers have loaded wait
+	 * for the driver probe routines to complete.
+	 */
+	wait_for_device_probe();
+
+	cxl_region_srmem_update();
+}
+
+DECLARE_WORK(cxl_sr_work, cxl_srmem_work_fn);
+
+static void cxl_srmem_update(void)
+{
+	schedule_work(&cxl_sr_work);
+}
+
 static int cxl_acpi_probe(struct platform_device *pdev)
 {
 	int rc;
@@ -887,6 +910,10 @@ static int cxl_acpi_probe(struct platform_device *pdev)
 
 	/* In case PCI is scanned before ACPI re-trigger memdev attach */
 	cxl_bus_rescan();
+
+	/* Update SOFT RESERVED resources that intersect with CXL regions */
+	cxl_srmem_update();
+
 	return 0;
 }
 
@@ -918,6 +945,7 @@ static int __init cxl_acpi_init(void)
 
 static void __exit cxl_acpi_exit(void)
 {
+	cancel_work_sync(&cxl_sr_work);
 	platform_driver_unregister(&cxl_acpi_driver);
 	cxl_bus_drain();
 }
drivers/cxl/core/Makefile +1 -1
index 086df97a0fcf..035864db8a32 100644
@@ -1,6 +1,6 @@
 # SPDX-License-Identifier: GPL-2.0
 obj-$(CONFIG_CXL_BUS) += cxl_core.o
-obj-$(CONFIG_CXL_SUSPEND) += suspend.o
+obj-y += suspend.o
 
 ccflags-y += -I$(srctree)/drivers/cxl
 CFLAGS_trace.o = -DTRACE_INCLUDE_PATH=. -I$(src)
drivers/cxl/core/region.c +23 -1
index c3f4dc244df7..25d70175f204 100644
@@ -10,6 +10,7 @@
 #include <linux/sort.h>
 #include <linux/idr.h>
 #include <linux/memory-tiers.h>
+#include <linux/ioport.h>
 #include <cxlmem.h>
 #include <cxl.h>
 #include "core.h"
@@ -2333,7 +2334,7 @@ const struct device_type cxl_region_type = {
 
 bool is_cxl_region(struct device *dev)
 {
-	return dev->type == &cxl_region_type;
+	return dev && dev->type == &cxl_region_type;
 }
 EXPORT_SYMBOL_NS_GPL(is_cxl_region, "CXL");
 
@@ -3443,6 +3444,27 @@ int cxl_add_to_region(struct cxl_port *root, struct cxl_endpoint_decoder *cxled)
 }
 EXPORT_SYMBOL_NS_GPL(cxl_add_to_region, "CXL");
 
+int cxl_region_srmem_update(void)
+{
+	struct device *dev = NULL;
+	struct cxl_region *cxlr;
+	struct resource *res;
+
+	do {
+		dev = bus_find_next_device(&cxl_bus_type, dev);
+		if (is_cxl_region(dev)) {
+			cxlr = to_cxl_region(dev);
+			res = cxlr->params.res;
+			release_srmem_region_adjustable(res->start,
+							resource_size(res));
+		}
+		put_device(dev);
+	} while (dev);
+
+	return 0;
+}
+EXPORT_SYMBOL_NS_GPL(cxl_region_srmem_update, "CXL");
+
 u64 cxl_port_get_spa_cache_alias(struct cxl_port *endpoint, u64 spa)
 {
 	struct cxl_region_ref *iter;
drivers/cxl/core/suspend.c +41 -0

59 lines not on this page · raw message

drivers/cxl/cxl.h +3 -0
index be8a7dc77719..40835ec692c8 100644
@@ -858,6 +858,7 @@ bool is_cxl_pmem_region(struct device *dev);
 struct cxl_pmem_region *to_cxl_pmem_region(struct device *dev);
 int cxl_add_to_region(struct cxl_port *root,
 		      struct cxl_endpoint_decoder *cxled);
+int cxl_region_srmem_update(void);
 struct cxl_dax_region *to_cxl_dax_region(struct device *dev);
 u64 cxl_port_get_spa_cache_alias(struct cxl_port *endpoint, u64 spa);
 #else
@@ -902,6 +903,8 @@ void cxl_coordinates_combine(struct access_coordinate *out,
 
 bool cxl_endpoint_decoder_reset_detected(struct cxl_port *port);
 
+void cxl_wait_for_pci_mem(void);
+
 /*
  * Unit test builds overrides this to __weak, find the 'strong' version
  * of these symbols in tools/testing/cxl/.
drivers/cxl/cxlmem.h +0 -9
index 3ec6b906371b..1bd1e88c4cc0 100644
@@ -853,17 +853,8 @@ int cxl_trigger_poison_list(struct cxl_memdev *cxlmd);
 int cxl_inject_poison(struct cxl_memdev *cxlmd, u64 dpa);
 int cxl_clear_poison(struct cxl_memdev *cxlmd, u64 dpa);
 
-#ifdef CONFIG_CXL_SUSPEND
 void cxl_mem_active_inc(void);
 void cxl_mem_active_dec(void);
-#else
-static inline void cxl_mem_active_inc(void)
-{
-}
-static inline void cxl_mem_active_dec(void)
-{
-}
-#endif
 
 int cxl_mem_sanitize(struct cxl_memdev *cxlmd, u16 cmd);
 
drivers/cxl/cxlpci.h +1 -0
index 54e219b0049e..5a811ac63fcf 100644
@@ -135,4 +135,5 @@ void read_cdat_data(struct cxl_port *port);
 void cxl_cor_error_detected(struct pci_dev *pdev);
 pci_ers_result_t cxl_error_detected(struct pci_dev *pdev,
 				    pci_channel_state_t state);
+void mark_cxl_pci_loaded(void);
 #endif /* __CXL_PCI_H__ */
drivers/cxl/pci.c +2 -0
index 4288f4814cc5..b784008489b3 100644
@@ -1185,6 +1185,8 @@ static int __init cxl_pci_driver_init(void)
 	if (rc)
 		pci_unregister_driver(&cxl_pci_driver);
 
+	mark_cxl_pci_loaded();
+
 	return rc;
 }
 
include/linux/pm.h +0 -7
index 78855d794342..11ff485c9722 100644
@@ -35,14 +35,7 @@ static inline void pm_vt_switch_unregister(struct device *dev)
 }
 #endif /* CONFIG_VT_CONSOLE_SLEEP */
 
-#ifdef CONFIG_CXL_SUSPEND
 bool cxl_mem_active(void);
-#else
-static inline bool cxl_mem_active(void)
-{
-	return false;
-}
-#endif
 
 /*
  * Device power management
Signature · 1 line
2.34.1
#10 lore raw

Re: [PATCH v3 2/4] cxl: Update Soft Reserved resources upon region creation

Quoted 9 lines · On Thu, 3 Apr 2025 13:33:13 -0500 Terry Bowman <[email protected]> wrote:

From: Nathan Fontenot <[email protected]>

Update handling of SOFT RESERVE iomem resources that intersect with CXL region resources to remove intersections from the SOFT RESERVE resources. The current approach of leaving SOFT RESERVE resources as is can cause failures during hotplug replace of CXL devices because the resource is not available for reuse after teardown of the CXL device.

To accomplish this the cxl acpi driver creates a worker thread at the

Inconsistent in capitalization. I'd just use CXL ACPI here given you used CXL PCI below.

Quoted 11 lines

end of cxl_acpi_probe(). This worker thread first waits for the CXL PCI CXL mem drivers have loaded. The cxl core/suspend.c code is updated to add a pci_loaded variable, in addition to the mem_active variable, that is updated when the pci driver loads. Remove CONFIG_CXL_SUSPEND Kconfig as it is no longer needed. A new cxl_wait_for_pci_mem() routine uses a waitqueue for both these driver to be loaded. The need to add this additional waitqueue is ensure the CXL PCI and CXL mem drivers have loaded before we wait for their probe, without it the cxl acpi probe worker thread calls wait_for_device_probe() before these drivers are loaded.

After the CXL PCI and CXL mem drivers load the cxl acpi worker thread

CXL ACPI

Quoted 2 lines

uses wait_for_device_probe() to ensure device probe routines have completed.

Does it matter if these drivers go away again? Everything seems to be one way at the moment.

Quoted 2 lines

Once probe completes and regions have been created, find all cxl

CXL

Quoted 14 lines

regions that have been created and trim any SOFT RESERVE resources that intersect with the region.

Update cxl_acpi_exit() to cancel pending waitqueue work.

drivers/cxl/cxl.h +0 -0
index be8a7dc77719..40835ec692c8 100644

drivers/cxl/cxl.h bool is_cxl_pmem_region(struct device *dev); · line 861

Quoted 4 lines
drivers/cxl/cxl.h +1 -0
@@ -858,6 +858,7 @@ bool is_cxl_pmem_region(struct device *dev);
 struct cxl_pmem_region *to_cxl_pmem_region(struct device *dev);
 int cxl_add_to_region(struct cxl_port *root,
 		      struct cxl_endpoint_decoder *cxled);
+int cxl_region_srmem_update(void);

As before: srmem is a bit obscure. Maybe spell it out more.

Quoted 8 lines
drivers/cxl/cxl.h +1 -0
 struct cxl_dax_region *to_cxl_dax_region(struct device *dev);
 u64 cxl_port_get_spa_cache_alias(struct cxl_port *endpoint, u64 spa);
 #else
@@ -902,6 +903,8 @@ void cxl_coordinates_combine(struct access_coordinate *out,
 
 bool cxl_endpoint_decoder_reset_detected(struct cxl_port *port);
 
+void cxl_wait_for_pci_mem(void);
#11 lore raw

Re: [PATCH v3 2/4] cxl: Update Soft Reserved resources upon region creation

Quoted 11 lines · Jonathan Cameron
Quoted 8 lines · On Thu, 3 Apr 2025 13:33:13 -0500 Terry Bowman <[email protected]> wrote:

From: Nathan Fontenot <[email protected]>

Update handling of SOFT RESERVE iomem resources that intersect with CXL region resources to remove intersections from the SOFT RESERVE resources. The current approach of leaving SOFT RESERVE resources as is can cause failures during hotplug replace of CXL devices because the resource is not available for reuse after teardown of the CXL device.

Quoted 5 lines
Quoted 1 line

To accomplish this the cxl acpi driver creates a worker thread at the

Inconsistent in capitalization. I'd just use CXL ACPI here given you used CXL PCI below.

Thanks. I will update.

Quoted 10 lines
Quoted 10 lines

end of cxl_acpi_probe(). This worker thread first waits for the CXL PCI CXL mem drivers have loaded. The cxl core/suspend.c code is updated to add a pci_loaded variable, in addition to the mem_active variable, that is updated when the pci driver loads. Remove CONFIG_CXL_SUSPEND Kconfig as it is no longer needed. A new cxl_wait_for_pci_mem() routine uses a waitqueue for both these driver to be loaded. The need to add this additional waitqueue is ensure the CXL PCI and CXL mem drivers have loaded before we wait for their probe, without it the cxl acpi probe worker thread calls wait_for_device_probe() before these drivers are loaded.

Quoted 9 lines
Quoted 1 line

After the CXL PCI and CXL mem drivers load the cxl acpi worker thread

CXL ACPI

Quoted 2 lines

uses wait_for_device_probe() to ensure device probe routines have completed.

Does it matter if these drivers go away again? Everything seems to be one way at the moment.

There is a maximum timeout wait period. I'll add these details to the commit message here.

Quoted 14 lines
Quoted 2 lines

Once probe completes and regions have been created, find all cxl

CXL

Quoted 7 lines

regions that have been created and trim any SOFT RESERVE resources that intersect with the region.

Update cxl_acpi_exit() to cancel pending waitqueue work.

Quoted 12 lines

drivers/cxl/cxl.h bool is_cxl_pmem_region(struct device *dev); · line 861

Quoted 9 lines
drivers/cxl/cxl.h +1 -0
index be8a7dc77719..40835ec692c8 100644
@@ -858,6 +858,7 @@ bool is_cxl_pmem_region(struct device *dev);
 struct cxl_pmem_region *to_cxl_pmem_region(struct device *dev);
 int cxl_add_to_region(struct cxl_port *root,
 		      struct cxl_endpoint_decoder *cxled);
+int cxl_region_srmem_update(void);

As before: srmem is a bit obscure. Maybe spell it out more.

Yes, will update.

-Terry

Quoted 9 lines
Quoted 8 lines
drivers/cxl/cxl.h +1 -0
 struct cxl_dax_region *to_cxl_dax_region(struct device *dev);
 u64 cxl_port_get_spa_cache_alias(struct cxl_port *endpoint, u64 spa);
 #else
@@ -902,6 +903,8 @@ void cxl_coordinates_combine(struct access_coordinate *out,
 
 bool cxl_endpoint_decoder_reset_detected(struct cxl_port *port);
 
+void cxl_wait_for_pci_mem(void);
#12 lore raw

Re: [PATCH v3 2/4] cxl: Update Soft Reserved resources upon region creation

In reply to Terry Bowman
kernel test robot noticed the following build errors:

Hi Terry,

kernel test robot noticed the following build errors:

[auto build test ERROR on aae0594a7053c60b82621136257c8b648c67b512]

url:    https://github.com/intel-lab-lkp/linux/commits/Terry-Bowman/kernel-resource-Provide-mem-region-release-for-SOFT-RESERVES/20250404-023601
base:   aae0594a7053c60b82621136257c8b648c67b512
patch link:    https://lore.kernel.org/r/20250403183315.286710-3-terry.bowman%40amd.com
patch subject: [PATCH v3 2/4] cxl: Update Soft Reserved resources upon region creation
config: hexagon-randconfig-001-20250404 (https://download.01.org/0day-ci/archive/20250404/[email protected]/config)
compiler: clang version 15.0.7 (https://github.com/llvm/llvm-project 8dfdcc7b7bf66834a761bd8de445840ef68e4d1a)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250404/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

In file included from drivers/cxl/core/suspend.c:7:

>> drivers/cxl/cxlpci.h:126:2: error: call to undeclared function 'pcie_capability_read_word'; ISO C99 and later do not support implicit function declarations [-Werror,-Wimplicit-function-declaration]
           pcie_capability_read_word(pdev, PCI_EXP_LNKSTA2, &lnksta2);
           ^

1 error generated.

vim +/pcie_capability_read_word +126 drivers/cxl/cxlpci.h

e0c818e00443ce Robert Richter 2024-02-16  116  
4d07a05397c8c1 Dave Jiang     2023-12-21  117  /*
4d07a05397c8c1 Dave Jiang     2023-12-21  118   * CXL v3.0 6.2.3 Table 6-4
4d07a05397c8c1 Dave Jiang     2023-12-21  119   * The table indicates that if PCIe Flit Mode is set, then CXL is in 256B flits
4d07a05397c8c1 Dave Jiang     2023-12-21  120   * mode, otherwise it's 68B flits mode.
4d07a05397c8c1 Dave Jiang     2023-12-21  121   */
4d07a05397c8c1 Dave Jiang     2023-12-21  122  static inline bool cxl_pci_flit_256(struct pci_dev *pdev)
4d07a05397c8c1 Dave Jiang     2023-12-21  123  {
4d07a05397c8c1 Dave Jiang     2023-12-21  124  	u16 lnksta2;
4d07a05397c8c1 Dave Jiang     2023-12-21  125  
4d07a05397c8c1 Dave Jiang     2023-12-21 @126  	pcie_capability_read_word(pdev, PCI_EXP_LNKSTA2, &lnksta2);
4d07a05397c8c1 Dave Jiang     2023-12-21  127  	return lnksta2 & PCI_EXP_LNKSTA2_FLIT;
4d07a05397c8c1 Dave Jiang     2023-12-21  128  }
4d07a05397c8c1 Dave Jiang     2023-12-21  129  
Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
#13 lore raw

PATCH v3 3/4 dax/mum: Save the dax mum platform device pointer

In reply to Terry Bowman

From: Nathan Fontenot <[email protected]>

In order to handle registering hmem devices for SOFT RESERVE resources after the dax hmem device initialization occurs we need to save a reference to the dax hmem platform device that will be used in a following patch.

Saving the platform device pointer also allows us to clean up the walk_hmem_resources() routine to no require the struct device argument.

There should be no functional changes.

---
 drivers/dax/hmem/device.c | 4 ++--
 drivers/dax/hmem/hmem.c   | 9 ++++++---
 include/linux/dax.h       | 5 ++---
 3 files changed, 10 insertions(+), 8 deletions(-)
drivers/dax/hmem/device.c +2 -2
index f9e1a76a04a9..59ad44761191 100644
@@ -17,14 +17,14 @@ static struct resource hmem_active = {
 	.flags = IORESOURCE_MEM,
 };
 
-int walk_hmem_resources(struct device *host, walk_hmem_fn fn)
+int walk_hmem_resources(walk_hmem_fn fn)
 {
 	struct resource *res;
 	int rc = 0;
 
 	mutex_lock(&hmem_resource_lock);
 	for (res = hmem_active.child; res; res = res->sibling) {
-		rc = fn(host, (int) res->desc, res);
+		rc = fn((int) res->desc, res);
 		if (rc)
 			break;
 	}
drivers/dax/hmem/hmem.c +6 -3
index 5e7c53f18491..3aedef5f1be1 100644
@@ -9,6 +9,8 @@
 static bool region_idle;
 module_param_named(region_idle, region_idle, bool, 0644);
 
+static struct platform_device *dax_hmem_pdev;
+
 static int dax_hmem_probe(struct platform_device *pdev)
 {
 	unsigned long flags = IORESOURCE_DAX_KMEM;
@@ -59,9 +61,9 @@ static void release_hmem(void *pdev)
 	platform_device_unregister(pdev);
 }
 
-static int hmem_register_device(struct device *host, int target_nid,
-				const struct resource *res)
+static int hmem_register_device(int target_nid, const struct resource *res)
 {
+	struct device *host = &dax_hmem_pdev->dev;
 	struct platform_device *pdev;
 	struct memregion_info info;
 	long id;
@@ -125,7 +127,8 @@ static int hmem_register_device(struct device *host, int target_nid,
 
 static int dax_hmem_platform_probe(struct platform_device *pdev)
 {
-	return walk_hmem_resources(&pdev->dev, hmem_register_device);
+	dax_hmem_pdev = pdev;
+	return walk_hmem_resources(hmem_register_device);
 }
 
 static struct platform_driver dax_hmem_platform_driver = {
include/linux/dax.h +2 -3
index df41a0017b31..4b4d16f94898 100644
@@ -277,7 +277,6 @@ static inline void hmem_register_resource(int target_nid, struct resource *r)
 }
 #endif
 
-typedef int (*walk_hmem_fn)(struct device *dev, int target_nid,
-			    const struct resource *res);
-int walk_hmem_resources(struct device *dev, walk_hmem_fn fn);
+typedef int (*walk_hmem_fn)(int target_nid, const struct resource *res);
+int walk_hmem_resources(walk_hmem_fn fn);
 #endif
Signature · 1 line
2.34.1
#14 lore raw

Re: [PATCH v3 3/4] dax/mum: Save the dax mum platform device pointer

Quoted 1 line · On Thu, 3 Apr 2025 13:33:14 -0500 Terry Bowman <[email protected]> wrote:

From: Nathan Fontenot <[email protected]>

mum?

Quoted 14 lines

In order to handle registering hmem devices for SOFT RESERVE resources after the dax hmem device initialization occurs we need to save a reference to the dax hmem platform device that will be used in a following patch.

Saving the platform device pointer also allows us to clean up the walk_hmem_resources() routine to no require the struct device argument.

There should be no functional changes.

#15 lore raw

Re: [PATCH v3 3/4] dax/mum: Save the dax mum platform device pointer

Quoted 22 lines · Jonathan Cameron
Quoted 1 line · On Thu, 3 Apr 2025 13:33:14 -0500 Terry Bowman <[email protected]> wrote:

From: Nathan Fontenot <[email protected]>

mum?

Quoted 14 lines

In order to handle registering hmem devices for SOFT RESERVE resources after the dax hmem device initialization occurs we need to save a reference to the dax hmem platform device that will be used in a following patch.

Saving the platform device pointer also allows us to clean up the walk_hmem_resources() routine to no require the struct device argument.

There should be no functional changes.

Thanks. Updated to be hmem.

-Terry

#16 lore raw

PATCH v3 4/4 cxl/dax: Delay consumption of SOFT RESERVE resources

In reply to Terry Bowman

From: Nathan Fontenot <[email protected]>

The dax hmem device initialization will consume any iomem SOFT RESERVE resources prior to CXL region creation. To allow for the CXL driver to complete region creation and trim any SOFT RESERVE resources before the dax driver consumes them we need to delay the dax driver's search for SOFT RESERVEs.

To do this the dax driver hmem device initialization code skips the walk of the iomem resource tree if the CXL ACPI driver is enabled. This allows the CXL driver to complete region creation and trim any SOFT RESERVES. Once the CXL driver completes this, the CXL driver then registers any remaining SOFT RESERVE resources with the dax hmem driver.

---
 drivers/cxl/core/region.c | 10 +++++++++
 drivers/dax/hmem/device.c | 43 ++++++++++++++++++++-------------------
 drivers/dax/hmem/hmem.c   |  3 ++-
 include/linux/dax.h       |  6 ++++++
 4 files changed, 40 insertions(+), 22 deletions(-)
drivers/cxl/core/region.c +10 -0
index 25d70175f204..bf4a4371d98b 100644
@@ -11,6 +11,7 @@
 #include <linux/idr.h>
 #include <linux/memory-tiers.h>
 #include <linux/ioport.h>
+#include <linux/dax.h>
 #include <cxlmem.h>
 #include <cxl.h>
 #include "core.h"
@@ -3444,6 +3445,11 @@ int cxl_add_to_region(struct cxl_port *root, struct cxl_endpoint_decoder *cxled)
 }
 EXPORT_SYMBOL_NS_GPL(cxl_add_to_region, "CXL");
 
+static int cxl_srmem_register(struct resource *res, void *unused)
+{
+	return hmem_register_device(phys_to_target_node(res->start), res);
+}
+
 int cxl_region_srmem_update(void)
 {
 	struct device *dev = NULL;
@@ -3461,6 +3467,10 @@ int cxl_region_srmem_update(void)
 		put_device(dev);
 	} while (dev);
 
+	/* Now register any remaining SOFT RESERVES with dax */
+	walk_iomem_res_desc(IORES_DESC_SOFT_RESERVED, IORESOURCE_MEM,
+			    0, -1, NULL, cxl_srmem_register);
+
 	return 0;
 }
 EXPORT_SYMBOL_NS_GPL(cxl_region_srmem_update, "CXL");
drivers/dax/hmem/device.c +22 -21

68 lines not on this page · raw message

drivers/dax/hmem/hmem.c +2 -1
index 3aedef5f1be1..a206b9b383e4 100644
@@ -61,7 +61,7 @@ static void release_hmem(void *pdev)
 	platform_device_unregister(pdev);
 }
 
-static int hmem_register_device(int target_nid, const struct resource *res)
+int hmem_register_device(int target_nid, const struct resource *res)
 {
 	struct device *host = &dax_hmem_pdev->dev;
 	struct platform_device *pdev;
@@ -124,6 +124,7 @@ static int hmem_register_device(int target_nid, const struct resource *res)
 	platform_device_put(pdev);
 	return rc;
 }
+EXPORT_SYMBOL_GPL(hmem_register_device);
 
 static int dax_hmem_platform_probe(struct platform_device *pdev)
 {
include/linux/dax.h +6 -0
index 4b4d16f94898..a1a75ade9ea7 100644
@@ -271,10 +271,16 @@ static inline int dax_mem2blk_err(int err)
 
 #ifdef CONFIG_DEV_DAX_HMEM_DEVICES
 void hmem_register_resource(int target_nid, struct resource *r);
+int hmem_register_device(int target_nid, const struct resource *res);
 #else
 static inline void hmem_register_resource(int target_nid, struct resource *r)
 {
 }
+
+static inline int hmem_register_device(int target_nid, const struct resource *res)
+{
+	return 0;
+}
 #endif
 
 typedef int (*walk_hmem_fn)(int target_nid, const struct resource *res);
Signature · 1 line
2.34.1
#17 lore raw

Re: [PATCH v3 4/4] cxl/dax: Delay consumption of SOFT RESERVE resources

Quoted 15 lines · On Thu, 3 Apr 2025 13:33:15 -0500 Terry Bowman <[email protected]> wrote:

From: Nathan Fontenot <[email protected]>

The dax hmem device initialization will consume any iomem SOFT RESERVE resources prior to CXL region creation. To allow for the CXL driver to complete region creation and trim any SOFT RESERVE resources before the dax driver consumes them we need to delay the dax driver's search for SOFT RESERVEs.

To do this the dax driver hmem device initialization code skips the walk of the iomem resource tree if the CXL ACPI driver is enabled. This allows the CXL driver to complete region creation and trim any SOFT RESERVES. Once the CXL driver completes this, the CXL driver then registers any remaining SOFT RESERVE resources with the dax hmem driver.

Quoted 2 lines

Looks fine to me, but I'm not feeling confident enough of this area of the kernel to give a tag.

Jonathan

#18 lore raw
In reply to Terry Bowman

Hi Terry,

If I understand correctly, this patch set has only considered the situation where the soft reserved area and the region are exactly the same, as in pattern 1.

However, I believe we also need to consider situations where these two are not equal, which are outlined in pattern 2 and 3 below. Let me explain them:

===========================================
Pattern 1:
- region0 will be created during OS booting due to programed hdm decoder
- After OS booted, region0 can be re-created again after destroy it

┌────────────────────┐
│       CFMW         │
└────────────────────┘
┌────────────────────┐
│    reserved0       │
└────────────────────┘
┌────────────────────┐
│       mem0         │
└────────────────────┘
┌────────────────────┐
│      region0       │
└────────────────────┘

Pattern 2:
The HDM decoder is not in a committed state, so during the kernel boot process, egion0 will not be created automatically. In this case, the soft reserved area will not be removed from the iomem tree. After the OS starts,
users cannot create a region (cxl create-region) either, as there should be an intersection between the soft reserved area and the region.

┌────────────────────┐
│       CFMW         │
└────────────────────┘
┌────────────────────┐
│    reserved0       │
└────────────────────┘
┌────────────────────┐
│       mem0*        │
└────────────────────┘
┌────────────────────┐
│      N/A           │ region0
└────────────────────┘

*HDM decoder in mem0 is not committed.

Pattern 3:
Region0 is a child of the soft reserved area. In this case, the soft reserved area will not be removed from the iomem tree, resulting in being unable to be recreated later after destroy.

┌────────────────────┐
│       CFMW         │
└────────────────────┘
┌────────────────────┐
│   reserved         │
└────────────────────┘
┌────────────────────┐
│ mem0    | mem1*    │
└────────────────────┘
┌────────────────────┐
│region0  |  N/A     │ region1
└────────────────────┘

*HDM decoder in mem1 is not committed.

Thanks
Zhijian

Quoted 68 lines · Terry Bowman

Add the ability to manage SOFT RESERVE iomem resources prior to them being added to the iomem resource tree. This allows drivers, such as CXL, to remove any pieces of the SOFT RESERVE resource that intersect with created CXL regions.

The current approach of leaving the SOFT RESERVE resources as is can cause failures during hotplug of devices, such as CXL, because the resource is not available for reuse after teardown of the device.

The approach is to add SOFT RESERVE resources to a separate tree during boot. This allows any drivers to update the SOFT RESERVE resources before they are merged into the iomem resource tree. In addition a notifier chain is added so that drivers can be notified when these SOFT RESERVE resources are added to the ioeme resource tree.

The CXL driver is modified to use a worker thread that waits for the CXL PCI and CXL mem drivers to be loaded and for their probe routine to complete. Then the driver walks through any created CXL regions to trim any intersections with SOFT RESERVE resources in the iomem tree.

The dax driver uses the new soft reserve notifier chain so it can consume any remaining SOFT RESERVES once they're added to the iomem tree.

V3 updates:
- Remove srmem resource tree from kernel/resource.c, this is no longer needed in the current implementation. All SOFT RESERVE resources now put on the iomem resource tree.
- Remove the no longer needed SOFT_RESERVED_MANAGED kernel config option.
- Add the 'nid' parameter back to hmem_register_resource();
- Remove the no longer used soft reserve notification chain (introduced in v2). The dax driver is now notified of SOFT RESERVED resources by the CXL driver.

v2 updates:
- Add config option SOFT_RESERVE_MANAGED to control use of the separate srmem resource tree at boot.
- Only add SOFT RESERVE resources to the soft reserve tree during boot, they go to the iomem resource tree after boot.
- Remove the resource trimming code in the previous patch to re-use the existing code in kernel/resource.c
- Add functionality for the cxl acpi driver to wait for the cxl PCI and me drivers to load.

Nathan Fontenot (4):
kernel/resource: Provide mem region release for SOFT RESERVES
cxl: Update Soft Reserved resources upon region creation dax/mum: Save the dax mum platform device pointer cxl/dax: Delay consumption of SOFT RESERVE resources

  drivers/cxl/Kconfig        |  4 ---
  drivers/cxl/acpi.c         | 28 +++++++++++++++++++
  drivers/cxl/core/Makefile  |  2 +-
  drivers/cxl/core/region.c  | 34 ++++++++++++++++++++++-
  drivers/cxl/core/suspend.c | 41 ++++++++++++++++++++++++++++
  drivers/cxl/cxl.h          |  3 +++
  drivers/cxl/cxlmem.h       |  9 -------
  drivers/cxl/cxlpci.h       |  1 +
  drivers/cxl/pci.c          |  2 ++
  drivers/dax/hmem/device.c  | 47 ++++++++++++++++----------------
  drivers/dax/hmem/hmem.c    | 10 ++++---
  include/linux/dax.h        | 11 +++++---
  include/linux/ioport.h     |  3 +++
  include/linux/pm.h         |  7 -----
  kernel/resource.c          | 55 +++++++++++++++++++++++++++++++++++---
  15 files changed, 202 insertions(+), 55 deletions(-)

base-commit: aae0594a7053c60b82621136257c8b648c67b512

#19 lore raw
Quoted 4 lines · Zhijian Li (Fujitsu)

Hi Terry,

If I understand correctly, this patch set has only considered the situation where the soft reserved area and the region are exactly the same, as in pattern 1.

Hi Zhijian,

I'm working on example test case(s) for your questions. I'll respond here.

Regards,
Terry

Quoted 135 lines

However, I believe we also need to consider situations where these two are not equal, which are outlined in pattern 2 and 3 below. Let me explain them:

===========================================
Pattern 1:
- region0 will be created during OS booting due to programed hdm decoder
- After OS booted, region0 can be re-created again after destroy it

┌────────────────────┐
│       CFMW         │
└────────────────────┘
┌────────────────────┐
│    reserved0       │
└────────────────────┘
┌────────────────────┐
│       mem0         │
└────────────────────┘
┌────────────────────┐
│      region0       │
└────────────────────┘

Pattern 2:
The HDM decoder is not in a committed state, so during the kernel boot process, egion0 will not be created automatically. In this case, the soft reserved area will not be removed from the iomem tree. After the OS starts,
users cannot create a region (cxl create-region) either, as there should be an intersection between the soft reserved area and the region.

┌────────────────────┐
│       CFMW         │
└────────────────────┘
┌────────────────────┐
│    reserved0       │
└────────────────────┘
┌────────────────────┐
│       mem0*        │
└────────────────────┘
┌────────────────────┐
│      N/A           │ region0
└────────────────────┘

*HDM decoder in mem0 is not committed.

Pattern 3:
Region0 is a child of the soft reserved area. In this case, the soft reserved area will not be removed from the iomem tree, resulting in being unable to be recreated later after destroy.

┌────────────────────┐
│       CFMW         │
└────────────────────┘
┌────────────────────┐
│   reserved         │
└────────────────────┘
┌────────────────────┐
│ mem0    | mem1*    │
└────────────────────┘
┌────────────────────┐
│region0  |  N/A     │ region1
└────────────────────┘

*HDM decoder in mem1 is not committed.

Thanks
Zhijian

Quoted 68 lines · Terry Bowman

Add the ability to manage SOFT RESERVE iomem resources prior to them being added to the iomem resource tree. This allows drivers, such as CXL, to remove any pieces of the SOFT RESERVE resource that intersect with created CXL regions.

The current approach of leaving the SOFT RESERVE resources as is can cause failures during hotplug of devices, such as CXL, because the resource is not available for reuse after teardown of the device.

The approach is to add SOFT RESERVE resources to a separate tree during boot. This allows any drivers to update the SOFT RESERVE resources before they are merged into the iomem resource tree. In addition a notifier chain is added so that drivers can be notified when these SOFT RESERVE resources are added to the ioeme resource tree.

The CXL driver is modified to use a worker thread that waits for the CXL PCI and CXL mem drivers to be loaded and for their probe routine to complete. Then the driver walks through any created CXL regions to trim any intersections with SOFT RESERVE resources in the iomem tree.

The dax driver uses the new soft reserve notifier chain so it can consume any remaining SOFT RESERVES once they're added to the iomem tree.

V3 updates:
- Remove srmem resource tree from kernel/resource.c, this is no longer needed in the current implementation. All SOFT RESERVE resources now put on the iomem resource tree.
- Remove the no longer needed SOFT_RESERVED_MANAGED kernel config option.
- Add the 'nid' parameter back to hmem_register_resource();
- Remove the no longer used soft reserve notification chain (introduced in v2). The dax driver is now notified of SOFT RESERVED resources by the CXL driver.

v2 updates:
- Add config option SOFT_RESERVE_MANAGED to control use of the separate srmem resource tree at boot.
- Only add SOFT RESERVE resources to the soft reserve tree during boot, they go to the iomem resource tree after boot.
- Remove the resource trimming code in the previous patch to re-use the existing code in kernel/resource.c
- Add functionality for the cxl acpi driver to wait for the cxl PCI and me drivers to load.

Nathan Fontenot (4):
kernel/resource: Provide mem region release for SOFT RESERVES
cxl: Update Soft Reserved resources upon region creation dax/mum: Save the dax mum platform device pointer cxl/dax: Delay consumption of SOFT RESERVE resources

  drivers/cxl/Kconfig        |  4 ---
  drivers/cxl/acpi.c         | 28 +++++++++++++++++++
  drivers/cxl/core/Makefile  |  2 +-
  drivers/cxl/core/region.c  | 34 ++++++++++++++++++++++-
  drivers/cxl/core/suspend.c | 41 ++++++++++++++++++++++++++++
  drivers/cxl/cxl.h          |  3 +++
  drivers/cxl/cxlmem.h       |  9 -------
  drivers/cxl/cxlpci.h       |  1 +
  drivers/cxl/pci.c          |  2 ++
  drivers/dax/hmem/device.c  | 47 ++++++++++++++++----------------
  drivers/dax/hmem/hmem.c    | 10 ++++---
  include/linux/dax.h        | 11 +++++---
  include/linux/ioport.h     |  3 +++
  include/linux/pm.h         |  7 -----
  kernel/resource.c          | 55 +++++++++++++++++++++++++++++++++++---
  15 files changed, 202 insertions(+), 55 deletions(-)

base-commit: aae0594a7053c60b82621136257c8b648c67b512

#20 lore raw
In reply to Zhijian Li (Fujitsu)

Hi Zhijian,

We recreated the failure for the cases you mentioned below. We will be adding the fix into v4 I am working on now.

Regards,
Terry

Quoted 140 lines · Zhijian Li (Fujitsu)

Hi Terry,

If I understand correctly, this patch set has only considered the situation where the soft reserved area and the region are exactly the same, as in pattern 1.

However, I believe we also need to consider situations where these two are not equal, which are outlined in pattern 2 and 3 below. Let me explain them:

===========================================
Pattern 1:
- region0 will be created during OS booting due to programed hdm decoder
- After OS booted, region0 can be re-created again after destroy it

┌────────────────────┐
│       CFMW         │
└────────────────────┘
┌────────────────────┐
│    reserved0       │
└────────────────────┘
┌────────────────────┐
│       mem0         │
└────────────────────┘
┌────────────────────┐
│      region0       │
└────────────────────┘

Pattern 2:
The HDM decoder is not in a committed state, so during the kernel boot process, egion0 will not be created automatically. In this case, the soft reserved area will not be removed from the iomem tree. After the OS starts,
users cannot create a region (cxl create-region) either, as there should be an intersection between the soft reserved area and the region.

┌────────────────────┐
│       CFMW         │
└────────────────────┘
┌────────────────────┐
│    reserved0       │
└────────────────────┘
┌────────────────────┐
│       mem0*        │
└────────────────────┘
┌────────────────────┐
│      N/A           │ region0
└────────────────────┘

*HDM decoder in mem0 is not committed.

Pattern 3:
Region0 is a child of the soft reserved area. In this case, the soft reserved area will not be removed from the iomem tree, resulting in being unable to be recreated later after destroy.

┌────────────────────┐
│       CFMW         │
└────────────────────┘
┌────────────────────┐
│   reserved         │
└────────────────────┘
┌────────────────────┐
│ mem0    | mem1*    │
└────────────────────┘
┌────────────────────┐
│region0  |  N/A     │ region1
└────────────────────┘

*HDM decoder in mem1 is not committed.

Thanks
Zhijian

Quoted 68 lines · Terry Bowman

Add the ability to manage SOFT RESERVE iomem resources prior to them being added to the iomem resource tree. This allows drivers, such as CXL, to remove any pieces of the SOFT RESERVE resource that intersect with created CXL regions.

The current approach of leaving the SOFT RESERVE resources as is can cause failures during hotplug of devices, such as CXL, because the resource is not available for reuse after teardown of the device.

The approach is to add SOFT RESERVE resources to a separate tree during boot. This allows any drivers to update the SOFT RESERVE resources before they are merged into the iomem resource tree. In addition a notifier chain is added so that drivers can be notified when these SOFT RESERVE resources are added to the ioeme resource tree.

The CXL driver is modified to use a worker thread that waits for the CXL PCI and CXL mem drivers to be loaded and for their probe routine to complete. Then the driver walks through any created CXL regions to trim any intersections with SOFT RESERVE resources in the iomem tree.

The dax driver uses the new soft reserve notifier chain so it can consume any remaining SOFT RESERVES once they're added to the iomem tree.

V3 updates:
- Remove srmem resource tree from kernel/resource.c, this is no longer needed in the current implementation. All SOFT RESERVE resources now put on the iomem resource tree.
- Remove the no longer needed SOFT_RESERVED_MANAGED kernel config option.
- Add the 'nid' parameter back to hmem_register_resource();
- Remove the no longer used soft reserve notification chain (introduced in v2). The dax driver is now notified of SOFT RESERVED resources by the CXL driver.

v2 updates:
- Add config option SOFT_RESERVE_MANAGED to control use of the separate srmem resource tree at boot.
- Only add SOFT RESERVE resources to the soft reserve tree during boot, they go to the iomem resource tree after boot.
- Remove the resource trimming code in the previous patch to re-use the existing code in kernel/resource.c
- Add functionality for the cxl acpi driver to wait for the cxl PCI and me drivers to load.

Nathan Fontenot (4):
kernel/resource: Provide mem region release for SOFT RESERVES
cxl: Update Soft Reserved resources upon region creation dax/mum: Save the dax mum platform device pointer cxl/dax: Delay consumption of SOFT RESERVE resources

  drivers/cxl/Kconfig        |  4 ---
  drivers/cxl/acpi.c         | 28 +++++++++++++++++++
  drivers/cxl/core/Makefile  |  2 +-
  drivers/cxl/core/region.c  | 34 ++++++++++++++++++++++-
  drivers/cxl/core/suspend.c | 41 ++++++++++++++++++++++++++++
  drivers/cxl/cxl.h          |  3 +++
  drivers/cxl/cxlmem.h       |  9 -------
  drivers/cxl/cxlpci.h       |  1 +
  drivers/cxl/pci.c          |  2 ++
  drivers/dax/hmem/device.c  | 47 ++++++++++++++++----------------
  drivers/dax/hmem/hmem.c    | 10 ++++---
  include/linux/dax.h        | 11 +++++---
  include/linux/ioport.h     |  3 +++
  include/linux/pm.h         |  7 -----
  kernel/resource.c          | 55 +++++++++++++++++++++++++++++++++++---
  15 files changed, 202 insertions(+), 55 deletions(-)

base-commit: aae0594a7053c60b82621136257c8b648c67b512