Skip to contentSkip to conversation

DCD: Add support for Dynamic Capacity Devices (DCD)

PATCH Superseded Ira Weiny 14 participants 134 messages Oct 7, 2024 – Oct 25, 2024 Page 1 of 2

Subsystems (9)

v4 of 9 Superseded Latest is v9 →

lore
Share & tools
lore

Ira Weiny

author
#0 lore raw

v4 cover DCD: Add support for Dynamic Capacity Devices (DCD)

A git tree of this series can be found here:

	https://github.com/weiny2/linux-kernel/tree/dcd-v4-2024-10-04

Series info
===========

This series has 5 parts:

Patch 1-3: Add %pra printk format for struct range Patch 4: Add core range_overlaps() function Patch 5-6: CXL clean up/prelim patches Patch 7-26: Core DCD support
Patch 27-28: cxl_test support

Background
==========

A Dynamic Capacity Device (DCD) (CXL 3.1 sec 9.13.3) is a CXL memory device that allows memory capacity within a region to change dynamically without the need for resetting the device, reconfiguring HDM decoders, or reconfiguring software DAX regions.

One of the biggest use cases for Dynamic Capacity is to allow hosts to share memory dynamically within a data center without increasing the per-host attached memory.

The general flow for the addition or removal of memory is to have an orchestrator coordinate the use of the memory. Generally there are 5 actors in such a system, the Orchestrator, Fabric Manager, the Logical device, the Host Kernel, and a Host User.

Typical work flows are shown below.

Orchestrator FM Device Host Kernel Host User

    |             |           |            |              |
    |-------------- Create region ----------------------->|
    |             |           |            |              |
    |             |           |            |<-- Create ---|
    |             |           |            |    Region    |
    |<------------- Signal done --------------------------|
    |             |           |            |              |
    |-- Add ----->|-- Add --->|--- Add --->|              |
    |  Capacity   |  Extent   |   Extent   |              |
    |             |           |            |              |
    |             |<- Accept -|<- Accept  -|              |
    |             |   Extent  |   Extent   |              |
    |             |           |            |<- Create --->|
    |             |           |            |   DAX dev    |-- Use memory
    |             |           |            |              |   |
    |             |           |            |              |   |
    |             |           |            |<- Release ---| <-+
    |             |           |            |   DAX dev    |
    |             |           |            |              |
    |<------------- Signal done --------------------------|
    |             |           |            |              |
    |-- Remove -->|- Release->|- Release ->|              |
    |  Capacity   |  Extent   |   Extent   |              |
    |             |           |            |              |
    |             |<- Release-|<- Release -|              |
    |             |   Extent  |   Extent   |              |
    |             |           |            |              |
    |-- Add ----->|-- Add --->|--- Add --->|              |
    |  Capacity   |  Extent   |   Extent   |              |
    |             |           |            |              |
    |             |<- Accept -|<- Accept  -|              |
    |             |   Extent  |   Extent   |              |
    |             |           |            |<- Create ----|
    |             |           |            |   DAX dev    |-- Use memory
    |             |           |            |              |   |
    |             |           |            |<- Release ---| <-+
    |             |           |            |   DAX dev    |
    |<------------- Signal done --------------------------|
    |             |           |            |              |
    |-- Remove -->|- Release->|- Release ->|              |
    |  Capacity   |  Extent   |   Extent   |              |
    |             |           |            |              |
    |             |<- Release-|<- Release -|              |
    |             |   Extent  |   Extent   |              |
    |             |           |            |              |
    |-- Add ----->|-- Add --->|--- Add --->|              |
    |  Capacity   |  Extent   |   Extent   |              |
    |             |           |            |<- Create ----|
    |             |           |            |   DAX dev    |-- Use memory
    |             |           |            |              |   |
    |-- Remove -->|- Release->|- Release ->|              |   |
    |  Capacity   |  Extent   |   Extent   |              |   |
    |             |           |            |              |   |
    |             |           |     (Release Ignored)     |   |
    |             |           |            |              |   |
    |             |           |            |<- Release ---| <-+
    |             |           |            |   DAX dev    |
    |<------------- Signal done --------------------------|
    |             |           |            |              |
    |             |- Release->|- Release ->|              |
    |             |  Extent   |   Extent   |              |
    |             |           |            |              |
    |             |<- Release-|<- Release -|              |
    |             |   Extent  |   Extent   |              |
    |             |           |            |<- Destroy ---|
    |             |           |            |   Region     |
    |             |           |            |              |

Implementation
==============

The series still requires the creation of regions and DAX devices to be closely synchronized with the Orchestrator and Fabric Manager. The host kernel will reject extents if a region is not yet created. It also ignores extent release if memory is in use (DAX device created). These synchronizations are not anticipated to be an issue with real applications.

In order to allow for capacity to be added and removed a new concept of a sparse DAX region is introduced. A sparse DAX region may have 0 or more bytes of available space. The total space depends on the number and size of the extents which have been added.

Initially it is anticipated that users of the memory will carefully coordinate the surfacing of additional capacity with the creation of DAX devices which use that capacity. Therefore, the allocation of the memory to DAX devices does not allow for specific associations between DAX device and extent. This keeps allocations very similar to existing DAX region behavior.

To keep the DAX memory allocation aligned with the existing DAX devices which do not have tags extents are not allowed to have tags. Future support for tags is planned.

Great care was taken to keep the extent tracking simple. Some xarray's needed to be added but extra software objects were kept to a minimum.

Region extents continue to be tracked as sub-devices of the DAX region. This ensures that region destruction cleans up all extent allocations properly.

Some review tags were kept if a patch did not change.

The major functionality of this series includes:

- Getting the dynamic capacity (DC) configuration information from cxl devices

- Configuring the DC partitions reported by hardware

- Enhancing the CXL and DAX regions for dynamic capacity support a. Maintain a logical separation between hardware extents and software managed region extents. This provides an abstraction between the layers and should allow for interleaving in the future

- Get hardware extent lists for endpoint decoders upon region creation.

- Adjust extent/region memory available on the following events. a. Add capacity Events
b. Release capacity events

- Host response for add capacity
a. do not accept the extent if:
If the region does not exist
or an error occurs realizing the extent
b. If the region does exist
realize a DAX region extent with 1:1 mapping (no interleave yet)
c. Support the event more bit by processing a list of extents
marked with the more bit together before setting up a response.

- Host response for remove capacity
a. If no DAX device references the extent; release the extent b. If a reference does exist, ignore the request.
(Require FM to issue release again.)

- Modify DAX device creation/resize to account for extents within a sparse DAX region

- Trace Dynamic Capacity events for debugging

- Add cxl-test infrastructure to allow for faster unit testing (See new ndctl branch for cxl-dcd.sh test[1])

- Only support 0 value extent tags

Fan Ni's upstream of Qemu DCD was used for testing.

Remaining work:

1) Allow mapping to specific extents (perhaps based on label/tag)
1a) devise region size reporting based on tags
2) Interleave support

Possible additional work depending on requirements:

1) Accept a new extent which extends (but overlaps) an existing extent(s)
2) Release extents when DAX devices are released if a release was previously seen from the device
3) Rework DAX device interfaces, memfd has been explored a bit

[1] https://github.com/weiny2/ndctl/tree/dcd-region2-2024-10-01

---
Major changes in v4:
- iweiny: rebase to 6.12-rc
- iweiny: Add qos data to regions
- Jonathan: Fix up shared region detection
- Jonathan/jgroves/djbw/iweiny: Ignore 0 value tags
- iweiny: Change DCD partition sysfs entries to allow for qos class and additional parameters per partition
- Petr/Andy: s/%par/%pra/
- Andy: Share logic between printing struct resource and struct range
- Link to v3: https://patch.msgid.link/[email protected]

---
Ira Weiny (14):
test printk: Add very basic struct resource tests
printk: Add print format (%pra) for struct range cxl/cdat: Use %pra for dpa range outputs
range: Add range_overlaps()
dax: Document dax dev range tuple
cxl/pci: Delay event buffer allocation
cxl/cdat: Gather DSMAS data for DCD regions cxl/region: Refactor common create region code cxl/events: Split event msgnum configuration from irq setup cxl/pci: Factor out interrupt policy check
cxl/core: Return endpoint decoder information from region search dax/bus: Factor out dev dax resize logic tools/testing/cxl: Make event logs dynamic tools/testing/cxl: Add DC Regions to mock mem data

Navneet Singh (14):
cxl/mbox: Flag support for Dynamic Capacity Devices (DCD) cxl/mem: Read dynamic capacity configuration from the device cxl/core: Separate region mode from decoder mode cxl/region: Add dynamic capacity decoder and region modes cxl/hdm: Add dynamic capacity size support to endpoint decoders cxl/mem: Expose DCD partition capabilities in sysfs cxl/port: Add endpoint decoder DC mode support to sysfs cxl/region: Add sparse DAX region support
cxl/mem: Configure dynamic capacity interrupts cxl/extent: Process DCD events and realize region extents cxl/region/extent: Expose region extent information in sysfs dax/region: Create resources on sparse DAX regions cxl/region: Read existing extents on region creation cxl/mem: Trace Dynamic capacity Event Record

 Documentation/ABI/testing/sysfs-bus-cxl   | 120 +++-
 Documentation/core-api/printk-formats.rst |  13 +
 drivers/cxl/core/Makefile                 |   2 +-
 drivers/cxl/core/cdat.c                   |  52 +-
 drivers/cxl/core/core.h                   |  33 +-
 drivers/cxl/core/extent.c                 | 486 +++++++++++++++
 drivers/cxl/core/hdm.c                    | 213 ++++++-
 drivers/cxl/core/mbox.c                   | 605 ++++++++++++++++++-
 drivers/cxl/core/memdev.c                 | 130 +++-
 drivers/cxl/core/port.c                   |  13 +-
 drivers/cxl/core/region.c                 | 170 ++++--
 drivers/cxl/core/trace.h                  |  65 ++
 drivers/cxl/cxl.h                         | 122 +++-
 drivers/cxl/cxlmem.h                      | 131 +++-
 drivers/cxl/pci.c                         | 123 +++-
 drivers/dax/bus.c                         | 352 +++++++++--
 drivers/dax/bus.h                         |   4 +-
 drivers/dax/cxl.c                         |  72 ++-
 drivers/dax/dax-private.h                 |  47 +-
 drivers/dax/hmem/hmem.c                   |   2 +-
 drivers/dax/pmem.c                        |   2 +-
 fs/btrfs/ordered-data.c                   |  10 +-
 include/acpi/actbl1.h                     |   2 +
 include/cxl/event.h                       |  32 +
 include/linux/range.h                     |   7 +
 lib/test_printf.c                         |  70 +++
 lib/vsprintf.c                            |  55 +-
 tools/testing/cxl/Kbuild                  |   3 +-
 tools/testing/cxl/test/mem.c              | 960 ++++++++++++++++++++++++++----
 29 files changed, 3576 insertions(+), 320 deletions(-)

---
base-commit: 9852d85ec9d492ebef56dc5f229416c925758edc
change-id: 20230604-dcd-type2-upstream-0cd15f6216fd

Best regards,

Signature · 1 line
Ira Weiny <[email protected]>
author
#1 lore raw

PATCH v4 1/28 test printk: Add very basic struct resource tests

The printk tests for struct resource were stubbed out. struct range printing will leverage the struct resource implementation.

To prevent regression add some basic sanity tests for struct resource.

To: Petr Mladek <[email protected]>
To: Steven Rostedt <[email protected]>
To: Andy Shevchenko <[email protected]>
To: Rasmus Villemoes <[email protected]>
To: Sergey Senozhatsky <[email protected]>

---
[lkp: ensure phys_addr_t is within limits for all arch's]

---
 lib/test_printf.c | 44 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 44 insertions(+)
lib/test_printf.c +44 -0
index 8448b6d02bd9..5afdf5efc627 100644
@@ -386,6 +386,50 @@ kernel_ptr(void)
 static void __init
 struct_resource(void)
 {
+	struct resource test_resource = {
+		.start = 0xc0ffee00,
+		.end = 0xc0ffee00,
+		.flags = IORESOURCE_MEM,
+	};
+
+	test("[mem 0xc0ffee00 flags 0x200]",
+	     "%pr", &test_resource);
+
+	test_resource = (struct resource) {
+		.start = 0xc0ffee,
+		.end = 0xba5eba11,
+		.flags = IORESOURCE_MEM,
+	};
+	test("[mem 0x00c0ffee-0xba5eba11 flags 0x200]",
+	     "%pr", &test_resource);
+
+	test_resource = (struct resource) {
+		.start = 0xba5eba11,
+		.end = 0xc0ffee,
+		.flags = IORESOURCE_MEM,
+	};
+	test("[mem 0xba5eba11-0x00c0ffee flags 0x200]",
+	     "%pr", &test_resource);
+
+	test_resource = (struct resource) {
+		.start = 0xba5eba11,
+		.end = 0xba5eca11,
+		.flags = IORESOURCE_MEM,
+	};
+
+	test("[mem 0xba5eba11-0xba5eca11 flags 0x200]",
+	     "%pr", &test_resource);
+
+	test_resource = (struct resource) {
+		.start = 0xba11,
+		.end = 0xca10,
+		.flags = IORESOURCE_IO |
+			 IORESOURCE_DISABLED |
+			 IORESOURCE_UNSET,
+	};
+
+	test("[io  size 0x1000 disabled]",
+	     "%pR", &test_resource);
 }
 
 static void __init
Signature · 1 line
2.46.0
reviewer Reviewed-by on v4 1/28
#2 lore raw

Re: [PATCH v4 01/28] test printk: Add very basic struct resource tests

Quoted 4 lines · Ira Weiny

The printk tests for struct resource were stubbed out. struct range printing will leverage the struct resource implementation.

To prevent regression add some basic sanity tests for struct resource.

Good we start having them!

Signature · 2 lines
With Best Regards,
Andy Shevchenko
Reviewed-by on v4 1/28
#3 lore raw

Re: [PATCH v4 01/28] test printk: Add very basic struct resource tests

In reply to Ira Weiny
Quoted 5 lines · On Mon, 07 Oct 2024 18:16:07 -0500 Ira Weiny <[email protected]> wrote:

The printk tests for struct resource were stubbed out. struct range printing will leverage the struct resource implementation.

To prevent regression add some basic sanity tests for struct resource.

Quoted 8 lines

To: Petr Mladek <[email protected]>
To: Steven Rostedt <[email protected]>
To: Andy Shevchenko <[email protected]>
To: Rasmus Villemoes <[email protected]>
To: Sergey Senozhatsky <[email protected]>

Looks sane to me.

Reviewed-by Tested-by on v4 1/28
#4 lore raw

Re: [PATCH v4 01/28] test printk: Add very basic struct resource tests

In reply to Ira Weiny
Quoted 5 lines · Ira Weiny

The printk tests for struct resource were stubbed out. struct range printing will leverage the struct resource implementation.

To prevent regression add some basic sanity tests for struct resource.

Quoted 7 lines

To: Petr Mladek <[email protected]>
To: Steven Rostedt <[email protected]>
To: Andy Shevchenko <[email protected]>
To: Rasmus Villemoes <[email protected]>
To: Sergey Senozhatsky <[email protected]>

Quoted 66 lines

---
[lkp: ensure phys_addr_t is within limits for all arch's]

---
 lib/test_printf.c | 44 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 44 insertions(+)
lib/test_printf.c +44 -0
index 8448b6d02bd9..5afdf5efc627 100644
@@ -386,6 +386,50 @@ kernel_ptr(void)
 static void __init
 struct_resource(void)
 {
+	struct resource test_resource = {
+		.start = 0xc0ffee00,
+		.end = 0xc0ffee00,
+		.flags = IORESOURCE_MEM,
+	};
+
+	test("[mem 0xc0ffee00 flags 0x200]",
+	     "%pr", &test_resource);
+
+	test_resource = (struct resource) {
+		.start = 0xc0ffee,
+		.end = 0xba5eba11,
+		.flags = IORESOURCE_MEM,
+	};
+	test("[mem 0x00c0ffee-0xba5eba11 flags 0x200]",
+	     "%pr", &test_resource);
+
+	test_resource = (struct resource) {
+		.start = 0xba5eba11,
+		.end = 0xc0ffee,
+		.flags = IORESOURCE_MEM,
+	};
+	test("[mem 0xba5eba11-0x00c0ffee flags 0x200]",
+	     "%pr", &test_resource);
+
+	test_resource = (struct resource) {
+		.start = 0xba5eba11,
+		.end = 0xba5eca11,
+		.flags = IORESOURCE_MEM,
+	};
+
+	test("[mem 0xba5eba11-0xba5eca11 flags 0x200]",
+	     "%pr", &test_resource);
+
+	test_resource = (struct resource) {
+		.start = 0xba11,
+		.end = 0xca10,
+		.flags = IORESOURCE_IO |
+			 IORESOURCE_DISABLED |
+			 IORESOURCE_UNSET,
+	};
+
+	test("[io  size 0x1000 disabled]",
+	     "%pR", &test_resource);
 }
 
 static void __init
Signature · 2 lines
-- 
2.46.0
Signature · 1 line
Fan Ni
maintainer Acked-by on v4 1/28
#5 lore raw

Re: [PATCH v4 01/28] test printk: Add very basic struct resource tests

In reply to Ira Weiny
Quoted 5 lines · Ira Weiny

The printk tests for struct resource were stubbed out. struct range printing will leverage the struct resource implementation.

To prevent regression add some basic sanity tests for struct resource.

Quoted 7 lines

To: Petr Mladek <[email protected]>
To: Steven Rostedt <[email protected]>
To: Andy Shevchenko <[email protected]>
To: Rasmus Villemoes <[email protected]>
To: Sergey Senozhatsky <[email protected]>

Thanks for adding them. They look good:

Best Regards,
Petr

author
#6 lore raw

Re: [PATCH v4 01/28] test printk: Add very basic struct resource tests

Quoted 14 lines · Petr Mladek
Quoted 5 lines · Ira Weiny

The printk tests for struct resource were stubbed out. struct range printing will leverage the struct resource implementation.

To prevent regression add some basic sanity tests for struct resource.

Quoted 7 lines

To: Petr Mladek <[email protected]>
To: Steven Rostedt <[email protected]>
To: Andy Shevchenko <[email protected]>
To: Rasmus Villemoes <[email protected]>
To: Sergey Senozhatsky <[email protected]>

Quoted 3 lines

Thanks for adding them. They look good:

Thanks I've queued them in cxl-next even if this series is not ready by then.

Ira

author
#7 lore raw

PATCH v4 2/28 printk: Add print format (%pra) for struct range

In reply to Ira Weiny

The use of struct range in the CXL subsystem is growing. In particular, the addition of Dynamic Capacity devices uses struct range in a number of places which are reported in debug and error messages.

To wit requiring the printing of the start/end fields in each print became cumbersome. Dan Williams mentions in [1] that it might be time to have a print specifier for struct range similar to struct resource

A few alternatives were considered including '%par', '%r', and '%pn'. %pra follows that struct range is similar to struct resource (%p[rR]) but need to be different. Based on discussions with Petr and Andy '%pra' was chosen.[2]

Andy also suggested to keep the range prints similar to struct resource though combined code. Add hex_range() to handle printing for both pointer types.

To: Petr Mladek <[email protected]>
To: Steven Rostedt <[email protected]>
To: Andy Shevchenko <[email protected]>
To: Rasmus Villemoes <[email protected]>
To: Sergey Senozhatsky <[email protected]>
To: Jonathan Corbet <[email protected]> (maintainer:DOCUMENTATION)

---
Changes:
[Andy: create new hex_range() and use it in both range/resource] [Petr/Andy: Use %pra]
[Andy: Add test case start > end]
[Petr: Update documentation]
[Petr: use 'range -']
[Petr: fixup printf_spec specifiers]
[Petr: add lib/test_printf test]

---
 Documentation/core-api/printk-formats.rst | 13 ++++++++
 lib/test_printf.c                         | 26 +++++++++++++++
 lib/vsprintf.c                            | 55 +++++++++++++++++++++++++++----
 3 files changed, 88 insertions(+), 6 deletions(-)
Documentation/core-api/printk-formats.rst +13 -0
index 14e093da3ccd..03b102fc60bb 100644
@@ -231,6 +231,19 @@ width of the CPU data path.
 
 Passed by reference.
 
+Struct Range
+------------
+
+::
+
+	%pra    [range 0x0000000060000000-0x000000006fffffff]
+	%pra    [range 0x0000000060000000]
+
+For printing struct range.  struct range holds an arbitrary range of u64
+values.  If start is equal to end only 1 value is printed.
+
+Passed by reference.
+
 DMA address types dma_addr_t
 ----------------------------
 
lib/test_printf.c +26 -0
index 5afdf5efc627..e3e75b6d10a0 100644
@@ -432,6 +432,31 @@ struct_resource(void)
 	     "%pR", &test_resource);
 }
 
+static void __init
+struct_range(void)
+{
+	struct range test_range = {
+		.start = 0xc0ffee00ba5eba11,
+		.end = 0xc0ffee00ba5eba11,
+	};
+
+	test("[range 0xc0ffee00ba5eba11]", "%pra", &test_range);
+
+	test_range = (struct range) {
+		.start = 0xc0ffee,
+		.end = 0xba5eba11,
+	};
+	test("[range 0x0000000000c0ffee-0x00000000ba5eba11]",
+	     "%pra", &test_range);
+
+	test_range = (struct range) {
+		.start = 0xba5eba11,
+		.end = 0xc0ffee,
+	};
+	test("[range 0x00000000ba5eba11-0x0000000000c0ffee]",
+	     "%pra", &test_range);
+}
+
 static void __init
 addr(void)
 {
@@ -807,6 +832,7 @@ test_pointer(void)
 	symbol_ptr();
 	kernel_ptr();
 	struct_resource();
+	struct_range();
 	addr();
 	escaped_str();
 	hex_string();
lib/vsprintf.c +49 -6

86 lines not on this page · raw message

Signature · 1 line
2.46.0
#8 lore raw

Re: [PATCH v4 02/28] printk: Add print format (%pra) for struct range

Quoted 16 lines · Ira Weiny

The use of struct range in the CXL subsystem is growing. In particular, the addition of Dynamic Capacity devices uses struct range in a number of places which are reported in debug and error messages.

To wit requiring the printing of the start/end fields in each print became cumbersome. Dan Williams mentions in [1] that it might be time to have a print specifier for struct range similar to struct resource

A few alternatives were considered including '%par', '%r', and '%pn'. %pra follows that struct range is similar to struct resource (%p[rR]) but need to be different. Based on discussions with Petr and Andy '%pra' was chosen.[2]

Andy also suggested to keep the range prints similar to struct resource though combined code. Add hex_range() to handle printing for both pointer types.

...

lib/test_printf.c struct_resource(void) · line 441

Quoted 7 lines
lib/test_printf.c +7 -0
+static void __init
+struct_range(void)
+{
+	struct range test_range = {
+		.start = 0xc0ffee00ba5eba11,
+		.end = 0xc0ffee00ba5eba11,
+	};

A side note, can we add something like

#define DEFINE_RANGE(start, end)	\
	(struct range) {		\
		.start = (start),	\
		.end = (end),		\
	}

in range.h and use here and in the similar cases?

Quoted 16 lines
lib/test_printf.c +16 -0
+	test("[range 0xc0ffee00ba5eba11]", "%pra", &test_range);
+
+	test_range = (struct range) {
+		.start = 0xc0ffee,
+		.end = 0xba5eba11,
+	};
+	test("[range 0x0000000000c0ffee-0x00000000ba5eba11]",
+	     "%pra", &test_range);
+
+	test_range = (struct range) {
+		.start = 0xba5eba11,
+		.end = 0xc0ffee,
+	};
+	test("[range 0x00000000ba5eba11-0x0000000000c0ffee]",
+	     "%pra", &test_range);
+}

...

lib/vsprintf.c static const struct printf_spec default_dec04_spec = { · line 1053

Quoted 11 lines
lib/vsprintf.c +11 -0
+char *hex_range(char *buf, char *end, u64 start_val, u64 end_val,
+		struct printf_spec spec)
+{
+	buf = number(buf, end, start_val, spec);
+	if (start_val != end_val) {
+		if (buf < end)
+			*buf++ = '-';
+		buf = number(buf, end, end_val, spec);
+	}
+	return buf;
+}

Perhaps

	buf = number(buf, end, start_val, spec);
	if (start_val == end_val)
		return buf;

	if (buf < end)
		*buf++ = '-';
	return number(buf, end, end_val, spec);

(yes, I have seen the original code)?

Quoted 27 lines
lib/vsprintf.c +27 -0
+static noinline_for_stack
+char *range_string(char *buf, char *end, const struct range *range,
+		   struct printf_spec spec, const char *fmt)
+{
+#define RANGE_DECODED_BUF_SIZE		((2 * sizeof(struct range)) + 4)
+#define RANGE_PRINT_BUF_SIZE		sizeof("[range -]")
+	char sym[RANGE_DECODED_BUF_SIZE + RANGE_PRINT_BUF_SIZE];
+	char *p = sym, *pend = sym + sizeof(sym);
+
+	struct printf_spec range_spec = {
+		.field_width = 2 + 2 * sizeof(range->start), /* 0x + 2 * 8 */
+		.flags = SPECIAL | SMALL | ZEROPAD,
+		.base = 16,
+		.precision = -1,
+	};
+
+	if (check_pointer(&buf, end, range, spec))
+		return buf;
+
+	*p++ = '[';
+	p = string_nocheck(p, pend, "range ", default_str_spec);
+	p = hex_range(p, pend, range->start, range->end, range_spec);
+	*p++ = ']';
+	*p = '\0';
+
+	return string_nocheck(buf, end, sym, spec);
+}

...

Quoted 1 line

+ * - 'ra' struct ranges [range 0x00 - 0xff]

Is it possible to get only bytes out of this? I thought we have always 64-bit values here, no?

...

lib/vsprintf.c char *pointer(const char *fmt, char *buf, char *end, void *ptr, · line 2447

Quoted 11 lines
lib/vsprintf.c +6 -1
 	case 'B':
 		return symbol_string(buf, end, ptr, spec, fmt);
-	case 'R':
 	case 'r':
+		switch (fmt[1]) {
+		case 'a':
+			return range_string(buf, end, ptr, spec, fmt);
+		}
+		fallthrough;
+	case 'R':
 		return resource_string(buf, end, ptr, spec, fmt);

Do we have default-less switches in the code (in this file)?

Actually I would suggest to move this to a wrapper like time_and_date().

Signature · 2 lines
With Best Regards,
Andy Shevchenko
#9 lore raw

Re: [PATCH v4 02/28] printk: Add print format (%pra) for struct range

Quoted 30 lines · On Tue, 8 Oct 2024 19:56:20 +0300 Andy Shevchenko…
Quoted 16 lines · Ira Weiny

The use of struct range in the CXL subsystem is growing. In particular, the addition of Dynamic Capacity devices uses struct range in a number of places which are reported in debug and error messages.

To wit requiring the printing of the start/end fields in each print became cumbersome. Dan Williams mentions in [1] that it might be time to have a print specifier for struct range similar to struct resource

A few alternatives were considered including '%par', '%r', and '%pn'. %pra follows that struct range is similar to struct resource (%p[rR]) but need to be different. Based on discussions with Petr and Andy '%pra' was chosen.[2]

Andy also suggested to keep the range prints similar to struct resource though combined code. Add hex_range() to handle printing for both pointer types.

...

lib/test_printf.c struct_resource(void) · line 441

Quoted 7 lines
lib/test_printf.c +7 -0
+static void __init
+struct_range(void)
+{
+	struct range test_range = {
+		.start = 0xc0ffee00ba5eba11,
+		.end = 0xc0ffee00ba5eba11,
+	};  

A side note, can we add something like

Quoted 7 lines
#define DEFINE_RANGE(start, end)	\
	(struct range) {		\
		.start = (start),	\
		.end = (end),		\
	}

in range.h and use here and in the similar cases?

DEFINE_XXXX at least sometimes is used in cases that create the variable as well. E.g. DEFINE_MUTEX()

INIT_RANGE() maybe?

#10 lore raw

Re: [PATCH v4 02/28] printk: Add print format (%pra) for struct range

Quoted 3 lines · Jonathan Cameron
Quoted 1 line · On Tue, 8 Oct 2024 19:56:20 +0300 Andy Shevchenko…

On Mon, Oct 07, 2024 at 06:16:08PM -0500, Ira Weiny wrote:

...

Quoted 16 lines
Quoted 16 lines

lib/test_printf.c struct_resource(void) · line 441

Quoted 7 lines
lib/test_printf.c +7 -0
+static void __init
+struct_range(void)
+{
+	struct range test_range = {
+		.start = 0xc0ffee00ba5eba11,
+		.end = 0xc0ffee00ba5eba11,
+	};  

A side note, can we add something like

#define DEFINE_RANGE(start, end)	\
	(struct range) {		\
		.start = (start),	\
		.end = (end),		\
	}
Quoted 4 lines
Quoted 1 line

in range.h and use here and in the similar cases?

DEFINE_XXXX at least sometimes is used in cases that create the variable as well. E.g. DEFINE_MUTEX()

I understand your point, but since there are many similarities to struct resource, I would stick with naming convention in ioport.h.

Quoted 1 line

INIT_RANGE() maybe?

Signature · 2 lines
With Best Regards,
Andy Shevchenko
#11 lore raw

Re: [PATCH v4 02/28] printk: Add print format (%pra) for struct range

In reply to Ira Weiny
Quoted 11 lines · Ira Weiny
---
 Documentation/core-api/printk-formats.rst | 13 ++++++++
 lib/test_printf.c                         | 26 +++++++++++++++
 lib/vsprintf.c                            | 55 +++++++++++++++++++++++++++----
 3 files changed, 88 insertions(+), 6 deletions(-)
Documentation/core-api/printk-formats.rst +0 -0
index 14e093da3ccd..03b102fc60bb 100644

Documentation/core-api/printk-formats.rst width of the CPU data path. · line 235

Quoted 5 lines
Documentation/core-api/printk-formats.rst +2 -0
@@ -231,6 +231,19 @@ width of the CPU data path.
 
 Passed by reference.
 
+Struct Range
+------------

Probably neither of those words should be capitalized.

Quoted 37 lines
Documentation/core-api/printk-formats.rst +11 -0
+
+::
+
+	%pra    [range 0x0000000060000000-0x000000006fffffff]
+	%pra    [range 0x0000000060000000]
+
+For printing struct range.  struct range holds an arbitrary range of u64
+values.  If start is equal to end only 1 value is printed.
+
+Passed by reference.
+
 DMA address types dma_addr_t
 ----------------------------
 
lib/test_printf.c +15 -0
index 5afdf5efc627..e3e75b6d10a0 100644
@@ -432,6 +432,31 @@ struct_resource(void)
 	     "%pR", &test_resource);
 }
 
+static void __init
+struct_range(void)
+{
+	struct range test_range = {
+		.start = 0xc0ffee00ba5eba11,
+		.end = 0xc0ffee00ba5eba11,
+	};
+
+	test("[range 0xc0ffee00ba5eba11]", "%pra", &test_range);
+
+	test_range = (struct range) {
+		.start = 0xc0ffee,
+		.end = 0xba5eba11,
+	};
+	test("[range 0x0000000000c0ffee-0x00000000ba5eba11]",

lib/test_printf.c struct_resource(void) · line 459

Quoted 10 lines
lib/test_printf.c +10 -0
+	     "%pra", &test_range);
+
+	test_range = (struct range) {
+		.start = 0xba5eba11,
+		.end = 0xc0ffee,
+	};
+	test("[range 0x00000000ba5eba11-0x0000000000c0ffee]",
+	     "%pra", &test_range);
+}
+

Thanks for including tests!

Rather than the struct assignments, I think it's easier to read if you just do

struct range r;

  r.start = 0xc0ffee00ba5eba11;
  r.end   = r.start;
  ...

  r.start = 0xc0ffee;
  r.end   = 0xba5eba11;
  ...

which saves two lines per test and for the first one makes it more obvious that the start and end values are identical.

Quoted 17 lines
lib/test_printf.c +1 -0
 static void __init
 addr(void)
 {
@@ -807,6 +832,7 @@ test_pointer(void)
 	symbol_ptr();
 	kernel_ptr();
 	struct_resource();
+	struct_range();
 	addr();
 	escaped_str();
 	hex_string();
lib/vsprintf.c +0 -0
index 09f022ba1c05..f8f5ed8f4d39 100644
@@ -1039,6 +1039,19 @@ static const struct printf_spec default_dec04_spec = {
 	.flags = ZEROPAD,

lib/vsprintf.c static const struct printf_spec default_dec04_spec = { · line 1049

Quoted 10 lines
lib/vsprintf.c +8 -0
 };
 
+static noinline_for_stack
+char *hex_range(char *buf, char *end, u64 start_val, u64 end_val,
+		struct printf_spec spec)
+{
+	buf = number(buf, end, start_val, spec);
+	if (start_val != end_val) {
+		if (buf < end)
+			*buf++ = '-';

No. Either all your callers pass a (probably stack-allocated) buffer which is guaranteed to be big enough, in which case you don't need the "if (buf < end)", or if some callers may "print" directly to the buffer passed to vsnprintf(), the buf++ must still be done unconditionally in order that vsnprintf(NULL, 0, ...) [used by fx kasprintf] can accurately determine how large the output string would be.

So, either

*buf++ = '-'

or

  if (buf < end)
    *buf = '-';
  buf++;

Please don't mix the two.

Quoted 22 lines
lib/vsprintf.c +6 -5
+		buf = number(buf, end, end_val, spec);
+	}
+	return buf;
+}
+
 static noinline_for_stack
 char *resource_string(char *buf, char *end, struct resource *res,
 		      struct printf_spec spec, const char *fmt)
@@ -1115,11 +1128,7 @@ char *resource_string(char *buf, char *end, struct resource *res,
 		p = string_nocheck(p, pend, "size ", str_spec);
 		p = number(p, pend, resource_size(res), *specp);
 	} else {
-		p = number(p, pend, res->start, *specp);
-		if (res->start != res->end) {
-			*p++ = '-';
-			p = number(p, pend, res->end, *specp);
-		}
+		p = hex_range(p, pend, res->start, res->end, *specp);
 	}
 	if (decode) {
 		if (res->flags & IORESOURCE_MEM_64)

lib/vsprintf.c char *resource_string(char *buf, char *end, struct resource *res, · line 1158

Quoted 10 lines
lib/vsprintf.c +7 -0
@@ -1140,6 +1149,34 @@ char *resource_string(char *buf, char *end, struct resource *res,
 	return string_nocheck(buf, end, sym, spec);
 }
 
+static noinline_for_stack
+char *range_string(char *buf, char *end, const struct range *range,
+		   struct printf_spec spec, const char *fmt)
+{
+#define RANGE_DECODED_BUF_SIZE		((2 * sizeof(struct range)) + 4)
+#define RANGE_PRINT_BUF_SIZE		sizeof("[range -]")
+	char sym[RANGE_DECODED_BUF_SIZE + RANGE_PRINT_BUF_SIZE];

I don't think these names or the split in two constants helps convincing that's the right amount. I have to think quite a bit to see that 2*sizeof is because struct range has two u64 and we're printing in hex so four-bits-per-char and probably the +4 are for two time "0x".

Why not just size the buffer directly using an "example" string?

char sym[sizeof("[range 0x0123456789abcdef-0x0123456789abcdef]")]

Quoted 4 lines
lib/vsprintf.c +4 -0
+	char *p = sym, *pend = sym + sizeof(sym);
+
+	struct printf_spec range_spec = {
+		.field_width = 2 + 2 * sizeof(range->start), /* 0x + 2 * 8 */

lib/vsprintf.c char *resource_string(char *buf, char *end, struct resource *res, · line 1172

Quoted 10 lines
lib/vsprintf.c +10 -0
+		.flags = SPECIAL | SMALL | ZEROPAD,
+		.base = 16,
+		.precision = -1,
+	};
+
+	if (check_pointer(&buf, end, range, spec))
+		return buf;
+
+	*p++ = '[';
+	p = string_nocheck(p, pend, "range ", default_str_spec);

We really should have mempcpy or stpcpy. I don't see the point of using string_nocheck here, or not including the [ in the string copy (however it's done). But yeah, without stpcpy() that's a bit awkward.

Rasmus

#12 lore raw

Re: [PATCH v4 02/28] printk: Add print format (%pra) for struct range

On Wed, Oct 09, 2024 at 03:30:14PM +0200, Rasmus Villemoes wrote:

...

Quoted 13 lines

Rather than the struct assignments, I think it's easier to read if you just do

struct range r;

  r.start = 0xc0ffee00ba5eba11;
  r.end   = r.start;
  ...

  r.start = 0xc0ffee;
  r.end   = 0xba5eba11;
  ...
Quoted 2 lines

which saves two lines per test and for the first one makes it more obvious that the start and end values are identical.

With DEFINE_RANGE() it will save even more lines!

..

Quoted 9 lines

lib/vsprintf.c static const struct printf_spec default_dec04_spec = { · line 1049

Quoted 2 lines
lib/vsprintf.c +2 -0
+		if (buf < end)
+			*buf++ = '-';

No. Either all your callers pass a (probably stack-allocated) buffer which is guaranteed to be big enough, in which case you don't need the "if (buf < end)", or if some callers may "print" directly to the buffer passed to vsnprintf(), the buf++ must still be done unconditionally in order that vsnprintf(NULL, 0, ...) [used by fx kasprintf] can accurately determine how large the output string would be.

Ah, good catch, I would add...

Quoted 9 lines

So, either

*buf++ = '-'

or

  if (buf < end)
    *buf = '-';
  buf++;

...that we use rather ++buf in such cases, but it doesn't really matter.

Quoted 1 line

Please don't mix the two.

Signature · 2 lines
With Best Regards,
Andy Shevchenko
author
#13 lore raw

Re: [PATCH v4 02/28] printk: Add print format (%pra) for struct range

Quoted 13 lines · Andy Shevchenko

On Wed, Oct 09, 2024 at 03:30:14PM +0200, Rasmus Villemoes wrote:

...

Quoted 9 lines

Rather than the struct assignments, I think it's easier to read if you just do

struct range r;

  r.start = 0xc0ffee00ba5eba11;
  r.end   = r.start;
  ...
Quoted 8 lines
Quoted 6 lines
  r.start = 0xc0ffee;
  r.end   = 0xba5eba11;
  ...

which saves two lines per test and for the first one makes it more obvious that the start and end values are identical.

With DEFINE_RANGE() it will save even more lines!

Yea I've added DEFINE_RANGE(). Thanks.

Quoted 19 lines

..

Quoted 9 lines

lib/vsprintf.c static const struct printf_spec default_dec04_spec = { · line 1049

Quoted 2 lines
lib/vsprintf.c +2 -0
+		if (buf < end)
+			*buf++ = '-';

No. Either all your callers pass a (probably stack-allocated) buffer which is guaranteed to be big enough, in which case you don't need the "if (buf < end)", or if some callers may "print" directly to the buffer passed to vsnprintf(), the buf++ must still be done unconditionally in order that vsnprintf(NULL, 0, ...) [used by fx kasprintf] can accurately determine how large the output string would be.

Ah, good catch, I would add...

Quoted 4 lines

So, either

*buf++ = '-'

Quoted 7 lines
Quoted 5 lines

or

  if (buf < end)
    *buf = '-';
  buf++;

...that we use rather ++buf in such cases, but it doesn't really matter.

Done.
Ira

author
#14 lore raw

Re: [PATCH v4 02/28] printk: Add print format (%pra) for struct range

In reply to Rasmus Villemoes
Quoted 14 lines · Rasmus Villemoes
Quoted 12 lines · Ira Weiny
---
 Documentation/core-api/printk-formats.rst | 13 ++++++++
 lib/test_printf.c                         | 26 +++++++++++++++
 lib/vsprintf.c                            | 55 +++++++++++++++++++++++++++----
 3 files changed, 88 insertions(+), 6 deletions(-)
Documentation/core-api/printk-formats.rst +0 -0
index 14e093da3ccd..03b102fc60bb 100644
@@ -231,6 +231,19 @@ width of the CPU data path.
 
Quoted 6 lines

Documentation/core-api/printk-formats.rst width of the CPU data path. · line 235

Quoted 4 lines
Documentation/core-api/printk-formats.rst +2 -0
 Passed by reference.
 
+Struct Range
+------------

Probably neither of those words should be capitalized.

I was following the format of the header of struct resource

	Struct Resources
	----------------

I can change it but I was trying to be consistent here.

[snip]

Quoted 28 lines
Quoted 15 lines
Documentation/core-api/printk-formats.rst +15 -0
+static void __init
+struct_range(void)
+{
+	struct range test_range = {
+		.start = 0xc0ffee00ba5eba11,
+		.end = 0xc0ffee00ba5eba11,
+	};
+
+	test("[range 0xc0ffee00ba5eba11]", "%pra", &test_range);
+
+	test_range = (struct range) {
+		.start = 0xc0ffee,
+		.end = 0xba5eba11,
+	};
+	test("[range 0x0000000000c0ffee-0x00000000ba5eba11]",

Documentation/core-api/printk-formats.rst width of the CPU data path.

Quoted 10 lines
Documentation/core-api/printk-formats.rst +10 -0
+	     "%pra", &test_range);
+
+	test_range = (struct range) {
+		.start = 0xba5eba11,
+		.end = 0xc0ffee,
+	};
+	test("[range 0x00000000ba5eba11-0x0000000000c0ffee]",
+	     "%pra", &test_range);
+}
+

Thanks for including tests!

Quoted 2 lines

Rather than the struct assignments, I think it's easier to read if you just do

I'm using Andy's suggestion of DEFINE_RANGE()

Quoted 53 lines

struct range r;

  r.start = 0xc0ffee00ba5eba11;
  r.end   = r.start;
  ...

  r.start = 0xc0ffee;
  r.end   = 0xba5eba11;
  ...

which saves two lines per test and for the first one makes it more obvious that the start and end values are identical.

Quoted 17 lines

static void __init
addr(void)
{

Documentation/core-api/printk-formats.rst +1 -0
@@ -807,6 +832,7 @@ test_pointer(void)
 	symbol_ptr();
 	kernel_ptr();
 	struct_resource();
+	struct_range();
 	addr();
 	escaped_str();
 	hex_string();
lib/vsprintf.c +0 -0
index 09f022ba1c05..f8f5ed8f4d39 100644
@@ -1039,6 +1039,19 @@ static const struct printf_spec default_dec04_spec = {
 	.flags = ZEROPAD,

lib/vsprintf.c static const struct printf_spec default_dec04_spec = { · line 1049

Quoted 10 lines
lib/vsprintf.c +8 -0
 };
 
+static noinline_for_stack
+char *hex_range(char *buf, char *end, u64 start_val, u64 end_val,
+		struct printf_spec spec)
+{
+	buf = number(buf, end, start_val, spec);
+	if (start_val != end_val) {
+		if (buf < end)
+			*buf++ = '-';

No. Either all your callers pass a (probably stack-allocated) buffer which is guaranteed to be big enough, in which case you don't need the "if (buf < end)", or if some callers may "print" directly to the buffer passed to vsnprintf(), the buf++ must still be done unconditionally in order that vsnprintf(NULL, 0, ...) [used by fx kasprintf] can accurately determine how large the output string would be.

So, either

*buf++ = '-'

Quoted 7 lines

or

  if (buf < end)
    *buf = '-';
  buf++;

Please don't mix the two.

Ah ok yea fixed building on Andy's comment.

lib/vsprintf.c +2 -1
index a7b5e4618f6a..7aa47f7d9d5b 100644
@@ -1048,7 +1048,8 @@ char *hex_range(char *buf, char *end, u64 start_val, u64 end_val,
                return buf;
 
        if (buf < end)
-               *buf++ = '-';
+               *buf = '-';
+       ++buf;
        return number(buf, end, end_val, spec);
 }
 
Quoted 28 lines
Quoted 28 lines
lib/vsprintf.c +6 -5
+		buf = number(buf, end, end_val, spec);
+	}
+	return buf;
+}
+
 static noinline_for_stack
 char *resource_string(char *buf, char *end, struct resource *res,
 		      struct printf_spec spec, const char *fmt)
@@ -1115,11 +1128,7 @@ char *resource_string(char *buf, char *end, struct resource *res,
 		p = string_nocheck(p, pend, "size ", str_spec);
 		p = number(p, pend, resource_size(res), *specp);
 	} else {
-		p = number(p, pend, res->start, *specp);
-		if (res->start != res->end) {
-			*p++ = '-';
-			p = number(p, pend, res->end, *specp);
-		}
+		p = hex_range(p, pend, res->start, res->end, *specp);
 	}
 	if (decode) {
 		if (res->flags & IORESOURCE_MEM_64)
@@ -1140,6 +1149,34 @@ char *resource_string(char *buf, char *end, struct resource *res,
 	return string_nocheck(buf, end, sym, spec);
 }
 
Quoted 12 lines

lib/vsprintf.c char *resource_string(char *buf, char *end, struct resource *res, · line 1158

Quoted 7 lines
lib/vsprintf.c +7 -0
+static noinline_for_stack
+char *range_string(char *buf, char *end, const struct range *range,
+		   struct printf_spec spec, const char *fmt)
+{
+#define RANGE_DECODED_BUF_SIZE		((2 * sizeof(struct range)) + 4)
+#define RANGE_PRINT_BUF_SIZE		sizeof("[range -]")
+	char sym[RANGE_DECODED_BUF_SIZE + RANGE_PRINT_BUF_SIZE];

I don't think these names or the split in two constants helps convincing that's the right amount. I have to think quite a bit to see that 2*sizeof is because struct range has two u64 and we're printing in hex so four-bits-per-char and probably the +4 are for two time "0x".

Yea.

Quoted 4 lines

Why not just size the buffer directly using an "example" string?

char sym[sizeof("[range 0x0123456789abcdef-0x0123456789abcdef]")]

Ok that is simpler.

Quoted 11 lines
Quoted 11 lines
lib/vsprintf.c +10 -0
+	char *p = sym, *pend = sym + sizeof(sym);
+
+	struct printf_spec range_spec = {
+		.field_width = 2 + 2 * sizeof(range->start), /* 0x + 2 * 8 */
+		.flags = SPECIAL | SMALL | ZEROPAD,
+		.base = 16,
+		.precision = -1,
+	};
+
+	if (check_pointer(&buf, end, range, spec))
Quoted 8 lines

lib/vsprintf.c char *resource_string(char *buf, char *end, struct resource *res, · line 1172

Quoted 4 lines
lib/vsprintf.c +4 -0
+		return buf;
+
+	*p++ = '[';
+	p = string_nocheck(p, pend, "range ", default_str_spec);

We really should have mempcpy or stpcpy. I don't see the point of using string_nocheck here, or not including the [ in the string copy (however it's done). But yeah, without stpcpy() that's a bit awkward.

Added '[' to the string. The prevalent use of string_nocheck() seems reasonable to me but it is pretty heavyweight for this case.

Ira

#15 lore raw

Re: [PATCH v4 02/28] printk: Add print format (%pra) for struct range

In reply to Ira Weiny
Quoted 102 lines · Ira Weiny

The use of struct range in the CXL subsystem is growing. In particular, the addition of Dynamic Capacity devices uses struct range in a number of places which are reported in debug and error messages.

To wit requiring the printing of the start/end fields in each print became cumbersome. Dan Williams mentions in [1] that it might be time to have a print specifier for struct range similar to struct resource

A few alternatives were considered including '%par', '%r', and '%pn'. %pra follows that struct range is similar to struct resource (%p[rR]) but need to be different. Based on discussions with Petr and Andy '%pra' was chosen.[2]

Andy also suggested to keep the range prints similar to struct resource though combined code. Add hex_range() to handle printing for both pointer types.

To: Petr Mladek <[email protected]>
To: Steven Rostedt <[email protected]>
To: Andy Shevchenko <[email protected]>
To: Rasmus Villemoes <[email protected]>
To: Sergey Senozhatsky <[email protected]>
To: Jonathan Corbet <[email protected]> (maintainer:DOCUMENTATION)

---
Changes:
[Andy: create new hex_range() and use it in both range/resource] [Petr/Andy: Use %pra]
[Andy: Add test case start > end]
[Petr: Update documentation]
[Petr: use 'range -']
[Petr: fixup printf_spec specifiers]
[Petr: add lib/test_printf test]

---
 Documentation/core-api/printk-formats.rst | 13 ++++++++
 lib/test_printf.c                         | 26 +++++++++++++++
 lib/vsprintf.c                            | 55 +++++++++++++++++++++++++++----
 3 files changed, 88 insertions(+), 6 deletions(-)
Documentation/core-api/printk-formats.rst +13 -0
index 14e093da3ccd..03b102fc60bb 100644
@@ -231,6 +231,19 @@ width of the CPU data path.
 
 Passed by reference.
 
+Struct Range
+------------
+
+::
+
+	%pra    [range 0x0000000060000000-0x000000006fffffff]
+	%pra    [range 0x0000000060000000]
+
+For printing struct range.  struct range holds an arbitrary range of u64
+values.  If start is equal to end only 1 value is printed.
+
+Passed by reference.
+
 DMA address types dma_addr_t
 ----------------------------
 
lib/test_printf.c +25 -0
index 5afdf5efc627..e3e75b6d10a0 100644
@@ -432,6 +432,31 @@ struct_resource(void)
 	     "%pR", &test_resource);
 }
 
+static void __init
+struct_range(void)
+{
+	struct range test_range = {
+		.start = 0xc0ffee00ba5eba11,
+		.end = 0xc0ffee00ba5eba11,
+	};
+
+	test("[range 0xc0ffee00ba5eba11]", "%pra", &test_range);
+
+	test_range = (struct range) {
+		.start = 0xc0ffee,
+		.end = 0xba5eba11,
+	};
+	test("[range 0x0000000000c0ffee-0x00000000ba5eba11]",
+	     "%pra", &test_range);
+
+	test_range = (struct range) {
+		.start = 0xba5eba11,
+		.end = 0xc0ffee,
+	};
+	test("[range 0x00000000ba5eba11-0x0000000000c0ffee]",
+	     "%pra", &test_range);
+}
+

...

lib/test_printf.c char *rust_fmt_argument(char *buf, char *end, void *ptr); · line 2317

Quoted 8 lines

static noinline_for_stack
char *hex_string(char *buf, char *end, u8 *addr, struct printf_spec spec,

lib/test_printf.c +1 -0
 		 const char *fmt)
@@ -2277,6 +2314,7 @@ char *rust_fmt_argument(char *buf, char *end, void *ptr);
  * - 'Bb' as above with module build ID (for use in backtraces)
  * - 'R' For decoded struct resource, e.g., [mem 0x0-0x1f 64bit pref]
  * - 'r' For raw struct resource, e.g., [mem 0x0-0x1f flags 0x201]
+ * - 'ra' struct ranges [range 0x00 - 0xff]

Maybe follow the existing examples here, like
'ra" For struct ranges, e.g., ...

fan

Quoted 22 lines
lib/test_printf.c +6 -1
  * - 'b[l]' For a bitmap, the number of bits is determined by the field
  *       width which must be explicitly specified either as part of the
  *       format string '%32b[l]' or through '%*b[l]', [l] selects
@@ -2399,8 +2437,13 @@ char *pointer(const char *fmt, char *buf, char *end, void *ptr,
 		fallthrough;
 	case 'B':
 		return symbol_string(buf, end, ptr, spec, fmt);
-	case 'R':
 	case 'r':
+		switch (fmt[1]) {
+		case 'a':
+			return range_string(buf, end, ptr, spec, fmt);
+		}
+		fallthrough;
+	case 'R':
 		return resource_string(buf, end, ptr, spec, fmt);
 	case 'h':
 		return hex_string(buf, end, ptr, spec, fmt);
Signature · 2 lines
-- 
2.46.0
Signature · 1 line
Fan Ni
#16 lore raw

Re: [PATCH v4 02/28] printk: Add print format (%pra) for struct range

In reply to Ira Weiny

Documentation/core-api/printk-formats.rst width of the CPU data path. · line 243

Quoted 10 lines · Ira Weiny
Documentation/core-api/printk-formats.rst +10 -0
+Struct Range
+------------
+
+::
+
+	%pra    [range 0x0000000060000000-0x000000006fffffff]
+	%pra    [range 0x0000000060000000]
+
+For printing struct range.  struct range holds an arbitrary range of u64
+values.  If start is equal to end only 1 value is printed.

Do you mean printing only start value in start=equal case?

Confused...

Signature · 1 line
An old man doll... just what I always wanted! - Clara
author
#17 lore raw

Re: [PATCH v4 02/28] printk: Add print format (%pra) for struct range

Quoted 13 lines · Bagas Sanjaya

Review

Quoted 10 lines · Ira Weiny
(quoted diff) +10 -0
+Struct Range
+------------
+
+::
+
+	%pra    [range 0x0000000060000000-0x000000006fffffff]
+	%pra    [range 0x0000000060000000]
+
+For printing struct range.  struct range holds an arbitrary range of u64
+values.  If start is equal to end only 1 value is printed.

Do you mean printing only start value in start=equal case?

Yes I'll change the verbiage.

Ira

Documentation/core-api/printk-formats.rst +1 -1
index 03b102fc60bb..e1ebf0376154 100644
@@ -240,7 +240,7 @@ Struct Range
        %pra    [range 0x0000000060000000]
 
 For printing struct range.  struct range holds an arbitrary range of u64
-values.  If start is equal to end only 1 value is printed.
+values.  If start is equal to end only print the start value.
 
 Passed by reference.
#18 lore raw

Re: [PATCH v4 02/28] printk: Add print format (%pra) for struct range

Quoted 24 lines · Ira Weiny
Quoted 13 lines · Bagas Sanjaya

Review

Quoted 10 lines · Ira Weiny
(quoted diff) +10 -0
+Struct Range
+------------
+
+::
+
+	%pra    [range 0x0000000060000000-0x000000006fffffff]
+	%pra    [range 0x0000000060000000]
+
+For printing struct range.  struct range holds an arbitrary range of u64
+values.  If start is equal to end only 1 value is printed.

Do you mean printing only start value in start=equal case?

Yes I'll change the verbiage.

Ira

Documentation/core-api/printk-formats.rst +0 -0
index 03b102fc60bb..e1ebf0376154 100644

Documentation/core-api/printk-formats.rst Struct Range · line 245

Quoted 7 lines
Documentation/core-api/printk-formats.rst +1 -1
@@ -240,7 +240,7 @@ Struct Range
        %pra    [range 0x0000000060000000]
 
 For printing struct range.  struct range holds an arbitrary range of u64
-values.  If start is equal to end only 1 value is printed.
+values.  If start is equal to end only print the start value.
 
 Passed by reference.

That's nice, thanks!

Signature · 1 line
An old man doll... just what I always wanted! - Clara
author
#19 lore raw

PATCH v4 3/28 cxl/cdat: Use %pra for dpa range outputs

In reply to Ira Weiny

Now that there is a printk specifier for struct range use it in debug output of CDAT data.

To: Petr Mladek <[email protected]>
To: Steven Rostedt <[email protected]>
To: Andy Shevchenko <[email protected]>
To: Rasmus Villemoes <[email protected]>
To: Sergey Senozhatsky <[email protected]>
To: Jonathan Corbet <[email protected]> (maintainer:DOCUMENTATION)

---
 drivers/cxl/core/cdat.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
drivers/cxl/core/cdat.c +4 -4
index ef1621d40f05..438869df241a 100644
@@ -247,8 +247,8 @@ static void update_perf_entry(struct device *dev, struct dsmas_entry *dent,
 	dpa_perf->dpa_range = dent->dpa_range;
 	dpa_perf->qos_class = dent->qos_class;
 	dev_dbg(dev,
-		"DSMAS: dpa: %#llx qos: %d read_bw: %d write_bw %d read_lat: %d write_lat: %d\n",
-		dent->dpa_range.start, dpa_perf->qos_class,
+		"DSMAS: dpa: %pra qos: %d read_bw: %d write_bw %d read_lat: %d write_lat: %d\n",
+		&dent->dpa_range, dpa_perf->qos_class,
 		dent->coord[ACCESS_COORDINATE_CPU].read_bandwidth,
 		dent->coord[ACCESS_COORDINATE_CPU].write_bandwidth,
 		dent->coord[ACCESS_COORDINATE_CPU].read_latency,
@@ -279,8 +279,8 @@ static void cxl_memdev_set_qos_class(struct cxl_dev_state *cxlds,
 			 range_contains(&pmem_range, &dent->dpa_range))
 			update_perf_entry(dev, dent, &mds->pmem_perf);
 		else
-			dev_dbg(dev, "no partition for dsmas dpa: %#llx\n",
-				dent->dpa_range.start);
+			dev_dbg(dev, "no partition for dsmas dpa: %pra\n",
+				&dent->dpa_range);
 	}
 }
 
Signature · 1 line
2.46.0
Reviewed-by on v4 3/28
#20 lore raw

Re: [PATCH v4 03/28] cxl/cdat: Use %pra for dpa range outputs

Quoted 9 lines · On Mon, 07 Oct 2024 18:16:09 -0500 Ira Weiny <[email protected]> wrote:

Now that there is a printk specifier for struct range use it in debug output of CDAT data.

To: Petr Mladek <[email protected]>
To: Steven Rostedt <[email protected]>
To: Andy Shevchenko <[email protected]>
To: Rasmus Villemoes <[email protected]>
To: Sergey Senozhatsky <[email protected]>
To: Jonathan Corbet <[email protected]> (maintainer:DOCUMENTATION)

Quoted 3 lines
Reviewed-by on v4 3/28
#21 lore raw

Re: [PATCH v4 03/28] cxl/cdat: Use %pra for dpa range outputs

In reply to Ira Weiny
Quoted 9 lines · Ira Weiny

Now that there is a printk specifier for struct range use it in debug output of CDAT data.

To: Petr Mladek <[email protected]>
To: Steven Rostedt <[email protected]>
To: Andy Shevchenko <[email protected]>
To: Rasmus Villemoes <[email protected]>
To: Sergey Senozhatsky <[email protected]>
To: Jonathan Corbet <[email protected]> (maintainer:DOCUMENTATION)

Quoted 4 lines

---

Quoted 33 lines
 drivers/cxl/core/cdat.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
drivers/cxl/core/cdat.c +4 -4
index ef1621d40f05..438869df241a 100644
@@ -247,8 +247,8 @@ static void update_perf_entry(struct device *dev, struct dsmas_entry *dent,
 	dpa_perf->dpa_range = dent->dpa_range;
 	dpa_perf->qos_class = dent->qos_class;
 	dev_dbg(dev,
-		"DSMAS: dpa: %#llx qos: %d read_bw: %d write_bw %d read_lat: %d write_lat: %d\n",
-		dent->dpa_range.start, dpa_perf->qos_class,
+		"DSMAS: dpa: %pra qos: %d read_bw: %d write_bw %d read_lat: %d write_lat: %d\n",
+		&dent->dpa_range, dpa_perf->qos_class,
 		dent->coord[ACCESS_COORDINATE_CPU].read_bandwidth,
 		dent->coord[ACCESS_COORDINATE_CPU].write_bandwidth,
 		dent->coord[ACCESS_COORDINATE_CPU].read_latency,
@@ -279,8 +279,8 @@ static void cxl_memdev_set_qos_class(struct cxl_dev_state *cxlds,
 			 range_contains(&pmem_range, &dent->dpa_range))
 			update_perf_entry(dev, dent, &mds->pmem_perf);
 		else
-			dev_dbg(dev, "no partition for dsmas dpa: %#llx\n",
-				dent->dpa_range.start);
+			dev_dbg(dev, "no partition for dsmas dpa: %pra\n",
+				&dent->dpa_range);
 	}
 }
 
Signature · 2 lines
-- 
2.46.0
Signature · 1 line
Fan Ni
author
#22 lore raw

PATCH v4 4/28 range: Add range_overlaps()

In reply to Ira Weiny

Code to support CXL Dynamic Capacity devices will have extent ranges which need to be compared for intersection not a subset as is being checked in range_contains().

range_overlaps() is defined in btrfs with a different meaning from what is required in the standard range code. Dan Williams pointed this out in [1]. Adjust the btrfs call according to his suggestion there.

Then add a generic range_overlaps().

Cc Dan Williams <[email protected]>
Cc Chris Mason <[email protected]>
Cc Josef Bacik <[email protected]>
Cc David Sterba <[email protected]>

[1] https://lore.kernel.org/all/[email protected]/

---
 fs/btrfs/ordered-data.c | 10 +++++-----
 include/linux/range.h   |  7 +++++++
 2 files changed, 12 insertions(+), 5 deletions(-)
fs/btrfs/ordered-data.c +5 -5
index 2104d60c2161..744c3375ee6a 100644
@@ -111,8 +111,8 @@ static struct rb_node *__tree_search(struct rb_root *root, u64 file_offset,
 	return NULL;
 }
 
-static int range_overlaps(struct btrfs_ordered_extent *entry, u64 file_offset,
-			  u64 len)
+static int btrfs_range_overlaps(struct btrfs_ordered_extent *entry, u64 file_offset,
+				u64 len)
 {
 	if (file_offset + len <= entry->file_offset ||
 	    entry->file_offset + entry->num_bytes <= file_offset)
@@ -985,7 +985,7 @@ struct btrfs_ordered_extent *btrfs_lookup_ordered_range(
 
 	while (1) {
 		entry = rb_entry(node, struct btrfs_ordered_extent, rb_node);
-		if (range_overlaps(entry, file_offset, len))
+		if (btrfs_range_overlaps(entry, file_offset, len))
 			break;
 
 		if (entry->file_offset >= file_offset + len) {
@@ -1114,12 +1114,12 @@ struct btrfs_ordered_extent *btrfs_lookup_first_ordered_range(
 	}
 	if (prev) {
 		entry = rb_entry(prev, struct btrfs_ordered_extent, rb_node);
-		if (range_overlaps(entry, file_offset, len))
+		if (btrfs_range_overlaps(entry, file_offset, len))
 			goto out;
 	}
 	if (next) {
 		entry = rb_entry(next, struct btrfs_ordered_extent, rb_node);
-		if (range_overlaps(entry, file_offset, len))
+		if (btrfs_range_overlaps(entry, file_offset, len))
 			goto out;
 	}
 	/* No ordered extent in the range */
include/linux/range.h +7 -0
index 6ad0b73cb7ad..9a46f3212965 100644
@@ -13,11 +13,18 @@ static inline u64 range_len(const struct range *range)
 	return range->end - range->start + 1;
 }
 
+/* True if r1 completely contains r2 */
 static inline bool range_contains(struct range *r1, struct range *r2)
 {
 	return r1->start <= r2->start && r1->end >= r2->end;
 }
 
+/* True if any part of r1 overlaps r2 */
+static inline bool range_overlaps(struct range *r1, struct range *r2)
+{
+	return r1->start <= r2->end && r1->end >= r2->start;
+}
+
 int add_range(struct range *range, int az, int nr_range,
 		u64 start, u64 end);
 
Signature · 1 line
2.46.0
#23 lore raw

Re: [PATCH v4 04/28] range: Add range_overlaps()

include/linux/range.h static inline u64 range_len(const struct range *range) · line 23

Quoted 4 lines · Ira Weiny
include/linux/range.h +2 -0
+/* True if any part of r1 overlaps r2 */
+static inline bool range_overlaps(struct range *r1, struct range *r2)

I've noticed only now, you can constify the arguments, but this applise to other range_* functions so that can be done later in one go.

Quoted 3 lines
include/linux/range.h +3 -0
+{
+	return r1->start <= r2->end && r1->end >= r2->start;
+}
#24 lore raw

Re: [PATCH v4 04/28] range: Add range_overlaps()

Quoted 1 line · David Sterba

On Mon, Oct 07, 2024 at 06:16:10PM -0500, Ira Weiny wrote:

...

Quoted 4 lines
Quoted 1 line

+static inline bool range_overlaps(struct range *r1, struct range *r2)

I've noticed only now, you can constify the arguments, but this applise to other range_* functions so that can be done later in one go.

Frankly you may add the same to each new API being added to the file and the "one go" will never happen. So, I support your first part with constifying, but I think it would be rather done now to start that "one go" to happen.

Signature · 2 lines
With Best Regards,
Andy Shevchenko
#25 lore raw

Re: [PATCH v4 04/28] range: Add range_overlaps()

Quoted 2 lines · Andy Shevchenko
Quoted 1 line · David Sterba

On Mon, Oct 07, 2024 at 06:16:10PM -0500, Ira Weiny wrote:

...

Quoted 9 lines
Quoted 4 lines
Quoted 1 line

+static inline bool range_overlaps(struct range *r1, struct range *r2)

I've noticed only now, you can constify the arguments, but this applise to other range_* functions so that can be done later in one go.

Frankly you may add the same to each new API being added to the file and the "one go" will never happen. So, I support your first part with constifying, but I think it would be rather done now to start that "one go" to happen.

Alternatively there is should be the patch _in this series_ to make it happen before extending an API. I leave the choice to Ira.

Signature · 2 lines
With Best Regards,
Andy Shevchenko
author
#26 lore raw

Re: [PATCH v4 04/28] range: Add range_overlaps()

Quoted 11 lines · Andy Shevchenko
Quoted 2 lines · Andy Shevchenko
Quoted 1 line · David Sterba

On Mon, Oct 07, 2024 at 06:16:10PM -0500, Ira Weiny wrote:

...

Quoted 5 lines
Quoted 4 lines
Quoted 1 line

+static inline bool range_overlaps(struct range *r1, struct range *r2)

I've noticed only now, you can constify the arguments, but this applise to other range_* functions so that can be done later in one go.

Quoted 7 lines
Quoted 4 lines

Frankly you may add the same to each new API being added to the file and the "one go" will never happen. So, I support your first part with constifying, but I think it would be rather done now to start that "one go" to happen.

Alternatively there is should be the patch _in this series_ to make it happen before extending an API. I leave the choice to Ira.

I'm not sure I follow what you are saying here but I think you are saying to make those calls const prior to adding new ones.

I agree see:

https://lore.kernel.org/all/[email protected]/

Hopefully this is what you meant and closes this issue. Ira

#27 lore raw

Re: [PATCH v4 04/28] range: Add range_overlaps()

In reply to Andy Shevchenko
Quoted 5 lines · Andy Shevchenko
Quoted 1 line · David Sterba

On Mon, Oct 07, 2024 at 06:16:10PM -0500, Ira Weiny wrote:

...

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

+static inline bool range_overlaps(struct range *r1, struct range *r2)

I've noticed only now, you can constify the arguments, but this applise to other range_* functions so that can be done later in one go.

Frankly you may add the same to each new API being added to the file and the "one go" will never happen.

Yeah, but it's a minor issue for a 28 patchset, I don't know if there are some other major things still to do so that a v5 is expected.

If anybody is interested, reviewing APIs and interfaces with focus on some data structure and const is relatively easy, compile test is typically enough. The hard part is to find the missing ones. There's no compiler aid thad I'd know of (-Wsuggest-attribute=const is not for parameters), so it's been reading a file top-down for me.

Quoted 3 lines

So, I support your first part with
constifying, but I think it would be rather done now to start that "one go" to happen.

Agreed, one patch on top is probably the least intrusive way.

#28 lore raw

Re: [PATCH v4 04/28] range: Add range_overlaps()

Quoted 3 lines · David Sterba
Quoted 2 lines · Andy Shevchenko
Quoted 1 line · David Sterba

On Mon, Oct 07, 2024 at 06:16:10PM -0500, Ira Weiny wrote:

...

Quoted 10 lines
Quoted 7 lines
Quoted 4 lines
Quoted 1 line

+static inline bool range_overlaps(struct range *r1, struct range *r2)

I've noticed only now, you can constify the arguments, but this applise to other range_* functions so that can be done later in one go.

Frankly you may add the same to each new API being added to the file and the "one go" will never happen.

Yeah, but it's a minor issue for a 28 patchset, I don't know if there are some other major things still to do so that a v5 is expected.

At least seems printf() changes have to be amended, so I think v5 is warranted anyway.

Quoted 3 lines

If anybody is interested, reviewing APIs and interfaces with focus on some data structure and const is relatively easy, compile test is typically enough.

Except the cases where a const pointer has to be passed thru non-const (or integer) field in a data structure. Tons of the existing examples is ID tables that wanted to have kernel_ulong_t instead of const void * in driver data field.

Quoted 3 lines

The hard part is to find the missing ones. There's no compiler aid thad I'd know of (-Wsuggest-attribute=const is not for parameters), so it's been reading a file top-down for me.

Yeah...

Quoted 5 lines
Quoted 3 lines

So, I support your first part with
constifying, but I think it would be rather done now to start that "one go" to happen.

Agreed, one patch on top is probably the least intrusive way.

Signature · 2 lines
With Best Regards,
Andy Shevchenko
author
#29 lore raw

Re: [PATCH v4 04/28] range: Add range_overlaps()

In reply to David Sterba
Quoted 8 lines · David Sterba

include/linux/range.h static inline u64 range_len(const struct range *range) · line 23

Quoted 4 lines · Ira Weiny
include/linux/range.h +2 -0
+/* True if any part of r1 overlaps r2 */
+static inline bool range_overlaps(struct range *r1, struct range *r2)

I've noticed only now, you can constify the arguments, but this applise to other range_* functions so that can be done later in one go.

Looks like there will be a v5. I'll do a separate cleanup patch for range_contains() and change this one.

Thanks!
Ira

Quoted 4 lines
Quoted 4 lines
include/linux/range.h +3 -0
+{
+	return r1->start <= r2->end && r1->end >= r2->start;
+}
author
#30 lore raw

PATCH v4 5/28 dax: Document dax dev range tuple

In reply to Ira Weiny

The device DAX structure is being enhanced to track additional DCD information.

The current range tuple was not fully documented. Document it prior to adding information for DC.

---
Changes:
[iweiny: move to start of series]

---
 drivers/dax/dax-private.h | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
drivers/dax/dax-private.h +4 -1
index 446617b73aea..ccde98c3d4e2 100644
@@ -58,7 +58,10 @@ struct dax_mapping {
  * @dev - device core
  * @pgmap - pgmap for memmap setup / lifetime (driver owned)
  * @nr_range: size of @ranges
- * @ranges: resource-span + pgoff tuples for the instance
+ * @ranges: range tuples of memory used
+ * @pgoff: page offset
+ * @range: resource-span
+ * @mapping: device to assist in interrogating the range layout
  */
 struct dev_dax {
 	struct dax_region *region;
Signature · 1 line
2.46.0
#31 lore raw

Re: [PATCH v4 05/28] dax: Document dax dev range tuple

Quoted 9 lines · On Mon, 07 Oct 2024 18:16:11 -0500 Ira Weiny <[email protected]> wrote:

The device DAX structure is being enhanced to track additional DCD information.

The current range tuple was not fully documented. Document it prior to adding information for DC.

Isn't this a nested struct?
https://docs.kernel.org/doc-guide/kernel-doc.html#nested-structs-unions

I'm not quite sure how we document when it's a nested pointer to a a structure. Is it the same as for a 'normal' nested struct?

Quoted 12 lines

---
Changes:
[iweiny: move to start of series]

---
 drivers/dax/dax-private.h | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
drivers/dax/dax-private.h +0 -0
index 446617b73aea..ccde98c3d4e2 100644

drivers/dax/dax-private.h struct dax_mapping { · line 62

Quoted 6 lines
drivers/dax/dax-private.h +2 -1
@@ -58,7 +58,10 @@ struct dax_mapping {
  * @dev - device core
  * @pgmap - pgmap for memmap setup / lifetime (driver owned)
  * @nr_range: size of @ranges
- * @ranges: resource-span + pgoff tuples for the instance
+ * @ranges: range tuples of memory used
+ * @pgoff: page offset

@ranges.pgoff?
etc

Quoted 6 lines
drivers/dax/dax-private.h +2 -0
+ * @range: resource-span
+ * @mapping: device to assist in interrogating the range layout
  */
 struct dev_dax {
 	struct dax_region *region;
author
#32 lore raw

Re: [PATCH v4 05/28] dax: Document dax dev range tuple

Quoted 9 lines · Jonathan Cameron
Quoted 6 lines · On Mon, 07 Oct 2024 18:16:11 -0500 Ira Weiny <[email protected]> wrote:

The device DAX structure is being enhanced to track additional DCD information.

The current range tuple was not fully documented. Document it prior to adding information for DC.

Quoted 8 lines
Quoted 3 lines

Isn't this a nested struct?
https://docs.kernel.org/doc-guide/kernel-doc.html#nested-structs-unions

I'm not quite sure how we document when it's a nested pointer to a a structure. Is it the same as for a 'normal' nested struct?

In this case I think it best to document the struct and just document the reference. See below.

Quoted 8 lines
Quoted 8 lines

---
Changes:
[iweiny: move to start of series]

---
 drivers/dax/dax-private.h | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
Quoted 13 lines

drivers/dax/dax-private.h struct dax_mapping { · line 62

Quoted 11 lines
drivers/dax/dax-private.h +2 -1
index 446617b73aea..ccde98c3d4e2 100644
@@ -58,7 +58,10 @@ struct dax_mapping {
  * @dev - device core
  * @pgmap - pgmap for memmap setup / lifetime (driver owned)
  * @nr_range: size of @ranges
- * @ranges: resource-span + pgoff tuples for the instance
+ * @ranges: range tuples of memory used
+ * @pgoff: page offset

@ranges.pgoff?
etc

Ok yea.

As for the pointer to a structure. I think the best thing to do is simply document that structure.

Something like this building on this patch:

drivers/dax/dax-private.h +9 -3
index ccde98c3d4e2..b9816c933575 100644
@@ -40,6 +40,12 @@ struct dax_region {
        struct device *youngest;
 };
 
+/**
+ * struct dax_mapping - device to display mapping range attributes
+ * @dev: device representing this range
+ * @range_id: index within dev_dax ranges array
+ * @id: ida of this mapping
+ */
 struct dax_mapping {
        struct device dev;
        int range_id;
@@ -59,9 +65,9 @@ struct dax_mapping {
  * @pgmap - pgmap for memmap setup / lifetime (driver owned)
  * @nr_range: size of @ranges
  * @ranges: range tuples of memory used
- * @pgoff: page offset
- * @range: resource-span
- * @mapping: device to assist in interrogating the range layout
+ * @ranges.pgoff: page offset
+ * @ranges.range: resource-span
+ * @ranges.mapping: reference to the dax_mapping for this range
  */
 struct dev_dax {
        struct dax_region *region;
#33 lore raw

Re: [PATCH v4 05/28] dax: Document dax dev range tuple

Quoted 70 lines · On Fri, 11 Oct 2024 15:40:58 -0500 Ira Weiny <[email protected]> wrote:
Quoted 9 lines · Jonathan Cameron
Quoted 6 lines · On Mon, 07 Oct 2024 18:16:11 -0500 Ira Weiny <[email protected]> wrote:

The device DAX structure is being enhanced to track additional DCD information.

The current range tuple was not fully documented. Document it prior to adding information for DC.

Quoted 8 lines
Quoted 3 lines

Isn't this a nested struct?
https://docs.kernel.org/doc-guide/kernel-doc.html#nested-structs-unions

I'm not quite sure how we document when it's a nested pointer to a a structure. Is it the same as for a 'normal' nested struct?

In this case I think it best to document the struct and just document the reference. See below.

Quoted 8 lines
Quoted 8 lines

---
Changes:
[iweiny: move to start of series]

---
 drivers/dax/dax-private.h | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
Quoted 13 lines

drivers/dax/dax-private.h struct dax_mapping { · line 62

Quoted 11 lines
drivers/dax/dax-private.h +2 -1
index 446617b73aea..ccde98c3d4e2 100644
@@ -58,7 +58,10 @@ struct dax_mapping {
  * @dev - device core
  * @pgmap - pgmap for memmap setup / lifetime (driver owned)
  * @nr_range: size of @ranges
- * @ranges: resource-span + pgoff tuples for the instance
+ * @ranges: range tuples of memory used
+ * @pgoff: page offset  

@ranges.pgoff?
etc

Ok yea.

As for the pointer to a structure. I think the best thing to do is simply document that structure.

Something like this building on this patch:

drivers/dax/dax-private.h +6 -0
index ccde98c3d4e2..b9816c933575 100644
@@ -40,6 +40,12 @@ struct dax_region {
        struct device *youngest;
 };
 
+/**
+ * struct dax_mapping - device to display mapping range attributes
+ * @dev: device representing this range
+ * @range_id: index within dev_dax ranges array
+ * @id: ida of this mapping
+ */
 struct dax_mapping {
        struct device dev;
        int range_id;

drivers/dax/dax-private.h struct dax_mapping { · line 70

Quoted 9 lines
drivers/dax/dax-private.h +3 -3
@@ -59,9 +65,9 @@ struct dax_mapping {
  * @pgmap - pgmap for memmap setup / lifetime (driver owned)
  * @nr_range: size of @ranges
  * @ranges: range tuples of memory used
- * @pgoff: page offset
- * @range: resource-span
- * @mapping: device to assist in interrogating the range layout
+ * @ranges.pgoff: page offset
+ * @ranges.range: resource-span
+ * @ranges.mapping: reference to the dax_mapping for this range

Maybe just pull out definition of struct dev_dax_range? Avoids this confusion and no particularly obvious reason why it is embedded in the definition of dev_dax.

Quoted 4 lines
drivers/dax/dax-private.h +0 -0
  */
 struct dev_dax {
        struct dax_region *region;
author
#34 lore raw

PATCH v4 6/28 cxl/pci: Delay event buffer allocation

In reply to Ira Weiny

The event buffer does not need to be allocated if something has failed in setting up event irq's.

In prep for adjusting event configuration for DCD events move the buffer allocation to the end of the event configuration.

---
Changes:
[iweiny: keep tags for early simple patch] [Davidlohr, Jonathan, djiang: move to beginning of series]
[Dave feel free to pick this up if you like]

---
 drivers/cxl/pci.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
drivers/cxl/pci.c +4 -4
index 37164174b5fb..0ccd6fd98b9d 100644
@@ -764,10 +764,6 @@ static int cxl_event_config(struct pci_host_bridge *host_bridge,
 		return 0;
 	}
 
-	rc = cxl_mem_alloc_event_buf(mds);
-	if (rc)
-		return rc;
-
 	rc = cxl_event_get_int_policy(mds, &policy);
 	if (rc)
 		return rc;
@@ -781,6 +777,10 @@ static int cxl_event_config(struct pci_host_bridge *host_bridge,
 		return -EBUSY;
 	}
 
+	rc = cxl_mem_alloc_event_buf(mds);
+	if (rc)
+		return rc;
+
 	rc = cxl_event_irqsetup(mds);
 	if (rc)
 		return rc;
Signature · 1 line
2.46.0
Reviewed-by on v4 6/28
#35 lore raw

Re: [PATCH v4 06/28] cxl/pci: Delay event buffer allocation

Quoted 11 lines · Ira Weiny

The event buffer does not need to be allocated if something has failed in setting up event irq's.

In prep for adjusting event configuration for DCD events move the buffer allocation to the end of the event configuration.

Quoted 39 lines

---
Changes:
[iweiny: keep tags for early simple patch] [Davidlohr, Jonathan, djiang: move to beginning of series]
[Dave feel free to pick this up if you like]

---
 drivers/cxl/pci.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
drivers/cxl/pci.c +4 -4
index 37164174b5fb..0ccd6fd98b9d 100644
@@ -764,10 +764,6 @@ static int cxl_event_config(struct pci_host_bridge *host_bridge,
 		return 0;
 	}
 
-	rc = cxl_mem_alloc_event_buf(mds);
-	if (rc)
-		return rc;
-
 	rc = cxl_event_get_int_policy(mds, &policy);
 	if (rc)
 		return rc;
@@ -781,6 +777,10 @@ static int cxl_event_config(struct pci_host_bridge *host_bridge,
 		return -EBUSY;
 	}
 
+	rc = cxl_mem_alloc_event_buf(mds);
+	if (rc)
+		return rc;
+
 	rc = cxl_event_irqsetup(mds);
 	if (rc)
 		return rc;
Signature · 2 lines
-- 
2.46.0
Signature · 1 line
Fan Ni
#36 lore raw

PATCH v4 7/28 cxl/mbox: Flag support for Dynamic Capacity Devices (DCD)

In reply to Ira Weiny

From: Navneet Singh <[email protected]>

Per the CXL 3.1 specification software must check the Command Effects Log (CEL) for dynamic capacity command support.

Detect support for the DCD commands while reading the CEL, including:

Get DC Config
Get DC Extent List
Add DC Response
Release DC

---
Changes:
[iweiny: Keep tags for this early simple patch]
[Davidlohr: update commit message]
[djiang: Fix misalignment]

---
 drivers/cxl/core/mbox.c | 33 +++++++++++++++++++++++++++++++++
 drivers/cxl/cxlmem.h    | 15 +++++++++++++++
 2 files changed, 48 insertions(+)
drivers/cxl/core/mbox.c +33 -0
index 946f8e44455f..8bd5bf1a746d 100644
@@ -164,6 +164,34 @@ static void cxl_set_security_cmd_enabled(struct cxl_security_state *security,
 	}
 }
 
+static bool cxl_is_dcd_command(u16 opcode)
+{
+#define CXL_MBOX_OP_DCD_CMDS 0x48
+
+	return (opcode >> 8) == CXL_MBOX_OP_DCD_CMDS;
+}
+
+static void cxl_set_dcd_cmd_enabled(struct cxl_memdev_state *mds,
+				    u16 opcode)
+{
+	switch (opcode) {
+	case CXL_MBOX_OP_GET_DC_CONFIG:
+		set_bit(CXL_DCD_ENABLED_GET_CONFIG, mds->dcd_cmds);
+		break;
+	case CXL_MBOX_OP_GET_DC_EXTENT_LIST:
+		set_bit(CXL_DCD_ENABLED_GET_EXTENT_LIST, mds->dcd_cmds);
+		break;
+	case CXL_MBOX_OP_ADD_DC_RESPONSE:
+		set_bit(CXL_DCD_ENABLED_ADD_RESPONSE, mds->dcd_cmds);
+		break;
+	case CXL_MBOX_OP_RELEASE_DC:
+		set_bit(CXL_DCD_ENABLED_RELEASE, mds->dcd_cmds);
+		break;
+	default:
+		break;
+	}
+}
+
 static bool cxl_is_poison_command(u16 opcode)
 {
 #define CXL_MBOX_OP_POISON_CMDS 0x43
@@ -751,6 +779,11 @@ static void cxl_walk_cel(struct cxl_memdev_state *mds, size_t size, u8 *cel)
 			enabled++;
 		}
 
+		if (cxl_is_dcd_command(opcode)) {
+			cxl_set_dcd_cmd_enabled(mds, opcode);
+			enabled++;
+		}
+
 		dev_dbg(dev, "Opcode 0x%04x %s\n", opcode,
 			enabled ? "enabled" : "unsupported by driver");
 	}
drivers/cxl/cxlmem.h +15 -0
index 2a25d1957ddb..e8907c403edb 100644
@@ -239,6 +239,15 @@ struct cxl_event_state {
 	struct mutex log_lock;
 };
 
+/* Device enabled DCD commands */
+enum dcd_cmd_enabled_bits {
+	CXL_DCD_ENABLED_GET_CONFIG,
+	CXL_DCD_ENABLED_GET_EXTENT_LIST,
+	CXL_DCD_ENABLED_ADD_RESPONSE,
+	CXL_DCD_ENABLED_RELEASE,
+	CXL_DCD_ENABLED_MAX
+};
+
 /* Device enabled poison commands */
 enum poison_cmd_enabled_bits {
 	CXL_POISON_ENABLED_LIST,
@@ -461,6 +470,7 @@ static inline struct cxl_dev_state *mbox_to_cxlds(struct cxl_mailbox *cxl_mbox)
  * @lsa_size: Size of Label Storage Area
  *                (CXL 2.0 8.2.9.5.1.1 Identify Memory Device)
  * @firmware_version: Firmware version for the memory device.
+ * @dcd_cmds: List of DCD commands implemented by memory device
  * @enabled_cmds: Hardware commands found enabled in CEL.
  * @exclusive_cmds: Commands that are kernel-internal only
  * @total_bytes: sum of all possible capacities
@@ -485,6 +495,7 @@ struct cxl_memdev_state {
 	struct cxl_dev_state cxlds;
 	size_t lsa_size;
 	char firmware_version[0x10];
+	DECLARE_BITMAP(dcd_cmds, CXL_DCD_ENABLED_MAX);
 	DECLARE_BITMAP(enabled_cmds, CXL_MEM_COMMAND_ID_MAX);
 	DECLARE_BITMAP(exclusive_cmds, CXL_MEM_COMMAND_ID_MAX);
 	u64 total_bytes;
@@ -554,6 +565,10 @@ enum cxl_opcode {
 	CXL_MBOX_OP_UNLOCK		= 0x4503,
 	CXL_MBOX_OP_FREEZE_SECURITY	= 0x4504,
 	CXL_MBOX_OP_PASSPHRASE_SECURE_ERASE	= 0x4505,
+	CXL_MBOX_OP_GET_DC_CONFIG	= 0x4800,
+	CXL_MBOX_OP_GET_DC_EXTENT_LIST	= 0x4801,
+	CXL_MBOX_OP_ADD_DC_RESPONSE	= 0x4802,
+	CXL_MBOX_OP_RELEASE_DC		= 0x4803,
 	CXL_MBOX_OP_MAX			= 0x10000
 };
 
Signature · 1 line
2.46.0
#37 lore raw

PATCH v4 8/28 cxl/mem: Read dynamic capacity configuration from the device

In reply to Ira Weiny

From: Navneet Singh <[email protected]>

Devices which optionally support Dynamic Capacity (DC) are configured via mailbox commands. CXL 3.1 requires the host to issue the Get DC Configuration command in order to properly configure DCDs. Without the Get DC Configuration command DCD can't be supported.

Implement the DC mailbox commands as specified in CXL 3.1 section 8.2.9.9.9 (opcodes 48XXh) to read and store the DCD configuration information. Disable DCD if DCD is not supported. Leverage the Get DC Configuration command supported bit to indicate if DCD support.

Linux has no use for the trailing fields of the Get Dynamic Capacity Configuration Output Payload (Total number of supported extents, number of available extents, total number of supported tags, and number of available tags). Avoid defining those fields to use the more useful dynamic C array.

---
Changes:
[iweiny: Adjust for mailbox split]
[djiang: remove setting region names to '<nil>']
[fan: Clean up dev_{err,dbg}]

---
 drivers/cxl/core/mbox.c | 166 +++++++++++++++++++++++++++++++++++++++++++++++-
 drivers/cxl/cxlmem.h    |  64 ++++++++++++++++++-
 drivers/cxl/pci.c       |   4 ++
 3 files changed, 232 insertions(+), 2 deletions(-)
drivers/cxl/core/mbox.c +165 -1
index 8bd5bf1a746d..4b51ddd1ff94 100644
@@ -1168,7 +1168,7 @@ int cxl_dev_state_identify(struct cxl_memdev_state *mds)
 	if (rc < 0)
 		return rc;
 
-	mds->total_bytes =
+	mds->static_bytes =
 		le64_to_cpu(id.total_capacity) * CXL_CAPACITY_MULTIPLIER;
 	mds->volatile_only_bytes =
 		le64_to_cpu(id.volatile_capacity) * CXL_CAPACITY_MULTIPLIER;
@@ -1274,6 +1274,154 @@ int cxl_mem_sanitize(struct cxl_memdev *cxlmd, u16 cmd)
 	return rc;
 }
 
+static int cxl_dc_save_region_info(struct cxl_memdev_state *mds, u8 index,
+				   struct cxl_dc_region_config *region_config)
+{
+	struct cxl_dc_region_info *dcr = &mds->dc_region[index];
+	struct device *dev = mds->cxlds.dev;
+
+	dcr->base = le64_to_cpu(region_config->region_base);
+	dcr->decode_len = le64_to_cpu(region_config->region_decode_length);
+	dcr->decode_len *= CXL_CAPACITY_MULTIPLIER;
+	dcr->len = le64_to_cpu(region_config->region_length);
+	dcr->blk_size = le64_to_cpu(region_config->region_block_size);
+	dcr->dsmad_handle = le32_to_cpu(region_config->region_dsmad_handle);
+	dcr->flags = region_config->flags;
+	snprintf(dcr->name, CXL_DC_REGION_STRLEN, "dc%d", index);
+
+	/* Check regions are in increasing DPA order */
+	if (index > 0) {
+		struct cxl_dc_region_info *prev_dcr = &mds->dc_region[index - 1];
+
+		if ((prev_dcr->base + prev_dcr->decode_len) > dcr->base) {
+			dev_err(dev,
+				"DPA ordering violation for DC region %d and %d\n",
+				index - 1, index);
+			return -EINVAL;
+		}
+	}
+
+	if (!IS_ALIGNED(dcr->base, SZ_256M) ||
+	    !IS_ALIGNED(dcr->base, dcr->blk_size)) {
+		dev_err(dev, "DC region %d invalid base %#llx blk size %#llx\n",
+			index, dcr->base, dcr->blk_size);
+		return -EINVAL;
+	}
+
+	if (dcr->decode_len == 0 || dcr->len == 0 || dcr->decode_len < dcr->len ||
+	    !IS_ALIGNED(dcr->len, dcr->blk_size)) {
+		dev_err(dev, "DC region %d invalid length; decode %#llx len %#llx blk size %#llx\n",
+			index, dcr->decode_len, dcr->len, dcr->blk_size);
+		return -EINVAL;
+	}
+
+	if (dcr->blk_size == 0 || dcr->blk_size % CXL_DCD_BLOCK_LINE_SIZE ||
+	    !is_power_of_2(dcr->blk_size)) {
+		dev_err(dev, "DC region %d invalid block size; %#llx\n",
+			index, dcr->blk_size);
+		return -EINVAL;
+	}
+
+	dev_dbg(dev,
+		"DC region %s base %#llx length %#llx block size %#llx\n",
+		dcr->name, dcr->base, dcr->decode_len, dcr->blk_size);
+
+	return 0;
+}
+
+/* Returns the number of regions in dc_resp or -ERRNO */
+static int cxl_get_dc_config(struct cxl_memdev_state *mds, u8 start_region,
+			     struct cxl_mbox_get_dc_config_out *dc_resp,
+			     size_t dc_resp_size)
+{
+	struct cxl_mbox_get_dc_config_in get_dc = (struct cxl_mbox_get_dc_config_in) {
+		.region_count = CXL_MAX_DC_REGION,
+		.start_region_index = start_region,
+	};
+	struct cxl_mbox_cmd mbox_cmd = (struct cxl_mbox_cmd) {
+		.opcode = CXL_MBOX_OP_GET_DC_CONFIG,
+		.payload_in = &get_dc,
+		.size_in = sizeof(get_dc),
+		.size_out = dc_resp_size,
+		.payload_out = dc_resp,
+		.min_out = 1,
+	};
+	struct device *dev = mds->cxlds.dev;
+	int rc;
+
+	rc = cxl_internal_send_cmd(&mds->cxlds.cxl_mbox, &mbox_cmd);
+	if (rc < 0)
+		return rc;
+
+	dev_dbg(dev, "Read %d/%d DC regions\n",
+		dc_resp->regions_returned, dc_resp->avail_region_count);
+	return dc_resp->regions_returned;
+}
+
+/**
+ * cxl_dev_dynamic_capacity_identify() - Reads the dynamic capacity
+ *					 information from the device.
+ * @mds: The memory device state
+ *
+ * Read Dynamic Capacity information from the device and populate the state
+ * structures for later use.
+ *
+ * Return: 0 if identify was executed successfully, -ERRNO on error.
+ */
+int cxl_dev_dynamic_capacity_identify(struct cxl_memdev_state *mds)
+{
+	size_t dc_resp_size = mds->cxlds.cxl_mbox.payload_size;
+	struct device *dev = mds->cxlds.dev;
+	u8 start_region, i;
+
+	if (!cxl_dcd_supported(mds)) {
+		dev_dbg(dev, "DCD not supported\n");
+		return 0;
+	}
+
+	struct cxl_mbox_get_dc_config_out *dc_resp __free(kfree) =
+					kvmalloc(dc_resp_size, GFP_KERNEL);
+	if (!dc_resp)
+		return -ENOMEM;
+
+	start_region = 0;
+	do {
+		int rc, j;
+
+		rc = cxl_get_dc_config(mds, start_region, dc_resp, dc_resp_size);
+		if (rc < 0) {
+			dev_err(dev, "Failed to get DC config: %d\n", rc);
+			return rc;
+		}
+
+		mds->nr_dc_region += rc;
+
+		if (mds->nr_dc_region < 1 || mds->nr_dc_region > CXL_MAX_DC_REGION) {
+			dev_err(dev, "Invalid num of dynamic capacity regions %d\n",
+				mds->nr_dc_region);
+			return -EINVAL;
+		}
+
+		for (i = start_region, j = 0; i < mds->nr_dc_region; i++, j++) {
+			rc = cxl_dc_save_region_info(mds, i, &dc_resp->region[j]);
+			if (rc)
+				return rc;
+		}
+
+		start_region = mds->nr_dc_region;
+
+	} while (mds->nr_dc_region < dc_resp->avail_region_count);
+
+	mds->dynamic_bytes =
+		mds->dc_region[mds->nr_dc_region - 1].base +
+		mds->dc_region[mds->nr_dc_region - 1].decode_len -
+		mds->dc_region[0].base;
+	dev_dbg(dev, "Total dynamic range: %#llx\n", mds->dynamic_bytes);
+
+	return 0;
+}
+EXPORT_SYMBOL_NS_GPL(cxl_dev_dynamic_capacity_identify, CXL);
+
 static int add_dpa_res(struct device *dev, struct resource *parent,
 		       struct resource *res, resource_size_t start,
 		       resource_size_t size, const char *type)
@@ -1304,8 +1452,15 @@ int cxl_mem_create_range_info(struct cxl_memdev_state *mds)
 {
 	struct cxl_dev_state *cxlds = &mds->cxlds;
 	struct device *dev = cxlds->dev;
+	size_t untenanted_mem;
 	int rc;
 
+	mds->total_bytes = mds->static_bytes;
+	if (mds->nr_dc_region) {
+		untenanted_mem = mds->dc_region[0].base - mds->static_bytes;
+		mds->total_bytes += untenanted_mem + mds->dynamic_bytes;
+	}
+
 	if (!cxlds->media_ready) {
 		cxlds->dpa_res = DEFINE_RES_MEM(0, 0);
 		cxlds->ram_res = DEFINE_RES_MEM(0, 0);
@@ -1315,6 +1470,15 @@ int cxl_mem_create_range_info(struct cxl_memdev_state *mds)
 
 	cxlds->dpa_res = DEFINE_RES_MEM(0, mds->total_bytes);
 
+	for (int i = 0; i < mds->nr_dc_region; i++) {
+		struct cxl_dc_region_info *dcr = &mds->dc_region[i];
+
+		rc = add_dpa_res(dev, &cxlds->dpa_res, &cxlds->dc_res[i],
+				 dcr->base, dcr->decode_len, dcr->name);
+		if (rc)
+			return rc;
+	}
+
 	if (mds->partition_align_bytes == 0) {
 		rc = add_dpa_res(dev, &cxlds->dpa_res, &cxlds->ram_res, 0,
 				 mds->volatile_only_bytes, "ram");
drivers/cxl/cxlmem.h +63 -1

129 lines not on this page · raw message

drivers/cxl/pci.c +4 -0
index 0ccd6fd98b9d..fc5ab74448cc 100644
@@ -899,6 +899,10 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 	if (rc)
 		return rc;
 
+	rc = cxl_dev_dynamic_capacity_identify(mds);
+	if (rc)
+		cxl_disable_dcd(mds);
+
 	rc = cxl_mem_create_range_info(mds);
 	if (rc)
 		return rc;
Signature · 1 line
2.46.0
Reviewed-by on v4 8/28
#38 lore raw

Re: [PATCH v4 08/28] cxl/mem: Read dynamic capacity configuration from the device

Quoted 18 lines · On Mon, 07 Oct 2024 18:16:14 -0500 [email protected] wrote:

From: Navneet Singh <[email protected]>

Devices which optionally support Dynamic Capacity (DC) are configured via mailbox commands. CXL 3.1 requires the host to issue the Get DC Configuration command in order to properly configure DCDs. Without the Get DC Configuration command DCD can't be supported.

Implement the DC mailbox commands as specified in CXL 3.1 section 8.2.9.9.9 (opcodes 48XXh) to read and store the DCD configuration information. Disable DCD if DCD is not supported. Leverage the Get DC Configuration command supported bit to indicate if DCD support.

Linux has no use for the trailing fields of the Get Dynamic Capacity Configuration Output Payload (Total number of supported extents, number of available extents, total number of supported tags, and number of available tags). Avoid defining those fields to use the more useful dynamic C array.

Quoted 4 lines

Looks fine to me. Trivial comment inline

Quoted 4 lines
drivers/cxl/cxlmem.h +0 -0
index e8907c403edb..0690b917b1e0 100644

...

Quoted 5 lines
drivers/cxl/cxlmem.h +5 -0
+/* See CXL 3.1 Table 8-164 get dynamic capacity config Output Payload */
+struct cxl_mbox_get_dc_config_out {
+	u8 avail_region_count;
+	u8 regions_returned;
+	u8 rsvd[6];

drivers/cxl/cxlmem.h struct cxl_mbox_set_partition_info { · line 755

Quoted 10 lines
drivers/cxl/cxlmem.h +10 -0
+	/* See CXL 3.1 Table 8-165 */
+	struct cxl_dc_region_config {
+		__le64 region_base;
+		__le64 region_decode_length;
+		__le64 region_length;
+		__le64 region_block_size;
+		__le32 region_dsmad_handle;
+		u8 flags;
+		u8 rsvd[3];
+	} __packed region[];

Could throw in a __counted_by I think?

Quoted 4 lines
drivers/cxl/cxlmem.h +4 -0
+	/* Trailing fields unused */
+} __packed;
+#define CXL_DYNAMIC_CAPACITY_SANITIZE_ON_RELEASE_FLAG BIT(0)
+#define CXL_DCD_BLOCK_LINE_SIZE 0x40
author
#39 lore raw

Re: [PATCH v4 08/28] cxl/mem: Read dynamic capacity configuration from the device

Quoted 21 lines · Jonathan Cameron
Quoted 18 lines · On Mon, 07 Oct 2024 18:16:14 -0500 [email protected] wrote:

From: Navneet Singh <[email protected]>

Devices which optionally support Dynamic Capacity (DC) are configured via mailbox commands. CXL 3.1 requires the host to issue the Get DC Configuration command in order to properly configure DCDs. Without the Get DC Configuration command DCD can't be supported.

Implement the DC mailbox commands as specified in CXL 3.1 section 8.2.9.9.9 (opcodes 48XXh) to read and store the DCD configuration information. Disable DCD if DCD is not supported. Leverage the Get DC Configuration command supported bit to indicate if DCD support.

Linux has no use for the trailing fields of the Get Dynamic Capacity Configuration Output Payload (Total number of supported extents, number of available extents, total number of supported tags, and number of available tags). Avoid defining those fields to use the more useful dynamic C array.

Quoted 6 lines
Quoted 4 lines

Looks fine to me. Trivial comment inline

Thanks.

Quoted 19 lines
Quoted 4 lines
drivers/cxl/cxlmem.h +0 -0
index e8907c403edb..0690b917b1e0 100644

...

Quoted 9 lines
drivers/cxl/cxlmem.h +9 -0
+/* See CXL 3.1 Table 8-164 get dynamic capacity config Output Payload */
+struct cxl_mbox_get_dc_config_out {
+	u8 avail_region_count;
+	u8 regions_returned;
+	u8 rsvd[6];
+	/* See CXL 3.1 Table 8-165 */
+	struct cxl_dc_region_config {
+		__le64 region_base;
+		__le64 region_decode_length;
Quoted 8 lines

drivers/cxl/cxlmem.h struct cxl_mbox_set_partition_info { · line 755

Quoted 6 lines
drivers/cxl/cxlmem.h +6 -0
+		__le64 region_length;
+		__le64 region_block_size;
+		__le32 region_dsmad_handle;
+		u8 flags;
+		u8 rsvd[3];
+	} __packed region[];

Could throw in a __counted_by I think?

I was not sure if this would work considering this is coming from the hardware. From what I have read I think it will but only because the region count can't be byte swapped.

Is this something we want to do with structs coming from hardware when we can?

Ira

#40 lore raw

Re: [PATCH v4 08/28] cxl/mem: Read dynamic capacity configuration from the device

Quoted 59 lines · On Sun, 13 Oct 2024 19:05:11 -0500 Ira Weiny <[email protected]> wrote:
Quoted 21 lines · Jonathan Cameron
Quoted 18 lines · On Mon, 07 Oct 2024 18:16:14 -0500 [email protected] wrote:

From: Navneet Singh <[email protected]>

Devices which optionally support Dynamic Capacity (DC) are configured via mailbox commands. CXL 3.1 requires the host to issue the Get DC Configuration command in order to properly configure DCDs. Without the Get DC Configuration command DCD can't be supported.

Implement the DC mailbox commands as specified in CXL 3.1 section 8.2.9.9.9 (opcodes 48XXh) to read and store the DCD configuration information. Disable DCD if DCD is not supported. Leverage the Get DC Configuration command supported bit to indicate if DCD support.

Linux has no use for the trailing fields of the Get Dynamic Capacity Configuration Output Payload (Total number of supported extents, number of available extents, total number of supported tags, and number of available tags). Avoid defining those fields to use the more useful dynamic C array.

Quoted 6 lines
Quoted 4 lines

Looks fine to me. Trivial comment inline

Thanks.

Quoted 19 lines
Quoted 4 lines
drivers/cxl/cxlmem.h +0 -0
index e8907c403edb..0690b917b1e0 100644

...

Quoted 9 lines
drivers/cxl/cxlmem.h +9 -0
+/* See CXL 3.1 Table 8-164 get dynamic capacity config Output Payload */
+struct cxl_mbox_get_dc_config_out {
+	u8 avail_region_count;
+	u8 regions_returned;
+	u8 rsvd[6];
+	/* See CXL 3.1 Table 8-165 */
+	struct cxl_dc_region_config {
+		__le64 region_base;
+		__le64 region_decode_length;
Quoted 8 lines

drivers/cxl/cxlmem.h struct cxl_mbox_set_partition_info { · line 755

Quoted 6 lines
drivers/cxl/cxlmem.h +6 -0
+		__le64 region_length;
+		__le64 region_block_size;
+		__le32 region_dsmad_handle;
+		u8 flags;
+		u8 rsvd[3];
+	} __packed region[];  

Could throw in a __counted_by I think?

Quoted 3 lines

I was not sure if this would work considering this is coming from the hardware. From what I have read I think it will but only because the region count can't be byte swapped.

__counted_by_le() deals potentially larger values by just making the __counted_by go away on big endian architectures.

Quoted 2 lines

Is this something we want to do with structs coming from hardware when we can?

I think we still do. Gustavo, Kees?

Quoted 2 lines

Ira

#41 lore raw

Re: [PATCH v4 08/28] cxl/mem: Read dynamic capacity configuration from the device

Quoted 66 lines · Jonathan Cameron
Quoted 59 lines · On Sun, 13 Oct 2024 19:05:11 -0500 Ira Weiny <[email protected]> wrote:
Quoted 21 lines · Jonathan Cameron
Quoted 18 lines · On Mon, 07 Oct 2024 18:16:14 -0500 [email protected] wrote:

From: Navneet Singh <[email protected]>

Devices which optionally support Dynamic Capacity (DC) are configured via mailbox commands. CXL 3.1 requires the host to issue the Get DC Configuration command in order to properly configure DCDs. Without the Get DC Configuration command DCD can't be supported.

Implement the DC mailbox commands as specified in CXL 3.1 section 8.2.9.9.9 (opcodes 48XXh) to read and store the DCD configuration information. Disable DCD if DCD is not supported. Leverage the Get DC Configuration command supported bit to indicate if DCD support.

Linux has no use for the trailing fields of the Get Dynamic Capacity Configuration Output Payload (Total number of supported extents, number of available extents, total number of supported tags, and number of available tags). Avoid defining those fields to use the more useful dynamic C array.

Quoted 6 lines
Quoted 4 lines

Looks fine to me. Trivial comment inline

Thanks.

Quoted 19 lines
Quoted 4 lines
drivers/cxl/cxlmem.h +0 -0
index e8907c403edb..0690b917b1e0 100644

...

Quoted 9 lines
drivers/cxl/cxlmem.h +9 -0
+/* See CXL 3.1 Table 8-164 get dynamic capacity config Output Payload */
+struct cxl_mbox_get_dc_config_out {
+	u8 avail_region_count;
+	u8 regions_returned;
+	u8 rsvd[6];
+	/* See CXL 3.1 Table 8-165 */
+	struct cxl_dc_region_config {
+		__le64 region_base;
+		__le64 region_decode_length;
Quoted 8 lines

drivers/cxl/cxlmem.h struct cxl_mbox_set_partition_info { · line 755

Quoted 6 lines
drivers/cxl/cxlmem.h +6 -0
+		__le64 region_length;
+		__le64 region_block_size;
+		__le32 region_dsmad_handle;
+		u8 flags;
+		u8 rsvd[3];
+	} __packed region[];  

Could throw in a __counted_by I think?

Quoted 3 lines

I was not sure if this would work considering this is coming from the hardware. From what I have read I think it will but only because the region count can't be byte swapped.

Quoted 8 lines

__counted_by_le() deals potentially larger values by just making the __counted_by go away on big endian architectures.

Quoted 2 lines

Is this something we want to do with structs coming from hardware when we can?

I think we still do. Gustavo, Kees?

As long as "regions_returned" got correctly set by hardware before anything in the kernel touches the "region" array, it can totally be used by a __counted_by annotation.

Signature · 1 line
Kees Cook
#42 lore raw

PATCH v4 9/28 cxl/core: Separate region mode from decoder mode

In reply to Ira Weiny

From: Navneet Singh <[email protected]>

Until now region modes and decoder modes were equivalent in that both modes were either PMEM or RAM. The addition of Dynamic Capacity partitions defines up to 8 DC partitions per device.

The region mode is thus no longer equivalent to the endpoint decoder mode. IOW the endpoint decoders may have modes of DC0-DC7 while the region mode is simply DC.

Define a new region mode enumeration which applies to regions separate from the decoder mode. Adjust the code to process these modes independently.

There is no equal to decoder mode dead in region modes. Avoid constructing regions with decoders which have been flagged as dead.

---
Changes:
[iweiny: rebase]
[Jonathan: remove dead code]
[Jonathan: clarify commit message]

---
 drivers/cxl/core/cdat.c   |  6 ++--
 drivers/cxl/core/region.c | 75 ++++++++++++++++++++++++++++++++++-------------
 drivers/cxl/cxl.h         | 26 ++++++++++++++--
 3 files changed, 82 insertions(+), 25 deletions(-)
drivers/cxl/core/cdat.c +3 -3
index 438869df241a..bd50bb655741 100644
@@ -571,17 +571,17 @@ static bool dpa_perf_contains(struct cxl_dpa_perf *perf,
 }
 
 static struct cxl_dpa_perf *cxled_get_dpa_perf(struct cxl_endpoint_decoder *cxled,
-					       enum cxl_decoder_mode mode)
+					       enum cxl_region_mode mode)
 {
 	struct cxl_memdev *cxlmd = cxled_to_memdev(cxled);
 	struct cxl_memdev_state *mds = to_cxl_memdev_state(cxlmd->cxlds);
 	struct cxl_dpa_perf *perf;
 
 	switch (mode) {
-	case CXL_DECODER_RAM:
+	case CXL_REGION_RAM:
 		perf = &mds->ram_perf;
 		break;
-	case CXL_DECODER_PMEM:
+	case CXL_REGION_PMEM:
 		perf = &mds->pmem_perf;
 		break;
 	default:
drivers/cxl/core/region.c +55 -20

177 lines not on this page · raw message

drivers/cxl/cxl.h +24 -2
index 0d8b810a51f0..5d74eb4ffab3 100644
@@ -388,6 +388,27 @@ static inline const char *cxl_decoder_mode_name(enum cxl_decoder_mode mode)
 	return "mixed";
 }
 
+enum cxl_region_mode {
+	CXL_REGION_NONE,
+	CXL_REGION_RAM,
+	CXL_REGION_PMEM,
+	CXL_REGION_MIXED,
+};
+
+static inline const char *cxl_region_mode_name(enum cxl_region_mode mode)
+{
+	static const char * const names[] = {
+		[CXL_REGION_NONE] = "none",
+		[CXL_REGION_RAM] = "ram",
+		[CXL_REGION_PMEM] = "pmem",
+		[CXL_REGION_MIXED] = "mixed",
+	};
+
+	if (mode >= CXL_REGION_NONE && mode <= CXL_REGION_MIXED)
+		return names[mode];
+	return "mixed";
+}
+
 /*
  * Track whether this decoder is reserved for region autodiscovery, or
  * free for userspace provisioning.
@@ -515,7 +536,8 @@ struct cxl_region_params {
  * struct cxl_region - CXL region
  * @dev: This region's device
  * @id: This region's id. Id is globally unique across all regions
- * @mode: Endpoint decoder allocation / access mode
+ * @mode: Region mode which defines which endpoint decoder modes the region is
+ *        compatible with
  * @type: Endpoint decoder target type
  * @cxl_nvb: nvdimm bridge for coordinating @cxlr_pmem setup / shutdown
  * @cxlr_pmem: (for pmem regions) cached copy of the nvdimm bridge
@@ -528,7 +550,7 @@ struct cxl_region_params {
 struct cxl_region {
 	struct device dev;
 	int id;
-	enum cxl_decoder_mode mode;
+	enum cxl_region_mode mode;
 	enum cxl_decoder_type type;
 	struct cxl_nvdimm_bridge *cxl_nvb;
 	struct cxl_pmem_region *cxlr_pmem;
Signature · 1 line
2.46.0
Reviewed-by on v4 9/28
#43 lore raw

Re: [PATCH v4 09/28] cxl/core: Separate region mode from decoder mode

Quoted 14 lines · On Mon, 07 Oct 2024 18:16:15 -0500 [email protected] wrote:

From: Navneet Singh <[email protected]>

Until now region modes and decoder modes were equivalent in that both modes were either PMEM or RAM. The addition of Dynamic Capacity partitions defines up to 8 DC partitions per device.

The region mode is thus no longer equivalent to the endpoint decoder mode. IOW the endpoint decoders may have modes of DC0-DC7 while the region mode is simply DC.

Define a new region mode enumeration which applies to regions separate from the decoder mode. Adjust the code to process these modes independently.

Quoted 7 lines

There is no equal to decoder mode dead in region modes. Avoid constructing regions with decoders which have been flagged as dead.

LGTM

Reviewed-by on v4 9/28
#44 lore raw

Re: [PATCH v4 09/28] cxl/core: Separate region mode from decoder mode

In reply to [email protected]
Quoted 14 lines · On Mon, Oct 07, 2024 at 06:16:15PM -0500, [email protected] wrote:

From: Navneet Singh <[email protected]>

Until now region modes and decoder modes were equivalent in that both modes were either PMEM or RAM. The addition of Dynamic Capacity partitions defines up to 8 DC partitions per device.

The region mode is thus no longer equivalent to the endpoint decoder mode. IOW the endpoint decoders may have modes of DC0-DC7 while the region mode is simply DC.

Define a new region mode enumeration which applies to regions separate from the decoder mode. Adjust the code to process these modes independently.

Quoted 8 lines

There is no equal to decoder mode dead in region modes. Avoid constructing regions with decoders which have been flagged as dead.

Quoted 287 lines

---
Changes:
[iweiny: rebase]
[Jonathan: remove dead code]
[Jonathan: clarify commit message]

---
 drivers/cxl/core/cdat.c   |  6 ++--
 drivers/cxl/core/region.c | 75 ++++++++++++++++++++++++++++++++++-------------
 drivers/cxl/cxl.h         | 26 ++++++++++++++--
 3 files changed, 82 insertions(+), 25 deletions(-)
drivers/cxl/core/cdat.c +3 -3
index 438869df241a..bd50bb655741 100644
@@ -571,17 +571,17 @@ static bool dpa_perf_contains(struct cxl_dpa_perf *perf,
 }
 
 static struct cxl_dpa_perf *cxled_get_dpa_perf(struct cxl_endpoint_decoder *cxled,
-					       enum cxl_decoder_mode mode)
+					       enum cxl_region_mode mode)
 {
 	struct cxl_memdev *cxlmd = cxled_to_memdev(cxled);
 	struct cxl_memdev_state *mds = to_cxl_memdev_state(cxlmd->cxlds);
 	struct cxl_dpa_perf *perf;
 
 	switch (mode) {
-	case CXL_DECODER_RAM:
+	case CXL_REGION_RAM:
 		perf = &mds->ram_perf;
 		break;
-	case CXL_DECODER_PMEM:
+	case CXL_REGION_PMEM:
 		perf = &mds->pmem_perf;
 		break;
 	default:
drivers/cxl/core/region.c +55 -20
index e701e4b04032..f3a56003edc1 100644
@@ -144,7 +144,7 @@ static ssize_t uuid_show(struct device *dev, struct device_attribute *attr,
 	rc = down_read_interruptible(&cxl_region_rwsem);
 	if (rc)
 		return rc;
-	if (cxlr->mode != CXL_DECODER_PMEM)
+	if (cxlr->mode != CXL_REGION_PMEM)
 		rc = sysfs_emit(buf, "\n");
 	else
 		rc = sysfs_emit(buf, "%pUb\n", &p->uuid);
@@ -457,7 +457,7 @@ static umode_t cxl_region_visible(struct kobject *kobj, struct attribute *a,
 	 * Support tooling that expects to find a 'uuid' attribute for all
 	 * regions regardless of mode.
 	 */
-	if (a == &dev_attr_uuid.attr && cxlr->mode != CXL_DECODER_PMEM)
+	if (a == &dev_attr_uuid.attr && cxlr->mode != CXL_REGION_PMEM)
 		return 0444;
 	return a->mode;
 }
@@ -620,7 +620,7 @@ static ssize_t mode_show(struct device *dev, struct device_attribute *attr,
 {
 	struct cxl_region *cxlr = to_cxl_region(dev);
 
-	return sysfs_emit(buf, "%s\n", cxl_decoder_mode_name(cxlr->mode));
+	return sysfs_emit(buf, "%s\n", cxl_region_mode_name(cxlr->mode));
 }
 static DEVICE_ATTR_RO(mode);
 
@@ -646,7 +646,7 @@ static int alloc_hpa(struct cxl_region *cxlr, resource_size_t size)
 
 	/* ways, granularity and uuid (if PMEM) need to be set before HPA */
 	if (!p->interleave_ways || !p->interleave_granularity ||
-	    (cxlr->mode == CXL_DECODER_PMEM && uuid_is_null(&p->uuid)))
+	    (cxlr->mode == CXL_REGION_PMEM && uuid_is_null(&p->uuid)))
 		return -ENXIO;
 
 	div64_u64_rem(size, (u64)SZ_256M * p->interleave_ways, &remainder);
@@ -1863,6 +1863,17 @@ static int cxl_region_sort_targets(struct cxl_region *cxlr)
 	return rc;
 }
 
+static bool cxl_modes_compatible(enum cxl_region_mode rmode,
+				 enum cxl_decoder_mode dmode)
+{
+	if (rmode == CXL_REGION_RAM && dmode == CXL_DECODER_RAM)
+		return true;
+	if (rmode == CXL_REGION_PMEM && dmode == CXL_DECODER_PMEM)
+		return true;
+
+	return false;
+}
+
 static int cxl_region_attach(struct cxl_region *cxlr,
 			     struct cxl_endpoint_decoder *cxled, int pos)
 {
@@ -1882,9 +1893,11 @@ static int cxl_region_attach(struct cxl_region *cxlr,
 		return rc;
 	}
 
-	if (cxled->mode != cxlr->mode) {
-		dev_dbg(&cxlr->dev, "%s region mode: %d mismatch: %d\n",
-			dev_name(&cxled->cxld.dev), cxlr->mode, cxled->mode);
+	if (!cxl_modes_compatible(cxlr->mode, cxled->mode)) {
+		dev_dbg(&cxlr->dev, "%s region mode: %s mismatch decoder: %s\n",
+			dev_name(&cxled->cxld.dev),
+			cxl_region_mode_name(cxlr->mode),
+			cxl_decoder_mode_name(cxled->mode));
 		return -EINVAL;
 	}
 
@@ -2446,7 +2459,7 @@ static int cxl_region_calculate_adistance(struct notifier_block *nb,
  * devm_cxl_add_region - Adds a region to a decoder
  * @cxlrd: root decoder
  * @id: memregion id to create, or memregion_free() on failure
- * @mode: mode for the endpoint decoders of this region
+ * @mode: mode of this region
  * @type: select whether this is an expander or accelerator (type-2 or type-3)
  *
  * This is the second step of region initialization. Regions exist within an
@@ -2457,7 +2470,7 @@ static int cxl_region_calculate_adistance(struct notifier_block *nb,
  */
 static struct cxl_region *devm_cxl_add_region(struct cxl_root_decoder *cxlrd,
 					      int id,
-					      enum cxl_decoder_mode mode,
+					      enum cxl_region_mode mode,
 					      enum cxl_decoder_type type)
 {
 	struct cxl_port *port = to_cxl_port(cxlrd->cxlsd.cxld.dev.parent);
@@ -2511,16 +2524,17 @@ static ssize_t create_ram_region_show(struct device *dev,
 }
 
 static struct cxl_region *__create_region(struct cxl_root_decoder *cxlrd,
-					  enum cxl_decoder_mode mode, int id)
+					  enum cxl_region_mode mode, int id)
 {
 	int rc;
 
 	switch (mode) {
-	case CXL_DECODER_RAM:
-	case CXL_DECODER_PMEM:
+	case CXL_REGION_RAM:
+	case CXL_REGION_PMEM:
 		break;
 	default:
-		dev_err(&cxlrd->cxlsd.cxld.dev, "unsupported mode %d\n", mode);
+		dev_err(&cxlrd->cxlsd.cxld.dev, "unsupported mode %s\n",
+			cxl_region_mode_name(mode));
 		return ERR_PTR(-EINVAL);
 	}
 
@@ -2548,7 +2562,7 @@ static ssize_t create_pmem_region_store(struct device *dev,
 	if (rc != 1)
 		return -EINVAL;
 
-	cxlr = __create_region(cxlrd, CXL_DECODER_PMEM, id);
+	cxlr = __create_region(cxlrd, CXL_REGION_PMEM, id);
 	if (IS_ERR(cxlr))
 		return PTR_ERR(cxlr);
 
@@ -2568,7 +2582,7 @@ static ssize_t create_ram_region_store(struct device *dev,
 	if (rc != 1)
 		return -EINVAL;
 
-	cxlr = __create_region(cxlrd, CXL_DECODER_RAM, id);
+	cxlr = __create_region(cxlrd, CXL_REGION_RAM, id);
 	if (IS_ERR(cxlr))
 		return PTR_ERR(cxlr);
 
@@ -3215,6 +3229,22 @@ static int match_region_by_range(struct device *dev, void *data)
 	return rc;
 }
 
+static enum cxl_region_mode
+cxl_decoder_to_region_mode(enum cxl_decoder_mode mode)
+{
+	switch (mode) {
+	case CXL_DECODER_NONE:
+		return CXL_REGION_NONE;
+	case CXL_DECODER_RAM:
+		return CXL_REGION_RAM;
+	case CXL_DECODER_PMEM:
+		return CXL_REGION_PMEM;
+	case CXL_DECODER_MIXED:
+	default:
+		return CXL_REGION_MIXED;
+	}
+}
+
 /* Establish an empty region covering the given HPA range */
 static struct cxl_region *construct_region(struct cxl_root_decoder *cxlrd,
 					   struct cxl_endpoint_decoder *cxled)
@@ -3223,12 +3253,17 @@ static struct cxl_region *construct_region(struct cxl_root_decoder *cxlrd,
 	struct cxl_port *port = cxlrd_to_port(cxlrd);
 	struct range *hpa = &cxled->cxld.hpa_range;
 	struct cxl_region_params *p;
+	enum cxl_region_mode mode;
 	struct cxl_region *cxlr;
 	struct resource *res;
 	int rc;
 
+	if (cxled->mode == CXL_DECODER_DEAD)
+		return ERR_PTR(-EINVAL);
+
+	mode = cxl_decoder_to_region_mode(cxled->mode);
 	do {
-		cxlr = __create_region(cxlrd, cxled->mode,
+		cxlr = __create_region(cxlrd, mode,
 				       atomic_read(&cxlrd->region_id));
 	} while (IS_ERR(cxlr) && PTR_ERR(cxlr) == -EBUSY);
 
@@ -3431,9 +3466,9 @@ static int cxl_region_probe(struct device *dev)
 		return rc;
 
 	switch (cxlr->mode) {
-	case CXL_DECODER_PMEM:
+	case CXL_REGION_PMEM:
 		return devm_cxl_add_pmem_region(cxlr);
-	case CXL_DECODER_RAM:
+	case CXL_REGION_RAM:
 		/*
 		 * The region can not be manged by CXL if any portion of
 		 * it is already online as 'System RAM'
@@ -3445,8 +3480,8 @@ static int cxl_region_probe(struct device *dev)
 			return 0;
 		return devm_cxl_add_dax_region(cxlr);
 	default:
-		dev_dbg(&cxlr->dev, "unsupported region mode: %d\n",
-			cxlr->mode);
+		dev_dbg(&cxlr->dev, "unsupported region mode: %s\n",
+			cxl_region_mode_name(cxlr->mode));
 		return -ENXIO;
 	}
 }
drivers/cxl/cxl.h +24 -2
index 0d8b810a51f0..5d74eb4ffab3 100644
@@ -388,6 +388,27 @@ static inline const char *cxl_decoder_mode_name(enum cxl_decoder_mode mode)
 	return "mixed";
 }
 
+enum cxl_region_mode {
+	CXL_REGION_NONE,
+	CXL_REGION_RAM,
+	CXL_REGION_PMEM,
+	CXL_REGION_MIXED,
+};
+
+static inline const char *cxl_region_mode_name(enum cxl_region_mode mode)
+{
+	static const char * const names[] = {
+		[CXL_REGION_NONE] = "none",
+		[CXL_REGION_RAM] = "ram",
+		[CXL_REGION_PMEM] = "pmem",
+		[CXL_REGION_MIXED] = "mixed",
+	};
+
+	if (mode >= CXL_REGION_NONE && mode <= CXL_REGION_MIXED)
+		return names[mode];
+	return "mixed";
+}
+
 /*
  * Track whether this decoder is reserved for region autodiscovery, or
  * free for userspace provisioning.
@@ -515,7 +536,8 @@ struct cxl_region_params {
  * struct cxl_region - CXL region
  * @dev: This region's device
  * @id: This region's id. Id is globally unique across all regions
- * @mode: Endpoint decoder allocation / access mode
+ * @mode: Region mode which defines which endpoint decoder modes the region is
+ *        compatible with
  * @type: Endpoint decoder target type
  * @cxl_nvb: nvdimm bridge for coordinating @cxlr_pmem setup / shutdown
  * @cxlr_pmem: (for pmem regions) cached copy of the nvdimm bridge
@@ -528,7 +550,7 @@ struct cxl_region_params {
 struct cxl_region {
 	struct device dev;
 	int id;
-	enum cxl_decoder_mode mode;
+	enum cxl_region_mode mode;
 	enum cxl_decoder_type type;
 	struct cxl_nvdimm_bridge *cxl_nvb;
 	struct cxl_pmem_region *cxlr_pmem;
Signature · 2 lines
-- 
2.46.0
Signature · 1 line
Fan Ni
#45 lore raw

PATCH v4 10/28 cxl/region: Add dynamic capacity decoder and region modes

In reply to Ira Weiny

From: Navneet Singh <[email protected]>

One or more decoders each pointing to a Dynamic Capacity (DC) partition form a CXL software region. The region mode reflects composition of that entire software region. Decoder mode reflects a specific DC partition. DC partitions are also known as DC regions per CXL specification r3.1.

Define the new modes and helper functions required to make the association between these new modes.

---
Changes:
[iweiny: keep tags on simple patch]
[Fan: s/partitions/partition/]
[djiang: New wording for the commit message]
[iweiny: reword commit message more]

---
 drivers/cxl/core/region.c |  4 ++++
 drivers/cxl/cxl.h         | 23 +++++++++++++++++++++++
 2 files changed, 27 insertions(+)
drivers/cxl/core/region.c +4 -0
index f3a56003edc1..ab00203f285a 100644
@@ -1870,6 +1870,8 @@ static bool cxl_modes_compatible(enum cxl_region_mode rmode,
 		return true;
 	if (rmode == CXL_REGION_PMEM && dmode == CXL_DECODER_PMEM)
 		return true;
+	if (rmode == CXL_REGION_DC && cxl_decoder_mode_is_dc(dmode))
+		return true;
 
 	return false;
 }
@@ -3239,6 +3241,8 @@ cxl_decoder_to_region_mode(enum cxl_decoder_mode mode)
 		return CXL_REGION_RAM;
 	case CXL_DECODER_PMEM:
 		return CXL_REGION_PMEM;
+	case CXL_DECODER_DC0 ... CXL_DECODER_DC7:
+		return CXL_REGION_DC;
 	case CXL_DECODER_MIXED:
 	default:
 		return CXL_REGION_MIXED;
drivers/cxl/cxl.h +23 -0
index 5d74eb4ffab3..f931ebdd36d0 100644
@@ -370,6 +370,14 @@ enum cxl_decoder_mode {
 	CXL_DECODER_NONE,
 	CXL_DECODER_RAM,
 	CXL_DECODER_PMEM,
+	CXL_DECODER_DC0,
+	CXL_DECODER_DC1,
+	CXL_DECODER_DC2,
+	CXL_DECODER_DC3,
+	CXL_DECODER_DC4,
+	CXL_DECODER_DC5,
+	CXL_DECODER_DC6,
+	CXL_DECODER_DC7,
 	CXL_DECODER_MIXED,
 	CXL_DECODER_DEAD,
 };
@@ -380,6 +388,14 @@ static inline const char *cxl_decoder_mode_name(enum cxl_decoder_mode mode)
 		[CXL_DECODER_NONE] = "none",
 		[CXL_DECODER_RAM] = "ram",
 		[CXL_DECODER_PMEM] = "pmem",
+		[CXL_DECODER_DC0] = "dc0",
+		[CXL_DECODER_DC1] = "dc1",
+		[CXL_DECODER_DC2] = "dc2",
+		[CXL_DECODER_DC3] = "dc3",
+		[CXL_DECODER_DC4] = "dc4",
+		[CXL_DECODER_DC5] = "dc5",
+		[CXL_DECODER_DC6] = "dc6",
+		[CXL_DECODER_DC7] = "dc7",
 		[CXL_DECODER_MIXED] = "mixed",
 	};
 
@@ -388,10 +404,16 @@ static inline const char *cxl_decoder_mode_name(enum cxl_decoder_mode mode)
 	return "mixed";
 }
 
+static inline bool cxl_decoder_mode_is_dc(enum cxl_decoder_mode mode)
+{
+	return (mode >= CXL_DECODER_DC0 && mode <= CXL_DECODER_DC7);
+}
+
 enum cxl_region_mode {
 	CXL_REGION_NONE,
 	CXL_REGION_RAM,
 	CXL_REGION_PMEM,
+	CXL_REGION_DC,
 	CXL_REGION_MIXED,
 };
 
@@ -401,6 +423,7 @@ static inline const char *cxl_region_mode_name(enum cxl_region_mode mode)
 		[CXL_REGION_NONE] = "none",
 		[CXL_REGION_RAM] = "ram",
 		[CXL_REGION_PMEM] = "pmem",
+		[CXL_REGION_DC] = "dc",
 		[CXL_REGION_MIXED] = "mixed",
 	};
 
Signature · 1 line
2.46.0
#46 lore raw

PATCH v4 11/28 cxl/hdm: Add dynamic capacity size support to endpoint decoders

In reply to Ira Weiny

From: Navneet Singh <[email protected]>

To support Dynamic Capacity Devices (DCD) endpoint decoders will need to map DC partitions (regions). In addition to assigning the size of the DC partition, the decoder must assign any skip value from the previous decoder. This must be done within a contiguous DPA space.

Two complications arise with Dynamic Capacity regions which did not exist with Ram and PMEM partitions. First, gaps in the DPA space can exist between and around the DC partitions. Second, the Linux resource tree does not allow a resource to be marked across existing nodes within a tree.

For clarity, below is an example of an 60GB device with 10GB of RAM, 10GB of PMEM and 10GB for each of 2 DC partitions. The desired CXL mapping is 5GB of RAM, 5GB of PMEM, and 5GB of DC1.

     DPA RANGE
     (dpa_res)
0GB        10GB       20GB       30GB       40GB       50GB       60GB
|----------|----------|----------|----------|----------|----------|
RAM         PMEM                  DC0                   DC1
 (ram_res)  (pmem_res)            (dc_res[0])           (dc_res[1])

|----------|----------| <gap> |----------| <gap> |----------|

RAM PMEM DC1

|XXXXX|----|XXXXX|----|----------|----------|----------|XXXXX-----|
0GB   5GB  10GB  15GB 20GB       30GB       40GB       50GB       60GB

The previous skip resource between RAM and PMEM was always a child of the RAM resource and fit nicely [see (S) below]. Because of this simplicity this skip resource reference was not stored in any CXL state. On release the skip range could be calculated based on the endpoint decoders stored values.

Now when DC1 is being mapped 4 skip resources must be created as children. One for the PMEM resource (A), two of the parent DPA resource (B,D), and one more child of the DC0 resource (C).

0GB        10GB       20GB       30GB       40GB       50GB       60GB
|----------|----------|----------|----------|----------|----------|
                           |                     |
|----------|----------|    |     |----------|    |     |----------|
        |          |       |          |          |
       (S)        (A)     (B)        (C)        (D)
	v          v       v          v          v
|XXXXX|----|XXXXX|----|----------|----------|----------|XXXXX-----|
       skip       skip  skip        skip      skip

Expand the calculation of DPA free space and enhance the logic to support this more complex skipping. To track the potential of multiple skip resources an xarray is attached to the endpoint decoder. The existing algorithm between RAM and PMEM is consolidated within the new one to streamline the code even though the result is the storage of a single skip resource in the xarray.

---
Changes:
[djiang: s/skip_res/skip_xa/]

---
 drivers/cxl/core/hdm.c  | 196 ++++++++++++++++++++++++++++++++++++++++++++----
 drivers/cxl/core/port.c |   2 +
 drivers/cxl/cxl.h       |   2 +
 3 files changed, 184 insertions(+), 16 deletions(-)
drivers/cxl/core/hdm.c +180 -16
index 3df10517a327..8c7f941eaba1 100644
@@ -223,6 +223,25 @@ void cxl_dpa_debug(struct seq_file *file, struct cxl_dev_state *cxlds)
 }
 EXPORT_SYMBOL_NS_GPL(cxl_dpa_debug, CXL);
 
+static void cxl_skip_release(struct cxl_endpoint_decoder *cxled)
+{
+	struct cxl_dev_state *cxlds = cxled_to_memdev(cxled)->cxlds;
+	struct cxl_port *port = cxled_to_port(cxled);
+	struct device *dev = &port->dev;
+	unsigned long index;
+	void *entry;
+
+	xa_for_each(&cxled->skip_xa, index, entry) {
+		struct resource *res = entry;
+
+		dev_dbg(dev, "decoder%d.%d: releasing skipped space; %pr\n",
+			port->id, cxled->cxld.id, res);
+		__release_region(&cxlds->dpa_res, res->start,
+				 resource_size(res));
+		xa_erase(&cxled->skip_xa, index);
+	}
+}
+
 /*
  * Must be called in a context that synchronizes against this decoder's
  * port ->remove() callback (like an endpoint decoder sysfs attribute)
@@ -233,15 +252,11 @@ static void __cxl_dpa_release(struct cxl_endpoint_decoder *cxled)
 	struct cxl_port *port = cxled_to_port(cxled);
 	struct cxl_dev_state *cxlds = cxlmd->cxlds;
 	struct resource *res = cxled->dpa_res;
-	resource_size_t skip_start;
 
 	lockdep_assert_held_write(&cxl_dpa_rwsem);
 
-	/* save @skip_start, before @res is released */
-	skip_start = res->start - cxled->skip;
 	__release_region(&cxlds->dpa_res, res->start, resource_size(res));
-	if (cxled->skip)
-		__release_region(&cxlds->dpa_res, skip_start, cxled->skip);
+	cxl_skip_release(cxled);
 	cxled->skip = 0;
 	cxled->dpa_res = NULL;
 	put_device(&cxled->cxld.dev);
@@ -268,6 +283,105 @@ static void devm_cxl_dpa_release(struct cxl_endpoint_decoder *cxled)
 	__cxl_dpa_release(cxled);
 }
 
+static int dc_mode_to_region_index(enum cxl_decoder_mode mode)
+{
+	return mode - CXL_DECODER_DC0;
+}
+
+static int cxl_request_skip(struct cxl_endpoint_decoder *cxled,
+			    resource_size_t skip_base, resource_size_t skip_len)
+{
+	struct cxl_dev_state *cxlds = cxled_to_memdev(cxled)->cxlds;
+	const char *name = dev_name(&cxled->cxld.dev);
+	struct cxl_port *port = cxled_to_port(cxled);
+	struct resource *dpa_res = &cxlds->dpa_res;
+	struct device *dev = &port->dev;
+	struct resource *res;
+	int rc;
+
+	res = __request_region(dpa_res, skip_base, skip_len, name, 0);
+	if (!res)
+		return -EBUSY;
+
+	rc = xa_insert(&cxled->skip_xa, skip_base, res, GFP_KERNEL);
+	if (rc) {
+		__release_region(dpa_res, skip_base, skip_len);
+		return rc;
+	}
+
+	dev_dbg(dev, "decoder%d.%d: skipped space; %pr\n",
+		port->id, cxled->cxld.id, res);
+	return 0;
+}
+
+static int cxl_reserve_dpa_skip(struct cxl_endpoint_decoder *cxled,
+				resource_size_t base, resource_size_t skipped)
+{
+	struct cxl_memdev *cxlmd = cxled_to_memdev(cxled);
+	struct cxl_port *port = cxled_to_port(cxled);
+	struct cxl_dev_state *cxlds = cxlmd->cxlds;
+	resource_size_t skip_base = base - skipped;
+	struct device *dev = &port->dev;
+	resource_size_t skip_len = 0;
+	int rc, index;
+
+	if (resource_size(&cxlds->ram_res) && skip_base <= cxlds->ram_res.end) {
+		skip_len = cxlds->ram_res.end - skip_base + 1;
+		rc = cxl_request_skip(cxled, skip_base, skip_len);
+		if (rc)
+			return rc;
+		skip_base += skip_len;
+	}
+
+	if (skip_base == base) {
+		dev_dbg(dev, "skip done ram!\n");
+		return 0;
+	}
+
+	if (resource_size(&cxlds->pmem_res) &&
+	    skip_base <= cxlds->pmem_res.end) {
+		skip_len = cxlds->pmem_res.end - skip_base + 1;
+		rc = cxl_request_skip(cxled, skip_base, skip_len);
+		if (rc)
+			return rc;
+		skip_base += skip_len;
+	}
+
+	index = dc_mode_to_region_index(cxled->mode);
+	for (int i = 0; i <= index; i++) {
+		struct resource *dcr = &cxlds->dc_res[i];
+
+		if (skip_base < dcr->start) {
+			skip_len = dcr->start - skip_base;
+			rc = cxl_request_skip(cxled, skip_base, skip_len);
+			if (rc)
+				return rc;
+			skip_base += skip_len;
+		}
+
+		if (skip_base == base) {
+			dev_dbg(dev, "skip done DC region %d!\n", i);
+			break;
+		}
+
+		if (resource_size(dcr) && skip_base <= dcr->end) {
+			if (skip_base > base) {
+				dev_err(dev, "Skip error DC region %d; skip_base %pa; base %pa\n",
+					i, &skip_base, &base);
+				return -ENXIO;
+			}
+
+			skip_len = dcr->end - skip_base + 1;
+			rc = cxl_request_skip(cxled, skip_base, skip_len);
+			if (rc)
+				return rc;
+			skip_base += skip_len;
+		}
+	}
+
+	return 0;
+}
+
 static int __cxl_dpa_reserve(struct cxl_endpoint_decoder *cxled,
 			     resource_size_t base, resource_size_t len,
 			     resource_size_t skipped)
@@ -305,13 +419,12 @@ static int __cxl_dpa_reserve(struct cxl_endpoint_decoder *cxled,
 	}
 
 	if (skipped) {
-		res = __request_region(&cxlds->dpa_res, base - skipped, skipped,
-				       dev_name(&cxled->cxld.dev), 0);
-		if (!res) {
-			dev_dbg(dev,
-				"decoder%d.%d: failed to reserve skipped space\n",
-				port->id, cxled->cxld.id);
-			return -EBUSY;
+		int rc = cxl_reserve_dpa_skip(cxled, base, skipped);
+
+		if (rc) {
+			dev_dbg(dev, "decoder%d.%d: failed to reserve skipped space; %pa - %pa\n",
+				port->id, cxled->cxld.id, &base, &skipped);
+			return rc;
 		}
 	}
 	res = __request_region(&cxlds->dpa_res, base, len,
@@ -319,14 +432,20 @@ static int __cxl_dpa_reserve(struct cxl_endpoint_decoder *cxled,
 	if (!res) {
 		dev_dbg(dev, "decoder%d.%d: failed to reserve allocation\n",
 			port->id, cxled->cxld.id);
-		if (skipped)
-			__release_region(&cxlds->dpa_res, base - skipped,
-					 skipped);
+		cxl_skip_release(cxled);
 		return -EBUSY;
 	}
 	cxled->dpa_res = res;
 	cxled->skip = skipped;
 
+	for (int mode = CXL_DECODER_DC0; mode <= CXL_DECODER_DC7; mode++) {
+		int index = dc_mode_to_region_index(mode);
+
+		if (resource_contains(&cxlds->dc_res[index], res)) {
+			cxled->mode = mode;
+			goto success;
+		}
+	}
 	if (resource_contains(&cxlds->pmem_res, res))
 		cxled->mode = CXL_DECODER_PMEM;
 	else if (resource_contains(&cxlds->ram_res, res))
@@ -337,6 +456,9 @@ static int __cxl_dpa_reserve(struct cxl_endpoint_decoder *cxled,
 		cxled->mode = CXL_DECODER_MIXED;
 	}
 
+success:
+	dev_dbg(dev, "decoder%d.%d: %pr mode: %d\n", port->id, cxled->cxld.id,
+		cxled->dpa_res, cxled->mode);
 	port->hdm_end++;
 	get_device(&cxled->cxld.dev);
 	return 0;
@@ -466,8 +588,8 @@ int cxl_dpa_set_mode(struct cxl_endpoint_decoder *cxled,
 
 int cxl_dpa_alloc(struct cxl_endpoint_decoder *cxled, unsigned long long size)
 {
-	struct cxl_memdev *cxlmd = cxled_to_memdev(cxled);
 	resource_size_t free_ram_start, free_pmem_start;
+	struct cxl_memdev *cxlmd = cxled_to_memdev(cxled);
 	struct cxl_port *port = cxled_to_port(cxled);
 	struct cxl_dev_state *cxlds = cxlmd->cxlds;
 	struct device *dev = &cxled->cxld.dev;
@@ -524,12 +646,54 @@ int cxl_dpa_alloc(struct cxl_endpoint_decoder *cxled, unsigned long long size)
 		else
 			skip_end = start - 1;
 		skip = skip_end - skip_start + 1;
+	} else if (cxl_decoder_mode_is_dc(cxled->mode)) {
+		int dc_index = dc_mode_to_region_index(cxled->mode);
+
+		for (p = cxlds->dc_res[dc_index].child, last = NULL; p; p = p->sibling)
+			last = p;
+
+		if (last) {
+			/*
+			 * Some capacity in this DC partition is already allocated,
+			 * that allocation already handled the skip.
+			 */
+			start = last->end + 1;
+			skip = 0;
+		} else {
+			/* Calculate skip */
+			resource_size_t skip_start, skip_end;
+
+			start = cxlds->dc_res[dc_index].start;
+
+			if ((resource_size(&cxlds->pmem_res) == 0) || !cxlds->pmem_res.child)
+				skip_start = free_ram_start;
+			else
+				skip_start = free_pmem_start;
+			/*
+			 * If any dc region is already mapped, then that allocation
+			 * already handled the RAM and PMEM skip.  Check for DC region
+			 * skip.
+			 */
+			for (int i = dc_index - 1; i >= 0 ; i--) {
+				if (cxlds->dc_res[i].child) {
+					skip_start = cxlds->dc_res[i].child->end + 1;
+					break;
+				}
+			}
+
+			skip_end = start - 1;
+			skip = skip_end - skip_start + 1;
+		}
+		avail = cxlds->dc_res[dc_index].end - start + 1;
 	} else {
 		dev_dbg(dev, "mode not set\n");
 		rc = -EINVAL;
 		goto out;
 	}
 
+	dev_dbg(dev, "DPA Allocation start: %pa len: %#llx Skip: %pa\n",
+		&start, size, &skip);
+
 	if (size > avail) {
 		dev_dbg(dev, "%pa exceeds available %s capacity: %pa\n", &size,
 			cxl_decoder_mode_name(cxled->mode), &avail);
drivers/cxl/core/port.c +2 -0
index e666ec6a9085..85b912c11f04 100644
@@ -419,6 +419,7 @@ static void cxl_endpoint_decoder_release(struct device *dev)
 	struct cxl_endpoint_decoder *cxled = to_cxl_endpoint_decoder(dev);
 
 	__cxl_decoder_release(&cxled->cxld);
+	xa_destroy(&cxled->skip_xa);
 	kfree(cxled);
 }
 
@@ -1899,6 +1900,7 @@ struct cxl_endpoint_decoder *cxl_endpoint_decoder_alloc(struct cxl_port *port)
 		return ERR_PTR(-ENOMEM);
 
 	cxled->pos = -1;
+	xa_init(&cxled->skip_xa);
 	cxld = &cxled->cxld;
 	rc = cxl_decoder_init(port, cxld);
 	if (rc)	 {
drivers/cxl/cxl.h +2 -0
index f931ebdd36d0..8b7099c38a40 100644
@@ -446,6 +446,7 @@ enum cxl_decoder_state {
  * @cxld: base cxl_decoder_object
  * @dpa_res: actively claimed DPA span of this decoder
  * @skip: offset into @dpa_res where @cxld.hpa_range maps
+ * @skip_xa: array of skipped resources from the previous decoder end
  * @mode: which memory type / access-mode-partition this decoder targets
  * @state: autodiscovery state
  * @pos: interleave position in @cxld.region
@@ -454,6 +455,7 @@ struct cxl_endpoint_decoder {
 	struct cxl_decoder cxld;
 	struct resource *dpa_res;
 	resource_size_t skip;
+	struct xarray skip_xa;
 	enum cxl_decoder_mode mode;
 	enum cxl_decoder_state state;
 	int pos;
Signature · 1 line
2.46.0
Reviewed-by on v4 11/28
#47 lore raw

Re: [PATCH v4 11/28] cxl/hdm: Add dynamic capacity size support to endpoint decoders

Quoted 57 lines · On Mon, 07 Oct 2024 18:16:17 -0500 [email protected] wrote:

From: Navneet Singh <[email protected]>

To support Dynamic Capacity Devices (DCD) endpoint decoders will need to map DC partitions (regions). In addition to assigning the size of the DC partition, the decoder must assign any skip value from the previous decoder. This must be done within a contiguous DPA space.

Two complications arise with Dynamic Capacity regions which did not exist with Ram and PMEM partitions. First, gaps in the DPA space can exist between and around the DC partitions. Second, the Linux resource tree does not allow a resource to be marked across existing nodes within a tree.

For clarity, below is an example of an 60GB device with 10GB of RAM, 10GB of PMEM and 10GB for each of 2 DC partitions. The desired CXL mapping is 5GB of RAM, 5GB of PMEM, and 5GB of DC1.

     DPA RANGE
     (dpa_res)
0GB        10GB       20GB       30GB       40GB       50GB       60GB
|----------|----------|----------|----------|----------|----------|
RAM         PMEM                  DC0                   DC1
 (ram_res)  (pmem_res)            (dc_res[0])           (dc_res[1])

|----------|----------| <gap> |----------| <gap> |----------|

RAM PMEM DC1

|XXXXX|----|XXXXX|----|----------|----------|----------|XXXXX-----|
0GB   5GB  10GB  15GB 20GB       30GB       40GB       50GB       60GB

The previous skip resource between RAM and PMEM was always a child of the RAM resource and fit nicely [see (S) below]. Because of this simplicity this skip resource reference was not stored in any CXL state. On release the skip range could be calculated based on the endpoint decoders stored values.

Now when DC1 is being mapped 4 skip resources must be created as children. One for the PMEM resource (A), two of the parent DPA resource (B,D), and one more child of the DC0 resource (C).

0GB        10GB       20GB       30GB       40GB       50GB       60GB
|----------|----------|----------|----------|----------|----------|
                           |                     |
|----------|----------|    |     |----------|    |     |----------|
        |          |       |          |          |
       (S)        (A)     (B)        (C)        (D)
	v          v       v          v          v
|XXXXX|----|XXXXX|----|----------|----------|----------|XXXXX-----|
       skip       skip  skip        skip      skip

Expand the calculation of DPA free space and enhance the logic to support this more complex skipping. To track the potential of multiple skip resources an xarray is attached to the endpoint decoder. The existing algorithm between RAM and PMEM is consolidated within the new one to streamline the code even though the result is the storage of a single skip resource in the xarray.

Quoted 3 lines

One trivial comment inline.

Quoted 18 lines

---
Changes:
[djiang: s/skip_res/skip_xa/]

---
 drivers/cxl/core/hdm.c  | 196 ++++++++++++++++++++++++++++++++++++++++++++----
 drivers/cxl/core/port.c |   2 +
 drivers/cxl/cxl.h       |   2 +
 3 files changed, 184 insertions(+), 16 deletions(-)
drivers/cxl/core/hdm.c +0 -0
index 3df10517a327..8c7f941eaba1 100644
@@ -223,6 +223,25 @@ void cxl_dpa_debug(struct seq_file *file, struct cxl_dev_state *cxlds)
 }
 EXPORT_SYMBOL_NS_GPL(cxl_dpa_debug, CXL);
 

drivers/cxl/core/hdm.c void cxl_dpa_debug(struct seq_file *file, struct cxl_dev_state *cxlds) · line 235

Quoted 10 lines
drivers/cxl/core/hdm.c +10 -0
+static void cxl_skip_release(struct cxl_endpoint_decoder *cxled)
+{
+	struct cxl_dev_state *cxlds = cxled_to_memdev(cxled)->cxlds;
+	struct cxl_port *port = cxled_to_port(cxled);
+	struct device *dev = &port->dev;
+	unsigned long index;
+	void *entry;
+
+	xa_for_each(&cxled->skip_xa, index, entry) {
+		struct resource *res = entry;
	struct resource *res;

	xa_for_each(&cxled->skip_xa, index, res) {

as can always cast form a pointer to a void * and avoiding the extra local variable is a nice to have.

Quoted 8 lines
drivers/cxl/core/hdm.c +8 -0
+
+		dev_dbg(dev, "decoder%d.%d: releasing skipped space; %pr\n",
+			port->id, cxled->cxld.id, res);
+		__release_region(&cxlds->dpa_res, res->start,
+				 resource_size(res));
+		xa_erase(&cxled->skip_xa, index);
+	}
+}
author
#48 lore raw

PATCH v4 12/28 cxl/cdat: Gather DSMAS data for DCD regions

In reply to Ira Weiny

Additional DCD region (partition) information is contained in the DSMAS CDAT tables, including performance, read only, and shareable attributes.

Match DCD partitions with DSMAS tables and store the meta data.

To: Robert Moore <[email protected]>
To: Rafael J. Wysocki <[email protected]>
To: Len Brown <[email protected]>

---
Changes:
[iweiny: new patch]
[iweiny: Gather shareable/read-only flags for later use]

---
 drivers/cxl/core/cdat.c | 38 ++++++++++++++++++++++++++++++++++++++
 drivers/cxl/core/mbox.c |  2 ++
 drivers/cxl/cxlmem.h    |  3 +++
 include/acpi/actbl1.h   |  2 ++
 4 files changed, 45 insertions(+)
drivers/cxl/core/cdat.c +38 -0
index bd50bb655741..9b2f717a16e5 100644
@@ -17,6 +17,8 @@ struct dsmas_entry {
 	struct access_coordinate cdat_coord[ACCESS_COORDINATE_MAX];
 	int entries;
 	int qos_class;
+	bool shareable;
+	bool read_only;
 };
 
 static u32 cdat_normalize(u16 entry, u64 base, u8 type)
@@ -74,6 +76,8 @@ static int cdat_dsmas_handler(union acpi_subtable_headers *header, void *arg,
 		return -ENOMEM;
 
 	dent->handle = dsmas->dsmad_handle;
+	dent->shareable = dsmas->flags & ACPI_CDAT_DSMAS_SHAREABLE;
+	dent->read_only = dsmas->flags & ACPI_CDAT_DSMAS_READ_ONLY;
 	dent->dpa_range.start = le64_to_cpu((__force __le64)dsmas->dpa_base_address);
 	dent->dpa_range.end = le64_to_cpu((__force __le64)dsmas->dpa_base_address) +
 			      le64_to_cpu((__force __le64)dsmas->dpa_length) - 1;
@@ -255,6 +259,38 @@ static void update_perf_entry(struct device *dev, struct dsmas_entry *dent,
 		dent->coord[ACCESS_COORDINATE_CPU].write_latency);
 }
 
+
+static void update_dcd_perf(struct cxl_dev_state *cxlds,
+			    struct dsmas_entry *dent)
+{
+	struct cxl_memdev_state *mds = to_cxl_memdev_state(cxlds);
+	struct device *dev = cxlds->dev;
+
+	for (int i = 0; i < mds->nr_dc_region; i++) {
+		/* CXL defines a u32 handle while cdat defines u8, ignore upper bits */
+		u8 dc_handle = mds->dc_region[i].dsmad_handle & 0xff;
+
+		if (resource_size(&cxlds->dc_res[i])) {
+			struct range dc_range = {
+				.start = cxlds->dc_res[i].start,
+				.end = cxlds->dc_res[i].end,
+			};
+
+			if (range_contains(&dent->dpa_range, &dc_range)) {
+				if (dent->handle != dc_handle)
+					dev_warn(dev, "DC Region/DSMAS mis-matched handle/range; region %pra (%u); dsmas %pra (%u)\n"
+						      "   setting DC region attributes regardless\n",
+						&dent->dpa_range, dent->handle,
+						&dc_range, dc_handle);
+
+				mds->dc_region[i].shareable = dent->shareable;
+				mds->dc_region[i].read_only = dent->read_only;
+				update_perf_entry(dev, dent, &mds->dc_perf[i]);
+			}
+		}
+	}
+}
+
 static void cxl_memdev_set_qos_class(struct cxl_dev_state *cxlds,
 				     struct xarray *dsmas_xa)
 {
@@ -278,6 +314,8 @@ static void cxl_memdev_set_qos_class(struct cxl_dev_state *cxlds,
 		else if (resource_size(&cxlds->pmem_res) &&
 			 range_contains(&pmem_range, &dent->dpa_range))
 			update_perf_entry(dev, dent, &mds->pmem_perf);
+		else if (cxl_dcd_supported(mds))
+			update_dcd_perf(cxlds, dent);
 		else
 			dev_dbg(dev, "no partition for dsmas dpa: %pra\n",
 				&dent->dpa_range);
drivers/cxl/core/mbox.c +2 -0
index 4b51ddd1ff94..3ba465823564 100644
@@ -1649,6 +1649,8 @@ struct cxl_memdev_state *cxl_memdev_state_create(struct device *dev)
 	mds->cxlds.type = CXL_DEVTYPE_CLASSMEM;
 	mds->ram_perf.qos_class = CXL_QOS_CLASS_INVALID;
 	mds->pmem_perf.qos_class = CXL_QOS_CLASS_INVALID;
+	for (int i = 0; i < CXL_MAX_DC_REGION; i++)
+		mds->dc_perf[i].qos_class = CXL_QOS_CLASS_INVALID;
 
 	return mds;
 }
drivers/cxl/cxlmem.h +3 -0
index 0690b917b1e0..c3b889a586d8 100644
@@ -466,6 +466,8 @@ struct cxl_dc_region_info {
 	u64 blk_size;
 	u32 dsmad_handle;
 	u8 flags;
+	bool shareable;
+	bool read_only;
 	u8 name[CXL_DC_REGION_STRLEN];
 };
 
@@ -533,6 +535,7 @@ struct cxl_memdev_state {
 
 	u8 nr_dc_region;
 	struct cxl_dc_region_info dc_region[CXL_MAX_DC_REGION];
+	struct cxl_dpa_perf dc_perf[CXL_MAX_DC_REGION];
 
 	struct cxl_event_state event;
 	struct cxl_poison_state poison;
include/acpi/actbl1.h +2 -0
index 199afc2cd122..387fc821703a 100644
@@ -403,6 +403,8 @@ struct acpi_cdat_dsmas {
 /* Flags for subtable above */
 
 #define ACPI_CDAT_DSMAS_NON_VOLATILE        (1 << 2)
+#define ACPI_CDAT_DSMAS_SHAREABLE           (1 << 3)
+#define ACPI_CDAT_DSMAS_READ_ONLY           (1 << 6)
 
 /* Subtable 1: Device scoped Latency and Bandwidth Information Structure (DSLBIS) */
 
Signature · 1 line
2.46.0
#49 lore raw

Re: [PATCH v4 12/28] cxl/cdat: Gather DSMAS data for DCD regions

Quoted 133 lines · Ira Weiny

Additional DCD region (partition) information is contained in the DSMAS CDAT tables, including performance, read only, and shareable attributes.

Match DCD partitions with DSMAS tables and store the meta data.

To: Robert Moore <[email protected]>
To: Rafael J. Wysocki <[email protected]>
To: Len Brown <[email protected]>

---
Changes:
[iweiny: new patch]
[iweiny: Gather shareable/read-only flags for later use]

---
 drivers/cxl/core/cdat.c | 38 ++++++++++++++++++++++++++++++++++++++
 drivers/cxl/core/mbox.c |  2 ++
 drivers/cxl/cxlmem.h    |  3 +++
 include/acpi/actbl1.h   |  2 ++
 4 files changed, 45 insertions(+)
drivers/cxl/core/cdat.c +38 -0
index bd50bb655741..9b2f717a16e5 100644
@@ -17,6 +17,8 @@ struct dsmas_entry {
        struct access_coordinate cdat_coord[ACCESS_COORDINATE_MAX];
        int entries;
        int qos_class;
+       bool shareable;
+       bool read_only;
 };
 
 static u32 cdat_normalize(u16 entry, u64 base, u8 type)
@@ -74,6 +76,8 @@ static int cdat_dsmas_handler(union acpi_subtable_headers *header, void *arg,
                return -ENOMEM;
 
        dent->handle = dsmas->dsmad_handle;
+       dent->shareable = dsmas->flags & ACPI_CDAT_DSMAS_SHAREABLE;
+       dent->read_only = dsmas->flags & ACPI_CDAT_DSMAS_READ_ONLY;
        dent->dpa_range.start = le64_to_cpu((__force __le64)dsmas->dpa_base_address);
        dent->dpa_range.end = le64_to_cpu((__force __le64)dsmas->dpa_base_address) +
                              le64_to_cpu((__force __le64)dsmas->dpa_length) - 1;
@@ -255,6 +259,38 @@ static void update_perf_entry(struct device *dev, struct dsmas_entry *dent,
                dent->coord[ACCESS_COORDINATE_CPU].write_latency);
 }
 
+
+static void update_dcd_perf(struct cxl_dev_state *cxlds,
+                           struct dsmas_entry *dent)
+{
+       struct cxl_memdev_state *mds = to_cxl_memdev_state(cxlds);
+       struct device *dev = cxlds->dev;
+
+       for (int i = 0; i < mds->nr_dc_region; i++) {
+               /* CXL defines a u32 handle while cdat defines u8, ignore upper bits */
+               u8 dc_handle = mds->dc_region[i].dsmad_handle & 0xff;
+
+               if (resource_size(&cxlds->dc_res[i])) {
+                       struct range dc_range = {
+                               .start = cxlds->dc_res[i].start,
+                               .end = cxlds->dc_res[i].end,
+                       };
+
+                       if (range_contains(&dent->dpa_range, &dc_range)) {
+                               if (dent->handle != dc_handle)
+                                       dev_warn(dev, "DC Region/DSMAS mis-matched handle/range; region %pra (%u); dsmas %pra (%u)\n"
+                                                     "   setting DC region attributes regardless\n",
+                                               &dent->dpa_range, dent->handle,
+                                               &dc_range, dc_handle);
+
+                               mds->dc_region[i].shareable = dent->shareable;
+                               mds->dc_region[i].read_only = dent->read_only;
+                               update_perf_entry(dev, dent, &mds->dc_perf[i]);
+                       }
+               }
+       }
+}
+
 static void cxl_memdev_set_qos_class(struct cxl_dev_state *cxlds,
                                     struct xarray *dsmas_xa)
 {
@@ -278,6 +314,8 @@ static void cxl_memdev_set_qos_class(struct cxl_dev_state *cxlds,
                else if (resource_size(&cxlds->pmem_res) &&
                         range_contains(&pmem_range, &dent->dpa_range))
                        update_perf_entry(dev, dent, &mds->pmem_perf);
+               else if (cxl_dcd_supported(mds))
+                       update_dcd_perf(cxlds, dent);
                else
                        dev_dbg(dev, "no partition for dsmas dpa: %pra\n",
                                &dent->dpa_range);
drivers/cxl/core/mbox.c +2 -0
index 4b51ddd1ff94..3ba465823564 100644
@@ -1649,6 +1649,8 @@ struct cxl_memdev_state *cxl_memdev_state_create(struct device *dev)
        mds->cxlds.type = CXL_DEVTYPE_CLASSMEM;
        mds->ram_perf.qos_class = CXL_QOS_CLASS_INVALID;
        mds->pmem_perf.qos_class = CXL_QOS_CLASS_INVALID;
+       for (int i = 0; i < CXL_MAX_DC_REGION; i++)
+               mds->dc_perf[i].qos_class = CXL_QOS_CLASS_INVALID;
 
        return mds;
 }
drivers/cxl/cxlmem.h +3 -0
index 0690b917b1e0..c3b889a586d8 100644
@@ -466,6 +466,8 @@ struct cxl_dc_region_info {
        u64 blk_size;
        u32 dsmad_handle;
        u8 flags;
+       bool shareable;
+       bool read_only;
        u8 name[CXL_DC_REGION_STRLEN];
 };
 
@@ -533,6 +535,7 @@ struct cxl_memdev_state {
 
        u8 nr_dc_region;
        struct cxl_dc_region_info dc_region[CXL_MAX_DC_REGION];
+       struct cxl_dpa_perf dc_perf[CXL_MAX_DC_REGION];
 
        struct cxl_event_state event;
        struct cxl_poison_state poison;
include/acpi/actbl1.h +0 -0
index 199afc2cd122..387fc821703a 100644

include/acpi/actbl1.h struct acpi_cdat_dsmas { · line 410

Quoted 8 lines
include/acpi/actbl1.h +2 -0
@@ -403,6 +403,8 @@ struct acpi_cdat_dsmas {
 /* Flags for subtable above */
 
 #define ACPI_CDAT_DSMAS_NON_VOLATILE        (1 << 2)
+#define ACPI_CDAT_DSMAS_SHAREABLE           (1 << 3)
+#define ACPI_CDAT_DSMAS_READ_ONLY           (1 << 6)
 
 /* Subtable 1: Device scoped Latency and Bandwidth Information Structure (DSLBIS) */
 

Is there an upstream ACPICA commit for this?

author
#50 lore raw

Re: [PATCH v4 12/28] cxl/cdat: Gather DSMAS data for DCD regions

Quoted 2 lines · Rafael J. Wysocki
Quoted 1 line · Ira Weiny

[snip]

Quoted 7 lines
Quoted 7 lines
include/acpi/actbl1.h +0 -0
index 199afc2cd122..387fc821703a 100644
@@ -403,6 +403,8 @@ struct acpi_cdat_dsmas {
 /* Flags for subtable above */
 
Quoted 8 lines

include/acpi/actbl1.h struct acpi_cdat_dsmas { · line 410

Quoted 6 lines
include/acpi/actbl1.h +2 -0
 #define ACPI_CDAT_DSMAS_NON_VOLATILE        (1 << 2)
+#define ACPI_CDAT_DSMAS_SHAREABLE           (1 << 3)
+#define ACPI_CDAT_DSMAS_READ_ONLY           (1 << 6)
 
 /* Subtable 1: Device scoped Latency and Bandwidth Information Structure (DSLBIS) */
 

Is there an upstream ACPICA commit for this?

There is a PR for it now.

	https://github.com/acpica/acpica/pull/976

Do I need to reference that in this patch? Or wait for it to be merged and drop this hunk?

Ira

#51 lore raw

Re: [PATCH v4 12/28] cxl/cdat: Gather DSMAS data for DCD regions

Quoted 18 lines · Ira Weiny
Quoted 1 line · Rafael J. Wysocki

On Tue, Oct 8, 2024 at 1:17 AM Ira Weiny <[email protected]> wrote:

[snip]

Quoted 7 lines
Quoted 7 lines
include/acpi/actbl1.h +0 -0
index 199afc2cd122..387fc821703a 100644
@@ -403,6 +403,8 @@ struct acpi_cdat_dsmas {
  /* Flags for subtable above */
 
Quoted 7 lines

include/acpi/actbl1.h struct acpi_cdat_dsmas { · line 410

Quoted 6 lines
include/acpi/actbl1.h +2 -0
  #define ACPI_CDAT_DSMAS_NON_VOLATILE        (1 << 2)
+#define ACPI_CDAT_DSMAS_SHAREABLE           (1 << 3)
+#define ACPI_CDAT_DSMAS_READ_ONLY           (1 << 6)
 
  /* Subtable 1: Device scoped Latency and Bandwidth Information Structure (DSLBIS) */
 

Is there an upstream ACPICA commit for this?

Quoted 6 lines

There is a PR for it now.

	https://github.com/acpica/acpica/pull/976

Do I need to reference that in this patch? Or wait for it to be merged and drop this hunk?

Wait for it to be merged first.  Then either drop this hunk and wait for an ACPICA release (that may not happen soon, though), or send a Linux patch corresponding to it with a Link tag pointing to the above.

Thanks!

#52 lore raw

Re: [PATCH v4 12/28] cxl/cdat: Gather DSMAS data for DCD regions

In reply to Ira Weiny
Quoted 12 lines · Ira Weiny

Additional DCD region (partition) information is contained in the DSMAS CDAT tables, including performance, read only, and shareable attributes.

Match DCD partitions with DSMAS tables and store the meta data.

To: Robert Moore <[email protected]>
To: Rafael J. Wysocki <[email protected]>
To: Len Brown <[email protected]>

One minor comment inline.

Quoted 34 lines

---
Changes:
[iweiny: new patch]
[iweiny: Gather shareable/read-only flags for later use]

---
 drivers/cxl/core/cdat.c | 38 ++++++++++++++++++++++++++++++++++++++
 drivers/cxl/core/mbox.c |  2 ++
 drivers/cxl/cxlmem.h    |  3 +++
 include/acpi/actbl1.h   |  2 ++
 4 files changed, 45 insertions(+)
drivers/cxl/core/cdat.c +4 -0
index bd50bb655741..9b2f717a16e5 100644
@@ -17,6 +17,8 @@ struct dsmas_entry {
 	struct access_coordinate cdat_coord[ACCESS_COORDINATE_MAX];
 	int entries;
 	int qos_class;
+	bool shareable;
+	bool read_only;
 };
 
 static u32 cdat_normalize(u16 entry, u64 base, u8 type)
@@ -74,6 +76,8 @@ static int cdat_dsmas_handler(union acpi_subtable_headers *header, void *arg,
 		return -ENOMEM;
 
 	dent->handle = dsmas->dsmad_handle;
+	dent->shareable = dsmas->flags & ACPI_CDAT_DSMAS_SHAREABLE;
+	dent->read_only = dsmas->flags & ACPI_CDAT_DSMAS_READ_ONLY;
 	dent->dpa_range.start = le64_to_cpu((__force __le64)dsmas->dpa_base_address);
 	dent->dpa_range.end = le64_to_cpu((__force __le64)dsmas->dpa_base_address) +
 			      le64_to_cpu((__force __le64)dsmas->dpa_length) - 1;

drivers/cxl/core/cdat.c static void update_perf_entry(struct device *dev, struct dsmas_entry *dent, · line 262

Quoted 4 lines
drivers/cxl/core/cdat.c +1 -0
@@ -255,6 +259,38 @@ static void update_perf_entry(struct device *dev, struct dsmas_entry *dent,
 		dent->coord[ACCESS_COORDINATE_CPU].write_latency);
 }
 
+

Unwanted blank line.

Fan

Quoted 94 lines
drivers/cxl/core/cdat.c +33 -0
+static void update_dcd_perf(struct cxl_dev_state *cxlds,
+			    struct dsmas_entry *dent)
+{
+	struct cxl_memdev_state *mds = to_cxl_memdev_state(cxlds);
+	struct device *dev = cxlds->dev;
+
+	for (int i = 0; i < mds->nr_dc_region; i++) {
+		/* CXL defines a u32 handle while cdat defines u8, ignore upper bits */
+		u8 dc_handle = mds->dc_region[i].dsmad_handle & 0xff;
+
+		if (resource_size(&cxlds->dc_res[i])) {
+			struct range dc_range = {
+				.start = cxlds->dc_res[i].start,
+				.end = cxlds->dc_res[i].end,
+			};
+
+			if (range_contains(&dent->dpa_range, &dc_range)) {
+				if (dent->handle != dc_handle)
+					dev_warn(dev, "DC Region/DSMAS mis-matched handle/range; region %pra (%u); dsmas %pra (%u)\n"
+						      "   setting DC region attributes regardless\n",
+						&dent->dpa_range, dent->handle,
+						&dc_range, dc_handle);
+
+				mds->dc_region[i].shareable = dent->shareable;
+				mds->dc_region[i].read_only = dent->read_only;
+				update_perf_entry(dev, dent, &mds->dc_perf[i]);
+			}
+		}
+	}
+}
+
 static void cxl_memdev_set_qos_class(struct cxl_dev_state *cxlds,
 				     struct xarray *dsmas_xa)
 {
@@ -278,6 +314,8 @@ static void cxl_memdev_set_qos_class(struct cxl_dev_state *cxlds,
 		else if (resource_size(&cxlds->pmem_res) &&
 			 range_contains(&pmem_range, &dent->dpa_range))
 			update_perf_entry(dev, dent, &mds->pmem_perf);
+		else if (cxl_dcd_supported(mds))
+			update_dcd_perf(cxlds, dent);
 		else
 			dev_dbg(dev, "no partition for dsmas dpa: %pra\n",
 				&dent->dpa_range);
drivers/cxl/core/mbox.c +2 -0
index 4b51ddd1ff94..3ba465823564 100644
@@ -1649,6 +1649,8 @@ struct cxl_memdev_state *cxl_memdev_state_create(struct device *dev)
 	mds->cxlds.type = CXL_DEVTYPE_CLASSMEM;
 	mds->ram_perf.qos_class = CXL_QOS_CLASS_INVALID;
 	mds->pmem_perf.qos_class = CXL_QOS_CLASS_INVALID;
+	for (int i = 0; i < CXL_MAX_DC_REGION; i++)
+		mds->dc_perf[i].qos_class = CXL_QOS_CLASS_INVALID;
 
 	return mds;
 }
drivers/cxl/cxlmem.h +3 -0
index 0690b917b1e0..c3b889a586d8 100644
@@ -466,6 +466,8 @@ struct cxl_dc_region_info {
 	u64 blk_size;
 	u32 dsmad_handle;
 	u8 flags;
+	bool shareable;
+	bool read_only;
 	u8 name[CXL_DC_REGION_STRLEN];
 };
 
@@ -533,6 +535,7 @@ struct cxl_memdev_state {
 
 	u8 nr_dc_region;
 	struct cxl_dc_region_info dc_region[CXL_MAX_DC_REGION];
+	struct cxl_dpa_perf dc_perf[CXL_MAX_DC_REGION];
 
 	struct cxl_event_state event;
 	struct cxl_poison_state poison;
include/acpi/actbl1.h +2 -0
index 199afc2cd122..387fc821703a 100644
@@ -403,6 +403,8 @@ struct acpi_cdat_dsmas {
 /* Flags for subtable above */
 
 #define ACPI_CDAT_DSMAS_NON_VOLATILE        (1 << 2)
+#define ACPI_CDAT_DSMAS_SHAREABLE           (1 << 3)
+#define ACPI_CDAT_DSMAS_READ_ONLY           (1 << 6)
 
 /* Subtable 1: Device scoped Latency and Bandwidth Information Structure (DSLBIS) */
 
Signature · 2 lines
-- 
2.46.0
Signature · 1 line
Fan Ni
author
#53 lore raw

Re: [PATCH v4 12/28] cxl/cdat: Gather DSMAS data for DCD regions

Quoted 53 lines · Fan Ni
Quoted 12 lines · Ira Weiny

Additional DCD region (partition) information is contained in the DSMAS CDAT tables, including performance, read only, and shareable attributes.

Match DCD partitions with DSMAS tables and store the meta data.

To: Robert Moore <[email protected]>
To: Rafael J. Wysocki <[email protected]>
To: Len Brown <[email protected]>

One minor comment inline.

Quoted 37 lines

---
Changes:
[iweiny: new patch]
[iweiny: Gather shareable/read-only flags for later use]

---
 drivers/cxl/core/cdat.c | 38 ++++++++++++++++++++++++++++++++++++++
 drivers/cxl/core/mbox.c |  2 ++
 drivers/cxl/cxlmem.h    |  3 +++
 include/acpi/actbl1.h   |  2 ++
 4 files changed, 45 insertions(+)
drivers/cxl/core/cdat.c +4 -0
index bd50bb655741..9b2f717a16e5 100644
@@ -17,6 +17,8 @@ struct dsmas_entry {
 	struct access_coordinate cdat_coord[ACCESS_COORDINATE_MAX];
 	int entries;
 	int qos_class;
+	bool shareable;
+	bool read_only;
 };
 
 static u32 cdat_normalize(u16 entry, u64 base, u8 type)
@@ -74,6 +76,8 @@ static int cdat_dsmas_handler(union acpi_subtable_headers *header, void *arg,
 		return -ENOMEM;
 
 	dent->handle = dsmas->dsmad_handle;
+	dent->shareable = dsmas->flags & ACPI_CDAT_DSMAS_SHAREABLE;
+	dent->read_only = dsmas->flags & ACPI_CDAT_DSMAS_READ_ONLY;
 	dent->dpa_range.start = le64_to_cpu((__force __le64)dsmas->dpa_base_address);
 	dent->dpa_range.end = le64_to_cpu((__force __le64)dsmas->dpa_base_address) +
 			      le64_to_cpu((__force __le64)dsmas->dpa_length) - 1;
@@ -255,6 +259,38 @@ static void update_perf_entry(struct device *dev, struct dsmas_entry *dent,
 		dent->coord[ACCESS_COORDINATE_CPU].write_latency);
 }
 
Quoted 2 lines

drivers/cxl/core/cdat.c static void update_perf_entry(struct device *dev, struct dsmas_entry *dent, · line 262

Quoted 1 line
drivers/cxl/core/cdat.c +1 -0
+

Unwanted blank line.

Fixed. Thanks.
Ira

Reviewed-by on v4 12/28
#54 lore raw

Re: [PATCH v4 12/28] cxl/cdat: Gather DSMAS data for DCD regions

In reply to Ira Weiny
Quoted 11 lines · On Mon, 07 Oct 2024 18:16:18 -0500 Ira Weiny <[email protected]> wrote:

Additional DCD region (partition) information is contained in the DSMAS CDAT tables, including performance, read only, and shareable attributes.

Match DCD partitions with DSMAS tables and store the meta data.

To: Robert Moore <[email protected]>
To: Rafael J. Wysocki <[email protected]>
To: Len Brown <[email protected]>

One trivial comment from me.
As Rafael has raised, the ACPICA dependency in here is going to be the blocker :(

drivers/cxl/core/cdat.c static void update_perf_entry(struct device *dev, struct dsmas_entry *dent, · line 270

Quoted 8 lines
drivers/cxl/core/cdat.c +8 -0
+static void update_dcd_perf(struct cxl_dev_state *cxlds,
+			    struct dsmas_entry *dent)
+{
+	struct cxl_memdev_state *mds = to_cxl_memdev_state(cxlds);
+	struct device *dev = cxlds->dev;
+
+	for (int i = 0; i < mds->nr_dc_region; i++) {
+		/* CXL defines a u32 handle while cdat defines u8, ignore upper bits */

CDAT

Quoted 22 lines
drivers/cxl/core/cdat.c +22 -0
+		u8 dc_handle = mds->dc_region[i].dsmad_handle & 0xff;
+
+		if (resource_size(&cxlds->dc_res[i])) {
+			struct range dc_range = {
+				.start = cxlds->dc_res[i].start,
+				.end = cxlds->dc_res[i].end,
+			};
+
+			if (range_contains(&dent->dpa_range, &dc_range)) {
+				if (dent->handle != dc_handle)
+					dev_warn(dev, "DC Region/DSMAS mis-matched handle/range; region %pra (%u); dsmas %pra (%u)\n"
+						      "   setting DC region attributes regardless\n",
+						&dent->dpa_range, dent->handle,
+						&dc_range, dc_handle);
+
+				mds->dc_region[i].shareable = dent->shareable;
+				mds->dc_region[i].read_only = dent->read_only;
+				update_perf_entry(dev, dent, &mds->dc_perf[i]);
+			}
+		}
+	}
+}
#55 lore raw

PATCH v4 13/28 cxl/mem: Expose DCD partition capabilities in sysfs

In reply to Ira Weiny

From: Navneet Singh <[email protected]>

To properly configure CXL regions on Dynamic Capacity Devices (DCD), user space will need to know the details of the DC partitions available.

Expose dynamic capacity capabilities through sysfs.

---
Changes:
[iweiny: Change .../memX/dc/* to .../memX/dcY/*] [iweiny: add read only and shareable attributes from DSMAS] [djiang: Split sysfs docs]
[iweiny: Adjust sysfs doc dates]
[iweiny: Add qos details]

---
 Documentation/ABI/testing/sysfs-bus-cxl |  45 ++++++++++++
 drivers/cxl/core/memdev.c               | 126 ++++++++++++++++++++++++++++++++
 2 files changed, 171 insertions(+)
Documentation/ABI/testing/sysfs-bus-cxl +45 -0
index 3f5627a1210a..b865eefdb74c 100644
@@ -54,6 +54,51 @@ Description:
 		identically named field in the Identify Memory Device Output
 		Payload in the CXL-2.0 specification.
 
+What:		/sys/bus/cxl/devices/memX/dcY/size
+Date:		December, 2024
+KernelVersion:	v6.13
+Contact:	[email protected]
+Description:
+		(RO) Dynamic Capacity (DC) region information.  Devices only
+		export dcY if DCD partition Y is supported.
+		dcY/size is the size of each of those partitions.
+
+What:		/sys/bus/cxl/devices/memX/dcY/read_only
+Date:		December, 2024
+KernelVersion:	v6.13
+Contact:	[email protected]
+Description:
+		(RO) Dynamic Capacity (DC) region information.  Devices only
+		export dcY if DCD partition Y is supported.
+		dcY/read_only indicates true if the region is exported
+		read_only from the device.
+
+What:		/sys/bus/cxl/devices/memX/dcY/shareable
+Date:		December, 2024
+KernelVersion:	v6.13
+Contact:	[email protected]
+Description:
+		(RO) Dynamic Capacity (DC) region information.  Devices only
+		export dcY if DCD partition Y is supported.
+		dcY/shareable indicates true if the region is exported
+		shareable from the device.
+
+What:		/sys/bus/cxl/devices/memX/dcY/qos_class
+Date:		December, 2024
+KernelVersion:	v6.13
+Contact:	[email protected]
+Description:
+		(RO) Dynamic Capacity (DC) region information.  Devices only
+		export dcY if DCD partition Y is supported.  For CXL host
+		platforms that support "QoS Telemmetry" this attribute conveys
+		a comma delimited list of platform specific cookies that
+		identifies a QoS performance class for the persistent partition
+		of the CXL mem device. These class-ids can be compared against
+		a similar "qos_class" published for a root decoder. While it is
+		not required that the endpoints map their local memory-class to
+		a matching platform class, mismatches are not recommended and
+		there are platform specific performance related side-effects
+		that may result. First class-id is displayed.
 
 What:		/sys/bus/cxl/devices/memX/pmem/qos_class
 Date:		May, 2023
drivers/cxl/core/memdev.c +126 -0

144 lines not on this page · raw message

Signature · 1 line
2.46.0
#56 lore raw

Re: [PATCH v4 13/28] cxl/mem: Expose DCD partition capabilities in sysfs

Quoted 98 lines · On Mon, Oct 07, 2024 at 06:16:19PM -0500, [email protected] wrote:

From: Navneet Singh <[email protected]>

To properly configure CXL regions on Dynamic Capacity Devices (DCD), user space will need to know the details of the DC partitions available.

Expose dynamic capacity capabilities through sysfs.

---
Changes:
[iweiny: Change .../memX/dc/* to .../memX/dcY/*] [iweiny: add read only and shareable attributes from DSMAS] [djiang: Split sysfs docs]
[iweiny: Adjust sysfs doc dates]
[iweiny: Add qos details]

---
 Documentation/ABI/testing/sysfs-bus-cxl |  45 ++++++++++++
 drivers/cxl/core/memdev.c               | 126 ++++++++++++++++++++++++++++++++
 2 files changed, 171 insertions(+)
Documentation/ABI/testing/sysfs-bus-cxl +45 -0
index 3f5627a1210a..b865eefdb74c 100644
@@ -54,6 +54,51 @@ Description:
 		identically named field in the Identify Memory Device Output
 		Payload in the CXL-2.0 specification.
 
+What:		/sys/bus/cxl/devices/memX/dcY/size
+Date:		December, 2024
+KernelVersion:	v6.13
+Contact:	[email protected]
+Description:
+		(RO) Dynamic Capacity (DC) region information.  Devices only
+		export dcY if DCD partition Y is supported.
+		dcY/size is the size of each of those partitions.
+
+What:		/sys/bus/cxl/devices/memX/dcY/read_only
+Date:		December, 2024
+KernelVersion:	v6.13
+Contact:	[email protected]
+Description:
+		(RO) Dynamic Capacity (DC) region information.  Devices only
+		export dcY if DCD partition Y is supported.
+		dcY/read_only indicates true if the region is exported
+		read_only from the device.
+
+What:		/sys/bus/cxl/devices/memX/dcY/shareable
+Date:		December, 2024
+KernelVersion:	v6.13
+Contact:	[email protected]
+Description:
+		(RO) Dynamic Capacity (DC) region information.  Devices only
+		export dcY if DCD partition Y is supported.
+		dcY/shareable indicates true if the region is exported
+		shareable from the device.
+
+What:		/sys/bus/cxl/devices/memX/dcY/qos_class
+Date:		December, 2024
+KernelVersion:	v6.13
+Contact:	[email protected]
+Description:
+		(RO) Dynamic Capacity (DC) region information.  Devices only
+		export dcY if DCD partition Y is supported.  For CXL host
+		platforms that support "QoS Telemmetry" this attribute conveys
+		a comma delimited list of platform specific cookies that
+		identifies a QoS performance class for the persistent partition
+		of the CXL mem device. These class-ids can be compared against
+		a similar "qos_class" published for a root decoder. While it is
+		not required that the endpoints map their local memory-class to
+		a matching platform class, mismatches are not recommended and
+		there are platform specific performance related side-effects
+		that may result. First class-id is displayed.
 
 What:		/sys/bus/cxl/devices/memX/pmem/qos_class
 Date:		May, 2023
drivers/cxl/core/memdev.c +4 -0
index 84fefb76dafa..2565b10a769c 100644
@@ -2,6 +2,7 @@
 /* Copyright(c) 2020 Intel Corporation. */
 
 #include <linux/io-64-nonatomic-lo-hi.h>
+#include <linux/string_choices.h>
 #include <linux/firmware.h>
 #include <linux/device.h>
 #include <linux/slab.h>
@@ -449,6 +450,123 @@ static struct attribute *cxl_memdev_security_attributes[] = {
 	NULL,
 };
 
+static ssize_t show_size_dcN(struct cxl_memdev *cxlmd, char *buf, int pos)
+{
+	struct cxl_memdev_state *mds = to_cxl_memdev_state(cxlmd->cxlds);

drivers/cxl/core/memdev.c static struct attribute *cxl_memdev_security_attributes[] = { · line 465

Quoted 10 lines
drivers/cxl/core/memdev.c +10 -0
+
+	return sysfs_emit(buf, "%#llx\n", mds->dc_region[pos].decode_len);
+}
+
+static ssize_t show_read_only_dcN(struct cxl_memdev *cxlmd, char *buf, int pos)
+{
+	struct cxl_memdev_state *mds = to_cxl_memdev_state(cxlmd->cxlds);
+
+	return sysfs_emit(buf, "%s\n",
+			  str_false_true(mds->dc_region[pos].read_only));

For this function and below, why str_false_true instead of str_true_false??

Fan

Quoted 126 lines
drivers/cxl/core/memdev.c +112 -0
+}
+
+static ssize_t show_shareable_dcN(struct cxl_memdev *cxlmd, char *buf, int pos)
+{
+	struct cxl_memdev_state *mds = to_cxl_memdev_state(cxlmd->cxlds);
+
+	return sysfs_emit(buf, "%s\n",
+			  str_false_true(mds->dc_region[pos].shareable));
+}
+
+static ssize_t show_qos_class_dcN(struct cxl_memdev *cxlmd, char *buf, int pos)
+{
+	struct cxl_memdev_state *mds = to_cxl_memdev_state(cxlmd->cxlds);
+
+	return sysfs_emit(buf, "%d\n", mds->dc_perf[pos].qos_class);
+}
+
+#define CXL_MEMDEV_DC_ATTR_GROUP(n)						\
+static ssize_t dc##n##_size_show(struct device *dev,				\
+				 struct device_attribute *attr,			\
+				 char *buf)					\
+{										\
+	return show_size_dcN(to_cxl_memdev(dev), buf, (n));			\
+}										\
+struct device_attribute dc##n##_size = {					\
+	.attr	= { .name = "size", .mode = 0444 },				\
+	.show	= dc##n##_size_show,						\
+};										\
+static ssize_t dc##n##_read_only_show(struct device *dev,			\
+				      struct device_attribute *attr,		\
+				      char *buf)				\
+{										\
+	return show_read_only_dcN(to_cxl_memdev(dev), buf, (n));		\
+}										\
+struct device_attribute dc##n##_read_only = {					\
+	.attr	= { .name = "read_only", .mode = 0444 },			\
+	.show	= dc##n##_read_only_show,					\
+};										\
+static ssize_t dc##n##_shareable_show(struct device *dev,			\
+				     struct device_attribute *attr,		\
+				     char *buf)					\
+{										\
+	return show_shareable_dcN(to_cxl_memdev(dev), buf, (n));		\
+}										\
+struct device_attribute dc##n##_shareable = {					\
+	.attr	= { .name = "shareable", .mode = 0444 },			\
+	.show	= dc##n##_shareable_show,					\
+};										\
+static ssize_t dc##n##_qos_class_show(struct device *dev,			\
+				      struct device_attribute *attr,		\
+				      char *buf)				\
+{										\
+	return show_qos_class_dcN(to_cxl_memdev(dev), buf, (n));		\
+}										\
+struct device_attribute dc##n##_qos_class = {					\
+	.attr	= { .name = "qos_class", .mode = 0444 },			\
+	.show	= dc##n##_qos_class_show,					\
+};										\
+static struct attribute *cxl_memdev_dc##n##_attributes[] = {			\
+	&dc##n##_size.attr,							\
+	&dc##n##_read_only.attr,						\
+	&dc##n##_shareable.attr,						\
+	&dc##n##_qos_class.attr,						\
+	NULL,									\
+};										\
+static umode_t cxl_memdev_dc##n##_attr_visible(struct kobject *kobj,		\
+					       struct attribute *a,		\
+					       int pos)				\
+{										\
+	struct device *dev = kobj_to_dev(kobj);					\
+	struct cxl_memdev *cxlmd = to_cxl_memdev(dev);				\
+	struct cxl_memdev_state *mds = to_cxl_memdev_state(cxlmd->cxlds);	\
+										\
+	/* Not a memory device */						\
+	if (!mds)								\
+		return 0;							\
+	return a->mode;								\
+}										\
+static umode_t cxl_memdev_dc##n##_group_visible(struct kobject *kobj)		\
+{										\
+	struct device *dev = kobj_to_dev(kobj);					\
+	struct cxl_memdev *cxlmd = to_cxl_memdev(dev);				\
+	struct cxl_memdev_state *mds = to_cxl_memdev_state(cxlmd->cxlds);	\
+										\
+	/* Not a memory device or partition not supported */			\
+	if (!mds || n >= mds->nr_dc_region)					\
+		return false;							\
+	return true;								\
+}										\
+DEFINE_SYSFS_GROUP_VISIBLE(cxl_memdev_dc##n);					\
+static struct attribute_group cxl_memdev_dc##n##_group = {			\
+	.name = "dc"#n,								\
+	.attrs = cxl_memdev_dc##n##_attributes,					\
+	.is_visible = SYSFS_GROUP_VISIBLE(cxl_memdev_dc##n),			\
+}
+CXL_MEMDEV_DC_ATTR_GROUP(0);
+CXL_MEMDEV_DC_ATTR_GROUP(1);
+CXL_MEMDEV_DC_ATTR_GROUP(2);
+CXL_MEMDEV_DC_ATTR_GROUP(3);
+CXL_MEMDEV_DC_ATTR_GROUP(4);
+CXL_MEMDEV_DC_ATTR_GROUP(5);
+CXL_MEMDEV_DC_ATTR_GROUP(6);
+CXL_MEMDEV_DC_ATTR_GROUP(7);
+
 static umode_t cxl_memdev_visible(struct kobject *kobj, struct attribute *a,
 				  int n)
 {
@@ -525,6 +643,14 @@ static struct attribute_group cxl_memdev_security_attribute_group = {
 };
 
 static const struct attribute_group *cxl_memdev_attribute_groups[] = {
+	&cxl_memdev_dc0_group,
+	&cxl_memdev_dc1_group,
+	&cxl_memdev_dc2_group,
+	&cxl_memdev_dc3_group,
+	&cxl_memdev_dc4_group,
+	&cxl_memdev_dc5_group,
+	&cxl_memdev_dc6_group,
+	&cxl_memdev_dc7_group,
 	&cxl_memdev_attribute_group,
 	&cxl_memdev_ram_attribute_group,
 	&cxl_memdev_pmem_attribute_group,
Signature · 2 lines
-- 
2.46.0
Signature · 1 line
Fan Ni
author
#57 lore raw

Re: [PATCH v4 13/28] cxl/mem: Expose DCD partition capabilities in sysfs

Quoted 3 lines · Fan Ni
Quoted 2 lines · On Mon, Oct 07, 2024 at 06:16:19PM -0500, [email protected] wrote:

From: Navneet Singh <[email protected]>

[snip]

Quoted 11 lines

drivers/cxl/core/memdev.c static struct attribute *cxl_memdev_security_attributes[] = { · line 465

Quoted 7 lines
drivers/cxl/core/memdev.c +7 -0
+
+static ssize_t show_read_only_dcN(struct cxl_memdev *cxlmd, char *buf, int pos)
+{
+	struct cxl_memdev_state *mds = to_cxl_memdev_state(cxlmd->cxlds);
+
+	return sysfs_emit(buf, "%s\n",
+			  str_false_true(mds->dc_region[pos].read_only));

For this function and below, why str_false_true instead of str_true_false??

Oh! I did not realize they were not the same. That API is tricky.

Yea str_true_false() is the correct call.

Thanks for noticing that.
Ira

Reviewed-by on v4 13/28
#58 lore raw

Re: [PATCH v4 13/28] cxl/mem: Expose DCD partition capabilities in sysfs

In reply to [email protected]
Quoted 10 lines · On Mon, 07 Oct 2024 18:16:19 -0500 [email protected] wrote:

From: Navneet Singh <[email protected]>

To properly configure CXL regions on Dynamic Capacity Devices (DCD), user space will need to know the details of the DC partitions available.

Expose dynamic capacity capabilities through sysfs.

Some trivial stuff inline that I'm not that bothered about either way.

Subject to answering Fan's query

Quoted 47 lines

---
Changes:
[iweiny: Change .../memX/dc/* to .../memX/dcY/*] [iweiny: add read only and shareable attributes from DSMAS] [djiang: Split sysfs docs]
[iweiny: Adjust sysfs doc dates]
[iweiny: Add qos details]

---
 Documentation/ABI/testing/sysfs-bus-cxl |  45 ++++++++++++
 drivers/cxl/core/memdev.c               | 126 ++++++++++++++++++++++++++++++++
 2 files changed, 171 insertions(+)
Documentation/ABI/testing/sysfs-bus-cxl +26 -0
index 3f5627a1210a..b865eefdb74c 100644
@@ -54,6 +54,51 @@ Description:
 		identically named field in the Identify Memory Device Output
 		Payload in the CXL-2.0 specification.
 
+What:		/sys/bus/cxl/devices/memX/dcY/size
+Date:		December, 2024
+KernelVersion:	v6.13
+Contact:	[email protected]
+Description:
+		(RO) Dynamic Capacity (DC) region information.  Devices only
+		export dcY if DCD partition Y is supported.
+		dcY/size is the size of each of those partitions.
+
+What:		/sys/bus/cxl/devices/memX/dcY/read_only
+Date:		December, 2024
+KernelVersion:	v6.13
+Contact:	[email protected]
+Description:
+		(RO) Dynamic Capacity (DC) region information.  Devices only
+		export dcY if DCD partition Y is supported.
+		dcY/read_only indicates true if the region is exported
+		read_only from the device.
+
+What:		/sys/bus/cxl/devices/memX/dcY/shareable
+Date:		December, 2024
+KernelVersion:	v6.13
+Contact:	[email protected]
+Description:
+		(RO) Dynamic Capacity (DC) region information.  Devices only
+		export dcY if DCD partition Y is supported.

Documentation/ABI/testing/sysfs-bus-cxl Description: · line 92

Quoted 10 lines
Documentation/ABI/testing/sysfs-bus-cxl +10 -0
+		dcY/shareable indicates true if the region is exported
+		shareable from the device.
+
+What:		/sys/bus/cxl/devices/memX/dcY/qos_class
+Date:		December, 2024
+KernelVersion:	v6.13
+Contact:	[email protected]
+Description:
+		(RO) Dynamic Capacity (DC) region information.  Devices only
+		export dcY if DCD partition Y is supported.  

You can document sysfs directories I think, e.g.
https://elixir.bootlin.com/linux/v6.12-rc2/source/Documentation/ABI/stable/sysfs-devices-node#L32
so maybe

What: /sys/bus/cxl/device/memX/dcY
Date: December, 2024
KernelVersion: v6.13
Contact: [email protected]
Description:
Directory containing Dynamic Capacity (DC) region information. Devices only export dcY if DCD partition Y is supported.

What: /sys/bus/cxl/devices/memX/dcY/qos_class
Date: December, 2024
KernelVersion: v6.13
Contact: [email protected]
Description:
For CXL host...

To avoid the repetition of first bit of docs?

Quoted 10 lines
Documentation/ABI/testing/sysfs-bus-cxl +9 -0
+		platforms that support "QoS Telemmetry" this attribute conveys
+		a comma delimited list of platform specific cookies that
+		identifies a QoS performance class for the persistent partition
+		of the CXL mem device. These class-ids can be compared against
+		a similar "qos_class" published for a root decoder. While it is
+		not required that the endpoints map their local memory-class to
+		a matching platform class, mismatches are not recommended and
+		there are platform specific performance related side-effects
+		that may result. First class-id is displayed.
 

Documentation/ABI/testing/sysfs-bus-cxl Description:

Quoted 10 lines
Documentation/ABI/testing/sysfs-bus-cxl +6 -0
 What:		/sys/bus/cxl/devices/memX/pmem/qos_class
 Date:		May, 2023
 
 
+static ssize_t show_shareable_dcN(struct cxl_memdev *cxlmd, char *buf, int pos)
+{
+	struct cxl_memdev_state *mds = to_cxl_memdev_state(cxlmd->cxlds);
+
+	return sysfs_emit(buf, "%s\n",
+			  str_false_true(mds->dc_region[pos].shareable));

Fan has already raised that these seem backwards.

Quoted 46 lines
Documentation/ABI/testing/sysfs-bus-cxl +46 -0
+}
+
+static ssize_t show_qos_class_dcN(struct cxl_memdev *cxlmd, char *buf, int pos)
+{
+	struct cxl_memdev_state *mds = to_cxl_memdev_state(cxlmd->cxlds);
+
+	return sysfs_emit(buf, "%d\n", mds->dc_perf[pos].qos_class);
+}
+
+#define CXL_MEMDEV_DC_ATTR_GROUP(n)						\
+static ssize_t dc##n##_size_show(struct device *dev,				\
+				 struct device_attribute *attr,			\
+				 char *buf)					\
+{										\
+	return show_size_dcN(to_cxl_memdev(dev), buf, (n));			\
+}										\
+struct device_attribute dc##n##_size = {					\
+	.attr	= { .name = "size", .mode = 0444 },				\
+	.show	= dc##n##_size_show,						\
+};										\
+static ssize_t dc##n##_read_only_show(struct device *dev,			\
+				      struct device_attribute *attr,		\
+				      char *buf)				\
+{										\
+	return show_read_only_dcN(to_cxl_memdev(dev), buf, (n));		\
+}										\
+struct device_attribute dc##n##_read_only = {					\
+	.attr	= { .name = "read_only", .mode = 0444 },			\
+	.show	= dc##n##_read_only_show,					\
+};										\
+static ssize_t dc##n##_shareable_show(struct device *dev,			\
+				     struct device_attribute *attr,		\
+				     char *buf)					\
+{										\
+	return show_shareable_dcN(to_cxl_memdev(dev), buf, (n));		\
+}										\
+struct device_attribute dc##n##_shareable = {					\
+	.attr	= { .name = "shareable", .mode = 0444 },			\
+	.show	= dc##n##_shareable_show,					\
+};										\
+static ssize_t dc##n##_qos_class_show(struct device *dev,			\
+				      struct device_attribute *attr,		\
+				      char *buf)				\
+{										\
+	return show_qos_class_dcN(to_cxl_memdev(dev), buf, (n));		\
+}										\

Documentation/ABI/testing/sysfs-bus-cxl

Quoted 10 lines
Documentation/ABI/testing/sysfs-bus-cxl +10 -0
+struct device_attribute dc##n##_qos_class = {					\
+	.attr	= { .name = "qos_class", .mode = 0444 },			\
+	.show	= dc##n##_qos_class_show,					\
+};										\
+static struct attribute *cxl_memdev_dc##n##_attributes[] = {			\
+	&dc##n##_size.attr,							\
+	&dc##n##_read_only.attr,						\
+	&dc##n##_shareable.attr,						\
+	&dc##n##_qos_class.attr,						\
+	NULL,									\

No comma needed on terminator.

Documentation/ABI/testing/sysfs-bus-cxl

Quoted 11 lines
Documentation/ABI/testing/sysfs-bus-cxl +11 -0
+};										\
+static umode_t cxl_memdev_dc##n##_attr_visible(struct kobject *kobj,		\
+					       struct attribute *a,		\
+					       int pos)				\
+{										\
+	struct device *dev = kobj_to_dev(kobj);					\
+	struct cxl_memdev *cxlmd = to_cxl_memdev(dev);				\
+	struct cxl_memdev_state *mds = to_cxl_memdev_state(cxlmd->cxlds);	\
+										\
+	/* Not a memory device */						\
+	if (!mds)								\
	if (!to_cxl_memdev_state(cxlmd->cxlds))
		return 0;

I dislike long macros so if we can shave them down that is always good!

We do have precedence in hdm.c for just checking the type directly so maybe

	if (cxlmd->cxlds->type != CXL_DEVTYPE_CLASSMEM)

but the above is also fine as compiler should be able to figure out it doesn't need to do the second half of the inline.

Documentation/ABI/testing/sysfs-bus-cxl

Quoted 13 lines
Documentation/ABI/testing/sysfs-bus-cxl +13 -0
+		return 0;							\
+	return a->mode;								\
+}										\
+static umode_t cxl_memdev_dc##n##_group_visible(struct kobject *kobj)		\
+{										\
+	struct device *dev = kobj_to_dev(kobj);					\
+	struct cxl_memdev *cxlmd = to_cxl_memdev(dev);				\
+	struct cxl_memdev_state *mds = to_cxl_memdev_state(cxlmd->cxlds);	\
+										\
+	/* Not a memory device or partition not supported */			\
+	if (!mds || n >= mds->nr_dc_region)					\
+		return false;							\
+	return true;								\
	/* Memory device and partition is supported */
	return mds && n < mds->nr_dc_region;
Quoted 2 lines

+} \

author
#59 lore raw

Re: [PATCH v4 13/28] cxl/mem: Expose DCD partition capabilities in sysfs

Quoted 18 lines · Jonathan Cameron
Quoted 10 lines · On Mon, 07 Oct 2024 18:16:19 -0500 [email protected] wrote:

From: Navneet Singh <[email protected]>

To properly configure CXL regions on Dynamic Capacity Devices (DCD), user space will need to know the details of the DC partitions available.

Expose dynamic capacity capabilities through sysfs.

Some trivial stuff inline that I'm not that bothered about either way.

Subject to answering Fan's query

[snip]

Quoted 58 lines
Quoted 35 lines
Documentation/ABI/testing/sysfs-bus-cxl +26 -0
index 3f5627a1210a..b865eefdb74c 100644
@@ -54,6 +54,51 @@ Description:
 		identically named field in the Identify Memory Device Output
 		Payload in the CXL-2.0 specification.
 
+What:		/sys/bus/cxl/devices/memX/dcY/size
+Date:		December, 2024
+KernelVersion:	v6.13
+Contact:	[email protected]
+Description:
+		(RO) Dynamic Capacity (DC) region information.  Devices only
+		export dcY if DCD partition Y is supported.
+		dcY/size is the size of each of those partitions.
+
+What:		/sys/bus/cxl/devices/memX/dcY/read_only
+Date:		December, 2024
+KernelVersion:	v6.13
+Contact:	[email protected]
+Description:
+		(RO) Dynamic Capacity (DC) region information.  Devices only
+		export dcY if DCD partition Y is supported.
+		dcY/read_only indicates true if the region is exported
+		read_only from the device.
+
+What:		/sys/bus/cxl/devices/memX/dcY/shareable
+Date:		December, 2024
+KernelVersion:	v6.13
+Contact:	[email protected]
+Description:
+		(RO) Dynamic Capacity (DC) region information.  Devices only
+		export dcY if DCD partition Y is supported.

Documentation/ABI/testing/sysfs-bus-cxl Description: · line 92

Quoted 10 lines
Documentation/ABI/testing/sysfs-bus-cxl +10 -0
+		dcY/shareable indicates true if the region is exported
+		shareable from the device.
+
+What:		/sys/bus/cxl/devices/memX/dcY/qos_class
+Date:		December, 2024
+KernelVersion:	v6.13
+Contact:	[email protected]
+Description:
+		(RO) Dynamic Capacity (DC) region information.  Devices only
+		export dcY if DCD partition Y is supported.  

You can document sysfs directories I think, e.g.
https://elixir.bootlin.com/linux/v6.12-rc2/source/Documentation/ABI/stable/sysfs-devices-node#L32
so maybe

What: /sys/bus/cxl/device/memX/dcY
Date: December, 2024
KernelVersion: v6.13
Contact: [email protected]
Description:
Directory containing Dynamic Capacity (DC) region information. Devices only export dcY if DCD partition Y is supported.

Quoted 8 lines

What: /sys/bus/cxl/devices/memX/dcY/qos_class
Date: December, 2024
KernelVersion: v6.13
Contact: [email protected]
Description:
For CXL host...

To avoid the repetition of first bit of docs?

The other docs don't do this. For example:

/sys/bus/cxl/devices/memX
/sys/bus/cxl/devices/memX/ram
/sys/bus/cxl/devices/memX/pmem

Are not documented like that. I'm inclined to leave it.

Quoted 15 lines
Quoted 13 lines
Documentation/ABI/testing/sysfs-bus-cxl +9 -0
+		platforms that support "QoS Telemmetry" this attribute conveys
+		a comma delimited list of platform specific cookies that
+		identifies a QoS performance class for the persistent partition
+		of the CXL mem device. These class-ids can be compared against
+		a similar "qos_class" published for a root decoder. While it is
+		not required that the endpoints map their local memory-class to
+		a matching platform class, mismatches are not recommended and
+		there are platform specific performance related side-effects
+		that may result. First class-id is displayed.
 
 What:		/sys/bus/cxl/devices/memX/pmem/qos_class
 Date:		May, 2023
Quoted 8 lines

Documentation/ABI/testing/sysfs-bus-cxl

Quoted 6 lines
Documentation/ABI/testing/sysfs-bus-cxl +6 -0
+static ssize_t show_shareable_dcN(struct cxl_memdev *cxlmd, char *buf, int pos)
+{
+	struct cxl_memdev_state *mds = to_cxl_memdev_state(cxlmd->cxlds);
+
+	return sysfs_emit(buf, "%s\n",
+			  str_false_true(mds->dc_region[pos].shareable));

Fan has already raised that these seem backwards.

Yep fixed.

[snip]

Quoted 8 lines

Documentation/ABI/testing/sysfs-bus-cxl

Quoted 6 lines
Documentation/ABI/testing/sysfs-bus-cxl +6 -0
+static struct attribute *cxl_memdev_dc##n##_attributes[] = {			\
+	&dc##n##_size.attr,							\
+	&dc##n##_read_only.attr,						\
+	&dc##n##_shareable.attr,						\
+	&dc##n##_qos_class.attr,						\
+	NULL,									\

No comma needed on terminator.

Fixed.

Quoted 15 lines

Documentation/ABI/testing/sysfs-bus-cxl

Quoted 12 lines
Documentation/ABI/testing/sysfs-bus-cxl +11 -0
+};										\
+static umode_t cxl_memdev_dc##n##_attr_visible(struct kobject *kobj,		\
+					       struct attribute *a,		\
+					       int pos)				\
+{										\
+	struct device *dev = kobj_to_dev(kobj);					\
+	struct cxl_memdev *cxlmd = to_cxl_memdev(dev);				\
+	struct cxl_memdev_state *mds = to_cxl_memdev_state(cxlmd->cxlds);	\
+										\
+	/* Not a memory device */						\
+	if (!mds)								\
	if (!to_cxl_memdev_state(cxlmd->cxlds))
		return 0;
Quoted 1 line

I dislike long macros so if we can shave them down that is always good!

Agreed but this was the most straight forward way to deal with this. I could perhaps break it up by having a 'master macro' which is made of smaller macros... But this works.

Quoted 2 lines

We do have precedence in hdm.c

Not in hdm.c directly but all the 'to_XXX()' calls have a type check. So it is modeled that way and is called from other places.

Quoted 6 lines

for just checking the type directly so maybe

	if (cxlmd->cxlds->type != CXL_DEVTYPE_CLASSMEM)

but the above is also fine as compiler should be able to figure out it doesn't need to do the second half of the inline.

I'm going to leave it.

Quoted 18 lines
Quoted 5 lines
Documentation/ABI/testing/sysfs-bus-cxl +3 -0
+		return 0;							\
+	return a->mode;								\
+}										\

Documentation/ABI/testing/sysfs-bus-cxl

Quoted 10 lines
Documentation/ABI/testing/sysfs-bus-cxl +10 -0
+static umode_t cxl_memdev_dc##n##_group_visible(struct kobject *kobj)		\
+{										\
+	struct device *dev = kobj_to_dev(kobj);					\
+	struct cxl_memdev *cxlmd = to_cxl_memdev(dev);				\
+	struct cxl_memdev_state *mds = to_cxl_memdev_state(cxlmd->cxlds);	\
+										\
+	/* Not a memory device or partition not supported */			\
+	if (!mds || n >= mds->nr_dc_region)					\
+		return false;							\
+	return true;								\
	/* Memory device and partition is supported */
	return mds && n < mds->nr_dc_region;

Done.
Ira

#60 lore raw

Re: [PATCH v4 13/28] cxl/mem: Expose DCD partition capabilities in sysfs

In reply to [email protected]
Quoted 5 lines · On Mon, Oct 07, 2024 at 06:16:19PM -0500, [email protected] wrote:
Documentation/ABI/testing/sysfs-bus-cxl +5 -0
+What:		/sys/bus/cxl/devices/memX/dcY/qos_class
+Date:		December, 2024
+KernelVersion:	v6.13
+Contact:	[email protected]
+Description:

Documentation/ABI/testing/sysfs-bus-cxl Description: · line 100

Quoted 10 lines
Documentation/ABI/testing/sysfs-bus-cxl +10 -0
+		(RO) Dynamic Capacity (DC) region information.  Devices only
+		export dcY if DCD partition Y is supported.  For CXL host
+		platforms that support "QoS Telemmetry" this attribute conveys
+		a comma delimited list of platform specific cookies that
+		identifies a QoS performance class for the persistent partition
+		of the CXL mem device. These class-ids can be compared against
+		a similar "qos_class" published for a root decoder. While it is
+		not required that the endpoints map their local memory-class to
+		a matching platform class, mismatches are not recommended and
+		there are platform specific performance related side-effects

"... mismatches are not recommended as there are ..."

Documentation/ABI/testing/sysfs-bus-cxl Description: · line 101

Quoted 2 lines
Documentation/ABI/testing/sysfs-bus-cxl +1 -0
+		that may result. First class-id is displayed.

Thanks.

Signature · 1 line
An old man doll... just what I always wanted! - Clara
#61 lore raw

PATCH v4 14/28 cxl/port: Add endpoint decoder DC mode support to sysfs

In reply to Ira Weiny

From: Navneet Singh <[email protected]>

Endpoint decoder mode is used to represent the partition the decoder points to such as ram or pmem.

Expand the mode to allow a decoder to point to a specific DC partition (Region).

---
Changes:
[iweiny: prevent creation of region on shareable DC partitions] [Fan: change mode range logic]
[Fan: use !resource_size()]
[djiang: use the static mode name string array in mode_store()] [Jonathan: remove rc check from mode to region index]
[Jonathan: clarify decoder mode 'mixed']
[djbw: drop cleanup patch and just follow the convention in cxl_dpa_set_mode()] [fan: make dcd resource size check similar to other partitions] [djbw, jonathan, fan: remove mode range check from dc_mode_to_region_index] [iweiny: push sysfs versions to 6.12]

---
 Documentation/ABI/testing/sysfs-bus-cxl | 21 ++++++++++----------
 drivers/cxl/core/hdm.c                  | 17 ++++++++++++++++
 drivers/cxl/core/port.c                 | 10 +++++-----
 drivers/cxl/cxl.h                       | 35 ++++++++++++++++++---------------
 4 files changed, 52 insertions(+), 31 deletions(-)
Documentation/ABI/testing/sysfs-bus-cxl +11 -10
index b865eefdb74c..661dab99183f 100644
@@ -361,23 +361,24 @@ Description:
 
 
 What:		/sys/bus/cxl/devices/decoderX.Y/mode
-Date:		May, 2022
-KernelVersion:	v6.0
+Date:		May, 2022, October 2024
+KernelVersion:	v6.0, v6.12 (dcY)
 Contact:	[email protected]
 Description:
 		(RW) When a CXL decoder is of devtype "cxl_decoder_endpoint" it
 		translates from a host physical address range, to a device local
 		address range. Device-local address ranges are further split
-		into a 'ram' (volatile memory) range and 'pmem' (persistent
-		memory) range. The 'mode' attribute emits one of 'ram', 'pmem',
-		'mixed', or 'none'. The 'mixed' indication is for error cases
-		when a decoder straddles the volatile/persistent partition
-		boundary, and 'none' indicates the decoder is not actively
-		decoding, or no DPA allocation policy has been set.
+		into a 'ram' (volatile memory) range, 'pmem' (persistent
+		memory) range, or Dynamic Capacity (DC) range. The 'mode'
+		attribute emits one of 'ram', 'pmem', 'dcY', 'mixed', or
+		'none'. The 'mixed' indication is for error cases when a
+		decoder straddles partition boundaries, and 'none' indicates
+		the decoder is not actively decoding, or no DPA allocation
+		policy has been set.
 
 		'mode' can be written, when the decoder is in the 'disabled'
-		state, with either 'ram' or 'pmem' to set the boundaries for the
-		next allocation.
+		state, with 'ram', 'pmem', or 'dcY' to set the boundaries for
+		the next allocation.
 
 
 What:		/sys/bus/cxl/devices/decoderX.Y/dpa_resource
drivers/cxl/core/hdm.c +17 -0
index 8c7f941eaba1..b368babb55d9 100644
@@ -551,6 +551,7 @@ int cxl_dpa_set_mode(struct cxl_endpoint_decoder *cxled,
 	switch (mode) {
 	case CXL_DECODER_RAM:
 	case CXL_DECODER_PMEM:
+	case CXL_DECODER_DC0 ... CXL_DECODER_DC7:
 		break;
 	default:
 		dev_dbg(dev, "unsupported mode: %d\n", mode);
@@ -578,6 +579,22 @@ int cxl_dpa_set_mode(struct cxl_endpoint_decoder *cxled,
 		goto out;
 	}
 
+	if (mode >= CXL_DECODER_DC0 && mode <= CXL_DECODER_DC7) {
+		struct cxl_memdev_state *mds = to_cxl_memdev_state(cxlds);
+
+		rc = dc_mode_to_region_index(mode);
+		if (!resource_size(&cxlds->dc_res[rc])) {
+			dev_dbg(dev, "no available dynamic capacity\n");
+			rc = -ENXIO;
+			goto out;
+		}
+		if (mds->dc_region[rc].shareable) {
+			dev_err(dev, "DC region %d is shareable\n", rc);
+			rc = -EINVAL;
+			goto out;
+		}
+	}
+
 	cxled->mode = mode;
 	rc = 0;
 out:
drivers/cxl/core/port.c +5 -5
index 85b912c11f04..23b4f266a83a 100644
@@ -205,11 +205,11 @@ static ssize_t mode_store(struct device *dev, struct device_attribute *attr,
 	enum cxl_decoder_mode mode;
 	ssize_t rc;
 
-	if (sysfs_streq(buf, "pmem"))
-		mode = CXL_DECODER_PMEM;
-	else if (sysfs_streq(buf, "ram"))
-		mode = CXL_DECODER_RAM;
-	else
+	for (mode = CXL_DECODER_RAM; mode < CXL_DECODER_MIXED; mode++)
+		if (sysfs_streq(buf, cxl_decoder_mode_names[mode]))
+			break;
+
+	if (mode >= CXL_DECODER_MIXED)
 		return -EINVAL;
 
 	rc = cxl_dpa_set_mode(cxled, mode);
drivers/cxl/cxl.h +19 -16
index 8b7099c38a40..cbaacbe0f36d 100644
@@ -365,6 +365,9 @@ struct cxl_decoder {
 /*
  * CXL_DECODER_DEAD prevents endpoints from being reattached to regions
  * while cxld_unregister() is running
+ *
+ * NOTE: CXL_DECODER_RAM must be second and CXL_DECODER_MIXED must be last.
+ *	 See mode_store()
  */
 enum cxl_decoder_mode {
 	CXL_DECODER_NONE,
@@ -382,25 +385,25 @@ enum cxl_decoder_mode {
 	CXL_DECODER_DEAD,
 };
 
+static const char * const cxl_decoder_mode_names[] = {
+	[CXL_DECODER_NONE] = "none",
+	[CXL_DECODER_RAM] = "ram",
+	[CXL_DECODER_PMEM] = "pmem",
+	[CXL_DECODER_DC0] = "dc0",
+	[CXL_DECODER_DC1] = "dc1",
+	[CXL_DECODER_DC2] = "dc2",
+	[CXL_DECODER_DC3] = "dc3",
+	[CXL_DECODER_DC4] = "dc4",
+	[CXL_DECODER_DC5] = "dc5",
+	[CXL_DECODER_DC6] = "dc6",
+	[CXL_DECODER_DC7] = "dc7",
+	[CXL_DECODER_MIXED] = "mixed",
+};
+
 static inline const char *cxl_decoder_mode_name(enum cxl_decoder_mode mode)
 {
-	static const char * const names[] = {
-		[CXL_DECODER_NONE] = "none",
-		[CXL_DECODER_RAM] = "ram",
-		[CXL_DECODER_PMEM] = "pmem",
-		[CXL_DECODER_DC0] = "dc0",
-		[CXL_DECODER_DC1] = "dc1",
-		[CXL_DECODER_DC2] = "dc2",
-		[CXL_DECODER_DC3] = "dc3",
-		[CXL_DECODER_DC4] = "dc4",
-		[CXL_DECODER_DC5] = "dc5",
-		[CXL_DECODER_DC6] = "dc6",
-		[CXL_DECODER_DC7] = "dc7",
-		[CXL_DECODER_MIXED] = "mixed",
-	};
-
 	if (mode >= CXL_DECODER_NONE && mode <= CXL_DECODER_MIXED)
-		return names[mode];
+		return cxl_decoder_mode_names[mode];
 	return "mixed";
 }
 
Signature · 1 line
2.46.0
#62 lore raw

Re: [PATCH v4 14/28] cxl/port: Add endpoint decoder DC mode support to sysfs

Quoted 11 lines · On Mon, 07 Oct 2024 18:16:20 -0500 [email protected] wrote:

From: Navneet Singh <[email protected]>

Endpoint decoder mode is used to represent the partition the decoder points to such as ram or pmem.

Expand the mode to allow a decoder to point to a specific DC partition (Region).

A few comments inline about ways that can make this a little tidier and less fragile.

Jonathan

Quoted 35 lines

---
Changes:
[iweiny: prevent creation of region on shareable DC partitions] [Fan: change mode range logic]
[Fan: use !resource_size()]
[djiang: use the static mode name string array in mode_store()] [Jonathan: remove rc check from mode to region index]
[Jonathan: clarify decoder mode 'mixed']
[djbw: drop cleanup patch and just follow the convention in cxl_dpa_set_mode()] [fan: make dcd resource size check similar to other partitions] [djbw, jonathan, fan: remove mode range check from dc_mode_to_region_index] [iweiny: push sysfs versions to 6.12]

---
 Documentation/ABI/testing/sysfs-bus-cxl | 21 ++++++++++----------
 drivers/cxl/core/hdm.c                  | 17 ++++++++++++++++
 drivers/cxl/core/port.c                 | 10 +++++-----
 drivers/cxl/cxl.h                       | 35 ++++++++++++++++++---------------
 4 files changed, 52 insertions(+), 31 deletions(-)
Documentation/ABI/testing/sysfs-bus-cxl +2 -2
index b865eefdb74c..661dab99183f 100644
@@ -361,23 +361,24 @@ Description:
 
 
 What:		/sys/bus/cxl/devices/decoderX.Y/mode
-Date:		May, 2022
-KernelVersion:	v6.0
+Date:		May, 2022, October 2024
+KernelVersion:	v6.0, v6.12 (dcY)
 Contact:	[email protected]
 Description:
 		(RW) When a CXL decoder is of devtype "cxl_decoder_endpoint" it

Documentation/ABI/testing/sysfs-bus-cxl Description: · line 372

Quoted 10 lines
Documentation/ABI/testing/sysfs-bus-cxl +2 -6
 		translates from a host physical address range, to a device local
 		address range. Device-local address ranges are further split
-		into a 'ram' (volatile memory) range and 'pmem' (persistent
-		memory) range. The 'mode' attribute emits one of 'ram', 'pmem',
-		'mixed', or 'none'. The 'mixed' indication is for error cases
-		when a decoder straddles the volatile/persistent partition
-		boundary, and 'none' indicates the decoder is not actively
-		decoding, or no DPA allocation policy has been set.
+		into a 'ram' (volatile memory) range, 'pmem' (persistent
+		memory) range, or Dynamic Capacity (DC) range.

memory) range, and Dynamic Capacity (DC) ranges.

(doesn't work with preceding text otherwise)

Quoted 29 lines

The 'mode'

Documentation/ABI/testing/sysfs-bus-cxl +7 -2
+		attribute emits one of 'ram', 'pmem', 'dcY', 'mixed', or
+		'none'. The 'mixed' indication is for error cases when a
+		decoder straddles partition boundaries, and 'none' indicates
+		the decoder is not actively decoding, or no DPA allocation
+		policy has been set.
 
 		'mode' can be written, when the decoder is in the 'disabled'
-		state, with either 'ram' or 'pmem' to set the boundaries for the
-		next allocation.
+		state, with 'ram', 'pmem', or 'dcY' to set the boundaries for
+		the next allocation.
 
 
 What:		/sys/bus/cxl/devices/decoderX.Y/dpa_resource
drivers/cxl/core/hdm.c +1 -0
index 8c7f941eaba1..b368babb55d9 100644
@@ -551,6 +551,7 @@ int cxl_dpa_set_mode(struct cxl_endpoint_decoder *cxled,
 	switch (mode) {
 	case CXL_DECODER_RAM:
 	case CXL_DECODER_PMEM:
+	case CXL_DECODER_DC0 ... CXL_DECODER_DC7:
 		break;
 	default:
 		dev_dbg(dev, "unsupported mode: %d\n", mode);
@@ -578,6 +579,22 @@ int cxl_dpa_set_mode(struct cxl_endpoint_decoder *cxled,
 		goto out;

drivers/cxl/core/hdm.c int cxl_dpa_set_mode(struct cxl_endpoint_decoder *cxled, · line 589

Quoted 10 lines
drivers/cxl/core/hdm.c +8 -0
 	}
 
+	if (mode >= CXL_DECODER_DC0 && mode <= CXL_DECODER_DC7) {
+		struct cxl_memdev_state *mds = to_cxl_memdev_state(cxlds);
+
+		rc = dc_mode_to_region_index(mode);
+		if (!resource_size(&cxlds->dc_res[rc])) {
+			dev_dbg(dev, "no available dynamic capacity\n");
+			rc = -ENXIO;
+			goto out;

Probably worth adding a precursor patch that uses guard(rwsem_write) on the cxl_dpa_rwsem
Allows for early returns simplifying existing code and this.

Quoted 18 lines
drivers/cxl/core/hdm.c +8 -0
+		}
+		if (mds->dc_region[rc].shareable) {
+			dev_err(dev, "DC region %d is shareable\n", rc);
+			rc = -EINVAL;
+			goto out;
+		}
+	}
+
 	cxled->mode = mode;
 	rc = 0;
 out:
drivers/cxl/core/port.c +0 -0
index 85b912c11f04..23b4f266a83a 100644
@@ -205,11 +205,11 @@ static ssize_t mode_store(struct device *dev, struct device_attribute *attr,
 	enum cxl_decoder_mode mode;
 	ssize_t rc;

drivers/cxl/core/port.c static ssize_t mode_store(struct device *dev, struct device_attribute *attr, · line 211

Quoted 10 lines
drivers/cxl/core/port.c +4 -5
 
-	if (sysfs_streq(buf, "pmem"))
-		mode = CXL_DECODER_PMEM;
-	else if (sysfs_streq(buf, "ram"))
-		mode = CXL_DECODER_RAM;
-	else
+	for (mode = CXL_DECODER_RAM; mode < CXL_DECODER_MIXED; mode++)
+		if (sysfs_streq(buf, cxl_decoder_mode_names[mode]))
+			break;
+

Loop over them all then do what you have here but explicit matches to reject the ones that can't be set.
Add a MODE_COUNT to the end of the options.

	for (mode = 0; mode < CXL_DECODER_MODE_COUNT; mode++)
		if (sysfs_streq(buf, cxl_decoder_mode_names[mode]))
			break;

	if (mode == CXL_DECODER_MODE_COUNT)
		return -EINVAL;

	if (mode == CXL_DECODER_NONE)
		return -EINVAL;

	/* Not yet supported */
	if (mode == CXL_DECODER_MIXED)
		return -EINVAL;

...

Quoted 9 lines
drivers/cxl/core/port.c +1 -0
+	if (mode >= CXL_DECODER_MIXED)
 		return -EINVAL;
 
 	rc = cxl_dpa_set_mode(cxled, mode);
drivers/cxl/cxl.h +0 -0
index 8b7099c38a40..cbaacbe0f36d 100644

drivers/cxl/cxl.h struct cxl_decoder { · line 369

Quoted 5 lines
drivers/cxl/cxl.h +2 -0
@@ -365,6 +365,9 @@ struct cxl_decoder {
 /*
  * CXL_DECODER_DEAD prevents endpoints from being reattached to regions
  * while cxld_unregister() is running
+ *
+ * NOTE: CXL_DECODER_RAM must be second and CXL_DECODER_MIXED must be last.

This is a bit ugly. I'd change the logic a bit to avoid it. The list of things we don't support is short so just check for them. See above.

Quoted 47 lines
drivers/cxl/cxl.h +17 -16
+ *	 See mode_store()
  */
 enum cxl_decoder_mode {
 	CXL_DECODER_NONE,
@@ -382,25 +385,25 @@ enum cxl_decoder_mode {
 	CXL_DECODER_DEAD,
 };
 
+static const char * const cxl_decoder_mode_names[] = {
+	[CXL_DECODER_NONE] = "none",
+	[CXL_DECODER_RAM] = "ram",
+	[CXL_DECODER_PMEM] = "pmem",
+	[CXL_DECODER_DC0] = "dc0",
+	[CXL_DECODER_DC1] = "dc1",
+	[CXL_DECODER_DC2] = "dc2",
+	[CXL_DECODER_DC3] = "dc3",
+	[CXL_DECODER_DC4] = "dc4",
+	[CXL_DECODER_DC5] = "dc5",
+	[CXL_DECODER_DC6] = "dc6",
+	[CXL_DECODER_DC7] = "dc7",
+	[CXL_DECODER_MIXED] = "mixed",
+};
+
 static inline const char *cxl_decoder_mode_name(enum cxl_decoder_mode mode)
 {
-	static const char * const names[] = {
-		[CXL_DECODER_NONE] = "none",
-		[CXL_DECODER_RAM] = "ram",
-		[CXL_DECODER_PMEM] = "pmem",
-		[CXL_DECODER_DC0] = "dc0",
-		[CXL_DECODER_DC1] = "dc1",
-		[CXL_DECODER_DC2] = "dc2",
-		[CXL_DECODER_DC3] = "dc3",
-		[CXL_DECODER_DC4] = "dc4",
-		[CXL_DECODER_DC5] = "dc5",
-		[CXL_DECODER_DC6] = "dc6",
-		[CXL_DECODER_DC7] = "dc7",
-		[CXL_DECODER_MIXED] = "mixed",
-	};
-
 	if (mode >= CXL_DECODER_NONE && mode <= CXL_DECODER_MIXED)
-		return names[mode];
+		return cxl_decoder_mode_names[mode];
 	return "mixed";
 }
 
author
#63 lore raw

Re: [PATCH v4 14/28] cxl/port: Add endpoint decoder DC mode support to sysfs

Quoted 11 lines · Jonathan Cameron
Quoted 8 lines · On Mon, 07 Oct 2024 18:16:20 -0500 [email protected] wrote:

From: Navneet Singh <[email protected]>

Endpoint decoder mode is used to represent the partition the decoder points to such as ram or pmem.

Expand the mode to allow a decoder to point to a specific DC partition (Region).

Quoted 6 lines
Quoted 3 lines

A few comments inline about ways that can make this a little tidier and less fragile.

All Done. Yea good idea on the enum.

Ira

[snip]

Quoted 15 lines

Documentation/ABI/testing/sysfs-bus-cxl Description:

Quoted 11 lines
Documentation/ABI/testing/sysfs-bus-cxl +2 -6
 		(RW) When a CXL decoder is of devtype "cxl_decoder_endpoint" it
 		translates from a host physical address range, to a device local
 		address range. Device-local address ranges are further split
-		into a 'ram' (volatile memory) range and 'pmem' (persistent
-		memory) range. The 'mode' attribute emits one of 'ram', 'pmem',
-		'mixed', or 'none'. The 'mixed' indication is for error cases
-		when a decoder straddles the volatile/persistent partition
-		boundary, and 'none' indicates the decoder is not actively
-		decoding, or no DPA allocation policy has been set.
+		into a 'ram' (volatile memory) range, 'pmem' (persistent
+		memory) range, or Dynamic Capacity (DC) range.

memory) range, and Dynamic Capacity (DC) ranges.

(doesn't work with preceding text otherwise)

[snip]

Quoted 11 lines

drivers/cxl/core/hdm.c int cxl_dpa_set_mode(struct cxl_endpoint_decoder *cxled, · line 589

Quoted 8 lines
drivers/cxl/core/hdm.c +8 -0
+	if (mode >= CXL_DECODER_DC0 && mode <= CXL_DECODER_DC7) {
+		struct cxl_memdev_state *mds = to_cxl_memdev_state(cxlds);
+
+		rc = dc_mode_to_region_index(mode);
+		if (!resource_size(&cxlds->dc_res[rc])) {
+			dev_dbg(dev, "no available dynamic capacity\n");
+			rc = -ENXIO;
+			goto out;

Probably worth adding a precursor patch that uses guard(rwsem_write) on the cxl_dpa_rwsem
Allows for early returns simplifying existing code and this.

[snip]

Quoted 54 lines
Quoted 7 lines
drivers/cxl/core/port.c +0 -0
index 85b912c11f04..23b4f266a83a 100644
@@ -205,11 +205,11 @@ static ssize_t mode_store(struct device *dev, struct device_attribute *attr,
 	enum cxl_decoder_mode mode;
 	ssize_t rc;

drivers/cxl/core/port.c static ssize_t mode_store(struct device *dev, struct device_attribute *attr, · line 211

Quoted 10 lines
drivers/cxl/core/port.c +4 -5
 
-	if (sysfs_streq(buf, "pmem"))
-		mode = CXL_DECODER_PMEM;
-	else if (sysfs_streq(buf, "ram"))
-		mode = CXL_DECODER_RAM;
-	else
+	for (mode = CXL_DECODER_RAM; mode < CXL_DECODER_MIXED; mode++)
+		if (sysfs_streq(buf, cxl_decoder_mode_names[mode]))
+			break;
+

Loop over them all then do what you have here but explicit matches to reject the ones that can't be set.
Add a MODE_COUNT to the end of the options.

	for (mode = 0; mode < CXL_DECODER_MODE_COUNT; mode++)
		if (sysfs_streq(buf, cxl_decoder_mode_names[mode]))
			break;

	if (mode == CXL_DECODER_MODE_COUNT)
		return -EINVAL;

	if (mode == CXL_DECODER_NONE)
		return -EINVAL;

	/* Not yet supported */
	if (mode == CXL_DECODER_MIXED)
		return -EINVAL;

...

Quoted 9 lines
drivers/cxl/core/port.c +1 -0
+	if (mode >= CXL_DECODER_MIXED)
 		return -EINVAL;
 
 	rc = cxl_dpa_set_mode(cxled, mode);
drivers/cxl/cxl.h +0 -0
index 8b7099c38a40..cbaacbe0f36d 100644

drivers/cxl/cxl.h struct cxl_decoder { · line 369

Quoted 5 lines
drivers/cxl/cxl.h +2 -0
@@ -365,6 +365,9 @@ struct cxl_decoder {
 /*
  * CXL_DECODER_DEAD prevents endpoints from being reattached to regions
  * while cxld_unregister() is running
+ *
+ * NOTE: CXL_DECODER_RAM must be second and CXL_DECODER_MIXED must be last.

This is a bit ugly. I'd change the logic a bit to avoid it. The list of things we don't support is short so just check for them. See above.

[snip]

author
#64 lore raw

PATCH v4 15/28 cxl/region: Refactor common create region code

In reply to Ira Weiny

create_pmem_region_store() and create_ram_region_store() are identical with the exception of the region mode. With the addition of DC region mode this would end up being 3 copies of the same code.

Refactor create_pmem_region_store() and create_ram_region_store() to use a single common function to be used in subsequent DC code.

---
 drivers/cxl/core/region.c | 28 +++++++++++-----------------
 1 file changed, 11 insertions(+), 17 deletions(-)
drivers/cxl/core/region.c +11 -17
index ab00203f285a..2ca6148d108c 100644
@@ -2552,9 +2552,8 @@ static struct cxl_region *__create_region(struct cxl_root_decoder *cxlrd,
 	return devm_cxl_add_region(cxlrd, id, mode, CXL_DECODER_HOSTONLYMEM);
 }
 
-static ssize_t create_pmem_region_store(struct device *dev,
-					struct device_attribute *attr,
-					const char *buf, size_t len)
+static ssize_t create_region_store(struct device *dev, const char *buf,
+				   size_t len, enum cxl_region_mode mode)
 {
 	struct cxl_root_decoder *cxlrd = to_cxl_root_decoder(dev);
 	struct cxl_region *cxlr;
@@ -2564,31 +2563,26 @@ static ssize_t create_pmem_region_store(struct device *dev,
 	if (rc != 1)
 		return -EINVAL;
 
-	cxlr = __create_region(cxlrd, CXL_REGION_PMEM, id);
+	cxlr = __create_region(cxlrd, mode, id);
 	if (IS_ERR(cxlr))
 		return PTR_ERR(cxlr);
 
 	return len;
 }
+
+static ssize_t create_pmem_region_store(struct device *dev,
+					struct device_attribute *attr,
+					const char *buf, size_t len)
+{
+	return create_region_store(dev, buf, len, CXL_REGION_PMEM);
+}
 DEVICE_ATTR_RW(create_pmem_region);
 
 static ssize_t create_ram_region_store(struct device *dev,
 				       struct device_attribute *attr,
 				       const char *buf, size_t len)
 {
-	struct cxl_root_decoder *cxlrd = to_cxl_root_decoder(dev);
-	struct cxl_region *cxlr;
-	int rc, id;
-
-	rc = sscanf(buf, "region%d\n", &id);
-	if (rc != 1)
-		return -EINVAL;
-
-	cxlr = __create_region(cxlrd, CXL_REGION_RAM, id);
-	if (IS_ERR(cxlr))
-		return PTR_ERR(cxlr);
-
-	return len;
+	return create_region_store(dev, buf, len, CXL_REGION_RAM);
 }
 DEVICE_ATTR_RW(create_ram_region);
 
Signature · 1 line
2.46.0
Reviewed-by on v4 15/28
#65 lore raw

Re: [PATCH v4 15/28] cxl/region: Refactor common create region code

Quoted 9 lines · On Mon, 07 Oct 2024 18:16:21 -0500 Ira Weiny <[email protected]> wrote:

create_pmem_region_store() and create_ram_region_store() are identical with the exception of the region mode. With the addition of DC region mode this would end up being 3 copies of the same code.

Refactor create_pmem_region_store() and create_ram_region_store() to use a single common function to be used in subsequent DC code.

Nice.

Is it worth dragging out cleanup like this to the start of the series so Dave can queue it up as 'good to have whatever' and reduce this set a bit?

Jonathan

Quoted 44 lines
---
 drivers/cxl/core/region.c | 28 +++++++++++-----------------
 1 file changed, 11 insertions(+), 17 deletions(-)
drivers/cxl/core/region.c +8 -13
index ab00203f285a..2ca6148d108c 100644
@@ -2552,9 +2552,8 @@ static struct cxl_region *__create_region(struct cxl_root_decoder *cxlrd,
 	return devm_cxl_add_region(cxlrd, id, mode, CXL_DECODER_HOSTONLYMEM);
 }
 
 
+
+static ssize_t create_pmem_region_store(struct device *dev,
+					struct device_attribute *attr,
+					const char *buf, size_t len)
+{
+	return create_region_store(dev, buf, len, CXL_REGION_PMEM);
+}
 DEVICE_ATTR_RW(create_pmem_region);
 
 static ssize_t create_ram_region_store(struct device *dev,
 				       struct device_attribute *attr,
 				       const char *buf, size_t len)
 {
-	struct cxl_root_decoder *cxlrd = to_cxl_root_decoder(dev);
-	struct cxl_region *cxlr;
-	int rc, id;
-
-	rc = sscanf(buf, "region%d\n", &id);
-	if (rc != 1)
-		return -EINVAL;
-
-	cxlr = __create_region(cxlrd, CXL_REGION_RAM, id);
-	if (IS_ERR(cxlr))
-		return PTR_ERR(cxlr);
-
-	return len;
+	return create_region_store(dev, buf, len, CXL_REGION_RAM);
 }
 DEVICE_ATTR_RW(create_ram_region);
author
#66 lore raw

Re: [PATCH v4 15/28] cxl/region: Refactor common create region code

Quoted 10 lines · Jonathan Cameron
Quoted 7 lines · On Mon, 07 Oct 2024 18:16:21 -0500 Ira Weiny <[email protected]> wrote:

create_pmem_region_store() and create_ram_region_store() are identical with the exception of the region mode. With the addition of DC region mode this would end up being 3 copies of the same code.

Refactor create_pmem_region_store() and create_ram_region_store() to use a single common function to be used in subsequent DC code.

Quoted 8 lines
Quoted 2 lines

Nice.

Is it worth dragging out cleanup like this to the start of the series so Dave can queue it up as 'good to have whatever' and reduce this set a bit?

The problem was that this patch depended on the region mode change... But that was an easy change.

I've moved it forward.

Ira

Reviewed-by on v4 15/28
#67 lore raw

Re: [PATCH v4 15/28] cxl/region: Refactor common create region code

In reply to Ira Weiny
Quoted 10 lines · Ira Weiny

create_pmem_region_store() and create_ram_region_store() are identical with the exception of the region mode. With the addition of DC region mode this would end up being 3 copies of the same code.

Refactor create_pmem_region_store() and create_ram_region_store() to use a single common function to be used in subsequent DC code.

---

Quoted 64 lines
 drivers/cxl/core/region.c | 28 +++++++++++-----------------
 1 file changed, 11 insertions(+), 17 deletions(-)
drivers/cxl/core/region.c +11 -17
index ab00203f285a..2ca6148d108c 100644
@@ -2552,9 +2552,8 @@ static struct cxl_region *__create_region(struct cxl_root_decoder *cxlrd,
 	return devm_cxl_add_region(cxlrd, id, mode, CXL_DECODER_HOSTONLYMEM);
 }
 
-static ssize_t create_pmem_region_store(struct device *dev,
-					struct device_attribute *attr,
-					const char *buf, size_t len)
+static ssize_t create_region_store(struct device *dev, const char *buf,
+				   size_t len, enum cxl_region_mode mode)
 {
 	struct cxl_root_decoder *cxlrd = to_cxl_root_decoder(dev);
 	struct cxl_region *cxlr;
@@ -2564,31 +2563,26 @@ static ssize_t create_pmem_region_store(struct device *dev,
 	if (rc != 1)
 		return -EINVAL;
 
-	cxlr = __create_region(cxlrd, CXL_REGION_PMEM, id);
+	cxlr = __create_region(cxlrd, mode, id);
 	if (IS_ERR(cxlr))
 		return PTR_ERR(cxlr);
 
 	return len;
 }
+
+static ssize_t create_pmem_region_store(struct device *dev,
+					struct device_attribute *attr,
+					const char *buf, size_t len)
+{
+	return create_region_store(dev, buf, len, CXL_REGION_PMEM);
+}
 DEVICE_ATTR_RW(create_pmem_region);
 
 static ssize_t create_ram_region_store(struct device *dev,
 				       struct device_attribute *attr,
 				       const char *buf, size_t len)
 {
-	struct cxl_root_decoder *cxlrd = to_cxl_root_decoder(dev);
-	struct cxl_region *cxlr;
-	int rc, id;
-
-	rc = sscanf(buf, "region%d\n", &id);
-	if (rc != 1)
-		return -EINVAL;
-
-	cxlr = __create_region(cxlrd, CXL_REGION_RAM, id);
-	if (IS_ERR(cxlr))
-		return PTR_ERR(cxlr);
-
-	return len;
+	return create_region_store(dev, buf, len, CXL_REGION_RAM);
 }
 DEVICE_ATTR_RW(create_ram_region);
 
Signature · 2 lines
-- 
2.46.0
Signature · 1 line
Fan Ni
maintainer Reviewed-by on v4 15/28
#68 lore raw

Re: [PATCH v4 15/28] cxl/region: Refactor common create region code

In reply to Ira Weiny
Quoted 6 lines · Ira Weiny

create_pmem_region_store() and create_ram_region_store() are identical with the exception of the region mode. With the addition of DC region mode this would end up being 3 copies of the same code.

Refactor create_pmem_region_store() and create_ram_region_store() to use a single common function to be used in subsequent DC code.

--snip

#69 lore raw

PATCH v4 16/28 cxl/region: Add sparse DAX region support

In reply to Ira Weiny

From: Navneet Singh <[email protected]>

Dynamic Capacity CXL regions must allow memory to be added or removed dynamically. In addition to the quantity of memory available the location of the memory within a DC partition is dynamic based on the extents offered by a device. CXL DAX regions must accommodate the sparseness of this memory in the management of DAX regions and devices.

Introduce the concept of a sparse DAX region. Add a create_dc_region() sysfs entry to create such regions. Special case DC capable regions to create a 0 sized seed DAX device to maintain compatibility which requires a default DAX device to hold a region reference.

Indicate 0 byte available capacity until such time that capacity is added.

Sparse regions complicate the range mapping of dax devices. There is no known use case for range mapping on sparse regions. Avoid the complication by preventing range mapping of dax devices on sparse regions.

Interleaving is deferred for now. Add checks.

---
Changes:
[Fan: use single function for dc region store]
[djiang: avoid setting dev_size twice]
[djbw: Check DCD support and interleave restriction on region creation] [iweiny: squash patch : dax/region: Prevent range mapping allocation on sparse regions] [iwieny: remove reviews]
[iweiny: rebase to master]
[iweiny: push sysfs version to 6.12]
[iweiny: make cxled_to_mds inline]

---
 Documentation/ABI/testing/sysfs-bus-cxl | 22 ++++++++--------
 drivers/cxl/core/core.h                 | 12 +++++++++
 drivers/cxl/core/port.c                 |  1 +
 drivers/cxl/core/region.c               | 46 +++++++++++++++++++++++++++++++--
 drivers/dax/bus.c                       | 10 +++++++
 drivers/dax/bus.h                       |  1 +
 drivers/dax/cxl.c                       | 16 ++++++++++--
 7 files changed, 93 insertions(+), 15 deletions(-)
Documentation/ABI/testing/sysfs-bus-cxl +11 -11
index 661dab99183f..b63ab622515f 100644
@@ -439,20 +439,20 @@ Description:
 		interleave_granularity).
 
 
-What:		/sys/bus/cxl/devices/decoderX.Y/create_{pmem,ram}_region
-Date:		May, 2022, January, 2023
-KernelVersion:	v6.0 (pmem), v6.3 (ram)
+What:		/sys/bus/cxl/devices/decoderX.Y/create_{pmem,ram,dc}_region
+Date:		May, 2022, January, 2023, August 2024
+KernelVersion:	v6.0 (pmem), v6.3 (ram), v6.12 (dc)
 Contact:	[email protected]
 Description:
 		(RW) Write a string in the form 'regionZ' to start the process
-		of defining a new persistent, or volatile memory region
-		(interleave-set) within the decode range bounded by root decoder
-		'decoderX.Y'. The value written must match the current value
-		returned from reading this attribute. An atomic compare exchange
-		operation is done on write to assign the requested id to a
-		region and allocate the region-id for the next creation attempt.
-		EBUSY is returned if the region name written does not match the
-		current cached value.
+		of defining a new persistent, volatile, or Dynamic Capacity
+		(DC) memory region (interleave-set) within the decode range
+		bounded by root decoder 'decoderX.Y'. The value written must
+		match the current value returned from reading this attribute.
+		An atomic compare exchange operation is done on write to assign
+		the requested id to a region and allocate the region-id for the
+		next creation attempt.  EBUSY is returned if the region name
+		written does not match the current cached value.
 
 
 What:		/sys/bus/cxl/devices/decoderX.Y/delete_region
drivers/cxl/core/core.h +12 -0
index 0c62b4069ba0..5d6fe7ab0a78 100644
@@ -4,15 +4,27 @@
 #ifndef __CXL_CORE_H__
 #define __CXL_CORE_H__
 
+#include <cxlmem.h>
+
 extern const struct device_type cxl_nvdimm_bridge_type;
 extern const struct device_type cxl_nvdimm_type;
 extern const struct device_type cxl_pmu_type;
 
 extern struct attribute_group cxl_base_attribute_group;
 
+static inline struct cxl_memdev_state *
+cxled_to_mds(struct cxl_endpoint_decoder *cxled)
+{
+	struct cxl_memdev *cxlmd = cxled_to_memdev(cxled);
+	struct cxl_dev_state *cxlds = cxlmd->cxlds;
+
+	return container_of(cxlds, struct cxl_memdev_state, cxlds);
+}
+
 #ifdef CONFIG_CXL_REGION
 extern struct device_attribute dev_attr_create_pmem_region;
 extern struct device_attribute dev_attr_create_ram_region;
+extern struct device_attribute dev_attr_create_dc_region;
 extern struct device_attribute dev_attr_delete_region;
 extern struct device_attribute dev_attr_region;
 extern const struct device_type cxl_pmem_region_type;
drivers/cxl/core/port.c +1 -0
index 23b4f266a83a..fefa592e9159 100644
@@ -320,6 +320,7 @@ static struct attribute *cxl_decoder_root_attrs[] = {
 	&dev_attr_qos_class.attr,
 	SET_CXL_REGION_ATTR(create_pmem_region)
 	SET_CXL_REGION_ATTR(create_ram_region)
+	SET_CXL_REGION_ATTR(create_dc_region)
 	SET_CXL_REGION_ATTR(delete_region)
 	NULL,
 };
drivers/cxl/core/region.c +44 -2

94 lines not on this page · raw message

drivers/dax/bus.c +10 -0
index fde29e0ad68b..d8cb5195a227 100644
@@ -178,6 +178,11 @@ static bool is_static(struct dax_region *dax_region)
 	return (dax_region->res.flags & IORESOURCE_DAX_STATIC) != 0;
 }
 
+static bool is_sparse(struct dax_region *dax_region)
+{
+	return (dax_region->res.flags & IORESOURCE_DAX_SPARSE_CAP) != 0;
+}
+
 bool static_dev_dax(struct dev_dax *dev_dax)
 {
 	return is_static(dev_dax->region);
@@ -301,6 +306,9 @@ static unsigned long long dax_region_avail_size(struct dax_region *dax_region)
 
 	lockdep_assert_held(&dax_region_rwsem);
 
+	if (is_sparse(dax_region))
+		return 0;
+
 	for_each_dax_region_resource(dax_region, res)
 		size -= resource_size(res);
 	return size;
@@ -1373,6 +1381,8 @@ static umode_t dev_dax_visible(struct kobject *kobj, struct attribute *a, int n)
 		return 0;
 	if (a == &dev_attr_mapping.attr && is_static(dax_region))
 		return 0;
+	if (a == &dev_attr_mapping.attr && is_sparse(dax_region))
+		return 0;
 	if ((a == &dev_attr_align.attr ||
 	     a == &dev_attr_size.attr) && is_static(dax_region))
 		return 0444;
drivers/dax/bus.h +1 -0
index cbbf64443098..783bfeef42cc 100644
@@ -13,6 +13,7 @@ struct dax_region;
 /* dax bus specific ioresource flags */
 #define IORESOURCE_DAX_STATIC BIT(0)
 #define IORESOURCE_DAX_KMEM BIT(1)
+#define IORESOURCE_DAX_SPARSE_CAP BIT(2)
 
 struct dax_region *alloc_dax_region(struct device *parent, int region_id,
 		struct range *range, int target_node, unsigned int align,
drivers/dax/cxl.c +14 -2
index 9b29e732b39a..367e86b1c22a 100644
@@ -13,19 +13,31 @@ static int cxl_dax_region_probe(struct device *dev)
 	struct cxl_region *cxlr = cxlr_dax->cxlr;
 	struct dax_region *dax_region;
 	struct dev_dax_data data;
+	resource_size_t dev_size;
+	unsigned long flags;
 
 	if (nid == NUMA_NO_NODE)
 		nid = memory_add_physaddr_to_nid(cxlr_dax->hpa_range.start);
 
+	flags = IORESOURCE_DAX_KMEM;
+	if (cxlr->mode == CXL_REGION_DC)
+		flags |= IORESOURCE_DAX_SPARSE_CAP;
+
 	dax_region = alloc_dax_region(dev, cxlr->id, &cxlr_dax->hpa_range, nid,
-				      PMD_SIZE, IORESOURCE_DAX_KMEM);
+				      PMD_SIZE, flags);
 	if (!dax_region)
 		return -ENOMEM;
 
+	if (cxlr->mode == CXL_REGION_DC)
+		/* Add empty seed dax device */
+		dev_size = 0;
+	else
+		dev_size = range_len(&cxlr_dax->hpa_range);
+
 	data = (struct dev_dax_data) {
 		.dax_region = dax_region,
 		.id = -1,
-		.size = range_len(&cxlr_dax->hpa_range),
+		.size = dev_size,
 		.memmap_on_memory = true,
 	};
 
Signature · 1 line
2.46.0
Reviewed-by on v4 16/28
#70 lore raw

Re: [PATCH v4 16/28] cxl/region: Add sparse DAX region support

Quoted 21 lines · On Mon, 07 Oct 2024 18:16:22 -0500 [email protected] wrote:

From: Navneet Singh <[email protected]>

Dynamic Capacity CXL regions must allow memory to be added or removed dynamically. In addition to the quantity of memory available the location of the memory within a DC partition is dynamic based on the extents offered by a device. CXL DAX regions must accommodate the sparseness of this memory in the management of DAX regions and devices.

Introduce the concept of a sparse DAX region. Add a create_dc_region() sysfs entry to create such regions. Special case DC capable regions to create a 0 sized seed DAX device to maintain compatibility which requires a default DAX device to hold a region reference.

Indicate 0 byte available capacity until such time that capacity is added.

Sparse regions complicate the range mapping of dax devices. There is no known use case for range mapping on sparse regions. Avoid the complication by preventing range mapping of dax devices on sparse regions.

Quoted 5 lines

Interleaving is deferred for now. Add checks.

Looks fine to me, though the DAX bit is not an area I know enough about.

#71 lore raw

Re: [PATCH v4 16/28] cxl/region: Add sparse DAX region support

In reply to [email protected]
Quoted 21 lines · On Mon, Oct 07, 2024 at 06:16:22PM -0500, [email protected] wrote:

From: Navneet Singh <[email protected]>

Dynamic Capacity CXL regions must allow memory to be added or removed dynamically. In addition to the quantity of memory available the location of the memory within a DC partition is dynamic based on the extents offered by a device. CXL DAX regions must accommodate the sparseness of this memory in the management of DAX regions and devices.

Introduce the concept of a sparse DAX region. Add a create_dc_region() sysfs entry to create such regions. Special case DC capable regions to create a 0 sized seed DAX device to maintain compatibility which requires a default DAX device to hold a region reference.

Indicate 0 byte available capacity until such time that capacity is added.

Sparse regions complicate the range mapping of dax devices. There is no known use case for range mapping on sparse regions. Avoid the complication by preventing range mapping of dax devices on sparse regions.

Quoted 6 lines

Interleaving is deferred for now. Add checks.

LGTM, and behaves as expected when tested with qemu setup.

Fan

Quoted 295 lines

---
Changes:
[Fan: use single function for dc region store]
[djiang: avoid setting dev_size twice]
[djbw: Check DCD support and interleave restriction on region creation] [iweiny: squash patch : dax/region: Prevent range mapping allocation on sparse regions] [iwieny: remove reviews]
[iweiny: rebase to master]
[iweiny: push sysfs version to 6.12]
[iweiny: make cxled_to_mds inline]

---
 Documentation/ABI/testing/sysfs-bus-cxl | 22 ++++++++--------
 drivers/cxl/core/core.h                 | 12 +++++++++
 drivers/cxl/core/port.c                 |  1 +
 drivers/cxl/core/region.c               | 46 +++++++++++++++++++++++++++++++--
 drivers/dax/bus.c                       | 10 +++++++
 drivers/dax/bus.h                       |  1 +
 drivers/dax/cxl.c                       | 16 ++++++++++--
 7 files changed, 93 insertions(+), 15 deletions(-)
Documentation/ABI/testing/sysfs-bus-cxl +11 -11
index 661dab99183f..b63ab622515f 100644
@@ -439,20 +439,20 @@ Description:
 		interleave_granularity).
 
 
-What:		/sys/bus/cxl/devices/decoderX.Y/create_{pmem,ram}_region
-Date:		May, 2022, January, 2023
-KernelVersion:	v6.0 (pmem), v6.3 (ram)
+What:		/sys/bus/cxl/devices/decoderX.Y/create_{pmem,ram,dc}_region
+Date:		May, 2022, January, 2023, August 2024
+KernelVersion:	v6.0 (pmem), v6.3 (ram), v6.12 (dc)
 Contact:	[email protected]
 Description:
 		(RW) Write a string in the form 'regionZ' to start the process
-		of defining a new persistent, or volatile memory region
-		(interleave-set) within the decode range bounded by root decoder
-		'decoderX.Y'. The value written must match the current value
-		returned from reading this attribute. An atomic compare exchange
-		operation is done on write to assign the requested id to a
-		region and allocate the region-id for the next creation attempt.
-		EBUSY is returned if the region name written does not match the
-		current cached value.
+		of defining a new persistent, volatile, or Dynamic Capacity
+		(DC) memory region (interleave-set) within the decode range
+		bounded by root decoder 'decoderX.Y'. The value written must
+		match the current value returned from reading this attribute.
+		An atomic compare exchange operation is done on write to assign
+		the requested id to a region and allocate the region-id for the
+		next creation attempt.  EBUSY is returned if the region name
+		written does not match the current cached value.
 
 
 What:		/sys/bus/cxl/devices/decoderX.Y/delete_region
drivers/cxl/core/core.h +12 -0
index 0c62b4069ba0..5d6fe7ab0a78 100644
@@ -4,15 +4,27 @@
 #ifndef __CXL_CORE_H__
 #define __CXL_CORE_H__
 
+#include <cxlmem.h>
+
 extern const struct device_type cxl_nvdimm_bridge_type;
 extern const struct device_type cxl_nvdimm_type;
 extern const struct device_type cxl_pmu_type;
 
 extern struct attribute_group cxl_base_attribute_group;
 
+static inline struct cxl_memdev_state *
+cxled_to_mds(struct cxl_endpoint_decoder *cxled)
+{
+	struct cxl_memdev *cxlmd = cxled_to_memdev(cxled);
+	struct cxl_dev_state *cxlds = cxlmd->cxlds;
+
+	return container_of(cxlds, struct cxl_memdev_state, cxlds);
+}
+
 #ifdef CONFIG_CXL_REGION
 extern struct device_attribute dev_attr_create_pmem_region;
 extern struct device_attribute dev_attr_create_ram_region;
+extern struct device_attribute dev_attr_create_dc_region;
 extern struct device_attribute dev_attr_delete_region;
 extern struct device_attribute dev_attr_region;
 extern const struct device_type cxl_pmem_region_type;
drivers/cxl/core/port.c +1 -0
index 23b4f266a83a..fefa592e9159 100644
@@ -320,6 +320,7 @@ static struct attribute *cxl_decoder_root_attrs[] = {
 	&dev_attr_qos_class.attr,
 	SET_CXL_REGION_ATTR(create_pmem_region)
 	SET_CXL_REGION_ATTR(create_ram_region)
+	SET_CXL_REGION_ATTR(create_dc_region)
 	SET_CXL_REGION_ATTR(delete_region)
 	NULL,
 };
drivers/cxl/core/region.c +44 -2
index 2ca6148d108c..34a6f447e75b 100644
@@ -496,6 +496,11 @@ static ssize_t interleave_ways_store(struct device *dev,
 	if (rc)
 		return rc;
 
+	if (cxlr->mode == CXL_REGION_DC && val != 1) {
+		dev_err(dev, "Interleaving and DCD not supported\n");
+		return -EINVAL;
+	}
+
 	rc = ways_to_eiw(val, &iw);
 	if (rc)
 		return rc;
@@ -2176,6 +2181,7 @@ static size_t store_targetN(struct cxl_region *cxlr, const char *buf, int pos,
 	if (sysfs_streq(buf, "\n"))
 		rc = detach_target(cxlr, pos);
 	else {
+		struct cxl_endpoint_decoder *cxled;
 		struct device *dev;
 
 		dev = bus_find_device_by_name(&cxl_bus_type, NULL, buf);
@@ -2187,8 +2193,13 @@ static size_t store_targetN(struct cxl_region *cxlr, const char *buf, int pos,
 			goto out;
 		}
 
-		rc = attach_target(cxlr, to_cxl_endpoint_decoder(dev), pos,
-				   TASK_INTERRUPTIBLE);
+		cxled = to_cxl_endpoint_decoder(dev);
+		if (cxlr->mode == CXL_REGION_DC &&
+		    !cxl_dcd_supported(cxled_to_mds(cxled))) {
+			dev_dbg(dev, "DCD unsupported\n");
+			return -EINVAL;
+		}
+		rc = attach_target(cxlr, cxled, pos, TASK_INTERRUPTIBLE);
 out:
 		put_device(dev);
 	}
@@ -2533,6 +2544,7 @@ static struct cxl_region *__create_region(struct cxl_root_decoder *cxlrd,
 	switch (mode) {
 	case CXL_REGION_RAM:
 	case CXL_REGION_PMEM:
+	case CXL_REGION_DC:
 		break;
 	default:
 		dev_err(&cxlrd->cxlsd.cxld.dev, "unsupported mode %s\n",
@@ -2586,6 +2598,20 @@ static ssize_t create_ram_region_store(struct device *dev,
 }
 DEVICE_ATTR_RW(create_ram_region);
 
+static ssize_t create_dc_region_show(struct device *dev,
+				     struct device_attribute *attr, char *buf)
+{
+	return __create_region_show(to_cxl_root_decoder(dev), buf);
+}
+
+static ssize_t create_dc_region_store(struct device *dev,
+				      struct device_attribute *attr,
+				      const char *buf, size_t len)
+{
+	return create_region_store(dev, buf, len, CXL_REGION_DC);
+}
+DEVICE_ATTR_RW(create_dc_region);
+
 static ssize_t region_show(struct device *dev, struct device_attribute *attr,
 			   char *buf)
 {
@@ -3168,6 +3194,11 @@ static int devm_cxl_add_dax_region(struct cxl_region *cxlr)
 	struct device *dev;
 	int rc;
 
+	if (cxlr->mode == CXL_REGION_DC && cxlr->params.interleave_ways != 1) {
+		dev_err(&cxlr->dev, "Interleaving DC not supported\n");
+		return -EINVAL;
+	}
+
 	cxlr_dax = cxl_dax_region_alloc(cxlr);
 	if (IS_ERR(cxlr_dax))
 		return PTR_ERR(cxlr_dax);
@@ -3260,6 +3291,16 @@ static struct cxl_region *construct_region(struct cxl_root_decoder *cxlrd,
 		return ERR_PTR(-EINVAL);
 
 	mode = cxl_decoder_to_region_mode(cxled->mode);
+	if (mode == CXL_REGION_DC) {
+		if (!cxl_dcd_supported(cxled_to_mds(cxled))) {
+			dev_err(&cxled->cxld.dev, "DCD unsupported\n");
+			return ERR_PTR(-EINVAL);
+		}
+		if (cxled->cxld.interleave_ways != 1) {
+			dev_err(&cxled->cxld.dev, "Interleaving and DCD not supported\n");
+			return ERR_PTR(-EINVAL);
+		}
+	}
 	do {
 		cxlr = __create_region(cxlrd, mode,
 				       atomic_read(&cxlrd->region_id));
@@ -3467,6 +3508,7 @@ static int cxl_region_probe(struct device *dev)
 	case CXL_REGION_PMEM:
 		return devm_cxl_add_pmem_region(cxlr);
 	case CXL_REGION_RAM:
+	case CXL_REGION_DC:
 		/*
 		 * The region can not be manged by CXL if any portion of
 		 * it is already online as 'System RAM'
drivers/dax/bus.c +10 -0
index fde29e0ad68b..d8cb5195a227 100644
@@ -178,6 +178,11 @@ static bool is_static(struct dax_region *dax_region)
 	return (dax_region->res.flags & IORESOURCE_DAX_STATIC) != 0;
 }
 
+static bool is_sparse(struct dax_region *dax_region)
+{
+	return (dax_region->res.flags & IORESOURCE_DAX_SPARSE_CAP) != 0;
+}
+
 bool static_dev_dax(struct dev_dax *dev_dax)
 {
 	return is_static(dev_dax->region);
@@ -301,6 +306,9 @@ static unsigned long long dax_region_avail_size(struct dax_region *dax_region)
 
 	lockdep_assert_held(&dax_region_rwsem);
 
+	if (is_sparse(dax_region))
+		return 0;
+
 	for_each_dax_region_resource(dax_region, res)
 		size -= resource_size(res);
 	return size;
@@ -1373,6 +1381,8 @@ static umode_t dev_dax_visible(struct kobject *kobj, struct attribute *a, int n)
 		return 0;
 	if (a == &dev_attr_mapping.attr && is_static(dax_region))
 		return 0;
+	if (a == &dev_attr_mapping.attr && is_sparse(dax_region))
+		return 0;
 	if ((a == &dev_attr_align.attr ||
 	     a == &dev_attr_size.attr) && is_static(dax_region))
 		return 0444;
drivers/dax/bus.h +1 -0
index cbbf64443098..783bfeef42cc 100644
@@ -13,6 +13,7 @@ struct dax_region;
 /* dax bus specific ioresource flags */
 #define IORESOURCE_DAX_STATIC BIT(0)
 #define IORESOURCE_DAX_KMEM BIT(1)
+#define IORESOURCE_DAX_SPARSE_CAP BIT(2)
 
 struct dax_region *alloc_dax_region(struct device *parent, int region_id,
 		struct range *range, int target_node, unsigned int align,
drivers/dax/cxl.c +14 -2
index 9b29e732b39a..367e86b1c22a 100644
@@ -13,19 +13,31 @@ static int cxl_dax_region_probe(struct device *dev)
 	struct cxl_region *cxlr = cxlr_dax->cxlr;
 	struct dax_region *dax_region;
 	struct dev_dax_data data;
+	resource_size_t dev_size;
+	unsigned long flags;
 
 	if (nid == NUMA_NO_NODE)
 		nid = memory_add_physaddr_to_nid(cxlr_dax->hpa_range.start);
 
+	flags = IORESOURCE_DAX_KMEM;
+	if (cxlr->mode == CXL_REGION_DC)
+		flags |= IORESOURCE_DAX_SPARSE_CAP;
+
 	dax_region = alloc_dax_region(dev, cxlr->id, &cxlr_dax->hpa_range, nid,
-				      PMD_SIZE, IORESOURCE_DAX_KMEM);
+				      PMD_SIZE, flags);
 	if (!dax_region)
 		return -ENOMEM;
 
+	if (cxlr->mode == CXL_REGION_DC)
+		/* Add empty seed dax device */
+		dev_size = 0;
+	else
+		dev_size = range_len(&cxlr_dax->hpa_range);
+
 	data = (struct dev_dax_data) {
 		.dax_region = dax_region,
 		.id = -1,
-		.size = range_len(&cxlr_dax->hpa_range),
+		.size = dev_size,
 		.memmap_on_memory = true,
 	};
 
Signature · 2 lines
-- 
2.46.0
Signature · 1 line
Fan Ni
author
#72 lore raw

PATCH v4 17/28 cxl/events: Split event msgnum configuration from irq setup

In reply to Ira Weiny

Dynamic Capacity Devices (DCD) require event interrupts to process memory addition or removal. BIOS may have control over non-DCD event processing. DCD interrupt configuration needs to be separate from memory event interrupt configuration.

Split cxl_event_config_msgnums() from irq setup in preparation for separate DCD interrupts configuration.

---
 drivers/cxl/pci.c | 24 ++++++++++++------------
 1 file changed, 12 insertions(+), 12 deletions(-)
drivers/cxl/pci.c +12 -12
index fc5ab74448cc..29a863331bec 100644
@@ -702,35 +702,31 @@ static int cxl_event_config_msgnums(struct cxl_memdev_state *mds,
 	return cxl_event_get_int_policy(mds, policy);
 }
 
-static int cxl_event_irqsetup(struct cxl_memdev_state *mds)
+static int cxl_event_irqsetup(struct cxl_memdev_state *mds,
+			      struct cxl_event_interrupt_policy *policy)
 {
 	struct cxl_dev_state *cxlds = &mds->cxlds;
-	struct cxl_event_interrupt_policy policy;
 	int rc;
 
-	rc = cxl_event_config_msgnums(mds, &policy);
-	if (rc)
-		return rc;
-
-	rc = cxl_event_req_irq(cxlds, policy.info_settings);
+	rc = cxl_event_req_irq(cxlds, policy->info_settings);
 	if (rc) {
 		dev_err(cxlds->dev, "Failed to get interrupt for event Info log\n");
 		return rc;
 	}
 
-	rc = cxl_event_req_irq(cxlds, policy.warn_settings);
+	rc = cxl_event_req_irq(cxlds, policy->warn_settings);
 	if (rc) {
 		dev_err(cxlds->dev, "Failed to get interrupt for event Warn log\n");
 		return rc;
 	}
 
-	rc = cxl_event_req_irq(cxlds, policy.failure_settings);
+	rc = cxl_event_req_irq(cxlds, policy->failure_settings);
 	if (rc) {
 		dev_err(cxlds->dev, "Failed to get interrupt for event Failure log\n");
 		return rc;
 	}
 
-	rc = cxl_event_req_irq(cxlds, policy.fatal_settings);
+	rc = cxl_event_req_irq(cxlds, policy->fatal_settings);
 	if (rc) {
 		dev_err(cxlds->dev, "Failed to get interrupt for event Fatal log\n");
 		return rc;
@@ -749,7 +745,7 @@ static bool cxl_event_int_is_fw(u8 setting)
 static int cxl_event_config(struct pci_host_bridge *host_bridge,
 			    struct cxl_memdev_state *mds, bool irq_avail)
 {
-	struct cxl_event_interrupt_policy policy;
+	struct cxl_event_interrupt_policy policy = { 0 };
 	int rc;
 
 	/*
@@ -777,11 +773,15 @@ static int cxl_event_config(struct pci_host_bridge *host_bridge,
 		return -EBUSY;
 	}
 
+	rc = cxl_event_config_msgnums(mds, &policy);
+	if (rc)
+		return rc;
+
 	rc = cxl_mem_alloc_event_buf(mds);
 	if (rc)
 		return rc;
 
-	rc = cxl_event_irqsetup(mds);
+	rc = cxl_event_irqsetup(mds, &policy);
 	if (rc)
 		return rc;
 
Signature · 1 line
2.46.0
Reviewed-by on v4 17/28
#73 lore raw

Re: [PATCH v4 17/28] cxl/events: Split event msgnum configuration from irq setup

Quoted 9 lines · On Mon, 07 Oct 2024 18:16:23 -0500 Ira Weiny <[email protected]> wrote:

Dynamic Capacity Devices (DCD) require event interrupts to process memory addition or removal. BIOS may have control over non-DCD event processing. DCD interrupt configuration needs to be separate from memory event interrupt configuration.

Split cxl_event_config_msgnums() from irq setup in preparation for separate DCD interrupts configuration.

Trivial comment inline

Quoted 19 lines
---
 drivers/cxl/pci.c | 24 ++++++++++++------------
 1 file changed, 12 insertions(+), 12 deletions(-)
drivers/cxl/pci.c +2 -2
index fc5ab74448cc..29a863331bec 100644
@@ -702,35 +702,31 @@ static int cxl_event_config_msgnums(struct cxl_memdev_state *mds,
 	return cxl_event_get_int_policy(mds, policy);
 }
 
-static int cxl_event_irqsetup(struct cxl_memdev_state *mds)
+static int cxl_event_irqsetup(struct cxl_memdev_state *mds,
+			      struct cxl_event_interrupt_policy *policy)
 {
 	struct cxl_dev_state *cxlds = &mds->cxlds;
-	struct cxl_event_interrupt_policy policy;
 	int rc;

drivers/cxl/pci.c static int cxl_event_config_msgnums(struct cxl_memdev_state *mds, · line 714

Quoted 10 lines
drivers/cxl/pci.c +1 -5
 
-	rc = cxl_event_config_msgnums(mds, &policy);
-	if (rc)
-		return rc;
-
-	rc = cxl_event_req_irq(cxlds, policy.info_settings);
+	rc = cxl_event_req_irq(cxlds, policy->info_settings);
 	if (rc) {
 		dev_err(cxlds->dev, "Failed to get interrupt for event Info log\n");
 		return rc;

At somepoint maybe dev_err_probe() is appropriate in here.

Quoted 48 lines
drivers/cxl/pci.c +9 -5
 	}
 
-	rc = cxl_event_req_irq(cxlds, policy.warn_settings);
+	rc = cxl_event_req_irq(cxlds, policy->warn_settings);
 	if (rc) {
 		dev_err(cxlds->dev, "Failed to get interrupt for event Warn log\n");
 		return rc;
 	}
 
-	rc = cxl_event_req_irq(cxlds, policy.failure_settings);
+	rc = cxl_event_req_irq(cxlds, policy->failure_settings);
 	if (rc) {
 		dev_err(cxlds->dev, "Failed to get interrupt for event Failure log\n");
 		return rc;
 	}
 
-	rc = cxl_event_req_irq(cxlds, policy.fatal_settings);
+	rc = cxl_event_req_irq(cxlds, policy->fatal_settings);
 	if (rc) {
 		dev_err(cxlds->dev, "Failed to get interrupt for event Fatal log\n");
 		return rc;
@@ -749,7 +745,7 @@ static bool cxl_event_int_is_fw(u8 setting)
 static int cxl_event_config(struct pci_host_bridge *host_bridge,
 			    struct cxl_memdev_state *mds, bool irq_avail)
 {
-	struct cxl_event_interrupt_policy policy;
+	struct cxl_event_interrupt_policy policy = { 0 };
 	int rc;
 
 	/*
@@ -777,11 +773,15 @@ static int cxl_event_config(struct pci_host_bridge *host_bridge,
 		return -EBUSY;
 	}
 
+	rc = cxl_event_config_msgnums(mds, &policy);
+	if (rc)
+		return rc;
+
 	rc = cxl_mem_alloc_event_buf(mds);
 	if (rc)
 		return rc;
 
-	rc = cxl_event_irqsetup(mds);
+	rc = cxl_event_irqsetup(mds, &policy);
 	if (rc)
 		return rc;
 
Reviewed-by on v4 17/28
#74 lore raw

Re: [PATCH v4 17/28] cxl/events: Split event msgnum configuration from irq setup

In reply to Ira Weiny
Quoted 10 lines · Ira Weiny

Dynamic Capacity Devices (DCD) require event interrupts to process memory addition or removal. BIOS may have control over non-DCD event processing. DCD interrupt configuration needs to be separate from memory event interrupt configuration.

Split cxl_event_config_msgnums() from irq setup in preparation for separate DCD interrupts configuration.

---

One minor comment inline; otherwise

Quoted 39 lines
 drivers/cxl/pci.c | 24 ++++++++++++------------
 1 file changed, 12 insertions(+), 12 deletions(-)
drivers/cxl/pci.c +5 -9
index fc5ab74448cc..29a863331bec 100644
@@ -702,35 +702,31 @@ static int cxl_event_config_msgnums(struct cxl_memdev_state *mds,
 	return cxl_event_get_int_policy(mds, policy);
 }
 
-static int cxl_event_irqsetup(struct cxl_memdev_state *mds)
+static int cxl_event_irqsetup(struct cxl_memdev_state *mds,
+			      struct cxl_event_interrupt_policy *policy)
 {
 	struct cxl_dev_state *cxlds = &mds->cxlds;
-	struct cxl_event_interrupt_policy policy;
 	int rc;
 
-	rc = cxl_event_config_msgnums(mds, &policy);
-	if (rc)
-		return rc;
-
-	rc = cxl_event_req_irq(cxlds, policy.info_settings);
+	rc = cxl_event_req_irq(cxlds, policy->info_settings);
 	if (rc) {
 		dev_err(cxlds->dev, "Failed to get interrupt for event Info log\n");
 		return rc;
 	}
 
-	rc = cxl_event_req_irq(cxlds, policy.warn_settings);
+	rc = cxl_event_req_irq(cxlds, policy->warn_settings);
 	if (rc) {
 		dev_err(cxlds->dev, "Failed to get interrupt for event Warn log\n");
 		return rc;
 	}
 
-	rc = cxl_event_req_irq(cxlds, policy.failure_settings);
+	rc = cxl_event_req_irq(cxlds, policy->failure_settings);

drivers/cxl/pci.c static int cxl_event_config_msgnums(struct cxl_memdev_state *mds, · line 732

Quoted 10 lines
drivers/cxl/pci.c +1 -1
 	if (rc) {
 		dev_err(cxlds->dev, "Failed to get interrupt for event Failure log\n");
 		return rc;
 	}
 
-	rc = cxl_event_req_irq(cxlds, policy.fatal_settings);
+	rc = cxl_event_req_irq(cxlds, policy->fatal_settings);
 	if (rc) {
 		dev_err(cxlds->dev, "Failed to get interrupt for event Fatal log\n");
 		return rc;

There is a lot of duplicate code here, can we simplify it by iteratting all setttings in cxl_event_interrrupt_policy like

for setting in policy:

    rc = cxl_event_req_irq(cxlds, setting);
    if (rc) {
        ...
    }

For DCD, handle the setup separately afterwards.

Fan

Quoted 30 lines
drivers/cxl/pci.c +6 -2
@@ -749,7 +745,7 @@ static bool cxl_event_int_is_fw(u8 setting)
 static int cxl_event_config(struct pci_host_bridge *host_bridge,
 			    struct cxl_memdev_state *mds, bool irq_avail)
 {
-	struct cxl_event_interrupt_policy policy;
+	struct cxl_event_interrupt_policy policy = { 0 };
 	int rc;
 
 	/*
@@ -777,11 +773,15 @@ static int cxl_event_config(struct pci_host_bridge *host_bridge,
 		return -EBUSY;
 	}
 
+	rc = cxl_event_config_msgnums(mds, &policy);
+	if (rc)
+		return rc;
+
 	rc = cxl_mem_alloc_event_buf(mds);
 	if (rc)
 		return rc;
 
-	rc = cxl_event_irqsetup(mds);
+	rc = cxl_event_irqsetup(mds, &policy);
 	if (rc)
 		return rc;
 
Signature · 2 lines
-- 
2.46.0
Signature · 1 line
Fan Ni
author
#75 lore raw

Re: [PATCH v4 17/28] cxl/events: Split event msgnum configuration from irq setup

Quoted 15 lines · Fan Ni
Quoted 10 lines · Ira Weiny

Dynamic Capacity Devices (DCD) require event interrupts to process memory addition or removal. BIOS may have control over non-DCD event processing. DCD interrupt configuration needs to be separate from memory event interrupt configuration.

Split cxl_event_config_msgnums() from irq setup in preparation for separate DCD interrupts configuration.

---

One minor comment inline; otherwise

[snip]

Quoted 10 lines

drivers/cxl/pci.c static int cxl_event_config_msgnums(struct cxl_memdev_state *mds, · line 732

Quoted 6 lines
drivers/cxl/pci.c +1 -1
-	rc = cxl_event_req_irq(cxlds, policy.fatal_settings);
+	rc = cxl_event_req_irq(cxlds, policy->fatal_settings);
 	if (rc) {
 		dev_err(cxlds->dev, "Failed to get interrupt for event Fatal log\n");
 		return rc;

There is a lot of duplicate code here, can we simplify it by iteratting all setttings in cxl_event_interrrupt_policy like

Quoted 6 lines

for setting in policy:

    rc = cxl_event_req_irq(cxlds, setting);
    if (rc) {
        ...
    }

Do you mean by treating struct cxl_event_interrupt_policy as an u8 array?

I'm not sure that is super beneficial.

Ira

Quoted 4 lines

For DCD, handle the setup separately afterwards.

Fan

[snip]

author
#76 lore raw

PATCH v4 18/28 cxl/pci: Factor out interrupt policy check

In reply to Ira Weiny

Dynamic Capacity Devices (DCD) require event interrupts to process memory addition or removal. BIOS may have control over non-DCD event processing. DCD interrupt configuration needs to be separate from memory event interrupt configuration.

Factor out event interrupt setting validation.

---
Changes:
[iweiny: reword commit message]
[iweiny: keep review tags on simple patch]

---
 drivers/cxl/pci.c | 23 ++++++++++++++++-------
 1 file changed, 16 insertions(+), 7 deletions(-)
drivers/cxl/pci.c +16 -7
index 29a863331bec..c6042db0653d 100644
@@ -742,6 +742,21 @@ static bool cxl_event_int_is_fw(u8 setting)
 	return mode == CXL_INT_FW;
 }
 
+static bool cxl_event_validate_mem_policy(struct cxl_memdev_state *mds,
+					  struct cxl_event_interrupt_policy *policy)
+{
+	if (cxl_event_int_is_fw(policy->info_settings) ||
+	    cxl_event_int_is_fw(policy->warn_settings) ||
+	    cxl_event_int_is_fw(policy->failure_settings) ||
+	    cxl_event_int_is_fw(policy->fatal_settings)) {
+		dev_err(mds->cxlds.dev,
+			"FW still in control of Event Logs despite _OSC settings\n");
+		return false;
+	}
+
+	return true;
+}
+
 static int cxl_event_config(struct pci_host_bridge *host_bridge,
 			    struct cxl_memdev_state *mds, bool irq_avail)
 {
@@ -764,14 +779,8 @@ static int cxl_event_config(struct pci_host_bridge *host_bridge,
 	if (rc)
 		return rc;
 
-	if (cxl_event_int_is_fw(policy.info_settings) ||
-	    cxl_event_int_is_fw(policy.warn_settings) ||
-	    cxl_event_int_is_fw(policy.failure_settings) ||
-	    cxl_event_int_is_fw(policy.fatal_settings)) {
-		dev_err(mds->cxlds.dev,
-			"FW still in control of Event Logs despite _OSC settings\n");
+	if (!cxl_event_validate_mem_policy(mds, &policy))
 		return -EBUSY;
-	}
 
 	rc = cxl_event_config_msgnums(mds, &policy);
 	if (rc)
Signature · 1 line
2.46.0
Reviewed-by on v4 18/28
#77 lore raw

Re: [PATCH v4 18/28] cxl/pci: Factor out interrupt policy check

Quoted 11 lines · Ira Weiny

Dynamic Capacity Devices (DCD) require event interrupts to process memory addition or removal. BIOS may have control over non-DCD event processing. DCD interrupt configuration needs to be separate from memory event interrupt configuration.

Factor out event interrupt setting validation.

Quoted 54 lines

---
Changes:
[iweiny: reword commit message]
[iweiny: keep review tags on simple patch]

---
 drivers/cxl/pci.c | 23 ++++++++++++++++-------
 1 file changed, 16 insertions(+), 7 deletions(-)
drivers/cxl/pci.c +16 -7
index 29a863331bec..c6042db0653d 100644
@@ -742,6 +742,21 @@ static bool cxl_event_int_is_fw(u8 setting)
 	return mode == CXL_INT_FW;
 }
 
+static bool cxl_event_validate_mem_policy(struct cxl_memdev_state *mds,
+					  struct cxl_event_interrupt_policy *policy)
+{
+	if (cxl_event_int_is_fw(policy->info_settings) ||
+	    cxl_event_int_is_fw(policy->warn_settings) ||
+	    cxl_event_int_is_fw(policy->failure_settings) ||
+	    cxl_event_int_is_fw(policy->fatal_settings)) {
+		dev_err(mds->cxlds.dev,
+			"FW still in control of Event Logs despite _OSC settings\n");
+		return false;
+	}
+
+	return true;
+}
+
 static int cxl_event_config(struct pci_host_bridge *host_bridge,
 			    struct cxl_memdev_state *mds, bool irq_avail)
 {
@@ -764,14 +779,8 @@ static int cxl_event_config(struct pci_host_bridge *host_bridge,
 	if (rc)
 		return rc;
 
-	if (cxl_event_int_is_fw(policy.info_settings) ||
-	    cxl_event_int_is_fw(policy.warn_settings) ||
-	    cxl_event_int_is_fw(policy.failure_settings) ||
-	    cxl_event_int_is_fw(policy.fatal_settings)) {
-		dev_err(mds->cxlds.dev,
-			"FW still in control of Event Logs despite _OSC settings\n");
+	if (!cxl_event_validate_mem_policy(mds, &policy))
 		return -EBUSY;
-	}
 
 	rc = cxl_event_config_msgnums(mds, &policy);
 	if (rc)
Signature · 2 lines
-- 
2.46.0
Signature · 1 line
Fan Ni
#78 lore raw

PATCH v4 19/28 cxl/mem: Configure dynamic capacity interrupts

In reply to Ira Weiny

From: Navneet Singh <[email protected]>

Dynamic Capacity Devices (DCD) support extent change notifications through the event log mechanism. The interrupt mailbox commands were extended in CXL 3.1 to support these notifications. Firmware can't configure DCD events to be FW controlled but can retain control of memory events.

Configure DCD event log interrupts on devices supporting dynamic capacity. Disable DCD if interrupts are not supported.

Care is taken to preserve the interrupt policy set by the FW if FW first has been selected by the BIOS.

---
Changes:
[iweiny: rebase on 6.12]

---
 drivers/cxl/cxlmem.h |  2 ++
 drivers/cxl/pci.c    | 72 +++++++++++++++++++++++++++++++++++++++++++---------
 2 files changed, 62 insertions(+), 12 deletions(-)
drivers/cxl/cxlmem.h +2 -0
index c3b889a586d8..2d2a1884a174 100644
@@ -226,7 +226,9 @@ struct cxl_event_interrupt_policy {
 	u8 warn_settings;
 	u8 failure_settings;
 	u8 fatal_settings;
+	u8 dcd_settings;
 } __packed;
+#define CXL_EVENT_INT_POLICY_BASE_SIZE 4 /* info, warn, failure, fatal */
 
 /**
  * struct cxl_event_state - Event log driver state
drivers/cxl/pci.c +60 -12

128 lines not on this page · raw message

Signature · 1 line
2.46.0
Reviewed-by on v4 19/28
#79 lore raw

Re: [PATCH v4 19/28] cxl/mem: Configure dynamic capacity interrupts

Quoted 11 lines · On Mon, 07 Oct 2024 18:16:25 -0500 [email protected] wrote:

From: Navneet Singh <[email protected]>

Dynamic Capacity Devices (DCD) support extent change notifications through the event log mechanism. The interrupt mailbox commands were extended in CXL 3.1 to support these notifications. Firmware can't configure DCD events to be FW controlled but can retain control of memory events.

Configure DCD event log interrupts on devices supporting dynamic capacity. Disable DCD if interrupts are not supported.

Quoted 6 lines

Care is taken to preserve the interrupt policy set by the FW if FW first has been selected by the BIOS.

Ah. I was wondering why policy needed to be initialize to zero outside this call. Maybe moving it in here with a memset() would be cleaner.

Either way

#80 lore raw

Re: [PATCH v4 19/28] cxl/mem: Configure dynamic capacity interrupts

In reply to [email protected]
Quoted 100 lines · On Mon, Oct 07, 2024 at 06:16:25PM -0500, [email protected] wrote:

From: Navneet Singh <[email protected]>

Dynamic Capacity Devices (DCD) support extent change notifications through the event log mechanism. The interrupt mailbox commands were extended in CXL 3.1 to support these notifications. Firmware can't configure DCD events to be FW controlled but can retain control of memory events.

Configure DCD event log interrupts on devices supporting dynamic capacity. Disable DCD if interrupts are not supported.

Care is taken to preserve the interrupt policy set by the FW if FW first has been selected by the BIOS.

---
Changes:
[iweiny: rebase on 6.12]

---
 drivers/cxl/cxlmem.h |  2 ++
 drivers/cxl/pci.c    | 72 +++++++++++++++++++++++++++++++++++++++++++---------
 2 files changed, 62 insertions(+), 12 deletions(-)
drivers/cxl/cxlmem.h +2 -0
index c3b889a586d8..2d2a1884a174 100644
@@ -226,7 +226,9 @@ struct cxl_event_interrupt_policy {
 	u8 warn_settings;
 	u8 failure_settings;
 	u8 fatal_settings;
+	u8 dcd_settings;
 } __packed;
+#define CXL_EVENT_INT_POLICY_BASE_SIZE 4 /* info, warn, failure, fatal */
 
 /**
  * struct cxl_event_state - Event log driver state
drivers/cxl/pci.c +28 -8
index c6042db0653d..2ba059d313c2 100644
@@ -672,23 +672,34 @@ static int cxl_event_get_int_policy(struct cxl_memdev_state *mds,
 }
 
 static int cxl_event_config_msgnums(struct cxl_memdev_state *mds,
-				    struct cxl_event_interrupt_policy *policy)
+				    struct cxl_event_interrupt_policy *policy,
+				    bool native_cxl)
 {
 	struct cxl_mailbox *cxl_mbox = &mds->cxlds.cxl_mbox;
+	size_t size_in = CXL_EVENT_INT_POLICY_BASE_SIZE;
 	struct cxl_mbox_cmd mbox_cmd;
 	int rc;
 
-	*policy = (struct cxl_event_interrupt_policy) {
-		.info_settings = CXL_INT_MSI_MSIX,
-		.warn_settings = CXL_INT_MSI_MSIX,
-		.failure_settings = CXL_INT_MSI_MSIX,
-		.fatal_settings = CXL_INT_MSI_MSIX,
-	};
+	/* memory event policy is left if FW has control */
+	if (native_cxl) {
+		*policy = (struct cxl_event_interrupt_policy) {
+			.info_settings = CXL_INT_MSI_MSIX,
+			.warn_settings = CXL_INT_MSI_MSIX,
+			.failure_settings = CXL_INT_MSI_MSIX,
+			.fatal_settings = CXL_INT_MSI_MSIX,
+			.dcd_settings = 0,
+		};
+	}
+
+	if (cxl_dcd_supported(mds)) {
+		policy->dcd_settings = CXL_INT_MSI_MSIX;
+		size_in += sizeof(policy->dcd_settings);
+	}
 
 	mbox_cmd = (struct cxl_mbox_cmd) {
 		.opcode = CXL_MBOX_OP_SET_EVT_INT_POLICY,
 		.payload_in = policy,
-		.size_in = sizeof(*policy),
+		.size_in = size_in,
 	};
 
 	rc = cxl_internal_send_cmd(cxl_mbox, &mbox_cmd);
@@ -735,6 +746,31 @@ static int cxl_event_irqsetup(struct cxl_memdev_state *mds,
 	return 0;
 }
 
+static int cxl_irqsetup(struct cxl_memdev_state *mds,
+			struct cxl_event_interrupt_policy *policy,
+			bool native_cxl)
+{
+	struct cxl_dev_state *cxlds = &mds->cxlds;
+	int rc;
+
+	if (native_cxl) {
+		rc = cxl_event_irqsetup(mds, policy);

drivers/cxl/pci.c static int cxl_event_irqsetup(struct cxl_memdev_state *mds, · line 767

Quoted 10 lines
drivers/cxl/pci.c +10 -0
+		if (rc)
+			return rc;
+	}
+
+	if (cxl_dcd_supported(mds)) {
+		rc = cxl_event_req_irq(cxlds, policy->dcd_settings);
+		if (rc) {
+			dev_err(cxlds->dev, "Failed to get interrupt for DCD event log\n");
+			cxl_disable_dcd(mds);
+			return rc;

If the device has both static and dynamic capacity, return an error code here will cause cxl_event_config() return early, and cxl_mem_get_event_records() will not be called, will it be an issue?

Fan

Quoted 71 lines
drivers/cxl/pci.c +22 -4
+		}
+	}
+
+	return 0;
+}
+
 static bool cxl_event_int_is_fw(u8 setting)
 {
 	u8 mode = FIELD_GET(CXLDEV_EVENT_INT_MODE_MASK, setting);
@@ -761,17 +797,25 @@ static int cxl_event_config(struct pci_host_bridge *host_bridge,
 			    struct cxl_memdev_state *mds, bool irq_avail)
 {
 	struct cxl_event_interrupt_policy policy = { 0 };
+	bool native_cxl = host_bridge->native_cxl_error;
 	int rc;
 
 	/*
 	 * When BIOS maintains CXL error reporting control, it will process
 	 * event records.  Only one agent can do so.
+	 *
+	 * If BIOS has control of events and DCD is not supported skip event
+	 * configuration.
 	 */
-	if (!host_bridge->native_cxl_error)
+	if (!native_cxl && !cxl_dcd_supported(mds))
 		return 0;
 
 	if (!irq_avail) {
 		dev_info(mds->cxlds.dev, "No interrupt support, disable event processing.\n");
+		if (cxl_dcd_supported(mds)) {
+			dev_info(mds->cxlds.dev, "DCD requires interrupts, disable DCD\n");
+			cxl_disable_dcd(mds);
+		}
 		return 0;
 	}
 
@@ -779,10 +823,10 @@ static int cxl_event_config(struct pci_host_bridge *host_bridge,
 	if (rc)
 		return rc;
 
-	if (!cxl_event_validate_mem_policy(mds, &policy))
+	if (native_cxl && !cxl_event_validate_mem_policy(mds, &policy))
 		return -EBUSY;
 
-	rc = cxl_event_config_msgnums(mds, &policy);
+	rc = cxl_event_config_msgnums(mds, &policy, native_cxl);
 	if (rc)
 		return rc;
 
@@ -790,12 +834,16 @@ static int cxl_event_config(struct pci_host_bridge *host_bridge,
 	if (rc)
 		return rc;
 
-	rc = cxl_event_irqsetup(mds, &policy);
+	rc = cxl_irqsetup(mds, &policy, native_cxl);
 	if (rc)
 		return rc;
 
 	cxl_mem_get_event_records(mds, CXLDEV_EVENT_STATUS_ALL);
 
+	dev_dbg(mds->cxlds.dev, "Event config : %s DCD %s\n",
+		native_cxl ? "OS" : "BIOS",
+		cxl_dcd_supported(mds) ? "supported" : "not supported");
+
 	return 0;
 }
 
Signature · 2 lines
-- 
2.46.0
Signature · 1 line
Fan Ni
author
#81 lore raw

Re: [PATCH v4 19/28] cxl/mem: Configure dynamic capacity interrupts

Quoted 3 lines · Fan Ni
Quoted 2 lines · On Mon, Oct 07, 2024 at 06:16:25PM -0500, [email protected] wrote:

From: Navneet Singh <[email protected]>

[snip]

Quoted 16 lines
Quoted 16 lines
drivers/cxl/pci.c +15 -0
+static int cxl_irqsetup(struct cxl_memdev_state *mds,
+			struct cxl_event_interrupt_policy *policy,
+			bool native_cxl)
+{
+	struct cxl_dev_state *cxlds = &mds->cxlds;
+	int rc;
+
+	if (native_cxl) {
+		rc = cxl_event_irqsetup(mds, policy);
+		if (rc)
+			return rc;
+	}
+
+	if (cxl_dcd_supported(mds)) {
+		rc = cxl_event_req_irq(cxlds, policy->dcd_settings);
Quoted 8 lines

drivers/cxl/pci.c static int cxl_event_irqsetup(struct cxl_memdev_state *mds, · line 767

Quoted 4 lines
drivers/cxl/pci.c +4 -0
+		if (rc) {
+			dev_err(cxlds->dev, "Failed to get interrupt for DCD event log\n");
+			cxl_disable_dcd(mds);
+			return rc;

If the device has both static and dynamic capacity, return an error code here will cause cxl_event_config() return early, and cxl_mem_get_event_records() will not be called, will it be an issue?

Good catch as it was not my intent to fail the device probe. Rather to disable DCD. In practice however, the device would be broken because it should not be advertising DCD support without allowing for the proper irq setup.

Current support will fail the probe if event configuration fails. So the error here is probably bothering to try to disable future DCD processing.

At this point I'm not sure which way to go...

Ira

author
#82 lore raw

PATCH v4 20/28 cxl/core: Return endpoint decoder information from region search

In reply to Ira Weiny

cxl_dpa_to_region() finds the region from a <DPA, device> tuple. The search involves finding the device endpoint decoder as well.

Dynamic capacity extent processing uses the endpoint decoder HPA information to calculate the HPA offset. In addition, well behaved extents should be contained within an endpoint decoder.

Return the endpoint decoder found to be used in subsequent DCD code.

---
 drivers/cxl/core/core.h   | 6 ++++--
 drivers/cxl/core/mbox.c   | 2 +-
 drivers/cxl/core/memdev.c | 4 ++--
 drivers/cxl/core/region.c | 8 +++++++-
 4 files changed, 14 insertions(+), 6 deletions(-)
drivers/cxl/core/core.h +4 -2
index 5d6fe7ab0a78..94ee06cfbdca 100644
@@ -39,7 +39,8 @@ void cxl_decoder_kill_region(struct cxl_endpoint_decoder *cxled);
 int cxl_region_init(void);
 void cxl_region_exit(void);
 int cxl_get_poison_by_endpoint(struct cxl_port *port);
-struct cxl_region *cxl_dpa_to_region(const struct cxl_memdev *cxlmd, u64 dpa);
+struct cxl_region *cxl_dpa_to_region(const struct cxl_memdev *cxlmd, u64 dpa,
+				     struct cxl_endpoint_decoder **cxled);
 u64 cxl_dpa_to_hpa(struct cxl_region *cxlr, const struct cxl_memdev *cxlmd,
 		   u64 dpa);
 
@@ -50,7 +51,8 @@ static inline u64 cxl_dpa_to_hpa(struct cxl_region *cxlr,
 	return ULLONG_MAX;
 }
 static inline
-struct cxl_region *cxl_dpa_to_region(const struct cxl_memdev *cxlmd, u64 dpa)
+struct cxl_region *cxl_dpa_to_region(const struct cxl_memdev *cxlmd, u64 dpa,
+				     struct cxl_endpoint_decoder **cxled)
 {
 	return NULL;
 }
drivers/cxl/core/mbox.c +1 -1
index 3ba465823564..584d7d282a97 100644
@@ -916,7 +916,7 @@ void cxl_event_trace_record(const struct cxl_memdev *cxlmd,
 		guard(rwsem_read)(&cxl_dpa_rwsem);
 
 		dpa = le64_to_cpu(evt->media_hdr.phys_addr) & CXL_DPA_MASK;
-		cxlr = cxl_dpa_to_region(cxlmd, dpa);
+		cxlr = cxl_dpa_to_region(cxlmd, dpa, NULL);
 		if (cxlr)
 			hpa = cxl_dpa_to_hpa(cxlr, cxlmd, dpa);
 
drivers/cxl/core/memdev.c +2 -2
index 2565b10a769c..31872c03006b 100644
@@ -313,7 +313,7 @@ int cxl_inject_poison(struct cxl_memdev *cxlmd, u64 dpa)
 	if (rc)
 		goto out;
 
-	cxlr = cxl_dpa_to_region(cxlmd, dpa);
+	cxlr = cxl_dpa_to_region(cxlmd, dpa, NULL);
 	if (cxlr)
 		dev_warn_once(cxl_mbox->host,
 			      "poison inject dpa:%#llx region: %s\n", dpa,
@@ -377,7 +377,7 @@ int cxl_clear_poison(struct cxl_memdev *cxlmd, u64 dpa)
 	if (rc)
 		goto out;
 
-	cxlr = cxl_dpa_to_region(cxlmd, dpa);
+	cxlr = cxl_dpa_to_region(cxlmd, dpa, NULL);
 	if (cxlr)
 		dev_warn_once(cxl_mbox->host,
 			      "poison clear dpa:%#llx region: %s\n", dpa,
drivers/cxl/core/region.c +7 -1
index 34a6f447e75b..a0c181cc33e4 100644
@@ -2827,6 +2827,7 @@ int cxl_get_poison_by_endpoint(struct cxl_port *port)
 struct cxl_dpa_to_region_context {
 	struct cxl_region *cxlr;
 	u64 dpa;
+	struct cxl_endpoint_decoder *cxled;
 };
 
 static int __cxl_dpa_to_region(struct device *dev, void *arg)
@@ -2860,11 +2861,13 @@ static int __cxl_dpa_to_region(struct device *dev, void *arg)
 			dev_name(dev));
 
 	ctx->cxlr = cxlr;
+	ctx->cxled = cxled;
 
 	return 1;
 }
 
-struct cxl_region *cxl_dpa_to_region(const struct cxl_memdev *cxlmd, u64 dpa)
+struct cxl_region *cxl_dpa_to_region(const struct cxl_memdev *cxlmd, u64 dpa,
+				     struct cxl_endpoint_decoder **cxled)
 {
 	struct cxl_dpa_to_region_context ctx;
 	struct cxl_port *port;
@@ -2876,6 +2879,9 @@ struct cxl_region *cxl_dpa_to_region(const struct cxl_memdev *cxlmd, u64 dpa)
 	if (port && is_cxl_endpoint(port) && cxl_num_decoders_committed(port))
 		device_for_each_child(&port->dev, &ctx, __cxl_dpa_to_region);
 
+	if (cxled)
+		*cxled = ctx.cxled;
+
 	return ctx.cxlr;
 }
 
Signature · 1 line
2.46.0
Reviewed-by on v4 20/28
#83 lore raw

Re: [PATCH v4 20/28] cxl/core: Return endpoint decoder information from region search

Quoted 10 lines · On Mon, 07 Oct 2024 18:16:26 -0500 Ira Weiny <[email protected]> wrote:

cxl_dpa_to_region() finds the region from a <DPA, device> tuple. The search involves finding the device endpoint decoder as well.

Dynamic capacity extent processing uses the endpoint decoder HPA information to calculate the HPA offset. In addition, well behaved extents should be contained within an endpoint decoder.

Return the endpoint decoder found to be used in subsequent DCD code.

Reviewed-by on v4 20/28
#84 lore raw

Re: [PATCH v4 20/28] cxl/core: Return endpoint decoder information from region search

In reply to Ira Weiny
Quoted 10 lines · Ira Weiny

cxl_dpa_to_region() finds the region from a <DPA, device> tuple. The search involves finding the device endpoint decoder as well.

Dynamic capacity extent processing uses the endpoint decoder HPA information to calculate the HPA offset. In addition, well behaved extents should be contained within an endpoint decoder.

Return the endpoint decoder found to be used in subsequent DCD code.

Quoted 107 lines
---
 drivers/cxl/core/core.h   | 6 ++++--
 drivers/cxl/core/mbox.c   | 2 +-
 drivers/cxl/core/memdev.c | 4 ++--
 drivers/cxl/core/region.c | 8 +++++++-
 4 files changed, 14 insertions(+), 6 deletions(-)
drivers/cxl/core/core.h +4 -2
index 5d6fe7ab0a78..94ee06cfbdca 100644
@@ -39,7 +39,8 @@ void cxl_decoder_kill_region(struct cxl_endpoint_decoder *cxled);
 int cxl_region_init(void);
 void cxl_region_exit(void);
 int cxl_get_poison_by_endpoint(struct cxl_port *port);
-struct cxl_region *cxl_dpa_to_region(const struct cxl_memdev *cxlmd, u64 dpa);
+struct cxl_region *cxl_dpa_to_region(const struct cxl_memdev *cxlmd, u64 dpa,
+				     struct cxl_endpoint_decoder **cxled);
 u64 cxl_dpa_to_hpa(struct cxl_region *cxlr, const struct cxl_memdev *cxlmd,
 		   u64 dpa);
 
@@ -50,7 +51,8 @@ static inline u64 cxl_dpa_to_hpa(struct cxl_region *cxlr,
 	return ULLONG_MAX;
 }
 static inline
-struct cxl_region *cxl_dpa_to_region(const struct cxl_memdev *cxlmd, u64 dpa)
+struct cxl_region *cxl_dpa_to_region(const struct cxl_memdev *cxlmd, u64 dpa,
+				     struct cxl_endpoint_decoder **cxled)
 {
 	return NULL;
 }
drivers/cxl/core/mbox.c +1 -1
index 3ba465823564..584d7d282a97 100644
@@ -916,7 +916,7 @@ void cxl_event_trace_record(const struct cxl_memdev *cxlmd,
 		guard(rwsem_read)(&cxl_dpa_rwsem);
 
 		dpa = le64_to_cpu(evt->media_hdr.phys_addr) & CXL_DPA_MASK;
-		cxlr = cxl_dpa_to_region(cxlmd, dpa);
+		cxlr = cxl_dpa_to_region(cxlmd, dpa, NULL);
 		if (cxlr)
 			hpa = cxl_dpa_to_hpa(cxlr, cxlmd, dpa);
 
drivers/cxl/core/memdev.c +2 -2
index 2565b10a769c..31872c03006b 100644
@@ -313,7 +313,7 @@ int cxl_inject_poison(struct cxl_memdev *cxlmd, u64 dpa)
 	if (rc)
 		goto out;
 
-	cxlr = cxl_dpa_to_region(cxlmd, dpa);
+	cxlr = cxl_dpa_to_region(cxlmd, dpa, NULL);
 	if (cxlr)
 		dev_warn_once(cxl_mbox->host,
 			      "poison inject dpa:%#llx region: %s\n", dpa,
@@ -377,7 +377,7 @@ int cxl_clear_poison(struct cxl_memdev *cxlmd, u64 dpa)
 	if (rc)
 		goto out;
 
-	cxlr = cxl_dpa_to_region(cxlmd, dpa);
+	cxlr = cxl_dpa_to_region(cxlmd, dpa, NULL);
 	if (cxlr)
 		dev_warn_once(cxl_mbox->host,
 			      "poison clear dpa:%#llx region: %s\n", dpa,
drivers/cxl/core/region.c +7 -1
index 34a6f447e75b..a0c181cc33e4 100644
@@ -2827,6 +2827,7 @@ int cxl_get_poison_by_endpoint(struct cxl_port *port)
 struct cxl_dpa_to_region_context {
 	struct cxl_region *cxlr;
 	u64 dpa;
+	struct cxl_endpoint_decoder *cxled;
 };
 
 static int __cxl_dpa_to_region(struct device *dev, void *arg)
@@ -2860,11 +2861,13 @@ static int __cxl_dpa_to_region(struct device *dev, void *arg)
 			dev_name(dev));
 
 	ctx->cxlr = cxlr;
+	ctx->cxled = cxled;
 
 	return 1;
 }
 
-struct cxl_region *cxl_dpa_to_region(const struct cxl_memdev *cxlmd, u64 dpa)
+struct cxl_region *cxl_dpa_to_region(const struct cxl_memdev *cxlmd, u64 dpa,
+				     struct cxl_endpoint_decoder **cxled)
 {
 	struct cxl_dpa_to_region_context ctx;
 	struct cxl_port *port;
@@ -2876,6 +2879,9 @@ struct cxl_region *cxl_dpa_to_region(const struct cxl_memdev *cxlmd, u64 dpa)
 	if (port && is_cxl_endpoint(port) && cxl_num_decoders_committed(port))
 		device_for_each_child(&port->dev, &ctx, __cxl_dpa_to_region);
 
+	if (cxled)
+		*cxled = ctx.cxled;
+
 	return ctx.cxlr;
 }
 
Signature · 2 lines
-- 
2.46.0
Signature · 1 line
Fan Ni
maintainer Reviewed-by on v4 20/28
#85 lore raw

Re: [PATCH v4 20/28] cxl/core: Return endpoint decoder information from region search

In reply to Ira Weiny
Quoted 8 lines · Ira Weiny

cxl_dpa_to_region() finds the region from a <DPA, device> tuple. The search involves finding the device endpoint decoder as well.

Dynamic capacity extent processing uses the endpoint decoder HPA information to calculate the HPA offset. In addition, well behaved extents should be contained within an endpoint decoder.

Return the endpoint decoder found to be used in subsequent DCD code.

BTW - I reviewed this patch when it first appeard in the DCD series and looked for other ways to layer the delivery of cxled and cxlr. Nothing clever appeared and looking at how DCD uses it in the future patch made this feel less yucky - it does want both cxled & cxlr so it can have them here all at once ;)

snip

#86 lore raw

PATCH v4 21/28 cxl/extent: Process DCD events and realize region extents

In reply to Ira Weiny

From: Navneet Singh <[email protected]>

A dynamic capacity device (DCD) sends events to signal the host for changes in the availability of Dynamic Capacity (DC) memory. These events contain extents describing a DPA range and meta data for memory to be added or removed. Events may be sent from the device at any time.

Three types of events can be signaled, Add, Release, and Force Release.

On add, the host may accept or reject the memory being offered. If no region exists, or the extent is invalid, the extent should be rejected. Add extent events may be grouped by a 'more' bit which indicates those extents should be processed as a group.

On remove, the host can delay the response until the host is safely not using the memory. If no region exists the release can be sent immediately. The host may also release extents (or partial extents) at any time. Thus the 'more' bit grouping of release events is of less value and can be ignored in favor of sending multiple release capacity responses for groups of release events.

Force removal is intended as a mechanism between the FM and the device and intended only when the host is unresponsive, out of sync, or otherwise broken. Purposely ignore force removal events.

Regions are made up of one or more devices which may be surfacing memory to the host. Once all devices in a region have surfaced an extent the region can expose a corresponding extent for the user to consume. Without interleaving a device extent forms a 1:1 relationship with the region extent. Immediately surface a region extent upon getting a device extent.

Per the specification the device is allowed to offer or remove extents at any time. However, anticipated use cases can expect extents to be offered, accepted, and removed in well defined chunks.

Simplify extent tracking with the following restrictions.

1) Flag for removal any extent which overlaps a requested release range.
2) Refuse the offer of extents which overlap already accepted memory ranges.
3) Accept again a range which has already been accepted by the host. Eating duplicates serves three purposes. First, this simplifies the code if the device should get out of sync with the host. And it should be safe to acknowledge the extent again. Second, this simplifies the code to process existing extents if the extent list should change while the extent list is being read. Third, duplicates for a given region which are seen during a race between the hardware surfacing an extent and the cxl dax driver scanning for existing extents will be ignored.

NOTE: Processing existing extents is done in a later patch.

Management of the region extent devices must be synchronized with potential uses of the memory within the DAX layer. Create region extent devices as children of the cxl_dax_region device such that the DAX region driver can co-drive them and synchronize with the DAX layer. Synchronization and management is handled in a subsequent patch.

Tag support within the DAX layer is not yet supported. To maintain compatibility legacy DAX/region processing only tags with a value of 0 are allowed. This defines existing DAX devices as having a 0 tag which makes the most logical sense as a default.

Process DCD events and create region devices.

---
Changes:
[Jonathan/jgroves/iweiny: restrict tags to 0] [iweiny: rebase to 6.12 and adjust mailbox commands] [Jonathan: remove setting cxlr <-> cxlr_dax links to NULL]

---
 drivers/cxl/core/Makefile |   2 +-
 drivers/cxl/core/core.h   |  13 ++
 drivers/cxl/core/extent.c | 366 ++++++++++++++++++++++++++++++++++++++++++++++
 drivers/cxl/core/mbox.c   | 293 ++++++++++++++++++++++++++++++++++++-
 drivers/cxl/core/region.c |   3 +
 drivers/cxl/cxl.h         |  52 ++++++-
 drivers/cxl/cxlmem.h      |  26 ++++
 include/cxl/event.h       |  32 ++++
 tools/testing/cxl/Kbuild  |   3 +-
 9 files changed, 786 insertions(+), 4 deletions(-)
drivers/cxl/core/Makefile +1 -1
index 9259bcc6773c..3b812515e725 100644
@@ -15,4 +15,4 @@ cxl_core-y += hdm.o
 cxl_core-y += pmu.o
 cxl_core-y += cdat.o
 cxl_core-$(CONFIG_TRACING) += trace.o
-cxl_core-$(CONFIG_CXL_REGION) += region.o
+cxl_core-$(CONFIG_CXL_REGION) += region.o extent.o
drivers/cxl/core/core.h +13 -0
index 94ee06cfbdca..0eccdd0b9261 100644
@@ -44,12 +44,24 @@ struct cxl_region *cxl_dpa_to_region(const struct cxl_memdev *cxlmd, u64 dpa,
 u64 cxl_dpa_to_hpa(struct cxl_region *cxlr, const struct cxl_memdev *cxlmd,
 		   u64 dpa);
 
+int cxl_add_extent(struct cxl_memdev_state *mds, struct cxl_extent *extent);
+int cxl_rm_extent(struct cxl_memdev_state *mds, struct cxl_extent *extent);
 #else
 static inline u64 cxl_dpa_to_hpa(struct cxl_region *cxlr,
 				 const struct cxl_memdev *cxlmd, u64 dpa)
 {
 	return ULLONG_MAX;
 }
+static inline int cxl_add_extent(struct cxl_memdev_state *mds,
+				   struct cxl_extent *extent)
+{
+	return 0;
+}
+static inline int cxl_rm_extent(struct cxl_memdev_state *mds,
+				struct cxl_extent *extent)
+{
+	return 0;
+}
 static inline
 struct cxl_region *cxl_dpa_to_region(const struct cxl_memdev *cxlmd, u64 dpa,
 				     struct cxl_endpoint_decoder **cxled)
@@ -123,5 +135,6 @@ int cxl_update_hmat_access_coordinates(int nid, struct cxl_region *cxlr,
 bool cxl_need_node_perf_attrs_update(int nid);
 int cxl_port_get_switch_dport_bandwidth(struct cxl_port *port,
 					struct access_coordinate *c);
+void memdev_release_extent(struct cxl_memdev_state *mds, struct range *range);
 
 #endif /* __CXL_CORE_H__ */
drivers/cxl/core/extent.c new file +366 -0

366 lines not on this page · raw message

drivers/cxl/core/mbox.c +292 -1

334 lines not on this page · raw message

drivers/cxl/core/region.c +3 -0
index a0c181cc33e4..6ae51fc2bdae 100644
@@ -3036,6 +3036,7 @@ static void cxl_dax_region_release(struct device *dev)
 {
 	struct cxl_dax_region *cxlr_dax = to_cxl_dax_region(dev);
 
+	ida_destroy(&cxlr_dax->extent_ida);
 	kfree(cxlr_dax);
 }
 
@@ -3089,6 +3090,8 @@ static struct cxl_dax_region *cxl_dax_region_alloc(struct cxl_region *cxlr)
 
 	dev = &cxlr_dax->dev;
 	cxlr_dax->cxlr = cxlr;
+	cxlr->cxlr_dax = cxlr_dax;
+	ida_init(&cxlr_dax->extent_ida);
 	device_initialize(dev);
 	lockdep_set_class(&dev->mutex, &cxl_dax_region_key);
 	device_set_pm_not_required(dev);
drivers/cxl/cxl.h +51 -1

98 lines not on this page · raw message

drivers/cxl/cxlmem.h +26 -0
index 2d2a1884a174..dd7cc0d373af 100644
@@ -506,6 +506,7 @@ static inline struct cxl_dev_state *mbox_to_cxlds(struct cxl_mailbox *cxl_mbox)
  * @pmem_perf: performance data entry matched to PMEM partition
  * @nr_dc_region: number of DC regions implemented in the memory device
  * @dc_region: array containing info about the DC regions
+ * @pending_extents: array of extents pending during more bit processing
  * @event: event log driver state
  * @poison: poison driver state info
  * @security: security driver state info
@@ -538,6 +539,7 @@ struct cxl_memdev_state {
 	u8 nr_dc_region;
 	struct cxl_dc_region_info dc_region[CXL_MAX_DC_REGION];
 	struct cxl_dpa_perf dc_perf[CXL_MAX_DC_REGION];
+	struct xarray pending_extents;
 
 	struct cxl_event_state event;
 	struct cxl_poison_state poison;
@@ -609,6 +611,21 @@ enum cxl_opcode {
 	UUID_INIT(0x5e1819d9, 0x11a9, 0x400c, 0x81, 0x1f, 0xd6, 0x07, 0x19,     \
 		  0x40, 0x3d, 0x86)
 
+/*
+ * Add Dynamic Capacity Response
+ * CXL rev 3.1 section 8.2.9.9.9.3; Table 8-168 & Table 8-169
+ */
+struct cxl_mbox_dc_response {
+	__le32 extent_list_size;
+	u8 flags;
+	u8 reserved[3];
+	struct updated_extent_list {
+		__le64 dpa_start;
+		__le64 length;
+		u8 reserved[8];
+	} __packed extent_list[];
+} __packed;
+
 struct cxl_mbox_get_supported_logs {
 	__le16 entries;
 	u8 rsvd[6];
@@ -671,6 +688,14 @@ struct cxl_mbox_identify {
 	UUID_INIT(0xfe927475, 0xdd59, 0x4339, 0xa5, 0x86, 0x79, 0xba, 0xb1, \
 		  0x13, 0xb7, 0x74)
 
+/*
+ * Dynamic Capacity Event Record
+ * CXL rev 3.1 section 8.2.9.2.1; Table 8-43
+ */
+#define CXL_EVENT_DC_EVENT_UUID                                             \
+	UUID_INIT(0xca95afa7, 0xf183, 0x4018, 0x8c, 0x2f, 0x95, 0x26, 0x8e, \
+		  0x10, 0x1a, 0x2a)
+
 /*
  * Get Event Records output payload
  * CXL rev 3.0 section 8.2.9.2.2; Table 8-50
@@ -696,6 +721,7 @@ enum cxl_event_log_type {
 	CXL_EVENT_TYPE_WARN,
 	CXL_EVENT_TYPE_FAIL,
 	CXL_EVENT_TYPE_FATAL,
+	CXL_EVENT_TYPE_DCD,
 	CXL_EVENT_TYPE_MAX
 };
 
include/cxl/event.h +32 -0
index 0bea1afbd747..eeda8059d81a 100644
@@ -96,11 +96,43 @@ struct cxl_event_mem_module {
 	u8 reserved[0x3d];
 } __packed;
 
+/*
+ * CXL rev 3.1 section 8.2.9.2.1.6; Table 8-51
+ */
+#define CXL_EXTENT_TAG_LEN 0x10
+struct cxl_extent {
+	__le64 start_dpa;
+	__le64 length;
+	u8 tag[CXL_EXTENT_TAG_LEN];
+	__le16 shared_extn_seq;
+	u8 reserved[0x6];
+} __packed;
+
+/*
+ * Dynamic Capacity Event Record
+ * CXL rev 3.1 section 8.2.9.2.1.6; Table 8-50
+ */
+#define CXL_DCD_EVENT_MORE			BIT(0)
+struct cxl_event_dcd {
+	struct cxl_event_record_hdr hdr;
+	u8 event_type;
+	u8 validity_flags;
+	__le16 host_id;
+	u8 region_index;
+	u8 flags;
+	u8 reserved1[0x2];
+	struct cxl_extent extent;
+	u8 reserved2[0x18];
+	__le32 num_avail_extents;
+	__le32 num_avail_tags;
+} __packed;
+
 union cxl_event {
 	struct cxl_event_generic generic;
 	struct cxl_event_gen_media gen_media;
 	struct cxl_event_dram dram;
 	struct cxl_event_mem_module mem_module;
+	struct cxl_event_dcd dcd;
 	/* dram & gen_media event header */
 	struct cxl_event_media_hdr media_hdr;
 } __packed;
tools/testing/cxl/Kbuild +2 -1
index b1256fee3567..bfa19587fd76 100644
@@ -62,7 +62,8 @@ cxl_core-y += $(CXL_CORE_SRC)/hdm.o
 cxl_core-y += $(CXL_CORE_SRC)/pmu.o
 cxl_core-y += $(CXL_CORE_SRC)/cdat.o
 cxl_core-$(CONFIG_TRACING) += $(CXL_CORE_SRC)/trace.o
-cxl_core-$(CONFIG_CXL_REGION) += $(CXL_CORE_SRC)/region.o
+cxl_core-$(CONFIG_CXL_REGION) += $(CXL_CORE_SRC)/region.o \
+				 $(CXL_CORE_SRC)/extent.o
 cxl_core-y += config_check.o
 cxl_core-y += cxl_core_test.o
 cxl_core-y += cxl_core_exports.o
Signature · 1 line
2.46.0
#87 lore raw

Re: [PATCH v4 21/28] cxl/extent: Process DCD events and realize region extents

Quoted 66 lines · On 10/8/2024 7:16 AM, [email protected] wrote:

From: Navneet Singh <[email protected]>

A dynamic capacity device (DCD) sends events to signal the host for changes in the availability of Dynamic Capacity (DC) memory. These events contain extents describing a DPA range and meta data for memory to be added or removed. Events may be sent from the device at any time.

Three types of events can be signaled, Add, Release, and Force Release.

On add, the host may accept or reject the memory being offered. If no region exists, or the extent is invalid, the extent should be rejected. Add extent events may be grouped by a 'more' bit which indicates those extents should be processed as a group.

On remove, the host can delay the response until the host is safely not using the memory. If no region exists the release can be sent immediately. The host may also release extents (or partial extents) at any time. Thus the 'more' bit grouping of release events is of less value and can be ignored in favor of sending multiple release capacity responses for groups of release events.

Force removal is intended as a mechanism between the FM and the device and intended only when the host is unresponsive, out of sync, or otherwise broken. Purposely ignore force removal events.

Regions are made up of one or more devices which may be surfacing memory to the host. Once all devices in a region have surfaced an extent the region can expose a corresponding extent for the user to consume. Without interleaving a device extent forms a 1:1 relationship with the region extent. Immediately surface a region extent upon getting a device extent.

Per the specification the device is allowed to offer or remove extents at any time. However, anticipated use cases can expect extents to be offered, accepted, and removed in well defined chunks.

Simplify extent tracking with the following restrictions.

1) Flag for removal any extent which overlaps a requested release range.
2) Refuse the offer of extents which overlap already accepted memory ranges.
3) Accept again a range which has already been accepted by the host. Eating duplicates serves three purposes. First, this simplifies the code if the device should get out of sync with the host. And it should be safe to acknowledge the extent again. Second, this simplifies the code to process existing extents if the extent list should change while the extent list is being read. Third, duplicates for a given region which are seen during a race between the hardware surfacing an extent and the cxl dax driver scanning for existing extents will be ignored.

NOTE: Processing existing extents is done in a later patch.

Management of the region extent devices must be synchronized with potential uses of the memory within the DAX layer. Create region extent devices as children of the cxl_dax_region device such that the DAX region driver can co-drive them and synchronize with the DAX layer. Synchronization and management is handled in a subsequent patch.

Tag support within the DAX layer is not yet supported. To maintain compatibility legacy DAX/region processing only tags with a value of 0 are allowed. This defines existing DAX devices as having a 0 tag which makes the most logical sense as a default.

Quoted 6 lines

Process DCD events and create region devices.

Hi Ira,

I guess you missed my comments for V3, I comment it again for this patch.

Quoted 5 lines
drivers/cxl/core/extent.c new file +5 -0
+static bool extents_contain(struct cxl_dax_region *cxlr_dax,
+			    struct cxl_endpoint_decoder *cxled,
+			    struct range *new_range)
+{
+	struct device *extent_device;

drivers/cxl/core/extent.c line 160

Quoted 10 lines
drivers/cxl/core/extent.c new file +10 -0
+	struct match_data md = {
+		.cxled = cxled,
+		.new_range = new_range,
+	};
+
+	extent_device = device_find_child(&cxlr_dax->dev, &md, match_contains);
+	if (!extent_device)
+		return false;
+
+	put_device(extent_device);

could use __free(put_device) to drop this 'put_device(extent_device)'

Quoted 2 lines
(quoted diff) +2 -0
+	return true;
+}

[...]

Quoted 5 lines
drivers/cxl/core/extent.c new file +5 -0
+static bool extents_overlap(struct cxl_dax_region *cxlr_dax,
+			    struct cxl_endpoint_decoder *cxled,
+			    struct range *new_range)
+{
+	struct device *extent_device;

drivers/cxl/core/extent.c line 197

Quoted 10 lines
drivers/cxl/core/extent.c new file +10 -0
+	struct match_data md = {
+		.cxled = cxled,
+		.new_range = new_range,
+	};
+
+	extent_device = device_find_child(&cxlr_dax->dev, &md, match_overlaps);
+	if (!extent_device)
+		return false;
+
+	put_device(extent_device);

Same as above.

Quoted 3 lines
drivers/cxl/core/extent.c new file +3 -0
+	return true;
+}
+

[...]

Quoted 35 lines
drivers/cxl/core/mbox.c +35 -0
+static int cxl_send_dc_response(struct cxl_memdev_state *mds, int opcode,
+				struct xarray *extent_array, int cnt)
+{
+	struct cxl_mailbox *cxl_mbox = &mds->cxlds.cxl_mbox;
+	struct cxl_mbox_dc_response *p;
+	struct cxl_mbox_cmd mbox_cmd;
+	struct cxl_extent *extent;
+	unsigned long index;
+	u32 pl_index;
+	int rc;
+
+	size_t pl_size = struct_size(p, extent_list, cnt);
+	u32 max_extents = cnt;
+
+	/* May have to use more bit on response. */
+	if (pl_size > cxl_mbox->payload_size) {
+		max_extents = (cxl_mbox->payload_size - sizeof(*p)) /
+			      sizeof(struct updated_extent_list);
+		pl_size = struct_size(p, extent_list, max_extents);
+	}
+
+	struct cxl_mbox_dc_response *response __free(kfree) =
+						kzalloc(pl_size, GFP_KERNEL);
+	if (!response)
+		return -ENOMEM;
+
+	pl_index = 0;
+	xa_for_each(extent_array, index, extent) {
+
+		response->extent_list[pl_index].dpa_start = extent->start_dpa;
+		response->extent_list[pl_index].length = extent->length;
+		pl_index++;
+		response->extent_list_size = cpu_to_le32(pl_index);
+
+		if (pl_index == max_extents) {

drivers/cxl/core/mbox.c static int cxl_clear_event_record(struct cxl_memdev_state *mds, · line 1116

Quoted 10 lines
drivers/cxl/core/mbox.c +10 -0
+			mbox_cmd = (struct cxl_mbox_cmd) {
+				.opcode = opcode,
+				.size_in = struct_size(response, extent_list,
+						       pl_index),
+				.payload_in = response,
+			};
+
+			response->flags = 0;
+			if (pl_index < cnt)
+				response->flags &= CXL_DCD_EVENT_MORE;

It should be 'response->flags |= CXL_DCD_EVENT_MORE' here.

Another issue is if 'cnt' is N times bigger than 'max_extents'(e,g. cnt=20, max_extents=10). all responses will be sent in this xa_for_each(), and CXL_DCD_EVENT_MORE will be set in the last response but it should not be set in these cases.

Quoted 25 lines
drivers/cxl/core/mbox.c +25 -0
+
+			rc = cxl_internal_send_cmd(cxl_mbox, &mbox_cmd);
+			if (rc)
+				return rc;
+			pl_index = 0;
+		}
+	}
+
+	if (cnt == 0 || pl_index) {
+		mbox_cmd = (struct cxl_mbox_cmd) {
+			.opcode = opcode,
+			.size_in = struct_size(response, extent_list,
+					       pl_index),
+			.payload_in = response,
+		};
+
+		response->flags = 0;
+		rc = cxl_internal_send_cmd(cxl_mbox, &mbox_cmd);
+		if (rc)
+			return rc;
+	}
+
+	return 0;
+}
+
author
#88 lore raw

Re: [PATCH v4 21/28] cxl/extent: Process DCD events and realize region extents

Quoted 3 lines · Li, Ming4 wrote:
Quoted 2 lines · On 10/8/2024 7:16 AM, [email protected] wrote:

From: Navneet Singh <[email protected]>

[snip]

Quoted 8 lines
Quoted 5 lines

Hi Ira,

I guess you missed my comments for V3, I comment it again for this patch.

Apologies. Yes I totally missed your reply. :-(

Quoted 17 lines
Quoted 6 lines
(quoted diff) +5 -0
+static bool extents_contain(struct cxl_dax_region *cxlr_dax,
+			    struct cxl_endpoint_decoder *cxled,
+			    struct range *new_range)
+{
+	struct device *extent_device;

Review

Quoted 10 lines
(quoted diff) +10 -0
+	struct match_data md = {
+		.cxled = cxled,
+		.new_range = new_range,
+	};
+
+	extent_device = device_find_child(&cxlr_dax->dev, &md, match_contains);
+	if (!extent_device)
+		return false;
+
+	put_device(extent_device);

could use __free(put_device) to drop this 'put_device(extent_device)'

Yep.

Quoted 19 lines
Quoted 2 lines
(quoted diff) +2 -0
+	return true;
+}

[...]

Quoted 5 lines
(quoted diff) +5 -0
+static bool extents_overlap(struct cxl_dax_region *cxlr_dax,
+			    struct cxl_endpoint_decoder *cxled,
+			    struct range *new_range)
+{
+	struct device *extent_device;

Review

Quoted 10 lines
(quoted diff) +10 -0
+	struct match_data md = {
+		.cxled = cxled,
+		.new_range = new_range,
+	};
+
+	extent_device = device_find_child(&cxlr_dax->dev, &md, match_overlaps);
+	if (!extent_device)
+		return false;
+
+	put_device(extent_device);

Same as above.

Done.

Quoted 43 lines
Quoted 3 lines
(quoted diff) +3 -0
+	return true;
+}
+

[...]

Quoted 39 lines
(quoted diff) +39 -0
+static int cxl_send_dc_response(struct cxl_memdev_state *mds, int opcode,
+				struct xarray *extent_array, int cnt)
+{
+	struct cxl_mailbox *cxl_mbox = &mds->cxlds.cxl_mbox;
+	struct cxl_mbox_dc_response *p;
+	struct cxl_mbox_cmd mbox_cmd;
+	struct cxl_extent *extent;
+	unsigned long index;
+	u32 pl_index;
+	int rc;
+
+	size_t pl_size = struct_size(p, extent_list, cnt);
+	u32 max_extents = cnt;
+
+	/* May have to use more bit on response. */
+	if (pl_size > cxl_mbox->payload_size) {
+		max_extents = (cxl_mbox->payload_size - sizeof(*p)) /
+			      sizeof(struct updated_extent_list);
+		pl_size = struct_size(p, extent_list, max_extents);
+	}
+
+	struct cxl_mbox_dc_response *response __free(kfree) =
+						kzalloc(pl_size, GFP_KERNEL);
+	if (!response)
+		return -ENOMEM;
+
+	pl_index = 0;
+	xa_for_each(extent_array, index, extent) {
+
+		response->extent_list[pl_index].dpa_start = extent->start_dpa;
+		response->extent_list[pl_index].length = extent->length;
+		pl_index++;
+		response->extent_list_size = cpu_to_le32(pl_index);
+
+		if (pl_index == max_extents) {
+			mbox_cmd = (struct cxl_mbox_cmd) {
+				.opcode = opcode,
+				.size_in = struct_size(response, extent_list,
+						       pl_index),
Quoted 8 lines

Review

Quoted 6 lines
(quoted diff) +6 -0
+				.payload_in = response,
+			};
+
+			response->flags = 0;
+			if (pl_index < cnt)
+				response->flags &= CXL_DCD_EVENT_MORE;

It should be 'response->flags |= CXL_DCD_EVENT_MORE' here.

Ah yea. Good catch.

Quoted 3 lines

Another issue is if 'cnt' is N times bigger than 'max_extents'(e,g. cnt=20, max_extents=10). all responses will be sent in this xa_for_each(), and CXL_DCD_EVENT_MORE will be set in the last response but it should not be set in these cases.

Ah yes. cnt must be decremented. As I looked at the patch just now the

	if (cnt == 0 || pl_index)

... seemed very wrong to me. That change masked this bug.

This should fix it:

drivers/cxl/core/mbox.c +2 -1
index d66beec687a0..99200274dea8 100644
@@ -1119,10 +1119,11 @@ static int cxl_send_dc_response(struct cxl_memdev_state *mds, int opcode,
                        if (rc)
                                return rc;
                        pl_index = 0;
+                       cnt -= pl_index;
                }
        }
 
-       if (cnt == 0 || pl_index) {
+       if (pl_index) {
                mbox_cmd = (struct cxl_mbox_cmd) {
                        .opcode = opcode,
                        .size_in = struct_size(response, extent_list,

Thank you, and sorry again for missing your feedback.

Ira

[snip]

#89 lore raw

Re: [PATCH v4 21/28] cxl/extent: Process DCD events and realize region extents

Quoted 4 lines · Ira Weiny
Quoted 3 lines · Li, Ming4 wrote:
Quoted 2 lines · On 10/8/2024 7:16 AM, [email protected] wrote:

From: Navneet Singh <[email protected]>

[snip]

Quoted 63 lines
Quoted 38 lines
Quoted 38 lines
(quoted diff) +38 -0
+static int cxl_send_dc_response(struct cxl_memdev_state *mds, int opcode,
+				struct xarray *extent_array, int cnt)
+{
+	struct cxl_mailbox *cxl_mbox = &mds->cxlds.cxl_mbox;
+	struct cxl_mbox_dc_response *p;
+	struct cxl_mbox_cmd mbox_cmd;
+	struct cxl_extent *extent;
+	unsigned long index;
+	u32 pl_index;
+	int rc;
+
+	size_t pl_size = struct_size(p, extent_list, cnt);
+	u32 max_extents = cnt;
+
+	/* May have to use more bit on response. */
+	if (pl_size > cxl_mbox->payload_size) {
+		max_extents = (cxl_mbox->payload_size - sizeof(*p)) /
+			      sizeof(struct updated_extent_list);
+		pl_size = struct_size(p, extent_list, max_extents);
+	}
+
+	struct cxl_mbox_dc_response *response __free(kfree) =
+						kzalloc(pl_size, GFP_KERNEL);
+	if (!response)
+		return -ENOMEM;
+
+	pl_index = 0;
+	xa_for_each(extent_array, index, extent) {
+
+		response->extent_list[pl_index].dpa_start = extent->start_dpa;
+		response->extent_list[pl_index].length = extent->length;
+		pl_index++;
+		response->extent_list_size = cpu_to_le32(pl_index);
+
+		if (pl_index == max_extents) {
+			mbox_cmd = (struct cxl_mbox_cmd) {
+				.opcode = opcode,
+				.size_in = struct_size(response, extent_list,
Quoted 8 lines

Review

Quoted 7 lines
(quoted diff) +7 -0
+						       pl_index),
+				.payload_in = response,
+			};
+
+			response->flags = 0;
+			if (pl_index < cnt)
+				response->flags &= CXL_DCD_EVENT_MORE;

It should be 'response->flags |= CXL_DCD_EVENT_MORE' here.

Ah yea. Good catch.

Quoted 2 lines

Another issue is if 'cnt' is N times bigger than 'max_extents'(e,g. cnt=20, max_extents=10). all responses will be sent in this xa_for_each(), and CXL_DCD_EVENT_MORE will be set in the last response but it should not be set in these cases.

Ah yes. cnt must be decremented. As I looked at the patch just now the

	if (cnt == 0 || pl_index)

... seemed very wrong to me. That change masked this bug.

This should fix it:

drivers/cxl/core/mbox.c +0 -0
index d66beec687a0..99200274dea8 100644

drivers/cxl/core/mbox.c static int cxl_send_dc_response(struct cxl_memdev_state *mds, int opcode, · line 1122

Quoted 4 lines
drivers/cxl/core/mbox.c +1 -0
@@ -1119,10 +1119,11 @@ static int cxl_send_dc_response(struct cxl_memdev_state *mds, int opcode,
                        if (rc)
                                return rc;
                        pl_index = 0;
+                       cnt -= pl_index;

should update cnt before pl_index is reset to 0.

the cnt is a one of parameters of cxl_send_dc_response(), that means the caller gives the value of cnt, is that possible if the size of extent_array is larger than cnt? Should skip remain extents in extent_array when cnt is equal to 0?

Quoted 15 lines
drivers/cxl/core/mbox.c +1 -1
                }
        }
 
-       if (cnt == 0 || pl_index) {
+       if (pl_index) {
                mbox_cmd = (struct cxl_mbox_cmd) {
                        .opcode = opcode,
                        .size_in = struct_size(response, extent_list,

Thank you, and sorry again for missing your feedback.

Ira

[snip]

author
#90 lore raw

Re: [PATCH v4 21/28] cxl/extent: Process DCD events and realize region extents

Quoted 4 lines · Li, Ming4 wrote:
Quoted 3 lines · Ira Weiny
Quoted 2 lines · Li, Ming4 wrote:
Quoted 1 line · On 10/8/2024 7:16 AM, [email protected] wrote:

From: Navneet Singh <[email protected]>

[snip]

Quoted 13 lines

drivers/cxl/core/mbox.c static int cxl_send_dc_response(struct cxl_memdev_state *mds, int opcode, · line 1122

Quoted 11 lines

This should fix it:

drivers/cxl/core/mbox.c +1 -0
index d66beec687a0..99200274dea8 100644
@@ -1119,10 +1119,11 @@ static int cxl_send_dc_response(struct cxl_memdev_state *mds, int opcode,
                        if (rc)
                                return rc;
                        pl_index = 0;
+                       cnt -= pl_index;

should update cnt before pl_index is reset to 0.

Of course.

Quoted 4 lines

the cnt is a one of parameters of cxl_send_dc_response(), that means the caller gives the value of cnt, is that possible if the size of extent_array is larger than cnt?

No both callers ensure cnt is equal to the number of elements in the array. Otherwise cxl_send_dc_response() would need to iterate the array to determine this size itself. It is just more efficient to pass that count which was already determined.

Quoted 3 lines

Should skip remain extents in
extent_array when cnt is equal to 0?

No not skip. But this makes me rethink this solution. The spec requires a response even if 0 extents are accepted. That is what drove me to the buggy solution before. I'll have to think on this a bit. I'd like to not have weird gotos but that is the easiest solution I see.

Ira

#91 lore raw

Re: [PATCH v4 21/28] cxl/extent: Process DCD events and realize region extents

In reply to Ira Weiny
Quoted 131 lines · On Wed, 9 Oct 2024 14:49:09 -0500 Ira Weiny <[email protected]> wrote:
Quoted 3 lines · Li, Ming4 wrote:
Quoted 2 lines · On 10/8/2024 7:16 AM, [email protected] wrote:

From: Navneet Singh <[email protected]>

[snip]

Quoted 8 lines
Quoted 5 lines

Hi Ira,

I guess you missed my comments for V3, I comment it again for this patch.

Apologies. Yes I totally missed your reply. :-(

Quoted 17 lines
Quoted 6 lines
(quoted diff) +5 -0
+static bool extents_contain(struct cxl_dax_region *cxlr_dax,
+			    struct cxl_endpoint_decoder *cxled,
+			    struct range *new_range)
+{
+	struct device *extent_device;

Review

Quoted 10 lines
(quoted diff) +10 -0
+	struct match_data md = {
+		.cxled = cxled,
+		.new_range = new_range,
+	};
+
+	extent_device = device_find_child(&cxlr_dax->dev, &md, match_contains);
+	if (!extent_device)
+		return false;
+
+	put_device(extent_device);  

could use __free(put_device) to drop this 'put_device(extent_device)'

Yep.

Quoted 19 lines
Quoted 2 lines
(quoted diff) +2 -0
+	return true;
+}  

[...]

Quoted 5 lines
(quoted diff) +5 -0
+static bool extents_overlap(struct cxl_dax_region *cxlr_dax,
+			    struct cxl_endpoint_decoder *cxled,
+			    struct range *new_range)
+{
+	struct device *extent_device;

Review

Quoted 10 lines
(quoted diff) +10 -0
+	struct match_data md = {
+		.cxled = cxled,
+		.new_range = new_range,
+	};
+
+	extent_device = device_find_child(&cxlr_dax->dev, &md, match_overlaps);
+	if (!extent_device)
+		return false;
+
+	put_device(extent_device);  

Same as above.

Done.

Quoted 43 lines
Quoted 3 lines
(quoted diff) +3 -0
+	return true;
+}
+  

[...]

Quoted 39 lines
(quoted diff) +39 -0
+static int cxl_send_dc_response(struct cxl_memdev_state *mds, int opcode,
+				struct xarray *extent_array, int cnt)
+{
+	struct cxl_mailbox *cxl_mbox = &mds->cxlds.cxl_mbox;
+	struct cxl_mbox_dc_response *p;
+	struct cxl_mbox_cmd mbox_cmd;
+	struct cxl_extent *extent;
+	unsigned long index;
+	u32 pl_index;
+	int rc;
+
+	size_t pl_size = struct_size(p, extent_list, cnt);
+	u32 max_extents = cnt;
+
+	/* May have to use more bit on response. */
+	if (pl_size > cxl_mbox->payload_size) {
+		max_extents = (cxl_mbox->payload_size - sizeof(*p)) /
+			      sizeof(struct updated_extent_list);
+		pl_size = struct_size(p, extent_list, max_extents);
+	}
+
+	struct cxl_mbox_dc_response *response __free(kfree) =
+						kzalloc(pl_size, GFP_KERNEL);
+	if (!response)
+		return -ENOMEM;
+
+	pl_index = 0;
+	xa_for_each(extent_array, index, extent) {
+
+		response->extent_list[pl_index].dpa_start = extent->start_dpa;
+		response->extent_list[pl_index].length = extent->length;
+		pl_index++;
+		response->extent_list_size = cpu_to_le32(pl_index);
+
+		if (pl_index == max_extents) {
+			mbox_cmd = (struct cxl_mbox_cmd) {
+				.opcode = opcode,
+				.size_in = struct_size(response, extent_list,
+						       pl_index),
Quoted 8 lines

Review

Quoted 6 lines
(quoted diff) +6 -0
+				.payload_in = response,
+			};
+
+			response->flags = 0;
+			if (pl_index < cnt)
+				response->flags &= CXL_DCD_EVENT_MORE;  

It should be 'response->flags |= CXL_DCD_EVENT_MORE' here.

Ah yea. Good catch.

Quoted 3 lines

Another issue is if 'cnt' is N times bigger than 'max_extents'(e,g. cnt=20, max_extents=10). all responses will be sent in this xa_for_each(), and CXL_DCD_EVENT_MORE will be set in the last response but it should not be set in these cases.

Ah yes. cnt must be decremented. As I looked at the patch just now the

	if (cnt == 0 || pl_index)

... seemed very wrong to me. That change masked this bug.

This should fix it:

drivers/cxl/core/mbox.c +0 -0
index d66beec687a0..99200274dea8 100644

drivers/cxl/core/mbox.c static int cxl_send_dc_response(struct cxl_memdev_state *mds, int opcode, · line 1125

Quoted 8 lines
drivers/cxl/core/mbox.c +1 -1
@@ -1119,10 +1119,11 @@ static int cxl_send_dc_response(struct cxl_memdev_state *mds, int opcode,
                        if (rc)
                                return rc;
                        pl_index = 0;
+                       cnt -= pl_index;
                }
        }
 
-       if (cnt == 0 || pl_index) {

I thought this cnt == 0 check was to deal with the no valid extents case where an empty reply is needed.

Quoted 12 lines
drivers/cxl/core/mbox.c +1 -0
+       if (pl_index) {
                mbox_cmd = (struct cxl_mbox_cmd) {
                        .opcode = opcode,
                        .size_in = struct_size(response, extent_list,

Thank you, and sorry again for missing your feedback.

Ira

[snip]

#92 lore raw

Re: [PATCH v4 21/28] cxl/extent: Process DCD events and realize region extents

Quoted 141 lines · Jonathan Cameron
Quoted 138 lines · On Wed, 9 Oct 2024 14:49:09 -0500 Ira Weiny <[email protected]> wrote:
Quoted 3 lines · Li, Ming4 wrote:
Quoted 2 lines · On 10/8/2024 7:16 AM, [email protected] wrote:

From: Navneet Singh <[email protected]>

[snip]

Quoted 8 lines
Quoted 5 lines

Hi Ira,

I guess you missed my comments for V3, I comment it again for this patch.

Apologies. Yes I totally missed your reply. :-(

Quoted 17 lines
Quoted 6 lines
(quoted diff) +5 -0
+static bool extents_contain(struct cxl_dax_region *cxlr_dax,
+			    struct cxl_endpoint_decoder *cxled,
+			    struct range *new_range)
+{
+	struct device *extent_device;

Review

Quoted 10 lines
(quoted diff) +10 -0
+	struct match_data md = {
+		.cxled = cxled,
+		.new_range = new_range,
+	};
+
+	extent_device = device_find_child(&cxlr_dax->dev, &md, match_contains);
+	if (!extent_device)
+		return false;
+
+	put_device(extent_device);  

could use __free(put_device) to drop this 'put_device(extent_device)'

Yep.

Quoted 19 lines
Quoted 2 lines
(quoted diff) +2 -0
+	return true;
+}  

[...]

Quoted 5 lines
(quoted diff) +5 -0
+static bool extents_overlap(struct cxl_dax_region *cxlr_dax,
+			    struct cxl_endpoint_decoder *cxled,
+			    struct range *new_range)
+{
+	struct device *extent_device;

Review

Quoted 10 lines
(quoted diff) +10 -0
+	struct match_data md = {
+		.cxled = cxled,
+		.new_range = new_range,
+	};
+
+	extent_device = device_find_child(&cxlr_dax->dev, &md, match_overlaps);
+	if (!extent_device)
+		return false;
+
+	put_device(extent_device);  

Same as above.

Done.

Quoted 43 lines
Quoted 3 lines
(quoted diff) +3 -0
+	return true;
+}
+  

[...]

Quoted 39 lines
(quoted diff) +39 -0
+static int cxl_send_dc_response(struct cxl_memdev_state *mds, int opcode,
+				struct xarray *extent_array, int cnt)
+{
+	struct cxl_mailbox *cxl_mbox = &mds->cxlds.cxl_mbox;
+	struct cxl_mbox_dc_response *p;
+	struct cxl_mbox_cmd mbox_cmd;
+	struct cxl_extent *extent;
+	unsigned long index;
+	u32 pl_index;
+	int rc;
+
+	size_t pl_size = struct_size(p, extent_list, cnt);
+	u32 max_extents = cnt;
+
+	/* May have to use more bit on response. */
+	if (pl_size > cxl_mbox->payload_size) {
+		max_extents = (cxl_mbox->payload_size - sizeof(*p)) /
+			      sizeof(struct updated_extent_list);
+		pl_size = struct_size(p, extent_list, max_extents);
+	}
+
+	struct cxl_mbox_dc_response *response __free(kfree) =
+						kzalloc(pl_size, GFP_KERNEL);
+	if (!response)
+		return -ENOMEM;
+
+	pl_index = 0;
+	xa_for_each(extent_array, index, extent) {
+
+		response->extent_list[pl_index].dpa_start = extent->start_dpa;
+		response->extent_list[pl_index].length = extent->length;
+		pl_index++;
+		response->extent_list_size = cpu_to_le32(pl_index);
+
+		if (pl_index == max_extents) {
+			mbox_cmd = (struct cxl_mbox_cmd) {
+				.opcode = opcode,
+				.size_in = struct_size(response, extent_list,
+						       pl_index),
Quoted 8 lines

Review

Quoted 6 lines
(quoted diff) +6 -0
+				.payload_in = response,
+			};
+
+			response->flags = 0;
+			if (pl_index < cnt)
+				response->flags &= CXL_DCD_EVENT_MORE;  

It should be 'response->flags |= CXL_DCD_EVENT_MORE' here.

Ah yea. Good catch.

Quoted 3 lines

Another issue is if 'cnt' is N times bigger than 'max_extents'(e,g. cnt=20, max_extents=10). all responses will be sent in this xa_for_each(), and CXL_DCD_EVENT_MORE will be set in the last response but it should not be set in these cases.

Ah yes. cnt must be decremented. As I looked at the patch just now the

	if (cnt == 0 || pl_index)

... seemed very wrong to me. That change masked this bug.

This should fix it:

drivers/cxl/core/mbox.c +1 -0
index d66beec687a0..99200274dea8 100644
@@ -1119,10 +1119,11 @@ static int cxl_send_dc_response(struct cxl_memdev_state *mds, int opcode,
                        if (rc)
                                return rc;
                        pl_index = 0;
+                       cnt -= pl_index;
                }
        }
 
Quoted 4 lines

drivers/cxl/core/mbox.c static int cxl_send_dc_response(struct cxl_memdev_state *mds, int opcode, · line 1125

Quoted 1 line
drivers/cxl/core/mbox.c +0 -1
-       if (cnt == 0 || pl_index) {

I thought this cnt == 0 check was to deal with the no valid extents case where an empty reply is needed.

Agreed. Based on current code logic, there are two cases that cnt == 0:
1. no extent is accepted so cnt is passed as 0;
2. cnt was decreased to 0 and response has already been sent.

For case 1, we still need to send a response with zero extents; For case 2, we do not need to handle.

Fan

Quoted 15 lines
Quoted 14 lines
drivers/cxl/core/mbox.c +1 -0
+       if (pl_index) {
                mbox_cmd = (struct cxl_mbox_cmd) {
                        .opcode = opcode,
                        .size_in = struct_size(response, extent_list,

Thank you, and sorry again for missing your feedback.

Ira

[snip]

Signature · 1 line
Fan Ni
author
#93 lore raw

Re: [PATCH v4 21/28] cxl/extent: Process DCD events and realize region extents

In reply to Jonathan Cameron
Quoted 7 lines · Jonathan Cameron
Quoted 4 lines · On Wed, 9 Oct 2024 14:49:09 -0500 Ira Weiny <[email protected]> wrote:
Quoted 3 lines · Li, Ming4 wrote:
Quoted 2 lines · On 10/8/2024 7:16 AM, [email protected] wrote:

From: Navneet Singh <[email protected]>

[snip]

Quoted 75 lines
Quoted 75 lines
Quoted 40 lines
Quoted 40 lines
(quoted diff) +39 -0
+static int cxl_send_dc_response(struct cxl_memdev_state *mds, int opcode,
+				struct xarray *extent_array, int cnt)
+{
+	struct cxl_mailbox *cxl_mbox = &mds->cxlds.cxl_mbox;
+	struct cxl_mbox_dc_response *p;
+	struct cxl_mbox_cmd mbox_cmd;
+	struct cxl_extent *extent;
+	unsigned long index;
+	u32 pl_index;
+	int rc;
+
+	size_t pl_size = struct_size(p, extent_list, cnt);
+	u32 max_extents = cnt;
+
+	/* May have to use more bit on response. */
+	if (pl_size > cxl_mbox->payload_size) {
+		max_extents = (cxl_mbox->payload_size - sizeof(*p)) /
+			      sizeof(struct updated_extent_list);
+		pl_size = struct_size(p, extent_list, max_extents);
+	}
+
+	struct cxl_mbox_dc_response *response __free(kfree) =
+						kzalloc(pl_size, GFP_KERNEL);
+	if (!response)
+		return -ENOMEM;
+
+	pl_index = 0;
+	xa_for_each(extent_array, index, extent) {
+
+		response->extent_list[pl_index].dpa_start = extent->start_dpa;
+		response->extent_list[pl_index].length = extent->length;
+		pl_index++;
+		response->extent_list_size = cpu_to_le32(pl_index);
+
+		if (pl_index == max_extents) {
+			mbox_cmd = (struct cxl_mbox_cmd) {
+				.opcode = opcode,
+				.size_in = struct_size(response, extent_list,
+						       pl_index),
Quoted 8 lines

Review

Quoted 6 lines
(quoted diff) +6 -0
+				.payload_in = response,
+			};
+
+			response->flags = 0;
+			if (pl_index < cnt)
+				response->flags &= CXL_DCD_EVENT_MORE;  

It should be 'response->flags |= CXL_DCD_EVENT_MORE' here.

Ah yea. Good catch.

Quoted 3 lines

Another issue is if 'cnt' is N times bigger than 'max_extents'(e,g. cnt=20, max_extents=10). all responses will be sent in this xa_for_each(), and CXL_DCD_EVENT_MORE will be set in the last response but it should not be set in these cases.

Ah yes. cnt must be decremented. As I looked at the patch just now the

	if (cnt == 0 || pl_index)

... seemed very wrong to me. That change masked this bug.

This should fix it:

drivers/cxl/core/mbox.c +1 -0
index d66beec687a0..99200274dea8 100644
@@ -1119,10 +1119,11 @@ static int cxl_send_dc_response(struct cxl_memdev_state *mds, int opcode,
                        if (rc)
                                return rc;
                        pl_index = 0;
+                       cnt -= pl_index;
                }
        }
 
Quoted 4 lines

drivers/cxl/core/mbox.c static int cxl_send_dc_response(struct cxl_memdev_state *mds, int opcode, · line 1125

Quoted 1 line
drivers/cxl/core/mbox.c +0 -1
-       if (cnt == 0 || pl_index) {

I thought this cnt == 0 check was to deal with the no valid extents case where an empty reply is needed.

Yes but the bug found by Ming needs to be handled too. I see Fan is also questioning this code.

So... for clarity among all of us here is the new function. I'm not thrilled with the use of a goto but I think it is ok here.

Ira

static int cxl_send_dc_response(struct cxl_memdev_state *mds, int opcode,
struct xarray *extent_array, int cnt)

{                                                                              
       struct cxl_mailbox *cxl_mbox = &mds->cxlds.cxl_mbox;                    
       struct cxl_mbox_dc_response *p;                                         
       struct cxl_mbox_cmd mbox_cmd;                                           
       struct cxl_extent *extent;                                              
       unsigned long index;                                                    
       u32 pl_index;                                                           
       int rc;                                                                 
                                                                               
       size_t pl_size = struct_size(p, extent_list, cnt);                      
       u32 max_extents = cnt;                                                  
                                                                               
       /* May have to use more bit on response. */                             
       if (pl_size > cxl_mbox->payload_size) {                                 
               max_extents = (cxl_mbox->payload_size - sizeof(*p)) /           
                             sizeof(struct updated_extent_list);               
               pl_size = struct_size(p, extent_list, max_extents);             
       }                                                                       
                                                                               
       struct cxl_mbox_dc_response *response __free(kfree) =                   
                                               kzalloc(pl_size, GFP_KERNEL);   
       if (!response)                                                          
               return -ENOMEM;                                                 
                                                                               
       pl_index = 0;                                                           
       if (cnt == 0)                                                           
               goto send_zero_accepted;                                        
                                                                               
       xa_for_each(extent_array, index, extent) {                              
               response->extent_list[pl_index].dpa_start = extent->start_dpa;  
               response->extent_list[pl_index].length = extent->length;        
               pl_index++;                                                     
               response->extent_list_size = cpu_to_le32(pl_index);             
                                                                               
               if (pl_index == max_extents) {                                  
                       mbox_cmd = (struct cxl_mbox_cmd) {                      
                               .opcode = opcode,                               
                               .size_in = struct_size(response, extent_list,   
                                                      pl_index),               
                               .payload_in = response,                         
                       };                                                      
                                                                               
                       response->flags = 0;                                    
                       if (pl_index < cnt)                                     
                               response->flags &= CXL_DCD_EVENT_MORE;          
                                                                               
                       rc = cxl_internal_send_cmd(cxl_mbox, &mbox_cmd);        
                       if (rc)                                                 
                               return rc;                                      
                       cnt -= pl_index;                                        
                       pl_index = 0;                                           
               }                                                               
       }                                                                       
                                                                               
       if (!pl_index)                                                          
               return 0;                                                       

send_zero_accepted:

       mbox_cmd = (struct cxl_mbox_cmd) {                                      
               .opcode = opcode,                                               
               .size_in = struct_size(response, extent_list,                   
                                      pl_index),                               
               .payload_in = response,                                         
       };                                                                      
                                                                               
       response->flags = 0;                                                    
       return cxl_internal_send_cmd(cxl_mbox, &mbox_cmd);                      
}                                                                              
#94 lore raw

Re: [PATCH v4 21/28] cxl/extent: Process DCD events and realize region extents

Quoted 91 lines · On Thu, 17 Oct 2024 16:15:03 -0500 Ira Weiny <[email protected]> wrote:
Quoted 7 lines · Jonathan Cameron
Quoted 4 lines · On Wed, 9 Oct 2024 14:49:09 -0500 Ira Weiny <[email protected]> wrote:
Quoted 3 lines · Li, Ming4 wrote:
Quoted 2 lines · On 10/8/2024 7:16 AM, [email protected] wrote:

From: Navneet Singh <[email protected]>

[snip]

Quoted 75 lines
Quoted 75 lines
Quoted 40 lines
Quoted 40 lines
(quoted diff) +39 -0
+static int cxl_send_dc_response(struct cxl_memdev_state *mds, int opcode,
+				struct xarray *extent_array, int cnt)
+{
+	struct cxl_mailbox *cxl_mbox = &mds->cxlds.cxl_mbox;
+	struct cxl_mbox_dc_response *p;
+	struct cxl_mbox_cmd mbox_cmd;
+	struct cxl_extent *extent;
+	unsigned long index;
+	u32 pl_index;
+	int rc;
+
+	size_t pl_size = struct_size(p, extent_list, cnt);
+	u32 max_extents = cnt;
+
+	/* May have to use more bit on response. */
+	if (pl_size > cxl_mbox->payload_size) {
+		max_extents = (cxl_mbox->payload_size - sizeof(*p)) /
+			      sizeof(struct updated_extent_list);
+		pl_size = struct_size(p, extent_list, max_extents);
+	}
+
+	struct cxl_mbox_dc_response *response __free(kfree) =
+						kzalloc(pl_size, GFP_KERNEL);
+	if (!response)
+		return -ENOMEM;
+
+	pl_index = 0;
+	xa_for_each(extent_array, index, extent) {
+
+		response->extent_list[pl_index].dpa_start = extent->start_dpa;
+		response->extent_list[pl_index].length = extent->length;
+		pl_index++;
+		response->extent_list_size = cpu_to_le32(pl_index);
+
+		if (pl_index == max_extents) {
+			mbox_cmd = (struct cxl_mbox_cmd) {
+				.opcode = opcode,
+				.size_in = struct_size(response, extent_list,
+						       pl_index),
Quoted 8 lines

Review

Quoted 6 lines
(quoted diff) +6 -0
+				.payload_in = response,
+			};
+
+			response->flags = 0;
+			if (pl_index < cnt)
+				response->flags &= CXL_DCD_EVENT_MORE;    

It should be 'response->flags |= CXL_DCD_EVENT_MORE' here.

Ah yea. Good catch.

Quoted 3 lines

Another issue is if 'cnt' is N times bigger than 'max_extents'(e,g. cnt=20, max_extents=10). all responses will be sent in this xa_for_each(), and CXL_DCD_EVENT_MORE will be set in the last response but it should not be set in these cases.

Ah yes. cnt must be decremented. As I looked at the patch just now the

	if (cnt == 0 || pl_index)

... seemed very wrong to me. That change masked this bug.

This should fix it:

drivers/cxl/core/mbox.c +1 -0
index d66beec687a0..99200274dea8 100644
@@ -1119,10 +1119,11 @@ static int cxl_send_dc_response(struct cxl_memdev_state *mds, int opcode,
                        if (rc)
                                return rc;
                        pl_index = 0;
+                       cnt -= pl_index;
                }
        }
 
Quoted 4 lines

drivers/cxl/core/mbox.c static int cxl_send_dc_response(struct cxl_memdev_state *mds, int opcode, · line 1125

Quoted 1 line
drivers/cxl/core/mbox.c +0 -1
-       if (cnt == 0 || pl_index) {  

I thought this cnt == 0 check was to deal with the no valid extents case where an empty reply is needed.

Quoted 5 lines

Yes but the bug found by Ming needs to be handled too. I see Fan is also questioning this code.

So... for clarity among all of us here is the new function. I'm not thrilled with the use of a goto but I think it is ok here.

Easy enough to avoid and I don't think it hurts readability much to do so.

Your code should work though.

Quoted 30 lines

Ira

static int cxl_send_dc_response(struct cxl_memdev_state *mds, int opcode,
struct xarray *extent_array, int cnt)

{                                                                              
       struct cxl_mailbox *cxl_mbox = &mds->cxlds.cxl_mbox;                    
       struct cxl_mbox_dc_response *p;                                         
       struct cxl_mbox_cmd mbox_cmd;                                           
       struct cxl_extent *extent;                                              
       unsigned long index;                                                    
       u32 pl_index;                                                           
       int rc;                                                                 
                                                                               
       size_t pl_size = struct_size(p, extent_list, cnt);                      
       u32 max_extents = cnt;                                              
                                                                               
       /* May have to use more bit on response. */                             
       if (pl_size > cxl_mbox->payload_size) {                                 
               max_extents = (cxl_mbox->payload_size - sizeof(*p)) /           
                             sizeof(struct updated_extent_list);               
               pl_size = struct_size(p, extent_list, max_extents);
             
       }                                                                       
                                                                               
       struct cxl_mbox_dc_response *response __free(kfree) =                   
                                               kzalloc(pl_size, GFP_KERNEL);   
       if (!response)                                                          
               return -ENOMEM;                                                 
Quoted 8 lines
       pl_index = 0;                                                           
       if (cnt == 0)                                                           
               goto send_zero_accepted;
       xa_for_each(extent_array, index, extent) {                              
               response->extent_list[pl_index].dpa_start = extent->start_dpa;  
               response->extent_list[pl_index].length = extent->length;        
               pl_index++;                                                     
               response->extent_list_size = cpu_to_le32(pl_index);    

Why set this here - to me makes more sense to set it only once but I can see the logic either way.

Quoted 25 lines
  
               if (pl_index == max_extents) {                                  
                       mbox_cmd = (struct cxl_mbox_cmd) {                      
                               .opcode = opcode,                               
                               .size_in = struct_size(response, extent_list,   
                                                      pl_index),               
                               .payload_in = response,                         
                       };                                                      
                                                                               
                       response->flags = 0;                                    
                       if (pl_index < cnt)                                     
                               response->flags &= CXL_DCD_EVENT_MORE;          
                                                                               
                       rc = cxl_internal_send_cmd(cxl_mbox, &mbox_cmd);        
                       if (rc)                                                 
                               return rc;                                      
                       cnt -= pl_index;                                        
                       pl_index = 0;                                          
               }                                                               
       }                                                                       
                                                                               
       if (!pl_index)                                                          
               return 0;                                                       

send_zero_accepted:

Quoted 10 lines
       mbox_cmd = (struct cxl_mbox_cmd) {                                      
               .opcode = opcode,                                               
               .size_in = struct_size(response, extent_list,                   
                                      pl_index),                               
               .payload_in = response,                                         
       };                                                                      
                                                                               
       response->flags = 0;                                                    
       return cxl_internal_send_cmd(cxl_mbox, &mbox_cmd);                      
}                

Alternative form for what you have...

	if (cnt != 0) { /* Something to send */
                 xa_for_each(extent_array, index, extent) {
			response->extent_list[pl_index].dpa_start = extent->start_dpa;
			response->extent_list[pl_index].length = extent->length;
			pl_index++;
			response->extent_list_size = cpu_to_le32(pl_index);                 
			if (pl_index != max_extents) /* Space for more? */
				continue;

			/* Send what we have */
			response->flags = 0; 
			if (pl_index < cnt)                                     
				response->flags &= CXL_DCD_EVENT_MORE;
			
			mbox_cmd = (struct cxl_mbox_cmd) {                      
				.opcode = opcode,                               
				.size_in = struct_size(response, extent_list,   
						       pl_index),               
				.payload_in = response,                         
			};

			rc = cxl_internal_send_cmd(cxl_mbox, &mbox_cmd);        
			if (rc)                                                 
				return rc;                                      
			cnt -= pl_index;                                        
			pl_index = 0;
		}
		if (!pl_index)
			return 0;
	}

	/* Catch left overs + send if zero length */
	response->flags = 0;
	mbox_cmd = (struct cxl_mbox_cmd) {                                      
		.opcode = opcode,                                               
		.size_in = struct_size(response, extent_list, pl_index),                               
		.payload_in = response,                                         
	};                                                                      
                                                                                                                                    
	return cxl_internal_send_cmd(cxl_mbox, &mbox_cmd);                      
}
Quoted 1 line
author
#95 lore raw

Re: [PATCH v4 21/28] cxl/extent: Process DCD events and realize region extents

Quoted 14 lines · Jonathan Cameron
Quoted 11 lines · On Thu, 17 Oct 2024 16:15:03 -0500 Ira Weiny <[email protected]> wrote:
Quoted 7 lines · Jonathan Cameron
Quoted 4 lines · On Wed, 9 Oct 2024 14:49:09 -0500 Ira Weiny <[email protected]> wrote:
Quoted 3 lines · Li, Ming4 wrote:
Quoted 2 lines · On 10/8/2024 7:16 AM, [email protected] wrote:

From: Navneet Singh <[email protected]>

[snip]

[snip]

Quoted 5 lines
Quoted 3 lines

So... for clarity among all of us here is the new function. I'm not thrilled with the use of a goto but I think it is ok here.

Easy enough to avoid and I don't think it hurts readability much to do so.

I disagree... See below.

Quoted 42 lines

Your code should work though.

Quoted 30 lines

Ira

static int cxl_send_dc_response(struct cxl_memdev_state *mds, int opcode,
struct xarray *extent_array, int cnt)

{                                                                              
       struct cxl_mailbox *cxl_mbox = &mds->cxlds.cxl_mbox;                    
       struct cxl_mbox_dc_response *p;                                         
       struct cxl_mbox_cmd mbox_cmd;                                           
       struct cxl_extent *extent;                                              
       unsigned long index;                                                    
       u32 pl_index;                                                           
       int rc;                                                                 
                                                                               
       size_t pl_size = struct_size(p, extent_list, cnt);                      
       u32 max_extents = cnt;                                              
                                                                               
       /* May have to use more bit on response. */                             
       if (pl_size > cxl_mbox->payload_size) {                                 
               max_extents = (cxl_mbox->payload_size - sizeof(*p)) /           
                             sizeof(struct updated_extent_list);               
               pl_size = struct_size(p, extent_list, max_extents);
             
       }                                                                       
                                                                               
       struct cxl_mbox_dc_response *response __free(kfree) =                   
                                               kzalloc(pl_size, GFP_KERNEL);   
       if (!response)                                                          
               return -ENOMEM;                                                 
Quoted 8 lines
       pl_index = 0;                                                           
       if (cnt == 0)                                                           
               goto send_zero_accepted;
       xa_for_each(extent_array, index, extent) {                              
               response->extent_list[pl_index].dpa_start = extent->start_dpa;  
               response->extent_list[pl_index].length = extent->length;        
               pl_index++;                                                     
               response->extent_list_size = cpu_to_le32(pl_index);    
Quoted 2 lines

Why set this here - to me makes more sense to set it only once but I can see the logic either way.

I put it here to group it with the changing of pl_index. It is extra work.

Since I'm resending I'll make the quick change.

Quoted 33 lines
Quoted 33 lines
         
  
               if (pl_index == max_extents) {                                  
                       mbox_cmd = (struct cxl_mbox_cmd) {                      
                               .opcode = opcode,                               
                               .size_in = struct_size(response, extent_list,   
                                                      pl_index),               
                               .payload_in = response,                         
                       };                                                      
                                                                               
                       response->flags = 0;                                    
                       if (pl_index < cnt)                                     
                               response->flags &= CXL_DCD_EVENT_MORE;          
                                                                               
                       rc = cxl_internal_send_cmd(cxl_mbox, &mbox_cmd);        
                       if (rc)                                                 
                               return rc;                                      
                       cnt -= pl_index;                                        
                       pl_index = 0;                                          
               }                                                               
       }                                                                       
                                                                               
       if (!pl_index)                                                          
               return 0;                                                       

send_zero_accepted:
mbox_cmd = (struct cxl_mbox_cmd) {
.opcode = opcode,
.size_in = struct_size(response, extent_list,
pl_index),
.payload_in = response,
};

Quoted 6 lines
Quoted 3 lines
       response->flags = 0;                                                    
       return cxl_internal_send_cmd(cxl_mbox, &mbox_cmd);                      
}                

Alternative form for what you have...

Sure but lots of indentation on the common path which I have grown to avoid... :-/

Looking at this fresh... A helper function works best.

static int send_one_response(struct cxl_mailbox *cxl_mbox,
struct cxl_mbox_dc_response *response, int opcode, u32 extent_list_size, u8 flags)

{
        struct cxl_mbox_cmd mbox_cmd = (struct cxl_mbox_cmd) {
                .opcode = opcode,
                .size_in = struct_size(response, extent_list, extent_list_size),
                .payload_in = response,
        };

        response->extent_list_size = cpu_to_le32(extent_list_size);
        response->flags = flags;
        return cxl_internal_send_cmd(cxl_mbox, &mbox_cmd);
}

static int cxl_send_dc_response(struct cxl_memdev_state *mds, int opcode,
struct xarray *extent_array, int cnt)

{
        struct cxl_mailbox *cxl_mbox = &mds->cxlds.cxl_mbox;
        struct cxl_mbox_dc_response *p;
        struct cxl_extent *extent;
        unsigned long index;
        u32 pl_index;

        size_t pl_size = struct_size(p, extent_list, cnt);
        u32 max_extents = cnt;

        /* May have to use more bit on response. */
        if (pl_size > cxl_mbox->payload_size) {
                max_extents = (cxl_mbox->payload_size - sizeof(*p)) /
                              sizeof(struct updated_extent_list);
                pl_size = struct_size(p, extent_list, max_extents);
        }

        struct cxl_mbox_dc_response *response __free(kfree) =
                                                kzalloc(pl_size, GFP_KERNEL);
        if (!response)
                return -ENOMEM;

        if (cnt == 0)
                return send_one_response(cxl_mbox, response, opcode, 0, 0);

        pl_index = 0;
        xa_for_each(extent_array, index, extent) {
                response->extent_list[pl_index].dpa_start = extent->start_dpa;
                response->extent_list[pl_index].length = extent->length;
                pl_index++;

                if (pl_index == max_extents) {
                        u8 flags = 0;
                        int rc;

                        if (pl_index < cnt)
                                flags &= CXL_DCD_EVENT_MORE;
                        rc = send_one_response(cxl_mbox, response, opcode,
                                               pl_index, flags);
                        if (rc) 
                                return rc;
                        cnt -= pl_index;
                        pl_index = 0;
                }
        }

        if (!pl_index) /* nothing more to do */
                return 0;
        return send_one_response(cxl_mbox, response, opcode, pl_index, 0);
}
#96 lore raw

Re: [PATCH v4 21/28] cxl/extent: Process DCD events and realize region extents

<snip>

Quoted 8 lines
Quoted 3 lines

Alternative form for what you have...

Sure but lots of indentation on the common path which I have grown to avoid... :-/

Looking at this fresh... A helper function works best.

Agreed this looks better

Jonathan

#97 lore raw

Re: [PATCH v4 21/28] cxl/extent: Process DCD events and realize region extents

In reply to [email protected]
Quoted 14 lines · On Mon, 07 Oct 2024 18:16:27 -0500 [email protected] wrote:

From: Navneet Singh <[email protected]>

A dynamic capacity device (DCD) sends events to signal the host for changes in the availability of Dynamic Capacity (DC) memory. These events contain extents describing a DPA range and meta data for memory to be added or removed. Events may be sent from the device at any time.

Three types of events can be signaled, Add, Release, and Force Release.

On add, the host may accept or reject the memory being offered. If no region exists, or the extent is invalid, the extent should be rejected. Add extent events may be grouped by a 'more' bit which indicates those extents should be processed as a group.

Quoted 6 lines

On remove, the host can delay the response until the host is safely not using the memory. If no region exists the release can be sent immediately. The host may also release extents (or partial extents) at any time. Thus the 'more' bit grouping of release events is of less value and can be ignored in favor of sending multiple release capacity responses for groups of release events.

True today - I think that would be an error for shared extents though as they need to be released in one go. We can deal with that when it matters.

Mind you patch seems to try to handle more bit anyway, so maybe just remove that discussion from this description?

Quoted 11 lines

Simplify extent tracking with the following restrictions.

1) Flag for removal any extent which overlaps a requested release range.
2) Refuse the offer of extents which overlap already accepted memory ranges.
3) Accept again a range which has already been accepted by the host. Eating duplicates serves three purposes. First, this simplifies the code if the device should get out of sync with the host.

Maybe scream about this a little. AFAIK that happening is a device bug.

Quoted 7 lines

And it should be safe to acknowledge the extent
again. Second, this simplifies the code to process existing extents if the extent list should change while the extent list is being read. Third, duplicates for a given region which are seen during a race between the hardware surfacing an extent and the cxl dax driver scanning for existing extents will be ignored.

This last one is a good justification.

Quoted 14 lines

NOTE: Processing existing extents is done in a later patch.

Management of the region extent devices must be synchronized with potential uses of the memory within the DAX layer. Create region extent devices as children of the cxl_dax_region device such that the DAX region driver can co-drive them and synchronize with the DAX layer. Synchronization and management is handled in a subsequent patch.

Tag support within the DAX layer is not yet supported. To maintain compatibility legacy DAX/region processing only tags with a value of 0 are allowed. This defines existing DAX devices as having a 0 tag which makes the most logical sense as a default.

Quoted 6 lines

Process DCD events and create region devices.

A couple of minor comments from me.

J

Quoted 23 lines
drivers/cxl/core/extent.c new file +0 -0
new file mode 100644
index 000000000000..69a7614ba6a9
drivers/cxl/core/mbox.c +5 -0
index 584d7d282a97..d66beec687a0 100644
@@ -889,6 +889,58 @@ int cxl_enumerate_cmds(struct cxl_memdev_state *mds)
 
 
@@ -1017,6 +1069,223 @@ static int cxl_clear_event_record(struct cxl_memdev_state *mds,
 	return rc;
 }
 
+static int cxl_send_dc_response(struct cxl_memdev_state *mds, int opcode,
+				struct xarray *extent_array, int cnt)
+{
+	struct cxl_mailbox *cxl_mbox = &mds->cxlds.cxl_mbox;
+	struct cxl_mbox_dc_response *p;

drivers/cxl/core/mbox.c static int cxl_clear_event_record(struct cxl_memdev_state *mds, · line 1086

Quoted 10 lines
drivers/cxl/core/mbox.c +10 -0
+	struct cxl_mbox_cmd mbox_cmd;
+	struct cxl_extent *extent;
+	unsigned long index;
+	u32 pl_index;
+	int rc;
+
+	size_t pl_size = struct_size(p, extent_list, cnt);
+	u32 max_extents = cnt;
+
+	/* May have to use more bit on response. */

I thought you argued in the patch description that it didn't matter if you didn't set it?

Quoted 20 lines
drivers/cxl/core/mbox.c +20 -0
+	if (pl_size > cxl_mbox->payload_size) {
+		max_extents = (cxl_mbox->payload_size - sizeof(*p)) /
+			      sizeof(struct updated_extent_list);
+		pl_size = struct_size(p, extent_list, max_extents);
+	}
+
+	struct cxl_mbox_dc_response *response __free(kfree) =
+						kzalloc(pl_size, GFP_KERNEL);
+	if (!response)
+		return -ENOMEM;
+
+	pl_index = 0;
+	xa_for_each(extent_array, index, extent) {
+
+		response->extent_list[pl_index].dpa_start = extent->start_dpa;
+		response->extent_list[pl_index].length = extent->length;
+		pl_index++;
+		response->extent_list_size = cpu_to_le32(pl_index);
+
+		if (pl_index == max_extents) {

drivers/cxl/core/mbox.c static int cxl_clear_event_record(struct cxl_memdev_state *mds, · line 1116

Quoted 10 lines
drivers/cxl/core/mbox.c +10 -0
+			mbox_cmd = (struct cxl_mbox_cmd) {
+				.opcode = opcode,
+				.size_in = struct_size(response, extent_list,
+						       pl_index),
+				.payload_in = response,
+			};
+
+			response->flags = 0;
+			if (pl_index < cnt)
+				response->flags &= CXL_DCD_EVENT_MORE;

Covered in other branch of thread.

Quoted 32 lines
drivers/cxl/core/mbox.c +24 -0
+
+			rc = cxl_internal_send_cmd(cxl_mbox, &mbox_cmd);
+			if (rc)
+				return rc;
+			pl_index = 0;
+		}
+	}
+
+	if (cnt == 0 || pl_index) {
+		mbox_cmd = (struct cxl_mbox_cmd) {
+			.opcode = opcode,
+			.size_in = struct_size(response, extent_list,
+					       pl_index),
+			.payload_in = response,
+		};
+
+		response->flags = 0;
+		rc = cxl_internal_send_cmd(cxl_mbox, &mbox_cmd);
+		if (rc)
+			return rc;
+	}
+
+	return 0;
+}
 
 
 
drivers/cxl/cxl.h +0 -0
index cbaacbe0f36d..b75653e9bc32 100644

drivers/cxl/cxl.h struct cxl_pmem_region { · line 642

Quoted 4 lines
drivers/cxl/cxl.h +1 -0
 
 
 
+/* See CXL 3.0 8.2.9.2.1.5 */

Maybe update to 3.1? Otherwise patch reviewer needs to open two spec versions! In 3.1 it is 8.2.9.2.1.6

Quoted 6 lines
drivers/cxl/cxl.h +6 -0
+enum dc_event {
+	DCD_ADD_CAPACITY,
+	DCD_RELEASE_CAPACITY,
+	DCD_FORCED_CAPACITY_RELEASE,
+	DCD_REGION_CONFIGURATION_UPDATED,
+};
author
#98 lore raw

Re: [PATCH v4 21/28] cxl/extent: Process DCD events and realize region extents

Quoted 24 lines · Jonathan Cameron
Quoted 14 lines · On Mon, 07 Oct 2024 18:16:27 -0500 [email protected] wrote:

From: Navneet Singh <[email protected]>

A dynamic capacity device (DCD) sends events to signal the host for changes in the availability of Dynamic Capacity (DC) memory. These events contain extents describing a DPA range and meta data for memory to be added or removed. Events may be sent from the device at any time.

Three types of events can be signaled, Add, Release, and Force Release.

On add, the host may accept or reject the memory being offered. If no region exists, or the extent is invalid, the extent should be rejected. Add extent events may be grouped by a 'more' bit which indicates those extents should be processed as a group.

Quoted 6 lines

On remove, the host can delay the response until the host is safely not using the memory. If no region exists the release can be sent immediately. The host may also release extents (or partial extents) at any time. Thus the 'more' bit grouping of release events is of less value and can be ignored in favor of sending multiple release capacity responses for groups of release events.

Quoted 7 lines

True today - I think that would be an error for shared extents though as they need to be released in one go. We can deal with that when it matters.

Mind you patch seems to try to handle more bit anyway, so maybe just remove that discussion from this description?

It only handles more bit response on ADD because on RELEASE the count is always 1.

+ if (cxl_send_dc_response(mds, CXL_MBOX_OP_RELEASE_DC, &extent_list, 1))
+ dev_dbg(dev, "Failed to release [range 0x%016llx-0x%016llx]\n",
+ range->start, range->end);

For shared; a flag will need to be added to the extents and additional logic to group these extents for checking use etc.

I agree, we need to handle that later on and get this basic support in. For now I think my comments are correct WRT the sending of release responses.

Quoted 12 lines
Quoted 11 lines

Simplify extent tracking with the following restrictions.

1) Flag for removal any extent which overlaps a requested release range.
2) Refuse the offer of extents which overlap already accepted memory ranges.
3) Accept again a range which has already been accepted by the host. Eating duplicates serves three purposes. First, this simplifies the code if the device should get out of sync with the host.

Quoted 2 lines

Maybe scream about this a little. AFAIK that happening is a device bug.

Agreed but because of the 2nd purpose this is difficult to scream about because this situation can come up in normal operation. Here is the scenario:

1) Device has 2 DCD partitions active, A and B
2) Host crashes
3) Region X is created on A
4) Region Y is created on B
5) Region Y scans for extents
6) Region X surfaces a new extent while Y is scanning
7) Gen number changes due to new extent in X
8) Region Y rescans for existing extents and sees duplicates.

These duplicates need to be ignored without signaling an error.

Quoted 5 lines
Quoted 5 lines

And it should be safe to acknowledge the extent
again. Second, this simplifies the code to process existing extents if the extent list should change while the extent list is being read.

This is the 'normal' case.

Quoted 6 lines
Quoted 4 lines

Third, duplicates for a given region
which are seen during a race between the hardware surfacing an extent and the cxl dax driver scanning for existing extents will be ignored.

This last one is a good justification.

I think the second justification is actually better than this one. Regardless this makes everything ok and should work.

Quoted 15 lines
Quoted 15 lines

NOTE: Processing existing extents is done in a later patch.

Management of the region extent devices must be synchronized with potential uses of the memory within the DAX layer. Create region extent devices as children of the cxl_dax_region device such that the DAX region driver can co-drive them and synchronize with the DAX layer. Synchronization and management is handled in a subsequent patch.

Tag support within the DAX layer is not yet supported. To maintain compatibility legacy DAX/region processing only tags with a value of 0 are allowed. This defines existing DAX devices as having a 0 tag which makes the most logical sense as a default.

Quoted 7 lines
Quoted 6 lines

Process DCD events and create region devices.

A couple of minor comments from me.

I do appreciate the review.

[snip]

Quoted 11 lines
Quoted 11 lines
drivers/cxl/core/mbox.c +10 -0
+static int cxl_send_dc_response(struct cxl_memdev_state *mds, int opcode,
+				struct xarray *extent_array, int cnt)
+{
+	struct cxl_mailbox *cxl_mbox = &mds->cxlds.cxl_mbox;
+	struct cxl_mbox_dc_response *p;
+	struct cxl_mbox_cmd mbox_cmd;
+	struct cxl_extent *extent;
+	unsigned long index;
+	u32 pl_index;
+	int rc;
Quoted 8 lines

drivers/cxl/core/mbox.c static int cxl_clear_event_record(struct cxl_memdev_state *mds, · line 1086

Quoted 5 lines
drivers/cxl/core/mbox.c +5 -0
+
+	size_t pl_size = struct_size(p, extent_list, cnt);
+	u32 max_extents = cnt;
+
+	/* May have to use more bit on response. */

I thought you argued in the patch description that it didn't matter if you didn't set it?

Only on RELEASE responses. ADD responses might need it depending on the payload size and number of extents being added.

Sorry that was not clear.

Quoted 24 lines
Quoted 24 lines
drivers/cxl/core/mbox.c +23 -0
+	if (pl_size > cxl_mbox->payload_size) {
+		max_extents = (cxl_mbox->payload_size - sizeof(*p)) /
+			      sizeof(struct updated_extent_list);
+		pl_size = struct_size(p, extent_list, max_extents);
+	}
+
+	struct cxl_mbox_dc_response *response __free(kfree) =
+						kzalloc(pl_size, GFP_KERNEL);
+	if (!response)
+		return -ENOMEM;
+
+	pl_index = 0;
+	xa_for_each(extent_array, index, extent) {
+
+		response->extent_list[pl_index].dpa_start = extent->start_dpa;
+		response->extent_list[pl_index].length = extent->length;
+		pl_index++;
+		response->extent_list_size = cpu_to_le32(pl_index);
+
+		if (pl_index == max_extents) {
+			mbox_cmd = (struct cxl_mbox_cmd) {
+				.opcode = opcode,
+				.size_in = struct_size(response, extent_list,
Quoted 8 lines

drivers/cxl/core/mbox.c static int cxl_clear_event_record(struct cxl_memdev_state *mds, · line 1116

Quoted 7 lines
drivers/cxl/core/mbox.c +7 -0
+						       pl_index),
+				.payload_in = response,
+			};
+
+			response->flags = 0;
+			if (pl_index < cnt)
+				response->flags &= CXL_DCD_EVENT_MORE;

Covered in other branch of thread.

Yep.

[snip]

Quoted 6 lines
Quoted 3 lines

+/* See CXL 3.0 8.2.9.2.1.5 */

Maybe update to 3.1? Otherwise patch reviewer needs to open two spec versions! In 3.1 it is 8.2.9.2.1.6

Yep missed this one. Thanks,
Ira

#99 lore raw

Re: [PATCH v4 21/28] cxl/extent: Process DCD events and realize region extents

Quoted 75 lines · On Thu, 17 Oct 2024 16:39:57 -0500 Ira Weiny <[email protected]> wrote:
Quoted 24 lines · Jonathan Cameron
Quoted 14 lines · On Mon, 07 Oct 2024 18:16:27 -0500 [email protected] wrote:

From: Navneet Singh <[email protected]>

A dynamic capacity device (DCD) sends events to signal the host for changes in the availability of Dynamic Capacity (DC) memory. These events contain extents describing a DPA range and meta data for memory to be added or removed. Events may be sent from the device at any time.

Three types of events can be signaled, Add, Release, and Force Release.

On add, the host may accept or reject the memory being offered. If no region exists, or the extent is invalid, the extent should be rejected. Add extent events may be grouped by a 'more' bit which indicates those extents should be processed as a group.

Quoted 6 lines

On remove, the host can delay the response until the host is safely not using the memory. If no region exists the release can be sent immediately. The host may also release extents (or partial extents) at any time. Thus the 'more' bit grouping of release events is of less value and can be ignored in favor of sending multiple release capacity responses for groups of release events.

Quoted 7 lines

True today - I think that would be an error for shared extents though as they need to be released in one go. We can deal with that when it matters.

Mind you patch seems to try to handle more bit anyway, so maybe just remove that discussion from this description?

It only handles more bit response on ADD because on RELEASE the count is always 1.

(quoted diff) +3 -0
+       if (cxl_send_dc_response(mds, CXL_MBOX_OP_RELEASE_DC, &extent_list, 1)) 
+               dev_dbg(dev, "Failed to release [range 0x%016llx-0x%016llx]\n", 
+                       range->start, range->end);                              

For shared; a flag will need to be added to the extents and additional logic to group these extents for checking use etc.

I agree, we need to handle that later on and get this basic support in. For now I think my comments are correct WRT the sending of release responses.

Quoted 12 lines
Quoted 11 lines

Simplify extent tracking with the following restrictions.

1) Flag for removal any extent which overlaps a requested release range.
2) Refuse the offer of extents which overlap already accepted memory ranges.
3) Accept again a range which has already been accepted by the host. Eating duplicates serves three purposes. First, this simplifies the code if the device should get out of sync with the host.

Quoted 2 lines

Maybe scream about this a little. AFAIK that happening is a device bug.

Agreed but because of the 2nd purpose this is difficult to scream about because this situation can come up in normal operation. Here is the scenario:

1) Device has 2 DCD partitions active, A and B
2) Host crashes
3) Region X is created on A
4) Region Y is created on B
5) Region Y scans for extents
6) Region X surfaces a new extent while Y is scanning
7) Gen number changes due to new extent in X
8) Region Y rescans for existing extents and sees duplicates.

Quoted 1 line

These duplicates need to be ignored without signaling an error.

Hmm. If we can know that path is the trigger (should be able to as it's a scan after a gen number change), can we just muffle the screams on that path? (Halloween is close, the analogies will get ever worse :)

Jonathan

author
#100 lore raw

Re: [PATCH v4 21/28] cxl/extent: Process DCD events and realize region extents

Quoted 7 lines · Jonathan Cameron
Quoted 4 lines · On Thu, 17 Oct 2024 16:39:57 -0500 Ira Weiny <[email protected]> wrote:
Quoted 3 lines · Jonathan Cameron

On Mon, 07 Oct 2024 18:16:27 -0500
[email protected] wrote:

[snip]

Quoted 26 lines
Quoted 26 lines
Quoted 11 lines
Quoted 10 lines

Simplify extent tracking with the following restrictions.

1) Flag for removal any extent which overlaps a requested release range.
2) Refuse the offer of extents which overlap already accepted memory ranges.
3) Accept again a range which has already been accepted by the host. Eating duplicates serves three purposes. First, this simplifies the code if the device should get out of sync with the host.

Quoted 2 lines

Maybe scream about this a little. AFAIK that happening is a device bug.

Agreed but because of the 2nd purpose this is difficult to scream about because this situation can come up in normal operation. Here is the scenario:

1) Device has 2 DCD partitions active, A and B
2) Host crashes
3) Region X is created on A
4) Region Y is created on B
5) Region Y scans for extents
6) Region X surfaces a new extent while Y is scanning
7) Gen number changes due to new extent in X
8) Region Y rescans for existing extents and sees duplicates.

Quoted 5 lines
Quoted 1 line

These duplicates need to be ignored without signaling an error.

Hmm. If we can know that path is the trigger (should be able to as it's a scan after a gen number change), can we just muffle the screams on that path? (Halloween is close, the analogies will get ever worse :)

Ok yea since this would be a device error we should do something here. But the code is going to be somewhat convoluted to print an error whenever this happens.

What if we make this a warning and change the rescan debug message to a warning as well? This would allow enough bread crumbs to determine if a device is failing without a lot of extra code to alter print messages on the fly?

Ira

#101 lore raw

Re: [PATCH v4 21/28] cxl/extent: Process DCD events and realize region extents

Quoted 43 lines · On Mon, 21 Oct 2024 13:45:57 -0500 Ira Weiny <[email protected]> wrote:
Quoted 7 lines · Jonathan Cameron
Quoted 4 lines · On Thu, 17 Oct 2024 16:39:57 -0500 Ira Weiny <[email protected]> wrote:
Quoted 3 lines · Jonathan Cameron

On Mon, 07 Oct 2024 18:16:27 -0500
[email protected] wrote:

[snip]

Quoted 26 lines
Quoted 26 lines
Quoted 11 lines
Quoted 10 lines

Simplify extent tracking with the following restrictions.

1) Flag for removal any extent which overlaps a requested release range.
2) Refuse the offer of extents which overlap already accepted memory ranges.
3) Accept again a range which has already been accepted by the host. Eating duplicates serves three purposes. First, this simplifies the code if the device should get out of sync with the host.

Quoted 2 lines

Maybe scream about this a little. AFAIK that happening is a device bug.

Agreed but because of the 2nd purpose this is difficult to scream about because this situation can come up in normal operation. Here is the scenario:

1) Device has 2 DCD partitions active, A and B
2) Host crashes
3) Region X is created on A
4) Region Y is created on B
5) Region Y scans for extents
6) Region X surfaces a new extent while Y is scanning
7) Gen number changes due to new extent in X
8) Region Y rescans for existing extents and sees duplicates.

Quoted 5 lines
Quoted 1 line

These duplicates need to be ignored without signaling an error.

Hmm. If we can know that path is the trigger (should be able to as it's a scan after a gen number change), can we just muffle the screams on that path? (Halloween is close, the analogies will get ever worse :)

Quoted 7 lines

Ok yea since this would be a device error we should do something here. But the code is going to be somewhat convoluted to print an error whenever this happens.

What if we make this a warning and change the rescan debug message to a warning as well? This would allow enough bread crumbs to determine if a device is failing without a lot of extra code to alter print messages on the fly?

Sounds ok to me.

Jonathan

Quoted 3 lines

Ira

#102 lore raw

PATCH v4 22/28 cxl/region/extent: Expose region extent information in sysfs

In reply to Ira Weiny

From: Navneet Singh <[email protected]>

Extent information can be helpful to the user to coordinate memory usage with the external orchestrator and FM.

Expose the details of region extents by creating the following sysfs entries.

/sys/bus/cxl/devices/dax_regionX/extentX.Y /sys/bus/cxl/devices/dax_regionX/extentX.Y/offset /sys/bus/cxl/devices/dax_regionX/extentX.Y/length /sys/bus/cxl/devices/dax_regionX/extentX.Y/tag

---
Changes:
[djiang: Split sysfs docs up]
[iweiny: Adjust sysfs docs dates]

---
 Documentation/ABI/testing/sysfs-bus-cxl | 32 ++++++++++++++++++
 drivers/cxl/core/extent.c               | 58 +++++++++++++++++++++++++++++++++
 2 files changed, 90 insertions(+)
Documentation/ABI/testing/sysfs-bus-cxl +32 -0
index b63ab622515f..64918180a3c9 100644
@@ -632,3 +632,35 @@ Description:
 		See Documentation/ABI/stable/sysfs-devices-node. access0 provides
 		the number to the closest initiator and access1 provides the
 		number to the closest CPU.
+
+What:		/sys/bus/cxl/devices/dax_regionX/extentX.Y/offset
+Date:		December, 2024
+KernelVersion:	v6.13
+Contact:	[email protected]
+Description:
+		(RO) [For Dynamic Capacity regions only] Users can use the
+		extent information to create DAX devices on specific extents.
+		This is done by creating and destroying DAX devices in specific
+		sequences and looking at the mappings created.  Extent offset
+		within the region.
+
+What:		/sys/bus/cxl/devices/dax_regionX/extentX.Y/length
+Date:		December, 2024
+KernelVersion:	v6.13
+Contact:	[email protected]
+Description:
+		(RO) [For Dynamic Capacity regions only] Users can use the
+		extent information to create DAX devices on specific extents.
+		This is done by creating and destroying DAX devices in specific
+		sequences and looking at the mappings created.  Extent length
+		within the region.
+
+What:		/sys/bus/cxl/devices/dax_regionX/extentX.Y/tag
+Date:		December, 2024
+KernelVersion:	v6.13
+Contact:	[email protected]
+Description:
+		(RO) [For Dynamic Capacity regions only] Users can use the
+		extent information to create DAX devices on specific extents.
+		This is done by creating and destroying DAX devices in specific
+		sequences and looking at the mappings created.  Extent tag.
drivers/cxl/core/extent.c +58 -0

70 lines not on this page · raw message

Signature · 1 line
2.46.0
Reviewed-by on v4 22/28
#103 lore raw

Re: [PATCH v4 22/28] cxl/region/extent: Expose region extent information in sysfs

Quoted 8 lines · On Mon, 07 Oct 2024 18:16:28 -0500 [email protected] wrote:

From: Navneet Singh <[email protected]>

Extent information can be helpful to the user to coordinate memory usage with the external orchestrator and FM.

Expose the details of region extents by creating the following sysfs entries.

Quoted 9 lines

/sys/bus/cxl/devices/dax_regionX/extentX.Y /sys/bus/cxl/devices/dax_regionX/extentX.Y/offset /sys/bus/cxl/devices/dax_regionX/extentX.Y/length /sys/bus/cxl/devices/dax_regionX/extentX.Y/tag

Trivial comments inline.

Quoted 17 lines

---
Changes:
[djiang: Split sysfs docs up]
[iweiny: Adjust sysfs docs dates]

---
 Documentation/ABI/testing/sysfs-bus-cxl | 32 ++++++++++++++++++
 drivers/cxl/core/extent.c               | 58 +++++++++++++++++++++++++++++++++
 2 files changed, 90 insertions(+)
Documentation/ABI/testing/sysfs-bus-cxl +0 -0
index b63ab622515f..64918180a3c9 100644
@@ -632,3 +632,35 @@ Description:
 		See Documentation/ABI/stable/sysfs-devices-node. access0 provides
 		the number to the closest initiator and access1 provides the
 		number to the closest CPU.

Documentation/ABI/testing/sysfs-bus-cxl Description: · line 644

Quoted 10 lines
Documentation/ABI/testing/sysfs-bus-cxl +10 -0
+
+What:		/sys/bus/cxl/devices/dax_regionX/extentX.Y/offset
+Date:		December, 2024
+KernelVersion:	v6.13
+Contact:	[email protected]
+Description:
+		(RO) [For Dynamic Capacity regions only] Users can use the
+		extent information to create DAX devices on specific extents.
+		This is done by creating and destroying DAX devices in specific
+		sequences and looking at the mappings created. 

Similar to earlier patch, maybe put this doc for the directory, then have much less duplication?

Quoted 13 lines

Extent offset

Documentation/ABI/testing/sysfs-bus-cxl +12 -0
+		within the region.
+
+What:		/sys/bus/cxl/devices/dax_regionX/extentX.Y/length
+Date:		December, 2024
+KernelVersion:	v6.13
+Contact:	[email protected]
+Description:
+		(RO) [For Dynamic Capacity regions only] Users can use the
+		extent information to create DAX devices on specific extents.
+		This is done by creating and destroying DAX devices in specific
+		sequences and looking at the mappings created.  Extent length
+		within the region.

Documentation/ABI/testing/sysfs-bus-cxl Description: · line 666

Quoted 10 lines
Documentation/ABI/testing/sysfs-bus-cxl +10 -0
+
+What:		/sys/bus/cxl/devices/dax_regionX/extentX.Y/tag
+Date:		December, 2024
+KernelVersion:	v6.13
+Contact:	[email protected]
+Description:
+		(RO) [For Dynamic Capacity regions only] Users can use the
+		extent information to create DAX devices on specific extents.
+		This is done by creating and destroying DAX devices in specific
+		sequences and looking at the mappings created.  Extent tag.

Maybe say we are treating it as a UUID?

drivers/cxl/core/extent.c line 11

Quoted 11 lines
drivers/cxl/core/extent.c +5 -0
index 69a7614ba6a9..a1eb6e8e4f1a 100644
@@ -6,6 +6,63 @@
 
+static struct attribute *region_extent_attrs[] = {
+	&dev_attr_offset.attr,
+	&dev_attr_length.attr,
+	&dev_attr_tag.attr,
+	NULL,

No need for trailing comma (one of my 'favourite' review comments :)

Quoted 1 line
drivers/cxl/core/extent.c +1 -0
+};
author
#104 lore raw

Re: [PATCH v4 22/28] cxl/region/extent: Expose region extent information in sysfs

Quoted 16 lines · Jonathan Cameron
Quoted 13 lines · On Mon, 07 Oct 2024 18:16:28 -0500 [email protected] wrote:

From: Navneet Singh <[email protected]>

Extent information can be helpful to the user to coordinate memory usage with the external orchestrator and FM.

Expose the details of region extents by creating the following sysfs entries.

/sys/bus/cxl/devices/dax_regionX/extentX.Y /sys/bus/cxl/devices/dax_regionX/extentX.Y/offset /sys/bus/cxl/devices/dax_regionX/extentX.Y/length /sys/bus/cxl/devices/dax_regionX/extentX.Y/tag

Quoted 7 lines
Quoted 4 lines

Trivial comments inline.

Thanks!

Quoted 23 lines
Quoted 23 lines

---
Changes:
[djiang: Split sysfs docs up]
[iweiny: Adjust sysfs docs dates]

---
 Documentation/ABI/testing/sysfs-bus-cxl | 32 ++++++++++++++++++
 drivers/cxl/core/extent.c               | 58 +++++++++++++++++++++++++++++++++
 2 files changed, 90 insertions(+)
Documentation/ABI/testing/sysfs-bus-cxl +5 -0
index b63ab622515f..64918180a3c9 100644
@@ -632,3 +632,35 @@ Description:
 		See Documentation/ABI/stable/sysfs-devices-node. access0 provides
 		the number to the closest initiator and access1 provides the
 		number to the closest CPU.
+
+What:		/sys/bus/cxl/devices/dax_regionX/extentX.Y/offset
+Date:		December, 2024
+KernelVersion:	v6.13
+Contact:	[email protected]
Quoted 9 lines

Documentation/ABI/testing/sysfs-bus-cxl Description:

Quoted 5 lines
Documentation/ABI/testing/sysfs-bus-cxl +5 -0
+Description:
+		(RO) [For Dynamic Capacity regions only] Users can use the
+		extent information to create DAX devices on specific extents.
+		This is done by creating and destroying DAX devices in specific
+		sequences and looking at the mappings created. 

Similar to earlier patch, maybe put this doc for the directory, then have much less duplication?

But none of the other directories are done this way so I'm inclined to keep it.

Quoted 18 lines
Quoted 18 lines

Extent offset

Documentation/ABI/testing/sysfs-bus-cxl +16 -0
+		within the region.
+
+What:		/sys/bus/cxl/devices/dax_regionX/extentX.Y/length
+Date:		December, 2024
+KernelVersion:	v6.13
+Contact:	[email protected]
+Description:
+		(RO) [For Dynamic Capacity regions only] Users can use the
+		extent information to create DAX devices on specific extents.
+		This is done by creating and destroying DAX devices in specific
+		sequences and looking at the mappings created.  Extent length
+		within the region.
+
+What:		/sys/bus/cxl/devices/dax_regionX/extentX.Y/tag
+Date:		December, 2024
+KernelVersion:	v6.13
Quoted 8 lines

Documentation/ABI/testing/sysfs-bus-cxl Description: · line 666

Quoted 6 lines
Documentation/ABI/testing/sysfs-bus-cxl +6 -0
+Contact:	[email protected]
+Description:
+		(RO) [For Dynamic Capacity regions only] Users can use the
+		extent information to create DAX devices on specific extents.
+		This is done by creating and destroying DAX devices in specific
+		sequences and looking at the mappings created.  Extent tag.

Maybe say we are treating it as a UUID?

ok... How about?

<quote>
... looking at the mappings created. UUID extent tag. </quote>

Quoted 6 lines
Quoted 5 lines
drivers/cxl/core/extent.c +0 -0
index 69a7614ba6a9..a1eb6e8e4f1a 100644
@@ -6,6 +6,63 @@
Quoted 6 lines

drivers/cxl/core/extent.c line 41

Quoted 5 lines
drivers/cxl/core/extent.c +5 -0
+static struct attribute *region_extent_attrs[] = {
+	&dev_attr_offset.attr,
+	&dev_attr_length.attr,
+	&dev_attr_tag.attr,
+	NULL,

No need for trailing comma (one of my 'favourite' review comments :)

I'm noticing... :-D

Ira

#105 lore raw

Re: [PATCH v4 22/28] cxl/region/extent: Expose region extent information in sysfs

Quoted 60 lines · On Fri, 18 Oct 2024 13:26:14 -0500 Ira Weiny <[email protected]> wrote:
Quoted 16 lines · Jonathan Cameron
Quoted 13 lines · On Mon, 07 Oct 2024 18:16:28 -0500 [email protected] wrote:

From: Navneet Singh <[email protected]>

Extent information can be helpful to the user to coordinate memory usage with the external orchestrator and FM.

Expose the details of region extents by creating the following sysfs entries.

/sys/bus/cxl/devices/dax_regionX/extentX.Y /sys/bus/cxl/devices/dax_regionX/extentX.Y/offset /sys/bus/cxl/devices/dax_regionX/extentX.Y/length /sys/bus/cxl/devices/dax_regionX/extentX.Y/tag

Quoted 7 lines
Quoted 4 lines

Trivial comments inline.

Thanks!

Quoted 23 lines
Quoted 23 lines

---
Changes:
[djiang: Split sysfs docs up]
[iweiny: Adjust sysfs docs dates]

---
 Documentation/ABI/testing/sysfs-bus-cxl | 32 ++++++++++++++++++
 drivers/cxl/core/extent.c               | 58 +++++++++++++++++++++++++++++++++
 2 files changed, 90 insertions(+)
Documentation/ABI/testing/sysfs-bus-cxl +5 -0
index b63ab622515f..64918180a3c9 100644
@@ -632,3 +632,35 @@ Description:
 		See Documentation/ABI/stable/sysfs-devices-node. access0 provides
 		the number to the closest initiator and access1 provides the
 		number to the closest CPU.
+
+What:		/sys/bus/cxl/devices/dax_regionX/extentX.Y/offset
+Date:		December, 2024
+KernelVersion:	v6.13
+Contact:	[email protected]
Quoted 9 lines

Documentation/ABI/testing/sysfs-bus-cxl Description:

Quoted 5 lines
Documentation/ABI/testing/sysfs-bus-cxl +5 -0
+Description:
+		(RO) [For Dynamic Capacity regions only] Users can use the
+		extent information to create DAX devices on specific extents.
+		This is done by creating and destroying DAX devices in specific
+		sequences and looking at the mappings created.   

Similar to earlier patch, maybe put this doc for the directory, then have much less duplication?

Quoted 1 line

But none of the other directories are done this way so I'm inclined to keep it.

Fair enough. Maybe a topic for a future cleanup to reduce duplication.

Quoted 28 lines
Quoted 19 lines
Quoted 19 lines

Extent offset

Documentation/ABI/testing/sysfs-bus-cxl +16 -0
+		within the region.
+
+What:		/sys/bus/cxl/devices/dax_regionX/extentX.Y/length
+Date:		December, 2024
+KernelVersion:	v6.13
+Contact:	[email protected]
+Description:
+		(RO) [For Dynamic Capacity regions only] Users can use the
+		extent information to create DAX devices on specific extents.
+		This is done by creating and destroying DAX devices in specific
+		sequences and looking at the mappings created.  Extent length
+		within the region.
+
+What:		/sys/bus/cxl/devices/dax_regionX/extentX.Y/tag
+Date:		December, 2024
+KernelVersion:	v6.13
Quoted 8 lines

Documentation/ABI/testing/sysfs-bus-cxl Description: · line 666

Quoted 6 lines
Documentation/ABI/testing/sysfs-bus-cxl +6 -0
+Contact:	[email protected]
+Description:
+		(RO) [For Dynamic Capacity regions only] Users can use the
+		extent information to create DAX devices on specific extents.
+		This is done by creating and destroying DAX devices in specific
+		sequences and looking at the mappings created.  Extent tag.  

Maybe say we are treating it as a UUID?

Quoted 4 lines

ok... How about?

<quote>
... looking at the mappings created. UUID extent tag.

That's fine.

Quoted 8 lines

</quote>

Quoted 6 lines
Quoted 5 lines
drivers/cxl/core/extent.c +0 -0
index 69a7614ba6a9..a1eb6e8e4f1a 100644
@@ -6,6 +6,63 @@  
Quoted 8 lines
Quoted 6 lines

drivers/cxl/core/extent.c line 41

Quoted 5 lines
drivers/cxl/core/extent.c +5 -0
+static struct attribute *region_extent_attrs[] = {
+	&dev_attr_offset.attr,
+	&dev_attr_length.attr,
+	&dev_attr_tag.attr,
+	NULL,  

No need for trailing comma (one of my 'favourite' review comments :)

I'm noticing... :-D

Maybe I'll one day add to checkpatch. If it weren't written in perl I'd do it now ;)

Jonathan

Quoted 3 lines

Ira

Reviewed-by Tested-by on v4 22/28
#106 lore raw

Re: [PATCH v4 22/28] cxl/region/extent: Expose region extent information in sysfs

In reply to [email protected]
Quoted 8 lines · On Mon, Oct 07, 2024 at 06:16:28PM -0500, [email protected] wrote:

From: Navneet Singh <[email protected]>

Extent information can be helpful to the user to coordinate memory usage with the external orchestrator and FM.

Expose the details of region extents by creating the following sysfs entries.

Quoted 9 lines

/sys/bus/cxl/devices/dax_regionX/extentX.Y /sys/bus/cxl/devices/dax_regionX/extentX.Y/offset /sys/bus/cxl/devices/dax_regionX/extentX.Y/length /sys/bus/cxl/devices/dax_regionX/extentX.Y/tag

Quoted 129 lines

---
Changes:
[djiang: Split sysfs docs up]
[iweiny: Adjust sysfs docs dates]

---
 Documentation/ABI/testing/sysfs-bus-cxl | 32 ++++++++++++++++++
 drivers/cxl/core/extent.c               | 58 +++++++++++++++++++++++++++++++++
 2 files changed, 90 insertions(+)
Documentation/ABI/testing/sysfs-bus-cxl +32 -0
index b63ab622515f..64918180a3c9 100644
@@ -632,3 +632,35 @@ Description:
 		See Documentation/ABI/stable/sysfs-devices-node. access0 provides
 		the number to the closest initiator and access1 provides the
 		number to the closest CPU.
+
+What:		/sys/bus/cxl/devices/dax_regionX/extentX.Y/offset
+Date:		December, 2024
+KernelVersion:	v6.13
+Contact:	[email protected]
+Description:
+		(RO) [For Dynamic Capacity regions only] Users can use the
+		extent information to create DAX devices on specific extents.
+		This is done by creating and destroying DAX devices in specific
+		sequences and looking at the mappings created.  Extent offset
+		within the region.
+
+What:		/sys/bus/cxl/devices/dax_regionX/extentX.Y/length
+Date:		December, 2024
+KernelVersion:	v6.13
+Contact:	[email protected]
+Description:
+		(RO) [For Dynamic Capacity regions only] Users can use the
+		extent information to create DAX devices on specific extents.
+		This is done by creating and destroying DAX devices in specific
+		sequences and looking at the mappings created.  Extent length
+		within the region.
+
+What:		/sys/bus/cxl/devices/dax_regionX/extentX.Y/tag
+Date:		December, 2024
+KernelVersion:	v6.13
+Contact:	[email protected]
+Description:
+		(RO) [For Dynamic Capacity regions only] Users can use the
+		extent information to create DAX devices on specific extents.
+		This is done by creating and destroying DAX devices in specific
+		sequences and looking at the mappings created.  Extent tag.
drivers/cxl/core/extent.c +58 -0
index 69a7614ba6a9..a1eb6e8e4f1a 100644
@@ -6,6 +6,63 @@
 
 #include "core.h"
 
+static ssize_t offset_show(struct device *dev, struct device_attribute *attr,
+			   char *buf)
+{
+	struct region_extent *region_extent = to_region_extent(dev);
+
+	return sysfs_emit(buf, "%#llx\n", region_extent->hpa_range.start);
+}
+static DEVICE_ATTR_RO(offset);
+
+static ssize_t length_show(struct device *dev, struct device_attribute *attr,
+			   char *buf)
+{
+	struct region_extent *region_extent = to_region_extent(dev);
+	u64 length = range_len(&region_extent->hpa_range);
+
+	return sysfs_emit(buf, "%#llx\n", length);
+}
+static DEVICE_ATTR_RO(length);
+
+static ssize_t tag_show(struct device *dev, struct device_attribute *attr,
+			char *buf)
+{
+	struct region_extent *region_extent = to_region_extent(dev);
+
+	return sysfs_emit(buf, "%pUb\n", &region_extent->tag);
+}
+static DEVICE_ATTR_RO(tag);
+
+static struct attribute *region_extent_attrs[] = {
+	&dev_attr_offset.attr,
+	&dev_attr_length.attr,
+	&dev_attr_tag.attr,
+	NULL,
+};
+
+static uuid_t empty_tag = { 0 };
+
+static umode_t region_extent_visible(struct kobject *kobj,
+				     struct attribute *a, int n)
+{
+	struct device *dev = kobj_to_dev(kobj);
+	struct region_extent *region_extent = to_region_extent(dev);
+
+	if (a == &dev_attr_tag.attr &&
+	    uuid_equal(&region_extent->tag, &empty_tag))
+		return 0;
+
+	return a->mode;
+}
+
+static const struct attribute_group region_extent_attribute_group = {
+	.attrs = region_extent_attrs,
+	.is_visible = region_extent_visible,
+};
+
+__ATTRIBUTE_GROUPS(region_extent_attribute);
+
 static void cxled_release_extent(struct cxl_endpoint_decoder *cxled,
 				 struct cxled_extent *ed_extent)
 {
@@ -44,6 +101,7 @@ static void region_extent_release(struct device *dev)
 static const struct device_type region_extent_type = {
 	.name = "extent",
 	.release = region_extent_release,
+	.groups = region_extent_attribute_groups,
 };
 
 bool is_region_extent(struct device *dev)
Signature · 2 lines
-- 
2.46.0
Signature · 1 line
Fan Ni
author
#107 lore raw

PATCH v4 23/28 dax/bus: Factor out dev dax resize logic

In reply to Ira Weiny

Dynamic Capacity regions must limit dev dax resources to those areas which have extents backing real memory. Such DAX regions are dubbed 'sparse' regions. In order to manage where memory is available four alternatives were considered:

1) Create a single region resource child on region creation which reserves the entire region. Then as extents are added punch holes in this reservation. This requires new resource manipulation to punch the holes and still requires an additional iteration over the extent areas which may already have existing dev dax resources used.

2) Maintain an ordered xarray of extents which can be queried while processing the resize logic. The issue is that existing region->res children may artificially limit the allocation size sent to alloc_dev_dax_range(). IE the resource children can't be directly used in the resize logic to find where space in the region is. This also poses a problem of managing the available size in 2 places.

3) Maintain a separate resource tree with extents. This option is the same as 2) but with the different data structure. Most ideally there should be a unified representation of the resource tree not two places to look for space.

4) Create region resource children for each extent. Manage the dax dev resize logic in the same way as before but use a region child (extent) resource as the parents to find space within each extent.

Option 4 can leverage the existing resize algorithm to find space within the extents. It manages the available space in a singular resource tree which is less complicated for finding space.

In preparation for this change, factor out the dev_dax_resize logic. For static regions use dax_region->res as the parent to find space for the dax ranges. Future patches will use the same algorithm with individual extent resources as the parent.

---
Changes:
[Jonathan: Fix handling of alloc]

---
 drivers/dax/bus.c | 129 +++++++++++++++++++++++++++++++++---------------------
 1 file changed, 79 insertions(+), 50 deletions(-)
drivers/dax/bus.c +79 -50
index d8cb5195a227..f0e3f8c787df 100644
@@ -844,11 +844,9 @@ static int devm_register_dax_mapping(struct dev_dax *dev_dax, int range_id)
 	return 0;
 }
 
-static int alloc_dev_dax_range(struct dev_dax *dev_dax, u64 start,
-		resource_size_t size)
+static int alloc_dev_dax_range(struct resource *parent, struct dev_dax *dev_dax,
+			       u64 start, resource_size_t size)
 {
-	struct dax_region *dax_region = dev_dax->region;
-	struct resource *res = &dax_region->res;
 	struct device *dev = &dev_dax->dev;
 	struct dev_dax_range *ranges;
 	unsigned long pgoff = 0;
@@ -866,14 +864,14 @@ static int alloc_dev_dax_range(struct dev_dax *dev_dax, u64 start,
 		return 0;
 	}
 
-	alloc = __request_region(res, start, size, dev_name(dev), 0);
+	alloc = __request_region(parent, start, size, dev_name(dev), 0);
 	if (!alloc)
 		return -ENOMEM;
 
 	ranges = krealloc(dev_dax->ranges, sizeof(*ranges)
 			* (dev_dax->nr_range + 1), GFP_KERNEL);
 	if (!ranges) {
-		__release_region(res, alloc->start, resource_size(alloc));
+		__release_region(parent, alloc->start, resource_size(alloc));
 		return -ENOMEM;
 	}
 
@@ -1026,50 +1024,45 @@ static bool adjust_ok(struct dev_dax *dev_dax, struct resource *res)
 	return true;
 }
 
-static ssize_t dev_dax_resize(struct dax_region *dax_region,
-		struct dev_dax *dev_dax, resource_size_t size)
+/**
+ * dev_dax_resize_static - Expand the device into the unused portion of the
+ * region. This may involve adjusting the end of an existing resource, or
+ * allocating a new resource.
+ *
+ * @parent: parent resource to allocate this range in
+ * @dev_dax: DAX device to be expanded
+ * @to_alloc: amount of space to alloc; must be <= space available in @parent
+ *
+ * Return the amount of space allocated or -ERRNO on failure
+ */
+static ssize_t dev_dax_resize_static(struct resource *parent,
+				     struct dev_dax *dev_dax,
+				     resource_size_t to_alloc)
 {
-	resource_size_t avail = dax_region_avail_size(dax_region), to_alloc;
-	resource_size_t dev_size = dev_dax_size(dev_dax);
-	struct resource *region_res = &dax_region->res;
-	struct device *dev = &dev_dax->dev;
 	struct resource *res, *first;
-	resource_size_t alloc = 0;
 	int rc;
 
-	if (dev->driver)
-		return -EBUSY;
-	if (size == dev_size)
-		return 0;
-	if (size > dev_size && size - dev_size > avail)
-		return -ENOSPC;
-	if (size < dev_size)
-		return dev_dax_shrink(dev_dax, size);
-
-	to_alloc = size - dev_size;
-	if (dev_WARN_ONCE(dev, !alloc_is_aligned(dev_dax, to_alloc),
-			"resize of %pa misaligned\n", &to_alloc))
-		return -ENXIO;
-
-	/*
-	 * Expand the device into the unused portion of the region. This
-	 * may involve adjusting the end of an existing resource, or
-	 * allocating a new resource.
-	 */
-retry:
-	first = region_res->child;
-	if (!first)
-		return alloc_dev_dax_range(dev_dax, dax_region->res.start, to_alloc);
+	first = parent->child;
+	if (!first) {
+		rc = alloc_dev_dax_range(parent, dev_dax,
+					   parent->start, to_alloc);
+		if (rc)
+			return rc;
+		return to_alloc;
+	}
 
-	rc = -ENOSPC;
 	for (res = first; res; res = res->sibling) {
 		struct resource *next = res->sibling;
+		resource_size_t alloc;
 
 		/* space at the beginning of the region */
-		if (res == first && res->start > dax_region->res.start) {
-			alloc = min(res->start - dax_region->res.start, to_alloc);
-			rc = alloc_dev_dax_range(dev_dax, dax_region->res.start, alloc);
-			break;
+		if (res == first && res->start > parent->start) {
+			alloc = min(res->start - parent->start, to_alloc);
+			rc = alloc_dev_dax_range(parent, dev_dax,
+						 parent->start, alloc);
+			if (rc)
+				return rc;
+			return alloc;
 		}
 
 		alloc = 0;
@@ -1078,21 +1071,55 @@ static ssize_t dev_dax_resize(struct dax_region *dax_region,
 			alloc = min(next->start - (res->end + 1), to_alloc);
 
 		/* space at the end of the region */
-		if (!alloc && !next && res->end < region_res->end)
-			alloc = min(region_res->end - res->end, to_alloc);
+		if (!alloc && !next && res->end < parent->end)
+			alloc = min(parent->end - res->end, to_alloc);
 
 		if (!alloc)
 			continue;
 
 		if (adjust_ok(dev_dax, res)) {
 			rc = adjust_dev_dax_range(dev_dax, res, resource_size(res) + alloc);
-			break;
+			if (rc)
+				return rc;
+			return alloc;
 		}
-		rc = alloc_dev_dax_range(dev_dax, res->end + 1, alloc);
-		break;
+		rc = alloc_dev_dax_range(parent, dev_dax, res->end + 1, alloc);
+		if (rc)
+			return rc;
+		return alloc;
 	}
-	if (rc)
-		return rc;
+
+	/* available was already calculated and should never be an issue */
+	dev_WARN_ONCE(&dev_dax->dev, 1, "space not found?");
+	return 0;
+}
+
+static ssize_t dev_dax_resize(struct dax_region *dax_region,
+		struct dev_dax *dev_dax, resource_size_t size)
+{
+	resource_size_t avail = dax_region_avail_size(dax_region), to_alloc;
+	resource_size_t dev_size = dev_dax_size(dev_dax);
+	struct device *dev = &dev_dax->dev;
+	resource_size_t alloc;
+
+	if (dev->driver)
+		return -EBUSY;
+	if (size == dev_size)
+		return 0;
+	if (size > dev_size && size - dev_size > avail)
+		return -ENOSPC;
+	if (size < dev_size)
+		return dev_dax_shrink(dev_dax, size);
+
+	to_alloc = size - dev_size;
+	if (dev_WARN_ONCE(dev, !alloc_is_aligned(dev_dax, to_alloc),
+			"resize of %pa misaligned\n", &to_alloc))
+		return -ENXIO;
+
+retry:
+	alloc = dev_dax_resize_static(&dax_region->res, dev_dax, to_alloc);
+	if (alloc <= 0)
+		return alloc;
 	to_alloc -= alloc;
 	if (to_alloc)
 		goto retry;
@@ -1198,7 +1225,8 @@ static ssize_t mapping_store(struct device *dev, struct device_attribute *attr,
 
 	to_alloc = range_len(&r);
 	if (alloc_is_aligned(dev_dax, to_alloc))
-		rc = alloc_dev_dax_range(dev_dax, r.start, to_alloc);
+		rc = alloc_dev_dax_range(&dax_region->res, dev_dax, r.start,
+					 to_alloc);
 	up_write(&dax_dev_rwsem);
 	up_write(&dax_region_rwsem);
 
@@ -1466,7 +1494,8 @@ static struct dev_dax *__devm_create_dev_dax(struct dev_dax_data *data)
 	device_initialize(dev);
 	dev_set_name(dev, "dax%d.%d", dax_region->id, dev_dax->id);
 
-	rc = alloc_dev_dax_range(dev_dax, dax_region->res.start, data->size);
+	rc = alloc_dev_dax_range(&dax_region->res, dev_dax, dax_region->res.start,
+				 data->size);
 	if (rc)
 		goto err_range;
 
Signature · 1 line
2.46.0
Reviewed-by on v4 23/28
#108 lore raw

Re: [PATCH v4 23/28] dax/bus: Factor out dev dax resize logic

Quoted 36 lines · On Mon, 07 Oct 2024 18:16:29 -0500 Ira Weiny <[email protected]> wrote:

Dynamic Capacity regions must limit dev dax resources to those areas which have extents backing real memory. Such DAX regions are dubbed 'sparse' regions. In order to manage where memory is available four alternatives were considered:

1) Create a single region resource child on region creation which reserves the entire region. Then as extents are added punch holes in this reservation. This requires new resource manipulation to punch the holes and still requires an additional iteration over the extent areas which may already have existing dev dax resources used.

2) Maintain an ordered xarray of extents which can be queried while processing the resize logic. The issue is that existing region->res children may artificially limit the allocation size sent to alloc_dev_dax_range(). IE the resource children can't be directly used in the resize logic to find where space in the region is. This also poses a problem of managing the available size in 2 places.

3) Maintain a separate resource tree with extents. This option is the same as 2) but with the different data structure. Most ideally there should be a unified representation of the resource tree not two places to look for space.

4) Create region resource children for each extent. Manage the dax dev resize logic in the same way as before but use a region child (extent) resource as the parents to find space within each extent.

Option 4 can leverage the existing resize algorithm to find space within the extents. It manages the available space in a singular resource tree which is less complicated for finding space.

In preparation for this change, factor out the dev_dax_resize logic. For static regions use dax_region->res as the parent to find space for the dax ranges. Future patches will use the same algorithm with individual extent resources as the parent.

Quoted 4 lines

---
Changes:
[Jonathan: Fix handling of alloc]

Trivial comments inline.
Not an area I know much about, so treat this one as a 'smells ok' type of tag.

drivers/dax/bus.c static ssize_t dev_dax_resize(struct dax_region *dax_region, · line 1100

Quoted 5 lines
drivers/dax/bus.c +5 -0
+
+static ssize_t dev_dax_resize(struct dax_region *dax_region,
+		struct dev_dax *dev_dax, resource_size_t size)
+{
+	resource_size_t avail = dax_region_avail_size(dax_region), to_alloc;
	resource_size_t to_alloc;

on it's own line. That was hard to spot all the way over there. Obviously this was in original code, but maybe slip a tidy up in whilst you are moving it?

Quoted 25 lines
drivers/dax/bus.c +22 -0
+	resource_size_t dev_size = dev_dax_size(dev_dax);
+	struct device *dev = &dev_dax->dev;
+	resource_size_t alloc;
+
+	if (dev->driver)
+		return -EBUSY;
+	if (size == dev_size)
+		return 0;
+	if (size > dev_size && size - dev_size > avail)
+		return -ENOSPC;
+	if (size < dev_size)
+		return dev_dax_shrink(dev_dax, size);
+
+	to_alloc = size - dev_size;
+	if (dev_WARN_ONCE(dev, !alloc_is_aligned(dev_dax, to_alloc),
+			"resize of %pa misaligned\n", &to_alloc))
+		return -ENXIO;
+
+retry:
+	alloc = dev_dax_resize_static(&dax_region->res, dev_dax, to_alloc);
+	if (alloc <= 0)
+		return alloc;
 	to_alloc -= alloc;
 	if (to_alloc)
 		goto retry;
author
#109 lore raw

Re: [PATCH v4 23/28] dax/bus: Factor out dev dax resize logic

Quoted 2 lines · Jonathan Cameron

On Mon, 07 Oct 2024 18:16:29 -0500
Ira Weiny <[email protected]> wrote:

[snip]

Quoted 8 lines
Quoted 4 lines

---
Changes:
[Jonathan: Fix handling of alloc]

Trivial comments inline.
Not an area I know much about, so treat this one as a 'smells ok' type of tag.

NP

Quoted 2 lines

Thanks

Quoted 11 lines

drivers/dax/bus.c static ssize_t dev_dax_resize(struct dax_region *dax_region, · line 1100

Quoted 6 lines
drivers/dax/bus.c +5 -0
+
+static ssize_t dev_dax_resize(struct dax_region *dax_region,
+		struct dev_dax *dev_dax, resource_size_t size)
+{
+	resource_size_t avail = dax_region_avail_size(dax_region), to_alloc;
	resource_size_t to_alloc;

on it's own line. That was hard to spot all the way over there. Obviously this was in original code, but maybe slip a tidy up in whilst you are moving it?

Yea fixed up.
Ira

#110 lore raw

Re: [PATCH v4 23/28] dax/bus: Factor out dev dax resize logic

In reply to Ira Weiny
Quoted 31 lines · Ira Weiny

Dynamic Capacity regions must limit dev dax resources to those areas which have extents backing real memory. Such DAX regions are dubbed 'sparse' regions. In order to manage where memory is available four alternatives were considered:

1) Create a single region resource child on region creation which reserves the entire region. Then as extents are added punch holes in this reservation. This requires new resource manipulation to punch the holes and still requires an additional iteration over the extent areas which may already have existing dev dax resources used.

2) Maintain an ordered xarray of extents which can be queried while processing the resize logic. The issue is that existing region->res children may artificially limit the allocation size sent to alloc_dev_dax_range(). IE the resource children can't be directly used in the resize logic to find where space in the region is. This also poses a problem of managing the available size in 2 places.

3) Maintain a separate resource tree with extents. This option is the same as 2) but with the different data structure. Most ideally there should be a unified representation of the resource tree not two places to look for space.

4) Create region resource children for each extent. Manage the dax dev resize logic in the same way as before but use a region child (extent) resource as the parents to find space within each extent.

Option 4 can leverage the existing resize algorithm to find space within the extents. It manages the available space in a singular resource tree which is less complicated for finding space.

Quoted 6 lines

In preparation for this change, factor out the dev_dax_resize logic. For static regions use dax_region->res as the parent to find space for the dax ranges. Future patches will use the same algorithm with individual extent resources as the parent.

LGTM based on the code logic, but not familiar with dax resource management.

Fan

Quoted 210 lines

---
Changes:
[Jonathan: Fix handling of alloc]

---
 drivers/dax/bus.c | 129 +++++++++++++++++++++++++++++++++---------------------
 1 file changed, 79 insertions(+), 50 deletions(-)
drivers/dax/bus.c +79 -50
index d8cb5195a227..f0e3f8c787df 100644
@@ -844,11 +844,9 @@ static int devm_register_dax_mapping(struct dev_dax *dev_dax, int range_id)
 	return 0;
 }
 
-static int alloc_dev_dax_range(struct dev_dax *dev_dax, u64 start,
-		resource_size_t size)
+static int alloc_dev_dax_range(struct resource *parent, struct dev_dax *dev_dax,
+			       u64 start, resource_size_t size)
 {
-	struct dax_region *dax_region = dev_dax->region;
-	struct resource *res = &dax_region->res;
 	struct device *dev = &dev_dax->dev;
 	struct dev_dax_range *ranges;
 	unsigned long pgoff = 0;
@@ -866,14 +864,14 @@ static int alloc_dev_dax_range(struct dev_dax *dev_dax, u64 start,
 		return 0;
 	}
 
-	alloc = __request_region(res, start, size, dev_name(dev), 0);
+	alloc = __request_region(parent, start, size, dev_name(dev), 0);
 	if (!alloc)
 		return -ENOMEM;
 
 	ranges = krealloc(dev_dax->ranges, sizeof(*ranges)
 			* (dev_dax->nr_range + 1), GFP_KERNEL);
 	if (!ranges) {
-		__release_region(res, alloc->start, resource_size(alloc));
+		__release_region(parent, alloc->start, resource_size(alloc));
 		return -ENOMEM;
 	}
 
@@ -1026,50 +1024,45 @@ static bool adjust_ok(struct dev_dax *dev_dax, struct resource *res)
 	return true;
 }
 
-static ssize_t dev_dax_resize(struct dax_region *dax_region,
-		struct dev_dax *dev_dax, resource_size_t size)
+/**
+ * dev_dax_resize_static - Expand the device into the unused portion of the
+ * region. This may involve adjusting the end of an existing resource, or
+ * allocating a new resource.
+ *
+ * @parent: parent resource to allocate this range in
+ * @dev_dax: DAX device to be expanded
+ * @to_alloc: amount of space to alloc; must be <= space available in @parent
+ *
+ * Return the amount of space allocated or -ERRNO on failure
+ */
+static ssize_t dev_dax_resize_static(struct resource *parent,
+				     struct dev_dax *dev_dax,
+				     resource_size_t to_alloc)
 {
-	resource_size_t avail = dax_region_avail_size(dax_region), to_alloc;
-	resource_size_t dev_size = dev_dax_size(dev_dax);
-	struct resource *region_res = &dax_region->res;
-	struct device *dev = &dev_dax->dev;
 	struct resource *res, *first;
-	resource_size_t alloc = 0;
 	int rc;
 
-	if (dev->driver)
-		return -EBUSY;
-	if (size == dev_size)
-		return 0;
-	if (size > dev_size && size - dev_size > avail)
-		return -ENOSPC;
-	if (size < dev_size)
-		return dev_dax_shrink(dev_dax, size);
-
-	to_alloc = size - dev_size;
-	if (dev_WARN_ONCE(dev, !alloc_is_aligned(dev_dax, to_alloc),
-			"resize of %pa misaligned\n", &to_alloc))
-		return -ENXIO;
-
-	/*
-	 * Expand the device into the unused portion of the region. This
-	 * may involve adjusting the end of an existing resource, or
-	 * allocating a new resource.
-	 */
-retry:
-	first = region_res->child;
-	if (!first)
-		return alloc_dev_dax_range(dev_dax, dax_region->res.start, to_alloc);
+	first = parent->child;
+	if (!first) {
+		rc = alloc_dev_dax_range(parent, dev_dax,
+					   parent->start, to_alloc);
+		if (rc)
+			return rc;
+		return to_alloc;
+	}
 
-	rc = -ENOSPC;
 	for (res = first; res; res = res->sibling) {
 		struct resource *next = res->sibling;
+		resource_size_t alloc;
 
 		/* space at the beginning of the region */
-		if (res == first && res->start > dax_region->res.start) {
-			alloc = min(res->start - dax_region->res.start, to_alloc);
-			rc = alloc_dev_dax_range(dev_dax, dax_region->res.start, alloc);
-			break;
+		if (res == first && res->start > parent->start) {
+			alloc = min(res->start - parent->start, to_alloc);
+			rc = alloc_dev_dax_range(parent, dev_dax,
+						 parent->start, alloc);
+			if (rc)
+				return rc;
+			return alloc;
 		}
 
 		alloc = 0;
@@ -1078,21 +1071,55 @@ static ssize_t dev_dax_resize(struct dax_region *dax_region,
 			alloc = min(next->start - (res->end + 1), to_alloc);
 
 		/* space at the end of the region */
-		if (!alloc && !next && res->end < region_res->end)
-			alloc = min(region_res->end - res->end, to_alloc);
+		if (!alloc && !next && res->end < parent->end)
+			alloc = min(parent->end - res->end, to_alloc);
 
 		if (!alloc)
 			continue;
 
 		if (adjust_ok(dev_dax, res)) {
 			rc = adjust_dev_dax_range(dev_dax, res, resource_size(res) + alloc);
-			break;
+			if (rc)
+				return rc;
+			return alloc;
 		}
-		rc = alloc_dev_dax_range(dev_dax, res->end + 1, alloc);
-		break;
+		rc = alloc_dev_dax_range(parent, dev_dax, res->end + 1, alloc);
+		if (rc)
+			return rc;
+		return alloc;
 	}
-	if (rc)
-		return rc;
+
+	/* available was already calculated and should never be an issue */
+	dev_WARN_ONCE(&dev_dax->dev, 1, "space not found?");
+	return 0;
+}
+
+static ssize_t dev_dax_resize(struct dax_region *dax_region,
+		struct dev_dax *dev_dax, resource_size_t size)
+{
+	resource_size_t avail = dax_region_avail_size(dax_region), to_alloc;
+	resource_size_t dev_size = dev_dax_size(dev_dax);
+	struct device *dev = &dev_dax->dev;
+	resource_size_t alloc;
+
+	if (dev->driver)
+		return -EBUSY;
+	if (size == dev_size)
+		return 0;
+	if (size > dev_size && size - dev_size > avail)
+		return -ENOSPC;
+	if (size < dev_size)
+		return dev_dax_shrink(dev_dax, size);
+
+	to_alloc = size - dev_size;
+	if (dev_WARN_ONCE(dev, !alloc_is_aligned(dev_dax, to_alloc),
+			"resize of %pa misaligned\n", &to_alloc))
+		return -ENXIO;
+
+retry:
+	alloc = dev_dax_resize_static(&dax_region->res, dev_dax, to_alloc);
+	if (alloc <= 0)
+		return alloc;
 	to_alloc -= alloc;
 	if (to_alloc)
 		goto retry;
@@ -1198,7 +1225,8 @@ static ssize_t mapping_store(struct device *dev, struct device_attribute *attr,
 
 	to_alloc = range_len(&r);
 	if (alloc_is_aligned(dev_dax, to_alloc))
-		rc = alloc_dev_dax_range(dev_dax, r.start, to_alloc);
+		rc = alloc_dev_dax_range(&dax_region->res, dev_dax, r.start,
+					 to_alloc);
 	up_write(&dax_dev_rwsem);
 	up_write(&dax_region_rwsem);
 
@@ -1466,7 +1494,8 @@ static struct dev_dax *__devm_create_dev_dax(struct dev_dax_data *data)
 	device_initialize(dev);
 	dev_set_name(dev, "dax%d.%d", dax_region->id, dev_dax->id);
 
-	rc = alloc_dev_dax_range(dev_dax, dax_region->res.start, data->size);
+	rc = alloc_dev_dax_range(&dax_region->res, dev_dax, dax_region->res.start,
+				 data->size);
 	if (rc)
 		goto err_range;
 
Signature · 2 lines
-- 
2.46.0
Signature · 1 line
Fan Ni
#111 lore raw

PATCH v4 24/28 dax/region: Create resources on sparse DAX regions

In reply to Ira Weiny

From: Navneet Singh <[email protected]>

DAX regions which map dynamic capacity partitions require that memory be allowed to come and go. Recall sparse regions were created for this purpose. Now that extents can be realized within DAX regions the DAX region driver can start tracking sub-resource information.

The tight relationship between DAX region operations and extent operations require memory changes to be controlled synchronously with the user of the region. Synchronize through the dax_region_rwsem and by having the region driver drive both the region device as well as the extent sub-devices.

Recall requests to remove extents can happen at any time and that a host is not obligated to release the memory until it is not being used. If an extent is not used allow a release response.

The DAX layer has no need for the details of the CXL memory extent devices. Expose extents to the DAX layer as device children of the DAX region device. A single callback from the driver aids the DAX layer to determine if the child device is an extent. The DAX layer also registers a devres function to automatically clean up when the device is removed from the region.

There is a race between extents being surfaced and the dax_cxl driver being loaded. The driver must therefore scan for any existing extents while still under the device lock.

Respond to extent notifications. Manage the DAX region resource tree based on the extents lifetime. Return the status of remove notifications to lower layers such that it can manage the hardware appropriately.

---
Changes:
[djiang: clarify the use of dax_region_{add,rm}_resource()
and dax_avail_size()]
[djiang: use guard(device)() and clean up notify function] [Jonathan: fix briefly]
[Jonathan: clarify the comment when dax_resource must be released] [Jonathan: fix bug with put_device]
[Jonathan: remove available_size variable]
[Jonathan: Fix error message]
[Jonathan: clarify error return of cxl_dax_region_notify()] [Jonathan: Fix bracket coding style]
[iweiny: s/%par/%pra/]

---
 drivers/cxl/core/extent.c |  74 ++++++++++++--
 drivers/cxl/cxl.h         |   6 ++
 drivers/dax/bus.c         | 243 +++++++++++++++++++++++++++++++++++++++++-----
 drivers/dax/bus.h         |   3 +-
 drivers/dax/cxl.c         |  62 +++++++++++-
 drivers/dax/dax-private.h |  42 ++++++++
 drivers/dax/hmem/hmem.c   |   2 +-
 drivers/dax/pmem.c        |   2 +-
 8 files changed, 396 insertions(+), 38 deletions(-)
drivers/cxl/core/extent.c +68 -6
index a1eb6e8e4f1a..75fb73ce2185 100644
@@ -270,20 +270,65 @@ static void calc_hpa_range(struct cxl_endpoint_decoder *cxled,
 	hpa_range->end = hpa_range->start + range_len(dpa_range) - 1;
 }
 
+static int cxlr_notify_extent(struct cxl_region *cxlr, enum dc_event event,
+			      struct region_extent *region_extent)
+{
+	struct device *dev = &cxlr->cxlr_dax->dev;
+	struct cxl_notify_data notify_data;
+	struct cxl_driver *driver;
+
+	dev_dbg(dev, "Trying notify: type %d HPA %pra\n",
+		event, &region_extent->hpa_range);
+
+	guard(device)(dev);
+
+	/*
+	 * The lack of a driver indicates a notification has failed.  No user
+	 * space coordiantion was possible.
+	 */
+	if (!dev->driver)
+		return 0;
+	driver = to_cxl_drv(dev->driver);
+	if (!driver->notify)
+		return 0;
+
+	notify_data = (struct cxl_notify_data) {
+		.event = event,
+		.region_extent = region_extent,
+	};
+
+	dev_dbg(dev, "Notify: type %d HPA %pra\n",
+		event, &region_extent->hpa_range);
+	return driver->notify(dev, &notify_data);
+}
+
+struct rm_data {
+	struct cxl_region *cxlr;
+	struct range *range;
+};
+
 static int cxlr_rm_extent(struct device *dev, void *data)
 {
 	struct region_extent *region_extent = to_region_extent(dev);
-	struct range *region_hpa_range = data;
+	struct rm_data *rm_data = data;
+	int rc;
 
 	if (!region_extent)
 		return 0;
 
 	/*
-	 * Any extent which 'touches' the released range is removed.
+	 * Any extent which 'touches' the released range is attempted to be
+	 * removed.
 	 */
-	if (range_overlaps(region_hpa_range, &region_extent->hpa_range)) {
+	if (range_overlaps(rm_data->range, &region_extent->hpa_range)) {
+		struct cxl_region *cxlr = rm_data->cxlr;
+
 		dev_dbg(dev, "Remove region extent HPA %pra\n",
 			&region_extent->hpa_range);
+		rc = cxlr_notify_extent(cxlr, DCD_RELEASE_CAPACITY, region_extent);
+		if (rc == -EBUSY)
+			return 0;
+		/* Extent not in use or error, remove it */
 		region_rm_extent(region_extent);
 	}
 	return 0;
@@ -328,8 +373,13 @@ int cxl_rm_extent(struct cxl_memdev_state *mds, struct cxl_extent *extent)
 
 	calc_hpa_range(cxled, cxlr->cxlr_dax, &dpa_range, &hpa_range);
 
+	struct rm_data rm_data = {
+		.cxlr = cxlr,
+		.range = &hpa_range,
+	};
+
 	/* Remove region extents which overlap */
-	return device_for_each_child(&cxlr->cxlr_dax->dev, &hpa_range,
+	return device_for_each_child(&cxlr->cxlr_dax->dev, &rm_data,
 				     cxlr_rm_extent);
 }
 
@@ -354,8 +404,20 @@ static int cxlr_add_extent(struct cxl_dax_region *cxlr_dax,
 		return rc;
 	}
 
-	/* device model handles freeing region_extent */
-	return online_region_extent(region_extent);
+	rc = online_region_extent(region_extent);
+	/* device model handled freeing region_extent */
+	if (rc)
+		return rc;
+
+	rc = cxlr_notify_extent(cxlr_dax->cxlr, DCD_ADD_CAPACITY, region_extent);
+	/*
+	 * The region device was briefly live but DAX layer ensures it was not
+	 * used
+	 */
+	if (rc)
+		region_rm_extent(region_extent);
+
+	return rc;
 }
 
 /* Callers are expected to ensure cxled has been attached to a region */
drivers/cxl/cxl.h +6 -0
index b75653e9bc32..fce007af98cf 100644
@@ -918,10 +918,16 @@ bool is_cxl_region(struct device *dev);
 
 extern struct bus_type cxl_bus_type;
 
+struct cxl_notify_data {
+	enum dc_event event;
+	struct region_extent *region_extent;
+};
+
 struct cxl_driver {
 	const char *name;
 	int (*probe)(struct device *dev);
 	void (*remove)(struct device *dev);
+	int (*notify)(struct device *dev, struct cxl_notify_data *notify_data);
 	struct device_driver drv;
 	int id;
 };
drivers/dax/bus.c +217 -26

376 lines not on this page · raw message

drivers/dax/bus.h +2 -1
index 783bfeef42cc..ae5029ea6047 100644
@@ -9,6 +9,7 @@ struct dev_dax;
 struct resource;
 struct dax_device;
 struct dax_region;
+struct dax_sparse_ops;
 
 /* dax bus specific ioresource flags */
 #define IORESOURCE_DAX_STATIC BIT(0)
@@ -17,7 +18,7 @@ struct dax_region;
 
 struct dax_region *alloc_dax_region(struct device *parent, int region_id,
 		struct range *range, int target_node, unsigned int align,
-		unsigned long flags);
+		unsigned long flags, struct dax_sparse_ops *sparse_ops);
 
 struct dev_dax_data {
 	struct dax_region *dax_region;
drivers/dax/cxl.c +59 -3

86 lines not on this page · raw message

drivers/dax/dax-private.h +42 -0

78 lines not on this page · raw message

drivers/dax/hmem/hmem.c +1 -1
index 5e7c53f18491..0eea65052874 100644
@@ -28,7 +28,7 @@ static int dax_hmem_probe(struct platform_device *pdev)
 
 	mri = dev->platform_data;
 	dax_region = alloc_dax_region(dev, pdev->id, &mri->range,
-				      mri->target_node, PMD_SIZE, flags);
+				      mri->target_node, PMD_SIZE, flags, NULL);
 	if (!dax_region)
 		return -ENOMEM;
 
drivers/dax/pmem.c +1 -1
index c8ebf4e281f2..f927e855f240 100644
@@ -54,7 +54,7 @@ static struct dev_dax *__dax_pmem_probe(struct device *dev)
 	range.start += offset;
 	dax_region = alloc_dax_region(dev, region_id, &range,
 			nd_region->target_node, le32_to_cpu(pfn_sb->align),
-			IORESOURCE_DAX_STATIC);
+			IORESOURCE_DAX_STATIC, NULL);
 	if (!dax_region)
 		return ERR_PTR(-ENOMEM);
 
Signature · 1 line
2.46.0
#112 lore raw

Re: [PATCH v4 24/28] dax/region: Create resources on sparse DAX regions

Quoted 28 lines · On Mon, 07 Oct 2024 18:16:30 -0500 [email protected] wrote:

From: Navneet Singh <[email protected]>

DAX regions which map dynamic capacity partitions require that memory be allowed to come and go. Recall sparse regions were created for this purpose. Now that extents can be realized within DAX regions the DAX region driver can start tracking sub-resource information.

The tight relationship between DAX region operations and extent operations require memory changes to be controlled synchronously with the user of the region. Synchronize through the dax_region_rwsem and by having the region driver drive both the region device as well as the extent sub-devices.

Recall requests to remove extents can happen at any time and that a host is not obligated to release the memory until it is not being used. If an extent is not used allow a release response.

The DAX layer has no need for the details of the CXL memory extent devices. Expose extents to the DAX layer as device children of the DAX region device. A single callback from the driver aids the DAX layer to determine if the child device is an extent. The DAX layer also registers a devres function to automatically clean up when the device is removed from the region.

There is a race between extents being surfaced and the dax_cxl driver being loaded. The driver must therefore scan for any existing extents while still under the device lock.

Quoted 9 lines

Respond to extent notifications. Manage the DAX region resource tree based on the extents lifetime. Return the status of remove notifications to lower layers such that it can manage the hardware appropriately.

More somewhat superficial review from me.
Needs DAX expert reviewers.

Jonathan

Quoted 24 lines
---
 drivers/cxl/core/extent.c |  74 ++++++++++++--
 drivers/cxl/cxl.h         |   6 ++
 drivers/dax/bus.c         | 243 +++++++++++++++++++++++++++++++++++++++++-----
 drivers/dax/bus.h         |   3 +-
 drivers/dax/cxl.c         |  62 +++++++++++-
 drivers/dax/dax-private.h |  42 ++++++++
 drivers/dax/hmem/hmem.c   |   2 +-
 drivers/dax/pmem.c        |   2 +-
 8 files changed, 396 insertions(+), 38 deletions(-)
drivers/cxl/core/extent.c +5 -0
index a1eb6e8e4f1a..75fb73ce2185 100644
@@ -270,20 +270,65 @@ static void calc_hpa_range(struct cxl_endpoint_decoder *cxled,
 	hpa_range->end = hpa_range->start + range_len(dpa_range) - 1;
 }
 
+static int cxlr_notify_extent(struct cxl_region *cxlr, enum dc_event event,
+			      struct region_extent *region_extent)
+{
+	struct device *dev = &cxlr->cxlr_dax->dev;
+	struct cxl_notify_data notify_data;

drivers/cxl/core/extent.c static void calc_hpa_range(struct cxl_endpoint_decoder *cxled, · line 287

Quoted 10 lines
drivers/cxl/core/extent.c +10 -0
+	struct cxl_driver *driver;
+
+	dev_dbg(dev, "Trying notify: type %d HPA %pra\n",
+		event, &region_extent->hpa_range);
+
+	guard(device)(dev);
+
+	/*
+	 * The lack of a driver indicates a notification has failed.  No user
+	 * space coordiantion was possible.

spell check.
coordination

Quoted 45 lines
drivers/cxl/core/extent.c +16 -0
+	 */
+	if (!dev->driver)
+		return 0;
+	driver = to_cxl_drv(dev->driver);
+	if (!driver->notify)
+		return 0;
+
+	notify_data = (struct cxl_notify_data) {
+		.event = event,
+		.region_extent = region_extent,
+	};
+
+	dev_dbg(dev, "Notify: type %d HPA %pra\n",
+		event, &region_extent->hpa_range);
+	return driver->notify(dev, &notify_data);
+}
 
drivers/dax/bus.c +20 -0
index f0e3f8c787df..4e19d18369de 100644
@@ -183,6 +183,86 @@ static bool is_sparse(struct dax_region *dax_region)
 	return (dax_region->res.flags & IORESOURCE_DAX_SPARSE_CAP) != 0;
 }
 
+
+int dax_region_add_resource(struct dax_region *dax_region,
+			    struct device *device,
+			    resource_size_t start, resource_size_t length)
+{
+	struct resource *new_resource;
+	int rc;
+
+	struct dax_resource *dax_resource __free(kfree) =
+				kzalloc(sizeof(*dax_resource), GFP_KERNEL);
+	if (!dax_resource)
+		return -ENOMEM;
+
+	guard(rwsem_write)(&dax_region_rwsem);
+
+	dev_dbg(dax_region->dev, "DAX region resource %pr\n", &dax_region->res);
+	new_resource = __request_region(&dax_region->res, start, length, "extent", 0);
+	if (!new_resource) {
+		dev_err(dax_region->dev, "Failed to add region s:%pa l:%pa\n",
+			&start, &length);

drivers/dax/bus.c static bool is_sparse(struct dax_region *dax_region) · line 215

Quoted 10 lines
drivers/dax/bus.c +10 -0
+		return -ENOSPC;
+	}
+
+	dev_dbg(dax_region->dev, "add resource %pr\n", new_resource);
+	dax_resource->region = dax_region;
+	dax_resource->res = new_resource;
+	dev_set_drvdata(device, dax_resource);
+	rc = devm_add_action_or_reset(device, dax_release_resource,
+				      no_free_ptr(dax_resource));
+	/*  On error; ensure driver data is cleared under semaphore */

It's not used in the dax_release_resource callback (that I can immediately spot) so could you just not set it until after this has succeeded?

drivers/dax/bus.c static bool is_sparse(struct dax_region *dax_region) · line 238

Quoted 2 lines
drivers/dax/bus.c +2 -0
+	if (rc)
+		dev_set_drvdata(device, NULL);

i.e. move

	dev_set_drvdata(device, dax_resource);

to here.

drivers/dax/bus.c static bool is_sparse(struct dax_region *dax_region) · line 241

Quoted 3 lines
drivers/dax/bus.c +3 -0
+	return rc;
+}
+EXPORT_SYMBOL_GPL(dax_region_add_resource);

Adding quite a few exports. Is it time to namespace DAX exports? Perhaps a follow up series.

drivers/dax/bus.c static ssize_t region_align_show(struct device *dev, · line 381

Quoted 10 lines
drivers/dax/bus.c +3 -0
 bool static_dev_dax(struct dev_dax *dev_dax)
 {
 	return is_static(dev_dax->region);
@@ -296,19 +376,44 @@ static ssize_t region_align_show(struct device *dev,
 static struct device_attribute dev_attr_region_align =
 		__ATTR(align, 0400, region_align_show, NULL);
 
+#define for_each_child_resource(extent, res) \
+	for (res = (extent)->child; res; res = res->sibling)
+

Extent naming in here is a little off for a general sounding macro. Maybe for_each_child_resource(parent, res) or something like that?

Seem generally useful. Maybe move to resource.h?

drivers/dax/bus.c static struct dev_dax *__devm_create_dev_dax(struct dev_dax_data *data) · line 1685

Quoted 8 lines
drivers/dax/bus.c +4 -0
@@ -1494,8 +1679,14 @@ static struct dev_dax *__devm_create_dev_dax(struct dev_dax_data *data)
 	device_initialize(dev);
 	dev_set_name(dev, "dax%d.%d", dax_region->id, dev_dax->id);
 
+	if (is_sparse(dax_region) && data->size) {
+		dev_err(parent, "Sparse DAX region devices must be created initially with 0 size");
+		rc = -EINVAL;
+		goto err_id;

Right label? This code doesn't have side effects and the next error path is goto err_range Looks like you fail to reverse the alloc_dev_dax_id() in this error path.

Quoted 36 lines
drivers/dax/bus.c +3 -1
+	}
+
 	rc = alloc_dev_dax_range(&dax_region->res, dev_dax, dax_region->res.start,
-				 data->size);
+				 data->size, NULL);
 	if (rc)
 		goto err_range;
 
drivers/dax/bus.h +2 -1
index 783bfeef42cc..ae5029ea6047 100644
@@ -9,6 +9,7 @@ struct dev_dax;
 struct resource;
 struct dax_device;
 struct dax_region;
+struct dax_sparse_ops;
 
 /* dax bus specific ioresource flags */
 #define IORESOURCE_DAX_STATIC BIT(0)
@@ -17,7 +18,7 @@ struct dax_region;
 
 struct dax_region *alloc_dax_region(struct device *parent, int region_id,
 		struct range *range, int target_node, unsigned int align,
-		unsigned long flags);
+		unsigned long flags, struct dax_sparse_ops *sparse_ops);
 
 struct dev_dax_data {
 	struct dax_region *dax_region;
drivers/dax/cxl.c +0 -0
index 367e86b1c22a..df979ea2cb59 100644
@@ -5,6 +5,58 @@
 
 #include "../cxl/cxl.h"

drivers/dax/cxl.c line 16

Quoted 10 lines
drivers/dax/cxl.c +9 -0
 #include "bus.h"
+#include "dax-private.h"
+
+static int __cxl_dax_add_resource(struct dax_region *dax_region,
+				  struct region_extent *region_extent)
+{
+	resource_size_t start, length;
+	struct device *dev;
+
+	dev = &region_extent->dev;

Might as well do

	struct device *dev = &region_extent->dev;
Quoted 10 lines
drivers/dax/cxl.c +4 -0
+	start = dax_region->res.start + region_extent->hpa_range.start;
+	length = range_len(&region_extent->hpa_range);
+	return dax_region_add_resource(dax_region, dev, start, length);
+}
 
 
drivers/dax/dax-private.h +0 -0
index ccde98c3d4e2..e3866115243e 100644

...

drivers/dax/dax-private.h struct dev_dax { · line 132

Quoted 12 lines
drivers/dax/dax-private.h +12 -0
+/*
+ * Similar to run_dax() dax_region_{add,rm}_resource() and dax_avail_size() are
+ * exported but are not intended to be generic operations outside the dax
+ * subsystem.  They are only generic between the dax layer and the dax drivers.
+ */
+int dax_region_add_resource(struct dax_region *dax_region, struct device *dev,
+			    resource_size_t start, resource_size_t length);
+int dax_region_rm_resource(struct dax_region *dax_region,
+			   struct device *dev);
+resource_size_t dax_avail_size(struct resource *dax_resource);
+
+typedef int (*match_cb)(struct device *dev, resource_size_t *size_avail);

Why is this here?

author
#113 lore raw

Re: [PATCH v4 24/28] dax/region: Create resources on sparse DAX regions

Quoted 4 lines · Jonathan Cameron
Quoted 1 line · On Mon, 07 Oct 2024 18:16:30 -0500 [email protected] wrote:

From: Navneet Singh <[email protected]>

[snip]

Quoted 9 lines
Quoted 9 lines
drivers/cxl/core/extent.c +9 -0
+static int cxlr_notify_extent(struct cxl_region *cxlr, enum dc_event event,
+			      struct region_extent *region_extent)
+{
+	struct device *dev = &cxlr->cxlr_dax->dev;
+	struct cxl_notify_data notify_data;
+	struct cxl_driver *driver;
+
+	dev_dbg(dev, "Trying notify: type %d HPA %pra\n",
+		event, &region_extent->hpa_range);
Quoted 8 lines

drivers/cxl/core/extent.c static void calc_hpa_range(struct cxl_endpoint_decoder *cxled, · line 287

Quoted 6 lines
drivers/cxl/core/extent.c +6 -0
+
+	guard(device)(dev);
+
+	/*
+	 * The lack of a driver indicates a notification has failed.  No user
+	 * space coordiantion was possible.

spell check.
coordination

Done.

[snip]

Quoted 35 lines
Quoted 20 lines
drivers/dax/bus.c +20 -0
+
+int dax_region_add_resource(struct dax_region *dax_region,
+			    struct device *device,
+			    resource_size_t start, resource_size_t length)
+{
+	struct resource *new_resource;
+	int rc;
+
+	struct dax_resource *dax_resource __free(kfree) =
+				kzalloc(sizeof(*dax_resource), GFP_KERNEL);
+	if (!dax_resource)
+		return -ENOMEM;
+
+	guard(rwsem_write)(&dax_region_rwsem);
+
+	dev_dbg(dax_region->dev, "DAX region resource %pr\n", &dax_region->res);
+	new_resource = __request_region(&dax_region->res, start, length, "extent", 0);
+	if (!new_resource) {
+		dev_err(dax_region->dev, "Failed to add region s:%pa l:%pa\n",
+			&start, &length);

drivers/dax/bus.c static bool is_sparse(struct dax_region *dax_region) · line 236

Quoted 10 lines
drivers/dax/bus.c +10 -0
+		return -ENOSPC;
+	}
+
+	dev_dbg(dax_region->dev, "add resource %pr\n", new_resource);
+	dax_resource->region = dax_region;
+	dax_resource->res = new_resource;
+	dev_set_drvdata(device, dax_resource);
+	rc = devm_add_action_or_reset(device, dax_release_resource,
+				      no_free_ptr(dax_resource));
+	/*  On error; ensure driver data is cleared under semaphore */

It's not used in the dax_release_resource callback (that I can immediately spot) so could you just not set it until after this has succeeded?

Quoted 5 lines

drivers/dax/bus.c static bool is_sparse(struct dax_region *dax_region) · line 238

Quoted 2 lines
drivers/dax/bus.c +2 -0
+	if (rc)
+		dev_set_drvdata(device, NULL);

i.e. move

	dev_set_drvdata(device, dax_resource);

to here.

Oh boy... I probably needed a better comment on this one. No we can't do that as written because no_free_ptr() was used to flag that the pointer was handed off. Thus at this point dax_resource is always NULL.

That said, I realize now this code has an issue with using devm_add_action_or_reset() because on error dax_region_rwsem will be taken for write recursively.

So I have to re-write this using devm_add_action() with a manual reset using __dax_release_resource()... in that case no_free_ptr() can be moved to a better place.

All that results in something much nicer:

...

        /*
         * open code devm_add_action_or_reset() to avoid recursive write lock
         * of dax_region_rwsem in the error case.
         */
        rc = devm_add_action(device, dax_release_resource, dax_resource);
        if (rc) {
                __dax_release_resource(dax_resource);
                return rc;
        }

        dev_set_drvdata(device, no_free_ptr(dax_resource));
        return 0;
}
Quoted 6 lines

drivers/dax/bus.c static bool is_sparse(struct dax_region *dax_region) · line 241

Quoted 4 lines
drivers/dax/bus.c +3 -0
+	return rc;
+}
+EXPORT_SYMBOL_GPL(dax_region_add_resource);

Adding quite a few exports. Is it time to namespace DAX exports? Perhaps a follow up series.

Perhaps. The calls have a dax_ prefix. In addition, I thought use of the export namespaces were out of favor?

Quoted 10 lines
Quoted 10 lines

bool static_dev_dax(struct dev_dax *dev_dax)
{

drivers/dax/bus.c +0 -0
 	return is_static(dev_dax->region);
@@ -296,19 +376,44 @@ static ssize_t region_align_show(struct device *dev,
 static struct device_attribute dev_attr_region_align =
 		__ATTR(align, 0400, region_align_show, NULL);
 
Quoted 7 lines

drivers/dax/bus.c static ssize_t region_align_show(struct device *dev, · line 381

Quoted 3 lines
drivers/dax/bus.c +3 -0
+#define for_each_child_resource(extent, res) \
+	for (res = (extent)->child; res; res = res->sibling)
+

Extent naming in here is a little off for a general sounding macro. Maybe for_each_child_resource(parent, res) or something like that?

Seem generally useful. Maybe move to resource.h?

I could (with the name change).

I guess the self review process ended up with something generic except for the 'extent' name.

Quoted 5 lines
Quoted 5 lines
drivers/dax/bus.c +0 -0
@@ -1494,8 +1679,14 @@ static struct dev_dax *__devm_create_dev_dax(struct dev_dax_data *data)
 	device_initialize(dev);
 	dev_set_name(dev, "dax%d.%d", dax_region->id, dev_dax->id);
 
Quoted 7 lines

drivers/dax/bus.c static struct dev_dax *__devm_create_dev_dax(struct dev_dax_data *data) · line 1685

Quoted 4 lines
drivers/dax/bus.c +4 -0
+	if (is_sparse(dax_region) && data->size) {
+		dev_err(parent, "Sparse DAX region devices must be created initially with 0 size");
+		rc = -EINVAL;
+		goto err_id;

Right label? This code doesn't have side effects and the next error path is goto err_range Looks like you fail to reverse the alloc_dev_dax_id() in this error path.

Yea.

Worse yet I think this check could be done much earlier before dev_dax allocation.

Let me work on that.

Quoted 48 lines
Quoted 37 lines
drivers/dax/bus.c +3 -1
+	}
+
 	rc = alloc_dev_dax_range(&dax_region->res, dev_dax, dax_region->res.start,
-				 data->size);
+				 data->size, NULL);
 	if (rc)
 		goto err_range;
 
drivers/dax/bus.h +2 -1
index 783bfeef42cc..ae5029ea6047 100644
@@ -9,6 +9,7 @@ struct dev_dax;
 struct resource;
 struct dax_device;
 struct dax_region;
+struct dax_sparse_ops;
 
 /* dax bus specific ioresource flags */
 #define IORESOURCE_DAX_STATIC BIT(0)
@@ -17,7 +18,7 @@ struct dax_region;
 
 struct dax_region *alloc_dax_region(struct device *parent, int region_id,
 		struct range *range, int target_node, unsigned int align,
-		unsigned long flags);
+		unsigned long flags, struct dax_sparse_ops *sparse_ops);
 
 struct dev_dax_data {
 	struct dax_region *dax_region;
drivers/dax/cxl.c +0 -0
index 367e86b1c22a..df979ea2cb59 100644
@@ -5,6 +5,58 @@
 
 #include "../cxl/cxl.h"

drivers/dax/cxl.c line 16

Quoted 10 lines
drivers/dax/cxl.c +9 -0
 #include "bus.h"
+#include "dax-private.h"
+
+static int __cxl_dax_add_resource(struct dax_region *dax_region,
+				  struct region_extent *region_extent)
+{
+	resource_size_t start, length;
+	struct device *dev;
+
+	dev = &region_extent->dev;

Might as well do

Quoted 1 line
	struct device *dev = &region_extent->dev;

sure.

Quoted 14 lines
Quoted 12 lines
drivers/dax/cxl.c +4 -0
+	start = dax_region->res.start + region_extent->hpa_range.start;
+	length = range_len(&region_extent->hpa_range);
+	return dax_region_add_resource(dax_region, dev, start, length);
+}
 
 
drivers/dax/dax-private.h +0 -0
index ccde98c3d4e2..e3866115243e 100644

...

Quoted 14 lines

drivers/dax/dax-private.h struct dev_dax { · line 132

Quoted 12 lines
drivers/dax/dax-private.h +12 -0
+/*
+ * Similar to run_dax() dax_region_{add,rm}_resource() and dax_avail_size() are
+ * exported but are not intended to be generic operations outside the dax
+ * subsystem.  They are only generic between the dax layer and the dax drivers.
+ */
+int dax_region_add_resource(struct dax_region *dax_region, struct device *dev,
+			    resource_size_t start, resource_size_t length);
+int dax_region_rm_resource(struct dax_region *dax_region,
+			   struct device *dev);
+resource_size_t dax_avail_size(struct resource *dax_resource);
+
+typedef int (*match_cb)(struct device *dev, resource_size_t *size_avail);

Why is this here?

Left over from a bygone implementation... :-/

Deleted

Ira

#114 lore raw

Re: [PATCH v4 24/28] dax/region: Create resources on sparse DAX regions

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

+EXPORT_SYMBOL_GPL(dax_region_add_resource);

Adding quite a few exports. Is it time to namespace DAX exports? Perhaps a follow up series.

Perhaps. The calls have a dax_ prefix. In addition, I thought use of the export namespaces were out of favor?

I guess I missed a change in the wind.
Any references?

Jonathan

author
#115 lore raw

Re: [PATCH v4 24/28] dax/region: Create resources on sparse DAX regions

Quoted 10 lines · Jonathan Cameron
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

+EXPORT_SYMBOL_GPL(dax_region_add_resource);

Adding quite a few exports. Is it time to namespace DAX exports? Perhaps a follow up series.

Perhaps. The calls have a dax_ prefix. In addition, I thought use of the export namespaces were out of favor?

I guess I missed a change in the wind.
Any references?

I think Dan mentioned this to me off-line WRT the CXL namespace. I'll double check with him. Regardless that is all something which should be done separate from this series as I need to reduce, not add, to it right now.

Ira

#116 lore raw

PATCH v4 25/28 cxl/region: Read existing extents on region creation

In reply to Ira Weiny

From: Navneet Singh <[email protected]>

Dynamic capacity device extents may be left in an accepted state on a device due to an unexpected host crash. In this case it is expected that the creation of a new region on top of a DC partition can read those extents and surface them for continued use.

Once all endpoint decoders are part of a region and the region is being realized, a read of the 'devices extent list' can reveal these previously accepted extents.

CXL r3.1 specifies the mailbox call Get Dynamic Capacity Extent List for this purpose. The call returns all the extents for all dynamic capacity partitions. If the fabric manager is adding extents to any DCD partition, the extent list for the recovered region may change. In this case the query must retry. Upon retry the query could encounter extents which were accepted on a previous list query. Adding such extents is ignored without error because they are entirely within a previous accepted extent.

The scan for existing extents races with the dax_cxl driver. This is synchronized through the region device lock. Extents which are found after the driver has loaded will surface through the normal notification path while extents seen prior to the driver are read during driver load.

---
Changes:
[iweiny: adjust for mailbox split]

[djiang: Update commit messages]
[djiang: s/cxl_read_extent_list/cxl_process_extent_list/] [djiang: #define CXL_READ_EXTENT_LIST_RETRY]

---
 drivers/cxl/core/core.h   |   2 +
 drivers/cxl/core/mbox.c   | 105 ++++++++++++++++++++++++++++++++++++++++++++++
 drivers/cxl/core/region.c |  12 ++++++
 drivers/cxl/cxlmem.h      |  21 ++++++++++
 4 files changed, 140 insertions(+)
drivers/cxl/core/core.h +2 -0
index 0eccdd0b9261..80d61f75161d 100644
@@ -21,6 +21,8 @@ cxled_to_mds(struct cxl_endpoint_decoder *cxled)
 	return container_of(cxlds, struct cxl_memdev_state, cxlds);
 }
 
+void cxl_process_extent_list(struct cxl_endpoint_decoder *cxled);
+
 #ifdef CONFIG_CXL_REGION
 extern struct device_attribute dev_attr_create_pmem_region;
 extern struct device_attribute dev_attr_create_ram_region;
drivers/cxl/core/mbox.c +105 -0

111 lines not on this page · raw message

drivers/cxl/core/region.c +12 -0
index 6ae51fc2bdae..5ed4a77491e5 100644
@@ -3190,6 +3190,15 @@ static int devm_cxl_add_pmem_region(struct cxl_region *cxlr)
 	return rc;
 }
 
+static void cxlr_add_existing_extents(struct cxl_region *cxlr)
+{
+	struct cxl_region_params *p = &cxlr->params;
+	int i;
+
+	for (i = 0; i < p->nr_targets; i++)
+		cxl_process_extent_list(p->targets[i]);
+}
+
 static void cxlr_dax_unregister(void *_cxlr_dax)
 {
 	struct cxl_dax_region *cxlr_dax = _cxlr_dax;
@@ -3224,6 +3233,9 @@ static int devm_cxl_add_dax_region(struct cxl_region *cxlr)
 	dev_dbg(&cxlr->dev, "%s: register %s\n", dev_name(dev->parent),
 		dev_name(dev));
 
+	if (cxlr->mode == CXL_REGION_DC)
+		cxlr_add_existing_extents(cxlr);
+
 	return devm_add_action_or_reset(&cxlr->dev, cxlr_dax_unregister,
 					cxlr_dax);
 err:
drivers/cxl/cxlmem.h +21 -0
index dd7cc0d373af..4272f134da8f 100644
@@ -626,6 +626,27 @@ struct cxl_mbox_dc_response {
 	} __packed extent_list[];
 } __packed;
 
+/*
+ * Get Dynamic Capacity Extent List; Input Payload
+ * CXL rev 3.1 section 8.2.9.9.9.2; Table 8-166
+ */
+struct cxl_mbox_get_extent_in {
+	__le32 extent_cnt;
+	__le32 start_extent_index;
+} __packed;
+
+/*
+ * Get Dynamic Capacity Extent List; Output Payload
+ * CXL rev 3.1 section 8.2.9.9.9.2; Table 8-167
+ */
+struct cxl_mbox_get_extent_out {
+	__le32 returned_extent_count;
+	__le32 total_extent_count;
+	__le32 generation_num;
+	u8 rsvd[4];
+	struct cxl_extent extent[];
+} __packed;
+
 struct cxl_mbox_get_supported_logs {
 	__le16 entries;
 	u8 rsvd[6];
Signature · 1 line
2.46.0
Reviewed-by on v4 25/28
#117 lore raw

Re: [PATCH v4 25/28] cxl/region: Read existing extents on region creation

Quoted 20 lines · On Mon, 07 Oct 2024 18:16:31 -0500 [email protected] wrote:

From: Navneet Singh <[email protected]>

Dynamic capacity device extents may be left in an accepted state on a device due to an unexpected host crash. In this case it is expected that the creation of a new region on top of a DC partition can read those extents and surface them for continued use.

Once all endpoint decoders are part of a region and the region is being realized, a read of the 'devices extent list' can reveal these previously accepted extents.

CXL r3.1 specifies the mailbox call Get Dynamic Capacity Extent List for this purpose. The call returns all the extents for all dynamic capacity partitions. If the fabric manager is adding extents to any DCD partition, the extent list for the recovered region may change. In this case the query must retry. Upon retry the query could encounter extents which were accepted on a previous list query. Adding such extents is ignored without error because they are entirely within a previous accepted extent.

Quoted 9 lines

The scan for existing extents races with the dax_cxl driver. This is synchronized through the region device lock. Extents which are found after the driver has loaded will surface through the normal notification path while extents seen prior to the driver are read during driver load.

One buglet, and a request for an error message.
With those.

Quoted 10 lines
drivers/cxl/core/mbox.c +2 -0
index d66beec687a0..6b25d15403a3 100644
@@ -1697,6 +1697,111 @@ int cxl_dev_dynamic_capacity_identify(struct cxl_memdev_state *mds)
 }
 EXPORT_SYMBOL_NS_GPL(cxl_dev_dynamic_capacity_identify, CXL);
 
+/* Return -EAGAIN if the extent list changes while reading */
+static int __cxl_process_extent_list(struct cxl_endpoint_decoder *cxled)

drivers/cxl/core/mbox.c int cxl_dev_dynamic_capacity_identify(struct cxl_memdev_state *mds) · line 1711

Quoted 10 lines
drivers/cxl/core/mbox.c +10 -0
+{
+	u32 current_index, total_read, total_expected, initial_gen_num;
+	struct cxl_memdev_state *mds = cxled_to_mds(cxled);
+	struct cxl_mailbox *cxl_mbox = &mds->cxlds.cxl_mbox;
+	struct device *dev = mds->cxlds.dev;
+	struct cxl_mbox_cmd mbox_cmd;
+	u32 max_extent_count;
+	bool first = true;
+
+	struct cxl_mbox_get_extent_out *extents __free(kfree) =

__free(kvfree)

Quoted 3 lines
drivers/cxl/core/mbox.c +3 -0
+				kvmalloc(cxl_mbox->payload_size, GFP_KERNEL);
+	if (!extents)
+		return -ENOMEM;

...

drivers/cxl/core/mbox.c static int devm_cxl_add_dax_region(struct cxl_region *cxlr) · line 3237

Quoted 11 lines
drivers/cxl/core/mbox.c +3 -0
+}
 
 static void cxlr_dax_unregister(void *_cxlr_dax)
 {
 	struct cxl_dax_region *cxlr_dax = _cxlr_dax;
@@ -3224,6 +3233,9 @@ static int devm_cxl_add_dax_region(struct cxl_region *cxlr)
 	dev_dbg(&cxlr->dev, "%s: register %s\n", dev_name(dev->parent),
 		dev_name(dev));
 
+	if (cxlr->mode == CXL_REGION_DC)
+		cxlr_add_existing_extents(cxlr);

Whilst there isn't a whole lot we can do if this fails, I'd like an error print to indicate something odd is going on. Probably pass any error up to here then print a message before carrying on.

Quoted 4 lines
drivers/cxl/core/mbox.c +1 -0
+
 	return devm_add_action_or_reset(&cxlr->dev, cxlr_dax_unregister,
 					cxlr_dax);
 err:
author
#118 lore raw

Re: [PATCH v4 25/28] cxl/region: Read existing extents on region creation

Quoted 5 lines · Jonathan Cameron
Quoted 2 lines · On Mon, 07 Oct 2024 18:16:31 -0500 [email protected] wrote:

From: Navneet Singh <[email protected]>

[snip]

Quoted 4 lines
Quoted 1 line

One buglet, and a request for an error message.
With those.

Thanks.

Quoted 9 lines
Quoted 9 lines
drivers/cxl/core/mbox.c +0 -0
index d66beec687a0..6b25d15403a3 100644
@@ -1697,6 +1697,111 @@ int cxl_dev_dynamic_capacity_identify(struct cxl_memdev_state *mds)
 }
 EXPORT_SYMBOL_NS_GPL(cxl_dev_dynamic_capacity_identify, CXL);
 
Quoted 14 lines

drivers/cxl/core/mbox.c int cxl_dev_dynamic_capacity_identify(struct cxl_memdev_state *mds) · line 1711

Quoted 12 lines
drivers/cxl/core/mbox.c +12 -0
+/* Return -EAGAIN if the extent list changes while reading */
+static int __cxl_process_extent_list(struct cxl_endpoint_decoder *cxled)
+{
+	u32 current_index, total_read, total_expected, initial_gen_num;
+	struct cxl_memdev_state *mds = cxled_to_mds(cxled);
+	struct cxl_mailbox *cxl_mbox = &mds->cxlds.cxl_mbox;
+	struct device *dev = mds->cxlds.dev;
+	struct cxl_mbox_cmd mbox_cmd;
+	u32 max_extent_count;
+	bool first = true;
+
+	struct cxl_mbox_get_extent_out *extents __free(kfree) =

__free(kvfree)

Yep fixed

Quoted 17 lines
Quoted 4 lines
drivers/cxl/core/mbox.c +3 -0
+				kvmalloc(cxl_mbox->payload_size, GFP_KERNEL);
+	if (!extents)
+		return -ENOMEM;

...

Quoted 9 lines
drivers/cxl/core/mbox.c +1 -0
+}
 
 static void cxlr_dax_unregister(void *_cxlr_dax)
 {
 	struct cxl_dax_region *cxlr_dax = _cxlr_dax;
@@ -3224,6 +3233,9 @@ static int devm_cxl_add_dax_region(struct cxl_region *cxlr)
 	dev_dbg(&cxlr->dev, "%s: register %s\n", dev_name(dev->parent),
 		dev_name(dev));
 
Quoted 6 lines

drivers/cxl/core/mbox.c static int devm_cxl_add_dax_region(struct cxl_region *cxlr) · line 3237

Quoted 2 lines
drivers/cxl/core/mbox.c +2 -0
+	if (cxlr->mode == CXL_REGION_DC)
+		cxlr_add_existing_extents(cxlr);

Whilst there isn't a whole lot we can do if this fails, I'd like an error print to indicate something odd is going on. Probably pass any error up to here then print a message before carrying on.

Bubbled up an error and added some dev_err() calls. I don't think they need to be rate limited since regions are not created very often.

Ira

[snip]

#119 lore raw

PATCH v4 26/28 cxl/mem: Trace Dynamic capacity Event Record

In reply to Ira Weiny

From: Navneet Singh <[email protected]>

CXL rev 3.1 section 8.2.9.2.1 adds the Dynamic Capacity Event Records. User space can use trace events for debugging of DC capacity changes.

Add DC trace points to the trace log.

---
Changes:
[djiang: Use 3.1 spec reference]

---
 drivers/cxl/core/mbox.c  |  4 +++
 drivers/cxl/core/trace.h | 65 ++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 69 insertions(+)
drivers/cxl/core/mbox.c +4 -0
index 6b25d15403a3..816e28cc5a40 100644
@@ -994,6 +994,10 @@ static void __cxl_event_trace_record(const struct cxl_memdev *cxlmd,
 		ev_type = CXL_CPER_EVENT_DRAM;
 	else if (uuid_equal(uuid, &CXL_EVENT_MEM_MODULE_UUID))
 		ev_type = CXL_CPER_EVENT_MEM_MODULE;
+	else if (uuid_equal(uuid, &CXL_EVENT_DC_EVENT_UUID)) {
+		trace_cxl_dynamic_capacity(cxlmd, type, &record->event.dcd);
+		return;
+	}
 
 	cxl_event_trace_record(cxlmd, type, ev_type, uuid, &record->event);
 }
drivers/cxl/core/trace.h +65 -0

71 lines not on this page · raw message

Signature · 1 line
2.46.0
Reviewed-by on v4 26/28
#120 lore raw

Re: [PATCH v4 26/28] cxl/mem: Trace Dynamic capacity Event Record

Quoted 9 lines · On Mon, 07 Oct 2024 18:16:32 -0500 [email protected] wrote:

From: Navneet Singh <[email protected]>

CXL rev 3.1 section 8.2.9.2.1 adds the Dynamic Capacity Event Records. User space can use trace events for debugging of DC capacity changes.

Add DC trace points to the trace log.

Minor comment inline about tag formatting.

Either way

Quoted 63 lines

---
Changes:
[djiang: Use 3.1 spec reference]

---
 drivers/cxl/core/mbox.c  |  4 +++
 drivers/cxl/core/trace.h | 65 ++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 69 insertions(+)
drivers/cxl/core/mbox.c +4 -0
index 6b25d15403a3..816e28cc5a40 100644
@@ -994,6 +994,10 @@ static void __cxl_event_trace_record(const struct cxl_memdev *cxlmd,
 		ev_type = CXL_CPER_EVENT_DRAM;
 	else if (uuid_equal(uuid, &CXL_EVENT_MEM_MODULE_UUID))
 		ev_type = CXL_CPER_EVENT_MEM_MODULE;
+	else if (uuid_equal(uuid, &CXL_EVENT_DC_EVENT_UUID)) {
+		trace_cxl_dynamic_capacity(cxlmd, type, &record->event.dcd);
+		return;
+	}
 
 	cxl_event_trace_record(cxlmd, type, ev_type, uuid, &record->event);
 }
drivers/cxl/core/trace.h +34 -0
index 9167cfba7f59..1303024b5239 100644
 
+TRACE_EVENT(cxl_dynamic_capacity,
+
+	TP_PROTO(const struct cxl_memdev *cxlmd, enum cxl_event_log_type log,
+		 struct cxl_event_dcd *rec),
+
+	TP_ARGS(cxlmd, log, rec),
+
+	TP_STRUCT__entry(
+		CXL_EVT_TP_entry
+
+		/* Dynamic capacity Event */
+		__field(u8, event_type)
+		__field(u16, hostid)
+		__field(u8, region_id)
+		__field(u64, dpa_start)
+		__field(u64, length)
+		__array(u8, tag, CXL_EXTENT_TAG_LEN)
+		__field(u16, sh_extent_seq)
+	),
+
+	TP_fast_assign(
+		CXL_EVT_TP_fast_assign(cxlmd, log, rec->hdr);
+
+		/* Dynamic_capacity Event */
+		__entry->event_type = rec->event_type;
+
+		/* DCD event record data */
+		__entry->hostid = le16_to_cpu(rec->host_id);
+		__entry->region_id = rec->region_index;
+		__entry->dpa_start = le64_to_cpu(rec->extent.start_dpa);
+		__entry->length = le64_to_cpu(rec->extent.length);
+		memcpy(__entry->tag, &rec->extent.tag, CXL_EXTENT_TAG_LEN);
+		__entry->sh_extent_seq = le16_to_cpu(rec->extent.shared_extn_seq);
+	),

drivers/cxl/core/trace.h TRACE_EVENT(cxl_poison, · line 794

Quoted 10 lines
drivers/cxl/core/trace.h +10 -0
+
+	CXL_EVT_TP_printk("event_type='%s' host_id='%d' region_id='%d' " \
+		"starting_dpa=%llx length=%llx tag=%s " \
+		"shared_extent_sequence=%d",
+		show_dc_evt_type(__entry->event_type),
+		__entry->hostid,
+		__entry->region_id,
+		__entry->dpa_start,
+		__entry->length,
+		__print_hex(__entry->tag, CXL_EXTENT_TAG_LEN),

%pU maybe?
https://elixir.bootlin.com/linux/v6.11.2/source/include/ras/ras_event.h#L248
uses it for the GUIDs in CPER etc.

I guess it depends on how strongly we want to push John's vision of these always being UUIDs! (I'm in favor and here is just formatting a debug print so that shouldn't be a problem even for those who want for some odd reason to use something else for tags :)

Quoted 8 lines
drivers/cxl/core/trace.h +4 -0
+		__entry->sh_extent_seq
+	)
+);
+
 #endif /* _CXL_EVENTS_H */
 
 #define TRACE_INCLUDE_FILE trace
author
#121 lore raw

Re: [PATCH v4 26/28] cxl/mem: Trace Dynamic capacity Event Record

Quoted 17 lines · Jonathan Cameron
Quoted 9 lines · On Mon, 07 Oct 2024 18:16:32 -0500 [email protected] wrote:

From: Navneet Singh <[email protected]>

CXL rev 3.1 section 8.2.9.2.1 adds the Dynamic Capacity Event Records. User space can use trace events for debugging of DC capacity changes.

Add DC trace points to the trace log.

Minor comment inline about tag formatting.

Either way

[snip]

Quoted 15 lines

drivers/cxl/core/trace.h TRACE_EVENT(cxl_poison, · line 794

Quoted 10 lines
drivers/cxl/core/trace.h +10 -0
+
+	CXL_EVT_TP_printk("event_type='%s' host_id='%d' region_id='%d' " \
+		"starting_dpa=%llx length=%llx tag=%s " \
+		"shared_extent_sequence=%d",
+		show_dc_evt_type(__entry->event_type),
+		__entry->hostid,
+		__entry->region_id,
+		__entry->dpa_start,
+		__entry->length,
+		__print_hex(__entry->tag, CXL_EXTENT_TAG_LEN),

%pU maybe?
https://elixir.bootlin.com/linux/v6.11.2/source/include/ras/ras_event.h#L248
uses it for the GUIDs in CPER etc.

Quoted 4 lines

I guess it depends on how strongly we want to push John's vision of these always being UUIDs! (I'm in favor and here is just formatting a debug print so that shouldn't be a problem even for those who want for some odd reason to use something else for tags :)

I'm not pushing against the UUID idea any more. Just forgot this print.

Changed.

Ira

[snip]

author
#122 lore raw

PATCH v4 27/28 tools/testing/cxl: Make event logs dynamic

In reply to Ira Weiny

The event logs test was created as static arrays as an easy way to mock events. Dynamic Capacity Device (DCD) test support requires events be generated dynamically when extents are created or destroyed.

The current event log test has specific checks for the number of events seen including log overflow.

Modify mock event logs to be dynamically allocated. Adjust array size and mock event entry data to match the output expected by the existing event test.

Use the static event data to create the dynamic events in the new logs without inventing complex event injection for the previous tests.

Simplify log processing by using the event log array index as the handle. Add a lock to manage concurrency required when user space is allowed to control DCD extents

---
Changes:
[iweiny: rebase to 6.12]

---
 tools/testing/cxl/test/mem.c | 268 ++++++++++++++++++++++++++-----------------
 1 file changed, 162 insertions(+), 106 deletions(-)
tools/testing/cxl/test/mem.c +162 -106
index ccdd6a504222..5e453aa2819b 100644
@@ -126,18 +126,26 @@ static struct {
 
 #define PASS_TRY_LIMIT 3
 
-#define CXL_TEST_EVENT_CNT_MAX 15
+#define CXL_TEST_EVENT_CNT_MAX 16
+/* 1 extra slot to accommodate that handles can't be 0 */
+#define CXL_TEST_EVENT_ARRAY_SIZE (CXL_TEST_EVENT_CNT_MAX + 1)
 
 /* Set a number of events to return at a time for simulation.  */
 #define CXL_TEST_EVENT_RET_MAX 4
 
+/*
+ * @last_handle: last handle (index) to have an entry stored
+ * @current_handle: current handle (index) to be returned to the user on get_event
+ * @nr_overflow: number of events added past the log size
+ * @lock: protect these state variables
+ * @events: array of pending events to be returned.
+ */
 struct mock_event_log {
-	u16 clear_idx;
-	u16 cur_idx;
-	u16 nr_events;
+	u16 last_handle;
+	u16 current_handle;
 	u16 nr_overflow;
-	u16 overflow_reset;
-	struct cxl_event_record_raw *events[CXL_TEST_EVENT_CNT_MAX];
+	rwlock_t lock;
+	struct cxl_event_record_raw *events[CXL_TEST_EVENT_ARRAY_SIZE];
 };
 
 struct mock_event_store {
@@ -172,56 +180,65 @@ static struct mock_event_log *event_find_log(struct device *dev, int log_type)
 	return &mdata->mes.mock_logs[log_type];
 }
 
-static struct cxl_event_record_raw *event_get_current(struct mock_event_log *log)
-{
-	return log->events[log->cur_idx];
-}
-
-static void event_reset_log(struct mock_event_log *log)
-{
-	log->cur_idx = 0;
-	log->clear_idx = 0;
-	log->nr_overflow = log->overflow_reset;
-}
-
 /* Handle can never be 0 use 1 based indexing for handle */
-static u16 event_get_clear_handle(struct mock_event_log *log)
+static u16 event_inc_handle(u16 handle)
 {
-	return log->clear_idx + 1;
+	handle = (handle + 1) % CXL_TEST_EVENT_ARRAY_SIZE;
+	if (!handle)
+		handle = handle + 1;
+	return handle;
 }
 
-/* Handle can never be 0 use 1 based indexing for handle */
-static __le16 event_get_cur_event_handle(struct mock_event_log *log)
-{
-	u16 cur_handle = log->cur_idx + 1;
-
-	return cpu_to_le16(cur_handle);
-}
-
-static bool event_log_empty(struct mock_event_log *log)
-{
-	return log->cur_idx == log->nr_events;
-}
-
-static void mes_add_event(struct mock_event_store *mes,
+/* Add the event or free it on overflow */
+static void mes_add_event(struct cxl_mockmem_data *mdata,
 			  enum cxl_event_log_type log_type,
 			  struct cxl_event_record_raw *event)
 {
+	struct device *dev = mdata->mds->cxlds.dev;
 	struct mock_event_log *log;
 
 	if (WARN_ON(log_type >= CXL_EVENT_TYPE_MAX))
 		return;
 
-	log = &mes->mock_logs[log_type];
+	log = &mdata->mes.mock_logs[log_type];
+
+	guard(write_lock)(&log->lock);
 
-	if ((log->nr_events + 1) > CXL_TEST_EVENT_CNT_MAX) {
+	dev_dbg(dev, "Add log %d cur %d last %d\n",
+		log_type, log->current_handle, log->last_handle);
+
+	/* Check next buffer */
+	if (event_inc_handle(log->last_handle) == log->current_handle) {
 		log->nr_overflow++;
-		log->overflow_reset = log->nr_overflow;
+		dev_dbg(dev, "Overflowing log %d nr %d\n",
+			log_type, log->nr_overflow);
+		devm_kfree(dev, event);
 		return;
 	}
 
-	log->events[log->nr_events] = event;
-	log->nr_events++;
+	dev_dbg(dev, "Log %d; handle %u\n", log_type, log->last_handle);
+	event->event.generic.hdr.handle = cpu_to_le16(log->last_handle);
+	log->events[log->last_handle] = event;
+	log->last_handle = event_inc_handle(log->last_handle);
+}
+
+static void mes_del_event(struct device *dev,
+			  struct mock_event_log *log,
+			  u16 handle)
+{
+	struct cxl_event_record_raw *record;
+
+	lockdep_assert(lockdep_is_held(&log->lock));
+
+	dev_dbg(dev, "Clearing event %u; record %u\n",
+		handle, log->current_handle);
+	record = log->events[handle];
+	if (!record)
+		dev_err(dev, "Mock event index %u empty?\n", handle);
+
+	log->events[handle] = NULL;
+	log->current_handle = event_inc_handle(log->current_handle);
+	devm_kfree(dev, record);
 }
 
 /*
@@ -234,7 +251,7 @@ static int mock_get_event(struct device *dev, struct cxl_mbox_cmd *cmd)
 {
 	struct cxl_get_event_payload *pl;
 	struct mock_event_log *log;
-	u16 nr_overflow;
+	u16 handle;
 	u8 log_type;
 	int i;
 
@@ -255,29 +272,38 @@ static int mock_get_event(struct device *dev, struct cxl_mbox_cmd *cmd)
 	memset(cmd->payload_out, 0, struct_size(pl, records, 0));
 
 	log = event_find_log(dev, log_type);
-	if (!log || event_log_empty(log))
+	if (!log)
 		return 0;
 
 	pl = cmd->payload_out;
 
-	for (i = 0; i < ret_limit && !event_log_empty(log); i++) {
-		memcpy(&pl->records[i], event_get_current(log),
-		       sizeof(pl->records[i]));
-		pl->records[i].event.generic.hdr.handle =
-				event_get_cur_event_handle(log);
-		log->cur_idx++;
+	guard(read_lock)(&log->lock);
+
+	handle = log->current_handle;
+	dev_dbg(dev, "Get log %d handle %u last %u\n",
+		log_type, handle, log->last_handle);
+	for (i = 0; i < ret_limit && handle != log->last_handle;
+	     i++, handle = event_inc_handle(handle)) {
+		struct cxl_event_record_raw *cur;
+
+		cur = log->events[handle];
+		dev_dbg(dev, "Sending event log %d handle %d idx %u\n",
+			log_type, le16_to_cpu(cur->event.generic.hdr.handle),
+			handle);
+		memcpy(&pl->records[i], cur, sizeof(pl->records[i]));
+		pl->records[i].event.generic.hdr.handle = cpu_to_le16(handle);
 	}
 
 	cmd->size_out = struct_size(pl, records, i);
 	pl->record_count = cpu_to_le16(i);
-	if (!event_log_empty(log))
+	if (handle != log->last_handle)
 		pl->flags |= CXL_GET_EVENT_FLAG_MORE_RECORDS;
 
 	if (log->nr_overflow) {
 		u64 ns;
 
 		pl->flags |= CXL_GET_EVENT_FLAG_OVERFLOW;
-		pl->overflow_err_count = cpu_to_le16(nr_overflow);
+		pl->overflow_err_count = cpu_to_le16(log->nr_overflow);
 		ns = ktime_get_real_ns();
 		ns -= 5000000000; /* 5s ago */
 		pl->first_overflow_timestamp = cpu_to_le64(ns);
@@ -292,8 +318,8 @@ static int mock_get_event(struct device *dev, struct cxl_mbox_cmd *cmd)
 static int mock_clear_event(struct device *dev, struct cxl_mbox_cmd *cmd)
 {
 	struct cxl_mbox_clear_event_payload *pl = cmd->payload_in;
-	struct mock_event_log *log;
 	u8 log_type = pl->event_log;
+	struct mock_event_log *log;
 	u16 handle;
 	int nr;
 
@@ -304,23 +330,20 @@ static int mock_clear_event(struct device *dev, struct cxl_mbox_cmd *cmd)
 	if (!log)
 		return 0; /* No mock data in this log */
 
-	/*
-	 * This check is technically not invalid per the specification AFAICS.
-	 * (The host could 'guess' handles and clear them in order).
-	 * However, this is not good behavior for the host so test it.
-	 */
-	if (log->clear_idx + pl->nr_recs > log->cur_idx) {
-		dev_err(dev,
-			"Attempting to clear more events than returned!\n");
-		return -EINVAL;
-	}
+	guard(write_lock)(&log->lock);
 
 	/* Check handle order prior to clearing events */
-	for (nr = 0, handle = event_get_clear_handle(log);
-	     nr < pl->nr_recs;
-	     nr++, handle++) {
+	handle = log->current_handle;
+	for (nr = 0; nr < pl->nr_recs && handle != log->last_handle;
+	     nr++, handle = event_inc_handle(handle)) {
+
+		dev_dbg(dev, "Checking clear of %d handle %u plhandle %u\n",
+			log_type, handle,
+			le16_to_cpu(pl->handles[nr]));
+
 		if (handle != le16_to_cpu(pl->handles[nr])) {
-			dev_err(dev, "Clearing events out of order\n");
+			dev_err(dev, "Clearing events out of order %u %u\n",
+				handle, le16_to_cpu(pl->handles[nr]));
 			return -EINVAL;
 		}
 	}
@@ -329,25 +352,12 @@ static int mock_clear_event(struct device *dev, struct cxl_mbox_cmd *cmd)
 		log->nr_overflow = 0;
 
 	/* Clear events */
-	log->clear_idx += pl->nr_recs;
-	return 0;
-}
-
-static void cxl_mock_event_trigger(struct device *dev)
-{
-	struct cxl_mockmem_data *mdata = dev_get_drvdata(dev);
-	struct mock_event_store *mes = &mdata->mes;
-	int i;
+	for (nr = 0; nr < pl->nr_recs; nr++)
+		mes_del_event(dev, log, le16_to_cpu(pl->handles[nr]));
+	dev_dbg(dev, "Delete log %d cur %d last %d\n",
+		log_type, log->current_handle, log->last_handle);
 
-	for (i = CXL_EVENT_TYPE_INFO; i < CXL_EVENT_TYPE_MAX; i++) {
-		struct mock_event_log *log;
-
-		log = event_find_log(dev, i);
-		if (log)
-			event_reset_log(log);
-	}
-
-	cxl_mem_get_event_records(mdata->mds, mes->ev_status);
+	return 0;
 }
 
 struct cxl_event_record_raw maint_needed = {
@@ -476,8 +486,27 @@ static int mock_set_timestamp(struct cxl_dev_state *cxlds,
 	return 0;
 }
 
-static void cxl_mock_add_event_logs(struct mock_event_store *mes)
+/* Create a dynamically allocated event out of a statically defined event. */
+static void add_event_from_static(struct cxl_mockmem_data *mdata,
+				  enum cxl_event_log_type log_type,
+				  struct cxl_event_record_raw *raw)
+{
+	struct device *dev = mdata->mds->cxlds.dev;
+	struct cxl_event_record_raw *rec;
+
+	rec = devm_kmemdup(dev, raw, sizeof(*rec), GFP_KERNEL);
+	if (!rec) {
+		dev_err(dev, "Failed to alloc event for log\n");
+		return;
+	}
+	mes_add_event(mdata, log_type, rec);
+}
+
+static void cxl_mock_add_event_logs(struct cxl_mockmem_data *mdata)
 {
+	struct mock_event_store *mes = &mdata->mes;
+	struct device *dev = mdata->mds->cxlds.dev;
+
 	put_unaligned_le16(CXL_GMER_VALID_CHANNEL | CXL_GMER_VALID_RANK,
 			   &gen_media.rec.media_hdr.validity_flags);
 
@@ -485,43 +514,60 @@ static void cxl_mock_add_event_logs(struct mock_event_store *mes)
 			   CXL_DER_VALID_BANK | CXL_DER_VALID_COLUMN,
 			   &dram.rec.media_hdr.validity_flags);
 
-	mes_add_event(mes, CXL_EVENT_TYPE_INFO, &maint_needed);
-	mes_add_event(mes, CXL_EVENT_TYPE_INFO,
+	dev_dbg(dev, "Generating fake event logs %d\n",
+		CXL_EVENT_TYPE_INFO);
+	add_event_from_static(mdata, CXL_EVENT_TYPE_INFO, &maint_needed);
+	add_event_from_static(mdata, CXL_EVENT_TYPE_INFO,
 		      (struct cxl_event_record_raw *)&gen_media);
-	mes_add_event(mes, CXL_EVENT_TYPE_INFO,
+	add_event_from_static(mdata, CXL_EVENT_TYPE_INFO,
 		      (struct cxl_event_record_raw *)&mem_module);
 	mes->ev_status |= CXLDEV_EVENT_STATUS_INFO;
 
-	mes_add_event(mes, CXL_EVENT_TYPE_FAIL, &maint_needed);
-	mes_add_event(mes, CXL_EVENT_TYPE_FAIL, &hardware_replace);
-	mes_add_event(mes, CXL_EVENT_TYPE_FAIL,
+	dev_dbg(dev, "Generating fake event logs %d\n",
+		CXL_EVENT_TYPE_FAIL);
+	add_event_from_static(mdata, CXL_EVENT_TYPE_FAIL, &maint_needed);
+	add_event_from_static(mdata, CXL_EVENT_TYPE_FAIL,
+		      (struct cxl_event_record_raw *)&mem_module);
+	add_event_from_static(mdata, CXL_EVENT_TYPE_FAIL, &hardware_replace);
+	add_event_from_static(mdata, CXL_EVENT_TYPE_FAIL,
 		      (struct cxl_event_record_raw *)&dram);
-	mes_add_event(mes, CXL_EVENT_TYPE_FAIL,
+	add_event_from_static(mdata, CXL_EVENT_TYPE_FAIL,
 		      (struct cxl_event_record_raw *)&gen_media);
-	mes_add_event(mes, CXL_EVENT_TYPE_FAIL,
+	add_event_from_static(mdata, CXL_EVENT_TYPE_FAIL,
 		      (struct cxl_event_record_raw *)&mem_module);
-	mes_add_event(mes, CXL_EVENT_TYPE_FAIL, &hardware_replace);
-	mes_add_event(mes, CXL_EVENT_TYPE_FAIL,
+	add_event_from_static(mdata, CXL_EVENT_TYPE_FAIL, &hardware_replace);
+	add_event_from_static(mdata, CXL_EVENT_TYPE_FAIL,
 		      (struct cxl_event_record_raw *)&dram);
 	/* Overflow this log */
-	mes_add_event(mes, CXL_EVENT_TYPE_FAIL, &hardware_replace);
-	mes_add_event(mes, CXL_EVENT_TYPE_FAIL, &hardware_replace);
-	mes_add_event(mes, CXL_EVENT_TYPE_FAIL, &hardware_replace);
-	mes_add_event(mes, CXL_EVENT_TYPE_FAIL, &hardware_replace);
-	mes_add_event(mes, CXL_EVENT_TYPE_FAIL, &hardware_replace);
-	mes_add_event(mes, CXL_EVENT_TYPE_FAIL, &hardware_replace);
-	mes_add_event(mes, CXL_EVENT_TYPE_FAIL, &hardware_replace);
-	mes_add_event(mes, CXL_EVENT_TYPE_FAIL, &hardware_replace);
-	mes_add_event(mes, CXL_EVENT_TYPE_FAIL, &hardware_replace);
-	mes_add_event(mes, CXL_EVENT_TYPE_FAIL, &hardware_replace);
+	add_event_from_static(mdata, CXL_EVENT_TYPE_FAIL, &hardware_replace);
+	add_event_from_static(mdata, CXL_EVENT_TYPE_FAIL, &hardware_replace);
+	add_event_from_static(mdata, CXL_EVENT_TYPE_FAIL, &hardware_replace);
+	add_event_from_static(mdata, CXL_EVENT_TYPE_FAIL, &hardware_replace);
+	add_event_from_static(mdata, CXL_EVENT_TYPE_FAIL, &hardware_replace);
+	add_event_from_static(mdata, CXL_EVENT_TYPE_FAIL, &hardware_replace);
+	add_event_from_static(mdata, CXL_EVENT_TYPE_FAIL, &hardware_replace);
+	add_event_from_static(mdata, CXL_EVENT_TYPE_FAIL, &hardware_replace);
+	add_event_from_static(mdata, CXL_EVENT_TYPE_FAIL, &hardware_replace);
+	add_event_from_static(mdata, CXL_EVENT_TYPE_FAIL, &hardware_replace);
 	mes->ev_status |= CXLDEV_EVENT_STATUS_FAIL;
 
-	mes_add_event(mes, CXL_EVENT_TYPE_FATAL, &hardware_replace);
-	mes_add_event(mes, CXL_EVENT_TYPE_FATAL,
+	dev_dbg(dev, "Generating fake event logs %d\n",
+		CXL_EVENT_TYPE_FATAL);
+	add_event_from_static(mdata, CXL_EVENT_TYPE_FATAL, &hardware_replace);
+	add_event_from_static(mdata, CXL_EVENT_TYPE_FATAL,
 		      (struct cxl_event_record_raw *)&dram);
 	mes->ev_status |= CXLDEV_EVENT_STATUS_FATAL;
 }
 
+static void cxl_mock_event_trigger(struct device *dev)
+{
+	struct cxl_mockmem_data *mdata = dev_get_drvdata(dev);
+	struct mock_event_store *mes = &mdata->mes;
+
+	cxl_mock_add_event_logs(mdata);
+	cxl_mem_get_event_records(mdata->mds, mes->ev_status);
+}
+
 static int mock_gsl(struct cxl_mbox_cmd *cmd)
 {
 	if (cmd->size_out < sizeof(mock_gsl_payload))
@@ -1469,6 +1515,14 @@ static int cxl_mock_mailbox_create(struct cxl_dev_state *cxlds)
 	return 0;
 }
 
+static void init_event_log(struct mock_event_log *log)
+{
+	rwlock_init(&log->lock);
+	/* Handle can never be 0 use 1 based indexing for handle */
+	log->current_handle = 1;
+	log->last_handle = 1;
+}
+
 static int cxl_mock_mem_probe(struct platform_device *pdev)
 {
 	struct device *dev = &pdev->dev;
@@ -1541,7 +1595,9 @@ static int cxl_mock_mem_probe(struct platform_device *pdev)
 	if (rc)
 		return rc;
 
-	cxl_mock_add_event_logs(&mdata->mes);
+	for (int i = 0; i < CXL_EVENT_TYPE_MAX; i++)
+		init_event_log(&mdata->mes.mock_logs[i]);
+	cxl_mock_add_event_logs(mdata);
 
 	cxlmd = devm_cxl_add_memdev(&pdev->dev, cxlds);
 	if (IS_ERR(cxlmd))
Signature · 1 line
2.46.0
Reviewed-by on v4 27/28
#123 lore raw

Re: [PATCH v4 27/28] tools/testing/cxl: Make event logs dynamic

Quoted 11 lines · On Mon, 07 Oct 2024 18:16:33 -0500 Ira Weiny <[email protected]> wrote:

The event logs test was created as static arrays as an easy way to mock events. Dynamic Capacity Device (DCD) test support requires events be generated dynamically when extents are created or destroyed.

The current event log test has specific checks for the number of events seen including log overflow.

Modify mock event logs to be dynamically allocated. Adjust array size and mock event entry data to match the output expected by the existing event test.

Quoted 8 lines

Use the static event data to create the dynamic events in the new logs without inventing complex event injection for the previous tests.

Simplify log processing by using the event log array index as the handle. Add a lock to manage concurrency required when user space is allowed to control DCD extents

Might be worth breaking up into refactor (the static cases) and then new stuff.

Otherwise one trivial comment inline.

Quoted 13 lines

---
Changes:
[iweiny: rebase to 6.12]

---
 tools/testing/cxl/test/mem.c | 268 ++++++++++++++++++++++++++-----------------
 1 file changed, 162 insertions(+), 106 deletions(-)
tools/testing/cxl/test/mem.c +0 -0
index ccdd6a504222..5e453aa2819b 100644

tools/testing/cxl/test/mem.c static struct { · line 132

Quoted 9 lines
tools/testing/cxl/test/mem.c +4 -2
@@ -126,18 +126,26 @@ static struct {
 
 /* Handle can never be 0 use 1 based indexing for handle */
-static u16 event_get_clear_handle(struct mock_event_log *log)
+static u16 event_inc_handle(u16 handle)
 {
-	return log->clear_idx + 1;
+	handle = (handle + 1) % CXL_TEST_EVENT_ARRAY_SIZE;
+	if (!handle)
+		handle = handle + 1;

That's a little confusing for me

	if (handle == 0)
		handle = 1;
Quoted 2 lines
tools/testing/cxl/test/mem.c +1 -0
+	return handle;
 }
author
#124 lore raw

Re: [PATCH v4 27/28] tools/testing/cxl: Make event logs dynamic

Quoted 3 lines · Jonathan Cameron

On Mon, 07 Oct 2024 18:16:33 -0500
Ira Weiny <[email protected]> wrote:

[snip]

Quoted 4 lines
Quoted 2 lines

Might be worth breaking up into refactor (the static cases) and then new stuff.

I had it split but the rework became difficult. And this is test code so I merged it. I'd rather keep it as is.

Quoted 4 lines

Otherwise one trivial comment inline.

Thanks.

Quoted 26 lines
Quoted 14 lines

---
Changes:
[iweiny: rebase to 6.12]

---
 tools/testing/cxl/test/mem.c | 268 ++++++++++++++++++++++++++-----------------
 1 file changed, 162 insertions(+), 106 deletions(-)
tools/testing/cxl/test/mem.c +0 -0
index ccdd6a504222..5e453aa2819b 100644

tools/testing/cxl/test/mem.c static struct { · line 132

Quoted 9 lines
tools/testing/cxl/test/mem.c +4 -2
@@ -126,18 +126,26 @@ static struct {
 
 /* Handle can never be 0 use 1 based indexing for handle */
-static u16 event_get_clear_handle(struct mock_event_log *log)
+static u16 event_inc_handle(u16 handle)
 {
-	return log->clear_idx + 1;
+	handle = (handle + 1) % CXL_TEST_EVENT_ARRAY_SIZE;
+	if (!handle)
+		handle = handle + 1;

That's a little confusing for me

Quoted 2 lines
	if (handle == 0)
		handle = 1;

Done.

Ira