Skip to contentSkip to conversation

cxl: Preparation of type2 accelerators support

PATCH In review Huang Ying 5 participants 19 messages Jul 29, 2024 – Aug 15, 2024 CXL

lore
Share & tools
lore

Huang Ying

author
#0 lore raw

cover cxl: Preparation of type2 accelerators support

There have been 2 series to add type 2 accelerators support in [1] and [2].

[1] https://lore.kernel.org/linux-cxl/168592149709.1948938.8663425987110396027.stgit@dwillia2-xfh.jf.intel.com/
[2] https://lore.kernel.org/linux-cxl/[email protected]/

Both provide relative complete support, but both are long too. To make it easier to review, some preparation of type2 accelerators support is implemented in this series. More complete support can be implemented based on it.

This series has been tested with cxl_test via mocking a type2 accelerator as in [1] above. Because more type2 accelerators support than that provided by this series is needed to simulate the device, the cxl_test patch isn't included in the series.

Huang Ying (3):
cxl: Set target type of root decoder based on CFMWS restrictions
cxl: Set target type of region with that of root decoder
cxl: Avoid to create dax regions for type2 accelerators

 drivers/cxl/acpi.c        |  5 ++++-
 drivers/cxl/core/region.c | 11 ++++++++++-
 2 files changed, 14 insertions(+), 2 deletions(-)

Best Regards,
Huang, Ying

author
#1 lore raw

PATCH 1/3 cxl: Set target type of root decoder based on CFMWS restrictions

Now, the target type of root decoder is hard-coded to HOSTONLYMEM, because only type3 expanders are supported. To support type2 accelerators, set the target type of root decoder based on the window restrictions field of CFMWS entry.

Cc Davidlohr Bueso <[email protected]>
Cc Jonathan Cameron <[email protected]>
Cc Dave Jiang <[email protected]>
Cc Alison Schofield <[email protected]>
Cc Vishal Verma <[email protected]>
Cc Ira Weiny <[email protected]>
Cc Alejandro Lucero <[email protected]>
---
 drivers/cxl/acpi.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
drivers/cxl/acpi.c +4 -1
index 82b78e331d8e..40c92ad29122 100644
@@ -382,7 +382,10 @@ static int __cxl_parse_cfmws(struct acpi_cedt_cfmws *cfmws,
 
 	cxld = &cxlrd->cxlsd.cxld;
 	cxld->flags = cfmws_to_decoder_flags(cfmws->restrictions);
-	cxld->target_type = CXL_DECODER_HOSTONLYMEM;
+	if (cxld->flags & CXL_DECODER_F_TYPE2)
+		cxld->target_type = CXL_DECODER_DEVMEM;
+	else
+		cxld->target_type = CXL_DECODER_HOSTONLYMEM;
 	cxld->hpa_range = (struct range) {
 		.start = cfmws->base_hpa,
 		.end = cfmws->base_hpa + cfmws->window_size - 1,
Signature · 1 line
2.39.2
maintainer Reviewed-by on v1 1/3
#2 lore raw

Re: [PATCH 1/3] cxl: Set target type of root decoder based on CFMWS restrictions

Quoted 5 lines · Ying Huang

Now, the target type of root decoder is hard-coded to HOSTONLYMEM, because only type3 expanders are supported. To support type2 accelerators, set the target type of root decoder based on the window restrictions field of CFMWS entry.

Quoted 32 lines
Cc Davidlohr Bueso <[email protected]>
Cc Jonathan Cameron <[email protected]>
Cc Dave Jiang <[email protected]>
Cc Alison Schofield <[email protected]>
Cc Vishal Verma <[email protected]>
Cc Ira Weiny <[email protected]>
Cc Alejandro Lucero <[email protected]>
---
 drivers/cxl/acpi.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
drivers/cxl/acpi.c +4 -1
index 82b78e331d8e..40c92ad29122 100644
@@ -382,7 +382,10 @@ static int __cxl_parse_cfmws(struct acpi_cedt_cfmws *cfmws,
 
 	cxld = &cxlrd->cxlsd.cxld;
 	cxld->flags = cfmws_to_decoder_flags(cfmws->restrictions);
-	cxld->target_type = CXL_DECODER_HOSTONLYMEM;
+	if (cxld->flags & CXL_DECODER_F_TYPE2)
+		cxld->target_type = CXL_DECODER_DEVMEM;
+	else
+		cxld->target_type = CXL_DECODER_HOSTONLYMEM;
 	cxld->hpa_range = (struct range) {
 		.start = cfmws->base_hpa,
 		.end = cfmws->base_hpa + cfmws->window_size - 1,
Signature · 1 line
2.39.2
#3 lore raw

Re: [PATCH 1/3] cxl: Set target type of root decoder based on CFMWS restrictions

In reply to Huang Ying
Quoted 23 lines · On Mon, 29 Jul 2024 16:46:09 +0800 Huang Ying <[email protected]> wrote:

Now, the target type of root decoder is hard-coded to HOSTONLYMEM, because only type3 expanders are supported. To support type2 accelerators, set the target type of root decoder based on the window restrictions field of CFMWS entry.

Cc Davidlohr Bueso <[email protected]>
Cc Jonathan Cameron <[email protected]>
Cc Dave Jiang <[email protected]>
Cc Alison Schofield <[email protected]>
Cc Vishal Verma <[email protected]>
Cc Ira Weiny <[email protected]>
Cc Alejandro Lucero <[email protected]>
---
 drivers/cxl/acpi.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
drivers/cxl/acpi.c +0 -0
index 82b78e331d8e..40c92ad29122 100644

drivers/cxl/acpi.c static int __cxl_parse_cfmws(struct acpi_cedt_cfmws *cfmws, · line 385

Quoted 5 lines
drivers/cxl/acpi.c +1 -1
@@ -382,7 +382,10 @@ static int __cxl_parse_cfmws(struct acpi_cedt_cfmws *cfmws,
 
 	cxld = &cxlrd->cxlsd.cxld;
 	cxld->flags = cfmws_to_decoder_flags(cfmws->restrictions);
-	cxld->target_type = CXL_DECODER_HOSTONLYMEM;
+	if (cxld->flags & CXL_DECODER_F_TYPE2)

These flags need updating or we are going to run into problems long term.

As of more recent specs, the distinction is messier than it was and it's device coherent HDM-D / HDM-DB (second one being type2 or type3 with BI support) and/or Host only coherent HDM-H.

I'm curious on whether anyone is support both on same CFWMS? I believe it is possible and the spec doesn't rule it out.

Jonathan

Quoted 6 lines
drivers/cxl/acpi.c +3 -0
+		cxld->target_type = CXL_DECODER_DEVMEM;
+	else
+		cxld->target_type = CXL_DECODER_HOSTONLYMEM;
 	cxld->hpa_range = (struct range) {
 		.start = cfmws->base_hpa,
 		.end = cfmws->base_hpa + cfmws->window_size - 1,
#4 lore raw

Re: [PATCH 1/3] cxl: Set target type of root decoder based on CFMWS restrictions

Hi, Jonathan,

Quoted 35 lines · Jonathan Cameron
Quoted 23 lines · On Mon, 29 Jul 2024 16:46:09 +0800 Huang Ying <[email protected]> wrote:

Now, the target type of root decoder is hard-coded to HOSTONLYMEM, because only type3 expanders are supported. To support type2 accelerators, set the target type of root decoder based on the window restrictions field of CFMWS entry.

Cc Davidlohr Bueso <[email protected]>
Cc Jonathan Cameron <[email protected]>
Cc Dave Jiang <[email protected]>
Cc Alison Schofield <[email protected]>
Cc Vishal Verma <[email protected]>
Cc Ira Weiny <[email protected]>
Cc Alejandro Lucero <[email protected]>
---
 drivers/cxl/acpi.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
drivers/cxl/acpi.c +0 -0
index 82b78e331d8e..40c92ad29122 100644

drivers/cxl/acpi.c static int __cxl_parse_cfmws(struct acpi_cedt_cfmws *cfmws, · line 385

Quoted 5 lines
drivers/cxl/acpi.c +1 -1
@@ -382,7 +382,10 @@ static int __cxl_parse_cfmws(struct acpi_cedt_cfmws *cfmws,
 
 	cxld = &cxlrd->cxlsd.cxld;
 	cxld->flags = cfmws_to_decoder_flags(cfmws->restrictions);
-	cxld->target_type = CXL_DECODER_HOSTONLYMEM;
+	if (cxld->flags & CXL_DECODER_F_TYPE2)

These flags need updating or we are going to run into problems long term.

Quoted 3 lines

As of more recent specs, the distinction is messier than it was and it's device coherent HDM-D / HDM-DB (second one being type2 or type3 with BI support) and/or Host only coherent HDM-H.

I got your idea. Previously, Device Coherent (HDM-D/DB) means type2 devices, while Host-only Coherent (HDM-H) means type3 devices. But in recent specs, type3 devices could be HDM-DB too. So, we should rename ACPI_CEDT_CFMWS_RESTRICT_TYPEX and CXL_DECODER_F_TYPEX. What's your suggestion for the new name? _DEVMEM and _HOSTONLYMEM?

Quoted 2 lines

I'm curious on whether anyone is support both on same CFWMS? I believe it is possible and the spec doesn't rule it out.

This sounds possible.

Quoted 9 lines

Jonathan

drivers/cxl/acpi.c static int __cxl_parse_cfmws(struct acpi_cedt_cfmws *cfmws, · line 391

Quoted 6 lines
drivers/cxl/acpi.c +3 -0
+		cxld->target_type = CXL_DECODER_DEVMEM;
+	else
+		cxld->target_type = CXL_DECODER_HOSTONLYMEM;
 	cxld->hpa_range = (struct range) {
 		.start = cfmws->base_hpa,
 		.end = cfmws->base_hpa + cfmws->window_size - 1,

--
Best Regards,
Huang, Ying

Reviewed-by on v1 1/3
#5 lore raw

Re: [PATCH 1/3] cxl: Set target type of root decoder based on CFMWS restrictions

In reply to Huang Ying
Quoted 14 lines · Huang Ying

Now, the target type of root decoder is hard-coded to HOSTONLYMEM, because only type3 expanders are supported. To support type2 accelerators, set the target type of root decoder based on the window restrictions field of CFMWS entry.

Cc Davidlohr Bueso <[email protected]>
Cc Jonathan Cameron <[email protected]>
Cc Dave Jiang <[email protected]>
Cc Alison Schofield <[email protected]>
Cc Vishal Verma <[email protected]>
Cc Ira Weiny <[email protected]>
Cc Alejandro Lucero <[email protected]>
Quoted 1 line

---

Quoted 22 lines
 drivers/cxl/acpi.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
drivers/cxl/acpi.c +4 -1
index 82b78e331d8e..40c92ad29122 100644
@@ -382,7 +382,10 @@ static int __cxl_parse_cfmws(struct acpi_cedt_cfmws *cfmws,
 
 	cxld = &cxlrd->cxlsd.cxld;
 	cxld->flags = cfmws_to_decoder_flags(cfmws->restrictions);
-	cxld->target_type = CXL_DECODER_HOSTONLYMEM;
+	if (cxld->flags & CXL_DECODER_F_TYPE2)
+		cxld->target_type = CXL_DECODER_DEVMEM;
+	else
+		cxld->target_type = CXL_DECODER_HOSTONLYMEM;
 	cxld->hpa_range = (struct range) {
 		.start = cfmws->base_hpa,
 		.end = cfmws->base_hpa + cfmws->window_size - 1,
Signature · 1 line
2.39.2
author
#6 lore raw

PATCH 2/3 cxl: Set target type of region with that of root decoder

In reply to Huang Ying

Now, the target type of region is hard-coded to HOSTONLYMEM, because only type3 expanders are supported. To support type2 accelerators, set the target type of region root decoder with that of the root decoder.

Cc Davidlohr Bueso <[email protected]>
Cc Jonathan Cameron <[email protected]>
Cc Dave Jiang <[email protected]>
Cc Alison Schofield <[email protected]>
Cc Vishal Verma <[email protected]>
Cc Ira Weiny <[email protected]>
Cc Alejandro Lucero <[email protected]>
---
 drivers/cxl/core/region.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/cxl/core/region.c +2 -1
index 21ad5f242875..9a483c8a32fd 100644
@@ -2545,7 +2545,8 @@ static struct cxl_region *__create_region(struct cxl_root_decoder *cxlrd,
 		return ERR_PTR(-EBUSY);
 	}
 
-	return devm_cxl_add_region(cxlrd, id, mode, CXL_DECODER_HOSTONLYMEM);
+	return devm_cxl_add_region(cxlrd, id, mode,
+				   cxlrd->cxlsd.cxld.target_type);
 }
 
 static ssize_t create_pmem_region_store(struct device *dev,
Signature · 1 line
2.39.2
maintainer
#7 lore raw

Re: [PATCH 2/3] cxl: Set target type of region with that of root decoder

Quoted 4 lines · Ying Huang

Now, the target type of region is hard-coded to HOSTONLYMEM, because only type3 expanders are supported. To support type2 accelerators, set the target type of region root decoder with that of the root decoder.

Hi Ying,

If the target type of a region is always the same as it's root decoder, (is it?) why do we store it as an attribute of the region. Can we look it up when needed?

A bit more below -

Quoted 19 lines
Cc Davidlohr Bueso <[email protected]>
Cc Jonathan Cameron <[email protected]>
Cc Dave Jiang <[email protected]>
Cc Alison Schofield <[email protected]>
Cc Vishal Verma <[email protected]>
Cc Ira Weiny <[email protected]>
Cc Alejandro Lucero <[email protected]>
---
 drivers/cxl/core/region.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/cxl/core/region.c +0 -0
index 21ad5f242875..9a483c8a32fd 100644

drivers/cxl/core/region.c static struct cxl_region *__create_region(struct cxl_root_decoder *cxlrd, · line 2550

Quoted 7 lines
drivers/cxl/core/region.c +2 -1
@@ -2545,7 +2545,8 @@ static struct cxl_region *__create_region(struct cxl_root_decoder *cxlrd,
 		return ERR_PTR(-EBUSY);
 	}
 
-	return devm_cxl_add_region(cxlrd, id, mode, CXL_DECODER_HOSTONLYMEM);
+	return devm_cxl_add_region(cxlrd, id, mode,
+				   cxlrd->cxlsd.cxld.target_type);
 }

Passing the 'cxlrd' and then a piece of the cxlrd (.target_type) looks redundant.

-- Alison

Quoted 5 lines
drivers/cxl/core/region.c +0 -0
 static ssize_t create_pmem_region_store(struct device *dev,
Signature · 1 line
2.39.2
#8 lore raw

Re: [PATCH 2/3] cxl: Set target type of region with that of root decoder

Quoted 10 lines · Alison Schofield
Quoted 4 lines · Ying Huang

Now, the target type of region is hard-coded to HOSTONLYMEM, because only type3 expanders are supported. To support type2 accelerators, set the target type of region root decoder with that of the root decoder.

Hi Ying,

If the target type of a region is always the same as it's root decoder, (is it?)

IIUC, it is. Do you know when they may be different?

Quoted 2 lines

why do we store it as an attribute of the region. Can we look it up when needed?

Yes. This is possible via to_cxl_root_decoder(). It's just a little inconvenient.

Quoted 24 lines

A bit more below -

Quoted 22 lines
Cc Davidlohr Bueso <[email protected]>
Cc Jonathan Cameron <[email protected]>
Cc Dave Jiang <[email protected]>
Cc Alison Schofield <[email protected]>
Cc Vishal Verma <[email protected]>
Cc Ira Weiny <[email protected]>
Cc Alejandro Lucero <[email protected]>
---
 drivers/cxl/core/region.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/cxl/core/region.c +0 -0
index 21ad5f242875..9a483c8a32fd 100644
@@ -2545,7 +2545,8 @@ static struct cxl_region *__create_region(struct cxl_root_decoder *cxlrd,
 		return ERR_PTR(-EBUSY);
 	}
 
Quoted 7 lines

drivers/cxl/core/region.c static struct cxl_region *__create_region(struct cxl_root_decoder *cxlrd, · line 2550

Quoted 4 lines
drivers/cxl/core/region.c +2 -1
-	return devm_cxl_add_region(cxlrd, id, mode, CXL_DECODER_HOSTONLYMEM);
+	return devm_cxl_add_region(cxlrd, id, mode,
+				   cxlrd->cxlsd.cxld.target_type);
 }

Passing the 'cxlrd' and then a piece of the cxlrd (.target_type) looks redundant.

Yes. We can remove the parameter. Will change this if we still need cxlr->type. Thanks!

--
Best Regards,
Huang, Ying

Quoted 8 lines

-- Alison

Quoted 5 lines
drivers/cxl/core/region.c +0 -0
 static ssize_t create_pmem_region_store(struct device *dev,
Signature · 1 line
2.39.2
#9 lore raw

Re: [PATCH 2/3] cxl: Set target type of region with that of root decoder

Quoted 8 lines · Ying
Quoted 6 lines · Alison Schofield
Quoted 4 lines · Ying Huang

Now, the target type of region is hard-coded to HOSTONLYMEM, because only type3 expanders are supported. To support type2 accelerators, set the target type of region root decoder with that of the root decoder.

Quoted 6 lines
Quoted 4 lines

Hi Ying,

If the target type of a region is always the same as it's root decoder, (is it?)

IIUC, it is. Do you know when they may be different?

Root decoder (CFMW I think) allows both and target device only one.

More likely when it's HDM-DB / HDM-H though I think than HDM-D / HDM-H. A host would do this because it has simple address routing (maybe a single root complex) and doesn't want to pay the HPA space cost of providing separate regions, so decisions on protocol etc derived from RC HDM decoder programming, not the fixed bit in in the host.

Note for those not so buried in CXL terms, HDM-DB includes back invalidate memory devices (for P2P or hardware coherent sharing), HDM-H is simpler memory devices that don't support back invalidate.

Jonathan

Quoted 56 lines
Quoted 3 lines

why do we store it as an attribute of the region. Can we look it up when needed?

Yes. This is possible via to_cxl_root_decoder(). It's just a little inconvenient.

Quoted 24 lines

A bit more below -

Quoted 22 lines
Cc Davidlohr Bueso <[email protected]>
Cc Jonathan Cameron <[email protected]>
Cc Dave Jiang <[email protected]>
Cc Alison Schofield <[email protected]>
Cc Vishal Verma <[email protected]>
Cc Ira Weiny <[email protected]>
Cc Alejandro Lucero <[email protected]>
---
 drivers/cxl/core/region.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/cxl/core/region.c +0 -0
index 21ad5f242875..9a483c8a32fd 100644
@@ -2545,7 +2545,8 @@ static struct cxl_region *__create_region(struct cxl_root_decoder *cxlrd,
 		return ERR_PTR(-EBUSY);
 	}
 
Quoted 7 lines

drivers/cxl/core/region.c static struct cxl_region *__create_region(struct cxl_root_decoder *cxlrd, · line 2550

Quoted 4 lines
drivers/cxl/core/region.c +2 -1
-	return devm_cxl_add_region(cxlrd, id, mode, CXL_DECODER_HOSTONLYMEM);
+	return devm_cxl_add_region(cxlrd, id, mode,
+				   cxlrd->cxlsd.cxld.target_type);
 }  

Passing the 'cxlrd' and then a piece of the cxlrd (.target_type) looks redundant.

Yes. We can remove the parameter. Will change this if we still need cxlr->type. Thanks!

--
Best Regards,
Huang, Ying

Quoted 8 lines

-- Alison

Quoted 5 lines
drivers/cxl/core/region.c +0 -0
 static ssize_t create_pmem_region_store(struct device *dev,
Signature · 1 line
2.39.2
Reviewed-by on v1 2/3
#10 lore raw

Re: [PATCH 2/3] cxl: Set target type of region with that of root decoder

In reply to Huang Ying
Quoted 14 lines · Huang Ying

Now, the target type of region is hard-coded to HOSTONLYMEM, because only type3 expanders are supported. To support type2 accelerators, set the target type of region root decoder with that of the root decoder.

Cc Davidlohr Bueso <[email protected]>
Cc Jonathan Cameron <[email protected]>
Cc Dave Jiang <[email protected]>
Cc Alison Schofield <[email protected]>
Cc Vishal Verma <[email protected]>
Cc Ira Weiny <[email protected]>
Cc Alejandro Lucero <[email protected]>
Quoted 1 line

---

Quoted 20 lines
 drivers/cxl/core/region.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/cxl/core/region.c +2 -1
index 21ad5f242875..9a483c8a32fd 100644
@@ -2545,7 +2545,8 @@ static struct cxl_region *__create_region(struct cxl_root_decoder *cxlrd,
 		return ERR_PTR(-EBUSY);
 	}
 
-	return devm_cxl_add_region(cxlrd, id, mode, CXL_DECODER_HOSTONLYMEM);
+	return devm_cxl_add_region(cxlrd, id, mode,
+				   cxlrd->cxlsd.cxld.target_type);
 }
 
 static ssize_t create_pmem_region_store(struct device *dev,
Signature · 1 line
2.39.2
author
#11 lore raw

PATCH 3/3 cxl: Avoid to create dax regions for type2 accelerators

In reply to Huang Ying

The memory range of a type2 accelerator should be managed by the type2 accelerator specific driver instead of the common dax region drivers, as discussed in [1].

[1] https://lore.kernel.org/linux-cxl/[email protected]/

So, in this patch, we skip dax regions creation for type2 accelerator device memory regions.

Based on: https://lore.kernel.org/linux-cxl/168592159835.1948938.1647215579839222774.stgit@dwillia2-xfh.jf.intel.com/

Cc Davidlohr Bueso <[email protected]>
Cc Jonathan Cameron <[email protected]>
Cc Dave Jiang <[email protected]>
Cc Alison Schofield <[email protected]>
Cc Vishal Verma <[email protected]>
Cc Ira Weiny <[email protected]>
Cc Alejandro Lucero <[email protected]>
---
 drivers/cxl/core/region.c | 8 ++++++++
 1 file changed, 8 insertions(+)
drivers/cxl/core/region.c +8 -0
index 9a483c8a32fd..b37e12bb4a35 100644
@@ -3435,6 +3435,14 @@ static int cxl_region_probe(struct device *dev)
 					p->res->start, p->res->end, cxlr,
 					is_system_ram) > 0)
 			return 0;
+		/*
+		 * HDM-D[B] (device-memory) regions have accelerator
+		 * specific usage, skip device-dax registration.
+		 */
+		if (cxlr->type == CXL_DECODER_DEVMEM)
+			return 0;
+
+		/* HDM-H routes to device-dax */
 		return devm_cxl_add_dax_region(cxlr);
 	default:
 		dev_dbg(&cxlr->dev, "unsupported region mode: %d\n",
Signature · 1 line
2.39.2
#12 lore raw

Re: [PATCH 3/3] cxl: Avoid to create dax regions for type2 accelerators

Quoted 30 lines · On Mon, 29 Jul 2024 16:46:11 +0800 Huang Ying <[email protected]> wrote:

The memory range of a type2 accelerator should be managed by the type2 accelerator specific driver instead of the common dax region drivers, as discussed in [1].

[1] https://lore.kernel.org/linux-cxl/[email protected]/

So, in this patch, we skip dax regions creation for type2 accelerator device memory regions.

Based on: https://lore.kernel.org/linux-cxl/168592159835.1948938.1647215579839222774.stgit@dwillia2-xfh.jf.intel.com/

Cc Davidlohr Bueso <[email protected]>
Cc Jonathan Cameron <[email protected]>
Cc Dave Jiang <[email protected]>
Cc Alison Schofield <[email protected]>
Cc Vishal Verma <[email protected]>
Cc Ira Weiny <[email protected]>
Cc Alejandro Lucero <[email protected]>
---
 drivers/cxl/core/region.c | 8 ++++++++
 1 file changed, 8 insertions(+)
drivers/cxl/core/region.c +0 -0
index 9a483c8a32fd..b37e12bb4a35 100644

drivers/cxl/core/region.c static int cxl_region_probe(struct device *dev) · line 3443

Quoted 9 lines
drivers/cxl/core/region.c +6 -0
@@ -3435,6 +3435,14 @@ static int cxl_region_probe(struct device *dev)
 					p->res->start, p->res->end, cxlr,
 					is_system_ram) > 0)
 			return 0;
+		/*
+		 * HDM-D[B] (device-memory) regions have accelerator
+		 * specific usage, skip device-dax registration.
+		 */
+		if (cxlr->type == CXL_DECODER_DEVMEM)
+			return 0;

As in previous need to be careful as that may not mean it's an accelerator.

However, we do need to deal with BI setup for HDM-DB type 3 devices etc and to check the HDM Decoder capability registers to make sure Supported Coherence model is appropriate. (e.g. 11 for host only or device coherency - HDM-H/HDM-DB)

Quoted 5 lines
drivers/cxl/core/region.c +2 -0
+
+		/* HDM-H routes to device-dax */
 		return devm_cxl_add_dax_region(cxlr);
 	default:
 		dev_dbg(&cxlr->dev, "unsupported region mode: %d\n",
#13 lore raw

Re: [PATCH 3/3] cxl: Avoid to create dax regions for type2 accelerators

Quoted 37 lines · Jonathan Cameron
Quoted 34 lines · On Mon, 29 Jul 2024 16:46:11 +0800 Huang Ying <[email protected]> wrote:

The memory range of a type2 accelerator should be managed by the type2 accelerator specific driver instead of the common dax region drivers, as discussed in [1].

[1] https://lore.kernel.org/linux-cxl/[email protected]/

So, in this patch, we skip dax regions creation for type2 accelerator device memory regions.

Based on: https://lore.kernel.org/linux-cxl/168592159835.1948938.1647215579839222774.stgit@dwillia2-xfh.jf.intel.com/

Cc Davidlohr Bueso <[email protected]>
Cc Jonathan Cameron <[email protected]>
Cc Dave Jiang <[email protected]>
Cc Alison Schofield <[email protected]>
Cc Vishal Verma <[email protected]>
Cc Ira Weiny <[email protected]>
Cc Alejandro Lucero <[email protected]>
---
 drivers/cxl/core/region.c | 8 ++++++++
 1 file changed, 8 insertions(+)
drivers/cxl/core/region.c +1 -0
index 9a483c8a32fd..b37e12bb4a35 100644
@@ -3435,6 +3435,14 @@ static int cxl_region_probe(struct device *dev)
 					p->res->start, p->res->end, cxlr,
 					is_system_ram) > 0)
 			return 0;
+		/*
Quoted 8 lines

drivers/cxl/core/region.c static int cxl_region_probe(struct device *dev) · line 3443

Quoted 5 lines
drivers/cxl/core/region.c +5 -0
+		 * HDM-D[B] (device-memory) regions have accelerator
+		 * specific usage, skip device-dax registration.
+		 */
+		if (cxlr->type == CXL_DECODER_DEVMEM)
+			return 0;

As in previous need to be careful as that may not mean it's an accelerator.

Yes. We need some other way to identify type2 devices.

Quoted 4 lines

However, we do need to deal with BI setup for HDM-DB type 3 devices etc and to check the HDM Decoder capability registers to make sure Supported Coherence model is appropriate. (e.g. 11 for host only or device coherency - HDM-H/HDM-DB)

Yes. We need to check BI configuration too.

Quoted 5 lines

drivers/cxl/core/region.c static int cxl_region_probe(struct device *dev) · line 3448

Quoted 5 lines
drivers/cxl/core/region.c +2 -0
+
+		/* HDM-H routes to device-dax */
 		return devm_cxl_add_dax_region(cxlr);
 	default:
 		dev_dbg(&cxlr->dev, "unsupported region mode: %d\n",

--
Best Regards,
Huang, Ying

#14 lore raw

Re: [PATCH 3/3] cxl: Avoid to create dax regions for type2 accelerators

I do not understand why you took this change from my patchset.

Maybe the other two patches have a pass, but this should not be removed from my patchset, not at least after discussing it publicly.

The reason you mentioned for doing it was for making things easier for the other changes in my larger patchset, but again, you should have discussed this first publicly, and second, I do not remember other large patchsets, far larger than mine, being partially picked up  by someone else but the patchset submitter, and sending those picked changes in another patchset.

Quoted 62 lines · Ying
Quoted 28 lines · Jonathan Cameron
Quoted 25 lines · On Mon, 29 Jul 2024 16:46:11 +0800 Huang Ying <[email protected]> wrote:

The memory range of a type2 accelerator should be managed by the type2 accelerator specific driver instead of the common dax region drivers, as discussed in [1].

[1] https://lore.kernel.org/linux-cxl/[email protected]/

So, in this patch, we skip dax regions creation for type2 accelerator device memory regions.

Based on: https://lore.kernel.org/linux-cxl/168592159835.1948938.1647215579839222774.stgit@dwillia2-xfh.jf.intel.com/

Cc Davidlohr Bueso <[email protected]>
Cc Jonathan Cameron <[email protected]>
Cc Dave Jiang <[email protected]>
Cc Alison Schofield <[email protected]>
Cc Vishal Verma <[email protected]>
Cc Ira Weiny <[email protected]>
Cc Alejandro Lucero <[email protected]>
---
  drivers/cxl/core/region.c | 8 ++++++++
  1 file changed, 8 insertions(+)
Quoted 16 lines
Quoted 5 lines
drivers/cxl/core/region.c +0 -0
index 9a483c8a32fd..b37e12bb4a35 100644

drivers/cxl/core/region.c static int cxl_region_probe(struct device *dev) · line 3443

Quoted 9 lines
drivers/cxl/core/region.c +6 -0
@@ -3435,6 +3435,14 @@ static int cxl_region_probe(struct device *dev)
  					p->res->start, p->res->end, cxlr,
  					is_system_ram) > 0)
  			return 0;
+		/*
+		 * HDM-D[B] (device-memory) regions have accelerator
+		 * specific usage, skip device-dax registration.
+		 */
+		if (cxlr->type == CXL_DECODER_DEVMEM)
+			return 0;

As in previous need to be careful as that may not mean it's an accelerator.

Yes. We need some other way to identify type2 devices.

Quoted 4 lines

However, we do need to deal with BI setup for HDM-DB type 3 devices etc and to check the HDM Decoder capability registers to make sure Supported Coherence model is appropriate. (e.g. 11 for host only or device coherency - HDM-H/HDM-DB)

Yes. We need to check BI configuration too.

Quoted 5 lines

drivers/cxl/core/region.c static int cxl_region_probe(struct device *dev) · line 3448

Quoted 5 lines
drivers/cxl/core/region.c +2 -0
+
+		/* HDM-H routes to device-dax */
  		return devm_cxl_add_dax_region(cxlr);
  	default:
  		dev_dbg(&cxlr->dev, "unsupported region mode: %d\n",

--
Best Regards,
Huang, Ying

#15 lore raw

Re: [PATCH 3/3] cxl: Avoid to create dax regions for type2 accelerators

In reply to Huang, Ying
Quoted 46 lines · Ying
Quoted 28 lines · Jonathan Cameron
Quoted 25 lines · On Mon, 29 Jul 2024 16:46:11 +0800 Huang Ying <[email protected]> wrote:

The memory range of a type2 accelerator should be managed by the type2 accelerator specific driver instead of the common dax region drivers, as discussed in [1].

[1] https://lore.kernel.org/linux-cxl/[email protected]/

So, in this patch, we skip dax regions creation for type2 accelerator device memory regions.

Based on: https://lore.kernel.org/linux-cxl/168592159835.1948938.1647215579839222774.stgit@dwillia2-xfh.jf.intel.com/

Cc Davidlohr Bueso <[email protected]>
Cc Jonathan Cameron <[email protected]>
Cc Dave Jiang <[email protected]>
Cc Alison Schofield <[email protected]>
Cc Vishal Verma <[email protected]>
Cc Ira Weiny <[email protected]>
Cc Alejandro Lucero <[email protected]>
---
  drivers/cxl/core/region.c | 8 ++++++++
  1 file changed, 8 insertions(+)
Quoted 16 lines
Quoted 5 lines
drivers/cxl/core/region.c +0 -0
index 9a483c8a32fd..b37e12bb4a35 100644

drivers/cxl/core/region.c static int cxl_region_probe(struct device *dev) · line 3443

Quoted 9 lines
drivers/cxl/core/region.c +6 -0
@@ -3435,6 +3435,14 @@ static int cxl_region_probe(struct device *dev)
  					p->res->start, p->res->end, cxlr,
  					is_system_ram) > 0)
  			return 0;
+		/*
+		 * HDM-D[B] (device-memory) regions have accelerator
+		 * specific usage, skip device-dax registration.
+		 */
+		if (cxlr->type == CXL_DECODER_DEVMEM)
+			return 0;

As in previous need to be careful as that may not mean it's an accelerator.

Quoted 1 line

Yes. We need some other way to identify type2 devices.

Maybe the easier option is the accel driver specifying if DAX should be used.

We are adding mailbox and hdm as optional for accel/type2, with the driver specifying what is supported. Another optional param could be this DAX usage.

Quoted 14 lines
Quoted 4 lines

However, we do need to deal with BI setup for HDM-DB type 3 devices etc and to check the HDM Decoder capability registers to make sure Supported Coherence model is appropriate. (e.g. 11 for host only or device coherency - HDM-H/HDM-DB)

Yes. We need to check BI configuration too.

Quoted 5 lines

drivers/cxl/core/region.c static int cxl_region_probe(struct device *dev) · line 3448

Quoted 5 lines
drivers/cxl/core/region.c +2 -0
+
+		/* HDM-H routes to device-dax */
  		return devm_cxl_add_dax_region(cxlr);
  	default:
  		dev_dbg(&cxlr->dev, "unsupported region mode: %d\n",

--
Best Regards,
Huang, Ying

#16 lore raw

Re: [PATCH 3/3] cxl: Avoid to create dax regions for type2 accelerators

Quoted 50 lines · Alejandro Lucero Palau
Quoted 46 lines · Ying
Quoted 28 lines · Jonathan Cameron
Quoted 25 lines · On Mon, 29 Jul 2024 16:46:11 +0800 Huang Ying <[email protected]> wrote:

The memory range of a type2 accelerator should be managed by the type2 accelerator specific driver instead of the common dax region drivers, as discussed in [1].

[1] https://lore.kernel.org/linux-cxl/[email protected]/

So, in this patch, we skip dax regions creation for type2 accelerator device memory regions.

Based on: https://lore.kernel.org/linux-cxl/168592159835.1948938.1647215579839222774.stgit@dwillia2-xfh.jf.intel.com/

Cc Davidlohr Bueso <[email protected]>
Cc Jonathan Cameron <[email protected]>
Cc Dave Jiang <[email protected]>
Cc Alison Schofield <[email protected]>
Cc Vishal Verma <[email protected]>
Cc Ira Weiny <[email protected]>
Cc Alejandro Lucero <[email protected]>
---
  drivers/cxl/core/region.c | 8 ++++++++
  1 file changed, 8 insertions(+)
Quoted 16 lines
Quoted 5 lines
drivers/cxl/core/region.c +0 -0
index 9a483c8a32fd..b37e12bb4a35 100644

drivers/cxl/core/region.c static int cxl_region_probe(struct device *dev) · line 3443

Quoted 9 lines
drivers/cxl/core/region.c +6 -0
@@ -3435,6 +3435,14 @@ static int cxl_region_probe(struct device *dev)
  					p->res->start, p->res->end, cxlr,
  					is_system_ram) > 0)
  			return 0;
+		/*
+		 * HDM-D[B] (device-memory) regions have accelerator
+		 * specific usage, skip device-dax registration.
+		 */
+		if (cxlr->type == CXL_DECODER_DEVMEM)
+			return 0;

As in previous need to be careful as that may not mean it's an accelerator.

Quoted 1 line

Yes. We need some other way to identify type2 devices.

Quoted 6 lines

Maybe the easier option is the accel driver specifying if DAX should be used.

We are adding mailbox and hdm as optional for accel/type2, with the driver specifying what is supported. Another optional param could be this DAX usage.

Another way is let accel/cxl_pci driver specify the device type (type2 vs. type3). cxl_pci_probe()->cxl_memdev_state_create() will set cxl_dev_state.type = CXL_DEVTYPE_CLASSMEM. And accel_X_probe()->cxl_accel_state_create() can set cxl_dev_state.type = CXL_DEVTYPE_DEVMEM. This can be passed to cxl_region. Then we can create cxl_dax_region for type3 devices only by default.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

However, we do need to deal with BI setup for HDM-DB type 3 devices etc and to check the HDM Decoder capability registers to make sure Supported Coherence model is appropriate. (e.g. 11 for host only or device coherency - HDM-H/HDM-DB)

Quoted 7 lines
Quoted 7 lines

Yes. We need to check BI configuration too.

Quoted 5 lines

drivers/cxl/core/region.c static int cxl_region_probe(struct device *dev) · line 3448

Quoted 5 lines
drivers/cxl/core/region.c +2 -0
+
+		/* HDM-H routes to device-dax */
  		return devm_cxl_add_dax_region(cxlr);
  	default:
  		dev_dbg(&cxlr->dev, "unsupported region mode: %d\n",

--
Best Regards,
Huang, Ying

#17 lore raw
In reply to Huang Ying

Hi,

I'm a bit "surprised" by this patchset. As you rightly say in this cover letter, there is a patchset under review, and I have not seen you commenting there on the concerns stated in this cover letter.

I would say that is the first thing you should do, at least, to comment there, suggest to do other way, pointing to other needed changes, and if things do not go well for reaching an agreement, then a patchset like this could make sense exposing another way.

Anyway, I think a CXL maintainer should say something about it, but after 24hours, I had to say something.

About testing these changes, I wonder how did you proceed. If you have used an emulated Type2 device, as I did with the first patchset version, you should trigger some of the problems I found, what makes any Type2 initialization for getting a memdev and finally a cxl region to fail. In other words, testing these changes can not be a partly initialised Type2 but a complete one. This does not mean a patch fixing a known and obvious issue for supporting Type2 should not be approved if not tested, since the Type2 support is not yet there, but mentioning testing makes things confusing, at least for me.

Quoted 26 lines · Huang Ying

There have been 2 series to add type 2 accelerators support in [1] and [2].

[1] https://lore.kernel.org/linux-cxl/168592149709.1948938.8663425987110396027.stgit@dwillia2-xfh.jf.intel.com/
[2] https://lore.kernel.org/linux-cxl/[email protected]/

Both provide relative complete support, but both are long too. To make it easier to review, some preparation of type2 accelerators support is implemented in this series. More complete support can be implemented based on it.

This series has been tested with cxl_test via mocking a type2 accelerator as in [1] above. Because more type2 accelerators support than that provided by this series is needed to simulate the device, the cxl_test patch isn't included in the series.

Huang Ying (3):
cxl: Set target type of root decoder based on CFMWS restrictions
cxl: Set target type of region with that of root decoder
cxl: Avoid to create dax regions for type2 accelerators

  drivers/cxl/acpi.c        |  5 ++++-
  drivers/cxl/core/region.c | 11 ++++++++++-
  2 files changed, 14 insertions(+), 2 deletions(-)

Best Regards,
Huang, Ying

#18 lore raw

Hi, Alejandro,

Quoted 8 lines · Alejandro Lucero Palau

Hi,

I'm a bit "surprised" by this patchset. As you rightly say in this cover letter, there is a patchset under review, and I have not seen you commenting there on the concerns stated in this cover letter.

Quoted 4 lines

I would say that is the first thing you should do, at least, to comment there, suggest to do other way, pointing to other needed changes, and if things do not go well for reaching an agreement, then a patchset like this could make sense exposing another way.

Sorry for confusing. The patchset is just some preparation work for type2 accelerator support. Your patchset is more complete. I just want to relief your overhead a bit.

Quoted 4 lines

Anyway, I think a CXL maintainer should say something about it, but after 24hours, I had to say something.

Quoted 9 lines

About testing these changes, I wonder how did you proceed. If you have used an emulated Type2 device, as I did with the first patchset version, you should trigger some of the problems I found, what makes any Type2 initialization for getting a memdev and finally a cxl region to fail. In other words, testing these changes can not be a partly initialised Type2 but a complete one. This does not mean a patch fixing a known and obvious issue for supporting Type2 should not be approved if not tested, since the Type2 support is not yet there, but mentioning testing makes things confusing, at least for me.

Yes. To test this patchset, I need more complete type2 support. I used some patches from your and Dan's patchset on top of this patchset to do the test.

Quoted 17 lines · Huang Ying
Quoted 15 lines

There have been 2 series to add type 2 accelerators support in [1] and [2].

[1] https://lore.kernel.org/linux-cxl/168592149709.1948938.8663425987110396027.stgit@dwillia2-xfh.jf.intel.com/
[2] https://lore.kernel.org/linux-cxl/[email protected]/

Both provide relative complete support, but both are long too. To make it easier to review, some preparation of type2 accelerators support is implemented in this series. More complete support can be implemented based on it.

This series has been tested with cxl_test via mocking a type2 accelerator as in [1] above. Because more type2 accelerators support than that provided by this series is needed to simulate the device, the cxl_test patch isn't included in the series.

Quoted 8 lines
Quoted 8 lines

Huang Ying (3):
cxl: Set target type of root decoder based on CFMWS restrictions
cxl: Set target type of region with that of root decoder
cxl: Avoid to create dax regions for type2 accelerators

  drivers/cxl/acpi.c        |  5 ++++-
  drivers/cxl/core/region.c | 11 ++++++++++-
  2 files changed, 14 insertions(+), 2 deletions(-)

--
Best Regards,
Huang, Ying