Skip to contentSkip to conversation

Harden HDM decoder enumeration

PATCH Superseded Alison Schofield 4 participants 12 messages Jul 17 – Aug 7 CXL

v3 of 5 Superseded Latest is v5 →

lore
Share & tools
lore

Alison Schofield

author maintainer
#0 lore raw

v3 cover Harden HDM decoder enumeration

Changes in v3:
P1: Move zero-target reject to init_hdm_decoder() (RichardC)
P3: Reset commit_end via a devres action after decoder teardown (RichardC)

Changes in v2:
P1: Handle uncommitted decoders (Sashiko)
P1: Reject switch decoders with zero targets (Sashiko)
P1: Add in code comment clarifying target_count (Sashiko)
P1: Update commit log
P3: Reset commit_end on the failure path only (Sashiko)
P3: Update commit log
No changes in P2

Here are 3 Sashiko reported cleanups.
Only the first includes a link to the complaint, since the others did not originally appear on Lore for me to add public links.

Srirangan: I believe the first 2 of these also address pre-existing issues Sashiko posted in reply to your reset series, Patch 02/11.

Alison Schofield (3):
cxl/hdm: Reject switch decoder interleave ways that overflow targets cxl/hdm: Make switch decoder target parsing endian-safe cxl/hdm: Restore commit_end when decoder enumeration fails

 drivers/cxl/core/hdm.c | 65 ++++++++++++++++++++++++++++++++++++------
 drivers/cxl/cxl.h      |  2 ++
 2 files changed, 58 insertions(+), 9 deletions(-)

base-commit: dc59e4fea9d83f03bad6bddf3fa2e52491777482

Signature · 1 line
2.37.3
author maintainer
#1 lore raw

PATCH v3 1/3 cxl/hdm: Reject switch decoder interleave ways that overflow targets

Switch decoder enumeration validates that the interleave ways encoding is legal, but not that the resulting number of ways fits the available targets. This can overrun the target arrays during enumeration.

Reject committed decoders whose interleave ways exceed either the hardware target list capacity or the reported target count. Reject switch decoders that report zero targets.

For uncommitted decoders, ignore the stale interleave ways value and reset it to one until the decoder is committed.

Add a clarifying comment that target_count is a direct count, not 0-based like decoder_count.

Fixes d17d0540a0db ("cxl/core/hdm: Add CXL standard decoder enumeration to the core")
---
 drivers/cxl/core/hdm.c | 33 +++++++++++++++++++++++++++++++++
 drivers/cxl/cxl.h      |  2 ++
 2 files changed, 35 insertions(+)
drivers/cxl/core/hdm.c +33 -0
index 0c80b76a5f9b..d81df45d8005 100644
@@ -76,6 +76,8 @@ static void parse_hdm_decoder_caps(struct cxl_hdm *cxlhdm)
 
 	hdm_cap = readl(cxlhdm->regs.hdm_decoder + CXL_HDM_DECODER_CAP_OFFSET);
 	cxlhdm->decoder_count = cxl_hdm_decoder_count(hdm_cap);
+
+	/* target_count is a direct count (1h..8h), not 0-based like decoder_count */
 	cxlhdm->target_count =
 		FIELD_GET(CXL_HDM_DECODER_TARGET_COUNT_MASK, hdm_cap);
 	if (FIELD_GET(CXL_HDM_DECODER_INTERLEAVE_11_8, hdm_cap))
@@ -1084,6 +1086,37 @@ static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld,
 		cxld->interleave_ways, cxld->interleave_granularity);
 
 	if (!cxled) {
+		struct cxl_switch_decoder *cxlsd =
+			to_cxl_switch_decoder(&cxld->dev);
+
+		if (cxlsd->nr_targets < 1) {
+			dev_err(&port->dev,
+				"decoder%d.%d: reports zero targets\n",
+				port->id, cxld->id);
+			return -ENXIO;
+		}
+
+		if (!committed) {
+			/* Ignore interleave ways until commit */
+			cxld->interleave_ways = 1;
+			return 0;
+		}
+
+		if (cxld->interleave_ways > CXL_HDM_DECODER0_TL_TARGETS) {
+			dev_err(&port->dev,
+				"decoder%d.%d: interleave ways: %d exceeds target list capacity: %d\n",
+				port->id, cxld->id, cxld->interleave_ways,
+				CXL_HDM_DECODER0_TL_TARGETS);
+			return -ENXIO;
+		}
+		if (cxld->interleave_ways > cxlsd->nr_targets) {
+			dev_err(&port->dev,
+				"decoder%d.%d: interleave ways: %d exceeds targets: %d\n",
+				port->id, cxld->id, cxld->interleave_ways,
+				cxlsd->nr_targets);
+			return -ENXIO;
+		}
+
 		lo = readl(hdm + CXL_HDM_DECODER0_TL_LOW(which));
 		hi = readl(hdm + CXL_HDM_DECODER0_TL_HIGH(which));
 		target_list.value = (hi << 32) + lo;
drivers/cxl/cxl.h +2 -0
index c0e5308e4d1b..291ada46b646 100644
@@ -67,6 +67,8 @@ extern const struct nvdimm_security_ops *cxl_security_ops;
 #define   CXL_HDM_DECODER0_CTRL_HOSTONLY BIT(12)
 #define CXL_HDM_DECODER0_TL_LOW(i) (0x20 * (i) + 0x24)
 #define CXL_HDM_DECODER0_TL_HIGH(i) (0x20 * (i) + 0x28)
+/* Two registers with one target ID per byte */
+#define CXL_HDM_DECODER0_TL_TARGETS 8
 #define CXL_HDM_DECODER0_SKIP_LOW(i) CXL_HDM_DECODER0_TL_LOW(i)
 #define CXL_HDM_DECODER0_SKIP_HIGH(i) CXL_HDM_DECODER0_TL_HIGH(i)
 
Signature · 1 line
2.37.3
#2 lore raw

Re: [PATCH v3 1/3] cxl/hdm: Reject switch decoder interleave ways that overflow targets

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Pre-existing issues:
- [High] The target_list union is not endian-safe, causing corrupted target mapping on big-endian architectures. --

--- Patch [1]: [PATCH v3 1/3] cxl/hdm: Reject switch decoder interleave ways that overflow targets --- commit 4abc7b385aec9c1d79e37f3577a9009b393c2333
Author: Alison Schofield <[email protected]>

cxl/hdm: Reject switch decoder interleave ways that overflow targets

This patch validates switch decoder enumeration to ensure the interleave ways encoding fits the available targets. It prevents overrunning the target arrays during enumeration and handles uncommitted decoders by ignoring stale interleave ways.

Quoted 4 lines
drivers/cxl/core/hdm.c +0 -0
@@ -1084,6 +1086,37 @@ static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld,

[ ... ]

drivers/cxl/core/hdm.c static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld, · line 1122

Quoted 11 lines
drivers/cxl/core/hdm.c +8 -0
+		if (cxld->interleave_ways > cxlsd->nr_targets) {
+			dev_err(&port->dev,
+				"decoder%d.%d: interleave ways: %d exceeds targets: %d\n",
+				port->id, cxld->id, cxld->interleave_ways,
+				cxlsd->nr_targets);
+			return -ENXIO;
+		}
+
 		lo = readl(hdm + CXL_HDM_DECODER0_TL_LOW(which));
 		hi = readl(hdm + CXL_HDM_DECODER0_TL_HIGH(which));
 		target_list.value = (hi << 32) + lo;

[Severity: High]
This is a pre-existing issue, but does this union overlay create an implicit memory dependency on the CPU's byte order?

In init_hdm_decoder(), the target mapping extracts bytes via the target_id array based on memory layout:

    target_list.value = (hi << 32) + lo;
    for (i = 0; i < cxld->interleave_ways; i++)
        cxld->target_map[i] = target_list.target_id[i];

On big-endian architectures, byte 0 of the 64-bit integer corresponds to the most significant byte (target 7) instead of the least significant byte (target 0).

Does this completely reverse the target mapping, potentially routing memory transactions to the wrong CXL targets and leading to memory corruption?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
reviewer
#3 lore raw

Re: [PATCH v3 1/3] cxl/hdm: Reject switch decoder interleave ways that overflow targets

In reply to Alison Schofield
Quoted 44 lines · Alison Schofield

Switch decoder enumeration validates that the interleave ways encoding is legal, but not that the resulting number of ways fits the available targets. This can overrun the target arrays during enumeration.

Reject committed decoders whose interleave ways exceed either the hardware target list capacity or the reported target count. Reject switch decoders that report zero targets.

For uncommitted decoders, ignore the stale interleave ways value and reset it to one until the decoder is committed.

Add a clarifying comment that target_count is a direct count, not 0-based like decoder_count.

Fixes d17d0540a0db ("cxl/core/hdm: Add CXL standard decoder enumeration to the core")
---
  drivers/cxl/core/hdm.c | 33 +++++++++++++++++++++++++++++++++
  drivers/cxl/cxl.h      |  2 ++
  2 files changed, 35 insertions(+)
drivers/cxl/core/hdm.c +7 -0
index 0c80b76a5f9b..d81df45d8005 100644
@@ -76,6 +76,8 @@ static void parse_hdm_decoder_caps(struct cxl_hdm *cxlhdm)
 
  	hdm_cap = readl(cxlhdm->regs.hdm_decoder + CXL_HDM_DECODER_CAP_OFFSET);
  	cxlhdm->decoder_count = cxl_hdm_decoder_count(hdm_cap);
+
+	/* target_count is a direct count (1h..8h), not 0-based like decoder_count */
  	cxlhdm->target_count =
  		FIELD_GET(CXL_HDM_DECODER_TARGET_COUNT_MASK, hdm_cap);
  	if (FIELD_GET(CXL_HDM_DECODER_INTERLEAVE_11_8, hdm_cap))
@@ -1084,6 +1086,37 @@ static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld,
  		cxld->interleave_ways, cxld->interleave_granularity);
 
  	if (!cxled) {
+		struct cxl_switch_decoder *cxlsd =
+			to_cxl_switch_decoder(&cxld->dev);
+
+		if (cxlsd->nr_targets < 1) {
+			dev_err(&port->dev,

drivers/cxl/core/hdm.c static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld, · line 1103

Quoted 10 lines
drivers/cxl/core/hdm.c +10 -0
+				"decoder%d.%d: reports zero targets\n",
+				port->id, cxld->id);
+			return -ENXIO;
+		}
+
+		if (!committed) {
+			/* Ignore interleave ways until commit */
+			cxld->interleave_ways = 1;
+			return 0;
+		}

Current init_hdm_decoder() reads IW and IG from CTRL register whatever a decoder is enabled or not, just wondering any reason for that. Because my understanding is when CXL driver programs a decoder, IW and IG will be rewritten, so is it not needed to read IW and IG from the register if a decoder is not enabled?

Quoted 32 lines
drivers/cxl/core/hdm.c +16 -0
+
+		if (cxld->interleave_ways > CXL_HDM_DECODER0_TL_TARGETS) {
+			dev_err(&port->dev,
+				"decoder%d.%d: interleave ways: %d exceeds target list capacity: %d\n",
+				port->id, cxld->id, cxld->interleave_ways,
+				CXL_HDM_DECODER0_TL_TARGETS);
+			return -ENXIO;
+		}
+		if (cxld->interleave_ways > cxlsd->nr_targets) {
+			dev_err(&port->dev,
+				"decoder%d.%d: interleave ways: %d exceeds targets: %d\n",
+				port->id, cxld->id, cxld->interleave_ways,
+				cxlsd->nr_targets);
+			return -ENXIO;
+		}
+
  		lo = readl(hdm + CXL_HDM_DECODER0_TL_LOW(which));
  		hi = readl(hdm + CXL_HDM_DECODER0_TL_HIGH(which));
  		target_list.value = (hi << 32) + lo;
drivers/cxl/cxl.h +2 -0
index c0e5308e4d1b..291ada46b646 100644
@@ -67,6 +67,8 @@ extern const struct nvdimm_security_ops *cxl_security_ops;
  #define   CXL_HDM_DECODER0_CTRL_HOSTONLY BIT(12)
  #define CXL_HDM_DECODER0_TL_LOW(i) (0x20 * (i) + 0x24)
  #define CXL_HDM_DECODER0_TL_HIGH(i) (0x20 * (i) + 0x28)
+/* Two registers with one target ID per byte */
+#define CXL_HDM_DECODER0_TL_TARGETS 8
  #define CXL_HDM_DECODER0_SKIP_LOW(i) CXL_HDM_DECODER0_TL_LOW(i)
  #define CXL_HDM_DECODER0_SKIP_HIGH(i) CXL_HDM_DECODER0_TL_HIGH(i)
 
author maintainer
#4 lore raw

Re: [PATCH v3 1/3] cxl/hdm: Reject switch decoder interleave ways that overflow targets

Quoted 51 lines · Li Ming
Quoted 49 lines · Alison Schofield

Switch decoder enumeration validates that the interleave ways encoding is legal, but not that the resulting number of ways fits the available targets. This can overrun the target arrays during enumeration.

Reject committed decoders whose interleave ways exceed either the hardware target list capacity or the reported target count. Reject switch decoders that report zero targets.

For uncommitted decoders, ignore the stale interleave ways value and reset it to one until the decoder is committed.

Add a clarifying comment that target_count is a direct count, not 0-based like decoder_count.

Fixes d17d0540a0db ("cxl/core/hdm: Add CXL standard decoder enumeration to the core")
---
  drivers/cxl/core/hdm.c | 33 +++++++++++++++++++++++++++++++++
  drivers/cxl/cxl.h      |  2 ++
  2 files changed, 35 insertions(+)
drivers/cxl/core/hdm.c +14 -0
index 0c80b76a5f9b..d81df45d8005 100644
@@ -76,6 +76,8 @@ static void parse_hdm_decoder_caps(struct cxl_hdm *cxlhdm)
  	hdm_cap = readl(cxlhdm->regs.hdm_decoder + CXL_HDM_DECODER_CAP_OFFSET);
  	cxlhdm->decoder_count = cxl_hdm_decoder_count(hdm_cap);
+
+	/* target_count is a direct count (1h..8h), not 0-based like decoder_count */
  	cxlhdm->target_count =
  		FIELD_GET(CXL_HDM_DECODER_TARGET_COUNT_MASK, hdm_cap);
  	if (FIELD_GET(CXL_HDM_DECODER_INTERLEAVE_11_8, hdm_cap))
@@ -1084,6 +1086,37 @@ static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld,
  		cxld->interleave_ways, cxld->interleave_granularity);
  	if (!cxled) {
+		struct cxl_switch_decoder *cxlsd =
+			to_cxl_switch_decoder(&cxld->dev);
+
+		if (cxlsd->nr_targets < 1) {
+			dev_err(&port->dev,
+				"decoder%d.%d: reports zero targets\n",
+				port->id, cxld->id);
+			return -ENXIO;
+		}
+
+		if (!committed) {
+			/* Ignore interleave ways until commit */
Quoted 8 lines

drivers/cxl/core/hdm.c static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld, · line 1103

Quoted 3 lines
drivers/cxl/core/hdm.c +3 -0
+			cxld->interleave_ways = 1;
+			return 0;
+		}

Current init_hdm_decoder() reads IW and IG from CTRL register whatever a decoder is enabled or not, just wondering any reason for that. Because my understanding is when CXL driver programs a decoder, IW and IG will be rewritten, so is it not needed to read IW and IG from the register if a decoder is not enabled?

Hi Ming,

Great catch! It's not only needless but stale bits could cause the whole enumeration through eiw_to,eig_to....to return -EINVAL.

I added a patch 2 to address this (another pre existing issue).

I do see that you used the word 'read' and I wanted to note that IW and IG are field of the ctrl register and that has to be read once to find the committed bit. So there is not separate read to drop. What the new patch stops doing is decoding and validating those fields.

Take a look please. Thanks,
-- Alison

maintainer
#5 lore raw

Re: [PATCH v3 1/3] cxl/hdm: Reject switch decoder interleave ways that overflow targets

In reply to Alison Schofield
Quoted 10 lines · On Fri, 17 Jul 2026 14:42:46 -0700 Alison Schofield…

Switch decoder enumeration validates that the interleave ways encoding is legal, but not that the resulting number of ways fits the available targets. This can overrun the target arrays during enumeration.

Reject committed decoders whose interleave ways exceed either the hardware target list capacity or the reported target count. Reject switch decoders that report zero targets.

For uncommitted decoders, ignore the stale interleave ways value and reset it to one until the decoder is committed.

Note on this below. Why 1? If it is invalid can we use a nonsense value like 0 so that any use of it will cause an obvious bug and make it easy to spot? (hopefully)

Quoted 26 lines

Add a clarifying comment that target_count is a direct count, not 0-based like decoder_count.

Fixes d17d0540a0db ("cxl/core/hdm: Add CXL standard decoder enumeration to the core")
---
 drivers/cxl/core/hdm.c | 33 +++++++++++++++++++++++++++++++++
 drivers/cxl/cxl.h      |  2 ++
 2 files changed, 35 insertions(+)
drivers/cxl/core/hdm.c +2 -0
index 0c80b76a5f9b..d81df45d8005 100644
@@ -76,6 +76,8 @@ static void parse_hdm_decoder_caps(struct cxl_hdm *cxlhdm)
 
 	hdm_cap = readl(cxlhdm->regs.hdm_decoder + CXL_HDM_DECODER_CAP_OFFSET);
 	cxlhdm->decoder_count = cxl_hdm_decoder_count(hdm_cap);
+
+	/* target_count is a direct count (1h..8h), not 0-based like decoder_count */
 	cxlhdm->target_count =
 		FIELD_GET(CXL_HDM_DECODER_TARGET_COUNT_MASK, hdm_cap);
 	if (FIELD_GET(CXL_HDM_DECODER_INTERLEAVE_11_8, hdm_cap))

drivers/cxl/core/hdm.c static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld, · line 1090

Quoted 5 lines
drivers/cxl/core/hdm.c +2 -0
@@ -1084,6 +1086,37 @@ static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld,
 		cxld->interleave_ways, cxld->interleave_granularity);
 
 	if (!cxled) {
+		struct cxl_switch_decoder *cxlsd =
+			to_cxl_switch_decoder(&cxld->dev);

Trivial but I'd go long and have that on one line. It's only just over 80 chars!

drivers/cxl/core/hdm.c static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld, · line 1100

Quoted 10 lines
drivers/cxl/core/hdm.c +10 -0
+
+		if (cxlsd->nr_targets < 1) {
+			dev_err(&port->dev,
+				"decoder%d.%d: reports zero targets\n",
+				port->id, cxld->id);
+			return -ENXIO;
+		}
+
+		if (!committed) {
+			/* Ignore interleave ways until commit */

Dumb question but why is = 1 equivalent to ignore? Can we make it explicit either by gating at all use of interleave_ways, or using an invalid value to mean we don't care yet. I like 0 as that's clearly nonsense for interleave ways.

drivers/cxl/core/hdm.c static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld, · line 1105

Quoted 5 lines
drivers/cxl/core/hdm.c +5 -0
+			cxld->interleave_ways = 1;
+			return 0;
+		}
+
+		if (cxld->interleave_ways > CXL_HDM_DECODER0_TL_TARGETS) {

This feels a bit belt and braces. Should we be verifying that nr_targets meets this constraint before setting it instead and only having the below test here?

Quoted 22 lines
drivers/cxl/core/hdm.c +14 -0
+			dev_err(&port->dev,
+				"decoder%d.%d: interleave ways: %d exceeds target list capacity: %d\n",
+				port->id, cxld->id, cxld->interleave_ways,
+				CXL_HDM_DECODER0_TL_TARGETS);
+			return -ENXIO;
+		}
+		if (cxld->interleave_ways > cxlsd->nr_targets) {
+			dev_err(&port->dev,
+				"decoder%d.%d: interleave ways: %d exceeds targets: %d\n",
+				port->id, cxld->id, cxld->interleave_ways,
+				cxlsd->nr_targets);
+			return -ENXIO;
+		}
+
 		lo = readl(hdm + CXL_HDM_DECODER0_TL_LOW(which));
 		hi = readl(hdm + CXL_HDM_DECODER0_TL_HIGH(which));
 		target_list.value = (hi << 32) + lo;
drivers/cxl/cxl.h +0 -0
index c0e5308e4d1b..291ada46b646 100644

drivers/cxl/cxl.h extern const struct nvdimm_security_ops *cxl_security_ops; · line 71

Quoted 5 lines
drivers/cxl/cxl.h +2 -0
@@ -67,6 +67,8 @@ extern const struct nvdimm_security_ops *cxl_security_ops;
 #define   CXL_HDM_DECODER0_CTRL_HOSTONLY BIT(12)
 #define CXL_HDM_DECODER0_TL_LOW(i) (0x20 * (i) + 0x24)
 #define CXL_HDM_DECODER0_TL_HIGH(i) (0x20 * (i) + 0x28)
+/* Two registers with one target ID per byte */
+#define CXL_HDM_DECODER0_TL_TARGETS 8

Whilst the comment is true, perhaps better to refer to IW definition and 8 being max allowed for anything that isn't a end point.

Maybe the spec would provide 2 registers and then only allow 6 decoders :) Also comment doesn't say how big registers are. Anyhow, to me the IW reference is crisper.

Quoted 3 lines
drivers/cxl/cxl.h +0 -0
 #define CXL_HDM_DECODER0_SKIP_LOW(i) CXL_HDM_DECODER0_TL_LOW(i)
 #define CXL_HDM_DECODER0_SKIP_HIGH(i) CXL_HDM_DECODER0_TL_HIGH(i)
 
author maintainer
#6 lore raw

Re: [PATCH v3 1/3] cxl/hdm: Reject switch decoder interleave ways that overflow targets

Quoted 7 lines · Jonathan Cameron
Quoted 4 lines · On Fri, 17 Jul 2026 14:42:46 -0700 Alison Schofield…

Switch decoder enumeration validates that the interleave ways encoding is legal, but not that the resulting number of ways fits the available targets. This can overrun the target arrays during enumeration.

Quoted 6 lines
Quoted 6 lines

Reject committed decoders whose interleave ways exceed either the hardware target list capacity or the reported target count. Reject switch decoders that report zero targets.

For uncommitted decoders, ignore the stale interleave ways value and reset it to one until the decoder is committed.

Thanks for the review Jonathan. I'm responding as I'm about to send v5, with all this considered and fixed up.

Quoted 4 lines

Note on this below. Why 1? If it is invalid can we use a nonsense value like 0 so that any use of it will cause an obvious bug and make it easy to spot? (hopefully)

replied below

Quoted 29 lines
Quoted 29 lines

Add a clarifying comment that target_count is a direct count, not 0-based like decoder_count.

Fixes d17d0540a0db ("cxl/core/hdm: Add CXL standard decoder enumeration to the core")
---
 drivers/cxl/core/hdm.c | 33 +++++++++++++++++++++++++++++++++
 drivers/cxl/cxl.h      |  2 ++
 2 files changed, 35 insertions(+)
drivers/cxl/core/hdm.c +2 -0
index 0c80b76a5f9b..d81df45d8005 100644
@@ -76,6 +76,8 @@ static void parse_hdm_decoder_caps(struct cxl_hdm *cxlhdm)
 
 	hdm_cap = readl(cxlhdm->regs.hdm_decoder + CXL_HDM_DECODER_CAP_OFFSET);
 	cxlhdm->decoder_count = cxl_hdm_decoder_count(hdm_cap);
+
+	/* target_count is a direct count (1h..8h), not 0-based like decoder_count */
 	cxlhdm->target_count =
 		FIELD_GET(CXL_HDM_DECODER_TARGET_COUNT_MASK, hdm_cap);
 	if (FIELD_GET(CXL_HDM_DECODER_INTERLEAVE_11_8, hdm_cap))
@@ -1084,6 +1086,37 @@ static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld,
 		cxld->interleave_ways, cxld->interleave_granularity);
 
Quoted 5 lines

drivers/cxl/core/hdm.c static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld, · line 1090

Quoted 3 lines
drivers/cxl/core/hdm.c +2 -0
 	if (!cxled) {
+		struct cxl_switch_decoder *cxlsd =
+			to_cxl_switch_decoder(&cxld->dev);

Trivial but I'd go long and have that on one line. It's only just over 80 chars!

Done.

Quoted 1 line

snip

Quoted 9 lines

drivers/cxl/core/hdm.c static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld, · line 1100

Quoted 3 lines
drivers/cxl/core/hdm.c +3 -0
+
+		if (!committed) {
+			/* Ignore interleave ways until commit */

Dumb question but why is = 1 equivalent to ignore? Can we make it explicit either by gating at all use of interleave_ways, or using an invalid value to mean we don't care yet. I like 0 as that's clearly nonsense for interleave ways.

Quoted 3 lines

drivers/cxl/core/hdm.c static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld, · line 1103

Quoted 3 lines
drivers/cxl/core/hdm.c +3 -0
+			cxld->interleave_ways = 1;
+			return 0;
+		}

0 is no usable because cxl_decoder_acc_locked() reject ways < 1, so every uncommitted switch decoder would fail to add. And 1 was wrong for a different reason that Davidlohr caught on v4/patch 2 - ways is a sysfs attribute that reports what the register field holds, so changing that changes what usersspace sees. In v5 we leave the uncommitted read path alone.

Quoted 6 lines

drivers/cxl/core/hdm.c

Quoted 2 lines
drivers/cxl/core/hdm.c +2 -0
+
+		if (cxld->interleave_ways > CXL_HDM_DECODER0_TL_TARGETS) {

This feels a bit belt and braces. Should we be verifying that nr_targets meets this constraint before setting it instead and only having the below test here?

Yes, done in v5. The target count is now bounded where it is established rather than at each use, so...
- devm_cxl_enumerate_decoders() rejects an HDM capability whose reported target count is outside 1..CXL_HDM_DECODER0_TL_TARGETS, once per port.
- cxl_switch_decoder_init() rejects nr_targets < 1 (Ming asked for that one).

init_hdm_decoder() is left with the ways > nr_targets test only.

snip

Quoted 5 lines
Quoted 5 lines
drivers/cxl/cxl.h +0 -0
index c0e5308e4d1b..291ada46b646 100644
@@ -67,6 +67,8 @@ extern const struct nvdimm_security_ops *cxl_security_ops;
Quoted 8 lines

drivers/cxl/cxl.h extern const struct nvdimm_security_ops *cxl_security_ops; · line 71

Quoted 5 lines
drivers/cxl/cxl.h +2 -0
 #define   CXL_HDM_DECODER0_CTRL_HOSTONLY BIT(12)
 #define CXL_HDM_DECODER0_TL_LOW(i) (0x20 * (i) + 0x24)
 #define CXL_HDM_DECODER0_TL_HIGH(i) (0x20 * (i) + 0x28)
+/* Two registers with one target ID per byte */
+#define CXL_HDM_DECODER0_TL_TARGETS 8

Whilst the comment is true, perhaps better to refer to IW definition and 8 being max allowed for anything that isn't a end point.

Done like this:

	/* Target list capacity, i.e. max interleave ways for a non-endpoint decoder */
        #define CXL_HDM_DECODER0_TL_TARGETS 8
Quoted 9 lines

Maybe the spec would provide 2 registers and then only allow 6 decoders :) Also comment doesn't say how big registers are. Anyhow, to me the IW reference is crisper.

Quoted 3 lines
drivers/cxl/cxl.h +0 -0
 #define CXL_HDM_DECODER0_SKIP_LOW(i) CXL_HDM_DECODER0_TL_LOW(i)
 #define CXL_HDM_DECODER0_SKIP_HIGH(i) CXL_HDM_DECODER0_TL_HIGH(i)
 
author maintainer
#7 lore raw

PATCH v3 2/3 cxl/hdm: Make switch decoder target parsing endian-safe

In reply to Alison Schofield

Switch decoder target IDs are stored one per byte, starting with target 0 in the least significant byte. The current code accesses those bytes in memory order, which only matches register order on little-endian hosts.

On big-endian hosts, the target IDs are reversed and the decoder is associated with the wrong downstream ports.

Extract each target ID from the register value with an explicit shift and mask so the target mapping is independent of host endianness.

Fixes d17d0540a0db ("cxl/core/hdm: Add CXL standard decoder enumeration to the core")
---
 drivers/cxl/core/hdm.c | 9 +++------
 1 file changed, 3 insertions(+), 6 deletions(-)
drivers/cxl/core/hdm.c +3 -6
index d81df45d8005..4aaadb842d90 100644
@@ -976,14 +976,11 @@ static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld,
 {
 	struct cxl_endpoint_decoder *cxled = NULL;
 	u64 size, base, skip, dpa_size, lo, hi;
+	u64 target_list;
 	bool committed;
 	u32 remainder;
 	int i, rc;
 	u32 ctrl;
-	union {
-		u64 value;
-		unsigned char target_id[8];
-	} target_list;
 
 	if (should_emulate_decoders(info))
 		return cxl_setup_hdm_decoder_from_dvsec(port, cxld, dpa_base,
@@ -1119,9 +1116,9 @@ static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld,
 
 		lo = readl(hdm + CXL_HDM_DECODER0_TL_LOW(which));
 		hi = readl(hdm + CXL_HDM_DECODER0_TL_HIGH(which));
-		target_list.value = (hi << 32) + lo;
+		target_list = (hi << 32) + lo;
 		for (i = 0; i < cxld->interleave_ways; i++)
-			cxld->target_map[i] = target_list.target_id[i];
+			cxld->target_map[i] = (target_list >> (i * 8)) & 0xff;
 
 		return 0;
 	}
Signature · 1 line
2.37.3
reviewer Reviewed-by on v3 2/3
#8 lore raw

Re: [PATCH v3 2/3] cxl/hdm: Make switch decoder target parsing endian-safe

Quoted 8 lines · Alison Schofield

Switch decoder target IDs are stored one per byte, starting with target 0 in the least significant byte. The current code accesses those bytes in memory order, which only matches register order on little-endian hosts.

On big-endian hosts, the target IDs are reversed and the decoder is associated with the wrong downstream ports.

Quoted 6 lines

Extract each target ID from the register value with an explicit shift and mask so the target mapping is independent of host endianness.

Fixes d17d0540a0db ("cxl/core/hdm: Add CXL standard decoder enumeration to the core")
Quoted 36 lines
---
  drivers/cxl/core/hdm.c | 9 +++------
  1 file changed, 3 insertions(+), 6 deletions(-)
drivers/cxl/core/hdm.c +3 -6
index d81df45d8005..4aaadb842d90 100644
@@ -976,14 +976,11 @@ static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld,
  {
  	struct cxl_endpoint_decoder *cxled = NULL;
  	u64 size, base, skip, dpa_size, lo, hi;
+	u64 target_list;
  	bool committed;
  	u32 remainder;
  	int i, rc;
  	u32 ctrl;
-	union {
-		u64 value;
-		unsigned char target_id[8];
-	} target_list;
 
  	if (should_emulate_decoders(info))
  		return cxl_setup_hdm_decoder_from_dvsec(port, cxld, dpa_base,
@@ -1119,9 +1116,9 @@ static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld,
 
  		lo = readl(hdm + CXL_HDM_DECODER0_TL_LOW(which));
  		hi = readl(hdm + CXL_HDM_DECODER0_TL_HIGH(which));
-		target_list.value = (hi << 32) + lo;
+		target_list = (hi << 32) + lo;
  		for (i = 0; i < cxld->interleave_ways; i++)
-			cxld->target_map[i] = target_list.target_id[i];
+			cxld->target_map[i] = (target_list >> (i * 8)) & 0xff;
 
  		return 0;
  	}
author maintainer
#9 lore raw

PATCH v3 3/3 cxl/hdm: Restore commit_end when decoder enumeration fails

In reply to Alison Schofield

commit_end tracks the highest committed decoder on a port. It gets advanced before decoder enumeration is complete, so a later failure leaves it pointing at a decoder that was never added. The next probe then rejects decoder0 as out of order and enumeration fails.

Reset commit_end to the none committed baseline once the decoders are torn down, so a subsequent probe rebuilds it from scratch. Register the reset before the decoders are added so that on unwind it runs after every decoder has been unregistered. This prevents commit_end from dropping below the highest committed decoder still registered and exposing an inconsistent value to a concurrent reader.

Protect the commit_end advance in the DVSEC emulation path against concurrent readers, matching the register-programmed path.

Fixes 176baefb2eb5 ("cxl/hdm: Commit decoder state to hardware")
Fixes b777e9bec960 ("cxl/hdm: Emulate HDM decoder from DVSEC range registers")
---
 drivers/cxl/core/hdm.c | 23 ++++++++++++++++++++---
 1 file changed, 20 insertions(+), 3 deletions(-)
drivers/cxl/core/hdm.c +20 -3
index 4aaadb842d90..6dc6e53c28c6 100644
@@ -955,7 +955,8 @@ static int cxl_setup_hdm_decoder_from_dvsec(
 	 * change the range registers at run time.
 	 */
 	cxld->flags |= CXL_DECODER_F_ENABLE | CXL_DECODER_F_LOCK;
-	port->commit_end = cxld->id;
+	scoped_guard(rwsem_write, &cxl_rwsem.region)
+		port->commit_end = cxld->id;
 
 	rc = devm_cxl_dpa_reserve(cxled, *dpa_base, len, 0);
 	if (rc) {
@@ -1183,18 +1184,34 @@ static void cxl_settle_decoders(struct cxl_hdm *cxlhdm)
  * @cxlhdm: Structure to populate with HDM capabilities
  * @info: cached DVSEC range register info
  */
+static void cxl_reset_commit_end(void *data)
+{
+	struct cxl_port *port = data;
+
+	guard(rwsem_write)(&cxl_rwsem.region);
+	port->commit_end = -1;
+}
+
 static int devm_cxl_enumerate_decoders(struct cxl_hdm *cxlhdm,
 				       struct cxl_endpoint_dvsec_info *info)
 {
 	void __iomem *hdm = cxlhdm->regs.hdm_decoder;
 	struct cxl_port *port = cxlhdm->port;
-	int i;
 	u64 dpa_base = 0;
+	int i, rc;
 
 	cxl_settle_decoders(cxlhdm);
 
+	/*
+	 * Reset commit_end after all decoders have been torn down so a
+	 * subsequent probe rebuilds it from scratch.
+	 */
+	rc = devm_add_action_or_reset(&port->dev, cxl_reset_commit_end, port);
+	if (rc)
+		return rc;
+
 	for (i = 0; i < cxlhdm->decoder_count; i++) {
-		int rc, target_count = cxlhdm->target_count;
+		int target_count = cxlhdm->target_count;
 		struct cxl_decoder *cxld;
 
 		if (is_cxl_endpoint(port)) {
Signature · 1 line
2.37.3
reviewer
#10 lore raw

Re: [PATCH v3 3/3] cxl/hdm: Restore commit_end when decoder enumeration fails

Quoted 56 lines · Alison Schofield

commit_end tracks the highest committed decoder on a port. It gets advanced before decoder enumeration is complete, so a later failure leaves it pointing at a decoder that was never added. The next probe then rejects decoder0 as out of order and enumeration fails.

Reset commit_end to the none committed baseline once the decoders are torn down, so a subsequent probe rebuilds it from scratch. Register the reset before the decoders are added so that on unwind it runs after every decoder has been unregistered. This prevents commit_end from dropping below the highest committed decoder still registered and exposing an inconsistent value to a concurrent reader.

Protect the commit_end advance in the DVSEC emulation path against concurrent readers, matching the register-programmed path.

Fixes 176baefb2eb5 ("cxl/hdm: Commit decoder state to hardware")
Fixes b777e9bec960 ("cxl/hdm: Emulate HDM decoder from DVSEC range registers")
---
  drivers/cxl/core/hdm.c | 23 ++++++++++++++++++++---
  1 file changed, 20 insertions(+), 3 deletions(-)
drivers/cxl/core/hdm.c +11 -2
index 4aaadb842d90..6dc6e53c28c6 100644
@@ -955,7 +955,8 @@ static int cxl_setup_hdm_decoder_from_dvsec(
  	 * change the range registers at run time.
  	 */
  	cxld->flags |= CXL_DECODER_F_ENABLE | CXL_DECODER_F_LOCK;
-	port->commit_end = cxld->id;
+	scoped_guard(rwsem_write, &cxl_rwsem.region)
+		port->commit_end = cxld->id;
 
  	rc = devm_cxl_dpa_reserve(cxled, *dpa_base, len, 0);
  	if (rc) {
@@ -1183,18 +1184,34 @@ static void cxl_settle_decoders(struct cxl_hdm *cxlhdm)
   * @cxlhdm: Structure to populate with HDM capabilities
   * @info: cached DVSEC range register info
   */
+static void cxl_reset_commit_end(void *data)
+{
+	struct cxl_port *port = data;
+
+	guard(rwsem_write)(&cxl_rwsem.region);
+	port->commit_end = -1;
+}
+
  static int devm_cxl_enumerate_decoders(struct cxl_hdm *cxlhdm,
  				       struct cxl_endpoint_dvsec_info *info)
  {
  	void __iomem *hdm = cxlhdm->regs.hdm_decoder;
  	struct cxl_port *port = cxlhdm->port;
-	int i;
  	u64 dpa_base = 0;
+	int i, rc;

drivers/cxl/core/hdm.c static void cxl_settle_decoders(struct cxl_hdm *cxlhdm) · line 1211

Quoted 10 lines
drivers/cxl/core/hdm.c +7 -0
 
  	cxl_settle_decoders(cxlhdm);
 
+	/*
+	 * Reset commit_end after all decoders have been torn down so a
+	 * subsequent probe rebuilds it from scratch.
+	 */
+	rc = devm_add_action_or_reset(&port->dev, cxl_reset_commit_end, port);
+	if (rc)
+		return rc;

Seems like devm_add_action() is enough here. port->commit_end is -1 already.

Quoted 7 lines
drivers/cxl/core/hdm.c +2 -1
+
  	for (i = 0; i < cxlhdm->decoder_count; i++) {
-		int rc, target_count = cxlhdm->target_count;
+		int target_count = cxlhdm->target_count;
  		struct cxl_decoder *cxld;
 
  		if (is_cxl_endpoint(port)) {
author maintainer
#11 lore raw

Re: [PATCH v3 3/3] cxl/hdm: Restore commit_end when decoder enumeration fails

Quoted 59 lines · Li Ming
Quoted 57 lines · Alison Schofield

commit_end tracks the highest committed decoder on a port. It gets advanced before decoder enumeration is complete, so a later failure leaves it pointing at a decoder that was never added. The next probe then rejects decoder0 as out of order and enumeration fails.

Reset commit_end to the none committed baseline once the decoders are torn down, so a subsequent probe rebuilds it from scratch. Register the reset before the decoders are added so that on unwind it runs after every decoder has been unregistered. This prevents commit_end from dropping below the highest committed decoder still registered and exposing an inconsistent value to a concurrent reader.

Protect the commit_end advance in the DVSEC emulation path against concurrent readers, matching the register-programmed path.

Fixes 176baefb2eb5 ("cxl/hdm: Commit decoder state to hardware")
Fixes b777e9bec960 ("cxl/hdm: Emulate HDM decoder from DVSEC range registers")
---
  drivers/cxl/core/hdm.c | 23 ++++++++++++++++++++---
  1 file changed, 20 insertions(+), 3 deletions(-)
drivers/cxl/core/hdm.c +12 -2
index 4aaadb842d90..6dc6e53c28c6 100644
@@ -955,7 +955,8 @@ static int cxl_setup_hdm_decoder_from_dvsec(
  	 * change the range registers at run time.
  	 */
  	cxld->flags |= CXL_DECODER_F_ENABLE | CXL_DECODER_F_LOCK;
-	port->commit_end = cxld->id;
+	scoped_guard(rwsem_write, &cxl_rwsem.region)
+		port->commit_end = cxld->id;
  	rc = devm_cxl_dpa_reserve(cxled, *dpa_base, len, 0);
  	if (rc) {
@@ -1183,18 +1184,34 @@ static void cxl_settle_decoders(struct cxl_hdm *cxlhdm)
   * @cxlhdm: Structure to populate with HDM capabilities
   * @info: cached DVSEC range register info
   */
+static void cxl_reset_commit_end(void *data)
+{
+	struct cxl_port *port = data;
+
+	guard(rwsem_write)(&cxl_rwsem.region);
+	port->commit_end = -1;
+}
+
  static int devm_cxl_enumerate_decoders(struct cxl_hdm *cxlhdm,
  				       struct cxl_endpoint_dvsec_info *info)
  {
  	void __iomem *hdm = cxlhdm->regs.hdm_decoder;
  	struct cxl_port *port = cxlhdm->port;
-	int i;
  	u64 dpa_base = 0;
+	int i, rc;
  	cxl_settle_decoders(cxlhdm);
+	/*
Quoted 8 lines

drivers/cxl/core/hdm.c static void cxl_settle_decoders(struct cxl_hdm *cxlhdm) · line 1211

Quoted 6 lines
drivers/cxl/core/hdm.c +6 -0
+	 * Reset commit_end after all decoders have been torn down so a
+	 * subsequent probe rebuilds it from scratch.
+	 */
+	rc = devm_add_action_or_reset(&port->dev, cxl_reset_commit_end, port);
+	if (rc)
+		return rc;

Seems like devm_add_action() is enough here. port->commit_end is -1 already.

Agree. The reset on fail would only write -1 over -1. Changed in v4. Please take a look.

-- Alison