Skip to contentSkip to conversation

cxl/region: Restore passthrough decoder enable on region re-assembly

PATCH In review Richard Cheng 3 participants 8 messages Jul 27 – Aug 5 CXL

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 cxl/region: Restore passthrough decoder enable on region re-assembly

After a CXL endpoint is PCI hot-removed and the bus rescanned, an auto-discovered region fails to rebiuld and is lost, evne though the device's HDM decoder is still committed and decoding.

A single-dport host bridge/root port has no HDM decoder capability, so its switch decoder is a SW-only passthrough. Its CXL_DECODER_F_ENABLE flag is cleared on region teardown and never restored on rescan, so cxl_port_setup_targets() fails with -ENXIO.

Re-enable the passthrough decoder when its interleave and HPA config still match the region, it holds no HW state.

---
 drivers/cxl/core/region.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)
drivers/cxl/core/region.c +13 -0
index 1e211542b6b6..011ceb5bae79 100644
@@ -1574,6 +1574,19 @@ static int cxl_port_setup_targets(struct cxl_port *port,
 	}
 
 	if (test_bit(CXL_REGION_F_AUTO, &cxlr->flags)) {
+		/*
+		 * A passthrough switch decoder holds no HW decode state.
+		 * It's CXL_DECODER_F_ENABLE flag is pure software bookkeeping
+		 * that is cleared when the region is torn down. On auto-discovery
+		 * re-assembly after a subsequent rescan the decode config still
+		 * matches the region, so restore the flag rather than fail to
+		 * rebuild a region that HW is in fact still decoding.
+		 */
+		if (!cxld->commit && cxld->interleave_ways == iw &&
+		    (iw <= 1 || cxld->interleave_granularity == ig) &&
+		    spa_maps_hpa(p, &cxld->hpa_range))
+			cxld->flags |= CXL_DECODER_F_ENABLE;
+
 		if (cxld->interleave_ways != iw ||
 		    (iw > 1 && cxld->interleave_granularity != ig) ||
 		    !spa_maps_hpa(p, &cxld->hpa_range) ||

base-commit: 4539944e515183668109bdf4d0c3d7d228383d88

Signature · 1 line
2.43.0
maintainer
#1 lore raw
Quoted 2 lines · Richard Cheng

After a CXL endpoint is PCI hot-removed and the bus rescanned, an auto-discovered region fails to rebiuld and is lost, evne though the

rebuild
even

Quoted 1 line

device's HDM decoder is still committed and decoding.

If we truly do a PCI hot-remove and re-insert, the endpoint device would be reset and disabled. Not sure how it would still be decoding. Also, a complete different device with possibly different size can be inserted. And the device showed up would be unconfigured. Given there's no BIOS to program the device since the OS has taken over, should it be still considered part of the auto-region?

Any thoughts Jonathan?

Quoted 40 lines

A single-dport host bridge/root port has no HDM decoder capability, so its switch decoder is a SW-only passthrough. Its CXL_DECODER_F_ENABLE flag is cleared on region teardown and never restored on rescan, so cxl_port_setup_targets() fails with -ENXIO.

Re-enable the passthrough decoder when its interleave and HPA config still match the region, it holds no HW state.

---
 drivers/cxl/core/region.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)
drivers/cxl/core/region.c +13 -0
index 1e211542b6b6..011ceb5bae79 100644
@@ -1574,6 +1574,19 @@ static int cxl_port_setup_targets(struct cxl_port *port,
 	}
 
 	if (test_bit(CXL_REGION_F_AUTO, &cxlr->flags)) {
+		/*
+		 * A passthrough switch decoder holds no HW decode state.
+		 * It's CXL_DECODER_F_ENABLE flag is pure software bookkeeping
+		 * that is cleared when the region is torn down. On auto-discovery
+		 * re-assembly after a subsequent rescan the decode config still
+		 * matches the region, so restore the flag rather than fail to
+		 * rebuild a region that HW is in fact still decoding.
+		 */
+		if (!cxld->commit && cxld->interleave_ways == iw &&
+		    (iw <= 1 || cxld->interleave_granularity == ig) &&
+		    spa_maps_hpa(p, &cxld->hpa_range))
+			cxld->flags |= CXL_DECODER_F_ENABLE;
+
 		if (cxld->interleave_ways != iw ||
 		    (iw > 1 && cxld->interleave_granularity != ig) ||
 		    !spa_maps_hpa(p, &cxld->hpa_range) ||

base-commit: 4539944e515183668109bdf4d0c3d7d228383d88

#2 lore raw
Quoted 9 lines · Dave Jiang
Quoted 2 lines · Richard Cheng

After a CXL endpoint is PCI hot-removed and the bus rescanned, an auto-discovered region fails to rebiuld and is lost, evne though the

rebuild
even

Hi Dave,
Thanks for the review, I'll fix the typo in v2.

Quoted 2 lines
Quoted 1 line

device's HDM decoder is still committed and decoding.

I think this part is talking about physical remove/insert ? sorry maybe I commit message was too vague about the scenario. The case here is a SW-only teardown via sysfs, this is what I did.

"""

$ echo 1 > /sys/bus/pci/devices/$BDF/remove
$ echo 1 > /sys/bus/pci/devices/$BDF/rescan
"""

No physical removal, no power cycle, no link-down, no reset. Linux drops and re-enumerates the same device, which never stopped running. The endpoint HDM decoder is still committed.

The device is byte-identical before and after, the kernel doesn't clear it either, the decoder is locked and cxl_decoder_reset() returns early for CXL_DECODER_F_LOCK before touching any register. The memory keeps decoding.

I don't think anything needs re-programming here? the only thing lost is kernel-side bookkeeping on the passthrough decoder, which is freed with port and reallocated with F_ENABLE clear on rescan. The endpoint recovers its state from HW.
The passthrough decoder has no HW to recover from, that asymmetry is the bug.

Quoted 2 lines

Also, a complete different device with possibly different size can be inserted. And the device showed up would be unconfigured. Given there's no BIOS to program the device since the OS has taken over, should it be still considered part of the auto-region?

Agreed that would be wrong, and I don't think this patch allows it. A different or freshly-inserted device fails the existing endpoint checks, its own decoder won't come back COMMITTED with a matching HPA range, so cxl_add_to_region() won't re-assemble the auto-region relardless of what the passthrough decoder's flag says. This patch guards on the decode config still matching (iw, ig, spa_maps_hap()), and on !cxld->commit, so it can only ever touch a SW-only stub.

So the coverage here is SW-only, I should state it more clearly in v2.

Btw, the check "!cxld->commit" is doing a lot of implicit work in the code base, I think that's worth fixing regardless of the patch. "commit == NULL" is not unique to passthrough decoders, cxl_setup_hdm_decoder_from_dvsec() also sets "commit = NULL" for DVSEC-emulated RCD endpoint decoders, and root decoder never set it. My call happens to be safe, but that isn't visible from the condition.

I would suggest an inline helper like the following, do you think it's reasonable ?

"""
static inline bool cxl_decoder_is_passthrough(struct cxl_decoder *cxld)

{
    if (cxld->commit)
        return false;
    if (!is_switch_decoder(&cxld->dev) || is_root_decoder(&cxld->dev))
        return false;
    return to_cxl_switch_decoder(&cxld->dev)->nr_targets <= 1;
}

"""

There're about 4 call site today, and it distinguishes "passthrough" from "has no commit routine" , which are currently the same check for different things.

Happy to send it as a separate patch and make this one a patch series if you prefer the shape.

Best regards,
Richard Cheng.

Quoted 43 lines

Any thoughts Jonathan?

Quoted 40 lines

A single-dport host bridge/root port has no HDM decoder capability, so its switch decoder is a SW-only passthrough. Its CXL_DECODER_F_ENABLE flag is cleared on region teardown and never restored on rescan, so cxl_port_setup_targets() fails with -ENXIO.

Re-enable the passthrough decoder when its interleave and HPA config still match the region, it holds no HW state.

---
 drivers/cxl/core/region.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)
drivers/cxl/core/region.c +13 -0
index 1e211542b6b6..011ceb5bae79 100644
@@ -1574,6 +1574,19 @@ static int cxl_port_setup_targets(struct cxl_port *port,
 	}
 
 	if (test_bit(CXL_REGION_F_AUTO, &cxlr->flags)) {
+		/*
+		 * A passthrough switch decoder holds no HW decode state.
+		 * It's CXL_DECODER_F_ENABLE flag is pure software bookkeeping
+		 * that is cleared when the region is torn down. On auto-discovery
+		 * re-assembly after a subsequent rescan the decode config still
+		 * matches the region, so restore the flag rather than fail to
+		 * rebuild a region that HW is in fact still decoding.
+		 */
+		if (!cxld->commit && cxld->interleave_ways == iw &&
+		    (iw <= 1 || cxld->interleave_granularity == ig) &&
+		    spa_maps_hpa(p, &cxld->hpa_range))
+			cxld->flags |= CXL_DECODER_F_ENABLE;
+
 		if (cxld->interleave_ways != iw ||
 		    (iw > 1 && cxld->interleave_granularity != ig) ||
 		    !spa_maps_hpa(p, &cxld->hpa_range) ||

base-commit: 4539944e515183668109bdf4d0c3d7d228383d88

#3 lore raw
Quoted 13 lines · Richard Cheng

On Mon, Jul 27, 2026 at 01:25:48PM +0800, Dave Jiang wrote:

I think this part is talking about physical remove/insert ? sorry maybe I commit message was too vague about the scenario. The case here is a SW-only teardown via sysfs, this is what I did.

"""

$ echo 1 > /sys/bus/pci/devices/$BDF/remove
$ echo 1 > /sys/bus/pci/devices/$BDF/rescan
"""

No physical removal, no power cycle, no link-down, no reset. Linux drops and re-enumerates the same device, which never stopped running. The endpoint HDM decoder is still committed.

Quoted 7 lines

The device is byte-identical before and after, the kernel doesn't clear it either, the decoder is locked and cxl_decoder_reset() returns early for CXL_DECODER_F_LOCK before touching any register. The memory keeps decoding.

I don't think anything needs re-programming here? the only thing lost is kernel-side bookkeeping on the passthrough decoder, which is freed with port and reallocated with F_ENABLE clear on rescan. The endpoint recovers its state from HW.
The passthrough decoder has no HW to recover from, that asymmetry is the bug.

Being a passthrough decoder is just a special state of a switch decoder, it doesn't necessarily imply programmability (Enable, Commit, Lock all technically still apply, it's just ambiguous what they should be).

Wonder if we're just not setting these intermediate decoders up sanely.

If the endpoint is locked / not reprogrammable (commit / lock) on the first go around, maybe the intermediate decoders should be force-locked and not have F_ENABLE cleared on teardown? It's not like this actually changes anything on the hardware, it's just bookkeeping.

(although i will say it's been a bit since i looked at the flag state machine here, so i could be misremembering what the rules around ENABLE are i this context).

Curious - what happens if you fully unload cxl_pci between remove/rescan?

~Gregory

#4 lore raw
Quoted 26 lines · Gregory Price
Quoted 13 lines · Richard Cheng

On Mon, Jul 27, 2026 at 01:25:48PM +0800, Dave Jiang wrote:

I think this part is talking about physical remove/insert ? sorry maybe I commit message was too vague about the scenario. The case here is a SW-only teardown via sysfs, this is what I did.

"""

$ echo 1 > /sys/bus/pci/devices/$BDF/remove
$ echo 1 > /sys/bus/pci/devices/$BDF/rescan
"""

No physical removal, no power cycle, no link-down, no reset. Linux drops and re-enumerates the same device, which never stopped running. The endpoint HDM decoder is still committed.

Quoted 7 lines

The device is byte-identical before and after, the kernel doesn't clear it either, the decoder is locked and cxl_decoder_reset() returns early for CXL_DECODER_F_LOCK before touching any register. The memory keeps decoding.

I don't think anything needs re-programming here? the only thing lost is kernel-side bookkeeping on the passthrough decoder, which is freed with port and reallocated with F_ENABLE clear on rescan. The endpoint recovers its state from HW.
The passthrough decoder has no HW to recover from, that asymmetry is the bug.

Being a passthrough decoder is just a special state of a switch decoder, it doesn't necessarily imply programmability (Enable, Commit, Lock all technically still apply, it's just ambiguous what they should be).

Quoted 7 lines

Wonder if we're just not setting these intermediate decoders up sanely.

If the endpoint is locked / not reprogrammable (commit / lock) on the first go around, maybe the intermediate decoders should be force-locked and not have F_ENABLE cleared on teardown? It's not like this actually changes anything on the hardware, it's just bookkeeping.

Agree on this part, I think what you are suggesting is a better approach than what I posted, if the EP is locked an not reprogrammable, the intermediate decoder describing that path has no business begin torn down either.

It also lines up with Dave's question about checking the EP decoder's lock state before proceeding.

Quoted 7 lines

(although i will say it's been a bit since i looked at the flag state machine here, so i could be misremembering what the rules around ENABLE are i this context).

Curious - what happens if you fully unload cxl_pci between remove/rescan?

Good qeustion, I'll test that as soon as possible before I sent v2.

Thanks for the pointer !

Best regards,
Richard Cheng.

Quoted 1 line

~Gregory

maintainer
#5 lore raw
In reply to Richard Cheng
Quoted 25 lines · Richard Cheng
Quoted 9 lines · Dave Jiang
Quoted 2 lines · Richard Cheng

After a CXL endpoint is PCI hot-removed and the bus rescanned, an auto-discovered region fails to rebiuld and is lost, evne though the

rebuild
even

Hi Dave,
Thanks for the review, I'll fix the typo in v2.

Quoted 2 lines
Quoted 1 line

device's HDM decoder is still committed and decoding.

I think this part is talking about physical remove/insert ? sorry maybe I commit message was too vague about the scenario. The case here is a SW-only teardown via sysfs, this is what I did.

"""

$ echo 1 > /sys/bus/pci/devices/$BDF/remove
$ echo 1 > /sys/bus/pci/devices/$BDF/rescan
"""
Quoted 5 lines

No physical removal, no power cycle, no link-down, no reset. Linux drops and re-enumerates the same device, which never stopped running. The endpoint HDM decoder is still committed.

The device is byte-identical before and after, the kernel doesn't clear it either, the decoder is locked and cxl_decoder_reset() returns early for CXL_DECODER_F_LOCK before touching any register. The memory keeps decoding.

Given this is a single target passthrough, should we check the EP decoder for lock before going forward?

Quoted 30 lines

I don't think anything needs re-programming here? the only thing lost is kernel-side bookkeeping on the passthrough decoder, which is freed with port and reallocated with F_ENABLE clear on rescan. The endpoint recovers its state from HW.
The passthrough decoder has no HW to recover from, that asymmetry is the bug.

Quoted 2 lines

Also, a complete different device with possibly different size can be inserted. And the device showed up would be unconfigured. Given there's no BIOS to program the device since the OS has taken over, should it be still considered part of the auto-region?

Agreed that would be wrong, and I don't think this patch allows it. A different or freshly-inserted device fails the existing endpoint checks, its own decoder won't come back COMMITTED with a matching HPA range, so cxl_add_to_region() won't re-assemble the auto-region relardless of what the passthrough decoder's flag says. This patch guards on the decode config still matching (iw, ig, spa_maps_hap()), and on !cxld->commit, so it can only ever touch a SW-only stub.

So the coverage here is SW-only, I should state it more clearly in v2.

Btw, the check "!cxld->commit" is doing a lot of implicit work in the code base, I think that's worth fixing regardless of the patch. "commit == NULL" is not unique to passthrough decoders, cxl_setup_hdm_decoder_from_dvsec() also sets "commit = NULL" for DVSEC-emulated RCD endpoint decoders, and root decoder never set it. My call happens to be safe, but that isn't visible from the condition.

I would suggest an inline helper like the following, do you think it's reasonable ?

"""
static inline bool cxl_decoder_is_passthrough(struct cxl_decoder *cxld)

{
    if (cxld->commit)
        return false;
    if (!is_switch_decoder(&cxld->dev) || is_root_decoder(&cxld->dev))
        return false;
    return to_cxl_switch_decoder(&cxld->dev)->nr_targets <= 1;
}
Quoted 6 lines

"""

There're about 4 call site today, and it distinguishes "passthrough" from "has no commit routine" , which are currently the same check for different things.

Happy to send it as a separate patch and make this one a patch series if you prefer the shape.

Yeah I think if we can improve the clarity that would be a good thing. Thanks Richard!

DJ

Quoted 47 lines

Best regards,
Richard Cheng.

Quoted 43 lines

Any thoughts Jonathan?

Quoted 40 lines

A single-dport host bridge/root port has no HDM decoder capability, so its switch decoder is a SW-only passthrough. Its CXL_DECODER_F_ENABLE flag is cleared on region teardown and never restored on rescan, so cxl_port_setup_targets() fails with -ENXIO.

Re-enable the passthrough decoder when its interleave and HPA config still match the region, it holds no HW state.

---
 drivers/cxl/core/region.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)
drivers/cxl/core/region.c +13 -0
index 1e211542b6b6..011ceb5bae79 100644
@@ -1574,6 +1574,19 @@ static int cxl_port_setup_targets(struct cxl_port *port,
 	}
 
 	if (test_bit(CXL_REGION_F_AUTO, &cxlr->flags)) {
+		/*
+		 * A passthrough switch decoder holds no HW decode state.
+		 * It's CXL_DECODER_F_ENABLE flag is pure software bookkeeping
+		 * that is cleared when the region is torn down. On auto-discovery
+		 * re-assembly after a subsequent rescan the decode config still
+		 * matches the region, so restore the flag rather than fail to
+		 * rebuild a region that HW is in fact still decoding.
+		 */
+		if (!cxld->commit && cxld->interleave_ways == iw &&
+		    (iw <= 1 || cxld->interleave_granularity == ig) &&
+		    spa_maps_hpa(p, &cxld->hpa_range))
+			cxld->flags |= CXL_DECODER_F_ENABLE;
+
 		if (cxld->interleave_ways != iw ||
 		    (iw > 1 && cxld->interleave_granularity != ig) ||
 		    !spa_maps_hpa(p, &cxld->hpa_range) ||

base-commit: 4539944e515183668109bdf4d0c3d7d228383d88

#6 lore raw
Quoted 67 lines · Dave Jiang
Quoted 25 lines · Richard Cheng
Quoted 9 lines · Dave Jiang
Quoted 2 lines · Richard Cheng

After a CXL endpoint is PCI hot-removed and the bus rescanned, an auto-discovered region fails to rebiuld and is lost, evne though the

rebuild
even

Hi Dave,
Thanks for the review, I'll fix the typo in v2.

Quoted 2 lines
Quoted 1 line

device's HDM decoder is still committed and decoding.

I think this part is talking about physical remove/insert ? sorry maybe I commit message was too vague about the scenario. The case here is a SW-only teardown via sysfs, this is what I did.

"""

$ echo 1 > /sys/bus/pci/devices/$BDF/remove
$ echo 1 > /sys/bus/pci/devices/$BDF/rescan
"""
Quoted 5 lines

No physical removal, no power cycle, no link-down, no reset. Linux drops and re-enumerates the same device, which never stopped running. The endpoint HDM decoder is still committed.

The device is byte-identical before and after, the kernel doesn't clear it either, the decoder is locked and cxl_decoder_reset() returns early for CXL_DECODER_F_LOCK before touching any register. The memory keeps decoding.

Given this is a single target passthrough, should we check the EP decoder for lock before going forward?

Quoted 32 lines

I don't think anything needs re-programming here? the only thing lost is kernel-side bookkeeping on the passthrough decoder, which is freed with port and reallocated with F_ENABLE clear on rescan. The endpoint recovers its state from HW.
The passthrough decoder has no HW to recover from, that asymmetry is the bug.

Quoted 2 lines

Also, a complete different device with possibly different size can be inserted. And the device showed up would be unconfigured. Given there's no BIOS to program the device since the OS has taken over, should it be still considered part of the auto-region?

Agreed that would be wrong, and I don't think this patch allows it. A different or freshly-inserted device fails the existing endpoint checks, its own decoder won't come back COMMITTED with a matching HPA range, so cxl_add_to_region() won't re-assemble the auto-region relardless of what the passthrough decoder's flag says. This patch guards on the decode config still matching (iw, ig, spa_maps_hap()), and on !cxld->commit, so it can only ever touch a SW-only stub.

So the coverage here is SW-only, I should state it more clearly in v2.

Btw, the check "!cxld->commit" is doing a lot of implicit work in the code base, I think that's worth fixing regardless of the patch. "commit == NULL" is not unique to passthrough decoders, cxl_setup_hdm_decoder_from_dvsec() also sets "commit = NULL" for DVSEC-emulated RCD endpoint decoders, and root decoder never set it. My call happens to be safe, but that isn't visible from the condition.

I would suggest an inline helper like the following, do you think it's reasonable ?

"""
static inline bool cxl_decoder_is_passthrough(struct cxl_decoder *cxld)

{
    if (cxld->commit)
        return false;
    if (!is_switch_decoder(&cxld->dev) || is_root_decoder(&cxld->dev))
        return false;
    return to_cxl_switch_decoder(&cxld->dev)->nr_targets <= 1;
}

"""

Quoted 57 lines
Quoted 4 lines

There're about 4 call site today, and it distinguishes "passthrough" from "has no commit routine" , which are currently the same check for different things.

Happy to send it as a separate patch and make this one a patch series if you prefer the shape.

Yeah I think if we can improve the clarity that would be a good thing. Thanks Richard!

DJ

Quoted 47 lines

Best regards,
Richard Cheng.

Quoted 43 lines

Any thoughts Jonathan?

Quoted 40 lines

A single-dport host bridge/root port has no HDM decoder capability, so its switch decoder is a SW-only passthrough. Its CXL_DECODER_F_ENABLE flag is cleared on region teardown and never restored on rescan, so cxl_port_setup_targets() fails with -ENXIO.

Re-enable the passthrough decoder when its interleave and HPA config still match the region, it holds no HW state.

---
 drivers/cxl/core/region.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)
drivers/cxl/core/region.c +13 -0
index 1e211542b6b6..011ceb5bae79 100644
@@ -1574,6 +1574,19 @@ static int cxl_port_setup_targets(struct cxl_port *port,
 	}
 
 	if (test_bit(CXL_REGION_F_AUTO, &cxlr->flags)) {
+		/*
+		 * A passthrough switch decoder holds no HW decode state.
+		 * It's CXL_DECODER_F_ENABLE flag is pure software bookkeeping
+		 * that is cleared when the region is torn down. On auto-discovery
+		 * re-assembly after a subsequent rescan the decode config still
+		 * matches the region, so restore the flag rather than fail to
+		 * rebuild a region that HW is in fact still decoding.
+		 */
+		if (!cxld->commit && cxld->interleave_ways == iw &&
+		    (iw <= 1 || cxld->interleave_granularity == ig) &&
+		    spa_maps_hpa(p, &cxld->hpa_range))
+			cxld->flags |= CXL_DECODER_F_ENABLE;
+
 		if (cxld->interleave_ways != iw ||
 		    (iw > 1 && cxld->interleave_granularity != ig) ||
 		    !spa_maps_hpa(p, &cxld->hpa_range) ||

base-commit: 4539944e515183668109bdf4d0c3d7d228383d88

So I re-tested those operations on v7.0-rc2 kernel without the patch, we find out

0a70b7cd397e ("cxl: Test CXL_DECODER_F_LOCK as a bitmask")

This patch is not needed, and Gregory is right that intermediate decoders should stay locked and not have F_ENABLE cleared on teardown.

For v2 I'll only send the cxl_decoder_is_passthrough() helper that Dave asked for. It's just for clarity and no behavior change. This mask v.s. index mistake already caused one bug, so a small helper that hides the raw flag from callers seems useful.

Best regards,
Richard Cheng.

#7 lore raw
Quoted 9 lines · Richard Cheng

On Tue, Jul 28, 2026 at 07:25:09AM +0800, Dave Jiang wrote:

So I re-tested those operations on v7.0-rc2 kernel without the patch, we find out

0a70b7cd397e ("cxl: Test CXL_DECODER_F_LOCK as a bitmask")

This patch is not needed, and Gregory is right that intermediate decoders should stay locked and not have F_ENABLE cleared on teardown.

Makes sense to me, it will look a little funky but auto decoders already got that funk all over them so it is what it is. :]

~Gregory