cxl/port: Fix target list setup for multiple decoders sharing the same dport
PATCH 1/1 cxl/port: Fix target list setup for multiple decoders sharing the same dport
If a switch port has more than one decoder that is using the same downstream port, the enumeration of the target lists may fail with:
# dmesg | grep target.list update_decoder_targets: cxl decoder1.0: dport3 found in target list, index 3 update_decoder_targets: cxl decoder1.0: dport2 found in target list, index 2 update_decoder_targets: cxl decoder1.0: dport0 found in target list, index 0 update_decoder_targets: cxl decoder2.0: dport3 found in target list, index 1 update_decoder_targets: cxl decoder4.0: dport3 found in target list, index 1 cxl_mem mem6: failed to find endpoint12:0000:00:01.4 in target list of decoder2.1 cxl_mem mem8: failed to find endpoint13:0000:20:01.4 in target list of decoder4.1
The case, that the same downstream port can be used in multiple target lists, is allowed and possible.
Fix the update of the target list. Enumerate all children of the switch port and do not stop the iteration after the first matching target was found.
With the fix applied:
# dmesg | grep target.list update_decoder_targets: cxl decoder1.0: dport2 found in target list, index 2 update_decoder_targets: cxl decoder1.0: dport0 found in target list, index 0 update_decoder_targets: cxl decoder1.0: dport3 found in target list, index 3 update_decoder_targets: cxl decoder2.0: dport3 found in target list, index 1 update_decoder_targets: cxl decoder2.1: dport3 found in target list, index 1 update_decoder_targets: cxl decoder4.0: dport3 found in target list, index 1 update_decoder_targets: cxl decoder4.1: dport3 found in target list, index 1
Analyzing the conditions when this happens:
1) A dport is shared by multiple decoders.
2) The decoders have interleaving configured (ways > 1).
The configuration above has the following hierarchy details (fixed version):
root0
|_ | | | decoder0.1 | ways: 2 | target_list: 0,1 |_______________________________________ | | | dport0 | dport1 | | port2 port4 | | |___________________ |_____________________ | | | | | | | decoder2.0 decoder2.1 | decoder4.0 decoder4.1 | ways: 2 ways: 2 | ways: 2 ways: 2 | target_list: 2,3 target_list: 2,3 | target_list: 2,3 target_list: 2,3 |___________________ |___________________ | | | | | dport2 | dport3 | dport2 | dport3 | | | | endpoint7 endpoint12 endpoint9 endpoint13 |_ |_ |_ |_ | | | | | | | | | decoder7.0 | decoder12.0 | decoder9.0 | decoder13.0 | decoder7.2 | decoder12.2 | decoder9.2 | decoder13.2 | | | | mem3 mem5 mem6 mem8
Note: Device numbers vary for every boot.
Current kernel fails to enumerate endpoint12 and endpoint13 as the target list is not updated for the second decoder.
--- drivers/cxl/core/port.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/cxl/core/port.c +1 -1
@@ -1590,7 +1590,7 @@ static int update_decoder_targets(struct device *dev, void *data) cxlsd->target[i] = dport;dev_dbg (dev,"dport%d found in target list, index %d\n" , dport->port_id, i); -return 1 ; +return 0 ; } }
base-commit: 88c72bab77aaf389beccf762e112828253ca0564
prerequisite-patch-id: a92b04b164c158c7f0f0dcde957bf5c02ffb3918
Signature · 1 line
2.47.3
Quoted 68 lines · Robert Richter
If a switch port has more than one decoder that is using the same downstream port, the enumeration of the target lists may fail with:
# dmesg | grep target.list update_decoder_targets: cxl decoder1.0: dport3 found in target list, index 3 update_decoder_targets: cxl decoder1.0: dport2 found in target list, index 2 update_decoder_targets: cxl decoder1.0: dport0 found in target list, index 0 update_decoder_targets: cxl decoder2.0: dport3 found in target list, index 1 update_decoder_targets: cxl decoder4.0: dport3 found in target list, index 1 cxl_mem mem6: failed to find endpoint12:0000:00:01.4 in target list of decoder2.1 cxl_mem mem8: failed to find endpoint13:0000:20:01.4 in target list of decoder4.1The case, that the same downstream port can be used in multiple target lists, is allowed and possible.
Fix the update of the target list. Enumerate all children of the switch port and do not stop the iteration after the first matching target was found.
With the fix applied:
# dmesg | grep target.list update_decoder_targets: cxl decoder1.0: dport2 found in target list, index 2 update_decoder_targets: cxl decoder1.0: dport0 found in target list, index 0 update_decoder_targets: cxl decoder1.0: dport3 found in target list, index 3 update_decoder_targets: cxl decoder2.0: dport3 found in target list, index 1 update_decoder_targets: cxl decoder2.1: dport3 found in target list, index 1 update_decoder_targets: cxl decoder4.0: dport3 found in target list, index 1 update_decoder_targets: cxl decoder4.1: dport3 found in target list, index 1Analyzing the conditions when this happens:
1) A dport is shared by multiple decoders.
2) The decoders have interleaving configured (ways > 1).
The configuration above has the following hierarchy details (fixed version):
root0
|_ | | | decoder0.1 | ways: 2 | target_list: 0,1 |_______________________________________ | | | dport0 | dport1 | | port2 port4 | | |___________________ |_____________________ | | | | | | | decoder2.0 decoder2.1 | decoder4.0 decoder4.1 | ways: 2 ways: 2 | ways: 2 ways: 2 | target_list: 2,3 target_list: 2,3 | target_list: 2,3 target_list: 2,3 |___________________ |___________________ | | | | | dport2 | dport3 | dport2 | dport3 | | | | endpoint7 endpoint12 endpoint9 endpoint13 |_ |_ |_ |_ | | | | | | | | | decoder7.0 | decoder12.0 | decoder9.0 | decoder13.0 | decoder7.2 | decoder12.2 | decoder9.2 | decoder13.2 | | | | mem3 mem5 mem6 mem8
Quoted 6 lines
Note: Device numbers vary for every boot.
Current kernel fails to enumerate endpoint12 and endpoint13 as the target list is not updated for the second decoder.
Signed-off-by Robert Richter <[email protected]>
Can you add a Fixes tag pls?
DJ
Quoted 20 lines
--- drivers/cxl/core/port.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/cxl/core/port.c +1
-1@@ -1590,7 +1590,7 @@ static int update_decoder_targets(struct device *dev, void *data) cxlsd->target[i] = dport;dev_dbg (dev,"dport%d found in target list, index %d\n" , dport->port_id, i); -return 1 ; +return 0 ; } }base-commit:
88c72bab77aaf389beccf762e112828253ca0564
prerequisite-patch-id:a92b04b164c158c7f0f0dcde957bf5c02ffb3918
Quoted 68 lines · Robert Richter
If a switch port has more than one decoder that is using the same downstream port, the enumeration of the target lists may fail with:
# dmesg | grep target.list update_decoder_targets: cxl decoder1.0: dport3 found in target list, index 3 update_decoder_targets: cxl decoder1.0: dport2 found in target list, index 2 update_decoder_targets: cxl decoder1.0: dport0 found in target list, index 0 update_decoder_targets: cxl decoder2.0: dport3 found in target list, index 1 update_decoder_targets: cxl decoder4.0: dport3 found in target list, index 1 cxl_mem mem6: failed to find endpoint12:0000:00:01.4 in target list of decoder2.1 cxl_mem mem8: failed to find endpoint13:0000:20:01.4 in target list of decoder4.1The case, that the same downstream port can be used in multiple target lists, is allowed and possible.
Fix the update of the target list. Enumerate all children of the switch port and do not stop the iteration after the first matching target was found.
With the fix applied:
# dmesg | grep target.list update_decoder_targets: cxl decoder1.0: dport2 found in target list, index 2 update_decoder_targets: cxl decoder1.0: dport0 found in target list, index 0 update_decoder_targets: cxl decoder1.0: dport3 found in target list, index 3 update_decoder_targets: cxl decoder2.0: dport3 found in target list, index 1 update_decoder_targets: cxl decoder2.1: dport3 found in target list, index 1 update_decoder_targets: cxl decoder4.0: dport3 found in target list, index 1 update_decoder_targets: cxl decoder4.1: dport3 found in target list, index 1Analyzing the conditions when this happens:
1) A dport is shared by multiple decoders.
2) The decoders have interleaving configured (ways > 1).
The configuration above has the following hierarchy details (fixed version):
root0
|_ | | | decoder0.1 | ways: 2 | target_list: 0,1 |_______________________________________ | | | dport0 | dport1 | | port2 port4 | | |___________________ |_____________________ | | | | | | | decoder2.0 decoder2.1 | decoder4.0 decoder4.1 | ways: 2 ways: 2 | ways: 2 ways: 2 | target_list: 2,3 target_list: 2,3 | target_list: 2,3 target_list: 2,3 |___________________ |___________________ | | | | | dport2 | dport3 | dport2 | dport3 | | | | endpoint7 endpoint12 endpoint9 endpoint13 |_ |_ |_ |_ | | | | | | | | | decoder7.0 | decoder12.0 | decoder9.0 | decoder13.0 | decoder7.2 | decoder12.2 | decoder9.2 | decoder13.2 | | | | mem3 mem5 mem6 mem8
Quoted 4 lines
Note: Device numbers vary for every boot.
Current kernel fails to enumerate endpoint12 and endpoint13 as the target list is not updated for the second decoder.
Using the cxl-debug messaging you used to explain this, I was able to see this in the default cxl-test topology. At 6.18 we stopped seeing all the expected dports and with this patch, they're back.