Skip to contentSkip to conversation

cxl/port: Fix target list setup for multiple decoders sharing the same dport

PATCH Superseded Robert Richter 3 participants 3 messages Jan 7 – Jan 8 CXL

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 cxl/port: Fix target list setup for multiple decoders sharing the same dport

If a switch port has more than one decoder that is using the same downstream port, the enumeration of the target lists may fail with:

 # dmesg | grep target.list
 update_decoder_targets: cxl decoder1.0: dport3 found in target list, index 3
 update_decoder_targets: cxl decoder1.0: dport2 found in target list, index 2
 update_decoder_targets: cxl decoder1.0: dport0 found in target list, index 0
 update_decoder_targets: cxl decoder2.0: dport3 found in target list, index 1
 update_decoder_targets: cxl decoder4.0: dport3 found in target list, index 1
 cxl_mem mem6: failed to find endpoint12:0000:00:01.4 in target list of decoder2.1
 cxl_mem mem8: failed to find endpoint13:0000:20:01.4 in target list of decoder4.1

The case, that the same downstream port can be used in multiple target lists, is allowed and possible.

Fix the update of the target list. Enumerate all children of the switch port and do not stop the iteration after the first matching target was found.

With the fix applied:

 # dmesg | grep target.list
 update_decoder_targets: cxl decoder1.0: dport2 found in target list, index 2
 update_decoder_targets: cxl decoder1.0: dport0 found in target list, index 0
 update_decoder_targets: cxl decoder1.0: dport3 found in target list, index 3
 update_decoder_targets: cxl decoder2.0: dport3 found in target list, index 1
 update_decoder_targets: cxl decoder2.1: dport3 found in target list, index 1
 update_decoder_targets: cxl decoder4.0: dport3 found in target list, index 1
 update_decoder_targets: cxl decoder4.1: dport3 found in target list, index 1

Analyzing the conditions when this happens:

1) A dport is shared by multiple decoders.

2) The decoders have interleaving configured (ways > 1).

The configuration above has the following hierarchy details (fixed version):

root0

 |_
 | |
 | decoder0.1
 | ways: 2
 | target_list: 0,1
 |_______________________________________
 |                                       |
 | dport0                                | dport1
 |                                       |
 port2                                   port4
 |                                       |
 |___________________                    |_____________________
 | |                 |                   | |                   |
 | decoder2.0        decoder2.1          | decoder4.0          decoder4.1
 | ways: 2           ways: 2             | ways: 2             ways: 2
 | target_list: 2,3  target_list: 2,3    | target_list: 2,3    target_list: 2,3
 |___________________                    |___________________
 |                   |                   |                   |
 | dport2            | dport3            | dport2            | dport3
 |                   |                   |                   |
 endpoint7           endpoint12          endpoint9           endpoint13
 |_                  |_                  |_                  |_
 | |                 | |                 | |                 | |
 | decoder7.0        | decoder12.0       | decoder9.0        | decoder13.0
 | decoder7.2        | decoder12.2       | decoder9.2        | decoder13.2
 |                   |                   |                   |
 mem3                mem5                mem6                mem8

Note: Device numbers vary for every boot.

Current kernel fails to enumerate endpoint12 and endpoint13 as the target list is not updated for the second decoder.

---
 drivers/cxl/core/port.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/cxl/core/port.c +1 -1
index fef3aa0c6680..3310dbfae9d6 100644
@@ -1590,7 +1590,7 @@ static int update_decoder_targets(struct device *dev, void *data)
 			cxlsd->target[i] = dport;
 			dev_dbg(dev, "dport%d found in target list, index %d\n",
 				dport->port_id, i);
-			return 1;
+			return 0;
 		}
 	}
 

base-commit: 88c72bab77aaf389beccf762e112828253ca0564
prerequisite-patch-id: a92b04b164c158c7f0f0dcde957bf5c02ffb3918

Signature · 1 line
2.47.3
maintainer Reviewed-by on v1 1/1
#1 lore raw
Quoted 68 lines · Robert Richter

If a switch port has more than one decoder that is using the same downstream port, the enumeration of the target lists may fail with:

 # dmesg | grep target.list
 update_decoder_targets: cxl decoder1.0: dport3 found in target list, index 3
 update_decoder_targets: cxl decoder1.0: dport2 found in target list, index 2
 update_decoder_targets: cxl decoder1.0: dport0 found in target list, index 0
 update_decoder_targets: cxl decoder2.0: dport3 found in target list, index 1
 update_decoder_targets: cxl decoder4.0: dport3 found in target list, index 1
 cxl_mem mem6: failed to find endpoint12:0000:00:01.4 in target list of decoder2.1
 cxl_mem mem8: failed to find endpoint13:0000:20:01.4 in target list of decoder4.1

The case, that the same downstream port can be used in multiple target lists, is allowed and possible.

Fix the update of the target list. Enumerate all children of the switch port and do not stop the iteration after the first matching target was found.

With the fix applied:

 # dmesg | grep target.list
 update_decoder_targets: cxl decoder1.0: dport2 found in target list, index 2
 update_decoder_targets: cxl decoder1.0: dport0 found in target list, index 0
 update_decoder_targets: cxl decoder1.0: dport3 found in target list, index 3
 update_decoder_targets: cxl decoder2.0: dport3 found in target list, index 1
 update_decoder_targets: cxl decoder2.1: dport3 found in target list, index 1
 update_decoder_targets: cxl decoder4.0: dport3 found in target list, index 1
 update_decoder_targets: cxl decoder4.1: dport3 found in target list, index 1

Analyzing the conditions when this happens:

1) A dport is shared by multiple decoders.

2) The decoders have interleaving configured (ways > 1).

The configuration above has the following hierarchy details (fixed version):

root0

 |_
 | |
 | decoder0.1
 | ways: 2
 | target_list: 0,1
 |_______________________________________
 |                                       |
 | dport0                                | dport1
 |                                       |
 port2                                   port4
 |                                       |
 |___________________                    |_____________________
 | |                 |                   | |                   |
 | decoder2.0        decoder2.1          | decoder4.0          decoder4.1
 | ways: 2           ways: 2             | ways: 2             ways: 2
 | target_list: 2,3  target_list: 2,3    | target_list: 2,3    target_list: 2,3
 |___________________                    |___________________
 |                   |                   |                   |
 | dport2            | dport3            | dport2            | dport3
 |                   |                   |                   |
 endpoint7           endpoint12          endpoint9           endpoint13
 |_                  |_                  |_                  |_
 | |                 | |                 | |                 | |
 | decoder7.0        | decoder12.0       | decoder9.0        | decoder13.0
 | decoder7.2        | decoder12.2       | decoder9.2        | decoder13.2
 |                   |                   |                   |
 mem3                mem5                mem6                mem8
Quoted 6 lines

Note: Device numbers vary for every boot.

Current kernel fails to enumerate endpoint12 and endpoint13 as the target list is not updated for the second decoder.

Can you add a Fixes tag pls?

DJ

Quoted 20 lines
---
 drivers/cxl/core/port.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/cxl/core/port.c +1 -1
index fef3aa0c6680..3310dbfae9d6 100644
@@ -1590,7 +1590,7 @@ static int update_decoder_targets(struct device *dev, void *data)
 			cxlsd->target[i] = dport;
 			dev_dbg(dev, "dport%d found in target list, index %d\n",
 				dport->port_id, i);
-			return 1;
+			return 0;
 		}
 	}
 

base-commit: 88c72bab77aaf389beccf762e112828253ca0564
prerequisite-patch-id: a92b04b164c158c7f0f0dcde957bf5c02ffb3918

maintainer Reviewed-by on v1 1/1
#2 lore raw
In reply to Robert Richter
Quoted 68 lines · Robert Richter

If a switch port has more than one decoder that is using the same downstream port, the enumeration of the target lists may fail with:

 # dmesg | grep target.list
 update_decoder_targets: cxl decoder1.0: dport3 found in target list, index 3
 update_decoder_targets: cxl decoder1.0: dport2 found in target list, index 2
 update_decoder_targets: cxl decoder1.0: dport0 found in target list, index 0
 update_decoder_targets: cxl decoder2.0: dport3 found in target list, index 1
 update_decoder_targets: cxl decoder4.0: dport3 found in target list, index 1
 cxl_mem mem6: failed to find endpoint12:0000:00:01.4 in target list of decoder2.1
 cxl_mem mem8: failed to find endpoint13:0000:20:01.4 in target list of decoder4.1

The case, that the same downstream port can be used in multiple target lists, is allowed and possible.

Fix the update of the target list. Enumerate all children of the switch port and do not stop the iteration after the first matching target was found.

With the fix applied:

 # dmesg | grep target.list
 update_decoder_targets: cxl decoder1.0: dport2 found in target list, index 2
 update_decoder_targets: cxl decoder1.0: dport0 found in target list, index 0
 update_decoder_targets: cxl decoder1.0: dport3 found in target list, index 3
 update_decoder_targets: cxl decoder2.0: dport3 found in target list, index 1
 update_decoder_targets: cxl decoder2.1: dport3 found in target list, index 1
 update_decoder_targets: cxl decoder4.0: dport3 found in target list, index 1
 update_decoder_targets: cxl decoder4.1: dport3 found in target list, index 1

Analyzing the conditions when this happens:

1) A dport is shared by multiple decoders.

2) The decoders have interleaving configured (ways > 1).

The configuration above has the following hierarchy details (fixed version):

root0

 |_
 | |
 | decoder0.1
 | ways: 2
 | target_list: 0,1
 |_______________________________________
 |                                       |
 | dport0                                | dport1
 |                                       |
 port2                                   port4
 |                                       |
 |___________________                    |_____________________
 | |                 |                   | |                   |
 | decoder2.0        decoder2.1          | decoder4.0          decoder4.1
 | ways: 2           ways: 2             | ways: 2             ways: 2
 | target_list: 2,3  target_list: 2,3    | target_list: 2,3    target_list: 2,3
 |___________________                    |___________________
 |                   |                   |                   |
 | dport2            | dport3            | dport2            | dport3
 |                   |                   |                   |
 endpoint7           endpoint12          endpoint9           endpoint13
 |_                  |_                  |_                  |_
 | |                 | |                 | |                 | |
 | decoder7.0        | decoder12.0       | decoder9.0        | decoder13.0
 | decoder7.2        | decoder12.2       | decoder9.2        | decoder13.2
 |                   |                   |                   |
 mem3                mem5                mem6                mem8
Quoted 4 lines

Note: Device numbers vary for every boot.

Current kernel fails to enumerate endpoint12 and endpoint13 as the target list is not updated for the second decoder.

Using the cxl-debug messaging you used to explain this, I was able to see this in the default cxl-test topology. At 6.18 we stopped seeing all the expected dports and with this patch, they're back.