cxl/port: Restart port enumeration when a sibling adds the dport first
Alison Schofield
maintainerPATCH 1/1 cxl/port: Restart port enumeration when a sibling adds the dport first
Endpoint probes can race while enumerating a shared switch. If a sibling probe adds the dport first, the losing probe treats the existing dport as an error and fails to enumerate the endpoint.
Treat this race the same as the existing port-created case by restarting the port walk, allowing it to find the existing dport and continue enumeration.
This race was discovered while testing a cxl_test mixed-granularity topology, where twelve endpoints behind shared switches are probed in parallel during module load.
--- drivers/cxl/core/port.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/cxl/core/port.c +2 -2
@@ -1749,8 +1749,8 @@ static int add_port_attach_ep(struct cxl_memdev *cxlmd, parent_dport, uport_dev, dport_dev);if (IS_ERR (dport)) { -/* Port already exists, restart iteration */ -if (PTR_ERR (dport) == -EAGAIN ) +/* Port or dport already exists, restart iteration */ +if (PTR_ERR (dport) == -EAGAIN ||PTR_ERR (dport) == -EBUSY )return 0 ;return PTR_ERR (dport); }
Signature · 1 line
2.37.3
Li Ming
reviewerQuoted 11 lines · Alison Schofield
Endpoint probes can race while enumerating a shared switch. If a sibling probe adds the dport first, the losing probe treats the existing dport as an error and fails to enumerate the endpoint.
Treat this race the same as the existing port-created case by restarting the port walk, allowing it to find the existing dport and continue enumeration.
This race was discovered while testing a
cxl_testmixed-granularity topology, where twelve endpoints behind shared switches are probed in parallel during module load.
Hi Alison,
Per commit log, I think the root cause is probe_dport() called in devm_cxl_create_port() twice, and the second time failed because the dport has been added.
But devm_cxl_create_port() is protected by its parent port's device lock, probe_dport() is called only when a new port is created in devm_cxl_create_port(). If a port can be found, will return -EAGAIN directly in devm_cxl_create_port().
So If probe_dport() was called twice, is it something wrong in find_cxl_port_by_uport()? Or I miss some details?
Ming
Quoted 21 lines
Fixes 4f06d81e7c6a ("cxl: Defer dport allocation for switch ports")Signed-off-by Alison Schofield <[email protected]>--- drivers/cxl/core/port.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)drivers/cxl/core/port.c +2
-2@@ -1749,8 +1749,8 @@ static int add_port_attach_ep(struct cxl_memdev *cxlmd, parent_dport, uport_dev, dport_dev);if (IS_ERR (dport)) { -/* Port already exists, restart iteration */ -if (PTR_ERR (dport) == -EAGAIN ) +/* Port or dport already exists, restart iteration */ +if (PTR_ERR (dport) == -EAGAIN ||PTR_ERR (dport) == -EBUSY )return 0 ;return PTR_ERR (dport); }
Alison Schofield
maintainerQuoted 13 lines · Li Ming
Quoted 11 lines · Alison Schofield
Endpoint probes can race while enumerating a shared switch. If a sibling probe adds the dport first, the losing probe treats the existing dport as an error and fails to enumerate the endpoint.
Treat this race the same as the existing port-created case by restarting the port walk, allowing it to find the existing dport and continue enumeration.
This race was discovered while testing a
cxl_testmixed-granularity topology, where twelve endpoints behind shared switches are probed in parallel during module load.
Quoted 8 lines
Hi Alison,
Per commit log, I think the root cause is
probe_dport()called indevm_cxl_create_port()twice, and the second time failed because the dport has been added.But
devm_cxl_create_port()is protected by its parent port's device lock,probe_dport()is called only when a new port is created indevm_cxl_create_port(). If a port can be found, will return-EAGAINdirectly indevm_cxl_create_port().So If
probe_dport()was called twice, is it something wrong infind_cxl_port_by_uport()? Or I miss some details?
Hi Ming,
Thanks for the review and good catch! The commit log is vague and I've reworded it for v2.
You're right about devm_cxl_create_port(): two probes both entering it serialize on parent_port->dev and the loser then finds the port and returns -EAGAIN without a second probe_dport(). find_cxl_port_by_uport() is fine.
The -EBUSY comes from a different path. Two endpoints behind the same switch race to add the shared dport and once one publishes the port the other takes the port-found path instead of devm_cxl_create_port():
cxl_mem probe (endpoint 0) cxl_mem probe (endpoint 1)
-------------------------- --------------------------
find(uport) == NULL
add_port_attach_ep()
devm_cxl_create_port()
devm_cxl_add_port()
/* port published, */
/* dport not added yet */
find(uport) succeeds
find_or_add_dport()
probe_dport()
/* adds shared dport */
return -EAGAIN
guard(device)(&port->dev)
probe_dport()
dport_exists() == true
return -EBUSYBefore this patch, add_port_attach_ep() treated that -EBUSY as fatal and dropped endpoint 0. Treating it like -EAGAIN restarts the walk, and that finds the now present dport and continues.
Sending a v2 w commit msg update. Please take a look.
Thanks,
Alison
Quoted 26 lines
Ming
Quoted 21 lines
Fixes 4f06d81e7c6a ("cxl: Defer dport allocation for switch ports")Signed-off-by Alison Schofield <[email protected]>--- drivers/cxl/core/port.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)drivers/cxl/core/port.c +2
-2@@ -1749,8 +1749,8 @@ static int add_port_attach_ep(struct cxl_memdev *cxlmd, parent_dport, uport_dev, dport_dev);if (IS_ERR (dport)) { -/* Port already exists, restart iteration */ -if (PTR_ERR (dport) == -EAGAIN ) +/* Port or dport already exists, restart iteration */ +if (PTR_ERR (dport) == -EAGAIN ||PTR_ERR (dport) == -EBUSY )return 0 ;return PTR_ERR (dport); }