cxl: Remove checking of iter in cxl_endpoint_get_perf_coordinates()
Dave Jiang
maintainerPATCH 1/2 cxl: Remove checking of iter in cxl_endpoint_get_perf_coordinates()
The while() loop in cxl_endpoint_get_perf_coordinates() checks to see if 'iter' is valid as part of the condition breaking out of the loop. However, iter is being used before the check at the end of the while loop before the next iteration starts. Given that the loop doesn't expect the iter to be NULL because it stops before the root port, remove the iter check.
--- drivers/cxl/core/port.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/cxl/core/port.c +1 -1
@@ -2142,7 +2142,7 @@ int cxl_endpoint_get_perf_coordinates(struct cxl_port *port,* port each iteration. If the parent is cxl root then there is * nothing to gather. */ -while (iter && !is_cxl_root (to_cxl_port (iter->dev.parent))) { +while (!is_cxl_root (to_cxl_port (iter->dev.parent))) {combine_coordinates (&c, &dport->sw_coord); c.write_latency += dport->link_latency; c.read_latency += dport->link_latency;
Signature · 1 line
2.43.0
Dave Jiang
maintainerPATCH 2/2 cxl: Add checks to access_coordinate calculation to fail missing data
Jonathan noted that when the coordinates for host bridge and switches can be 0s if no actual data are retrieved and the calculation continues. The resulting number would be inaccurate. Add checks to ensure that the calculation would complete only if the numbers are valid.
--- drivers/cxl/core/port.c | 31 +++++++++++++++++++++++++++---- 1 file changed, 27 insertions(+), 4 deletions(-)
drivers/cxl/core/port.c +27 -4
@@ -2110,6 +2110,20 @@ static void combine_coordinates(struct access_coordinate *c1, c1->read_latency += c2->read_latency; } +static bool coordinates_invalid (struct access_coordinate *c) +{ +if (!c->read_bandwidth && !c->write_bandwidth && + !c->read_latency && !c->write_latency) +return true; + +return false; +} + +static bool parent_port_is_cxl_root (struct cxl_port *port) +{ +return is_cxl_root (to_cxl_port (port->dev.parent)); +} +/** * cxl_endpoint_get_perf_coordinates - Retrieve performance numbers stored in dports * of CXL path
@@ -2142,16 +2156,25 @@ int cxl_endpoint_get_perf_coordinates(struct cxl_port *port,* port each iteration. If the parent is cxl root then there is * nothing to gather. */ -while (!is_cxl_root (to_cxl_port (iter->dev.parent))) { -combine_coordinates (&c, &dport->sw_coord); +while (!parent_port_is_cxl_root (iter)) { + iter =to_cxl_port (iter->dev.parent); + +/* There's no CDAT for the host bridge, so skip if so. */ +if (!parent_port_is_cxl_root (iter)) { +if (coordinates_invalid (&dport->sw_coord)) +return -EINVAL ; + +combine_coordinates (&c, &dport->sw_coord); + } + c.write_latency += dport->link_latency; c.read_latency += dport->link_latency; - - iter =to_cxl_port (iter->dev.parent); dport = iter->parent_dport; }/* Augment with the generic port (host bridge) perf data */ +if (coordinates_invalid (&dport->hb_coord)) +return -EINVAL ;combine_coordinates (&c, &dport->hb_coord);/* Get the calculated PCI paths bandwidth */
Signature · 1 line
2.43.0
RE: [PATCH 2/2] cxl: Add checks to access_coordinate calculation to fail missing data
Quoted 4 lines · Dave Jiang
Jonathan noted that when the coordinates for host bridge and switches can be 0s if no actual data are retrieved and the calculation continues. The resulting number would be inaccurate. Add checks to ensure that the calculation would complete only if the numbers are valid.
Similar comment as patch1. This smells like a fix, is this an urgent thing to get into v6.8, i.e. most configurations are busted without this, or is a nice to have fixup for a QEMU effect that may or may not show up in physical systems?
Dave Jiang
maintainerRe: [PATCH 2/2] cxl: Add checks to access_coordinate calculation to fail missing data
Quoted 10 lines · Dan Williams
Quoted 4 lines · Dave Jiang
Jonathan noted that when the coordinates for host bridge and switches can be 0s if no actual data are retrieved and the calculation continues. The resulting number would be inaccurate. Add checks to ensure that the calculation would complete only if the numbers are valid.
Similar comment as patch1. This smells like a fix, is this an urgent thing to get into v6.8, i.e. most configurations are busted without this, or is a nice to have fixup for a QEMU effect that may or may not show up in physical systems?
This would only be an issue if the switches do not supply a proper CDAT and/or if BIOS does not provide proper ACPI tables. I think it is only experienced on QEMU currently. I'll add a fix tag if you think it should be a fix.
Re: [PATCH 2/2] cxl: Add checks to access_coordinate calculation to fail missing data
Quoted 14 lines · Dave Jiang
Quoted 10 lines · Dan Williams
Quoted 4 lines · Dave Jiang
Jonathan noted that when the coordinates for host bridge and switches can be 0s if no actual data are retrieved and the calculation continues. The resulting number would be inaccurate. Add checks to ensure that the calculation would complete only if the numbers are valid.
Similar comment as patch1. This smells like a fix, is this an urgent thing to get into v6.8, i.e. most configurations are busted without this, or is a nice to have fixup for a QEMU effect that may or may not show up in physical systems?
Quoted 4 lines
This would only be an issue if the switches do not supply a proper CDAT and/or if BIOS does not provide proper ACPI tables. I think it is only experienced on QEMU currently. I'll add a fix tag if you think it should be a fix.
No, but please add that relevant clarification of impact to the changelog. It would have made it clear this is a nice to have thing for v6.9, nothing to worry about for v6.8.
Re: [PATCH 2/2] cxl: Add checks to access_coordinate calculation to fail missing data
Quoted 15 lines · On Wed, 28 Feb 2024 16:44:29 -0800 Dan Williams <[email protected]>…
Quoted 14 lines · Dave Jiang
Quoted 10 lines · Dan Williams
Quoted 4 lines · Dave Jiang
Jonathan noted that when the coordinates for host bridge and switches can be 0s if no actual data are retrieved and the calculation continues. The resulting number would be inaccurate. Add checks to ensure that the calculation would complete only if the numbers are valid.
Similar comment as patch1. This smells like a fix, is this an urgent thing to get into v6.8, i.e. most configurations are busted without this, or is a nice to have fixup for a QEMU effect that may or may not show up in physical systems?
Quoted 8 lines
Quoted 4 lines
This would only be an issue if the switches do not supply a proper CDAT and/or if BIOS does not provide proper ACPI tables. I think it is only experienced on QEMU currently. I'll add a fix tag if you think it should be a fix.
No, but please add that relevant clarification of impact to the changelog. It would have made it clear this is a nice to have thing for v6.9, nothing to worry about for v6.8.
Not seen in wild, but it might show up with a BIOS that reported CXL root ports via Generic Ports (via a PCI handle in the SRAT entry) rather than CXL Host Bridge (via an ACPI Handle int he SRAT entry). The code doesn't yet support this case (which is fine) but it should cleanly not support it rather than giving wrong numbers. There are other fun corners like embedded CXL switches in the host where the reporting will also be more 'creative'. I expect we will see these eventually but not in first generation or two.
I 'think' this would be a reasonable system choice as there is nothing to say that a CXL hostbridge doesn't have a bunch of root ports (potentially on different dies) that have different host initiator to port exit latency and bandwidth characteristics.
Whilst no known hardware. I don't think this is a FW bug case only.
Not a rush job though, fixes tag + 6.9 will be fine.
Jonathan
Re: [PATCH 2/2] cxl: Add checks to access_coordinate calculation to fail missing data
In reply to Dave JiangQuoted 7 lines · On Wed, 28 Feb 2024 17:25:42 -0700 Dave Jiang <[email protected]> wrote:
Jonathan noted that when the coordinates for host bridge and switches can be 0s if no actual data are retrieved and the calculation continues. The resulting number would be inaccurate. Add checks to ensure that the calculation would complete only if the numbers are valid.
Reported-by Jonathan Cameron <[email protected]>Signed-off-by Dave Jiang <[email protected]>
Hi Dave,
Whilst I think the fix is right, it is getting hard to read. Maybe a rethink is needed on how that iteration works?
Quoted 30 lines
--- drivers/cxl/core/port.c | 31 +++++++++++++++++++++++++++---- 1 file changed, 27 insertions(+), 4 deletions(-)drivers/cxl/core/port.c +14
-0@@ -2110,6 +2110,20 @@ static void combine_coordinates(struct access_coordinate *c1, c1->read_latency += c2->read_latency; } +static bool coordinates_invalid (struct access_coordinate *c) +{ +if (!c->read_bandwidth && !c->write_bandwidth && + !c->read_latency && !c->write_latency) +return true; + +return false; +} + +static bool parent_port_is_cxl_root (struct cxl_port *port) +{ +return is_cxl_root (to_cxl_port (port->dev.parent)); +} +/** * cxl_endpoint_get_perf_coordinates - Retrieve performance numbers stored in dports * of CXL path
drivers/cxl/core/port.c int cxl_endpoint_get_perf_coordinates(struct cxl_port *port, · line 2162
Quoted 9 lines
drivers/cxl/core/port.c +4
-2@@ -2142,16 +2156,25 @@ int cxl_endpoint_get_perf_coordinates(struct cxl_port *port,* port each iteration. If the parent is cxl root then there is * nothing to gather. */ -while (!is_cxl_root (to_cxl_port (iter->dev.parent))) { -combine_coordinates (&c, &dport->sw_coord); +while (!parent_port_is_cxl_root (iter)) { + iter =to_cxl_port (iter->dev.parent); + +/* There's no CDAT for the host bridge, so skip if so. */
Comment refers to skipping whereas code is 'doing more' for the other case so this is confusing to me.
The inverse of this only occurs on the last iteration I think.
Possibly a do / while instead of a while will do it. I'm far from confident though as all the levels of look up have me too confused.
do {
if (coordinates_invalid(&dport->sw_coord))
return -EINVAL;
combine_coordinates(&c, &dport->sw_coord);
iter = to_cxl_port(iter->dev.parent);
dport = iter->parent_dport;
} while (!parent_port_is_cxl_root(iter));
/* Do final link updates */
c.write_latency += dport->link_latency;
c.read_latency += dport->link_latency;Quoted 20 lines
drivers/cxl/core/port.c +9
-2+if (!parent_port_is_cxl_root (iter)) { +if (coordinates_invalid (&dport->sw_coord)) +return -EINVAL ; + +combine_coordinates (&c, &dport->sw_coord); + } + c.write_latency += dport->link_latency; c.read_latency += dport->link_latency; - - iter =to_cxl_port (iter->dev.parent); dport = iter->parent_dport; }/* Augment with the generic port (host bridge) perf data */ +if (coordinates_invalid (&dport->hb_coord)) +return -EINVAL ;combine_coordinates (&c, &dport->hb_coord);/* Get the calculated PCI paths bandwidth */
Dave Jiang
maintainerRe: [PATCH 2/2] cxl: Add checks to access_coordinate calculation to fail missing data
Quoted 59 lines · Jonathan Cameron
Quoted 7 lines · On Wed, 28 Feb 2024 17:25:42 -0700 Dave Jiang <[email protected]> wrote:
Jonathan noted that when the coordinates for host bridge and switches can be 0s if no actual data are retrieved and the calculation continues. The resulting number would be inaccurate. Add checks to ensure that the calculation would complete only if the numbers are valid.
Reported-by Jonathan Cameron <[email protected]>Signed-off-by Dave Jiang <[email protected]>Hi Dave,
Whilst I think the fix is right, it is getting hard to read. Maybe a rethink is needed on how that iteration works?
Quoted 30 lines
--- drivers/cxl/core/port.c | 31 +++++++++++++++++++++++++++---- 1 file changed, 27 insertions(+), 4 deletions(-)drivers/cxl/core/port.c +14
-0@@ -2110,6 +2110,20 @@ static void combine_coordinates(struct access_coordinate *c1, c1->read_latency += c2->read_latency; } +static bool coordinates_invalid (struct access_coordinate *c) +{ +if (!c->read_bandwidth && !c->write_bandwidth && + !c->read_latency && !c->write_latency) +return true; + +return false; +} + +static bool parent_port_is_cxl_root (struct cxl_port *port) +{ +return is_cxl_root (to_cxl_port (port->dev.parent)); +} +/** * cxl_endpoint_get_perf_coordinates - Retrieve performance numbers stored in dports * of CXL path drivers/cxl/core/port.c int cxl_endpoint_get_perf_coordinates(struct cxl_port *port, · line 2162
Quoted 9 lines
drivers/cxl/core/port.c +4
-2@@ -2142,16 +2156,25 @@ int cxl_endpoint_get_perf_coordinates(struct cxl_port *port,* port each iteration. If the parent is cxl root then there is * nothing to gather. */ -while (!is_cxl_root (to_cxl_port (iter->dev.parent))) { -combine_coordinates (&c, &dport->sw_coord); +while (!parent_port_is_cxl_root (iter)) { + iter =to_cxl_port (iter->dev.parent); + +/* There's no CDAT for the host bridge, so skip if so. */ Comment refers to skipping whereas code is 'doing more' for the other case so this is confusing to me.
Quoted 5 lines
The inverse of this only occurs on the last iteration I think.
Possibly a do / while instead of a while will do it. I'm far from confident though as all the levels of look up have me too confused.
So this is somewhat tricky. For example:
devices/pci0000:35/0000:35:01.0/0000:37:00.0/mem5
In this case the endpoint is attached to the host bridge without any switches. The endpoint is 0000:37:00.0 and the host bridge down stream port is 0000:35:01.0. In this instance there is no switch and therefore switch CDAT, but there is a valid dport. So we would skip the dport->sw_coord. However, we do need to pick up the link_latency between endpoint and downstream port. So we spend 1 iteration in the loop and skips the dport->sw_coord.
devices/pci0000:bf/0000:bf:00.0/0000:c0:00.0/0000:c1:00.0/0000:c2:00.0/mem8
Now in this case there's a CXL switch in between. So in first iteration, we pick up the dport->sw_coord. And in second iteration, we skip the dport->sw_coord. However, we would be adding two link_latency. For the link between 0000:c2:00.0 (endpoint) and 0000:c1:00.0 (switch downstream port), and the link between 0000:c0:00.0 (switch upstream port) and 0000:bf:00.0 (host bridge down stream port). So therefore we can't put the sum of link_latency outside of the loop.
Not sure how much better this is:
do {
struct cxl_port *parent_port = to_cxl_port(iter->dev.parent);
dport = iter->parent_dport;
if (!parent_port_is_cxl_root(parent_port)) {
if (coordinates_invalid(&dport->sw_coord))
return -EINVAL;
combine_coordinates(&c, &dport->sw_coord);
}
c.write_latency += dport->link_latency;
c.read_latency += dport->link_latency;
iter = to_cxl_port(iter->dev.parent);
} while (!parent_port_is_cxl_root(iter));or:
do {
struct cxl_port *parent_port = to_cxl_port(iter->dev.parent);
dport = iter->parent_dport;
c.write_latency += dport->link_latency;
c.read_latency += dport->link_latency;
if (parent_port_is_cxl_root(parent_port))
break;
if (coordinates_invalid(&dport->sw_coord))
return -EINVAL;
combine_coordinates(&c, &dport->sw_coord);
iter = to_cxl_port(iter->dev.parent);
} while (!parent_port_is_cxl_root(iter));Quoted 35 lines
do { if (coordinates_invalid(&dport->sw_coord)) return -EINVAL; combine_coordinates(&c, &dport->sw_coord); iter = to_cxl_port(iter->dev.parent); dport = iter->parent_dport; } while (!parent_port_is_cxl_root(iter)); /* Do final link updates */ c.write_latency += dport->link_latency; c.read_latency += dport->link_latency;Quoted 20 lines
drivers/cxl/core/port.c +9
-2+if (!parent_port_is_cxl_root (iter)) { +if (coordinates_invalid (&dport->sw_coord)) +return -EINVAL ; + +combine_coordinates (&c, &dport->sw_coord); + } + c.write_latency += dport->link_latency; c.read_latency += dport->link_latency; - - iter =to_cxl_port (iter->dev.parent); dport = iter->parent_dport; }/* Augment with the generic port (host bridge) perf data */ +if (coordinates_invalid (&dport->hb_coord)) +return -EINVAL ;combine_coordinates (&c, &dport->hb_coord);/* Get the calculated PCI paths bandwidth */
Dave Jiang
maintainerRe: [PATCH 2/2] cxl: Add checks to access_coordinate calculation to fail missing data
Quoted 97 lines · Dave Jiang
Quoted 59 lines · Jonathan Cameron
Quoted 7 lines · On Wed, 28 Feb 2024 17:25:42 -0700 Dave Jiang <[email protected]> wrote:
Jonathan noted that when the coordinates for host bridge and switches can be 0s if no actual data are retrieved and the calculation continues. The resulting number would be inaccurate. Add checks to ensure that the calculation would complete only if the numbers are valid.
Reported-by Jonathan Cameron <[email protected]>Signed-off-by Dave Jiang <[email protected]>Hi Dave,
Whilst I think the fix is right, it is getting hard to read. Maybe a rethink is needed on how that iteration works?
Quoted 30 lines
--- drivers/cxl/core/port.c | 31 +++++++++++++++++++++++++++---- 1 file changed, 27 insertions(+), 4 deletions(-)drivers/cxl/core/port.c +14
-0@@ -2110,6 +2110,20 @@ static void combine_coordinates(struct access_coordinate *c1, c1->read_latency += c2->read_latency; } +static bool coordinates_invalid (struct access_coordinate *c) +{ +if (!c->read_bandwidth && !c->write_bandwidth && + !c->read_latency && !c->write_latency) +return true; + +return false; +} + +static bool parent_port_is_cxl_root (struct cxl_port *port) +{ +return is_cxl_root (to_cxl_port (port->dev.parent)); +} +/** * cxl_endpoint_get_perf_coordinates - Retrieve performance numbers stored in dports * of CXL path drivers/cxl/core/port.c int cxl_endpoint_get_perf_coordinates(struct cxl_port *port, · line 2162
Quoted 9 lines
drivers/cxl/core/port.c +4
-2@@ -2142,16 +2156,25 @@ int cxl_endpoint_get_perf_coordinates(struct cxl_port *port,* port each iteration. If the parent is cxl root then there is * nothing to gather. */ -while (!is_cxl_root (to_cxl_port (iter->dev.parent))) { -combine_coordinates (&c, &dport->sw_coord); +while (!parent_port_is_cxl_root (iter)) { + iter =to_cxl_port (iter->dev.parent); + +/* There's no CDAT for the host bridge, so skip if so. */ Comment refers to skipping whereas code is 'doing more' for the other case so this is confusing to me.
Quoted 5 lines
The inverse of this only occurs on the last iteration I think.
Possibly a do / while instead of a while will do it. I'm far from confident though as all the levels of look up have me too confused.
So this is somewhat tricky. For example:
devices/pci0000:35/0000:35:01.0/0000:37:00.0/mem5In this case the endpoint is attached to the host bridge without any switches. The endpoint is 0000:37:00.0 and the host bridge down stream port is 0000:35:01.0. In this instance there is no switch and therefore switch CDAT, but there is a valid dport. So we would skip the
dport->sw_coord. However, we do need to pick up thelink_latencybetween endpoint and downstream port. So we spend 1 iteration in the loop and skips thedport->sw_coord.devices/pci0000:bf/0000:bf:00.0/0000:c0:00.0/0000:c1:00.0/0000:c2:00.0/mem8
Now in this case there's a CXL switch in between. So in first iteration, we pick up the
dport->sw_coord. And in second iteration, we skip thedport->sw_coord. However, we would be adding twolink_latency. For the link between 0000:c2:00.0 (endpoint) and 0000:c1:00.0 (switch downstream port), and the link between 0000:c0:00.0 (switch upstream port) and 0000:bf:00.0 (host bridge down stream port). So therefore we can't put the sum oflink_latencyoutside of the loop.Not sure how much better this is:
do { struct cxl_port *parent_port = to_cxl_port(iter->dev.parent); dport = iter->parent_dport; if (!parent_port_is_cxl_root(parent_port)) { if (coordinates_invalid(&dport->sw_coord)) return -EINVAL; combine_coordinates(&c, &dport->sw_coord); } c.write_latency += dport->link_latency; c.read_latency += dport->link_latency; iter = to_cxl_port(iter->dev.parent); } while (!parent_port_is_cxl_root(iter));or:
Quoted 16 lines
do {
struct cxl_port *parent_port = to_cxl_port(iter->dev.parent);
dport = iter->parent_dport;
c.write_latency += dport->link_latency;
c.read_latency += dport->link_latency;
if (parent_port_is_cxl_root(parent_port))
break;
if (coordinates_invalid(&dport->sw_coord))
return -EINVAL;
combine_coordinates(&c, &dport->sw_coord);
iter = to_cxl_port(iter->dev.parent);
} while (!parent_port_is_cxl_root(iter));Actually, I think if we just make it dport->coord instead of dport->sw_coord and dport->hb_coord, we can remove the check and everything should work out properly.
Quoted 38 lines
Quoted 37 lines
do { if (coordinates_invalid(&dport->sw_coord)) return -EINVAL; combine_coordinates(&c, &dport->sw_coord); iter = to_cxl_port(iter->dev.parent); dport = iter->parent_dport; } while (!parent_port_is_cxl_root(iter)); /* Do final link updates */ c.write_latency += dport->link_latency; c.read_latency += dport->link_latency;Quoted 20 lines
drivers/cxl/core/port.c +9
-2+if (!parent_port_is_cxl_root (iter)) { +if (coordinates_invalid (&dport->sw_coord)) +return -EINVAL ; + +combine_coordinates (&c, &dport->sw_coord); + } + c.write_latency += dport->link_latency; c.read_latency += dport->link_latency; - - iter =to_cxl_port (iter->dev.parent); dport = iter->parent_dport; }/* Augment with the generic port (host bridge) perf data */ +if (coordinates_invalid (&dport->hb_coord)) +return -EINVAL ;combine_coordinates (&c, &dport->hb_coord);/* Get the calculated PCI paths bandwidth */
Quoted 5 lines · Dave Jiang
The
while()loop incxl_endpoint_get_perf_coordinates()checks to see if 'iter' is valid as part of the condition breaking out of the loop. However, iter is being used before the check at the end of the while loop before the next iteration starts. Given that the loop doesn't expect the iter to be NULL because it stops before the root port, remove the iter check.
This smells like a fix, but no "Fix" in the title or "Fixes" tag. So, is this a cleanup or a fix, and if it a fix what are the user visible effects of the bug?
Dave Jiang
maintainerQuoted 10 lines · Dan Williams
Quoted 5 lines · Dave Jiang
The
while()loop incxl_endpoint_get_perf_coordinates()checks to see if 'iter' is valid as part of the condition breaking out of the loop. However, iter is being used before the check at the end of the while loop before the next iteration starts. Given that the loop doesn't expect the iter to be NULL because it stops before the root port, remove the iter check.This smells like a fix, but no "Fix" in the title or "Fixes" tag. So, is this a cleanup or a fix, and if it a fix what are the user visible effects of the bug?
I debated on this and in the end decided that it doesn't need to be a fix because there is no impact whether we check the iter or not in the loop since iter should not ever be NULL.
Quoted 7 lines · On Wed, 28 Feb 2024 17:25:41 -0700 Dave Jiang <[email protected]> wrote:
The
while()loop incxl_endpoint_get_perf_coordinates()checks to see if 'iter' is valid as part of the condition breaking out of the loop. However, iter is being used before the check at the end of the while loop before the next iteration starts. Given that the loop doesn't expect the iter to be NULL because it stops before the root port, remove the iter check.Signed-off-by Dave Jiang <[email protected]>
Quoted 17 lines
--- drivers/cxl/core/port.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/cxl/core/port.c +1
-1@@ -2142,7 +2142,7 @@ int cxl_endpoint_get_perf_coordinates(struct cxl_port *port,* port each iteration. If the parent is cxl root then there is * nothing to gather. */ -while (iter && !is_cxl_root (to_cxl_port (iter->dev.parent))) { +while (!is_cxl_root (to_cxl_port (iter->dev.parent))) {combine_coordinates (&c, &dport->sw_coord); c.write_latency += dport->link_latency; c.read_latency += dport->link_latency;