cxl: access_coordinate validity fixes for 6.9
Dave Jiang
maintainerv7 cover cxl: access_coordinate validity fixes for 6.9
v7:
- Add bug fix for not skipping RP coords that don't exist. (Jonathan)
- Remove min_not_zero() as it doesn't apply here. DIV_ROUNDUP() is sufficient. (Jonathan)
- Reflowed patch ordering for easier backport.
A group of fixes to make CXL access_coordinate calculation function properly.
[1/5] cxl: Remove checking of iter in cxl_endpoint_get_perf_coordinates() [2/5] cxl: Fix retrieving of access_coordinates in PCIe path
[3/5] cxl: Fix incorrect region perf data calculation
[4/5] cxl: Consolidate dport access_coordinate ->hb_coord and ->sw_coord into ->coord [5/5] cxl: Add checks to access_coordinate calculation to fail missing data
Dave Jiang
maintainerPATCH v7 1/5 cxl: Remove checking of iter in cxl_endpoint_get_perf_coordinates()
The while() loop in cxl_endpoint_get_perf_coordinates() checks to see if 'iter' is valid as part of the condition breaking out of the loop. is_cxl_root() will stop the loop before the next iteration could go NULL. Remove the iter check.
The presence of the iter or removing the iter does not impact the behavior of the code. This is a code clean up and not a bug fix.
--- drivers/cxl/core/port.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/cxl/core/port.c +1 -1
@@ -2197,7 +2197,7 @@ int cxl_endpoint_get_perf_coordinates(struct cxl_port *port,* port each iteration. If the parent is cxl root then there is * nothing to gather. */ -while (iter && !is_cxl_root (to_cxl_port (iter->dev.parent))) { +while (!is_cxl_root (to_cxl_port (iter->dev.parent))) {cxl_coordinates_combine (&c, &c, &dport->sw_coord); c.write_latency += dport->link_latency; c.read_latency += dport->link_latency;
Signature · 1 line
2.44.0
Re: [PATCH v7 1/5] cxl: Remove checking of iter in cxl_endpoint_get_perf_coordinates()
Quoted 21 lines · Dave Jiang
The
while()loop incxl_endpoint_get_perf_coordinates()checks to see if 'iter' is valid as part of the condition breaking out of the loop.is_cxl_root()will stop the loop before the next iteration could go NULL. Remove the iter check.The presence of the iter or removing the iter does not impact the behavior of the code. This is a code clean up and not a bug fix.
Reviewed-by Jonathan Cameron <[email protected]>Reviewed-by Davidlohr Bueso <[email protected]>Reviewed-by Dan Williams <[email protected]>Signed-off-by Dave Jiang <[email protected]>--- drivers/cxl/core/port.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/cxl/core/port.c +0
-0
drivers/cxl/core/port.c int cxl_endpoint_get_perf_coordinates(struct cxl_port *port, · line 2200
Quoted 5 lines
drivers/cxl/core/port.c +1
-1@@ -2197,7 +2197,7 @@ int cxl_endpoint_get_perf_coordinates(struct cxl_port *port,* port each iteration. If the parent is cxl root then there is * nothing to gather. */ -while (iter && !is_cxl_root (to_cxl_port (iter->dev.parent))) { +while (!is_cxl_root (to_cxl_port (iter->dev.parent))) {
I am seeing the following mainline after [1]:
[ 39.815379] cxl_acpi ACPI0017:00: not a cxl_port device [ 39.827123] WARNING: CPU: 46 PID: 1754 at drivers/cxl/core/port.c:592 to_cxl_port+0x56/0x70 [cxl_core]
... plus some related subsequent NULL pointer dereference:
[ 40.718708] BUG: kernel NULL pointer dereference, address: 00000000000002d8
This changes looks related. I am going to dig deeper here but just a headsup in advance.
Note this is tested on an RCH topology.
Thanks,
-Robert
[1] commit 586b5dfb51b9 ("Merge tag 'cxl-fixes-6.9-rc4' of
git://git.kernel.org/pub/scm/linux/kernel/git/cxl/cxl"):
Quoted 6 lines
drivers/cxl/core/port.c +0
-0cxl_coordinates_combine (&c, &c, &dport->sw_coord); c.write_latency += dport->link_latency; c.read_latency += dport->link_latency;Signature · 1 line
2.44.0
Dave Jiang
maintainerRe: [PATCH v7 1/5] cxl: Remove checking of iter in cxl_endpoint_get_perf_coordinates()
Quoted 33 lines · Robert Richter
Quoted 21 lines · Dave Jiang
The
while()loop incxl_endpoint_get_perf_coordinates()checks to see if 'iter' is valid as part of the condition breaking out of the loop.is_cxl_root()will stop the loop before the next iteration could go NULL. Remove the iter check.The presence of the iter or removing the iter does not impact the behavior of the code. This is a code clean up and not a bug fix.
Reviewed-by Jonathan Cameron <[email protected]>Reviewed-by Davidlohr Bueso <[email protected]>Reviewed-by Dan Williams <[email protected]>Signed-off-by Dave Jiang <[email protected]>--- drivers/cxl/core/port.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/cxl/core/port.c +0
-0drivers/cxl/core/port.c int cxl_endpoint_get_perf_coordinates(struct cxl_port *port, · line 2200
Quoted 5 lines
drivers/cxl/core/port.c +1
-1@@ -2197,7 +2197,7 @@ int cxl_endpoint_get_perf_coordinates(struct cxl_port *port,* port each iteration. If the parent is cxl root then there is * nothing to gather. */ -while (iter && !is_cxl_root (to_cxl_port (iter->dev.parent))) { +while (!is_cxl_root (to_cxl_port (iter->dev.parent))) {I am seeing the following mainline after [1]:
[ 39.815379] cxl_acpi ACPI0017:00: not a cxl_port device [ 39.827123] WARNING: CPU: 46 PID: 1754 at drivers/cxl/core/port.c:592 to_cxl_port+0x56/0x70 [cxl_core]
Quoted 8 lines
... plus some related subsequent NULL pointer dereference:
[ 40.718708] BUG: kernel NULL pointer dereference, address: 00000000000002d8This changes looks related. I am going to dig deeper here but just a headsup in advance.
Note this is tested on an RCH topology.
Hi Robert,
Can you please provide the 'ls -l /sys/bus/cxl/devices/$memX' of the RCH device? Maybe the iterator may need to be tweaked for RCH topology support.
Quoted 15 lines
Thanks,
-Robert
[1] commit
586b5dfb51b9("Merge tag 'cxl-fixes-6.9-rc4' ofgit://git.kernel.org/pub/scm/linux/kernel/git/cxl/cxl"):Quoted 6 lines
drivers/cxl/core/port.c +0
-0cxl_coordinates_combine (&c, &c, &dport->sw_coord); c.write_latency += dport->link_latency; c.read_latency += dport->link_latency;Signature · 1 line
2.44.0
Dave Jiang
maintainerPATCH v7 2/5 cxl: Fix retrieving of access_coordinates in PCIe path
In reply to Dave JiangCurrent loop in cxl_endpoint_get_perf_coordinates() incorrectly assumes the Root Port (RP) dport is the one with generic port access_coordinate. However those coordinates are one level up in the Host Bridge (HB). Current code causes the computation code to pick up 0s as the coordinates and cause minimal bandwidth to result in 0.
Add check to skip RP when combining coordinates.
--- drivers/cxl/core/port.c | 35 ++++++++++++++++++++++------------- 1 file changed, 22 insertions(+), 13 deletions(-)
drivers/cxl/core/port.c +22 -13
@@ -2165,6 +2165,11 @@ int cxl_hb_get_perf_coordinates(struct cxl_port *port,return 0 ; } +static bool parent_port_is_cxl_root (struct cxl_port *port) +{ +return is_cxl_root (to_cxl_port (port->dev.parent)); +} +/** * cxl_endpoint_get_perf_coordinates - Retrieve performance numbers stored in dports * of CXL path
@@ -2184,27 +2189,31 @@ int cxl_endpoint_get_perf_coordinates(struct cxl_port *port,struct cxl_dport *dport;struct pci_dev *pdev;unsigned int bw; +bool is_cxl_root;if (!is_cxl_endpoint (port))return -EINVAL ; - dport = iter->parent_dport; -/* -* Exit the loop when the parent port of the current port is cxl root. -* The iterative loop starts at the endpoint and gathers the -* latency of the CXL link from the current iter to the next downstream -* port each iteration. If the parent is cxl root then there is -* nothing to gather. +* Exit the loop when the parent port of the current iter port is cxl +* root. The iterative loop starts at the endpoint and gathers the +* latency of the CXL link from the current device/port to the connected +* downstream port each iteration. */ -while (!is_cxl_root (to_cxl_port (iter->dev.parent))) { -cxl_coordinates_combine (&c, &c, &dport->sw_coord); +do { + dport = iter->parent_dport; + iter =to_cxl_port (iter->dev.parent); + is_cxl_root =parent_port_is_cxl_root (iter); + +/* +* There's no valid access_coordinate for a root port since RPs do not +* have CDAT and therefore needs to be skipped. +*/ +if (!is_cxl_root) +cxl_coordinates_combine (&c, &c, &dport->sw_coord); c.write_latency += dport->link_latency; c.read_latency += dport->link_latency; - - iter =to_cxl_port (iter->dev.parent); - dport = iter->parent_dport; - } + }while (!is_cxl_root);/* Get the calculated PCI paths bandwidth */ pdev =to_pci_dev (port->uport_dev->parent);
Signature · 1 line
2.44.0
Re: [PATCH v7 2/5] cxl: Fix retrieving of access_coordinates in PCIe path
Quoted 11 lines · On Wed, 3 Apr 2024 08:47:13 -0700 Dave Jiang <[email protected]> wrote:
Current loop in
cxl_endpoint_get_perf_coordinates()incorrectly assumes the Root Port (RP) dport is the one with generic portaccess_coordinate. However those coordinates are one level up in the Host Bridge (HB). Current code causes the computation code to pick up 0s as the coordinates and cause minimal bandwidth to result in 0.Add check to skip RP when combining coordinates.
Fixes 4d59ba915318 ("cxl: Fix retrieving of access_coordinates in PCIe path")Reported-by Jonathan Cameron <[email protected]>Signed-off-by Dave Jiang <[email protected]>
I've only tested this with direct connect, but can't see why it wouldn't work with switches in the path. So
Quoted 65 lines
--- drivers/cxl/core/port.c | 35 ++++++++++++++++++++++------------- 1 file changed, 22 insertions(+), 13 deletions(-)drivers/cxl/core/port.c +22
-13@@ -2165,6 +2165,11 @@ int cxl_hb_get_perf_coordinates(struct cxl_port *port,return 0 ; } +static bool parent_port_is_cxl_root (struct cxl_port *port) +{ +return is_cxl_root (to_cxl_port (port->dev.parent)); +} +/** * cxl_endpoint_get_perf_coordinates - Retrieve performance numbers stored in dports * of CXL path @@ -2184,27 +2189,31 @@ int cxl_endpoint_get_perf_coordinates(struct cxl_port *port,struct cxl_dport *dport;struct pci_dev *pdev;unsigned int bw; +bool is_cxl_root;if (!is_cxl_endpoint (port))return -EINVAL ; - dport = iter->parent_dport; -/* -* Exit the loop when the parent port of the current port is cxl root. -* The iterative loop starts at the endpoint and gathers the -* latency of the CXL link from the current iter to the next downstream -* port each iteration. If the parent is cxl root then there is -* nothing to gather. +* Exit the loop when the parent port of the current iter port is cxl +* root. The iterative loop starts at the endpoint and gathers the +* latency of the CXL link from the current device/port to the connected +* downstream port each iteration. */ -while (!is_cxl_root (to_cxl_port (iter->dev.parent))) { -cxl_coordinates_combine (&c, &c, &dport->sw_coord); +do { + dport = iter->parent_dport; + iter =to_cxl_port (iter->dev.parent); + is_cxl_root =parent_port_is_cxl_root (iter); + +/* +* There's no valid access_coordinate for a root port since RPs do not +* have CDAT and therefore needs to be skipped. +*/ +if (!is_cxl_root) +cxl_coordinates_combine (&c, &c, &dport->sw_coord); c.write_latency += dport->link_latency; c.read_latency += dport->link_latency; - - iter =to_cxl_port (iter->dev.parent); - dport = iter->parent_dport; - } + }while (!is_cxl_root);/* Get the calculated PCI paths bandwidth */ pdev =to_pci_dev (port->uport_dev->parent);
RE: [PATCH v7 2/5] cxl: Fix retrieving of access_coordinates in PCIe path
In reply to Dave JiangQuoted 11 lines · Dave Jiang
Current loop in
cxl_endpoint_get_perf_coordinates()incorrectly assumes the Root Port (RP) dport is the one with generic portaccess_coordinate. However those coordinates are one level up in the Host Bridge (HB). Current code causes the computation code to pick up 0s as the coordinates and cause minimal bandwidth to result in 0.Add check to skip RP when combining coordinates.
Fixes 4d59ba915318 ("cxl: Fix retrieving of access_coordinates in PCIe path")Reported-by Jonathan Cameron <[email protected]>Signed-off-by Dave Jiang <[email protected]>
Looks ok to me, and I see the Generic Port details are considered right after this walk to the root-port.
Dave Jiang
maintainerPATCH v7 3/5 cxl: Fix incorrect region perf data calculation
In reply to Dave JiangCurrent math in cxl_region_perf_data_calculate divides the latency by 1000 every time the function gets called. This causes the region latency to be divided by 1000 per memory device and the math is incorrect. This is user visible as the latency access_coordinate exposed via sysfs will show incorrect latency data.
Normalize values from CDAT to nanoseconds. Adjust sub-nanoseconds latency to at least 1. Remove adjustment of perf numbers from the generic target since hmat handling code has already normalized those numbers. Now all computation and stored numbers should be in nanoseconds.
cxl_hb_get_perf_coordinates() is removed and HB coords are calculated in the port access_coordinate calculation path since it no longer need to be treated special.
---
v7:
- Remove min_not_zero(). Incorrectly set everything to 1. DIV_ROUNDUP() will ensure sub-nanoseconds values not 0 unless value 0 to begin with. (Jonathan)
- Reflowed patch order
- Remove cxl_hb_get_perf_coordinates() as change made function unnessary.
- Add hb access_coordinate back to port caclculation.
--- drivers/cxl/acpi.c | 13 +----- drivers/cxl/core/cdat.c | 89 ++++++++++++++++------------------------- drivers/cxl/core/port.c | 36 ++--------------- drivers/cxl/cxl.h | 2 - 4 files changed, 40 insertions(+), 100 deletions(-)
drivers/cxl/acpi.c +1 -12
@@ -525,22 +525,11 @@ static int get_genport_coordinates(struct device *dev, struct cxl_dport *dport) {struct acpi_device *hb =to_cxl_host_bridge (NULL , dev);u32 uid; -int rc;if (kstrtou32 (acpi_device_uid (hb),0 , &uid))return -EINVAL ; - rc =acpi_get_genport_coordinates (uid, dport->hb_coord); -if (rc <0 ) -return rc; - -/* Adjust back to picoseconds from nanoseconds */ -for (int i =0 ; i <ACCESS_COORDINATE_MAX ; i++) { - dport->hb_coord[i].read_latency *=1000 ; - dport->hb_coord[i].write_latency *=1000 ; - } - -return 0 ; +return acpi_get_genport_coordinates (uid, dport->hb_coord); }static int add_host_bridge_dport (struct device *match,void *arg)
drivers/cxl/core/cdat.c +35 -54
157 lines not on this page · raw message
drivers/cxl/core/port.c +4 -32
@@ -2133,38 +2133,6 @@ bool schedule_cxl_memdev_detach(struct cxl_memdev *cxlmd) }EXPORT_SYMBOL_NS_GPL (schedule_cxl_memdev_detach,CXL ); -/** -* cxl_hb_get_perf_coordinates - Retrieve performance numbers between initiator -* and host bridge -* -* @port: endpoint cxl_port -* @coord: output access coordinates -* -* Return: errno on failure, 0 on success. -*/ -int cxl_hb_get_perf_coordinates (struct cxl_port *port, -struct access_coordinate *coord) -{ -struct cxl_port *iter = port; -struct cxl_dport *dport; - -if (!is_cxl_endpoint (port)) -return -EINVAL ; - - dport = iter->parent_dport; -while (iter && !is_cxl_root (to_cxl_port (iter->dev.parent))) { - iter =to_cxl_port (iter->dev.parent); - dport = iter->parent_dport; - } - - coord[ACCESS_COORDINATE_LOCAL ] = - dport->hb_coord[ACCESS_COORDINATE_LOCAL ]; - coord[ACCESS_COORDINATE_CPU ] = - dport->hb_coord[ACCESS_COORDINATE_CPU ]; - -return 0 ; -} -static bool parent_port_is_cxl_root (struct cxl_port *port) {return is_cxl_root (to_cxl_port (port->dev.parent));
@@ -2215,6 +2183,10 @@ int cxl_endpoint_get_perf_coordinates(struct cxl_port *port, c.read_latency += dport->link_latency; }while (!is_cxl_root); + dport = iter->parent_dport; +/* Retrieve HB coords */ +cxl_coordinates_combine (&c, &c, dport->hb_coord); +/* Get the calculated PCI paths bandwidth */ pdev =to_pci_dev (port->uport_dev->parent); bw =pcie_bandwidth_available (pdev,NULL ,NULL ,NULL );
drivers/cxl/cxl.h +0 -2
@@ -884,8 +884,6 @@ void cxl_switch_parse_cdat(struct cxl_port *port);int cxl_endpoint_get_perf_coordinates (struct cxl_port *port,struct access_coordinate *coord); -int cxl_hb_get_perf_coordinates (struct cxl_port *port, -struct access_coordinate *coord);void cxl_region_perf_data_calculate (struct cxl_region *cxlr,struct cxl_endpoint_decoder *cxled);
Signature · 1 line
2.44.0
Re: [PATCH v7 3/5] cxl: Fix incorrect region perf data calculation
Quoted 11 lines · On Wed, 3 Apr 2024 08:47:14 -0700 Dave Jiang <[email protected]> wrote:
Current math in
cxl_region_perf_data_calculatedivides the latency by 1000 every time the function gets called. This causes the region latency to be divided by 1000 per memory device and the math is incorrect. This is user visible as the latencyaccess_coordinateexposed via sysfs will show incorrect latency data.Normalize values from CDAT to nanoseconds. Adjust sub-nanoseconds latency to at least 1. Remove adjustment of perf numbers from the generic target since hmat handling code has already normalized those numbers. Now all computation and stored numbers should be in nanoseconds.
Quoted 6 lines
cxl_hb_get_perf_coordinates()is removed and HB coords are calculated in the portaccess_coordinatecalculation path since it no longer need to be treated special.Fixes 3d9f4a197230 ("cxl/region: Calculate performance data for a region")Signed-off-by Dave Jiang <[email protected]>
I should stop reading this code...
What happens with the bandwidth if the minimum point on path to EP is shared? Gets more complex as maybe the shared bit wasn't the minimum bandwidth previously but when it's 'split' across multiple paths it becomes so.
E.g. HEre the min on each path is 5, but the bottleneck is actually the RP to switch at 8 once we are interleaving across EP0 and EP1.
CPU
|
HB
|
RP
|
<min BW here = 8>
|
SWITCH
| |<each of these BW 5>
EP0 EP1
None of this mattered with traditional HMAT entries because they are point to point so if such interleaving is going on it was a problem for the BIOS writer...
Not related to what you are fixing here though so
Quoted 23 lines
drivers/cxl/core/cdat.c +0
-5@@ -521,17 +525,13 @@ void cxl_region_perf_data_calculate(struct cxl_region *cxlr, struct cxl_endpoint_decoder *cxled) {struct cxl_memdev *cxlmd =cxled_to_memdev (cxled); -struct cxl_port *port = cxlmd->endpoint;struct cxl_dev_state *cxlds = cxlmd->cxlds;struct cxl_memdev_state *mds =to_cxl_memdev_state (cxlds); -struct access_coordinate hb_coord[ACCESS_COORDINATE_MAX ]; -struct access_coordinate coord;struct range dpa = { .start = cxled->dpa_res->start, .end = cxled->dpa_res->end, };struct cxl_dpa_perf *perf; -int rc;switch (cxlr->mode) {case CXL_DECODER_RAM :@@ -549,35 +549,16 @@ void cxl_region_perf_data_calculate(struct cxl_region *cxlr,if (!range_contains (&perf->dpa_range, &dpa))return ; - rc =cxl_hb_get_perf_coordinates (port, hb_coord);
drivers/cxl/core/cdat.c void cxl_region_perf_data_calculate(struct cxl_region *cxlr, · line 553
Quoted 10 lines
drivers/cxl/core/cdat.c +0
-8-if (rc) { -dev_dbg (&port->dev,"Failed to retrieve hb perf coordinates.\n" ); -return ; - } -for (int i =0 ; i <ACCESS_COORDINATE_MAX ; i++) { -/* Pickup the host bridge coords */ -cxl_coordinates_combine (&coord, &hb_coord[i], &perf->coord); -/* Get total bandwidth and the worst latency for the cxl region */
Worst latency from what set of choices? Perhaps useful to call that out (multiple EP paths?)
Quoted 12 lines
drivers/cxl/core/cdat.c +1
-7cxlr->coord[i].read_latency = max_t(unsigned int, cxlr->coord[i].read_latency, - coord.read_latency); + perf->coord.read_latency); cxlr->coord[i].write_latency = max_t(unsigned int, cxlr->coord[i].write_latency, - coord.write_latency); - cxlr->coord[i].read_bandwidth += coord.read_bandwidth; - cxlr->coord[i].write_bandwidth += coord.write_bandwidth; - -/* -* Convert latency to nanosec from picosec to be consistent
drivers/cxl/core/cdat.c void cxl_region_perf_data_calculate(struct cxl_region *cxlr, · line 561
Quoted 10 lines
drivers/cxl/core/cdat.c +3
-7-* with the resulting latency coordinates computed by the -* HMAT_REPORTING code. -*/ - cxlr->coord[i].read_latency = -DIV_ROUND_UP (cxlr->coord[i].read_latency,1000 ); - cxlr->coord[i].write_latency = -DIV_ROUND_UP (cxlr->coord[i].write_latency,1000 ); + perf->coord.write_latency); + cxlr->coord[i].read_bandwidth += perf->coord.read_bandwidth; + cxlr->coord[i].write_bandwidth += perf->coord.write_bandwidth;
As above, this might be the same bandwidth we are double counting...
Quoted 4 lines
(quoted diff) +0
-0} }
RE: [PATCH v7 3/5] cxl: Fix incorrect region perf data calculation
In reply to Dave JiangQuoted 25 lines · Dave Jiang
Current math in
cxl_region_perf_data_calculatedivides the latency by 1000 every time the function gets called. This causes the region latency to be divided by 1000 per memory device and the math is incorrect. This is user visible as the latencyaccess_coordinateexposed via sysfs will show incorrect latency data.Normalize values from CDAT to nanoseconds. Adjust sub-nanoseconds latency to at least 1. Remove adjustment of perf numbers from the generic target since hmat handling code has already normalized those numbers. Now all computation and stored numbers should be in nanoseconds.
cxl_hb_get_perf_coordinates()is removed and HB coords are calculated in the portaccess_coordinatecalculation path since it no longer need to be treated special.Fixes 3d9f4a197230 ("cxl/region: Calculate performance data for a region")Signed-off-by Dave Jiang <[email protected]>---
v7:
- Removemin_not_zero(). Incorrectly set everything to 1.DIV_ROUNDUP()will ensure sub-nanoseconds values not 0 unless value 0 to begin with. (Jonathan)
- Reflowed patch order
- Removecxl_hb_get_perf_coordinates()as change made function unnessary.
- Add hbaccess_coordinateback to port caclculation.
Quoted 6 lines
--- drivers/cxl/acpi.c | 13 +----- drivers/cxl/core/cdat.c | 89 ++++++++++++++++------------------------- drivers/cxl/core/port.c | 36 ++--------------- drivers/cxl/cxl.h | 2 - 4 files changed, 40 insertions(+), 100 deletions(-)
yum... bug fix that removes more than double the code it adds.
[..]
Quoted 11 lines
drivers/cxl/core/cdat.c +3
-0@@ -20,6 +20,31 @@ struct dsmas_entry {int qos_class; }; +static u32 cdat_normalize (u16 entry,u64 base,u8 type) +{ +u32 value;
drivers/cxl/core/cdat.c struct dsmas_entry { · line 35
Quoted 10 lines
drivers/cxl/core/cdat.c +10
-0+ +/* +* Check for invalid and overflow values +*/ +if (entry ==0xffff || !entry) +return 0 ; +else if (base > (UINT_MAX / (entry))) +return 0 ; + + value = entry * base;
Might be worth a reminder comment here that CDAT fields follow the format of HMAT fields when a future reader wonders why these type names are not CDAT_ACCESS_LATENCY, etc. Bonus points for a "see Table 5 Device Scoped Latency and Bandwidth Information Structure in Coherent Device Attribute Table (CDAT) Specification v1.01"
Either way you can add:
Re: [PATCH v7 3/5] cxl: Fix incorrect region perf data calculation
Quoted 55 lines · On Fri, 5 Apr 2024 15:34:25 -0700 Dan Williams <[email protected]> wrote:
Quoted 25 lines · Dave Jiang
Current math in
cxl_region_perf_data_calculatedivides the latency by 1000 every time the function gets called. This causes the region latency to be divided by 1000 per memory device and the math is incorrect. This is user visible as the latencyaccess_coordinateexposed via sysfs will show incorrect latency data.Normalize values from CDAT to nanoseconds. Adjust sub-nanoseconds latency to at least 1. Remove adjustment of perf numbers from the generic target since hmat handling code has already normalized those numbers. Now all computation and stored numbers should be in nanoseconds.
cxl_hb_get_perf_coordinates()is removed and HB coords are calculated in the portaccess_coordinatecalculation path since it no longer need to be treated special.Fixes 3d9f4a197230 ("cxl/region: Calculate performance data for a region")Signed-off-by Dave Jiang <[email protected]>---
v7:
- Removemin_not_zero(). Incorrectly set everything to 1.DIV_ROUNDUP()will ensure sub-nanoseconds values not 0 unless value 0 to begin with. (Jonathan)
- Reflowed patch order
- Removecxl_hb_get_perf_coordinates()as change made function unnessary.
- Add hbaccess_coordinateback to port caclculation.Quoted 6 lines
--- drivers/cxl/acpi.c | 13 +----- drivers/cxl/core/cdat.c | 89 ++++++++++++++++------------------------- drivers/cxl/core/port.c | 36 ++--------------- drivers/cxl/cxl.h | 2 - 4 files changed, 40 insertions(+), 100 deletions(-)yum... bug fix that removes more than double the code it adds.
[..]
Quoted 19 lines
drivers/cxl/core/cdat.c +11
-0@@ -20,6 +20,31 @@ struct dsmas_entry {int qos_class; }; +static u32 cdat_normalize (u16 entry,u64 base,u8 type) +{ +u32 value; + +/* +* Check for invalid and overflow values +*/ +if (entry ==0xffff || !entry) +return 0 ; +else if (base > (UINT_MAX / (entry))) +return 0 ;
Quoted 8 lines
drivers/cxl/core/cdat.c struct dsmas_entry { · line 35
Quoted 2 lines
drivers/cxl/core/cdat.c +2
-0+ + value = entry * base;Might be worth a reminder comment here that CDAT fields follow the format of HMAT fields when a future reader wonders why these type names are not
CDAT_ACCESS_LATENCY, etc. Bonus points for a "see Table 5 Device Scoped Latency and Bandwidth Information Structure in Coherent Device Attribute Table (CDAT) Specification v1.01"
Call out a specific HMAT version if you do add such a comment. We don't want anyone to happen to look at an early version and get even more confused :(
Quoted 4 lines
Either way you can add:
Reviewed-by Dan Williams <[email protected]>
Dave Jiang
maintainerPATCH v7 4/5 cxl: Consolidate dport access_coordinate ->hb_coord and ->sw_coord into ->coord
In reply to Dave JiangThe driver stores access_coordinate for host bridge in ->hb_coord and switch CDAT access_coordinate in ->sw_coord. Since neither of these access_coordinate clobber each other, the variable name can be consolidated into ->coord to simplify the code.
---
v7:
- Adjusted based on code changes from pervious patches
--- drivers/cxl/acpi.c | 2 +- drivers/cxl/core/cdat.c | 74 +++++++++++++++++++++++------------- drivers/cxl/core/port.c | 48 +++++++++++++++++------ drivers/cxl/cxl.h | 6 +-- drivers/cxl/cxlmem.h | 2 +- tools/testing/cxl/test/cxl.c | 10 +++-- 6 files changed, 94 insertions(+), 48 deletions(-)
drivers/cxl/acpi.c +1 -1
@@ -529,7 +529,7 @@ static int get_genport_coordinates(struct device *dev, struct cxl_dport *dport)if (kstrtou32 (acpi_device_uid (hb),0 , &uid))return -EINVAL ; -return acpi_get_genport_coordinates (uid, dport->hb_coord); +return acpi_get_genport_coordinates (uid, dport->coord); }static int add_host_bridge_dport (struct device *match,void *arg)
drivers/cxl/core/cdat.c +47 -27
155 lines not on this page · raw message
drivers/cxl/core/port.c +37 -11
76 lines not on this page · raw message
drivers/cxl/cxl.h +2 -4
@@ -663,8 +663,7 @@ struct cxl_rcrb_info {* @rch: Indicate whether this dport was enumerated in RCH or VH mode * @port: reference to cxl_port that contains this downstream port * @regs: Dport parsed register blocks -* @sw_coord: access coordinates (performance) for switch from CDAT -* @hb_coord: access coordinates (performance) from ACPI generic port (host bridge) +* @coord: access coordinates (bandwidth and latency performance attributes) * @link_latency: calculated PCIe downstream latency */ struct cxl_dport {
@@ -675,8 +674,7 @@ struct cxl_dport {bool rch;struct cxl_port *port;struct cxl_regs regs; -struct access_coordinate sw_coord; -struct access_coordinate hb_coord[ACCESS_COORDINATE_MAX ]; +struct access_coordinate coord[ACCESS_COORDINATE_MAX ];long link_latency; };
drivers/cxl/cxlmem.h +1 -1
@@ -401,7 +401,7 @@ enum cxl_devtype {*/ struct cxl_dpa_perf {struct range dpa_range; -struct access_coordinate coord; +struct access_coordinate coord[ACCESS_COORDINATE_MAX ];int qos_class; };
tools/testing/cxl/test/cxl.c +6 -4
@@ -986,10 +986,12 @@ static void dpa_perf_setup(struct cxl_port *endpoint, struct range *range, { dpa_perf->qos_class =FAKE_QTG_ID ; dpa_perf->dpa_range = *range; - dpa_perf->coord.read_latency =500 ; - dpa_perf->coord.write_latency =500 ; - dpa_perf->coord.read_bandwidth =1000 ; - dpa_perf->coord.write_bandwidth =1000 ; +for (int i =0 ; i <ACCESS_COORDINATE_MAX ; i++) { + dpa_perf->coord[i].read_latency =500 ; + dpa_perf->coord[i].write_latency =500 ; + dpa_perf->coord[i].read_bandwidth =1000 ; + dpa_perf->coord[i].write_bandwidth =1000 ; + } }static void mock_cxl_endpoint_parse_cdat (struct cxl_port *port)
Signature · 1 line
2.44.0
Dave Jiang
maintainerPATCH v7 5/5 cxl: Add checks to access_coordinate calculation to fail missing data
In reply to Dave JiangJonathan noted that when the coordinates for host bridge and switches can be 0s if no actual data are retrieved and the calculation continues. The resulting number would be inaccurate. Add checks to ensure that the calculation would complete only if the numbers are valid.
While not seen in the wild, issue may show up with a BIOS that reported CXL root ports via Generic Ports (via a PCI handle in the SRAT entry).
--- drivers/cxl/core/port.c | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-)
drivers/cxl/core/port.c +18 -1
@@ -2141,6 +2141,18 @@ static void add_latency(struct access_coordinate *c, long latency) } } +static bool coordinates_valid (struct access_coordinate *c) +{ +for (int i =0 ; i <ACCESS_COORDINATE_MAX ; i++) { +if (c[i].read_bandwidth && c[i].write_bandwidth && + c[i].read_latency && c[i].write_latency) +continue ; +return false; + } + +return true; +} +static void set_min_bandwidth (struct access_coordinate *c,unsigned int bw) {for (int i =0 ; i <ACCESS_COORDINATE_MAX ; i++) {
@@ -2206,13 +2218,18 @@ int cxl_endpoint_get_perf_coordinates(struct cxl_port *port,* There's no valid access_coordinate for a root port since RPs do not * have CDAT and therefore needs to be skipped. */ -if (!is_cxl_root) +if (!is_cxl_root) { +if (!coordinates_valid (dport->coord)) +return -EINVAL ;cxl_coordinates_combine (c, c, dport->coord); + }add_latency (c, dport->link_latency); }while (!is_cxl_root); dport = iter->parent_dport;/* Retrieve HB coords */ +if (!coordinates_valid (dport->coord)) +return -EINVAL ;cxl_coordinates_combine (c, c, dport->coord);/* Get the calculated PCI paths bandwidth */
Signature · 1 line
2.44.0
RE: [PATCH v7 5/5] cxl: Add checks to access_coordinate calculation to fail missing data
Quoted 5 lines · Dave Jiang
Jonathan noted that when the coordinates for host bridge and switches can be 0s if no actual data are retrieved and the calculation continues. The resulting number would be inaccurate. Add checks to ensure that the calculation would complete only if the numbers are valid.
Quoted 8 lines
While not seen in the wild, issue may show up with a BIOS that reported CXL root ports via Generic Ports (via a PCI handle in the SRAT entry).
Fixes 14a6960b3e92 ("cxl: Add helper function that calculate performance data for downstream ports")Reported-by Jonathan Cameron <[email protected]>Reviewed-by Jonathan Cameron <[email protected]>Reviewed-by Davidlohr Bueso <[email protected]>Signed-off-by Dave Jiang <[email protected]>
Looks good,