mlxsw: Improvements

PATCH Applied Petr Machata 3 participants 6 messages Jul 8, 2024 – Jul 10, 2024

Subsystems (4)

Networking Network drivers PCI Mellanox Ethernet switch drivers

lore

Share & tools

lore

Petr Machata

author maintainer

2024-07-08 14:26 UTC #0 lore raw

v2 cover mlxsw: Improvements

This patchset contains assortments of improvements to the mlxsw driver. Please see individual patches for details.

v2:
- Patch #1:
- changed to WARN_ONCE() with some prints
- Patch #2:
- call thermal_cooling_device_unregister() unconditionally and mention it in the commit message
- Patch #3:
- reword the commit message to reflect the fact that the change both suppresses a warning and avoid concurrent access

Ido Schimmel (2):
mlxsw: core_thermal: Report valid current state during cooling device
registration
mlxsw: pci: Lock configuration space of upstream bridge during reset

Petr Machata (1):
mlxsw: Warn about invalid accesses to array fields

 .../ethernet/mellanox/mlxsw/core_thermal.c    | 51 +++++++++----------
 drivers/net/ethernet/mellanox/mlxsw/item.h    |  4 ++
 drivers/net/ethernet/mellanox/mlxsw/pci.c     |  6 +++
 3 files changed, 35 insertions(+), 26 deletions(-)

Signature · 1 line

2.45.0

Petr Machata

author maintainer

2024-07-08 14:26 UTC #1 lore raw

PATCH v2 1/3 mlxsw: Warn about invalid accesses to array fields

A forgotten or buggy variable initialization can cause out-of-bounds access to a register or other item array field. For an overflow, such access would mangle adjacent parts of the register payload. For an underflow, due to all variables being unsigned, the access would likely trample unrelated memory. Since neither is correct, replace these accesses with accesses at the index of 0, and warn about the issue.

Signed-off-by Petr Machata <[email protected]>

Reviewed-by Ido Schimmel <[email protected]>

---

Notes:
v2:
- changed to WARN_ONCE() with some prints

 drivers/net/ethernet/mellanox/mlxsw/item.h | 4 ++++
 1 file changed, 4 insertions(+)

drivers/net/ethernet/mellanox/mlxsw/item.h +4 -0

index cfafbeb42586..a619a0736bd1 100644
@@ -218,6 +218,10 @@ __mlxsw_item_bit_array_offset(const struct mlxsw_item *item,
 	}
 
 	max_index = (item->size.bytes << 3) / item->element_size - 1;
+	if (WARN_ONCE(index > max_index,
+		      "name=%s,index=%u,max_index=%u\n", item->name, index,
+		      max_index))
+		index = 0;
 	be_index = max_index - index;
 	offset = be_index * item->element_size >> 3;
 	in_byte_index  = index % (BITS_PER_BYTE / item->element_size);

Signature · 1 line

2.45.0

Petr Machata

author maintainer

2024-07-08 14:26 UTC #2 lore raw

PATCH v2 2/3 mlxsw: core_thermal: Report valid current state during cooling device registration

In reply to Petr Machata

From: Ido Schimmel <[email protected]>

Commit 31a0fa0019b0 ("thermal/debugfs: Pass cooling device state to thermal_debug_cdev_add()") changed the thermal core to read the current state of the cooling device as part of the cooling device's registration. This is incompatible with the current implementation of the cooling device operations in mlxsw, leading to initialization failure with errors such as:

mlxsw_spectrum 0000:01:00.0: Failed to register cooling device mlxsw_spectrum 0000:01:00.0: cannot register bus device

The reason for the failure is that when the get current state operation is invoked the driver tries to derive the index of the cooling device by walking a per thermal zone array and looking for the matching cooling device pointer. However, the pointer is returned from the registration function and therefore only set in the array after the registration.

The issue was later fixed by commit 1af89dedc8a5 ("thermal: core: Do not fail cdev registration because of invalid initial state") by not failing the registration of the cooling device if it cannot report a valid current state during registration, although drivers are responsible for ensuring that this will not happen.

Therefore, make sure the driver is able to report a valid current state for the cooling device during registration by passing to the registration function a per cooling device private data that already has the cooling device index populated.

While at it, call thermal_cooling_device_unregister() unconditionally since the function returns immediately if the cooling device pointer is NULL.

Cc [email protected]

Reviewed-by Vadim Pasternak <[email protected]>

Signed-off-by Ido Schimmel <[email protected]>

Signed-off-by Petr Machata <[email protected]>

---

Notes:
v2:
- call thermal_cooling_device_unregister() unconditionally and mention it in the commit message

 .../ethernet/mellanox/mlxsw/core_thermal.c    | 51 +++++++++----------
 1 file changed, 25 insertions(+), 26 deletions(-)

drivers/net/ethernet/mellanox/mlxsw/core_thermal.c +25 ~~-26~~

index 5c511e1a8efa..d61478c0c632 100644
@@ -100,6 +100,12 @@ static const struct mlxsw_cooling_states default_cooling_states[] = {
 
 struct mlxsw_thermal;
 
+struct mlxsw_thermal_cooling_device {
+	struct mlxsw_thermal *thermal;
+	struct thermal_cooling_device *cdev;
+	unsigned int idx;
+};
+
 struct mlxsw_thermal_module {
 	struct mlxsw_thermal *parent;
 	struct thermal_zone_device *tzdev;

@@ -123,7 +129,7 @@ struct mlxsw_thermal {
 	const struct mlxsw_bus_info *bus_info;
 	struct thermal_zone_device *tzdev;
 	int polling_delay;
-	struct thermal_cooling_device *cdevs[MLXSW_MFCR_PWMS_MAX];
+	struct mlxsw_thermal_cooling_device cdevs[MLXSW_MFCR_PWMS_MAX];
 	struct thermal_trip trips[MLXSW_THERMAL_NUM_TRIPS];
 	struct mlxsw_cooling_states cooling_states[MLXSW_THERMAL_NUM_TRIPS];
 	struct mlxsw_thermal_area line_cards[];

@@ -147,7 +153,7 @@ static int mlxsw_get_cooling_device_idx(struct mlxsw_thermal *thermal,
 	int i;
 
 	for (i = 0; i < MLXSW_MFCR_PWMS_MAX; i++)
-		if (thermal->cdevs[i] == cdev)
+		if (thermal->cdevs[i].cdev == cdev)
 			return i;
 
 	/* Allow mlxsw thermal zone binding to an external cooling device */

@@ -352,17 +358,14 @@ static int mlxsw_thermal_get_cur_state(struct thermal_cooling_device *cdev,
 				       unsigned long *p_state)
 
 {
-	struct mlxsw_thermal *thermal = cdev->devdata;
+	struct mlxsw_thermal_cooling_device *mlxsw_cdev = cdev->devdata;
+	struct mlxsw_thermal *thermal = mlxsw_cdev->thermal;
 	struct device *dev = thermal->bus_info->dev;
 	char mfsc_pl[MLXSW_REG_MFSC_LEN];
-	int err, idx;
 	u8 duty;
+	int err;
 
-	idx = mlxsw_get_cooling_device_idx(thermal, cdev);
-	if (idx < 0)
-		return idx;
-
-	mlxsw_reg_mfsc_pack(mfsc_pl, idx, 0);
+	mlxsw_reg_mfsc_pack(mfsc_pl, mlxsw_cdev->idx, 0);
 	err = mlxsw_reg_query(thermal->core, MLXSW_REG(mfsc), mfsc_pl);
 	if (err) {
 		dev_err(dev, "Failed to query PWM duty\n");

@@ -378,22 +381,19 @@ static int mlxsw_thermal_set_cur_state(struct thermal_cooling_device *cdev,
 				       unsigned long state)
 
 {
-	struct mlxsw_thermal *thermal = cdev->devdata;
+	struct mlxsw_thermal_cooling_device *mlxsw_cdev = cdev->devdata;
+	struct mlxsw_thermal *thermal = mlxsw_cdev->thermal;
 	struct device *dev = thermal->bus_info->dev;
 	char mfsc_pl[MLXSW_REG_MFSC_LEN];
-	int idx;
 	int err;
 
 	if (state > MLXSW_THERMAL_MAX_STATE)
 		return -EINVAL;
 
-	idx = mlxsw_get_cooling_device_idx(thermal, cdev);
-	if (idx < 0)
-		return idx;
-
 	/* Normalize the state to the valid speed range. */
 	state = max_t(unsigned long, MLXSW_THERMAL_MIN_STATE, state);
-	mlxsw_reg_mfsc_pack(mfsc_pl, idx, mlxsw_state_to_duty(state));
+	mlxsw_reg_mfsc_pack(mfsc_pl, mlxsw_cdev->idx,
+			    mlxsw_state_to_duty(state));
 	err = mlxsw_reg_write(thermal->core, MLXSW_REG(mfsc), mfsc_pl);
 	if (err) {
 		dev_err(dev, "Failed to write PWM duty\n");

@@ -753,17 +753,21 @@ int mlxsw_thermal_init(struct mlxsw_core *core,
 	}
 	for (i = 0; i < MLXSW_MFCR_PWMS_MAX; i++) {
 		if (pwm_active & BIT(i)) {
+			struct mlxsw_thermal_cooling_device *mlxsw_cdev;
 			struct thermal_cooling_device *cdev;
 
+			mlxsw_cdev = &thermal->cdevs[i];
+			mlxsw_cdev->thermal = thermal;
+			mlxsw_cdev->idx = i;
 			cdev = thermal_cooling_device_register("mlxsw_fan",
-							       thermal,
+							       mlxsw_cdev,
 							       &mlxsw_cooling_ops);
 			if (IS_ERR(cdev)) {
 				err = PTR_ERR(cdev);
 				dev_err(dev, "Failed to register cooling device\n");
 				goto err_thermal_cooling_device_register;
 			}
-			thermal->cdevs[i] = cdev;
+			mlxsw_cdev->cdev = cdev;
 		}
 	}

@@ -824,8 +828,7 @@ int mlxsw_thermal_init(struct mlxsw_core *core,
 err_thermal_zone_device_register:
 err_thermal_cooling_device_register:
 	for (i = 0; i < MLXSW_MFCR_PWMS_MAX; i++)
-		if (thermal->cdevs[i])
-			thermal_cooling_device_unregister(thermal->cdevs[i]);
+		thermal_cooling_device_unregister(thermal->cdevs[i].cdev);
 err_reg_write:
 err_reg_query:
 	kfree(thermal);

@@ -847,12 +850,8 @@ void mlxsw_thermal_fini(struct mlxsw_thermal *thermal)
 		thermal->tzdev = NULL;
 	}
 
-	for (i = 0; i < MLXSW_MFCR_PWMS_MAX; i++) {
-		if (thermal->cdevs[i]) {
-			thermal_cooling_device_unregister(thermal->cdevs[i]);
-			thermal->cdevs[i] = NULL;
-		}
-	}
+	for (i = 0; i < MLXSW_MFCR_PWMS_MAX; i++)
+		thermal_cooling_device_unregister(thermal->cdevs[i].cdev);
 
 	kfree(thermal);
 }

Signature · 1 line

2.45.0

Petr Machata

author maintainer

2024-07-08 14:26 UTC #3 lore raw

PATCH v2 3/3 mlxsw: pci: Lock configuration space of upstream bridge during reset

In reply to Petr Machata

From: Ido Schimmel <[email protected]>

The driver triggers a "Secondary Bus Reset" (SBR) by calling __pci_reset_function_locked() which asserts the SBR bit in the "Bridge Control Register" in the configuration space of the upstream bridge for 2ms. This is done without locking the configuration space of the upstream bridge port, allowing user space to access it concurrently.

Linux 6.11 will start warning about such unlocked resets [1][2]:

pcieport 0000:00:01.0: unlocked secondary bus reset via: pci_reset_bus_function+0x51c/0x6a0

Avoid the warning and the concurrent access by locking the configuration space of the upstream bridge prior to the reset and unlocking it afterwards.

[1] https://lore.kernel.org/all/171711746953.1628941.4692125082286867825.stgit@dwillia2-xfh.jf.intel.com/
[2] https://lore.kernel.org/all/20240531213150.GA610983@bhelgaas/

Cc [email protected]

Signed-off-by Ido Schimmel <[email protected]>

Signed-off-by Petr Machata <[email protected]>

---

Notes:
v2:
- reword the commit message to reflect the fact that the change both suppresses a warning and avoid concurrent access

 drivers/net/ethernet/mellanox/mlxsw/pci.c | 6 ++++++
 1 file changed, 6 insertions(+)

drivers/net/ethernet/mellanox/mlxsw/pci.c +6 -0

index 0320dabd1380..060e5b939211 100644
@@ -1784,6 +1784,7 @@ static int mlxsw_pci_reset_at_pci_disable(struct mlxsw_pci *mlxsw_pci,
 {
 	struct pci_dev *pdev = mlxsw_pci->pdev;
 	char mrsr_pl[MLXSW_REG_MRSR_LEN];
+	struct pci_dev *bridge;
 	int err;
 
 	if (!pci_reset_sbr_supported) {

@@ -1800,6 +1801,9 @@ static int mlxsw_pci_reset_at_pci_disable(struct mlxsw_pci *mlxsw_pci,
 sbr:
 	device_lock_assert(&pdev->dev);
 
+	bridge = pci_upstream_bridge(pdev);
+	if (bridge)
+		pci_cfg_access_lock(bridge);
 	pci_cfg_access_lock(pdev);
 	pci_save_state(pdev);

@@ -1809,6 +1813,8 @@ static int mlxsw_pci_reset_at_pci_disable(struct mlxsw_pci *mlxsw_pci,
 
 	pci_restore_state(pdev);
 	pci_cfg_access_unlock(pdev);
+	if (bridge)
+		pci_cfg_access_unlock(bridge);
 
 	return err;
 }

Signature · 1 line

2.45.0

Przemek Kitszel

Reviewed-by on the v2 series

2024-07-09 08:50 UTC #4 lore raw

In reply to Petr Machata

Quoted 18 lines · Petr Machata

This patchset contains assortments of improvements to the mlxsw driver. Please see individual patches for details.
v2:
- Patch #1:
- changed to WARN_ONCE() with some prints
- Patch #2:
- call thermal_cooling_device_unregister() unconditionally and mention it in the commit message
- Patch #3:
- reword the commit message to reflect the fact that the change both suppresses a warning and avoid concurrent access
Ido Schimmel (2):
mlxsw: core_thermal: Report valid current state during cooling device
registration
mlxsw: pci: Lock configuration space of upstream bridge during reset

Quoted 8 lines

Petr Machata (1):
mlxsw: Warn about invalid accesses to array fields

  .../ethernet/mellanox/mlxsw/core_thermal.c    | 51 +++++++++----------
  drivers/net/ethernet/mellanox/mlxsw/item.h    |  4 ++
  drivers/net/ethernet/mellanox/mlxsw/pci.c     |  6 +++
  3 files changed, 35 insertions(+), 26 deletions(-)

for the series:

Reviewed-by Przemek Kitszel <[email protected]>

[email protected]

CI Applied

2024-07-10 02:10 UTC #5 lore raw

In reply to Petr Machata

This series was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:

Hello:

This series was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:

Quoted 13 lines · On Mon, 8 Jul 2024 16:23:39 +0200 you wrote:

This patchset contains assortments of improvements to the mlxsw driver. Please see individual patches for details.
v2:
- Patch #1:
- changed to WARN_ONCE() with some prints
- Patch #2:
- call thermal_cooling_device_unregister() unconditionally and mention it in the commit message
- Patch #3:
- reword the commit message to reflect the fact that the change both suppresses a warning and avoid concurrent access

Quoted 1 line

[...]

Here is the summary with links:
- [net-next,v2,1/3] mlxsw: Warn about invalid accesses to array fields

    https://git.kernel.org/netdev/net-next/c/b45c76e5f43f
  - [net-next,v2,2/3] mlxsw: core_thermal: Report valid current state during cooling device registration
    https://git.kernel.org/netdev/net-next/c/a22f3bc80075
  - [net-next,v2,3/3] mlxsw: pci: Lock configuration space of upstream bridge during reset
    https://git.kernel.org/netdev/net-next/c/0970836c348b

You are awesome, thank you!

Signature · 2 lines

Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html