Skip to contentSkip to conversation

mlxsw: Initialize register query payloads

PATCH Applied Til Kaiser 4 participants 7 messages Oct 14 – Oct 20

Subsystems (4)
lore
Share & tools
lore

Til Kaiser

author
#0 lore raw

PATCH 1/1 mlxsw: Initialize register query payloads

MFGD register queries can exhaust EMAD retries and leave Mellanox Spectrum switches stuck during boot or shutdown when running OpenWrt built with GCC 15.

An example failure is shown below with the second line wrapped:

EMAD retries (5/5) (tid=9b544fc100001f35)
EMAD reg access failed (tid=9b544fc100001f35,reg_id=90f0(mfgd),
type=query,status=0(operation performed))

On an SN2100, the failure occurs during driver teardown on reboot. On an SN3420, it occurs while probing mlxsw_spectrum during boot.

The register query API transmits the supplied payload to the device before replacing it with the response. Both MFGD callers pass an uninitialized stack buffer, so the request contains arbitrary register field values and reserved bits. This makes the request depend on stack contents and compiler code generation.

Initialize both MFGD query buffers in the firmware fatal event configuration and reporter test paths. The initialization has been confirmed to eliminate the boot and shutdown hangs on both switches. The subsequent read-modify-write still preserves the other fields returned by the device.

Also initialize the FORE fan fault query and the MOGCR mirroring policer base query, which pass entirely uninitialized payloads. Restore the MTMP query initialization in the temperature history reset path, where only the slot and sensor index fields are set before querying the register.

This follows commit 5b0907407e7f ("mlxsw: core: Zero payload buffers for couple of registers"), which initialized other query payloads after a firmware field length bug caused firmware to consume uninitialized data.

Fixes 7d83ee111008 ("mlxsw: core: Introduce fw_fatal health reporter")
Fixes 2c6a33cd334b ("mlxsw: core: Extend hwmon interface with fan fault attribute")
Fixes 4039504e6a0c ("mlxsw: spectrum_span: Allow setting policer on a SPAN agent")
Fixes b890ad418e1f ("mlxsw: core_hwmon: Extend internal structures to support multi hwmon objects")
---
 drivers/net/ethernet/mellanox/mlxsw/core.c          | 4 ++--
 drivers/net/ethernet/mellanox/mlxsw/core_hwmon.c    | 4 ++--
 drivers/net/ethernet/mellanox/mlxsw/spectrum_span.c | 2 +-
 3 files changed, 5 insertions(+), 5 deletions(-)
drivers/net/ethernet/mellanox/mlxsw/core.c +2 -2
index 308d8a94865f..42e9689d7f35 100644
@@ -2001,7 +2001,7 @@ mlxsw_core_health_fw_fatal_test(struct devlink_health_reporter *reporter,
 				struct netlink_ext_ack *extack)
 {
 	struct mlxsw_core *mlxsw_core = devlink_health_reporter_priv(reporter);
-	char mfgd_pl[MLXSW_REG_MFGD_LEN];
+	char mfgd_pl[MLXSW_REG_MFGD_LEN] = {0};
 	int err;
 
 	/* Read the register first to make sure no other bits are changed. */
@@ -2022,7 +2022,7 @@ mlxsw_core_health_fw_fatal_ops = {
 static int mlxsw_core_health_fw_fatal_config(struct mlxsw_core *mlxsw_core,
 					     bool enable)
 {
-	char mfgd_pl[MLXSW_REG_MFGD_LEN];
+	char mfgd_pl[MLXSW_REG_MFGD_LEN] = {0};
 	int err;
 
 	/* Read the register first to make sure no other bits are changed. */
drivers/net/ethernet/mellanox/mlxsw/core_hwmon.c +2 -2
index 30f172ce822c..a5e7aee730d6 100644
@@ -120,7 +120,7 @@ static ssize_t mlxsw_hwmon_temp_rst_store(struct device *dev,
 			container_of(attr, struct mlxsw_hwmon_attr, dev_attr);
 	struct mlxsw_hwmon_dev *mlxsw_hwmon_dev = mlxsw_hwmon_attr->mlxsw_hwmon_dev;
 	struct mlxsw_hwmon *mlxsw_hwmon = mlxsw_hwmon_dev->hwmon;
-	char mtmp_pl[MLXSW_REG_MTMP_LEN];
+	char mtmp_pl[MLXSW_REG_MTMP_LEN] = {0};
 	unsigned long val;
 	int index;
 	int err;
@@ -177,7 +177,7 @@ static ssize_t mlxsw_hwmon_fan_fault_show(struct device *dev,
 			container_of(attr, struct mlxsw_hwmon_attr, dev_attr);
 	struct mlxsw_hwmon_dev *mlxsw_hwmon_dev = mlxsw_hwmon_attr->mlxsw_hwmon_dev;
 	struct mlxsw_hwmon *mlxsw_hwmon = mlxsw_hwmon_dev->hwmon;
-	char fore_pl[MLXSW_REG_FORE_LEN];
+	char fore_pl[MLXSW_REG_FORE_LEN] = {0};
 	bool fault;
 	int err;
 
drivers/net/ethernet/mellanox/mlxsw/spectrum_span.c +1 -1
index ae63d549b542..d54fe5e61296 100644
@@ -1720,7 +1720,7 @@ static int mlxsw_sp2_span_init(struct mlxsw_sp *mlxsw_sp)
 static int mlxsw_sp2_span_policer_id_base_set(struct mlxsw_sp *mlxsw_sp,
 					      u16 policer_id_base)
 {
-	char mogcr_pl[MLXSW_REG_MOGCR_LEN];
+	char mogcr_pl[MLXSW_REG_MOGCR_LEN] = {0};
 	int err;
 
 	err = mlxsw_reg_query(mlxsw_sp->core, MLXSW_REG(mogcr), mogcr_pl);

base-commit: 513444cc62e231a11c9b15ea88de0890da9b2b4f

Signature · 1 line
2.55.0
#1 lore raw
Quoted 25 lines · Til Kaiser

MFGD register queries can exhaust EMAD retries and leave Mellanox Spectrum switches stuck during boot or shutdown when running OpenWrt built with GCC 15.

An example failure is shown below with the second line wrapped:

EMAD retries (5/5) (tid=9b544fc100001f35)
EMAD reg access failed (tid=9b544fc100001f35,reg_id=90f0(mfgd),
type=query,status=0(operation performed))

On an SN2100, the failure occurs during driver teardown on reboot. On an SN3420, it occurs while probing mlxsw_spectrum during boot.

The register query API transmits the supplied payload to the device before replacing it with the response. Both MFGD callers pass an uninitialized stack buffer, so the request contains arbitrary register field values and reserved bits. This makes the request depend on stack contents and compiler code generation.

Initialize both MFGD query buffers in the firmware fatal event configuration and reporter test paths. The initialization has been confirmed to eliminate the boot and shutdown hangs on both switches. The subsequent read-modify-write still preserves the other fields returned by the device.

Quoted 8 lines

Also initialize the FORE fan fault query and the MOGCR mirroring policer base query, which pass entirely uninitialized payloads. Restore the MTMP query initialization in the temperature history reset path, where only the slot and sensor index fields are set before querying the register.

This follows commit 5b0907407e7f ("mlxsw: core: Zero payload buffers for couple of registers"), which initialized other query payloads after a firmware field length bug caused firmware to consume uninitialized data.

Hmm, isn't this pointing to that the query should be more robust, instead of relying on zeroing receive buffers pre-call? Maybe just fixup query instead?

Björn

maintainer
#2 lore raw
Quoted 3 lines · Björn Töpel

Hmm, isn't this pointing to that the query should be more robust, instead of relying on zeroing receive buffers pre-call? Maybe just fixup query instead?

Most registers require an index, so the common pattern is:

mlxsw_reg_*_pack(...);
mlxsw_reg_query(...);

Where the pack function zeroes the payload. We can't zero in the core because then we would overwrite the indexes.

#3 lore raw
Quoted 5 lines · Ido Schimmel
Quoted 3 lines · Björn Töpel

Hmm, isn't this pointing to that the query should be more robust, instead of relying on zeroing receive buffers pre-call? Maybe just fixup query instead?

Quoted 7 lines

Most registers require an index, so the common pattern is:

mlxsw_reg_*_pack(...);
mlxsw_reg_query(...);

Where the pack function zeroes the payload. We can't zero in the core because then we would overwrite the indexes.

Got it! Thanks for clearing that up!

Björn

maintainer Reviewed-by on v1 1/1
#4 lore raw
In reply to Til Kaiser
Quoted 3 lines · Til Kaiser

MFGD register queries can exhaust EMAD retries and leave Mellanox Spectrum switches stuck during boot or shutdown when running OpenWrt built with GCC 15.

I assume that you don't have CONFIG_INIT_STACK_ALL_ZERO enabled?

Quoted 9 lines

An example failure is shown below with the second line wrapped:

EMAD retries (5/5) (tid=9b544fc100001f35)
EMAD reg access failed (tid=9b544fc100001f35,reg_id=90f0(mfgd),
type=query,status=0(operation performed))

On an SN2100, the failure occurs during driver teardown on reboot. On an SN3420, it occurs while probing mlxsw_spectrum during boot.

Can you share the firmware versions?

Quoted 22 lines

The register query API transmits the supplied payload to the device before replacing it with the response. Both MFGD callers pass an uninitialized stack buffer, so the request contains arbitrary register field values and reserved bits. This makes the request depend on stack contents and compiler code generation.

Initialize both MFGD query buffers in the firmware fatal event configuration and reporter test paths. The initialization has been confirmed to eliminate the boot and shutdown hangs on both switches. The subsequent read-modify-write still preserves the other fields returned by the device.

Also initialize the FORE fan fault query and the MOGCR mirroring policer base query, which pass entirely uninitialized payloads. Restore the MTMP query initialization in the temperature history reset path, where only the slot and sensor index fields are set before querying the register.

This follows commit 5b0907407e7f ("mlxsw: core: Zero payload buffers for couple of registers"), which initialized other query payloads after a firmware field length bug caused firmware to consume uninitialized data.

Quoted 6 lines
Fixes 7d83ee111008 ("mlxsw: core: Introduce fw_fatal health reporter")
Fixes 2c6a33cd334b ("mlxsw: core: Extend hwmon interface with fan fault attribute")
Fixes 4039504e6a0c ("mlxsw: spectrum_span: Allow setting policer on a SPAN agent")
Fixes b890ad418e1f ("mlxsw: core_hwmon: Extend internal structures to support multi hwmon objects")
author
#5 lore raw
Quoted 3 lines · Ido Schimmel

I assume that you don't have CONFIG_INIT_STACK_ALL_ZERO enabled?

Correct, CONFIG_INIT_STACK_ALL_ZERO is not enabled by default in OpenWrt's x86 builds.

Quoted 3 lines

Can you share the firmware versions?

I tested with firmware 2020.1148 and with 2016.3900 from linux-firmware.

Til

#6 lore raw
In reply to Til Kaiser
This patch was applied to netdev/net.git (main) by Jakub Kicinski <[email protected]>:

Hello:

This patch was applied to netdev/net.git (main)
by Jakub Kicinski <[email protected]>:

Quoted 11 lines · On Wed, 14 Oct 2026 19:31:17 +0200 you wrote:

MFGD register queries can exhaust EMAD retries and leave Mellanox Spectrum switches stuck during boot or shutdown when running OpenWrt built with GCC 15.

An example failure is shown below with the second line wrapped:

EMAD retries (5/5) (tid=9b544fc100001f35)
EMAD reg access failed (tid=9b544fc100001f35,reg_id=90f0(mfgd),
type=query,status=0(operation performed))

[...]

Here is the summary with links:
- [net] mlxsw: Initialize register query payloads

    https://git.kernel.org/netdev/net/c/15b7a86a8888

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html