mlxsw: Initialize register query payloads
PATCH 1/1 mlxsw: Initialize register query payloads
MFGD register queries can exhaust EMAD retries and leave Mellanox Spectrum switches stuck during boot or shutdown when running OpenWrt built with GCC 15.
An example failure is shown below with the second line wrapped:
EMAD retries (5/5) (tid=9b544fc100001f35)
EMAD reg access failed (tid=9b544fc100001f35,reg_id=90f0(mfgd),
type=query,status=0(operation performed))
On an SN2100, the failure occurs during driver teardown on reboot. On an SN3420, it occurs while probing mlxsw_spectrum during boot.
The register query API transmits the supplied payload to the device before replacing it with the response. Both MFGD callers pass an uninitialized stack buffer, so the request contains arbitrary register field values and reserved bits. This makes the request depend on stack contents and compiler code generation.
Initialize both MFGD query buffers in the firmware fatal event configuration and reporter test paths. The initialization has been confirmed to eliminate the boot and shutdown hangs on both switches. The subsequent read-modify-write still preserves the other fields returned by the device.
Also initialize the FORE fan fault query and the MOGCR mirroring policer base query, which pass entirely uninitialized payloads. Restore the MTMP query initialization in the temperature history reset path, where only the slot and sensor index fields are set before querying the register.
This follows commit 5b0907407e7f ("mlxsw: core: Zero payload buffers for couple of registers"), which initialized other query payloads after a firmware field length bug caused firmware to consume uninitialized data.
--- drivers/net/ethernet/mellanox/mlxsw/core.c | 4 ++-- drivers/net/ethernet/mellanox/mlxsw/core_hwmon.c | 4 ++-- drivers/net/ethernet/mellanox/mlxsw/spectrum_span.c | 2 +- 3 files changed, 5 insertions(+), 5 deletions(-)
drivers/net/ethernet/mellanox/mlxsw/core.c +2 -2
@@ -2001,7 +2001,7 @@ mlxsw_core_health_fw_fatal_test(struct devlink_health_reporter *reporter, struct netlink_ext_ack *extack) {struct mlxsw_core *mlxsw_core =devlink_health_reporter_priv (reporter); -char mfgd_pl[MLXSW_REG_MFGD_LEN ]; +char mfgd_pl[MLXSW_REG_MFGD_LEN ] = {0 };int err;/* Read the register first to make sure no other bits are changed. */
@@ -2022,7 +2022,7 @@ mlxsw_core_health_fw_fatal_ops = {static int mlxsw_core_health_fw_fatal_config (struct mlxsw_core *mlxsw_core,bool enable) { -char mfgd_pl[MLXSW_REG_MFGD_LEN ]; +char mfgd_pl[MLXSW_REG_MFGD_LEN ] = {0 };int err;/* Read the register first to make sure no other bits are changed. */
drivers/net/ethernet/mellanox/mlxsw/core_hwmon.c +2 -2
@@ -120,7 +120,7 @@ static ssize_t mlxsw_hwmon_temp_rst_store(struct device *dev, container_of(attr, struct mlxsw_hwmon_attr, dev_attr);struct mlxsw_hwmon_dev *mlxsw_hwmon_dev = mlxsw_hwmon_attr->mlxsw_hwmon_dev;struct mlxsw_hwmon *mlxsw_hwmon = mlxsw_hwmon_dev->hwmon; -char mtmp_pl[MLXSW_REG_MTMP_LEN ]; +char mtmp_pl[MLXSW_REG_MTMP_LEN ] = {0 };unsigned long val;int index;int err;
@@ -177,7 +177,7 @@ static ssize_t mlxsw_hwmon_fan_fault_show(struct device *dev, container_of(attr, struct mlxsw_hwmon_attr, dev_attr);struct mlxsw_hwmon_dev *mlxsw_hwmon_dev = mlxsw_hwmon_attr->mlxsw_hwmon_dev;struct mlxsw_hwmon *mlxsw_hwmon = mlxsw_hwmon_dev->hwmon; -char fore_pl[MLXSW_REG_FORE_LEN ]; +char fore_pl[MLXSW_REG_FORE_LEN ] = {0 };bool fault;int err;
drivers/net/ethernet/mellanox/mlxsw/spectrum_span.c +1 -1
@@ -1720,7 +1720,7 @@ static int mlxsw_sp2_span_init(struct mlxsw_sp *mlxsw_sp)static int mlxsw_sp2_span_policer_id_base_set (struct mlxsw_sp *mlxsw_sp,u16 policer_id_base) { -char mogcr_pl[MLXSW_REG_MOGCR_LEN ]; +char mogcr_pl[MLXSW_REG_MOGCR_LEN ] = {0 };int err; err =mlxsw_reg_query (mlxsw_sp->core,MLXSW_REG (mogcr), mogcr_pl);
base-commit: 513444cc62e231a11c9b15ea88de0890da9b2b4f
Signature · 1 line
2.55.0
Quoted 25 lines · Til Kaiser
MFGD register queries can exhaust EMAD retries and leave Mellanox Spectrum switches stuck during boot or shutdown when running OpenWrt built with GCC 15.
An example failure is shown below with the second line wrapped:
EMAD retries (5/5) (tid=9b544fc100001f35)
EMAD reg access failed (tid=9b544fc100001f35,reg_id=90f0(mfgd),
type=query,status=0(operation performed))On an SN2100, the failure occurs during driver teardown on reboot. On an SN3420, it occurs while probing
mlxsw_spectrumduring boot.The register query API transmits the supplied payload to the device before replacing it with the response. Both MFGD callers pass an uninitialized stack buffer, so the request contains arbitrary register field values and reserved bits. This makes the request depend on stack contents and compiler code generation.
Initialize both MFGD query buffers in the firmware fatal event configuration and reporter test paths. The initialization has been confirmed to eliminate the boot and shutdown hangs on both switches. The subsequent read-modify-write still preserves the other fields returned by the device.
Quoted 8 lines
Also initialize the FORE fan fault query and the MOGCR mirroring policer base query, which pass entirely uninitialized payloads. Restore the MTMP query initialization in the temperature history reset path, where only the slot and sensor index fields are set before querying the register.
This follows commit
5b0907407e7f("mlxsw: core: Zero payload buffers for couple of registers"), which initialized other query payloads after a firmware field length bug caused firmware to consume uninitialized data.
Hmm, isn't this pointing to that the query should be more robust, instead of relying on zeroing receive buffers pre-call? Maybe just fixup query instead?
Björn
Ido Schimmel
maintainerQuoted 3 lines · Björn Töpel
Hmm, isn't this pointing to that the query should be more robust, instead of relying on zeroing receive buffers pre-call? Maybe just fixup query instead?
Most registers require an index, so the common pattern is:
mlxsw_reg_*_pack(...);mlxsw_reg_query(...);
Where the pack function zeroes the payload. We can't zero in the core because then we would overwrite the indexes.
Quoted 5 lines · Ido Schimmel
Quoted 3 lines · Björn Töpel
Hmm, isn't this pointing to that the query should be more robust, instead of relying on zeroing receive buffers pre-call? Maybe just fixup query instead?
Quoted 7 lines
Most registers require an index, so the common pattern is:
mlxsw_reg_*_pack(...);
mlxsw_reg_query(...);Where the pack function zeroes the payload. We can't zero in the core because then we would overwrite the indexes.
Got it! Thanks for clearing that up!
Björn
Quoted 3 lines · Til Kaiser
MFGD register queries can exhaust EMAD retries and leave Mellanox Spectrum switches stuck during boot or shutdown when running OpenWrt built with GCC 15.
I assume that you don't have CONFIG_INIT_STACK_ALL_ZERO enabled?
Quoted 9 lines
An example failure is shown below with the second line wrapped:
EMAD retries (5/5) (tid=9b544fc100001f35)
EMAD reg access failed (tid=9b544fc100001f35,reg_id=90f0(mfgd),
type=query,status=0(operation performed))On an SN2100, the failure occurs during driver teardown on reboot. On an SN3420, it occurs while probing
mlxsw_spectrumduring boot.
Can you share the firmware versions?
Quoted 22 lines
The register query API transmits the supplied payload to the device before replacing it with the response. Both MFGD callers pass an uninitialized stack buffer, so the request contains arbitrary register field values and reserved bits. This makes the request depend on stack contents and compiler code generation.
Initialize both MFGD query buffers in the firmware fatal event configuration and reporter test paths. The initialization has been confirmed to eliminate the boot and shutdown hangs on both switches. The subsequent read-modify-write still preserves the other fields returned by the device.
Also initialize the FORE fan fault query and the MOGCR mirroring policer base query, which pass entirely uninitialized payloads. Restore the MTMP query initialization in the temperature history reset path, where only the slot and sensor index fields are set before querying the register.
This follows commit
5b0907407e7f("mlxsw: core: Zero payload buffers for couple of registers"), which initialized other query payloads after a firmware field length bug caused firmware to consume uninitialized data.
Quoted 6 lines
Fixes 7d83ee111008 ("mlxsw: core: Introduce fw_fatal health reporter")Fixes 2c6a33cd334b ("mlxsw: core: Extend hwmon interface with fan fault attribute")Fixes 4039504e6a0c ("mlxsw: spectrum_span: Allow setting policer on a SPAN agent")Fixes b890ad418e1f ("mlxsw: core_hwmon: Extend internal structures to support multi hwmon objects")Signed-off-by Til Kaiser <[email protected]>
Quoted 3 lines · Ido Schimmel
I assume that you don't have
CONFIG_INIT_STACK_ALL_ZEROenabled?
Correct, CONFIG_INIT_STACK_ALL_ZERO is not enabled by default in OpenWrt's x86 builds.
Quoted 3 lines
Can you share the firmware versions?
I tested with firmware 2020.1148 and with 2016.3900 from linux-firmware.
Til
This patch was applied to netdev/net.git (main) by Jakub Kicinski <[email protected]>:
Hello:
This patch was applied to netdev/net.git (main)
by Jakub Kicinski <[email protected]>:
Quoted 11 lines · On Wed, 14 Oct 2026 19:31:17 +0200 you wrote:
MFGD register queries can exhaust EMAD retries and leave Mellanox Spectrum switches stuck during boot or shutdown when running OpenWrt built with GCC 15.
An example failure is shown below with the second line wrapped:
EMAD retries (5/5) (tid=9b544fc100001f35)
EMAD reg access failed (tid=9b544fc100001f35,reg_id=90f0(mfgd),
type=query,status=0(operation performed))[...]
Here is the summary with links:
- [net] mlxsw: Initialize register query payloads
https://git.kernel.org/netdev/net/c/15b7a86a8888
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html