devlink: add generic max_sfs parameter and mlx5 support
Subsystems (6)
Tariq Toukan
maintainerv2 cover devlink: add generic max_sfs parameter and mlx5 support
Hi,
This series by Nikolay introduces a new generic devlink device parameter, max_sfs, to control the number of light-weight NIC subfunctions (SFs) that can be created on a device.
The first patch adds the generic devlink parameter and infrastructure support.
The second patch implements support for the parameter in the mlx5 driver.
With this addition, users can enable or disable SF creation directly via devlink, without relying on external vendor-specific tools.
Regards,
Tariq
V2:
- Add missing ` (Aleksandr Loktionov).
- Add review tag to patch 1.
V1:
https://lore.kernel.org/all/[email protected]/
Nikolay Aleksandrov (2):
devlink: add generic device max_sfs parameter
net/mlx5: implement max_sfs parameter
.../networking/devlink/devlink-params.rst | 6 ++ Documentation/networking/devlink/mlx5.rst | 7 +- .../mellanox/mlx5/core/lib/nv_param.c | 83 ++++++++++++++++++- include/net/devlink.h | 4 + net/devlink/param.c | 5 ++ 5 files changed, 101 insertions(+), 4 deletions(-)
base-commit: 9bf93cb2e180a58d5984ba13daee95903ff4fc14
Signature · 1 line
2.44.0
Tariq Toukan
maintainerPATCH v2 1/2 devlink: add generic device max_sfs parameter
From: Nikolay Aleksandrov <[email protected]>
Add a new generic devlink device parameter (max_sfs) to control if and how many light-weight NIC subfunctions can be created. Subfunctions are a light-weight network functions backed by an underlying PCI function. Their lifecycle can already be managed by devlink, but currently users cannot enable them in the device. They can be enabled/disabled only via external vendor tools. This parameter allows subfunctions to be enabled (>0) or disabled (0) via devlink. A subsequent patch will add support for max_sfs to the mlx5 driver.
--- Documentation/networking/devlink/devlink-params.rst | 6 ++++++ include/net/devlink.h | 4 ++++ net/devlink/param.c | 5 +++++ 3 files changed, 15 insertions(+)
Documentation/networking/devlink/devlink-params.rst +6 -0
@@ -165,3 +165,9 @@ own name. - u32 - Controls the maximum number of MAC address filters that can be assigned to a Virtual Function (VF). + * - ``max_sfs`` + - u32 + - The maximum number of subfunctions which can be created on the device. + Modifying this parameter may require a device restart and PCI bus + rescanning because the BAR layout may change. A value of 0 disables + subfunction creation.
include/net/devlink.h +4 -0
@@ -546,6 +546,7 @@ enum devlink_param_generic_id {DEVLINK_PARAM_GENERIC_ID_TOTAL_VFS ,DEVLINK_PARAM_GENERIC_ID_NUM_DOORBELLS ,DEVLINK_PARAM_GENERIC_ID_MAX_MAC_PER_VF , +DEVLINK_PARAM_GENERIC_ID_MAX_SFS ,/* add new param generic ids above here*/ __DEVLINK_PARAM_GENERIC_ID_MAX,
@@ -619,6 +620,9 @@ enum devlink_param_generic_id {#define DEVLINK_PARAM_GENERIC_MAX_MAC_PER_VF_NAME "max_mac_per_vf"#define DEVLINK_PARAM_GENERIC_MAX_MAC_PER_VF_TYPE DEVLINK_PARAM_TYPE_U32 +#define DEVLINK_PARAM_GENERIC_MAX_SFS_NAME "max_sfs" +#define DEVLINK_PARAM_GENERIC_MAX_SFS_TYPE DEVLINK_PARAM_TYPE_U32 +#define DEVLINK_PARAM_GENERIC (_id, _cmodes, _get, _set, _validate) \ { \ .id = DEVLINK_PARAM_GENERIC_ID_##_id, \
net/devlink/param.c +5 -0
@@ -117,6 +117,11 @@ static const struct devlink_param devlink_param_generic[] = { .name = DEVLINK_PARAM_GENERIC_MAX_MAC_PER_VF_NAME, .type = DEVLINK_PARAM_GENERIC_MAX_MAC_PER_VF_TYPE, }, + { + .id = DEVLINK_PARAM_GENERIC_ID_MAX_SFS, + .name = DEVLINK_PARAM_GENERIC_MAX_SFS_NAME, + .type = DEVLINK_PARAM_GENERIC_MAX_SFS_TYPE, + }, };static int devlink_param_generic_verify (const struct devlink_param *param)
Signature · 1 line
2.44.0
Tariq Toukan
maintainerPATCH v2 2/2 net/mlx5: implement max_sfs parameter
In reply to Tariq ToukanFrom: Nikolay Aleksandrov <[email protected]>
Implement max_sfs generic parameter to allow users to control the total light-weight NIC subfunctions that can be created using devlink instead of external vendor tools. A value of 0 will effectively disable creation of new subfunction devices. A warning is sent to user-space via extack (returning extack without error code is interpreted as a warning by user-space tools).
--- Documentation/networking/devlink/mlx5.rst | 7 +- .../mellanox/mlx5/core/lib/nv_param.c | 83 ++++++++++++++++++- 2 files changed, 86 insertions(+), 4 deletions(-)
Documentation/networking/devlink/mlx5.rst +6 -1
@@ -45,8 +45,13 @@ Parameters - The range is between 1 and a device-specific max. - Applies to each physical function (PF) independently, if the device supports it. Otherwise, it applies symmetrically to all PFs. + * - ``max_sfs`` + - permanent + - The range is between 0 and a device-specific max. + - Applies to each physical function (PF) independently. -Note: permanent parameters such as ``enable_sriov`` and ``total_vfs`` require FW reset to take effect +Note: permanent parameters such as ``enable_sriov``, ``total_vfs`` and ``max_sfs`` + require FW reset to take effect .. code-block:: bash
drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c +80 -3
108 lines not on this page · raw message
Signature · 1 line
2.44.0
Jakub Kicinski
maintainerRe: [PATCH net-next V2 2/2] net/mlx5: implement max_sfs parameter
This is an AI-generated review of your patch. The human sending this email has considered the AI review valid, or at least plausible.
---
net/mlx5: implement max_sfs parameter
This patch adds a generic devlink permanent parameter named max_sfs to the mlx5 driver so users can cap the number of light-weight NIC subfunctions via devlink instead of vendor tooling. The setter rewrites non-volatile firmware fields in nv_global_pci_conf and nv_pf_pci_conf, and a reboot is required for the change to take effect.
A few questions on the implementation below.
Documentation/networking/devlink/mlx5.rst Parameters · line 51
Quoted 12 lines
Documentation/networking/devlink/mlx5.rst +4
-0@@ -45,8 +45,13 @@ Parameters - The range is between 1 and a device-specific max. - Applies to each physical function (PF) independently, if the device supports it. Otherwise, it applies symmetrically to all PFs. + * - ``max_sfs`` + - permanent + - The range is between 0 and a device-specific max. + - Applies to each physical function (PF) independently.
[Medium, Medium]
The documentation states the range is "between 0 and a device-specific max", and the entry says max_sfs "Applies to each physical function (PF) independently" with no "if the device supports it" qualifier like total_vfs has.
Looking at the implementation in mlx5_devlink_max_sfs_set(), there is no read of any nv_global_pci_cap field to obtain a device-reported maximum, and the registration uses NULL for the validate callback:
DEVLINK_PARAM_GENERIC(MAX_SFS, BIT(DEVLINK_PARAM_CMODE_PERMANENT),
mlx5_devlink_max_sfs_get, mlx5_devlink_max_sfs_set, NULL),
Should there be a validate callback analogous to mlx5_devlink_total_vfs_validate() that reads the cap and rejects values above the device-reported maximum, so the documented "device-specific max" is actually enforced?
Should the doc also include the "if the device supports it" qualifier to match how the setter behaves on devices without the relevant capability?
Quoted 14 lines
drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c +3
-1@@ -68,7 +68,9 @@ struct mlx5_ifc_mnvda_reg_bits {struct mlx5_ifc_nv_global_pci_conf_bits {u8 sriov_valid[0x1 ]; -u8 reserved_at_1[0x10 ]; +u8 reserved_at_1[0xa ]; +u8 per_pf_num_sf[0x1 ]; +u8 reserved_at_c[0x5 ];u8 per_pf_total_vf[0x1 ];u8 reserved_at_12[0xe ];
[ ... ]
drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c struct mlx5_ifc_nv_global_pci_cap_bits { · line 104
Quoted 13 lines
drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c +4
-2@@ -93,9 +95,11 @@ struct mlx5_ifc_nv_global_pci_cap_bits { };struct mlx5_ifc_nv_pf_pci_conf_bits { -u8 reserved_at_0[0x9 ]; +u8 log_sf_bar_size[0x8 ]; +u8 pf_total_sf_en[0x1 ];u8 pf_total_vf_en[0x1 ]; -u8 reserved_at_a[0x16 ]; +u8 reserved_at_a[0x6 ]; +u8 total_sf[0x10 ];u8 reserved_at_20[0x20 ];
[Medium]
The on-wire field total_sf is 16 bits wide, but the devlink parameter is registered as DEVLINK_PARAM_TYPE_U32 (max_sfs uses DEVLINK_PARAM_GENERIC_MAX_SFS_TYPE, which is u32) and the setter writes ctx->val.vu32 directly via MLX5_SET(nv_pf_pci_conf, data, total_sf, ctx->val.vu32).
MLX5_SET masks the value with __mlx5_mask(typ, fld) before storing, so any user-supplied value above 0xffff is silently truncated. For example, "devlink dev param set ... name max_sfs value 70000" would persistently store 4464.
Without a validate callback, is there anything that prevents this silent truncation, and should the setter or a validate hook reject values that do not fit in 16 bits?
Quoted 14 lines
drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c +10
-0@@ -755,6 +759,76 @@ static int mlx5_devlink_total_vfs_validate(struct devlink *devlink, u32 id,return 0 ; } +static int mlx5_devlink_max_sfs_get (struct devlink *devlink,u32 id, +struct devlink_param_gset_ctx *ctx, +struct netlink_ext_ack *extack) +{ +struct mlx5_core_dev *dev =devlink_priv (devlink); +u32 mnvda[MLX5_ST_SZ_DW (mnvda_reg)] = {}; +void *data; +int err; + + err =mlx5_nv_param_read_per_host_pf_conf (dev, mnvda,sizeof (mnvda));
drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c static int mlx5_devlink_total_vfs_validate(struct devlink *devlink, u32 id, · line 781
Quoted 10 lines
drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c +10
-0+if (err) { +NL_SET_ERR_MSG_MOD (extack,"Failed to read PF configuration" ); +return err; + } + + data =MLX5_ADDR_OF (mnvda_reg, mnvda, configuration_item_data); + ctx->val.vu32 =MLX5_GET (nv_pf_pci_conf, data, total_sf); + +return 0 ; +}
[Medium]
Should max_sfs_get() mirror mlx5_devlink_total_vfs_get() and consult nv_global_pci_conf.per_pf_num_sf before reporting nv_pf_pci_conf.total_sf?
The setter explicitly conditions the meaning of per-PF total_sf on per_pf_num_sf:
MLX5_SET(nv_global_pci_conf, data, per_pf_num_sf, !!ctx->val.vu32); ... MLX5_SET(nv_pf_pci_conf, data, total_sf, ctx->val.vu32);
But the getter reads total_sf unconditionally, so on a device whose firmware was previously configured by another tool with per_pf_num_sf=0 and a stale per-host total_sf, "devlink dev param show" would return a non-zero value that does not represent the active SF policy. Is that intentional?
Quoted 15 lines
drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c +15
-0+static int mlx5_devlink_max_sfs_set (struct devlink *devlink,u32 id, +struct devlink_param_gset_ctx *ctx, +struct netlink_ext_ack *extack) +{ +struct mlx5_core_dev *dev =devlink_priv (devlink); +u32 mnvda[MLX5_ST_SZ_DW (mnvda_reg)] = {}; +void *data; +int err; + + err =mlx5_nv_param_read_global_pci_conf (dev, mnvda,sizeof (mnvda)); +if (err) { +NL_SET_ERR_MSG_MOD (extack, +"Failed to read global PCI configuration" ); +return err; + }
drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c static int mlx5_devlink_total_vfs_validate(struct devlink *devlink, u32 id, · line 807
Quoted 10 lines
drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c +10
-0+ + data =MLX5_ADDR_OF (mnvda_reg, mnvda, configuration_item_data); +MLX5_SET (nv_global_pci_conf, data, per_pf_num_sf, !!ctx->val.vu32); + + err =mlx5_nv_param_write (dev, mnvda,sizeof (mnvda)); +if (err) { +NL_SET_ERR_MSG_MOD (extack, +"Failed to change per_pf_num_sf global PCI configuration" ); +return err; + }
[Medium]
Should this path first read nv_global_pci_cap and reject the operation when the device does not advertise SF NV configuration, the way mlx5_devlink_total_vfs_set() does for SR-IOV?
mlx5_devlink_total_vfs_set() rejects with -EOPNOTSUPP and an extack "SRIOV is not per PF on this device" when sriov_support or per_pf_total_vf_supported is clear, but no equivalent per_pf_num_sf_supported (or any SF-related) capability bit is added to nv_global_pci_cap_bits or queried here. On hardware that lacks the feature, the user only sees the generic firmware error "Failed to change ... global PCI configuration".
[Medium]
At this point, a successful write of per_pf_num_sf has already been committed to non-volatile firmware storage. If the subsequent mlx5_nv_param_read_per_host_pf_conf() or the second mlx5_nv_param_write() fails (for example, transient firmware/PCIe issue), is there a path that rolls back per_pf_num_sf?
Because these are permanent parameters that "require a reboot to take effect", a half-applied state (e.g., per_pf_num_sf=1 but pf_total_sf_en=0/total_sf=0) appears to persist across reboots until the user issues another successful set.
drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c static int mlx5_devlink_total_vfs_validate(struct devlink *devlink, u32 id, · line 817
Quoted 10 lines
drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c +10
-0+ +memset (mnvda,0 ,sizeof (mnvda)); + err =mlx5_nv_param_read_per_host_pf_conf (dev, mnvda,sizeof (mnvda)); +if (err) { +NL_SET_ERR_MSG_MOD (extack,"Failed to read PF configuration" ); +return err; + } + + data =MLX5_ADDR_OF (mnvda_reg, mnvda, configuration_item_data); +MLX5_SET (nv_pf_pci_conf, data, log_sf_bar_size, ctx->val.vu32 ?12 :0 );
[Low]
What is the meaning of the literal 12 here, and why is it correct on all supported devices?
Elsewhere in the driver, the SF BAR length is computed from a hardware-reported capability rather than a fixed value:
drivers/net/ethernet/mellanox/mlx5/core/sf/dev/dev.c:
sf_bar_length = 1 << (MLX5_CAP_GEN(dev, log_min_sf_size) + 12);Should log_sf_bar_size be derived from MLX5_CAP_GEN(dev, log_min_sf_size) (or another cap) rather than hardcoded?
The commit message says max_sfs is to "control the total light-weight NIC subfunctions"; the BAR-size side-effect is not mentioned, and any previously configured log_sf_bar_size is overwritten on every max_sfs set. Should that behavior be documented or split out from the count knob?
Quoted 13 lines
drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c +13
-0+MLX5_SET (nv_pf_pci_conf, data, pf_total_sf_en, !!ctx->val.vu32); +MLX5_SET (nv_pf_pci_conf, data, total_sf, ctx->val.vu32); + + err =mlx5_nv_param_write (dev, mnvda,sizeof (mnvda)); +if (err) { +NL_SET_ERR_MSG_MOD (extack, +"Failed to change PF PCI configuration" ); +return err; + } +NL_SET_ERR_MSG (extack,"Modifying max_sfs requires a reboot" ); + +return 0 ; +}
Signature · 1 line
pw-bot: cr
Re: [PATCH net-next V2 2/2] net/mlx5: implement max_sfs parameter
Quoted 36 lines · Jakub Kicinski
This is an AI-generated review of your patch. The human sending this email has considered the AI review valid, or at least plausible.
---
net/mlx5: implementmax_sfsparameterThis patch adds a generic devlink permanent parameter named
max_sfsto the mlx5 driver so users can cap the number of light-weight NIC subfunctions via devlink instead of vendor tooling. The setter rewrites non-volatile firmware fields innv_global_pci_confandnv_pf_pci_conf, and a reboot is required for the change to take effect.A few questions on the implementation below.
Documentation/networking/devlink/mlx5.rst Parameters · line 51
Quoted 12 lines
Documentation/networking/devlink/mlx5.rst +4
-0@@ -45,8 +45,13 @@ Parameters - The range is between 1 and a device-specific max. - Applies to each physical function (PF) independently, if the device supports it. Otherwise, it applies symmetrically to all PFs. + * - ``max_sfs`` + - permanent + - The range is between 0 and a device-specific max. + - Applies to each physical function (PF) independently.[Medium, Medium]
The documentation states the range is "between 0 and a device-specific max", and the entry saysmax_sfs"Applies to each physical function (PF) independently" with no "if the device supports it" qualifier liketotal_vfshas.Looking at the implementation in
mlx5_devlink_max_sfs_set(), there is no read of anynv_global_pci_capfield to obtain a device-reported maximum, and the registration uses NULL for the validate callback:
Quoted 9 lines
DEVLINK_PARAM_GENERIC(MAX_SFS,
BIT(DEVLINK_PARAM_CMODE_PERMANENT),
mlx5_devlink_max_sfs_get,mlx5_devlink_max_sfs_set, NULL),Should there be a validate callback analogous to
mlx5_devlink_total_vfs_validate()that reads the cap and rejects values above the device-reported maximum, so the documented "device-specific max" is actually enforced?
I don't know of such cap, if there was I'd have added it.
Quoted 42 lines
Should the doc also include the "if the device supports it" qualifier to match how the setter behaves on devices without the relevant capability?
Quoted 14 lines
drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c +3
-1@@ -68,7 +68,9 @@ struct mlx5_ifc_mnvda_reg_bits {struct mlx5_ifc_nv_global_pci_conf_bits {u8 sriov_valid[0x1 ]; -u8 reserved_at_1[0x10 ]; +u8 reserved_at_1[0xa ]; +u8 per_pf_num_sf[0x1 ]; +u8 reserved_at_c[0x5 ];u8 per_pf_total_vf[0x1 ];u8 reserved_at_12[0xe ];[ ... ]
drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c struct mlx5_ifc_nv_global_pci_cap_bits { · line 104
Quoted 13 lines
drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c +4
-2@@ -93,9 +95,11 @@ struct mlx5_ifc_nv_global_pci_cap_bits { };struct mlx5_ifc_nv_pf_pci_conf_bits { -u8 reserved_at_0[0x9 ]; +u8 log_sf_bar_size[0x8 ]; +u8 pf_total_sf_en[0x1 ];u8 pf_total_vf_en[0x1 ]; -u8 reserved_at_a[0x16 ]; +u8 reserved_at_a[0x6 ]; +u8 total_sf[0x10 ];u8 reserved_at_20[0x20 ];[Medium]
The on-wire fieldtotal_sfis 16 bits wide, but the devlink parameter is registered asDEVLINK_PARAM_TYPE_U32(max_sfsusesDEVLINK_PARAM_GENERIC_MAX_SFS_TYPE, which is u32) and the setter writesctx->val.vu32directly viaMLX5_SET(nv_pf_pci_conf, data, total_sf, ctx->val.vu32).
Quoted 9 lines
MLX5_SETmasks the value with__mlx5_mask(typ, fld)before storing, so any user-supplied value above0xffffis silently truncated. For example, "devlink dev param set ... namemax_sfsvalue 70000" would persistently store 4464.Without a validate callback, is there anything that prevents this silent truncation, and should the setter or a validate hook reject values that do not fit in 16 bits?
This is benign but I'll cap it at 16 bits for the next version.
Quoted 36 lines
Quoted 14 lines
drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c +10
-0@@ -755,6 +759,76 @@ static int mlx5_devlink_total_vfs_validate(struct devlink *devlink, u32 id,return 0 ; } +static int mlx5_devlink_max_sfs_get (struct devlink *devlink,u32 id, +struct devlink_param_gset_ctx *ctx, +struct netlink_ext_ack *extack) +{ +struct mlx5_core_dev *dev =devlink_priv (devlink); +u32 mnvda[MLX5_ST_SZ_DW (mnvda_reg)] = {}; +void *data; +int err; + + err =mlx5_nv_param_read_per_host_pf_conf (dev, mnvda,sizeof (mnvda));drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c static int mlx5_devlink_total_vfs_validate(struct devlink *devlink, u32 id, · line 781
Quoted 10 lines
drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c +10
-0+if (err) { +NL_SET_ERR_MSG_MOD (extack,"Failed to read PF configuration" ); +return err; + } + + data =MLX5_ADDR_OF (mnvda_reg, mnvda, configuration_item_data); + ctx->val.vu32 =MLX5_GET (nv_pf_pci_conf, data, total_sf); + +return 0 ; +}[Medium]
Shouldmax_sfs_get()mirrormlx5_devlink_total_vfs_get()and consult nv_global_pci_conf.per_pf_num_sf before reporting nv_pf_pci_conf.total_sf?The setter explicitly conditions the meaning of per-PF
total_sfonper_pf_num_sf:MLX5_SET(nv_global_pci_conf, data, per_pf_num_sf, !!ctx->val.vu32); ... MLX5_SET(nv_pf_pci_conf, data, total_sf, ctx->val.vu32);
Quoted 6 lines
But the getter reads
total_sfunconditionally, so on a device whose firmware was previously configured by another tool with per_pf_num_sf=0 and a stale per-hosttotal_sf, "devlink dev param show" would return a non-zero value that does not represent the active SF policy. Is that intentional?
No.
Quoted 31 lines
Quoted 15 lines
drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c +15
-0+static int mlx5_devlink_max_sfs_set (struct devlink *devlink,u32 id, +struct devlink_param_gset_ctx *ctx, +struct netlink_ext_ack *extack) +{ +struct mlx5_core_dev *dev =devlink_priv (devlink); +u32 mnvda[MLX5_ST_SZ_DW (mnvda_reg)] = {}; +void *data; +int err; + + err =mlx5_nv_param_read_global_pci_conf (dev, mnvda,sizeof (mnvda)); +if (err) { +NL_SET_ERR_MSG_MOD (extack, +"Failed to read global PCI configuration" ); +return err; + }drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c static int mlx5_devlink_total_vfs_validate(struct devlink *devlink, u32 id, · line 807
Quoted 10 lines
drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c +10
-0+ + data =MLX5_ADDR_OF (mnvda_reg, mnvda, configuration_item_data); +MLX5_SET (nv_global_pci_conf, data, per_pf_num_sf, !!ctx->val.vu32); + + err =mlx5_nv_param_write (dev, mnvda,sizeof (mnvda)); +if (err) { +NL_SET_ERR_MSG_MOD (extack, +"Failed to change per_pf_num_sf global PCI configuration" ); +return err; + }[Medium]
Should this path first readnv_global_pci_capand reject the operation when the device does not advertise SF NV configuration, the waymlx5_devlink_total_vfs_set()does for SR-IOV?
Quoted 8 lines
mlx5_devlink_total_vfs_set()rejects with-EOPNOTSUPPand an extack "SRIOV is not per PF on this device" whensriov_supportorper_pf_total_vf_supportedis clear, but no equivalentper_pf_num_sf_supported(or any SF-related) capability bit is added tonv_global_pci_cap_bitsor queried here. On hardware that lacks the feature, the user only sees the generic firmware error "Failed to change ... global PCI configuration".
I don't know of such bit, if there was such bit I'd have added it.
Quoted 7 lines
[Medium]
At this point, a successful write ofper_pf_num_sfhas already been committed to non-volatile firmware storage. If the subsequentmlx5_nv_param_read_per_host_pf_conf()or the secondmlx5_nv_param_write()fails (for example, transientfirmware/PCIeissue), is there a path that rolls backper_pf_num_sf?
Quoted 5 lines
Because these are permanent parameters that "require a reboot to take effect", a half-applied state (e.g., per_pf_num_sf=1 but pf_total_sf_en=0/total_sf=0) appears to persist across reboots until the user issues another successful set.
sure, that is expected
Quoted 6 lines
Quoted 6 lines
drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c +6
-0+ +memset (mnvda,0 ,sizeof (mnvda)); + err =mlx5_nv_param_read_per_host_pf_conf (dev, mnvda,sizeof (mnvda)); +if (err) { +NL_SET_ERR_MSG_MOD (extack,"Failed to read PF configuration" ); +return err;
Quoted 9 lines
drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c static int mlx5_devlink_total_vfs_validate(struct devlink *devlink, u32 id, · line 817
Quoted 4 lines
drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c +4
-0+ } + + data =MLX5_ADDR_OF (mnvda_reg, mnvda, configuration_item_data); +MLX5_SET (nv_pf_pci_conf, data, log_sf_bar_size, ctx->val.vu32 ?12 :0 );[Low]
What is the meaning of the literal 12 here, and why is it correct on all supported devices?
It is a sane default, some day we can make it configurable as well.
Quoted 9 lines
Elsewhere in the driver, the SF BAR length is computed from a hardware-reported capability rather than a fixed value:
drivers/net/ethernet/mellanox/mlx5/core/sf/dev/dev.c: sf_bar_length = 1 << (MLX5_CAP_GEN(dev, log_min_sf_size) + 12);Should
log_sf_bar_sizebe derived fromMLX5_CAP_GEN(dev, log_min_sf_size)(or another cap) rather than hardcoded?
Interesting, I'll check this out.
Quoted 6 lines
The commit message says
max_sfsis to "control the total light-weight NIC subfunctions"; the BAR-size side-effect is not mentioned, and any previously configuredlog_sf_bar_sizeis overwritten on everymax_sfsset. Should that behavior be documented or split out from the count knob?
It should be documented when it is split out, at present time - no.
Quoted 13 lines
Quoted 13 lines
drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c +13
-0+MLX5_SET (nv_pf_pci_conf, data, pf_total_sf_en, !!ctx->val.vu32); +MLX5_SET (nv_pf_pci_conf, data, total_sf, ctx->val.vu32); + + err =mlx5_nv_param_write (dev, mnvda,sizeof (mnvda)); +if (err) { +NL_SET_ERR_MSG_MOD (extack, +"Failed to change PF PCI configuration" ); +return err; + } +NL_SET_ERR_MSG (extack,"Modifying max_sfs requires a reboot" ); + +return 0 ; +}