devlink: add generic device max_sfs parameter
Subsystems (6)
Tariq Toukan
maintainerv4 cover devlink: add generic device max_sfs parameter
Hi,
This series by Nikolay introduces a new generic devlink device parameter, max_sfs, to control the number of light-weight NIC subfunctions (SFs) that can be created on a device.
The first patch adds the generic devlink parameter and infrastructure support.
The second patch implements support for the parameter in the mlx5 driver.
With this addition, users can enable or disable SF creation directly via devlink, without relying on external vendor-specific tools.
Regards,
Tariq
Notes about a few items from a previous Sashiko review:
Quoted 4 lines
Should there be a validate callback analogous to
mlx5_devlink_total_vfs_validate()that reads the cap and rejects values above the device-reported maximum, so the documented "device-specific max" is actually enforced?
N: I don't know of such cap, if there was I'd have added it.
Quoted 7 lines
mlx5_devlink_total_vfs_set()rejects with-EOPNOTSUPPand an extack "SRIOV is not per PF on this device" whensriov_supportorper_pf_total_vf_supportedis clear, but no equivalentper_pf_num_sf_supported(or any SF-related) capability bit is added tonv_global_pci_cap_bitsor queried here. On hardware that lacks the feature, the user only sees the generic firmware error "Failed to change ... global PCI configuration".
N: I don't know of such bit, if there was such bit I'd have added it.
Quoted 10 lines
At this point, a successful write of
per_pf_num_sfhas already been committed to non-volatile firmware storage. If the subsequentmlx5_nv_param_read_per_host_pf_conf()or the secondmlx5_nv_param_write()fails (for example, transientfirmware/PCIeissue), is there a path that rolls backper_pf_num_sf?Because these are permanent parameters that "require a reboot to take effect", a half-applied state (e.g., per_pf_num_sf=1 but pf_total_sf_en=0/total_sf=0) appears to persist across reboots until the user issues another successful set.
N: That is expected and in line with the rest of the code.
Quoted 5 lines
The commit message says
max_sfsis to "control the total light-weight NIC subfunctions"; the BAR-size side-effect is not mentioned, and any previously configuredlog_sf_bar_sizeis overwritten on everymax_sfsset. Should that behavior be documented or split out from the count knob?
N: It should be documented when it is split out, at present time - no.
V4:
- rebase due to build failure because of commit:
d603517771d8 ("devlink: pass param values by pointer") fixed in patch 02
- added Alexander Lobakin's reviewed-by tags (please let me know if that is not ok, the change in patch 02 is minor and mechanical due to the commit above)
V3, all changes are in patch 02 (mlx5 implementation):
https://lore.kernel.org/netdev/[email protected]
- Cap max_sfs at U16_MAX using a validate callback (sashiko)
- Change the warning message to match the docs (sashiko)
- On get verify that per_pf_num_sf & pf_total_sf_en are set otherwise return 0 since it means SFs are not properly enabled (sashiko)
- Define the default log bar size and use it instead of raw value (sashiko)
V2:
https://lore.kernel.org/netdev/[email protected]/
- Add missing ` (Aleksandr Loktionov).
- Add review tag to patch 1.
V1:
https: //lore.kernel.org/all/[email protected]/
Nikolay Aleksandrov (2):
devlink: add generic device max_sfs parameter
net/mlx5: implement max_sfs parameter
.../networking/devlink/devlink-params.rst | 6 + Documentation/networking/devlink/mlx5.rst | 7 +- .../mellanox/mlx5/core/lib/nv_param.c | 118 +++++++++++++++++- include/net/devlink.h | 4 + net/devlink/param.c | 5 + 5 files changed, 136 insertions(+), 4 deletions(-)
base-commit: 1df10cef2d1e7f9f2fb7eddb67fc70d3abf101f9
Signature · 1 line
2.44.0
Tariq Toukan
maintainerPATCH v4 1/2 devlink: add generic device max_sfs parameter
From: Nikolay Aleksandrov <[email protected]>
Add a new generic devlink device parameter (max_sfs) to control if and how many light-weight NIC subfunctions can be created. Subfunctions are a light-weight network functions backed by an underlying PCI function. Their lifecycle can already be managed by devlink, but currently users cannot enable them in the device. They can be enabled/disabled only via external vendor tools. This parameter allows subfunctions to be enabled (>0) or disabled (0) via devlink. A subsequent patch will add support for max_sfs to the mlx5 driver.
--- Documentation/networking/devlink/devlink-params.rst | 6 ++++++ include/net/devlink.h | 4 ++++ net/devlink/param.c | 5 +++++ 3 files changed, 15 insertions(+)
Documentation/networking/devlink/devlink-params.rst +6 -0
@@ -165,3 +165,9 @@ own name. - u32 - Controls the maximum number of MAC address filters that can be assigned to a Virtual Function (VF). + * - ``max_sfs`` + - u32 + - The maximum number of subfunctions which can be created on the device. + Modifying this parameter may require a device restart and PCI bus + rescanning because the BAR layout may change. A value of 0 disables + subfunction creation.
include/net/devlink.h +4 -0
@@ -554,6 +554,7 @@ enum devlink_param_generic_id {DEVLINK_PARAM_GENERIC_ID_TOTAL_VFS ,DEVLINK_PARAM_GENERIC_ID_NUM_DOORBELLS ,DEVLINK_PARAM_GENERIC_ID_MAX_MAC_PER_VF , +DEVLINK_PARAM_GENERIC_ID_MAX_SFS ,/* add new param generic ids above here*/ __DEVLINK_PARAM_GENERIC_ID_MAX,
@@ -627,6 +628,9 @@ enum devlink_param_generic_id {#define DEVLINK_PARAM_GENERIC_MAX_MAC_PER_VF_NAME "max_mac_per_vf"#define DEVLINK_PARAM_GENERIC_MAX_MAC_PER_VF_TYPE DEVLINK_PARAM_TYPE_U32 +#define DEVLINK_PARAM_GENERIC_MAX_SFS_NAME "max_sfs" +#define DEVLINK_PARAM_GENERIC_MAX_SFS_TYPE DEVLINK_PARAM_TYPE_U32 +#define DEVLINK_PARAM_GENERIC (_id, _cmodes, _get, _set, _validate) \ { \ .id = DEVLINK_PARAM_GENERIC_ID_##_id, \
net/devlink/param.c +5 -0
@@ -117,6 +117,11 @@ static const struct devlink_param devlink_param_generic[] = { .name = DEVLINK_PARAM_GENERIC_MAX_MAC_PER_VF_NAME, .type = DEVLINK_PARAM_GENERIC_MAX_MAC_PER_VF_TYPE, }, + { + .id = DEVLINK_PARAM_GENERIC_ID_MAX_SFS, + .name = DEVLINK_PARAM_GENERIC_MAX_SFS_NAME, + .type = DEVLINK_PARAM_GENERIC_MAX_SFS_TYPE, + }, };static int devlink_param_generic_verify (const struct devlink_param *param)
Signature · 1 line
2.44.0
Tariq Toukan
maintainerPATCH v4 2/2 net/mlx5: implement max_sfs parameter
In reply to Tariq ToukanFrom: Nikolay Aleksandrov <[email protected]>
Implement max_sfs generic parameter to allow users to control the total light-weight NIC subfunctions that can be created using devlink instead of external vendor tools. A value of 0 will effectively disable creation of new subfunction devices. A warning is sent to user-space via extack (returning extack without error code is interpreted as a warning by user-space tools). The maximum value is capped at U16_MAX.
--- Documentation/networking/devlink/mlx5.rst | 7 +- .../mellanox/mlx5/core/lib/nv_param.c | 118 +++++++++++++++++- 2 files changed, 121 insertions(+), 4 deletions(-)
Documentation/networking/devlink/mlx5.rst +6 -1
@@ -45,8 +45,13 @@ Parameters - The range is between 1 and a device-specific max. - Applies to each physical function (PF) independently, if the device supports it. Otherwise, it applies symmetrically to all PFs. + * - ``max_sfs`` + - permanent + - The range is between 0 and a device-specific max. + - Applies to each physical function (PF) independently. -Note: permanent parameters such as ``enable_sriov`` and ``total_vfs`` require FW reset to take effect +Note: permanent parameters such as ``enable_sriov``, ``total_vfs`` and ``max_sfs`` + require FW reset to take effect .. code-block:: bash
drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c +115 -3
149 lines not on this page · raw message
Signature · 1 line
2.44.0
Re: [PATCH net-next V4 2/2] net/mlx5: implement max_sfs parameter
Quoted 9 lines · Tariq Toukan
From: Nikolay Aleksandrov <[email protected]>
Implement
max_sfsgeneric parameter to allow users to control the total light-weight NIC subfunctions that can be created using devlink instead of external vendor tools. A value of 0 will effectively disable creation of new subfunction devices. A warning is sent to user-space via extack (returning extack without error code is interpreted as a warning by user-space tools). The maximum value is capped atU16_MAX.
Quoted 9 lines
Signed-off-by Nikolay Aleksandrov <[email protected]>Reviewed-by David Ahern <[email protected]>Reviewed-by Alexander Lobakin <[email protected]>Signed-off-by Tariq Toukan <[email protected]>--- Documentation/networking/devlink/mlx5.rst | 7 +- .../mellanox/mlx5/core/lib/nv_param.c | 118 +++++++++++++++++- 2 files changed, 121 insertions(+), 4 deletions(-)
Sashiko has 2 comments:
Review
Quoted 2 lines
(quoted diff) +2
-0+ data = MLX5_ADDR_OF(mnvda_reg, mnvda, configuration_item_data); + MLX5_SET(nv_global_pci_conf, data, per_pf_num_sf, !!ctx->val.vu32);
If max_sfs is set to 0 for a single Physical Function (PF), does this unintentionally clear the global per_pf_num_sf bit?
The commit message and documentation indicate max_sfs applies to each PF independently. However, clearing this global device configuration bit would silently disable Subfunction capabilities for all other PFs on the entire device.
Could this break Subfunction configurations on independent ports?
Answer: Yes, it does. I just tested it and it switches the config for both
PFs on a device. Somehow I missed this, and it was in earlier reviews as well. The !!ctx->val.vu32 should just be 1. I'm sorry for the noise, I'll fix it and queue v5.
and
Does this unconditionally overwrite any existing configuration for the SubFunction BAR size?
When a user configures a custom log_sf_bar_size via firmware tools like mlxconfig, and then later uses devlink to change max_sfs, it appears this line discards any non-zero value already present in the configuration data and forcibly replaces it with MLX5_DEFAULT_LOG_SF_BAR_SIZE. Should this code preserve the existing log_sf_bar_size if it was already configured?
Answer: Yes, it does and it is okay. That was intentional.