devlink: Add port function attribute for IO EQs
Subsystems (6)
v2 cover devlink: Add port function attribute for IO EQs
Currently, PCI SFs and VFs use IO event queues to deliver netdev per channel events. The number of netdev channels is a function of IO event queues. In the second scenario of an RDMA device, the completion vectors are also a function of IO event queues. Currently, an administrator on the hypervisor has no means to provision the number of IO event queues for the SF device or the VF device. Device/firmware determines some arbitrary value for these IO event queues. Due to this, the SF netdev channels are unpredictable, and consequently, the performance is too.
This short series introduces a new port function attribute: max_io_eqs. The goal is to provide administrators at the hypervisor level with the ability to provision the maximum number of IO event queues for a function. This gives the control to the administrator to provision right number of IO event queues and have predictable performance.
Examples of when an administrator provisions (set) maximum number of IO event queues when using switchdev mode:
$ devlink port show pci/0000:06:00.0/1
pci/0000:06:00.0/1: type eth netdev enp6s0pf0vf0 flavour pcivf pfnum 0 vfnum 0
function:
hw_addr 00:00:00:00:00:00 roce enable max_io_eqs 10$ devlink port function set pci/0000:06:00.0/1 max_io_eqs 20
$ devlink port show pci/0000:06:00.0/1
pci/0000:06:00.0/1: type eth netdev enp6s0pf0vf0 flavour pcivf pfnum 0 vfnum 0
function:
hw_addr 00:00:00:00:00:00 roce enable max_io_eqs 20This sets the corresponding maximum IO event queues of the function before it is enumerated. Thus, when the VF/SF driver reads the capability from the device, it sees the value provisioned by the hypervisor. The driver is then able to configure the number of channels for the net device, as well as the number of completion vectors for the RDMA device. The device/firmware also honors the provisioned value, hence any VF/SF driver attempting to create IO EQs beyond provisioned value results in an error.
With above setting now, the administrator is able to achieve the 2x performance on SFs with 20 channels. In second example when SF was provisioned for a container with 2 cpus, the administrator provisioned only 2 IO event queues, thereby saving device resources.
With the above settings now in place, the administrator achieved 2x performance with the SF device with 20 channels. In the second example, when the SF was provisioned for a container with 2 CPUs, the administrator provisioned only 2 IO event queues, thereby saving device resources.
changelog:v1->v2:
- limited comment to 80 chars per line in header file
- fixed set function variables for reverse christmas tree
- fixed comments from Kalesh
- fixed missing kfree in get call
- returning error code for get cmd failure
- fixed error msg copy paste error in set on cmd failure
Parav Pandit (2):
devlink: Support setting max_io_eqs
mlx5/core: Support max_io_eqs for a function
.../networking/devlink/devlink-port.rst | 25 +++++ .../mellanox/mlx5/core/esw/devlink_port.c | 2 + .../net/ethernet/mellanox/mlx5/core/eswitch.h | 7 ++ .../mellanox/mlx5/core/eswitch_offloads.c | 97 +++++++++++++++++++ include/net/devlink.h | 14 +++ include/uapi/linux/devlink.h | 1 + net/devlink/port.c | 52 ++++++++++ 7 files changed, 198 insertions(+)
Signature · 1 line
2.26.2
PATCH v2 1/2 devlink: Support setting max_io_eqs
Many devices send event notifications for the IO queues, such as tx and rx queues, through event queues.
Enable a privileged owner, such as a hypervisor PF, to set the number of IO event queues for the VF and SF during the provisioning stage.
example:
Get maximum IO event queues of the VF device::
$ devlink port show pci/0000:06:00.0/2
pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1
function:
hw_addr 00:00:00:00:00:00 ipsec_packet disabled max_io_eqs 10Set maximum IO event queues of the VF device::
$ devlink port function set pci/0000:06:00.0/2 max_io_eqs 32
$ devlink port show pci/0000:06:00.0/2
pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1
function:
hw_addr 00:00:00:00:00:00 ipsec_packet disabled max_io_eqs 32---
changelog:v1->v2:
- limited comment to 80 chars per line in header file
--- .../networking/devlink/devlink-port.rst | 25 +++++++++ include/net/devlink.h | 14 +++++ include/uapi/linux/devlink.h | 1 + net/devlink/port.c | 52 +++++++++++++++++++ 4 files changed, 92 insertions(+)
Documentation/networking/devlink/devlink-port.rst +25 -0
@@ -134,6 +134,9 @@ Users may also set the IPsec crypto capability of the function using Users may also set the IPsec packet capability of the function using `devlink port function set ipsec_packet` command. +Users may also set the maximum IO event queues of the function +using `devlink port function set max_io_eqs` command. + Function attributes ===================
@@ -295,6 +298,28 @@ policy is processed in software by the kernel. function: hw_addr 00:00:00:00:00:00 ipsec_packet enabled +Maximum IO events queues setup +------------------------------ +When user sets maximum number of IO event queues for a SF or +a VF, such function driver is limited to consume only enforced +number of IO event queues. + +- Get maximum IO event queues of the VF device:: + + $ devlink port show pci/0000:06:00.0/2 + pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1 + function: + hw_addr 00:00:00:00:00:00 ipsec_packet disabled max_io_eqs 10 + +- Set maximum IO event queues of the VF device:: + + $ devlink port function set pci/0000:06:00.0/2 max_io_eqs 32 + + $ devlink port show pci/0000:06:00.0/2 + pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1 + function: + hw_addr 00:00:00:00:00:00 ipsec_packet disabled max_io_eqs 32 + Subfunction ============
include/net/devlink.h +14 -0
@@ -1602,6 +1602,14 @@ void devlink_free(struct devlink *devlink);* capability. Should be used by device drivers to * enable/disable ipsec_packet capability of a * function managed by the devlink port. +* @port_fn_max_io_eqs_get: Callback used to get port function's maximum number +* of event queues. Should be used by device drivers to +* report the maximum event queues of a function +* managed by the devlink port. +* @port_fn_max_io_eqs_set: Callback used to set port function's maximum number +* of event queues. Should be used by device drivers to +* configure maximum number of event queues +* of a function managed by the devlink port. * * Note: Driver should return -EOPNOTSUPP if it doesn't support * port function (@port_fn_*) handling for a particular port.
@@ -1651,6 +1659,12 @@ struct devlink_port_ops {int (*port_fn_ipsec_packet_set)(struct devlink_port *devlink_port,bool enable,struct netlink_ext_ack *extack); +int (*port_fn_max_io_eqs_get)(struct devlink_port *devlink_port, +u32 *max_eqs, +struct netlink_ext_ack *extack); +int (*port_fn_max_io_eqs_set)(struct devlink_port *devlink_port, +u32 max_eqs, +struct netlink_ext_ack *extack); }; void devlink_port_init(struct devlink *devlink,
include/uapi/linux/devlink.h +1 -0
@@ -686,6 +686,7 @@ enum devlink_port_function_attr {DEVLINK_PORT_FN_ATTR_OPSTATE ,/* u8 */ DEVLINK_PORT_FN_ATTR_CAPS ,/* bitfield32 */ DEVLINK_PORT_FN_ATTR_DEVLINK ,/* nested */ +DEVLINK_PORT_FN_ATTR_MAX_IO_EQS ,/* u32 */ __DEVLINK_PORT_FUNCTION_ATTR_MAX,DEVLINK_PORT_FUNCTION_ATTR_MAX = __DEVLINK_PORT_FUNCTION_ATTR_MAX -1
net/devlink/port.c +52 -0
88 lines not on this page · raw message
Signature · 1 line
2.26.2
PATCH v2 2/2 mlx5/core: Support max_io_eqs for a function
In reply to Parav PanditImplement get and set for the maximum IO event queues for SF and VF. This enables administrator on the hypervisor to control the maximum IO event queues which are typically used to derive the maximum and default number of net device channels or rdma device completion vectors.
---
changelog:v1->v2:
- fixed comments from Kalesh
- fixed missing kfree in get call
- returning error code for get cmd failure
- fixed error msg copy paste error in set on cmd failure
- limited code to 80 chars limit
- fixed set function variables for reverse christmas tree
--- .../mellanox/mlx5/core/esw/devlink_port.c | 2 + .../net/ethernet/mellanox/mlx5/core/eswitch.h | 7 ++ .../mellanox/mlx5/core/eswitch_offloads.c | 97 +++++++++++++++++++ 3 files changed, 106 insertions(+)
drivers/net/ethernet/mellanox/mlx5/core/esw/devlink_port.c +2 -0
@@ -98,6 +98,8 @@ static const struct devlink_port_ops mlx5_esw_pf_vf_dl_port_ops = { .port_fn_ipsec_packet_get = mlx5_devlink_port_fn_ipsec_packet_get, .port_fn_ipsec_packet_set = mlx5_devlink_port_fn_ipsec_packet_set,#endif /* CONFIG_XFRM_OFFLOAD */ + .port_fn_max_io_eqs_get = mlx5_devlink_port_fn_max_io_eqs_get, + .port_fn_max_io_eqs_set = mlx5_devlink_port_fn_max_io_eqs_set, }; static void mlx5_esw_offloads_sf_devlink_port_attrs_set(struct mlx5_eswitch *esw,
drivers/net/ethernet/mellanox/mlx5/core/eswitch.h +7 -0
@@ -573,6 +573,13 @@ int mlx5_devlink_port_fn_ipsec_packet_get(struct devlink_port *port, bool *is_enint mlx5_devlink_port_fn_ipsec_packet_set (struct devlink_port *port,bool enable,struct netlink_ext_ack *extack);#endif /* CONFIG_XFRM_OFFLOAD */ +int mlx5_devlink_port_fn_max_io_eqs_get (struct devlink_port *port, +u32 *max_io_eqs, +struct netlink_ext_ack *extack); +int mlx5_devlink_port_fn_max_io_eqs_set (struct devlink_port *port, +u32 max_io_eqs, +struct netlink_ext_ack *extack); +void *mlx5_eswitch_get_uplink_priv (struct mlx5_eswitch *esw,u8 rep_type); int __mlx5_eswitch_set_vport_vlan(struct mlx5_eswitch *esw,
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +97 -0
106 lines not on this page · raw message
Signature · 1 line
2.26.2
Jakub Kicinski
maintainerRe: [net-next v2 2/2] mlx5/core: Support max_io_eqs for a function
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c int mlx5_devlink_port_fn_ipsec_packet_set(struct devlink_port *port, · line 4592
Quoted 10 lines · On Tue, 2 Apr 2024 14:25:49 +0300 Parav Pandit wrote:
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +10
-0+ query_ctx =kzalloc (query_out_sz,GFP_KERNEL ); +if (!query_ctx) +return -ENOMEM ; + +mutex_lock (&esw->state_lock); + err =mlx5_vport_get_other_func_cap (esw->dev, vport_num, query_ctx, +MLX5_CAP_GENERAL ); +if (err) { +NL_SET_ERR_MSG_MOD (extack,"Failed getting HCA caps" ); + goto out;
missing unlock
And before someone suggests we need guards, even make coccicheck catches this...
Quoted 14 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +14
-0+ } + + hca_caps =MLX5_ADDR_OF (query_hca_cap_out, query_ctx, capability); +MLX5_SET (cmd_hca_cap, hca_caps, max_num_eqs, max_eqs); + + err =mlx5_vport_set_other_func_cap (esw->dev, hca_caps, vport_num, +MLX5_SET_HCA_CAP_OP_MOD_GENERAL_DEVICE ); +mutex_unlock (&esw->state_lock); +if (err) +NL_SET_ERR_MSG_MOD (extack,"Failed setting HCA caps" ); + +out: +kfree (query_ctx); +return err;
Signature · 1 line
pw-bot: cr
RE: [net-next v2 2/2] mlx5/core: Support max_io_eqs for a function
Quoted 19 lines
From: Jakub Kicinski <[email protected]>
Sent: Tuesday, April 2, 2024 8:49 PMdrivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c int mlx5_devlink_port_fn_ipsec_packet_set(struct devlink_port *port,
Quoted 6 lines · On Tue, 2 Apr 2024 14:25:49 +0300 Parav Pandit wrote:
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +6
-0+ query_ctx =kzalloc (query_out_sz,GFP_KERNEL ); +if (!query_ctx) +return -ENOMEM ; + +mutex_lock (&esw->state_lock); + err = mlx5_vport_get_other_func_cap(esw->dev, vport_num,
query_ctx,drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c int mlx5_devlink_port_fn_ipsec_packet_set(struct devlink_port *port, · line 4592
Quoted 4 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +4
-0+ MLX5_CAP_GENERAL); +if (err) { +NL_SET_ERR_MSG_MOD (extack,"Failed getting HCA caps" ); + goto out;missing unlock
Quoted 3 lines
And before someone suggests we need guards, even make coccicheck catches this...
I am sorry about it.
I didn't wait for the coccicheck to finish in internal build. Fixing it.
Quoted 19 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c int mlx5_devlink_port_fn_ipsec_packet_set(struct devlink_port *port,
Quoted 3 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +3
-0+ } + + hca_caps = MLX5_ADDR_OF(query_hca_cap_out, query_ctx,capability);
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c int mlx5_devlink_port_fn_ipsec_packet_set(struct devlink_port *port,
Quoted 3 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +3
-0+MLX5_SET (cmd_hca_cap, hca_caps, max_num_eqs, max_eqs); + + err = mlx5_vport_set_other_func_cap(esw->dev, hca_caps,
vport_num,Quoted 1 line
+
MLX5_SET_HCA_CAP_OP_MOD_GENERAL_DEVICE);drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c int mlx5_devlink_port_fn_ipsec_packet_set(struct devlink_port *port, · line 4655
Quoted 7 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +7
-0+mutex_unlock (&esw->state_lock); +if (err) +NL_SET_ERR_MSG_MOD (extack,"Failed setting HCA caps" ); + +out: +kfree (query_ctx); +return err;--
pw-bot: cr