devlink: Add port function attribute for IO EQs
Subsystems (6)
v3 cover devlink: Add port function attribute for IO EQs
Currently, PCI SFs and VFs use IO event queues to deliver netdev per channel events. The number of netdev channels is a function of IO event queues. In the second scenario of an RDMA device, the completion vectors are also a function of IO event queues. Currently, an administrator on the hypervisor has no means to provision the number of IO event queues for the SF device or the VF device. Device/firmware determines some arbitrary value for these IO event queues. Due to this, the SF netdev channels are unpredictable, and consequently, the performance is too.
This short series introduces a new port function attribute: max_io_eqs. The goal is to provide administrators at the hypervisor level with the ability to provision the maximum number of IO event queues for a function. This gives the control to the administrator to provision right number of IO event queues and have predictable performance.
Examples of when an administrator provisions (set) maximum number of IO event queues when using switchdev mode:
$ devlink port show pci/0000:06:00.0/1
pci/0000:06:00.0/1: type eth netdev enp6s0pf0vf0 flavour pcivf pfnum 0 vfnum 0
function:
hw_addr 00:00:00:00:00:00 roce enable max_io_eqs 10$ devlink port function set pci/0000:06:00.0/1 max_io_eqs 20
$ devlink port show pci/0000:06:00.0/1
pci/0000:06:00.0/1: type eth netdev enp6s0pf0vf0 flavour pcivf pfnum 0 vfnum 0
function:
hw_addr 00:00:00:00:00:00 roce enable max_io_eqs 20This sets the corresponding maximum IO event queues of the function before it is enumerated. Thus, when the VF/SF driver reads the capability from the device, it sees the value provisioned by the hypervisor. The driver is then able to configure the number of channels for the net device, as well as the number of completion vectors for the RDMA device. The device/firmware also honors the provisioned value, hence any VF/SF driver attempting to create IO EQs beyond provisioned value results in an error.
With above setting now, the administrator is able to achieve the 2x performance on SFs with 20 channels. In second example when SF was provisioned for a container with 2 cpus, the administrator provisioned only 2 IO event queues, thereby saving device resources.
With the above settings now in place, the administrator achieved 2x performance with the SF device with 20 channels. In the second example, when the SF was provisioned for a container with 2 CPUs, the administrator provisioned only 2 IO event queues, thereby saving device resources.
changelog:v2->v3:
- limited to 80 chars per line in devlink
- fixed comments from Jakub in mlx5 driver to fix missing mutex unlock on error pathv1->v2:
- limited comment to 80 chars per line in header file
- fixed set function variables for reverse christmas tree
- fixed comments from Kalesh
- fixed missing kfree in get call
- returning error code for get cmd failure
- fixed error msg copy paste error in set on cmd failure
Parav Pandit (2):
devlink: Support setting max_io_eqs
mlx5/core: Support max_io_eqs for a function
.../networking/devlink/devlink-port.rst | 25 +++++ .../mellanox/mlx5/core/esw/devlink_port.c | 4 + .../net/ethernet/mellanox/mlx5/core/eswitch.h | 7 ++ .../mellanox/mlx5/core/eswitch_offloads.c | 97 +++++++++++++++++++ include/net/devlink.h | 14 +++ include/uapi/linux/devlink.h | 1 + net/devlink/port.c | 53 ++++++++++ 7 files changed, 201 insertions(+)
Signature · 1 line
2.26.2
PATCH v3 1/2 devlink: Support setting max_io_eqs
Many devices send event notifications for the IO queues, such as tx and rx queues, through event queues.
Enable a privileged owner, such as a hypervisor PF, to set the number of IO event queues for the VF and SF during the provisioning stage.
example:
Get maximum IO event queues of the VF device::
$ devlink port show pci/0000:06:00.0/2
pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1
function:
hw_addr 00:00:00:00:00:00 ipsec_packet disabled max_io_eqs 10Set maximum IO event queues of the VF device::
$ devlink port function set pci/0000:06:00.0/2 max_io_eqs 32
$ devlink port show pci/0000:06:00.0/2
pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1
function:
hw_addr 00:00:00:00:00:00 ipsec_packet disabled max_io_eqs 32---
changelog:v2->v3:
- limited 80 chars per linev1->v2:
- limited comment to 80 chars per line in header file
--- .../networking/devlink/devlink-port.rst | 25 +++++++++ include/net/devlink.h | 14 +++++ include/uapi/linux/devlink.h | 1 + net/devlink/port.c | 53 +++++++++++++++++++ 4 files changed, 93 insertions(+)
Documentation/networking/devlink/devlink-port.rst +25 -0
@@ -134,6 +134,9 @@ Users may also set the IPsec crypto capability of the function using Users may also set the IPsec packet capability of the function using `devlink port function set ipsec_packet` command. +Users may also set the maximum IO event queues of the function +using `devlink port function set max_io_eqs` command. + Function attributes ===================
@@ -295,6 +298,28 @@ policy is processed in software by the kernel. function: hw_addr 00:00:00:00:00:00 ipsec_packet enabled +Maximum IO events queues setup +------------------------------ +When user sets maximum number of IO event queues for a SF or +a VF, such function driver is limited to consume only enforced +number of IO event queues. + +- Get maximum IO event queues of the VF device:: + + $ devlink port show pci/0000:06:00.0/2 + pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1 + function: + hw_addr 00:00:00:00:00:00 ipsec_packet disabled max_io_eqs 10 + +- Set maximum IO event queues of the VF device:: + + $ devlink port function set pci/0000:06:00.0/2 max_io_eqs 32 + + $ devlink port show pci/0000:06:00.0/2 + pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1 + function: + hw_addr 00:00:00:00:00:00 ipsec_packet disabled max_io_eqs 32 + Subfunction ============
include/net/devlink.h +14 -0
@@ -1602,6 +1602,14 @@ void devlink_free(struct devlink *devlink);* capability. Should be used by device drivers to * enable/disable ipsec_packet capability of a * function managed by the devlink port. +* @port_fn_max_io_eqs_get: Callback used to get port function's maximum number +* of event queues. Should be used by device drivers to +* report the maximum event queues of a function +* managed by the devlink port. +* @port_fn_max_io_eqs_set: Callback used to set port function's maximum number +* of event queues. Should be used by device drivers to +* configure maximum number of event queues +* of a function managed by the devlink port. * * Note: Driver should return -EOPNOTSUPP if it doesn't support * port function (@port_fn_*) handling for a particular port.
@@ -1651,6 +1659,12 @@ struct devlink_port_ops {int (*port_fn_ipsec_packet_set)(struct devlink_port *devlink_port,bool enable,struct netlink_ext_ack *extack); +int (*port_fn_max_io_eqs_get)(struct devlink_port *devlink_port, +u32 *max_eqs, +struct netlink_ext_ack *extack); +int (*port_fn_max_io_eqs_set)(struct devlink_port *devlink_port, +u32 max_eqs, +struct netlink_ext_ack *extack); }; void devlink_port_init(struct devlink *devlink,
include/uapi/linux/devlink.h +1 -0
@@ -686,6 +686,7 @@ enum devlink_port_function_attr {DEVLINK_PORT_FN_ATTR_OPSTATE ,/* u8 */ DEVLINK_PORT_FN_ATTR_CAPS ,/* bitfield32 */ DEVLINK_PORT_FN_ATTR_DEVLINK ,/* nested */ +DEVLINK_PORT_FN_ATTR_MAX_IO_EQS ,/* u32 */ __DEVLINK_PORT_FUNCTION_ATTR_MAX,DEVLINK_PORT_FUNCTION_ATTR_MAX = __DEVLINK_PORT_FUNCTION_ATTR_MAX -1
net/devlink/port.c +53 -0
89 lines not on this page · raw message
Signature · 1 line
2.26.2
Re: [net-next v3 1/2] devlink: Support setting max_io_eqs
Quoted 147 lines · Parav Pandit
Many devices send event notifications for the IO queues, such as tx and rx queues, through event queues.
Enable a privileged owner, such as a hypervisor PF, to set the number of IO event queues for the VF and SF during the provisioning stage.
example:
Get maximum IO event queues of the VF device::$ devlink port show pci/0000:06:00.0/2 pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1 function: hw_addr 00:00:00:00:00:00 ipsec_packet disabled max_io_eqs 10Set maximum IO event queues of the VF device::
$ devlink port function set pci/0000:06:00.0/2 max_io_eqs 32$ devlink port show pci/0000:06:00.0/2 pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1 function: hw_addr 00:00:00:00:00:00 ipsec_packet disabled max_io_eqs 32Reviewed-by Jiri Pirko <[email protected]>Reviewed-by Shay Drory <[email protected]>Signed-off-by Parav Pandit <[email protected]>---
changelog:v2->v3:
- limited 80 chars per linev1->v2:
- limited comment to 80 chars per line in header file--- .../networking/devlink/devlink-port.rst | 25 +++++++++ include/net/devlink.h | 14 +++++ include/uapi/linux/devlink.h | 1 + net/devlink/port.c | 53 +++++++++++++++++++ 4 files changed, 93 insertions(+)Documentation/networking/devlink/devlink-port.rst +25
-0@@ -134,6 +134,9 @@ Users may also set the IPsec crypto capability of the function using Users may also set the IPsec packet capability of the function using `devlink port function set ipsec_packet` command. +Users may also set the maximum IO event queues of the function +using `devlink port function set max_io_eqs` command. + Function attributes ===================@@ -295,6 +298,28 @@ policy is processed in software by the kernel. function: hw_addr 00:00:00:00:00:00 ipsec_packet enabled +Maximum IO events queues setup +------------------------------ +When user sets maximum number of IO event queues for a SF or +a VF, such function driver is limited to consume only enforced +number of IO event queues. + +- Get maximum IO event queues of the VF device:: + + $ devlink port show pci/0000:06:00.0/2 + pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1 + function: + hw_addr 00:00:00:00:00:00 ipsec_packet disabled max_io_eqs 10 + +- Set maximum IO event queues of the VF device:: + + $ devlink port function set pci/0000:06:00.0/2 max_io_eqs 32 + + $ devlink port show pci/0000:06:00.0/2 + pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1 + function: + hw_addr 00:00:00:00:00:00 ipsec_packet disabled max_io_eqs 32 + Subfunction ============include/net/devlink.h +14
-0@@ -1602,6 +1602,14 @@ void devlink_free(struct devlink *devlink);* capability. Should be used by device drivers to * enable/disable ipsec_packet capability of a * function managed by the devlink port. +* @port_fn_max_io_eqs_get: Callback used to get port function's maximum number +* of event queues. Should be used by device drivers to +* report the maximum event queues of a function +* managed by the devlink port. +* @port_fn_max_io_eqs_set: Callback used to set port function's maximum number +* of event queues. Should be used by device drivers to +* configure maximum number of event queues +* of a function managed by the devlink port. * * Note: Driver should return -EOPNOTSUPP if it doesn't support * port function (@port_fn_*) handling for a particular port. @@ -1651,6 +1659,12 @@ struct devlink_port_ops {int (*port_fn_ipsec_packet_set)(struct devlink_port *devlink_port,bool enable,struct netlink_ext_ack *extack); +int (*port_fn_max_io_eqs_get)(struct devlink_port *devlink_port, +u32 *max_eqs, +struct netlink_ext_ack *extack); +int (*port_fn_max_io_eqs_set)(struct devlink_port *devlink_port, +u32 max_eqs, +struct netlink_ext_ack *extack); }; void devlink_port_init(struct devlink *devlink,include/uapi/linux/devlink.h +1
-0@@ -686,6 +686,7 @@ enum devlink_port_function_attr {DEVLINK_PORT_FN_ATTR_OPSTATE ,/* u8 */ DEVLINK_PORT_FN_ATTR_CAPS ,/* bitfield32 */ DEVLINK_PORT_FN_ATTR_DEVLINK ,/* nested */ +DEVLINK_PORT_FN_ATTR_MAX_IO_EQS ,/* u32 */ __DEVLINK_PORT_FUNCTION_ATTR_MAX,DEVLINK_PORT_FUNCTION_ATTR_MAX = __DEVLINK_PORT_FUNCTION_ATTR_MAX -1 net/devlink/port.c +6
-0@@ -16,6 +16,7 @@ static const struct nla_policy devlink_function_nl_policy[DEVLINK_PORT_FUNCTION_ DEVLINK_PORT_FN_STATE_ACTIVE), [DEVLINK_PORT_FN_ATTR_CAPS ] = NLA_POLICY_BITFIELD32(DEVLINK_PORT_FN_CAPS_VALID_MASK), + [DEVLINK_PORT_FN_ATTR_MAX_IO_EQS] = { .type = NLA_U32 }, };#define ASSERT_DEVLINK_PORT_REGISTERED (devlink_port) \@@ -182,6 +183,30 @@ static int devlink_port_fn_caps_fill(struct devlink_port *devlink_port,return 0 ; } +static int devlink_port_fn_max_io_eqs_fill (struct devlink_port *port, +struct sk_buff *msg, +struct netlink_ext_ack *extack, +bool *msg_updated) +{
net/devlink/port.c static int devlink_port_fn_caps_fill(struct devlink_port *devlink_port, · line 200
Quoted 10 lines
net/devlink/port.c +10
-0+u32 max_io_eqs; +int err; + +if (!port->ops->port_fn_max_io_eqs_get) +return 0 ; + + err = port->ops->port_fn_max_io_eqs_get (port, &max_io_eqs, extack); +if (err) { +if (err == -EOPNOTSUPP ) +return 0 ;
Docs above says:
* Note: Driver should return -EOPNOTSUPP if it doesn't support
* port function (@port_fn_*) handling for a particular port.
But here you're returning 0 in both cases of no port_fn_max_io_eqs_get or port_fn_max_io_eqs_get() returns EOPNOTSUPP.
Quoted 68 lines
net/devlink/port.c +37
-0+return err; + } + err =nla_put_u32 (msg,DEVLINK_PORT_FN_ATTR_MAX_IO_EQS , max_io_eqs); +if (err) +return err; + *msg_updated = true; +return 0 ; +} +int devlink_nl_port_handle_fill (struct sk_buff *msg,struct devlink_port *devlink_port) {if (devlink_nl_put_handle (msg, devlink_port->devlink))@@ -409,6 +434,18 @@ static int devlink_port_fn_caps_set(struct devlink_port *devlink_port,return 0 ; } +static int +devlink_port_fn_max_io_eqs_set (struct devlink_port *devlink_port, +const struct nlattr *attr, +struct netlink_ext_ack *extack) +{ +u32 max_io_eqs; + + max_io_eqs =nla_get_u32 (attr); +return devlink_port->ops->port_fn_max_io_eqs_set (devlink_port, + max_io_eqs, extack); +} +static int devlink_nl_port_function_attrs_put (struct sk_buff *msg,struct devlink_port *port,struct netlink_ext_ack *extack)@@ -428,6 +465,9 @@ devlink_nl_port_function_attrs_put(struct sk_buff *msg, struct devlink_port *porif (err) goto out; err =devlink_port_fn_state_fill (port, msg, extack, &msg_updated); +if (err) + goto out; + err =devlink_port_fn_max_io_eqs_fill (port, msg, extack, &msg_updated);if (err) goto out; err = devlink_rel_devlink_handle_put(msg, port->devlink,@@ -726,6 +766,12 @@ static int devlink_port_function_validate(struct devlink_port *devlink_port, } } } +if (tb[DEVLINK_PORT_FN_ATTR_MAX_IO_EQS ] && + !ops->port_fn_max_io_eqs_set) { +NL_SET_ERR_MSG_ATTR (extack, tb[DEVLINK_PORT_FN_ATTR_MAX_IO_EQS ], +"Function does not support max_io_eqs setting" ); +return -EOPNOTSUPP ; + }return 0 ; }@@ -761,6 +807,13 @@ static int devlink_port_function_set(struct devlink_port *port,return err; } + attr = tb[DEVLINK_PORT_FN_ATTR_MAX_IO_EQS ]; +if (attr) { + err =devlink_port_fn_max_io_eqs_set (port, attr, extack); +if (err) +return err; + } +/* Keep this as the last function attribute set, so that when * multiple port function attributes are set along with state, * Those can be applied first before activating the state.
RE: [net-next v3 1/2] devlink: Support setting max_io_eqs
Hi David,
Quoted 185 lines
From: David Wei <[email protected]>
Sent: Thursday, April 4, 2024 10:29 PMQuoted 11 lines · Parav Pandit
Many devices send event notifications for the IO queues, such as tx and rx queues, through event queues.
Enable a privileged owner, such as a hypervisor PF, to set the number of IO event queues for the VF and SF during the provisioning stage.
example:
Get maximum IO event queues of the VF device::$ devlink port show pci/0000:06:00.0/2 pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0vfnum 1
Quoted 10 lines
function: hw_addr 00:00:00:00:00:00 ipsec_packet disabled max_io_eqs10
Set maximum IO event queues of the VF device::
$ devlink port function set pci/0000:06:00.0/2 max_io_eqs 32$ devlink port show pci/0000:06:00.0/2 pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0vfnum 1
Quoted 39 lines
function: hw_addr 00:00:00:00:00:00 ipsec_packet disabled max_io_eqs32
Reviewed-by Jiri Pirko <[email protected]>Reviewed-by Shay Drory <[email protected]>Signed-off-by Parav Pandit <[email protected]>---
changelog:v2->v3:
- limited 80 chars per linev1->v2:
- limited comment to 80 chars per line in header file--- .../networking/devlink/devlink-port.rst | 25 +++++++++ include/net/devlink.h | 14 +++++ include/uapi/linux/devlink.h | 1 + net/devlink/port.c | 53 +++++++++++++++++++ 4 files changed, 93 insertions(+)Documentation/networking/devlink/devlink-port.rst +0
-0@@ -134,6 +134,9 @@ Users may also set the IPsec crypto capability ofthe function using Users may also set the IPsec packet capability of the function using
devlink port function set ipsec_packetcommand.Documentation/networking/devlink/devlink-port.rst +3
-0+Users may also set the maximum IO event queues of the function using +`devlink port function set max_io_eqs` command. + Function attributes ===================@@ -295,6 +298,28 @@ policy is processed in software by the kernel. function: hw_addr 00:00:00:00:00:00 ipsec_packet enabledDocumentation/networking/devlink/devlink-port.rst policy is processed in software by the kernel. · line 310
Quoted 10 lines
Documentation/networking/devlink/devlink-port.rst +10
-0+Maximum IO events queues setup +------------------------------ +When user sets maximum number of IO event queues for a SF or a VF, +such function driver is limited to consume only enforced number of IO +event queues. + +- Get maximum IO event queues of the VF device:: + + $ devlink port show pci/0000:06:00.0/2 + pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum0 vfnum 1
Documentation/networking/devlink/devlink-port.rst policy is processed in software by the kernel.
Quoted 10 lines
Documentation/networking/devlink/devlink-port.rst +10
-0+ function: + hw_addr 00:00:00:00:00:00 ipsec_packet disabled + max_io_eqs 10 + +- Set maximum IO event queues of the VF device:: + + $ devlink port function set pci/0000:06:00.0/2 max_io_eqs 32 + + $ devlink port show pci/0000:06:00.0/2 + pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum0 vfnum 1
Quoted 12 lines
Documentation/networking/devlink/devlink-port.rst +4
-0+ function: + hw_addr 00:00:00:00:00:00 ipsec_packet disabled + max_io_eqs 32 + Subfunction ============include/net/devlink.h index renamed from include/net/devlink.h +0
-0
9ac394bdfbe4..bb1af599d101100644include/net/devlink.h void devlink_free(struct devlink *devlink); · line 1605
Quoted 4 lines
include/net/devlink.h +1
-0@@ -1602,6 +1602,14 @@ void devlink_free(struct devlink *devlink);* capability. Should be used by device drivers to * enable/disable ipsec_packet capability of a * function managed by the devlink port. +* @port_fn_max_io_eqs_get: Callback used to get port function's maximum number
include/net/devlink.h void devlink_free(struct devlink *devlink); · line 1606
Quoted 1 line
include/net/devlink.h +1
-0+* of event queues. Should be used by device drivers to
include/net/devlink.h void devlink_free(struct devlink *devlink);
Quoted 3 lines
include/net/devlink.h +3
-0+* report the maximum event queues of a function +* managed by the devlink port. +* @port_fn_max_io_eqs_set: Callback used to set port function's maximum number
include/net/devlink.h void devlink_free(struct devlink *devlink); · line 1610
Quoted 1 line
include/net/devlink.h +1
-0+* of event queues. Should be used by device drivers to
Quoted 23 lines
include/net/devlink.h +8
-0+* configure maximum number of event queues +* of a function managed by the devlink port. * * Note: Driver should return -EOPNOTSUPP if it doesn't support * port function (@port_fn_*) handling for a particular port. @@ -1651,6 +1659,12 @@ struct devlink_port_ops {int (*port_fn_ipsec_packet_set)(struct devlink_port *devlink_port,bool enable,struct netlink_ext_ack *extack); +int (*port_fn_max_io_eqs_get)(struct devlink_port *devlink_port, +u32 *max_eqs, +struct netlink_ext_ack *extack); +int (*port_fn_max_io_eqs_set)(struct devlink_port *devlink_port, +u32 max_eqs, +struct netlink_ext_ack *extack); }; void devlink_port_init(struct devlink *devlink, diff --git
a/include/uapi/linux/devlink.hb/include/uapi/linux/devlink.hindex2da0c7eb6710..9401aa343673100644include/uapi/linux/devlink.h enum devlink_port_function_attr { · line 692
Quoted 7 lines
include/uapi/linux/devlink.h +1
-0@@ -686,6 +686,7 @@ enum devlink_port_function_attr {DEVLINK_PORT_FN_ATTR_OPSTATE ,/* u8 */ DEVLINK_PORT_FN_ATTR_CAPS ,/* bitfield32 */ DEVLINK_PORT_FN_ATTR_DEVLINK ,/* nested */ +DEVLINK_PORT_FN_ATTR_MAX_IO_EQS ,/* u32 */ __DEVLINK_PORT_FUNCTION_ATTR_MAX, DEVLINK_PORT_FUNCTION_ATTR_MAX =
__DEVLINK_PORT_FUNCTION_ATTR_MAX-net/devlink/port.c static const struct nla_policy
Quoted 5 lines
1 diff --git
a/net/devlink/port.cb/net/devlink/port.cindex118d130d2afd..be9158b4453c100644net/devlink/port.c +0
-0@@ -16,6 +16,7 @@ static const struct nla_policydevlink_function_nl_policy[DEVLINK_PORT_FUNCTION_
net/devlink/port.c static const struct nla_policy devlink_function_nl_policy[DEVLINK_PORT_FUNCTION_ · line 21
Quoted 3 lines
net/devlink/port.c +0
-0DEVLINK_PORT_FN_STATE_ACTIVE), [DEVLINK_PORT_FN_ATTR_CAPS] =
NLA_POLICY_BITFIELD32(DEVLINK_PORT_FN_CAPS_VALID_MASK),net/devlink/port.c static const struct nla_policy devlink_function_nl_policy[DEVLINK_PORT_FUNCTION_
Quoted 4 lines
net/devlink/port.c +1
-0+ [DEVLINK_PORT_FN_ATTR_MAX_IO_EQS] = { .type = NLA_U32 }, };#define ASSERT_DEVLINK_PORT_REGISTERED (devlink_port)\net/devlink/port.c static int devlink_port_fn_caps_fill(struct
Quoted 1 line
net/devlink/port.c +0
-0@@ -182,6 +183,30 @@ static int devlink_port_fn_caps_fill(struct
devlink_port*devlink_port,Quoted 5 lines
net/devlink/port.c +2
-0return 0 ; } +static int devlink_port_fn_max_io_eqs_fill(struct devlink_port *port, + struct sk_buff *msg,net/devlink/port.c static int devlink_port_fn_caps_fill(struct devlink_port *devlink_port,
Quoted 10 lines
net/devlink/port.c +10
-0+struct netlink_ext_ack *extack, + bool *msg_updated) +{ + u32 max_io_eqs; +int err; + +if (!port->ops->port_fn_max_io_eqs_get) +return 0 ; + + err = port->ops->port_fn_max_io_eqs_get(port, &max_io_eqs,extack);
net/devlink/port.c static int devlink_port_fn_caps_fill(struct devlink_port *devlink_port, · line 200
Quoted 3 lines
net/devlink/port.c +3
-0+if (err) { +if (err == -EOPNOTSUPP ) +return 0 ;Docs above says:
* Note: Driver should return-EOPNOTSUPPif it doesn't support
* port function (@port_fn_*) handling for a particular port.
Quoted 3 lines
But here you're returning 0 in both cases of no
port_fn_max_io_eqs_getorport_fn_max_io_eqs_get()returns EOPNOTSUPP.
When the port does not support this op, the function pointer is null and, 0 is returned as expected.
When the port for some reason has the ops function pointer set for a port, but if the port does not support the ops, it will return ENOPNOTSUPP. This may be possible when the driver has chosen to use same ops callback structure for multiple port flavors.
This code pattern is likely left over code of relatively recent work that moved ops from devlink instance to per port ops.
I propose to keep the current check as done in this patch,
and run a full audit of all the drivers, if all drivers have moved to per port ops, then simplify the code to drop the check for EOPNOTSUPP in a new series that may touch more drivers. Otherwise, we may end up failing the port show operation when it returns - ENOPNOTSUPP.
Quoted 77 lines
net/devlink/port.c static int devlink_port_fn_caps_fill(struct devlink_port *devlink_port,
Quoted 3 lines
net/devlink/port.c +3
-0+return err; + } + err = nla_put_u32(msg, DEVLINK_PORT_FN_ATTR_MAX_IO_EQS,
max_io_eqs);net/devlink/port.c static int devlink_port_fn_caps_fill(struct devlink_port *devlink_port,
Quoted 10 lines
net/devlink/port.c +7
-0+if (err) +return err; + *msg_updated = true; +return 0 ; +} +int devlink_nl_port_handle_fill (struct sk_buff *msg,struct devlink_port *devlink_port) { if (devlink_nl_put_handle(msg, devlink_port->devlink)) @@ -409,6 +434,18 @@ static int devlink_port_fn_caps_set(struct devlink_port
*devlink_port,Quoted 6 lines
net/devlink/port.c +3
-0return 0 ; } +static int +devlink_port_fn_max_io_eqs_set(struct devlink_port *devlink_port, + const struct nlattr *attr,net/devlink/port.c static int devlink_port_fn_caps_set(struct devlink_port *devlink_port,
Quoted 10 lines
net/devlink/port.c +8
-0+ struct netlink_ext_ack *extack) { +u32 max_io_eqs; + + max_io_eqs =nla_get_u32 (attr); +return devlink_port->ops->port_fn_max_io_eqs_set (devlink_port, + max_io_eqs, extack); +} + static int devlink_nl_port_function_attrs_put(struct sk_buff *msg, struct
devlink_port*port,Quoted 1 line
struct
netlink_ext_ack*extack) @@ -428,6+465,9 @@
Quoted 1 line
devlink_nl_port_function_attrs_put(struct
sk_buff*msg, struct
devlink_port*pornet/devlink/port.c devlink_nl_port_function_attrs_put(struct sk_buff *msg, struct devlink_port *por
Quoted 11 lines
net/devlink/port.c +5
-0if (err) goto out; err =devlink_port_fn_state_fill (port, msg, extack, &msg_updated); +if (err) + goto out; + err =devlink_port_fn_max_io_eqs_fill (port, msg, extack, +&msg_updated);if (err) goto out; err = devlink_rel_devlink_handle_put(msg, port->devlink, @@ -726,6 +766,12 @@ static int devlink_port_function_validate(struct devlink_port
*devlink_port,net/devlink/port.c static int devlink_port_function_validate(struct devlink_port *devlink_port,
Quoted 6 lines
net/devlink/port.c +3
-0} } } +if (tb[DEVLINK_PORT_FN_ATTR_MAX_IO_EQS ] && + !ops->port_fn_max_io_eqs_set) { + NL_SET_ERR_MSG_ATTR(extack,tb[DEVLINK_PORT_FN_ATTR_MAX_IO_EQS],
net/devlink/port.c
Quoted 1 line
net/devlink/port.c +1
-0+ "Function does not support max_io_eqssetting");
net/devlink/port.c static int devlink_port_function_set(struct
Quoted 6 lines
net/devlink/port.c +2
-0+return -EOPNOTSUPP ; + }return 0 ; }@@ -761,6 +807,13 @@ static int devlink_port_function_set(struct
devlink_port*port,Quoted 13 lines
net/devlink/port.c +7
-0return err; } + attr = tb[DEVLINK_PORT_FN_ATTR_MAX_IO_EQS ]; +if (attr) { + err =devlink_port_fn_max_io_eqs_set (port, attr, extack); +if (err) +return err; + } +/* Keep this as the last function attribute set, so that when * multiple port function attributes are set along with state, * Those can be applied first before activating the state.
Re: [net-next v3 1/2] devlink: Support setting max_io_eqs
Quoted 192 lines · Parav Pandit
Hi David,
Quoted 185 lines
From: David Wei <[email protected]>
Sent: Thursday, April 4, 2024 10:29 PMQuoted 11 lines · Parav Pandit
Many devices send event notifications for the IO queues, such as tx and rx queues, through event queues.
Enable a privileged owner, such as a hypervisor PF, to set the number of IO event queues for the VF and SF during the provisioning stage.
example:
Get maximum IO event queues of the VF device::$ devlink port show pci/0000:06:00.0/2 pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0vfnum 1
Quoted 10 lines
function: hw_addr 00:00:00:00:00:00 ipsec_packet disabled max_io_eqs10
Set maximum IO event queues of the VF device::
$ devlink port function set pci/0000:06:00.0/2 max_io_eqs 32$ devlink port show pci/0000:06:00.0/2 pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0vfnum 1
Quoted 39 lines
function: hw_addr 00:00:00:00:00:00 ipsec_packet disabled max_io_eqs32
Reviewed-by Jiri Pirko <[email protected]>Reviewed-by Shay Drory <[email protected]>Signed-off-by Parav Pandit <[email protected]>---
changelog:v2->v3:
- limited 80 chars per linev1->v2:
- limited comment to 80 chars per line in header file--- .../networking/devlink/devlink-port.rst | 25 +++++++++ include/net/devlink.h | 14 +++++ include/uapi/linux/devlink.h | 1 + net/devlink/port.c | 53 +++++++++++++++++++ 4 files changed, 93 insertions(+)Documentation/networking/devlink/devlink-port.rst +0
-0@@ -134,6 +134,9 @@ Users may also set the IPsec crypto capability ofthe function using Users may also set the IPsec packet capability of the function using
devlink port function set ipsec_packetcommand.Documentation/networking/devlink/devlink-port.rst +3
-0+Users may also set the maximum IO event queues of the function using +`devlink port function set max_io_eqs` command. + Function attributes ===================@@ -295,6 +298,28 @@ policy is processed in software by the kernel. function: hw_addr 00:00:00:00:00:00 ipsec_packet enabledDocumentation/networking/devlink/devlink-port.rst policy is processed in software by the kernel. · line 310
Quoted 10 lines
Documentation/networking/devlink/devlink-port.rst +10
-0+Maximum IO events queues setup +------------------------------ +When user sets maximum number of IO event queues for a SF or a VF, +such function driver is limited to consume only enforced number of IO +event queues. + +- Get maximum IO event queues of the VF device:: + + $ devlink port show pci/0000:06:00.0/2 + pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum0 vfnum 1
Documentation/networking/devlink/devlink-port.rst policy is processed in software by the kernel.
Quoted 10 lines
Documentation/networking/devlink/devlink-port.rst +10
-0+ function: + hw_addr 00:00:00:00:00:00 ipsec_packet disabled + max_io_eqs 10 + +- Set maximum IO event queues of the VF device:: + + $ devlink port function set pci/0000:06:00.0/2 max_io_eqs 32 + + $ devlink port show pci/0000:06:00.0/2 + pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum0 vfnum 1
Quoted 12 lines
Documentation/networking/devlink/devlink-port.rst +4
-0+ function: + hw_addr 00:00:00:00:00:00 ipsec_packet disabled + max_io_eqs 32 + Subfunction ============include/net/devlink.h index renamed from include/net/devlink.h +0
-0
9ac394bdfbe4..bb1af599d101100644include/net/devlink.h void devlink_free(struct devlink *devlink); · line 1605
Quoted 4 lines
include/net/devlink.h +1
-0@@ -1602,6 +1602,14 @@ void devlink_free(struct devlink *devlink);* capability. Should be used by device drivers to * enable/disable ipsec_packet capability of a * function managed by the devlink port. +* @port_fn_max_io_eqs_get: Callback used to get port function's maximum number
include/net/devlink.h void devlink_free(struct devlink *devlink); · line 1606
Quoted 1 line
include/net/devlink.h +1
-0+* of event queues. Should be used by device drivers to
include/net/devlink.h void devlink_free(struct devlink *devlink);
Quoted 3 lines
include/net/devlink.h +3
-0+* report the maximum event queues of a function +* managed by the devlink port. +* @port_fn_max_io_eqs_set: Callback used to set port function's maximum number
include/net/devlink.h void devlink_free(struct devlink *devlink); · line 1610
Quoted 1 line
include/net/devlink.h +1
-0+* of event queues. Should be used by device drivers to
Quoted 23 lines
include/net/devlink.h +8
-0+* configure maximum number of event queues +* of a function managed by the devlink port. * * Note: Driver should return -EOPNOTSUPP if it doesn't support * port function (@port_fn_*) handling for a particular port. @@ -1651,6 +1659,12 @@ struct devlink_port_ops {int (*port_fn_ipsec_packet_set)(struct devlink_port *devlink_port,bool enable,struct netlink_ext_ack *extack); +int (*port_fn_max_io_eqs_get)(struct devlink_port *devlink_port, +u32 *max_eqs, +struct netlink_ext_ack *extack); +int (*port_fn_max_io_eqs_set)(struct devlink_port *devlink_port, +u32 max_eqs, +struct netlink_ext_ack *extack); }; void devlink_port_init(struct devlink *devlink, diff --git
a/include/uapi/linux/devlink.hb/include/uapi/linux/devlink.hindex2da0c7eb6710..9401aa343673100644include/uapi/linux/devlink.h enum devlink_port_function_attr { · line 692
Quoted 7 lines
include/uapi/linux/devlink.h +1
-0@@ -686,6 +686,7 @@ enum devlink_port_function_attr {DEVLINK_PORT_FN_ATTR_OPSTATE ,/* u8 */ DEVLINK_PORT_FN_ATTR_CAPS ,/* bitfield32 */ DEVLINK_PORT_FN_ATTR_DEVLINK ,/* nested */ +DEVLINK_PORT_FN_ATTR_MAX_IO_EQS ,/* u32 */ __DEVLINK_PORT_FUNCTION_ATTR_MAX, DEVLINK_PORT_FUNCTION_ATTR_MAX =
__DEVLINK_PORT_FUNCTION_ATTR_MAX-net/devlink/port.c static const struct nla_policy
Quoted 5 lines
1 diff --git
a/net/devlink/port.cb/net/devlink/port.cindex118d130d2afd..be9158b4453c100644net/devlink/port.c +0
-0@@ -16,6 +16,7 @@ static const struct nla_policydevlink_function_nl_policy[DEVLINK_PORT_FUNCTION_
net/devlink/port.c static const struct nla_policy devlink_function_nl_policy[DEVLINK_PORT_FUNCTION_ · line 21
Quoted 3 lines
net/devlink/port.c +0
-0DEVLINK_PORT_FN_STATE_ACTIVE), [DEVLINK_PORT_FN_ATTR_CAPS] =
NLA_POLICY_BITFIELD32(DEVLINK_PORT_FN_CAPS_VALID_MASK),net/devlink/port.c static const struct nla_policy devlink_function_nl_policy[DEVLINK_PORT_FUNCTION_
Quoted 4 lines
net/devlink/port.c +1
-0+ [DEVLINK_PORT_FN_ATTR_MAX_IO_EQS] = { .type = NLA_U32 }, };#define ASSERT_DEVLINK_PORT_REGISTERED (devlink_port)\net/devlink/port.c static int devlink_port_fn_caps_fill(struct
Quoted 1 line
net/devlink/port.c +0
-0@@ -182,6 +183,30 @@ static int devlink_port_fn_caps_fill(struct
devlink_port*devlink_port,Quoted 5 lines
net/devlink/port.c +2
-0return 0 ; } +static int devlink_port_fn_max_io_eqs_fill(struct devlink_port *port, + struct sk_buff *msg,net/devlink/port.c static int devlink_port_fn_caps_fill(struct devlink_port *devlink_port,
Quoted 10 lines
net/devlink/port.c +10
-0+struct netlink_ext_ack *extack, + bool *msg_updated) +{ + u32 max_io_eqs; +int err; + +if (!port->ops->port_fn_max_io_eqs_get) +return 0 ; + + err = port->ops->port_fn_max_io_eqs_get(port, &max_io_eqs,extack);
net/devlink/port.c static int devlink_port_fn_caps_fill(struct devlink_port *devlink_port, · line 200
Quoted 3 lines
net/devlink/port.c +3
-0+if (err) { +if (err == -EOPNOTSUPP ) +return 0 ;Docs above says:
* Note: Driver should return-EOPNOTSUPPif it doesn't support
* port function (@port_fn_*) handling for a particular port.Quoted 3 lines
But here you're returning 0 in both cases of no
port_fn_max_io_eqs_getorport_fn_max_io_eqs_get()returns EOPNOTSUPP.When the port does not support this op, the function pointer is null and, 0 is returned as expected.
Quoted 7 lines
When the port for some reason has the ops function pointer set for a port, but if the port does not support the ops, it will return ENOPNOTSUPP. This may be possible when the driver has chosen to use same ops callback structure for multiple port flavors.
This code pattern is likely left over code of relatively recent work that moved ops from devlink instance to per port ops.
I propose to keep the current check as done in this patch,
and run a full audit of all the drivers, if all drivers have moved to per port ops, then simplify the code to drop the check for EOPNOTSUPP in a new series that may touch more drivers. Otherwise, we may end up failing the port show operation when it returns - ENOPNOTSUPP.
Thanks for the explanation. So ideally each port flavour has its own unique set of struct ops, and if something is not supported then don't set the func ptr in the struct ops.
Yes, I see that 0 has to be returned for devlink_port_fn_caps_fill() to succeed.
Had a brief look and there's only a handful of drivers (mlx, nfp, ice) that use devlink_port_ops.
Quoted 78 lines
Quoted 78 lines
net/devlink/port.c static int devlink_port_fn_caps_fill(struct devlink_port *devlink_port,
Quoted 4 lines
net/devlink/port.c +3
-0+return err; + } + err = nla_put_u32(msg, DEVLINK_PORT_FN_ATTR_MAX_IO_EQS,
max_io_eqs);net/devlink/port.c static int devlink_port_fn_caps_fill(struct devlink_port *devlink_port,
Quoted 10 lines
net/devlink/port.c +7
-0+if (err) +return err; + *msg_updated = true; +return 0 ; +} +int devlink_nl_port_handle_fill (struct sk_buff *msg,struct devlink_port *devlink_port) { if (devlink_nl_put_handle(msg, devlink_port->devlink)) @@ -409,6 +434,18 @@ static int devlink_port_fn_caps_set(struct devlink_port
*devlink_port,Quoted 6 lines
net/devlink/port.c +3
-0return 0 ; } +static int +devlink_port_fn_max_io_eqs_set(struct devlink_port *devlink_port, + const struct nlattr *attr,net/devlink/port.c static int devlink_port_fn_caps_set(struct devlink_port *devlink_port,
Quoted 10 lines
net/devlink/port.c +8
-0+ struct netlink_ext_ack *extack) { +u32 max_io_eqs; + + max_io_eqs =nla_get_u32 (attr); +return devlink_port->ops->port_fn_max_io_eqs_set (devlink_port, + max_io_eqs, extack); +} + static int devlink_nl_port_function_attrs_put(struct sk_buff *msg, struct
devlink_port*port,Quoted 1 line
struct
netlink_ext_ack*extack) @@ -428,6+465,9 @@
Quoted 1 line
devlink_nl_port_function_attrs_put(struct
sk_buff*msg, struct
devlink_port*pornet/devlink/port.c devlink_nl_port_function_attrs_put(struct sk_buff *msg, struct devlink_port *por
Quoted 11 lines
net/devlink/port.c +5
-0if (err) goto out; err =devlink_port_fn_state_fill (port, msg, extack, &msg_updated); +if (err) + goto out; + err =devlink_port_fn_max_io_eqs_fill (port, msg, extack, +&msg_updated);if (err) goto out; err = devlink_rel_devlink_handle_put(msg, port->devlink, @@ -726,6 +766,12 @@ static int devlink_port_function_validate(struct devlink_port
*devlink_port,net/devlink/port.c static int devlink_port_function_validate(struct devlink_port *devlink_port,
Quoted 6 lines
net/devlink/port.c +3
-0} } } +if (tb[DEVLINK_PORT_FN_ATTR_MAX_IO_EQS ] && + !ops->port_fn_max_io_eqs_set) { + NL_SET_ERR_MSG_ATTR(extack,tb[DEVLINK_PORT_FN_ATTR_MAX_IO_EQS],
net/devlink/port.c
Quoted 1 line
net/devlink/port.c +1
-0+ "Function does not support max_io_eqssetting");
net/devlink/port.c static int devlink_port_function_set(struct
Quoted 6 lines
net/devlink/port.c +2
-0+return -EOPNOTSUPP ; + }return 0 ; }@@ -761,6 +807,13 @@ static int devlink_port_function_set(struct
devlink_port*port,Quoted 13 lines
net/devlink/port.c +7
-0return err; } + attr = tb[DEVLINK_PORT_FN_ATTR_MAX_IO_EQS ]; +if (attr) { + err =devlink_port_fn_max_io_eqs_set (port, attr, extack); +if (err) +return err; + } +/* Keep this as the last function attribute set, so that when * multiple port function attributes are set along with state, * Those can be applied first before activating the state.
RE: [net-next v3 1/2] devlink: Support setting max_io_eqs
Quoted 220 lines
From: David Wei <[email protected]>
Sent: Friday, April 5, 2024 12:10 AMQuoted 11 lines · Parav Pandit
Hi David,
Quoted 9 lines
From: David Wei <[email protected]>
Sent: Thursday, April 4, 2024 10:29 PMQuoted 5 lines · Parav Pandit
Many devices send event notifications for the IO queues, such as tx and rx queues, through event queues.
Enable a privileged owner, such as a hypervisor PF, to set the number of IO event queues for the VF and SF during the provisioning
stage.
Quoted 40 lines
Quoted 40 lines
Quoted 7 lines
example:
Get maximum IO event queues of the VF device::$ devlink port show pci/0000:06:00.0/2 pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0vfnum 1
Quoted 11 lines
function: hw_addr 00:00:00:00:00:00 ipsec_packet disabled max_io_eqs10
Set maximum IO event queues of the VF device::
$ devlink port function set pci/0000:06:00.0/2 max_io_eqs 32$ devlink port show pci/0000:06:00.0/2 pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0vfnum 1
Quoted 20 lines
function: hw_addr 00:00:00:00:00:00 ipsec_packet disabled max_io_eqs32
Reviewed-by Jiri Pirko <[email protected]>Reviewed-by Shay Drory <[email protected]>Signed-off-by Parav Pandit <[email protected]>---
changelog:v2->v3:
- limited 80 chars per linev1->v2:
- limited comment to 80 chars per line in header file--- .../networking/devlink/devlink-port.rst | 25 +++++++++ include/net/devlink.h | 14 +++++ include/uapi/linux/devlink.h | 1 + net/devlink/port.c | 53 +++++++++++++++++++ 4 files changed, 93 insertions(+)Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Documentation/networking/devlink/devlink-port.rst +0
-0@@ -134,6 +134,9 @@ Users may also set the IPsec crypto capabilityof the function using Users may also set the IPsec packet capability of the function using
devlink port function set ipsec_packetcommand.
Quoted 132 lines
Quoted 132 lines
Quoted 12 lines
Documentation/networking/devlink/devlink-port.rst +4
-0+Users may also set the maximum IO event queues of the function +using `devlink port function set max_io_eqs` command. + Function attributes ===================@@ -295,6 +298,28 @@ policy is processed in software by the kernel. function: hw_addr 00:00:00:00:00:00 ipsec_packet enabled +Maximum IO events queues setupDocumentation/networking/devlink/devlink-port.rst policy is processed in software by the kernel. · line 311
Quoted 10 lines
Documentation/networking/devlink/devlink-port.rst +10
-0+------------------------------ +When user sets maximum number of IO event queues for a SF or a VF, +such function driver is limited to consume only enforced number of +IO event queues. + +- Get maximum IO event queues of the VF device:: + + $ devlink port show pci/0000:06:00.0/2 + pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf + pfnum0 vfnum 1
Documentation/networking/devlink/devlink-port.rst policy is processed in software by the kernel.
Quoted 11 lines
Documentation/networking/devlink/devlink-port.rst +11
-0+ function: + hw_addr 00:00:00:00:00:00 ipsec_packet disabled + max_io_eqs 10 + +- Set maximum IO event queues of the VF device:: + + $ devlink port function set pci/0000:06:00.0/2 max_io_eqs 32 + + $ devlink port show pci/0000:06:00.0/2 + pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf + pfnum0 vfnum 1
Quoted 12 lines
Documentation/networking/devlink/devlink-port.rst +4
-0+ function: + hw_addr 00:00:00:00:00:00 ipsec_packet disabled + max_io_eqs 32 + Subfunction ============include/net/devlink.h index renamed from include/net/devlink.h +0
-0
9ac394bdfbe4..bb1af599d101100644include/net/devlink.h void devlink_free(struct devlink *devlink); · line 1605
Quoted 4 lines
include/net/devlink.h +1
-0@@ -1602,6 +1602,14 @@ void devlink_free(struct devlink *devlink);* capability. Should be used by device drivers to * enable/disable ipsec_packet capability of a * function managed by the devlink port. +* @port_fn_max_io_eqs_get: Callback used to get port function's maximum number
include/net/devlink.h void devlink_free(struct devlink *devlink); · line 1606
Quoted 1 line
include/net/devlink.h +1
-0+* of event queues. Should be used by device drivers to
include/net/devlink.h void devlink_free(struct devlink *devlink);
Quoted 3 lines
include/net/devlink.h +3
-0+* report the maximum event queues of a function +* managed by the devlink port. +* @port_fn_max_io_eqs_set: Callback used to set port function's maximum number
include/net/devlink.h void devlink_free(struct devlink *devlink); · line 1610
Quoted 1 line
include/net/devlink.h +1
-0+* of event queues. Should be used by device drivers to
Quoted 23 lines
include/net/devlink.h +8
-0+* configure maximum number of event queues +* of a function managed by the devlink port. * * Note: Driver should return -EOPNOTSUPP if it doesn't support * port function (@port_fn_*) handling for a particular port. @@ -1651,6 +1659,12 @@ struct devlink_port_ops {int (*port_fn_ipsec_packet_set)(struct devlink_port *devlink_port,bool enable,struct netlink_ext_ack *extack); +int (*port_fn_max_io_eqs_get)(struct devlink_port *devlink_port, +u32 *max_eqs, +struct netlink_ext_ack *extack); +int (*port_fn_max_io_eqs_set)(struct devlink_port *devlink_port, +u32 max_eqs, +struct netlink_ext_ack *extack); }; void devlink_port_init(struct devlink *devlink, diff --git
a/include/uapi/linux/devlink.hb/include/uapi/linux/devlink.hindex2da0c7eb6710..9401aa343673100644include/uapi/linux/devlink.h enum devlink_port_function_attr { · line 692
Quoted 7 lines
include/uapi/linux/devlink.h +1
-0@@ -686,6 +686,7 @@ enum devlink_port_function_attr {DEVLINK_PORT_FN_ATTR_OPSTATE ,/* u8 */ DEVLINK_PORT_FN_ATTR_CAPS ,/* bitfield32 */ DEVLINK_PORT_FN_ATTR_DEVLINK ,/* nested */ +DEVLINK_PORT_FN_ATTR_MAX_IO_EQS ,/* u32 */ __DEVLINK_PORT_FUNCTION_ATTR_MAX, DEVLINK_PORT_FUNCTION_ATTR_MAX =
__DEVLINK_PORT_FUNCTION_ATTR_MAX-net/devlink/port.c static const struct nla_policy
Quoted 5 lines
1 diff --git
a/net/devlink/port.cb/net/devlink/port.cindex118d130d2afd..be9158b4453c100644net/devlink/port.c +0
-0@@ -16,6 +16,7 @@ static const struct nla_policydevlink_function_nl_policy[DEVLINK_PORT_FUNCTION_
net/devlink/port.c static const struct nla_policy devlink_function_nl_policy[DEVLINK_PORT_FUNCTION_ · line 21
Quoted 3 lines
net/devlink/port.c +0
-0DEVLINK_PORT_FN_STATE_ACTIVE), [DEVLINK_PORT_FN_ATTR_CAPS] =
NLA_POLICY_BITFIELD32(DEVLINK_PORT_FN_CAPS_VALID_MASK),net/devlink/port.c static const struct nla_policy devlink_function_nl_policy[DEVLINK_PORT_FUNCTION_
Quoted 4 lines
net/devlink/port.c +1
-0+ [DEVLINK_PORT_FN_ATTR_MAX_IO_EQS] = { .type = NLA_U32 }, };#define ASSERT_DEVLINK_PORT_REGISTERED (devlink_port)\net/devlink/port.c static int devlink_port_fn_caps_fill(struct
Quoted 1 line
net/devlink/port.c +0
-0@@ -182,6 +183,30 @@ static int devlink_port_fn_caps_fill(struct
devlink_port*devlink_port,Quoted 5 lines
net/devlink/port.c +2
-0return 0 ; } +static int devlink_port_fn_max_io_eqs_fill(struct devlink_port *port, + struct sk_buff *msg,net/devlink/port.c static int devlink_port_fn_caps_fill(struct devlink_port *devlink_port,
Quoted 10 lines
net/devlink/port.c +10
-0+struct netlink_ext_ack *extack, + bool *msg_updated) +{ + u32 max_io_eqs; +int err; + +if (!port->ops->port_fn_max_io_eqs_get) +return 0 ; + + err = port->ops->port_fn_max_io_eqs_get(port, &max_io_eqs,extack);
net/devlink/port.c static int devlink_port_fn_caps_fill(struct devlink_port *devlink_port, · line 200
Quoted 3 lines
net/devlink/port.c +3
-0+if (err) { +if (err == -EOPNOTSUPP ) +return 0 ;Docs above says:
* Note: Driver should return-EOPNOTSUPPif it doesn't support
* port function (@port_fn_*) handling for a particular port.Quoted 5 lines
Quoted 4 lines
But here you're returning 0 in both cases of no
port_fn_max_io_eqs_getorport_fn_max_io_eqs_get()returns EOPNOTSUPP.When the port does not support this op, the function pointer is null and, 0
is returned as expected.
Quoted 2 lines
When the port for some reason has the ops function pointer set for a port,
but if the port does not support the ops, it will return ENOPNOTSUPP.
Quoted 1 line
This may be possible when the driver has chosen to use same ops callback
structure for multiple port flavors.
Quoted 2 lines
This code pattern is likely left over code of relatively recent work that
moved ops from devlink instance to per port ops.
Quoted 2 lines
I propose to keep the current check as done in this patch, and run a full audit of all the drivers, if all drivers have moved to per port ops, then
simplify the code to drop the check for EOPNOTSUPP in a new series that may touch more drivers.
Quoted 1 line
Otherwise, we may end up failing the port show operation when it returns
- ENOPNOTSUPP.
Quoted 9 lines
Thanks for the explanation. So ideally each port flavour has its own unique set of struct ops, and if something is not supported then don't set the func ptr in the struct ops.
Yes, I see that 0 has to be returned for
devlink_port_fn_caps_fill()to succeed.Had a brief look and there's only a handful of drivers (mlx, nfp, ice) that use
devlink_port_ops.
And netdevsim too. :)
Can we please the cleanup in a separate series?
Post this may be later in the month, I may have cycles to audit and simplify. Would it be ok with you?
Quoted 79 lines
Quoted 79 lines
Quoted 79 lines
net/devlink/port.c static int devlink_port_fn_caps_fill(struct devlink_port *devlink_port,
Quoted 4 lines
net/devlink/port.c +3
-0+return err; + } + err = nla_put_u32(msg, DEVLINK_PORT_FN_ATTR_MAX_IO_EQS,
max_io_eqs);net/devlink/port.c static int devlink_port_fn_caps_fill(struct devlink_port *devlink_port,
Quoted 10 lines
net/devlink/port.c +7
-0+if (err) +return err; + *msg_updated = true; +return 0 ; +} +int devlink_nl_port_handle_fill (struct sk_buff *msg,struct devlink_port *devlink_port) { if (devlink_nl_put_handle(msg, devlink_port->devlink)) @@ -409,6 +434,18 @@ static int devlink_port_fn_caps_set(struct devlink_port
*devlink_port,Quoted 6 lines
net/devlink/port.c +3
-0return 0 ; } +static int +devlink_port_fn_max_io_eqs_set(struct devlink_port *devlink_port, + const struct nlattr *attr,net/devlink/port.c static int devlink_port_fn_caps_set(struct devlink_port *devlink_port,
Quoted 10 lines
net/devlink/port.c +8
-0+ struct netlink_ext_ack *extack) { +u32 max_io_eqs; + + max_io_eqs =nla_get_u32 (attr); +return devlink_port->ops->port_fn_max_io_eqs_set (devlink_port, + max_io_eqs, extack); +} + static int devlink_nl_port_function_attrs_put(struct sk_buff *msg, struct
devlink_port*port,Quoted 1 line
struct
netlink_ext_ack*extack) @@ -428,6+465,9 @@
Quoted 1 line
devlink_nl_port_function_attrs_put(struct
sk_buff*msg, struct
devlink_port*pornet/devlink/port.c devlink_nl_port_function_attrs_put(struct sk_buff *msg, struct devlink_port *por
Quoted 12 lines
net/devlink/port.c +6
-0if (err) goto out; err =devlink_port_fn_state_fill (port, msg, extack, &msg_updated); +if (err) + goto out; + err =devlink_port_fn_max_io_eqs_fill (port, msg, extack, +&msg_updated);if (err) goto out; err = devlink_rel_devlink_handle_put(msg, port->devlink, @@ -726,6 +766,12 @@ static int devlink_port_function_validate(struct +devlink_port
*devlink_port,net/devlink/port.c static int devlink_port_function_validate(struct devlink_port *devlink_port,
Quoted 6 lines
net/devlink/port.c +3
-0} } } +if (tb[DEVLINK_PORT_FN_ATTR_MAX_IO_EQS ] && + !ops->port_fn_max_io_eqs_set) { + NL_SET_ERR_MSG_ATTR(extack,tb[DEVLINK_PORT_FN_ATTR_MAX_IO_EQS],
net/devlink/port.c
Quoted 1 line
net/devlink/port.c +1
-0+ "Function does not support max_io_eqssetting");
net/devlink/port.c static int devlink_port_function_set(struct
Quoted 6 lines
net/devlink/port.c +2
-0+return -EOPNOTSUPP ; + }return 0 ; }@@ -761,6 +807,13 @@ static int devlink_port_function_set(struct
devlink_port*port,Quoted 13 lines
net/devlink/port.c +7
-0return err; } + attr = tb[DEVLINK_PORT_FN_ATTR_MAX_IO_EQS ]; +if (attr) { + err =devlink_port_fn_max_io_eqs_set (port, attr, extack); +if (err) +return err; + } +/* Keep this as the last function attribute set, so that when * multiple port function attributes are set along with state, * Those can be applied first before activating the state.
Re: [net-next v3 1/2] devlink: Support setting max_io_eqs
Quoted 226 lines · Parav Pandit
Quoted 226 lines
From: David Wei <[email protected]>
Sent: Friday, April 5, 2024 12:10 AMQuoted 11 lines · Parav Pandit
Hi David,
Quoted 9 lines
From: David Wei <[email protected]>
Sent: Thursday, April 4, 2024 10:29 PMQuoted 5 lines · Parav Pandit
Many devices send event notifications for the IO queues, such as tx and rx queues, through event queues.
Enable a privileged owner, such as a hypervisor PF, to set the number of IO event queues for the VF and SF during the provisioning
stage.
Quoted 40 lines
Quoted 40 lines
Quoted 7 lines
example:
Get maximum IO event queues of the VF device::$ devlink port show pci/0000:06:00.0/2 pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0vfnum 1
Quoted 11 lines
function: hw_addr 00:00:00:00:00:00 ipsec_packet disabled max_io_eqs10
Set maximum IO event queues of the VF device::
$ devlink port function set pci/0000:06:00.0/2 max_io_eqs 32$ devlink port show pci/0000:06:00.0/2 pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0vfnum 1
Quoted 20 lines
function: hw_addr 00:00:00:00:00:00 ipsec_packet disabled max_io_eqs32
Reviewed-by Jiri Pirko <[email protected]>Reviewed-by Shay Drory <[email protected]>Signed-off-by Parav Pandit <[email protected]>---
changelog:v2->v3:
- limited 80 chars per linev1->v2:
- limited comment to 80 chars per line in header file--- .../networking/devlink/devlink-port.rst | 25 +++++++++ include/net/devlink.h | 14 +++++ include/uapi/linux/devlink.h | 1 + net/devlink/port.c | 53 +++++++++++++++++++ 4 files changed, 93 insertions(+)Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Documentation/networking/devlink/devlink-port.rst +0
-0@@ -134,6 +134,9 @@ Users may also set the IPsec crypto capabilityof the function using Users may also set the IPsec packet capability of the function using
devlink port function set ipsec_packetcommand.
Quoted 132 lines
Quoted 132 lines
Quoted 12 lines
Documentation/networking/devlink/devlink-port.rst +4
-0+Users may also set the maximum IO event queues of the function +using `devlink port function set max_io_eqs` command. + Function attributes ===================@@ -295,6 +298,28 @@ policy is processed in software by the kernel. function: hw_addr 00:00:00:00:00:00 ipsec_packet enabled +Maximum IO events queues setupDocumentation/networking/devlink/devlink-port.rst policy is processed in software by the kernel. · line 311
Quoted 10 lines
Documentation/networking/devlink/devlink-port.rst +10
-0+------------------------------ +When user sets maximum number of IO event queues for a SF or a VF, +such function driver is limited to consume only enforced number of +IO event queues. + +- Get maximum IO event queues of the VF device:: + + $ devlink port show pci/0000:06:00.0/2 + pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf + pfnum0 vfnum 1
Documentation/networking/devlink/devlink-port.rst policy is processed in software by the kernel.
Quoted 11 lines
Documentation/networking/devlink/devlink-port.rst +11
-0+ function: + hw_addr 00:00:00:00:00:00 ipsec_packet disabled + max_io_eqs 10 + +- Set maximum IO event queues of the VF device:: + + $ devlink port function set pci/0000:06:00.0/2 max_io_eqs 32 + + $ devlink port show pci/0000:06:00.0/2 + pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf + pfnum0 vfnum 1
Quoted 12 lines
Documentation/networking/devlink/devlink-port.rst +4
-0+ function: + hw_addr 00:00:00:00:00:00 ipsec_packet disabled + max_io_eqs 32 + Subfunction ============include/net/devlink.h index renamed from include/net/devlink.h +0
-0
9ac394bdfbe4..bb1af599d101100644include/net/devlink.h void devlink_free(struct devlink *devlink); · line 1605
Quoted 4 lines
include/net/devlink.h +1
-0@@ -1602,6 +1602,14 @@ void devlink_free(struct devlink *devlink);* capability. Should be used by device drivers to * enable/disable ipsec_packet capability of a * function managed by the devlink port. +* @port_fn_max_io_eqs_get: Callback used to get port function's maximum number
include/net/devlink.h void devlink_free(struct devlink *devlink); · line 1606
Quoted 1 line
include/net/devlink.h +1
-0+* of event queues. Should be used by device drivers to
include/net/devlink.h void devlink_free(struct devlink *devlink);
Quoted 3 lines
include/net/devlink.h +3
-0+* report the maximum event queues of a function +* managed by the devlink port. +* @port_fn_max_io_eqs_set: Callback used to set port function's maximum number
include/net/devlink.h void devlink_free(struct devlink *devlink); · line 1610
Quoted 1 line
include/net/devlink.h +1
-0+* of event queues. Should be used by device drivers to
Quoted 23 lines
include/net/devlink.h +8
-0+* configure maximum number of event queues +* of a function managed by the devlink port. * * Note: Driver should return -EOPNOTSUPP if it doesn't support * port function (@port_fn_*) handling for a particular port. @@ -1651,6 +1659,12 @@ struct devlink_port_ops {int (*port_fn_ipsec_packet_set)(struct devlink_port *devlink_port,bool enable,struct netlink_ext_ack *extack); +int (*port_fn_max_io_eqs_get)(struct devlink_port *devlink_port, +u32 *max_eqs, +struct netlink_ext_ack *extack); +int (*port_fn_max_io_eqs_set)(struct devlink_port *devlink_port, +u32 max_eqs, +struct netlink_ext_ack *extack); }; void devlink_port_init(struct devlink *devlink, diff --git
a/include/uapi/linux/devlink.hb/include/uapi/linux/devlink.hindex2da0c7eb6710..9401aa343673100644include/uapi/linux/devlink.h enum devlink_port_function_attr { · line 692
Quoted 7 lines
include/uapi/linux/devlink.h +1
-0@@ -686,6 +686,7 @@ enum devlink_port_function_attr {DEVLINK_PORT_FN_ATTR_OPSTATE ,/* u8 */ DEVLINK_PORT_FN_ATTR_CAPS ,/* bitfield32 */ DEVLINK_PORT_FN_ATTR_DEVLINK ,/* nested */ +DEVLINK_PORT_FN_ATTR_MAX_IO_EQS ,/* u32 */ __DEVLINK_PORT_FUNCTION_ATTR_MAX, DEVLINK_PORT_FUNCTION_ATTR_MAX =
__DEVLINK_PORT_FUNCTION_ATTR_MAX-net/devlink/port.c static const struct nla_policy
Quoted 5 lines
1 diff --git
a/net/devlink/port.cb/net/devlink/port.cindex118d130d2afd..be9158b4453c100644net/devlink/port.c +0
-0@@ -16,6 +16,7 @@ static const struct nla_policydevlink_function_nl_policy[DEVLINK_PORT_FUNCTION_
net/devlink/port.c static const struct nla_policy devlink_function_nl_policy[DEVLINK_PORT_FUNCTION_ · line 21
Quoted 3 lines
net/devlink/port.c +0
-0DEVLINK_PORT_FN_STATE_ACTIVE), [DEVLINK_PORT_FN_ATTR_CAPS] =
NLA_POLICY_BITFIELD32(DEVLINK_PORT_FN_CAPS_VALID_MASK),net/devlink/port.c static const struct nla_policy devlink_function_nl_policy[DEVLINK_PORT_FUNCTION_
Quoted 4 lines
net/devlink/port.c +1
-0+ [DEVLINK_PORT_FN_ATTR_MAX_IO_EQS] = { .type = NLA_U32 }, };#define ASSERT_DEVLINK_PORT_REGISTERED (devlink_port)\net/devlink/port.c static int devlink_port_fn_caps_fill(struct
Quoted 1 line
net/devlink/port.c +0
-0@@ -182,6 +183,30 @@ static int devlink_port_fn_caps_fill(struct
devlink_port*devlink_port,Quoted 5 lines
net/devlink/port.c +2
-0return 0 ; } +static int devlink_port_fn_max_io_eqs_fill(struct devlink_port *port, + struct sk_buff *msg,net/devlink/port.c static int devlink_port_fn_caps_fill(struct devlink_port *devlink_port,
Quoted 10 lines
net/devlink/port.c +10
-0+struct netlink_ext_ack *extack, + bool *msg_updated) +{ + u32 max_io_eqs; +int err; + +if (!port->ops->port_fn_max_io_eqs_get) +return 0 ; + + err = port->ops->port_fn_max_io_eqs_get(port, &max_io_eqs,extack);
net/devlink/port.c static int devlink_port_fn_caps_fill(struct devlink_port *devlink_port, · line 200
Quoted 3 lines
net/devlink/port.c +3
-0+if (err) { +if (err == -EOPNOTSUPP ) +return 0 ;Docs above says:
* Note: Driver should return-EOPNOTSUPPif it doesn't support
* port function (@port_fn_*) handling for a particular port.Quoted 5 lines
Quoted 4 lines
But here you're returning 0 in both cases of no
port_fn_max_io_eqs_getorport_fn_max_io_eqs_get()returns EOPNOTSUPP.When the port does not support this op, the function pointer is null and, 0
is returned as expected.
Quoted 2 lines
When the port for some reason has the ops function pointer set for a port,
but if the port does not support the ops, it will return ENOPNOTSUPP.
Quoted 1 line
This may be possible when the driver has chosen to use same ops callback
structure for multiple port flavors.
Quoted 2 lines
This code pattern is likely left over code of relatively recent work that
moved ops from devlink instance to per port ops.
Quoted 2 lines
I propose to keep the current check as done in this patch, and run a full audit of all the drivers, if all drivers have moved to per port ops, then
simplify the code to drop the check for EOPNOTSUPP in a new series that may touch more drivers.
Quoted 1 line
Otherwise, we may end up failing the port show operation when it returns
- ENOPNOTSUPP.
Thanks for the explanation. So ideally each port flavour has its own unique set of struct ops, and if something is not supported then don't set the func ptr in the struct ops.
Yes, I see that 0 has to be returned for
devlink_port_fn_caps_fill()to succeed.
Quoted 7 lines
Quoted 3 lines
Had a brief look and there's only a handful of drivers (mlx, nfp, ice) that use
devlink_port_ops.And netdevsim too. :)
Can we please the cleanup in a separate series?
Post this may be later in the month, I may have cycles to audit and simplify. Would it be ok with you?
Yeah of course, thanks. I'm making some changes in netdevsim and can do a drive-by cleanup there too.
Quoted 80 lines
Quoted 80 lines
Quoted 80 lines
Quoted 80 lines
net/devlink/port.c static int devlink_port_fn_caps_fill(struct devlink_port *devlink_port,
Quoted 5 lines
net/devlink/port.c +3
-0+return err; + } + err = nla_put_u32(msg, DEVLINK_PORT_FN_ATTR_MAX_IO_EQS,
max_io_eqs);net/devlink/port.c static int devlink_port_fn_caps_fill(struct devlink_port *devlink_port,
Quoted 10 lines
net/devlink/port.c +7
-0+if (err) +return err; + *msg_updated = true; +return 0 ; +} +int devlink_nl_port_handle_fill (struct sk_buff *msg,struct devlink_port *devlink_port) { if (devlink_nl_put_handle(msg, devlink_port->devlink)) @@ -409,6 +434,18 @@ static int devlink_port_fn_caps_set(struct devlink_port
*devlink_port,Quoted 6 lines
net/devlink/port.c +3
-0return 0 ; } +static int +devlink_port_fn_max_io_eqs_set(struct devlink_port *devlink_port, + const struct nlattr *attr,net/devlink/port.c static int devlink_port_fn_caps_set(struct devlink_port *devlink_port,
Quoted 10 lines
net/devlink/port.c +8
-0+ struct netlink_ext_ack *extack) { +u32 max_io_eqs; + + max_io_eqs =nla_get_u32 (attr); +return devlink_port->ops->port_fn_max_io_eqs_set (devlink_port, + max_io_eqs, extack); +} + static int devlink_nl_port_function_attrs_put(struct sk_buff *msg, struct
devlink_port*port,Quoted 1 line
struct
netlink_ext_ack*extack) @@ -428,6+465,9 @@
Quoted 1 line
devlink_nl_port_function_attrs_put(struct
sk_buff*msg, struct
devlink_port*pornet/devlink/port.c devlink_nl_port_function_attrs_put(struct sk_buff *msg, struct devlink_port *por
Quoted 12 lines
net/devlink/port.c +6
-0if (err) goto out; err =devlink_port_fn_state_fill (port, msg, extack, &msg_updated); +if (err) + goto out; + err =devlink_port_fn_max_io_eqs_fill (port, msg, extack, +&msg_updated);if (err) goto out; err = devlink_rel_devlink_handle_put(msg, port->devlink, @@ -726,6 +766,12 @@ static int devlink_port_function_validate(struct +devlink_port
*devlink_port,net/devlink/port.c static int devlink_port_function_validate(struct devlink_port *devlink_port,
Quoted 6 lines
net/devlink/port.c +3
-0} } } +if (tb[DEVLINK_PORT_FN_ATTR_MAX_IO_EQS ] && + !ops->port_fn_max_io_eqs_set) { + NL_SET_ERR_MSG_ATTR(extack,tb[DEVLINK_PORT_FN_ATTR_MAX_IO_EQS],
net/devlink/port.c
Quoted 1 line
net/devlink/port.c +1
-0+ "Function does not support max_io_eqssetting");
net/devlink/port.c static int devlink_port_function_set(struct
Quoted 6 lines
net/devlink/port.c +2
-0+return -EOPNOTSUPP ; + }return 0 ; }@@ -761,6 +807,13 @@ static int devlink_port_function_set(struct
devlink_port*port,Quoted 13 lines
net/devlink/port.c +7
-0return err; } + attr = tb[DEVLINK_PORT_FN_ATTR_MAX_IO_EQS ]; +if (attr) { + err =devlink_port_fn_max_io_eqs_set (port, attr, extack); +if (err) +return err; + } +/* Keep this as the last function attribute set, so that when * multiple port function attributes are set along with state, * Those can be applied first before activating the state.
RE: [net-next v3 1/2] devlink: Support setting max_io_eqs
Quoted 258 lines
From: David Wei <[email protected]>
Sent: Friday, April 5, 2024 12:23 AMQuoted 113 lines · Parav Pandit
Quoted 113 lines
From: David Wei <[email protected]>
Sent: Friday, April 5, 2024 12:10 AMQuoted 12 lines · Parav Pandit
Hi David,
Quoted 10 lines
From: David Wei <[email protected]>
Sent: Thursday, April 4, 2024 10:29 PMQuoted 6 lines · Parav Pandit
Many devices send event notifications for the IO queues, such as tx and rx queues, through event queues.
Enable a privileged owner, such as a hypervisor PF, to set the number of IO event queues for the VF and SF during the provisioning
stage.
Quoted 42 lines
Quoted 42 lines
Quoted 7 lines
example:
Get maximum IO event queues of the VF device::$ devlink port show pci/0000:06:00.0/2 pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0vfnum 1
Quoted 5 lines
function: hw_addr 00:00:00:00:00:00 ipsec_packet disabled
max_io_eqs
10Quoted 7 lines
Set maximum IO event queues of the VF device::
$ devlink port function set pci/0000:06:00.0/2 max_io_eqs 32$ devlink port show pci/0000:06:00.0/2 pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0vfnum 1
Quoted 21 lines
function: hw_addr 00:00:00:00:00:00 ipsec_packet disabled
max_io_eqs
32Reviewed-by Jiri Pirko <[email protected]>Reviewed-by Shay Drory <[email protected]>Signed-off-by Parav Pandit <[email protected]>---
changelog:v2->v3:
- limited 80 chars per linev1->v2:
- limited comment to 80 chars per line in header file--- .../networking/devlink/devlink-port.rst | 25 +++++++++ include/net/devlink.h | 14 +++++ include/uapi/linux/devlink.h | 1 + net/devlink/port.c | 53 +++++++++++++++++++ 4 files changed, 93 insertions(+)Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Documentation/networking/devlink/devlink-port.rst +0
-0@@ -134,6 +134,9 @@ Users may also set the IPsec crypto capabilityof the function using Users may also set the IPsec packet capability of the function using
devlink port function set ipsec_packetcommand.
Quoted 42 lines
Quoted 42 lines
Quoted 12 lines
Documentation/networking/devlink/devlink-port.rst +4
-0+Users may also set the maximum IO event queues of the function +using `devlink port function set max_io_eqs` command. + Function attributes ===================@@ -295,6 +298,28 @@ policy is processed in software by the kernel. function: hw_addr 00:00:00:00:00:00 ipsec_packet enabled +Maximum IO events queues setupDocumentation/networking/devlink/devlink-port.rst policy is processed in software by the kernel. · line 311
Quoted 10 lines
Documentation/networking/devlink/devlink-port.rst +10
-0+------------------------------ +When user sets maximum number of IO event queues for a SF or a +VF, such function driver is limited to consume only enforced +number of IO event queues. + +- Get maximum IO event queues of the VF device:: + + $ devlink port show pci/0000:06:00.0/2 + pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour + pcivf pfnum0 vfnum 1
Documentation/networking/devlink/devlink-port.rst policy is processed in software by the kernel.
Quoted 11 lines
Documentation/networking/devlink/devlink-port.rst +11
-0+ function: + hw_addr 00:00:00:00:00:00 ipsec_packet disabled + max_io_eqs 10 + +- Set maximum IO event queues of the VF device:: + + $ devlink port function set pci/0000:06:00.0/2 max_io_eqs 32 + + $ devlink port show pci/0000:06:00.0/2 + pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour + pcivf pfnum0 vfnum 1
Quoted 7 lines
Documentation/networking/devlink/devlink-port.rst +4
-0+ function: + hw_addr 00:00:00:00:00:00 ipsec_packet disabled + max_io_eqs 32 + Subfunction ============Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
include/net/devlink.h void devlink_free(struct devlink *devlink); · line 1602
Quoted 6 lines
include/net/devlink.h index renamed from include/net/devlink.h +0
-0
9ac394bdfbe4..bb1af599d101100644include/net/devlink.h +0
-0@@ -1602,6 +1602,14 @@ void devlink_free(struct devlink *devlink);* capability. Should be used by device drivers to
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
include/net/devlink.h void devlink_free(struct devlink *devlink); · line 1603
Quoted 1 line
include/net/devlink.h +0
-0* enable/disable ipsec_packet capability of a
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
include/net/devlink.h void devlink_free(struct devlink *devlink); · line 1605
Quoted 2 lines
include/net/devlink.h +1
-0* function managed by the devlink port. +* @port_fn_max_io_eqs_get: Callback used to get port function's maximum number
Quoted 1 line
include/net/devlink.h +1
-0+* of event queues. Should be used by device drivers
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
to
Quoted 1 line
include/net/devlink.h +1
-0+* report the maximum event queues of a function
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
include/net/devlink.h void devlink_free(struct devlink *devlink); · line 1609
Quoted 2 lines
include/net/devlink.h +2
-0+* managed by the devlink port. +* @port_fn_max_io_eqs_set: Callback used to set port function's maximum number
Quoted 1 line
include/net/devlink.h +1
-0+* of event queues. Should be used by device drivers
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
to
Quoted 1 line
include/net/devlink.h +1
-0+* configure maximum number of event queues
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
include/net/devlink.h struct devlink_port_ops { · line 1662
Quoted 9 lines
include/net/devlink.h +2
-0+* of a function managed by the devlink port. * * Note: Driver should return -EOPNOTSUPP if it doesn't support * port function (@port_fn_*) handling for a particular port. @@ -1651,6 +1659,12 @@ struct devlink_port_ops {int (*port_fn_ipsec_packet_set)(struct devlink_port *devlink_port,bool enable,struct netlink_ext_ack *extack); + int (*port_fn_max_io_eqs_get)(struct devlink_port
*devlink_port,Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
include/net/devlink.h struct devlink_port_ops {
Quoted 3 lines
include/net/devlink.h +3
-0+u32 *max_eqs, + struct netlink_ext_ack *extack); + int (*port_fn_max_io_eqs_set)(struct devlink_port
*devlink_port,Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 11 lines
include/net/devlink.h +2
-0+u32 max_eqs, + struct netlink_ext_ack *extack); }; void devlink_port_init(struct devlink *devlink, diff --git
a/include/uapi/linux/devlink.hb/include/uapi/linux/devlink.hindex2da0c7eb6710..9401aa343673100644include/uapi/linux/devlink.h enum devlink_port_function_attr { · line 692
Quoted 7 lines
include/uapi/linux/devlink.h +1
-0@@ -686,6 +686,7 @@ enum devlink_port_function_attr {DEVLINK_PORT_FN_ATTR_OPSTATE ,/* u8 */ DEVLINK_PORT_FN_ATTR_CAPS ,/* bitfield32 */ DEVLINK_PORT_FN_ATTR_DEVLINK ,/* nested */ +DEVLINK_PORT_FN_ATTR_MAX_IO_EQS ,/* u32 */ __DEVLINK_PORT_FUNCTION_ATTR_MAX, DEVLINK_PORT_FUNCTION_ATTR_MAX =
__DEVLINK_PORT_FUNCTION_ATTR_MAX-net/devlink/port.c static const struct nla_policy
Quoted 5 lines
1 diff --git
a/net/devlink/port.cb/net/devlink/port.cindex118d130d2afd..be9158b4453c100644net/devlink/port.c +0
-0@@ -16,6 +16,7 @@ static const struct nla_policydevlink_function_nl_policy[DEVLINK_PORT_FUNCTION_
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
net/devlink/port.c static const struct nla_policy devlink_function_nl_policy[DEVLINK_PORT_FUNCTION_ · line 21
Quoted 3 lines
net/devlink/port.c +0
-0DEVLINK_PORT_FN_STATE_ACTIVE), [DEVLINK_PORT_FN_ATTR_CAPS] =
NLA_POLICY_BITFIELD32(DEVLINK_PORT_FN_CAPS_VALID_MASK),net/devlink/port.c static const struct nla_policy · line 19
Quoted 1 line
net/devlink/port.c +1
-0+ [DEVLINK_PORT_FN_ATTR_MAX_IO_EQS] = { .type =
NLA_U32},Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
net/devlink/port.c static const struct nla_policy · line 22
Quoted 3 lines
net/devlink/port.c +0
-0};#define ASSERT_DEVLINK_PORT_REGISTERED (devlink_port)\Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
net/devlink/port.c static int devlink_port_fn_caps_fill(struct
Quoted 1 line
net/devlink/port.c +0
-0@@ -182,6 +183,30 @@ static int devlink_port_fn_caps_fill(struct
devlink_port*devlink_port,net/devlink/port.c static int devlink_port_fn_caps_fill(struct devlink_port *devlink_port,
Quoted 6 lines
net/devlink/port.c +3
-0return 0 ; } +static int devlink_port_fn_max_io_eqs_fill (struct devlink_port *port, +struct sk_buff *msg, + struct netlink_ext_ack*extack,
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
net/devlink/port.c static int devlink_port_fn_caps_fill(struct devlink_port *devlink_port,
Quoted 9 lines
net/devlink/port.c +9
-0+ bool *msg_updated) +{ +u32 max_io_eqs; +int err; + +if (!port->ops->port_fn_max_io_eqs_get) +return 0 ; + + err = port->ops->port_fn_max_io_eqs_get(port,
&max_io_eqs,Quoted 34 lines
Quoted 34 lines
Quoted 9 lines
Quoted 9 lines
extack);
net/devlink/port.c static int devlink_port_fn_caps_fill(struct devlink_port *devlink_port, · line 200
Quoted 3 lines
net/devlink/port.c +3
-0+if (err) { +if (err == -EOPNOTSUPP ) +return 0 ;Docs above says:
* Note: Driver should return-EOPNOTSUPPif it doesn't support
* port function (@port_fn_*) handling for a particular port.Quoted 6 lines
Quoted 4 lines
But here you're returning 0 in both cases of no
port_fn_max_io_eqs_getorport_fn_max_io_eqs_get()returns EOPNOTSUPP.When the port does not support this op, the function pointer is null and, 0
is returned as expected.
Quoted 3 lines
When the port for some reason has the ops function pointer set for a port,
but if the port does not support the ops, it will return ENOPNOTSUPP.
Quoted 2 lines
This may be possible when the driver has chosen to use same ops callback
structure for multiple port flavors.
Quoted 3 lines
This code pattern is likely left over code of relatively recent work that
moved ops from devlink instance to per port ops.
Quoted 3 lines
I propose to keep the current check as done in this patch, and run a full audit of all the drivers, if all drivers have moved to per port ops, then
simplify the code to drop the check for EOPNOTSUPP in a new series that may touch more drivers.
Quoted 2 lines
Otherwise, we may end up failing the port show operation when it returns
Quoted 7 lines
Quoted 7 lines
- ENOPNOTSUPP.
Thanks for the explanation. So ideally each port flavour has its own unique set of struct ops, and if something is not supported then don't set the func ptr in the struct ops.
Yes, I see that 0 has to be returned for
devlink_port_fn_caps_fill()tosucceed.
Quoted 4 lines
Quoted 4 lines
Had a brief look and there's only a handful of drivers (mlx, nfp, ice) that use
devlink_port_ops.
Quoted 8 lines
Quoted 4 lines
And netdevsim too. :)
Can we please the cleanup in a separate series?
Post this may be later in the month, I may have cycles to audit and simplify. Would it be ok with you?Yeah of course, thanks. I'm making some changes in netdevsim and can do a drive-by cleanup there too.
Ok. yes, that will be helpful. Thanks.
Quoted 87 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
net/devlink/port.c static int devlink_port_fn_caps_fill(struct devlink_port *devlink_port,
Quoted 5 lines
net/devlink/port.c +3
-0+return err; + } + err = nla_put_u32(msg,
DEVLINK_PORT_FN_ATTR_MAX_IO_EQS,Quoted 24 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
max_io_eqs);net/devlink/port.c static int devlink_port_fn_caps_fill(struct devlink_port *devlink_port,
Quoted 10 lines
net/devlink/port.c +7
-0+if (err) +return err; + *msg_updated = true; +return 0 ; +} +int devlink_nl_port_handle_fill (struct sk_buff *msg,struct devlink_port *devlink_port) { if (devlink_nl_put_handle(msg, devlink_port->devlink)) @@ -409,6 +434,18 @@ static int devlink_port_fn_caps_set(struct devlink_port
*devlink_port,Quoted 3 lines
return 0;
}Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
net/devlink/port.c static int devlink_port_fn_caps_set(struct devlink_port *devlink_port,
Quoted 8 lines
net/devlink/port.c +8
-0+static int +devlink_port_fn_max_io_eqs_set (struct devlink_port *devlink_port, +const struct nlattr *attr, +struct netlink_ext_ack *extack) { +u32 max_io_eqs; + + max_io_eqs =nla_get_u32 (attr); + return devlink_port->ops-port_fn_max_io_eqs_set(devlink_port,
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
net/devlink/port.c
Quoted 1 line
net/devlink/port.c +1
-0+ max_io_eqs,extack);
Quoted 24 lines
Quoted 24 lines
Quoted 24 lines
Quoted 24 lines
net/devlink/port.c static int devlink_port_fn_caps_set(struct devlink_port *devlink_port,
Quoted 4 lines
net/devlink/port.c +2
-0+} + static int devlink_nl_port_function_attrs_put(struct sk_buff *msg, struct
devlink_port*port,Quoted 1 line
struct
netlink_ext_ack*extack) @@ -428,6+465,9 @@
Quoted 1 line
devlink_nl_port_function_attrs_put(struct
sk_buff*msg, struct
devlink_port*porQuoted 4 lines
net/devlink/port.c +0
-0if (err) goto out; err = devlink_port_fn_state_fill(port, msg, extack,
&msg_updated);net/devlink/port.c
Quoted 10 lines
net/devlink/port.c +6
-1+if (err) + goto out; + err =devlink_port_fn_max_io_eqs_fill (port, msg, extack, +&msg_updated);if (err) goto out; err = devlink_rel_devlink_handle_put(msg, port->devlink, @@ -726,6 +766,12 @@ static int devlink_port_function_validate(struct +devlink_port
*devlink_port,Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
net/devlink/port.c static int devlink_port_function_validate(struct devlink_port *devlink_port,
Quoted 6 lines
net/devlink/port.c +3
-0} } } +if (tb[DEVLINK_PORT_FN_ATTR_MAX_IO_EQS ] && + !ops->port_fn_max_io_eqs_set) { + NL_SET_ERR_MSG_ATTR(extack,tb[DEVLINK_PORT_FN_ATTR_MAX_IO_EQS],
net/devlink/port.c
Quoted 1 line
net/devlink/port.c +1
-0+ "Function does not support
max_io_eqsQuoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
setting");
Quoted 5 lines
net/devlink/port.c +2
-0+return -EOPNOTSUPP ; + }return 0 ; }Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
net/devlink/port.c static int devlink_port_function_set(struct
Quoted 1 line
net/devlink/port.c +0
-0@@ -761,6 +807,13 @@ static int devlink_port_function_set(struct
devlink_port*port,net/devlink/port.c static int devlink_port_function_set(struct devlink_port *port,
Quoted 6 lines
net/devlink/port.c +3
-0return err; } + attr = tb[DEVLINK_PORT_FN_ATTR_MAX_IO_EQS ]; +if (attr) { + err = devlink_port_fn_max_io_eqs_set(port, attr,extack);
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
net/devlink/port.c +4
-0+if (err) +return err; + } +/* Keep this as the last function attribute set, so that when * multiple port function attributes are set along with state, * Those can be applied first before activating the state.
Jakub Kicinski
maintainerRe: [net-next v3 1/2] devlink: Support setting max_io_eqs
Quoted 4 lines · On Thu, 4 Apr 2024 18:55:09 +0000 Parav Pandit wrote:
Quoted 3 lines
Yeah of course, thanks. I'm making some changes in netdevsim and can do a drive-by cleanup there too.
Ok. yes, that will be helpful. Thanks.
I think that part is fine, TBH.
Drivers may not support particular control on particular HW/FW for any number of reasons, we should let them return EOPNOTSUPP. Not sure if there's much benefit for splitting the ops..
Jakub Kicinski
maintainerRe: [net-next v3 1/2] devlink: Support setting max_io_eqs
In reply to Parav PanditQuoted 5 lines · On Wed, 3 Apr 2024 20:41:32 +0300 Parav Pandit wrote:
Many devices send event notifications for the IO queues, such as tx and rx queues, through event queues.
Enable a privileged owner, such as a hypervisor PF, to set the number of IO event queues for the VF and SF during the provisioning stage.
What's the relationship between EQ and queue pairs and IRQs? The next patch says "maximum IO event queues which are typically used to derive the maximum and default number of net device channels" It may not be obvious to non-mlx5 experts, I think it needs to be better documented.
RE: [net-next v3 1/2] devlink: Support setting max_io_eqs
Hi Jakub,
Quoted 11 lines
From: Jakub Kicinski <[email protected]>
Sent: Friday, April 5, 2024 7:51 AMQuoted 5 lines · On Wed, 3 Apr 2024 20:41:32 +0300 Parav Pandit wrote:
Many devices send event notifications for the IO queues, such as tx and rx queues, through event queues.
Enable a privileged owner, such as a hypervisor PF, to set the number of IO event queues for the VF and SF during the provisioning stage.
What's the relationship between EQ and queue pairs and IRQs?
Netdev qps (txq, rxq pair) channels created are typically upto num cpus by driver, provided it has enough IO event queues upto cpu count.
Rdma qps are far more than netdev qps as multiple process uses them and they are per user space process resource. Those applications use number of QPs based on number of cpus and number of event channels to deliver notifications to user space.
Driver uses the IRQs dynamically upto the PCI's limit based on supported IO event queues.
Quoted 4 lines
The next patch says "maximum IO event queues which are typically used to derive the maximum and default number of net device channels" It may not be obvious to non-mlx5 experts, I think it needs to be better documented.
I will expand the documentation in .../networking/devlink/devlink-port.rst.
I will add below change to the v4 that has David's comments also addressed. Is this ok for you?
Documentation/networking/devlink/devlink-port.rst +5 -0
@@ -304,6 +304,11 @@ When user sets maximum number of IO event queues for a SF or a VF, such function driver is limited to consume only enforced number of IO event queues. +IO event queues deliver events related to IO queues, including network +device transmit and receive queues (txq and rxq) and RDMA Queue Pairs (QPs). +For example, the number of netdevice channels and RDMA device completion +vectors are derived from the function's IO event queues. +
Jakub Kicinski
maintainerRe: [net-next v3 1/2] devlink: Support setting max_io_eqs
Quoted 4 lines · On Fri, 5 Apr 2024 03:13:36 +0000 Parav Pandit wrote:
Netdev qps (txq, rxq pair) channels created are typically upto num cpus by driver, provided it has enough IO event queues upto cpu count.
Rdma qps are far more than netdev qps as multiple process uses them and they are per user space process resource. Those applications use number of QPs based on number of cpus and number of event channels to deliver notifications to user space.
Some other drivers (e.g. intel) support multiple queues per core in netdev. For mlx5 I think AF_XDP may be a good example (or used to be) where there may be more than one queue?
So I think the question still stands even for netdev.
We should document whether number of EQs contains the number of Rx/Tx queues.
Quoted 1 line
Driver uses the IRQs dynamically upto the PCI's limit based on supported IO event queues.
Right but one IRQ <> one EQ? Typically / always? SFs "share" the IRQs with PF IIRC, do they share EQs?
Quoted 8 lines
Quoted 4 lines
The next patch says "maximum IO event queues which are typically used to derive the maximum and default number of net device channels" It may not be obvious to non-mlx5 experts, I think it needs to be better documented.
I will expand the documentation in
.../networking/devlink/devlink-port.rst.I will add below change to the v4 that has David's comments also addressed. Is this ok for you?
Looks like a good start but I think a few more sentences describing the relation to other resources would be good.
Quoted 10 lines
Documentation/networking/devlink/devlink-port.rst +4
-0@@ -304,6 +304,11 @@ When user sets maximum number of IO event queues for a SF or a VF, such function driver is limited to consume only enforced number of IO event queues. +IO event queues deliver events related to IO queues, including network +device transmit and receive queues (txq and rxq) and RDMA Queue Pairs (QPs). +For example, the number of netdevice channels and RDMA device completion +vectors are derived from the function's IO event queues.
RE: [net-next v3 1/2] devlink: Support setting max_io_eqs
Quoted 9 lines
From: Jakub Kicinski <[email protected]>
Sent: Friday, April 5, 2024 7:44 PMQuoted 1 line · On Fri, 5 Apr 2024 03:13:36 +0000 Parav Pandit wrote:
Netdev qps (txq, rxq pair) channels created are typically upto num cpus by
driver, provided it has enough IO event queues upto cpu count.
Quoted 1 line
Quoted 9 lines
Quoted 1 line
Rdma qps are far more than netdev qps as multiple process uses them and
they are per user space process resource.
Quoted 1 line
Those applications use number of QPs based on number of cpus and
number of event channels to deliver notifications to user space.
Some other drivers (e.g. intel) support multiple queues per core in netdev. For mlx5 I think
AF_XDPmay be a good example (or used to be) where there may be more than one queue?
Yes, there may be multiple netdev queues which can be connected to a eq.
For example, as you described, mlx5 xdp, also mlx5 creates a multiple txq per traffic class per channel which are linked to a single eq of the channel. But still those txq are per channel AFAIK.
Quoted 4 lines
So I think the question still stands even for netdev.
We should document whether number of EQs contains the number of Rx/Tx queues.
I believe number of txq, rxqs can be more than the number of EQs connecting to same EQ. Netdev channels have more accurate linkage to EQs, compared to raw txq/rxqs.
Quoted 4 lines
Quoted 1 line
Driver uses the IRQs dynamically upto the PCI's limit based on supported IO
event queues.
Right but one IRQ <> one EQ? Typically / always?
Typically yes, one IRQ <> one EQ.
Quoted 2 lines
SFs "share" the IRQs with PF IIRC, do they share EQs?
SFs do not share EQs. Yes, SFs have their own dedicated EQs. You remember right, that they share IRQs.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
The next patch says "maximum IO event queues which are typically used to derive the maximum and default number of net device channels" It may not be obvious to non-mlx5 experts, I think it needs to be better documented.
Quoted 9 lines
Quoted 3 lines
I will expand the documentation in
.../networking/devlink/devlink-port.rst.I will add below change to the v4 that has David's comments also
addressed.
Quoted 1 line
Is this ok for you?
Looks like a good start but I think a few more sentences describing the relation to other resources would be good.
I think EQs limited object that does not have more wider relation in the stack. Relation to IRQ is probably a good addition to do.
Along with below changes, will add the reference to IRQ too in v4.
Quoted 12 lines
Documentation/networking/devlink/devlink-port.rst When user sets maximum number of IO event
Quoted 3 lines
Documentation/networking/devlink/devlink-port.rst +0
-0@@ -304,6 +304,11 @@ When user sets maximum number of IO eventqueues
Documentation/networking/devlink/devlink-port.rst
Quoted 5 lines
for a SF or a VF, such function driver is limited to consume only enforced number of IO event queues.
Documentation/networking/devlink/devlink-port.rst +2
-0+IO event queues deliver events related to IO queues, including +network device transmit and receive queues (txq and rxq) and RDMAQueue Pairs (QPs).
Quoted 2 lines
Documentation/networking/devlink/devlink-port.rst +2
-0+For example, the number of netdevice channels and RDMA device +completion vectors are derived from the function's IO event queues.
PATCH v3 2/2 mlx5/core: Support max_io_eqs for a function
In reply to Parav PanditImplement get and set for the maximum IO event queues for SF and VF. This enables administrator on the hypervisor to control the maximum IO event queues which are typically used to derive the maximum and default number of net device channels or rdma device completion vectors.
---
changelog:v2->v3:
- limited to 80 chars per line in devlink
- fixed comments from Jakub in mlx5 driver to fix missing mutex unlock on error pathv1->v2:
- fixed comments from Kalesh
- fixed missing kfree in get call
- returning error code for get cmd failure
- fixed error msg copy paste error in set on cmd failure
- limited code to 80 chars limit
- fixed set function variables for reverse christmas tree
--- .../mellanox/mlx5/core/esw/devlink_port.c | 4 + .../net/ethernet/mellanox/mlx5/core/eswitch.h | 7 ++ .../mellanox/mlx5/core/eswitch_offloads.c | 97 +++++++++++++++++++ 3 files changed, 108 insertions(+)
drivers/net/ethernet/mellanox/mlx5/core/esw/devlink_port.c +4 -0
@@ -98,6 +98,8 @@ static const struct devlink_port_ops mlx5_esw_pf_vf_dl_port_ops = { .port_fn_ipsec_packet_get = mlx5_devlink_port_fn_ipsec_packet_get, .port_fn_ipsec_packet_set = mlx5_devlink_port_fn_ipsec_packet_set,#endif /* CONFIG_XFRM_OFFLOAD */ + .port_fn_max_io_eqs_get = mlx5_devlink_port_fn_max_io_eqs_get, + .port_fn_max_io_eqs_set = mlx5_devlink_port_fn_max_io_eqs_set, }; static void mlx5_esw_offloads_sf_devlink_port_attrs_set(struct mlx5_eswitch *esw,
@@ -143,6 +145,8 @@ static const struct devlink_port_ops mlx5_esw_dl_sf_port_ops = { .port_fn_state_get = mlx5_devlink_sf_port_fn_state_get, .port_fn_state_set = mlx5_devlink_sf_port_fn_state_set, #endif + .port_fn_max_io_eqs_get = mlx5_devlink_port_fn_max_io_eqs_get, + .port_fn_max_io_eqs_set = mlx5_devlink_port_fn_max_io_eqs_set, };int mlx5_esw_offloads_devlink_port_register (struct mlx5_eswitch *esw,struct mlx5_vport *vport)
drivers/net/ethernet/mellanox/mlx5/core/eswitch.h +7 -0
@@ -573,6 +573,13 @@ int mlx5_devlink_port_fn_ipsec_packet_get(struct devlink_port *port, bool *is_enint mlx5_devlink_port_fn_ipsec_packet_set (struct devlink_port *port,bool enable,struct netlink_ext_ack *extack);#endif /* CONFIG_XFRM_OFFLOAD */ +int mlx5_devlink_port_fn_max_io_eqs_get (struct devlink_port *port, +u32 *max_io_eqs, +struct netlink_ext_ack *extack); +int mlx5_devlink_port_fn_max_io_eqs_set (struct devlink_port *port, +u32 max_io_eqs, +struct netlink_ext_ack *extack); +void *mlx5_eswitch_get_uplink_priv (struct mlx5_eswitch *esw,u8 rep_type); int __mlx5_eswitch_set_vport_vlan(struct mlx5_eswitch *esw,
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +97 -0
106 lines not on this page · raw message
Signature · 1 line
2.26.2
Re: [net-next v3 2/2] mlx5/core: Support max_io_eqs for a function
Quoted 8 lines · Parav Pandit
Implement get and set for the maximum IO event queues for SF and VF. This enables administrator on the hypervisor to control the maximum IO event queues which are typically used to derive the maximum and default number of net device channels or rdma device completion vectors.
Reviewed-by Shay Drory <[email protected]>Signed-off-by Parav Pandit <[email protected]>
Thanks Parav for addressing the comments. Changes look good to me.
Quoted 174 lines
---
changelog:v2->v3:
- limited to 80 chars per line in devlink
- fixed comments from Jakub in mlx5 driver to fix missing mutex unlock on error pathv1->v2:
- fixed comments from Kalesh
- fixed missing kfree in get call
- returning error code for get cmd failure
- fixed error msg copy paste error in set on cmd failure
- limited code to 80 chars limit
- fixed set function variables for reverse christmas tree--- .../mellanox/mlx5/core/esw/devlink_port.c | 4 + .../net/ethernet/mellanox/mlx5/core/eswitch.h | 7 ++ .../mellanox/mlx5/core/eswitch_offloads.c | 97 +++++++++++++++++++ 3 files changed, 108 insertions(+)drivers/net/ethernet/mellanox/mlx5/core/esw/devlink_port.c +4
-0@@ -98,6 +98,8 @@ static const struct devlink_port_ops mlx5_esw_pf_vf_dl_port_ops = { .port_fn_ipsec_packet_get = mlx5_devlink_port_fn_ipsec_packet_get, .port_fn_ipsec_packet_set = mlx5_devlink_port_fn_ipsec_packet_set,#endif /* CONFIG_XFRM_OFFLOAD */ + .port_fn_max_io_eqs_get = mlx5_devlink_port_fn_max_io_eqs_get, + .port_fn_max_io_eqs_set = mlx5_devlink_port_fn_max_io_eqs_set, }; static void mlx5_esw_offloads_sf_devlink_port_attrs_set(struct mlx5_eswitch *esw,@@ -143,6 +145,8 @@ static const struct devlink_port_ops mlx5_esw_dl_sf_port_ops = { .port_fn_state_get = mlx5_devlink_sf_port_fn_state_get, .port_fn_state_set = mlx5_devlink_sf_port_fn_state_set, #endif + .port_fn_max_io_eqs_get = mlx5_devlink_port_fn_max_io_eqs_get, + .port_fn_max_io_eqs_set = mlx5_devlink_port_fn_max_io_eqs_set, };int mlx5_esw_offloads_devlink_port_register (struct mlx5_eswitch *esw,struct mlx5_vport *vport)drivers/net/ethernet/mellanox/mlx5/core/eswitch.h +7
-0@@ -573,6 +573,13 @@ int mlx5_devlink_port_fn_ipsec_packet_get(struct devlink_port *port, bool *is_enint mlx5_devlink_port_fn_ipsec_packet_set (struct devlink_port *port,bool enable,struct netlink_ext_ack *extack);#endif /* CONFIG_XFRM_OFFLOAD */ +int mlx5_devlink_port_fn_max_io_eqs_get (struct devlink_port *port, +u32 *max_io_eqs, +struct netlink_ext_ack *extack); +int mlx5_devlink_port_fn_max_io_eqs_set (struct devlink_port *port, +u32 max_io_eqs, +struct netlink_ext_ack *extack); +void *mlx5_eswitch_get_uplink_priv (struct mlx5_eswitch *esw,u8 rep_type); int __mlx5_eswitch_set_vport_vlan(struct mlx5_eswitch *esw,drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +97
-1@@ -66,6 +66,8 @@#define MLX5_ESW_FT_OFFLOADS_DROP_RULE (1) +#define MLX5_ESW_MAX_CTRL_EQS 4 +static struct esw_vport_tbl_namespace mlx5_esw_vport_tbl_mirror_ns = { .max_fte =MLX5_ESW_VPORT_TBL_SIZE , .max_num_groups =MLX5_ESW_VPORT_TBL_NUM_GROUPS ,@@ -4557,3 +4559,98 @@ int mlx5_devlink_port_fn_ipsec_packet_set(struct devlink_port *port,return err; }#endif /* CONFIG_XFRM_OFFLOAD */ + +int +mlx5_devlink_port_fn_max_io_eqs_get (struct devlink_port *port,u32 *max_io_eqs, +struct netlink_ext_ack *extack) +{ +struct mlx5_vport *vport =mlx5_devlink_port_vport_get (port); +int query_out_sz =MLX5_ST_SZ_BYTES (query_hca_cap_out); +u16 vport_num = vport->vport; +struct mlx5_eswitch *esw; +void *query_ctx; +void *hca_caps; +u32 max_eqs; +int err; + + esw =mlx5_devlink_eswitch_nocheck_get (port->devlink); +if (!MLX5_CAP_GEN (esw->dev, vhca_resource_manager)) { +NL_SET_ERR_MSG_MOD (extack, +"Device doesn't support VHCA management" ); +return -EOPNOTSUPP ; + } + + query_ctx =kzalloc (query_out_sz,GFP_KERNEL ); +if (!query_ctx) +return -ENOMEM ; + +mutex_lock (&esw->state_lock); + err =mlx5_vport_get_other_func_cap (esw->dev, vport_num, query_ctx, +MLX5_CAP_GENERAL ); +if (err) { +NL_SET_ERR_MSG_MOD (extack,"Failed getting HCA caps" ); + goto out; + } + + hca_caps =MLX5_ADDR_OF (query_hca_cap_out, query_ctx, capability); + max_eqs =MLX5_GET (cmd_hca_cap, hca_caps, max_num_eqs); +if (max_eqs <MLX5_ESW_MAX_CTRL_EQS ) + *max_io_eqs =0 ; +else + *max_io_eqs = max_eqs -MLX5_ESW_MAX_CTRL_EQS ; +out: +mutex_unlock (&esw->state_lock); +kfree (query_ctx); +return err; +} + +int +mlx5_devlink_port_fn_max_io_eqs_set (struct devlink_port *port,u32 max_io_eqs, +struct netlink_ext_ack *extack) +{ +struct mlx5_vport *vport =mlx5_devlink_port_vport_get (port); +int query_out_sz =MLX5_ST_SZ_BYTES (query_hca_cap_out); +u16 max_eqs = max_io_eqs +MLX5_ESW_MAX_CTRL_EQS ; +u16 vport_num = vport->vport; +struct mlx5_eswitch *esw; +void *query_ctx; +void *hca_caps; +int err; + + esw =mlx5_devlink_eswitch_nocheck_get (port->devlink); +if (!MLX5_CAP_GEN (esw->dev, vhca_resource_manager)) { +NL_SET_ERR_MSG_MOD (extack, +"Device doesn't support VHCA management" ); +return -EOPNOTSUPP ; + } + +if (max_io_eqs +MLX5_ESW_MAX_CTRL_EQS >USHRT_MAX ) { +NL_SET_ERR_MSG_MOD (extack,"Supplied value out of range" ); +return -EINVAL ; + } + + query_ctx =kzalloc (query_out_sz,GFP_KERNEL ); +if (!query_ctx) +return -ENOMEM ; + +mutex_lock (&esw->state_lock); + err =mlx5_vport_get_other_func_cap (esw->dev, vport_num, query_ctx, +MLX5_CAP_GENERAL ); +if (err) { +NL_SET_ERR_MSG_MOD (extack,"Failed getting HCA caps" ); + goto out; + } + + hca_caps =MLX5_ADDR_OF (query_hca_cap_out, query_ctx, capability); +MLX5_SET (cmd_hca_cap, hca_caps, max_num_eqs, max_eqs); + + err =mlx5_vport_set_other_func_cap (esw->dev, hca_caps, vport_num, +MLX5_SET_HCA_CAP_OP_MOD_GENERAL_DEVICE ); +if (err) +NL_SET_ERR_MSG_MOD (extack,"Failed setting HCA caps" ); + +out: +mutex_unlock (&esw->state_lock); +kfree (query_ctx); +return err; +} --2.26.2
Signature · 2 lines
Regards, Kalesh A P
Re: [net-next v3 2/2] mlx5/core: Support max_io_eqs for a function
In reply to Parav PanditQuoted 139 lines · Parav Pandit
Implement get and set for the maximum IO event queues for SF and VF. This enables administrator on the hypervisor to control the maximum IO event queues which are typically used to derive the maximum and default number of net device channels or rdma device completion vectors.
Reviewed-by Shay Drory <[email protected]>Signed-off-by Parav Pandit <[email protected]>---
changelog:v2->v3:
- limited to 80 chars per line in devlink
- fixed comments from Jakub in mlx5 driver to fix missing mutex unlock on error pathv1->v2:
- fixed comments from Kalesh
- fixed missing kfree in get call
- returning error code for get cmd failure
- fixed error msg copy paste error in set on cmd failure
- limited code to 80 chars limit
- fixed set function variables for reverse christmas tree--- .../mellanox/mlx5/core/esw/devlink_port.c | 4 + .../net/ethernet/mellanox/mlx5/core/eswitch.h | 7 ++ .../mellanox/mlx5/core/eswitch_offloads.c | 97 +++++++++++++++++++ 3 files changed, 108 insertions(+)drivers/net/ethernet/mellanox/mlx5/core/esw/devlink_port.c +4
-0@@ -98,6 +98,8 @@ static const struct devlink_port_ops mlx5_esw_pf_vf_dl_port_ops = { .port_fn_ipsec_packet_get = mlx5_devlink_port_fn_ipsec_packet_get, .port_fn_ipsec_packet_set = mlx5_devlink_port_fn_ipsec_packet_set,#endif /* CONFIG_XFRM_OFFLOAD */ + .port_fn_max_io_eqs_get = mlx5_devlink_port_fn_max_io_eqs_get, + .port_fn_max_io_eqs_set = mlx5_devlink_port_fn_max_io_eqs_set, }; static void mlx5_esw_offloads_sf_devlink_port_attrs_set(struct mlx5_eswitch *esw,@@ -143,6 +145,8 @@ static const struct devlink_port_ops mlx5_esw_dl_sf_port_ops = { .port_fn_state_get = mlx5_devlink_sf_port_fn_state_get, .port_fn_state_set = mlx5_devlink_sf_port_fn_state_set, #endif + .port_fn_max_io_eqs_get = mlx5_devlink_port_fn_max_io_eqs_get, + .port_fn_max_io_eqs_set = mlx5_devlink_port_fn_max_io_eqs_set, };int mlx5_esw_offloads_devlink_port_register (struct mlx5_eswitch *esw,struct mlx5_vport *vport)drivers/net/ethernet/mellanox/mlx5/core/eswitch.h +7
-0@@ -573,6 +573,13 @@ int mlx5_devlink_port_fn_ipsec_packet_get(struct devlink_port *port, bool *is_enint mlx5_devlink_port_fn_ipsec_packet_set (struct devlink_port *port,bool enable,struct netlink_ext_ack *extack);#endif /* CONFIG_XFRM_OFFLOAD */ +int mlx5_devlink_port_fn_max_io_eqs_get (struct devlink_port *port, +u32 *max_io_eqs, +struct netlink_ext_ack *extack); +int mlx5_devlink_port_fn_max_io_eqs_set (struct devlink_port *port, +u32 max_io_eqs, +struct netlink_ext_ack *extack); +void *mlx5_eswitch_get_uplink_priv (struct mlx5_eswitch *esw,u8 rep_type); int __mlx5_eswitch_set_vport_vlan(struct mlx5_eswitch *esw,drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +58
-0@@ -66,6 +66,8 @@#define MLX5_ESW_FT_OFFLOADS_DROP_RULE (1) +#define MLX5_ESW_MAX_CTRL_EQS 4 +static struct esw_vport_tbl_namespace mlx5_esw_vport_tbl_mirror_ns = { .max_fte =MLX5_ESW_VPORT_TBL_SIZE , .max_num_groups =MLX5_ESW_VPORT_TBL_NUM_GROUPS ,@@ -4557,3 +4559,98 @@ int mlx5_devlink_port_fn_ipsec_packet_set(struct devlink_port *port,return err; }#endif /* CONFIG_XFRM_OFFLOAD */ + +int +mlx5_devlink_port_fn_max_io_eqs_get (struct devlink_port *port,u32 *max_io_eqs, +struct netlink_ext_ack *extack) +{ +struct mlx5_vport *vport =mlx5_devlink_port_vport_get (port); +int query_out_sz =MLX5_ST_SZ_BYTES (query_hca_cap_out); +u16 vport_num = vport->vport; +struct mlx5_eswitch *esw; +void *query_ctx; +void *hca_caps; +u32 max_eqs; +int err; + + esw =mlx5_devlink_eswitch_nocheck_get (port->devlink); +if (!MLX5_CAP_GEN (esw->dev, vhca_resource_manager)) { +NL_SET_ERR_MSG_MOD (extack, +"Device doesn't support VHCA management" ); +return -EOPNOTSUPP ; + } + + query_ctx =kzalloc (query_out_sz,GFP_KERNEL ); +if (!query_ctx) +return -ENOMEM ; + +mutex_lock (&esw->state_lock); + err =mlx5_vport_get_other_func_cap (esw->dev, vport_num, query_ctx, +MLX5_CAP_GENERAL ); +if (err) { +NL_SET_ERR_MSG_MOD (extack,"Failed getting HCA caps" ); + goto out; + } + + hca_caps =MLX5_ADDR_OF (query_hca_cap_out, query_ctx, capability); + max_eqs =MLX5_GET (cmd_hca_cap, hca_caps, max_num_eqs); +if (max_eqs <MLX5_ESW_MAX_CTRL_EQS ) + *max_io_eqs =0 ; +else + *max_io_eqs = max_eqs -MLX5_ESW_MAX_CTRL_EQS ; +out: +mutex_unlock (&esw->state_lock); +kfree (query_ctx); +return err; +} + +int +mlx5_devlink_port_fn_max_io_eqs_set (struct devlink_port *port,u32 max_io_eqs, +struct netlink_ext_ack *extack) +{ +struct mlx5_vport *vport =mlx5_devlink_port_vport_get (port); +int query_out_sz =MLX5_ST_SZ_BYTES (query_hca_cap_out); +u16 max_eqs = max_io_eqs +MLX5_ESW_MAX_CTRL_EQS ; +u16 vport_num = vport->vport; +struct mlx5_eswitch *esw; +void *query_ctx; +void *hca_caps;
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c int mlx5_devlink_port_fn_ipsec_packet_set(struct devlink_port *port, · line 4627
Quoted 10 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +10
-0+int err; + + esw =mlx5_devlink_eswitch_nocheck_get (port->devlink); +if (!MLX5_CAP_GEN (esw->dev, vhca_resource_manager)) { +NL_SET_ERR_MSG_MOD (extack, +"Device doesn't support VHCA management" ); +return -EOPNOTSUPP ; + } + +if (max_io_eqs +MLX5_ESW_MAX_CTRL_EQS >USHRT_MAX ) {
nit: this is the same as max_eqs
Quoted 29 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +29
-0+NL_SET_ERR_MSG_MOD (extack,"Supplied value out of range" ); +return -EINVAL ; + } + + query_ctx =kzalloc (query_out_sz,GFP_KERNEL ); +if (!query_ctx) +return -ENOMEM ; + +mutex_lock (&esw->state_lock); + err =mlx5_vport_get_other_func_cap (esw->dev, vport_num, query_ctx, +MLX5_CAP_GENERAL ); +if (err) { +NL_SET_ERR_MSG_MOD (extack,"Failed getting HCA caps" ); + goto out; + } + + hca_caps =MLX5_ADDR_OF (query_hca_cap_out, query_ctx, capability); +MLX5_SET (cmd_hca_cap, hca_caps, max_num_eqs, max_eqs); + + err =mlx5_vport_set_other_func_cap (esw->dev, hca_caps, vport_num, +MLX5_SET_HCA_CAP_OP_MOD_GENERAL_DEVICE ); +if (err) +NL_SET_ERR_MSG_MOD (extack,"Failed setting HCA caps" ); + +out: +mutex_unlock (&esw->state_lock); +kfree (query_ctx); +return err; +}
RE: [net-next v3 2/2] mlx5/core: Support max_io_eqs for a function
Quoted 169 lines
From: David Wei <[email protected]>
Sent: Thursday, April 4, 2024 10:27 PMdrivers/net/ethernet/mellanox/mlx5/core/esw/devlink_port.c static const struct devlink_port_ops
Quoted 33 lines · Parav Pandit
Implement get and set for the maximum IO event queues for SF and VF. This enables administrator on the hypervisor to control the maximum IO event queues which are typically used to derive the maximum and default number of net device channels or rdma device completion vectors.
Reviewed-by Shay Drory <[email protected]>Signed-off-by Parav Pandit <[email protected]>---
changelog:v2->v3:
- limited to 80 chars per line in devlink
- fixed comments from Jakub in mlx5 driver to fix missing mutex unlock on error pathv1->v2:
- fixed comments from Kalesh
- fixed missing kfree in get call
- returning error code for get cmd failure
- fixed error msg copy paste error in set on cmd failure
- limited code to 80 chars limit
- fixed set function variables for reverse christmas tree--- .../mellanox/mlx5/core/esw/devlink_port.c | 4 + .../net/ethernet/mellanox/mlx5/core/eswitch.h | 7 ++ .../mellanox/mlx5/core/eswitch_offloads.c | 97 +++++++++++++++++++ 3 files changed, 108 insertions(+)diff --git
a/drivers/net/ethernet/mellanox/mlx5/core/esw/devlink_port.cb/drivers/net/ethernet/mellanox/mlx5/core/esw/devlink_port.cindexd8e739cbcbce..f8869c9b6802100644drivers/net/ethernet/mellanox/mlx5/core/esw/devlink_port.c +0
-0@@ -98,6 +98,8 @@ static const struct devlink_port_ops
mlx5_esw_pf_vf_dl_port_ops= {drivers/net/ethernet/mellanox/mlx5/core/esw/devlink_port.c static const struct devlink_port_ops · line 98
Quoted 1 line
drivers/net/ethernet/mellanox/mlx5/core/esw/devlink_port.c +0
-0.port_fn_ipsec_packet_get =
mlx5_devlink_port_fn_ipsec_packet_get,drivers/net/ethernet/mellanox/mlx5/core/esw/devlink_port.c static const struct devlink_port_ops · line 99
Quoted 1 line
drivers/net/ethernet/mellanox/mlx5/core/esw/devlink_port.c +0
-0.port_fn_ipsec_packet_set =
mlx5_devlink_port_fn_ipsec_packet_set,Quoted 7 lines
drivers/net/ethernet/mellanox/mlx5/core/esw/devlink_port.c +2
-0#endif /* CONFIG_XFRM_OFFLOAD */ + .port_fn_max_io_eqs_get = mlx5_devlink_port_fn_max_io_eqs_get, + .port_fn_max_io_eqs_set = mlx5_devlink_port_fn_max_io_eqs_set, }; static void mlx5_esw_offloads_sf_devlink_port_attrs_set(struct
mlx5_eswitch*esw, @@ -143,6 +145,8 @@ static const struct
devlink_port_opsmlx5_esw_dl_sf_port_ops= {Quoted 15 lines
drivers/net/ethernet/mellanox/mlx5/core/esw/devlink_port.c +2
-0.port_fn_state_get = mlx5_devlink_sf_port_fn_state_get, .port_fn_state_set = mlx5_devlink_sf_port_fn_state_set, #endif + .port_fn_max_io_eqs_get = mlx5_devlink_port_fn_max_io_eqs_get, + .port_fn_max_io_eqs_set = mlx5_devlink_port_fn_max_io_eqs_set, }; int mlx5_esw_offloads_devlink_port_register(struct mlx5_eswitch *esw,struct
mlx5_vport*vport) diff --gita/drivers/net/ethernet/mellanox/mlx5/core/eswitch.hb/drivers/net/ethernet/mellanox/mlx5/core/eswitch.hindex349e28a6dd8d..50ce1ea20dd4100644drivers/net/ethernet/mellanox/mlx5/core/eswitch.h +0
-0@@ -573,6 +573,13 @@ int mlx5_devlink_port_fn_ipsec_packet_get(structQuoted 1 line
devlink_port*port, bool*is_enintmlx5_devlink_port_fn_ipsec_packet_set(struct
devlink_port*port, bool enable,Quoted 1 line
struct
netlink_ext_ack*extack);#endif /*
Quoted 18 lines
CONFIG_XFRM_OFFLOAD*/drivers/net/ethernet/mellanox/mlx5/core/eswitch.h +7
-0+int mlx5_devlink_port_fn_max_io_eqs_get (struct devlink_port *port, +u32 *max_io_eqs, +struct netlink_ext_ack *extack);int +mlx5_devlink_port_fn_max_io_eqs_set (struct devlink_port *port, +u32 max_io_eqs, +struct netlink_ext_ack *extack); + void *mlx5_eswitch_get_uplink_priv(struct mlx5_eswitch *esw, u8
rep_type);int __mlx5_eswitch_set_vport_vlan(struct
mlx5_eswitch*esw, diff
--gita/drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.cb/drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.cindexbaaae628b0a0..2ad50634b401100644drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c line 73
Quoted 8 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +2
-0@@ -66,6 +66,8 @@#define MLX5_ESW_FT_OFFLOADS_DROP_RULE (1) +#define MLX5_ESW_MAX_CTRL_EQS 4 +static struct esw_vport_tbl_namespace mlx5_esw_vport_tbl_mirror_ns = { .max_fte =MLX5_ESW_VPORT_TBL_SIZE , .max_num_groups = MLX5_ESW_VPORT_TBL_NUM_GROUPS, @@ -4557,3 +4559,98
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c int mlx5_devlink_port_fn_ipsec_packet_set(struct devlink_port *port,
Quoted 7 lines
@@ int mlx5_devlink_port_fn_ipsec_packet_set(struct
devlink_port*port,drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +3
-0return err; }#endif /* CONFIG_XFRM_OFFLOAD */ + +int +mlx5_devlink_port_fn_max_io_eqs_get(struct devlink_port *port, u32
*max_io_eqs,Quoted 4 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +4
-0+ struct netlink_ext_ack *extack) { +struct mlx5_vport *vport =mlx5_devlink_port_vport_get (port); +int query_out_sz =MLX5_ST_SZ_BYTES (query_hca_cap_out); +u16 vport_num = vport->vport;drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c int mlx5_devlink_port_fn_ipsec_packet_set(struct devlink_port *port,
Quoted 10 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +10
-0+struct mlx5_eswitch *esw; +void *query_ctx; +void *hca_caps; +u32 max_eqs; +int err; + + esw =mlx5_devlink_eswitch_nocheck_get (port->devlink); +if (!MLX5_CAP_GEN (esw->dev, vhca_resource_manager)) { + NL_SET_ERR_MSG_MOD(extack, + "Device doesn't support VHCAmanagement");
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c int mlx5_devlink_port_fn_ipsec_packet_set(struct devlink_port *port,
Quoted 9 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +9
-0+return -EOPNOTSUPP ; + } + + query_ctx =kzalloc (query_out_sz,GFP_KERNEL ); +if (!query_ctx) +return -ENOMEM ; + +mutex_lock (&esw->state_lock); + err = mlx5_vport_get_other_func_cap(esw->dev, vport_num,
query_ctx,drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c int mlx5_devlink_port_fn_ipsec_packet_set(struct devlink_port *port,
Quoted 7 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +7
-0+ MLX5_CAP_GENERAL); +if (err) { +NL_SET_ERR_MSG_MOD (extack,"Failed getting HCA caps" ); + goto out; + } + + hca_caps = MLX5_ADDR_OF(query_hca_cap_out, query_ctx,capability);
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c int mlx5_devlink_port_fn_ipsec_packet_set(struct devlink_port *port,
Quoted 13 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +13
-0+ max_eqs =MLX5_GET (cmd_hca_cap, hca_caps, max_num_eqs); +if (max_eqs <MLX5_ESW_MAX_CTRL_EQS ) + *max_io_eqs =0 ; +else + *max_io_eqs = max_eqs -MLX5_ESW_MAX_CTRL_EQS ; +out: +mutex_unlock (&esw->state_lock); +kfree (query_ctx); +return err; +} + +int +mlx5_devlink_port_fn_max_io_eqs_set(struct devlink_port *port, u32
max_io_eqs,Quoted 4 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +4
-0+ struct netlink_ext_ack *extack) { +struct mlx5_vport *vport =mlx5_devlink_port_vport_get (port); +int query_out_sz =MLX5_ST_SZ_BYTES (query_hca_cap_out); +u16 max_eqs = max_io_eqs +MLX5_ESW_MAX_CTRL_EQS ;drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c int mlx5_devlink_port_fn_ipsec_packet_set(struct devlink_port *port,
Quoted 10 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +10
-0+u16 vport_num = vport->vport; +struct mlx5_eswitch *esw; +void *query_ctx; +void *hca_caps; +int err; + + esw =mlx5_devlink_eswitch_nocheck_get (port->devlink); +if (!MLX5_CAP_GEN (esw->dev, vhca_resource_manager)) { + NL_SET_ERR_MSG_MOD(extack, + "Device doesn't support VHCAmanagement");
Quoted 7 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c int mlx5_devlink_port_fn_ipsec_packet_set(struct devlink_port *port, · line 4627
Quoted 4 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +4
-0+return -EOPNOTSUPP ; + } + +if (max_io_eqs +MLX5_ESW_MAX_CTRL_EQS >USHRT_MAX ) {nit: this is the same as
max_eqs
Yes.max_eqs may overflow and we may miss the overflow check by reusing max_eqs.
Above if condition avoids that bug.
That reminds me that I can replace above max_eqs line and this with check_add_overflow() and store the result in max_io_eqs. And save one line. :)
Shall I resend with this change?
Quoted 34 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c
Quoted 1 line
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +1
-0+ NL_SET_ERR_MSG_MOD(extack, "Supplied value out ofrange");
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c int mlx5_devlink_port_fn_ipsec_packet_set(struct devlink_port *port,
Quoted 9 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +9
-0+return -EINVAL ; + } + + query_ctx =kzalloc (query_out_sz,GFP_KERNEL ); +if (!query_ctx) +return -ENOMEM ; + +mutex_lock (&esw->state_lock); + err = mlx5_vport_get_other_func_cap(esw->dev, vport_num,
query_ctx,drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c int mlx5_devlink_port_fn_ipsec_packet_set(struct devlink_port *port,
Quoted 7 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +7
-0+ MLX5_CAP_GENERAL); +if (err) { +NL_SET_ERR_MSG_MOD (extack,"Failed getting HCA caps" ); + goto out; + } + + hca_caps = MLX5_ADDR_OF(query_hca_cap_out, query_ctx,capability);
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c int mlx5_devlink_port_fn_ipsec_packet_set(struct devlink_port *port,
Quoted 3 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +3
-0+MLX5_SET (cmd_hca_cap, hca_caps, max_num_eqs, max_eqs); + + err = mlx5_vport_set_other_func_cap(esw->dev, hca_caps,
vport_num,Quoted 1 line
+
MLX5_SET_HCA_CAP_OP_MOD_GENERAL_DEVICE);Quoted 8 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +8
-0+if (err) +NL_SET_ERR_MSG_MOD (extack,"Failed setting HCA caps" ); + +out: +mutex_unlock (&esw->state_lock); +kfree (query_ctx); +return err; +}
Re: [net-next v3 2/2] mlx5/core: Support max_io_eqs for a function
Quoted 176 lines · Parav Pandit
Quoted 169 lines
From: David Wei <[email protected]>
Sent: Thursday, April 4, 2024 10:27 PMdrivers/net/ethernet/mellanox/mlx5/core/esw/devlink_port.c static const struct devlink_port_ops
Quoted 33 lines · Parav Pandit
Implement get and set for the maximum IO event queues for SF and VF. This enables administrator on the hypervisor to control the maximum IO event queues which are typically used to derive the maximum and default number of net device channels or rdma device completion vectors.
Reviewed-by Shay Drory <[email protected]>Signed-off-by Parav Pandit <[email protected]>---
changelog:v2->v3:
- limited to 80 chars per line in devlink
- fixed comments from Jakub in mlx5 driver to fix missing mutex unlock on error pathv1->v2:
- fixed comments from Kalesh
- fixed missing kfree in get call
- returning error code for get cmd failure
- fixed error msg copy paste error in set on cmd failure
- limited code to 80 chars limit
- fixed set function variables for reverse christmas tree--- .../mellanox/mlx5/core/esw/devlink_port.c | 4 + .../net/ethernet/mellanox/mlx5/core/eswitch.h | 7 ++ .../mellanox/mlx5/core/eswitch_offloads.c | 97 +++++++++++++++++++ 3 files changed, 108 insertions(+)diff --git
a/drivers/net/ethernet/mellanox/mlx5/core/esw/devlink_port.cb/drivers/net/ethernet/mellanox/mlx5/core/esw/devlink_port.cindexd8e739cbcbce..f8869c9b6802100644drivers/net/ethernet/mellanox/mlx5/core/esw/devlink_port.c +0
-0@@ -98,6 +98,8 @@ static const struct devlink_port_ops
mlx5_esw_pf_vf_dl_port_ops= {drivers/net/ethernet/mellanox/mlx5/core/esw/devlink_port.c static const struct devlink_port_ops · line 98
Quoted 1 line
drivers/net/ethernet/mellanox/mlx5/core/esw/devlink_port.c +0
-0.port_fn_ipsec_packet_get =
mlx5_devlink_port_fn_ipsec_packet_get,drivers/net/ethernet/mellanox/mlx5/core/esw/devlink_port.c static const struct devlink_port_ops · line 99
Quoted 1 line
drivers/net/ethernet/mellanox/mlx5/core/esw/devlink_port.c +0
-0.port_fn_ipsec_packet_set =
mlx5_devlink_port_fn_ipsec_packet_set,Quoted 7 lines
drivers/net/ethernet/mellanox/mlx5/core/esw/devlink_port.c +2
-0#endif /* CONFIG_XFRM_OFFLOAD */ + .port_fn_max_io_eqs_get = mlx5_devlink_port_fn_max_io_eqs_get, + .port_fn_max_io_eqs_set = mlx5_devlink_port_fn_max_io_eqs_set, }; static void mlx5_esw_offloads_sf_devlink_port_attrs_set(struct
mlx5_eswitch*esw, @@ -143,6 +145,8 @@ static const struct
devlink_port_opsmlx5_esw_dl_sf_port_ops= {Quoted 15 lines
drivers/net/ethernet/mellanox/mlx5/core/esw/devlink_port.c +2
-0.port_fn_state_get = mlx5_devlink_sf_port_fn_state_get, .port_fn_state_set = mlx5_devlink_sf_port_fn_state_set, #endif + .port_fn_max_io_eqs_get = mlx5_devlink_port_fn_max_io_eqs_get, + .port_fn_max_io_eqs_set = mlx5_devlink_port_fn_max_io_eqs_set, }; int mlx5_esw_offloads_devlink_port_register(struct mlx5_eswitch *esw,struct
mlx5_vport*vport) diff --gita/drivers/net/ethernet/mellanox/mlx5/core/eswitch.hb/drivers/net/ethernet/mellanox/mlx5/core/eswitch.hindex349e28a6dd8d..50ce1ea20dd4100644drivers/net/ethernet/mellanox/mlx5/core/eswitch.h +0
-0@@ -573,6 +573,13 @@ int mlx5_devlink_port_fn_ipsec_packet_get(structQuoted 1 line
devlink_port*port, bool*is_enintmlx5_devlink_port_fn_ipsec_packet_set(struct
devlink_port*port, bool enable,Quoted 1 line
struct
netlink_ext_ack*extack);#endif /*
Quoted 18 lines
CONFIG_XFRM_OFFLOAD*/drivers/net/ethernet/mellanox/mlx5/core/eswitch.h +7
-0+int mlx5_devlink_port_fn_max_io_eqs_get (struct devlink_port *port, +u32 *max_io_eqs, +struct netlink_ext_ack *extack);int +mlx5_devlink_port_fn_max_io_eqs_set (struct devlink_port *port, +u32 max_io_eqs, +struct netlink_ext_ack *extack); + void *mlx5_eswitch_get_uplink_priv(struct mlx5_eswitch *esw, u8
rep_type);int __mlx5_eswitch_set_vport_vlan(struct
mlx5_eswitch*esw, diff
--gita/drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.cb/drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.cindexbaaae628b0a0..2ad50634b401100644drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c line 73
Quoted 8 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +2
-0@@ -66,6 +66,8 @@#define MLX5_ESW_FT_OFFLOADS_DROP_RULE (1) +#define MLX5_ESW_MAX_CTRL_EQS 4 +static struct esw_vport_tbl_namespace mlx5_esw_vport_tbl_mirror_ns = { .max_fte =MLX5_ESW_VPORT_TBL_SIZE , .max_num_groups = MLX5_ESW_VPORT_TBL_NUM_GROUPS, @@ -4557,3 +4559,98
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c int mlx5_devlink_port_fn_ipsec_packet_set(struct devlink_port *port,
Quoted 7 lines
@@ int mlx5_devlink_port_fn_ipsec_packet_set(struct
devlink_port*port,drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +3
-0return err; }#endif /* CONFIG_XFRM_OFFLOAD */ + +int +mlx5_devlink_port_fn_max_io_eqs_get(struct devlink_port *port, u32
*max_io_eqs,Quoted 4 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +4
-0+ struct netlink_ext_ack *extack) { +struct mlx5_vport *vport =mlx5_devlink_port_vport_get (port); +int query_out_sz =MLX5_ST_SZ_BYTES (query_hca_cap_out); +u16 vport_num = vport->vport;drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c int mlx5_devlink_port_fn_ipsec_packet_set(struct devlink_port *port,
Quoted 10 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +10
-0+struct mlx5_eswitch *esw; +void *query_ctx; +void *hca_caps; +u32 max_eqs; +int err; + + esw =mlx5_devlink_eswitch_nocheck_get (port->devlink); +if (!MLX5_CAP_GEN (esw->dev, vhca_resource_manager)) { + NL_SET_ERR_MSG_MOD(extack, + "Device doesn't support VHCAmanagement");
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c int mlx5_devlink_port_fn_ipsec_packet_set(struct devlink_port *port,
Quoted 9 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +9
-0+return -EOPNOTSUPP ; + } + + query_ctx =kzalloc (query_out_sz,GFP_KERNEL ); +if (!query_ctx) +return -ENOMEM ; + +mutex_lock (&esw->state_lock); + err = mlx5_vport_get_other_func_cap(esw->dev, vport_num,
query_ctx,drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c int mlx5_devlink_port_fn_ipsec_packet_set(struct devlink_port *port,
Quoted 7 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +7
-0+ MLX5_CAP_GENERAL); +if (err) { +NL_SET_ERR_MSG_MOD (extack,"Failed getting HCA caps" ); + goto out; + } + + hca_caps = MLX5_ADDR_OF(query_hca_cap_out, query_ctx,capability);
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c int mlx5_devlink_port_fn_ipsec_packet_set(struct devlink_port *port,
Quoted 13 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +13
-0+ max_eqs =MLX5_GET (cmd_hca_cap, hca_caps, max_num_eqs); +if (max_eqs <MLX5_ESW_MAX_CTRL_EQS ) + *max_io_eqs =0 ; +else + *max_io_eqs = max_eqs -MLX5_ESW_MAX_CTRL_EQS ; +out: +mutex_unlock (&esw->state_lock); +kfree (query_ctx); +return err; +} + +int +mlx5_devlink_port_fn_max_io_eqs_set(struct devlink_port *port, u32
max_io_eqs,Quoted 4 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +4
-0+ struct netlink_ext_ack *extack) { +struct mlx5_vport *vport =mlx5_devlink_port_vport_get (port); +int query_out_sz =MLX5_ST_SZ_BYTES (query_hca_cap_out); +u16 max_eqs = max_io_eqs +MLX5_ESW_MAX_CTRL_EQS ;drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c int mlx5_devlink_port_fn_ipsec_packet_set(struct devlink_port *port,
Quoted 10 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +10
-0+u16 vport_num = vport->vport; +struct mlx5_eswitch *esw; +void *query_ctx; +void *hca_caps; +int err; + + esw =mlx5_devlink_eswitch_nocheck_get (port->devlink); +if (!MLX5_CAP_GEN (esw->dev, vhca_resource_manager)) { + NL_SET_ERR_MSG_MOD(extack, + "Device doesn't support VHCAmanagement");
Quoted 7 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c int mlx5_devlink_port_fn_ipsec_packet_set(struct devlink_port *port, · line 4627
Quoted 4 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +4
-0+return -EOPNOTSUPP ; + } + +if (max_io_eqs +MLX5_ESW_MAX_CTRL_EQS >USHRT_MAX ) {nit: this is the same as
max_eqs
Quoted 3 lines
Yes.
max_eqsmay overflow and we may miss the overflow check by reusingmax_eqs. Above if condition avoids that bug.
Thanks, I missed that max_io_eqs is u32.
Quoted 4 lines
That reminds me that I can replace above
max_eqsline and this withcheck_add_overflow()and store the result inmax_io_eqs. And save one line. :)Shall I resend with this change?
Yes that sounds good, thank you.
Quoted 35 lines
Quoted 35 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c
Quoted 2 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +1
-0+ NL_SET_ERR_MSG_MOD(extack, "Supplied value out ofrange");
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c int mlx5_devlink_port_fn_ipsec_packet_set(struct devlink_port *port,
Quoted 9 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +9
-0+return -EINVAL ; + } + + query_ctx =kzalloc (query_out_sz,GFP_KERNEL ); +if (!query_ctx) +return -ENOMEM ; + +mutex_lock (&esw->state_lock); + err = mlx5_vport_get_other_func_cap(esw->dev, vport_num,
query_ctx,drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c int mlx5_devlink_port_fn_ipsec_packet_set(struct devlink_port *port,
Quoted 7 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +7
-0+ MLX5_CAP_GENERAL); +if (err) { +NL_SET_ERR_MSG_MOD (extack,"Failed getting HCA caps" ); + goto out; + } + + hca_caps = MLX5_ADDR_OF(query_hca_cap_out, query_ctx,capability);
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c int mlx5_devlink_port_fn_ipsec_packet_set(struct devlink_port *port,
Quoted 3 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +3
-0+MLX5_SET (cmd_hca_cap, hca_caps, max_num_eqs, max_eqs); + + err = mlx5_vport_set_other_func_cap(esw->dev, hca_caps,
vport_num,Quoted 1 line
+
MLX5_SET_HCA_CAP_OP_MOD_GENERAL_DEVICE);Quoted 8 lines
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +8
-0+if (err) +NL_SET_ERR_MSG_MOD (extack,"Failed setting HCA caps" ); + +out: +mutex_unlock (&esw->state_lock); +kfree (query_ctx); +return err; +}