devlink: Add port function attribute to enable/disable roce
Subsystems (5)
cover devlink: Add port function attribute to enable/disable roce
Currently mlx5 PCI VF and SF are enabled by default for RoCE functionality.
Currently a user does not have the ability to disable RoCE for a PCI VF/SF device before such device is enumerated by the driver.
User is also incapable to do such setting from smartnic scenario for a VF from the smartnic.
Current 'enable_roce' device knob is limited to do setting only at driverinit time. By this time device is already created and firmware has already allocated necessary system memory for supporting RoCE.
This is a hyper visor level control, to restrict the functionality of devices passed through to guests.
This is achieved by extending existing 'port function' object to control capabilities of a function. This enables users to control capability of the device before enumeration.
Examples when user prefers to disable RoCE for a VF when using switchdev mode:
$ devlink port show pci/0000:06:00.0/1
pci/0000:06:00.0/1: type eth netdev pf0vf0 flavour pcivf controller 0
pfnum 0 vfnum 0 external false splittable false
function:
hw_addr 00:00:00:00:00:00 roce on$ devlink port function set pci/0000:06:00.0/1 roce off
$ devlink port show pci/0000:06:00.0/1
pci/0000:06:00.0/1: type eth netdev pf0vf0 flavour pcivf controller 0
pfnum 0 vfnum 0 external false splittable false
function:
hw_addr 00:00:00:00:00:00 roce offFAQs:
-----
1. What does roce on/off do?
Ans: It disables RoCE capability of the function before its enumerated, so when driver reads the capability from the device firmware, it is disabled.
At this point RDMA stack will not be able to create UD, QP1, RC, XRC type of QPs. When RoCE is disabled, the GID table of all ports of the device is disabled in the device and software stack.
2. How is the roce 'port function' option different from existing devlink param?
Ans: RoCE attribute at the port function level disables the RoCE capability at the specific function level; while enable_roce only does at the software level.
3. Why is this option for disabling only RoCE and not the whole RDMA device?
Ans: Because user still wants to use the RDMA device for non RoCE commands in more memory efficient way.
Patch summary:
Patch-1 adds devlink attribute to control roce Patch-2 implements mlx5 callbacks for roce control
Yishai Hadas (2):
devlink: Expose port function commands to control roce net/mlx5: E-Switch, Implement devlink port function cmds to control
roce
.../device_drivers/ethernet/mellanox/mlx5.rst | 32 ++++++ .../networking/devlink/devlink-port.rst | 5 +- .../net/ethernet/mellanox/mlx5/core/devlink.c | 2 + .../net/ethernet/mellanox/mlx5/core/eswitch.c | 34 ++++++ .../net/ethernet/mellanox/mlx5/core/eswitch.h | 8 +- .../mellanox/mlx5/core/eswitch_offloads.c | 105 ++++++++++++++++++ .../ethernet/mellanox/mlx5/core/mlx5_core.h | 2 + .../net/ethernet/mellanox/mlx5/core/vport.c | 23 ++++ include/net/devlink.h | 20 ++++ include/uapi/linux/devlink.h | 1 + net/core/devlink.c | 61 ++++++++++ 11 files changed, 291 insertions(+), 2 deletions(-)
Signature · 1 line
2.27.0
PATCH 1/2 devlink: Expose port function commands to control roce
From: Yishai Hadas <[email protected]>
Expose port function commands to turn on / off roce, this is used to control the port roce device capabilities.
When roce is disabled for a function of the port, function cannot create any roce specific resources (e.g GID table).
It also saves system memory utilization. For example disabling roce on a VF/SF saves 1 Mbytes of system memory per function.
Example of a PCI VF port which supports function configuration:
Set roce of the VF's port function.
$ devlink port show pci/0000:06:00.0/2
pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1
function:
hw_addr 00:00:00:00:00:00 roce on$ devlink port function set pci/0000:06:00.0/2 roce off
$ devlink port show pci/0000:06:00.0/2
pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1
function:
hw_addr 00:11:22:33:44:55 roce off--- .../networking/devlink/devlink-port.rst | 5 +- include/net/devlink.h | 20 ++++++ include/uapi/linux/devlink.h | 1 + net/core/devlink.c | 61 +++++++++++++++++++ 4 files changed, 86 insertions(+), 1 deletion(-)
Documentation/networking/devlink/devlink-port.rst +4 -1
@@ -110,7 +110,7 @@ devlink ports for both the controllers. Function configuration ====================== -A user can configure the function attribute before enumerating the PCI +A user can configure one or more function attributes before enumerating the PCI function. Usually it means, user should configure function attribute before a bus specific device for the function is created. However, when SRIOV is enabled, virtual function devices are created on the PCI bus.
@@ -122,6 +122,9 @@ A user may set the hardware address of the function using 'devlink port function set hw_addr' command. For Ethernet port function this means a MAC address. +A user may set also the roce capability of the function using +'devlink port function set roce' command. + Subfunction ============
include/net/devlink.h +20 -0
@@ -1414,6 +1414,26 @@ struct devlink_ops {int (*port_function_hw_addr_set)(struct devlink_port *port,const u8 *hw_addr,int hw_addr_len,struct netlink_ext_ack *extack); +/** +* @port_function_roce_get: Port function's roce get function. +* +* Should be used by device drivers to report the roce state of +* a function managed by the devlink port. Driver should return +* -EOPNOTSUPP if it doesn't support port function handling for +* a particular port. +*/ +int (*port_function_roce_get)(struct devlink_port *port,bool *on, +struct netlink_ext_ack *extack); +/** +* @port_function_roce_set: Port function's roce set function. +* +* Should be used by device drivers to enable/disable the roce state of +* a function managed by the devlink port. Driver should return +* -EOPNOTSUPP if it doesn't support port function handling for +* a particular port. +*/ +int (*port_function_roce_set)(struct devlink_port *port,bool on, +struct netlink_ext_ack *extack);/** * port_new() - Add a new port function of a specified flavor * @devlink: Devlink instance
include/uapi/linux/devlink.h +1 -0
@@ -658,6 +658,7 @@ enum devlink_port_function_attr {DEVLINK_PORT_FUNCTION_ATTR_HW_ADDR ,/* binary */ DEVLINK_PORT_FN_ATTR_STATE ,/* u8 */ DEVLINK_PORT_FN_ATTR_OPSTATE ,/* u8 */ +DEVLINK_PORT_FN_ATTR_ROCE ,/* u8 */ __DEVLINK_PORT_FUNCTION_ATTR_MAX,DEVLINK_PORT_FUNCTION_ATTR_MAX = __DEVLINK_PORT_FUNCTION_ATTR_MAX -1
net/core/devlink.c +61 -0
91 lines not on this page · raw message
Signature · 1 line
2.27.0
PATCH 2/2 net/mlx5: E-Switch, Implement devlink port function cmds to control roce
In reply to Daniel JurgensFrom: Yishai Hadas <[email protected]>
Implement devlink port function commands to enable / disable roce. This is used to control the roce device capabilities.
--- .../device_drivers/ethernet/mellanox/mlx5.rst | 32 ++++++ .../net/ethernet/mellanox/mlx5/core/devlink.c | 2 + .../net/ethernet/mellanox/mlx5/core/eswitch.c | 34 ++++++ .../net/ethernet/mellanox/mlx5/core/eswitch.h | 8 +- .../mellanox/mlx5/core/eswitch_offloads.c | 105 ++++++++++++++++++ .../ethernet/mellanox/mlx5/core/mlx5_core.h | 2 + .../net/ethernet/mellanox/mlx5/core/vport.c | 23 ++++ 7 files changed, 205 insertions(+), 1 deletion(-)
Documentation/networking/device_drivers/ethernet/mellanox/mlx5.rst +32 -0
@@ -388,6 +388,38 @@ device created for the PCI VF/SF. function: hw_addr 00:00:00:00:88:88 +RoCE capability setup +--------------------- +Not all mlx5 PCI VFs/SFs require RoCE capability. + +When RoCE capability is disabled, it saves 1 Mbytes worth of system memory per +PCI VF/SF. + +mlx5 driver provides mechanism to setup RoCE capability. + +When user disables RoCE capability for a VF/SF, user application cannot send or +receive any RoCE packets through this VF/SF and RoCE GID table for this PCI +will be empty. + +When RoCE capability is disabled in the device using port function attribute, +VF/SF driver cannot override it. + +- Get RoCE capability of the VF device:: + + $ devlink port show pci/0000:06:00.0/2 + pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1 + function: + hw_addr 00:00:00:00:00:00 roce on + +- Set RoCE capability of the VF device:: + + $ devlink port function set pci/0000:06:00.0/2 roce off + + $ devlink port show pci/0000:06:00.0/2 + pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1 + function: + hw_addr 00:00:00:00:00:00 roce off + SF state setup -------------- To use the SF, the user must active the SF using the SF function state
drivers/net/ethernet/mellanox/mlx5/core/devlink.c +2 -0
@@ -318,6 +318,8 @@ static const struct devlink_ops mlx5_devlink_ops = { .rate_node_new = mlx5_esw_devlink_rate_node_new, .rate_node_del = mlx5_esw_devlink_rate_node_del, .rate_leaf_parent_set = mlx5_esw_devlink_rate_parent_set, + .port_function_roce_get = mlx5_devlink_port_function_roce_get, + .port_function_roce_set = mlx5_devlink_port_function_roce_set,#endif #ifdef CONFIG_MLX5_SF_MANAGER .port_new = mlx5_devlink_sf_port_new,
drivers/net/ethernet/mellanox/mlx5/core/eswitch.c +34 -0
@@ -772,6 +772,32 @@ static void esw_vport_cleanup_acl(struct mlx5_eswitch *esw,esw_vport_destroy_offloads_acl_tables (esw, vport); } +static int mlx5_esw_vport_roce_cap_get (struct mlx5_eswitch *esw,struct mlx5_vport *vport) +{ +int query_out_sz =MLX5_ST_SZ_BYTES (query_hca_cap_out); +void *query_ctx; +void *hca_caps; +int err; + +if (!MLX5_CAP_GEN (esw->dev, vhca_resource_manager)) +return 0 ; + + query_ctx =kzalloc (query_out_sz,GFP_KERNEL ); +if (!query_ctx) +return -ENOMEM ; + + err =mlx5_vport_get_other_func_cap (esw->dev, vport->vport, query_ctx); +if (err) + goto out_free; + + hca_caps =MLX5_ADDR_OF (query_hca_cap_out, query_ctx, capability); + vport->info.roce_enabled =MLX5_GET (cmd_hca_cap, hca_caps, roce); + +out_free: +kfree (query_ctx); +return err; +} +static int esw_vport_setup (struct mlx5_eswitch *esw,struct mlx5_vport *vport) {u16 vport_num = vport->vport;
@@ -785,6 +811,10 @@ static int esw_vport_setup(struct mlx5_eswitch *esw, struct mlx5_vport *vport)if (mlx5_esw_is_manager_vport (esw, vport_num))return 0 ; + err =mlx5_esw_vport_roce_cap_get (esw, vport); +if (err) + goto err_roce; + mlx5_modify_vport_admin_state(esw->dev, MLX5_VPORT_STATE_OP_MOD_ESW_VPORT, vport_num, 1,
@@ -804,6 +834,10 @@ static int esw_vport_setup(struct mlx5_eswitch *esw, struct mlx5_vport *vport) vport->info.qos, flags);return 0 ; + +err_roce: +esw_vport_cleanup_acl (esw, vport); +return err; }/* Don't cleanup vport->info, it's needed to restore vport configuration */
drivers/net/ethernet/mellanox/mlx5/core/eswitch.h +7 -1
@@ -153,6 +153,7 @@ struct mlx5_vport_info {u8 qos; u8 spoofchk: 1; u8 trusted: 1; + u8 roce_enabled: 1; };/* Vport context events */
@@ -508,7 +509,12 @@ int mlx5_devlink_port_function_hw_addr_get(struct devlink_port *port,int mlx5_devlink_port_function_hw_addr_set (struct devlink_port *port,const u8 *hw_addr,int hw_addr_len,struct netlink_ext_ack *extack); - +int mlx5_devlink_port_function_roce_get (struct devlink_port *port, +bool *is_enabled, +struct netlink_ext_ack *extack); +int mlx5_devlink_port_function_roce_set (struct devlink_port *port, +bool enable, +struct netlink_ext_ack *extack);void *mlx5_eswitch_get_uplink_priv (struct mlx5_eswitch *esw,u8 rep_type); int mlx5_eswitch_add_vlan_action(struct mlx5_eswitch *esw,
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +105 -0
108 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h +2 -0
@@ -309,6 +309,8 @@ enum {u8 mlx5_get_nic_state (struct mlx5_core_dev *dev);void mlx5_set_nic_state (struct mlx5_core_dev *dev,u8 state); +int mlx5_vport_get_other_func_cap (struct mlx5_core_dev *dev,u16 function_id,void *out); +int mlx5_vport_set_other_func_cap (struct mlx5_core_dev *dev,const void *hca_cap,u16 function_id);static inline bool mlx5_core_is_sf (const struct mlx5_core_dev *dev) {
drivers/net/ethernet/mellanox/mlx5/core/vport.c +23 -0
@@ -1171,3 +1171,26 @@ int mlx5_vport_get_other_func_cap(struct mlx5_core_dev *dev, u16 function_id, voMLX5_SET (query_hca_cap_in, in, other_function, true);return mlx5_cmd_exec_inout (dev, query_hca_cap, in, out); } + +int mlx5_vport_set_other_func_cap (struct mlx5_core_dev *dev,const void *hca_cap,u16 function_id) +{ +int set_sz =MLX5_ST_SZ_BYTES (set_hca_cap_in); +void *set_hca_cap; +void *set_ctx; +int ret; + + set_ctx =kzalloc (set_sz,GFP_KERNEL ); +if (!set_ctx) +return -ENOMEM ; + +MLX5_SET (set_hca_cap_in, set_ctx, opcode,MLX5_CMD_OP_SET_HCA_CAP ); +MLX5_SET (set_hca_cap_in, set_ctx, op_mod,MLX5_SET_HCA_CAP_OP_MOD_GENERAL_DEVICE <<1 ); + set_hca_cap =MLX5_ADDR_OF (set_hca_cap_in, set_ctx, capability); +memcpy (set_hca_cap, hca_cap,MLX5_ST_SZ_BYTES (cmd_hca_cap)); +MLX5_SET (set_hca_cap_in, set_ctx, function_id, function_id); +MLX5_SET (set_hca_cap_in, set_ctx, other_function, true); + ret =mlx5_cmd_exec_in (dev, set_hca_cap, set_ctx); + +kfree (set_ctx); +return ret; +}
Signature · 1 line
2.27.0
Jakub Kicinski
maintainerRe: [PATCH 1/2] devlink: Expose port function commands to control roce
In reply to Daniel JurgensQuoted 25 lines · On Wed, 2 Nov 2022 18:39:53 +0200 Daniel Jurgens wrote:
From: Yishai Hadas <[email protected]>
Expose port function commands to turn on / off roce, this is used to control the port roce device capabilities.
When roce is disabled for a function of the port, function cannot create any roce specific resources (e.g GID table).
It also saves system memory utilization. For example disabling roce on a VF/SF saves 1 Mbytes of system memory per function.Example of a PCI VF port which supports function configuration:
Set roce of the VF's port function.$ devlink port show pci/0000:06:00.0/2 pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1 function: hw_addr 00:00:00:00:00:00 roce on$ devlink port function set pci/0000:06:00.0/2 roce off$ devlink port show pci/0000:06:00.0/2 pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1 function: hw_addr 00:11:22:33:44:55 roce off
Quoted 5 lines
Signed-off-by Yishai Hadas <[email protected]>Signed-off-by Daniel Jurgens <[email protected]>Reviewed-by Parav Pandit <[email protected]>Reviewed-by Jiri Pirko <[email protected]>Signed-off-by Saeed Mahameed <[email protected]>
LGTM, handful of minor nit picks:
Documentation/networking/devlink/devlink-port.rst devlink ports for both the controllers. · line 113
Quoted 10 lines
Documentation/networking/devlink/devlink-port.rst +1
-1@@ -110,7 +110,7 @@ devlink ports for both the controllers. Function configuration ====================== -A user can configure the function attribute before enumerating the PCI +A user can configure one or more function attributes before enumerating the PCI
I'm not an expert on English grammar, but this sounds odd. I think it is a generic reference, so the most suitable form would to use plural "Users". Since we're touching this line anyway...
Documentation/networking/devlink/devlink-port.rst A user may set the hardware address of the function using · line 125
Quoted 8 lines
Documentation/networking/devlink/devlink-port.rst +1
-0function. Usually it means, user should configure function attribute before a bus specific device for the function is created. However, when SRIOV is enabled, virtual function devices are created on the PCI bus.@@ -122,6 +122,9 @@ A user may set the hardware address of the function using 'devlink port function set hw_addr' command. For Ethernet port function this means a MAC address. +A user may set also the roce capability of the function using
... and adding another instance here. "also" before "set". roce -> RoCE.
Documentation/networking/devlink/devlink-port.rst A user may set the hardware address of the function using
Quoted 12 lines
Documentation/networking/devlink/devlink-port.rst +9
-0+'devlink port function set roce' command. + Subfunction ============ + /** + * @port_function_roce_get: Port function's roce get function. + * + * Should be used by device drivers to report the roce state of + * a function managed by the devlink port. Driver should return + * -EOPNOTSUPP if it doesn't support port function handling for + * a particular port.
Use imperative:
* Query RoCE state of a function managed ...
* Return -EOPNOTSUPP if port function handing is not supported.
Documentation/networking/devlink/devlink-port.rst
Quoted 13 lines
Documentation/networking/devlink/devlink-port.rst +12
-0+ int (*port_function_roce_get)(struct devlink_port *port, bool *on, + struct netlink_ext_ack *extack); + /** + * @port_function_roce_set: Port function's roce set function. + * + * Should be used by device drivers to enable/disable the roce state of + * a function managed by the devlink port. Driver should return + * -EOPNOTSUPP if it doesn't support port function handling for + * a particular port. + */ + int (*port_function_roce_set)(struct devlink_port *port, bool on, + DEVLINK_PORT_FN_ATTR_ROCE, /* u8 */
Please use u32, forget u8 and u16 exist, netlink rounds up the size of attributes to 4B, anyway.
Quoted 6 lines
Documentation/networking/devlink/devlink-port.rst +2
-0__DEVLINK_PORT_FUNCTION_ATTR_MAX, DEVLINK_PORT_FUNCTION_ATTR_MAX = __DEVLINK_PORT_FUNCTION_ATTR_MAX - 1 +static int +devlink_port_fn_roce_set(struct devlink_port *port,
Documentation/networking/devlink/devlink-port.rst
Quoted 10 lines
Documentation/networking/devlink/devlink-port.rst +10
-0+ const struct nlattr *attr, + struct netlink_ext_ack *extack) +{ + const struct devlink_ops *ops = port->devlink->ops; + bool on; + + on = nla_get_u8(attr); + + if (!ops->port_function_roce_set) { + NL_SET_ERR_MSG_MOD(extack,
NL_SET_ERR_MSG_ATTR(), please
Quoted 24 lines
Documentation/networking/devlink/devlink-port.rst +13
-0+ "Port doesn't support roce function attribute"); + return -EOPNOTSUPP; + } + + return ops->port_function_roce_set(port, on, extack); +} + static int devlink_nl_port_function_attrs_put(struct sk_buff *msg, struct devlink_port *port, struct netlink_ext_ack *extack)@@ -1266,6 +1313,12 @@ devlink_nl_port_function_attrs_put(struct sk_buff *msg, struct devlink_port *por &msg_updated); if (err) goto out; + + err = devlink_port_function_roce_fill(ops, port, msg, extack, + &msg_updated); + if (err) + goto out; + err = devlink_port_fn_state_fill(ops, port, msg, extack, &msg_updated); out: if (err || !msg_updated)
Documentation/networking/devlink/devlink-port.rst static int devlink_port_function_set(struct devlink_port *port, · line 1729
Quoted 7 lines
Documentation/networking/devlink/devlink-port.rst +4
-0@@ -1670,6 +1723,14 @@ static int devlink_port_function_set(struct devlink_port *port, if (err) return err; } + + attr = tb[DEVLINK_PORT_FN_ATTR_ROCE]; + if (attr) { + err = devlink_port_fn_roce_set(port, attr, extack);
We should try a little harder to avoid partial request processing. Let's check if the device has the callbacks for all the settings in the request upfront?
Quoted 7 lines
Documentation/networking/devlink/devlink-port.rst +4
-0+ if (err) + return err; + } + /* Keep this as the last function attribute set, so that when * multiple port function attributes are set along with state, * Those can be applied first before activating the state.