Add mlx5 subfunction support
Subsystems (8)
v4 cover Add mlx5 subfunction support
Hi Dave, Jakub, Jason,
This series form Parav was the theme of this mlx5 release cycle, we've been waiting anxiously for the auxbus infrastructure to make it into the kernel, and now as the auxbus is in and all the stars are aligned, I can finally submit this V2 of the devlink and mlx5 subfunction support.
Subfunctions came to solve the scaling issue of virtualization and switchdev environments, where SRIOV failed to deliver and users ran out of VFs very quickly as SRIOV demands huge amount of physical resources in both of the servers and the NIC.
Subfunction provide the same functionality as SRIOV but in a very lightweight manner, please see the thorough and detailed documentation from Parav below, in the commit messages and the Networking documentation patches at the end of this series.
Sending V4 as a continuation to V1 that was sent Last month [0],
[0] https://lore.kernel.org/linux-rdma/[email protected]/
---
Changelog:v3->v4:
- Fix 32bit compilation issue
v2->v3:
- added header file sf/priv.h to cmd.c to avoid missing prototype warning
- made mlx5_sf_table_disable as static function as its used only in one file
v1->v2:
- added documentation for subfunction and its mlx5 implementation
- add MLX5_SF config option documentation
- rebased
- dropped devlink global lock improvement patch as mlx5 doesn't support reload while SFs are allocated
- dropped devlink reload lock patch as mlx5 doesn't support reload when SFs are allocated
- using updated vhca event from device to add remove auxiliary device
- split sf devlink port allocation and sf hardware context allocation
Parav Pandit Says:
=================
This patchset introduces support for mlx5 subfunction (SF).
A subfunction is a lightweight function that has a parent PCI function on which it is deployed. mlx5 subfunction has its own function capabilities and its own resources. This means a subfunction has its own dedicated queues(txq, rxq, cq, eq). These queues are neither shared nor stealed from the parent PCI function.
When subfunction is RDMA capable, it has its own QP1, GID table and rdma resources neither shared nor stealed from the parent PCI function.
A subfunction has dedicated window in PCI BAR space that is not shared with ther other subfunctions or parent PCI function. This ensures that all class devices of the subfunction accesses only assigned PCI BAR space.
A Subfunction supports eswitch representation through which it supports tc offloads. User must configure eswitch to send/receive packets from/to subfunction port.
Subfunctions share PCI level resources such as PCI MSI-X IRQs with their other subfunctions and/or with its parent PCI function.
Patch summary:
--------------
Patch 1 to 4 prepares devlink
patch 5 to 7 mlx5 adds SF device support
Patch 8 to 11 mlx5 adds SF devlink port support
Patch 12 and 14 adds documentation
Patch-1 prepares code to handle multiple port function attributes Patch-2 introduces devlink pcisf port flavour similar to pcipf and pcivf Patch-3 adds port add and delete driver callbacks
Patch-4 adds port function state get and set callbacks Patch-5 mlx5 vhca event notifier support to distribute subfunction
state change notification
Patch-6 adds SF auxiliary device
Patch-7 adds SF auxiliary driver
Patch-8 prepares eswitch to handler SF vport
Patch-9 adds eswitch helpers to add/remove SF vport
Patch-10 implements devlink port add/del callbacks
Patch-11 implements devlink port function get/set callbacks Patch-12 to 14 adds documentation
Patch-12 added mlx5 port function documentation
Patch-13 adds subfunction documentation
Patch-14 adds mlx5 subfunction documentation
Subfunction support is discussed in detail in RFC [1] and [2]. RFC [1] and extension [2] describes requirements, design and proposed plumbing using devlink, auxiliary bus and sysfs for systemd/udev support. Functionality of this patchset is best explained using real examples further below.
overview:
--------
A subfunction can be created and deleted by a user using devlink port add/delete interface.
A subfunction can be configured using devlink port function attribute before its activated.
When a subfunction is activated, it results in an auxiliary device on the host PCI device where it is deployed. A driver binds to the auxiliary device that further creates supported class devices.
example subfunction usage sequence:
-----------------------------------
Change device to switchdev mode:
$ devlink dev eswitch set pci/0000:06:00.0 mode switchdev
Add a devlink port of subfunction flaovur:
$ devlink port add pci/0000:06:00.0 flavour pcisf pfnum 0 sfnum 88
Configure mac address of the port function:
$ devlink port function set ens2f0npf0sf88 hw_addr 00:00:00:00:88:88
Now activate the function:
$ devlink port function set ens2f0npf0sf88 state active
Now use the auxiliary device and class devices:
$ devlink dev show pci/0000:06:00.0 auxiliary/mlx5_core.sf.4
$ ip link show
127: ens2f0np0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 24:8a:07:b3:d1:12 brd ff:ff:ff:ff:ff:ff
altname enp6s0f0np0
129: p0sf88: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 00:00:00:00:88:88 brd ff:ff:ff:ff:ff:ff$ rdma dev show 43: rdmap6s0f0: node_type ca fw 16.29.0550 node_guid 248a:0703:00b3:d112 sys_image_guid 248a:0703:00b3:d112 44: mlx5_0: node_type ca fw 16.29.0550 node_guid 0000:00ff:fe00:8888 sys_image_guid 248a:0703:00b3:d112
After use inactivate the function:
$ devlink port function set ens2f0npf0sf88 state inactive
Now delete the subfunction port:
$ devlink port del ens2f0npf0sf88
[1] https://lore.kernel.org/netdev/20200519092258.GF4655@nanopsycho/
[2] https://marc.info/?l=linux-netdev&m=158555928517777&w=2
=================
Parav Pandit (14):
net/mlx5: Fix compilation warning for 32-bit platform
devlink: Prepare code to fill multiple port function attributes
devlink: Introduce PCI SF port flavour and port attribute
devlink: Support add and delete devlink port
devlink: Support get and set state of port function net/mlx5: Introduce vhca state event notifier net/mlx5: SF, Add auxiliary device support
net/mlx5: SF, Add auxiliary device driver
net/mlx5: E-switch, Add eswitch helpers for SF vport net/mlx5: SF, Add port add delete functionality net/mlx5: SF, Port function state change support
devlink: Add devlink port documentation
devlink: Extend devlink port documentation for subfunctions net/mlx5: Add devlink subfunction port documentation
Vu Pham (1):
net/mlx5: E-switch, Prepare eswitch to handle SF vport
Documentation/driver-api/auxiliary_bus.rst | 2 + .../device_drivers/ethernet/mellanox/mlx5.rst | 209 +++++++ .../networking/devlink/devlink-port.rst | 199 +++++++ Documentation/networking/devlink/index.rst | 1 + .../net/ethernet/mellanox/mlx5/core/Kconfig | 19 + .../net/ethernet/mellanox/mlx5/core/Makefile | 9 + drivers/net/ethernet/mellanox/mlx5/core/cmd.c | 8 + .../net/ethernet/mellanox/mlx5/core/devlink.c | 19 + drivers/net/ethernet/mellanox/mlx5/core/eq.c | 5 +- .../mellanox/mlx5/core/esw/acl/egress_ofld.c | 2 +- .../mellanox/mlx5/core/esw/devlink_port.c | 41 ++ .../net/ethernet/mellanox/mlx5/core/eswitch.c | 48 +- .../net/ethernet/mellanox/mlx5/core/eswitch.h | 78 +++ .../mellanox/mlx5/core/eswitch_offloads.c | 47 +- .../net/ethernet/mellanox/mlx5/core/events.c | 7 + .../net/ethernet/mellanox/mlx5/core/main.c | 60 +- .../ethernet/mellanox/mlx5/core/mlx5_core.h | 12 + .../net/ethernet/mellanox/mlx5/core/pci_irq.c | 20 + .../net/ethernet/mellanox/mlx5/core/sf/cmd.c | 49 ++ .../ethernet/mellanox/mlx5/core/sf/dev/dev.c | 271 +++++++++ .../ethernet/mellanox/mlx5/core/sf/dev/dev.h | 55 ++ .../mellanox/mlx5/core/sf/dev/driver.c | 101 ++++ .../ethernet/mellanox/mlx5/core/sf/devlink.c | 552 ++++++++++++++++++ .../ethernet/mellanox/mlx5/core/sf/hw_table.c | 233 ++++++++ .../mlx5/core/sf/mlx5_ifc_vhca_event.h | 82 +++ .../net/ethernet/mellanox/mlx5/core/sf/priv.h | 21 + .../net/ethernet/mellanox/mlx5/core/sf/sf.h | 92 +++ .../mellanox/mlx5/core/sf/vhca_event.c | 189 ++++++ .../mellanox/mlx5/core/sf/vhca_event.h | 57 ++ .../net/ethernet/mellanox/mlx5/core/vport.c | 3 +- include/linux/mlx5/driver.h | 16 +- include/linux/mlx5/mlx5_ifc.h | 6 +- include/net/devlink.h | 79 +++ include/uapi/linux/devlink.h | 26 + net/core/devlink.c | 266 ++++++++- 35 files changed, 2834 insertions(+), 50 deletions(-) create mode 100644 Documentation/networking/devlink/devlink-port.rst create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/sf/cmd.c create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/sf/dev/dev.c create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/sf/dev/dev.h create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/sf/dev/driver.c create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/sf/devlink.c create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/sf/hw_table.c create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/sf/mlx5_ifc_vhca_event.h create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/sf/priv.h create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/sf/sf.h create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/sf/vhca_event.c create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/sf/vhca_event.h
Signature · 1 line
2.26.2
PATCH v4 11/15 net/mlx5: SF, Add port add delete functionality
From: Parav Pandit <[email protected]>
To handle SF port management outside of the eswitch as independent software layer, introduce eswitch notifier APIs so that upper layer who wish to support sf port management in switchdev mode can perform its task whenever eswitch mode is set to switchdev or before eswitch is disabled.
Initialize sf port table on such eswitch event.
Add SF port add and delete functionality in switchdev mode. Destroy all SF ports when eswitch is disabled. Expose SF port add and delete to user via devlink commands.
$ devlink dev eswitch set pci/0000:06:00.0 mode switchdev
$ devlink port show pci/0000:06:00.0/65535: type eth netdev ens2f0np0 flavour physical port 0 splittable false
$ devlink port add pci/0000:06:00.0 flavour pcisf pfnum 0 sfnum 88
$ devlink port show ens2f0npf0sf88
pci/0000:06:00.0/32768: type eth netdev ens2f0npf0sf88 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false
function:
hw_addr 00:00:00:00:88:88 state inactive opstate detached$ devlink port show ens2f0npf0sf88 -jp
{
"port": {
"pci/0000:06:00.0/32768": {
"type": "eth",
"netdev": "ens2f0npf0sf88",
"flavour": "pcisf",
"controller": 0,
"pfnum": 0,
"sfnum": 88,
"external": false,
"splittable": false,
"function": {
"hw_addr": "00:00:00:00:88:88",
"state": "inactive",
"opstate": "detached"
}
}
}
}--- .../net/ethernet/mellanox/mlx5/core/Makefile | 5 + drivers/net/ethernet/mellanox/mlx5/core/cmd.c | 4 + .../net/ethernet/mellanox/mlx5/core/devlink.c | 5 + .../net/ethernet/mellanox/mlx5/core/eswitch.c | 25 ++ .../net/ethernet/mellanox/mlx5/core/eswitch.h | 12 + .../net/ethernet/mellanox/mlx5/core/main.c | 18 + .../net/ethernet/mellanox/mlx5/core/sf/cmd.c | 27 ++ .../ethernet/mellanox/mlx5/core/sf/devlink.c | 312 ++++++++++++++++++ .../ethernet/mellanox/mlx5/core/sf/hw_table.c | 125 +++++++ .../net/ethernet/mellanox/mlx5/core/sf/priv.h | 17 + .../net/ethernet/mellanox/mlx5/core/sf/sf.h | 28 ++ include/linux/mlx5/driver.h | 6 + 12 files changed, 584 insertions(+) create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/sf/cmd.c create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/sf/devlink.c create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/sf/hw_table.c create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/sf/priv.h
drivers/net/ethernet/mellanox/mlx5/core/Makefile +5 -0
@@ -89,3 +89,8 @@ mlx5_core-$(CONFIG_MLX5_SW_STEERING) += steering/dr_domain.o steering/dr_table.o # SF device # mlx5_core-$(CONFIG_MLX5_SF) += sf/vhca_event.o sf/dev/dev.o sf/dev/driver.o + +# +# SF manager +# +mlx5_core-$(CONFIG_MLX5_SF_MANAGER) += sf/cmd.o sf/hw_table.o sf/devlink.o
drivers/net/ethernet/mellanox/mlx5/core/cmd.c +4 -0
@@ -333,6 +333,7 @@ static int mlx5_internal_err_ret_value(struct mlx5_core_dev *dev, u16 op,case MLX5_CMD_OP_DEALLOC_MEMIC :case MLX5_CMD_OP_PAGE_FAULT_RESUME :case MLX5_CMD_OP_QUERY_ESW_FUNCTIONS : +case MLX5_CMD_OP_DEALLOC_SF :return MLX5_CMD_STAT_OK ;case MLX5_CMD_OP_QUERY_HCA_CAP :
@@ -466,6 +467,7 @@ static int mlx5_internal_err_ret_value(struct mlx5_core_dev *dev, u16 op,case MLX5_CMD_OP_RELEASE_XRQ_ERROR :case MLX5_CMD_OP_QUERY_VHCA_STATE :case MLX5_CMD_OP_MODIFY_VHCA_STATE : +case MLX5_CMD_OP_ALLOC_SF : *status =MLX5_DRIVER_STATUS_ABORTED ; *synd =MLX5_DRIVER_SYND ;return -EIO ;
@@ -661,6 +663,8 @@ const char *mlx5_command_str(int command)MLX5_COMMAND_STR_CASE (MODIFY_XRQ );MLX5_COMMAND_STR_CASE (QUERY_VHCA_STATE );MLX5_COMMAND_STR_CASE (MODIFY_VHCA_STATE ); +MLX5_COMMAND_STR_CASE (ALLOC_SF ); +MLX5_COMMAND_STR_CASE (DEALLOC_SF );default :return "unknown command opcode" ; } }
drivers/net/ethernet/mellanox/mlx5/core/devlink.c +5 -0
@@ -8,6 +8,7 @@#include "fs_core.h" #include "eswitch.h" #include "sf/dev/dev.h" +#include "sf/sf.h" static int mlx5_devlink_flash_update(struct devlink *devlink, struct devlink_flash_update_params *params,
@@ -190,6 +191,10 @@ static const struct devlink_ops mlx5_devlink_ops = { .eswitch_encap_mode_get = mlx5_devlink_eswitch_encap_mode_get, .port_function_hw_addr_get = mlx5_devlink_port_function_hw_addr_get, .port_function_hw_addr_set = mlx5_devlink_port_function_hw_addr_set, +#endif +#ifdef CONFIG_MLX5_SF_MANAGER + .port_new = mlx5_devlink_sf_port_new, + .port_del = mlx5_devlink_sf_port_del, #endif .flash_update = mlx5_devlink_flash_update, .info_get = mlx5_devlink_info_get,
drivers/net/ethernet/mellanox/mlx5/core/eswitch.c +25 -0
@@ -1600,6 +1600,15 @@ mlx5_eswitch_update_num_of_vfs(struct mlx5_eswitch *esw, int num_vfs)kvfree (out); } +static void mlx5_esw_mode_change_notify (struct mlx5_eswitch *esw,u16 mode) +{ +struct mlx5_esw_event_info info = {}; + + info.new_mode = mode; + +blocking_notifier_call_chain (&esw->n_head,0 , &info); +} +/** * mlx5_eswitch_enable_locked - Enable eswitch * @esw: Pointer to eswitch
@@ -1660,6 +1669,8 @@ int mlx5_eswitch_enable_locked(struct mlx5_eswitch *esw, int mode, int num_vfs) mode ==MLX5_ESWITCH_LEGACY ?"LEGACY" :"OFFLOADS" , esw->esw_funcs.num_vfs, esw->enabled_vports); +mlx5_esw_mode_change_notify (esw, mode); +return 0 ; abort:
@@ -1716,6 +1727,11 @@ void mlx5_eswitch_disable_locked(struct mlx5_eswitch *esw, bool clear_vf) esw->mode ==MLX5_ESWITCH_LEGACY ?"LEGACY" :"OFFLOADS" , esw->esw_funcs.num_vfs, esw->enabled_vports); +/* Notify eswitch users that it is exiting from current mode. +* So that it can do necessary cleanup before the eswitch is disabled. +*/ +mlx5_esw_mode_change_notify (esw,MLX5_ESWITCH_NONE ); +mlx5_eswitch_event_handlers_unregister (esw);if (esw->mode ==MLX5_ESWITCH_LEGACY )
@@ -1816,6 +1832,7 @@ int mlx5_eswitch_init(struct mlx5_core_dev *dev) esw->offloads.inline_mode =MLX5_INLINE_MODE_NONE ; dev->priv.eswitch = esw; +BLOCKING_INIT_NOTIFIER_HEAD (&esw->n_head);return 0 ; abort:if (esw->work_queue)
@@ -2507,4 +2524,12 @@ bool mlx5_esw_multipath_prereq(struct mlx5_core_dev *dev0, dev1->priv.eswitch->mode == MLX5_ESWITCH_OFFLOADS); } +int mlx5_esw_event_notifier_register (struct mlx5_eswitch *esw,struct notifier_block *nb) +{ +return blocking_notifier_chain_register (&esw->n_head, nb); +} +void mlx5_esw_event_notifier_unregister (struct mlx5_eswitch *esw,struct notifier_block *nb) +{ +blocking_notifier_chain_unregister (&esw->n_head, nb); +}
drivers/net/ethernet/mellanox/mlx5/core/eswitch.h +12 -0
@@ -278,6 +278,7 @@ struct mlx5_eswitch {struct {u32 large_group_num; } params; +struct blocking_notifier_head n_head; };void esw_offloads_disable (struct mlx5_eswitch *esw);
@@ -733,6 +734,17 @@ int mlx5_esw_offloads_sf_vport_enable(struct mlx5_eswitch *esw, struct devlink_p u16 vport_num, u32 sfnum);void mlx5_esw_offloads_sf_vport_disable (struct mlx5_eswitch *esw,u16 vport_num); +/** +* mlx5_esw_event_info - Indicates eswitch mode changed/changing. +* +* @new_mode: New mode of eswitch. +*/ +struct mlx5_esw_event_info { +u16 new_mode; +}; + +int mlx5_esw_event_notifier_register (struct mlx5_eswitch *esw,struct notifier_block *n); +void mlx5_esw_event_notifier_unregister (struct mlx5_eswitch *esw,struct notifier_block *n);#else /* CONFIG_MLX5_ESWITCH */ /* eswitch API stubs */ static inline int mlx5_eswitch_init (struct mlx5_core_dev *dev) {return 0 ; }
drivers/net/ethernet/mellanox/mlx5/core/main.c +18 -0
@@ -893,6 +893,18 @@ static int mlx5_init_once(struct mlx5_core_dev *dev) goto err_fpga_cleanup; } + err =mlx5_sf_hw_table_init (dev); +if (err) { +mlx5_core_err (dev,"Failed to init SF HW table %d\n" , err); + goto err_sf_hw_table_cleanup; + } + + err =mlx5_sf_table_init (dev); +if (err) { +mlx5_core_err (dev,"Failed to init SF table %d\n" , err); + goto err_sf_table_cleanup; + } + dev->dm =mlx5_dm_create (dev);if (IS_ERR (dev->dm))mlx5_core_warn (dev,"Failed to init device memory%d\n" , err);
@@ -903,6 +915,10 @@ static int mlx5_init_once(struct mlx5_core_dev *dev)return 0 ; +err_sf_table_cleanup: +mlx5_sf_hw_table_cleanup (dev); +err_sf_hw_table_cleanup: +mlx5_vhca_event_cleanup (dev); err_fpga_cleanup:mlx5_fpga_cleanup (dev); err_eswitch_cleanup:
@@ -936,6 +952,8 @@ static void mlx5_cleanup_once(struct mlx5_core_dev *dev)mlx5_hv_vhca_destroy (dev->hv_vhca);mlx5_fw_tracer_destroy (dev->tracer);mlx5_dm_cleanup (dev); +mlx5_sf_table_cleanup (dev); +mlx5_sf_hw_table_cleanup (dev);mlx5_vhca_event_cleanup (dev);mlx5_fpga_cleanup (dev);mlx5_eswitch_cleanup (dev->priv.eswitch);
drivers/net/ethernet/mellanox/mlx5/core/sf/cmd.c new file +27 -0
@@ -0,0 +1,27 @@ +// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB +/* Copyright (c) 2020 Mellanox Technologies Ltd */ + +#include <linux/mlx5/driver.h> +#include "priv.h" + +int mlx5_cmd_alloc_sf (struct mlx5_core_dev *dev,u16 function_id) +{ +u32 out[MLX5_ST_SZ_DW (alloc_sf_out)] = {}; +u32 in[MLX5_ST_SZ_DW (alloc_sf_in)] = {}; + +MLX5_SET (alloc_sf_in, in, opcode,MLX5_CMD_OP_ALLOC_SF ); +MLX5_SET (alloc_sf_in, in, function_id, function_id); + +return mlx5_cmd_exec (dev, in,sizeof (in), out,sizeof (out)); +} + +int mlx5_cmd_dealloc_sf (struct mlx5_core_dev *dev,u16 function_id) +{ +u32 out[MLX5_ST_SZ_DW (dealloc_sf_out)] = {}; +u32 in[MLX5_ST_SZ_DW (dealloc_sf_in)] = {}; + +MLX5_SET (dealloc_sf_in, in, opcode,MLX5_CMD_OP_DEALLOC_SF ); +MLX5_SET (dealloc_sf_in, in, function_id, function_id); + +return mlx5_cmd_exec (dev, in,sizeof (in), out,sizeof (out)); +}
drivers/net/ethernet/mellanox/mlx5/core/sf/devlink.c new file +312 -0
312 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/sf/hw_table.c new file +125 -0
125 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/sf/priv.h new file +17 -0
@@ -0,0 +1,17 @@ +/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */ +/* Copyright (c) 2020 Mellanox Technologies Ltd */ + +#ifndef __MLX5_SF_PRIV_H__ +#define __MLX5_SF_PRIV_H__ + +#include <linux/mlx5/driver.h> + +int mlx5_cmd_alloc_sf (struct mlx5_core_dev *dev,u16 function_id); +int mlx5_cmd_dealloc_sf (struct mlx5_core_dev *dev,u16 function_id); + +u16 mlx5_sf_sw_to_hw_id (const struct mlx5_core_dev *dev,u16 sw_id); + +int mlx5_sf_hw_table_sf_alloc (struct mlx5_core_dev *dev,u32 usr_sfnum); +void mlx5_sf_hw_table_sf_free (struct mlx5_core_dev *dev,u16 id); + +#endif
drivers/net/ethernet/mellanox/mlx5/core/sf/sf.h +28 -0
@@ -28,6 +28,16 @@ static inline u16 mlx5_sf_max_functions(const struct mlx5_core_dev *dev) return 1 << MLX5_CAP_GEN(dev, log_max_sf); } +int mlx5_sf_hw_table_init (struct mlx5_core_dev *dev); +void mlx5_sf_hw_table_cleanup (struct mlx5_core_dev *dev); + +int mlx5_sf_table_init (struct mlx5_core_dev *dev); +void mlx5_sf_table_cleanup (struct mlx5_core_dev *dev); + +int mlx5_devlink_sf_port_new (struct devlink *devlink,const struct devlink_port_new_attrs *add_attr, +struct netlink_ext_ack *extack); +int mlx5_devlink_sf_port_del (struct devlink *devlink,unsigned int port_index, +struct netlink_ext_ack *extack);#else static inline bool mlx5_sf_supported (const struct mlx5_core_dev *dev)
@@ -40,6 +50,24 @@ static inline u16 mlx5_sf_max_functions(const struct mlx5_core_dev *dev)return 0 ; } +static inline int mlx5_sf_hw_table_init (struct mlx5_core_dev *dev) +{ +return 0 ; +} + +static inline void mlx5_sf_hw_table_cleanup (struct mlx5_core_dev *dev) +{ +} + +static inline int mlx5_sf_table_init (struct mlx5_core_dev *dev) +{ +return 0 ; +} + +static inline void mlx5_sf_table_cleanup (struct mlx5_core_dev *dev) +{ +} +#endif #endif
include/linux/mlx5/driver.h +6 -0
@@ -510,6 +510,8 @@ struct mlx5_eq_table;struct mlx5_irq_table ;struct mlx5_vhca_state_notifier ;struct mlx5_sf_dev_table ; +struct mlx5_sf_hw_table ; +struct mlx5_sf_table ;struct mlx5_rate_limit {u32 rate;
@@ -611,6 +613,10 @@ struct mlx5_priv {struct mlx5_sf_dev_table *sf_dev_table;struct mlx5_core_dev *parent_mdev;#endif +#ifdef CONFIG_MLX5_SF_MANAGER +struct mlx5_sf_hw_table *sf_hw_table; +struct mlx5_sf_table *sf_table; +#endif };enum mlx5_device_state {
Signature · 1 line
2.26.2
PATCH v4 8/15 net/mlx5: SF, Add auxiliary device driver
In reply to Saeed MahameedFrom: Parav Pandit <[email protected]>
Add auxiliary device driver for mlx5 subfunction auxiliary device.
A mlx5 subfunction is similar to PCI PF and VF. For a subfunction an auxiliary device is created.
As a result, when mlx5 SF auxiliary device binds to the driver, its netdev and rdma device are created, they appear as
$ ls -l /sys/bus/auxiliary/devices/ mlx5_core.sf.4 -> ../../../devices/pci0000:00/0000:00:03.0/0000:06:00.0/mlx5_core.sf.4
$ ls -l /sys/class/net/eth1/device /sys/class/net/eth1/device -> ../../../mlx5_core.sf.4
$ cat /sys/bus/auxiliary/devices/mlx5_core.sf.4/sfnum 88
$ devlink dev show pci/0000:06:00.0 auxiliary/mlx5_core.sf.4
$ devlink port show auxiliary/mlx5_core.sf.4/1 auxiliary/mlx5_core.sf.4/1: type eth netdev p0sf88 flavour virtual port 0 splittable false
$ rdma link show mlx5_0/1 link mlx5_0/1 state ACTIVE physical_state LINK_UP netdev p0sf88
$ rdma dev show 8: rocep6s0f1: node_type ca fw 16.29.0550 node_guid 248a:0703:00b3:d113 sys_image_guid 248a:0703:00b3:d112 13: mlx5_0: node_type ca fw 16.29.0550 node_guid 0000:00ff:fe00:8888 sys_image_guid 248a:0703:00b3:d112
In future, devlink device instance name will adapt to have sfnum annotation using either an alias or as devlink instance name described in RFC [1].
[1] https://lore.kernel.org/netdev/20200519092258.GF4655@nanopsycho/
--- .../net/ethernet/mellanox/mlx5/core/Makefile | 2 +- .../net/ethernet/mellanox/mlx5/core/devlink.c | 12 +++ drivers/net/ethernet/mellanox/mlx5/core/eq.c | 2 +- .../net/ethernet/mellanox/mlx5/core/main.c | 12 ++- .../ethernet/mellanox/mlx5/core/mlx5_core.h | 10 ++ .../net/ethernet/mellanox/mlx5/core/pci_irq.c | 20 ++++ .../ethernet/mellanox/mlx5/core/sf/dev/dev.c | 10 ++ .../ethernet/mellanox/mlx5/core/sf/dev/dev.h | 20 ++++ .../mellanox/mlx5/core/sf/dev/driver.c | 101 ++++++++++++++++++ include/linux/mlx5/driver.h | 4 +- 10 files changed, 187 insertions(+), 6 deletions(-) create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/sf/dev/driver.c
drivers/net/ethernet/mellanox/mlx5/core/Makefile +1 -1
@@ -88,4 +88,4 @@ mlx5_core-$(CONFIG_MLX5_SW_STEERING) += steering/dr_domain.o steering/dr_table.o # # SF device # -mlx5_core-$(CONFIG_MLX5_SF) += sf/vhca_event.o sf/dev/dev.o +mlx5_core-$(CONFIG_MLX5_SF) += sf/vhca_event.o sf/dev/dev.o sf/dev/driver.o
drivers/net/ethernet/mellanox/mlx5/core/devlink.c +12 -0
@@ -7,6 +7,7 @@#include "fw_reset.h" #include "fs_core.h" #include "eswitch.h" +#include "sf/dev/dev.h" static int mlx5_devlink_flash_update(struct devlink *devlink, struct devlink_flash_update_params *params,
@@ -127,6 +128,17 @@ static int mlx5_devlink_reload_down(struct devlink *devlink, bool netns_change, struct netlink_ext_ack *extack) {struct mlx5_core_dev *dev =devlink_priv (devlink); +bool sf_dev_allocated; + + sf_dev_allocated =mlx5_sf_dev_allocated (dev); +if (sf_dev_allocated) { +/* Reload results in deleting SF device which further results in +* unregistering devlink instance while holding devlink_mutext. +* Hence, do not support reload. +*/ +NL_SET_ERR_MSG_MOD (extack,"reload is unsupported when SFs are allocated\n" ); +return -EOPNOTSUPP ; + }switch (action) {case DEVLINK_RELOAD_ACTION_DRIVER_REINIT :
drivers/net/ethernet/mellanox/mlx5/core/eq.c +1 -1
@@ -467,7 +467,7 @@ int mlx5_eq_table_init(struct mlx5_core_dev *dev)for (i =0 ; i <MLX5_EVENT_TYPE_MAX ; i++)ATOMIC_INIT_NOTIFIER_HEAD (&eq_table->nh[i]); - eq_table->irq_table = dev->priv.irq_table; + eq_table->irq_table =mlx5_irq_table_get (dev);return 0 ; }
drivers/net/ethernet/mellanox/mlx5/core/main.c +9 -3
@@ -84,7 +84,6 @@ unsigned int mlx5_core_debug_mask;module_param_named (debug_mask, mlx5_core_debug_mask, uint,0644 );MODULE_PARM_DESC (debug_mask,"debug mask: 1 = dump cmd data, 2 = dump cmd exec time, 3 = both. Default=0" ); -#define MLX5_DEFAULT_PROF 2static unsigned int prof_sel =MLX5_DEFAULT_PROF ;module_param_named (prof_sel, prof_sel, uint,0444 );MODULE_PARM_DESC (prof_sel,"profile selector. Valid range 0 - 2" );
@@ -1303,7 +1302,7 @@ void mlx5_unload_one(struct mlx5_core_dev *dev, bool cleanup)mutex_unlock (&dev->intf_state_mutex); } -static int mlx5_mdev_init (struct mlx5_core_dev *dev,int profile_idx) +int mlx5_mdev_init (struct mlx5_core_dev *dev,int profile_idx) {struct mlx5_priv *priv = &dev->priv;int err;
@@ -1353,7 +1352,7 @@ static int mlx5_mdev_init(struct mlx5_core_dev *dev, int profile_idx)return err; } -static void mlx5_mdev_uninit (struct mlx5_core_dev *dev) +void mlx5_mdev_uninit (struct mlx5_core_dev *dev) {struct mlx5_priv *priv = &dev->priv;
@@ -1693,6 +1692,10 @@ static int __init init(void)if (err) goto err_debug; + err =mlx5_sf_driver_register (); +if (err) + goto err_sf; +#ifdef CONFIG_MLX5_CORE_EN err =mlx5e_init ();if (err) {
@@ -1703,6 +1706,8 @@ static int __init init(void)return 0 ; +err_sf: +pci_unregister_driver (&mlx5_core_driver); err_debug:mlx5_unregister_debugfs ();return err;
@@ -1713,6 +1718,7 @@ static void __exit cleanup(void)#ifdef CONFIG_MLX5_CORE_EN mlx5e_cleanup ();#endif +mlx5_sf_driver_unregister ();pci_unregister_driver (&mlx5_core_driver);mlx5_unregister_debugfs (); }
drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h +10 -0
@@ -117,6 +117,8 @@ enum mlx5_semaphore_space_address { MLX5_SEMAPHORE_SW_RESET = 0x20, }; +#define MLX5_DEFAULT_PROF 2 +int mlx5_query_hca_caps (struct mlx5_core_dev *dev);int mlx5_query_board_id (struct mlx5_core_dev *dev);int mlx5_cmd_init (struct mlx5_core_dev *dev);
@@ -176,6 +178,7 @@ struct cpumask * mlx5_irq_get_affinity_mask(struct mlx5_irq_table *irq_table, int vecidx);struct cpu_rmap *mlx5_irq_get_rmap (struct mlx5_irq_table *table);int mlx5_irq_get_num_comp (struct mlx5_irq_table *table); +struct mlx5_irq_table *mlx5_irq_table_get (struct mlx5_core_dev *dev);int mlx5_events_init (struct mlx5_core_dev *dev);void mlx5_events_cleanup (struct mlx5_core_dev *dev);
@@ -257,6 +260,13 @@ enum {u8 mlx5_get_nic_state (struct mlx5_core_dev *dev);void mlx5_set_nic_state (struct mlx5_core_dev *dev,u8 state); +static inline bool mlx5_core_is_sf (const struct mlx5_core_dev *dev) +{ +return dev->coredev_type ==MLX5_COREDEV_SF ; +} + +int mlx5_mdev_init (struct mlx5_core_dev *dev,int profile_idx); +void mlx5_mdev_uninit (struct mlx5_core_dev *dev);void mlx5_unload_one (struct mlx5_core_dev *dev,bool cleanup);int mlx5_load_one (struct mlx5_core_dev *dev,bool boot);
drivers/net/ethernet/mellanox/mlx5/core/pci_irq.c +20 -0
@@ -30,6 +30,9 @@ int mlx5_irq_table_init(struct mlx5_core_dev *dev) {struct mlx5_irq_table *irq_table; +if (mlx5_core_is_sf (dev)) +return 0 ; + irq_table =kvzalloc (sizeof (*irq_table),GFP_KERNEL );if (!irq_table)return -ENOMEM ;
@@ -40,6 +43,9 @@ int mlx5_irq_table_init(struct mlx5_core_dev *dev)void mlx5_irq_table_cleanup (struct mlx5_core_dev *dev) { +if (mlx5_core_is_sf (dev)) +return ; +kvfree (dev->priv.irq_table); }
@@ -268,6 +274,9 @@ int mlx5_irq_table_create(struct mlx5_core_dev *dev)int nvec;int err; +if (mlx5_core_is_sf (dev)) +return 0 ; + nvec =MLX5_CAP_GEN (dev, num_ports) *num_online_cpus () +MLX5_IRQ_VEC_COMP_BASE ; nvec =min_t (int, nvec, num_eqs);
@@ -319,6 +328,9 @@ void mlx5_irq_table_destroy(struct mlx5_core_dev *dev)struct mlx5_irq_table *table = dev->priv.irq_table;int i; +if (mlx5_core_is_sf (dev)) +return ; +/* free_irq requires that affinity and rmap will be cleared * before calling it. This is why there is asymmetry with set_rmap * which should be called after alloc_irq but before request_irq.
@@ -332,3 +344,11 @@ void mlx5_irq_table_destroy(struct mlx5_core_dev *dev)kfree (table->irq); } +struct mlx5_irq_table *mlx5_irq_table_get (struct mlx5_core_dev *dev) +{ +#ifdef CONFIG_MLX5_SF +if (mlx5_core_is_sf (dev)) +return dev->priv.parent_mdev->priv.irq_table; +#endif +return dev->priv.irq_table; +}
drivers/net/ethernet/mellanox/mlx5/core/sf/dev/dev.c +10 -0
@@ -24,6 +24,16 @@ static bool mlx5_sf_dev_supported(const struct mlx5_core_dev *dev)return MLX5_CAP_GEN (dev, sf) &&mlx5_vhca_event_supported (dev); } +bool mlx5_sf_dev_allocated (const struct mlx5_core_dev *dev) +{ +struct mlx5_sf_dev_table *table = dev->priv.sf_dev_table; + +if (!mlx5_sf_dev_supported (dev)) +return false; + +return xa_empty (&table->devices); +} +static ssize_t sfnum_show (struct device *dev,struct device_attribute *attr,char *buf) { struct auxiliary_device *adev = container_of(dev, struct auxiliary_device, dev);
drivers/net/ethernet/mellanox/mlx5/core/sf/dev/dev.h +20 -0
@@ -13,6 +13,7 @@struct mlx5_sf_dev {struct auxiliary_device adev;struct mlx5_core_dev *parent_mdev; +struct mlx5_core_dev *mdev;phys_addr_t bar_base_addr;u32 sfnum; };
@@ -20,6 +21,11 @@ struct mlx5_sf_dev {void mlx5_sf_dev_table_create (struct mlx5_core_dev *dev);void mlx5_sf_dev_table_destroy (struct mlx5_core_dev *dev); +int mlx5_sf_driver_register (void ); +void mlx5_sf_driver_unregister (void ); + +bool mlx5_sf_dev_allocated (const struct mlx5_core_dev *dev); +#else static inline void mlx5_sf_dev_table_create (struct mlx5_core_dev *dev)
@@ -30,6 +36,20 @@ static inline void mlx5_sf_dev_table_destroy(struct mlx5_core_dev *dev) { } +static inline int mlx5_sf_driver_register (void ) +{ +return 0 ; +} + +static inline void mlx5_sf_driver_unregister (void ) +{ +} + +static inline bool mlx5_sf_dev_allocated (const struct mlx5_core_dev *dev) +{ +return 0 ; +} +#endif #endif
drivers/net/ethernet/mellanox/mlx5/core/sf/dev/driver.c new file +101 -0
101 lines not on this page · raw message
include/linux/mlx5/driver.h +3 -1
@@ -193,7 +193,8 @@ enum port_state_policy {enum mlx5_coredev_type {MLX5_COREDEV_PF , -MLX5_COREDEV_VF +MLX5_COREDEV_VF , +MLX5_COREDEV_SF , };struct mlx5_field_desc {
@@ -608,6 +609,7 @@ struct mlx5_priv {#ifdef CONFIG_MLX5_SF struct mlx5_vhca_state_notifier *vhca_state_notifier;struct mlx5_sf_dev_table *sf_dev_table; +struct mlx5_core_dev *parent_mdev;#endif };
Signature · 1 line
2.26.2
PATCH v4 5/15 devlink: Support get and set state of port function
In reply to Saeed MahameedFrom: Parav Pandit <[email protected]>
devlink port function can be in active or inactive state. Allow users to get and set port function's state.
When the port function it activated, its operational state may change after a while when the device is created and driver binds to it. Similarly on deactivation flow.
To clearly describe the state of the port function and its device's operational state in the host system, define state and opstate attributes.
Example of a PCI SF port which supports a port function:
Create a device with ID=10 and one physical port.
$ devlink dev eswitch set pci/0000:06:00.0 mode switchdev
$ devlink port show pci/0000:06:00.0/65535: type eth netdev ens2f0np0 flavour physical port 0 splittable false
$ devlink port add pci/0000:06:00.0 flavour pcisf pfnum 0 sfnum 88
$ devlink port show pci/0000:06:00.0/32768
pci/0000:06:00.0/32768: type eth netdev ens2f0npf0sf88 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false
function:
hw_addr 00:00:00:00:88:88 state inactive opstate detached$ devlink port function set pci/0000:06:00.0/32768 hw_addr 00:00:00:00:88:88 state active
$ devlink port show pci/0000:06:00.0/32768 -jp
{
"port": {
"pci/0000:06:00.0/32768": {
"type": "eth",
"netdev": "ens2f0npf0sf88",
"flavour": "pcisf",
"controller": 0,
"pfnum": 0,
"sfnum": 88,
"external": false,
"splittable": false,
"function": {
"hw_addr": "00:00:00:00:88:88",
"state": "active",
"opstate": "attached"
}
}
}
}--- include/net/devlink.h | 23 +++++++++ include/uapi/linux/devlink.h | 21 +++++++++ net/core/devlink.c | 90 +++++++++++++++++++++++++++++++++++- 3 files changed, 133 insertions(+), 1 deletion(-)
include/net/devlink.h +23 -0
@@ -1374,6 +1374,29 @@ struct devlink_ops {int (*port_function_hw_addr_set)(struct devlink *devlink,struct devlink_port *port,const u8 *hw_addr,int hw_addr_len,struct netlink_ext_ack *extack); +/** +* @port_function_state_get: Port function's state get function. +* +* Should be used by device drivers to report the state of a function +* managed by the devlink port. Driver should return -EOPNOTSUPP if it +* doesn't support port function handling for a particular port. +*/ +int (*port_function_state_get)(struct devlink *devlink, +struct devlink_port *port, +enum devlink_port_function_state *state, +enum devlink_port_function_opstate *opstate, +struct netlink_ext_ack *extack); +/** +* @port_function_state_set: Port function's state set function. +* +* Should be used by device drivers to set the state of a function +* managed by the devlink port. Driver should return -EOPNOTSUPP if it +* doesn't support port function handling for a particular port. +*/ +int (*port_function_state_set)(struct devlink *devlink, +struct devlink_port *port, +enum devlink_port_function_state state, +struct netlink_ext_ack *extack);/** * @port_new: Port add function. *
include/uapi/linux/devlink.h +21 -0
@@ -583,9 +583,30 @@ enum devlink_resource_unit {enum devlink_port_function_attr {DEVLINK_PORT_FUNCTION_ATTR_UNSPEC ,DEVLINK_PORT_FUNCTION_ATTR_HW_ADDR ,/* binary */ +DEVLINK_PORT_FUNCTION_ATTR_STATE ,/* u8 */ +DEVLINK_PORT_FUNCTION_ATTR_OPSTATE ,/* u8 */ __DEVLINK_PORT_FUNCTION_ATTR_MAX,DEVLINK_PORT_FUNCTION_ATTR_MAX = __DEVLINK_PORT_FUNCTION_ATTR_MAX -1 }; +enum devlink_port_function_state { +DEVLINK_PORT_FUNCTION_STATE_INACTIVE , +DEVLINK_PORT_FUNCTION_STATE_ACTIVE , +}; + +/** +* enum devlink_port_function_opstate - indicates operational state of port function +* @DEVLINK_PORT_FUNCTION_OPSTATE_ATTACHED: Driver is attached to the function of port, for +* gracefufl tear down of the function, after +* inactivation of the port function, user should wait +* for operational state to turn DETACHED. +* @DEVLINK_PORT_FUNCTION_OPSTATE_DETACHED: Driver is detached from the function of port; it is +* safe to delete the port. +*/ +enum devlink_port_function_opstate { +DEVLINK_PORT_FUNCTION_OPSTATE_DETACHED , +DEVLINK_PORT_FUNCTION_OPSTATE_ATTACHED , +}; +#endif /* _UAPI_LINUX_DEVLINK_H_ */
net/core/devlink.c +89 -1
121 lines not on this page · raw message
Signature · 1 line
2.26.2
PATCH v4 9/15 net/mlx5: E-switch, Prepare eswitch to handle SF vport
In reply to Saeed MahameedFrom: Vu Pham <[email protected]>
Prepare eswitch to handle SF vport during
(a) querying eswitch functions
(b) egress ACL creation
(c) account for SF vports in total vports calculation
Assign a dedicated placeholder for SFs vports and their representors. They are placed after VFs vports and before ECPF vports as below:
[PF,VF0,...,VFn,SF0,...SFm,ECPF,UPLINK].
Change functions to map SF's vport numbers to indices when accessing the vports or representors arrays, and vice versa.
--- .../net/ethernet/mellanox/mlx5/core/Kconfig | 10 ++++ .../mellanox/mlx5/core/esw/acl/egress_ofld.c | 2 +- .../net/ethernet/mellanox/mlx5/core/eswitch.c | 11 +++- .../net/ethernet/mellanox/mlx5/core/eswitch.h | 50 +++++++++++++++++++ .../mellanox/mlx5/core/eswitch_offloads.c | 11 ++++ .../net/ethernet/mellanox/mlx5/core/vport.c | 3 +- 6 files changed, 83 insertions(+), 4 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/Kconfig +10 -0
@@ -212,3 +212,13 @@ config MLX5_SF Build support for subfuction device in the NIC. A Mellanox subfunction device can support RDMA, netdevice and vdpa device. It is similar to a SRIOV VF but it doesn't require SRIOV support. + +config MLX5_SF_MANAGER + bool + depends on MLX5_SF && MLX5_ESWITCH + default y + help + Build support for subfuction port in the NIC. A Mellanox subfunction + port is managed through devlink. A subfunction supports RDMA, netdevice + and vdpa device. It is similar to a SRIOV VF but it doesn't require + SRIOV support.
drivers/net/ethernet/mellanox/mlx5/core/esw/acl/egress_ofld.c +1 -1
@@ -150,7 +150,7 @@ static void esw_acl_egress_ofld_groups_destroy(struct mlx5_vport *vport)static bool esw_acl_egress_needed (const struct mlx5_eswitch *esw,u16 vport_num) { -return mlx5_eswitch_is_vf_vport (esw, vport_num); +return mlx5_eswitch_is_vf_vport (esw, vport_num) ||mlx5_esw_is_sf_vport (esw, vport_num); }int esw_acl_egress_ofld_setup (struct mlx5_eswitch *esw,struct mlx5_vport *vport)
drivers/net/ethernet/mellanox/mlx5/core/eswitch.c +9 -2
@@ -1366,9 +1366,15 @@ const u32 *mlx5_esw_query_functions(struct mlx5_core_dev *dev) {int outlen =MLX5_ST_SZ_BYTES (query_esw_functions_out);u32 in[MLX5_ST_SZ_DW (query_esw_functions_in)] = {}; +u16 max_sf_vports;u32 *out;int err; + max_sf_vports =mlx5_sf_max_functions (dev); +/* Device interface is array of 64-bits */ +if (max_sf_vports) + outlen +=DIV_ROUND_UP (max_sf_vports,BITS_PER_TYPE (__be64)) *sizeof (__be64); + out =kvzalloc (outlen,GFP_KERNEL );if (!out)return ERR_PTR (-ENOMEM );
@@ -1376,7 +1382,7 @@ const u32 *mlx5_esw_query_functions(struct mlx5_core_dev *dev)MLX5_SET (query_esw_functions_in, in, opcode,MLX5_CMD_OP_QUERY_ESW_FUNCTIONS ); - err =mlx5_cmd_exec_inout (dev, query_esw_functions, in, out); + err =mlx5_cmd_exec (dev, in,sizeof (in), out, outlen);if (!err)return out;
@@ -1899,7 +1905,8 @@ static bool is_port_function_supported(const struct mlx5_eswitch *esw, u16 vport_num) {return vport_num ==MLX5_VPORT_PF || -mlx5_eswitch_is_vf_vport (esw, vport_num); +mlx5_eswitch_is_vf_vport (esw, vport_num) || +mlx5_esw_is_sf_vport (esw, vport_num); } int mlx5_devlink_port_function_hw_addr_get(struct devlink *devlink,
drivers/net/ethernet/mellanox/mlx5/core/eswitch.h +50 -0
80 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +11 -0
@@ -1800,11 +1800,22 @@ static void __esw_offloads_unload_rep(struct mlx5_eswitch *esw, esw->offloads.rep_ops[rep_type]->unload (rep); } +static void __unload_reps_sf_vport (struct mlx5_eswitch *esw,u8 rep_type) +{ +struct mlx5_eswitch_rep *rep; +int i; + +mlx5_esw_for_each_sf_rep (esw, i, rep) +__esw_offloads_unload_rep (esw, rep, rep_type); +} +static void __unload_reps_all_vport (struct mlx5_eswitch *esw,u8 rep_type) {struct mlx5_eswitch_rep *rep;int i; +__unload_reps_sf_vport (esw, rep_type); +mlx5_esw_for_each_vf_rep_reverse (esw, i, rep, esw->esw_funcs.num_vfs)__esw_offloads_unload_rep (esw, rep, rep_type);
drivers/net/ethernet/mellanox/mlx5/core/vport.c +2 -1
@@ -36,6 +36,7 @@#include <linux/mlx5/vport.h> #include <linux/mlx5/eswitch.h> #include "mlx5_core.h" +#include "sf/sf.h" /* Mutex to hold while enabling or disabling RoCE */ static DEFINE_MUTEX (mlx5_roce_en_lock);
@@ -1160,6 +1161,6 @@ EXPORT_SYMBOL_GPL(mlx5_query_nic_system_image_guid);*/ u16 mlx5_eswitch_get_total_vports (const struct mlx5_core_dev *dev) { -return MLX5_SPECIAL_VPORTS (dev) +mlx5_core_max_vfs (dev); +return MLX5_SPECIAL_VPORTS (dev) +mlx5_core_max_vfs (dev) +mlx5_sf_max_functions (dev); }EXPORT_SYMBOL_GPL (mlx5_eswitch_get_total_vports);
Signature · 1 line
2.26.2
PATCH v4 7/15 net/mlx5: SF, Add auxiliary device support
In reply to Saeed MahameedFrom: Parav Pandit <[email protected]>
Introduce API to add and delete an auxiliary device for an SF. Each SF has its own dedicated window in the PCI BAR 2.
SF device is similar to PCI PF and VF that supports multiple class of devices such as net, rdma and vdpa.
SF device will be added or removed in subsequent patch during SF devlink port function state change command.
A subfunction device exposes user supplied subfunction number which will be further used by systemd/udev to have deterministic name for its netdevice and rdma device.
An mlx5 subfunction auxiliary device example:
$ devlink dev eswitch set pci/0000:06:00.0 mode switchdev
$ devlink port show pci/0000:06:00.0/65535: type eth netdev ens2f0np0 flavour physical port 0 splittable false
$ devlink port add pci/0000:06:00.0 flavour pcisf pfnum 0 sfnum 88
$ devlink port show ens2f0npf0sf88
pci/0000:06:00.0/32768: type eth netdev ens2f0npf0sf88 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false
function:
hw_addr 00:00:00:00:88:88 state inactive opstate detached$ devlink port function set ens2f0npf0sf88 hw_addr 00:00:00:00:88:88 state active
On activation,
$ ls -l /sys/bus/auxiliary/devices/ mlx5_core.sf.4 -> ../../../devices/pci0000:00/0000:00:03.0/0000:06:00.0/mlx5_core.sf.4
$ cat /sys/bus/auxiliary/devices/mlx5_core.sf.4/sfnum 88
--- .../device_drivers/ethernet/mellanox/mlx5.rst | 5 + .../net/ethernet/mellanox/mlx5/core/Makefile | 2 +- .../net/ethernet/mellanox/mlx5/core/main.c | 4 + .../ethernet/mellanox/mlx5/core/sf/dev/dev.c | 261 ++++++++++++++++++ .../ethernet/mellanox/mlx5/core/sf/dev/dev.h | 35 +++ include/linux/mlx5/driver.h | 2 + 6 files changed, 308 insertions(+), 1 deletion(-) create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/sf/dev/dev.c create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/sf/dev/dev.h
Documentation/networking/device_drivers/ethernet/mellanox/mlx5.rst +5 -0
@@ -97,6 +97,11 @@ Enabling the driver and kconfig options | Provides low-level InfiniBand/RDMA and `RoCE <https://community.mellanox.com/s/article/recommended-network-configuration-examples-for-roce-deployment>`_ support. +**CONFIG_MLX5_SF=(y/n)** + +| Build support for subfunction. +| Subfunctons are more light weight than PCI SRIOV VFs. Choosing this option +| will enable support for creating subfunction devices. **External options** ( Choose if the corresponding mlx5 feature is required )
drivers/net/ethernet/mellanox/mlx5/core/Makefile +1 -1
@@ -88,4 +88,4 @@ mlx5_core-$(CONFIG_MLX5_SW_STEERING) += steering/dr_domain.o steering/dr_table.o # # SF device # -mlx5_core-$(CONFIG_MLX5_SF) += sf/vhca_event.o +mlx5_core-$(CONFIG_MLX5_SF) += sf/vhca_event.o sf/dev/dev.o
drivers/net/ethernet/mellanox/mlx5/core/main.c +4 -0
@@ -74,6 +74,7 @@#include "lib/hv_vhca.h" #include "diag/rsc_dump.h" #include "sf/vhca_event.h" +#include "sf/dev/dev.h" MODULE_AUTHOR ("Eli Cohen <[email protected]>" );MODULE_DESCRIPTION ("Mellanox 5th generation network adapters (ConnectX series) core driver" );
@@ -1155,6 +1156,8 @@ static int mlx5_load(struct mlx5_core_dev *dev) goto err_sriov; } +mlx5_sf_dev_table_create (dev); +return 0 ; err_sriov:
@@ -1186,6 +1189,7 @@ static int mlx5_load(struct mlx5_core_dev *dev)static void mlx5_unload (struct mlx5_core_dev *dev) { +mlx5_sf_dev_table_destroy (dev);mlx5_sriov_detach (dev);mlx5_ec_cleanup (dev);mlx5_vhca_event_stop (dev);
drivers/net/ethernet/mellanox/mlx5/core/sf/dev/dev.c new file +261 -0
261 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/sf/dev/dev.h new file +35 -0
@@ -0,0 +1,35 @@ +/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */ +/* Copyright (c) 2020 Mellanox Technologies Ltd */ + +#ifndef __MLX5_SF_DEV_H__ +#define __MLX5_SF_DEV_H__ + +#ifdef CONFIG_MLX5_SF + +#include <linux/auxiliary_bus.h> + +#define MLX5_SF_DEV_ID_NAME "sf" + +struct mlx5_sf_dev { +struct auxiliary_device adev; +struct mlx5_core_dev *parent_mdev; +phys_addr_t bar_base_addr; +u32 sfnum; +}; + +void mlx5_sf_dev_table_create (struct mlx5_core_dev *dev); +void mlx5_sf_dev_table_destroy (struct mlx5_core_dev *dev); + +#else + +static inline void mlx5_sf_dev_table_create (struct mlx5_core_dev *dev) +{ +} + +static inline void mlx5_sf_dev_table_destroy (struct mlx5_core_dev *dev) +{ +} + +#endif + +#endif
include/linux/mlx5/driver.h +2 -0
@@ -508,6 +508,7 @@ struct mlx5_fw_reset;struct mlx5_eq_table ;struct mlx5_irq_table ;struct mlx5_vhca_state_notifier ; +struct mlx5_sf_dev_table ;struct mlx5_rate_limit {u32 rate;
@@ -606,6 +607,7 @@ struct mlx5_priv {struct mlx5_uars_page *uar;#ifdef CONFIG_MLX5_SF struct mlx5_vhca_state_notifier *vhca_state_notifier; +struct mlx5_sf_dev_table *sf_dev_table;#endif };
Signature · 1 line
2.26.2
PATCH v4 15/15 net/mlx5: Add devlink subfunction port documentation
In reply to Saeed MahameedFrom: Parav Pandit <[email protected]>
Add documentation for subfunction management using devlink port.
--- .../device_drivers/ethernet/mellanox/mlx5.rst | 204 ++++++++++++++++++ 1 file changed, 204 insertions(+)
Documentation/networking/device_drivers/ethernet/mellanox/mlx5.rst +204 -0
@@ -12,6 +12,8 @@ Contents - `Enabling the driver and kconfig options`_ - `Devlink info`_ - `Devlink parameters`_ +- `mlx5 subfunction`_ +- `mlx5 port function`_ - `Devlink health reporters`_ - `mlx5 tracepoints`_
@@ -181,6 +183,208 @@ User command examples: values: cmode driverinit value true +mlx5 subfunction +================ +mlx5 supports subfunctions management using devlink port (see :ref:`Documentation/networking/devlink/devlink-port.rst <devlink_port>`) interface. + +A Subfunction has its own function capabilities and its own resources. This +means a subfunction has its own dedicated queues(txq, rxq, cq, eq). These queues +are neither shared nor stealed from the parent PCI function. + +When subfunction is RDMA capable, it has its own QP1, GID table and rdma +resources neither shared nor stealed from the parent PCI function. + +A subfunction has dedicated window in PCI BAR space that is not shared +with ther other subfunctions or parent PCI function. This ensures that all +class devices of the subfunction accesses only assigned PCI BAR space. + +A Subfunction supports eswitch representation through which it supports tc +offloads. User must configure eswitch to send/receive packets from/to +subfunction port. + +Subfunctions share PCI level resources such as PCI MSI-X IRQs with +ther other subfunctions and/or with its parent PCI function. + +Example mlx5 software, system and device view:: + + _______ + | admin | + | user |---------- + |_______| | + | | + ____|____ __|______ _________________ + | | | | | | + | devlink | | tc tool | | user | + | tool | |_________| | applications | + |_________| | |_________________| + | | | | + | | | | Userspace + +---------|-------------|-------------------|----------|--------------------+ + | | +----------+ +----------+ Kernel + | | | netdev | | rdma dev | + | | +----------+ +----------+ + (devlink port add/del | ^ ^ + port function set) | | | + | | +---------------| + _____|___ | | _______|_______ + | | | | | mlx5 class | + | devlink | +------------+ | | drivers | + | kernel | | rep netdev | | |(mlx5_core,ib) | + |_________| +------------+ | |_______________| + | | | ^ + (devlink ops) | | (probe/remove) + _________|________ | | ____|________ + | subfunction | | +---------------+ | subfunction | + | management driver|----- | subfunction |---| driver | + | (mlx5_core) | | auxiliary dev | | (mlx5_core) | + |__________________| +---------------+ |_____________| + | ^ + (sf add/del, vhca events) | + | (device add/del) + _____|____ ____|________ + | | | subfunction | + | PCI NIC |---- activate/deactive events---->| host driver | + |__________| | (mlx5_core) | + |_____________| + +Subfunction is created using devlink port interface. + +- Change device to switchdev mode:: + + $ devlink dev eswitch set pci/0000:06:00.0 mode switchdev + +- Add a devlink port of subfunction flaovur:: + + $ devlink port add pci/0000:06:00.0 flavour pcisf pfnum 0 sfnum 88 + +- Show a devlink port of the subfunction:: + + $ devlink port show pci/0000:06:00.0/32768 + pci/0000:06:00.0/32768: type eth netdev enp6s0pf0sf88 flavour pcisf pfnum 0 sfnum 88 + function: + hw_addr 00:00:00:00:00:00 + +- Delete a devlink port of subfunction after use:: + + $ devlink port del pci/0000:06:00.0 flavour pcisf pfnum 0 sfnum 88 + +mlx5 port function +================== +mlx5 driver provides mechanism to setup PCI VF/SF port function +attributes in unified way for smartnic and non-smartnic NICs. + +This is supported only when eswitch mode is set to switchdev. Port function +configuration of the PCI VF/SF is supported through devlink eswitch port. + +Port function attributes should be set before PCI VF/SF is enumerated by the +driver. + +MAC address setup +----------------- +mlx5 driver provides mechanism to setup the MAC address of the PCI VF/SF. + +Configured MAC address of the PCI VF/SF will be used by netdevice and rdma +device created for the PCI VF/SF. + +- Get MAC address of the VF identified by its unique devlink port index:: + + $ devlink port show pci/0000:06:00.0/2 + pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1 + function: + hw_addr 00:00:00:00:00:00 + +- Set MAC address of the VF identified by its unique devlink port index:: + + $ devlink port function set pci/0000:06:00.0/2 hw_addr 00:11:22:33:44:55 + + $ devlink port show pci/0000:06:00.0/2 + pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1 + function: + hw_addr 00:11:22:33:44:55 + +- Get MAC address of the SF identified by its unique devlink port index:: + + $ devlink port show pci/0000:06:00.0/32768 + pci/0000:06:00.0/32768: type eth netdev enp6s0pf0sf88 flavour pcisf pfnum 0 sfnum 88 + function: + hw_addr 00:00:00:00:00:00 + +- Set MAC address of the VF identified by its unique devlink port index:: + + $ devlink port function set pci/0000:06:00.0/32768 hw_addr 00:00:00:00:88:88 + + $ devlink port show pci/0000:06:00.0/32768 + pci/0000:06:00.0/32768: type eth netdev enp6s0pf0sf88 flavour pcivf pfnum 0 sfnum 88 + function: + hw_addr 00:00:00:00:88:88 + +SF state setup +-------------- +To use the SF, user must active the SF using SF port function state attribute. + +- Get state of the SF identified by its unique devlink port index:: + + $ devlink port show ens2f0npf0sf88 + pci/0000:06:00.0/32768: type eth netdev ens2f0npf0sf88 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false + function: + hw_addr 00:00:00:00:88:88 state inactive opstate detached + +- Activate the function and verify its state is active:: + + $ devlink port function set ens2f0npf0sf88 state active + + $ devlink port show ens2f0npf0sf88 + pci/0000:06:00.0/32768: type eth netdev ens2f0npf0sf88 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false + function: + hw_addr 00:00:00:00:88:88 state active opstate detached + +Upon function activation, PF driver instance gets the event from the device that +particular SF was activated. It's the cue to put the device on bus, probe it and +instantiate devlink instance and class specific auxiliary devices for it. + +- Show the auxiliary device and port of the subfunction:: + + $ devlink dev show + devlink dev show auxiliary/mlx5_core.sf.4 + + $ devlink port show auxiliary/mlx5_core.sf.4/1 + auxiliary/mlx5_core.sf.4/1: type eth netdev p0sf88 flavour virtual port 0 splittable false + + $ rdma link show mlx5_0/1 + link mlx5_0/1 state ACTIVE physical_state LINK_UP netdev p0sf88 + + $ rdma dev show + 8: rocep6s0f1: node_type ca fw 16.29.0550 node_guid 248a:0703:00b3:d113 sys_image_guid 248a:0703:00b3:d112 + 13: mlx5_0: node_type ca fw 16.29.0550 node_guid 0000:00ff:fe00:8888 sys_image_guid 248a:0703:00b3:d112 + +- Subfunction auxilary device and class device hierarchy:: + + mlx5_core.sf.4 + (subfunction auxilary device) + /\ + / \ + / \ + / \ + / \ + mlx5_core.eth.4 mlx5_core.rdma.4 + (sf eth aux dev) (sf rdma aux dev) + | | + | | + p0sf88 mlx5_0 + (sf netdev) (sf rdma device) + +Additionally SF port also gets the event when the driver attaches to the +auxiliary device of the subfunction. This results in changing the operational +state of the function. This provides visiblity to user to decide when it is +safe to delete the SF port for graceful termination of the subfunction. + +- Show the SF port operational state:: + + $ devlink port show ens2f0npf0sf88 + pci/0000:06:00.0/32768: type eth netdev ens2f0npf0sf88 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false + function: + hw_addr 00:00:00:00:88:88 state active opstate attached + Devlink health reporters ========================
Signature · 1 line
2.26.2
PATCH v4 13/15 devlink: Add devlink port documentation
In reply to Saeed MahameedFrom: Parav Pandit <[email protected]>
Added documentation for devlink port and port function related commands.
--- .../networking/devlink/devlink-port.rst | 118 ++++++++++++++++++ Documentation/networking/devlink/index.rst | 1 + 2 files changed, 119 insertions(+) create mode 100644 Documentation/networking/devlink/devlink-port.rst
Documentation/networking/devlink/devlink-port.rst new file +118 -0
@@ -0,0 +1,118 @@ +.. SPDX-License-Identifier: GPL-2.0 + +.. _devlink_port: + +============ +Devlink Port +============ + +``devlink-port`` is a port that exists on the device. It has a logically +separate ingress/egress point of the device. A devlink port can be any one +of many flavours. A devlink port flavour along with port attributes +describe what a port represents. + +A device driver that intends to publish a devlink port sets the +devlink port attributes and registers the devlink port. + +Devlink port flavours are described below. + +.. list-table:: List of devlink port flavours + :widths: 33 90 + + * - Flavour + - Description + * - ``DEVLINK_PORT_FLAVOUR_PHYSICAL`` + - Any kind of physical port. This can be an eswitch physical port or any + other physical port on the device. + * - ``DEVLINK_PORT_FLAVOUR_DSA`` + - This indicates a DSA interconnect port. + * - ``DEVLINK_PORT_FLAVOUR_CPU`` + - This indicates a CPU port applicable only to DSA. + * - ``DEVLINK_PORT_FLAVOUR_PCI_PF`` + - This indicates an eswitch port representing a port of PCI + physical function (PF). + * - ``DEVLINK_PORT_FLAVOUR_PCI_VF`` + - This indicates an eswitch port representing a port of PCI + virtual function (VF). + * - ``DEVLINK_PORT_FLAVOUR_VIRTUAL`` + - This indicates a virtual port for the PCI virtual function. + +Devlink port can have a different type based on the link layer described below. + +.. list-table:: List of devlink port types + :widths: 23 90 + + * - Type + - Description + * - ``DEVLINK_PORT_TYPE_ETH`` + - Driver should set this port type when a link layer of the port is + Ethernet. + * - ``DEVLINK_PORT_TYPE_IB`` + - Driver should set this port type when a link layer of the port is + InfiniBand. + * - ``DEVLINK_PORT_TYPE_AUTO`` + - This type is indicated by the user when driver should detect the port + type automatically. + +PCI controllers +--------------- +In most cases a PCI device has only one controller. A controller consists of +potentially multiple physical and virtual functions. Such PCI function consists +of one or more ports. This port of the function is represented by the devlink +eswitch port. + +A PCI Device connected to multiple CPUs or multiple PCI root complexes or +SmartNIC, however, may have multiple controllers. For a device with multiple +controllers, each controller is distinguished by a unique controller number. +An eswitch on the PCI device support ports of multiple controllers. + +An example view of a system with two controllers:: + + --------------------------------------------------------- + | | + | --------- --------- ------- ------- | + ----------- | | vf(s) | | sf(s) | |vf(s)| |sf(s)| | + | server | | ------- ----/---- ---/----- ------- ---/--- ---/--- | + | pci rc |=== | pf0 |______/________/ | pf1 |___/_______/ | + | connect | | ------- ------- | + ----------- | | controller_num=1 (no eswitch) | + ------|-------------------------------------------------- + (internal wire) + | + --------------------------------------------------------- + | devlink eswitch ports and reps | + | ----------------------------------------------------- | + | |ctrl-0 | ctrl-0 | ctrl-0 | ctrl-0 | ctrl-0 |ctrl-0 | | + | |pf0 | pf0vfN | pf0sfN | pf1 | pf1vfN |pf1sfN | | + | ----------------------------------------------------- | + | |ctrl-1 | ctrl-1 | ctrl-1 | ctrl-1 | ctrl-1 |ctrl-1 | | + | |pf0 | pf0vfN | pf0sfN | pf1 | pf1vfN |pf1sfN | | + | ----------------------------------------------------- | + | | + | | + ----------- | --------- --------- ------- ------- | + | smartNIC| | | vf(s) | | sf(s) | |vf(s)| |sf(s)| | + | pci rc |==| ------- ----/---- ---/----- ------- ---/--- ---/--- | + | connect | | | pf0 |______/________/ | pf1 |___/_______/ | + ----------- | ------- ------- | + | | + | local controller_num=0 (eswitch) | + --------------------------------------------------------- + +In above example, external controller (identified by controller number = 1) +doesn't have eswitch. Local controller (identified by controller number = 0) +has the eswitch. Devlink instance on local controller has eswitch devlink +ports representing ports for both the controllers. + +Port function configuration +=========================== + +A user can configure the port function attribute before enumerating the +PCI function. Usually it means, user should configure port function attribute +before a bus specific device for the function is created. However, when +SRIOV is enabled, virtual function devices are created on the PCI bus. +Hence, function attribute should be configured before binding virtual +function device to the driver. + +User may set the hardware address of the function represented by the devlink +port function. For Ethernet port function this means a MAC address.
Documentation/networking/devlink/index.rst +1 -0
@@ -18,6 +18,7 @@ general. devlink-info devlink-flash devlink-params + devlink-port devlink-region devlink-resource devlink-reload
Signature · 1 line
2.26.2
PATCH v4 14/15 devlink: Extend devlink port documentation for subfunctions
In reply to Saeed MahameedFrom: Parav Pandit <[email protected]>
Add devlink port documentation for subfunction management.
--- Documentation/driver-api/auxiliary_bus.rst | 2 + .../networking/devlink/devlink-port.rst | 89 ++++++++++++++++++- 2 files changed, 87 insertions(+), 4 deletions(-)
Documentation/driver-api/auxiliary_bus.rst +2 -0
@@ -1,5 +1,7 @@ .. SPDX-License-Identifier: GPL-2.0-only +.. _auxiliary_bus: + ============= Auxiliary Bus =============
Documentation/networking/devlink/devlink-port.rst +85 -4
107 lines not on this page · raw message
Signature · 1 line
2.26.2
PATCH v4 6/15 net/mlx5: Introduce vhca state event notifier
In reply to Saeed MahameedFrom: Parav Pandit <[email protected]>
vhca state events indicates change in the state of the vhca that may occur due to a SF allocation, deallocation or enabling/disabling the SF HCA.
Introduce vhca state event handler which will be used by SF devlink port manager and SF hardware id allocator in subsequent patches to act on the event.
This enables single entity to subscribe, query and rearm the event for a function.
--- .../net/ethernet/mellanox/mlx5/core/Kconfig | 9 + .../net/ethernet/mellanox/mlx5/core/Makefile | 4 + drivers/net/ethernet/mellanox/mlx5/core/cmd.c | 4 + drivers/net/ethernet/mellanox/mlx5/core/eq.c | 3 + .../net/ethernet/mellanox/mlx5/core/events.c | 7 + .../net/ethernet/mellanox/mlx5/core/main.c | 16 ++ .../ethernet/mellanox/mlx5/core/mlx5_core.h | 2 + .../mlx5/core/sf/mlx5_ifc_vhca_event.h | 82 ++++++++ .../net/ethernet/mellanox/mlx5/core/sf/sf.h | 45 +++++ .../mellanox/mlx5/core/sf/vhca_event.c | 189 ++++++++++++++++++ .../mellanox/mlx5/core/sf/vhca_event.h | 57 ++++++ include/linux/mlx5/driver.h | 4 + 12 files changed, 422 insertions(+) create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/sf/mlx5_ifc_vhca_event.h create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/sf/sf.h create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/sf/vhca_event.c create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/sf/vhca_event.h
drivers/net/ethernet/mellanox/mlx5/core/Kconfig +9 -0
@@ -203,3 +203,12 @@ config MLX5_SW_STEERING default y help Build support for software-managed steering in the NIC. + +config MLX5_SF + bool "Mellanox Technologies subfunction device support using auxiliary device" + depends on MLX5_CORE && MLX5_CORE_EN + default n + help + Build support for subfuction device in the NIC. A Mellanox subfunction + device can support RDMA, netdevice and vdpa device. + It is similar to a SRIOV VF but it doesn't require SRIOV support.
drivers/net/ethernet/mellanox/mlx5/core/Makefile +4 -0
@@ -85,3 +85,7 @@ mlx5_core-$(CONFIG_MLX5_SW_STEERING) += steering/dr_domain.o steering/dr_table.o steering/dr_ste.o steering/dr_send.o \ steering/dr_cmd.o steering/dr_fw.o \ steering/dr_action.o steering/fs_dr.o +# +# SF device +# +mlx5_core-$(CONFIG_MLX5_SF) += sf/vhca_event.o
drivers/net/ethernet/mellanox/mlx5/core/cmd.c +4 -0
@@ -464,6 +464,8 @@ static int mlx5_internal_err_ret_value(struct mlx5_core_dev *dev, u16 op,case MLX5_CMD_OP_ALLOC_MEMIC :case MLX5_CMD_OP_MODIFY_XRQ :case MLX5_CMD_OP_RELEASE_XRQ_ERROR : +case MLX5_CMD_OP_QUERY_VHCA_STATE : +case MLX5_CMD_OP_MODIFY_VHCA_STATE : *status =MLX5_DRIVER_STATUS_ABORTED ; *synd =MLX5_DRIVER_SYND ;return -EIO ;
@@ -657,6 +659,8 @@ const char *mlx5_command_str(int command)MLX5_COMMAND_STR_CASE (DESTROY_UMEM );MLX5_COMMAND_STR_CASE (RELEASE_XRQ_ERROR );MLX5_COMMAND_STR_CASE (MODIFY_XRQ ); +MLX5_COMMAND_STR_CASE (QUERY_VHCA_STATE ); +MLX5_COMMAND_STR_CASE (MODIFY_VHCA_STATE );default :return "unknown command opcode" ; } }
drivers/net/ethernet/mellanox/mlx5/core/eq.c +3 -0
@@ -595,6 +595,9 @@ static void gather_async_events_mask(struct mlx5_core_dev *dev, u64 mask[4]) async_event_mask |= (1ull <<MLX5_EVENT_TYPE_ESW_FUNCTIONS_CHANGED ); +if (MLX5_CAP_GEN_MAX (dev, vhca_state)) + async_event_mask |= (1ull <<MLX5_EVENT_TYPE_VHCA_STATE_CHANGE ); + mask[0 ] = async_event_mask;if (MLX5_CAP_GEN (dev, event_cap))
drivers/net/ethernet/mellanox/mlx5/core/events.c +7 -0
@@ -110,6 +110,8 @@ static const char *eqe_type_str(u8 type)return "MLX5_EVENT_TYPE_CMD" ;case MLX5_EVENT_TYPE_ESW_FUNCTIONS_CHANGED :return "MLX5_EVENT_TYPE_ESW_FUNCTIONS_CHANGED" ; +case MLX5_EVENT_TYPE_VHCA_STATE_CHANGE : +return "MLX5_EVENT_TYPE_VHCA_STATE_CHANGE" ;case MLX5_EVENT_TYPE_PAGE_REQUEST :return "MLX5_EVENT_TYPE_PAGE_REQUEST" ;case MLX5_EVENT_TYPE_PAGE_FAULT :
@@ -403,3 +405,8 @@ int mlx5_notifier_call_chain(struct mlx5_events *events, unsigned int event, voi {return atomic_notifier_call_chain (&events->nh, event, data); } + +void mlx5_events_work_enqueue (struct mlx5_core_dev *dev,struct work_struct *work) +{ +queue_work (dev->priv.events->wq, work); +}
drivers/net/ethernet/mellanox/mlx5/core/main.c +16 -0
@@ -73,6 +73,7 @@#include "ecpf.h" #include "lib/hv_vhca.h" #include "diag/rsc_dump.h" +#include "sf/vhca_event.h" MODULE_AUTHOR ("Eli Cohen <[email protected]>" );MODULE_DESCRIPTION ("Mellanox 5th generation network adapters (ConnectX series) core driver" );
@@ -567,6 +568,8 @@ static int handle_hca_cap(struct mlx5_core_dev *dev, void *set_ctx)if (MLX5_CAP_GEN_MAX (dev, mkey_by_name))MLX5_SET (cmd_hca_cap, set_hca_cap, mkey_by_name,1 ); +mlx5_vhca_state_cap_handle (dev, set_hca_cap); +return set_caps (dev, set_ctx,MLX5_SET_HCA_CAP_OP_MOD_GENERAL_DEVICE ); }
@@ -884,6 +887,12 @@ static int mlx5_init_once(struct mlx5_core_dev *dev) goto err_eswitch_cleanup; } + err =mlx5_vhca_event_init (dev); +if (err) { +mlx5_core_err (dev,"Failed to init vhca event notifier %d\n" , err); + goto err_fpga_cleanup; + } + dev->dm =mlx5_dm_create (dev);if (IS_ERR (dev->dm))mlx5_core_warn (dev,"Failed to init device memory%d\n" , err);
@@ -894,6 +903,8 @@ static int mlx5_init_once(struct mlx5_core_dev *dev)return 0 ; +err_fpga_cleanup: +mlx5_fpga_cleanup (dev); err_eswitch_cleanup:mlx5_eswitch_cleanup (dev->priv.eswitch); err_sriov_cleanup:
@@ -925,6 +936,7 @@ static void mlx5_cleanup_once(struct mlx5_core_dev *dev)mlx5_hv_vhca_destroy (dev->hv_vhca);mlx5_fw_tracer_destroy (dev->tracer);mlx5_dm_cleanup (dev); +mlx5_vhca_event_cleanup (dev);mlx5_fpga_cleanup (dev);mlx5_eswitch_cleanup (dev->priv.eswitch);mlx5_sriov_cleanup (dev);
@@ -1129,6 +1141,8 @@ static int mlx5_load(struct mlx5_core_dev *dev) goto err_sriov; } +mlx5_vhca_event_start (dev); + err =mlx5_ec_init (dev);if (err) {mlx5_core_err (dev,"Failed to init embedded CPU\n" );
@@ -1146,6 +1160,7 @@ static int mlx5_load(struct mlx5_core_dev *dev) err_sriov:mlx5_ec_cleanup (dev); err_ec: +mlx5_vhca_event_stop (dev);mlx5_cleanup_fs (dev); err_fs:mlx5_accel_tls_cleanup (dev);
@@ -1173,6 +1188,7 @@ static void mlx5_unload(struct mlx5_core_dev *dev) {mlx5_sriov_detach (dev);mlx5_ec_cleanup (dev); +mlx5_vhca_event_stop (dev);mlx5_cleanup_fs (dev);mlx5_accel_ipsec_cleanup (dev);mlx5_accel_tls_cleanup (dev);
drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h +2 -0
@@ -259,4 +259,6 @@ void mlx5_set_nic_state(struct mlx5_core_dev *dev, u8 state);void mlx5_unload_one (struct mlx5_core_dev *dev,bool cleanup);int mlx5_load_one (struct mlx5_core_dev *dev,bool boot); + +void mlx5_events_work_enqueue (struct mlx5_core_dev *dev,struct work_struct *work);#endif /* __MLX5_CORE_H__ */
drivers/net/ethernet/mellanox/mlx5/core/sf/mlx5_ifc_vhca_event.h new file +82 -0
82 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/sf/sf.h new file +45 -0
45 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/sf/vhca_event.c new file +189 -0
189 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/sf/vhca_event.h new file +57 -0
57 lines not on this page · raw message
include/linux/mlx5/driver.h +4 -0
@@ -507,6 +507,7 @@ struct mlx5_devcom;struct mlx5_fw_reset ;struct mlx5_eq_table ;struct mlx5_irq_table ; +struct mlx5_vhca_state_notifier ;struct mlx5_rate_limit {u32 rate;
@@ -603,6 +604,9 @@ struct mlx5_priv {struct mlx5_bfreg_data bfregs;struct mlx5_uars_page *uar; +#ifdef CONFIG_MLX5_SF +struct mlx5_vhca_state_notifier *vhca_state_notifier; +#endif };enum mlx5_device_state {
Signature · 1 line
2.26.2
PATCH v4 10/15 net/mlx5: E-switch, Add eswitch helpers for SF vport
In reply to Saeed MahameedFrom: Parav Pandit <[email protected]>
Add helpers to enable/disable eswitch port, register its devlink port and load its representor.
--- .../mellanox/mlx5/core/esw/devlink_port.c | 41 +++++++++++++++++++ .../net/ethernet/mellanox/mlx5/core/eswitch.c | 12 +++--- .../net/ethernet/mellanox/mlx5/core/eswitch.h | 16 ++++++++ .../mellanox/mlx5/core/eswitch_offloads.c | 36 +++++++++++++++- 4 files changed, 97 insertions(+), 8 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/esw/devlink_port.c +41 -0
@@ -122,3 +122,44 @@ struct devlink_port *mlx5_esw_offloads_devlink_port(struct mlx5_eswitch *esw, u1 vport =mlx5_eswitch_get_vport (esw, vport_num);return vport->dl_port; } + +int mlx5_esw_devlink_sf_port_register (struct mlx5_eswitch *esw,struct devlink_port *dl_port, +u16 vport_num,u32 sfnum) +{ +struct mlx5_core_dev *dev = esw->dev; +struct netdev_phys_item_id ppid = {}; +unsigned int dl_port_index; +struct mlx5_vport *vport; +struct devlink *devlink; +u16 pfnum; +int err; + + vport =mlx5_eswitch_get_vport (esw, vport_num); +if (IS_ERR (vport)) +return PTR_ERR (vport); + + pfnum =PCI_FUNC (dev->pdev->devfn); +mlx5_esw_get_port_parent_id (dev, &ppid); +memcpy (dl_port->attrs.switch_id.id, &ppid.id[0 ], ppid.id_len); + dl_port->attrs.switch_id.id_len = ppid.id_len; +devlink_port_attrs_pci_sf_set (dl_port,0 , pfnum, sfnum, false); + devlink =priv_to_devlink (dev); + dl_port_index =mlx5_esw_vport_to_devlink_port_index (dev, vport_num); + err =devlink_port_register (devlink, dl_port, dl_port_index); +if (err) +return err; + + vport->dl_port = dl_port; +return 0 ; +} + +void mlx5_esw_devlink_sf_port_unregister (struct mlx5_eswitch *esw,u16 vport_num) +{ +struct mlx5_vport *vport; + + vport =mlx5_eswitch_get_vport (esw, vport_num); +if (IS_ERR (vport)) +return ; +devlink_port_unregister (vport->dl_port); + vport->dl_port =NULL ; +}
drivers/net/ethernet/mellanox/mlx5/core/eswitch.c +6 -6
@@ -1273,8 +1273,8 @@ static void esw_vport_cleanup(struct mlx5_eswitch *esw, struct mlx5_vport *vportesw_vport_cleanup_acl (esw, vport); } -static int esw_enable_vport (struct mlx5_eswitch *esw,u16 vport_num, -enum mlx5_eswitch_vport_event enabled_events) +int mlx5_esw_vport_enable (struct mlx5_eswitch *esw,u16 vport_num, +enum mlx5_eswitch_vport_event enabled_events) {struct mlx5_vport *vport;int ret;
@@ -1310,7 +1310,7 @@ static int esw_enable_vport(struct mlx5_eswitch *esw, u16 vport_num,return ret; } -static void esw_disable_vport (struct mlx5_eswitch *esw,u16 vport_num) +void mlx5_esw_vport_disable (struct mlx5_eswitch *esw,u16 vport_num) {struct mlx5_vport *vport;
@@ -1432,7 +1432,7 @@ int mlx5_eswitch_load_vport(struct mlx5_eswitch *esw, u16 vport_num, {int err; - err =esw_enable_vport (esw, vport_num, enabled_events); + err =mlx5_esw_vport_enable (esw, vport_num, enabled_events);if (err)return err;
@@ -1443,14 +1443,14 @@ int mlx5_eswitch_load_vport(struct mlx5_eswitch *esw, u16 vport_num,return err; err_rep: -esw_disable_vport (esw, vport_num); +mlx5_esw_vport_disable (esw, vport_num);return err; }void mlx5_eswitch_unload_vport (struct mlx5_eswitch *esw,u16 vport_num) {esw_offloads_unload_rep (esw, vport_num); -esw_disable_vport (esw, vport_num); +mlx5_esw_vport_disable (esw, vport_num); }void mlx5_eswitch_unload_vf_vports (struct mlx5_eswitch *esw,u16 num_vfs)
drivers/net/ethernet/mellanox/mlx5/core/eswitch.h +16 -0
@@ -688,6 +688,10 @@ mlx5_eswitch_enable_pf_vf_vports(struct mlx5_eswitch *esw, enum mlx5_eswitch_vport_event enabled_events);void mlx5_eswitch_disable_pf_vf_vports (struct mlx5_eswitch *esw); +int mlx5_esw_vport_enable (struct mlx5_eswitch *esw,u16 vport_num, +enum mlx5_eswitch_vport_event enabled_events); +void mlx5_esw_vport_disable (struct mlx5_eswitch *esw,u16 vport_num); +int esw_vport_create_offloads_acl_tables (struct mlx5_eswitch *esw,struct mlx5_vport *vport);
@@ -706,6 +710,9 @@ esw_get_max_restore_tag(struct mlx5_eswitch *esw);int esw_offloads_load_rep (struct mlx5_eswitch *esw,u16 vport_num);void esw_offloads_unload_rep (struct mlx5_eswitch *esw,u16 vport_num); +int mlx5_esw_offloads_rep_load (struct mlx5_eswitch *esw,u16 vport_num); +void mlx5_esw_offloads_rep_unload (struct mlx5_eswitch *esw,u16 vport_num); +int mlx5_eswitch_load_vport (struct mlx5_eswitch *esw,u16 vport_num,enum mlx5_eswitch_vport_event enabled_events);void mlx5_eswitch_unload_vport (struct mlx5_eswitch *esw,u16 vport_num);
@@ -717,6 +724,15 @@ void mlx5_eswitch_unload_vf_vports(struct mlx5_eswitch *esw, u16 num_vfs);int mlx5_esw_offloads_devlink_port_register (struct mlx5_eswitch *esw,u16 vport_num);void mlx5_esw_offloads_devlink_port_unregister (struct mlx5_eswitch *esw,u16 vport_num);struct devlink_port *mlx5_esw_offloads_devlink_port (struct mlx5_eswitch *esw,u16 vport_num); + +int mlx5_esw_devlink_sf_port_register (struct mlx5_eswitch *esw,struct devlink_port *dl_port, +u16 vport_num,u32 sfnum); +void mlx5_esw_devlink_sf_port_unregister (struct mlx5_eswitch *esw,u16 vport_num); + +int mlx5_esw_offloads_sf_vport_enable (struct mlx5_eswitch *esw,struct devlink_port *dl_port, +u16 vport_num,u32 sfnum); +void mlx5_esw_offloads_sf_vport_disable (struct mlx5_eswitch *esw,u16 vport_num); +#else /* CONFIG_MLX5_ESWITCH */ /* eswitch API stubs */ static inline int mlx5_eswitch_init (struct mlx5_core_dev *dev) {return 0 ; }
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +34 -2
@@ -1833,7 +1833,7 @@ static void __unload_reps_all_vport(struct mlx5_eswitch *esw, u8 rep_type)__esw_offloads_unload_rep (esw, rep, rep_type); } -static int mlx5_esw_offloads_rep_load (struct mlx5_eswitch *esw,u16 vport_num) +int mlx5_esw_offloads_rep_load (struct mlx5_eswitch *esw,u16 vport_num) {struct mlx5_eswitch_rep *rep;int rep_type;
@@ -1857,7 +1857,7 @@ static int mlx5_esw_offloads_rep_load(struct mlx5_eswitch *esw, u16 vport_num)return err; } -static void mlx5_esw_offloads_rep_unload (struct mlx5_eswitch *esw,u16 vport_num) +void mlx5_esw_offloads_rep_unload (struct mlx5_eswitch *esw,u16 vport_num) {struct mlx5_eswitch_rep *rep;int rep_type;
@@ -2835,3 +2835,35 @@ u32 mlx5_eswitch_get_vport_metadata_for_match(struct mlx5_eswitch *esw,return vport->metadata << (32 -ESW_SOURCE_PORT_METADATA_BITS ); }EXPORT_SYMBOL (mlx5_eswitch_get_vport_metadata_for_match); + +int mlx5_esw_offloads_sf_vport_enable (struct mlx5_eswitch *esw,struct devlink_port *dl_port, +u16 vport_num,u32 sfnum) +{ +int err; + + err =mlx5_esw_vport_enable (esw, vport_num,MLX5_VPORT_UC_ADDR_CHANGE ); +if (err) +return err; + + err =mlx5_esw_devlink_sf_port_register (esw, dl_port, vport_num, sfnum); +if (err) + goto devlink_err; + + err =mlx5_esw_offloads_rep_load (esw, vport_num); +if (err) + goto rep_err; +return 0 ; + +rep_err: +mlx5_esw_devlink_sf_port_unregister (esw, vport_num); +devlink_err: +mlx5_esw_vport_disable (esw, vport_num); +return err; +} + +void mlx5_esw_offloads_sf_vport_disable (struct mlx5_eswitch *esw,u16 vport_num) +{ +mlx5_esw_offloads_rep_unload (esw, vport_num); +mlx5_esw_devlink_sf_port_unregister (esw, vport_num); +mlx5_esw_vport_disable (esw, vport_num); +}
Signature · 1 line
2.26.2
PATCH v4 12/15 net/mlx5: SF, Port function state change support
In reply to Saeed MahameedFrom: Parav Pandit <[email protected]>
Support changing the state of the SF port's function through devlink. When activating the SF port's function, enable the hca in the device followed by adding its auxiliary device.
When deactivating the SF port's function, delete its auxiliary device followed by disabling the vHCA.
Port function attributes get/set callbacks are invoked with devlink instance lock held. Such callbacks need to synchronize with sf port table getting disabled either via sriov sysfs callback. Such callbacks synchronize with table disable context holding table refcount.
$ devlink dev eswitch set pci/0000:06:00.0 mode switchdev
$ devlink port show pci/0000:06:00.0/65535: type eth netdev ens2f0np0 flavour physical port 0 splittable false
$ devlink port add pci/0000:06:00.0 flavour pcisf pfnum 0 sfnum 88
$ devlink port show ens2f0npf0sf88
pci/0000:06:00.0/32768: type eth netdev ens2f0npf0sf88 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false
function:
hw_addr 00:00:00:00:88:88 state inactive opstate detached$ devlink port function set pci/0000:06:00.0/32768 hw_addr 00:00:00:00:88:88 state active
$ devlink port show ens2f0npf0sf88 -jp
{
"port": {
"pci/0000:06:00.0/32768": {
"type": "eth",
"netdev": "ens2f0npf0sf88",
"flavour": "pcisf",
"controller": 0,
"pfnum": 0,
"sfnum": 88,
"external": false,
"splittable": false,
"function": {
"hw_addr": "00:00:00:00:88:88",
"state": "active",
"opstate": "attached"
}
}
}
}On port function activation, an auxiliary device is created in below example.
$ devlink dev show devlink dev show auxiliary/mlx5_core.sf.4
$ devlink port show auxiliary/mlx5_core.sf.4/1 auxiliary/mlx5_core.sf.4/1: type eth netdev p0sf88 flavour virtual port 0 splittable false
--- .../net/ethernet/mellanox/mlx5/core/devlink.c | 2 + .../net/ethernet/mellanox/mlx5/core/main.c | 10 + .../net/ethernet/mellanox/mlx5/core/sf/cmd.c | 22 ++ .../ethernet/mellanox/mlx5/core/sf/devlink.c | 284 ++++++++++++++++-- .../ethernet/mellanox/mlx5/core/sf/hw_table.c | 116 ++++++- .../net/ethernet/mellanox/mlx5/core/sf/priv.h | 4 + .../net/ethernet/mellanox/mlx5/core/sf/sf.h | 19 ++ 7 files changed, 431 insertions(+), 26 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/devlink.c +2 -0
@@ -195,6 +195,8 @@ static const struct devlink_ops mlx5_devlink_ops = {#ifdef CONFIG_MLX5_SF_MANAGER .port_new = mlx5_devlink_sf_port_new, .port_del = mlx5_devlink_sf_port_del, + .port_function_state_get = mlx5_devlink_sf_port_fn_state_get, + .port_function_state_set = mlx5_devlink_sf_port_fn_state_set, #endif .flash_update = mlx5_devlink_flash_update, .info_get = mlx5_devlink_info_get,
drivers/net/ethernet/mellanox/mlx5/core/main.c +10 -0
@@ -75,6 +75,7 @@#include "diag/rsc_dump.h" #include "sf/vhca_event.h" #include "sf/dev/dev.h" +#include "sf/sf.h" MODULE_AUTHOR ("Eli Cohen <[email protected]>" );MODULE_DESCRIPTION ("Mellanox 5th generation network adapters (ConnectX series) core driver" );
@@ -1161,6 +1162,12 @@ static int mlx5_load(struct mlx5_core_dev *dev)mlx5_vhca_event_start (dev); + err =mlx5_sf_hw_table_create (dev); +if (err) { +mlx5_core_err (dev,"sf table create failed %d\n" , err); + goto err_vhca; + } + err =mlx5_ec_init (dev);if (err) {mlx5_core_err (dev,"Failed to init embedded CPU\n" );
@@ -1180,6 +1187,8 @@ static int mlx5_load(struct mlx5_core_dev *dev) err_sriov:mlx5_ec_cleanup (dev); err_ec: +mlx5_sf_hw_table_destroy (dev); +err_vhca:mlx5_vhca_event_stop (dev);mlx5_cleanup_fs (dev); err_fs:
@@ -1209,6 +1218,7 @@ static void mlx5_unload(struct mlx5_core_dev *dev)mlx5_sf_dev_table_destroy (dev);mlx5_sriov_detach (dev);mlx5_ec_cleanup (dev); +mlx5_sf_hw_table_destroy (dev);mlx5_vhca_event_stop (dev);mlx5_cleanup_fs (dev);mlx5_accel_ipsec_cleanup (dev);
drivers/net/ethernet/mellanox/mlx5/core/sf/cmd.c +22 -0
@@ -25,3 +25,25 @@ int mlx5_cmd_dealloc_sf(struct mlx5_core_dev *dev, u16 function_id)return mlx5_cmd_exec (dev, in,sizeof (in), out,sizeof (out)); } + +int mlx5_cmd_sf_enable_hca (struct mlx5_core_dev *dev,u16 func_id) +{ +u32 out[MLX5_ST_SZ_DW (enable_hca_out)] = {}; +u32 in[MLX5_ST_SZ_DW (enable_hca_in)] = {}; + +MLX5_SET (enable_hca_in, in, opcode,MLX5_CMD_OP_ENABLE_HCA ); +MLX5_SET (enable_hca_in, in, function_id, func_id); +MLX5_SET (enable_hca_in, in, embedded_cpu_function,0 ); +return mlx5_cmd_exec (dev, &in,sizeof (in), &out,sizeof (out)); +} + +int mlx5_cmd_sf_disable_hca (struct mlx5_core_dev *dev,u16 func_id) +{ +u32 out[MLX5_ST_SZ_DW (disable_hca_out)] = {}; +u32 in[MLX5_ST_SZ_DW (disable_hca_in)] = {}; + +MLX5_SET (disable_hca_in, in, opcode,MLX5_CMD_OP_DISABLE_HCA ); +MLX5_SET (disable_hca_in, in, function_id, func_id); +MLX5_SET (enable_hca_in, in, embedded_cpu_function,0 ); +return mlx5_cmd_exec (dev, in,sizeof (in), out,sizeof (out)); +}
drivers/net/ethernet/mellanox/mlx5/core/sf/devlink.c +262 -22
395 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/sf/hw_table.c +112 -4
186 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/sf/priv.h +4 -0
@@ -9,9 +9,13 @@int mlx5_cmd_alloc_sf (struct mlx5_core_dev *dev,u16 function_id);int mlx5_cmd_dealloc_sf (struct mlx5_core_dev *dev,u16 function_id); +int mlx5_cmd_sf_enable_hca (struct mlx5_core_dev *dev,u16 func_id); +int mlx5_cmd_sf_disable_hca (struct mlx5_core_dev *dev,u16 func_id); +u16 mlx5_sf_sw_to_hw_id (const struct mlx5_core_dev *dev,u16 sw_id);int mlx5_sf_hw_table_sf_alloc (struct mlx5_core_dev *dev,u32 usr_sfnum);void mlx5_sf_hw_table_sf_free (struct mlx5_core_dev *dev,u16 id); +void mlx5_sf_hw_table_sf_deferred_free (struct mlx5_core_dev *dev,u16 id);#endif
drivers/net/ethernet/mellanox/mlx5/core/sf/sf.h +19 -0
@@ -31,6 +31,9 @@ static inline u16 mlx5_sf_max_functions(const struct mlx5_core_dev *dev)int mlx5_sf_hw_table_init (struct mlx5_core_dev *dev);void mlx5_sf_hw_table_cleanup (struct mlx5_core_dev *dev); +int mlx5_sf_hw_table_create (struct mlx5_core_dev *dev); +void mlx5_sf_hw_table_destroy (struct mlx5_core_dev *dev); +int mlx5_sf_table_init (struct mlx5_core_dev *dev);void mlx5_sf_table_cleanup (struct mlx5_core_dev *dev);
@@ -38,6 +41,13 @@ int mlx5_devlink_sf_port_new(struct devlink *devlink, const struct devlink_port_ struct netlink_ext_ack *extack);int mlx5_devlink_sf_port_del (struct devlink *devlink,unsigned int port_index,struct netlink_ext_ack *extack); +int mlx5_devlink_sf_port_fn_state_get (struct devlink *devlink,struct devlink_port *dl_port, +enum devlink_port_function_state *state, +enum devlink_port_function_opstate *opstate, +struct netlink_ext_ack *extack); +int mlx5_devlink_sf_port_fn_state_set (struct devlink *devlink,struct devlink_port *dl_port, +enum devlink_port_function_state state, +struct netlink_ext_ack *extack);#else static inline bool mlx5_sf_supported (const struct mlx5_core_dev *dev)
@@ -59,6 +69,15 @@ static inline void mlx5_sf_hw_table_cleanup(struct mlx5_core_dev *dev) { } +static inline int mlx5_sf_hw_table_create (struct mlx5_core_dev *dev) +{ +return 0 ; +} + +static inline void mlx5_sf_hw_table_destroy (struct mlx5_core_dev *dev) +{ +} +static inline int mlx5_sf_table_init (struct mlx5_core_dev *dev) {return 0 ;
Signature · 1 line
2.26.2
PATCH v4 1/15 net/mlx5: Fix compilation warning for 32-bit platform
In reply to Saeed MahameedFrom: Parav Pandit <[email protected]>
MLX5_GENERAL_OBJECT_TYPES types bitfield is 64-bit field.
Defining an enum for such bit fields on 32-bit platform results in below warning.
./include/vdso/bits.h:7:26: warning: left shift count >= width of type [-Wshift-count-overflow]
^
./include/linux/mlx5/mlx5_ifc.h:10716:46: note: in expansion of macro ‘BIT’
MLX5_HCA_CAP_GENERAL_OBJECT_TYPES_SAMPLER = BIT(0x20),
^~~Use 32-bit friendly left shift.
--- include/linux/mlx5/mlx5_ifc.h | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
include/linux/mlx5/mlx5_ifc.h +3 -3
@@ -10711,9 +10711,9 @@ struct mlx5_ifc_affiliated_event_header_bits { };enum { -MLX5_HCA_CAP_GENERAL_OBJECT_TYPES_ENCRYPTION_KEY =BIT (0xc ), -MLX5_HCA_CAP_GENERAL_OBJECT_TYPES_IPSEC =BIT (0x13 ), -MLX5_HCA_CAP_GENERAL_OBJECT_TYPES_SAMPLER =BIT (0x20 ), +MLX5_HCA_CAP_GENERAL_OBJECT_TYPES_ENCRYPTION_KEY =1ULL <<0xc , +MLX5_HCA_CAP_GENERAL_OBJECT_TYPES_IPSEC =1ULL <<0x13 , +MLX5_HCA_CAP_GENERAL_OBJECT_TYPES_SAMPLER =1ULL <<0x20 , };enum {
Signature · 1 line
2.26.2
PATCH v4 2/15 devlink: Prepare code to fill multiple port function attributes
In reply to Saeed MahameedFrom: Parav Pandit <[email protected]>
Prepare code to fill zero or more port function optional attributes. Subsequent patch makes use of this to fill more port function attributes.
--- net/core/devlink.c | 63 +++++++++++++++++++++++----------------------- 1 file changed, 32 insertions(+), 31 deletions(-)
net/core/devlink.c +32 -31
@@ -712,6 +712,31 @@ static int devlink_nl_port_attrs_put(struct sk_buff *msg,return 0 ; } +static int +devlink_port_function_hw_addr_fill (struct devlink *devlink,const struct devlink_ops *ops, +struct devlink_port *port,struct sk_buff *msg, +struct netlink_ext_ack *extack,bool *msg_updated) +{ +u8 hw_addr[MAX_ADDR_LEN ]; +int hw_addr_len; +int err; + +if (!ops->port_function_hw_addr_get) +return 0 ; + + err = ops->port_function_hw_addr_get (devlink, port, hw_addr, &hw_addr_len, extack); +if (err) { +if (err == -EOPNOTSUPP ) +return 0 ; +return err; + } + err =nla_put (msg,DEVLINK_PORT_FUNCTION_ATTR_HW_ADDR , hw_addr_len, hw_addr); +if (err) +return err; + *msg_updated = true; +return 0 ; +} +static int devlink_nl_port_function_attrs_put (struct sk_buff *msg,struct devlink_port *port,struct netlink_ext_ack *extack)
@@ -719,36 +744,16 @@ devlink_nl_port_function_attrs_put(struct sk_buff *msg, struct devlink_port *porstruct devlink *devlink = port->devlink;const struct devlink_ops *ops;struct nlattr *function_attr; -bool empty_nest = true; -int err =0 ; +bool msg_updated = false; +int err; function_attr =nla_nest_start_noflag (msg,DEVLINK_ATTR_PORT_FUNCTION );if (!function_attr)return -EMSGSIZE ; ops = devlink->ops; -if (ops->port_function_hw_addr_get) { -int hw_addr_len; -u8 hw_addr[MAX_ADDR_LEN ]; - - err = ops->port_function_hw_addr_get (devlink, port, hw_addr, &hw_addr_len, extack); -if (err == -EOPNOTSUPP ) { -/* Port function attributes are optional for a port. If port doesn't -* support function attribute, returning -EOPNOTSUPP is not an error. -*/ - err =0 ; - goto out; - }else if (err) { - goto out; - } - err =nla_put (msg,DEVLINK_PORT_FUNCTION_ATTR_HW_ADDR , hw_addr_len, hw_addr); -if (err) - goto out; - empty_nest = false; - } - -out: -if (err || empty_nest) + err =devlink_port_function_hw_addr_fill (devlink, ops, port, msg, extack, &msg_updated); +if (err || !msg_updated)nla_nest_cancel (msg, function_attr);else nla_nest_end (msg, function_attr);
@@ -986,7 +991,6 @@ devlink_port_function_hw_addr_set(struct devlink *devlink, struct devlink_port *const struct devlink_ops *ops;const u8 *hw_addr;int hw_addr_len; -int err; hw_addr =nla_data (attr); hw_addr_len =nla_len (attr);
@@ -1011,12 +1015,7 @@ devlink_port_function_hw_addr_set(struct devlink *devlink, struct devlink_port *return -EOPNOTSUPP ; } - err = ops->port_function_hw_addr_set (devlink, port, hw_addr, hw_addr_len, extack); -if (err) -return err; - -devlink_port_notify (port,DEVLINK_CMD_PORT_NEW ); -return 0 ; +return ops->port_function_hw_addr_set (devlink, port, hw_addr, hw_addr_len, extack); } static int
@@ -1037,6 +1036,8 @@ devlink_port_function_set(struct devlink *devlink, struct devlink_port *port,if (attr) err =devlink_port_function_hw_addr_set (devlink, port, attr, extack); +if (!err) +devlink_port_notify (port,DEVLINK_CMD_PORT_NEW );return err; }
Signature · 1 line
2.26.2
PATCH v4 3/15 devlink: Introduce PCI SF port flavour and port attribute
In reply to Saeed MahameedFrom: Parav Pandit <[email protected]>
A PCI sub-function (SF) represents a portion of the device similar to PCI VF.
In an eswitch, PCI SF may have port which is normally represented using a representor netdevice.
To have better visibility of eswitch port, its association with SF, and its representor netdevice, introduce a PCI SF port flavour.
When devlink port flavour is PCI SF, fill up PCI SF attributes of the port.
Extend port name creation using PCI PF and SF number scheme on best effort basis, so that vendor drivers can skip defining their own scheme.
This is done as cApfNSfM, where A, N and M are controller, PCI PF and PCI SF number respectively.
This is similar to existing naming for PCI PF and PCI VF ports.
An example view of a PCI SF port:
$ devlink port show pci/0000:06:00.0/32768
pci/0000:06:00.0/32768: type eth netdev ens2f0npf0sf88 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false
function:
hw_addr 00:00:00:00:88:88 state active opstate attached$ devlink port show pci/0000:06:00.0/32768 -jp
{
"port": {
"pci/0000:06:00.0/32768": {
"type": "eth",
"netdev": "ens2f0npf0sf88",
"flavour": "pcisf",
"controller": 0,
"pfnum": 0,
"sfnum": 88,
"external": false,
"splittable": false,
"function": {
"hw_addr": "00:00:00:00:88:88",
"state": "active",
"opstate": "attached"
}
}
}
}--- include/net/devlink.h | 17 +++++++++++++ include/uapi/linux/devlink.h | 5 ++++ net/core/devlink.c | 46 ++++++++++++++++++++++++++++++++++++ 3 files changed, 68 insertions(+)
include/net/devlink.h +17 -0
@@ -93,6 +93,20 @@ struct devlink_port_pci_vf_attrs { u8 external:1; }; +/** +* struct devlink_port_pci_sf_attrs - devlink port's PCI SF attributes +* @controller: Associated controller number +* @pf: Associated PCI PF number for this port. +* @sf: Associated PCI SF for of the PCI PF for this port. +* @external: when set, indicates if a port is for an external controller +*/ +struct devlink_port_pci_sf_attrs { +u32 controller; +u16 pf; +u32 sf; +u8 external:1 ; +}; +/** * struct devlink_port_attrs - devlink port object * @flavour: flavour of the port
@@ -114,6 +128,7 @@ struct devlink_port_attrs {struct devlink_port_phys_attrs phys;struct devlink_port_pci_pf_attrs pci_pf;struct devlink_port_pci_vf_attrs pci_vf; +struct devlink_port_pci_sf_attrs pci_sf; }; };
@@ -1404,6 +1419,8 @@ void devlink_port_attrs_pci_pf_set(struct devlink_port *devlink_port, u32 contro u16 pf, bool external);void devlink_port_attrs_pci_vf_set (struct devlink_port *devlink_port,u32 controller,u16 pf,u16 vf,bool external); +void devlink_port_attrs_pci_sf_set (struct devlink_port *devlink_port,u32 controller, +u16 pf,u32 sf,bool external); int devlink_sb_register(struct devlink *devlink, unsigned int sb_index, u32 size, u16 ingress_pools_count, u16 egress_pools_count, u16 ingress_tc_count,
include/uapi/linux/devlink.h +5 -0
@@ -200,6 +200,10 @@ enum devlink_port_flavour {DEVLINK_PORT_FLAVOUR_UNUSED ,/* Port which exists in the switch, but * is not used in any way. */ + DEVLINK_PORT_FLAVOUR_PCI_SF, /* Represents eswitch port + * for the PCI SF. It is an internal + * port that faces the PCI SF. + */ }; enum devlink_param_cmode {
@@ -529,6 +533,7 @@ enum devlink_attr {DEVLINK_ATTR_RELOAD_ACTION_INFO ,/* nested */ DEVLINK_ATTR_RELOAD_ACTION_STATS ,/* nested */ +DEVLINK_ATTR_PORT_PCI_SF_NUMBER ,/* u32 */ /* add new attributes above here, update the policy in devlink.c */ __DEVLINK_ATTR_MAX,
net/core/devlink.c +46 -0
64 lines not on this page · raw message
Signature · 1 line
2.26.2
PATCH v4 4/15 devlink: Support add and delete devlink port
In reply to Saeed MahameedFrom: Parav Pandit <[email protected]>
Extended devlink interface for the user to add and delete port. Extend devlink to connect user requests to driver to add/delete such port in the device.
When driver routines are invoked, devlink instance lock is not held. This enables driver to perform several devlink objects registration, unregistration such as (port, health reporter, resource etc) by using exising devlink APIs.
This also helps to uniformly use the code for port unregistration during driver unload and during port deletion initiated by user.
Examples of add, show and delete commands:
$ devlink dev eswitch set pci/0000:06:00.0 mode switchdev
$ devlink port show pci/0000:06:00.0/65535: type eth netdev ens2f0np0 flavour physical port 0 splittable false
$ devlink port add pci/0000:06:00.0 flavour pcisf pfnum 0 sfnum 88
$ devlink port show pci/0000:06:00.0/32768
pci/0000:06:00.0/32768: type eth netdev eth0 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false
function:
hw_addr 00:00:00:00:88:88 state inactive opstate detached$ udevadm test-builtin net_id /sys/class/net/eth0 Load module index Parsed configuration file /usr/lib/systemd/network/99-default.link Created link configuration context. Using default interface naming scheme 'v245'. ID_NET_NAMING_SCHEME=v245 ID_NET_NAME_PATH=enp6s0f0npf0sf88 ID_NET_NAME_SLOT=ens2f0npf0sf88 Unload module index Unloaded link configuration context.
--- include/net/devlink.h | 39 ++++++++++++++++++++++++ net/core/devlink.c | 71 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 110 insertions(+)
include/net/devlink.h +39 -0
@@ -153,6 +153,17 @@ struct devlink_port {struct mutex reporters_lock;/* Protects reporter_list */ }; +struct devlink_port_new_attrs { +enum devlink_port_flavour flavour; +unsigned int port_index; +u32 controller; +u32 sfnum; +u16 pfnum; +u8 port_index_valid:1 , + controller_valid:1 , + sfnum_valid:1 ; +}; +struct devlink_sb_pool_info {enum devlink_sb_pool_type pool_type;u32 size;
@@ -1363,6 +1374,34 @@ struct devlink_ops {int (*port_function_hw_addr_set)(struct devlink *devlink,struct devlink_port *port,const u8 *hw_addr,int hw_addr_len,struct netlink_ext_ack *extack); +/** +* @port_new: Port add function. +* +* Should be used by device driver to let caller add new port of a +* specified flavour with optional attributes. +* Driver should return -EOPNOTSUPP if it doesn't support port addition +* of a specified flavour or specified attributes. Driver should set +* extack error message in case of fail to add the port. Devlink core +* does not hold a devlink instance lock when this callback is invoked. +* Driver must ensures synchronization when adding or deleting a port. +* Driver must register a port with devlink core. +*/ +int (*port_new)(struct devlink *devlink, +const struct devlink_port_new_attrs *attrs, +struct netlink_ext_ack *extack); +/** +* @port_del: Port delete function. +* +* Should be used by device driver to let caller delete port which was +* previously created using port_new() callback. +* Driver should return -EOPNOTSUPP if it doesn't support port deletion. +* Driver should set extack error message in case of fail to delete the +* port. Devlink core does not hold a devlink instance lock when this +* callback is invoked. Driver must ensures synchronization when adding +* or deleting a port. Driver must register a port with devlink core. +*/ +int (*port_del)(struct devlink *devlink,unsigned int port_index, +struct netlink_ext_ack *extack); };static inline void *devlink_priv (struct devlink *devlink)
net/core/devlink.c +71 -0
89 lines not on this page · raw message
Signature · 1 line
2.26.2
Re: [net-next v4 01/15] net/mlx5: Fix compilation warning for 32-bit platform
In reply to Saeed MahameedQuoted 29 lines · Saeed Mahameed
From: Parav Pandit <[email protected]>
MLX5_GENERAL_OBJECT_TYPEStypes bitfield is 64-bit field.Defining an enum for such bit fields on 32-bit platform results in below warning.
./include/vdso/bits.h:7:26: warning: left shift count >= width of type [-Wshift-count-overflow]^ ./include/linux/mlx5/mlx5_ifc.h:10716:46: note: in expansion of macro ‘BIT’ MLX5_HCA_CAP_GENERAL_OBJECT_TYPES_SAMPLER = BIT(0x20), ^~~Use 32-bit friendly left shift.
Fixes 2a2970891647 ("net/mlx5: Add sample offload hardware bits and structures")Signed-off-by Parav Pandit <[email protected]>Reported-by Stephen Rothwell <[email protected]>Signed-off-by Leon Romanovsky <[email protected]>Signed-off-by Saeed Mahameed <[email protected]>--- include/linux/mlx5/mlx5_ifc.h | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)include/linux/mlx5/mlx5_ifc.h +0
-0
include/linux/mlx5/mlx5_ifc.h struct mlx5_ifc_affiliated_event_header_bits { · line 10717
Quoted 10 lines
include/linux/mlx5/mlx5_ifc.h +3
-3@@ -10711,9 +10711,9 @@ struct mlx5_ifc_affiliated_event_header_bits { };enum { -MLX5_HCA_CAP_GENERAL_OBJECT_TYPES_ENCRYPTION_KEY =BIT (0xc ), -MLX5_HCA_CAP_GENERAL_OBJECT_TYPES_IPSEC =BIT (0x13 ), -MLX5_HCA_CAP_GENERAL_OBJECT_TYPES_SAMPLER =BIT (0x20 ), +MLX5_HCA_CAP_GENERAL_OBJECT_TYPES_ENCRYPTION_KEY =1ULL <<0xc , +MLX5_HCA_CAP_GENERAL_OBJECT_TYPES_IPSEC =1ULL <<0x13 , +MLX5_HCA_CAP_GENERAL_OBJECT_TYPES_SAMPLER =1ULL <<0x20 , };
Why not just use BIT_ULL?
Re: [net-next v4 01/15] net/mlx5: Fix compilation warning for 32-bit platform
Quoted 38 lines · Alexander Duyck
Quoted 36 lines · Saeed Mahameed
From: Parav Pandit <[email protected]>
MLX5_GENERAL_OBJECT_TYPEStypes bitfield is 64-bit field.Defining an enum for such bit fields on 32-bit platform results in below
warning.
./include/vdso/bits.h:7:26: warning: left shift count >= width of type [-Wshift-count-overflow]^ ./include/linux/mlx5/mlx5_ifc.h:10716:46: note: in expansion of macro ‘BIT’ MLX5_HCA_CAP_GENERAL_OBJECT_TYPES_SAMPLER = BIT(0x20), ^~~Use 32-bit friendly left shift.
Fixes 2a2970891647 ("net/mlx5: Add sample offload hardware bitsand structures")
Signed-off-by Parav Pandit <[email protected]>Reported-by Stephen Rothwell <[email protected]>Signed-off-by Leon Romanovsky <[email protected]>Signed-off-by Saeed Mahameed <[email protected]>--- include/linux/mlx5/mlx5_ifc.h | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)include/linux/mlx5/mlx5_ifc.h +0
-0@@ -10711,9 +10711,9 @@ struct mlx5_ifc_affiliated_event_header_bits { };
Quoted 12 lines
Quoted 10 lines
include/linux/mlx5/mlx5_ifc.h +3
-3enum { -MLX5_HCA_CAP_GENERAL_OBJECT_TYPES_ENCRYPTION_KEY = BIT(0xc), -MLX5_HCA_CAP_GENERAL_OBJECT_TYPES_IPSEC =BIT (0x13 ), -MLX5_HCA_CAP_GENERAL_OBJECT_TYPES_SAMPLER =BIT (0x20 ), +MLX5_HCA_CAP_GENERAL_OBJECT_TYPES_ENCRYPTION_KEY =1ULL <<0xc , +MLX5_HCA_CAP_GENERAL_OBJECT_TYPES_IPSEC =1ULL <<0x13 , +MLX5_HCA_CAP_GENERAL_OBJECT_TYPES_SAMPLER =1ULL <<0x20 ,};
Why not just use
BIT_ULL?
I was following the file convention where we use 1ULL/1UL in all of the places, I will consider changing the whole file to use BIT macros in another patch.
Quoted 13 lines · Saeed Mahameed
Hi Dave, Jakub, Jason,
This series form Parav was the theme of this mlx5 release cycle, we've been waiting anxiously for the auxbus infrastructure to make it into the kernel, and now as the auxbus is in and all the stars are aligned, I can finally submit this V2 of the devlink and mlx5 subfunction support.
Subfunctions came to solve the scaling issue of virtualization and switchdev environments, where SRIOV failed to deliver and users ran out of VFs very quickly as SRIOV demands huge amount of physical resources in both of the servers and the NIC.
Quoted 5 lines
Subfunction provide the same functionality as SRIOV but in a very lightweight manner, please see the thorough and detailed documentation from Parav below, in the commit messages and the Networking documentation patches at the end of this series.
Just to clarify a few things for myself. You mention virtualization and SR-IOV in your patch description but you cannot support direct assignment with this correct? The idea here is simply logical partitioning of an existing network interface, correct? So this isn't so much a solution for virtualization, but may work better for containers. I view this as an important distinction to make as the first thing that came to mind when I read this was mediated devices which is similar, but focused only on the virtualization case:
https://www.kernel.org/doc/html/v5.9/driver-api/vfio-mediated-device.html
Quoted 10 lines
Parav Pandit Says:
=================This patchset introduces support for mlx5 subfunction (SF).
A subfunction is a lightweight function that has a parent PCI function on which it is deployed. mlx5 subfunction has its own function capabilities and its own resources. This means a subfunction has its own dedicated
queues(txq, rxq, cq, eq). These queues are neither shared nor stealed from the parent PCI function.
Rather than calling this a subfunction, would it make more sense to call it something such as a queue set? It seems like this is exposing some of the same functionality we did in the Intel drivers such as ixgbe and i40e via the macvlan offload interface. However the ixgbe/i40e hardware was somewhat limited in that we were only able to expose Ethernet interfaces via this sort of VMQ/VMDQ feature, and even with that we have seen some limitations to the interface. It sounds like you are able to break out RDMA capable devices this way as well. So in terms of ways to go I would argue this is likely better. However one downside is that we are going to end up seeing each subfunction being different from driver to driver and vendor to vendor which I would argue was also one of the problems with SR-IOV as you end up with a bit of vendor lock-in as a result of this feature since each vendor will be providing a different interface.
Quoted 7 lines
When subfunction is RDMA capable, it has its own QP1, GID table and rdma resources neither shared nor stealed from the parent PCI function.
A subfunction has dedicated window in PCI BAR space that is not shared with ther other subfunctions or parent PCI function. This ensures that all class devices of the subfunction accesses only assigned PCI BAR space.
Quoted 6 lines
A Subfunction supports eswitch representation through which it supports tc offloads. User must configure eswitch to send/receive packets from/to subfunction port.
Subfunctions share PCI level resources such as PCI MSI-X IRQs with their other subfunctions and/or with its parent PCI function.
This piece to the architecture for this has me somewhat concerned. If all your resources are shared and you are allowing devices to be created incrementally you either have to pre-partition the entire function which usually results in limited resources for your base setup, or free resources from existing interfaces and redistribute them as things change. I would be curious which approach you are taking here? So for example if you hit a certain threshold will you need to reset the port and rebalance the IRQs between the various functions?
Quoted 42 lines
Patch summary:
--------------
Patch 1 to 4 prepares devlink
patch 5 to 7 mlx5 adds SF device support
Patch 8 to 11 mlx5 adds SF devlink port support
Patch 12 and 14 adds documentationPatch-1 prepares code to handle multiple port function attributes Patch-2 introduces devlink pcisf port flavour similar to pcipf and pcivf Patch-3 adds port add and delete driver callbacks
Patch-4 adds port function state get and set callbacks Patch-5 mlx5 vhca event notifier support to distribute subfunction
state change notification
Patch-6 adds SF auxiliary device
Patch-7 adds SF auxiliary driver
Patch-8 prepares eswitch to handler SF vport
Patch-9 adds eswitch helpers to add/remove SF vport
Patch-10 implements devlink port add/del callbacks
Patch-11 implements devlink port function get/set callbacks Patch-12 to 14 adds documentation
Patch-12 added mlx5 port function documentation
Patch-13 adds subfunction documentation
Patch-14 adds mlx5 subfunction documentationSubfunction support is discussed in detail in RFC [1] and [2]. RFC [1] and extension [2] describes requirements, design and proposed plumbing using devlink, auxiliary bus and sysfs for systemd/udev support. Functionality of this patchset is best explained using real examples further below.
overview:
--------
A subfunction can be created and deleted by a user using devlink port add/delete interface.A subfunction can be configured using devlink port function attribute before its activated.
When a subfunction is activated, it results in an auxiliary device on the host PCI device where it is deployed. A driver binds to the auxiliary device that further creates supported class devices.
Quoted 7 lines
example subfunction usage sequence:
-----------------------------------
Change device to switchdev mode:$ devlink dev eswitch set pci/0000:06:00.0 mode switchdevAdd a devlink port of subfunction flaovur:
$ devlink port add pci/0000:06:00.0 flavour pcisf pfnum 0 sfnum 88
Typo in your description. Also I don't know if you want to stick with "flavour" or just shorten it to the U.S. spelling which is "flavor".
Quoted 11 lines
Configure mac address of the port function:
$ devlink port function set ens2f0npf0sf88 hw_addr 00:00:00:00:88:88Now activate the function:
$ devlink port function set ens2f0npf0sf88 state activeNow use the auxiliary device and class devices:
$ devlink dev show pci/0000:06:00.0 auxiliary/mlx5_core.sf.4
Quoted 6 lines
$ ip link show
127: ens2f0np0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 24:8a:07:b3:d1:12 brd ff:ff:ff:ff:ff:ff
altname enp6s0f0np0
129: p0sf88: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 00:00:00:00:88:88 brd ff:ff:ff:ff:ff:ff>I assume that p0sf88 is supposed to be the newly created subfunction. However I thought the naming was supposed to be the same as what you are referring to in the devlink, or did I miss something?
Quoted 9 lines
$ rdma dev show 43: rdmap6s0f0: node_type ca fw 16.29.0550 node_guid 248a:0703:00b3:d112 sys_image_guid 248a:0703:00b3:d112 44: mlx5_0: node_type ca fw 16.29.0550 node_guid 0000:00ff:fe00:8888 sys_image_guid 248a:0703:00b3:d112After use inactivate the function:
$ devlink port function set ens2f0npf0sf88 state inactiveNow delete the subfunction port:
$ devlink port del ens2f0npf0sf88
This seems wrong to me as it breaks the symmetry with the port add command and assumes you have ownership of the interface in the host. I would much prefer to to see the same arguments that were passed to the add command being used to do the teardown as that would allow for the parent function to create the object, assign it to a container namespace, and not need to pull it back in order to destroy it.
Quoted 10 lines · Alexander Duyck
Quoted 7 lines
example subfunction usage sequence:
-----------------------------------
Change device to switchdev mode:$ devlink dev eswitch set pci/0000:06:00.0 mode switchdevAdd a devlink port of subfunction flaovur:
$ devlink port add pci/0000:06:00.0 flavour pcisf pfnum 0 sfnum 88Typo in your description. Also I don't know if you want to stick with "flavour" or just shorten it to the
U.S. spelling which is "flavor".
The term exists in devlink today (since 2018). When support was added to iproute2 I decided there was no reason to require the US spelling over the British spelling, so I accepted the patch.
Leon Romanovsky
maintainerRe: [net-next v4 01/15] net/mlx5: Fix compilation warning for 32-bit platform
In reply to Alexander DuyckQuoted 32 lines · Alexander Duyck
Quoted 31 lines · Saeed Mahameed
From: Parav Pandit <[email protected]>
MLX5_GENERAL_OBJECT_TYPEStypes bitfield is 64-bit field.Defining an enum for such bit fields on 32-bit platform results in below warning.
./include/vdso/bits.h:7:26: warning: left shift count >= width of type [-Wshift-count-overflow]^ ./include/linux/mlx5/mlx5_ifc.h:10716:46: note: in expansion of macro ‘BIT’ MLX5_HCA_CAP_GENERAL_OBJECT_TYPES_SAMPLER = BIT(0x20), ^~~Use 32-bit friendly left shift.
Fixes 2a2970891647 ("net/mlx5: Add sample offload hardware bits and structures")Signed-off-by Parav Pandit <[email protected]>Reported-by Stephen Rothwell <[email protected]>Signed-off-by Leon Romanovsky <[email protected]>Signed-off-by Saeed Mahameed <[email protected]>--- include/linux/mlx5/mlx5_ifc.h | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)include/linux/mlx5/mlx5_ifc.h +0
-0@@ -10711,9 +10711,9 @@ struct mlx5_ifc_affiliated_event_header_bits { };
Quoted 10 lines
include/linux/mlx5/mlx5_ifc.h struct mlx5_ifc_affiliated_event_header_bits { · line 10717
Quoted 8 lines
include/linux/mlx5/mlx5_ifc.h +3
-3enum { -MLX5_HCA_CAP_GENERAL_OBJECT_TYPES_ENCRYPTION_KEY =BIT (0xc ), -MLX5_HCA_CAP_GENERAL_OBJECT_TYPES_IPSEC =BIT (0x13 ), -MLX5_HCA_CAP_GENERAL_OBJECT_TYPES_SAMPLER =BIT (0x20 ), +MLX5_HCA_CAP_GENERAL_OBJECT_TYPES_ENCRYPTION_KEY =1ULL <<0xc , +MLX5_HCA_CAP_GENERAL_OBJECT_TYPES_IPSEC =1ULL <<0x13 , +MLX5_HCA_CAP_GENERAL_OBJECT_TYPES_SAMPLER =1ULL <<0x20 , };Why not just use
BIT_ULL?
mlx5_ifc.h doesn't include bits.h on-purpose and there are "*.c" files that include that ifc header file, but don't include bits.h either.
It can cause to build failures in random builds.
The mlx5_ifc.h is our main hardware definition file that we are using in other projects outside of the kernel (rdma-core) too, so it is preferable to keep it as plain-C without any extra dependencies.
Thanks
Quoted 7 lines
From: Alexander Duyck <[email protected]>
Sent: Tuesday, December 15, 2020 7:24 AMQuoted 1 line · Saeed Mahameed
Quoted 6 lines
Quoted 2 lines
Hi Dave, Jakub, Jason,
Just to clarify a few things for myself. You mention virtualization and SR-IOV in your patch description but you cannot support direct assignment with this correct?
Correct. it cannot be directly assigned.
Quoted 2 lines
The idea here is simply logical partitioning of an existing network interface, correct?
No. Idea is to spawn multiple functions from a single PCI device.
These functions are not born in PCI device and in OS until they are created by user.
Jason and Saeed explained this in great detail few weeks back in v0 version of the patchset at [1], [2] and [3]. I better not repeat all of it here again. Please go through it.
If you may want to read precursor to it, RFC from Jiri at [4] is also explains this in great detail.
Quoted 7 lines
So this isn't so much a solution for virtualization, but may
work better for containers. I view this as an important distinction to make as the first thing that came to mind when I read this was mediated devices which is similar, but focused only on the virtualization case:
https://www.kernel.org/doc/html/v5.9/driver-api/vfio-mediated-
device.html
Managing subfunction using medicated device is already ruled out last year at [5] as it is the abuse of the mdev bus for this purpose + has severe limitations of managing the subfunction device. We are not going back to it anymore.
It will be duplicating lot of the plumbing which exists in devlink, netlink, auxiliary bus and more.
Quoted 2 lines
Rather than calling this a subfunction, would it make more sense to call it something such as a queue set?
No, queue is just one way to send and receive data/packets.
Jason and Saeed explained and discussed this piece to you and others during v0 few weeks back at [1], [2], [3]. Please take a look.
Quoted 7 lines
So in terms of ways to go I would argue this is likely better. However one downside is that we are going to end up seeing each subfunction being different from driver to driver and vendor to vendor which I would argue was also one of the problems with SR-IOV as you end up with a bit of vendor lock-in as a result of this feature since each vendor will be providing a different interface.
Each and several vendors provided unified interface for managing VFs. i.e.
(a) enable/disable was via vendor neutral sysfs
(b) sriov capability exposed via standard pci capability and sysfs
(c) sriov vf config (mac, vlan, rss, tx rate, spoof check trust) are using vendor agnostic netlink Even though the driver's internal implementation largely differs on how trust, spoof, mac, vlan rate etc are enforced.
So subfunction feature/attribute/functionality will be implemented differently internally in the driver matching vendor's device, for reasonably abstract concept of 'subfunction'.
Quoted 9 lines
Quoted 6 lines
A Subfunction supports eswitch representation through which it supports tc offloads. User must configure eswitch to send/receive packets from/to subfunction port.
Subfunctions share PCI level resources such as PCI MSI-X IRQs with their other subfunctions and/or with its parent PCI function.
This piece to the architecture for this has me somewhat concerned. If all your resources are shared and
All resources are not shared.
Quoted 7 lines
you are allowing devices to be created
incrementally you either have to pre-partition the entire function which usually results in limited resources for your base setup, or free resources from existing interfaces and redistribute them as things change. I would be curious which approach you are taking here? So for example if you hit a certain threshold will you need to reset the port and rebalance the IRQs between the various functions?
No. Its works bit differently for mlx5 device.
When base function is started, it started as if it doesn't have any subfunctions.
When subfunction is instantiated, it spawns new resources in device (hw, fw, memory) depending on how much a function wants.
For example, PCI PF uses BAR 0, while subfunctions uses BAR 2.
For IRQs, subfunction instance shares the IRQ with its parent/hosting PCI PF. In future, yes, a dedicated IRQs per SF is likely desired.
Sridhar also talked about limiting number of queues to a subfunction. I believe there will be resources/attributes of the function to be controlled. devlink already provides rich interface to achieve that using devlink resources [8].
[..]
Quoted 6 lines
Quoted 2 lines
$ ip link show 127: ens2f0np0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop stateDOWN mode DEFAULT group default qlen 1000
Quoted 3 lines
link/ether 24:8a:07:b3:d1:12 brd ff:ff:ff:ff:ff:ff altname enp6s0f0np0129: p0sf88: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN
Quoted 7 lines
mode DEFAULT group default qlen 1000
Quoted 1 line
link/ether 00:00:00:00:88:88 brd ff:ff:ff:ff:ff:ff>I assume that p0sf88 is supposed to be the newly created subfunction. However I thought the naming was supposed to be the same as what you are referring to in the devlink, or did I miss something?
I believe you are confused with the representor netdevice of subfuction with devices of subfunction. (netdev, rdma, vdpa etc). I suggest that please refer to the diagram in patch_15 in [7] to see the stack, modules, objects.
Hope below description clarifies a bit.
There are two netdevices.
(a) representor netdevice, attached to the devlink port of the eswitch
(b) netdevice of the SF used by the end application (in your example, this is assigned to container).
Both netdevice follow obviously a different naming scheme.
Representor netdevice follows naming scheme well defined in kernel + systemd/udev v245 and higher. It is based on phys_port_name sysfs attribute.
This is same for existing PF and SF representors exist for year+ now. Further used by subfunction.
For subfunction netdevice (p0s88), system/udev will be extended. I put example based on my few lines of udev rule that reads phys_port_name and user supplied sfnum, so that user exactly knows which interface to assign to container.
Quoted 8 lines
Quoted 5 lines
After use inactivate the function:
$ devlink port function set ens2f0npf0sf88 state inactiveNow delete the subfunction port:
$ devlink port del ens2f0npf0sf88This seems wrong to me as it breaks the symmetry with the port add command and
Example of the representor device is only to make life easier for the user.
Devlink port del command works based on the devlink port index, just like existing devlink port commands (get,set,split,unsplit). I explained this in a thread with Sridhar at [6].
In short devlink port del <bus/device_name/port_index command is just fine.
Port index is unique handle for the devlink instance that user refers to delete, get, set port and port function attributes post its creation. I choose the representor netdev example because it is more intuitive to related to, but port index is equally fine and supported.
Quoted 5 lines
assumes you have ownership of the interface in the host. I
would much prefer to to see the same arguments that were passed to the add command being used to do the teardown as that would allow for the parent function to create the object, assign it to a container namespace, and not need to pull it back in order to destroy it.
Parent function will not have same netdevice name as that of representor netdevice, because both devices exist in single system for large part of the use cases. So port delete command works on the port index.
Host doesn't need to pull it back to destroy it. It is destroyed via port del command.
[1] https://lore.kernel.org/netdev/[email protected]/
[2] https://lore.kernel.org/netdev/[email protected]/
[3] https://lore.kernel.org/netdev/[email protected]/
[4] https://lore.kernel.org/netdev/[email protected]/
[5] https://lore.kernel.org/netdev/[email protected]/
[6] https://lore.kernel.org/netdev/BY5PR12MB43227784BB34D929CA64E315DCCA0@BY5PR12MB4322.namprd12.prod.outlook.com/
[7] https://lore.kernel.org/netdev/[email protected]/T/#u
[8] https://man7.org/linux/man-pages/man8/devlink-resource.8.html
Quoted 25 lines · Alexander Duyck
Quoted 17 lines · Saeed Mahameed
Hi Dave, Jakub, Jason,
This series form Parav was the theme of this mlx5 release cycle, we've been waiting anxiously for the auxbus infrastructure to make it into
the kernel, and now as the auxbus is in and all the stars are aligned, I
can finally submit this V2 of the devlink and mlx5 subfunction support.Subfunctions came to solve the scaling issue of virtualization and switchdev environments, where SRIOV failed to deliver and users ran
out of VFs very quickly as SRIOV demands huge amount of physical resources
in both of the servers and the NIC.Quoted 5 lines
Subfunction provide the same functionality as SRIOV but in a very lightweight manner, please see the thorough and detailed documentation from Parav below, in the commit messages and the Networking documentation patches at the end of this series.
Quoted 6 lines
Just to clarify a few things for myself. You mention virtualization and SR-IOV in your patch description but you cannot support direct assignment with this correct? The idea here is simply logical partitioning of an existing network interface, correct? So this isn't so much a solution for virtualization, but may work better for containers. I view this as an important distinction to make as the
at the current state yes, but the SF solution can be extended to support direct assignment, so this is why i think SF solution can do better and eventually replace SRIOV.
also many customers are currently using SRIOV with containers to get the performance and isolation features since there was no other options.
Quoted 19 lines
first thing that came to mind when I read this was mediated devices which is similar, but focused only on the virtualization case:
https://www.kernel.org/doc/html/v5.9/driver-api/vfio-mediated-device.htmlQuoted 5 lines
Parav Pandit Says:
=================This patchset introduces support for mlx5 subfunction (SF).
Quoted 9 lines
A subfunction is a lightweight function that has a parent PCI function on
which it is deployed. mlx5 subfunction has its own function capabilities
and its own resources. This means a subfunction has its own dedicatedqueues(txq, rxq, cq, eq). These queues are neither shared nor stealed from
the parent PCI function.
Quoted 10 lines
Rather than calling this a subfunction, would it make more sense to call it something such as a queue set? It seems like this is exposing some of the same functionality we did in the Intel drivers such as ixgbe and i40e via the macvlan offload interface. However the ixgbe/i40e hardware was somewhat limited in that we were only able to expose Ethernet interfaces via this sort of VMQ/VMDQ feature, and even
with that we have seen some limitations to the interface. It sounds like you are able to break out RDMA capable devices this way as well. So in terms of ways to go I would argue this is likely better.
We've discussed this thoroughly on V0, the SF solutions is closer to a VF than a VMDQ, this is not just a set of queues.
https://lore.kernel.org/linux-rdma/[email protected]/
Quoted 7 lines
However
one downside is that we are going to end up seeing each subfunction being different from driver to driver and vendor to vendor which I would argue was also one of the problems with SR-IOV as you end up with a bit of vendor lock-in as a result of this feature since each vendor will be providing a different interface.
I disagree, SFs are tightly coupled with switchdev model and devlink functions port, they are backed with the a well defined model, i can say the same about sriov with switchdev mode, this sort of vendor lock- in issues is eliminated when you migrate to switchdev mode.
Quoted 17 lines
Quoted 17 lines
When subfunction is RDMA capable, it has its own QP1, GID table and rdma
resources neither shared nor stealed from the parent PCI function.A subfunction has dedicated window in PCI BAR space that is not shared
with ther other subfunctions or parent PCI function. This ensures that all
class devices of the subfunction accesses only assigned PCI BAR space.A Subfunction supports eswitch representation through which it supports tc
offloads. User must configure eswitch to send/receive packets from/to
subfunction port.
Quoted 5 lines
Quoted 2 lines
Subfunctions share PCI level resources such as PCI MSI-X IRQs with their other subfunctions and/or with its parent PCI function.
This piece to the architecture for this has me somewhat concerned. If all your resources are shared and you are allowing devices to be
not all, only PCI MSIX, for now..
Quoted 8 lines
created incrementally you either have to pre-partition the entire function which usually results in limited resources for your base setup, or free resources from existing interfaces and redistribute them as things change. I would be curious which approach you are taking here? So for example if you hit a certain threshold will you need to reset the port and rebalance the IRQs between the various functions?
Currently SFs will use whatever IRQs the PF has pre-allocated for itself, so there is no IRQ limit issue at the moment, we are considering a dynamic IRQ pool with dynamic balancing, or even better us the IMS approach, which perfectly fits the SF architecture.
https://patchwork.kernel.org/project/linux-pci/cover/[email protected]/
for internal resources the are fully isolated (not shared) and they are internally managed by FW exactly like a VF internal resources.
Quoted 10 lines · David Ahern
Quoted 8 lines · Alexander Duyck
Quoted 7 lines
example subfunction usage sequence:
-----------------------------------
Change device to switchdev mode:$ devlink dev eswitch set pci/0000:06:00.0 mode switchdevAdd a devlink port of subfunction flaovur:
$ devlink port add pci/0000:06:00.0 flavour pcisf pfnum 0 sfnum 88
Quoted 6 lines
Quoted 2 lines
Typo in your description. Also I don't know if you want to stick with "flavour" or just shorten it to the
U.S. spelling which is "flavor".The term exists in devlink today (since 2018). When support was added to iproute2 I decided there was no reason to require the US spelling over the British spelling, so I accepted the patch.
Okay. The only reason why I noticed is because "flaovur" is definitely a wrong spelling. If it is already in the interface then no need to change it.
Quoted 19 lines
From: Alexander Duyck <[email protected]>
Sent: Tuesday, December 15, 2020 9:47 PMQuoted 13 lines · David Ahern
Quoted 10 lines · Alexander Duyck
Quoted 7 lines
example subfunction usage sequence:
-----------------------------------
Change device to switchdev mode:$ devlink dev eswitch set pci/0000:06:00.0 mode switchdevAdd a devlink port of subfunction flaovur:
$ devlink port add pci/0000:06:00.0 flavour pcisf pfnum 0 sfnum 88Typo in your description. Also I don't know if you want to stick with "flavour" or just shorten it to the
U.S. spelling which is "flavor".
Quoted 7 lines
Quoted 4 lines
The term exists in devlink today (since 2018). When support was added to
iproute2 I decided there was no reason to require the US spelling over the British spelling, so I accepted the patch.Okay. The only reason why I noticed is because "flaovur" is definitely a wrong spelling. If it is already in the interface then no need to change it.
I am using to write "flavor" and I realized that I should probably say "flavour" because in devlink it is that way. So I added 'u' and typo added it at wrong location. :-)
Thanks for catching it.
Saeed sent the v5 fixing this along with few more English corrections.
Quoted 14 lines · Parav Pandit
Quoted 8 lines
From: Alexander Duyck <[email protected]>
Sent: Tuesday, December 15, 2020 7:24 AMQuoted 1 line · Saeed Mahameed
Quoted 6 lines
Quoted 2 lines
Hi Dave, Jakub, Jason,
Just to clarify a few things for myself. You mention virtualization and SR-IOV in your patch description but you cannot support direct assignment with this correct?
Quoted 6 lines
Correct. it cannot be directly assigned.
Quoted 2 lines
The idea here is simply logical partitioning of an existing network interface, correct?
No. Idea is to spawn multiple functions from a single PCI device. These functions are not born in PCI device and in OS until they are created by user.
That is the definition of logical partitioning. You are essentially taking one physical PCIe function and splitting up the resources over multiple logical devices. With something like an MFD driver you would partition the device as soon as the driver loads, but with this you are peeling our resources and defining the devices on demand.
Quoted 3 lines
Jason and Saeed explained this in great detail few weeks back in v0 version of the patchset at [1], [2] and [3]. I better not repeat all of it here again. Please go through it.
If you may want to read precursor to it, RFC from Jiri at [4] is also explains this in great detail.
I think I have a pretty good idea of how the feature works. My concern is more the use of marketing speak versus actual functionality. The way this is being setup it sounds like it is useful for virtualization and it is not, at least in its current state. It may be at some point in the future but I worry that it is really going to muddy the waters as we end up with yet another way to partition devices.
Quoted 8 lines
Quoted 7 lines
So this isn't so much a solution for virtualization, but may
work better for containers. I view this as an important distinction to make as the first thing that came to mind when I read this was mediated devices which is similar, but focused only on the virtualization case:
https://www.kernel.org/doc/html/v5.9/driver-api/vfio-mediated-
device.htmlManaging subfunction using medicated device is already ruled out last year at [5] as it is the abuse of the mdev bus for this purpose + has severe limitations of managing the subfunction device.
I agree with you on that. My thought was more the fact that the two can be easily confused. If we are going to do this we need to define that for networking devices perhaps that using the mdev interface would be deprecated and we would need to go through devlink. However before we do that we need to make sure we have this completely standardized.
Quoted 2 lines
We are not going back to it anymore.
It will be duplicating lot of the plumbing which exists in devlink, netlink, auxiliary bus and more.
That is kind of my point. It is already in the kernel. What you are adding is the stuff that is duplicating it. I'm assuming that in order to be able to virtualize your interfaces in the future you are going to have to make use of the same vfio plumbing that the mediated devices do.
Quoted 5 lines
Quoted 2 lines
Rather than calling this a subfunction, would it make more sense to call it something such as a queue set?
No, queue is just one way to send and receive data/packets.
Jason and Saeed explained and discussed this piece to you and others during v0 few weeks back at [1], [2], [3]. Please take a look.
Yeah, I recall that. However I feel like it is being oversold. It isn't "SR-IOV done right" it seems more like "VMDq done better". The fact that interrupts are shared between the subfunctions is telling. That is exactly how things work for Intel parts when they do VMDq as well. The queues are split up into pools and a block of queues belongs to a specific queue. From what I can can tell the only difference is that there is isolation of the pool into specific pages in the BAR. Which is essentially a requirement for mediated devices so that they can be direct assigned.
Quoted 7 lines
Quoted 7 lines
So in terms of ways to go I would argue this is likely better. However one downside is that we are going to end up seeing each subfunction being different from driver to driver and vendor to vendor which I would argue was also one of the problems with SR-IOV as you end up with a bit of vendor lock-in as a result of this feature since each vendor will be providing a different interface.
Quoted 7 lines
Each and several vendors provided unified interface for managing VFs. i.e.
(a) enable/disable was via vendor neutral sysfs
(b) sriov capability exposed via standard pci capability and sysfs
(c) sriov vf config (mac, vlan, rss, tx rate, spoof check trust) are using vendor agnostic netlink Even though the driver's internal implementation largely differs on how trust, spoof, mac, vlan rate etc are enforced.So subfunction feature/attribute/functionality will be implemented differently internally in the driver matching vendor's device, for reasonably abstract concept of 'subfunction'.
I think you are missing the point. The biggest issue with SR-IOV adoption was the fact that the drivers all created different VF interfaces and those interfaces didn't support migration. That was the two biggest drawbacks for SR-IOV. I don't really see this approach resolving either of those and so that is one of the reasons why I say this is closer to "VMDq done better" rather than "SR-IOV done right". Assuming at some point one of the flavours is a virtio-net style interface you could eventually get to the point of something similar to what seems to have been the goal of mdev which was meant to address these two points.
Quoted 10 lines
Quoted 9 lines
Quoted 6 lines
A Subfunction supports eswitch representation through which it supports tc offloads. User must configure eswitch to send/receive packets from/to subfunction port.
Subfunctions share PCI level resources such as PCI MSI-X IRQs with their other subfunctions and/or with its parent PCI function.
This piece to the architecture for this has me somewhat concerned. If all your resources are shared and
All resources are not shared.
Just to clarify, when I say "shared" I mean that they are all coming from the same function. So if for example I were to direct-assign the PF then all the resources for the subfunctions would go with it.
Quoted 7 lines
Quoted 7 lines
you are allowing devices to be created
incrementally you either have to pre-partition the entire function which usually results in limited resources for your base setup, or free resources from existing interfaces and redistribute them as things change. I would be curious which approach you are taking here? So for example if you hit a certain threshold will you need to reset the port and rebalance the IRQs between the various functions?
Quoted 5 lines
No. Its works bit differently for mlx5 device.
When base function is started, it started as if it doesn't have any subfunctions.
When subfunction is instantiated, it spawns new resources in device (hw, fw, memory) depending on how much a function wants.For example, PCI PF uses BAR 0, while subfunctions uses BAR 2.
In the grand scheme BAR doesn't really matter much. The assumption here is that resources are page aligned so that you can map the pages into a guest eventually.
Quoted 7 lines
For IRQs, subfunction instance shares the IRQ with its parent/hosting PCI PF. In future, yes, a dedicated IRQs per SF is likely desired.
Sridhar also talked about limiting number of queues to a subfunction. I believe there will be resources/attributes of the function to be controlled. devlink already provides rich interface to achieve that using devlink resources [8].[..]
So it sounds like the device firmware is pre-partitioining the resources and splitting them up between the PCI PF and your subfunctions. Although in your case it sounds like there are significantly more resources than you might find in an ixgbe interface for instance. :)
The point is that we should probably define some sort of standard and/or expectations on what should happen when you spawn a new interface. Would it be acceptable for the PF and existing subfunctions to have to reset if you need to rebalance the IRQ distribution, or should they not be disrupted when you spawn a new interface?
One of the things I prefer about the mediated device setup is the fact that it has essentially pre-partitioned things beforehand and you know how many of each type of interface you can spawn. Is there any information like that provided by this interface?
Quoted 13 lines
Quoted 6 lines
Quoted 2 lines
$ ip link show 127: ens2f0np0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop stateDOWN mode DEFAULT group default qlen 1000
Quoted 3 lines
link/ether 24:8a:07:b3:d1:12 brd ff:ff:ff:ff:ff:ff altname enp6s0f0np0129: p0sf88: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN
Quoted 7 lines
mode DEFAULT group default qlen 1000
Quoted 1 line
link/ether 00:00:00:00:88:88 brd ff:ff:ff:ff:ff:ff>I assume that p0sf88 is supposed to be the newly created subfunction. However I thought the naming was supposed to be the same as what you are referring to in the devlink, or did I miss something?
Quoted 6 lines
I believe you are confused with the representor netdevice of subfuction with devices of subfunction. (netdev, rdma, vdpa etc). I suggest that please refer to the diagram in
patch_15in [7] to see the stack, modules, objects.
Hope below description clarifies a bit.
There are two netdevices.
(a) representor netdevice, attached to the devlink port of the eswitch
(b) netdevice of the SF used by the end application (in your example, this is assigned to container).
Sorry, that wasn't clear from your example. So in this case you started in a namespace and the new device you created via devlink was spawned in the root namespace?
Quoted 7 lines
Both netdevice follow obviously a different naming scheme.
Representor netdevice follows naming scheme well defined in kernel + systemd/udev v245 and higher. It is based onphys_port_namesysfs attribute.
This is same for existing PF and SF representors exist for year+ now. Further used by subfunction.For subfunction netdevice (p0s88), system/udev will be extended. I put example based on my few lines of udev rule that reads
phys_port_nameand user supplied sfnum, so that user exactly knows which interface to assign to container.
Admittedly I have been out of the loop for the last couple years since I had switched over to memory management work for a while. It would be useful to include something that shows your created network interface in the example in addition to your switchdev port.
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
After use inactivate the function:
$ devlink port function set ens2f0npf0sf88 state inactiveNow delete the subfunction port:
$ devlink port del ens2f0npf0sf88
Quoted 8 lines
Quoted 2 lines
This seems wrong to me as it breaks the symmetry with the port add command and
Example of the representor device is only to make life easier for the user.
Devlink port del command works based on the devlink port index, just like existing devlink port commands (get,set,split,unsplit). I explained this in a thread with Sridhar at [6].
In short devlink port del <bus/device_name/port_index command is just fine.
Port index is unique handle for the devlink instance that user refers to delete, get, set port and port function attributes post its creation. I choose the representor netdev example because it is more intuitive to related to, but port index is equally fine and supported.
Okay then, that addresses my concern. I just wanted to make sure we weren't in some situation where you had to have the interface in order to remove it.
Quoted 18 lines
Quoted 5 lines
assumes you have ownership of the interface in the host. I
would much prefer to to see the same arguments that were passed to the add command being used to do the teardown as that would allow for the parent function to create the object, assign it to a container namespace, and not need to pull it back in order to destroy it.Parent function will not have same netdevice name as that of representor netdevice, because both devices exist in single system for large part of the use cases. So port delete command works on the port index.
Host doesn't need to pull it back to destroy it. It is destroyed via port del command.[1] https://lore.kernel.org/netdev/[email protected]/
[2] https://lore.kernel.org/netdev/[email protected]/
[3] https://lore.kernel.org/netdev/[email protected]/
[4] https://lore.kernel.org/netdev/[email protected]/
[5] https://lore.kernel.org/netdev/[email protected]/
[6] https://lore.kernel.org/netdev/BY5PR12MB43227784BB34D929CA64E315DCCA0@BY5PR12MB4322.namprd12.prod.outlook.com/
[7] https://lore.kernel.org/netdev/[email protected]/T/#u
[8] https://man7.org/linux/man-pages/man8/devlink-resource.8.html
Quoted 34 lines · Saeed Mahameed
Quoted 25 lines · Alexander Duyck
Quoted 17 lines · Saeed Mahameed
Hi Dave, Jakub, Jason,
This series form Parav was the theme of this mlx5 release cycle, we've been waiting anxiously for the auxbus infrastructure to make it into
the kernel, and now as the auxbus is in and all the stars are aligned, I
can finally submit this V2 of the devlink and mlx5 subfunction support.Subfunctions came to solve the scaling issue of virtualization and switchdev environments, where SRIOV failed to deliver and users ran
out of VFs very quickly as SRIOV demands huge amount of physical resources
in both of the servers and the NIC.Quoted 5 lines
Subfunction provide the same functionality as SRIOV but in a very lightweight manner, please see the thorough and detailed documentation from Parav below, in the commit messages and the Networking documentation patches at the end of this series.
Quoted 6 lines
Just to clarify a few things for myself. You mention virtualization and SR-IOV in your patch description but you cannot support direct assignment with this correct? The idea here is simply logical partitioning of an existing network interface, correct? So this isn't so much a solution for virtualization, but may work better for containers. I view this as an important distinction to make as the
Quoted 3 lines
at the current state yes, but the SF solution can be extended to support direct assignment, so this is why i think SF solution can do better and eventually replace SRIOV.
My only real concern is that this and mediated devices are essentially the same thing. When you start making this work for direct-assignment the only real difference becomes the switchdev and devlink interfaces. Basically this is netdev specific mdev versus the PCIe specific mdev.
Quoted 3 lines
also many customers are currently using SRIOV with containers to get the performance and isolation features since there was no other options.
There were, but you hadn't implemented them. The fact is the approach Intel had taken for that was offloaded macvlan.
I think the big thing we really should do if we are going to go this route is to look at standardizing what the flavours are that get created by the parent netdevice. Otherwise we are just creating the same mess we had with SRIOV all over again and muddying the waters of mediated devices.
Quoted 30 lines
Quoted 19 lines
first thing that came to mind when I read this was mediated devices which is similar, but focused only on the virtualization case:
https://www.kernel.org/doc/html/v5.9/driver-api/vfio-mediated-device.htmlQuoted 5 lines
Parav Pandit Says:
=================This patchset introduces support for mlx5 subfunction (SF).
Quoted 9 lines
A subfunction is a lightweight function that has a parent PCI function on
which it is deployed. mlx5 subfunction has its own function capabilities
and its own resources. This means a subfunction has its own dedicatedqueues(txq, rxq, cq, eq). These queues are neither shared nor stealed from
the parent PCI function.Quoted 10 lines
Rather than calling this a subfunction, would it make more sense to call it something such as a queue set? It seems like this is exposing some of the same functionality we did in the Intel drivers such as ixgbe and i40e via the macvlan offload interface. However the ixgbe/i40e hardware was somewhat limited in that we were only able to expose Ethernet interfaces via this sort of VMQ/VMDQ feature, and even
with that we have seen some limitations to the interface. It sounds like you are able to break out RDMA capable devices this way as well. So in terms of ways to go I would argue this is likely better.
Quoted 4 lines
We've discussed this thoroughly on V0, the SF solutions is closer to a VF than a VMDQ, this is not just a set of queues.
VMDq is more than just a set of queues. The fact is it is a pool of resources that get created to handle the requests for a specific VM. The extra bits that are added here are essentially stuff that was required to support mediated devices.
Quoted 8 lines
Quoted 7 lines
However
one downside is that we are going to end up seeing each subfunction being different from driver to driver and vendor to vendor which I would argue was also one of the problems with SR-IOV as you end up with a bit of vendor lock-in as a result of this feature since each vendor will be providing a different interface.
Quoted 4 lines
I disagree, SFs are tightly coupled with switchdev model and devlink functions port, they are backed with the a well defined model, i can say the same about sriov with switchdev mode, this sort of vendor lock- in issues is eliminated when you migrate to switchdev mode.
What you are talking about is the backend. I am talking about what is exposed to the user. The user is going to see a Mellanox device having to be placed into their container in order to support this. One of the advantages of the Intel approach was that the macvlan interface was generic so you could have an offloaded interface or not and the user wouldn't necessarily know. The offload could be disabled and the user would be none the wiser as it is moved from one interface to another. I see that as a big thing that is missing in this solution.
Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
When subfunction is RDMA capable, it has its own QP1, GID table and rdma
resources neither shared nor stealed from the parent PCI function.A subfunction has dedicated window in PCI BAR space that is not shared
with ther other subfunctions or parent PCI function. This ensures that all
class devices of the subfunction accesses only assigned PCI BAR space.A Subfunction supports eswitch representation through which it supports tc
offloads. User must configure eswitch to send/receive packets from/to
subfunction port.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Subfunctions share PCI level resources such as PCI MSI-X IRQs with their other subfunctions and/or with its parent PCI function.
This piece to the architecture for this has me somewhat concerned. If all your resources are shared and you are allowing devices to be
not all, only PCI MSIX, for now..
They aren't shared after you partition them but they are coming from the same device. Basically you are subdividing the BAR2 in order to generate the subfunctions. BAR2 is a shared resource in my point of view.
Quoted 9 lines
Quoted 8 lines
created incrementally you either have to pre-partition the entire function which usually results in limited resources for your base setup, or free resources from existing interfaces and redistribute them as things change. I would be curious which approach you are taking here? So for example if you hit a certain threshold will you need to reset the port and rebalance the IRQs between the various functions?
Quoted 5 lines
Currently SFs will use whatever IRQs the PF has pre-allocated for itself, so there is no IRQ limit issue at the moment, we are considering a dynamic IRQ pool with dynamic balancing, or even better us the IMS approach, which perfectly fits the SF architecture.
https://patchwork.kernel.org/project/linux-pci/cover/[email protected]/
When you say you are using the PF's interrupts are you just using that as a pool of resources or having the interrupt process interrupts for both the PF and SFs? Without IMS you are limited to 2048 interrupts. Moving over to that would make sense since SF is similar to mdev in the way it partitions up the device and resources.
Quoted 2 lines
for internal resources the are fully isolated (not shared) and they are internally managed by FW exactly like a VF internal resources.
I assume by isolated you mean they are contained within page aligned blocks like what was required for mdev?
Quoted 25 lines · Alexander Duyck
Quoted 14 lines · Parav Pandit
Quoted 6 lines
From: Alexander Duyck <[email protected]>
Sent: Tuesday, December 15, 2020 7:24 AMOn Mon, Dec 14, 2020 at 1:49 PM Saeed Mahameed <[email protected]> wrote:
Quoted 8 lines
Quoted 2 lines
Hi Dave, Jakub, Jason,
Just to clarify a few things for myself. You mention virtualization and SR-IOV
in your patch description but you cannot support direct assignment with this
correct?Quoted 8 lines
Correct. it cannot be directly assigned.
Quoted 3 lines
The idea here is simply logical partitioning of an existing network
interface, correct?No. Idea is to spawn multiple functions from a single PCI device. These functions are not born in PCI device and in OS until they are created by user.
Quoted 6 lines
That is the definition of logical partitioning. You are essentially taking one physical PCIe function and splitting up the resources over multiple logical devices. With something like an MFD driver you would partition the device as soon as the driver loads, but with this you are peeling our resources and defining the devices on demand.
Sure, same for SRIOV and same for VMDQ. they are all logical partitioning and they are all sharing the same resources of the system. our point here is that the SF mechanisms are more similar to SRIOV than VMDQ, other than sharing the MSIX vectors and slicing up the BARs, everything else works exactly like SRIOV.
Quoted 6 lines
Quoted 5 lines
Jason and Saeed explained this in great detail few weeks back in v0 version of the patchset at [1], [2] and [3].
I better not repeat all of it here again. Please go through it. If you may want to read precursor to it, RFC from Jiri at [4] is also explains this in great detail.
Quoted 9 lines
I think I have a pretty good idea of how the feature works. My concern
is more the use of marketing speak versus actual functionality. The way this is being setup it sounds like it is useful for virtualization
and it is not, at least in its current state. It may be at some point in the future but I worry that it is really going to muddy the waters as we end up with yet another way to partition devices.
Ok, maybe we have different views on the feature and use cases, this is useful for visualization for various reasons, take vdpa for an instance, but anyway, we can improve documentation to address your concerns and at some point in the future when we add the direct assignment we can add the necessary documentation.
Quoted 14 lines
Quoted 9 lines
Quoted 9 lines
So this isn't so much a solution for virtualization, but may work better for containers. I view this as an important distinction to make as
the first thing that came to mind when I read this was mediated devices
which is similar, but focused only on the virtualization case:
https://www.kernel.org/doc/html/v5.9/driver-api/vfio-mediated-
device.htmlQuoted 4 lines
Managing subfunction using medicated device is already ruled out last year at [5] as it is the abuse of the mdev bus for this purpose + has severe limitations of managing the subfunction device.
Quoted 7 lines
I agree with you on that. My thought was more the fact that the two can be easily confused. If we are going to do this we need to define that for networking devices perhaps that using the mdev interface would be deprecated and we would need to go through devlink. However before we do that we need to make sure we have this completely standardized.
Then lets keep this discussion for later, when we add the direct assignment support :)
Quoted 14 lines
Quoted 3 lines
We are not going back to it anymore.
It will be duplicating lot of the plumbing which exists in devlink, netlink, auxiliary bus and more.That is kind of my point. It is already in the kernel. What you are adding is the stuff that is duplicating it. I'm assuming that in order
to be able to virtualize your interfaces in the future you are going to have to make use of the same vfio plumbing that the mediated devices do.Quoted 3 lines
Quoted 3 lines
Rather than calling this a subfunction, would it make more sense to call it
something such as a queue set?
Quoted 7 lines
Quoted 4 lines
No, queue is just one way to send and receive data/packets. Jason and Saeed explained and discussed this piece to you and others during v0 few weeks back at [1], [2], [3]. Please take a look.
Yeah, I recall that. However I feel like it is being oversold. It isn't "SR-IOV done right" it seems more like "VMDq done better". The
Ok then will improve documentation to not oversell this as "SRIOV done right".
Quoted 9 lines
fact that interrupts are shared between the subfunctions is telling. That is exactly how things work for Intel parts when they do VMDq as well. The queues are split up into pools and a block of queues belongs
to a specific queue. From what I can can tell the only difference is that there is isolation of the pool into specific pages in the BAR. Which is essentially a requirement for mediated devices so that they can be direct assigned.
I disagree, this is very dissimilar to VDMQ.
SF is a VF without a unique PCI function and bar, the BAR is split up to give the "SF" access to its own partition in the HW, and then it will access the HW exactly like a VF would do, there are no "pools" involved here or any PF resource/queue management, from our HW architecture perspective there is no difference between SF and VF.. really.
Quoted 25 lines
Quoted 21 lines
Quoted 12 lines
So in terms of ways to go I would argue this is likely better. However one
downside is that we are going to end up seeing each subfunction being
different from driver to driver and vendor to vendor which I would argue
was also one of the problems with SR-IOV as you end up with a bit of vendor
lock-in as a result of this feature since each vendor will be providing a
different interface.Each and several vendors provided unified interface for managing VFs. i.e.
(a) enable/disable was via vendor neutral sysfs
(b) sriov capability exposed via standard pci capability and sysfs
(c) sriov vf config (mac, vlan, rss, tx rate, spoof check trust) are using vendor agnostic netlink
Even though the driver's internal implementation largely differs on how trust, spoof, mac, vlan rate etc are enforced.Quoted 3 lines
So subfunction feature/attribute/functionality will be implemented differently internally in the driver matching vendor's device, for reasonably abstract concept of 'subfunction'.
Quoted 14 lines
I think you are missing the point. The biggest issue with SR-IOV adoption was the fact that the drivers all created different VF interfaces and those interfaces didn't support migration. That was the
two biggest drawbacks for SR-IOV. I don't really see this approach resolving either of those and so that is one of the reasons why I say this is closer to "VMDq done better" rather than "SR-IOV done right".
Assuming at some point one of the flavours is a virtio-net style interface you could eventually get to the point of something similar to what seems to have been the goal of mdev which was meant to address
these two points.
Improving documentation will address these concerns ?
but to be clear it is much easier to solve sriov live migration when SFs are involved.
Quoted 184 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
A Subfunction supports eswitch representation through which it supports tc offloads. User must configure eswitch to send/receive
packets from/to subfunction port.Quoted 8 lines
Quoted 7 lines
Quoted 3 lines
Subfunctions share PCI level resources such as PCI MSI-X IRQs with
their other subfunctions and/or with its parent PCI function.This piece to the architecture for this has me somewhat concerned. If all your
resources are shared andAll resources are not shared.
Just to clarify, when I say "shared" I mean that they are all coming from the same function. So if for example I were to direct-assign the PF then all the resources for the subfunctions would go with it.
Quoted 12 lines
Quoted 4 lines
you are allowing devices to be created
incrementally you either have to pre-partition the entire function which
usually results in limited resources for your base setup, or freeQuoted 8 lines
resources
from existing interfaces and redistribute them as things change. I would be
curious which approach you are taking here? So for example if you hit a
certain threshold will you need to reset the port and rebalance the IRQs
between the various functions?Quoted 7 lines
No. Its works bit differently for mlx5 device.
When base function is started, it started as if it doesn't have any subfunctions.
When subfunction is instantiated, it spawns new resources in device (hw, fw, memory) depending on how much a function wants.For example, PCI PF uses BAR 0, while subfunctions uses BAR 2.
In the grand scheme BAR doesn't really matter much. The assumption here is that resources are page aligned so that you can map the pages into a guest eventually.
Quoted 11 lines
For IRQs, subfunction instance shares the IRQ with its parent/hosting PCI PF.
In future, yes, a dedicated IRQs per SF is likely desired. Sridhar also talked about limiting number of queues to a subfunction.
I believe there will be resources/attributes of the function to be controlled.
devlink already provides rich interface to achieve that using devlink resources [8].[..]
So it sounds like the device firmware is pre-partitioining the resources and splitting them up between the PCI PF and your subfunctions. Although in your case it sounds like there are significantly more resources than you might find in an ixgbe interface
for instance. :)The point is that we should probably define some sort of standard and/or expectations on what should happen when you spawn a new interface. Would it be acceptable for the PF and existing subfunctions
to have to reset if you need to rebalance the IRQ distribution, or should they not be disrupted when you spawn a new interface?One of the things I prefer about the mediated device setup is the fact
that it has essentially pre-partitioned things beforehand and you know
how many of each type of interface you can spawn. Is there any information like that provided by this interface?Quoted 16 lines
Quoted 7 lines
Quoted 2 lines
$ ip link show 127: ens2f0np0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop stateDOWN mode DEFAULT group default qlen 1000
Quoted 4 lines
link/ether 24:8a:07:b3:d1:12 brd ff:ff:ff:ff:ff:ff altname enp6s0f0np0129: p0sf88: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN
Quoted 9 lines
mode DEFAULT group default qlen 1000
Quoted 1 line
link/ether 00:00:00:00:88:88 brd ff:ff:ff:ff:ff:ff>I assume that p0sf88 is supposed to be the newly created subfunction.
However I thought the naming was supposed to be the same as what you are
referring to in the devlink, or did I miss something?Quoted 10 lines
I believe you are confused with the representor netdevice of subfuction with devices of subfunction. (netdev, rdma, vdpa etc). I suggest that please refer to the diagram in
patch_15in [7] to see the stack, modules, objects.
Hope below description clarifies a bit.
There are two netdevices.
(a) representor netdevice, attached to the devlink port of the eswitch
(b) netdevice of the SF used by the end application (in your example, this is assigned to container).Sorry, that wasn't clear from your example. So in this case you started in a namespace and the new device you created via devlink was spawned in the root namespace?
Quoted 11 lines
Both netdevice follow obviously a different naming scheme. Representor netdevice follows naming scheme well defined in kernel
+ systemd/udev v245 and higher.
It is based onphys_port_namesysfs attribute.
This is same for existing PF and SF representors exist for year+ now. Further used by subfunction.For subfunction netdevice (p0s88), system/udev will be extended. I put example based on my few lines of udev rule that reads
phys_port_nameand user supplied sfnum, so that user exactly knows which interface to assign to container.Admittedly I have been out of the loop for the last couple years since
I had switched over to memory management work for a while. It would be
useful to include something that shows your created network interface in the example in addition to your switchdev port.Quoted 9 lines
Quoted 9 lines
Quoted 5 lines
After use inactivate the function:
$ devlink port function set ens2f0npf0sf88 state inactiveNow delete the subfunction port:
$ devlink port del ens2f0npf0sf88This seems wrong to me as it breaks the symmetry with the port add
command andQuoted 13 lines
Example of the representor device is only to make life easier for the user.
Devlink port del command works based on the devlink port index, just like existing devlink port commands (get,set,split,unsplit). I explained this in a thread with Sridhar at [6].
In short devlink port del <bus/device_name/port_index command is just fine.
Port index is unique handle for the devlink instance that user refers to delete, get, set port and port function attributes post its creation.
I choose the representor netdev example because it is more intuitive to related to, but port index is equally fine and supported.Okay then, that addresses my concern. I just wanted to make sure we weren't in some situation where you had to have the interface in order
to remove it.Quoted 31 lines
Quoted 8 lines
assumes you have ownership of the interface in the host. I would much prefer to to see the same arguments that were passed to the
add command being used to do the teardown as that would allow for the
parent function to create the object, assign it to a container namespace, and
not need to pull it back in order to destroy it.Parent function will not have same netdevice name as that of representor netdevice, because both devices exist in single system for large part of the use cases.
So port delete command works on the port index.
Host doesn't need to pull it back to destroy it. It is destroyed via port del command.[1]
https://lore.kernel.org/netdev/[email protected]/
[2]
https://lore.kernel.org/netdev/[email protected]/
[3]
https://lore.kernel.org/netdev/[email protected]/
[4]
https://lore.kernel.org/netdev/[email protected]/
[5]
https://lore.kernel.org/netdev/[email protected]/
[6]
https://lore.kernel.org/netdev/BY5PR12MB43227784BB34D929CA64E315DCCA0@BY5PR12MB4322.namprd12.prod.outlook.com/
[7]
https://lore.kernel.org/netdev/[email protected]/T/#u
[8] https://man7.org/linux/man-pages/man8/devlink-resource.8.html
Quoted 49 lines · Alexander Duyck
Quoted 42 lines · Saeed Mahameed
Quoted 30 lines · Alexander Duyck
Quoted 21 lines · Saeed Mahameed
Hi Dave, Jakub, Jason,
This series form Parav was the theme of this mlx5 release cycle,
we've been waiting anxiously for the auxbus infrastructure to make
it into
the kernel, and now as the auxbus is in and all the stars are aligned, I
can finally submit this V2 of the devlink and mlx5 subfunction support.Subfunctions came to solve the scaling issue of virtualization and switchdev environments, where SRIOV failed to deliver and users
ran
out of VFs very quickly as SRIOV demands huge amount of physical
resources
in both of the servers and the NIC.Quoted 6 lines
Subfunction provide the same functionality as SRIOV but in a very
lightweight manner, please see the thorough and detailed documentation from Parav below, in the commit messages and the Networking documentation patches at the end of this series.Quoted 10 lines
Just to clarify a few things for myself. You mention virtualization
and SR-IOV in your patch description but you cannot support direct
assignment with this correct? The idea here is simply logical partitioning of an existing network interface, correct? So this isn't
so much a solution for virtualization, but may work better for containers. I view this as an important distinction to make as theQuoted 4 lines
at the current state yes, but the SF solution can be extended to support direct assignment, so this is why i think SF solution can do
better and eventually replace SRIOV.
Quoted 5 lines
My only real concern is that this and mediated devices are essentially
the same thing. When you start making this work for direct-assignment the only real difference becomes the switchdev and devlink interfaces.
not just devlink and switchdev, auxbus was also introduced to standardize some of the interfaces.
Quoted 2 lines
Basically this is netdev specific mdev versus the PCIe specific mdev.
SF is not netdev specific mdev .. :/
Quoted 8 lines
Quoted 4 lines
also many customers are currently using SRIOV with containers to get
the performance and isolation features since there was no other options.There were, but you hadn't implemented them. The fact is the approach Intel had taken for that was offloaded macvlan.
offloaded macvlan is just a macvlan with checksum/tso and gro.
macvlan can't provide RDMA, TC offloads, ethtool steering, PTP, vdpa
...
our SF provides the same set of features a VF can provide
Quoted 6 lines
I think the big thing we really should do if we are going to go this route is to look at standardizing what the flavours are that get created by the parent netdevice. Otherwise we are just creating the same mess we had with SRIOV all over again and muddying the waters of mediated devices.
yes in the near future we will be working on auxbus interfaces for auto-probing and user flavor selection, this is a must have feature for us.
Quoted 41 lines
Quoted 41 lines
Quoted 20 lines
first thing that came to mind when I read this was mediated devices
which is similar, but focused only on the virtualization case:
https://www.kernel.org/doc/html/v5.9/driver-api/vfio-mediated-device.htmlQuoted 5 lines
Parav Pandit Says:
=================This patchset introduces support for mlx5 subfunction (SF).
Quoted 9 lines
A subfunction is a lightweight function that has a parent PCI function on
which it is deployed. mlx5 subfunction has its own function capabilities
and its own resources. This means a subfunction has its own dedicatedqueues(txq, rxq, cq, eq). These queues are neither shared nor stealed from
the parent PCI function.Quoted 16 lines
Rather than calling this a subfunction, would it make more sense to
call it something such as a queue set? It seems like this is exposing
some of the same functionality we did in the Intel drivers such as
ixgbe and i40e via the macvlan offload interface. However the ixgbe/i40e hardware was somewhat limited in that we were only able to
expose Ethernet interfaces via this sort of VMQ/VMDQ feature, and even
with that we have seen some limitations to the interface. It sounds
like you are able to break out RDMA capable devices this way as well.
So in terms of ways to go I would argue this is likely better.We've discussed this thoroughly on V0, the SF solutions is closer to a
VF than a VMDQ, this is not just a set of queues.
Quoted 7 lines
VMDq is more than just a set of queues. The fact is it is a pool of resources that get created to handle the requests for a specific VM. The extra bits that are added here are essentially stuff that was required to support mediated devices.
VMDq pools are managed by the driver and only logically isolated in the kernel, SFs has no shared pool for transport resources (queues), SFs have their own isolated steering domains, processing engines, and HW objects, exactly like a VF.
Quoted 20 lines
Quoted 12 lines
Quoted 11 lines
However
one downside is that we are going to end up seeing each subfunction
being different from driver to driver and vendor to vendor which I
would argue was also one of the problems with SR-IOV as you end up
with a bit of vendor lock-in as a result of this feature since each
vendor will be providing a different interface.Quoted 7 lines
I disagree, SFs are tightly coupled with switchdev model and devlink
functions port, they are backed with the a well defined model, i can
say the same about sriov with switchdev mode, this sort of vendor lock-
in issues is eliminated when you migrate to switchdev mode.
Quoted 11 lines
What you are talking about is the backend. I am talking about what is exposed to the user. The user is going to see a Mellanox device having
to be placed into their container in order to support this. One of the
advantages of the Intel approach was that the macvlan interface was generic so you could have an offloaded interface or not and the user wouldn't necessarily know. The offload could be disabled and the user would be none the wiser as it is moved from one interface to another. I see that as a big thing that is missing in this solution.
You are talking about the basic netdev users, Sure there are users who would want a more generic netdev, so yes. but most of my customers are not like that, they want vdpa/rdma and heavy netdev offload such as encap/decap/crypto and driver xdp in their containers, the SF approach will make more sense to them than sriov and VMDq.
Quoted 28 lines
Quoted 28 lines
Quoted 27 lines
Quoted 20 lines
When subfunction is RDMA capable, it has its own QP1, GID table and
rdma
resources neither shared nor stealed from the parent PCI function.A subfunction has dedicated window in PCI BAR space that is not shared
with ther other subfunctions or parent PCI function. This ensures
that all
class devices of the subfunction accesses only assigned PCI BAR space.A Subfunction supports eswitch representation through which it supports tc
offloads. User must configure eswitch to send/receive packets from/to
subfunction port.Quoted 3 lines
Subfunctions share PCI level resources such as PCI MSI-X IRQs with
their other subfunctions and/or with its parent PCI function.This piece to the architecture for this has me somewhat concerned. If
all your resources are shared and you are allowing devices to be
Quoted 7 lines
Quoted 1 line
not all, only PCI MSIX, for now..
They aren't shared after you partition them but they are coming from the same device. Basically you are subdividing the BAR2 in order to generate the subfunctions. BAR2 is a shared resource in my point of view.
Sure, but it doesn't host any actual resources, only the communication channel with the HW partition, so other than the BAR and the msix the actual HW resources, steering pipelines, offloads and queues are totlay isolated and separated.
Quoted 18 lines
Quoted 11 lines
Quoted 10 lines
created incrementally you either have to pre-partition the entire function which usually results in limited resources for your base setup, or free resources from existing interfaces and redistribute
them as things change. I would be curious which approach you are taking here? So for example if you hit a certain threshold will you
need to reset the port and rebalance the IRQs between the various functions?Quoted 6 lines
Currently SFs will use whatever IRQs the PF has pre-allocated for itself, so there is no IRQ limit issue at the moment, we are considering a dynamic IRQ pool with dynamic balancing, or even better
us the IMS approach, which perfectly fits the SF architecture.
https://patchwork.kernel.org/project/linux-pci/cover/[email protected]/
Quoted 7 lines
When you say you are using the PF's interrupts are you just using that
as a pool of resources or having the interrupt process interrupts for both the PF and SFs? Without IMS you are limited to 2048 interrupts. Moving over to that would make sense since SF is similar to mdev in the way it partitions up the device and resources.
Yes moving to IMS is on the top of our priorities.
Quoted 6 lines
Quoted 3 lines
for internal resources the are fully isolated (not shared) and they are internally managed by FW exactly like a VF internal resources.
I assume by isolated you mean they are contained within page aligned blocks like what was required for mdev?
I mean they are isolated and abstracted in the FW, we don't really expose any resource directly to the BAR. the BAR is only used for communicating with the device, so VF and SF will work exactly the same the only difference is where they get their BAR and offsets from, everything else is just similar.
Quoted 138 lines · Parav Pandit
Quoted 7 lines
From: Alexander Duyck <[email protected]>
Sent: Tuesday, December 15, 2020 7:24 AMQuoted 1 line · Saeed Mahameed
Quoted 6 lines
Quoted 2 lines
Hi Dave, Jakub, Jason,
Just to clarify a few things for myself. You mention virtualization and SR-IOV in your patch description but you cannot support direct assignment with this correct?
Correct. it cannot be directly assigned.
Quoted 2 lines
The idea here is simply logical partitioning of an existing network interface, correct?
No. Idea is to spawn multiple functions from a single PCI device.
These functions are not born in PCI device and in OS until they are created by user.
Jason and Saeed explained this in great detail few weeks back in v0 version of the patchset at [1], [2] and [3]. I better not repeat all of it here again. Please go through it.
If you may want to read precursor to it, RFC from Jiri at [4] is also explains this in great detail.Quoted 7 lines
So this isn't so much a solution for virtualization, but may
work better for containers. I view this as an important distinction to make as the first thing that came to mind when I read this was mediated devices which is similar, but focused only on the virtualization case:
https://www.kernel.org/doc/html/v5.9/driver-api/vfio-mediated-
device.htmlManaging subfunction using medicated device is already ruled out last year at [5] as it is the abuse of the mdev bus for this purpose + has severe limitations of managing the subfunction device. We are not going back to it anymore.
It will be duplicating lot of the plumbing which exists in devlink, netlink, auxiliary bus and more.Quoted 2 lines
Rather than calling this a subfunction, would it make more sense to call it something such as a queue set?
No, queue is just one way to send and receive data/packets.
Jason and Saeed explained and discussed this piece to you and others during v0 few weeks back at [1], [2], [3]. Please take a look.Quoted 7 lines
So in terms of ways to go I would argue this is likely better. However one downside is that we are going to end up seeing each subfunction being different from driver to driver and vendor to vendor which I would argue was also one of the problems with SR-IOV as you end up with a bit of vendor lock-in as a result of this feature since each vendor will be providing a different interface.
Each and several vendors provided unified interface for managing VFs. i.e.
(a) enable/disable was via vendor neutral sysfs
(b) sriov capability exposed via standard pci capability and sysfs
(c) sriov vf config (mac, vlan, rss, tx rate, spoof check trust) are using vendor agnostic netlink Even though the driver's internal implementation largely differs on how trust, spoof, mac, vlan rate etc are enforced.So subfunction feature/attribute/functionality will be implemented differently internally in the driver matching vendor's device, for reasonably abstract concept of 'subfunction'.
Quoted 9 lines
Quoted 6 lines
A Subfunction supports eswitch representation through which it supports tc offloads. User must configure eswitch to send/receive packets from/to subfunction port.
Subfunctions share PCI level resources such as PCI MSI-X IRQs with their other subfunctions and/or with its parent PCI function.
This piece to the architecture for this has me somewhat concerned. If all your resources are shared and
All resources are not shared.
Quoted 7 lines
you are allowing devices to be created
incrementally you either have to pre-partition the entire function which usually results in limited resources for your base setup, or free resources from existing interfaces and redistribute them as things change. I would be curious which approach you are taking here? So for example if you hit a certain threshold will you need to reset the port and rebalance the IRQs between the various functions?No. Its works bit differently for mlx5 device.
When base function is started, it started as if it doesn't have any subfunctions.
When subfunction is instantiated, it spawns new resources in device (hw, fw, memory) depending on how much a function wants.For example, PCI PF uses BAR 0, while subfunctions uses BAR 2.
For IRQs, subfunction instance shares the IRQ with its parent/hosting PCI PF. In future, yes, a dedicated IRQs per SF is likely desired.
Sridhar also talked about limiting number of queues to a subfunction. I believe there will be resources/attributes of the function to be controlled. devlink already provides rich interface to achieve that using devlink resources [8].[..]
Quoted 6 lines
Quoted 2 lines
$ ip link show 127: ens2f0np0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop stateDOWN mode DEFAULT group default qlen 1000
Quoted 3 lines
link/ether 24:8a:07:b3:d1:12 brd ff:ff:ff:ff:ff:ff altname enp6s0f0np0129: p0sf88: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN
Quoted 7 lines
mode DEFAULT group default qlen 1000
Quoted 1 line
link/ether 00:00:00:00:88:88 brd ff:ff:ff:ff:ff:ff>I assume that p0sf88 is supposed to be the newly created subfunction. However I thought the naming was supposed to be the same as what you are referring to in the devlink, or did I miss something?
I believe you are confused with the representor netdevice of subfuction with devices of subfunction. (netdev, rdma, vdpa etc). I suggest that please refer to the diagram in
patch_15in [7] to see the stack, modules, objects.
Hope below description clarifies a bit.
There are two netdevices.
(a) representor netdevice, attached to the devlink port of the eswitch
(b) netdevice of the SF used by the end application (in your example, this is assigned to container).Both netdevice follow obviously a different naming scheme.
Representor netdevice follows naming scheme well defined in kernel + systemd/udev v245 and higher. It is based onphys_port_namesysfs attribute.
This is same for existing PF and SF representors exist for year+ now. Further used by subfunction.For subfunction netdevice (p0s88), system/udev will be extended. I put example based on my few lines of udev rule that reads
phys_port_nameand user supplied sfnum, so that user exactly knows which interface to assign to container.Quoted 8 lines
Quoted 5 lines
After use inactivate the function:
$ devlink port function set ens2f0npf0sf88 state inactiveNow delete the subfunction port:
$ devlink port del ens2f0npf0sf88This seems wrong to me as it breaks the symmetry with the port add command and
Example of the representor device is only to make life easier for the user.
Devlink port del command works based on the devlink port index, just like existing devlink port commands (get,set,split,unsplit). I explained this in a thread with Sridhar at [6].
In short devlink port del <bus/device_name/port_index command is just fine.
Port index is unique handle for the devlink instance that user refers to delete, get, set port and port function attributes post its creation. I choose the representor netdev example because it is more intuitive to related to, but port index is equally fine and supported.Quoted 5 lines
assumes you have ownership of the interface in the host. I
would much prefer to to see the same arguments that were passed to the add command being used to do the teardown as that would allow for the parent function to create the object, assign it to a container namespace, and not need to pull it back in order to destroy it.Parent function will not have same netdevice name as that of representor netdevice, because both devices exist in single system for large part of the use cases. So port delete command works on the port index.
Host doesn't need to pull it back to destroy it. It is destroyed via port del command.
Quoted 9 lines
[1] https://lore.kernel.org/netdev/[email protected]/
[2] https://lore.kernel.org/netdev/[email protected]/
[3] https://lore.kernel.org/netdev/[email protected]/
[4] https://lore.kernel.org/netdev/[email protected]/
[5] https://lore.kernel.org/netdev/[email protected]/
[6] https://lore.kernel.org/netdev/BY5PR12MB43227784BB34D929CA64E315DCCA0@BY5PR12MB4322.namprd12.prod.outlook.com/
[7] https://lore.kernel.org/netdev/[email protected]/T/#u
[8] https://man7.org/linux/man-pages/man8/devlink-resource.8.html
Seems to be a repeated line of questions. You might want to add these FAQs, responses and references to the subfunction document once this set gets merged.
Jason Gunthorpe
maintainerQuoted 10 lines · Alexander Duyck
Quoted 3 lines
Jason and Saeed explained this in great detail few weeks back in v0 version of the patchset at [1], [2] and [3]. I better not repeat all of it here again. Please go through it.
If you may want to read precursor to it, RFC from Jiri at [4] is also explains this in great detail.I think I have a pretty good idea of how the feature works. My concern is more the use of marketing speak versus actual functionality. The way this is being setup it sounds like it is useful for virtualization and it is not, at least in its current state. It may be at some point in the future but I worry that it is really going to muddy the waters as we end up with yet another way to partition devices.
If we do a virtualization version then it will take a SF and instead of loading a mlx5_core on the SF aux device, we will load some vfio_mdev_mlx5 driver which will convert the SF aux device into a /dev/vfio/*
This is essentially the same as how you'd take a PCI VF and replace mlx5_core with vfio-pci to get /dev/vfio/*. It has to be a special mdev driver because it sits on the SF aux device, not on the VF PCI device.
The vfio_mdev_mlx5 driver will create what Intel calls an SIOV ADI from the SF, in other words the SF is already a superset of what a SIOV ADI should be.
This matches very nicely the driver model in Linux, and I don't think it becomes more muddied as we go along. If anything it is becoming more clear and sane as things progress.
Quoted 6 lines
I agree with you on that. My thought was more the fact that the two can be easily confused. If we are going to do this we need to define that for networking devices perhaps that using the mdev interface would be deprecated and we would need to go through devlink. However before we do that we need to make sure we have this completely standardized.
mdev is for creating /dev/vfio/* interfaces in userspace. Using it for anything else is a bad abuse of the driver model.
We had this debate endlessly already.
AFAIK, there is nothing to deprecate, there are no mdev_drivers in drivers/net, and none should ever be added. The only mdev_driver that should ever exists is in vfio_mdev.c
If someone is using a mdev_driver in drivers/net out of tree then they will need to convert to an aux driver for in-tree.
Quoted 3 lines
Yeah, I recall that. However I feel like it is being oversold. It isn't "SR-IOV done right" it seems more like "VMDq done better". The fact that interrupts are shared between the subfunctions is telling.
The interrupt sharing is a consequence of having an ADI-like model without relying on IMS. When IMS works then shared interrupts won't be very necessary. Otherwise there is no choice but to share the MSI table of the function.
Quoted 6 lines
That is exactly how things work for Intel parts when they do VMDq as well. The queues are split up into pools and a block of queues belongs to a specific queue. From what I can can tell the only difference is that there is isolation of the pool into specific pages in the BAR. Which is essentially a requirement for mediated devices so that they can be direct assigned.
No, I said this to Jakub, mlx5 SFs have very little to do with queues. There is no some 'queue' HW element that needs partitioning.
The SF is a hardware security boundary that wraps every operation a mlx5 device can do. This is why it is an ADI. It is not a crappy ADI that relies on hypervisor emulation, it is the real thing, just like a SRIOV VF. You stick it in the VM and the guest can directly talk to the HW. The HW provides the security.
I can't put focus on this enough: A mlx5 SF can run a *full RDMA stack*. This means the driver can create all the RDMA HW objects and resources under the SF. This is *not* just steering some ethernet traffic to a few different ethernet queues like VMDq is.
The Intel analog to a SF is a *full virtual function* on one of the Intel iWarp capable NICs, not VMDq.
Quoted 4 lines
Assuming at some point one of the flavours is a virtio-net style interface you could eventually get to the point of something similar to what seems to have been the goal of mdev which was meant to address these two points.
mlx5 already supports VDPA virtio-net on PF/VF and with this series SF too.
ie you can take a SF, bind the vdpa_mlx5 driver, and get a fully HW accelerated "ADI" that does virtio-net. This can be assigned to a guest and shows up as a PCI virtio-net netdev. With VT-d guest packet tx/rx on this netdev never uses the hypervisor CPU.
Quoted 5 lines
The point is that we should probably define some sort of standard and/or expectations on what should happen when you spawn a new interface. Would it be acceptable for the PF and existing subfunctions to have to reset if you need to rebalance the IRQ distribution, or should they not be disrupted when you spawn a new interface?
It is best to think of the SF as an ADI, so if you change something in the PF and that causes the driver attached to the ADI in a VM to reset, is that OK? I'd say no.
Jason
Jakub Kicinski
maintainerQuoted 9 lines · On Tue, 15 Dec 2020 12:35:20 -0800 Saeed Mahameed wrote:
Quoted 5 lines
I think the big thing we really should do if we are going to go this route is to look at standardizing what the flavours are that get created by the parent netdevice. Otherwise we are just creating the same mess we had with SRIOV all over again and muddying the waters of mediated devices.
yes in the near future we will be working on auxbus interfaces for auto-probing and user flavor selection, this is a must have feature for us.
Can you elaborate? I thought config would be via devlink.
Quoted 51 lines · Saeed Mahameed
Quoted 49 lines · Alexander Duyck
Quoted 46 lines · Saeed Mahameed
Quoted 30 lines · Alexander Duyck
Quoted 21 lines · Saeed Mahameed
Hi Dave, Jakub, Jason,
This series form Parav was the theme of this mlx5 release cycle,
we've been waiting anxiously for the auxbus infrastructure to make
it into
the kernel, and now as the auxbus is in and all the stars are aligned, I
can finally submit this V2 of the devlink and mlx5 subfunction support.Subfunctions came to solve the scaling issue of virtualization and switchdev environments, where SRIOV failed to deliver and users
ran
out of VFs very quickly as SRIOV demands huge amount of physical
resources
in both of the servers and the NIC.Quoted 6 lines
Subfunction provide the same functionality as SRIOV but in a very
lightweight manner, please see the thorough and detailed documentation from Parav below, in the commit messages and the Networking documentation patches at the end of this series.Quoted 10 lines
Just to clarify a few things for myself. You mention virtualization
and SR-IOV in your patch description but you cannot support direct
assignment with this correct? The idea here is simply logical partitioning of an existing network interface, correct? So this isn't
so much a solution for virtualization, but may work better for containers. I view this as an important distinction to make as theat the current state yes, but the SF solution can be extended to support direct assignment, so this is why i think SF solution can do
better and eventually replace SRIOV.
Quoted 8 lines
Quoted 5 lines
My only real concern is that this and mediated devices are essentially
the same thing. When you start making this work for direct-assignment the only real difference becomes the switchdev and devlink interfaces.not just devlink and switchdev, auxbus was also introduced to standardize some of the interfaces.
The auxbus is just there to make up for the fact that there isn't another bus type for this though. I would imagine otherwise this would be on some sort of platform bus.
Quoted 4 lines
Quoted 2 lines
Basically this is netdev specific mdev versus the PCIe specific mdev.
SF is not netdev specific mdev .. :/
I agree it is not. However there are just a few extensions to it. What I would really like to see is a solid standardization of what this is. Otherwise the comparison is going to be made. Especially since a year ago Mellanox was pushing this as an mdev type interface. There is more here than just mdev, however my concern is that we may be also losing some of the advantages of mdev.
It would be much easier for me to go along with this if we had more than one vendor pushing it. My concern is that this is becoming something that may end up being vendor specific.
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
also many customers are currently using SRIOV with containers to get
the performance and isolation features since there was no other options.
Quoted 7 lines
Quoted 3 lines
There were, but you hadn't implemented them. The fact is the approach Intel had taken for that was offloaded macvlan.
offloaded macvlan is just a macvlan with checksum/tso and gro.
macvlan can't provide RDMA, TC offloads, ethtool steering, PTP, vdpa
Agreed. I have already acknowledged that macvlan couldn't meet the needs for all use cases. However at the same time it provides a consistent interface regardless of vendors.
If we decide to go with the vendor specific drivers for subfunctions that is fine, however I see that going down the same path as SR-IOV and ultimately ending in obscurity since I don't see many being willing to adopt it.
Quoted 2 lines
...
our SF provides the same set of features a VF can provide
That is all well and good. However if we agree that SR-IOV wasn't done right saying that you are spinning up something that works just like SR-IOV isn't all that appealing, is it?
Quoted 10 lines
Quoted 6 lines
I think the big thing we really should do if we are going to go this route is to look at standardizing what the flavours are that get created by the parent netdevice. Otherwise we are just creating the same mess we had with SRIOV all over again and muddying the waters of mediated devices.
yes in the near future we will be working on auxbus interfaces for auto-probing and user flavor selection, this is a must have feature for us.
I would take this one step further. If we are going to have flavours maybe we should have interfaces pre-defined that are vendor agnostic that can represent the possible flavors. Basically an Ethernet interface for that case, and RDMA interface for that case and so on. It limits what functionality can be exposed, however it frees up the containers and/or guests to work on whatever NIC you want as long as it supports that interface.
Quoted 49 lines
Quoted 41 lines
Quoted 41 lines
Quoted 20 lines
first thing that came to mind when I read this was mediated devices
which is similar, but focused only on the virtualization case:
https://www.kernel.org/doc/html/v5.9/driver-api/vfio-mediated-device.htmlQuoted 5 lines
Parav Pandit Says:
=================This patchset introduces support for mlx5 subfunction (SF).
Quoted 9 lines
A subfunction is a lightweight function that has a parent PCI function on
which it is deployed. mlx5 subfunction has its own function capabilities
and its own resources. This means a subfunction has its own dedicatedqueues(txq, rxq, cq, eq). These queues are neither shared nor stealed from
the parent PCI function.Quoted 16 lines
Rather than calling this a subfunction, would it make more sense to
call it something such as a queue set? It seems like this is exposing
some of the same functionality we did in the Intel drivers such as
ixgbe and i40e via the macvlan offload interface. However the ixgbe/i40e hardware was somewhat limited in that we were only able to
expose Ethernet interfaces via this sort of VMQ/VMDQ feature, and even
with that we have seen some limitations to the interface. It sounds
like you are able to break out RDMA capable devices this way as well.
So in terms of ways to go I would argue this is likely better.We've discussed this thoroughly on V0, the SF solutions is closer to a
VF than a VMDQ, this is not just a set of queues.Quoted 7 lines
VMDq is more than just a set of queues. The fact is it is a pool of resources that get created to handle the requests for a specific VM. The extra bits that are added here are essentially stuff that was required to support mediated devices.
Quoted 4 lines
VMDq pools are managed by the driver and only logically isolated in the kernel, SFs has no shared pool for transport resources (queues), SFs have their own isolated steering domains, processing engines, and HW objects, exactly like a VF.
You are describing your specific implementation. That may not apply to others. What you are defining as the differences of VMDq and SR-IOV are not the same as other vendors.
You are essentially arguing implementation semantics, if it is configured by the driver or firmware it doesn't really make any difference. Being fully isolated versus only logically isolated only really matters in terms of direct assignment. In the grand scheme of things the only real difference between SR-IOV and VMDq is the spawning of the PCIe device with its own BAR to access the resources. Isolating the queues to their own 4K bounded subset of a BAR is pretty straightforward and I assume that and the firmware is what is giving you most of your isolation in this case.
Quoted 32 lines
Quoted 20 lines
Quoted 12 lines
Quoted 11 lines
However
one downside is that we are going to end up seeing each subfunction
being different from driver to driver and vendor to vendor which I
would argue was also one of the problems with SR-IOV as you end up
with a bit of vendor lock-in as a result of this feature since each
vendor will be providing a different interface.Quoted 7 lines
I disagree, SFs are tightly coupled with switchdev model and devlink
functions port, they are backed with the a well defined model, i can
say the same about sriov with switchdev mode, this sort of vendor lock-
in issues is eliminated when you migrate to switchdev mode.Quoted 11 lines
What you are talking about is the backend. I am talking about what is exposed to the user. The user is going to see a Mellanox device having
to be placed into their container in order to support this. One of the
advantages of the Intel approach was that the macvlan interface was generic so you could have an offloaded interface or not and the user wouldn't necessarily know. The offload could be disabled and the user would be none the wiser as it is moved from one interface to another. I see that as a big thing that is missing in this solution.
Quoted 5 lines
You are talking about the basic netdev users, Sure there are users who would want a more generic netdev, so yes. but most of my customers are not like that, they want vdpa/rdma and heavy netdev offload such as encap/decap/crypto and driver xdp in their containers, the SF approach will make more sense to them than sriov and VMDq.
I am talking about my perspective. From what I have seen, one-off features that are only available from specific vendors are a pain to deal with and difficult to enable when you have to support multiple vendors within your ecosystem. What you end up going for is usually the lowest common denominator because you ideally want to be able to configure all your devices the same and have one recipe for setup.
I'm not saying you cannot enable those features. However at the same time I am saying it would be nice to have a vendor neutral way of dealing with those if we are going to support SF, ideally with some sort of software fallback that may not perform as well but will at least get us the same functionality.
I'm trying to remember which netdev conference it was. I referred to this as a veth switchdev offload when something like this was first brought up. The more I think about it now it would almost make more sense to have something like that as a flavor. The way I view it we have a few different use cases floating around which will have different needs. My thought is having a standardized interface that could address those needs would be a good way to go for this as it would force everyone to come together and define a standardized feature set that all of the vendors would want to expose.
Quoted 36 lines
Quoted 28 lines
Quoted 28 lines
Quoted 27 lines
Quoted 20 lines
When subfunction is RDMA capable, it has its own QP1, GID table and
rdma
resources neither shared nor stealed from the parent PCI function.A subfunction has dedicated window in PCI BAR space that is not shared
with ther other subfunctions or parent PCI function. This ensures
that all
class devices of the subfunction accesses only assigned PCI BAR space.A Subfunction supports eswitch representation through which it supports tc
offloads. User must configure eswitch to send/receive packets from/to
subfunction port.Quoted 3 lines
Subfunctions share PCI level resources such as PCI MSI-X IRQs with
their other subfunctions and/or with its parent PCI function.This piece to the architecture for this has me somewhat concerned. If
all your resources are shared and you are allowing devices to beQuoted 7 lines
Quoted 1 line
not all, only PCI MSIX, for now..
They aren't shared after you partition them but they are coming from the same device. Basically you are subdividing the BAR2 in order to generate the subfunctions. BAR2 is a shared resource in my point of view.
Quoted 4 lines
Sure, but it doesn't host any actual resources, only the communication channel with the HW partition, so other than the BAR and the msix the actual HW resources, steering pipelines, offloads and queues are totlay isolated and separated.
I understand what you are trying to say, however this is semantics specific to the implementation. Ultimately you are having to share the function.
Quoted 32 lines
Quoted 18 lines
Quoted 11 lines
Quoted 10 lines
created incrementally you either have to pre-partition the entire function which usually results in limited resources for your base setup, or free resources from existing interfaces and redistribute
them as things change. I would be curious which approach you are taking here? So for example if you hit a certain threshold will you
need to reset the port and rebalance the IRQs between the various functions?Quoted 6 lines
Currently SFs will use whatever IRQs the PF has pre-allocated for itself, so there is no IRQ limit issue at the moment, we are considering a dynamic IRQ pool with dynamic balancing, or even better
us the IMS approach, which perfectly fits the SF architecture.
https://patchwork.kernel.org/project/linux-pci/cover/[email protected]/Quoted 7 lines
When you say you are using the PF's interrupts are you just using that
as a pool of resources or having the interrupt process interrupts for both the PF and SFs? Without IMS you are limited to 2048 interrupts. Moving over to that would make sense since SF is similar to mdev in the way it partitions up the device and resources.Yes moving to IMS is on the top of our priorities.
Quoted 4 lines
Quoted 3 lines
for internal resources the are fully isolated (not shared) and they are internally managed by FW exactly like a VF internal resources.
Quoted 8 lines
Quoted 2 lines
I assume by isolated you mean they are contained within page aligned blocks like what was required for mdev?
I mean they are isolated and abstracted in the FW, we don't really expose any resource directly to the BAR. the BAR is only used for communicating with the device, so VF and SF will work exactly the same the only difference is where they get their BAR and offsets from, everything else is just similar.
I think where you and I differ is our use of the term "resource". I would consider the address space a "resource", while you argue that the resources are hidden behind the BAR.
I agree with you that the firmware should be managing most of the resources in the device. So it isn't surprising that it would be splitting them up and then dolling out pieces as needed to put together a subfunction.
Jason Gunthorpe
maintainerQuoted 6 lines · Alexander Duyck
Quoted 2 lines
not just devlink and switchdev, auxbus was also introduced to standardize some of the interfaces.
The auxbus is just there to make up for the fact that there isn't another bus type for this though. I would imagine otherwise this would be on some sort of platform bus.
Please lets not start this again. This was gone over with Greg for literally a year and a half and he explicitly NAK'd platform bus for this purpose.
Aux bus exists to connect different kernel subsystems that touch the same HW block together. Here we have the mlx5_core subsystem, vdpa, rdma, and netdev all being linked together using auxbus.
It is kind of like what MFD does, but again, using MFD for this was also NAK'd by Greg.
At the very worst we might sometime find out there is some common stuff between ADIs that we might get an ADI bus, but I'm not optimistic. So far it looks like there is no commonality.
Aux bus has at least 4 users already in various stages of submission, and many other target areas that should be replaced by it.
Quoted 3 lines
I would really like to see is a solid standardization of what this is. Otherwise the comparison is going to be made. Especially since a year ago Mellanox was pushing this as an mdev type interface.
mdev was NAK'd too.
mdev is only for creating /dev/vfio/*.
Quoted 3 lines
That is all well and good. However if we agree that SR-IOV wasn't done right saying that you are spinning up something that works just like SR-IOV isn't all that appealing, is it?
Fitting into some universal least-common-denominator was never a goal for SR-IOV, so I wouldn't agree it was done wrong.
Quoted 6 lines
I am talking about my perspective. From what I have seen, one-off features that are only available from specific vendors are a pain to deal with and difficult to enable when you have to support multiple vendors within your ecosystem. What you end up going for is usually the lowest common denominator because you ideally want to be able to configure all your devices the same and have one recipe for setup.
So encourage other vendors to support the switchdev model for managing VFs and ADIs!
Quoted 5 lines
I'm not saying you cannot enable those features. However at the same time I am saying it would be nice to have a vendor neutral way of dealing with those if we are going to support SF, ideally with some sort of software fallback that may not perform as well but will at least get us the same functionality.
Is it really true there is no way to create a software device on a switchdev today? I looked for a while and couldn't find anything. openvswitch can do this, so it does seem like a gap, but this has nothing to do with this series.
A software switchdev path should still end up with the representor and user facing netdev, and the behavior of the two netdevs should be identical to the VF switchdev flow we already have today.
SF doesn't change any of this, it just shines a light that, yes, people actually have been using VFs with netdevs in containers and switchdev, as part of their operations.
FWIW, I view this as a positive because it shows the switchdev model is working very well and seeing adoption beyond the original idea of controlling VMs with SRIOV.
Quoted 3 lines
I'm trying to remember which netdev conference it was. I referred to this as a veth switchdev offload when something like this was first brought up.
Sure, though I think the way you'd create such a thing might be different. These APIs are really about creating an ADI that might be assigned to a VM and never have a netdev.
It would be nonsense to create a veth-switchdev thing with out a netdev, and there have been various past attempts already NAK'd to transform a netdev into an ADI.
Anyhow, if such a thing exists someday it could make sense to automatically substitute the HW version using a SF, if available.
Quoted 3 lines
could address those needs would be a good way to go for this as it would force everyone to come together and define a standardized feature set that all of the vendors would want to expose.
I would say switchdev is already the standard feature set.
Jason
Quoted 1 line · Alexander Duyck
It isn't "SR-IOV done right" it seems more like "VMDq done better".
I don't think I agree with that assertion. The fact that VMDq can talk to a common driver still makes VMDq preferable in some respects. Thus, subfunctions do appear to be more of a better SR-IOV than a better VMDq, but I'm similarly not sold on whether a better SR-IOV is sufficient benefit to warrant the additional complexity this introduces. If I understand correctly, subfunctions buy two things:
1) More than 256 SFs are possible: Maybe it's about time PCI-SIG addresses this limit for VFs? If that were the only problem with VFs, then fixing it once there would be cleaner. The devlink interface for configuring a SF is certainly more sexy than legacy SR-IOV, but it shouldn't be fundamentally impossible to zhuzh up VFs either. One can also imagine possibilities around remapping multiple PFs (and their VFs) in a clever way to get around the limited number of PCI resources exposed to the host.
2) More flexible division of resources: It's not clear that device firmwarre can't perform smarter allocation than N/<num VFs>, but subfunctions also appear to allow sharing of certain resources by the PF driver, if desirable. To the extent that resources are shared, how are workloads isolated from each other?
I'm not sure I like the idea of having to support another resource allocation model in our driver just to support this, at least not without a clearer understanding of what is being gained.
Like you, I would also prefer a more common infrastructure for exposing something based on VirtIO/VMDq as the container/VM facing netdevs. Is the lowest common denominator that a VMDq based interface would constrain things to really unsuitable for container use cases? Is the added complexity and confusion around VF vs SF vs VMDq really warranted? I also don't see how this tackles container/VF portability, migration of workloads, kernel network stack bypass, or any of the other legacy limitations regarding SR-IOV VFs when we have vendor specific aux bus drivers talking directly to vendor specific backend hardware resources. In this regard, don't subfunctions, by definition, have most of the same limitations as SR-IOV VFs?
Regards,
Edwin Peer
Signature · 11 lines
This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the use of the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or otherwise restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it.
Quoted 14 lines · Jason Gunthorpe
Quoted 6 lines · Alexander Duyck
Quoted 2 lines
not just devlink and switchdev, auxbus was also introduced to standardize some of the interfaces.
The auxbus is just there to make up for the fact that there isn't another bus type for this though. I would imagine otherwise this would be on some sort of platform bus.
Please lets not start this again. This was gone over with Greg for literally a year and a half and he explicitly NAK'd platform bus for this purpose.
Quoted 6 lines
Aux bus exists to connect different kernel subsystems that touch the same HW block together. Here we have the
mlx5_coresubsystem, vdpa, rdma, and netdev all being linked together using auxbus.It is kind of like what MFD does, but again, using MFD for this was also NAK'd by Greg.
Sorry I wasn't saying we needed to use MFD or platform bus. I'm aware of the discussions I was just saying there are some parallels, not that we needed to go through and use that.
Quoted 6 lines
At the very worst we might sometime find out there is some common stuff between ADIs that we might get an ADI bus, but I'm not optimistic. So far it looks like there is no commonality.
Aux bus has at least 4 users already in various stages of submission, and many other target areas that should be replaced by it.
Aux bus is fine and I am happy with it. I was just saying that auxbus isn't something that should really be used to say that this is significantly different from mdev as they both rely on a bus topology.
Quoted 7 lines
Quoted 3 lines
I would really like to see is a solid standardization of what this is. Otherwise the comparison is going to be made. Especially since a year ago Mellanox was pushing this as an mdev type interface.
mdev was NAK'd too.
mdev is only for creating
/dev/vfio/*.
Agreed. However my worry is that as we start looking to make this support virtualization it will still end up swinging more toward mdev. I would much prefer to make certain that any mdev is a flavor and doesn't end up coloring the entire interface.
Quoted 6 lines
Quoted 3 lines
That is all well and good. However if we agree that SR-IOV wasn't done right saying that you are spinning up something that works just like SR-IOV isn't all that appealing, is it?
Fitting into some universal least-common-denominator was never a goal for SR-IOV, so I wouldn't agree it was done wrong.
It isn't so much about right or wrong but he use cases. My experience has been that SR-IOV ends up being used for very niche use cases where you are direct assigning it into either DPDK or some NFV VM and you are essentially building the application around the NIC. It is all well and good, but for general virtualization it never really caught on.
My thought is that if we are going to do this we need to do this in a way that improves the usability, otherwise we are looking at more niche use cases.
Quoted 9 lines
Quoted 6 lines
I am talking about my perspective. From what I have seen, one-off features that are only available from specific vendors are a pain to deal with and difficult to enable when you have to support multiple vendors within your ecosystem. What you end up going for is usually the lowest common denominator because you ideally want to be able to configure all your devices the same and have one recipe for setup.
So encourage other vendors to support the switchdev model for managing VFs and ADIs!
Ugh, don't get me started on switchdev. The biggest issue as I see it with switchev is that you have to have a true switch in order to really be able to use it. As such dumbed down hardware like the ixgbe for instance cannot use it since it defaults to outputting anything that doesn't have an existing rule to the external port. If we could tweak the design to allow for more dumbed down hardware it would probably be much easier to get wider adoption.
Honestly, the switchdev interface isn't what I was talking about. I was talking about the SF interface, not the switchdev side of it. In my mind you can place your complexity into the switchdev side of the interface, but keep the SF interface simple. Then you can back it with whatever you want, but without having to have a vendor specific version of the interface being plugged into the guest or container.
One of the reasons why virtio-net is being pushed as a common interface for vendors is for this reason. It is an interface that can be emulated by software or hardware and it allows the guest to run on any arbitrary hardware.
Quoted 10 lines
Quoted 5 lines
I'm not saying you cannot enable those features. However at the same time I am saying it would be nice to have a vendor neutral way of dealing with those if we are going to support SF, ideally with some sort of software fallback that may not perform as well but will at least get us the same functionality.
Is it really true there is no way to create a software device on a switchdev today? I looked for a while and couldn't find anything. openvswitch can do this, so it does seem like a gap, but this has nothing to do with this series.
It has plenty to do with this series. This topic has been under discussion since something like 2017 when Mellanox first brought it up at Netdev 2.1. At the time I told them they should implement this as a veth offload. Then it becomes obvious what the fallback becomes as you can place packets into one end of a veth and it comes out the other, just like a switchdev representor and the SF in this case. It would make much more sense to do it this way rather than setting up yet another vendor proprietary interface pair.
Quoted 11 lines
A software switchdev path should still end up with the representor and user facing netdev, and the behavior of the two netdevs should be identical to the VF switchdev flow we already have today.
SF doesn't change any of this, it just shines a light that, yes, people actually have been using VFs with netdevs in containers and switchdev, as part of their operations.
FWIW, I view this as a positive because it shows the switchdev model is working very well and seeing adoption beyond the original idea of controlling VMs with SRIOV.
PF/VF isolation is a given. So the existing switchdev behavior is fine in that regard. I wouldn't expect that to change. The fact is we actually support something similar in the macvlan approach we put in the Intel drivers since the macvlan itself provided an option for isolation or to hairpin the traffic if you wanted to allow the VMDq instances to be bypassed. That was another thing I view as a huge feature as broadcast/multicast traffic can really get ugly when the devices are all separate pipelines and being able to switch that off and just do software replication and hairpinning can be extremely useful.
Quoted 8 lines
Quoted 3 lines
I'm trying to remember which netdev conference it was. I referred to this as a veth switchdev offload when something like this was first brought up.
Sure, though I think the way you'd create such a thing might be different. These APIs are really about creating an ADI that might be assigned to a VM and never have a netdev.
Quoted 6 lines
It would be nonsense to create a veth-switchdev thing with out a netdev, and there have been various past attempts already NAK'd to transform a netdev into an ADI.
Anyhow, if such a thing exists someday it could make sense to automatically substitute the HW version using a SF, if available.
The main problem as I see it is the fact that the SF interface is bound too tightly to the hardware. The direct pass-thru with no hairpin is always an option but if we are going to have an interface where both ends are in the same system there are many cases where always pushing all the packets off to the hardware don't necessarily make sense.
Quoted 5 lines
Quoted 3 lines
could address those needs would be a good way to go for this as it would force everyone to come together and define a standardized feature set that all of the vendors would want to expose.
I would say switchdev is already the standard feature set.
Yes, it is a standard feature set for the control plane. However for the data-path it is somewhat limited as I feel it only describes what goes through the switch. Not the interfaces that are exposed as the endpoints. It is the problem of that last bit and how it is handled that can make things ugly. For example the multicast/broadcast replication problem that just occurred to me while writing this up. The fact is for east-west traffic there has always been a problem with the switchdev model as it limits everything to PCIe/DMA so there are cases where software switches can outperform the hardware ones.
Jason Gunthorpe
maintainerQuoted 2 lines · Edwin Peer
1) More than 256 SFs are possible: Maybe it's about time PCI-SIG addresses this limit for VFs?
They can't, the Bus/Device/Function is limited by protocol and changing that would upend the entire PCI world.
Instead PCI-SIG said PASID is the way forward.
Quoted 2 lines
If that were the only problem with VFs, then fixing it once there would be cleaner.
Maybe, but half the problem with VFs is how HW expensive they are. The mlx5 SF version is not such a good example, but Intel has shown in other recent patches, like for their idxd, that the HW side of an ADI can be very simple and hypervisor emulation can build a simple HW capability into a full ADI for assignment to a guest.
A lot of the trappings that PCI-SIG requires to be implemented in HW for a VF, like PCI config space, MSI tables, BAR space, etc. is all just dead weight when scaling up to 1000's of VFs.
The ADI scheme is not that bad, the very simplest HW is just a queue that can have all DMA contained by a PASID and can trigger an addr/data interrupt message. Much less HW costly than a SRIOV VF.
Regardless, Intel kicked this path off years ago when they published their SIOV cookbook and everyone started integrating PASID support into their IOMMUs and working on ADIs. The mlx5 SFs are kind of early because the HW is flexible enough to avoid the parts of SIOV that are not ready or widely deployed yet, like IMS and PASID.
Quoted 3 lines
Like you, I would also prefer a more common infrastructure for exposing something based on VirtIO/VMDq as the container/VM facing netdevs.
A major point is to get switchdev.
Quoted 3 lines
I also don't see how this tackles container/VF portability, migration of workloads, kernel network stack bypass, or any of the other legacy limitations regarding SR-IOV VFs
It isn't ment too. SF/ADI are just a way to have more VF's than PCI SIG can support..
Jason
Jason Gunthorpe
maintainerQuoted 11 lines · Alexander Duyck
Quoted 7 lines
Quoted 3 lines
I would really like to see is a solid standardization of what this is. Otherwise the comparison is going to be made. Especially since a year ago Mellanox was pushing this as an mdev type interface.
mdev was NAK'd too.
mdev is only for creating
/dev/vfio/*.Agreed. However my worry is that as we start looking to make this support virtualization it will still end up swinging more toward mdev.
Of course. mdev is also the only way to create a /dev/vfio/* :)
So all paths that want to use vfio must end up creating a mdev.
Here we would choose to create the mdev on top of the SF aux device. There isn't really anything mlx5 specific about that decision.
The SF models the vendor specific ADI in the driver model.
Quoted 6 lines
It isn't so much about right or wrong but he use cases. My experience has been that SR-IOV ends up being used for very niche use cases where you are direct assigning it into either DPDK or some NFV VM and you are essentially building the application around the NIC. It is all well and good, but for general virtualization it never really caught on.
Sure
Quoted 6 lines
Quoted 2 lines
So encourage other vendors to support the switchdev model for managing VFs and ADIs!
Ugh, don't get me started on switchdev. The biggest issue as I see it with switchev is that you have to have a true switch in order to really be able to use it.
That cuts both ways, suggesting HW with a true switch model itself with VMDq is equally problematic.
Quoted 5 lines
As such dumbed down hardware like the ixgbe for instance cannot use it since it defaults to outputting anything that doesn't have an existing rule to the external port. If we could tweak the design to allow for more dumbed down hardware it would probably be much easier to get wider adoption.
I'd agree with this
Quoted 3 lines
interface, but keep the SF interface simple. Then you can back it with whatever you want, but without having to have a vendor specific version of the interface being plugged into the guest or container.
The entire point *is* to create the vendor version because that serves the niche cases where SRIOV assignment is already being used.
Having a general solution that can't do vendor SRIOV is useful for other application, but doesn't eliminate the need for the SRIOV case.
Quoted 4 lines
One of the reasons why virtio-net is being pushed as a common interface for vendors is for this reason. It is an interface that can be emulated by software or hardware and it allows the guest to run on any arbitrary hardware.
Yes, and there is mlx5_vdpa to support this usecase, and it binds to the SF. Of course all of that is vendor specific too, the driver to convert HW specifc register programming into a virio-net ADI has to live *somewhere*
Quoted 4 lines
It has plenty to do with this series. This topic has been under discussion since something like 2017 when Mellanox first brought it up at Netdev 2.1. At the time I told them they should implement this as a veth offload.
veth doesn't give an ADI, it is useless for these niche cases.
veth offload might be interesting for some container case, but feels like writing an enormous amount of code to accomplish nothing new...
Quoted 5 lines
Then it becomes obvious what the fallback becomes as you can place packets into one end of a veth and it comes out the other, just like a switchdev representor and the SF in this case. It would make much more sense to do it this way rather than setting up yet another vendor proprietary interface pair.
I agree it makes sense to have an all SW veth-like option, but I wouldn't try to make that as the entry point for all the HW acceleration or to serve the niche SRIOV use cases, or to represent an ADI.
It just can't do that and it would make a huge mess if you tried to force it. Didn't Intel already try this once with trying to use the macvlan netdev and its queue offload to build an ADI?
Quoted 5 lines
Quoted 2 lines
Anyhow, if such a thing exists someday it could make sense to automatically substitute the HW version using a SF, if available.
The main problem as I see it is the fact that the SF interface is bound too tightly to the hardware.
That is goal here. This is not about creating just a netdev, this is about the whole kit: rdma, netdev, vdpa virtio-net, virtio-mdev.
The SF has to support all of that completely. Focusing only on the one use case of netdevs in containers misses the bigger picture.
Yes, lots of this stuff is niche, but niche stuff needs to be supported too.
Quoted 3 lines
Yes, it is a standard feature set for the control plane. However for the data-path it is somewhat limited as I feel it only describes what goes through the switch.
Sure, I think that is its main point.
Quoted 1 line
Not the interfaces that are exposed as the endpoints.
It came from modeling physical HW so the endports are 'physical' things like actual HW switch ports, or SRIOV VFs, ADI, etc.
Quoted 6 lines
It is the problem of that last bit and how it is handled that can make things ugly. For example the multicast/broadcast replication problem that just occurred to me while writing this up. The fact is for east-west traffic there has always been a problem with the switchdev model as it limits everything to PCIe/DMA so there are cases where software switches can outperform the hardware ones.
Yes, but, mixing CPU and DMA in the same packet delivery scheme is very complicated :)
Jason
Quoted 13 lines · Edwin Peer
Quoted 1 line · Alexander Duyck
It isn't "SR-IOV done right" it seems more like "VMDq done better".
I don't think I agree with that assertion. The fact that VMDq can talk to a common driver still makes VMDq preferable in some respects. Thus, subfunctions do appear to be more of a better SR-IOV than a better VMDq, but I'm similarly not sold on whether a better SR-IOV is sufficient benefit to warrant the additional complexity this introduces. If I understand correctly, subfunctions buy two things:
Quoted 8 lines
1) More than 256 SFs are possible: Maybe it's about time PCI-SIG addresses this limit for VFs? If that were the only problem with VFs, then fixing it once there would be cleaner. The devlink interface for configuring a SF is certainly more sexy than legacy SR-IOV, but it shouldn't be fundamentally impossible to zhuzh up VFs either. One can also imagine possibilities around remapping multiple PFs (and their VFs) in a clever way to get around the limited number of PCI resources exposed to the host.
The fact is SR-IOV just wasn't designed to scale well. I think we are probably going to see most vendors move away from it.
The fact is what we are talking about now is the future of all this and how to implement Scalable I/O Virtualization
(https://software.intel.com/content/www/us/en/develop/download/intel-scalable-io-virtualization-technical-specification.html).
The document is a good primer to many of the features we are discussing as it describes how to compose a device.
The problem is as it was with SR-IOV that the S-IOV specification is very PCIe centric and doesn't do a good job explaining how to deal with the network as it relates to all this. Then to complicate things the S-IOV expected this to be used with direct assigned devices for guests/applications and instead we are talking about using the devices in the host which makes things a bit messier.
Quoted 9 lines
2) More flexible division of resources: It's not clear that device firmwarre can't perform smarter allocation than N/<num VFs>, but subfunctions also appear to allow sharing of certain resources by the PF driver, if desirable. To the extent that resources are shared, how are workloads isolated from each other?
I'm not sure I like the idea of having to support another resource allocation model in our driver just to support this, at least not without a clearer understanding of what is being gained.
I view this as the future alternative to SR-IOV. It is just a matter of how we define it. Eventually we probably would be dropping the SR-IOV implementation and instead moving over to S-IOV as an alternative instead. As such if this is done right I don't see this as a thing where we need to support both. Really we should be able to drop support for one if we have the other.
Quoted 36 lines · Jason Gunthorpe
Quoted 11 lines · Alexander Duyck
Quoted 7 lines
Quoted 3 lines
I would really like to see is a solid standardization of what this is. Otherwise the comparison is going to be made. Especially since a year ago Mellanox was pushing this as an mdev type interface.
mdev was NAK'd too.
mdev is only for creating
/dev/vfio/*.Agreed. However my worry is that as we start looking to make this support virtualization it will still end up swinging more toward mdev.
Of course. mdev is also the only way to create a
/dev/vfio/*:)So all paths that want to use vfio must end up creating a mdev.
Here we would choose to create the mdev on top of the SF aux device. There isn't really anything mlx5 specific about that decision.
The SF models the vendor specific ADI in the driver model.
Quoted 6 lines
It isn't so much about right or wrong but he use cases. My experience has been that SR-IOV ends up being used for very niche use cases where you are direct assigning it into either DPDK or some NFV VM and you are essentially building the application around the NIC. It is all well and good, but for general virtualization it never really caught on.
Sure
Quoted 3 lines
Quoted 2 lines
So encourage other vendors to support the switchdev model for managing VFs and ADIs!
Quoted 6 lines
Quoted 3 lines
Ugh, don't get me started on switchdev. The biggest issue as I see it with switchev is that you have to have a true switch in order to really be able to use it.
That cuts both ways, suggesting HW with a true switch model itself with VMDq is equally problematic.
Yes and no. For example the macvlan offload I had setup could be configured both ways and it made use of VMDq. I'm not necessarily arguing that we need to do VMDq here, however at the same time saying that this is only meant to replace SR-IOV becomes problematic since we already have SR-IOV so why replace it with something that has many of the same limitations?
Quoted 12 lines
Quoted 5 lines
As such dumbed down hardware like the ixgbe for instance cannot use it since it defaults to outputting anything that doesn't have an existing rule to the external port. If we could tweak the design to allow for more dumbed down hardware it would probably be much easier to get wider adoption.
I'd agree with this
Quoted 3 lines
interface, but keep the SF interface simple. Then you can back it with whatever you want, but without having to have a vendor specific version of the interface being plugged into the guest or container.
Quoted 5 lines
The entire point *is* to create the vendor version because that serves the niche cases where SRIOV assignment is already being used.
Having a general solution that can't do vendor SRIOV is useful for other application, but doesn't eliminate the need for the SRIOV case.
So part of the problem here is we already have SR-IOV. So we don't need to repeat the mistakes. Rather, we need to have a solution to the existing problems and then we can look at eliminating it.
That said I understand your argument, however I view the elimination of SR-IOV to be something we do after we get this interface right and can justify doing so. I don't have a problem necessarily with vendor specific instances, unless we are only able to get vendor specific instances. Thus I would prefer that we have a solution in place before we allow the switch over.
Quoted 9 lines
Quoted 4 lines
One of the reasons why virtio-net is being pushed as a common interface for vendors is for this reason. It is an interface that can be emulated by software or hardware and it allows the guest to run on any arbitrary hardware.
Yes, and there is
mlx5_vdpato support this usecase, and it binds to the SF. Of course all of that is vendor specific too, the driver to convert HW specifc register programming into a virio-net ADI has to live *somewhere*
Right, but this is more the model I am in favor of. The backend is hidden from the guest and lives somewhere on the host.
Also it might be useful to call out the flavours and planned flavours in the cover page. Admittedly the description is somewhat lacking in that regard.
Quoted 9 lines
Quoted 4 lines
It has plenty to do with this series. This topic has been under discussion since something like 2017 when Mellanox first brought it up at Netdev 2.1. At the time I told them they should implement this as a veth offload.
veth doesn't give an ADI, it is useless for these niche cases.
veth offload might be interesting for some container case, but feels like writing an enormous amount of code to accomplish nothing new...
My concern is if we are going to start partitioning up a PF on the host we might as well make the best use of it. I would argue that it would make more sense to have some standardized mechanism in place for the PF to communicate and interact with the SFs. I would argue that is one of the reasons why this keeps being compared to either VMDq or VMQ as it is something that SR-IOV has yet to fully replace and has many features that would be useful in an interface that is a subpartition of an existing interface.
Quoted 6 lines
Quoted 5 lines
Then it becomes obvious what the fallback becomes as you can place packets into one end of a veth and it comes out the other, just like a switchdev representor and the SF in this case. It would make much more sense to do it this way rather than setting up yet another vendor proprietary interface pair.
Quoted 8 lines
I agree it makes sense to have an all SW veth-like option, but I wouldn't try to make that as the entry point for all the HW acceleration or to serve the niche SRIOV use cases, or to represent an ADI.
It just can't do that and it would make a huge mess if you tried to force it. Didn't Intel already try this once with trying to use the macvlan netdev and its queue offload to build an ADI?
The Intel drivers still have the macvlan as the assignable ADI and make use of VMDq to enable it. Actually I would consider it an example of the kind of thing I am talking about. It is capable of doing software switching between interfaces, broadcast/multicast replication in software, and makes use of the hardware interfaces to allow for receiving directly from the driver into the macvlan interface.
The limitation as I see it is that the macvlan interface doesn't allow for much in the way of custom offloads and the Intel hardware doesn't support switchdev. As such it is good for a basic interface, but doesn't really do well in terms of supporting advanced vendor-specific features.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Anyhow, if such a thing exists someday it could make sense to automatically substitute the HW version using a SF, if available.
The main problem as I see it is the fact that the SF interface is bound too tightly to the hardware.
That is goal here. This is not about creating just a netdev, this is about the whole kit: rdma, netdev, vdpa virtio-net, virtio-mdev.
One issue is right now we are only seeing the rdma and netdev. It is kind of backwards as it is using the ADIs on the host when this was really meant to be used for things like mdev.
Quoted 5 lines
The SF has to support all of that completely. Focusing only on the one use case of netdevs in containers misses the bigger picture.
Yes, lots of this stuff is niche, but niche stuff needs to be supported too.
I have no problem with niche stuff, however we need to address the basics before we move on to the niche stuff.
Quoted 10 lines
Quoted 3 lines
Yes, it is a standard feature set for the control plane. However for the data-path it is somewhat limited as I feel it only describes what goes through the switch.
Sure, I think that is its main point.
Quoted 1 line
Not the interfaces that are exposed as the endpoints.
It came from modeling physical HW so the endports are 'physical' things like actual HW switch ports, or SRIOV VFs, ADI, etc.
The problem is the "physical things" such as the SRIOV VFs and ADI aren't really defined in the specification and are left up to the implementers interpretation. These specs have always been fuzzy since they are essentially PCI specifications and don't explain anything about how the network on such a device should be configured or expected to work. The swtichdev API puts some restrictions in place but there still ends up being parts without any definition.
Quoted 9 lines
Quoted 6 lines
It is the problem of that last bit and how it is handled that can make things ugly. For example the multicast/broadcast replication problem that just occurred to me while writing this up. The fact is for east-west traffic there has always been a problem with the switchdev model as it limits everything to PCIe/DMA so there are cases where software switches can outperform the hardware ones.
Yes, but, mixing CPU and DMA in the same packet delivery scheme is very complicated :)
I'm not necessarily saying we need to mix the two. However there are cases such as multicast/broadcast where it would make much more sense to avoid the duplication of packets and instead simply send one copy and have it replicated by the software.
What would probably make sense for now would be to look at splitting the netdev into two pieces. The frontend which would provide the netdev and be a common driver for subfunction netdevs in this case, and a backend which would be a common point for all the subfunctions that are being used directly on the host. This is essentially what we have with the macvlan model. The idea is that if we wanted to do software switching or duplication of traffic we could, but if not then we wouldn't.
Quoted 6 lines
From: Alexander Duyck <[email protected]>
Sent: Wednesday, December 16, 2020 9:43 AMQuoted 1 line
Quoted 7 lines
Quoted 2 lines
That is goal here. This is not about creating just a netdev, this is about the whole kit: rdma, netdev, vdpa virtio-net, virtio-mdev.
One issue is right now we are only seeing the rdma and netdev. It is kind of backwards as it is using the ADIs on the host when this was really meant to be used for things like mdev.
mdev is just yet another _use_ of subfunction.
There are users of subfunction who want to use it without mapping subfunction as mdev to guest VM. i.e. as netdev, rdma, vdpa.
In future direct assignment can be via mdev of subfunction, like how rest of the above devices are done.
Creating a subfunction for non vfio purpose via vfio mdev is just not right. If I understand you correctly, I hope you are not suggesting that.
Leon Romanovsky
maintainerQuoted 7 lines · Jakub Kicinski
Quoted 6 lines · On Tue, 15 Dec 2020 12:35:20 -0800 Saeed Mahameed wrote:
Quoted 5 lines
I think the big thing we really should do if we are going to go this route is to look at standardizing what the flavours are that get created by the parent netdevice. Otherwise we are just creating the same mess we had with SRIOV all over again and muddying the waters of mediated devices.
Quoted 5 lines
Quoted 3 lines
yes in the near future we will be working on auxbus interfaces for auto-probing and user flavor selection, this is a must have feature for us.
Can you elaborate? I thought config would be via devlink.
Yes, everything continues to be done through devlink.
One of the immediate features is an ability to disable/enable creation of specific SF types.
For example, if user doesn't want RDMA, the SF RDMA won't be created.
Thanks
Jason Gunthorpe
maintainerQuoted 7 lines · Alexander Duyck
Quoted 6 lines
Quoted 3 lines
Ugh, don't get me started on switchdev. The biggest issue as I see it with switchev is that you have to have a true switch in order to really be able to use it.
That cuts both ways, suggesting HW with a true switch model itself with VMDq is equally problematic.
Quoted 6 lines
Yes and no. For example the macvlan offload I had setup could be configured both ways and it made use of VMDq. I'm not necessarily arguing that we need to do VMDq here, however at the same time saying that this is only meant to replace SR-IOV becomes problematic since we already have SR-IOV so why replace it with something that has many of the same limitations?
Why? Because SR-IOV is the *only* option for many use cases. Still. I said this already, something more generic does not magicaly eliminate SR-IOV.
The SIOV ADI model is a small refinement to the existing VF scheme, it is completely parallel to making more generic things.
It is not "repeating mistakes" it is accepting the limitations of SR-IOV because benefits exist and applications need those benefits.
Quoted 3 lines
That said I understand your argument, however I view the elimination of SR-IOV to be something we do after we get this interface right and can justify doing so.
Elimination of SR-IOV isn't even a goal here!
Quoted 3 lines
Also it might be useful to call out the flavours and planned flavours in the cover page. Admittedly the description is somewhat lacking in that regard.
This is more of a general switchdev remark though. In the swithdev model you have a the switch and a switch port. Each port has a swichdev representor on the switch side and a "user port" of some kind.
It can be a physical thing:
- SFP
- QSFP
- WiFi Antennae
It could be a semi-physical thing outside the view of the kernel:
- SmartNIC VF/SF attached to another CPU
It can be a semi-physical thing in view of this kernel:
- SRIOV VF (struct pci device)
- SF (struct aux device)
It could be a SW construct in this kernel:
- netdev (struct net device)
*all* of these different port types are needed. Probably more down the road!
Notice I don't have VPDA, VF/SF netdev, or virtio-mdev as a "user port" type here. Instead creating the user port pci or aux device allows the user to use the Linux driver model to control what happens to the pci/aux device next.
Quoted 4 lines
I would argue that is one of the reasons why this keeps being compared to either VMDq or VMQ as it is something that SR-IOV has yet to fully replace and has many features that would be useful in an interface that is a subpartition of an existing interface.
In what sense does switchdev and a VF not fully replace macvlan VMDq?
Quoted 2 lines
The Intel drivers still have the macvlan as the assignable ADI and make use of VMDq to enable it.
Is this in-tree or only in the proprietary driver? AFAIK there is no in-tree way to extract the DMA queue from the macvlan netdev into userspace..
Remeber all this VF/SF/VDPA stuff results in a HW dataplane, not a SW one. It doesn't really make sense to compare a SW dataplane to a HW one. HW dataplanes come with limitations and require special driver code.
Quoted 5 lines
The limitation as I see it is that the macvlan interface doesn't allow for much in the way of custom offloads and the Intel hardware doesn't support switchdev. As such it is good for a basic interface, but doesn't really do well in terms of supporting advanced vendor-specific features.
I don't know what it is that prevents Intel from modeling their selector HW in switchdev, but I think it is on them to work with the switchdev folks to figure something out.
I'm a bit surprised HW that can do macvlan can't be modeled with switchdev? What is missing?
Quoted 6 lines
Quoted 2 lines
That is goal here. This is not about creating just a netdev, this is about the whole kit: rdma, netdev, vdpa virtio-net, virtio-mdev.
One issue is right now we are only seeing the rdma and netdev. It is kind of backwards as it is using the ADIs on the host when this was really meant to be used for things like mdev.
This is second 15 patch series on this path already. It is not possible to pack every single thing into this series. This is the micro step of introducing the SF idea and using SF==VF to show how the driver stack works. The minimal changing to the existing drivers implies this can support an ADI as well.
Further, this does already show an ADI! vdpa_mlx5 will run on the VF/SF and eventually causes qemu to build a virtio-net ADI that directly passes HW DMA rings into the guest.
Isn't this exactly the kind of generic SRIOV replacement option you have been asking for? Doesn't this completely supersede stuff built on macvlan?
Quoted 2 lines
expected to work. The swtichdev API puts some restrictions in place but there still ends up being parts without any definition.
I'm curious what you see as needing definition here?
The SRIOV model has the HW register programming API is device specific.
The switchdev model is: no matter what HW register programing is done on the VF/SF all the packets tx/rx'd will flow through the switchdev.
The purpose of switchdev/SRIOV/SIOV has never been to define a single "one register set to rule them all".
That is the area that VDPA virtio-net and others are covering.
Jason
Quoted 21 lines · Jason Gunthorpe
Quoted 7 lines · Alexander Duyck
Quoted 6 lines
Quoted 3 lines
Ugh, don't get me started on switchdev. The biggest issue as I see it with switchev is that you have to have a true switch in order to really be able to use it.
That cuts both ways, suggesting HW with a true switch model itself with VMDq is equally problematic.
Quoted 6 lines
Yes and no. For example the macvlan offload I had setup could be configured both ways and it made use of VMDq. I'm not necessarily arguing that we need to do VMDq here, however at the same time saying that this is only meant to replace SR-IOV becomes problematic since we already have SR-IOV so why replace it with something that has many of the same limitations?
Why? Because SR-IOV is the *only* option for many use cases. Still. I said this already, something more generic does not magicaly eliminate SR-IOV.
Quoted 5 lines
The SIOV ADI model is a small refinement to the existing VF scheme, it is completely parallel to making more generic things.
It is not "repeating mistakes" it is accepting the limitations of SR-IOV because benefits exist and applications need those benefits.
If we have two interfaces, both with pretty much the same limitations then many would view it as "repeating mistakes". The fact is we already have SR-IOV. Why introduce yet another interface that has the same functionality?
You say this will scale better but I am not even sure about that. The fact is SR-IOV could scale to 256 VFs, but for networking I kind of doubt the limitation would have been the bus number and would more likely be issues with packet replication and PCIe throughput, especially when you start dealing with east-west traffic within the same system.
Quoted 5 lines
Quoted 3 lines
That said I understand your argument, however I view the elimination of SR-IOV to be something we do after we get this interface right and can justify doing so.
Elimination of SR-IOV isn't even a goal here!
Sorry you used the word "replace", and my assumption here was that the goal is to get something in place that can take the place of SR-IOV so that you wouldn't be maintaining the two systems at the same time. That is my concern as I don't want us having SR-IOV, and then several flavors of SIOV. We need to decide on one thing that will be the way forward.
Quoted 24 lines
Quoted 3 lines
Also it might be useful to call out the flavours and planned flavours in the cover page. Admittedly the description is somewhat lacking in that regard.
This is more of a general switchdev remark though. In the swithdev model you have a the switch and a switch port. Each port has a swichdev representor on the switch side and a "user port" of some kind.
It can be a physical thing:
- SFP
- QSFP
- WiFi AntennaeIt could be a semi-physical thing outside the view of the kernel:
- SmartNIC VF/SF attached to another CPUIt can be a semi-physical thing in view of this kernel:
- SRIOV VF (struct pci device)
- SF (struct aux device)It could be a SW construct in this kernel:
- netdev (struct net device)
Quoted 7 lines
*all* of these different port types are needed. Probably more down the road!
Notice I don't have VPDA, VF/SF netdev, or virtio-mdev as a "user port" type here. Instead creating the user port pci or aux device allows the user to use the Linux driver model to control what happens to the pci/aux device next.
I get that. That is why I said switchdev isn't a standard for the endpoint. One of the biggest issues with SR-IOV that I have seen is the fact that the last piece isn't really defined. We never did a good job of defining how the ADI should look to the guest and as a result it kind of stalled in adoption.
Quoted 6 lines
Quoted 4 lines
I would argue that is one of the reasons why this keeps being compared to either VMDq or VMQ as it is something that SR-IOV has yet to fully replace and has many features that would be useful in an interface that is a subpartition of an existing interface.
In what sense does switchdev and a VF not fully replace macvlan VMDq?
One of the biggest is east-west traffic. You quickly run up against the PCIe bandwidth bottleneck and then the performance tanks. I have seen a number of cases where peer-to-peer on the same host swamps the network interface.
Quoted 11 lines
Quoted 2 lines
The Intel drivers still have the macvlan as the assignable ADI and make use of VMDq to enable it.
Is this in-tree or only in the proprietary driver? AFAIK there is no in-tree way to extract the DMA queue from the macvlan netdev into userspace..
Remeber all this VF/SF/VDPA stuff results in a HW dataplane, not a SW one. It doesn't really make sense to compare a SW dataplane to a HW one. HW dataplanes come with limitations and require special driver code.
I get that. At the same time we can mask some of those limitations by allowing for the backend to be somewhat abstract so you have the possibility of augmenting the hardware dataplane with a software one if needed.
Quoted 9 lines
Quoted 5 lines
The limitation as I see it is that the macvlan interface doesn't allow for much in the way of custom offloads and the Intel hardware doesn't support switchdev. As such it is good for a basic interface, but doesn't really do well in terms of supporting advanced vendor-specific features.
I don't know what it is that prevents Intel from modeling their selector HW in switchdev, but I think it is on them to work with the switchdev folks to figure something out.
They tried for the ixgbe and i40e. The problem is the hardware couldn't conform to what was asked for if I recall. It has been a few years since I worked in the Ethernet group at intel so I don't recall the exact details.
Quoted 2 lines
I'm a bit surprised HW that can do macvlan can't be modeled with switchdev? What is missing?
If I recall it was the fact that the hardware defaults to transmitting everything that doesn't match an existing rule to the external port unless it comes from the external port.
Quoted 13 lines
Quoted 6 lines
Quoted 2 lines
That is goal here. This is not about creating just a netdev, this is about the whole kit: rdma, netdev, vdpa virtio-net, virtio-mdev.
One issue is right now we are only seeing the rdma and netdev. It is kind of backwards as it is using the ADIs on the host when this was really meant to be used for things like mdev.
This is second 15 patch series on this path already. It is not possible to pack every single thing into this series. This is the micro step of introducing the SF idea and using SF==VF to show how the driver stack works. The minimal changing to the existing drivers implies this can support an ADI as well.
Quoted 7 lines
Further, this does already show an ADI!
vdpa_mlx5will run on the VF/SF and eventually causes qemu to build a virtio-net ADI that directly passes HW DMA rings into the guest.Isn't this exactly the kind of generic SRIOV replacement option you have been asking for? Doesn't this completely supersede stuff built on macvlan?
Something like the vdpa model is more like what I had in mind. Only vdpa only works for the userspace networking case.
Basically the idea is to have an assignable device interface that isn't directly tied to the hardware. Instead it is making use of a slice of it and referencing the PF as the parent leaving the PF as the owner of the slice. If at some point in the future we could make changes to allow for software to step in and do some switching if needed. The key bit is the abstraction of the assignable interface so that it is vendor agnostic and could be switched over to pure software backing if needed.
Quoted 8 lines
Quoted 2 lines
expected to work. The swtichdev API puts some restrictions in place but there still ends up being parts without any definition.
I'm curious what you see as needing definition here?
The SRIOV model has the HW register programming API is device specific.
Quoted 7 lines
The switchdev model is: no matter what HW register programing is done on the VF/SF all the packets tx/rx'd will flow through the switchdev.
The purpose of switchdev/SRIOV/SIOV has never been to define a single "one register set to rule them all".
That is the area that VDPA virtio-net and others are covering.
That is fine and that covers it for direct assigned devices. However that doesn't cover the container case. My thought is if we are going to partition a PF into multiple netdevices we should have some generic interface that can be provided to represent the netdevs so that if they are pushed into containers you don't have to rip them out if for some reason you need to change the network configuration. For the Intel NICs we did that with macvlan in the VMDq case. I see no reason why you couldn't do something like that here with the subfunction case.
Jason Gunthorpe
maintainerQuoted 6 lines · Alexander Duyck
You say this will scale better but I am not even sure about that. The fact is SR-IOV could scale to 256 VFs, but for networking I kind of doubt the limitation would have been the bus number and would more likely be issues with packet replication and PCIe throughput, especially when you start dealing with east-west traffic within the same system.
We have been seeing deployments already hitting the 256 limit. This is not a "theoretical use" patch set. There are already VM and container farms with SW networking that support much more than 256 VM/containers per server.
The optimization here is to reduce the hypervisor workload and free up CPU cycles for the VMs/containers to consume. This means less handling of packets in the CPU, especially for VM cases w/ SRIOV or VDPA.
Even the extra DMA on the NIC is not really a big deal. These are 400G NICs with big fast PCI. If you top them out you are already doing an aggregate of 400G of network traffic. That is a big number for a single sever, it is OK.
Someone might feel differently if they did this on a 10/40G NIC, in which case this is not the solution for their application.
Quoted 6 lines
Sorry you used the word "replace", and my assumption here was that the goal is to get something in place that can take the place of SR-IOV so that you wouldn't be maintaining the two systems at the same time. That is my concern as I don't want us having SR-IOV, and then several flavors of SIOV. We need to decide on one thing that will be the way forward.
SRIOV has to continue until the PASID and IMS platform features are widely available and mature. It will probably be 10 years before we see most people able to use SIOV for everything they want.
I think we will see lots of SIOV varients, I know Intel is already pushing SIOV parts outside netdev.
Quoted 5 lines
I get that. That is why I said switchdev isn't a standard for the endpoint. One of the biggest issues with SR-IOV that I have seen is the fact that the last piece isn't really defined. We never did a good job of defining how the ADI should look to the guest and as a result it kind of stalled in adoption.
The ADI is supposed to present the HW programming API that is desired. It is always up to the implementation.
SIOV was never a project to standardize HW programming models like virtio-net, NVMe, etc.
Quoted 6 lines
Quoted 2 lines
I'm a bit surprised HW that can do macvlan can't be modeled with switchdev? What is missing?
If I recall it was the fact that the hardware defaults to transmitting everything that doesn't match an existing rule to the external port unless it comes from the external port.
That seems small enough it should be resolvable, IMHO. eg some new switch rule that matches that specific HW behavior?
Quoted 2 lines
Something like the vdpa model is more like what I had in mind. Only vdpa only works for the userspace networking case.
That's because making a driver that converts the native HW to VDPA and then running a generic netdev on the resulting virtio-net is a pretty wild thing to do. I can't really think of an actual use case.
Quoted 2 lines
Basically the idea is to have an assignable device interface that isn't directly tied to the hardware.
The switchdev model is to create a switch port. As I explained in Linux we see "pci device" and "aux device" as being some "user port" options to access to that switch.
If you want a "generic device" that is fine, but what exactly is that programming interface in Linux? Sketch out an API, where does the idea go? What does the driver that implement it look like? What consumes it?
Should this be a great idea, then a mlx5 version of this will still be to create an SF aux device, bind mlx5_core, then bind "generic device" on top of that. This is simply a reflection of how the mlx5 HW/SW layering works. Squashing all of this into a single layer is work with no bad ROI.
Quoted 2 lines
they are pushed into containers you don't have to rip them out if for some reason you need to change the network configuration.
Why would you need to rip them out?
Jason
Quoted 20 lines · Leon Romanovsky
Quoted 11 lines · Jakub Kicinski
Quoted 10 lines · On Tue, 15 Dec 2020 12:35:20 -0800 Saeed Mahameed wrote:
Quoted 9 lines
I think the big thing we really should do if we are going to go this
route is to look at standardizing what the flavours are that get
created by the parent netdevice. Otherwise we are just creating the
same mess we had with SRIOV all over again and muddying the waters of
mediated devices.Quoted 7 lines
Quoted 5 lines
yes in the near future we will be working on auxbus interfaces for
auto-probing and user flavor selection, this is a must have feature for
us.Can you elaborate? I thought config would be via devlink.
Quoted 8 lines
Yes, everything continues to be done through devlink.
One of the immediate features is an ability to disable/enable creation
of specific SF types.For example, if user doesn't want RDMA, the SF RDMA won't be created.
Devlink is an option too, we still don't have our mind set on a specific API, we are considering both as a valuable solutions since devlink make sense as a go to interface for everything SF, but on the other hand, auto-probing and device instantiating is done at the auxbus level, so it also make sense to have some sort of "device type" user selection api in the auxbus, anyway this discussion is for a future patch.
Quoted 10 lines · Jason Gunthorpe
Quoted 6 lines · Alexander Duyck
You say this will scale better but I am not even sure about that. The fact is SR-IOV could scale to 256 VFs, but for networking I kind of doubt the limitation would have been the bus number and would more likely be issues with packet replication and PCIe throughput, especially when you start dealing with east-west traffic within the same system.
Quoted 4 lines
We have been seeing deployments already hitting the 256 limit. This is not a "theoretical use" patch set. There are already VM and container farms with SW networking that support much more than 256 VM/containers per server.
That has been the case for a long time. However it had been my experience that SR-IOV never scaled well to meet those needs and so it hadn't been used in such deployments.
Quoted 8 lines
The optimization here is to reduce the hypervisor workload and free up CPU cycles for the VMs/containers to consume. This means less handling of packets in the CPU, especially for VM cases w/ SRIOV or VDPA.
Even the extra DMA on the NIC is not really a big deal. These are 400G NICs with big fast PCI. If you top them out you are already doing an aggregate of 400G of network traffic. That is a big number for a single sever, it is OK.
Yes, but at a certain point you start bumping up against memory throughput limitations as well. Doubling up the memory footprint by having the device have to write to new pages instead of being able to do something like pinning and zero-copy would be expensive.
For something like an NFV use case it might make sense, but in the general high server count case it seems like it is a setup that would be detrimental.
Quoted 2 lines
Someone might feel differently if they did this on a 10/40G NIC, in which case this is not the solution for their application.
My past experience was with 10/40G NIC with tens of VFs. When we start talking about hundreds I would imagine the overhead becomes orders of magnitudes worse as the problem becomes more of an n^2 issue since you will have n times more systems sending to n times more systems receiving. As such things like broadcast traffic would end up consuming a fair bit of traffic.
Quoted 7 lines
Quoted 6 lines
Sorry you used the word "replace", and my assumption here was that the goal is to get something in place that can take the place of SR-IOV so that you wouldn't be maintaining the two systems at the same time. That is my concern as I don't want us having SR-IOV, and then several flavors of SIOV. We need to decide on one thing that will be the way forward.
Quoted 6 lines
SRIOV has to continue until the PASID and IMS platform features are widely available and mature. It will probably be 10 years before we see most people able to use SIOV for everything they want.
I think we will see lots of SIOV varients, I know Intel is already pushing SIOV parts outside netdev.
The key bit here is outside of netdev. Like I said, SIOV and SR-IOV tend to be PCIe specific specifications. What we are defining here is how the network interfaces presented by such devices will work.
Quoted 11 lines
Quoted 5 lines
I get that. That is why I said switchdev isn't a standard for the endpoint. One of the biggest issues with SR-IOV that I have seen is the fact that the last piece isn't really defined. We never did a good job of defining how the ADI should look to the guest and as a result it kind of stalled in adoption.
The ADI is supposed to present the HW programming API that is desired. It is always up to the implementation.
SIOV was never a project to standardize HW programming models like virtio-net, NVMe, etc.
I agree. Just like SR-IOV never spelled out that network devices should be using switchdev. That is something we decided on as a community. What I am explaining here is that we should be thinking about the implications of how the network interface is exposed in the host in the case of subfunctions that are associated with a given switchdev device.
Quoted 9 lines
Quoted 6 lines
Quoted 2 lines
I'm a bit surprised HW that can do macvlan can't be modeled with switchdev? What is missing?
If I recall it was the fact that the hardware defaults to transmitting everything that doesn't match an existing rule to the external port unless it comes from the external port.
That seems small enough it should be resolvable, IMHO. eg some new switch rule that matches that specific HW behavior?
I would have to go digging to find the conversation. It was about 3 or 4 years ago. I seem to recall mentioning the idea of having some static rules but it was a no-go at the time. If we wanted to spin off this conversation and pull in some Intel folks I would be up for us revisiting it. However I'm not with Intel anymore so it would mostly be something I would be working on as a hobby project instead of anything serious.
Quoted 6 lines
Quoted 2 lines
Something like the vdpa model is more like what I had in mind. Only vdpa only works for the userspace networking case.
That's because making a driver that converts the native HW to VDPA and then running a generic netdev on the resulting virtio-net is a pretty wild thing to do. I can't really think of an actual use case.
I'm not talking about us drastically changing existing models. I would still expect the mlx5 driver to be running on top of the aux device. However it may be that the aux device is associated with something like the switchdev port as a parent and the output from the traffic is then going to the subfunction netdev.
Quoted 12 lines
Quoted 2 lines
Basically the idea is to have an assignable device interface that isn't directly tied to the hardware.
The switchdev model is to create a switch port. As I explained in Linux we see "pci device" and "aux device" as being some "user port" options to access to that switch.
If you want a "generic device" that is fine, but what exactly is that programming interface in Linux? Sketch out an API, where does the idea go? What does the driver that implement it look like? What consumes it?
Quoted 5 lines
Should this be a great idea, then a mlx5 version of this will still be to create an SF aux device, bind
mlx5_core, then bind "generic device" on top of that. This is simply a reflection of how the mlx5 HW/SW layering works. Squashing all of this into a single layer is work with no bad ROI.
In my mind the mlx5_core is still binding to the SF aux device so I am fine with that. The way I view this working is that it would work sort of like the macvlan approach that was taken with the Intel parts. Although instead of binding to the PF it might make more sense to have it bound to the switchdev port associated with the subfunction and looking like a veth pair from the host perspective in terms of behavior. The basic idea is most of the control is still in the switchdev port, but it becomes more visible that the switchdev port and the subfunction netdev are linked with the switchdev port as the parent.
Quoted 4 lines
Quoted 2 lines
they are pushed into containers you don't have to rip them out if for some reason you need to change the network configuration.
Why would you need to rip them out?
Because they are specifically tied to the mlx5 device. So if for example I need to hotplug out the mlx5 and replace it, it would be useful to allow the interface in the containers to stay in place and fail over to some other software backing interface in the switchdev namespace. One of the issues with VFs is that we have always had to push some sort of bond on top, or switch over to a virtio-net interface in order to support fail-over. If we can resolve that in the host case that would go a long way toward solving one of the main issues of SR-IOV.
Jason Gunthorpe
maintainerQuoted 3 lines · Alexander Duyck
That has been the case for a long time. However it had been my experience that SR-IOV never scaled well to meet those needs and so it hadn't been used in such deployments.
Seems to be going quite well here, perhaps the applications are different.
Quoted 9 lines
Quoted 8 lines
The optimization here is to reduce the hypervisor workload and free up CPU cycles for the VMs/containers to consume. This means less handling of packets in the CPU, especially for VM cases w/ SRIOV or VDPA.
Even the extra DMA on the NIC is not really a big deal. These are 400G NICs with big fast PCI. If you top them out you are already doing an aggregate of 400G of network traffic. That is a big number for a single sever, it is OK.
Quoted 4 lines
Yes, but at a certain point you start bumping up against memory throughput limitations as well. Doubling up the memory footprint by having the device have to write to new pages instead of being able to do something like pinning and zero-copy would be expensive.
You can't zero-copy when using VMs.
And when using containers every skb still has to go through all the switching and encapsulation logic, which is not free in SW.
At a certain point the gains of avoiding the DMA copy are lost by the costs of all the extra CPU work. The factor being optimized here is CPU capacity.
Quoted 7 lines
Quoted 2 lines
Someone might feel differently if they did this on a 10/40G NIC, in which case this is not the solution for their application.
My past experience was with 10/40G NIC with tens of VFs. When we start talking about hundreds I would imagine the overhead becomes orders of magnitudes worse as the problem becomes more of an n^2 issue since you will have n times more systems sending to n times more systems
The traffic demand is application dependent. If an application has an n^2 traffic pattern then it needs a network to sustain that cross sectional bandwidth regardless of how the VMs are packed.
It just becomes a design factor of the network and now the network includes that switching component on the PCIe NIC as part of the capacity for cross sectional BW.
There is some balance where a VM can only generate so much traffic based on the CPU it has available, and you can design the entire infrastructure to balance the CPU with the NIC with the switches and come to some packing factor of VMs.
As CPU constrains VM performance, removing CPU overheads from the system will improve packing density. A HW network data path in the VMs is one such case that can turn to a net win if the CPU bottleneck is bigger than the network bottleneck.
It is really over simplifiying to just say PCIe DMA copies are bad.
Quoted 2 lines
receiving. As such things like broadcast traffic would end up consuming a fair bit of traffic.
I think you have a lot bigger network problems if your broadcast traffic is so high that you start to worry about DMA copy performance in a 400G NIC.
Quoted 3 lines
The key bit here is outside of netdev. Like I said, SIOV and SR-IOV tend to be PCIe specific specifications. What we are defining here is how the network interfaces presented by such devices will work.
I think we've achieved this..
Quoted 10 lines
Quoted 2 lines
That seems small enough it should be resolvable, IMHO. eg some new switch rule that matches that specific HW behavior?
I would have to go digging to find the conversation. It was about 3 or 4 years ago. I seem to recall mentioning the idea of having some static rules but it was a no-go at the time. If we wanted to spin off this conversation and pull in some Intel folks I would be up for us revisiting it. However I'm not with Intel anymore so it would mostly be something I would be working on as a hobby project instead of anything serious.
Personally I welcome getting more drivers to implement the switchdev model, I think it is only good for the netdev community as as a whole to understand and standardize on this.
Quoted 11 lines
Quoted 6 lines
Quoted 2 lines
Something like the vdpa model is more like what I had in mind. Only vdpa only works for the userspace networking case.
That's because making a driver that converts the native HW to VDPA and then running a generic netdev on the resulting virtio-net is a pretty wild thing to do. I can't really think of an actual use case.
I'm not talking about us drastically changing existing models. I would still expect the mlx5 driver to be running on top of the aux device. However it may be that the aux device is associated with something like the switchdev port as a parent
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
That is exactly how this works. The switchdev representor and the aux device are paired and form the analog of the veth tunnel. IIRC this relationship with the aux device is shown in the devlink output for the switchdev ports.
I still can't understand what you think should be changed here.
We can't get rid of the aux device, it is integral to the software layering and essential to support any device assignment flow.
We can't add a mandatory netdev, because that is just pure waste for any device assignment flow.
Quoted 3 lines
The basic idea is most of the control is still in the switchdev port, but it becomes more visible that the switchdev port and the subfunction netdev are linked with the switchdev port as the parent.
If a user netdev is created, say for a container, then it should have as a parent the aux device.
If you inspect the port configuration on the switchdev side it will show the aux device associated with the port.
Are you just asking that the userspace tools give a little help and show that switchdev port XX is visable as netdev YYY by cross matching the auxbus?
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
they are pushed into containers you don't have to rip them out if for some reason you need to change the network configuration.
Why would you need to rip them out?
Because they are specifically tied to the mlx5 device. So if for example I need to hotplug out the mlx5 and replace it,
Uhh, I think we are very very far away from being able to hot unplug a switchdev driver, keep the switchdev running, and drop in a different driver.
That isn't even on the radar, AFAIK.
Quoted 5 lines
namespace. One of the issues with VFs is that we have always had to push some sort of bond on top, or switch over to a virtio-net interface in order to support fail-over. If we can resolve that in the host case that would go a long way toward solving one of the main issues of SR-IOV.
This is all solved already, virtio-net is the answer.
qemu can swap back ends under the virtio-net ADI it created on the fly. This means it can go from processing a virtio-net queue in mlx5 HW, to full SW, to some other HW on another machine. All hitlessly and transparently to the guest VM.
Direct HW processing of a queue inside a VM without any downsides for VM migration. Check.
I have the feeling this stuff you are asking for is already done..
Jason
Quoted 54 lines · Jason Gunthorpe
Quoted 3 lines · Alexander Duyck
That has been the case for a long time. However it had been my experience that SR-IOV never scaled well to meet those needs and so it hadn't been used in such deployments.
Seems to be going quite well here, perhaps the applications are different.
Quoted 9 lines
Quoted 8 lines
The optimization here is to reduce the hypervisor workload and free up CPU cycles for the VMs/containers to consume. This means less handling of packets in the CPU, especially for VM cases w/ SRIOV or VDPA.
Even the extra DMA on the NIC is not really a big deal. These are 400G NICs with big fast PCI. If you top them out you are already doing an aggregate of 400G of network traffic. That is a big number for a single sever, it is OK.
Quoted 4 lines
Yes, but at a certain point you start bumping up against memory throughput limitations as well. Doubling up the memory footprint by having the device have to write to new pages instead of being able to do something like pinning and zero-copy would be expensive.
You can't zero-copy when using VMs.
And when using containers every skb still has to go through all the switching and encapsulation logic, which is not free in SW.
At a certain point the gains of avoiding the DMA copy are lost by the costs of all the extra CPU work. The factor being optimized here is CPU capacity.
Quoted 7 lines
Quoted 2 lines
Someone might feel differently if they did this on a 10/40G NIC, in which case this is not the solution for their application.
My past experience was with 10/40G NIC with tens of VFs. When we start talking about hundreds I would imagine the overhead becomes orders of magnitudes worse as the problem becomes more of an n^2 issue since you will have n times more systems sending to n times more systems
The traffic demand is application dependent. If an application has an n^2 traffic pattern then it needs a network to sustain that cross sectional bandwidth regardless of how the VMs are packed.
It just becomes a design factor of the network and now the network includes that switching component on the PCIe NIC as part of the capacity for cross sectional BW.
There is some balance where a VM can only generate so much traffic based on the CPU it has available, and you can design the entire infrastructure to balance the CPU with the NIC with the switches and come to some packing factor of VMs.
Quoted 6 lines
As CPU constrains VM performance, removing CPU overheads from the system will improve packing density. A HW network data path in the VMs is one such case that can turn to a net win if the CPU bottleneck is bigger than the network bottleneck.
It is really over simplifiying to just say PCIe DMA copies are bad.
I'm not saying the copies are bad. However they can be limiting. As you said it all depends on the use case. If you have nothing but functions that are performing bump-in-the-wire type operations odds are the PCIe bandwidth won't be the problem. It all depends on the use case and that is why I would prefer the interface to be more flexible rather than just repeating what has been done with SR-IOV.
The problem in my case was based on a past experience where east-west traffic became a problem and it was easily shown that bypassing the NIC for traffic was significantly faster.
Quoted 6 lines
Quoted 2 lines
receiving. As such things like broadcast traffic would end up consuming a fair bit of traffic.
I think you have a lot bigger network problems if your broadcast traffic is so high that you start to worry about DMA copy performance in a 400G NIC.
Usually the problems were more multicast rather than broadcast, but yeah this typically isn't an issue. However still at 256 VFs you would be talking about a replication rate such that at 1-2Gbps one VF could saturate the entire 400G device.
Quoted 5 lines
Quoted 3 lines
The key bit here is outside of netdev. Like I said, SIOV and SR-IOV tend to be PCIe specific specifications. What we are defining here is how the network interfaces presented by such devices will work.
I think we've achieved this..
Somewhat. We have it explained for the control plane. What we are defining now is how it will appear in the guest/container.
Quoted 11 lines
Quoted 10 lines
Quoted 2 lines
That seems small enough it should be resolvable, IMHO. eg some new switch rule that matches that specific HW behavior?
I would have to go digging to find the conversation. It was about 3 or 4 years ago. I seem to recall mentioning the idea of having some static rules but it was a no-go at the time. If we wanted to spin off this conversation and pull in some Intel folks I would be up for us revisiting it. However I'm not with Intel anymore so it would mostly be something I would be working on as a hobby project instead of anything serious.
Quoted 3 lines
Personally I welcome getting more drivers to implement the switchdev model, I think it is only good for the netdev community as as a whole to understand and standardize on this.
Agreed.
Quoted 18 lines
Quoted 11 lines
Quoted 6 lines
Quoted 2 lines
Something like the vdpa model is more like what I had in mind. Only vdpa only works for the userspace networking case.
That's because making a driver that converts the native HW to VDPA and then running a generic netdev on the resulting virtio-net is a pretty wild thing to do. I can't really think of an actual use case.
I'm not talking about us drastically changing existing models. I would still expect the mlx5 driver to be running on top of the aux device. However it may be that the aux device is associated with something like the switchdev port as a parent
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
That is exactly how this works. The switchdev representor and the aux device are paired and form the analog of the veth tunnel. IIRC this relationship with the aux device is shown in the devlink output for the switchdev ports.
Quoted 7 lines
I still can't understand what you think should be changed here.
We can't get rid of the aux device, it is integral to the software layering and essential to support any device assignment flow.
We can't add a mandatory netdev, because that is just pure waste for any device assignment flow.
I'm not saying to get rid of the netdev. I'm saying the netdev created should be a generic interface that could be reused by other vendors. The whole idea behind using something like macvlan is to hide the underlying device. It becomes the namespace assignable interface. What I would like to see is us get rid of that step and instead just have a generic interface spawned in the first place and push the driver specific bits back to the switchdev port.
The problem right now is that the switchdev netdev and the subfunction netdev are treated as peers. In my mind it should work somewhere in between the macvlan and veth. Basically you have the switchedev port handling both ends of the traffic and handling the aux device directly, and the subfunction device is floating on top of it sometimes acting like a macvlan when traffic is going to/from the aux device, and acting like a veth pair in some cases perhaps such as broadcast/multicast allowing the swtichdev to take care of that locally.
Quoted 7 lines
Quoted 3 lines
The basic idea is most of the control is still in the switchdev port, but it becomes more visible that the switchdev port and the subfunction netdev are linked with the switchdev port as the parent.
If a user netdev is created, say for a container, then it should have as a parent the aux device.
Quoted 6 lines
If you inspect the port configuration on the switchdev side it will show the aux device associated with the port.
Are you just asking that the userspace tools give a little help and show that switchdev port XX is visable as netdev YYY by cross matching the auxbus?
It isn't about the association, it is about who is handling the traffic. Going back to the macvlan model what we did is we had a group of rings on the device that would automatically forward unicast packets to the macvlan interface and would be reserved for transmitting packets from the macvlan interface. We took care of multicast and broadcast replication in software.
In my mind it should be possible to do something similar for the swtichdev case. Basically the packets would be routed from the subfunction netdev, to the switchdev netdev, and could then be transmitted through the aux device. In order to make this work in the case of ixgbe I had come up with the concept of a subordinate device, "sb_dev", which was a pointer used in the Tx case to identify Tx rings and qdiscs that had been given to macvlan interface.
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Quoted 2 lines
they are pushed into containers you don't have to rip them out if for some reason you need to change the network configuration.
Why would you need to rip them out?
Quoted 8 lines
Quoted 2 lines
Because they are specifically tied to the mlx5 device. So if for example I need to hotplug out the mlx5 and replace it,
Uhh, I think we are very very far away from being able to hot unplug a switchdev driver, keep the switchdev running, and drop in a different driver.
That isn't even on the radar, AFAIK.
That might be a bad example, I was thinking of the issues we have had with VFs and direct assignment to Qemu based guests in the past. Essentially what I am getting at is that the setup in the container should be vendor agnostic. The interface exposed shouldn't be specific to any one vendor. So if I want to fire up a container or Mellanox, Broadcom, or some other vendor it shouldn't matter or be visible to the user. They should just see a vendor agnostic subfunction netdevice.
Something like that is doable already using something like a macvlan on top of a subfunction interface, but I feel like that is an unnecessary step and creates unnecessary netdevices.
Quoted 13 lines
Quoted 5 lines
namespace. One of the issues with VFs is that we have always had to push some sort of bond on top, or switch over to a virtio-net interface in order to support fail-over. If we can resolve that in the host case that would go a long way toward solving one of the main issues of SR-IOV.
This is all solved already, virtio-net is the answer.
qemu can swap back ends under the virtio-net ADI it created on the fly. This means it can go from processing a virtio-net queue in mlx5 HW, to full SW, to some other HW on another machine. All hitlessly and transparently to the guest VM.
Quoted 4 lines
Direct HW processing of a queue inside a VM without any downsides for VM migration. Check.
I have the feeling this stuff you are asking for is already done..
The case you are describing has essentially solved it for Qemu virtualization and direct assignment. It still doesn't necessarily solve it for the container case though.
Jason Gunthorpe
maintainerQuoted 6 lines · Alexander Duyck
It isn't about the association, it is about who is handling the traffic. Going back to the macvlan model what we did is we had a group of rings on the device that would automatically forward unicast packets to the macvlan interface and would be reserved for transmitting packets from the macvlan interface. We took care of multicast and broadcast replication in software.
Okay, maybe I'm starting to see where you are coming from.
First, I think some clarity here, as I see it the devlink infrastructure is all about creating the auxdevice for a switchdev port.
What goes into that auxdevice is *completely* up to the driver. mlx5 is doing a SF which == VF, but that is not a requirement of the design at all.
If an Intel driver wants to put a queue block into the aux device and that is != VF, it is just fine.
The Intel netdev that binds to the auxdevice can transform the queue block and specific switchdev config into a netdev identical to accelerated macvlan. Nothing about the breaks the switchdev model.
Essentially think of it as generalizing the acceleration plugin for a netdev. Instead of making something specific to limited macvlan, the driver gets to provide exactly the structure that matches its HW to provide the netdev as the user side of the switchdev port. I see no limitation here so long as the switchdev model for controlling traffic is followed.
Let me segue into a short story from RDMA.. We've had a netdev called IPoIB for a long time. It is actually kind of similar to this general thing you are talking about, in that there is a programming layer under the IPOIB netdev called RDMA verbs that generalizes the actual HW. Over the years this became more complicated because every new netdev offloaded needed mirroring into the RDMA verbs general API. TSO, GSO, checksum offload, endlessly onwards. It became quite dumb in the end. We gave up and said the HW driver should directly implement netdev. Implementing a middle API layer makes zero sense when netdev is already perfectly suited to implement ontop of HW. Removing SW layers caused performance to go up something like 2x.
The hard earned lesson I take from that is don't put software layers between a struct net_device and the actual HW. The closest coupling is really the best thing. Provide libary code in the kernel to help drivers implement common patterns when making their netdevs, do not provide wrapper netdevs around drivers.
IMHO the approach of macvlan accleration made some sense in 2013, but today I would say it is mashing unrelated layers together and polluting what should be a pure SW implementation with HW hooks.
I see from the mailing list comments this was done because creating a device specific netdev via 'ip link add' was rightly rejected. However here we *can* create a device specific vmdq *auxdevice*. This is OK because the netdev is controlling and containing the aux device via switchdev.
So, Intel can get the "VMDQ link type" that was originally desired more or less directly, so long as the associated switchdev port controls the MAC filter process, not "ip link add".
And if you want to make the vmdq auxdevice into an ADI by user DMA to queues, then sure, that model is completely sane too (vs hacking up macvlan to expose user queues) - so long as the kernel controls the selection of traffic into those queues and follows the switchdev model. I would recommend creating a simple RDMA raw ethernet queue driver over the aux device for something like this :)
Quoted 2 lines
That might be a bad example, I was thinking of the issues we have had with VFs and direct assignment to Qemu based guests in the past.
As described, this is solved by VDPA.
Quoted 6 lines
Essentially what I am getting at is that the setup in the container should be vendor agnostic. The interface exposed shouldn't be specific to any one vendor. So if I want to fire up a container or Mellanox, Broadcom, or some other vendor it shouldn't matter or be visible to the user. They should just see a vendor agnostic subfunction netdevice.
Agree. The agnostic container user interface here is 'struct net_device'.
Quoted 5 lines
Quoted 1 line
I have the feeling this stuff you are asking for is already done..
The case you are describing has essentially solved it for Qemu virtualization and direct assignment. It still doesn't necessarily solve it for the container case though.
The container case doesn't need solving.
Any scheme I've heard for container live migration, like CRIU, essentially hot plugs the entire kernel in/out of a user process. We rely on the kernel providing low leakage of the implementation details of the struct net_device as part of it's uAPI contract. When CRIU swaps the kernel the new kernel can have any implementation of the container netdev it wants.
I've never heard of a use case to hot swap the implemention *under* a netdev from a container. macvlan can't do this today. If you have a use case here, it really has nothing to do with with this series.
Jason
Quoted 20 lines · Jason Gunthorpe
Quoted 6 lines · Alexander Duyck
It isn't about the association, it is about who is handling the traffic. Going back to the macvlan model what we did is we had a group of rings on the device that would automatically forward unicast packets to the macvlan interface and would be reserved for transmitting packets from the macvlan interface. We took care of multicast and broadcast replication in software.
Okay, maybe I'm starting to see where you are coming from.
First, I think some clarity here, as I see it the devlink infrastructure is all about creating the auxdevice for a switchdev port.
What goes into that auxdevice is *completely* up to the driver. mlx5 is doing a SF which == VF, but that is not a requirement of the design at all.
Quoted 6 lines
If an Intel driver wants to put a queue block into the aux device and that is != VF, it is just fine.
The Intel netdev that binds to the auxdevice can transform the queue block and specific switchdev config into a netdev identical to accelerated macvlan. Nothing about the breaks the switchdev model.
Just to clarify I am not with Intel, nor do I plan to work on any Intel drivers related to this.
My concern has more to do with how this is being plumbed and the fact that the basic architecture is somewhat limiting.
Quoted 6 lines
Essentially think of it as generalizing the acceleration plugin for a netdev. Instead of making something specific to limited macvlan, the driver gets to provide exactly the structure that matches its HW to provide the netdev as the user side of the switchdev port. I see no limitation here so long as the switchdev model for controlling traffic is followed.
I see plenty. The problem is it just sets up more vendor lock-in and features that have to be thrown away when you have to settle for least-common denominator in order to maintain functionality across vendors.
Quoted 19 lines
Let me segue into a short story from RDMA.. We've had a netdev called IPoIB for a long time. It is actually kind of similar to this general thing you are talking about, in that there is a programming layer under the IPOIB netdev called RDMA verbs that generalizes the actual HW. Over the years this became more complicated because every new netdev offloaded needed mirroring into the RDMA verbs general API. TSO, GSO, checksum offload, endlessly onwards. It became quite dumb in the end. We gave up and said the HW driver should directly implement netdev. Implementing a middle API layer makes zero sense when netdev is already perfectly suited to implement ontop of HW. Removing SW layers caused performance to go up something like 2x.
The hard earned lesson I take from that is don't put software layers between a struct
net_deviceand the actual HW. The closest coupling is really the best thing. Provide libary code in the kernel to help drivers implement common patterns when making their netdevs, do not provide wrapper netdevs around drivers.
Quoted 3 lines
IMHO the approach of macvlan accleration made some sense in 2013, but today I would say it is mashing unrelated layers together and polluting what should be a pure SW implementation with HW hooks.
I disagree here. In my mind a design where two interfaces, which both exist in the kernel, have to go to hardware in order to communicate is very limiting. The main thing I am wanting to see is the option of being able to pass traffic directly between the switchdev and the SF without the need to touch the hardware.
An easy example of such traffic that would likely benefit from this is multicast/broadcast traffic. Instead of having to process each and every broadcast packet in hardware you could very easily process it at the switchdev and then directly hand it off from the switchdev to the SF in this case instead of having to send it to hardware for each switchdev instance.
Quoted 10 lines
I see from the mailing list comments this was done because creating a device specific netdev via 'ip link add' was rightly rejected. However here we *can* create a device specific vmdq *auxdevice*. This is OK because the netdev is controlling and containing the aux device via switchdev.
So, Intel can get the "VMDQ link type" that was originally desired more or less directly, so long as the associated switchdev port controls the MAC filter process, not "ip link add".
Quoted 6 lines
And if you want to make the vmdq auxdevice into an ADI by user DMA to queues, then sure, that model is completely sane too (vs hacking up macvlan to expose user queues) - so long as the kernel controls the selection of traffic into those queues and follows the switchdev model. I would recommend creating a simple RDMA raw ethernet queue driver over the aux device for something like this :)
You lost me here, I'm not seeing how RDMA and macvlan are connected.
Quoted 12 lines
Quoted 2 lines
That might be a bad example, I was thinking of the issues we have had with VFs and direct assignment to Qemu based guests in the past.
As described, this is solved by VDPA.
Quoted 6 lines
Essentially what I am getting at is that the setup in the container should be vendor agnostic. The interface exposed shouldn't be specific to any one vendor. So if I want to fire up a container or Mellanox, Broadcom, or some other vendor it shouldn't matter or be visible to the user. They should just see a vendor agnostic subfunction netdevice.
Quoted 2 lines
Agree. The agnostic container user interface here is 'struct
net_device'.
I disagree here. The fact is a mellanox netdev, versus a broadcom netdev, versus an intel netdev all have a very different look at feel as the netdev is essentially just the base device you are building around.
In addition it still doesn't address my concern as called out above which is the east-west traffic problem.
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
I have the feeling this stuff you are asking for is already done..
The case you are describing has essentially solved it for Qemu virtualization and direct assignment. It still doesn't necessarily solve it for the container case though.
The container case doesn't need solving.
I disagree and that is at the heart where you and I have different views. I view there being two advantages to having the container case solved:
1. A standardized set of features that can be provided regardless of vendor
2. Allowing for the case where east-west traffic can avoid having to touch hardware
Quoted 6 lines
Any scheme I've heard for container live migration, like CRIU, essentially hot plugs the entire kernel in/out of a user process. We rely on the kernel providing low leakage of the implementation details of the struct
net_deviceas part of it's uAPI contract. When CRIU swaps the kernel the new kernel can have any implementation of the container netdev it wants.
I'm not thinking about migration. I am thinking more about the user experience. In my mind if I set up a container I shouldn't need to know which vendor provided the network interface when I set it up. The problem is most NICs have so many one-off proprietary tweaks needed that it gets annoying. That is why in my mind it would make much more sense to have a simple vendor agnostic interface. That is why I would prefer to avoid the VF model.
Quoted 3 lines
I've never heard of a use case to hot swap the implemention *under* a netdev from a container. macvlan can't do this today. If you have a use case here, it really has nothing to do with with this series.
Again, the hot-swap isn't necessarily what I am talking about. I am talking about setting up a config for a set of containers in a datacenter. What I don't want to do is have to have one set of configs for an mlx5 SF, another for a broadcom SF, and yet another set for any other vendors out there. I would much rather have all of that dealt with within the namespace that is handling the switchdev setup.
In addition, the east-west traffic is the other bit I would like to see addressed. I am okay excusing this in the case of direct assignment since the resources for the SF will not be available to the host. However if the SF will be operating in the same kernel as the PF/switchev it would make much more sense to enable an east/west channel which would allow for hardware bypass under certain circumstances without having to ever leave the kernel.
Jason Gunthorpe
maintainerQuoted 2 lines · Alexander Duyck
Just to clarify I am not with Intel, nor do I plan to work on any Intel drivers related to this.
Sure
Quoted 5 lines
I disagree here. In my mind a design where two interfaces, which both exist in the kernel, have to go to hardware in order to communicate is very limiting. The main thing I am wanting to see is the option of being able to pass traffic directly between the switchdev and the SF without the need to touch the hardware.
I view the SW bypass path you are talking about similarly to GSO/etc. It should be accessed by the HW driver as an optional service provided by the core netdev, not implemented as some wrapper netdev around a HW implementation.
If you feel strongly it is needed then there is nothing standing in the way to implement it in the switchdev auxdevice model.
It is simple enough, the HW driver's tx path would somehow detect east/west and queue it differently, and the rx path would somehow be able to mux in skbs from a SW queue. Not seeing any blockers here.
Quoted 4 lines
Quoted 2 lines
model. I would recommend creating a simple RDMA raw ethernet queue driver over the aux device for something like this :)
You lost me here, I'm not seeing how RDMA and macvlan are connected.
RDMA is the standard uAPI to get a userspace HW DMA queue for ethernet packets.
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Essentially what I am getting at is that the setup in the container should be vendor agnostic. The interface exposed shouldn't be specific to any one vendor. So if I want to fire up a container or Mellanox, Broadcom, or some other vendor it shouldn't matter or be visible to the user. They should just see a vendor agnostic subfunction netdevice.
Quoted 7 lines
Quoted 2 lines
Agree. The agnostic container user interface here is 'struct
net_device'.I disagree here. The fact is a mellanox netdev, versus a broadcom netdev, versus an intel netdev all have a very different look at feel as the netdev is essentially just the base device you are building around.
Then fix the lack of standardization of netdev implementations!
Adding more abstraction layers isn't going to fix that fundamental problem.
Frankly it seems a bit absurd to complain that the very basic element of the common kernel uAPI - struct net_device - is so horribly fragmented and vendor polluted that we can't rely on it as a stable interface for containers.
Even if that is true, I don't belive for a second that adding a different HW abstraction layer is going to somehow undo the mistakes of the last 20 years.
Quoted 6 lines
Again, the hot-swap isn't necessarily what I am talking about. I am talking about setting up a config for a set of containers in a datacenter. What I don't want to do is have to have one set of configs for an mlx5 SF, another for a broadcom SF, and yet another set for any other vendors out there. I would much rather have all of that dealt with within the namespace that is handling the switchdev setup.
If there is real problems here then I very much encourage you to start an effort to push all the vendors to implement a consistent user experience for the HW netdevs.
I don't know what your issues are, but it sounds like it would be a very interesting conference presentation.
But it has nothing to do with this series.
Jason
Quoted 14 lines · Jason Gunthorpe
Quoted 2 lines · Alexander Duyck
Just to clarify I am not with Intel, nor do I plan to work on any Intel drivers related to this.
Sure
Quoted 5 lines
I disagree here. In my mind a design where two interfaces, which both exist in the kernel, have to go to hardware in order to communicate is very limiting. The main thing I am wanting to see is the option of being able to pass traffic directly between the switchdev and the SF without the need to touch the hardware.
Quoted 4 lines
I view the SW bypass path you are talking about similarly to GSO/etc. It should be accessed by the HW driver as an optional service provided by the core netdev, not implemented as some wrapper netdev around a HW implementation.
I view it as being something that would be a part of the switchdev API itself. Basically the switchev and endpoint would need to be able to control something like this because if XDP were enabled on one end or the other you would need to be able to switch it off so that all of the packets followed the same flow and could be scanned by the XDP program.
Quoted 6 lines
If you feel strongly it is needed then there is nothing standing in the way to implement it in the switchdev auxdevice model.
It is simple enough, the HW driver's tx path would somehow detect east/west and queue it differently, and the rx path would somehow be able to mux in skbs from a SW queue. Not seeing any blockers here.
In my mind the simple proof of concept for this would be to check for the multicast bit being set in the destination MAC address for packets coming from the subfunction. If it is then shunt to this bypass route, and if not then you transmit to the hardware queues. In the case of packets coming from the switchdev port it would probably depend. The part I am not sure about is if any packets need to be actually transmitted to the hardware in the standard case for packets going from the switchdev port to the subfunction. If there is no XDP or anything like that present in the subfunction it probably wouldn't matter and you could just shunt it straight across and bypass the hardware, however if XDP is present you would need to get the packets into the ring which would force the bypass to be turned off.
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
model. I would recommend creating a simple RDMA raw ethernet queue driver over the aux device for something like this :)
You lost me here, I'm not seeing how RDMA and macvlan are connected.
RDMA is the standard uAPI to get a userspace HW DMA queue for ethernet packets.
Ah, I think you are talking about device assignment. In my mind I was just talking about the interface assigned to the container which as you have stated is basically just a netdev.
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Quoted 6 lines
Essentially what I am getting at is that the setup in the container should be vendor agnostic. The interface exposed shouldn't be specific to any one vendor. So if I want to fire up a container or Mellanox, Broadcom, or some other vendor it shouldn't matter or be visible to the user. They should just see a vendor agnostic subfunction netdevice.
Agree. The agnostic container user interface here is 'struct
net_device'.
Quoted 6 lines
Quoted 4 lines
I disagree here. The fact is a mellanox netdev, versus a broadcom netdev, versus an intel netdev all have a very different look at feel as the netdev is essentially just the base device you are building around.
Then fix the lack of standardization of netdev implementations!
We're trying to work on that, but trying to fix it after the fact is like herding cats.
Quoted 7 lines
Adding more abstraction layers isn't going to fix that fundamental problem.
Frankly it seems a bit absurd to complain that the very basic element of the common kernel uAPI - struct
net_device- is so horribly fragmented and vendor polluted that we can't rely on it as a stable interface for containers.
The problem isn't necessarily the net_device it is more the net_device_ops and the fact that there are so many different ways to get things done. Arguably the flexibility of the netd_device is great for allowing vendors to expose their features. However at the same time it allows for features to be left out so what you end up with a wide variety of things that are net_devices.
Quoted 3 lines
Even if that is true, I don't belive for a second that adding a different HW abstraction layer is going to somehow undo the mistakes of the last 20 years.
It depends on how it is done. The general idea is to address the biggest limitation that has occured, which is the fact that in many cases we don't have software offloads to take care of things when the hardware offloads provided by a certain piece of hardware are not present. It would basically allow us to reset the feature set. If something cannot be offloaded in software in a reasonable way, it is not allowed to be present in the interface provided to a container. That way instead of having to do all the custom configuration in the container recipe it can be centralized to one container handling all of the switching and hardware configuration.
Quoted 10 lines
Quoted 6 lines
Again, the hot-swap isn't necessarily what I am talking about. I am talking about setting up a config for a set of containers in a datacenter. What I don't want to do is have to have one set of configs for an mlx5 SF, another for a broadcom SF, and yet another set for any other vendors out there. I would much rather have all of that dealt with within the namespace that is handling the switchdev setup.
If there is real problems here then I very much encourage you to start an effort to push all the vendors to implement a consistent user experience for the HW netdevs.
To some extent that has been going on for some time. It is one of the reasons why there is supposed to be software offloads for any datapath features that get added to the hardware. Such as GSO to offset TSO. However there always ends up the occasional thing that ends up getting past and that is where the frustration comes in.
Quoted 6 lines
I don't know what your issues are, but it sounds like it would be a very interesting conference presentation.
But it has nothing to do with this series.
Jason
There I disagree. Now I can agree that most of the series is about presenting the aux device and that part I am fine with. However when the aux device is a netdev and that netdev is being loaded into the same kernel as the switchdev port is where the red flags start flying, especially when we start talking about how it is the same as a VF.
In my mind we are talking about how the switchdev will behave and it makes sense to see about defining if a east-west bypass makes sense and how it could be implemented, rather than saying we won't bother for now and potentially locking in the subfunction to virtual function equality. In my mind we need more than just the increased count to justify going to subfunctions, and I think being able to solve the east-west problem at least in terms of containers would be such a thing.
Jason Gunthorpe
maintainerQuoted 11 lines · Alexander Duyck
Quoted 4 lines
I view the SW bypass path you are talking about similarly to GSO/etc. It should be accessed by the HW driver as an optional service provided by the core netdev, not implemented as some wrapper netdev around a HW implementation.
I view it as being something that would be a part of the switchdev API itself. Basically the switchev and endpoint would need to be able to control something like this because if XDP were enabled on one end or the other you would need to be able to switch it off so that all of the packets followed the same flow and could be scanned by the XDP program.
To me that still all comes down to being something like an optional offload that the HW driver can trigger if the conditions are met.
Quoted 8 lines
Quoted 3 lines
It is simple enough, the HW driver's tx path would somehow detect east/west and queue it differently, and the rx path would somehow be able to mux in skbs from a SW queue. Not seeing any blockers here.
In my mind the simple proof of concept for this would be to check for the multicast bit being set in the destination MAC address for packets coming from the subfunction. If it is then shunt to this bypass route, and if not then you transmit to the hardware queues.
Sure, not sure multicast optimization like this isn't incredibly niche too, but it would be an interesting path to explore.
But again, there is nothing fundamental about the model here that precludes this optional optimization.
Quoted 9 lines
Quoted 3 lines
Even if that is true, I don't belive for a second that adding a different HW abstraction layer is going to somehow undo the mistakes of the last 20 years.
It depends on how it is done. The general idea is to address the biggest limitation that has occured, which is the fact that in many cases we don't have software offloads to take care of things when the hardware offloads provided by a certain piece of hardware are not present.
This is really disappointing to hear. Admittedly I don't follow all the twists and turns on the mailing list, but I thought having a SW version of everything was one of the fundamental tenants of netdev that truly distinguished it from something like RDMA.
Quoted 6 lines
It would basically allow us to reset the feature set. If something cannot be offloaded in software in a reasonable way, it is not allowed to be present in the interface provided to a container. That way instead of having to do all the custom configuration in the container recipe it can be centralized to one container handling all of the switching and hardware configuration.
Well, you could start by blocking stuff without a SW fallback..
Quoted 5 lines
There I disagree. Now I can agree that most of the series is about presenting the aux device and that part I am fine with. However when the aux device is a netdev and that netdev is being loaded into the same kernel as the switchdev port is where the red flags start flying, especially when we start talking about how it is the same as a VF.
Well, it happens for the same reason a VF can create a netdev, stopping it would actually be more patches. As I said before, people are already doing this model with VFs.
I can agree with some of our points, but this is not the series to argue them. What you want is to start some new thread on optimizing switchdev for the container user case.
Quoted 5 lines
In my mind we are talking about how the switchdev will behave and it makes sense to see about defining if a east-west bypass makes sense and how it could be implemented, rather than saying we won't bother for now and potentially locking in the subfunction to virtual function equality.
At least for mlx5 SF == VF, that is a consequence of the HW. Any SW bypass would need to be specially built in the mlx5 netdev running on a VF/SF attached to a switchdev port.
I don't see anything about this part of the model that precludes ever doing that, and I also don't see this optimization as being valuable enough to block things "just to be sure"
Quoted 3 lines
In my mind we need more than just the increased count to justify going to subfunctions, and I think being able to solve the east-west problem at least in terms of containers would be such a thing.
Increased count is pretty important for users with SRIOV.
Jason
Quoted 3 lines · Alexander Duyck
The problem in my case was based on a past experience where east-west traffic became a problem and it was easily shown that bypassing the NIC for traffic was significantly faster.
If a deployment expects a lot of east-west traffic *within a host* why is it using hardware based isolation like a VF. That is a side effect of a design choice that is remedied by other options.
Quoted 9 lines · David Ahern
Quoted 3 lines · Alexander Duyck
The problem in my case was based on a past experience where east-west traffic became a problem and it was easily shown that bypassing the NIC for traffic was significantly faster.
If a deployment expects a lot of east-west traffic *within a host* why is it using hardware based isolation like a VF. That is a side effect of a design choice that is remedied by other options.
I am mostly talking about this from past experience as I had seen a few instances when I was at Intel when it became an issue. Sales and marketing people aren't exactly happy when you tell them "don't sell that" in response to them trying to sell a feature into an area where it doesn't belong. Generally they want a solution. The macvlan offload addressed these issues as the replication and local switching can be handled in software.
The problem is PCIe DMA wasn't designed to function as a network switch fabric and when we start talking about a 400Gb NIC trying to handle over 256 subfunctions it will quickly reduce the receive/transmit throughput to gigabit or less speeds when encountering hardware multicast/broadcast replication. With 256 subfunctions a simple 60B ARP could consume more than 19KB of PCIe bandwidth due to the packet having to be duplicated so many times. In my mind it should be simpler to simply clone a single skb 256 times, forward that to the switchdev ports, and have them perform a bypass (if available) to deliver it to the subfunctions. That's why I was thinking it might be a good time to look at addressing it.
Quoted 7 lines · Alexander Duyck
Quoted 6 lines · David Ahern
Quoted 3 lines · Alexander Duyck
The problem in my case was based on a past experience where east-west traffic became a problem and it was easily shown that bypassing the NIC for traffic was significantly faster.
Quoted 8 lines
Quoted 3 lines
If a deployment expects a lot of east-west traffic *within a host* why is it using hardware based isolation like a VF. That is a side effect of a design choice that is remedied by other options.
I am mostly talking about this from past experience as I had seen a few instances when I was at Intel when it became an issue. Sales and marketing people aren't exactly happy when you tell them "don't sell that" in response to them trying to sell a feature into an area where
that's a problem engineers can never solve...
Quoted 3 lines
it doesn't belong. Generally they want a solution. The macvlan offload addressed these issues as the replication and local switching can be handled in software.
well, I guess almost never. :-)
Quoted 13 lines
The problem is PCIe DMA wasn't designed to function as a network switch fabric and when we start talking about a 400Gb NIC trying to handle over 256 subfunctions it will quickly reduce the receive/transmit throughput to gigabit or less speeds when encountering hardware multicast/broadcast replication. With 256 subfunctions a simple 60B ARP could consume more than 19KB of PCIe bandwidth due to the packet having to be duplicated so many times. In my mind it should be simpler to simply clone a single skb 256 times, forward that to the switchdev ports, and have them perform a bypass (if available) to deliver it to the subfunctions. That's why I was thinking it might be a good time to look at addressing it.
east-west traffic within a host is more than likely the same tenant in which case a proper VPC is a better solution than the s/w stack trying to detect and guess that a bypass is needed. Guesses cost cycles in the fast path which is a net loss - and even more so as speeds increase.
Quoted 7 lines
From: Alexander Duyck <[email protected]>
Sent: Friday, December 18, 2020 8:41 AMQuoted 2 lines · David Ahern
On 12/16/20 3:53 PM, Alexander Duyck wrote:
Quoted 10 lines
The problem is PCIe DMA wasn't designed to function as a network switch fabric and when we start talking about a 400Gb NIC trying to handle over 256 subfunctions it will quickly reduce the receive/transmit throughput to gigabit or less speeds when encountering hardware multicast/broadcast replication. With 256 subfunctions a simple 60B ARP could consume more than 19KB of PCIe bandwidth due to the packet having to be duplicated so many times. In my mind it should be simpler to simply clone a single skb 256 times, forward that to the switchdev ports, and have them perform a bypass (if available) to deliver it to the subfunctions. That's why I was thinking it might be a good time to look at addressing it.
Linux tc framework is rich to address this and already used by openvswich for years now. Today arp broadcasts are not offloaded. They go through software patch and replicated in the L2 domain. It is a solved problem for many years now.
Quoted 22 lines
From: Parav Pandit <[email protected]>
Sent: Friday, December 18, 2020 10:51 AMQuoted 10 lines
From: Alexander Duyck <[email protected]>
Sent: Friday, December 18, 2020 8:41 AMQuoted 2 lines · David Ahern
On 12/16/20 3:53 PM, Alexander Duyck wrote:
The problem is PCIe DMA wasn't designed to function as a network switch fabric and when we start talking about a 400Gb NIC trying to handle over 256 subfunctions it will quickly reduce the receive/transmit throughput to gigabit or less speeds when encountering
hardware multicast/broadcast replication.
Quoted 6 lines
With 256 subfunctions a simple 60B ARP could consume more than 19KB of PCIe bandwidth due to the packet having to be duplicated so many times. In my mind it should be simpler to simply clone a single skb 256 times, forward that to the switchdev ports, and have them perform a bypass (if available) to deliver it to the subfunctions. That's why I was thinking it might be a good time to look at addressing it.
Quoted 3 lines
Linux tc framework is rich to address this and already used by openvswich for years now.
Today arp broadcasts are not offloaded. They go through software patch and
s/patch/path
Quoted 2 lines
replicated in the L2 domain.
It is a solved problem for many years now.
Quoted 40 lines · David Ahern
Quoted 7 lines · Alexander Duyck
Quoted 6 lines · David Ahern
Quoted 3 lines · Alexander Duyck
The problem in my case was based on a past experience where east-west traffic became a problem and it was easily shown that bypassing the NIC for traffic was significantly faster.
Quoted 8 lines
Quoted 3 lines
If a deployment expects a lot of east-west traffic *within a host* why is it using hardware based isolation like a VF. That is a side effect of a design choice that is remedied by other options.
I am mostly talking about this from past experience as I had seen a few instances when I was at Intel when it became an issue. Sales and marketing people aren't exactly happy when you tell them "don't sell that" in response to them trying to sell a feature into an area where
that's a problem engineers can never solve...
Quoted 3 lines
it doesn't belong. Generally they want a solution. The macvlan offload addressed these issues as the replication and local switching can be handled in software.
well, I guess almost never. :-)
Quoted 13 lines
The problem is PCIe DMA wasn't designed to function as a network switch fabric and when we start talking about a 400Gb NIC trying to handle over 256 subfunctions it will quickly reduce the receive/transmit throughput to gigabit or less speeds when encountering hardware multicast/broadcast replication. With 256 subfunctions a simple 60B ARP could consume more than 19KB of PCIe bandwidth due to the packet having to be duplicated so many times. In my mind it should be simpler to simply clone a single skb 256 times, forward that to the switchdev ports, and have them perform a bypass (if available) to deliver it to the subfunctions. That's why I was thinking it might be a good time to look at addressing it.
Quoted 4 lines
east-west traffic within a host is more than likely the same tenant in which case a proper VPC is a better solution than the s/w stack trying to detect and guess that a bypass is needed. Guesses cost cycles in the fast path which is a net loss - and even more so as speeds increase.
Yes, but this becomes the hardware limitations deciding the layout of the network. I lean towards more flexibility to allow more configuration options being a good thing rather than us needing to dictate how a network has to be constructed based on the limitations of the hardware and software.
For broadcast/multicast it isn't so much a guess. It would be a single bit test. My understanding is the switchdev setup is already making special cases for things like broadcast/multicast due to the extra overhead incurred. I mentioned ARP because in many cases it has to be offloaded specifically due to these sorts of issues.
Quoted 18 lines · Parav Pandit
Quoted 8 lines
From: Alexander Duyck <[email protected]>
Sent: Friday, December 18, 2020 8:41 AMQuoted 2 lines · David Ahern
On 12/16/20 3:53 PM, Alexander Duyck wrote:
Quoted 10 lines
The problem is PCIe DMA wasn't designed to function as a network switch fabric and when we start talking about a 400Gb NIC trying to handle over 256 subfunctions it will quickly reduce the receive/transmit throughput to gigabit or less speeds when encountering hardware multicast/broadcast replication. With 256 subfunctions a simple 60B ARP could consume more than 19KB of PCIe bandwidth due to the packet having to be duplicated so many times. In my mind it should be simpler to simply clone a single skb 256 times, forward that to the switchdev ports, and have them perform a bypass (if available) to deliver it to the subfunctions. That's why I was thinking it might be a good time to look at addressing it.
Quoted 3 lines
Linux tc framework is rich to address this and already used by openvswich for years now. Today arp broadcasts are not offloaded. They go through software path and replicated in the L2 domain. It is a solved problem for many years now.
When you say they are replicated in the L2 domain I assume you are talking about the software switch connected to the switchdev ports. My question is what are you doing with them after you have replicated them? I'm assuming they are being sent to the other switchdev ports which will require a DMA to transmit them, and another to receive them on the VF/SF, or are you saying something else is going on here?
My argument is that this cuts into both the transmit and receive DMA bandwidth of the NIC, and could easily be avoided in the case where SF exists in the same kernel as the switchdev port by identifying the multicast bit being set and simply bypassing the device.
Quoted 25 lines
From: Alexander Duyck <[email protected]>
Sent: Friday, December 18, 2020 9:31 PMQuoted 6 lines · Parav Pandit
Quoted 6 lines
From: Alexander Duyck <[email protected]>
Sent: Friday, December 18, 2020 8:41 AMOn Thu, Dec 17, 2020 at 5:30 PM David Ahern <[email protected]>
Quoted 6 lines · wrote:
Quoted 6 lines
Quoted 2 lines · Alexander Duyck
The problem is PCIe DMA wasn't designed to function as a network switch fabric and when we start talking about a 400Gb NIC trying to handle over 256 subfunctions it will quickly reduce the receive/transmit throughput to gigabit or less speeds when encountering
hardware multicast/broadcast replication.
Quoted 6 lines
Quoted 6 lines
With 256 subfunctions a simple 60B ARP could consume more than 19KB of PCIe bandwidth due to the packet having to be duplicated so many times. In my mind it should be simpler to simply clone a single skb 256 times, forward that to the switchdev ports, and have them perform a bypass (if available) to deliver it to the subfunctions. That's why I was thinking it might be a good time to look at addressing it.
Quoted 8 lines
Quoted 1 line
Linux tc framework is rich to address this and already used by openvswich
for years now.
Quoted 1 line
Today arp broadcasts are not offloaded. They go through software path
and replicated in the L2 domain.
Quoted 1 line
It is a solved problem for many years now.
When you say they are replicated in the L2 domain I assume you are talking about the software switch connected to the switchdev ports.
Yes.
Quoted 6 lines
My question is
what are you doing with them after you have replicated them? I'm assuming they are being sent to the other switchdev ports which will require a DMA to transmit them, and another to receive them on the VF/SF, or are you saying something else is going on here?
Yes, that is correct.
Quoted 4 lines
My argument is that this cuts into both the transmit and receive DMA bandwidth of the NIC, and could easily be avoided in the case where SF exists in the same kernel as the switchdev port by identifying the multicast bit being set and simply bypassing the device.
It probably can be avoided but its probably not worth for occasional ARP packets on neighbor cache miss.
If I am not mistaken, even some recent HW can forward such ARP packets to multiple switchcev ports with commit 7ee3f6d2486e without following the above described DMA path.
Quoted 42 lines · Parav Pandit
Quoted 26 lines
From: Alexander Duyck <[email protected]>
Sent: Friday, December 18, 2020 9:31 PMQuoted 6 lines · Parav Pandit
Quoted 6 lines
From: Alexander Duyck <[email protected]>
Sent: Friday, December 18, 2020 8:41 AMOn Thu, Dec 17, 2020 at 5:30 PM David Ahern <[email protected]>
Quoted 6 lines · wrote:
Quoted 6 lines
Quoted 2 lines · Alexander Duyck
The problem is PCIe DMA wasn't designed to function as a network switch fabric and when we start talking about a 400Gb NIC trying to handle over 256 subfunctions it will quickly reduce the receive/transmit throughput to gigabit or less speeds when encountering
hardware multicast/broadcast replication.
Quoted 6 lines
Quoted 6 lines
With 256 subfunctions a simple 60B ARP could consume more than 19KB of PCIe bandwidth due to the packet having to be duplicated so many times. In my mind it should be simpler to simply clone a single skb 256 times, forward that to the switchdev ports, and have them perform a bypass (if available) to deliver it to the subfunctions. That's why I was thinking it might be a good time to look at addressing it.
Quoted 8 lines
Quoted 1 line
Linux tc framework is rich to address this and already used by openvswich
for years now.
Quoted 1 line
Today arp broadcasts are not offloaded. They go through software path
and replicated in the L2 domain.
Quoted 1 line
It is a solved problem for many years now.
When you say they are replicated in the L2 domain I assume you are talking about the software switch connected to the switchdev ports.
Yes.
Quoted 6 lines
My question is
what are you doing with them after you have replicated them? I'm assuming they are being sent to the other switchdev ports which will require a DMA to transmit them, and another to receive them on the VF/SF, or are you saying something else is going on here?
Quoted 8 lines
Yes, that is correct.
Quoted 4 lines
My argument is that this cuts into both the transmit and receive DMA bandwidth of the NIC, and could easily be avoided in the case where SF exists in the same kernel as the switchdev port by identifying the multicast bit being set and simply bypassing the device.
It probably can be avoided but its probably not worth for occasional ARP packets on neighbor cache miss.
If I am not mistaken, even some recent HW can forward such ARP packets to multiple switchdev ports with commit7ee3f6d2486ewithout following the above described DMA path.
Even with that it sounds like it will have to DMA the packet to multiple Rx destinations even if it is only performing the Tx DMA once. The Intel NICs did all this replication in hardware as well so that is what I was thinking of when I was talking about the replication behavior seen with SR-IOV.
Basically what I am getting at is that this could be used as an architectural feature for switchdev to avoid creating increased DMA overhead for broadcast, multicast and unknown-unicast traffic. I'm not saying this is anything mandatory, and I would be perfectly okay with something like this being optional and defaulted to off. In my mind the setup only has the interfaces handling traffic to single point destinations so that at most you are only looking at a 2x bump in PCIe bandwidth for those cases where the packet ends up needing to go out the physical port. It would essentially be a software offload to avoid saturating the PCIe bus.
This setup would only work if both interfaces are present in the same kernel though so that is why I chose now to bring it up as historically SR-IOV hasn't normally been associated with containers due to the limited number of interfaces that could be created.
Also as far as the patch count complaints I have seen in a few threads I would be fine with splitting things up so that the devlink and aux device creation get handled in one set, and then we work out the details of mlx5 attaching to the devices and spawning of the SF netdevs in another since that seems to be where the debate is.
Quoted 5 lines · Alexander Duyck
Also as far as the patch count complaints I have seen in a few threads I would be fine with splitting things up so that the devlink and aux device creation get handled in one set, and then we work out the details of mlx5 attaching to the devices and spawning of the SF netdevs in another since that seems to be where the debate is.
It doesn't work like that. The aux device creates a mlx5_core and every mlx5_core can run mlx5_en.
This really isn't the series to raise this feature request. Adding an optional short cut path to VF/SF is something that can be done later if up to date benchmarks show it has value. There is no blocker in this model to doing that.
Jason
Quoted 11 lines · Jason Gunthorpe
Quoted 5 lines · Alexander Duyck
Also as far as the patch count complaints I have seen in a few threads I would be fine with splitting things up so that the devlink and aux device creation get handled in one set, and then we work out the details of mlx5 attaching to the devices and spawning of the SF netdevs in another since that seems to be where the debate is.
It doesn't work like that. The aux device creates a
mlx5_coreand everymlx5_corecan runmlx5_en.
The aux device is still just a device isn't it? Until you register a driver that latches on to "MLX5_SF_DEV_ID_NAME" the device is there and it should function like an unbound/idle VF.
Quoted 4 lines
This really isn't the series to raise this feature request. Adding an optional short cut path to VF/SF is something that can be done later if up to date benchmarks show it has value. There is no blocker in this model to doing that.
That is the point of contention that we probably won't solve. I feel like if we are going to put out something that is an alternative to the macvlan and SR-IOV approaches it should address the issues resolved in both. Right now this just throws up yet another virtualization solution that ends up reintroducing many of the existing problems we already had with SR-IOV, and possibly exacerbating them by allowing for an even greater number of subfunctions.
Anyway I have made my opinion known, and I am not in the position to block the patches since I am not the maintainer.