Add mlx5 subfunction support
Get series with b4
b4 am [email protected]Versions newest first
- v1 Latest13 patches 43 repliesIn review
Patches in v1 No applies recorded yet
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/13 | devlink: Prepare code to fill multiple port function attributesnet/ | +32 | 0 replies | |
| 2/13 | devlink: Introduce PCI SF port flavour and port attributeinclude/ include/ net/ | +68 | 0 replies | |
| 3/13 | devlink: Support add and delete devlink portinclude/ net/ | +105 | 11 replies | |
| 4/13 | devlink: Support get and set state of port functioninclude/ include/ net/ | +130 | 0 replies | |
| 5/13 | devlink: Avoid global devlink mutex, use per instance reload lockinclude/ net/ | +16 | 0 replies | |
| 6/13 | devlink: Introduce devlink refcount to reduce scope of global devlink_mutexinclude/ net/ | +67 | 0 replies | |
| 7/13 | net/mlx5: SF, Add auxiliary device supportdrivers/ drivers/ drivers/ +3 more | +297 | 2 replies | |
| 8/13 | net/mlx5: SF, Add auxiliary device driverdrivers/ drivers/ drivers/ +5 more | +168 | 0 replies | |
| 9/13 | net/mlx5: E-switch, Prepare eswitch to handle SF vportdrivers/ drivers/ drivers/ +4 more | +123 | 0 replies | |
| 10/13 | net/mlx5: E-switch, Add eswitch helpers for SF vportdrivers/ drivers/ drivers/ +1 more | +100 | 0 replies | |
| 11/13 | net/mlx5: SF, Add SF configuration hardware commandsdrivers/ drivers/ drivers/ +2 more | +76 | 0 replies | |
| 12/13 | net/mlx5: SF, Add port add delete functionalitydrivers/ drivers/ drivers/ +5 more | +443 | 0 replies | |
| 13/13 | net/mlx5: SF, Port function state change supportdrivers/ drivers/ drivers/ | +137 | 0 replies |
Reviews trailers per patch
12 of 13 patches have a Reviewed-by or Acked-by.
| Patch | Vu Pham | Jiri Pirko | Roi Dayan |
|---|---|---|---|
| 1/13 | R-b | ||
| 2/13 | R-b | R-b | |
| 3/13 | R-b | R-b | |
| 4/13 | R-b | R-b | |
| 5/13 | R-b | ||
| 6/13 | R-b | ||
| 7/13 | R-b | ||
| 8/13 | R-b | ||
| 9/13 | R-b | ||
| 10/13 | R-b | ||
| 11/13 | |||
| 12/13 | R-b | ||
| 13/13 | R-b |
Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 3 reviewers.
Cover letter [PATCH net-next 00/13] Add mlx5 subfunction support
Hi Dave, Jakub, Greg,
This series introduces support for mlx5 subfunction (SF). A subfunction is a portion of a PCI device that supports multiple classes of devices such as netdev, RDMA and more.
This patchset is based on Leon's series [3].
It is a third user of proposed auxiliary bus [4].
Subfunction support is discussed in detail in RFC [1] and [2]. RFC [1] and extension [2] describes requirements, design, and proposed plumbing using devlink, auxiliary bus and sysfs for systemd/udev support.
Patch summary:
--------------
Patch 1 to 6 prepares devlink:
Patch-1 prepares code to handle multiple port function attributes Patch-2 introduces devlink pcisf port flavour similar to pcipf and pcivf Patch-3 adds port add and delete driver callbacks
Patch-4 adds port function state get and set callbacks Patch-5 refactors devlink to avoid using global mutext Patch-6 uses refcount to allow creating devlink instance from existing one
Patch 7 to 13 implements mlx5 pieces for SF support.
Patch-7 adds SF auxiliary device
Patch-8 adds SF auxiliary driver
Patch-9 prepares eswitch to handler SF vport
PAtch-10 adds eswitch helpers to add/remove SF vport Patch-11 adds SF device configuration commands
Patch-12 implements devlink port add/del callbacks
Patch-13 implements devlink port function get/set callbacks
More on SF plumbing below.
overview:
--------
A subfunction can be created and deleted by a user using devlink port add/delete interface.
A subfunction can be configured using devlink port function attributes before its activated.
When a subfunction is activated, it results in an auxiliary device. A driver binds to the auxiliary device that further creates supported class devices.
short example sequence:
-----------------------
Change device to switchdev mode:
$ devlink dev eswitch set pci/0000:06:00.0 mode switchdev
Add a devlink port of subfunction flaovur:
$ devlink port add pci/0000:06:00.0 flavour pcisf pfnum 0 sfnum 88
Configure mac address of the port function:
$ devlink port function set ens2f0npf0sf88 hw_addr 00:00:00:00:88:88
Now activate the function:
$ devlink port function set ens2f0npf0sf88 state active
Now use the auxiliary device and class devices:
$ devlink dev show pci/0000:06:00.0 auxiliary/mlx5_core.sf.4
$ ip link show
127: ens2f0np0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 24:8a:07:b3:d1:12 brd ff:ff:ff:ff:ff:ff
altname enp6s0f0np0
129: p0sf88: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 00:00:00:00:88:88 brd ff:ff:ff:ff:ff:ff$ rdma dev show 43: rdmap6s0f0: node_type ca fw 16.28.1002 node_guid 248a:0703:00b3:d112 sys_image_guid 248a:0703:00b3:d112 44: mlx5_0: node_type ca fw 16.28.1002 node_guid 0000:00ff:fe00:8888 sys_image_guid 248a:0703:00b3:d112
subfunction (SF) in detail:
---------------------------
- A sub-function is a portion of the PCI device which supports multiple classes of devices such as netdev, RDMA and more.
- A SF netdev has its own dedicated queues(txq, rxq).
- A SF RDMA device has its own QP1, GID table and other RDMA resources.
- A SF supports eswitch representation and tc offload support similar to existing PF and VF representors.
- User must configure eswitch to send/receive SF's packets.
- A SF shares PCI level resources with other SFs and/or with its parent PCI function.
For example, an SF shares IRQ vectors with other SFs and its PCI function.
In future it may have dedicated IRQ vector per SF.
A SF has dedicated window in PCI BAR space that is not shared with other SFs or PF. This ensures that when a SF is assigned to an application, only that application can access device resources.
- SF's auxiliary device exposes sfnum sysfs attribute. This will be used by systemd/udev to deterministic names for its netdev and RDMA device.
[1] https://lore.kernel.org/netdev/20200519092258.GF4655@nanopsycho/
[2] https://marc.info/?l=linux-netdev&m=158555928517777&w=2
[3] https://lists.linuxfoundation.org/pipermail/virtualization/2020-November/050473.html
[4] https://lore.kernel.org/linux-rdma/[email protected]/
Parav Pandit (11):
devlink: Prepare code to fill multiple port function attributes
devlink: Introduce PCI SF port flavour and port attribute
devlink: Support add and delete devlink port
devlink: Support get and set state of port function
devlink: Avoid global devlink mutex, use per instance reload lock
devlink: Introduce devlink refcount to reduce scope of global
devlink_mutex
net/mlx5: SF, Add auxiliary device support
net/mlx5: SF, Add auxiliary device driver
net/mlx5: E-switch, Add eswitch helpers for SF vport net/mlx5: SF, Add port add delete functionality net/mlx5: SF, Port function state change support
Vu Pham (2):
net/mlx5: E-switch, Prepare eswitch to handle SF vport net/mlx5: SF, Add SF configuration hardware commands
.../net/ethernet/mellanox/mlx5/core/Kconfig | 19 + .../net/ethernet/mellanox/mlx5/core/Makefile | 9 + drivers/net/ethernet/mellanox/mlx5/core/cmd.c | 4 + .../net/ethernet/mellanox/mlx5/core/devlink.c | 7 + drivers/net/ethernet/mellanox/mlx5/core/eq.c | 2 +- .../mellanox/mlx5/core/esw/acl/egress_ofld.c | 2 +- .../mellanox/mlx5/core/esw/devlink_port.c | 41 ++ .../net/ethernet/mellanox/mlx5/core/eswitch.c | 46 +- .../net/ethernet/mellanox/mlx5/core/eswitch.h | 82 +++ .../mellanox/mlx5/core/eswitch_offloads.c | 47 +- .../net/ethernet/mellanox/mlx5/core/main.c | 48 +- .../ethernet/mellanox/mlx5/core/mlx5_core.h | 10 + .../net/ethernet/mellanox/mlx5/core/pci_irq.c | 20 + .../net/ethernet/mellanox/mlx5/core/sf/cmd.c | 48 ++ .../ethernet/mellanox/mlx5/core/sf/dev/dev.c | 213 ++++++++ .../ethernet/mellanox/mlx5/core/sf/dev/dev.h | 68 +++ .../mellanox/mlx5/core/sf/dev/driver.c | 105 ++++ .../net/ethernet/mellanox/mlx5/core/sf/priv.h | 14 + .../net/ethernet/mellanox/mlx5/core/sf/sf.c | 498 ++++++++++++++++++ .../net/ethernet/mellanox/mlx5/core/sf/sf.h | 59 +++ .../net/ethernet/mellanox/mlx5/core/vport.c | 3 +- include/linux/mlx5/driver.h | 12 +- include/net/devlink.h | 82 +++ include/uapi/linux/devlink.h | 26 + net/core/devlink.c | 362 +++++++++++-- 25 files changed, 1754 insertions(+), 73 deletions(-) create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/sf/cmd.c create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/sf/dev/dev.c create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/sf/dev/dev.h create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/sf/dev/driver.c create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/sf/priv.h create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/sf/sf.c create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/sf/sf.h
Signature · 1 line
2.26.2