Add mlx5 subfunction support
Subsystems (6)
cover Add mlx5 subfunction support
Hi Dave, Jakub, Greg,
This series introduces support for mlx5 subfunction (SF). A subfunction is a portion of a PCI device that supports multiple classes of devices such as netdev, RDMA and more.
This patchset is based on Leon's series [3].
It is a third user of proposed auxiliary bus [4].
Subfunction support is discussed in detail in RFC [1] and [2]. RFC [1] and extension [2] describes requirements, design, and proposed plumbing using devlink, auxiliary bus and sysfs for systemd/udev support.
Patch summary:
--------------
Patch 1 to 6 prepares devlink:
Patch-1 prepares code to handle multiple port function attributes Patch-2 introduces devlink pcisf port flavour similar to pcipf and pcivf Patch-3 adds port add and delete driver callbacks
Patch-4 adds port function state get and set callbacks Patch-5 refactors devlink to avoid using global mutext Patch-6 uses refcount to allow creating devlink instance from existing one
Patch 7 to 13 implements mlx5 pieces for SF support.
Patch-7 adds SF auxiliary device
Patch-8 adds SF auxiliary driver
Patch-9 prepares eswitch to handler SF vport
PAtch-10 adds eswitch helpers to add/remove SF vport Patch-11 adds SF device configuration commands
Patch-12 implements devlink port add/del callbacks
Patch-13 implements devlink port function get/set callbacks
More on SF plumbing below.
overview:
--------
A subfunction can be created and deleted by a user using devlink port add/delete interface.
A subfunction can be configured using devlink port function attributes before its activated.
When a subfunction is activated, it results in an auxiliary device. A driver binds to the auxiliary device that further creates supported class devices.
short example sequence:
-----------------------
Change device to switchdev mode:
$ devlink dev eswitch set pci/0000:06:00.0 mode switchdev
Add a devlink port of subfunction flaovur:
$ devlink port add pci/0000:06:00.0 flavour pcisf pfnum 0 sfnum 88
Configure mac address of the port function:
$ devlink port function set ens2f0npf0sf88 hw_addr 00:00:00:00:88:88
Now activate the function:
$ devlink port function set ens2f0npf0sf88 state active
Now use the auxiliary device and class devices:
$ devlink dev show pci/0000:06:00.0 auxiliary/mlx5_core.sf.4
$ ip link show
127: ens2f0np0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 24:8a:07:b3:d1:12 brd ff:ff:ff:ff:ff:ff
altname enp6s0f0np0
129: p0sf88: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 00:00:00:00:88:88 brd ff:ff:ff:ff:ff:ff$ rdma dev show 43: rdmap6s0f0: node_type ca fw 16.28.1002 node_guid 248a:0703:00b3:d112 sys_image_guid 248a:0703:00b3:d112 44: mlx5_0: node_type ca fw 16.28.1002 node_guid 0000:00ff:fe00:8888 sys_image_guid 248a:0703:00b3:d112
subfunction (SF) in detail:
---------------------------
- A sub-function is a portion of the PCI device which supports multiple classes of devices such as netdev, RDMA and more.
- A SF netdev has its own dedicated queues(txq, rxq).
- A SF RDMA device has its own QP1, GID table and other RDMA resources.
- A SF supports eswitch representation and tc offload support similar to existing PF and VF representors.
- User must configure eswitch to send/receive SF's packets.
- A SF shares PCI level resources with other SFs and/or with its parent PCI function.
For example, an SF shares IRQ vectors with other SFs and its PCI function.
In future it may have dedicated IRQ vector per SF.
A SF has dedicated window in PCI BAR space that is not shared with other SFs or PF. This ensures that when a SF is assigned to an application, only that application can access device resources.
- SF's auxiliary device exposes sfnum sysfs attribute. This will be used by systemd/udev to deterministic names for its netdev and RDMA device.
[1] https://lore.kernel.org/netdev/20200519092258.GF4655@nanopsycho/
[2] https://marc.info/?l=linux-netdev&m=158555928517777&w=2
[3] https://lists.linuxfoundation.org/pipermail/virtualization/2020-November/050473.html
[4] https://lore.kernel.org/linux-rdma/[email protected]/
Parav Pandit (11):
devlink: Prepare code to fill multiple port function attributes
devlink: Introduce PCI SF port flavour and port attribute
devlink: Support add and delete devlink port
devlink: Support get and set state of port function
devlink: Avoid global devlink mutex, use per instance reload lock
devlink: Introduce devlink refcount to reduce scope of global
devlink_mutex
net/mlx5: SF, Add auxiliary device support
net/mlx5: SF, Add auxiliary device driver
net/mlx5: E-switch, Add eswitch helpers for SF vport net/mlx5: SF, Add port add delete functionality net/mlx5: SF, Port function state change support
Vu Pham (2):
net/mlx5: E-switch, Prepare eswitch to handle SF vport net/mlx5: SF, Add SF configuration hardware commands
.../net/ethernet/mellanox/mlx5/core/Kconfig | 19 + .../net/ethernet/mellanox/mlx5/core/Makefile | 9 + drivers/net/ethernet/mellanox/mlx5/core/cmd.c | 4 + .../net/ethernet/mellanox/mlx5/core/devlink.c | 7 + drivers/net/ethernet/mellanox/mlx5/core/eq.c | 2 +- .../mellanox/mlx5/core/esw/acl/egress_ofld.c | 2 +- .../mellanox/mlx5/core/esw/devlink_port.c | 41 ++ .../net/ethernet/mellanox/mlx5/core/eswitch.c | 46 +- .../net/ethernet/mellanox/mlx5/core/eswitch.h | 82 +++ .../mellanox/mlx5/core/eswitch_offloads.c | 47 +- .../net/ethernet/mellanox/mlx5/core/main.c | 48 +- .../ethernet/mellanox/mlx5/core/mlx5_core.h | 10 + .../net/ethernet/mellanox/mlx5/core/pci_irq.c | 20 + .../net/ethernet/mellanox/mlx5/core/sf/cmd.c | 48 ++ .../ethernet/mellanox/mlx5/core/sf/dev/dev.c | 213 ++++++++ .../ethernet/mellanox/mlx5/core/sf/dev/dev.h | 68 +++ .../mellanox/mlx5/core/sf/dev/driver.c | 105 ++++ .../net/ethernet/mellanox/mlx5/core/sf/priv.h | 14 + .../net/ethernet/mellanox/mlx5/core/sf/sf.c | 498 ++++++++++++++++++ .../net/ethernet/mellanox/mlx5/core/sf/sf.h | 59 +++ .../net/ethernet/mellanox/mlx5/core/vport.c | 3 +- include/linux/mlx5/driver.h | 12 +- include/net/devlink.h | 82 +++ include/uapi/linux/devlink.h | 26 + net/core/devlink.c | 362 +++++++++++-- 25 files changed, 1754 insertions(+), 73 deletions(-) create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/sf/cmd.c create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/sf/dev/dev.c create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/sf/dev/dev.h create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/sf/dev/driver.c create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/sf/priv.h create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/sf/sf.c create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/sf/sf.h
Signature · 1 line
2.26.2
PATCH 1/13 devlink: Prepare code to fill multiple port function attributes
Prepare code to fill zero or more port function optional attributes. Subsequent patch makes use of this to fill more port function attributes.
--- net/core/devlink.c | 63 +++++++++++++++++++++++----------------------- 1 file changed, 32 insertions(+), 31 deletions(-)
net/core/devlink.c +32 -31
@@ -695,6 +695,31 @@ static int devlink_nl_port_attrs_put(struct sk_buff *msg,return 0 ; } +static int +devlink_port_function_hw_addr_fill (struct devlink *devlink,const struct devlink_ops *ops, +struct devlink_port *port,struct sk_buff *msg, +struct netlink_ext_ack *extack,bool *msg_updated) +{ +u8 hw_addr[MAX_ADDR_LEN ]; +int hw_addr_len; +int err; + +if (!ops->port_function_hw_addr_get) +return 0 ; + + err = ops->port_function_hw_addr_get (devlink, port, hw_addr, &hw_addr_len, extack); +if (err) { +if (err == -EOPNOTSUPP ) +return 0 ; +return err; + } + err =nla_put (msg,DEVLINK_PORT_FUNCTION_ATTR_HW_ADDR , hw_addr_len, hw_addr); +if (err) +return err; + *msg_updated = true; +return 0 ; +} +static int devlink_nl_port_function_attrs_put (struct sk_buff *msg,struct devlink_port *port,struct netlink_ext_ack *extack)
@@ -702,36 +727,16 @@ devlink_nl_port_function_attrs_put(struct sk_buff *msg, struct devlink_port *porstruct devlink *devlink = port->devlink;const struct devlink_ops *ops;struct nlattr *function_attr; -bool empty_nest = true; -int err =0 ; +bool msg_updated = false; +int err; function_attr =nla_nest_start_noflag (msg,DEVLINK_ATTR_PORT_FUNCTION );if (!function_attr)return -EMSGSIZE ; ops = devlink->ops; -if (ops->port_function_hw_addr_get) { -int hw_addr_len; -u8 hw_addr[MAX_ADDR_LEN ]; - - err = ops->port_function_hw_addr_get (devlink, port, hw_addr, &hw_addr_len, extack); -if (err == -EOPNOTSUPP ) { -/* Port function attributes are optional for a port. If port doesn't -* support function attribute, returning -EOPNOTSUPP is not an error. -*/ - err =0 ; - goto out; - }else if (err) { - goto out; - } - err =nla_put (msg,DEVLINK_PORT_FUNCTION_ATTR_HW_ADDR , hw_addr_len, hw_addr); -if (err) - goto out; - empty_nest = false; - } - -out: -if (err || empty_nest) + err =devlink_port_function_hw_addr_fill (devlink, ops, port, msg, extack, &msg_updated); +if (err || !msg_updated)nla_nest_cancel (msg, function_attr);else nla_nest_end (msg, function_attr);
@@ -964,7 +969,6 @@ devlink_port_function_hw_addr_set(struct devlink *devlink, struct devlink_port *const struct devlink_ops *ops;const u8 *hw_addr;int hw_addr_len; -int err; hw_addr =nla_data (attr); hw_addr_len =nla_len (attr);
@@ -989,12 +993,7 @@ devlink_port_function_hw_addr_set(struct devlink *devlink, struct devlink_port *return -EOPNOTSUPP ; } - err = ops->port_function_hw_addr_set (devlink, port, hw_addr, hw_addr_len, extack); -if (err) -return err; - -devlink_port_notify (port,DEVLINK_CMD_PORT_NEW ); -return 0 ; +return ops->port_function_hw_addr_set (devlink, port, hw_addr, hw_addr_len, extack); } static int
@@ -1015,6 +1014,8 @@ devlink_port_function_set(struct devlink *devlink, struct devlink_port *port,if (attr) err =devlink_port_function_hw_addr_set (devlink, port, attr, extack); +if (!err) +devlink_port_notify (port,DEVLINK_CMD_PORT_NEW );return err; }
Signature · 1 line
2.26.2
PATCH 2/13 devlink: Introduce PCI SF port flavour and port attribute
In reply to Parav PanditA PCI sub-function (SF) represents a portion of the device similar to PCI VF.
In an eswitch, PCI SF may have port which is normally represented using a representor netdevice.
To have better visibility of eswitch port, its association with SF, and its representor netdevice, introduce a PCI SF port flavour.
When devlink port flavour is PCI SF, fill up PCI SF attributes of the port.
Extend port name creation using PCI PF and SF number scheme on best effort basis, so that vendor drivers can skip defining their own scheme.
This is done as cApfNSfM, where A, N and M are controller, PCI PF and PCI SF number respectively.
This is similar to existing naming for PCI PF and PCI VF ports.
An example view of a PCI SF port:
$ devlink port show pci/0000:06:00.0/32768
pci/0000:06:00.0/32768: type eth netdev eth0 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false
function:
hw_addr 00:00:00:00:88:88 state active opstate attached$ devlink port show pci/0000:06:00.0/32768 -jp
{
"port": {
"pci/0000:06:00.0/32768": {
"type": "eth",
"netdev": "eth0",
"flavour": "pcisf",
"controller": 0,
"pfnum": 0,
"sfnum": 88,
"external": false,
"splittable": false,
"function": {
"hw_addr": "00:00:00:00:88:88",
"state": "active",
"opstate": "attached"
}
}
}
}--- include/net/devlink.h | 17 +++++++++++++ include/uapi/linux/devlink.h | 5 ++++ net/core/devlink.c | 46 ++++++++++++++++++++++++++++++++++++ 3 files changed, 68 insertions(+)
include/net/devlink.h +17 -0
@@ -92,6 +92,20 @@ struct devlink_port_pci_vf_attrs { u8 external:1; }; +/** +* struct devlink_port_pci_sf_attrs - devlink port's PCI SF attributes +* @controller: Associated controller number +* @pf: Associated PCI PF number for this port. +* @sf: Associated PCI SF for of the PCI PF for this port. +* @external: when set, indicates if a port is for an external controller +*/ +struct devlink_port_pci_sf_attrs { +u32 controller; +u16 pf; +u32 sf; +u8 external:1 ; +}; +/** * struct devlink_port_attrs - devlink port object * @flavour: flavour of the port
@@ -113,6 +127,7 @@ struct devlink_port_attrs {struct devlink_port_phys_attrs phys;struct devlink_port_pci_pf_attrs pci_pf;struct devlink_port_pci_vf_attrs pci_vf; +struct devlink_port_pci_sf_attrs pci_sf; }; };
@@ -1401,6 +1416,8 @@ void devlink_port_attrs_pci_pf_set(struct devlink_port *devlink_port, u32 contro u16 pf, bool external);void devlink_port_attrs_pci_vf_set (struct devlink_port *devlink_port,u32 controller,u16 pf,u16 vf,bool external); +void devlink_port_attrs_pci_sf_set (struct devlink_port *devlink_port,u32 controller, +u16 pf,u32 sf,bool external); int devlink_sb_register(struct devlink *devlink, unsigned int sb_index, u32 size, u16 ingress_pools_count, u16 egress_pools_count, u16 ingress_tc_count,
include/uapi/linux/devlink.h +5 -0
@@ -200,6 +200,10 @@ enum devlink_port_flavour {DEVLINK_PORT_FLAVOUR_UNUSED ,/* Port which exists in the switch, but * is not used in any way. */ + DEVLINK_PORT_FLAVOUR_PCI_SF, /* Represents eswitch port + * for the PCI SF. It is an internal + * port that faces the PCI SF. + */ }; enum devlink_param_cmode {
@@ -527,6 +531,7 @@ enum devlink_attr {DEVLINK_ATTR_RELOAD_STATS_VALUE ,/* u32 */ DEVLINK_ATTR_REMOTE_RELOAD_STATS ,/* nested */ +DEVLINK_ATTR_PORT_PCI_SF_NUMBER ,/* u32 */ /* add new attributes above here, update the policy in devlink.c */ __DEVLINK_ATTR_MAX,
net/core/devlink.c +46 -0
64 lines not on this page · raw message
Signature · 1 line
2.26.2
PATCH 3/13 devlink: Support add and delete devlink port
In reply to Parav PanditExtended devlink interface for the user to add and delete port. Extend devlink to connect user requests to driver to add/delete such port in the device.
When driver routines are invoked, devlink instance lock is not held. This enables driver to perform several devlink objects registration, unregistration such as (port, health reporter, resource etc) by using exising devlink APIs.
This also helps to uniformly use the code for port unregistration during driver unload and during port deletion initiated by user.
Examples of add, show and delete commands:
$ devlink dev eswitch set pci/0000:06:00.0 mode switchdev
$ devlink port show pci/0000:06:00.0/65535: type eth netdev ens2f0np0 flavour physical port 0 splittable false
$ devlink port add pci/0000:06:00.0 flavour pcisf pfnum 0 sfnum 88
$ devlink port show pci/0000:06:00.0/32768
pci/0000:06:00.0/32768: type eth netdev eth0 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false
function:
hw_addr 00:00:00:00:88:88 state inactive opstate detached$ udevadm test-builtin net_id /sys/class/net/eth0 Load module index Parsed configuration file /usr/lib/systemd/network/99-default.link Created link configuration context. Using default interface naming scheme 'v245'. ID_NET_NAMING_SCHEME=v245 ID_NET_NAME_PATH=enp6s0f0npf0sf88 ID_NET_NAME_SLOT=ens2f0npf0sf88 Unload module index Unloaded link configuration context.
$ devlink port del netdevsim/netdevsim10/32768
--- include/net/devlink.h | 38 ++++++++++++++++++++++++ net/core/devlink.c | 67 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 105 insertions(+)
include/net/devlink.h +38 -0
@@ -152,6 +152,17 @@ struct devlink_port {struct mutex reporters_lock;/* Protects reporter_list */ }; +struct devlink_port_new_attrs { +enum devlink_port_flavour flavour; +unsigned int port_index; +u32 controller; +u32 sfnum; +u16 pfnum; +u8 port_index_valid:1 , + controller_valid:1 , + sfnum_valid:1 ; +}; +struct devlink_sb_pool_info {enum devlink_sb_pool_type pool_type;u32 size;
@@ -1360,6 +1371,33 @@ struct devlink_ops {int (*port_function_hw_addr_set)(struct devlink *devlink,struct devlink_port *port,const u8 *hw_addr,int hw_addr_len,struct netlink_ext_ack *extack); +/** +* @port_new: Port add function. +* +* Should be used by device driver to let caller add new port of a specified flavour +* with optional attributes. +* Driver should return -EOPNOTSUPP if it doesn't support port addition of a specified +* flavour or specified attributes. Driver should set extack error message in case of fail +* to add the port. +* devlink core does not hold a devlink instance lock when this callback is invoked. +* Driver must ensures synchronization when adding or deleting a port. Driver must +* register a port with devlink core. +*/ +int (*port_new)(struct devlink *devlink,const struct devlink_port_new_attrs *attrs, +struct netlink_ext_ack *extack); +/** +* @port_del: Port delete function. +* +* Should be used by device driver to let caller delete port which was previously created +* using port_new() callback. +* Driver should return -EOPNOTSUPP if it doesn't support port deletion. +* Driver should set extack error message in case of fail to delete the port. +* devlink core does not hold a devlink instance lock when this callback is invoked. +* Driver must ensures synchronization when adding or deleting a port. Driver must +* register a port with devlink core. +*/ +int (*port_del)(struct devlink *devlink,unsigned int port_index, +struct netlink_ext_ack *extack); };static inline void *devlink_priv (struct devlink *devlink)
net/core/devlink.c +67 -0
85 lines not on this page · raw message
Signature · 1 line
2.26.2
Re: [PATCH net-next 03/13] devlink: Support add and delete devlink port
Quoted 17 lines · Parav Pandit
Extended devlink interface for the user to add and delete port. Extend devlink to connect user requests to driver to add/delete such port in the device.
When driver routines are invoked, devlink instance lock is not held. This enables driver to perform several devlink objects registration, unregistration such as (port, health reporter, resource etc) by using exising devlink APIs.
This also helps to uniformly use the code for port unregistration during driver unload and during port deletion initiated by user.Examples of add, show and delete commands:
$ devlink dev eswitch set pci/0000:06:00.0 mode switchdev$ devlink port show pci/0000:06:00.0/65535: type eth netdev ens2f0np0 flavour physical port 0 splittable false
Quoted 7 lines
$ devlink port add pci/0000:06:00.0 flavour pcisf pfnum 0 sfnum 88$ devlink port show pci/0000:06:00.0/32768 pci/0000:06:00.0/32768: type eth netdev eth0 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false function: hw_addr 00:00:00:00:88:88 state inactive opstate detached
There has to be limits on the number of sub functions that can be created for a device. How does a user find that limit?
Also, seems like there are hardware constraint at play. e.g., can a user reduce the number of queues used by the physical function to support more sub-functions? If so how does a user programmatically learn about this limitation? e.g., devlink could have support to show resource sizing and configure constraints similar to what mlxsw has.
RE: [PATCH net-next 03/13] devlink: Support add and delete devlink port
Quoted 27 lines
From: David Ahern <[email protected]>
Sent: Wednesday, November 18, 2020 9:51 PMQuoted 14 lines · Parav Pandit
Extended devlink interface for the user to add and delete port. Extend devlink to connect user requests to driver to add/delete such port in the device.
When driver routines are invoked, devlink instance lock is not held. This enables driver to perform several devlink objects registration, unregistration such as (port, health reporter, resource etc) by using exising devlink APIs.
This also helps to uniformly use the code for port unregistration during driver unload and during port deletion initiated by user.Examples of add, show and delete commands:
$ devlink dev eswitch set pci/0000:06:00.0 mode switchdevQuoted 8 lines
$ devlink port show pci/0000:06:00.0/65535: type eth netdev ens2f0np0 flavour physical port 0 splittable false$ devlink port add pci/0000:06:00.0 flavour pcisf pfnum 0 sfnum 88$ devlink port show pci/0000:06:00.0/32768 pci/0000:06:00.0/32768: type eth netdev eth0 flavour pcisf controller 0
Quoted 7 lines
pfnum 0 sfnum 88 external false splittable false
Quoted 3 lines
function:
hw_addr 00:00:00:00:88:88 state inactive opstate detachedThere has to be limits on the number of sub functions that can be created for a device. How does a user find that limit?
Yes, this came up internally, but didn't really converged.
The devlink resource looked too verbose for an average or simple use cases. But it may be fine.
The hurdle I faced with devlink resource is with defining the granularity.
For example one devlink instance deploys sub functions on multiple pci functions. So how to name them? Currently we have controller and PFs in port annotation. So resource name asc0pf0_subfunctions -> for controller 0, pf 0c1pf2_subfunctions -> for controller 1, pf 2
Couldn't convince my self to name it this way.
Below example looked simpler to use but plumbing doesn’t exist for it.
$ devlink resource show pci/0000:03:00.0 pci/0000:03:00.0/1: name max_sfs count 256 controller 0 pf 0 pci/0000:03:00.0/2: name max_sfs count 100 controller 1 pf 0 pci/0000:03:00.0/3: name max_sfs count 64 controller 1 pf 1
$ devlink resource set pci/0000:03:00.0/1 max_sfs 100
Second option I was considering was use port params which doesn't sound so right as resource.
Quoted 6 lines
Also, seems like there are hardware constraint at play. e.g., can a user reduce the number of queues used by the physical function to support more sub- functions? If so how does a user programmatically learn about this limitation? e.g., devlink could have support to show resource sizing and configure constraints similar to what mlxsw has.
Yes, need to figure out its naming. For mlx5 num queues doesn't have relation to subfunctions.
But PCI resource has relation and this is something we want to do in future, as you said may be using devlink resource.
Re: [PATCH net-next 03/13] devlink: Support add and delete devlink port
Quoted 59 lines · Parav Pandit
Quoted 27 lines
From: David Ahern <[email protected]>
Sent: Wednesday, November 18, 2020 9:51 PMQuoted 14 lines · Parav Pandit
Extended devlink interface for the user to add and delete port. Extend devlink to connect user requests to driver to add/delete such port in the device.
When driver routines are invoked, devlink instance lock is not held. This enables driver to perform several devlink objects registration, unregistration such as (port, health reporter, resource etc) by using exising devlink APIs.
This also helps to uniformly use the code for port unregistration during driver unload and during port deletion initiated by user.Examples of add, show and delete commands:
$ devlink dev eswitch set pci/0000:06:00.0 mode switchdevQuoted 8 lines
$ devlink port show pci/0000:06:00.0/65535: type eth netdev ens2f0np0 flavour physical port 0 splittable false$ devlink port add pci/0000:06:00.0 flavour pcisf pfnum 0 sfnum 88$ devlink port show pci/0000:06:00.0/32768 pci/0000:06:00.0/32768: type eth netdev eth0 flavour pcisf controller 0Quoted 7 lines
pfnum 0 sfnum 88 external false splittable false
Quoted 3 lines
function:
hw_addr 00:00:00:00:88:88 state inactive opstate detachedThere has to be limits on the number of sub functions that can be created for a device. How does a user find that limit?
Yes, this came up internally, but didn't really converged.
The devlink resource looked too verbose for an average or simple use cases. But it may be fine.
The hurdle I faced with devlink resource is with defining the granularity.For example one devlink instance deploys sub functions on multiple pci functions. So how to name them? Currently we have controller and PFs in port annotation. So resource name as
c0pf0_subfunctions-> for controller 0, pf 0c1pf2_subfunctions-> for controller 1, pf 2Couldn't convince my self to name it this way.
Below example looked simpler to use but plumbing doesn’t exist for it.
$ devlink resource show pci/0000:03:00.0 pci/0000:03:00.0/1: name max_sfs count 256 controller 0 pf 0 pci/0000:03:00.0/2: name max_sfs count 100 controller 1 pf 0 pci/0000:03:00.0/3: name max_sfs count 64 controller 1 pf 1$ devlink resource set pci/0000:03:00.0/1 max_sfs 100Second option I was considering was use port params which doesn't sound so right as resource.
Quoted 1 line
Quoted 8 lines
Quoted 5 lines
Also, seems like there are hardware constraint at play. e.g., can a user reduce the number of queues used by the physical function to support more sub- functions? If so how does a user programmatically learn about this limitation? e.g., devlink could have support to show resource sizing and configure constraints similar to what mlxsw has.
Yes, need to figure out its naming. For mlx5 num queues doesn't have relation to subfunctions.
But PCI resource has relation and this is something we want to do in future, as you said may be using devlink resource.
With Connectx-4 Lx for example the netdev can have at most 63 queues leaving 96 cpu servers a bit short - as an example of the limited number of queues that a nic can handle (or currently exposes to the OS not sure which). If I create a subfunction for ethernet traffic, how many queues are allocated to it by default, is it managed via ethtool like the pf and is there an impact to the resources used by / available to the primary device?
Jason Gunthorpe
maintainerRe: [PATCH net-next 03/13] devlink: Support add and delete devlink port
Quoted 1 line · David Ahern
With Connectx-4 Lx for example the netdev can have at most 63 queues
What netdev calls a queue is really a "can the device deliver interrupts and packets to a given per-CPU queue" and covers a whole spectrum of smaller limits like RSS scheme, # of available interrupts, ability of the device to create queues, etc.
CX4Lx can create a huge number of queues, but hits one of these limits that mean netdev's specific usage can't scale up. Other stuff like RDMA doesn't have the same limits, and has tonnes of queues.
What seems to be needed is a resource controller concept like cgroup has for processes. The system is really organized into a tree:
physical device
mlx5_core
/ | \ \ (aux bus)
netdev rdma vdpa SF etc
| (aux bus)
mlx5_core
/ \ (aux bus)
netdev vdpa
And it does make a lot of sense to start to talk about limits at each tree level.
eg the top of the tree may have 128 physical interrupts. With 128 CPU cores that isn't enough interrupts to support all of those things concurrently.
So the user may want to configure:
- The first level netdev only gets 64,
- 3rd level mlx5_core gets 32
- Final level vdpa gets 8
Other stuff has to fight it out with the remaining shared interrupts.
In netdev land # of interrupts governs # of queues
For RDMA # of interrupts limits the CPU affinities for queues
VPDA limits the # of VMs that can use VT-d
The same story repeats for other less general resources, mlx5 also has consumption of limited BAR space, and consumption of some limited memory elements. These numbers are much bigger and may not need explicit governing, but the general concept holds.
It would be very nice if the limit could be injected when the aux device is created but before the driver is bound. I'm not sure how to manage that though..
I assume other devices will be different, maybe some devices have a limit on the number of total queues, or a limit on the number of VDPA or RDMA devices.
Jason
Re: [PATCH net-next 03/13] devlink: Support add and delete devlink port
Quoted 54 lines · Jason Gunthorpe
Quoted 1 line · David Ahern
With Connectx-4 Lx for example the netdev can have at most 63 queues
What netdev calls a queue is really a "can the device deliver interrupts and packets to a given per-CPU queue" and covers a whole spectrum of smaller limits like RSS scheme, # of available interrupts, ability of the device to create queues, etc.
CX4Lx can create a huge number of queues, but hits one of these limits that mean netdev's specific usage can't scale up. Other stuff like RDMA doesn't have the same limits, and has tonnes of queues.
What seems to be needed is a resource controller concept like cgroup has for processes. The system is really organized into a tree:
physical device
mlx5_core
/ | \ \ (aux bus)
netdev rdma vdpa SF etc
| (aux bus)
mlx5_core
/ \ (aux bus)
netdev vdpaAnd it does make a lot of sense to start to talk about limits at each tree level.
eg the top of the tree may have 128 physical interrupts. With 128 CPU cores that isn't enough interrupts to support all of those things concurrently.
So the user may want to configure:
- The first level netdev only gets 64,
- 3rd levelmlx5_coregets 32
- Final level vdpa gets 8Other stuff has to fight it out with the remaining shared interrupts.
In netdev land # of interrupts governs # of queues
For RDMA # of interrupts limits the CPU affinities for queues
VPDA limits the # of VMs that can use VT-d
The same story repeats for other less general resources, mlx5 also has consumption of limited BAR space, and consumption of some limited memory elements. These numbers are much bigger and may not need explicit governing, but the general concept holds.
It would be very nice if the limit could be injected when the aux device is created but before the driver is bound. I'm not sure how to manage that though..
Quoted 6 lines
I assume other devices will be different, maybe some devices have a limit on the number of total queues, or a limit on the number of VDPA or RDMA devices.
Jason
A lot of low level resource details that need to be summarized into a nicer user / config perspective to specify limits / allocations.
Thanks for the detailed response.
Jason Gunthorpe
maintainerRe: [PATCH net-next 03/13] devlink: Support add and delete devlink port
Quoted 55 lines · David Ahern
Quoted 54 lines · Jason Gunthorpe
Quoted 1 line · David Ahern
With Connectx-4 Lx for example the netdev can have at most 63 queues
What netdev calls a queue is really a "can the device deliver interrupts and packets to a given per-CPU queue" and covers a whole spectrum of smaller limits like RSS scheme, # of available interrupts, ability of the device to create queues, etc.
CX4Lx can create a huge number of queues, but hits one of these limits that mean netdev's specific usage can't scale up. Other stuff like RDMA doesn't have the same limits, and has tonnes of queues.
What seems to be needed is a resource controller concept like cgroup has for processes. The system is really organized into a tree:
physical device
mlx5_core
/ | \ \ (aux bus)
netdev rdma vdpa SF etc
| (aux bus)
mlx5_core
/ \ (aux bus)
netdev vdpaAnd it does make a lot of sense to start to talk about limits at each tree level.
eg the top of the tree may have 128 physical interrupts. With 128 CPU cores that isn't enough interrupts to support all of those things concurrently.
So the user may want to configure:
- The first level netdev only gets 64,
- 3rd levelmlx5_coregets 32
- Final level vdpa gets 8Other stuff has to fight it out with the remaining shared interrupts.
In netdev land # of interrupts governs # of queues
For RDMA # of interrupts limits the CPU affinities for queues
VPDA limits the # of VMs that can use VT-d
The same story repeats for other less general resources, mlx5 also has consumption of limited BAR space, and consumption of some limited memory elements. These numbers are much bigger and may not need explicit governing, but the general concept holds.
It would be very nice if the limit could be injected when the aux device is created but before the driver is bound. I'm not sure how to manage that though..
Quoted 6 lines
Quoted 3 lines
I assume other devices will be different, maybe some devices have a limit on the number of total queues, or a limit on the number of VDPA or RDMA devices.
A lot of low level resource details that need to be summarized into a nicer user / config perspective to specify limits / allocations.
Well, now that we have the aux bus stuff there is a nice natural place to put things..
The aux bus owner device (mlx5_core) could have a list of available resources
Each aux bus device (netdev/rdma/vdpa) could have a list of consumed resources
Some API to place a limit on the consumed resources at each aux bus device.
The tricky bit is the auto-probing/configure. By the time the user has a chance to apply a limit the drivers are already bound and have already done their setup. So each subsystem has to support dynamically imposing a limit..
And I simplified things a bit above too, we actually have two kinds of interrupt demand: sharable and dedicated. The actual need is to carve out a bunch of dedicated interrupts and only allow subsystems that are doing VT-d guest interrupt assignment to consume them (eg VDPA)
Jason
RE: [PATCH net-next 03/13] devlink: Support add and delete devlink port
In reply to David AhernQuoted 6 lines
From: David Ahern <[email protected]>
Sent: Wednesday, November 18, 2020 11:33 PM
Quoted 6 lines
With Connectx-4 Lx for example the netdev can have at most 63 queues leaving 96 cpu servers a bit short - as an example of the limited number of queues that a nic can handle (or currently exposes to the OS not sure which). If I create a subfunction for ethernet traffic, how many queues are allocated to it by default, is it managed via ethtool like the pf and is there an impact to the resources used by / available to the primary device?
Jason already answered it with details.
Thanks a lot Jason.
Short answer to ethtool question, yes, ethtool can change num queues for subfunction like PF. Default is same number of queues for subfunction as that of PF in this patchset.
Re: [PATCH net-next 03/13] devlink: Support add and delete devlink port
Quoted 13 lines · Parav Pandit
Quoted 6 lines
From: David Ahern <[email protected]>
Sent: Wednesday, November 18, 2020 11:33 PMQuoted 6 lines
With Connectx-4 Lx for example the netdev can have at most 63 queues leaving 96 cpu servers a bit short - as an example of the limited number of queues that a nic can handle (or currently exposes to the OS not sure which). If I create a subfunction for ethernet traffic, how many queues are allocated to it by default, is it managed via ethtool like the pf and is there an impact to the resources used by / available to the primary device?
Quoted 6 lines
Jason already answered it with details.
Thanks a lot Jason.Short answer to ethtool question, yes, ethtool can change num queues for subfunction like PF. Default is same number of queues for subfunction as that of PF in this patchset.
But what is the mechanism for partitioning the global resources of the device into each sub function?
Is it just evenly divided into the subfunctions? is there some maximum limit per sub function?
Re: [PATCH net-next 03/13] devlink: Support add and delete devlink port
Quoted 23 lines · Jacob Keller
Quoted 13 lines · Parav Pandit
Quoted 6 lines
From: David Ahern <[email protected]>
Sent: Wednesday, November 18, 2020 11:33 PMQuoted 6 lines
With Connectx-4 Lx for example the netdev can have at most 63 queues leaving 96 cpu servers a bit short - as an example of the limited number of queues that a nic can handle (or currently exposes to the OS not sure which). If I create a subfunction for ethernet traffic, how many queues are allocated to it by default, is it managed via ethtool like the pf and is there an impact to the resources used by / available to the primary device?
Quoted 6 lines
Jason already answered it with details.
Thanks a lot Jason.Short answer to ethtool question, yes, ethtool can change num queues for subfunction like PF. Default is same number of queues for subfunction as that of PF in this patchset.
Quoted 6 lines
But what is the mechanism for partitioning the global resources of the device into each sub function?
Is it just evenly divided into the subfunctions? is there some maximum limit per sub function?
I hope it is not just evenly divided; it should be user controllable. If I create a subfunction for say a container's networking, I may want to only assign 1 Rx and 1 Tx queue pair (or 1 channel depending on terminology where channel includes Rx, Tx and CQ).
Re: [PATCH net-next 03/13] devlink: Support add and delete devlink port
Quoted 27 lines · David Ahern
Quoted 26 lines · Jacob Keller
Quoted 13 lines · Parav Pandit
Quoted 6 lines
From: David Ahern <[email protected]>
Sent: Wednesday, November 18, 2020 11:33 PMQuoted 6 lines
With Connectx-4 Lx for example the netdev can have at most 63 queues leaving 96 cpu servers a bit short - as an example of the limited number of queues that a nic can handle (or currently exposes to the OS not sure which). If I create a subfunction for ethernet traffic, how many queues are allocated to it by default, is it managed via ethtool like the pf and is there an impact to the resources used by / available to the primary device?
Quoted 6 lines
Jason already answered it with details.
Thanks a lot Jason.Short answer to ethtool question, yes, ethtool can change num queues for subfunction like PF. Default is same number of queues for subfunction as that of PF in this patchset.
But what is the mechanism for partitioning the global resources of the device into each sub function?
Quoted 8 lines
Quoted 3 lines
Is it just evenly divided into the subfunctions? is there some maximum limit per sub function?
I hope it is not just evenly divided; it should be user controllable. If I create a subfunction for say a container's networking, I may want to only assign 1 Rx and 1 Tx queue pair (or 1 channel depending on terminology where channel includes Rx, Tx and CQ).
I think we need a way to expose and configure policy for resources associated with each type of auxiliary device.
For ex: default, min and max queues and interrupt vectors.
Once an auxiliary device is created, the user should be able to configure the resources within the allowed min-max values.
Re: [PATCH net-next 03/13] devlink: Support add and delete devlink port
In reply to Parav PanditQuoted 39 lines · Parav Pandit
Quoted 27 lines
From: David Ahern <[email protected]>
Sent: Wednesday, November 18, 2020 9:51 PMQuoted 14 lines · Parav Pandit
Extended devlink interface for the user to add and delete port. Extend devlink to connect user requests to driver to add/delete such port in the device.
When driver routines are invoked, devlink instance lock is not held. This enables driver to perform several devlink objects registration, unregistration such as (port, health reporter, resource etc) by using exising devlink APIs.
This also helps to uniformly use the code for port unregistration during driver unload and during port deletion initiated by user.Examples of add, show and delete commands:
$ devlink dev eswitch set pci/0000:06:00.0 mode switchdevQuoted 8 lines
$ devlink port show pci/0000:06:00.0/65535: type eth netdev ens2f0np0 flavour physical port 0 splittable false$ devlink port add pci/0000:06:00.0 flavour pcisf pfnum 0 sfnum 88$ devlink port show pci/0000:06:00.0/32768 pci/0000:06:00.0/32768: type eth netdev eth0 flavour pcisf controller 0Quoted 7 lines
pfnum 0 sfnum 88 external false splittable false
Quoted 3 lines
function:
hw_addr 00:00:00:00:88:88 state inactive opstate detachedThere has to be limits on the number of sub functions that can be created for a device. How does a user find that limit?
Yes, this came up internally, but didn't really converged.
The devlink resource looked too verbose for an average or simple use cases. But it may be fine.
The hurdle I faced with devlink resource is with defining the granularity.
Quoted 7 lines
For example one devlink instance deploys sub functions on multiple pci functions. So how to name them? Currently we have controller and PFs in port annotation. So resource name as
c0pf0_subfunctions-> for controller 0, pf 0c1pf2_subfunctions-> for controller 1, pf 2Couldn't convince my self to name it this way.
Yea, I think we need to extend the plumbing of resources to allow specifying or assigning parent resources to a subfunction.
Quoted 12 lines
Below example looked simpler to use but plumbing doesn’t exist for it.
$ devlink resource show pci/0000:03:00.0 pci/0000:03:00.0/1: name max_sfs count 256 controller 0 pf 0 pci/0000:03:00.0/2: name max_sfs count 100 controller 1 pf 0 pci/0000:03:00.0/3: name max_sfs count 64 controller 1 pf 1$ devlink resource set pci/0000:03:00.0/1 max_sfs 100Second option I was considering was use port params which doesn't sound so right as resource.
I don't think port parameters make sense here. They only encapsulate single name -> value pairs, and don't really help show the relationships between the subfunction ports and the parent device.
Quoted 9 lines
Quoted 6 lines
Also, seems like there are hardware constraint at play. e.g., can a user reduce the number of queues used by the physical function to support more sub- functions? If so how does a user programmatically learn about this limitation? e.g., devlink could have support to show resource sizing and configure constraints similar to what mlxsw has.
Yes, need to figure out its naming. For mlx5 num queues doesn't have relation to subfunctions.
But PCI resource has relation and this is something we want to do in future, as you said may be using devlink resource.
I've been looking into queue management and being able to add and remove queue groups and queues. I'm leaning towards building on top of devlink resource for this.
Specifically I have been looking at picking up the work started by Magnus last year, around creating interface for representing queues to the stack better for AF_XDP, but it also has other possible uses.
I'd like to make sure it aligns with the ideas here for partitioning resources. It seems like that should be best done at the devlink level, where the main devlink instance knows about all the part limitations and can then have new commands for allowing assignment of resources to ports.
PATCH 4/13 devlink: Support get and set state of port function
In reply to Parav Panditdevlink port function can be in active or inactive state. Allow users to get and set port function's state.
When the port function it activated, its operational state may change after a while when the device is created and driver binds to it. Similarly on deactivation flow.
To clearly describe the state of the port function and its device's operational state in the host system, define state and opstate attributes.
Example of a PCI SF port which supports a port function:
Create a device with ID=10 and one physical port.
$ devlink dev eswitch set pci/0000:06:00.0 mode switchdev
$ devlink port show pci/0000:06:00.0/65535: type eth netdev ens2f0np0 flavour physical port 0 splittable false
$ devlink port add pci/0000:06:00.0 flavour pcisf pfnum 0 sfnum 88
$ devlink port show pci/0000:06:00.0/32768
pci/0000:06:00.0/32768: type eth netdev eth0 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false
function:
hw_addr 00:00:00:00:88:88 state inactive opstate detached$ devlink port function set pci/0000:06:00.0/32768 hw_addr 00:00:00:00:88:88 state active
$ devlink port show pci/0000:06:00.0/32768 -jp
{
"port": {
"pci/0000:06:00.0/32768": {
"type": "eth",
"netdev": "eth0",
"flavour": "pcisf",
"controller": 0,
"pfnum": 0,
"sfnum": 88,
"external": false,
"splittable": false,
"function": {
"hw_addr": "00:00:00:00:88:88",
"state": "active",
"opstate": "attached"
}
}
}
}--- include/net/devlink.h | 21 +++++++++ include/uapi/linux/devlink.h | 21 +++++++++ net/core/devlink.c | 89 +++++++++++++++++++++++++++++++++++- 3 files changed, 130 insertions(+), 1 deletion(-)
include/net/devlink.h +21 -0
@@ -1371,6 +1371,27 @@ struct devlink_ops {int (*port_function_hw_addr_set)(struct devlink *devlink,struct devlink_port *port,const u8 *hw_addr,int hw_addr_len,struct netlink_ext_ack *extack); +/** +* @port_function_state_get: Port function's state get function. +* +* Should be used by device drivers to report the state of a function managed +* by the devlink port. Driver should return -EOPNOTSUPP if it doesn't support port +* function handling for a particular port. +*/ +int (*port_function_state_get)(struct devlink *devlink,struct devlink_port *port, +enum devlink_port_function_state *state, +enum devlink_port_function_opstate *opstate, +struct netlink_ext_ack *extack); +/** +* @port_function_state_set: Port function's state set function. +* +* Should be used by device drivers to set the state of a function managed +* by the devlink port. Driver should return -EOPNOTSUPP if it doesn't support port +* function handling for a particular port. +*/ +int (*port_function_state_set)(struct devlink *devlink,struct devlink_port *port, +enum devlink_port_function_state state, +struct netlink_ext_ack *extack);/** * @port_new: Port add function. *
include/uapi/linux/devlink.h +21 -0
@@ -581,9 +581,30 @@ enum devlink_resource_unit {enum devlink_port_function_attr {DEVLINK_PORT_FUNCTION_ATTR_UNSPEC ,DEVLINK_PORT_FUNCTION_ATTR_HW_ADDR ,/* binary */ +DEVLINK_PORT_FUNCTION_ATTR_STATE ,/* u8 */ +DEVLINK_PORT_FUNCTION_ATTR_OPSTATE ,/* u8 */ __DEVLINK_PORT_FUNCTION_ATTR_MAX,DEVLINK_PORT_FUNCTION_ATTR_MAX = __DEVLINK_PORT_FUNCTION_ATTR_MAX -1 }; +enum devlink_port_function_state { +DEVLINK_PORT_FUNCTION_STATE_INACTIVE , +DEVLINK_PORT_FUNCTION_STATE_ACTIVE , +}; + +/** +* enum devlink_port_function_opstate - indicates operational state of port function +* @DEVLINK_PORT_FUNCTION_OPSTATE_ATTACHED: Driver is attached to the function of port, for +* gracefufl tear down of the function, after +* inactivation of the port function, user should wait +* for operational state to turn DETACHED. +* @DEVLINK_PORT_FUNCTION_OPSTATE_DETACHED: Driver is detached from the function of port; it is +* safe to delete the port. +*/ +enum devlink_port_function_opstate { +DEVLINK_PORT_FUNCTION_OPSTATE_DETACHED , +DEVLINK_PORT_FUNCTION_OPSTATE_ATTACHED , +}; +#endif /* _UAPI_LINUX_DEVLINK_H_ */
net/core/devlink.c +88 -1
120 lines not on this page · raw message
Signature · 1 line
2.26.2
PATCH 5/13 devlink: Avoid global devlink mutex, use per instance reload lock
In reply to Parav Panditdevlink device reload is a special operation which brings down and up the device. Such operation will unregister devlink device of sub function port.
During devlink_reload() with devlink_mutex held leads to cyclic dependency. For example,
devlink_reload()
mutex_lock(&devlink_mutex); <- First lock acquire mlx5_reload_down(PCI PF device)
disable_sf_devices();
sf_state_set(inactive);
ancillary_dev->remove();
mlx5_adev_remove(adev);
devlink_unregister(adev->devlink_instance);
mutex_lock(&devlink_mutex); <- Second lock acquireHence devlink_reload() operation cannot be done under global devlink_mutex mutex.
In second such instance reload_down() callback likely to disable reload on child devlink device. This also prevents devlink_reload() to use the overloaded global devlink_mutex.
devlink_reload()
mutex_lock(&devlink_mutex); <- First lock acquire
mlx5_reload_down(PCI PF device)
disable_sf_devices();
ancillary_dev->remove();
mlx5_adev_remove(adev);
devlink_reload_disable(adev->devlink_instance);
mutex_lock(&devlink_mutex); <- Second lock acquireTherefore, introduce a reload_lock per devlink instance which is held when performing devlink device reload.
--- include/net/devlink.h | 1 + net/core/devlink.c | 25 +++++++++++++++---------- 2 files changed, 16 insertions(+), 10 deletions(-)
include/net/devlink.h +1 -0
@@ -52,6 +52,7 @@ struct devlink {struct mutex lock;/* Serializes access to devlink instance specific objects such as * port, sb, dpipe, resource, params, region, traps and more. */ +struct mutex reload_lock;/* Protects reload operation */ u8 reload_failed:1, reload_enabled:1, registered:1;
net/core/devlink.c +15 -10
@@ -3307,29 +3307,32 @@ static int devlink_reload(struct devlink *devlink, struct net *dest_net,u32 remote_reload_stats[DEVLINK_RELOAD_STATS_ARRAY_SIZE ];int err; -if (!devlink->reload_enabled) -return -EOPNOTSUPP ; +mutex_lock (&devlink->reload_lock); +if (!devlink->reload_enabled) { + err = -EOPNOTSUPP ; + goto done; + }memcpy (remote_reload_stats, devlink->stats.remote_reload_stats,sizeof (remote_reload_stats)); err = devlink->ops->reload_down (devlink, !!dest_net, action, limit, extack);if (err) -return err; + goto done;if (dest_net && !net_eq (dest_net,devlink_net (devlink)))devlink_reload_netns_change (devlink, dest_net); err = devlink->ops->reload_up (devlink, action, limit, actions_performed, extack);devlink_reload_failed_set (devlink, !!err); -if (err) -return err;WARN_ON (!(*actions_performed &BIT (action)));/* Catch driver on updating the remote action within devlink reload */ WARN_ON (memcmp (remote_reload_stats, devlink->stats.remote_reload_stats,sizeof (remote_reload_stats)));devlink_reload_stats_update (devlink, limit, *actions_performed); -return 0 ; +done: +mutex_unlock (&devlink->reload_lock); +return err; } static int
@@ -8118,6 +8121,7 @@ struct devlink *devlink_alloc(const struct devlink_ops *ops, size_t priv_size)INIT_LIST_HEAD (&devlink->trap_policer_list);mutex_init (&devlink->lock);mutex_init (&devlink->reporters_lock); +mutex_init (&devlink->reload_lock);return devlink; }EXPORT_SYMBOL_GPL (devlink_alloc);
@@ -8166,9 +8170,9 @@ EXPORT_SYMBOL_GPL(devlink_unregister);*/ void devlink_reload_enable (struct devlink *devlink) { -mutex_lock (&devlink_mutex); +mutex_lock (&devlink->reload_lock); devlink->reload_enabled = true; -mutex_unlock (&devlink_mutex); +mutex_unlock (&devlink->reload_lock); }EXPORT_SYMBOL_GPL (devlink_reload_enable);
@@ -8182,12 +8186,12 @@ EXPORT_SYMBOL_GPL(devlink_reload_enable);*/ void devlink_reload_disable (struct devlink *devlink) { -mutex_lock (&devlink_mutex); +mutex_lock (&devlink->reload_lock);/* Mutex is taken which ensures that no reload operation is in * progress while setting up forbidded flag. */ devlink->reload_enabled = false; -mutex_unlock (&devlink_mutex); +mutex_unlock (&devlink->reload_lock); }EXPORT_SYMBOL_GPL (devlink_reload_disable);
@@ -8198,6 +8202,7 @@ EXPORT_SYMBOL_GPL(devlink_reload_disable);*/ void devlink_free (struct devlink *devlink) { +mutex_destroy (&devlink->reload_lock);mutex_destroy (&devlink->reporters_lock);mutex_destroy (&devlink->lock);WARN_ON (!list_empty (&devlink->trap_policer_list));
Signature · 1 line
2.26.2
PATCH 6/13 devlink: Introduce devlink refcount to reduce scope of global devlink_mutex
In reply to Parav PanditCurrently global devlink_mutex is held while a doit() operation is progress. This brings a limitation.
A Driver cannot perform devlink_register()/unregister() calls during devlink doit() callback functions.
This is typically required when a port state change described in RFC [1] callback wants to delete an active SF port or wants to activate a SF port that results into unregistering or registering a devlink instance on different bus such as ancillary bus.
An example flow:
devlink_predoit()
mutex_lock(&devlink_mutex); <- First lock acquire devlink_reload()
driver->reload_down(inactive)
adev->remove();
mlx5_adev_remove(ancillary_dev);
devlink_unregister(ancillary_dev->devlink_instance);
mutex_lock(&devlink_mutex); <- Second lock acquireThis patch is preparation patch to enable drivers to achieve this.
It achieves this by maintaining a per devlink instance refcount to prevent devlink device unregistration while user command are in progress or while devlink device is migration to init_net net namespace.
devlink_nl_family continue to remain registered with parallel_ops disabled. So even after removing devlink_mutex during doit commands, it doesn't enable userspace to run multiple devlink commands for one or multiple devlink instance.
[1] https://lore.kernel.org/netdev/20200519092258.GF4655@nanopsycho
--- include/net/devlink.h | 5 +++ net/core/devlink.c | 84 +++++++++++++++++++++++++++++++------------ 2 files changed, 67 insertions(+), 22 deletions(-)
include/net/devlink.h +5 -0
@@ -53,6 +53,11 @@ struct devlink {* port, sb, dpipe, resource, params, region, traps and more. */ struct mutex reload_lock;/* Protects reload operation */ +struct list_head reload_list; +refcount_t refcount;/* Serializes user doit commands and netns command +* with device unregistration. +*/ +struct completion unregister_complete; u8 reload_failed:1, reload_enabled:1, registered:1;
net/core/devlink.c +62 -22
163 lines not on this page · raw message
Signature · 1 line
2.26.2
PATCH 7/13 net/mlx5: SF, Add auxiliary device support
In reply to Parav PanditIntroduce API to add and delete an auxiliary device for an SF. Each SF has its own dedicated window in the PCI BAR 2.
SF device is similar to PCI PF and VF that supports multiple class of devices such as net, rdma and vdpa.
SF device will be added or removed in subsequent patch during SF devlink port function state change command.
A subfunction device exposes user supplied subfunction number which will be further used by systemd/udev to have deterministic name for its netdevice and rdma device.
An mlx5 subfunction auxiliary device example:
$ devlink dev eswitch set pci/0000:06:00.0 mode switchdev
$ devlink port show pci/0000:06:00.0/65535: type eth netdev ens2f0np0 flavour physical port 0 splittable false
$ devlink port add pci/0000:06:00.0 flavour pcisf pfnum 0 sfnum 88
$ devlink port show ens2f0npf0sf88
pci/0000:06:00.0/32768: type eth netdev ens2f0npf0sf88 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false
function:
hw_addr 00:00:00:00:88:88 state inactive opstate detached$ devlink port function set ens2f0npf0sf88 hw_addr 00:00:00:00:88:88 state active
On activation,
$ ls -l /sys/bus/auxiliary/devices/ mlx5_core.sf.0 -> ../../../devices/pci0000:00/0000:00:03.0/0000:06:00.0/mlx5_core.sf.0
$ cat /sys/bus/auxiliary/devices/mlx5_core.sf.0/sfnum 88
--- .../net/ethernet/mellanox/mlx5/core/Kconfig | 9 + .../net/ethernet/mellanox/mlx5/core/Makefile | 4 + .../net/ethernet/mellanox/mlx5/core/main.c | 12 + .../ethernet/mellanox/mlx5/core/sf/dev/dev.c | 213 ++++++++++++++++++ .../ethernet/mellanox/mlx5/core/sf/dev/dev.h | 55 +++++ include/linux/mlx5/driver.h | 4 + 6 files changed, 297 insertions(+) create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/sf/dev/dev.c create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/sf/dev/dev.h
drivers/net/ethernet/mellanox/mlx5/core/Kconfig +9 -0
@@ -202,3 +202,12 @@ config MLX5_SW_STEERING default y help Build support for software-managed steering in the NIC. + +config MLX5_SF + bool "Mellanox Technologies subfunction device support using auxiliary device" + depends on MLX5_CORE && MLX5_CORE_EN + default n + help + Build support for subfuction device in the NIC. A Mellanox subfunction + device can support RDMA, netdevice and vdpa device. + It is similar to a SRIOV VF but it doesn't require SRIOV support.
drivers/net/ethernet/mellanox/mlx5/core/Makefile +4 -0
@@ -85,3 +85,7 @@ mlx5_core-$(CONFIG_MLX5_SW_STEERING) += steering/dr_domain.o steering/dr_table.o steering/dr_ste.o steering/dr_send.o \ steering/dr_cmd.o steering/dr_fw.o \ steering/dr_action.o steering/fs_dr.o +# +# SF device +# +mlx5_core-$(CONFIG_MLX5_SF) += sf/dev/dev.o
drivers/net/ethernet/mellanox/mlx5/core/main.c +12 -0
@@ -73,6 +73,7 @@#include "ecpf.h" #include "lib/hv_vhca.h" #include "diag/rsc_dump.h" +#include "sf/dev/dev.h" MODULE_AUTHOR ("Eli Cohen <[email protected]>" );MODULE_DESCRIPTION ("Mellanox 5th generation network adapters (ConnectX series) core driver" );
@@ -884,6 +885,12 @@ static int mlx5_init_once(struct mlx5_core_dev *dev) goto err_eswitch_cleanup; } + err =mlx5_sf_dev_table_init (dev); +if (err) { +mlx5_core_err (dev,"Failed to init SF device table %d\n" , err); + goto err_sf_dev_cleanup; + } + dev->dm =mlx5_dm_create (dev);if (IS_ERR (dev->dm))mlx5_core_warn (dev,"Failed to init device memory%d\n" , err);
@@ -894,6 +901,8 @@ static int mlx5_init_once(struct mlx5_core_dev *dev)return 0 ; +err_sf_dev_cleanup: +mlx5_fpga_cleanup (dev); err_eswitch_cleanup:mlx5_eswitch_cleanup (dev->priv.eswitch); err_sriov_cleanup:
@@ -925,6 +934,7 @@ static void mlx5_cleanup_once(struct mlx5_core_dev *dev)mlx5_hv_vhca_destroy (dev->hv_vhca);mlx5_fw_tracer_destroy (dev->tracer);mlx5_dm_cleanup (dev); +mlx5_sf_dev_table_cleanup (dev);mlx5_fpga_cleanup (dev);mlx5_eswitch_cleanup (dev->priv.eswitch);mlx5_sriov_cleanup (dev);
@@ -1141,6 +1151,7 @@ static int mlx5_load(struct mlx5_core_dev *dev) goto err_ec; } +mlx5_sf_dev_table_create (dev);return 0 ; err_ec:
@@ -1171,6 +1182,7 @@ static int mlx5_load(struct mlx5_core_dev *dev)static void mlx5_unload (struct mlx5_core_dev *dev) { +mlx5_sf_dev_table_destroy (dev);mlx5_ec_cleanup (dev);mlx5_sriov_detach (dev);mlx5_cleanup_fs (dev);
drivers/net/ethernet/mellanox/mlx5/core/sf/dev/dev.c new file +213 -0
213 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/sf/dev/dev.h new file +55 -0
55 lines not on this page · raw message
include/linux/mlx5/driver.h +4 -0
@@ -507,6 +507,7 @@ struct mlx5_devcom;struct mlx5_fw_reset ;struct mlx5_eq_table ;struct mlx5_irq_table ; +struct mlx5_sf_dev_table ;struct mlx5_rate_limit {u32 rate;
@@ -603,6 +604,9 @@ struct mlx5_priv {struct mlx5_bfreg_data bfregs;struct mlx5_uars_page *uar; +#ifdef CONFIG_MLX5_SF +struct mlx5_sf_dev_table *sf_dev_table; +#endif };enum mlx5_device_state {
Signature · 1 line
2.26.2
Re: [PATCH net-next 07/13] net/mlx5: SF, Add auxiliary device support
Quoted 7 lines · Parav Pandit
drivers/net/ethernet/mellanox/mlx5/core/Kconfig +0
-0@@ -202,3 +202,12 @@ config MLX5_SW_STEERING default y help
drivers/net/ethernet/mellanox/mlx5/core/Kconfig config MLX5_SW_STEERING · line 213
Quoted 10 lines
drivers/net/ethernet/mellanox/mlx5/core/Kconfig +9
-0Build support for software-managed steering in the NIC. + +config MLX5_SF + bool "Mellanox Technologies subfunction device support using auxiliary device" + depends on MLX5_CORE && MLX5_CORE_EN + default n + help + Build support for subfuction device in the NIC. A Mellanox subfunction + device can support RDMA, netdevice and vdpa device. + It is similar to a SRIOV VF but it doesn't require SRIOV support.
per Dan's comment about AUXILIARY_BUS being select only, should this config select AUXILIARY_BUS?
RE: [PATCH net-next 07/13] net/mlx5: SF, Add auxiliary device support
Quoted 19 lines
From: David Ahern <[email protected]>
Sent: Monday, December 7, 2020 8:19 AMdrivers/net/ethernet/mellanox/mlx5/core/Kconfig config MLX5_SW_STEERING · line 207
Quoted 12 lines · Parav Pandit
drivers/net/ethernet/mellanox/mlx5/core/Kconfig +3
-0@@ -202,3 +202,12 @@ config MLX5_SW_STEERING default y help Build support for software-managed steering in the NIC. + +config MLX5_SF + bool "Mellanox Technologies subfunction device support using auxiliarydevice"
Quoted 9 lines
drivers/net/ethernet/mellanox/mlx5/core/Kconfig config MLX5_SW_STEERING · line 213
Quoted 6 lines
drivers/net/ethernet/mellanox/mlx5/core/Kconfig +6
-0+ depends on MLX5_CORE && MLX5_CORE_EN + default n + help + Build support for subfuction device in the NIC. A Mellanox subfunction + device can support RDMA, netdevice and vdpa device. + It is similar to a SRIOV VF but it doesn't require SRIOV support.per Dan's comment about
AUXILIARY_BUSbeing select only, should this config selectAUXILIARY_BUS?
Yes.
However, my patchset depends on patchset [1].
With introduction of patchset [2], MLX5_CORE depends on AUXILIARY_BUS. MLX5_SF depends on MLX5_CORE.
So I omitted explicitly selecting AUXBUS by MLX5_SF.
[1] https://lore.kernel.org/linux-rdma/[email protected]/
[2] https://lore.kernel.org/alsa-devel/[email protected]/
PATCH 8/13 net/mlx5: SF, Add auxiliary device driver
In reply to Parav PanditAdd auxiliary device driver for mlx5 subfunction auxiliary device.
A mlx5 subfunction is similar to PCI PF and VF. For a subfunction an auxiliary device is created.
As a result, when mlx5 SF auxiliary device binds to the driver, its netdev and rdma device are created, they appear as
$ ls -l /sys/bus/auxiliary/devices/ mlx5_core.sf.0 -> ../../../devices/pci0000:00/0000:00:03.0/0000:06:00.0/mlx5_core.sf.0
$ ls -l /sys/class/net/eth1/device /sys/class/net/eth1/device -> ../../../mlx5_core.sf.0
$ cat /sys/bus/auxiliary/devices/mlx5_core.sf.0/sfnum 88
$ devlink dev show pci/0000:06:00.0 auxiliary/mlx5_core.sf.0
$ devlink port show auxiliary/mlx5_core.sf.0/1 auxiliary/mlx5_core.sf.0/1: type eth netdev p0sf88 flavour virtual port 0 splittable false
$ rdma link show mlx5_0/1 link mlx5_0/1 state ACTIVE physical_state LINK_UP netdev p0sf88
$ rdma dev show 8: rocep6s0f1: node_type ca fw 16.27.1017 node_guid 248a:0703:00b3:d113 sys_image_guid 248a:0703:00b3:d112 13: mlx5_0: node_type ca fw 16.27.1017 node_guid 0000:00ff:fe00:8888 sys_image_guid 248a:0703:00b3:d112
In future, devlink device instance name will adapt to have sfnum annotation using either an alias or as devlink instance name described in RFC [1].
[1] https://lore.kernel.org/netdev/20200519092258.GF4655@nanopsycho/
--- .../net/ethernet/mellanox/mlx5/core/Makefile | 2 +- drivers/net/ethernet/mellanox/mlx5/core/eq.c | 2 +- .../net/ethernet/mellanox/mlx5/core/main.c | 22 ++-- .../ethernet/mellanox/mlx5/core/mlx5_core.h | 10 ++ .../net/ethernet/mellanox/mlx5/core/pci_irq.c | 20 ++++ .../ethernet/mellanox/mlx5/core/sf/dev/dev.h | 13 +++ .../mellanox/mlx5/core/sf/dev/driver.c | 105 ++++++++++++++++++ include/linux/mlx5/driver.h | 4 +- 8 files changed, 168 insertions(+), 10 deletions(-) create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/sf/dev/driver.c
drivers/net/ethernet/mellanox/mlx5/core/Makefile +1 -1
@@ -88,4 +88,4 @@ mlx5_core-$(CONFIG_MLX5_SW_STEERING) += steering/dr_domain.o steering/dr_table.o # # SF device # -mlx5_core-$(CONFIG_MLX5_SF) += sf/dev/dev.o +mlx5_core-$(CONFIG_MLX5_SF) += sf/dev/dev.o sf/dev/driver.o
drivers/net/ethernet/mellanox/mlx5/core/eq.c +1 -1
@@ -465,7 +465,7 @@ int mlx5_eq_table_init(struct mlx5_core_dev *dev)for (i =0 ; i <MLX5_EVENT_TYPE_MAX ; i++)ATOMIC_INIT_NOTIFIER_HEAD (&eq_table->nh[i]); - eq_table->irq_table = dev->priv.irq_table; + eq_table->irq_table =mlx5_irq_table_get (dev);return 0 ; }
drivers/net/ethernet/mellanox/mlx5/core/main.c +15 -7
@@ -83,7 +83,6 @@ unsigned int mlx5_core_debug_mask;module_param_named (debug_mask, mlx5_core_debug_mask, uint,0644 );MODULE_PARM_DESC (debug_mask,"debug mask: 1 = dump cmd data, 2 = dump cmd exec time, 3 = both. Default=0" ); -#define MLX5_DEFAULT_PROF 2static unsigned int prof_sel =MLX5_DEFAULT_PROF ;module_param_named (prof_sel, prof_sel, uint,0444 );MODULE_PARM_DESC (prof_sel,"profile selector. Valid range 0 - 2" );
@@ -1295,7 +1294,7 @@ void mlx5_unload_one(struct mlx5_core_dev *dev, bool cleanup)mutex_unlock (&dev->intf_state_mutex); } -static int mlx5_mdev_init (struct mlx5_core_dev *dev,int profile_idx) +int mlx5_mdev_init (struct mlx5_core_dev *dev,int profile_idx) {struct mlx5_priv *priv = &dev->priv;int err;
@@ -1345,7 +1344,7 @@ static int mlx5_mdev_init(struct mlx5_core_dev *dev, int profile_idx)return err; } -static void mlx5_mdev_uninit (struct mlx5_core_dev *dev) +void mlx5_mdev_uninit (struct mlx5_core_dev *dev) {struct mlx5_priv *priv = &dev->priv;
@@ -1684,16 +1683,24 @@ static int __init init(void)if (err) goto err_debug; + err =mlx5_sf_driver_register (); +if (err) + goto err_sf; +#ifdef CONFIG_MLX5_CORE_EN err =mlx5e_init (); -if (err) { -pci_unregister_driver (&mlx5_core_driver); - goto err_debug; - } +if (err) + goto err_eth;#endif return 0 ; +#ifdef CONFIG_MLX5_CORE_EN +err_eth: +mlx5_sf_driver_unregister (); +#endif +err_sf: +pci_unregister_driver (&mlx5_core_driver); err_debug:mlx5_unregister_debugfs ();return err;
@@ -1704,6 +1711,7 @@ static void __exit cleanup(void)#ifdef CONFIG_MLX5_CORE_EN mlx5e_cleanup ();#endif +mlx5_sf_driver_unregister ();pci_unregister_driver (&mlx5_core_driver);mlx5_unregister_debugfs (); }
drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h +10 -0
@@ -117,6 +117,8 @@ enum mlx5_semaphore_space_address { MLX5_SEMAPHORE_SW_RESET = 0x20, }; +#define MLX5_DEFAULT_PROF 2 +int mlx5_query_hca_caps (struct mlx5_core_dev *dev);int mlx5_query_board_id (struct mlx5_core_dev *dev);int mlx5_cmd_init_hca (struct mlx5_core_dev *dev,uint32_t *sw_owner_id);
@@ -172,6 +174,7 @@ struct cpumask * mlx5_irq_get_affinity_mask(struct mlx5_irq_table *irq_table, int vecidx);struct cpu_rmap *mlx5_irq_get_rmap (struct mlx5_irq_table *table);int mlx5_irq_get_num_comp (struct mlx5_irq_table *table); +struct mlx5_irq_table *mlx5_irq_table_get (struct mlx5_core_dev *dev);int mlx5_events_init (struct mlx5_core_dev *dev);void mlx5_events_cleanup (struct mlx5_core_dev *dev);
@@ -253,6 +256,13 @@ enum {u8 mlx5_get_nic_state (struct mlx5_core_dev *dev);void mlx5_set_nic_state (struct mlx5_core_dev *dev,u8 state); +static inline bool mlx5_core_is_sf (const struct mlx5_core_dev *dev) +{ +return dev->coredev_type ==MLX5_COREDEV_SF ; +} + +int mlx5_mdev_init (struct mlx5_core_dev *dev,int profile_idx); +void mlx5_mdev_uninit (struct mlx5_core_dev *dev);void mlx5_unload_one (struct mlx5_core_dev *dev,bool cleanup);int mlx5_load_one (struct mlx5_core_dev *dev,bool boot);#endif /* __MLX5_CORE_H__ */
drivers/net/ethernet/mellanox/mlx5/core/pci_irq.c +20 -0
@@ -30,6 +30,9 @@ int mlx5_irq_table_init(struct mlx5_core_dev *dev) {struct mlx5_irq_table *irq_table; +if (mlx5_core_is_sf (dev)) +return 0 ; + irq_table =kvzalloc (sizeof (*irq_table),GFP_KERNEL );if (!irq_table)return -ENOMEM ;
@@ -40,6 +43,9 @@ int mlx5_irq_table_init(struct mlx5_core_dev *dev)void mlx5_irq_table_cleanup (struct mlx5_core_dev *dev) { +if (mlx5_core_is_sf (dev)) +return ; +kvfree (dev->priv.irq_table); }
@@ -268,6 +274,9 @@ int mlx5_irq_table_create(struct mlx5_core_dev *dev)int nvec;int err; +if (mlx5_core_is_sf (dev)) +return 0 ; + nvec =MLX5_CAP_GEN (dev, num_ports) *num_online_cpus () +MLX5_IRQ_VEC_COMP_BASE ; nvec =min_t (int, nvec, num_eqs);
@@ -319,6 +328,9 @@ void mlx5_irq_table_destroy(struct mlx5_core_dev *dev)struct mlx5_irq_table *table = dev->priv.irq_table;int i; +if (mlx5_core_is_sf (dev)) +return ; +/* free_irq requires that affinity and rmap will be cleared * before calling it. This is why there is asymmetry with set_rmap * which should be called after alloc_irq but before request_irq.
@@ -332,3 +344,11 @@ void mlx5_irq_table_destroy(struct mlx5_core_dev *dev)kfree (table->irq); } +struct mlx5_irq_table *mlx5_irq_table_get (struct mlx5_core_dev *dev) +{ +#ifdef CONFIG_MLX5_SF +if (mlx5_core_is_sf (dev)) +return dev->priv.parent_mdev->priv.irq_table; +#endif +return dev->priv.irq_table; +}
drivers/net/ethernet/mellanox/mlx5/core/sf/dev/dev.h +13 -0
@@ -13,6 +13,7 @@struct mlx5_sf_dev {struct auxiliary_device adev;struct mlx5_core_dev *parent_mdev; +struct mlx5_core_dev *mdev;phys_addr_t bar_base_addr;u32 sfnum; };
@@ -26,6 +27,9 @@ void mlx5_sf_dev_table_destroy(struct mlx5_core_dev *dev);int mlx5_sf_dev_add (struct mlx5_core_dev *dev,u16 sf_index,u32 sfnum);void mlx5_sf_dev_del (struct mlx5_core_dev *dev,u16 sf_index); +int mlx5_sf_driver_register (void ); +void mlx5_sf_driver_unregister (void ); +#else static inline void __exit mlx5_sf_dev_exit(void)
@@ -50,6 +54,15 @@ static inline void mlx5_sf_dev_table_destroy(struct mlx5_core_dev *dev) { } +static inline int mlx5_sf_driver_register (void ) +{ +return 0 ; +} + +static inline void mlx5_sf_driver_unregister (void ) +{ +} +#endif #endif
drivers/net/ethernet/mellanox/mlx5/core/sf/dev/driver.c new file +105 -0
105 lines not on this page · raw message
include/linux/mlx5/driver.h +3 -1
@@ -193,7 +193,8 @@ enum port_state_policy {enum mlx5_coredev_type {MLX5_COREDEV_PF , -MLX5_COREDEV_VF +MLX5_COREDEV_VF , +MLX5_COREDEV_SF , };struct mlx5_field_desc {
@@ -606,6 +607,7 @@ struct mlx5_priv {struct mlx5_uars_page *uar;#ifdef CONFIG_MLX5_SF struct mlx5_sf_dev_table *sf_dev_table; +struct mlx5_core_dev *parent_mdev;#endif };
Signature · 1 line
2.26.2
PATCH 9/13 net/mlx5: E-switch, Prepare eswitch to handle SF vport
In reply to Parav PanditFrom: Vu Pham <[email protected]>
Prepare eswitch to handle SF vport during
(a) querying eswitch functions
(b) egress ACL creation
(c) account for SF vports in total vports calculation
Assign a dedicated placeholder for SFs vports and their representors. They are placed after VFs vports and before ECPF vports as below:
[PF,VF0,...,VFn,SF0,...SFm,ECPF,UPLINK].
Change functions to map SF's vport numbers to indices when accessing the vports or representors arrays, and vice versa.
--- .../net/ethernet/mellanox/mlx5/core/Kconfig | 10 ++++ .../mellanox/mlx5/core/esw/acl/egress_ofld.c | 2 +- .../net/ethernet/mellanox/mlx5/core/eswitch.c | 11 +++- .../net/ethernet/mellanox/mlx5/core/eswitch.h | 55 +++++++++++++++++++ .../mellanox/mlx5/core/eswitch_offloads.c | 11 ++++ .../net/ethernet/mellanox/mlx5/core/sf/sf.h | 35 ++++++++++++ .../net/ethernet/mellanox/mlx5/core/vport.c | 3 +- 7 files changed, 123 insertions(+), 4 deletions(-) create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/sf/sf.h
drivers/net/ethernet/mellanox/mlx5/core/Kconfig +10 -0
@@ -211,3 +211,13 @@ config MLX5_SF Build support for subfuction device in the NIC. A Mellanox subfunction device can support RDMA, netdevice and vdpa device. It is similar to a SRIOV VF but it doesn't require SRIOV support. + +config MLX5_SF_MANAGER + bool + depends on MLX5_SF && MLX5_ESWITCH + default y + help + Build support for subfuction port in the NIC. A Mellanox subfunction + port is managed through devlink. A subfunction supports RDMA, netdevice + and vdpa device. It is similar to a SRIOV VF but it doesn't require + SRIOV support.
drivers/net/ethernet/mellanox/mlx5/core/esw/acl/egress_ofld.c +1 -1
@@ -150,7 +150,7 @@ static void esw_acl_egress_ofld_groups_destroy(struct mlx5_vport *vport)static bool esw_acl_egress_needed (const struct mlx5_eswitch *esw,u16 vport_num) { -return mlx5_eswitch_is_vf_vport (esw, vport_num); +return mlx5_eswitch_is_vf_vport (esw, vport_num) ||mlx5_esw_is_sf_vport (esw, vport_num); }int esw_acl_egress_ofld_setup (struct mlx5_eswitch *esw,struct mlx5_vport *vport)
drivers/net/ethernet/mellanox/mlx5/core/eswitch.c +9 -2
@@ -1369,9 +1369,15 @@ const u32 *mlx5_esw_query_functions(struct mlx5_core_dev *dev) {int outlen =MLX5_ST_SZ_BYTES (query_esw_functions_out);u32 in[MLX5_ST_SZ_DW (query_esw_functions_in)] = {}; +u16 max_sf_vports;u32 *out;int err; + max_sf_vports =mlx5_sf_max_ports (dev); +/* Device interface is array of 64-bits */ +if (max_sf_vports) + outlen +=DIV_ROUND_UP (max_sf_vports,BITS_PER_TYPE (__be64)) *sizeof (__be64); + out =kvzalloc (outlen,GFP_KERNEL );if (!out)return ERR_PTR (-ENOMEM );
@@ -1379,7 +1385,7 @@ const u32 *mlx5_esw_query_functions(struct mlx5_core_dev *dev)MLX5_SET (query_esw_functions_in, in, opcode,MLX5_CMD_OP_QUERY_ESW_FUNCTIONS ); - err =mlx5_cmd_exec_inout (dev, query_esw_functions, in, out); + err =mlx5_cmd_exec (dev, in,sizeof (in), out, outlen);if (!err)return out;
@@ -1874,7 +1880,8 @@ static bool is_port_function_supported(const struct mlx5_eswitch *esw, u16 vport_num) {return vport_num ==MLX5_VPORT_PF || -mlx5_eswitch_is_vf_vport (esw, vport_num); +mlx5_eswitch_is_vf_vport (esw, vport_num) || +mlx5_esw_is_sf_vport (esw, vport_num); } int mlx5_devlink_port_function_hw_addr_get(struct devlink *devlink,
drivers/net/ethernet/mellanox/mlx5/core/eswitch.h +55 -0
85 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +11 -0
@@ -1801,11 +1801,22 @@ static void __esw_offloads_unload_rep(struct mlx5_eswitch *esw, esw->offloads.rep_ops[rep_type]->unload (rep); } +static void __unload_reps_sf_vport (struct mlx5_eswitch *esw,u8 rep_type) +{ +struct mlx5_eswitch_rep *rep; +int i; + +mlx5_esw_for_each_sf_rep (esw, i, rep) +__esw_offloads_unload_rep (esw, rep, rep_type); +} +static void __unload_reps_all_vport (struct mlx5_eswitch *esw,u8 rep_type) {struct mlx5_eswitch_rep *rep;int i; +__unload_reps_sf_vport (esw, rep_type); +mlx5_esw_for_each_vf_rep_reverse (esw, i, rep, esw->esw_funcs.num_vfs)__esw_offloads_unload_rep (esw, rep, rep_type);
drivers/net/ethernet/mellanox/mlx5/core/sf/sf.h new file +35 -0
@@ -0,0 +1,35 @@ +/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */ +/* Copyright (c) 2020 Mellanox Technologies Ltd */ + +#ifndef __MLX5_SF_H__ +#define __MLX5_SF_H__ + +#include <linux/mlx5/driver.h> + +#ifdef CONFIG_MLX5_SF_MANAGER + +static inline bool mlx5_sf_supported (const struct mlx5_core_dev *dev) +{ +return MLX5_CAP_GEN (dev, sf); +} + +static inline u16 mlx5_sf_max_ports (const struct mlx5_core_dev *dev) +{ +return mlx5_sf_supported (dev) ?1 <<MLX5_CAP_GEN (dev, log_max_sf) :0 ; +} + +#else + +static inline bool mlx5_sf_supported (const struct mlx5_core_dev *dev) +{ +return false; +} + +static inline u16 mlx5_sf_max_ports (const struct mlx5_core_dev *dev) +{ +return 0 ; +} + +#endif + +#endif
drivers/net/ethernet/mellanox/mlx5/core/vport.c +2 -1
@@ -36,6 +36,7 @@#include <linux/mlx5/vport.h> #include <linux/mlx5/eswitch.h> #include "mlx5_core.h" +#include "sf/sf.h" /* Mutex to hold while enabling or disabling RoCE */ static DEFINE_MUTEX (mlx5_roce_en_lock);
@@ -1160,6 +1161,6 @@ EXPORT_SYMBOL_GPL(mlx5_query_nic_system_image_guid);*/ u16 mlx5_eswitch_get_total_vports (const struct mlx5_core_dev *dev) { -return MLX5_SPECIAL_VPORTS (dev) +mlx5_core_max_vfs (dev); +return MLX5_SPECIAL_VPORTS (dev) +mlx5_core_max_vfs (dev) +mlx5_sf_max_ports (dev); }EXPORT_SYMBOL_GPL (mlx5_eswitch_get_total_vports);
Signature · 1 line
2.26.2
PATCH 10/13 net/mlx5: E-switch, Add eswitch helpers for SF vport
In reply to Parav PanditAdd helpers to enable/disable eswitch port, register its devlink port and load its representor.
--- .../mellanox/mlx5/core/esw/devlink_port.c | 41 +++++++++++++++++++ .../net/ethernet/mellanox/mlx5/core/eswitch.c | 10 +++++ .../net/ethernet/mellanox/mlx5/core/eswitch.h | 15 +++++++ .../mellanox/mlx5/core/eswitch_offloads.c | 36 +++++++++++++++- 4 files changed, 100 insertions(+), 2 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/esw/devlink_port.c +41 -0
@@ -122,3 +122,44 @@ struct devlink_port *mlx5_esw_offloads_devlink_port(struct mlx5_eswitch *esw, u1 vport =mlx5_eswitch_get_vport (esw, vport_num);return IS_ERR (vport) ?ERR_CAST (vport) : vport->dl_port; } + +int mlx5_esw_devlink_sf_port_register (struct mlx5_eswitch *esw,struct devlink_port *dl_port, +u16 vport_num,u32 sfnum) +{ +struct mlx5_core_dev *dev = esw->dev; +struct netdev_phys_item_id ppid = {}; +unsigned int dl_port_index; +struct mlx5_vport *vport; +struct devlink *devlink; +u16 pfnum; +int err; + + vport =mlx5_eswitch_get_vport (esw, vport_num); +if (IS_ERR (vport)) +return PTR_ERR (vport); + + pfnum =PCI_FUNC (dev->pdev->devfn); +mlx5_esw_get_port_parent_id (dev, &ppid); +memcpy (dl_port->attrs.switch_id.id, &ppid.id[0 ], ppid.id_len); + dl_port->attrs.switch_id.id_len = ppid.id_len; +devlink_port_attrs_pci_sf_set (dl_port,0 , pfnum, sfnum, false); + devlink =priv_to_devlink (dev); + dl_port_index =mlx5_esw_vport_to_devlink_port_index (dev, vport_num); + err =devlink_port_register (devlink, dl_port, dl_port_index); +if (err) +return err; + + vport->dl_port = dl_port; +return 0 ; +} + +void mlx5_esw_devlink_sf_port_unregister (struct mlx5_eswitch *esw,u16 vport_num) +{ +struct mlx5_vport *vport; + + vport =mlx5_eswitch_get_vport (esw, vport_num); +if (IS_ERR (vport)) +return ; +devlink_port_unregister (vport->dl_port); + vport->dl_port =NULL ; +}
drivers/net/ethernet/mellanox/mlx5/core/eswitch.c +10 -0
@@ -1342,6 +1342,16 @@ static void esw_disable_vport(struct mlx5_eswitch *esw, u16 vport_num)mutex_unlock (&esw->state_lock); } +int mlx5_esw_vport_enable (struct mlx5_eswitch *esw,u16 vport_num) +{ +return esw_enable_vport (esw, vport_num,MLX5_VPORT_UC_ADDR_CHANGE ); +} + +void mlx5_esw_vport_disable (struct mlx5_eswitch *esw,u16 vport_num) +{ +esw_disable_vport (esw, vport_num); +} +static int eswitch_vport_event (struct notifier_block *nb,unsigned long type,void *data) {
drivers/net/ethernet/mellanox/mlx5/core/eswitch.h +15 -0
@@ -711,6 +711,9 @@ esw_get_max_restore_tag(struct mlx5_eswitch *esw);int esw_offloads_load_rep (struct mlx5_eswitch *esw,u16 vport_num);void esw_offloads_unload_rep (struct mlx5_eswitch *esw,u16 vport_num); +int mlx5_esw_offloads_rep_load (struct mlx5_eswitch *esw,u16 vport_num); +void mlx5_esw_offloads_rep_unload (struct mlx5_eswitch *esw,u16 vport_num); +int mlx5_eswitch_load_vport (struct mlx5_eswitch *esw,u16 vport_num,enum mlx5_eswitch_vport_event enabled_events);void mlx5_eswitch_unload_vport (struct mlx5_eswitch *esw,u16 vport_num);
@@ -722,6 +725,18 @@ void mlx5_eswitch_unload_vf_vports(struct mlx5_eswitch *esw, u16 num_vfs);int mlx5_esw_offloads_devlink_port_register (struct mlx5_eswitch *esw,u16 vport_num);void mlx5_esw_offloads_devlink_port_unregister (struct mlx5_eswitch *esw,u16 vport_num);struct devlink_port *mlx5_esw_offloads_devlink_port (struct mlx5_eswitch *esw,u16 vport_num); + +int mlx5_esw_devlink_sf_port_register (struct mlx5_eswitch *esw,struct devlink_port *dl_port, +u16 vport_num,u32 sfnum); +void mlx5_esw_devlink_sf_port_unregister (struct mlx5_eswitch *esw,u16 vport_num); + +int mlx5_esw_vport_enable (struct mlx5_eswitch *esw,u16 vport_num); +void mlx5_esw_vport_disable (struct mlx5_eswitch *esw,u16 vport_num); + +int mlx5_esw_offloads_sf_vport_enable (struct mlx5_eswitch *esw,struct devlink_port *dl_port, +u16 vport_num,u32 sfnum); +void mlx5_esw_offloads_sf_vport_disable (struct mlx5_eswitch *esw,u16 vport_num); +#else /* CONFIG_MLX5_ESWITCH */ /* eswitch API stubs */ static inline int mlx5_eswitch_init (struct mlx5_core_dev *dev) {return 0 ; }
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +34 -2
@@ -1834,7 +1834,7 @@ static void __unload_reps_all_vport(struct mlx5_eswitch *esw, u8 rep_type)__esw_offloads_unload_rep (esw, rep, rep_type); } -static int mlx5_esw_offloads_rep_load (struct mlx5_eswitch *esw,u16 vport_num) +int mlx5_esw_offloads_rep_load (struct mlx5_eswitch *esw,u16 vport_num) {struct mlx5_eswitch_rep *rep;int rep_type;
@@ -1858,7 +1858,7 @@ static int mlx5_esw_offloads_rep_load(struct mlx5_eswitch *esw, u16 vport_num)return err; } -static void mlx5_esw_offloads_rep_unload (struct mlx5_eswitch *esw,u16 vport_num) +void mlx5_esw_offloads_rep_unload (struct mlx5_eswitch *esw,u16 vport_num) {struct mlx5_eswitch_rep *rep;int rep_type;
@@ -2842,3 +2842,35 @@ u32 mlx5_eswitch_get_vport_metadata_for_match(struct mlx5_eswitch *esw,return vport->metadata << (32 -ESW_SOURCE_PORT_METADATA_BITS ); }EXPORT_SYMBOL (mlx5_eswitch_get_vport_metadata_for_match); + +int mlx5_esw_offloads_sf_vport_enable (struct mlx5_eswitch *esw,struct devlink_port *dl_port, +u16 vport_num,u32 sfnum) +{ +int err; + + err =mlx5_esw_vport_enable (esw, vport_num); +if (err) +return err; + + err =mlx5_esw_devlink_sf_port_register (esw, dl_port, vport_num, sfnum); +if (err) + goto devlink_err; + + err =mlx5_esw_offloads_rep_load (esw, vport_num); +if (err) + goto rep_err; +return 0 ; + +rep_err: +mlx5_esw_devlink_sf_port_unregister (esw, vport_num); +devlink_err: +mlx5_esw_vport_disable (esw, vport_num); +return err; +} + +void mlx5_esw_offloads_sf_vport_disable (struct mlx5_eswitch *esw,u16 vport_num) +{ +mlx5_esw_offloads_rep_unload (esw, vport_num); +mlx5_esw_devlink_sf_port_unregister (esw, vport_num); +mlx5_esw_vport_disable (esw, vport_num); +}
Signature · 1 line
2.26.2
PATCH 11/13 net/mlx5: SF, Add SF configuration hardware commands
In reply to Parav PanditFrom: Vu Pham <[email protected]>
Add command helpers to access SF port and function in device.
Enable SF HCA port capability when such configuration is enabled and supported in a device.
Use them in subsequent patches.
--- .../net/ethernet/mellanox/mlx5/core/Makefile | 5 ++ drivers/net/ethernet/mellanox/mlx5/core/cmd.c | 4 ++ .../net/ethernet/mellanox/mlx5/core/main.c | 5 ++ .../net/ethernet/mellanox/mlx5/core/sf/cmd.c | 48 +++++++++++++++++++ .../net/ethernet/mellanox/mlx5/core/sf/priv.h | 14 ++++++ 5 files changed, 76 insertions(+) create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/sf/cmd.c create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/sf/priv.h
drivers/net/ethernet/mellanox/mlx5/core/Makefile +5 -0
@@ -89,3 +89,8 @@ mlx5_core-$(CONFIG_MLX5_SW_STEERING) += steering/dr_domain.o steering/dr_table.o # SF device # mlx5_core-$(CONFIG_MLX5_SF) += sf/dev/dev.o sf/dev/driver.o + +# +# SF manager +# +mlx5_core-$(CONFIG_MLX5_SF_MANAGER) += sf/cmd.o
drivers/net/ethernet/mellanox/mlx5/core/cmd.c +4 -0
@@ -333,6 +333,7 @@ static int mlx5_internal_err_ret_value(struct mlx5_core_dev *dev, u16 op,case MLX5_CMD_OP_DEALLOC_MEMIC :case MLX5_CMD_OP_PAGE_FAULT_RESUME :case MLX5_CMD_OP_QUERY_ESW_FUNCTIONS : +case MLX5_CMD_OP_DEALLOC_SF :return MLX5_CMD_STAT_OK ;case MLX5_CMD_OP_QUERY_HCA_CAP :
@@ -464,6 +465,7 @@ static int mlx5_internal_err_ret_value(struct mlx5_core_dev *dev, u16 op,case MLX5_CMD_OP_ALLOC_MEMIC :case MLX5_CMD_OP_MODIFY_XRQ :case MLX5_CMD_OP_RELEASE_XRQ_ERROR : +case MLX5_CMD_OP_ALLOC_SF : *status =MLX5_DRIVER_STATUS_ABORTED ; *synd =MLX5_DRIVER_SYND ;return -EIO ;
@@ -657,6 +659,8 @@ const char *mlx5_command_str(int command)MLX5_COMMAND_STR_CASE (DESTROY_UMEM );MLX5_COMMAND_STR_CASE (RELEASE_XRQ_ERROR );MLX5_COMMAND_STR_CASE (MODIFY_XRQ ); +MLX5_COMMAND_STR_CASE (ALLOC_SF ); +MLX5_COMMAND_STR_CASE (DEALLOC_SF );default :return "unknown command opcode" ; } }
drivers/net/ethernet/mellanox/mlx5/core/main.c +5 -0
@@ -567,6 +567,11 @@ static int handle_hca_cap(struct mlx5_core_dev *dev, void *set_ctx)if (MLX5_CAP_GEN_MAX (dev, mkey_by_name))MLX5_SET (cmd_hca_cap, set_hca_cap, mkey_by_name,1 ); +#ifdef CONFIG_MLX5_SF_MANAGER +if (MLX5_CAP_GEN_MAX (dev, sf) &&MLX5_ESWITCH_MANAGER (dev)) +MLX5_SET (cmd_hca_cap, set_hca_cap, sf,1 ); +#endif +return set_caps (dev, set_ctx,MLX5_SET_HCA_CAP_OP_MOD_GENERAL_DEVICE ); }
drivers/net/ethernet/mellanox/mlx5/core/sf/cmd.c new file +48 -0
48 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/sf/priv.h new file +14 -0
@@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */ +/* Copyright (c) 2020 Mellanox Technologies Ltd */ + +#ifndef __MLX5_SF_PRIV_H__ +#define __MLX5_SF_PRIV_H__ + +#include <linux/mlx5/driver.h> + +int mlx5_cmd_alloc_sf (struct mlx5_core_dev *dev,u16 function_id); +int mlx5_cmd_dealloc_sf (struct mlx5_core_dev *dev,u16 function_id); +int mlx5_cmd_sf_enable_hca (struct mlx5_core_dev *dev,u16 func_id); +int mlx5_cmd_sf_disable_hca (struct mlx5_core_dev *dev,u16 func_id); + +#endif
Signature · 1 line
2.26.2
PATCH 12/13 net/mlx5: SF, Add port add delete functionality
In reply to Parav PanditTo handle SF port management outside of the eswitch as independent software layer, introduce eswitch notifier APIs so that upper layer who wish to support sf port management in switchdev mode can perform its task whenever eswitch mode is set to switchdev or before eswitch is disabled.
(a) Initialize sf port table on such eswitch event.
(b) Add SF port add and delete functionality in switchdev mode.
(c) Destroy all SF ports when eswitch is disabled.
(d) Expose SF port add and delete to user via devlink commands.
$ devlink dev eswitch set pci/0000:06:00.0 mode switchdev
$ devlink port show pci/0000:06:00.0/65535: type eth netdev ens2f0np0 flavour physical port 0 splittable false
$ devlink port add pci/0000:06:00.0 flavour pcisf pfnum 0 sfnum 88
$ devlink port show ens2f0npf0sf88
pci/0000:06:00.0/32768: type eth netdev ens2f0npf0sf88 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false
function:
hw_addr 00:00:00:00:88:88 state inactive opstate detached$ devlink port show ens2f0npf0sf88 -jp
{
"port": {
"pci/0000:06:00.0/32768": {
"type": "eth",
"netdev": "ens2f0npf0sf88",
"flavour": "pcisf",
"controller": 0,
"pfnum": 0,
"sfnum": 88,
"external": false,
"splittable": false,
"function": {
"hw_addr": "00:00:00:00:88:88",
"state": "inactive",
"opstate": "detached"
}
}
}
}--- .../net/ethernet/mellanox/mlx5/core/Makefile | 2 +- .../net/ethernet/mellanox/mlx5/core/devlink.c | 5 + .../net/ethernet/mellanox/mlx5/core/eswitch.c | 25 ++ .../net/ethernet/mellanox/mlx5/core/eswitch.h | 12 + .../net/ethernet/mellanox/mlx5/core/main.c | 9 + .../net/ethernet/mellanox/mlx5/core/sf/sf.c | 370 ++++++++++++++++++ .../net/ethernet/mellanox/mlx5/core/sf/sf.h | 17 + include/linux/mlx5/driver.h | 4 + 8 files changed, 443 insertions(+), 1 deletion(-) create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/sf/sf.c
drivers/net/ethernet/mellanox/mlx5/core/Makefile +1 -1
@@ -93,4 +93,4 @@ mlx5_core-$(CONFIG_MLX5_SF) += sf/dev/dev.o sf/dev/driver.o # # SF manager # -mlx5_core-$(CONFIG_MLX5_SF_MANAGER) += sf/cmd.o +mlx5_core-$(CONFIG_MLX5_SF_MANAGER) += sf/cmd.o sf/sf.o
drivers/net/ethernet/mellanox/mlx5/core/devlink.c +5 -0
@@ -7,6 +7,7 @@#include "fw_reset.h" #include "fs_core.h" #include "eswitch.h" +#include "sf/sf.h" static int mlx5_devlink_flash_update(struct devlink *devlink, struct devlink_flash_update_params *params,
@@ -187,6 +188,10 @@ static const struct devlink_ops mlx5_devlink_ops = { .eswitch_encap_mode_get = mlx5_devlink_eswitch_encap_mode_get, .port_function_hw_addr_get = mlx5_devlink_port_function_hw_addr_get, .port_function_hw_addr_set = mlx5_devlink_port_function_hw_addr_set, +#endif +#ifdef CONFIG_MLX5_SF_MANAGER + .port_new = mlx5_devlink_sf_port_new, + .port_del = mlx5_devlink_sf_port_del, #endif .flash_update = mlx5_devlink_flash_update, .info_get = mlx5_devlink_info_get,
drivers/net/ethernet/mellanox/mlx5/core/eswitch.c +25 -0
@@ -1585,6 +1585,15 @@ mlx5_eswitch_update_num_of_vfs(struct mlx5_eswitch *esw, int num_vfs)kvfree (out); } +static void mlx5_esw_mode_change_notify (struct mlx5_eswitch *esw,u16 mode) +{ +struct mlx5_esw_event_info info = {}; + + info.new_mode = mode; + +blocking_notifier_call_chain (&esw->n_head,0 , &info); +} +/** * mlx5_eswitch_enable_locked - Enable eswitch * @esw: Pointer to eswitch
@@ -1645,6 +1654,8 @@ int mlx5_eswitch_enable_locked(struct mlx5_eswitch *esw, int mode, int num_vfs) mode ==MLX5_ESWITCH_LEGACY ?"LEGACY" :"OFFLOADS" , esw->esw_funcs.num_vfs, esw->enabled_vports); +mlx5_esw_mode_change_notify (esw, mode); +return 0 ; abort:
@@ -1701,6 +1712,11 @@ void mlx5_eswitch_disable_locked(struct mlx5_eswitch *esw, bool clear_vf) esw->mode ==MLX5_ESWITCH_LEGACY ?"LEGACY" :"OFFLOADS" , esw->esw_funcs.num_vfs, esw->enabled_vports); +/* Notify eswitch users that it is exiting from current mode. +* So that it can do necessary cleanup before the eswitch is disabled. +*/ +mlx5_esw_mode_change_notify (esw,MLX5_ESWITCH_NONE ); +mlx5_eswitch_event_handlers_unregister (esw);if (esw->mode ==MLX5_ESWITCH_LEGACY )
@@ -1801,6 +1817,7 @@ int mlx5_eswitch_init(struct mlx5_core_dev *dev) esw->offloads.inline_mode =MLX5_INLINE_MODE_NONE ; dev->priv.eswitch = esw; +BLOCKING_INIT_NOTIFIER_HEAD (&esw->n_head);return 0 ; abort:if (esw->work_queue)
@@ -2493,4 +2510,12 @@ bool mlx5_esw_multipath_prereq(struct mlx5_core_dev *dev0, dev1->priv.eswitch->mode == MLX5_ESWITCH_OFFLOADS); } +int mlx5_esw_event_notifier_register (struct mlx5_eswitch *esw,struct notifier_block *nb) +{ +return blocking_notifier_chain_register (&esw->n_head, nb); +} +void mlx5_esw_event_notifier_unregister (struct mlx5_eswitch *esw,struct notifier_block *nb) +{ +blocking_notifier_chain_unregister (&esw->n_head, nb); +}
drivers/net/ethernet/mellanox/mlx5/core/eswitch.h +12 -0
@@ -278,6 +278,7 @@ struct mlx5_eswitch {struct {u32 large_group_num; } params; +struct blocking_notifier_head n_head; };void esw_offloads_disable (struct mlx5_eswitch *esw);
@@ -737,6 +738,17 @@ int mlx5_esw_offloads_sf_vport_enable(struct mlx5_eswitch *esw, struct devlink_p u16 vport_num, u32 sfnum);void mlx5_esw_offloads_sf_vport_disable (struct mlx5_eswitch *esw,u16 vport_num); +/** +* mlx5_esw_event_info - Indicates eswitch mode changed/changing. +* +* @new_mode: New mode of eswitch. +*/ +struct mlx5_esw_event_info { +u16 new_mode; +}; + +int mlx5_esw_event_notifier_register (struct mlx5_eswitch *esw,struct notifier_block *n); +void mlx5_esw_event_notifier_unregister (struct mlx5_eswitch *esw,struct notifier_block *n);#else /* CONFIG_MLX5_ESWITCH */ /* eswitch API stubs */ static inline int mlx5_eswitch_init (struct mlx5_core_dev *dev) {return 0 ; }
drivers/net/ethernet/mellanox/mlx5/core/main.c +9 -0
@@ -889,6 +889,12 @@ static int mlx5_init_once(struct mlx5_core_dev *dev) goto err_eswitch_cleanup; } + err =mlx5_sf_table_init (dev); +if (err) { +mlx5_core_err (dev,"Failed to init SF table %d\n" , err); + goto err_sf_table_cleanup; + } + err =mlx5_sf_dev_table_init (dev);if (err) {mlx5_core_err (dev,"Failed to init SF device table %d\n" , err);
@@ -906,6 +912,8 @@ static int mlx5_init_once(struct mlx5_core_dev *dev)return 0 ; err_sf_dev_cleanup: +mlx5_sf_table_cleanup (dev); +err_sf_table_cleanup:mlx5_fpga_cleanup (dev); err_eswitch_cleanup:mlx5_eswitch_cleanup (dev->priv.eswitch);
@@ -939,6 +947,7 @@ static void mlx5_cleanup_once(struct mlx5_core_dev *dev)mlx5_fw_tracer_destroy (dev->tracer);mlx5_dm_cleanup (dev);mlx5_sf_dev_table_cleanup (dev); +mlx5_sf_table_cleanup (dev);mlx5_fpga_cleanup (dev);mlx5_eswitch_cleanup (dev->priv.eswitch);mlx5_sriov_cleanup (dev);
drivers/net/ethernet/mellanox/mlx5/core/sf/sf.c new file +370 -0
370 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/sf/sf.h +17 -0
@@ -18,6 +18,14 @@ static inline u16 mlx5_sf_max_ports(const struct mlx5_core_dev *dev) return mlx5_sf_supported(dev) ? 1 << MLX5_CAP_GEN(dev, log_max_sf) : 0; } +int mlx5_sf_table_init (struct mlx5_core_dev *dev); +void mlx5_sf_table_cleanup (struct mlx5_core_dev *dev); + +int mlx5_devlink_sf_port_new (struct devlink *devlink,const struct devlink_port_new_attrs *add_attr, +struct netlink_ext_ack *extack); +int mlx5_devlink_sf_port_del (struct devlink *devlink,unsigned int port_index, +struct netlink_ext_ack *extack); +#else static inline bool mlx5_sf_supported (const struct mlx5_core_dev *dev)
@@ -30,6 +38,15 @@ static inline u16 mlx5_sf_max_ports(const struct mlx5_core_dev *dev)return 0 ; } +static inline int mlx5_sf_table_init (struct mlx5_core_dev *dev) +{ +return 0 ; +} + +static inline void mlx5_sf_table_cleanup (struct mlx5_core_dev *dev) +{ +} +#endif #endif
include/linux/mlx5/driver.h +4 -0
@@ -509,6 +509,7 @@ struct mlx5_fw_reset;struct mlx5_eq_table ;struct mlx5_irq_table ;struct mlx5_sf_dev_table ; +struct mlx5_sf_table ;struct mlx5_rate_limit {u32 rate;
@@ -609,6 +610,9 @@ struct mlx5_priv {struct mlx5_sf_dev_table *sf_dev_table;struct mlx5_core_dev *parent_mdev;#endif +#ifdef CONFIG_MLX5_SF_MANAGER +struct mlx5_sf_table *sf_table; +#endif };enum mlx5_device_state {
Signature · 1 line
2.26.2
PATCH 13/13 net/mlx5: SF, Port function state change support
In reply to Parav PanditSupport changing the state of the SF port's function through devlink. When activating the SF port's function, enable the hca in the device followed by adding its auxiliary device.
When deactivating the SF port's function, delete its auxiliary device followed by disabling the HCA.
Port function attributes get/set callbacks are invoked with devlink instance lock held. Such callbacks need to synchronize with sf port table getting disabled. These callbacks while operating on the devlink port, synchronize with table disable context by holding table refcount.
$ devlink dev eswitch set pci/0000:06:00.0 mode switchdev
$ devlink port show pci/0000:06:00.0/65535: type eth netdev ens2f0np0 flavour physical port 0 splittable false
$ devlink port add pci/0000:06:00.0 flavour pcisf pfnum 0 sfnum 88
$ devlink port show ens2f0npf0sf88
pci/0000:06:00.0/32768: type eth netdev ens2f0npf0sf88 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false
function:
hw_addr 00:00:00:00:88:88 state inactive opstate detached$ devlink port function set pci/0000:06:00.0/32768 hw_addr 00:00:00:00:88:88 state active
$ devlink port show ens2f0npf0sf88 -jp
{
"port": {
"pci/0000:06:00.0/32768": {
"type": "eth",
"netdev": "ens2f0npf0sf88",
"flavour": "pcisf",
"controller": 0,
"pfnum": 0,
"sfnum": 88,
"external": false,
"splittable": false,
"function": {
"hw_addr": "00:00:00:00:88:88",
"state": "active",
"opstate": "attached"
}
}
}
}On port function activation, an auxiliary device is created in below example.
$ devlink dev show devlink dev show auxiliary/mlx5_core.sf.0
$ devlink port show auxiliary/mlx5_core.sf.0/1 auxiliary/mlx5_core.sf.0/1: type eth netdev p0sf88 flavour virtual port 0 splittable false
--- .../net/ethernet/mellanox/mlx5/core/devlink.c | 2 + .../net/ethernet/mellanox/mlx5/core/sf/sf.c | 128 ++++++++++++++++++ .../net/ethernet/mellanox/mlx5/core/sf/sf.h | 7 + 3 files changed, 137 insertions(+)
drivers/net/ethernet/mellanox/mlx5/core/devlink.c +2 -0
@@ -192,6 +192,8 @@ static const struct devlink_ops mlx5_devlink_ops = {#ifdef CONFIG_MLX5_SF_MANAGER .port_new = mlx5_devlink_sf_port_new, .port_del = mlx5_devlink_sf_port_del, + .port_function_state_get = mlx5_devlink_sf_port_fn_state_get, + .port_function_state_set = mlx5_devlink_sf_port_fn_state_set, #endif .flash_update = mlx5_devlink_flash_update, .info_get = mlx5_devlink_info_get,
drivers/net/ethernet/mellanox/mlx5/core/sf/sf.c +128 -0
164 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/sf/sf.h +7 -0
@@ -25,6 +25,13 @@ int mlx5_devlink_sf_port_new(struct devlink *devlink, const struct devlink_port_ struct netlink_ext_ack *extack);int mlx5_devlink_sf_port_del (struct devlink *devlink,unsigned int port_index,struct netlink_ext_ack *extack); +int mlx5_devlink_sf_port_fn_state_get (struct devlink *devlink,struct devlink_port *dl_port, +enum devlink_port_function_state *state, +enum devlink_port_function_opstate *opstate, +struct netlink_ext_ack *extack); +int mlx5_devlink_sf_port_fn_state_set (struct devlink *devlink,struct devlink_port *dl_port, +enum devlink_port_function_state state, +struct netlink_ext_ack *extack);#else
Signature · 1 line
2.26.2
Jakub Kicinski
maintainerQuoted 11 lines · On Thu, 12 Nov 2020 21:24:10 +0200 Parav Pandit wrote:
This series introduces support for mlx5 subfunction (SF). A subfunction is a portion of a PCI device that supports multiple classes of devices such as netdev, RDMA and more.
This patchset is based on Leon's series [3].
It is a third user of proposed auxiliary bus [4].Subfunction support is discussed in detail in RFC [1] and [2]. RFC [1] and extension [2] describes requirements, design, and proposed plumbing using devlink, auxiliary bus and sysfs for systemd/udev support.
So we're going to have two ways of adding subdevs? Via devlink and via the new vdpa netlink thing?
Question number two - is this supposed to be ready to be applied to net-next? It seems there is a conflict.
Also could you please wrap your code at 80 chars?
Thanks.
Quoted 14 lines · Jakub Kicinski
Quoted 4 lines · On Thu, 12 Nov 2020 21:24:10 +0200 Parav Pandit wrote:
This series introduces support for mlx5 subfunction (SF). A subfunction is a portion of a PCI device that supports multiple classes of devices such as netdev, RDMA and more.
Quoted 8 lines
This patchset is based on Leon's series [3].
It is a third user of proposed auxiliary bus [4].Subfunction support is discussed in detail in RFC [1] and [2]. RFC [1] and extension [2] describes requirements, design, and proposed
plumbing using devlink, auxiliary bus and sysfs for systemd/udev support.
Quoted 4 lines
So we're going to have two ways of adding subdevs? Via devlink and via
the new vdpa netlink thing?
Via devlink you add the Sub-function bus device - think of it as spawning a new VF - but has no actual characteristics (netdev/vpda/rdma) "yet" until user admin decides to load an interface on it via aux sysfs.
Basically devlink adds a new eswitch port (the SF port) and loading the drivers and the interfaces is done via the auxbus subsystem only after the SF is spawned by FW.
Quoted 3 lines
Question number two - is this supposed to be ready to be applied to net-next? It seems there is a conflict.
This series requires other mlx5 and auxbus infrastructure dependencies that was already submitted by leon 2-3 weeks ago and pending Greg's review, once finalized it will be merged into mlx5-next, then I will ask you to pull mlx5-next and only after, you can apply this series cleanly to net-next, sorry for the mess but we had to move forward and show how auxdev subsystem is being actually used.
Leon's series:
https://patchwork.ozlabs.org/project/netdev/cover/[email protected]/
Quoted 2 lines
Also could you please wrap your code at 80 chars?
I prefer no to do this in mlx5, in mlx5 we follow a 95 chars rule. But if you insist :) ..
Thanks,
Saeed.
Jakub Kicinski
maintainerQuoted 15 lines · On Mon, 16 Nov 2020 16:06:02 -0800 Saeed Mahameed wrote:
Quoted 14 lines · Jakub Kicinski
Quoted 12 lines · On Thu, 12 Nov 2020 21:24:10 +0200 Parav Pandit wrote:
This series introduces support for mlx5 subfunction (SF). A subfunction is a portion of a PCI device that supports multiple classes of devices such as netdev, RDMA and more.
This patchset is based on Leon's series [3].
It is a third user of proposed auxiliary bus [4].Subfunction support is discussed in detail in RFC [1] and [2]. RFC [1] and extension [2] describes requirements, design, and proposed
plumbing using devlink, auxiliary bus and sysfs for systemd/udev support.
Quoted 7 lines
Quoted 2 lines
So we're going to have two ways of adding subdevs? Via devlink and via the new vdpa netlink thing?
Via devlink you add the Sub-function bus device - think of it as spawning a new VF - but has no actual characteristics (netdev/vpda/rdma) "yet" until user admin decides to load an interface on it via aux sysfs.
By which you mean it doesn't get probed or the device type is not set (IOW it can still become a block device or netdev depending on the vdpa request)?
Quoted 3 lines
Basically devlink adds a new eswitch port (the SF port) and loading the drivers and the interfaces is done via the auxbus subsystem only after the SF is spawned by FW.
But why?
Is this for the SmartNIC / bare metal case? The flow for spawning on the local host gets highly convoluted.
Quoted 4 lines
Quoted 1 line
Also could you please wrap your code at 80 chars?
I prefer no to do this in mlx5, in mlx5 we follow a 95 chars rule. But if you insist :) ..
Oh yeah, I meant the devlink patches!
Quoted 12 lines
From: Jakub Kicinski <[email protected]>
Sent: Tuesday, November 17, 2020 7:28 AMQuoted 7 lines · On Mon, 16 Nov 2020 16:06:02 -0800 Saeed Mahameed wrote:
Quoted 7 lines
Quoted 4 lines
Subfunction support is discussed in detail in RFC [1] and [2]. RFC [1] and extension [2] describes requirements, design, and proposed plumbing using devlink, auxiliary bus and sysfs for systemd/udev support.
So we're going to have two ways of adding subdevs? Via devlink and via the new vdpa netlink thing?
Nop.
Subfunctions (subdevs) are added only one way,
i.e. devlink port as settled in RFC [1].
Just to refresh all our memory, we discussed and settled on the flow in [2]; RFC [1] followed this discussion.
vdpa tool of [3] can add one or more vdpa device(s) on top of already spawned PF, VF, SF device.
Quoted 9 lines
Quoted 5 lines
Via devlink you add the Sub-function bus device - think of it as spawning a new VF - but has no actual characteristics (netdev/vpda/rdma) "yet" until user admin decides to load an interface on it via aux sysfs.
By which you mean it doesn't get probed or the device type is not set (IOW it can still become a block device or netdev depending on the vdpa request)?
Quoted 9 lines
Quoted 3 lines
Basically devlink adds a new eswitch port (the SF port) and loading the drivers and the interfaces is done via the auxbus subsystem only after the SF is spawned by FW.
But why?
Is this for the SmartNIC / bare metal case? The flow for spawning on the local host gets highly convoluted.
The flow of spawning for (a) local host or (b) for external host controller from smartnic is same.
$ devlink port add.. [..] Followed by $ devlink port function set state...
Only change would be to specify the destination where to spawn it. (controller number, pf, sf num etc) Please refer to the detailed examples in individual patch.
Patch 12 and 13 mostly covers the complete view.
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Also could you please wrap your code at 80 chars?
I prefer no to do this in mlx5, in mlx5 we follow a 95 chars rule. But if you insist :) ..
Oh yeah, I meant the devlink patches!
May I ask why?
Past few devlink patches [4] followed 100 chars rule. When did we revert back to 80?
If so, any pointers to the thread for 80? checkpatch.pl with --strict mode didn't complain me when I prepared the patches.
[1] https://lore.kernel.org/netdev/20200519092258.GF4655@nanopsycho/
[2] https://lore.kernel.org/netdev/[email protected]/
[3] https://lists.linuxfoundation.org/pipermail/virtualization/2020-November/050623.html
[4] commits dc64cc7c6310, 77069ba2e3ad, a1e8ae907c8d, 2a916ecc4056, ba356c90985d
Jakub Kicinski
maintainerQuoted 12 lines · On Tue, 17 Nov 2020 04:08:57 +0000 Parav Pandit wrote:
Quoted 8 lines · On Mon, 16 Nov 2020 16:06:02 -0800 Saeed Mahameed wrote:
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Subfunction support is discussed in detail in RFC [1] and [2]. RFC [1] and extension [2] describes requirements, design, and proposed plumbing using devlink, auxiliary bus and sysfs for systemd/udev support.
So we're going to have two ways of adding subdevs? Via devlink and via the new vdpa netlink thing?
Nop.
Subfunctions (subdevs) are added only one way,
i.e. devlink port as settled in RFC [1].
Quoted 5 lines
Just to refresh all our memory, we discussed and settled on the flow in [2]; RFC [1] followed this discussion.
vdpa tool of [3] can add one or more vdpa device(s) on top of already spawned PF, VF, SF device.
Nack for the networking part of that. It'd basically be VMDq.
Quoted 25 lines
Quoted 8 lines
Quoted 4 lines
Via devlink you add the Sub-function bus device - think of it as spawning a new VF - but has no actual characteristics (netdev/vpda/rdma) "yet" until user admin decides to load an interface on it via aux sysfs.
By which you mean it doesn't get probed or the device type is not set (IOW it can still become a block device or netdev depending on the vdpa request)?
Quoted 8 lines
Quoted 3 lines
Basically devlink adds a new eswitch port (the SF port) and loading the drivers and the interfaces is done via the auxbus subsystem only after the SF is spawned by FW.
But why?
Is this for the SmartNIC / bare metal case? The flow for spawning on the local host gets highly convoluted.
The flow of spawning for (a) local host or (b) for external host controller from smartnic is same.
$ devlink port add.. [..] Followed by $ devlink port function set state...
Quoted 4 lines
Only change would be to specify the destination where to spawn it. (controller number, pf, sf num etc) Please refer to the detailed examples in individual patch. Patch 12 and 13 mostly covers the complete view.
Please share full examples of the workflow.
I'm asking how the vdpa API fits in with this, and you're showing me the two devlink commands we already talked about in the past.
Jason Gunthorpe
maintainerQuoted 7 lines · Jakub Kicinski
Quoted 5 lines
Just to refresh all our memory, we discussed and settled on the flow in [2]; RFC [1] followed this discussion.
vdpa tool of [3] can add one or more vdpa device(s) on top of already spawned PF, VF, SF device.
Nack for the networking part of that. It'd basically be VMDq.
What are you NAK'ing?
It is consistent with the multi-subsystem device sharing model we've had for ages now.
The physical ethernet port is shared between multiple accelerator subsystems. netdev gets its slice of traffic, so does RDMA, iSCSI, VDPA, etc.
Jason
Jakub Kicinski
maintainerQuoted 11 lines · On Tue, 17 Nov 2020 14:49:54 -0400 Jason Gunthorpe wrote:
Quoted 7 lines · Jakub Kicinski
Quoted 5 lines
Just to refresh all our memory, we discussed and settled on the flow in [2]; RFC [1] followed this discussion.
vdpa tool of [3] can add one or more vdpa device(s) on top of already spawned PF, VF, SF device.
Nack for the networking part of that. It'd basically be VMDq.
What are you NAK'ing?
Spawning multiple netdevs from one device by slicing up its queues.
Quoted 6 lines
It is consistent with the multi-subsystem device sharing model we've had for ages now.
The physical ethernet port is shared between multiple accelerator subsystems. netdev gets its slice of traffic, so does RDMA, iSCSI, VDPA, etc.
Right, devices of other subsystems are fine, I don't care.
Sorry for not being crystal clear but quite frankly IDK what else can be expected from me given the submissions have little to no context and documentation. This comes up every damn time with the SF patches, I'm tired of having to ask for a basic workflow.
Quoted 6 lines · Jakub Kicinski
Quoted 5 lines · On Tue, 17 Nov 2020 14:49:54 -0400 Jason Gunthorpe wrote:
Quoted 3 lines · Jakub Kicinski
Quoted 3 lines
Just to refresh all our memory, we discussed and settled on the flow in [2]; RFC [1] followed this discussion.
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
vdpa tool of [3] can add one or more vdpa device(s) on top of already spawned PF, VF, SF device.
Nack for the networking part of that. It'd basically be VMDq.
What are you NAK'ing?
Spawning multiple netdevs from one device by slicing up its queues.
Why do you object to that? Slicing up h/w resources for virtual what ever has been common practice for a long time.
Quoted 12 lines · David Ahern
Quoted 11 lines · Jakub Kicinski
Quoted 10 lines · On Tue, 17 Nov 2020 14:49:54 -0400 Jason Gunthorpe wrote:
Quoted 8 lines · Jakub Kicinski
Quoted 7 lines
Just to refresh all our memory, we discussed and settled on the flow
in [2]; RFC [1] followed this discussion.vdpa tool of [3] can add one or more vdpa device(s) on top of already
spawned PF, VF, SF device.
Quoted 10 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
Nack for the networking part of that. It'd basically be VMDq.
What are you NAK'ing?
Spawning multiple netdevs from one device by slicing up its queues.
Why do you object to that? Slicing up h/w resources for virtual what ever has been common practice for a long time.
We are not slicing up any queues, from our HW and FW perspective SF == VF literally, a full blown HW slice (Function), with isolated control and data plane of its own, this is very different from VMDq and more generic and secure. an SF device is exactly like a VF, doesn't steal or share any HW resources or control/data path with others. SF is basically SRIOV done right.
this series has nothing to do with netdev, if you look at the list of files Parav is touching, there is 0 change in our netdev stack :) .. all Parav is doing is adding the API to create/destroy SFs and represents the low level SF function to devlink as a device, just like a VF.
Jakub Kicinski
maintainerQuoted 29 lines · On Wed, 18 Nov 2020 21:57:57 -0800 Saeed Mahameed wrote:
Quoted 12 lines · David Ahern
Quoted 11 lines · Jakub Kicinski
Quoted 10 lines · On Tue, 17 Nov 2020 14:49:54 -0400 Jason Gunthorpe wrote:
Quoted 8 lines · Jakub Kicinski
Quoted 7 lines
Just to refresh all our memory, we discussed and settled on the flow
in [2]; RFC [1] followed this discussion.vdpa tool of [3] can add one or more vdpa device(s) on top of already
spawned PF, VF, SF device.Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
Nack for the networking part of that. It'd basically be VMDq.
What are you NAK'ing?
Spawning multiple netdevs from one device by slicing up its queues.
Why do you object to that? Slicing up h/w resources for virtual what ever has been common practice for a long time.
We are not slicing up any queues, from our HW and FW perspective SF == VF literally, a full blown HW slice (Function), with isolated control and data plane of its own, this is very different from VMDq and more generic and secure. an SF device is exactly like a VF, doesn't steal or share any HW resources or control/data path with others. SF is basically SRIOV done right.
Quoted 5 lines
this series has nothing to do with netdev, if you look at the list of files Parav is touching, there is 0 change in our netdev stack :) .. all Parav is doing is adding the API to create/destroy SFs and represents the low level SF function to devlink as a device, just like a VF.
Ack, the concern is about the vdpa, not SF.
So not really this patch set.
Quoted 7 lines · Saeed Mahameed
We are not slicing up any queues, from our HW and FW perspective SF == VF literally, a full blown HW slice (Function), with isolated control and data plane of its own, this is very different from VMDq and more generic and secure. an SF device is exactly like a VF, doesn't steal or share any HW resources or control/data path with others. SF is basically SRIOV done right.
What does that mean with respect to mac filtering and ntuple rules?
Also, Tx is fairly easy to imagine, but how does hardware know how to direct packets for the Rx path? As an example, consider 2 VMs or containers with the same destination ip both using subfunction devices. How does the nic know how to direct the ingress flows to the right queues for the subfunction?
Hi David,
Quoted 12 lines
From: David Ahern <[email protected]>
Sent: Wednesday, November 25, 2020 11:04 AMQuoted 7 lines · Saeed Mahameed
We are not slicing up any queues, from our HW and FW perspective SF == VF literally, a full blown HW slice (Function), with isolated control and data plane of its own, this is very different from VMDq and more generic and secure. an SF device is exactly like a VF, doesn't steal or share any HW resources or control/data path with others. SF is basically SRIOV done right.
Quoted 5 lines
What does that mean with respect to mac filtering and ntuple rules?
Also, Tx is fairly easy to imagine, but how does hardware know how to direct packets for the Rx path? As an example, consider 2 VMs or containers with the same destination ip both using subfunction devices.
Since both VM/containers are having same IP, it is better to place them in different L2 domains via vlan, vxlan etc.
Quoted 3 lines
How does the nic know how to direct the ingress flows to the right queues for the subfunction?
Rx steering occurs through tc filters via representor netdev of SF. Exactly same way as VF representor netdev operation.
When devlink eswitch port is created as shown in example in cover letter, and also in patch-12, it creates the representor netdevice. Below is the snippet of it.
Add a devlink port of subfunction flavour:
$ devlink port add pci/0000:06:00.0 flavour pcisf pfnum 0 sfnum 88
Configure mac address of the port function:
$ devlink port function set ens2f0npf0sf88 hw_addr 00:00:00:00:88:88
^^^^^^^^^^^^^^
This is the representor netdevice. It is created by port add command.
This name is setup by systemd/udev v245 and higher by utilizing the existing phys_port_name infrastructure already exists for PF and VF representors.Now user can add unicast rx tc rule for example,
$ tc filter add dev ens2f0np0 parent ffff: prio 1 flower dst_mac 00:00:00:00:88:88 action mirred egress redirect dev ens2f0npf0sf88
I didn't cover this tc example in cover letter, to keep it short.
But I had a one line description as below in the 'detail' section of cover-letter. Hope it helps.
- A SF supports eswitch representation and tc offload support similar to existing PF and VF representors.
Now above portion answers, how to forward the packet to subfunction.
But how to forward to the right rx queue out of multiple rxqueues?
This is done by the rss configuration done by the user, number of channels from ethtool. Just like VF and PF.
The driver defaults are similar to VF, which user can change via ethtool.
Quoted 14 lines · Parav Pandit
Hi David,
Quoted 12 lines
From: David Ahern <[email protected]>
Sent: Wednesday, November 25, 2020 11:04 AMQuoted 7 lines · Saeed Mahameed
We are not slicing up any queues, from our HW and FW perspective SF == VF literally, a full blown HW slice (Function), with isolated control and data plane of its own, this is very different from VMDq and more generic and secure. an SF device is exactly like a VF, doesn't steal or share any HW resources or control/data path with others. SF is basically SRIOV done right.
Quoted 6 lines
Quoted 5 lines
What does that mean with respect to mac filtering and ntuple rules?
Also, Tx is fairly easy to imagine, but how does hardware know how to direct packets for the Rx path? As an example, consider 2 VMs or containers with the same destination ip both using subfunction devices.
Since both VM/containers are having same IP, it is better to place them in different L2 domains via vlan, vxlan etc.
ok, so relying on <vlan, dmac> pairs.
Quoted 10 lines
Quoted 4 lines
How does the nic know how to direct the ingress flows to the right queues for the subfunction?
Rx steering occurs through tc filters via representor netdev of SF. Exactly same way as VF representor netdev operation.
When devlink eswitch port is created as shown in example in cover letter, and also in patch-12, it creates the representor netdevice. Below is the snippet of it.
Quoted 8 lines
Add a devlink port of subfunction flavour:
$ devlink port add pci/0000:06:00.0 flavour pcisf pfnum 0 sfnum 88Configure mac address of the port function:
$ devlink port function set ens2f0npf0sf88 hw_addr 00:00:00:00:88:88 ^^^^^^^^^^^^^^ This is the representor netdevice. It is created by port add command. This name is setup by systemd/udev v245 and higher by utilizing the existing phys_port_name infrastructure already exists for PF and VF representors.
hardware ensures only packets with that dmac are sent to the subfunction device.
Quoted 9 lines
Now user can add unicast rx tc rule for example,
$ tc filter add dev ens2f0np0 parent ffff: prio 1 flower dst_mac 00:00:00:00:88:88 action mirred egress redirect dev ens2f0npf0sf88I didn't cover this tc example in cover letter, to keep it short.
But I had a one line description as below in the 'detail' section of cover-letter. Hope it helps.
Quoted 9 lines
- A SF supports eswitch representation and tc offload support similar to existing PF and VF representors.
Now above portion answers, how to forward the packet to subfunction.
But how to forward to the right rx queue out of multiple rxqueues?
This is done by the rss configuration done by the user, number of channels from ethtool. Just like VF and PF.
The driver defaults are similar to VF, which user can change via ethtool.
so users can add flow steering or drop rules to SF devices.
thanks,
Jakub Kicinski
maintainerQuoted 10 lines · On Wed, 18 Nov 2020 21:35:29 -0700 David Ahern wrote:
Quoted 9 lines · Jakub Kicinski
Quoted 8 lines · On Tue, 17 Nov 2020 14:49:54 -0400 Jason Gunthorpe wrote:
Quoted 6 lines · Jakub Kicinski
Quoted 5 lines
Just to refresh all our memory, we discussed and settled on the flow in [2]; RFC [1] followed this discussion.
vdpa tool of [3] can add one or more vdpa device(s) on top of already spawned PF, VF, SF device.
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
Nack for the networking part of that. It'd basically be VMDq.
What are you NAK'ing?
Spawning multiple netdevs from one device by slicing up its queues.
Why do you object to that? Slicing up h/w resources for virtual what ever has been common practice for a long time.
My memory of the VMDq debate is hazy, let me rope in Alex into this. I believe the argument was that we should offload software constructs, not create HW-specific APIs which depend on HW availability and implementation. So the path we took was offloading macvlan.
Quoted 18 lines · Jakub Kicinski
Quoted 16 lines · On Wed, 18 Nov 2020 21:35:29 -0700 David Ahern wrote:
Quoted 14 lines · Jakub Kicinski
Quoted 11 lines · On Tue, 17 Nov 2020 14:49:54 -0400 Jason Gunthorpe wrote:
Quoted 7 lines · Jakub Kicinski
Quoted 5 lines
Just to refresh all our memory, we discussed and settled on the flow in [2]; RFC [1] followed this discussion.
vdpa tool of [3] can add one or more vdpa device(s) on top of already spawned PF, VF, SF device.
Nack for the networking part of that. It'd basically be VMDq.
What are you NAK'ing?
Spawning multiple netdevs from one device by slicing up its queues.
Quoted 7 lines
Quoted 2 lines
Why do you object to that? Slicing up h/w resources for virtual what ever has been common practice for a long time.
My memory of the VMDq debate is hazy, let me rope in Alex into this. I believe the argument was that we should offload software constructs, not create HW-specific APIs which depend on HW availability and implementation. So the path we took was offloading macvlan.
I think it somewhat depends on the type of interface we are talking about. What we were wanting to avoid was drivers spawning their own unique VMDq netdevs and each having a different way of doing it. The approach Intel went with was to use a MACVLAN offload to approach it. Although I would imagine many would argue the approach is somewhat dated and limiting since you cannot do many offloads on a MACVLAN interface.
With the VDPA case I believe there is a set of predefined virtio devices that are being emulated and presented so it isn't as if they are creating a totally new interface for this.
What I would be interested in seeing is if there are any other vendors that have reviewed this and sign off on this approach. What we don't want to see is Nivida/Mellanox do this one way, then Broadcom or Intel come along later and have yet another way of doing this. We need an interface and feature set that will work for everyone in terms of how this will look going forward.
Quoted 27 lines · Alexander Duyck
Quoted 18 lines · Jakub Kicinski
Quoted 16 lines · On Wed, 18 Nov 2020 21:35:29 -0700 David Ahern wrote:
Quoted 14 lines · Jakub Kicinski
Quoted 11 lines · On Tue, 17 Nov 2020 14:49:54 -0400 Jason Gunthorpe wrote:
Quoted 7 lines · Jakub Kicinski
Quoted 5 lines
Just to refresh all our memory, we discussed and settled on the flow in [2]; RFC [1] followed this discussion.
vdpa tool of [3] can add one or more vdpa device(s) on top of already spawned PF, VF, SF device.
Nack for the networking part of that. It'd basically be VMDq.
What are you NAK'ing?
Spawning multiple netdevs from one device by slicing up its queues.
Quoted 7 lines
Quoted 2 lines
Why do you object to that? Slicing up h/w resources for virtual what ever has been common practice for a long time.
My memory of the VMDq debate is hazy, let me rope in Alex into this. I believe the argument was that we should offload software constructs, not create HW-specific APIs which depend on HW availability and implementation. So the path we took was offloading macvlan.
Quoted 7 lines
I think it somewhat depends on the type of interface we are talking about. What we were wanting to avoid was drivers spawning their own unique VMDq netdevs and each having a different way of doing it. The approach Intel went with was to use a MACVLAN offload to approach it. Although I would imagine many would argue the approach is somewhat dated and limiting since you cannot do many offloads on a MACVLAN interface.
Yes. We talked about this at netdev 0x14 and the limitations of macvlan based offloads.
https://netdevconf.info/0x14/session.html?talk-hardware-acceleration-of-container-networking-interfaces
Subfunction seems to be a good model to expose VMDq VSI or SIOV ADI as a netdev for kernel containers. AF_XDP ZC in a container is one of the usecase this would address. Today we have to pass the entire PF/VF to a container to do AF_XDP.
Looks like the current model is to create a subfunction of a specific type on auxiliary bus, do some configuration to assign resources and then activate the subfunction.
Quoted 11 lines
With the VDPA case I believe there is a set of predefined virtio devices that are being emulated and presented so it isn't as if they are creating a totally new interface for this.
What I would be interested in seeing is if there are any other vendors that have reviewed this and sign off on this approach. What we don't want to see is Nivida/Mellanox do this one way, then Broadcom or Intel come along later and have yet another way of doing this. We need an interface and feature set that will work for everyone in terms of how this will look going forward.
Quoted 37 lines · Sridhar
Quoted 35 lines · Alexander Duyck
Quoted 32 lines · Jakub Kicinski
Quoted 16 lines · On Wed, 18 Nov 2020 21:35:29 -0700 David Ahern wrote:
Quoted 15 lines · Jakub Kicinski
Quoted 14 lines · On Tue, 17 Nov 2020 14:49:54 -0400 Jason Gunthorpe wrote:
Quoted 10 lines · Jakub Kicinski
Quoted 7 lines
Just to refresh all our memory, we discussed and
settled on the flow
in [2]; RFC [1] followed this discussion.vdpa tool of [3] can add one or more vdpa device(s) on top of already
spawned PF, VF, SF device.Nack for the networking part of that. It'd basically be VMDq.
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
What are you NAK'ing?
Spawning multiple netdevs from one device by slicing up its queues.
Why do you object to that? Slicing up h/w resources for virtual what
ever has been common practice for a long time.My memory of the VMDq debate is hazy, let me rope in Alex into this.
I believe the argument was that we should offload software constructs,
not create HW-specific APIs which depend on HW availability and implementation. So the path we took was offloading macvlan.
Quoted 3 lines
Quoted 3 lines
I think it somewhat depends on the type of interface we are talking about. What we were wanting to avoid was drivers spawning their own unique VMDq netdevs and each having a different way of doing it.
Agreed, but SF netdevs are not a VMDq netdevs, they are avaiable in the switchdev model where they correspond to a full blown port (HW domain).
Quoted 7 lines
Quoted 6 lines
The
approach Intel went with was to use a MACVLAN offload to approach it.
Although I would imagine many would argue the approach is somewhat dated and limiting since you cannot do many offloads on a MACVLAN interface.
Quoted 7 lines
Yes. We talked about this at netdev
0x14and the limitations of macvlan
based offloads.
https://netdevconf.info/0x14/session.html?talk-hardware-acceleration-of-container-networking-interfacesSubfunction seems to be a good model to expose VMDq VSI or SIOV ADI as a
Exactly, Subfunctions is the most generic model to overcome any SW model limitations e.g macvtap offload, all HW vendors are already creating netdevs on a given PF/VF .. all we need is to model the SF and all the rest is the same! most likely every thing else comes for free like in the mlx5 model where the netdev/rmda interfaces are abstracted from the underlying HW, same netdev loads on a PF/VF/SF or even an embedded function !
Quoted 5 lines
netdev for kernel containers.
AF_XDPZC in a container is one of the usecase this would address. Today we have to pass the entire PF/VF to a
container to doAF_XDP.
this will be supported out of the box for free with SFs.
Quoted 20 lines
Looks like the current model is to create a subfunction of a specific
type on auxiliary bus, do some configuration to assign resources and then activate the subfunction.Quoted 15 lines
With the VDPA case I believe there is a set of predefined virtio devices that are being emulated and presented so it isn't as if they
are creating a totally new interface for this.What I would be interested in seeing is if there are any other vendors
that have reviewed this and sign off on this approach. What we don't
want to see is Nivida/Mellanox do this one way, then Broadcom or Intel
come along later and have yet another way of doing this. We need an interface and feature set that will work for everyone in terms of how
this will look going forward.
Well, the vdpa interface was created by the virtio community and especially redhat, i am not sure mellanox were even involved in the initial development stages :-)
anyway historically speaking vDPA was originally created for DPDK, but same API applies to device drivers who can deliver the same set of queues and API while bypassing the whole DPDK stack, enters Kernel vDPA which was created to overcome some of the userspace limitations and complexity and to leverage some of the kernel great feature such as eBPF.
https://www.redhat.com/en/blog/introduction-vdpa-kernel-framework
Quoted 38 lines · Sridhar
Quoted 27 lines · Alexander Duyck
Quoted 18 lines · Jakub Kicinski
Quoted 16 lines · On Wed, 18 Nov 2020 21:35:29 -0700 David Ahern wrote:
Quoted 14 lines · Jakub Kicinski
Quoted 11 lines · On Tue, 17 Nov 2020 14:49:54 -0400 Jason Gunthorpe wrote:
Quoted 7 lines · Jakub Kicinski
Quoted 5 lines
Just to refresh all our memory, we discussed and settled on the flow in [2]; RFC [1] followed this discussion.
vdpa tool of [3] can add one or more vdpa device(s) on top of already spawned PF, VF, SF device.
Nack for the networking part of that. It'd basically be VMDq.
What are you NAK'ing?
Spawning multiple netdevs from one device by slicing up its queues.
Quoted 7 lines
Quoted 2 lines
Why do you object to that? Slicing up h/w resources for virtual what ever has been common practice for a long time.
My memory of the VMDq debate is hazy, let me rope in Alex into this. I believe the argument was that we should offload software constructs, not create HW-specific APIs which depend on HW availability and implementation. So the path we took was offloading macvlan.
Quoted 7 lines
I think it somewhat depends on the type of interface we are talking about. What we were wanting to avoid was drivers spawning their own unique VMDq netdevs and each having a different way of doing it. The approach Intel went with was to use a MACVLAN offload to approach it. Although I would imagine many would argue the approach is somewhat dated and limiting since you cannot do many offloads on a MACVLAN interface.
Quoted 12 lines
Yes. We talked about this at netdev
0x14and the limitations of macvlan based offloads.
https://netdevconf.info/0x14/session.html?talk-hardware-acceleration-of-container-networking-interfacesSubfunction seems to be a good model to expose VMDq VSI or SIOV ADI as a netdev for kernel containers.
AF_XDPZC in a container is one of the usecase this would address. Today we have to pass the entire PF/VF to a container to doAF_XDP.Looks like the current model is to create a subfunction of a specific type on auxiliary bus, do some configuration to assign resources and then activate the subfunction.
Quoted 4 lines
With the VDPA case I believe there is a set of predefined virtio devices that are being emulated and presented so it isn't as if they are creating a totally new interface for this.
vDPA doesn't have any limitation of how the devices is created or implemented. It could be predefined or created dynamically. vDPA leaves all of those to the parent device with the help of a unified management API[1]. E.g It could be a PCI device (PF or VF), sub-function or software emulated devices.
Quoted 3 lines
Quoted 3 lines
What I would be interested in seeing is if there are any other vendors that have reviewed this and sign off on this approach.
For "this approach" do you mean vDPA subfucntion? My understanding is that it's totally vendor specific, vDPA subsystem don't want to be limited by a specific type of device.
Quoted 5 lines
Quoted 5 lines
What we don't
want to see is Nivida/Mellanox do this one way, then Broadcom or Intel come along later and have yet another way of doing this. We need an interface and feature set that will work for everyone in terms of how this will look going forward.
For feature set, it would be hard to force (we can have a recommendation set of features) vendors to implement a common set of features consider they can be negotiated. So the management interface is expected to implement features like cpu clusters in order to make sure the migration compatibility, or qemu can assist for the missing feature with performance lose.
Thanks
Quoted 49 lines · Jason Wang
Quoted 47 lines · Sridhar
Quoted 27 lines · Alexander Duyck
Quoted 18 lines · Jakub Kicinski
Quoted 16 lines · On Wed, 18 Nov 2020 21:35:29 -0700 David Ahern wrote:
Quoted 14 lines · Jakub Kicinski
Quoted 11 lines · On Tue, 17 Nov 2020 14:49:54 -0400 Jason Gunthorpe wrote:
Quoted 7 lines · Jakub Kicinski
Quoted 5 lines
Just to refresh all our memory, we discussed and settled on the flow in [2]; RFC [1] followed this discussion.
vdpa tool of [3] can add one or more vdpa device(s) on top of already spawned PF, VF, SF device.
Nack for the networking part of that. It'd basically be VMDq.
What are you NAK'ing?
Spawning multiple netdevs from one device by slicing up its queues.
Quoted 7 lines
Quoted 2 lines
Why do you object to that? Slicing up h/w resources for virtual what ever has been common practice for a long time.
My memory of the VMDq debate is hazy, let me rope in Alex into this. I believe the argument was that we should offload software constructs, not create HW-specific APIs which depend on HW availability and implementation. So the path we took was offloading macvlan.
Quoted 7 lines
I think it somewhat depends on the type of interface we are talking about. What we were wanting to avoid was drivers spawning their own unique VMDq netdevs and each having a different way of doing it. The approach Intel went with was to use a MACVLAN offload to approach it. Although I would imagine many would argue the approach is somewhat dated and limiting since you cannot do many offloads on a MACVLAN interface.
Yes. We talked about this at netdev
0x14and the limitations of macvlan based offloads.
https://netdevconf.info/0x14/session.html?talk-hardware-acceleration-of-container-networking-interfacesSubfunction seems to be a good model to expose VMDq VSI or SIOV ADI as a netdev for kernel containers.
AF_XDPZC in a container is one of the usecase this would address. Today we have to pass the entire PF/VF to a container to doAF_XDP.Looks like the current model is to create a subfunction of a specific type on auxiliary bus, do some configuration to assign resources and then activate the subfunction.
Quoted 1 line
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines
With the VDPA case I believe there is a set of predefined virtio devices that are being emulated and presented so it isn't as if they are creating a totally new interface for this.
vDPA doesn't have any limitation of how the devices is created or implemented. It could be predefined or created dynamically. vDPA leaves all of those to the parent device with the help of a unified management API[1]. E.g It could be a PCI device (PF or VF), sub-function or software emulated devices.
Miss the link, https://www.spinics.net/lists/netdev/msg699374.html.
Thanks
Quoted 21 lines
Quoted 5 lines
Quoted 5 lines
What I would be interested in seeing is if there are any other vendors that have reviewed this and sign off on this approach.
For "this approach" do you mean vDPA subfucntion? My understanding is that it's totally vendor specific, vDPA subsystem don't want to be limited by a specific type of device.
Quoted 5 lines
Quoted 5 lines
What we don't
want to see is Nivida/Mellanox do this one way, then Broadcom or Intel come along later and have yet another way of doing this. We need an interface and feature set that will work for everyone in terms of how this will look going forward.For feature set, it would be hard to force (we can have a recommendation set of features) vendors to implement a common set of features consider they can be negotiated. So the management interface is expected to implement features like cpu clusters in order to make sure the migration compatibility, or qemu can assist for the missing feature with performance lose.
Thanks
Quoted 10 lines · Jakub Kicinski
Quoted 9 lines · On Tue, 17 Nov 2020 14:49:54 -0400 Jason Gunthorpe wrote:
On Tue, Nov 17, 2020 at 09:11:20AM -0800, Jakub Kicinski wrote:
It is consistent with the multi-subsystem device sharing model we've
had for ages now.The physical ethernet port is shared between multiple accelerator subsystems. netdev gets its slice of traffic, so does RDMA, iSCSI, VDPA, etc.
not just a slice of traffic, a whole HW domain.
Quoted 3 lines
Right, devices of other subsystems are fine, I don't care.
But a netdev will be loaded on SF automatically just through the current driver design and modularity, since SF == VF and our netdev is abstract and doesn't know if it runs on a PF/VF/SF .. we literally have to add code to not load a netdev on a SF. why ? :/
Quoted 5 lines
Sorry for not being crystal clear but quite frankly IDK what else can be expected from me given the submissions have little to no context and
documentation. This comes up every damn time with the SF patches, I'm tired of having to ask for a basic workflow.
From how this discussion is going, i think you are right, we need to clarify what we are doing in a more high level simplified and generic documentation to give some initial context, Parav, let's add the missing documentation, we can also add some comments regarding how this is very different from VMDq, but i would like to avoid that, since it is different in almost every way:) ..
Quoted 10 lines
From: Saeed Mahameed <[email protected]>
Sent: Thursday, November 19, 2020 11:42 AMFrom how this discussion is going, i think you are right, we need to clarify what we are doing in a more high level simplified and generic documentation to give some initial context, Parav, let's add the missing documentation, we can also add some comments regarding how this is very different from VMDq, but i would like to avoid that, since it is different in almost every way:)
Sure I will add Documentation/networking/subfunction.rst in v2 describing subfunction details.
Jakub Kicinski
maintainerQuoted 6 lines · On Wed, 18 Nov 2020 22:12:22 -0800 Saeed Mahameed wrote:
Quoted 1 line
Right, devices of other subsystems are fine, I don't care.
But a netdev will be loaded on SF automatically just through the current driver design and modularity, since SF == VF and our netdev is abstract and doesn't know if it runs on a PF/VF/SF .. we literally have to add code to not load a netdev on a SF. why ? :/
A netdev is fine, but the examples so far don't make it clear (to me) if it's expected/supported to spawn _multiple_ netdevs from a single "vdpa parentdev".
Quoted 7 lines
From: Jakub Kicinski <[email protected]>
Sent: Friday, November 20, 2020 7:05 AMQuoted 2 lines · On Wed, 18 Nov 2020 22:12:22 -0800 Saeed Mahameed wrote:
Quoted 1 line
Right, devices of other subsystems are fine, I don't care.
Quoted 8 lines
Quoted 4 lines
But a netdev will be loaded on SF automatically just through the current driver design and modularity, since SF == VF and our netdev is abstract and doesn't know if it runs on a PF/VF/SF .. we literally have to add code to not load a netdev on a SF. why ? :/
A netdev is fine, but the examples so far don't make it clear (to me) if it's expected/supported to spawn _multiple_ netdevs from a single "vdpa parentdev".
We do not create Netdev from vdpa parentdev.
From vdpa parentdev, only vdpa device(s) are created which is 'struct device' residing in /sys/bus/vdpa/<device>.
Currently such vdpa device is already created on mlx5_vpa.ko driver load, however user has no way to inspect, stats, get/set features of this device, hence the vdpa tool.
Hi Jakub,
Quoted 16 lines
From: Jakub Kicinski <[email protected]>
Sent: Tuesday, November 17, 2020 10:41 PMQuoted 12 lines · On Tue, 17 Nov 2020 04:08:57 +0000 Parav Pandit wrote:
Quoted 8 lines · On Mon, 16 Nov 2020 16:06:02 -0800 Saeed Mahameed wrote:
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Subfunction support is discussed in detail in RFC [1] and [2]. RFC [1] and extension [2] describes requirements, design, and proposed plumbing using devlink, auxiliary bus and sysfs for systemd/udev support.
So we're going to have two ways of adding subdevs? Via devlink and via the new vdpa netlink thing?
Nop.
Subfunctions (subdevs) are added only one way, i.e. devlink port as settled in RFC [1].
Quoted 8 lines
Quoted 5 lines
Just to refresh all our memory, we discussed and settled on the flow in [2]; RFC [1] followed this discussion.
vdpa tool of [3] can add one or more vdpa device(s) on top of already spawned PF, VF, SF device.
Nack for the networking part of that. It'd basically be VMDq.
Can you please clarify which networking part do you mean? Which patches exactly in this patchset?
Quoted 25 lines
Quoted 25 lines
Quoted 8 lines
Quoted 4 lines
Via devlink you add the Sub-function bus device - think of it as spawning a new VF - but has no actual characteristics (netdev/vpda/rdma) "yet" until user admin decides to load an interface on it via aux sysfs.
By which you mean it doesn't get probed or the device type is not set (IOW it can still become a block device or netdev depending on the vdpa request)?
Quoted 8 lines
Quoted 3 lines
Basically devlink adds a new eswitch port (the SF port) and loading the drivers and the interfaces is done via the auxbus subsystem only after the SF is spawned by FW.
But why?
Is this for the SmartNIC / bare metal case? The flow for spawning on the local host gets highly convoluted.
The flow of spawning for (a) local host or (b) for external host controller from smartnic is same.
$ devlink port add.. [..] Followed by $ devlink port function set state...
Quoted 7 lines
Quoted 4 lines
Only change would be to specify the destination where to spawn it. (controller number, pf, sf num etc) Please refer to the detailed examples in individual patch. Patch 12 and 13 mostly covers the complete view.
Please share full examples of the workflow.
Please find the full example sequence below, taken from this cover letter and from the respective patches 12 and 13.
Change device to switchdev mode:
$ devlink dev eswitch set pci/0000:06:00.0 mode switchdev
Add a devlink port of subfunction flavour:
$ devlink port add pci/0000:06:00.0 flavour pcisf pfnum 0 sfnum 88
Configure mac address of the port function: (existing API).
$ devlink port function set ens2f0npf0sf88 hw_addr 00:00:00:00:88:88
Now activate the function:
$ devlink port function set ens2f0npf0sf88 state active
Now use the auxiliary device and class devices:
$ devlink dev show pci/0000:06:00.0 auxiliary/mlx5_core.sf.4
$ devlink port show auxiliary/mlx5_core.sf.4/1 auxiliary/mlx5_core.sf.4/1: type eth netdev p0sf88 flavour virtual port 0 splittable false
$ ip link show
127: ens2f0np0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 24:8a:07:b3:d1:12 brd ff:ff:ff:ff:ff:ff
altname enp6s0f0np0
129: p0sf88: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 00:00:00:00:88:88 brd ff:ff:ff:ff:ff:ff$ rdma dev show 43: rdmap6s0f0: node_type ca fw 16.28.1002 node_guid 248a:0703:00b3:d112 sys_image_guid 248a:0703:00b3:d112 44: mlx5_0: node_type ca fw 16.28.1002 node_guid 0000:00ff:fe00:8888 sys_image_guid 248a:0703:00b3:d112
At this point vdpa tool of [1] can create one or more vdpa net devices on this subfunction device in below sequence.
$ vdpa parentdev list
auxiliary/mlx5_core.sf.4
supported_classes
net$ vdpa dev add parentdev auxiliary/mlx5_core.sf.4 type net name foo0
$ vdpa dev show foo0 foo0: parentdev auxiliary/mlx5_core.sf.4 type network parentdev vdpasim vendor_id 0 max_vqs 2 max_vq_size 256
Quoted 2 lines
I'm asking how the vdpa API fits in with this, and you're showing me the two devlink commands we already talked about in the past.
Oh ok, sorry, my bad. I understood your question now about relation of vdpa commands with this. Please look at the above example sequence that covers the vdpa example also.
Jakub Kicinski
maintainerQuoted 12 lines · On Tue, 17 Nov 2020 18:50:57 +0000 Parav Pandit wrote:
At this point vdpa tool of [1] can create one or more vdpa net devices on this subfunction device in below sequence.
$ vdpa parentdev list auxiliary/mlx5_core.sf.4 supported_classes net$ vdpa dev add parentdev auxiliary/mlx5_core.sf.4 type net name foo0$ vdpa dev show foo0 foo0: parentdev auxiliary/mlx5_core.sf.4 type network parentdev vdpasim vendor_id 0 max_vqs 2 max_vq_size 256
Quoted 6 lines
Quoted 2 lines
I'm asking how the vdpa API fits in with this, and you're showing me the two devlink commands we already talked about in the past.
Oh ok, sorry, my bad. I understood your question now about relation of vdpa commands with this. Please look at the above example sequence that covers the vdpa example also.
I think the biggest missing piece in my understanding is what's the technical difference between an SF and a VDPA device.
Isn't a VDPA device an SF with a particular descriptor format for the queues?
Quoted 24 lines · Jakub Kicinski
Quoted 23 lines · On Tue, 17 Nov 2020 18:50:57 +0000 Parav Pandit wrote:
At this point vdpa tool of [1] can create one or more vdpa net devices on this subfunction device in below sequence.
$ vdpa parentdev list auxiliary/mlx5_core.sf.4 supported_classes net$ vdpa dev add parentdev auxiliary/mlx5_core.sf.4 type net name foo0$ vdpa dev show foo0 foo0: parentdev auxiliary/mlx5_core.sf.4 type network parentdev vdpasim vendor_id 0 max_vqs 2 max_vq_size 256Quoted 3 lines
I'm asking how the vdpa API fits in with this, and you're showing me the two
devlink commands we already talked about in the past.Oh ok, sorry, my bad. I understood your question now about relation of vdpa commands with this.
Please look at the above example sequence that covers the vdpa example also.
Quoted 6 lines
Quoted 2 lines
I think the biggest missing piece in my understanding is what's the technical difference between an SF and a VDPA device.
Same difference as between a VF and netdev.
SF == VF, so a full HW function. VDPA/RDMA/netdev/SCSI/nvme/etc.. are just interfaces (ULPs) sharing the same functions as always been, nothing new about this.
Today on a VF we load a RDMA/VDPA/netdev interfaces
SF will do exactly the same and the ULPs will simply load, and we don't need to modify them.
Quoted 2 lines
Isn't a VDPA device an SF with a particular descriptor format for the queues?
No :/,
I hope the above answer clarifies things a bit. SF is a device function that provides all kinds of queues.
Jason Gunthorpe
maintainerQuoted 7 lines · Saeed Mahameed
Quoted 2 lines
I think the biggest missing piece in my understanding is what's the technical difference between an SF and a VDPA device.
Same difference as between a VF and netdev.
SF == VF, so a full HW function. VDPA/RDMA/netdev/SCSI/nvme/etc.. are just interfaces (ULPs) sharing the same functions as always been, nothing new about this.
All the implementation details are very different, but this white paper from Intel goes into some detail the basic elements and rational for the SF concept:
What we are calling a sub-function here is a close cousin to what Intel calls an Assignable Device Interface. I expect to see other drivers following this general pattern eventually.
A SF will eventually be assignable to a VM and the VM won't be able to tell the difference between a VF or SF providing the assignable PCI resources.
VDPA is also assignable to a guest, but the key difference between mlx5's SF and VDPA is what guest driver binds to the virtual PCI function. For a SF the guest will bind mlx5_core, for VDPA the guest will bind virtio-net.
So, the driver stack for a VM using VDPA might be
Physical device [pci] -> mlx5_core -> [aux] -> SF -> [aux] -> mlx5_core -> [aux] -> mlx5_vdpa -> QEMU -> |VM| -> [pci] -> virtio_net
When Parav is talking about creating VDPA devices he means attaching the VDPA accelerator subsystem to a mlx5_core, where ever that mlx5_core might be attached to.
To your other remark:
Quoted 2 lines
Quoted 1 line
What are you NAK'ing?
Spawning multiple netdevs from one device by slicing up its queues.
This is a bit vauge. In SRIOV a device spawns multiple netdevs for a physical port by "slicing up its physical queues" - where do you see the cross over between VMDq (bad) and SRIOV (ok)?
I thought the issue with VMDq was more on the horrid management to configure the traffic splitting, not the actual splitting itself?
In classic SRIOV the traffic is split by a simple non-configurable HW switch based on MAC address of the VF.
mlx5 already has the extended version of that idea, we can run in switchdev mode and use switchdev to configure the HW switch. Now configurable switchdev rules split the traffic for VFs.
This SF step replaces the VF in the above, but everything else is the same. The switchdev still splits the traffic, it still ends up in same nested netdev queue structure & RSS a VF/PF would use, etc, etc. No queues are "stolen" to create the nested netdev.
From the driver perspective there is no significant difference between sticking a netdev on a mlx5 VF or sticking a netdev on a mlx5 SF. A SF netdev is not going in and doing deep surgery to the PF netdev to steal queues or something.
Both VF and SF will be eventually assignable to guests, both can support all the accelerator subsystems - VDPA, RDMA, etc. Both can support netdev.
Compared to VMDq, I think it is really no comparison. SF/ADI is an evolution of a SRIOV VF from something PCI-SGI controlled to something device specific and lighter weight.
SF/ADI come with a architectural security boundary suitable for assignment to an untrusted guest. It is not just a jumble of queues.
VMDq is .. not that.
Actually it has been one of the open debates in the virtualization userspace world. The approach to use switchdev to control the traffic splitting to VMs is elegant but many drivers are are not following this design. :(
Finally, in the mlx5 model VDPA is just an "application". It asks the device to create a 'RDMA' raw ethernet packet QP that is uses rings formed in the virtio-net specification. We can create it in the kernel using mlx5_vdpa, and we can create it in userspace through the RDMA subsystem. Like any "RDMA" application it is contained by the security boundary of the PF/VF/SF the mlx5_core is running on.
Jason
Jakub Kicinski
maintainerQuoted 6 lines · On Thu, 19 Nov 2020 10:00:17 -0400 Jason Gunthorpe wrote:
Finally, in the mlx5 model VDPA is just an "application". It asks the device to create a 'RDMA' raw ethernet packet QP that is uses rings formed in the virtio-net specification. We can create it in the kernel using
mlx5_vdpa, and we can create it in userspace through the RDMA subsystem. Like any "RDMA" application it is contained by the security boundary of the PF/VF/SF themlx5_coreis running on.
Thanks for the write up!
The SF part is pretty clear to me, it is what it is. DPDK camp has been pretty excited about ADI/PASID for a while now.
The part that's blurry to me is VDPA.
I was under the impression that for VDPA the device is supposed to support native virtio 2.0 (or whatever the "HW friendly" spec was).
I believe that's what the early patches from Intel did.
You're saying it's a client application like any other - do I understand it right that the hypervisor driver will be translating descriptors between virtio and device-native then?
The vdpa parent is in the hypervisor correct?
Can a VDPA device have multiple children of the same type?
Why do we have a representor for a SF, if the interface is actually VDPA? Block and net traffic can't reasonably be treated the same by the switch.
Also I'm confused how block device can bind to mlx5_core - in that case I'm assuming the QP is bound 1:1 with a QP on the SmartNIC side, and that QP is plugged into an appropriate backend?
Quoted 19 lines
From: Jakub Kicinski <[email protected]>
Sent: Friday, November 20, 2020 9:05 AMQuoted 6 lines · On Thu, 19 Nov 2020 10:00:17 -0400 Jason Gunthorpe wrote:
Finally, in the mlx5 model VDPA is just an "application". It asks the device to create a 'RDMA' raw ethernet packet QP that is uses rings formed in the virtio-net specification. We can create it in the kernel using
mlx5_vdpa, and we can create it in userspace through the RDMA subsystem. Like any "RDMA" application it is contained by the security boundary of the PF/VF/SF themlx5_coreis running on.Thanks for the write up!
The SF part is pretty clear to me, it is what it is. DPDK camp has been pretty excited about ADI/PASID for a while now.
The part that's blurry to me is VDPA.
Quoted 9 lines
I was under the impression that for VDPA the device is supposed to support native virtio 2.0 (or whatever the "HW friendly" spec was).
I believe that's what the early patches from Intel did.
You're saying it's a client application like any other - do I understand it right that the hypervisor driver will be translating descriptors between virtio and device- native then?
mlx5 device support virtio descriptors natively. So no need of translation.
Quoted 2 lines
The vdpa parent is in the hypervisor correct?
Yep.
Quoted 2 lines
Can a VDPA device have multiple children of the same type?
I guess, you mean VDPA parentdev? If so, yes, however at present we see only one_to_one mapping of vdpa device and parent dev.
Quoted 1 line
Why do we have a representor for a SF, if the interface is actually VDPA?
Because vdpa is just one client out of multiple.
At the moment there is one to one relation of vdpa device to a SF/VF.
Quoted 5 lines
Block and net traffic can't reasonably be treated the same by the switch.
Also I'm confused how block device can bind to
mlx5_core- in that case I'm assuming the QP is bound 1:1 with a QP on the SmartNIC side, and that QP is plugged into an appropriate backend?
So far there isn't mlx5_vdpa.ko or plan to do block. But yes, in future for block, it needs to bind to a QP in backend in smartnic.
Jason Gunthorpe
maintainerQuoted 9 lines · Jakub Kicinski
Quoted 6 lines · On Thu, 19 Nov 2020 10:00:17 -0400 Jason Gunthorpe wrote:
Finally, in the mlx5 model VDPA is just an "application". It asks the device to create a 'RDMA' raw ethernet packet QP that is uses rings formed in the virtio-net specification. We can create it in the kernel using
mlx5_vdpa, and we can create it in userspace through the RDMA subsystem. Like any "RDMA" application it is contained by the security boundary of the PF/VF/SF themlx5_coreis running on.Thanks for the write up!
No problem!
Quoted 1 line
The part that's blurry to me is VDPA.
Okay, I think I see where the gap is, I'm going to elaborate below so we are clear.
Quoted 2 lines
I was under the impression that for VDPA the device is supposed to support native virtio 2.0 (or whatever the "HW friendly" spec was).
I think VDPA covers a wide range of things.
The basic idea is starting with the all SW virtio-net implementation we can move parts to HW. Each implementation will probably be a little different here. The kernel vdpa subsystem is a toolbox to mix the required emulation and HW capability to build a virtio-net PCI interface.
The most key question to ask of any VDPA design is "what does the VDPA FW do with the packet once the HW accelerator has parsed the virtio-net descriptor?".
The VDPA world has refused to agree on this due to vendor squabbling, but mlx5 has a clear answer:
VDPA Tx generates an ethernet packet and sends it out the SF/VF port through a tunnel to the representor and then on to the switchdev.
Other VDPA designs have a different answer!!
This concept is so innate to how Mellanox views the world it is not surprising me that the cover letters and patch descriptions don't belabor this point much :)
I'm going to deep dive through this answer below. I think you'll see this is the most sane and coherent architecture with the tools available in netdev.. Mellanox thinks the VDPA world should standardize on this design so we can have a standard control plane.
Quoted 3 lines
You're saying it's a client application like any other - do I understand it right that the hypervisor driver will be translating descriptors between virtio and device-native then?
No, the hypervisor creates a QP and tells the HW that this QP's descriptor format follows virtio-net. The QP processes those descriptors in HW and generates ethernet packets.
A "client application like any other" means that the ethernet packets VDPA forms are identical to the ones netdev or RDMA forms. They are all delivered into the tunnel on the SF/VF to the representor and on to the switch. See below
Quoted 3 lines
The vdpa parent is in the hypervisor correct?
Can a VDPA device have multiple children of the same type?
I'm not sure parent/child are good words here.
The VDPA emulation runs in the hypervisor, and the virtio-net netdev driver runs in the guest. The VDPA is attached to a switchdev port and representor tunnel by virtue of its QPs being created under a SF/VF.
If we imagine a virtio-rdma, then you might have a SF/VF hosting both VDPA and VDPA-RDMA which emulate two PCI devices assigned to a VM. Both of these peer virtio's would generate ethernet packets for TX on the SF/VF port into the tunnel through the represntor and to the switch.
Quoted 3 lines
Why do we have a representor for a SF, if the interface is actually VDPA? Block and net traffic can't reasonably be treated the same by the switch.
I think you are focusing on queues, the architecture at PF/SF/VF is not queue based, it is packet based.
At the physical mlx5 the netdev has a switchdev. On that switch I can create a *switch port*.
The switch port is composed of a representor and a SF/VF. They form a tunnel for packets.
The representor is the hypervisor side of the tunnel and contains all packets coming out of and into the SF/VF.
The SF/VF is the guest side of the tunnel and has a full NIC.
The SF/VF can be:
- Used in the same OS as the switch
- Assigned to a guest VM as a PCI device
- Assigned to another processor in the SmartNIC case.
In all cases if I use a queue on a SF/VF to generate an ethernet packet then that packet *always* goes into the tunnel to the representor and goes into a switch. It is always contained by any rules on the switch side. If the switch is set so the representor is VLAN tagged then a queue on a SF/VF *cannot* escape the VLAN tag.
Similarly SF/VF cannot Rx any packets that are not sent into the tunnel, meaning the switch controls what packets go into the representor, through the tunnel and to the SF.
Yes, block and net traffic are all reduced to ethernet packets, sent through the tunnel to the representor and treated by the switch. It is no different than a physical switch. If there is to be some net/block difference it has to be represented in the ethernet packets, eg with vlan or something.
This is the fundamental security boundary of the architecture. The SF/VF is a security domain and the only exchange of information from that security domain to the hypervisor security domain is the tunnel to the representor. The exchange across the boundary is only *packets* not queues.
Essentially it exactly models the physical world. If I phyically plug in a NIC to a switch then the "representor" is the switch port in the physical switch OS and the "SF/VF" is the NIC in the server.
The switch OS does not know or care what the NIC is doing. It does not know or care if the NIC is doing VDPA, or if the packets are "block" or "net" - they are all just packets by the time it gets to switching.
Quoted 3 lines
Also I'm confused how block device can bind to
mlx5_core- in that case I'm assuming the QP is bound 1:1 with a QP on the SmartNIC side, and that QP is plugged into an appropriate backend?
Every mlx5_core is a full multi-queue instance. It can have a huge number of queues with no problems. Do not focus on the queues. *queues* are irrelevant here.
Queues always have two ends. In this model one end is at the CPU and the other is just ethernet packets. The purpose of the queue is to convert CPU stuff into ethernet packets and vice versa. A mlx5 device has a wide range of accelerators that can do all sorts of transformations between CPU and packets built into the queues.
A queue can only be attached to a single mlx5_core, meaning all the ethernet packets the queue sources/sinks must come from the PF/SF/VF port. For SF/VF this port is connected to a tunnel to a representor to the switch. Thus every queue has its packet side connected to the switch.
However, the *queue* is an opaque detail of how the ethernet packets are created from CPU data.
It doesn't matter if the queue is running VDPA, RDMA, netdev, or block traffic - all of these things inherently result in ethernet packets, and the hypervisor can't tell how the packet was created.
The architecture is *not* like virtio. virtio queues are individual tunnels between hypervisor and guest.
This is the key detail: A VDPA queue is *not a tunnel*. It is a engine to covert CPU data in virtio-net format to ethernet packets and deliver those packet to the SF/VF end of the tunnel to the representor and then to the switch. The tunnel is the SF/VF and representor pairing, NOT the VDPA queue.
Looking at the logical life of a Tx packet from a VM doing VDPA:
- VM's netdev builds the skb and writes a vitio-net formed descriptor to a send qeuue
- VM triggers a doorbell via write to a BAR. In mlx5 this write goes to the device - qemu mmaps part of the device BAR to the guest
- The HW begins processing a queue. The queue is in virtio-net format so it fetches the descriptor and now has the skb data
- The HW forms the skb into an ethernet packet and delivers it to the representor through the tunnel, which immediately sends it to the HW switch. The VDPA QP in the SF/VF is now done.
- In the switch the HW determines the packet is an exception. It applies RSS rules/etc and dynamically identifies on a per-packet basis what hypervisor queue the packet should be delivered to. This queue is in the hypervisor, and is in mlx5 native format.
- The choosen hypervisor queue recives this packet and begins processing. It gets a receive buffer and writes the packet, triggers an interrupts. This queue is now done.
- hypervisor netdev now has the packet. It does the exception path in netdev and puts the SKB back on another queue for TX to the physical port. This queue is in mlx5 native format, the packet goes to the physical port.
It traversed three queues. The HW dynamically selected the hypervisor queue the VDPA packet is delivered to based *entirely* on switch rules. The originating queue only informs the switch of what SF/VF (and thus switch port) generated the packet.
At no point does the hypervisor know the packet originated from a VDPA QP.
The RX side the similar, each PF/SF/VF port has a selector that chooses which queue each packet goes to. That chooses how the packet is converted to CPU. Each PF/SF/VF can have a huge number of selectors, and SF/VF source their packets from the logical tunnel attached to a representor which receives packets from the switch.
The selector is how the cross subsystem sharing of the ethernet port works, regardless of PF/SF/VF.
Again the hypervisor side has *no idea* what queue the packet will be selected to when it delivers the packet to the representor side of the tunnel.
Jason