mlx5 single FDB for lag
Subsystems (6)
cover mlx5 single FDB for lag
From: Saeed Mahameed <[email protected]>
This series is aimed at mlx5-next branch to be pulled later by both rdma and netdev subsystems as it contains patches to both trees.
The series provides support for single shared FDB table for lag:
Shared FDB allows to direct traffic from all the vports in the HCA to a single E-Switch, as opposed to an E-Switch per up-link, a single E-switch will improve the lag logic as the traffic will be handled by a single point on the device, which allows more flexibility and natural management of FDB rules when lag is ON.
Before shared FDB in order to control traffic from a vport when lag was ON a FDB rule had to be duplicated (on both E-switches), with single FDB duplication is not required.
To achieve single FDB:
1) Point the ingress ACL of the slave uplink to that of the master. With this, wire traffic from both uplinks will reach the same eswitch with the same metadata where a single steering rule can catch traffic from both ports.
2) Set the FDB root flow table of the slave's eswitch to that of the master. As this flow table can change dynamically make sure to sync it on any set root flow table FDB command.
This will make sure traffic from SFs, VFs, ECPFs and PFs reach the master eswitch.
3) Split wire traffic at the eswitch manager egress ACL so that it's directed to the native eswitch manager. We only treat wire traffic from both ports the same at the eswitch level. If such traffic wasn't handled in the eswitch it needs to reach the right representor to be processed by software. For example LACP packets should *always* reach the right uplink representor for correct operation.
---
Ariel Levkovich (1):
net/mlx5: E-Switch, set flow source for send to uplink rule
Mark Bloch (11):
net/mlx5: Return mdev from eswitch
net/mlx5: Lag, add initial logic for shared FDB RDMA/mlx5: Fill port info based on the relevant eswitch {net, RDMA}/mlx5: Extend send to vport rules RDMA/mlx5: Add shared FDB support
net/mlx5: E-Switch, Add event callback for representors net/mlx5: Add send to vport rules on paired device net/mlx5: Lag, properly lock eswitch if needed net/mlx5: Lag, move lag destruction to a workqueue net/mlx5/ E-Switch, add logic to enable shared FDB net/mlx5: Lag, Create shared FDB when in switchdev mode
Roi Dayan (2):
net/mlx5e: Add an option to create a shared mapping net/mlx5e: Use shared mappings for restoring from metadata
drivers/infiniband/hw/mlx5/ib_rep.c | 77 +++- drivers/infiniband/hw/mlx5/main.c | 44 +- drivers/infiniband/hw/mlx5/std_types.c | 10 +- .../ethernet/mellanox/mlx5/core/en/mapping.c | 45 ++ .../ethernet/mellanox/mlx5/core/en/mapping.h | 5 + .../ethernet/mellanox/mlx5/core/en/tc_ct.c | 9 +- .../net/ethernet/mellanox/mlx5/core/en_rep.c | 88 +++- .../net/ethernet/mellanox/mlx5/core/en_rep.h | 2 + .../net/ethernet/mellanox/mlx5/core/en_tc.c | 21 +- .../mellanox/mlx5/core/esw/acl/egress_ofld.c | 16 + .../net/ethernet/mellanox/mlx5/core/eswitch.c | 36 +- .../net/ethernet/mellanox/mlx5/core/eswitch.h | 38 ++ .../mellanox/mlx5/core/eswitch_offloads.c | 383 +++++++++++++++++- .../net/ethernet/mellanox/mlx5/core/fs_cmd.c | 58 ++- .../net/ethernet/mellanox/mlx5/core/fs_core.c | 2 +- .../net/ethernet/mellanox/mlx5/core/fs_core.h | 2 + drivers/net/ethernet/mellanox/mlx5/core/lag.c | 267 ++++++++++-- drivers/net/ethernet/mellanox/mlx5/core/lag.h | 5 +- .../net/ethernet/mellanox/mlx5/core/lag_mp.c | 2 +- .../net/ethernet/mellanox/mlx5/core/main.c | 5 +- .../ethernet/mellanox/mlx5/core/mlx5_core.h | 2 + include/linux/mlx5/driver.h | 3 + include/linux/mlx5/eswitch.h | 16 + 23 files changed, 1043 insertions(+), 93 deletions(-)
Signature · 1 line
2.31.1
PATCH 1/14 net/mlx5: Return mdev from eswitch
From: Mark Bloch <[email protected]>
Export a function so users can retrieve the mellanox device that manages the eswitch from the eswitch device.
--- drivers/net/ethernet/mellanox/mlx5/core/eswitch.c | 12 ++++++++++++ include/linux/mlx5/eswitch.h | 6 ++++++ 2 files changed, 18 insertions(+)
drivers/net/ethernet/mellanox/mlx5/core/eswitch.c +12 -0
@@ -2384,3 +2384,15 @@ u16 mlx5_eswitch_get_total_vports(const struct mlx5_core_dev *dev)return mlx5_esw_allowed (esw) ? esw->total_vports :0 ; }EXPORT_SYMBOL_GPL (mlx5_eswitch_get_total_vports); + +/** +* mlx5_eswitch_get_core_dev - Get the mdev device +* @esw : eswitch device. +* +* Return the mellanox core device which manages the eswitch. +*/ +struct mlx5_core_dev *mlx5_eswitch_get_core_dev (struct mlx5_eswitch *esw) +{ +return mlx5_esw_allowed (esw) ? esw->dev :NULL ; +} +EXPORT_SYMBOL (mlx5_eswitch_get_core_dev);
include/linux/mlx5/eswitch.h +6 -0
@@ -128,6 +128,7 @@ u32 mlx5_eswitch_get_vport_metadata_for_set(struct mlx5_eswitch *esw,u8 mlx5_eswitch_mode (struct mlx5_core_dev *dev);u16 mlx5_eswitch_get_total_vports (const struct mlx5_core_dev *dev); +struct mlx5_core_dev *mlx5_eswitch_get_core_dev (struct mlx5_eswitch *esw);#else /* CONFIG_MLX5_ESWITCH */
@@ -171,6 +172,11 @@ static inline u16 mlx5_eswitch_get_total_vports(const struct mlx5_core_dev *dev) return 0; } +static inline struct mlx5_core_dev *mlx5_eswitch_get_core_dev (struct mlx5_eswitch *esw) +{ +return NULL ; +} +#endif /* CONFIG_MLX5_ESWITCH */ static inline bool is_mdev_switchdev_mode (struct mlx5_core_dev *dev)
Signature · 1 line
2.31.1
PATCH 2/14 net/mlx5: Lag, add initial logic for shared FDB
In reply to Saeed MahameedFrom: Mark Bloch <[email protected]>
As shared FDB requires changes in two subsystems first expose the needed core functions so the RDMA side can be changed.
mlx5_lag_is_master(): return true if a given mlx5 device is the lag master. mlx5_lag_is_shared_fdb(): Returns true if the lag mode is shared FDB. mlx5_lag_get_peer_mdev(): Return the peer mdev in lag.
The mentioned functions will be used by downstream patches in order to add support for shared FDB for the RDMA side.
--- drivers/net/ethernet/mellanox/mlx5/core/lag.c | 49 +++++++++++++++++++ drivers/net/ethernet/mellanox/mlx5/core/lag.h | 1 + include/linux/mlx5/driver.h | 3 ++ 3 files changed, 53 insertions(+)
drivers/net/ethernet/mellanox/mlx5/core/lag.c +49 -0
@@ -746,6 +746,21 @@ bool mlx5_lag_is_active(struct mlx5_core_dev *dev) }EXPORT_SYMBOL (mlx5_lag_is_active); +bool mlx5_lag_is_master (struct mlx5_core_dev *dev) +{ +struct mlx5_lag *ldev; +bool res; + +spin_lock (&lag_lock); + ldev =mlx5_lag_dev (dev); + res = ldev &&__mlx5_lag_is_active (ldev) && + dev == ldev->pf[MLX5_LAG_P1 ].dev; +spin_unlock (&lag_lock); + +return res; +} +EXPORT_SYMBOL (mlx5_lag_is_master); +bool mlx5_lag_is_sriov (struct mlx5_core_dev *dev) {struct mlx5_lag *ldev;
@@ -760,6 +775,20 @@ bool mlx5_lag_is_sriov(struct mlx5_core_dev *dev) }EXPORT_SYMBOL (mlx5_lag_is_sriov); +bool mlx5_lag_is_shared_fdb (struct mlx5_core_dev *dev) +{ +struct mlx5_lag *ldev; +bool res; + +spin_lock (&lag_lock); + ldev =mlx5_lag_dev (dev); + res = ldev &&__mlx5_lag_is_sriov (ldev) && ldev->shared_fdb; +spin_unlock (&lag_lock); + +return res; +} +EXPORT_SYMBOL (mlx5_lag_is_shared_fdb); +void mlx5_lag_update (struct mlx5_core_dev *dev) {struct mlx5_lag *ldev;
@@ -827,6 +856,26 @@ u8 mlx5_lag_get_slave_port(struct mlx5_core_dev *dev, }EXPORT_SYMBOL (mlx5_lag_get_slave_port); +struct mlx5_core_dev *mlx5_lag_get_peer_mdev (struct mlx5_core_dev *dev) +{ +struct mlx5_core_dev *peer_dev =NULL ; +struct mlx5_lag *ldev; + +spin_lock (&lag_lock); + ldev =mlx5_lag_dev (dev); +if (!ldev) + goto unlock; + + peer_dev = ldev->pf[MLX5_LAG_P1 ].dev == dev ? + ldev->pf[MLX5_LAG_P2 ].dev : + ldev->pf[MLX5_LAG_P1 ].dev; + +unlock: +spin_unlock (&lag_lock); +return peer_dev; +} +EXPORT_SYMBOL (mlx5_lag_get_peer_mdev); + int mlx5_lag_query_cong_counters(struct mlx5_core_dev *dev, u64 *values, int num_counters,
drivers/net/ethernet/mellanox/mlx5/core/lag.h +1 -0
@@ -39,6 +39,7 @@ struct lag_tracker {*/ struct mlx5_lag {u8 flags; +bool shared_fdb;u8 v2p_map[MLX5_MAX_PORTS ];struct kref ref;struct lag_func pf[MLX5_MAX_PORTS ];
include/linux/mlx5/driver.h +3 -0
@@ -1138,6 +1138,8 @@ bool mlx5_lag_is_roce(struct mlx5_core_dev *dev);bool mlx5_lag_is_sriov (struct mlx5_core_dev *dev);bool mlx5_lag_is_multipath (struct mlx5_core_dev *dev);bool mlx5_lag_is_active (struct mlx5_core_dev *dev); +bool mlx5_lag_is_master (struct mlx5_core_dev *dev); +bool mlx5_lag_is_shared_fdb (struct mlx5_core_dev *dev);struct net_device *mlx5_lag_get_roce_netdev (struct mlx5_core_dev *dev);u8 mlx5_lag_get_slave_port (struct mlx5_core_dev *dev,struct net_device *slave);
@@ -1145,6 +1147,7 @@ int mlx5_lag_query_cong_counters(struct mlx5_core_dev *dev,u64 *values, int num_counters, size_t *offsets); +struct mlx5_core_dev *mlx5_lag_get_peer_mdev (struct mlx5_core_dev *dev);struct mlx5_uars_page *mlx5_get_uars_page (struct mlx5_core_dev *mdev);void mlx5_put_uars_page (struct mlx5_core_dev *mdev,struct mlx5_uars_page *up); int mlx5_dm_sw_icm_alloc(struct mlx5_core_dev *dev, enum mlx5_sw_icm_type type,
Signature · 1 line
2.31.1
PATCH 3/14 RDMA/mlx5: Fill port info based on the relevant eswitch
In reply to Saeed MahameedFrom: Mark Bloch <[email protected]>
In shared FDB a single RDMA device can have representors that are connected to two different eswitches. Use the right eswitch when preparing the response to userspace.
--- drivers/infiniband/hw/mlx5/std_types.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-)
drivers/infiniband/hw/mlx5/std_types.c +7 -3
@@ -114,14 +114,18 @@ static int fill_vport_vhca_id(struct mlx5_core_dev *mdev, u16 vport,static int fill_switchdev_info (struct mlx5_ib_dev *dev,u32 port_num,struct mlx5_ib_uapi_query_port *info) { -struct mlx5_core_dev *mdev = dev->mdev;struct mlx5_eswitch_rep *rep; +struct mlx5_core_dev *mdev;int err; rep = dev->port[port_num -1 ].rep;if (!rep)return -EOPNOTSUPP ; + mdev =mlx5_eswitch_get_core_dev (rep->esw); +if (!mdev) +return -EINVAL ; + info->vport = rep->vport; info->flags |=MLX5_IB_UAPI_QUERY_PORT_VPORT ;
@@ -138,9 +142,9 @@ static int fill_switchdev_info(struct mlx5_ib_dev *dev, u32 port_num,if (err)return err; -if (mlx5_eswitch_vport_match_metadata_enabled (mdev->priv.eswitch)) { +if (mlx5_eswitch_vport_match_metadata_enabled (rep->esw)) { info->reg_c0.value =mlx5_eswitch_get_vport_metadata_for_match ( - mdev->priv.eswitch, rep->vport); + rep->esw, rep->vport); info->reg_c0.mask =mlx5_eswitch_get_vport_metadata_mask (); info->flags |=MLX5_IB_UAPI_QUERY_PORT_VPORT_REG_C0 ; }
Signature · 1 line
2.31.1
PATCH 4/14 {net, RDMA}/mlx5: Extend send to vport rules
In reply to Saeed MahameedFrom: Mark Bloch <[email protected]>
In shared FDB there is only one eswitch which is active and it receives traffic from all representors and all vports in the HCA.
While the Ethernet representor will always reside on its native PF the IB representor will not. Extend send to vport rule creation to support such flows. Need to account for source vport that sends the traffic (on which the representors resides) and the target eswitch the traffic which reach.
--- drivers/infiniband/hw/mlx5/ib_rep.c | 2 +- drivers/net/ethernet/mellanox/mlx5/core/en_rep.c | 2 +- drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c | 5 +++-- include/linux/mlx5/eswitch.h | 1 + 4 files changed, 6 insertions(+), 4 deletions(-)
drivers/infiniband/hw/mlx5/ib_rep.c +1 -1
@@ -123,7 +123,7 @@ struct mlx5_flow_handle *create_flow_rule_vport_sq(struct mlx5_ib_dev *dev, rep = dev->port[port -1 ].rep; -return mlx5_eswitch_add_send_to_vport_rule (esw, rep, sq->base.mqp.qpn); +return mlx5_eswitch_add_send_to_vport_rule (esw, esw, rep, sq->base.mqp.qpn); } static int mlx5r_rep_probe(struct auxiliary_device *adev,
drivers/net/ethernet/mellanox/mlx5/core/en_rep.c +1 -1
@@ -337,7 +337,7 @@ static int mlx5e_sqs2vport_start(struct mlx5_eswitch *esw, }/* Add re-inject rule to the PF/representor sqs */ - flow_rule =mlx5_eswitch_add_send_to_vport_rule (esw, rep, + flow_rule =mlx5_eswitch_add_send_to_vport_rule (esw, esw, rep, sqns_array[i]);if (IS_ERR (flow_rule)) { err =PTR_ERR (flow_rule);
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +3 -2
@@ -925,6 +925,7 @@ int mlx5_eswitch_del_vlan_action(struct mlx5_eswitch *esw,struct mlx5_flow_handle *mlx5_eswitch_add_send_to_vport_rule (struct mlx5_eswitch *on_esw, +struct mlx5_eswitch *from_esw,struct mlx5_eswitch_rep *rep,u32 sqn) {
@@ -943,10 +944,10 @@ mlx5_eswitch_add_send_to_vport_rule(struct mlx5_eswitch *on_esw, misc =MLX5_ADDR_OF (fte_match_param, spec->match_value, misc_parameters);MLX5_SET (fte_match_set_misc, misc, source_sqn, sqn);/* source vport is the esw manager */ -MLX5_SET (fte_match_set_misc, misc, source_port, rep->esw->manager_vport); +MLX5_SET (fte_match_set_misc, misc, source_port, from_esw->manager_vport);if (MLX5_CAP_ESW (on_esw->dev, merged_eswitch))MLX5_SET (fte_match_set_misc, misc, source_eswitch_owner_vhca_id, -MLX5_CAP_GEN (rep->esw->dev, vhca_id)); +MLX5_CAP_GEN (from_esw->dev, vhca_id)); misc =MLX5_ADDR_OF (fte_match_param, spec->match_criteria, misc_parameters);MLX5_SET_TO_ONES (fte_match_set_misc, misc, source_sqn);
include/linux/mlx5/eswitch.h +1 -0
@@ -63,6 +63,7 @@ struct mlx5_eswitch_rep *mlx5_eswitch_vport_rep(struct mlx5_eswitch *esw,void *mlx5_eswitch_uplink_get_proto_dev (struct mlx5_eswitch *esw,u8 rep_type);struct mlx5_flow_handle *mlx5_eswitch_add_send_to_vport_rule (struct mlx5_eswitch *on_esw, +struct mlx5_eswitch *from_esw,struct mlx5_eswitch_rep *rep,u32 sqn);#ifdef CONFIG_MLX5_ESWITCH
Signature · 1 line
2.31.1
PATCH 5/14 RDMA/mlx5: Add shared FDB support
In reply to Saeed MahameedFrom: Mark Bloch <[email protected]>
Shared FDB allows to create a single RDMA device that holds representors from both eswitches. As shared FDB is only active when both uplink representors are enslaved there is a single RDMA port that represents both uplinks.
The number of ports is the number of vports on both eswitches minus one as we only need 1 port for both uplinks.
--- drivers/infiniband/hw/mlx5/ib_rep.c | 75 ++++++++++++++++++++++++++--- drivers/infiniband/hw/mlx5/main.c | 44 ++++++++++------- 2 files changed, 95 insertions(+), 24 deletions(-)
drivers/infiniband/hw/mlx5/ib_rep.c +69 -6
@@ -8,13 +8,15 @@#include "srq.h" static int -mlx5_ib_set_vport_rep (struct mlx5_core_dev *dev,struct mlx5_eswitch_rep *rep) +mlx5_ib_set_vport_rep (struct mlx5_core_dev *dev, +struct mlx5_eswitch_rep *rep, +int vport_index) {struct mlx5_ib_dev *ibdev; -int vport_index; ibdev =mlx5_eswitch_uplink_get_proto_dev (dev->priv.eswitch,REP_IB ); - vport_index = rep->vport_index; +if (!ibdev) +return -EINVAL ; ibdev->port[vport_index].rep = rep; rep->rep_data[REP_IB ].priv = ibdev;
@@ -26,19 +28,39 @@ mlx5_ib_set_vport_rep(struct mlx5_core_dev *dev, struct mlx5_eswitch_rep *rep)return 0 ; } +static void mlx5_ib_register_peer_vport_reps (struct mlx5_core_dev *mdev); +static int mlx5_ib_vport_rep_load (struct mlx5_core_dev *dev,struct mlx5_eswitch_rep *rep) {u32 num_ports =mlx5_eswitch_get_total_vports (dev);const struct mlx5_ib_profile *profile; +struct mlx5_core_dev *peer_dev;struct mlx5_ib_dev *ibdev; +u32 peer_num_ports;int vport_index;int ret; + vport_index = rep->vport_index; + +if (mlx5_lag_is_shared_fdb (dev)) { + peer_dev =mlx5_lag_get_peer_mdev (dev); + peer_num_ports =mlx5_eswitch_get_total_vports (peer_dev); +if (mlx5_lag_is_master (dev)) { +/* Only 1 ib port is the representor for both uplinks */ + num_ports += peer_num_ports -1 ; + }else { +if (rep->vport ==MLX5_VPORT_UPLINK ) +return 0 ; + vport_index += peer_num_ports; + dev = peer_dev; + } + } +if (rep->vport ==MLX5_VPORT_UPLINK ) profile = &raw_eth_profile;else -return mlx5_ib_set_vport_rep (dev, rep); +return mlx5_ib_set_vport_rep (dev, rep, vport_index); ibdev =ib_alloc_device (mlx5_ib_dev, ib_dev);if (!ibdev)
@@ -64,6 +86,8 @@ mlx5_ib_vport_rep_load(struct mlx5_core_dev *dev, struct mlx5_eswitch_rep *rep) goto fail_add; rep->rep_data[REP_IB ].priv = ibdev; +if (mlx5_lag_is_shared_fdb (dev)) +mlx5_ib_register_peer_vport_reps (dev);return 0 ;
@@ -82,18 +106,45 @@ static void *mlx5_ib_rep_to_dev(struct mlx5_eswitch_rep *rep)static void mlx5_ib_vport_rep_unload (struct mlx5_eswitch_rep *rep) { +struct mlx5_core_dev *mdev =mlx5_eswitch_get_core_dev (rep->esw);struct mlx5_ib_dev *dev =mlx5_ib_rep_to_dev (rep); +int vport_index = rep->vport_index;struct mlx5_ib_port *port; - port = &dev->port[rep->vport_index]; +if (WARN_ON (!mdev)) +return ; + +if (mlx5_lag_is_shared_fdb (mdev) && + !mlx5_lag_is_master (mdev)) { +struct mlx5_core_dev *peer_mdev; + +if (rep->vport ==MLX5_VPORT_UPLINK ) +return ; + peer_mdev =mlx5_lag_get_peer_mdev (mdev); + vport_index +=mlx5_eswitch_get_total_vports (peer_mdev); + } + +if (!dev) +return ; + + port = &dev->port[vport_index];write_lock (&port->roce.netdev_lock); port->roce.netdev =NULL ;write_unlock (&port->roce.netdev_lock); rep->rep_data[REP_IB ].priv =NULL ; port->rep =NULL ; -if (rep->vport ==MLX5_VPORT_UPLINK ) +if (rep->vport ==MLX5_VPORT_UPLINK ) { +struct mlx5_core_dev *peer_mdev; +struct mlx5_eswitch *esw; + +if (mlx5_lag_is_shared_fdb (mdev)) { + peer_mdev =mlx5_lag_get_peer_mdev (mdev); + esw = peer_mdev->priv.eswitch; +mlx5_eswitch_unregister_vport_reps (esw,REP_IB ); + }__mlx5_ib_remove (dev, dev->profile,MLX5_IB_STAGE_MAX ); + } }static const struct mlx5_eswitch_rep_ops rep_ops = {
@@ -102,6 +153,18 @@ static const struct mlx5_eswitch_rep_ops rep_ops = { .get_proto_dev = mlx5_ib_rep_to_dev, }; +static void mlx5_ib_register_peer_vport_reps (struct mlx5_core_dev *mdev) +{ +struct mlx5_core_dev *peer_mdev =mlx5_lag_get_peer_mdev (mdev); +struct mlx5_eswitch *esw; + +if (!peer_mdev) +return ; + + esw = peer_mdev->priv.eswitch; +mlx5_eswitch_register_vport_reps (esw, &rep_ops,REP_IB ); +} +struct net_device *mlx5_ib_get_rep_netdev (struct mlx5_eswitch *esw,u16 vport_num) {
drivers/infiniband/hw/mlx5/main.c +26 -18
106 lines not on this page · raw message
Signature · 1 line
2.31.1
PATCH 6/14 net/mlx5: E-Switch, set flow source for send to uplink rule
In reply to Saeed MahameedFrom: Ariel Levkovich <[email protected]>
Set the flow source param to local vport for the uplink rep send-to-vport rule.
This will comply with the recent changes in SW steering that use the flow source as an indication for the rule type - rx or tx.
Since the uplink send-to-vport rule is forwarding traffic to the wire it has to indicate that it is an sx rule and can't use the any port value in the flow source.
--- drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c | 3 +++ 1 file changed, 3 insertions(+)
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +3 -0
@@ -963,6 +963,9 @@ mlx5_eswitch_add_send_to_vport_rule(struct mlx5_eswitch *on_esw, dest.vport.flags |=MLX5_FLOW_DEST_VPORT_VHCA_ID ; flow_act.action =MLX5_FLOW_CONTEXT_ACTION_FWD_DEST ; +if (rep->vport ==MLX5_VPORT_UPLINK ) + spec->flow_context.flow_source =MLX5_FLOW_CONTEXT_FLOW_SOURCE_LOCAL_VPORT ; + flow_rule =mlx5_add_flow_rules (on_esw->fdb_table.offloads.slow_fdb, spec, &flow_act, &dest,1 );if (IS_ERR (flow_rule))
Signature · 1 line
2.31.1
PATCH 7/14 net/mlx5e: Add an option to create a shared mapping
In reply to Saeed MahameedFrom: Roi Dayan <[email protected]>
The shared mapping is identified by an id and type.
--- .../ethernet/mellanox/mlx5/core/en/mapping.c | 45 +++++++++++++++++++ .../ethernet/mellanox/mlx5/core/en/mapping.h | 5 +++ 2 files changed, 50 insertions(+)
drivers/net/ethernet/mellanox/mlx5/core/en/mapping.c +45 -0
@@ -5,11 +5,15 @@#include <linux/slab.h> #include <linux/xarray.h> #include <linux/hashtable.h> +#include <linux/refcount.h> #include "mapping.h" #define MAPPING_GRACE_PERIOD 2000 +static LIST_HEAD (shared_ctx_list); +static DEFINE_MUTEX (shared_ctx_lock); +struct mapping_ctx {struct xarray xarray; DECLARE_HASHTABLE(ht, 8);
@@ -20,6 +24,10 @@ struct mapping_ctx {struct delayed_work dwork;struct list_head pending_list;spinlock_t pending_list_lock;/* Guards pending list */ +u64 id; +u8 type; +struct list_head list; +refcount_t refcount; };struct mapping_item {
@@ -205,11 +213,48 @@ mapping_create(size_t data_size, u32 max_id, bool delayed_removal)mutex_init (&ctx->lock);xa_init_flags (&ctx->xarray,XA_FLAGS_ALLOC1 ); +refcount_set (&ctx->refcount,1 ); +INIT_LIST_HEAD (&ctx->list); + +return ctx; +} + +struct mapping_ctx * +mapping_create_for_id (u64 id,u8 type,size_t data_size,u32 max_id,bool delayed_removal) +{ +struct mapping_ctx *ctx; + +mutex_lock (&shared_ctx_lock); +list_for_each_entry (ctx, &shared_ctx_list, list) { +if (ctx->id == id && ctx->type == type) { +if (refcount_inc_not_zero (&ctx->refcount)) + goto unlock; +break ; + } + } + + ctx =mapping_create (data_size, max_id, delayed_removal); +if (IS_ERR (ctx)) + goto unlock; + + ctx->id = id; + ctx->type = type; +list_add (&ctx->list, &shared_ctx_list); + +unlock: +mutex_unlock (&shared_ctx_lock);return ctx; }void mapping_destroy (struct mapping_ctx *ctx) { +if (!refcount_dec_and_test (&ctx->refcount)) +return ; + +mutex_lock (&shared_ctx_lock); +list_del (&ctx->list); +mutex_unlock (&shared_ctx_lock); +mapping_flush_work (ctx);xa_destroy (&ctx->xarray);mutex_destroy (&ctx->lock);
drivers/net/ethernet/mellanox/mlx5/core/en/mapping.h +5 -0
@@ -24,4 +24,9 @@ struct mapping_ctx *mapping_create(size_t data_size, u32 max_id, bool delayed_removal);void mapping_destroy (struct mapping_ctx *ctx); +/* adds mapping with an id or get an existing mapping with the same id +*/ +struct mapping_ctx * +mapping_create_for_id (u64 id,u8 type,size_t data_size,u32 max_id,bool delayed_removal); +#endif /* __MLX5_MAPPING_H__ */
Signature · 1 line
2.31.1
PATCH 8/14 net/mlx5e: Use shared mappings for restoring from metadata
In reply to Saeed MahameedFrom: Roi Dayan <[email protected]>
FTEs are added with mapped metadata which is saved per eswitch. When uplink reps are bonded and we are in a single FDB mode, we could fail to find metadata which was stored on one eswitch mapping but not the other or with a different id.
To resolve this issue use shared mapping between eswitch ports. We do not have any conflict using a single mapping, for a type, between the ports.
--- .../ethernet/mellanox/mlx5/core/en/tc_ct.c | 9 ++++++-- .../net/ethernet/mellanox/mlx5/core/en_tc.c | 21 ++++++++++++++----- .../net/ethernet/mellanox/mlx5/core/eswitch.h | 8 +++++++ .../mellanox/mlx5/core/eswitch_offloads.c | 11 +++++++--- 4 files changed, 39 insertions(+), 10 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en/tc_ct.c +7 -2
@@ -2138,6 +2138,7 @@ mlx5_tc_ct_init(struct mlx5e_priv *priv, struct mlx5_fs_chains *chains,struct mlx5_tc_ct_priv *ct_priv;struct mlx5_core_dev *dev;const char *msg; +u64 mapping_id;int err; dev = priv->mdev;
@@ -2153,13 +2154,17 @@ mlx5_tc_ct_init(struct mlx5e_priv *priv, struct mlx5_fs_chains *chains,if (!ct_priv) goto err_alloc; - ct_priv->zone_mapping =mapping_create (sizeof (u16),0 , true); + mapping_id =mlx5_query_nic_system_image_guid (dev); + + ct_priv->zone_mapping =mapping_create_for_id (mapping_id,MAPPING_TYPE_ZONE , +sizeof (u16),0 , true);if (IS_ERR (ct_priv->zone_mapping)) { err =PTR_ERR (ct_priv->zone_mapping); goto err_mapping_zone; } - ct_priv->labels_mapping =mapping_create (sizeof (u32) *4 ,0 , true); + ct_priv->labels_mapping =mapping_create_for_id (mapping_id,MAPPING_TYPE_LABELS , +sizeof (u32) *4 ,0 , true);if (IS_ERR (ct_priv->labels_mapping)) { err =PTR_ERR (ct_priv->labels_mapping); goto err_mapping_labels;
drivers/net/ethernet/mellanox/mlx5/core/en_tc.c +16 -5
@@ -4848,6 +4848,7 @@ int mlx5e_tc_nic_init(struct mlx5e_priv *priv)struct mlx5_core_dev *dev = priv->mdev;struct mapping_ctx *chains_mapping;struct mlx5_chains_attr attr = {}; +u64 mapping_id;int err;mlx5e_mod_hdr_tbl_init (&tc->mod_hdr);
@@ -4861,8 +4862,12 @@ int mlx5e_tc_nic_init(struct mlx5e_priv *priv)lockdep_set_class (&tc->ht.mutex, &tc_ht_lock_key); - chains_mapping =mapping_create (sizeof (struct mlx5_mapped_obj ), -MLX5E_TC_TABLE_CHAIN_TAG_MASK , true); + mapping_id =mlx5_query_nic_system_image_guid (dev); + + chains_mapping =mapping_create_for_id (mapping_id,MAPPING_TYPE_CHAIN , +sizeof (struct mlx5_mapped_obj ), +MLX5E_TC_TABLE_CHAIN_TAG_MASK , true); +if (IS_ERR (chains_mapping)) { err =PTR_ERR (chains_mapping); goto err_mapping;
@@ -4951,6 +4956,7 @@ int mlx5e_tc_esw_init(struct rhashtable *tc_ht)struct mapping_ctx *mapping;struct mlx5_eswitch *esw;struct mlx5e_priv *priv; +u64 mapping_id;int err =0 ; uplink_priv = container_of(tc_ht, struct mlx5_rep_uplink_priv, tc_ht);
@@ -4967,8 +4973,12 @@ int mlx5e_tc_esw_init(struct rhashtable *tc_ht) uplink_priv->esw_psample =mlx5_esw_sample_init (netdev_priv (priv->netdev));#endif - mapping =mapping_create (sizeof (struct tunnel_match_key ), -TUNNEL_INFO_BITS_MASK , true); + mapping_id =mlx5_query_nic_system_image_guid (esw->dev); + + mapping =mapping_create_for_id (mapping_id,MAPPING_TYPE_TUNNEL , +sizeof (struct tunnel_match_key ), +TUNNEL_INFO_BITS_MASK , true); +if (IS_ERR (mapping)) { err =PTR_ERR (mapping); goto err_tun_mapping;
@@ -4976,7 +4986,8 @@ int mlx5e_tc_esw_init(struct rhashtable *tc_ht) uplink_priv->tunnel_mapping = mapping;/* 0xFFF is reserved for stack devices slow path table mark */ - mapping =mapping_create (sz_enc_opts,ENC_OPTS_BITS_MASK -1 , true); + mapping =mapping_create_for_id (mapping_id,MAPPING_TYPE_TUNNEL_ENC_OPTS , + sz_enc_opts,ENC_OPTS_BITS_MASK -1 , true);if (IS_ERR (mapping)) { err =PTR_ERR (mapping); goto err_enc_opts_mapping;
drivers/net/ethernet/mellanox/mlx5/core/eswitch.h +8 -0
@@ -86,6 +86,14 @@ struct mlx5_mapped_obj {#define esw_chains (esw) \ ((esw)->fdb_table.offloads.esw_chains_priv) +enum { +MAPPING_TYPE_CHAIN , +MAPPING_TYPE_TUNNEL , +MAPPING_TYPE_TUNNEL_ENC_OPTS , +MAPPING_TYPE_LABELS , +MAPPING_TYPE_ZONE , +}; +struct vport_ingress {struct mlx5_flow_table *acl;struct mlx5_flow_handle *allow_rule;
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +8 -3
@@ -2787,6 +2787,7 @@ int esw_offloads_enable(struct mlx5_eswitch *esw)struct mapping_ctx *reg_c0_obj_pool;struct mlx5_vport *vport;unsigned long i; +u64 mapping_id;int err; if (MLX5_CAP_ESW_FLOWTABLE_FDB(esw->dev, reformat) &&
@@ -2810,9 +2811,13 @@ int esw_offloads_enable(struct mlx5_eswitch *esw)if (err) goto err_vport_metadata; - reg_c0_obj_pool =mapping_create (sizeof (struct mlx5_mapped_obj ), -ESW_REG_C0_USER_DATA_METADATA_MASK , - true); + mapping_id =mlx5_query_nic_system_image_guid (esw->dev); + + reg_c0_obj_pool =mapping_create_for_id (mapping_id,MAPPING_TYPE_CHAIN , +sizeof (struct mlx5_mapped_obj ), +ESW_REG_C0_USER_DATA_METADATA_MASK , + true); +if (IS_ERR (reg_c0_obj_pool)) { err =PTR_ERR (reg_c0_obj_pool); goto err_pool;
Signature · 1 line
2.31.1
PATCH 9/14 net/mlx5: E-Switch, Add event callback for representors
In reply to Saeed MahameedFrom: Mark Bloch <[email protected]>
This callback will allow to notify representors about relevant events when in OFFLOADS mode. In downstream patches, this will be used to notify about PAIR/UNPAIR devcom events.
--- .../mellanox/mlx5/core/eswitch_offloads.c | 50 +++++++++++++++++-- include/linux/mlx5/eswitch.h | 9 ++++ 2 files changed, 56 insertions(+), 3 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +47 -3
@@ -2316,11 +2316,22 @@ void esw_offloads_unload_rep(struct mlx5_eswitch *esw, u16 vport_num)#define ESW_OFFLOADS_DEVCOM_PAIR (0)#define ESW_OFFLOADS_DEVCOM_UNPAIR (1) -static int mlx5_esw_offloads_pair (struct mlx5_eswitch *esw, -struct mlx5_eswitch *peer_esw) +static void mlx5_esw_offloads_rep_event_unpair (struct mlx5_eswitch *esw) { +const struct mlx5_eswitch_rep_ops *ops; +struct mlx5_eswitch_rep *rep; +unsigned long i; +u8 rep_type; -return esw_add_fdb_peer_miss_rules (esw, peer_esw->dev); +mlx5_esw_for_each_rep (esw, i, rep) { + rep_type =NUM_REP_TYPES ; +while (rep_type--) { + ops = esw->offloads.rep_ops[rep_type]; +if (atomic_read (&rep->rep_data[rep_type].state) ==REP_LOADED && + ops->event) + ops->event (esw, rep,MLX5_SWITCHDEV_EVENT_UNPAIR ,NULL ); + } + } }static void mlx5_esw_offloads_unpair (struct mlx5_eswitch *esw)
@@ -2328,9 +2339,42 @@ static void mlx5_esw_offloads_unpair(struct mlx5_eswitch *esw)#if IS_ENABLED (CONFIG_MLX5_CLS_ACT )mlx5e_tc_clean_fdb_peer_flows (esw);#endif +mlx5_esw_offloads_rep_event_unpair (esw);esw_del_fdb_peer_miss_rules (esw); } +static int mlx5_esw_offloads_pair (struct mlx5_eswitch *esw, +struct mlx5_eswitch *peer_esw) +{ +const struct mlx5_eswitch_rep_ops *ops; +struct mlx5_eswitch_rep *rep; +unsigned long i; +u8 rep_type; +int err; + + err =esw_add_fdb_peer_miss_rules (esw, peer_esw->dev); +if (err) +return err; + +mlx5_esw_for_each_rep (esw, i, rep) { +for (rep_type =0 ; rep_type <NUM_REP_TYPES ; rep_type++) { + ops = esw->offloads.rep_ops[rep_type]; +if (atomic_read (&rep->rep_data[rep_type].state) ==REP_LOADED && + ops->event) { + err = ops->event (esw, rep,MLX5_SWITCHDEV_EVENT_PAIR , peer_esw); +if (err) + goto err_out; + } + } + } + +return 0 ; + +err_out: +mlx5_esw_offloads_unpair (esw); +return err; +} +static int mlx5_esw_offloads_set_ns_peer (struct mlx5_eswitch *esw,struct mlx5_eswitch *peer_esw,bool pair)
include/linux/mlx5/eswitch.h +9 -0
@@ -29,11 +29,20 @@ enum { REP_LOADED, }; +enum mlx5_switchdev_event { +MLX5_SWITCHDEV_EVENT_PAIR , +MLX5_SWITCHDEV_EVENT_UNPAIR , +}; +struct mlx5_eswitch_rep ;struct mlx5_eswitch_rep_ops {int (*load)(struct mlx5_core_dev *dev,struct mlx5_eswitch_rep *rep);void (*unload)(struct mlx5_eswitch_rep *rep);void *(*get_proto_dev)(struct mlx5_eswitch_rep *rep); +int (*event)(struct mlx5_eswitch *esw, +struct mlx5_eswitch_rep *rep, +enum mlx5_switchdev_event event, +void *data); };struct mlx5_eswitch_rep_data {
Signature · 1 line
2.31.1
PATCH 10/14 net/mlx5: Add send to vport rules on paired device
In reply to Saeed MahameedFrom: Mark Bloch <[email protected]>
When two mlx5 devices are paired in switchdev mode, always offload the send-to-vport rule to the peer E-Switch. This allows to abstract the logic when this is really necessary (single FDB) and combine the logic of both cases into one.
--- .../net/ethernet/mellanox/mlx5/core/en_rep.c | 86 ++++++++++++++++++- .../net/ethernet/mellanox/mlx5/core/en_rep.h | 2 + .../mellanox/mlx5/core/eswitch_offloads.c | 16 +++- 3 files changed, 101 insertions(+), 3 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en_rep.c +85 -1
@@ -49,6 +49,7 @@#include "en/devlink.h" #include "fs_core.h" #include "lib/mlx5.h" +#include "lib/devcom.h" #define CREATE_TRACE_POINTS #include "diag/en_rep_tracepoint.h" #include "en_accel/ipsec.h"
@@ -310,6 +311,8 @@ static void mlx5e_sqs2vport_stop(struct mlx5_eswitch *esw, rpriv =mlx5e_rep_to_rep_priv (rep);list_for_each_entry_safe (rep_sq, tmp, &rpriv->vport_sqs_list, list) {mlx5_eswitch_del_send_to_vport_rule (rep_sq->send_to_vport_rule); +if (rep_sq->send_to_vport_rule_peer) +mlx5_eswitch_del_send_to_vport_rule (rep_sq->send_to_vport_rule_peer);list_del (&rep_sq->list);kfree (rep_sq); }
@@ -319,6 +322,7 @@ static int mlx5e_sqs2vport_start(struct mlx5_eswitch *esw,struct mlx5_eswitch_rep *rep, u32 *sqns_array, int sqns_num) { + struct mlx5_eswitch *peer_esw = NULL;struct mlx5_flow_handle *flow_rule;struct mlx5e_rep_priv *rpriv;struct mlx5e_rep_sq *rep_sq;
@@ -329,6 +333,10 @@ static int mlx5e_sqs2vport_start(struct mlx5_eswitch *esw,return 0 ; rpriv =mlx5e_rep_to_rep_priv (rep); +if (mlx5_devcom_is_paired (esw->dev->priv.devcom,MLX5_DEVCOM_ESW_OFFLOADS )) + peer_esw =mlx5_devcom_get_peer_data (esw->dev->priv.devcom, +MLX5_DEVCOM_ESW_OFFLOADS ); +for (i =0 ; i < sqns_num; i++) { rep_sq =kzalloc (sizeof (*rep_sq),GFP_KERNEL );if (!rep_sq) {
@@ -345,12 +353,34 @@ static int mlx5e_sqs2vport_start(struct mlx5_eswitch *esw, goto out_err; } rep_sq->send_to_vport_rule = flow_rule; + rep_sq->sqn = sqns_array[i]; + +if (peer_esw) { + flow_rule =mlx5_eswitch_add_send_to_vport_rule (peer_esw, esw, + rep, sqns_array[i]); +if (IS_ERR (flow_rule)) { + err =PTR_ERR (flow_rule); +mlx5_eswitch_del_send_to_vport_rule (rep_sq->send_to_vport_rule); +kfree (rep_sq); + goto out_err; + } + rep_sq->send_to_vport_rule_peer = flow_rule; + } +list_add (&rep_sq->list, &rpriv->vport_sqs_list); } + +if (peer_esw) +mlx5_devcom_release_peer_data (esw->dev->priv.devcom,MLX5_DEVCOM_ESW_OFFLOADS ); +return 0 ; out_err:mlx5e_sqs2vport_stop (esw, rep); + +if (peer_esw) +mlx5_devcom_release_peer_data (esw->dev->priv.devcom,MLX5_DEVCOM_ESW_OFFLOADS ); +return err; }
@@ -1264,10 +1294,64 @@ static void *mlx5e_vport_rep_get_proto_dev(struct mlx5_eswitch_rep *rep)return rpriv->netdev; } +static void mlx5e_vport_rep_event_unpair (struct mlx5_eswitch_rep *rep) +{ +struct mlx5e_rep_priv *rpriv; +struct mlx5e_rep_sq *rep_sq; + + rpriv =mlx5e_rep_to_rep_priv (rep); +list_for_each_entry (rep_sq, &rpriv->vport_sqs_list, list) { +if (!rep_sq->send_to_vport_rule_peer) +continue ; +mlx5_eswitch_del_send_to_vport_rule (rep_sq->send_to_vport_rule_peer); + rep_sq->send_to_vport_rule_peer =NULL ; + } +} + +static int mlx5e_vport_rep_event_pair (struct mlx5_eswitch *esw, +struct mlx5_eswitch_rep *rep, +struct mlx5_eswitch *peer_esw) +{ +struct mlx5_flow_handle *flow_rule; +struct mlx5e_rep_priv *rpriv; +struct mlx5e_rep_sq *rep_sq; + + rpriv =mlx5e_rep_to_rep_priv (rep); +list_for_each_entry (rep_sq, &rpriv->vport_sqs_list, list) { +if (rep_sq->send_to_vport_rule_peer) +continue ; + flow_rule =mlx5_eswitch_add_send_to_vport_rule (peer_esw, esw, rep, rep_sq->sqn); +if (IS_ERR (flow_rule)) + goto err_out; + rep_sq->send_to_vport_rule_peer = flow_rule; + } + +return 0 ; +err_out: +mlx5e_vport_rep_event_unpair (rep); +return PTR_ERR (flow_rule); +} + +static int mlx5e_vport_rep_event (struct mlx5_eswitch *esw, +struct mlx5_eswitch_rep *rep, +enum mlx5_switchdev_event event, +void *data) +{ +int err =0 ; + +if (event ==MLX5_SWITCHDEV_EVENT_PAIR ) + err =mlx5e_vport_rep_event_pair (esw, rep, data); +else if (event ==MLX5_SWITCHDEV_EVENT_UNPAIR ) +mlx5e_vport_rep_event_unpair (rep); + +return err; +} +static const struct mlx5_eswitch_rep_ops rep_ops = { .load = mlx5e_vport_rep_load, .unload = mlx5e_vport_rep_unload, - .get_proto_dev = mlx5e_vport_rep_get_proto_dev + .get_proto_dev = mlx5e_vport_rep_get_proto_dev, + .event = mlx5e_vport_rep_event, }; static int mlx5e_rep_probe(struct auxiliary_device *adev,
drivers/net/ethernet/mellanox/mlx5/core/en_rep.h +2 -0
@@ -207,6 +207,8 @@ struct mlx5e_encap_entry {struct mlx5e_rep_sq {struct mlx5_flow_handle *send_to_vport_rule; +struct mlx5_flow_handle *send_to_vport_rule_peer; +u32 sqn;struct list_head list; };
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +14 -2
@@ -1616,7 +1616,18 @@ static int esw_create_offloads_fdb_tables(struct mlx5_eswitch *esw) goto ns_err; } - table_size = esw->total_vports *MAX_SQ_NVPORTS +MAX_PF_SQ + +/* To be strictly correct: +* MLX5_MAX_PORTS * (esw->total_vports * MAX_SQ_NVPORTS + MAX_PF_SQ) +* should be: +* esw->total_vports * MAX_SQ_NVPORTS + MAX_PF_SQ + +* peer_esw->total_vports * MAX_SQ_NVPORTS + MAX_PF_SQ +* but as the peer device might not be in switchdev mode it's not +* possible. We use the fact that by default FW sets max vfs and max sfs +* to the same value on both devices. If it needs to be changed in the future note +* the peer miss group should also be created based on the number of +* total vports of the peer (currently is also uses esw->total_vports). +*/ + table_size =MLX5_MAX_PORTS * (esw->total_vports *MAX_SQ_NVPORTS +MAX_PF_SQ ) +MLX5_ESW_MISS_FLOWS + esw->total_vports + esw->esw_funcs.num_vfs;/* create the slow path fdb with encap set, so further table instances
@@ -1673,7 +1684,8 @@ static int esw_create_offloads_fdb_tables(struct mlx5_eswitch *esw) source_eswitch_owner_vhca_id_valid, 1); } - ix = esw->total_vports *MAX_SQ_NVPORTS +MAX_PF_SQ ; +/* See comment above table_size calculation */ + ix =MLX5_MAX_PORTS * (esw->total_vports *MAX_SQ_NVPORTS +MAX_PF_SQ );MLX5_SET (create_flow_group_in, flow_group_in, start_flow_index,0 );MLX5_SET (create_flow_group_in, flow_group_in, end_flow_index, ix -1 );
Signature · 1 line
2.31.1
PATCH 11/14 net/mlx5: Lag, properly lock eswitch if needed
In reply to Saeed MahameedFrom: Mark Bloch <[email protected]>
Currently when doing hardware lag we check the eswitch mode but as this isn't done under a lock the check isn't valid.
As the code needs to sync between two different devices an extra care is needed.
- When going to change eswitch mode, if hardware lag is active destroy it.
- While changing eswitch modes block any hardware bond creation.
- Delay handling bonding events until there are no mode changes in progress.
- When attaching a new mdev to lag, block until there is no mode change in progress. In order for the mode change to finish the interface lock will have to be taken. Release the lock and sleep for 100ms to allow forward progress. As this is a very rare condition (can happen if the user unbinds and binds a PCI function while also changing eswitch mode of the other PCI function) it has no real world impact.
As taking multiple eswitch mode locks is now required lockdep will complain about a possible deadlock. Register a key per eswitch to make lockdep happy.
--- .../net/ethernet/mellanox/mlx5/core/eswitch.c | 24 +++++- .../net/ethernet/mellanox/mlx5/core/eswitch.h | 5 ++ .../mellanox/mlx5/core/eswitch_offloads.c | 5 +- drivers/net/ethernet/mellanox/mlx5/core/lag.c | 83 ++++++++++++++++--- drivers/net/ethernet/mellanox/mlx5/core/lag.h | 1 + .../net/ethernet/mellanox/mlx5/core/main.c | 5 +- .../ethernet/mellanox/mlx5/core/mlx5_core.h | 2 + 7 files changed, 107 insertions(+), 18 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/eswitch.c +20 -4
@@ -1458,8 +1458,6 @@ int mlx5_eswitch_enable_locked(struct mlx5_eswitch *esw, int mode, int num_vfs) esw->mode = mode; -mlx5_lag_update (esw->dev); -if (mode ==MLX5_ESWITCH_LEGACY ) { err =esw_legacy_enable (esw); }else {
@@ -1506,6 +1504,7 @@ int mlx5_eswitch_enable(struct mlx5_eswitch *esw, int num_vfs)if (!mlx5_esw_allowed (esw))return 0 ; +mlx5_lag_disable_change (esw->dev);down_write (&esw->mode_lock);if (esw->mode ==MLX5_ESWITCH_NONE ) { ret =mlx5_eswitch_enable_locked (esw,MLX5_ESWITCH_LEGACY , num_vfs);
@@ -1519,6 +1518,7 @@ int mlx5_eswitch_enable(struct mlx5_eswitch *esw, int num_vfs) esw->esw_funcs.num_vfs = num_vfs; }up_write (&esw->mode_lock); +mlx5_lag_enable_change (esw->dev);return ret; }
@@ -1550,8 +1550,6 @@ void mlx5_eswitch_disable_locked(struct mlx5_eswitch *esw, bool clear_vf) old_mode = esw->mode; esw->mode =MLX5_ESWITCH_NONE ; -mlx5_lag_update (esw->dev); -if (old_mode ==MLX5_ESWITCH_OFFLOADS )mlx5_rescan_drivers (esw->dev);
@@ -1567,10 +1565,12 @@ void mlx5_eswitch_disable(struct mlx5_eswitch *esw, bool clear_vf)if (!mlx5_esw_allowed (esw))return ; +mlx5_lag_disable_change (esw->dev);down_write (&esw->mode_lock);mlx5_eswitch_disable_locked (esw, clear_vf); esw->esw_funcs.num_vfs =0 ;up_write (&esw->mode_lock); +mlx5_lag_enable_change (esw->dev); }static int mlx5_query_hca_cap_host_pf (struct mlx5_core_dev *dev,void *out)
@@ -1759,7 +1759,9 @@ int mlx5_eswitch_init(struct mlx5_core_dev *dev)ida_init (&esw->offloads.vport_metadata_ida);xa_init_flags (&esw->offloads.vhca_map,XA_FLAGS_ALLOC );mutex_init (&esw->state_lock); +lockdep_register_key (&esw->mode_lock_key);init_rwsem (&esw->mode_lock); +lockdep_set_class (&esw->mode_lock, &esw->mode_lock_key); esw->enabled_vports =0 ; esw->mode =MLX5_ESWITCH_NONE ;
@@ -1793,6 +1795,7 @@ void mlx5_eswitch_cleanup(struct mlx5_eswitch *esw) esw->dev->priv.eswitch =NULL ;destroy_workqueue (esw->work_queue); +lockdep_unregister_key (&esw->mode_lock_key);mutex_destroy (&esw->state_lock);WARN_ON (!xa_empty (&esw->offloads.vhca_map));xa_destroy (&esw->offloads.vhca_map);
@@ -2366,9 +2369,22 @@ int mlx5_esw_try_lock(struct mlx5_eswitch *esw)*/ void mlx5_esw_unlock (struct mlx5_eswitch *esw) { +if (!mlx5_esw_allowed (esw)) +return ;up_write (&esw->mode_lock); } +/** +* mlx5_esw_lock() - Take write lock on esw mode lock +* @esw: eswitch device. +*/ +void mlx5_esw_lock (struct mlx5_eswitch *esw) +{ +if (!mlx5_esw_allowed (esw)) +return ; +down_write (&esw->mode_lock); +} +/** * mlx5_eswitch_get_total_vports - Get total vports of the eswitch *
drivers/net/ethernet/mellanox/mlx5/core/eswitch.h +5 -0
@@ -323,6 +323,7 @@ struct mlx5_eswitch {u32 large_group_num; } params;struct blocking_notifier_head n_head; +struct lock_class_key mode_lock_key; };void esw_offloads_disable (struct mlx5_eswitch *esw);
@@ -707,6 +708,7 @@ void mlx5_esw_get(struct mlx5_core_dev *dev);void mlx5_esw_put (struct mlx5_core_dev *dev);int mlx5_esw_try_lock (struct mlx5_eswitch *esw);void mlx5_esw_unlock (struct mlx5_eswitch *esw); +void mlx5_esw_lock (struct mlx5_eswitch *esw);void esw_vport_change_handle_locked (struct mlx5_vport *vport);
@@ -727,6 +729,9 @@ static inline const u32 *mlx5_esw_query_functions(struct mlx5_core_dev *dev)return ERR_PTR (-EOPNOTSUPP ); } +static inline void mlx5_esw_unlock (struct mlx5_eswitch *esw) {return ; } +static inline void mlx5_esw_lock (struct mlx5_eswitch *esw) {return ; } +static inline struct mlx5_flow_handle *esw_add_restore_rule (struct mlx5_eswitch *esw,u32 tag) {
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +4 -1
@@ -3051,10 +3051,11 @@ int mlx5_devlink_eswitch_mode_set(struct devlink *devlink, u16 mode,if (esw_mode_from_devlink (mode, &mlx5_mode))return -EINVAL ; +mlx5_lag_disable_change (esw->dev); err =mlx5_esw_try_lock (esw);if (err <0 ) {NL_SET_ERR_MSG_MOD (extack,"Can't change mode, E-Switch is busy" ); -return err; + goto enable_lag; } cur_mlx5_mode = err; err =0 ;
@@ -3071,6 +3072,8 @@ int mlx5_devlink_eswitch_mode_set(struct devlink *devlink, u16 mode, unlock:mlx5_esw_unlock (esw); +enable_lag: +mlx5_lag_enable_change (esw->dev);return err; }
drivers/net/ethernet/mellanox/mlx5/core/lag.c +72 -11
162 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/lag.h +1 -0
@@ -39,6 +39,7 @@ struct lag_tracker {*/ struct mlx5_lag {u8 flags; +int mode_changes_in_progress;bool shared_fdb;u8 v2p_map[MLX5_MAX_PORTS ];struct kref ref;
drivers/net/ethernet/mellanox/mlx5/core/main.c +3 -2
@@ -1179,6 +1179,7 @@ static int mlx5_load(struct mlx5_core_dev *dev) goto err_ec; } +mlx5_lag_add_mdev (dev); err =mlx5_sriov_attach (dev);if (err) {mlx5_core_err (dev,"sriov init failed %d\n" , err);
@@ -1186,11 +1187,11 @@ static int mlx5_load(struct mlx5_core_dev *dev) }mlx5_sf_dev_table_create (dev); -mlx5_lag_add_mdev (dev);return 0 ; err_sriov: +mlx5_lag_remove_mdev (dev);mlx5_ec_cleanup (dev); err_ec:mlx5_sf_hw_table_destroy (dev);
@@ -1222,9 +1223,9 @@ static int mlx5_load(struct mlx5_core_dev *dev)static void mlx5_unload (struct mlx5_core_dev *dev) { -mlx5_lag_remove_mdev (dev);mlx5_sf_dev_table_destroy (dev);mlx5_sriov_detach (dev); +mlx5_lag_remove_mdev (dev);mlx5_ec_cleanup (dev);mlx5_sf_hw_table_destroy (dev);mlx5_vhca_event_stop (dev);
drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h +2 -0
@@ -168,6 +168,8 @@ void mlx5_lag_add_netdev(struct mlx5_core_dev *dev, struct net_device *netdev);void mlx5_lag_remove_netdev (struct mlx5_core_dev *dev,struct net_device *netdev);void mlx5_lag_add_mdev (struct mlx5_core_dev *dev);void mlx5_lag_remove_mdev (struct mlx5_core_dev *dev); +void mlx5_lag_disable_change (struct mlx5_core_dev *dev); +void mlx5_lag_enable_change (struct mlx5_core_dev *dev);int mlx5_events_init (struct mlx5_core_dev *dev);void mlx5_events_cleanup (struct mlx5_core_dev *dev);
Signature · 1 line
2.31.1
PATCH 12/14 net/mlx5: Lag, move lag destruction to a workqueue
In reply to Saeed MahameedFrom: Mark Bloch <[email protected]>
If a netdev is removed from the lag the lag should be destroyed. With downstream patches this might trigger a reconfiguration of representors on a different eswitch and such we don't have the proper locking to so from this path. Move the destruction to be done by the workqueue.
As the destruction won't affect the netdev side it okay to do so. The RDMA side will be reconfigured and it already coded to handle such reconfiguration.
--- drivers/net/ethernet/mellanox/mlx5/core/lag.c | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/lag.c +9 -8
@@ -371,12 +371,13 @@ static void mlx5_do_bond(struct mlx5_lag *ldev)bool do_bond, roce_lag;int err; -if (!mlx5_lag_is_ready (ldev)) -return ; - - tracker = ldev->tracker; +if (!mlx5_lag_is_ready (ldev)) { + do_bond = false; + }else { + tracker = ldev->tracker; - do_bond = tracker.is_bonded &&mlx5_lag_check_prereq (ldev); + do_bond = tracker.is_bonded &&mlx5_lag_check_prereq (ldev); + }if (do_bond && !__mlx5_lag_is_active (ldev)) { roce_lag = !mlx5_sriov_is_enabled(dev0) &&
@@ -733,11 +734,11 @@ void mlx5_lag_remove_netdev(struct mlx5_core_dev *dev,if (!ldev)return ; -if (__mlx5_lag_is_active (ldev)) -mlx5_disable_lag (ldev); -mlx5_ldev_remove_netdev (ldev, netdev); ldev->flags &= ~MLX5_LAG_FLAG_READY ; + +if (__mlx5_lag_is_active (ldev)) +mlx5_queue_bond_work (ldev,0 ); }/* Must be called with intf_mutex held */
Signature · 1 line
2.31.1
PATCH 13/14 net/mlx5/ E-Switch, add logic to enable shared FDB
In reply to Saeed MahameedFrom: Mark Bloch <[email protected]>
Shared FDB allows to direct traffic from all the vports in the HCA to a single eswitch. In order to do that three things are needed.
1) Point the ingress ACL of the slave uplink to that of the master. With this, wire traffic from both uplinks will reach the same eswitch with the same metadata where a single steering rule can catch traffic from both ports.
2) Set the FDB root flow table of the slave's eswitch to that of the master. As this flow table can change dynamically make sure to sync it on any set root flow table FDB command.
This will make sure traffic from SFs, VFs, ECPFs and PFs reach the master eswitch.
3) Split wire traffic at the eswitch manager egress ACL so that it's directed to the native eswitch manager. We only treat wire traffic from both ports the same at the eswitch level. If such traffic wasn't handled in the eswitch it needs to reach the right representor to be processed by software. For example LACP packets should *always* reach the right uplink representor for correct operation.
--- .../mellanox/mlx5/core/esw/acl/egress_ofld.c | 16 + .../net/ethernet/mellanox/mlx5/core/eswitch.h | 25 ++ .../mellanox/mlx5/core/eswitch_offloads.c | 293 ++++++++++++++++++ .../net/ethernet/mellanox/mlx5/core/fs_cmd.c | 58 +++- .../net/ethernet/mellanox/mlx5/core/fs_core.c | 2 +- .../net/ethernet/mellanox/mlx5/core/fs_core.h | 2 + 6 files changed, 394 insertions(+), 2 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/esw/acl/egress_ofld.c +16 -0
@@ -15,6 +15,15 @@ static void esw_acl_egress_ofld_fwd2vport_destroy(struct mlx5_vport *vport) vport->egress.offloads.fwd_rule =NULL ; } +static void esw_acl_egress_ofld_bounce_rule_destroy (struct mlx5_vport *vport) +{ +if (!vport->egress.offloads.bounce_rule) +return ; + +mlx5_del_flow_rules (vport->egress.offloads.bounce_rule); + vport->egress.offloads.bounce_rule =NULL ; +} +static int esw_acl_egress_ofld_fwd2vport_create (struct mlx5_eswitch *esw,struct mlx5_vport *vport,struct mlx5_flow_destination *fwd_dest)
@@ -87,6 +96,7 @@ static void esw_acl_egress_ofld_rules_destroy(struct mlx5_vport *vport) {esw_acl_egress_vlan_destroy (vport);esw_acl_egress_ofld_fwd2vport_destroy (vport); +esw_acl_egress_ofld_bounce_rule_destroy (vport); } static int esw_acl_egress_ofld_groups_create(struct mlx5_eswitch *esw,
@@ -145,6 +155,12 @@ static void esw_acl_egress_ofld_groups_destroy(struct mlx5_vport *vport)mlx5_destroy_flow_group (vport->egress.offloads.fwd_grp); vport->egress.offloads.fwd_grp =NULL ; } + +if (!IS_ERR_OR_NULL (vport->egress.offloads.bounce_grp)) { +mlx5_destroy_flow_group (vport->egress.offloads.bounce_grp); + vport->egress.offloads.bounce_grp =NULL ; + } +esw_acl_egress_vlan_grp_destroy (vport); }
drivers/net/ethernet/mellanox/mlx5/core/eswitch.h +25 -0
@@ -132,6 +132,8 @@ struct vport_egress {struct {struct mlx5_flow_group *fwd_grp;struct mlx5_flow_handle *fwd_rule; +struct mlx5_flow_handle *bounce_rule; +struct mlx5_flow_group *bounce_grp; } offloads; }; };
@@ -714,6 +716,12 @@ void esw_vport_change_handle_locked(struct mlx5_vport *vport);bool mlx5_esw_offloads_controller_valid (const struct mlx5_eswitch *esw,u32 controller); +int mlx5_eswitch_offloads_config_single_fdb (struct mlx5_eswitch *master_esw, +struct mlx5_eswitch *slave_esw); +void mlx5_eswitch_offloads_destroy_single_fdb (struct mlx5_eswitch *master_esw, +struct mlx5_eswitch *slave_esw); +int mlx5_eswitch_reload_reps (struct mlx5_eswitch *esw); +#else /* CONFIG_MLX5_ESWITCH */ /* eswitch API stubs */ static inline int mlx5_eswitch_init (struct mlx5_core_dev *dev) {return 0 ; }
@@ -744,6 +752,23 @@ mlx5_esw_vport_to_devlink_port_index(const struct mlx5_core_dev *dev, {return vport_num; } + +static inline int +mlx5_eswitch_offloads_config_single_fdb (struct mlx5_eswitch *master_esw, +struct mlx5_eswitch *slave_esw) +{ +return 0 ; +} + +static inline void +mlx5_eswitch_offloads_destroy_single_fdb (struct mlx5_eswitch *master_esw, +struct mlx5_eswitch *slave_esw) {} + +static inline int +mlx5_eswitch_reload_reps (struct mlx5_eswitch *esw) +{ +return 0 ; +}#endif /* CONFIG_MLX5_ESWITCH */ #endif /* __MLX5_ESWITCH_H__ */
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +293 -0
305 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/fs_cmd.c +57 -1
81 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/fs_core.c +1 -1
@@ -413,7 +413,7 @@ static bool check_valid_spec(const struct mlx5_flow_spec *spec)return true; } -static struct mlx5_flow_root_namespace *find_root (struct fs_node *node) +struct mlx5_flow_root_namespace *find_root (struct fs_node *node) {struct fs_node *root;struct mlx5_flow_namespace *ns;
drivers/net/ethernet/mellanox/mlx5/core/fs_core.h +2 -0
@@ -294,6 +294,8 @@ void mlx5_fs_egress_acls_cleanup(struct mlx5_core_dev *dev);int mlx5_fs_ingress_acls_init (struct mlx5_core_dev *dev,int total_vports);void mlx5_fs_ingress_acls_cleanup (struct mlx5_core_dev *dev); +struct mlx5_flow_root_namespace *find_root (struct fs_node *node); +#define fs_get_obj (v, _node) {v = container_of((_node), typeof(*v), node); }#define fs_list_for_each_entry (pos, root) \
Signature · 1 line
2.31.1
PATCH 14/14 net/mlx5: Lag, Create shared FDB when in switchdev mode
In reply to Saeed MahameedFrom: Mark Bloch <[email protected]>
If both eswitches are in switchdev mode and the uplink representors are enslaved to the same bond device create a shared FDB configuration.
When moving to shared FDB mode not only the hardware needs be configured but the RDMA driver needs to reconfigure itself.
When such change is done, unload the RDMA devices, configure the hardware and load the RDMA representors.
When destroying the lag (can happen if a PCI function is unbinded, driver is unloaded or by just removing a netdev from the bond) make sure to restore the system to the previous state only if possible.
For example, if a PCI function is unbinded there is no need to load the representors as the device is going away.
--- drivers/net/ethernet/mellanox/mlx5/core/lag.c | 118 +++++++++++++++--- drivers/net/ethernet/mellanox/mlx5/core/lag.h | 3 +- .../net/ethernet/mellanox/mlx5/core/lag_mp.c | 2 +- 3 files changed, 105 insertions(+), 18 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/lag.c +102 -16
@@ -32,7 +32,9 @@#include <linux/netdevice.h> #include <linux/mlx5/driver.h> +#include <linux/mlx5/eswitch.h> #include <linux/mlx5/vport.h> +#include "lib/devcom.h" #include "mlx5_core.h" #include "eswitch.h" #include "lag.h"
@@ -45,7 +47,7 @@static DEFINE_SPINLOCK (lag_lock);static int mlx5_cmd_create_lag (struct mlx5_core_dev *dev,u8 remap_port1, -u8 remap_port2) +u8 remap_port2,bool shared_fdb) {u32 in[MLX5_ST_SZ_DW (create_lag_in)] = {};void *lag_ctx =MLX5_ADDR_OF (create_lag_in, in, ctx);
@@ -54,6 +56,7 @@ static int mlx5_cmd_create_lag(struct mlx5_core_dev *dev, u8 remap_port1,MLX5_SET (lagc, lag_ctx, tx_remap_affinity_1, remap_port1);MLX5_SET (lagc, lag_ctx, tx_remap_affinity_2, remap_port2); +MLX5_SET (lagc, lag_ctx, fdb_selection_mode, shared_fdb);return mlx5_cmd_exec_in (dev, create_lag, in); }
@@ -224,35 +227,59 @@ void mlx5_modify_lag(struct mlx5_lag *ldev, }static int mlx5_create_lag (struct mlx5_lag *ldev, -struct lag_tracker *tracker) +struct lag_tracker *tracker, +bool shared_fdb) {struct mlx5_core_dev *dev0 = ldev->pf[MLX5_LAG_P1 ].dev; +struct mlx5_core_dev *dev1 = ldev->pf[MLX5_LAG_P2 ].dev; +u32 in[MLX5_ST_SZ_DW (destroy_lag_in)] = {};int err;mlx5_infer_tx_affinity_mapping (tracker, &ldev->v2p_map[MLX5_LAG_P1 ], &ldev->v2p_map[MLX5_LAG_P2 ]); -mlx5_core_info (dev0,"lag map port 1:%d port 2:%d" , - ldev->v2p_map[MLX5_LAG_P1 ], ldev->v2p_map[MLX5_LAG_P2 ]); +mlx5_core_info (dev0,"lag map port 1:%d port 2:%d shared_fdb:%d" , + ldev->v2p_map[MLX5_LAG_P1 ], ldev->v2p_map[MLX5_LAG_P2 ], + shared_fdb); err =mlx5_cmd_create_lag (dev0, ldev->v2p_map[MLX5_LAG_P1 ], - ldev->v2p_map[MLX5_LAG_P2 ]); -if (err) + ldev->v2p_map[MLX5_LAG_P2 ], shared_fdb); +if (err) {mlx5_core_err (dev0,"Failed to create LAG (%d)\n" , err); +return err; + } + +if (shared_fdb) { + err =mlx5_eswitch_offloads_config_single_fdb (dev0->priv.eswitch, + dev1->priv.eswitch); +if (err) +mlx5_core_err (dev0,"Can't enable single FDB mode\n" ); +else +mlx5_core_info (dev0,"Operation mode is single FDB\n" ); + } + +if (err) { +MLX5_SET (destroy_lag_in, in, opcode,MLX5_CMD_OP_DESTROY_LAG ); +if (mlx5_cmd_exec_in (dev0, destroy_lag, in)) +mlx5_core_err (dev0, +"Failed to deactivate RoCE LAG; driver restart required\n" ); + } +return err; }int mlx5_activate_lag (struct mlx5_lag *ldev,struct lag_tracker *tracker, -u8 flags) +u8 flags, +bool shared_fdb) {bool roce_lag = !!(flags &MLX5_LAG_FLAG_ROCE );struct mlx5_core_dev *dev0 = ldev->pf[MLX5_LAG_P1 ].dev;int err; - err =mlx5_create_lag (ldev, tracker); + err =mlx5_create_lag (ldev, tracker, shared_fdb);if (err) {if (roce_lag) { mlx5_core_err(dev0,
@@ -266,6 +293,7 @@ int mlx5_activate_lag(struct mlx5_lag *ldev, } ldev->flags |= flags; + ldev->shared_fdb = shared_fdb;return 0 ; }
@@ -278,6 +306,12 @@ static int mlx5_deactivate_lag(struct mlx5_lag *ldev) ldev->flags &= ~MLX5_LAG_MODE_FLAGS ; +if (ldev->shared_fdb) { +mlx5_eswitch_offloads_destroy_single_fdb (ldev->pf[MLX5_LAG_P1 ].dev->priv.eswitch, + ldev->pf[MLX5_LAG_P2 ].dev->priv.eswitch); + ldev->shared_fdb = false; + } +MLX5_SET (destroy_lag_in, in, opcode,MLX5_CMD_OP_DESTROY_LAG ); err =mlx5_cmd_exec_in (dev0, destroy_lag, in);if (err) {
@@ -333,6 +367,10 @@ static void mlx5_lag_remove_devices(struct mlx5_lag *ldev)if (!ldev->pf[i].dev)continue ; +if (ldev->pf[i].dev->priv.flags & +MLX5_PRIV_FLAGS_DISABLE_ALL_ADEV ) +continue ; + ldev->pf[i].dev->priv.flags |=MLX5_PRIV_FLAGS_DISABLE_IB_ADEV ;mlx5_rescan_drivers_locked (ldev->pf[i].dev); }
@@ -342,12 +380,15 @@ static void mlx5_disable_lag(struct mlx5_lag *ldev) {struct mlx5_core_dev *dev0 = ldev->pf[MLX5_LAG_P1 ].dev;struct mlx5_core_dev *dev1 = ldev->pf[MLX5_LAG_P2 ].dev; +bool shared_fdb = ldev->shared_fdb;bool roce_lag;int err; roce_lag =__mlx5_lag_is_roce (ldev); -if (roce_lag) { +if (shared_fdb) { +mlx5_lag_remove_devices (ldev); + }else if (roce_lag) {if (!(dev0->priv.flags &MLX5_PRIV_FLAGS_DISABLE_ALL_ADEV )) { dev0->priv.flags |=MLX5_PRIV_FLAGS_DISABLE_IB_ADEV ;mlx5_rescan_drivers_locked (dev0);
@@ -359,8 +400,34 @@ static void mlx5_disable_lag(struct mlx5_lag *ldev)if (err)return ; -if (roce_lag) +if (shared_fdb || roce_lag)mlx5_lag_add_devices (ldev); + +if (shared_fdb) { +if (!(dev0->priv.flags &MLX5_PRIV_FLAGS_DISABLE_ALL_ADEV )) +mlx5_eswitch_reload_reps (dev0->priv.eswitch); +if (!(dev1->priv.flags &MLX5_PRIV_FLAGS_DISABLE_ALL_ADEV )) +mlx5_eswitch_reload_reps (dev1->priv.eswitch); + } +} + +static bool mlx5_shared_fdb_supported (struct mlx5_lag *ldev) +{ +struct mlx5_core_dev *dev0 = ldev->pf[MLX5_LAG_P1 ].dev; +struct mlx5_core_dev *dev1 = ldev->pf[MLX5_LAG_P2 ].dev; + +if (is_mdev_switchdev_mode (dev0) && +is_mdev_switchdev_mode (dev1) && +mlx5_eswitch_vport_match_metadata_enabled (dev0->priv.eswitch) && +mlx5_eswitch_vport_match_metadata_enabled (dev1->priv.eswitch) && +mlx5_devcom_is_paired (dev0->priv.devcom, +MLX5_DEVCOM_ESW_OFFLOADS ) && +MLX5_CAP_GEN (dev1, lag_native_fdb_selection) && +MLX5_CAP_ESW (dev1, root_ft_on_other_esw) && +MLX5_CAP_ESW (dev0, esw_shared_ingress_acl)) +return true; + +return false; }static void mlx5_do_bond (struct mlx5_lag *ldev)
@@ -380,6 +447,8 @@ static void mlx5_do_bond(struct mlx5_lag *ldev) }if (do_bond && !__mlx5_lag_is_active (ldev)) { +bool shared_fdb =mlx5_shared_fdb_supported (ldev); + roce_lag = !mlx5_sriov_is_enabled (dev0) && !mlx5_sriov_is_enabled (dev1);
@@ -389,23 +458,40 @@ static void mlx5_do_bond(struct mlx5_lag *ldev) dev1->priv.eswitch->mode ==MLX5_ESWITCH_NONE ;#endif -if (roce_lag) +if (shared_fdb || roce_lag)mlx5_lag_remove_devices (ldev); err =mlx5_activate_lag (ldev, &tracker, roce_lag ?MLX5_LAG_FLAG_ROCE : -MLX5_LAG_FLAG_SRIOV ); +MLX5_LAG_FLAG_SRIOV , + shared_fdb);if (err) { -if (roce_lag) +if (shared_fdb || roce_lag)mlx5_lag_add_devices (ldev);return ; - } - -if (roce_lag) { + }else if (roce_lag) { dev0->priv.flags &= ~MLX5_PRIV_FLAGS_DISABLE_IB_ADEV ;mlx5_rescan_drivers_locked (dev0);mlx5_nic_vport_enable_roce (dev1); + }else if (shared_fdb) { + dev0->priv.flags &= ~MLX5_PRIV_FLAGS_DISABLE_IB_ADEV ; +mlx5_rescan_drivers_locked (dev0); + + err =mlx5_eswitch_reload_reps (dev0->priv.eswitch); +if (!err) + err =mlx5_eswitch_reload_reps (dev1->priv.eswitch); + +if (err) { + dev0->priv.flags |=MLX5_PRIV_FLAGS_DISABLE_IB_ADEV ; +mlx5_rescan_drivers_locked (dev0); +mlx5_deactivate_lag (ldev); +mlx5_lag_add_devices (ldev); +mlx5_eswitch_reload_reps (dev0->priv.eswitch); +mlx5_eswitch_reload_reps (dev1->priv.eswitch); +mlx5_core_err (dev0,"Failed to enable lag\n" ); +return ; + } } }else if (do_bond &&__mlx5_lag_is_active (ldev)) {mlx5_modify_lag (ldev, &tracker);
drivers/net/ethernet/mellanox/mlx5/core/lag.h +2 -1
@@ -73,7 +73,8 @@ void mlx5_modify_lag(struct mlx5_lag *ldev, struct lag_tracker *tracker);int mlx5_activate_lag (struct mlx5_lag *ldev,struct lag_tracker *tracker, -u8 flags); +u8 flags, +bool shared_fdb);int mlx5_lag_dev_get_netdev_idx (struct mlx5_lag *ldev,struct net_device *ndev);
drivers/net/ethernet/mellanox/mlx5/core/lag_mp.c +1 -1
@@ -161,7 +161,7 @@ static void mlx5_lag_fib_route_event(struct mlx5_lag *ldev,struct lag_tracker tracker; tracker = ldev->tracker; -mlx5_activate_lag (ldev, &tracker,MLX5_LAG_FLAG_MULTIPATH ); +mlx5_activate_lag (ldev, &tracker,MLX5_LAG_FLAG_MULTIPATH , false); }mlx5_lag_set_port_affinity (ldev,MLX5_LAG_NORMAL_AFFINITY );
Signature · 1 line
2.31.1
Saeed Mahameed
AppliedQuoted 6 lines · Saeed Mahameed
From: Saeed Mahameed <[email protected]>
This series is aimed at mlx5-next branch to be pulled later by both rdma and netdev subsystems as it contains patches to both trees.
The series provides support for single shared FDB table for lag:
applied to mlx5-next, Thanks !