Move IB representors to single IB device multiple ports
Subsystems (5)
cover Move IB representors to single IB device multiple ports
From: Leon Romanovsky <[email protected]>
Quoted 1 line
From Mark,
Hi,
This series starts from two refactoring patches performed in mlx5_core and they are part of a larger series intended to expose the ability to insert FDB steering rules by the RDMA side.
They are followed by patches which move IB representors to single IB device. Currently when moving to switchdev mode an IB device is created per virtual function (and one for the uplink). While initially this was right approach, the limitation of this model requires us to change it.
Such limitations includes:
- Inability to share resources between IB devices.
- Lack of DEVX to allow more control and device specific functionality.
- Interacting with the E-Switch.
- Doesn't work well with namespaces.
In this series we move to a single IB device with multiple ports where each port represents a different virtual function (port 1 is for the uplink). All the limitations listed above are now resolved and the software model exposed to users is more cohesive.
Thanks
Mark Bloch (12):
net/mlx5: E-Switch, don't use hardcoded values for FDB prios net/mlx5: E-Switch, add a new prio to be used by the RDMA side RDMA/mlx5: Move netdev info into the port struct RDMA/mlx5: Free IB device on remove
RDMA/mlx5: Move ports allocation to outside of INIT stage RDMA/mlx5: Use correct size for device resources RDMA/mlx5: Move rep into port struct
RDMA/mlx5: Move default representors SQ steering to rule to modify QP RDMA/mlx5: Refactor netdev affinity code
RDMA/mlx5: Move SMI caps logic
RDMA/mlx5: Move to single device multiport ports in switchdev mode RDMA/mlx5: Remove VF representor profile
drivers/infiniband/hw/mlx5/devx.c | 2 +- drivers/infiniband/hw/mlx5/flow.c | 2 +- drivers/infiniband/hw/mlx5/ib_rep.c | 109 +++++---- drivers/infiniband/hw/mlx5/ib_rep.h | 13 +- drivers/infiniband/hw/mlx5/main.c | 207 ++++++++++-------- drivers/infiniband/hw/mlx5/mlx5_ib.h | 35 +-- drivers/infiniband/hw/mlx5/mr.c | 6 +- drivers/infiniband/hw/mlx5/qp.c | 46 ++-- .../mellanox/mlx5/core/eswitch_offloads.c | 5 - .../net/ethernet/mellanox/mlx5/core/fs_core.c | 12 +- include/linux/mlx5/fs.h | 6 + 11 files changed, 239 insertions(+), 204 deletions(-)
PATCH 1/12 net/mlx5: E-Switch, don't use hardcoded values for FDB prios
From: Mark Bloch <[email protected]>
When creating the FDB prios, use the enum values already defined and not the hardcoded values.
--- drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c | 5 ----- drivers/net/ethernet/mellanox/mlx5/core/fs_core.c | 5 +++-- include/linux/mlx5/fs.h | 5 +++++ 3 files changed, 8 insertions(+), 7 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +0 -5
@@ -43,11 +43,6 @@#include "ecpf.h" #include "lib/eq.h" -enum { -FDB_FAST_PATH =0 , -FDB_SLOW_PATH -}; -/* There are two match-all miss flows, one for unicast dst mac and * one for multicast. */
drivers/net/ethernet/mellanox/mlx5/core/fs_core.c +3 -2
@@ -2517,7 +2517,8 @@ static int init_fdb_root_ns(struct mlx5_flow_steering *steering)return -ENOMEM ; levels =2 *FDB_MAX_PRIO * (FDB_MAX_CHAIN +1 ); - maj_prio =fs_create_prio_chained (&steering->fdb_root_ns->ns,0 , + maj_prio =fs_create_prio_chained (&steering->fdb_root_ns->ns, +FDB_FAST_PATH , levels);if (IS_ERR (maj_prio)) { err =PTR_ERR (maj_prio);
@@ -2542,7 +2543,7 @@ static int init_fdb_root_ns(struct mlx5_flow_steering *steering) steering->fdb_sub_ns[chain] = ns; } - maj_prio =fs_create_prio (&steering->fdb_root_ns->ns,1 ,1 ); + maj_prio =fs_create_prio (&steering->fdb_root_ns->ns,FDB_SLOW_PATH ,1 );if (IS_ERR (maj_prio)) { err =PTR_ERR (maj_prio); goto out_err;
include/linux/mlx5/fs.h +5 -0
@@ -75,6 +75,11 @@ enum mlx5_flow_namespace_type { MLX5_FLOW_NAMESPACE_EGRESS, }; +enum { +FDB_FAST_PATH , +FDB_SLOW_PATH , +}; +struct mlx5_flow_table ;struct mlx5_flow_group ;struct mlx5_flow_namespace ;
Signature · 1 line
2.20.1
PATCH 2/12 net/mlx5: E-Switch, add a new prio to be used by the RDMA side
In reply to Leon RomanovskyFrom: Mark Bloch <[email protected]>
Create a new prio in the FDB, it will be used when inserting steering rules into the FDB from the RDMA side. We create a new PRIO so rules from the net side and rules from the RDMA side won't be inserted to the same PRIO, each side has it's own sandbox to play in.
--- drivers/net/ethernet/mellanox/mlx5/core/fs_core.c | 7 +++++++ include/linux/mlx5/fs.h | 1 + 2 files changed, 8 insertions(+)
drivers/net/ethernet/mellanox/mlx5/core/fs_core.c +7 -0
@@ -2516,6 +2516,13 @@ static int init_fdb_root_ns(struct mlx5_flow_steering *steering)if (!steering->fdb_sub_ns)return -ENOMEM ; + maj_prio =fs_create_prio (&steering->fdb_root_ns->ns,FDB_BYPASS_PATH , +1 ); +if (IS_ERR (maj_prio)) { + err =PTR_ERR (maj_prio); + goto out_err; + } + levels =2 *FDB_MAX_PRIO * (FDB_MAX_CHAIN +1 ); maj_prio = fs_create_prio_chained(&steering->fdb_root_ns->ns, FDB_FAST_PATH,
include/linux/mlx5/fs.h +1 -0
@@ -76,6 +76,7 @@ enum mlx5_flow_namespace_type { };enum { +FDB_BYPASS_PATH ,FDB_FAST_PATH ,FDB_SLOW_PATH , };
Signature · 1 line
2.20.1
PATCH 3/12 RDMA/mlx5: Move netdev info into the port struct
In reply to Leon RomanovskyFrom: Mark Bloch <[email protected]>
Netdev info is stored in a separate array and holds data relevant on a per port basis, move it to be part of the port struct.
--- drivers/infiniband/hw/mlx5/main.c | 30 ++++++++++++++-------------- drivers/infiniband/hw/mlx5/mlx5_ib.h | 14 ++++++------- drivers/infiniband/hw/mlx5/qp.c | 2 +- 3 files changed, 23 insertions(+), 23 deletions(-)
drivers/infiniband/hw/mlx5/main.c +15 -15
@@ -257,11 +257,11 @@ static struct net_device *mlx5_ib_get_netdev(struct ib_device *device,/* Ensure ndev does not disappear before we invoke dev_hold() */ -read_lock (&ibdev->roce[port_num -1 ].netdev_lock); - ndev = ibdev->roce[port_num -1 ].netdev; +read_lock (&ibdev->port[port_num -1 ].roce.netdev_lock); + ndev = ibdev->port[port_num -1 ].roce.netdev;if (ndev)dev_hold (ndev); -read_unlock (&ibdev->roce[port_num -1 ].netdev_lock); +read_unlock (&ibdev->port[port_num -1 ].roce.netdev_lock); out:mlx5_ib_put_native_port_mdev (ibdev, port_num);
@@ -1956,7 +1956,7 @@ static int mlx5_ib_alloc_ucontext(struct ib_ucontext *uctx,atomic_set (&context->tx_port_affinity,atomic_add_return ( -1 , &dev->roce[port].tx_port_affinity)); +1 , &dev->port[port].roce.tx_port_affinity)); }return 0 ;
@@ -5020,10 +5020,10 @@ static int mlx5_add_netdev_notifier(struct mlx5_ib_dev *dev, u8 port_num) {int err; - dev->roce[port_num].nb.notifier_call = mlx5_netdev_event; - err =register_netdevice_notifier (&dev->roce[port_num].nb); + dev->port[port_num].roce.nb.notifier_call = mlx5_netdev_event; + err =register_netdevice_notifier (&dev->port[port_num].roce.nb);if (err) { - dev->roce[port_num].nb.notifier_call =NULL ; + dev->port[port_num].roce.nb.notifier_call =NULL ;return err; }
@@ -5032,9 +5032,9 @@ static int mlx5_add_netdev_notifier(struct mlx5_ib_dev *dev, u8 port_num)static void mlx5_remove_netdev_notifier (struct mlx5_ib_dev *dev,u8 port_num) { -if (dev->roce[port_num].nb.notifier_call) { -unregister_netdevice_notifier (&dev->roce[port_num].nb); - dev->roce[port_num].nb.notifier_call =NULL ; +if (dev->port[port_num].roce.nb.notifier_call) { +unregister_netdevice_notifier (&dev->port[port_num].roce.nb); + dev->port[port_num].roce.nb.notifier_call =NULL ; } }
@@ -5657,7 +5657,7 @@ static void mlx5_ib_unbind_slave_port(struct mlx5_ib_dev *ibdev,mlx5_ib_err (ibdev,"Failed to unaffiliate port %u\n" , port_num +1 ); - ibdev->roce[port_num].last_port_state =IB_PORT_DOWN ; + ibdev->port[port_num].roce.last_port_state =IB_PORT_DOWN ; }/* The mlx5_ib_multiport_mutex should be held when calling this function */
@@ -5930,7 +5930,7 @@ int mlx5_ib_stage_init_init(struct mlx5_ib_dev *dev)for (i =0 ; i < dev->num_ports; i++) {spin_lock_init (&dev->port[i].mp.mpi_lock); -rwlock_init (&dev->roce[i].netdev_lock); +rwlock_init (&dev->port[i].roce.netdev_lock); } err =mlx5_ib_init_multiport_master (dev);
@@ -6233,9 +6233,9 @@ static int mlx5_ib_stage_common_roce_init(struct mlx5_ib_dev *dev)int i;for (i =0 ; i < dev->num_ports; i++) { - dev->roce[i].dev = dev; - dev->roce[i].native_port_num = i +1 ; - dev->roce[i].last_port_state =IB_PORT_DOWN ; + dev->port[i].roce.dev = dev; + dev->port[i].roce.native_port_num = i +1 ; + dev->port[i].roce.last_port_state =IB_PORT_DOWN ; } dev->ib_dev.uverbs_ex_cmd_mask |=
drivers/infiniband/hw/mlx5/mlx5_ib.h +7 -7
@@ -706,12 +706,6 @@ struct mlx5_ib_multiport {spinlock_t mpi_lock; }; -struct mlx5_ib_port { -struct mlx5_ib_counters cnts; -struct mlx5_ib_multiport mp; -struct mlx5_ib_dbg_cc_params *dbg_cc_params; -}; -struct mlx5_roce {/* Protect mlx5_ib_get_netdev from invoking dev_hold() with a NULL * netdev pointer
@@ -725,6 +719,13 @@ struct mlx5_roce {u8 native_port_num; }; +struct mlx5_ib_port { +struct mlx5_ib_counters cnts; +struct mlx5_ib_multiport mp; +struct mlx5_ib_dbg_cc_params *dbg_cc_params; +struct mlx5_roce roce; +}; +struct mlx5_ib_dbg_param {int offset;struct mlx5_ib_dev *dev;
@@ -909,7 +910,6 @@ struct mlx5_ib_dev {struct ib_device ib_dev;struct mlx5_core_dev *mdev;struct notifier_block mdev_events; -struct mlx5_roce roce[MLX5_MAX_PORTS ];int num_ports;/* serialize update of capability mask */
drivers/infiniband/hw/mlx5/qp.c +1 -1
@@ -3296,7 +3296,7 @@ static unsigned int get_tx_affinity(struct mlx5_ib_dev *dev, }else { tx_port_affinity = (unsigned int )atomic_add_return ( -1 , &dev->roce[port_num].tx_port_affinity) % +1 , &dev->port[port_num].roce.tx_port_affinity) %MLX5_MAX_PORTS +1 ; mlx5_ib_dbg(dev, "Set tx affinity 0x%x to qpn 0x%x\n",
Signature · 1 line
2.20.1
PATCH 4/12 RDMA/mlx5: Free IB device on remove
In reply to Leon RomanovskyFrom: Mark Bloch <[email protected]>
Simplify the code and move the deallocation of the IB device into the remove function.
--- drivers/infiniband/hw/mlx5/ib_rep.c | 5 +---- drivers/infiniband/hw/mlx5/main.c | 4 ++-- 2 files changed, 3 insertions(+), 6 deletions(-)
drivers/infiniband/hw/mlx5/ib_rep.c +1 -4
@@ -65,10 +65,8 @@ mlx5_ib_vport_rep_load(struct mlx5_core_dev *dev, struct mlx5_eswitch_rep *rep) ibdev->mdev = dev; ibdev->num_ports =max (MLX5_CAP_GEN (dev, num_ports),MLX5_CAP_GEN (dev, num_vhca_ports)); -if (!__mlx5_ib_add (ibdev, profile)) { -ib_dealloc_device (&ibdev->ib_dev); +if (!__mlx5_ib_add (ibdev, profile))return -EINVAL ; - } rep->rep_if[REP_IB ].priv = ibdev;
@@ -86,7 +84,6 @@ mlx5_ib_vport_rep_unload(struct mlx5_eswitch_rep *rep) dev =mlx5_ib_rep_to_dev (rep);__mlx5_ib_remove (dev, dev->profile,MLX5_IB_STAGE_MAX ); rep->rep_if[REP_IB ].priv =NULL ; -ib_dealloc_device (&dev->ib_dev); }static void *mlx5_ib_vport_get_proto_dev (struct mlx5_eswitch_rep *rep)
drivers/infiniband/hw/mlx5/main.c +2 -2
@@ -6491,6 +6491,8 @@ void __mlx5_ib_remove(struct mlx5_ib_dev *dev,if (profile->stage[stage].cleanup) profile->stage[stage].cleanup (dev); } + +ib_dealloc_device (&dev->ib_dev); } void *__mlx5_ib_add(struct mlx5_ib_dev *dev,
@@ -6713,8 +6715,6 @@ static void mlx5_ib_remove(struct mlx5_core_dev *mdev, void *context) dev = context;__mlx5_ib_remove (dev, dev->profile,MLX5_IB_STAGE_MAX ); - -ib_dealloc_device ((struct ib_device *)dev); }static struct mlx5_interface mlx5_ib_interface = {
Signature · 1 line
2.20.1
PATCH 5/12 RDMA/mlx5: Move ports allocation to outside of INIT stage
In reply to Leon RomanovskyFrom: Mark Bloch <[email protected]>
In downstream patches we will need access to the ports before doing any stages, in order to set net device per representor.
--- drivers/infiniband/hw/mlx5/ib_rep.c | 12 ++++++++++-- drivers/infiniband/hw/mlx5/main.c | 24 ++++++++++++------------ 2 files changed, 22 insertions(+), 14 deletions(-)
drivers/infiniband/hw/mlx5/ib_rep.c +10 -2
@@ -51,6 +51,7 @@ mlx5_ib_vport_rep_load(struct mlx5_core_dev *dev, struct mlx5_eswitch_rep *rep) {const struct mlx5_ib_profile *profile;struct mlx5_ib_dev *ibdev; +int num_ports =1 ;if (rep->vport ==MLX5_VPORT_UPLINK ) profile = &uplink_rep_profile;
@@ -61,10 +62,17 @@ mlx5_ib_vport_rep_load(struct mlx5_core_dev *dev, struct mlx5_eswitch_rep *rep)if (!ibdev)return -ENOMEM ; + ibdev->port =kcalloc (num_ports,sizeof (*ibdev->port), +GFP_KERNEL ); +if (!ibdev->port) { +ib_dealloc_device (&ibdev->ib_dev); +return -ENOMEM ; + } + ibdev->rep = rep; ibdev->mdev = dev; - ibdev->num_ports =max (MLX5_CAP_GEN (dev, num_ports), -MLX5_CAP_GEN (dev, num_vhca_ports)); + ibdev->num_ports = num_ports; +if (!__mlx5_ib_add (ibdev, profile))return -EINVAL ;
drivers/infiniband/hw/mlx5/main.c +12 -12
@@ -5914,7 +5914,6 @@ void mlx5_ib_stage_init_cleanup(struct mlx5_ib_dev *dev)srcu_barrier (&dev->mr_srcu);cleanup_srcu_struct (&dev->mr_srcu); } -kfree (dev->port); }int mlx5_ib_stage_init_init (struct mlx5_ib_dev *dev)
@@ -5923,11 +5922,6 @@ int mlx5_ib_stage_init_init(struct mlx5_ib_dev *dev)int err;int i; - dev->port =kcalloc (dev->num_ports,sizeof (*dev->port), -GFP_KERNEL ); -if (!dev->port) -return -ENOMEM ; -for (i =0 ; i < dev->num_ports; i++) {spin_lock_init (&dev->port[i].mp.mpi_lock);rwlock_init (&dev->port[i].roce.netdev_lock);
@@ -5935,7 +5929,7 @@ int mlx5_ib_stage_init_init(struct mlx5_ib_dev *dev) err =mlx5_ib_init_multiport_master (dev);if (err) - goto err_free_port; +return err;if (!mlx5_core_mp_enabled (mdev)) {for (i =1 ; i <= dev->num_ports; i++) {
@@ -5976,9 +5970,6 @@ int mlx5_ib_stage_init_init(struct mlx5_ib_dev *dev) err_mp:mlx5_ib_cleanup_multiport_master (dev); -err_free_port: -kfree (dev->port); -return -ENOMEM ; }
@@ -6492,6 +6483,7 @@ void __mlx5_ib_remove(struct mlx5_ib_dev *dev, profile->stage[stage].cleanup (dev); } +kfree (dev->port);ib_dealloc_device (&dev->ib_dev); }
@@ -6667,6 +6659,7 @@ static void *mlx5_ib_add(struct mlx5_core_dev *mdev)enum rdma_link_layer ll;struct mlx5_ib_dev *dev;int port_type_cap; +int num_ports;printk_once (KERN_INFO "%s" , mlx5_version);
@@ -6682,13 +6675,20 @@ static void *mlx5_ib_add(struct mlx5_core_dev *mdev)if (mlx5_core_is_mp_slave (mdev) && ll ==IB_LINK_LAYER_ETHERNET )return mlx5_ib_add_slave_port (mdev); + num_ports =max (MLX5_CAP_GEN (mdev, num_ports), +MLX5_CAP_GEN (mdev, num_vhca_ports)); dev =ib_alloc_device (mlx5_ib_dev, ib_dev);if (!dev)return NULL ; + dev->port =kcalloc (num_ports,sizeof (*dev->port), +GFP_KERNEL ); +if (!dev->port) { +ib_dealloc_device ((struct ib_device *)dev); +return NULL ; + } dev->mdev = mdev; - dev->num_ports =max (MLX5_CAP_GEN (mdev, num_ports), -MLX5_CAP_GEN (mdev, num_vhca_ports)); + dev->num_ports = num_ports;return __mlx5_ib_add (dev, &pf_profile); }
Signature · 1 line
2.20.1
PATCH 6/12 RDMA/mlx5: Use correct size for device resources
In reply to Leon RomanovskyFrom: Mark Bloch <[email protected]>
On allocation we use the array size and on destruction num_ports, use the array size of destruction as well, in this context the array corresponds to the native/actual ports on the NIC so no need to adjust this logic for representors.
--- drivers/infiniband/hw/mlx5/main.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)
drivers/infiniband/hw/mlx5/main.c +1 -3
@@ -4862,8 +4862,6 @@ static int create_dev_resources(struct mlx5_ib_resources *devr)static void destroy_dev_resources (struct mlx5_ib_resources *devr) { -struct mlx5_ib_dev *dev = - container_of(devr, struct mlx5_ib_dev, devr);int port;mlx5_ib_destroy_srq (devr->s1);
@@ -4877,7 +4875,7 @@ static void destroy_dev_resources(struct mlx5_ib_resources *devr)kfree (devr->p0);/* Make sure no change P_Key work items are still executing */ -for (port =0 ; port < dev->num_ports; ++port) +for (port =0 ; port <ARRAY_SIZE (devr->ports); ++port)cancel_work_sync (&devr->ports[port].pkey_change_work); }
Signature · 1 line
2.20.1
PATCH 7/12 RDMA/mlx5: Move rep into port struct
In reply to Leon RomanovskyFrom: Mark Bloch <[email protected]>
In preparation of moving into a model of single IB device multiple ports move rep to be part of the port structure. We mark a representor device by setting is_rep, no functional change with this patch.
--- drivers/infiniband/hw/mlx5/devx.c | 2 +- drivers/infiniband/hw/mlx5/flow.c | 2 +- drivers/infiniband/hw/mlx5/ib_rep.c | 7 ++++--- drivers/infiniband/hw/mlx5/main.c | 22 +++++++++++++--------- drivers/infiniband/hw/mlx5/mlx5_ib.h | 3 ++- drivers/infiniband/hw/mlx5/mr.c | 6 +++--- drivers/infiniband/hw/mlx5/qp.c | 4 ++-- 7 files changed, 26 insertions(+), 20 deletions(-)
drivers/infiniband/hw/mlx5/devx.c +1 -1
@@ -1900,7 +1900,7 @@ static bool devx_is_supported(struct ib_device *device) {struct mlx5_ib_dev *dev =to_mdev (device); -return !dev->rep &&MLX5_CAP_GEN (dev->mdev, log_max_uctx); +return !dev->is_rep &&MLX5_CAP_GEN (dev->mdev, log_max_uctx); }const struct uapi_definition mlx5_ib_devx_defs[] = {
drivers/infiniband/hw/mlx5/flow.c +1 -1
@@ -621,7 +621,7 @@ DECLARE_UVERBS_NAMED_OBJECT(MLX5_IB_OBJECT_FLOW_MATCHER,static bool flow_is_supported (struct ib_device *device) { -return !to_mdev (device)->rep; +return !to_mdev (device)->is_rep; }const struct uapi_definition mlx5_ib_flow_defs[] = {
drivers/infiniband/hw/mlx5/ib_rep.c +4 -3
@@ -69,7 +69,8 @@ mlx5_ib_vport_rep_load(struct mlx5_core_dev *dev, struct mlx5_eswitch_rep *rep)return -ENOMEM ; } - ibdev->rep = rep; + ibdev->is_rep = true; + ibdev->port[0 ].rep = rep; ibdev->mdev = dev; ibdev->num_ports = num_ports;
@@ -151,12 +152,12 @@ int create_flow_rule_vport_sq(struct mlx5_ib_dev *dev,struct mlx5_flow_handle *flow_rule;struct mlx5_eswitch *esw = dev->mdev->priv.eswitch; -if (!dev->rep) +if (!dev->is_rep)return 0 ; flow_rule =mlx5_eswitch_add_send_to_vport_rule (esw, - dev->rep->vport, + dev->port[0 ].rep->vport, sq->base.mqp.qpn);if (IS_ERR (flow_rule))return PTR_ERR (flow_rule);
drivers/infiniband/hw/mlx5/main.c +13 -9
@@ -173,12 +173,12 @@ static int mlx5_netdev_event(struct notifier_block *this,switch (event) {case NETDEV_REGISTER :write_lock (&roce->netdev_lock); -if (ibdev->rep) { +if (ibdev->is_rep) {struct mlx5_eswitch *esw = ibdev->mdev->priv.eswitch; +struct mlx5_eswitch_rep *rep = ibdev->port[0 ].rep;struct net_device *rep_ndev; - rep_ndev =mlx5_ib_get_rep_netdev (esw, - ibdev->rep->vport); + rep_ndev =mlx5_ib_get_rep_netdev (esw, rep->vport);if (rep_ndev == ndev) roce->netdev = ndev; }else if (ndev->dev.parent == &mdev->pdev->dev) {
@@ -3149,10 +3149,10 @@ static struct mlx5_ib_flow_prio *get_flow_table(struct mlx5_ib_dev *dev,if (ft_type ==MLX5_IB_FT_RX ) { fn_type =MLX5_FLOW_NAMESPACE_BYPASS ; prio = &dev->flow_db->prios[priority]; -if (!dev->rep && +if (!dev->is_rep &&MLX5_CAP_FLOWTABLE_NIC_RX (dev->mdev, decap)) flags |=MLX5_FLOW_TABLE_TUNNEL_EN_DECAP ; -if (!dev->rep && +if (!dev->is_rep &&MLX5_CAP_FLOWTABLE_NIC_RX (dev->mdev, reformat_l3_tunnel_to_l2)) flags |=MLX5_FLOW_TABLE_TUNNEL_EN_REFORMAT ;
@@ -3162,7 +3162,7 @@ static struct mlx5_ib_flow_prio *get_flow_table(struct mlx5_ib_dev *dev, log_max_ft_size)); fn_type =MLX5_FLOW_NAMESPACE_EGRESS ; prio = &dev->flow_db->egress_prios[priority]; -if (!dev->rep && +if (!dev->is_rep &&MLX5_CAP_FLOWTABLE_NIC_TX (dev->mdev, reformat)) flags |=MLX5_FLOW_TABLE_TUNNEL_EN_REFORMAT ; }
@@ -3368,7 +3368,7 @@ static struct mlx5_ib_flow_handler *_create_flow_rule(struct mlx5_ib_dev *dev,if (!is_valid_attr (dev->mdev, flow_attr))return ERR_PTR (-EINVAL ); -if (dev->rep && is_egress) +if (dev->is_rep && is_egress)return ERR_PTR (-EINVAL ); spec =kvzalloc (sizeof (*spec),GFP_KERNEL );
@@ -3399,13 +3399,17 @@ static struct mlx5_ib_flow_handler *_create_flow_rule(struct mlx5_ib_dev *dev,if (!flow_is_multicast_only (flow_attr))set_underlay_qp (dev, spec, underlay_qpn); -if (dev->rep) { +if (dev->is_rep) {void *misc; +if (!dev->port[flow_attr->port -1 ].rep) { + err = -EINVAL ; + goto free; + } misc =MLX5_ADDR_OF (fte_match_param, spec->match_value, misc_parameters);MLX5_SET (fte_match_set_misc, misc, source_port, - dev->rep->vport); + dev->port[flow_attr->port -1 ].rep->vport); misc =MLX5_ADDR_OF (fte_match_param, spec->match_criteria, misc_parameters);MLX5_SET_TO_ONES (fte_match_set_misc, misc, source_port);
drivers/infiniband/hw/mlx5/mlx5_ib.h +2 -1
@@ -724,6 +724,7 @@ struct mlx5_ib_port {struct mlx5_ib_multiport mp;struct mlx5_ib_dbg_cc_params *dbg_cc_params;struct mlx5_roce roce; +struct mlx5_eswitch_rep *rep; };struct mlx5_ib_dbg_param {
@@ -944,7 +945,7 @@ struct mlx5_ib_dev {struct mlx5_sq_bfreg fp_bfreg;struct mlx5_ib_delay_drop delay_drop;const struct mlx5_ib_profile *profile; -struct mlx5_eswitch_rep *rep; +bool is_rep;int lag_active;struct mlx5_ib_lb_state lb;
drivers/infiniband/hw/mlx5/mr.c +3 -3
@@ -600,7 +600,7 @@ static void clean_keys(struct mlx5_ib_dev *dev, int c)static void mlx5_mr_cache_debugfs_cleanup (struct mlx5_ib_dev *dev) { -if (!mlx5_debugfs_root || dev->rep) +if (!mlx5_debugfs_root || dev->is_rep)return ;debugfs_remove_recursive (dev->cache.root);
@@ -614,7 +614,7 @@ static void mlx5_mr_cache_debugfs_init(struct mlx5_ib_dev *dev)struct dentry *dir;int i; -if (!mlx5_debugfs_root || dev->rep) +if (!mlx5_debugfs_root || dev->is_rep)return ; cache->root =debugfs_create_dir ("mr_cache" , dev->mdev->priv.dbg_root);
@@ -677,7 +677,7 @@ int mlx5_mr_cache_init(struct mlx5_ib_dev *dev)MLX5_IB_UMR_OCTOWORD ; ent->access_mode =MLX5_MKC_ACCESS_MODE_MTT ;if ((dev->mdev->profile->mask &MLX5_PROF_MASK_MR_CACHE ) && - !dev->rep && + !dev->is_rep &&mlx5_core_is_pf (dev->mdev)) ent->limit = dev->mdev->profile->mr_cache[i].limit;else
drivers/infiniband/hw/mlx5/qp.c +2 -2
@@ -1430,7 +1430,7 @@ static int create_raw_packet_qp_tir(struct mlx5_ib_dev *dev,if (*qp_flags_en &MLX5_QP_FLAG_TIR_ALLOW_SELF_LB_MC ) lb_flag |=MLX5_TIRC_SELF_LB_BLOCK_BLOCK_MULTICAST ; -if (dev->rep) { +if (dev->is_rep) { lb_flag |=MLX5_TIRC_SELF_LB_BLOCK_BLOCK_UNICAST ; *qp_flags_en |=MLX5_QP_FLAG_TIR_ALLOW_SELF_LB_UC ; }
@@ -1642,7 +1642,7 @@ static int create_rss_raw_qp_tir(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp,return -EOPNOTSUPP ; } -if (ucmd.flags &MLX5_QP_FLAG_TIR_ALLOW_SELF_LB_UC || dev->rep) { +if (ucmd.flags &MLX5_QP_FLAG_TIR_ALLOW_SELF_LB_UC || dev->is_rep) { lb_flag |=MLX5_TIRC_SELF_LB_BLOCK_BLOCK_UNICAST ; qp->flags_en |=MLX5_QP_FLAG_TIR_ALLOW_SELF_LB_UC ; }
Signature · 1 line
2.20.1
PATCH 8/12 RDMA/mlx5: Move default representors SQ steering to rule to modify QP
In reply to Leon RomanovskyFrom: Mark Bloch <[email protected]>
Currently the steering for SQs created on representors is done on creation, once we move to representors as ports of an IB device we need the port argument which is given only at the modify QP stage, adjust the code appropriately.
--- drivers/infiniband/hw/mlx5/ib_rep.c | 25 +++++++++--------- drivers/infiniband/hw/mlx5/ib_rep.h | 13 ++++++---- drivers/infiniband/hw/mlx5/qp.c | 40 ++++++++++++++++++++--------- 3 files changed, 48 insertions(+), 30 deletions(-)
drivers/infiniband/hw/mlx5/ib_rep.c +12 -13
@@ -146,22 +146,21 @@ struct mlx5_eswitch_rep *mlx5_ib_vport_rep(struct mlx5_eswitch *esw, int vport)return mlx5_eswitch_vport_rep (esw, vport); } -int create_flow_rule_vport_sq (struct mlx5_ib_dev *dev, -struct mlx5_ib_sq *sq) +struct mlx5_flow_handle *create_flow_rule_vport_sq (struct mlx5_ib_dev *dev, +struct mlx5_ib_sq *sq, +u16 port) { -struct mlx5_flow_handle *flow_rule;struct mlx5_eswitch *esw = dev->mdev->priv.eswitch; +struct mlx5_eswitch_rep *rep; -if (!dev->is_rep) -return 0 ; +if (!dev->is_rep || !port) +return NULL ; - flow_rule = -mlx5_eswitch_add_send_to_vport_rule (esw, - dev->port[0 ].rep->vport, - sq->base.mqp.qpn); -if (IS_ERR (flow_rule)) -return PTR_ERR (flow_rule); - sq->flow_rule = flow_rule; +if (!dev->port[port -1 ].rep) +return ERR_PTR (-EINVAL ); -return 0 ; + rep = dev->port[port -1 ].rep; + +return mlx5_eswitch_add_send_to_vport_rule (esw, rep->vport, + sq->base.mqp.qpn); }
drivers/infiniband/hw/mlx5/ib_rep.h +8 -5
@@ -20,8 +20,9 @@ struct mlx5_eswitch_rep *mlx5_ib_vport_rep(struct mlx5_eswitch *esw, int vport_index);void mlx5_ib_register_vport_reps (struct mlx5_core_dev *mdev);void mlx5_ib_unregister_vport_reps (struct mlx5_core_dev *mdev); -int create_flow_rule_vport_sq (struct mlx5_ib_dev *dev, -struct mlx5_ib_sq *sq); +struct mlx5_flow_handle *create_flow_rule_vport_sq (struct mlx5_ib_dev *dev, +struct mlx5_ib_sq *sq, +u16 port);struct net_device *mlx5_ib_get_rep_netdev (struct mlx5_eswitch *esw,int vport_index);#else /* CONFIG_MLX5_ESWITCH */
@@ -52,10 +53,12 @@ struct mlx5_eswitch_rep *mlx5_ib_vport_rep(struct mlx5_eswitch *esw,static inline void mlx5_ib_register_vport_reps (struct mlx5_core_dev *mdev) {}static inline void mlx5_ib_unregister_vport_reps (struct mlx5_core_dev *mdev) {} -static inline int create_flow_rule_vport_sq (struct mlx5_ib_dev *dev, -struct mlx5_ib_sq *sq) +static inline +struct mlx5_flow_handle *create_flow_rule_vport_sq (struct mlx5_ib_dev *dev, +struct mlx5_ib_sq *sq, +u16 port) { -return 0 ; +return NULL ; } static inline
drivers/infiniband/hw/mlx5/qp.c +28 -12
@@ -93,6 +93,7 @@ struct mlx5_modify_raw_qp_param {struct mlx5_rate_limit rl;u8 rq_q_ctr_id; +u16 port; }; static void get_cqs(enum ib_qp_type qp_type,
@@ -1207,11 +1208,11 @@ static void destroy_raw_packet_qp_tis(struct mlx5_ib_dev *dev,mlx5_cmd_destroy_tis (dev->mdev, sq->tisn,to_mpd (pd)->uid); } -static void destroy_flow_rule_vport_sq (struct mlx5_ib_dev *dev, -struct mlx5_ib_sq *sq) +static void destroy_flow_rule_vport_sq (struct mlx5_ib_sq *sq) {if (sq->flow_rule)mlx5_del_flow_rules (sq->flow_rule); + sq->flow_rule =NULL ; } static int create_raw_packet_qp_sq(struct mlx5_ib_dev *dev,
@@ -1279,15 +1280,8 @@ static int create_raw_packet_qp_sq(struct mlx5_ib_dev *dev,if (err) goto err_umem; - err =create_flow_rule_vport_sq (dev, sq); -if (err) - goto err_flow; -return 0 ; -err_flow: -mlx5_core_destroy_sq_tracked (dev->mdev, &sq->base.mqp); - err_umem:ib_umem_release (sq->ubuffer.umem); sq->ubuffer.umem =NULL ;
@@ -1298,7 +1292,7 @@ static int create_raw_packet_qp_sq(struct mlx5_ib_dev *dev,static void destroy_raw_packet_qp_sq (struct mlx5_ib_dev *dev,struct mlx5_ib_sq *sq) { -destroy_flow_rule_vport_sq (dev, sq); +destroy_flow_rule_vport_sq (sq);mlx5_core_destroy_sq_tracked (dev->mdev, &sq->base.mqp);ib_umem_release (sq->ubuffer.umem); }
@@ -3262,6 +3256,8 @@ static int modify_raw_packet_qp(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp, }if (modify_sq) { +struct mlx5_flow_handle *flow_rule; +if (tx_affinity) { err = modify_raw_packet_tx_affinity(dev->mdev, sq, tx_affinity,
@@ -3270,8 +3266,25 @@ static int modify_raw_packet_qp(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp,return err; } -return modify_raw_packet_qp_sq (dev->mdev, sq, sq_state, - raw_qp_param, qp->ibqp.pd); + flow_rule =create_flow_rule_vport_sq (dev, sq, + raw_qp_param->port); +if (IS_ERR (flow_rule)) +return err; + + err =modify_raw_packet_qp_sq (dev->mdev, sq, sq_state, + raw_qp_param, qp->ibqp.pd); +if (err) { +if (flow_rule) +mlx5_del_flow_rules (flow_rule); +return err; + } + +if (flow_rule) { +destroy_flow_rule_vport_sq (sq); + sq->flow_rule = flow_rule; + } + +return err; }return 0 ;
@@ -3588,6 +3601,9 @@ static int __mlx5_ib_modify_qp(struct ib_qp *ibqp, raw_qp_param.set_mask |=MLX5_RAW_QP_MOD_SET_RQ_Q_CTR_ID ; } +if (attr_mask &IB_QP_PORT ) + raw_qp_param.port = attr->port_num; +if (attr_mask &IB_QP_RATE_LIMIT ) { raw_qp_param.rl.rate = attr->rate_limit;
Signature · 1 line
2.20.1
PATCH 9/12 RDMA/mlx5: Refactor netdev affinity code
In reply to Leon RomanovskyFrom: Mark Bloch <[email protected]>
The design of representors is such that once an IB representor is created, the netdev of representor already exists, we can use that fact to simplify the netdev affinity code.
--- drivers/infiniband/hw/mlx5/ib_rep.c | 2 ++ drivers/infiniband/hw/mlx5/main.c | 47 +++++++++++++++++++++++------ 2 files changed, 39 insertions(+), 10 deletions(-)
drivers/infiniband/hw/mlx5/ib_rep.c +2 -0
@@ -71,6 +71,8 @@ mlx5_ib_vport_rep_load(struct mlx5_core_dev *dev, struct mlx5_eswitch_rep *rep) ibdev->is_rep = true; ibdev->port[0 ].rep = rep; + ibdev->port[0 ].roce.netdev = +mlx5_ib_get_rep_netdev (dev->priv.eswitch, rep->vport); ibdev->mdev = dev; ibdev->num_ports = num_ports;
drivers/infiniband/hw/mlx5/main.c +37 -10
71 lines not on this page · raw message
Signature · 1 line
2.20.1
PATCH 10/12 RDMA/mlx5: Move SMI caps logic
In reply to Leon RomanovskyFrom: Mark Bloch <[email protected]>
We store the SMI information in the core device's struct, make sure we set that information only once (and not per port), while here make the for loop based on the actual size of the array.
--- drivers/infiniband/hw/mlx5/main.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
drivers/infiniband/hw/mlx5/main.c +5 -5
@@ -4534,7 +4534,7 @@ static int set_has_smi_cap(struct mlx5_ib_dev *dev)int err;int port; -for (port =1 ; port <= dev->num_ports; port++) { +for (port =1 ; port <=ARRAY_SIZE (dev->mdev->port_caps); port++) { dev->mdev->port_caps[port -1 ].has_smi = false;if (MLX5_CAP_GEN (dev->mdev, port_type) ==MLX5_CAP_PORT_TYPE_IB ) {
@@ -4580,10 +4580,6 @@ static int get_port_caps(struct mlx5_ib_dev *dev, u8 port)if (!dprops) goto out; - err =set_has_smi_cap (dev); -if (err) - goto out; - err =mlx5_ib_query_device (&dev->ib_dev, dprops, &uhw);if (err) {mlx5_ib_warn (dev,"query_device failed %d\n" , err);
@@ -5960,6 +5956,10 @@ int mlx5_ib_stage_init_init(struct mlx5_ib_dev *dev)if (err)return err; + err =set_has_smi_cap (dev); +if (err) +return err; +if (!mlx5_core_mp_enabled (mdev)) {for (i =1 ; i <= dev->num_ports; i++) { err =get_port_caps (dev, i);
Signature · 1 line
2.20.1
PATCH 11/12 RDMA/mlx5: Move to single device multiport ports in switchdev mode
In reply to Leon RomanovskyFrom: Mark Bloch <[email protected]>
Move from IB device (representor) per virtual function to single IB device with port per virtual function (port 1 represents the uplink). As number of ports is a static property of an IB device, declare the IB device with as many port as the possible according to the PCI bus.
--- drivers/infiniband/hw/mlx5/ib_rep.c | 31 +++++++++++++++++++++++----- drivers/infiniband/hw/mlx5/main.c | 26 +++++++++++++++++++---- drivers/infiniband/hw/mlx5/mlx5_ib.h | 1 + 3 files changed, 49 insertions(+), 9 deletions(-)
drivers/infiniband/hw/mlx5/ib_rep.c +26 -5
@@ -46,17 +46,36 @@ static const struct mlx5_ib_profile vf_rep_profile = { NULL), }; +static int +mlx5_ib_set_vport_rep (struct mlx5_core_dev *dev,struct mlx5_eswitch_rep *rep) +{ +struct mlx5_ib_dev *ibdev; +int vport_index; + + ibdev =mlx5_ib_get_uplink_ibdev (dev->priv.eswitch); + vport_index = ibdev->free_port++; + + ibdev->port[vport_index].rep = rep; +write_lock (&ibdev->port[vport_index].roce.netdev_lock); + ibdev->port[vport_index].roce.netdev = +mlx5_ib_get_rep_netdev (dev->priv.eswitch, rep->vport); +write_unlock (&ibdev->port[vport_index].roce.netdev_lock); + +return 0 ; +} +static int mlx5_ib_vport_rep_load (struct mlx5_core_dev *dev,struct mlx5_eswitch_rep *rep) { +int num_ports =MLX5_TOTAL_VPORTS (dev);const struct mlx5_ib_profile *profile;struct mlx5_ib_dev *ibdev; -int num_ports =1 ; +int vport_index;if (rep->vport ==MLX5_VPORT_UPLINK ) profile = &uplink_rep_profile;else - profile = &vf_rep_profile; +return mlx5_ib_set_vport_rep (dev, rep); ibdev =ib_alloc_device (mlx5_ib_dev, ib_dev);if (!ibdev)
@@ -70,8 +89,9 @@ mlx5_ib_vport_rep_load(struct mlx5_core_dev *dev, struct mlx5_eswitch_rep *rep) } ibdev->is_rep = true; - ibdev->port[0 ].rep = rep; - ibdev->port[0 ].roce.netdev = + vport_index = ibdev->free_port++; + ibdev->port[vport_index].rep = rep; + ibdev->port[vport_index].roce.netdev =mlx5_ib_get_rep_netdev (dev->priv.eswitch, rep->vport); ibdev->mdev = dev; ibdev->num_ports = num_ports;
@@ -89,7 +109,8 @@ mlx5_ib_vport_rep_unload(struct mlx5_eswitch_rep *rep) {struct mlx5_ib_dev *dev; -if (!rep->rep_if[REP_IB ].priv) +if (!rep->rep_if[REP_IB ].priv || + rep->vport !=MLX5_VPORT_UPLINK )return ; dev =mlx5_ib_rep_to_dev (rep);
drivers/infiniband/hw/mlx5/main.c +22 -4
@@ -506,9 +506,14 @@ static int mlx5_query_port_roce(struct ib_device *device, u8 port_num,/* Possible bad flows are checked before filling out props so in case * of an error it will still be zeroed out. +* Use native port in case of reps */ - err =mlx5_query_port_ptys (mdev, out,sizeof (out),MLX5_PTYS_EN , - mdev_port_num); +if (dev->is_rep) + err =mlx5_query_port_ptys (mdev, out,sizeof (out),MLX5_PTYS_EN , +1 ); +else + err =mlx5_query_port_ptys (mdev, out,sizeof (out),MLX5_PTYS_EN , + mdev_port_num);if (err) goto out; ext =MLX5_CAP_PCAM_FEATURE (dev->mdev, ptys_extended_ethernet);
@@ -1432,7 +1437,9 @@ static int mlx5_ib_rep_query_port(struct ib_device *ibdev, u8 port, {int ret; -/* Only link layer == ethernet is valid for representors */ +/* Only link layer == ethernet is valid for representors +* and we always use port 1 +*/ ret =mlx5_query_port_roce (ibdev, port, props);if (ret || !props)return ret;
@@ -4565,7 +4572,7 @@ static void get_ext_port_caps(struct mlx5_ib_dev *dev)mlx5_query_ext_port_caps (dev, port); } -static int get_port_caps (struct mlx5_ib_dev *dev,u8 port) +static int __get_port_caps (struct mlx5_ib_dev *dev,u8 port) {struct ib_device_attr *dprops =NULL ;struct ib_port_attr *pprops =NULL ;
@@ -4608,6 +4615,16 @@ static int get_port_caps(struct mlx5_ib_dev *dev, u8 port)return err; } +static int get_port_caps (struct mlx5_ib_dev *dev,u8 port) +{ +/* For representors use port 1, is this is the only native +* port +*/ +if (dev->is_rep) +return __get_port_caps (dev,1 ); +return __get_port_caps (dev, port); +} +static void destroy_umrc_res (struct mlx5_ib_dev *dev) {int err;
@@ -6268,6 +6285,7 @@ static int mlx5_ib_stage_common_roce_init(struct mlx5_ib_dev *dev) port_num =mlx5_core_native_port_num (dev->mdev) -1 ; +/* Register only for native ports */ return mlx5_add_netdev_notifier (dev, port_num); }
drivers/infiniband/hw/mlx5/mlx5_ib.h +1 -0
@@ -956,6 +956,7 @@ struct mlx5_ib_dev {u16 devx_whitelist_uid;struct mlx5_srq_table srq_table;struct mlx5_async_ctx async_ctx; +int free_port; };static inline struct mlx5_ib_cq *to_mibcq (struct mlx5_core_cq *mcq)
Signature · 1 line
2.20.1
PATCH 12/12 RDMA/mlx5: Remove VF representor profile
In reply to Leon RomanovskyFrom: Mark Bloch <[email protected]>
Now that we have a single IB device with multipul ports we can remove the VF representor profile.
--- drivers/infiniband/hw/mlx5/ib_rep.c | 39 ----------------------- drivers/infiniband/hw/mlx5/main.c | 46 ++++++++++------------------ drivers/infiniband/hw/mlx5/mlx5_ib.h | 17 ---------- 3 files changed, 16 insertions(+), 86 deletions(-)
drivers/infiniband/hw/mlx5/ib_rep.c +0 -39
@@ -7,45 +7,6 @@#include "ib_rep.h" #include "srq.h" -static const struct mlx5_ib_profile vf_rep_profile = { -STAGE_CREATE (MLX5_IB_STAGE_INIT , - mlx5_ib_stage_init_init, - mlx5_ib_stage_init_cleanup), -STAGE_CREATE (MLX5_IB_STAGE_FLOW_DB , - mlx5_ib_stage_rep_flow_db_init, -NULL ), -STAGE_CREATE (MLX5_IB_STAGE_CAPS , - mlx5_ib_stage_caps_init, -NULL ), -STAGE_CREATE (MLX5_IB_STAGE_NON_DEFAULT_CB , - mlx5_ib_stage_rep_non_default_cb, -NULL ), -STAGE_CREATE (MLX5_IB_STAGE_ROCE , - mlx5_ib_stage_rep_roce_init, - mlx5_ib_stage_rep_roce_cleanup), -STAGE_CREATE (MLX5_IB_STAGE_SRQ , - mlx5_init_srq_table, - mlx5_cleanup_srq_table), -STAGE_CREATE (MLX5_IB_STAGE_DEVICE_RESOURCES , - mlx5_ib_stage_dev_res_init, - mlx5_ib_stage_dev_res_cleanup), -STAGE_CREATE (MLX5_IB_STAGE_COUNTERS , - mlx5_ib_stage_counters_init, - mlx5_ib_stage_counters_cleanup), -STAGE_CREATE (MLX5_IB_STAGE_BFREG , - mlx5_ib_stage_bfrag_init, - mlx5_ib_stage_bfrag_cleanup), -STAGE_CREATE (MLX5_IB_STAGE_PRE_IB_REG_UMR , -NULL , - mlx5_ib_stage_pre_ib_reg_umr_cleanup), -STAGE_CREATE (MLX5_IB_STAGE_IB_REG , - mlx5_ib_stage_ib_reg_init, - mlx5_ib_stage_ib_reg_cleanup), -STAGE_CREATE (MLX5_IB_STAGE_POST_IB_REG_UMR , - mlx5_ib_stage_post_ib_reg_umr_init, -NULL ), -}; -static int mlx5_ib_set_vport_rep (struct mlx5_core_dev *dev,struct mlx5_eswitch_rep *rep) {
drivers/infiniband/hw/mlx5/main.c +16 -30
141 lines not on this page · raw message
drivers/infiniband/hw/mlx5/mlx5_ib.h +0 -17
@@ -1235,23 +1235,6 @@ static inline void mlx5_ib_invalidate_range(struct ib_umem_odp *umem_odp,#endif /* CONFIG_INFINIBAND_ON_DEMAND_PAGING */ /* Needed for rep profile */ -int mlx5_ib_stage_init_init (struct mlx5_ib_dev *dev); -void mlx5_ib_stage_init_cleanup (struct mlx5_ib_dev *dev); -int mlx5_ib_stage_rep_flow_db_init (struct mlx5_ib_dev *dev); -int mlx5_ib_stage_caps_init (struct mlx5_ib_dev *dev); -int mlx5_ib_stage_rep_non_default_cb (struct mlx5_ib_dev *dev); -int mlx5_ib_stage_rep_roce_init (struct mlx5_ib_dev *dev); -void mlx5_ib_stage_rep_roce_cleanup (struct mlx5_ib_dev *dev); -int mlx5_ib_stage_dev_res_init (struct mlx5_ib_dev *dev); -void mlx5_ib_stage_dev_res_cleanup (struct mlx5_ib_dev *dev); -int mlx5_ib_stage_counters_init (struct mlx5_ib_dev *dev); -void mlx5_ib_stage_counters_cleanup (struct mlx5_ib_dev *dev); -int mlx5_ib_stage_bfrag_init (struct mlx5_ib_dev *dev); -void mlx5_ib_stage_bfrag_cleanup (struct mlx5_ib_dev *dev); -void mlx5_ib_stage_pre_ib_reg_umr_cleanup (struct mlx5_ib_dev *dev); -int mlx5_ib_stage_ib_reg_init (struct mlx5_ib_dev *dev); -void mlx5_ib_stage_ib_reg_cleanup (struct mlx5_ib_dev *dev); -int mlx5_ib_stage_post_ib_reg_umr_init (struct mlx5_ib_dev *dev);void __mlx5_ib_remove (struct mlx5_ib_dev *dev,const struct mlx5_ib_profile *profile,int stage);
Signature · 1 line
2.20.1
Quoted 28 lines · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
Quoted 1 line
From Mark,
Hi,
This series starts from two refactoring patches performed in
mlx5_coreand they are part of a larger series intended to expose the ability to insert FDB steering rules by the RDMA side.They are followed by patches which move IB representors to single IB device. Currently when moving to switchdev mode an IB device is created per virtual function (and one for the uplink). While initially this was right approach, the limitation of this model requires us to change it.
Such limitations includes:
- Inability to share resources between IB devices.
- Lack of DEVX to allow more control and device specific functionality.
- Interacting with the E-Switch.
- Doesn't work well with namespaces.In this series we move to a single IB device with multiple ports where each port represents a different virtual function (port 1 is for the uplink). All the limitations listed above are now resolved and the software model exposed to users is more cohesive.
Thanks
Quoted 13 lines
Mark Bloch (12):
net/mlx5: E-Switch, don't use hardcoded values for FDB prios net/mlx5: E-Switch, add a new prio to be used by the RDMA side RDMA/mlx5: Move netdev info into the port struct RDMA/mlx5: Free IB device on remove
RDMA/mlx5: Move ports allocation to outside of INIT stage RDMA/mlx5: Use correct size for device resources RDMA/mlx5: Move rep into port struct
RDMA/mlx5: Move default representors SQ steering to rule to modify QP RDMA/mlx5: Refactor netdev affinity code
RDMA/mlx5: Move SMI caps logic
RDMA/mlx5: Move to single device multiport ports in switchdev mode RDMA/mlx5: Remove VF representor profile
This looks Ok, can you update the shared branch please
Jason
Quoted 43 lines · Jason Gunthorpe
Quoted 28 lines · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
Quoted 1 line
From Mark,
Hi,
This series starts from two refactoring patches performed in
mlx5_coreand they are part of a larger series intended to expose the ability to insert FDB steering rules by the RDMA side.They are followed by patches which move IB representors to single IB device. Currently when moving to switchdev mode an IB device is created per virtual function (and one for the uplink). While initially this was right approach, the limitation of this model requires us to change it.
Such limitations includes:
- Inability to share resources between IB devices.
- Lack of DEVX to allow more control and device specific functionality.
- Interacting with the E-Switch.
- Doesn't work well with namespaces.In this series we move to a single IB device with multiple ports where each port represents a different virtual function (port 1 is for the uplink). All the limitations listed above are now resolved and the software model exposed to users is more cohesive.
Thanks
Quoted 13 lines
Mark Bloch (12):
net/mlx5: E-Switch, don't use hardcoded values for FDB prios net/mlx5: E-Switch, add a new prio to be used by the RDMA side RDMA/mlx5: Move netdev info into the port struct RDMA/mlx5: Free IB device on remove
RDMA/mlx5: Move ports allocation to outside of INIT stage RDMA/mlx5: Use correct size for device resources RDMA/mlx5: Move rep into port struct
RDMA/mlx5: Move default representors SQ steering to rule to modify QP RDMA/mlx5: Refactor netdev affinity code
RDMA/mlx5: Move SMI caps logic
RDMA/mlx5: Move to single device multiport ports in switchdev mode RDMA/mlx5: Remove VF representor profile
Quoted 1 line
This looks Ok, can you update the shared branch please
Thanks Jason,
I'll slightly delay update till Sunday, internally, we have doubts about "RDMA/mlx5: Use correct size for device resources" patch.
Thanks
Quoted 2 lines
Jason
Quoted 44 lines · Leon Romanovsky
Quoted 43 lines · Jason Gunthorpe
Quoted 41 lines · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
Quoted 1 line
From Mark,
Hi,
This series starts from two refactoring patches performed in
mlx5_coreand they are part of a larger series intended to expose the ability to insert FDB steering rules by the RDMA side.They are followed by patches which move IB representors to single IB device. Currently when moving to switchdev mode an IB device is created per virtual function (and one for the uplink). While initially this was right approach, the limitation of this model requires us to change it.
Such limitations includes:
- Inability to share resources between IB devices.
- Lack of DEVX to allow more control and device specific functionality.
- Interacting with the E-Switch.
- Doesn't work well with namespaces.In this series we move to a single IB device with multiple ports where each port represents a different virtual function (port 1 is for the uplink). All the limitations listed above are now resolved and the software model exposed to users is more cohesive.
Thanks
Mark Bloch (12):
net/mlx5: E-Switch, don't use hardcoded values for FDB prios net/mlx5: E-Switch, add a new prio to be used by the RDMA side RDMA/mlx5: Move netdev info into the port struct RDMA/mlx5: Free IB device on remove
RDMA/mlx5: Move ports allocation to outside of INIT stage RDMA/mlx5: Use correct size for device resources RDMA/mlx5: Move rep into port struct
RDMA/mlx5: Move default representors SQ steering to rule to modify QP RDMA/mlx5: Refactor netdev affinity code
RDMA/mlx5: Move SMI caps logic
RDMA/mlx5: Move to single device multiport ports in switchdev mode RDMA/mlx5: Remove VF representor profile
Quoted 6 lines
Quoted 1 line
This looks Ok, can you update the shared branch please
Thanks Jason,
I'll slightly delay update till Sunday, internally, we have doubts about "RDMA/mlx5: Use correct size for device resources" patch.
Hi Jason,
I updated our mlx5-next with first two patches.d9cb06759eca net/mlx5: E-Switch, add a new prio to be used by the RDMA side b6d9ccb11250 net/mlx5: E-Switch, don't use hardcoded values for FDB prios
We found the reason for instability over bond interface and it was related to the patch "RDMA/mlx5: Move netdev info into the port struct". The following fixup is needed. It will be helpful if you can apply it internally, or I can resend rest of the series.
commit 775efa8dd53e5148d8d4942d2d0b7e72ca606d40
Author: Mark Bloch <[email protected]> Date: Tue Apr 9 03:28:59 2019 +0000
fixup! RDMA/mlx5: Move netdev info into the port struct
drivers/infiniband/hw/mlx5/main.c +1 -1
@@ -1958,7 +1958,7 @@ static int mlx5_ib_alloc_ucontext(struct ib_ucontext *uctx,print_lib_caps (dev, context->lib_caps);if (dev->lag_active) { -u8 port =mlx5_core_native_port_num (dev->mdev); +u8 port =mlx5_core_native_port_num (dev->mdev) -1 ; atomic_set(&context->tx_port_affinity, atomic_add_return(
drivers/infiniband/hw/mlx5/qp.c +1 -1
@@ -3491,7 +3491,7 @@ static int __mlx5_ib_modify_qp(struct ib_qp *ibqp, (ibqp->qp_type == IB_QPT_XRC_INI) || (ibqp->qp_type == IB_QPT_XRC_TGT)) {if (dev->lag_active) { -u8 p =mlx5_core_native_port_num (dev->mdev); +u8 p =mlx5_core_native_port_num (dev->mdev) -1 ; tx_affinity =get_tx_affinity (dev, pd, base, p, udata); context->flags |=cpu_to_be32 (tx_affinity <<24 );
Quoted 5 lines
Thanks
Quoted 2 lines
Jason
Quoted 53 lines · Leon Romanovsky
Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
Mark Bloch (12):
net/mlx5: E-Switch, don't use hardcoded values for FDB prios net/mlx5: E-Switch, add a new prio to be used by the RDMA side RDMA/mlx5: Move netdev info into the port struct RDMA/mlx5: Free IB device on remove
RDMA/mlx5: Move ports allocation to outside of INIT stage RDMA/mlx5: Use correct size for device resources RDMA/mlx5: Move rep into port struct
RDMA/mlx5: Move default representors SQ steering to rule to modify QP RDMA/mlx5: Refactor netdev affinity code
RDMA/mlx5: Move SMI caps logic
RDMA/mlx5: Move to single device multiport ports in switchdev mode RDMA/mlx5: Remove VF representor profileQuoted 6 lines
Quoted 1 line
This looks Ok, can you update the shared branch please
Thanks Jason,
I'll slightly delay update till Sunday, internally, we have doubts about "RDMA/mlx5: Use correct size for device resources" patch.
Hi Jason,
I updated our mlx5-next with first two patches.
d9cb06759ecanet/mlx5: E-Switch, add a new prio to be used by the RDMA sideb6d9ccb11250net/mlx5: E-Switch, don't use hardcoded values for FDB priosWe found the reason for instability over bond interface and it was related to the patch "RDMA/mlx5: Move netdev info into the port struct". The following fixup is needed. It will be helpful if you can apply it internally, or I can resend rest of the series.
commit
775efa8dd53e5148d8d4942d2d0b7e72ca606d40Author: Mark Bloch <[email protected]> Date: Tue Apr 9 03:28:59 2019 +0000fixup! RDMA/mlx5: Move netdev info into the port struct
drivers/infiniband/hw/mlx5/main.c +2
-1+++ b/drivers/infiniband/hw/mlx5/main.c@@ -1958,7 +1958,7 @@ static int mlx5_ib_alloc_ucontext(struct ib_ucontext *uctx,print_lib_caps (dev, context->lib_caps);if (dev->lag_active) { -u8 port =mlx5_core_native_port_num (dev->mdev); +u8 port =mlx5_core_native_port_num (dev->mdev) -1 ; atomic_set(&context->tx_port_affinity, atomic_add_return(drivers/infiniband/hw/mlx5/qp.c +1
-0+++ b/drivers/infiniband/hw/mlx5/qp.c
drivers/infiniband/hw/mlx5/qp.c static int __mlx5_ib_modify_qp(struct ib_qp *ibqp, · line 3497
Quoted 8 lines
drivers/infiniband/hw/mlx5/qp.c +1
-1@@ -3491,7 +3491,7 @@ static int __mlx5_ib_modify_qp(struct ib_qp *ibqp, (ibqp->qp_type == IB_QPT_XRC_INI) || (ibqp->qp_type == IB_QPT_XRC_TGT)) {if (dev->lag_active) { -u8 p =mlx5_core_native_port_num (dev->mdev); +u8 p =mlx5_core_native_port_num (dev->mdev) -1 ; tx_affinity =get_tx_affinity (dev, pd, base, p, udata); context->flags |=cpu_to_be32 (tx_affinity <<24 );
Okay, done
Jason