Skip to contentSkip to conversation

Move IB representors to single IB device multiple ports

PATCH In review Leon Romanovsky 2 participants 17 messages Mar 28, 2019 – Apr 10, 2019

Subsystems (5)
lore
Share & tools
lore
#0 lore raw

cover Move IB representors to single IB device multiple ports

From: Leon Romanovsky <[email protected]>

Quoted 1 line

From Mark,

Hi,

This series starts from two refactoring patches performed in mlx5_core and they are part of a larger series intended to expose the ability to insert FDB steering rules by the RDMA side.

They are followed by patches which move IB representors to single IB device. Currently when moving to switchdev mode an IB device is created per virtual function (and one for the uplink). While initially this was right approach, the limitation of this model requires us to change it.

Such limitations includes:
- Inability to share resources between IB devices.
- Lack of DEVX to allow more control and device specific functionality.
- Interacting with the E-Switch.
- Doesn't work well with namespaces.

In this series we move to a single IB device with multiple ports where each port represents a different virtual function (port 1 is for the uplink). All the limitations listed above are now resolved and the software model exposed to users is more cohesive.

Thanks

Mark Bloch (12):
net/mlx5: E-Switch, don't use hardcoded values for FDB prios net/mlx5: E-Switch, add a new prio to be used by the RDMA side RDMA/mlx5: Move netdev info into the port struct RDMA/mlx5: Free IB device on remove
RDMA/mlx5: Move ports allocation to outside of INIT stage RDMA/mlx5: Use correct size for device resources RDMA/mlx5: Move rep into port struct
RDMA/mlx5: Move default representors SQ steering to rule to modify QP RDMA/mlx5: Refactor netdev affinity code
RDMA/mlx5: Move SMI caps logic
RDMA/mlx5: Move to single device multiport ports in switchdev mode RDMA/mlx5: Remove VF representor profile

 drivers/infiniband/hw/mlx5/devx.c             |   2 +-
 drivers/infiniband/hw/mlx5/flow.c             |   2 +-
 drivers/infiniband/hw/mlx5/ib_rep.c           | 109 +++++----
 drivers/infiniband/hw/mlx5/ib_rep.h           |  13 +-
 drivers/infiniband/hw/mlx5/main.c             | 207 ++++++++++--------
 drivers/infiniband/hw/mlx5/mlx5_ib.h          |  35 +--
 drivers/infiniband/hw/mlx5/mr.c               |   6 +-
 drivers/infiniband/hw/mlx5/qp.c               |  46 ++--
 .../mellanox/mlx5/core/eswitch_offloads.c     |   5 -
 .../net/ethernet/mellanox/mlx5/core/fs_core.c |  12 +-
 include/linux/mlx5/fs.h                       |   6 +
 11 files changed, 239 insertions(+), 204 deletions(-)
#1 lore raw

PATCH 1/12 net/mlx5: E-Switch, don't use hardcoded values for FDB prios

From: Mark Bloch <[email protected]>

When creating the FDB prios, use the enum values already defined and not the hardcoded values.

---
 drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c | 5 -----
 drivers/net/ethernet/mellanox/mlx5/core/fs_core.c          | 5 +++--
 include/linux/mlx5/fs.h                                    | 5 +++++
 3 files changed, 8 insertions(+), 7 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +0 -5
index f2260391be5b..6c8a17ca236e 100644
@@ -43,11 +43,6 @@
 #include "ecpf.h"
 #include "lib/eq.h"
 
-enum {
-	FDB_FAST_PATH = 0,
-	FDB_SLOW_PATH
-};
-
 /* There are two match-all miss flows, one for unicast dst mac and
  * one for multicast.
  */
drivers/net/ethernet/mellanox/mlx5/core/fs_core.c +3 -2
index 0be3eb86dd84..f34515d03a3b 100644
@@ -2517,7 +2517,8 @@ static int init_fdb_root_ns(struct mlx5_flow_steering *steering)
 		return -ENOMEM;
 
 	levels = 2 * FDB_MAX_PRIO * (FDB_MAX_CHAIN + 1);
-	maj_prio = fs_create_prio_chained(&steering->fdb_root_ns->ns, 0,
+	maj_prio = fs_create_prio_chained(&steering->fdb_root_ns->ns,
+					  FDB_FAST_PATH,
 					  levels);
 	if (IS_ERR(maj_prio)) {
 		err = PTR_ERR(maj_prio);
@@ -2542,7 +2543,7 @@ static int init_fdb_root_ns(struct mlx5_flow_steering *steering)
 		steering->fdb_sub_ns[chain] = ns;
 	}
 
-	maj_prio = fs_create_prio(&steering->fdb_root_ns->ns, 1, 1);
+	maj_prio = fs_create_prio(&steering->fdb_root_ns->ns, FDB_SLOW_PATH, 1);
 	if (IS_ERR(maj_prio)) {
 		err = PTR_ERR(maj_prio);
 		goto out_err;
include/linux/mlx5/fs.h +5 -0
index 9df51da04621..3eeb04154317 100644
@@ -75,6 +75,11 @@ enum mlx5_flow_namespace_type {
 	MLX5_FLOW_NAMESPACE_EGRESS,
 };
 
+enum {
+	FDB_FAST_PATH,
+	FDB_SLOW_PATH,
+};
+
 struct mlx5_flow_table;
 struct mlx5_flow_group;
 struct mlx5_flow_namespace;
Signature · 1 line
2.20.1
#2 lore raw

PATCH 2/12 net/mlx5: E-Switch, add a new prio to be used by the RDMA side

In reply to Leon Romanovsky

From: Mark Bloch <[email protected]>

Create a new prio in the FDB, it will be used when inserting steering rules into the FDB from the RDMA side. We create a new PRIO so rules from the net side and rules from the RDMA side won't be inserted to the same PRIO, each side has it's own sandbox to play in.

---
 drivers/net/ethernet/mellanox/mlx5/core/fs_core.c | 7 +++++++
 include/linux/mlx5/fs.h                           | 1 +
 2 files changed, 8 insertions(+)
drivers/net/ethernet/mellanox/mlx5/core/fs_core.c +7 -0
index f34515d03a3b..f0b460879c91 100644
@@ -2516,6 +2516,13 @@ static int init_fdb_root_ns(struct mlx5_flow_steering *steering)
 	if (!steering->fdb_sub_ns)
 		return -ENOMEM;
 
+	maj_prio = fs_create_prio(&steering->fdb_root_ns->ns, FDB_BYPASS_PATH,
+				  1);
+	if (IS_ERR(maj_prio)) {
+		err = PTR_ERR(maj_prio);
+		goto out_err;
+	}
+
 	levels = 2 * FDB_MAX_PRIO * (FDB_MAX_CHAIN + 1);
 	maj_prio = fs_create_prio_chained(&steering->fdb_root_ns->ns,
 					  FDB_FAST_PATH,
include/linux/mlx5/fs.h +1 -0
index 3eeb04154317..fd91df3a4e09 100644
@@ -76,6 +76,7 @@ enum mlx5_flow_namespace_type {
 };
 
 enum {
+	FDB_BYPASS_PATH,
 	FDB_FAST_PATH,
 	FDB_SLOW_PATH,
 };
Signature · 1 line
2.20.1
#3 lore raw

PATCH 3/12 RDMA/mlx5: Move netdev info into the port struct

In reply to Leon Romanovsky

From: Mark Bloch <[email protected]>

Netdev info is stored in a separate array and holds data relevant on a per port basis, move it to be part of the port struct.

---
 drivers/infiniband/hw/mlx5/main.c    | 30 ++++++++++++++--------------
 drivers/infiniband/hw/mlx5/mlx5_ib.h | 14 ++++++-------
 drivers/infiniband/hw/mlx5/qp.c      |  2 +-
 3 files changed, 23 insertions(+), 23 deletions(-)
drivers/infiniband/hw/mlx5/main.c +15 -15
index e58ba0851983..a8522a0dedae 100644
@@ -257,11 +257,11 @@ static struct net_device *mlx5_ib_get_netdev(struct ib_device *device,
 
 	/* Ensure ndev does not disappear before we invoke dev_hold()
 	 */
-	read_lock(&ibdev->roce[port_num - 1].netdev_lock);
-	ndev = ibdev->roce[port_num - 1].netdev;
+	read_lock(&ibdev->port[port_num - 1].roce.netdev_lock);
+	ndev = ibdev->port[port_num - 1].roce.netdev;
 	if (ndev)
 		dev_hold(ndev);
-	read_unlock(&ibdev->roce[port_num - 1].netdev_lock);
+	read_unlock(&ibdev->port[port_num - 1].roce.netdev_lock);
 
 out:
 	mlx5_ib_put_native_port_mdev(ibdev, port_num);
@@ -1956,7 +1956,7 @@ static int mlx5_ib_alloc_ucontext(struct ib_ucontext *uctx,
 
 		atomic_set(&context->tx_port_affinity,
 			   atomic_add_return(
-				   1, &dev->roce[port].tx_port_affinity));
+				   1, &dev->port[port].roce.tx_port_affinity));
 	}
 
 	return 0;
@@ -5020,10 +5020,10 @@ static int mlx5_add_netdev_notifier(struct mlx5_ib_dev *dev, u8 port_num)
 {
 	int err;
 
-	dev->roce[port_num].nb.notifier_call = mlx5_netdev_event;
-	err = register_netdevice_notifier(&dev->roce[port_num].nb);
+	dev->port[port_num].roce.nb.notifier_call = mlx5_netdev_event;
+	err = register_netdevice_notifier(&dev->port[port_num].roce.nb);
 	if (err) {
-		dev->roce[port_num].nb.notifier_call = NULL;
+		dev->port[port_num].roce.nb.notifier_call = NULL;
 		return err;
 	}
 
@@ -5032,9 +5032,9 @@ static int mlx5_add_netdev_notifier(struct mlx5_ib_dev *dev, u8 port_num)
 
 static void mlx5_remove_netdev_notifier(struct mlx5_ib_dev *dev, u8 port_num)
 {
-	if (dev->roce[port_num].nb.notifier_call) {
-		unregister_netdevice_notifier(&dev->roce[port_num].nb);
-		dev->roce[port_num].nb.notifier_call = NULL;
+	if (dev->port[port_num].roce.nb.notifier_call) {
+		unregister_netdevice_notifier(&dev->port[port_num].roce.nb);
+		dev->port[port_num].roce.nb.notifier_call = NULL;
 	}
 }
 
@@ -5657,7 +5657,7 @@ static void mlx5_ib_unbind_slave_port(struct mlx5_ib_dev *ibdev,
 		mlx5_ib_err(ibdev, "Failed to unaffiliate port %u\n",
 			    port_num + 1);
 
-	ibdev->roce[port_num].last_port_state = IB_PORT_DOWN;
+	ibdev->port[port_num].roce.last_port_state = IB_PORT_DOWN;
 }
 
 /* The mlx5_ib_multiport_mutex should be held when calling this function */
@@ -5930,7 +5930,7 @@ int mlx5_ib_stage_init_init(struct mlx5_ib_dev *dev)
 
 	for (i = 0; i < dev->num_ports; i++) {
 		spin_lock_init(&dev->port[i].mp.mpi_lock);
-		rwlock_init(&dev->roce[i].netdev_lock);
+		rwlock_init(&dev->port[i].roce.netdev_lock);
 	}
 
 	err = mlx5_ib_init_multiport_master(dev);
@@ -6233,9 +6233,9 @@ static int mlx5_ib_stage_common_roce_init(struct mlx5_ib_dev *dev)
 	int i;
 
 	for (i = 0; i < dev->num_ports; i++) {
-		dev->roce[i].dev = dev;
-		dev->roce[i].native_port_num = i + 1;
-		dev->roce[i].last_port_state = IB_PORT_DOWN;
+		dev->port[i].roce.dev = dev;
+		dev->port[i].roce.native_port_num = i + 1;
+		dev->port[i].roce.last_port_state = IB_PORT_DOWN;
 	}
 
 	dev->ib_dev.uverbs_ex_cmd_mask |=
drivers/infiniband/hw/mlx5/mlx5_ib.h +7 -7
index 16311cb05cae..662cc73b7b6b 100644
@@ -706,12 +706,6 @@ struct mlx5_ib_multiport {
 	spinlock_t mpi_lock;
 };
 
-struct mlx5_ib_port {
-	struct mlx5_ib_counters cnts;
-	struct mlx5_ib_multiport mp;
-	struct mlx5_ib_dbg_cc_params	*dbg_cc_params;
-};
-
 struct mlx5_roce {
 	/* Protect mlx5_ib_get_netdev from invoking dev_hold() with a NULL
 	 * netdev pointer
@@ -725,6 +719,13 @@ struct mlx5_roce {
 	u8			native_port_num;
 };
 
+struct mlx5_ib_port {
+	struct mlx5_ib_counters cnts;
+	struct mlx5_ib_multiport mp;
+	struct mlx5_ib_dbg_cc_params *dbg_cc_params;
+	struct mlx5_roce roce;
+};
+
 struct mlx5_ib_dbg_param {
 	int			offset;
 	struct mlx5_ib_dev	*dev;
@@ -909,7 +910,6 @@ struct mlx5_ib_dev {
 	struct ib_device		ib_dev;
 	struct mlx5_core_dev		*mdev;
 	struct notifier_block		mdev_events;
-	struct mlx5_roce		roce[MLX5_MAX_PORTS];
 	int				num_ports;
 	/* serialize update of capability mask
 	 */
drivers/infiniband/hw/mlx5/qp.c +1 -1
index 9a4fca35875c..d84c7f478513 100644
@@ -3296,7 +3296,7 @@ static unsigned int get_tx_affinity(struct mlx5_ib_dev *dev,
 	} else {
 		tx_port_affinity =
 			(unsigned int)atomic_add_return(
-				1, &dev->roce[port_num].tx_port_affinity) %
+				1, &dev->port[port_num].roce.tx_port_affinity) %
 				MLX5_MAX_PORTS +
 			1;
 		mlx5_ib_dbg(dev, "Set tx affinity 0x%x to qpn 0x%x\n",
Signature · 1 line
2.20.1
#4 lore raw

PATCH 4/12 RDMA/mlx5: Free IB device on remove

In reply to Leon Romanovsky

From: Mark Bloch <[email protected]>

Simplify the code and move the deallocation of the IB device into the remove function.

---
 drivers/infiniband/hw/mlx5/ib_rep.c | 5 +----
 drivers/infiniband/hw/mlx5/main.c   | 4 ++--
 2 files changed, 3 insertions(+), 6 deletions(-)
drivers/infiniband/hw/mlx5/ib_rep.c +1 -4
index b8639ac71336..87d553396fb4 100644
@@ -65,10 +65,8 @@ mlx5_ib_vport_rep_load(struct mlx5_core_dev *dev, struct mlx5_eswitch_rep *rep)
 	ibdev->mdev = dev;
 	ibdev->num_ports = max(MLX5_CAP_GEN(dev, num_ports),
 			       MLX5_CAP_GEN(dev, num_vhca_ports));
-	if (!__mlx5_ib_add(ibdev, profile)) {
-		ib_dealloc_device(&ibdev->ib_dev);
+	if (!__mlx5_ib_add(ibdev, profile))
 		return -EINVAL;
-	}
 
 	rep->rep_if[REP_IB].priv = ibdev;
 
@@ -86,7 +84,6 @@ mlx5_ib_vport_rep_unload(struct mlx5_eswitch_rep *rep)
 	dev = mlx5_ib_rep_to_dev(rep);
 	__mlx5_ib_remove(dev, dev->profile, MLX5_IB_STAGE_MAX);
 	rep->rep_if[REP_IB].priv = NULL;
-	ib_dealloc_device(&dev->ib_dev);
 }
 
 static void *mlx5_ib_vport_get_proto_dev(struct mlx5_eswitch_rep *rep)
drivers/infiniband/hw/mlx5/main.c +2 -2
index a8522a0dedae..df38e24119ff 100644
@@ -6491,6 +6491,8 @@ void __mlx5_ib_remove(struct mlx5_ib_dev *dev,
 		if (profile->stage[stage].cleanup)
 			profile->stage[stage].cleanup(dev);
 	}
+
+	ib_dealloc_device(&dev->ib_dev);
 }
 
 void *__mlx5_ib_add(struct mlx5_ib_dev *dev,
@@ -6713,8 +6715,6 @@ static void mlx5_ib_remove(struct mlx5_core_dev *mdev, void *context)
 
 	dev = context;
 	__mlx5_ib_remove(dev, dev->profile, MLX5_IB_STAGE_MAX);
-
-	ib_dealloc_device((struct ib_device *)dev);
 }
 
 static struct mlx5_interface mlx5_ib_interface = {
Signature · 1 line
2.20.1
#5 lore raw

PATCH 5/12 RDMA/mlx5: Move ports allocation to outside of INIT stage

In reply to Leon Romanovsky

From: Mark Bloch <[email protected]>

In downstream patches we will need access to the ports before doing any stages, in order to set net device per representor.

---
 drivers/infiniband/hw/mlx5/ib_rep.c | 12 ++++++++++--
 drivers/infiniband/hw/mlx5/main.c   | 24 ++++++++++++------------
 2 files changed, 22 insertions(+), 14 deletions(-)
drivers/infiniband/hw/mlx5/ib_rep.c +10 -2
index 87d553396fb4..14ac728b460c 100644
@@ -51,6 +51,7 @@ mlx5_ib_vport_rep_load(struct mlx5_core_dev *dev, struct mlx5_eswitch_rep *rep)
 {
 	const struct mlx5_ib_profile *profile;
 	struct mlx5_ib_dev *ibdev;
+	int num_ports = 1;
 
 	if (rep->vport == MLX5_VPORT_UPLINK)
 		profile = &uplink_rep_profile;
@@ -61,10 +62,17 @@ mlx5_ib_vport_rep_load(struct mlx5_core_dev *dev, struct mlx5_eswitch_rep *rep)
 	if (!ibdev)
 		return -ENOMEM;
 
+	ibdev->port = kcalloc(num_ports, sizeof(*ibdev->port),
+			      GFP_KERNEL);
+	if (!ibdev->port) {
+		ib_dealloc_device(&ibdev->ib_dev);
+		return -ENOMEM;
+	}
+
 	ibdev->rep = rep;
 	ibdev->mdev = dev;
-	ibdev->num_ports = max(MLX5_CAP_GEN(dev, num_ports),
-			       MLX5_CAP_GEN(dev, num_vhca_ports));
+	ibdev->num_ports = num_ports;
+
 	if (!__mlx5_ib_add(ibdev, profile))
 		return -EINVAL;
 
drivers/infiniband/hw/mlx5/main.c +12 -12
index df38e24119ff..e3483ecc2f98 100644
@@ -5914,7 +5914,6 @@ void mlx5_ib_stage_init_cleanup(struct mlx5_ib_dev *dev)
 		srcu_barrier(&dev->mr_srcu);
 		cleanup_srcu_struct(&dev->mr_srcu);
 	}
-	kfree(dev->port);
 }
 
 int mlx5_ib_stage_init_init(struct mlx5_ib_dev *dev)
@@ -5923,11 +5922,6 @@ int mlx5_ib_stage_init_init(struct mlx5_ib_dev *dev)
 	int err;
 	int i;
 
-	dev->port = kcalloc(dev->num_ports, sizeof(*dev->port),
-			    GFP_KERNEL);
-	if (!dev->port)
-		return -ENOMEM;
-
 	for (i = 0; i < dev->num_ports; i++) {
 		spin_lock_init(&dev->port[i].mp.mpi_lock);
 		rwlock_init(&dev->port[i].roce.netdev_lock);
@@ -5935,7 +5929,7 @@ int mlx5_ib_stage_init_init(struct mlx5_ib_dev *dev)
 
 	err = mlx5_ib_init_multiport_master(dev);
 	if (err)
-		goto err_free_port;
+		return err;
 
 	if (!mlx5_core_mp_enabled(mdev)) {
 		for (i = 1; i <= dev->num_ports; i++) {
@@ -5976,9 +5970,6 @@ int mlx5_ib_stage_init_init(struct mlx5_ib_dev *dev)
 err_mp:
 	mlx5_ib_cleanup_multiport_master(dev);
 
-err_free_port:
-	kfree(dev->port);
-
 	return -ENOMEM;
 }
 
@@ -6492,6 +6483,7 @@ void __mlx5_ib_remove(struct mlx5_ib_dev *dev,
 			profile->stage[stage].cleanup(dev);
 	}
 
+	kfree(dev->port);
 	ib_dealloc_device(&dev->ib_dev);
 }
 
@@ -6667,6 +6659,7 @@ static void *mlx5_ib_add(struct mlx5_core_dev *mdev)
 	enum rdma_link_layer ll;
 	struct mlx5_ib_dev *dev;
 	int port_type_cap;
+	int num_ports;
 
 	printk_once(KERN_INFO "%s", mlx5_version);
 
@@ -6682,13 +6675,20 @@ static void *mlx5_ib_add(struct mlx5_core_dev *mdev)
 	if (mlx5_core_is_mp_slave(mdev) && ll == IB_LINK_LAYER_ETHERNET)
 		return mlx5_ib_add_slave_port(mdev);
 
+	num_ports = max(MLX5_CAP_GEN(mdev, num_ports),
+			MLX5_CAP_GEN(mdev, num_vhca_ports));
 	dev = ib_alloc_device(mlx5_ib_dev, ib_dev);
 	if (!dev)
 		return NULL;
+	dev->port = kcalloc(num_ports, sizeof(*dev->port),
+			     GFP_KERNEL);
+	if (!dev->port) {
+		ib_dealloc_device((struct ib_device *)dev);
+		return NULL;
+	}
 
 	dev->mdev = mdev;
-	dev->num_ports = max(MLX5_CAP_GEN(mdev, num_ports),
-			     MLX5_CAP_GEN(mdev, num_vhca_ports));
+	dev->num_ports = num_ports;
 
 	return __mlx5_ib_add(dev, &pf_profile);
 }
Signature · 1 line
2.20.1
#6 lore raw

PATCH 6/12 RDMA/mlx5: Use correct size for device resources

In reply to Leon Romanovsky

From: Mark Bloch <[email protected]>

On allocation we use the array size and on destruction num_ports, use the array size of destruction as well, in this context the array corresponds to the native/actual ports on the NIC so no need to adjust this logic for representors.

---
 drivers/infiniband/hw/mlx5/main.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)
drivers/infiniband/hw/mlx5/main.c +1 -3
index e3483ecc2f98..14cc4c56e2b1 100644
@@ -4862,8 +4862,6 @@ static int create_dev_resources(struct mlx5_ib_resources *devr)
 
 static void destroy_dev_resources(struct mlx5_ib_resources *devr)
 {
-	struct mlx5_ib_dev *dev =
-		container_of(devr, struct mlx5_ib_dev, devr);
 	int port;
 
 	mlx5_ib_destroy_srq(devr->s1);
@@ -4877,7 +4875,7 @@ static void destroy_dev_resources(struct mlx5_ib_resources *devr)
 	kfree(devr->p0);
 
 	/* Make sure no change P_Key work items are still executing */
-	for (port = 0; port < dev->num_ports; ++port)
+	for (port = 0; port < ARRAY_SIZE(devr->ports); ++port)
 		cancel_work_sync(&devr->ports[port].pkey_change_work);
 }
 
Signature · 1 line
2.20.1
#7 lore raw

PATCH 7/12 RDMA/mlx5: Move rep into port struct

In reply to Leon Romanovsky

From: Mark Bloch <[email protected]>

In preparation of moving into a model of single IB device multiple ports move rep to be part of the port structure. We mark a representor device by setting is_rep, no functional change with this patch.

---
 drivers/infiniband/hw/mlx5/devx.c    |  2 +-
 drivers/infiniband/hw/mlx5/flow.c    |  2 +-
 drivers/infiniband/hw/mlx5/ib_rep.c  |  7 ++++---
 drivers/infiniband/hw/mlx5/main.c    | 22 +++++++++++++---------
 drivers/infiniband/hw/mlx5/mlx5_ib.h |  3 ++-
 drivers/infiniband/hw/mlx5/mr.c      |  6 +++---
 drivers/infiniband/hw/mlx5/qp.c      |  4 ++--
 7 files changed, 26 insertions(+), 20 deletions(-)
drivers/infiniband/hw/mlx5/devx.c +1 -1
index 9e08df7914aa..687425ccc870 100644
@@ -1900,7 +1900,7 @@ static bool devx_is_supported(struct ib_device *device)
 {
 	struct mlx5_ib_dev *dev = to_mdev(device);
 
-	return !dev->rep && MLX5_CAP_GEN(dev->mdev, log_max_uctx);
+	return !dev->is_rep && MLX5_CAP_GEN(dev->mdev, log_max_uctx);
 }
 
 const struct uapi_definition mlx5_ib_devx_defs[] = {
drivers/infiniband/hw/mlx5/flow.c +1 -1
index 798591a18484..969fafb6ed9b 100644
@@ -621,7 +621,7 @@ DECLARE_UVERBS_NAMED_OBJECT(MLX5_IB_OBJECT_FLOW_MATCHER,
 
 static bool flow_is_supported(struct ib_device *device)
 {
-	return !to_mdev(device)->rep;
+	return !to_mdev(device)->is_rep;
 }
 
 const struct uapi_definition mlx5_ib_flow_defs[] = {
drivers/infiniband/hw/mlx5/ib_rep.c +4 -3
index 14ac728b460c..64256dc1d1de 100644
@@ -69,7 +69,8 @@ mlx5_ib_vport_rep_load(struct mlx5_core_dev *dev, struct mlx5_eswitch_rep *rep)
 		return -ENOMEM;
 	}
 
-	ibdev->rep = rep;
+	ibdev->is_rep = true;
+	ibdev->port[0].rep = rep;
 	ibdev->mdev = dev;
 	ibdev->num_ports = num_ports;
 
@@ -151,12 +152,12 @@ int create_flow_rule_vport_sq(struct mlx5_ib_dev *dev,
 	struct mlx5_flow_handle *flow_rule;
 	struct mlx5_eswitch *esw = dev->mdev->priv.eswitch;
 
-	if (!dev->rep)
+	if (!dev->is_rep)
 		return 0;
 
 	flow_rule =
 		mlx5_eswitch_add_send_to_vport_rule(esw,
-						    dev->rep->vport,
+						    dev->port[0].rep->vport,
 						    sq->base.mqp.qpn);
 	if (IS_ERR(flow_rule))
 		return PTR_ERR(flow_rule);
drivers/infiniband/hw/mlx5/main.c +13 -9
index 14cc4c56e2b1..d9dfc9975164 100644
@@ -173,12 +173,12 @@ static int mlx5_netdev_event(struct notifier_block *this,
 	switch (event) {
 	case NETDEV_REGISTER:
 		write_lock(&roce->netdev_lock);
-		if (ibdev->rep) {
+		if (ibdev->is_rep) {
 			struct mlx5_eswitch *esw = ibdev->mdev->priv.eswitch;
+			struct mlx5_eswitch_rep	*rep = ibdev->port[0].rep;
 			struct net_device *rep_ndev;
 
-			rep_ndev = mlx5_ib_get_rep_netdev(esw,
-							  ibdev->rep->vport);
+			rep_ndev = mlx5_ib_get_rep_netdev(esw, rep->vport);
 			if (rep_ndev == ndev)
 				roce->netdev = ndev;
 		} else if (ndev->dev.parent == &mdev->pdev->dev) {
@@ -3149,10 +3149,10 @@ static struct mlx5_ib_flow_prio *get_flow_table(struct mlx5_ib_dev *dev,
 		if (ft_type == MLX5_IB_FT_RX) {
 			fn_type = MLX5_FLOW_NAMESPACE_BYPASS;
 			prio = &dev->flow_db->prios[priority];
-			if (!dev->rep &&
+			if (!dev->is_rep &&
 			    MLX5_CAP_FLOWTABLE_NIC_RX(dev->mdev, decap))
 				flags |= MLX5_FLOW_TABLE_TUNNEL_EN_DECAP;
-			if (!dev->rep &&
+			if (!dev->is_rep &&
 			    MLX5_CAP_FLOWTABLE_NIC_RX(dev->mdev,
 					reformat_l3_tunnel_to_l2))
 				flags |= MLX5_FLOW_TABLE_TUNNEL_EN_REFORMAT;
@@ -3162,7 +3162,7 @@ static struct mlx5_ib_flow_prio *get_flow_table(struct mlx5_ib_dev *dev,
 							      log_max_ft_size));
 			fn_type = MLX5_FLOW_NAMESPACE_EGRESS;
 			prio = &dev->flow_db->egress_prios[priority];
-			if (!dev->rep &&
+			if (!dev->is_rep &&
 			    MLX5_CAP_FLOWTABLE_NIC_TX(dev->mdev, reformat))
 				flags |= MLX5_FLOW_TABLE_TUNNEL_EN_REFORMAT;
 		}
@@ -3368,7 +3368,7 @@ static struct mlx5_ib_flow_handler *_create_flow_rule(struct mlx5_ib_dev *dev,
 	if (!is_valid_attr(dev->mdev, flow_attr))
 		return ERR_PTR(-EINVAL);
 
-	if (dev->rep && is_egress)
+	if (dev->is_rep && is_egress)
 		return ERR_PTR(-EINVAL);
 
 	spec = kvzalloc(sizeof(*spec), GFP_KERNEL);
@@ -3399,13 +3399,17 @@ static struct mlx5_ib_flow_handler *_create_flow_rule(struct mlx5_ib_dev *dev,
 	if (!flow_is_multicast_only(flow_attr))
 		set_underlay_qp(dev, spec, underlay_qpn);
 
-	if (dev->rep) {
+	if (dev->is_rep) {
 		void *misc;
 
+		if (!dev->port[flow_attr->port - 1].rep) {
+			err = -EINVAL;
+			goto free;
+		}
 		misc = MLX5_ADDR_OF(fte_match_param, spec->match_value,
 				    misc_parameters);
 		MLX5_SET(fte_match_set_misc, misc, source_port,
-			 dev->rep->vport);
+			 dev->port[flow_attr->port - 1].rep->vport);
 		misc = MLX5_ADDR_OF(fte_match_param, spec->match_criteria,
 				    misc_parameters);
 		MLX5_SET_TO_ONES(fte_match_set_misc, misc, source_port);
drivers/infiniband/hw/mlx5/mlx5_ib.h +2 -1
index 662cc73b7b6b..885f87d0a3f9 100644
@@ -724,6 +724,7 @@ struct mlx5_ib_port {
 	struct mlx5_ib_multiport mp;
 	struct mlx5_ib_dbg_cc_params *dbg_cc_params;
 	struct mlx5_roce roce;
+	struct mlx5_eswitch_rep		*rep;
 };
 
 struct mlx5_ib_dbg_param {
@@ -944,7 +945,7 @@ struct mlx5_ib_dev {
 	struct mlx5_sq_bfreg	fp_bfreg;
 	struct mlx5_ib_delay_drop	delay_drop;
 	const struct mlx5_ib_profile	*profile;
-	struct mlx5_eswitch_rep		*rep;
+	bool			is_rep;
 	int				lag_active;
 
 	struct mlx5_ib_lb_state		lb;
drivers/infiniband/hw/mlx5/mr.c +3 -3
index c85f00255884..02119fe8338f 100644
@@ -600,7 +600,7 @@ static void clean_keys(struct mlx5_ib_dev *dev, int c)
 
 static void mlx5_mr_cache_debugfs_cleanup(struct mlx5_ib_dev *dev)
 {
-	if (!mlx5_debugfs_root || dev->rep)
+	if (!mlx5_debugfs_root || dev->is_rep)
 		return;
 
 	debugfs_remove_recursive(dev->cache.root);
@@ -614,7 +614,7 @@ static void mlx5_mr_cache_debugfs_init(struct mlx5_ib_dev *dev)
 	struct dentry *dir;
 	int i;
 
-	if (!mlx5_debugfs_root || dev->rep)
+	if (!mlx5_debugfs_root || dev->is_rep)
 		return;
 
 	cache->root = debugfs_create_dir("mr_cache", dev->mdev->priv.dbg_root);
@@ -677,7 +677,7 @@ int mlx5_mr_cache_init(struct mlx5_ib_dev *dev)
 			   MLX5_IB_UMR_OCTOWORD;
 		ent->access_mode = MLX5_MKC_ACCESS_MODE_MTT;
 		if ((dev->mdev->profile->mask & MLX5_PROF_MASK_MR_CACHE) &&
-		    !dev->rep &&
+		    !dev->is_rep &&
 		    mlx5_core_is_pf(dev->mdev))
 			ent->limit = dev->mdev->profile->mr_cache[i].limit;
 		else
drivers/infiniband/hw/mlx5/qp.c +2 -2
index d84c7f478513..c597e0b7f904 100644
@@ -1430,7 +1430,7 @@ static int create_raw_packet_qp_tir(struct mlx5_ib_dev *dev,
 	if (*qp_flags_en & MLX5_QP_FLAG_TIR_ALLOW_SELF_LB_MC)
 		lb_flag |= MLX5_TIRC_SELF_LB_BLOCK_BLOCK_MULTICAST;
 
-	if (dev->rep) {
+	if (dev->is_rep) {
 		lb_flag |= MLX5_TIRC_SELF_LB_BLOCK_BLOCK_UNICAST;
 		*qp_flags_en |= MLX5_QP_FLAG_TIR_ALLOW_SELF_LB_UC;
 	}
@@ -1642,7 +1642,7 @@ static int create_rss_raw_qp_tir(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp,
 		return -EOPNOTSUPP;
 	}
 
-	if (ucmd.flags & MLX5_QP_FLAG_TIR_ALLOW_SELF_LB_UC || dev->rep) {
+	if (ucmd.flags & MLX5_QP_FLAG_TIR_ALLOW_SELF_LB_UC || dev->is_rep) {
 		lb_flag |= MLX5_TIRC_SELF_LB_BLOCK_BLOCK_UNICAST;
 		qp->flags_en |= MLX5_QP_FLAG_TIR_ALLOW_SELF_LB_UC;
 	}
Signature · 1 line
2.20.1
#8 lore raw

PATCH 8/12 RDMA/mlx5: Move default representors SQ steering to rule to modify QP

In reply to Leon Romanovsky

From: Mark Bloch <[email protected]>

Currently the steering for SQs created on representors is done on creation, once we move to representors as ports of an IB device we need the port argument which is given only at the modify QP stage, adjust the code appropriately.

---
 drivers/infiniband/hw/mlx5/ib_rep.c | 25 +++++++++---------
 drivers/infiniband/hw/mlx5/ib_rep.h | 13 ++++++----
 drivers/infiniband/hw/mlx5/qp.c     | 40 ++++++++++++++++++++---------
 3 files changed, 48 insertions(+), 30 deletions(-)
drivers/infiniband/hw/mlx5/ib_rep.c +12 -13
index 64256dc1d1de..d3988f6ae2ae 100644
@@ -146,22 +146,21 @@ struct mlx5_eswitch_rep *mlx5_ib_vport_rep(struct mlx5_eswitch *esw, int vport)
 	return mlx5_eswitch_vport_rep(esw, vport);
 }
 
-int create_flow_rule_vport_sq(struct mlx5_ib_dev *dev,
-			      struct mlx5_ib_sq *sq)
+struct mlx5_flow_handle *create_flow_rule_vport_sq(struct mlx5_ib_dev *dev,
+						   struct mlx5_ib_sq *sq,
+						   u16 port)
 {
-	struct mlx5_flow_handle *flow_rule;
 	struct mlx5_eswitch *esw = dev->mdev->priv.eswitch;
+	struct mlx5_eswitch_rep *rep;
 
-	if (!dev->is_rep)
-		return 0;
+	if (!dev->is_rep || !port)
+		return NULL;
 
-	flow_rule =
-		mlx5_eswitch_add_send_to_vport_rule(esw,
-						    dev->port[0].rep->vport,
-						    sq->base.mqp.qpn);
-	if (IS_ERR(flow_rule))
-		return PTR_ERR(flow_rule);
-	sq->flow_rule = flow_rule;
+	if (!dev->port[port - 1].rep)
+		return ERR_PTR(-EINVAL);
 
-	return 0;
+	rep = dev->port[port - 1].rep;
+
+	return mlx5_eswitch_add_send_to_vport_rule(esw, rep->vport,
+						   sq->base.mqp.qpn);
 }
drivers/infiniband/hw/mlx5/ib_rep.h +8 -5
index 798d41e61fb4..1d9778da8a50 100644
@@ -20,8 +20,9 @@ struct mlx5_eswitch_rep *mlx5_ib_vport_rep(struct mlx5_eswitch *esw,
 					   int vport_index);
 void mlx5_ib_register_vport_reps(struct mlx5_core_dev *mdev);
 void mlx5_ib_unregister_vport_reps(struct mlx5_core_dev *mdev);
-int create_flow_rule_vport_sq(struct mlx5_ib_dev *dev,
-			      struct mlx5_ib_sq *sq);
+struct mlx5_flow_handle *create_flow_rule_vport_sq(struct mlx5_ib_dev *dev,
+						   struct mlx5_ib_sq *sq,
+						   u16 port);
 struct net_device *mlx5_ib_get_rep_netdev(struct mlx5_eswitch *esw,
 					  int vport_index);
 #else /* CONFIG_MLX5_ESWITCH */
@@ -52,10 +53,12 @@ struct mlx5_eswitch_rep *mlx5_ib_vport_rep(struct mlx5_eswitch *esw,
 
 static inline void mlx5_ib_register_vport_reps(struct mlx5_core_dev *mdev) {}
 static inline void mlx5_ib_unregister_vport_reps(struct mlx5_core_dev *mdev) {}
-static inline int create_flow_rule_vport_sq(struct mlx5_ib_dev *dev,
-					    struct mlx5_ib_sq *sq)
+static inline
+struct mlx5_flow_handle *create_flow_rule_vport_sq(struct mlx5_ib_dev *dev,
+						   struct mlx5_ib_sq *sq,
+						   u16 port)
 {
-	return 0;
+	return NULL;
 }
 
 static inline
drivers/infiniband/hw/mlx5/qp.c +28 -12
index c597e0b7f904..3ff7c32207b9 100644
@@ -93,6 +93,7 @@ struct mlx5_modify_raw_qp_param {
 	struct mlx5_rate_limit rl;
 
 	u8 rq_q_ctr_id;
+	u16 port;
 };
 
 static void get_cqs(enum ib_qp_type qp_type,
@@ -1207,11 +1208,11 @@ static void destroy_raw_packet_qp_tis(struct mlx5_ib_dev *dev,
 	mlx5_cmd_destroy_tis(dev->mdev, sq->tisn, to_mpd(pd)->uid);
 }
 
-static void destroy_flow_rule_vport_sq(struct mlx5_ib_dev *dev,
-				       struct mlx5_ib_sq *sq)
+static void destroy_flow_rule_vport_sq(struct mlx5_ib_sq *sq)
 {
 	if (sq->flow_rule)
 		mlx5_del_flow_rules(sq->flow_rule);
+	sq->flow_rule = NULL;
 }
 
 static int create_raw_packet_qp_sq(struct mlx5_ib_dev *dev,
@@ -1279,15 +1280,8 @@ static int create_raw_packet_qp_sq(struct mlx5_ib_dev *dev,
 	if (err)
 		goto err_umem;
 
-	err = create_flow_rule_vport_sq(dev, sq);
-	if (err)
-		goto err_flow;
-
 	return 0;
 
-err_flow:
-	mlx5_core_destroy_sq_tracked(dev->mdev, &sq->base.mqp);
-
 err_umem:
 	ib_umem_release(sq->ubuffer.umem);
 	sq->ubuffer.umem = NULL;
@@ -1298,7 +1292,7 @@ static int create_raw_packet_qp_sq(struct mlx5_ib_dev *dev,
 static void destroy_raw_packet_qp_sq(struct mlx5_ib_dev *dev,
 				     struct mlx5_ib_sq *sq)
 {
-	destroy_flow_rule_vport_sq(dev, sq);
+	destroy_flow_rule_vport_sq(sq);
 	mlx5_core_destroy_sq_tracked(dev->mdev, &sq->base.mqp);
 	ib_umem_release(sq->ubuffer.umem);
 }
@@ -3262,6 +3256,8 @@ static int modify_raw_packet_qp(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp,
 	}
 
 	if (modify_sq) {
+		struct mlx5_flow_handle *flow_rule;
+
 		if (tx_affinity) {
 			err = modify_raw_packet_tx_affinity(dev->mdev, sq,
 							    tx_affinity,
@@ -3270,8 +3266,25 @@ static int modify_raw_packet_qp(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp,
 				return err;
 		}
 
-		return modify_raw_packet_qp_sq(dev->mdev, sq, sq_state,
-					       raw_qp_param, qp->ibqp.pd);
+		flow_rule = create_flow_rule_vport_sq(dev, sq,
+						      raw_qp_param->port);
+		if (IS_ERR(flow_rule))
+			return err;
+
+		err = modify_raw_packet_qp_sq(dev->mdev, sq, sq_state,
+					      raw_qp_param, qp->ibqp.pd);
+		if (err) {
+			if (flow_rule)
+				mlx5_del_flow_rules(flow_rule);
+			return err;
+		}
+
+		if (flow_rule) {
+			destroy_flow_rule_vport_sq(sq);
+			sq->flow_rule = flow_rule;
+		}
+
+		return err;
 	}
 
 	return 0;
@@ -3588,6 +3601,9 @@ static int __mlx5_ib_modify_qp(struct ib_qp *ibqp,
 			raw_qp_param.set_mask |= MLX5_RAW_QP_MOD_SET_RQ_Q_CTR_ID;
 		}
 
+		if (attr_mask & IB_QP_PORT)
+			raw_qp_param.port = attr->port_num;
+
 		if (attr_mask & IB_QP_RATE_LIMIT) {
 			raw_qp_param.rl.rate = attr->rate_limit;
 
Signature · 1 line
2.20.1
#9 lore raw

PATCH 9/12 RDMA/mlx5: Refactor netdev affinity code

In reply to Leon Romanovsky

From: Mark Bloch <[email protected]>

The design of representors is such that once an IB representor is created, the netdev of representor already exists, we can use that fact to simplify the netdev affinity code.

---
 drivers/infiniband/hw/mlx5/ib_rep.c |  2 ++
 drivers/infiniband/hw/mlx5/main.c   | 47 +++++++++++++++++++++++------
 2 files changed, 39 insertions(+), 10 deletions(-)
drivers/infiniband/hw/mlx5/ib_rep.c +2 -0
index d3988f6ae2ae..7946cf26421b 100644
@@ -71,6 +71,8 @@ mlx5_ib_vport_rep_load(struct mlx5_core_dev *dev, struct mlx5_eswitch_rep *rep)
 
 	ibdev->is_rep = true;
 	ibdev->port[0].rep = rep;
+	ibdev->port[0].roce.netdev =
+		mlx5_ib_get_rep_netdev(dev->priv.eswitch, rep->vport);
 	ibdev->mdev = dev;
 	ibdev->num_ports = num_ports;
 
drivers/infiniband/hw/mlx5/main.c +37 -10

71 lines not on this page · raw message

Signature · 1 line
2.20.1
#10 lore raw

PATCH 10/12 RDMA/mlx5: Move SMI caps logic

In reply to Leon Romanovsky

From: Mark Bloch <[email protected]>

We store the SMI information in the core device's struct, make sure we set that information only once (and not per port), while here make the for loop based on the actual size of the array.

---
 drivers/infiniband/hw/mlx5/main.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)
drivers/infiniband/hw/mlx5/main.c +5 -5
index 9ce8ae5565a3..7eca3978c50c 100644
@@ -4534,7 +4534,7 @@ static int set_has_smi_cap(struct mlx5_ib_dev *dev)
 	int err;
 	int port;
 
-	for (port = 1; port <= dev->num_ports; port++) {
+	for (port = 1; port <= ARRAY_SIZE(dev->mdev->port_caps); port++) {
 		dev->mdev->port_caps[port - 1].has_smi = false;
 		if (MLX5_CAP_GEN(dev->mdev, port_type) ==
 		    MLX5_CAP_PORT_TYPE_IB) {
@@ -4580,10 +4580,6 @@ static int get_port_caps(struct mlx5_ib_dev *dev, u8 port)
 	if (!dprops)
 		goto out;
 
-	err = set_has_smi_cap(dev);
-	if (err)
-		goto out;
-
 	err = mlx5_ib_query_device(&dev->ib_dev, dprops, &uhw);
 	if (err) {
 		mlx5_ib_warn(dev, "query_device failed %d\n", err);
@@ -5960,6 +5956,10 @@ int mlx5_ib_stage_init_init(struct mlx5_ib_dev *dev)
 	if (err)
 		return err;
 
+	err = set_has_smi_cap(dev);
+	if (err)
+		return err;
+
 	if (!mlx5_core_mp_enabled(mdev)) {
 		for (i = 1; i <= dev->num_ports; i++) {
 			err = get_port_caps(dev, i);
Signature · 1 line
2.20.1
#11 lore raw

PATCH 11/12 RDMA/mlx5: Move to single device multiport ports in switchdev mode

In reply to Leon Romanovsky

From: Mark Bloch <[email protected]>

Move from IB device (representor) per virtual function to single IB device with port per virtual function (port 1 represents the uplink). As number of ports is a static property of an IB device, declare the IB device with as many port as the possible according to the PCI bus.

---
 drivers/infiniband/hw/mlx5/ib_rep.c  | 31 +++++++++++++++++++++++-----
 drivers/infiniband/hw/mlx5/main.c    | 26 +++++++++++++++++++----
 drivers/infiniband/hw/mlx5/mlx5_ib.h |  1 +
 3 files changed, 49 insertions(+), 9 deletions(-)
drivers/infiniband/hw/mlx5/ib_rep.c +26 -5
index 7946cf26421b..224ef6c88d17 100644
@@ -46,17 +46,36 @@ static const struct mlx5_ib_profile vf_rep_profile = {
 		     NULL),
 };
 
+static int
+mlx5_ib_set_vport_rep(struct mlx5_core_dev *dev, struct mlx5_eswitch_rep *rep)
+{
+	struct mlx5_ib_dev *ibdev;
+	int vport_index;
+
+	ibdev = mlx5_ib_get_uplink_ibdev(dev->priv.eswitch);
+	vport_index = ibdev->free_port++;
+
+	ibdev->port[vport_index].rep = rep;
+	write_lock(&ibdev->port[vport_index].roce.netdev_lock);
+	ibdev->port[vport_index].roce.netdev =
+		mlx5_ib_get_rep_netdev(dev->priv.eswitch, rep->vport);
+	write_unlock(&ibdev->port[vport_index].roce.netdev_lock);
+
+	return 0;
+}
+
 static int
 mlx5_ib_vport_rep_load(struct mlx5_core_dev *dev, struct mlx5_eswitch_rep *rep)
 {
+	int num_ports = MLX5_TOTAL_VPORTS(dev);
 	const struct mlx5_ib_profile *profile;
 	struct mlx5_ib_dev *ibdev;
-	int num_ports = 1;
+	int vport_index;
 
 	if (rep->vport == MLX5_VPORT_UPLINK)
 		profile = &uplink_rep_profile;
 	else
-		profile = &vf_rep_profile;
+		return mlx5_ib_set_vport_rep(dev, rep);
 
 	ibdev = ib_alloc_device(mlx5_ib_dev, ib_dev);
 	if (!ibdev)
@@ -70,8 +89,9 @@ mlx5_ib_vport_rep_load(struct mlx5_core_dev *dev, struct mlx5_eswitch_rep *rep)
 	}
 
 	ibdev->is_rep = true;
-	ibdev->port[0].rep = rep;
-	ibdev->port[0].roce.netdev =
+	vport_index = ibdev->free_port++;
+	ibdev->port[vport_index].rep = rep;
+	ibdev->port[vport_index].roce.netdev =
 		mlx5_ib_get_rep_netdev(dev->priv.eswitch, rep->vport);
 	ibdev->mdev = dev;
 	ibdev->num_ports = num_ports;
@@ -89,7 +109,8 @@ mlx5_ib_vport_rep_unload(struct mlx5_eswitch_rep *rep)
 {
 	struct mlx5_ib_dev *dev;
 
-	if (!rep->rep_if[REP_IB].priv)
+	if (!rep->rep_if[REP_IB].priv ||
+	    rep->vport != MLX5_VPORT_UPLINK)
 		return;
 
 	dev = mlx5_ib_rep_to_dev(rep);
drivers/infiniband/hw/mlx5/main.c +22 -4
index 7eca3978c50c..57d269c1b874 100644
@@ -506,9 +506,14 @@ static int mlx5_query_port_roce(struct ib_device *device, u8 port_num,
 
 	/* Possible bad flows are checked before filling out props so in case
 	 * of an error it will still be zeroed out.
+	 * Use native port in case of reps
 	 */
-	err = mlx5_query_port_ptys(mdev, out, sizeof(out), MLX5_PTYS_EN,
-				   mdev_port_num);
+	if (dev->is_rep)
+		err = mlx5_query_port_ptys(mdev, out, sizeof(out), MLX5_PTYS_EN,
+					   1);
+	else
+		err = mlx5_query_port_ptys(mdev, out, sizeof(out), MLX5_PTYS_EN,
+					   mdev_port_num);
 	if (err)
 		goto out;
 	ext = MLX5_CAP_PCAM_FEATURE(dev->mdev, ptys_extended_ethernet);
@@ -1432,7 +1437,9 @@ static int mlx5_ib_rep_query_port(struct ib_device *ibdev, u8 port,
 {
 	int ret;
 
-	/* Only link layer == ethernet is valid for representors */
+	/* Only link layer == ethernet is valid for representors
+	 * and we always use port 1
+	 */
 	ret = mlx5_query_port_roce(ibdev, port, props);
 	if (ret || !props)
 		return ret;
@@ -4565,7 +4572,7 @@ static void get_ext_port_caps(struct mlx5_ib_dev *dev)
 		mlx5_query_ext_port_caps(dev, port);
 }
 
-static int get_port_caps(struct mlx5_ib_dev *dev, u8 port)
+static int __get_port_caps(struct mlx5_ib_dev *dev, u8 port)
 {
 	struct ib_device_attr *dprops = NULL;
 	struct ib_port_attr *pprops = NULL;
@@ -4608,6 +4615,16 @@ static int get_port_caps(struct mlx5_ib_dev *dev, u8 port)
 	return err;
 }
 
+static int get_port_caps(struct mlx5_ib_dev *dev, u8 port)
+{
+	/* For representors use port 1, is this is the only native
+	 * port
+	 */
+	if (dev->is_rep)
+		return __get_port_caps(dev, 1);
+	return __get_port_caps(dev, port);
+}
+
 static void destroy_umrc_res(struct mlx5_ib_dev *dev)
 {
 	int err;
@@ -6268,6 +6285,7 @@ static int mlx5_ib_stage_common_roce_init(struct mlx5_ib_dev *dev)
 
 	port_num = mlx5_core_native_port_num(dev->mdev) - 1;
 
+	/* Register only for native ports */
 	return mlx5_add_netdev_notifier(dev, port_num);
 }
 
drivers/infiniband/hw/mlx5/mlx5_ib.h +1 -0
index 885f87d0a3f9..7daa20102b1d 100644
@@ -956,6 +956,7 @@ struct mlx5_ib_dev {
 	u16			devx_whitelist_uid;
 	struct mlx5_srq_table   srq_table;
 	struct mlx5_async_ctx   async_ctx;
+	int			free_port;
 };
 
 static inline struct mlx5_ib_cq *to_mibcq(struct mlx5_core_cq *mcq)
Signature · 1 line
2.20.1
#12 lore raw

PATCH 12/12 RDMA/mlx5: Remove VF representor profile

In reply to Leon Romanovsky

From: Mark Bloch <[email protected]>

Now that we have a single IB device with multipul ports we can remove the VF representor profile.

---
 drivers/infiniband/hw/mlx5/ib_rep.c  | 39 -----------------------
 drivers/infiniband/hw/mlx5/main.c    | 46 ++++++++++------------------
 drivers/infiniband/hw/mlx5/mlx5_ib.h | 17 ----------
 3 files changed, 16 insertions(+), 86 deletions(-)
drivers/infiniband/hw/mlx5/ib_rep.c +0 -39
index 224ef6c88d17..cbcc40d776b9 100644
@@ -7,45 +7,6 @@
 #include "ib_rep.h"
 #include "srq.h"
 
-static const struct mlx5_ib_profile vf_rep_profile = {
-	STAGE_CREATE(MLX5_IB_STAGE_INIT,
-		     mlx5_ib_stage_init_init,
-		     mlx5_ib_stage_init_cleanup),
-	STAGE_CREATE(MLX5_IB_STAGE_FLOW_DB,
-		     mlx5_ib_stage_rep_flow_db_init,
-		     NULL),
-	STAGE_CREATE(MLX5_IB_STAGE_CAPS,
-		     mlx5_ib_stage_caps_init,
-		     NULL),
-	STAGE_CREATE(MLX5_IB_STAGE_NON_DEFAULT_CB,
-		     mlx5_ib_stage_rep_non_default_cb,
-		     NULL),
-	STAGE_CREATE(MLX5_IB_STAGE_ROCE,
-		     mlx5_ib_stage_rep_roce_init,
-		     mlx5_ib_stage_rep_roce_cleanup),
-	STAGE_CREATE(MLX5_IB_STAGE_SRQ,
-		     mlx5_init_srq_table,
-		     mlx5_cleanup_srq_table),
-	STAGE_CREATE(MLX5_IB_STAGE_DEVICE_RESOURCES,
-		     mlx5_ib_stage_dev_res_init,
-		     mlx5_ib_stage_dev_res_cleanup),
-	STAGE_CREATE(MLX5_IB_STAGE_COUNTERS,
-		     mlx5_ib_stage_counters_init,
-		     mlx5_ib_stage_counters_cleanup),
-	STAGE_CREATE(MLX5_IB_STAGE_BFREG,
-		     mlx5_ib_stage_bfrag_init,
-		     mlx5_ib_stage_bfrag_cleanup),
-	STAGE_CREATE(MLX5_IB_STAGE_PRE_IB_REG_UMR,
-		     NULL,
-		     mlx5_ib_stage_pre_ib_reg_umr_cleanup),
-	STAGE_CREATE(MLX5_IB_STAGE_IB_REG,
-		     mlx5_ib_stage_ib_reg_init,
-		     mlx5_ib_stage_ib_reg_cleanup),
-	STAGE_CREATE(MLX5_IB_STAGE_POST_IB_REG_UMR,
-		     mlx5_ib_stage_post_ib_reg_umr_init,
-		     NULL),
-};
-
 static int
 mlx5_ib_set_vport_rep(struct mlx5_core_dev *dev, struct mlx5_eswitch_rep *rep)
 {
drivers/infiniband/hw/mlx5/main.c +16 -30

141 lines not on this page · raw message

drivers/infiniband/hw/mlx5/mlx5_ib.h +0 -17
index 7daa20102b1d..505332015ff2 100644
@@ -1235,23 +1235,6 @@ static inline void mlx5_ib_invalidate_range(struct ib_umem_odp *umem_odp,
 #endif /* CONFIG_INFINIBAND_ON_DEMAND_PAGING */
 
 /* Needed for rep profile */
-int mlx5_ib_stage_init_init(struct mlx5_ib_dev *dev);
-void mlx5_ib_stage_init_cleanup(struct mlx5_ib_dev *dev);
-int mlx5_ib_stage_rep_flow_db_init(struct mlx5_ib_dev *dev);
-int mlx5_ib_stage_caps_init(struct mlx5_ib_dev *dev);
-int mlx5_ib_stage_rep_non_default_cb(struct mlx5_ib_dev *dev);
-int mlx5_ib_stage_rep_roce_init(struct mlx5_ib_dev *dev);
-void mlx5_ib_stage_rep_roce_cleanup(struct mlx5_ib_dev *dev);
-int mlx5_ib_stage_dev_res_init(struct mlx5_ib_dev *dev);
-void mlx5_ib_stage_dev_res_cleanup(struct mlx5_ib_dev *dev);
-int mlx5_ib_stage_counters_init(struct mlx5_ib_dev *dev);
-void mlx5_ib_stage_counters_cleanup(struct mlx5_ib_dev *dev);
-int mlx5_ib_stage_bfrag_init(struct mlx5_ib_dev *dev);
-void mlx5_ib_stage_bfrag_cleanup(struct mlx5_ib_dev *dev);
-void mlx5_ib_stage_pre_ib_reg_umr_cleanup(struct mlx5_ib_dev *dev);
-int mlx5_ib_stage_ib_reg_init(struct mlx5_ib_dev *dev);
-void mlx5_ib_stage_ib_reg_cleanup(struct mlx5_ib_dev *dev);
-int mlx5_ib_stage_post_ib_reg_umr_init(struct mlx5_ib_dev *dev);
 void __mlx5_ib_remove(struct mlx5_ib_dev *dev,
 		      const struct mlx5_ib_profile *profile,
 		      int stage);
Signature · 1 line
2.20.1
#13 lore raw
In reply to Leon Romanovsky
Quoted 28 lines · Leon Romanovsky

From: Leon Romanovsky <[email protected]>

Quoted 1 line

From Mark,

Hi,

This series starts from two refactoring patches performed in mlx5_core and they are part of a larger series intended to expose the ability to insert FDB steering rules by the RDMA side.

They are followed by patches which move IB representors to single IB device. Currently when moving to switchdev mode an IB device is created per virtual function (and one for the uplink). While initially this was right approach, the limitation of this model requires us to change it.

Such limitations includes:
- Inability to share resources between IB devices.
- Lack of DEVX to allow more control and device specific functionality.
- Interacting with the E-Switch.
- Doesn't work well with namespaces.

In this series we move to a single IB device with multiple ports where each port represents a different virtual function (port 1 is for the uplink). All the limitations listed above are now resolved and the software model exposed to users is more cohesive.

Thanks

Quoted 13 lines

Mark Bloch (12):
net/mlx5: E-Switch, don't use hardcoded values for FDB prios net/mlx5: E-Switch, add a new prio to be used by the RDMA side RDMA/mlx5: Move netdev info into the port struct RDMA/mlx5: Free IB device on remove
RDMA/mlx5: Move ports allocation to outside of INIT stage RDMA/mlx5: Use correct size for device resources RDMA/mlx5: Move rep into port struct
RDMA/mlx5: Move default representors SQ steering to rule to modify QP RDMA/mlx5: Refactor netdev affinity code
RDMA/mlx5: Move SMI caps logic
RDMA/mlx5: Move to single device multiport ports in switchdev mode RDMA/mlx5: Remove VF representor profile

This looks Ok, can you update the shared branch please

Jason

#14 lore raw
Quoted 43 lines · Jason Gunthorpe
Quoted 28 lines · Leon Romanovsky

From: Leon Romanovsky <[email protected]>

Quoted 1 line

From Mark,

Hi,

This series starts from two refactoring patches performed in mlx5_core and they are part of a larger series intended to expose the ability to insert FDB steering rules by the RDMA side.

They are followed by patches which move IB representors to single IB device. Currently when moving to switchdev mode an IB device is created per virtual function (and one for the uplink). While initially this was right approach, the limitation of this model requires us to change it.

Such limitations includes:
- Inability to share resources between IB devices.
- Lack of DEVX to allow more control and device specific functionality.
- Interacting with the E-Switch.
- Doesn't work well with namespaces.

In this series we move to a single IB device with multiple ports where each port represents a different virtual function (port 1 is for the uplink). All the limitations listed above are now resolved and the software model exposed to users is more cohesive.

Thanks

Quoted 13 lines

Mark Bloch (12):
net/mlx5: E-Switch, don't use hardcoded values for FDB prios net/mlx5: E-Switch, add a new prio to be used by the RDMA side RDMA/mlx5: Move netdev info into the port struct RDMA/mlx5: Free IB device on remove
RDMA/mlx5: Move ports allocation to outside of INIT stage RDMA/mlx5: Use correct size for device resources RDMA/mlx5: Move rep into port struct
RDMA/mlx5: Move default representors SQ steering to rule to modify QP RDMA/mlx5: Refactor netdev affinity code
RDMA/mlx5: Move SMI caps logic
RDMA/mlx5: Move to single device multiport ports in switchdev mode RDMA/mlx5: Remove VF representor profile

Quoted 1 line

This looks Ok, can you update the shared branch please

Thanks Jason,

I'll slightly delay update till Sunday, internally, we have doubts about "RDMA/mlx5: Use correct size for device resources" patch.

Thanks

Quoted 2 lines

Jason

#15 lore raw
Quoted 44 lines · Leon Romanovsky
Quoted 43 lines · Jason Gunthorpe
Quoted 41 lines · Leon Romanovsky

From: Leon Romanovsky <[email protected]>

Quoted 1 line

From Mark,

Hi,

This series starts from two refactoring patches performed in mlx5_core and they are part of a larger series intended to expose the ability to insert FDB steering rules by the RDMA side.

They are followed by patches which move IB representors to single IB device. Currently when moving to switchdev mode an IB device is created per virtual function (and one for the uplink). While initially this was right approach, the limitation of this model requires us to change it.

Such limitations includes:
- Inability to share resources between IB devices.
- Lack of DEVX to allow more control and device specific functionality.
- Interacting with the E-Switch.
- Doesn't work well with namespaces.

In this series we move to a single IB device with multiple ports where each port represents a different virtual function (port 1 is for the uplink). All the limitations listed above are now resolved and the software model exposed to users is more cohesive.

Thanks

Mark Bloch (12):
net/mlx5: E-Switch, don't use hardcoded values for FDB prios net/mlx5: E-Switch, add a new prio to be used by the RDMA side RDMA/mlx5: Move netdev info into the port struct RDMA/mlx5: Free IB device on remove
RDMA/mlx5: Move ports allocation to outside of INIT stage RDMA/mlx5: Use correct size for device resources RDMA/mlx5: Move rep into port struct
RDMA/mlx5: Move default representors SQ steering to rule to modify QP RDMA/mlx5: Refactor netdev affinity code
RDMA/mlx5: Move SMI caps logic
RDMA/mlx5: Move to single device multiport ports in switchdev mode RDMA/mlx5: Remove VF representor profile

Quoted 6 lines
Quoted 1 line

This looks Ok, can you update the shared branch please

Thanks Jason,

I'll slightly delay update till Sunday, internally, we have doubts about "RDMA/mlx5: Use correct size for device resources" patch.

Hi Jason,

I updated our mlx5-next with first two patches.
d9cb06759eca net/mlx5: E-Switch, add a new prio to be used by the RDMA side b6d9ccb11250 net/mlx5: E-Switch, don't use hardcoded values for FDB prios

We found the reason for instability over bond interface and it was related to the patch "RDMA/mlx5: Move netdev info into the port struct". The following fixup is needed. It will be helpful if you can apply it internally, or I can resend rest of the series.

commit 775efa8dd53e5148d8d4942d2d0b7e72ca606d40

Author: Mark Bloch <[email protected]>
Date:   Tue Apr 9 03:28:59 2019 +0000

fixup! RDMA/mlx5: Move netdev info into the port struct

drivers/infiniband/hw/mlx5/main.c +1 -1
index fdcc76383ae9..6f479dc66b78 100644
@@ -1958,7 +1958,7 @@ static int mlx5_ib_alloc_ucontext(struct ib_ucontext *uctx,
 	print_lib_caps(dev, context->lib_caps);
 
 	if (dev->lag_active) {
-		u8 port = mlx5_core_native_port_num(dev->mdev);
+		u8 port = mlx5_core_native_port_num(dev->mdev) - 1;
 
 		atomic_set(&context->tx_port_affinity,
 			   atomic_add_return(
drivers/infiniband/hw/mlx5/qp.c +1 -1
index 832f6d176e25..31039fe1dd20 100644
@@ -3491,7 +3491,7 @@ static int __mlx5_ib_modify_qp(struct ib_qp *ibqp,
 		    (ibqp->qp_type == IB_QPT_XRC_INI) ||
 		    (ibqp->qp_type == IB_QPT_XRC_TGT)) {
 			if (dev->lag_active) {
-				u8 p = mlx5_core_native_port_num(dev->mdev);
+				u8 p = mlx5_core_native_port_num(dev->mdev) - 1;
 				tx_affinity = get_tx_affinity(dev, pd, base, p,
 							      udata);
 				context->flags |= cpu_to_be32(tx_affinity << 24);
Quoted 5 lines

Thanks

Quoted 2 lines

Jason

#16 lore raw
Quoted 53 lines · Leon Romanovsky
Quoted 14 lines
Quoted 14 lines
Quoted 13 lines

Mark Bloch (12):
net/mlx5: E-Switch, don't use hardcoded values for FDB prios net/mlx5: E-Switch, add a new prio to be used by the RDMA side RDMA/mlx5: Move netdev info into the port struct RDMA/mlx5: Free IB device on remove
RDMA/mlx5: Move ports allocation to outside of INIT stage RDMA/mlx5: Use correct size for device resources RDMA/mlx5: Move rep into port struct
RDMA/mlx5: Move default representors SQ steering to rule to modify QP RDMA/mlx5: Refactor netdev affinity code
RDMA/mlx5: Move SMI caps logic
RDMA/mlx5: Move to single device multiport ports in switchdev mode RDMA/mlx5: Remove VF representor profile

Quoted 6 lines
Quoted 1 line

This looks Ok, can you update the shared branch please

Thanks Jason,

I'll slightly delay update till Sunday, internally, we have doubts about "RDMA/mlx5: Use correct size for device resources" patch.

Hi Jason,

I updated our mlx5-next with first two patches.
d9cb06759eca net/mlx5: E-Switch, add a new prio to be used by the RDMA side b6d9ccb11250 net/mlx5: E-Switch, don't use hardcoded values for FDB prios

We found the reason for instability over bond interface and it was related to the patch "RDMA/mlx5: Move netdev info into the port struct". The following fixup is needed. It will be helpful if you can apply it internally, or I can resend rest of the series.

commit 775efa8dd53e5148d8d4942d2d0b7e72ca606d40

Author: Mark Bloch <[email protected]>
Date:   Tue Apr 9 03:28:59 2019 +0000

fixup! RDMA/mlx5: Move netdev info into the port struct

drivers/infiniband/hw/mlx5/main.c +2 -1
index fdcc76383ae9..6f479dc66b78 100644
+++ b/drivers/infiniband/hw/mlx5/main.c
@@ -1958,7 +1958,7 @@ static int mlx5_ib_alloc_ucontext(struct ib_ucontext *uctx,
 	print_lib_caps(dev, context->lib_caps);
 
 	if (dev->lag_active) {
-		u8 port = mlx5_core_native_port_num(dev->mdev);
+		u8 port = mlx5_core_native_port_num(dev->mdev) - 1;
 
 		atomic_set(&context->tx_port_affinity,
 			   atomic_add_return(
drivers/infiniband/hw/mlx5/qp.c +1 -0
index 832f6d176e25..31039fe1dd20 100644
+++ b/drivers/infiniband/hw/mlx5/qp.c

drivers/infiniband/hw/mlx5/qp.c static int __mlx5_ib_modify_qp(struct ib_qp *ibqp, · line 3497

Quoted 8 lines
drivers/infiniband/hw/mlx5/qp.c +1 -1
@@ -3491,7 +3491,7 @@ static int __mlx5_ib_modify_qp(struct ib_qp *ibqp,
 		    (ibqp->qp_type == IB_QPT_XRC_INI) ||
 		    (ibqp->qp_type == IB_QPT_XRC_TGT)) {
 			if (dev->lag_active) {
-				u8 p = mlx5_core_native_port_num(dev->mdev);
+				u8 p = mlx5_core_native_port_num(dev->mdev) - 1;
 				tx_affinity = get_tx_affinity(dev, pd, base, p,
 							      udata);
 				context->flags |= cpu_to_be32(tx_affinity << 24);

Okay, done

Jason