Skip to contentSkip to conversation

RDMA/mlx4: Provide port number for special QPs

PATCH In review Leon Romanovsky 2 participants 5 messages Sep 14, 2020 – Sep 18, 2020

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH v3 1/1 RDMA/mlx4: Provide port number for special QPs

From: Leon Romanovsky <[email protected]>

Special QPs created by mlx4 have same QP port borrowed from the context, while they are expected to have different ones.

Fix it by using HW physical port instead.

It fixes the following error during driver init:

[   12.074150] mlx4_core 0000:05:00.0: mlx4_ib: initializing demux service for 128 qp1 clients
[   12.084036] <mlx4_ib> create_pv_sqp: Couldn't create special QP (-16)
[   12.085123] <mlx4_ib> create_pv_resources: Couldn't create  QP1 (-16)
[   12.088300] mlx4_en: Mellanox ConnectX HCA Ethernet driver v4.0-0

---
Changelog:
v3: mlx4 devices create 2 special QPs in SRIOV mode, separate them by port number and special bit. The mlx4 is limited to two ports and not going to be extended, and the port_num is not forwarded to FW too, so it is safe.
v2: https://lore.kernel.org/linux-rdma/[email protected]/#r

---
 drivers/infiniband/hw/mlx4/mad.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)
drivers/infiniband/hw/mlx4/mad.c +5 -4
index 8bd16474708f..4b565640ba85 100644
@@ -1792,7 +1792,7 @@ static void pv_qp_event_handler(struct ib_event *event, void *qp_context)
 }
 
 static int create_pv_sqp(struct mlx4_ib_demux_pv_ctx *ctx,
-			    enum ib_qp_type qp_type, int create_tun)
+			 enum ib_qp_type qp_type, int port, int create_tun)
 {
 	int i, ret;
 	struct mlx4_ib_demux_pv_qp *tun_qp;
@@ -1822,12 +1822,13 @@ static int create_pv_sqp(struct mlx4_ib_demux_pv_ctx *ctx,
 		qp_init_attr.proxy_qp_type = qp_type;
 		qp_attr_mask_INIT = IB_QP_STATE | IB_QP_PKEY_INDEX |
 			   IB_QP_QKEY | IB_QP_PORT;
+		qp_init_attr.init_attr.port_num = ctx->port;
 	} else {
 		qp_init_attr.init_attr.qp_type = qp_type;
 		qp_init_attr.init_attr.create_flags = MLX4_IB_SRIOV_SQP;
 		qp_attr_mask_INIT = IB_QP_STATE | IB_QP_PKEY_INDEX | IB_QP_QKEY;
+		qp_init_attr.init_attr.port_num = port | 1 << 7;
 	}
-	qp_init_attr.init_attr.port_num = ctx->port;
 	qp_init_attr.init_attr.qp_context = ctx;
 	qp_init_attr.init_attr.event_handler = pv_qp_event_handler;
 	tun_qp->qp = ib_create_qp(ctx->pd, &qp_init_attr.init_attr);
@@ -2026,7 +2027,7 @@ static int create_pv_resources(struct ib_device *ibdev, int slave, int port,
 	}
 
 	if (ctx->has_smi) {
-		ret = create_pv_sqp(ctx, IB_QPT_SMI, create_tun);
+		ret = create_pv_sqp(ctx, IB_QPT_SMI, port, create_tun);
 		if (ret) {
 			pr_err("Couldn't create %s QP0 (%d)\n",
 			       create_tun ? "tunnel for" : "",  ret);
@@ -2034,7 +2035,7 @@ static int create_pv_resources(struct ib_device *ibdev, int slave, int port,
 		}
 	}
 
-	ret = create_pv_sqp(ctx, IB_QPT_GSI, create_tun);
+	ret = create_pv_sqp(ctx, IB_QPT_GSI, port, create_tun);
 	if (ret) {
 		pr_err("Couldn't create %s QP1 (%d)\n",
 		       create_tun ? "tunnel for" : "",  ret);

--
2.26.2

maintainer
#1 lore raw
Quoted 21 lines · Leon Romanovsky

From: Leon Romanovsky <[email protected]>

Special QPs created by mlx4 have same QP port borrowed from the context, while they are expected to have different ones.

Fix it by using HW physical port instead.

It fixes the following error during driver init:

[   12.074150] mlx4_core 0000:05:00.0: mlx4_ib: initializing demux service for 128 qp1 clients
[   12.084036] <mlx4_ib> create_pv_sqp: Couldn't create special QP (-16)
[   12.085123] <mlx4_ib> create_pv_resources: Couldn't create  QP1 (-16)
[   12.088300] mlx4_en: Mellanox ConnectX HCA Ethernet driver v4.0-0

---
Changelog:
v3: mlx4 devices create 2 special QPs in SRIOV mode, separate them by port number and special bit. The mlx4 is limited to two ports and not going to be extended, and the port_num is not forwarded to FW too, so it is safe.
v2: https://lore.kernel.org/linux-rdma/[email protected]/#r

Quoted 3 lines
---
 drivers/infiniband/hw/mlx4/mad.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

I didn't understand why this was in the restrack series, and the commit doesn't say when the error can be hit

No fixes line?

Jason

#2 lore raw
Quoted 26 lines · Jason Gunthorpe
Quoted 21 lines · Leon Romanovsky

From: Leon Romanovsky <[email protected]>

Special QPs created by mlx4 have same QP port borrowed from the context, while they are expected to have different ones.

Fix it by using HW physical port instead.

It fixes the following error during driver init:

[   12.074150] mlx4_core 0000:05:00.0: mlx4_ib: initializing demux service for 128 qp1 clients
[   12.084036] <mlx4_ib> create_pv_sqp: Couldn't create special QP (-16)
[   12.085123] <mlx4_ib> create_pv_resources: Couldn't create  QP1 (-16)
[   12.088300] mlx4_en: Mellanox ConnectX HCA Ethernet driver v4.0-0

---
Changelog:
v3: mlx4 devices create 2 special QPs in SRIOV mode, separate them by port number and special bit. The mlx4 is limited to two ports and not going to be extended, and the port_num is not forwarded to FW too, so it is safe.
v2: https://lore.kernel.org/linux-rdma/[email protected]/#r

Quoted 3 lines
---
 drivers/infiniband/hw/mlx4/mad.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)
Quoted 2 lines

I didn't understand why this was in the restrack series, and the commit doesn't say when the error can be hit

restrack changes revealed it when I added these special QPs to the DB. It is not fix in traditional sense, so no Fixes line.

Thanks

Quoted 4 lines

No fixes line?

Jason

#3 lore raw
Quoted 27 lines · Leon Romanovsky
Quoted 26 lines · Jason Gunthorpe
Quoted 24 lines · Leon Romanovsky

From: Leon Romanovsky <[email protected]>

Special QPs created by mlx4 have same QP port borrowed from the context, while they are expected to have different ones.

Fix it by using HW physical port instead.

It fixes the following error during driver init:

[   12.074150] mlx4_core 0000:05:00.0: mlx4_ib: initializing demux service for 128 qp1 clients
[   12.084036] <mlx4_ib> create_pv_sqp: Couldn't create special QP (-16)
[   12.085123] <mlx4_ib> create_pv_resources: Couldn't create  QP1 (-16)
[   12.088300] mlx4_en: Mellanox ConnectX HCA Ethernet driver v4.0-0

---
Changelog:
v3: mlx4 devices create 2 special QPs in SRIOV mode, separate them by port number and special bit. The mlx4 is limited to two ports and not going to be extended, and the port_num is not forwarded to FW too, so it is safe.
v2: https://lore.kernel.org/linux-rdma/[email protected]/#r

---
 drivers/infiniband/hw/mlx4/mad.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)
Quoted 5 lines
Quoted 2 lines

I didn't understand why this was in the restrack series, and the commit doesn't say when the error can be hit

restrack changes revealed it when I added these special QPs to the DB. It is not fix in traditional sense, so no Fixes line.

Anyway, please wait with this patch, it makes troubles in some mlx4 SRIOV tests.

Thanks

Quoted 7 lines

Thanks

Quoted 4 lines

No fixes line?

Jason

#4 lore raw
Quoted 28 lines · Leon Romanovsky
Quoted 27 lines · Leon Romanovsky
Quoted 26 lines · Jason Gunthorpe
Quoted 24 lines · Leon Romanovsky

From: Leon Romanovsky <[email protected]>

Special QPs created by mlx4 have same QP port borrowed from the context, while they are expected to have different ones.

Fix it by using HW physical port instead.

It fixes the following error during driver init:

[   12.074150] mlx4_core 0000:05:00.0: mlx4_ib: initializing demux service for 128 qp1 clients
[   12.084036] <mlx4_ib> create_pv_sqp: Couldn't create special QP (-16)
[   12.085123] <mlx4_ib> create_pv_resources: Couldn't create  QP1 (-16)
[   12.088300] mlx4_en: Mellanox ConnectX HCA Ethernet driver v4.0-0

---
Changelog:
v3: mlx4 devices create 2 special QPs in SRIOV mode, separate them by port number and special bit. The mlx4 is limited to two ports and not going to be extended, and the port_num is not forwarded to FW too, so it is safe.
v2: https://lore.kernel.org/linux-rdma/[email protected]/#r

---
 drivers/infiniband/hw/mlx4/mad.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

I didn't understand why this was in the restrack series, and the commit doesn't say when the error can be hit

restrack changes revealed it when I added these special QPs to the DB. It is not fix in traditional sense, so no Fixes line.

Anyway, please wait with this patch, it makes troubles in some mlx4 SRIOV tests.

The end result that this patch is going to be replaced by this fixup to "RDMA/core: Allow drivers to disable restrack DB" patch.

drivers/infiniband/hw/mlx4/qp.c +5 -0
index 7f0290112db7..e306d2aab510 100644
@@ -1562,6 +1562,11 @@ static int _mlx4_ib_create_qp(struct ib_pd *pd, struct mlx4_ib_qp *qp,
 		if (err)
 			return err;
 
+		if (init_attr->create_flags &
+		    (MLX4_IB_SRIOV_SQP | MLX4_IB_SRIOV_TUNNEL_QP))
+			/* Internal QP created with ib_create_qp */
+			rdma_restrack_no_track(&qp->ibqp.res);
+
 		qp->port	= init_attr->port_num;
 		qp->ibqp.qp_num = init_attr->qp_type == IB_QPT_SMI ? 0 :
 			init_attr->create_flags & MLX4_IB_QP_CREATE_ROCE_V2_GSI ? sqpn : 1;
Quoted 10 lines

Thanks

Quoted 7 lines

Thanks

Quoted 4 lines

No fixes line?

Jason