RDMA/mlx4: Provide port number for special QPs
Subsystems (2)
PATCH v3 1/1 RDMA/mlx4: Provide port number for special QPs
From: Leon Romanovsky <[email protected]>
Special QPs created by mlx4 have same QP port borrowed from the context, while they are expected to have different ones.
Fix it by using HW physical port instead.
It fixes the following error during driver init:
[ 12.074150] mlx4_core 0000:05:00.0: mlx4_ib: initializing demux service for 128 qp1 clients [ 12.084036] <mlx4_ib> create_pv_sqp: Couldn't create special QP (-16) [ 12.085123] <mlx4_ib> create_pv_resources: Couldn't create QP1 (-16) [ 12.088300] mlx4_en: Mellanox ConnectX HCA Ethernet driver v4.0-0
---
Changelog:
v3: mlx4 devices create 2 special QPs in SRIOV mode, separate them by port number and special bit. The mlx4 is limited to two ports and not going to be extended, and the port_num is not forwarded to FW too, so it is safe.
v2: https://lore.kernel.org/linux-rdma/[email protected]/#r
--- drivers/infiniband/hw/mlx4/mad.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
drivers/infiniband/hw/mlx4/mad.c +5 -4
@@ -1792,7 +1792,7 @@ static void pv_qp_event_handler(struct ib_event *event, void *qp_context) }static int create_pv_sqp (struct mlx4_ib_demux_pv_ctx *ctx, -enum ib_qp_type qp_type,int create_tun) +enum ib_qp_type qp_type,int port,int create_tun) {int i, ret;struct mlx4_ib_demux_pv_qp *tun_qp;
@@ -1822,12 +1822,13 @@ static int create_pv_sqp(struct mlx4_ib_demux_pv_ctx *ctx, qp_init_attr.proxy_qp_type = qp_type; qp_attr_mask_INIT =IB_QP_STATE |IB_QP_PKEY_INDEX |IB_QP_QKEY |IB_QP_PORT ; + qp_init_attr.init_attr.port_num = ctx->port; }else { qp_init_attr.init_attr.qp_type = qp_type; qp_init_attr.init_attr.create_flags =MLX4_IB_SRIOV_SQP ; qp_attr_mask_INIT =IB_QP_STATE |IB_QP_PKEY_INDEX |IB_QP_QKEY ; + qp_init_attr.init_attr.port_num = port |1 <<7 ; } - qp_init_attr.init_attr.port_num = ctx->port; qp_init_attr.init_attr.qp_context = ctx; qp_init_attr.init_attr.event_handler = pv_qp_event_handler; tun_qp->qp =ib_create_qp (ctx->pd, &qp_init_attr.init_attr);
@@ -2026,7 +2027,7 @@ static int create_pv_resources(struct ib_device *ibdev, int slave, int port, }if (ctx->has_smi) { - ret =create_pv_sqp (ctx,IB_QPT_SMI , create_tun); + ret =create_pv_sqp (ctx,IB_QPT_SMI , port, create_tun);if (ret) {pr_err ("Couldn't create %s QP0 (%d)\n" , create_tun ?"tunnel for" :"" , ret);
@@ -2034,7 +2035,7 @@ static int create_pv_resources(struct ib_device *ibdev, int slave, int port, } } - ret =create_pv_sqp (ctx,IB_QPT_GSI , create_tun); + ret =create_pv_sqp (ctx,IB_QPT_GSI , port, create_tun);if (ret) {pr_err ("Couldn't create %s QP1 (%d)\n" , create_tun ?"tunnel for" :"" , ret);
--
2.26.2
Quoted 26 lines · Jason Gunthorpe
Quoted 21 lines · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
Special QPs created by mlx4 have same QP port borrowed from the context, while they are expected to have different ones.
Fix it by using HW physical port instead.
It fixes the following error during driver init:
[ 12.074150] mlx4_core 0000:05:00.0: mlx4_ib: initializing demux service for 128 qp1 clients [ 12.084036] <mlx4_ib> create_pv_sqp: Couldn't create special QP (-16) [ 12.085123] <mlx4_ib> create_pv_resources: Couldn't create QP1 (-16) [ 12.088300] mlx4_en: Mellanox ConnectX HCA Ethernet driver v4.0-0Signed-off-by Leon Romanovsky <[email protected]>---
Changelog:
v3: mlx4 devices create 2 special QPs in SRIOV mode, separate them by port number and special bit. The mlx4 is limited to two ports and not going to be extended, and theport_numis not forwarded to FW too, so it is safe.
v2: https://lore.kernel.org/linux-rdma/[email protected]/#rQuoted 3 lines
--- drivers/infiniband/hw/mlx4/mad.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
Quoted 2 lines
I didn't understand why this was in the restrack series, and the commit doesn't say when the error can be hit
restrack changes revealed it when I added these special QPs to the DB. It is not fix in traditional sense, so no Fixes line.
Thanks
Quoted 4 lines
No fixes line?
Jason
Jason Gunthorpe
maintainerQuoted 21 lines · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
Special QPs created by mlx4 have same QP port borrowed from the context, while they are expected to have different ones.
Fix it by using HW physical port instead.
It fixes the following error during driver init:
[ 12.074150] mlx4_core 0000:05:00.0: mlx4_ib: initializing demux service for 128 qp1 clients [ 12.084036] <mlx4_ib> create_pv_sqp: Couldn't create special QP (-16) [ 12.085123] <mlx4_ib> create_pv_resources: Couldn't create QP1 (-16) [ 12.088300] mlx4_en: Mellanox ConnectX HCA Ethernet driver v4.0-0Signed-off-by Leon Romanovsky <[email protected]>---
Changelog:
v3: mlx4 devices create 2 special QPs in SRIOV mode, separate them by port number and special bit. The mlx4 is limited to two ports and not going to be extended, and theport_numis not forwarded to FW too, so it is safe.
v2: https://lore.kernel.org/linux-rdma/[email protected]/#r
Quoted 3 lines
--- drivers/infiniband/hw/mlx4/mad.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
I didn't understand why this was in the restrack series, and the commit doesn't say when the error can be hit
No fixes line?
Jason
Quoted 27 lines · Leon Romanovsky
Quoted 26 lines · Jason Gunthorpe
Quoted 24 lines · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
Special QPs created by mlx4 have same QP port borrowed from the context, while they are expected to have different ones.
Fix it by using HW physical port instead.
It fixes the following error during driver init:
[ 12.074150] mlx4_core 0000:05:00.0: mlx4_ib: initializing demux service for 128 qp1 clients [ 12.084036] <mlx4_ib> create_pv_sqp: Couldn't create special QP (-16) [ 12.085123] <mlx4_ib> create_pv_resources: Couldn't create QP1 (-16) [ 12.088300] mlx4_en: Mellanox ConnectX HCA Ethernet driver v4.0-0Signed-off-by Leon Romanovsky <[email protected]>---
Changelog:
v3: mlx4 devices create 2 special QPs in SRIOV mode, separate them by port number and special bit. The mlx4 is limited to two ports and not going to be extended, and theport_numis not forwarded to FW too, so it is safe.
v2: https://lore.kernel.org/linux-rdma/[email protected]/#r--- drivers/infiniband/hw/mlx4/mad.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
Quoted 5 lines
Quoted 2 lines
I didn't understand why this was in the restrack series, and the commit doesn't say when the error can be hit
restrack changes revealed it when I added these special QPs to the DB. It is not fix in traditional sense, so no Fixes line.
Anyway, please wait with this patch, it makes troubles in some mlx4 SRIOV tests.
Thanks
Quoted 7 lines
Thanks
Quoted 4 lines
No fixes line?
Jason
Quoted 28 lines · Leon Romanovsky
Quoted 27 lines · Leon Romanovsky
Quoted 26 lines · Jason Gunthorpe
Quoted 24 lines · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
Special QPs created by mlx4 have same QP port borrowed from the context, while they are expected to have different ones.
Fix it by using HW physical port instead.
It fixes the following error during driver init:
[ 12.074150] mlx4_core 0000:05:00.0: mlx4_ib: initializing demux service for 128 qp1 clients [ 12.084036] <mlx4_ib> create_pv_sqp: Couldn't create special QP (-16) [ 12.085123] <mlx4_ib> create_pv_resources: Couldn't create QP1 (-16) [ 12.088300] mlx4_en: Mellanox ConnectX HCA Ethernet driver v4.0-0Signed-off-by Leon Romanovsky <[email protected]>---
Changelog:
v3: mlx4 devices create 2 special QPs in SRIOV mode, separate them by port number and special bit. The mlx4 is limited to two ports and not going to be extended, and theport_numis not forwarded to FW too, so it is safe.
v2: https://lore.kernel.org/linux-rdma/[email protected]/#r--- drivers/infiniband/hw/mlx4/mad.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
I didn't understand why this was in the restrack series, and the commit doesn't say when the error can be hit
restrack changes revealed it when I added these special QPs to the DB. It is not fix in traditional sense, so no Fixes line.
Anyway, please wait with this patch, it makes troubles in some mlx4 SRIOV tests.
The end result that this patch is going to be replaced by this fixup to "RDMA/core: Allow drivers to disable restrack DB" patch.
drivers/infiniband/hw/mlx4/qp.c +5 -0
@@ -1562,6 +1562,11 @@ static int _mlx4_ib_create_qp(struct ib_pd *pd, struct mlx4_ib_qp *qp,if (err)return err; +if (init_attr->create_flags & + (MLX4_IB_SRIOV_SQP |MLX4_IB_SRIOV_TUNNEL_QP )) +/* Internal QP created with ib_create_qp */ +rdma_restrack_no_track (&qp->ibqp.res); + qp->port = init_attr->port_num; qp->ibqp.qp_num = init_attr->qp_type ==IB_QPT_SMI ?0 : init_attr->create_flags &MLX4_IB_QP_CREATE_ROCE_V2_GSI ? sqpn :1 ;
Quoted 10 lines
Thanks
Quoted 7 lines
Thanks
Quoted 4 lines
No fixes line?
Jason