RDMA/irdma: Make the source udp port vary
Subsystems (2)
PATCH v2 1/1 RDMA/irdma: Make the source udp port vary
From: Zhu Yanjun <[email protected]>
Based on the link https://www.spinics.net/lists/linux-rdma/msg73735.html,
get the source udp port number for a QP based on the grh.flow_label or lqpn/rqpn. This provides a better spread of traffic across NIC RX queues. The method in the commit 2b880b2e5e03 ("RDMA/mlx5: Define RoCEv2 udp source port when set path") is a standard way. So it is also adopted in this commit.
---V1->V2: Adopt a standard method to get udp source port.
--- drivers/infiniband/hw/irdma/verbs.c | 13 +++++++++++++ 1 file changed, 13 insertions(+)
drivers/infiniband/hw/irdma/verbs.c +13 -0
@@ -1094,6 +1094,15 @@ static int irdma_query_pkey(struct ib_device *ibdev, u32 port, u16 index,return 0 ; } + +static u16 irdma_get_udp_sport (u32 fl,u32 lqpn,u32 rqpn) +{ +if (!fl) + fl =rdma_calc_flow_label (lqpn, rqpn); + +return rdma_flow_label_to_udp_sport (fl); +} +/** * irdma_modify_qp_roce - modify qp request * @ibqp: qp's pointer for modify
@@ -1159,6 +1168,10 @@ int irdma_modify_qp_roce(struct ib_qp *ibqp, struct ib_qp_attr *attr, ctx_info->roce_info->pd_id = iwpd->sc_pd.pd_id; + udp_info->src_port =irdma_get_udp_sport (udp_info->flow_label, + ibqp->qp_num, + roce_info->dest_qp); +if (attr_mask &IB_QP_AV ) {struct irdma_av *av = &iwqp->roce_ah.av;const struct ib_gid_attr *sgid_attr;
Signature · 1 line
2.27.0
Quoted 40 lines · On 12/19/2021 4:44 AM, [email protected] wrote:
External email: Use caution opening links or attachments
From: Zhu Yanjun <[email protected]>
Based on the link https://www.spinics.net/lists/linux-rdma/msg73735.html,
get the source udp port number for a QP based on the grh.flow_label or lqpn/rqpn. This provides a better spread of traffic across NIC RX queues. The method in the commit2b880b2e5e03("RDMA/mlx5: Define RoCEv2 udp source port when set path") is a standard way. So it is also adopted in this commit.Signed-off-by Zhu Yanjun <[email protected]>---
V1->V2: Adopt a standard method to get udp source port.--- drivers/infiniband/hw/irdma/verbs.c | 13 +++++++++++++ 1 file changed, 13 insertions(+)drivers/infiniband/hw/irdma/verbs.c +9
-0@@ -1094,6 +1094,15 @@ static int irdma_query_pkey(struct ib_device *ibdev, u32 port, u16 index,return 0 ; } + +static u16 irdma_get_udp_sport (u32 fl,u32 lqpn,u32 rqpn) +{ +if (!fl) + fl =rdma_calc_flow_label (lqpn, rqpn); + +return rdma_flow_label_to_udp_sport (fl); +} +/** * irdma_modify_qp_roce - modify qp request * @ibqp: qp's pointer for modify
drivers/infiniband/hw/irdma/verbs.c int irdma_modify_qp_roce(struct ib_qp *ibqp, struct ib_qp_attr *attr, · line 1174
Quoted 7 lines
drivers/infiniband/hw/irdma/verbs.c +4
-0@@ -1159,6 +1168,10 @@ int irdma_modify_qp_roce(struct ib_qp *ibqp, struct ib_qp_attr *attr, ctx_info->roce_info->pd_id = iwpd->sc_pd.pd_id; + udp_info->src_port =irdma_get_udp_sport (udp_info->flow_label, + ibqp->qp_num, + roce_info->dest_qp); +
FYI that in mlx5 driver the udp_sport is only set when address vector and dest qpn (IB_QP_AV and IB_QP_DEST_QPN) is provided.
Quoted 6 lines
drivers/infiniband/hw/irdma/verbs.c +0
-1if (attr_mask &IB_QP_AV ) {struct irdma_av *av = &iwqp->roce_ah.av;const struct ib_gid_attr *sgid_attr; --2.27.0
Quoted 46 lines · Mark Zhang
Quoted 45 lines · On 12/19/2021 4:44 AM, [email protected] wrote:
External email: Use caution opening links or attachments
From: Zhu Yanjun <[email protected]>
Based on the link https://www.spinics.net/lists/linux-rdma/msg73735.html,
get the source udp port number for a QP based on the grh.flow_label or lqpn/rqpn. This provides a better spread of traffic across NIC RX queues. The method in the commit2b880b2e5e03("RDMA/mlx5: Define RoCEv2 udp source port when set path") is a standard way. So it is also adopted in this commit.Signed-off-by Zhu Yanjun <[email protected]>---
V1->V2: Adopt a standard method to get udp source port.
---
drivers/infiniband/hw/irdma/verbs.c| 13 +++++++++++++1 file changed, 13 insertions(+)drivers/infiniband/hw/irdma/verbs.c +9
-0@@ -1094,6 +1094,15 @@ static int irdma_query_pkey(struct ib_device *ibdev, u32 port, u16 index, return 0; } + +static u16 irdma_get_udp_sport (u32 fl,u32 lqpn,u32 rqpn) +{ + if (!fl) + fl = rdma_calc_flow_label(lqpn, rqpn); + + return rdma_flow_label_to_udp_sport(fl); +} + /*** irdma_modify_qp_roce - modify qp request * @ibqp: qp's pointer for modify
drivers/infiniband/hw/irdma/verbs.c +0
-0@@ -1159,6 +1168,10 @@ int irdma_modify_qp_roce(struct ib_qp *ibqp,struct ib_qp_attr *attr, ctx_info->roce_info->pd_id = iwpd->sc_pd.pd_id;
Quoted 7 lines
drivers/infiniband/hw/irdma/verbs.c int irdma_modify_qp_roce(struct ib_qp *ibqp, struct ib_qp_attr *attr, · line 1174
Quoted 4 lines
drivers/infiniband/hw/irdma/verbs.c +4
-0+ udp_info->src_port = irdma_get_udp_sport(udp_info->flow_label, + ibqp->qp_num, + roce_info->dest_qp); +FYI that in mlx5 driver the
udp_sportis only set when address vector and dest qpn (IB_QP_AVandIB_QP_DEST_QPN) is provided.
Sure. From my tests, when this function irdma_get_udp_sport is called, dest qpn is ready.
Zhu Yanjun
Quoted 8 lines
Quoted 7 lines
drivers/infiniband/hw/irdma/verbs.c +0
-0if (attr_mask & IB_QP_AV) { struct irdma_av *av = &iwqp->roce_ah.av;const struct
ib_gid_attr*sgid_attr;Signature · 1 line
2.27.0
Quoted 60 lines
Subject: Re: [PATCHv2 1/1] RDMA/irdma: Make the source udp port vary
Quoted 47 lines · Mark Zhang
Quoted 46 lines · On 12/19/2021 4:44 AM, [email protected] wrote:
External email: Use caution opening links or attachments
From: Zhu Yanjun <[email protected]>
Based on the link
https://www.spinics.net/lists/linux-rdma/msg73735.html,
get the source udp port number for a QP based on the grh.flow_label or lqpn/rqpn. This provides a better spread of traffic across NIC RX queues.
The method in the commit2b880b2e5e03("RDMA/mlx5: Define RoCEv2 udp source port when set path") is a standard way. So it is also adopted in this commit.Signed-off-by Zhu Yanjun <[email protected]>---
V1->V2: Adopt a standard method to get udp source port.
---
drivers/infiniband/hw/irdma/verbs.c| 13 +++++++++++++1 file changed, 13 insertions(+)drivers/infiniband/hw/irdma/verbs.c +7
-0@@ -1094,6 +1094,15 @@ static int irdma_query_pkey(struct ib_device *ibdev, u32 port, u16 index, return 0; } + +static u16 irdma_get_udp_sport (u32 fl,u32 lqpn,u32 rqpn) { + if (!fl) + fl = rdma_calc_flow_label(lqpn, rqpn); + + return rdma_flow_label_to_udp_sport(fl); } + /*** irdma_modify_qp_roce - modify qp request * @ibqp: qp's pointer for modify
drivers/infiniband/hw/irdma/verbs.c +0
-0@@ -1159,6 +1168,10 @@ int irdma_modify_qp_roce(struct ib_qp *ibqp,struct ib_qp_attr *attr, ctx_info->roce_info->pd_id = iwpd->sc_pd.pd_id;Quoted 8 lines
drivers/infiniband/hw/irdma/verbs.c int irdma_modify_qp_roce(struct ib_qp *ibqp, struct ib_qp_attr *attr, · line 1174
Quoted 5 lines
drivers/infiniband/hw/irdma/verbs.c +5
-0+ udp_info->src_port = +irdma_get_udp_sport (udp_info->flow_label, + ibqp->qp_num, + roce_info->dest_qp); +FYI that in mlx5 driver the
udp_sportis only set when address vector and dest qpn (IB_QP_AVandIB_QP_DEST_QPN) is provided.
Quoted 3 lines
Sure. From my tests, when this function
irdma_get_udp_sportis called, dest qpn is ready.
I think this needs to move to when we have IB_QP_AV in the mask and IB_AH_GRH in ah_flags.
So, in this block,
https://elixir.bootlin.com/linux/v5.16-rc6/source/drivers/infiniband/hw/irdma/verbs.c#L1171
Shiraz
Quoted 60 lines · Shiraz
Quoted 60 lines
Subject: Re: [PATCHv2 1/1] RDMA/irdma: Make the source udp port vary
Quoted 55 lines · Mark Zhang
Quoted 42 lines · On 12/19/2021 4:44 AM, [email protected] wrote:
External email: Use caution opening links or attachments
From: Zhu Yanjun <[email protected]>
Based on the link
https://www.spinics.net/lists/linux-rdma/msg73735.html,
get the source udp port number for a QP based on the grh.flow_label or lqpn/rqpn. This provides a better spread of traffic across NIC RX queues.
The method in the commit2b880b2e5e03("RDMA/mlx5: Define RoCEv2 udp source port when set path") is a standard way. So it is also adopted in this commit.Signed-off-by Zhu Yanjun <[email protected]>---
V1->V2: Adopt a standard method to get udp source port.--- drivers/infiniband/hw/irdma/verbs.c | 13 +++++++++++++ 1 file changed, 13 insertions(+)drivers/infiniband/hw/irdma/verbs.c +7
-0@@ -1094,6 +1094,15 @@ static int irdma_query_pkey(struct ib_device *ibdev, u32 port, u16 index, return 0; } + +static u16 irdma_get_udp_sport (u32 fl,u32 lqpn,u32 rqpn) { + if (!fl) + fl = rdma_calc_flow_label(lqpn, rqpn); + + return rdma_flow_label_to_udp_sport(fl); } + /*** irdma_modify_qp_roce - modify qp request * @ibqp: qp's pointer for modify drivers/infiniband/hw/irdma/verbs.c int irdma_modify_qp_roce(struct ib_qp *ibqp, · line 1176
Quoted 9 lines
drivers/infiniband/hw/irdma/verbs.c +5
-0@@ -1159,6 +1168,10 @@ int irdma_modify_qp_roce(struct ib_qp *ibqp,struct ib_qp_attr *attr, ctx_info->roce_info->pd_id = iwpd->sc_pd.pd_id; + udp_info->src_port = +irdma_get_udp_sport (udp_info->flow_label, + ibqp->qp_num, + roce_info->dest_qp); +FYI that in mlx5 driver the
udp_sportis only set when address vector and dest qpn (IB_QP_AVandIB_QP_DEST_QPN) is provided.
Quoted 8 lines
Quoted 3 lines
Sure. From my tests, when this function
irdma_get_udp_sportis called, dest qpn is ready.I think this needs to move to when we have
IB_QP_AVin the mask andIB_AH_GRHinah_flags.So, in this block,
https://elixir.bootlin.com/linux/v5.16-rc6/source/drivers/infiniband/hw/irdma/verbs.c#L1171
Thanks. The latest patch has been sent following your suggestions.
Zhu Yanjun
Quoted 2 lines
Shiraz