Skip to contentSkip to conversation

RDMA/irdma: Make the source udp port vary

PATCH Superseded [email protected] 3 participants 5 messages Dec 18, 2021 – Dec 21, 2021

Subsystems (2)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 RDMA/irdma: Make the source udp port vary

From: Zhu Yanjun <[email protected]>

Based on the link https://www.spinics.net/lists/linux-rdma/msg73735.html,
get the source udp port number for a QP based on the grh.flow_label or lqpn/rqpn. This provides a better spread of traffic across NIC RX queues. The method in the commit 2b880b2e5e03 ("RDMA/mlx5: Define RoCEv2 udp source port when set path") is a standard way. So it is also adopted in this commit.

---
V1->V2: Adopt a standard method to get udp source port.

---
 drivers/infiniband/hw/irdma/verbs.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)
drivers/infiniband/hw/irdma/verbs.c +13 -0
index 8cd5f9261692..9fe73d1db0d9 100644
@@ -1094,6 +1094,15 @@ static int irdma_query_pkey(struct ib_device *ibdev, u32 port, u16 index,
 	return 0;
 }
 
+
+static u16 irdma_get_udp_sport(u32 fl, u32 lqpn, u32 rqpn)
+{
+	if (!fl)
+		fl = rdma_calc_flow_label(lqpn, rqpn);
+
+	return rdma_flow_label_to_udp_sport(fl);
+}
+
 /**
  * irdma_modify_qp_roce - modify qp request
  * @ibqp: qp's pointer for modify
@@ -1159,6 +1168,10 @@ int irdma_modify_qp_roce(struct ib_qp *ibqp, struct ib_qp_attr *attr,
 
 	ctx_info->roce_info->pd_id = iwpd->sc_pd.pd_id;
 
+	udp_info->src_port = irdma_get_udp_sport(udp_info->flow_label,
+						 ibqp->qp_num,
+						 roce_info->dest_qp);
+
 	if (attr_mask & IB_QP_AV) {
 		struct irdma_av *av = &iwqp->roce_ah.av;
 		const struct ib_gid_attr *sgid_attr;
Signature · 1 line
2.27.0
#1 lore raw
Quoted 40 lines · On 12/19/2021 4:44 AM, [email protected] wrote:

External email: Use caution opening links or attachments

From: Zhu Yanjun <[email protected]>

Based on the link https://www.spinics.net/lists/linux-rdma/msg73735.html,
get the source udp port number for a QP based on the grh.flow_label or lqpn/rqpn. This provides a better spread of traffic across NIC RX queues. The method in the commit 2b880b2e5e03 ("RDMA/mlx5: Define RoCEv2 udp source port when set path") is a standard way. So it is also adopted in this commit.

---
V1->V2: Adopt a standard method to get udp source port.

---
  drivers/infiniband/hw/irdma/verbs.c | 13 +++++++++++++
  1 file changed, 13 insertions(+)
drivers/infiniband/hw/irdma/verbs.c +9 -0
index 8cd5f9261692..9fe73d1db0d9 100644
@@ -1094,6 +1094,15 @@ static int irdma_query_pkey(struct ib_device *ibdev, u32 port, u16 index,
         return 0;
  }
 
+
+static u16 irdma_get_udp_sport(u32 fl, u32 lqpn, u32 rqpn)
+{
+       if (!fl)
+               fl = rdma_calc_flow_label(lqpn, rqpn);
+
+       return rdma_flow_label_to_udp_sport(fl);
+}
+
  /**
   * irdma_modify_qp_roce - modify qp request
   * @ibqp: qp's pointer for modify

drivers/infiniband/hw/irdma/verbs.c int irdma_modify_qp_roce(struct ib_qp *ibqp, struct ib_qp_attr *attr, · line 1174

Quoted 7 lines
drivers/infiniband/hw/irdma/verbs.c +4 -0
@@ -1159,6 +1168,10 @@ int irdma_modify_qp_roce(struct ib_qp *ibqp, struct ib_qp_attr *attr,
 
         ctx_info->roce_info->pd_id = iwpd->sc_pd.pd_id;
 
+       udp_info->src_port = irdma_get_udp_sport(udp_info->flow_label,
+                                                ibqp->qp_num,
+                                                roce_info->dest_qp);
+

FYI that in mlx5 driver the udp_sport is only set when address vector and dest qpn (IB_QP_AV and IB_QP_DEST_QPN) is provided.

Quoted 6 lines
drivers/infiniband/hw/irdma/verbs.c +0 -1
         if (attr_mask & IB_QP_AV) {
                 struct irdma_av *av = &iwqp->roce_ah.av;
                 const struct ib_gid_attr *sgid_attr;
--

2.27.0

author
#2 lore raw
Quoted 46 lines · Mark Zhang
Quoted 45 lines · On 12/19/2021 4:44 AM, [email protected] wrote:

External email: Use caution opening links or attachments

From: Zhu Yanjun <[email protected]>

Based on the link https://www.spinics.net/lists/linux-rdma/msg73735.html,
get the source udp port number for a QP based on the grh.flow_label or lqpn/rqpn. This provides a better spread of traffic across NIC RX queues. The method in the commit 2b880b2e5e03 ("RDMA/mlx5: Define RoCEv2 udp source port when set path") is a standard way. So it is also adopted in this commit.

---
V1->V2: Adopt a standard method to get udp source port.
---
  drivers/infiniband/hw/irdma/verbs.c | 13 +++++++++++++

  1 file changed, 13 insertions(+)
drivers/infiniband/hw/irdma/verbs.c +9 -0
index 8cd5f9261692..9fe73d1db0d9 100644
@@ -1094,6 +1094,15 @@ static int irdma_query_pkey(struct ib_device 
 *ibdev, u32 port, u16 index,
          return 0;
   }
 
+
+static u16 irdma_get_udp_sport(u32 fl, u32 lqpn, u32 rqpn)
+{
+       if (!fl)
+               fl = rdma_calc_flow_label(lqpn, rqpn);
+
+       return rdma_flow_label_to_udp_sport(fl);
+}
+
   /**
    * irdma_modify_qp_roce - modify qp request

   * @ibqp: qp's pointer for modify

drivers/infiniband/hw/irdma/verbs.c +0 -0
@@ -1159,6 +1168,10 @@ int irdma_modify_qp_roce(struct ib_qp *ibqp, 
 struct ib_qp_attr *attr,
 
          ctx_info->roce_info->pd_id = iwpd->sc_pd.pd_id;
 
Quoted 7 lines

drivers/infiniband/hw/irdma/verbs.c int irdma_modify_qp_roce(struct ib_qp *ibqp, struct ib_qp_attr *attr, · line 1174

Quoted 4 lines
drivers/infiniband/hw/irdma/verbs.c +4 -0
+       udp_info->src_port = irdma_get_udp_sport(udp_info->flow_label,
+ ibqp->qp_num,
+ roce_info->dest_qp);
+

FYI that in mlx5 driver the udp_sport is only set when address vector and dest qpn (IB_QP_AV and IB_QP_DEST_QPN) is provided.

Sure. From my tests, when this function irdma_get_udp_sport is called,  dest qpn is ready.

Zhu Yanjun

Quoted 8 lines
Quoted 7 lines
drivers/infiniband/hw/irdma/verbs.c +0 -0
          if (attr_mask & IB_QP_AV) {
                  struct irdma_av *av = &iwqp->roce_ah.av;

                 const struct ib_gid_attr *sgid_attr;

Signature · 1 line
2.27.0
#3 lore raw
Quoted 60 lines

Subject: Re: [PATCHv2 1/1] RDMA/irdma: Make the source udp port vary

Quoted 47 lines · Mark Zhang
Quoted 46 lines · On 12/19/2021 4:44 AM, [email protected] wrote:

External email: Use caution opening links or attachments

From: Zhu Yanjun <[email protected]>

Based on the link
https://www.spinics.net/lists/linux-rdma/msg73735.html,
get the source udp port number for a QP based on the grh.flow_label or lqpn/rqpn. This provides a better spread of traffic across NIC RX queues.
The method in the commit 2b880b2e5e03 ("RDMA/mlx5: Define RoCEv2 udp source port when set path") is a standard way. So it is also adopted in this commit.

---
V1->V2: Adopt a standard method to get udp source port.
---
  drivers/infiniband/hw/irdma/verbs.c | 13 +++++++++++++

  1 file changed, 13 insertions(+)
drivers/infiniband/hw/irdma/verbs.c +7 -0
index 8cd5f9261692..9fe73d1db0d9 100644
@@ -1094,6 +1094,15 @@ static int irdma_query_pkey(struct ib_device
 *ibdev, u32 port, u16 index,
          return 0;
   }
 
+
+static u16 irdma_get_udp_sport(u32 fl, u32 lqpn, u32 rqpn) {
+       if (!fl)
+               fl = rdma_calc_flow_label(lqpn, rqpn);
+
+       return rdma_flow_label_to_udp_sport(fl); }
+
   /**
    * irdma_modify_qp_roce - modify qp request

   * @ibqp: qp's pointer for modify

drivers/infiniband/hw/irdma/verbs.c +0 -0
@@ -1159,6 +1168,10 @@ int irdma_modify_qp_roce(struct ib_qp *ibqp,
 struct ib_qp_attr *attr,
 
          ctx_info->roce_info->pd_id = iwpd->sc_pd.pd_id;
 
Quoted 8 lines

drivers/infiniband/hw/irdma/verbs.c int irdma_modify_qp_roce(struct ib_qp *ibqp, struct ib_qp_attr *attr, · line 1174

Quoted 5 lines
drivers/infiniband/hw/irdma/verbs.c +5 -0
+       udp_info->src_port =
+irdma_get_udp_sport(udp_info->flow_label,
+ ibqp->qp_num,
+ roce_info->dest_qp);
+

FYI that in mlx5 driver the udp_sport is only set when address vector and dest qpn (IB_QP_AV and IB_QP_DEST_QPN) is provided.

Quoted 3 lines

Sure. From my tests, when this function irdma_get_udp_sport is called, dest qpn is ready.

I think this needs to move to when we have IB_QP_AV in the mask and IB_AH_GRH in ah_flags.

So, in this block,
https://elixir.bootlin.com/linux/v5.16-rc6/source/drivers/infiniband/hw/irdma/verbs.c#L1171

Shiraz

author
#4 lore raw
Quoted 60 lines · Shiraz
Quoted 60 lines

Subject: Re: [PATCHv2 1/1] RDMA/irdma: Make the source udp port vary

Quoted 55 lines · Mark Zhang
Quoted 42 lines · On 12/19/2021 4:44 AM, [email protected] wrote:

External email: Use caution opening links or attachments

From: Zhu Yanjun <[email protected]>

Based on the link
https://www.spinics.net/lists/linux-rdma/msg73735.html,
get the source udp port number for a QP based on the grh.flow_label or lqpn/rqpn. This provides a better spread of traffic across NIC RX queues.
The method in the commit 2b880b2e5e03 ("RDMA/mlx5: Define RoCEv2 udp source port when set path") is a standard way. So it is also adopted in this commit.

---
V1->V2: Adopt a standard method to get udp source port.

---
   drivers/infiniband/hw/irdma/verbs.c | 13 +++++++++++++
   1 file changed, 13 insertions(+)
drivers/infiniband/hw/irdma/verbs.c +7 -0
index 8cd5f9261692..9fe73d1db0d9 100644
@@ -1094,6 +1094,15 @@ static int irdma_query_pkey(struct ib_device
 *ibdev, u32 port, u16 index,
          return 0;
   }
 
+
+static u16 irdma_get_udp_sport(u32 fl, u32 lqpn, u32 rqpn) {
+       if (!fl)
+               fl = rdma_calc_flow_label(lqpn, rqpn);
+
+       return rdma_flow_label_to_udp_sport(fl); }
+
   /**
    * irdma_modify_qp_roce - modify qp request
    * @ibqp: qp's pointer for modify

drivers/infiniband/hw/irdma/verbs.c int irdma_modify_qp_roce(struct ib_qp *ibqp, · line 1176

Quoted 9 lines
drivers/infiniband/hw/irdma/verbs.c +5 -0
@@ -1159,6 +1168,10 @@ int irdma_modify_qp_roce(struct ib_qp *ibqp,
 struct ib_qp_attr *attr,
 
          ctx_info->roce_info->pd_id = iwpd->sc_pd.pd_id;
 
+       udp_info->src_port =
+irdma_get_udp_sport(udp_info->flow_label,
+ ibqp->qp_num,
+ roce_info->dest_qp);
+

FYI that in mlx5 driver the udp_sport is only set when address vector and dest qpn (IB_QP_AV and IB_QP_DEST_QPN) is provided.

Quoted 8 lines
Quoted 3 lines

Sure. From my tests, when this function irdma_get_udp_sport is called, dest qpn is ready.

I think this needs to move to when we have IB_QP_AV in the mask and IB_AH_GRH in ah_flags.

So, in this block,
https://elixir.bootlin.com/linux/v5.16-rc6/source/drivers/infiniband/hw/irdma/verbs.c#L1171

Thanks. The latest patch has been sent following your suggestions.

Zhu Yanjun

Quoted 2 lines

Shiraz