Skip to contentSkip to conversation

RDMA/irdma: Make the source udp port vary

PATCH In review [email protected] 4 participants 7 messages Dec 21, 2021 – Jan 4, 2022

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH v3 1/1 RDMA/irdma: Make the source udp port vary

From: Zhu Yanjun <[email protected]>

Based on the link https://www.spinics.net/lists/linux-rdma/msg73735.html,
get the source udp port number for a QP based on the grh.flow_label or lqpn/rqrpn. This provides a better spread of traffic across NIC RX queues. The method in the commit 2b880b2e5e03 ("RDMA/mlx5: Define RoCEv2 udp source port when set path") is a standard way. So it is also adopted in this commit.

---
V2->V3: Move to the block of IB_QP_AV in the mask and IB_AH_GRH in ah_flags V1->V2: Adopt a standard method to get udp source port.

---
 drivers/infiniband/hw/irdma/verbs.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
drivers/infiniband/hw/irdma/verbs.c +14 -0
index 8cd5f9261692..31039b295206 100644
@@ -1094,6 +1094,15 @@ static int irdma_query_pkey(struct ib_device *ibdev, u32 port, u16 index,
 	return 0;
 }
 
+
+static u16 irdma_get_udp_sport(u32 fl, u32 lqpn, u32 rqpn)
+{
+	if (!fl)
+		fl = rdma_calc_flow_label(lqpn, rqpn);
+
+	return rdma_flow_label_to_udp_sport(fl);
+}
+
 /**
  * irdma_modify_qp_roce - modify qp request
  * @ibqp: qp's pointer for modify
@@ -1167,6 +1176,11 @@ int irdma_modify_qp_roce(struct ib_qp *ibqp, struct ib_qp_attr *attr,
 
 		memset(&iwqp->roce_ah, 0, sizeof(iwqp->roce_ah));
 		if (attr->ah_attr.ah_flags & IB_AH_GRH) {
+			u32 fl = udp_info->flow_label;
+			u32 lqp = ibqp->qp_num;
+			u32 rqp = roce_info->dest_qp;
+
+			udp_info->src_port = irdma_get_udp_sport(fl, lqp, rqp);
 			udp_info->ttl = attr->ah_attr.grh.hop_limit;
 			udp_info->flow_label = attr->ah_attr.grh.flow_label;
 			udp_info->tos = attr->ah_attr.grh.traffic_class;
Signature · 1 line
2.27.0
Reviewed-by on v3 1/1
#1 lore raw
Quoted 11 lines

Subject: [PATCHv3 1/1] RDMA/irdma: Make the source udp port vary

From: Zhu Yanjun <[email protected]>

Based on the link https://www.spinics.net/lists/linux-rdma/msg73735.html,
get the source udp port number for a QP based on the grh.flow_label or lqpn/rqrpn. This provides a better spread of traffic across NIC RX queues. The method in the commit 2b880b2e5e03 ("RDMA/mlx5: Define RoCEv2 udp source port when set path") is a standard way. So it is also adopted in this commit.

Quoted 9 lines

---
V2->V3: Move to the block of IB_QP_AV in the mask and IB_AH_GRH in V2->ah_flags
V1->V2: Adopt a standard method to get udp source port.

---
 drivers/infiniband/hw/irdma/verbs.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
#2 lore raw
In reply to [email protected]
Quoted 3 lines · On Tue, Dec 21, 2021 at 12:39:13PM -0500, [email protected] wrote:

From: Zhu Yanjun <[email protected]>

Based on the link https://www.spinics.net/lists/linux-rdma/msg73735.html,

Please use lore.kernel.org links. They have all chances to outlive spinics.

Quoted 35 lines

get the source udp port number for a QP based on the grh.flow_label or lqpn/rqrpn. This provides a better spread of traffic across NIC RX queues. The method in the commit 2b880b2e5e03 ("RDMA/mlx5: Define RoCEv2 udp source port when set path") is a standard way. So it is also adopted in this commit.

---
V2->V3: Move to the block of IB_QP_AV in the mask and IB_AH_GRH in ah_flags V1->V2: Adopt a standard method to get udp source port.

---
 drivers/infiniband/hw/irdma/verbs.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
drivers/infiniband/hw/irdma/verbs.c +9 -0
index 8cd5f9261692..31039b295206 100644
@@ -1094,6 +1094,15 @@ static int irdma_query_pkey(struct ib_device *ibdev, u32 port, u16 index,
 	return 0;
 }
 
+
+static u16 irdma_get_udp_sport(u32 fl, u32 lqpn, u32 rqpn)
+{
+	if (!fl)
+		fl = rdma_calc_flow_label(lqpn, rqpn);
+
+	return rdma_flow_label_to_udp_sport(fl);
+}
+
 /**
  * irdma_modify_qp_roce - modify qp request
  * @ibqp: qp's pointer for modify

drivers/infiniband/hw/irdma/verbs.c int irdma_modify_qp_roce(struct ib_qp *ibqp, struct ib_qp_attr *attr, · line 1182

Quoted 7 lines
drivers/infiniband/hw/irdma/verbs.c +4 -0
@@ -1167,6 +1176,11 @@ int irdma_modify_qp_roce(struct ib_qp *ibqp, struct ib_qp_attr *attr,
 
 		memset(&iwqp->roce_ah, 0, sizeof(iwqp->roce_ah));
 		if (attr->ah_attr.ah_flags & IB_AH_GRH) {
+			u32 fl = udp_info->flow_label;
+			u32 lqp = ibqp->qp_num;
+			u32 rqp = roce_info->dest_qp;
+

I don't see too much value in these extra variables and extra function that is the same as get_udp_sport() from hns.

It is worth to add new function to ib_verbs.h and reuse in both drivers.

Thanks

Quoted 7 lines
drivers/infiniband/hw/irdma/verbs.c +1 -0
+			udp_info->src_port = irdma_get_udp_sport(fl, lqp, rqp);
 			udp_info->ttl = attr->ah_attr.grh.hop_limit;
 			udp_info->flow_label = attr->ah_attr.grh.flow_label;
 			udp_info->tos = attr->ah_attr.grh.traffic_class;
Signature · 1 line
2.27.0
#3 lore raw
Quoted 54 lines · Leon Romanovsky
Quoted 3 lines · On Tue, Dec 21, 2021 at 12:39:13PM -0500, [email protected] wrote:

From: Zhu Yanjun <[email protected]>

Based on the link https://www.spinics.net/lists/linux-rdma/msg73735.html,

Please use lore.kernel.org links. They have all chances to outlive spinics.

Quoted 35 lines

get the source udp port number for a QP based on the grh.flow_label or lqpn/rqrpn. This provides a better spread of traffic across NIC RX queues. The method in the commit 2b880b2e5e03 ("RDMA/mlx5: Define RoCEv2 udp source port when set path") is a standard way. So it is also adopted in this commit.

---
V2->V3: Move to the block of IB_QP_AV in the mask and IB_AH_GRH in ah_flags V1->V2: Adopt a standard method to get udp source port.
---
drivers/infiniband/hw/irdma/verbs.c | 14 ++++++++++++++

1 file changed, 14 insertions(+)
drivers/infiniband/hw/irdma/verbs.c +9 -0
index 8cd5f9261692..31039b295206 100644
@@ -1094,6 +1094,15 @@ static int irdma_query_pkey(struct ib_device *ibdev, u32 port, u16 index,
 return 0;
 }
 
+
+static u16 irdma_get_udp_sport(u32 fl, u32 lqpn, u32 rqpn)
+{
+ if (!fl)
+ fl = rdma_calc_flow_label(lqpn, rqpn);
+
+ return rdma_flow_label_to_udp_sport(fl);
+}
+
 /**
 * irdma_modify_qp_roce - modify qp request
 * @ibqp: qp's pointer for modify

drivers/infiniband/hw/irdma/verbs.c int irdma_modify_qp_roce(struct ib_qp *ibqp, struct ib_qp_attr *attr, · line 1182

Quoted 7 lines
drivers/infiniband/hw/irdma/verbs.c +4 -0
@@ -1167,6 +1176,11 @@ int irdma_modify_qp_roce(struct ib_qp *ibqp, struct ib_qp_attr *attr,
 
 memset(&iwqp->roce_ah, 0, sizeof(iwqp->roce_ah));
 if (attr->ah_attr.ah_flags & IB_AH_GRH) {
+ u32 fl = udp_info->flow_label;
+ u32 lqp = ibqp->qp_num;
+ u32 rqp = roce_info->dest_qp;
+

I don't see too much value in these extra variables and extra function that is the same as get_udp_sport() from hns.

Quoted 1 line

It is worth to add new function to ib_verbs.h and reuse in both drivers.

Do you mean the following function should be added into ib_verbs.h?

"
static inline u16 rdma_get_udp_sport(u32 fl, u32 lqpn, u32 rqpn)

{
        if (!fl)
                fl = rdma_calc_flow_label(lqpn, rqpn);

        return rdma_flow_label_to_udp_sport(fl);
}

"
Then in hns, rxe and irdma, this function is called to get udp source port? If so, I will send new patches.

Thanks.
Zhu Yanjun

Quoted 9 lines

Thanks

Quoted 6 lines
drivers/infiniband/hw/irdma/verbs.c +1 -1
+ udp_info->src_port = irdma_get_udp_sport(fl, lqp, rqp);
 udp_info->ttl = attr->ah_attr.grh.hop_limit;
 udp_info->flow_label = attr->ah_attr.grh.flow_label;
 udp_info->tos = attr->ah_attr.grh.traffic_class;
--

2.27.0

#4 lore raw
Quoted 68 lines · On Tue, Jan 04, 2022 at 07:51:54AM +0000, [email protected] wrote:
Quoted 54 lines · Leon Romanovsky
Quoted 3 lines · On Tue, Dec 21, 2021 at 12:39:13PM -0500, [email protected] wrote:

From: Zhu Yanjun <[email protected]>

Based on the link https://www.spinics.net/lists/linux-rdma/msg73735.html,

Please use lore.kernel.org links. They have all chances to outlive spinics.

Quoted 35 lines

get the source udp port number for a QP based on the grh.flow_label or lqpn/rqrpn. This provides a better spread of traffic across NIC RX queues. The method in the commit 2b880b2e5e03 ("RDMA/mlx5: Define RoCEv2 udp source port when set path") is a standard way. So it is also adopted in this commit.

---
V2->V3: Move to the block of IB_QP_AV in the mask and IB_AH_GRH in ah_flags V1->V2: Adopt a standard method to get udp source port.
---
drivers/infiniband/hw/irdma/verbs.c | 14 ++++++++++++++

1 file changed, 14 insertions(+)
drivers/infiniband/hw/irdma/verbs.c +9 -0
index 8cd5f9261692..31039b295206 100644
@@ -1094,6 +1094,15 @@ static int irdma_query_pkey(struct ib_device *ibdev, u32 port, u16 index,
 return 0;
 }
 
+
+static u16 irdma_get_udp_sport(u32 fl, u32 lqpn, u32 rqpn)
+{
+ if (!fl)
+ fl = rdma_calc_flow_label(lqpn, rqpn);
+
+ return rdma_flow_label_to_udp_sport(fl);
+}
+
 /**
 * irdma_modify_qp_roce - modify qp request
 * @ibqp: qp's pointer for modify

drivers/infiniband/hw/irdma/verbs.c int irdma_modify_qp_roce(struct ib_qp *ibqp, struct ib_qp_attr *attr, · line 1182

Quoted 7 lines
drivers/infiniband/hw/irdma/verbs.c +4 -0
@@ -1167,6 +1176,11 @@ int irdma_modify_qp_roce(struct ib_qp *ibqp, struct ib_qp_attr *attr,
 
 memset(&iwqp->roce_ah, 0, sizeof(iwqp->roce_ah));
 if (attr->ah_attr.ah_flags & IB_AH_GRH) {
+ u32 fl = udp_info->flow_label;
+ u32 lqp = ibqp->qp_num;
+ u32 rqp = roce_info->dest_qp;
+

I don't see too much value in these extra variables and extra function that is the same as get_udp_sport() from hns.

Quoted 1 line

It is worth to add new function to ib_verbs.h and reuse in both drivers.

Do you mean the following function should be added into ib_verbs.h?

"
static inline u16 rdma_get_udp_sport(u32 fl, u32 lqpn, u32 rqpn)

{
        if (!fl)
                fl = rdma_calc_flow_label(lqpn, rqpn);

        return rdma_flow_label_to_udp_sport(fl);
}
Quoted 3 lines

"
Then in hns, rxe and irdma, this function is called to get udp source port? If so, I will send new patches.

Yes and group all these patches in one series, please.

Thanks

Quoted 13 lines

Thanks.
Zhu Yanjun

Quoted 9 lines

Thanks

Quoted 6 lines
drivers/infiniband/hw/irdma/verbs.c +1 -1
+ udp_info->src_port = irdma_get_udp_sport(fl, lqp, rqp);
 udp_info->ttl = attr->ah_attr.grh.hop_limit;
 udp_info->flow_label = attr->ah_attr.grh.flow_label;
 udp_info->tos = attr->ah_attr.grh.traffic_class;
--

2.27.0

#5 lore raw
In reply to [email protected]
Quoted 41 lines

Subject: [PATCHv3 1/1] RDMA/irdma: Make the source udp port vary

From: Zhu Yanjun <[email protected]>

Based on the link https://www.spinics.net/lists/linux-rdma/msg73735.html,
get the source udp port number for a QP based on the grh.flow_label or lqpn/rqrpn. This provides a better spread of traffic across NIC RX queues. The method in the commit 2b880b2e5e03 ("RDMA/mlx5: Define RoCEv2 udp source port when set path") is a standard way. So it is also adopted in this commit.

---
V2->V3: Move to the block of IB_QP_AV in the mask and IB_AH_GRH in V2->ah_flags
V1->V2: Adopt a standard method to get udp source port.

---
 drivers/infiniband/hw/irdma/verbs.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
drivers/infiniband/hw/irdma/verbs.c +7 -0
index 8cd5f9261692..31039b295206 100644
@@ -1094,6 +1094,15 @@ static int irdma_query_pkey(struct ib_device *ibdev,
 u32 port, u16 index,
 	return 0;
 }
 
+
+static u16 irdma_get_udp_sport(u32 fl, u32 lqpn, u32 rqpn) {
+	if (!fl)
+		fl = rdma_calc_flow_label(lqpn, rqpn);
+
+	return rdma_flow_label_to_udp_sport(fl); }
+
 /**
  * irdma_modify_qp_roce - modify qp request
  * @ibqp: qp's pointer for modify

drivers/infiniband/hw/irdma/verbs.c int irdma_modify_qp_roce(struct ib_qp *ibqp, struct · line 1184

Quoted 9 lines
drivers/infiniband/hw/irdma/verbs.c +5 -0
@@ -1167,6 +1176,11 @@ int irdma_modify_qp_roce(struct ib_qp *ibqp, struct
 ib_qp_attr *attr,
 
 		memset(&iwqp->roce_ah, 0, sizeof(iwqp->roce_ah));
 		if (attr->ah_attr.ah_flags & IB_AH_GRH) {
+			u32 fl = udp_info->flow_label;
+			u32 lqp = ibqp->qp_num;
+			u32 rqp = roce_info->dest_qp;
+
+			udp_info->src_port = irdma_get_udp_sport(fl, lqp, rqp);

The flow label used in function is not right as udp_info->flow_label is only set a few lines below. Kindly fold this fix into your next revision.

Quoted 5 lines
drivers/infiniband/hw/irdma/verbs.c +0 -0
 			udp_info->ttl = attr->ah_attr.grh.hop_limit;
 			udp_info->flow_label = attr->ah_attr.grh.flow_label;

udp_info->tos = attr->ah_attr.grh.traffic_class;
--
2.27.0

author
#6 lore raw
Quoted 42 lines · Shiraz
Quoted 42 lines

Subject: [PATCHv3 1/1] RDMA/irdma: Make the source udp port vary

From: Zhu Yanjun <[email protected]>

Based on the link https://www.spinics.net/lists/linux-rdma/msg73735.html,
get the source udp port number for a QP based on the grh.flow_label or lqpn/rqrpn. This provides a better spread of traffic across NIC RX queues. The method in the commit 2b880b2e5e03 ("RDMA/mlx5: Define RoCEv2 udp source port when set path") is a standard way. So it is also adopted in this commit.

---
V2->V3: Move to the block of IB_QP_AV in the mask and IB_AH_GRH in V2->ah_flags
V1->V2: Adopt a standard method to get udp source port.

---
  drivers/infiniband/hw/irdma/verbs.c | 14 ++++++++++++++
  1 file changed, 14 insertions(+)
drivers/infiniband/hw/irdma/verbs.c +7 -0
index 8cd5f9261692..31039b295206 100644
@@ -1094,6 +1094,15 @@ static int irdma_query_pkey(struct ib_device *ibdev,
 u32 port, u16 index,
  	return 0;
  }
 
+
+static u16 irdma_get_udp_sport(u32 fl, u32 lqpn, u32 rqpn) {
+	if (!fl)
+		fl = rdma_calc_flow_label(lqpn, rqpn);
+
+	return rdma_flow_label_to_udp_sport(fl); }
+
  /**
   * irdma_modify_qp_roce - modify qp request
   * @ibqp: qp's pointer for modify
@@ -1167,6 +1176,11 @@ int irdma_modify_qp_roce(struct ib_qp *ibqp, struct
 ib_qp_attr *attr,
 
Quoted 8 lines

drivers/infiniband/hw/irdma/verbs.c int irdma_modify_qp_roce(struct ib_qp *ibqp, struct ib_qp_attr *attr, · line 1183

Quoted 7 lines
drivers/infiniband/hw/irdma/verbs.c +5 -0
  		memset(&iwqp->roce_ah, 0, sizeof(iwqp->roce_ah));
  		if (attr->ah_attr.ah_flags & IB_AH_GRH) {
+			u32 fl = udp_info->flow_label;
+			u32 lqp = ibqp->qp_num;
+			u32 rqp = roce_info->dest_qp;
+
+			udp_info->src_port = irdma_get_udp_sport(fl, lqp, rqp);

The flow label used in function is not right as udp_info->flow_label is only set a few lines below. Kindly fold this fix into your next revision.

Thanks.  " attr->ah_attr.grh.flow_label " should be OK.

Zhu Yanjun

Quoted 6 lines
Quoted 6 lines
drivers/infiniband/hw/irdma/verbs.c +0 -0
  			udp_info->ttl = attr->ah_attr.grh.hop_limit;
  			udp_info->flow_label = attr->ah_attr.grh.flow_label;

udp_info->tos = attr->ah_attr.grh.traffic_class;
--
2.27.0