Skip to contentSkip to conversation

Generate UDP src port with flow label or lqpn/rqpn

PATCH Superseded [email protected] 4 participants 19 messages Jan 5, 2022

Subsystems (4)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

v2 cover Generate UDP src port with flow label or lqpn/rqpn

From: Zhu Yanjun <[email protected]>

Follow the advice from Leon Romanovsky, rdma_get_udp_sport is moved to ib_verbs.h. several drivers generate udp source port with this function.

---
v1->v2:Remove the local variables in commits "RDMA/irdma: Make the source
udp port vary" and "RDMA/rxe: Use the standard method to produce udp source port". A new commit is added to remove the redundant randomization for UDP source port in RXE.
---

Zhu Yanjun (5):
RDMA/core: Calculate UDP source port based on flow label or lqpn/rqpn RDMA/hns: Replace get_udp_sport with rdma_get_udp_sport RDMA/irdma: Make the source udp port vary
RDMA/rxe: Use the standard method to produce udp source port RDMA/rxe: Remove the redundant randomization for UDP source port

 drivers/infiniband/hw/hns/hns_roce_hw_v2.c | 12 ++----------
 drivers/infiniband/hw/irdma/verbs.c        |  4 ++++
 drivers/infiniband/sw/rxe/rxe_qp.c         | 10 ++--------
 drivers/infiniband/sw/rxe/rxe_verbs.c      |  6 ++++++
 include/rdma/ib_verbs.h                    | 17 +++++++++++++++++
 5 files changed, 31 insertions(+), 18 deletions(-)
Signature · 1 line
2.27.0
#1 lore raw

PATCH v2 1/5 RDMA/core: Calculate UDP source port based on flow label or lqpn/rqpn

From: Zhu Yanjun <[email protected]>

Calculate and set UDP source port based on the flow label. If flow label is not defined in GRH then calculate it based on lqpn/rqpn.

---
 include/rdma/ib_verbs.h | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
include/rdma/ib_verbs.h +17 -0
index 6e9ad656ecb7..69d883f7fb41 100644
@@ -4749,6 +4749,23 @@ static inline u32 rdma_calc_flow_label(u32 lqpn, u32 rqpn)
 	return (u32)(v & IB_GRH_FLOWLABEL_MASK);
 }
 
+/**
+ * rdma_get_udp_sport - Calculate and set UDP source port based on the flow
+ *                      label. If flow label is not defined in GRH then
+ *                      calculate it based on lqpn/rqpn.
+ *
+ * @fl:                 flow label from GRH
+ * @lqpn:               local qp number
+ * @rqpn:               remote qp number
+ */
+static inline u16 rdma_get_udp_sport(u32 fl, u32 lqpn, u32 rqpn)
+{
+	if (!fl)
+		fl = rdma_calc_flow_label(lqpn, rqpn);
+
+	return rdma_flow_label_to_udp_sport(fl);
+}
+
 const struct ib_port_immutable*
 ib_port_immutable_read(struct ib_device *dev, unsigned int port);
 #endif /* IB_VERBS_H */
Signature · 1 line
2.27.0
#3 lore raw

PATCH v2 2/5 RDMA/hns: Replace get_udp_sport with rdma_get_udp_sport

In reply to [email protected]

From: Zhu Yanjun <[email protected]>

Several drivers have the same function xxx_get_udp_sport. So this function is moved to ib_verbs.h.

---
 drivers/infiniband/hw/hns/hns_roce_hw_v2.c | 12 ++----------
 1 file changed, 2 insertions(+), 10 deletions(-)
drivers/infiniband/hw/hns/hns_roce_hw_v2.c +2 -10
index eb0defa80d0d..cb795663b813 100644
@@ -4488,14 +4488,6 @@ static int modify_qp_rtr_to_rts(struct ib_qp *ibqp,
 	return 0;
 }
 
-static inline u16 get_udp_sport(u32 fl, u32 lqpn, u32 rqpn)
-{
-	if (!fl)
-		fl = rdma_calc_flow_label(lqpn, rqpn);
-
-	return rdma_flow_label_to_udp_sport(fl);
-}
-
 static int get_dip_ctx_idx(struct ib_qp *ibqp, const struct ib_qp_attr *attr,
 			   u32 *dip_idx)
 {
@@ -4712,8 +4704,8 @@ static int hns_roce_v2_set_path(struct ib_qp *ibqp,
 	}
 
 	hr_reg_write(context, QPC_UDPSPN,
-		     is_udp ? get_udp_sport(grh->flow_label, ibqp->qp_num,
-					    attr->dest_qp_num) : 0);
+		     is_udp ? rdma_get_udp_sport(grh->flow_label, ibqp->qp_num,
+						 attr->dest_qp_num) : 0);
 
 	hr_reg_clear(qpc_mask, QPC_UDPSPN);
 
Signature · 1 line
2.27.0
#5 lore raw

PATCH v2 3/5 RDMA/irdma: Make the source udp port vary

In reply to [email protected]

From: Zhu Yanjun <[email protected]>

Get the source udp port number for a QP based on the grh.flow_label or lqpn/rqrpn. This provides a better spread of traffic across NIC RX queues.

---
 drivers/infiniband/hw/irdma/verbs.c | 4 ++++
 1 file changed, 4 insertions(+)
drivers/infiniband/hw/irdma/verbs.c +4 -0
index 8cd5f9261692..8234a101e752 100644
@@ -1170,6 +1170,10 @@ int irdma_modify_qp_roce(struct ib_qp *ibqp, struct ib_qp_attr *attr,
 			udp_info->ttl = attr->ah_attr.grh.hop_limit;
 			udp_info->flow_label = attr->ah_attr.grh.flow_label;
 			udp_info->tos = attr->ah_attr.grh.traffic_class;
+			udp_info->src_port = rdma_get_udp_sport(
+						udp_info->flow_label,
+						ibqp->qp_num,
+						roce_info->dest_qp);
 			irdma_qp_rem_qos(&iwqp->sc_qp);
 			dev->ws_remove(iwqp->sc_qp.vsi, ctx_info->user_pri);
 			ctx_info->user_pri = rt_tos2priority(udp_info->tos);
Signature · 1 line
2.27.0
#8 lore raw

PATCH v2 4/5 RDMA/rxe: Use the standard method to produce udp source port

In reply to [email protected]

From: Zhu Yanjun <[email protected]>

Use the standard method to produce udp source port.

---
 drivers/infiniband/sw/rxe/rxe_verbs.c | 6 ++++++
 1 file changed, 6 insertions(+)
drivers/infiniband/sw/rxe/rxe_verbs.c +6 -0
index 0aa0d7e52773..42fa81b455de 100644
@@ -469,6 +469,12 @@ static int rxe_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr,
 	if (err)
 		goto err1;
 
+	if ((mask & IB_QP_AV) && (attr->ah_attr.ah_flags & IB_AH_GRH))
+		qp->src_port = rdma_get_udp_sport(attr->ah_attr.grh.flow_label,
+						  qp->ibqp.qp_num,
+						  qp->attr.dest_qp_num);
+
+
 	return 0;
 
 err1:
Signature · 1 line
2.27.0
#14 lore raw

PATCH v2 5/5 RDMA/rxe: Remove the redundant randomization for UDP source port

In reply to [email protected]

From: Zhu Yanjun <[email protected]>

Since the UDP source port is modified in rxe_modify_qp, the randomization for UDP source port is redundant in this function. So remove it.

---
 drivers/infiniband/sw/rxe/rxe_qp.c | 10 ++--------
 1 file changed, 2 insertions(+), 8 deletions(-)
drivers/infiniband/sw/rxe/rxe_qp.c +2 -8
index 54b8711321c1..84d6ffe7350a 100644
@@ -210,15 +210,9 @@ static int rxe_qp_init_req(struct rxe_dev *rxe, struct rxe_qp *qp,
 		return err;
 	qp->sk->sk->sk_user_data = qp;
 
-	/* pick a source UDP port number for this QP based on
-	 * the source QPN. this spreads traffic for different QPs
-	 * across different NIC RX queues (while using a single
-	 * flow for a given QP to maintain packet order).
-	 * the port number must be in the Dynamic Ports range
-	 * (0xc000 - 0xffff).
+	/* Source UDP port number for this QP is modified in rxe_qp_modify.
 	 */
-	qp->src_port = RXE_ROCE_V2_SPORT +
-		(hash_32_generic(qp_num(qp), 14) & 0x3fff);
+	qp->src_port		= RXE_ROCE_V2_SPORT;
 	qp->sq.max_wr		= init->cap.max_send_wr;
 
 	/* These caps are limited by rxe_qp_chk_cap() done by the caller */
Signature · 1 line
2.27.0
#15 lore raw

Re: [PATCH 5/5] RDMA/rxe: Remove the redundant randomization for UDP source port

Quoted 16 lines · On Wed, Jan 05, 2022 at 05:12:37PM -0500, [email protected] wrote:

From: Zhu Yanjun <[email protected]>

Since the UDP source port is modified in rxe_modify_qp, the randomization for UDP source port is redundant in this function. So remove it.

---
 drivers/infiniband/sw/rxe/rxe_qp.c | 10 ++--------
 1 file changed, 2 insertions(+), 8 deletions(-)
drivers/infiniband/sw/rxe/rxe_qp.c +0 -0
index 54b8711321c1..84d6ffe7350a 100644
@@ -210,15 +210,9 @@ static int rxe_qp_init_req(struct rxe_dev *rxe, struct rxe_qp *qp,
 		return err;

drivers/infiniband/sw/rxe/rxe_qp.c static int rxe_qp_init_req(struct rxe_dev *rxe, struct rxe_qp *qp, · line 214

Quoted 10 lines
drivers/infiniband/sw/rxe/rxe_qp.c +1 -6
 	qp->sk->sk->sk_user_data = qp;
 
-	/* pick a source UDP port number for this QP based on
-	 * the source QPN. this spreads traffic for different QPs
-	 * across different NIC RX queues (while using a single
-	 * flow for a given QP to maintain packet order).
-	 * the port number must be in the Dynamic Ports range
-	 * (0xc000 - 0xffff).
+	/* Source UDP port number for this QP is modified in rxe_qp_modify.
 	 */

This makes me wonder why do we set this src_port here? Are we using this field before modify QP?

Thanks

Quoted 9 lines
drivers/infiniband/sw/rxe/rxe_qp.c +1 -2
-	qp->src_port = RXE_ROCE_V2_SPORT +
-		(hash_32_generic(qp_num(qp), 14) & 0x3fff);
+	qp->src_port		= RXE_ROCE_V2_SPORT;
 	qp->sq.max_wr		= init->cap.max_send_wr;
 
 	/* These caps are limited by rxe_qp_chk_cap() done by the caller */
Signature · 1 line
2.27.0
Reviewed-by on v2 1/5
#2 lore raw

Re: [PATCH 1/5] RDMA/core: Calculate UDP source port based on flow label or lqpn/rqpn

In reply to [email protected]
Quoted 10 lines · On Wed, Jan 05, 2022 at 05:12:33PM -0500, [email protected] wrote:

From: Zhu Yanjun <[email protected]>

Calculate and set UDP source port based on the flow label. If flow label is not defined in GRH then calculate it based on lqpn/rqpn.

---
 include/rdma/ib_verbs.h | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

Thanks,

Reviewed-by on v2 2/5
#4 lore raw

Re: [PATCH 2/5] RDMA/hns: Replace get_udp_sport with rdma_get_udp_sport

In reply to [email protected]
Quoted 10 lines · On Wed, Jan 05, 2022 at 05:12:34PM -0500, [email protected] wrote:

From: Zhu Yanjun <[email protected]>

Several drivers have the same function xxx_get_udp_sport. So this function is moved to ib_verbs.h.

---
 drivers/infiniband/hw/hns/hns_roce_hw_v2.c | 12 ++----------
 1 file changed, 2 insertions(+), 10 deletions(-)

Thanks,

Reviewed-by on v2 3/5
#6 lore raw

Re: [PATCH 3/5] RDMA/irdma: Make the source udp port vary

In reply to [email protected]
Quoted 10 lines · On Wed, Jan 05, 2022 at 05:12:35PM -0500, [email protected] wrote:

From: Zhu Yanjun <[email protected]>

Get the source udp port number for a QP based on the grh.flow_label or lqpn/rqrpn. This provides a better spread of traffic across NIC RX queues.

---
 drivers/infiniband/hw/irdma/verbs.c | 4 ++++
 1 file changed, 4 insertions(+)

Thanks,

#9 lore raw

Re: [PATCH 4/5] RDMA/rxe: Use the standard method to produce udp source port

In reply to [email protected]
Quoted 14 lines · On Wed, Jan 05, 2022 at 05:12:36PM -0500, [email protected] wrote:

From: Zhu Yanjun <[email protected]>

Use the standard method to produce udp source port.

---
 drivers/infiniband/sw/rxe/rxe_verbs.c | 6 ++++++
 1 file changed, 6 insertions(+)
drivers/infiniband/sw/rxe/rxe_verbs.c +0 -0
index 0aa0d7e52773..42fa81b455de 100644

drivers/infiniband/sw/rxe/rxe_verbs.c static int rxe_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr, · line 472

Quoted 4 lines
drivers/infiniband/sw/rxe/rxe_verbs.c +1 -0
@@ -469,6 +469,12 @@ static int rxe_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr,
 	if (err)
 		goto err1;
 
+	if ((mask & IB_QP_AV) && (attr->ah_attr.ah_flags & IB_AH_GRH))

You are leaving src_port default and wired to same port as other QPs without any randomization.

Thanks

Quoted 11 lines
drivers/infiniband/sw/rxe/rxe_verbs.c +5 -0
+		qp->src_port = rdma_get_udp_sport(attr->ah_attr.grh.flow_label,
+						  qp->ibqp.qp_num,
+						  qp->attr.dest_qp_num);
+
+
 	return 0;
 
 err1:
Signature · 1 line
2.27.0
maintainer
#10 lore raw

Re: [PATCH 4/5] RDMA/rxe: Use the standard method to produce udp source port

Quoted 19 lines · Leon Romanovsky
Quoted 17 lines · On Wed, Jan 05, 2022 at 05:12:36PM -0500, [email protected] wrote:

From: Zhu Yanjun <[email protected]>

Use the standard method to produce udp source port.

---
 drivers/infiniband/sw/rxe/rxe_verbs.c | 6 ++++++
 1 file changed, 6 insertions(+)
drivers/infiniband/sw/rxe/rxe_verbs.c +0 -0
index 0aa0d7e52773..42fa81b455de 100644
@@ -469,6 +469,12 @@ static int rxe_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr,
      if (err)
              goto err1;
 
Quoted 4 lines

drivers/infiniband/sw/rxe/rxe_verbs.c static int rxe_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr, · line 472

Quoted 1 line
drivers/infiniband/sw/rxe/rxe_verbs.c +1 -0
+     if ((mask & IB_QP_AV) && (attr->ah_attr.ah_flags & IB_AH_GRH))

You are leaving src_port default and wired to same port as other QPs without any randomization.

Hi,

I do not get you. Why do you think I am leaving src_pport default? Thanks.

Zhu Yanjun

Quoted 14 lines

Thanks

Quoted 11 lines
drivers/infiniband/sw/rxe/rxe_verbs.c +5 -1
+             qp->src_port = rdma_get_udp_sport(attr->ah_attr.grh.flow_label,
+                                               qp->ibqp.qp_num,
+                                               qp->attr.dest_qp_num);
+
+
      return 0;
 
 err1:
--

2.27.0

#16 lore raw

Re: [PATCH 5/5] RDMA/rxe: Remove the redundant randomization for UDP source port

In reply to Leon Romanovsky
Quoted 20 lines · Leon Romanovsky
Quoted 18 lines · On Wed, Jan 05, 2022 at 05:12:37PM -0500, [email protected] wrote:

From: Zhu Yanjun <[email protected]>

Since the UDP source port is modified in rxe_modify_qp, the randomization for UDP source port is redundant in this function. So remove it.

---
drivers/infiniband/sw/rxe/rxe_qp.c | 10 ++--------

1 file changed, 2 insertions(+), 8 deletions(-)
drivers/infiniband/sw/rxe/rxe_qp.c +0 -0
index 54b8711321c1..84d6ffe7350a 100644
@@ -210,15 +210,9 @@ static int rxe_qp_init_req(struct rxe_dev *rxe, struct rxe_qp *qp,
 return err;
 qp->sk->sk->sk_user_data = qp;
 
Quoted 11 lines

drivers/infiniband/sw/rxe/rxe_qp.c static int rxe_qp_init_req(struct rxe_dev *rxe, struct rxe_qp *qp, · line 214

Quoted 8 lines
drivers/infiniband/sw/rxe/rxe_qp.c +1 -6
- /* pick a source UDP port number for this QP based on
- * the source QPN. this spreads traffic for different QPs
- * across different NIC RX queues (while using a single
- * flow for a given QP to maintain packet order).
- * the port number must be in the Dynamic Ports range
- * (0xc000 - 0xffff).
+ /* Source UDP port number for this QP is modified in rxe_qp_modify.
 */

This makes me wonder why do we set this src_port here? Are we using this field before modify QP?

The commit d3c04a3a6870 ("IB/rxe: vary the source udp port for receive scaling") sets this src_port here.

The advantage of setting src_port here is: before rxe_modify_qp, the src port is randomized, not 0xc000. So after/before rxe_modify_qp, the src port is the same value.

If the src port is changed in rxe_modify_qp, before rxe_modify_qp, the src port is 0xc000, after rxe_modify_qp, the src port is randomized, for example, src port is 0xF043.

So when the new method is adopted, I removed this.

Zhu Yanjun

Quoted 11 lines

Thanks

Quoted 8 lines
drivers/infiniband/sw/rxe/rxe_qp.c +1 -3
- qp->src_port = RXE_ROCE_V2_SPORT +
- (hash_32_generic(qp_num(qp), 14) & 0x3fff);
+ qp->src_port = RXE_ROCE_V2_SPORT;
 qp->sq.max_wr = init->cap.max_send_wr;
 
 /* These caps are limited by rxe_qp_chk_cap() done by the caller */
--

2.27.0

#11 lore raw

Re: [PATCH 4/5] RDMA/rxe: Use the standard method to produce udp source port

In reply to Zhu Yanjun
Quoted 20 lines · Zhu Yanjun
Quoted 19 lines · Leon Romanovsky
Quoted 17 lines · On Wed, Jan 05, 2022 at 05:12:36PM -0500, [email protected] wrote:

From: Zhu Yanjun <[email protected]>

Use the standard method to produce udp source port.

---
 drivers/infiniband/sw/rxe/rxe_verbs.c | 6 ++++++
 1 file changed, 6 insertions(+)
drivers/infiniband/sw/rxe/rxe_verbs.c +0 -0
index 0aa0d7e52773..42fa81b455de 100644
@@ -469,6 +469,12 @@ static int rxe_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr,
      if (err)
              goto err1;
 
Quoted 8 lines
Quoted 4 lines

drivers/infiniband/sw/rxe/rxe_verbs.c static int rxe_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr, · line 472

Quoted 1 line
drivers/infiniband/sw/rxe/rxe_verbs.c +1 -0
+     if ((mask & IB_QP_AV) && (attr->ah_attr.ah_flags & IB_AH_GRH))

You are leaving src_port default and wired to same port as other QPs without any randomization.

Hi,

I do not get you. Why do you think I am leaving src_pport default?

Because in original code, you randomized src_port without any relation to mask flags.

       qp->src_port = RXE_ROCE_V2_SPORT +
               (hash_32_generic(qp_num(qp), 14) & 0x3fff);

After patch #5, if user doesn't pass "proper" mask, you will leave qp->src_port to be equal to RXE_ROCE_V2_SPORT, which is different from the current behaviour.

Thanks

Quoted 18 lines

Thanks.

Zhu Yanjun

Quoted 14 lines

Thanks

Quoted 11 lines
drivers/infiniband/sw/rxe/rxe_verbs.c +5 -1
+             qp->src_port = rdma_get_udp_sport(attr->ah_attr.grh.flow_label,
+                                               qp->ibqp.qp_num,
+                                               qp->attr.dest_qp_num);
+
+
      return 0;
 
 err1:
--

2.27.0

#17 lore raw

Re: [PATCH 5/5] RDMA/rxe: Remove the redundant randomization for UDP source port

In reply to [email protected]
Quoted 34 lines · On Wed, Jan 05, 2022 at 08:42:03AM +0000, [email protected] wrote:
Quoted 20 lines · Leon Romanovsky
Quoted 18 lines · On Wed, Jan 05, 2022 at 05:12:37PM -0500, [email protected] wrote:

From: Zhu Yanjun <[email protected]>

Since the UDP source port is modified in rxe_modify_qp, the randomization for UDP source port is redundant in this function. So remove it.

---
drivers/infiniband/sw/rxe/rxe_qp.c | 10 ++--------

1 file changed, 2 insertions(+), 8 deletions(-)
drivers/infiniband/sw/rxe/rxe_qp.c +0 -0
index 54b8711321c1..84d6ffe7350a 100644
@@ -210,15 +210,9 @@ static int rxe_qp_init_req(struct rxe_dev *rxe, struct rxe_qp *qp,
 return err;
 qp->sk->sk->sk_user_data = qp;
 
Quoted 11 lines

drivers/infiniband/sw/rxe/rxe_qp.c static int rxe_qp_init_req(struct rxe_dev *rxe, struct rxe_qp *qp, · line 214

Quoted 8 lines
drivers/infiniband/sw/rxe/rxe_qp.c +1 -6
- /* pick a source UDP port number for this QP based on
- * the source QPN. this spreads traffic for different QPs
- * across different NIC RX queues (while using a single
- * flow for a given QP to maintain packet order).
- * the port number must be in the Dynamic Ports range
- * (0xc000 - 0xffff).
+ /* Source UDP port number for this QP is modified in rxe_qp_modify.
 */

This makes me wonder why do we set this src_port here? Are we using this field before modify QP?

Quoted 7 lines

The commit d3c04a3a6870 ("IB/rxe: vary the source udp port for receive scaling") sets this src_port here.

The advantage of setting src_port here is: before rxe_modify_qp, the src port is randomized, not 0xc000. So after/before rxe_modify_qp, the src port is the same value.

If the src port is changed in rxe_modify_qp, before rxe_modify_qp, the src port is 0xc000, after rxe_modify_qp, the src port is randomized, for example, src port is 0xF043.

I'm asking if you use qp->src_port between this line and rxe_modify_qp?

Thanks

Quoted 16 lines

So when the new method is adopted, I removed this.

Zhu Yanjun

Quoted 11 lines

Thanks

Quoted 8 lines
drivers/infiniband/sw/rxe/rxe_qp.c +1 -3
- qp->src_port = RXE_ROCE_V2_SPORT +
- (hash_32_generic(qp_num(qp), 14) & 0x3fff);
+ qp->src_port = RXE_ROCE_V2_SPORT;
 qp->sq.max_wr = init->cap.max_send_wr;
 
 /* These caps are limited by rxe_qp_chk_cap() done by the caller */
--

2.27.0

#18 lore raw

Re: [PATCH 5/5] RDMA/rxe: Remove the redundant randomization for UDP source port

Quoted 43 lines · Leon Romanovsky
Quoted 41 lines · On Wed, Jan 05, 2022 at 08:42:03AM +0000, [email protected] wrote:

January 5, 2022 3:49 PM, "Leon Romanovsky" <[email protected]> wrote:

On Wed, Jan 05, 2022 at 05:12:37PM -0500, [email protected] wrote:

From: Zhu Yanjun <[email protected]>

Since the UDP source port is modified in rxe_modify_qp, the randomization for UDP source port is redundant in this function. So remove it.

---
drivers/infiniband/sw/rxe/rxe_qp.c | 10 ++--------

1 file changed, 2 insertions(+), 8 deletions(-)
drivers/infiniband/sw/rxe/rxe_qp.c +1 -6
index 54b8711321c1..84d6ffe7350a 100644
@@ -210,15 +210,9 @@ static int rxe_qp_init_req(struct rxe_dev *rxe, struct rxe_qp *qp,
 return err;
 qp->sk->sk->sk_user_data = qp;
 
- /* pick a source UDP port number for this QP based on
- * the source QPN. this spreads traffic for different QPs
- * across different NIC RX queues (while using a single
- * flow for a given QP to maintain packet order).
- * the port number must be in the Dynamic Ports range
- * (0xc000 - 0xffff).
+ /* Source UDP port number for this QP is modified in rxe_qp_modify.
 */
 

This makes me wonder why do we set this src_port here? Are we using this field before modify QP?

The commit d3c04a3a6870 ("IB/rxe: vary the source udp port for receive scaling") sets this src_port here.

The advantage of setting src_port here is: before rxe_modify_qp, the src port is randomized, not 0xc000.
So after/before rxe_modify_qp, the src port is the same value.

Quoted 5 lines
Quoted 3 lines

If the src port is changed in rxe_modify_qp, before rxe_modify_qp, the src port is 0xc000, after rxe_modify_qp,
the src port is randomized, for example, src port is 0xF043.

I'm asking if you use qp->src_port between this line and rxe_modify_qp?

There are only 2 udp packets between this line and rxe_modify_qp.

Zhu Yanjun

Quoted 17 lines

Thanks

Quoted 14 lines

So when the new method is adopted, I removed this.

Zhu Yanjun

Thanks

- qp->src_port = RXE_ROCE_V2_SPORT +
- (hash_32_generic(qp_num(qp), 14) & 0x3fff);
+ qp->src_port = RXE_ROCE_V2_SPORT;
qp->sq.max_wr = init->cap.max_send_wr;

/* These caps are limited by rxe_qp_chk_cap() done by the caller */ --
2.27.0

#12 lore raw

Re: [PATCH 4/5] RDMA/rxe: Use the standard method to produce udp source port

In reply to Leon Romanovsky
Quoted 34 lines · Leon Romanovsky
Quoted 20 lines · Zhu Yanjun

On Wed, Jan 5, 2022 at 3:52 PM Leon Romanovsky <[email protected]> wrote:

Quoted 17 lines · On Wed, Jan 05, 2022 at 05:12:36PM -0500, [email protected] wrote:

From: Zhu Yanjun <[email protected]>

Use the standard method to produce udp source port.

---
drivers/infiniband/sw/rxe/rxe_verbs.c | 6 ++++++

1 file changed, 6 insertions(+)
drivers/infiniband/sw/rxe/rxe_verbs.c +0 -0
index 0aa0d7e52773..42fa81b455de 100644
@@ -469,6 +469,12 @@ static int rxe_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr,
 if (err)
 goto err1;
 
Quoted 8 lines

drivers/infiniband/sw/rxe/rxe_verbs.c static int rxe_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr, · line 472

Quoted 1 line
drivers/infiniband/sw/rxe/rxe_verbs.c +1 -0
+ if ((mask & IB_QP_AV) && (attr->ah_attr.ah_flags & IB_AH_GRH))

You are leaving src_port default and wired to same port as other QPs without any randomization.

Hi,

I do not get you. Why do you think I am leaving src_pport default?

Because in original code, you randomized src_port without any relation to mask flags.

Quoted 6 lines

qp->src_port = RXE_ROCE_V2_SPORT +
(hash_32_generic(qp_num(qp), 14) & 0x3fff);

After patch #5, if user doesn't pass "proper" mask, you will leave qp->src_port to be equal to RXE_ROCE_V2_SPORT, which is different from the current behaviour.

About the "proper" mask, please check this link https://patchwork.kernel.org/project/linux-rdma/patch/[email protected]/

"the udp_sport is only set when address vector and dest qpn (IB_QP_AV and IB_QP_DEST_QPN) is provided."

Zhu Yanjun

Quoted 20 lines

Thanks

Quoted 17 lines

Thanks.

Zhu Yanjun

Thanks

Quoted 11 lines
drivers/infiniband/sw/rxe/rxe_verbs.c +5 -0
+ qp->src_port = rdma_get_udp_sport(attr->ah_attr.grh.flow_label,
+ qp->ibqp.qp_num,
+ qp->attr.dest_qp_num);
+
+
 return 0;
 

err1:
--
2.27.0

maintainer
#13 lore raw

Re: [PATCH 4/5] RDMA/rxe: Use the standard method to produce udp source port

In reply to Leon Romanovsky
Quoted 34 lines · Leon Romanovsky
Quoted 20 lines · Zhu Yanjun
Quoted 19 lines · Leon Romanovsky
Quoted 17 lines · On Wed, Jan 05, 2022 at 05:12:36PM -0500, [email protected] wrote:

From: Zhu Yanjun <[email protected]>

Use the standard method to produce udp source port.

---
 drivers/infiniband/sw/rxe/rxe_verbs.c | 6 ++++++
 1 file changed, 6 insertions(+)
drivers/infiniband/sw/rxe/rxe_verbs.c +0 -0
index 0aa0d7e52773..42fa81b455de 100644
@@ -469,6 +469,12 @@ static int rxe_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr,
      if (err)
              goto err1;
 
Quoted 8 lines
Quoted 4 lines

drivers/infiniband/sw/rxe/rxe_verbs.c static int rxe_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr, · line 472

Quoted 1 line
drivers/infiniband/sw/rxe/rxe_verbs.c +1 -0
+     if ((mask & IB_QP_AV) && (attr->ah_attr.ah_flags & IB_AH_GRH))

You are leaving src_port default and wired to same port as other QPs without any randomization.

Hi,

I do not get you. Why do you think I am leaving src_pport default?

Because in original code, you randomized src_port without any relation to mask flags.

Quoted 6 lines
       qp->src_port = RXE_ROCE_V2_SPORT +
               (hash_32_generic(qp_num(qp), 14) & 0x3fff);

After patch #5, if user doesn't pass "proper" mask, you will leave qp->src_port to be equal to RXE_ROCE_V2_SPORT, which is different from the current behaviour.

Hi, Leon Romanovsky

I read your comments again and checked the source code.
And I found this https://lkml.org/lkml/2015/12/15/566, Jason commented:
"
...
The GRH is optional for in-subnet communications.
...
"
I agree with you. When in-subnet communications, GRH is optional. It is possible that qp->src_port is set to RXE_ROCE_V2_SPORT.

So I will remove patch #5 and send the patch series again. Thanks.
Zhu Yanjun

Quoted 22 lines

Thanks

Quoted 18 lines

Thanks.

Zhu Yanjun

Quoted 14 lines

Thanks

Quoted 11 lines
drivers/infiniband/sw/rxe/rxe_verbs.c +5 -1
+             qp->src_port = rdma_get_udp_sport(attr->ah_attr.grh.flow_label,
+                                               qp->ibqp.qp_num,
+                                               qp->attr.dest_qp_num);
+
+
      return 0;
 
 err1:
--

2.27.0

Acked-by on v2 3/5
#7 lore raw

RE: [PATCH 3/5] RDMA/irdma: Make the source udp port vary

In reply to [email protected]
Quoted 4 lines

Subject: [PATCH 3/5] RDMA/irdma: Make the source udp port vary

From: Zhu Yanjun <[email protected]>

Quoted 8 lines

Get the source udp port number for a QP based on the grh.flow_label or lqpn/rqrpn. This provides a better spread of traffic across NIC RX queues.

---
 drivers/infiniband/hw/irdma/verbs.c | 4 ++++
 1 file changed, 4 insertions(+)