Skip to contentSkip to conversation

RDMA/irdma: Make the source udp port vary

PATCH Superseded [email protected] 4 participants 7 messages Dec 13, 2021 – Dec 18, 2021

Subsystems (2)

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 RDMA/irdma: Make the source udp port vary

From: Zhu Yanjun <[email protected]>

Based on the link https://www.spinics.net/lists/linux-rdma/msg73735.html,
get the source udp port number for a QP based on the local QPN. This provides a better spread of traffic across NIC RX queues. The method in the commit d3c04a3a6870 ("IB/rxe: vary the source udp port for receive scaling") is stable. So it is also adopted in this commit.

---
 drivers/infiniband/hw/irdma/verbs.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
drivers/infiniband/hw/irdma/verbs.c +6 -1
index 102dc9342f2a..2697b40a539e 100644
@@ -690,6 +690,11 @@ static int irdma_cqp_create_qp_cmd(struct irdma_qp *iwqp)
 	return status ? -ENOMEM : 0;
 }
 
+static inline u16 irdma_get_src_port(struct irdma_qp *iwqp)
+{
+	return 0xc000 + (hash_32_generic(iwqp->ibqp.qp_num, 14) & 0x3fff);
+}
+
 static void irdma_roce_fill_and_set_qpctx_info(struct irdma_qp *iwqp,
 					       struct irdma_qp_host_ctx_info *ctx_info)
 {
@@ -703,7 +708,7 @@ static void irdma_roce_fill_and_set_qpctx_info(struct irdma_qp *iwqp,
 	udp_info->cwnd = iwdev->roce_cwnd;
 	udp_info->rexmit_thresh = 2;
 	udp_info->rnr_nak_thresh = 2;
-	udp_info->src_port = 0xc000;
+	udp_info->src_port = irdma_get_src_port(iwqp);
 	udp_info->dst_port = ROCE_V2_UDP_DPORT;
 	roce_info = &iwqp->roce_info;
 	ether_addr_copy(roce_info->mac_addr, iwdev->netdev->dev_addr);
Signature · 1 line
2.27.0
#1 lore raw
Quoted 18 lines · On 12/14/2021 12:42 AM, [email protected] wrote:

From: Zhu Yanjun <[email protected]>

Based on the link https://www.spinics.net/lists/linux-rdma/msg73735.html,
get the source udp port number for a QP based on the local QPN. This provides a better spread of traffic across NIC RX queues. The method in the commit d3c04a3a6870 ("IB/rxe: vary the source udp port for receive scaling") is stable. So it is also adopted in this commit.

---
  drivers/infiniband/hw/irdma/verbs.c | 7 ++++++-
  1 file changed, 6 insertions(+), 1 deletion(-)
drivers/infiniband/hw/irdma/verbs.c +0 -0
index 102dc9342f2a..2697b40a539e 100644

drivers/infiniband/hw/irdma/verbs.c static int irdma_cqp_create_qp_cmd(struct irdma_qp *iwqp) · line 696

Quoted 7 lines
drivers/infiniband/hw/irdma/verbs.c +4 -0
@@ -690,6 +690,11 @@ static int irdma_cqp_create_qp_cmd(struct irdma_qp *iwqp)
  	return status ? -ENOMEM : 0;
  }
 
+static inline u16 irdma_get_src_port(struct irdma_qp *iwqp)
+{
+	return 0xc000 + (hash_32_generic(iwqp->ibqp.qp_num, 14) & 0x3fff);
+}

How do you ensure the resulting port number is not already in use?

Tom.

Quoted 13 lines
drivers/infiniband/hw/irdma/verbs.c +2 -1
+
  static void irdma_roce_fill_and_set_qpctx_info(struct irdma_qp *iwqp,
  					       struct irdma_qp_host_ctx_info *ctx_info)
  {
@@ -703,7 +708,7 @@ static void irdma_roce_fill_and_set_qpctx_info(struct irdma_qp *iwqp,
  	udp_info->cwnd = iwdev->roce_cwnd;
  	udp_info->rexmit_thresh = 2;
  	udp_info->rnr_nak_thresh = 2;
-	udp_info->src_port = 0xc000;
+	udp_info->src_port = irdma_get_src_port(iwqp);
  	udp_info->dst_port = ROCE_V2_UDP_DPORT;
  	roce_info = &iwqp->roce_info;
  	ether_addr_copy(roce_info->mac_addr, iwdev->netdev->dev_addr);
#2 lore raw
Quoted 13 lines · Tom Talpey
Quoted 12 lines · On 12/14/2021 12:42 AM, [email protected] wrote:

From: Zhu Yanjun <[email protected]>

Based on the link https://www.spinics.net/lists/linux-rdma/msg73735.html,
get the source udp port number for a QP based on the local QPN. This provides a better spread of traffic across NIC RX queues. The method in the commit d3c04a3a6870 ("IB/rxe: vary the source udp port for receive scaling") is stable. So it is also adopted in this commit.

  drivers/infiniband/hw/irdma/verbs.c | 7 ++++++-
  1 file changed, 6 insertions(+), 1 deletion(-)
Quoted 12 lines

drivers/infiniband/hw/irdma/verbs.c static int irdma_cqp_create_qp_cmd(struct irdma_qp *iwqp) · line 695

Quoted 10 lines
drivers/infiniband/hw/irdma/verbs.c +5 -0
index 102dc9342f2a..2697b40a539e 100644
+++ b/drivers/infiniband/hw/irdma/verbs.c
@@ -690,6 +690,11 @@ static int irdma_cqp_create_qp_cmd(struct irdma_qp *iwqp)
  	return status ? -ENOMEM : 0;
  }
+static inline u16 irdma_get_src_port(struct irdma_qp *iwqp)
+{
+	return 0xc000 + (hash_32_generic(iwqp->ibqp.qp_num, 14) & 0x3fff);
+}

How do you ensure the resulting port number is not already in use?

It doesn't matter, it is never used by anything, the receiver captures all data with the roce dport and ignores the sport

Jason

#3 lore raw
Quoted 27 lines · Jason Gunthorpe
Quoted 13 lines · Tom Talpey
Quoted 12 lines · On 12/14/2021 12:42 AM, [email protected] wrote:

From: Zhu Yanjun <[email protected]>

Based on the link https://www.spinics.net/lists/linux-rdma/msg73735.html,
get the source udp port number for a QP based on the local QPN. This provides a better spread of traffic across NIC RX queues. The method in the commit d3c04a3a6870 ("IB/rxe: vary the source udp port for receive scaling") is stable. So it is also adopted in this commit.

   drivers/infiniband/hw/irdma/verbs.c | 7 ++++++-
   1 file changed, 6 insertions(+), 1 deletion(-)
Quoted 12 lines

drivers/infiniband/hw/irdma/verbs.c static int irdma_cqp_create_qp_cmd(struct irdma_qp *iwqp) · line 695

Quoted 10 lines
drivers/infiniband/hw/irdma/verbs.c +5 -0
index 102dc9342f2a..2697b40a539e 100644
+++ b/drivers/infiniband/hw/irdma/verbs.c
@@ -690,6 +690,11 @@ static int irdma_cqp_create_qp_cmd(struct irdma_qp *iwqp)
   	return status ? -ENOMEM : 0;
   }
+static inline u16 irdma_get_src_port(struct irdma_qp *iwqp)
+{
+	return 0xc000 + (hash_32_generic(iwqp->ibqp.qp_num, 14) & 0x3fff);
+}

How do you ensure the resulting port number is not already in use?

Quoted 2 lines

It doesn't matter, it is never used by anything, the receiver captures all data with the roce dport and ignores the sport

It still violates core networking addressing principles, and will mightily confuse a network capture that's filtering on source ports. Firewalls, ICMP, and similar fabric behaviors may also interfere.

SoftRoCE is forced to register/reserve the source port, isn't it?

Tom.

#4 lore raw
Quoted 28 lines · Tom Talpey
Quoted 27 lines · Jason Gunthorpe
Quoted 25 lines · Tom Talpey
Quoted 16 lines · On 12/14/2021 12:42 AM, [email protected] wrote:

From: Zhu Yanjun <[email protected]>

Based on the link https://www.spinics.net/lists/linux-rdma/msg73735.html,
get the source udp port number for a QP based on the local QPN. This provides a better spread of traffic across NIC RX queues. The method in the commit d3c04a3a6870 ("IB/rxe: vary the source udp port for receive scaling") is stable. So it is also adopted in this commit.

   drivers/infiniband/hw/irdma/verbs.c | 7 ++++++-
   1 file changed, 6 insertions(+), 1 deletion(-)
drivers/infiniband/hw/irdma/verbs.c +1 -0
index 102dc9342f2a..2697b40a539e 100644
+++ b/drivers/infiniband/hw/irdma/verbs.c

drivers/infiniband/hw/irdma/verbs.c static int irdma_cqp_create_qp_cmd(struct irdma_qp *iwqp) · line 695

Quoted 6 lines
drivers/infiniband/hw/irdma/verbs.c +4 -0
@@ -690,6 +690,11 @@ static int irdma_cqp_create_qp_cmd(struct irdma_qp *iwqp)
   	return status ? -ENOMEM : 0;
   }
+static inline u16 irdma_get_src_port(struct irdma_qp *iwqp)
+{
+	return 0xc000 + (hash_32_generic(iwqp->ibqp.qp_num, 14) & 0x3fff);
+}

How do you ensure the resulting port number is not already in use?

Quoted 6 lines
Quoted 2 lines

It doesn't matter, it is never used by anything, the receiver captures all data with the roce dport and ignores the sport

It still violates core networking addressing principles, and will mightily confuse a network capture that's filtering on source ports. Firewalls, ICMP, and similar fabric behaviors may also interfere.

Maybe, but most of that stuff doesn't work with roce anyhow.

Quoted 1 line

SoftRoCE is forced to register/reserve the source port, isn't it?

Logically it has to register the dest port, it receives from any source port.

Due to the way the netstack works softroce can't do this trick either, IIRC.

Jason

#5 lore raw
In reply to [email protected]
Quoted 20 lines

Subject: [PATCH 1/1] RDMA/irdma: Make the source udp port vary

From: Zhu Yanjun <[email protected]>

Based on the link https://www.spinics.net/lists/linux-rdma/msg73735.html,
get the source udp port number for a QP based on the local QPN. This provides a better spread of traffic across NIC RX queues. The method in the commit d3c04a3a6870 ("IB/rxe: vary the source udp port for receive scaling") is stable. So it is also adopted in this commit.

---
 drivers/infiniband/hw/irdma/verbs.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
drivers/infiniband/hw/irdma/verbs.c +0 -0
index 102dc9342f2a..2697b40a539e 100644

drivers/infiniband/hw/irdma/verbs.c static int irdma_cqp_create_qp_cmd(struct irdma_qp · line 696

Quoted 7 lines
drivers/infiniband/hw/irdma/verbs.c +3 -0
@@ -690,6 +690,11 @@ static int irdma_cqp_create_qp_cmd(struct irdma_qp
 *iwqp)
 	return status ? -ENOMEM : 0;
 }
 
+static inline u16 irdma_get_src_port(struct irdma_qp *iwqp) {
+	return 0xc000 + (hash_32_generic(iwqp->ibqp.qp_num, 14) & 0x3fff); }
+

There are core hash function helpers based on the grh.flow_label or lqpn/rqrpn that RoCEv2 drivers could use the to get the UDP src port?

https://elixir.bootlin.com/linux/v5.16-rc5/source/include/rdma/ib_verbs.h#L4719

Why don't we use them instead to set the udp_info->src_port in irdma_modify_qp_roce when the path address vector is provided?

Shiraz

author
#6 lore raw
Quoted 30 lines · Shiraz
Quoted 20 lines

Subject: [PATCH 1/1] RDMA/irdma: Make the source udp port vary

From: Zhu Yanjun <[email protected]>

Based on the link https://www.spinics.net/lists/linux-rdma/msg73735.html,
get the source udp port number for a QP based on the local QPN. This provides a better spread of traffic across NIC RX queues. The method in the commit d3c04a3a6870 ("IB/rxe: vary the source udp port for receive scaling") is stable. So it is also adopted in this commit.

---
  drivers/infiniband/hw/irdma/verbs.c | 7 ++++++-
  1 file changed, 6 insertions(+), 1 deletion(-)
drivers/infiniband/hw/irdma/verbs.c +0 -0
index 102dc9342f2a..2697b40a539e 100644

drivers/infiniband/hw/irdma/verbs.c static int irdma_cqp_create_qp_cmd(struct irdma_qp · line 696

Quoted 7 lines
drivers/infiniband/hw/irdma/verbs.c +3 -0
@@ -690,6 +690,11 @@ static int irdma_cqp_create_qp_cmd(struct irdma_qp
 *iwqp)
  	return status ? -ENOMEM : 0;
  }
 
+static inline u16 irdma_get_src_port(struct irdma_qp *iwqp) {
+	return 0xc000 + (hash_32_generic(iwqp->ibqp.qp_num, 14) & 0x3fff); }
+

There are core hash function helpers based on the grh.flow_label or lqpn/rqrpn that RoCEv2 drivers could use the to get the UDP src port?

Quoted 3 lines

https://elixir.bootlin.com/linux/v5.16-rc5/source/include/rdma/ib_verbs.h#L4719

Why don't we use them instead to set the udp_info->src_port in irdma_modify_qp_roce when the path address vector is provided?

Got it. I will send a new patch based on your suggestion. Thanks.
Zhu Yanjun

Quoted 2 lines

Shiraz