Skip to contentSkip to conversation

net: mana: Batch ringing RX queue doorbell on receiving packets

PATCH Superseded [email protected] 5 participants 5 messages Jun 22, 2023 – Jun 26, 2023

Subsystems (6)

v2 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 net: mana: Batch ringing RX queue doorbell on receiving packets

From: Long Li <[email protected]>

It's inefficient to ring the doorbell page every time a WQE is posted to the received queue.

Move the code for ringing doorbell page to where after we have posted all WQEs to the receive queue during a callback from napi_poll().

Tests showed no regression in network latency benchmarks.

Fixes ca9c54d2d6a5 ("net: mana: Add a driver for Microsoft Azure Network Adapter (MANA)")

---
Change log:
v2:
Check for comp_read > 0 as it might be negative on completion error. Set rq.wqe_cnt to 0 according to BNIC spec.

 drivers/net/ethernet/microsoft/mana/gdma_main.c |  5 ++++-
 drivers/net/ethernet/microsoft/mana/mana_en.c   | 10 ++++++++--
 2 files changed, 12 insertions(+), 3 deletions(-)
drivers/net/ethernet/microsoft/mana/gdma_main.c +4 -1
index 8f3f78b68592..ef11d09a3655 100644
@@ -300,8 +300,11 @@ static void mana_gd_ring_doorbell(struct gdma_context *gc, u32 db_index,
 
 void mana_gd_wq_ring_doorbell(struct gdma_context *gc, struct gdma_queue *queue)
 {
+	/* BNIC Spec specifies that client should set 0 for rq.wqe_cnt
+	 * This value is not used in sq
+	 */
 	mana_gd_ring_doorbell(gc, queue->gdma_dev->doorbell, queue->type,
-			      queue->id, queue->head * GDMA_WQE_BU_SIZE, 1);
+			      queue->id, queue->head * GDMA_WQE_BU_SIZE, 0);
 }
 
 void mana_gd_ring_cq(struct gdma_queue *cq, u8 arm_bit)
drivers/net/ethernet/microsoft/mana/mana_en.c +8 -2
index cd4d5ceb9f2d..1d8abe63fcb8 100644
@@ -1383,8 +1383,8 @@ static void mana_post_pkt_rxq(struct mana_rxq *rxq)
 
 	recv_buf_oob = &rxq->rx_oobs[curr_index];
 
-	err = mana_gd_post_and_ring(rxq->gdma_rq, &recv_buf_oob->wqe_req,
-				    &recv_buf_oob->wqe_inf);
+	err = mana_gd_post_work_request(rxq->gdma_rq, &recv_buf_oob->wqe_req,
+					&recv_buf_oob->wqe_inf);
 	if (WARN_ON_ONCE(err))
 		return;
 
@@ -1654,6 +1654,12 @@ static void mana_poll_rx_cq(struct mana_cq *cq)
 		mana_process_rx_cqe(rxq, cq, &comp[i]);
 	}
 
+	if (comp_read > 0) {
+		struct gdma_context *gc = rxq->gdma_rq->gdma_dev->gdma_context;
+
+		mana_gd_wq_ring_doorbell(gc, rxq->gdma_rq);
+	}
+
 	if (rxq->xdp_flush)
 		xdp_do_flush();
 }
Signature · 1 line
2.34.1
maintainer Reviewed-by on v2 1/1
#1 lore raw
Quoted 25 lines

-----Original Message-----
From: [email protected] <[email protected]>
Sent: Thursday, June 22, 2023 12:23 PM
To: Jason Gunthorpe <[email protected]>; Leon Romanovsky <[email protected]>; Ajay Sharma <[email protected]>; Dexuan Cui <[email protected]>; KY Srinivasan <[email protected]>; Haiyang Zhang <[email protected]>; Wei Liu <[email protected]>; David S. Miller <[email protected]>; Eric Dumazet <[email protected]>; Jakub Kicinski <[email protected]>; Paolo Abeni <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Long Li <[email protected]>; [email protected]
Subject: [PATCH v2] net: mana: Batch ringing RX queue doorbell on receiving packets

From: Long Li <[email protected]>

It's inefficient to ring the doorbell page every time a WQE is posted to the received queue.

Move the code for ringing doorbell page to where after we have posted all WQEs to the receive queue during a callback from napi_poll().

Quoted 6 lines

Tests showed no regression in network latency benchmarks.

Fixes ca9c54d2d6a5 ("net: mana: Add a driver for Microsoft Azure Network

Adapter (MANA)")

maintainer Reviewed-by on v2 1/1
#2 lore raw
In reply to [email protected]
Quoted 10 lines

From: [email protected]
Sent: Thursday, June 22, 2023 9:23 AM
...

It's inefficient to ring the doorbell page every time a WQE is posted to the received queue.

Move the code for ringing doorbell page to where after we have posted all WQEs to the receive queue during a callback from napi_poll().

Quoted 6 lines

Tests showed no regression in network latency benchmarks.

Fixes ca9c54d2d6a5 ("net: mana: Add a driver for Microsoft Azure Network

Adapter (MANA)")

maintainer
#3 lore raw
In reply to [email protected]
Quoted 10 lines · On Thu, 22 Jun 2023 09:22:36 -0700 [email protected] wrote:

It's inefficient to ring the doorbell page every time a WQE is posted to the received queue.

Move the code for ringing doorbell page to where after we have posted all WQEs to the receive queue during a callback from napi_poll().

Tests showed no regression in network latency benchmarks.

Fixes ca9c54d2d6a5 ("net: mana: Add a driver for Microsoft Azure Network Adapter (MANA)")

If this is supposed to be a fix, you need to clearly explain what the performance loss was, so that backporters can make an informed decision.

Quoted 9 lines
 drivers/net/ethernet/microsoft/mana/gdma_main.c |  5 ++++-
 drivers/net/ethernet/microsoft/mana/mana_en.c   | 10 ++++++++--
 2 files changed, 12 insertions(+), 3 deletions(-)
drivers/net/ethernet/microsoft/mana/gdma_main.c +0 -0
index 8f3f78b68592..ef11d09a3655 100644

drivers/net/ethernet/microsoft/mana/gdma_main.c static void mana_gd_ring_doorbell(struct gdma_context *gc, u32 db_index, · line 308

Quoted 10 lines
drivers/net/ethernet/microsoft/mana/gdma_main.c +4 -1
@@ -300,8 +300,11 @@ static void mana_gd_ring_doorbell(struct gdma_context *gc, u32 db_index,
 
 void mana_gd_wq_ring_doorbell(struct gdma_context *gc, struct gdma_queue *queue)
 {
+	/* BNIC Spec specifies that client should set 0 for rq.wqe_cnt
+	 * This value is not used in sq
+	 */
 	mana_gd_ring_doorbell(gc, queue->gdma_dev->doorbell, queue->type,
-			      queue->id, queue->head * GDMA_WQE_BU_SIZE, 1);
+			      queue->id, queue->head * GDMA_WQE_BU_SIZE, 0);
 }

This change needs to be explained in the commit message, or should be a separate patch.

Signature · 1 line
pw-bot: cr
#4 lore raw
Quoted 10 lines

Subject: Re: [PATCH v2] net: mana: Batch ringing RX queue doorbell on receiving packets

Quoted 6 lines · On Thu, 22 Jun 2023 09:22:36 -0700 [email protected] wrote:

It's inefficient to ring the doorbell page every time a WQE is posted to the received queue.

Move the code for ringing doorbell page to where after we have posted all WQEs to the receive queue during a callback from napi_poll().

Quoted 8 lines
Quoted 5 lines

Tests showed no regression in network latency benchmarks.

Fixes ca9c54d2d6a5 ("net: mana: Add a driver for Microsoft Azure

Network Adapter (MANA)")

If this is supposed to be a fix, you need to clearly explain what the performance loss was, so that backporters can make an informed decision.

I'm sending v3 to include the details.

Quoted 22 lines
Quoted 11 lines
 drivers/net/ethernet/microsoft/mana/gdma_main.c |  5 ++++-
 drivers/net/ethernet/microsoft/mana/mana_en.c   | 10 ++++++++--
 2 files changed, 12 insertions(+), 3 deletions(-)
drivers/net/ethernet/microsoft/mana/gdma_main.c +0 -0
index 8f3f78b68592..ef11d09a3655 100644

drivers/net/ethernet/microsoft/mana/gdma_main.c static void mana_gd_ring_doorbell(struct · line 306

Quoted 10 lines
drivers/net/ethernet/microsoft/mana/gdma_main.c +3 -1
@@ -300,8 +300,11 @@ static void mana_gd_ring_doorbell(struct
 gdma_context *gc, u32 db_index,
 
 void mana_gd_wq_ring_doorbell(struct gdma_context *gc, struct
 gdma_queue *queue)  {
+	/* BNIC Spec specifies that client should set 0 for rq.wqe_cnt
+	 * This value is not used in sq
+	 */
 	mana_gd_ring_doorbell(gc, queue->gdma_dev->doorbell, queue-
 type,
-			      queue->id, queue->head * GDMA_WQE_BU_SIZE,

1);

Quoted 6 lines

drivers/net/ethernet/microsoft/mana/gdma_main.c static void mana_gd_ring_doorbell(struct · line 309

Quoted 1 line
drivers/net/ethernet/microsoft/mana/gdma_main.c +1 -0
+			      queue->id, queue->head * GDMA_WQE_BU_SIZE,

0);

drivers/net/ethernet/microsoft/mana/gdma_main.c static void mana_gd_ring_doorbell(struct · line 310

Quoted 1 line
drivers/net/ethernet/microsoft/mana/gdma_main.c +0 -0
 }

This change needs to be explained in the commit message, or should be a separate patch.

Will explain the change in the commit message.

Thanks,
Long

Quoted 2 lines

--
pw-bot: cr