net: mana: Batch ringing RX queue doorbell on receiving packets
Subsystems (6)
PATCH v2 1/1 net: mana: Batch ringing RX queue doorbell on receiving packets
From: Long Li <[email protected]>
It's inefficient to ring the doorbell page every time a WQE is posted to the received queue.
Move the code for ringing doorbell page to where after we have posted all WQEs to the receive queue during a callback from napi_poll().
Tests showed no regression in network latency benchmarks.
---
Change log:
v2:
Check for comp_read > 0 as it might be negative on completion error. Set rq.wqe_cnt to 0 according to BNIC spec.
drivers/net/ethernet/microsoft/mana/gdma_main.c | 5 ++++- drivers/net/ethernet/microsoft/mana/mana_en.c | 10 ++++++++-- 2 files changed, 12 insertions(+), 3 deletions(-)
drivers/net/ethernet/microsoft/mana/gdma_main.c +4 -1
@@ -300,8 +300,11 @@ static void mana_gd_ring_doorbell(struct gdma_context *gc, u32 db_index,void mana_gd_wq_ring_doorbell (struct gdma_context *gc,struct gdma_queue *queue) { +/* BNIC Spec specifies that client should set 0 for rq.wqe_cnt +* This value is not used in sq +*/ mana_gd_ring_doorbell (gc, queue->gdma_dev->doorbell, queue->type, - queue->id, queue->head *GDMA_WQE_BU_SIZE ,1 ); + queue->id, queue->head *GDMA_WQE_BU_SIZE ,0 ); }void mana_gd_ring_cq (struct gdma_queue *cq,u8 arm_bit)
drivers/net/ethernet/microsoft/mana/mana_en.c +8 -2
@@ -1383,8 +1383,8 @@ static void mana_post_pkt_rxq(struct mana_rxq *rxq) recv_buf_oob = &rxq->rx_oobs[curr_index]; - err =mana_gd_post_and_ring (rxq->gdma_rq, &recv_buf_oob->wqe_req, - &recv_buf_oob->wqe_inf); + err =mana_gd_post_work_request (rxq->gdma_rq, &recv_buf_oob->wqe_req, + &recv_buf_oob->wqe_inf);if (WARN_ON_ONCE (err))return ;
@@ -1654,6 +1654,12 @@ static void mana_poll_rx_cq(struct mana_cq *cq)mana_process_rx_cqe (rxq, cq, &comp[i]); } +if (comp_read >0 ) { +struct gdma_context *gc = rxq->gdma_rq->gdma_dev->gdma_context; + +mana_gd_wq_ring_doorbell (gc, rxq->gdma_rq); + } +if (rxq->xdp_flush)xdp_do_flush (); }
Signature · 1 line
2.34.1
Quoted 25 lines
-----Original Message-----
From: [email protected] <[email protected]>
Sent: Thursday, June 22, 2023 12:23 PM
To: Jason Gunthorpe <[email protected]>; Leon Romanovsky <[email protected]>; Ajay Sharma <[email protected]>; Dexuan Cui <[email protected]>; KY Srinivasan <[email protected]>; Haiyang Zhang <[email protected]>; Wei Liu <[email protected]>; David S. Miller <[email protected]>; Eric Dumazet <[email protected]>; Jakub Kicinski <[email protected]>; Paolo Abeni <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Long Li <[email protected]>; [email protected]
Subject: [PATCH v2] net: mana: Batch ringing RX queue doorbell on receiving packetsFrom: Long Li <[email protected]>
It's inefficient to ring the doorbell page every time a WQE is posted to the received queue.
Move the code for ringing doorbell page to where after we have posted all WQEs to the receive queue during a callback from
napi_poll().
Quoted 6 lines
Tests showed no regression in network latency benchmarks.
Fixes ca9c54d2d6a5 ("net: mana: Add a driver for Microsoft Azure NetworkAdapter (MANA)")
Signed-off-by Long Li <[email protected]>
Quoted 10 lines
From: [email protected]
Sent: Thursday, June 22, 2023 9:23 AM
...It's inefficient to ring the doorbell page every time a WQE is posted to the received queue.
Move the code for ringing doorbell page to where after we have posted all WQEs to the receive queue during a callback from
napi_poll().
Quoted 6 lines
Tests showed no regression in network latency benchmarks.
Fixes ca9c54d2d6a5 ("net: mana: Add a driver for Microsoft Azure NetworkAdapter (MANA)")
Signed-off-by Long Li <[email protected]>
Jakub Kicinski
maintainerQuoted 10 lines · On Thu, 22 Jun 2023 09:22:36 -0700 [email protected] wrote:
It's inefficient to ring the doorbell page every time a WQE is posted to the received queue.
Move the code for ringing doorbell page to where after we have posted all WQEs to the receive queue during a callback from
napi_poll().Tests showed no regression in network latency benchmarks.
Fixes ca9c54d2d6a5 ("net: mana: Add a driver for Microsoft Azure Network Adapter (MANA)")
If this is supposed to be a fix, you need to clearly explain what the performance loss was, so that backporters can make an informed decision.
Quoted 9 lines
drivers/net/ethernet/microsoft/mana/gdma_main.c | 5 ++++- drivers/net/ethernet/microsoft/mana/mana_en.c | 10 ++++++++-- 2 files changed, 12 insertions(+), 3 deletions(-)drivers/net/ethernet/microsoft/mana/gdma_main.c +0
-0
drivers/net/ethernet/microsoft/mana/gdma_main.c static void mana_gd_ring_doorbell(struct gdma_context *gc, u32 db_index, · line 308
Quoted 10 lines
drivers/net/ethernet/microsoft/mana/gdma_main.c +4
-1@@ -300,8 +300,11 @@ static void mana_gd_ring_doorbell(struct gdma_context *gc, u32 db_index,void mana_gd_wq_ring_doorbell (struct gdma_context *gc,struct gdma_queue *queue) { +/* BNIC Spec specifies that client should set 0 for rq.wqe_cnt +* This value is not used in sq +*/ mana_gd_ring_doorbell (gc, queue->gdma_dev->doorbell, queue->type, - queue->id, queue->head *GDMA_WQE_BU_SIZE ,1 ); + queue->id, queue->head *GDMA_WQE_BU_SIZE ,0 ); }
This change needs to be explained in the commit message, or should be a separate patch.
Signature · 1 line
pw-bot: cr
Quoted 10 lines
Subject: Re: [PATCH v2] net: mana: Batch ringing RX queue doorbell on receiving packets
Quoted 6 lines · On Thu, 22 Jun 2023 09:22:36 -0700 [email protected] wrote:
It's inefficient to ring the doorbell page every time a WQE is posted to the received queue.
Move the code for ringing doorbell page to where after we have posted all WQEs to the receive queue during a callback from
napi_poll().
Quoted 8 lines
Quoted 5 lines
Tests showed no regression in network latency benchmarks.
Fixes ca9c54d2d6a5 ("net: mana: Add a driver for Microsoft AzureNetwork Adapter (MANA)")
If this is supposed to be a fix, you need to clearly explain what the performance loss was, so that backporters can make an informed decision.
I'm sending v3 to include the details.
Quoted 22 lines
Quoted 11 lines
drivers/net/ethernet/microsoft/mana/gdma_main.c | 5 ++++- drivers/net/ethernet/microsoft/mana/mana_en.c | 10 ++++++++-- 2 files changed, 12 insertions(+), 3 deletions(-)drivers/net/ethernet/microsoft/mana/gdma_main.c +0
-0drivers/net/ethernet/microsoft/mana/gdma_main.c static void mana_gd_ring_doorbell(struct · line 306
Quoted 10 lines
drivers/net/ethernet/microsoft/mana/gdma_main.c +3
-1@@ -300,8 +300,11 @@ static void mana_gd_ring_doorbell(struct gdma_context *gc, u32 db_index, void mana_gd_wq_ring_doorbell(struct gdma_context *gc, struct gdma_queue *queue) { + /* BNIC Spec specifies that client should set 0 for rq.wqe_cnt + * This value is not used in sq + */ mana_gd_ring_doorbell(gc, queue->gdma_dev->doorbell, queue- type, - queue->id, queue->head * GDMA_WQE_BU_SIZE,1);
Quoted 6 lines
drivers/net/ethernet/microsoft/mana/gdma_main.c static void mana_gd_ring_doorbell(struct · line 309
Quoted 1 line
drivers/net/ethernet/microsoft/mana/gdma_main.c +1
-0+ queue->id, queue->head * GDMA_WQE_BU_SIZE,0);
drivers/net/ethernet/microsoft/mana/gdma_main.c static void mana_gd_ring_doorbell(struct · line 310
Quoted 1 line
drivers/net/ethernet/microsoft/mana/gdma_main.c +0
-0}This change needs to be explained in the commit message, or should be a separate patch.
Will explain the change in the commit message.
Thanks,
Long
Quoted 2 lines
--
pw-bot: cr