net: mana: Add Interrupt Moderation support
Subsystems (5)
PATCH v4 1/1 net: mana: Add Interrupt Moderation support
From: Haiyang Zhang <[email protected]>
Add Static and Dynamic Interrupt Moderation (DIM) support for Rx and Tx.
Update queue creation procedure with new data struct with the related settings.
Add functions to collect stat for DIM, and workers to update DIM data and settings.
Update ethtool handler to get/set the moderation settings from a user. To avoid detach/re-attach ops, ring DIM doorbell to change settings at run time.
By default, adaptive-rx/tx (DIM) are enabled if supported by HW.
---
v4:
Fixed tx stat, concurrency, and mb issues from Simon's review.
v3:
Updated to avoid detach/re-attach ops as suggested by Paolo.
v2:
Updated with comments from Jedrzej.
--- drivers/net/ethernet/microsoft/Kconfig | 1 + .../net/ethernet/microsoft/mana/gdma_main.c | 29 +++ drivers/net/ethernet/microsoft/mana/mana_en.c | 171 ++++++++++++++++++ .../ethernet/microsoft/mana/mana_ethtool.c | 167 ++++++++++++++++- include/net/mana/gdma.h | 24 ++- include/net/mana/mana.h | 54 ++++++ 6 files changed, 437 insertions(+), 9 deletions(-)
drivers/net/ethernet/microsoft/Kconfig +1 -0
@@ -21,6 +21,7 @@ config MICROSOFT_MANA depends on X86_64 || (ARM64 && !CPU_BIG_ENDIAN) depends on PCI_HYPERV select AUXILIARY_BUS + select DIMLIB select PAGE_POOL select NET_SHAPER help
drivers/net/ethernet/microsoft/mana/gdma_main.c +29 -0
@@ -1,6 +1,7 @@// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause /* Copyright (c) 2021, Microsoft Corporation. */ +#include <linux/bitfield.h> #include <linux/debugfs.h> #include <linux/module.h> #include <linux/pci.h>
@@ -466,6 +467,7 @@ static int mana_gd_disable_queue(struct gdma_queue *queue)#define DOORBELL_OFFSET_RQ 0x400#define DOORBELL_OFFSET_CQ 0x800#define DOORBELL_OFFSET_EQ 0xFF8 +#define DOORBELL_OFFSET_DIM 0x820 static void mana_gd_ring_doorbell(struct gdma_context *gc, u32 db_index, enum gdma_queue_type q_type, u32 qid,
@@ -506,6 +508,16 @@ static void mana_gd_ring_doorbell(struct gdma_context *gc, u32 db_index, addr +=DOORBELL_OFFSET_SQ ;break ; +case GDMA_DIM : + e.dim.id = qid; + e.dim.mod_usec =FIELD_GET (MANA_INTR_MODR_USEC_MAX , tail_ptr); + e.dim.mod_usec_vld = !!(tail_ptr &MANA_INTR_MODR_USEC_VLD ); + e.dim.mod_comps =FIELD_GET (MANA_INTR_MODR_COMP_MASK , tail_ptr); + e.dim.mod_comps_vld = num_req; + + addr +=DOORBELL_OFFSET_DIM ; +break ; +default :WARN_ON (1 );return ;
@@ -540,6 +552,23 @@ void mana_gd_ring_cq(struct gdma_queue *cq, u8 arm_bit) }EXPORT_SYMBOL_NS (mana_gd_ring_cq,"NET_MANA" ); +void mana_gd_ring_dim (struct gdma_queue *cq,u32 mod_usec,bool mod_usec_vld, +u32 mod_comps,bool mod_comps_vld) +{ +struct gdma_context *gc = cq->gdma_dev->gdma_context; +u32 dim_val; + +/* Convert the DIM values to doorbell parameters */ + dim_val =FIELD_PREP (MANA_INTR_MODR_USEC_MAX , mod_usec) | +FIELD_PREP (MANA_INTR_MODR_COMP_MASK , mod_comps); +if (mod_usec_vld) + dim_val |=MANA_INTR_MODR_USEC_VLD ; + +mana_gd_ring_doorbell (gc, cq->gdma_dev->doorbell,GDMA_DIM , cq->id, + dim_val, mod_comps_vld); +} +EXPORT_SYMBOL_NS (mana_gd_ring_dim,"NET_MANA" ); +#define MANA_SERVICE_PERIOD 10static void mana_serv_rescan (struct pci_dev *pdev)
drivers/net/ethernet/microsoft/mana/mana_en.c +171 -0
261 lines not on this page · raw message
drivers/net/ethernet/microsoft/mana/mana_ethtool.c +159 -8
199 lines not on this page · raw message
include/net/mana/gdma.h +23 -1
@@ -47,6 +47,7 @@ enum gdma_queue_type {GDMA_RQ ,GDMA_CQ ,GDMA_EQ , + GDMA_DIM, }; enum gdma_work_request_flags {
@@ -126,6 +127,17 @@ union gdma_doorbell_entry { u64 tail_ptr : 31; u64 arm : 1; } eq; + +struct { +u64 id :24 ; +u64 reserved :8 ; +u64 mod_usec :10 ; +u64 reserve1 :5 ; +u64 mod_usec_vld :1 ; +u64 mod_comps :8 ; +u64 reserve2 :7 ; +u64 mod_comps_vld:1 ; + } dim; };/* HW DATA */ struct gdma_msg_hdr {
@@ -502,6 +514,9 @@ void mana_gd_ring_cq(struct gdma_queue *cq, u8 arm_bit);int mana_schedule_serv_work (struct gdma_context *gc,enum gdma_eqe_type type); +void mana_gd_ring_dim (struct gdma_queue *cq,u32 mod_usec,bool mod_usec_vld, +u32 mod_comps,bool mod_comps_vld); +struct gdma_wqe {u32 reserved :24 ;u32 last_vbytes :8 ;
@@ -650,6 +665,9 @@ enum {/* Driver supports self recovery on Hardware Channel timeouts */ #define GDMA_DRV_CAP_FLAG_1_HWC_TIMEOUT_RECOVERY BIT(25) +/* Driver supports dynamic interrupt moderation - DIM */ +#define GDMA_DRV_CAP_FLAG_1_DYN_INTERRUPT_MODERATION BIT(28) +#define GDMA_DRV_CAP_FLAGS1 \ (GDMA_DRV_CAP_FLAG_1_EQ_SHARING_MULTI_VPORT | \ GDMA_DRV_CAP_FLAG_1_NAPI_WKDONE_FIX | \
@@ -665,7 +683,8 @@ enum {GDMA_DRV_CAP_FLAG_1_PROBE_RECOVERY | \GDMA_DRV_CAP_FLAG_1_HANDLE_STALL_SQ_RECOVERY | \GDMA_DRV_CAP_FLAG_1_HWC_TIMEOUT_RECOVERY | \ - GDMA_DRV_CAP_FLAG_1_EQ_MSI_UNSHARE_MULTI_VPORT) +GDMA_DRV_CAP_FLAG_1_EQ_MSI_UNSHARE_MULTI_VPORT | \ + GDMA_DRV_CAP_FLAG_1_DYN_INTERRUPT_MODERATION) #define GDMA_DRV_CAP_FLAGS2 0
@@ -701,6 +720,9 @@ struct gdma_verify_ver_req {u8 os_ver_str4[128 ]; };/* HW DATA */ +/* HW supports dynamic interrupt moderation - DIM */ +#define GDMA_PF_CAP_FLAG_1_DYN_INTERRUPT_MODERATION BIT(15) +struct gdma_verify_ver_resp {struct gdma_resp_hdr hdr;u64 gdma_protocol_ver;
include/net/mana/mana.h +54 -0
102 lines not on this page · raw message
Signature · 1 line
2.34.1
Paolo Abeni
maintainerdrivers/net/ethernet/microsoft/mana/mana_en.c int mana_create_wq_obj(struct mana_port_context *apc, · line 1596
Quoted 11 lines · Haiyang Zhang
drivers/net/ethernet/microsoft/mana/mana_en.c +3
-0@@ -1591,6 +1591,9 @@ int mana_create_wq_obj(struct mana_port_context *apc,mana_gd_init_req_hdr (&req.hdr,MANA_CREATE_WQ_OBJ ,sizeof (req),sizeof (resp)); + + req.hdr.req.msg_version =GDMA_MESSAGE_V3 ; + req.hdr.resp.msg_version =GDMA_MESSAGE_V2 ;
Double checking the above is intentional; it feels strange to me that request and reply use different versions. Possibly a comment for future memory would make sense.
Quoted 76 lines
drivers/net/ethernet/microsoft/mana/mana_en.c +41
-0req.vport = vport; req.wq_type = wq_type; req.wq_gdma_region = wq_spec->gdma_region;@@ -1599,6 +1602,9 @@ int mana_create_wq_obj(struct mana_port_context *apc, req.cq_size = cq_spec->queue_size; req.cq_moderation_ctx_id = cq_spec->modr_ctx_id; req.cq_parent_qid = cq_spec->attached_eq; + req.req_cq_moderation = cq_spec->req_cq_moderation; + req.cq_moderation_comp = cq_spec->cq_moderation_comp; + req.cq_moderation_usec = cq_spec->cq_moderation_usec; err =mana_send_request (apc->ac, &req,sizeof (req), &resp,sizeof (resp));@@ -1856,6 +1862,7 @@ static void mana_poll_tx_cq(struct mana_cq *cq)struct gdma_posted_wqe_info *wqe_info;unsigned int pkt_transmitted =0 ;unsigned int wqe_unit_cnt =0 ; +unsigned int tx_bytes =0 ;struct mana_txq *txq = cq->txq;struct mana_port_context *apc;struct netdev_queue *net_txq;@@ -1937,6 +1944,8 @@ static void mana_poll_tx_cq(struct mana_cq *cq)mana_unmap_skb (skb, apc); + tx_bytes += skb->len; +napi_consume_skb (skb, cq->budget); pkt_transmitted++;@@ -1967,6 +1976,10 @@ static void mana_poll_tx_cq(struct mana_cq *cq)if (atomic_sub_return (pkt_transmitted, &txq->pending_sends) <0 )WARN_ON_ONCE (1 ); +/* Feed DIM with the completion rate observed here, in NAPI context. */ + cq->tx_dim_pkts += pkt_transmitted; + cq->tx_dim_bytes += tx_bytes; + cq->work_done = pkt_transmitted; }@@ -2318,6 +2331,119 @@ static void mana_poll_rx_cq(struct mana_cq *cq)xdp_do_flush (); } +static void mana_rx_dim_work (struct work_struct *work) +{ + struct dim *dim = container_of(work, struct dim, work); +struct dim_cq_moder cur_moder; +struct mana_cq *cq; + + cur_moder =net_dim_get_rx_moderation (dim->mode, dim->profile_ix); + cq = container_of(dim, struct mana_cq, dim); + + cur_moder.usec =min_t (u16, cur_moder.usec,MANA_INTR_MODR_USEC_MAX ); + cur_moder.pkts =min_t (u16, cur_moder.pkts,MANA_INTR_MODR_COMP_MAX ); + +mana_gd_ring_dim (cq->gdma_cq, cur_moder.usec, true, + cur_moder.pkts, true); + + dim->state =DIM_START_MEASURE ; +} + +static void mana_tx_dim_work (struct work_struct *work) +{ + struct dim *dim = container_of(work, struct dim, work); +struct dim_cq_moder cur_moder; +struct mana_cq *cq; + + cur_moder =net_dim_get_tx_moderation (dim->mode, dim->profile_ix); + cq = container_of(dim, struct mana_cq, dim); + + cur_moder.usec =min_t (u16, cur_moder.usec,MANA_INTR_MODR_USEC_MAX ); + cur_moder.pkts =min_t (u16, cur_moder.pkts,MANA_INTR_MODR_COMP_MAX ); + + mana_gd_ring_dim(cq->gdma_cq, cur_moder.usec, true,
drivers/net/ethernet/microsoft/mana/mana_en.c static void mana_poll_rx_cq(struct mana_cq *cq) · line 2374
Quoted 10 lines
drivers/net/ethernet/microsoft/mana/mana_en.c +10
-0+ cur_moder.pkts, true); + + dim->state =DIM_START_MEASURE ; +} + +/* The caller must update apc->rx/tx_dim_enabled before disabling and +* after enabling. And synchronize_net() before draining the DIM work, +* so that NAPI cannot observe a stale flag. +*/ +int mana_dim_change (struct mana_cq *cq,bool enable)
This always return 0, and the return value is not checked by the callers; return type should likelly changed to void
/P
Haiyang Zhang
maintainerQuoted 30 lines
-----Original Message-----
From: Paolo Abeni <[email protected]>
Sent: Thursday, July 2, 2026 4:57 AM
To: Haiyang Zhang <[email protected]>; linux- [email protected]; [email protected]; KY Srinivasan <[email protected]>; Haiyang Zhang <[email protected]>; Wei Liu <[email protected]>; Dexuan Cui <[email protected]>; Long Li <[email protected]>; Andrew Lunn <[email protected]>; David S. Miller <[email protected]>; Eric Dumazet <[email protected]>; Jakub Kicinski <[email protected]>; Konstantin Taranov <[email protected]>; Simon Horman <[email protected]>; Erni Sri Satya Vennela <[email protected]>; Dipayaan Roy <[email protected]>; Aditya Garg <[email protected]>; Breno Leitao <[email protected]>; linux- [email protected]; [email protected]
Cc: Paul Rosswurm <[email protected]>
Subject: [EXTERNAL] Re: [PATCH net-next v4] net: mana: Add Interrupt Moderation supportReview
Quoted 1 line · Haiyang Zhang
(quoted diff) +0
-0
b/drivers/net/ethernet/microsoft/mana/mana_en.cdrivers/net/ethernet/microsoft/mana/mana_en.c int mana_create_wq_obj(struct mana_port_context
Quoted 4 lines
index
7438ea6b3f26..9391e9564605100644drivers/net/ethernet/microsoft/mana/mana_en.c +0
-0@@ -1591,6 +1591,9 @@ int mana_create_wq_obj(struct mana_port_context*apc,
Quoted 1 line
Quoted 9 lines
drivers/net/ethernet/microsoft/mana/mana_en.c int mana_create_wq_obj(struct mana_port_context *apc, · line 1596
Quoted 5 lines
drivers/net/ethernet/microsoft/mana/mana_en.c +3
-0mana_gd_init_req_hdr (&req.hdr,MANA_CREATE_WQ_OBJ ,sizeof (req),sizeof (resp)); + + req.hdr.req.msg_version =GDMA_MESSAGE_V3 ; + req.hdr.resp.msg_version =GDMA_MESSAGE_V2 ;Double checking the above is intentional; it feels strange to me that request and reply use different versions. Possibly a comment for future memory would make sense.
Yes, it's intentional. The request and reply versions can be different. I will add comments.
Quoted 9 lines
drivers/net/ethernet/microsoft/mana/mana_en.c static void mana_poll_rx_cq(struct mana_cq *cq) · line 2374
Quoted 6 lines
drivers/net/ethernet/microsoft/mana/mana_en.c +6
-0+ +/* The caller must update apc->rx/tx_dim_enabled before disabling and +* after enabling. And synchronize_net() before draining the DIM work, +* so that NAPI cannot observe a stale flag. +*/ +int mana_dim_change (struct mana_cq *cq,bool enable)This always return 0, and the return value is not checked by the callers; return type should likelly changed to void
Will update.
Thanks,
- Haiyang