net/gve: out of order completion processing for DQO
cover net/gve: out of order completion processing for DQO
Both RX and TX processing on DQ were originally implemented with the asusmption that descriptor completions will be written by the hardware in the order that they are posted. In certain cases, such as RSC on RX and double completions on TX, this will not necessarily be the case.
Depends-on: series-35656 ("net/gve: Tx datapath fixes for GVE DQO")
Joshua Washington (4):
net/gve: free Rx mbufs if allocation fails on ring setup net/gve: add datapath-specific logging for gve net/gve: support for out of order completions on DQ Tx net/gve: support for out of order completions on DQ Rx
drivers/net/gve/base/gve_adminq.c | 2 +- drivers/net/gve/gve_ethdev.h | 20 ++- drivers/net/gve/gve_logs.h | 3 + drivers/net/gve/gve_rx_dqo.c | 135 +++++++++++----- drivers/net/gve/gve_tx_dqo.c | 250 ++++++++++++++++++------------ 5 files changed, 267 insertions(+), 143 deletions(-)
Signature · 1 line
2.51.0.rc1.167.g924127e9c0-goog
PATCH 1/4 net/gve: free Rx mbufs if allocation fails on ring setup
When creating new RX rings, one less than the number of buffers in the ring need to be allocated. It is possible that only a part of the allocation is successful, resulting in a failure to create the rings. In this case, the driver should free the buffers which were successfully allocated to avoid a memory leak in case the application does not automatically exit.
--- drivers/net/gve/gve_rx_dqo.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-)
drivers/net/gve/gve_rx_dqo.c +5 -6
@@ -376,14 +376,13 @@ gve_rxq_mbufs_alloc_dqo(struct gve_rx_queue *rxq) rxq->stats.no_mbufs_bulk++;for (i =0 ; i < rx_mask; i++) { nmb =rte_pktmbuf_alloc (rxq->mpool); -if (!nmb) -break ; +if (!nmb) { + rxq->stats.no_mbufs++; +gve_release_rxq_mbufs_dqo (rxq); +return -ENOMEM ; + } rxq->sw_ring[i] = nmb; } -if (i < rxq->nb_rx_desc -1 ) { - rxq->stats.no_mbufs += rx_mask - i; -return -ENOMEM ; - } }for (i =0 ; i < rx_mask; i++) {
Signature · 1 line
2.51.0.rc1.167.g924127e9c0-goog
PATCH 2/4 net/gve: add datapath-specific logging for gve
In reply to Joshua WashingtonThis would allow for separate control of datapath-related logs vs non-datapath logs.
--- drivers/net/gve/gve_logs.h | 3 +++ drivers/net/gve/gve_rx_dqo.c | 5 +++-- drivers/net/gve/gve_tx_dqo.c | 11 ++++++----- 3 files changed, 12 insertions(+), 7 deletions(-)
drivers/net/gve/gve_logs.h +3 -0
@@ -11,4 +11,7 @@ extern int gve_logtype_driver;#define PMD_DRV_LOG (level, ...) \ RTE_LOG_LINE_PREFIX(level, GVE_DRIVER, "%s(): ", __func__, __VA_ARGS__) +#define PMD_DRV_DP_LOG (level, ...) \ + RTE_LOG_DP_LINE_PREFIX(level, GVE_DRIVER, "%s(): ", __func__, __VA_ARGS__) +#endif
drivers/net/gve/gve_rx_dqo.c +3 -2
@@ -27,8 +27,9 @@ gve_rx_refill_dqo(struct gve_rx_queue *rxq) rxq->stats.no_mbufs += nb_refill; dev = &rte_eth_devices[rxq->port_id]; dev->data->rx_mbuf_alloc_failed += nb_refill; -PMD_DRV_LOG (DEBUG ,"RX mbuf alloc failed port_id=%u queue_id=%u" , - rxq->port_id, rxq->queue_id); +PMD_DRV_DP_LOG (DEBUG , +"RX mbuf alloc failed port_id=%u queue_id=%u" , + rxq->port_id, rxq->queue_id);return ; }
drivers/net/gve/gve_tx_dqo.c +6 -5
@@ -42,7 +42,7 @@ gve_tx_clean_dqo(struct gve_tx_queue *txq) aim_txq->last_desc_cleaned = compl_tag;break ;case GVE_COMPL_TYPE_DQO_REINJECTION : -PMD_DRV_LOG (DEBUG ,"GVE_COMPL_TYPE_DQO_REINJECTION !!!" ); +PMD_DRV_DP_LOG (DEBUG ,"GVE_COMPL_TYPE_DQO_REINJECTION !!!" );/* FALLTHROUGH */ case GVE_COMPL_TYPE_DQO_PKT :/* free all segments. */
@@ -58,10 +58,10 @@ gve_tx_clean_dqo(struct gve_tx_queue *txq)break ;case GVE_COMPL_TYPE_DQO_MISS :rte_delay_us_sleep (1 ); -PMD_DRV_LOG (DEBUG ,"GVE_COMPL_TYPE_DQO_MISS ignored !!!" ); +PMD_DRV_DP_LOG (DEBUG ,"GVE_COMPL_TYPE_DQO_MISS ignored !!!" );break ;default : -PMD_DRV_LOG (ERR ,"unknown completion type." ); +PMD_DRV_DP_LOG (ERR ,"unknown completion type." );return ; }
@@ -206,7 +206,7 @@ gve_tx_burst_dqo(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)if (rte_mbuf_check (tx_pkt, true, &reason)) { -PMD_DRV_LOG (DEBUG ,"Invalid mbuf: %s" , reason); +PMD_DRV_DP_LOG (DEBUG ,"Invalid mbuf: %s" , reason);break ; }
@@ -243,7 +243,8 @@ gve_tx_burst_dqo(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts) do { if (sw_ring[sw_id] != NULL) - PMD_DRV_LOG(DEBUG, "Overwriting an entry in sw_ring"); + PMD_DRV_DP_LOG(DEBUG, + "Overwriting an entry in sw_ring"); /* Skip writing descriptor if mbuf has no data. */ if (!tx_pkt->data_len)
Signature · 1 line
2.51.0.rc1.167.g924127e9c0-goog
PATCH 3/4 net/gve: support for out of order completions on DQ Tx
In reply to Joshua WashingtonPrior to this patch, the DQ format made use of two separate arrays which dictate the TX submission queue, named tx_ring and sw_ring. The tx_ring represented the actual hardware descriptor ring, while the sw_ring represented a ring of DMA'd buffers to be used in transmission.
Given that the DQ format makes use of out-of-order completions, the use of a queue for keeping track of buffers DMA'd to the hardware is sub-optimal at best, and error-prone in the worst case. Because completions can come out of order, it was possible to hit a case where the next buffer to be posted in the submission ring to have not yet gotten a completion. If this is the case, the driver would have to pause transmission until the completion is received or risk leaving a pointer unfreed. In this sense, to prevent memory leaks, the driver must act as if the completions come in-order, sometimes pausing traffic even though there are buffers ready to be used, which can lead to degraded performance.
To combat this, the GVE driver made a 4x multiplier for the sw_ring size, relative to the size of the hardware ring. However, this mitigation had no true guarantee against memory leaks, and should not have been relied on for a correctly behaving driver.
DQ completions happen at a packet granularity, not a buffer granularity. As such, this patch introduces a new pkt_ring_dqo struct, allowing multidescriptor packets not to take up as many entries in the "sw_ring". Additionally, new free list stack is introduced to obtain available completion tags, allowing for packet completions to be properly processed out of order. Where the sw_ring would have at once been overrun, there is a new limitation on the number of available completion tags, which aligns with the number of possible outstanding packets.
--- drivers/net/gve/base/gve_adminq.c | 2 +- drivers/net/gve/gve_ethdev.h | 16 +- drivers/net/gve/gve_tx_dqo.c | 243 ++++++++++++++++++------------ 3 files changed, 160 insertions(+), 101 deletions(-)
drivers/net/gve/base/gve_adminq.c +1 -1
@@ -582,7 +582,7 @@ static int gve_adminq_create_tx_queue(struct gve_priv *priv, u32 queue_index) cmd.create_tx_queue.tx_comp_ring_addr =cpu_to_be64 (txq->compl_ring_phys_addr); cmd.create_tx_queue.tx_comp_ring_size = -cpu_to_be16 (txq->sw_size); +cpu_to_be16 (txq->nb_complq_desc); }return gve_adminq_issue_cmd (priv, &cmd);
drivers/net/gve/gve_ethdev.h +15 -1
@@ -119,16 +119,25 @@ struct gve_xstats_name_offset {unsigned int offset; }; +struct gve_tx_pkt { +struct rte_mbuf *mbuf; +int16_t next_avail_pkt;/* Entry in software ring for next free packet slot */ +}; +struct gve_tx_queue {volatile union gve_tx_desc *tx_desc_ring;const struct rte_memzone *mz;uint64_t tx_ring_phys_addr; -struct rte_mbuf **sw_ring; +union { +struct rte_mbuf **sw_ring; +struct gve_tx_pkt *pkt_ring_dqo; + };volatile rte_be32_t *qtx_tail;volatile rte_be32_t *qtx_head;uint32_t tx_tail;uint16_t nb_tx_desc; +uint16_t nb_complq_desc;uint16_t nb_free;uint16_t nb_used;uint32_t next_to_clean;
@@ -162,6 +171,11 @@ struct gve_tx_queue {/* newly added for DQO */ volatile union gve_tx_desc_dqo *tx_ring;struct gve_tx_compl_desc *compl_ring; + +/* List of free completion tags that map into sw_ring. */ +int16_t free_compl_tags_head; +uint16_t num_free_compl_tags; +const struct rte_memzone *compl_ring_mz;uint64_t compl_ring_phys_addr;uint32_t complq_tail;
drivers/net/gve/gve_tx_dqo.c +144 -99
424 lines not on this page · raw message
Signature · 1 line
2.51.0.rc1.167.g924127e9c0-goog
PATCH 4/4 net/gve: support for out of order completions on DQ Rx
In reply to Joshua WashingtonThe DPDK DQ driver made an implicit assumption that buffers will be returned from the NIC in the order which they are posted. While this is generally the case, there are certain edge cases, such as HW GRO, where buffers may be completed out of order. In such cases, the driver would have returned the wrong mbuf to the program, which would cause issues if the mbuf is freed while the NIC still expects to own the buffer or the application receives a buffer which contains garbage data.
This patch updates the RX path behavior more in line with what is expected by the NIC by utilizing the buf_id in the completion descriptor to decide which buffer to complete. A stack is introduced to hold the available buf_ids when posting to handle the fact that buf_ids can come back from the hardware out of order.
--- drivers/net/gve/gve_ethdev.h | 4 ++ drivers/net/gve/gve_rx_dqo.c | 119 ++++++++++++++++++++++++++--------- 2 files changed, 92 insertions(+), 31 deletions(-)
drivers/net/gve/gve_ethdev.h +4 -0
@@ -242,6 +242,10 @@ struct gve_rx_queue {uint8_t cur_gen_bit;uint16_t bufq_tail; +/* List of buffers which are known to be completed by the hardware. */ +int16_t *completed_buf_list; +int16_t completed_buf_list_head; +/* Only valid for DQO_RDA queue format */ struct gve_rx_queue *bufq;
drivers/net/gve/gve_rx_dqo.c +88 -31
235 lines not on this page · raw message
Signature · 1 line
2.51.0.rc1.167.g924127e9c0-goog
Stephen Hemminger
AppliedQuoted 13 lines · On Mon, 25 Aug 2025 17:03:33 -0700 Joshua Washington <[email protected]>…
Both RX and TX processing on DQ were originally implemented with the asusmption that descriptor completions will be written by the hardware in the order that they are posted. In certain cases, such as RSC on RX and double completions on TX, this will not necessarily be the case.
Depends-on: series-35656 ("net/gve: Tx datapath fixes for GVE DQO")
Joshua Washington (4):
net/gve: free Rx mbufs if allocation fails on ring setup net/gve: add datapath-specific logging for gve net/gve: support for out of order completions on DQ Tx net/gve: support for out of order completions on DQ Rx
Quoted 7 lines
drivers/net/gve/base/gve_adminq.c | 2 +- drivers/net/gve/gve_ethdev.h | 20 ++- drivers/net/gve/gve_logs.h | 3 + drivers/net/gve/gve_rx_dqo.c | 135 +++++++++++----- drivers/net/gve/gve_tx_dqo.c | 250 ++++++++++++++++++------------ 5 files changed, 267 insertions(+), 143 deletions(-)
Queued to next-net