Skip to contentSkip to conversation

net/gve: out of order completion processing for DQO

PATCH Applied Joshua Washington 2 participants 6 messages Aug 26, 2025 Network drivers

lore
Share & tools
lore
#0 lore raw

cover net/gve: out of order completion processing for DQO

Both RX and TX processing on DQ were originally implemented with the asusmption that descriptor completions will be written by the hardware in the order that they are posted. In certain cases, such as RSC on RX and double completions on TX, this will not necessarily be the case.

Depends-on: series-35656 ("net/gve: Tx datapath fixes for GVE DQO")

Joshua Washington (4):
net/gve: free Rx mbufs if allocation fails on ring setup net/gve: add datapath-specific logging for gve net/gve: support for out of order completions on DQ Tx net/gve: support for out of order completions on DQ Rx

 drivers/net/gve/base/gve_adminq.c |   2 +-
 drivers/net/gve/gve_ethdev.h      |  20 ++-
 drivers/net/gve/gve_logs.h        |   3 +
 drivers/net/gve/gve_rx_dqo.c      | 135 +++++++++++-----
 drivers/net/gve/gve_tx_dqo.c      | 250 ++++++++++++++++++------------
 5 files changed, 267 insertions(+), 143 deletions(-)
Signature · 1 line
2.51.0.rc1.167.g924127e9c0-goog
#1 lore raw

PATCH 1/4 net/gve: free Rx mbufs if allocation fails on ring setup

When creating new RX rings, one less than the number of buffers in the ring need to be allocated. It is possible that only a part of the allocation is successful, resulting in a failure to create the rings. In this case, the driver should free the buffers which were successfully allocated to avoid a memory leak in case the application does not automatically exit.

Fixes 265daac8a53a ("net/gve: fix mbuf allocation memory leak for DQ Rx")
---
 drivers/net/gve/gve_rx_dqo.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)
drivers/net/gve/gve_rx_dqo.c +5 -6
index 0103add985..cd85d90bb6 100644
@@ -376,14 +376,13 @@ gve_rxq_mbufs_alloc_dqo(struct gve_rx_queue *rxq)
 		rxq->stats.no_mbufs_bulk++;
 		for (i = 0; i < rx_mask; i++) {
 			nmb = rte_pktmbuf_alloc(rxq->mpool);
-			if (!nmb)
-				break;
+			if (!nmb) {
+				rxq->stats.no_mbufs++;
+				gve_release_rxq_mbufs_dqo(rxq);
+				return -ENOMEM;
+			}
 			rxq->sw_ring[i] = nmb;
 		}
-		if (i < rxq->nb_rx_desc - 1) {
-			rxq->stats.no_mbufs += rx_mask - i;
-			return -ENOMEM;
-		}
 	}
 
 	for (i = 0; i < rx_mask; i++) {
Signature · 1 line
2.51.0.rc1.167.g924127e9c0-goog
#2 lore raw

PATCH 2/4 net/gve: add datapath-specific logging for gve

In reply to Joshua Washington

This would allow for separate control of datapath-related logs vs non-datapath logs.

---
 drivers/net/gve/gve_logs.h   |  3 +++
 drivers/net/gve/gve_rx_dqo.c |  5 +++--
 drivers/net/gve/gve_tx_dqo.c | 11 ++++++-----
 3 files changed, 12 insertions(+), 7 deletions(-)
drivers/net/gve/gve_logs.h +3 -0
index a3d50fa45c..beada20e02 100644
@@ -11,4 +11,7 @@ extern int gve_logtype_driver;
 #define PMD_DRV_LOG(level, ...) \
 	RTE_LOG_LINE_PREFIX(level, GVE_DRIVER, "%s(): ", __func__, __VA_ARGS__)
 
+#define PMD_DRV_DP_LOG(level, ...) \
+	RTE_LOG_DP_LINE_PREFIX(level, GVE_DRIVER, "%s(): ", __func__,  __VA_ARGS__)
+
 #endif
drivers/net/gve/gve_rx_dqo.c +3 -2
index cd85d90bb6..ccaca1b0ea 100644
@@ -27,8 +27,9 @@ gve_rx_refill_dqo(struct gve_rx_queue *rxq)
 		rxq->stats.no_mbufs += nb_refill;
 		dev = &rte_eth_devices[rxq->port_id];
 		dev->data->rx_mbuf_alloc_failed += nb_refill;
-		PMD_DRV_LOG(DEBUG, "RX mbuf alloc failed port_id=%u queue_id=%u",
-			    rxq->port_id, rxq->queue_id);
+		PMD_DRV_DP_LOG(DEBUG,
+			       "RX mbuf alloc failed port_id=%u queue_id=%u",
+			       rxq->port_id, rxq->queue_id);
 		return;
 	}
 
drivers/net/gve/gve_tx_dqo.c +6 -5
index c36c215b94..10ef645802 100644
@@ -42,7 +42,7 @@ gve_tx_clean_dqo(struct gve_tx_queue *txq)
 		aim_txq->last_desc_cleaned = compl_tag;
 		break;
 	case GVE_COMPL_TYPE_DQO_REINJECTION:
-		PMD_DRV_LOG(DEBUG, "GVE_COMPL_TYPE_DQO_REINJECTION !!!");
+		PMD_DRV_DP_LOG(DEBUG, "GVE_COMPL_TYPE_DQO_REINJECTION !!!");
 		/* FALLTHROUGH */
 	case GVE_COMPL_TYPE_DQO_PKT:
 		/* free all segments. */
@@ -58,10 +58,10 @@ gve_tx_clean_dqo(struct gve_tx_queue *txq)
 		break;
 	case GVE_COMPL_TYPE_DQO_MISS:
 		rte_delay_us_sleep(1);
-		PMD_DRV_LOG(DEBUG, "GVE_COMPL_TYPE_DQO_MISS ignored !!!");
+		PMD_DRV_DP_LOG(DEBUG, "GVE_COMPL_TYPE_DQO_MISS ignored !!!");
 		break;
 	default:
-		PMD_DRV_LOG(ERR, "unknown completion type.");
+		PMD_DRV_DP_LOG(ERR, "unknown completion type.");
 		return;
 	}
 
@@ -206,7 +206,7 @@ gve_tx_burst_dqo(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
 
 
 		if (rte_mbuf_check(tx_pkt, true, &reason)) {
-			PMD_DRV_LOG(DEBUG, "Invalid mbuf: %s", reason);
+			PMD_DRV_DP_LOG(DEBUG, "Invalid mbuf: %s", reason);
 			break;
 		}
 
@@ -243,7 +243,8 @@ gve_tx_burst_dqo(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
 
 		do {
 			if (sw_ring[sw_id] != NULL)
-				PMD_DRV_LOG(DEBUG, "Overwriting an entry in sw_ring");
+				PMD_DRV_DP_LOG(DEBUG,
+					       "Overwriting an entry in sw_ring");
 
 			/* Skip writing descriptor if mbuf has no data. */
 			if (!tx_pkt->data_len)
Signature · 1 line
2.51.0.rc1.167.g924127e9c0-goog
#3 lore raw

PATCH 3/4 net/gve: support for out of order completions on DQ Tx

In reply to Joshua Washington

Prior to this patch, the DQ format made use of two separate arrays which dictate the TX submission queue, named tx_ring and sw_ring. The tx_ring represented the actual hardware descriptor ring, while the sw_ring represented a ring of DMA'd buffers to be used in transmission.

Given that the DQ format makes use of out-of-order completions, the use of a queue for keeping track of buffers DMA'd to the hardware is sub-optimal at best, and error-prone in the worst case. Because completions can come out of order, it was possible to hit a case where the next buffer to be posted in the submission ring to have not yet gotten a completion. If this is the case, the driver would have to pause transmission until the completion is received or risk leaving a pointer unfreed. In this sense, to prevent memory leaks, the driver must act as if the completions come in-order, sometimes pausing traffic even though there are buffers ready to be used, which can lead to degraded performance.

To combat this, the GVE driver made a 4x multiplier for the sw_ring size, relative to the size of the hardware ring. However, this mitigation had no true guarantee against memory leaks, and should not have been relied on for a correctly behaving driver.

DQ completions happen at a packet granularity, not a buffer granularity. As such, this patch introduces a new pkt_ring_dqo struct, allowing multidescriptor packets not to take up as many entries in the "sw_ring". Additionally, new free list stack is introduced to obtain available completion tags, allowing for packet completions to be properly processed out of order. Where the sw_ring would have at once been overrun, there is a new limitation on the number of available completion tags, which aligns with the number of possible outstanding packets.

---
 drivers/net/gve/base/gve_adminq.c |   2 +-
 drivers/net/gve/gve_ethdev.h      |  16 +-
 drivers/net/gve/gve_tx_dqo.c      | 243 ++++++++++++++++++------------
 3 files changed, 160 insertions(+), 101 deletions(-)
drivers/net/gve/base/gve_adminq.c +1 -1
index b1fe33080a..5d3270106c 100644
@@ -582,7 +582,7 @@ static int gve_adminq_create_tx_queue(struct gve_priv *priv, u32 queue_index)
 		cmd.create_tx_queue.tx_comp_ring_addr =
 			cpu_to_be64(txq->compl_ring_phys_addr);
 		cmd.create_tx_queue.tx_comp_ring_size =
-			cpu_to_be16(txq->sw_size);
+			cpu_to_be16(txq->nb_complq_desc);
 	}
 
 	return gve_adminq_issue_cmd(priv, &cmd);
drivers/net/gve/gve_ethdev.h +15 -1
index 195dadc4d4..b36f0ff746 100644
@@ -119,16 +119,25 @@ struct gve_xstats_name_offset {
 	unsigned int offset;
 };
 
+struct gve_tx_pkt {
+	struct rte_mbuf *mbuf;
+	int16_t next_avail_pkt; /* Entry in software ring for next free packet slot */
+};
+
 struct gve_tx_queue {
 	volatile union gve_tx_desc *tx_desc_ring;
 	const struct rte_memzone *mz;
 	uint64_t tx_ring_phys_addr;
-	struct rte_mbuf **sw_ring;
+	union {
+		struct rte_mbuf **sw_ring;
+		struct gve_tx_pkt *pkt_ring_dqo;
+	};
 	volatile rte_be32_t *qtx_tail;
 	volatile rte_be32_t *qtx_head;
 
 	uint32_t tx_tail;
 	uint16_t nb_tx_desc;
+	uint16_t nb_complq_desc;
 	uint16_t nb_free;
 	uint16_t nb_used;
 	uint32_t next_to_clean;
@@ -162,6 +171,11 @@ struct gve_tx_queue {
 	/* newly added for DQO */
 	volatile union gve_tx_desc_dqo *tx_ring;
 	struct gve_tx_compl_desc *compl_ring;
+
+	/* List of free completion tags that map into sw_ring. */
+	int16_t free_compl_tags_head;
+	uint16_t num_free_compl_tags;
+
 	const struct rte_memzone *compl_ring_mz;
 	uint64_t compl_ring_phys_addr;
 	uint32_t complq_tail;
drivers/net/gve/gve_tx_dqo.c +144 -99

424 lines not on this page · raw message

Signature · 1 line
2.51.0.rc1.167.g924127e9c0-goog
#4 lore raw

PATCH 4/4 net/gve: support for out of order completions on DQ Rx

In reply to Joshua Washington

The DPDK DQ driver made an implicit assumption that buffers will be returned from the NIC in the order which they are posted. While this is generally the case, there are certain edge cases, such as HW GRO, where buffers may be completed out of order. In such cases, the driver would have returned the wrong mbuf to the program, which would cause issues if the mbuf is freed while the NIC still expects to own the buffer or the application receives a buffer which contains garbage data.

This patch updates the RX path behavior more in line with what is expected by the NIC by utilizing the buf_id in the completion descriptor to decide which buffer to complete. A stack is introduced to hold the available buf_ids when posting to handle the fact that buf_ids can come back from the hardware out of order.

---
 drivers/net/gve/gve_ethdev.h |   4 ++
 drivers/net/gve/gve_rx_dqo.c | 119 ++++++++++++++++++++++++++---------
 2 files changed, 92 insertions(+), 31 deletions(-)
drivers/net/gve/gve_ethdev.h +4 -0
index b36f0ff746..f7cc781640 100644
@@ -242,6 +242,10 @@ struct gve_rx_queue {
 	uint8_t cur_gen_bit;
 	uint16_t bufq_tail;
 
+	/* List of buffers which are known to be completed by the hardware. */
+	int16_t *completed_buf_list;
+	int16_t completed_buf_list_head;
+
 	/* Only valid for DQO_RDA queue format */
 	struct gve_rx_queue *bufq;
 
drivers/net/gve/gve_rx_dqo.c +88 -31

235 lines not on this page · raw message

Signature · 1 line
2.51.0.rc1.167.g924127e9c0-goog
#5 lore raw
In reply to Joshua Washington
Quoted 13 lines · On Mon, 25 Aug 2025 17:03:33 -0700 Joshua Washington <[email protected]>…

Both RX and TX processing on DQ were originally implemented with the asusmption that descriptor completions will be written by the hardware in the order that they are posted. In certain cases, such as RSC on RX and double completions on TX, this will not necessarily be the case.

Depends-on: series-35656 ("net/gve: Tx datapath fixes for GVE DQO")

Joshua Washington (4):
net/gve: free Rx mbufs if allocation fails on ring setup net/gve: add datapath-specific logging for gve net/gve: support for out of order completions on DQ Tx net/gve: support for out of order completions on DQ Rx

Quoted 7 lines
 drivers/net/gve/base/gve_adminq.c |   2 +-
 drivers/net/gve/gve_ethdev.h      |  20 ++-
 drivers/net/gve/gve_logs.h        |   3 +
 drivers/net/gve/gve_rx_dqo.c      | 135 +++++++++++-----
 drivers/net/gve/gve_tx_dqo.c      | 250 ++++++++++++++++++------------
 5 files changed, 267 insertions(+), 143 deletions(-)

Queued to next-net