net/gve: advertise Rx scatter in GQI queue format

Since jumbo frame support for GQI, a frame that does not fit in one Rx buffer arrives as several descriptors with GVE_RXF_PKT_CONT set on all but the last, and gve_rx_burst() chains them into one multi-segment mbuf. The driver never reports RTE_ETH_RX_OFFLOAD_SCATTER, though, so an application that enables multi-segment receive only when the port claims it treats every packet as one segment. VPP is one: it masks its requested Rx offloads with rx_offload_capa, so on a GQI gVNIC it never sets its multi-segment flag, reads only the first buffer of a jumbo frame and drops it as malformed. With 4 KB buffers that caps the usable MTU near 4000 bytes on a VPC whose MTU is 8896.

Report SCATTER for GQI. DQO posts one buffer per packet and does not chain, so it is unchanged.

---
 .mailmap                         | 1 +
 doc/guides/nics/features/gve.ini | 1 +
 drivers/net/gve/gve_ethdev.c     | 5 +++++
 3 files changed, 7 insertions(+)

index 6d9e262cfb..87c544947c 100644
@@ -1477,6 +1477,7 @@ Ruifeng Wang <[email protected]>
 Rupesh Chiluka <[email protected]>
 Rushil Gupta <[email protected]>
 Ryan E Hall <[email protected]>
+Ryan Wolfe <[email protected]>
 Sabyasachi Sengupta <[email protected]>
 Sachin Saxena <[email protected]> <[email protected]>
 Sagar Abhang <[email protected]>

index 9f3835edf6..5566511b6b 100644
@@ -7,6 +7,7 @@
 Speed capabilities   = Y
 Queue start/stop     = Y
 MTU update           = Y
+Scattered Rx         = P
 TSO                  = Y
 RSS hash             = Y
 RSS key update       = Y

index 3c8d1a76e4..92a5a7679f 100644
@@ -897,6 +897,11 @@ gve_dev_info_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info)
 	dev_info->min_mtu = RTE_ETHER_MIN_MTU;
 
 	dev_info->rx_offload_capa = RTE_ETH_RX_OFFLOAD_RSS_HASH;
+	/* GQI receives a frame larger than one Rx buffer as a chain of
+	 * buffers (gve_rx_burst follows GVE_RXF_PKT_CONT).
+	 */
+	if (gve_is_gqi(priv))
+		dev_info->rx_offload_capa |= RTE_ETH_RX_OFFLOAD_SCATTER;
 	if (priv->clk_read_type != GVE_DEV_CLK_UNSUPPORTED &&
 	    priv->nic_ts_report_mz != NULL)
 		dev_info->rx_offload_capa |= RTE_ETH_RX_OFFLOAD_TIMESTAMP;

Subject: Re: [PATCH] net/gve: advertise Rx scatter in GQI queue format

Applies to main and builds with -Dwerror=true. The Fixes hash resolves.

Warnings

1. GQI chains buffers whether or not the application enabled scatter.

With this patch an application can request RTE_ETH_RX_OFFLOAD_SCATTER, but gve_rx() follows GVE_RXF_PKT_CONT regardless, so one that does not still gets multi-segment mbufs whenever the MTU exceeds the Rx buffer. Advertising the capability is only half the contract; without SCATTER the driver must refuse the configuration rather than hand back chains the application did not ask for. Check it in gve_rx_queue_setup() before the rxq allocation:

       uint64_t offloads = dev->data->dev_conf.rxmode.offloads |
                           conf->offloads;
       uint32_t buf_len = RTE_MIN((uint32_t)GVE_RX_MAX_BUF_SIZE_GQI,
               RTE_ALIGN_FLOOR(rte_pktmbuf_data_room_size(pool) -
                               RTE_PKTMBUF_HEADROOM,
                               GVE_RX_BUF_ALIGN_GQI));

       if (!(offloads & RTE_ETH_RX_OFFLOAD_SCATTER) &&
           dev->data->mtu + RTE_ETHER_HDR_LEN + GVE_RX_PAD > buf_len) {
               PMD_DRV_LOG(ERR, "MTU %u needs Rx scatter",
                           dev->data->mtu);
               return -EINVAL;
       }

and apply the same test in gve_dev_mtu_set() against the configured Rx queues, since the MTU can change after queue setup. A separate patch in the same series is fine.

Info

2. The commit message says DQO "does not chain, so it is unchanged". DQO Rx ignores end_of_packet, so with buffers smaller than the frame (2 KB mbufs, 8896 MTU) each buffer is delivered as its own packet. Not introduced here, but DQO should either chain or refuse that configuration too.

GQI receives a frame larger than one Rx buffer as a chain of mbufs, but the driver doesn't report RTE_ETH_RX_OFFLOAD_SCATTER, and it chains whether or not the application enabled it. 1/2 reports SCATTER for GQI so an application that gates multi-segment receive on the capability, such as VPP, can turn it on. 2/2 refuses an Rx queue setup or an MTU change that would need scatter when the application hasn't enabled it.

Both patches change only the GQI queue format. DQO has a related problem that this series leaves alone: its Rx path ignores end_of_packet, so with Rx buffers smaller than the frame each buffer is delivered as its own packet. DQO should either chain those buffers or refuse that configuration too; that is left for a follow-up.

One more gap 2/2 doesn't close: the check reads dev->data->mtu, but gve never pushes that value to the device. rte_eth_dev_configure() sets it without calling mtu_set, so an application that never calls rte_eth_dev_set_mtu() passes the check at 1500 while the device keeps the MTU from its device descriptor, and on a jumbo VPC it can still get chained frames. Applying dev->data->mtu at configure or start would fix that but changes behavior and needs hardware to test, so it is left for the same follow-up.

v2:
- add 2/2 to refuse MTUs that need scatter when it isn't enabled (Stephen)
- correct the DQO statement in 1/2

Ryan Wolfe (2):
net/gve: advertise Rx scatter in GQI queue format net/gve: refuse an MTU that needs Rx scatter when it is off

 .mailmap                         |  1 +
 doc/guides/nics/features/gve.ini |  1 +
 drivers/net/gve/gve_ethdev.c     | 23 +++++++++++++++++++++++
 drivers/net/gve/gve_ethdev.h     |  9 +++++++++
 drivers/net/gve/gve_rx.c         | 31 +++++++++++++++++++++++++------
 5 files changed, 59 insertions(+), 6 deletions(-)

2.56.0.218.g1ec904b173-goog

Since jumbo frame support for GQI, a frame that does not fit in one Rx buffer arrives as several descriptors with GVE_RXF_PKT_CONT set on all but the last, and gve_rx_burst() chains them into one multi-segment mbuf. The driver never reports RTE_ETH_RX_OFFLOAD_SCATTER, though, so an application that enables multi-segment receive only when the port claims it treats every packet as one segment. VPP is one: it masks its requested Rx offloads with rx_offload_capa, so on a GQI gVNIC it never sets its multi-segment flag, reads only the first buffer of a jumbo frame and drops it as malformed. With 4 KB buffers that caps the usable MTU near 4000 bytes on a VPC whose MTU is 8896.

Report SCATTER for GQI only. The DQO Rx path is not touched by this patch.

---
 .mailmap                         | 1 +
 doc/guides/nics/features/gve.ini | 1 +
 drivers/net/gve/gve_ethdev.c     | 5 +++++
 3 files changed, 7 insertions(+)

index 6d9e262cfb..87c544947c 100644
@@ -1477,6 +1477,7 @@ Ruifeng Wang <[email protected]>
 Rupesh Chiluka <[email protected]>
 Rushil Gupta <[email protected]>
 Ryan E Hall <[email protected]>
+Ryan Wolfe <[email protected]>
 Sabyasachi Sengupta <[email protected]>
 Sachin Saxena <[email protected]> <[email protected]>
 Sagar Abhang <[email protected]>

index 9f3835edf6..5566511b6b 100644
@@ -7,6 +7,7 @@
 Speed capabilities   = Y
 Queue start/stop     = Y
 MTU update           = Y
+Scattered Rx         = P
 TSO                  = Y
 RSS hash             = Y
 RSS key update       = Y

index 3c8d1a76e4..92a5a7679f 100644
@@ -897,6 +897,11 @@ gve_dev_info_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info)
 	dev_info->min_mtu = RTE_ETHER_MIN_MTU;
 
 	dev_info->rx_offload_capa = RTE_ETH_RX_OFFLOAD_RSS_HASH;
+	/* GQI receives a frame larger than one Rx buffer as a chain of
+	 * buffers (gve_rx_burst follows GVE_RXF_PKT_CONT).
+	 */
+	if (gve_is_gqi(priv))
+		dev_info->rx_offload_capa |= RTE_ETH_RX_OFFLOAD_SCATTER;
 	if (priv->clk_read_type != GVE_DEV_CLK_UNSUPPORTED &&
 	    priv->nic_ts_report_mz != NULL)
 		dev_info->rx_offload_capa |= RTE_ETH_RX_OFFLOAD_TIMESTAMP;

2.56.0.218.g1ec904b173-goog

GQI chains a frame larger than one Rx buffer into a multi-segment mbuf whether or not the application enabled RTE_ETH_RX_OFFLOAD_SCATTER, so an application that did not ask for scatter can still receive chained mbufs once the MTU exceeds the Rx buffer.

Refuse such a configuration on GQI. Rx queue setup fails when the current MTU plus the Ethernet header and the GVE_RX_PAD bytes ahead of the frame does not fit in the queue's Rx buffer, and an MTU change fails on the same test against each configured Rx queue. The buffer length comes from one helper shared with queue setup, so the check matches the size the device is given. DQO is unchanged.

---
 drivers/net/gve/gve_ethdev.c | 18 ++++++++++++++++++
 drivers/net/gve/gve_ethdev.h |  9 +++++++++
 drivers/net/gve/gve_rx.c     | 31 +++++++++++++++++++++++++------
 3 files changed, 52 insertions(+), 6 deletions(-)

index 92a5a7679f..e47ec0c55b 100644
@@ -1031,6 +1031,24 @@ gve_dev_mtu_set(struct rte_eth_dev *dev, uint16_t mtu)
 		return -EBUSY;
 	}
 
+	/* GQI chains a frame larger than one Rx buffer into a multi-segment
+	 * mbuf. No Rx offload is per-queue here, so check the port offloads.
+	 */
+	if (gve_is_gqi(priv) &&
+	    !(dev->data->dev_conf.rxmode.offloads & RTE_ETH_RX_OFFLOAD_SCATTER)) {
+		for (uint16_t i = 0; i < dev->data->nb_rx_queues; i++) {
+			struct gve_rx_queue *rxq = dev->data->rx_queues[i];
+
+			if (rxq != NULL &&
+			    gve_rx_gqi_needs_scatter(mtu, rxq->rx_buf_len)) {
+				PMD_DRV_LOG(ERR,
+					    "MTU %u needs Rx scatter with %u byte Rx buffers",
+					    mtu, rxq->rx_buf_len);
+				return -EINVAL;
+			}
+		}
+	}
+
 	err = process_priv->ctrl_ops->set_mtu(priv, mtu);
 	if (err) {
 		PMD_DRV_LOG(ERR, "Failed to set mtu as %u err = %d", mtu, err);

index 19eac9ec0b..f4b72a9159 100644
@@ -454,6 +454,15 @@ gve_is_gqi(struct gve_priv *priv)
 		priv->queue_format == GVE_GQI_QPL_FORMAT;
 }
 
+/* GQI places GVE_RX_PAD bytes ahead of the frame in its first Rx buffer
+ * and chains the rest of a larger frame into further buffers.
+ */
+static inline bool
+gve_rx_gqi_needs_scatter(uint16_t mtu, uint16_t rx_buf_len)
+{
+	return (uint32_t)mtu + RTE_ETHER_HDR_LEN + GVE_RX_PAD > rx_buf_len;
+}
+
 static inline bool
 gve_get_admin_queue_ok(struct gve_priv *priv)
 {

index 09f9d8d480..4a455b3067 100644
@@ -297,6 +297,16 @@ gve_rx_queue_release(struct rte_eth_dev *dev, uint16_t qid)
 	dev->data->rx_queues[qid] = NULL;
 }
 
+static uint16_t
+gve_rx_buf_len_gqi(struct rte_mempool *pool)
+{
+	uint32_t mbuf_len =
+		rte_pktmbuf_data_room_size(pool) - RTE_PKTMBUF_HEADROOM;
+
+	return RTE_MIN((uint16_t)GVE_RX_MAX_BUF_SIZE_GQI,
+		       RTE_ALIGN_FLOOR(mbuf_len, GVE_RX_BUF_ALIGN_GQI));
+}
+
 int
 gve_rx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_id,
 		uint16_t nb_desc, unsigned int socket_id,

@@ -306,7 +316,8 @@ gve_rx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_id,
 	const struct rte_memzone *mz;
 	struct gve_rx_queue *rxq;
 	uint16_t free_thresh;
-	uint32_t mbuf_len;
+	uint16_t rx_buf_len;
+	uint64_t offloads;
 	int err = 0;
 
 	/* Ring size is required to be a power of two. */

@@ -316,6 +327,18 @@ gve_rx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_id,
 		return -EINVAL;
 	}
 
+	/* A frame larger than one Rx buffer is chained into a multi-segment
+	 * mbuf, which the application must have asked for.
+	 */
+	offloads = dev->data->dev_conf.rxmode.offloads | conf->offloads;
+	rx_buf_len = gve_rx_buf_len_gqi(pool);
+	if (!(offloads & RTE_ETH_RX_OFFLOAD_SCATTER) &&
+	    gve_rx_gqi_needs_scatter(dev->data->mtu, rx_buf_len)) {
+		PMD_DRV_LOG(ERR, "MTU %u needs Rx scatter with %u byte Rx buffers",
+			    dev->data->mtu, rx_buf_len);
+		return -EINVAL;
+	}
+
 	/* Free memory if needed. */
 	if (dev->data->rx_queues[queue_id]) {
 		gve_rx_queue_release(dev, queue_id);

@@ -351,11 +374,7 @@ gve_rx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_id,
 	rxq->hw = hw;
 	rxq->ntfy_addr = &hw->db_bar[rte_be_to_cpu_32(hw->irq_dbs[rxq->ntfy_id].id)];
 
-	mbuf_len =
-		rte_pktmbuf_data_room_size(rxq->mpool) - RTE_PKTMBUF_HEADROOM;
-	rxq->rx_buf_len =
-		RTE_MIN((uint16_t)GVE_RX_MAX_BUF_SIZE_GQI,
-			RTE_ALIGN_FLOOR(mbuf_len, GVE_RX_BUF_ALIGN_GQI));
+	rxq->rx_buf_len = rx_buf_len;
 
 	/* Allocate software ring */
 	rxq->sw_ring = rte_zmalloc_socket("gve rx sw ring", sizeof(struct rte_mbuf *) * nb_desc,

2.56.0.218.g1ec904b173-goog

Applied to next-net

AI review had a followup suggestion:

Subject: Re: [PATCH v2 0/2] net/gve: Rx scatter for GQI

Applies to main; both commits build with -Dwerror=true.

Info

[PATCH v2 2/2] net/gve: refuse an MTU that needs Rx scatter when it is off

1. rte_eth_dev_configure() sets dev->data->mtu directly from rxmode.mtu without calling mtu_set, and keeps already set up Rx queues when the queue count is unchanged. So stop, reconfigure with a larger MTU (or without SCATTER), and start without redoing queue setup passes both checks. Moving the mtu_set loop into a helper and calling it from gve_dev_start() as well covers that path:

       static int
       gve_rx_check_scatter(struct rte_eth_dev *dev, uint16_t mtu)
       {
               struct gve_priv *priv = dev->data->dev_private;
               uint16_t i;

               if (!gve_is_gqi(priv) ||
                   (dev->data->dev_conf.rxmode.offloads &
                    RTE_ETH_RX_OFFLOAD_SCATTER))
                       return 0;

               for (i = 0; i < dev->data->nb_rx_queues; i++) {
                       struct gve_rx_queue *q = dev->data->rx_queues[i];

                       if (q != NULL &&
                           gve_rx_gqi_needs_scatter(mtu, q->rx_buf_len)) {
                               PMD_DRV_LOG(ERR,
                                       "MTU %u needs Rx scatter", mtu);
                               return -EINVAL;
                       }
               }
               return 0;
       }

Not required for this series; can be a follow up.

rte_eth_dev_configure() sets the MTU from rxmode.mtu without calling mtu_set, and keeps Rx queues that are already set up when the queue count doesn't change. Stopping the port and reconfiguring it with a larger MTU, or without RTE_ETH_RX_OFFLOAD_SCATTER, therefore skipped both the MTU change check and the Rx queue setup check, and GQI went on to chain large frames for an application that hadn't enabled scatter.

Move the check into gve_rx_check_scatter() and call it from gve_dev_start() as well as gve_dev_mtu_set(), so starting such a configuration fails with an error log instead.

---
Built with -Dwerror=true (gcc and clang); not yet run on gVNIC.

 drivers/net/gve/gve_ethdev.c | 53 ++++++++++++++++++++++++------------
 1 file changed, 36 insertions(+), 17 deletions(-)

index 7506c46d9a..269950fde0 100644
@@ -459,6 +459,32 @@ gve_start_queues(struct rte_eth_dev *dev)
 	return ret;
 }
 
+/* GQI chains a frame larger than one Rx buffer into a multi-segment
+ * mbuf. No Rx offload is per-queue here, so check the port offloads.
+ */
+static int
+gve_rx_check_scatter(struct rte_eth_dev *dev, uint16_t mtu)
+{
+	struct gve_priv *priv = dev->data->dev_private;
+	uint16_t i;
+
+	if (!gve_is_gqi(priv) ||
+	    (dev->data->dev_conf.rxmode.offloads & RTE_ETH_RX_OFFLOAD_SCATTER))
+		return 0;
+
+	for (i = 0; i < dev->data->nb_rx_queues; i++) {
+		struct gve_rx_queue *rxq = dev->data->rx_queues[i];
+
+		if (rxq != NULL && gve_rx_gqi_needs_scatter(mtu, rxq->rx_buf_len)) {
+			PMD_DRV_LOG(ERR, "MTU %u needs Rx scatter with %u byte Rx buffers",
+				    mtu, rxq->rx_buf_len);
+			return -EINVAL;
+		}
+	}
+
+	return 0;
+}
+
 static int
 gve_dev_start(struct rte_eth_dev *dev)
 {

@@ -471,6 +497,13 @@ gve_dev_start(struct rte_eth_dev *dev)
 		return -EPERM;
 	}
 
+	/* rte_eth_dev_configure() sets the MTU and the Rx offloads without
+	 * calling mtu_set, and keeps Rx queues already set up, so check again.
+	 */
+	ret = gve_rx_check_scatter(dev, dev->data->mtu);
+	if (ret != 0)
+		return ret;
+
 	ret = gve_start_queues(dev);
 	if (ret != 0) {
 		PMD_DRV_LOG(ERR, "Failed to start queues");

@@ -1031,23 +1064,9 @@ gve_dev_mtu_set(struct rte_eth_dev *dev, uint16_t mtu)
 		return -EBUSY;
 	}
 
-	/* GQI chains a frame larger than one Rx buffer into a multi-segment
-	 * mbuf. No Rx offload is per-queue here, so check the port offloads.
-	 */
-	if (gve_is_gqi(priv) &&
-	    !(dev->data->dev_conf.rxmode.offloads & RTE_ETH_RX_OFFLOAD_SCATTER)) {
-		for (uint16_t i = 0; i < dev->data->nb_rx_queues; i++) {
-			struct gve_rx_queue *rxq = dev->data->rx_queues[i];
-
-			if (rxq != NULL &&
-			    gve_rx_gqi_needs_scatter(mtu, rxq->rx_buf_len)) {
-				PMD_DRV_LOG(ERR,
-					    "MTU %u needs Rx scatter with %u byte Rx buffers",
-					    mtu, rxq->rx_buf_len);
-				return -EINVAL;
-			}
-		}
-	}
+	err = gve_rx_check_scatter(dev, mtu);
+	if (err != 0)
+		return err;
 
 	err = process_priv->ctrl_ops->set_mtu(priv, mtu);
 	if (err) {

2.56.0.218.g1ec904b173-goog

AI found a flaw in this logic

Review: [PATCH] net/gve: check the Rx scatter MTU at device start
Patchwork: 171827

Applied on top of cffdca824388 (next-net-for-main); builds clean with -Dwerror=true for net/gve.

Warning

gve_dev_start() checks dev->data->mtu, but gve never gives that MTU to the device. The only caller of ctrl_ops->set_mtu() is gve_dev_mtu_set(), and rte_eth_dev_configure() just assigns dev->data->mtu from rxmode.mtu. After configure the gVNIC keeps whatever MTU it had (its default, or the last mtu_set), so the start check validates a number the hardware is not using. With rxmode.mtu left at 0 (1500) on a device running a larger MTU, the check passes and GQI still chains frames up to the device MTU without scatter.

Program the MTU at start, after the check, so the two agree:

	ret = gve_rx_check_scatter(dev, dev->data->mtu);
	if (ret != 0)
		return ret;

	ret = process_priv->ctrl_ops->set_mtu(priv, dev->data->mtu);
	if (ret != 0) {
		PMD_DRV_LOG(ERR, "Failed to set MTU %u: %d",
			    dev->data->mtu, ret);
		return ret;
	}

(move "priv = dev->data->dev_private;" to the top of the function.) This also fixes rxmode.mtu being ignored by gve, which is the larger bug here.

"Not yet run on gVNIC". Please test on GQI before v2: testpmd, port stop, port config mtu / reconfigure without scatter, port start must fail; and with the set_mtu change, confirm a 1500 MTU config drops jumbo frames.

Info

Commit message: ethdev keeps already set up Rx queues below the new count whether or not the count changes (eth_dev_rx_queue_config only releases queues >= nb_queues). Drop "when the queue count doesn't change".

cffdca824388 is only in next-net-for-main, not in any release. This can be squashed into it when merging; if kept separate, Cc: stable is not needed since the Fixes target already carries it.

rte_eth_dev_configure() sets the MTU from rxmode.mtu without calling mtu_set, and keeps Rx queues that are already set up. Stopping the port and reconfiguring it with a larger MTU, or without RTE_ETH_RX_OFFLOAD_SCATTER, therefore skipped both the MTU change check and the Rx queue setup check. gve also never gave the device that MTU.

Even with the MTU programmed, a GQI device does not limit what it receives to it: on a network with a larger MTU it still delivers bigger frames, chained over several Rx buffers. So the configuration checks alone cannot keep chained mbufs from an application that did not enable scatter.

Move the configuration check into gve_rx_check_scatter() and call it from gve_dev_start() as well as gve_dev_mtu_set(), program the configured MTU into the device at start, and have the GQI Rx path drop a frame that spans more than one buffer, counted in the Rx errors, when the port has not enabled scatter. A dropped descriptor's mbuf is now freed, since the refill replaces it.

---
v2:
- program the configured MTU into the device at start (Stephen)
- on GQI-QPL hardware the device does not enforce the driver MTU on Rx:
8000-byte frames still arrive chained over 4 buffers at MTU 1500, so also drop and count chained frames in the Rx path when scatter is off, and free a dropped descriptor's mbuf
- retitled from "net/gve: check the Rx scatter MTU at device start"
- commit message fix and no Cc: stable (Stephen)

Tested on a GQI-QPL gVNIC with testpmd. At MTU 1500 without scatter, 25000 8000-byte and 25000 1000-byte UDP datagrams, interleaved, gave 25000 Rx errors and 25000 packets received, with no nombuf on a 16384-mbuf pool and traffic still flowing afterwards; with scatter on, the 8042-byte frames arrive as 4-segment mbufs. A small ethdev program that reconfigures a started-and-stopped port to MTU 8000 without scatter, keeping its Rx queues, now gets -EINVAL from rte_eth_dev_start() (it started before).

 drivers/net/gve/gve_ethdev.c | 75 +++++++++++++++++++++++++++---------
 drivers/net/gve/gve_ethdev.h |  2 +
 drivers/net/gve/gve_rx.c     | 20 +++++++++-
 3 files changed, 76 insertions(+), 21 deletions(-)

index 7506c46d9a..8ee52fdd47 100644
@@ -459,11 +459,38 @@ gve_start_queues(struct rte_eth_dev *dev)
 	return ret;
 }
 
+/* GQI chains a frame larger than one Rx buffer into a multi-segment
+ * mbuf. No Rx offload is per-queue here, so check the port offloads.
+ */
+static int
+gve_rx_check_scatter(struct rte_eth_dev *dev, uint16_t mtu)
+{
+	struct gve_priv *priv = dev->data->dev_private;
+	uint16_t i;
+
+	if (!gve_is_gqi(priv) ||
+	    (dev->data->dev_conf.rxmode.offloads & RTE_ETH_RX_OFFLOAD_SCATTER))
+		return 0;
+
+	for (i = 0; i < dev->data->nb_rx_queues; i++) {
+		struct gve_rx_queue *rxq = dev->data->rx_queues[i];
+
+		if (rxq != NULL && gve_rx_gqi_needs_scatter(mtu, rxq->rx_buf_len)) {
+			PMD_DRV_LOG(ERR, "MTU %u needs Rx scatter with %u byte Rx buffers",
+				    mtu, rxq->rx_buf_len);
+			return -EINVAL;
+		}
+	}
+
+	return 0;
+}
+
 static int
 gve_dev_start(struct rte_eth_dev *dev)
 {
 	struct gve_process_private *process_priv = dev->process_private;
-	struct gve_priv *priv;
+	struct gve_priv *priv = dev->data->dev_private;
+	uint16_t i;
 	int ret;
 
 	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {

@@ -471,6 +498,31 @@ gve_dev_start(struct rte_eth_dev *dev)
 		return -EPERM;
 	}
 
+	/* rte_eth_dev_configure() sets the MTU and the Rx offloads without
+	 * calling mtu_set, and keeps Rx queues already set up, so check them
+	 * again and give the device the MTU the check was made against.
+	 */
+	ret = gve_rx_check_scatter(dev, dev->data->mtu);
+	if (ret != 0)
+		return ret;
+
+	ret = process_priv->ctrl_ops->set_mtu(priv, dev->data->mtu);
+	if (ret != 0) {
+		PMD_DRV_LOG(ERR, "Failed to set MTU %u: %d", dev->data->mtu, ret);
+		return ret;
+	}
+
+	/* The device chains frames up to its own MTU, not the one it was
+	 * given; the Rx path drops them unless scatter is on.
+	 */
+	for (i = 0; i < dev->data->nb_rx_queues; i++) {
+		struct gve_rx_queue *rxq = dev->data->rx_queues[i];
+
+		if (rxq != NULL)
+			rxq->scatter = !!(dev->data->dev_conf.rxmode.offloads &
+					  RTE_ETH_RX_OFFLOAD_SCATTER);
+	}
+
 	ret = gve_start_queues(dev);
 	if (ret != 0) {
 		PMD_DRV_LOG(ERR, "Failed to start queues");

@@ -480,7 +532,6 @@ gve_dev_start(struct rte_eth_dev *dev)
 	dev->data->dev_started = 1;
 	gve_link_update(dev, 0);
 
-	priv = dev->data->dev_private;
 	/* No stats available yet for Dqo. */
 	if (gve_is_gqi(priv)) {
 		ret = gve_alloc_stats_report(priv,

@@ -1031,23 +1082,9 @@ gve_dev_mtu_set(struct rte_eth_dev *dev, uint16_t mtu)
 		return -EBUSY;
 	}
 
-	/* GQI chains a frame larger than one Rx buffer into a multi-segment
-	 * mbuf. No Rx offload is per-queue here, so check the port offloads.
-	 */
-	if (gve_is_gqi(priv) &&
-	    !(dev->data->dev_conf.rxmode.offloads & RTE_ETH_RX_OFFLOAD_SCATTER)) {
-		for (uint16_t i = 0; i < dev->data->nb_rx_queues; i++) {
-			struct gve_rx_queue *rxq = dev->data->rx_queues[i];
-
-			if (rxq != NULL &&
-			    gve_rx_gqi_needs_scatter(mtu, rxq->rx_buf_len)) {
-				PMD_DRV_LOG(ERR,
-					    "MTU %u needs Rx scatter with %u byte Rx buffers",
-					    mtu, rxq->rx_buf_len);
-				return -EINVAL;
-			}
-		}
-	}
+	err = gve_rx_check_scatter(dev, mtu);
+	if (err != 0)
+		return err;
 
 	err = process_priv->ctrl_ops->set_mtu(priv, mtu);
 	if (err) {

index 26c6a304a6..3a8ddfbbb6 100644
@@ -263,6 +263,8 @@ struct gve_rx_queue {
 	struct rte_mbuf **refill_bufs;
 
 	uint8_t is_gqi_qpl;
+	/* the port enabled RTE_ETH_RX_OFFLOAD_SCATTER (GQI) */
+	bool scatter;
 	bool timestamp_enabled;
 };
 

index 4a455b3067..31c187ca83 100644
@@ -139,12 +139,21 @@ gve_rx(struct gve_rx_queue *rxq, volatile struct gve_rx_desc *rxd, uint16_t rx_i
 	struct rte_mbuf *rxe;
 
 	if (ctx->drop_pkt)
-		goto finish_frag;
+		goto drop_frag;
 
 	if (rxd->flags_seq & GVE_RXF_ERR) {
 		ctx->drop_pkt = true;
 		rxq->stats.errors++;
-		goto finish_frag;
+		goto drop_frag;
+	}
+
+	/* The device chains a frame larger than one Rx buffer whatever MTU
+	 * it was given, so drop it unless the application enabled scatter.
+	 */
+	if (is_first_frag && !is_last_frag && !rxq->scatter) {
+		ctx->drop_pkt = true;
+		rxq->stats.errors++;
+		goto drop_frag;
 	}
 
 	if (is_first_frag)

@@ -171,6 +180,12 @@ gve_rx(struct gve_rx_queue *rxq, volatile struct gve_rx_desc *rxd, uint16_t rx_i
 		}
 	}
 
+	goto finish_frag;
+
+drop_frag:
+	/* The refill puts a new mbuf in this slot, so free the old one. */
+	rte_pktmbuf_free(rxq->sw_ring[rx_id]);
+	rxq->sw_ring[rx_id] = NULL;
 finish_frag:
 	ctx->total_frags++;
 	return is_last_frag;

@@ -375,6 +390,7 @@ gve_rx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_id,
 	rxq->ntfy_addr = &hw->db_bar[rte_be_to_cpu_32(hw->irq_dbs[rxq->ntfy_id].id)];
 
 	rxq->rx_buf_len = rx_buf_len;
+	rxq->scatter = !!(offloads & RTE_ETH_RX_OFFLOAD_SCATTER);
 
 	/* Allocate software ring */
 	rxq->sw_ring = rte_zmalloc_socket("gve rx sw ring", sizeof(struct rte_mbuf *) * nb_desc,

2.56.0.257.gf7497c7d27-goog

Looks good applied to next-net