Skip to contentSkip to conversation

e1000e: use page pool

PATCH In review Matt Vollrath 3 participants 14 messages Aug 30 – Sep 3

Subsystems (5)
lore
Share & tools
lore
#0 lore raw

cover e1000e: use page pool

This series converges the three e1000e Rx paths into one and then makes it use a page pool. The jumbo path is chosen as the golden path because it is the only one that can handle every combination of MTU and page size. Any performance regression caused by using that path for all configurations is remedied by eliminating allocations and DMA mapping from the hot path, as shown in benchmarks below.

There are some general modernizations in this conversion, intended to make this driver match conventions found in other Intel drivers. The copybreak parameter is removed. The NAPI skb cache is used. The netmem API is used for chained frames. Prefetch is used optimistically in the cleaner loop to pull in the next descriptor and the frame headers.

User-visible changes:
* The copybreak module parameter and its documentation are removed.
* The rx_header_split ethtool stat is removed.
* Small packets are charged at the buffer's full 2 KB truesize at standard MTU.
* Jumbo MTU frames are delivered as 2 KB chunks. At MTU > 2026 this increases userspace copy overhead by gathering more fragments and increases truesize accounting per frame. This is described in more detail below.

The series begins by fixing some existing bugs which would either be more reachable or have worse consequences after the conversion. Missing support for CrcStripping=0 is added. The page dump routine's mis-read of jumbo buffers is remedied. A race between the reset task and runtime PM is guarded.

Then the standard and packet-split paths are removed, making the jumbo path the only way to Rx. At this point standard-MTU Rx allocates and maps a page per packet, until the page pool conversion removes that cost.

Next NAPI is disabled while the adapter is down, which is required to pass assertions when the page pool is destroyed.

Next is the main event, the singular Rx path is converted to use a page pool with support from libeth.

Finally, a small change to Tx to return skbs to the NAPI cache, which the Rx side now pulls from when constructing frame heads.

Benchmarks were run on an I218-V and Xeon E3-1240v3 pinned at 3.8GHz. NAPI was isolated on one core and iperf3 on another. ITR was fixed at 4000/s because the adaptive scaling can take a few minutes to settle (to be fixed separately). C-states above C1 and EEE were disabled. Kernel version was 7.1.12. Direct link from sender to receiver.

The sender was an 82574L in a Xeon E3-1270 pinned at 3.4GHz, NIC vectors and iperf3 -s pinned to separate cores, ITR fixed.

For cycle measurements, each cell is perf stat cycles on the core divided by packets received over 30s, averaged over three runs. IOMMU "lazy" is the default DMA-FQ. Throughput was identical between stock and series, with one exception noted below. The stock driver is configured at the default copybreak=256. Note that the stock copybreak features is flawed; it always remaps DMA when sending an already-mapped buffer back to the h/w.

Bulk TCP, segments sized to the MTU (line rate at both MTUs):

NAPI core cycles/pkt consumer core cycles/pkt

  MTU   IOMMU   kpps    stock  series  change   stock  series  change
  1500  off     81.4     2523    1784    -29%    2193    2193      0%
  1500  lazy    81.4     5170    1938    -63%    2369    2323     -2%
  9000  off     13.9    11590    7574    -35%   11764   12792     +9%
  9000  lazy    13.9    21781    8423    -61%   12129   13396    +10%

The increase in consumer cost at MTU 9000 is probably because the frame arrives as five 2 KB chunks instead of three pieces from the legacy packet-split path. The copy to userspace walks more fragments and GRO merges fewer frames per skb.

TCP with an 88-byte MSS (iperf3 -M 88, ~142-byte frames):

NAPI core cycles/pkt consumer core cycles/pkt

  MTU   IOMMU   kpps    stock  series  change   stock  series  change
  1500  off    754.3     1453     793    -45%     446     484     +8%
  1500  lazy   754.3     3819     809    -79%     387     496    +28%
  9000  off    754.4     2231     780    -65%     498     509     +2%
  9000  lazy 520/754     6920     798    -88%     430     523    +22%

At MTU 9000 with IOMMU on, the stock driver can't keep up with the stream at line rate. It polls continuously at 95% of the core and can't send descriptors back to h/w fast enough, so it sends pause frames and flow control throttles the sender.

Differences in consumer cost are likely explained by two things:
* The stock copybreak=256 leaves the payload in the NAPI core's cache.
* The high IOMMU cost in the stock driver causes it to return more bytes per read.

UDP, one full-size datagram per frame (1472/8972 bytes, line rate):

NAPI core cycles/pkt consumer core cycles/pkt

  MTU   IOMMU   kpps    stock  series  change   stock  series  change
  1500  off     81.4     4991    4417    -11%    6409    6381      0%
  1500  lazy    81.4     7426    4397    -41%    6619    6395     -3%
  9000  off     13.9    12070    8286    -31%   12861   13624     +6%
  9000  lazy    13.9    21494    8289    -61%   12566   13772    +10%

Here we see a similar tax on the jumbo MTU consumer due to gathering more fragments.

UDP, 64-byte datagrams paced at 150 kpps:

NAPI core cycles/pkt consumer core cycles/pkt

  MTU   IOMMU   kpps    stock  series  change   stock  series  change
  1500  off    150.2     4202    3939     -6%    5541    5525      0%
  1500  lazy   150.3     6663    3919    -41%    6151    5465    -11%
  9000  off    150.2     4376    3956    -10%    5535    5428     -2%
  9000  lazy   150.2     6821    3953    -42%    6154    5453    -11%

At MTU 9000, this rate of datagrams runs up against the socket receive buffer's limits in the series, causing a 0.003% drop rate. Because each datagram sits in a full page buffer, the default 213 KB socket buffer only holds 48 vs. stock's ~275. This can be remedied by increasing SO_RCVBUF or rmem_default. This is another example of copybreak truesize compression value in the stock driver.

Payload Density:

This table shows pieces per frame and density at every MTU breakpoint. Stock is the standard path at 1500 and packet-split above it.

stock (std / packet split) series (2 KB chunks)
MTU pieces truesize density pieces truesize density

<= 1500           1     2688      0.56         1     2304      0.66
1501-2026         1     4864   0.31-0.42       1     4352   0.35-0.47
2027-4074         1     4864   0.42-0.84       2     8448   0.24-0.48
4075-4096         1     4864      0.84         3    12544      0.33
4097-6122         2     8960   0.46-0.68       3    12544   0.33-0.49
6123-8170         2     8960   0.68-0.91       4    16640   0.37-0.49
8171-8192         2     8960   0.91-0.92       5    20736   0.39-0.40
8193-9212         3    13056   0.63-0.71       5    20736   0.40-0.44

Series chunk counts assume the worst-case frame (MTU + 22: VLAN tag and FCS present).

This series is hungrier for socket buffer size under two conditions:
* At higher-than-standard MTU, some page size is wasted because RCTL.BSIZE only accepts powers of two, unlike the more granular settings of later h/w.
* For small packets at any MTU, copybreak no longer moves the payload into an appropriately-sized buffer.

Matt Vollrath (8):
e1000e: add jumbo Rx CRC stripping
e1000e: dump pages for jumbo Rx buffers
e1000e: prevent race between PM and reset task
e1000e: remove packet-split Rx path
e1000e: always use jumbo Rx path
e1000e: disable NAPI while interface is down
e1000e: use libeth page_pool for Rx
e1000e: return skbs to NAPI cache

 .../device_drivers/ethernet/intel/e1000e.rst  |   15 -
 drivers/net/ethernet/intel/Kconfig            |    1 +
 drivers/net/ethernet/intel/e1000e/e1000.h     |   53 +-
 drivers/net/ethernet/intel/e1000e/ethtool.c   |    1 -
 drivers/net/ethernet/intel/e1000e/netdev.c    | 1374 +++++------------
 drivers/net/ethernet/intel/e1000e/param.c     |    6 -
 6 files changed, 382 insertions(+), 1068 deletions(-)

base-commit: 1b78070aaef63512688aebfbc82365ef9d6660f1

Signature · 1 line
2.43.0
#1 lore raw

PATCH 1/8 e1000e: add jumbo Rx CRC stripping

When the hardware is configured to not strip the FCS (CrcStripping=0), the jumbo Rx path would deliver frames with the FCS still attached. Follow the standard path's convention by stripping the FCS at EOP and leaving it out of the byte count.

The jumbo path was omitted when the CrcStripping feature was added. Since v3.3[1], the jumbo path has only been used where the page size is over 16K, which practically rules out LOMs, BMC sideband, and the need to leave FCS on frames. The bug is only reachable only by setting CrcStripping=0 on a platform with nothing that needs it.

This change prepares for convergence of Rx onto the jumbo path in following patches. The set of h/w routed through this path will expand to include LOMs.

[1] Commit 79d4e9087a6e ("e1000e: disable Early Receive DMA on ICH LOMs")

Fixes eb7c3adb1ca9 ("e1000e: fix IPMI traffic")
Assisted-by Claude:claude-5-fable
---
 drivers/net/ethernet/intel/e1000e/netdev.c | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)
drivers/net/ethernet/intel/e1000e/netdev.c +11 -1
index 844f31ab37ad..599600ad695c 100644
@@ -1631,13 +1631,23 @@ static bool e1000_clean_jumbo_rx_irq(struct e1000_ring *rx_ring, int *work_done,
 			}
 		}
 
+		/* strip the Ethernet CRC; it may span fragments */
+		if (!(adapter->flags2 & FLAG2_CRC_STRIPPING) &&
+		    !(netdev->features & NETIF_F_RXFCS))
+			pskb_trim(skb, skb->len - 4);
+
 		/* Receive Checksum Offload */
 		e1000_rx_checksum(adapter, staterr, skb);
 
 		e1000_rx_hash(netdev, rx_desc->wb.lower.hi_dword.rss, skb);
 
-		/* probably a little skewed due to removing CRC */
 		total_rx_bytes += skb->len;
+		/* If configured to store CRC, keep the FCS bytes out of the
+		 * total_rx_bytes counter
+		 */
+		if (!(adapter->flags2 & FLAG2_CRC_STRIPPING) &&
+		    (netdev->features & NETIF_F_RXFCS))
+			total_rx_bytes -= 4;
 		total_rx_packets++;
 
 		/* eth type trans needs skb->data to point to something */
Signature · 1 line
2.43.0
#5 lore raw

PATCH 2/8 e1000e: dump pages for jumbo Rx buffers

In reply to Matt Vollrath

The jumbo Rx path keeps its data in buffer_info->page rather than the skb, which is only a shell. Previously data beyond the end of the skb allocation would be dumped. The jumbo path would at best dump useless garbage. At worst it would dump arbitrary kernel memory. This OOB read is only reachable when page size is >16K and MTU is >1518.

Dump the page instead when it exists. Skip dumping the shell skb left behind when a jumbo slot is cleaned.

Fixes f0c5dadff3fb ("e1000e: fix panic while dumping packets on Tx hang with IOMMU")
Assisted-by Claude:claude-5-fable
---
 drivers/net/ethernet/intel/e1000e/netdev.c | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)
drivers/net/ethernet/intel/e1000e/netdev.c +16 -1
index 599600ad695c..47ff3c6ab451 100644
@@ -465,8 +465,23 @@ static void e1000e_dump(struct e1000_adapter *adapter)
 					(unsigned long long)buffer_info->dma,
 					buffer_info->skb, next_desc);
 
+				/* Jumbo buffers land in the page; a cleaned
+				 * jumbo slot keeps only its small shell skb
+				 * until it is refilled, so only dump an skb
+				 * that can hold a whole buffer.
+				 */
 				if (netif_msg_pktdata(adapter) &&
-				    buffer_info->skb)
+				    buffer_info->page)
+					print_hex_dump(KERN_INFO, "",
+						       DUMP_PREFIX_ADDRESS, 16,
+						       1,
+						       page_address(buffer_info->page),
+						       adapter->rx_buffer_len,
+						       true);
+				else if (netif_msg_pktdata(adapter) &&
+					 buffer_info->skb &&
+					 skb_tailroom(buffer_info->skb) >=
+					 adapter->rx_buffer_len)
 					print_hex_dump(KERN_INFO, "",
 						       DUMP_PREFIX_ADDRESS, 16,
 						       1,
Signature · 1 line
2.43.0
#6 lore raw

PATCH 3/8 e1000e: prevent race between PM and reset task

In reply to Matt Vollrath

e1000e_down could be entered simultaneously by the reset task and the runtime PM suspend callback. Prevent this by acquiring a runtime PM reference in the reset task.

The intention is that:
* A reset attempted during runtime suspend should drop silently; the interface is already going down.
* A runtime suspend attempted during a reset will be retried later.
* All callers of e1000e_down are now serialized.

Fixes 23606cf5d119 ("e1000e / PCI / PM: Add basic runtime PM support (rev. 4)")
Assisted-by Claude:claude-5-fable
---
 drivers/net/ethernet/intel/e1000e/netdev.c | 25 ++++++++++++++++++----
 1 file changed, 21 insertions(+), 4 deletions(-)
drivers/net/ethernet/intel/e1000e/netdev.c +21 -4
index 47ff3c6ab451..4c15ca307b08 100644
@@ -6004,20 +6004,37 @@ static void e1000_tx_timeout(struct net_device *netdev, unsigned int __always_un
 static void e1000_reset_task(struct work_struct *work)
 {
 	struct e1000_adapter *adapter;
+	struct device *dev;
+	int rc;
+
 	adapter = container_of(work, struct e1000_adapter, reset_task);
+	dev = &adapter->pdev->dev;
 
 	rtnl_lock();
+
+	/* Runtime suspend downs the device without holding rtnl.  Hold a
+	 * runtime PM reference so it cannot start underneath the reset, and
+	 * skip the reset if the device is already suspending or suspended:
+	 * resuming resets the hardware anyway.
+	 */
+	rc = pm_runtime_get_if_active(dev);
+	if (!rc)
+		goto out_unlock;
+
 	/* don't run the task if already down */
-	if (test_bit(__E1000_DOWN, &adapter->state)) {
-		rtnl_unlock();
-		return;
-	}
+	if (test_bit(__E1000_DOWN, &adapter->state))
+		goto out_put;
 
 	if (!(adapter->flags & FLAG_RESTART_NOW)) {
 		e1000e_dump(adapter);
 		e_err("Reset adapter unexpectedly\n");
 	}
 	e1000e_reinit_locked(adapter);
+
+out_put:
+	if (rc > 0)
+		pm_runtime_put(dev);
+out_unlock:
 	rtnl_unlock();
 }
 
Signature · 1 line
2.43.0
#7 lore raw

PATCH 4/8 e1000e: remove packet-split Rx path

In reply to Matt Vollrath

This approach to Rx has not stood the test of time. All other Intel drivers have removed it. Do the same for e1000e.

Use the standard path for any MTU <=1518.

Use the jumbo path for any MTU >1518.

This may not be an ideal breakpoint, but the next step is factoring out the breakpoint by converging on the jumbo Rx path with page pool.

Changes to e1000e_dump should only be removal of the ps path and un-indentation of the other path from the switch, except the correct sized u0 overlay is used instead of u1.

The rx_header_split statistic has been removed.

Assisted-by Claude:claude-5-fable
---
 drivers/net/ethernet/intel/e1000e/e1000.h   |  16 +-
 drivers/net/ethernet/intel/e1000e/ethtool.c |   1 -
 drivers/net/ethernet/intel/e1000e/netdev.c  | 637 +++-----------------
 3 files changed, 81 insertions(+), 573 deletions(-)
drivers/net/ethernet/intel/e1000e/e1000.h +1 -15
index 63ebe00376f5..533d1981eb5e 100644
@@ -121,11 +121,6 @@ enum e1000_boards {
 	board_pch_ptp
 };
 
-struct e1000_ps_page {
-	struct page *page;
-	u64 dma; /* must be u64 - written to hw */
-};
-
 /* wrappers around a pointer to a socket buffer,
  * so a DMA handle can be stored along with the buffer
  */
@@ -143,11 +138,7 @@ struct e1000_buffer {
 			u16 mapped_as_page;
 		};
 		/* Rx */
-		struct {
-			/* arrays of page information for packet split */
-			struct e1000_ps_page *ps_pages;
-			struct page *page;
-		};
+		struct page *page;
 	};
 };
 
@@ -265,15 +256,12 @@ struct e1000_adapter {
 	/* Rx stats */
 	u64 hw_csum_err;
 	u64 hw_csum_good;
-	u64 rx_hdr_split;
 	u32 gorc;
 	u64 gorc_old;
 	u32 alloc_rx_buff_failed;
 	u32 rx_dma_failed;
 	u32 rx_hwtstamp_cleared;
 
-	unsigned int rx_ps_pages;
-	u16 rx_ps_bsize0;
 	u32 max_frame_size;
 	u32 min_frame_size;
 
@@ -464,8 +452,6 @@ s32 e1000e_get_base_timinca(struct e1000_adapter *adapter, u32 *timinca);
 #define FLAG2_ENABLE_S0IX_FLOWS           BIT(15)
 #define FLAG2_DISABLE_K1		   BIT(16)
 
-#define E1000_RX_DESC_PS(R, i)	    \
-	(&(((union e1000_rx_desc_packet_split *)((R).desc))[i]))
 #define E1000_RX_DESC_EXT(R, i)	    \
 	(&(((union e1000_rx_desc_extended *)((R).desc))[i]))
 #define E1000_GET_DESC(R, i, type)	(&(((struct type *)((R).desc))[i]))
drivers/net/ethernet/intel/e1000e/ethtool.c +0 -1
index a8b35ae41141..b209b0c3da5f 100644
@@ -85,7 +85,6 @@ static const struct e1000_stats e1000_gstrings_stats[] = {
 	E1000_STAT("tx_flow_control_xoff", stats.xofftxc),
 	E1000_STAT("rx_csum_offload_good", hw_csum_good),
 	E1000_STAT("rx_csum_offload_errors", hw_csum_err),
-	E1000_STAT("rx_header_split", rx_hdr_split),
 	E1000_STAT("alloc_rx_buff_failed", alloc_rx_buff_failed),
 	E1000_STAT("tx_smbus", stats.mgptc),
 	E1000_STAT("rx_smbus", stats.mgprc),
drivers/net/ethernet/intel/e1000e/netdev.c +80 -557

756 lines not on this page · raw message

Signature · 1 line
2.43.0
#8 lore raw

PATCH 5/8 e1000e: always use jumbo Rx path

In reply to Matt Vollrath

Converge on the Rx path which can support any combination of page size and MTU.

Rename the jumbo path's clean and allocate functions to the standard path's names. Also, drop the unused return value from the clean function.

Factor out the function pointers for cleaning and allocation.

Remove the discarding flag because we no longer desire to discard frames that don't fit in a single descriptor.

Remove rx_buffer_len clamping because buffers are now whole pages.

Additional cost for standard MTU operation will be addressed by the proceeding page pool conversion.

Assisted-by Claude:claude-5-fable
---
 drivers/net/ethernet/intel/e1000e/e1000.h  |   7 +-
 drivers/net/ethernet/intel/e1000e/netdev.c | 311 ++-------------------
 2 files changed, 18 insertions(+), 300 deletions(-)
drivers/net/ethernet/intel/e1000e/e1000.h +1 -6
index 533d1981eb5e..e0e4b72c13f3 100644
@@ -244,11 +244,7 @@ struct e1000_adapter {
 	u32 tx_hwtstamp_skipped;
 
 	/* Rx */
-	bool (*clean_rx)(struct e1000_ring *ring, int *work_done,
-			 int work_to_do) ____cacheline_aligned_in_smp;
-	void (*alloc_rx_buf)(struct e1000_ring *ring, int cleaned_count,
-			     gfp_t gfp);
-	struct e1000_ring *rx_ring;
+	struct e1000_ring *rx_ring ____cacheline_aligned_in_smp;
 
 	u32 rx_int_delay;
 	u32 rx_abs_int_delay;
@@ -436,7 +432,6 @@ s32 e1000e_get_base_timinca(struct e1000_adapter *adapter, u32 *timinca);
 
 #define FLAG2_CRC_STRIPPING               BIT(0)
 #define FLAG2_HAS_PHY_WAKEUP              BIT(1)
-#define FLAG2_IS_DISCARDING               BIT(2)
 #define FLAG2_DISABLE_ASPM_L1             BIT(3)
 #define FLAG2_HAS_PHY_STATS               BIT(4)
 #define FLAG2_HAS_EEE                     BIT(5)
drivers/net/ethernet/intel/e1000e/netdev.c +17 -294

425 lines not on this page · raw message

Signature · 1 line
2.43.0
#9 lore raw

PATCH 6/8 e1000e: disable NAPI while interface is down

In reply to Matt Vollrath

The upcoming page pool conversion requires that NAPI is disabled during reconfiguration windows. Otherwise pool destruction would fail the assertion that NAPI will not race, causing a benign but loud warning.

Move napi_disable into e1000e_down and add a napi_enable to e1000e_up. This follows convention of all other Intel drivers.

Because napi_disable would hang if called twice, all callers of e1000e_down must now be serialized. As of this patch, I'm convinced this is true.

Assisted-by Claude:claude-5-fable
---
 drivers/net/ethernet/intel/e1000e/netdev.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
drivers/net/ethernet/intel/e1000e/netdev.c +3 -2
index 194166531bc8..6d846cacb026 100644
@@ -3527,6 +3527,8 @@ void e1000e_up(struct e1000_adapter *adapter)
 	/* hardware has been reset, we need to reload some things */
 	e1000_configure(adapter);
 
+	napi_enable(&adapter->napi);
+
 	clear_bit(__E1000_DOWN, &adapter->state);
 
 	if (adapter->msix_entries)
@@ -3601,7 +3603,7 @@ void e1000e_down(struct e1000_adapter *adapter, bool reset)
 
 	e1000_irq_disable(adapter);
 
-	napi_synchronize(&adapter->napi);
+	napi_disable(&adapter->napi);
 
 	timer_delete_sync(&adapter->watchdog_timer);
 	timer_delete_sync(&adapter->phy_info_timer);
@@ -4055,7 +4057,6 @@ int e1000e_close(struct net_device *netdev)
 
 	netif_queue_set_napi(netdev, 0, NETDEV_QUEUE_TYPE_RX, NULL);
 	netif_queue_set_napi(netdev, 0, NETDEV_QUEUE_TYPE_TX, NULL);
-	napi_disable(&adapter->napi);
 
 	e1000e_free_tx_resources(adapter->tx_ring);
 	e1000e_free_rx_resources(adapter->rx_ring);
Signature · 1 line
2.43.0
#12 lore raw

PATCH 7/8 e1000e: use libeth page_pool for Rx

In reply to Matt Vollrath

Convert this driver's Rx to use a page pool managed by libeth. This change potentially eliminates most DMA map/unmap operations from the Rx hot path, which was most of the overhead with IOMMU enabled.

The previous standard path's behavior of publishing descriptors back to the hardware every 16 iterations is retained because it was the safest choice.

libeth is pulled in as a dependency because it is a convenient wrapper around pool creation, destruction, and geometry.

The copybreak feature has been removed because it no longer consistently avoids the cost of DMA remapping (which was broken in this driver).

Assisted-by Claude:claude-5-fable
---
 .../device_drivers/ethernet/intel/e1000e.rst  |  15 -
 drivers/net/ethernet/intel/Kconfig            |   1 +
 drivers/net/ethernet/intel/e1000e/e1000.h     |  32 +-
 drivers/net/ethernet/intel/e1000e/netdev.c    | 567 ++++++++++--------
 drivers/net/ethernet/intel/e1000e/param.c     |   6 -
 5 files changed, 332 insertions(+), 289 deletions(-)
Documentation/networking/device_drivers/ethernet/intel/e1000e.rst +0 -15
index d8f810afdd49..fab6ebc0cf95 100644
@@ -163,21 +163,6 @@ It ensures that an interrupt is generated after the initial Packet is sent on
 the wire within the set amount of time. Proper tuning, along with TxIntDelay,
 may improve traffic throughput in specific network conditions.
 
-copybreak
----------
-:Valid Range: 0-xxxxxxx (0=off)
-:Default Value: 256
-
-The driver copies all packets below or equaling this size to a fresh receive
-buffer before handing it up the stack.
-This parameter differs from other parameters because it is a single (not 1,1,1
-etc.) parameter applied to all driver instances and it is also available
-during runtime at /sys/module/e1000e/parameters/copybreak.
-
-To use copybreak, type::
-
-    modprobe e1000e.ko copybreak=128
-
 SmartPowerDownEnable
 --------------------
 :Valid Range: 0,1
drivers/net/ethernet/intel/Kconfig +1 -0
index 780f113986ea..86fddcdabda3 100644
@@ -63,6 +63,7 @@ config E1000E
 	depends on PCI && (!SPARC32 || BROKEN)
 	depends on PTP_1588_CLOCK_OPTIONAL
 	select CRC32
+	select LIBETH
 	help
 	  This driver supports the PCI-Express Intel(R) PRO/1000 gigabit
 	  ethernet family of adapters. For PCI or PCI-X e1000 adapters,
drivers/net/ethernet/intel/e1000e/e1000.h +15 -17
index e0e4b72c13f3..f8e9f3fecf3e 100644
@@ -23,6 +23,7 @@
 #include <linux/mdio.h>
 #include <linux/mutex.h>
 #include <linux/pm_qos.h>
+#include <net/libeth/rx.h>
 #include "hw.h"
 
 struct e1000_info;
@@ -121,25 +122,18 @@ enum e1000_boards {
 	board_pch_ptp
 };
 
-/* wrappers around a pointer to a socket buffer,
+/* wrapper around a pointer to a Tx socket buffer,
  * so a DMA handle can be stored along with the buffer
  */
 struct e1000_buffer {
 	dma_addr_t dma;
 	struct sk_buff *skb;
-	union {
-		/* Tx */
-		struct {
-			unsigned long time_stamp;
-			u16 length;
-			u16 next_to_watch;
-			unsigned int segs;
-			unsigned int bytecount;
-			u16 mapped_as_page;
-		};
-		/* Rx */
-		struct page *page;
-	};
+	unsigned long time_stamp;
+	u16 length;
+	u16 next_to_watch;
+	unsigned int segs;
+	unsigned int bytecount;
+	u16 mapped_as_page;
 };
 
 struct e1000_ring {
@@ -158,6 +152,13 @@ struct e1000_ring {
 	/* array of buffer information structs */
 	struct e1000_buffer *buffer_info;
 
+	/* libeth fill queue backing the Rx path */
+	struct page_pool *pp;
+	struct libeth_fqe *rx_fqes;
+	u32 rx_truesize;
+	u32 rx_buf_len;
+	u32 rx_fq_mtu;			/* MTU the fill queue was sized for */
+
 	char name[IFNAMSIZ + 5];
 	u32 ims_val;
 	u32 itr_val;
@@ -192,7 +193,6 @@ struct e1000_adapter {
 
 	unsigned long active_vlans[BITS_TO_LONGS(VLAN_N_VID)];
 	u32 bd_number;
-	u32 rx_buffer_len;
 	u16 mng_vlan_id;
 	u16 link_speed;
 	u16 link_duplex;
@@ -491,8 +491,6 @@ void e1000e_get_hw_control(struct e1000_adapter *adapter);
 void e1000e_release_hw_control(struct e1000_adapter *adapter);
 void e1000e_write_itr(struct e1000_adapter *adapter, u32 itr);
 
-extern unsigned int copybreak;
-
 extern const struct e1000_info e1000_82571_info;
 extern const struct e1000_info e1000_82572_info;
 extern const struct e1000_info e1000_82573_info;
drivers/net/ethernet/intel/e1000e/netdev.c +316 -251

795 lines not on this page · raw message

drivers/net/ethernet/intel/e1000e/param.c +0 -6
index 3132d8f2f207..bf8a006686d5 100644
@@ -16,12 +16,6 @@
 #define OPTION_DISABLED 0
 #define OPTION_ENABLED  1
 
-#define COPYBREAK_DEFAULT 256
-unsigned int copybreak = COPYBREAK_DEFAULT;
-module_param(copybreak, uint, 0644);
-MODULE_PARM_DESC(copybreak,
-		 "Maximum size of packet that is copied to a new buffer on receive");
-
 /* All parameters are treated the same, as an integer array of values.
  * This macro just reduces the need to repeat the same declaration code
  * over and over (plus this helps to avoid typo bugs).
Signature · 1 line
2.43.0
#13 lore raw

PATCH 8/8 e1000e: return skbs to NAPI cache

In reply to Matt Vollrath

Use napi_consume_skb to hand skbs back to the NAPI cache. This closes the loop with changes to the Rx path.

Assisted-by Claude:claude-5-fable
---
 drivers/net/ethernet/intel/e1000e/netdev.c | 18 ++++++++++--------
 1 file changed, 10 insertions(+), 8 deletions(-)
drivers/net/ethernet/intel/e1000e/netdev.c +10 -8
index c906175ad38d..a28ee0750f4b 100644
@@ -713,7 +713,7 @@ static struct sk_buff *e1000_build_rx_skb(const struct libeth_fqe *fqe,
 
 static void e1000_put_txbuf(struct e1000_ring *tx_ring,
 			    struct e1000_buffer *buffer_info,
-			    bool drop)
+			    bool drop, int budget)
 {
 	struct e1000_adapter *adapter = tx_ring->adapter;
 
@@ -730,7 +730,7 @@ static void e1000_put_txbuf(struct e1000_ring *tx_ring,
 		if (drop)
 			dev_kfree_skb_any(buffer_info->skb);
 		else
-			dev_consume_skb_any(buffer_info->skb);
+			napi_consume_skb(buffer_info->skb, budget);
 		buffer_info->skb = NULL;
 	}
 	buffer_info->time_stamp = 0;
@@ -860,11 +860,12 @@ static void e1000e_tx_hwtstamp_work(struct work_struct *work)
 /**
  * e1000_clean_tx_irq - Reclaim resources after transmit completes
  * @tx_ring: Tx descriptor ring
+ * @napi_budget: NAPI polling budget, or 0 when called outside NAPI context
  *
  * the return value indicates whether actual cleaning was done, there
  * is no guarantee that everything was cleaned
  **/
-static bool e1000_clean_tx_irq(struct e1000_ring *tx_ring)
+static bool e1000_clean_tx_irq(struct e1000_ring *tx_ring, int napi_budget)
 {
 	struct e1000_adapter *adapter = tx_ring->adapter;
 	struct net_device *netdev = adapter->netdev;
@@ -899,7 +900,8 @@ static bool e1000_clean_tx_irq(struct e1000_ring *tx_ring)
 				}
 			}
 
-			e1000_put_txbuf(tx_ring, buffer_info, false);
+			e1000_put_txbuf(tx_ring, buffer_info, false,
+					napi_budget);
 			tx_desc->upper.data = 0;
 
 			i++;
@@ -1340,7 +1342,7 @@ static irqreturn_t e1000_intr_msix_tx(int __always_unused irq, void *data)
 	adapter->total_tx_bytes = 0;
 	adapter->total_tx_packets = 0;
 
-	if (!e1000_clean_tx_irq(tx_ring))
+	if (!e1000_clean_tx_irq(tx_ring, 0))
 		/* Ring was not completely cleaned, so fire another interrupt */
 		ew32(ICS, tx_ring->ims_val);
 
@@ -1816,7 +1818,7 @@ static void e1000_clean_tx_ring(struct e1000_ring *tx_ring)
 
 	for (i = 0; i < tx_ring->count; i++) {
 		buffer_info = &tx_ring->buffer_info[i];
-		e1000_put_txbuf(tx_ring, buffer_info, false);
+		e1000_put_txbuf(tx_ring, buffer_info, false, 0);
 	}
 
 	netdev_reset_queue(adapter->netdev);
@@ -2060,7 +2062,7 @@ static int e1000e_poll(struct napi_struct *napi, int budget)
 
 	if (!adapter->msix_entries ||
 	    (adapter->rx_ring->ims_val & adapter->tx_ring->ims_val))
-		tx_cleaned = e1000_clean_tx_irq(adapter->tx_ring);
+		tx_cleaned = e1000_clean_tx_irq(adapter->tx_ring, budget);
 
 	e1000_clean_rx_irq(adapter->rx_ring, &work_done, budget);
 
@@ -5019,7 +5021,7 @@ static int e1000_tx_map(struct e1000_ring *tx_ring, struct sk_buff *skb,
 			i += tx_ring->count;
 		i--;
 		buffer_info = &tx_ring->buffer_info[i];
-		e1000_put_txbuf(tx_ring, buffer_info, true);
+		e1000_put_txbuf(tx_ring, buffer_info, true, 0);
 	}
 
 	return 0;
Signature · 1 line
2.43.0
Reviewed-by on v1 1/8
#2 lore raw

RE: [PATCH iwl-next 1/8] e1000e: add jumbo Rx CRC stripping

In reply to Matt Vollrath
Quoted 23 lines

-----Original Message-----
From: Matt Vollrath <[email protected]>
Sent: Monday, August 31, 2026 1:22 AM
To: [email protected]
Cc: [email protected]; Nguyen, Anthony L <[email protected]>; Kitszel, Przemyslaw <[email protected]>; Lobakin, Aleksander <[email protected]>; Andrew Lunn <[email protected]>; David S . Miller <[email protected]>; Eric Dumazet <[email protected]>; Jakub Kicinski <[email protected]>; Paolo Abeni <[email protected]>; Simon Horman <[email protected]>; Jonathan Corbet <[email protected]>; Shuah Khan <[email protected]>; linux- [email protected]; [email protected]; Matt Vollrath <[email protected]>
Subject: [PATCH iwl-next 1/8] e1000e: add jumbo Rx CRC stripping

When the hardware is configured to not strip the FCS (CrcStripping=0), the jumbo Rx path would deliver frames with the FCS still attached. Follow the standard path's convention by stripping the FCS at EOP and leaving it out of the byte count.

Quoted 4 lines

The jumbo path was omitted when the CrcStripping feature was added. Since v3.3[1], the jumbo path has only been used where the page size is over 16K, which practically rules out LOMs, BMC sideband, and the need to leave FCS on frames. The bug is only reachable only by setting

"only" is duplicated

Quoted 52 lines

CrcStripping=0 on a platform with nothing that needs it.

This change prepares for convergence of Rx onto the jumbo path in following patches. The set of h/w routed through this path will expand to include LOMs.

[1] Commit 79d4e9087a6e ("e1000e: disable Early Receive DMA on ICH LOMs")

Fixes eb7c3adb1ca9 ("e1000e: fix IPMI traffic")
Assisted-by Claude:claude-5-fable
---
 drivers/net/ethernet/intel/e1000e/netdev.c | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)
drivers/net/ethernet/intel/e1000e/netdev.c +6 -1
index 844f31ab37ad..599600ad695c 100644
@@ -1631,13 +1631,23 @@ static bool e1000_clean_jumbo_rx_irq(struct
 e1000_ring *rx_ring, int *work_done,
 			}
 		}
 
+		/* strip the Ethernet CRC; it may span fragments */
+		if (!(adapter->flags2 & FLAG2_CRC_STRIPPING) &&
+		    !(netdev->features & NETIF_F_RXFCS))
+			pskb_trim(skb, skb->len - 4);
+
 		/* Receive Checksum Offload */
 		e1000_rx_checksum(adapter, staterr, skb);
 
 		e1000_rx_hash(netdev, rx_desc->wb.lower.hi_dword.rss,
 skb);
 
-		/* probably a little skewed due to removing CRC */
 		total_rx_bytes += skb->len;
+		/* If configured to store CRC, keep the FCS bytes out of

the

drivers/net/ethernet/intel/e1000e/netdev.c +5 -0
+		 * total_rx_bytes counter
+		 */
+		if (!(adapter->flags2 & FLAG2_CRC_STRIPPING) &&
+		    (netdev->features & NETIF_F_RXFCS))
+			total_rx_bytes -= 4;
 		total_rx_packets++;
 
 		/* eth type trans needs skb->data to point to something

*/
--
2.43.0

reviewer
#3 lore raw

Re: [PATCH iwl-next 1/8] e1000e: add jumbo Rx CRC stripping

In reply to Matt Vollrath

This is an AI-generated review of your patch. The human sending this
email has considered the AI review valid, or at least plausible.
Full review at: https://sashiko.dev/#/patchset/[email protected]?part=1
---
e1000e: add jumbo Rx CRC stripping

When the hardware is configured to not strip the FCS (CrcStripping=0), the jumbo Rx path would deliver frames with the FCS still attached. Follow the standard path's convention by stripping the FCS at EOP and leaving it out of the byte count.

Quoted 16 lines
drivers/net/ethernet/intel/e1000e/netdev.c +5 -0
index 844f31ab37ad4..599600ad695cd 100644
@@ -1631,13 +1631,23 @@ static bool e1000_clean_jumbo_rx_irq(struct e1000_ring *rx_ring, int *work_done,
 			}
 		}
 
+		/* strip the Ethernet CRC; it may span fragments */
+		if (!(adapter->flags2 & FLAG2_CRC_STRIPPING) &&
+		    !(netdev->features & NETIF_F_RXFCS))
+			pskb_trim(skb, skb->len - 4);
+
 		/* Receive Checksum Offload */
 		e1000_rx_checksum(adapter, staterr, skb);
 

drivers/net/ethernet/intel/e1000e/netdev.c static bool e1000_clean_jumbo_rx_irq(struct e1000_ring *rx_ring, int *work_done, · line 1650

Quoted 10 lines
drivers/net/ethernet/intel/e1000e/netdev.c +6 -1
 		e1000_rx_hash(netdev, rx_desc->wb.lower.hi_dword.rss, skb);
 
-		/* probably a little skewed due to removing CRC */
 		total_rx_bytes += skb->len;
+		/* If configured to store CRC, keep the FCS bytes out of the
+		 * total_rx_bytes counter
+		 */
+		if (!(adapter->flags2 & FLAG2_CRC_STRIPPING) &&
+		    (netdev->features & NETIF_F_RXFCS))
+			total_rx_bytes -= 4;

[Severity: Medium]
Could these lockless evaluations of adapter->flags2 and netdev->features get out of sync if userspace changes features via ethtool while a packet is being processed?

If a concurrent change occurs between the first check and this second check, it appears it could lead to an inconsistent state where a packet's payload is incorrectly truncated by 4 bytes, or where the byte statistics are overcounted or undercounted by 4 bytes.

Quoted 3 lines
drivers/net/ethernet/intel/e1000e/netdev.c +0 -0
 		total_rx_packets++;
 
 		/* eth type trans needs skb->data to point to something */
reviewer
#10 lore raw

Re: [PATCH iwl-next 6/8] e1000e: disable NAPI while interface is down

In reply to Matt Vollrath

This is an AI-generated review of your patch. The human sending this
email has considered the AI review valid, or at least plausible.
Full review at: https://sashiko.dev/#/patchset/[email protected]?part=6
---
e1000e: disable NAPI while interface is down

This commit moves napi_disable into e1000e_down and adds napi_enable to e1000e_up to ensure NAPI is disabled during reconfiguration windows.

Quoted 4 lines
drivers/net/ethernet/intel/e1000e/netdev.c +0 -0
index 194166531bc84..6d846cacb026e 100644

[ ... ]

drivers/net/ethernet/intel/e1000e/netdev.c void e1000e_down(struct e1000_adapter *adapter, bool reset) · line 3606

Quoted 6 lines
drivers/net/ethernet/intel/e1000e/netdev.c +1 -1
@@ -3601,7 +3603,7 @@ void e1000e_down(struct e1000_adapter *adapter, bool reset)
 
 	e1000_irq_disable(adapter);
 
-	napi_synchronize(&adapter->napi);
+	napi_disable(&adapter->napi);

[Severity: High]
Does this change introduce an infinite hang in napi_disable() due to concurrent or duplicate e1000e_down() calls?

A runtime PM suspend callback (e1000e_pm_runtime_suspend) can execute concurrently with an ethtool operation like e1000_set_ringparam. The ethtool operation sets the __E1000_RESETTING bit and holds it while allocating resources.

The PM suspend callback waits on this bit but uses a bounded timeout (300ms). If the ethtool operation takes longer than 300ms, the PM callback times out, prints a WARN_ON, and unconditionally calls e1000e_down():

e1000e_pm_runtime_suspend() {
        while (test_bit(__E1000_RESETTING, &adapter->state) && count--)
                usleep_range(10000, 11000);

        WARN_ON(test_bit(__E1000_RESETTING, &adapter->state));

        /* Down the device without resetting the hardware */
        e1000e_down(adapter, false);
}

Because the ethtool operation also called (or is calling) e1000e_down(), napi_disable() is executed either concurrently or sequentially on an already-disabled NAPI struct, leading to an infinite hang waiting for NAPI_STATE_SCHED.

[Severity: High]
Could this also lead to a sequential double napi_disable() hang due to state mismatches where e1000e_down() is called but netif_running() remains true?

The e1000e_pm_thaw() function attempts to bring the device back up after a freeze/suspend. If e1000_request_irq() fails, it exits via the err_irq label without calling e1000e_up(), leaving NAPI disabled:

e1000e_pm_thaw() {
        rc = e1000_request_irq(adapter);
        if (rc)
                goto err_irq;

        e1000e_up(adapter);
}

However, the device's IFF_UP flag is untouched, so netif_running() remains true. Subsequently, if an administrator changes the MTU via e1000_change_mtu(), the driver checks netif_running() and blindly calls e1000e_down():

e1000_change_mtu() {
        if (netif_running(netdev))
                e1000e_down(adapter, true);
}

This invokes napi_disable() a second time sequentially, which hangs indefinitely because the NAPI instance was never re-enabled.

Quoted 3 lines
drivers/net/ethernet/intel/e1000e/netdev.c +0 -0
 	timer_delete_sync(&adapter->watchdog_timer);
 	timer_delete_sync(&adapter->phy_info_timer);
#11 lore raw

Re: [PATCH iwl-next 6/8] e1000e: disable NAPI while interface is down

Quoted 25 lines · Simon Horman

This is an AI-generated review of your patch. The human sending this
email has considered the AI review valid, or at least plausible.
Full review at: https://sashiko.dev/#/patchset/[email protected]?part=6
---
e1000e: disable NAPI while interface is down

This commit moves napi_disable into e1000e_down and adds napi_enable to e1000e_up to ensure NAPI is disabled during reconfiguration windows.

Quoted 4 lines
drivers/net/ethernet/intel/e1000e/netdev.c +0 -0
index 194166531bc84..6d846cacb026e 100644

[ ... ]

drivers/net/ethernet/intel/e1000e/netdev.c void e1000e_down(struct e1000_adapter *adapter, bool reset) · line 3606

Quoted 6 lines
drivers/net/ethernet/intel/e1000e/netdev.c +1 -1
@@ -3601,7 +3603,7 @@ void e1000e_down(struct e1000_adapter *adapter, bool reset)
 
  	e1000_irq_disable(adapter);
 
-	napi_synchronize(&adapter->napi);
+	napi_disable(&adapter->napi);

[Severity: High]
Does this change introduce an infinite hang in napi_disable() due to concurrent or duplicate e1000e_down() calls?

Quoted 4 lines

A runtime PM suspend callback (e1000e_pm_runtime_suspend) can execute concurrently with an ethtool operation like e1000_set_ringparam. The ethtool operation sets the __E1000_RESETTING bit and holds it while allocating resources.

No, ethtool and runtime PM suspend are synchronized:
* ethtool core calls pm_runtime_get_sync() before any operation. This increments dev->power.usage_count and waits for any ongoing suspend to finish.
* PM core rpm_suspend exits without retry (rpm_check_suspend_allowed()).

Quoted 41 lines

The PM suspend callback waits on this bit but uses a bounded timeout (300ms). If the ethtool operation takes longer than 300ms, the PM callback times out, prints a WARN_ON, and unconditionally calls e1000e_down():

e1000e_pm_runtime_suspend() {
         while (test_bit(__E1000_RESETTING, &adapter->state) && count--)
                 usleep_range(10000, 11000);

         WARN_ON(test_bit(__E1000_RESETTING, &adapter->state));

         /* Down the device without resetting the hardware */
         e1000e_down(adapter, false);
}

Because the ethtool operation also called (or is calling) e1000e_down(), napi_disable() is executed either concurrently or sequentially on an already-disabled NAPI struct, leading to an infinite hang waiting for NAPI_STATE_SCHED.

[Severity: High]
Could this also lead to a sequential double napi_disable() hang due to state mismatches where e1000e_down() is called but netif_running() remains true?

The e1000e_pm_thaw() function attempts to bring the device back up after a freeze/suspend. If e1000_request_irq() fails, it exits via the err_irq label without calling e1000e_up(), leaving NAPI disabled:

e1000e_pm_thaw() {
         rc = e1000_request_irq(adapter);
         if (rc)
                 goto err_irq;

         e1000e_up(adapter);
}

However, the device's IFF_UP flag is untouched, so netif_running() remains true. Subsequently, if an administrator changes the MTU via e1000_change_mtu(), the driver checks netif_running() and blindly calls e1000e_down():

Quoted 4 lines
e1000_change_mtu() {
         if (netif_running(netdev))
                 e1000e_down(adapter, true);
}

Core does not call ndo_change_mtu on detached devices, it checks netif_device_present() in the typical path. Bonding and team paths will close the device before changing MTU.

Same netif_device_present() check in ethtool core and e1000e_pm_freeze().

However, the PM runtime suspend and resume ops only check IFF_UP and not netif_device_present(). PM core does not check this or gate it on a known thaw failure (by design). That is a real pre-existing bug made more consequential by this change.

Quoted 7 lines

This invokes napi_disable() a second time sequentially, which hangs indefinitely because the NAPI instance was never re-enabled.

Quoted 3 lines
drivers/net/ethernet/intel/e1000e/netdev.c +0 -0
  	timer_delete_sync(&adapter->watchdog_timer);
  	timer_delete_sync(&adapter->phy_info_timer);
#4 lore raw

Re: [PATCH iwl-next 1/8] e1000e: add jumbo Rx CRC stripping

In reply to Simon Horman
Quoted 29 lines · Simon Horman

This is an AI-generated review of your patch. The human sending this
email has considered the AI review valid, or at least plausible.
Full review at: https://sashiko.dev/#/patchset/[email protected]?part=1
---
e1000e: add jumbo Rx CRC stripping

When the hardware is configured to not strip the FCS (CrcStripping=0), the jumbo Rx path would deliver frames with the FCS still attached. Follow the standard path's convention by stripping the FCS at EOP and leaving it out of the byte count.

Quoted 18 lines
drivers/net/ethernet/intel/e1000e/netdev.c +5 -0
index 844f31ab37ad4..599600ad695cd 100644
@@ -1631,13 +1631,23 @@ static bool e1000_clean_jumbo_rx_irq(struct e1000_ring *rx_ring, int *work_done,
  			}
  		}
 
+		/* strip the Ethernet CRC; it may span fragments */
+		if (!(adapter->flags2 & FLAG2_CRC_STRIPPING) &&
+		    !(netdev->features & NETIF_F_RXFCS))
+			pskb_trim(skb, skb->len - 4);
+
  		/* Receive Checksum Offload */
  		e1000_rx_checksum(adapter, staterr, skb);
 
  		e1000_rx_hash(netdev, rx_desc->wb.lower.hi_dword.rss, skb);
 
Quoted 13 lines

drivers/net/ethernet/intel/e1000e/netdev.c static bool e1000_clean_jumbo_rx_irq(struct e1000_ring *rx_ring, int *work_done, · line 1650

Quoted 8 lines
drivers/net/ethernet/intel/e1000e/netdev.c +6 -1
-		/* probably a little skewed due to removing CRC */
  		total_rx_bytes += skb->len;
+		/* If configured to store CRC, keep the FCS bytes out of the
+		 * total_rx_bytes counter
+		 */
+		if (!(adapter->flags2 & FLAG2_CRC_STRIPPING) &&
+		    (netdev->features & NETIF_F_RXFCS))
+			total_rx_bytes -= 4;

[Severity: Medium]
Could these lockless evaluations of adapter->flags2 and netdev->features get out of sync if userspace changes features via ethtool while a packet is being processed?

Yes, this could be prevented by bringing down the adapter before changing bits in e1000_set_features().

Quoted 9 lines

If a concurrent change occurs between the first check and this second check, it appears it could lead to an inconsistent state where a packet's payload is incorrectly truncated by 4 bytes, or where the byte statistics are overcounted or undercounted by 4 bytes.

Quoted 3 lines
drivers/net/ethernet/intel/e1000e/netdev.c +0 -0
  		total_rx_packets++;
 
  		/* eth type trans needs skb->data to point to something */