Skip to content

e1000e: use page pool

Matt Vollrath intel-wired-lan linux-doc lkml netdev stable 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 Latest8 patches 5 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/8e1000e: add jumbo Rx CRC strippingdrivers/net/ethernet/intel/e1000e/netdev.c+11 -13 replies
2/8e1000e: dump pages for jumbo Rx buffersdrivers/net/ethernet/intel/e1000e/netdev.c+16 -10 replies
3/8e1000e: prevent race between PM and reset taskdrivers/net/ethernet/intel/e1000e/netdev.c+21 -40 replies
4/8e1000e: remove packet-split Rx pathdrivers/net/ethernet/intel/e1000e/e1000.h drivers/net/ethernet/intel/e1000e/ethtool.c drivers/net/ethernet/intel/e1000e/netdev.c+81 -5730 replies
5/8e1000e: always use jumbo Rx pathdrivers/net/ethernet/intel/e1000e/e1000.h drivers/net/ethernet/intel/e1000e/netdev.c+18 -3000 replies
6/8e1000e: disable NAPI while interface is downdrivers/net/ethernet/intel/e1000e/netdev.c+3 -22 replies
7/8e1000e: use libeth page_pool for RxDocumentation/networking/device_drivers/ethernet/intel/e1000e.rst drivers/net/ethernet/intel/Kconfig drivers/net/ethernet/intel/e1000e/e1000.h +2 more+332 -2890 replies
8/8e1000e: return skbs to NAPI cachedrivers/net/ethernet/intel/e1000e/netdev.c+10 -80 replies

Reviews trailers per patch

1 of 8 patches has a Reviewed-by or Acked-by.

PatchAleksandr Loktionov
1/8R-b
2/8
3/8
4/8
5/8
6/8
7/8
8/8

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v1 thread

Cover letter [PATCH iwl-next 0/8] e1000e: use page pool

This series converges the three e1000e Rx paths into one and then makes it use a page pool. The jumbo path is chosen as the golden path because it is the only one that can handle every combination of MTU and page size. Any performance regression caused by using that path for all configurations is remedied by eliminating allocations and DMA mapping from the hot path, as shown in benchmarks below.

There are some general modernizations in this conversion, intended to make this driver match conventions found in other Intel drivers. The copybreak parameter is removed. The NAPI skb cache is used. The netmem API is used for chained frames. Prefetch is used optimistically in the cleaner loop to pull in the next descriptor and the frame headers.

User-visible changes:
* The copybreak module parameter and its documentation are removed.
* The rx_header_split ethtool stat is removed.
* Small packets are charged at the buffer's full 2 KB truesize at standard MTU.
* Jumbo MTU frames are delivered as 2 KB chunks. At MTU > 2026 this increases userspace copy overhead by gathering more fragments and increases truesize accounting per frame. This is described in more detail below.

The series begins by fixing some existing bugs which would either be more reachable or have worse consequences after the conversion. Missing support for CrcStripping=0 is added. The page dump routine's mis-read of jumbo buffers is remedied. A race between the reset task and runtime PM is guarded.

Then the standard and packet-split paths are removed, making the jumbo path the only way to Rx. At this point standard-MTU Rx allocates and maps a page per packet, until the page pool conversion removes that cost.

Next NAPI is disabled while the adapter is down, which is required to pass assertions when the page pool is destroyed.

Next is the main event, the singular Rx path is converted to use a page pool with support from libeth.

Finally, a small change to Tx to return skbs to the NAPI cache, which the Rx side now pulls from when constructing frame heads.

Benchmarks were run on an I218-V and Xeon E3-1240v3 pinned at 3.8GHz. NAPI was isolated on one core and iperf3 on another. ITR was fixed at 4000/s because the adaptive scaling can take a few minutes to settle (to be fixed separately). C-states above C1 and EEE were disabled. Kernel version was 7.1.12. Direct link from sender to receiver.

The sender was an 82574L in a Xeon E3-1270 pinned at 3.4GHz, NIC vectors and iperf3 -s pinned to separate cores, ITR fixed.

For cycle measurements, each cell is perf stat cycles on the core divided by packets received over 30s, averaged over three runs. IOMMU "lazy" is the default DMA-FQ. Throughput was identical between stock and series, with one exception noted below. The stock driver is configured at the default copybreak=256. Note that the stock copybreak features is flawed; it always remaps DMA when sending an already-mapped buffer back to the h/w.

Bulk TCP, segments sized to the MTU (line rate at both MTUs):

NAPI core cycles/pkt consumer core cycles/pkt

  MTU   IOMMU   kpps    stock  series  change   stock  series  change
  1500  off     81.4     2523    1784    -29%    2193    2193      0%
  1500  lazy    81.4     5170    1938    -63%    2369    2323     -2%
  9000  off     13.9    11590    7574    -35%   11764   12792     +9%
  9000  lazy    13.9    21781    8423    -61%   12129   13396    +10%

The increase in consumer cost at MTU 9000 is probably because the frame arrives as five 2 KB chunks instead of three pieces from the legacy packet-split path. The copy to userspace walks more fragments and GRO merges fewer frames per skb.

TCP with an 88-byte MSS (iperf3 -M 88, ~142-byte frames):

NAPI core cycles/pkt consumer core cycles/pkt

  MTU   IOMMU   kpps    stock  series  change   stock  series  change
  1500  off    754.3     1453     793    -45%     446     484     +8%
  1500  lazy   754.3     3819     809    -79%     387     496    +28%
  9000  off    754.4     2231     780    -65%     498     509     +2%
  9000  lazy 520/754     6920     798    -88%     430     523    +22%

At MTU 9000 with IOMMU on, the stock driver can't keep up with the stream at line rate. It polls continuously at 95% of the core and can't send descriptors back to h/w fast enough, so it sends pause frames and flow control throttles the sender.

Differences in consumer cost are likely explained by two things:
* The stock copybreak=256 leaves the payload in the NAPI core's cache.
* The high IOMMU cost in the stock driver causes it to return more bytes per read.

UDP, one full-size datagram per frame (1472/8972 bytes, line rate):

NAPI core cycles/pkt consumer core cycles/pkt

  MTU   IOMMU   kpps    stock  series  change   stock  series  change
  1500  off     81.4     4991    4417    -11%    6409    6381      0%
  1500  lazy    81.4     7426    4397    -41%    6619    6395     -3%
  9000  off     13.9    12070    8286    -31%   12861   13624     +6%
  9000  lazy    13.9    21494    8289    -61%   12566   13772    +10%

Here we see a similar tax on the jumbo MTU consumer due to gathering more fragments.

UDP, 64-byte datagrams paced at 150 kpps:

NAPI core cycles/pkt consumer core cycles/pkt

  MTU   IOMMU   kpps    stock  series  change   stock  series  change
  1500  off    150.2     4202    3939     -6%    5541    5525      0%
  1500  lazy   150.3     6663    3919    -41%    6151    5465    -11%
  9000  off    150.2     4376    3956    -10%    5535    5428     -2%
  9000  lazy   150.2     6821    3953    -42%    6154    5453    -11%

At MTU 9000, this rate of datagrams runs up against the socket receive buffer's limits in the series, causing a 0.003% drop rate. Because each datagram sits in a full page buffer, the default 213 KB socket buffer only holds 48 vs. stock's ~275. This can be remedied by increasing SO_RCVBUF or rmem_default. This is another example of copybreak truesize compression value in the stock driver.

Payload Density:

This table shows pieces per frame and density at every MTU breakpoint. Stock is the standard path at 1500 and packet-split above it.

stock (std / packet split) series (2 KB chunks)
MTU pieces truesize density pieces truesize density

<= 1500           1     2688      0.56         1     2304      0.66
1501-2026         1     4864   0.31-0.42       1     4352   0.35-0.47
2027-4074         1     4864   0.42-0.84       2     8448   0.24-0.48
4075-4096         1     4864      0.84         3    12544      0.33
4097-6122         2     8960   0.46-0.68       3    12544   0.33-0.49
6123-8170         2     8960   0.68-0.91       4    16640   0.37-0.49
8171-8192         2     8960   0.91-0.92       5    20736   0.39-0.40
8193-9212         3    13056   0.63-0.71       5    20736   0.40-0.44

Series chunk counts assume the worst-case frame (MTU + 22: VLAN tag and FCS present).

This series is hungrier for socket buffer size under two conditions:
* At higher-than-standard MTU, some page size is wasted because RCTL.BSIZE only accepts powers of two, unlike the more granular settings of later h/w.
* For small packets at any MTU, copybreak no longer moves the payload into an appropriately-sized buffer.

Matt Vollrath (8):
e1000e: add jumbo Rx CRC stripping
e1000e: dump pages for jumbo Rx buffers
e1000e: prevent race between PM and reset task
e1000e: remove packet-split Rx path
e1000e: always use jumbo Rx path
e1000e: disable NAPI while interface is down
e1000e: use libeth page_pool for Rx
e1000e: return skbs to NAPI cache

 .../device_drivers/ethernet/intel/e1000e.rst  |   15 -
 drivers/net/ethernet/intel/Kconfig            |    1 +
 drivers/net/ethernet/intel/e1000e/e1000.h     |   53 +-
 drivers/net/ethernet/intel/e1000e/ethtool.c   |    1 -
 drivers/net/ethernet/intel/e1000e/netdev.c    | 1374 +++++------------
 drivers/net/ethernet/intel/e1000e/param.c     |    6 -
 6 files changed, 382 insertions(+), 1068 deletions(-)

base-commit: 1b78070aaef63512688aebfbc82365ef9d6660f1

Signature · 1 line
2.43.0