Skip to contentSkip to conversation

net/af_packet: cleanups and optimizations

RFC In review Stephen Hemminger 3 participants 18 messages Jan 28 – Feb 3

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover net/af_packet: cleanups and optimizations

This series contains several improvements to the AF_PACKET PMD:

1. Remove unnecessary volatile qualifiers from statistics counters and standardize on uint64_t for consistency.

2. Add a unit test for the AF_PACKET PMD. The test creates a TAP interface and uses it to verify basic send/receive functionality, statistics, and device operations.

3. Implement VPP-style single/dual/quad loop pattern with software prefetching in the receive path. This optimization processes packets in batches of 4, 2, and 1, prefetching upcoming frame headers and packet data to improve cache utilization and hide memory latency.

Note: The test code in patch 2 and the prefetch optimization in patch 4 were generated with the assistance of Claude AI (Anthropic), with manual review and cleanup afterwards.

The prefetch pattern is modeled after FD.IO VPP's packet processing loops, which have proven effective in high-performance networking applications. The key techniques are:
- Loop unrolling to reduce branch overhead
- Prefetching next iteration's data while processing current batch
- Graceful fallback from quad to dual to single packet processing

Testing was performed using the new unit test on a TAP interface. Performance testing on real hardware with high packet rates would be valuable to quantify the prefetch benefits.

Stephen Hemminger (4):
net/af_packet: remove volatile from statistics
test: add test for af_packet
net/af_packet: fix indentation
net/af_packet: add VPP-style prefetching to receive path

 app/test/meson.build                      |   1 +
 app/test/test_pmd_af_packet.c             | 875 ++++++++++++++++++++++
 drivers/net/af_packet/rte_eth_af_packet.c | 286 +++++--
 3 files changed, 1079 insertions(+), 83 deletions(-)
 create mode 100644 app/test/test_pmd_af_packet.c
Signature · 1 line
2.51.0
#1 lore raw

RFC 1/4 net/af_packet: remove volatile from statistics

Statistics are only updated from a single thread at a time and the compiler should not have to worry about optimizing them. The statistics returned are 64 bit so use uint64_t instead of arch specific unsigned long.

---
 drivers/net/af_packet/rte_eth_af_packet.c | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)
drivers/net/af_packet/rte_eth_af_packet.c +7 -7
index ef11b8fb6b..158393dd70 100644
@@ -57,10 +57,10 @@ struct __rte_cache_aligned pkt_rx_queue {
 	uint8_t vlan_strip;
 	uint8_t timestamp_offloading;
 
-	volatile unsigned long rx_pkts;
-	volatile unsigned long rx_bytes;
-	volatile unsigned long rx_nombuf;
-	volatile unsigned long rx_dropped_pkts;
+	uint64_t rx_pkts;
+	uint64_t rx_bytes;
+	uint64_t rx_nombuf;
+	uint64_t rx_dropped_pkts;
 };
 
 struct __rte_cache_aligned pkt_tx_queue {
@@ -72,9 +72,9 @@ struct __rte_cache_aligned pkt_tx_queue {
 	unsigned int framecount;
 	unsigned int framenum;
 
-	volatile unsigned long tx_pkts;
-	volatile unsigned long err_pkts;
-	volatile unsigned long tx_bytes;
+	uint64_t tx_pkts;
+	uint64_t tx_bytes;
+	uint64_t err_pkts;
 };
 
 struct pmd_internals {
Signature · 1 line
2.51.0
#2 lore raw

Re: [RFC 1/4] net/af_packet: remove volatile from statistics

Can you clarify why removal of volatile is safe (visibility, ordering) when the producer thread and consumer thread can be different?

Quoted 46 lines · Stephen Hemminger

Statistics are only updated from a single thread at a time and the compiler should not have to worry about optimizing them. The statistics returned are 64 bit so use uint64_t instead of arch specific unsigned long.

---
 drivers/net/af_packet/rte_eth_af_packet.c | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)
drivers/net/af_packet/rte_eth_af_packet.c +7 -8
index ef11b8fb6b..158393dd70 100644
@@ -57,10 +57,10 @@ struct __rte_cache_aligned pkt_rx_queue {
        uint8_t vlan_strip;
        uint8_t timestamp_offloading;
 
-       volatile unsigned long rx_pkts;
-       volatile unsigned long rx_bytes;
-       volatile unsigned long rx_nombuf;
-       volatile unsigned long rx_dropped_pkts;
+       uint64_t rx_pkts;
+       uint64_t rx_bytes;
+       uint64_t rx_nombuf;
+       uint64_t rx_dropped_pkts;
 };
 
 struct __rte_cache_aligned pkt_tx_queue {
@@ -72,9 +72,9 @@ struct __rte_cache_aligned pkt_tx_queue {
        unsigned int framecount;
        unsigned int framenum;
 
-       volatile unsigned long tx_pkts;
-       volatile unsigned long err_pkts;
-       volatile unsigned long tx_bytes;
+       uint64_t tx_pkts;
+       uint64_t tx_bytes;
+       uint64_t err_pkts;
 };
 
 struct pmd_internals {
--

2.51.0

#3 lore raw

Re: [RFC 1/4] net/af_packet: remove volatile from statistics

Quoted 2 lines · On Wed, 28 Jan 2026 11:57:16 -0800 Scott Mitchell <[email protected]>…

Can you clarify why removal of volatile is safe (visibility, ordering) when the producer thread and consumer thread can be different?

The short version is, volatile protects against compiler reordering code where it thinks variable can't change. The atomic primitives protect against load/store ordering issues on non-cache coherent CPU architectures. The DPDK design pattern has been that statistics don't need to be protected since producer is single thread and consumer should be ok with slightly out of date data. The point is that using atomic requires going out of cache on ARM (where it would matter) and on x86 locked operations cause a pipeline stall. At the high data rates in DPDK any stall or cache miss matters.

The one exception would be drivers that advertise lock free transmit would need to use atomic on the Tx stats to avoid multi-producer problems.

#4 lore raw

Re: [RFC 1/4] net/af_packet: remove volatile from statistics

Without volatile, is there any guarantee the consumer thread will see the writes from the producer thread? For example, couldn't the compiler cache rx_pkts in a register on the consumer side? // Consumer thread:
uint64_t prev = 0;

while (monitoring) {
    uint64_t curr = rx_pkts;  // Compiler may cache in register
    if (curr != prev) {       // Without volatile: may always be false
      update_display();       // Never executes
      prev = curr;
    }
}

I created a godbolt comparing different approaches
(https://godbolt.org/z/1Gaz6jPxh) showing assembly for x86-64 and ARM at -O3. Summary:
1. Plain load/store: Fastest. No visibility guarantees. May tear on 32-bit.
2. Volatile: Provides visibility. May tear on 32-bit.
3. Atomic load/store (relaxed): Same performance as volatile. Provides visibility and guarantees no tearing (even on 32-bit).
4. Atomic_fetch_add: Heaviest cost (LOCK on x86).

My concern is that Option 1 (Plain) has no guaranteed visibility - the compiler may optimize away loads entirely. Since Option 3 (Atomic Load/Store) has identical instruction cost to Volatile but provides formal guarantees (visibility + no tearing), would that be the preferred solution?

#5 lore raw

Re: [RFC 1/4] net/af_packet: remove volatile from statistics

Quoted 22 lines · On Sun, 1 Feb 2026 23:02:53 -0800 Scott Mitchell <[email protected]>…

Without volatile, is there any guarantee the consumer thread will see the writes from the producer thread? For example, couldn't the compiler cache rx_pkts in a register on the consumer side? // Consumer thread:
uint64_t prev = 0;

while (monitoring) {
    uint64_t curr = rx_pkts;  // Compiler may cache in register
    if (curr != prev) {       // Without volatile: may always be false
      update_display();       // Never executes
      prev = curr;
    }
}

I created a godbolt comparing different approaches
(https://godbolt.org/z/1Gaz6jPxh) showing assembly for x86-64 and ARM at -O3. Summary:
1. Plain load/store: Fastest. No visibility guarantees. May tear on 32-bit.
2. Volatile: Provides visibility. May tear on 32-bit.
3. Atomic load/store (relaxed): Same performance as volatile. Provides visibility and guarantees no tearing (even on 32-bit).
4. Atomic_fetch_add: Heaviest cost (LOCK on x86).

Quoted 5 lines

My concern is that Option 1 (Plain) has no guaranteed visibility - the compiler may optimize away loads entirely. Since Option 3 (Atomic Load/Store) has identical instruction cost to Volatile but provides formal guarantees (visibility + no tearing), would that be the preferred solution?

In normal case, compiler isn't going to be able to see across function boundary especially through indirection of eth_dev_ops table. With LTO it might be possible but unlikely.

#6 lore raw

Re: [RFC 1/4] net/af_packet: remove volatile from statistics

Thank you for the clarification about the function pointer indirection acting as a compiler barrier - that makes sense for the typical case.

I have one remaining question about 32-bit architectures: even with the implicit barrier, plain uint64_t reads aren't atomic on 32-bit platforms (reader could see torn high/low halves). Is this acceptable for stats counters in DPDK, or is 32-bit support not a concern?

For context, atomic_load/store(memory_order_relaxed) formally guarantees both visibility and no tearing across architectures, with minimal (GCC) or zero (Clang) overhead for x86-64. I see there's precedent in DPDK for plain uint64_t stats, and I'm hoping to understand the assumptions/trade-offs better.

#7 lore raw

Re: [RFC 1/4] net/af_packet: remove volatile from statistics

Quoted 7 lines · On Mon, 2 Feb 2026 11:12:31 -0800 Scott Mitchell <[email protected]>…

Thank you for the clarification about the function pointer indirection acting as a compiler barrier - that makes sense for the typical case.

I have one remaining question about 32-bit architectures: even with the implicit barrier, plain uint64_t reads aren't atomic on 32-bit platforms (reader could see torn high/low halves). Is this acceptable for stats counters in DPDK, or is 32-bit support not a concern?

DPDK has not worried about torn counters on 32 bit architectures. Whether it should or not is open question.
People don't care much about 32 bit anymore; other than not breaking it.

Quoted 5 lines

For context, atomic_load/store(memory_order_relaxed) formally guarantees both visibility and no tearing across architectures, with minimal (GCC) or zero (Clang) overhead for x86-64. I see there's precedent in DPDK for plain uint64_t stats, and I'm hoping to understand the assumptions/trade-offs better.

The problems is that on relaxed ordering architectures like ARM it makes the counter update more expensive in the fast path.

#8 lore raw

RFC 2/4 test: add test for af_packet

In reply to Stephen Hemminger

Add a test for AF_PACKET PMD.
This test was generated with Claude AI based off of existing test_pmd_ring.c with some cleanup afterwards.

---
 app/test/meson.build          |   1 +
 app/test/test_pmd_af_packet.c | 875 ++++++++++++++++++++++++++++++++++
 2 files changed, 876 insertions(+)
 create mode 100644 app/test/test_pmd_af_packet.c
app/test/meson.build +1 -0
index f4d04a6e42..9fb7067b4b 100644
@@ -141,6 +141,7 @@ source_file_deps = {
     'test_per_lcore.c': [],
     'test_pflock.c': [],
     'test_pie.c': ['sched'],
+    'test_pmd_af_packet.c': ['net_af_packet', 'ethdev', 'bus_vdev'],
     'test_pmd_perf.c': ['ethdev', 'net'] + packet_burst_generator_deps,
     'test_pmd_ring.c': ['net_ring', 'ethdev', 'bus_vdev'],
     'test_pmd_ring_perf.c': ['ethdev', 'net_ring', 'bus_vdev'],
app/test/test_pmd_af_packet.c new file +875 -0

875 lines not on this page · raw message

Signature · 1 line
2.51.0
#9 lore raw

Re: [RFC 2/4] test: add test for af_packet

Awesome! Some suggestions below but could easily be done in a followup for incremental progress.

app/test/test_pmd_af_packet.c line 412

Quoted 9 lines
app/test/test_pmd_af_packet.c new file +9 -0
+       /* TX burst */
+       nb_tx = rte_eth_tx_burst(port_id, 0, bufs, BURST_SIZE);
+
+       /* Free any unsent mbufs */
+       for (i = nb_tx; i < BURST_SIZE; i++)
+               rte_pktmbuf_free(bufs[i]);
+
+       /* Small delay to allow stats update */
+       rte_delay_us_block(1000);

Consider polling rte_eth_stats_get at a more frequent interval, break when you see status update, and apply overall timeout/repeat limit -> can make test more reliable and take less time when data is usually available quickly

app/test/test_pmd_af_packet.c line 450

Quoted 9 lines
app/test/test_pmd_af_packet.c new file +9 -0
+       /* Try to receive packets (non-blocking) */
+       nb_rx = rte_eth_rx_burst(port_id, 0, bufs, BURST_SIZE);
+
+       /* Free any received mbufs */
+       for (i = 0; i < nb_rx; i++)
+               rte_pktmbuf_free(bufs[i]);
+
+       /* RX from tap interface without external traffic will return 0 */
+       /* This test just verifies the RX path doesn't crash */

Consider adding a test that round trips data (tx -> rx) to exercise the read loop.

app/test/test_pmd_af_packet.c line 539

Quoted 6 lines
app/test/test_pmd_af_packet.c new file +6 -0
+       /* Get current MTU */
+       ret = rte_eth_dev_get_mtu(port_id, &mtu);
+       TEST_ASSERT(ret == 0, "Failed to get MTU");
+
+       /* Try to set a smaller MTU */
+       ret = rte_eth_dev_set_mtu(port_id, 1400);

Consider getting the min/max supported MTU and setting it to verify it works. My patch fixed some issues related to MTU and overhead of frames that may have been caught by this.

app/test/test_pmd_af_packet.c line 343

Quoted 4 lines
app/test/test_pmd_af_packet.c new file +4 -0
+
+       /* Reset stats */
+       ret = rte_eth_stats_reset(port_id);
+       TEST_ASSERT(ret == 0, "Failed to reset stats");

Consider enhancing to introduce multiple threads (writer and reader) to replicate

Review

Quoted 2 lines
(quoted diff) +2 -0
+static int
+test_af_packet_multi_queue(void)

Consider sharing rx/tx logic in common functions for single/multi-queue cases.

#10 lore raw

Re: [RFC 2/4] test: add test for af_packet

Quoted 47 lines · On Wed, 28 Jan 2026 12:36:41 -0800 Scott Mitchell <[email protected]>…

Awesome! Some suggestions below but could easily be done in a followup for incremental progress.

app/test/test_pmd_af_packet.c line 412

Quoted 9 lines
app/test/test_pmd_af_packet.c new file +9 -0
+       /* TX burst */
+       nb_tx = rte_eth_tx_burst(port_id, 0, bufs, BURST_SIZE);
+
+       /* Free any unsent mbufs */
+       for (i = nb_tx; i < BURST_SIZE; i++)
+               rte_pktmbuf_free(bufs[i]);
+
+       /* Small delay to allow stats update */
+       rte_delay_us_block(1000);  

Consider polling rte_eth_stats_get at a more frequent interval, break when you see status update, and apply overall timeout/repeat limit -> can make test more reliable and take less time when data is usually available quickly

app/test/test_pmd_af_packet.c line 450

Quoted 9 lines
app/test/test_pmd_af_packet.c new file +9 -0
+       /* Try to receive packets (non-blocking) */
+       nb_rx = rte_eth_rx_burst(port_id, 0, bufs, BURST_SIZE);
+
+       /* Free any received mbufs */
+       for (i = 0; i < nb_rx; i++)
+               rte_pktmbuf_free(bufs[i]);
+
+       /* RX from tap interface without external traffic will return 0 */
+       /* This test just verifies the RX path doesn't crash */  

Consider adding a test that round trips data (tx -> rx) to exercise the read loop.

app/test/test_pmd_af_packet.c line 539

Quoted 6 lines
app/test/test_pmd_af_packet.c new file +6 -0
+       /* Get current MTU */
+       ret = rte_eth_dev_get_mtu(port_id, &mtu);
+       TEST_ASSERT(ret == 0, "Failed to get MTU");
+
+       /* Try to set a smaller MTU */
+       ret = rte_eth_dev_set_mtu(port_id, 1400);  

Consider getting the min/max supported MTU and setting it to verify it works. My patch fixed some issues related to MTU and overhead of frames that may have been caught by this.

app/test/test_pmd_af_packet.c line 343

Quoted 4 lines
app/test/test_pmd_af_packet.c new file +4 -0
+
+       /* Reset stats */
+       ret = rte_eth_stats_reset(port_id);
+       TEST_ASSERT(ret == 0, "Failed to reset stats");  
Quoted 7 lines

Consider enhancing to introduce multiple threads (writer and reader) to replicate

Review

Quoted 2 lines
(quoted diff) +2 -0
+static int
+test_af_packet_multi_queue(void)  

Consider sharing rx/tx logic in common functions for single/multi-queue cases.

Wow, AI could take that feedback and revise for a new version.

#11 lore raw

RFC 3/4 net/af_packet: fix indentation

In reply to Stephen Hemminger

DPDK code is supposed to be indented with TABS not spaces. Also, used "unsigned int" to keep checkpatch happy.

---
 drivers/net/af_packet/rte_eth_af_packet.c | 64 +++++++++++------------
 1 file changed, 32 insertions(+), 32 deletions(-)
drivers/net/af_packet/rte_eth_af_packet.c +32 -32
index 158393dd70..5847e14d80 100644
@@ -78,7 +78,7 @@ struct __rte_cache_aligned pkt_tx_queue {
 };
 
 struct pmd_internals {
-	unsigned nb_queues;
+	unsigned int nb_queues;
 
 	int if_index;
 	char *if_name;
@@ -537,7 +537,7 @@ eth_dev_close(struct rte_eth_dev *dev)
 
 static int
 eth_link_update(struct rte_eth_dev *dev,
-                int wait_to_complete __rte_unused)
+		int wait_to_complete __rte_unused)
 {
 	const struct pmd_internals *internals = dev->data->dev_private;
 	struct rte_eth_link *dev_link = &dev->data->dev_link;
@@ -557,11 +557,11 @@ eth_link_update(struct rte_eth_dev *dev,
 
 static int
 eth_rx_queue_setup(struct rte_eth_dev *dev,
-                   uint16_t rx_queue_id,
-                   uint16_t nb_rx_desc __rte_unused,
-                   unsigned int socket_id __rte_unused,
-                   const struct rte_eth_rxconf *rx_conf __rte_unused,
-                   struct rte_mempool *mb_pool)
+		   uint16_t rx_queue_id,
+		   uint16_t nb_rx_desc __rte_unused,
+		   unsigned int socket_id __rte_unused,
+		   const struct rte_eth_rxconf *rx_conf __rte_unused,
+		   struct rte_mempool *mb_pool)
 {
 	struct pmd_internals *internals = dev->data->dev_private;
 	struct pkt_rx_queue *pkt_q = &internals->rx_queue[rx_queue_id];
@@ -592,10 +592,10 @@ eth_rx_queue_setup(struct rte_eth_dev *dev,
 
 static int
 eth_tx_queue_setup(struct rte_eth_dev *dev,
-                   uint16_t tx_queue_id,
-                   uint16_t nb_tx_desc __rte_unused,
-                   unsigned int socket_id __rte_unused,
-                   const struct rte_eth_txconf *tx_conf __rte_unused)
+		   uint16_t tx_queue_id,
+		   uint16_t nb_tx_desc __rte_unused,
+		   unsigned int socket_id __rte_unused,
+		   const struct rte_eth_txconf *tx_conf __rte_unused)
 {
 
 	struct pmd_internals *internals = dev->data->dev_private;
@@ -722,8 +722,8 @@ static const struct eth_dev_ops ops = {
  */
 static int
 open_packet_iface(const char *key __rte_unused,
-                  const char *value __rte_unused,
-                  void *extra_args)
+		  const char *value __rte_unused,
+		  void *extra_args)
 {
 	int *sockfd = extra_args;
 
@@ -786,17 +786,17 @@ get_fanout(const char *fanout_mode, int if_index)
 
 static int
 rte_pmd_init_internals(struct rte_vdev_device *dev,
-                       const int sockfd,
-                       const unsigned nb_queues,
-                       unsigned int blocksize,
-                       unsigned int blockcnt,
-                       unsigned int framesize,
-                       unsigned int framecnt,
+		       const int sockfd,
+		       const unsigned int nb_queues,
+		       unsigned int blocksize,
+		       unsigned int blockcnt,
+		       unsigned int framesize,
+		       unsigned int framecnt,
 		       unsigned int qdisc_bypass,
 		       const char *fanout_mode,
-                       struct pmd_internals **internals,
-                       struct rte_eth_dev **eth_dev,
-                       struct rte_kvargs *kvlist)
+		       struct pmd_internals **internals,
+		       struct rte_eth_dev **eth_dev,
+		       struct rte_kvargs *kvlist)
 {
 	const char *name = rte_vdev_device_name(dev);
 	const unsigned int numa_node = dev->device.numa_node;
@@ -822,7 +822,7 @@ rte_pmd_init_internals(struct rte_vdev_device *dev,
 	if (pair == NULL) {
 		PMD_LOG(ERR,
 			"%s: no interface specified for AF_PACKET ethdev",
-		        name);
+			name);
 		return -1;
 	}
 
@@ -831,7 +831,7 @@ rte_pmd_init_internals(struct rte_vdev_device *dev,
 		name, numa_node);
 
 	*internals = rte_zmalloc_socket(name, sizeof(**internals),
-	                                0, numa_node);
+					0, numa_node);
 	if (*internals == NULL)
 		return -1;
 
@@ -1072,8 +1072,8 @@ rte_pmd_init_internals(struct rte_vdev_device *dev,
 
 static int
 rte_eth_from_packet(struct rte_vdev_device *dev,
-                    int const *sockfd,
-                    struct rte_kvargs *kvlist)
+		    int const *sockfd,
+		    struct rte_kvargs *kvlist)
 {
 	const char *name = rte_vdev_device_name(dev);
 	struct pmd_internals *internals = NULL;
@@ -1104,7 +1104,7 @@ rte_eth_from_packet(struct rte_vdev_device *dev,
 			if (qpairs < 1) {
 				PMD_LOG(ERR,
 					"%s: invalid qpairs value",
-				        name);
+					name);
 				return -1;
 			}
 			continue;
@@ -1114,7 +1114,7 @@ rte_eth_from_packet(struct rte_vdev_device *dev,
 			if (!blocksize) {
 				PMD_LOG(ERR,
 					"%s: invalid blocksize value",
-				        name);
+					name);
 				return -1;
 			}
 			continue;
@@ -1124,7 +1124,7 @@ rte_eth_from_packet(struct rte_vdev_device *dev,
 			if (!framesize) {
 				PMD_LOG(ERR,
 					"%s: invalid framesize value",
-				        name);
+					name);
 				return -1;
 			}
 			continue;
@@ -1134,7 +1134,7 @@ rte_eth_from_packet(struct rte_vdev_device *dev,
 			if (!framecount) {
 				PMD_LOG(ERR,
 					"%s: invalid framecount value",
-				        name);
+					name);
 				return -1;
 			}
 			continue;
@@ -1158,7 +1158,7 @@ rte_eth_from_packet(struct rte_vdev_device *dev,
 	if (framesize > blocksize) {
 		PMD_LOG(ERR,
 			"%s: AF_PACKET MMAP frame size exceeds block size!",
-		        name);
+			name);
 		return -1;
 	}
 
@@ -1233,7 +1233,7 @@ rte_pmd_af_packet_probe(struct rte_vdev_device *dev)
 	if (rte_kvargs_count(kvlist, ETH_AF_PACKET_IFACE_ARG) == 1) {
 
 		ret = rte_kvargs_process(kvlist, ETH_AF_PACKET_IFACE_ARG,
-		                         &open_packet_iface, &sockfd);
+					 &open_packet_iface, &sockfd);
 		if (ret < 0)
 			goto exit;
 	}
Signature · 1 line
2.51.0
#12 lore raw

RFC 4/4 net/af_packet: add VPP-style prefetching to receive path

In reply to Stephen Hemminger

Implement the single/dual/quad loop design pattern from FD.IO VPP to improve cache efficiency in the af_packet PMD receive path.

The original implementation processes packets one at a time in a simple loop, which can result in cache misses when accessing frame headers and packet data. The new implementation:

- Processes packets in batches of 4 (quad), 2 (dual), and 1 (single)
- Prefetches next batch of frame headers while processing current batch
- Prefetches packet data before memcpy to hide memory latency
- Reduces loop overhead through partial unrolling

Two helper functions are introduced:
- af_packet_get_frame(): Returns frame pointer at index with wraparound
- af_packet_rx_one(): Common per-packet processing (mbuf alloc, memcpy, VLAN handling, timestamp offload)

The quad loop checks availability of all 4 frames before processing, falling through to dual/single loops when fewer frames are ready. Early exit paths (out_advance1/2/3) ensure correct frame index tracking when mbuf allocation fails mid-batch.

Prefetch strategy:
- Frame headers: prefetch N+4..N+7 while processing N..N+3
- Packet data: prefetch at tp_mac offset before memcpy

This pattern is well-established in high-performance packet processing and should improve throughput by better utilizing CPU cache hierarchy, particularly beneficial when processing bursts of packets.

---
 drivers/net/af_packet/rte_eth_af_packet.c | 208 +++++++++++++++++-----
 1 file changed, 164 insertions(+), 44 deletions(-)
drivers/net/af_packet/rte_eth_af_packet.c +164 -44
index 5847e14d80..946c21d878 100644
@@ -14,6 +14,7 @@
 #include <rte_malloc.h>
 #include <rte_kvargs.h>
 #include <bus_vdev_driver.h>
+#include <rte_prefetch.h>
 
 #include <errno.h>
 #include <linux/if_ether.h>
@@ -120,75 +121,194 @@ RTE_LOG_REGISTER_DEFAULT(af_packet_logtype, NOTICE);
 	RTE_LOG_LINE(level, AFPACKET, "%s(): " fmt ":%s", __func__, \
 		## __VA_ARGS__, strerror(errno))
 
-static uint16_t
-eth_af_packet_rx(void *queue, struct rte_mbuf **bufs, uint16_t nb_pkts)
+/*
+ * Helper to get the frame pointer at a given index with wraparound
+ */
+static inline struct tpacket2_hdr *
+af_packet_get_frame(struct pkt_rx_queue *pkt_q, unsigned int idx)
+{
+	if (idx >= pkt_q->framecount)
+		idx -= pkt_q->framecount;
+	return (struct tpacket2_hdr *)pkt_q->rd[idx].iov_base;
+}
+
+/*
+ * Process a single received packet - common code for all loop variants
+ */
+static inline int
+af_packet_rx_one(struct pkt_rx_queue *pkt_q,
+		 struct tpacket2_hdr *ppd,
+		 struct rte_mbuf **mbuf_out,
+		 unsigned long *rx_bytes)
 {
-	unsigned i;
-	struct tpacket2_hdr *ppd;
 	struct rte_mbuf *mbuf;
 	uint8_t *pbuf;
+
+	mbuf = rte_pktmbuf_alloc(pkt_q->mb_pool);
+	if (unlikely(mbuf == NULL)) {
+		pkt_q->rx_nombuf++;
+		return -1;
+	}
+
+	rte_pktmbuf_pkt_len(mbuf) = rte_pktmbuf_data_len(mbuf) = ppd->tp_snaplen;
+	pbuf = (uint8_t *)ppd + ppd->tp_mac;
+	memcpy(rte_pktmbuf_mtod(mbuf, void *), pbuf, rte_pktmbuf_data_len(mbuf));
+
+	if (ppd->tp_status & TP_STATUS_VLAN_VALID) {
+		mbuf->vlan_tci = ppd->tp_vlan_tci;
+		mbuf->ol_flags |= (RTE_MBUF_F_RX_VLAN | RTE_MBUF_F_RX_VLAN_STRIPPED);
+		if (!pkt_q->vlan_strip && rte_vlan_insert(&mbuf))
+			PMD_LOG(ERR, "Failed to reinsert VLAN tag");
+	}
+
+	if (pkt_q->timestamp_offloading) {
+		*RTE_MBUF_DYNFIELD(mbuf, timestamp_dynfield_offset,
+			rte_mbuf_timestamp_t *) =
+				(uint64_t)ppd->tp_sec * 1000000000 + ppd->tp_nsec;
+		mbuf->ol_flags |= timestamp_dynflag;
+	}
+
+	mbuf->port = pkt_q->in_port;
+	*mbuf_out = mbuf;
+	*rx_bytes += mbuf->pkt_len;
+	ppd->tp_status = TP_STATUS_KERNEL;
+
+	return 0;
+}
+
+/*
+ * Receive packets using VPP-style single/dual/quad loop pattern with prefetching.
+ */
+static uint16_t
+eth_af_packet_rx(void *queue, struct rte_mbuf **bufs, uint16_t nb_pkts)
+{
 	struct pkt_rx_queue *pkt_q = queue;
+	struct tpacket2_hdr *ppd0, *ppd1, *ppd2, *ppd3;
 	uint16_t num_rx = 0;
 	unsigned long num_rx_bytes = 0;
 	unsigned int framecount, framenum;
+	uint16_t n_left;
 
 	if (unlikely(nb_pkts == 0))
 		return 0;
 
-	/*
-	 * Reads the given number of packets from the AF_PACKET socket one by
-	 * one and copies the packet data into a newly allocated mbuf.
-	 */
 	framecount = pkt_q->framecount;
 	framenum = pkt_q->framenum;
-	for (i = 0; i < nb_pkts; i++) {
-		/* point at the next incoming frame */
-		ppd = (struct tpacket2_hdr *) pkt_q->rd[framenum].iov_base;
-		if ((ppd->tp_status & TP_STATUS_USER) == 0)
+	n_left = nb_pkts;
+
+	/* Quad loop: Process 4 packets at a time with prefetching */
+	while (n_left >= 4) {
+		ppd0 = af_packet_get_frame(pkt_q, framenum);
+		ppd1 = af_packet_get_frame(pkt_q, framenum + 1);
+		ppd2 = af_packet_get_frame(pkt_q, framenum + 2);
+		ppd3 = af_packet_get_frame(pkt_q, framenum + 3);
+
+		if ((ppd0->tp_status & TP_STATUS_USER) == 0)
 			break;
+		if ((ppd1->tp_status & TP_STATUS_USER) == 0)
+			goto dual_loop;
+		if ((ppd2->tp_status & TP_STATUS_USER) == 0)
+			goto dual_loop;
+		if ((ppd3->tp_status & TP_STATUS_USER) == 0)
+			goto dual_loop;
+
+		/* Prefetch next 4 frame headers */
+		rte_prefetch0(af_packet_get_frame(pkt_q, framenum + 4));
+		rte_prefetch0(af_packet_get_frame(pkt_q, framenum + 5));
+		rte_prefetch0(af_packet_get_frame(pkt_q, framenum + 6));
+		rte_prefetch0(af_packet_get_frame(pkt_q, framenum + 7));
+
+		/* Prefetch packet data */
+		rte_prefetch0((uint8_t *)ppd0 + ppd0->tp_mac);
+		rte_prefetch0((uint8_t *)ppd1 + ppd1->tp_mac);
+		rte_prefetch0((uint8_t *)ppd2 + ppd2->tp_mac);
+		rte_prefetch0((uint8_t *)ppd3 + ppd3->tp_mac);
+
+		if (unlikely(af_packet_rx_one(pkt_q, ppd0, &bufs[num_rx], &num_rx_bytes) < 0))
+			goto out;
+		num_rx++;
+		if (unlikely(af_packet_rx_one(pkt_q, ppd1, &bufs[num_rx], &num_rx_bytes) < 0))
+			goto out_advance1;
+		num_rx++;
+		if (unlikely(af_packet_rx_one(pkt_q, ppd2, &bufs[num_rx], &num_rx_bytes) < 0))
+			goto out_advance2;
+		num_rx++;
+		if (unlikely(af_packet_rx_one(pkt_q, ppd3, &bufs[num_rx], &num_rx_bytes) < 0))
+			goto out_advance3;
+		num_rx++;
 
-		/* allocate the next mbuf */
-		mbuf = rte_pktmbuf_alloc(pkt_q->mb_pool);
-		if (unlikely(mbuf == NULL)) {
-			pkt_q->rx_nombuf++;
+		framenum += 4;
+		if (framenum >= framecount)
+			framenum -= framecount;
+		n_left -= 4;
+	}
+
+dual_loop:
+	/* Dual loop: Process 2 packets at a time */
+	while (n_left >= 2) {
+		ppd0 = af_packet_get_frame(pkt_q, framenum);
+		ppd1 = af_packet_get_frame(pkt_q, framenum + 1);
+
+		if ((ppd0->tp_status & TP_STATUS_USER) == 0)
 			break;
-		}
+		if ((ppd1->tp_status & TP_STATUS_USER) == 0)
+			goto single_loop;
 
-		/* packet will fit in the mbuf, go ahead and receive it */
-		rte_pktmbuf_pkt_len(mbuf) = rte_pktmbuf_data_len(mbuf) = ppd->tp_snaplen;
-		pbuf = (uint8_t *) ppd + ppd->tp_mac;
-		memcpy(rte_pktmbuf_mtod(mbuf, void *), pbuf, rte_pktmbuf_data_len(mbuf));
+		rte_prefetch0(af_packet_get_frame(pkt_q, framenum + 2));
+		rte_prefetch0(af_packet_get_frame(pkt_q, framenum + 3));
+		rte_prefetch0((uint8_t *)ppd0 + ppd0->tp_mac);
+		rte_prefetch0((uint8_t *)ppd1 + ppd1->tp_mac);
 
-		/* check for vlan info */
-		if (ppd->tp_status & TP_STATUS_VLAN_VALID) {
-			mbuf->vlan_tci = ppd->tp_vlan_tci;
-			mbuf->ol_flags |= (RTE_MBUF_F_RX_VLAN | RTE_MBUF_F_RX_VLAN_STRIPPED);
+		if (unlikely(af_packet_rx_one(pkt_q, ppd0, &bufs[num_rx], &num_rx_bytes) < 0))
+			goto out;
+		num_rx++;
+		if (unlikely(af_packet_rx_one(pkt_q, ppd1, &bufs[num_rx], &num_rx_bytes) < 0))
+			goto out_advance1;
+		num_rx++;
 
-			if (!pkt_q->vlan_strip && rte_vlan_insert(&mbuf))
-				PMD_LOG(ERR, "Failed to reinsert VLAN tag");
-		}
+		framenum += 2;
+		if (framenum >= framecount)
+			framenum -= framecount;
+		n_left -= 2;
+	}
 
-		/* add kernel provided timestamp when offloading is enabled */
-		if (pkt_q->timestamp_offloading) {
-			/* since TPACKET_V2 timestamps are provided in nanoseconds resolution */
-			*RTE_MBUF_DYNFIELD(mbuf, timestamp_dynfield_offset,
-				rte_mbuf_timestamp_t *) =
-					(uint64_t)ppd->tp_sec * 1000000000 + ppd->tp_nsec;
+single_loop:
+	/* Single loop: Process remaining packets */
+	while (n_left >= 1) {
+		ppd0 = af_packet_get_frame(pkt_q, framenum);
 
-			mbuf->ol_flags |= timestamp_dynflag;
-		}
+		if ((ppd0->tp_status & TP_STATUS_USER) == 0)
+			break;
 
-		/* release incoming frame and advance ring buffer */
-		ppd->tp_status = TP_STATUS_KERNEL;
-		if (++framenum >= framecount)
-			framenum = 0;
-		mbuf->port = pkt_q->in_port;
+		rte_prefetch0(af_packet_get_frame(pkt_q, framenum + 1));
+		rte_prefetch0((uint8_t *)ppd0 + ppd0->tp_mac);
 
-		/* account for the receive frame */
-		bufs[i] = mbuf;
+		if (unlikely(af_packet_rx_one(pkt_q, ppd0, &bufs[num_rx], &num_rx_bytes) < 0))
+			goto out;
 		num_rx++;
-		num_rx_bytes += mbuf->pkt_len;
+
+		if (++framenum >= framecount)
+			framenum = 0;
+		n_left--;
 	}
+
+	goto out;
+
+out_advance3:
+	framenum += 3;
+	if (framenum >= framecount)
+		framenum -= framecount;
+	goto out;
+out_advance2:
+	framenum += 2;
+	if (framenum >= framecount)
+		framenum -= framecount;
+	goto out;
+out_advance1:
+	framenum += 1;
+	if (framenum >= framecount)
+		framenum -= framecount;
+out:
 	pkt_q->framenum = framenum;
 	pkt_q->rx_pkts += num_rx;
 	pkt_q->rx_bytes += num_rx_bytes;
Signature · 1 line
2.51.0
#13 lore raw

Re: [RFC 4/4] net/af_packet: add VPP-style prefetching to receive path

Quoted 26 lines · On Wed, 28 Jan 2026 09:30:20 -0800 Stephen Hemminger…

Implement the single/dual/quad loop design pattern from FD.IO VPP to improve cache efficiency in the af_packet PMD receive path.

The original implementation processes packets one at a time in a simple loop, which can result in cache misses when accessing frame headers and packet data. The new implementation:

- Processes packets in batches of 4 (quad), 2 (dual), and 1 (single)
- Prefetches next batch of frame headers while processing current batch
- Prefetches packet data before memcpy to hide memory latency
- Reduces loop overhead through partial unrolling

Two helper functions are introduced:
- af_packet_get_frame(): Returns frame pointer at index with wraparound
- af_packet_rx_one(): Common per-packet processing (mbuf alloc, memcpy, VLAN handling, timestamp offload)

The quad loop checks availability of all 4 frames before processing, falling through to dual/single loops when fewer frames are ready. Early exit paths (out_advance1/2/3) ensure correct frame index tracking when mbuf allocation fails mid-batch.

Prefetch strategy:
- Frame headers: prefetch N+4..N+7 while processing N..N+3
- Packet data: prefetch at tp_mac offset before memcpy

Quoted 5 lines

This pattern is well-established in high-performance packet processing and should improve throughput by better utilizing CPU cache hierarchy, particularly beneficial when processing bursts of packets.

This and previous proposal to prefetch have no impact on performance. Rolled a simple perf test and all three versions come out the same. The bottleneck is not here, probably at system call and copies now.

Original Prefetch Quad/Dual
TX 1.427 Mpps 1.426 Mpps 1.426 Mpps

RX 0.529 Mpps 0.530 Mpps 0.533 Mpps
loss 87.93% 87.98% 88.0%

Original Prefetch Quad/Dual
TX 1.427 Mpps 1.426 Mpps 1.426 Mpps

RX 0.529 Mpps 0.530 Mpps 0.533 Mpps
loss 87.93% 87.98% 88.0%

Will put the test in the next version of this series, and drop this patch.

#14 lore raw

RE: [RFC 4/4] net/af_packet: add VPP-style prefetching to receive path

Quoted 42 lines

From: Stephen Hemminger [mailto:[email protected]]
Sent: Thursday, 29 January 2026 02.06

Quoted 4 lines · On Wed, 28 Jan 2026 09:30:20 -0800 Stephen Hemminger…

Implement the single/dual/quad loop design pattern from FD.IO VPP to improve cache efficiency in the af_packet PMD receive path.

The original implementation processes packets one at a time in a

simple

Quoted 1 line

loop, which can result in cache misses when accessing frame headers

and

Quoted 4 lines

packet data. The new implementation:

- Processes packets in batches of 4 (quad), 2 (dual), and 1 (single)
- Prefetches next batch of frame headers while processing current

batch

Quoted 5 lines

- Prefetches packet data before memcpy to hide memory latency
- Reduces loop overhead through partial unrolling

Two helper functions are introduced:
- af_packet_get_frame(): Returns frame pointer at index with

wraparound

Quoted 1 line

- af_packet_rx_one(): Common per-packet processing (mbuf alloc,

memcpy,

Quoted 4 lines

VLAN handling, timestamp offload)

The quad loop checks availability of all 4 frames before processing, falling through to dual/single loops when fewer frames are ready.

Early

Quoted 1 line

exit paths (out_advance1/2/3) ensure correct frame index tracking

when

Quoted 7 lines

mbuf allocation fails mid-batch.

Prefetch strategy:
- Frame headers: prefetch N+4..N+7 while processing N..N+3
- Packet data: prefetch at tp_mac offset before memcpy

This pattern is well-established in high-performance packet

processing

Quoted 1 line

and should improve throughput by better utilizing CPU cache

Quoted 8 lines

hierarchy,

Quoted 3 lines

particularly beneficial when processing bursts of packets.

This and previous proposal to prefetch have no impact on performance. Rolled a simple perf test and all three versions come out the same.

Please be aware that many test cases are inadvertently designed in a way where mbufs unintendedly are hot in the cache, so prefetching does not provide the expected performance gain. E.g. when working on a newly allocated mbuf, the mbuf should be cold.
But if it came from the mempool cache, and was recently worked on and then freed into the mempool cache, then it will be hot.

Quoted 1 line

The bottleneck is not here, probably at system call and copies now.

The most important bottleneck might be elsewhere.
But this optimization might not be as irrelevant as the test results indicate.

Anyway, I agree that dropping the patch (for now) makes sense.

Quoted 17 lines

Original Prefetch Quad/Dual
TX 1.427 Mpps 1.426 Mpps 1.426 Mpps

RX 0.529 Mpps 0.530 Mpps 0.533 Mpps
loss 87.93% 87.98% 88.0%

Original Prefetch Quad/Dual
TX 1.427 Mpps 1.426 Mpps 1.426 Mpps

RX 0.529 Mpps 0.530 Mpps 0.533 Mpps
loss 87.93% 87.98% 88.0%

Will put the test in the next version of this series, and drop this patch.

#15 lore raw

Re: [RFC 4/4] net/af_packet: add VPP-style prefetching to receive path

Thanks for running benchmarks! Dropping for now sounds good (I did the same in my patch).

#16 lore raw

Re: [RFC 4/4] net/af_packet: add VPP-style prefetching to receive path

In reply to Morten Brørup
Quoted 7 lines · On Thu, 29 Jan 2026 10:00:15 +0100 Morten Brørup <[email protected]>…
Quoted 2 lines

This and previous proposal to prefetch have no impact on performance. Rolled a simple perf test and all three versions come out the same.

Please be aware that many test cases are inadvertently designed in a way where mbufs unintendedly are hot in the cache, so prefetching does not provide the expected performance gain. E.g. when working on a newly allocated mbuf, the mbuf should be cold.
But if it came from the mempool cache, and was recently worked on and then freed into the mempool cache, then it will be hot.

Quoted 6 lines
Quoted 1 line

The bottleneck is not here, probably at system call and copies now.

The most important bottleneck might be elsewhere.
But this optimization might not be as irrelevant as the test results indicate.

Anyway, I agree that dropping the patch (for now) makes sense.

I doubt pre-fetch will matter much in a driver like this because:
- on tx the data is still in cache since just setup by caller
- on rx the data is still in cache since kernel just copied it into the buffer.

#17 lore raw

RE: [RFC 4/4] net/af_packet: add VPP-style prefetching to receive path

Quoted 19 lines

From: Stephen Hemminger [mailto:[email protected]]
Sent: Monday, 2 February 2026 19.44

Quoted 1 line · On Thu, 29 Jan 2026 10:00:15 +0100 Morten Brørup <[email protected]>…
Quoted 1 line

This and previous proposal to prefetch have no impact on

performance.

Quoted 3 lines
Quoted 1 line

Rolled a simple perf test and all three versions come out the same.

Please be aware that many test cases are inadvertently designed in a

way where mbufs unintendedly are hot in the cache, so prefetching does not provide the expected performance gain.

Quoted 2 lines

E.g. when working on a newly allocated mbuf, the mbuf should be cold. But if it came from the mempool cache, and was recently worked on and

then freed into the mempool cache, then it will be hot.

Quoted 3 lines
Quoted 2 lines

The bottleneck is not here, probably at system call and copies now.

Quoted 8 lines
Quoted 2 lines

The most important bottleneck might be elsewhere.
But this optimization might not be as irrelevant as the test results

indicate.

Quoted 2 lines

Anyway, I agree that dropping the patch (for now) makes sense.

I doubt pre-fetch will matter much in a driver like this because:
- on tx the data is still in cache since just setup by caller

If the packet has been sitting in a queue, e.g. for QoS buffering, it may have been evicted from the cache.

Quoted 2 lines

- on rx the data is still in cache since kernel just copied it into the buffer.

If there's any delay between the signaling and the handling, it could be evicted from the cache in the meantime. (The delay could come from e.g. the handling CPU core being busy with something else before handling the RX ready signal.) And it could be in the cache of a different CPU core.

Anyway, I still agree with dropping the patch for now, since testing shows no difference.