net/af_packet: cleanups and optimizations
Subsystems (2)
cover net/af_packet: cleanups and optimizations
This series contains several improvements to the AF_PACKET PMD:
1. Remove unnecessary volatile qualifiers from statistics counters and standardize on uint64_t for consistency.
2. Add a unit test for the AF_PACKET PMD. The test creates a TAP interface and uses it to verify basic send/receive functionality, statistics, and device operations.
3. Implement VPP-style single/dual/quad loop pattern with software prefetching in the receive path. This optimization processes packets in batches of 4, 2, and 1, prefetching upcoming frame headers and packet data to improve cache utilization and hide memory latency.
Note: The test code in patch 2 and the prefetch optimization in patch 4 were generated with the assistance of Claude AI (Anthropic), with manual review and cleanup afterwards.
The prefetch pattern is modeled after FD.IO VPP's packet processing loops, which have proven effective in high-performance networking applications. The key techniques are:
- Loop unrolling to reduce branch overhead
- Prefetching next iteration's data while processing current batch
- Graceful fallback from quad to dual to single packet processing
Testing was performed using the new unit test on a TAP interface. Performance testing on real hardware with high packet rates would be valuable to quantify the prefetch benefits.
Stephen Hemminger (4):
net/af_packet: remove volatile from statistics
test: add test for af_packet
net/af_packet: fix indentation
net/af_packet: add VPP-style prefetching to receive path
app/test/meson.build | 1 + app/test/test_pmd_af_packet.c | 875 ++++++++++++++++++++++ drivers/net/af_packet/rte_eth_af_packet.c | 286 +++++-- 3 files changed, 1079 insertions(+), 83 deletions(-) create mode 100644 app/test/test_pmd_af_packet.c
Signature · 1 line
2.51.0
RFC 1/4 net/af_packet: remove volatile from statistics
Statistics are only updated from a single thread at a time and the compiler should not have to worry about optimizing them. The statistics returned are 64 bit so use uint64_t instead of arch specific unsigned long.
--- drivers/net/af_packet/rte_eth_af_packet.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-)
drivers/net/af_packet/rte_eth_af_packet.c +7 -7
@@ -57,10 +57,10 @@ struct __rte_cache_aligned pkt_rx_queue {uint8_t vlan_strip;uint8_t timestamp_offloading; -volatile unsigned long rx_pkts; -volatile unsigned long rx_bytes; -volatile unsigned long rx_nombuf; -volatile unsigned long rx_dropped_pkts; +uint64_t rx_pkts; +uint64_t rx_bytes; +uint64_t rx_nombuf; +uint64_t rx_dropped_pkts; };struct __rte_cache_aligned pkt_tx_queue {
@@ -72,9 +72,9 @@ struct __rte_cache_aligned pkt_tx_queue {unsigned int framecount;unsigned int framenum; -volatile unsigned long tx_pkts; -volatile unsigned long err_pkts; -volatile unsigned long tx_bytes; +uint64_t tx_pkts; +uint64_t tx_bytes; +uint64_t err_pkts; };struct pmd_internals {
Signature · 1 line
2.51.0
Re: [RFC 1/4] net/af_packet: remove volatile from statistics
Can you clarify why removal of volatile is safe (visibility, ordering) when the producer thread and consumer thread can be different?
Quoted 46 lines · Stephen Hemminger
Statistics are only updated from a single thread at a time and the compiler should not have to worry about optimizing them. The statistics returned are 64 bit so use
uint64_tinstead of arch specific unsigned long.Signed-off-by Stephen Hemminger <[email protected]>--- drivers/net/af_packet/rte_eth_af_packet.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-)drivers/net/af_packet/rte_eth_af_packet.c +7
-8@@ -57,10 +57,10 @@ struct __rte_cache_aligned pkt_rx_queue {uint8_t vlan_strip;uint8_t timestamp_offloading; -volatile unsigned long rx_pkts; -volatile unsigned long rx_bytes; -volatile unsigned long rx_nombuf; -volatile unsigned long rx_dropped_pkts; +uint64_t rx_pkts; +uint64_t rx_bytes; +uint64_t rx_nombuf; +uint64_t rx_dropped_pkts; };struct __rte_cache_aligned pkt_tx_queue {@@ -72,9 +72,9 @@ struct __rte_cache_aligned pkt_tx_queue {unsigned int framecount;unsigned int framenum; -volatile unsigned long tx_pkts; -volatile unsigned long err_pkts; -volatile unsigned long tx_bytes; +uint64_t tx_pkts; +uint64_t tx_bytes; +uint64_t err_pkts; };struct pmd_internals { --2.51.0
Re: [RFC 1/4] net/af_packet: remove volatile from statistics
Quoted 2 lines · On Wed, 28 Jan 2026 11:57:16 -0800 Scott Mitchell <[email protected]>…
Can you clarify why removal of volatile is safe (visibility, ordering) when the producer thread and consumer thread can be different?
The short version is, volatile protects against compiler reordering code where it thinks variable can't change. The atomic primitives protect against load/store ordering issues on non-cache coherent CPU architectures. The DPDK design pattern has been that statistics don't need to be protected since producer is single thread and consumer should be ok with slightly out of date data. The point is that using atomic requires going out of cache on ARM (where it would matter) and on x86 locked operations cause a pipeline stall. At the high data rates in DPDK any stall or cache miss matters.
The one exception would be drivers that advertise lock free transmit would need to use atomic on the Tx stats to avoid multi-producer problems.
Re: [RFC 1/4] net/af_packet: remove volatile from statistics
Without volatile, is there any guarantee the consumer thread will see the writes from the producer thread? For example, couldn't the compiler cache rx_pkts in a register on the consumer side? // Consumer thread:uint64_t prev = 0;
while (monitoring) {
uint64_t curr = rx_pkts; // Compiler may cache in register
if (curr != prev) { // Without volatile: may always be false
update_display(); // Never executes
prev = curr;
}
}I created a godbolt comparing different approaches
(https://godbolt.org/z/1Gaz6jPxh) showing assembly for x86-64 and ARM at -O3. Summary:
1. Plain load/store: Fastest. No visibility guarantees. May tear on 32-bit.
2. Volatile: Provides visibility. May tear on 32-bit.
3. Atomic load/store (relaxed): Same performance as volatile. Provides visibility and guarantees no tearing (even on 32-bit).
4. Atomic_fetch_add: Heaviest cost (LOCK on x86).
My concern is that Option 1 (Plain) has no guaranteed visibility - the compiler may optimize away loads entirely. Since Option 3 (Atomic Load/Store) has identical instruction cost to Volatile but provides formal guarantees (visibility + no tearing), would that be the preferred solution?
Re: [RFC 1/4] net/af_packet: remove volatile from statistics
Quoted 22 lines · On Sun, 1 Feb 2026 23:02:53 -0800 Scott Mitchell <[email protected]>…
Without volatile, is there any guarantee the consumer thread will see the writes from the producer thread? For example, couldn't the compiler cache
rx_pktsin a register on the consumer side? // Consumer thread:uint64_tprev = 0;while (monitoring) { uint64_t curr = rx_pkts; // Compiler may cache in register if (curr != prev) { // Without volatile: may always be false update_display(); // Never executes prev = curr; } }I created a godbolt comparing different approaches
(https://godbolt.org/z/1Gaz6jPxh) showing assembly for x86-64 and ARM at -O3. Summary:
1. Plain load/store: Fastest. No visibility guarantees. May tear on 32-bit.
2. Volatile: Provides visibility. May tear on 32-bit.
3. Atomic load/store (relaxed): Same performance as volatile. Provides visibility and guarantees no tearing (even on 32-bit).
4.Atomic_fetch_add: Heaviest cost (LOCK on x86).
Quoted 5 lines
My concern is that Option 1 (Plain) has no guaranteed visibility - the compiler may optimize away loads entirely. Since Option 3 (Atomic Load/Store) has identical instruction cost to Volatile but provides formal guarantees (visibility + no tearing), would that be the preferred solution?
In normal case, compiler isn't going to be able to see across function boundary especially through indirection of eth_dev_ops table. With LTO it might be possible but unlikely.
Re: [RFC 1/4] net/af_packet: remove volatile from statistics
Thank you for the clarification about the function pointer indirection acting as a compiler barrier - that makes sense for the typical case.
I have one remaining question about 32-bit architectures: even with the implicit barrier, plain uint64_t reads aren't atomic on 32-bit platforms (reader could see torn high/low halves). Is this acceptable for stats counters in DPDK, or is 32-bit support not a concern?
For context, atomic_load/store(memory_order_relaxed) formally guarantees both visibility and no tearing across architectures, with minimal (GCC) or zero (Clang) overhead for x86-64. I see there's precedent in DPDK for plain uint64_t stats, and I'm hoping to understand the assumptions/trade-offs better.
Re: [RFC 1/4] net/af_packet: remove volatile from statistics
Quoted 7 lines · On Mon, 2 Feb 2026 11:12:31 -0800 Scott Mitchell <[email protected]>…
Thank you for the clarification about the function pointer indirection acting as a compiler barrier - that makes sense for the typical case.
I have one remaining question about 32-bit architectures: even with the implicit barrier, plain
uint64_treads aren't atomic on 32-bit platforms (reader could see torn high/low halves). Is this acceptable for stats counters in DPDK, or is 32-bit support not a concern?
DPDK has not worried about torn counters on 32 bit architectures. Whether it should or not is open question.
People don't care much about 32 bit anymore; other than not breaking it.
Quoted 5 lines
For context, atomic_load/store(memory_order_relaxed) formally guarantees both visibility and no tearing across architectures, with minimal (GCC) or zero (Clang) overhead for x86-64. I see there's precedent in DPDK for plain
uint64_tstats, and I'm hoping to understand the assumptions/trade-offs better.
The problems is that on relaxed ordering architectures like ARM it makes the counter update more expensive in the fast path.
RFC 2/4 test: add test for af_packet
In reply to Stephen HemmingerAdd a test for AF_PACKET PMD.
This test was generated with Claude AI based off of existing test_pmd_ring.c with some cleanup afterwards.
--- app/test/meson.build | 1 + app/test/test_pmd_af_packet.c | 875 ++++++++++++++++++++++++++++++++++ 2 files changed, 876 insertions(+) create mode 100644 app/test/test_pmd_af_packet.c
app/test/meson.build +1 -0
@@ -141,6 +141,7 @@ source_file_deps = { 'test_per_lcore.c': [], 'test_pflock.c': [], 'test_pie.c': ['sched'], + 'test_pmd_af_packet.c': ['net_af_packet', 'ethdev', 'bus_vdev'], 'test_pmd_perf.c': ['ethdev', 'net'] + packet_burst_generator_deps, 'test_pmd_ring.c': ['net_ring', 'ethdev', 'bus_vdev'], 'test_pmd_ring_perf.c': ['ethdev', 'net_ring', 'bus_vdev'],
app/test/test_pmd_af_packet.c new file +875 -0
875 lines not on this page · raw message
Signature · 1 line
2.51.0
Re: [RFC 2/4] test: add test for af_packet
Awesome! Some suggestions below but could easily be done in a followup for incremental progress.
app/test/test_pmd_af_packet.c line 412
Quoted 9 lines
app/test/test_pmd_af_packet.c new file +9
-0+/* TX burst */ + nb_tx =rte_eth_tx_burst (port_id,0 , bufs,BURST_SIZE ); + +/* Free any unsent mbufs */ +for (i = nb_tx; i <BURST_SIZE ; i++) +rte_pktmbuf_free (bufs[i]); + +/* Small delay to allow stats update */ +rte_delay_us_block (1000 );
Consider polling rte_eth_stats_get at a more frequent interval, break when you see status update, and apply overall timeout/repeat limit -> can make test more reliable and take less time when data is usually available quickly
app/test/test_pmd_af_packet.c line 450
Quoted 9 lines
app/test/test_pmd_af_packet.c new file +9
-0+/* Try to receive packets (non-blocking) */ + nb_rx =rte_eth_rx_burst (port_id,0 , bufs,BURST_SIZE ); + +/* Free any received mbufs */ +for (i =0 ; i < nb_rx; i++) +rte_pktmbuf_free (bufs[i]); + +/* RX from tap interface without external traffic will return 0 */ +/* This test just verifies the RX path doesn't crash */
Consider adding a test that round trips data (tx -> rx) to exercise the read loop.
app/test/test_pmd_af_packet.c line 539
Quoted 6 lines
app/test/test_pmd_af_packet.c new file +6
-0+/* Get current MTU */ + ret =rte_eth_dev_get_mtu (port_id, &mtu); +TEST_ASSERT (ret ==0 ,"Failed to get MTU" ); + +/* Try to set a smaller MTU */ + ret =rte_eth_dev_set_mtu (port_id,1400 );
Consider getting the min/max supported MTU and setting it to verify it works. My patch fixed some issues related to MTU and overhead of frames that may have been caught by this.
app/test/test_pmd_af_packet.c line 343
Quoted 4 lines
app/test/test_pmd_af_packet.c new file +4
-0+ +/* Reset stats */ + ret =rte_eth_stats_reset (port_id); +TEST_ASSERT (ret ==0 ,"Failed to reset stats" );
Consider enhancing to introduce multiple threads (writer and reader) to replicate
Review
Quoted 2 lines
(quoted diff) +2
-0+static int +test_af_packet_multi_queue(void)
Consider sharing rx/tx logic in common functions for single/multi-queue cases.
Re: [RFC 2/4] test: add test for af_packet
Quoted 47 lines · On Wed, 28 Jan 2026 12:36:41 -0800 Scott Mitchell <[email protected]>…
Awesome! Some suggestions below but could easily be done in a followup for incremental progress.
app/test/test_pmd_af_packet.c line 412
Quoted 9 lines
app/test/test_pmd_af_packet.c new file +9
-0+/* TX burst */ + nb_tx =rte_eth_tx_burst (port_id,0 , bufs,BURST_SIZE ); + +/* Free any unsent mbufs */ +for (i = nb_tx; i <BURST_SIZE ; i++) +rte_pktmbuf_free (bufs[i]); + +/* Small delay to allow stats update */ +rte_delay_us_block (1000 );Consider polling
rte_eth_stats_getat a more frequent interval, break when you see status update, and apply overall timeout/repeat limit -> can make test more reliable and take less time when data is usually available quicklyapp/test/test_pmd_af_packet.c line 450
Quoted 9 lines
app/test/test_pmd_af_packet.c new file +9
-0+/* Try to receive packets (non-blocking) */ + nb_rx =rte_eth_rx_burst (port_id,0 , bufs,BURST_SIZE ); + +/* Free any received mbufs */ +for (i =0 ; i < nb_rx; i++) +rte_pktmbuf_free (bufs[i]); + +/* RX from tap interface without external traffic will return 0 */ +/* This test just verifies the RX path doesn't crash */ Consider adding a test that round trips data (tx -> rx) to exercise the read loop.
app/test/test_pmd_af_packet.c line 539
Quoted 6 lines
app/test/test_pmd_af_packet.c new file +6
-0+/* Get current MTU */ + ret =rte_eth_dev_get_mtu (port_id, &mtu); +TEST_ASSERT (ret ==0 ,"Failed to get MTU" ); + +/* Try to set a smaller MTU */ + ret =rte_eth_dev_set_mtu (port_id,1400 );Consider getting the min/max supported MTU and setting it to verify it works. My patch fixed some issues related to MTU and overhead of frames that may have been caught by this.
app/test/test_pmd_af_packet.c line 343
Quoted 4 lines
app/test/test_pmd_af_packet.c new file +4
-0+ +/* Reset stats */ + ret =rte_eth_stats_reset (port_id); +TEST_ASSERT (ret ==0 ,"Failed to reset stats" );
Quoted 7 lines
Consider enhancing to introduce multiple threads (writer and reader) to replicate
Review
Quoted 2 lines
(quoted diff) +2
-0+static int +test_af_packet_multi_queue(void)Consider sharing rx/tx logic in common functions for single/multi-queue cases.
Wow, AI could take that feedback and revise for a new version.
RFC 3/4 net/af_packet: fix indentation
In reply to Stephen HemmingerDPDK code is supposed to be indented with TABS not spaces. Also, used "unsigned int" to keep checkpatch happy.
--- drivers/net/af_packet/rte_eth_af_packet.c | 64 +++++++++++------------ 1 file changed, 32 insertions(+), 32 deletions(-)
drivers/net/af_packet/rte_eth_af_packet.c +32 -32
@@ -78,7 +78,7 @@ struct __rte_cache_aligned pkt_tx_queue { };struct pmd_internals { -unsigned nb_queues; +unsigned int nb_queues;int if_index;char *if_name;
@@ -537,7 +537,7 @@ eth_dev_close(struct rte_eth_dev *dev)static int eth_link_update (struct rte_eth_dev *dev, - int wait_to_complete __rte_unused) + int wait_to_complete __rte_unused) {const struct pmd_internals *internals = dev->data->dev_private;struct rte_eth_link *dev_link = &dev->data->dev_link;
@@ -557,11 +557,11 @@ eth_link_update(struct rte_eth_dev *dev,static int eth_rx_queue_setup (struct rte_eth_dev *dev, -uint16_t rx_queue_id, - uint16_t nb_rx_desc __rte_unused, - unsigned int socket_id __rte_unused, - const struct rte_eth_rxconf *rx_conf __rte_unused, -struct rte_mempool *mb_pool) +uint16_t rx_queue_id, + uint16_t nb_rx_desc __rte_unused, + unsigned int socket_id __rte_unused, + const struct rte_eth_rxconf *rx_conf __rte_unused, +struct rte_mempool *mb_pool) {struct pmd_internals *internals = dev->data->dev_private;struct pkt_rx_queue *pkt_q = &internals->rx_queue[rx_queue_id];
@@ -592,10 +592,10 @@ eth_rx_queue_setup(struct rte_eth_dev *dev,static int eth_tx_queue_setup (struct rte_eth_dev *dev, -uint16_t tx_queue_id, - uint16_t nb_tx_desc __rte_unused, - unsigned int socket_id __rte_unused, - const struct rte_eth_txconf *tx_conf __rte_unused) +uint16_t tx_queue_id, + uint16_t nb_tx_desc __rte_unused, + unsigned int socket_id __rte_unused, + const struct rte_eth_txconf *tx_conf __rte_unused) {struct pmd_internals *internals = dev->data->dev_private;
@@ -722,8 +722,8 @@ static const struct eth_dev_ops ops = {*/ static int open_packet_iface(const char *key __rte_unused, - const char *value __rte_unused, -void *extra_args) + const char *value __rte_unused, +void *extra_args) {int *sockfd = extra_args;
@@ -786,17 +786,17 @@ get_fanout(const char *fanout_mode, int if_index)static int rte_pmd_init_internals (struct rte_vdev_device *dev, -const int sockfd, -const unsigned nb_queues, -unsigned int blocksize, -unsigned int blockcnt, -unsigned int framesize, -unsigned int framecnt, +const int sockfd, +const unsigned int nb_queues, +unsigned int blocksize, +unsigned int blockcnt, +unsigned int framesize, +unsigned int framecnt,unsigned int qdisc_bypass,const char *fanout_mode, -struct pmd_internals **internals, -struct rte_eth_dev **eth_dev, -struct rte_kvargs *kvlist) +struct pmd_internals **internals, +struct rte_eth_dev **eth_dev, +struct rte_kvargs *kvlist) {const char *name =rte_vdev_device_name (dev);const unsigned int numa_node = dev->device.numa_node;
@@ -822,7 +822,7 @@ rte_pmd_init_internals(struct rte_vdev_device *dev,if (pair ==NULL ) {PMD_LOG (ERR ,"%s: no interface specified for AF_PACKET ethdev" , - name); + name);return -1 ; }
@@ -831,7 +831,7 @@ rte_pmd_init_internals(struct rte_vdev_device *dev, name, numa_node); *internals =rte_zmalloc_socket (name,sizeof (**internals), -0 , numa_node); +0 , numa_node);if (*internals ==NULL )return -1 ;
@@ -1072,8 +1072,8 @@ rte_pmd_init_internals(struct rte_vdev_device *dev,static int rte_eth_from_packet (struct rte_vdev_device *dev, -int const *sockfd, -struct rte_kvargs *kvlist) +int const *sockfd, +struct rte_kvargs *kvlist) {const char *name =rte_vdev_device_name (dev);struct pmd_internals *internals =NULL ;
@@ -1104,7 +1104,7 @@ rte_eth_from_packet(struct rte_vdev_device *dev,if (qpairs <1 ) {PMD_LOG (ERR ,"%s: invalid qpairs value" , - name); + name);return -1 ; }continue ;
@@ -1114,7 +1114,7 @@ rte_eth_from_packet(struct rte_vdev_device *dev,if (!blocksize) {PMD_LOG (ERR ,"%s: invalid blocksize value" , - name); + name);return -1 ; }continue ;
@@ -1124,7 +1124,7 @@ rte_eth_from_packet(struct rte_vdev_device *dev,if (!framesize) {PMD_LOG (ERR ,"%s: invalid framesize value" , - name); + name);return -1 ; }continue ;
@@ -1134,7 +1134,7 @@ rte_eth_from_packet(struct rte_vdev_device *dev,if (!framecount) {PMD_LOG (ERR ,"%s: invalid framecount value" , - name); + name);return -1 ; }continue ;
@@ -1158,7 +1158,7 @@ rte_eth_from_packet(struct rte_vdev_device *dev,if (framesize > blocksize) {PMD_LOG (ERR ,"%s: AF_PACKET MMAP frame size exceeds block size!" , - name); + name);return -1 ; }
@@ -1233,7 +1233,7 @@ rte_pmd_af_packet_probe(struct rte_vdev_device *dev)if (rte_kvargs_count (kvlist,ETH_AF_PACKET_IFACE_ARG ) ==1 ) { ret =rte_kvargs_process (kvlist,ETH_AF_PACKET_IFACE_ARG , - &open_packet_iface, &sockfd); + &open_packet_iface, &sockfd);if (ret <0 ) goto exit; }
Signature · 1 line
2.51.0
RFC 4/4 net/af_packet: add VPP-style prefetching to receive path
In reply to Stephen HemmingerImplement the single/dual/quad loop design pattern from FD.IO VPP to improve cache efficiency in the af_packet PMD receive path.
The original implementation processes packets one at a time in a simple loop, which can result in cache misses when accessing frame headers and packet data. The new implementation:
- Processes packets in batches of 4 (quad), 2 (dual), and 1 (single)
- Prefetches next batch of frame headers while processing current batch
- Prefetches packet data before memcpy to hide memory latency
- Reduces loop overhead through partial unrolling
Two helper functions are introduced:
- af_packet_get_frame(): Returns frame pointer at index with wraparound
- af_packet_rx_one(): Common per-packet processing (mbuf alloc, memcpy, VLAN handling, timestamp offload)
The quad loop checks availability of all 4 frames before processing, falling through to dual/single loops when fewer frames are ready. Early exit paths (out_advance1/2/3) ensure correct frame index tracking when mbuf allocation fails mid-batch.
Prefetch strategy:
- Frame headers: prefetch N+4..N+7 while processing N..N+3
- Packet data: prefetch at tp_mac offset before memcpy
This pattern is well-established in high-performance packet processing and should improve throughput by better utilizing CPU cache hierarchy, particularly beneficial when processing bursts of packets.
--- drivers/net/af_packet/rte_eth_af_packet.c | 208 +++++++++++++++++----- 1 file changed, 164 insertions(+), 44 deletions(-)
drivers/net/af_packet/rte_eth_af_packet.c +164 -44
@@ -14,6 +14,7 @@#include <rte_malloc.h> #include <rte_kvargs.h> #include <bus_vdev_driver.h> +#include <rte_prefetch.h> #include <errno.h> #include <linux/if_ether.h>
@@ -120,75 +121,194 @@ RTE_LOG_REGISTER_DEFAULT(af_packet_logtype, NOTICE);RTE_LOG_LINE (level,AFPACKET ,"%s(): " fmt":%s" , __func__, \ ## __VA_ARGS__, strerror(errno)) -static uint16_t -eth_af_packet_rx (void *queue,struct rte_mbuf **bufs,uint16_t nb_pkts) +/* +* Helper to get the frame pointer at a given index with wraparound + */ +static inline struct tpacket2_hdr * +af_packet_get_frame (struct pkt_rx_queue *pkt_q,unsigned int idx) +{ +if (idx >= pkt_q->framecount) + idx -= pkt_q->framecount; +return (struct tpacket2_hdr *)pkt_q->rd[idx].iov_base; +} + +/* +* Process a single received packet - common code for all loop variants +*/ +static inline int +af_packet_rx_one (struct pkt_rx_queue *pkt_q, +struct tpacket2_hdr *ppd, +struct rte_mbuf **mbuf_out, +unsigned long *rx_bytes) { -unsigned i; -struct tpacket2_hdr *ppd;struct rte_mbuf *mbuf;uint8_t *pbuf; + + mbuf =rte_pktmbuf_alloc (pkt_q->mb_pool); +if (unlikely (mbuf ==NULL )) { + pkt_q->rx_nombuf++; +return -1 ; + } + +rte_pktmbuf_pkt_len (mbuf) =rte_pktmbuf_data_len (mbuf) = ppd->tp_snaplen; + pbuf = (uint8_t *)ppd + ppd->tp_mac; + memcpy(rte_pktmbuf_mtod(mbuf, void *), pbuf, rte_pktmbuf_data_len(mbuf)); + +if (ppd->tp_status &TP_STATUS_VLAN_VALID ) { + mbuf->vlan_tci = ppd->tp_vlan_tci; + mbuf->ol_flags |= (RTE_MBUF_F_RX_VLAN |RTE_MBUF_F_RX_VLAN_STRIPPED ); +if (!pkt_q->vlan_strip &&rte_vlan_insert (&mbuf)) +PMD_LOG (ERR ,"Failed to reinsert VLAN tag" ); + } + +if (pkt_q->timestamp_offloading) { + *RTE_MBUF_DYNFIELD (mbuf, timestamp_dynfield_offset, + rte_mbuf_timestamp_t *) = + (uint64_t )ppd->tp_sec *1000000000 + ppd->tp_nsec; + mbuf->ol_flags |= timestamp_dynflag; + } + + mbuf->port = pkt_q->in_port; + *mbuf_out = mbuf; + *rx_bytes += mbuf->pkt_len; + ppd->tp_status =TP_STATUS_KERNEL ; + +return 0 ; +} + +/* +* Receive packets using VPP-style single/dual/quad loop pattern with prefetching. +*/ +static uint16_t +eth_af_packet_rx (void *queue,struct rte_mbuf **bufs,uint16_t nb_pkts) +{struct pkt_rx_queue *pkt_q = queue; +struct tpacket2_hdr *ppd0, *ppd1, *ppd2, *ppd3;uint16_t num_rx =0 ;unsigned long num_rx_bytes =0 ;unsigned int framecount, framenum; +uint16_t n_left;if (unlikely (nb_pkts ==0 ))return 0 ; -/* -* Reads the given number of packets from the AF_PACKET socket one by -* one and copies the packet data into a newly allocated mbuf. -*/ framecount = pkt_q->framecount; framenum = pkt_q->framenum; -for (i =0 ; i < nb_pkts; i++) { -/* point at the next incoming frame */ - ppd = (struct tpacket2_hdr *) pkt_q->rd[framenum].iov_base; -if ((ppd->tp_status &TP_STATUS_USER ) ==0 ) + n_left = nb_pkts; + +/* Quad loop: Process 4 packets at a time with prefetching */ +while (n_left >=4 ) { + ppd0 =af_packet_get_frame (pkt_q, framenum); + ppd1 =af_packet_get_frame (pkt_q, framenum +1 ); + ppd2 =af_packet_get_frame (pkt_q, framenum +2 ); + ppd3 =af_packet_get_frame (pkt_q, framenum +3 ); + +if ((ppd0->tp_status &TP_STATUS_USER ) ==0 )break ; +if ((ppd1->tp_status &TP_STATUS_USER ) ==0 ) + goto dual_loop; +if ((ppd2->tp_status &TP_STATUS_USER ) ==0 ) + goto dual_loop; +if ((ppd3->tp_status &TP_STATUS_USER ) ==0 ) + goto dual_loop; + +/* Prefetch next 4 frame headers */ +rte_prefetch0 (af_packet_get_frame (pkt_q, framenum +4 )); +rte_prefetch0 (af_packet_get_frame (pkt_q, framenum +5 )); +rte_prefetch0 (af_packet_get_frame (pkt_q, framenum +6 )); +rte_prefetch0 (af_packet_get_frame (pkt_q, framenum +7 )); + +/* Prefetch packet data */ +rte_prefetch0 ((uint8_t *)ppd0 + ppd0->tp_mac); +rte_prefetch0 ((uint8_t *)ppd1 + ppd1->tp_mac); +rte_prefetch0 ((uint8_t *)ppd2 + ppd2->tp_mac); +rte_prefetch0 ((uint8_t *)ppd3 + ppd3->tp_mac); + +if (unlikely (af_packet_rx_one (pkt_q, ppd0, &bufs[num_rx], &num_rx_bytes) <0 )) + goto out; + num_rx++; +if (unlikely (af_packet_rx_one (pkt_q, ppd1, &bufs[num_rx], &num_rx_bytes) <0 )) + goto out_advance1; + num_rx++; +if (unlikely (af_packet_rx_one (pkt_q, ppd2, &bufs[num_rx], &num_rx_bytes) <0 )) + goto out_advance2; + num_rx++; +if (unlikely (af_packet_rx_one (pkt_q, ppd3, &bufs[num_rx], &num_rx_bytes) <0 )) + goto out_advance3; + num_rx++; -/* allocate the next mbuf */ - mbuf =rte_pktmbuf_alloc (pkt_q->mb_pool); -if (unlikely (mbuf ==NULL )) { - pkt_q->rx_nombuf++; + framenum +=4 ; +if (framenum >= framecount) + framenum -= framecount; + n_left -=4 ; + } + +dual_loop: +/* Dual loop: Process 2 packets at a time */ +while (n_left >=2 ) { + ppd0 =af_packet_get_frame (pkt_q, framenum); + ppd1 =af_packet_get_frame (pkt_q, framenum +1 ); + +if ((ppd0->tp_status &TP_STATUS_USER ) ==0 )break ; - } +if ((ppd1->tp_status &TP_STATUS_USER ) ==0 ) + goto single_loop; -/* packet will fit in the mbuf, go ahead and receive it */ -rte_pktmbuf_pkt_len (mbuf) =rte_pktmbuf_data_len (mbuf) = ppd->tp_snaplen; - pbuf = (uint8_t *) ppd + ppd->tp_mac; - memcpy(rte_pktmbuf_mtod(mbuf, void *), pbuf, rte_pktmbuf_data_len(mbuf)); +rte_prefetch0 (af_packet_get_frame (pkt_q, framenum +2 )); +rte_prefetch0 (af_packet_get_frame (pkt_q, framenum +3 )); +rte_prefetch0 ((uint8_t *)ppd0 + ppd0->tp_mac); +rte_prefetch0 ((uint8_t *)ppd1 + ppd1->tp_mac); -/* check for vlan info */ -if (ppd->tp_status &TP_STATUS_VLAN_VALID ) { - mbuf->vlan_tci = ppd->tp_vlan_tci; - mbuf->ol_flags |= (RTE_MBUF_F_RX_VLAN |RTE_MBUF_F_RX_VLAN_STRIPPED ); +if (unlikely (af_packet_rx_one (pkt_q, ppd0, &bufs[num_rx], &num_rx_bytes) <0 )) + goto out; + num_rx++; +if (unlikely (af_packet_rx_one (pkt_q, ppd1, &bufs[num_rx], &num_rx_bytes) <0 )) + goto out_advance1; + num_rx++; -if (!pkt_q->vlan_strip &&rte_vlan_insert (&mbuf)) -PMD_LOG (ERR ,"Failed to reinsert VLAN tag" ); - } + framenum +=2 ; +if (framenum >= framecount) + framenum -= framecount; + n_left -=2 ; + } -/* add kernel provided timestamp when offloading is enabled */ -if (pkt_q->timestamp_offloading) { -/* since TPACKET_V2 timestamps are provided in nanoseconds resolution */ - *RTE_MBUF_DYNFIELD (mbuf, timestamp_dynfield_offset, - rte_mbuf_timestamp_t *) = - (uint64_t )ppd->tp_sec *1000000000 + ppd->tp_nsec; +single_loop: +/* Single loop: Process remaining packets */ +while (n_left >=1 ) { + ppd0 =af_packet_get_frame (pkt_q, framenum); - mbuf->ol_flags |= timestamp_dynflag; - } +if ((ppd0->tp_status &TP_STATUS_USER ) ==0 ) +break ; -/* release incoming frame and advance ring buffer */ - ppd->tp_status =TP_STATUS_KERNEL ; -if (++framenum >= framecount) - framenum =0 ; - mbuf->port = pkt_q->in_port; +rte_prefetch0 (af_packet_get_frame (pkt_q, framenum +1 )); +rte_prefetch0 ((uint8_t *)ppd0 + ppd0->tp_mac); -/* account for the receive frame */ - bufs[i] = mbuf; +if (unlikely (af_packet_rx_one (pkt_q, ppd0, &bufs[num_rx], &num_rx_bytes) <0 )) + goto out; num_rx++; - num_rx_bytes += mbuf->pkt_len; + +if (++framenum >= framecount) + framenum =0 ; + n_left--; } + + goto out; + +out_advance3: + framenum +=3 ; +if (framenum >= framecount) + framenum -= framecount; + goto out; +out_advance2: + framenum +=2 ; +if (framenum >= framecount) + framenum -= framecount; + goto out; +out_advance1: + framenum +=1 ; +if (framenum >= framecount) + framenum -= framecount; +out: pkt_q->framenum = framenum; pkt_q->rx_pkts += num_rx; pkt_q->rx_bytes += num_rx_bytes;
Signature · 1 line
2.51.0
Re: [RFC 4/4] net/af_packet: add VPP-style prefetching to receive path
Quoted 26 lines · On Wed, 28 Jan 2026 09:30:20 -0800 Stephen Hemminger…
Implement the single/dual/quad loop design pattern from FD.IO VPP to improve cache efficiency in the
af_packetPMD receive path.The original implementation processes packets one at a time in a simple loop, which can result in cache misses when accessing frame headers and packet data. The new implementation:
- Processes packets in batches of 4 (quad), 2 (dual), and 1 (single)
- Prefetches next batch of frame headers while processing current batch
- Prefetches packet data before memcpy to hide memory latency
- Reduces loop overhead through partial unrollingTwo helper functions are introduced:
-af_packet_get_frame(): Returns frame pointer at index with wraparound
-af_packet_rx_one(): Common per-packet processing (mbuf alloc, memcpy, VLAN handling, timestamp offload)The quad loop checks availability of all 4 frames before processing, falling through to dual/single loops when fewer frames are ready. Early exit paths (out_advance1/2/3) ensure correct frame index tracking when mbuf allocation fails mid-batch.
Prefetch strategy:
- Frame headers: prefetch N+4..N+7 while processing N..N+3
- Packet data: prefetch attp_macoffset before memcpy
Quoted 5 lines
This pattern is well-established in high-performance packet processing and should improve throughput by better utilizing CPU cache hierarchy, particularly beneficial when processing bursts of packets.
Signed-off-by Stephen Hemminger <[email protected]>
This and previous proposal to prefetch have no impact on performance. Rolled a simple perf test and all three versions come out the same. The bottleneck is not here, probably at system call and copies now.
Original Prefetch Quad/Dual
TX 1.427 Mpps 1.426 Mpps 1.426 Mpps
RX 0.529 Mpps 0.530 Mpps 0.533 Mpps
loss 87.93% 87.98% 88.0%
Original Prefetch Quad/Dual
TX 1.427 Mpps 1.426 Mpps 1.426 Mpps
RX 0.529 Mpps 0.530 Mpps 0.533 Mpps
loss 87.93% 87.98% 88.0%
Will put the test in the next version of this series, and drop this patch.
RE: [RFC 4/4] net/af_packet: add VPP-style prefetching to receive path
Quoted 42 lines
From: Stephen Hemminger [mailto:[email protected]]
Sent: Thursday, 29 January 2026 02.06Quoted 4 lines · On Wed, 28 Jan 2026 09:30:20 -0800 Stephen Hemminger…
Implement the single/dual/quad loop design pattern from FD.IO VPP to improve cache efficiency in the
af_packetPMD receive path.The original implementation processes packets one at a time in a
simple
Quoted 1 line
loop, which can result in cache misses when accessing frame headers
and
Quoted 4 lines
packet data. The new implementation:
- Processes packets in batches of 4 (quad), 2 (dual), and 1 (single)
- Prefetches next batch of frame headers while processing currentbatch
Quoted 5 lines
- Prefetches packet data before memcpy to hide memory latency
- Reduces loop overhead through partial unrollingTwo helper functions are introduced:
-af_packet_get_frame(): Returns frame pointer at index withwraparound
Quoted 1 line
-
af_packet_rx_one(): Common per-packet processing (mbuf alloc,memcpy,
Quoted 4 lines
VLAN handling, timestamp offload)
The quad loop checks availability of all 4 frames before processing, falling through to dual/single loops when fewer frames are ready.
Early
Quoted 1 line
exit paths (out_advance1/2/3) ensure correct frame index tracking
when
Quoted 7 lines
mbuf allocation fails mid-batch.
Prefetch strategy:
- Frame headers: prefetch N+4..N+7 while processing N..N+3
- Packet data: prefetch attp_macoffset before memcpyThis pattern is well-established in high-performance packet
processing
Quoted 1 line
and should improve throughput by better utilizing CPU cache
Quoted 8 lines
hierarchy,
Quoted 3 lines
particularly beneficial when processing bursts of packets.
Signed-off-by Stephen Hemminger <[email protected]>This and previous proposal to prefetch have no impact on performance. Rolled a simple perf test and all three versions come out the same.
Please be aware that many test cases are inadvertently designed in a way where mbufs unintendedly are hot in the cache, so prefetching does not provide the expected performance gain. E.g. when working on a newly allocated mbuf, the mbuf should be cold.
But if it came from the mempool cache, and was recently worked on and then freed into the mempool cache, then it will be hot.
Quoted 1 line
The bottleneck is not here, probably at system call and copies now.
The most important bottleneck might be elsewhere.
But this optimization might not be as irrelevant as the test results indicate.
Anyway, I agree that dropping the patch (for now) makes sense.
Quoted 17 lines
Original Prefetch Quad/Dual
TX 1.427 Mpps 1.426 Mpps 1.426 MppsRX 0.529 Mpps 0.530 Mpps 0.533 Mpps
loss 87.93% 87.98% 88.0%Original Prefetch Quad/Dual
TX 1.427 Mpps 1.426 Mpps 1.426 MppsRX 0.529 Mpps 0.530 Mpps 0.533 Mpps
loss 87.93% 87.98% 88.0%Will put the test in the next version of this series, and drop this patch.
Re: [RFC 4/4] net/af_packet: add VPP-style prefetching to receive path
Thanks for running benchmarks! Dropping for now sounds good (I did the same in my patch).
Re: [RFC 4/4] net/af_packet: add VPP-style prefetching to receive path
In reply to Morten BrørupQuoted 7 lines · On Thu, 29 Jan 2026 10:00:15 +0100 Morten Brørup <[email protected]>…
Quoted 2 lines
This and previous proposal to prefetch have no impact on performance. Rolled a simple perf test and all three versions come out the same.
Please be aware that many test cases are inadvertently designed in a way where mbufs unintendedly are hot in the cache, so prefetching does not provide the expected performance gain. E.g. when working on a newly allocated mbuf, the mbuf should be cold.
But if it came from the mempool cache, and was recently worked on and then freed into the mempool cache, then it will be hot.
Quoted 6 lines
Quoted 1 line
The bottleneck is not here, probably at system call and copies now.
The most important bottleneck might be elsewhere.
But this optimization might not be as irrelevant as the test results indicate.Anyway, I agree that dropping the patch (for now) makes sense.
I doubt pre-fetch will matter much in a driver like this because:
- on tx the data is still in cache since just setup by caller
- on rx the data is still in cache since kernel just copied it into the buffer.
RE: [RFC 4/4] net/af_packet: add VPP-style prefetching to receive path
Quoted 19 lines
From: Stephen Hemminger [mailto:[email protected]]
Sent: Monday, 2 February 2026 19.44Quoted 1 line · On Thu, 29 Jan 2026 10:00:15 +0100 Morten Brørup <[email protected]>…
Quoted 1 line
This and previous proposal to prefetch have no impact on
performance.
Quoted 3 lines
Quoted 1 line
Rolled a simple perf test and all three versions come out the same.
Please be aware that many test cases are inadvertently designed in a
way where mbufs unintendedly are hot in the cache, so prefetching does not provide the expected performance gain.
Quoted 2 lines
E.g. when working on a newly allocated mbuf, the mbuf should be cold. But if it came from the mempool cache, and was recently worked on and
then freed into the mempool cache, then it will be hot.
Quoted 3 lines
Quoted 2 lines
The bottleneck is not here, probably at system call and copies now.
Quoted 8 lines
Quoted 2 lines
The most important bottleneck might be elsewhere.
But this optimization might not be as irrelevant as the test resultsindicate.
Quoted 2 lines
Anyway, I agree that dropping the patch (for now) makes sense.
I doubt pre-fetch will matter much in a driver like this because:
- on tx the data is still in cache since just setup by caller
If the packet has been sitting in a queue, e.g. for QoS buffering, it may have been evicted from the cache.
Quoted 2 lines
- on rx the data is still in cache since kernel just copied it into the buffer.
If there's any delay between the signaling and the handling, it could be evicted from the cache in the meantime. (The delay could come from e.g. the handling CPU core being busy with something else before handling the RX ready signal.) And it could be in the cache of a different CPU core.
Anyway, I still agree with dropping the patch for now, since testing shows no difference.