iavf: defer loading netmem_desc
cover iavf: defer loading netmem_desc
In a few places libeth and the iavf driver inspect which page pool is attached to a netmem_ref. This forces an immediate load of the netmem_desc struct for each buffer in the Rx loop.
Defer or eliminate these loads of netmem_desc from the driver fast path by using the page_pool ref in the first cache line of iavf_ring instead.
There are only two paths out of the Rx loop where netmem_desc needs to be consumed:
* The "very rare" case of libeth_rx_sync_for_cpu calling libeth_rx_recycle_slow and indicating that there was no data. This could be similarly factored out, but not by this series.
* GRO merging a frame into an existing aggregate stream. In this case, the cold load of netmem_desc may overlap the payload prefetch started by iavf_build_skb, which is now no longer dependent on netmem_desc to start.
Matt Vollrath (3):
libeth: add __libeth_rx_sync_for_cpu
iavf: use __libeth_rx_sync_for_cpu
iavf: use cached page_pool ref in skb helpers
drivers/net/ethernet/intel/iavf/iavf_txrx.c | 18 ++++++---- include/net/libeth/rx.h | 37 +++++++++++++++------ 2 files changed, 37 insertions(+), 18 deletions(-)
base-commit: 5c458073553f0ef74f5c8db1bd459c87c722a299
Signature · 1 line
2.43.0
PATCH 1/3 libeth: add __libeth_rx_sync_for_cpu
Deriving a netmem's page pool from the ref requires loading a pointer from the netmem_desc, which may not be cached. The driver may already have a cached reference to the correct page pool, so add a bypass.
No functional change for existing callers.
--- include/net/libeth/rx.h | 37 ++++++++++++++++++++++++++----------- 1 file changed, 26 insertions(+), 11 deletions(-)
include/net/libeth/rx.h +26 -11
@@ -118,36 +118,51 @@ static inline dma_addr_t libeth_rx_alloc(const struct libeth_fq_fp *fq, u32 i)void libeth_rx_recycle_slow (netmem_ref netmem);/** -* libeth_rx_sync_for_cpu - synchronize or recycle buffer post DMA +* __libeth_rx_sync_for_cpu - synchronize or recycle buffer post DMA +* @pool: &page_pool the buffer was allocated from * @fqe: buffer to process * @len: frame length from the descriptor * -* Process the buffer after it's written by HW. The regular path is to -* synchronize DMA for CPU, but in case of no data it will be immediately -* recycled back to its PP. +* Variant for callers which already know the buffer's pool, typically +* their Rx queue's fill queue pool. * * Return: true when there's data to process, false otherwise. */ -static inline bool libeth_rx_sync_for_cpu (const struct libeth_fqe *fqe, -u32 len) +static inline bool __libeth_rx_sync_for_cpu (const struct page_pool *pool, +const struct libeth_fqe *fqe, +u32 len) { -netmem_ref netmem = fqe->netmem; -/* Very rare, but possible case. The most common reason: * the last fragment contained FCS only, which was then * stripped by the HW. */ if (unlikely (!len)) { -libeth_rx_recycle_slow (netmem); +libeth_rx_recycle_slow (fqe->netmem);return false; } -page_pool_dma_sync_netmem_for_cpu (netmem_get_pp (netmem), netmem, - fqe->offset, len); +page_pool_dma_sync_netmem_for_cpu (pool, fqe->netmem, fqe->offset, len);return true; } +/** +* libeth_rx_sync_for_cpu - synchronize or recycle buffer post DMA +* @fqe: buffer to process +* @len: frame length from the descriptor +* +* Process the buffer after it's written by HW. The regular path is to +* synchronize DMA for CPU, but in case of no data it will be immediately +* recycled back to its PP. +* +* Return: true when there's data to process, false otherwise. +*/ +static inline bool libeth_rx_sync_for_cpu (const struct libeth_fqe *fqe, +u32 len) +{ +return __libeth_rx_sync_for_cpu (netmem_get_pp (fqe->netmem), fqe, len); +} +/* Converting abstract packet type numbers into a software structure with * the packet parameters to do O(1) lookup on Rx. */
Signature · 1 line
2.43.0
RE: [Intel-wired-lan] [PATCH iwl-next 1/3] libeth: add __libeth_rx_sync_for_cpu
Quoted 39 lines
-----Original Message-----
From: Intel-wired-lan <[email protected]> On Behalf Of Matt Vollrath
Sent: Friday, July 31, 2026 2:41 PM
To: [email protected]
Cc: [email protected]; Lobakin, Aleksander <[email protected]>; Nguyen, Anthony L <[email protected]>; Kitszel, Przemyslaw <[email protected]>; Andrew Lunn <[email protected]>; David S . Miller <[email protected]>; Eric Dumazet <[email protected]>; Jakub Kicinski <[email protected]>; Paolo Abeni <[email protected]>; Simon Horman <[email protected]>; Matt Vollrath <[email protected]>
Subject: [Intel-wired-lan] [PATCH iwl-next 1/3] libeth: add__libeth_rx_sync_for_cpuDeriving a netmem's page pool from the ref requires loading a pointer from the
netmem_desc, which may not be cached. The driver may already have a cached reference to the correct page pool, so add a bypass.No functional change for existing callers.
Signed-off-by Matt Vollrath <[email protected]>Assisted-by Claude:claude-5-fable--- include/net/libeth/rx.h | 37 ++++++++++++++++++++++++++----------- 1 file changed, 26 insertions(+), 11 deletions(-)include/net/libeth/rx.h index renamed from include/net/libeth/rx.h +0
-0
5d991404845e..fddac084e001100644include/net/libeth/rx.h +0
-0@@ -118,36 +118,51 @@ static inline dma_addr_t libeth_rx_alloc(conststruct
libeth_fq_fp*fq, u32 i) voidlibeth_rx_recycle_slow(netmem_ref netmem);/**
...
Quoted 6 lines
/* Converting abstract packet type numbers into a software structure
with
* the packet parameters to do O(1) lookup on Rx.
*/
--
2.43.0
PATCH 2/3 iavf: use __libeth_rx_sync_for_cpu
In reply to Matt VollrathThe Rx ring already has a reference to its page pool adjacent to fields touched every iteration. Avoid loading the netmem_desc for sync by taking the shortcut.
--- drivers/net/ethernet/intel/iavf/iavf_txrx.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/net/ethernet/intel/iavf/iavf_txrx.c +2 -1
@@ -1387,6 +1387,7 @@ static int iavf_clean_rx_irq(struct iavf_ring *rx_ring, int budget) {bool flex = rx_ring->rxdid ==VIRTCHNL_RXDID_2_FLEX_SQ_NIC ;unsigned int total_rx_bytes =0 , total_rx_packets =0 ; +const struct page_pool *pp = rx_ring->pp;struct sk_buff *skb = rx_ring->skb;u16 cleaned_count =IAVF_DESC_UNUSED (rx_ring);bool failure = false;
@@ -1424,7 +1425,7 @@ static int iavf_clean_rx_irq(struct iavf_ring *rx_ring, int budget)iavf_trace (clean_rx_irq, rx_ring, rx_desc, skb); rx_buffer = &rx_ring->rx_fqes[rx_ring->next_to_clean]; -if (!libeth_rx_sync_for_cpu (rx_buffer, fields.len)) +if (!__libeth_rx_sync_for_cpu (pp, rx_buffer, fields.len)) goto skip_data;/* retrieve a buffer from the ring */
Signature · 1 line
2.43.0
RE: [Intel-wired-lan] [PATCH iwl-next 2/3] iavf: use __libeth_rx_sync_for_cpu
Quoted 51 lines
-----Original Message-----
From: Intel-wired-lan <[email protected]> On Behalf Of Matt Vollrath
Sent: Friday, July 31, 2026 2:41 PM
To: [email protected]
Cc: [email protected]; Lobakin, Aleksander <[email protected]>; Nguyen, Anthony L <[email protected]>; Kitszel, Przemyslaw <[email protected]>; Andrew Lunn <[email protected]>; David S . Miller <[email protected]>; Eric Dumazet <[email protected]>; Jakub Kicinski <[email protected]>; Paolo Abeni <[email protected]>; Simon Horman <[email protected]>; Matt Vollrath <[email protected]>
Subject: [Intel-wired-lan] [PATCH iwl-next 2/3] iavf: use__libeth_rx_sync_for_cpuThe Rx ring already has a reference to its page pool adjacent to fields touched every iteration. Avoid loading the
netmem_descfor sync by taking the shortcut.Signed-off-by Matt Vollrath <[email protected]>Assisted-by Claude:claude-5-fable--- drivers/net/ethernet/intel/iavf/iavf_txrx.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)drivers/net/ethernet/intel/iavf/iavf_txrx.c +2
-1@@ -1387,6 +1387,7 @@ static int iavf_clean_rx_irq(struct iavf_ring *rx_ring, int budget) {bool flex = rx_ring->rxdid ==VIRTCHNL_RXDID_2_FLEX_SQ_NIC ;unsigned int total_rx_bytes =0 , total_rx_packets =0 ; +const struct page_pool *pp = rx_ring->pp;struct sk_buff *skb = rx_ring->skb;u16 cleaned_count =IAVF_DESC_UNUSED (rx_ring);bool failure = false;@@ -1424,7 +1425,7 @@ static int iavf_clean_rx_irq(struct iavf_ring *rx_ring, int budget)iavf_trace (clean_rx_irq, rx_ring, rx_desc, skb); rx_buffer = &rx_ring->rx_fqes[rx_ring->next_to_clean]; -if (!libeth_rx_sync_for_cpu (rx_buffer, fields.len)) +if (!__libeth_rx_sync_for_cpu (pp, rx_buffer, fields.len)) goto skip_data;
Quoted 3 lines
/* retrieve a buffer from the ring */
--
2.43.0
PATCH 3/3 iavf: use cached page_pool ref in skb helpers
In reply to Matt VollrathDeriving a reference to the page pool from netmem_desc requires loading the struct. We already have a cached reference to the pool, so use that instead.
--- drivers/net/ethernet/intel/iavf/iavf_txrx.c | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-)
drivers/net/ethernet/intel/iavf/iavf_txrx.c +9 -6
@@ -1184,6 +1184,7 @@ static bool iavf_cleanup_headers(struct iavf_ring *rx_ring, struct sk_buff *skb)/** * iavf_add_rx_frag - Add contents of Rx buffer to sk_buff +* @pp: page pool the buffer was allocated from * @skb: sk_buff to place the data into * @rx_buffer: buffer containing page to add * @size: packet length from rx_desc
@@ -1193,11 +1194,11 @@ static bool iavf_cleanup_headers(struct iavf_ring *rx_ring, struct sk_buff *skb)* * The function will then update the page offset. **/ -static void iavf_add_rx_frag (struct sk_buff *skb, +static void iavf_add_rx_frag (const struct page_pool *pp,struct sk_buff *skb,const struct libeth_fqe *rx_buffer,unsigned int size) { -u32 hr =netmem_get_pp (rx_buffer->netmem)->p.offset; +u32 hr = pp->p.offset; skb_add_rx_frag_netmem(skb, skb_shinfo(skb)->nr_frags, rx_buffer->netmem, rx_buffer->offset + hr,
@@ -1206,17 +1207,19 @@ static void iavf_add_rx_frag(struct sk_buff *skb,/** * iavf_build_skb - Build skb around an existing buffer +* @pp: page pool the buffer was allocated from * @rx_buffer: Rx buffer to pull data from * @size: size of buffer to add to skb * * This function builds an skb around an existing Rx buffer, taking care * to set up the skb correctly and avoid any memcpy overhead. */ -static struct sk_buff *iavf_build_skb (const struct libeth_fqe *rx_buffer, +static struct sk_buff *iavf_build_skb (const struct page_pool *pp, +const struct libeth_fqe *rx_buffer,unsigned int size) {struct page *buf_page =__netmem_to_page (rx_buffer->netmem); -u32 hr =pp_page_to_nmdesc (buf_page)->pp->p.offset; +u32 hr = pp->p.offset;struct sk_buff *skb;void *va;
@@ -1430,9 +1433,9 @@ static int iavf_clean_rx_irq(struct iavf_ring *rx_ring, int budget)/* retrieve a buffer from the ring */ if (skb) -iavf_add_rx_frag (skb, rx_buffer, fields.len); +iavf_add_rx_frag (pp, skb, rx_buffer, fields.len);else - skb =iavf_build_skb (rx_buffer, fields.len); + skb =iavf_build_skb (pp, rx_buffer, fields.len);/* exit if we failed to retrieve a buffer */ if (!skb) {
Signature · 1 line
2.43.0
RE: [Intel-wired-lan] [PATCH iwl-next 3/3] iavf: use cached page_pool ref in skb helpers
Quoted 96 lines
-----Original Message-----
From: Intel-wired-lan <[email protected]> On Behalf Of Matt Vollrath
Sent: Friday, July 31, 2026 2:41 PM
To: [email protected]
Cc: [email protected]; Lobakin, Aleksander <[email protected]>; Nguyen, Anthony L <[email protected]>; Kitszel, Przemyslaw <[email protected]>; Andrew Lunn <[email protected]>; David S . Miller <[email protected]>; Eric Dumazet <[email protected]>; Jakub Kicinski <[email protected]>; Paolo Abeni <[email protected]>; Simon Horman <[email protected]>; Matt Vollrath <[email protected]>
Subject: [Intel-wired-lan] [PATCH iwl-next 3/3] iavf: use cachedpage_poolref in skb helpersDeriving a reference to the page pool from
netmem_descrequires loading the struct. We already have a cached reference to the pool, so use that instead.Signed-off-by Matt Vollrath <[email protected]>Assisted-by Claude:claude-5-fable--- drivers/net/ethernet/intel/iavf/iavf_txrx.c | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-)drivers/net/ethernet/intel/iavf/iavf_txrx.c +1
-0@@ -1184,6 +1184,7 @@ static bool iavf_cleanup_headers(struct iavf_ring *rx_ring, struct sk_buff *skb)/** * iavf_add_rx_frag - Add contents of Rx buffer to sk_buff +* @pp: page pool the buffer was allocated from * @skb: sk_buff to place the data into * @rx_buffer: buffer containing page to add * @size: packet length from
rx_descdrivers/net/ethernet/intel/iavf/iavf_txrx.c +1
-1@@ -1193,11 +1194,11 @@ static bool iavf_cleanup_headers(structiavf_ring *rx_ring, struct sk_buff *skb) * * The function will then update the page offset. **/ -static void iavf_add_rx_frag(struct sk_buff *skb, +static void iavf_add_rx_frag(const struct page_pool *pp, struct
sk_buffdrivers/net/ethernet/intel/iavf/iavf_txrx.c +3
-1+*skb, const struct libeth_fqe *rx_buffer, unsigned int size) { - u32 hr = netmem_get_pp(rx_buffer->netmem)->p.offset; +u32 hr = pp->p.offset; skb_add_rx_frag_netmem(skb, skb_shinfo(skb)->nr_frags, rx_buffer->netmem, rx_buffer->offset + hr,@@ -1206,17 +1207,19 @@ static void iavf_add_rx_frag(struct sk_buff *skb,/** * iavf_build_skb - Build skb around an existing buffer +* @pp: page pool the buffer was allocated from * @rx_buffer: Rx buffer to pull data from * @size: size of buffer to add to skb * * This function builds an skb around an existing Rx buffer, taking care
* to set up the skb correctly and avoid any memcpy overhead. */
-static structsk_buff*iavf_build_skb(const structlibeth_fqe*rx_buffer,drivers/net/ethernet/intel/iavf/iavf_txrx.c +5
-3+static struct sk_buff *iavf_build_skb (const struct page_pool *pp, +const struct libeth_fqe *rx_buffer,unsigned int size) {struct page *buf_page =__netmem_to_page (rx_buffer->netmem); -u32 hr =pp_page_to_nmdesc (buf_page)->pp->p.offset; +u32 hr = pp->p.offset;struct sk_buff *skb;void *va;@@ -1430,9 +1433,9 @@ static int iavf_clean_rx_irq(struct iavf_ring *rx_ring, int budget)/* retrieve a buffer from the ring */ if (skb) -iavf_add_rx_frag (skb, rx_buffer, fields.len); +iavf_add_rx_frag (pp, skb, rx_buffer, fields.len);else - skb =iavf_build_skb (rx_buffer, fields.len); + skb =iavf_build_skb (pp, rx_buffer, fields.len);/* exit if we failed to retrieve a buffer */
Quoted 3 lines
if (!skb) {
--
2.43.0
Alexander Lobakin
maintainerFrom: Matt Vollrath <[email protected]>
Date: Fri, 31 Jul 2026 08:41:06 -0400
Quoted 6 lines
In a few places libeth and the iavf driver inspect which page pool is attached to a
netmem_ref. This forces an immediate load of thenetmem_descstruct for each buffer in the Rx loop.Defer or eliminate these loads of
netmem_descfrom the driver fast path by using thepage_poolref in the first cache line ofiavf_ringinstead.
I was thinking of this when implementing the current design, but:
Do you have any data to prove this actually helps performance?
Quoted 10 lines
There are only two paths out of the Rx loop where
netmem_descneeds to be consumed:
* The "very rare" case oflibeth_rx_sync_for_cpucallinglibeth_rx_recycle_slowand indicating that there was no data. This could be similarly factored out, but not by this series.
* GRO merging a frame into an existing aggregate stream. In this case, the cold load ofnetmem_descmay overlap the payload prefetch started byiavf_build_skb, which is now no longer dependent onnetmem_descto start.
Thanks,
Olek
Quoted 7 lines · Alexander Lobakin
From: Matt Vollrath <[email protected]>
Date: Fri, 31 Jul 2026 08:41:06 -0400Quoted 4 lines
In a few places libeth and the iavf driver inspect which page pool is attached to a
netmem_ref. This forces an immediate load of thenetmem_descstruct for each buffer in the Rx loop.
Quoted 6 lines
Quoted 2 lines
Defer or eliminate these loads of
netmem_descfrom the driver fast path by using thepage_poolref in the first cache line ofiavf_ringinstead.I was thinking of this when implementing the current design, but:
Do you have any data to prove this actually helps performance?
The optimization was profile-driven, but not on an iavf. I was working on a libeth port of e1000e, because I'm an obsessive optimizer, and this quirk of the interface fell out. I'm working with a Xeon E3-1240 v3 and I218-V.
With a mirror of the changes in this patch series, on my platform, it's a net zero overall change in Rx performance. This is not surprising, because the cold read is just deferred until later. The time saved in eth_type_trans (where the payload is first read) is instead spent in skb_gro_receive and pool puts.
The additional thing that does make an improvement, which was omitted from this series, is prefetching netmem_desc early in the loop. This makes about 22 cycles/packet, 3% difference.
I left it where it is because I don't have hardware to perf this on, and didn't know if anyone would even be interested in looking at something that I can't produce data for on the actual hardware. For iavf the prefetch would be conditional, only when the EOP flag is set, because in this case netmem_desc will always be needed either for GRO merge or pool put.
I can't guarantee that this will be worth 3% on an iavf platform. In any case, it'll be a couple weeks before I can work on this again.
Quoted 11 lines
Quoted 11 lines
There are only two paths out of the Rx loop where
netmem_descneeds to be consumed:
* The "very rare" case oflibeth_rx_sync_for_cpucallinglibeth_rx_recycle_slowand indicating that there was no data. This could be similarly factored out, but not by this series.
* GRO merging a frame into an existing aggregate stream. In this case, the cold load ofnetmem_descmay overlap the payload prefetch started byiavf_build_skb, which is now no longer dependent onnetmem_descto start.
I see now this isn't right, anything going to GRO will end up back in the pool in the same iteration, and netmem_desc will be loaded. It just happens later.
Quoted 3 lines
Thanks,
Olek
Alexander Lobakin
maintainerFrom: Matt Vollrath <[email protected]>
Date: Fri, 31 Jul 2026 15:27:37 -0400
Quoted 29 lines · Alexander Lobakin
Quoted 7 lines
From: Matt Vollrath <[email protected]>
Date: Fri, 31 Jul 2026 08:41:06 -0400Quoted 4 lines
In a few places libeth and the iavf driver inspect which page pool is attached to a
netmem_ref. This forces an immediate load of thenetmem_descstruct for each buffer in the Rx loop.Quoted 6 lines
Quoted 2 lines
Defer or eliminate these loads of
netmem_descfrom the driver fast path by using thepage_poolref in the first cache line ofiavf_ringinstead.I was thinking of this when implementing the current design, but:
Do you have any data to prove this actually helps performance?
The optimization was profile-driven, but not on an iavf. I was working on a libeth port of e1000e, because I'm an obsessive optimizer, and this quirk of the interface fell out. I'm working with a Xeon E3-1240 v3 and I218-V.
With a mirror of the changes in this patch series, on my platform, it's a net zero overall change in Rx performance. This is not surprising, because the cold read is just deferred until later. The time saved in
eth_type_trans(where the payload is first read) is instead spent inskb_gro_receiveand pool puts.The additional thing that does make an improvement, which was omitted from this series, is prefetching
netmem_descearly in the loop. This makes about 22 cycles/packet, 3% difference.
Quoted 8 lines
I left it where it is because I don't have hardware to perf this on, and didn't know if anyone would even be interested in looking at something that I can't produce data for on the actual hardware. For iavf the prefetch would be conditional, only when the EOP flag is set, because in this case
netmem_descwill always be needed either for GRO merge or pool put.I can't guarantee that this will be worth 3% on an iavf platform. In any case, it'll be a couple weeks before I can work on this again.
I asked because I was testing this approach on iavf and idpf when developing the initial libeth code. But turned out that there was [almost] no difference, while passing struct page_pool * to each libeth Rx helper was very inconvenient and ugly. So I left it as it is currently.
Moreover, most of libeth_xdp code couldn't take a separate PP pointer, so it would make no sense at all since we're converting more and more drivers to libeth_xdp helpers directly instead of raw libeth_rx. And also, with XDP at least, the core either way accesses netmem_desc->pp.
So honestly I wouldn't make this change for some ghost benefit (the current code is either way capable of handling 50-60 Mpps on XDP_DROP despite that it reads netmem_desc->pp for each frame).
Quoted 13 lines
Quoted 12 lines
Quoted 12 lines
There are only two paths out of the Rx loop where
netmem_descneeds to be consumed:
* The "very rare" case oflibeth_rx_sync_for_cpucallinglibeth_rx_recycle_slowand indicating that there was no data. This could be similarly factored out, but not by this series.
* GRO merging a frame into an existing aggregate stream. In this case, the cold load ofnetmem_descmay overlap the payload prefetch started byiavf_build_skb, which is now no longer dependent onnetmem_descto start.
Quoted 3 lines
I see now this isn't right, anything going to GRO will end up back in the pool in the same iteration, and
netmem_descwill be loaded. It just happens later.
Thanks,
Olek