mbuf fast-free requirements analysis
Executive Summary:
My analysis shows that the mbuf library is not a barrier for fast-freeing segmented packet mbufs, and thus fast-free of jumbo frames is possible.
Detailed Analysis:
The purpose of the mbuf fast-free Tx optimization is to reduce rte_pktmbuf_free_seg() to something much simpler in the ethdev drivers, by eliminating the code path related to indirect mbufs. Optimally, we want to simplify the ethdev driver's function that frees the transmitted mbufs, so it can free them directly to their mempool without accessing the mbufs themselves.
If the driver cannot access the mbuf itself, it cannot determine which mempool it belongs to.
We don't want the driver to access every mbuf being freed; but if all mbufs of a Tx queue belong to the same mempool, the driver can determine which mempool by looking into just one of the mbufs.
REQUIREMENT 1: The mbufs of a Tx queue must come from the same mempool.
When an mbuf is freed to its mempool, some of the fields in the mbuf must be initialized.
So, for fast-free, this must be done by the driver's function that prepares the Tx descriptor.
This is a requirement to the driver, not a requirement to the application.
Now, let's dig into the code for freeing an mbuf.
Note: For readability purposes, I'll cut out some code and comments unrelated to this topic.
static __rte_always_inline voidrte_pktmbuf_free_seg(struct rte_mbuf *m)
{
m = rte_pktmbuf_prefree_seg(m);
if (likely(m != NULL))
rte_mbuf_raw_free(m);
}rte_mbuf_raw_free(m) is simple, so nothing to gain there:
/**
* Put mbuf back into its original mempool.
*
* The caller must ensure that the mbuf is direct and properly
* reinitialized (refcnt=1, next=NULL, nb_segs=1), as done by
* rte_pktmbuf_prefree_seg().
*/
static __rte_always_inline void rte_mbuf_raw_free(struct rte_mbuf *m)
{
rte_mbuf_history_mark(m, RTE_MBUF_HISTORY_OP_LIB_FREE);
rte_mempool_put(m->pool, m);
}Note that the description says that the mbuf must be direct. This is not entirely accurate; the mbuf is allowed to use a pinned external buffer, if the mbuf holds the only reference to it. (Most of the mbuf library functions have this documentation inaccuracy, which should be fixed some day.)
So, the fast-free optimization really comes down to rte_pktmbuf_prefree_seg(m), which must not return NULL.
Let's dig into that.
/**
* Decrease reference counter and unlink a mbuf segment *
* This function does the same than a free, except that it does not
* return the segment to its pool.
* It decreases the reference counter, and if it reaches 0, it is
* detached from its parent for an indirect mbuf.
*
* @return
* - (m) if it is the last reference. It can be recycled or freed.
* - (NULL) if the mbuf still has remaining references on it.
*/
static __rte_always_inline struct rte_mbuf * rte_pktmbuf_prefree_seg(struct rte_mbuf *m)
{
bool refcnt_not_one;
refcnt_not_one = unlikely(rte_mbuf_refcnt_read(m) != 1);
if (refcnt_not_one && __rte_mbuf_refcnt_update(m, -1) != 0)
return NULL;
if (unlikely(!RTE_MBUF_DIRECT(m))) {
rte_pktmbuf_detach(m);
if (RTE_MBUF_HAS_EXTBUF(m) &&
RTE_MBUF_HAS_PINNED_EXTBUF(m) &&
__rte_pktmbuf_pinned_extbuf_decref(m))
return NULL;
}
if (refcnt_not_one)
rte_mbuf_refcnt_set(m, 1);
if (m->nb_segs != 1)
m->nb_segs = 1;
if (m->next != NULL)
m->next = NULL;
return m;
}This function can only succeed (i.e. return non-NULL) when 'refcnt' is 1 (or reaches 0).
REQUIREMENT 2: The driver must hold the only reference to the mbuf, i.e. 'm->refcnt' must be 1.
When the function succeeds, it initializes the mbuf fields as required by rte_mbuf_raw_free() before returning.
Now, since the driver has exclusive access to the mbuf, it is free to initialize the 'm->next' and 'm->nb_segs' at any time.
It could do that when preparing the Tx descriptor.
This is very interesting, because it means that fast-free does not prohibit segmented packets!
(But the driver must have sufficient Tx descriptors for all segments in the mbuf.)
Now, lets dig into rte_pktmbuf_prefree_seg()'s block handling non-direct mbufs, i.e. cloned mbufs and mbufs with external buffer:
if (unlikely(!RTE_MBUF_DIRECT(m))) {
rte_pktmbuf_detach(m);
if (RTE_MBUF_HAS_EXTBUF(m) &&
RTE_MBUF_HAS_PINNED_EXTBUF(m) &&
__rte_pktmbuf_pinned_extbuf_decref(m))
return NULL;
}Starting with rte_pktmbuf_detach():
static inline void rte_pktmbuf_detach(struct rte_mbuf *m)
{
struct rte_mempool *mp = m->pool;
uint32_t mbuf_size, buf_len;
uint16_t priv_size;
if (RTE_MBUF_HAS_EXTBUF(m)) {
/*
* The mbuf has the external attached buffer,
* we should check the type of the memory pool where
* the mbuf was allocated from to detect the pinned
* external buffer.
*/
uint32_t flags = rte_pktmbuf_priv_flags(mp);
if (flags & RTE_PKTMBUF_POOL_F_PINNED_EXT_BUF) {
/*
* The pinned external buffer should not be
* detached from its backing mbuf, just exit.
*/
return;
}
__rte_pktmbuf_free_extbuf(m);
} else {
__rte_pktmbuf_free_direct(m);
}
priv_size = rte_pktmbuf_priv_size(mp);
mbuf_size = (uint32_t)(sizeof(struct rte_mbuf) + priv_size);
buf_len = rte_pktmbuf_data_room_size(mp);
m->priv_size = priv_size;
m->buf_addr = (char *)m + mbuf_size;
rte_mbuf_iova_set(m, rte_mempool_virt2iova(m) + mbuf_size);
m->buf_len = (uint16_t)buf_len;
rte_pktmbuf_reset_headroom(m);
m->data_len = 0;
m->ol_flags = 0;
}The only quick and simple code path through this function is when the mbuf uses a pinned external buffer:
if (RTE_MBUF_HAS_EXTBUF(m)) {
uint32_t flags = rte_pktmbuf_priv_flags(mp);
if (flags & RTE_PKTMBUF_POOL_F_PINNED_EXT_BUF)
return;REQUIREMENT 3: The mbuf must not be cloned or use a non-pinned external buffer.
Continuing with the next part of rte_pktmbuf_prefree_seg()'s block:
if (RTE_MBUF_HAS_EXTBUF(m) &&
RTE_MBUF_HAS_PINNED_EXTBUF(m) && __rte_pktmbuf_pinned_extbuf_decref(m)) return NULL;
Continuing with the next part of the block in rte_pktmbuf_prefree_seg():
/**
* @internal Handle the packet mbufs with attached pinned external buffer
* on the mbuf freeing:
*
* - return zero if reference counter in shinfo is one. It means there is * no more reference to this pinned buffer and mbuf can be returned to * the pool
*
* - otherwise (if reference counter is not one), decrement reference
* counter and return non-zero value to prevent freeing the backing mbuf. *
* Returns non zero if mbuf should not be freed.
*/
static inline int __rte_pktmbuf_pinned_extbuf_decref(struct rte_mbuf *m)
{
struct rte_mbuf_ext_shared_info *shinfo;
/* Clear flags, mbuf is being freed. */
m->ol_flags = RTE_MBUF_F_EXTERNAL;
shinfo = m->shinfo;
/* Optimize for performance - do not dec/reinit */
if (likely(rte_mbuf_ext_refcnt_read(shinfo) == 1))
return 0;
/*
* Direct usage of add primitive to avoid
* duplication of comparing with one.
*/
if (likely(rte_atomic_fetch_add_explicit(&shinfo->refcnt, -1,
rte_memory_order_acq_rel) - 1))
return 1;
/* Reinitialize counter before mbuf freeing. */
rte_mbuf_ext_refcnt_set(shinfo, 1);
return 0;
}Essentially, if the mbuf does use a pinned external buffer, rte_pktmbuf_prefree_seg() only succeeds if that pinned external buffer is only referred to by the mbuf.
REQUIREMENT 4: If the mbuf uses a pinned external buffer, the mbuf must hold the only reference to that pinned external buffer, i.e. in that case, 'm->shinfo->refcnt' must be 1.
Please review.
If I'm not mistaken, the mbuf library is not a barrier for fast-freeing segmented packet mbufs, and thus fast-free of jumbo frames is possible.
We need a driver developer to confirm that my suggested approach - resetting the mbuf fields, incl. 'm->nb_segs' and 'm->next', when preparing the Tx descriptor - is viable.
Quoted 246 lines · Morten Brørup
Executive Summary:
My analysis shows that the mbuf library is not a barrier for fast-freeing segmented packet mbufs, and thus fast-free of jumbo frames is possible.
Detailed Analysis:
The purpose of the mbuf fast-free Tx optimization is to reduce
rte_pktmbuf_free_seg()to something much simpler in the ethdev drivers, by eliminating the code path related to indirect mbufs. Optimally, we want to simplify the ethdev driver's function that frees the transmitted mbufs, so it can free them directly to their mempool without accessing the mbufs themselves.If the driver cannot access the mbuf itself, it cannot determine which mempool it belongs to.
We don't want the driver to access every mbuf being freed; but if all mbufs of a Tx queue belong to the same mempool, the driver can determine which mempool by looking into just one of the mbufs.REQUIREMENT 1: The mbufs of a Tx queue must come from the same mempool.
When an mbuf is freed to its mempool, some of the fields in the mbuf must be initialized.
So, for fast-free, this must be done by the driver's function that prepares the Tx descriptor.
This is a requirement to the driver, not a requirement to the application.Now, let's dig into the code for freeing an mbuf.
Note: For readability purposes, I'll cut out some code and comments unrelated to this topic.static
__rte_always_inlinevoidrte_pktmbuf_free_seg(struct rte_mbuf *m){ m = rte_pktmbuf_prefree_seg(m); if (likely(m != NULL)) rte_mbuf_raw_free(m); }
rte_mbuf_raw_free(m)is simple, so nothing to gain there:/**
* Put mbuf back into its original mempool.
*
* The caller must ensure that the mbuf is direct and properly
* reinitialized (refcnt=1, next=NULL, nb_segs=1), as done by
*rte_pktmbuf_prefree_seg().
*/
static__rte_always_inlinevoidrte_mbuf_raw_free(struct rte_mbuf *m){ rte_mbuf_history_mark(m, RTE_MBUF_HISTORY_OP_LIB_FREE); rte_mempool_put(m->pool, m); }Note that the description says that the mbuf must be direct. This is not entirely accurate; the mbuf is allowed to use a pinned external buffer, if the mbuf holds the only reference to it. (Most of the mbuf library functions have this documentation inaccuracy, which should be fixed some day.)
So, the fast-free optimization really comes down to
rte_pktmbuf_prefree_seg(m), which must not return NULL.Let's dig into that.
/**
* Decrease reference counter and unlink a mbuf segment *
* This function does the same than a free, except that it does not
* return the segment to its pool.
* It decreases the reference counter, and if it reaches 0, it is
* detached from its parent for an indirect mbuf.
*
* @return
* - (m) if it is the last reference. It can be recycled or freed.
* - (NULL) if the mbuf still has remaining references on it.
*/
static__rte_always_inlinestructrte_mbuf*rte_pktmbuf_prefree_seg(struct rte_mbuf *m){ bool refcnt_not_one; refcnt_not_one = unlikely(rte_mbuf_refcnt_read(m) != 1); if (refcnt_not_one && __rte_mbuf_refcnt_update(m, -1) != 0) return NULL; if (unlikely(!RTE_MBUF_DIRECT(m))) { rte_pktmbuf_detach(m); if (RTE_MBUF_HAS_EXTBUF(m) && RTE_MBUF_HAS_PINNED_EXTBUF(m) && __rte_pktmbuf_pinned_extbuf_decref(m)) return NULL; } if (refcnt_not_one) rte_mbuf_refcnt_set(m, 1); if (m->nb_segs != 1) m->nb_segs = 1; if (m->next != NULL) m->next = NULL; return m; }This function can only succeed (i.e. return non-NULL) when 'refcnt' is 1 (or reaches 0).
REQUIREMENT 2: The driver must hold the only reference to the mbuf, i.e. '
m->refcnt' must be 1.When the function succeeds, it initializes the mbuf fields as required by
rte_mbuf_raw_free()before returning.Now, since the driver has exclusive access to the mbuf, it is free to initialize the '
m->next' and 'm->nb_segs' at any time.
It could do that when preparing the Tx descriptor.This is very interesting, because it means that fast-free does not prohibit segmented packets!
(But the driver must have sufficient Tx descriptors for all segments in the mbuf.)Now, lets dig into rte_pktmbuf_prefree_seg()'s block handling non-direct mbufs, i.e. cloned mbufs and mbufs with external buffer:
if (unlikely(!RTE_MBUF_DIRECT(m))) { rte_pktmbuf_detach(m); if (RTE_MBUF_HAS_EXTBUF(m) && RTE_MBUF_HAS_PINNED_EXTBUF(m) && __rte_pktmbuf_pinned_extbuf_decref(m)) return NULL; }Starting with
rte_pktmbuf_detach():static inline void
rte_pktmbuf_detach(struct rte_mbuf *m){ struct rte_mempool *mp = m->pool; uint32_t mbuf_size, buf_len; uint16_t priv_size; if (RTE_MBUF_HAS_EXTBUF(m)) { /* * The mbuf has the external attached buffer, * we should check the type of the memory pool where * the mbuf was allocated from to detect the pinned * external buffer. */ uint32_t flags = rte_pktmbuf_priv_flags(mp); if (flags & RTE_PKTMBUF_POOL_F_PINNED_EXT_BUF) { /* * The pinned external buffer should not be * detached from its backing mbuf, just exit. */ return; } __rte_pktmbuf_free_extbuf(m); } else { __rte_pktmbuf_free_direct(m); } priv_size = rte_pktmbuf_priv_size(mp); mbuf_size = (uint32_t)(sizeof(struct rte_mbuf) + priv_size); buf_len = rte_pktmbuf_data_room_size(mp); m->priv_size = priv_size; m->buf_addr = (char *)m + mbuf_size; rte_mbuf_iova_set(m, rte_mempool_virt2iova(m) + mbuf_size); m->buf_len = (uint16_t)buf_len; rte_pktmbuf_reset_headroom(m); m->data_len = 0; m->ol_flags = 0; }The only quick and simple code path through this function is when the mbuf uses a pinned external buffer:
if (RTE_MBUF_HAS_EXTBUF(m)) { uint32_t flags = rte_pktmbuf_priv_flags(mp); if (flags & RTE_PKTMBUF_POOL_F_PINNED_EXT_BUF) return;REQUIREMENT 3: The mbuf must not be cloned or use a non-pinned external buffer.
Continuing with the next part of rte_pktmbuf_prefree_seg()'s block:
if (RTE_MBUF_HAS_EXTBUF(m)&&RTE_MBUF_HAS_PINNED_EXTBUF(m) && __rte_pktmbuf_pinned_extbuf_decref(m)) return NULL;Continuing with the next part of the block in
rte_pktmbuf_prefree_seg():/**
* @internal Handle the packet mbufs with attached pinned external buffer
* on the mbuf freeing:
** - return zero if reference counter in shinfo is one. It means there is * no more reference to this pinned buffer and mbuf can be returned to * the pool*
* - otherwise (if reference counter is not one), decrement reference
* counter and return non-zero value to prevent freeing the backing mbuf. *
* Returns non zero if mbuf should not be freed.
*/
static inline int__rte_pktmbuf_pinned_extbuf_decref(struct rte_mbuf *m){ struct rte_mbuf_ext_shared_info *shinfo; /* Clear flags, mbuf is being freed. */ m->ol_flags = RTE_MBUF_F_EXTERNAL; shinfo = m->shinfo; /* Optimize for performance - do not dec/reinit */ if (likely(rte_mbuf_ext_refcnt_read(shinfo) == 1)) return 0; /* * Direct usage of add primitive to avoid * duplication of comparing with one. */ if (likely(rte_atomic_fetch_add_explicit(&shinfo->refcnt, -1, rte_memory_order_acq_rel) - 1)) return 1; /* Reinitialize counter before mbuf freeing. */ rte_mbuf_ext_refcnt_set(shinfo, 1); return 0; }Essentially, if the mbuf does use a pinned external buffer,
rte_pktmbuf_prefree_seg()only succeeds if that pinned external buffer is only referred to by the mbuf.REQUIREMENT 4: If the mbuf uses a pinned external buffer, the mbuf must hold the only reference to that pinned external buffer, i.e. in that case, '
m->shinfo->refcnt' must be 1.
Quoted 9 lines
Please review.
If I'm not mistaken, the mbuf library is not a barrier for fast-freeing segmented packet mbufs, and thus fast-free of jumbo frames is possible.
We need a driver developer to confirm that my suggested approach - resetting the mbuf fields, incl. '
m->nb_segs' and 'm->next', when preparing the Tx descriptor - is viable.
Excellent analysis, Morten. If I get a chance some time this release cycle, I will try implementing this change in our drivers, see if any difference is made.
/Bruce
Quoted 9 lines
Quoted 1 line
If I'm not mistaken, the mbuf library is not a barrier for fast-
freeing
Quoted 1 line
segmented packet mbufs, and thus fast-free of jumbo frames is
possible.
Quoted 5 lines
We need a driver developer to confirm that my suggested approach - resetting the mbuf fields, incl. '
m->nb_segs' and 'm->next', when preparing the Tx descriptor - is viable.
Quoted 5 lines
Excellent analysis, Morten. If I get a chance some time this release cycle,
I will try implementing this change in our drivers, see if any difference
is made.
Bruce,
Have you had a chance to look into the driver change requirements? If not, could you please try scratching the surface, to build a gut feeling.
I wonder if the vector implementations have strong requirements that packets are not segmented...
The i40 driver only sets "tx_simple_allowed" and "tx_vec_allowed" flags when MBUF_FAST_FREE is set:
https://elixir.bootlin.com/dpdk/v25.11/source/drivers/net/intel/i40e/i40e_rxtx.c#L3502
And only when these two flags are set, it uses a vector Tx function:
https://elixir.bootlin.com/dpdk/v25.11/source/drivers/net/intel/i40e/i40e_rxtx.c#L3550
And a special Tx Prep function:
https://elixir.bootlin.com/dpdk/v25.11/source/drivers/net/intel/i40e/i40e_rxtx.c#L3584
Which fails if nb_segs != 1:
https://elixir.bootlin.com/dpdk/v25.11/source/drivers/net/intel/i40e/i40e_rxtx.c#L1675
So currently it does.
But does it need to?... That is the question.
Paraphrasing:
Can the Tx function only be vectorized when the code path doesn't have branches depending on the number of segments? If so, then this may be the main reason for not supporting segmented packets with FAST_FREE.
In that case, we cannot remove the single-segment requirement from FAST_FREE without sacrificing the performance boost from vectorizing.
But then we can proceed pursuing alternative optimizations, as suggested by Konstantin.
Here's another idea:
The Tx function could pre-scan each Tx burst for multi-segment packets, to decide if the burst should be processed by the vector code path or a fallback code path (which can also handle multi-segment packets).
-Morten
Quoted 15 lines · Morten Brørup
Quoted 9 lines
Quoted 1 line
If I'm not mistaken, the mbuf library is not a barrier for fast-
freeing
Quoted 1 line
segmented packet mbufs, and thus fast-free of jumbo frames is
possible.
Quoted 5 lines
We need a driver developer to confirm that my suggested approach - resetting the mbuf fields, incl. '
m->nb_segs' and 'm->next', when preparing the Tx descriptor - is viable.Quoted 5 lines
Excellent analysis, Morten. If I get a chance some time this release cycle,
I will try implementing this change in our drivers, see if any difference
is made.
Quoted 4 lines
Bruce,
Have you had a chance to look into the driver change requirements? If not, could you please try scratching the surface, to build a gut feeling.
I'll try and take a look this week. Juggling a few things at the moment, so I had forgotten about this. Sorry.
More comments inline below.
/Bruce
Quoted 6 lines
I wonder if the vector implementations have strong requirements that packets are not segmented...
The i40 driver only sets "
tx_simple_allowed" and "tx_vec_allowed" flags whenMBUF_FAST_FREEis set:
https://elixir.bootlin.com/dpdk/v25.11/source/drivers/net/intel/i40e/i40e_rxtx.c#L3502
Actually, it allows but does not require FAST_FREE. The check is just verifying that the flags with everything *but* FAST_FREE masked out is the same as the original flags, i.e. FAST_FREE is just ignored.
Quoted 7 lines
And only when these two flags are set, it uses a vector Tx function:
https://elixir.bootlin.com/dpdk/v25.11/source/drivers/net/intel/i40e/i40e_rxtx.c#L3550
And a special Tx Prep function:
https://elixir.bootlin.com/dpdk/v25.11/source/drivers/net/intel/i40e/i40e_rxtx.c#L3584
Which fails ifnb_segs!= 1:
https://elixir.bootlin.com/dpdk/v25.11/source/drivers/net/intel/i40e/i40e_rxtx.c#L1675
Quoted 7 lines
So currently it does.
But does it need to?... That is the question.
Paraphrasing:
Can the Tx function only be vectorized when the code path doesn't have branches depending on the number of segments? If so, then this may be the main reason for not supporting segmented packets withFAST_FREE.In that case, we cannot remove the single-segment requirement from
FAST_FREEwithout sacrificing the performance boost from vectorizing.
No, based on what I state above, this should not be a blocker. The vector paths do require us to guarantee only one segment per packet - without additional context descriptors - so only one descriptor per packet (generally, or always one + ctx, in one code-path case). FAST_FREE can be used in conjunction with that but should not be a requirement. See [1] where in vector cleanup we explicitly check for FAST_FREE.
Similarly for scalar code path, in my latest rework, I am attempting to standardize the use of FAST_FREE optimizations even when we have a slightly slower Tx path [2].
[1] https://github.com/DPDK/dpdk/blob/main/drivers/net/intel/common/tx.h
[2] https://patches.dpdk.org/project/dpdk/patch/[email protected]/
Quoted 9 lines
But then we can proceed pursuing alternative optimizations, as suggested by Konstantin.
Here's another idea:
The Tx function could pre-scan each Tx burst for multi-segment packets, to decide if the burst should be processed by the vector code path or a fallback code path (which can also handle multi-segment packets).-Morten
Quoted 42 lines
From: Bruce Richardson [mailto:[email protected]]
Sent: Wednesday, 14 January 2026 17.36Quoted 6 lines · Morten Brørup
Quoted 6 lines
Quoted 1 line
If I'm not mistaken, the mbuf library is not a barrier for fast-
freeing
Quoted 1 line
segmented packet mbufs, and thus fast-free of jumbo frames is
possible.
Quoted 2 lines
We need a driver developer to confirm that my suggested approach
-
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
resetting the mbuf fields, incl. '
m->nb_segs' and 'm->next', when preparing the Tx descriptor - is viable.Excellent analysis, Morten. If I get a chance some time this
release
Quoted 9 lines
Quoted 4 lines
cycle,
I will try implementing this change in our drivers, see if any difference
is made.Bruce,
Have you had a chance to look into the driver change requirements? If not, could you please try scratching the surface, to build a gut
feeling.
I'll try and take a look this week. Juggling a few things at the moment, so
I had forgotten about this. Sorry.More comments inline below.
/Bruce
Quoted 2 lines
I wonder if the vector implementations have strong requirements that
packets are not segmented...
Quoted 2 lines
The i40 driver only sets "
tx_simple_allowed" and "tx_vec_allowed"flags when
MBUF_FAST_FREEis set:Quoted 1 line
Quoted 8 lines
https://elixir.bootlin.com/dpdk/v25.11/source/drivers/net/intel/i40e/i4
0e_rxtx.c#L3502Quoted 1 line
Actually, it allows but does not require
FAST_FREE. The check is just verifying that the flags with everything *but*FAST_FREEmasked out is the
same as the original flags, i.e.FAST_FREEis just ignored.
That's not how I read the code:ad->tx_simple_allowed =
(txq->offloads == (txq->offloads & RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE) && txq->tx_rs_thresh >= I40E_TX_MAX_BURST);
Look at it with offloads=(MULTI_SEGS|FAST_FREE):simple_allowed = (MULTI_SEGS|FAST_FREE) == (MULTI_SEGS|FAST_FREE) & FAST_FREE i.e.:simple_allowed = (MULTI_SEGS|FAST_FREE) == FAST_FREE
i.e.: false
Quoted 34 lines
Quoted 3 lines
And only when these two flags are set, it uses a vector Tx function:
https://elixir.bootlin.com/dpdk/v25.11/source/drivers/net/intel/i40e/i4
0e_rxtx.c#L3550Quoted 2 lines
And a special Tx Prep function:
https://elixir.bootlin.com/dpdk/v25.11/source/drivers/net/intel/i40e/i4
0e_rxtx.c#L3584Quoted 2 lines
Which fails if
nb_segs!= 1:https://elixir.bootlin.com/dpdk/v25.11/source/drivers/net/intel/i40e/i4
0e_rxtx.c#L1675Quoted 5 lines
So currently it does.
But does it need to?... That is the question. Paraphrasing:
Can the Tx function only be vectorized when the code path doesn'thave branches depending on the number of segments?
Quoted 1 line
If so, then this may be the main reason for not supporting segmented
packets with
FAST_FREE.Quoted 2 lines
In that case, we cannot remove the single-segment requirement from
FAST_FREEwithout sacrificing the performance boost from vectorizing.No, based on what I state above, this should not be a blocker. The vector
paths do require us to guarantee only one segment per packet - without additional context descriptors - so only one descriptor per packet (generally, or always one + ctx, in one code-path case).FAST_FREEcan be
used in conjunction with that but should not be a requirement. See [1] where in vector cleanup we explicitly check forFAST_FREE.
Quoted 4 lines
Similarly for scalar code path, in my latest rework, I am attempting to standardize the use of
FAST_FREEoptimizations even when we have a slightly
slower Tx path [2].
Good point:
The Tx path has two steps:
1) Pre-transmission Tx descriptor setup.
2) Post-transmission mbuf free.
FAST_FREE requirements for optimizing each of these two steps might differ.
As suggested in my other email, hopefully the post-transmission step can be vectorized (also for multi-segment packets) by assisting it in the pre-transmission step - i.e. by preparing the FAST_FREE segments for direct release to the mempool.
Then we can consider single-segment requirements for the pre-transmission step.
Quoted 19 lines
[1]
https://github.com/DPDK/dpdk/blob/main/drivers/net/intel/common/tx.h
[2] https://patches.dpdk.org/project/dpdk/patch/20260113151505.1871271-
[email protected]/Quoted 2 lines
But then we can proceed pursuing alternative optimizations, as
suggested by Konstantin.
Quoted 3 lines
Here's another idea:
The Tx function could pre-scan each Tx burst for multi-segmentpackets, to decide if the burst should be processed by the vector code path or a fallback code path (which can also handle multi-segment packets).
Quoted 4 lines
-Morten
Quoted 57 lines · Morten Brørup
Quoted 42 lines
From: Bruce Richardson [mailto:[email protected]]
Sent: Wednesday, 14 January 2026 17.36Quoted 6 lines · Morten Brørup
Quoted 6 lines
Quoted 1 line
If I'm not mistaken, the mbuf library is not a barrier for fast-
freeing
Quoted 1 line
segmented packet mbufs, and thus fast-free of jumbo frames is
possible.
Quoted 2 lines
We need a driver developer to confirm that my suggested approach
-
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
resetting the mbuf fields, incl. '
m->nb_segs' and 'm->next', when preparing the Tx descriptor - is viable.Excellent analysis, Morten. If I get a chance some time this
release
Quoted 9 lines
Quoted 4 lines
cycle,
I will try implementing this change in our drivers, see if any difference
is made.Bruce,
Have you had a chance to look into the driver change requirements? If not, could you please try scratching the surface, to build a gut
feeling.
I'll try and take a look this week. Juggling a few things at the moment, so
I had forgotten about this. Sorry.More comments inline below.
/Bruce
Quoted 2 lines
I wonder if the vector implementations have strong requirements that
packets are not segmented...
Quoted 2 lines
The i40 driver only sets "
tx_simple_allowed" and "tx_vec_allowed"flags when
MBUF_FAST_FREEis set:Quoted 1 line
Quoted 8 lines
https://elixir.bootlin.com/dpdk/v25.11/source/drivers/net/intel/i40e/i4
0e_rxtx.c#L3502Quoted 1 line
Actually, it allows but does not require
FAST_FREE. The check is just verifying that the flags with everything *but*FAST_FREEmasked out is the
same as the original flags, i.e.FAST_FREEis just ignored.That's not how I read the code:
ad->tx_simple_allowed=(txq->offloads == (txq->offloads & RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE) && txq->tx_rs_thresh >= I40E_TX_MAX_BURST);
Quoted 6 lines
Look at it with offloads=(MULTI_SEGS|FAST_FREE):
simple_allowed= (MULTI_SEGS|FAST_FREE) == (MULTI_SEGS|FAST_FREE) &FAST_FREEi.e.:simple_allowed= (MULTI_SEGS|FAST_FREE) ==FAST_FREE
i.e.: false
Which is correct. The only flag allowed is FAST_FREE, but its not required. If the input flags were just MULTI_SEGS, it would end up as:
simple_allowed = (MULTI_SEGS) == 0
i.e. also false
So the FAST_FREE flag does not affect the result.
/Bruce
Quoted 33 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
I wonder if the vector implementations have strong requirements
that
Quoted 6 lines
Quoted 6 lines
packets are not segmented...
Quoted 2 lines
The i40 driver only sets "
tx_simple_allowed" and "tx_vec_allowed"flags when
MBUF_FAST_FREEis set:Quoted 1 line
https://elixir.bootlin.com/dpdk/v25.11/source/drivers/net/intel/i40e/i4
Quoted 4 lines
Quoted 4 lines
0e_rxtx.c#L3502
Quoted 1 line
Actually, it allows but does not require
FAST_FREE. The check isjust
Quoted 1 line
Quoted 1 line
verifying that the flags with everything *but*
FAST_FREEmasked outis
Quoted 11 lines
Quoted 2 lines
the
same as the original flags, i.e.FAST_FREEis just ignored.That's not how I read the code:
ad->tx_simple_allowed=(txq->offloads == (txq->offloads & RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE) && txq->tx_rs_thresh >= I40E_TX_MAX_BURST);Look at it with offloads=(MULTI_SEGS|FAST_FREE):
simple_allowed= (MULTI_SEGS|FAST_FREE) == (MULTI_SEGS|FAST_FREE) &
FAST_FREEQuoted 4 lines
i.e.:
simple_allowed= (MULTI_SEGS|FAST_FREE) ==FAST_FREEi.e.: false
Quoted 8 lines
Which is correct. The only flag allowed is
FAST_FREE, but its not required.
If the input flags were justMULTI_SEGS, it would end up as:
simple_allowed= (MULTI_SEGS) == 0
i.e. also falseSo the
FAST_FREEflag does not affect the result.
OK, now I get it.
It's an obfuscated way of writing:simple_allowed = (offloads & ~RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE) == 0
Suggest updating for readability next time that code is touched.
The comment "only fast free is allowed" [1] didn't help me either; I interpreted it as "it's only allowed when fast free is set". Now that I understand the code, I can also interpret the comment as intended.
[1]: https://elixir.bootlin.com/dpdk/v25.11/source/drivers/net/intel/i40e/i40e_rxtx.c#L3502
Thanks for clarifying, Bruce!
PATCH 1/1 net/intel: optimize for fast-free hint
In reply to Morten BrørupWhen the fast-free hint is provided to the driver we know that the mbufs have refcnt of 1 and are from the same mempool. Therefore, we can optimize a bit for this case by:
* resetting the necessary mbuf fields, ie. nb_seg and next pointer when we are accessing the mbuf on writing the descriptor.
* on cleanup of buffers after transmit, we can just write those buffers straight to the mempool without accessing them.
--- drivers/net/intel/common/tx.h | 16 +++-- drivers/net/intel/common/tx_scalar_fns.h | 81 +++++++++++++++++++----- 2 files changed, 76 insertions(+), 21 deletions(-)
drivers/net/intel/common/tx.h +12 -4
@@ -345,12 +345,20 @@ ci_txq_release_all_mbufs(struct ci_tx_queue *txq, bool use_ctx)return ;if (!txq->vector_tx) { -for (uint16_t i =0 ; i < txq->nb_tx_desc; i++) { -if (txq->sw_ring[i].mbuf !=NULL ) { +/* Free mbufs from (last_desc_cleaned + 1) to (tx_tail - 1). */ +const uint16_t start = (txq->last_desc_cleaned +1 ) % txq->nb_tx_desc; +const uint16_t nb_desc = txq->nb_tx_desc; +const uint16_t end = txq->tx_tail; + +uint16_t i = start; +if (end < i) { +for (; i < nb_desc; i++)rte_pktmbuf_free_seg (txq->sw_ring[i].mbuf); - txq->sw_ring[i].mbuf =NULL ; - } + i =0 ; } +for (; i < end; i++) +rte_pktmbuf_free_seg (txq->sw_ring[i].mbuf); +memset (txq->sw_ring,0 ,sizeof (txq->sw_ring[0 ]) * nb_desc);return ; }
drivers/net/intel/common/tx_scalar_fns.h +64 -17
132 lines not on this page · raw message
Signature · 1 line
2.51.0
RE: [PATCH] net/intel: optimize for fast-free hint
I haven't looked into the details yet, but have a quick question inline below.
drivers/net/intel/common/tx.h ci_txq_release_all_mbufs(struct ci_tx_queue *txq, · line 350
Quoted 7 lines
drivers/net/intel/common/tx.h +0
-2@@ -345,12 +345,20 @@ ci_txq_release_all_mbufs(struct ci_tx_queue *txq, bool use_ctx) return;if (!txq->vector_tx) { -for (uint16_t i =0 ; i < txq->nb_tx_desc; i++) { -if (txq->sw_ring[i].mbuf !=NULL ) {
You changed this loop to only operate on not-yet-cleaned descriptors.
Here comes the first part of my question:
You removed the NULL check for txq->sw_ring[i].mbuf, thereby assuming that it is never NULL for not-yet-cleaned descriptors.
Quoted 60 lines
(quoted diff) +1
-0+ /* Free mbufs from (last_desc_cleaned + 1) to (tx_tail -1). */
(quoted diff) +1
-0+ const uint16_t start = (txq->last_desc_cleaned + 1) % txq-Quoted 1 line
nb_tx_desc;drivers/net/intel/common/tx.h +10
-2+const uint16_t nb_desc = txq->nb_tx_desc; +const uint16_t end = txq->tx_tail; + +uint16_t i = start; +if (end < i) { +for (; i < nb_desc; i++)rte_pktmbuf_free_seg (txq->sw_ring[i].mbuf); - txq->sw_ring[i].mbuf =NULL ; - } + i =0 ; } +for (; i < end; i++) +rte_pktmbuf_free_seg (txq->sw_ring[i].mbuf); +memset (txq->sw_ring,0 ,sizeof (txq->sw_ring[0 ]) * nb_desc);return ; }drivers/net/intel/common/tx_scalar_fns.h +2
-1@@ -30,16 +30,60 @@ ci_tx_xmit_cleanup(struct ci_tx_queue *txq)const uint16_t rs_idx = (last_desc_cleaned == nb_tx_desc -1 ) ?0 : (last_desc_cleaned +1 ) >> txq->log2_rs_thresh; -uint16_t desc_to_clean_to = (rs_idx << txq->log2_rs_thresh) + (txq->tx_rs_thresh -1 ); +const uint16_t dd_idx = txq->rs_last_id[rs_idx]; +const uint16_t first_to_clean = rs_idx << txq->log2_rs_thresh;/* Check if descriptor is done - all drivers use 0xF as done value in bits 3:0 */
drivers/net/intel/common/tx_scalar_fns.h +0
-1- if ((txd[txq->rs_last_id[rs_idx]].cmd_type_offset_bsz &
rte_cpu_to_le_64(CI_TXD_QW1_DTYPE_M)) !=drivers/net/intel/common/tx_scalar_fns.h +1
-0+ if ((txd[dd_idx].cmd_type_offset_bsz &
rte_cpu_to_le_64(CI_TXD_QW1_DTYPE_M)) !=drivers/net/intel/common/tx_scalar_fns.h +2
-0rte_cpu_to_le_64(CI_TX_DESC_DTYPE_DESC_DONE))/* Descriptor not yet processed by hardware */ return -1 ; +/* DD bit is set, descriptors are done. Now free the mbufs. */ +/* Note: nb_tx_desc is guaranteed to be a multiple of
tx_rs_thresh,drivers/net/intel/common/tx_scalar_fns.h +1
-0+* validated during queue setup. This means cleanup never wraps around
drivers/net/intel/common/tx_scalar_fns.h +1
-0+* the ring within a single burst (e.g., ring=256, rs_thresh=32 gives
drivers/net/intel/common/tx_scalar_fns.h +7
-0+* bursts of 0-31, 32-63, ..., 224-255). +*/ +const uint16_t nb_to_clean = txq->tx_rs_thresh; +struct ci_tx_entry *sw_ring = txq->sw_ring; + +if (txq->offloads &RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE ) { +/* FAST_FREE path: mbufs are already reset, just return to pool */
drivers/net/intel/common/tx_scalar_fns.h ci_tx_xmit_cleanup(struct ci_tx_queue *txq) · line 61
Quoted 9 lines
drivers/net/intel/common/tx_scalar_fns.h +9
-0+uint16_t nb_free =0 ; + +/* Get cached mempool pointer, or cache it on first use */ +struct rte_mempool *mp = +likely (txq->fast_free_mp != (void *)UINTPTR_MAX ) ? + txq->fast_free_mp : + (txq->fast_free_mp = sw_ring[dd_idx].mbuf->pool); + +/* Pack non-NULL mbufs in-place at start of sw_ring range.
Here is the second part of my question:
How can they (sw_ring[X].mbuf) be NULL here, when they cannot be NULL in ci_txq_release_all_mbufs()?
Quoted 36 lines
drivers/net/intel/common/tx_scalar_fns.h +1
-0+* No modulo needed in loop since we're guaranteed not to wrap.
drivers/net/intel/common/tx_scalar_fns.h +3
-0+*/ +for (uint16_t i =0 ; i < nb_to_clean; i++) { + struct rte_mbuf *m = sw_ring[first_to_clean +i].mbuf;
drivers/net/intel/common/tx_scalar_fns.h +8
-0+if (m !=NULL ) { +/* Pack into sw_ring at packed position */ + sw_ring[first_to_clean + nb_free].mbuf = m; + nb_free++; + } + } + +/* Bulk return to mempool using packed sw_ring entries directly */
drivers/net/intel/common/tx_scalar_fns.h +2
-0+if (nb_free >0 ) + rte_mempool_put_bulk(mp, (void**)&sw_ring[first_to_clean].mbuf,
nb_free);drivers/net/intel/common/tx_scalar_fns.h +2
-0+ }else { +/* Non-FAST_FREE path: use prefree_seg for refcount checks */
drivers/net/intel/common/tx_scalar_fns.h +2
-0+for (uint16_t i =0 ; i < nb_to_clean; i++) { + struct rte_mbuf *m = sw_ring[first_to_clean +i].mbuf;
drivers/net/intel/common/tx_scalar_fns.h +5
-0+if (m !=NULL ) +rte_pktmbuf_free_seg (m); + } + } +/* Update the txq to reflect the last descriptor that was cleaned */
drivers/net/intel/common/tx_scalar_fns.h +1
-1- txq->last_desc_cleaned = desc_to_clean_to; + txq->last_desc_cleaned = first_to_clean + txq->tx_rs_thresh -1 ; txq->nb_tx_free += txq->tx_rs_thresh;return 0 ;
Re: [PATCH] net/intel: optimize for fast-free hint
Quoted 6 lines · Morten Brørup
I haven't looked into the details yet, but have a quick question inline below.
Quoted 4 lines
(quoted diff) +0
-0@@ -345,12 +345,20 @@ ci_txq_release_all_mbufs(struct ci_tx_queue *txq, bool use_ctx) return;
Quoted 9 lines
drivers/net/intel/common/tx.h ci_txq_release_all_mbufs(struct ci_tx_queue *txq, bool use_ctx) · line 349
Quoted 3 lines
drivers/net/intel/common/tx.h +0
-2if (!txq->vector_tx) { -for (uint16_t i =0 ; i < txq->nb_tx_desc; i++) { -if (txq->sw_ring[i].mbuf !=NULL ) {You changed this loop to only operate on not-yet-cleaned descriptors.
Here comes the first part of my question:
You removed the NULL check for txq->sw_ring[i].mbuf, thereby assuming that it is never NULL for not-yet-cleaned descriptors.
Good point, I was quite focused on making this block and the vector block the same, I forgot that we can have NULL pointers for context descriptors. That was a silly mistake (and AI never caught it for me either.)
Quoted 64 lines
Quoted 64 lines
(quoted diff) +1
-0+ /* Free mbufs from (last_desc_cleaned + 1) to (tx_tail -1). */
(quoted diff) +1
-0+ const uint16_t start = (txq->last_desc_cleaned + 1) % txq-Quoted 1 line
nb_tx_desc;drivers/net/intel/common/tx.h +10
-2+const uint16_t nb_desc = txq->nb_tx_desc; +const uint16_t end = txq->tx_tail; + +uint16_t i = start; +if (end < i) { +for (; i < nb_desc; i++)rte_pktmbuf_free_seg (txq->sw_ring[i].mbuf); - txq->sw_ring[i].mbuf =NULL ; - } + i =0 ; } +for (; i < end; i++) +rte_pktmbuf_free_seg (txq->sw_ring[i].mbuf); +memset (txq->sw_ring,0 ,sizeof (txq->sw_ring[0 ]) * nb_desc);return ; }drivers/net/intel/common/tx_scalar_fns.h +2
-1@@ -30,16 +30,60 @@ ci_tx_xmit_cleanup(struct ci_tx_queue *txq)const uint16_t rs_idx = (last_desc_cleaned == nb_tx_desc -1 ) ?0 : (last_desc_cleaned +1 ) >> txq->log2_rs_thresh; -uint16_t desc_to_clean_to = (rs_idx << txq->log2_rs_thresh) + (txq->tx_rs_thresh -1 ); +const uint16_t dd_idx = txq->rs_last_id[rs_idx]; +const uint16_t first_to_clean = rs_idx << txq->log2_rs_thresh;/* Check if descriptor is done - all drivers use 0xF as done value in bits 3:0 */
drivers/net/intel/common/tx_scalar_fns.h +0
-1- if ((txd[txq->rs_last_id[rs_idx]].cmd_type_offset_bsz &
rte_cpu_to_le_64(CI_TXD_QW1_DTYPE_M)) !=drivers/net/intel/common/tx_scalar_fns.h +1
-0+ if ((txd[dd_idx].cmd_type_offset_bsz &
rte_cpu_to_le_64(CI_TXD_QW1_DTYPE_M)) !=drivers/net/intel/common/tx_scalar_fns.h +2
-0rte_cpu_to_le_64(CI_TX_DESC_DTYPE_DESC_DONE))/* Descriptor not yet processed by hardware */ return -1 ; +/* DD bit is set, descriptors are done. Now free the mbufs. */ +/* Note: nb_tx_desc is guaranteed to be a multiple of
tx_rs_thresh,drivers/net/intel/common/tx_scalar_fns.h +1
-0+* validated during queue setup. This means cleanup never wraps around
drivers/net/intel/common/tx_scalar_fns.h +1
-0+* the ring within a single burst (e.g., ring=256, rs_thresh=32 gives
drivers/net/intel/common/tx_scalar_fns.h +7
-0+* bursts of 0-31, 32-63, ..., 224-255). +*/ +const uint16_t nb_to_clean = txq->tx_rs_thresh; +struct ci_tx_entry *sw_ring = txq->sw_ring; + +if (txq->offloads &RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE ) { +/* FAST_FREE path: mbufs are already reset, just return to pool */
drivers/net/intel/common/tx_scalar_fns.h +4
-0+uint16_t nb_free =0 ; + +/* Get cached mempool pointer, or cache it on first use */ + struct rte_mempool *mp =
Quoted 8 lines
drivers/net/intel/common/tx_scalar_fns.h ci_tx_xmit_cleanup(struct ci_tx_queue *txq) · line 61
Quoted 5 lines
drivers/net/intel/common/tx_scalar_fns.h +5
-0+likely (txq->fast_free_mp != (void *)UINTPTR_MAX ) ? + txq->fast_free_mp : + (txq->fast_free_mp = sw_ring[dd_idx].mbuf->pool); + +/* Pack non-NULL mbufs in-place at start of sw_ring range. Here is the second part of my question:
How can they (sw_ring[X].mbuf) be NULL here, when they cannot be NULL inci_txq_release_all_mbufs()?
Because the latter function is wrong! :-)
Quoted 37 lines
Quoted 37 lines
drivers/net/intel/common/tx_scalar_fns.h +1
-0+* No modulo needed in loop since we're guaranteed not to wrap.
drivers/net/intel/common/tx_scalar_fns.h +3
-0+*/ +for (uint16_t i =0 ; i < nb_to_clean; i++) { + struct rte_mbuf *m = sw_ring[first_to_clean +i].mbuf;
drivers/net/intel/common/tx_scalar_fns.h +8
-0+if (m !=NULL ) { +/* Pack into sw_ring at packed position */ + sw_ring[first_to_clean + nb_free].mbuf = m; + nb_free++; + } + } + +/* Bulk return to mempool using packed sw_ring entries directly */
drivers/net/intel/common/tx_scalar_fns.h +2
-0+if (nb_free >0 ) + rte_mempool_put_bulk(mp, (void**)&sw_ring[first_to_clean].mbuf,
nb_free);drivers/net/intel/common/tx_scalar_fns.h +2
-0+ }else { +/* Non-FAST_FREE path: use prefree_seg for refcount checks */
drivers/net/intel/common/tx_scalar_fns.h +2
-0+for (uint16_t i =0 ; i < nb_to_clean; i++) { + struct rte_mbuf *m = sw_ring[first_to_clean +i].mbuf;
drivers/net/intel/common/tx_scalar_fns.h +5
-0+if (m !=NULL ) +rte_pktmbuf_free_seg (m); + } + } +/* Update the txq to reflect the last descriptor that was cleaned */
drivers/net/intel/common/tx_scalar_fns.h +1
-1- txq->last_desc_cleaned = desc_to_clean_to; + txq->last_desc_cleaned = first_to_clean + txq->tx_rs_thresh -1 ; txq->nb_tx_free += txq->tx_rs_thresh;return 0 ;
RE: [PATCH] net/intel: optimize for fast-free hint
Quoted 29 lines
From: Bruce Richardson [mailto:[email protected]]
Sent: Friday, 23 January 2026 13.09Quoted 1 line · Morten Brørup
I haven't looked into the details yet, but have a quick question
inline below.
Quoted 2 lines
Review ci_txq_release_all_mbufs(struct ci_tx_queue
Quoted 2 lines
(quoted diff) +0
-0@@ -345,12 +345,20 @@ ci_txq_release_all_mbufs(struct ci_tx_queue*txq,
Quoted 11 lines
drivers/net/intel/common/tx.h ci_txq_release_all_mbufs(struct ci_tx_queue *txq, bool use_ctx) · line 349
Quoted 6 lines
drivers/net/intel/common/tx.h +0
-2bool use_ctx) return;if (!txq->vector_tx) { -for (uint16_t i =0 ; i < txq->nb_tx_desc; i++) { -if (txq->sw_ring[i].mbuf !=NULL ) {You changed this loop to only operate on not-yet-cleaned descriptors.
Here comes the first part of my question:
You removed the NULL check for txq->sw_ring[i].mbuf, thereby assumingthat it is never NULL for not-yet-cleaned descriptors.
Quoted 1 line
Good point, I was quite focused on making this block and the vector block
the same, I forgot that we can have NULL pointers for context descriptors.
That was a silly mistake (and AI never caught it for me either.)
Quoted 8 lines
Quoted 8 lines
drivers/net/intel/common/tx.h ci_txq_release_all_mbufs(struct ci_tx_queue *txq, bool use_ctx) · line 353
Quoted 8 lines
(quoted diff) +1
-0+ /* Free mbufs from (last_desc_cleaned + 1) to (tx_tail -1). */
(quoted diff) +1
-0+ const uint16_t start = (txq->last_desc_cleaned + 1) % txq-Quoted 1 line
nb_tx_desc;drivers/net/intel/common/tx.h +4
-0+const uint16_t nb_desc = txq->nb_tx_desc; +const uint16_t end = txq->tx_tail; + +uint16_t i = start;
Suggest getting rid of "start"; it is only used for initializing "i".
Quoted 10 lines
Quoted 10 lines
drivers/net/intel/common/tx.h ci_txq_release_all_mbufs(struct ci_tx_queue *txq, bool use_ctx) · line 361
Quoted 10 lines
drivers/net/intel/common/tx.h +6
-2+if (end < i) { +for (; i < nb_desc; i++)rte_pktmbuf_free_seg (txq->sw_ring[i].mbuf); - txq->sw_ring[i].mbuf =NULL ; - } + i =0 ; } +for (; i < end; i++) +rte_pktmbuf_free_seg (txq->sw_ring[i].mbuf); +memset (txq->sw_ring,0 ,sizeof (txq->sw_ring[0 ]) * nb_desc);
Consider also splitting this memset() into two, one for each of the two for loops. Then you might need to keep "start" and make it non-const. :-)
Quoted 2 lines
Quoted 2 lines
Review
Quoted 2 lines
(quoted diff) +0
-0return; }
Or just keep the original version, looping over all descriptors.
Re: [PATCH] net/intel: optimize for fast-free hint
Quoted 32 lines · Morten Brørup
Quoted 32 lines
From: Bruce Richardson [mailto:[email protected]]
Sent: Friday, 23 January 2026 13.09Quoted 1 line · Morten Brørup
I haven't looked into the details yet, but have a quick question
inline below.
Quoted 2 lines
Review ci_txq_release_all_mbufs(struct ci_tx_queue
Quoted 2 lines
(quoted diff) +0
-0@@ -345,12 +345,20 @@ ci_txq_release_all_mbufs(struct ci_tx_queue*txq,
Quoted 11 lines
drivers/net/intel/common/tx.h ci_txq_release_all_mbufs(struct ci_tx_queue *txq, bool use_ctx) · line 349
Quoted 6 lines
drivers/net/intel/common/tx.h +0
-2bool use_ctx) return;if (!txq->vector_tx) { -for (uint16_t i =0 ; i < txq->nb_tx_desc; i++) { -if (txq->sw_ring[i].mbuf !=NULL ) {You changed this loop to only operate on not-yet-cleaned descriptors.
Here comes the first part of my question:
You removed the NULL check for txq->sw_ring[i].mbuf, thereby assumingthat it is never NULL for not-yet-cleaned descriptors.
Quoted 1 line
Good point, I was quite focused on making this block and the vector block
the same, I forgot that we can have NULL pointers for context descriptors.
That was a silly mistake (and AI never caught it for me either.)Quoted 3 lines
Quoted 3 lines
(quoted diff) +1
-0+ /* Free mbufs from (last_desc_cleaned + 1) to (tx_tail -1). */
(quoted diff) +1
-0+ const uint16_t start = (txq->last_desc_cleaned + 1) % txq-
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
drivers/net/intel/common/tx.h ci_txq_release_all_mbufs(struct ci_tx_queue *txq, bool use_ctx) · line 353
Quoted 5 lines
Quoted 1 line
nb_tx_desc;drivers/net/intel/common/tx.h +4
-0+const uint16_t nb_desc = txq->nb_tx_desc; +const uint16_t end = txq->tx_tail; + +uint16_t i = start;Suggest getting rid of "start"; it is only used for initializing "i".
Not sure it's worth doing. I quite like having an explicit start and end values for clarity.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
drivers/net/intel/common/tx.h +2
-2+if (end < i) { +for (; i < nb_desc; i++)rte_pktmbuf_free_seg (txq->sw_ring[i].mbuf); - txq->sw_ring[i].mbuf =NULL ; - }
Quoted 9 lines
Quoted 5 lines
Quoted 5 lines
drivers/net/intel/common/tx.h ci_txq_release_all_mbufs(struct ci_tx_queue *txq, bool use_ctx) · line 361
Quoted 5 lines
drivers/net/intel/common/tx.h +4
-0+ i =0 ; } +for (; i < end; i++) +rte_pktmbuf_free_seg (txq->sw_ring[i].mbuf); +memset (txq->sw_ring,0 ,sizeof (txq->sw_ring[0 ]) * nb_desc);Consider also splitting this
memset()into two, one for each of the two for loops. Then you might need to keep "start" and make it non-const. :-)
Don't see the point of that. The memset just zeros the whole array, ignoring wraparound so no point in doing two memsets when one will do.
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
return; }
Or just keep the original version, looping over all descriptors.
The reason for this whole change is that after the refactor the old code was wrong.
The original code used the fact that all mbuf pointers were zereod or overwritten after being freed, but that no longer applies, because we free the mbufs in bulk after we check the dd bits, rather than doing so individually later immediately before reuse. Instead, in both the datapath and this release path, we must use the index values to track what mbuf entries are valid or invalid. (We go from having two states, NULL or non-NULL, to 3; invalid i.e. freed or NULL, valid-NULL i.e. in slot used by context descriptor, valid-non-NULL i.e. a pointer to a not-yet-cleaned-up mbuf).
/Bruce
RE: [PATCH] net/intel: optimize for fast-free hint
Quoted 50 lines
From: Bruce Richardson [mailto:[email protected]]
Sent: Friday, 23 January 2026 13.54Quoted 8 lines · Morten Brørup
Quoted 8 lines
From: Bruce Richardson [mailto:[email protected]]
Sent: Friday, 23 January 2026 13.09Quoted 1 line · Morten Brørup
I haven't looked into the details yet, but have a quick question
inline below.
Quoted 2 lines
Review ci_txq_release_all_mbufs(struct
Quoted 2 lines
(quoted diff) +0
-0@@ -345,12 +345,20 @@ ci_txq_release_all_mbufs(struct
ci_tx_queueQuoted 9 lines
Quoted 9 lines
*txq,
Quoted 8 lines
drivers/net/intel/common/tx.h ci_txq_release_all_mbufs(struct ci_tx_queue *txq, bool use_ctx) · line 349
Quoted 6 lines
drivers/net/intel/common/tx.h +0
-2bool use_ctx) return;if (!txq->vector_tx) { -for (uint16_t i =0 ; i < txq->nb_tx_desc; i++) { -if (txq->sw_ring[i].mbuf !=NULL ) {You changed this loop to only operate on not-yet-cleaned
descriptors.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Here comes the first part of my question:
You removed the NULL check for txq->sw_ring[i].mbuf, therebyassuming
Quoted 10 lines
Quoted 10 lines
that it is never NULL for not-yet-cleaned descriptors.
Quoted 1 line
Good point, I was quite focused on making this block and the vector block
the same, I forgot that we can have NULL pointers for context descriptors.
That was a silly mistake (and AI never caught it for me either.)Quoted 1 line
Review ci_txq_release_all_mbufs(struct · line 349
Quoted 1 line
(quoted diff) +1
-0+ /* Free mbufs from (last_desc_cleaned + 1) to(
tx_tail-Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Review
Quoted 2 lines
1). */
(quoted diff) +1
-0+ const uint16_t start = (txq->last_desc_cleaned + 1) %txq-
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
drivers/net/intel/common/tx.h ci_txq_release_all_mbufs(struct ci_tx_queue *txq, bool use_ctx) · line 353
Quoted 5 lines
Quoted 1 line
nb_tx_desc;drivers/net/intel/common/tx.h +4
-0+const uint16_t nb_desc = txq->nb_tx_desc; +const uint16_t end = txq->tx_tail; + +uint16_t i = start;Suggest getting rid of "start"; it is only used for initializing "i".
Quoted 3 lines
Not sure it's worth doing. I quite like having an explicit start and end
values for clarity.
I have no preference, and it's a matter of taste, so the choice is yours. :-)
Quoted 31 lines
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines
drivers/net/intel/common/tx.h ci_txq_release_all_mbufs(struct ci_tx_queue *txq, bool use_ctx)
Quoted 4 lines
drivers/net/intel/common/tx.h +2
-0+if (end < i) { +for (; i < nb_desc; i++) rte_pktmbuf_free_seg(txq-sw_ring[i].mbuf);
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
drivers/net/intel/common/tx.h ci_txq_release_all_mbufs(struct ci_tx_queue *txq, bool use_ctx)
Quoted 7 lines
drivers/net/intel/common/tx.h +4
-2- txq->sw_ring[i].mbuf =NULL ; - } + i =0 ; } +for (; i < end; i++) +rte_pktmbuf_free_seg (txq->sw_ring[i].mbuf); + memset(txq->sw_ring, 0, sizeof(txq->sw_ring[0]) *
nb_desc);Quoted 2 lines
Consider also splitting this
memset()into two, one for each of thetwo for loops.
Quoted 2 lines
Then you might need to keep "start" and make it non-const. :-)
Don't see the point of that. The memset just zeros the whole array, ignoring wraparound so no point in doing two memsets when one will do.
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
return; }
Or just keep the original version, looping over all descriptors.
The reason for this whole change is that after the refactor the old code
was wrong.
Quoted 13 lines
The original code used the fact that all mbuf pointers were zereod or overwritten after being freed, but that no longer applies, because we free
the mbufs in bulk after we check the dd bits, rather than doing so individually later immediately before reuse. Instead, in both the datapath
and this release path, we must use the index values to track what mbuf entries are valid or invalid. (We go from having two states, NULL or non-NULL, to 3; invalid i.e. freed or NULL, valid-NULL i.e. in slot used by
context descriptor, valid-non-NULL i.e. a pointer to a not-yet-cleaned- up
mbuf).
Thank you for clarifying.
I thought of the two for loops as a kind of performance optimization, skipping the sub-array of already freed descriptors. In that case, memsetting only the two remaining sub-arrays might have been a good idea. That's not the case, so having one memset for the whole array is perfectly fine.
PATCH v2 1/1 net/intel: optimize for fast-free hint
In reply to Bruce RichardsonWhen the fast-free hint is provided to the driver we know that the mbufs have refcnt of 1 and are from the same mempool. Therefore, we can optimize a bit for this case by:
* resetting the necessary mbuf fields, ie. nb_seg and next pointer when we are accessing the mbuf on writing the descriptor.
* on cleanup of buffers after transmit, we can just write those buffers straight to the mempool without accessing them.
---
V2: Fix issues with original submission:
* missed check for NULL mbufs
* fixed issue with freeing directly from sw_ring in scalar path which doesn't work as thats not a flag array of pointers
* fixed missing null assignment in case of large segments for TSO
--- drivers/net/intel/common/tx.h | 21 ++++-- drivers/net/intel/common/tx_scalar.h | 95 ++++++++++++++++++++++------ 2 files changed, 90 insertions(+), 26 deletions(-)
drivers/net/intel/common/tx.h +15 -6
@@ -363,13 +363,22 @@ ci_txq_release_all_mbufs(struct ci_tx_queue *txq, bool use_ctx)return ;if (!txq->use_vec_entry) { -/* Regular scalar path uses sw_ring with ci_tx_entry */ -for (uint16_t i =0 ; i < txq->nb_tx_desc; i++) { -if (txq->sw_ring[i].mbuf !=NULL ) { -rte_pktmbuf_free_seg (txq->sw_ring[i].mbuf); - txq->sw_ring[i].mbuf =NULL ; - } +/* Free mbufs from (last_desc_cleaned + 1) to (tx_tail - 1). */ +const uint16_t start = (txq->last_desc_cleaned +1 ) % txq->nb_tx_desc; +const uint16_t nb_desc = txq->nb_tx_desc; +const uint16_t end = txq->tx_tail; + +uint16_t i = start; +if (end < i) { +for (; i < nb_desc; i++) +if (txq->sw_ring[i].mbuf !=NULL ) +rte_pktmbuf_free_seg (txq->sw_ring[i].mbuf); + i =0 ; } +for (; i < end; i++) +if (txq->sw_ring[i].mbuf !=NULL ) +rte_pktmbuf_free_seg (txq->sw_ring[i].mbuf); +memset (txq->sw_ring,0 ,sizeof (txq->sw_ring[0 ]) * nb_desc);return ; }
drivers/net/intel/common/tx_scalar.h +75 -20
154 lines not on this page · raw message
Signature · 1 line
2.51.0
RE: [PATCH v2] net/intel: optimize for fast-free hint
Quoted 8 lines
From: Bruce Richardson [mailto:[email protected]]
Sent: Wednesday, 8 April 2026 15.25When the fast-free hint is provided to the driver we know that the mbufs
have refcnt of 1 and are from the same mempool. Therefore, we can optimize a bit for this case by:
Quoted 6 lines
* resetting the necessary mbuf fields, ie.
nb_segand next pointer when we are accessing the mbuf on writing the descriptor.
* on cleanup of buffers after transmit, we can just write those buffers straight to the mempool without accessing them.Signed-off-by Bruce Richardson <[email protected]>
A bunch of review thoughts inline below. The ones regarding instrumentation should be fixed. The rest might be irrelevant and/or nonsense.
Quoted 40 lines
---
V2: Fix issues with original submission:
* missed check for NULL mbufs
* fixed issue with freeing directly fromsw_ringin scalar path which doesn't work as thats not a flag array of pointers
* fixed missing null assignment in case of large segments for TSO--- drivers/net/intel/common/tx.h | 21 ++++-- drivers/net/intel/common/tx_scalar.h | 95 ++++++++++++++++++++++------ 2 files changed, 90 insertions(+), 26 deletions(-)drivers/net/intel/common/tx.h +1
-6@@ -363,13 +363,22 @@ ci_txq_release_all_mbufs(struct ci_tx_queue *txq, bool use_ctx) return;if (!txq->use_vec_entry) { -/* Regular scalar path uses sw_ring with ci_tx_entry */ -for (uint16_t i =0 ; i < txq->nb_tx_desc; i++) { -if (txq->sw_ring[i].mbuf !=NULL ) { -rte_pktmbuf_free_seg (txq->sw_ring[i].mbuf); - txq->sw_ring[i].mbuf =NULL ; - } +/* Free mbufs from (last_desc_cleaned + 1) to (tx_tail - 1). */
drivers/net/intel/common/tx.h +1
-0+ const uint16_t start = (txq->last_desc_cleaned + 1) % txq-Quoted 1 line
nb_tx_desc;drivers/net/intel/common/tx.h +8
-0+const uint16_t nb_desc = txq->nb_tx_desc; +const uint16_t end = txq->tx_tail; + +uint16_t i = start; +if (end < i) { +for (; i < nb_desc; i++) +if (txq->sw_ring[i].mbuf !=NULL ) + rte_pktmbuf_free_seg(txq-Quoted 1 line
sw_ring[i].mbuf);
drivers/net/intel/common/tx.h ci_txq_release_all_mbufs(struct ci_tx_queue *txq, bool use_ctx) · line 383
Quoted 8 lines
drivers/net/intel/common/tx.h +5
-0+ i =0 ; } +for (; i < end; i++) +if (txq->sw_ring[i].mbuf !=NULL ) +rte_pktmbuf_free_seg (txq->sw_ring[i].mbuf); +memset (txq->sw_ring,0 ,sizeof (txq->sw_ring[0 ]) * nb_desc);return ; }
The above LGTM.
IIRC, we already discussed it - or something very similar.
Quoted 41 lines
drivers/net/intel/common/tx_scalar.h +3
-5@@ -197,16 +197,63 @@ ci_tx_xmit_cleanup(struct ci_tx_queue *txq)const uint16_t rs_idx = (last_desc_cleaned == nb_tx_desc -1 ) ?0 : (last_desc_cleaned +1 ) >> txq->log2_rs_thresh; -uint16_t desc_to_clean_to = (rs_idx << txq->log2_rs_thresh) + (txq->tx_rs_thresh -1 ); +const uint16_t dd_idx = txq->rs_last_id[rs_idx]; +const uint16_t first_to_clean = rs_idx << txq->log2_rs_thresh; -/* Check if descriptor is done */ -if ((txd[txq->rs_last_id[rs_idx]].cmd_type_offset_bsz & -rte_cpu_to_le_64 (CI_TXD_QW1_DTYPE_M )) != -rte_cpu_to_le_64 (CI_TX_DESC_DTYPE_DESC_DONE )) +/* Check if descriptor is done - all drivers use 0xF as done value in bits 3:0 */
drivers/net/intel/common/tx_scalar.h +1
-0+ if ((txd[dd_idx].cmd_type_offset_bsz &
rte_cpu_to_le_64(CI_TXD_QW1_DTYPE_M)) !=drivers/net/intel/common/tx_scalar.h +4
-0+ rte_cpu_to_le_64(CI_TX_DESC_DTYPE_DESC_DONE)) +/* Descriptor not yet processed by hardware */ return -1 ; +/* DD bit is set, descriptors are done. Now free the mbufs. */ +/* Note: nb_tx_desc is guaranteed to be a multiple of
tx_rs_thresh,drivers/net/intel/common/tx_scalar.h +1
-0+* validated during queue setup. This means cleanup never wraps around
drivers/net/intel/common/tx_scalar.h +1
-0+* the ring within a single burst (e.g., ring=256, rs_thresh=32 gives
drivers/net/intel/common/tx_scalar.h +7
-0+* bursts of 0-31, 32-63, ..., 224-255). +*/ +const uint16_t nb_to_clean = txq->tx_rs_thresh; +struct ci_tx_entry *sw_ring = txq->sw_ring; + +if (txq->offloads &RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE ) { +/* FAST_FREE path: mbufs are already reset, just return to
Quoted 1 line
pool */
Depending on which cache lines from txq have already been loaded, unless txq->offloads is hot in the CPU cache and txq->fast_free_mp is not, consider testing (mp != NULL) instead of (txq->offloads & RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE). Like here:
https://elixir.bootlin.com/dpdk/v26.03/source/drivers/net/intel/common/tx.h#L281
Quoted 16 lines
drivers/net/intel/common/tx_scalar.h +11
-0+void *free[CI_TX_MAX_FREE_BUF_SZ ]; +uint16_t nb_free =0 ; + +/* Get cached mempool pointer, or cache it on first use */ +struct rte_mempool *mp = +likely (txq->fast_free_mp != (void *)UINTPTR_MAX ) ? + txq->fast_free_mp : + (txq->fast_free_mp = sw_ring[dd_idx].mbuf->pool); + +/* Pack non-NULL mbufs in-place at start of sw_ring range. +* No modulo needed in loop since we're guaranteed not to wrap.
drivers/net/intel/common/tx_scalar.h +3
-0+*/ +for (uint16_t i =0 ; i < nb_to_clean; i++) { + struct rte_mbuf *m = sw_ring[first_to_clean +i].mbuf;
drivers/net/intel/common/tx_scalar.h ci_tx_xmit_cleanup(struct ci_tx_queue *txq) · line 236
Quoted 3 lines
drivers/net/intel/common/tx_scalar.h +3
-0+if (m ==NULL ) +continue ; + free[nb_free++] = m;
Should sw_ring[first_to_clean + i].mbuf be set to NULL here, instead of in ci_xmit_pkts()? I don't know, just want you to consider it.
drivers/net/intel/common/tx_scalar.h
Quoted 2 lines
drivers/net/intel/common/tx_scalar.h +2
-0+if (unlikely (nb_free ==CI_TX_MAX_FREE_BUF_SZ )) { +rte_mempool_put_bulk (mp, free, nb_free);
rte_mempool_put_bulk() -> rte_mbuf_raw_free_bulk(), for instrumentation.
drivers/net/intel/common/tx_scalar.h ci_tx_xmit_cleanup(struct ci_tx_queue *txq) · line 245
Quoted 8 lines
drivers/net/intel/common/tx_scalar.h +5
-0+ nb_free =0 ; + } + } + +/* Bulk return to mempool using packed sw_ring entries directly */
drivers/net/intel/common/tx_scalar.h +2
-0+if (nb_free >0 ) +rte_mempool_put_bulk (mp, free, nb_free);
rte_mempool_put_bulk() -> rte_mbuf_raw_free_bulk(), for instrumentation.
drivers/net/intel/common/tx_scalar.h ci_tx_xmit_cleanup(struct ci_tx_queue *txq) · line 251
Quoted 8 lines
drivers/net/intel/common/tx_scalar.h +2
-0+ }else { +/* Non-FAST_FREE path: use prefree_seg for refcount checks */
drivers/net/intel/common/tx_scalar.h +2
-0+for (uint16_t i =0 ; i < nb_to_clean; i++) { + struct rte_mbuf *m = sw_ring[first_to_clean +i].mbuf;
drivers/net/intel/common/tx_scalar.h +2
-0+if (m !=NULL ) +rte_pktmbuf_free_seg (m);
Should sw_ring[first_to_clean + i].mbuf be set to NULL here, instead of in ci_xmit_pkts()? I don't know, just want you to consider it.
Quoted 20 lines
drivers/net/intel/common/tx_scalar.h +3
-0+ } + } +/* Update the txq to reflect the last descriptor that was cleaned */
drivers/net/intel/common/tx_scalar.h +1
-3- txq->last_desc_cleaned = desc_to_clean_to; + txq->last_desc_cleaned = first_to_clean + txq->tx_rs_thresh -1 ; txq->nb_tx_free += txq->tx_rs_thresh;return 0 ;@@ -450,8 +497,6 @@ ci_xmit_pkts(struct ci_tx_queue *txq, txd = &ci_tx_ring[tx_id]; tx_id = txe->next_id; -if (txe->mbuf) -rte_pktmbuf_free_seg (txe->mbuf); txe->mbuf = tx_pkt;/* Setup TX Descriptor */ td_cmd |=CI_TX_DESC_CMD_EOP ;
drivers/net/intel/common/tx_scalar.h ci_xmit_pkts(struct ci_tx_queue *txq, · line 519
Quoted 3 lines
drivers/net/intel/common/tx_scalar.h +0
-0@@ -472,10 +517,7 @@ ci_xmit_pkts(struct ci_tx_queue *txq, txn = &sw_ring[txe->next_id];RTE_MBUF_PREFETCH_TO_FREE (txn->mbuf);
RTE_MBUF_PREFETCH_TO_FREE() doesn't seem relevant here anymore. I don't know if it fits into ci_tx_xmit_cleanup() instead.
drivers/net/intel/common/tx_scalar.h ci_xmit_pkts(struct ci_tx_queue *txq, · line 520
Quoted 5 lines
drivers/net/intel/common/tx_scalar.h +1
-4-if (txe->mbuf) { -rte_pktmbuf_free_seg (txe->mbuf); - txe->mbuf =NULL ; - } + txe->mbuf =NULL ;
Already mentioned: Should txe->mbuf be set to NULL in ci_tx_xmit_cleanup() instead of in ci_tx_xmit_pkts()?
drivers/net/intel/common/tx_scalar.h ci_xmit_pkts(struct ci_tx_queue *txq, · line 533
Quoted 7 lines
drivers/net/intel/common/tx_scalar.h +0
-0write_txd (ctx_txd, cd_qw0, cd_qw1);@@ -489,10 +531,7 @@ ci_xmit_pkts(struct ci_tx_queue *txq, txn = &sw_ring[txe->next_id];RTE_MBUF_PREFETCH_TO_FREE (txn->mbuf);
RTE_MBUF_PREFETCH_TO_FREE() doesn't seem relevant here anymore. I don't know if it fits into ci_tx_xmit_cleanup() instead.
drivers/net/intel/common/tx_scalar.h ci_xmit_pkts(struct ci_tx_queue *txq, · line 520
Quoted 5 lines
drivers/net/intel/common/tx_scalar.h +1
-4-if (txe->mbuf) { -rte_pktmbuf_free_seg (txe->mbuf); - txe->mbuf =NULL ; - } + txe->mbuf =NULL ;
Already mentioned: Should txe->mbuf be set to NULL in ci_tx_xmit_cleanup() instead of in ci_tx_xmit_pkts()?
Quoted 22 lines
drivers/net/intel/common/tx_scalar.h +1
-2ipsec_txd[0 ] = ipsec_qw0; ipsec_txd[1 ] = ipsec_qw1;@@ -507,10 +546,21 @@ ci_xmit_pkts(struct ci_tx_queue *txq, txd = &ci_tx_ring[tx_id]; txn = &sw_ring[txe->next_id]; -if (txe->mbuf) -rte_pktmbuf_free_seg (txe->mbuf); txe->mbuf = m_seg; +/* For FAST_FREE: reset mbuf fields while we have it in cache.
drivers/net/intel/common/tx_scalar.h +1
-0+* FAST_FREE guarantees refcnt=1 and direct mbufs, so we only
drivers/net/intel/common/tx_scalar.h +1
-0+* need to reset nb_segs and next pointer as per
rte_pktmbuf_prefree_seg.drivers/net/intel/common/tx_scalar.h +1
-0+* Save next pointer before resetting since we need it for loop iteration.
drivers/net/intel/common/tx_scalar.h +3
-0+*/ +struct rte_mbuf *next_seg = m_seg->next; + if (txq->offloads &
Quoted 1 line
RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE) {
Similar to comment further above: Is txq->offloads or txq->fast_free_mp hotter in the CPU cache here?
Quoted 53 lines
drivers/net/intel/common/tx_scalar.h +6
-0+if (m_seg->nb_segs !=1 ) + m_seg->nb_segs =1 ; +if (next_seg !=NULL ) + m_seg->next =NULL ; + } +/* Setup TX Descriptor */ /* Calculate segment length, using IPsec callback if provided */
drivers/net/intel/common/tx_scalar.h +0
-0if (ipsec_ops !=NULL )@@ -528,18 +578,23 @@ ci_xmit_pkts(struct ci_tx_queue *txq, ((uint64_t)CI_MAX_DATA_PER_TXD <<
CI_TXD_QW1_TX_BUF_SZ_S) |
((uint64_t)td_tag <<CI_TXD_QW1_L2TAG1_S);
write_txd(txd,buf_dma_addr,cmd_type_offset_bsz);drivers/net/intel/common/tx_scalar.h +1
-0+/* txe for this slot has already been written (e.g. above outside
drivers/net/intel/common/tx_scalar.h +1
-0+* loop), so we write the extra NULL mbuf pointer for this
drivers/net/intel/common/tx_scalar.h +3
-0+* descriptor after we increment txe below. +*/ buf_dma_addr +=CI_MAX_DATA_PER_TXD ; slen -=CI_MAX_DATA_PER_TXD ; tx_id = txe->next_id; txe = txn; + txe->mbuf =NULL ; txd = &ci_tx_ring[tx_id]; txn = &sw_ring[txe->next_id]; }/* fill the last descriptor with End of Packet (EOP) bit */
drivers/net/intel/common/tx_scalar.h +1
-1-if (m_seg->next ==NULL ) +if (next_seg ==NULL ) td_cmd |=CI_TX_DESC_CMD_EOP ; const uint64_t cmd_type_offset_bsz =
CI_TX_DESC_DTYPE_DATA|drivers/net/intel/common/tx_scalar.h +1
-2@@ -551,7 +606,7 @@ ci_xmit_pkts(struct ci_tx_queue *txq, tx_id = txe->next_id; txe = txn; - m_seg = m_seg->next; + m_seg = next_seg; }while (m_seg); end_pkt: txq->nb_tx_free = (uint16_t )(txq->nb_tx_free - nb_used); --2.51.0
Re: [PATCH v2] net/intel: optimize for fast-free hint
Quoted 13 lines · Morten Brørup
Quoted 13 lines
From: Bruce Richardson [mailto:[email protected]]
Sent: Wednesday, 8 April 2026 15.25When the fast-free hint is provided to the driver we know that the mbufs
have refcnt of 1 and are from the same mempool. Therefore, we can optimize a bit for this case by:* resetting the necessary mbuf fields, ie.
nb_segand next pointer when we are accessing the mbuf on writing the descriptor.
* on cleanup of buffers after transmit, we can just write those buffers straight to the mempool without accessing them.
Quoted 6 lines
Quoted 1 line
Signed-off-by Bruce Richardson <[email protected]>A bunch of review thoughts inline below. The ones regarding instrumentation should be fixed. The rest might be irrelevant and/or nonsense.
See inline below. Summary:
* ack for mempool functions for instrumentation
* ack for referencing the mp pointer rather than the offload flags
* nak for setting the mbuf pointers for null, it's unnecessary with this design.
* ack for removing the prefetches
Thanks for the detailed review. Preparing v3 now.
/Bruce
Quoted 90 lines
Quoted 40 lines
---
V2: Fix issues with original submission:
* missed check for NULL mbufs
* fixed issue with freeing directly fromsw_ringin scalar path which doesn't work as thats not a flag array of pointers
* fixed missing null assignment in case of large segments for TSO--- drivers/net/intel/common/tx.h | 21 ++++-- drivers/net/intel/common/tx_scalar.h | 95 ++++++++++++++++++++++------ 2 files changed, 90 insertions(+), 26 deletions(-)drivers/net/intel/common/tx.h +1
-6@@ -363,13 +363,22 @@ ci_txq_release_all_mbufs(struct ci_tx_queue *txq, bool use_ctx) return;if (!txq->use_vec_entry) { -/* Regular scalar path uses sw_ring with ci_tx_entry */ -for (uint16_t i =0 ; i < txq->nb_tx_desc; i++) { -if (txq->sw_ring[i].mbuf !=NULL ) { -rte_pktmbuf_free_seg (txq->sw_ring[i].mbuf); - txq->sw_ring[i].mbuf =NULL ; - } +/* Free mbufs from (last_desc_cleaned + 1) to (tx_tail - 1). */
drivers/net/intel/common/tx.h +1
-0+ const uint16_t start = (txq->last_desc_cleaned + 1) % txq-Quoted 1 line
nb_tx_desc;drivers/net/intel/common/tx.h +8
-0+const uint16_t nb_desc = txq->nb_tx_desc; +const uint16_t end = txq->tx_tail; + +uint16_t i = start; +if (end < i) { +for (; i < nb_desc; i++) +if (txq->sw_ring[i].mbuf !=NULL ) + rte_pktmbuf_free_seg(txq-Quoted 1 line
sw_ring[i].mbuf);
drivers/net/intel/common/tx.h ci_txq_release_all_mbufs(struct ci_tx_queue *txq, bool use_ctx) · line 383
Quoted 8 lines
drivers/net/intel/common/tx.h +5
-0+ i =0 ; } +for (; i < end; i++) +if (txq->sw_ring[i].mbuf !=NULL ) +rte_pktmbuf_free_seg (txq->sw_ring[i].mbuf); +memset (txq->sw_ring,0 ,sizeof (txq->sw_ring[0 ]) * nb_desc);return ; }The above LGTM.
IIRC, we already discussed it - or something very similar.Quoted 38 lines
drivers/net/intel/common/tx_scalar.h +3
-5@@ -197,16 +197,63 @@ ci_tx_xmit_cleanup(struct ci_tx_queue *txq)const uint16_t rs_idx = (last_desc_cleaned == nb_tx_desc -1 ) ?0 : (last_desc_cleaned +1 ) >> txq->log2_rs_thresh; -uint16_t desc_to_clean_to = (rs_idx << txq->log2_rs_thresh) + (txq->tx_rs_thresh -1 ); +const uint16_t dd_idx = txq->rs_last_id[rs_idx]; +const uint16_t first_to_clean = rs_idx << txq->log2_rs_thresh; -/* Check if descriptor is done */ -if ((txd[txq->rs_last_id[rs_idx]].cmd_type_offset_bsz & -rte_cpu_to_le_64 (CI_TXD_QW1_DTYPE_M )) != -rte_cpu_to_le_64 (CI_TX_DESC_DTYPE_DESC_DONE )) +/* Check if descriptor is done - all drivers use 0xF as done value in bits 3:0 */
drivers/net/intel/common/tx_scalar.h +1
-0+ if ((txd[dd_idx].cmd_type_offset_bsz &
rte_cpu_to_le_64(CI_TXD_QW1_DTYPE_M)) !=drivers/net/intel/common/tx_scalar.h +4
-0+ rte_cpu_to_le_64(CI_TX_DESC_DTYPE_DESC_DONE)) +/* Descriptor not yet processed by hardware */ return -1 ; +/* DD bit is set, descriptors are done. Now free the mbufs. */ +/* Note: nb_tx_desc is guaranteed to be a multiple of
tx_rs_thresh,drivers/net/intel/common/tx_scalar.h +1
-0+* validated during queue setup. This means cleanup never wraps around
drivers/net/intel/common/tx_scalar.h +1
-0+* the ring within a single burst (e.g., ring=256, rs_thresh=32 gives
drivers/net/intel/common/tx_scalar.h +4
-0+* bursts of 0-31, 32-63, ..., 224-255). +*/ +const uint16_t nb_to_clean = txq->tx_rs_thresh; +struct ci_tx_entry *sw_ring = txq->sw_ring;
Quoted 9 lines
Quoted 4 lines
drivers/net/intel/common/tx_scalar.h +3
-0+ +if (txq->offloads &RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE ) { +/* FAST_FREE path: mbufs are already reset, just return to pool */
Depending on which cache lines from txq have already been loaded, unless
txq->offloadsis hot in the CPU cache andtxq->fast_free_mpis not, consider testing (mp != NULL) instead of (txq->offloads&RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE). Like here:
https://elixir.bootlin.com/dpdk/v26.03/source/drivers/net/intel/common/tx.h#L281
Ack. The offloads are on the second cacheline of the txq, I think, while the fast_free_mp is on the first one. Therefore, checking the mp pointer for null is slightly better.
Quoted 14 lines
Quoted 14 lines
drivers/net/intel/common/tx_scalar.h +11
-0+void *free[CI_TX_MAX_FREE_BUF_SZ ]; +uint16_t nb_free =0 ; + +/* Get cached mempool pointer, or cache it on first use */ +struct rte_mempool *mp = +likely (txq->fast_free_mp != (void *)UINTPTR_MAX ) ? + txq->fast_free_mp : + (txq->fast_free_mp = sw_ring[dd_idx].mbuf->pool); + +/* Pack non-NULL mbufs in-place at start of sw_ring range. +* No modulo needed in loop since we're guaranteed not to wrap.
drivers/net/intel/common/tx_scalar.h +2
-0+*/ +for (uint16_t i =0 ; i < nb_to_clean; i++) {
Quoted 8 lines
drivers/net/intel/common/tx_scalar.h ci_tx_xmit_cleanup(struct ci_tx_queue *txq) · line 236
Quoted 5 lines
drivers/net/intel/common/tx_scalar.h +1
-0+ struct rte_mbuf *m = sw_ring[first_to_clean +i].mbuf;
drivers/net/intel/common/tx_scalar.h +3
-0+if (m ==NULL ) +continue ; + free[nb_free++] = m;Should sw_ring[first_to_clean + i].mbuf be set to NULL here, instead of in
ci_xmit_pkts()? I don't know, just want you to consider it.
No, we are ok just setting it there. The changes in this patch to the ci_txq_release_all_mbufs() function mean that we assume all buffers between tx_tail and last_desc_cleaned are invalid/freed (i.e. we free from last-cleaned to tx_tail only), and so we don't need to set the invalid slots to NULL. The NULL sentinel is only used where we have a valid slot but which does not have an associated mbuf.
Quoted 7 lines
drivers/net/intel/common/tx_scalar.h
Quoted 3 lines
drivers/net/intel/common/tx_scalar.h +2
-0+if (unlikely (nb_free ==CI_TX_MAX_FREE_BUF_SZ )) { +rte_mempool_put_bulk (mp, free, nb_free);
rte_mempool_put_bulk()->rte_mbuf_raw_free_bulk(), for instrumentation.
Ack
Quoted 11 lines
drivers/net/intel/common/tx_scalar.h ci_tx_xmit_cleanup(struct ci_tx_queue *txq) · line 245
Quoted 8 lines
drivers/net/intel/common/tx_scalar.h +5
-0+ nb_free =0 ; + } + } + +/* Bulk return to mempool using packed sw_ring entries directly */
drivers/net/intel/common/tx_scalar.h +2
-0+if (nb_free >0 ) +rte_mempool_put_bulk (mp, free, nb_free);
rte_mempool_put_bulk()->rte_mbuf_raw_free_bulk(), for instrumentation.
Ack
Quoted 13 lines
drivers/net/intel/common/tx_scalar.h ci_tx_xmit_cleanup(struct ci_tx_queue *txq) · line 251
Quoted 9 lines
drivers/net/intel/common/tx_scalar.h +2
-0+ }else { +/* Non-FAST_FREE path: use prefree_seg for refcount checks */
drivers/net/intel/common/tx_scalar.h +2
-0+for (uint16_t i =0 ; i < nb_to_clean; i++) { + struct rte_mbuf *m = sw_ring[first_to_clean +i].mbuf;
drivers/net/intel/common/tx_scalar.h +2
-0+if (m !=NULL ) +rte_pktmbuf_free_seg (m);Should sw_ring[first_to_clean + i].mbuf be set to NULL here, instead of in
ci_xmit_pkts()? I don't know, just want you to consider it.
As above, reason should hold for both fast-free and non-fast-free paths.
Quoted 21 lines
Quoted 21 lines
drivers/net/intel/common/tx_scalar.h +3
-0+ } + } +/* Update the txq to reflect the last descriptor that was cleaned */
drivers/net/intel/common/tx_scalar.h +1
-3- txq->last_desc_cleaned = desc_to_clean_to; + txq->last_desc_cleaned = first_to_clean + txq->tx_rs_thresh -1 ; txq->nb_tx_free += txq->tx_rs_thresh;return 0 ;@@ -450,8 +497,6 @@ ci_xmit_pkts(struct ci_tx_queue *txq, txd = &ci_tx_ring[tx_id]; tx_id = txe->next_id; -if (txe->mbuf) -rte_pktmbuf_free_seg (txe->mbuf); txe->mbuf = tx_pkt;/* Setup TX Descriptor */ td_cmd |=CI_TX_DESC_CMD_EOP ;@@ -472,10 +517,7 @@ ci_xmit_pkts(struct ci_tx_queue *txq,
Quoted 5 lines
drivers/net/intel/common/tx_scalar.h ci_xmit_pkts(struct ci_tx_queue *txq, · line 519
Quoted 2 lines
drivers/net/intel/common/tx_scalar.h +0
-0txn = &sw_ring[txe->next_id];RTE_MBUF_PREFETCH_TO_FREE (txn->mbuf);
RTE_MBUF_PREFETCH_TO_FREE()doesn't seem relevant here anymore. I don't know if it fits intoci_tx_xmit_cleanup()instead.
Yes, dropping.
Quoted 39 lines
drivers/net/intel/common/tx_scalar.h ci_xmit_pkts(struct ci_tx_queue *txq, · line 520
Quoted 6 lines
drivers/net/intel/common/tx_scalar.h +1
-4-if (txe->mbuf) { -rte_pktmbuf_free_seg (txe->mbuf); - txe->mbuf =NULL ; - } + txe->mbuf =NULL ;Already mentioned: Should
txe->mbufbe set to NULL inci_tx_xmit_cleanup()instead of inci_tx_xmit_pkts()?drivers/net/intel/common/tx_scalar.h ci_xmit_pkts(struct ci_tx_queue *txq, · line 533
Quoted 7 lines
drivers/net/intel/common/tx_scalar.h +0
-0write_txd (ctx_txd, cd_qw0, cd_qw1);@@ -489,10 +531,7 @@ ci_xmit_pkts(struct ci_tx_queue *txq, txn = &sw_ring[txe->next_id];RTE_MBUF_PREFETCH_TO_FREE (txn->mbuf);
RTE_MBUF_PREFETCH_TO_FREE()doesn't seem relevant here anymore. I don't know if it fits intoci_tx_xmit_cleanup()instead.drivers/net/intel/common/tx_scalar.h ci_xmit_pkts(struct ci_tx_queue *txq, · line 520
Quoted 5 lines
drivers/net/intel/common/tx_scalar.h +1
-4-if (txe->mbuf) { -rte_pktmbuf_free_seg (txe->mbuf); - txe->mbuf =NULL ; - } + txe->mbuf =NULL ;Already mentioned: Should
txe->mbufbe set to NULL inci_tx_xmit_cleanup()instead of inci_tx_xmit_pkts()?Quoted 11 lines
drivers/net/intel/common/tx_scalar.h +0
-2ipsec_txd[0 ] = ipsec_qw0; ipsec_txd[1 ] = ipsec_qw1;@@ -507,10 +546,21 @@ ci_xmit_pkts(struct ci_tx_queue *txq, txd = &ci_tx_ring[tx_id]; txn = &sw_ring[txe->next_id]; -if (txe->mbuf) -rte_pktmbuf_free_seg (txe->mbuf); txe->mbuf = m_seg;
Quoted 15 lines
Quoted 11 lines
drivers/net/intel/common/tx_scalar.h +1
-0+/* For FAST_FREE: reset mbuf fields while we have it in cache.
drivers/net/intel/common/tx_scalar.h +1
-0+* FAST_FREE guarantees refcnt=1 and direct mbufs, so we only
drivers/net/intel/common/tx_scalar.h +1
-0+* need to reset nb_segs and next pointer as per
rte_pktmbuf_prefree_seg.drivers/net/intel/common/tx_scalar.h +1
-0+* Save next pointer before resetting since we need it for loop iteration.
drivers/net/intel/common/tx_scalar.h +3
-0+*/ +struct rte_mbuf *next_seg = m_seg->next; + if (txq->offloads &Quoted 1 line
RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE) {Similar to comment further above: Is
txq->offloadsortxq->fast_free_mphotter in the CPU cache here?
Updating as you suggest.
Quoted 54 lines
Quoted 53 lines
drivers/net/intel/common/tx_scalar.h +6
-0+if (m_seg->nb_segs !=1 ) + m_seg->nb_segs =1 ; +if (next_seg !=NULL ) + m_seg->next =NULL ; + } +/* Setup TX Descriptor */ /* Calculate segment length, using IPsec callback if provided */
drivers/net/intel/common/tx_scalar.h +0
-0if (ipsec_ops !=NULL )@@ -528,18 +578,23 @@ ci_xmit_pkts(struct ci_tx_queue *txq, ((uint64_t)CI_MAX_DATA_PER_TXD <<
CI_TXD_QW1_TX_BUF_SZ_S) |
((uint64_t)td_tag <<CI_TXD_QW1_L2TAG1_S);
write_txd(txd,buf_dma_addr,cmd_type_offset_bsz);drivers/net/intel/common/tx_scalar.h +1
-0+/* txe for this slot has already been written (e.g. above outside
drivers/net/intel/common/tx_scalar.h +1
-0+* loop), so we write the extra NULL mbuf pointer for this
drivers/net/intel/common/tx_scalar.h +3
-0+* descriptor after we increment txe below. +*/ buf_dma_addr +=CI_MAX_DATA_PER_TXD ; slen -=CI_MAX_DATA_PER_TXD ; tx_id = txe->next_id; txe = txn; + txe->mbuf =NULL ; txd = &ci_tx_ring[tx_id]; txn = &sw_ring[txe->next_id]; }/* fill the last descriptor with End of Packet (EOP) bit */
drivers/net/intel/common/tx_scalar.h +1
-1-if (m_seg->next ==NULL ) +if (next_seg ==NULL ) td_cmd |=CI_TX_DESC_CMD_EOP ; const uint64_t cmd_type_offset_bsz =
CI_TX_DESC_DTYPE_DATA|drivers/net/intel/common/tx_scalar.h +1
-2@@ -551,7 +606,7 @@ ci_xmit_pkts(struct ci_tx_queue *txq, tx_id = txe->next_id; txe = txn; - m_seg = m_seg->next; + m_seg = next_seg; }while (m_seg); end_pkt: txq->nb_tx_free = (uint16_t )(txq->nb_tx_free - nb_used); --2.51.0
PATCH v3 1/1 net/intel: optimize for fast-free hint
In reply to Bruce RichardsonWhen the fast-free hint is provided to the driver we know that the mbufs have refcnt of 1 and are from the same mempool. Therefore, we can optimize a bit for this case by:
* resetting the necessary mbuf fields, ie. nb_seg and next pointer when we are accessing the mbuf on writing the descriptor.
* on cleanup of buffers after transmit, we can just write those buffers straight to the mempool without accessing them.
* when doing buffer cleanup, we can follow the model of the vector driver where we cleanup in fixed bursts which don't wrap, and where we track the mbufs between tx_tail and last_cleaned as invalid, so we don't need to set them to NULL. [We only set to NULL valid mbuf slots where we have a context descriptor or segment not backed by an mbuf]
---
v3:
* used mbuf_raw_free_bulk rather than mempool function directly
* check for fast_free via mp pointer rather than flags
* remove unnecessary prefetches
V2: Fix issues with original submission:
* missed check for NULL mbufs
* fixed issue with freeing directly from sw_ring in scalar path which doesn't work as thats not a flag array of pointers
* fixed missing null assignment in case of large segments for TSO
--- drivers/net/intel/common/tx.h | 21 ++++-- drivers/net/intel/common/tx_scalar.h | 99 +++++++++++++++++++++------- 2 files changed, 92 insertions(+), 28 deletions(-)
drivers/net/intel/common/tx.h +15 -6
@@ -329,13 +329,22 @@ ci_txq_release_all_mbufs(struct ci_tx_queue *txq, bool use_ctx)return ;if (!txq->use_vec_entry) { -/* Regular scalar path uses sw_ring with ci_tx_entry */ -for (uint16_t i =0 ; i < txq->nb_tx_desc; i++) { -if (txq->sw_ring[i].mbuf !=NULL ) { -rte_pktmbuf_free_seg (txq->sw_ring[i].mbuf); - txq->sw_ring[i].mbuf =NULL ; - } +/* Free mbufs from (last_desc_cleaned + 1) to (tx_tail - 1). */ +const uint16_t start = (txq->last_desc_cleaned +1 ) % txq->nb_tx_desc; +const uint16_t nb_desc = txq->nb_tx_desc; +const uint16_t end = txq->tx_tail; + +uint16_t i = start; +if (end < i) { +for (; i < nb_desc; i++) +if (txq->sw_ring[i].mbuf !=NULL ) +rte_pktmbuf_free_seg (txq->sw_ring[i].mbuf); + i =0 ; } +for (; i < end; i++) +if (txq->sw_ring[i].mbuf !=NULL ) +rte_pktmbuf_free_seg (txq->sw_ring[i].mbuf); +memset (txq->sw_ring,0 ,sizeof (txq->sw_ring[0 ]) * nb_desc);return ; }
drivers/net/intel/common/tx_scalar.h +77 -22
158 lines not on this page · raw message
Signature · 1 line
2.53.0
RE: [PATCH v3] net/intel: optimize for fast-free hint
Quoted 2 lines
Subject: [PATCH v3] net/intel: optimize for fast-free hint
snip
Quoted 41 lines
drivers/net/intel/common/tx_scalar.h +3
-5@@ -197,16 +197,64 @@ ci_tx_xmit_cleanup(struct ci_tx_queue *txq)const uint16_t rs_idx = (last_desc_cleaned == nb_tx_desc -1 ) ?0 : (last_desc_cleaned +1 ) >> txq->log2_rs_thresh; -uint16_t desc_to_clean_to = (rs_idx << txq->log2_rs_thresh) + (txq- >tx_rs_thresh - 1); +const uint16_t dd_idx = txq->rs_last_id[rs_idx]; +const uint16_t first_to_clean = rs_idx << txq->log2_rs_thresh; -/* Check if descriptor is done */ -if ((txd[txq->rs_last_id[rs_idx]].cmd_type_offset_bsz & -rte_cpu_to_le_64 (CI_TXD_QW1_DTYPE_M )) != - rte_cpu_to_le_64(CI_TX_DESC_DTYPE_DESC_DONE)) +/* Check if descriptor is done - all drivers use 0xF as done value in bits 3:0 */
drivers/net/intel/common/tx_scalar.h +1
-0+ if ((txd[dd_idx].cmd_type_offset_bsz &
rte_cpu_to_le_64(CI_TXD_QW1_DTYPE_M)) !=drivers/net/intel/common/tx_scalar.h +5
-0+ rte_cpu_to_le_64(CI_TX_DESC_DTYPE_DESC_DONE)) +/* Descriptor not yet processed by hardware */ return -1 ; +/* DD bit is set, descriptors are done. Now free the mbufs. */ +/* Note: nb_tx_desc is guaranteed to be a multiple of tx_rs_thresh, +* validated during queue setup. This means cleanup never wraps around
drivers/net/intel/common/tx_scalar.h +9
-0+* the ring within a single burst (e.g., ring=256, rs_thresh=32 gives +* bursts of 0-31, 32-63, ..., 224-255). +*/ +const uint16_t nb_to_clean = txq->tx_rs_thresh; +struct ci_tx_entry *sw_ring = txq->sw_ring; + +/* fast_free_mp is NULL only when the fast free is disabled*/ +if (txq->fast_free_mp !=NULL ) { +/* FAST_FREE path: mbufs are already reset, just return to pool */
drivers/net/intel/common/tx_scalar.h ci_tx_xmit_cleanup(struct ci_tx_queue *txq) · line 228
Quoted 8 lines
drivers/net/intel/common/tx_scalar.h +8
-0+struct rte_mbuf *free[CI_TX_MAX_FREE_BUF_SZ ]; +uint16_t nb_free =0 ; + +/* Get cached mempool pointer, or cache it on first use */ +struct rte_mempool *mp = +likely (txq->fast_free_mp != (void *)UINTPTR_MAX ) ? + txq->fast_free_mp : + (txq->fast_free_mp = sw_ring[dd_idx].mbuf->pool);
Is the mbuf you are dereferencing here guaranteed to be non NULL?
Other than that concern:
Quoted 6 lines
drivers/net/intel/common/tx_scalar.h +3
-0+ +/* Pack non-NULL mbufs in-place at start of sw_ring range. +* No modulo needed in loop since we're guaranteed not to wrap.
drivers/net/intel/common/tx_scalar.h +2
-0+*/ +for (uint16_t i =0 ; i < nb_to_clean; i++) {
Re: [PATCH v3] net/intel: optimize for fast-free hint
Quoted 48 lines · Ciara
Quoted 2 lines
Subject: [PATCH v3] net/intel: optimize for fast-free hint
snip
Quoted 43 lines
drivers/net/intel/common/tx_scalar.h +3
-5@@ -197,16 +197,64 @@ ci_tx_xmit_cleanup(struct ci_tx_queue *txq)const uint16_t rs_idx = (last_desc_cleaned == nb_tx_desc -1 ) ?0 : (last_desc_cleaned +1 ) >> txq->log2_rs_thresh; -uint16_t desc_to_clean_to = (rs_idx << txq->log2_rs_thresh) + (txq- >tx_rs_thresh - 1); +const uint16_t dd_idx = txq->rs_last_id[rs_idx]; +const uint16_t first_to_clean = rs_idx << txq->log2_rs_thresh; -/* Check if descriptor is done */ -if ((txd[txq->rs_last_id[rs_idx]].cmd_type_offset_bsz & -rte_cpu_to_le_64 (CI_TXD_QW1_DTYPE_M )) != - rte_cpu_to_le_64(CI_TX_DESC_DTYPE_DESC_DONE)) +/* Check if descriptor is done - all drivers use 0xF as done value in bits 3:0 */
drivers/net/intel/common/tx_scalar.h +1
-0+ if ((txd[dd_idx].cmd_type_offset_bsz &
rte_cpu_to_le_64(CI_TXD_QW1_DTYPE_M)) !=drivers/net/intel/common/tx_scalar.h +5
-0+ rte_cpu_to_le_64(CI_TX_DESC_DTYPE_DESC_DONE)) +/* Descriptor not yet processed by hardware */ return -1 ; +/* DD bit is set, descriptors are done. Now free the mbufs. */ +/* Note: nb_tx_desc is guaranteed to be a multiple of tx_rs_thresh, +* validated during queue setup. This means cleanup never wraps around
drivers/net/intel/common/tx_scalar.h +9
-0+* the ring within a single burst (e.g., ring=256, rs_thresh=32 gives +* bursts of 0-31, 32-63, ..., 224-255). +*/ +const uint16_t nb_to_clean = txq->tx_rs_thresh; +struct ci_tx_entry *sw_ring = txq->sw_ring; + +/* fast_free_mp is NULL only when the fast free is disabled*/ +if (txq->fast_free_mp !=NULL ) { +/* FAST_FREE path: mbufs are already reset, just return to pool */
drivers/net/intel/common/tx_scalar.h +2
-0+struct rte_mbuf *free[CI_TX_MAX_FREE_BUF_SZ ]; +uint16_t nb_free =0 ;
Quoted 9 lines
drivers/net/intel/common/tx_scalar.h ci_tx_xmit_cleanup(struct ci_tx_queue *txq) · line 228
Quoted 6 lines
drivers/net/intel/common/tx_scalar.h +6
-0+ +/* Get cached mempool pointer, or cache it on first use */ +struct rte_mempool *mp = +likely (txq->fast_free_mp != (void *)UINTPTR_MAX ) ? + txq->fast_free_mp : + (txq->fast_free_mp = sw_ring[dd_idx].mbuf->pool);Is the mbuf you are dereferencing here guaranteed to be non NULL?
Good point. It almost certainly is, but checking through the logic of the code, the one possible error condition is where we have the last segment of the packet being a large segment using TSO which is to be split across multiple data descriptors. In that case (only) the mbuf pointer can be NULL. In all other cases, it should be valid, because the DD bit can only be set on a data descriptor.
Rather than trying to fix that issue here on cleanup, I think the better solution is to adjust the large segment handling so that the last descriptor of the segment, rather than the first, has the mbuf pointer. That way we can always know that the descriptor that has DD set on it will have a valid mbuf. Will implement this as a separate patch in v4.
/Bruce
v4 cover net/intel: optimize for fast-free hint
In reply to Bruce RichardsonWhen the fast-free hint is provided to the driver we know that the mbufs have refcnt of 1 and are from the same mempool. Therefore, we can optimize a bit for this case even in the scalar path of our drivers.
---
v4:
* add precursor patch to adjust mbuf pointers so that the DD bit is written to a descriptor with a valid mbuf pointer associated with it.
v3:
* used mbuf_raw_free_bulk rather than mempool function directly
* check for fast_free via mp pointer rather than flags
* remove unnecessary prefetches
V2: Fix issues with original submission:
* missed check for NULL mbufs
* fixed issue with freeing directly from sw_ring in scalar path which doesn't work as thats not a flag array of pointers
* fixed missing null assignment in case of large segments for TSO
Bruce Richardson (2):
net/intel: write mbuf for last Tx desc of segment net/intel: optimize for fast-free hint
drivers/net/intel/common/tx.h | 21 ++++-- drivers/net/intel/common/tx_scalar.h | 98 +++++++++++++++++++++------- 2 files changed, 90 insertions(+), 29 deletions(-)
--
2.53.0
PATCH v4 1/2 net/intel: write mbuf for last Tx desc of segment
When a single mbuf segment has more data than can be handled by a single Tx data descriptor in the TSO case, adjust how the storing of mbufs is being done. Rather than putting the mbuf pointer in the first slot for that segment, store it in the last slot instead. This guarantees for us that the descriptor for which we have desc-done (DD) writeback always has a valid mbuf associated with it.
--- drivers/net/intel/common/tx_scalar.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/net/intel/common/tx_scalar.h +2 -1
@@ -509,7 +509,6 @@ ci_xmit_pkts(struct ci_tx_queue *txq,if (txe->mbuf)rte_pktmbuf_free_seg (txe->mbuf); - txe->mbuf = m_seg;/* Setup TX Descriptor */ /* Calculate segment length, using IPsec callback if provided */
@@ -528,6 +527,7 @@ ci_xmit_pkts(struct ci_tx_queue *txq, ((uint64_t )CI_MAX_DATA_PER_TXD <<CI_TXD_QW1_TX_BUF_SZ_S ) | ((uint64_t )td_tag <<CI_TXD_QW1_L2TAG1_S );write_txd (txd, buf_dma_addr, cmd_type_offset_bsz); + txe->mbuf =NULL ; buf_dma_addr +=CI_MAX_DATA_PER_TXD ; slen -=CI_MAX_DATA_PER_TXD ;
@@ -548,6 +548,7 @@ ci_xmit_pkts(struct ci_tx_queue *txq, ((uint64_t )slen <<CI_TXD_QW1_TX_BUF_SZ_S ) | ((uint64_t )td_tag <<CI_TXD_QW1_L2TAG1_S );write_txd (txd, buf_dma_addr, cmd_type_offset_bsz); + txe->mbuf = m_seg; tx_id = txe->next_id; txe = txn;
Signature · 1 line
2.53.0
RE: [PATCH v4 1/2] net/intel: write mbuf for last Tx desc of segment
Quoted 10 lines
Subject: [PATCH v4 1/2] net/intel: write mbuf for last Tx desc of segment
When a single mbuf segment has more data than can be handled by a single Tx data descriptor in the TSO case, adjust how the storing of mbufs is being done. Rather than putting the mbuf pointer in the first slot for that segment, store it in the last slot instead. This guarantees for us that the descriptor for which we have desc-done (DD) writeback always has a valid mbuf associated with it.
Signed-off-by Bruce Richardson <[email protected]>
The fix looks good to me.
Quoted 41 lines
--- drivers/net/intel/common/tx_scalar.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)drivers/net/intel/common/tx_scalar.h +0
-1@@ -509,7 +509,6 @@ ci_xmit_pkts(struct ci_tx_queue *txq,if (txe->mbuf)rte_pktmbuf_free_seg (txe->mbuf); - txe->mbuf = m_seg;/* Setup TX Descriptor */ /* Calculate segment length, using IPsec callback if provided */
drivers/net/intel/common/tx_scalar.h +0
-0@@ -528,6 +527,7 @@ ci_xmit_pkts(struct ci_tx_queue *txq, ((uint64_t)CI_MAX_DATA_PER_TXD<<
CI_TXD_QW1_TX_BUF_SZ_S) |
((uint64_t)td_tag <<CI_TXD_QW1_L2TAG1_S);
write_txd(txd,buf_dma_addr,cmd_type_offset_bsz);drivers/net/intel/common/tx_scalar.h +1
-0+ txe->mbuf =NULL ; buf_dma_addr +=CI_MAX_DATA_PER_TXD ; slen -=CI_MAX_DATA_PER_TXD ;@@ -548,6 +548,7 @@ ci_xmit_pkts(struct ci_tx_queue *txq, ((uint64_t)slen <<
CI_TXD_QW1_TX_BUF_SZ_S) |
((uint64_t)td_tag <<CI_TXD_QW1_L2TAG1_S);drivers/net/intel/common/tx_scalar.h +1
-1write_txd (txd, buf_dma_addr, cmd_type_offset_bsz); + txe->mbuf = m_seg; tx_id = txe->next_id; txe = txn; --2.53.0
PATCH v4 2/2 net/intel: optimize for fast-free hint
In reply to Bruce RichardsonWhen the fast-free hint is provided to the driver we know that the mbufs have refcnt of 1 and are from the same mempool. Therefore, we can optimize a bit for this case by:
* resetting the necessary mbuf fields, ie. nb_seg and next pointer when we are accessing the mbuf on writing the descriptor.
* on cleanup of buffers after transmit, we can just write those buffers straight to the mempool without accessing them.
* when doing buffer cleanup, we can follow the model of the vector driver where we cleanup in fixed bursts which don't wrap, and where we track the mbufs between tx_tail and last_cleaned as invalid, so we don't need to set them to NULL. [We only set to NULL valid mbuf slots where we have a context descriptor or segment not backed by an mbuf]
--- drivers/net/intel/common/tx.h | 21 ++++-- drivers/net/intel/common/tx_scalar.h | 95 +++++++++++++++++++++------- 2 files changed, 88 insertions(+), 28 deletions(-)
drivers/net/intel/common/tx.h +15 -6
@@ -329,13 +329,22 @@ ci_txq_release_all_mbufs(struct ci_tx_queue *txq, bool use_ctx)return ;if (!txq->use_vec_entry) { -/* Regular scalar path uses sw_ring with ci_tx_entry */ -for (uint16_t i =0 ; i < txq->nb_tx_desc; i++) { -if (txq->sw_ring[i].mbuf !=NULL ) { -rte_pktmbuf_free_seg (txq->sw_ring[i].mbuf); - txq->sw_ring[i].mbuf =NULL ; - } +/* Free mbufs from (last_desc_cleaned + 1) to (tx_tail - 1). */ +const uint16_t start = (txq->last_desc_cleaned +1 ) % txq->nb_tx_desc; +const uint16_t nb_desc = txq->nb_tx_desc; +const uint16_t end = txq->tx_tail; + +uint16_t i = start; +if (end < i) { +for (; i < nb_desc; i++) +if (txq->sw_ring[i].mbuf !=NULL ) +rte_pktmbuf_free_seg (txq->sw_ring[i].mbuf); + i =0 ; } +for (; i < end; i++) +if (txq->sw_ring[i].mbuf !=NULL ) +rte_pktmbuf_free_seg (txq->sw_ring[i].mbuf); +memset (txq->sw_ring,0 ,sizeof (txq->sw_ring[0 ]) * nb_desc);return ; }
drivers/net/intel/common/tx_scalar.h +73 -22
141 lines not on this page · raw message
Signature · 1 line
2.53.0
RE: [PATCH v4 0/2] net/intel: optimize for fast-free hint
In reply to Bruce RichardsonQuoted 30 lines
From: Bruce Richardson [mailto:[email protected]]
Sent: Tuesday, 2 June 2026 17.45When the fast-free hint is provided to the driver we know that the mbufs
have refcnt of 1 and are from the same mempool. Therefore, we can optimize a bit for this case even in the scalar path of our drivers.---
v4:
* add precursor patch to adjust mbuf pointers so that the DD bit is written to a descriptor with a valid mbuf pointer associated with it.v3:
* usedmbuf_raw_free_bulkrather than mempool function directly
* check forfast_freevia mp pointer rather than flags
* remove unnecessary prefetchesV2: Fix issues with original submission:
* missed check for NULL mbufs
* fixed issue with freeing directly fromsw_ringin scalar path which doesn't work as thats not a flag array of pointers
* fixed missing null assignment in case of large segments for TSOBruce Richardson (2):
net/intel: write mbuf for last Tx desc of segment net/intel: optimize for fast-free hint
Quoted 6 lines
drivers/net/intel/common/tx.h | 21 ++++-- drivers/net/intel/common/tx_scalar.h | 98 +++++++++++++++++++++------- 2 files changed, 90 insertions(+), 29 deletions(-)--
2.53.0
Good catch by Ciara, and good solution to it.
Bruce Richardson
AppliedRe: [PATCH v4 0/2] net/intel: optimize for fast-free hint
Quoted 34 lines · Morten Brørup
Quoted 34 lines
From: Bruce Richardson [mailto:[email protected]]
Sent: Tuesday, 2 June 2026 17.45When the fast-free hint is provided to the driver we know that the mbufs
have refcnt of 1 and are from the same mempool. Therefore, we can optimize a bit for this case even in the scalar path of our drivers.---
v4:
* add precursor patch to adjust mbuf pointers so that the DD bit is written to a descriptor with a valid mbuf pointer associated with it.v3:
* usedmbuf_raw_free_bulkrather than mempool function directly
* check forfast_freevia mp pointer rather than flags
* remove unnecessary prefetchesV2: Fix issues with original submission:
* missed check for NULL mbufs
* fixed issue with freeing directly fromsw_ringin scalar path which doesn't work as thats not a flag array of pointers
* fixed missing null assignment in case of large segments for TSOBruce Richardson (2):
net/intel: write mbuf for last Tx desc of segment net/intel: optimize for fast-free hintdrivers/net/intel/common/tx.h | 21 ++++-- drivers/net/intel/common/tx_scalar.h | 98 +++++++++++++++++++++------- 2 files changed, 90 insertions(+), 29 deletions(-)
Quoted 6 lines
Quoted 2 lines
--
2.53.0Good catch by Ciara, and good solution to it.
Series-Acked-by Morten Brørup <[email protected]>
Applied to dpdk-next-net-intel.
Thanks for all the reviews folks. Quick tests on next-net-intel tree with scalar driver show a nice perf bump from this change.
/Bruce
Quoted 15 lines · Morten Brørup
Quoted 9 lines
Quoted 1 line
If I'm not mistaken, the mbuf library is not a barrier for fast-
freeing
Quoted 1 line
segmented packet mbufs, and thus fast-free of jumbo frames is
possible.
Quoted 5 lines
We need a driver developer to confirm that my suggested approach - resetting the mbuf fields, incl. '
m->nb_segs' and 'm->next', when preparing the Tx descriptor - is viable.Quoted 5 lines
Excellent analysis, Morten. If I get a chance some time this release cycle,
I will try implementing this change in our drivers, see if any difference
is made.
Quoted 5 lines
Bruce,
Have you had a chance to look into the driver change requirements? If not, could you please try scratching the surface, to build a gut feeling.
Prototype implementation done and posted as a patch to this thread[1]. (It applies on top of my "combine multiple Intel scalar Tx paths" patchset[2]).
The logic is not terribly complicated, and running a quick perf test with scattered Tx enabled (to force use of scalar path, this optimization doesn't apply to vector path*) shows a small perf increase from it using testpmd iofwd.
/Bruce
[1] https://patches.dpdk.org/project/dpdk/patch/[email protected]/
[2] https://patches.dpdk.org/project/dpdk/list/?series=37033
* Vector path already has support for fast-free, and disallows use of scattered packets, so there is nothing to optimize for here.
Quoted 250 lines
Executive Summary:
My analysis shows that the mbuf library is not a barrier for fast-freeing segmented packet mbufs, and thus fast-free of jumbo frames is possible.
Detailed Analysis:
The purpose of the mbuf fast-free Tx optimization is to reduce
rte_pktmbuf_free_seg()to something much simpler in the ethdev drivers, by eliminating the code path related to indirect mbufs. Optimally, we want to simplify the ethdev driver's function that frees the transmitted mbufs, so it can free them directly to their mempool without accessing the mbufs themselves.If the driver cannot access the mbuf itself, it cannot determine which mempool it belongs to.
We don't want the driver to access every mbuf being freed; but if all mbufs of a Tx queue belong to the same mempool, the driver can determine which mempool by looking into just one of the mbufs.REQUIREMENT 1: The mbufs of a Tx queue must come from the same mempool.
When an mbuf is freed to its mempool, some of the fields in the mbuf must be initialized.
So, for fast-free, this must be done by the driver's function that prepares the Tx descriptor.
This is a requirement to the driver, not a requirement to the application.Now, let's dig into the code for freeing an mbuf.
Note: For readability purposes, I'll cut out some code and comments unrelated to this topic.static
__rte_always_inlinevoidrte_pktmbuf_free_seg(struct rte_mbuf *m){ m = rte_pktmbuf_prefree_seg(m); if (likely(m != NULL)) rte_mbuf_raw_free(m); }
rte_mbuf_raw_free(m)is simple, so nothing to gain there:/**
* Put mbuf back into its original mempool.
*
* The caller must ensure that the mbuf is direct and properly
* reinitialized (refcnt=1, next=NULL, nb_segs=1), as done by
*rte_pktmbuf_prefree_seg().
*/
static__rte_always_inlinevoidrte_mbuf_raw_free(struct rte_mbuf *m){ rte_mbuf_history_mark(m, RTE_MBUF_HISTORY_OP_LIB_FREE); rte_mempool_put(m->pool, m); }Note that the description says that the mbuf must be direct. This is not entirely accurate; the mbuf is allowed to use a pinned external buffer, if the mbuf holds the only reference to it. (Most of the mbuf library functions have this documentation inaccuracy, which should be fixed some day.)
So, the fast-free optimization really comes down to
rte_pktmbuf_prefree_seg(m), which must not return NULL.Let's dig into that.
/**
* Decrease reference counter and unlink a mbuf segment *
* This function does the same than a free, except that it does not
* return the segment to its pool.
* It decreases the reference counter, and if it reaches 0, it is
* detached from its parent for an indirect mbuf.
*
* @return
* - (m) if it is the last reference. It can be recycled or freed.
* - (NULL) if the mbuf still has remaining references on it.
*/
static__rte_always_inlinestructrte_mbuf*rte_pktmbuf_prefree_seg(struct rte_mbuf *m){ bool refcnt_not_one; refcnt_not_one = unlikely(rte_mbuf_refcnt_read(m) != 1); if (refcnt_not_one && __rte_mbuf_refcnt_update(m, -1) != 0) return NULL; if (unlikely(!RTE_MBUF_DIRECT(m))) { rte_pktmbuf_detach(m); if (RTE_MBUF_HAS_EXTBUF(m) && RTE_MBUF_HAS_PINNED_EXTBUF(m) && __rte_pktmbuf_pinned_extbuf_decref(m)) return NULL; } if (refcnt_not_one) rte_mbuf_refcnt_set(m, 1); if (m->nb_segs != 1) m->nb_segs = 1; if (m->next != NULL) m->next = NULL; return m; }This function can only succeed (i.e. return non-NULL) when 'refcnt' is 1 (or reaches 0).
REQUIREMENT 2: The driver must hold the only reference to the mbuf, i.e. '
m->refcnt' must be 1.When the function succeeds, it initializes the mbuf fields as required by
rte_mbuf_raw_free()before returning.Now, since the driver has exclusive access to the mbuf, it is free to initialize the '
m->next' and 'm->nb_segs' at any time.
It could do that when preparing the Tx descriptor.This is very interesting, because it means that fast-free does not prohibit segmented packets!
(But the driver must have sufficient Tx descriptors for all segments in the mbuf.)Now, lets dig into rte_pktmbuf_prefree_seg()'s block handling non-direct mbufs, i.e. cloned mbufs and mbufs with external buffer:
if (unlikely(!RTE_MBUF_DIRECT(m))) { rte_pktmbuf_detach(m); if (RTE_MBUF_HAS_EXTBUF(m) && RTE_MBUF_HAS_PINNED_EXTBUF(m) && __rte_pktmbuf_pinned_extbuf_decref(m)) return NULL; }Starting with
rte_pktmbuf_detach():static inline void
rte_pktmbuf_detach(struct rte_mbuf *m){ struct rte_mempool *mp = m->pool; uint32_t mbuf_size, buf_len; uint16_t priv_size; if (RTE_MBUF_HAS_EXTBUF(m)) { /* * The mbuf has the external attached buffer, * we should check the type of the memory pool where * the mbuf was allocated from to detect the pinned * external buffer. */ uint32_t flags = rte_pktmbuf_priv_flags(mp); if (flags & RTE_PKTMBUF_POOL_F_PINNED_EXT_BUF) { /* * The pinned external buffer should not be * detached from its backing mbuf, just exit. */ return; } __rte_pktmbuf_free_extbuf(m); } else { __rte_pktmbuf_free_direct(m); } priv_size = rte_pktmbuf_priv_size(mp); mbuf_size = (uint32_t)(sizeof(struct rte_mbuf) + priv_size); buf_len = rte_pktmbuf_data_room_size(mp); m->priv_size = priv_size; m->buf_addr = (char *)m + mbuf_size; rte_mbuf_iova_set(m, rte_mempool_virt2iova(m) + mbuf_size); m->buf_len = (uint16_t)buf_len; rte_pktmbuf_reset_headroom(m); m->data_len = 0; m->ol_flags = 0; }The only quick and simple code path through this function is when the mbuf uses a pinned external buffer:
if (RTE_MBUF_HAS_EXTBUF(m)) { uint32_t flags = rte_pktmbuf_priv_flags(mp); if (flags & RTE_PKTMBUF_POOL_F_PINNED_EXT_BUF) return;REQUIREMENT 3: The mbuf must not be cloned or use a non-pinned external buffer.
Continuing with the next part of rte_pktmbuf_prefree_seg()'s block:
if (RTE_MBUF_HAS_EXTBUF(m)&&RTE_MBUF_HAS_PINNED_EXTBUF(m) && __rte_pktmbuf_pinned_extbuf_decref(m)) return NULL;Continuing with the next part of the block in
rte_pktmbuf_prefree_seg():/**
* @internal Handle the packet mbufs with attached pinned external buffer
* on the mbuf freeing:
** - return zero if reference counter in shinfo is one. It means there is * no more reference to this pinned buffer and mbuf can be returned to * the pool*
* - otherwise (if reference counter is not one), decrement reference
* counter and return non-zero value to prevent freeing the backing mbuf. *
* Returns non zero if mbuf should not be freed.
*/
static inline int__rte_pktmbuf_pinned_extbuf_decref(struct rte_mbuf *m){ struct rte_mbuf_ext_shared_info *shinfo; /* Clear flags, mbuf is being freed. */ m->ol_flags = RTE_MBUF_F_EXTERNAL; shinfo = m->shinfo; /* Optimize for performance - do not dec/reinit */ if (likely(rte_mbuf_ext_refcnt_read(shinfo) == 1)) return 0; /* * Direct usage of add primitive to avoid * duplication of comparing with one. */ if (likely(rte_atomic_fetch_add_explicit(&shinfo->refcnt, -1, rte_memory_order_acq_rel) - 1)) return 1; /* Reinitialize counter before mbuf freeing. */ rte_mbuf_ext_refcnt_set(shinfo, 1); return 0; }Essentially, if the mbuf does use a pinned external buffer,
rte_pktmbuf_prefree_seg()only succeeds if that pinned external buffer is only referred to by the mbuf.REQUIREMENT 4: If the mbuf uses a pinned external buffer, the mbuf must hold the only reference to that pinned external buffer, i.e. in that case, '
m->shinfo->refcnt' must be 1.
Quoted 8 lines
Please review.
If I'm not mistaken, the mbuf library is not a barrier for fast-freeing segmented packet mbufs, and thus fast-free of jumbo frames is possible.
We need a driver developer to confirm that my suggested approach - resetting the mbuf fields, incl. '
m->nb_segs' and 'm->next', when preparing the Tx descriptor - is viable.
Great analysis, makes a lot of sense to me.
Shall we add then a special API to make PMD maintainers life a bit easier:
Something like rte_mbuf_fast_free_prep(mp, mb), that will optionally check that requirements outlined above are satisfied for given mbuf and also reset mbuf fields to expected values?
Konstantin
Quoted 289 lines
From: Konstantin Ananyev [mailto:[email protected]]
Sent: Monday, 15 December 2025 15.41Quoted 4 lines
Executive Summary:
My analysis shows that the mbuf library is not a barrier for fast-
freeing
Quoted 1 line
segmented packet mbufs, and thus fast-free of jumbo frames is
possible.
Quoted 6 lines
Detailed Analysis:
The purpose of the mbuf fast-free Tx optimization is to reduce
rte_pktmbuf_free_seg()to something much simpler in the ethdevdrivers, by
Quoted 2 lines
eliminating the code path related to indirect mbufs. Optimally, we want to simplify the ethdev driver's function that
frees the
Quoted 1 line
transmitted mbufs, so it can free them directly to their mempool
without
Quoted 3 lines
accessing the mbufs themselves.
If the driver cannot access the mbuf itself, it cannot determine
which
Quoted 3 lines
mempool it belongs to.
We don't want the driver to access every mbuf being freed; but if all mbufs of a Tx queue belong to the same mempool, the driver candetermine
Quoted 3 lines
which mempool by looking into just one of the mbufs.
REQUIREMENT 1: The mbufs of a Tx queue must come from the same
mempool.
Quoted 3 lines
When an mbuf is freed to its mempool, some of the fields in the mbuf
must
Quoted 4 lines
be initialized.
So, for fast-free, this must be done by the driver's function that prepares the Tx descriptor.
This is a requirement to the driver, not a requirement to theapplication.
Quoted 30 lines
Now, let's dig into the code for freeing an mbuf.
Note: For readability purposes, I'll cut out some code and comments unrelated to this topic.static
__rte_always_inlinevoidrte_pktmbuf_free_seg(struct rte_mbuf *m){ m = rte_pktmbuf_prefree_seg(m); if (likely(m != NULL)) rte_mbuf_raw_free(m); }
rte_mbuf_raw_free(m)is simple, so nothing to gain there:/**
* Put mbuf back into its original mempool.
*
* The caller must ensure that the mbuf is direct and properly
* reinitialized (refcnt=1, next=NULL, nb_segs=1), as done by
*rte_pktmbuf_prefree_seg().
*/
static__rte_always_inlinevoidrte_mbuf_raw_free(struct rte_mbuf *m){ rte_mbuf_history_mark(m, RTE_MBUF_HISTORY_OP_LIB_FREE); rte_mempool_put(m->pool, m); }Quoted 4 lines
Note that the description says that the mbuf must be direct. This is not entirely accurate; the mbuf is allowed to use a pinned external buffer, if the mbuf holds the only reference to it. (Most of the mbuf library functions have this documentation
inaccuracy,
Quoted 46 lines
which should be fixed some day.)
So, the fast-free optimization really comes down to
rte_pktmbuf_prefree_seg(m), which must not return NULL.Let's dig into that.
/**
* Decrease reference counter and unlink a mbuf segment *
* This function does the same than a free, except that it does not
* return the segment to its pool.
* It decreases the reference counter, and if it reaches 0, it is
* detached from its parent for an indirect mbuf.
*
* @return
* - (m) if it is the last reference. It can be recycled or freed.
* - (NULL) if the mbuf still has remaining references on it.
*/
static__rte_always_inlinestructrte_mbuf*rte_pktmbuf_prefree_seg(struct rte_mbuf *m){ bool refcnt_not_one; refcnt_not_one = unlikely(rte_mbuf_refcnt_read(m) != 1); if (refcnt_not_one && __rte_mbuf_refcnt_update(m, -1) != 0) return NULL; if (unlikely(!RTE_MBUF_DIRECT(m))) { rte_pktmbuf_detach(m); if (RTE_MBUF_HAS_EXTBUF(m) && RTE_MBUF_HAS_PINNED_EXTBUF(m) && __rte_pktmbuf_pinned_extbuf_decref(m)) return NULL; } if (refcnt_not_one) rte_mbuf_refcnt_set(m, 1); if (m->nb_segs != 1) m->nb_segs = 1; if (m->next != NULL) m->next = NULL; return m; }Quoted 1 line
This function can only succeed (i.e. return non-NULL) when 'refcnt'
is 1
Quoted 7 lines
(or reaches 0).
REQUIREMENT 2: The driver must hold the only reference to the mbuf, i.e. '
m->refcnt' must be 1.When the function succeeds, it initializes the mbuf fields as
required by
Quoted 9 lines
rte_mbuf_raw_free()before returning.Now, since the driver has exclusive access to the mbuf, it is free to initialize the '
m->next' and 'm->nb_segs' at any time.
It could do that when preparing the Tx descriptor.This is very interesting, because it means that fast-free does not prohibit segmented packets!
(But the driver must have sufficient Tx descriptors for all segmentsin
Quoted 4 lines
the mbuf.)
Now, lets dig into rte_pktmbuf_prefree_seg()'s block handling non-
direct
Quoted 51 lines
mbufs, i.e. cloned mbufs and mbufs with external buffer:
if (unlikely(!RTE_MBUF_DIRECT(m))) { rte_pktmbuf_detach(m); if (RTE_MBUF_HAS_EXTBUF(m) && RTE_MBUF_HAS_PINNED_EXTBUF(m) && __rte_pktmbuf_pinned_extbuf_decref(m)) return NULL; }Starting with
rte_pktmbuf_detach():static inline void
rte_pktmbuf_detach(struct rte_mbuf *m){ struct rte_mempool *mp = m->pool; uint32_t mbuf_size, buf_len; uint16_t priv_size; if (RTE_MBUF_HAS_EXTBUF(m)) { /* * The mbuf has the external attached buffer, * we should check the type of the memory pool where * the mbuf was allocated from to detect the pinned * external buffer. */ uint32_t flags = rte_pktmbuf_priv_flags(mp); if (flags & RTE_PKTMBUF_POOL_F_PINNED_EXT_BUF) { /* * The pinned external buffer should not be * detached from its backing mbuf, just exit. */ return; } __rte_pktmbuf_free_extbuf(m); } else { __rte_pktmbuf_free_direct(m); } priv_size = rte_pktmbuf_priv_size(mp); mbuf_size = (uint32_t)(sizeof(struct rte_mbuf) + priv_size); buf_len = rte_pktmbuf_data_room_size(mp); m->priv_size = priv_size; m->buf_addr = (char *)m + mbuf_size; rte_mbuf_iova_set(m, rte_mempool_virt2iova(m) + mbuf_size); m->buf_len = (uint16_t)buf_len; rte_pktmbuf_reset_headroom(m); m->data_len = 0; m->ol_flags = 0; }Quoted 1 line
The only quick and simple code path through this function is when the
mbuf
Quoted 7 lines
uses a pinned external buffer:
if (RTE_MBUF_HAS_EXTBUF(m)) { uint32_t flags = rte_pktmbuf_priv_flags(mp); if (flags & RTE_PKTMBUF_POOL_F_PINNED_EXT_BUF) return;REQUIREMENT 3: The mbuf must not be cloned or use a non-pinned
external
Quoted 10 lines
buffer.
Continuing with the next part of rte_pktmbuf_prefree_seg()'s block:
if (RTE_MBUF_HAS_EXTBUF(m)&&RTE_MBUF_HAS_PINNED_EXTBUF(m) && __rte_pktmbuf_pinned_extbuf_decref(m)) return NULL;Continuing with the next part of the block in
rte_pktmbuf_prefree_seg():Quoted 3 lines
/**
* @internal Handle the packet mbufs with attached pinned externalbuffer
Quoted 3 lines
* on the mbuf freeing:
*
* - return zero if reference counter in shinfo is one. It meansthere is
Quoted 1 line
* no more reference to this pinned buffer and mbuf can be returned
to
Quoted 3 lines
* the pool
*
* - otherwise (if reference counter is not one), decrementreference
Quoted 1 line
* counter and return non-zero value to prevent freeing the backing
mbuf.
Quoted 4 lines
*
* Returns non zero if mbuf should not be freed.
*/
static inline int __rte_pktmbuf_pinned_extbuf_decref(structrte_mbuf*m)
Quoted 24 lines
{ struct rte_mbuf_ext_shared_info *shinfo; /* Clear flags, mbuf is being freed. */ m->ol_flags = RTE_MBUF_F_EXTERNAL; shinfo = m->shinfo; /* Optimize for performance - do not dec/reinit */ if (likely(rte_mbuf_ext_refcnt_read(shinfo) == 1)) return 0; /* * Direct usage of add primitive to avoid * duplication of comparing with one. */ if (likely(rte_atomic_fetch_add_explicit(&shinfo->refcnt, -1, rte_memory_order_acq_rel) - 1)) return 1; /* Reinitialize counter before mbuf freeing. */ rte_mbuf_ext_refcnt_set(shinfo, 1); return 0; }Quoted 2 lines
Essentially, if the mbuf does use a pinned external buffer,
rte_pktmbuf_prefree_seg()only succeeds if that pinned externalbuffer is
Quoted 3 lines
only referred to by the mbuf.
REQUIREMENT 4: If the mbuf uses a pinned external buffer, the mbuf
must
Quoted 1 line
hold the only reference to that pinned external buffer, i.e. in that
case,
Quoted 6 lines
'
m->shinfo->refcnt' must be 1.Please review.
If I'm not mistaken, the mbuf library is not a barrier for fast-
freeing
Quoted 1 line
segmented packet mbufs, and thus fast-free of jumbo frames is
possible.
Quoted 4 lines
We need a driver developer to confirm that my suggested approach - resetting the mbuf fields, incl. '
m->nb_segs' and 'm->next', when preparing the Tx descriptor - is viable.
Quoted 7 lines
Great analysis, makes a lot of sense to me.
Shall we add then a special API to make PMD maintainers life a bit easier:
Something likerte_mbuf_fast_free_prep(mp, mb), that will optionally check
that requirements outlined above are satisfied for given mbuf and also reset mbuf fields to expected values?
Good idea, Konstantin.
Detailed suggestion below.
Note that __rte_mbuf_raw_sanity_check_mp() is used to checks the requirements after 'nb_segs' and 'next' have been initialized.
/**
* Reinitialize an mbuf for freeing back into the mempool.
*
* The caller must ensure that the mbuf comes from the specified mempool,
* is direct and only referred to by the caller (refcnt=1).
*
* This function is used by drivers in their transmit function for mbuf fast release
* when the transmit descriptor is initialized,
* so the driver can call rte_mbuf_raw_free()
* when the packet segment has been transmitted.
*
* @see RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE
*
* @param mp
* The mempool to which the mbuf belong.
* @param m
* The mbuf being reinitialized.
*/
static __rte_always_inline voidrte_mbuf_raw_prefree_seg(const struct rte_mempool *mp, struct rte_mbuf *m)
{
if (m->nb_segs != 1)
m->nb_segs = 1;
if (m->next != NULL)
m->next = NULL;
__rte_mbuf_raw_sanity_check_mp(m, mp);
rte_mbuf_history_mark(mbuf, RTE_MBUF_HISTORY_OP_LIB_PREFREE_RAW);
}/**
* Reinitialize a bulk of mbufs for freeing back into the mempool.
*
* The caller must ensure that the mbufs come from the specified mempool,
* are direct and only referred to by the caller (refcnt=1).
*
* This function is used by drivers in their transmit function for mbuf fast release
* when the transmit descriptors are initialized,
* so the driver can call rte_mbuf_raw_free_bulk()
* when the packet segments have been transmitted.
*
* @see RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE
*
* @param mp
* The mempool to which the mbufs belong.
* @param mbufs
* Array of pointers to mbufs being reinitialized.
* The array must not contain NULL pointers.
* @param count
* Array size.
*/
static __rte_always_inline voidrte_mbuf_raw_prefree_seg_bulk(const struct rte_mempool *mp, struct rte_mbuf **mbufs, unsigned int count)
{
for (unsigned int idx = 0; idx < count; idx++) {
struct rte_mbuf *m = mbufs[idx];
if (m->nb_segs != 1)
m->nb_segs = 1;
if (m->next != NULL)
m->next = NULL;
__rte_mbuf_raw_sanity_check_mp(m, mp);
}
rte_mbuf_history_mark_bulk(mbufs, count, RTE_MBUF_HISTORY_OP_LIB_PREFREE_RAW);
}Quoted 3 lines
Konstantin
Quoted 332 lines
Quoted 290 lines
From: Konstantin Ananyev [mailto:[email protected]]
Sent: Monday, 15 December 2025 15.41Quoted 4 lines
Executive Summary:
My analysis shows that the mbuf library is not a barrier for fast-
freeing
Quoted 1 line
segmented packet mbufs, and thus fast-free of jumbo frames is
possible.
Quoted 6 lines
Detailed Analysis:
The purpose of the mbuf fast-free Tx optimization is to reduce
rte_pktmbuf_free_seg()to something much simpler in the ethdevdrivers, by
Quoted 2 lines
eliminating the code path related to indirect mbufs. Optimally, we want to simplify the ethdev driver's function that
frees the
Quoted 1 line
transmitted mbufs, so it can free them directly to their mempool
without
Quoted 3 lines
accessing the mbufs themselves.
If the driver cannot access the mbuf itself, it cannot determine
which
Quoted 3 lines
mempool it belongs to.
We don't want the driver to access every mbuf being freed; but if all mbufs of a Tx queue belong to the same mempool, the driver candetermine
Quoted 3 lines
which mempool by looking into just one of the mbufs.
REQUIREMENT 1: The mbufs of a Tx queue must come from the same
mempool.
Quoted 3 lines
When an mbuf is freed to its mempool, some of the fields in the mbuf
must
Quoted 4 lines
be initialized.
So, for fast-free, this must be done by the driver's function that prepares the Tx descriptor.
This is a requirement to the driver, not a requirement to theapplication.
Quoted 30 lines
Now, let's dig into the code for freeing an mbuf.
Note: For readability purposes, I'll cut out some code and comments unrelated to this topic.static
__rte_always_inlinevoidrte_pktmbuf_free_seg(struct rte_mbuf *m){ m = rte_pktmbuf_prefree_seg(m); if (likely(m != NULL)) rte_mbuf_raw_free(m); }
rte_mbuf_raw_free(m)is simple, so nothing to gain there:/**
* Put mbuf back into its original mempool.
*
* The caller must ensure that the mbuf is direct and properly
* reinitialized (refcnt=1, next=NULL, nb_segs=1), as done by
*rte_pktmbuf_prefree_seg().
*/
static__rte_always_inlinevoidrte_mbuf_raw_free(struct rte_mbuf *m){ rte_mbuf_history_mark(m, RTE_MBUF_HISTORY_OP_LIB_FREE); rte_mempool_put(m->pool, m); }Quoted 4 lines
Note that the description says that the mbuf must be direct. This is not entirely accurate; the mbuf is allowed to use a pinned external buffer, if the mbuf holds the only reference to it. (Most of the mbuf library functions have this documentation
inaccuracy,
Quoted 46 lines
which should be fixed some day.)
So, the fast-free optimization really comes down to
rte_pktmbuf_prefree_seg(m), which must not return NULL.Let's dig into that.
/**
* Decrease reference counter and unlink a mbuf segment *
* This function does the same than a free, except that it does not
* return the segment to its pool.
* It decreases the reference counter, and if it reaches 0, it is
* detached from its parent for an indirect mbuf.
*
* @return
* - (m) if it is the last reference. It can be recycled or freed.
* - (NULL) if the mbuf still has remaining references on it.
*/
static__rte_always_inlinestructrte_mbuf*rte_pktmbuf_prefree_seg(struct rte_mbuf *m){ bool refcnt_not_one; refcnt_not_one = unlikely(rte_mbuf_refcnt_read(m) != 1); if (refcnt_not_one && __rte_mbuf_refcnt_update(m, -1) != 0) return NULL; if (unlikely(!RTE_MBUF_DIRECT(m))) { rte_pktmbuf_detach(m); if (RTE_MBUF_HAS_EXTBUF(m) && RTE_MBUF_HAS_PINNED_EXTBUF(m) && __rte_pktmbuf_pinned_extbuf_decref(m)) return NULL; } if (refcnt_not_one) rte_mbuf_refcnt_set(m, 1); if (m->nb_segs != 1) m->nb_segs = 1; if (m->next != NULL) m->next = NULL; return m; }Quoted 1 line
This function can only succeed (i.e. return non-NULL) when 'refcnt'
is 1
Quoted 7 lines
(or reaches 0).
REQUIREMENT 2: The driver must hold the only reference to the mbuf, i.e. '
m->refcnt' must be 1.When the function succeeds, it initializes the mbuf fields as
required by
Quoted 9 lines
rte_mbuf_raw_free()before returning.Now, since the driver has exclusive access to the mbuf, it is free to initialize the '
m->next' and 'm->nb_segs' at any time.
It could do that when preparing the Tx descriptor.This is very interesting, because it means that fast-free does not prohibit segmented packets!
(But the driver must have sufficient Tx descriptors for all segmentsin
Quoted 4 lines
the mbuf.)
Now, lets dig into rte_pktmbuf_prefree_seg()'s block handling non-
direct
Quoted 51 lines
mbufs, i.e. cloned mbufs and mbufs with external buffer:
if (unlikely(!RTE_MBUF_DIRECT(m))) { rte_pktmbuf_detach(m); if (RTE_MBUF_HAS_EXTBUF(m) && RTE_MBUF_HAS_PINNED_EXTBUF(m) && __rte_pktmbuf_pinned_extbuf_decref(m)) return NULL; }Starting with
rte_pktmbuf_detach():static inline void
rte_pktmbuf_detach(struct rte_mbuf *m){ struct rte_mempool *mp = m->pool; uint32_t mbuf_size, buf_len; uint16_t priv_size; if (RTE_MBUF_HAS_EXTBUF(m)) { /* * The mbuf has the external attached buffer, * we should check the type of the memory pool where * the mbuf was allocated from to detect the pinned * external buffer. */ uint32_t flags = rte_pktmbuf_priv_flags(mp); if (flags & RTE_PKTMBUF_POOL_F_PINNED_EXT_BUF) { /* * The pinned external buffer should not be * detached from its backing mbuf, just exit. */ return; } __rte_pktmbuf_free_extbuf(m); } else { __rte_pktmbuf_free_direct(m); } priv_size = rte_pktmbuf_priv_size(mp); mbuf_size = (uint32_t)(sizeof(struct rte_mbuf) + priv_size); buf_len = rte_pktmbuf_data_room_size(mp); m->priv_size = priv_size; m->buf_addr = (char *)m + mbuf_size; rte_mbuf_iova_set(m, rte_mempool_virt2iova(m) + mbuf_size); m->buf_len = (uint16_t)buf_len; rte_pktmbuf_reset_headroom(m); m->data_len = 0; m->ol_flags = 0; }Quoted 1 line
The only quick and simple code path through this function is when the
mbuf
Quoted 7 lines
uses a pinned external buffer:
if (RTE_MBUF_HAS_EXTBUF(m)) { uint32_t flags = rte_pktmbuf_priv_flags(mp); if (flags & RTE_PKTMBUF_POOL_F_PINNED_EXT_BUF) return;REQUIREMENT 3: The mbuf must not be cloned or use a non-pinned
external
Quoted 10 lines
buffer.
Continuing with the next part of rte_pktmbuf_prefree_seg()'s block:
if (RTE_MBUF_HAS_EXTBUF(m)&&RTE_MBUF_HAS_PINNED_EXTBUF(m) && __rte_pktmbuf_pinned_extbuf_decref(m)) return NULL;Continuing with the next part of the block in
rte_pktmbuf_prefree_seg():Quoted 3 lines
/**
* @internal Handle the packet mbufs with attached pinned externalbuffer
Quoted 3 lines
* on the mbuf freeing:
*
* - return zero if reference counter in shinfo is one. It meansthere is
Quoted 1 line
* no more reference to this pinned buffer and mbuf can be returned
to
Quoted 3 lines
* the pool
*
* - otherwise (if reference counter is not one), decrementreference
Quoted 1 line
* counter and return non-zero value to prevent freeing the backing
mbuf.
Quoted 4 lines
*
* Returns non zero if mbuf should not be freed.
*/
static inline int __rte_pktmbuf_pinned_extbuf_decref(structrte_mbuf*m)
Quoted 24 lines
{ struct rte_mbuf_ext_shared_info *shinfo; /* Clear flags, mbuf is being freed. */ m->ol_flags = RTE_MBUF_F_EXTERNAL; shinfo = m->shinfo; /* Optimize for performance - do not dec/reinit */ if (likely(rte_mbuf_ext_refcnt_read(shinfo) == 1)) return 0; /* * Direct usage of add primitive to avoid * duplication of comparing with one. */ if (likely(rte_atomic_fetch_add_explicit(&shinfo->refcnt, -1, rte_memory_order_acq_rel) - 1)) return 1; /* Reinitialize counter before mbuf freeing. */ rte_mbuf_ext_refcnt_set(shinfo, 1); return 0; }Quoted 2 lines
Essentially, if the mbuf does use a pinned external buffer,
rte_pktmbuf_prefree_seg()only succeeds if that pinned externalbuffer is
Quoted 3 lines
only referred to by the mbuf.
REQUIREMENT 4: If the mbuf uses a pinned external buffer, the mbuf
must
Quoted 1 line
hold the only reference to that pinned external buffer, i.e. in that
case,
Quoted 6 lines
'
m->shinfo->refcnt' must be 1.Please review.
If I'm not mistaken, the mbuf library is not a barrier for fast-
freeing
Quoted 1 line
segmented packet mbufs, and thus fast-free of jumbo frames is
possible.
Quoted 4 lines
We need a driver developer to confirm that my suggested approach - resetting the mbuf fields, incl. '
m->nb_segs' and 'm->next', when preparing the Tx descriptor - is viable.Quoted 7 lines
Great analysis, makes a lot of sense to me.
Shall we add then a special API to make PMD maintainers life a bit easier:
Something likerte_mbuf_fast_free_prep(mp, mb), that will optionally check
that requirements outlined above are satisfied for given mbuf and also reset mbuf fields to expected values?Good idea, Konstantin.
Detailed suggestion below.
Note that__rte_mbuf_raw_sanity_check_mp()is used to checks the requirements after 'nb_segs' and 'next' have been initialized./**
* Reinitialize an mbuf for freeing back into the mempool.
*
* The caller must ensure that the mbuf comes from the specified mempool,
* is direct and only referred to by the caller (refcnt=1).
*
* This function is used by drivers in their transmit function for mbuf fast release
* when the transmit descriptor is initialized,
* so the driver can callrte_mbuf_raw_free()
* when the packet segment has been transmitted.
*
* @seeRTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE
*
* @param mp
* The mempool to which the mbuf belong.
* @param m
* The mbuf being reinitialized.
*/
static__rte_always_inlinevoidrte_mbuf_raw_prefree_seg(const struct rte_mempool *mp, struct rte_mbuf *m){ if (m->nb_segs != 1) m->nb_segs = 1; if (m->next != NULL) m->next = NULL; __rte_mbuf_raw_sanity_check_mp(m, mp); rte_mbuf_history_mark(mbuf,
Quoted 2 lines
RTE_MBUF_HISTORY_OP_LIB_PREFREE_RAW);
}
Thanks Morten, though should we really panic if condition is not met? Might be just do check first and return an error.
Quoted 43 lines
/**
* Reinitialize a bulk of mbufs for freeing back into the mempool.
*
* The caller must ensure that the mbufs come from the specified mempool,
* are direct and only referred to by the caller (refcnt=1).
*
* This function is used by drivers in their transmit function for mbuf fast release
* when the transmit descriptors are initialized,
* so the driver can callrte_mbuf_raw_free_bulk()
* when the packet segments have been transmitted.
*
* @seeRTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE
*
* @param mp
* The mempool to which the mbufs belong.
* @param mbufs
* Array of pointers to mbufs being reinitialized.
* The array must not contain NULL pointers.
* @param count
* Array size.
*/
static__rte_always_inlinevoidrte_mbuf_raw_prefree_seg_bulk(const struct rte_mempool *mp, struct rte_mbuf **mbufs, unsigned int count){ for (unsigned int idx = 0; idx < count; idx++) { struct rte_mbuf *m = mbufs[idx]; if (m->nb_segs != 1) m->nb_segs = 1; if (m->next != NULL) m->next = NULL; __rte_mbuf_raw_sanity_check_mp(m, mp); } rte_mbuf_history_mark_bulk(mbufs, count,
RTE_MBUF_HISTORY_OP_LIB_PREFREE_RAW);
}Quoted 3 lines
Konstantin
Quoted 358 lines
From: Konstantin Ananyev [mailto:[email protected]]
Sent: Friday, 19 December 2025 18.08Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Executive Summary:
My analysis shows that the mbuf library is not a barrier for
fast-
Quoted 13 lines
Quoted 13 lines
freeing
Quoted 1 line
segmented packet mbufs, and thus fast-free of jumbo frames is
possible.
Quoted 6 lines
Detailed Analysis:
The purpose of the mbuf fast-free Tx optimization is to reduce
rte_pktmbuf_free_seg()to something much simpler in the ethdevdrivers, by
Quoted 2 lines
eliminating the code path related to indirect mbufs. Optimally, we want to simplify the ethdev driver's function that
frees the
Quoted 8 lines
Quoted 8 lines
Quoted 1 line
transmitted mbufs, so it can free them directly to their mempool
without
Quoted 3 lines
accessing the mbufs themselves.
If the driver cannot access the mbuf itself, it cannot determine
which
Quoted 2 lines
mempool it belongs to.
We don't want the driver to access every mbuf being freed; but ifall
Quoted 9 lines
Quoted 9 lines
Quoted 1 line
mbufs of a Tx queue belong to the same mempool, the driver can
determine
Quoted 3 lines
which mempool by looking into just one of the mbufs.
REQUIREMENT 1: The mbufs of a Tx queue must come from the same
mempool.
Quoted 3 lines
When an mbuf is freed to its mempool, some of the fields in the
mbuf
Quoted 3 lines
Quoted 3 lines
must
Quoted 2 lines
be initialized.
So, for fast-free, this must be done by the driver's functionthat
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
prepares the Tx descriptor.
This is a requirement to the driver, not a requirement to theapplication.
Quoted 3 lines
Now, let's dig into the code for freeing an mbuf.
Note: For readability purposes, I'll cut out some code andcomments
Quoted 27 lines
Quoted 27 lines
Quoted 27 lines
unrelated to this topic.
static
__rte_always_inlinevoidrte_pktmbuf_free_seg(struct rte_mbuf *m){ m = rte_pktmbuf_prefree_seg(m); if (likely(m != NULL)) rte_mbuf_raw_free(m); }
rte_mbuf_raw_free(m)is simple, so nothing to gain there:/**
* Put mbuf back into its original mempool.
*
* The caller must ensure that the mbuf is direct and properly
* reinitialized (refcnt=1, next=NULL, nb_segs=1), as done by
*rte_pktmbuf_prefree_seg().
*/
static__rte_always_inlinevoidrte_mbuf_raw_free(struct rte_mbuf *m){ rte_mbuf_history_mark(m, RTE_MBUF_HISTORY_OP_LIB_FREE); rte_mempool_put(m->pool, m); }Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Note that the description says that the mbuf must be direct. This is not entirely accurate; the mbuf is allowed to use a
pinned
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
external buffer, if the mbuf holds the only reference to it. (Most of the mbuf library functions have this documentation
inaccuracy,
Quoted 5 lines
which should be fixed some day.)
So, the fast-free optimization really comes down to
rte_pktmbuf_prefree_seg(m), which must not return NULL.Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Let's dig into that.
/**
* Decrease reference counter and unlink a mbuf segment *
* This function does the same than a free, except that it doesnot
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
* return the segment to its pool.
* It decreases the reference counter, and if it reaches 0, it is
* detached from its parent for an indirect mbuf.
*
* @return
* - (m) if it is the last reference. It can be recycled orfreed.
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
* - (NULL) if the mbuf still has remaining references on it.
*/
static__rte_always_inlinestructrte_mbuf*rte_pktmbuf_prefree_seg(struct rte_mbuf *m){ bool refcnt_not_one; refcnt_not_one = unlikely(rte_mbuf_refcnt_read(m) != 1); if (refcnt_not_one && __rte_mbuf_refcnt_update(m, -1) != 0) return NULL; if (unlikely(!RTE_MBUF_DIRECT(m))) { rte_pktmbuf_detach(m); if (RTE_MBUF_HAS_EXTBUF(m) && RTE_MBUF_HAS_PINNED_EXTBUF(m) && __rte_pktmbuf_pinned_extbuf_decref(m)) return NULL; } if (refcnt_not_one) rte_mbuf_refcnt_set(m, 1); if (m->nb_segs != 1) m->nb_segs = 1; if (m->next != NULL) m->next = NULL;Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
return m; }This function can only succeed (i.e. return non-NULL) when
'refcnt'
Quoted 4 lines
Quoted 4 lines
is 1
Quoted 3 lines
(or reaches 0).
REQUIREMENT 2: The driver must hold the only reference to the
mbuf,
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
i.e. '
m->refcnt' must be 1.When the function succeeds, it initializes the mbuf fields as
required by
Quoted 3 lines
rte_mbuf_raw_free()before returning.Now, since the driver has exclusive access to the mbuf, it is
free to
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
initialize the '
m->next' and 'm->nb_segs' at any time. It could do that when preparing the Tx descriptor.This is very interesting, because it means that fast-free does
not
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
prohibit segmented packets!
(But the driver must have sufficient Tx descriptors for allsegments
Quoted 5 lines
Quoted 5 lines
in
Quoted 4 lines
the mbuf.)
Now, lets dig into rte_pktmbuf_prefree_seg()'s block handling
non-
Quoted 28 lines
Quoted 28 lines
direct
Quoted 27 lines
mbufs, i.e. cloned mbufs and mbufs with external buffer:
if (unlikely(!RTE_MBUF_DIRECT(m))) { rte_pktmbuf_detach(m); if (RTE_MBUF_HAS_EXTBUF(m) && RTE_MBUF_HAS_PINNED_EXTBUF(m) && __rte_pktmbuf_pinned_extbuf_decref(m)) return NULL; }Starting with
rte_pktmbuf_detach():static inline void
rte_pktmbuf_detach(struct rte_mbuf *m){ struct rte_mempool *mp = m->pool; uint32_t mbuf_size, buf_len; uint16_t priv_size; if (RTE_MBUF_HAS_EXTBUF(m)) { /* * The mbuf has the external attached buffer, * we should check the type of the memory pool where * the mbuf was allocated from to detect the pinned * external buffer. */ uint32_t flags = rte_pktmbuf_priv_flags(mp);Quoted 13 lines
Quoted 5 lines
Quoted 5 lines
if (flags & RTE_PKTMBUF_POOL_F_PINNED_EXT_BUF) { /* * The pinned external buffer should not be * detached from its backing mbuf, just exit. */Quoted 8 lines
Quoted 8 lines
return; } __rte_pktmbuf_free_extbuf(m); } else { __rte_pktmbuf_free_direct(m); } priv_size = rte_pktmbuf_priv_size(mp); mbuf_size = (uint32_t)(sizeof(struct rte_mbuf) +
priv_size);Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
buf_len = rte_pktmbuf_data_room_size(mp); m->priv_size = priv_size; m->buf_addr = (char *)m + mbuf_size; rte_mbuf_iova_set(m, rte_mempool_virt2iova(m) + mbuf_size); m->buf_len = (uint16_t)buf_len; rte_pktmbuf_reset_headroom(m); m->data_len = 0; m->ol_flags = 0; }Quoted 1 line
Quoted 1 line
Quoted 1 line
The only quick and simple code path through this function is when
the
Quoted 7 lines
Quoted 7 lines
mbuf
Quoted 6 lines
uses a pinned external buffer:
if (RTE_MBUF_HAS_EXTBUF(m)) { uint32_t flags = rte_pktmbuf_priv_flags(mp); if (flags & RTE_PKTMBUF_POOL_F_PINNED_EXT_BUF) return;Quoted 6 lines
Quoted 6 lines
Quoted 1 line
REQUIREMENT 3: The mbuf must not be cloned or use a non-pinned
external
Quoted 4 lines
buffer.
Continuing with the next part of rte_pktmbuf_prefree_seg()'s
block:
Quoted 10 lines
Quoted 10 lines
Quoted 6 lines
if (
RTE_MBUF_HAS_EXTBUF(m)&&RTE_MBUF_HAS_PINNED_EXTBUF(m) && __rte_pktmbuf_pinned_extbuf_decref(m)) return NULL;Continuing with the next part of the block in
rte_pktmbuf_prefree_seg():Quoted 3 lines
/**
* @internal Handle the packet mbufs with attached pinnedexternal
Quoted 6 lines
Quoted 6 lines
buffer
Quoted 3 lines
* on the mbuf freeing:
*
* - return zero if reference counter in shinfo is one. It meansthere is
Quoted 1 line
* no more reference to this pinned buffer and mbuf can be
returned
Quoted 6 lines
Quoted 6 lines
to
Quoted 3 lines
* the pool
*
* - otherwise (if reference counter is not one), decrementreference
Quoted 1 line
* counter and return non-zero value to prevent freeing the
backing
Quoted 5 lines
Quoted 5 lines
mbuf.
Quoted 4 lines
*
* Returns non zero if mbuf should not be freed. */
static inline int __rte_pktmbuf_pinned_extbuf_decref(struct
rte_mbufQuoted 12 lines
Quoted 12 lines
*m)
Quoted 11 lines
{ struct rte_mbuf_ext_shared_info *shinfo; /* Clear flags, mbuf is being freed. */ m->ol_flags = RTE_MBUF_F_EXTERNAL; shinfo = m->shinfo; /* Optimize for performance - do not dec/reinit */ if (likely(rte_mbuf_ext_refcnt_read(shinfo) == 1)) return 0;Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
/* * Direct usage of add primitive to avoid * duplication of comparing with one. */ if (likely(rte_atomic_fetch_add_explicit(&shinfo->refcnt, -1,
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
rte_memory_order_acq_rel) - 1)) return 1; /* Reinitialize counter before mbuf freeing. */ rte_mbuf_ext_refcnt_set(shinfo, 1); return 0; }Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
Essentially, if the mbuf does use a pinned external buffer,
rte_pktmbuf_prefree_seg()only succeeds if that pinned externalbuffer is
Quoted 3 lines
only referred to by the mbuf.
REQUIREMENT 4: If the mbuf uses a pinned external buffer, the
mbuf
Quoted 2 lines
Quoted 2 lines
must
Quoted 1 line
hold the only reference to that pinned external buffer, i.e. in
that
Quoted 4 lines
Quoted 4 lines
case,
Quoted 3 lines
'
m->shinfo->refcnt' must be 1.Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
Please review.
If I'm not mistaken, the mbuf library is not a barrier for fast-
freeing
Quoted 1 line
segmented packet mbufs, and thus fast-free of jumbo frames is
possible.
Quoted 2 lines
We need a driver developer to confirm that my suggested approach
-
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
resetting the mbuf fields, incl. '
m->nb_segs' and 'm->next', when preparing the Tx descriptor - is viable.Great analysis, makes a lot of sense to me.
Shall we add then a special API to make PMD maintainers life a bit easier:
Something likerte_mbuf_fast_free_prep(mp, mb), that willoptionally
Quoted 8 lines
Quoted 3 lines
check
that requirements outlined above are satisfied for given mbuf and also reset mbuf fields to expected values?Good idea, Konstantin.
Detailed suggestion below.
Note that__rte_mbuf_raw_sanity_check_mp()is used to checks therequirements
Quoted 6 lines
after '
nb_segs' and 'next' have been initialized./**
* Reinitialize an mbuf for freeing back into the mempool. *
* The caller must ensure that the mbuf comes from the specifiedmempool,
Quoted 3 lines
* is direct and only referred to by the caller (refcnt=1). *
* This function is used by drivers in their transmit function formbuf fast release
Quoted 5 lines
* when the transmit descriptor is initialized,
* so the driver can callrte_mbuf_raw_free()
* when the packet segment has been transmitted.
*
* @seeRTE_ETH_TX_OFFLOAD_MBUF_FAST_FREEQuoted 8 lines
*
* @param mp
* The mempool to which the mbuf belong.
* @param m
* The mbuf being reinitialized.
*/
static__rte_always_inlinevoid rte_mbuf_raw_prefree_seg(const structrte_mempool*mp, struct
rte_mbuf*m)Quoted 6 lines
{ if (m->nb_segs != 1) m->nb_segs = 1; if (m->next != NULL) m->next = NULL;
Quoted 7 lines
Quoted 4 lines
__rte_mbuf_raw_sanity_check_mp(m, mp); rte_mbuf_history_mark(mbuf,
RTE_MBUF_HISTORY_OP_LIB_PREFREE_RAW);
}Thanks Morten, though should we really panic if condition is not met? Might be just do check first and return an error.
__rte_mbuf_raw_sanity_check_mp() is a no-op unless RTE_LIBRTE_MBUF_DEBUG is enabled.
Using it everywhere in alloc/free in the mbuf library is the convention.
And if we don't do it here, the __rte_mbuf_raw_sanity_check_mp() in rte_mbuf_raw_free() will panic later instead. Better to fail early.
Quoted 47 lines
Quoted 6 lines
/**
* Reinitialize a bulk of mbufs for freeing back into the mempool. *
* The caller must ensure that the mbufs come from the specifiedmempool,
Quoted 3 lines
* are direct and only referred to by the caller (refcnt=1). *
* This function is used by drivers in their transmit function formbuf fast release
Quoted 8 lines
* when the transmit descriptors are initialized,
* so the driver can callrte_mbuf_raw_free_bulk()
* when the packet segments have been transmitted. *
* @seeRTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE*
* @param mp
* The mempool to which the mbufs belong.Quoted 8 lines
* @param mbufs
* Array of pointers to mbufs being reinitialized.
* The array must not contain NULL pointers.
* @param count
* Array size.
*/
static__rte_always_inlinevoid rte_mbuf_raw_prefree_seg_bulk(const structrte_mempool*mp, struct
rte_mbufQuoted 19 lines
**mbufs, unsigned int count)
{ for (unsigned int idx = 0; idx < count; idx++) { struct rte_mbuf *m = mbufs[idx]; if (m->nb_segs != 1) m->nb_segs = 1; if (m->next != NULL) m->next = NULL; __rte_mbuf_raw_sanity_check_mp(m, mp); } rte_mbuf_history_mark_bulk(mbufs, count,
RTE_MBUF_HISTORY_OP_LIB_PREFREE_RAW);
}Quoted 3 lines
Konstantin
Quoted 365 lines
Quoted 365 lines
From: Konstantin Ananyev [mailto:[email protected]]
Sent: Friday, 19 December 2025 18.08Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Executive Summary:
My analysis shows that the mbuf library is not a barrier for
fast-
Quoted 13 lines
Quoted 13 lines
freeing
Quoted 1 line
segmented packet mbufs, and thus fast-free of jumbo frames is
possible.
Quoted 6 lines
Detailed Analysis:
The purpose of the mbuf fast-free Tx optimization is to reduce
rte_pktmbuf_free_seg()to something much simpler in the ethdevdrivers, by
Quoted 2 lines
eliminating the code path related to indirect mbufs. Optimally, we want to simplify the ethdev driver's function that
frees the
Quoted 8 lines
Quoted 8 lines
Quoted 1 line
transmitted mbufs, so it can free them directly to their mempool
without
Quoted 3 lines
accessing the mbufs themselves.
If the driver cannot access the mbuf itself, it cannot determine
which
Quoted 2 lines
mempool it belongs to.
We don't want the driver to access every mbuf being freed; but ifall
Quoted 9 lines
Quoted 9 lines
Quoted 1 line
mbufs of a Tx queue belong to the same mempool, the driver can
determine
Quoted 3 lines
which mempool by looking into just one of the mbufs.
REQUIREMENT 1: The mbufs of a Tx queue must come from the same
mempool.
Quoted 3 lines
When an mbuf is freed to its mempool, some of the fields in the
mbuf
Quoted 3 lines
Quoted 3 lines
must
Quoted 2 lines
be initialized.
So, for fast-free, this must be done by the driver's functionthat
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
prepares the Tx descriptor.
This is a requirement to the driver, not a requirement to theapplication.
Quoted 3 lines
Now, let's dig into the code for freeing an mbuf.
Note: For readability purposes, I'll cut out some code andcomments
Quoted 27 lines
Quoted 27 lines
Quoted 27 lines
unrelated to this topic.
static
__rte_always_inlinevoidrte_pktmbuf_free_seg(struct rte_mbuf *m){ m = rte_pktmbuf_prefree_seg(m); if (likely(m != NULL)) rte_mbuf_raw_free(m); }
rte_mbuf_raw_free(m)is simple, so nothing to gain there:/**
* Put mbuf back into its original mempool.
*
* The caller must ensure that the mbuf is direct and properly
* reinitialized (refcnt=1, next=NULL, nb_segs=1), as done by
*rte_pktmbuf_prefree_seg().
*/
static__rte_always_inlinevoidrte_mbuf_raw_free(struct rte_mbuf *m){ rte_mbuf_history_mark(m, RTE_MBUF_HISTORY_OP_LIB_FREE); rte_mempool_put(m->pool, m); }Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Note that the description says that the mbuf must be direct. This is not entirely accurate; the mbuf is allowed to use a
pinned
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
external buffer, if the mbuf holds the only reference to it. (Most of the mbuf library functions have this documentation
inaccuracy,
Quoted 5 lines
which should be fixed some day.)
So, the fast-free optimization really comes down to
rte_pktmbuf_prefree_seg(m), which must not return NULL.Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Let's dig into that.
/**
* Decrease reference counter and unlink a mbuf segment *
* This function does the same than a free, except that it doesnot
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
* return the segment to its pool.
* It decreases the reference counter, and if it reaches 0, it is
* detached from its parent for an indirect mbuf.
*
* @return
* - (m) if it is the last reference. It can be recycled orfreed.
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
* - (NULL) if the mbuf still has remaining references on it.
*/
static__rte_always_inlinestructrte_mbuf*rte_pktmbuf_prefree_seg(struct rte_mbuf *m){ bool refcnt_not_one; refcnt_not_one = unlikely(rte_mbuf_refcnt_read(m) != 1); if (refcnt_not_one && __rte_mbuf_refcnt_update(m, -1) != 0) return NULL; if (unlikely(!RTE_MBUF_DIRECT(m))) { rte_pktmbuf_detach(m); if (RTE_MBUF_HAS_EXTBUF(m) && RTE_MBUF_HAS_PINNED_EXTBUF(m) && __rte_pktmbuf_pinned_extbuf_decref(m)) return NULL; } if (refcnt_not_one) rte_mbuf_refcnt_set(m, 1); if (m->nb_segs != 1) m->nb_segs = 1; if (m->next != NULL) m->next = NULL;Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
return m; }This function can only succeed (i.e. return non-NULL) when
'refcnt'
Quoted 4 lines
Quoted 4 lines
is 1
Quoted 3 lines
(or reaches 0).
REQUIREMENT 2: The driver must hold the only reference to the
mbuf,
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
i.e. '
m->refcnt' must be 1.When the function succeeds, it initializes the mbuf fields as
required by
Quoted 3 lines
rte_mbuf_raw_free()before returning.Now, since the driver has exclusive access to the mbuf, it is
free to
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
initialize the '
m->next' and 'm->nb_segs' at any time. It could do that when preparing the Tx descriptor.This is very interesting, because it means that fast-free does
not
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
prohibit segmented packets!
(But the driver must have sufficient Tx descriptors for allsegments
Quoted 5 lines
Quoted 5 lines
in
Quoted 4 lines
the mbuf.)
Now, lets dig into rte_pktmbuf_prefree_seg()'s block handling
non-
Quoted 28 lines
Quoted 28 lines
direct
Quoted 27 lines
mbufs, i.e. cloned mbufs and mbufs with external buffer:
if (unlikely(!RTE_MBUF_DIRECT(m))) { rte_pktmbuf_detach(m); if (RTE_MBUF_HAS_EXTBUF(m) && RTE_MBUF_HAS_PINNED_EXTBUF(m) && __rte_pktmbuf_pinned_extbuf_decref(m)) return NULL; }Starting with
rte_pktmbuf_detach():static inline void
rte_pktmbuf_detach(struct rte_mbuf *m){ struct rte_mempool *mp = m->pool; uint32_t mbuf_size, buf_len; uint16_t priv_size; if (RTE_MBUF_HAS_EXTBUF(m)) { /* * The mbuf has the external attached buffer, * we should check the type of the memory pool where * the mbuf was allocated from to detect the pinned * external buffer. */ uint32_t flags = rte_pktmbuf_priv_flags(mp);Quoted 13 lines
Quoted 5 lines
Quoted 5 lines
if (flags & RTE_PKTMBUF_POOL_F_PINNED_EXT_BUF) { /* * The pinned external buffer should not be * detached from its backing mbuf, just exit. */Quoted 8 lines
Quoted 8 lines
return; } __rte_pktmbuf_free_extbuf(m); } else { __rte_pktmbuf_free_direct(m); } priv_size = rte_pktmbuf_priv_size(mp); mbuf_size = (uint32_t)(sizeof(struct rte_mbuf) +
priv_size);Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
buf_len = rte_pktmbuf_data_room_size(mp); m->priv_size = priv_size; m->buf_addr = (char *)m + mbuf_size; rte_mbuf_iova_set(m, rte_mempool_virt2iova(m) + mbuf_size); m->buf_len = (uint16_t)buf_len; rte_pktmbuf_reset_headroom(m); m->data_len = 0; m->ol_flags = 0; }Quoted 1 line
Quoted 1 line
Quoted 1 line
The only quick and simple code path through this function is when
the
Quoted 7 lines
Quoted 7 lines
mbuf
Quoted 6 lines
uses a pinned external buffer:
if (RTE_MBUF_HAS_EXTBUF(m)) { uint32_t flags = rte_pktmbuf_priv_flags(mp); if (flags & RTE_PKTMBUF_POOL_F_PINNED_EXT_BUF) return;Quoted 6 lines
Quoted 6 lines
Quoted 1 line
REQUIREMENT 3: The mbuf must not be cloned or use a non-pinned
external
Quoted 4 lines
buffer.
Continuing with the next part of rte_pktmbuf_prefree_seg()'s
block:
Quoted 10 lines
Quoted 10 lines
Quoted 6 lines
if (
RTE_MBUF_HAS_EXTBUF(m)&&RTE_MBUF_HAS_PINNED_EXTBUF(m) && __rte_pktmbuf_pinned_extbuf_decref(m)) return NULL;Continuing with the next part of the block in
rte_pktmbuf_prefree_seg():Quoted 3 lines
/**
* @internal Handle the packet mbufs with attached pinnedexternal
Quoted 6 lines
Quoted 6 lines
buffer
Quoted 3 lines
* on the mbuf freeing:
*
* - return zero if reference counter in shinfo is one. It meansthere is
Quoted 1 line
* no more reference to this pinned buffer and mbuf can be
returned
Quoted 6 lines
Quoted 6 lines
to
Quoted 3 lines
* the pool
*
* - otherwise (if reference counter is not one), decrementreference
Quoted 1 line
* counter and return non-zero value to prevent freeing the
backing
Quoted 5 lines
Quoted 5 lines
mbuf.
Quoted 4 lines
*
* Returns non zero if mbuf should not be freed. */
static inline int __rte_pktmbuf_pinned_extbuf_decref(struct
rte_mbufQuoted 12 lines
Quoted 12 lines
*m)
Quoted 11 lines
{ struct rte_mbuf_ext_shared_info *shinfo; /* Clear flags, mbuf is being freed. */ m->ol_flags = RTE_MBUF_F_EXTERNAL; shinfo = m->shinfo; /* Optimize for performance - do not dec/reinit */ if (likely(rte_mbuf_ext_refcnt_read(shinfo) == 1)) return 0;Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
/* * Direct usage of add primitive to avoid * duplication of comparing with one. */ if (likely(rte_atomic_fetch_add_explicit(&shinfo->refcnt, -1,
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
rte_memory_order_acq_rel) - 1)) return 1; /* Reinitialize counter before mbuf freeing. */ rte_mbuf_ext_refcnt_set(shinfo, 1); return 0; }Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
Essentially, if the mbuf does use a pinned external buffer,
rte_pktmbuf_prefree_seg()only succeeds if that pinned externalbuffer is
Quoted 3 lines
only referred to by the mbuf.
REQUIREMENT 4: If the mbuf uses a pinned external buffer, the
mbuf
Quoted 2 lines
Quoted 2 lines
must
Quoted 1 line
hold the only reference to that pinned external buffer, i.e. in
that
Quoted 4 lines
Quoted 4 lines
case,
Quoted 3 lines
'
m->shinfo->refcnt' must be 1.Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
Please review.
If I'm not mistaken, the mbuf library is not a barrier for fast-
freeing
Quoted 1 line
segmented packet mbufs, and thus fast-free of jumbo frames is
possible.
Quoted 2 lines
We need a driver developer to confirm that my suggested approach
-
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
resetting the mbuf fields, incl. '
m->nb_segs' and 'm->next', when preparing the Tx descriptor - is viable.Great analysis, makes a lot of sense to me.
Shall we add then a special API to make PMD maintainers life a bit easier:
Something likerte_mbuf_fast_free_prep(mp, mb), that willoptionally
Quoted 8 lines
Quoted 3 lines
check
that requirements outlined above are satisfied for given mbuf and also reset mbuf fields to expected values?Good idea, Konstantin.
Detailed suggestion below.
Note that__rte_mbuf_raw_sanity_check_mp()is used to checks therequirements
Quoted 6 lines
after '
nb_segs' and 'next' have been initialized./**
* Reinitialize an mbuf for freeing back into the mempool. *
* The caller must ensure that the mbuf comes from the specifiedmempool,
Quoted 3 lines
* is direct and only referred to by the caller (refcnt=1). *
* This function is used by drivers in their transmit function formbuf fast release
Quoted 5 lines
* when the transmit descriptor is initialized,
* so the driver can callrte_mbuf_raw_free()
* when the packet segment has been transmitted.
*
* @seeRTE_ETH_TX_OFFLOAD_MBUF_FAST_FREEQuoted 8 lines
*
* @param mp
* The mempool to which the mbuf belong.
* @param m
* The mbuf being reinitialized.
*/
static__rte_always_inlinevoid rte_mbuf_raw_prefree_seg(const structrte_mempool*mp, struct
rte_mbuf*m)Quoted 10 lines
{ if (m->nb_segs != 1) m->nb_segs = 1; if (m->next != NULL) m->next = NULL; __rte_mbuf_raw_sanity_check_mp(m, mp); rte_mbuf_history_mark(mbuf,
RTE_MBUF_HISTORY_OP_LIB_PREFREE_RAW);
}
Quoted 5 lines
Quoted 2 lines
Thanks Morten, though should we really panic if condition is not met? Might be just do check first and return an error.
__rte_mbuf_raw_sanity_check_mp()is a no-op unlessRTE_LIBRTE_MBUF_DEBUGis enabled.
Yep, I noticed.
Quoted 5 lines
Using it everywhere in alloc/free in the mbuf library is the convention.
And if we don't do it here, the
__rte_mbuf_raw_sanity_check_mp()inrte_mbuf_raw_free()will panic later instead.
Why?
This new routine (rte_mbuf_raw_prefree_seg) can check that mbuf satisfies fast-free criteria before updating it, and if conditions are not met simply return an error. Then the driver has several options:
1) Drop the packet silently
2) Refuse to send it
3) Switch to some slower but always working code-path (without fast-free)
4) Panic
Quoted 49 lines
Better to fail early.
Quoted 47 lines
Quoted 6 lines
/**
* Reinitialize a bulk of mbufs for freeing back into the mempool. *
* The caller must ensure that the mbufs come from the specifiedmempool,
Quoted 3 lines
* are direct and only referred to by the caller (refcnt=1). *
* This function is used by drivers in their transmit function formbuf fast release
Quoted 8 lines
* when the transmit descriptors are initialized,
* so the driver can callrte_mbuf_raw_free_bulk()
* when the packet segments have been transmitted. *
* @seeRTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE*
* @param mp
* The mempool to which the mbufs belong.Quoted 8 lines
* @param mbufs
* Array of pointers to mbufs being reinitialized.
* The array must not contain NULL pointers.
* @param count
* Array size.
*/
static__rte_always_inlinevoid rte_mbuf_raw_prefree_seg_bulk(const structrte_mempool*mp, struct
rte_mbufQuoted 19 lines
**mbufs, unsigned int count)
{ for (unsigned int idx = 0; idx < count; idx++) { struct rte_mbuf *m = mbufs[idx]; if (m->nb_segs != 1) m->nb_segs = 1; if (m->next != NULL) m->next = NULL; __rte_mbuf_raw_sanity_check_mp(m, mp); } rte_mbuf_history_mark_bulk(mbufs, count,
RTE_MBUF_HISTORY_OP_LIB_PREFREE_RAW);
}Quoted 3 lines
Konstantin
Quoted 357 lines
From: Konstantin Ananyev [mailto:[email protected]]
Sent: Monday, 22 December 2025 16.22Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Executive Summary:
My analysis shows that the mbuf library is not a barrier for
fast-
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
freeing
Quoted 1 line
segmented packet mbufs, and thus fast-free of jumbo frames is
possible.
Quoted 5 lines
Detailed Analysis:
The purpose of the mbuf fast-free Tx optimization is to
reduce
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
rte_pktmbuf_free_seg()to something much simpler in theethdev
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
drivers, by
Quoted 2 lines
eliminating the code path related to indirect mbufs. Optimally, we want to simplify the ethdev driver's function
that
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
frees the
Quoted 1 line
transmitted mbufs, so it can free them directly to their
mempool
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
without
Quoted 3 lines
accessing the mbufs themselves.
If the driver cannot access the mbuf itself, it cannot
determine
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
which
Quoted 2 lines
mempool it belongs to.
We don't want the driver to access every mbuf being freed;but if
Quoted 2 lines
Quoted 2 lines
all
Quoted 1 line
Quoted 1 line
Quoted 1 line
mbufs of a Tx queue belong to the same mempool, the driver
can
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
determine
Quoted 3 lines
which mempool by looking into just one of the mbufs.
REQUIREMENT 1: The mbufs of a Tx queue must come from the
same
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
mempool.
Quoted 3 lines
When an mbuf is freed to its mempool, some of the fields in
the
Quoted 23 lines
Quoted 23 lines
mbuf
Quoted 3 lines
Quoted 3 lines
must
Quoted 2 lines
be initialized.
So, for fast-free, this must be done by the driver's functionthat
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
prepares the Tx descriptor.
This is a requirement to the driver, not a requirement to theapplication.
Quoted 3 lines
Now, let's dig into the code for freeing an mbuf.
Note: For readability purposes, I'll cut out some code andcomments
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
unrelated to this topic.
static
__rte_always_inlinevoidrte_pktmbuf_free_seg(struct rte_mbuf *m){ m = rte_pktmbuf_prefree_seg(m); if (likely(m != NULL)) rte_mbuf_raw_free(m); }Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
rte_mbuf_raw_free(m)is simple, so nothing to gain there:/**
* Put mbuf back into its original mempool.
*
* The caller must ensure that the mbuf is direct andproperly
Quoted 21 lines
Quoted 21 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
* reinitialized (refcnt=1, next=NULL, nb_segs=1), as done by
*rte_pktmbuf_prefree_seg().
*/
static__rte_always_inlinevoidrte_mbuf_raw_free(struct rte_mbuf *m){ rte_mbuf_history_mark(m, RTE_MBUF_HISTORY_OP_LIB_FREE); rte_mempool_put(m->pool, m); }Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Note that the description says that the mbuf must be direct. This is not entirely accurate; the mbuf is allowed to use a
pinned
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
external buffer, if the mbuf holds the only reference to it. (Most of the mbuf library functions have this documentation
inaccuracy,
Quoted 5 lines
which should be fixed some day.)
So, the fast-free optimization really comes down to
rte_pktmbuf_prefree_seg(m), which must not return NULL.Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Let's dig into that.
/**
* Decrease reference counter and unlink a mbuf segment *
* This function does the same than a free, except that itdoes
Quoted 3 lines
Quoted 3 lines
not
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
* return the segment to its pool.
* It decreases the reference counter, and if it reaches 0,it is
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
* detached from its parent for an indirect mbuf. *
* @return
* - (m) if it is the last reference. It can be recycled orfreed.
Quoted 1 line
Quoted 1 line
Quoted 1 line
* - (NULL) if the mbuf still has remaining references on
it.
Quoted 40 lines
Quoted 40 lines
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
*/
static__rte_always_inlinestructrte_mbuf*rte_pktmbuf_prefree_seg(struct rte_mbuf *m){ bool refcnt_not_one; refcnt_not_one = unlikely(rte_mbuf_refcnt_read(m) != 1); if (refcnt_not_one && __rte_mbuf_refcnt_update(m, -1) != 0) return NULL; if (unlikely(!RTE_MBUF_DIRECT(m))) { rte_pktmbuf_detach(m); if (RTE_MBUF_HAS_EXTBUF(m) && RTE_MBUF_HAS_PINNED_EXTBUF(m) && __rte_pktmbuf_pinned_extbuf_decref(m)) return NULL; } if (refcnt_not_one) rte_mbuf_refcnt_set(m, 1); if (m->nb_segs != 1) m->nb_segs = 1; if (m->next != NULL) m->next = NULL;Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
return m; }This function can only succeed (i.e. return non-NULL) when
'refcnt'
Quoted 4 lines
Quoted 4 lines
is 1
Quoted 3 lines
(or reaches 0).
REQUIREMENT 2: The driver must hold the only reference to the
mbuf,
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
i.e. '
m->refcnt' must be 1.When the function succeeds, it initializes the mbuf fields as
required by
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
rte_mbuf_raw_free()before returning.Now, since the driver has exclusive access to the mbuf, it is
free to
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
initialize the '
m->next' and 'm->nb_segs' at any time. It could do that when preparing the Tx descriptor.This is very interesting, because it means that fast-free
does
Quoted 63 lines
Quoted 63 lines
not
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
prohibit segmented packets!
(But the driver must have sufficient Tx descriptors for allsegments
Quoted 5 lines
Quoted 5 lines
in
Quoted 4 lines
the mbuf.)
Now, lets dig into rte_pktmbuf_prefree_seg()'s block handling
non-
Quoted 28 lines
Quoted 28 lines
direct
Quoted 27 lines
mbufs, i.e. cloned mbufs and mbufs with external buffer:
if (unlikely(!RTE_MBUF_DIRECT(m))) { rte_pktmbuf_detach(m); if (RTE_MBUF_HAS_EXTBUF(m) && RTE_MBUF_HAS_PINNED_EXTBUF(m) && __rte_pktmbuf_pinned_extbuf_decref(m)) return NULL; }Starting with
rte_pktmbuf_detach():static inline void
rte_pktmbuf_detach(struct rte_mbuf *m){ struct rte_mempool *mp = m->pool; uint32_t mbuf_size, buf_len; uint16_t priv_size; if (RTE_MBUF_HAS_EXTBUF(m)) { /* * The mbuf has the external attached buffer, * we should check the type of the memory pool where * the mbuf was allocated from to detect the pinned * external buffer. */ uint32_t flags = rte_pktmbuf_priv_flags(mp);Quoted 13 lines
Quoted 5 lines
Quoted 5 lines
if (flags & RTE_PKTMBUF_POOL_F_PINNED_EXT_BUF) { /* * The pinned external buffer should not be * detached from its backing mbuf, just exit. */Quoted 8 lines
Quoted 8 lines
return; } __rte_pktmbuf_free_extbuf(m); } else { __rte_pktmbuf_free_direct(m); } priv_size = rte_pktmbuf_priv_size(mp); mbuf_size = (uint32_t)(sizeof(struct rte_mbuf) +
priv_size);Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
buf_len = rte_pktmbuf_data_room_size(mp); m->priv_size = priv_size; m->buf_addr = (char *)m + mbuf_size; rte_mbuf_iova_set(m, rte_mempool_virt2iova(m) + mbuf_size); m->buf_len = (uint16_t)buf_len; rte_pktmbuf_reset_headroom(m); m->data_len = 0; m->ol_flags = 0; }Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
The only quick and simple code path through this function is
when
Quoted 9 lines
Quoted 9 lines
the
Quoted 8 lines
Quoted 8 lines
mbuf
Quoted 7 lines
uses a pinned external buffer:
if (RTE_MBUF_HAS_EXTBUF(m)) { uint32_t flags = rte_pktmbuf_priv_flags(mp); if (flags & RTE_PKTMBUF_POOL_F_PINNED_EXT_BUF) return;REQUIREMENT 3: The mbuf must not be cloned or use a non-
pinned
Quoted 14 lines
Quoted 14 lines
Quoted 5 lines
Quoted 5 lines
external
Quoted 4 lines
buffer.
Continuing with the next part of rte_pktmbuf_prefree_seg()'s
block:
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
if (
RTE_MBUF_HAS_EXTBUF(m)&&RTE_MBUF_HAS_PINNED_EXTBUF(m) && __rte_pktmbuf_pinned_extbuf_decref(m)) return NULL;Continuing with the next part of the block in
rte_pktmbuf_prefree_seg():Quoted 1 line
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
/**
* @internal Handle the packet mbufs with attached pinnedexternal
Quoted 4 lines
Quoted 4 lines
buffer
Quoted 3 lines
* on the mbuf freeing:
*
* - return zero if reference counter in shinfo is one. Itmeans
Quoted 37 lines
Quoted 37 lines
Quoted 2 lines
Quoted 2 lines
there is
Quoted 1 line
* no more reference to this pinned buffer and mbuf can be
returned
Quoted 6 lines
Quoted 6 lines
to
Quoted 3 lines
* the pool
*
* - otherwise (if reference counter is not one), decrementreference
Quoted 1 line
* counter and return non-zero value to prevent freeing the
backing
Quoted 5 lines
Quoted 5 lines
mbuf.
Quoted 4 lines
*
* Returns non zero if mbuf should not be freed. */
static inline int __rte_pktmbuf_pinned_extbuf_decref(struct
rte_mbufQuoted 12 lines
Quoted 12 lines
*m)
Quoted 11 lines
{ struct rte_mbuf_ext_shared_info *shinfo; /* Clear flags, mbuf is being freed. */ m->ol_flags = RTE_MBUF_F_EXTERNAL; shinfo = m->shinfo; /* Optimize for performance - do not dec/reinit */ if (likely(rte_mbuf_ext_refcnt_read(shinfo) == 1)) return 0;Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
/* * Direct usage of add primitive to avoid * duplication of comparing with one. */ if (likely(rte_atomic_fetch_add_explicit(&shinfo->refcnt, -1,
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
rte_memory_order_acq_rel) - 1)) return 1;Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
/* Reinitialize counter before mbuf freeing. */ rte_mbuf_ext_refcnt_set(shinfo, 1); return 0; }Essentially, if the mbuf does use a pinned external buffer,
rte_pktmbuf_prefree_seg()only succeeds if that pinnedexternal
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
buffer is
Quoted 3 lines
only referred to by the mbuf.
REQUIREMENT 4: If the mbuf uses a pinned external buffer, the
mbuf
Quoted 2 lines
Quoted 2 lines
must
Quoted 1 line
hold the only reference to that pinned external buffer, i.e.
in
Quoted 8 lines
Quoted 8 lines
that
Quoted 7 lines
Quoted 7 lines
case,
Quoted 6 lines
'
m->shinfo->refcnt' must be 1.Please review.
If I'm not mistaken, the mbuf library is not a barrier for
fast-
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
freeing
Quoted 1 line
segmented packet mbufs, and thus fast-free of jumbo frames is
possible.
Quoted 2 lines
We need a driver developer to confirm that my suggested
approach
Quoted 2 lines
Quoted 2 lines
-
Quoted 1 line
Quoted 1 line
Quoted 1 line
resetting the mbuf fields, incl. '
m->nb_segs' and 'm->next',when
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
preparing the Tx descriptor - is viable.
Great analysis, makes a lot of sense to me.
Shall we add then a special API to make PMD maintainers life abit
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
easier:
Something likerte_mbuf_fast_free_prep(mp, mb), that willoptionally
Quoted 2 lines
Quoted 2 lines
check
that requirements outlined above are satisfied for given mbufand
Quoted 13 lines
Quoted 13 lines
Quoted 6 lines
Quoted 1 line
also reset mbuf fields to expected values?
Good idea, Konstantin.
Detailed suggestion below.
Note that__rte_mbuf_raw_sanity_check_mp()is used to checks therequirements
Quoted 6 lines
after '
nb_segs' and 'next' have been initialized./**
* Reinitialize an mbuf for freeing back into the mempool. *
* The caller must ensure that the mbuf comes from the specified
Quoted 25 lines
Quoted 4 lines
Quoted 4 lines
mempool,
Quoted 3 lines
* is direct and only referred to by the caller (refcnt=1). *
* This function is used by drivers in their transmit functionfor
Quoted 20 lines
Quoted 20 lines
mbuf fast release
Quoted 5 lines
* when the transmit descriptor is initialized,
* so the driver can callrte_mbuf_raw_free()
* when the packet segment has been transmitted.
*
* @seeRTE_ETH_TX_OFFLOAD_MBUF_FAST_FREEQuoted 8 lines
*
* @param mp
* The mempool to which the mbuf belong.
* @param m
* The mbuf being reinitialized.
*/
static__rte_always_inlinevoid rte_mbuf_raw_prefree_seg(const structrte_mempool*mp, struct
rte_mbuf*m)Quoted 5 lines
{ if (m->nb_segs != 1) m->nb_segs = 1; if (m->next != NULL) m->next = NULL;
A comment could mention that when the driver calls this function, it has just loaded the mbuf to initialize the transmit descriptor, so m->nb_segs and m->next are already hot in the cache. And no store operations on the mbuf itself will occur for non-segmented packets.
Quoted 22 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
__rte_mbuf_raw_sanity_check_mp(m, mp); rte_mbuf_history_mark(mbuf,
RTE_MBUF_HISTORY_OP_LIB_PREFREE_RAW);
}Thanks Morten, though should we really panic if condition is not
met?
Quoted 4 lines
Quoted 1 line
Might be just do check first and return an error.
__rte_mbuf_raw_sanity_check_mp()is a no-op unlessRTE_LIBRTE_MBUF_DEBUGis enabled.Yep, I noticed.
Quoted 2 lines
Using it everywhere in alloc/free in the mbuf library is the
convention.
Quoted 3 lines
And if we don't do it here, the
__rte_mbuf_raw_sanity_check_mp()inrte_mbuf_raw_free()will panic later instead.
Quoted 11 lines
Why?
This new routine (rte_mbuf_raw_prefree_seg) can check that mbuf satisfies fast-free criteria
before updating it, and if conditions are not met simply return an error.
Then the driver has several options:
1) Drop the packet silently
2) Refuse to send it
3) Switch to some slower but always working code-path (without fast- free)
4) Panic
It boils down to purpose.
The function is designed for use in the transmit code path designated for fast-free, where the application has promised/hinted that packets are fast-free compliant. Violating preconditions in the fast path (by passing packets not compliant to fast-free requirements to this function) is a serious type of bug, for which DPDK usually doesn't provide graceful fallback. I don't want to make an exception (and introduce graceful fallback) for the designated fast-free code path.
My answer would be completely different if we were designing an API for standard packet transmission, whereby some packets living up to certain criteria could take a faster code path for being freed. If that was the case, I would agree with you about returning a value to indicate how to proceed, like rte_pktmbuf_prefree_seg() does.
Quoted 53 lines
Quoted 6 lines
Better to fail early.
Quoted 4 lines
Quoted 4 lines
/**
* Reinitialize a bulk of mbufs for freeing back into themempool.
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
*
* The caller must ensure that the mbufs come from the specifiedmempool,
Quoted 3 lines
* are direct and only referred to by the caller (refcnt=1). *
* This function is used by drivers in their transmit functionfor
Quoted 17 lines
Quoted 17 lines
mbuf fast release
Quoted 16 lines
* when the transmit descriptors are initialized,
* so the driver can callrte_mbuf_raw_free_bulk()
* when the packet segments have been transmitted. *
* @seeRTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE
*
* @param mp
* The mempool to which the mbufs belong.
* @param mbufs
* Array of pointers to mbufs being reinitialized.
* The array must not contain NULL pointers.
* @param count
* Array size.
*/
static__rte_always_inlinevoid rte_mbuf_raw_prefree_seg_bulk(const structrte_mempool*mp,struct
Quoted 21 lines
Quoted 20 lines
rte_mbufQuoted 19 lines
**mbufs, unsigned int count)
{ for (unsigned int idx = 0; idx < count; idx++) { struct rte_mbuf *m = mbufs[idx]; if (m->nb_segs != 1) m->nb_segs = 1; if (m->next != NULL) m->next = NULL; __rte_mbuf_raw_sanity_check_mp(m, mp); } rte_mbuf_history_mark_bulk(mbufs, count,
RTE_MBUF_HISTORY_OP_LIB_PREFREE_RAW);
}Quoted 3 lines
Konstantin
Quoted 3 lines · Morten Brørup
Quoted 3 lines
From: Konstantin Ananyev [mailto:[email protected]]
Sent: Monday, 22 December 2025 16.22Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
<snip>
Quoted 34 lines
Quoted 22 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
__rte_mbuf_raw_sanity_check_mp(m, mp); rte_mbuf_history_mark(mbuf,
RTE_MBUF_HISTORY_OP_LIB_PREFREE_RAW);
}Thanks Morten, though should we really panic if condition is not
met?
Quoted 4 lines
Quoted 1 line
Might be just do check first and return an error.
__rte_mbuf_raw_sanity_check_mp()is a no-op unlessRTE_LIBRTE_MBUF_DEBUGis enabled.Yep, I noticed.
Quoted 2 lines
Using it everywhere in alloc/free in the mbuf library is the
convention.
Quoted 3 lines
And if we don't do it here, the
__rte_mbuf_raw_sanity_check_mp()inrte_mbuf_raw_free()will panic later instead.Quoted 11 lines
Why?
This new routine (rte_mbuf_raw_prefree_seg) can check that mbuf satisfies fast-free criteria
before updating it, and if conditions are not met simply return an error.
Then the driver has several options:
1) Drop the packet silently
2) Refuse to send it
3) Switch to some slower but always working code-path (without fast- free)
4) Panic
Quoted 9 lines
It boils down to purpose.
The function is designed for use in the transmit code path designated for fast-free, where the application has promised/hinted that packets are fast-free compliant. Violating preconditions in the fast path (by passing packets not compliant to fast-free requirements to this function) is a serious type of bug, for which DPDK usually doesn't provide graceful fallback. I don't want to make an exception (and introduce graceful fallback) for the designated fast-free code path.
My answer would be completely different if we were designing an API for standard packet transmission, whereby some packets living up to certain criteria could take a faster code path for being freed. If that was the case, I would agree with you about returning a value to indicate how to proceed, like
rte_pktmbuf_prefree_seg()does.
I would tend to agree. The extra handling for those cases just expands the code and adds more potential branches to the resulting binary. Lots of the fastpath code just assumes you know what you are doing, and violating constraints will lead to panics and segfaults generally. Therefore panicing in this case doesn't seem a bit deal to me.
/Bruce
Quoted 47 lines
Quoted 34 lines
Quoted 22 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
__rte_mbuf_raw_sanity_check_mp(m, mp); rte_mbuf_history_mark(mbuf,
RTE_MBUF_HISTORY_OP_LIB_PREFREE_RAW);
}Thanks Morten, though should we really panic if condition is not
met?
Quoted 4 lines
Quoted 1 line
Might be just do check first and return an error.
__rte_mbuf_raw_sanity_check_mp()is a no-op unlessRTE_LIBRTE_MBUF_DEBUGis enabled.Yep, I noticed.
Quoted 2 lines
Using it everywhere in alloc/free in the mbuf library is the
convention.
Quoted 3 lines
And if we don't do it here, the
__rte_mbuf_raw_sanity_check_mp()inrte_mbuf_raw_free()will panic later instead.Quoted 11 lines
Why?
This new routine (rte_mbuf_raw_prefree_seg) can check that mbuf satisfies fast-free criteria
before updating it, and if conditions are not met simply return an error.
Then the driver has several options:
1) Drop the packet silently
2) Refuse to send it
3) Switch to some slower but always working code-path (without fast- free)
4) PanicQuoted 3 lines
It boils down to purpose.
The function is designed for use in the transmit code path designated for fast-free,
where the application has promised/hinted that packets are fast-free compliant.
Quoted 1 line
Violating preconditions in the fast path (by passing packets not compliant to fast-
free requirements to this function) is a serious type of bug, for which DPDK usually doesn't provide graceful fallback.
Quoted 1 line
I don't want to make an exception (and introduce graceful fallback) for the
designated fast-free code path.
Quoted 2 lines
My answer would be completely different if we were designing an API for standard
packet transmission, whereby some packets living up to certain criteria could take a faster code path for being freed.
Quoted 9 lines
Quoted 1 line
If that was the case, I would agree with you about returning a value to indicate
how to proceed, like
rte_pktmbuf_prefree_seg()does.Quoted 1 line
I would tend to agree. The extra handling for those cases just expands the code and adds more potential branches to the resulting binary. Lots of the fastpath code just assumes you know what you are doing, and violating constraints will lead to panics and segfaults generally. Therefore panicing in this case doesn't seem a bit deal to me.
I understand your point lads, and somewhat agree: if we plan to keep FAST_FREE flag forever, then it is probably the simplest and safest approach.
My hope was that such function will open a possibility for the PMDs to implement similar perf improvement completely transparent to the user (without need for special flags and/or pre-requirements). But might be I am looking forward way too far with it.
Even what Morten proposed above is a big step, so ok - let's deal with it first.
Konstantin
Quoted 56 lines · On Tue, 13 Jan 2026 14:48:19 +0000 Konstantin Ananyev…
Quoted 47 lines
Quoted 34 lines
Quoted 22 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
__rte_mbuf_raw_sanity_check_mp(m, mp); rte_mbuf_history_mark(mbuf,
RTE_MBUF_HISTORY_OP_LIB_PREFREE_RAW);
}Thanks Morten, though should we really panic if condition is not
met?
Quoted 4 lines
Quoted 1 line
Might be just do check first and return an error.
__rte_mbuf_raw_sanity_check_mp()is a no-op unlessRTE_LIBRTE_MBUF_DEBUGis enabled.Yep, I noticed.
Quoted 2 lines
Using it everywhere in alloc/free in the mbuf library is the
convention.
Quoted 3 lines
And if we don't do it here, the
__rte_mbuf_raw_sanity_check_mp()inrte_mbuf_raw_free()will panic later instead.Quoted 11 lines
Why?
This new routine (rte_mbuf_raw_prefree_seg) can check that mbuf satisfies fast-free criteria
before updating it, and if conditions are not met simply return an error.
Then the driver has several options:
1) Drop the packet silently
2) Refuse to send it
3) Switch to some slower but always working code-path (without fast- free)
4) PanicQuoted 3 lines
It boils down to purpose.
The function is designed for use in the transmit code path designated for fast-free,
where the application has promised/hinted that packets are fast-free compliant.
Quoted 1 line
Violating preconditions in the fast path (by passing packets not compliant to fast-
free requirements to this function) is a serious type of bug, for which DPDK usually doesn't provide graceful fallback.
Quoted 1 line
I don't want to make an exception (and introduce graceful fallback) for the
designated fast-free code path.
Quoted 2 lines
My answer would be completely different if we were designing an API for standard
packet transmission, whereby some packets living up to certain criteria could take a faster code path for being freed.
Quoted 9 lines
Quoted 1 line
If that was the case, I would agree with you about returning a value to indicate
how to proceed, like
rte_pktmbuf_prefree_seg()does.Quoted 1 line
I would tend to agree. The extra handling for those cases just expands the code and adds more potential branches to the resulting binary. Lots of the fastpath code just assumes you know what you are doing, and violating constraints will lead to panics and segfaults generally. Therefore panicing in this case doesn't seem a bit deal to me.
Quoted 8 lines
I understand your point lads, and somewhat agree: if we plan to keep
FAST_FREEflag forever, then it is probably the simplest and safest approach.
My hope was that such function will open a possibility for the PMDs to implement similar perf improvement completely transparent to the user (without need for special flags and/or pre-requirements). But might be I am looking forward way too far with it.
Even what Morten proposed above is a big step, so ok - let's deal with it first.
Konstantin
We should document any restrictions on this and other semantic assumptions. Ideally, then AI review tools could help new and old developers see places where the assumptions are violated.
Quoted 21 lines · Morten Brørup
Executive Summary:
My analysis shows that the mbuf library is not a barrier for fast-freeing segmented packet mbufs, and thus fast-free of jumbo frames is possible.
Detailed Analysis:
The purpose of the mbuf fast-free Tx optimization is to reduce
rte_pktmbuf_free_seg()to something much simpler in the ethdev drivers, by eliminating the code path related to indirect mbufs. Optimally, we want to simplify the ethdev driver's function that frees the transmitted mbufs, so it can free them directly to their mempool without accessing the mbufs themselves.If the driver cannot access the mbuf itself, it cannot determine which mempool it belongs to.
We don't want the driver to access every mbuf being freed; but if all mbufs of a Tx queue belong to the same mempool, the driver can determine which mempool by looking into just one of the mbufs.
<snip>
Quoted 8 lines
If I'm not mistaken, the mbuf library is not a barrier for fast-freeing segmented packet mbufs, and thus fast-free of jumbo frames is possible.
We need a driver developer to confirm that my suggested approach - resetting the mbuf fields, incl. '
m->nb_segs' and 'm->next', when preparing the Tx descriptor - is viable.
Just to make sure I understand things correctly here, the suggestion to prototype is:
- When FAST_FREE flag is set:
- reset the m->nb_segs and m->next fields (if necessary) when accessing the mbuf to write the descriptor
- skip calling pre-free seg on cleanup and instead
- just free all buffers directly to the mempool
Is that correct?
/Bruce
Quoted 49 lines
From: Bruce Richardson [mailto:[email protected]]
Sent: Wednesday, 14 January 2026 18.01Quoted 3 lines · Morten Brørup
Executive Summary:
My analysis shows that the mbuf library is not a barrier for fast-
freeing
Quoted 1 line
segmented packet mbufs, and thus fast-free of jumbo frames is
possible.
Quoted 6 lines
Detailed Analysis:
The purpose of the mbuf fast-free Tx optimization is to reduce
rte_pktmbuf_free_seg()to something much simpler in the ethdevdrivers, by
Quoted 2 lines
eliminating the code path related to indirect mbufs. Optimally, we want to simplify the ethdev driver's function that
frees the
Quoted 1 line
transmitted mbufs, so it can free them directly to their mempool
without
Quoted 3 lines
accessing the mbufs themselves.
If the driver cannot access the mbuf itself, it cannot determine
which
Quoted 3 lines
mempool it belongs to.
We don't want the driver to access every mbuf being freed; but if all mbufs of a Tx queue belong to the same mempool, the driver candetermine
Quoted 2 lines
which mempool by looking into just one of the mbufs.
<snip>
Quoted 2 lines
If I'm not mistaken, the mbuf library is not a barrier for fast-
freeing
Quoted 1 line
segmented packet mbufs, and thus fast-free of jumbo frames is
possible.
Quoted 5 lines
We need a driver developer to confirm that my suggested approach - resetting the mbuf fields, incl. '
m->nb_segs' and 'm->next', when preparing the Tx descriptor - is viable.Just to make sure I understand things correctly here, the suggestion to prototype is:
Quoted 8 lines
- When
FAST_FREEflag is set:
- reset them->nb_segsandm->nextfields (if necessary) when
accessing
the mbuf to write the descriptor
- skip calling pre-free seg on cleanup and instead
- just free all buffers directly to the mempoolIs that correct?
Yes.
If this can be done with multi-segment packets, we should be able to eliminate the single-segment requirement to FAST_FREE. (Unless something in the code that writes the descriptor requires single-segment to be super performant, as I suspected of vectorization.)
Quoted 56 lines · Morten Brørup
Quoted 56 lines
From: Bruce Richardson [mailto:[email protected]]
Sent: Wednesday, 14 January 2026 18.01Quoted 3 lines · Morten Brørup
Executive Summary:
My analysis shows that the mbuf library is not a barrier for fast-
freeing
Quoted 1 line
segmented packet mbufs, and thus fast-free of jumbo frames is
possible.
Quoted 6 lines
Detailed Analysis:
The purpose of the mbuf fast-free Tx optimization is to reduce
rte_pktmbuf_free_seg()to something much simpler in the ethdevdrivers, by
Quoted 2 lines
eliminating the code path related to indirect mbufs. Optimally, we want to simplify the ethdev driver's function that
frees the
Quoted 1 line
transmitted mbufs, so it can free them directly to their mempool
without
Quoted 3 lines
accessing the mbufs themselves.
If the driver cannot access the mbuf itself, it cannot determine
which
Quoted 3 lines
mempool it belongs to.
We don't want the driver to access every mbuf being freed; but if all mbufs of a Tx queue belong to the same mempool, the driver candetermine
Quoted 2 lines
which mempool by looking into just one of the mbufs.
<snip>
Quoted 2 lines
If I'm not mistaken, the mbuf library is not a barrier for fast-
freeing
Quoted 1 line
segmented packet mbufs, and thus fast-free of jumbo frames is
possible.
Quoted 5 lines
We need a driver developer to confirm that my suggested approach - resetting the mbuf fields, incl. '
m->nb_segs' and 'm->next', when preparing the Tx descriptor - is viable.Just to make sure I understand things correctly here, the suggestion to prototype is:
- When
FAST_FREEflag is set:
- reset them->nb_segsandm->nextfields (if necessary) when
accessing
the mbuf to write the descriptor
- skip calling pre-free seg on cleanup and instead
- just free all buffers directly to the mempool
Quoted 6 lines
Quoted 1 line
Is that correct?
Yes.
If this can be done with multi-segment packets, we should be able to eliminate the single-segment requirement toFAST_FREE. (Unless something in the code that writes the descriptor requires single-segment to be super performant, as I suspected of vectorization.)
Yes, the vector paths do require single-segment, but there is a separate offload flag for that [1]. That flag is part of the overall vector/non-vector Tx flags check.
/Bruce
[1] https://github.com/DPDK/dpdk/blob/main/lib/ethdev/rte_ethdev.h#L1643