Skip to contentSkip to conversation

mbuf fast-free requirements analysis

PATCH Superseded Morten Brørup 5 participants 38 messages Dec 15, 2025 – Jun 3 Network drivers

lore
Share & tools
lore
#0 lore raw

Executive Summary:

My analysis shows that the mbuf library is not a barrier for fast-freeing segmented packet mbufs, and thus fast-free of jumbo frames is possible.

Detailed Analysis:

The purpose of the mbuf fast-free Tx optimization is to reduce rte_pktmbuf_free_seg() to something much simpler in the ethdev drivers, by eliminating the code path related to indirect mbufs. Optimally, we want to simplify the ethdev driver's function that frees the transmitted mbufs, so it can free them directly to their mempool without accessing the mbufs themselves.

If the driver cannot access the mbuf itself, it cannot determine which mempool it belongs to.
We don't want the driver to access every mbuf being freed; but if all mbufs of a Tx queue belong to the same mempool, the driver can determine which mempool by looking into just one of the mbufs.

REQUIREMENT 1: The mbufs of a Tx queue must come from the same mempool.

When an mbuf is freed to its mempool, some of the fields in the mbuf must be initialized.
So, for fast-free, this must be done by the driver's function that prepares the Tx descriptor.
This is a requirement to the driver, not a requirement to the application.

Now, let's dig into the code for freeing an mbuf.
Note: For readability purposes, I'll cut out some code and comments unrelated to this topic.

static __rte_always_inline void
rte_pktmbuf_free_seg(struct rte_mbuf *m)

{
	m = rte_pktmbuf_prefree_seg(m);
	if (likely(m != NULL))
		rte_mbuf_raw_free(m);
}

rte_mbuf_raw_free(m) is simple, so nothing to gain there:

/**
* Put mbuf back into its original mempool.
*
* The caller must ensure that the mbuf is direct and properly
* reinitialized (refcnt=1, next=NULL, nb_segs=1), as done by
* rte_pktmbuf_prefree_seg().
*/
static __rte_always_inline void rte_mbuf_raw_free(struct rte_mbuf *m)

{
	rte_mbuf_history_mark(m, RTE_MBUF_HISTORY_OP_LIB_FREE);
	rte_mempool_put(m->pool, m);
}

Note that the description says that the mbuf must be direct. This is not entirely accurate; the mbuf is allowed to use a pinned external buffer, if the mbuf holds the only reference to it. (Most of the mbuf library functions have this documentation inaccuracy, which should be fixed some day.)

So, the fast-free optimization really comes down to rte_pktmbuf_prefree_seg(m), which must not return NULL.

Let's dig into that.

/**
* Decrease reference counter and unlink a mbuf segment *
* This function does the same than a free, except that it does not
* return the segment to its pool.
* It decreases the reference counter, and if it reaches 0, it is
* detached from its parent for an indirect mbuf.
*
* @return
* - (m) if it is the last reference. It can be recycled or freed.
* - (NULL) if the mbuf still has remaining references on it.
*/
static __rte_always_inline struct rte_mbuf * rte_pktmbuf_prefree_seg(struct rte_mbuf *m)

{
	bool refcnt_not_one;

	refcnt_not_one = unlikely(rte_mbuf_refcnt_read(m) != 1);
	if (refcnt_not_one && __rte_mbuf_refcnt_update(m, -1) != 0)
		return NULL;

	if (unlikely(!RTE_MBUF_DIRECT(m))) {
		rte_pktmbuf_detach(m);
		if (RTE_MBUF_HAS_EXTBUF(m) &&
				RTE_MBUF_HAS_PINNED_EXTBUF(m) &&
				__rte_pktmbuf_pinned_extbuf_decref(m))
			return NULL;
	}

	if (refcnt_not_one)
		rte_mbuf_refcnt_set(m, 1);
	if (m->nb_segs != 1)
		m->nb_segs = 1;
	if (m->next != NULL)
		m->next = NULL;

	return m;
}

This function can only succeed (i.e. return non-NULL) when 'refcnt' is 1 (or reaches 0).

REQUIREMENT 2: The driver must hold the only reference to the mbuf, i.e. 'm->refcnt' must be 1.

When the function succeeds, it initializes the mbuf fields as required by rte_mbuf_raw_free() before returning.

Now, since the driver has exclusive access to the mbuf, it is free to initialize the 'm->next' and 'm->nb_segs' at any time.
It could do that when preparing the Tx descriptor.

This is very interesting, because it means that fast-free does not prohibit segmented packets!
(But the driver must have sufficient Tx descriptors for all segments in the mbuf.)

Now, lets dig into rte_pktmbuf_prefree_seg()'s block handling non-direct mbufs, i.e. cloned mbufs and mbufs with external buffer:

	if (unlikely(!RTE_MBUF_DIRECT(m))) {
		rte_pktmbuf_detach(m);
		if (RTE_MBUF_HAS_EXTBUF(m) &&
				RTE_MBUF_HAS_PINNED_EXTBUF(m) &&
				__rte_pktmbuf_pinned_extbuf_decref(m))
			return NULL;
	}

Starting with rte_pktmbuf_detach():

static inline void rte_pktmbuf_detach(struct rte_mbuf *m)

{
	struct rte_mempool *mp = m->pool;
	uint32_t mbuf_size, buf_len;
	uint16_t priv_size;

	if (RTE_MBUF_HAS_EXTBUF(m)) {
		/*
		 * The mbuf has the external attached buffer,
		 * we should check the type of the memory pool where
		 * the mbuf was allocated from to detect the pinned
		 * external buffer.
		 */
		uint32_t flags = rte_pktmbuf_priv_flags(mp);

		if (flags & RTE_PKTMBUF_POOL_F_PINNED_EXT_BUF) {
			/*
			 * The pinned external buffer should not be
			 * detached from its backing mbuf, just exit.
			 */
			return;
		}
		__rte_pktmbuf_free_extbuf(m);
	} else {
		__rte_pktmbuf_free_direct(m);
	}
	priv_size = rte_pktmbuf_priv_size(mp);
	mbuf_size = (uint32_t)(sizeof(struct rte_mbuf) + priv_size);
	buf_len = rte_pktmbuf_data_room_size(mp);

	m->priv_size = priv_size;
	m->buf_addr = (char *)m + mbuf_size;
	rte_mbuf_iova_set(m, rte_mempool_virt2iova(m) + mbuf_size);
	m->buf_len = (uint16_t)buf_len;
	rte_pktmbuf_reset_headroom(m);
	m->data_len = 0;
	m->ol_flags = 0;
}

The only quick and simple code path through this function is when the mbuf uses a pinned external buffer:

	if (RTE_MBUF_HAS_EXTBUF(m)) {
		uint32_t flags = rte_pktmbuf_priv_flags(mp);
		if (flags & RTE_PKTMBUF_POOL_F_PINNED_EXT_BUF)
			return;

REQUIREMENT 3: The mbuf must not be cloned or use a non-pinned external buffer.

Continuing with the next part of rte_pktmbuf_prefree_seg()'s block:
if (RTE_MBUF_HAS_EXTBUF(m) &&

				RTE_MBUF_HAS_PINNED_EXTBUF(m) &&
				__rte_pktmbuf_pinned_extbuf_decref(m))
			return NULL;

Continuing with the next part of the block in rte_pktmbuf_prefree_seg():

/**
* @internal Handle the packet mbufs with attached pinned external buffer
* on the mbuf freeing:
*

 *  - return zero if reference counter in shinfo is one. It means there is
 *  no more reference to this pinned buffer and mbuf can be returned to
 *  the pool

*
* - otherwise (if reference counter is not one), decrement reference
* counter and return non-zero value to prevent freeing the backing mbuf. *
* Returns non zero if mbuf should not be freed.
*/
static inline int __rte_pktmbuf_pinned_extbuf_decref(struct rte_mbuf *m)

{
	struct rte_mbuf_ext_shared_info *shinfo;

	/* Clear flags, mbuf is being freed. */
	m->ol_flags = RTE_MBUF_F_EXTERNAL;
	shinfo = m->shinfo;

	/* Optimize for performance - do not dec/reinit */
	if (likely(rte_mbuf_ext_refcnt_read(shinfo) == 1))
		return 0;

	/*
	 * Direct usage of add primitive to avoid
	 * duplication of comparing with one.
	 */
	if (likely(rte_atomic_fetch_add_explicit(&shinfo->refcnt, -1,
				     rte_memory_order_acq_rel) - 1))
		return 1;

	/* Reinitialize counter before mbuf freeing. */
	rte_mbuf_ext_refcnt_set(shinfo, 1);
	return 0;
}

Essentially, if the mbuf does use a pinned external buffer, rte_pktmbuf_prefree_seg() only succeeds if that pinned external buffer is only referred to by the mbuf.

REQUIREMENT 4: If the mbuf uses a pinned external buffer, the mbuf must hold the only reference to that pinned external buffer, i.e. in that case, 'm->shinfo->refcnt' must be 1.

Please review.

If I'm not mistaken, the mbuf library is not a barrier for fast-freeing segmented packet mbufs, and thus fast-free of jumbo frames is possible.

We need a driver developer to confirm that my suggested approach - resetting the mbuf fields, incl. 'm->nb_segs' and 'm->next', when preparing the Tx descriptor - is viable.

#1 lore raw
Quoted 246 lines · Morten Brørup

Executive Summary:

My analysis shows that the mbuf library is not a barrier for fast-freeing segmented packet mbufs, and thus fast-free of jumbo frames is possible.

Detailed Analysis:

The purpose of the mbuf fast-free Tx optimization is to reduce rte_pktmbuf_free_seg() to something much simpler in the ethdev drivers, by eliminating the code path related to indirect mbufs. Optimally, we want to simplify the ethdev driver's function that frees the transmitted mbufs, so it can free them directly to their mempool without accessing the mbufs themselves.

If the driver cannot access the mbuf itself, it cannot determine which mempool it belongs to.
We don't want the driver to access every mbuf being freed; but if all mbufs of a Tx queue belong to the same mempool, the driver can determine which mempool by looking into just one of the mbufs.

REQUIREMENT 1: The mbufs of a Tx queue must come from the same mempool.

When an mbuf is freed to its mempool, some of the fields in the mbuf must be initialized.
So, for fast-free, this must be done by the driver's function that prepares the Tx descriptor.
This is a requirement to the driver, not a requirement to the application.

Now, let's dig into the code for freeing an mbuf.
Note: For readability purposes, I'll cut out some code and comments unrelated to this topic.

static __rte_always_inline void
rte_pktmbuf_free_seg(struct rte_mbuf *m)

{
	m = rte_pktmbuf_prefree_seg(m);
	if (likely(m != NULL))
		rte_mbuf_raw_free(m);
}

rte_mbuf_raw_free(m) is simple, so nothing to gain there:

/**
* Put mbuf back into its original mempool.
*
* The caller must ensure that the mbuf is direct and properly
* reinitialized (refcnt=1, next=NULL, nb_segs=1), as done by
* rte_pktmbuf_prefree_seg().
*/
static __rte_always_inline void rte_mbuf_raw_free(struct rte_mbuf *m)

{
	rte_mbuf_history_mark(m, RTE_MBUF_HISTORY_OP_LIB_FREE);
	rte_mempool_put(m->pool, m);
}

Note that the description says that the mbuf must be direct. This is not entirely accurate; the mbuf is allowed to use a pinned external buffer, if the mbuf holds the only reference to it. (Most of the mbuf library functions have this documentation inaccuracy, which should be fixed some day.)

So, the fast-free optimization really comes down to rte_pktmbuf_prefree_seg(m), which must not return NULL.

Let's dig into that.

/**
* Decrease reference counter and unlink a mbuf segment *
* This function does the same than a free, except that it does not
* return the segment to its pool.
* It decreases the reference counter, and if it reaches 0, it is
* detached from its parent for an indirect mbuf.
*
* @return
* - (m) if it is the last reference. It can be recycled or freed.
* - (NULL) if the mbuf still has remaining references on it.
*/
static __rte_always_inline struct rte_mbuf * rte_pktmbuf_prefree_seg(struct rte_mbuf *m)

{
	bool refcnt_not_one;

	refcnt_not_one = unlikely(rte_mbuf_refcnt_read(m) != 1);
	if (refcnt_not_one && __rte_mbuf_refcnt_update(m, -1) != 0)
		return NULL;

	if (unlikely(!RTE_MBUF_DIRECT(m))) {
		rte_pktmbuf_detach(m);
		if (RTE_MBUF_HAS_EXTBUF(m) &&
				RTE_MBUF_HAS_PINNED_EXTBUF(m) &&
				__rte_pktmbuf_pinned_extbuf_decref(m))
			return NULL;
	}

	if (refcnt_not_one)
		rte_mbuf_refcnt_set(m, 1);
	if (m->nb_segs != 1)
		m->nb_segs = 1;
	if (m->next != NULL)
		m->next = NULL;

	return m;
}

This function can only succeed (i.e. return non-NULL) when 'refcnt' is 1 (or reaches 0).

REQUIREMENT 2: The driver must hold the only reference to the mbuf, i.e. 'm->refcnt' must be 1.

When the function succeeds, it initializes the mbuf fields as required by rte_mbuf_raw_free() before returning.

Now, since the driver has exclusive access to the mbuf, it is free to initialize the 'm->next' and 'm->nb_segs' at any time.
It could do that when preparing the Tx descriptor.

This is very interesting, because it means that fast-free does not prohibit segmented packets!
(But the driver must have sufficient Tx descriptors for all segments in the mbuf.)

Now, lets dig into rte_pktmbuf_prefree_seg()'s block handling non-direct mbufs, i.e. cloned mbufs and mbufs with external buffer:

	if (unlikely(!RTE_MBUF_DIRECT(m))) {
		rte_pktmbuf_detach(m);
		if (RTE_MBUF_HAS_EXTBUF(m) &&
				RTE_MBUF_HAS_PINNED_EXTBUF(m) &&
				__rte_pktmbuf_pinned_extbuf_decref(m))
			return NULL;
	}

Starting with rte_pktmbuf_detach():

static inline void rte_pktmbuf_detach(struct rte_mbuf *m)

{
	struct rte_mempool *mp = m->pool;
	uint32_t mbuf_size, buf_len;
	uint16_t priv_size;

	if (RTE_MBUF_HAS_EXTBUF(m)) {
		/*
		 * The mbuf has the external attached buffer,
		 * we should check the type of the memory pool where
		 * the mbuf was allocated from to detect the pinned
		 * external buffer.
		 */
		uint32_t flags = rte_pktmbuf_priv_flags(mp);

		if (flags & RTE_PKTMBUF_POOL_F_PINNED_EXT_BUF) {
			/*
			 * The pinned external buffer should not be
			 * detached from its backing mbuf, just exit.
			 */
			return;
		}
		__rte_pktmbuf_free_extbuf(m);
	} else {
		__rte_pktmbuf_free_direct(m);
	}
	priv_size = rte_pktmbuf_priv_size(mp);
	mbuf_size = (uint32_t)(sizeof(struct rte_mbuf) + priv_size);
	buf_len = rte_pktmbuf_data_room_size(mp);

	m->priv_size = priv_size;
	m->buf_addr = (char *)m + mbuf_size;
	rte_mbuf_iova_set(m, rte_mempool_virt2iova(m) + mbuf_size);
	m->buf_len = (uint16_t)buf_len;
	rte_pktmbuf_reset_headroom(m);
	m->data_len = 0;
	m->ol_flags = 0;
}

The only quick and simple code path through this function is when the mbuf uses a pinned external buffer:

	if (RTE_MBUF_HAS_EXTBUF(m)) {
		uint32_t flags = rte_pktmbuf_priv_flags(mp);
		if (flags & RTE_PKTMBUF_POOL_F_PINNED_EXT_BUF)
			return;

REQUIREMENT 3: The mbuf must not be cloned or use a non-pinned external buffer.

Continuing with the next part of rte_pktmbuf_prefree_seg()'s block:
if (RTE_MBUF_HAS_EXTBUF(m) &&

				RTE_MBUF_HAS_PINNED_EXTBUF(m) &&
				__rte_pktmbuf_pinned_extbuf_decref(m))
			return NULL;

Continuing with the next part of the block in rte_pktmbuf_prefree_seg():

/**
* @internal Handle the packet mbufs with attached pinned external buffer
* on the mbuf freeing:
*

 *  - return zero if reference counter in shinfo is one. It means there is
 *  no more reference to this pinned buffer and mbuf can be returned to
 *  the pool

*
* - otherwise (if reference counter is not one), decrement reference
* counter and return non-zero value to prevent freeing the backing mbuf. *
* Returns non zero if mbuf should not be freed.
*/
static inline int __rte_pktmbuf_pinned_extbuf_decref(struct rte_mbuf *m)

{
	struct rte_mbuf_ext_shared_info *shinfo;

	/* Clear flags, mbuf is being freed. */
	m->ol_flags = RTE_MBUF_F_EXTERNAL;
	shinfo = m->shinfo;

	/* Optimize for performance - do not dec/reinit */
	if (likely(rte_mbuf_ext_refcnt_read(shinfo) == 1))
		return 0;

	/*
	 * Direct usage of add primitive to avoid
	 * duplication of comparing with one.
	 */
	if (likely(rte_atomic_fetch_add_explicit(&shinfo->refcnt, -1,
				     rte_memory_order_acq_rel) - 1))
		return 1;

	/* Reinitialize counter before mbuf freeing. */
	rte_mbuf_ext_refcnt_set(shinfo, 1);
	return 0;
}

Essentially, if the mbuf does use a pinned external buffer, rte_pktmbuf_prefree_seg() only succeeds if that pinned external buffer is only referred to by the mbuf.

REQUIREMENT 4: If the mbuf uses a pinned external buffer, the mbuf must hold the only reference to that pinned external buffer, i.e. in that case, 'm->shinfo->refcnt' must be 1.

Quoted 9 lines

Please review.

If I'm not mistaken, the mbuf library is not a barrier for fast-freeing segmented packet mbufs, and thus fast-free of jumbo frames is possible.

We need a driver developer to confirm that my suggested approach - resetting the mbuf fields, incl. 'm->nb_segs' and 'm->next', when preparing the Tx descriptor - is viable.

Excellent analysis, Morten. If I get a chance some time this release cycle, I will try implementing this change in our drivers, see if any difference is made.

/Bruce

#2 lore raw
Quoted 9 lines
Quoted 1 line

If I'm not mistaken, the mbuf library is not a barrier for fast-

freeing

Quoted 1 line

segmented packet mbufs, and thus fast-free of jumbo frames is

possible.

Quoted 5 lines

We need a driver developer to confirm that my suggested approach - resetting the mbuf fields, incl. 'm->nb_segs' and 'm->next', when preparing the Tx descriptor - is viable.

Quoted 5 lines

Excellent analysis, Morten. If I get a chance some time this release cycle,
I will try implementing this change in our drivers, see if any difference
is made.

Bruce,

Have you had a chance to look into the driver change requirements? If not, could you please try scratching the surface, to build a gut feeling.

I wonder if the vector implementations have strong requirements that packets are not segmented...

The i40 driver only sets "tx_simple_allowed" and "tx_vec_allowed" flags when MBUF_FAST_FREE is set:
https://elixir.bootlin.com/dpdk/v25.11/source/drivers/net/intel/i40e/i40e_rxtx.c#L3502

And only when these two flags are set, it uses a vector Tx function:
https://elixir.bootlin.com/dpdk/v25.11/source/drivers/net/intel/i40e/i40e_rxtx.c#L3550
And a special Tx Prep function:
https://elixir.bootlin.com/dpdk/v25.11/source/drivers/net/intel/i40e/i40e_rxtx.c#L3584
Which fails if nb_segs != 1:
https://elixir.bootlin.com/dpdk/v25.11/source/drivers/net/intel/i40e/i40e_rxtx.c#L1675

So currently it does.
But does it need to?... That is the question.
Paraphrasing:
Can the Tx function only be vectorized when the code path doesn't have branches depending on the number of segments? If so, then this may be the main reason for not supporting segmented packets with FAST_FREE.

In that case, we cannot remove the single-segment requirement from FAST_FREE without sacrificing the performance boost from vectorizing.

But then we can proceed pursuing alternative optimizations, as suggested by Konstantin.

Here's another idea:
The Tx function could pre-scan each Tx burst for multi-segment packets, to decide if the burst should be processed by the vector code path or a fallback code path (which can also handle multi-segment packets).

-Morten

#3 lore raw
Quoted 15 lines · Morten Brørup
Quoted 9 lines
Quoted 1 line

If I'm not mistaken, the mbuf library is not a barrier for fast-

freeing

Quoted 1 line

segmented packet mbufs, and thus fast-free of jumbo frames is

possible.

Quoted 5 lines

We need a driver developer to confirm that my suggested approach - resetting the mbuf fields, incl. 'm->nb_segs' and 'm->next', when preparing the Tx descriptor - is viable.

Quoted 5 lines

Excellent analysis, Morten. If I get a chance some time this release cycle,
I will try implementing this change in our drivers, see if any difference
is made.

Quoted 4 lines

Bruce,

Have you had a chance to look into the driver change requirements? If not, could you please try scratching the surface, to build a gut feeling.

I'll try and take a look this week. Juggling a few things at the moment, so I had forgotten about this. Sorry.

More comments inline below.

/Bruce

Quoted 6 lines

I wonder if the vector implementations have strong requirements that packets are not segmented...

The i40 driver only sets "tx_simple_allowed" and "tx_vec_allowed" flags when MBUF_FAST_FREE is set:
https://elixir.bootlin.com/dpdk/v25.11/source/drivers/net/intel/i40e/i40e_rxtx.c#L3502

Actually, it allows but does not require FAST_FREE. The check is just verifying that the flags with everything *but* FAST_FREE masked out is the same as the original flags, i.e. FAST_FREE is just ignored.

Quoted 7 lines

And only when these two flags are set, it uses a vector Tx function:
https://elixir.bootlin.com/dpdk/v25.11/source/drivers/net/intel/i40e/i40e_rxtx.c#L3550
And a special Tx Prep function:
https://elixir.bootlin.com/dpdk/v25.11/source/drivers/net/intel/i40e/i40e_rxtx.c#L3584
Which fails if nb_segs != 1:
https://elixir.bootlin.com/dpdk/v25.11/source/drivers/net/intel/i40e/i40e_rxtx.c#L1675

Quoted 7 lines

So currently it does.
But does it need to?... That is the question.
Paraphrasing:
Can the Tx function only be vectorized when the code path doesn't have branches depending on the number of segments? If so, then this may be the main reason for not supporting segmented packets with FAST_FREE.

In that case, we cannot remove the single-segment requirement from FAST_FREE without sacrificing the performance boost from vectorizing.

No, based on what I state above, this should not be a blocker. The vector paths do require us to guarantee only one segment per packet - without additional context descriptors - so only one descriptor per packet (generally, or always one + ctx, in one code-path case). FAST_FREE can be used in conjunction with that but should not be a requirement. See [1] where in vector cleanup we explicitly check for FAST_FREE.

Similarly for scalar code path, in my latest rework, I am attempting to standardize the use of FAST_FREE optimizations even when we have a slightly slower Tx path [2].

[1] https://github.com/DPDK/dpdk/blob/main/drivers/net/intel/common/tx.h
[2] https://patches.dpdk.org/project/dpdk/patch/[email protected]/

Quoted 9 lines

But then we can proceed pursuing alternative optimizations, as suggested by Konstantin.

Here's another idea:
The Tx function could pre-scan each Tx burst for multi-segment packets, to decide if the burst should be processed by the vector code path or a fallback code path (which can also handle multi-segment packets).

-Morten

#4 lore raw
Quoted 42 lines

From: Bruce Richardson [mailto:[email protected]]
Sent: Wednesday, 14 January 2026 17.36

Quoted 6 lines · Morten Brørup
Quoted 6 lines
Quoted 1 line

If I'm not mistaken, the mbuf library is not a barrier for fast-

freeing

Quoted 1 line

segmented packet mbufs, and thus fast-free of jumbo frames is

possible.

Quoted 2 lines

We need a driver developer to confirm that my suggested approach

-

Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

resetting the mbuf fields, incl. 'm->nb_segs' and 'm->next', when preparing the Tx descriptor - is viable.

Excellent analysis, Morten. If I get a chance some time this

release

Quoted 9 lines
Quoted 4 lines

cycle,
I will try implementing this change in our drivers, see if any difference
is made.

Bruce,

Have you had a chance to look into the driver change requirements? If not, could you please try scratching the surface, to build a gut

feeling.

I'll try and take a look this week. Juggling a few things at the moment, so
I had forgotten about this. Sorry.

More comments inline below.

/Bruce

Quoted 2 lines

I wonder if the vector implementations have strong requirements that

packets are not segmented...

Quoted 2 lines

The i40 driver only sets "tx_simple_allowed" and "tx_vec_allowed"

flags when MBUF_FAST_FREE is set:

Quoted 1 line
Quoted 8 lines

https://elixir.bootlin.com/dpdk/v25.11/source/drivers/net/intel/i40e/i4
0e_rxtx.c#L3502

Quoted 1 line

Actually, it allows but does not require FAST_FREE. The check is just verifying that the flags with everything *but* FAST_FREE masked out is the
same as the original flags, i.e. FAST_FREE is just ignored.

That's not how I read the code:
ad->tx_simple_allowed =

	(txq->offloads ==
	 (txq->offloads & RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE) &&
	 txq->tx_rs_thresh >= I40E_TX_MAX_BURST);

Look at it with offloads=(MULTI_SEGS|FAST_FREE):
simple_allowed = (MULTI_SEGS|FAST_FREE) == (MULTI_SEGS|FAST_FREE) & FAST_FREE i.e.:
simple_allowed = (MULTI_SEGS|FAST_FREE) == FAST_FREE
i.e.: false

Quoted 34 lines
Quoted 3 lines

And only when these two flags are set, it uses a vector Tx function:

https://elixir.bootlin.com/dpdk/v25.11/source/drivers/net/intel/i40e/i4
0e_rxtx.c#L3550

Quoted 2 lines

And a special Tx Prep function:

https://elixir.bootlin.com/dpdk/v25.11/source/drivers/net/intel/i40e/i4
0e_rxtx.c#L3584

Quoted 2 lines

Which fails if nb_segs != 1:

https://elixir.bootlin.com/dpdk/v25.11/source/drivers/net/intel/i40e/i4
0e_rxtx.c#L1675

Quoted 5 lines

So currently it does.
But does it need to?... That is the question. Paraphrasing:
Can the Tx function only be vectorized when the code path doesn't

have branches depending on the number of segments?

Quoted 1 line

If so, then this may be the main reason for not supporting segmented

packets with FAST_FREE.

Quoted 2 lines

In that case, we cannot remove the single-segment requirement from

FAST_FREE without sacrificing the performance boost from vectorizing.

No, based on what I state above, this should not be a blocker. The vector
paths do require us to guarantee only one segment per packet - without additional context descriptors - so only one descriptor per packet (generally, or always one + ctx, in one code-path case). FAST_FREE can be
used in conjunction with that but should not be a requirement. See [1] where in vector cleanup we explicitly check for FAST_FREE.

Quoted 4 lines

Similarly for scalar code path, in my latest rework, I am attempting to standardize the use of FAST_FREE optimizations even when we have a slightly
slower Tx path [2].

Good point:
The Tx path has two steps:
1) Pre-transmission Tx descriptor setup.
2) Post-transmission mbuf free.

FAST_FREE requirements for optimizing each of these two steps might differ.

As suggested in my other email, hopefully the post-transmission step can be vectorized (also for multi-segment packets) by assisting it in the pre-transmission step - i.e. by preparing the FAST_FREE segments for direct release to the mempool.

Then we can consider single-segment requirements for the pre-transmission step.

Quoted 19 lines

[1]
https://github.com/DPDK/dpdk/blob/main/drivers/net/intel/common/tx.h
[2] https://patches.dpdk.org/project/dpdk/patch/20260113151505.1871271-
[email protected]/

Quoted 2 lines

But then we can proceed pursuing alternative optimizations, as

suggested by Konstantin.

Quoted 3 lines

Here's another idea:
The Tx function could pre-scan each Tx burst for multi-segment

packets, to decide if the burst should be processed by the vector code path or a fallback code path (which can also handle multi-segment packets).

Quoted 4 lines

-Morten

#5 lore raw
Quoted 57 lines · Morten Brørup
Quoted 42 lines

From: Bruce Richardson [mailto:[email protected]]
Sent: Wednesday, 14 January 2026 17.36

Quoted 6 lines · Morten Brørup
Quoted 6 lines
Quoted 1 line

If I'm not mistaken, the mbuf library is not a barrier for fast-

freeing

Quoted 1 line

segmented packet mbufs, and thus fast-free of jumbo frames is

possible.

Quoted 2 lines

We need a driver developer to confirm that my suggested approach

-

Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

resetting the mbuf fields, incl. 'm->nb_segs' and 'm->next', when preparing the Tx descriptor - is viable.

Excellent analysis, Morten. If I get a chance some time this

release

Quoted 9 lines
Quoted 4 lines

cycle,
I will try implementing this change in our drivers, see if any difference
is made.

Bruce,

Have you had a chance to look into the driver change requirements? If not, could you please try scratching the surface, to build a gut

feeling.

I'll try and take a look this week. Juggling a few things at the moment, so
I had forgotten about this. Sorry.

More comments inline below.

/Bruce

Quoted 2 lines

I wonder if the vector implementations have strong requirements that

packets are not segmented...

Quoted 2 lines

The i40 driver only sets "tx_simple_allowed" and "tx_vec_allowed"

flags when MBUF_FAST_FREE is set:

Quoted 1 line
Quoted 8 lines

https://elixir.bootlin.com/dpdk/v25.11/source/drivers/net/intel/i40e/i4
0e_rxtx.c#L3502

Quoted 1 line

Actually, it allows but does not require FAST_FREE. The check is just verifying that the flags with everything *but* FAST_FREE masked out is the
same as the original flags, i.e. FAST_FREE is just ignored.

That's not how I read the code:
ad->tx_simple_allowed =

	(txq->offloads ==
	 (txq->offloads & RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE) &&
	 txq->tx_rs_thresh >= I40E_TX_MAX_BURST);
Quoted 6 lines

Look at it with offloads=(MULTI_SEGS|FAST_FREE):
simple_allowed = (MULTI_SEGS|FAST_FREE) == (MULTI_SEGS|FAST_FREE) & FAST_FREE i.e.:
simple_allowed = (MULTI_SEGS|FAST_FREE) == FAST_FREE
i.e.: false

Which is correct. The only flag allowed is FAST_FREE, but its not required. If the input flags were just MULTI_SEGS, it would end up as:

simple_allowed = (MULTI_SEGS) == 0
i.e. also false

So the FAST_FREE flag does not affect the result.

/Bruce

#6 lore raw
Quoted 33 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

I wonder if the vector implementations have strong requirements

that

Quoted 6 lines
Quoted 6 lines

packets are not segmented...

Quoted 2 lines

The i40 driver only sets "tx_simple_allowed" and "tx_vec_allowed"

flags when MBUF_FAST_FREE is set:

Quoted 1 line

https://elixir.bootlin.com/dpdk/v25.11/source/drivers/net/intel/i40e/i4

Quoted 4 lines
Quoted 4 lines

0e_rxtx.c#L3502

Quoted 1 line

Actually, it allows but does not require FAST_FREE. The check is

just

Quoted 1 line
Quoted 1 line

verifying that the flags with everything *but* FAST_FREE masked out

is

Quoted 11 lines
Quoted 2 lines

the
same as the original flags, i.e. FAST_FREE is just ignored.

That's not how I read the code:
ad->tx_simple_allowed =

	(txq->offloads ==
	 (txq->offloads & RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE) &&
	 txq->tx_rs_thresh >= I40E_TX_MAX_BURST);

Look at it with offloads=(MULTI_SEGS|FAST_FREE):
simple_allowed = (MULTI_SEGS|FAST_FREE) == (MULTI_SEGS|FAST_FREE) &

FAST_FREE

Quoted 4 lines

i.e.:
simple_allowed = (MULTI_SEGS|FAST_FREE) == FAST_FREE i.e.: false

Quoted 8 lines

Which is correct. The only flag allowed is FAST_FREE, but its not required.
If the input flags were just MULTI_SEGS, it would end up as:

simple_allowed = (MULTI_SEGS) == 0
i.e. also false

So the FAST_FREE flag does not affect the result.

OK, now I get it.
It's an obfuscated way of writing:
simple_allowed = (offloads & ~RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE) == 0

Suggest updating for readability next time that code is touched.

The comment "only fast free is allowed" [1] didn't help me either; I interpreted it as "it's only allowed when fast free is set". Now that I understand the code, I can also interpret the comment as intended.

[1]: https://elixir.bootlin.com/dpdk/v25.11/source/drivers/net/intel/i40e/i40e_rxtx.c#L3502

Thanks for clarifying, Bruce!

#7 lore raw

PATCH 1/1 net/intel: optimize for fast-free hint

In reply to Morten Brørup

When the fast-free hint is provided to the driver we know that the mbufs have refcnt of 1 and are from the same mempool. Therefore, we can optimize a bit for this case by:

* resetting the necessary mbuf fields, ie. nb_seg and next pointer when we are accessing the mbuf on writing the descriptor.
* on cleanup of buffers after transmit, we can just write those buffers straight to the mempool without accessing them.

---
 drivers/net/intel/common/tx.h            | 16 +++--
 drivers/net/intel/common/tx_scalar_fns.h | 81 +++++++++++++++++++-----
 2 files changed, 76 insertions(+), 21 deletions(-)
drivers/net/intel/common/tx.h +12 -4
index 7fe4022f35..9bda2c8f59 100644
@@ -345,12 +345,20 @@ ci_txq_release_all_mbufs(struct ci_tx_queue *txq, bool use_ctx)
 		return;
 
 	if (!txq->vector_tx) {
-		for (uint16_t i = 0; i < txq->nb_tx_desc; i++) {
-			if (txq->sw_ring[i].mbuf != NULL) {
+		/* Free mbufs from (last_desc_cleaned + 1) to (tx_tail - 1). */
+		const uint16_t start = (txq->last_desc_cleaned + 1) % txq->nb_tx_desc;
+		const uint16_t nb_desc = txq->nb_tx_desc;
+		const uint16_t end = txq->tx_tail;
+
+		uint16_t i = start;
+		if (end < i) {
+			for (; i < nb_desc; i++)
 				rte_pktmbuf_free_seg(txq->sw_ring[i].mbuf);
-				txq->sw_ring[i].mbuf = NULL;
-			}
+			i = 0;
 		}
+		for (; i < end; i++)
+			rte_pktmbuf_free_seg(txq->sw_ring[i].mbuf);
+		memset(txq->sw_ring, 0, sizeof(txq->sw_ring[0]) * nb_desc);
 		return;
 	}
 
drivers/net/intel/common/tx_scalar_fns.h +64 -17

132 lines not on this page · raw message

Signature · 1 line
2.51.0
#8 lore raw

RE: [PATCH] net/intel: optimize for fast-free hint

I haven't looked into the details yet, but have a quick question inline below.

drivers/net/intel/common/tx.h ci_txq_release_all_mbufs(struct ci_tx_queue *txq, · line 350

Quoted 7 lines
drivers/net/intel/common/tx.h +0 -2
@@ -345,12 +345,20 @@ ci_txq_release_all_mbufs(struct ci_tx_queue *txq,
 bool use_ctx)
 		return;
 
 	if (!txq->vector_tx) {
-		for (uint16_t i = 0; i < txq->nb_tx_desc; i++) {
-			if (txq->sw_ring[i].mbuf != NULL) {

You changed this loop to only operate on not-yet-cleaned descriptors.

Here comes the first part of my question:
You removed the NULL check for txq->sw_ring[i].mbuf, thereby assuming that it is never NULL for not-yet-cleaned descriptors.

Quoted 60 lines
(quoted diff) +1 -0
+		/* Free mbufs from (last_desc_cleaned + 1) to (tx_tail -

1). */

(quoted diff) +1 -0
+		const uint16_t start = (txq->last_desc_cleaned + 1) % txq-
Quoted 1 line

nb_tx_desc;

drivers/net/intel/common/tx.h +10 -2
+		const uint16_t nb_desc = txq->nb_tx_desc;
+		const uint16_t end = txq->tx_tail;
+
+		uint16_t i = start;
+		if (end < i) {
+			for (; i < nb_desc; i++)
 				rte_pktmbuf_free_seg(txq->sw_ring[i].mbuf);
-				txq->sw_ring[i].mbuf = NULL;
-			}
+			i = 0;
 		}
+		for (; i < end; i++)
+			rte_pktmbuf_free_seg(txq->sw_ring[i].mbuf);
+		memset(txq->sw_ring, 0, sizeof(txq->sw_ring[0]) * nb_desc);
 		return;
 	}
drivers/net/intel/common/tx_scalar_fns.h +2 -1
index 82dc54438f..47ddcf411b 100644
@@ -30,16 +30,60 @@ ci_tx_xmit_cleanup(struct ci_tx_queue *txq)
 	const uint16_t rs_idx = (last_desc_cleaned == nb_tx_desc - 1) ?
 			0 :
 			(last_desc_cleaned + 1) >> txq->log2_rs_thresh;
-	uint16_t desc_to_clean_to = (rs_idx << txq->log2_rs_thresh) +
 (txq->tx_rs_thresh - 1);
+	const uint16_t dd_idx = txq->rs_last_id[rs_idx];
+	const uint16_t first_to_clean = rs_idx << txq->log2_rs_thresh;
 
 	/* Check if descriptor is done - all drivers use 0xF as done

value in bits 3:0 */

drivers/net/intel/common/tx_scalar_fns.h +0 -1
-	if ((txd[txq->rs_last_id[rs_idx]].cmd_type_offset_bsz &

rte_cpu_to_le_64(CI_TXD_QW1_DTYPE_M)) !=

drivers/net/intel/common/tx_scalar_fns.h +1 -0
+	if ((txd[dd_idx].cmd_type_offset_bsz &

rte_cpu_to_le_64(CI_TXD_QW1_DTYPE_M)) !=

drivers/net/intel/common/tx_scalar_fns.h +2 -0
 			rte_cpu_to_le_64(CI_TX_DESC_DTYPE_DESC_DONE))
 		/* Descriptor not yet processed by hardware */
 		return -1;
 
+	/* DD bit is set, descriptors are done. Now free the mbufs. */
+	/* Note: nb_tx_desc is guaranteed to be a multiple of

tx_rs_thresh,

drivers/net/intel/common/tx_scalar_fns.h +1 -0
+	 * validated during queue setup. This means cleanup never wraps

around

drivers/net/intel/common/tx_scalar_fns.h +1 -0
+	 * the ring within a single burst (e.g., ring=256, rs_thresh=32

gives

drivers/net/intel/common/tx_scalar_fns.h +7 -0
+	 * bursts of 0-31, 32-63, ..., 224-255).
+	 */
+	const uint16_t nb_to_clean = txq->tx_rs_thresh;
+	struct ci_tx_entry *sw_ring = txq->sw_ring;
+
+	if (txq->offloads & RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE) {
+		/* FAST_FREE path: mbufs are already reset, just return to

pool */

drivers/net/intel/common/tx_scalar_fns.h ci_tx_xmit_cleanup(struct ci_tx_queue *txq) · line 61

Quoted 9 lines
drivers/net/intel/common/tx_scalar_fns.h +9 -0
+		uint16_t nb_free = 0;
+
+		/* Get cached mempool pointer, or cache it on first use */
+		struct rte_mempool *mp =
+			likely(txq->fast_free_mp != (void *)UINTPTR_MAX) ?
+			txq->fast_free_mp :
+			(txq->fast_free_mp = sw_ring[dd_idx].mbuf->pool);
+
+		/* Pack non-NULL mbufs in-place at start of sw_ring range.

Here is the second part of my question:
How can they (sw_ring[X].mbuf) be NULL here, when they cannot be NULL in ci_txq_release_all_mbufs()?

Quoted 36 lines
drivers/net/intel/common/tx_scalar_fns.h +1 -0
+		 * No modulo needed in loop since we're guaranteed not to

wrap.

drivers/net/intel/common/tx_scalar_fns.h +3 -0
+		 */
+		for (uint16_t i = 0; i < nb_to_clean; i++) {
+			struct rte_mbuf *m = sw_ring[first_to_clean +

i].mbuf;

drivers/net/intel/common/tx_scalar_fns.h +8 -0
+			if (m != NULL) {
+				/* Pack into sw_ring at packed position */
+				sw_ring[first_to_clean + nb_free].mbuf = m;
+				nb_free++;
+			}
+		}
+
+		/* Bulk return to mempool using packed sw_ring entries

directly */

drivers/net/intel/common/tx_scalar_fns.h +2 -0
+		if (nb_free > 0)
+			rte_mempool_put_bulk(mp, (void

**)&sw_ring[first_to_clean].mbuf, nb_free);

drivers/net/intel/common/tx_scalar_fns.h +2 -0
+	} else {
+		/* Non-FAST_FREE path: use prefree_seg for refcount checks

*/

drivers/net/intel/common/tx_scalar_fns.h +2 -0
+		for (uint16_t i = 0; i < nb_to_clean; i++) {
+			struct rte_mbuf *m = sw_ring[first_to_clean +

i].mbuf;

drivers/net/intel/common/tx_scalar_fns.h +5 -0
+			if (m != NULL)
+				rte_pktmbuf_free_seg(m);
+		}
+	}
+
 	/* Update the txq to reflect the last descriptor that was cleaned

*/

drivers/net/intel/common/tx_scalar_fns.h +1 -1
-	txq->last_desc_cleaned = desc_to_clean_to;
+	txq->last_desc_cleaned = first_to_clean + txq->tx_rs_thresh - 1;
 	txq->nb_tx_free += txq->tx_rs_thresh;
 
 	return 0;
#9 lore raw

Re: [PATCH] net/intel: optimize for fast-free hint

Quoted 6 lines · Morten Brørup

I haven't looked into the details yet, but have a quick question inline below.

Quoted 4 lines
(quoted diff) +0 -0
@@ -345,12 +345,20 @@ ci_txq_release_all_mbufs(struct ci_tx_queue *txq,
 bool use_ctx)
 		return;
 
Quoted 9 lines

drivers/net/intel/common/tx.h ci_txq_release_all_mbufs(struct ci_tx_queue *txq, bool use_ctx) · line 349

Quoted 3 lines
drivers/net/intel/common/tx.h +0 -2
 	if (!txq->vector_tx) {
-		for (uint16_t i = 0; i < txq->nb_tx_desc; i++) {
-			if (txq->sw_ring[i].mbuf != NULL) {

You changed this loop to only operate on not-yet-cleaned descriptors.

Here comes the first part of my question:
You removed the NULL check for txq->sw_ring[i].mbuf, thereby assuming that it is never NULL for not-yet-cleaned descriptors.

Good point, I was quite focused on making this block and the vector block the same, I forgot that we can have NULL pointers for context descriptors. That was a silly mistake (and AI never caught it for me either.)

Quoted 64 lines
Quoted 64 lines
(quoted diff) +1 -0
+		/* Free mbufs from (last_desc_cleaned + 1) to (tx_tail -

1). */

(quoted diff) +1 -0
+		const uint16_t start = (txq->last_desc_cleaned + 1) % txq-
Quoted 1 line

nb_tx_desc;

drivers/net/intel/common/tx.h +10 -2
+		const uint16_t nb_desc = txq->nb_tx_desc;
+		const uint16_t end = txq->tx_tail;
+
+		uint16_t i = start;
+		if (end < i) {
+			for (; i < nb_desc; i++)
 				rte_pktmbuf_free_seg(txq->sw_ring[i].mbuf);
-				txq->sw_ring[i].mbuf = NULL;
-			}
+			i = 0;
 		}
+		for (; i < end; i++)
+			rte_pktmbuf_free_seg(txq->sw_ring[i].mbuf);
+		memset(txq->sw_ring, 0, sizeof(txq->sw_ring[0]) * nb_desc);
 		return;
 	}
drivers/net/intel/common/tx_scalar_fns.h +2 -1
index 82dc54438f..47ddcf411b 100644
@@ -30,16 +30,60 @@ ci_tx_xmit_cleanup(struct ci_tx_queue *txq)
 	const uint16_t rs_idx = (last_desc_cleaned == nb_tx_desc - 1) ?
 			0 :
 			(last_desc_cleaned + 1) >> txq->log2_rs_thresh;
-	uint16_t desc_to_clean_to = (rs_idx << txq->log2_rs_thresh) +
 (txq->tx_rs_thresh - 1);
+	const uint16_t dd_idx = txq->rs_last_id[rs_idx];
+	const uint16_t first_to_clean = rs_idx << txq->log2_rs_thresh;
 
 	/* Check if descriptor is done - all drivers use 0xF as done

value in bits 3:0 */

drivers/net/intel/common/tx_scalar_fns.h +0 -1
-	if ((txd[txq->rs_last_id[rs_idx]].cmd_type_offset_bsz &

rte_cpu_to_le_64(CI_TXD_QW1_DTYPE_M)) !=

drivers/net/intel/common/tx_scalar_fns.h +1 -0
+	if ((txd[dd_idx].cmd_type_offset_bsz &

rte_cpu_to_le_64(CI_TXD_QW1_DTYPE_M)) !=

drivers/net/intel/common/tx_scalar_fns.h +2 -0
 			rte_cpu_to_le_64(CI_TX_DESC_DTYPE_DESC_DONE))
 		/* Descriptor not yet processed by hardware */
 		return -1;
 
+	/* DD bit is set, descriptors are done. Now free the mbufs. */
+	/* Note: nb_tx_desc is guaranteed to be a multiple of

tx_rs_thresh,

drivers/net/intel/common/tx_scalar_fns.h +1 -0
+	 * validated during queue setup. This means cleanup never wraps

around

drivers/net/intel/common/tx_scalar_fns.h +1 -0
+	 * the ring within a single burst (e.g., ring=256, rs_thresh=32

gives

drivers/net/intel/common/tx_scalar_fns.h +7 -0
+	 * bursts of 0-31, 32-63, ..., 224-255).
+	 */
+	const uint16_t nb_to_clean = txq->tx_rs_thresh;
+	struct ci_tx_entry *sw_ring = txq->sw_ring;
+
+	if (txq->offloads & RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE) {
+		/* FAST_FREE path: mbufs are already reset, just return to

pool */

drivers/net/intel/common/tx_scalar_fns.h +4 -0
+		uint16_t nb_free = 0;
+
+		/* Get cached mempool pointer, or cache it on first use */
+		struct rte_mempool *mp =
Quoted 8 lines

drivers/net/intel/common/tx_scalar_fns.h ci_tx_xmit_cleanup(struct ci_tx_queue *txq) · line 61

Quoted 5 lines
drivers/net/intel/common/tx_scalar_fns.h +5 -0
+			likely(txq->fast_free_mp != (void *)UINTPTR_MAX) ?
+			txq->fast_free_mp :
+			(txq->fast_free_mp = sw_ring[dd_idx].mbuf->pool);
+
+		/* Pack non-NULL mbufs in-place at start of sw_ring range.

Here is the second part of my question:
How can they (sw_ring[X].mbuf) be NULL here, when they cannot be NULL in ci_txq_release_all_mbufs()?

Because the latter function is wrong! :-)

Quoted 37 lines
Quoted 37 lines
drivers/net/intel/common/tx_scalar_fns.h +1 -0
+		 * No modulo needed in loop since we're guaranteed not to

wrap.

drivers/net/intel/common/tx_scalar_fns.h +3 -0
+		 */
+		for (uint16_t i = 0; i < nb_to_clean; i++) {
+			struct rte_mbuf *m = sw_ring[first_to_clean +

i].mbuf;

drivers/net/intel/common/tx_scalar_fns.h +8 -0
+			if (m != NULL) {
+				/* Pack into sw_ring at packed position */
+				sw_ring[first_to_clean + nb_free].mbuf = m;
+				nb_free++;
+			}
+		}
+
+		/* Bulk return to mempool using packed sw_ring entries

directly */

drivers/net/intel/common/tx_scalar_fns.h +2 -0
+		if (nb_free > 0)
+			rte_mempool_put_bulk(mp, (void

**)&sw_ring[first_to_clean].mbuf, nb_free);

drivers/net/intel/common/tx_scalar_fns.h +2 -0
+	} else {
+		/* Non-FAST_FREE path: use prefree_seg for refcount checks

*/

drivers/net/intel/common/tx_scalar_fns.h +2 -0
+		for (uint16_t i = 0; i < nb_to_clean; i++) {
+			struct rte_mbuf *m = sw_ring[first_to_clean +

i].mbuf;

drivers/net/intel/common/tx_scalar_fns.h +5 -0
+			if (m != NULL)
+				rte_pktmbuf_free_seg(m);
+		}
+	}
+
 	/* Update the txq to reflect the last descriptor that was cleaned

*/

drivers/net/intel/common/tx_scalar_fns.h +1 -1
-	txq->last_desc_cleaned = desc_to_clean_to;
+	txq->last_desc_cleaned = first_to_clean + txq->tx_rs_thresh - 1;
 	txq->nb_tx_free += txq->tx_rs_thresh;
 
 	return 0;
#10 lore raw

RE: [PATCH] net/intel: optimize for fast-free hint

Quoted 29 lines

From: Bruce Richardson [mailto:[email protected]]
Sent: Friday, 23 January 2026 13.09

Quoted 1 line · Morten Brørup

I haven't looked into the details yet, but have a quick question

inline below.

Quoted 2 lines

Review ci_txq_release_all_mbufs(struct ci_tx_queue

Quoted 2 lines
(quoted diff) +0 -0
@@ -345,12 +345,20 @@ ci_txq_release_all_mbufs(struct ci_tx_queue

*txq,

Quoted 11 lines

drivers/net/intel/common/tx.h ci_txq_release_all_mbufs(struct ci_tx_queue *txq, bool use_ctx) · line 349

Quoted 6 lines
drivers/net/intel/common/tx.h +0 -2
 bool use_ctx)
 		return;
 
 	if (!txq->vector_tx) {
-		for (uint16_t i = 0; i < txq->nb_tx_desc; i++) {
-			if (txq->sw_ring[i].mbuf != NULL) {

You changed this loop to only operate on not-yet-cleaned descriptors.

Here comes the first part of my question:
You removed the NULL check for txq->sw_ring[i].mbuf, thereby assuming

that it is never NULL for not-yet-cleaned descriptors.

Quoted 1 line

Good point, I was quite focused on making this block and the vector block
the same, I forgot that we can have NULL pointers for context descriptors.
That was a silly mistake (and AI never caught it for me either.)

Quoted 8 lines
Quoted 8 lines

drivers/net/intel/common/tx.h ci_txq_release_all_mbufs(struct ci_tx_queue *txq, bool use_ctx) · line 353

Quoted 8 lines
(quoted diff) +1 -0
+		/* Free mbufs from (last_desc_cleaned + 1) to (tx_tail -

1). */

(quoted diff) +1 -0
+		const uint16_t start = (txq->last_desc_cleaned + 1) % txq-
Quoted 1 line

nb_tx_desc;

drivers/net/intel/common/tx.h +4 -0
+		const uint16_t nb_desc = txq->nb_tx_desc;
+		const uint16_t end = txq->tx_tail;
+
+		uint16_t i = start;

Suggest getting rid of "start"; it is only used for initializing "i".

Quoted 10 lines
Quoted 10 lines

drivers/net/intel/common/tx.h ci_txq_release_all_mbufs(struct ci_tx_queue *txq, bool use_ctx) · line 361

Quoted 10 lines
drivers/net/intel/common/tx.h +6 -2
+		if (end < i) {
+			for (; i < nb_desc; i++)
 				rte_pktmbuf_free_seg(txq->sw_ring[i].mbuf);
-				txq->sw_ring[i].mbuf = NULL;
-			}
+			i = 0;
 		}
+		for (; i < end; i++)
+			rte_pktmbuf_free_seg(txq->sw_ring[i].mbuf);
+		memset(txq->sw_ring, 0, sizeof(txq->sw_ring[0]) * nb_desc);

Consider also splitting this memset() into two, one for each of the two for loops. Then you might need to keep "start" and make it non-const. :-)

Quoted 2 lines
Quoted 2 lines

Review

Quoted 2 lines
(quoted diff) +0 -0
 		return;
 	}

Or just keep the original version, looping over all descriptors.

#11 lore raw

Re: [PATCH] net/intel: optimize for fast-free hint

Quoted 32 lines · Morten Brørup
Quoted 32 lines

From: Bruce Richardson [mailto:[email protected]]
Sent: Friday, 23 January 2026 13.09

Quoted 1 line · Morten Brørup

I haven't looked into the details yet, but have a quick question

inline below.

Quoted 2 lines

Review ci_txq_release_all_mbufs(struct ci_tx_queue

Quoted 2 lines
(quoted diff) +0 -0
@@ -345,12 +345,20 @@ ci_txq_release_all_mbufs(struct ci_tx_queue

*txq,

Quoted 11 lines

drivers/net/intel/common/tx.h ci_txq_release_all_mbufs(struct ci_tx_queue *txq, bool use_ctx) · line 349

Quoted 6 lines
drivers/net/intel/common/tx.h +0 -2
 bool use_ctx)
 		return;
 
 	if (!txq->vector_tx) {
-		for (uint16_t i = 0; i < txq->nb_tx_desc; i++) {
-			if (txq->sw_ring[i].mbuf != NULL) {

You changed this loop to only operate on not-yet-cleaned descriptors.

Here comes the first part of my question:
You removed the NULL check for txq->sw_ring[i].mbuf, thereby assuming

that it is never NULL for not-yet-cleaned descriptors.

Quoted 1 line

Good point, I was quite focused on making this block and the vector block
the same, I forgot that we can have NULL pointers for context descriptors.
That was a silly mistake (and AI never caught it for me either.)

Quoted 3 lines
Quoted 3 lines
(quoted diff) +1 -0
+		/* Free mbufs from (last_desc_cleaned + 1) to (tx_tail -

1). */

(quoted diff) +1 -0
+		const uint16_t start = (txq->last_desc_cleaned + 1) % txq-
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines

drivers/net/intel/common/tx.h ci_txq_release_all_mbufs(struct ci_tx_queue *txq, bool use_ctx) · line 353

Quoted 5 lines
Quoted 1 line

nb_tx_desc;

drivers/net/intel/common/tx.h +4 -0
+		const uint16_t nb_desc = txq->nb_tx_desc;
+		const uint16_t end = txq->tx_tail;
+
+		uint16_t i = start;

Suggest getting rid of "start"; it is only used for initializing "i".

Not sure it's worth doing. I quite like having an explicit start and end values for clarity.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
drivers/net/intel/common/tx.h +2 -2
+		if (end < i) {
+			for (; i < nb_desc; i++)
 				rte_pktmbuf_free_seg(txq->sw_ring[i].mbuf);
-				txq->sw_ring[i].mbuf = NULL;
-			}
Quoted 9 lines
Quoted 5 lines
Quoted 5 lines

drivers/net/intel/common/tx.h ci_txq_release_all_mbufs(struct ci_tx_queue *txq, bool use_ctx) · line 361

Quoted 5 lines
drivers/net/intel/common/tx.h +4 -0
+			i = 0;
 		}
+		for (; i < end; i++)
+			rte_pktmbuf_free_seg(txq->sw_ring[i].mbuf);
+		memset(txq->sw_ring, 0, sizeof(txq->sw_ring[0]) * nb_desc);

Consider also splitting this memset() into two, one for each of the two for loops. Then you might need to keep "start" and make it non-const. :-)

Don't see the point of that. The memset just zeros the whole array, ignoring wraparound so no point in doing two memsets when one will do.

Quoted 4 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

return; }

Or just keep the original version, looping over all descriptors.

The reason for this whole change is that after the refactor the old code was wrong.

The original code used the fact that all mbuf pointers were zereod or overwritten after being freed, but that no longer applies, because we free the mbufs in bulk after we check the dd bits, rather than doing so individually later immediately before reuse. Instead, in both the datapath and this release path, we must use the index values to track what mbuf entries are valid or invalid. (We go from having two states, NULL or non-NULL, to 3; invalid i.e. freed or NULL, valid-NULL i.e. in slot used by context descriptor, valid-non-NULL i.e. a pointer to a not-yet-cleaned-up mbuf).

/Bruce

#12 lore raw

RE: [PATCH] net/intel: optimize for fast-free hint

Quoted 50 lines

From: Bruce Richardson [mailto:[email protected]]
Sent: Friday, 23 January 2026 13.54

Quoted 8 lines · Morten Brørup
Quoted 8 lines

From: Bruce Richardson [mailto:[email protected]]
Sent: Friday, 23 January 2026 13.09

Quoted 1 line · Morten Brørup

I haven't looked into the details yet, but have a quick question

inline below.

Quoted 2 lines

Review ci_txq_release_all_mbufs(struct

Quoted 2 lines
(quoted diff) +0 -0
@@ -345,12 +345,20 @@ ci_txq_release_all_mbufs(struct

ci_tx_queue

Quoted 9 lines
Quoted 9 lines

*txq,

Quoted 8 lines

drivers/net/intel/common/tx.h ci_txq_release_all_mbufs(struct ci_tx_queue *txq, bool use_ctx) · line 349

Quoted 6 lines
drivers/net/intel/common/tx.h +0 -2
 bool use_ctx)
 		return;
 
 	if (!txq->vector_tx) {
-		for (uint16_t i = 0; i < txq->nb_tx_desc; i++) {
-			if (txq->sw_ring[i].mbuf != NULL) {

You changed this loop to only operate on not-yet-cleaned

descriptors.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Here comes the first part of my question:
You removed the NULL check for txq->sw_ring[i].mbuf, thereby

assuming

Quoted 10 lines
Quoted 10 lines

that it is never NULL for not-yet-cleaned descriptors.

Quoted 1 line

Good point, I was quite focused on making this block and the vector block
the same, I forgot that we can have NULL pointers for context descriptors.
That was a silly mistake (and AI never caught it for me either.)

Quoted 1 line

Review ci_txq_release_all_mbufs(struct · line 349

Quoted 1 line
(quoted diff) +1 -0
+		/* Free mbufs from (last_desc_cleaned + 1) to

(tx_tail -

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Review

Quoted 2 lines

1). */

(quoted diff) +1 -0
+		const uint16_t start = (txq->last_desc_cleaned + 1) %

txq-

Quoted 8 lines
Quoted 5 lines
Quoted 5 lines

drivers/net/intel/common/tx.h ci_txq_release_all_mbufs(struct ci_tx_queue *txq, bool use_ctx) · line 353

Quoted 5 lines
Quoted 1 line

nb_tx_desc;

drivers/net/intel/common/tx.h +4 -0
+		const uint16_t nb_desc = txq->nb_tx_desc;
+		const uint16_t end = txq->tx_tail;
+
+		uint16_t i = start;

Suggest getting rid of "start"; it is only used for initializing "i".

Quoted 3 lines

Not sure it's worth doing. I quite like having an explicit start and end
values for clarity.

I have no preference, and it's a matter of taste, so the choice is yours. :-)

Quoted 31 lines
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines

drivers/net/intel/common/tx.h ci_txq_release_all_mbufs(struct ci_tx_queue *txq, bool use_ctx)

Quoted 4 lines
drivers/net/intel/common/tx.h +2 -0
+		if (end < i) {
+			for (; i < nb_desc; i++)
 				rte_pktmbuf_free_seg(txq-

sw_ring[i].mbuf);

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

drivers/net/intel/common/tx.h ci_txq_release_all_mbufs(struct ci_tx_queue *txq, bool use_ctx)

Quoted 7 lines
drivers/net/intel/common/tx.h +4 -2
-				txq->sw_ring[i].mbuf = NULL;
-			}
+			i = 0;
 		}
+		for (; i < end; i++)
+			rte_pktmbuf_free_seg(txq->sw_ring[i].mbuf);
+		memset(txq->sw_ring, 0, sizeof(txq->sw_ring[0]) *

nb_desc);

Quoted 2 lines

Consider also splitting this memset() into two, one for each of the

two for loops.

Quoted 2 lines

Then you might need to keep "start" and make it non-const. :-)

Don't see the point of that. The memset just zeros the whole array, ignoring wraparound so no point in doing two memsets when one will do.

Quoted 4 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

return; }

Or just keep the original version, looping over all descriptors.

The reason for this whole change is that after the refactor the old code
was wrong.

Quoted 13 lines

The original code used the fact that all mbuf pointers were zereod or overwritten after being freed, but that no longer applies, because we free
the mbufs in bulk after we check the dd bits, rather than doing so individually later immediately before reuse. Instead, in both the datapath
and this release path, we must use the index values to track what mbuf entries are valid or invalid. (We go from having two states, NULL or non-NULL, to 3; invalid i.e. freed or NULL, valid-NULL i.e. in slot used by
context descriptor, valid-non-NULL i.e. a pointer to a not-yet-cleaned- up
mbuf).

Thank you for clarifying.

I thought of the two for loops as a kind of performance optimization, skipping the sub-array of already freed descriptors. In that case, memsetting only the two remaining sub-arrays might have been a good idea. That's not the case, so having one memset for the whole array is perfectly fine.

#13 lore raw

PATCH v2 1/1 net/intel: optimize for fast-free hint

In reply to Bruce Richardson

When the fast-free hint is provided to the driver we know that the mbufs have refcnt of 1 and are from the same mempool. Therefore, we can optimize a bit for this case by:

* resetting the necessary mbuf fields, ie. nb_seg and next pointer when we are accessing the mbuf on writing the descriptor.
* on cleanup of buffers after transmit, we can just write those buffers straight to the mempool without accessing them.

---
V2: Fix issues with original submission:
* missed check for NULL mbufs
* fixed issue with freeing directly from sw_ring in scalar path which doesn't work as thats not a flag array of pointers
* fixed missing null assignment in case of large segments for TSO

---
 drivers/net/intel/common/tx.h        | 21 ++++--
 drivers/net/intel/common/tx_scalar.h | 95 ++++++++++++++++++++++------
 2 files changed, 90 insertions(+), 26 deletions(-)
drivers/net/intel/common/tx.h +15 -6
index 283bd58d5d..f2123f069c 100644
@@ -363,13 +363,22 @@ ci_txq_release_all_mbufs(struct ci_tx_queue *txq, bool use_ctx)
 		return;
 
 	if (!txq->use_vec_entry) {
-		/* Regular scalar path uses sw_ring with ci_tx_entry */
-		for (uint16_t i = 0; i < txq->nb_tx_desc; i++) {
-			if (txq->sw_ring[i].mbuf != NULL) {
-				rte_pktmbuf_free_seg(txq->sw_ring[i].mbuf);
-				txq->sw_ring[i].mbuf = NULL;
-			}
+		/* Free mbufs from (last_desc_cleaned + 1) to (tx_tail - 1). */
+		const uint16_t start = (txq->last_desc_cleaned + 1) % txq->nb_tx_desc;
+		const uint16_t nb_desc = txq->nb_tx_desc;
+		const uint16_t end = txq->tx_tail;
+
+		uint16_t i = start;
+		if (end < i) {
+			for (; i < nb_desc; i++)
+				if (txq->sw_ring[i].mbuf != NULL)
+					rte_pktmbuf_free_seg(txq->sw_ring[i].mbuf);
+			i = 0;
 		}
+		for (; i < end; i++)
+			if (txq->sw_ring[i].mbuf != NULL)
+				rte_pktmbuf_free_seg(txq->sw_ring[i].mbuf);
+		memset(txq->sw_ring, 0, sizeof(txq->sw_ring[0]) * nb_desc);
 		return;
 	}
 
drivers/net/intel/common/tx_scalar.h +75 -20

154 lines not on this page · raw message

Signature · 1 line
2.51.0
#14 lore raw

RE: [PATCH v2] net/intel: optimize for fast-free hint

Quoted 8 lines

From: Bruce Richardson [mailto:[email protected]]
Sent: Wednesday, 8 April 2026 15.25

When the fast-free hint is provided to the driver we know that the mbufs
have refcnt of 1 and are from the same mempool. Therefore, we can optimize a bit for this case by:

Quoted 6 lines

* resetting the necessary mbuf fields, ie. nb_seg and next pointer when we are accessing the mbuf on writing the descriptor.
* on cleanup of buffers after transmit, we can just write those buffers straight to the mempool without accessing them.

A bunch of review thoughts inline below. The ones regarding instrumentation should be fixed. The rest might be irrelevant and/or nonsense.

Quoted 40 lines

---
V2: Fix issues with original submission:
* missed check for NULL mbufs
* fixed issue with freeing directly from sw_ring in scalar path which doesn't work as thats not a flag array of pointers
* fixed missing null assignment in case of large segments for TSO

---
 drivers/net/intel/common/tx.h        | 21 ++++--
 drivers/net/intel/common/tx_scalar.h | 95 ++++++++++++++++++++++------
 2 files changed, 90 insertions(+), 26 deletions(-)
drivers/net/intel/common/tx.h +1 -6
index 283bd58d5d..f2123f069c 100644
@@ -363,13 +363,22 @@ ci_txq_release_all_mbufs(struct ci_tx_queue *txq,
 bool use_ctx)
 		return;
 
 	if (!txq->use_vec_entry) {
-		/* Regular scalar path uses sw_ring with ci_tx_entry */
-		for (uint16_t i = 0; i < txq->nb_tx_desc; i++) {
-			if (txq->sw_ring[i].mbuf != NULL) {
-				rte_pktmbuf_free_seg(txq->sw_ring[i].mbuf);
-				txq->sw_ring[i].mbuf = NULL;
-			}
+		/* Free mbufs from (last_desc_cleaned + 1) to (tx_tail -

1). */

drivers/net/intel/common/tx.h +1 -0
+		const uint16_t start = (txq->last_desc_cleaned + 1) % txq-
Quoted 1 line

nb_tx_desc;

drivers/net/intel/common/tx.h +8 -0
+		const uint16_t nb_desc = txq->nb_tx_desc;
+		const uint16_t end = txq->tx_tail;
+
+		uint16_t i = start;
+		if (end < i) {
+			for (; i < nb_desc; i++)
+				if (txq->sw_ring[i].mbuf != NULL)
+					rte_pktmbuf_free_seg(txq-
Quoted 1 line

sw_ring[i].mbuf);

drivers/net/intel/common/tx.h ci_txq_release_all_mbufs(struct ci_tx_queue *txq, bool use_ctx) · line 383

Quoted 8 lines
drivers/net/intel/common/tx.h +5 -0
+			i = 0;
 		}
+		for (; i < end; i++)
+			if (txq->sw_ring[i].mbuf != NULL)
+				rte_pktmbuf_free_seg(txq->sw_ring[i].mbuf);
+		memset(txq->sw_ring, 0, sizeof(txq->sw_ring[0]) * nb_desc);
 		return;
 	}

The above LGTM.
IIRC, we already discussed it - or something very similar.

Quoted 41 lines
drivers/net/intel/common/tx_scalar.h +3 -5
index 9fcd2e4733..adbc4bafee 100644
@@ -197,16 +197,63 @@ ci_tx_xmit_cleanup(struct ci_tx_queue *txq)
 	const uint16_t rs_idx = (last_desc_cleaned == nb_tx_desc - 1) ?
 			0 :
 			(last_desc_cleaned + 1) >> txq->log2_rs_thresh;
-	uint16_t desc_to_clean_to = (rs_idx << txq->log2_rs_thresh) +
 (txq->tx_rs_thresh - 1);
+	const uint16_t dd_idx = txq->rs_last_id[rs_idx];
+	const uint16_t first_to_clean = rs_idx << txq->log2_rs_thresh;
 
-	/* Check if descriptor is done  */
-	if ((txd[txq->rs_last_id[rs_idx]].cmd_type_offset_bsz &
-			rte_cpu_to_le_64(CI_TXD_QW1_DTYPE_M)) !=
-				rte_cpu_to_le_64(CI_TX_DESC_DTYPE_DESC_DONE))
+	/* Check if descriptor is done - all drivers use 0xF as done

value in bits 3:0 */

drivers/net/intel/common/tx_scalar.h +1 -0
+	if ((txd[dd_idx].cmd_type_offset_bsz &

rte_cpu_to_le_64(CI_TXD_QW1_DTYPE_M)) !=

drivers/net/intel/common/tx_scalar.h +4 -0
+			rte_cpu_to_le_64(CI_TX_DESC_DTYPE_DESC_DONE))
+		/* Descriptor not yet processed by hardware */
 		return -1;
 
+	/* DD bit is set, descriptors are done. Now free the mbufs. */
+	/* Note: nb_tx_desc is guaranteed to be a multiple of

tx_rs_thresh,

drivers/net/intel/common/tx_scalar.h +1 -0
+	 * validated during queue setup. This means cleanup never wraps

around

drivers/net/intel/common/tx_scalar.h +1 -0
+	 * the ring within a single burst (e.g., ring=256, rs_thresh=32

gives

drivers/net/intel/common/tx_scalar.h +7 -0
+	 * bursts of 0-31, 32-63, ..., 224-255).
+	 */
+	const uint16_t nb_to_clean = txq->tx_rs_thresh;
+	struct ci_tx_entry *sw_ring = txq->sw_ring;
+
+	if (txq->offloads & RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE) {
+		/* FAST_FREE path: mbufs are already reset, just return to
Quoted 1 line

pool */

Depending on which cache lines from txq have already been loaded, unless txq->offloads is hot in the CPU cache and txq->fast_free_mp is not, consider testing (mp != NULL) instead of (txq->offloads & RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE). Like here:
https://elixir.bootlin.com/dpdk/v26.03/source/drivers/net/intel/common/tx.h#L281

Quoted 16 lines
drivers/net/intel/common/tx_scalar.h +11 -0
+		void *free[CI_TX_MAX_FREE_BUF_SZ];
+		uint16_t nb_free = 0;
+
+		/* Get cached mempool pointer, or cache it on first use */
+		struct rte_mempool *mp =
+			likely(txq->fast_free_mp != (void *)UINTPTR_MAX) ?
+			txq->fast_free_mp :
+			(txq->fast_free_mp = sw_ring[dd_idx].mbuf->pool);
+
+		/* Pack non-NULL mbufs in-place at start of sw_ring range.
+		 * No modulo needed in loop since we're guaranteed not to

wrap.

drivers/net/intel/common/tx_scalar.h +3 -0
+		 */
+		for (uint16_t i = 0; i < nb_to_clean; i++) {
+			struct rte_mbuf *m = sw_ring[first_to_clean +

i].mbuf;

drivers/net/intel/common/tx_scalar.h ci_tx_xmit_cleanup(struct ci_tx_queue *txq) · line 236

Quoted 3 lines
drivers/net/intel/common/tx_scalar.h +3 -0
+			if (m == NULL)
+				continue;
+			free[nb_free++] = m;

Should sw_ring[first_to_clean + i].mbuf be set to NULL here, instead of in ci_xmit_pkts()? I don't know, just want you to consider it.

drivers/net/intel/common/tx_scalar.h

Quoted 2 lines
drivers/net/intel/common/tx_scalar.h +2 -0
+			if (unlikely(nb_free == CI_TX_MAX_FREE_BUF_SZ)) {
+				rte_mempool_put_bulk(mp, free, nb_free);

rte_mempool_put_bulk() -> rte_mbuf_raw_free_bulk(), for instrumentation.

drivers/net/intel/common/tx_scalar.h ci_tx_xmit_cleanup(struct ci_tx_queue *txq) · line 245

Quoted 8 lines
drivers/net/intel/common/tx_scalar.h +5 -0
+				nb_free = 0;
+			}
+		}
+
+		/* Bulk return to mempool using packed sw_ring entries

directly */

drivers/net/intel/common/tx_scalar.h +2 -0
+		if (nb_free > 0)
+			rte_mempool_put_bulk(mp, free, nb_free);

rte_mempool_put_bulk() -> rte_mbuf_raw_free_bulk(), for instrumentation.

drivers/net/intel/common/tx_scalar.h ci_tx_xmit_cleanup(struct ci_tx_queue *txq) · line 251

Quoted 8 lines
drivers/net/intel/common/tx_scalar.h +2 -0
+	} else {
+		/* Non-FAST_FREE path: use prefree_seg for refcount checks

*/

drivers/net/intel/common/tx_scalar.h +2 -0
+		for (uint16_t i = 0; i < nb_to_clean; i++) {
+			struct rte_mbuf *m = sw_ring[first_to_clean +

i].mbuf;

drivers/net/intel/common/tx_scalar.h +2 -0
+			if (m != NULL)
+				rte_pktmbuf_free_seg(m);

Should sw_ring[first_to_clean + i].mbuf be set to NULL here, instead of in ci_xmit_pkts()? I don't know, just want you to consider it.

Quoted 20 lines
drivers/net/intel/common/tx_scalar.h +3 -0
+		}
+	}
+
 	/* Update the txq to reflect the last descriptor that was cleaned

*/

drivers/net/intel/common/tx_scalar.h +1 -3
-	txq->last_desc_cleaned = desc_to_clean_to;
+	txq->last_desc_cleaned = first_to_clean + txq->tx_rs_thresh - 1;
 	txq->nb_tx_free += txq->tx_rs_thresh;
 
 	return 0;
@@ -450,8 +497,6 @@ ci_xmit_pkts(struct ci_tx_queue *txq,
 			txd = &ci_tx_ring[tx_id];
 			tx_id = txe->next_id;
 
-			if (txe->mbuf)
-				rte_pktmbuf_free_seg(txe->mbuf);
 			txe->mbuf = tx_pkt;
 			/* Setup TX Descriptor */
 			td_cmd |= CI_TX_DESC_CMD_EOP;

drivers/net/intel/common/tx_scalar.h ci_xmit_pkts(struct ci_tx_queue *txq, · line 519

Quoted 3 lines
drivers/net/intel/common/tx_scalar.h +0 -0
@@ -472,10 +517,7 @@ ci_xmit_pkts(struct ci_tx_queue *txq,
 
 			txn = &sw_ring[txe->next_id];
 			RTE_MBUF_PREFETCH_TO_FREE(txn->mbuf);

RTE_MBUF_PREFETCH_TO_FREE() doesn't seem relevant here anymore. I don't know if it fits into ci_tx_xmit_cleanup() instead.

drivers/net/intel/common/tx_scalar.h ci_xmit_pkts(struct ci_tx_queue *txq, · line 520

Quoted 5 lines
drivers/net/intel/common/tx_scalar.h +1 -4
-			if (txe->mbuf) {
-				rte_pktmbuf_free_seg(txe->mbuf);
-				txe->mbuf = NULL;
-			}
+			txe->mbuf = NULL;

Already mentioned: Should txe->mbuf be set to NULL in ci_tx_xmit_cleanup() instead of in ci_tx_xmit_pkts()?

drivers/net/intel/common/tx_scalar.h ci_xmit_pkts(struct ci_tx_queue *txq, · line 533

Quoted 7 lines
drivers/net/intel/common/tx_scalar.h +0 -0
 			write_txd(ctx_txd, cd_qw0, cd_qw1);
 
@@ -489,10 +531,7 @@ ci_xmit_pkts(struct ci_tx_queue *txq,
 
 			txn = &sw_ring[txe->next_id];
 			RTE_MBUF_PREFETCH_TO_FREE(txn->mbuf);

RTE_MBUF_PREFETCH_TO_FREE() doesn't seem relevant here anymore. I don't know if it fits into ci_tx_xmit_cleanup() instead.

drivers/net/intel/common/tx_scalar.h ci_xmit_pkts(struct ci_tx_queue *txq, · line 520

Quoted 5 lines
drivers/net/intel/common/tx_scalar.h +1 -4
-			if (txe->mbuf) {
-				rte_pktmbuf_free_seg(txe->mbuf);
-				txe->mbuf = NULL;
-			}
+			txe->mbuf = NULL;

Already mentioned: Should txe->mbuf be set to NULL in ci_tx_xmit_cleanup() instead of in ci_tx_xmit_pkts()?

Quoted 22 lines
drivers/net/intel/common/tx_scalar.h +1 -2
 			ipsec_txd[0] = ipsec_qw0;
 			ipsec_txd[1] = ipsec_qw1;
@@ -507,10 +546,21 @@ ci_xmit_pkts(struct ci_tx_queue *txq,
 			txd = &ci_tx_ring[tx_id];
 			txn = &sw_ring[txe->next_id];
 
-			if (txe->mbuf)
-				rte_pktmbuf_free_seg(txe->mbuf);
 			txe->mbuf = m_seg;
 
+			/* For FAST_FREE: reset mbuf fields while we have it

in cache.

drivers/net/intel/common/tx_scalar.h +1 -0
+			 * FAST_FREE guarantees refcnt=1 and direct mbufs, so

we only

drivers/net/intel/common/tx_scalar.h +1 -0
+			 * need to reset nb_segs and next pointer as per

rte_pktmbuf_prefree_seg.

drivers/net/intel/common/tx_scalar.h +1 -0
+			 * Save next pointer before resetting since we need

it for loop iteration.

drivers/net/intel/common/tx_scalar.h +3 -0
+			 */
+			struct rte_mbuf *next_seg = m_seg->next;
+			if (txq->offloads &
Quoted 1 line

RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE) {

Similar to comment further above: Is txq->offloads or txq->fast_free_mp hotter in the CPU cache here?

Quoted 53 lines
drivers/net/intel/common/tx_scalar.h +6 -0
+				if (m_seg->nb_segs != 1)
+					m_seg->nb_segs = 1;
+				if (next_seg != NULL)
+					m_seg->next = NULL;
+			}
+
 			/* Setup TX Descriptor */
 			/* Calculate segment length, using IPsec callback if

provided */

drivers/net/intel/common/tx_scalar.h +0 -0
 			if (ipsec_ops != NULL)
@@ -528,18 +578,23 @@ ci_xmit_pkts(struct ci_tx_queue *txq,
 					((uint64_t)CI_MAX_DATA_PER_TXD <<

CI_TXD_QW1_TX_BUF_SZ_S) |
((uint64_t)td_tag <<
CI_TXD_QW1_L2TAG1_S);
write_txd(txd, buf_dma_addr,
cmd_type_offset_bsz);

drivers/net/intel/common/tx_scalar.h +1 -0
+				/* txe for this slot has already been written

(e.g. above outside

drivers/net/intel/common/tx_scalar.h +1 -0
+				 * loop), so we write the extra NULL mbuf

pointer for this

drivers/net/intel/common/tx_scalar.h +3 -0
+				 * descriptor after we increment txe below.
+				 */
 
 				buf_dma_addr += CI_MAX_DATA_PER_TXD;
 				slen -= CI_MAX_DATA_PER_TXD;
 
 				tx_id = txe->next_id;
 				txe = txn;
+				txe->mbuf = NULL;
 				txd = &ci_tx_ring[tx_id];
 				txn = &sw_ring[txe->next_id];
 			}
 
 			/* fill the last descriptor with End of Packet (EOP)

bit */

drivers/net/intel/common/tx_scalar.h +1 -1
-			if (m_seg->next == NULL)
+			if (next_seg == NULL)
 				td_cmd |= CI_TX_DESC_CMD_EOP;
 
 			const uint64_t cmd_type_offset_bsz =

CI_TX_DESC_DTYPE_DATA |

drivers/net/intel/common/tx_scalar.h +1 -2
@@ -551,7 +606,7 @@ ci_xmit_pkts(struct ci_tx_queue *txq,
 
 			tx_id = txe->next_id;
 			txe = txn;
-			m_seg = m_seg->next;
+			m_seg = next_seg;
 		} while (m_seg);
 end_pkt:
 		txq->nb_tx_free = (uint16_t)(txq->nb_tx_free - nb_used);
--

2.51.0

#15 lore raw

Re: [PATCH v2] net/intel: optimize for fast-free hint

Quoted 13 lines · Morten Brørup
Quoted 13 lines

From: Bruce Richardson [mailto:[email protected]]
Sent: Wednesday, 8 April 2026 15.25

When the fast-free hint is provided to the driver we know that the mbufs
have refcnt of 1 and are from the same mempool. Therefore, we can optimize a bit for this case by:

* resetting the necessary mbuf fields, ie. nb_seg and next pointer when we are accessing the mbuf on writing the descriptor.
* on cleanup of buffers after transmit, we can just write those buffers straight to the mempool without accessing them.

Quoted 6 lines
Quoted 1 line

A bunch of review thoughts inline below. The ones regarding instrumentation should be fixed. The rest might be irrelevant and/or nonsense.

See inline below. Summary:
* ack for mempool functions for instrumentation
* ack for referencing the mp pointer rather than the offload flags
* nak for setting the mbuf pointers for null, it's unnecessary with this design.
* ack for removing the prefetches

Thanks for the detailed review. Preparing v3 now.

/Bruce

Quoted 90 lines
Quoted 40 lines

---
V2: Fix issues with original submission:
* missed check for NULL mbufs
* fixed issue with freeing directly from sw_ring in scalar path which doesn't work as thats not a flag array of pointers
* fixed missing null assignment in case of large segments for TSO

---
 drivers/net/intel/common/tx.h        | 21 ++++--
 drivers/net/intel/common/tx_scalar.h | 95 ++++++++++++++++++++++------
 2 files changed, 90 insertions(+), 26 deletions(-)
drivers/net/intel/common/tx.h +1 -6
index 283bd58d5d..f2123f069c 100644
@@ -363,13 +363,22 @@ ci_txq_release_all_mbufs(struct ci_tx_queue *txq,
 bool use_ctx)
 		return;
 
 	if (!txq->use_vec_entry) {
-		/* Regular scalar path uses sw_ring with ci_tx_entry */
-		for (uint16_t i = 0; i < txq->nb_tx_desc; i++) {
-			if (txq->sw_ring[i].mbuf != NULL) {
-				rte_pktmbuf_free_seg(txq->sw_ring[i].mbuf);
-				txq->sw_ring[i].mbuf = NULL;
-			}
+		/* Free mbufs from (last_desc_cleaned + 1) to (tx_tail -

1). */

drivers/net/intel/common/tx.h +1 -0
+		const uint16_t start = (txq->last_desc_cleaned + 1) % txq-
Quoted 1 line

nb_tx_desc;

drivers/net/intel/common/tx.h +8 -0
+		const uint16_t nb_desc = txq->nb_tx_desc;
+		const uint16_t end = txq->tx_tail;
+
+		uint16_t i = start;
+		if (end < i) {
+			for (; i < nb_desc; i++)
+				if (txq->sw_ring[i].mbuf != NULL)
+					rte_pktmbuf_free_seg(txq-
Quoted 1 line

sw_ring[i].mbuf);

drivers/net/intel/common/tx.h ci_txq_release_all_mbufs(struct ci_tx_queue *txq, bool use_ctx) · line 383

Quoted 8 lines
drivers/net/intel/common/tx.h +5 -0
+			i = 0;
 		}
+		for (; i < end; i++)
+			if (txq->sw_ring[i].mbuf != NULL)
+				rte_pktmbuf_free_seg(txq->sw_ring[i].mbuf);
+		memset(txq->sw_ring, 0, sizeof(txq->sw_ring[0]) * nb_desc);
 		return;
 	}

The above LGTM.
IIRC, we already discussed it - or something very similar.

Quoted 38 lines
drivers/net/intel/common/tx_scalar.h +3 -5
index 9fcd2e4733..adbc4bafee 100644
@@ -197,16 +197,63 @@ ci_tx_xmit_cleanup(struct ci_tx_queue *txq)
 	const uint16_t rs_idx = (last_desc_cleaned == nb_tx_desc - 1) ?
 			0 :
 			(last_desc_cleaned + 1) >> txq->log2_rs_thresh;
-	uint16_t desc_to_clean_to = (rs_idx << txq->log2_rs_thresh) +
 (txq->tx_rs_thresh - 1);
+	const uint16_t dd_idx = txq->rs_last_id[rs_idx];
+	const uint16_t first_to_clean = rs_idx << txq->log2_rs_thresh;
 
-	/* Check if descriptor is done  */
-	if ((txd[txq->rs_last_id[rs_idx]].cmd_type_offset_bsz &
-			rte_cpu_to_le_64(CI_TXD_QW1_DTYPE_M)) !=
-				rte_cpu_to_le_64(CI_TX_DESC_DTYPE_DESC_DONE))
+	/* Check if descriptor is done - all drivers use 0xF as done

value in bits 3:0 */

drivers/net/intel/common/tx_scalar.h +1 -0
+	if ((txd[dd_idx].cmd_type_offset_bsz &

rte_cpu_to_le_64(CI_TXD_QW1_DTYPE_M)) !=

drivers/net/intel/common/tx_scalar.h +4 -0
+			rte_cpu_to_le_64(CI_TX_DESC_DTYPE_DESC_DONE))
+		/* Descriptor not yet processed by hardware */
 		return -1;
 
+	/* DD bit is set, descriptors are done. Now free the mbufs. */
+	/* Note: nb_tx_desc is guaranteed to be a multiple of

tx_rs_thresh,

drivers/net/intel/common/tx_scalar.h +1 -0
+	 * validated during queue setup. This means cleanup never wraps

around

drivers/net/intel/common/tx_scalar.h +1 -0
+	 * the ring within a single burst (e.g., ring=256, rs_thresh=32

gives

drivers/net/intel/common/tx_scalar.h +4 -0
+	 * bursts of 0-31, 32-63, ..., 224-255).
+	 */
+	const uint16_t nb_to_clean = txq->tx_rs_thresh;
+	struct ci_tx_entry *sw_ring = txq->sw_ring;
Quoted 9 lines
Quoted 4 lines
drivers/net/intel/common/tx_scalar.h +3 -0
+
+	if (txq->offloads & RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE) {
+		/* FAST_FREE path: mbufs are already reset, just return to

pool */

Depending on which cache lines from txq have already been loaded, unless txq->offloads is hot in the CPU cache and txq->fast_free_mp is not, consider testing (mp != NULL) instead of (txq->offloads & RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE). Like here:
https://elixir.bootlin.com/dpdk/v26.03/source/drivers/net/intel/common/tx.h#L281

Ack. The offloads are on the second cacheline of the txq, I think, while the fast_free_mp is on the first one. Therefore, checking the mp pointer for null is slightly better.

Quoted 14 lines
Quoted 14 lines
drivers/net/intel/common/tx_scalar.h +11 -0
+		void *free[CI_TX_MAX_FREE_BUF_SZ];
+		uint16_t nb_free = 0;
+
+		/* Get cached mempool pointer, or cache it on first use */
+		struct rte_mempool *mp =
+			likely(txq->fast_free_mp != (void *)UINTPTR_MAX) ?
+			txq->fast_free_mp :
+			(txq->fast_free_mp = sw_ring[dd_idx].mbuf->pool);
+
+		/* Pack non-NULL mbufs in-place at start of sw_ring range.
+		 * No modulo needed in loop since we're guaranteed not to

wrap.

drivers/net/intel/common/tx_scalar.h +2 -0
+		 */
+		for (uint16_t i = 0; i < nb_to_clean; i++) {
Quoted 8 lines

drivers/net/intel/common/tx_scalar.h ci_tx_xmit_cleanup(struct ci_tx_queue *txq) · line 236

Quoted 5 lines
drivers/net/intel/common/tx_scalar.h +1 -0
+			struct rte_mbuf *m = sw_ring[first_to_clean +

i].mbuf;

drivers/net/intel/common/tx_scalar.h +3 -0
+			if (m == NULL)
+				continue;
+			free[nb_free++] = m;

Should sw_ring[first_to_clean + i].mbuf be set to NULL here, instead of in ci_xmit_pkts()? I don't know, just want you to consider it.

No, we are ok just setting it there. The changes in this patch to the ci_txq_release_all_mbufs() function mean that we assume all buffers between tx_tail and last_desc_cleaned are invalid/freed (i.e. we free from last-cleaned to tx_tail only), and so we don't need to set the invalid slots to NULL. The NULL sentinel is only used where we have a valid slot but which does not have an associated mbuf.

Quoted 7 lines

drivers/net/intel/common/tx_scalar.h

Quoted 3 lines
drivers/net/intel/common/tx_scalar.h +2 -0
+			if (unlikely(nb_free == CI_TX_MAX_FREE_BUF_SZ)) {
+				rte_mempool_put_bulk(mp, free, nb_free);

rte_mempool_put_bulk() -> rte_mbuf_raw_free_bulk(), for instrumentation.

Ack

Quoted 11 lines

drivers/net/intel/common/tx_scalar.h ci_tx_xmit_cleanup(struct ci_tx_queue *txq) · line 245

Quoted 8 lines
drivers/net/intel/common/tx_scalar.h +5 -0
+				nb_free = 0;
+			}
+		}
+
+		/* Bulk return to mempool using packed sw_ring entries

directly */

drivers/net/intel/common/tx_scalar.h +2 -0
+		if (nb_free > 0)
+			rte_mempool_put_bulk(mp, free, nb_free);

rte_mempool_put_bulk() -> rte_mbuf_raw_free_bulk(), for instrumentation.

Ack

Quoted 13 lines

drivers/net/intel/common/tx_scalar.h ci_tx_xmit_cleanup(struct ci_tx_queue *txq) · line 251

Quoted 9 lines
drivers/net/intel/common/tx_scalar.h +2 -0
+	} else {
+		/* Non-FAST_FREE path: use prefree_seg for refcount checks

*/

drivers/net/intel/common/tx_scalar.h +2 -0
+		for (uint16_t i = 0; i < nb_to_clean; i++) {
+			struct rte_mbuf *m = sw_ring[first_to_clean +

i].mbuf;

drivers/net/intel/common/tx_scalar.h +2 -0
+			if (m != NULL)
+				rte_pktmbuf_free_seg(m);

Should sw_ring[first_to_clean + i].mbuf be set to NULL here, instead of in ci_xmit_pkts()? I don't know, just want you to consider it.

As above, reason should hold for both fast-free and non-fast-free paths.

Quoted 21 lines
Quoted 21 lines
drivers/net/intel/common/tx_scalar.h +3 -0
+		}
+	}
+
 	/* Update the txq to reflect the last descriptor that was cleaned

*/

drivers/net/intel/common/tx_scalar.h +1 -3
-	txq->last_desc_cleaned = desc_to_clean_to;
+	txq->last_desc_cleaned = first_to_clean + txq->tx_rs_thresh - 1;
 	txq->nb_tx_free += txq->tx_rs_thresh;
 
 	return 0;
@@ -450,8 +497,6 @@ ci_xmit_pkts(struct ci_tx_queue *txq,
 			txd = &ci_tx_ring[tx_id];
 			tx_id = txe->next_id;
 
-			if (txe->mbuf)
-				rte_pktmbuf_free_seg(txe->mbuf);
 			txe->mbuf = tx_pkt;
 			/* Setup TX Descriptor */
 			td_cmd |= CI_TX_DESC_CMD_EOP;
@@ -472,10 +517,7 @@ ci_xmit_pkts(struct ci_tx_queue *txq,
 
Quoted 5 lines

drivers/net/intel/common/tx_scalar.h ci_xmit_pkts(struct ci_tx_queue *txq, · line 519

Quoted 2 lines
drivers/net/intel/common/tx_scalar.h +0 -0
 			txn = &sw_ring[txe->next_id];
 			RTE_MBUF_PREFETCH_TO_FREE(txn->mbuf);

RTE_MBUF_PREFETCH_TO_FREE() doesn't seem relevant here anymore. I don't know if it fits into ci_tx_xmit_cleanup() instead.

Yes, dropping.

Quoted 39 lines

drivers/net/intel/common/tx_scalar.h ci_xmit_pkts(struct ci_tx_queue *txq, · line 520

Quoted 6 lines
drivers/net/intel/common/tx_scalar.h +1 -4
-			if (txe->mbuf) {
-				rte_pktmbuf_free_seg(txe->mbuf);
-				txe->mbuf = NULL;
-			}
+			txe->mbuf = NULL;

Already mentioned: Should txe->mbuf be set to NULL in ci_tx_xmit_cleanup() instead of in ci_tx_xmit_pkts()?

drivers/net/intel/common/tx_scalar.h ci_xmit_pkts(struct ci_tx_queue *txq, · line 533

Quoted 7 lines
drivers/net/intel/common/tx_scalar.h +0 -0
 			write_txd(ctx_txd, cd_qw0, cd_qw1);
 
@@ -489,10 +531,7 @@ ci_xmit_pkts(struct ci_tx_queue *txq,
 
 			txn = &sw_ring[txe->next_id];
 			RTE_MBUF_PREFETCH_TO_FREE(txn->mbuf);

RTE_MBUF_PREFETCH_TO_FREE() doesn't seem relevant here anymore. I don't know if it fits into ci_tx_xmit_cleanup() instead.

drivers/net/intel/common/tx_scalar.h ci_xmit_pkts(struct ci_tx_queue *txq, · line 520

Quoted 5 lines
drivers/net/intel/common/tx_scalar.h +1 -4
-			if (txe->mbuf) {
-				rte_pktmbuf_free_seg(txe->mbuf);
-				txe->mbuf = NULL;
-			}
+			txe->mbuf = NULL;

Already mentioned: Should txe->mbuf be set to NULL in ci_tx_xmit_cleanup() instead of in ci_tx_xmit_pkts()?

Quoted 11 lines
drivers/net/intel/common/tx_scalar.h +0 -2
 			ipsec_txd[0] = ipsec_qw0;
 			ipsec_txd[1] = ipsec_qw1;
@@ -507,10 +546,21 @@ ci_xmit_pkts(struct ci_tx_queue *txq,
 			txd = &ci_tx_ring[tx_id];
 			txn = &sw_ring[txe->next_id];
 
-			if (txe->mbuf)
-				rte_pktmbuf_free_seg(txe->mbuf);
 			txe->mbuf = m_seg;
 
Quoted 15 lines
Quoted 11 lines
drivers/net/intel/common/tx_scalar.h +1 -0
+			/* For FAST_FREE: reset mbuf fields while we have it

in cache.

drivers/net/intel/common/tx_scalar.h +1 -0
+			 * FAST_FREE guarantees refcnt=1 and direct mbufs, so

we only

drivers/net/intel/common/tx_scalar.h +1 -0
+			 * need to reset nb_segs and next pointer as per

rte_pktmbuf_prefree_seg.

drivers/net/intel/common/tx_scalar.h +1 -0
+			 * Save next pointer before resetting since we need

it for loop iteration.

drivers/net/intel/common/tx_scalar.h +3 -0
+			 */
+			struct rte_mbuf *next_seg = m_seg->next;
+			if (txq->offloads &
Quoted 1 line

RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE) {

Similar to comment further above: Is txq->offloads or txq->fast_free_mp hotter in the CPU cache here?

Updating as you suggest.

Quoted 54 lines
Quoted 53 lines
drivers/net/intel/common/tx_scalar.h +6 -0
+				if (m_seg->nb_segs != 1)
+					m_seg->nb_segs = 1;
+				if (next_seg != NULL)
+					m_seg->next = NULL;
+			}
+
 			/* Setup TX Descriptor */
 			/* Calculate segment length, using IPsec callback if

provided */

drivers/net/intel/common/tx_scalar.h +0 -0
 			if (ipsec_ops != NULL)
@@ -528,18 +578,23 @@ ci_xmit_pkts(struct ci_tx_queue *txq,
 					((uint64_t)CI_MAX_DATA_PER_TXD <<

CI_TXD_QW1_TX_BUF_SZ_S) |
((uint64_t)td_tag <<
CI_TXD_QW1_L2TAG1_S);
write_txd(txd, buf_dma_addr,
cmd_type_offset_bsz);

drivers/net/intel/common/tx_scalar.h +1 -0
+				/* txe for this slot has already been written

(e.g. above outside

drivers/net/intel/common/tx_scalar.h +1 -0
+				 * loop), so we write the extra NULL mbuf

pointer for this

drivers/net/intel/common/tx_scalar.h +3 -0
+				 * descriptor after we increment txe below.
+				 */
 
 				buf_dma_addr += CI_MAX_DATA_PER_TXD;
 				slen -= CI_MAX_DATA_PER_TXD;
 
 				tx_id = txe->next_id;
 				txe = txn;
+				txe->mbuf = NULL;
 				txd = &ci_tx_ring[tx_id];
 				txn = &sw_ring[txe->next_id];
 			}
 
 			/* fill the last descriptor with End of Packet (EOP)

bit */

drivers/net/intel/common/tx_scalar.h +1 -1
-			if (m_seg->next == NULL)
+			if (next_seg == NULL)
 				td_cmd |= CI_TX_DESC_CMD_EOP;
 
 			const uint64_t cmd_type_offset_bsz =

CI_TX_DESC_DTYPE_DATA |

drivers/net/intel/common/tx_scalar.h +1 -2
@@ -551,7 +606,7 @@ ci_xmit_pkts(struct ci_tx_queue *txq,
 
 			tx_id = txe->next_id;
 			txe = txn;
-			m_seg = m_seg->next;
+			m_seg = next_seg;
 		} while (m_seg);
 end_pkt:
 		txq->nb_tx_free = (uint16_t)(txq->nb_tx_free - nb_used);
--

2.51.0

#16 lore raw

PATCH v3 1/1 net/intel: optimize for fast-free hint

In reply to Bruce Richardson

When the fast-free hint is provided to the driver we know that the mbufs have refcnt of 1 and are from the same mempool. Therefore, we can optimize a bit for this case by:

* resetting the necessary mbuf fields, ie. nb_seg and next pointer when we are accessing the mbuf on writing the descriptor.
* on cleanup of buffers after transmit, we can just write those buffers straight to the mempool without accessing them.
* when doing buffer cleanup, we can follow the model of the vector driver where we cleanup in fixed bursts which don't wrap, and where we track the mbufs between tx_tail and last_cleaned as invalid, so we don't need to set them to NULL. [We only set to NULL valid mbuf slots where we have a context descriptor or segment not backed by an mbuf]

---
v3:
* used mbuf_raw_free_bulk rather than mempool function directly
* check for fast_free via mp pointer rather than flags
* remove unnecessary prefetches

V2: Fix issues with original submission:
* missed check for NULL mbufs
* fixed issue with freeing directly from sw_ring in scalar path which doesn't work as thats not a flag array of pointers
* fixed missing null assignment in case of large segments for TSO

---
 drivers/net/intel/common/tx.h        | 21 ++++--
 drivers/net/intel/common/tx_scalar.h | 99 +++++++++++++++++++++-------
 2 files changed, 92 insertions(+), 28 deletions(-)
drivers/net/intel/common/tx.h +15 -6
index 23a8c39cf2..5fe71aed12 100644
@@ -329,13 +329,22 @@ ci_txq_release_all_mbufs(struct ci_tx_queue *txq, bool use_ctx)
 		return;
 
 	if (!txq->use_vec_entry) {
-		/* Regular scalar path uses sw_ring with ci_tx_entry */
-		for (uint16_t i = 0; i < txq->nb_tx_desc; i++) {
-			if (txq->sw_ring[i].mbuf != NULL) {
-				rte_pktmbuf_free_seg(txq->sw_ring[i].mbuf);
-				txq->sw_ring[i].mbuf = NULL;
-			}
+		/* Free mbufs from (last_desc_cleaned + 1) to (tx_tail - 1). */
+		const uint16_t start = (txq->last_desc_cleaned + 1) % txq->nb_tx_desc;
+		const uint16_t nb_desc = txq->nb_tx_desc;
+		const uint16_t end = txq->tx_tail;
+
+		uint16_t i = start;
+		if (end < i) {
+			for (; i < nb_desc; i++)
+				if (txq->sw_ring[i].mbuf != NULL)
+					rte_pktmbuf_free_seg(txq->sw_ring[i].mbuf);
+			i = 0;
 		}
+		for (; i < end; i++)
+			if (txq->sw_ring[i].mbuf != NULL)
+				rte_pktmbuf_free_seg(txq->sw_ring[i].mbuf);
+		memset(txq->sw_ring, 0, sizeof(txq->sw_ring[0]) * nb_desc);
 		return;
 	}
 
drivers/net/intel/common/tx_scalar.h +77 -22

158 lines not on this page · raw message

Signature · 1 line
2.53.0
Acked-by on v3 1/1
#17 lore raw

RE: [PATCH v3] net/intel: optimize for fast-free hint

Quoted 2 lines

Subject: [PATCH v3] net/intel: optimize for fast-free hint

snip

Quoted 41 lines
drivers/net/intel/common/tx_scalar.h +3 -5
index 9fcd2e4733..d27df34dfa 100644
@@ -197,16 +197,64 @@ ci_tx_xmit_cleanup(struct ci_tx_queue *txq)
 	const uint16_t rs_idx = (last_desc_cleaned == nb_tx_desc - 1) ?
 			0 :
 			(last_desc_cleaned + 1) >> txq->log2_rs_thresh;
-	uint16_t desc_to_clean_to = (rs_idx << txq->log2_rs_thresh) + (txq-
 >tx_rs_thresh - 1);
+	const uint16_t dd_idx = txq->rs_last_id[rs_idx];
+	const uint16_t first_to_clean = rs_idx << txq->log2_rs_thresh;
 
-	/* Check if descriptor is done  */
-	if ((txd[txq->rs_last_id[rs_idx]].cmd_type_offset_bsz &
-			rte_cpu_to_le_64(CI_TXD_QW1_DTYPE_M)) !=
-
 	rte_cpu_to_le_64(CI_TX_DESC_DTYPE_DESC_DONE))
+	/* Check if descriptor is done - all drivers use 0xF as done value in bits

3:0 */

drivers/net/intel/common/tx_scalar.h +1 -0
+	if ((txd[dd_idx].cmd_type_offset_bsz &

rte_cpu_to_le_64(CI_TXD_QW1_DTYPE_M)) !=

drivers/net/intel/common/tx_scalar.h +5 -0
+			rte_cpu_to_le_64(CI_TX_DESC_DTYPE_DESC_DONE))
+		/* Descriptor not yet processed by hardware */
 		return -1;
 
+	/* DD bit is set, descriptors are done. Now free the mbufs. */
+	/* Note: nb_tx_desc is guaranteed to be a multiple of tx_rs_thresh,
+	 * validated during queue setup. This means cleanup never wraps

around

drivers/net/intel/common/tx_scalar.h +9 -0
+	 * the ring within a single burst (e.g., ring=256, rs_thresh=32 gives
+	 * bursts of 0-31, 32-63, ..., 224-255).
+	 */
+	const uint16_t nb_to_clean = txq->tx_rs_thresh;
+	struct ci_tx_entry *sw_ring = txq->sw_ring;
+
+	/* fast_free_mp is NULL only when the fast free is disabled*/
+	if (txq->fast_free_mp != NULL) {
+		/* FAST_FREE path: mbufs are already reset, just return to

pool */

drivers/net/intel/common/tx_scalar.h ci_tx_xmit_cleanup(struct ci_tx_queue *txq) · line 228

Quoted 8 lines
drivers/net/intel/common/tx_scalar.h +8 -0
+		struct rte_mbuf *free[CI_TX_MAX_FREE_BUF_SZ];
+		uint16_t nb_free = 0;
+
+		/* Get cached mempool pointer, or cache it on first use */
+		struct rte_mempool *mp =
+			likely(txq->fast_free_mp != (void *)UINTPTR_MAX) ?
+			txq->fast_free_mp :
+			(txq->fast_free_mp = sw_ring[dd_idx].mbuf->pool);

Is the mbuf you are dereferencing here guaranteed to be non NULL?

Other than that concern:

Quoted 6 lines
drivers/net/intel/common/tx_scalar.h +3 -0
+
+		/* Pack non-NULL mbufs in-place at start of sw_ring range.
+		 * No modulo needed in loop since we're guaranteed not to

wrap.

drivers/net/intel/common/tx_scalar.h +2 -0
+		 */
+		for (uint16_t i = 0; i < nb_to_clean; i++) {
#18 lore raw

Re: [PATCH v3] net/intel: optimize for fast-free hint

Quoted 48 lines · Ciara
Quoted 2 lines

Subject: [PATCH v3] net/intel: optimize for fast-free hint

snip

Quoted 43 lines
drivers/net/intel/common/tx_scalar.h +3 -5
index 9fcd2e4733..d27df34dfa 100644
@@ -197,16 +197,64 @@ ci_tx_xmit_cleanup(struct ci_tx_queue *txq)
 	const uint16_t rs_idx = (last_desc_cleaned == nb_tx_desc - 1) ?
 			0 :
 			(last_desc_cleaned + 1) >> txq->log2_rs_thresh;
-	uint16_t desc_to_clean_to = (rs_idx << txq->log2_rs_thresh) + (txq-
 >tx_rs_thresh - 1);
+	const uint16_t dd_idx = txq->rs_last_id[rs_idx];
+	const uint16_t first_to_clean = rs_idx << txq->log2_rs_thresh;
 
-	/* Check if descriptor is done  */
-	if ((txd[txq->rs_last_id[rs_idx]].cmd_type_offset_bsz &
-			rte_cpu_to_le_64(CI_TXD_QW1_DTYPE_M)) !=
-
 	rte_cpu_to_le_64(CI_TX_DESC_DTYPE_DESC_DONE))
+	/* Check if descriptor is done - all drivers use 0xF as done value in bits

3:0 */

drivers/net/intel/common/tx_scalar.h +1 -0
+	if ((txd[dd_idx].cmd_type_offset_bsz &

rte_cpu_to_le_64(CI_TXD_QW1_DTYPE_M)) !=

drivers/net/intel/common/tx_scalar.h +5 -0
+			rte_cpu_to_le_64(CI_TX_DESC_DTYPE_DESC_DONE))
+		/* Descriptor not yet processed by hardware */
 		return -1;
 
+	/* DD bit is set, descriptors are done. Now free the mbufs. */
+	/* Note: nb_tx_desc is guaranteed to be a multiple of tx_rs_thresh,
+	 * validated during queue setup. This means cleanup never wraps

around

drivers/net/intel/common/tx_scalar.h +9 -0
+	 * the ring within a single burst (e.g., ring=256, rs_thresh=32 gives
+	 * bursts of 0-31, 32-63, ..., 224-255).
+	 */
+	const uint16_t nb_to_clean = txq->tx_rs_thresh;
+	struct ci_tx_entry *sw_ring = txq->sw_ring;
+
+	/* fast_free_mp is NULL only when the fast free is disabled*/
+	if (txq->fast_free_mp != NULL) {
+		/* FAST_FREE path: mbufs are already reset, just return to

pool */

drivers/net/intel/common/tx_scalar.h +2 -0
+		struct rte_mbuf *free[CI_TX_MAX_FREE_BUF_SZ];
+		uint16_t nb_free = 0;
Quoted 9 lines

drivers/net/intel/common/tx_scalar.h ci_tx_xmit_cleanup(struct ci_tx_queue *txq) · line 228

Quoted 6 lines
drivers/net/intel/common/tx_scalar.h +6 -0
+
+		/* Get cached mempool pointer, or cache it on first use */
+		struct rte_mempool *mp =
+			likely(txq->fast_free_mp != (void *)UINTPTR_MAX) ?
+			txq->fast_free_mp :
+			(txq->fast_free_mp = sw_ring[dd_idx].mbuf->pool);

Is the mbuf you are dereferencing here guaranteed to be non NULL?

Good point. It almost certainly is, but checking through the logic of the code, the one possible error condition is where we have the last segment of the packet being a large segment using TSO which is to be split across multiple data descriptors. In that case (only) the mbuf pointer can be NULL. In all other cases, it should be valid, because the DD bit can only be set on a data descriptor.

Rather than trying to fix that issue here on cleanup, I think the better solution is to adjust the large segment handling so that the last descriptor of the segment, rather than the first, has the mbuf pointer. That way we can always know that the descriptor that has DD set on it will have a valid mbuf. Will implement this as a separate patch in v4.

/Bruce

#19 lore raw

v4 cover net/intel: optimize for fast-free hint

In reply to Bruce Richardson

When the fast-free hint is provided to the driver we know that the mbufs have refcnt of 1 and are from the same mempool. Therefore, we can optimize a bit for this case even in the scalar path of our drivers.

---
v4:
* add precursor patch to adjust mbuf pointers so that the DD bit is written to a descriptor with a valid mbuf pointer associated with it.

v3:
* used mbuf_raw_free_bulk rather than mempool function directly
* check for fast_free via mp pointer rather than flags
* remove unnecessary prefetches

V2: Fix issues with original submission:
* missed check for NULL mbufs
* fixed issue with freeing directly from sw_ring in scalar path which doesn't work as thats not a flag array of pointers
* fixed missing null assignment in case of large segments for TSO

Bruce Richardson (2):
net/intel: write mbuf for last Tx desc of segment net/intel: optimize for fast-free hint

 drivers/net/intel/common/tx.h        | 21 ++++--
 drivers/net/intel/common/tx_scalar.h | 98 +++++++++++++++++++++-------
 2 files changed, 90 insertions(+), 29 deletions(-)

--
2.53.0

#20 lore raw

PATCH v4 1/2 net/intel: write mbuf for last Tx desc of segment

When a single mbuf segment has more data than can be handled by a single Tx data descriptor in the TSO case, adjust how the storing of mbufs is being done. Rather than putting the mbuf pointer in the first slot for that segment, store it in the last slot instead. This guarantees for us that the descriptor for which we have desc-done (DD) writeback always has a valid mbuf associated with it.

---
 drivers/net/intel/common/tx_scalar.h | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/net/intel/common/tx_scalar.h +2 -1
index 9fcd2e4733..7809bd53e8 100644
@@ -509,7 +509,6 @@ ci_xmit_pkts(struct ci_tx_queue *txq,
 
 			if (txe->mbuf)
 				rte_pktmbuf_free_seg(txe->mbuf);
-			txe->mbuf = m_seg;
 
 			/* Setup TX Descriptor */
 			/* Calculate segment length, using IPsec callback if provided */
@@ -528,6 +527,7 @@ ci_xmit_pkts(struct ci_tx_queue *txq,
 					((uint64_t)CI_MAX_DATA_PER_TXD << CI_TXD_QW1_TX_BUF_SZ_S) |
 					((uint64_t)td_tag << CI_TXD_QW1_L2TAG1_S);
 				write_txd(txd, buf_dma_addr, cmd_type_offset_bsz);
+				txe->mbuf = NULL;
 
 				buf_dma_addr += CI_MAX_DATA_PER_TXD;
 				slen -= CI_MAX_DATA_PER_TXD;
@@ -548,6 +548,7 @@ ci_xmit_pkts(struct ci_tx_queue *txq,
 				((uint64_t)slen << CI_TXD_QW1_TX_BUF_SZ_S) |
 				((uint64_t)td_tag << CI_TXD_QW1_L2TAG1_S);
 			write_txd(txd, buf_dma_addr, cmd_type_offset_bsz);
+			txe->mbuf = m_seg;
 
 			tx_id = txe->next_id;
 			txe = txn;
Signature · 1 line
2.53.0
Acked-by on v4 1/2
#21 lore raw

RE: [PATCH v4 1/2] net/intel: write mbuf for last Tx desc of segment

Quoted 10 lines

Subject: [PATCH v4 1/2] net/intel: write mbuf for last Tx desc of segment

When a single mbuf segment has more data than can be handled by a single Tx data descriptor in the TSO case, adjust how the storing of mbufs is being done. Rather than putting the mbuf pointer in the first slot for that segment, store it in the last slot instead. This guarantees for us that the descriptor for which we have desc-done (DD) writeback always has a valid mbuf associated with it.

The fix looks good to me.

Quoted 41 lines
---
 drivers/net/intel/common/tx_scalar.h | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/net/intel/common/tx_scalar.h +0 -1
index 9fcd2e4733..7809bd53e8 100644
@@ -509,7 +509,6 @@ ci_xmit_pkts(struct ci_tx_queue *txq,
 
 			if (txe->mbuf)
 				rte_pktmbuf_free_seg(txe->mbuf);
-			txe->mbuf = m_seg;
 
 			/* Setup TX Descriptor */
 			/* Calculate segment length, using IPsec callback if

provided */

drivers/net/intel/common/tx_scalar.h +0 -0
@@ -528,6 +527,7 @@ ci_xmit_pkts(struct ci_tx_queue *txq,
 					((uint64_t)CI_MAX_DATA_PER_TXD

<< CI_TXD_QW1_TX_BUF_SZ_S) |
((uint64_t)td_tag <<
CI_TXD_QW1_L2TAG1_S);
write_txd(txd, buf_dma_addr,
cmd_type_offset_bsz);

drivers/net/intel/common/tx_scalar.h +1 -0
+				txe->mbuf = NULL;
 
 				buf_dma_addr += CI_MAX_DATA_PER_TXD;
 				slen -= CI_MAX_DATA_PER_TXD;
@@ -548,6 +548,7 @@ ci_xmit_pkts(struct ci_tx_queue *txq,
 				((uint64_t)slen <<

CI_TXD_QW1_TX_BUF_SZ_S) |
((uint64_t)td_tag <<
CI_TXD_QW1_L2TAG1_S);

drivers/net/intel/common/tx_scalar.h +1 -1
 			write_txd(txd, buf_dma_addr, cmd_type_offset_bsz);
+			txe->mbuf = m_seg;
 
 			tx_id = txe->next_id;
 			txe = txn;
--

2.53.0

#22 lore raw

PATCH v4 2/2 net/intel: optimize for fast-free hint

In reply to Bruce Richardson

When the fast-free hint is provided to the driver we know that the mbufs have refcnt of 1 and are from the same mempool. Therefore, we can optimize a bit for this case by:

* resetting the necessary mbuf fields, ie. nb_seg and next pointer when we are accessing the mbuf on writing the descriptor.
* on cleanup of buffers after transmit, we can just write those buffers straight to the mempool without accessing them.
* when doing buffer cleanup, we can follow the model of the vector driver where we cleanup in fixed bursts which don't wrap, and where we track the mbufs between tx_tail and last_cleaned as invalid, so we don't need to set them to NULL. [We only set to NULL valid mbuf slots where we have a context descriptor or segment not backed by an mbuf]

---
 drivers/net/intel/common/tx.h        | 21 ++++--
 drivers/net/intel/common/tx_scalar.h | 95 +++++++++++++++++++++-------
 2 files changed, 88 insertions(+), 28 deletions(-)
drivers/net/intel/common/tx.h +15 -6
index 23a8c39cf2..5fe71aed12 100644
@@ -329,13 +329,22 @@ ci_txq_release_all_mbufs(struct ci_tx_queue *txq, bool use_ctx)
 		return;
 
 	if (!txq->use_vec_entry) {
-		/* Regular scalar path uses sw_ring with ci_tx_entry */
-		for (uint16_t i = 0; i < txq->nb_tx_desc; i++) {
-			if (txq->sw_ring[i].mbuf != NULL) {
-				rte_pktmbuf_free_seg(txq->sw_ring[i].mbuf);
-				txq->sw_ring[i].mbuf = NULL;
-			}
+		/* Free mbufs from (last_desc_cleaned + 1) to (tx_tail - 1). */
+		const uint16_t start = (txq->last_desc_cleaned + 1) % txq->nb_tx_desc;
+		const uint16_t nb_desc = txq->nb_tx_desc;
+		const uint16_t end = txq->tx_tail;
+
+		uint16_t i = start;
+		if (end < i) {
+			for (; i < nb_desc; i++)
+				if (txq->sw_ring[i].mbuf != NULL)
+					rte_pktmbuf_free_seg(txq->sw_ring[i].mbuf);
+			i = 0;
 		}
+		for (; i < end; i++)
+			if (txq->sw_ring[i].mbuf != NULL)
+				rte_pktmbuf_free_seg(txq->sw_ring[i].mbuf);
+		memset(txq->sw_ring, 0, sizeof(txq->sw_ring[0]) * nb_desc);
 		return;
 	}
 
drivers/net/intel/common/tx_scalar.h +73 -22

141 lines not on this page · raw message

Signature · 1 line
2.53.0
#23 lore raw

RE: [PATCH v4 0/2] net/intel: optimize for fast-free hint

In reply to Bruce Richardson
Quoted 30 lines

From: Bruce Richardson [mailto:[email protected]]
Sent: Tuesday, 2 June 2026 17.45

When the fast-free hint is provided to the driver we know that the mbufs
have refcnt of 1 and are from the same mempool. Therefore, we can optimize a bit for this case even in the scalar path of our drivers.

---
v4:
* add precursor patch to adjust mbuf pointers so that the DD bit is written to a descriptor with a valid mbuf pointer associated with it.

v3:
* used mbuf_raw_free_bulk rather than mempool function directly
* check for fast_free via mp pointer rather than flags
* remove unnecessary prefetches

V2: Fix issues with original submission:
* missed check for NULL mbufs
* fixed issue with freeing directly from sw_ring in scalar path which doesn't work as thats not a flag array of pointers
* fixed missing null assignment in case of large segments for TSO

Bruce Richardson (2):
net/intel: write mbuf for last Tx desc of segment net/intel: optimize for fast-free hint

Quoted 6 lines
 drivers/net/intel/common/tx.h        | 21 ++++--
 drivers/net/intel/common/tx_scalar.h | 98 +++++++++++++++++++++-------
 2 files changed, 90 insertions(+), 29 deletions(-)

--
2.53.0

Good catch by Ciara, and good solution to it.

#24 lore raw

Re: [PATCH v4 0/2] net/intel: optimize for fast-free hint

Quoted 34 lines · Morten Brørup
Quoted 34 lines

From: Bruce Richardson [mailto:[email protected]]
Sent: Tuesday, 2 June 2026 17.45

When the fast-free hint is provided to the driver we know that the mbufs
have refcnt of 1 and are from the same mempool. Therefore, we can optimize a bit for this case even in the scalar path of our drivers.

---
v4:
* add precursor patch to adjust mbuf pointers so that the DD bit is written to a descriptor with a valid mbuf pointer associated with it.

v3:
* used mbuf_raw_free_bulk rather than mempool function directly
* check for fast_free via mp pointer rather than flags
* remove unnecessary prefetches

V2: Fix issues with original submission:
* missed check for NULL mbufs
* fixed issue with freeing directly from sw_ring in scalar path which doesn't work as thats not a flag array of pointers
* fixed missing null assignment in case of large segments for TSO

Bruce Richardson (2):
net/intel: write mbuf for last Tx desc of segment net/intel: optimize for fast-free hint

 drivers/net/intel/common/tx.h        | 21 ++++--
 drivers/net/intel/common/tx_scalar.h | 98 +++++++++++++++++++++-------
 2 files changed, 90 insertions(+), 29 deletions(-)
Quoted 6 lines
Quoted 2 lines

--
2.53.0

Good catch by Ciara, and good solution to it.

Applied to dpdk-next-net-intel.
Thanks for all the reviews folks. Quick tests on next-net-intel tree with scalar driver show a nice perf bump from this change.

/Bruce

#25 lore raw
In reply to Morten Brørup
Quoted 15 lines · Morten Brørup
Quoted 9 lines
Quoted 1 line

If I'm not mistaken, the mbuf library is not a barrier for fast-

freeing

Quoted 1 line

segmented packet mbufs, and thus fast-free of jumbo frames is

possible.

Quoted 5 lines

We need a driver developer to confirm that my suggested approach - resetting the mbuf fields, incl. 'm->nb_segs' and 'm->next', when preparing the Tx descriptor - is viable.

Quoted 5 lines

Excellent analysis, Morten. If I get a chance some time this release cycle,
I will try implementing this change in our drivers, see if any difference
is made.

Quoted 5 lines

Bruce,

Have you had a chance to look into the driver change requirements? If not, could you please try scratching the surface, to build a gut feeling.

Prototype implementation done and posted as a patch to this thread[1]. (It applies on top of my "combine multiple Intel scalar Tx paths" patchset[2]).

The logic is not terribly complicated, and running a quick perf test with scattered Tx enabled (to force use of scalar path, this optimization doesn't apply to vector path*) shows a small perf increase from it using testpmd iofwd.

/Bruce

[1] https://patches.dpdk.org/project/dpdk/patch/[email protected]/
[2] https://patches.dpdk.org/project/dpdk/list/?series=37033

* Vector path already has support for fast-free, and disallows use of scattered packets, so there is nothing to optimize for here.

#26 lore raw
In reply to Morten Brørup
Quoted 250 lines

Executive Summary:

My analysis shows that the mbuf library is not a barrier for fast-freeing segmented packet mbufs, and thus fast-free of jumbo frames is possible.

Detailed Analysis:

The purpose of the mbuf fast-free Tx optimization is to reduce rte_pktmbuf_free_seg() to something much simpler in the ethdev drivers, by eliminating the code path related to indirect mbufs. Optimally, we want to simplify the ethdev driver's function that frees the transmitted mbufs, so it can free them directly to their mempool without accessing the mbufs themselves.

If the driver cannot access the mbuf itself, it cannot determine which mempool it belongs to.
We don't want the driver to access every mbuf being freed; but if all mbufs of a Tx queue belong to the same mempool, the driver can determine which mempool by looking into just one of the mbufs.

REQUIREMENT 1: The mbufs of a Tx queue must come from the same mempool.

When an mbuf is freed to its mempool, some of the fields in the mbuf must be initialized.
So, for fast-free, this must be done by the driver's function that prepares the Tx descriptor.
This is a requirement to the driver, not a requirement to the application.

Now, let's dig into the code for freeing an mbuf.
Note: For readability purposes, I'll cut out some code and comments unrelated to this topic.

static __rte_always_inline void
rte_pktmbuf_free_seg(struct rte_mbuf *m)

{
	m = rte_pktmbuf_prefree_seg(m);
	if (likely(m != NULL))
		rte_mbuf_raw_free(m);
}

rte_mbuf_raw_free(m) is simple, so nothing to gain there:

/**
* Put mbuf back into its original mempool.
*
* The caller must ensure that the mbuf is direct and properly
* reinitialized (refcnt=1, next=NULL, nb_segs=1), as done by
* rte_pktmbuf_prefree_seg().
*/
static __rte_always_inline void rte_mbuf_raw_free(struct rte_mbuf *m)

{
	rte_mbuf_history_mark(m, RTE_MBUF_HISTORY_OP_LIB_FREE);
	rte_mempool_put(m->pool, m);
}

Note that the description says that the mbuf must be direct. This is not entirely accurate; the mbuf is allowed to use a pinned external buffer, if the mbuf holds the only reference to it. (Most of the mbuf library functions have this documentation inaccuracy, which should be fixed some day.)

So, the fast-free optimization really comes down to rte_pktmbuf_prefree_seg(m), which must not return NULL.

Let's dig into that.

/**
* Decrease reference counter and unlink a mbuf segment *
* This function does the same than a free, except that it does not
* return the segment to its pool.
* It decreases the reference counter, and if it reaches 0, it is
* detached from its parent for an indirect mbuf.
*
* @return
* - (m) if it is the last reference. It can be recycled or freed.
* - (NULL) if the mbuf still has remaining references on it.
*/
static __rte_always_inline struct rte_mbuf * rte_pktmbuf_prefree_seg(struct rte_mbuf *m)

{
	bool refcnt_not_one;

	refcnt_not_one = unlikely(rte_mbuf_refcnt_read(m) != 1);
	if (refcnt_not_one && __rte_mbuf_refcnt_update(m, -1) != 0)
		return NULL;

	if (unlikely(!RTE_MBUF_DIRECT(m))) {
		rte_pktmbuf_detach(m);
		if (RTE_MBUF_HAS_EXTBUF(m) &&
				RTE_MBUF_HAS_PINNED_EXTBUF(m) &&
				__rte_pktmbuf_pinned_extbuf_decref(m))
			return NULL;
	}

	if (refcnt_not_one)
		rte_mbuf_refcnt_set(m, 1);
	if (m->nb_segs != 1)
		m->nb_segs = 1;
	if (m->next != NULL)
		m->next = NULL;

	return m;
}

This function can only succeed (i.e. return non-NULL) when 'refcnt' is 1 (or reaches 0).

REQUIREMENT 2: The driver must hold the only reference to the mbuf, i.e. 'm->refcnt' must be 1.

When the function succeeds, it initializes the mbuf fields as required by rte_mbuf_raw_free() before returning.

Now, since the driver has exclusive access to the mbuf, it is free to initialize the 'm->next' and 'm->nb_segs' at any time.
It could do that when preparing the Tx descriptor.

This is very interesting, because it means that fast-free does not prohibit segmented packets!
(But the driver must have sufficient Tx descriptors for all segments in the mbuf.)

Now, lets dig into rte_pktmbuf_prefree_seg()'s block handling non-direct mbufs, i.e. cloned mbufs and mbufs with external buffer:

	if (unlikely(!RTE_MBUF_DIRECT(m))) {
		rte_pktmbuf_detach(m);
		if (RTE_MBUF_HAS_EXTBUF(m) &&
				RTE_MBUF_HAS_PINNED_EXTBUF(m) &&
				__rte_pktmbuf_pinned_extbuf_decref(m))
			return NULL;
	}

Starting with rte_pktmbuf_detach():

static inline void rte_pktmbuf_detach(struct rte_mbuf *m)

{
	struct rte_mempool *mp = m->pool;
	uint32_t mbuf_size, buf_len;
	uint16_t priv_size;

	if (RTE_MBUF_HAS_EXTBUF(m)) {
		/*
		 * The mbuf has the external attached buffer,
		 * we should check the type of the memory pool where
		 * the mbuf was allocated from to detect the pinned
		 * external buffer.
		 */
		uint32_t flags = rte_pktmbuf_priv_flags(mp);

		if (flags & RTE_PKTMBUF_POOL_F_PINNED_EXT_BUF) {
			/*
			 * The pinned external buffer should not be
			 * detached from its backing mbuf, just exit.
			 */
			return;
		}
		__rte_pktmbuf_free_extbuf(m);
	} else {
		__rte_pktmbuf_free_direct(m);
	}
	priv_size = rte_pktmbuf_priv_size(mp);
	mbuf_size = (uint32_t)(sizeof(struct rte_mbuf) + priv_size);
	buf_len = rte_pktmbuf_data_room_size(mp);

	m->priv_size = priv_size;
	m->buf_addr = (char *)m + mbuf_size;
	rte_mbuf_iova_set(m, rte_mempool_virt2iova(m) + mbuf_size);
	m->buf_len = (uint16_t)buf_len;
	rte_pktmbuf_reset_headroom(m);
	m->data_len = 0;
	m->ol_flags = 0;
}

The only quick and simple code path through this function is when the mbuf uses a pinned external buffer:

	if (RTE_MBUF_HAS_EXTBUF(m)) {
		uint32_t flags = rte_pktmbuf_priv_flags(mp);
		if (flags & RTE_PKTMBUF_POOL_F_PINNED_EXT_BUF)
			return;

REQUIREMENT 3: The mbuf must not be cloned or use a non-pinned external buffer.

Continuing with the next part of rte_pktmbuf_prefree_seg()'s block:
if (RTE_MBUF_HAS_EXTBUF(m) &&

				RTE_MBUF_HAS_PINNED_EXTBUF(m) &&
				__rte_pktmbuf_pinned_extbuf_decref(m))
			return NULL;

Continuing with the next part of the block in rte_pktmbuf_prefree_seg():

/**
* @internal Handle the packet mbufs with attached pinned external buffer
* on the mbuf freeing:
*

 *  - return zero if reference counter in shinfo is one. It means there is
 *  no more reference to this pinned buffer and mbuf can be returned to
 *  the pool

*
* - otherwise (if reference counter is not one), decrement reference
* counter and return non-zero value to prevent freeing the backing mbuf. *
* Returns non zero if mbuf should not be freed.
*/
static inline int __rte_pktmbuf_pinned_extbuf_decref(struct rte_mbuf *m)

{
	struct rte_mbuf_ext_shared_info *shinfo;

	/* Clear flags, mbuf is being freed. */
	m->ol_flags = RTE_MBUF_F_EXTERNAL;
	shinfo = m->shinfo;

	/* Optimize for performance - do not dec/reinit */
	if (likely(rte_mbuf_ext_refcnt_read(shinfo) == 1))
		return 0;

	/*
	 * Direct usage of add primitive to avoid
	 * duplication of comparing with one.
	 */
	if (likely(rte_atomic_fetch_add_explicit(&shinfo->refcnt, -1,
				     rte_memory_order_acq_rel) - 1))
		return 1;

	/* Reinitialize counter before mbuf freeing. */
	rte_mbuf_ext_refcnt_set(shinfo, 1);
	return 0;
}

Essentially, if the mbuf does use a pinned external buffer, rte_pktmbuf_prefree_seg() only succeeds if that pinned external buffer is only referred to by the mbuf.

REQUIREMENT 4: If the mbuf uses a pinned external buffer, the mbuf must hold the only reference to that pinned external buffer, i.e. in that case, 'm->shinfo->refcnt' must be 1.

Quoted 8 lines

Please review.

If I'm not mistaken, the mbuf library is not a barrier for fast-freeing segmented packet mbufs, and thus fast-free of jumbo frames is possible.

We need a driver developer to confirm that my suggested approach - resetting the mbuf fields, incl. 'm->nb_segs' and 'm->next', when preparing the Tx descriptor - is viable.

Great analysis, makes a lot of sense to me.
Shall we add then a special API to make PMD maintainers life a bit easier:
Something like rte_mbuf_fast_free_prep(mp, mb), that will optionally check that requirements outlined above are satisfied for given mbuf and also reset mbuf fields to expected values?
Konstantin

#27 lore raw
Quoted 289 lines

From: Konstantin Ananyev [mailto:[email protected]]
Sent: Monday, 15 December 2025 15.41

Quoted 4 lines

Executive Summary:

My analysis shows that the mbuf library is not a barrier for fast-

freeing

Quoted 1 line

segmented packet mbufs, and thus fast-free of jumbo frames is

possible.

Quoted 6 lines

Detailed Analysis:

The purpose of the mbuf fast-free Tx optimization is to reduce rte_pktmbuf_free_seg() to something much simpler in the ethdev

drivers, by

Quoted 2 lines

eliminating the code path related to indirect mbufs. Optimally, we want to simplify the ethdev driver's function that

frees the

Quoted 1 line

transmitted mbufs, so it can free them directly to their mempool

without

Quoted 3 lines

accessing the mbufs themselves.

If the driver cannot access the mbuf itself, it cannot determine

which

Quoted 3 lines

mempool it belongs to.
We don't want the driver to access every mbuf being freed; but if all mbufs of a Tx queue belong to the same mempool, the driver can

determine

Quoted 3 lines

which mempool by looking into just one of the mbufs.

REQUIREMENT 1: The mbufs of a Tx queue must come from the same

mempool.

Quoted 3 lines

When an mbuf is freed to its mempool, some of the fields in the mbuf

must

Quoted 4 lines

be initialized.
So, for fast-free, this must be done by the driver's function that prepares the Tx descriptor.
This is a requirement to the driver, not a requirement to the

application.

Quoted 30 lines

Now, let's dig into the code for freeing an mbuf.
Note: For readability purposes, I'll cut out some code and comments unrelated to this topic.

static __rte_always_inline void
rte_pktmbuf_free_seg(struct rte_mbuf *m)

{
	m = rte_pktmbuf_prefree_seg(m);
	if (likely(m != NULL))
		rte_mbuf_raw_free(m);
}

rte_mbuf_raw_free(m) is simple, so nothing to gain there:

/**
* Put mbuf back into its original mempool.
*
* The caller must ensure that the mbuf is direct and properly
* reinitialized (refcnt=1, next=NULL, nb_segs=1), as done by
* rte_pktmbuf_prefree_seg().
*/
static __rte_always_inline void rte_mbuf_raw_free(struct rte_mbuf *m)

{
	rte_mbuf_history_mark(m, RTE_MBUF_HISTORY_OP_LIB_FREE);
	rte_mempool_put(m->pool, m);
}
Quoted 4 lines

Note that the description says that the mbuf must be direct. This is not entirely accurate; the mbuf is allowed to use a pinned external buffer, if the mbuf holds the only reference to it. (Most of the mbuf library functions have this documentation

inaccuracy,

Quoted 46 lines

which should be fixed some day.)

So, the fast-free optimization really comes down to rte_pktmbuf_prefree_seg(m), which must not return NULL.

Let's dig into that.

/**
* Decrease reference counter and unlink a mbuf segment *
* This function does the same than a free, except that it does not
* return the segment to its pool.
* It decreases the reference counter, and if it reaches 0, it is
* detached from its parent for an indirect mbuf.
*
* @return
* - (m) if it is the last reference. It can be recycled or freed.
* - (NULL) if the mbuf still has remaining references on it.
*/
static __rte_always_inline struct rte_mbuf * rte_pktmbuf_prefree_seg(struct rte_mbuf *m)

{
	bool refcnt_not_one;

	refcnt_not_one = unlikely(rte_mbuf_refcnt_read(m) != 1);
	if (refcnt_not_one && __rte_mbuf_refcnt_update(m, -1) != 0)
		return NULL;

	if (unlikely(!RTE_MBUF_DIRECT(m))) {
		rte_pktmbuf_detach(m);
		if (RTE_MBUF_HAS_EXTBUF(m) &&
				RTE_MBUF_HAS_PINNED_EXTBUF(m) &&
				__rte_pktmbuf_pinned_extbuf_decref(m))
			return NULL;
	}

	if (refcnt_not_one)
		rte_mbuf_refcnt_set(m, 1);
	if (m->nb_segs != 1)
		m->nb_segs = 1;
	if (m->next != NULL)
		m->next = NULL;

	return m;
}
Quoted 1 line

This function can only succeed (i.e. return non-NULL) when 'refcnt'

is 1

Quoted 7 lines

(or reaches 0).

REQUIREMENT 2: The driver must hold the only reference to the mbuf, i.e. 'm->refcnt' must be 1.

When the function succeeds, it initializes the mbuf fields as

required by

Quoted 9 lines

rte_mbuf_raw_free() before returning.

Now, since the driver has exclusive access to the mbuf, it is free to initialize the 'm->next' and 'm->nb_segs' at any time.
It could do that when preparing the Tx descriptor.

This is very interesting, because it means that fast-free does not prohibit segmented packets!
(But the driver must have sufficient Tx descriptors for all segments

in

Quoted 4 lines

the mbuf.)

Now, lets dig into rte_pktmbuf_prefree_seg()'s block handling non-

direct

Quoted 51 lines

mbufs, i.e. cloned mbufs and mbufs with external buffer:

	if (unlikely(!RTE_MBUF_DIRECT(m))) {
		rte_pktmbuf_detach(m);
		if (RTE_MBUF_HAS_EXTBUF(m) &&
				RTE_MBUF_HAS_PINNED_EXTBUF(m) &&
				__rte_pktmbuf_pinned_extbuf_decref(m))
			return NULL;
	}

Starting with rte_pktmbuf_detach():

static inline void rte_pktmbuf_detach(struct rte_mbuf *m)

{
	struct rte_mempool *mp = m->pool;
	uint32_t mbuf_size, buf_len;
	uint16_t priv_size;

	if (RTE_MBUF_HAS_EXTBUF(m)) {
		/*
		 * The mbuf has the external attached buffer,
		 * we should check the type of the memory pool where
		 * the mbuf was allocated from to detect the pinned
		 * external buffer.
		 */
		uint32_t flags = rte_pktmbuf_priv_flags(mp);

		if (flags & RTE_PKTMBUF_POOL_F_PINNED_EXT_BUF) {
			/*
			 * The pinned external buffer should not be
			 * detached from its backing mbuf, just exit.
			 */
			return;
		}
		__rte_pktmbuf_free_extbuf(m);
	} else {
		__rte_pktmbuf_free_direct(m);
	}
	priv_size = rte_pktmbuf_priv_size(mp);
	mbuf_size = (uint32_t)(sizeof(struct rte_mbuf) + priv_size);
	buf_len = rte_pktmbuf_data_room_size(mp);

	m->priv_size = priv_size;
	m->buf_addr = (char *)m + mbuf_size;
	rte_mbuf_iova_set(m, rte_mempool_virt2iova(m) + mbuf_size);
	m->buf_len = (uint16_t)buf_len;
	rte_pktmbuf_reset_headroom(m);
	m->data_len = 0;
	m->ol_flags = 0;
}
Quoted 1 line

The only quick and simple code path through this function is when the

mbuf

Quoted 7 lines

uses a pinned external buffer:

	if (RTE_MBUF_HAS_EXTBUF(m)) {
		uint32_t flags = rte_pktmbuf_priv_flags(mp);
		if (flags & RTE_PKTMBUF_POOL_F_PINNED_EXT_BUF)
			return;

REQUIREMENT 3: The mbuf must not be cloned or use a non-pinned

external

Quoted 10 lines

buffer.

Continuing with the next part of rte_pktmbuf_prefree_seg()'s block:
if (RTE_MBUF_HAS_EXTBUF(m) &&

				RTE_MBUF_HAS_PINNED_EXTBUF(m) &&
				__rte_pktmbuf_pinned_extbuf_decref(m))
			return NULL;

Continuing with the next part of the block in

rte_pktmbuf_prefree_seg():

Quoted 3 lines

/**
* @internal Handle the packet mbufs with attached pinned external

buffer

Quoted 3 lines

* on the mbuf freeing:
*
* - return zero if reference counter in shinfo is one. It means

there is

Quoted 1 line

* no more reference to this pinned buffer and mbuf can be returned

to

Quoted 3 lines

* the pool
*
* - otherwise (if reference counter is not one), decrement

reference

Quoted 1 line

* counter and return non-zero value to prevent freeing the backing

mbuf.

Quoted 4 lines

*
* Returns non zero if mbuf should not be freed.
*/
static inline int __rte_pktmbuf_pinned_extbuf_decref(struct rte_mbuf

*m)

Quoted 24 lines
{
	struct rte_mbuf_ext_shared_info *shinfo;

	/* Clear flags, mbuf is being freed. */
	m->ol_flags = RTE_MBUF_F_EXTERNAL;
	shinfo = m->shinfo;

	/* Optimize for performance - do not dec/reinit */
	if (likely(rte_mbuf_ext_refcnt_read(shinfo) == 1))
		return 0;

	/*
	 * Direct usage of add primitive to avoid
	 * duplication of comparing with one.
	 */
	if (likely(rte_atomic_fetch_add_explicit(&shinfo->refcnt, -1,
				     rte_memory_order_acq_rel) - 1))
		return 1;

	/* Reinitialize counter before mbuf freeing. */
	rte_mbuf_ext_refcnt_set(shinfo, 1);
	return 0;
}
Quoted 2 lines

Essentially, if the mbuf does use a pinned external buffer, rte_pktmbuf_prefree_seg() only succeeds if that pinned external

buffer is

Quoted 3 lines

only referred to by the mbuf.

REQUIREMENT 4: If the mbuf uses a pinned external buffer, the mbuf

must

Quoted 1 line

hold the only reference to that pinned external buffer, i.e. in that

case,

Quoted 6 lines

'm->shinfo->refcnt' must be 1.

Please review.

If I'm not mistaken, the mbuf library is not a barrier for fast-

freeing

Quoted 1 line

segmented packet mbufs, and thus fast-free of jumbo frames is

possible.

Quoted 4 lines

We need a driver developer to confirm that my suggested approach - resetting the mbuf fields, incl. 'm->nb_segs' and 'm->next', when preparing the Tx descriptor - is viable.

Quoted 7 lines

Great analysis, makes a lot of sense to me.
Shall we add then a special API to make PMD maintainers life a bit easier:
Something like rte_mbuf_fast_free_prep(mp, mb), that will optionally check
that requirements outlined above are satisfied for given mbuf and also reset mbuf fields to expected values?

Good idea, Konstantin.

Detailed suggestion below.
Note that __rte_mbuf_raw_sanity_check_mp() is used to checks the requirements after 'nb_segs' and 'next' have been initialized.

/**
* Reinitialize an mbuf for freeing back into the mempool.
*
* The caller must ensure that the mbuf comes from the specified mempool,
* is direct and only referred to by the caller (refcnt=1).
*
* This function is used by drivers in their transmit function for mbuf fast release
* when the transmit descriptor is initialized,
* so the driver can call rte_mbuf_raw_free()
* when the packet segment has been transmitted.
*
* @see RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE
*
* @param mp
* The mempool to which the mbuf belong.
* @param m
* The mbuf being reinitialized.
*/
static __rte_always_inline void
rte_mbuf_raw_prefree_seg(const struct rte_mempool *mp, struct rte_mbuf *m)

{
	if (m->nb_segs != 1)
		m->nb_segs = 1;
	if (m->next != NULL)
		m->next = NULL;

	__rte_mbuf_raw_sanity_check_mp(m, mp);
	rte_mbuf_history_mark(mbuf, RTE_MBUF_HISTORY_OP_LIB_PREFREE_RAW);
}

/**
* Reinitialize a bulk of mbufs for freeing back into the mempool.
*
* The caller must ensure that the mbufs come from the specified mempool,
* are direct and only referred to by the caller (refcnt=1).
*
* This function is used by drivers in their transmit function for mbuf fast release
* when the transmit descriptors are initialized,
* so the driver can call rte_mbuf_raw_free_bulk()
* when the packet segments have been transmitted.
*
* @see RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE
*
* @param mp
* The mempool to which the mbufs belong.
* @param mbufs
* Array of pointers to mbufs being reinitialized.
* The array must not contain NULL pointers.
* @param count
* Array size.
*/
static __rte_always_inline void
rte_mbuf_raw_prefree_seg_bulk(const struct rte_mempool *mp, struct rte_mbuf **mbufs, unsigned int count)

{
	for (unsigned int idx = 0; idx < count; idx++) {
		struct rte_mbuf *m = mbufs[idx];

		if (m->nb_segs != 1)
			m->nb_segs = 1;
		if (m->next != NULL)
			m->next = NULL;

		__rte_mbuf_raw_sanity_check_mp(m, mp);
	}
	rte_mbuf_history_mark_bulk(mbufs, count, RTE_MBUF_HISTORY_OP_LIB_PREFREE_RAW);
}
Quoted 3 lines

Konstantin

#28 lore raw
Quoted 332 lines
Quoted 290 lines

From: Konstantin Ananyev [mailto:[email protected]]
Sent: Monday, 15 December 2025 15.41

Quoted 4 lines

Executive Summary:

My analysis shows that the mbuf library is not a barrier for fast-

freeing

Quoted 1 line

segmented packet mbufs, and thus fast-free of jumbo frames is

possible.

Quoted 6 lines

Detailed Analysis:

The purpose of the mbuf fast-free Tx optimization is to reduce rte_pktmbuf_free_seg() to something much simpler in the ethdev

drivers, by

Quoted 2 lines

eliminating the code path related to indirect mbufs. Optimally, we want to simplify the ethdev driver's function that

frees the

Quoted 1 line

transmitted mbufs, so it can free them directly to their mempool

without

Quoted 3 lines

accessing the mbufs themselves.

If the driver cannot access the mbuf itself, it cannot determine

which

Quoted 3 lines

mempool it belongs to.
We don't want the driver to access every mbuf being freed; but if all mbufs of a Tx queue belong to the same mempool, the driver can

determine

Quoted 3 lines

which mempool by looking into just one of the mbufs.

REQUIREMENT 1: The mbufs of a Tx queue must come from the same

mempool.

Quoted 3 lines

When an mbuf is freed to its mempool, some of the fields in the mbuf

must

Quoted 4 lines

be initialized.
So, for fast-free, this must be done by the driver's function that prepares the Tx descriptor.
This is a requirement to the driver, not a requirement to the

application.

Quoted 30 lines

Now, let's dig into the code for freeing an mbuf.
Note: For readability purposes, I'll cut out some code and comments unrelated to this topic.

static __rte_always_inline void
rte_pktmbuf_free_seg(struct rte_mbuf *m)

{
	m = rte_pktmbuf_prefree_seg(m);
	if (likely(m != NULL))
		rte_mbuf_raw_free(m);
}

rte_mbuf_raw_free(m) is simple, so nothing to gain there:

/**
* Put mbuf back into its original mempool.
*
* The caller must ensure that the mbuf is direct and properly
* reinitialized (refcnt=1, next=NULL, nb_segs=1), as done by
* rte_pktmbuf_prefree_seg().
*/
static __rte_always_inline void rte_mbuf_raw_free(struct rte_mbuf *m)

{
	rte_mbuf_history_mark(m, RTE_MBUF_HISTORY_OP_LIB_FREE);
	rte_mempool_put(m->pool, m);
}
Quoted 4 lines

Note that the description says that the mbuf must be direct. This is not entirely accurate; the mbuf is allowed to use a pinned external buffer, if the mbuf holds the only reference to it. (Most of the mbuf library functions have this documentation

inaccuracy,

Quoted 46 lines

which should be fixed some day.)

So, the fast-free optimization really comes down to rte_pktmbuf_prefree_seg(m), which must not return NULL.

Let's dig into that.

/**
* Decrease reference counter and unlink a mbuf segment *
* This function does the same than a free, except that it does not
* return the segment to its pool.
* It decreases the reference counter, and if it reaches 0, it is
* detached from its parent for an indirect mbuf.
*
* @return
* - (m) if it is the last reference. It can be recycled or freed.
* - (NULL) if the mbuf still has remaining references on it.
*/
static __rte_always_inline struct rte_mbuf * rte_pktmbuf_prefree_seg(struct rte_mbuf *m)

{
	bool refcnt_not_one;

	refcnt_not_one = unlikely(rte_mbuf_refcnt_read(m) != 1);
	if (refcnt_not_one && __rte_mbuf_refcnt_update(m, -1) != 0)
		return NULL;

	if (unlikely(!RTE_MBUF_DIRECT(m))) {
		rte_pktmbuf_detach(m);
		if (RTE_MBUF_HAS_EXTBUF(m) &&
				RTE_MBUF_HAS_PINNED_EXTBUF(m) &&
				__rte_pktmbuf_pinned_extbuf_decref(m))
			return NULL;
	}

	if (refcnt_not_one)
		rte_mbuf_refcnt_set(m, 1);
	if (m->nb_segs != 1)
		m->nb_segs = 1;
	if (m->next != NULL)
		m->next = NULL;

	return m;
}
Quoted 1 line

This function can only succeed (i.e. return non-NULL) when 'refcnt'

is 1

Quoted 7 lines

(or reaches 0).

REQUIREMENT 2: The driver must hold the only reference to the mbuf, i.e. 'm->refcnt' must be 1.

When the function succeeds, it initializes the mbuf fields as

required by

Quoted 9 lines

rte_mbuf_raw_free() before returning.

Now, since the driver has exclusive access to the mbuf, it is free to initialize the 'm->next' and 'm->nb_segs' at any time.
It could do that when preparing the Tx descriptor.

This is very interesting, because it means that fast-free does not prohibit segmented packets!
(But the driver must have sufficient Tx descriptors for all segments

in

Quoted 4 lines

the mbuf.)

Now, lets dig into rte_pktmbuf_prefree_seg()'s block handling non-

direct

Quoted 51 lines

mbufs, i.e. cloned mbufs and mbufs with external buffer:

	if (unlikely(!RTE_MBUF_DIRECT(m))) {
		rte_pktmbuf_detach(m);
		if (RTE_MBUF_HAS_EXTBUF(m) &&
				RTE_MBUF_HAS_PINNED_EXTBUF(m) &&
				__rte_pktmbuf_pinned_extbuf_decref(m))
			return NULL;
	}

Starting with rte_pktmbuf_detach():

static inline void rte_pktmbuf_detach(struct rte_mbuf *m)

{
	struct rte_mempool *mp = m->pool;
	uint32_t mbuf_size, buf_len;
	uint16_t priv_size;

	if (RTE_MBUF_HAS_EXTBUF(m)) {
		/*
		 * The mbuf has the external attached buffer,
		 * we should check the type of the memory pool where
		 * the mbuf was allocated from to detect the pinned
		 * external buffer.
		 */
		uint32_t flags = rte_pktmbuf_priv_flags(mp);

		if (flags & RTE_PKTMBUF_POOL_F_PINNED_EXT_BUF) {
			/*
			 * The pinned external buffer should not be
			 * detached from its backing mbuf, just exit.
			 */
			return;
		}
		__rte_pktmbuf_free_extbuf(m);
	} else {
		__rte_pktmbuf_free_direct(m);
	}
	priv_size = rte_pktmbuf_priv_size(mp);
	mbuf_size = (uint32_t)(sizeof(struct rte_mbuf) + priv_size);
	buf_len = rte_pktmbuf_data_room_size(mp);

	m->priv_size = priv_size;
	m->buf_addr = (char *)m + mbuf_size;
	rte_mbuf_iova_set(m, rte_mempool_virt2iova(m) + mbuf_size);
	m->buf_len = (uint16_t)buf_len;
	rte_pktmbuf_reset_headroom(m);
	m->data_len = 0;
	m->ol_flags = 0;
}
Quoted 1 line

The only quick and simple code path through this function is when the

mbuf

Quoted 7 lines

uses a pinned external buffer:

	if (RTE_MBUF_HAS_EXTBUF(m)) {
		uint32_t flags = rte_pktmbuf_priv_flags(mp);
		if (flags & RTE_PKTMBUF_POOL_F_PINNED_EXT_BUF)
			return;

REQUIREMENT 3: The mbuf must not be cloned or use a non-pinned

external

Quoted 10 lines

buffer.

Continuing with the next part of rte_pktmbuf_prefree_seg()'s block:
if (RTE_MBUF_HAS_EXTBUF(m) &&

				RTE_MBUF_HAS_PINNED_EXTBUF(m) &&
				__rte_pktmbuf_pinned_extbuf_decref(m))
			return NULL;

Continuing with the next part of the block in

rte_pktmbuf_prefree_seg():

Quoted 3 lines

/**
* @internal Handle the packet mbufs with attached pinned external

buffer

Quoted 3 lines

* on the mbuf freeing:
*
* - return zero if reference counter in shinfo is one. It means

there is

Quoted 1 line

* no more reference to this pinned buffer and mbuf can be returned

to

Quoted 3 lines

* the pool
*
* - otherwise (if reference counter is not one), decrement

reference

Quoted 1 line

* counter and return non-zero value to prevent freeing the backing

mbuf.

Quoted 4 lines

*
* Returns non zero if mbuf should not be freed.
*/
static inline int __rte_pktmbuf_pinned_extbuf_decref(struct rte_mbuf

*m)

Quoted 24 lines
{
	struct rte_mbuf_ext_shared_info *shinfo;

	/* Clear flags, mbuf is being freed. */
	m->ol_flags = RTE_MBUF_F_EXTERNAL;
	shinfo = m->shinfo;

	/* Optimize for performance - do not dec/reinit */
	if (likely(rte_mbuf_ext_refcnt_read(shinfo) == 1))
		return 0;

	/*
	 * Direct usage of add primitive to avoid
	 * duplication of comparing with one.
	 */
	if (likely(rte_atomic_fetch_add_explicit(&shinfo->refcnt, -1,
				     rte_memory_order_acq_rel) - 1))
		return 1;

	/* Reinitialize counter before mbuf freeing. */
	rte_mbuf_ext_refcnt_set(shinfo, 1);
	return 0;
}
Quoted 2 lines

Essentially, if the mbuf does use a pinned external buffer, rte_pktmbuf_prefree_seg() only succeeds if that pinned external

buffer is

Quoted 3 lines

only referred to by the mbuf.

REQUIREMENT 4: If the mbuf uses a pinned external buffer, the mbuf

must

Quoted 1 line

hold the only reference to that pinned external buffer, i.e. in that

case,

Quoted 6 lines

'm->shinfo->refcnt' must be 1.

Please review.

If I'm not mistaken, the mbuf library is not a barrier for fast-

freeing

Quoted 1 line

segmented packet mbufs, and thus fast-free of jumbo frames is

possible.

Quoted 4 lines

We need a driver developer to confirm that my suggested approach - resetting the mbuf fields, incl. 'm->nb_segs' and 'm->next', when preparing the Tx descriptor - is viable.

Quoted 7 lines

Great analysis, makes a lot of sense to me.
Shall we add then a special API to make PMD maintainers life a bit easier:
Something like rte_mbuf_fast_free_prep(mp, mb), that will optionally check
that requirements outlined above are satisfied for given mbuf and also reset mbuf fields to expected values?

Good idea, Konstantin.

Detailed suggestion below.
Note that __rte_mbuf_raw_sanity_check_mp() is used to checks the requirements after 'nb_segs' and 'next' have been initialized.

/**
* Reinitialize an mbuf for freeing back into the mempool.
*
* The caller must ensure that the mbuf comes from the specified mempool,
* is direct and only referred to by the caller (refcnt=1).
*
* This function is used by drivers in their transmit function for mbuf fast release
* when the transmit descriptor is initialized,
* so the driver can call rte_mbuf_raw_free()
* when the packet segment has been transmitted.
*
* @see RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE
*
* @param mp
* The mempool to which the mbuf belong.
* @param m
* The mbuf being reinitialized.
*/
static __rte_always_inline void
rte_mbuf_raw_prefree_seg(const struct rte_mempool *mp, struct rte_mbuf *m)

{
	if (m->nb_segs != 1)
		m->nb_segs = 1;
	if (m->next != NULL)
		m->next = NULL;

	__rte_mbuf_raw_sanity_check_mp(m, mp);
	rte_mbuf_history_mark(mbuf,
Quoted 2 lines

RTE_MBUF_HISTORY_OP_LIB_PREFREE_RAW);
}

Thanks Morten, though should we really panic if condition is not met? Might be just do check first and return an error.

Quoted 43 lines

/**
* Reinitialize a bulk of mbufs for freeing back into the mempool.
*
* The caller must ensure that the mbufs come from the specified mempool,
* are direct and only referred to by the caller (refcnt=1).
*
* This function is used by drivers in their transmit function for mbuf fast release
* when the transmit descriptors are initialized,
* so the driver can call rte_mbuf_raw_free_bulk()
* when the packet segments have been transmitted.
*
* @see RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE
*
* @param mp
* The mempool to which the mbufs belong.
* @param mbufs
* Array of pointers to mbufs being reinitialized.
* The array must not contain NULL pointers.
* @param count
* Array size.
*/
static __rte_always_inline void
rte_mbuf_raw_prefree_seg_bulk(const struct rte_mempool *mp, struct rte_mbuf **mbufs, unsigned int count)

{
	for (unsigned int idx = 0; idx < count; idx++) {
		struct rte_mbuf *m = mbufs[idx];

		if (m->nb_segs != 1)
			m->nb_segs = 1;
		if (m->next != NULL)
			m->next = NULL;

		__rte_mbuf_raw_sanity_check_mp(m, mp);
	}
	rte_mbuf_history_mark_bulk(mbufs, count,

RTE_MBUF_HISTORY_OP_LIB_PREFREE_RAW);
}

Quoted 3 lines

Konstantin

#29 lore raw
Quoted 358 lines

From: Konstantin Ananyev [mailto:[email protected]]
Sent: Friday, 19 December 2025 18.08

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Executive Summary:

My analysis shows that the mbuf library is not a barrier for

fast-

Quoted 13 lines
Quoted 13 lines

freeing

Quoted 1 line

segmented packet mbufs, and thus fast-free of jumbo frames is

possible.

Quoted 6 lines

Detailed Analysis:

The purpose of the mbuf fast-free Tx optimization is to reduce rte_pktmbuf_free_seg() to something much simpler in the ethdev

drivers, by

Quoted 2 lines

eliminating the code path related to indirect mbufs. Optimally, we want to simplify the ethdev driver's function that

frees the

Quoted 8 lines
Quoted 8 lines
Quoted 1 line

transmitted mbufs, so it can free them directly to their mempool

without

Quoted 3 lines

accessing the mbufs themselves.

If the driver cannot access the mbuf itself, it cannot determine

which

Quoted 2 lines

mempool it belongs to.
We don't want the driver to access every mbuf being freed; but if

all

Quoted 9 lines
Quoted 9 lines
Quoted 1 line

mbufs of a Tx queue belong to the same mempool, the driver can

determine

Quoted 3 lines

which mempool by looking into just one of the mbufs.

REQUIREMENT 1: The mbufs of a Tx queue must come from the same

mempool.

Quoted 3 lines

When an mbuf is freed to its mempool, some of the fields in the

mbuf

Quoted 3 lines
Quoted 3 lines

must

Quoted 2 lines

be initialized.
So, for fast-free, this must be done by the driver's function

that

Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

prepares the Tx descriptor.
This is a requirement to the driver, not a requirement to the

application.

Quoted 3 lines

Now, let's dig into the code for freeing an mbuf.
Note: For readability purposes, I'll cut out some code and

comments

Quoted 27 lines
Quoted 27 lines
Quoted 27 lines

unrelated to this topic.

static __rte_always_inline void
rte_pktmbuf_free_seg(struct rte_mbuf *m)

{
	m = rte_pktmbuf_prefree_seg(m);
	if (likely(m != NULL))
		rte_mbuf_raw_free(m);
}

rte_mbuf_raw_free(m) is simple, so nothing to gain there:

/**
* Put mbuf back into its original mempool.
*
* The caller must ensure that the mbuf is direct and properly
* reinitialized (refcnt=1, next=NULL, nb_segs=1), as done by
* rte_pktmbuf_prefree_seg().
*/
static __rte_always_inline void rte_mbuf_raw_free(struct rte_mbuf *m)

{
	rte_mbuf_history_mark(m, RTE_MBUF_HISTORY_OP_LIB_FREE);
	rte_mempool_put(m->pool, m);
}
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Note that the description says that the mbuf must be direct. This is not entirely accurate; the mbuf is allowed to use a

pinned

Quoted 8 lines
Quoted 8 lines
Quoted 2 lines

external buffer, if the mbuf holds the only reference to it. (Most of the mbuf library functions have this documentation

inaccuracy,

Quoted 5 lines

which should be fixed some day.)

So, the fast-free optimization really comes down to rte_pktmbuf_prefree_seg(m), which must not return NULL.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Let's dig into that.

/**
* Decrease reference counter and unlink a mbuf segment *
* This function does the same than a free, except that it does

not

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

* return the segment to its pool.
* It decreases the reference counter, and if it reaches 0, it is
* detached from its parent for an indirect mbuf.
*
* @return
* - (m) if it is the last reference. It can be recycled or

freed.

Quoted 26 lines
Quoted 26 lines
Quoted 26 lines

* - (NULL) if the mbuf still has remaining references on it.
*/
static __rte_always_inline struct rte_mbuf *
rte_pktmbuf_prefree_seg(struct rte_mbuf *m)

{
	bool refcnt_not_one;

	refcnt_not_one = unlikely(rte_mbuf_refcnt_read(m) != 1);
	if (refcnt_not_one && __rte_mbuf_refcnt_update(m, -1) != 0)
		return NULL;

	if (unlikely(!RTE_MBUF_DIRECT(m))) {
		rte_pktmbuf_detach(m);
		if (RTE_MBUF_HAS_EXTBUF(m) &&
				RTE_MBUF_HAS_PINNED_EXTBUF(m) &&
				__rte_pktmbuf_pinned_extbuf_decref(m))
			return NULL;
	}

	if (refcnt_not_one)
		rte_mbuf_refcnt_set(m, 1);
	if (m->nb_segs != 1)
		m->nb_segs = 1;
	if (m->next != NULL)
		m->next = NULL;
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
	return m;
}

This function can only succeed (i.e. return non-NULL) when

'refcnt'

Quoted 4 lines
Quoted 4 lines

is 1

Quoted 3 lines

(or reaches 0).

REQUIREMENT 2: The driver must hold the only reference to the

mbuf,

Quoted 8 lines
Quoted 8 lines
Quoted 4 lines

i.e. 'm->refcnt' must be 1.

When the function succeeds, it initializes the mbuf fields as

required by

Quoted 3 lines

rte_mbuf_raw_free() before returning.

Now, since the driver has exclusive access to the mbuf, it is

free to

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

initialize the 'm->next' and 'm->nb_segs' at any time. It could do that when preparing the Tx descriptor.

This is very interesting, because it means that fast-free does

not

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

prohibit segmented packets!
(But the driver must have sufficient Tx descriptors for all

segments

Quoted 5 lines
Quoted 5 lines

in

Quoted 4 lines

the mbuf.)

Now, lets dig into rte_pktmbuf_prefree_seg()'s block handling

non-

Quoted 28 lines
Quoted 28 lines

direct

Quoted 27 lines

mbufs, i.e. cloned mbufs and mbufs with external buffer:

	if (unlikely(!RTE_MBUF_DIRECT(m))) {
		rte_pktmbuf_detach(m);
		if (RTE_MBUF_HAS_EXTBUF(m) &&
				RTE_MBUF_HAS_PINNED_EXTBUF(m) &&
				__rte_pktmbuf_pinned_extbuf_decref(m))
			return NULL;
	}

Starting with rte_pktmbuf_detach():

static inline void rte_pktmbuf_detach(struct rte_mbuf *m)

{
	struct rte_mempool *mp = m->pool;
	uint32_t mbuf_size, buf_len;
	uint16_t priv_size;

	if (RTE_MBUF_HAS_EXTBUF(m)) {
		/*
		 * The mbuf has the external attached buffer,
		 * we should check the type of the memory pool where
		 * the mbuf was allocated from to detect the pinned
		 * external buffer.
		 */
		uint32_t flags = rte_pktmbuf_priv_flags(mp);
Quoted 13 lines
Quoted 5 lines
Quoted 5 lines
		if (flags & RTE_PKTMBUF_POOL_F_PINNED_EXT_BUF) {
			/*
			 * The pinned external buffer should not be
			 * detached from its backing mbuf, just exit.
			 */
Quoted 8 lines
Quoted 8 lines
			return;
		}
		__rte_pktmbuf_free_extbuf(m);
	} else {
		__rte_pktmbuf_free_direct(m);
	}
	priv_size = rte_pktmbuf_priv_size(mp);
	mbuf_size = (uint32_t)(sizeof(struct rte_mbuf) +

priv_size);

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
	buf_len = rte_pktmbuf_data_room_size(mp);

	m->priv_size = priv_size;
	m->buf_addr = (char *)m + mbuf_size;
	rte_mbuf_iova_set(m, rte_mempool_virt2iova(m) + mbuf_size);
	m->buf_len = (uint16_t)buf_len;
	rte_pktmbuf_reset_headroom(m);
	m->data_len = 0;
	m->ol_flags = 0;
}
Quoted 1 line
Quoted 1 line
Quoted 1 line

The only quick and simple code path through this function is when

the

Quoted 7 lines
Quoted 7 lines

mbuf

Quoted 6 lines

uses a pinned external buffer:

	if (RTE_MBUF_HAS_EXTBUF(m)) {
		uint32_t flags = rte_pktmbuf_priv_flags(mp);
		if (flags & RTE_PKTMBUF_POOL_F_PINNED_EXT_BUF)
			return;
Quoted 6 lines
Quoted 6 lines
Quoted 1 line

REQUIREMENT 3: The mbuf must not be cloned or use a non-pinned

external

Quoted 4 lines

buffer.

Continuing with the next part of rte_pktmbuf_prefree_seg()'s

block:

Quoted 10 lines
Quoted 10 lines
Quoted 6 lines

if (RTE_MBUF_HAS_EXTBUF(m) &&

				RTE_MBUF_HAS_PINNED_EXTBUF(m) &&
				__rte_pktmbuf_pinned_extbuf_decref(m))
			return NULL;

Continuing with the next part of the block in

rte_pktmbuf_prefree_seg():

Quoted 3 lines

/**
* @internal Handle the packet mbufs with attached pinned

external

Quoted 6 lines
Quoted 6 lines

buffer

Quoted 3 lines

* on the mbuf freeing:
*
* - return zero if reference counter in shinfo is one. It means

there is

Quoted 1 line

* no more reference to this pinned buffer and mbuf can be

returned

Quoted 6 lines
Quoted 6 lines

to

Quoted 3 lines

* the pool
*
* - otherwise (if reference counter is not one), decrement

reference

Quoted 1 line

* counter and return non-zero value to prevent freeing the

backing

Quoted 5 lines
Quoted 5 lines

mbuf.

Quoted 4 lines

*
* Returns non zero if mbuf should not be freed. */
static inline int __rte_pktmbuf_pinned_extbuf_decref(struct

rte_mbuf

Quoted 12 lines
Quoted 12 lines

*m)

Quoted 11 lines
{
	struct rte_mbuf_ext_shared_info *shinfo;

	/* Clear flags, mbuf is being freed. */
	m->ol_flags = RTE_MBUF_F_EXTERNAL;
	shinfo = m->shinfo;

	/* Optimize for performance - do not dec/reinit */
	if (likely(rte_mbuf_ext_refcnt_read(shinfo) == 1))
		return 0;
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
	/*
	 * Direct usage of add primitive to avoid
	 * duplication of comparing with one.
	 */
	if (likely(rte_atomic_fetch_add_explicit(&shinfo->refcnt, -

1,

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
				     rte_memory_order_acq_rel) - 1))
		return 1;

	/* Reinitialize counter before mbuf freeing. */
	rte_mbuf_ext_refcnt_set(shinfo, 1);
	return 0;
}
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

Essentially, if the mbuf does use a pinned external buffer, rte_pktmbuf_prefree_seg() only succeeds if that pinned external

buffer is

Quoted 3 lines

only referred to by the mbuf.

REQUIREMENT 4: If the mbuf uses a pinned external buffer, the

mbuf

Quoted 2 lines
Quoted 2 lines

must

Quoted 1 line

hold the only reference to that pinned external buffer, i.e. in

that

Quoted 4 lines
Quoted 4 lines

case,

Quoted 3 lines

'm->shinfo->refcnt' must be 1.

Quoted 8 lines
Quoted 8 lines
Quoted 3 lines

Please review.

If I'm not mistaken, the mbuf library is not a barrier for fast-

freeing

Quoted 1 line

segmented packet mbufs, and thus fast-free of jumbo frames is

possible.

Quoted 2 lines

We need a driver developer to confirm that my suggested approach

-

Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

resetting the mbuf fields, incl. 'm->nb_segs' and 'm->next', when preparing the Tx descriptor - is viable.

Great analysis, makes a lot of sense to me.
Shall we add then a special API to make PMD maintainers life a bit easier:
Something like rte_mbuf_fast_free_prep(mp, mb), that will

optionally

Quoted 8 lines
Quoted 3 lines

check
that requirements outlined above are satisfied for given mbuf and also reset mbuf fields to expected values?

Good idea, Konstantin.

Detailed suggestion below.
Note that __rte_mbuf_raw_sanity_check_mp() is used to checks the

requirements

Quoted 6 lines

after 'nb_segs' and 'next' have been initialized.

/**
* Reinitialize an mbuf for freeing back into the mempool. *
* The caller must ensure that the mbuf comes from the specified

mempool,

Quoted 3 lines

* is direct and only referred to by the caller (refcnt=1). *
* This function is used by drivers in their transmit function for

mbuf fast release

Quoted 5 lines

* when the transmit descriptor is initialized,
* so the driver can call rte_mbuf_raw_free()
* when the packet segment has been transmitted.
*
* @see RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE

Quoted 8 lines

*
* @param mp
* The mempool to which the mbuf belong.
* @param m
* The mbuf being reinitialized.
*/
static __rte_always_inline void rte_mbuf_raw_prefree_seg(const struct rte_mempool *mp, struct

rte_mbuf *m)

Quoted 6 lines
{
	if (m->nb_segs != 1)
		m->nb_segs = 1;
	if (m->next != NULL)
		m->next = NULL;
Quoted 7 lines
Quoted 4 lines
	__rte_mbuf_raw_sanity_check_mp(m, mp);
	rte_mbuf_history_mark(mbuf,

RTE_MBUF_HISTORY_OP_LIB_PREFREE_RAW);
}

Thanks Morten, though should we really panic if condition is not met? Might be just do check first and return an error.

__rte_mbuf_raw_sanity_check_mp() is a no-op unless RTE_LIBRTE_MBUF_DEBUG is enabled.

Using it everywhere in alloc/free in the mbuf library is the convention.

And if we don't do it here, the __rte_mbuf_raw_sanity_check_mp() in rte_mbuf_raw_free() will panic later instead. Better to fail early.

Quoted 47 lines
Quoted 6 lines

/**
* Reinitialize a bulk of mbufs for freeing back into the mempool. *
* The caller must ensure that the mbufs come from the specified

mempool,

Quoted 3 lines

* are direct and only referred to by the caller (refcnt=1). *
* This function is used by drivers in their transmit function for

mbuf fast release

Quoted 8 lines

* when the transmit descriptors are initialized,
* so the driver can call rte_mbuf_raw_free_bulk()
* when the packet segments have been transmitted. *
* @see RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE *
* @param mp
* The mempool to which the mbufs belong.

Quoted 8 lines

* @param mbufs
* Array of pointers to mbufs being reinitialized.
* The array must not contain NULL pointers.
* @param count
* Array size.
*/
static __rte_always_inline void rte_mbuf_raw_prefree_seg_bulk(const struct rte_mempool *mp, struct

rte_mbuf

Quoted 19 lines

**mbufs, unsigned int count)

{
	for (unsigned int idx = 0; idx < count; idx++) {
		struct rte_mbuf *m = mbufs[idx];

		if (m->nb_segs != 1)
			m->nb_segs = 1;
		if (m->next != NULL)
			m->next = NULL;

		__rte_mbuf_raw_sanity_check_mp(m, mp);
	}
	rte_mbuf_history_mark_bulk(mbufs, count,

RTE_MBUF_HISTORY_OP_LIB_PREFREE_RAW);
}

Quoted 3 lines

Konstantin

#30 lore raw
Quoted 365 lines
Quoted 365 lines

From: Konstantin Ananyev [mailto:[email protected]]
Sent: Friday, 19 December 2025 18.08

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Executive Summary:

My analysis shows that the mbuf library is not a barrier for

fast-

Quoted 13 lines
Quoted 13 lines

freeing

Quoted 1 line

segmented packet mbufs, and thus fast-free of jumbo frames is

possible.

Quoted 6 lines

Detailed Analysis:

The purpose of the mbuf fast-free Tx optimization is to reduce rte_pktmbuf_free_seg() to something much simpler in the ethdev

drivers, by

Quoted 2 lines

eliminating the code path related to indirect mbufs. Optimally, we want to simplify the ethdev driver's function that

frees the

Quoted 8 lines
Quoted 8 lines
Quoted 1 line

transmitted mbufs, so it can free them directly to their mempool

without

Quoted 3 lines

accessing the mbufs themselves.

If the driver cannot access the mbuf itself, it cannot determine

which

Quoted 2 lines

mempool it belongs to.
We don't want the driver to access every mbuf being freed; but if

all

Quoted 9 lines
Quoted 9 lines
Quoted 1 line

mbufs of a Tx queue belong to the same mempool, the driver can

determine

Quoted 3 lines

which mempool by looking into just one of the mbufs.

REQUIREMENT 1: The mbufs of a Tx queue must come from the same

mempool.

Quoted 3 lines

When an mbuf is freed to its mempool, some of the fields in the

mbuf

Quoted 3 lines
Quoted 3 lines

must

Quoted 2 lines

be initialized.
So, for fast-free, this must be done by the driver's function

that

Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

prepares the Tx descriptor.
This is a requirement to the driver, not a requirement to the

application.

Quoted 3 lines

Now, let's dig into the code for freeing an mbuf.
Note: For readability purposes, I'll cut out some code and

comments

Quoted 27 lines
Quoted 27 lines
Quoted 27 lines

unrelated to this topic.

static __rte_always_inline void
rte_pktmbuf_free_seg(struct rte_mbuf *m)

{
	m = rte_pktmbuf_prefree_seg(m);
	if (likely(m != NULL))
		rte_mbuf_raw_free(m);
}

rte_mbuf_raw_free(m) is simple, so nothing to gain there:

/**
* Put mbuf back into its original mempool.
*
* The caller must ensure that the mbuf is direct and properly
* reinitialized (refcnt=1, next=NULL, nb_segs=1), as done by
* rte_pktmbuf_prefree_seg().
*/
static __rte_always_inline void rte_mbuf_raw_free(struct rte_mbuf *m)

{
	rte_mbuf_history_mark(m, RTE_MBUF_HISTORY_OP_LIB_FREE);
	rte_mempool_put(m->pool, m);
}
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Note that the description says that the mbuf must be direct. This is not entirely accurate; the mbuf is allowed to use a

pinned

Quoted 8 lines
Quoted 8 lines
Quoted 2 lines

external buffer, if the mbuf holds the only reference to it. (Most of the mbuf library functions have this documentation

inaccuracy,

Quoted 5 lines

which should be fixed some day.)

So, the fast-free optimization really comes down to rte_pktmbuf_prefree_seg(m), which must not return NULL.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Let's dig into that.

/**
* Decrease reference counter and unlink a mbuf segment *
* This function does the same than a free, except that it does

not

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

* return the segment to its pool.
* It decreases the reference counter, and if it reaches 0, it is
* detached from its parent for an indirect mbuf.
*
* @return
* - (m) if it is the last reference. It can be recycled or

freed.

Quoted 26 lines
Quoted 26 lines
Quoted 26 lines

* - (NULL) if the mbuf still has remaining references on it.
*/
static __rte_always_inline struct rte_mbuf *
rte_pktmbuf_prefree_seg(struct rte_mbuf *m)

{
	bool refcnt_not_one;

	refcnt_not_one = unlikely(rte_mbuf_refcnt_read(m) != 1);
	if (refcnt_not_one && __rte_mbuf_refcnt_update(m, -1) != 0)
		return NULL;

	if (unlikely(!RTE_MBUF_DIRECT(m))) {
		rte_pktmbuf_detach(m);
		if (RTE_MBUF_HAS_EXTBUF(m) &&
				RTE_MBUF_HAS_PINNED_EXTBUF(m) &&
				__rte_pktmbuf_pinned_extbuf_decref(m))
			return NULL;
	}

	if (refcnt_not_one)
		rte_mbuf_refcnt_set(m, 1);
	if (m->nb_segs != 1)
		m->nb_segs = 1;
	if (m->next != NULL)
		m->next = NULL;
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
	return m;
}

This function can only succeed (i.e. return non-NULL) when

'refcnt'

Quoted 4 lines
Quoted 4 lines

is 1

Quoted 3 lines

(or reaches 0).

REQUIREMENT 2: The driver must hold the only reference to the

mbuf,

Quoted 8 lines
Quoted 8 lines
Quoted 4 lines

i.e. 'm->refcnt' must be 1.

When the function succeeds, it initializes the mbuf fields as

required by

Quoted 3 lines

rte_mbuf_raw_free() before returning.

Now, since the driver has exclusive access to the mbuf, it is

free to

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

initialize the 'm->next' and 'm->nb_segs' at any time. It could do that when preparing the Tx descriptor.

This is very interesting, because it means that fast-free does

not

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

prohibit segmented packets!
(But the driver must have sufficient Tx descriptors for all

segments

Quoted 5 lines
Quoted 5 lines

in

Quoted 4 lines

the mbuf.)

Now, lets dig into rte_pktmbuf_prefree_seg()'s block handling

non-

Quoted 28 lines
Quoted 28 lines

direct

Quoted 27 lines

mbufs, i.e. cloned mbufs and mbufs with external buffer:

	if (unlikely(!RTE_MBUF_DIRECT(m))) {
		rte_pktmbuf_detach(m);
		if (RTE_MBUF_HAS_EXTBUF(m) &&
				RTE_MBUF_HAS_PINNED_EXTBUF(m) &&
				__rte_pktmbuf_pinned_extbuf_decref(m))
			return NULL;
	}

Starting with rte_pktmbuf_detach():

static inline void rte_pktmbuf_detach(struct rte_mbuf *m)

{
	struct rte_mempool *mp = m->pool;
	uint32_t mbuf_size, buf_len;
	uint16_t priv_size;

	if (RTE_MBUF_HAS_EXTBUF(m)) {
		/*
		 * The mbuf has the external attached buffer,
		 * we should check the type of the memory pool where
		 * the mbuf was allocated from to detect the pinned
		 * external buffer.
		 */
		uint32_t flags = rte_pktmbuf_priv_flags(mp);
Quoted 13 lines
Quoted 5 lines
Quoted 5 lines
		if (flags & RTE_PKTMBUF_POOL_F_PINNED_EXT_BUF) {
			/*
			 * The pinned external buffer should not be
			 * detached from its backing mbuf, just exit.
			 */
Quoted 8 lines
Quoted 8 lines
			return;
		}
		__rte_pktmbuf_free_extbuf(m);
	} else {
		__rte_pktmbuf_free_direct(m);
	}
	priv_size = rte_pktmbuf_priv_size(mp);
	mbuf_size = (uint32_t)(sizeof(struct rte_mbuf) +

priv_size);

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
	buf_len = rte_pktmbuf_data_room_size(mp);

	m->priv_size = priv_size;
	m->buf_addr = (char *)m + mbuf_size;
	rte_mbuf_iova_set(m, rte_mempool_virt2iova(m) + mbuf_size);
	m->buf_len = (uint16_t)buf_len;
	rte_pktmbuf_reset_headroom(m);
	m->data_len = 0;
	m->ol_flags = 0;
}
Quoted 1 line
Quoted 1 line
Quoted 1 line

The only quick and simple code path through this function is when

the

Quoted 7 lines
Quoted 7 lines

mbuf

Quoted 6 lines

uses a pinned external buffer:

	if (RTE_MBUF_HAS_EXTBUF(m)) {
		uint32_t flags = rte_pktmbuf_priv_flags(mp);
		if (flags & RTE_PKTMBUF_POOL_F_PINNED_EXT_BUF)
			return;
Quoted 6 lines
Quoted 6 lines
Quoted 1 line

REQUIREMENT 3: The mbuf must not be cloned or use a non-pinned

external

Quoted 4 lines

buffer.

Continuing with the next part of rte_pktmbuf_prefree_seg()'s

block:

Quoted 10 lines
Quoted 10 lines
Quoted 6 lines

if (RTE_MBUF_HAS_EXTBUF(m) &&

				RTE_MBUF_HAS_PINNED_EXTBUF(m) &&
				__rte_pktmbuf_pinned_extbuf_decref(m))
			return NULL;

Continuing with the next part of the block in

rte_pktmbuf_prefree_seg():

Quoted 3 lines

/**
* @internal Handle the packet mbufs with attached pinned

external

Quoted 6 lines
Quoted 6 lines

buffer

Quoted 3 lines

* on the mbuf freeing:
*
* - return zero if reference counter in shinfo is one. It means

there is

Quoted 1 line

* no more reference to this pinned buffer and mbuf can be

returned

Quoted 6 lines
Quoted 6 lines

to

Quoted 3 lines

* the pool
*
* - otherwise (if reference counter is not one), decrement

reference

Quoted 1 line

* counter and return non-zero value to prevent freeing the

backing

Quoted 5 lines
Quoted 5 lines

mbuf.

Quoted 4 lines

*
* Returns non zero if mbuf should not be freed. */
static inline int __rte_pktmbuf_pinned_extbuf_decref(struct

rte_mbuf

Quoted 12 lines
Quoted 12 lines

*m)

Quoted 11 lines
{
	struct rte_mbuf_ext_shared_info *shinfo;

	/* Clear flags, mbuf is being freed. */
	m->ol_flags = RTE_MBUF_F_EXTERNAL;
	shinfo = m->shinfo;

	/* Optimize for performance - do not dec/reinit */
	if (likely(rte_mbuf_ext_refcnt_read(shinfo) == 1))
		return 0;
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
	/*
	 * Direct usage of add primitive to avoid
	 * duplication of comparing with one.
	 */
	if (likely(rte_atomic_fetch_add_explicit(&shinfo->refcnt, -

1,

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
				     rte_memory_order_acq_rel) - 1))
		return 1;

	/* Reinitialize counter before mbuf freeing. */
	rte_mbuf_ext_refcnt_set(shinfo, 1);
	return 0;
}
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

Essentially, if the mbuf does use a pinned external buffer, rte_pktmbuf_prefree_seg() only succeeds if that pinned external

buffer is

Quoted 3 lines

only referred to by the mbuf.

REQUIREMENT 4: If the mbuf uses a pinned external buffer, the

mbuf

Quoted 2 lines
Quoted 2 lines

must

Quoted 1 line

hold the only reference to that pinned external buffer, i.e. in

that

Quoted 4 lines
Quoted 4 lines

case,

Quoted 3 lines

'm->shinfo->refcnt' must be 1.

Quoted 8 lines
Quoted 8 lines
Quoted 3 lines

Please review.

If I'm not mistaken, the mbuf library is not a barrier for fast-

freeing

Quoted 1 line

segmented packet mbufs, and thus fast-free of jumbo frames is

possible.

Quoted 2 lines

We need a driver developer to confirm that my suggested approach

-

Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

resetting the mbuf fields, incl. 'm->nb_segs' and 'm->next', when preparing the Tx descriptor - is viable.

Great analysis, makes a lot of sense to me.
Shall we add then a special API to make PMD maintainers life a bit easier:
Something like rte_mbuf_fast_free_prep(mp, mb), that will

optionally

Quoted 8 lines
Quoted 3 lines

check
that requirements outlined above are satisfied for given mbuf and also reset mbuf fields to expected values?

Good idea, Konstantin.

Detailed suggestion below.
Note that __rte_mbuf_raw_sanity_check_mp() is used to checks the

requirements

Quoted 6 lines

after 'nb_segs' and 'next' have been initialized.

/**
* Reinitialize an mbuf for freeing back into the mempool. *
* The caller must ensure that the mbuf comes from the specified

mempool,

Quoted 3 lines

* is direct and only referred to by the caller (refcnt=1). *
* This function is used by drivers in their transmit function for

mbuf fast release

Quoted 5 lines

* when the transmit descriptor is initialized,
* so the driver can call rte_mbuf_raw_free()
* when the packet segment has been transmitted.
*
* @see RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE

Quoted 8 lines

*
* @param mp
* The mempool to which the mbuf belong.
* @param m
* The mbuf being reinitialized.
*/
static __rte_always_inline void rte_mbuf_raw_prefree_seg(const struct rte_mempool *mp, struct

rte_mbuf *m)

Quoted 10 lines
{
	if (m->nb_segs != 1)
		m->nb_segs = 1;
	if (m->next != NULL)
		m->next = NULL;

	__rte_mbuf_raw_sanity_check_mp(m, mp);
	rte_mbuf_history_mark(mbuf,

RTE_MBUF_HISTORY_OP_LIB_PREFREE_RAW);
}

Quoted 5 lines
Quoted 2 lines

Thanks Morten, though should we really panic if condition is not met? Might be just do check first and return an error.

__rte_mbuf_raw_sanity_check_mp() is a no-op unless RTE_LIBRTE_MBUF_DEBUG is enabled.

Yep, I noticed.

Quoted 5 lines

Using it everywhere in alloc/free in the mbuf library is the convention.

And if we don't do it here, the __rte_mbuf_raw_sanity_check_mp() in rte_mbuf_raw_free() will panic later instead.

Why?
This new routine (rte_mbuf_raw_prefree_seg) can check that mbuf satisfies fast-free criteria before updating it, and if conditions are not met simply return an error. Then the driver has several options:
1) Drop the packet silently
2) Refuse to send it
3) Switch to some slower but always working code-path (without fast-free)
4) Panic

Quoted 49 lines

Better to fail early.

Quoted 47 lines
Quoted 6 lines

/**
* Reinitialize a bulk of mbufs for freeing back into the mempool. *
* The caller must ensure that the mbufs come from the specified

mempool,

Quoted 3 lines

* are direct and only referred to by the caller (refcnt=1). *
* This function is used by drivers in their transmit function for

mbuf fast release

Quoted 8 lines

* when the transmit descriptors are initialized,
* so the driver can call rte_mbuf_raw_free_bulk()
* when the packet segments have been transmitted. *
* @see RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE *
* @param mp
* The mempool to which the mbufs belong.

Quoted 8 lines

* @param mbufs
* Array of pointers to mbufs being reinitialized.
* The array must not contain NULL pointers.
* @param count
* Array size.
*/
static __rte_always_inline void rte_mbuf_raw_prefree_seg_bulk(const struct rte_mempool *mp, struct

rte_mbuf

Quoted 19 lines

**mbufs, unsigned int count)

{
	for (unsigned int idx = 0; idx < count; idx++) {
		struct rte_mbuf *m = mbufs[idx];

		if (m->nb_segs != 1)
			m->nb_segs = 1;
		if (m->next != NULL)
			m->next = NULL;

		__rte_mbuf_raw_sanity_check_mp(m, mp);
	}
	rte_mbuf_history_mark_bulk(mbufs, count,

RTE_MBUF_HISTORY_OP_LIB_PREFREE_RAW);
}

Quoted 3 lines

Konstantin

#31 lore raw
Quoted 357 lines

From: Konstantin Ananyev [mailto:[email protected]]
Sent: Monday, 22 December 2025 16.22

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Executive Summary:

My analysis shows that the mbuf library is not a barrier for

fast-

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

freeing

Quoted 1 line

segmented packet mbufs, and thus fast-free of jumbo frames is

possible.

Quoted 5 lines

Detailed Analysis:

The purpose of the mbuf fast-free Tx optimization is to

reduce

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

rte_pktmbuf_free_seg() to something much simpler in the

ethdev

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

drivers, by

Quoted 2 lines

eliminating the code path related to indirect mbufs. Optimally, we want to simplify the ethdev driver's function

that

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

frees the

Quoted 1 line

transmitted mbufs, so it can free them directly to their

mempool

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

without

Quoted 3 lines

accessing the mbufs themselves.

If the driver cannot access the mbuf itself, it cannot

determine

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

which

Quoted 2 lines

mempool it belongs to.
We don't want the driver to access every mbuf being freed;

but if

Quoted 2 lines
Quoted 2 lines

all

Quoted 1 line
Quoted 1 line
Quoted 1 line

mbufs of a Tx queue belong to the same mempool, the driver

can

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

determine

Quoted 3 lines

which mempool by looking into just one of the mbufs.

REQUIREMENT 1: The mbufs of a Tx queue must come from the

same

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

mempool.

Quoted 3 lines

When an mbuf is freed to its mempool, some of the fields in

the

Quoted 23 lines
Quoted 23 lines

mbuf

Quoted 3 lines
Quoted 3 lines

must

Quoted 2 lines

be initialized.
So, for fast-free, this must be done by the driver's function

that

Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

prepares the Tx descriptor.
This is a requirement to the driver, not a requirement to the

application.

Quoted 3 lines

Now, let's dig into the code for freeing an mbuf.
Note: For readability purposes, I'll cut out some code and

comments

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

unrelated to this topic.

static __rte_always_inline void
rte_pktmbuf_free_seg(struct rte_mbuf *m)

{
	m = rte_pktmbuf_prefree_seg(m);
	if (likely(m != NULL))
		rte_mbuf_raw_free(m);
}
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

rte_mbuf_raw_free(m) is simple, so nothing to gain there:

/**
* Put mbuf back into its original mempool.
*
* The caller must ensure that the mbuf is direct and

properly

Quoted 21 lines
Quoted 21 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

* reinitialized (refcnt=1, next=NULL, nb_segs=1), as done by
* rte_pktmbuf_prefree_seg().
*/
static __rte_always_inline void rte_mbuf_raw_free(struct rte_mbuf *m)

{
	rte_mbuf_history_mark(m, RTE_MBUF_HISTORY_OP_LIB_FREE);
	rte_mempool_put(m->pool, m);
}
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Note that the description says that the mbuf must be direct. This is not entirely accurate; the mbuf is allowed to use a

pinned

Quoted 8 lines
Quoted 8 lines
Quoted 2 lines

external buffer, if the mbuf holds the only reference to it. (Most of the mbuf library functions have this documentation

inaccuracy,

Quoted 5 lines

which should be fixed some day.)

So, the fast-free optimization really comes down to rte_pktmbuf_prefree_seg(m), which must not return NULL.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Let's dig into that.

/**
* Decrease reference counter and unlink a mbuf segment *
* This function does the same than a free, except that it

does

Quoted 3 lines
Quoted 3 lines

not

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

* return the segment to its pool.
* It decreases the reference counter, and if it reaches 0,

it is

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

* detached from its parent for an indirect mbuf. *
* @return
* - (m) if it is the last reference. It can be recycled or

freed.

Quoted 1 line
Quoted 1 line
Quoted 1 line

* - (NULL) if the mbuf still has remaining references on

it.

Quoted 40 lines
Quoted 40 lines
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines

*/
static __rte_always_inline struct rte_mbuf *
rte_pktmbuf_prefree_seg(struct rte_mbuf *m)

{
	bool refcnt_not_one;

	refcnt_not_one = unlikely(rte_mbuf_refcnt_read(m) != 1);
	if (refcnt_not_one && __rte_mbuf_refcnt_update(m, -1) != 0)
		return NULL;

	if (unlikely(!RTE_MBUF_DIRECT(m))) {
		rte_pktmbuf_detach(m);
		if (RTE_MBUF_HAS_EXTBUF(m) &&
				RTE_MBUF_HAS_PINNED_EXTBUF(m) &&
				__rte_pktmbuf_pinned_extbuf_decref(m))
			return NULL;
	}

	if (refcnt_not_one)
		rte_mbuf_refcnt_set(m, 1);
	if (m->nb_segs != 1)
		m->nb_segs = 1;
	if (m->next != NULL)
		m->next = NULL;
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
	return m;
}

This function can only succeed (i.e. return non-NULL) when

'refcnt'

Quoted 4 lines
Quoted 4 lines

is 1

Quoted 3 lines

(or reaches 0).

REQUIREMENT 2: The driver must hold the only reference to the

mbuf,

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

i.e. 'm->refcnt' must be 1.

When the function succeeds, it initializes the mbuf fields as

required by

Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

rte_mbuf_raw_free() before returning.

Now, since the driver has exclusive access to the mbuf, it is

free to

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

initialize the 'm->next' and 'm->nb_segs' at any time. It could do that when preparing the Tx descriptor.

This is very interesting, because it means that fast-free

does

Quoted 63 lines
Quoted 63 lines

not

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

prohibit segmented packets!
(But the driver must have sufficient Tx descriptors for all

segments

Quoted 5 lines
Quoted 5 lines

in

Quoted 4 lines

the mbuf.)

Now, lets dig into rte_pktmbuf_prefree_seg()'s block handling

non-

Quoted 28 lines
Quoted 28 lines

direct

Quoted 27 lines

mbufs, i.e. cloned mbufs and mbufs with external buffer:

	if (unlikely(!RTE_MBUF_DIRECT(m))) {
		rte_pktmbuf_detach(m);
		if (RTE_MBUF_HAS_EXTBUF(m) &&
				RTE_MBUF_HAS_PINNED_EXTBUF(m) &&
				__rte_pktmbuf_pinned_extbuf_decref(m))
			return NULL;
	}

Starting with rte_pktmbuf_detach():

static inline void rte_pktmbuf_detach(struct rte_mbuf *m)

{
	struct rte_mempool *mp = m->pool;
	uint32_t mbuf_size, buf_len;
	uint16_t priv_size;

	if (RTE_MBUF_HAS_EXTBUF(m)) {
		/*
		 * The mbuf has the external attached buffer,
		 * we should check the type of the memory pool where
		 * the mbuf was allocated from to detect the pinned
		 * external buffer.
		 */
		uint32_t flags = rte_pktmbuf_priv_flags(mp);
Quoted 13 lines
Quoted 5 lines
Quoted 5 lines
		if (flags & RTE_PKTMBUF_POOL_F_PINNED_EXT_BUF) {
			/*
			 * The pinned external buffer should not be
			 * detached from its backing mbuf, just exit.
			 */
Quoted 8 lines
Quoted 8 lines
			return;
		}
		__rte_pktmbuf_free_extbuf(m);
	} else {
		__rte_pktmbuf_free_direct(m);
	}
	priv_size = rte_pktmbuf_priv_size(mp);
	mbuf_size = (uint32_t)(sizeof(struct rte_mbuf) +

priv_size);

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
	buf_len = rte_pktmbuf_data_room_size(mp);

	m->priv_size = priv_size;
	m->buf_addr = (char *)m + mbuf_size;
	rte_mbuf_iova_set(m, rte_mempool_virt2iova(m) + mbuf_size);
	m->buf_len = (uint16_t)buf_len;
	rte_pktmbuf_reset_headroom(m);
	m->data_len = 0;
	m->ol_flags = 0;
}
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

The only quick and simple code path through this function is

when

Quoted 9 lines
Quoted 9 lines

the

Quoted 8 lines
Quoted 8 lines

mbuf

Quoted 7 lines

uses a pinned external buffer:

	if (RTE_MBUF_HAS_EXTBUF(m)) {
		uint32_t flags = rte_pktmbuf_priv_flags(mp);
		if (flags & RTE_PKTMBUF_POOL_F_PINNED_EXT_BUF)
			return;

REQUIREMENT 3: The mbuf must not be cloned or use a non-

pinned

Quoted 14 lines
Quoted 14 lines
Quoted 5 lines
Quoted 5 lines

external

Quoted 4 lines

buffer.

Continuing with the next part of rte_pktmbuf_prefree_seg()'s

block:

Quoted 8 lines
Quoted 8 lines
Quoted 6 lines

if (RTE_MBUF_HAS_EXTBUF(m) &&

				RTE_MBUF_HAS_PINNED_EXTBUF(m) &&
				__rte_pktmbuf_pinned_extbuf_decref(m))
			return NULL;

Continuing with the next part of the block in

rte_pktmbuf_prefree_seg():

Quoted 1 line
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

/**
* @internal Handle the packet mbufs with attached pinned

external

Quoted 4 lines
Quoted 4 lines

buffer

Quoted 3 lines

* on the mbuf freeing:
*
* - return zero if reference counter in shinfo is one. It

means

Quoted 37 lines
Quoted 37 lines
Quoted 2 lines
Quoted 2 lines

there is

Quoted 1 line

* no more reference to this pinned buffer and mbuf can be

returned

Quoted 6 lines
Quoted 6 lines

to

Quoted 3 lines

* the pool
*
* - otherwise (if reference counter is not one), decrement

reference

Quoted 1 line

* counter and return non-zero value to prevent freeing the

backing

Quoted 5 lines
Quoted 5 lines

mbuf.

Quoted 4 lines

*
* Returns non zero if mbuf should not be freed. */
static inline int __rte_pktmbuf_pinned_extbuf_decref(struct

rte_mbuf

Quoted 12 lines
Quoted 12 lines

*m)

Quoted 11 lines
{
	struct rte_mbuf_ext_shared_info *shinfo;

	/* Clear flags, mbuf is being freed. */
	m->ol_flags = RTE_MBUF_F_EXTERNAL;
	shinfo = m->shinfo;

	/* Optimize for performance - do not dec/reinit */
	if (likely(rte_mbuf_ext_refcnt_read(shinfo) == 1))
		return 0;
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
	/*
	 * Direct usage of add primitive to avoid
	 * duplication of comparing with one.
	 */
	if (likely(rte_atomic_fetch_add_explicit(&shinfo->refcnt, -

1,

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
				     rte_memory_order_acq_rel) - 1))
		return 1;
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
	/* Reinitialize counter before mbuf freeing. */
	rte_mbuf_ext_refcnt_set(shinfo, 1);
	return 0;
}

Essentially, if the mbuf does use a pinned external buffer, rte_pktmbuf_prefree_seg() only succeeds if that pinned

external

Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines

buffer is

Quoted 3 lines

only referred to by the mbuf.

REQUIREMENT 4: If the mbuf uses a pinned external buffer, the

mbuf

Quoted 2 lines
Quoted 2 lines

must

Quoted 1 line

hold the only reference to that pinned external buffer, i.e.

in

Quoted 8 lines
Quoted 8 lines

that

Quoted 7 lines
Quoted 7 lines

case,

Quoted 6 lines

'm->shinfo->refcnt' must be 1.

Please review.

If I'm not mistaken, the mbuf library is not a barrier for

fast-

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

freeing

Quoted 1 line

segmented packet mbufs, and thus fast-free of jumbo frames is

possible.

Quoted 2 lines

We need a driver developer to confirm that my suggested

approach

Quoted 2 lines
Quoted 2 lines

-

Quoted 1 line
Quoted 1 line
Quoted 1 line

resetting the mbuf fields, incl. 'm->nb_segs' and 'm->next',

when

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line

preparing the Tx descriptor - is viable.

Great analysis, makes a lot of sense to me.
Shall we add then a special API to make PMD maintainers life a

bit

Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines

easier:
Something like rte_mbuf_fast_free_prep(mp, mb), that will

optionally

Quoted 2 lines
Quoted 2 lines

check
that requirements outlined above are satisfied for given mbuf

and

Quoted 13 lines
Quoted 13 lines
Quoted 6 lines
Quoted 1 line

also reset mbuf fields to expected values?

Good idea, Konstantin.

Detailed suggestion below.
Note that __rte_mbuf_raw_sanity_check_mp() is used to checks the

requirements

Quoted 6 lines

after 'nb_segs' and 'next' have been initialized.

/**
* Reinitialize an mbuf for freeing back into the mempool. *
* The caller must ensure that the mbuf comes from the specified

Quoted 25 lines
Quoted 4 lines
Quoted 4 lines

mempool,

Quoted 3 lines

* is direct and only referred to by the caller (refcnt=1). *
* This function is used by drivers in their transmit function

for

Quoted 20 lines
Quoted 20 lines

mbuf fast release

Quoted 5 lines

* when the transmit descriptor is initialized,
* so the driver can call rte_mbuf_raw_free()
* when the packet segment has been transmitted.
*
* @see RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE

Quoted 8 lines

*
* @param mp
* The mempool to which the mbuf belong.
* @param m
* The mbuf being reinitialized.
*/
static __rte_always_inline void rte_mbuf_raw_prefree_seg(const struct rte_mempool *mp, struct

rte_mbuf *m)

Quoted 5 lines
{
	if (m->nb_segs != 1)
		m->nb_segs = 1;
	if (m->next != NULL)
		m->next = NULL;

A comment could mention that when the driver calls this function, it has just loaded the mbuf to initialize the transmit descriptor, so m->nb_segs and m->next are already hot in the cache. And no store operations on the mbuf itself will occur for non-segmented packets.

Quoted 22 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
	__rte_mbuf_raw_sanity_check_mp(m, mp);
	rte_mbuf_history_mark(mbuf,

RTE_MBUF_HISTORY_OP_LIB_PREFREE_RAW);
}

Thanks Morten, though should we really panic if condition is not

met?

Quoted 4 lines
Quoted 1 line

Might be just do check first and return an error.

__rte_mbuf_raw_sanity_check_mp() is a no-op unless RTE_LIBRTE_MBUF_DEBUG is enabled.

Yep, I noticed.

Quoted 2 lines

Using it everywhere in alloc/free in the mbuf library is the

convention.

Quoted 3 lines

And if we don't do it here, the __rte_mbuf_raw_sanity_check_mp() in rte_mbuf_raw_free() will panic later instead.

Quoted 11 lines

Why?
This new routine (rte_mbuf_raw_prefree_seg) can check that mbuf satisfies fast-free criteria
before updating it, and if conditions are not met simply return an error.
Then the driver has several options:
1) Drop the packet silently
2) Refuse to send it
3) Switch to some slower but always working code-path (without fast- free)
4) Panic

It boils down to purpose.

The function is designed for use in the transmit code path designated for fast-free, where the application has promised/hinted that packets are fast-free compliant. Violating preconditions in the fast path (by passing packets not compliant to fast-free requirements to this function) is a serious type of bug, for which DPDK usually doesn't provide graceful fallback. I don't want to make an exception (and introduce graceful fallback) for the designated fast-free code path.

My answer would be completely different if we were designing an API for standard packet transmission, whereby some packets living up to certain criteria could take a faster code path for being freed. If that was the case, I would agree with you about returning a value to indicate how to proceed, like rte_pktmbuf_prefree_seg() does.

Quoted 53 lines
Quoted 6 lines

Better to fail early.

Quoted 4 lines
Quoted 4 lines

/**
* Reinitialize a bulk of mbufs for freeing back into the

mempool.

Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

*
* The caller must ensure that the mbufs come from the specified

mempool,

Quoted 3 lines

* are direct and only referred to by the caller (refcnt=1). *
* This function is used by drivers in their transmit function

for

Quoted 17 lines
Quoted 17 lines

mbuf fast release

Quoted 16 lines

* when the transmit descriptors are initialized,
* so the driver can call rte_mbuf_raw_free_bulk()
* when the packet segments have been transmitted. *
* @see RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE
*
* @param mp
* The mempool to which the mbufs belong.
* @param mbufs
* Array of pointers to mbufs being reinitialized.
* The array must not contain NULL pointers.
* @param count
* Array size.
*/
static __rte_always_inline void rte_mbuf_raw_prefree_seg_bulk(const struct rte_mempool *mp,

struct

Quoted 21 lines
Quoted 20 lines

rte_mbuf

Quoted 19 lines

**mbufs, unsigned int count)

{
	for (unsigned int idx = 0; idx < count; idx++) {
		struct rte_mbuf *m = mbufs[idx];

		if (m->nb_segs != 1)
			m->nb_segs = 1;
		if (m->next != NULL)
			m->next = NULL;

		__rte_mbuf_raw_sanity_check_mp(m, mp);
	}
	rte_mbuf_history_mark_bulk(mbufs, count,

RTE_MBUF_HISTORY_OP_LIB_PREFREE_RAW);
}

Quoted 3 lines

Konstantin

#32 lore raw
Quoted 3 lines · Morten Brørup
Quoted 3 lines

From: Konstantin Ananyev [mailto:[email protected]]
Sent: Monday, 22 December 2025 16.22

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

<snip>

Quoted 34 lines
Quoted 22 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
	__rte_mbuf_raw_sanity_check_mp(m, mp);
	rte_mbuf_history_mark(mbuf,

RTE_MBUF_HISTORY_OP_LIB_PREFREE_RAW);
}

Thanks Morten, though should we really panic if condition is not

met?

Quoted 4 lines
Quoted 1 line

Might be just do check first and return an error.

__rte_mbuf_raw_sanity_check_mp() is a no-op unless RTE_LIBRTE_MBUF_DEBUG is enabled.

Yep, I noticed.

Quoted 2 lines

Using it everywhere in alloc/free in the mbuf library is the

convention.

Quoted 3 lines

And if we don't do it here, the __rte_mbuf_raw_sanity_check_mp() in rte_mbuf_raw_free() will panic later instead.

Quoted 11 lines

Why?
This new routine (rte_mbuf_raw_prefree_seg) can check that mbuf satisfies fast-free criteria
before updating it, and if conditions are not met simply return an error.
Then the driver has several options:
1) Drop the packet silently
2) Refuse to send it
3) Switch to some slower but always working code-path (without fast- free)
4) Panic

Quoted 9 lines

It boils down to purpose.

The function is designed for use in the transmit code path designated for fast-free, where the application has promised/hinted that packets are fast-free compliant. Violating preconditions in the fast path (by passing packets not compliant to fast-free requirements to this function) is a serious type of bug, for which DPDK usually doesn't provide graceful fallback. I don't want to make an exception (and introduce graceful fallback) for the designated fast-free code path.

My answer would be completely different if we were designing an API for standard packet transmission, whereby some packets living up to certain criteria could take a faster code path for being freed. If that was the case, I would agree with you about returning a value to indicate how to proceed, like rte_pktmbuf_prefree_seg() does.

I would tend to agree. The extra handling for those cases just expands the code and adds more potential branches to the resulting binary. Lots of the fastpath code just assumes you know what you are doing, and violating constraints will lead to panics and segfaults generally. Therefore panicing in this case doesn't seem a bit deal to me.

/Bruce

#33 lore raw
Quoted 47 lines
Quoted 34 lines
Quoted 22 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
	__rte_mbuf_raw_sanity_check_mp(m, mp);
	rte_mbuf_history_mark(mbuf,

RTE_MBUF_HISTORY_OP_LIB_PREFREE_RAW);
}

Thanks Morten, though should we really panic if condition is not

met?

Quoted 4 lines
Quoted 1 line

Might be just do check first and return an error.

__rte_mbuf_raw_sanity_check_mp() is a no-op unless RTE_LIBRTE_MBUF_DEBUG is enabled.

Yep, I noticed.

Quoted 2 lines

Using it everywhere in alloc/free in the mbuf library is the

convention.

Quoted 3 lines

And if we don't do it here, the __rte_mbuf_raw_sanity_check_mp() in rte_mbuf_raw_free() will panic later instead.

Quoted 11 lines

Why?
This new routine (rte_mbuf_raw_prefree_seg) can check that mbuf satisfies fast-free criteria
before updating it, and if conditions are not met simply return an error.
Then the driver has several options:
1) Drop the packet silently
2) Refuse to send it
3) Switch to some slower but always working code-path (without fast- free)
4) Panic

Quoted 3 lines

It boils down to purpose.

The function is designed for use in the transmit code path designated for fast-free,

where the application has promised/hinted that packets are fast-free compliant.

Quoted 1 line

Violating preconditions in the fast path (by passing packets not compliant to fast-

free requirements to this function) is a serious type of bug, for which DPDK usually doesn't provide graceful fallback.

Quoted 1 line

I don't want to make an exception (and introduce graceful fallback) for the

designated fast-free code path.

Quoted 2 lines

My answer would be completely different if we were designing an API for standard

packet transmission, whereby some packets living up to certain criteria could take a faster code path for being freed.

Quoted 9 lines
Quoted 1 line

If that was the case, I would agree with you about returning a value to indicate

how to proceed, like rte_pktmbuf_prefree_seg() does.

Quoted 1 line

I would tend to agree. The extra handling for those cases just expands the code and adds more potential branches to the resulting binary. Lots of the fastpath code just assumes you know what you are doing, and violating constraints will lead to panics and segfaults generally. Therefore panicing in this case doesn't seem a bit deal to me.

I understand your point lads, and somewhat agree: if we plan to keep FAST_FREE flag forever, then it is probably the simplest and safest approach.
My hope was that such function will open a possibility for the PMDs to implement similar perf improvement completely transparent to the user (without need for special flags and/or pre-requirements). But might be I am looking forward way too far with it.
Even what Morten proposed above is a big step, so ok - let's deal with it first.
Konstantin

#34 lore raw
Quoted 56 lines · On Tue, 13 Jan 2026 14:48:19 +0000 Konstantin Ananyev…
Quoted 47 lines
Quoted 34 lines
Quoted 22 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
	__rte_mbuf_raw_sanity_check_mp(m, mp);
	rte_mbuf_history_mark(mbuf,

RTE_MBUF_HISTORY_OP_LIB_PREFREE_RAW);
}

Thanks Morten, though should we really panic if condition is not

met?

Quoted 4 lines
Quoted 1 line

Might be just do check first and return an error.

__rte_mbuf_raw_sanity_check_mp() is a no-op unless RTE_LIBRTE_MBUF_DEBUG is enabled.

Yep, I noticed.

Quoted 2 lines

Using it everywhere in alloc/free in the mbuf library is the

convention.

Quoted 3 lines

And if we don't do it here, the __rte_mbuf_raw_sanity_check_mp() in rte_mbuf_raw_free() will panic later instead.

Quoted 11 lines

Why?
This new routine (rte_mbuf_raw_prefree_seg) can check that mbuf satisfies fast-free criteria
before updating it, and if conditions are not met simply return an error.
Then the driver has several options:
1) Drop the packet silently
2) Refuse to send it
3) Switch to some slower but always working code-path (without fast- free)
4) Panic

Quoted 3 lines

It boils down to purpose.

The function is designed for use in the transmit code path designated for fast-free,

where the application has promised/hinted that packets are fast-free compliant.

Quoted 1 line

Violating preconditions in the fast path (by passing packets not compliant to fast-

free requirements to this function) is a serious type of bug, for which DPDK usually doesn't provide graceful fallback.

Quoted 1 line

I don't want to make an exception (and introduce graceful fallback) for the

designated fast-free code path.

Quoted 2 lines

My answer would be completely different if we were designing an API for standard

packet transmission, whereby some packets living up to certain criteria could take a faster code path for being freed.

Quoted 9 lines
Quoted 1 line

If that was the case, I would agree with you about returning a value to indicate

how to proceed, like rte_pktmbuf_prefree_seg() does.

Quoted 1 line

I would tend to agree. The extra handling for those cases just expands the code and adds more potential branches to the resulting binary. Lots of the fastpath code just assumes you know what you are doing, and violating constraints will lead to panics and segfaults generally. Therefore panicing in this case doesn't seem a bit deal to me.

Quoted 8 lines

I understand your point lads, and somewhat agree: if we plan to keep FAST_FREE flag forever, then it is probably the simplest and safest approach.
My hope was that such function will open a possibility for the PMDs to implement similar perf improvement completely transparent to the user (without need for special flags and/or pre-requirements). But might be I am looking forward way too far with it.
Even what Morten proposed above is a big step, so ok - let's deal with it first.
Konstantin

We should document any restrictions on this and other semantic assumptions. Ideally, then AI review tools could help new and old developers see places where the assumptions are violated.

#35 lore raw
In reply to Morten Brørup
Quoted 21 lines · Morten Brørup

Executive Summary:

My analysis shows that the mbuf library is not a barrier for fast-freeing segmented packet mbufs, and thus fast-free of jumbo frames is possible.

Detailed Analysis:

The purpose of the mbuf fast-free Tx optimization is to reduce rte_pktmbuf_free_seg() to something much simpler in the ethdev drivers, by eliminating the code path related to indirect mbufs. Optimally, we want to simplify the ethdev driver's function that frees the transmitted mbufs, so it can free them directly to their mempool without accessing the mbufs themselves.

If the driver cannot access the mbuf itself, it cannot determine which mempool it belongs to.
We don't want the driver to access every mbuf being freed; but if all mbufs of a Tx queue belong to the same mempool, the driver can determine which mempool by looking into just one of the mbufs.

<snip>

Quoted 8 lines

If I'm not mistaken, the mbuf library is not a barrier for fast-freeing segmented packet mbufs, and thus fast-free of jumbo frames is possible.

We need a driver developer to confirm that my suggested approach - resetting the mbuf fields, incl. 'm->nb_segs' and 'm->next', when preparing the Tx descriptor - is viable.

Just to make sure I understand things correctly here, the suggestion to prototype is:

- When FAST_FREE flag is set:
- reset the m->nb_segs and m->next fields (if necessary) when accessing the mbuf to write the descriptor
- skip calling pre-free seg on cleanup and instead
- just free all buffers directly to the mempool

Is that correct?

/Bruce

#36 lore raw
Quoted 49 lines

From: Bruce Richardson [mailto:[email protected]]
Sent: Wednesday, 14 January 2026 18.01

Quoted 3 lines · Morten Brørup

Executive Summary:

My analysis shows that the mbuf library is not a barrier for fast-

freeing

Quoted 1 line

segmented packet mbufs, and thus fast-free of jumbo frames is

possible.

Quoted 6 lines

Detailed Analysis:

The purpose of the mbuf fast-free Tx optimization is to reduce rte_pktmbuf_free_seg() to something much simpler in the ethdev

drivers, by

Quoted 2 lines

eliminating the code path related to indirect mbufs. Optimally, we want to simplify the ethdev driver's function that

frees the

Quoted 1 line

transmitted mbufs, so it can free them directly to their mempool

without

Quoted 3 lines

accessing the mbufs themselves.

If the driver cannot access the mbuf itself, it cannot determine

which

Quoted 3 lines

mempool it belongs to.
We don't want the driver to access every mbuf being freed; but if all mbufs of a Tx queue belong to the same mempool, the driver can

determine

Quoted 2 lines

which mempool by looking into just one of the mbufs.

<snip>

Quoted 2 lines

If I'm not mistaken, the mbuf library is not a barrier for fast-

freeing

Quoted 1 line

segmented packet mbufs, and thus fast-free of jumbo frames is

possible.

Quoted 5 lines

We need a driver developer to confirm that my suggested approach - resetting the mbuf fields, incl. 'm->nb_segs' and 'm->next', when preparing the Tx descriptor - is viable.

Just to make sure I understand things correctly here, the suggestion to prototype is:

Quoted 8 lines

- When FAST_FREE flag is set:
- reset the m->nb_segs and m->next fields (if necessary) when
accessing
the mbuf to write the descriptor
- skip calling pre-free seg on cleanup and instead
- just free all buffers directly to the mempool

Is that correct?

Yes.
If this can be done with multi-segment packets, we should be able to eliminate the single-segment requirement to FAST_FREE. (Unless something in the code that writes the descriptor requires single-segment to be super performant, as I suspected of vectorization.)

#37 lore raw
Quoted 56 lines · Morten Brørup
Quoted 56 lines

From: Bruce Richardson [mailto:[email protected]]
Sent: Wednesday, 14 January 2026 18.01

Quoted 3 lines · Morten Brørup

Executive Summary:

My analysis shows that the mbuf library is not a barrier for fast-

freeing

Quoted 1 line

segmented packet mbufs, and thus fast-free of jumbo frames is

possible.

Quoted 6 lines

Detailed Analysis:

The purpose of the mbuf fast-free Tx optimization is to reduce rte_pktmbuf_free_seg() to something much simpler in the ethdev

drivers, by

Quoted 2 lines

eliminating the code path related to indirect mbufs. Optimally, we want to simplify the ethdev driver's function that

frees the

Quoted 1 line

transmitted mbufs, so it can free them directly to their mempool

without

Quoted 3 lines

accessing the mbufs themselves.

If the driver cannot access the mbuf itself, it cannot determine

which

Quoted 3 lines

mempool it belongs to.
We don't want the driver to access every mbuf being freed; but if all mbufs of a Tx queue belong to the same mempool, the driver can

determine

Quoted 2 lines

which mempool by looking into just one of the mbufs.

<snip>

Quoted 2 lines

If I'm not mistaken, the mbuf library is not a barrier for fast-

freeing

Quoted 1 line

segmented packet mbufs, and thus fast-free of jumbo frames is

possible.

Quoted 5 lines

We need a driver developer to confirm that my suggested approach - resetting the mbuf fields, incl. 'm->nb_segs' and 'm->next', when preparing the Tx descriptor - is viable.

Just to make sure I understand things correctly here, the suggestion to prototype is:

- When FAST_FREE flag is set:
- reset the m->nb_segs and m->next fields (if necessary) when
accessing
the mbuf to write the descriptor
- skip calling pre-free seg on cleanup and instead
- just free all buffers directly to the mempool

Quoted 6 lines
Quoted 1 line

Is that correct?

Yes.
If this can be done with multi-segment packets, we should be able to eliminate the single-segment requirement to FAST_FREE. (Unless something in the code that writes the descriptor requires single-segment to be super performant, as I suspected of vectorization.)

Yes, the vector paths do require single-segment, but there is a separate offload flag for that [1]. That flag is part of the overall vector/non-vector Tx flags check.

/Bruce

[1] https://github.com/DPDK/dpdk/blob/main/lib/ethdev/rte_ethdev.h#L1643