Skip to contentSkip to conversation

Free up completed TX buffers

DISCUSSION Zoltan Kiss 2 participants 3 messages May 29, 2015 – Jun 1, 2015

lore
Share & tools
lore
#0 lore raw

Hi,

I've came across an another problem while sorting out the one fixed by my patch "ixgbe: fix checking for tx_free_thresh". Even when the threshold check is correct it can happen that the application run out of free buffers, and the only solution would be to get back the ones from the TX rings. But if their number is still less than tx_free_thresh (per queue), currently there is no interface to achieve that. The bad way is to set tx_free_thresh to 1, but it has a very bad performance penalty. The easy way is just to increase your buffer pool's size to make sure that doesn't happen. But there is no bulletproof way to calculate such a number, and based on my experience it's hard to debug if it causes problem.
I'm thinking about a foolproof way, which is exposing functions like ixgbe_tx_free_bufs from the PMDs, so the application can call it as a last resort to avoid deadlock. Instead it causes probably worse performance, but at least fools like me will easily see that from e.g. oprofile.
How does that sound? Or is there a better way to solve this problem?

Regards,

Zoli

#1 lore raw

Hi Zoltan,

Quoted 2 lines · Zoltan Kiss

The easy way is just to increase your buffer pool's size to make sure that doesn't happen.

Go for it!

Quoted 2 lines

But there is no bulletproof way to calculate such
a number

Yeah, there are many places for mbufs to stay :( I would try:

Mempool size = sum(numbers of all TX descriptors)
+ sum(rx_free_thresh)
+ (mempool cache size * (number of lcores - 1))
+ (burst size * number of lcores)

Quoted 3 lines

I'm thinking about a foolproof way, which is exposing functions like ixgbe_tx_free_bufs from the PMDs, so the application can call it as a last resort to avoid deadlock.

Have a look at rte_eth_dev_tx_queue_stop()/start(). Some NICs (i.e. ixgbe) do reset the queue and free all the mbufs.

Regards,
Andriy

#2 lore raw
Quoted 7 lines · Andriy Berestovskyy

Hi Zoltan,

Quoted 2 lines · Zoltan Kiss

The easy way is just to increase your buffer pool's size to make sure that doesn't happen.

Go for it!

I went for it, my question is whether is it a good and worthwhile idea to give the applications a last resort option for rainy days? It's a problem which probably won't occur very often, but when it does, I think it can take painfully long until you figure out what's wrong.

Quoted 10 lines
Quoted 3 lines

But there is no bulletproof way to calculate such
a number

Yeah, there are many places for mbufs to stay :( I would try:

Mempool size = sum(numbers of all TX descriptors)
+ sum(rx_free_thresh)
+ (mempool cache size * (number of lcores - 1))
+ (burst size * number of lcores)

It heavily depends on what your application does, and I think it's easy to make a mistake in these calculations.

Quoted 7 lines
Quoted 4 lines

I'm thinking about a foolproof way, which is exposing functions like ixgbe_tx_free_bufs from the PMDs, so the application can call it as a last resort to avoid deadlock.

Have a look at rte_eth_dev_tx_queue_stop()/start(). Some NICs (i.e. ixgbe) do reset the queue and free all the mbufs.

That's a bit drastic, I just want to flush the finished TX buffers, even if tx_free_thresh were not reached.
An easy option would be to use rte_eth_tx_burst(..., nb_pkts=0), I'm already using this to enforce TX completion if it's really needed. It checks for tx_free_thresh, like this:

	/* Check if the descriptor ring needs to be cleaned. */
	if ((txq->nb_tx_desc - txq->nb_tx_free) > txq->tx_free_thresh)
		i40e_xmit_cleanup(txq);

My idea is to extend this condition and add " || nb_pkts == 0", so you can force cleanup. But there might be others who uses this same way to do manual TX completion, and they expect that it only happens when tx_free_thresh is reached.

Quoted 4 lines

Regards,
Andriy