Skip to contentSkip to conversation

net/i40e: improve free mbuf

PATCH Superseded Feifei Wang 3 participants 16 messages May 27, 2021 – Jul 1, 2021 Network drivers

lore
Share & tools
lore
#0 lore raw

cover net/i40e: improve free mbuf

For i40e Tx path, use bulk free of the buffers when mbuf fast free mode is enabled. This can efficiently improve the performance.

Feifei Wang (2):
net/i40e: improve performance for scalar Tx
net/i40e: improve performance for vector Tx

 drivers/net/i40e/i40e_rxtx.c            |  5 ++++-
 drivers/net/i40e/i40e_rxtx_vec_common.h | 11 +++++++++++
 2 files changed, 15 insertions(+), 1 deletion(-)
Signature · 1 line
2.25.1
#1 lore raw

PATCH 1/2 net/i40e: improve performance for scalar Tx

For i40e scalar Tx path, if implement FAST_FREE_MBUF mode, it means per-queue all mbufs come from the same mempool and have refcnt = 1.

Thus we can use bulk free of the buffers when mbuf fast free mode is enabled.

For scalar path in arm platform:
In n1sdp, performance is improved by 7.8%;
In thunderx2, performance is improved by 6.7%.

For scalar path in x86 platform,
performance is improved by 6%.

---
 drivers/net/i40e/i40e_rxtx.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
drivers/net/i40e/i40e_rxtx.c +4 -1
index 6c58decece..fe7b20f750 100644
@@ -1295,6 +1295,7 @@ i40e_tx_free_bufs(struct i40e_tx_queue *txq)
 {
 	struct i40e_tx_entry *txep;
 	uint16_t i;
+	struct rte_mbuf *free[RTE_I40E_TX_MAX_FREE_BUF_SZ];
 
 	if ((txq->tx_ring[txq->tx_next_dd].cmd_type_offset_bsz &
 			rte_cpu_to_le_64(I40E_TXD_QW1_DTYPE_MASK)) !=
@@ -1308,9 +1309,11 @@ i40e_tx_free_bufs(struct i40e_tx_queue *txq)
 
 	if (txq->offloads & DEV_TX_OFFLOAD_MBUF_FAST_FREE) {
 		for (i = 0; i < txq->tx_rs_thresh; ++i, ++txep) {
-			rte_mempool_put(txep->mbuf->pool, txep->mbuf);
+			free[i] = txep->mbuf;
 			txep->mbuf = NULL;
 		}
+		rte_mempool_put_bulk(free[0]->pool, (void **)free,
+					txq->tx_rs_thresh);
 	} else {
 		for (i = 0; i < txq->tx_rs_thresh; ++i, ++txep) {
 			rte_pktmbuf_free_seg(txep->mbuf);
Signature · 1 line
2.25.1
#2 lore raw

Re: [dpdk-dev] [PATCH v1 1/2] net/i40e: improve performance for scalar Tx

Quoted 41 lines

-----Original Message-----
From: Feifei Wang <[email protected]>
Sent: Thursday, May 27, 2021 4:17 PM
To: Xing, Beilei <[email protected]>
Cc: [email protected]; [email protected]; Feifei Wang <[email protected]>; Ruifeng Wang <[email protected]>
Subject: [PATCH v1 1/2] net/i40e: improve performance for scalar Tx

For i40e scalar Tx path, if implement FAST_FREE_MBUF mode, it means per- queue all mbufs come from the same mempool and have refcnt = 1.

Thus we can use bulk free of the buffers when mbuf fast free mode is enabled.

For scalar path in arm platform:
In n1sdp, performance is improved by 7.8%; In thunderx2, performance is improved by 6.7%.

For scalar path in x86 platform,
performance is improved by 6%.

---
 drivers/net/i40e/i40e_rxtx.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
drivers/net/i40e/i40e_rxtx.c index renamed from drivers/net/i40e/i40e_rxtx.c +0 -0

6c58decece..fe7b20f750 100644

drivers/net/i40e/i40e_rxtx.c +1 -0
@@ -1295,6 +1295,7 @@ i40e_tx_free_bufs(struct i40e_tx_queue *txq)  {
 	struct i40e_tx_entry *txep;
 	uint16_t i;
+	struct rte_mbuf *free[RTE_I40E_TX_MAX_FREE_BUF_SZ];
 
 	if ((txq->tx_ring[txq->tx_next_dd].cmd_type_offset_bsz &
 			rte_cpu_to_le_64(I40E_TXD_QW1_DTYPE_MASK)) !=

drivers/net/i40e/i40e_rxtx.c i40e_tx_free_bufs(struct i40e_tx_queue *txq) · line 1312

Quoted 5 lines
drivers/net/i40e/i40e_rxtx.c +1 -1
@@ -1308,9 +1309,11 @@ i40e_tx_free_bufs(struct i40e_tx_queue *txq)
 
 	if (txq->offloads & DEV_TX_OFFLOAD_MBUF_FAST_FREE) {
 		for (i = 0; i < txq->tx_rs_thresh; ++i, ++txep) {
-			rte_mempool_put(txep->mbuf->pool, txep->mbuf);
+			free[i] = txep->mbuf;

The tx_rs_thresh can be 'nb_desc - 3', so if tx_rs_thres > RTE_I40E_TX_MAX_FREE_BUF_SZ, there'll be out of bounds, right?

Quoted 9 lines
drivers/net/i40e/i40e_rxtx.c +2 -1
 			txep->mbuf = NULL;
 		}
+		rte_mempool_put_bulk(free[0]->pool, (void **)free,
+					txq->tx_rs_thresh);
 	} else {
 		for (i = 0; i < txq->tx_rs_thresh; ++i, ++txep) {
 			rte_pktmbuf_free_seg(txep->mbuf);
--

2.25.1

#3 lore raw

[dpdk-dev] 回复: [PATCH v1 1/2] net/i40e: improve performance for scalar Tx

Hi, Beilei

Thanks for your comments, please see below.

Quoted 53 lines

-----邮件原件-----
发件人: Xing, Beilei <[email protected]>
发送时间: 2021年6月22日 14:08
收件人: Feifei Wang <[email protected]>
抄送: [email protected]; nd <[email protected]>; Ruifeng Wang <[email protected]>
主题: RE: [PATCH v1 1/2] net/i40e: improve performance for scalar Tx

Quoted 33 lines

-----Original Message-----
From: Feifei Wang <[email protected]>
Sent: Thursday, May 27, 2021 4:17 PM
To: Xing, Beilei <[email protected]>
Cc: [email protected]; [email protected]; Feifei Wang <[email protected]>; Ruifeng Wang <[email protected]>
Subject: [PATCH v1 1/2] net/i40e: improve performance for scalar Tx

For i40e scalar Tx path, if implement FAST_FREE_MBUF mode, it means per- queue all mbufs come from the same mempool and have refcnt = 1.

Thus we can use bulk free of the buffers when mbuf fast free mode is enabled.

For scalar path in arm platform:
In n1sdp, performance is improved by 7.8%; In thunderx2, performance is improved by 6.7%.

For scalar path in x86 platform,
performance is improved by 6%.

---
 drivers/net/i40e/i40e_rxtx.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
(quoted diff) +0 -0

b/drivers/net/i40e/i40e_rxtx.c index
6c58decece..fe7b20f750 100644

drivers/net/i40e/i40e_rxtx.c i40e_tx_free_bufs(struct i40e_tx_queue *txq) { · line 1300

Quoted 6 lines
drivers/net/i40e/i40e_rxtx.c +1 -0
@@ -1295,6 +1295,7 @@ i40e_tx_free_bufs(struct i40e_tx_queue *txq)  {
 	struct i40e_tx_entry *txep;
 	uint16_t i;
+	struct rte_mbuf *free[RTE_I40E_TX_MAX_FREE_BUF_SZ];
 
 	if ((txq->tx_ring[txq->tx_next_dd].cmd_type_offset_bsz &
 
	rte_cpu_to_le_64(I40E_TXD_QW1_DTYPE_MASK)) != @@ -1308,9

+1309,11

Quoted 2 lines
drivers/net/i40e/i40e_rxtx.c +0 -0
 @@ i40e_tx_free_bufs(struct i40e_tx_queue *txq)
Quoted 7 lines

drivers/net/i40e/i40e_rxtx.c i40e_tx_free_bufs(struct i40e_tx_queue *txq) · line 1312

Quoted 4 lines
drivers/net/i40e/i40e_rxtx.c +1 -1
 	if (txq->offloads & DEV_TX_OFFLOAD_MBUF_FAST_FREE) {
 		for (i = 0; i < txq->tx_rs_thresh; ++i, ++txep) {
-			rte_mempool_put(txep->mbuf->pool, txep->mbuf);
+			free[i] = txep->mbuf;

The tx_rs_thresh can be 'nb_desc - 3', so if tx_rs_thres > RTE_I40E_TX_MAX_FREE_BUF_SZ, there'll be out of bounds, right?

Actually tx_rs_thresh <= tx__free_thresh < nb_desc - 3 (i40e_dev_tx_queue_setup). However, I don't know how it affects the relationship between tx_rs_thresh and RTE_I40E_TX_MAX_FREE_BUF_SZ.

Furthermore, I think you are right that tx_rs_thres can be greater than RTE_I40E_TX_MAX_FREE_BUF_SZ in tx_simple_mode (i40e_set_tx_function_flag).

Thus, in scalar path, we can change like:
---------------------------------------------------------------------------------------------------------------
int n = txq->tx_rs_thresh;
int32_t i = 0, j = 0;
const int32_t k = RTE_ALIGN_FLOOR(n, RTE_I40E_TX_MAX_FREE_BUF_SZ); const int32_t m = n % RTE_I40E_TX_MAX_FREE_BUF_SZ; struct rte_mbuf *free[RTE_I40E_TX_MAX_FREE_BUF_SZ];

For FAST_FREE_MODE:

if (k) {
	for (j = 0; j != k - RTE_I40E_TX_MAX_FREE_BUF_SZ;
			j += RTE_I40E_TX_MAX_FREE_BUF_SZ) {
		for (i = 0; i <RTE_I40E_TX_MAX_FREE_BUF_SZ; ++i, ++txep) {
			free[i] = txep->mbuf;
			txep->mbuf = NULL;
		}
		rte_mempool_put_bulk(free[0]->pool, (void **)free,
					RTE_I40E_TX_MAX_FREE_BUF_SZ);
	}
} else {
	for (i = 0; i < m; ++i, ++txep) {
		free[i] = txep->mbuf;
		txep->mbuf = NULL;
	}
	rte_mempool_put_bulk(free[0]->pool, (void **)free, m);
}

---------------------------------------------------------------------------------------------------------------

Best Regards
Feifei

#4 lore raw

[dpdk-dev] 回复: [PATCH v1 1/2] net/i40e: improve performance for scalar Tx

Sorry for a mistake for the code, it should be:
------------------------------------------------------------------------------------------------
int n = txq->tx_rs_thresh;
int32_t i = 0, j = 0;
const int32_t k = RTE_ALIGN_FLOOR(n, RTE_I40E_TX_MAX_FREE_BUF_SZ); const int32_t m = n % RTE_I40E_TX_MAX_FREE_BUF_SZ; struct rte_mbuf *free[RTE_I40E_TX_MAX_FREE_BUF_SZ];

For FAST_FREE_MODE:

if (k) {
for (j = 0; j != k - RTE_I40E_TX_MAX_FREE_BUF_SZ;

 			j += RTE_I40E_TX_MAX_FREE_BUF_SZ) {
		for (i = 0; i <RTE_I40E_TX_MAX_FREE_BUF_SZ; ++i, ++txep) {
			free[i] = txep->mbuf;
			txep->mbuf = NULL;
		}

rte_mempool_put_bulk(free[0]->pool, (void **)free,
RTE_I40E_TX_MAX_FREE_BUF_SZ);
}
}

if (m) {

 	for (i = 0; i < m; ++i, ++txep) {
		free[i] = txep->mbuf;
 		txep->mbuf = NULL;
	}

}
rte_mempool_put_bulk(free[0]->pool, (void **)free, m); }
------------------------------------------------------------------------------------------------

#5 lore raw

Re: [dpdk-dev] [PATCH v1 1/2] net/i40e: improve performance for scalar Tx

Quoted 32 lines

-----Original Message-----
From: Feifei Wang <[email protected]>
Sent: Tuesday, June 22, 2021 6:08 PM
To: Xing, Beilei <[email protected]>
Cc: [email protected]; nd <[email protected]>; Ruifeng Wang <[email protected]>; nd <[email protected]>
Subject: 回复: [PATCH v1 1/2] net/i40e: improve performance for scalar Tx

Sorry for a mistake for the code, it should be:
------------------------------------------------------------------------------------------------
int n = txq->tx_rs_thresh;
int32_t i = 0, j = 0;
const int32_t k = RTE_ALIGN_FLOOR(n, RTE_I40E_TX_MAX_FREE_BUF_SZ); const int32_t m = n % RTE_I40E_TX_MAX_FREE_BUF_SZ; struct rte_mbuf *free[RTE_I40E_TX_MAX_FREE_BUF_SZ];

For FAST_FREE_MODE:

if (k) {

(quoted diff) +0 -0
 	for (j = 0; j != k - RTE_I40E_TX_MAX_FREE_BUF_SZ;
 			j += RTE_I40E_TX_MAX_FREE_BUF_SZ) {
		for (i = 0; i <RTE_I40E_TX_MAX_FREE_BUF_SZ; ++i, ++txep) {
			free[i] = txep->mbuf;
			txep->mbuf = NULL;
		}
(quoted diff) +0 -0
 		rte_mempool_put_bulk(free[0]->pool, (void **)free,
 					RTE_I40E_TX_MAX_FREE_BUF_SZ);
 	}
 }
Quoted 8 lines

if (m) {

 	for (i = 0; i < m; ++i, ++txep) {
		free[i] = txep->mbuf;
 		txep->mbuf = NULL;
	}

}
rte_mempool_put_bulk(free[0]->pool, (void **)free, m); }
------------------------------------------------------------------------------------------------

Seems no logical problem, but the code looks heavy due to for loops.
Did you run performance with this change when tx_rs_thresh > RTE_I40E_TX_MAX_FREE_BUF_SZ?

#6 lore raw

[dpdk-dev] 回复: [PATCH v1 1/2] net/i40e: improve performance for scalar Tx

<snip>

Quoted 28 lines
Quoted 20 lines

int n = txq->tx_rs_thresh;
int32_t i = 0, j = 0;
const int32_t k = RTE_ALIGN_FLOOR(n, RTE_I40E_TX_MAX_FREE_BUF_SZ); const int32_t m = n % RTE_I40E_TX_MAX_FREE_BUF_SZ; struct rte_mbuf *free[RTE_I40E_TX_MAX_FREE_BUF_SZ];

For FAST_FREE_MODE:

if (k) {

(quoted diff) +0 -0
 	for (j = 0; j != k - RTE_I40E_TX_MAX_FREE_BUF_SZ;
 			j += RTE_I40E_TX_MAX_FREE_BUF_SZ) {
		for (i = 0; i <RTE_I40E_TX_MAX_FREE_BUF_SZ; ++i, ++txep) {
			free[i] = txep->mbuf;
			txep->mbuf = NULL;
		}
(quoted diff) +0 -0
 		rte_mempool_put_bulk(free[0]->pool, (void **)free,
 					RTE_I40E_TX_MAX_FREE_BUF_SZ);
 	}
 }
Quoted 7 lines

if (m) {

 	for (i = 0; i < m; ++i, ++txep) {
		free[i] = txep->mbuf;
 		txep->mbuf = NULL;
	}

}
rte_mempool_put_bulk(free[0]->pool, (void **)free, m); }

Quoted 3 lines

Seems no logical problem, but the code looks heavy due to for loops. Did you run performance with this change when tx_rs_thresh > RTE_I40E_TX_MAX_FREE_BUF_SZ?

Sorry for my late rely. It takes me some time to do the test for this path and following is my test results:

First, I come up with another way to solve this bug and compare it with "loop"(size of 'free' is 64). That is set the size of 'free' as a large constant. We know:
tx_rs_thresh < ring_desc_size < I40E_MAX_RING_DESC(4096), so we can directly define as:
struct rte_mbuf *free[RTE_I40E_TX_MAX_FREE_BUF_SZ];

[1]Test Config:
MRR Test: two porst & bi-directional flows & one core RX API: i40e_recv_pkts_bulk_alloc
TX API: i40e_xmit_pkts_simple
ring_descs_size: 1024
Ring_I40E_TX_MAX_FREE_SZ: 64

[2]Scheme:
tx_rs_thresh = I40E_DEFAULT_TX_RSBIT_THRESH
tx_free_thresh = I40E_DEFAULT_TX_FREE_THRESH
tx_rs_thresh <= tx_free_thresh < nb_tx_desc
So we change the value of 'tx_rs_thresh' by adjust I40E_DEFAULT_TX_RSBIT_THRESH

[3]Test Results (performance improve):
In X86:
tx_rs_thresh/ tx_free_thresh 32/32 256/256 512/512
1.mempool_put(base) 0 0 0
2.mempool_put_bulk:loop +4.7% +5.6% +7.0%
3.mempool_put_bulk:large size for free +3.8% +2.3% -2.0%
(free[I40E_MAX_RING_DESC])

In Arm:
N1SDP:
tx_rs_thresh/ tx_free_thresh 32/32 256/256 512/512
1.mempool_put(base) 0 0 0
2.mempool_put_bulk:loop +7.9% +9.1% +2.9%
3.mempool_put_bulk:large size for free +7.1% +8.7% +3.4%
(free[I40E_MAX_RING_DESC])

Thunderx2:
tx_rs_thresh/ tx_free_thresh 32/32 256/256 512/512
1.mempool_put(base) 0 0 0
2.mempool_put_bulk:loop +7.6% +10.5% +7.6%
3.mempool_put_bulk:large size for free +1.7% +18.4% +10.2%
(free[I40E_MAX_RING_DESC])

As a result, I feel maybe 'loop' is better and it seems not very heavy according to the test. What about your views and look forward to your reply.
Thanks a lot.

#7 lore raw

Re: [dpdk-dev] [PATCH v1 1/2] net/i40e: improve performance for scalar Tx

Quoted 89 lines

-----Original Message-----
From: Feifei Wang <[email protected]>
Sent: Friday, June 25, 2021 5:40 PM
To: Xing, Beilei <[email protected]>
Cc: [email protected]; nd <[email protected]>; Ruifeng Wang <[email protected]>; nd <[email protected]>; nd <[email protected]>
Subject: 回复: [PATCH v1 1/2] net/i40e: improve performance for scalar Tx

<snip>

Quoted 28 lines
Quoted 20 lines

int n = txq->tx_rs_thresh;
int32_t i = 0, j = 0;
const int32_t k = RTE_ALIGN_FLOOR(n, RTE_I40E_TX_MAX_FREE_BUF_SZ); const int32_t m = n % RTE_I40E_TX_MAX_FREE_BUF_SZ; struct rte_mbuf *free[RTE_I40E_TX_MAX_FREE_BUF_SZ];

For FAST_FREE_MODE:

if (k) {

(quoted diff) +0 -0
 	for (j = 0; j != k - RTE_I40E_TX_MAX_FREE_BUF_SZ;
 			j += RTE_I40E_TX_MAX_FREE_BUF_SZ) {
		for (i = 0; i <RTE_I40E_TX_MAX_FREE_BUF_SZ; ++i, ++txep) {
			free[i] = txep->mbuf;
			txep->mbuf = NULL;
		}
(quoted diff) +0 -0
 		rte_mempool_put_bulk(free[0]->pool, (void **)free,
 					RTE_I40E_TX_MAX_FREE_BUF_SZ);
 	}
 }
Quoted 7 lines

if (m) {

 	for (i = 0; i < m; ++i, ++txep) {
		free[i] = txep->mbuf;
 		txep->mbuf = NULL;
	}

}
rte_mempool_put_bulk(free[0]->pool, (void **)free, m); }

Quoted 3 lines

Seems no logical problem, but the code looks heavy due to for loops. Did you run performance with this change when tx_rs_thresh > RTE_I40E_TX_MAX_FREE_BUF_SZ?

Sorry for my late rely. It takes me some time to do the test for this path and following is my test results:

First, I come up with another way to solve this bug and compare it with "loop"(size of 'free' is 64).
That is set the size of 'free' as a large constant. We know:
tx_rs_thresh < ring_desc_size < I40E_MAX_RING_DESC(4096), so we can directly define as:
struct rte_mbuf *free[RTE_I40E_TX_MAX_FREE_BUF_SZ];

[1]Test Config:
MRR Test: two porst & bi-directional flows & one core RX API:
i40e_recv_pkts_bulk_alloc TX API: i40e_xmit_pkts_simple
ring_descs_size: 1024
Ring_I40E_TX_MAX_FREE_SZ: 64

[2]Scheme:
tx_rs_thresh = I40E_DEFAULT_TX_RSBIT_THRESH tx_free_thresh = I40E_DEFAULT_TX_FREE_THRESH tx_rs_thresh <= tx_free_thresh < nb_tx_desc So we change the value of 'tx_rs_thresh' by adjust I40E_DEFAULT_TX_RSBIT_THRESH

[3]Test Results (performance improve):
In X86:
tx_rs_thresh/ tx_free_thresh 32/32 256/256 512/512
1.mempool_put(base) 0 0 0
2.mempool_put_bulk:loop +4.7% +5.6% +7.0%
3.mempool_put_bulk:large size for free +3.8% +2.3% -2.0%
(free[I40E_MAX_RING_DESC])

In Arm:
N1SDP:
tx_rs_thresh/ tx_free_thresh 32/32 256/256 512/512
1.mempool_put(base) 0 0 0
2.mempool_put_bulk:loop +7.9% +9.1% +2.9%
3.mempool_put_bulk:large size for free +7.1% +8.7% +3.4%
(free[I40E_MAX_RING_DESC])

Thunderx2:
tx_rs_thresh/ tx_free_thresh 32/32 256/256 512/512
1.mempool_put(base) 0 0 0
2.mempool_put_bulk:loop +7.6% +10.5% +7.6%
3.mempool_put_bulk:large size for free +1.7% +18.4% +10.2%
(free[I40E_MAX_RING_DESC])

Quoted 4 lines

As a result, I feel maybe 'loop' is better and it seems not very heavy according to the test.
What about your views and look forward to your reply. Thanks a lot.

Thanks for your patch and test.
It looks OK for me, please send V2.

#8 lore raw

[dpdk-dev] 回复: [PATCH v1 1/2] net/i40e: improve performance for scalar Tx

Quoted 101 lines

-----邮件原件-----
发件人: Xing, Beilei <[email protected]>
发送时间: 2021年6月28日 10:27
收件人: Feifei Wang <[email protected]>
抄送: [email protected]; nd <[email protected]>; Ruifeng Wang <[email protected]>; nd <[email protected]>; nd <[email protected]> 主题: RE: [PATCH v1 1/2] net/i40e: improve performance for scalar Tx

Quoted 9 lines

-----Original Message-----
From: Feifei Wang <[email protected]>
Sent: Friday, June 25, 2021 5:40 PM
To: Xing, Beilei <[email protected]>
Cc: [email protected]; nd <[email protected]>; Ruifeng Wang <[email protected]>; nd <[email protected]>; nd <[email protected]>
Subject: 回复: [PATCH v1 1/2] net/i40e: improve performance for scalar Tx

Quoted 5 lines

<snip>

Quoted 3 lines
Quoted 3 lines

int n = txq->tx_rs_thresh;
int32_t i = 0, j = 0;
const int32_t k = RTE_ALIGN_FLOOR(n,

RTE_I40E_TX_MAX_FREE_BUF_SZ);

Quoted 1 line
Quoted 1 line
Quoted 1 line

const int32_t m = n % RTE_I40E_TX_MAX_FREE_BUF_SZ; struct

rte_mbuf

Quoted 73 lines
Quoted 24 lines
Quoted 16 lines

*free[RTE_I40E_TX_MAX_FREE_BUF_SZ];

For FAST_FREE_MODE:

if (k) {

(quoted diff) +0 -0
 	for (j = 0; j != k - RTE_I40E_TX_MAX_FREE_BUF_SZ;
 			j += RTE_I40E_TX_MAX_FREE_BUF_SZ) {
		for (i = 0; i <RTE_I40E_TX_MAX_FREE_BUF_SZ; ++i, ++txep) {
			free[i] = txep->mbuf;
			txep->mbuf = NULL;
		}
(quoted diff) +0 -0
 		rte_mempool_put_bulk(free[0]->pool, (void **)free,
 					RTE_I40E_TX_MAX_FREE_BUF_SZ);
 	}
 }
Quoted 7 lines

if (m) {

 	for (i = 0; i < m; ++i, ++txep) {
		free[i] = txep->mbuf;
 		txep->mbuf = NULL;
	}

}
rte_mempool_put_bulk(free[0]->pool, (void **)free, m); }

Quoted 3 lines

Seems no logical problem, but the code looks heavy due to for loops. Did you run performance with this change when tx_rs_thresh > RTE_I40E_TX_MAX_FREE_BUF_SZ?

Sorry for my late rely. It takes me some time to do the test for this path and following is my test results:

First, I come up with another way to solve this bug and compare it with "loop"(size of 'free' is 64).
That is set the size of 'free' as a large constant. We know:
tx_rs_thresh < ring_desc_size < I40E_MAX_RING_DESC(4096), so we can directly define as:
struct rte_mbuf *free[RTE_I40E_TX_MAX_FREE_BUF_SZ];

[1]Test Config:
MRR Test: two porst & bi-directional flows & one core RX API:
i40e_recv_pkts_bulk_alloc TX API: i40e_xmit_pkts_simple
ring_descs_size: 1024
Ring_I40E_TX_MAX_FREE_SZ: 64

[2]Scheme:
tx_rs_thresh = I40E_DEFAULT_TX_RSBIT_THRESH tx_free_thresh = I40E_DEFAULT_TX_FREE_THRESH tx_rs_thresh <= tx_free_thresh < nb_tx_desc So we change the value of 'tx_rs_thresh' by adjust I40E_DEFAULT_TX_RSBIT_THRESH

[3]Test Results (performance improve):
In X86:
tx_rs_thresh/ tx_free_thresh 32/32 256/256 512/512
1.mempool_put(base) 0 0 0
2.mempool_put_bulk:loop +4.7% +5.6% +7.0%
3.mempool_put_bulk:large size for free +3.8% +2.3% -2.0%
(free[I40E_MAX_RING_DESC])

In Arm:
N1SDP:
tx_rs_thresh/ tx_free_thresh 32/32 256/256 512/512
1.mempool_put(base) 0 0 0
2.mempool_put_bulk:loop +7.9% +9.1% +2.9%
3.mempool_put_bulk:large size for free +7.1% +8.7% +3.4%
(free[I40E_MAX_RING_DESC])

Thunderx2:
tx_rs_thresh/ tx_free_thresh 32/32 256/256 512/512
1.mempool_put(base) 0 0 0
2.mempool_put_bulk:loop +7.6% +10.5% +7.6%
3.mempool_put_bulk:large size for free +1.7% +18.4% +10.2%
(free[I40E_MAX_RING_DESC])

Quoted 7 lines
Quoted 4 lines

As a result, I feel maybe 'loop' is better and it seems not very heavy according to the test.
What about your views and look forward to your reply. Thanks a lot.

Thanks for your patch and test.
It looks OK for me, please send V2.

Thanks for the reviewing, I will update the V2 version.

#9 lore raw

PATCH 2/2 net/i40e: improve performance for vector Tx

In reply to Feifei Wang

For i40e vector Tx path, if tx_offload is set as FAST_FREE_MBUF mode, no mbuf fast free operations are executed. To fix this, add mbuf fast free mode for vector Tx path.

Furthermore, for i40e vector Tx path, if implement FAST_FREE_MBUF mode, it means per-queue all mbufs come from the same mempool and have refcnt = 1. Thus we can use bulk free of the buffers when mbuf fast free mode is enabled.

For vector path in arm platform:
In n1sdp, performance is improved by 18.4%;
In thunderx2, performance is improved by 23%.

For vector path in x86 platform:
No performance changes.

---
 drivers/net/i40e/i40e_rxtx_vec_common.h | 11 +++++++++++
 1 file changed, 11 insertions(+)
drivers/net/i40e/i40e_rxtx_vec_common.h +11 -0
index 16fcf0aec6..f52ed98d62 100644
@@ -99,6 +99,16 @@ i40e_tx_free_bufs(struct i40e_tx_queue *txq)
 	  * tx_next_dd - (tx_rs_thresh-1)
 	  */
 	txep = &txq->sw_ring[txq->tx_next_dd - (n - 1)];
+
+	if (txq->offloads & DEV_TX_OFFLOAD_MBUF_FAST_FREE) {
+		for (i = 0; i < n; i++) {
+			free[i] = txep[i].mbuf;
+			txep[i].mbuf = NULL;
+		}
+		rte_mempool_put_bulk(free[0]->pool, (void **)free, n);
+		goto done;
+	}
+
 	m = rte_pktmbuf_prefree_seg(txep[0].mbuf);
 	if (likely(m != NULL)) {
 		free[0] = m;
@@ -126,6 +136,7 @@ i40e_tx_free_bufs(struct i40e_tx_queue *txq)
 		}
 	}
 
+done:
 	/* buffers were freed, update counters */
 	txq->nb_tx_free = (uint16_t)(txq->nb_tx_free + txq->tx_rs_thresh);
 	txq->tx_next_dd = (uint16_t)(txq->tx_next_dd + txq->tx_rs_thresh);
Signature · 1 line
2.25.1
#10 lore raw
In reply to Feifei Wang

Hi, Qi

Can you help review these patches?
Thanks very much.

Best Regards
Feifei

Quoted 20 lines

-----邮件原件-----
发件人: Feifei Wang <[email protected]>
发送时间: 2021年5月27日 16:17
抄送: [email protected]; nd <[email protected]>; Feifei Wang
<[email protected]>
主题: [PATCH v1 0/2] net/i40e: improve free mbuf

For i40e Tx path, use bulk free of the buffers when mbuf fast free mode is enabled. This can efficiently improve the performance.

Feifei Wang (2):
net/i40e: improve performance for scalar Tx
net/i40e: improve performance for vector Tx

 drivers/net/i40e/i40e_rxtx.c            |  5 ++++-
 drivers/net/i40e/i40e_rxtx_vec_common.h | 11 +++++++++++
 2 files changed, 15 insertions(+), 1 deletion(-)

--
2.25.1

#11 lore raw

v3 cover net/i40e: improve free mbuf for Tx

In reply to Feifei Wang

For i40e Tx path, use bulk free of the buffers when mbuf fast free mode is enabled. This can efficiently improve the performance.

v2:
1. fix bug when tx_rs_thres > RTE_I40E_TX_MAX_FREE_BUF_SZ (Beilei)

v3:
1. change variable name for more readable (Beilei)

Feifei Wang (2):
net/i40e: improve performance for scalar Tx
net/i40e: improve performance for vector Tx

 drivers/net/i40e/i40e_rxtx.c            | 30 ++++++++++++++++++++-----
 drivers/net/i40e/i40e_rxtx_vec_common.h | 11 +++++++++
 2 files changed, 35 insertions(+), 6 deletions(-)
Signature · 1 line
2.25.1
#12 lore raw

PATCH v3 1/2 net/i40e: improve performance for scalar Tx

For i40e scalar Tx path, if implement FAST_FREE_MBUF mode, it means per-queue all mbufs come from the same mempool and have refcnt = 1.

Thus we can use bulk free of the buffers when mbuf fast free mode is enabled.

Following are the test results with this patch:

MRR L3FWD Test:
two ports & bi-directional flows & one core RX API: i40e_recv_pkts_bulk_alloc
TX API: i40e_xmit_pkts_simple ring_descs_size = 1024; Ring_I40E_TX_MAX_FREE_SZ = 64; tx_rs_thresh = I40E_DEFAULT_TX_RSBIT_THRESH = 32; tx_free_thresh = I40E_DEFAULT_TX_FREE_THRESH = 32;

For scalar path in arm platform with default 'tx_rs_thresh':
In n1sdp, performance is improved by 7.9%;
In thunderx2, performance is improved by 7.6%.

For scalar path in x86 platform with default 'tx_rs_thresh':
performance is improved by 4.7%.

---
 drivers/net/i40e/i40e_rxtx.c | 30 ++++++++++++++++++++++++------
 1 file changed, 24 insertions(+), 6 deletions(-)
drivers/net/i40e/i40e_rxtx.c +24 -6
index 6c58decece..0d3482a9d2 100644
@@ -1294,22 +1294,40 @@ static __rte_always_inline int
 i40e_tx_free_bufs(struct i40e_tx_queue *txq)
 {
 	struct i40e_tx_entry *txep;
-	uint16_t i;
+	uint16_t tx_rs_thresh = txq->tx_rs_thresh;
+	uint16_t i = 0, j = 0;
+	struct rte_mbuf *free[RTE_I40E_TX_MAX_FREE_BUF_SZ];
+	const uint16_t k = RTE_ALIGN_FLOOR(tx_rs_thresh, RTE_I40E_TX_MAX_FREE_BUF_SZ);
+	const uint16_t m = tx_rs_thresh % RTE_I40E_TX_MAX_FREE_BUF_SZ;
 
 	if ((txq->tx_ring[txq->tx_next_dd].cmd_type_offset_bsz &
 			rte_cpu_to_le_64(I40E_TXD_QW1_DTYPE_MASK)) !=
 			rte_cpu_to_le_64(I40E_TX_DESC_DTYPE_DESC_DONE))
 		return 0;
 
-	txep = &(txq->sw_ring[txq->tx_next_dd - (txq->tx_rs_thresh - 1)]);
+	txep = &txq->sw_ring[txq->tx_next_dd - (tx_rs_thresh - 1)];
 
-	for (i = 0; i < txq->tx_rs_thresh; i++)
+	for (i = 0; i < tx_rs_thresh; i++)
 		rte_prefetch0((txep + i)->mbuf);
 
 	if (txq->offloads & DEV_TX_OFFLOAD_MBUF_FAST_FREE) {
-		for (i = 0; i < txq->tx_rs_thresh; ++i, ++txep) {
-			rte_mempool_put(txep->mbuf->pool, txep->mbuf);
-			txep->mbuf = NULL;
+		if (k) {
+			for (j = 0; j != k; j += RTE_I40E_TX_MAX_FREE_BUF_SZ) {
+				for (i = 0; i < RTE_I40E_TX_MAX_FREE_BUF_SZ; ++i, ++txep) {
+					free[i] = txep->mbuf;
+					txep->mbuf = NULL;
+				}
+				rte_mempool_put_bulk(free[0]->pool, (void **)free,
+						RTE_I40E_TX_MAX_FREE_BUF_SZ);
+			}
+		}
+
+		if (m) {
+			for (i = 0; i < m; ++i, ++txep) {
+				free[i] = txep->mbuf;
+				txep->mbuf = NULL;
+			}
+			rte_mempool_put_bulk(free[0]->pool, (void **)free, m);
 		}
 	} else {
 		for (i = 0; i < txq->tx_rs_thresh; ++i, ++txep) {
Signature · 1 line
2.25.1
Acked-by on v3 1/2
#13 lore raw

Re: [dpdk-dev] [PATCH v3 1/2] net/i40e: improve performance for scalar Tx

Quoted 97 lines

-----Original Message-----
From: Feifei Wang <[email protected]>
Sent: Wednesday, June 30, 2021 2:41 PM
To: Xing, Beilei <[email protected]>
Cc: [email protected]; [email protected]; Feifei Wang <[email protected]>; Ruifeng Wang <[email protected]>
Subject: [PATCH v3 1/2] net/i40e: improve performance for scalar Tx

For i40e scalar Tx path, if implement FAST_FREE_MBUF mode, it means per- queue all mbufs come from the same mempool and have refcnt = 1.

Thus we can use bulk free of the buffers when mbuf fast free mode is enabled.

Following are the test results with this patch:

MRR L3FWD Test:
two ports & bi-directional flows & one core RX API:
i40e_recv_pkts_bulk_alloc TX API: i40e_xmit_pkts_simple ring_descs_size = 1024; Ring_I40E_TX_MAX_FREE_SZ = 64; tx_rs_thresh = I40E_DEFAULT_TX_RSBIT_THRESH = 32; tx_free_thresh = I40E_DEFAULT_TX_FREE_THRESH = 32;

For scalar path in arm platform with default 'tx_rs_thresh':
In n1sdp, performance is improved by 7.9%; In thunderx2, performance is improved by 7.6%.

For scalar path in x86 platform with default 'tx_rs_thresh':
performance is improved by 4.7%.

---
 drivers/net/i40e/i40e_rxtx.c | 30 ++++++++++++++++++++++++------
 1 file changed, 24 insertions(+), 6 deletions(-)
drivers/net/i40e/i40e_rxtx.c index renamed from drivers/net/i40e/i40e_rxtx.c +0 -0

6c58decece..0d3482a9d2 100644

drivers/net/i40e/i40e_rxtx.c +16 -6
@@ -1294,22 +1294,40 @@ static __rte_always_inline int
 i40e_tx_free_bufs(struct i40e_tx_queue *txq)  {
 	struct i40e_tx_entry *txep;
-	uint16_t i;
+	uint16_t tx_rs_thresh = txq->tx_rs_thresh;
+	uint16_t i = 0, j = 0;
+	struct rte_mbuf *free[RTE_I40E_TX_MAX_FREE_BUF_SZ];
+	const uint16_t k = RTE_ALIGN_FLOOR(tx_rs_thresh,
 RTE_I40E_TX_MAX_FREE_BUF_SZ);
+	const uint16_t m = tx_rs_thresh % RTE_I40E_TX_MAX_FREE_BUF_SZ;
 
 	if ((txq->tx_ring[txq->tx_next_dd].cmd_type_offset_bsz &
 			rte_cpu_to_le_64(I40E_TXD_QW1_DTYPE_MASK)) !=
 
 	rte_cpu_to_le_64(I40E_TX_DESC_DTYPE_DESC_DONE))
 		return 0;
 
-	txep = &(txq->sw_ring[txq->tx_next_dd - (txq->tx_rs_thresh - 1)]);
+	txep = &txq->sw_ring[txq->tx_next_dd - (tx_rs_thresh - 1)];
 
-	for (i = 0; i < txq->tx_rs_thresh; i++)
+	for (i = 0; i < tx_rs_thresh; i++)
 		rte_prefetch0((txep + i)->mbuf);
 
 	if (txq->offloads & DEV_TX_OFFLOAD_MBUF_FAST_FREE) {
-		for (i = 0; i < txq->tx_rs_thresh; ++i, ++txep) {
-			rte_mempool_put(txep->mbuf->pool, txep->mbuf);
-			txep->mbuf = NULL;
+		if (k) {
+			for (j = 0; j != k; j += RTE_I40E_TX_MAX_FREE_BUF_SZ)
 {
+				for (i = 0; i < RTE_I40E_TX_MAX_FREE_BUF_SZ;
++i, ++txep) {
+					free[i] = txep->mbuf;
+					txep->mbuf = NULL;
+				}
+				rte_mempool_put_bulk(free[0]->pool, (void
 **)free,
+
	RTE_I40E_TX_MAX_FREE_BUF_SZ);
drivers/net/i40e/i40e_rxtx.c +9 -0
+			}
+		}
+
+		if (m) {
+			for (i = 0; i < m; ++i, ++txep) {
+				free[i] = txep->mbuf;
+				txep->mbuf = NULL;
+			}
+			rte_mempool_put_bulk(free[0]->pool, (void **)free,

m);

drivers/net/i40e/i40e_rxtx.c +0 -1
 		}
 	} else {
 		for (i = 0; i < txq->tx_rs_thresh; ++i, ++txep) {
--
Quoted 1 line

2.25.1

#14 lore raw

PATCH v3 2/2 net/i40e: improve performance for vector Tx

In reply to Feifei Wang

For i40e vector Tx path, if tx_offload is set as FAST_FREE_MBUF mode, no mbuf fast free operations are executed. To fix this, add mbuf fast free mode for vector Tx path.

Furthermore, for i40e vector Tx path, if implement FAST_FREE_MBUF mode, it means per-queue all mbufs come from the same mempool and have refcnt = 1. Thus we can use bulk free of the buffers when mbuf fast free mode is enabled.

For vector path in arm platform:
In n1sdp, performance is improved by 18.4%;
In thunderx2, performance is improved by 23%.

For vector path in x86 platform:
No performance changes.

---
 drivers/net/i40e/i40e_rxtx_vec_common.h | 11 +++++++++++
 1 file changed, 11 insertions(+)
drivers/net/i40e/i40e_rxtx_vec_common.h +11 -0
index 16fcf0aec6..f52ed98d62 100644
@@ -99,6 +99,16 @@ i40e_tx_free_bufs(struct i40e_tx_queue *txq)
 	  * tx_next_dd - (tx_rs_thresh-1)
 	  */
 	txep = &txq->sw_ring[txq->tx_next_dd - (n - 1)];
+
+	if (txq->offloads & DEV_TX_OFFLOAD_MBUF_FAST_FREE) {
+		for (i = 0; i < n; i++) {
+			free[i] = txep[i].mbuf;
+			txep[i].mbuf = NULL;
+		}
+		rte_mempool_put_bulk(free[0]->pool, (void **)free, n);
+		goto done;
+	}
+
 	m = rte_pktmbuf_prefree_seg(txep[0].mbuf);
 	if (likely(m != NULL)) {
 		free[0] = m;
@@ -126,6 +136,7 @@ i40e_tx_free_bufs(struct i40e_tx_queue *txq)
 		}
 	}
 
+done:
 	/* buffers were freed, update counters */
 	txq->nb_tx_free = (uint16_t)(txq->nb_tx_free + txq->tx_rs_thresh);
 	txq->tx_next_dd = (uint16_t)(txq->tx_next_dd + txq->tx_rs_thresh);
Signature · 1 line
2.25.1
Applied
#15 lore raw

Re: [dpdk-dev] [PATCH v3 0/2] net/i40e: improve free mbuf for Tx

In reply to Feifei Wang
Quoted 21 lines

-----Original Message-----
From: dev <[email protected]> On Behalf Of Feifei Wang
Sent: Wednesday, June 30, 2021 2:41 PM
Cc: [email protected]; [email protected]; Feifei Wang <[email protected]>
Subject: [dpdk-dev] [PATCH v3 0/2] net/i40e: improve free mbuf for Tx

For i40e Tx path, use bulk free of the buffers when mbuf fast free mode is enabled. This can efficiently improve the performance.

v2:
1. fix bug when tx_rs_thres > RTE_I40E_TX_MAX_FREE_BUF_SZ (Beilei)

v3:
1. change variable name for more readable (Beilei)

Feifei Wang (2):
net/i40e: improve performance for scalar Tx
net/i40e: improve performance for vector Tx

Quoted 6 lines
 drivers/net/i40e/i40e_rxtx.c            | 30 ++++++++++++++++++++-----
 drivers/net/i40e/i40e_rxtx_vec_common.h | 11 +++++++++
 2 files changed, 35 insertions(+), 6 deletions(-)

--
2.25.1

Applied to dpdk-next-net-intel.

Thanks
Qi