net/i40e: improve free mbuf
cover net/i40e: improve free mbuf
For i40e Tx path, use bulk free of the buffers when mbuf fast free mode is enabled. This can efficiently improve the performance.
Feifei Wang (2):
net/i40e: improve performance for scalar Tx
net/i40e: improve performance for vector Tx
drivers/net/i40e/i40e_rxtx.c | 5 ++++- drivers/net/i40e/i40e_rxtx_vec_common.h | 11 +++++++++++ 2 files changed, 15 insertions(+), 1 deletion(-)
Signature · 1 line
2.25.1
PATCH 1/2 net/i40e: improve performance for scalar Tx
For i40e scalar Tx path, if implement FAST_FREE_MBUF mode, it means per-queue all mbufs come from the same mempool and have refcnt = 1.
Thus we can use bulk free of the buffers when mbuf fast free mode is enabled.
For scalar path in arm platform:
In n1sdp, performance is improved by 7.8%;
In thunderx2, performance is improved by 6.7%.
For scalar path in x86 platform,
performance is improved by 6%.
--- drivers/net/i40e/i40e_rxtx.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
drivers/net/i40e/i40e_rxtx.c +4 -1
@@ -1295,6 +1295,7 @@ i40e_tx_free_bufs(struct i40e_tx_queue *txq) {struct i40e_tx_entry *txep;uint16_t i; +struct rte_mbuf *free[RTE_I40E_TX_MAX_FREE_BUF_SZ ]; if ((txq->tx_ring[txq->tx_next_dd].cmd_type_offset_bsz & rte_cpu_to_le_64(I40E_TXD_QW1_DTYPE_MASK)) !=
@@ -1308,9 +1309,11 @@ i40e_tx_free_bufs(struct i40e_tx_queue *txq)if (txq->offloads &DEV_TX_OFFLOAD_MBUF_FAST_FREE ) {for (i =0 ; i < txq->tx_rs_thresh; ++i, ++txep) { -rte_mempool_put (txep->mbuf->pool, txep->mbuf); + free[i] = txep->mbuf; txep->mbuf =NULL ; } +rte_mempool_put_bulk (free[0 ]->pool, (void **)free, + txq->tx_rs_thresh); }else {for (i =0 ; i < txq->tx_rs_thresh; ++i, ++txep) {rte_pktmbuf_free_seg (txep->mbuf);
Signature · 1 line
2.25.1
Re: [dpdk-dev] [PATCH v1 1/2] net/i40e: improve performance for scalar Tx
Quoted 41 lines
-----Original Message-----
From: Feifei Wang <[email protected]>
Sent: Thursday, May 27, 2021 4:17 PM
To: Xing, Beilei <[email protected]>
Cc: [email protected]; [email protected]; Feifei Wang <[email protected]>; Ruifeng Wang <[email protected]>
Subject: [PATCH v1 1/2] net/i40e: improve performance for scalar TxFor i40e scalar Tx path, if implement
FAST_FREE_MBUFmode, it means per- queue all mbufs come from the same mempool and have refcnt = 1.Thus we can use bulk free of the buffers when mbuf fast free mode is enabled.
For scalar path in arm platform:
In n1sdp, performance is improved by 7.8%; In thunderx2, performance is improved by 6.7%.For scalar path in x86 platform,
performance is improved by 6%.Suggested-by Ruifeng Wang <[email protected]>Signed-off-by Feifei Wang <[email protected]>--- drivers/net/i40e/i40e_rxtx.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)drivers/net/i40e/i40e_rxtx.c index renamed from drivers/net/i40e/i40e_rxtx.c +0
-0
6c58decece..fe7b20f750100644drivers/net/i40e/i40e_rxtx.c +1
-0@@ -1295,6 +1295,7 @@ i40e_tx_free_bufs(struct i40e_tx_queue *txq) {struct i40e_tx_entry *txep;uint16_t i; +struct rte_mbuf *free[RTE_I40E_TX_MAX_FREE_BUF_SZ ]; if ((txq->tx_ring[txq->tx_next_dd].cmd_type_offset_bsz & rte_cpu_to_le_64(I40E_TXD_QW1_DTYPE_MASK)) !=
drivers/net/i40e/i40e_rxtx.c i40e_tx_free_bufs(struct i40e_tx_queue *txq) · line 1312
Quoted 5 lines
drivers/net/i40e/i40e_rxtx.c +1
-1@@ -1308,9 +1309,11 @@ i40e_tx_free_bufs(struct i40e_tx_queue *txq)if (txq->offloads &DEV_TX_OFFLOAD_MBUF_FAST_FREE ) {for (i =0 ; i < txq->tx_rs_thresh; ++i, ++txep) { -rte_mempool_put (txep->mbuf->pool, txep->mbuf); + free[i] = txep->mbuf;
The tx_rs_thresh can be 'nb_desc - 3', so if tx_rs_thres > RTE_I40E_TX_MAX_FREE_BUF_SZ, there'll be out of bounds, right?
Quoted 9 lines
drivers/net/i40e/i40e_rxtx.c +2
-1txep->mbuf =NULL ; } +rte_mempool_put_bulk (free[0 ]->pool, (void **)free, + txq->tx_rs_thresh); }else {for (i =0 ; i < txq->tx_rs_thresh; ++i, ++txep) {rte_pktmbuf_free_seg (txep->mbuf); --2.25.1
[dpdk-dev] 回复: [PATCH v1 1/2] net/i40e: improve performance for scalar Tx
Hi, Beilei
Thanks for your comments, please see below.
Quoted 53 lines
-----邮件原件-----
发件人: Xing, Beilei <[email protected]>
发送时间: 2021年6月22日 14:08
收件人: Feifei Wang <[email protected]>
抄送: [email protected]; nd <[email protected]>; Ruifeng Wang <[email protected]>
主题: RE: [PATCH v1 1/2] net/i40e: improve performance for scalar TxQuoted 33 lines
-----Original Message-----
From: Feifei Wang <[email protected]>
Sent: Thursday, May 27, 2021 4:17 PM
To: Xing, Beilei <[email protected]>
Cc: [email protected]; [email protected]; Feifei Wang <[email protected]>; Ruifeng Wang <[email protected]>
Subject: [PATCH v1 1/2] net/i40e: improve performance for scalar TxFor i40e scalar Tx path, if implement
FAST_FREE_MBUFmode, it means per- queue all mbufs come from the same mempool and have refcnt = 1.Thus we can use bulk free of the buffers when mbuf fast free mode is enabled.
For scalar path in arm platform:
In n1sdp, performance is improved by 7.8%; In thunderx2, performance is improved by 6.7%.For scalar path in x86 platform,
performance is improved by 6%.Suggested-by Ruifeng Wang <[email protected]>Signed-off-by Feifei Wang <[email protected]>--- drivers/net/i40e/i40e_rxtx.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)(quoted diff) +0
-0
b/drivers/net/i40e/i40e_rxtx.cindex6c58decece..fe7b20f750100644drivers/net/i40e/i40e_rxtx.c i40e_tx_free_bufs(struct i40e_tx_queue *txq) { · line 1300
Quoted 6 lines
drivers/net/i40e/i40e_rxtx.c +1
-0@@ -1295,6 +1295,7 @@ i40e_tx_free_bufs(struct i40e_tx_queue *txq) {struct i40e_tx_entry *txep;uint16_t i; +struct rte_mbuf *free[RTE_I40E_TX_MAX_FREE_BUF_SZ ]; if ((txq->tx_ring[txq->tx_next_dd].cmd_type_offset_bsz &rte_cpu_to_le_64(I40E_TXD_QW1_DTYPE_MASK)) != @@ -1308,9+1309,11
Quoted 2 lines
drivers/net/i40e/i40e_rxtx.c +0
-0@@ i40e_tx_free_bufs(struct i40e_tx_queue *txq)
Quoted 7 lines
drivers/net/i40e/i40e_rxtx.c i40e_tx_free_bufs(struct i40e_tx_queue *txq) · line 1312
Quoted 4 lines
drivers/net/i40e/i40e_rxtx.c +1
-1if (txq->offloads &DEV_TX_OFFLOAD_MBUF_FAST_FREE ) {for (i =0 ; i < txq->tx_rs_thresh; ++i, ++txep) { -rte_mempool_put (txep->mbuf->pool, txep->mbuf); + free[i] = txep->mbuf;The
tx_rs_threshcan be 'nb_desc- 3', so iftx_rs_thres>RTE_I40E_TX_MAX_FREE_BUF_SZ, there'll be out of bounds, right?
Actually tx_rs_thresh <= tx__free_thresh < nb_desc - 3 (i40e_dev_tx_queue_setup). However, I don't know how it affects the relationship between tx_rs_thresh and RTE_I40E_TX_MAX_FREE_BUF_SZ.
Furthermore, I think you are right that tx_rs_thres can be greater than RTE_I40E_TX_MAX_FREE_BUF_SZ in tx_simple_mode (i40e_set_tx_function_flag).
Thus, in scalar path, we can change like:
---------------------------------------------------------------------------------------------------------------
int n = txq->tx_rs_thresh;int32_t i = 0, j = 0;
const int32_t k = RTE_ALIGN_FLOOR(n, RTE_I40E_TX_MAX_FREE_BUF_SZ); const int32_t m = n % RTE_I40E_TX_MAX_FREE_BUF_SZ; struct rte_mbuf *free[RTE_I40E_TX_MAX_FREE_BUF_SZ];
For FAST_FREE_MODE:
if (k) {
for (j = 0; j != k - RTE_I40E_TX_MAX_FREE_BUF_SZ;
j += RTE_I40E_TX_MAX_FREE_BUF_SZ) {
for (i = 0; i <RTE_I40E_TX_MAX_FREE_BUF_SZ; ++i, ++txep) {
free[i] = txep->mbuf;
txep->mbuf = NULL;
}
rte_mempool_put_bulk(free[0]->pool, (void **)free,
RTE_I40E_TX_MAX_FREE_BUF_SZ);
}
} else {
for (i = 0; i < m; ++i, ++txep) {
free[i] = txep->mbuf;
txep->mbuf = NULL;
}
rte_mempool_put_bulk(free[0]->pool, (void **)free, m);
}---------------------------------------------------------------------------------------------------------------
Best Regards
Feifei
[dpdk-dev] 回复: [PATCH v1 1/2] net/i40e: improve performance for scalar Tx
Sorry for a mistake for the code, it should be:
------------------------------------------------------------------------------------------------
int n = txq->tx_rs_thresh;
int32_t i = 0, j = 0;
const int32_t k = RTE_ALIGN_FLOOR(n, RTE_I40E_TX_MAX_FREE_BUF_SZ); const int32_t m = n % RTE_I40E_TX_MAX_FREE_BUF_SZ; struct rte_mbuf *free[RTE_I40E_TX_MAX_FREE_BUF_SZ];
For FAST_FREE_MODE:
if (k) {
for (j = 0; j != k - RTE_I40E_TX_MAX_FREE_BUF_SZ;
j += RTE_I40E_TX_MAX_FREE_BUF_SZ) {
for (i = 0; i <RTE_I40E_TX_MAX_FREE_BUF_SZ; ++i, ++txep) {
free[i] = txep->mbuf;
txep->mbuf = NULL;
} rte_mempool_put_bulk(free[0]->pool, (void **)free,
RTE_I40E_TX_MAX_FREE_BUF_SZ);
}
}
if (m) {
for (i = 0; i < m; ++i, ++txep) {
free[i] = txep->mbuf;
txep->mbuf = NULL;
} }
rte_mempool_put_bulk(free[0]->pool, (void **)free, m); }
------------------------------------------------------------------------------------------------
Re: [dpdk-dev] [PATCH v1 1/2] net/i40e: improve performance for scalar Tx
Quoted 32 lines
-----Original Message-----
From: Feifei Wang <[email protected]>
Sent: Tuesday, June 22, 2021 6:08 PM
To: Xing, Beilei <[email protected]>
Cc: [email protected]; nd <[email protected]>; Ruifeng Wang <[email protected]>; nd <[email protected]>
Subject: 回复: [PATCH v1 1/2] net/i40e: improve performance for scalar TxSorry for a mistake for the code, it should be:
------------------------------------------------------------------------------------------------
int n =txq->tx_rs_thresh;
int32_ti = 0, j = 0;
constint32_tk =RTE_ALIGN_FLOOR(n, RTE_I40E_TX_MAX_FREE_BUF_SZ); constint32_tm = n %RTE_I40E_TX_MAX_FREE_BUF_SZ; structrte_mbuf*free[RTE_I40E_TX_MAX_FREE_BUF_SZ];For
FAST_FREE_MODE:if (k) {
(quoted diff) +0
-0for (j = 0; j != k - RTE_I40E_TX_MAX_FREE_BUF_SZ; j += RTE_I40E_TX_MAX_FREE_BUF_SZ) {for (i = 0; i <RTE_I40E_TX_MAX_FREE_BUF_SZ; ++i, ++txep) { free[i] = txep->mbuf; txep->mbuf = NULL; }(quoted diff) +0
-0rte_mempool_put_bulk(free[0]->pool, (void **)free, RTE_I40E_TX_MAX_FREE_BUF_SZ); } }
Quoted 8 lines
if (m) {
for (i = 0; i < m; ++i, ++txep) { free[i] = txep->mbuf; txep->mbuf = NULL; }}
rte_mempool_put_bulk(free[0]->pool, (void **)free, m); }
------------------------------------------------------------------------------------------------
Seems no logical problem, but the code looks heavy due to for loops.
Did you run performance with this change when tx_rs_thresh > RTE_I40E_TX_MAX_FREE_BUF_SZ?
[dpdk-dev] 回复: [PATCH v1 1/2] net/i40e: improve performance for scalar Tx
<snip>
Quoted 28 lines
Quoted 20 lines
int n =
txq->tx_rs_thresh;
int32_ti = 0, j = 0;
constint32_tk =RTE_ALIGN_FLOOR(n, RTE_I40E_TX_MAX_FREE_BUF_SZ); constint32_tm = n %RTE_I40E_TX_MAX_FREE_BUF_SZ; structrte_mbuf*free[RTE_I40E_TX_MAX_FREE_BUF_SZ];For
FAST_FREE_MODE:if (k) {
(quoted diff) +0
-0for (j = 0; j != k - RTE_I40E_TX_MAX_FREE_BUF_SZ; j += RTE_I40E_TX_MAX_FREE_BUF_SZ) {for (i = 0; i <RTE_I40E_TX_MAX_FREE_BUF_SZ; ++i, ++txep) { free[i] = txep->mbuf; txep->mbuf = NULL; }(quoted diff) +0
-0rte_mempool_put_bulk(free[0]->pool, (void **)free, RTE_I40E_TX_MAX_FREE_BUF_SZ); } }Quoted 7 lines
if (m) {
for (i = 0; i < m; ++i, ++txep) { free[i] = txep->mbuf; txep->mbuf = NULL; }}
rte_mempool_put_bulk(free[0]->pool, (void **)free, m); }
Quoted 3 lines
Seems no logical problem, but the code looks heavy due to for loops. Did you run performance with this change when
tx_rs_thresh>RTE_I40E_TX_MAX_FREE_BUF_SZ?
Sorry for my late rely. It takes me some time to do the test for this path and following is my test results:
First, I come up with another way to solve this bug and compare it with "loop"(size of 'free' is 64). That is set the size of 'free' as a large constant. We know:tx_rs_thresh < ring_desc_size < I40E_MAX_RING_DESC(4096), so we can directly define as:
struct rte_mbuf *free[RTE_I40E_TX_MAX_FREE_BUF_SZ];
[1]Test Config:
MRR Test: two porst & bi-directional flows & one core RX API: i40e_recv_pkts_bulk_alloc
TX API: i40e_xmit_pkts_simplering_descs_size: 1024Ring_I40E_TX_MAX_FREE_SZ: 64
[2]Scheme:tx_rs_thresh = I40E_DEFAULT_TX_RSBIT_THRESHtx_free_thresh = I40E_DEFAULT_TX_FREE_THRESHtx_rs_thresh <= tx_free_thresh < nb_tx_desc
So we change the value of 'tx_rs_thresh' by adjust I40E_DEFAULT_TX_RSBIT_THRESH
[3]Test Results (performance improve):
In X86:
tx_rs_thresh/ tx_free_thresh 32/32 256/256 512/512
1.mempool_put(base) 0 0 0
2.mempool_put_bulk:loop +4.7% +5.6% +7.0%
3.mempool_put_bulk:large size for free +3.8% +2.3% -2.0%
(free[I40E_MAX_RING_DESC])
In Arm:
N1SDP:
tx_rs_thresh/ tx_free_thresh 32/32 256/256 512/512
1.mempool_put(base) 0 0 0
2.mempool_put_bulk:loop +7.9% +9.1% +2.9%
3.mempool_put_bulk:large size for free +7.1% +8.7% +3.4%
(free[I40E_MAX_RING_DESC])
Thunderx2:
tx_rs_thresh/ tx_free_thresh 32/32 256/256 512/512
1.mempool_put(base) 0 0 0
2.mempool_put_bulk:loop +7.6% +10.5% +7.6%
3.mempool_put_bulk:large size for free +1.7% +18.4% +10.2%
(free[I40E_MAX_RING_DESC])
As a result, I feel maybe 'loop' is better and it seems not very heavy according to the test. What about your views and look forward to your reply.
Thanks a lot.
Re: [dpdk-dev] [PATCH v1 1/2] net/i40e: improve performance for scalar Tx
Quoted 89 lines
-----Original Message-----
From: Feifei Wang <[email protected]>
Sent: Friday, June 25, 2021 5:40 PM
To: Xing, Beilei <[email protected]>
Cc: [email protected]; nd <[email protected]>; Ruifeng Wang <[email protected]>; nd <[email protected]>; nd <[email protected]>
Subject: 回复: [PATCH v1 1/2] net/i40e: improve performance for scalar Tx<snip>
Quoted 28 lines
Quoted 20 lines
int n =
txq->tx_rs_thresh;
int32_ti = 0, j = 0;
constint32_tk =RTE_ALIGN_FLOOR(n, RTE_I40E_TX_MAX_FREE_BUF_SZ); constint32_tm = n %RTE_I40E_TX_MAX_FREE_BUF_SZ; structrte_mbuf*free[RTE_I40E_TX_MAX_FREE_BUF_SZ];For
FAST_FREE_MODE:if (k) {
(quoted diff) +0
-0for (j = 0; j != k - RTE_I40E_TX_MAX_FREE_BUF_SZ; j += RTE_I40E_TX_MAX_FREE_BUF_SZ) {for (i = 0; i <RTE_I40E_TX_MAX_FREE_BUF_SZ; ++i, ++txep) { free[i] = txep->mbuf; txep->mbuf = NULL; }(quoted diff) +0
-0rte_mempool_put_bulk(free[0]->pool, (void **)free, RTE_I40E_TX_MAX_FREE_BUF_SZ); } }Quoted 7 lines
if (m) {
for (i = 0; i < m; ++i, ++txep) { free[i] = txep->mbuf; txep->mbuf = NULL; }}
rte_mempool_put_bulk(free[0]->pool, (void **)free, m); }Quoted 3 lines
Seems no logical problem, but the code looks heavy due to for loops. Did you run performance with this change when
tx_rs_thresh>RTE_I40E_TX_MAX_FREE_BUF_SZ?Sorry for my late rely. It takes me some time to do the test for this path and following is my test results:
First, I come up with another way to solve this bug and compare it with "loop"(size of 'free' is 64).
That is set the size of 'free' as a large constant. We know:tx_rs_thresh<ring_desc_size<I40E_MAX_RING_DESC(4096), so we can directly define as:
structrte_mbuf*free[RTE_I40E_TX_MAX_FREE_BUF_SZ];[1]Test Config:
MRR Test: two porst & bi-directional flows & one core RX API:i40e_recv_pkts_bulk_allocTX API:i40e_xmit_pkts_simplering_descs_size: 1024Ring_I40E_TX_MAX_FREE_SZ: 64[2]Scheme:
tx_rs_thresh=I40E_DEFAULT_TX_RSBIT_THRESHtx_free_thresh=I40E_DEFAULT_TX_FREE_THRESHtx_rs_thresh<=tx_free_thresh<nb_tx_descSo we change the value of 'tx_rs_thresh' by adjustI40E_DEFAULT_TX_RSBIT_THRESH[3]Test Results (performance improve):
In X86:
tx_rs_thresh/tx_free_thresh32/32 256/256 512/512
1.mempool_put(base) 0 0 0
2.mempool_put_bulk:loop +4.7% +5.6% +7.0%
3.mempool_put_bulk:large size for free +3.8% +2.3% -2.0%
(free[I40E_MAX_RING_DESC])In Arm:
N1SDP:
tx_rs_thresh/tx_free_thresh32/32 256/256 512/512
1.mempool_put(base) 0 0 0
2.mempool_put_bulk:loop +7.9% +9.1% +2.9%
3.mempool_put_bulk:large size for free +7.1% +8.7% +3.4%
(free[I40E_MAX_RING_DESC])Thunderx2:
tx_rs_thresh/tx_free_thresh32/32 256/256 512/512
1.mempool_put(base) 0 0 0
2.mempool_put_bulk:loop +7.6% +10.5% +7.6%
3.mempool_put_bulk:large size for free +1.7% +18.4% +10.2%
(free[I40E_MAX_RING_DESC])
Quoted 4 lines
As a result, I feel maybe 'loop' is better and it seems not very heavy according to the test.
What about your views and look forward to your reply. Thanks a lot.
Thanks for your patch and test.
It looks OK for me, please send V2.
[dpdk-dev] 回复: [PATCH v1 1/2] net/i40e: improve performance for scalar Tx
Quoted 101 lines
-----邮件原件-----
发件人: Xing, Beilei <[email protected]>
发送时间: 2021年6月28日 10:27
收件人: Feifei Wang <[email protected]>
抄送: [email protected]; nd <[email protected]>; Ruifeng Wang <[email protected]>; nd <[email protected]>; nd <[email protected]> 主题: RE: [PATCH v1 1/2] net/i40e: improve performance for scalar TxQuoted 9 lines
-----Original Message-----
From: Feifei Wang <[email protected]>
Sent: Friday, June 25, 2021 5:40 PM
To: Xing, Beilei <[email protected]>
Cc: [email protected]; nd <[email protected]>; Ruifeng Wang <[email protected]>; nd <[email protected]>; nd <[email protected]>
Subject: 回复: [PATCH v1 1/2] net/i40e: improve performance for scalar TxQuoted 5 lines
<snip>
Quoted 3 lines
Quoted 3 lines
int n =
txq->tx_rs_thresh;
int32_ti = 0, j = 0;
constint32_tk = RTE_ALIGN_FLOOR(n,
RTE_I40E_TX_MAX_FREE_BUF_SZ);Quoted 1 line
Quoted 1 line
Quoted 1 line
const
int32_tm = n %RTE_I40E_TX_MAX_FREE_BUF_SZ; struct
rte_mbufQuoted 73 lines
Quoted 24 lines
Quoted 16 lines
*free[RTE_I40E_TX_MAX_FREE_BUF_SZ];
For
FAST_FREE_MODE:if (k) {
(quoted diff) +0
-0for (j = 0; j != k - RTE_I40E_TX_MAX_FREE_BUF_SZ; j += RTE_I40E_TX_MAX_FREE_BUF_SZ) {for (i = 0; i <RTE_I40E_TX_MAX_FREE_BUF_SZ; ++i, ++txep) { free[i] = txep->mbuf; txep->mbuf = NULL; }(quoted diff) +0
-0rte_mempool_put_bulk(free[0]->pool, (void **)free, RTE_I40E_TX_MAX_FREE_BUF_SZ); } }Quoted 7 lines
if (m) {
for (i = 0; i < m; ++i, ++txep) { free[i] = txep->mbuf; txep->mbuf = NULL; }}
rte_mempool_put_bulk(free[0]->pool, (void **)free, m); }Quoted 3 lines
Seems no logical problem, but the code looks heavy due to for loops. Did you run performance with this change when
tx_rs_thresh>RTE_I40E_TX_MAX_FREE_BUF_SZ?Sorry for my late rely. It takes me some time to do the test for this path and following is my test results:
First, I come up with another way to solve this bug and compare it with "loop"(size of 'free' is 64).
That is set the size of 'free' as a large constant. We know:tx_rs_thresh<ring_desc_size<I40E_MAX_RING_DESC(4096), so we can directly define as:
structrte_mbuf*free[RTE_I40E_TX_MAX_FREE_BUF_SZ];[1]Test Config:
MRR Test: two porst & bi-directional flows & one core RX API:i40e_recv_pkts_bulk_allocTX API:i40e_xmit_pkts_simplering_descs_size: 1024Ring_I40E_TX_MAX_FREE_SZ: 64[2]Scheme:
tx_rs_thresh=I40E_DEFAULT_TX_RSBIT_THRESHtx_free_thresh=I40E_DEFAULT_TX_FREE_THRESHtx_rs_thresh<=tx_free_thresh<nb_tx_descSo we change the value of 'tx_rs_thresh' by adjustI40E_DEFAULT_TX_RSBIT_THRESH[3]Test Results (performance improve):
In X86:
tx_rs_thresh/tx_free_thresh32/32 256/256 512/512
1.mempool_put(base) 0 0 0
2.mempool_put_bulk:loop +4.7% +5.6% +7.0%
3.mempool_put_bulk:large size for free +3.8% +2.3% -2.0%
(free[I40E_MAX_RING_DESC])In Arm:
N1SDP:
tx_rs_thresh/tx_free_thresh32/32 256/256 512/512
1.mempool_put(base) 0 0 0
2.mempool_put_bulk:loop +7.9% +9.1% +2.9%
3.mempool_put_bulk:large size for free +7.1% +8.7% +3.4%
(free[I40E_MAX_RING_DESC])Thunderx2:
tx_rs_thresh/tx_free_thresh32/32 256/256 512/512
1.mempool_put(base) 0 0 0
2.mempool_put_bulk:loop +7.6% +10.5% +7.6%
3.mempool_put_bulk:large size for free +1.7% +18.4% +10.2%
(free[I40E_MAX_RING_DESC])
Quoted 7 lines
Quoted 4 lines
As a result, I feel maybe 'loop' is better and it seems not very heavy according to the test.
What about your views and look forward to your reply. Thanks a lot.Thanks for your patch and test.
It looks OK for me, please send V2.
Thanks for the reviewing, I will update the V2 version.
PATCH 2/2 net/i40e: improve performance for vector Tx
In reply to Feifei WangFor i40e vector Tx path, if tx_offload is set as FAST_FREE_MBUF mode, no mbuf fast free operations are executed. To fix this, add mbuf fast free mode for vector Tx path.
Furthermore, for i40e vector Tx path, if implement FAST_FREE_MBUF mode, it means per-queue all mbufs come from the same mempool and have refcnt = 1. Thus we can use bulk free of the buffers when mbuf fast free mode is enabled.
For vector path in arm platform:
In n1sdp, performance is improved by 18.4%;
In thunderx2, performance is improved by 23%.
For vector path in x86 platform:
No performance changes.
--- drivers/net/i40e/i40e_rxtx_vec_common.h | 11 +++++++++++ 1 file changed, 11 insertions(+)
drivers/net/i40e/i40e_rxtx_vec_common.h +11 -0
@@ -99,6 +99,16 @@ i40e_tx_free_bufs(struct i40e_tx_queue *txq)* tx_next_dd - (tx_rs_thresh-1) */ txep = &txq->sw_ring[txq->tx_next_dd - (n -1 )]; + +if (txq->offloads &DEV_TX_OFFLOAD_MBUF_FAST_FREE ) { +for (i =0 ; i < n; i++) { + free[i] = txep[i].mbuf; + txep[i].mbuf =NULL ; + } +rte_mempool_put_bulk (free[0 ]->pool, (void **)free, n); + goto done; + } + m =rte_pktmbuf_prefree_seg (txep[0 ].mbuf);if (likely (m !=NULL )) { free[0 ] = m;
@@ -126,6 +136,7 @@ i40e_tx_free_bufs(struct i40e_tx_queue *txq) } } +done:/* buffers were freed, update counters */ txq->nb_tx_free = (uint16_t )(txq->nb_tx_free + txq->tx_rs_thresh); txq->tx_next_dd = (uint16_t )(txq->tx_next_dd + txq->tx_rs_thresh);
Signature · 1 line
2.25.1
Hi, Qi
Can you help review these patches?
Thanks very much.
Best Regards
Feifei
Quoted 20 lines
-----邮件原件-----
发件人: Feifei Wang <[email protected]>
发送时间: 2021年5月27日 16:17
抄送: [email protected]; nd <[email protected]>; Feifei Wang
<[email protected]>
主题: [PATCH v1 0/2] net/i40e: improve free mbufFor i40e Tx path, use bulk free of the buffers when mbuf fast free mode is enabled. This can efficiently improve the performance.
Feifei Wang (2):
net/i40e: improve performance for scalar Tx
net/i40e: improve performance for vector Txdrivers/net/i40e/i40e_rxtx.c | 5 ++++- drivers/net/i40e/i40e_rxtx_vec_common.h | 11 +++++++++++ 2 files changed, 15 insertions(+), 1 deletion(-)--
2.25.1
v3 cover net/i40e: improve free mbuf for Tx
In reply to Feifei WangFor i40e Tx path, use bulk free of the buffers when mbuf fast free mode is enabled. This can efficiently improve the performance.
v2:
1. fix bug when tx_rs_thres > RTE_I40E_TX_MAX_FREE_BUF_SZ (Beilei)
v3:
1. change variable name for more readable (Beilei)
Feifei Wang (2):
net/i40e: improve performance for scalar Tx
net/i40e: improve performance for vector Tx
drivers/net/i40e/i40e_rxtx.c | 30 ++++++++++++++++++++----- drivers/net/i40e/i40e_rxtx_vec_common.h | 11 +++++++++ 2 files changed, 35 insertions(+), 6 deletions(-)
Signature · 1 line
2.25.1
PATCH v3 1/2 net/i40e: improve performance for scalar Tx
For i40e scalar Tx path, if implement FAST_FREE_MBUF mode, it means per-queue all mbufs come from the same mempool and have refcnt = 1.
Thus we can use bulk free of the buffers when mbuf fast free mode is enabled.
Following are the test results with this patch:
MRR L3FWD Test:
two ports & bi-directional flows & one core RX API: i40e_recv_pkts_bulk_alloc
TX API: i40e_xmit_pkts_simple ring_descs_size = 1024; Ring_I40E_TX_MAX_FREE_SZ = 64; tx_rs_thresh = I40E_DEFAULT_TX_RSBIT_THRESH = 32; tx_free_thresh = I40E_DEFAULT_TX_FREE_THRESH = 32;
For scalar path in arm platform with default 'tx_rs_thresh':
In n1sdp, performance is improved by 7.9%;
In thunderx2, performance is improved by 7.6%.
For scalar path in x86 platform with default 'tx_rs_thresh':
performance is improved by 4.7%.
--- drivers/net/i40e/i40e_rxtx.c | 30 ++++++++++++++++++++++++------ 1 file changed, 24 insertions(+), 6 deletions(-)
drivers/net/i40e/i40e_rxtx.c +24 -6
@@ -1294,22 +1294,40 @@ static __rte_always_inline int i40e_tx_free_bufs(struct i40e_tx_queue *txq) {struct i40e_tx_entry *txep; -uint16_t i; +uint16_t tx_rs_thresh = txq->tx_rs_thresh; +uint16_t i =0 , j =0 ; +struct rte_mbuf *free[RTE_I40E_TX_MAX_FREE_BUF_SZ ]; +const uint16_t k =RTE_ALIGN_FLOOR (tx_rs_thresh,RTE_I40E_TX_MAX_FREE_BUF_SZ ); +const uint16_t m = tx_rs_thresh %RTE_I40E_TX_MAX_FREE_BUF_SZ ;if ((txq->tx_ring[txq->tx_next_dd].cmd_type_offset_bsz &rte_cpu_to_le_64 (I40E_TXD_QW1_DTYPE_MASK )) !=rte_cpu_to_le_64 (I40E_TX_DESC_DTYPE_DESC_DONE ))return 0 ; - txep = &(txq->sw_ring[txq->tx_next_dd - (txq->tx_rs_thresh -1 )]); + txep = &txq->sw_ring[txq->tx_next_dd - (tx_rs_thresh -1 )]; -for (i =0 ; i < txq->tx_rs_thresh; i++) +for (i =0 ; i < tx_rs_thresh; i++)rte_prefetch0 ((txep + i)->mbuf);if (txq->offloads &DEV_TX_OFFLOAD_MBUF_FAST_FREE ) { -for (i =0 ; i < txq->tx_rs_thresh; ++i, ++txep) { -rte_mempool_put (txep->mbuf->pool, txep->mbuf); - txep->mbuf =NULL ; +if (k) { +for (j =0 ; j != k; j +=RTE_I40E_TX_MAX_FREE_BUF_SZ ) { +for (i =0 ; i <RTE_I40E_TX_MAX_FREE_BUF_SZ ; ++i, ++txep) { + free[i] = txep->mbuf; + txep->mbuf =NULL ; + } +rte_mempool_put_bulk (free[0 ]->pool, (void **)free, +RTE_I40E_TX_MAX_FREE_BUF_SZ ); + } + } + +if (m) { +for (i =0 ; i < m; ++i, ++txep) { + free[i] = txep->mbuf; + txep->mbuf =NULL ; + } +rte_mempool_put_bulk (free[0 ]->pool, (void **)free, m); } }else {for (i =0 ; i < txq->tx_rs_thresh; ++i, ++txep) {
Signature · 1 line
2.25.1
Re: [dpdk-dev] [PATCH v3 1/2] net/i40e: improve performance for scalar Tx
Quoted 97 lines
-----Original Message-----
From: Feifei Wang <[email protected]>
Sent: Wednesday, June 30, 2021 2:41 PM
To: Xing, Beilei <[email protected]>
Cc: [email protected]; [email protected]; Feifei Wang <[email protected]>; Ruifeng Wang <[email protected]>
Subject: [PATCH v3 1/2] net/i40e: improve performance for scalar TxFor i40e scalar Tx path, if implement
FAST_FREE_MBUFmode, it means per- queue all mbufs come from the same mempool and have refcnt = 1.Thus we can use bulk free of the buffers when mbuf fast free mode is enabled.
Following are the test results with this patch:
MRR L3FWD Test:
two ports & bi-directional flows & one core RX API:i40e_recv_pkts_bulk_allocTX API:i40e_xmit_pkts_simplering_descs_size= 1024;Ring_I40E_TX_MAX_FREE_SZ= 64;tx_rs_thresh=I40E_DEFAULT_TX_RSBIT_THRESH= 32;tx_free_thresh=I40E_DEFAULT_TX_FREE_THRESH= 32;For scalar path in arm platform with default '
tx_rs_thresh':
In n1sdp, performance is improved by 7.9%; In thunderx2, performance is improved by 7.6%.For scalar path in x86 platform with default '
tx_rs_thresh':
performance is improved by 4.7%.Suggested-by Ruifeng Wang <[email protected]>Signed-off-by Feifei Wang <[email protected]>Reviewed-by Ruifeng Wang <[email protected]>--- drivers/net/i40e/i40e_rxtx.c | 30 ++++++++++++++++++++++++------ 1 file changed, 24 insertions(+), 6 deletions(-)drivers/net/i40e/i40e_rxtx.c index renamed from drivers/net/i40e/i40e_rxtx.c +0
-0
6c58decece..0d3482a9d2100644drivers/net/i40e/i40e_rxtx.c +16
-6@@ -1294,22 +1294,40 @@ static __rte_always_inline int i40e_tx_free_bufs(struct i40e_tx_queue *txq) {struct i40e_tx_entry *txep; -uint16_t i; +uint16_t tx_rs_thresh = txq->tx_rs_thresh; +uint16_t i =0 , j =0 ; +struct rte_mbuf *free[RTE_I40E_TX_MAX_FREE_BUF_SZ ]; +const uint16_t k =RTE_ALIGN_FLOOR (tx_rs_thresh,RTE_I40E_TX_MAX_FREE_BUF_SZ ); +const uint16_t m = tx_rs_thresh %RTE_I40E_TX_MAX_FREE_BUF_SZ ;if ((txq->tx_ring[txq->tx_next_dd].cmd_type_offset_bsz &rte_cpu_to_le_64 (I40E_TXD_QW1_DTYPE_MASK )) !=rte_cpu_to_le_64 (I40E_TX_DESC_DTYPE_DESC_DONE ))return 0 ; - txep = &(txq->sw_ring[txq->tx_next_dd - (txq->tx_rs_thresh -1 )]); + txep = &txq->sw_ring[txq->tx_next_dd - (tx_rs_thresh -1 )]; -for (i =0 ; i < txq->tx_rs_thresh; i++) +for (i =0 ; i < tx_rs_thresh; i++)rte_prefetch0 ((txep + i)->mbuf);if (txq->offloads &DEV_TX_OFFLOAD_MBUF_FAST_FREE ) { -for (i =0 ; i < txq->tx_rs_thresh; ++i, ++txep) { -rte_mempool_put (txep->mbuf->pool, txep->mbuf); - txep->mbuf =NULL ; +if (k) { +for (j =0 ; j != k; j +=RTE_I40E_TX_MAX_FREE_BUF_SZ ) { +for (i =0 ; i <RTE_I40E_TX_MAX_FREE_BUF_SZ ; ++i, ++txep) { + free[i] = txep->mbuf; + txep->mbuf =NULL ; + } + rte_mempool_put_bulk(free[0]->pool, (void **)free, +RTE_I40E_TX_MAX_FREE_BUF_SZ);drivers/net/i40e/i40e_rxtx.c +9
-0+ } + } + +if (m) { +for (i =0 ; i < m; ++i, ++txep) { + free[i] = txep->mbuf; + txep->mbuf =NULL ; + } + rte_mempool_put_bulk(free[0]->pool, (void **)free,m);
drivers/net/i40e/i40e_rxtx.c +0
-1} }else {for (i =0 ; i < txq->tx_rs_thresh; ++i, ++txep) { --
Quoted 1 line
2.25.1
PATCH v3 2/2 net/i40e: improve performance for vector Tx
In reply to Feifei WangFor i40e vector Tx path, if tx_offload is set as FAST_FREE_MBUF mode, no mbuf fast free operations are executed. To fix this, add mbuf fast free mode for vector Tx path.
Furthermore, for i40e vector Tx path, if implement FAST_FREE_MBUF mode, it means per-queue all mbufs come from the same mempool and have refcnt = 1. Thus we can use bulk free of the buffers when mbuf fast free mode is enabled.
For vector path in arm platform:
In n1sdp, performance is improved by 18.4%;
In thunderx2, performance is improved by 23%.
For vector path in x86 platform:
No performance changes.
--- drivers/net/i40e/i40e_rxtx_vec_common.h | 11 +++++++++++ 1 file changed, 11 insertions(+)
drivers/net/i40e/i40e_rxtx_vec_common.h +11 -0
@@ -99,6 +99,16 @@ i40e_tx_free_bufs(struct i40e_tx_queue *txq)* tx_next_dd - (tx_rs_thresh-1) */ txep = &txq->sw_ring[txq->tx_next_dd - (n -1 )]; + +if (txq->offloads &DEV_TX_OFFLOAD_MBUF_FAST_FREE ) { +for (i =0 ; i < n; i++) { + free[i] = txep[i].mbuf; + txep[i].mbuf =NULL ; + } +rte_mempool_put_bulk (free[0 ]->pool, (void **)free, n); + goto done; + } + m =rte_pktmbuf_prefree_seg (txep[0 ].mbuf);if (likely (m !=NULL )) { free[0 ] = m;
@@ -126,6 +136,7 @@ i40e_tx_free_bufs(struct i40e_tx_queue *txq) } } +done:/* buffers were freed, update counters */ txq->nb_tx_free = (uint16_t )(txq->nb_tx_free + txq->tx_rs_thresh); txq->tx_next_dd = (uint16_t )(txq->tx_next_dd + txq->tx_rs_thresh);
Signature · 1 line
2.25.1
Zhang, Qi Z
AppliedRe: [dpdk-dev] [PATCH v3 0/2] net/i40e: improve free mbuf for Tx
In reply to Feifei WangQuoted 21 lines
-----Original Message-----
From: dev <[email protected]> On Behalf Of Feifei Wang
Sent: Wednesday, June 30, 2021 2:41 PM
Cc: [email protected]; [email protected]; Feifei Wang <[email protected]>
Subject: [dpdk-dev] [PATCH v3 0/2] net/i40e: improve free mbuf for TxFor i40e Tx path, use bulk free of the buffers when mbuf fast free mode is enabled. This can efficiently improve the performance.
v2:
1. fix bug whentx_rs_thres>RTE_I40E_TX_MAX_FREE_BUF_SZ(Beilei)v3:
1. change variable name for more readable (Beilei)Feifei Wang (2):
net/i40e: improve performance for scalar Tx
net/i40e: improve performance for vector Tx
Quoted 6 lines
drivers/net/i40e/i40e_rxtx.c | 30 ++++++++++++++++++++----- drivers/net/i40e/i40e_rxtx_vec_common.h | 11 +++++++++ 2 files changed, 35 insertions(+), 6 deletions(-)--
2.25.1
Applied to dpdk-next-net-intel.
Thanks
Qi