Consolidate ice and iavf vector Tx paths
Subsystems (2)
cover Consolidate ice and iavf vector Tx paths
This patchset consolidates the ice and iavf vector Tx paths, moving the code to net/intel/common. The iavf code is used as the basis for the move, because it has a superset of the features. When the common code is adopted by the ice driver, it then gains features such as QinQ support and tunneling support.
Depends-on: series-39162 ("VLAN and QinQ fixes for iavf")
Bruce Richardson (13):
net/iavf: remove unnecessary alignment calls net/intel: make Tx context flag common
net/iavf: use separate params for VLAN and QinQ position net/iavf: deduplicate tunnel field fill functions net/intel: define common macros for tunneling bit-shifts net/intel: move iavf descriptor writing functions to common net/intel: use function callback for lldp net/ice: use common descriptor creation functions net/intel: move vector Tx paths to common net/ice: use common AVX Tx functions
net/intel: add common vector Tx fns with context handling net/intel: improve Tx path selection logic net/ice: enable context desc offloads for vector Tx
doc/guides/rel_notes/release_26_11.rst | 6 + drivers/net/intel/common/tx.h | 77 +- drivers/net/intel/common/tx_vec_x86.h | 783 ++++++++++++++++++ drivers/net/intel/cpfl/cpfl_rxtx.c | 3 +- drivers/net/intel/i40e/i40e_rxtx.c | 2 +- drivers/net/intel/iavf/iavf_rxtx.c | 76 +- drivers/net/intel/iavf/iavf_rxtx.h | 34 - drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c | 496 +---------- drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c | 495 +---------- drivers/net/intel/iavf/iavf_rxtx_vec_common.h | 76 +- drivers/net/intel/ice/ice_ethdev.h | 3 + drivers/net/intel/ice/ice_rxtx.c | 34 +- drivers/net/intel/ice/ice_rxtx.h | 11 + drivers/net/intel/ice/ice_rxtx_vec_avx2.c | 159 +--- drivers/net/intel/ice/ice_rxtx_vec_avx512.c | 154 +--- drivers/net/intel/ice/ice_rxtx_vec_common.h | 59 -- drivers/net/intel/idpf/idpf_rxtx.c | 3 +- 17 files changed, 1025 insertions(+), 1446 deletions(-) create mode 100644 drivers/net/intel/common/tx_vec_x86.h
--
2.53.0
PATCH 1/13 net/iavf: remove unnecessary alignment calls
When we have a context descriptor on Tx of a packet from the vector paths, that means that we always have 32-bytes being written per descriptor/per packet, so the stores are always 32-bit aligned. This means we can use aligned stores for each single descriptor store, and that we never need to do an initial descriptor write for alignment when doing a burst of descriptors.
Signed-off-by Bruce Richardson <[email protected]>
--- drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c | 9 ++------- drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c | 9 ++------- 2 files changed, 4 insertions(+), 14 deletions(-)
drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c +2 -7
@@ -1933,7 +1933,8 @@ ctx_vtx1(volatile struct ci_tx_desc *txdp, struct rte_mbuf *pkt,__m256i ctx_data_desc =_mm256_set_epi64x (high_data_qw, pkt->buf_iova + pkt->data_off, high_ctx_qw, low_ctx_qw); - _mm256_storeu_si256(RTE_CAST_PTR(__m256i *, txdp), ctx_data_desc); +/* tx_id is always even in ctx mode, so txdp is always 32-byte aligned */ + _mm256_store_si256(RTE_CAST_PTR(__m256i *, txdp), ctx_data_desc); }static __rte_always_inline void
@@ -1944,12 +1945,6 @@ ctx_vtx(volatile struct ci_tx_desc *txdp,uint64_t hi_data_qw_tmpl = (IAVF_TX_DESC_DTYPE_DATA | ((uint64_t )flags <<IAVF_TXD_QW1_CMD_SHIFT )); -/* if unaligned on 32-bit boundary, do one to align */ -if (((uintptr_t )txdp &0x1F ) !=0 && nb_pkts !=0 ) { -ctx_vtx1 (txdp, *pkt, flags, offload, vlan_flag, ptype_lldp_enabled); - nb_pkts--; txdp++; pkt++; - } -for (; nb_pkts >1 ; txdp +=4 , pkt +=2 , nb_pkts -=2 ) {uint64_t hi_ctx_qw1 =IAVF_TX_DESC_DTYPE_CONTEXT ;uint64_t hi_ctx_qw0 =IAVF_TX_DESC_DTYPE_CONTEXT ;
drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c +2 -7
@@ -2078,7 +2078,8 @@ ctx_vtx1(volatile struct ci_tx_desc *txdp, struct rte_mbuf *pkt,__m256i ctx_data_desc =_mm256_set_epi64x (high_data_qw, pkt->buf_iova + pkt->data_off, high_ctx_qw, low_ctx_qw); - _mm256_storeu_si256(RTE_CAST_PTR(__m256i *, txdp), ctx_data_desc); +/* tx_id is always even in ctx mode, so txdp is always 32-byte aligned */ + _mm256_store_si256(RTE_CAST_PTR(__m256i *, txdp), ctx_data_desc); }static __rte_always_inline void
@@ -2088,12 +2089,6 @@ ctx_vtx(volatile struct ci_tx_desc *txdp, {uint64_t hi_data_qw_tmpl = (CI_TX_DESC_DTYPE_DATA | (flags <<CI_TXD_QW1_CMD_S )); -/* if unaligned on 32-bit boundary, do one to align */ -if (((uintptr_t )txdp &0x1F ) !=0 && nb_pkts !=0 ) { -ctx_vtx1 (txdp, *pkt, flags, offload, vlan_flag, lldp_enabled); - nb_pkts--; txdp++; pkt++; - } -for (; nb_pkts >1 ; txdp +=4 , pkt +=2 , nb_pkts -=2 ) {uint64_t hi_ctx_qw1 =IAVF_TX_DESC_DTYPE_CONTEXT ;uint64_t hi_ctx_qw0 =IAVF_TX_DESC_DTYPE_CONTEXT ;
Signature · 1 line
2.53.0
RE: [PATCH 01/13] net/iavf: remove unnecessary alignment calls
Quoted 10 lines
Subject: [PATCH 01/13] net/iavf: remove unnecessary alignment calls
When we have a context descriptor on Tx of a packet from the vector paths, that means that we always have 32-bytes being written per descriptor/per packet, so the stores are always 32-bit aligned. This means we can use aligned stores for each single descriptor store, and that we never need to do an initial descriptor write for alignment when doing a burst of descriptors.
Signed-off-by Bruce Richardson <[email protected]>
Acked-by Ciara Loftus <[email protected]>
Quoted 5 lines
--- drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c | 9 ++------- drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c | 9 ++------- 2 files changed, 4 insertions(+), 14 deletions(-)
PATCH 2/13 net/intel: make Tx context flag common
In reply to Bruce RichardsonThe flag for enabling context descriptors in the vector paths should not be limited to just the iavf driver, as other drivers may use context paths in future also. Therefore move the flag out of the iavf-specific part of the Tx queue structure, to put with other standard queue flags.
Signed-off-by Bruce Richardson <[email protected]>
--- drivers/net/intel/common/tx.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/net/intel/common/tx.h +1 -1
@@ -173,6 +173,7 @@ struct ci_tx_queue {bool tx_deferred_start;/* don't start this queue in dev start */ bool q_set;/* indicate if tx queue has been configured */ bool use_vec_entry;/* use sw_ring_vec (true for vector and simple paths) */ +bool use_ctx;/* with ctx info, each pkt needs two desc in vec paths */ union {/* the VSI this queue belongs to */ struct i40e_vsi *i40e_vsi;struct iavf_vsi *iavf_vsi;
@@ -194,7 +195,6 @@ struct ci_tx_queue {#define IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG2 BIT(1)uint8_t vlan_flag;uint8_t tc; -bool use_ctx;/* with ctx info, each pkt needs two descriptors */ bool lldp_enabled; };struct {/* ixgbe specific values */
Signature · 1 line
2.53.0
RE: [PATCH 02/13] net/intel: make Tx context flag common
Quoted 8 lines
Subject: [PATCH 02/13] net/intel: make Tx context flag common
The flag for enabling context descriptors in the vector paths should not be limited to just the iavf driver, as other drivers may use context paths in future also. Therefore move the flag out of the iavf-specific part of the Tx queue structure, to put with other standard queue flags.
Signed-off-by Bruce Richardson <[email protected]>
Acked-by Ciara Loftus <[email protected]>
Quoted 27 lines
--- drivers/net/intel/common/tx.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/net/intel/common/tx.h +0
-0@@ -173,6 +173,7 @@ struct ci_tx_queue {bool tx_deferred_start;/* don't start this queue in dev start */ bool q_set;/* indicate if tx queue has been configured */ bool use_vec_entry;/* use sw_ring_vec (true for vector and simple paths) */
drivers/net/intel/common/tx.h +1
-0+bool use_ctx;/* with ctx info, each pkt needs two desc in vec paths */
drivers/net/intel/common/tx.h +0
-1union {/* the VSI this queue belongs to */ struct i40e_vsi *i40e_vsi;struct iavf_vsi *iavf_vsi;@@ -194,7 +195,6 @@ struct ci_tx_queue {#define IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG2 BIT(1)uint8_t vlan_flag;uint8_t tc; -bool use_ctx;/* with ctx info, each pkt needs two descriptors */
drivers/net/intel/common/tx.h +0
-0bool lldp_enabled; };struct {/* ixgbe specific values */
PATCH 3/13 net/iavf: use separate params for VLAN and QinQ position
In reply to Bruce RichardsonLike is the case with the scalar path, use separate offload parameters for the vector path to distinguish between the position of the single VLAN tag and the QinQ outer tag. With iavf these are currently always the same, but other drivers - with whom we may want to share code - have different positions for these.
Signed-off-by Bruce Richardson <[email protected]>
--- drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c | 64 +++++++++++-------- drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c | 62 ++++++++++-------- drivers/net/intel/iavf/iavf_rxtx_vec_common.h | 13 ++-- 3 files changed, 79 insertions(+), 60 deletions(-)
drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c +37 -27
@@ -1619,13 +1619,14 @@ iavf_recv_scattered_pkts_vec_avx2_flex_rxd_offload(void *rx_queue, static __rte_always_inline voidiavf_vtx1 (volatile struct ci_tx_desc *txdp, -struct rte_mbuf *pkt,uint64_t flags,bool offload,uint8_t vlan_flag) +struct rte_mbuf *pkt,uint64_t flags,bool offload, +enum ci_l2tag_pos single_vlan_pos,enum ci_l2tag_pos qinq_outer_pos) {uint64_t high_qw = (CI_TX_DESC_DTYPE_DATA | ((uint64_t )flags <<CI_TXD_QW1_CMD_S ) | ((uint64_t )pkt->data_len <<CI_TXD_QW1_TX_BUF_SZ_S ));if (offload) -iavf_txd_enable_offload (pkt, &high_qw, vlan_flag); +iavf_txd_enable_offload (pkt, &high_qw, single_vlan_pos, qinq_outer_pos);__m128i descriptor =_mm_set_epi64x (high_qw, pkt->buf_iova + pkt->data_off);
@@ -1634,13 +1635,14 @@ iavf_vtx1(volatile struct ci_tx_desc *txdp, static __rte_always_inline voidiavf_vtx (volatile struct ci_tx_desc *txdp, -struct rte_mbuf **pkt,uint16_t nb_pkts,uint64_t flags,bool offload,uint8_t vlan_flag) +struct rte_mbuf **pkt,uint16_t nb_pkts,uint64_t flags,bool offload, +enum ci_l2tag_pos single_vlan_pos,enum ci_l2tag_pos qinq_outer_pos) {const uint64_t hi_qw_tmpl = (CI_TX_DESC_DTYPE_DATA | (flags <<CI_TXD_QW1_CMD_S ));/* if unaligned on 32-bit boundary, do one to align */ if (((uintptr_t )txdp &0x1F ) !=0 && nb_pkts !=0 ) { -iavf_vtx1 (txdp, *pkt, flags, offload, vlan_flag); +iavf_vtx1 (txdp, *pkt, flags, offload, single_vlan_pos, qinq_outer_pos); nb_pkts--; txdp++; pkt++; }
@@ -1649,19 +1651,19 @@ iavf_vtx(volatile struct ci_tx_desc *txdp,uint64_t hi_qw3 = hi_qw_tmpl | ((uint64_t )pkt[3 ]->data_len <<CI_TXD_QW1_TX_BUF_SZ_S );if (offload) -iavf_txd_enable_offload (pkt[3 ], &hi_qw3, vlan_flag); +iavf_txd_enable_offload (pkt[3 ], &hi_qw3, single_vlan_pos, qinq_outer_pos);uint64_t hi_qw2 = hi_qw_tmpl | ((uint64_t )pkt[2 ]->data_len <<CI_TXD_QW1_TX_BUF_SZ_S );if (offload) -iavf_txd_enable_offload (pkt[2 ], &hi_qw2, vlan_flag); +iavf_txd_enable_offload (pkt[2 ], &hi_qw2, single_vlan_pos, qinq_outer_pos);uint64_t hi_qw1 = hi_qw_tmpl | ((uint64_t )pkt[1 ]->data_len <<CI_TXD_QW1_TX_BUF_SZ_S );if (offload) -iavf_txd_enable_offload (pkt[1 ], &hi_qw1, vlan_flag); +iavf_txd_enable_offload (pkt[1 ], &hi_qw1, single_vlan_pos, qinq_outer_pos);uint64_t hi_qw0 = hi_qw_tmpl | ((uint64_t )pkt[0 ]->data_len <<CI_TXD_QW1_TX_BUF_SZ_S );if (offload) -iavf_txd_enable_offload (pkt[0 ], &hi_qw0, vlan_flag); +iavf_txd_enable_offload (pkt[0 ], &hi_qw0, single_vlan_pos, qinq_outer_pos);__m256i desc2_3 = _mm256_set_epi64x
@@ -1681,7 +1683,7 @@ iavf_vtx(volatile struct ci_tx_desc *txdp,/* do any last ones */ while (nb_pkts) { -iavf_vtx1 (txdp, *pkt, flags, offload, vlan_flag); +iavf_vtx1 (txdp, *pkt, flags, offload, single_vlan_pos, qinq_outer_pos); txdp++; pkt++; nb_pkts--; } }
@@ -1697,6 +1699,9 @@ iavf_xmit_fixed_burst_vec_avx2(void *tx_queue, struct rte_mbuf **tx_pkts,/* bit2 is reserved and must be set to 1 according to Spec */ uint64_t flags =CI_TX_DESC_CMD_EOP |CI_TX_DESC_CMD_ICRC ;uint64_t rs =CI_TX_DESC_CMD_RS | flags; +/* vlan_flag gives both the single-VLAN and the QinQ outer tag position */ +enum ci_l2tag_pos vlan_pos = (txq->vlan_flag &IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG1 ) ? +CI_TAG_IN_DATA_DESC :CI_TAG_IN_CTX_DESC ;if (txq->nb_tx_free < txq->tx_free_thresh)ci_tx_free_bufs_vec (txq, iavf_tx_desc_done, false);
@@ -1716,11 +1721,11 @@ iavf_xmit_fixed_burst_vec_avx2(void *tx_queue, struct rte_mbuf **tx_pkts,if (nb_commit >= n) {ci_tx_backlog_entry_vec (txep, tx_pkts, n); -iavf_vtx (txdp, tx_pkts, n -1 , flags, offload, txq->vlan_flag); +iavf_vtx (txdp, tx_pkts, n -1 , flags, offload, vlan_pos, vlan_pos); tx_pkts += (n -1 ); txdp += (n -1 ); -iavf_vtx1 (txdp, *tx_pkts++, rs, offload, txq->vlan_flag); +iavf_vtx1 (txdp, *tx_pkts++, rs, offload, vlan_pos, vlan_pos); nb_commit = (uint16_t )(nb_commit - n);
@@ -1734,7 +1739,7 @@ iavf_xmit_fixed_burst_vec_avx2(void *tx_queue, struct rte_mbuf **tx_pkts,ci_tx_backlog_entry_vec (txep, tx_pkts, nb_commit); -iavf_vtx (txdp, tx_pkts, nb_commit, flags, offload, txq->vlan_flag); +iavf_vtx (txdp, tx_pkts, nb_commit, flags, offload, vlan_pos, vlan_pos); tx_id = (uint16_t )(tx_id + nb_commit);if (tx_id > txq->tx_next_rs) {
@@ -1903,7 +1908,8 @@ iavf_fill_ctx_desc_tunneling_field(volatile uint64_t *qw0, static __rte_always_inline voidctx_vtx1 (volatile struct ci_tx_desc *txdp,struct rte_mbuf *pkt, -uint64_t flags,bool offload,uint8_t vlan_flag,bool ptype_lldp_enabled) +uint64_t flags,bool offload,enum ci_l2tag_pos single_vlan_pos, +enum ci_l2tag_pos qinq_outer_pos,bool ptype_lldp_enabled) {uint64_t high_ctx_qw =IAVF_TX_DESC_DTYPE_CONTEXT ;uint64_t low_ctx_qw =0 ;
@@ -1911,13 +1917,13 @@ ctx_vtx1(volatile struct ci_tx_desc *txdp, struct rte_mbuf *pkt,if (offload) {iavf_fill_ctx_desc_tunneling_avx2 (&low_ctx_qw, pkt);if (pkt->ol_flags &RTE_MBUF_F_TX_QINQ ) { -uint64_t qinq_tag = vlan_flag &IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG2 ? +uint64_t qinq_tag = qinq_outer_pos ==CI_TAG_IN_CTX_DESC ? (uint64_t )pkt->vlan_tci_outer : (uint64_t )pkt->vlan_tci; high_ctx_qw |=IAVF_TX_CTX_DESC_IL2TAG2 <<IAVF_TXD_CTX_QW1_CMD_SHIFT ; low_ctx_qw |= qinq_tag <<IAVF_TXD_CTX_QW0_L2TAG2_PARAM ; }else if ((pkt->ol_flags &RTE_MBUF_F_TX_VLAN ) && - vlan_flag &IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG2 ) { + single_vlan_pos ==CI_TAG_IN_CTX_DESC ) { high_ctx_qw |=IAVF_TX_CTX_DESC_IL2TAG2 <<IAVF_TXD_CTX_QW1_CMD_SHIFT ; low_ctx_qw |= (uint64_t )pkt->vlan_tci <<IAVF_TXD_CTX_QW0_L2TAG2_PARAM ; }
@@ -1928,7 +1934,7 @@ ctx_vtx1(volatile struct ci_tx_desc *txdp, struct rte_mbuf *pkt, ((uint64_t )flags <<IAVF_TXD_QW1_CMD_SHIFT ) | ((uint64_t)pkt->data_len << IAVF_TXD_QW1_TX_BUF_SZ_SHIFT));if (offload) -iavf_txd_enable_offload (pkt, &high_data_qw, vlan_flag); +iavf_txd_enable_offload (pkt, &high_data_qw, single_vlan_pos, qinq_outer_pos);__m256i ctx_data_desc =_mm256_set_epi64x (high_data_qw, pkt->buf_iova + pkt->data_off, high_ctx_qw, low_ctx_qw);
@@ -1940,7 +1946,8 @@ ctx_vtx1(volatile struct ci_tx_desc *txdp, struct rte_mbuf *pkt, static __rte_always_inline voidctx_vtx (volatile struct ci_tx_desc *txdp,struct rte_mbuf **pkt,uint16_t nb_pkts,uint64_t flags, -bool offload,uint8_t vlan_flag,bool ptype_lldp_enabled) +bool offload,enum ci_l2tag_pos single_vlan_pos,enum ci_l2tag_pos qinq_outer_pos, +bool ptype_lldp_enabled) {uint64_t hi_data_qw_tmpl = (IAVF_TX_DESC_DTYPE_DATA | ((uint64_t )flags <<IAVF_TXD_QW1_CMD_SHIFT ));
@@ -1964,14 +1971,14 @@ ctx_vtx(volatile struct ci_tx_desc *txdp,/* tunnel fill assigns low_ctx_qw1; must run before QinQ/VLAN OR below */ iavf_fill_ctx_desc_tunneling_field (&low_ctx_qw1, pkt[1 ]);if (pkt[1 ]->ol_flags &RTE_MBUF_F_TX_QINQ ) { -uint64_t qinq_tag = vlan_flag &IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG2 ? +uint64_t qinq_tag = qinq_outer_pos ==CI_TAG_IN_CTX_DESC ? (uint64_t )pkt[1 ]->vlan_tci_outer : (uint64_t )pkt[1 ]->vlan_tci; hi_ctx_qw1 |=IAVF_TX_CTX_DESC_IL2TAG2 <<IAVF_TXD_CTX_QW1_CMD_SHIFT ; low_ctx_qw1 |= qinq_tag <<IAVF_TXD_CTX_QW0_L2TAG2_PARAM ; }else if (pkt[1 ]->ol_flags &RTE_MBUF_F_TX_VLAN && - vlan_flag &IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG2 ) { + single_vlan_pos ==CI_TAG_IN_CTX_DESC ) { hi_ctx_qw1 |=IAVF_TX_CTX_DESC_IL2TAG2 <<IAVF_TXD_CTX_QW1_CMD_SHIFT ; low_ctx_qw1 |=
@@ -1985,14 +1992,14 @@ ctx_vtx(volatile struct ci_tx_desc *txdp,/* tunnel fill assigns low_ctx_qw0; must run before QinQ/VLAN OR below */ iavf_fill_ctx_desc_tunneling_field (&low_ctx_qw0, pkt[0 ]);if (pkt[0 ]->ol_flags &RTE_MBUF_F_TX_QINQ ) { -uint64_t qinq_tag = vlan_flag &IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG2 ? +uint64_t qinq_tag = qinq_outer_pos ==CI_TAG_IN_CTX_DESC ? (uint64_t )pkt[0 ]->vlan_tci_outer : (uint64_t )pkt[0 ]->vlan_tci; hi_ctx_qw0 |=IAVF_TX_CTX_DESC_IL2TAG2 <<IAVF_TXD_CTX_QW1_CMD_SHIFT ; low_ctx_qw0 |= qinq_tag <<IAVF_TXD_CTX_QW0_L2TAG2_PARAM ; }else if (pkt[0 ]->ol_flags &RTE_MBUF_F_TX_VLAN && - vlan_flag &IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG2 ) { + single_vlan_pos ==CI_TAG_IN_CTX_DESC ) { hi_ctx_qw0 |=IAVF_TX_CTX_DESC_IL2TAG2 <<IAVF_TXD_CTX_QW1_CMD_SHIFT ; low_ctx_qw0 |=
@@ -2003,8 +2010,8 @@ ctx_vtx(volatile struct ci_tx_desc *txdp, hi_ctx_qw0 |=IAVF_TX_CTX_DESC_SWTCH_UPLINK <<IAVF_TXD_CTX_QW1_CMD_SHIFT ;if (offload) { -iavf_txd_enable_offload (pkt[1 ], &hi_data_qw1, vlan_flag); -iavf_txd_enable_offload (pkt[0 ], &hi_data_qw0, vlan_flag); +iavf_txd_enable_offload (pkt[1 ], &hi_data_qw1, single_vlan_pos, qinq_outer_pos); +iavf_txd_enable_offload (pkt[0 ], &hi_data_qw0, single_vlan_pos, qinq_outer_pos); } __m256i desc2_3 =
@@ -2020,7 +2027,7 @@ ctx_vtx(volatile struct ci_tx_desc *txdp, }if (nb_pkts) -ctx_vtx1 (txdp, *pkt, flags, offload, vlan_flag, ptype_lldp_enabled); +ctx_vtx1 (txdp, *pkt, flags, offload, single_vlan_pos, qinq_outer_pos, ptype_lldp_enabled); }static __rte_always_inline uint16_t
@@ -2035,6 +2042,9 @@ iavf_xmit_fixed_burst_vec_avx2_ctx(void *tx_queue, struct rte_mbuf **tx_pkts,uint64_t flags =IAVF_TX_DESC_CMD_EOP |IAVF_TX_DESC_CMD_ICRC ;uint64_t rs =IAVF_TX_DESC_CMD_RS | flags;bool lldp_enabled = txq->lldp_enabled; +/* vlan_flag gives both the single-VLAN and the QinQ outer tag position */ +enum ci_l2tag_pos vlan_pos = (txq->vlan_flag &IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG1 ) ? +CI_TAG_IN_DATA_DESC :CI_TAG_IN_CTX_DESC ;if (txq->nb_tx_free < txq->tx_free_thresh)ci_tx_free_bufs_vec (txq, iavf_tx_desc_done, true);
@@ -2057,10 +2067,10 @@ iavf_xmit_fixed_burst_vec_avx2_ctx(void *tx_queue, struct rte_mbuf **tx_pkts, nb_mbuf = n >>1 ;ci_tx_backlog_entry_vec (txep, tx_pkts, nb_mbuf); -ctx_vtx (txdp, tx_pkts, nb_mbuf -1 , flags, offload, txq->vlan_flag, lldp_enabled); +ctx_vtx (txdp, tx_pkts, nb_mbuf -1 , flags, offload, vlan_pos, vlan_pos, lldp_enabled); tx_pkts += (nb_mbuf -1 ); txdp += (n -2 ); -ctx_vtx1 (txdp, *tx_pkts++, rs, offload, txq->vlan_flag, lldp_enabled); +ctx_vtx1 (txdp, *tx_pkts++, rs, offload, vlan_pos, vlan_pos, lldp_enabled); nb_commit = (uint16_t )(nb_commit - n);
@@ -2074,7 +2084,7 @@ iavf_xmit_fixed_burst_vec_avx2_ctx(void *tx_queue, struct rte_mbuf **tx_pkts, nb_mbuf = nb_commit >>1 ;ci_tx_backlog_entry_vec (txep, tx_pkts, nb_mbuf); -ctx_vtx (txdp, tx_pkts, nb_mbuf, flags, offload, txq->vlan_flag, lldp_enabled); +ctx_vtx (txdp, tx_pkts, nb_mbuf, flags, offload, vlan_pos, vlan_pos, lldp_enabled); tx_id = (uint16_t )(tx_id + nb_commit);if (tx_id > txq->tx_next_rs) {
drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c +35 -27
200 lines not on this page · raw message
drivers/net/intel/iavf/iavf_rxtx_vec_common.h +7 -6
@@ -121,14 +121,13 @@ iavf_tx_vec_dev_check_default(struct rte_eth_dev *dev) static __rte_always_inline void iavf_txd_enable_offload(__rte_unused struct rte_mbuf *tx_pkt, -uint64_t *txd_hi,uint8_t vlan_flag) +uint64_t *txd_hi,enum ci_l2tag_pos single_vlan_pos, +enum ci_l2tag_pos qinq_outer_pos) {uint64_t ol_flags = tx_pkt->ol_flags;uint32_t td_cmd =0 ;uint32_t td_offset =0 ; -RTE_SET_USED (vlan_flag); -/* Set MACLEN */ if (ol_flags &RTE_MBUF_F_TX_TUNNEL_MASK ) td_offset |= (tx_pkt->outer_l2_len >>1 )
@@ -179,12 +178,14 @@ iavf_txd_enable_offload(__rte_unused struct rte_mbuf *tx_pkt,if (ol_flags &RTE_MBUF_F_TX_QINQ ) { td_cmd |=IAVF_TX_DESC_CMD_IL2TAG1 ; -/* vlan_flag specifies outer tag location for QinQ. */ -if (vlan_flag &IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG1 ) +/* L2Tag1 always carries a tag for QinQ: the outer tag if that's +* where it is placed, otherwise the inner. +*/ +if (qinq_outer_pos ==CI_TAG_IN_DATA_DESC ) *txd_hi |= ((uint64_t )tx_pkt->vlan_tci_outer <<CI_TXD_QW1_L2TAG1_S );else *txd_hi |= ((uint64_t )tx_pkt->vlan_tci <<CI_TXD_QW1_L2TAG1_S ); - }else if (ol_flags &RTE_MBUF_F_TX_VLAN && vlan_flag &IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG1 ) { + }else if (ol_flags &RTE_MBUF_F_TX_VLAN && single_vlan_pos ==CI_TAG_IN_DATA_DESC ) { td_cmd |=CI_TX_DESC_CMD_IL2TAG1 ; *txd_hi |= ((uint64_t )tx_pkt->vlan_tci <<CI_TXD_QW1_L2TAG1_S ); }
Signature · 1 line
2.53.0
RE: [PATCH 03/13] net/iavf: use separate params for VLAN and QinQ position
Quoted 16 lines
Subject: [PATCH 03/13] net/iavf: use separate params for VLAN and QinQ position
Like is the case with the scalar path, use separate offload parameters for the vector path to distinguish between the position of the single VLAN tag and the QinQ outer tag. With iavf these are currently always the same, but other drivers - with whom we may want to share code - have different positions for these.
Signed-off-by Bruce Richardson <[email protected]>--- drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c | 64 +++++++++++-------- drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c | 62 ++++++++++-------- drivers/net/intel/iavf/iavf_rxtx_vec_common.h | 13 ++-- 3 files changed, 79 insertions(+), 60 deletions(-)
<snip>
Quoted 24 lines
drivers/net/intel/iavf/iavf_rxtx_vec_common.h +2
-3@@ -121,14 +121,13 @@ iavf_tx_vec_dev_check_default(struct rte_eth_dev *dev) static __rte_always_inline void iavf_txd_enable_offload(__rte_unused struct rte_mbuf *tx_pkt, -uint64_t *txd_hi,uint8_t vlan_flag) +uint64_t *txd_hi,enum ci_l2tag_pos single_vlan_pos, +enum ci_l2tag_pos qinq_outer_pos) {uint64_t ol_flags = tx_pkt->ol_flags;uint32_t td_cmd =0 ;uint32_t td_offset =0 ; -RTE_SET_USED (vlan_flag); -/* Set MACLEN */ if (ol_flags &RTE_MBUF_F_TX_TUNNEL_MASK )drivers/net/intel/iavf/iavf_rxtx_vec_common.h +0
-0td_offset |= (tx_pkt->outer_l2_len >>1 )
drivers/net/intel/iavf/iavf_rxtx_vec_common.h iavf_txd_enable_offload(__rte_unused struct · line 184
Quoted 9 lines
drivers/net/intel/iavf/iavf_rxtx_vec_common.h +3
-2@@ -179,12 +178,14 @@ iavf_txd_enable_offload(__rte_unused struct rte_mbuf *tx_pkt, if (ol_flags & RTE_MBUF_F_TX_QINQ) { td_cmd |=IAVF_TX_DESC_CMD_IL2TAG1 ; -/* vlan_flag specifies outer tag location for QinQ. */ -if (vlan_flag &IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG1 ) +/* L2Tag1 always carries a tag for QinQ: the outer tag if that's +* where it is placed, otherwise the inner. +*/
Nit: I feel like the sentence after the colon is unnecessary and states the obvious. Code LGTM.
Acked-by Ciara Loftus <[email protected]>
Quoted 14 lines
drivers/net/intel/iavf/iavf_rxtx_vec_common.h +1
-1+if (qinq_outer_pos ==CI_TAG_IN_DATA_DESC ) *txd_hi |= ((uint64_t )tx_pkt->vlan_tci_outer <<CI_TXD_QW1_L2TAG1_S );else *txd_hi |= ((uint64_t )tx_pkt->vlan_tci <<CI_TXD_QW1_L2TAG1_S ); - } else if (ol_flags & RTE_MBUF_F_TX_VLAN && vlan_flag &
IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG1) {drivers/net/intel/iavf/iavf_rxtx_vec_common.h +1
-0+ } else if (ol_flags & RTE_MBUF_F_TX_VLAN && single_vlan_pos ==
CI_TAG_IN_DATA_DESC) {drivers/net/intel/iavf/iavf_rxtx_vec_common.h +0
-0td_cmd |=CI_TX_DESC_CMD_IL2TAG1 ; *txd_hi |= ((uint64_t)tx_pkt->vlan_tci <<
CI_TXD_QW1_L2TAG1_S);
}
PATCH 4/13 net/iavf: deduplicate tunnel field fill functions
In reply to Bruce RichardsonEach of the avx2 and avx512 Tx functions used two separate subfunctions to insert the tunneling fields into the context descriptor. Except for some minor differences (a flags guard on one) the two were identical, so remove the one missing the additional guard.
Signed-off-by Bruce Richardson <[email protected]>
--- drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c | 77 +------------------ drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c | 77 +------------------ 2 files changed, 4 insertions(+), 150 deletions(-)
drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c +2 -75
@@ -1833,79 +1833,6 @@ iavf_fill_ctx_desc_tunneling_avx2(uint64_t *low_ctx_qw, struct rte_mbuf *pkt) } } -static inline void -iavf_fill_ctx_desc_tunneling_field (volatile uint64_t *qw0, -const struct rte_mbuf *m) -{ -uint64_t eip_typ =IAVF_TX_CTX_DESC_EIPT_NONE ; -uint64_t eip_len =0 ; -uint64_t eip_noinc =0 ; -/* Default - IP_ID is increment in each segment of LSO */ - -switch (m->ol_flags & (RTE_MBUF_F_TX_OUTER_IPV4 | -RTE_MBUF_F_TX_OUTER_IPV6 | -RTE_MBUF_F_TX_OUTER_IP_CKSUM )) { -case RTE_MBUF_F_TX_OUTER_IPV4 : - eip_typ =IAVF_TX_CTX_DESC_EIPT_IPV4_NO_CHECKSUM_OFFLOAD ; - eip_len = m->outer_l3_len >>2 ; -break ; -case RTE_MBUF_F_TX_OUTER_IPV4 |RTE_MBUF_F_TX_OUTER_IP_CKSUM : - eip_typ =IAVF_TX_CTX_DESC_EIPT_IPV4_CHECKSUM_OFFLOAD ; - eip_len = m->outer_l3_len >>2 ; -break ; -case RTE_MBUF_F_TX_OUTER_IPV6 : - eip_typ =IAVF_TX_CTX_DESC_EIPT_IPV6 ; - eip_len = m->outer_l3_len >>2 ; -break ; - } - -/* L4TUNT: L4 Tunneling Type */ -switch (m->ol_flags &RTE_MBUF_F_TX_TUNNEL_MASK ) { -case RTE_MBUF_F_TX_TUNNEL_IPIP : -/* for non UDP / GRE tunneling, set to 00b */ -break ; -case RTE_MBUF_F_TX_TUNNEL_VXLAN : -case RTE_MBUF_F_TX_TUNNEL_VXLAN_GPE : -case RTE_MBUF_F_TX_TUNNEL_GTP : -case RTE_MBUF_F_TX_TUNNEL_GENEVE : - eip_typ |=IAVF_TXD_CTX_UDP_TUNNELING ; -break ; -case RTE_MBUF_F_TX_TUNNEL_GRE : - eip_typ |=IAVF_TXD_CTX_GRE_TUNNELING ; -break ; -default : -PMD_TX_LOG (ERR ,"Tunnel type not supported" ); -return ; - } - -/* L4TUNLEN: L4 Tunneling Length, in Words -* -* We depend on app to set rte_mbuf.l2_len correctly. -* For IP in GRE it should be set to the length of the GRE -* header; -* For MAC in GRE or MAC in UDP it should be set to the length -* of the GRE or UDP headers plus the inner MAC up to including -* its last Ethertype. -* If MPLS labels exists, it should include them as well. -*/ - eip_typ |= (m->l2_len >>1 ) <<IAVF_TXD_CTX_QW0_NATLEN_SHIFT ; - -/** -* Calculate the tunneling UDP checksum. -* Shall be set only if L4TUNT = 01b and EIPT is not zero -*/ -if ((eip_typ & (IAVF_TX_CTX_EXT_IP_IPV6 | -IAVF_TX_CTX_EXT_IP_IPV4 | -IAVF_TX_CTX_EXT_IP_IPV4_NO_CSUM )) && - (eip_typ &IAVF_TXD_CTX_UDP_TUNNELING ) && - (m->ol_flags &RTE_MBUF_F_TX_OUTER_UDP_CKSUM )) - eip_typ |=IAVF_TXD_CTX_QW0_L4T_CS_MASK ; - - *qw0 = eip_typ <<IAVF_TXD_CTX_QW0_TUN_PARAMS_EIPT_SHIFT | - eip_len <<IAVF_TXD_CTX_QW0_TUN_PARAMS_EIPLEN_SHIFT | - eip_noinc <<IAVF_TXD_CTX_QW0_TUN_PARAMS_EIP_NOINC_SHIFT ; -} - static __rte_always_inline void ctx_vtx1(volatile struct ci_tx_desc *txdp, struct rte_mbuf *pkt, uint64_t flags, bool offload, enum ci_l2tag_pos single_vlan_pos,
@@ -1969,7 +1896,7 @@ ctx_vtx(volatile struct ci_tx_desc *txdp,if (offload) {/* tunnel fill assigns low_ctx_qw1; must run before QinQ/VLAN OR below */ -iavf_fill_ctx_desc_tunneling_field (&low_ctx_qw1, pkt[1 ]); +iavf_fill_ctx_desc_tunneling_avx2 (&low_ctx_qw1, pkt[1 ]);if (pkt[1 ]->ol_flags &RTE_MBUF_F_TX_QINQ ) {uint64_t qinq_tag = qinq_outer_pos ==CI_TAG_IN_CTX_DESC ? (uint64_t)pkt[1]->vlan_tci_outer :
@@ -1990,7 +1917,7 @@ ctx_vtx(volatile struct ci_tx_desc *txdp,if (offload) {/* tunnel fill assigns low_ctx_qw0; must run before QinQ/VLAN OR below */ -iavf_fill_ctx_desc_tunneling_field (&low_ctx_qw0, pkt[0 ]); +iavf_fill_ctx_desc_tunneling_avx2 (&low_ctx_qw0, pkt[0 ]);if (pkt[0 ]->ol_flags &RTE_MBUF_F_TX_QINQ ) {uint64_t qinq_tag = qinq_outer_pos ==CI_TAG_IN_CTX_DESC ? (uint64_t)pkt[0]->vlan_tci_outer :
drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c +2 -75
95 lines not on this page · raw message
Signature · 1 line
2.53.0
Re: [PATCH 04/13] net/iavf: deduplicate tunnel field fill functions
Quoted 7 lines · Bruce Richardson
Each of the avx2 and avx512 Tx functions used two separate subfunctions to insert the tunneling fields into the context descriptor. Except for some minor differences (a flags guard on one) the two were identical, so remove the one missing the additional guard.
Signed-off-by Bruce Richardson <[email protected]>
Thanks for the cleanup.
Reviewed-by David Marchand <[email protected]>
Signature · 1 line
David Marchand
PATCH 5/13 net/intel: define common macros for tunneling bit-shifts
In reply to Bruce RichardsonDefine common macros for use in iavf (for now) and other drivers (future) defining the shifts needed for tunnel packet Tx
Signed-off-by Bruce Richardson <[email protected]>
--- drivers/net/intel/common/tx.h | 24 +++++++++++++ drivers/net/intel/iavf/iavf_rxtx.c | 30 ++++++++-------- drivers/net/intel/iavf/iavf_rxtx.h | 34 ------------------- drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c | 30 ++++++++-------- drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c | 30 ++++++++-------- 5 files changed, 69 insertions(+), 79 deletions(-)
drivers/net/intel/common/tx.h +24 -0
@@ -45,6 +45,30 @@#define CI_TX_CTX_DESC_TSYN 0x02#define CI_TX_CTX_DESC_IL2TAG2 0x04 +/* Common TX Context Descriptor QW0 Field Shifts */ +#define CI_TXD_CTX_QW0_L2TAG2_S 32 +#define CI_TXD_CTX_QW0_EIPT_S 0/* 2 BITS - external IP type */ +#define CI_TXD_CTX_QW0_EIPLEN_S 2/* 7 BITS - external IP header length */ +#define CI_TXD_CTX_QW0_NATT_S 9/* 2 BITS - L4 tunnel type */ +#define CI_TXD_CTX_QW0_EIP_NOINC_S 11/* 1 BIT - external IP ID not incremented */ +#define CI_TXD_CTX_QW0_NATLEN_S 12/* 7 BITS - L4 tunnel header length */ +#define CI_TXD_CTX_QW0_L4T_CS_S 23/* 1 BIT - tunnel UDP checksum enable */ + +/** +* Enum for the external (outer) IP type field of a Tx context descriptor's +* tunnel parameters (QW0 bits 0-1). +*/ +enum ci_tx_ctx_eipt { +CI_TX_CTX_EIPT_NONE , +CI_TX_CTX_EIPT_IPV6 , +CI_TX_CTX_EIPT_IPV4_NO_CSUM , +CI_TX_CTX_EIPT_IPV4 , +}; + +#define CI_TXD_CTX_UDP_TUNNELING (0x1ULL << CI_TXD_CTX_QW0_NATT_S) +#define CI_TXD_CTX_GRE_TUNNELING (0x2ULL << CI_TXD_CTX_QW0_NATT_S) +#define CI_TXD_CTX_QW0_L4T_CS_M (0x1ULL << CI_TXD_CTX_QW0_L4T_CS_S) +/** * Enum to specify where a VLAN tag is to be placed for packet Tx. */
drivers/net/intel/iavf/iavf_rxtx.c +15 -15
@@ -2345,7 +2345,7 @@ iavf_calc_context_desc(const struct rte_mbuf *mb, uint8_t vlan_flag, bool lldp_estatic inline void iavf_fill_ctx_desc_tunnelling_field (uint64_t *qw0,uint64_t ol_flags,const struct rte_mbuf *m) { -uint64_t eip_typ =IAVF_TX_CTX_DESC_EIPT_NONE ; +uint64_t eip_typ =CI_TX_CTX_EIPT_NONE ;uint64_t eip_len =0 ;uint64_t eip_noinc =0 ;/* Default - IP_ID is increment in each segment of LSO */
@@ -2354,15 +2354,15 @@ iavf_fill_ctx_desc_tunnelling_field(uint64_t *qw0, uint64_t ol_flags, const stru RTE_MBUF_F_TX_OUTER_IPV6 | RTE_MBUF_F_TX_OUTER_IP_CKSUM)) {case RTE_MBUF_F_TX_OUTER_IPV4 : - eip_typ =IAVF_TX_CTX_DESC_EIPT_IPV4_NO_CHECKSUM_OFFLOAD ; + eip_typ =CI_TX_CTX_EIPT_IPV4_NO_CSUM ; eip_len = m->outer_l3_len >>2 ;break ;case RTE_MBUF_F_TX_OUTER_IPV4 |RTE_MBUF_F_TX_OUTER_IP_CKSUM : - eip_typ =IAVF_TX_CTX_DESC_EIPT_IPV4_CHECKSUM_OFFLOAD ; + eip_typ =CI_TX_CTX_EIPT_IPV4 ; eip_len = m->outer_l3_len >>2 ;break ;case RTE_MBUF_F_TX_OUTER_IPV6 : - eip_typ =IAVF_TX_CTX_DESC_EIPT_IPV6 ; + eip_typ =CI_TX_CTX_EIPT_IPV6 ; eip_len = m->outer_l3_len >>2 ;break ; }
@@ -2377,10 +2377,10 @@ iavf_fill_ctx_desc_tunnelling_field(uint64_t *qw0, uint64_t ol_flags, const strucase RTE_MBUF_F_TX_TUNNEL_VXLAN_GPE :case RTE_MBUF_F_TX_TUNNEL_GTP :case RTE_MBUF_F_TX_TUNNEL_GENEVE : - eip_typ |=IAVF_TXD_CTX_UDP_TUNNELING ; + eip_typ |=CI_TXD_CTX_UDP_TUNNELING ;break ;case RTE_MBUF_F_TX_TUNNEL_GRE : - eip_typ |=IAVF_TXD_CTX_GRE_TUNNELING ; + eip_typ |=CI_TXD_CTX_GRE_TUNNELING ;break ;default :PMD_TX_LOG (ERR ,"Tunnel type not supported" );
@@ -2397,23 +2397,23 @@ iavf_fill_ctx_desc_tunnelling_field(uint64_t *qw0, uint64_t ol_flags, const stru* its last Ethertype. * If MPLS labels exists, it should include them as well. */ - eip_typ |= (m->l2_len >>1 ) <<IAVF_TXD_CTX_QW0_NATLEN_SHIFT ; + eip_typ |= (m->l2_len >>1 ) <<CI_TXD_CTX_QW0_NATLEN_S ;/** * Calculate the tunneling UDP checksum. * Shall be set only if L4TUNT = 01b and EIPT is not zero */ -if ((eip_typ & (IAVF_TX_CTX_EXT_IP_IPV6 | -IAVF_TX_CTX_EXT_IP_IPV4 | -IAVF_TX_CTX_EXT_IP_IPV4_NO_CSUM )) && - (eip_typ &IAVF_TXD_CTX_UDP_TUNNELING ) && +if ((eip_typ & (CI_TX_CTX_EIPT_IPV6 | +CI_TX_CTX_EIPT_IPV4 | +CI_TX_CTX_EIPT_IPV4_NO_CSUM )) && + (eip_typ &CI_TXD_CTX_UDP_TUNNELING ) && (ol_flags &RTE_MBUF_F_TX_OUTER_UDP_CKSUM )) - eip_typ |=IAVF_TXD_CTX_QW0_L4T_CS_MASK ; + eip_typ |=CI_TXD_CTX_QW0_L4T_CS_M ; } - *qw0 = eip_typ <<IAVF_TXD_CTX_QW0_TUN_PARAMS_EIPT_SHIFT | - eip_len <<IAVF_TXD_CTX_QW0_TUN_PARAMS_EIPLEN_SHIFT | - eip_noinc <<IAVF_TXD_CTX_QW0_TUN_PARAMS_EIP_NOINC_SHIFT ; + *qw0 = eip_typ <<CI_TXD_CTX_QW0_EIPT_S | + eip_len <<CI_TXD_CTX_QW0_EIPLEN_S | + eip_noinc <<CI_TXD_CTX_QW0_EIP_NOINC_S ; } static inline uint16_t
drivers/net/intel/iavf/iavf_rxtx.h +0 -34
@@ -444,40 +444,6 @@ enum iavf_rx_flex_desc_ipsec_crypto_status {#define IAVF_TXD_CTX_QW1_SEG_PARAMS_MSS_MASK \ (0x3FFFUL << IAVF_TXD_CTX_QW1_SEG_PARAMS_MSS_SHIFT) -#define IAVF_TXD_CTX_QW0_TUN_PARAMS_EIPT_SHIFT (0) -#define IAVF_TXD_CTX_QW0_TUN_PARAMS_EIPT_MASK (0x3UL) - -enum iavf_tx_ctx_desc_tunnel_external_ip_type { -IAVF_TX_CTX_DESC_EIPT_NONE , -IAVF_TX_CTX_DESC_EIPT_IPV6 , -IAVF_TX_CTX_DESC_EIPT_IPV4_NO_CHECKSUM_OFFLOAD , -IAVF_TX_CTX_DESC_EIPT_IPV4_CHECKSUM_OFFLOAD -}; - -#define IAVF_TXD_CTX_QW0_TUN_PARAMS_EIPLEN_SHIFT (2) -#define IAVF_TXD_CTX_QW0_TUN_PARAMS_EIPLEN_MASK (0x7FUL) - -#define IAVF_TXD_CTX_QW0_TUN_PARAMS_L4TUNT_SHIFT (9) -#define IAVF_TXD_CTX_QW0_TUN_PARAMS_L4TUNT_MASK (0x3UL) - -enum iavf_tx_ctx_desc_tunnel_l4_tunnel_type { -IAVF_TX_CTX_DESC_L4_TUN_TYP_NO_UDP_GRE , -IAVF_TX_CTX_DESC_L4_TUN_TYP_UDP , -IAVF_TX_CTX_DESC_L4_TUN_TYP_GRE -}; - -#define IAVF_TXD_CTX_QW0_TUN_PARAMS_EIP_NOINC_SHIFT (11) -#define IAVF_TXD_CTX_QW0_TUN_PARAMS_EIP_NOINC_MASK (0x1UL) - -#define IAVF_TXD_CTX_QW0_TUN_PARAMS_L4TUNLEN_SHIFT (12) -#define IAVF_TXD_CTX_QW0_TUN_PARAMS_L4TUNLEN_MASK (0x7FUL) - -#define IAVF_TXD_CTX_QW0_TUN_PARAMS_DECTTL_SHIFT (19) -#define IAVF_TXD_CTX_QW0_TUN_PARAMS_DECTTL_MASK (0xFUL) - -#define IAVF_TXD_CTX_QW0_TUN_PARAMS_L4T_CS_SHIFT (23) -#define IAVF_TXD_CTX_QW0_TUN_PARAMS_L4T_CS_MASK (0x1UL) -#define IAVF_TXD_CTX_QW0_L2TAG2_PARAM (32)#define IAVF_TXD_CTX_QW0_L2TAG2_MASK (0xFFFFUL)
drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c +15 -15
@@ -1760,7 +1760,7 @@ static inline void iavf_fill_ctx_desc_tunneling_avx2(uint64_t *low_ctx_qw, struct rte_mbuf *pkt) {if (pkt->ol_flags &RTE_MBUF_F_TX_TUNNEL_MASK ) { -uint64_t eip_typ =IAVF_TX_CTX_DESC_EIPT_NONE ; +uint64_t eip_typ =CI_TX_CTX_EIPT_NONE ;uint64_t eip_len =0 ;uint64_t eip_noinc =0 ;/* Default - IP_ID is increment in each segment of LSO */
@@ -1769,15 +1769,15 @@ iavf_fill_ctx_desc_tunneling_avx2(uint64_t *low_ctx_qw, struct rte_mbuf *pkt) RTE_MBUF_F_TX_OUTER_IPV6 | RTE_MBUF_F_TX_OUTER_IP_CKSUM)) {case RTE_MBUF_F_TX_OUTER_IPV4 : - eip_typ =IAVF_TX_CTX_DESC_EIPT_IPV4_NO_CHECKSUM_OFFLOAD ; + eip_typ =CI_TX_CTX_EIPT_IPV4_NO_CSUM ; eip_len = pkt->outer_l3_len >>2 ;break ;case RTE_MBUF_F_TX_OUTER_IPV4 |RTE_MBUF_F_TX_OUTER_IP_CKSUM : - eip_typ =IAVF_TX_CTX_DESC_EIPT_IPV4_CHECKSUM_OFFLOAD ; + eip_typ =CI_TX_CTX_EIPT_IPV4 ; eip_len = pkt->outer_l3_len >>2 ;break ;case RTE_MBUF_F_TX_OUTER_IPV6 : - eip_typ =IAVF_TX_CTX_DESC_EIPT_IPV6 ; + eip_typ =CI_TX_CTX_EIPT_IPV6 ; eip_len = pkt->outer_l3_len >>2 ;break ; }
@@ -1791,10 +1791,10 @@ iavf_fill_ctx_desc_tunneling_avx2(uint64_t *low_ctx_qw, struct rte_mbuf *pkt)case RTE_MBUF_F_TX_TUNNEL_VXLAN_GPE :case RTE_MBUF_F_TX_TUNNEL_GTP :case RTE_MBUF_F_TX_TUNNEL_GENEVE : - eip_typ |=IAVF_TXD_CTX_UDP_TUNNELING ; + eip_typ |=CI_TXD_CTX_UDP_TUNNELING ;break ;case RTE_MBUF_F_TX_TUNNEL_GRE : - eip_typ |=IAVF_TXD_CTX_GRE_TUNNELING ; + eip_typ |=CI_TXD_CTX_GRE_TUNNELING ;break ;default :PMD_TX_LOG (ERR ,"Tunnel type not supported" );
@@ -1811,22 +1811,22 @@ iavf_fill_ctx_desc_tunneling_avx2(uint64_t *low_ctx_qw, struct rte_mbuf *pkt)* its last Ethertype. * If MPLS labels exists, it should include them as well. */ - eip_typ |= (pkt->l2_len >>1 ) <<IAVF_TXD_CTX_QW0_NATLEN_SHIFT ; + eip_typ |= (pkt->l2_len >>1 ) <<CI_TXD_CTX_QW0_NATLEN_S ;/** * Calculate the tunneling UDP checksum. * Shall be set only if L4TUNT = 01b and EIPT is not zero */ -if ((eip_typ & (IAVF_TX_CTX_EXT_IP_IPV4 | -IAVF_TX_CTX_EXT_IP_IPV6 | -IAVF_TX_CTX_EXT_IP_IPV4_NO_CSUM )) && - (eip_typ &IAVF_TXD_CTX_UDP_TUNNELING ) && +if ((eip_typ & (CI_TX_CTX_EIPT_IPV4 | +CI_TX_CTX_EIPT_IPV6 | +CI_TX_CTX_EIPT_IPV4_NO_CSUM )) && + (eip_typ &CI_TXD_CTX_UDP_TUNNELING ) && (pkt->ol_flags &RTE_MBUF_F_TX_OUTER_UDP_CKSUM )) - eip_typ |=IAVF_TXD_CTX_QW0_L4T_CS_MASK ; + eip_typ |=CI_TXD_CTX_QW0_L4T_CS_M ; - *low_ctx_qw = eip_typ <<IAVF_TXD_CTX_QW0_TUN_PARAMS_EIPT_SHIFT | - eip_len <<IAVF_TXD_CTX_QW0_TUN_PARAMS_EIPLEN_SHIFT | - eip_noinc <<IAVF_TXD_CTX_QW0_TUN_PARAMS_EIP_NOINC_SHIFT ; + *low_ctx_qw = eip_typ <<CI_TXD_CTX_QW0_EIPT_S | + eip_len <<CI_TXD_CTX_QW0_EIPLEN_S | + eip_noinc <<CI_TXD_CTX_QW0_EIP_NOINC_S ; }else { *low_ctx_qw =0 ;
drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c +15 -15
@@ -1899,7 +1899,7 @@ static __rte_always_inline void iavf_fill_ctx_desc_tunneling_avx512(uint64_t *low_ctx_qw, struct rte_mbuf *pkt) {if (pkt->ol_flags &RTE_MBUF_F_TX_TUNNEL_MASK ) { -uint64_t eip_typ =IAVF_TX_CTX_DESC_EIPT_NONE ; +uint64_t eip_typ =CI_TX_CTX_EIPT_NONE ;uint64_t eip_len =0 ;uint64_t eip_noinc =0 ;/* Default - IP_ID is increment in each segment of LSO */
@@ -1908,15 +1908,15 @@ iavf_fill_ctx_desc_tunneling_avx512(uint64_t *low_ctx_qw, struct rte_mbuf *pkt) RTE_MBUF_F_TX_OUTER_IPV6 | RTE_MBUF_F_TX_OUTER_IP_CKSUM)) {case RTE_MBUF_F_TX_OUTER_IPV4 : - eip_typ =IAVF_TX_CTX_DESC_EIPT_IPV4_NO_CHECKSUM_OFFLOAD ; + eip_typ =CI_TX_CTX_EIPT_IPV4_NO_CSUM ; eip_len = pkt->outer_l3_len >>2 ;break ;case RTE_MBUF_F_TX_OUTER_IPV4 |RTE_MBUF_F_TX_OUTER_IP_CKSUM : - eip_typ =IAVF_TX_CTX_DESC_EIPT_IPV4_CHECKSUM_OFFLOAD ; + eip_typ =CI_TX_CTX_EIPT_IPV4 ; eip_len = pkt->outer_l3_len >>2 ;break ;case RTE_MBUF_F_TX_OUTER_IPV6 : - eip_typ =IAVF_TX_CTX_DESC_EIPT_IPV6 ; + eip_typ =CI_TX_CTX_EIPT_IPV6 ; eip_len = pkt->outer_l3_len >>2 ;break ; }
@@ -1930,10 +1930,10 @@ iavf_fill_ctx_desc_tunneling_avx512(uint64_t *low_ctx_qw, struct rte_mbuf *pkt)case RTE_MBUF_F_TX_TUNNEL_VXLAN_GPE :case RTE_MBUF_F_TX_TUNNEL_GTP :case RTE_MBUF_F_TX_TUNNEL_GENEVE : - eip_typ |=IAVF_TXD_CTX_UDP_TUNNELING ; + eip_typ |=CI_TXD_CTX_UDP_TUNNELING ;break ;case RTE_MBUF_F_TX_TUNNEL_GRE : - eip_typ |=IAVF_TXD_CTX_GRE_TUNNELING ; + eip_typ |=CI_TXD_CTX_GRE_TUNNELING ;break ;default :PMD_TX_LOG (ERR ,"Tunnel type not supported" );
@@ -1950,22 +1950,22 @@ iavf_fill_ctx_desc_tunneling_avx512(uint64_t *low_ctx_qw, struct rte_mbuf *pkt)* its last Ethertype. * If MPLS labels exists, it should include them as well. */ - eip_typ |= (pkt->l2_len >>1 ) <<IAVF_TXD_CTX_QW0_NATLEN_SHIFT ; + eip_typ |= (pkt->l2_len >>1 ) <<CI_TXD_CTX_QW0_NATLEN_S ;/** * Calculate the tunneling UDP checksum. * Shall be set only if L4TUNT = 01b and EIPT is not zero */ -if ((eip_typ & (IAVF_TX_CTX_EXT_IP_IPV4 | -IAVF_TX_CTX_EXT_IP_IPV6 | -IAVF_TX_CTX_EXT_IP_IPV4_NO_CSUM )) && - (eip_typ &IAVF_TXD_CTX_UDP_TUNNELING ) && +if ((eip_typ & (CI_TX_CTX_EIPT_IPV4 | +CI_TX_CTX_EIPT_IPV6 | +CI_TX_CTX_EIPT_IPV4_NO_CSUM )) && + (eip_typ &CI_TXD_CTX_UDP_TUNNELING ) && (pkt->ol_flags &RTE_MBUF_F_TX_OUTER_UDP_CKSUM )) - eip_typ |=IAVF_TXD_CTX_QW0_L4T_CS_MASK ; + eip_typ |=CI_TXD_CTX_QW0_L4T_CS_M ; - *low_ctx_qw = eip_typ <<IAVF_TXD_CTX_QW0_TUN_PARAMS_EIPT_SHIFT | - eip_len <<IAVF_TXD_CTX_QW0_TUN_PARAMS_EIPLEN_SHIFT | - eip_noinc <<IAVF_TXD_CTX_QW0_TUN_PARAMS_EIP_NOINC_SHIFT ; + *low_ctx_qw = eip_typ <<CI_TXD_CTX_QW0_EIPT_S | + eip_len <<CI_TXD_CTX_QW0_EIPLEN_S | + eip_noinc <<CI_TXD_CTX_QW0_EIP_NOINC_S ; }else { *low_ctx_qw =0 ;
Signature · 1 line
2.53.0
RE: [PATCH 05/13] net/intel: define common macros for tunneling bit-shifts
Quoted 7 lines
Subject: [PATCH 05/13] net/intel: define common macros for tunneling bit- shifts
Define common macros for use in iavf (for now) and other drivers (future) defining the shifts needed for tunnel packet Tx
Signed-off-by Bruce Richardson <[email protected]>
Acked-by Ciara Loftus <[email protected]>
Quoted 8 lines
--- drivers/net/intel/common/tx.h | 24 +++++++++++++ drivers/net/intel/iavf/iavf_rxtx.c | 30 ++++++++-------- drivers/net/intel/iavf/iavf_rxtx.h | 34 ------------------- drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c | 30 ++++++++-------- drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c | 30 ++++++++-------- 5 files changed, 69 insertions(+), 79 deletions(-)
PATCH 6/13 net/intel: move iavf descriptor writing functions to common
In reply to Bruce RichardsonIn order to promote reuse of code, move the Tx vector functions for writing descriptors from the iavf driver to the common folder in a new tx_vec_x86.h file. This will allow reuse in later commits by other drivers such as ice.
Signed-off-by Bruce Richardson <[email protected]>
--- drivers/net/intel/common/tx_vec_x86.h | 513 ++++++++++++++++++ drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c | 285 +--------- drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c | 272 +--------- drivers/net/intel/iavf/iavf_rxtx_vec_common.h | 77 +-- 4 files changed, 532 insertions(+), 615 deletions(-) create mode 100644 drivers/net/intel/common/tx_vec_x86.h
drivers/net/intel/common/tx_vec_x86.h new file +513 -0
@@ -0,0 +1,513 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2026 Intel Corporation +*/ + +#ifndef _COMMON_INTEL_TX_VEC_X86_H_ +#define _COMMON_INTEL_TX_VEC_X86_H_ + +#include <stdint.h> + +#include <rte_mbuf.h> + +#include "tx.h" + +static __rte_always_inline void +ci_fill_ctx_desc_tunneling (uint64_t *low_ctx_qw,struct rte_mbuf *pkt) +{ +if (pkt->ol_flags &RTE_MBUF_F_TX_TUNNEL_MASK ) { +uint64_t eip_typ =CI_TX_CTX_EIPT_NONE ; +uint64_t eip_len =0 ; +uint64_t eip_noinc =0 ; +/* Default - IP_ID is increment in each segment of LSO */ + +switch (pkt->ol_flags & (RTE_MBUF_F_TX_OUTER_IPV4 | +RTE_MBUF_F_TX_OUTER_IPV6 | +RTE_MBUF_F_TX_OUTER_IP_CKSUM )) { +case RTE_MBUF_F_TX_OUTER_IPV4 : + eip_typ =CI_TX_CTX_EIPT_IPV4_NO_CSUM ; + eip_len = pkt->outer_l3_len >>2 ; +break ; +case RTE_MBUF_F_TX_OUTER_IPV4 |RTE_MBUF_F_TX_OUTER_IP_CKSUM : + eip_typ =CI_TX_CTX_EIPT_IPV4 ; + eip_len = pkt->outer_l3_len >>2 ; +break ; +case RTE_MBUF_F_TX_OUTER_IPV6 : + eip_typ =CI_TX_CTX_EIPT_IPV6 ; + eip_len = pkt->outer_l3_len >>2 ; +break ; + } + +/* L4TUNT: L4 Tunneling Type */ +switch (pkt->ol_flags &RTE_MBUF_F_TX_TUNNEL_MASK ) { +case RTE_MBUF_F_TX_TUNNEL_IPIP : +/* for non UDP / GRE tunneling, set to 00b */ +break ; +case RTE_MBUF_F_TX_TUNNEL_VXLAN : +case RTE_MBUF_F_TX_TUNNEL_VXLAN_GPE : +case RTE_MBUF_F_TX_TUNNEL_GTP : +case RTE_MBUF_F_TX_TUNNEL_GENEVE : + eip_typ |=CI_TXD_CTX_UDP_TUNNELING ; +break ; +case RTE_MBUF_F_TX_TUNNEL_GRE : + eip_typ |=CI_TXD_CTX_GRE_TUNNELING ; +break ; +default : +PMD_TX_LOG (ERR ,"Tunnel type not supported" ); +return ; + } + +/* L4TUNLEN: L4 Tunneling Length, in Words +* +* We depend on app to set rte_mbuf.l2_len correctly. +* For IP in GRE it should be set to the length of the GRE +* header; +* For MAC in GRE or MAC in UDP it should be set to the length +* of the GRE or UDP headers plus the inner MAC up to including +* its last Ethertype. +* If MPLS labels exists, it should include them as well. +*/ + eip_typ |= (pkt->l2_len >>1 ) <<CI_TXD_CTX_QW0_NATLEN_S ; + +/** +* Calculate the tunneling UDP checksum. +* Shall be set only if L4TUNT = 01b and EIPT is not zero +*/ +if ((eip_typ & (CI_TX_CTX_EIPT_IPV4 | +CI_TX_CTX_EIPT_IPV6 | +CI_TX_CTX_EIPT_IPV4_NO_CSUM )) && + (eip_typ &CI_TXD_CTX_UDP_TUNNELING ) && + (pkt->ol_flags &RTE_MBUF_F_TX_OUTER_UDP_CKSUM )) + eip_typ |=CI_TXD_CTX_QW0_L4T_CS_M ; + + *low_ctx_qw = eip_typ <<CI_TXD_CTX_QW0_EIPT_S | + eip_len <<CI_TXD_CTX_QW0_EIPLEN_S | + eip_noinc <<CI_TXD_CTX_QW0_EIP_NOINC_S ; + + }else { + *low_ctx_qw =0 ; + } +} + +static __rte_always_inline void +ci_tx_vec_offload (struct rte_mbuf *tx_pkt,uint64_t *txd_hi, +enum ci_l2tag_pos single_vlan_pos,enum ci_l2tag_pos qinq_outer_pos) +{ +uint64_t ol_flags = tx_pkt->ol_flags; +uint32_t td_cmd =0 ; +uint32_t td_offset =0 ; + +/* Set MACLEN */ +if (ol_flags &RTE_MBUF_F_TX_TUNNEL_MASK ) + td_offset |= (tx_pkt->outer_l2_len >>1 ) <<CI_TX_DESC_LEN_MACLEN_S ; +else + td_offset |= (tx_pkt->l2_len >>1 ) <<CI_TX_DESC_LEN_MACLEN_S ; + +/* Enable L3 checksum offloads */ +if (ol_flags &RTE_MBUF_F_TX_IP_CKSUM ) { +if (ol_flags &RTE_MBUF_F_TX_IPV4 ) { + td_cmd |=CI_TX_DESC_CMD_IIPT_IPV4_CSUM ; + td_offset |= (tx_pkt->l3_len >>2 ) <<CI_TX_DESC_LEN_IPLEN_S ; + } + }else if (ol_flags &RTE_MBUF_F_TX_IPV4 ) { + td_cmd |=CI_TX_DESC_CMD_IIPT_IPV4 ; + td_offset |= (tx_pkt->l3_len >>2 ) <<CI_TX_DESC_LEN_IPLEN_S ; + }else if (ol_flags &RTE_MBUF_F_TX_IPV6 ) { + td_cmd |=CI_TX_DESC_CMD_IIPT_IPV6 ; + td_offset |= (tx_pkt->l3_len >>2 ) <<CI_TX_DESC_LEN_IPLEN_S ; + } + +/* Enable L4 checksum offloads */ +switch (ol_flags &RTE_MBUF_F_TX_L4_MASK ) { +case RTE_MBUF_F_TX_TCP_CKSUM : + td_cmd |=CI_TX_DESC_CMD_L4T_EOFT_TCP ; + td_offset |= (sizeof (struct rte_tcp_hdr ) >>2 ) <<CI_TX_DESC_LEN_L4_LEN_S ; +break ; +case RTE_MBUF_F_TX_SCTP_CKSUM : + td_cmd |=CI_TX_DESC_CMD_L4T_EOFT_SCTP ; + td_offset |= (sizeof (struct rte_sctp_hdr ) >>2 ) <<CI_TX_DESC_LEN_L4_LEN_S ; +break ; +case RTE_MBUF_F_TX_UDP_CKSUM : + td_cmd |=CI_TX_DESC_CMD_L4T_EOFT_UDP ; + td_offset |= (sizeof (struct rte_udp_hdr ) >>2 ) <<CI_TX_DESC_LEN_L4_LEN_S ; +break ; +default : +break ; + } + + *txd_hi |= ((uint64_t )td_offset) <<CI_TXD_QW1_OFFSET_S ; + +if (ol_flags &RTE_MBUF_F_TX_QINQ ) { + td_cmd |=CI_TX_DESC_CMD_IL2TAG1 ; +/* L2Tag1 always carries a tag for QinQ: the outer tag if that's +* where it is placed, otherwise the inner. +*/ +if (qinq_outer_pos ==CI_TAG_IN_DATA_DESC ) + *txd_hi |= ((uint64_t )tx_pkt->vlan_tci_outer <<CI_TXD_QW1_L2TAG1_S ); +else + *txd_hi |= ((uint64_t )tx_pkt->vlan_tci <<CI_TXD_QW1_L2TAG1_S ); + }else if (ol_flags &RTE_MBUF_F_TX_VLAN && single_vlan_pos ==CI_TAG_IN_DATA_DESC ) { + td_cmd |=CI_TX_DESC_CMD_IL2TAG1 ; + *txd_hi |= ((uint64_t )tx_pkt->vlan_tci <<CI_TXD_QW1_L2TAG1_S ); + } + + *txd_hi |= ((uint64_t )td_cmd) <<CI_TXD_QW1_CMD_S ; +} + +static __rte_always_inline void +ci_vtx1 (volatile struct ci_tx_desc *txdp, +struct rte_mbuf *pkt,uint64_t flags,bool offload, +enum ci_l2tag_pos single_vlan_pos,enum ci_l2tag_pos qinq_outer_pos) +{ +uint64_t high_qw = (CI_TX_DESC_DTYPE_DATA | + ((uint64_t )flags <<CI_TXD_QW1_CMD_S ) | + ((uint64_t )pkt->data_len <<CI_TXD_QW1_TX_BUF_SZ_S )); +if (offload) +ci_tx_vec_offload (pkt, &high_qw, single_vlan_pos, qinq_outer_pos); + +__m128i descriptor =_mm_set_epi64x (high_qw, pkt->buf_iova + pkt->data_off); + _mm_store_si128(RTE_CAST_PTR(__m128i *, txdp), descriptor); +} + +#ifdef __AVX2__ + +static __rte_always_inline void +ci_vtx_avx2 (volatile struct ci_tx_desc *txdp, +struct rte_mbuf **pkt,uint16_t nb_pkts,uint64_t flags,bool offload, +enum ci_l2tag_pos single_vlan_pos,enum ci_l2tag_pos qinq_outer_pos) +{ +const uint64_t hi_qw_tmpl = (CI_TX_DESC_DTYPE_DATA | (flags <<CI_TXD_QW1_CMD_S )); + +/* if unaligned on 32-bit boundary, do one to align */ +if (((uintptr_t )txdp &0x1F ) !=0 && nb_pkts !=0 ) { +ci_vtx1 (txdp, *pkt, flags, offload, single_vlan_pos, qinq_outer_pos); + nb_pkts--; txdp++; pkt++; + } + +/* do two at a time while possible, in bursts */ +for (; nb_pkts >3 ; txdp +=4 , pkt +=4 , nb_pkts -=4 ) { +uint64_t hi_qw3 = hi_qw_tmpl | + ((uint64_t )pkt[3 ]->data_len <<CI_TXD_QW1_TX_BUF_SZ_S ); +if (offload) +ci_tx_vec_offload (pkt[3 ], &hi_qw3, single_vlan_pos, qinq_outer_pos); +uint64_t hi_qw2 = hi_qw_tmpl | + ((uint64_t )pkt[2 ]->data_len <<CI_TXD_QW1_TX_BUF_SZ_S ); +if (offload) +ci_tx_vec_offload (pkt[2 ], &hi_qw2, single_vlan_pos, qinq_outer_pos); +uint64_t hi_qw1 = hi_qw_tmpl | + ((uint64_t )pkt[1 ]->data_len <<CI_TXD_QW1_TX_BUF_SZ_S ); +if (offload) +ci_tx_vec_offload (pkt[1 ], &hi_qw1, single_vlan_pos, qinq_outer_pos); +uint64_t hi_qw0 = hi_qw_tmpl | + ((uint64_t )pkt[0 ]->data_len <<CI_TXD_QW1_TX_BUF_SZ_S ); +if (offload) +ci_tx_vec_offload (pkt[0 ], &hi_qw0, single_vlan_pos, qinq_outer_pos); + +__m256i desc2_3 = +_mm256_set_epi64x + (hi_qw3, + pkt[3 ]->buf_iova + pkt[3 ]->data_off, + hi_qw2, + pkt[2 ]->buf_iova + pkt[2 ]->data_off); +__m256i desc0_1 = +_mm256_set_epi64x + (hi_qw1, + pkt[1 ]->buf_iova + pkt[1 ]->data_off, + hi_qw0, + pkt[0 ]->buf_iova + pkt[0 ]->data_off); + _mm256_store_si256(RTE_CAST_PTR(__m256i *, txdp + 2), desc2_3); + _mm256_store_si256(RTE_CAST_PTR(__m256i *, txdp), desc0_1); + } + +/* do any last ones */ +while (nb_pkts) { +ci_vtx1 (txdp, *pkt, flags, offload, single_vlan_pos, qinq_outer_pos); + txdp++; pkt++; nb_pkts--; + } +} + +static __rte_always_inline void +ci_vtx1_ctx_avx2 (volatile struct ci_tx_desc *txdp,struct rte_mbuf *pkt, +uint64_t flags,bool offload,enum ci_l2tag_pos single_vlan_pos, +enum ci_l2tag_pos qinq_outer_pos,bool ptype_lldp_enabled) +{ +uint64_t high_ctx_qw =CI_TX_DESC_DTYPE_CTX ; +uint64_t low_ctx_qw =0 ; + +if (offload) { +ci_fill_ctx_desc_tunneling (&low_ctx_qw, pkt); +if (pkt->ol_flags &RTE_MBUF_F_TX_QINQ ) { +uint64_t qinq_tag = qinq_outer_pos ==CI_TAG_IN_CTX_DESC ? + (uint64_t )pkt->vlan_tci_outer : + (uint64_t )pkt->vlan_tci; + high_ctx_qw |=CI_TX_CTX_DESC_IL2TAG2 <<CI_TXD_QW1_CMD_S ; + low_ctx_qw |= qinq_tag <<CI_TXD_CTX_QW0_L2TAG2_S ; + }else if ((pkt->ol_flags &RTE_MBUF_F_TX_VLAN ) && + single_vlan_pos ==CI_TAG_IN_CTX_DESC ) { + high_ctx_qw |=CI_TX_CTX_DESC_IL2TAG2 <<CI_TXD_QW1_CMD_S ; + low_ctx_qw |= (uint64_t )pkt->vlan_tci <<CI_TXD_CTX_QW0_L2TAG2_S ; + } + } +if (IAVF_CHECK_TX_LLDP (pkt, ptype_lldp_enabled)) + high_ctx_qw |=IAVF_TX_CTX_DESC_SWTCH_UPLINK <<CI_TXD_QW1_CMD_S ; +uint64_t high_data_qw = (CI_TX_DESC_DTYPE_DATA | + ((uint64_t )flags <<CI_TXD_QW1_CMD_S ) | + ((uint64_t )pkt->data_len <<CI_TXD_QW1_TX_BUF_SZ_S )); +if (offload) +ci_tx_vec_offload (pkt, &high_data_qw, single_vlan_pos, qinq_outer_pos); + +__m256i ctx_data_desc =_mm256_set_epi64x (high_data_qw, pkt->buf_iova + pkt->data_off, + high_ctx_qw, low_ctx_qw); + +/* tx_id is always even in ctx mode, so txdp is always 32-byte aligned */ + _mm256_store_si256(RTE_CAST_PTR(__m256i *, txdp), ctx_data_desc); +} + +static __rte_always_inline void +ci_vtx_ctx_avx2 (volatile struct ci_tx_desc *txdp, +struct rte_mbuf **pkt,uint16_t nb_pkts,uint64_t flags, +bool offload,enum ci_l2tag_pos single_vlan_pos,enum ci_l2tag_pos qinq_outer_pos, +bool ptype_lldp_enabled) +{ +uint64_t hi_data_qw_tmpl = (CI_TX_DESC_DTYPE_DATA | (flags <<CI_TXD_QW1_CMD_S )); + +for (; nb_pkts >1 ; txdp +=4 , pkt +=2 , nb_pkts -=2 ) { +uint64_t hi_ctx_qw1 =CI_TX_DESC_DTYPE_CTX ; +uint64_t hi_ctx_qw0 =CI_TX_DESC_DTYPE_CTX ; +uint64_t low_ctx_qw1 =0 ; +uint64_t low_ctx_qw0 =0 ; +uint64_t hi_data_qw1 =0 ; +uint64_t hi_data_qw0 =0 ; + + hi_data_qw1 = hi_data_qw_tmpl | + ((uint64_t )pkt[1 ]->data_len << +CI_TXD_QW1_TX_BUF_SZ_S ); + hi_data_qw0 = hi_data_qw_tmpl | + ((uint64_t )pkt[0 ]->data_len << +CI_TXD_QW1_TX_BUF_SZ_S ); + +if (offload) { +/* tunnel fill assigns low_ctx_qw1; must run before QinQ/VLAN OR below */ +ci_fill_ctx_desc_tunneling (&low_ctx_qw1, pkt[1 ]); +if (pkt[1 ]->ol_flags &RTE_MBUF_F_TX_QINQ ) { +uint64_t qinq_tag = qinq_outer_pos ==CI_TAG_IN_CTX_DESC ? + (uint64_t )pkt[1 ]->vlan_tci_outer : + (uint64_t )pkt[1 ]->vlan_tci; + hi_ctx_qw1 |=CI_TX_CTX_DESC_IL2TAG2 <<CI_TXD_QW1_CMD_S ; + low_ctx_qw1 |= qinq_tag <<CI_TXD_CTX_QW0_L2TAG2_S ; + }else if (pkt[1 ]->ol_flags &RTE_MBUF_F_TX_VLAN && + single_vlan_pos ==CI_TAG_IN_CTX_DESC ) { + hi_ctx_qw1 |=CI_TX_CTX_DESC_IL2TAG2 <<CI_TXD_QW1_CMD_S ; + low_ctx_qw1 |= (uint64_t )pkt[1 ]->vlan_tci <<CI_TXD_CTX_QW0_L2TAG2_S ; + } + } +if (IAVF_CHECK_TX_LLDP (pkt[1 ], ptype_lldp_enabled)) + hi_ctx_qw1 |=IAVF_TX_CTX_DESC_SWTCH_UPLINK <<CI_TXD_QW1_CMD_S ; + +if (offload) { +/* tunnel fill assigns low_ctx_qw0; must run before QinQ/VLAN OR below */ +ci_fill_ctx_desc_tunneling (&low_ctx_qw0, pkt[0 ]); +if (pkt[0 ]->ol_flags &RTE_MBUF_F_TX_QINQ ) { +uint64_t qinq_tag = qinq_outer_pos ==CI_TAG_IN_CTX_DESC ? + (uint64_t )pkt[0 ]->vlan_tci_outer : + (uint64_t )pkt[0 ]->vlan_tci; + hi_ctx_qw0 |=CI_TX_CTX_DESC_IL2TAG2 <<CI_TXD_QW1_CMD_S ; + low_ctx_qw0 |= qinq_tag <<CI_TXD_CTX_QW0_L2TAG2_S ; + }else if (pkt[0 ]->ol_flags &RTE_MBUF_F_TX_VLAN && + single_vlan_pos ==CI_TAG_IN_CTX_DESC ) { + hi_ctx_qw0 |=CI_TX_CTX_DESC_IL2TAG2 <<CI_TXD_QW1_CMD_S ; + low_ctx_qw0 |= (uint64_t )pkt[0 ]->vlan_tci <<CI_TXD_CTX_QW0_L2TAG2_S ; + } + } +if (IAVF_CHECK_TX_LLDP (pkt[0 ], ptype_lldp_enabled)) + hi_ctx_qw0 |=IAVF_TX_CTX_DESC_SWTCH_UPLINK <<CI_TXD_QW1_CMD_S ; + +if (offload) { +ci_tx_vec_offload (pkt[1 ], &hi_data_qw1, single_vlan_pos, qinq_outer_pos); +ci_tx_vec_offload (pkt[0 ], &hi_data_qw0, single_vlan_pos, qinq_outer_pos); + } + +__m256i desc2_3 =_mm256_set_epi64x + (hi_data_qw1, pkt[1 ]->buf_iova + pkt[1 ]->data_off, + hi_ctx_qw1, low_ctx_qw1); +__m256i desc0_1 =_mm256_set_epi64x + (hi_data_qw0, pkt[0 ]->buf_iova + pkt[0 ]->data_off, + hi_ctx_qw0, low_ctx_qw0); + _mm256_store_si256(RTE_CAST_PTR(__m256i *, txdp + 2), desc2_3); + _mm256_store_si256(RTE_CAST_PTR(__m256i *, txdp), desc0_1); + } + +if (nb_pkts) +ci_vtx1_ctx_avx2 (txdp, *pkt, flags, offload, + single_vlan_pos, qinq_outer_pos, ptype_lldp_enabled); +} + +#endif /* __AVX2__ */ + +#ifdef __AVX512VL__ + +static __rte_always_inline void +ci_vtx_avx512 (volatile struct ci_tx_desc *txdp, +struct rte_mbuf **pkt,uint16_t nb_pkts,uint64_t flags, +bool offload,enum ci_l2tag_pos single_vlan_pos,enum ci_l2tag_pos qinq_outer_pos) +{ +const uint64_t hi_qw_tmpl = (CI_TX_DESC_DTYPE_DATA | (flags <<CI_TXD_QW1_CMD_S )); + +/* if unaligned on 32-bit boundary, do one to align */ +if (((uintptr_t )txdp &0x1F ) !=0 && nb_pkts !=0 ) { +ci_vtx1 (txdp, *pkt, flags, offload, single_vlan_pos, qinq_outer_pos); + nb_pkts--; txdp++; pkt++; + } + +/* do 4 at a time while possible, in bursts */ +for (; nb_pkts >3 ; txdp +=4 , pkt +=4 , nb_pkts -=4 ) { +uint64_t hi_qw3 = hi_qw_tmpl | + ((uint64_t )pkt[3 ]->data_len <<CI_TXD_QW1_TX_BUF_SZ_S ); +uint64_t hi_qw2 = hi_qw_tmpl | + ((uint64_t )pkt[2 ]->data_len <<CI_TXD_QW1_TX_BUF_SZ_S ); +uint64_t hi_qw1 = hi_qw_tmpl | + ((uint64_t )pkt[1 ]->data_len <<CI_TXD_QW1_TX_BUF_SZ_S ); +uint64_t hi_qw0 = hi_qw_tmpl | + ((uint64_t )pkt[0 ]->data_len <<CI_TXD_QW1_TX_BUF_SZ_S ); +if (offload) { +ci_tx_vec_offload (pkt[3 ], &hi_qw3, single_vlan_pos, qinq_outer_pos); +ci_tx_vec_offload (pkt[2 ], &hi_qw2, single_vlan_pos, qinq_outer_pos); +ci_tx_vec_offload (pkt[1 ], &hi_qw1, single_vlan_pos, qinq_outer_pos); +ci_tx_vec_offload (pkt[0 ], &hi_qw0, single_vlan_pos, qinq_outer_pos); + } + +__m512i desc0_3 = +_mm512_set_epi64 + (hi_qw3, + pkt[3 ]->buf_iova + pkt[3 ]->data_off, + hi_qw2, + pkt[2 ]->buf_iova + pkt[2 ]->data_off, + hi_qw1, + pkt[1 ]->buf_iova + pkt[1 ]->data_off, + hi_qw0, + pkt[0 ]->buf_iova + pkt[0 ]->data_off); + _mm512_storeu_si512(RTE_CAST_PTR(void *, txdp), desc0_3); + } + +/* do any last ones */ +while (nb_pkts) { +ci_vtx1 (txdp, *pkt, flags, offload, single_vlan_pos, qinq_outer_pos); + txdp++; pkt++; nb_pkts--; + } +} + +static __rte_always_inline void +ci_vtx1_ctx_avx512 (volatile struct ci_tx_desc *txdp,struct rte_mbuf *pkt, +uint64_t flags,bool offload,enum ci_l2tag_pos single_vlan_pos, +enum ci_l2tag_pos qinq_outer_pos,bool lldp_enabled) +{ +uint64_t high_ctx_qw =CI_TX_DESC_DTYPE_CTX ; +uint64_t low_ctx_qw =0 ; + +if (offload) { +ci_fill_ctx_desc_tunneling (&low_ctx_qw, pkt); +if (pkt->ol_flags &RTE_MBUF_F_TX_QINQ ) { +uint64_t qinq_tag = qinq_outer_pos ==CI_TAG_IN_CTX_DESC ? + (uint64_t )pkt->vlan_tci_outer : + (uint64_t )pkt->vlan_tci; + high_ctx_qw |=CI_TX_CTX_DESC_IL2TAG2 <<CI_TXD_QW1_CMD_S ; + low_ctx_qw |= qinq_tag <<CI_TXD_CTX_QW0_L2TAG2_S ; + }else if ((pkt->ol_flags &RTE_MBUF_F_TX_VLAN ) && + single_vlan_pos ==CI_TAG_IN_CTX_DESC ) { + high_ctx_qw |=CI_TX_CTX_DESC_IL2TAG2 <<CI_TXD_QW1_CMD_S ; + low_ctx_qw |= (uint64_t )pkt->vlan_tci <<CI_TXD_CTX_QW0_L2TAG2_S ; + } + } +if (IAVF_CHECK_TX_LLDP (pkt, lldp_enabled)) + high_ctx_qw |=IAVF_TX_CTX_DESC_SWTCH_UPLINK <<CI_TXD_QW1_CMD_S ; +uint64_t high_data_qw = (CI_TX_DESC_DTYPE_DATA | + ((uint64_t )flags <<CI_TXD_QW1_CMD_S ) | + ((uint64_t )pkt->data_len <<CI_TXD_QW1_TX_BUF_SZ_S )); +if (offload) +ci_tx_vec_offload (pkt, &high_data_qw, single_vlan_pos, qinq_outer_pos); + +__m256i ctx_data_desc =_mm256_set_epi64x + (high_data_qw, pkt->buf_iova + pkt->data_off, + high_ctx_qw, low_ctx_qw); + +/* tx_id is always even in ctx mode, so txdp is always 32-byte aligned */ + _mm256_store_si256(RTE_CAST_PTR(__m256i *, txdp), ctx_data_desc); +} + +static __rte_always_inline void +ci_vtx_ctx_avx512 (volatile struct ci_tx_desc *txdp, +struct rte_mbuf **pkt,uint16_t nb_pkts,uint64_t flags, +bool offload,enum ci_l2tag_pos single_vlan_pos,enum ci_l2tag_pos qinq_outer_pos, +bool lldp_enabled) +{ +uint64_t hi_data_qw_tmpl = (CI_TX_DESC_DTYPE_DATA | (flags <<CI_TXD_QW1_CMD_S )); + +for (; nb_pkts >1 ; txdp +=4 , pkt +=2 , nb_pkts -=2 ) { +uint64_t hi_ctx_qw1 =CI_TX_DESC_DTYPE_CTX ; +uint64_t hi_ctx_qw0 =CI_TX_DESC_DTYPE_CTX ; +uint64_t low_ctx_qw1 =0 ; +uint64_t low_ctx_qw0 =0 ; +uint64_t hi_data_qw1 =0 ; +uint64_t hi_data_qw0 =0 ; + + hi_data_qw1 = hi_data_qw_tmpl | + ((uint64_t )pkt[1 ]->data_len <<CI_TXD_QW1_TX_BUF_SZ_S ); + hi_data_qw0 = hi_data_qw_tmpl | + ((uint64_t )pkt[0 ]->data_len <<CI_TXD_QW1_TX_BUF_SZ_S ); + +if (offload) { +/* tunnel fill assigns low_ctx_qw1; must run before QinQ/VLAN OR below */ +ci_fill_ctx_desc_tunneling (&low_ctx_qw1, pkt[1 ]); +if (pkt[1 ]->ol_flags &RTE_MBUF_F_TX_QINQ ) { +uint64_t qinq_tag = qinq_outer_pos ==CI_TAG_IN_CTX_DESC ? + (uint64_t )pkt[1 ]->vlan_tci_outer : + (uint64_t )pkt[1 ]->vlan_tci; + hi_ctx_qw1 |=CI_TX_CTX_DESC_IL2TAG2 <<CI_TXD_QW1_CMD_S ; + low_ctx_qw1 |= qinq_tag <<CI_TXD_CTX_QW0_L2TAG2_S ; + }else if (pkt[1 ]->ol_flags &RTE_MBUF_F_TX_VLAN && + single_vlan_pos ==CI_TAG_IN_CTX_DESC ) { + hi_ctx_qw1 |=CI_TX_CTX_DESC_IL2TAG2 <<CI_TXD_QW1_CMD_S ; + low_ctx_qw1 |= (uint64_t )pkt[1 ]->vlan_tci <<CI_TXD_CTX_QW0_L2TAG2_S ; + } + } +if (IAVF_CHECK_TX_LLDP (pkt[1 ], lldp_enabled)) + hi_ctx_qw1 |=IAVF_TX_CTX_DESC_SWTCH_UPLINK <<CI_TXD_QW1_CMD_S ; + +if (offload) { +/* tunnel fill assigns low_ctx_qw0; must run before QinQ/VLAN OR below */ +ci_fill_ctx_desc_tunneling (&low_ctx_qw0, pkt[0 ]); +if (pkt[0 ]->ol_flags &RTE_MBUF_F_TX_QINQ ) { +uint64_t qinq_tag = qinq_outer_pos ==CI_TAG_IN_CTX_DESC ? + (uint64_t )pkt[0 ]->vlan_tci_outer : + (uint64_t )pkt[0 ]->vlan_tci; + hi_ctx_qw0 |=CI_TX_CTX_DESC_IL2TAG2 <<CI_TXD_QW1_CMD_S ; + low_ctx_qw0 |= qinq_tag <<CI_TXD_CTX_QW0_L2TAG2_S ; + }else if (pkt[0 ]->ol_flags &RTE_MBUF_F_TX_VLAN && + single_vlan_pos ==CI_TAG_IN_CTX_DESC ) { + hi_ctx_qw0 |=CI_TX_CTX_DESC_IL2TAG2 <<CI_TXD_QW1_CMD_S ; + low_ctx_qw0 |= (uint64_t )pkt[0 ]->vlan_tci <<CI_TXD_CTX_QW0_L2TAG2_S ; + } + } +if (IAVF_CHECK_TX_LLDP (pkt[0 ], lldp_enabled)) + hi_ctx_qw0 |=IAVF_TX_CTX_DESC_SWTCH_UPLINK <<CI_TXD_QW1_CMD_S ; + +if (offload) { +ci_tx_vec_offload (pkt[1 ], &hi_data_qw1, single_vlan_pos, qinq_outer_pos); +ci_tx_vec_offload (pkt[0 ], &hi_data_qw0, single_vlan_pos, qinq_outer_pos); + } + +__m512i desc0_3 =_mm512_set_epi64 + (hi_data_qw1, pkt[1 ]->buf_iova + pkt[1 ]->data_off, + hi_ctx_qw1, low_ctx_qw1, + hi_data_qw0, pkt[0 ]->buf_iova + pkt[0 ]->data_off, + hi_ctx_qw0, low_ctx_qw0); + _mm512_storeu_si512(RTE_CAST_PTR(void *, txdp), desc0_3); + } + +if (nb_pkts) +ci_vtx1_ctx_avx512 (txdp, *pkt, flags, offload, + single_vlan_pos, qinq_outer_pos, lldp_enabled); +} + +#endif /* __AVX512VL__ */ + +#endif /* _COMMON_INTEL_TX_VEC_X86_H_ */
drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c +7 -278
326 lines not on this page · raw message
drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c +8 -264
309 lines not on this page · raw message
drivers/net/intel/iavf/iavf_rxtx_vec_common.h +4 -73
87 lines not on this page · raw message
Signature · 1 line
2.53.0
RE: [PATCH 06/13] net/intel: move iavf descriptor writing functions to common
Quoted 9 lines
Subject: [PATCH 06/13] net/intel: move iavf descriptor writing functions to common
In order to promote reuse of code, move the Tx vector functions for writing descriptors from the iavf driver to the common folder in a new
tx_vec_x86.hfile. This will allow reuse in later commits by other drivers such as ice.Signed-off-by Bruce Richardson <[email protected]>
Acked-by Ciara Loftus <[email protected]>
Quoted 8 lines
--- drivers/net/intel/common/tx_vec_x86.h | 513 ++++++++++++++++++ drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c | 285 +--------- drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c | 272 +--------- drivers/net/intel/iavf/iavf_rxtx_vec_common.h | 77 +-- 4 files changed, 532 insertions(+), 615 deletions(-) create mode 100644 drivers/net/intel/common/tx_vec_x86.h
PATCH 7/13 net/intel: use function callback for lldp
In reply to Bruce RichardsonLimit LLDP support for iavf drivers for now, by using a callback to handle its processing. Drivers not supporting this pass a NULL pointer and the compiler will optimize away the branches. For iavf, the compiler can inline the subfunction directly as it's known at compile-time.
Signed-off-by Bruce Richardson <[email protected]>
--- drivers/net/intel/common/tx_vec_x86.h | 42 +++++++++++-------- drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c | 16 +++++-- drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c | 17 +++++--- 3 files changed, 48 insertions(+), 27 deletions(-)
drivers/net/intel/common/tx_vec_x86.h +24 -18
@@ -11,6 +11,12 @@#include "tx.h" +/* Optional per-driver LLDP switch-uplink check for ctx descriptors. +* Takes the ctx-desc high qword and returns it with the LLDP bit applied +* if appropriate. NULL disables the check. +*/ +typedef uint64_t (*ci_tx_ctx_lldp_fn )(struct rte_mbuf *pkt,uint64_t high_ctx_qw); + static __rte_always_inline voidci_fill_ctx_desc_tunneling (uint64_t *low_ctx_qw,struct rte_mbuf *pkt) {
@@ -228,7 +234,7 @@ ci_vtx_avx2(volatile struct ci_tx_desc *txdp, static __rte_always_inline voidci_vtx1_ctx_avx2 (volatile struct ci_tx_desc *txdp,struct rte_mbuf *pkt,uint64_t flags,bool offload,enum ci_l2tag_pos single_vlan_pos, -enum ci_l2tag_pos qinq_outer_pos,bool ptype_lldp_enabled) +enum ci_l2tag_pos qinq_outer_pos,ci_tx_ctx_lldp_fn lldp_check) {uint64_t high_ctx_qw =CI_TX_DESC_DTYPE_CTX ;uint64_t low_ctx_qw =0 ;
@@ -247,8 +253,8 @@ ci_vtx1_ctx_avx2(volatile struct ci_tx_desc *txdp, struct rte_mbuf *pkt, low_ctx_qw |= (uint64_t )pkt->vlan_tci <<CI_TXD_CTX_QW0_L2TAG2_S ; } } -if (IAVF_CHECK_TX_LLDP (pkt, ptype_lldp_enabled)) - high_ctx_qw |=IAVF_TX_CTX_DESC_SWTCH_UPLINK <<CI_TXD_QW1_CMD_S ; +if (lldp_check !=NULL ) + high_ctx_qw =lldp_check (pkt, high_ctx_qw);uint64_t high_data_qw = (CI_TX_DESC_DTYPE_DATA | ((uint64_t )flags <<CI_TXD_QW1_CMD_S ) | ((uint64_t )pkt->data_len <<CI_TXD_QW1_TX_BUF_SZ_S ));
@@ -266,7 +272,7 @@ static __rte_always_inline void ci_vtx_ctx_avx2(volatile struct ci_tx_desc *txdp, struct rte_mbuf **pkt, uint16_t nb_pkts, uint64_t flags, bool offload, enum ci_l2tag_pos single_vlan_pos, enum ci_l2tag_pos qinq_outer_pos, - bool ptype_lldp_enabled) + ci_tx_ctx_lldp_fn lldp_check) { uint64_t hi_data_qw_tmpl = (CI_TX_DESC_DTYPE_DATA | (flags << CI_TXD_QW1_CMD_S));
@@ -300,8 +306,8 @@ ci_vtx_ctx_avx2(volatile struct ci_tx_desc *txdp, low_ctx_qw1 |= (uint64_t )pkt[1 ]->vlan_tci <<CI_TXD_CTX_QW0_L2TAG2_S ; } } -if (IAVF_CHECK_TX_LLDP (pkt[1 ], ptype_lldp_enabled)) - hi_ctx_qw1 |=IAVF_TX_CTX_DESC_SWTCH_UPLINK <<CI_TXD_QW1_CMD_S ; +if (lldp_check !=NULL ) + hi_ctx_qw1 =lldp_check (pkt[1 ], hi_ctx_qw1);if (offload) {/* tunnel fill assigns low_ctx_qw0; must run before QinQ/VLAN OR below */
@@ -318,8 +324,8 @@ ci_vtx_ctx_avx2(volatile struct ci_tx_desc *txdp, low_ctx_qw0 |= (uint64_t )pkt[0 ]->vlan_tci <<CI_TXD_CTX_QW0_L2TAG2_S ; } } -if (IAVF_CHECK_TX_LLDP (pkt[0 ], ptype_lldp_enabled)) - hi_ctx_qw0 |=IAVF_TX_CTX_DESC_SWTCH_UPLINK <<CI_TXD_QW1_CMD_S ; +if (lldp_check !=NULL ) + hi_ctx_qw0 =lldp_check (pkt[0 ], hi_ctx_qw0);if (offload) {ci_tx_vec_offload (pkt[1 ], &hi_data_qw1, single_vlan_pos, qinq_outer_pos);
@@ -338,7 +344,7 @@ ci_vtx_ctx_avx2(volatile struct ci_tx_desc *txdp,if (nb_pkts)ci_vtx1_ctx_avx2 (txdp, *pkt, flags, offload, - single_vlan_pos, qinq_outer_pos, ptype_lldp_enabled); + single_vlan_pos, qinq_outer_pos, lldp_check); }#endif /* __AVX2__ */
@@ -398,7 +404,7 @@ ci_vtx_avx512(volatile struct ci_tx_desc *txdp, static __rte_always_inline voidci_vtx1_ctx_avx512 (volatile struct ci_tx_desc *txdp,struct rte_mbuf *pkt,uint64_t flags,bool offload,enum ci_l2tag_pos single_vlan_pos, -enum ci_l2tag_pos qinq_outer_pos,bool lldp_enabled) +enum ci_l2tag_pos qinq_outer_pos,ci_tx_ctx_lldp_fn lldp_check) {uint64_t high_ctx_qw =CI_TX_DESC_DTYPE_CTX ;uint64_t low_ctx_qw =0 ;
@@ -417,8 +423,8 @@ ci_vtx1_ctx_avx512(volatile struct ci_tx_desc *txdp, struct rte_mbuf *pkt, low_ctx_qw |= (uint64_t )pkt->vlan_tci <<CI_TXD_CTX_QW0_L2TAG2_S ; } } -if (IAVF_CHECK_TX_LLDP (pkt, lldp_enabled)) - high_ctx_qw |=IAVF_TX_CTX_DESC_SWTCH_UPLINK <<CI_TXD_QW1_CMD_S ; +if (lldp_check !=NULL ) + high_ctx_qw =lldp_check (pkt, high_ctx_qw);uint64_t high_data_qw = (CI_TX_DESC_DTYPE_DATA | ((uint64_t )flags <<CI_TXD_QW1_CMD_S ) | ((uint64_t )pkt->data_len <<CI_TXD_QW1_TX_BUF_SZ_S ));
@@ -437,7 +443,7 @@ static __rte_always_inline void ci_vtx_ctx_avx512(volatile struct ci_tx_desc *txdp, struct rte_mbuf **pkt, uint16_t nb_pkts, uint64_t flags, bool offload, enum ci_l2tag_pos single_vlan_pos, enum ci_l2tag_pos qinq_outer_pos, - bool lldp_enabled) + ci_tx_ctx_lldp_fn lldp_check) { uint64_t hi_data_qw_tmpl = (CI_TX_DESC_DTYPE_DATA | (flags << CI_TXD_QW1_CMD_S));
@@ -469,8 +475,8 @@ ci_vtx_ctx_avx512(volatile struct ci_tx_desc *txdp, low_ctx_qw1 |= (uint64_t )pkt[1 ]->vlan_tci <<CI_TXD_CTX_QW0_L2TAG2_S ; } } -if (IAVF_CHECK_TX_LLDP (pkt[1 ], lldp_enabled)) - hi_ctx_qw1 |=IAVF_TX_CTX_DESC_SWTCH_UPLINK <<CI_TXD_QW1_CMD_S ; +if (lldp_check !=NULL ) + hi_ctx_qw1 =lldp_check (pkt[1 ], hi_ctx_qw1);if (offload) {/* tunnel fill assigns low_ctx_qw0; must run before QinQ/VLAN OR below */
@@ -487,8 +493,8 @@ ci_vtx_ctx_avx512(volatile struct ci_tx_desc *txdp, low_ctx_qw0 |= (uint64_t )pkt[0 ]->vlan_tci <<CI_TXD_CTX_QW0_L2TAG2_S ; } } -if (IAVF_CHECK_TX_LLDP (pkt[0 ], lldp_enabled)) - hi_ctx_qw0 |=IAVF_TX_CTX_DESC_SWTCH_UPLINK <<CI_TXD_QW1_CMD_S ; +if (lldp_check !=NULL ) + hi_ctx_qw0 =lldp_check (pkt[0 ], hi_ctx_qw0);if (offload) {ci_tx_vec_offload (pkt[1 ], &hi_data_qw1, single_vlan_pos, qinq_outer_pos);
@@ -505,7 +511,7 @@ ci_vtx_ctx_avx512(volatile struct ci_tx_desc *txdp,if (nb_pkts)ci_vtx1_ctx_avx512 (txdp, *pkt, flags, offload, - single_vlan_pos, qinq_outer_pos, lldp_enabled); + single_vlan_pos, qinq_outer_pos, lldp_check); }#endif /* __AVX512VL__ */
drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c +12 -4
@@ -1685,6 +1685,14 @@ iavf_xmit_fixed_burst_vec_avx2(void *tx_queue, struct rte_mbuf **tx_pkts,return nb_pkts; } +static __rte_always_inline uint64_t +iavf_tx_ctx_lldp_check (struct rte_mbuf *pkt,uint64_t high_ctx_qw) +{ +if (IAVF_CHECK_TX_LLDP (pkt, true)) + high_ctx_qw |=IAVF_TX_CTX_DESC_SWTCH_UPLINK <<CI_TXD_QW1_CMD_S ; +return high_ctx_qw; +} + static __rte_always_inline uint16_tiavf_xmit_fixed_burst_vec_avx2_ctx (void *tx_queue,struct rte_mbuf **tx_pkts,uint16_t nb_pkts,bool offload)
@@ -1696,7 +1704,7 @@ iavf_xmit_fixed_burst_vec_avx2_ctx(void *tx_queue, struct rte_mbuf **tx_pkts,/* bit2 is reserved and must be set to 1 according to Spec */ uint64_t flags =IAVF_TX_DESC_CMD_EOP |IAVF_TX_DESC_CMD_ICRC ;uint64_t rs =IAVF_TX_DESC_CMD_RS | flags; -bool lldp_enabled = txq->lldp_enabled; +ci_tx_ctx_lldp_fn lldp_check = txq->lldp_enabled ? iavf_tx_ctx_lldp_check :NULL ;/* vlan_flag gives both the single-VLAN and the QinQ outer tag position */ enum ci_l2tag_pos vlan_pos = (txq->vlan_flag &IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG1 ) ?CI_TAG_IN_DATA_DESC :CI_TAG_IN_CTX_DESC ;
@@ -1723,10 +1731,10 @@ iavf_xmit_fixed_burst_vec_avx2_ctx(void *tx_queue, struct rte_mbuf **tx_pkts,ci_tx_backlog_entry_vec (txep, tx_pkts, nb_mbuf);ci_vtx_ctx_avx2 (txdp, tx_pkts, nb_mbuf -1 , flags, offload, - vlan_pos, vlan_pos, lldp_enabled); + vlan_pos, vlan_pos, lldp_check); tx_pkts += (nb_mbuf -1 ); txdp += (n -2 ); -ci_vtx1_ctx_avx2 (txdp, *tx_pkts++, rs, offload, vlan_pos, vlan_pos, lldp_enabled); +ci_vtx1_ctx_avx2 (txdp, *tx_pkts++, rs, offload, vlan_pos, vlan_pos, lldp_check); nb_commit = (uint16_t )(nb_commit - n);
@@ -1740,7 +1748,7 @@ iavf_xmit_fixed_burst_vec_avx2_ctx(void *tx_queue, struct rte_mbuf **tx_pkts, nb_mbuf = nb_commit >>1 ;ci_tx_backlog_entry_vec (txep, tx_pkts, nb_mbuf); -ci_vtx_ctx_avx2 (txdp, tx_pkts, nb_mbuf, flags, offload, vlan_pos, vlan_pos, lldp_enabled); +ci_vtx_ctx_avx2 (txdp, tx_pkts, nb_mbuf, flags, offload, vlan_pos, vlan_pos, lldp_check); tx_id = (uint16_t )(tx_id + nb_commit);if (tx_id > txq->tx_next_rs) {
drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c +12 -5
@@ -1900,6 +1900,14 @@ iavf_xmit_fixed_burst_vec_avx512(void *tx_queue, struct rte_mbuf **tx_pkts,return nb_pkts; } +static __rte_always_inline uint64_t +iavf_tx_ctx_lldp_check (struct rte_mbuf *pkt,uint64_t high_ctx_qw) +{ +if (IAVF_CHECK_TX_LLDP (pkt, true)) + high_ctx_qw |=IAVF_TX_CTX_DESC_SWTCH_UPLINK <<CI_TXD_QW1_CMD_S ; +return high_ctx_qw; +} + static __rte_always_inline uint16_tiavf_xmit_fixed_burst_vec_avx512_ctx (void *tx_queue,struct rte_mbuf **tx_pkts,uint16_t nb_pkts,bool offload)
@@ -1911,7 +1919,7 @@ iavf_xmit_fixed_burst_vec_avx512_ctx(void *tx_queue, struct rte_mbuf **tx_pkts,/* bit2 is reserved and must be set to 1 according to Spec */ uint64_t flags =CI_TX_DESC_CMD_EOP |CI_TX_DESC_CMD_ICRC ;uint64_t rs =CI_TX_DESC_CMD_RS | flags; -bool lldp_enabled = txq->lldp_enabled; +ci_tx_ctx_lldp_fn lldp_check = txq->lldp_enabled ? iavf_tx_ctx_lldp_check :NULL ;/* vlan_flag gives both the single-VLAN and the QinQ outer tag position */ enum ci_l2tag_pos vlan_pos = (txq->vlan_flag &IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG1 ) ?CI_TAG_IN_DATA_DESC :CI_TAG_IN_CTX_DESC ;
@@ -1938,11 +1946,10 @@ iavf_xmit_fixed_burst_vec_avx512_ctx(void *tx_queue, struct rte_mbuf **tx_pkts,tx_backlog_entry_avx512 (txep, tx_pkts, nb_mbuf);ci_vtx_ctx_avx512 (txdp, tx_pkts, nb_mbuf -1 , flags, offload, - vlan_pos, vlan_pos, lldp_enabled); + vlan_pos, vlan_pos, lldp_check); tx_pkts += (nb_mbuf -1 ); txdp += (n -2 ); -ci_vtx1_ctx_avx512 (txdp, *tx_pkts++, rs, offload, - vlan_pos, vlan_pos, lldp_enabled); +ci_vtx1_ctx_avx512 (txdp, *tx_pkts++, rs, offload, vlan_pos, vlan_pos, lldp_check); nb_commit = (uint16_t )(nb_commit - n);
@@ -1956,7 +1963,7 @@ iavf_xmit_fixed_burst_vec_avx512_ctx(void *tx_queue, struct rte_mbuf **tx_pkts, nb_mbuf = nb_commit >>1 ;tx_backlog_entry_avx512 (txep, tx_pkts, nb_mbuf); -ci_vtx_ctx_avx512 (txdp, tx_pkts, nb_mbuf, flags, offload, vlan_pos, vlan_pos, lldp_enabled); +ci_vtx_ctx_avx512 (txdp, tx_pkts, nb_mbuf, flags, offload, vlan_pos, vlan_pos, lldp_check); tx_id = (uint16_t )(tx_id + nb_commit);if (tx_id > txq->tx_next_rs) {
Signature · 1 line
2.53.0
PATCH 8/13 net/ice: use common descriptor creation functions
In reply to Bruce RichardsonUpdate ice driver to use the descriptor creation functions from common rather than having it maintain its own copies of them.
Signed-off-by Bruce Richardson <[email protected]>
--- drivers/net/intel/ice/ice_rxtx_vec_avx2.c | 73 ++------------------- drivers/net/intel/ice/ice_rxtx_vec_avx512.c | 64 ++---------------- drivers/net/intel/ice/ice_rxtx_vec_common.h | 59 ----------------- 3 files changed, 12 insertions(+), 184 deletions(-)
drivers/net/intel/ice/ice_rxtx_vec_avx2.c +6 -67
@@ -5,6 +5,7 @@#include "ice_rxtx_vec_common.h" #include "../common/rx_vec_x86.h" +#include "../common/tx_vec_x86.h" #include <rte_vect.h>
@@ -773,70 +774,6 @@ ice_recv_scattered_pkts_vec_avx2_offload(void *rx_queue, true); } -static __rte_always_inline void -ice_vtx1 (volatile struct ci_tx_desc *txdp, -struct rte_mbuf *pkt,uint64_t flags,bool offload) -{ -uint64_t high_qw = (CI_TX_DESC_DTYPE_DATA | - ((uint64_t )flags <<CI_TXD_QW1_CMD_S ) | - ((uint64_t )pkt->data_len <<CI_TXD_QW1_TX_BUF_SZ_S )); -if (offload) -ice_txd_enable_offload (pkt, &high_qw); - -__m128i descriptor =_mm_set_epi64x (high_qw,rte_pktmbuf_iova (pkt)); - _mm_store_si128(RTE_CAST_PTR(__m128i *, txdp), descriptor); -} - -static __rte_always_inline void -ice_vtx (volatile struct ci_tx_desc *txdp, -struct rte_mbuf **pkt,uint16_t nb_pkts,uint64_t flags,bool offload) -{ -const uint64_t hi_qw_tmpl = (CI_TX_DESC_DTYPE_DATA | (flags <<CI_TXD_QW1_CMD_S )); - -/* if unaligned on 32-bit boundary, do one to align */ -if (((uintptr_t )txdp &0x1F ) !=0 && nb_pkts !=0 ) { -ice_vtx1 (txdp, *pkt, flags, offload); - nb_pkts--; txdp++; pkt++; - } - -/* do four at a time while possible, in bursts */ -for (; nb_pkts >3 ; txdp +=4 , pkt +=4 , nb_pkts -=4 ) { -uint64_t hi_qw3 = hi_qw_tmpl | - ((uint64_t )pkt[3 ]->data_len <<CI_TXD_QW1_TX_BUF_SZ_S ); -if (offload) -ice_txd_enable_offload (pkt[3 ], &hi_qw3); -uint64_t hi_qw2 = hi_qw_tmpl | - ((uint64_t )pkt[2 ]->data_len <<CI_TXD_QW1_TX_BUF_SZ_S ); -if (offload) -ice_txd_enable_offload (pkt[2 ], &hi_qw2); -uint64_t hi_qw1 = hi_qw_tmpl | - ((uint64_t )pkt[1 ]->data_len <<CI_TXD_QW1_TX_BUF_SZ_S ); -if (offload) -ice_txd_enable_offload (pkt[1 ], &hi_qw1); -uint64_t hi_qw0 = hi_qw_tmpl | - ((uint64_t )pkt[0 ]->data_len <<CI_TXD_QW1_TX_BUF_SZ_S ); -if (offload) -ice_txd_enable_offload (pkt[0 ], &hi_qw0); - -__m256i desc2_3 = -_mm256_set_epi64x - (hi_qw3,rte_pktmbuf_iova (pkt[3 ]), - hi_qw2,rte_pktmbuf_iova (pkt[2 ])); -__m256i desc0_1 = -_mm256_set_epi64x - (hi_qw1,rte_pktmbuf_iova (pkt[1 ]), - hi_qw0,rte_pktmbuf_iova (pkt[0 ])); - _mm256_store_si256(RTE_CAST_PTR(__m256i *, txdp + 2), desc2_3); - _mm256_store_si256(RTE_CAST_PTR(__m256i *, txdp), desc0_1); - } - -/* do any last ones */ -while (nb_pkts) { -ice_vtx1 (txdp, *pkt, flags, offload); - txdp++; pkt++; nb_pkts--; - } -} - static __rte_always_inline uint16_tice_xmit_fixed_burst_vec_avx2 (void *tx_queue,struct rte_mbuf **tx_pkts,uint16_t nb_pkts,bool offload)
@@ -868,11 +805,12 @@ ice_xmit_fixed_burst_vec_avx2(void *tx_queue, struct rte_mbuf **tx_pkts,if (nb_commit >= n) {ci_tx_backlog_entry_vec (txep, tx_pkts, n); -ice_vtx (txdp, tx_pkts, n -1 , flags, offload); +ci_vtx_avx2 (txdp, tx_pkts, n -1 , flags, offload, +CI_TAG_IN_DATA_DESC ,CI_TAG_IN_DATA_DESC ); tx_pkts += (n -1 ); txdp += (n -1 ); -ice_vtx1 (txdp, *tx_pkts++, rs, offload); +ci_vtx1 (txdp, *tx_pkts++, rs, offload,CI_TAG_IN_DATA_DESC ,CI_TAG_IN_DATA_DESC ); nb_commit = (uint16_t )(nb_commit - n);
@@ -886,7 +824,8 @@ ice_xmit_fixed_burst_vec_avx2(void *tx_queue, struct rte_mbuf **tx_pkts,ci_tx_backlog_entry_vec (txep, tx_pkts, nb_commit); -ice_vtx (txdp, tx_pkts, nb_commit, flags, offload); +ci_vtx_avx2 (txdp, tx_pkts, nb_commit, flags, offload, +CI_TAG_IN_DATA_DESC ,CI_TAG_IN_DATA_DESC ); tx_id = (uint16_t )(tx_id + nb_commit);if (tx_id > txq->tx_next_rs) {
drivers/net/intel/ice/ice_rxtx_vec_avx512.c +6 -58
91 lines not on this page · raw message
drivers/net/intel/ice/ice_rxtx_vec_common.h +0 -59
63 lines not on this page · raw message
Signature · 1 line
2.53.0
RE: [PATCH 08/13] net/ice: use common descriptor creation functions
Quoted 32 lines
Subject: [PATCH 08/13] net/ice: use common descriptor creation functions
Update ice driver to use the descriptor creation functions from common rather than having it maintain its own copies of them.
Signed-off-by Bruce Richardson <[email protected]>--- drivers/net/intel/ice/ice_rxtx_vec_avx2.c | 73 ++------------------- drivers/net/intel/ice/ice_rxtx_vec_avx512.c | 64 ++---------------- drivers/net/intel/ice/ice_rxtx_vec_common.h | 59 ----------------- 3 files changed, 12 insertions(+), 184 deletions(-)drivers/net/intel/ice/ice_rxtx_vec_avx2.c +1
-2@@ -5,6 +5,7 @@#include "ice_rxtx_vec_common.h" #include "../common/rx_vec_x86.h" +#include "../common/tx_vec_x86.h" #include <rte_vect.h> @@ -773,70 +774,6 @@ ice_recv_scattered_pkts_vec_avx2_offload(void *rx_queue, true); } -static __rte_always_inline void -ice_vtx1(volatile struct ci_tx_desc *txdp,
drivers/net/intel/ice/ice_rxtx_vec_avx2.c ice_recv_scattered_pkts_vec_avx2_offload(void · line 778
Quoted 10 lines
drivers/net/intel/ice/ice_rxtx_vec_avx2.c +0
-9- struct rte_mbuf *pkt, uint64_t flags, bool offload) -{ - uint64_t high_qw = (CI_TX_DESC_DTYPE_DATA | - ((uint64_t )flags <<CI_TXD_QW1_CMD_S ) | - ((uint64_t )pkt->data_len <<CI_TXD_QW1_TX_BUF_SZ_S )); -if (offload) -ice_txd_enable_offload (pkt, &high_qw); - -__m128i descriptor =_mm_set_epi64x (high_qw, rte_pktmbuf_iova(pkt));
AI caught this for me on review, and I was able to reproduce it:
When you build with -Denable_iova_as_pa=false the ice driver build will fail. "require_iova_in_mbuf = false" is set in the ice meson.build file. So either that needs to be flipped to true, or ice needs to keep using rte_pktmbuf_iova.
Quoted 7 lines
drivers/net/intel/ice/ice_rxtx_vec_avx2.c +0
-7- _mm_store_si128(RTE_CAST_PTR(__m128i *, txdp), descriptor); -} - -static __rte_always_inline void -ice_vtx (volatile struct ci_tx_desc *txdp, -struct rte_mbuf **pkt,uint16_t nb_pkts,uint64_t flags,bool offload) -{
Re: [PATCH 08/13] net/ice: use common descriptor creation functions
Quoted 39 lines · Ciara
Quoted 39 lines
Subject: [PATCH 08/13] net/ice: use common descriptor creation functions
Update ice driver to use the descriptor creation functions from common rather than having it maintain its own copies of them.
Signed-off-by Bruce Richardson <[email protected]>--- drivers/net/intel/ice/ice_rxtx_vec_avx2.c | 73 ++------------------- drivers/net/intel/ice/ice_rxtx_vec_avx512.c | 64 ++---------------- drivers/net/intel/ice/ice_rxtx_vec_common.h | 59 ----------------- 3 files changed, 12 insertions(+), 184 deletions(-)drivers/net/intel/ice/ice_rxtx_vec_avx2.c +1
-9@@ -5,6 +5,7 @@#include "ice_rxtx_vec_common.h" #include "../common/rx_vec_x86.h" +#include "../common/tx_vec_x86.h" #include <rte_vect.h> @@ -773,70 +774,6 @@ ice_recv_scattered_pkts_vec_avx2_offload(void *rx_queue, true); } -static __rte_always_inline void -ice_vtx1 (volatile struct ci_tx_desc *txdp, -struct rte_mbuf *pkt,uint64_t flags,bool offload) -{ -uint64_t high_qw = (CI_TX_DESC_DTYPE_DATA | - ((uint64_t )flags <<CI_TXD_QW1_CMD_S ) | - ((uint64_t )pkt->data_len <<CI_TXD_QW1_TX_BUF_SZ_S )); -if (offload) -ice_txd_enable_offload (pkt, &high_qw);
Quoted 9 lines
drivers/net/intel/ice/ice_rxtx_vec_avx2.c ice_recv_scattered_pkts_vec_avx2_offload(void · line 778
Quoted 3 lines
drivers/net/intel/ice/ice_rxtx_vec_avx2.c +0
-2- -__m128i descriptor =_mm_set_epi64x (high_qw, rte_pktmbuf_iova(pkt));AI caught this for me on review, and I was able to reproduce it:
When you build with -Denable_iova_as_pa=false the ice driver build will fail. "require_iova_in_mbuf= false" is set in the ice meson.build file. So either that needs to be flipped to true, or ice needs to keep usingrte_pktmbuf_iova.
Good catch by the AI. Changing the code to use the proper macro not only ensure ice isn't broken, but also allows us to enable iavf builds when the iova-in-mbuf setting is disabled. Will fix in next version.
/Bruce
PATCH 9/13 net/intel: move vector Tx paths to common
In reply to Bruce RichardsonThe main loops for vector Tx, both avx2 and avx512 are almost identical across a couple of drivers, so move the iavf copies to common for later reuse by other drivers.
Signed-off-by Bruce Richardson <[email protected]>
--- drivers/net/intel/common/tx.h | 10 ++ drivers/net/intel/common/tx_vec_x86.h | 128 ++++++++++++++++++ drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c | 75 +--------- drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c | 77 +---------- 4 files changed, 148 insertions(+), 142 deletions(-)
drivers/net/intel/common/tx.h +10 -0
@@ -9,6 +9,8 @@#include <rte_mbuf.h> #include <rte_ethdev.h> #include <rte_vect.h> +#include <rte_io.h> +#include <rte_byteorder.h> /* Common TX Descriptor QW1 Field Definitions */ #define CI_TXD_QW1_DTYPE_S 0
@@ -273,6 +275,14 @@ ci_tx_backlog_entry_vec(struct ci_tx_entry_vec *txep, struct rte_mbuf **tx_pkts, txep[i].mbuf = tx_pkts[i]; } + +/* Write the Tx tail register, byte-swapped for hardware regardless of host endianness. */ +static __rte_always_inline void +ci_tx_qtx_tail_write (struct ci_tx_queue *txq,uint16_t tx_id) +{ +rte_write32_wc (rte_cpu_to_le_32 ((uint32_t )tx_id), txq->qtx_tail); +} +#define IETH_VPMD_TX_MAX_FREE_BUF 64typedef int (*ci_desc_done_fn )(struct ci_tx_queue *txq,uint16_t idx);
drivers/net/intel/common/tx_vec_x86.h +128 -0
140 lines not on this page · raw message
drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c +5 -70
93 lines not on this page · raw message
drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c +5 -72
95 lines not on this page · raw message
Signature · 1 line
2.53.0
RE: [PATCH 09/13] net/intel: move vector Tx paths to common
Quoted 7 lines
Subject: [PATCH 09/13] net/intel: move vector Tx paths to common
The main loops for vector Tx, both avx2 and avx512 are almost identical across a couple of drivers, so move the iavf copies to common for later reuse by other drivers.
Signed-off-by Bruce Richardson <[email protected]>
Acked-by Ciara Loftus <[email protected]>
Quoted 7 lines
--- drivers/net/intel/common/tx.h | 10 ++ drivers/net/intel/common/tx_vec_x86.h | 128 ++++++++++++++++++ drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c | 75 +--------- drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c | 77 +---------- 4 files changed, 148 insertions(+), 142 deletions(-)
PATCH 10/13 net/ice: use common AVX Tx functions
In reply to Bruce RichardsonUse the newly created common Tx functions in the vector path.
Signed-off-by Bruce Richardson <[email protected]>
--- drivers/net/intel/ice/ice_rxtx_vec_avx2.c | 72 +------------- drivers/net/intel/ice/ice_rxtx_vec_avx512.c | 102 +++----------------- 2 files changed, 14 insertions(+), 160 deletions(-)
drivers/net/intel/ice/ice_rxtx_vec_avx2.c +2 -70
@@ -774,74 +774,6 @@ ice_recv_scattered_pkts_vec_avx2_offload(void *rx_queue, true); } -static __rte_always_inline uint16_t -ice_xmit_fixed_burst_vec_avx2 (void *tx_queue,struct rte_mbuf **tx_pkts, -uint16_t nb_pkts,bool offload) -{ -struct ci_tx_queue *txq = (struct ci_tx_queue *)tx_queue; -volatile struct ci_tx_desc *txdp; -struct ci_tx_entry_vec *txep; -uint16_t n, nb_commit, tx_id; -uint64_t flags =CI_TX_DESC_CMD_DEFAULT ; -uint64_t rs =CI_TX_DESC_CMD_RS |CI_TX_DESC_CMD_DEFAULT ; - -/* cross rx_thresh boundary is not allowed */ - nb_pkts =RTE_MIN (nb_pkts, txq->tx_rs_thresh); - -if (txq->nb_tx_free < txq->tx_free_thresh) -ci_tx_free_bufs_vec (txq, ice_tx_desc_done, false); - - nb_commit = nb_pkts = (uint16_t )RTE_MIN (txq->nb_tx_free, nb_pkts); -if (unlikely (nb_pkts ==0 )) -return 0 ; - - tx_id = txq->tx_tail; - txdp = &txq->ci_tx_ring[tx_id]; - txep = &txq->sw_ring_vec[tx_id]; - - txq->nb_tx_free = (uint16_t )(txq->nb_tx_free - nb_pkts); - - n = (uint16_t )(txq->nb_tx_desc - tx_id); -if (nb_commit >= n) { -ci_tx_backlog_entry_vec (txep, tx_pkts, n); - -ci_vtx_avx2 (txdp, tx_pkts, n -1 , flags, offload, -CI_TAG_IN_DATA_DESC ,CI_TAG_IN_DATA_DESC ); - tx_pkts += (n -1 ); - txdp += (n -1 ); - -ci_vtx1 (txdp, *tx_pkts++, rs, offload,CI_TAG_IN_DATA_DESC ,CI_TAG_IN_DATA_DESC ); - - nb_commit = (uint16_t )(nb_commit - n); - - tx_id =0 ; - txq->tx_next_rs = (uint16_t )(txq->tx_rs_thresh -1 ); - -/* avoid reach the end of ring */ - txdp = &txq->ci_tx_ring[tx_id]; - txep = &txq->sw_ring_vec[tx_id]; - } - -ci_tx_backlog_entry_vec (txep, tx_pkts, nb_commit); - -ci_vtx_avx2 (txdp, tx_pkts, nb_commit, flags, offload, -CI_TAG_IN_DATA_DESC ,CI_TAG_IN_DATA_DESC ); - - tx_id = (uint16_t )(tx_id + nb_commit); -if (tx_id > txq->tx_next_rs) { - txq->ci_tx_ring[txq->tx_next_rs].cmd_type_offset_bsz |= -rte_cpu_to_le_64 (((uint64_t )CI_TX_DESC_CMD_RS ) <<CI_TXD_QW1_CMD_S ); - txq->tx_next_rs = - (uint16_t )(txq->tx_next_rs + txq->tx_rs_thresh); - } - - txq->tx_tail = tx_id; - -ICE_PCI_REG_WC_WRITE (txq->qtx_tail, txq->tx_tail); - -return nb_pkts; -} - static __rte_always_inline uint16_tice_xmit_pkts_vec_avx2_common (void *tx_queue,struct rte_mbuf **tx_pkts,uint16_t nb_pkts,bool offload)
@@ -853,8 +785,8 @@ ice_xmit_pkts_vec_avx2_common(void *tx_queue, struct rte_mbuf **tx_pkts,uint16_t ret, num; num = (uint16_t )RTE_MIN (nb_pkts, txq->tx_rs_thresh); - ret =ice_xmit_fixed_burst_vec_avx2 (tx_queue, &tx_pkts[nb_tx], - num, offload); + ret =ci_xmit_fixed_burst_vec_avx2 (tx_queue, &tx_pkts[nb_tx], num, + offload,CI_TAG_IN_DATA_DESC ,CI_TAG_IN_DATA_DESC ); nb_tx += ret; nb_pkts -= ret;if (ret < num)
drivers/net/intel/ice/ice_rxtx_vec_avx512.c +12 -90
122 lines not on this page · raw message
Signature · 1 line
2.53.0
RE: [PATCH 10/13] net/ice: use common AVX Tx functions
Quoted 5 lines
Subject: [PATCH 10/13] net/ice: use common AVX Tx functions
Use the newly created common Tx functions in the vector path.
Signed-off-by Bruce Richardson <[email protected]>
Acked-by Ciara Loftus <[email protected]>
Quoted 5 lines
--- drivers/net/intel/ice/ice_rxtx_vec_avx2.c | 72 +------------- drivers/net/intel/ice/ice_rxtx_vec_avx512.c | 102 +++----------------- 2 files changed, 14 insertions(+), 160 deletions(-)
PATCH 11/13 net/intel: add common vector Tx fns with context handling
In reply to Bruce RichardsonThe iavf vector driver has support for more Tx offloads because it has Tx functions which handle transmit context descriptors. Move those context-handling functions to the common folder for reuse by other drivers.
Signed-off-by Bruce Richardson <[email protected]>
--- drivers/net/intel/common/tx_vec_x86.h | 136 ++++++++++++++++++ drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c | 78 +--------- drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c | 89 +----------- 3 files changed, 147 insertions(+), 156 deletions(-)
drivers/net/intel/common/tx_vec_x86.h +136 -0
@@ -410,6 +410,74 @@ ci_xmit_fixed_burst_vec_avx2(struct ci_tx_queue *txq, struct rte_mbuf **tx_pkts,return nb_pkts; } +static __rte_always_inline uint16_t +ci_xmit_fixed_burst_vec_ctx_avx2 (struct ci_tx_queue *txq,struct rte_mbuf **tx_pkts, +uint16_t nb_pkts,bool offload, +enum ci_l2tag_pos single_vlan_pos,enum ci_l2tag_pos qinq_outer_pos, +ci_tx_ctx_lldp_fn lldp_check) +{ +volatile struct ci_tx_desc *txdp; +struct ci_tx_entry_vec *txep; +uint16_t n, nb_commit, nb_mbuf, tx_id; +uint64_t flags =CI_TX_DESC_CMD_DEFAULT ; +uint64_t rs =CI_TX_DESC_CMD_RS | flags; + +if (txq->nb_tx_free < txq->tx_free_thresh) +ci_tx_free_bufs_vec (txq, ci_tx_desc_done_simple, true); + + nb_commit = (uint16_t )RTE_MIN (txq->nb_tx_free, nb_pkts <<1 ); + nb_commit &=0xFFFE ; +if (unlikely (nb_commit ==0 )) +return 0 ; + + nb_pkts = nb_commit >>1 ; + tx_id = txq->tx_tail; + txdp = &txq->ci_tx_ring[tx_id]; + txep = (void *)txq->sw_ring; + txep += (tx_id >>1 ); + + txq->nb_tx_free = (uint16_t )(txq->nb_tx_free - nb_commit); + n = (uint16_t )(txq->nb_tx_desc - tx_id); + +if (n !=0 && nb_commit >= n) { + nb_mbuf = n >>1 ; +ci_tx_backlog_entry_vec (txep, tx_pkts, nb_mbuf); + +ci_vtx_ctx_avx2 (txdp, tx_pkts, nb_mbuf -1 , flags, offload, + single_vlan_pos, qinq_outer_pos, lldp_check); + tx_pkts += (nb_mbuf -1 ); + txdp += (n -2 ); +ci_vtx1_ctx_avx2 (txdp, *tx_pkts++, rs, offload, + single_vlan_pos, qinq_outer_pos, lldp_check); + + nb_commit = (uint16_t )(nb_commit - n); + + txq->tx_next_rs = (uint16_t )(txq->tx_rs_thresh -1 ); + tx_id =0 ; +/* avoid reach the end of ring */ + txdp = txq->ci_tx_ring; + txep = (void *)txq->sw_ring; + } + + nb_mbuf = nb_commit >>1 ; +ci_tx_backlog_entry_vec (txep, tx_pkts, nb_mbuf); + +ci_vtx_ctx_avx2 (txdp, tx_pkts, nb_mbuf, flags, offload, + single_vlan_pos, qinq_outer_pos, lldp_check); + tx_id = (uint16_t )(tx_id + nb_commit); + +if (tx_id > txq->tx_next_rs) { + txq->ci_tx_ring[txq->tx_next_rs].cmd_type_offset_bsz |= +rte_cpu_to_le_64 (((uint64_t )CI_TX_DESC_CMD_RS ) <<CI_TXD_QW1_CMD_S ); + txq->tx_next_rs = (uint16_t )(txq->tx_next_rs + txq->tx_rs_thresh); + } + + txq->tx_tail = tx_id; + +ci_tx_qtx_tail_write (txq, tx_id); +return nb_pkts; +} +#endif /* __AVX2__ */ #ifdef __AVX512VL__
@@ -642,6 +710,74 @@ ci_xmit_fixed_burst_vec_avx512(struct ci_tx_queue *txq, struct rte_mbuf **tx_pktreturn nb_pkts; } +static __rte_always_inline uint16_t +ci_xmit_fixed_burst_vec_ctx_avx512 (struct ci_tx_queue *txq,struct rte_mbuf **tx_pkts, +uint16_t nb_pkts,bool offload, +enum ci_l2tag_pos single_vlan_pos,enum ci_l2tag_pos qinq_outer_pos, +ci_tx_ctx_lldp_fn lldp_check) +{ +volatile struct ci_tx_desc *txdp; +struct ci_tx_entry_vec *txep; +uint16_t n, nb_commit, nb_mbuf, tx_id; +uint64_t flags =CI_TX_DESC_CMD_DEFAULT ; +uint64_t rs =CI_TX_DESC_CMD_RS | flags; + +if (txq->nb_tx_free < txq->tx_free_thresh) +ci_tx_free_bufs_vec (txq, ci_tx_desc_done_simple, true); + + nb_commit = (uint16_t )RTE_MIN (txq->nb_tx_free, nb_pkts <<1 ); + nb_commit &=0xFFFE ; +if (unlikely (nb_commit ==0 )) +return 0 ; + + nb_pkts = nb_commit >>1 ; + tx_id = txq->tx_tail; + txdp = &txq->ci_tx_ring[tx_id]; + txep = (void *)txq->sw_ring; + txep += (tx_id >>1 ); + + txq->nb_tx_free = (uint16_t )(txq->nb_tx_free - nb_commit); + n = (uint16_t )(txq->nb_tx_desc - tx_id); + +if (n !=0 && nb_commit >= n) { + nb_mbuf = n >>1 ; +ci_tx_backlog_entry_vec (txep, tx_pkts, nb_mbuf); + +ci_vtx_ctx_avx512 (txdp, tx_pkts, nb_mbuf -1 , flags, offload, + single_vlan_pos, qinq_outer_pos, lldp_check); + tx_pkts += (nb_mbuf -1 ); + txdp += (n -2 ); +ci_vtx1_ctx_avx512 (txdp, *tx_pkts++, rs, offload, + single_vlan_pos, qinq_outer_pos, lldp_check); + + nb_commit = (uint16_t )(nb_commit - n); + + txq->tx_next_rs = (uint16_t )(txq->tx_rs_thresh -1 ); + tx_id =0 ; +/* avoid reach the end of ring */ + txdp = txq->ci_tx_ring; + txep = (void *)txq->sw_ring; + } + + nb_mbuf = nb_commit >>1 ; +ci_tx_backlog_entry_vec (txep, tx_pkts, nb_mbuf); + +ci_vtx_ctx_avx512 (txdp, tx_pkts, nb_mbuf, flags, offload, + single_vlan_pos, qinq_outer_pos, lldp_check); + tx_id = (uint16_t )(tx_id + nb_commit); + +if (tx_id > txq->tx_next_rs) { + txq->ci_tx_ring[txq->tx_next_rs].cmd_type_offset_bsz |= +rte_cpu_to_le_64 (((uint64_t )CI_TX_DESC_CMD_RS ) <<CI_TXD_QW1_CMD_S ); + txq->tx_next_rs = (uint16_t )(txq->tx_next_rs + txq->tx_rs_thresh); + } + + txq->tx_tail = tx_id; + +ci_tx_qtx_tail_write (txq, tx_id); +return nb_pkts; +} +#endif /* __AVX512VL__ */ #endif /* _COMMON_INTEL_TX_VEC_X86_H_ */
drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c +5 -73
97 lines not on this page · raw message
drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c +6 -83
113 lines not on this page · raw message
Signature · 1 line
2.53.0
RE: [PATCH 11/13] net/intel: add common vector Tx fns with context handling
Quoted 9 lines
Subject: [PATCH 11/13] net/intel: add common vector Tx fns with context handling
The iavf vector driver has support for more Tx offloads because it has Tx functions which handle transmit context descriptors. Move those context-handling functions to the common folder for reuse by other drivers.
Signed-off-by Bruce Richardson <[email protected]>
Acked-by Ciara Loftus <[email protected]>
Quoted 6 lines
--- drivers/net/intel/common/tx_vec_x86.h | 136 ++++++++++++++++++ drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c | 78 +--------- drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c | 89 +----------- 3 files changed, 147 insertions(+), 156 deletions(-)
PATCH 12/13 net/intel: improve Tx path selection logic
In reply to Bruce RichardsonRework the relevant priority of context descriptor vs SIMD width when selecting a Tx path so that presence of context descriptors is only considered when all else is equal.
Normally, path select takes place without having to worry about context descriptors, since offloads requiring context descriptors are specified using the offloads bitmap for each Tx path. However, in some cases, the need to use a context descriptor is not encodable via this offload path, e.g. VLAN offload for iavf can sometimes use the data descriptor and sometimes the context depending on what PF specifies at runtime. To account for those cases, we separate out the use of contexts from the path features, but instead note it as part of the path information, and add an additional parameter to the Tx path selection for drivers to mandate a context descriptor be present. Otherwise, ctx is largely ignored in path selection, except as a last-resort tie-breaker between two paths.
Signed-off-by Bruce Richardson <[email protected]>
--- drivers/net/intel/common/tx.h | 41 +++++++++++++++++--------- drivers/net/intel/cpfl/cpfl_rxtx.c | 3 +- drivers/net/intel/i40e/i40e_rxtx.c | 2 +- drivers/net/intel/iavf/iavf_rxtx.c | 46 ++++++++++++++++-------------- drivers/net/intel/ice/ice_rxtx.c | 3 +- drivers/net/intel/idpf/idpf_rxtx.c | 3 +- 6 files changed, 58 insertions(+), 40 deletions(-)
drivers/net/intel/common/tx.h +27 -14
@@ -249,7 +249,6 @@ struct ci_tx_path_features {uint32_t tx_offloads;enum rte_vect_max_simd simd_width;bool simple_tx; -bool ctx_desc;bool disabled;bool single_queue; };
@@ -259,6 +258,7 @@ struct ci_tx_path_info {const char *info;struct ci_tx_path_features features;eth_tx_prep_t pkt_prep; +bool supports_ctx; };static __rte_always_inline void
@@ -411,6 +411,11 @@ ci_txq_release_all_mbufs(struct ci_tx_queue *txq, bool use_ctx)* Number of available paths in the infos array * @param default_path * Index of the default path to use if no suitable path is found +* @param force_ctx +* If true, only paths that support context descriptors may be selected. +* Use this for offloads that require a context descriptor but cannot be +* discovered purely from req_features->tx_offloads (e.g. a driver-specific +* devarg, or a runtime/hardware-negotiated tag placement). * * @return * The packet burst function index that best matches the requested features,
@@ -420,10 +425,12 @@ static inline int ci_tx_path_select(const struct ci_tx_path_features *req_features, const struct ci_tx_path_info *infos, size_t num_paths, - int default_path) + int default_path, + bool force_ctx) { int idx = default_path;const struct ci_tx_path_features *chosen_path_features =NULL ; +bool chosen_supports_ctx = false;for (unsigned int i =0 ; i < num_paths; i++) {const struct ci_tx_path_features *path_features = &infos[i].features;
@@ -440,8 +447,8 @@ ci_tx_path_select(const struct ci_tx_path_features *req_features,if (path_features->simple_tx && !req_features->simple_tx)continue ; -/* If a context descriptor is requested, ensure the path supports it. */ -if (!path_features->ctx_desc && req_features->ctx_desc) +/* If a context descriptor is required, ensure the path supports it. */ +if (!infos[i].supports_ctx && force_ctx)continue ;/* If requested, ensure the path supports single queue TX. */
@@ -462,22 +469,28 @@ ci_tx_path_select(const struct ci_tx_path_features *req_features,/* Do not select paths with lower SIMD width than the chosen path. */ if (path_features->simd_width < chosen_path_features->simd_width)continue ; -/* Do not select paths with more offloads enabled than the chosen path if -* the SIMD widths are the same. +/* The following tie-breaks only matter when SIMD widths are tied; +* a strictly wider path is always preferred regardless of offload +* count or ctx-descriptor use. */ -if (path_features->simd_width == chosen_path_features->simd_width && -rte_popcount32 (path_features->tx_offloads) > -rte_popcount32 (chosen_path_features->tx_offloads)) -continue ; - -/* Don't use a context descriptor unless necessary */ -if (path_features->ctx_desc && !chosen_path_features->ctx_desc) -continue ; +if (path_features->simd_width == chosen_path_features->simd_width) { +/* Do not select paths with more offloads enabled than the +* chosen path. +*/ +if (rte_popcount32 (path_features->tx_offloads) > +rte_popcount32 (chosen_path_features->tx_offloads)) +continue ; + +/* Don't use a context descriptor unless necessary */ +if (infos[i].supports_ctx && !chosen_supports_ctx) +continue ; + } }/* Finally, select the path since it has met all the requirements. */ idx = i; chosen_path_features = &infos[idx].features; + chosen_supports_ctx = infos[idx].supports_ctx; }return idx;
drivers/net/intel/cpfl/cpfl_rxtx.c +2 -1
@@ -1542,7 +1542,8 @@ cpfl_set_tx_function(struct rte_eth_dev *dev) ad->tx_func_type =ci_tx_path_select (&req_features, &idpf_tx_path_infos[0 ],IDPF_TX_MAX , -IDPF_TX_DEFAULT ); +IDPF_TX_DEFAULT , + false);/* Set use_vec_entry for single queue mode - only IDPF_TX_SINGLEQ uses regular entries */ if (vport->txq_model ==VIRTCHNL2_QUEUE_MODEL_SINGLE ) {
drivers/net/intel/i40e/i40e_rxtx.c +1 -1
@@ -3120,7 +3120,7 @@ i40e_set_tx_function(struct rte_eth_dev *dev) } ad->tx_func_type =ci_tx_path_select (&req_features, &i40e_tx_path_infos[0 ], -RTE_DIM (i40e_tx_path_infos),I40E_TX_DEFAULT ); +RTE_DIM (i40e_tx_path_infos),I40E_TX_DEFAULT , false); out: dev->tx_pkt_burst = mbuf_check ? i40e_xmit_pkts_check :
drivers/net/intel/iavf/iavf_rxtx.c +24 -22
112 lines not on this page · raw message
drivers/net/intel/ice/ice_rxtx.c +2 -1
@@ -3777,7 +3777,8 @@ ice_set_tx_function(struct rte_eth_dev *dev) ad->tx_func_type =ci_tx_path_select (&req_features, &ice_tx_path_infos[0 ],RTE_DIM (ice_tx_path_infos), -ICE_TX_DEFAULT ); +ICE_TX_DEFAULT , + false); out:#if defined(RTE_ARCH_X86 )
drivers/net/intel/idpf/idpf_rxtx.c +2 -1
@@ -887,7 +887,8 @@ idpf_set_tx_function(struct rte_eth_dev *dev) ad->tx_func_type =ci_tx_path_select (&req_features, &idpf_tx_path_infos[0 ],IDPF_TX_MAX , -IDPF_TX_DEFAULT ); +IDPF_TX_DEFAULT , + false);/* Set use_vec_entry for single queue mode - only IDPF_TX_SINGLEQ uses regular entries */ if (vport->txq_model ==VIRTCHNL2_QUEUE_MODEL_SINGLE ) {
Signature · 1 line
2.53.0
RE: [PATCH 12/13] net/intel: improve Tx path selection logic
Quoted 19 lines
Subject: [PATCH 12/13] net/intel: improve Tx path selection logic
Rework the relevant priority of context descriptor vs SIMD width when selecting a Tx path so that presence of context descriptors is only considered when all else is equal.
Normally, path select takes place without having to worry about context descriptors, since offloads requiring context descriptors are specified using the offloads bitmap for each Tx path. However, in some cases, the need to use a context descriptor is not encodable via this offload path, e.g. VLAN offload for iavf can sometimes use the data descriptor and sometimes the context depending on what PF specifies at runtime. To account for those cases, we separate out the use of contexts from the path features, but instead note it as part of the path information, and add an additional parameter to the Tx path selection for drivers to mandate a context descriptor be present. Otherwise, ctx is largely ignored in path selection, except as a last-resort tie-breaker between two paths.
Quoted 1 line
Signed-off-by Bruce Richardson <[email protected]>
Acked-by Ciara Loftus <[email protected]>
Quoted 9 lines
--- drivers/net/intel/common/tx.h | 41 +++++++++++++++++--------- drivers/net/intel/cpfl/cpfl_rxtx.c | 3 +- drivers/net/intel/i40e/i40e_rxtx.c | 2 +- drivers/net/intel/iavf/iavf_rxtx.c | 46 ++++++++++++++++-------------- drivers/net/intel/ice/ice_rxtx.c | 3 +- drivers/net/intel/idpf/idpf_rxtx.c | 3 +- 6 files changed, 58 insertions(+), 40 deletions(-)
PATCH 13/13 net/ice: enable context desc offloads for vector Tx
In reply to Bruce RichardsonUse the context descriptor Tx vector paths from intel/common in ice driver to enable more offloads on the vector path for ice driver.
Signed-off-by Bruce Richardson <[email protected]>
--- doc/guides/rel_notes/release_26_11.rst | 6 ++++ drivers/net/intel/ice/ice_ethdev.h | 3 ++ drivers/net/intel/ice/ice_rxtx.c | 31 ++++++++++++++++++--- drivers/net/intel/ice/ice_rxtx.h | 11 ++++++++ drivers/net/intel/ice/ice_rxtx_vec_avx2.c | 24 ++++++++++++++++ drivers/net/intel/ice/ice_rxtx_vec_avx512.c | 24 ++++++++++++++++ 6 files changed, 95 insertions(+), 4 deletions(-)
doc/guides/rel_notes/release_26_11.rst +6 -0
@@ -64,6 +64,12 @@ New Features * Renamed the ``enable_ptype_lldp`` devarg to ``enable_lldp``. The old name is no longer accepted. +* **Updated Intel ice driver.** + + * Added Tx context descriptor support to the AVX2 and AVX512 vector Tx + paths, enabling QinQ tag insertion and outer IPv4/UDP checksum + offloads on those paths. + Removed Items -------------
drivers/net/intel/ice/ice_ethdev.h +3 -0
@@ -213,8 +213,10 @@ enum ice_tx_func_type {ICE_TX_SIMPLE ,ICE_TX_AVX2 ,ICE_TX_AVX2_OFFLOAD , +ICE_TX_AVX2_CTX_OFFLOAD ,ICE_TX_AVX512 ,ICE_TX_AVX512_OFFLOAD , +ICE_TX_AVX512_CTX_OFFLOAD , ICE_TX_NEON, };
@@ -673,6 +675,7 @@ struct ice_adapter {bool rx_bulk_alloc_allowed;bool tx_vec_allowed;bool tx_simple_allowed; +bool use_ctx;enum ice_rx_func_type rx_func_type;enum ice_tx_func_type tx_func_type;/* ptype mapping table */
drivers/net/intel/ice/ice_rxtx.c +27 -4
@@ -883,6 +883,7 @@ ice_tx_queue_start(struct rte_eth_dev *dev, uint16_t tx_queue_id)/* record what kind of descriptor cleanup we need on teardown */ txq->use_vec_entry = ad->tx_vec_allowed || ad->tx_simple_allowed; + txq->use_ctx = ad->use_ctx;if (txq->tsq !=NULL && txq->tsq->ts_flag >0 ) {struct ice_aqc_set_txtime_qgrp *ts_elem;
@@ -1193,7 +1194,7 @@ ice_tx_queue_stop(struct rte_eth_dev *dev, uint16_t tx_queue_id)return -EINVAL ; } -ci_txq_release_all_mbufs (txq, false); +ci_txq_release_all_mbufs (txq, txq->use_ctx);ice_reset_tx_queue (txq); dev->data->tx_queue_state[tx_queue_id] =RTE_ETH_QUEUE_STATE_STOPPED ;
@@ -1256,7 +1257,7 @@ ice_fdir_tx_queue_stop(struct rte_eth_dev *dev, uint16_t tx_queue_id)return -EINVAL ; } -ci_txq_release_all_mbufs (txq, false); +ci_txq_release_all_mbufs (txq, txq->use_ctx); txq->qtx_tail =NULL ;return 0 ;
@@ -1744,7 +1745,7 @@ ice_tx_queue_release(void *txq)return ; } -ci_txq_release_all_mbufs (q, false); +ci_txq_release_all_mbufs (q, q->use_ctx);rte_free (q->sw_ring);rte_free (q->rs_last_id);if (q->tsq) {
@@ -3526,7 +3527,8 @@ static const struct ci_tx_path_info ice_tx_path_infos[] = { .features = { .tx_offloads =ICE_TX_SCALAR_OFFLOADS }, - .pkt_prep = ice_prep_pkts + .pkt_prep = ice_prep_pkts, + .supports_ctx = true }, [ICE_TX_SIMPLE] = { .pkt_burst = ice_xmit_pkts_simple,
@@ -3556,6 +3558,16 @@ static const struct ci_tx_path_info ice_tx_path_infos[] = { }, .pkt_prep = ice_prep_pkts }, + [ICE_TX_AVX2_CTX_OFFLOAD] = { + .pkt_burst = ice_xmit_pkts_vec_avx2_ctx_offload, + .info = "Offload Vector AVX2 Ctx", + .features = { + .tx_offloads = ICE_TX_VECTOR_CTX_OFFLOAD_OFFLOADS, + .simd_width =RTE_VECT_SIMD_256 + }, + .pkt_prep = ice_prep_pkts, + .supports_ctx = true + }, #ifdef CC_AVX512_SUPPORT [ICE_TX_AVX512] = { .pkt_burst = ice_xmit_pkts_vec_avx512,
@@ -3575,6 +3587,16 @@ static const struct ci_tx_path_info ice_tx_path_infos[] = { }, .pkt_prep = ice_prep_pkts }, + [ICE_TX_AVX512_CTX_OFFLOAD] = { + .pkt_burst = ice_xmit_pkts_vec_avx512_ctx_offload, + .info = "Offload Vector AVX512 Ctx", + .features = { + .tx_offloads = ICE_TX_VECTOR_CTX_OFFLOAD_OFFLOADS, + .simd_width = RTE_VECT_SIMD_512 + }, + .pkt_prep = ice_prep_pkts, + .supports_ctx = true + },#endif #elif defined(RTE_ARCH_ARM64) [ICE_TX_NEON] = {
@@ -3788,6 +3810,7 @@ ice_set_tx_function(struct rte_eth_dev *dev) ad->tx_vec_allowed = (ice_tx_path_infos[ad->tx_func_type].features.simd_width >=RTE_VECT_SIMD_128 );#endif + ad->use_ctx = ice_tx_path_infos[ad->tx_func_type].supports_ctx; dev->tx_pkt_burst = mbuf_check ? ice_xmit_pkts_check : ice_tx_path_infos[ad->tx_func_type].pkt_burst;
drivers/net/intel/ice/ice_rxtx.h +11 -0
@@ -135,6 +135,12 @@ RTE_ETH_TX_OFFLOAD_UDP_CKSUM | \ RTE_ETH_TX_OFFLOAD_TCP_CKSUM | \ RTE_ETH_TX_OFFLOAD_SCTP_CKSUM) +/* vector ctx offload path: QinQ outer tag + outer/tunnel checksum via ctx desc */ +#define ICE_TX_VECTOR_CTX_OFFLOAD_OFFLOADS ( \ + ICE_TX_VECTOR_OFFLOAD_OFFLOADS | \ + RTE_ETH_TX_OFFLOAD_OUTER_IPV4_CKSUM | \ + RTE_ETH_TX_OFFLOAD_OUTER_UDP_CKSUM | \ + RTE_ETH_TX_OFFLOAD_QINQ_INSERT)/* Max header size can be 2K - 64 bytes */ #define ICE_RX_HDR_BUF_SIZE (2048 - 64)
@@ -284,6 +290,8 @@ uint16_t ice_xmit_pkts_vec_avx2(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts);uint16_t ice_xmit_pkts_vec_avx2_offload (void *tx_queue,struct rte_mbuf **tx_pkts,uint16_t nb_pkts); +uint16_t ice_xmit_pkts_vec_avx2_ctx_offload (void *tx_queue,struct rte_mbuf **tx_pkts, +uint16_t nb_pkts);uint16_t ice_recv_pkts_vec_avx512 (void *rx_queue,struct rte_mbuf **rx_pkts,uint16_t nb_pkts); uint16_t ice_recv_pkts_vec_avx512_offload(void *rx_queue,
@@ -300,6 +308,9 @@ uint16_t ice_xmit_pkts_vec_avx512(void *tx_queue, struct rte_mbuf **tx_pkts,uint16_t ice_xmit_pkts_vec_avx512_offload (void *tx_queue,struct rte_mbuf **tx_pkts,uint16_t nb_pkts); +uint16_t ice_xmit_pkts_vec_avx512_ctx_offload (void *tx_queue, +struct rte_mbuf **tx_pkts, +uint16_t nb_pkts);int ice_fdir_programming (struct ice_pf *pf,struct ice_fltr_desc *fdir_desc);int ice_tx_done_cleanup (void *txq,uint32_t free_cnt);int ice_get_monitor_addr (void *rx_queue,struct rte_power_monitor_cond *pmc);
drivers/net/intel/ice/ice_rxtx_vec_avx2.c +24 -0
@@ -810,6 +810,30 @@ ice_xmit_pkts_vec_avx2_offload(void *tx_queue, struct rte_mbuf **tx_pkts,return ice_xmit_pkts_vec_avx2_common (tx_queue, tx_pkts, nb_pkts, true); } +uint16_t +ice_xmit_pkts_vec_avx2_ctx_offload (void *tx_queue,struct rte_mbuf **tx_pkts, +uint16_t nb_pkts) +{ +uint16_t nb_tx =0 ; +struct ci_tx_queue *txq = (struct ci_tx_queue *)tx_queue; + +while (nb_pkts) { +uint16_t ret, num; + +/* cross rs_thresh boundary is not allowed */ + num = (uint16_t )RTE_MIN (nb_pkts <<1 , txq->tx_rs_thresh); + num = num >>1 ; + ret =ci_xmit_fixed_burst_vec_ctx_avx2 (tx_queue, &tx_pkts[nb_tx], num, + true,CI_TAG_IN_DATA_DESC ,CI_TAG_IN_CTX_DESC ,NULL ); + nb_tx += ret; + nb_pkts -= ret; +if (ret < num) +break ; + } + +return nb_tx; +} + int __rte_coldice_rxq_vec_setup (struct ci_rx_queue *rxq) {
drivers/net/intel/ice/ice_rxtx_vec_avx512.c +24 -0
@@ -882,3 +882,27 @@ ice_xmit_pkts_vec_avx512_offload(void *tx_queue, struct rte_mbuf **tx_pkts, {return ice_xmit_pkts_vec_avx512_common (tx_queue, tx_pkts, nb_pkts, true); } + +uint16_t +ice_xmit_pkts_vec_avx512_ctx_offload (void *tx_queue,struct rte_mbuf **tx_pkts, +uint16_t nb_pkts) +{ +uint16_t nb_tx =0 ; +struct ci_tx_queue *txq = (struct ci_tx_queue *)tx_queue; + +while (nb_pkts) { +uint16_t ret, num; + +/* cross rs_thresh boundary is not allowed */ + num = (uint16_t )RTE_MIN (nb_pkts <<1 , txq->tx_rs_thresh); + num = num >>1 ; + ret =ci_xmit_fixed_burst_vec_ctx_avx512 (tx_queue, &tx_pkts[nb_tx], num, + true,CI_TAG_IN_DATA_DESC ,CI_TAG_IN_CTX_DESC ,NULL ); + nb_tx += ret; + nb_pkts -= ret; +if (ret < num) +break ; + } + +return nb_tx; +}
Signature · 1 line
2.53.0
RE: [PATCH 13/13] net/ice: enable context desc offloads for vector Tx
Quoted 15 lines
Subject: [PATCH 13/13] net/ice: enable context desc offloads for vector Tx
Use the context descriptor Tx vector paths from intel/common in ice driver to enable more offloads on the vector path for ice driver.
Signed-off-by Bruce Richardson <[email protected]>--- doc/guides/rel_notes/release_26_11.rst | 6 ++++ drivers/net/intel/ice/ice_ethdev.h | 3 ++ drivers/net/intel/ice/ice_rxtx.c | 31 ++++++++++++++++++--- drivers/net/intel/ice/ice_rxtx.h | 11 ++++++++ drivers/net/intel/ice/ice_rxtx_vec_avx2.c | 24 ++++++++++++++++ drivers/net/intel/ice/ice_rxtx_vec_avx512.c | 24 ++++++++++++++++ 6 files changed, 95 insertions(+), 4 deletions(-)
<snip>
Quoted 10 lines
drivers/net/intel/ice/ice_rxtx.h +1
-0@@ -135,6 +135,12 @@RTE_ETH_TX_OFFLOAD_UDP_CKSUM | \RTE_ETH_TX_OFFLOAD_TCP_CKSUM | \ RTE_ETH_TX_OFFLOAD_SCTP_CKSUM) +/* vector ctx offload path: QinQ outer tag + outer/tunnel checksum via ctx desc */
drivers/net/intel/ice/ice_rxtx.h line 143
Quoted 5 lines
drivers/net/intel/ice/ice_rxtx.h +5
-0+#define ICE_TX_VECTOR_CTX_OFFLOAD_OFFLOADS ( \ + ICE_TX_VECTOR_OFFLOAD_OFFLOADS | \ + RTE_ETH_TX_OFFLOAD_OUTER_IPV4_CKSUM | \ + RTE_ETH_TX_OFFLOAD_OUTER_UDP_CKSUM | \ + RTE_ETH_TX_OFFLOAD_QINQ_INSERT)
Now that the paths and thus offloads are shared, is it worth creating common OFFLOADS masks? I assume IAVF_TX_VECTOR_CTX_OFFLOAD_OFFLOADS is an exact match for ICE_TX_VECTOR_CTX_OFFLOAD_OFFLOADS
Quoted 1 line
Re: [PATCH 13/13] net/ice: enable context desc offloads for vector Tx
Quoted 30 lines · Ciara
Quoted 15 lines
Subject: [PATCH 13/13] net/ice: enable context desc offloads for vector Tx
Use the context descriptor Tx vector paths from intel/common in ice driver to enable more offloads on the vector path for ice driver.
Signed-off-by Bruce Richardson <[email protected]>--- doc/guides/rel_notes/release_26_11.rst | 6 ++++ drivers/net/intel/ice/ice_ethdev.h | 3 ++ drivers/net/intel/ice/ice_rxtx.c | 31 ++++++++++++++++++--- drivers/net/intel/ice/ice_rxtx.h | 11 ++++++++ drivers/net/intel/ice/ice_rxtx_vec_avx2.c | 24 ++++++++++++++++ drivers/net/intel/ice/ice_rxtx_vec_avx512.c | 24 ++++++++++++++++ 6 files changed, 95 insertions(+), 4 deletions(-)<snip>
Quoted 12 lines
drivers/net/intel/ice/ice_rxtx.h +1
-0@@ -135,6 +135,12 @@RTE_ETH_TX_OFFLOAD_UDP_CKSUM | \RTE_ETH_TX_OFFLOAD_TCP_CKSUM | \ RTE_ETH_TX_OFFLOAD_SCTP_CKSUM) +/* vector ctx offload path: QinQ outer tag + outer/tunnel checksum via ctx desc */
drivers/net/intel/ice/ice_rxtx.h +2
-0+#define ICE_TX_VECTOR_CTX_OFFLOAD_OFFLOADS ( \ + ICE_TX_VECTOR_OFFLOAD_OFFLOADS | \
Quoted 9 lines
drivers/net/intel/ice/ice_rxtx.h line 143
Quoted 3 lines
drivers/net/intel/ice/ice_rxtx.h +3
-0+RTE_ETH_TX_OFFLOAD_OUTER_IPV4_CKSUM | \ +RTE_ETH_TX_OFFLOAD_OUTER_UDP_CKSUM | \ + RTE_ETH_TX_OFFLOAD_QINQ_INSERT)Now that the paths and thus offloads are shared, is it worth creating common OFFLOADS masks? I assume
IAVF_TX_VECTOR_CTX_OFFLOAD_OFFLOADSis an exact match forICE_TX_VECTOR_CTX_OFFLOAD_OFFLOADS
Yes, it is probably worth doing. Not sure if it's best in this patchset or not. Let me investigate.
/Bruce
Quoted 35 lines · Bruce Richardson
This patchset consolidates the ice and iavf vector Tx paths, moving the code to
net/intel/common. The iavf code is used as the basis for the move, because it has a superset of the features. When the common code is adopted by the ice driver, it then gains features such as QinQ support and tunneling support.Depends-on: series-39162 ("VLAN and QinQ fixes for iavf")
Bruce Richardson (13):
net/iavf: remove unnecessary alignment calls net/intel: make Tx context flag common
net/iavf: use separate params for VLAN and QinQ position net/iavf: deduplicate tunnel field fill functions net/intel: define common macros for tunneling bit-shifts net/intel: move iavf descriptor writing functions to common net/intel: use function callback for lldp net/ice: use common descriptor creation functions net/intel: move vector Tx paths to common net/ice: use common AVX Tx functions
net/intel: add common vector Tx fns with context handling net/intel: improve Tx path selection logic net/ice: enable context desc offloads for vector Txdoc/guides/rel_notes/release_26_11.rst | 6 + drivers/net/intel/common/tx.h | 77 +- drivers/net/intel/common/tx_vec_x86.h | 783 ++++++++++++++++++ drivers/net/intel/cpfl/cpfl_rxtx.c | 3 +- drivers/net/intel/i40e/i40e_rxtx.c | 2 +- drivers/net/intel/iavf/iavf_rxtx.c | 76 +- drivers/net/intel/iavf/iavf_rxtx.h | 34 - drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c | 496 +---------- drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c | 495 +---------- drivers/net/intel/iavf/iavf_rxtx_vec_common.h | 76 +- drivers/net/intel/ice/ice_ethdev.h | 3 +
Quoted 8 lines
drivers/net/intel/ice/ice_rxtx.c | 34 +- drivers/net/intel/ice/ice_rxtx.h | 11 + drivers/net/intel/ice/ice_rxtx_vec_avx2.c | 159 +--- drivers/net/intel/ice/ice_rxtx_vec_avx512.c | 154 +--- drivers/net/intel/ice/ice_rxtx_vec_common.h | 59 -- drivers/net/intel/idpf/idpf_rxtx.c | 3 +- 17 files changed, 1025 insertions(+), 1446 deletions(-) create mode 100644 drivers/net/intel/common/tx_vec_x86.h
I do not have the time to review in detail but this series looks good to me. My bench setup is unavailable and I am busy on other topics, so I won't be able to test it soon.
But in any case, thank you Bruce, nice cleanup again.
Signature · 1 line
David Marchand
v2 cover consolidate ice and iavf Tx vector paths
In reply to Bruce RichardsonThis patchset consolidates the ice and iavf vector Tx paths, moving the code to net/intel/common. The iavf code is used as the basis for the move, because it has a superset of the features. When the common code is adopted by the ice driver, it then gains features such as QinQ support and tunneling support.
V2:
* fix build when iova not in mbuf
* add extra patch to consolidate the offload flags
Bruce Richardson (15):
net/iavf: remove unnecessary alignment calls net/intel: make Tx context flag common
net/iavf: use separate params for VLAN and QinQ position net/iavf: deduplicate tunnel field fill functions net/intel: define common macros for tunneling bit-shifts net/intel: move iavf descriptor writing functions to common net/intel: use function callback for lldp net/intel: allow building without IOVA in mbuf net/ice: use common descriptor creation functions net/intel: move vector Tx paths to common net/ice: use common AVX Tx functions
net/intel: add common vector Tx fns with context handling net/intel: improve Tx path selection logic net/ice: enable context desc offloads for vector Tx net/intel: consolidate Tx vector offload path checks
doc/guides/rel_notes/release_26_11.rst | 6 + drivers/net/intel/common/tx.h | 107 ++- drivers/net/intel/common/tx_vec_x86.h | 781 ++++++++++++++++++ drivers/net/intel/cpfl/cpfl_rxtx.c | 3 +- drivers/net/intel/i40e/i40e_rxtx.c | 2 +- drivers/net/intel/iavf/iavf_rxtx.c | 94 +-- drivers/net/intel/iavf/iavf_rxtx.h | 53 -- drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c | 496 +---------- drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c | 495 +---------- drivers/net/intel/iavf/iavf_rxtx_vec_common.h | 79 +- drivers/net/intel/iavf/meson.build | 4 +- drivers/net/intel/ice/ice_ethdev.h | 3 + drivers/net/intel/ice/ice_rxtx.c | 44 +- drivers/net/intel/ice/ice_rxtx.h | 17 +- drivers/net/intel/ice/ice_rxtx_vec_avx2.c | 159 +--- drivers/net/intel/ice/ice_rxtx_vec_avx512.c | 154 +--- drivers/net/intel/ice/ice_rxtx_vec_common.h | 62 +- drivers/net/intel/idpf/idpf_rxtx.c | 3 +- 18 files changed, 1064 insertions(+), 1498 deletions(-) create mode 100644 drivers/net/intel/common/tx_vec_x86.h
--
2.53.0
PATCH v2 1/15 net/iavf: remove unnecessary alignment calls
When we have a context descriptor on Tx of a packet from the vector paths, that means that we always have 32-bytes being written per descriptor/per packet, so the stores are always 32-bit aligned. This means we can use aligned stores for each single descriptor store, and that we never need to do an initial descriptor write for alignment when doing a burst of descriptors.
Signed-off-by Bruce Richardson <[email protected]>
Acked-by Ciara Loftus <[email protected]>
--- drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c | 9 ++------- drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c | 9 ++------- 2 files changed, 4 insertions(+), 14 deletions(-)
drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c +2 -7
@@ -1933,7 +1933,8 @@ ctx_vtx1(volatile struct ci_tx_desc *txdp, struct rte_mbuf *pkt,__m256i ctx_data_desc =_mm256_set_epi64x (high_data_qw, pkt->buf_iova + pkt->data_off, high_ctx_qw, low_ctx_qw); - _mm256_storeu_si256(RTE_CAST_PTR(__m256i *, txdp), ctx_data_desc); +/* tx_id is always even in ctx mode, so txdp is always 32-byte aligned */ + _mm256_store_si256(RTE_CAST_PTR(__m256i *, txdp), ctx_data_desc); }static __rte_always_inline void
@@ -1944,12 +1945,6 @@ ctx_vtx(volatile struct ci_tx_desc *txdp,uint64_t hi_data_qw_tmpl = (IAVF_TX_DESC_DTYPE_DATA | ((uint64_t )flags <<IAVF_TXD_QW1_CMD_SHIFT )); -/* if unaligned on 32-bit boundary, do one to align */ -if (((uintptr_t )txdp &0x1F ) !=0 && nb_pkts !=0 ) { -ctx_vtx1 (txdp, *pkt, flags, offload, vlan_flag, ptype_lldp_enabled); - nb_pkts--; txdp++; pkt++; - } -for (; nb_pkts >1 ; txdp +=4 , pkt +=2 , nb_pkts -=2 ) {uint64_t hi_ctx_qw1 =IAVF_TX_DESC_DTYPE_CONTEXT ;uint64_t hi_ctx_qw0 =IAVF_TX_DESC_DTYPE_CONTEXT ;
drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c +2 -7
@@ -2078,7 +2078,8 @@ ctx_vtx1(volatile struct ci_tx_desc *txdp, struct rte_mbuf *pkt,__m256i ctx_data_desc =_mm256_set_epi64x (high_data_qw, pkt->buf_iova + pkt->data_off, high_ctx_qw, low_ctx_qw); - _mm256_storeu_si256(RTE_CAST_PTR(__m256i *, txdp), ctx_data_desc); +/* tx_id is always even in ctx mode, so txdp is always 32-byte aligned */ + _mm256_store_si256(RTE_CAST_PTR(__m256i *, txdp), ctx_data_desc); }static __rte_always_inline void
@@ -2088,12 +2089,6 @@ ctx_vtx(volatile struct ci_tx_desc *txdp, {uint64_t hi_data_qw_tmpl = (CI_TX_DESC_DTYPE_DATA | (flags <<CI_TXD_QW1_CMD_S )); -/* if unaligned on 32-bit boundary, do one to align */ -if (((uintptr_t )txdp &0x1F ) !=0 && nb_pkts !=0 ) { -ctx_vtx1 (txdp, *pkt, flags, offload, vlan_flag, lldp_enabled); - nb_pkts--; txdp++; pkt++; - } -for (; nb_pkts >1 ; txdp +=4 , pkt +=2 , nb_pkts -=2 ) {uint64_t hi_ctx_qw1 =IAVF_TX_DESC_DTYPE_CONTEXT ;uint64_t hi_ctx_qw0 =IAVF_TX_DESC_DTYPE_CONTEXT ;
Signature · 1 line
2.53.0
PATCH v2 2/15 net/intel: make Tx context flag common
In reply to Bruce RichardsonThe flag for enabling context descriptors in the vector paths should not be limited to just the iavf driver, as other drivers may use context paths in future also. Therefore move the flag out of the iavf-specific part of the Tx queue structure, to put with other standard queue flags.
Signed-off-by Bruce Richardson <[email protected]>
Acked-by Ciara Loftus <[email protected]>
--- drivers/net/intel/common/tx.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/net/intel/common/tx.h +1 -1
@@ -173,6 +173,7 @@ struct ci_tx_queue {bool tx_deferred_start;/* don't start this queue in dev start */ bool q_set;/* indicate if tx queue has been configured */ bool use_vec_entry;/* use sw_ring_vec (true for vector and simple paths) */ +bool use_ctx;/* with ctx info, each pkt needs two desc in vec paths */ union {/* the VSI this queue belongs to */ struct i40e_vsi *i40e_vsi;struct iavf_vsi *iavf_vsi;
@@ -194,7 +195,6 @@ struct ci_tx_queue {#define IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG2 BIT(1)uint8_t vlan_flag;uint8_t tc; -bool use_ctx;/* with ctx info, each pkt needs two descriptors */ bool lldp_enabled; };struct {/* ixgbe specific values */
Signature · 1 line
2.53.0
PATCH v2 3/15 net/iavf: use separate params for VLAN and QinQ position
In reply to Bruce RichardsonLike is the case with the scalar path, use separate offload parameters for the vector path to distinguish between the position of the single VLAN tag and the QinQ outer tag. With iavf these are currently always the same, but other drivers - with whom we may want to share code - have different positions for these.
Signed-off-by Bruce Richardson <[email protected]>
Acked-by Ciara Loftus <[email protected]>
--- drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c | 64 +++++++++++-------- drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c | 62 ++++++++++-------- drivers/net/intel/iavf/iavf_rxtx_vec_common.h | 11 ++-- 3 files changed, 77 insertions(+), 60 deletions(-)
drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c +37 -27
@@ -1619,13 +1619,14 @@ iavf_recv_scattered_pkts_vec_avx2_flex_rxd_offload(void *rx_queue, static __rte_always_inline voidiavf_vtx1 (volatile struct ci_tx_desc *txdp, -struct rte_mbuf *pkt,uint64_t flags,bool offload,uint8_t vlan_flag) +struct rte_mbuf *pkt,uint64_t flags,bool offload, +enum ci_l2tag_pos single_vlan_pos,enum ci_l2tag_pos qinq_outer_pos) {uint64_t high_qw = (CI_TX_DESC_DTYPE_DATA | ((uint64_t )flags <<CI_TXD_QW1_CMD_S ) | ((uint64_t )pkt->data_len <<CI_TXD_QW1_TX_BUF_SZ_S ));if (offload) -iavf_txd_enable_offload (pkt, &high_qw, vlan_flag); +iavf_txd_enable_offload (pkt, &high_qw, single_vlan_pos, qinq_outer_pos);__m128i descriptor =_mm_set_epi64x (high_qw, pkt->buf_iova + pkt->data_off);
@@ -1634,13 +1635,14 @@ iavf_vtx1(volatile struct ci_tx_desc *txdp, static __rte_always_inline voidiavf_vtx (volatile struct ci_tx_desc *txdp, -struct rte_mbuf **pkt,uint16_t nb_pkts,uint64_t flags,bool offload,uint8_t vlan_flag) +struct rte_mbuf **pkt,uint16_t nb_pkts,uint64_t flags,bool offload, +enum ci_l2tag_pos single_vlan_pos,enum ci_l2tag_pos qinq_outer_pos) {const uint64_t hi_qw_tmpl = (CI_TX_DESC_DTYPE_DATA | (flags <<CI_TXD_QW1_CMD_S ));/* if unaligned on 32-bit boundary, do one to align */ if (((uintptr_t )txdp &0x1F ) !=0 && nb_pkts !=0 ) { -iavf_vtx1 (txdp, *pkt, flags, offload, vlan_flag); +iavf_vtx1 (txdp, *pkt, flags, offload, single_vlan_pos, qinq_outer_pos); nb_pkts--; txdp++; pkt++; }
@@ -1649,19 +1651,19 @@ iavf_vtx(volatile struct ci_tx_desc *txdp,uint64_t hi_qw3 = hi_qw_tmpl | ((uint64_t )pkt[3 ]->data_len <<CI_TXD_QW1_TX_BUF_SZ_S );if (offload) -iavf_txd_enable_offload (pkt[3 ], &hi_qw3, vlan_flag); +iavf_txd_enable_offload (pkt[3 ], &hi_qw3, single_vlan_pos, qinq_outer_pos);uint64_t hi_qw2 = hi_qw_tmpl | ((uint64_t )pkt[2 ]->data_len <<CI_TXD_QW1_TX_BUF_SZ_S );if (offload) -iavf_txd_enable_offload (pkt[2 ], &hi_qw2, vlan_flag); +iavf_txd_enable_offload (pkt[2 ], &hi_qw2, single_vlan_pos, qinq_outer_pos);uint64_t hi_qw1 = hi_qw_tmpl | ((uint64_t )pkt[1 ]->data_len <<CI_TXD_QW1_TX_BUF_SZ_S );if (offload) -iavf_txd_enable_offload (pkt[1 ], &hi_qw1, vlan_flag); +iavf_txd_enable_offload (pkt[1 ], &hi_qw1, single_vlan_pos, qinq_outer_pos);uint64_t hi_qw0 = hi_qw_tmpl | ((uint64_t )pkt[0 ]->data_len <<CI_TXD_QW1_TX_BUF_SZ_S );if (offload) -iavf_txd_enable_offload (pkt[0 ], &hi_qw0, vlan_flag); +iavf_txd_enable_offload (pkt[0 ], &hi_qw0, single_vlan_pos, qinq_outer_pos);__m256i desc2_3 = _mm256_set_epi64x
@@ -1681,7 +1683,7 @@ iavf_vtx(volatile struct ci_tx_desc *txdp,/* do any last ones */ while (nb_pkts) { -iavf_vtx1 (txdp, *pkt, flags, offload, vlan_flag); +iavf_vtx1 (txdp, *pkt, flags, offload, single_vlan_pos, qinq_outer_pos); txdp++; pkt++; nb_pkts--; } }
@@ -1697,6 +1699,9 @@ iavf_xmit_fixed_burst_vec_avx2(void *tx_queue, struct rte_mbuf **tx_pkts,/* bit2 is reserved and must be set to 1 according to Spec */ uint64_t flags =CI_TX_DESC_CMD_EOP |CI_TX_DESC_CMD_ICRC ;uint64_t rs =CI_TX_DESC_CMD_RS | flags; +/* vlan_flag gives both the single-VLAN and the QinQ outer tag position */ +enum ci_l2tag_pos vlan_pos = (txq->vlan_flag &IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG1 ) ? +CI_TAG_IN_DATA_DESC :CI_TAG_IN_CTX_DESC ;if (txq->nb_tx_free < txq->tx_free_thresh)ci_tx_free_bufs_vec (txq, iavf_tx_desc_done, false);
@@ -1716,11 +1721,11 @@ iavf_xmit_fixed_burst_vec_avx2(void *tx_queue, struct rte_mbuf **tx_pkts,if (nb_commit >= n) {ci_tx_backlog_entry_vec (txep, tx_pkts, n); -iavf_vtx (txdp, tx_pkts, n -1 , flags, offload, txq->vlan_flag); +iavf_vtx (txdp, tx_pkts, n -1 , flags, offload, vlan_pos, vlan_pos); tx_pkts += (n -1 ); txdp += (n -1 ); -iavf_vtx1 (txdp, *tx_pkts++, rs, offload, txq->vlan_flag); +iavf_vtx1 (txdp, *tx_pkts++, rs, offload, vlan_pos, vlan_pos); nb_commit = (uint16_t )(nb_commit - n);
@@ -1734,7 +1739,7 @@ iavf_xmit_fixed_burst_vec_avx2(void *tx_queue, struct rte_mbuf **tx_pkts,ci_tx_backlog_entry_vec (txep, tx_pkts, nb_commit); -iavf_vtx (txdp, tx_pkts, nb_commit, flags, offload, txq->vlan_flag); +iavf_vtx (txdp, tx_pkts, nb_commit, flags, offload, vlan_pos, vlan_pos); tx_id = (uint16_t )(tx_id + nb_commit);if (tx_id > txq->tx_next_rs) {
@@ -1903,7 +1908,8 @@ iavf_fill_ctx_desc_tunneling_field(volatile uint64_t *qw0, static __rte_always_inline voidctx_vtx1 (volatile struct ci_tx_desc *txdp,struct rte_mbuf *pkt, -uint64_t flags,bool offload,uint8_t vlan_flag,bool ptype_lldp_enabled) +uint64_t flags,bool offload,enum ci_l2tag_pos single_vlan_pos, +enum ci_l2tag_pos qinq_outer_pos,bool ptype_lldp_enabled) {uint64_t high_ctx_qw =IAVF_TX_DESC_DTYPE_CONTEXT ;uint64_t low_ctx_qw =0 ;
@@ -1911,13 +1917,13 @@ ctx_vtx1(volatile struct ci_tx_desc *txdp, struct rte_mbuf *pkt,if (offload) {iavf_fill_ctx_desc_tunneling_avx2 (&low_ctx_qw, pkt);if (pkt->ol_flags &RTE_MBUF_F_TX_QINQ ) { -uint64_t qinq_tag = vlan_flag &IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG2 ? +uint64_t qinq_tag = qinq_outer_pos ==CI_TAG_IN_CTX_DESC ? (uint64_t )pkt->vlan_tci_outer : (uint64_t )pkt->vlan_tci; high_ctx_qw |=IAVF_TX_CTX_DESC_IL2TAG2 <<IAVF_TXD_CTX_QW1_CMD_SHIFT ; low_ctx_qw |= qinq_tag <<IAVF_TXD_CTX_QW0_L2TAG2_PARAM ; }else if ((pkt->ol_flags &RTE_MBUF_F_TX_VLAN ) && - vlan_flag &IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG2 ) { + single_vlan_pos ==CI_TAG_IN_CTX_DESC ) { high_ctx_qw |=IAVF_TX_CTX_DESC_IL2TAG2 <<IAVF_TXD_CTX_QW1_CMD_SHIFT ; low_ctx_qw |= (uint64_t )pkt->vlan_tci <<IAVF_TXD_CTX_QW0_L2TAG2_PARAM ; }
@@ -1928,7 +1934,7 @@ ctx_vtx1(volatile struct ci_tx_desc *txdp, struct rte_mbuf *pkt, ((uint64_t )flags <<IAVF_TXD_QW1_CMD_SHIFT ) | ((uint64_t)pkt->data_len << IAVF_TXD_QW1_TX_BUF_SZ_SHIFT));if (offload) -iavf_txd_enable_offload (pkt, &high_data_qw, vlan_flag); +iavf_txd_enable_offload (pkt, &high_data_qw, single_vlan_pos, qinq_outer_pos);__m256i ctx_data_desc =_mm256_set_epi64x (high_data_qw, pkt->buf_iova + pkt->data_off, high_ctx_qw, low_ctx_qw);
@@ -1940,7 +1946,8 @@ ctx_vtx1(volatile struct ci_tx_desc *txdp, struct rte_mbuf *pkt, static __rte_always_inline voidctx_vtx (volatile struct ci_tx_desc *txdp,struct rte_mbuf **pkt,uint16_t nb_pkts,uint64_t flags, -bool offload,uint8_t vlan_flag,bool ptype_lldp_enabled) +bool offload,enum ci_l2tag_pos single_vlan_pos,enum ci_l2tag_pos qinq_outer_pos, +bool ptype_lldp_enabled) {uint64_t hi_data_qw_tmpl = (IAVF_TX_DESC_DTYPE_DATA | ((uint64_t )flags <<IAVF_TXD_QW1_CMD_SHIFT ));
@@ -1964,14 +1971,14 @@ ctx_vtx(volatile struct ci_tx_desc *txdp,/* tunnel fill assigns low_ctx_qw1; must run before QinQ/VLAN OR below */ iavf_fill_ctx_desc_tunneling_field (&low_ctx_qw1, pkt[1 ]);if (pkt[1 ]->ol_flags &RTE_MBUF_F_TX_QINQ ) { -uint64_t qinq_tag = vlan_flag &IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG2 ? +uint64_t qinq_tag = qinq_outer_pos ==CI_TAG_IN_CTX_DESC ? (uint64_t )pkt[1 ]->vlan_tci_outer : (uint64_t )pkt[1 ]->vlan_tci; hi_ctx_qw1 |=IAVF_TX_CTX_DESC_IL2TAG2 <<IAVF_TXD_CTX_QW1_CMD_SHIFT ; low_ctx_qw1 |= qinq_tag <<IAVF_TXD_CTX_QW0_L2TAG2_PARAM ; }else if (pkt[1 ]->ol_flags &RTE_MBUF_F_TX_VLAN && - vlan_flag &IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG2 ) { + single_vlan_pos ==CI_TAG_IN_CTX_DESC ) { hi_ctx_qw1 |=IAVF_TX_CTX_DESC_IL2TAG2 <<IAVF_TXD_CTX_QW1_CMD_SHIFT ; low_ctx_qw1 |=
@@ -1985,14 +1992,14 @@ ctx_vtx(volatile struct ci_tx_desc *txdp,/* tunnel fill assigns low_ctx_qw0; must run before QinQ/VLAN OR below */ iavf_fill_ctx_desc_tunneling_field (&low_ctx_qw0, pkt[0 ]);if (pkt[0 ]->ol_flags &RTE_MBUF_F_TX_QINQ ) { -uint64_t qinq_tag = vlan_flag &IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG2 ? +uint64_t qinq_tag = qinq_outer_pos ==CI_TAG_IN_CTX_DESC ? (uint64_t )pkt[0 ]->vlan_tci_outer : (uint64_t )pkt[0 ]->vlan_tci; hi_ctx_qw0 |=IAVF_TX_CTX_DESC_IL2TAG2 <<IAVF_TXD_CTX_QW1_CMD_SHIFT ; low_ctx_qw0 |= qinq_tag <<IAVF_TXD_CTX_QW0_L2TAG2_PARAM ; }else if (pkt[0 ]->ol_flags &RTE_MBUF_F_TX_VLAN && - vlan_flag &IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG2 ) { + single_vlan_pos ==CI_TAG_IN_CTX_DESC ) { hi_ctx_qw0 |=IAVF_TX_CTX_DESC_IL2TAG2 <<IAVF_TXD_CTX_QW1_CMD_SHIFT ; low_ctx_qw0 |=
@@ -2003,8 +2010,8 @@ ctx_vtx(volatile struct ci_tx_desc *txdp, hi_ctx_qw0 |=IAVF_TX_CTX_DESC_SWTCH_UPLINK <<IAVF_TXD_CTX_QW1_CMD_SHIFT ;if (offload) { -iavf_txd_enable_offload (pkt[1 ], &hi_data_qw1, vlan_flag); -iavf_txd_enable_offload (pkt[0 ], &hi_data_qw0, vlan_flag); +iavf_txd_enable_offload (pkt[1 ], &hi_data_qw1, single_vlan_pos, qinq_outer_pos); +iavf_txd_enable_offload (pkt[0 ], &hi_data_qw0, single_vlan_pos, qinq_outer_pos); } __m256i desc2_3 =
@@ -2020,7 +2027,7 @@ ctx_vtx(volatile struct ci_tx_desc *txdp, }if (nb_pkts) -ctx_vtx1 (txdp, *pkt, flags, offload, vlan_flag, ptype_lldp_enabled); +ctx_vtx1 (txdp, *pkt, flags, offload, single_vlan_pos, qinq_outer_pos, ptype_lldp_enabled); }static __rte_always_inline uint16_t
@@ -2035,6 +2042,9 @@ iavf_xmit_fixed_burst_vec_avx2_ctx(void *tx_queue, struct rte_mbuf **tx_pkts,uint64_t flags =IAVF_TX_DESC_CMD_EOP |IAVF_TX_DESC_CMD_ICRC ;uint64_t rs =IAVF_TX_DESC_CMD_RS | flags;bool lldp_enabled = txq->lldp_enabled; +/* vlan_flag gives both the single-VLAN and the QinQ outer tag position */ +enum ci_l2tag_pos vlan_pos = (txq->vlan_flag &IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG1 ) ? +CI_TAG_IN_DATA_DESC :CI_TAG_IN_CTX_DESC ;if (txq->nb_tx_free < txq->tx_free_thresh)ci_tx_free_bufs_vec (txq, iavf_tx_desc_done, true);
@@ -2057,10 +2067,10 @@ iavf_xmit_fixed_burst_vec_avx2_ctx(void *tx_queue, struct rte_mbuf **tx_pkts, nb_mbuf = n >>1 ;ci_tx_backlog_entry_vec (txep, tx_pkts, nb_mbuf); -ctx_vtx (txdp, tx_pkts, nb_mbuf -1 , flags, offload, txq->vlan_flag, lldp_enabled); +ctx_vtx (txdp, tx_pkts, nb_mbuf -1 , flags, offload, vlan_pos, vlan_pos, lldp_enabled); tx_pkts += (nb_mbuf -1 ); txdp += (n -2 ); -ctx_vtx1 (txdp, *tx_pkts++, rs, offload, txq->vlan_flag, lldp_enabled); +ctx_vtx1 (txdp, *tx_pkts++, rs, offload, vlan_pos, vlan_pos, lldp_enabled); nb_commit = (uint16_t )(nb_commit - n);
@@ -2074,7 +2084,7 @@ iavf_xmit_fixed_burst_vec_avx2_ctx(void *tx_queue, struct rte_mbuf **tx_pkts, nb_mbuf = nb_commit >>1 ;ci_tx_backlog_entry_vec (txep, tx_pkts, nb_mbuf); -ctx_vtx (txdp, tx_pkts, nb_mbuf, flags, offload, txq->vlan_flag, lldp_enabled); +ctx_vtx (txdp, tx_pkts, nb_mbuf, flags, offload, vlan_pos, vlan_pos, lldp_enabled); tx_id = (uint16_t )(tx_id + nb_commit);if (tx_id > txq->tx_next_rs) {
drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c +35 -27
200 lines not on this page · raw message
drivers/net/intel/iavf/iavf_rxtx_vec_common.h +5 -6
@@ -121,14 +121,13 @@ iavf_tx_vec_dev_check_default(struct rte_eth_dev *dev) static __rte_always_inline void iavf_txd_enable_offload(__rte_unused struct rte_mbuf *tx_pkt, -uint64_t *txd_hi,uint8_t vlan_flag) +uint64_t *txd_hi,enum ci_l2tag_pos single_vlan_pos, +enum ci_l2tag_pos qinq_outer_pos) {uint64_t ol_flags = tx_pkt->ol_flags;uint32_t td_cmd =0 ;uint32_t td_offset =0 ; -RTE_SET_USED (vlan_flag); -/* Set MACLEN */ if (ol_flags &RTE_MBUF_F_TX_TUNNEL_MASK ) td_offset |= (tx_pkt->outer_l2_len >>1 )
@@ -179,12 +178,12 @@ iavf_txd_enable_offload(__rte_unused struct rte_mbuf *tx_pkt,if (ol_flags &RTE_MBUF_F_TX_QINQ ) { td_cmd |=IAVF_TX_DESC_CMD_IL2TAG1 ; -/* vlan_flag specifies outer tag location for QinQ. */ -if (vlan_flag &IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG1 ) +/* L2Tag1 always carries a tag for QinQ */ +if (qinq_outer_pos ==CI_TAG_IN_DATA_DESC ) *txd_hi |= ((uint64_t )tx_pkt->vlan_tci_outer <<CI_TXD_QW1_L2TAG1_S );else *txd_hi |= ((uint64_t )tx_pkt->vlan_tci <<CI_TXD_QW1_L2TAG1_S ); - }else if (ol_flags &RTE_MBUF_F_TX_VLAN && vlan_flag &IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG1 ) { + }else if (ol_flags &RTE_MBUF_F_TX_VLAN && single_vlan_pos ==CI_TAG_IN_DATA_DESC ) { td_cmd |=CI_TX_DESC_CMD_IL2TAG1 ; *txd_hi |= ((uint64_t )tx_pkt->vlan_tci <<CI_TXD_QW1_L2TAG1_S ); }
Signature · 1 line
2.53.0
PATCH v2 4/15 net/iavf: deduplicate tunnel field fill functions
In reply to Bruce RichardsonEach of the avx2 and avx512 Tx functions used two separate subfunctions to insert the tunneling fields into the context descriptor. Except for some minor differences (a flags guard on one) the two were identical, so remove the one missing the additional guard.
Signed-off-by Bruce Richardson <[email protected]>
Reviewed-by David Marchand <[email protected]>
--- drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c | 77 +------------------ drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c | 77 +------------------ 2 files changed, 4 insertions(+), 150 deletions(-)
drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c +2 -75
@@ -1833,79 +1833,6 @@ iavf_fill_ctx_desc_tunneling_avx2(uint64_t *low_ctx_qw, struct rte_mbuf *pkt) } } -static inline void -iavf_fill_ctx_desc_tunneling_field (volatile uint64_t *qw0, -const struct rte_mbuf *m) -{ -uint64_t eip_typ =IAVF_TX_CTX_DESC_EIPT_NONE ; -uint64_t eip_len =0 ; -uint64_t eip_noinc =0 ; -/* Default - IP_ID is increment in each segment of LSO */ - -switch (m->ol_flags & (RTE_MBUF_F_TX_OUTER_IPV4 | -RTE_MBUF_F_TX_OUTER_IPV6 | -RTE_MBUF_F_TX_OUTER_IP_CKSUM )) { -case RTE_MBUF_F_TX_OUTER_IPV4 : - eip_typ =IAVF_TX_CTX_DESC_EIPT_IPV4_NO_CHECKSUM_OFFLOAD ; - eip_len = m->outer_l3_len >>2 ; -break ; -case RTE_MBUF_F_TX_OUTER_IPV4 |RTE_MBUF_F_TX_OUTER_IP_CKSUM : - eip_typ =IAVF_TX_CTX_DESC_EIPT_IPV4_CHECKSUM_OFFLOAD ; - eip_len = m->outer_l3_len >>2 ; -break ; -case RTE_MBUF_F_TX_OUTER_IPV6 : - eip_typ =IAVF_TX_CTX_DESC_EIPT_IPV6 ; - eip_len = m->outer_l3_len >>2 ; -break ; - } - -/* L4TUNT: L4 Tunneling Type */ -switch (m->ol_flags &RTE_MBUF_F_TX_TUNNEL_MASK ) { -case RTE_MBUF_F_TX_TUNNEL_IPIP : -/* for non UDP / GRE tunneling, set to 00b */ -break ; -case RTE_MBUF_F_TX_TUNNEL_VXLAN : -case RTE_MBUF_F_TX_TUNNEL_VXLAN_GPE : -case RTE_MBUF_F_TX_TUNNEL_GTP : -case RTE_MBUF_F_TX_TUNNEL_GENEVE : - eip_typ |=IAVF_TXD_CTX_UDP_TUNNELING ; -break ; -case RTE_MBUF_F_TX_TUNNEL_GRE : - eip_typ |=IAVF_TXD_CTX_GRE_TUNNELING ; -break ; -default : -PMD_TX_LOG (ERR ,"Tunnel type not supported" ); -return ; - } - -/* L4TUNLEN: L4 Tunneling Length, in Words -* -* We depend on app to set rte_mbuf.l2_len correctly. -* For IP in GRE it should be set to the length of the GRE -* header; -* For MAC in GRE or MAC in UDP it should be set to the length -* of the GRE or UDP headers plus the inner MAC up to including -* its last Ethertype. -* If MPLS labels exists, it should include them as well. -*/ - eip_typ |= (m->l2_len >>1 ) <<IAVF_TXD_CTX_QW0_NATLEN_SHIFT ; - -/** -* Calculate the tunneling UDP checksum. -* Shall be set only if L4TUNT = 01b and EIPT is not zero -*/ -if ((eip_typ & (IAVF_TX_CTX_EXT_IP_IPV6 | -IAVF_TX_CTX_EXT_IP_IPV4 | -IAVF_TX_CTX_EXT_IP_IPV4_NO_CSUM )) && - (eip_typ &IAVF_TXD_CTX_UDP_TUNNELING ) && - (m->ol_flags &RTE_MBUF_F_TX_OUTER_UDP_CKSUM )) - eip_typ |=IAVF_TXD_CTX_QW0_L4T_CS_MASK ; - - *qw0 = eip_typ <<IAVF_TXD_CTX_QW0_TUN_PARAMS_EIPT_SHIFT | - eip_len <<IAVF_TXD_CTX_QW0_TUN_PARAMS_EIPLEN_SHIFT | - eip_noinc <<IAVF_TXD_CTX_QW0_TUN_PARAMS_EIP_NOINC_SHIFT ; -} - static __rte_always_inline void ctx_vtx1(volatile struct ci_tx_desc *txdp, struct rte_mbuf *pkt, uint64_t flags, bool offload, enum ci_l2tag_pos single_vlan_pos,
@@ -1969,7 +1896,7 @@ ctx_vtx(volatile struct ci_tx_desc *txdp,if (offload) {/* tunnel fill assigns low_ctx_qw1; must run before QinQ/VLAN OR below */ -iavf_fill_ctx_desc_tunneling_field (&low_ctx_qw1, pkt[1 ]); +iavf_fill_ctx_desc_tunneling_avx2 (&low_ctx_qw1, pkt[1 ]);if (pkt[1 ]->ol_flags &RTE_MBUF_F_TX_QINQ ) {uint64_t qinq_tag = qinq_outer_pos ==CI_TAG_IN_CTX_DESC ? (uint64_t)pkt[1]->vlan_tci_outer :
@@ -1990,7 +1917,7 @@ ctx_vtx(volatile struct ci_tx_desc *txdp,if (offload) {/* tunnel fill assigns low_ctx_qw0; must run before QinQ/VLAN OR below */ -iavf_fill_ctx_desc_tunneling_field (&low_ctx_qw0, pkt[0 ]); +iavf_fill_ctx_desc_tunneling_avx2 (&low_ctx_qw0, pkt[0 ]);if (pkt[0 ]->ol_flags &RTE_MBUF_F_TX_QINQ ) {uint64_t qinq_tag = qinq_outer_pos ==CI_TAG_IN_CTX_DESC ? (uint64_t)pkt[0]->vlan_tci_outer :
drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c +2 -75
95 lines not on this page · raw message
Signature · 1 line
2.53.0
PATCH v2 5/15 net/intel: define common macros for tunneling bit-shifts
In reply to Bruce RichardsonDefine common macros for use in iavf (for now) and other drivers (future) defining the shifts needed for tunnel packet Tx
Signed-off-by Bruce Richardson <[email protected]>
Acked-by Ciara Loftus <[email protected]>
--- drivers/net/intel/common/tx.h | 24 +++++++++++++ drivers/net/intel/iavf/iavf_rxtx.c | 30 ++++++++-------- drivers/net/intel/iavf/iavf_rxtx.h | 34 ------------------- drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c | 30 ++++++++-------- drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c | 30 ++++++++-------- 5 files changed, 69 insertions(+), 79 deletions(-)
drivers/net/intel/common/tx.h +24 -0
@@ -45,6 +45,30 @@#define CI_TX_CTX_DESC_TSYN 0x02#define CI_TX_CTX_DESC_IL2TAG2 0x04 +/* Common TX Context Descriptor QW0 Field Shifts */ +#define CI_TXD_CTX_QW0_L2TAG2_S 32 +#define CI_TXD_CTX_QW0_EIPT_S 0/* 2 BITS - external IP type */ +#define CI_TXD_CTX_QW0_EIPLEN_S 2/* 7 BITS - external IP header length */ +#define CI_TXD_CTX_QW0_NATT_S 9/* 2 BITS - L4 tunnel type */ +#define CI_TXD_CTX_QW0_EIP_NOINC_S 11/* 1 BIT - external IP ID not incremented */ +#define CI_TXD_CTX_QW0_NATLEN_S 12/* 7 BITS - L4 tunnel header length */ +#define CI_TXD_CTX_QW0_L4T_CS_S 23/* 1 BIT - tunnel UDP checksum enable */ + +/** +* Enum for the external (outer) IP type field of a Tx context descriptor's +* tunnel parameters (QW0 bits 0-1). +*/ +enum ci_tx_ctx_eipt { +CI_TX_CTX_EIPT_NONE , +CI_TX_CTX_EIPT_IPV6 , +CI_TX_CTX_EIPT_IPV4_NO_CSUM , +CI_TX_CTX_EIPT_IPV4 , +}; + +#define CI_TXD_CTX_UDP_TUNNELING (0x1ULL << CI_TXD_CTX_QW0_NATT_S) +#define CI_TXD_CTX_GRE_TUNNELING (0x2ULL << CI_TXD_CTX_QW0_NATT_S) +#define CI_TXD_CTX_QW0_L4T_CS_M (0x1ULL << CI_TXD_CTX_QW0_L4T_CS_S) +/** * Enum to specify where a VLAN tag is to be placed for packet Tx. */
drivers/net/intel/iavf/iavf_rxtx.c +15 -15
@@ -2345,7 +2345,7 @@ iavf_calc_context_desc(const struct rte_mbuf *mb, uint8_t vlan_flag, bool lldp_estatic inline void iavf_fill_ctx_desc_tunnelling_field (uint64_t *qw0,uint64_t ol_flags,const struct rte_mbuf *m) { -uint64_t eip_typ =IAVF_TX_CTX_DESC_EIPT_NONE ; +uint64_t eip_typ =CI_TX_CTX_EIPT_NONE ;uint64_t eip_len =0 ;uint64_t eip_noinc =0 ;/* Default - IP_ID is increment in each segment of LSO */
@@ -2354,15 +2354,15 @@ iavf_fill_ctx_desc_tunnelling_field(uint64_t *qw0, uint64_t ol_flags, const stru RTE_MBUF_F_TX_OUTER_IPV6 | RTE_MBUF_F_TX_OUTER_IP_CKSUM)) {case RTE_MBUF_F_TX_OUTER_IPV4 : - eip_typ =IAVF_TX_CTX_DESC_EIPT_IPV4_NO_CHECKSUM_OFFLOAD ; + eip_typ =CI_TX_CTX_EIPT_IPV4_NO_CSUM ; eip_len = m->outer_l3_len >>2 ;break ;case RTE_MBUF_F_TX_OUTER_IPV4 |RTE_MBUF_F_TX_OUTER_IP_CKSUM : - eip_typ =IAVF_TX_CTX_DESC_EIPT_IPV4_CHECKSUM_OFFLOAD ; + eip_typ =CI_TX_CTX_EIPT_IPV4 ; eip_len = m->outer_l3_len >>2 ;break ;case RTE_MBUF_F_TX_OUTER_IPV6 : - eip_typ =IAVF_TX_CTX_DESC_EIPT_IPV6 ; + eip_typ =CI_TX_CTX_EIPT_IPV6 ; eip_len = m->outer_l3_len >>2 ;break ; }
@@ -2377,10 +2377,10 @@ iavf_fill_ctx_desc_tunnelling_field(uint64_t *qw0, uint64_t ol_flags, const strucase RTE_MBUF_F_TX_TUNNEL_VXLAN_GPE :case RTE_MBUF_F_TX_TUNNEL_GTP :case RTE_MBUF_F_TX_TUNNEL_GENEVE : - eip_typ |=IAVF_TXD_CTX_UDP_TUNNELING ; + eip_typ |=CI_TXD_CTX_UDP_TUNNELING ;break ;case RTE_MBUF_F_TX_TUNNEL_GRE : - eip_typ |=IAVF_TXD_CTX_GRE_TUNNELING ; + eip_typ |=CI_TXD_CTX_GRE_TUNNELING ;break ;default :PMD_TX_LOG (ERR ,"Tunnel type not supported" );
@@ -2397,23 +2397,23 @@ iavf_fill_ctx_desc_tunnelling_field(uint64_t *qw0, uint64_t ol_flags, const stru* its last Ethertype. * If MPLS labels exists, it should include them as well. */ - eip_typ |= (m->l2_len >>1 ) <<IAVF_TXD_CTX_QW0_NATLEN_SHIFT ; + eip_typ |= (m->l2_len >>1 ) <<CI_TXD_CTX_QW0_NATLEN_S ;/** * Calculate the tunneling UDP checksum. * Shall be set only if L4TUNT = 01b and EIPT is not zero */ -if ((eip_typ & (IAVF_TX_CTX_EXT_IP_IPV6 | -IAVF_TX_CTX_EXT_IP_IPV4 | -IAVF_TX_CTX_EXT_IP_IPV4_NO_CSUM )) && - (eip_typ &IAVF_TXD_CTX_UDP_TUNNELING ) && +if ((eip_typ & (CI_TX_CTX_EIPT_IPV6 | +CI_TX_CTX_EIPT_IPV4 | +CI_TX_CTX_EIPT_IPV4_NO_CSUM )) && + (eip_typ &CI_TXD_CTX_UDP_TUNNELING ) && (ol_flags &RTE_MBUF_F_TX_OUTER_UDP_CKSUM )) - eip_typ |=IAVF_TXD_CTX_QW0_L4T_CS_MASK ; + eip_typ |=CI_TXD_CTX_QW0_L4T_CS_M ; } - *qw0 = eip_typ <<IAVF_TXD_CTX_QW0_TUN_PARAMS_EIPT_SHIFT | - eip_len <<IAVF_TXD_CTX_QW0_TUN_PARAMS_EIPLEN_SHIFT | - eip_noinc <<IAVF_TXD_CTX_QW0_TUN_PARAMS_EIP_NOINC_SHIFT ; + *qw0 = eip_typ <<CI_TXD_CTX_QW0_EIPT_S | + eip_len <<CI_TXD_CTX_QW0_EIPLEN_S | + eip_noinc <<CI_TXD_CTX_QW0_EIP_NOINC_S ; } static inline uint16_t
drivers/net/intel/iavf/iavf_rxtx.h +0 -34
@@ -444,40 +444,6 @@ enum iavf_rx_flex_desc_ipsec_crypto_status {#define IAVF_TXD_CTX_QW1_SEG_PARAMS_MSS_MASK \ (0x3FFFUL << IAVF_TXD_CTX_QW1_SEG_PARAMS_MSS_SHIFT) -#define IAVF_TXD_CTX_QW0_TUN_PARAMS_EIPT_SHIFT (0) -#define IAVF_TXD_CTX_QW0_TUN_PARAMS_EIPT_MASK (0x3UL) - -enum iavf_tx_ctx_desc_tunnel_external_ip_type { -IAVF_TX_CTX_DESC_EIPT_NONE , -IAVF_TX_CTX_DESC_EIPT_IPV6 , -IAVF_TX_CTX_DESC_EIPT_IPV4_NO_CHECKSUM_OFFLOAD , -IAVF_TX_CTX_DESC_EIPT_IPV4_CHECKSUM_OFFLOAD -}; - -#define IAVF_TXD_CTX_QW0_TUN_PARAMS_EIPLEN_SHIFT (2) -#define IAVF_TXD_CTX_QW0_TUN_PARAMS_EIPLEN_MASK (0x7FUL) - -#define IAVF_TXD_CTX_QW0_TUN_PARAMS_L4TUNT_SHIFT (9) -#define IAVF_TXD_CTX_QW0_TUN_PARAMS_L4TUNT_MASK (0x3UL) - -enum iavf_tx_ctx_desc_tunnel_l4_tunnel_type { -IAVF_TX_CTX_DESC_L4_TUN_TYP_NO_UDP_GRE , -IAVF_TX_CTX_DESC_L4_TUN_TYP_UDP , -IAVF_TX_CTX_DESC_L4_TUN_TYP_GRE -}; - -#define IAVF_TXD_CTX_QW0_TUN_PARAMS_EIP_NOINC_SHIFT (11) -#define IAVF_TXD_CTX_QW0_TUN_PARAMS_EIP_NOINC_MASK (0x1UL) - -#define IAVF_TXD_CTX_QW0_TUN_PARAMS_L4TUNLEN_SHIFT (12) -#define IAVF_TXD_CTX_QW0_TUN_PARAMS_L4TUNLEN_MASK (0x7FUL) - -#define IAVF_TXD_CTX_QW0_TUN_PARAMS_DECTTL_SHIFT (19) -#define IAVF_TXD_CTX_QW0_TUN_PARAMS_DECTTL_MASK (0xFUL) - -#define IAVF_TXD_CTX_QW0_TUN_PARAMS_L4T_CS_SHIFT (23) -#define IAVF_TXD_CTX_QW0_TUN_PARAMS_L4T_CS_MASK (0x1UL) -#define IAVF_TXD_CTX_QW0_L2TAG2_PARAM (32)#define IAVF_TXD_CTX_QW0_L2TAG2_MASK (0xFFFFUL)
drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c +15 -15
@@ -1760,7 +1760,7 @@ static inline void iavf_fill_ctx_desc_tunneling_avx2(uint64_t *low_ctx_qw, struct rte_mbuf *pkt) {if (pkt->ol_flags &RTE_MBUF_F_TX_TUNNEL_MASK ) { -uint64_t eip_typ =IAVF_TX_CTX_DESC_EIPT_NONE ; +uint64_t eip_typ =CI_TX_CTX_EIPT_NONE ;uint64_t eip_len =0 ;uint64_t eip_noinc =0 ;/* Default - IP_ID is increment in each segment of LSO */
@@ -1769,15 +1769,15 @@ iavf_fill_ctx_desc_tunneling_avx2(uint64_t *low_ctx_qw, struct rte_mbuf *pkt) RTE_MBUF_F_TX_OUTER_IPV6 | RTE_MBUF_F_TX_OUTER_IP_CKSUM)) {case RTE_MBUF_F_TX_OUTER_IPV4 : - eip_typ =IAVF_TX_CTX_DESC_EIPT_IPV4_NO_CHECKSUM_OFFLOAD ; + eip_typ =CI_TX_CTX_EIPT_IPV4_NO_CSUM ; eip_len = pkt->outer_l3_len >>2 ;break ;case RTE_MBUF_F_TX_OUTER_IPV4 |RTE_MBUF_F_TX_OUTER_IP_CKSUM : - eip_typ =IAVF_TX_CTX_DESC_EIPT_IPV4_CHECKSUM_OFFLOAD ; + eip_typ =CI_TX_CTX_EIPT_IPV4 ; eip_len = pkt->outer_l3_len >>2 ;break ;case RTE_MBUF_F_TX_OUTER_IPV6 : - eip_typ =IAVF_TX_CTX_DESC_EIPT_IPV6 ; + eip_typ =CI_TX_CTX_EIPT_IPV6 ; eip_len = pkt->outer_l3_len >>2 ;break ; }
@@ -1791,10 +1791,10 @@ iavf_fill_ctx_desc_tunneling_avx2(uint64_t *low_ctx_qw, struct rte_mbuf *pkt)case RTE_MBUF_F_TX_TUNNEL_VXLAN_GPE :case RTE_MBUF_F_TX_TUNNEL_GTP :case RTE_MBUF_F_TX_TUNNEL_GENEVE : - eip_typ |=IAVF_TXD_CTX_UDP_TUNNELING ; + eip_typ |=CI_TXD_CTX_UDP_TUNNELING ;break ;case RTE_MBUF_F_TX_TUNNEL_GRE : - eip_typ |=IAVF_TXD_CTX_GRE_TUNNELING ; + eip_typ |=CI_TXD_CTX_GRE_TUNNELING ;break ;default :PMD_TX_LOG (ERR ,"Tunnel type not supported" );
@@ -1811,22 +1811,22 @@ iavf_fill_ctx_desc_tunneling_avx2(uint64_t *low_ctx_qw, struct rte_mbuf *pkt)* its last Ethertype. * If MPLS labels exists, it should include them as well. */ - eip_typ |= (pkt->l2_len >>1 ) <<IAVF_TXD_CTX_QW0_NATLEN_SHIFT ; + eip_typ |= (pkt->l2_len >>1 ) <<CI_TXD_CTX_QW0_NATLEN_S ;/** * Calculate the tunneling UDP checksum. * Shall be set only if L4TUNT = 01b and EIPT is not zero */ -if ((eip_typ & (IAVF_TX_CTX_EXT_IP_IPV4 | -IAVF_TX_CTX_EXT_IP_IPV6 | -IAVF_TX_CTX_EXT_IP_IPV4_NO_CSUM )) && - (eip_typ &IAVF_TXD_CTX_UDP_TUNNELING ) && +if ((eip_typ & (CI_TX_CTX_EIPT_IPV4 | +CI_TX_CTX_EIPT_IPV6 | +CI_TX_CTX_EIPT_IPV4_NO_CSUM )) && + (eip_typ &CI_TXD_CTX_UDP_TUNNELING ) && (pkt->ol_flags &RTE_MBUF_F_TX_OUTER_UDP_CKSUM )) - eip_typ |=IAVF_TXD_CTX_QW0_L4T_CS_MASK ; + eip_typ |=CI_TXD_CTX_QW0_L4T_CS_M ; - *low_ctx_qw = eip_typ <<IAVF_TXD_CTX_QW0_TUN_PARAMS_EIPT_SHIFT | - eip_len <<IAVF_TXD_CTX_QW0_TUN_PARAMS_EIPLEN_SHIFT | - eip_noinc <<IAVF_TXD_CTX_QW0_TUN_PARAMS_EIP_NOINC_SHIFT ; + *low_ctx_qw = eip_typ <<CI_TXD_CTX_QW0_EIPT_S | + eip_len <<CI_TXD_CTX_QW0_EIPLEN_S | + eip_noinc <<CI_TXD_CTX_QW0_EIP_NOINC_S ; }else { *low_ctx_qw =0 ;
drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c +15 -15
@@ -1899,7 +1899,7 @@ static __rte_always_inline void iavf_fill_ctx_desc_tunneling_avx512(uint64_t *low_ctx_qw, struct rte_mbuf *pkt) {if (pkt->ol_flags &RTE_MBUF_F_TX_TUNNEL_MASK ) { -uint64_t eip_typ =IAVF_TX_CTX_DESC_EIPT_NONE ; +uint64_t eip_typ =CI_TX_CTX_EIPT_NONE ;uint64_t eip_len =0 ;uint64_t eip_noinc =0 ;/* Default - IP_ID is increment in each segment of LSO */
@@ -1908,15 +1908,15 @@ iavf_fill_ctx_desc_tunneling_avx512(uint64_t *low_ctx_qw, struct rte_mbuf *pkt) RTE_MBUF_F_TX_OUTER_IPV6 | RTE_MBUF_F_TX_OUTER_IP_CKSUM)) {case RTE_MBUF_F_TX_OUTER_IPV4 : - eip_typ =IAVF_TX_CTX_DESC_EIPT_IPV4_NO_CHECKSUM_OFFLOAD ; + eip_typ =CI_TX_CTX_EIPT_IPV4_NO_CSUM ; eip_len = pkt->outer_l3_len >>2 ;break ;case RTE_MBUF_F_TX_OUTER_IPV4 |RTE_MBUF_F_TX_OUTER_IP_CKSUM : - eip_typ =IAVF_TX_CTX_DESC_EIPT_IPV4_CHECKSUM_OFFLOAD ; + eip_typ =CI_TX_CTX_EIPT_IPV4 ; eip_len = pkt->outer_l3_len >>2 ;break ;case RTE_MBUF_F_TX_OUTER_IPV6 : - eip_typ =IAVF_TX_CTX_DESC_EIPT_IPV6 ; + eip_typ =CI_TX_CTX_EIPT_IPV6 ; eip_len = pkt->outer_l3_len >>2 ;break ; }
@@ -1930,10 +1930,10 @@ iavf_fill_ctx_desc_tunneling_avx512(uint64_t *low_ctx_qw, struct rte_mbuf *pkt)case RTE_MBUF_F_TX_TUNNEL_VXLAN_GPE :case RTE_MBUF_F_TX_TUNNEL_GTP :case RTE_MBUF_F_TX_TUNNEL_GENEVE : - eip_typ |=IAVF_TXD_CTX_UDP_TUNNELING ; + eip_typ |=CI_TXD_CTX_UDP_TUNNELING ;break ;case RTE_MBUF_F_TX_TUNNEL_GRE : - eip_typ |=IAVF_TXD_CTX_GRE_TUNNELING ; + eip_typ |=CI_TXD_CTX_GRE_TUNNELING ;break ;default :PMD_TX_LOG (ERR ,"Tunnel type not supported" );
@@ -1950,22 +1950,22 @@ iavf_fill_ctx_desc_tunneling_avx512(uint64_t *low_ctx_qw, struct rte_mbuf *pkt)* its last Ethertype. * If MPLS labels exists, it should include them as well. */ - eip_typ |= (pkt->l2_len >>1 ) <<IAVF_TXD_CTX_QW0_NATLEN_SHIFT ; + eip_typ |= (pkt->l2_len >>1 ) <<CI_TXD_CTX_QW0_NATLEN_S ;/** * Calculate the tunneling UDP checksum. * Shall be set only if L4TUNT = 01b and EIPT is not zero */ -if ((eip_typ & (IAVF_TX_CTX_EXT_IP_IPV4 | -IAVF_TX_CTX_EXT_IP_IPV6 | -IAVF_TX_CTX_EXT_IP_IPV4_NO_CSUM )) && - (eip_typ &IAVF_TXD_CTX_UDP_TUNNELING ) && +if ((eip_typ & (CI_TX_CTX_EIPT_IPV4 | +CI_TX_CTX_EIPT_IPV6 | +CI_TX_CTX_EIPT_IPV4_NO_CSUM )) && + (eip_typ &CI_TXD_CTX_UDP_TUNNELING ) && (pkt->ol_flags &RTE_MBUF_F_TX_OUTER_UDP_CKSUM )) - eip_typ |=IAVF_TXD_CTX_QW0_L4T_CS_MASK ; + eip_typ |=CI_TXD_CTX_QW0_L4T_CS_M ; - *low_ctx_qw = eip_typ <<IAVF_TXD_CTX_QW0_TUN_PARAMS_EIPT_SHIFT | - eip_len <<IAVF_TXD_CTX_QW0_TUN_PARAMS_EIPLEN_SHIFT | - eip_noinc <<IAVF_TXD_CTX_QW0_TUN_PARAMS_EIP_NOINC_SHIFT ; + *low_ctx_qw = eip_typ <<CI_TXD_CTX_QW0_EIPT_S | + eip_len <<CI_TXD_CTX_QW0_EIPLEN_S | + eip_noinc <<CI_TXD_CTX_QW0_EIP_NOINC_S ; }else { *low_ctx_qw =0 ;
Signature · 1 line
2.53.0
PATCH v2 6/15 net/intel: move iavf descriptor writing functions to common
In reply to Bruce RichardsonIn order to promote reuse of code, move the Tx vector functions for writing descriptors from the iavf driver to the common folder in a new tx_vec_x86.h file. This will allow reuse in later commits by other drivers such as ice.
Signed-off-by Bruce Richardson <[email protected]>
Acked-by Ciara Loftus <[email protected]>
--- drivers/net/intel/common/tx_vec_x86.h | 511 ++++++++++++++++++ drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c | 285 +--------- drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c | 272 +--------- drivers/net/intel/iavf/iavf_rxtx_vec_common.h | 75 +-- 4 files changed, 530 insertions(+), 613 deletions(-) create mode 100644 drivers/net/intel/common/tx_vec_x86.h
drivers/net/intel/common/tx_vec_x86.h new file +511 -0
@@ -0,0 +1,511 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2026 Intel Corporation +*/ + +#ifndef _COMMON_INTEL_TX_VEC_X86_H_ +#define _COMMON_INTEL_TX_VEC_X86_H_ + +#include <stdint.h> + +#include <rte_mbuf.h> + +#include "tx.h" + +static __rte_always_inline void +ci_fill_ctx_desc_tunneling (uint64_t *low_ctx_qw,struct rte_mbuf *pkt) +{ +if (pkt->ol_flags &RTE_MBUF_F_TX_TUNNEL_MASK ) { +uint64_t eip_typ =CI_TX_CTX_EIPT_NONE ; +uint64_t eip_len =0 ; +uint64_t eip_noinc =0 ; +/* Default - IP_ID is increment in each segment of LSO */ + +switch (pkt->ol_flags & (RTE_MBUF_F_TX_OUTER_IPV4 | +RTE_MBUF_F_TX_OUTER_IPV6 | +RTE_MBUF_F_TX_OUTER_IP_CKSUM )) { +case RTE_MBUF_F_TX_OUTER_IPV4 : + eip_typ =CI_TX_CTX_EIPT_IPV4_NO_CSUM ; + eip_len = pkt->outer_l3_len >>2 ; +break ; +case RTE_MBUF_F_TX_OUTER_IPV4 |RTE_MBUF_F_TX_OUTER_IP_CKSUM : + eip_typ =CI_TX_CTX_EIPT_IPV4 ; + eip_len = pkt->outer_l3_len >>2 ; +break ; +case RTE_MBUF_F_TX_OUTER_IPV6 : + eip_typ =CI_TX_CTX_EIPT_IPV6 ; + eip_len = pkt->outer_l3_len >>2 ; +break ; + } + +/* L4TUNT: L4 Tunneling Type */ +switch (pkt->ol_flags &RTE_MBUF_F_TX_TUNNEL_MASK ) { +case RTE_MBUF_F_TX_TUNNEL_IPIP : +/* for non UDP / GRE tunneling, set to 00b */ +break ; +case RTE_MBUF_F_TX_TUNNEL_VXLAN : +case RTE_MBUF_F_TX_TUNNEL_VXLAN_GPE : +case RTE_MBUF_F_TX_TUNNEL_GTP : +case RTE_MBUF_F_TX_TUNNEL_GENEVE : + eip_typ |=CI_TXD_CTX_UDP_TUNNELING ; +break ; +case RTE_MBUF_F_TX_TUNNEL_GRE : + eip_typ |=CI_TXD_CTX_GRE_TUNNELING ; +break ; +default : +PMD_TX_LOG (ERR ,"Tunnel type not supported" ); +return ; + } + +/* L4TUNLEN: L4 Tunneling Length, in Words +* +* We depend on app to set rte_mbuf.l2_len correctly. +* For IP in GRE it should be set to the length of the GRE +* header; +* For MAC in GRE or MAC in UDP it should be set to the length +* of the GRE or UDP headers plus the inner MAC up to including +* its last Ethertype. +* If MPLS labels exists, it should include them as well. +*/ + eip_typ |= (pkt->l2_len >>1 ) <<CI_TXD_CTX_QW0_NATLEN_S ; + +/** +* Calculate the tunneling UDP checksum. +* Shall be set only if L4TUNT = 01b and EIPT is not zero +*/ +if ((eip_typ & (CI_TX_CTX_EIPT_IPV4 | +CI_TX_CTX_EIPT_IPV6 | +CI_TX_CTX_EIPT_IPV4_NO_CSUM )) && + (eip_typ &CI_TXD_CTX_UDP_TUNNELING ) && + (pkt->ol_flags &RTE_MBUF_F_TX_OUTER_UDP_CKSUM )) + eip_typ |=CI_TXD_CTX_QW0_L4T_CS_M ; + + *low_ctx_qw = eip_typ <<CI_TXD_CTX_QW0_EIPT_S | + eip_len <<CI_TXD_CTX_QW0_EIPLEN_S | + eip_noinc <<CI_TXD_CTX_QW0_EIP_NOINC_S ; + + }else { + *low_ctx_qw =0 ; + } +} + +static __rte_always_inline void +ci_tx_vec_offload (struct rte_mbuf *tx_pkt,uint64_t *txd_hi, +enum ci_l2tag_pos single_vlan_pos,enum ci_l2tag_pos qinq_outer_pos) +{ +uint64_t ol_flags = tx_pkt->ol_flags; +uint32_t td_cmd =0 ; +uint32_t td_offset =0 ; + +/* Set MACLEN */ +if (ol_flags &RTE_MBUF_F_TX_TUNNEL_MASK ) + td_offset |= (tx_pkt->outer_l2_len >>1 ) <<CI_TX_DESC_LEN_MACLEN_S ; +else + td_offset |= (tx_pkt->l2_len >>1 ) <<CI_TX_DESC_LEN_MACLEN_S ; + +/* Enable L3 checksum offloads */ +if (ol_flags &RTE_MBUF_F_TX_IP_CKSUM ) { +if (ol_flags &RTE_MBUF_F_TX_IPV4 ) { + td_cmd |=CI_TX_DESC_CMD_IIPT_IPV4_CSUM ; + td_offset |= (tx_pkt->l3_len >>2 ) <<CI_TX_DESC_LEN_IPLEN_S ; + } + }else if (ol_flags &RTE_MBUF_F_TX_IPV4 ) { + td_cmd |=CI_TX_DESC_CMD_IIPT_IPV4 ; + td_offset |= (tx_pkt->l3_len >>2 ) <<CI_TX_DESC_LEN_IPLEN_S ; + }else if (ol_flags &RTE_MBUF_F_TX_IPV6 ) { + td_cmd |=CI_TX_DESC_CMD_IIPT_IPV6 ; + td_offset |= (tx_pkt->l3_len >>2 ) <<CI_TX_DESC_LEN_IPLEN_S ; + } + +/* Enable L4 checksum offloads */ +switch (ol_flags &RTE_MBUF_F_TX_L4_MASK ) { +case RTE_MBUF_F_TX_TCP_CKSUM : + td_cmd |=CI_TX_DESC_CMD_L4T_EOFT_TCP ; + td_offset |= (sizeof (struct rte_tcp_hdr ) >>2 ) <<CI_TX_DESC_LEN_L4_LEN_S ; +break ; +case RTE_MBUF_F_TX_SCTP_CKSUM : + td_cmd |=CI_TX_DESC_CMD_L4T_EOFT_SCTP ; + td_offset |= (sizeof (struct rte_sctp_hdr ) >>2 ) <<CI_TX_DESC_LEN_L4_LEN_S ; +break ; +case RTE_MBUF_F_TX_UDP_CKSUM : + td_cmd |=CI_TX_DESC_CMD_L4T_EOFT_UDP ; + td_offset |= (sizeof (struct rte_udp_hdr ) >>2 ) <<CI_TX_DESC_LEN_L4_LEN_S ; +break ; +default : +break ; + } + + *txd_hi |= ((uint64_t )td_offset) <<CI_TXD_QW1_OFFSET_S ; + +if (ol_flags &RTE_MBUF_F_TX_QINQ ) { + td_cmd |=CI_TX_DESC_CMD_IL2TAG1 ; +/* L2Tag1 always carries a tag for QinQ */ +if (qinq_outer_pos ==CI_TAG_IN_DATA_DESC ) + *txd_hi |= ((uint64_t )tx_pkt->vlan_tci_outer <<CI_TXD_QW1_L2TAG1_S ); +else + *txd_hi |= ((uint64_t )tx_pkt->vlan_tci <<CI_TXD_QW1_L2TAG1_S ); + }else if (ol_flags &RTE_MBUF_F_TX_VLAN && single_vlan_pos ==CI_TAG_IN_DATA_DESC ) { + td_cmd |=CI_TX_DESC_CMD_IL2TAG1 ; + *txd_hi |= ((uint64_t )tx_pkt->vlan_tci <<CI_TXD_QW1_L2TAG1_S ); + } + + *txd_hi |= ((uint64_t )td_cmd) <<CI_TXD_QW1_CMD_S ; +} + +static __rte_always_inline void +ci_vtx1 (volatile struct ci_tx_desc *txdp, +struct rte_mbuf *pkt,uint64_t flags,bool offload, +enum ci_l2tag_pos single_vlan_pos,enum ci_l2tag_pos qinq_outer_pos) +{ +uint64_t high_qw = (CI_TX_DESC_DTYPE_DATA | + ((uint64_t )flags <<CI_TXD_QW1_CMD_S ) | + ((uint64_t )pkt->data_len <<CI_TXD_QW1_TX_BUF_SZ_S )); +if (offload) +ci_tx_vec_offload (pkt, &high_qw, single_vlan_pos, qinq_outer_pos); + +__m128i descriptor =_mm_set_epi64x (high_qw, pkt->buf_iova + pkt->data_off); + _mm_store_si128(RTE_CAST_PTR(__m128i *, txdp), descriptor); +} + +#ifdef __AVX2__ + +static __rte_always_inline void +ci_vtx_avx2 (volatile struct ci_tx_desc *txdp, +struct rte_mbuf **pkt,uint16_t nb_pkts,uint64_t flags,bool offload, +enum ci_l2tag_pos single_vlan_pos,enum ci_l2tag_pos qinq_outer_pos) +{ +const uint64_t hi_qw_tmpl = (CI_TX_DESC_DTYPE_DATA | (flags <<CI_TXD_QW1_CMD_S )); + +/* if unaligned on 32-bit boundary, do one to align */ +if (((uintptr_t )txdp &0x1F ) !=0 && nb_pkts !=0 ) { +ci_vtx1 (txdp, *pkt, flags, offload, single_vlan_pos, qinq_outer_pos); + nb_pkts--; txdp++; pkt++; + } + +/* do two at a time while possible, in bursts */ +for (; nb_pkts >3 ; txdp +=4 , pkt +=4 , nb_pkts -=4 ) { +uint64_t hi_qw3 = hi_qw_tmpl | + ((uint64_t )pkt[3 ]->data_len <<CI_TXD_QW1_TX_BUF_SZ_S ); +if (offload) +ci_tx_vec_offload (pkt[3 ], &hi_qw3, single_vlan_pos, qinq_outer_pos); +uint64_t hi_qw2 = hi_qw_tmpl | + ((uint64_t )pkt[2 ]->data_len <<CI_TXD_QW1_TX_BUF_SZ_S ); +if (offload) +ci_tx_vec_offload (pkt[2 ], &hi_qw2, single_vlan_pos, qinq_outer_pos); +uint64_t hi_qw1 = hi_qw_tmpl | + ((uint64_t )pkt[1 ]->data_len <<CI_TXD_QW1_TX_BUF_SZ_S ); +if (offload) +ci_tx_vec_offload (pkt[1 ], &hi_qw1, single_vlan_pos, qinq_outer_pos); +uint64_t hi_qw0 = hi_qw_tmpl | + ((uint64_t )pkt[0 ]->data_len <<CI_TXD_QW1_TX_BUF_SZ_S ); +if (offload) +ci_tx_vec_offload (pkt[0 ], &hi_qw0, single_vlan_pos, qinq_outer_pos); + +__m256i desc2_3 = +_mm256_set_epi64x + (hi_qw3, + pkt[3 ]->buf_iova + pkt[3 ]->data_off, + hi_qw2, + pkt[2 ]->buf_iova + pkt[2 ]->data_off); +__m256i desc0_1 = +_mm256_set_epi64x + (hi_qw1, + pkt[1 ]->buf_iova + pkt[1 ]->data_off, + hi_qw0, + pkt[0 ]->buf_iova + pkt[0 ]->data_off); + _mm256_store_si256(RTE_CAST_PTR(__m256i *, txdp + 2), desc2_3); + _mm256_store_si256(RTE_CAST_PTR(__m256i *, txdp), desc0_1); + } + +/* do any last ones */ +while (nb_pkts) { +ci_vtx1 (txdp, *pkt, flags, offload, single_vlan_pos, qinq_outer_pos); + txdp++; pkt++; nb_pkts--; + } +} + +static __rte_always_inline void +ci_vtx1_ctx_avx2 (volatile struct ci_tx_desc *txdp,struct rte_mbuf *pkt, +uint64_t flags,bool offload,enum ci_l2tag_pos single_vlan_pos, +enum ci_l2tag_pos qinq_outer_pos,bool ptype_lldp_enabled) +{ +uint64_t high_ctx_qw =CI_TX_DESC_DTYPE_CTX ; +uint64_t low_ctx_qw =0 ; + +if (offload) { +ci_fill_ctx_desc_tunneling (&low_ctx_qw, pkt); +if (pkt->ol_flags &RTE_MBUF_F_TX_QINQ ) { +uint64_t qinq_tag = qinq_outer_pos ==CI_TAG_IN_CTX_DESC ? + (uint64_t )pkt->vlan_tci_outer : + (uint64_t )pkt->vlan_tci; + high_ctx_qw |=CI_TX_CTX_DESC_IL2TAG2 <<CI_TXD_QW1_CMD_S ; + low_ctx_qw |= qinq_tag <<CI_TXD_CTX_QW0_L2TAG2_S ; + }else if ((pkt->ol_flags &RTE_MBUF_F_TX_VLAN ) && + single_vlan_pos ==CI_TAG_IN_CTX_DESC ) { + high_ctx_qw |=CI_TX_CTX_DESC_IL2TAG2 <<CI_TXD_QW1_CMD_S ; + low_ctx_qw |= (uint64_t )pkt->vlan_tci <<CI_TXD_CTX_QW0_L2TAG2_S ; + } + } +if (IAVF_CHECK_TX_LLDP (pkt, ptype_lldp_enabled)) + high_ctx_qw |=IAVF_TX_CTX_DESC_SWTCH_UPLINK <<CI_TXD_QW1_CMD_S ; +uint64_t high_data_qw = (CI_TX_DESC_DTYPE_DATA | + ((uint64_t )flags <<CI_TXD_QW1_CMD_S ) | + ((uint64_t )pkt->data_len <<CI_TXD_QW1_TX_BUF_SZ_S )); +if (offload) +ci_tx_vec_offload (pkt, &high_data_qw, single_vlan_pos, qinq_outer_pos); + +__m256i ctx_data_desc =_mm256_set_epi64x (high_data_qw, pkt->buf_iova + pkt->data_off, + high_ctx_qw, low_ctx_qw); + +/* tx_id is always even in ctx mode, so txdp is always 32-byte aligned */ + _mm256_store_si256(RTE_CAST_PTR(__m256i *, txdp), ctx_data_desc); +} + +static __rte_always_inline void +ci_vtx_ctx_avx2 (volatile struct ci_tx_desc *txdp, +struct rte_mbuf **pkt,uint16_t nb_pkts,uint64_t flags, +bool offload,enum ci_l2tag_pos single_vlan_pos,enum ci_l2tag_pos qinq_outer_pos, +bool ptype_lldp_enabled) +{ +uint64_t hi_data_qw_tmpl = (CI_TX_DESC_DTYPE_DATA | (flags <<CI_TXD_QW1_CMD_S )); + +for (; nb_pkts >1 ; txdp +=4 , pkt +=2 , nb_pkts -=2 ) { +uint64_t hi_ctx_qw1 =CI_TX_DESC_DTYPE_CTX ; +uint64_t hi_ctx_qw0 =CI_TX_DESC_DTYPE_CTX ; +uint64_t low_ctx_qw1 =0 ; +uint64_t low_ctx_qw0 =0 ; +uint64_t hi_data_qw1 =0 ; +uint64_t hi_data_qw0 =0 ; + + hi_data_qw1 = hi_data_qw_tmpl | + ((uint64_t )pkt[1 ]->data_len << +CI_TXD_QW1_TX_BUF_SZ_S ); + hi_data_qw0 = hi_data_qw_tmpl | + ((uint64_t )pkt[0 ]->data_len << +CI_TXD_QW1_TX_BUF_SZ_S ); + +if (offload) { +/* tunnel fill assigns low_ctx_qw1; must run before QinQ/VLAN OR below */ +ci_fill_ctx_desc_tunneling (&low_ctx_qw1, pkt[1 ]); +if (pkt[1 ]->ol_flags &RTE_MBUF_F_TX_QINQ ) { +uint64_t qinq_tag = qinq_outer_pos ==CI_TAG_IN_CTX_DESC ? + (uint64_t )pkt[1 ]->vlan_tci_outer : + (uint64_t )pkt[1 ]->vlan_tci; + hi_ctx_qw1 |=CI_TX_CTX_DESC_IL2TAG2 <<CI_TXD_QW1_CMD_S ; + low_ctx_qw1 |= qinq_tag <<CI_TXD_CTX_QW0_L2TAG2_S ; + }else if (pkt[1 ]->ol_flags &RTE_MBUF_F_TX_VLAN && + single_vlan_pos ==CI_TAG_IN_CTX_DESC ) { + hi_ctx_qw1 |=CI_TX_CTX_DESC_IL2TAG2 <<CI_TXD_QW1_CMD_S ; + low_ctx_qw1 |= (uint64_t )pkt[1 ]->vlan_tci <<CI_TXD_CTX_QW0_L2TAG2_S ; + } + } +if (IAVF_CHECK_TX_LLDP (pkt[1 ], ptype_lldp_enabled)) + hi_ctx_qw1 |=IAVF_TX_CTX_DESC_SWTCH_UPLINK <<CI_TXD_QW1_CMD_S ; + +if (offload) { +/* tunnel fill assigns low_ctx_qw0; must run before QinQ/VLAN OR below */ +ci_fill_ctx_desc_tunneling (&low_ctx_qw0, pkt[0 ]); +if (pkt[0 ]->ol_flags &RTE_MBUF_F_TX_QINQ ) { +uint64_t qinq_tag = qinq_outer_pos ==CI_TAG_IN_CTX_DESC ? + (uint64_t )pkt[0 ]->vlan_tci_outer : + (uint64_t )pkt[0 ]->vlan_tci; + hi_ctx_qw0 |=CI_TX_CTX_DESC_IL2TAG2 <<CI_TXD_QW1_CMD_S ; + low_ctx_qw0 |= qinq_tag <<CI_TXD_CTX_QW0_L2TAG2_S ; + }else if (pkt[0 ]->ol_flags &RTE_MBUF_F_TX_VLAN && + single_vlan_pos ==CI_TAG_IN_CTX_DESC ) { + hi_ctx_qw0 |=CI_TX_CTX_DESC_IL2TAG2 <<CI_TXD_QW1_CMD_S ; + low_ctx_qw0 |= (uint64_t )pkt[0 ]->vlan_tci <<CI_TXD_CTX_QW0_L2TAG2_S ; + } + } +if (IAVF_CHECK_TX_LLDP (pkt[0 ], ptype_lldp_enabled)) + hi_ctx_qw0 |=IAVF_TX_CTX_DESC_SWTCH_UPLINK <<CI_TXD_QW1_CMD_S ; + +if (offload) { +ci_tx_vec_offload (pkt[1 ], &hi_data_qw1, single_vlan_pos, qinq_outer_pos); +ci_tx_vec_offload (pkt[0 ], &hi_data_qw0, single_vlan_pos, qinq_outer_pos); + } + +__m256i desc2_3 =_mm256_set_epi64x + (hi_data_qw1, pkt[1 ]->buf_iova + pkt[1 ]->data_off, + hi_ctx_qw1, low_ctx_qw1); +__m256i desc0_1 =_mm256_set_epi64x + (hi_data_qw0, pkt[0 ]->buf_iova + pkt[0 ]->data_off, + hi_ctx_qw0, low_ctx_qw0); + _mm256_store_si256(RTE_CAST_PTR(__m256i *, txdp + 2), desc2_3); + _mm256_store_si256(RTE_CAST_PTR(__m256i *, txdp), desc0_1); + } + +if (nb_pkts) +ci_vtx1_ctx_avx2 (txdp, *pkt, flags, offload, + single_vlan_pos, qinq_outer_pos, ptype_lldp_enabled); +} + +#endif /* __AVX2__ */ + +#ifdef __AVX512VL__ + +static __rte_always_inline void +ci_vtx_avx512 (volatile struct ci_tx_desc *txdp, +struct rte_mbuf **pkt,uint16_t nb_pkts,uint64_t flags, +bool offload,enum ci_l2tag_pos single_vlan_pos,enum ci_l2tag_pos qinq_outer_pos) +{ +const uint64_t hi_qw_tmpl = (CI_TX_DESC_DTYPE_DATA | (flags <<CI_TXD_QW1_CMD_S )); + +/* if unaligned on 32-bit boundary, do one to align */ +if (((uintptr_t )txdp &0x1F ) !=0 && nb_pkts !=0 ) { +ci_vtx1 (txdp, *pkt, flags, offload, single_vlan_pos, qinq_outer_pos); + nb_pkts--; txdp++; pkt++; + } + +/* do 4 at a time while possible, in bursts */ +for (; nb_pkts >3 ; txdp +=4 , pkt +=4 , nb_pkts -=4 ) { +uint64_t hi_qw3 = hi_qw_tmpl | + ((uint64_t )pkt[3 ]->data_len <<CI_TXD_QW1_TX_BUF_SZ_S ); +uint64_t hi_qw2 = hi_qw_tmpl | + ((uint64_t )pkt[2 ]->data_len <<CI_TXD_QW1_TX_BUF_SZ_S ); +uint64_t hi_qw1 = hi_qw_tmpl | + ((uint64_t )pkt[1 ]->data_len <<CI_TXD_QW1_TX_BUF_SZ_S ); +uint64_t hi_qw0 = hi_qw_tmpl | + ((uint64_t )pkt[0 ]->data_len <<CI_TXD_QW1_TX_BUF_SZ_S ); +if (offload) { +ci_tx_vec_offload (pkt[3 ], &hi_qw3, single_vlan_pos, qinq_outer_pos); +ci_tx_vec_offload (pkt[2 ], &hi_qw2, single_vlan_pos, qinq_outer_pos); +ci_tx_vec_offload (pkt[1 ], &hi_qw1, single_vlan_pos, qinq_outer_pos); +ci_tx_vec_offload (pkt[0 ], &hi_qw0, single_vlan_pos, qinq_outer_pos); + } + +__m512i desc0_3 = +_mm512_set_epi64 + (hi_qw3, + pkt[3 ]->buf_iova + pkt[3 ]->data_off, + hi_qw2, + pkt[2 ]->buf_iova + pkt[2 ]->data_off, + hi_qw1, + pkt[1 ]->buf_iova + pkt[1 ]->data_off, + hi_qw0, + pkt[0 ]->buf_iova + pkt[0 ]->data_off); + _mm512_storeu_si512(RTE_CAST_PTR(void *, txdp), desc0_3); + } + +/* do any last ones */ +while (nb_pkts) { +ci_vtx1 (txdp, *pkt, flags, offload, single_vlan_pos, qinq_outer_pos); + txdp++; pkt++; nb_pkts--; + } +} + +static __rte_always_inline void +ci_vtx1_ctx_avx512 (volatile struct ci_tx_desc *txdp,struct rte_mbuf *pkt, +uint64_t flags,bool offload,enum ci_l2tag_pos single_vlan_pos, +enum ci_l2tag_pos qinq_outer_pos,bool lldp_enabled) +{ +uint64_t high_ctx_qw =CI_TX_DESC_DTYPE_CTX ; +uint64_t low_ctx_qw =0 ; + +if (offload) { +ci_fill_ctx_desc_tunneling (&low_ctx_qw, pkt); +if (pkt->ol_flags &RTE_MBUF_F_TX_QINQ ) { +uint64_t qinq_tag = qinq_outer_pos ==CI_TAG_IN_CTX_DESC ? + (uint64_t )pkt->vlan_tci_outer : + (uint64_t )pkt->vlan_tci; + high_ctx_qw |=CI_TX_CTX_DESC_IL2TAG2 <<CI_TXD_QW1_CMD_S ; + low_ctx_qw |= qinq_tag <<CI_TXD_CTX_QW0_L2TAG2_S ; + }else if ((pkt->ol_flags &RTE_MBUF_F_TX_VLAN ) && + single_vlan_pos ==CI_TAG_IN_CTX_DESC ) { + high_ctx_qw |=CI_TX_CTX_DESC_IL2TAG2 <<CI_TXD_QW1_CMD_S ; + low_ctx_qw |= (uint64_t )pkt->vlan_tci <<CI_TXD_CTX_QW0_L2TAG2_S ; + } + } +if (IAVF_CHECK_TX_LLDP (pkt, lldp_enabled)) + high_ctx_qw |=IAVF_TX_CTX_DESC_SWTCH_UPLINK <<CI_TXD_QW1_CMD_S ; +uint64_t high_data_qw = (CI_TX_DESC_DTYPE_DATA | + ((uint64_t )flags <<CI_TXD_QW1_CMD_S ) | + ((uint64_t )pkt->data_len <<CI_TXD_QW1_TX_BUF_SZ_S )); +if (offload) +ci_tx_vec_offload (pkt, &high_data_qw, single_vlan_pos, qinq_outer_pos); + +__m256i ctx_data_desc =_mm256_set_epi64x + (high_data_qw, pkt->buf_iova + pkt->data_off, + high_ctx_qw, low_ctx_qw); + +/* tx_id is always even in ctx mode, so txdp is always 32-byte aligned */ + _mm256_store_si256(RTE_CAST_PTR(__m256i *, txdp), ctx_data_desc); +} + +static __rte_always_inline void +ci_vtx_ctx_avx512 (volatile struct ci_tx_desc *txdp, +struct rte_mbuf **pkt,uint16_t nb_pkts,uint64_t flags, +bool offload,enum ci_l2tag_pos single_vlan_pos,enum ci_l2tag_pos qinq_outer_pos, +bool lldp_enabled) +{ +uint64_t hi_data_qw_tmpl = (CI_TX_DESC_DTYPE_DATA | (flags <<CI_TXD_QW1_CMD_S )); + +for (; nb_pkts >1 ; txdp +=4 , pkt +=2 , nb_pkts -=2 ) { +uint64_t hi_ctx_qw1 =CI_TX_DESC_DTYPE_CTX ; +uint64_t hi_ctx_qw0 =CI_TX_DESC_DTYPE_CTX ; +uint64_t low_ctx_qw1 =0 ; +uint64_t low_ctx_qw0 =0 ; +uint64_t hi_data_qw1 =0 ; +uint64_t hi_data_qw0 =0 ; + + hi_data_qw1 = hi_data_qw_tmpl | + ((uint64_t )pkt[1 ]->data_len <<CI_TXD_QW1_TX_BUF_SZ_S ); + hi_data_qw0 = hi_data_qw_tmpl | + ((uint64_t )pkt[0 ]->data_len <<CI_TXD_QW1_TX_BUF_SZ_S ); + +if (offload) { +/* tunnel fill assigns low_ctx_qw1; must run before QinQ/VLAN OR below */ +ci_fill_ctx_desc_tunneling (&low_ctx_qw1, pkt[1 ]); +if (pkt[1 ]->ol_flags &RTE_MBUF_F_TX_QINQ ) { +uint64_t qinq_tag = qinq_outer_pos ==CI_TAG_IN_CTX_DESC ? + (uint64_t )pkt[1 ]->vlan_tci_outer : + (uint64_t )pkt[1 ]->vlan_tci; + hi_ctx_qw1 |=CI_TX_CTX_DESC_IL2TAG2 <<CI_TXD_QW1_CMD_S ; + low_ctx_qw1 |= qinq_tag <<CI_TXD_CTX_QW0_L2TAG2_S ; + }else if (pkt[1 ]->ol_flags &RTE_MBUF_F_TX_VLAN && + single_vlan_pos ==CI_TAG_IN_CTX_DESC ) { + hi_ctx_qw1 |=CI_TX_CTX_DESC_IL2TAG2 <<CI_TXD_QW1_CMD_S ; + low_ctx_qw1 |= (uint64_t )pkt[1 ]->vlan_tci <<CI_TXD_CTX_QW0_L2TAG2_S ; + } + } +if (IAVF_CHECK_TX_LLDP (pkt[1 ], lldp_enabled)) + hi_ctx_qw1 |=IAVF_TX_CTX_DESC_SWTCH_UPLINK <<CI_TXD_QW1_CMD_S ; + +if (offload) { +/* tunnel fill assigns low_ctx_qw0; must run before QinQ/VLAN OR below */ +ci_fill_ctx_desc_tunneling (&low_ctx_qw0, pkt[0 ]); +if (pkt[0 ]->ol_flags &RTE_MBUF_F_TX_QINQ ) { +uint64_t qinq_tag = qinq_outer_pos ==CI_TAG_IN_CTX_DESC ? + (uint64_t )pkt[0 ]->vlan_tci_outer : + (uint64_t )pkt[0 ]->vlan_tci; + hi_ctx_qw0 |=CI_TX_CTX_DESC_IL2TAG2 <<CI_TXD_QW1_CMD_S ; + low_ctx_qw0 |= qinq_tag <<CI_TXD_CTX_QW0_L2TAG2_S ; + }else if (pkt[0 ]->ol_flags &RTE_MBUF_F_TX_VLAN && + single_vlan_pos ==CI_TAG_IN_CTX_DESC ) { + hi_ctx_qw0 |=CI_TX_CTX_DESC_IL2TAG2 <<CI_TXD_QW1_CMD_S ; + low_ctx_qw0 |= (uint64_t )pkt[0 ]->vlan_tci <<CI_TXD_CTX_QW0_L2TAG2_S ; + } + } +if (IAVF_CHECK_TX_LLDP (pkt[0 ], lldp_enabled)) + hi_ctx_qw0 |=IAVF_TX_CTX_DESC_SWTCH_UPLINK <<CI_TXD_QW1_CMD_S ; + +if (offload) { +ci_tx_vec_offload (pkt[1 ], &hi_data_qw1, single_vlan_pos, qinq_outer_pos); +ci_tx_vec_offload (pkt[0 ], &hi_data_qw0, single_vlan_pos, qinq_outer_pos); + } + +__m512i desc0_3 =_mm512_set_epi64 + (hi_data_qw1, pkt[1 ]->buf_iova + pkt[1 ]->data_off, + hi_ctx_qw1, low_ctx_qw1, + hi_data_qw0, pkt[0 ]->buf_iova + pkt[0 ]->data_off, + hi_ctx_qw0, low_ctx_qw0); + _mm512_storeu_si512(RTE_CAST_PTR(void *, txdp), desc0_3); + } + +if (nb_pkts) +ci_vtx1_ctx_avx512 (txdp, *pkt, flags, offload, + single_vlan_pos, qinq_outer_pos, lldp_enabled); +} + +#endif /* __AVX512VL__ */ + +#endif /* _COMMON_INTEL_TX_VEC_X86_H_ */
drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c +7 -278
326 lines not on this page · raw message
drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c +8 -264
309 lines not on this page · raw message
drivers/net/intel/iavf/iavf_rxtx_vec_common.h +4 -71
85 lines not on this page · raw message
Signature · 1 line
2.53.0
PATCH v2 7/15 net/intel: use function callback for lldp
In reply to Bruce RichardsonLimit LLDP support for iavf drivers for now, by using a callback to handle its processing. Drivers not supporting this pass a NULL pointer and the compiler will optimize away the branches. For iavf, the compiler can inline the subfunction directly as it's known at compile-time.
Signed-off-by Bruce Richardson <[email protected]>
--- drivers/net/intel/common/tx_vec_x86.h | 42 +++++++++++-------- drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c | 16 +++++-- drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c | 17 +++++--- 3 files changed, 48 insertions(+), 27 deletions(-)
drivers/net/intel/common/tx_vec_x86.h +24 -18
@@ -11,6 +11,12 @@#include "tx.h" +/* Optional per-driver LLDP switch-uplink check for ctx descriptors. +* Takes the ctx-desc high qword and returns it with the LLDP bit applied +* if appropriate. NULL disables the check. +*/ +typedef uint64_t (*ci_tx_ctx_lldp_fn )(struct rte_mbuf *pkt,uint64_t high_ctx_qw); + static __rte_always_inline voidci_fill_ctx_desc_tunneling (uint64_t *low_ctx_qw,struct rte_mbuf *pkt) {
@@ -226,7 +232,7 @@ ci_vtx_avx2(volatile struct ci_tx_desc *txdp, static __rte_always_inline voidci_vtx1_ctx_avx2 (volatile struct ci_tx_desc *txdp,struct rte_mbuf *pkt,uint64_t flags,bool offload,enum ci_l2tag_pos single_vlan_pos, -enum ci_l2tag_pos qinq_outer_pos,bool ptype_lldp_enabled) +enum ci_l2tag_pos qinq_outer_pos,ci_tx_ctx_lldp_fn lldp_check) {uint64_t high_ctx_qw =CI_TX_DESC_DTYPE_CTX ;uint64_t low_ctx_qw =0 ;
@@ -245,8 +251,8 @@ ci_vtx1_ctx_avx2(volatile struct ci_tx_desc *txdp, struct rte_mbuf *pkt, low_ctx_qw |= (uint64_t )pkt->vlan_tci <<CI_TXD_CTX_QW0_L2TAG2_S ; } } -if (IAVF_CHECK_TX_LLDP (pkt, ptype_lldp_enabled)) - high_ctx_qw |=IAVF_TX_CTX_DESC_SWTCH_UPLINK <<CI_TXD_QW1_CMD_S ; +if (lldp_check !=NULL ) + high_ctx_qw =lldp_check (pkt, high_ctx_qw);uint64_t high_data_qw = (CI_TX_DESC_DTYPE_DATA | ((uint64_t )flags <<CI_TXD_QW1_CMD_S ) | ((uint64_t )pkt->data_len <<CI_TXD_QW1_TX_BUF_SZ_S ));
@@ -264,7 +270,7 @@ static __rte_always_inline void ci_vtx_ctx_avx2(volatile struct ci_tx_desc *txdp, struct rte_mbuf **pkt, uint16_t nb_pkts, uint64_t flags, bool offload, enum ci_l2tag_pos single_vlan_pos, enum ci_l2tag_pos qinq_outer_pos, - bool ptype_lldp_enabled) + ci_tx_ctx_lldp_fn lldp_check) { uint64_t hi_data_qw_tmpl = (CI_TX_DESC_DTYPE_DATA | (flags << CI_TXD_QW1_CMD_S));
@@ -298,8 +304,8 @@ ci_vtx_ctx_avx2(volatile struct ci_tx_desc *txdp, low_ctx_qw1 |= (uint64_t )pkt[1 ]->vlan_tci <<CI_TXD_CTX_QW0_L2TAG2_S ; } } -if (IAVF_CHECK_TX_LLDP (pkt[1 ], ptype_lldp_enabled)) - hi_ctx_qw1 |=IAVF_TX_CTX_DESC_SWTCH_UPLINK <<CI_TXD_QW1_CMD_S ; +if (lldp_check !=NULL ) + hi_ctx_qw1 =lldp_check (pkt[1 ], hi_ctx_qw1);if (offload) {/* tunnel fill assigns low_ctx_qw0; must run before QinQ/VLAN OR below */
@@ -316,8 +322,8 @@ ci_vtx_ctx_avx2(volatile struct ci_tx_desc *txdp, low_ctx_qw0 |= (uint64_t )pkt[0 ]->vlan_tci <<CI_TXD_CTX_QW0_L2TAG2_S ; } } -if (IAVF_CHECK_TX_LLDP (pkt[0 ], ptype_lldp_enabled)) - hi_ctx_qw0 |=IAVF_TX_CTX_DESC_SWTCH_UPLINK <<CI_TXD_QW1_CMD_S ; +if (lldp_check !=NULL ) + hi_ctx_qw0 =lldp_check (pkt[0 ], hi_ctx_qw0);if (offload) {ci_tx_vec_offload (pkt[1 ], &hi_data_qw1, single_vlan_pos, qinq_outer_pos);
@@ -336,7 +342,7 @@ ci_vtx_ctx_avx2(volatile struct ci_tx_desc *txdp,if (nb_pkts)ci_vtx1_ctx_avx2 (txdp, *pkt, flags, offload, - single_vlan_pos, qinq_outer_pos, ptype_lldp_enabled); + single_vlan_pos, qinq_outer_pos, lldp_check); }#endif /* __AVX2__ */
@@ -396,7 +402,7 @@ ci_vtx_avx512(volatile struct ci_tx_desc *txdp, static __rte_always_inline voidci_vtx1_ctx_avx512 (volatile struct ci_tx_desc *txdp,struct rte_mbuf *pkt,uint64_t flags,bool offload,enum ci_l2tag_pos single_vlan_pos, -enum ci_l2tag_pos qinq_outer_pos,bool lldp_enabled) +enum ci_l2tag_pos qinq_outer_pos,ci_tx_ctx_lldp_fn lldp_check) {uint64_t high_ctx_qw =CI_TX_DESC_DTYPE_CTX ;uint64_t low_ctx_qw =0 ;
@@ -415,8 +421,8 @@ ci_vtx1_ctx_avx512(volatile struct ci_tx_desc *txdp, struct rte_mbuf *pkt, low_ctx_qw |= (uint64_t )pkt->vlan_tci <<CI_TXD_CTX_QW0_L2TAG2_S ; } } -if (IAVF_CHECK_TX_LLDP (pkt, lldp_enabled)) - high_ctx_qw |=IAVF_TX_CTX_DESC_SWTCH_UPLINK <<CI_TXD_QW1_CMD_S ; +if (lldp_check !=NULL ) + high_ctx_qw =lldp_check (pkt, high_ctx_qw);uint64_t high_data_qw = (CI_TX_DESC_DTYPE_DATA | ((uint64_t )flags <<CI_TXD_QW1_CMD_S ) | ((uint64_t )pkt->data_len <<CI_TXD_QW1_TX_BUF_SZ_S ));
@@ -435,7 +441,7 @@ static __rte_always_inline void ci_vtx_ctx_avx512(volatile struct ci_tx_desc *txdp, struct rte_mbuf **pkt, uint16_t nb_pkts, uint64_t flags, bool offload, enum ci_l2tag_pos single_vlan_pos, enum ci_l2tag_pos qinq_outer_pos, - bool lldp_enabled) + ci_tx_ctx_lldp_fn lldp_check) { uint64_t hi_data_qw_tmpl = (CI_TX_DESC_DTYPE_DATA | (flags << CI_TXD_QW1_CMD_S));
@@ -467,8 +473,8 @@ ci_vtx_ctx_avx512(volatile struct ci_tx_desc *txdp, low_ctx_qw1 |= (uint64_t )pkt[1 ]->vlan_tci <<CI_TXD_CTX_QW0_L2TAG2_S ; } } -if (IAVF_CHECK_TX_LLDP (pkt[1 ], lldp_enabled)) - hi_ctx_qw1 |=IAVF_TX_CTX_DESC_SWTCH_UPLINK <<CI_TXD_QW1_CMD_S ; +if (lldp_check !=NULL ) + hi_ctx_qw1 =lldp_check (pkt[1 ], hi_ctx_qw1);if (offload) {/* tunnel fill assigns low_ctx_qw0; must run before QinQ/VLAN OR below */
@@ -485,8 +491,8 @@ ci_vtx_ctx_avx512(volatile struct ci_tx_desc *txdp, low_ctx_qw0 |= (uint64_t )pkt[0 ]->vlan_tci <<CI_TXD_CTX_QW0_L2TAG2_S ; } } -if (IAVF_CHECK_TX_LLDP (pkt[0 ], lldp_enabled)) - hi_ctx_qw0 |=IAVF_TX_CTX_DESC_SWTCH_UPLINK <<CI_TXD_QW1_CMD_S ; +if (lldp_check !=NULL ) + hi_ctx_qw0 =lldp_check (pkt[0 ], hi_ctx_qw0);if (offload) {ci_tx_vec_offload (pkt[1 ], &hi_data_qw1, single_vlan_pos, qinq_outer_pos);
@@ -503,7 +509,7 @@ ci_vtx_ctx_avx512(volatile struct ci_tx_desc *txdp,if (nb_pkts)ci_vtx1_ctx_avx512 (txdp, *pkt, flags, offload, - single_vlan_pos, qinq_outer_pos, lldp_enabled); + single_vlan_pos, qinq_outer_pos, lldp_check); }#endif /* __AVX512VL__ */
drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c +12 -4
@@ -1685,6 +1685,14 @@ iavf_xmit_fixed_burst_vec_avx2(void *tx_queue, struct rte_mbuf **tx_pkts,return nb_pkts; } +static __rte_always_inline uint64_t +iavf_tx_ctx_lldp_check (struct rte_mbuf *pkt,uint64_t high_ctx_qw) +{ +if (IAVF_CHECK_TX_LLDP (pkt, true)) + high_ctx_qw |=IAVF_TX_CTX_DESC_SWTCH_UPLINK <<CI_TXD_QW1_CMD_S ; +return high_ctx_qw; +} + static __rte_always_inline uint16_tiavf_xmit_fixed_burst_vec_avx2_ctx (void *tx_queue,struct rte_mbuf **tx_pkts,uint16_t nb_pkts,bool offload)
@@ -1696,7 +1704,7 @@ iavf_xmit_fixed_burst_vec_avx2_ctx(void *tx_queue, struct rte_mbuf **tx_pkts,/* bit2 is reserved and must be set to 1 according to Spec */ uint64_t flags =IAVF_TX_DESC_CMD_EOP |IAVF_TX_DESC_CMD_ICRC ;uint64_t rs =IAVF_TX_DESC_CMD_RS | flags; -bool lldp_enabled = txq->lldp_enabled; +ci_tx_ctx_lldp_fn lldp_check = txq->lldp_enabled ? iavf_tx_ctx_lldp_check :NULL ;/* vlan_flag gives both the single-VLAN and the QinQ outer tag position */ enum ci_l2tag_pos vlan_pos = (txq->vlan_flag &IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG1 ) ?CI_TAG_IN_DATA_DESC :CI_TAG_IN_CTX_DESC ;
@@ -1723,10 +1731,10 @@ iavf_xmit_fixed_burst_vec_avx2_ctx(void *tx_queue, struct rte_mbuf **tx_pkts,ci_tx_backlog_entry_vec (txep, tx_pkts, nb_mbuf);ci_vtx_ctx_avx2 (txdp, tx_pkts, nb_mbuf -1 , flags, offload, - vlan_pos, vlan_pos, lldp_enabled); + vlan_pos, vlan_pos, lldp_check); tx_pkts += (nb_mbuf -1 ); txdp += (n -2 ); -ci_vtx1_ctx_avx2 (txdp, *tx_pkts++, rs, offload, vlan_pos, vlan_pos, lldp_enabled); +ci_vtx1_ctx_avx2 (txdp, *tx_pkts++, rs, offload, vlan_pos, vlan_pos, lldp_check); nb_commit = (uint16_t )(nb_commit - n);
@@ -1740,7 +1748,7 @@ iavf_xmit_fixed_burst_vec_avx2_ctx(void *tx_queue, struct rte_mbuf **tx_pkts, nb_mbuf = nb_commit >>1 ;ci_tx_backlog_entry_vec (txep, tx_pkts, nb_mbuf); -ci_vtx_ctx_avx2 (txdp, tx_pkts, nb_mbuf, flags, offload, vlan_pos, vlan_pos, lldp_enabled); +ci_vtx_ctx_avx2 (txdp, tx_pkts, nb_mbuf, flags, offload, vlan_pos, vlan_pos, lldp_check); tx_id = (uint16_t )(tx_id + nb_commit);if (tx_id > txq->tx_next_rs) {
drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c +12 -5
@@ -1900,6 +1900,14 @@ iavf_xmit_fixed_burst_vec_avx512(void *tx_queue, struct rte_mbuf **tx_pkts,return nb_pkts; } +static __rte_always_inline uint64_t +iavf_tx_ctx_lldp_check (struct rte_mbuf *pkt,uint64_t high_ctx_qw) +{ +if (IAVF_CHECK_TX_LLDP (pkt, true)) + high_ctx_qw |=IAVF_TX_CTX_DESC_SWTCH_UPLINK <<CI_TXD_QW1_CMD_S ; +return high_ctx_qw; +} + static __rte_always_inline uint16_tiavf_xmit_fixed_burst_vec_avx512_ctx (void *tx_queue,struct rte_mbuf **tx_pkts,uint16_t nb_pkts,bool offload)
@@ -1911,7 +1919,7 @@ iavf_xmit_fixed_burst_vec_avx512_ctx(void *tx_queue, struct rte_mbuf **tx_pkts,/* bit2 is reserved and must be set to 1 according to Spec */ uint64_t flags =CI_TX_DESC_CMD_EOP |CI_TX_DESC_CMD_ICRC ;uint64_t rs =CI_TX_DESC_CMD_RS | flags; -bool lldp_enabled = txq->lldp_enabled; +ci_tx_ctx_lldp_fn lldp_check = txq->lldp_enabled ? iavf_tx_ctx_lldp_check :NULL ;/* vlan_flag gives both the single-VLAN and the QinQ outer tag position */ enum ci_l2tag_pos vlan_pos = (txq->vlan_flag &IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG1 ) ?CI_TAG_IN_DATA_DESC :CI_TAG_IN_CTX_DESC ;
@@ -1938,11 +1946,10 @@ iavf_xmit_fixed_burst_vec_avx512_ctx(void *tx_queue, struct rte_mbuf **tx_pkts,tx_backlog_entry_avx512 (txep, tx_pkts, nb_mbuf);ci_vtx_ctx_avx512 (txdp, tx_pkts, nb_mbuf -1 , flags, offload, - vlan_pos, vlan_pos, lldp_enabled); + vlan_pos, vlan_pos, lldp_check); tx_pkts += (nb_mbuf -1 ); txdp += (n -2 ); -ci_vtx1_ctx_avx512 (txdp, *tx_pkts++, rs, offload, - vlan_pos, vlan_pos, lldp_enabled); +ci_vtx1_ctx_avx512 (txdp, *tx_pkts++, rs, offload, vlan_pos, vlan_pos, lldp_check); nb_commit = (uint16_t )(nb_commit - n);
@@ -1956,7 +1963,7 @@ iavf_xmit_fixed_burst_vec_avx512_ctx(void *tx_queue, struct rte_mbuf **tx_pkts, nb_mbuf = nb_commit >>1 ;tx_backlog_entry_avx512 (txep, tx_pkts, nb_mbuf); -ci_vtx_ctx_avx512 (txdp, tx_pkts, nb_mbuf, flags, offload, vlan_pos, vlan_pos, lldp_enabled); +ci_vtx_ctx_avx512 (txdp, tx_pkts, nb_mbuf, flags, offload, vlan_pos, vlan_pos, lldp_check); tx_id = (uint16_t )(tx_id + nb_commit);if (tx_id > txq->tx_next_rs) {
Signature · 1 line
2.53.0
RE: [PATCH v2 07/15] net/intel: use function callback for lldp
Quoted 8 lines
Subject: [PATCH v2 07/15] net/intel: use function callback for lldp
Limit LLDP support for iavf drivers for now, by using a callback to handle its processing. Drivers not supporting this pass a NULL pointer and the compiler will optimize away the branches. For iavf, the compiler can inline the subfunction directly as it's known at compile-time.
Signed-off-by Bruce Richardson <[email protected]>
Acked-by Ciara Loftus <[email protected]>
Quoted 6 lines
--- drivers/net/intel/common/tx_vec_x86.h | 42 +++++++++++-------- drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c | 16 +++++-- drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c | 17 +++++--- 3 files changed, 48 insertions(+), 27 deletions(-)
PATCH v2 8/15 net/intel: allow building without IOVA in mbuf
In reply to Bruce RichardsonEnsure correct macros are used to build the vector Tx code without the IOVA stored explicitly in the mbuf. This allows the iavf driver to build when the config setting for IOVA in mbuf is disabled.
Signed-off-by Bruce Richardson <[email protected]>
--- drivers/net/intel/common/tx_vec_x86.h | 30 +++++++++++++-------------- drivers/net/intel/iavf/meson.build | 4 +--- 2 files changed, 16 insertions(+), 18 deletions(-)
drivers/net/intel/common/tx_vec_x86.h +15 -15
@@ -168,7 +168,7 @@ ci_vtx1(volatile struct ci_tx_desc *txdp,if (offload)ci_tx_vec_offload (pkt, &high_qw, single_vlan_pos, qinq_outer_pos); -__m128i descriptor =_mm_set_epi64x (high_qw, pkt->buf_iova + pkt->data_off); +__m128i descriptor =_mm_set_epi64x (high_qw,rte_pktmbuf_iova (pkt)); _mm_store_si128(RTE_CAST_PTR(__m128i *, txdp), descriptor); }
@@ -209,15 +209,15 @@ ci_vtx_avx2(volatile struct ci_tx_desc *txdp,__m256i desc2_3 =_mm256_set_epi64x (hi_qw3, - pkt[3 ]->buf_iova + pkt[3 ]->data_off, +rte_pktmbuf_iova (pkt[3 ]), hi_qw2, - pkt[2 ]->buf_iova + pkt[2 ]->data_off); +rte_pktmbuf_iova (pkt[2 ]));__m256i desc0_1 =_mm256_set_epi64x (hi_qw1, - pkt[1 ]->buf_iova + pkt[1 ]->data_off, +rte_pktmbuf_iova (pkt[1 ]), hi_qw0, - pkt[0 ]->buf_iova + pkt[0 ]->data_off); +rte_pktmbuf_iova (pkt[0 ])); _mm256_store_si256(RTE_CAST_PTR(__m256i *, txdp + 2), desc2_3); _mm256_store_si256(RTE_CAST_PTR(__m256i *, txdp), desc0_1); }
@@ -259,7 +259,7 @@ ci_vtx1_ctx_avx2(volatile struct ci_tx_desc *txdp, struct rte_mbuf *pkt,if (offload)ci_tx_vec_offload (pkt, &high_data_qw, single_vlan_pos, qinq_outer_pos); -__m256i ctx_data_desc =_mm256_set_epi64x (high_data_qw, pkt->buf_iova + pkt->data_off, +__m256i ctx_data_desc =_mm256_set_epi64x (high_data_qw,rte_pktmbuf_iova (pkt), high_ctx_qw, low_ctx_qw);/* tx_id is always even in ctx mode, so txdp is always 32-byte aligned */
@@ -331,10 +331,10 @@ ci_vtx_ctx_avx2(volatile struct ci_tx_desc *txdp, }__m256i desc2_3 =_mm256_set_epi64x - (hi_data_qw1, pkt[1 ]->buf_iova + pkt[1 ]->data_off, + (hi_data_qw1,rte_pktmbuf_iova (pkt[1 ]), hi_ctx_qw1, low_ctx_qw1);__m256i desc0_1 =_mm256_set_epi64x - (hi_data_qw0, pkt[0 ]->buf_iova + pkt[0 ]->data_off, + (hi_data_qw0,rte_pktmbuf_iova (pkt[0 ]), hi_ctx_qw0, low_ctx_qw0); _mm256_store_si256(RTE_CAST_PTR(__m256i *, txdp + 2), desc2_3); _mm256_store_si256(RTE_CAST_PTR(__m256i *, txdp), desc0_1);
@@ -382,13 +382,13 @@ ci_vtx_avx512(volatile struct ci_tx_desc *txdp,__m512i desc0_3 =_mm512_set_epi64 (hi_qw3, - pkt[3 ]->buf_iova + pkt[3 ]->data_off, +rte_pktmbuf_iova (pkt[3 ]), hi_qw2, - pkt[2 ]->buf_iova + pkt[2 ]->data_off, +rte_pktmbuf_iova (pkt[2 ]), hi_qw1, - pkt[1 ]->buf_iova + pkt[1 ]->data_off, +rte_pktmbuf_iova (pkt[1 ]), hi_qw0, - pkt[0 ]->buf_iova + pkt[0 ]->data_off); +rte_pktmbuf_iova (pkt[0 ])); _mm512_storeu_si512(RTE_CAST_PTR(void *, txdp), desc0_3); }
@@ -430,7 +430,7 @@ ci_vtx1_ctx_avx512(volatile struct ci_tx_desc *txdp, struct rte_mbuf *pkt,ci_tx_vec_offload (pkt, &high_data_qw, single_vlan_pos, qinq_outer_pos);__m256i ctx_data_desc =_mm256_set_epi64x - (high_data_qw, pkt->buf_iova + pkt->data_off, + (high_data_qw,rte_pktmbuf_iova (pkt), high_ctx_qw, low_ctx_qw);/* tx_id is always even in ctx mode, so txdp is always 32-byte aligned */
@@ -500,9 +500,9 @@ ci_vtx_ctx_avx512(volatile struct ci_tx_desc *txdp, }__m512i desc0_3 =_mm512_set_epi64 - (hi_data_qw1, pkt[1 ]->buf_iova + pkt[1 ]->data_off, + (hi_data_qw1,rte_pktmbuf_iova (pkt[1 ]), hi_ctx_qw1, low_ctx_qw1, - hi_data_qw0, pkt[0 ]->buf_iova + pkt[0 ]->data_off, + hi_data_qw0,rte_pktmbuf_iova (pkt[0 ]), hi_ctx_qw0, low_ctx_qw0); _mm512_storeu_si512(RTE_CAST_PTR(void *, txdp), desc0_3); }
drivers/net/intel/iavf/meson.build +1 -3
@@ -1,9 +1,7 @@ # SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2018 Luca Boccassi <[email protected]> -if dpdk_conf.get('RTE_IOVA_IN_MBUF') == 0 - subdir_done() -endif +require_iova_in_mbuf = false testpmd_sources = files('iavf_testpmd.c')
Signature · 1 line
2.53.0
RE: [PATCH v2 08/15] net/intel: allow building without IOVA in mbuf
Quoted 7 lines
Subject: [PATCH v2 08/15] net/intel: allow building without IOVA in mbuf
Ensure correct macros are used to build the vector Tx code without the IOVA stored explicitly in the mbuf. This allows the iavf driver to build when the config setting for IOVA in mbuf is disabled.
Signed-off-by Bruce Richardson <[email protected]>
Acked-by Ciara Loftus <[email protected]>
Quoted 5 lines
--- drivers/net/intel/common/tx_vec_x86.h | 30 +++++++++++++-------------- drivers/net/intel/iavf/meson.build | 4 +--- 2 files changed, 16 insertions(+), 18 deletions(-)
PATCH v2 9/15 net/ice: use common descriptor creation functions
In reply to Bruce RichardsonUpdate ice driver to use the descriptor creation functions from common rather than having it maintain its own copies of them.
Signed-off-by Bruce Richardson <[email protected]>
--- drivers/net/intel/ice/ice_rxtx_vec_avx2.c | 73 ++------------------- drivers/net/intel/ice/ice_rxtx_vec_avx512.c | 64 ++---------------- drivers/net/intel/ice/ice_rxtx_vec_common.h | 59 ----------------- 3 files changed, 12 insertions(+), 184 deletions(-)
drivers/net/intel/ice/ice_rxtx_vec_avx2.c +6 -67
@@ -5,6 +5,7 @@#include "ice_rxtx_vec_common.h" #include "../common/rx_vec_x86.h" +#include "../common/tx_vec_x86.h" #include <rte_vect.h>
@@ -773,70 +774,6 @@ ice_recv_scattered_pkts_vec_avx2_offload(void *rx_queue, true); } -static __rte_always_inline void -ice_vtx1 (volatile struct ci_tx_desc *txdp, -struct rte_mbuf *pkt,uint64_t flags,bool offload) -{ -uint64_t high_qw = (CI_TX_DESC_DTYPE_DATA | - ((uint64_t )flags <<CI_TXD_QW1_CMD_S ) | - ((uint64_t )pkt->data_len <<CI_TXD_QW1_TX_BUF_SZ_S )); -if (offload) -ice_txd_enable_offload (pkt, &high_qw); - -__m128i descriptor =_mm_set_epi64x (high_qw,rte_pktmbuf_iova (pkt)); - _mm_store_si128(RTE_CAST_PTR(__m128i *, txdp), descriptor); -} - -static __rte_always_inline void -ice_vtx (volatile struct ci_tx_desc *txdp, -struct rte_mbuf **pkt,uint16_t nb_pkts,uint64_t flags,bool offload) -{ -const uint64_t hi_qw_tmpl = (CI_TX_DESC_DTYPE_DATA | (flags <<CI_TXD_QW1_CMD_S )); - -/* if unaligned on 32-bit boundary, do one to align */ -if (((uintptr_t )txdp &0x1F ) !=0 && nb_pkts !=0 ) { -ice_vtx1 (txdp, *pkt, flags, offload); - nb_pkts--; txdp++; pkt++; - } - -/* do four at a time while possible, in bursts */ -for (; nb_pkts >3 ; txdp +=4 , pkt +=4 , nb_pkts -=4 ) { -uint64_t hi_qw3 = hi_qw_tmpl | - ((uint64_t )pkt[3 ]->data_len <<CI_TXD_QW1_TX_BUF_SZ_S ); -if (offload) -ice_txd_enable_offload (pkt[3 ], &hi_qw3); -uint64_t hi_qw2 = hi_qw_tmpl | - ((uint64_t )pkt[2 ]->data_len <<CI_TXD_QW1_TX_BUF_SZ_S ); -if (offload) -ice_txd_enable_offload (pkt[2 ], &hi_qw2); -uint64_t hi_qw1 = hi_qw_tmpl | - ((uint64_t )pkt[1 ]->data_len <<CI_TXD_QW1_TX_BUF_SZ_S ); -if (offload) -ice_txd_enable_offload (pkt[1 ], &hi_qw1); -uint64_t hi_qw0 = hi_qw_tmpl | - ((uint64_t )pkt[0 ]->data_len <<CI_TXD_QW1_TX_BUF_SZ_S ); -if (offload) -ice_txd_enable_offload (pkt[0 ], &hi_qw0); - -__m256i desc2_3 = -_mm256_set_epi64x - (hi_qw3,rte_pktmbuf_iova (pkt[3 ]), - hi_qw2,rte_pktmbuf_iova (pkt[2 ])); -__m256i desc0_1 = -_mm256_set_epi64x - (hi_qw1,rte_pktmbuf_iova (pkt[1 ]), - hi_qw0,rte_pktmbuf_iova (pkt[0 ])); - _mm256_store_si256(RTE_CAST_PTR(__m256i *, txdp + 2), desc2_3); - _mm256_store_si256(RTE_CAST_PTR(__m256i *, txdp), desc0_1); - } - -/* do any last ones */ -while (nb_pkts) { -ice_vtx1 (txdp, *pkt, flags, offload); - txdp++; pkt++; nb_pkts--; - } -} - static __rte_always_inline uint16_tice_xmit_fixed_burst_vec_avx2 (void *tx_queue,struct rte_mbuf **tx_pkts,uint16_t nb_pkts,bool offload)
@@ -868,11 +805,12 @@ ice_xmit_fixed_burst_vec_avx2(void *tx_queue, struct rte_mbuf **tx_pkts,if (nb_commit >= n) {ci_tx_backlog_entry_vec (txep, tx_pkts, n); -ice_vtx (txdp, tx_pkts, n -1 , flags, offload); +ci_vtx_avx2 (txdp, tx_pkts, n -1 , flags, offload, +CI_TAG_IN_DATA_DESC ,CI_TAG_IN_DATA_DESC ); tx_pkts += (n -1 ); txdp += (n -1 ); -ice_vtx1 (txdp, *tx_pkts++, rs, offload); +ci_vtx1 (txdp, *tx_pkts++, rs, offload,CI_TAG_IN_DATA_DESC ,CI_TAG_IN_DATA_DESC ); nb_commit = (uint16_t )(nb_commit - n);
@@ -886,7 +824,8 @@ ice_xmit_fixed_burst_vec_avx2(void *tx_queue, struct rte_mbuf **tx_pkts,ci_tx_backlog_entry_vec (txep, tx_pkts, nb_commit); -ice_vtx (txdp, tx_pkts, nb_commit, flags, offload); +ci_vtx_avx2 (txdp, tx_pkts, nb_commit, flags, offload, +CI_TAG_IN_DATA_DESC ,CI_TAG_IN_DATA_DESC ); tx_id = (uint16_t )(tx_id + nb_commit);if (tx_id > txq->tx_next_rs) {
drivers/net/intel/ice/ice_rxtx_vec_avx512.c +6 -58
91 lines not on this page · raw message
drivers/net/intel/ice/ice_rxtx_vec_common.h +0 -59
63 lines not on this page · raw message
Signature · 1 line
2.53.0
RE: [PATCH v2 09/15] net/ice: use common descriptor creation functions
Quoted 6 lines
Subject: [PATCH v2 09/15] net/ice: use common descriptor creation functions
Update ice driver to use the descriptor creation functions from common rather than having it maintain its own copies of them.
Signed-off-by Bruce Richardson <[email protected]>
Acked-by Ciara Loftus <[email protected]>
Quoted 6 lines
--- drivers/net/intel/ice/ice_rxtx_vec_avx2.c | 73 ++------------------- drivers/net/intel/ice/ice_rxtx_vec_avx512.c | 64 ++---------------- drivers/net/intel/ice/ice_rxtx_vec_common.h | 59 ----------------- 3 files changed, 12 insertions(+), 184 deletions(-)
PATCH v2 10/15 net/intel: move vector Tx paths to common
In reply to Bruce RichardsonThe main loops for vector Tx, both avx2 and avx512 are almost identical across a couple of drivers, so move the iavf copies to common for later reuse by other drivers.
Signed-off-by Bruce Richardson <[email protected]>
Acked-by Ciara Loftus <[email protected]>
--- drivers/net/intel/common/tx.h | 10 ++ drivers/net/intel/common/tx_vec_x86.h | 128 ++++++++++++++++++ drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c | 75 +--------- drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c | 77 +---------- 4 files changed, 148 insertions(+), 142 deletions(-)
drivers/net/intel/common/tx.h +10 -0
@@ -9,6 +9,8 @@#include <rte_mbuf.h> #include <rte_ethdev.h> #include <rte_vect.h> +#include <rte_io.h> +#include <rte_byteorder.h> /* Common TX Descriptor QW1 Field Definitions */ #define CI_TXD_QW1_DTYPE_S 0
@@ -273,6 +275,14 @@ ci_tx_backlog_entry_vec(struct ci_tx_entry_vec *txep, struct rte_mbuf **tx_pkts, txep[i].mbuf = tx_pkts[i]; } + +/* Write the Tx tail register, byte-swapped for hardware regardless of host endianness. */ +static __rte_always_inline void +ci_tx_qtx_tail_write (struct ci_tx_queue *txq,uint16_t tx_id) +{ +rte_write32_wc (rte_cpu_to_le_32 ((uint32_t )tx_id), txq->qtx_tail); +} +#define IETH_VPMD_TX_MAX_FREE_BUF 64typedef int (*ci_desc_done_fn )(struct ci_tx_queue *txq,uint16_t idx);
drivers/net/intel/common/tx_vec_x86.h +128 -0
140 lines not on this page · raw message
drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c +5 -70
93 lines not on this page · raw message
drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c +5 -72
95 lines not on this page · raw message
Signature · 1 line
2.53.0
PATCH v2 11/15 net/ice: use common AVX Tx functions
In reply to Bruce RichardsonUse the newly created common Tx functions in the vector path.
Signed-off-by Bruce Richardson <[email protected]>
Acked-by Ciara Loftus <[email protected]>
--- drivers/net/intel/ice/ice_rxtx_vec_avx2.c | 72 +------------- drivers/net/intel/ice/ice_rxtx_vec_avx512.c | 102 +++----------------- 2 files changed, 14 insertions(+), 160 deletions(-)
drivers/net/intel/ice/ice_rxtx_vec_avx2.c +2 -70
@@ -774,74 +774,6 @@ ice_recv_scattered_pkts_vec_avx2_offload(void *rx_queue, true); } -static __rte_always_inline uint16_t -ice_xmit_fixed_burst_vec_avx2 (void *tx_queue,struct rte_mbuf **tx_pkts, -uint16_t nb_pkts,bool offload) -{ -struct ci_tx_queue *txq = (struct ci_tx_queue *)tx_queue; -volatile struct ci_tx_desc *txdp; -struct ci_tx_entry_vec *txep; -uint16_t n, nb_commit, tx_id; -uint64_t flags =CI_TX_DESC_CMD_DEFAULT ; -uint64_t rs =CI_TX_DESC_CMD_RS |CI_TX_DESC_CMD_DEFAULT ; - -/* cross rx_thresh boundary is not allowed */ - nb_pkts =RTE_MIN (nb_pkts, txq->tx_rs_thresh); - -if (txq->nb_tx_free < txq->tx_free_thresh) -ci_tx_free_bufs_vec (txq, ice_tx_desc_done, false); - - nb_commit = nb_pkts = (uint16_t )RTE_MIN (txq->nb_tx_free, nb_pkts); -if (unlikely (nb_pkts ==0 )) -return 0 ; - - tx_id = txq->tx_tail; - txdp = &txq->ci_tx_ring[tx_id]; - txep = &txq->sw_ring_vec[tx_id]; - - txq->nb_tx_free = (uint16_t )(txq->nb_tx_free - nb_pkts); - - n = (uint16_t )(txq->nb_tx_desc - tx_id); -if (nb_commit >= n) { -ci_tx_backlog_entry_vec (txep, tx_pkts, n); - -ci_vtx_avx2 (txdp, tx_pkts, n -1 , flags, offload, -CI_TAG_IN_DATA_DESC ,CI_TAG_IN_DATA_DESC ); - tx_pkts += (n -1 ); - txdp += (n -1 ); - -ci_vtx1 (txdp, *tx_pkts++, rs, offload,CI_TAG_IN_DATA_DESC ,CI_TAG_IN_DATA_DESC ); - - nb_commit = (uint16_t )(nb_commit - n); - - tx_id =0 ; - txq->tx_next_rs = (uint16_t )(txq->tx_rs_thresh -1 ); - -/* avoid reach the end of ring */ - txdp = &txq->ci_tx_ring[tx_id]; - txep = &txq->sw_ring_vec[tx_id]; - } - -ci_tx_backlog_entry_vec (txep, tx_pkts, nb_commit); - -ci_vtx_avx2 (txdp, tx_pkts, nb_commit, flags, offload, -CI_TAG_IN_DATA_DESC ,CI_TAG_IN_DATA_DESC ); - - tx_id = (uint16_t )(tx_id + nb_commit); -if (tx_id > txq->tx_next_rs) { - txq->ci_tx_ring[txq->tx_next_rs].cmd_type_offset_bsz |= -rte_cpu_to_le_64 (((uint64_t )CI_TX_DESC_CMD_RS ) <<CI_TXD_QW1_CMD_S ); - txq->tx_next_rs = - (uint16_t )(txq->tx_next_rs + txq->tx_rs_thresh); - } - - txq->tx_tail = tx_id; - -ICE_PCI_REG_WC_WRITE (txq->qtx_tail, txq->tx_tail); - -return nb_pkts; -} - static __rte_always_inline uint16_tice_xmit_pkts_vec_avx2_common (void *tx_queue,struct rte_mbuf **tx_pkts,uint16_t nb_pkts,bool offload)
@@ -853,8 +785,8 @@ ice_xmit_pkts_vec_avx2_common(void *tx_queue, struct rte_mbuf **tx_pkts,uint16_t ret, num; num = (uint16_t )RTE_MIN (nb_pkts, txq->tx_rs_thresh); - ret =ice_xmit_fixed_burst_vec_avx2 (tx_queue, &tx_pkts[nb_tx], - num, offload); + ret =ci_xmit_fixed_burst_vec_avx2 (tx_queue, &tx_pkts[nb_tx], num, + offload,CI_TAG_IN_DATA_DESC ,CI_TAG_IN_DATA_DESC ); nb_tx += ret; nb_pkts -= ret;if (ret < num)
drivers/net/intel/ice/ice_rxtx_vec_avx512.c +12 -90
122 lines not on this page · raw message
Signature · 1 line
2.53.0
PATCH v2 12/15 net/intel: add common vector Tx fns with context handling
In reply to Bruce RichardsonThe iavf vector driver has support for more Tx offloads because it has Tx functions which handle transmit context descriptors. Move those context-handling functions to the common folder for reuse by other drivers.
Signed-off-by Bruce Richardson <[email protected]>
Acked-by Ciara Loftus <[email protected]>
--- drivers/net/intel/common/tx_vec_x86.h | 136 ++++++++++++++++++ drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c | 78 +--------- drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c | 89 +----------- 3 files changed, 147 insertions(+), 156 deletions(-)
drivers/net/intel/common/tx_vec_x86.h +136 -0
@@ -408,6 +408,74 @@ ci_xmit_fixed_burst_vec_avx2(struct ci_tx_queue *txq, struct rte_mbuf **tx_pkts,return nb_pkts; } +static __rte_always_inline uint16_t +ci_xmit_fixed_burst_vec_ctx_avx2 (struct ci_tx_queue *txq,struct rte_mbuf **tx_pkts, +uint16_t nb_pkts,bool offload, +enum ci_l2tag_pos single_vlan_pos,enum ci_l2tag_pos qinq_outer_pos, +ci_tx_ctx_lldp_fn lldp_check) +{ +volatile struct ci_tx_desc *txdp; +struct ci_tx_entry_vec *txep; +uint16_t n, nb_commit, nb_mbuf, tx_id; +uint64_t flags =CI_TX_DESC_CMD_DEFAULT ; +uint64_t rs =CI_TX_DESC_CMD_RS | flags; + +if (txq->nb_tx_free < txq->tx_free_thresh) +ci_tx_free_bufs_vec (txq, ci_tx_desc_done_simple, true); + + nb_commit = (uint16_t )RTE_MIN (txq->nb_tx_free, nb_pkts <<1 ); + nb_commit &=0xFFFE ; +if (unlikely (nb_commit ==0 )) +return 0 ; + + nb_pkts = nb_commit >>1 ; + tx_id = txq->tx_tail; + txdp = &txq->ci_tx_ring[tx_id]; + txep = (void *)txq->sw_ring; + txep += (tx_id >>1 ); + + txq->nb_tx_free = (uint16_t )(txq->nb_tx_free - nb_commit); + n = (uint16_t )(txq->nb_tx_desc - tx_id); + +if (n !=0 && nb_commit >= n) { + nb_mbuf = n >>1 ; +ci_tx_backlog_entry_vec (txep, tx_pkts, nb_mbuf); + +ci_vtx_ctx_avx2 (txdp, tx_pkts, nb_mbuf -1 , flags, offload, + single_vlan_pos, qinq_outer_pos, lldp_check); + tx_pkts += (nb_mbuf -1 ); + txdp += (n -2 ); +ci_vtx1_ctx_avx2 (txdp, *tx_pkts++, rs, offload, + single_vlan_pos, qinq_outer_pos, lldp_check); + + nb_commit = (uint16_t )(nb_commit - n); + + txq->tx_next_rs = (uint16_t )(txq->tx_rs_thresh -1 ); + tx_id =0 ; +/* avoid reach the end of ring */ + txdp = txq->ci_tx_ring; + txep = (void *)txq->sw_ring; + } + + nb_mbuf = nb_commit >>1 ; +ci_tx_backlog_entry_vec (txep, tx_pkts, nb_mbuf); + +ci_vtx_ctx_avx2 (txdp, tx_pkts, nb_mbuf, flags, offload, + single_vlan_pos, qinq_outer_pos, lldp_check); + tx_id = (uint16_t )(tx_id + nb_commit); + +if (tx_id > txq->tx_next_rs) { + txq->ci_tx_ring[txq->tx_next_rs].cmd_type_offset_bsz |= +rte_cpu_to_le_64 (((uint64_t )CI_TX_DESC_CMD_RS ) <<CI_TXD_QW1_CMD_S ); + txq->tx_next_rs = (uint16_t )(txq->tx_next_rs + txq->tx_rs_thresh); + } + + txq->tx_tail = tx_id; + +ci_tx_qtx_tail_write (txq, tx_id); +return nb_pkts; +} +#endif /* __AVX2__ */ #ifdef __AVX512VL__
@@ -640,6 +708,74 @@ ci_xmit_fixed_burst_vec_avx512(struct ci_tx_queue *txq, struct rte_mbuf **tx_pktreturn nb_pkts; } +static __rte_always_inline uint16_t +ci_xmit_fixed_burst_vec_ctx_avx512 (struct ci_tx_queue *txq,struct rte_mbuf **tx_pkts, +uint16_t nb_pkts,bool offload, +enum ci_l2tag_pos single_vlan_pos,enum ci_l2tag_pos qinq_outer_pos, +ci_tx_ctx_lldp_fn lldp_check) +{ +volatile struct ci_tx_desc *txdp; +struct ci_tx_entry_vec *txep; +uint16_t n, nb_commit, nb_mbuf, tx_id; +uint64_t flags =CI_TX_DESC_CMD_DEFAULT ; +uint64_t rs =CI_TX_DESC_CMD_RS | flags; + +if (txq->nb_tx_free < txq->tx_free_thresh) +ci_tx_free_bufs_vec (txq, ci_tx_desc_done_simple, true); + + nb_commit = (uint16_t )RTE_MIN (txq->nb_tx_free, nb_pkts <<1 ); + nb_commit &=0xFFFE ; +if (unlikely (nb_commit ==0 )) +return 0 ; + + nb_pkts = nb_commit >>1 ; + tx_id = txq->tx_tail; + txdp = &txq->ci_tx_ring[tx_id]; + txep = (void *)txq->sw_ring; + txep += (tx_id >>1 ); + + txq->nb_tx_free = (uint16_t )(txq->nb_tx_free - nb_commit); + n = (uint16_t )(txq->nb_tx_desc - tx_id); + +if (n !=0 && nb_commit >= n) { + nb_mbuf = n >>1 ; +ci_tx_backlog_entry_vec (txep, tx_pkts, nb_mbuf); + +ci_vtx_ctx_avx512 (txdp, tx_pkts, nb_mbuf -1 , flags, offload, + single_vlan_pos, qinq_outer_pos, lldp_check); + tx_pkts += (nb_mbuf -1 ); + txdp += (n -2 ); +ci_vtx1_ctx_avx512 (txdp, *tx_pkts++, rs, offload, + single_vlan_pos, qinq_outer_pos, lldp_check); + + nb_commit = (uint16_t )(nb_commit - n); + + txq->tx_next_rs = (uint16_t )(txq->tx_rs_thresh -1 ); + tx_id =0 ; +/* avoid reach the end of ring */ + txdp = txq->ci_tx_ring; + txep = (void *)txq->sw_ring; + } + + nb_mbuf = nb_commit >>1 ; +ci_tx_backlog_entry_vec (txep, tx_pkts, nb_mbuf); + +ci_vtx_ctx_avx512 (txdp, tx_pkts, nb_mbuf, flags, offload, + single_vlan_pos, qinq_outer_pos, lldp_check); + tx_id = (uint16_t )(tx_id + nb_commit); + +if (tx_id > txq->tx_next_rs) { + txq->ci_tx_ring[txq->tx_next_rs].cmd_type_offset_bsz |= +rte_cpu_to_le_64 (((uint64_t )CI_TX_DESC_CMD_RS ) <<CI_TXD_QW1_CMD_S ); + txq->tx_next_rs = (uint16_t )(txq->tx_next_rs + txq->tx_rs_thresh); + } + + txq->tx_tail = tx_id; + +ci_tx_qtx_tail_write (txq, tx_id); +return nb_pkts; +} +#endif /* __AVX512VL__ */ #endif /* _COMMON_INTEL_TX_VEC_X86_H_ */
drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c +5 -73
97 lines not on this page · raw message
drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c +6 -83
113 lines not on this page · raw message
Signature · 1 line
2.53.0
PATCH v2 13/15 net/intel: improve Tx path selection logic
In reply to Bruce RichardsonRework the relevant priority of context descriptor vs SIMD width when selecting a Tx path so that presence of context descriptors is only considered when all else is equal.
Normally, path select takes place without having to worry about context descriptors, since offloads requiring context descriptors are specified using the offloads bitmap for each Tx path. However, in some cases, the need to use a context descriptor is not encodable via this offload path, e.g. VLAN offload for iavf can sometimes use the data descriptor and sometimes the context depending on what PF specifies at runtime. To account for those cases, we separate out the use of contexts from the path features, but instead note it as part of the path information, and add an additional parameter to the Tx path selection for drivers to mandate a context descriptor be present. Otherwise, ctx is largely ignored in path selection, except as a last-resort tie-breaker between two paths.
Signed-off-by Bruce Richardson <[email protected]>
Acked-by Ciara Loftus <[email protected]>
--- drivers/net/intel/common/tx.h | 41 +++++++++++++++++--------- drivers/net/intel/cpfl/cpfl_rxtx.c | 3 +- drivers/net/intel/i40e/i40e_rxtx.c | 2 +- drivers/net/intel/iavf/iavf_rxtx.c | 46 ++++++++++++++++-------------- drivers/net/intel/ice/ice_rxtx.c | 3 +- drivers/net/intel/idpf/idpf_rxtx.c | 3 +- 6 files changed, 58 insertions(+), 40 deletions(-)
drivers/net/intel/common/tx.h +27 -14
@@ -249,7 +249,6 @@ struct ci_tx_path_features {uint32_t tx_offloads;enum rte_vect_max_simd simd_width;bool simple_tx; -bool ctx_desc;bool disabled;bool single_queue; };
@@ -259,6 +258,7 @@ struct ci_tx_path_info {const char *info;struct ci_tx_path_features features;eth_tx_prep_t pkt_prep; +bool supports_ctx; };static __rte_always_inline void
@@ -411,6 +411,11 @@ ci_txq_release_all_mbufs(struct ci_tx_queue *txq, bool use_ctx)* Number of available paths in the infos array * @param default_path * Index of the default path to use if no suitable path is found +* @param force_ctx +* If true, only paths that support context descriptors may be selected. +* Use this for offloads that require a context descriptor but cannot be +* discovered purely from req_features->tx_offloads (e.g. a driver-specific +* devarg, or a runtime/hardware-negotiated tag placement). * * @return * The packet burst function index that best matches the requested features,
@@ -420,10 +425,12 @@ static inline int ci_tx_path_select(const struct ci_tx_path_features *req_features, const struct ci_tx_path_info *infos, size_t num_paths, - int default_path) + int default_path, + bool force_ctx) { int idx = default_path;const struct ci_tx_path_features *chosen_path_features =NULL ; +bool chosen_supports_ctx = false;for (unsigned int i =0 ; i < num_paths; i++) {const struct ci_tx_path_features *path_features = &infos[i].features;
@@ -440,8 +447,8 @@ ci_tx_path_select(const struct ci_tx_path_features *req_features,if (path_features->simple_tx && !req_features->simple_tx)continue ; -/* If a context descriptor is requested, ensure the path supports it. */ -if (!path_features->ctx_desc && req_features->ctx_desc) +/* If a context descriptor is required, ensure the path supports it. */ +if (!infos[i].supports_ctx && force_ctx)continue ;/* If requested, ensure the path supports single queue TX. */
@@ -462,22 +469,28 @@ ci_tx_path_select(const struct ci_tx_path_features *req_features,/* Do not select paths with lower SIMD width than the chosen path. */ if (path_features->simd_width < chosen_path_features->simd_width)continue ; -/* Do not select paths with more offloads enabled than the chosen path if -* the SIMD widths are the same. +/* The following tie-breaks only matter when SIMD widths are tied; +* a strictly wider path is always preferred regardless of offload +* count or ctx-descriptor use. */ -if (path_features->simd_width == chosen_path_features->simd_width && -rte_popcount32 (path_features->tx_offloads) > -rte_popcount32 (chosen_path_features->tx_offloads)) -continue ; - -/* Don't use a context descriptor unless necessary */ -if (path_features->ctx_desc && !chosen_path_features->ctx_desc) -continue ; +if (path_features->simd_width == chosen_path_features->simd_width) { +/* Do not select paths with more offloads enabled than the +* chosen path. +*/ +if (rte_popcount32 (path_features->tx_offloads) > +rte_popcount32 (chosen_path_features->tx_offloads)) +continue ; + +/* Don't use a context descriptor unless necessary */ +if (infos[i].supports_ctx && !chosen_supports_ctx) +continue ; + } }/* Finally, select the path since it has met all the requirements. */ idx = i; chosen_path_features = &infos[idx].features; + chosen_supports_ctx = infos[idx].supports_ctx; }return idx;
drivers/net/intel/cpfl/cpfl_rxtx.c +2 -1
@@ -1542,7 +1542,8 @@ cpfl_set_tx_function(struct rte_eth_dev *dev) ad->tx_func_type =ci_tx_path_select (&req_features, &idpf_tx_path_infos[0 ],IDPF_TX_MAX , -IDPF_TX_DEFAULT ); +IDPF_TX_DEFAULT , + false);/* Set use_vec_entry for single queue mode - only IDPF_TX_SINGLEQ uses regular entries */ if (vport->txq_model ==VIRTCHNL2_QUEUE_MODEL_SINGLE ) {
drivers/net/intel/i40e/i40e_rxtx.c +1 -1
@@ -3120,7 +3120,7 @@ i40e_set_tx_function(struct rte_eth_dev *dev) } ad->tx_func_type =ci_tx_path_select (&req_features, &i40e_tx_path_infos[0 ], -RTE_DIM (i40e_tx_path_infos),I40E_TX_DEFAULT ); +RTE_DIM (i40e_tx_path_infos),I40E_TX_DEFAULT , false); out: dev->tx_pkt_burst = mbuf_check ? i40e_xmit_pkts_check :
drivers/net/intel/iavf/iavf_rxtx.c +24 -22
112 lines not on this page · raw message
drivers/net/intel/ice/ice_rxtx.c +2 -1
@@ -3784,7 +3784,8 @@ ice_set_tx_function(struct rte_eth_dev *dev) ad->tx_func_type =ci_tx_path_select (&req_features, &ice_tx_path_infos[0 ],RTE_DIM (ice_tx_path_infos), -ICE_TX_DEFAULT ); +ICE_TX_DEFAULT , + false); out:#if defined(RTE_ARCH_X86 )
drivers/net/intel/idpf/idpf_rxtx.c +2 -1
@@ -887,7 +887,8 @@ idpf_set_tx_function(struct rte_eth_dev *dev) ad->tx_func_type =ci_tx_path_select (&req_features, &idpf_tx_path_infos[0 ],IDPF_TX_MAX , -IDPF_TX_DEFAULT ); +IDPF_TX_DEFAULT , + false);/* Set use_vec_entry for single queue mode - only IDPF_TX_SINGLEQ uses regular entries */ if (vport->txq_model ==VIRTCHNL2_QUEUE_MODEL_SINGLE ) {
Signature · 1 line
2.53.0
PATCH v2 14/15 net/ice: enable context desc offloads for vector Tx
In reply to Bruce RichardsonUse the context descriptor Tx vector paths from intel/common in ice driver to enable more offloads on the vector path for ice driver.
Signed-off-by Bruce Richardson <[email protected]>
--- doc/guides/rel_notes/release_26_11.rst | 6 ++++ drivers/net/intel/ice/ice_ethdev.h | 3 ++ drivers/net/intel/ice/ice_rxtx.c | 31 ++++++++++++++++++--- drivers/net/intel/ice/ice_rxtx.h | 11 ++++++++ drivers/net/intel/ice/ice_rxtx_vec_avx2.c | 24 ++++++++++++++++ drivers/net/intel/ice/ice_rxtx_vec_avx512.c | 24 ++++++++++++++++ 6 files changed, 95 insertions(+), 4 deletions(-)
doc/guides/rel_notes/release_26_11.rst +6 -0
@@ -69,6 +69,12 @@ New Features Added ``fdir_buffer_size`` devarg to select the Flow Director table size (``64k``, ``128k`` or ``256k``) at probe time. The default remains ``64k``. +* **Updated Intel ice driver.** + + * Added Tx context descriptor support to the AVX2 and AVX512 vector Tx + paths, enabling QinQ tag insertion and outer IPv4/UDP checksum + offloads on those paths. + Removed Items -------------
drivers/net/intel/ice/ice_ethdev.h +3 -0
@@ -213,8 +213,10 @@ enum ice_tx_func_type {ICE_TX_SIMPLE ,ICE_TX_AVX2 ,ICE_TX_AVX2_OFFLOAD , +ICE_TX_AVX2_CTX_OFFLOAD ,ICE_TX_AVX512 ,ICE_TX_AVX512_OFFLOAD , +ICE_TX_AVX512_CTX_OFFLOAD , ICE_TX_NEON, };
@@ -676,6 +678,7 @@ struct ice_adapter {bool rx_bulk_alloc_allowed;bool tx_vec_allowed;bool tx_simple_allowed; +bool use_ctx;enum ice_rx_func_type rx_func_type;enum ice_tx_func_type tx_func_type;/* ptype mapping table */
drivers/net/intel/ice/ice_rxtx.c +27 -4
@@ -890,6 +890,7 @@ ice_tx_queue_start(struct rte_eth_dev *dev, uint16_t tx_queue_id)/* record what kind of descriptor cleanup we need on teardown */ txq->use_vec_entry = ad->tx_vec_allowed || ad->tx_simple_allowed; + txq->use_ctx = ad->use_ctx;if (txq->tsq !=NULL && txq->tsq->ts_flag >0 ) {struct ice_aqc_set_txtime_qgrp *ts_elem;
@@ -1200,7 +1201,7 @@ ice_tx_queue_stop(struct rte_eth_dev *dev, uint16_t tx_queue_id)return -EINVAL ; } -ci_txq_release_all_mbufs (txq, false); +ci_txq_release_all_mbufs (txq, txq->use_ctx);ice_reset_tx_queue (txq); dev->data->tx_queue_state[tx_queue_id] =RTE_ETH_QUEUE_STATE_STOPPED ;
@@ -1263,7 +1264,7 @@ ice_fdir_tx_queue_stop(struct rte_eth_dev *dev, uint16_t tx_queue_id)return -EINVAL ; } -ci_txq_release_all_mbufs (txq, false); +ci_txq_release_all_mbufs (txq, txq->use_ctx); txq->qtx_tail =NULL ;return 0 ;
@@ -1751,7 +1752,7 @@ ice_tx_queue_release(void *txq)return ; } -ci_txq_release_all_mbufs (q, false); +ci_txq_release_all_mbufs (q, q->use_ctx);rte_free (q->sw_ring);rte_free (q->rs_last_id);if (q->tsq) {
@@ -3533,7 +3534,8 @@ static const struct ci_tx_path_info ice_tx_path_infos[] = { .features = { .tx_offloads =ICE_TX_SCALAR_OFFLOADS }, - .pkt_prep = ice_prep_pkts + .pkt_prep = ice_prep_pkts, + .supports_ctx = true }, [ICE_TX_SIMPLE] = { .pkt_burst = ice_xmit_pkts_simple,
@@ -3563,6 +3565,16 @@ static const struct ci_tx_path_info ice_tx_path_infos[] = { }, .pkt_prep = ice_prep_pkts }, + [ICE_TX_AVX2_CTX_OFFLOAD] = { + .pkt_burst = ice_xmit_pkts_vec_avx2_ctx_offload, + .info = "Offload Vector AVX2 Ctx", + .features = { + .tx_offloads = ICE_TX_VECTOR_CTX_OFFLOAD_OFFLOADS, + .simd_width =RTE_VECT_SIMD_256 + }, + .pkt_prep = ice_prep_pkts, + .supports_ctx = true + }, #ifdef CC_AVX512_SUPPORT [ICE_TX_AVX512] = { .pkt_burst = ice_xmit_pkts_vec_avx512,
@@ -3582,6 +3594,16 @@ static const struct ci_tx_path_info ice_tx_path_infos[] = { }, .pkt_prep = ice_prep_pkts }, + [ICE_TX_AVX512_CTX_OFFLOAD] = { + .pkt_burst = ice_xmit_pkts_vec_avx512_ctx_offload, + .info = "Offload Vector AVX512 Ctx", + .features = { + .tx_offloads = ICE_TX_VECTOR_CTX_OFFLOAD_OFFLOADS, + .simd_width = RTE_VECT_SIMD_512 + }, + .pkt_prep = ice_prep_pkts, + .supports_ctx = true + },#endif #elif defined(RTE_ARCH_ARM64) [ICE_TX_NEON] = {
@@ -3795,6 +3817,7 @@ ice_set_tx_function(struct rte_eth_dev *dev) ad->tx_vec_allowed = (ice_tx_path_infos[ad->tx_func_type].features.simd_width >=RTE_VECT_SIMD_128 );#endif + ad->use_ctx = ice_tx_path_infos[ad->tx_func_type].supports_ctx; dev->tx_pkt_burst = mbuf_check ? ice_xmit_pkts_check : ice_tx_path_infos[ad->tx_func_type].pkt_burst;
drivers/net/intel/ice/ice_rxtx.h +11 -0
@@ -135,6 +135,12 @@ RTE_ETH_TX_OFFLOAD_UDP_CKSUM | \ RTE_ETH_TX_OFFLOAD_TCP_CKSUM | \ RTE_ETH_TX_OFFLOAD_SCTP_CKSUM) +/* vector ctx offload path: QinQ outer tag + outer/tunnel checksum via ctx desc */ +#define ICE_TX_VECTOR_CTX_OFFLOAD_OFFLOADS ( \ + ICE_TX_VECTOR_OFFLOAD_OFFLOADS | \ + RTE_ETH_TX_OFFLOAD_OUTER_IPV4_CKSUM | \ + RTE_ETH_TX_OFFLOAD_OUTER_UDP_CKSUM | \ + RTE_ETH_TX_OFFLOAD_QINQ_INSERT)/* Max header size can be 2K - 64 bytes */ #define ICE_RX_HDR_BUF_SIZE (2048 - 64)
@@ -284,6 +290,8 @@ uint16_t ice_xmit_pkts_vec_avx2(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts);uint16_t ice_xmit_pkts_vec_avx2_offload (void *tx_queue,struct rte_mbuf **tx_pkts,uint16_t nb_pkts); +uint16_t ice_xmit_pkts_vec_avx2_ctx_offload (void *tx_queue,struct rte_mbuf **tx_pkts, +uint16_t nb_pkts);uint16_t ice_recv_pkts_vec_avx512 (void *rx_queue,struct rte_mbuf **rx_pkts,uint16_t nb_pkts); uint16_t ice_recv_pkts_vec_avx512_offload(void *rx_queue,
@@ -300,6 +308,9 @@ uint16_t ice_xmit_pkts_vec_avx512(void *tx_queue, struct rte_mbuf **tx_pkts,uint16_t ice_xmit_pkts_vec_avx512_offload (void *tx_queue,struct rte_mbuf **tx_pkts,uint16_t nb_pkts); +uint16_t ice_xmit_pkts_vec_avx512_ctx_offload (void *tx_queue, +struct rte_mbuf **tx_pkts, +uint16_t nb_pkts);int ice_fdir_programming (struct ice_pf *pf,struct ice_fltr_desc *fdir_desc);int ice_tx_done_cleanup (void *txq,uint32_t free_cnt);int ice_get_monitor_addr (void *rx_queue,struct rte_power_monitor_cond *pmc);
drivers/net/intel/ice/ice_rxtx_vec_avx2.c +24 -0
@@ -810,6 +810,30 @@ ice_xmit_pkts_vec_avx2_offload(void *tx_queue, struct rte_mbuf **tx_pkts,return ice_xmit_pkts_vec_avx2_common (tx_queue, tx_pkts, nb_pkts, true); } +uint16_t +ice_xmit_pkts_vec_avx2_ctx_offload (void *tx_queue,struct rte_mbuf **tx_pkts, +uint16_t nb_pkts) +{ +uint16_t nb_tx =0 ; +struct ci_tx_queue *txq = (struct ci_tx_queue *)tx_queue; + +while (nb_pkts) { +uint16_t ret, num; + +/* cross rs_thresh boundary is not allowed */ + num = (uint16_t )RTE_MIN (nb_pkts <<1 , txq->tx_rs_thresh); + num = num >>1 ; + ret =ci_xmit_fixed_burst_vec_ctx_avx2 (tx_queue, &tx_pkts[nb_tx], num, + true,CI_TAG_IN_DATA_DESC ,CI_TAG_IN_CTX_DESC ,NULL ); + nb_tx += ret; + nb_pkts -= ret; +if (ret < num) +break ; + } + +return nb_tx; +} + int __rte_coldice_rxq_vec_setup (struct ci_rx_queue *rxq) {
drivers/net/intel/ice/ice_rxtx_vec_avx512.c +24 -0
@@ -882,3 +882,27 @@ ice_xmit_pkts_vec_avx512_offload(void *tx_queue, struct rte_mbuf **tx_pkts, {return ice_xmit_pkts_vec_avx512_common (tx_queue, tx_pkts, nb_pkts, true); } + +uint16_t +ice_xmit_pkts_vec_avx512_ctx_offload (void *tx_queue,struct rte_mbuf **tx_pkts, +uint16_t nb_pkts) +{ +uint16_t nb_tx =0 ; +struct ci_tx_queue *txq = (struct ci_tx_queue *)tx_queue; + +while (nb_pkts) { +uint16_t ret, num; + +/* cross rs_thresh boundary is not allowed */ + num = (uint16_t )RTE_MIN (nb_pkts <<1 , txq->tx_rs_thresh); + num = num >>1 ; + ret =ci_xmit_fixed_burst_vec_ctx_avx512 (tx_queue, &tx_pkts[nb_tx], num, + true,CI_TAG_IN_DATA_DESC ,CI_TAG_IN_CTX_DESC ,NULL ); + nb_tx += ret; + nb_pkts -= ret; +if (ret < num) +break ; + } + +return nb_tx; +}
Signature · 1 line
2.53.0
RE: [PATCH v2 14/15] net/ice: enable context desc offloads for vector Tx
Quoted 21 lines
Subject: [PATCH v2 14/15] net/ice: enable context desc offloads for vector Tx
Use the context descriptor Tx vector paths from intel/common in ice driver to enable more offloads on the vector path for ice driver.
Signed-off-by Bruce Richardson <[email protected]>--- doc/guides/rel_notes/release_26_11.rst | 6 ++++ drivers/net/intel/ice/ice_ethdev.h | 3 ++ drivers/net/intel/ice/ice_rxtx.c | 31 ++++++++++++++++++--- drivers/net/intel/ice/ice_rxtx.h | 11 ++++++++ drivers/net/intel/ice/ice_rxtx_vec_avx2.c | 24 ++++++++++++++++ drivers/net/intel/ice/ice_rxtx_vec_avx512.c | 24 ++++++++++++++++ 6 files changed, 95 insertions(+), 4 deletions(-)doc/guides/rel_notes/release_26_11.rst +0
-0
doc/guides/rel_notes/release_26_11.rst New Features · line 76
Quoted 8 lines
doc/guides/rel_notes/release_26_11.rst +5
-0@@ -69,6 +69,12 @@ New Features Added ``fdir_buffer_size`` devarg to select the Flow Director table size (``64k``, ``128k`` or ``256k``) at probe time. The default remains ``64k``. +* **Updated Intel ice driver.** + + * Added Tx context descriptor support to the AVX2 and AVX512 vector Tx + paths, enabling QinQ tag insertion and outer IPv4/UDP checksum + offloads on those paths.
Check if any updates are needed to the ice.ini features file, now that these new offloads are supported.
Acked-by Ciara Loftus <[email protected]>
Quoted 2 lines
doc/guides/rel_notes/release_26_11.rst +1
-0+
PATCH v2 15/15 net/intel: consolidate Tx vector offload path checks
In reply to Bruce RichardsonSince iavf and ice drivers now have common Tx vector paths, we can consolidate the path decisions a little, including replacing the per-driver offload flags with common ones.
Signed-off-by Bruce Richardson <[email protected]>
--- drivers/net/intel/common/tx.h | 30 +++++++++++++++++++ drivers/net/intel/iavf/iavf_rxtx.c | 18 +++++------ drivers/net/intel/iavf/iavf_rxtx.h | 19 ------------ drivers/net/intel/iavf/iavf_rxtx_vec_common.h | 3 +- drivers/net/intel/ice/ice_rxtx.c | 14 ++++----- drivers/net/intel/ice/ice_rxtx.h | 18 ----------- drivers/net/intel/ice/ice_rxtx_vec_common.h | 3 +- 7 files changed, 48 insertions(+), 57 deletions(-)
drivers/net/intel/common/tx.h +30 -0
@@ -12,6 +12,27 @@#include <rte_io.h> #include <rte_byteorder.h> +#define CI_VPMD_TX_BURST 32 +#define CI_VPMD_TX_MAX_FREE_BUF 64 + +/* basic vector path */ +#define CI_TX_VECTOR_OFFLOADS RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE +/* offload vector path */ +#define CI_TX_VEC_OFFLOAD_PATH_OFFLOADS ( \ + CI_TX_VECTOR_OFFLOADS | \ + RTE_ETH_TX_OFFLOAD_VLAN_INSERT | \ + RTE_ETH_TX_OFFLOAD_IPV4_CKSUM | \ + RTE_ETH_TX_OFFLOAD_SCTP_CKSUM | \ + RTE_ETH_TX_OFFLOAD_UDP_CKSUM | \ + RTE_ETH_TX_OFFLOAD_TCP_CKSUM) +/* offload vector path with context descriptor */ +#define CI_TX_VEC_CTX_OFFLOAD_PATH_OFFLOADS ( \ + CI_TX_VECTOR_OFFLOADS | \ + CI_TX_VEC_OFFLOAD_PATH_OFFLOADS | \ + RTE_ETH_TX_OFFLOAD_OUTER_IPV4_CKSUM | \ + RTE_ETH_TX_OFFLOAD_OUTER_UDP_CKSUM | \ + RTE_ETH_TX_OFFLOAD_QINQ_INSERT) +/* Common TX Descriptor QW1 Field Definitions */ #define CI_TXD_QW1_DTYPE_S 0#define CI_TXD_QW1_DTYPE_M (0xFUL << CI_TXD_QW1_DTYPE_S)
@@ -261,6 +282,15 @@ struct ci_tx_path_info {bool supports_ctx; }; +/* basic check for a vector-driver capable Tx queue. +* Individual drivers may have other further tests beyond this. +*/ +static inline bool +ci_txq_vec_capable (uint16_t tx_rs_thresh) +{ +return tx_rs_thresh >=CI_VPMD_TX_BURST && tx_rs_thresh <=CI_VPMD_TX_MAX_FREE_BUF ; +} + static __rte_always_inline voidci_tx_backlog_entry (struct ci_tx_entry *txep,struct rte_mbuf **tx_pkts,uint16_t nb_pkts) {
drivers/net/intel/iavf/iavf_rxtx.c +9 -9
@@ -3620,7 +3620,7 @@ static const struct ci_tx_path_info iavf_tx_path_infos[] = { .pkt_burst = iavf_xmit_pkts_vec_avx2, .info = "Vector AVX2", .features = { - .tx_offloads = IAVF_TX_VECTOR_OFFLOADS, + .tx_offloads = CI_TX_VECTOR_OFFLOADS, .simd_width = RTE_VECT_SIMD_256 } },
@@ -3628,7 +3628,7 @@ static const struct ci_tx_path_info iavf_tx_path_infos[] = { .pkt_burst = iavf_xmit_pkts_vec_avx2_offload, .info = "Vector AVX2 Offload", .features = { - .tx_offloads = IAVF_TX_VECTOR_OFFLOAD_OFFLOADS, + .tx_offloads = CI_TX_VEC_OFFLOAD_PATH_OFFLOADS, .simd_width = RTE_VECT_SIMD_256 } },
@@ -3636,7 +3636,7 @@ static const struct ci_tx_path_info iavf_tx_path_infos[] = { .pkt_burst = iavf_xmit_pkts_vec_avx2_ctx, .info = "Vector AVX2 Ctx", .features = { - .tx_offloads = IAVF_TX_VECTOR_OFFLOADS, + .tx_offloads = CI_TX_VECTOR_OFFLOADS, .simd_width =RTE_VECT_SIMD_256 }, .supports_ctx = true
@@ -3645,7 +3645,7 @@ static const struct ci_tx_path_info iavf_tx_path_infos[] = { .pkt_burst = iavf_xmit_pkts_vec_avx2_ctx_offload, .info = "Vector AVX2 Ctx Offload", .features = { - .tx_offloads = IAVF_TX_VECTOR_CTX_OFFLOAD_OFFLOADS, + .tx_offloads = CI_TX_VEC_CTX_OFFLOAD_PATH_OFFLOADS, .simd_width =RTE_VECT_SIMD_256 }, .supports_ctx = true
@@ -3655,7 +3655,7 @@ static const struct ci_tx_path_info iavf_tx_path_infos[] = { .pkt_burst = iavf_xmit_pkts_vec_avx512, .info = "Vector AVX512", .features = { - .tx_offloads = IAVF_TX_VECTOR_OFFLOADS, + .tx_offloads = CI_TX_VECTOR_OFFLOADS, .simd_width = RTE_VECT_SIMD_512 } },
@@ -3663,7 +3663,7 @@ static const struct ci_tx_path_info iavf_tx_path_infos[] = { .pkt_burst = iavf_xmit_pkts_vec_avx512_offload, .info = "Vector AVX512 Offload", .features = { - .tx_offloads = IAVF_TX_VECTOR_OFFLOAD_OFFLOADS, + .tx_offloads = CI_TX_VEC_OFFLOAD_PATH_OFFLOADS, .simd_width = RTE_VECT_SIMD_512 } },
@@ -3671,7 +3671,7 @@ static const struct ci_tx_path_info iavf_tx_path_infos[] = { .pkt_burst = iavf_xmit_pkts_vec_avx512_ctx, .info = "Vector AVX512 Ctx", .features = { - .tx_offloads = IAVF_TX_VECTOR_OFFLOADS, + .tx_offloads = CI_TX_VECTOR_OFFLOADS, .simd_width =RTE_VECT_SIMD_512 }, .supports_ctx = true
@@ -3680,7 +3680,7 @@ static const struct ci_tx_path_info iavf_tx_path_infos[] = { .pkt_burst = iavf_xmit_pkts_vec_avx512_ctx_offload, .info = "Vector AVX512 Ctx Offload", .features = { - .tx_offloads = IAVF_TX_VECTOR_CTX_OFFLOAD_OFFLOADS, + .tx_offloads = CI_TX_VEC_CTX_OFFLOAD_PATH_OFFLOADS, .simd_width =RTE_VECT_SIMD_512 }, .supports_ctx = true
@@ -3691,7 +3691,7 @@ static const struct ci_tx_path_info iavf_tx_path_infos[] = { .pkt_burst = iavf_xmit_pkts_vec, .info = "Vector Neon", .features = { - .tx_offloads = IAVF_TX_VECTOR_OFFLOADS, + .tx_offloads = CI_TX_VECTOR_OFFLOADS, .simd_width = RTE_VECT_SIMD_128 } },
drivers/net/intel/iavf/iavf_rxtx.h +0 -19
@@ -29,11 +29,9 @@/* used for Vector PMD */ #define IAVF_VPMD_RX_BURST CI_VPMD_RX_BURST -#define IAVF_VPMD_TX_BURST 32#define IAVF_VPMD_RXQ_REARM_THRESH CI_VPMD_RX_REARM_THRESH#define IAVF_VPMD_DESCS_PER_LOOP CI_VPMD_DESCS_PER_LOOP#define IAVF_VPMD_DESCS_PER_LOOP_WIDE CI_VPMD_DESCS_PER_LOOP_WIDE -#define IAVF_VPMD_TX_MAX_FREE_BUF 64/* basic scalar path */ #define IAVF_TX_SCALAR_OFFLOADS ( \
@@ -54,23 +52,6 @@ RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE | \ RTE_ETH_TX_OFFLOAD_SECURITY | \ RTE_ETH_TX_OFFLOAD_OUTER_UDP_CKSUM) -/* basic vector path */ -#define IAVF_TX_VECTOR_OFFLOADS RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE -/* offload vector path */ -#define IAVF_TX_VECTOR_OFFLOAD_OFFLOADS ( \ - IAVF_TX_VECTOR_OFFLOADS | \ - RTE_ETH_TX_OFFLOAD_VLAN_INSERT | \ - RTE_ETH_TX_OFFLOAD_IPV4_CKSUM | \ - RTE_ETH_TX_OFFLOAD_SCTP_CKSUM | \ - RTE_ETH_TX_OFFLOAD_UDP_CKSUM | \ - RTE_ETH_TX_OFFLOAD_TCP_CKSUM) -/* offload vector path with context descriptor */ -#define IAVF_TX_VECTOR_CTX_OFFLOAD_OFFLOADS ( \ - IAVF_TX_VECTOR_OFFLOADS | \ - IAVF_TX_VECTOR_OFFLOAD_OFFLOADS | \ - RTE_ETH_TX_OFFLOAD_OUTER_IPV4_CKSUM | \ - RTE_ETH_TX_OFFLOAD_OUTER_UDP_CKSUM | \ - RTE_ETH_TX_OFFLOAD_QINQ_INSERT)/* basic scalar path */ #define IAVF_RX_SCALAR_OFFLOADS ( \
drivers/net/intel/iavf/iavf_rxtx_vec_common.h +1 -2
@@ -80,8 +80,7 @@ iavf_tx_vec_queue_default(struct ci_tx_queue *txq)if (!txq)return -1 ; -if (txq->tx_rs_thresh <IAVF_VPMD_TX_BURST || - txq->tx_rs_thresh >IAVF_VPMD_TX_MAX_FREE_BUF ) +if (!ci_txq_vec_capable (txq->tx_rs_thresh))return -1 ;return 0 ;
drivers/net/intel/ice/ice_rxtx.c +7 -7
@@ -3551,7 +3551,7 @@ static const struct ci_tx_path_info ice_tx_path_infos[] = { .pkt_burst = ice_xmit_pkts_vec_avx2, .info = "Vector AVX2", .features = { - .tx_offloads = ICE_TX_VECTOR_OFFLOADS, + .tx_offloads = CI_TX_VECTOR_OFFLOADS, .simd_width =RTE_VECT_SIMD_256 }, .pkt_prep = rte_eth_tx_pkt_prepare_dummy
@@ -3560,7 +3560,7 @@ static const struct ci_tx_path_info ice_tx_path_infos[] = { .pkt_burst = ice_xmit_pkts_vec_avx2_offload, .info = "Offload Vector AVX2", .features = { - .tx_offloads = ICE_TX_VECTOR_OFFLOAD_OFFLOADS, + .tx_offloads = CI_TX_VEC_OFFLOAD_PATH_OFFLOADS, .simd_width =RTE_VECT_SIMD_256 }, .pkt_prep = ice_prep_pkts
@@ -3569,7 +3569,7 @@ static const struct ci_tx_path_info ice_tx_path_infos[] = { .pkt_burst = ice_xmit_pkts_vec_avx2_ctx_offload, .info = "Offload Vector AVX2 Ctx", .features = { - .tx_offloads = ICE_TX_VECTOR_CTX_OFFLOAD_OFFLOADS, + .tx_offloads = CI_TX_VEC_CTX_OFFLOAD_PATH_OFFLOADS, .simd_width =RTE_VECT_SIMD_256 }, .pkt_prep = ice_prep_pkts,
@@ -3580,7 +3580,7 @@ static const struct ci_tx_path_info ice_tx_path_infos[] = { .pkt_burst = ice_xmit_pkts_vec_avx512, .info = "Vector AVX512", .features = { - .tx_offloads = ICE_TX_VECTOR_OFFLOADS, + .tx_offloads = CI_TX_VECTOR_OFFLOADS, .simd_width =RTE_VECT_SIMD_512 }, .pkt_prep = rte_eth_tx_pkt_prepare_dummy
@@ -3589,7 +3589,7 @@ static const struct ci_tx_path_info ice_tx_path_infos[] = { .pkt_burst = ice_xmit_pkts_vec_avx512_offload, .info = "Offload Vector AVX512", .features = { - .tx_offloads = ICE_TX_VECTOR_OFFLOAD_OFFLOADS, + .tx_offloads = CI_TX_VEC_OFFLOAD_PATH_OFFLOADS, .simd_width =RTE_VECT_SIMD_512 }, .pkt_prep = ice_prep_pkts
@@ -3598,7 +3598,7 @@ static const struct ci_tx_path_info ice_tx_path_infos[] = { .pkt_burst = ice_xmit_pkts_vec_avx512_ctx_offload, .info = "Offload Vector AVX512 Ctx", .features = { - .tx_offloads = ICE_TX_VECTOR_CTX_OFFLOAD_OFFLOADS, + .tx_offloads = CI_TX_VEC_CTX_OFFLOAD_PATH_OFFLOADS, .simd_width =RTE_VECT_SIMD_512 }, .pkt_prep = ice_prep_pkts,
@@ -3610,7 +3610,7 @@ static const struct ci_tx_path_info ice_tx_path_infos[] = { .pkt_burst = ice_xmit_pkts_vec, .info = "Vector Neon", .features = { - .tx_offloads = ICE_TX_VECTOR_OFFLOADS, + .tx_offloads = CI_TX_VECTOR_OFFLOADS, .simd_width =RTE_VECT_SIMD_128 }, .pkt_prep = rte_eth_tx_pkt_prepare_dummy
drivers/net/intel/ice/ice_rxtx.h +0 -18
@@ -47,9 +47,7 @@#define ICE_SUPPORT_CHAIN_NUM 5#define ICE_VPMD_RX_BURST CI_VPMD_RX_BURST -#define ICE_VPMD_TX_BURST 32#define ICE_VPMD_RXQ_REARM_THRESH CI_VPMD_RX_REARM_THRESH -#define ICE_TX_MAX_FREE_BUF_SZ 64#define ICE_VPMD_DESCS_PER_LOOP CI_VPMD_DESCS_PER_LOOP#define ICE_VPMD_DESCS_PER_LOOP_WIDE CI_VPMD_DESCS_PER_LOOP_WIDE
@@ -125,22 +123,6 @@ RTE_ETH_TX_OFFLOAD_IPIP_TNL_TSO | \ RTE_ETH_TX_OFFLOAD_GENEVE_TNL_TSO | \ RTE_ETH_TX_OFFLOAD_SEND_ON_TIMESTAMP) -/* basic vector path */ -#define ICE_TX_VECTOR_OFFLOADS RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE -/* vector offload paths */ -#define ICE_TX_VECTOR_OFFLOAD_OFFLOADS ( \ - ICE_TX_VECTOR_OFFLOADS | \ - RTE_ETH_TX_OFFLOAD_VLAN_INSERT | \ - RTE_ETH_TX_OFFLOAD_IPV4_CKSUM | \ - RTE_ETH_TX_OFFLOAD_UDP_CKSUM | \ - RTE_ETH_TX_OFFLOAD_TCP_CKSUM | \ - RTE_ETH_TX_OFFLOAD_SCTP_CKSUM) -/* vector ctx offload path: QinQ outer tag + outer/tunnel checksum via ctx desc */ -#define ICE_TX_VECTOR_CTX_OFFLOAD_OFFLOADS ( \ - ICE_TX_VECTOR_OFFLOAD_OFFLOADS | \ - RTE_ETH_TX_OFFLOAD_OUTER_IPV4_CKSUM | \ - RTE_ETH_TX_OFFLOAD_OUTER_UDP_CKSUM | \ - RTE_ETH_TX_OFFLOAD_QINQ_INSERT)/* Max header size can be 2K - 64 bytes */ #define ICE_RX_HDR_BUF_SIZE (2048 - 64)
drivers/net/intel/ice/ice_rxtx_vec_common.h +1 -2
@@ -72,8 +72,7 @@ ice_tx_vec_queue_default(struct ci_tx_queue *txq)if (!txq)return -1 ; -if (txq->tx_rs_thresh <ICE_VPMD_TX_BURST || - txq->tx_rs_thresh >ICE_TX_MAX_FREE_BUF_SZ ) +if (!ci_txq_vec_capable (txq->tx_rs_thresh))return -1 ;return 0 ;
Signature · 1 line
2.53.0
RE: [PATCH v2 15/15] net/intel: consolidate Tx vector offload path checks
Quoted 7 lines
Subject: [PATCH v2 15/15] net/intel: consolidate Tx vector offload path checks
Since iavf and ice drivers now have common Tx vector paths, we can consolidate the path decisions a little, including replacing the per-driver offload flags with common ones.
Signed-off-by Bruce Richardson <[email protected]>
Acked-by Ciara Loftus <[email protected]>
Quoted 10 lines
--- drivers/net/intel/common/tx.h | 30 +++++++++++++++++++ drivers/net/intel/iavf/iavf_rxtx.c | 18 +++++------ drivers/net/intel/iavf/iavf_rxtx.h | 19 ------------ drivers/net/intel/iavf/iavf_rxtx_vec_common.h | 3 +- drivers/net/intel/ice/ice_rxtx.c | 14 ++++----- drivers/net/intel/ice/ice_rxtx.h | 18 ----------- drivers/net/intel/ice/ice_rxtx_vec_common.h | 3 +- 7 files changed, 48 insertions(+), 57 deletions(-)
v3 cover consolidate ice and iavf Tx vector paths
In reply to Bruce RichardsonThis patchset consolidates the ice and iavf vector Tx paths, moving the code to net/intel/common. The iavf code is used as the basis for the move, because it has a superset of the features. When the common code is adopted by the ice driver, it then gains features such as QinQ support and tunneling support.
V3:
* fix variable shadowing issue in patch 15 (ci_txq_vec_capable param)
* update ice.ini with new features
V2:
* fix build when iova not in mbuf
* add extra patch to consolidate the offload flags
Bruce Richardson (15):
net/iavf: remove unnecessary alignment calls net/intel: make Tx context flag common
net/iavf: use separate params for VLAN and QinQ position net/iavf: deduplicate tunnel field fill functions net/intel: define common macros for tunneling bit-shifts net/intel: move iavf descriptor writing functions to common net/intel: use function callback for lldp net/intel: allow building without IOVA in mbuf net/ice: use common descriptor creation functions net/intel: move vector Tx paths to common net/ice: use common AVX Tx functions
net/intel: add common vector Tx fns with context handling net/intel: improve Tx path selection logic net/ice: enable context desc offloads for vector Tx net/intel: consolidate Tx vector offload path checks
doc/guides/nics/features/ice.ini | 8 +- doc/guides/rel_notes/release_26_11.rst | 6 + drivers/net/intel/common/tx.h | 107 ++- drivers/net/intel/common/tx_vec_x86.h | 781 ++++++++++++++++++ drivers/net/intel/cpfl/cpfl_rxtx.c | 3 +- drivers/net/intel/i40e/i40e_rxtx.c | 2 +- drivers/net/intel/iavf/iavf_rxtx.c | 94 +-- drivers/net/intel/iavf/iavf_rxtx.h | 53 -- drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c | 496 +---------- drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c | 495 +---------- drivers/net/intel/iavf/iavf_rxtx_vec_common.h | 79 +- drivers/net/intel/iavf/meson.build | 4 +- drivers/net/intel/ice/ice_ethdev.h | 3 + drivers/net/intel/ice/ice_rxtx.c | 44 +- drivers/net/intel/ice/ice_rxtx.h | 17 +- drivers/net/intel/ice/ice_rxtx_vec_avx2.c | 159 +--- drivers/net/intel/ice/ice_rxtx_vec_avx512.c | 154 +--- drivers/net/intel/ice/ice_rxtx_vec_common.h | 62 +- drivers/net/intel/idpf/idpf_rxtx.c | 3 +- 19 files changed, 1068 insertions(+), 1502 deletions(-) create mode 100644 drivers/net/intel/common/tx_vec_x86.h
--
2.53.0
PATCH v3 1/15 net/iavf: remove unnecessary alignment calls
When we have a context descriptor on Tx of a packet from the vector paths, that means that we always have 32-bytes being written per descriptor/per packet, so the stores are always 32-bit aligned. This means we can use aligned stores for each single descriptor store, and that we never need to do an initial descriptor write for alignment when doing a burst of descriptors.
Signed-off-by Bruce Richardson <[email protected]>
Acked-by Ciara Loftus <[email protected]>
--- drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c | 9 ++------- drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c | 9 ++------- 2 files changed, 4 insertions(+), 14 deletions(-)
drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c +2 -7
@@ -1933,7 +1933,8 @@ ctx_vtx1(volatile struct ci_tx_desc *txdp, struct rte_mbuf *pkt,__m256i ctx_data_desc =_mm256_set_epi64x (high_data_qw, pkt->buf_iova + pkt->data_off, high_ctx_qw, low_ctx_qw); - _mm256_storeu_si256(RTE_CAST_PTR(__m256i *, txdp), ctx_data_desc); +/* tx_id is always even in ctx mode, so txdp is always 32-byte aligned */ + _mm256_store_si256(RTE_CAST_PTR(__m256i *, txdp), ctx_data_desc); }static __rte_always_inline void
@@ -1944,12 +1945,6 @@ ctx_vtx(volatile struct ci_tx_desc *txdp,uint64_t hi_data_qw_tmpl = (IAVF_TX_DESC_DTYPE_DATA | ((uint64_t )flags <<IAVF_TXD_QW1_CMD_SHIFT )); -/* if unaligned on 32-bit boundary, do one to align */ -if (((uintptr_t )txdp &0x1F ) !=0 && nb_pkts !=0 ) { -ctx_vtx1 (txdp, *pkt, flags, offload, vlan_flag, ptype_lldp_enabled); - nb_pkts--; txdp++; pkt++; - } -for (; nb_pkts >1 ; txdp +=4 , pkt +=2 , nb_pkts -=2 ) {uint64_t hi_ctx_qw1 =IAVF_TX_DESC_DTYPE_CONTEXT ;uint64_t hi_ctx_qw0 =IAVF_TX_DESC_DTYPE_CONTEXT ;
drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c +2 -7
@@ -2078,7 +2078,8 @@ ctx_vtx1(volatile struct ci_tx_desc *txdp, struct rte_mbuf *pkt,__m256i ctx_data_desc =_mm256_set_epi64x (high_data_qw, pkt->buf_iova + pkt->data_off, high_ctx_qw, low_ctx_qw); - _mm256_storeu_si256(RTE_CAST_PTR(__m256i *, txdp), ctx_data_desc); +/* tx_id is always even in ctx mode, so txdp is always 32-byte aligned */ + _mm256_store_si256(RTE_CAST_PTR(__m256i *, txdp), ctx_data_desc); }static __rte_always_inline void
@@ -2088,12 +2089,6 @@ ctx_vtx(volatile struct ci_tx_desc *txdp, {uint64_t hi_data_qw_tmpl = (CI_TX_DESC_DTYPE_DATA | (flags <<CI_TXD_QW1_CMD_S )); -/* if unaligned on 32-bit boundary, do one to align */ -if (((uintptr_t )txdp &0x1F ) !=0 && nb_pkts !=0 ) { -ctx_vtx1 (txdp, *pkt, flags, offload, vlan_flag, lldp_enabled); - nb_pkts--; txdp++; pkt++; - } -for (; nb_pkts >1 ; txdp +=4 , pkt +=2 , nb_pkts -=2 ) {uint64_t hi_ctx_qw1 =IAVF_TX_DESC_DTYPE_CONTEXT ;uint64_t hi_ctx_qw0 =IAVF_TX_DESC_DTYPE_CONTEXT ;
Signature · 1 line
2.53.0
PATCH v3 2/15 net/intel: make Tx context flag common
In reply to Bruce RichardsonThe flag for enabling context descriptors in the vector paths should not be limited to just the iavf driver, as other drivers may use context paths in future also. Therefore move the flag out of the iavf-specific part of the Tx queue structure, to put with other standard queue flags.
Signed-off-by Bruce Richardson <[email protected]>
Acked-by Ciara Loftus <[email protected]>
--- drivers/net/intel/common/tx.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/net/intel/common/tx.h +1 -1
@@ -173,6 +173,7 @@ struct ci_tx_queue {bool tx_deferred_start;/* don't start this queue in dev start */ bool q_set;/* indicate if tx queue has been configured */ bool use_vec_entry;/* use sw_ring_vec (true for vector and simple paths) */ +bool use_ctx;/* with ctx info, each pkt needs two desc in vec paths */ union {/* the VSI this queue belongs to */ struct i40e_vsi *i40e_vsi;struct iavf_vsi *iavf_vsi;
@@ -194,7 +195,6 @@ struct ci_tx_queue {#define IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG2 BIT(1)uint8_t vlan_flag;uint8_t tc; -bool use_ctx;/* with ctx info, each pkt needs two descriptors */ bool lldp_enabled; };struct {/* ixgbe specific values */
Signature · 1 line
2.53.0
PATCH v3 3/15 net/iavf: use separate params for VLAN and QinQ position
In reply to Bruce RichardsonLike is the case with the scalar path, use separate offload parameters for the vector path to distinguish between the position of the single VLAN tag and the QinQ outer tag. With iavf these are currently always the same, but other drivers - with whom we may want to share code - have different positions for these.
Signed-off-by Bruce Richardson <[email protected]>
Acked-by Ciara Loftus <[email protected]>
--- drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c | 64 +++++++++++-------- drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c | 62 ++++++++++-------- drivers/net/intel/iavf/iavf_rxtx_vec_common.h | 11 ++-- 3 files changed, 77 insertions(+), 60 deletions(-)
drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c +37 -27
@@ -1619,13 +1619,14 @@ iavf_recv_scattered_pkts_vec_avx2_flex_rxd_offload(void *rx_queue, static __rte_always_inline voidiavf_vtx1 (volatile struct ci_tx_desc *txdp, -struct rte_mbuf *pkt,uint64_t flags,bool offload,uint8_t vlan_flag) +struct rte_mbuf *pkt,uint64_t flags,bool offload, +enum ci_l2tag_pos single_vlan_pos,enum ci_l2tag_pos qinq_outer_pos) {uint64_t high_qw = (CI_TX_DESC_DTYPE_DATA | ((uint64_t )flags <<CI_TXD_QW1_CMD_S ) | ((uint64_t )pkt->data_len <<CI_TXD_QW1_TX_BUF_SZ_S ));if (offload) -iavf_txd_enable_offload (pkt, &high_qw, vlan_flag); +iavf_txd_enable_offload (pkt, &high_qw, single_vlan_pos, qinq_outer_pos);__m128i descriptor =_mm_set_epi64x (high_qw, pkt->buf_iova + pkt->data_off);
@@ -1634,13 +1635,14 @@ iavf_vtx1(volatile struct ci_tx_desc *txdp, static __rte_always_inline voidiavf_vtx (volatile struct ci_tx_desc *txdp, -struct rte_mbuf **pkt,uint16_t nb_pkts,uint64_t flags,bool offload,uint8_t vlan_flag) +struct rte_mbuf **pkt,uint16_t nb_pkts,uint64_t flags,bool offload, +enum ci_l2tag_pos single_vlan_pos,enum ci_l2tag_pos qinq_outer_pos) {const uint64_t hi_qw_tmpl = (CI_TX_DESC_DTYPE_DATA | (flags <<CI_TXD_QW1_CMD_S ));/* if unaligned on 32-bit boundary, do one to align */ if (((uintptr_t )txdp &0x1F ) !=0 && nb_pkts !=0 ) { -iavf_vtx1 (txdp, *pkt, flags, offload, vlan_flag); +iavf_vtx1 (txdp, *pkt, flags, offload, single_vlan_pos, qinq_outer_pos); nb_pkts--; txdp++; pkt++; }
@@ -1649,19 +1651,19 @@ iavf_vtx(volatile struct ci_tx_desc *txdp,uint64_t hi_qw3 = hi_qw_tmpl | ((uint64_t )pkt[3 ]->data_len <<CI_TXD_QW1_TX_BUF_SZ_S );if (offload) -iavf_txd_enable_offload (pkt[3 ], &hi_qw3, vlan_flag); +iavf_txd_enable_offload (pkt[3 ], &hi_qw3, single_vlan_pos, qinq_outer_pos);uint64_t hi_qw2 = hi_qw_tmpl | ((uint64_t )pkt[2 ]->data_len <<CI_TXD_QW1_TX_BUF_SZ_S );if (offload) -iavf_txd_enable_offload (pkt[2 ], &hi_qw2, vlan_flag); +iavf_txd_enable_offload (pkt[2 ], &hi_qw2, single_vlan_pos, qinq_outer_pos);uint64_t hi_qw1 = hi_qw_tmpl | ((uint64_t )pkt[1 ]->data_len <<CI_TXD_QW1_TX_BUF_SZ_S );if (offload) -iavf_txd_enable_offload (pkt[1 ], &hi_qw1, vlan_flag); +iavf_txd_enable_offload (pkt[1 ], &hi_qw1, single_vlan_pos, qinq_outer_pos);uint64_t hi_qw0 = hi_qw_tmpl | ((uint64_t )pkt[0 ]->data_len <<CI_TXD_QW1_TX_BUF_SZ_S );if (offload) -iavf_txd_enable_offload (pkt[0 ], &hi_qw0, vlan_flag); +iavf_txd_enable_offload (pkt[0 ], &hi_qw0, single_vlan_pos, qinq_outer_pos);__m256i desc2_3 = _mm256_set_epi64x
@@ -1681,7 +1683,7 @@ iavf_vtx(volatile struct ci_tx_desc *txdp,/* do any last ones */ while (nb_pkts) { -iavf_vtx1 (txdp, *pkt, flags, offload, vlan_flag); +iavf_vtx1 (txdp, *pkt, flags, offload, single_vlan_pos, qinq_outer_pos); txdp++; pkt++; nb_pkts--; } }
@@ -1697,6 +1699,9 @@ iavf_xmit_fixed_burst_vec_avx2(void *tx_queue, struct rte_mbuf **tx_pkts,/* bit2 is reserved and must be set to 1 according to Spec */ uint64_t flags =CI_TX_DESC_CMD_EOP |CI_TX_DESC_CMD_ICRC ;uint64_t rs =CI_TX_DESC_CMD_RS | flags; +/* vlan_flag gives both the single-VLAN and the QinQ outer tag position */ +enum ci_l2tag_pos vlan_pos = (txq->vlan_flag &IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG1 ) ? +CI_TAG_IN_DATA_DESC :CI_TAG_IN_CTX_DESC ;if (txq->nb_tx_free < txq->tx_free_thresh)ci_tx_free_bufs_vec (txq, iavf_tx_desc_done, false);
@@ -1716,11 +1721,11 @@ iavf_xmit_fixed_burst_vec_avx2(void *tx_queue, struct rte_mbuf **tx_pkts,if (nb_commit >= n) {ci_tx_backlog_entry_vec (txep, tx_pkts, n); -iavf_vtx (txdp, tx_pkts, n -1 , flags, offload, txq->vlan_flag); +iavf_vtx (txdp, tx_pkts, n -1 , flags, offload, vlan_pos, vlan_pos); tx_pkts += (n -1 ); txdp += (n -1 ); -iavf_vtx1 (txdp, *tx_pkts++, rs, offload, txq->vlan_flag); +iavf_vtx1 (txdp, *tx_pkts++, rs, offload, vlan_pos, vlan_pos); nb_commit = (uint16_t )(nb_commit - n);
@@ -1734,7 +1739,7 @@ iavf_xmit_fixed_burst_vec_avx2(void *tx_queue, struct rte_mbuf **tx_pkts,ci_tx_backlog_entry_vec (txep, tx_pkts, nb_commit); -iavf_vtx (txdp, tx_pkts, nb_commit, flags, offload, txq->vlan_flag); +iavf_vtx (txdp, tx_pkts, nb_commit, flags, offload, vlan_pos, vlan_pos); tx_id = (uint16_t )(tx_id + nb_commit);if (tx_id > txq->tx_next_rs) {
@@ -1903,7 +1908,8 @@ iavf_fill_ctx_desc_tunneling_field(volatile uint64_t *qw0, static __rte_always_inline voidctx_vtx1 (volatile struct ci_tx_desc *txdp,struct rte_mbuf *pkt, -uint64_t flags,bool offload,uint8_t vlan_flag,bool ptype_lldp_enabled) +uint64_t flags,bool offload,enum ci_l2tag_pos single_vlan_pos, +enum ci_l2tag_pos qinq_outer_pos,bool ptype_lldp_enabled) {uint64_t high_ctx_qw =IAVF_TX_DESC_DTYPE_CONTEXT ;uint64_t low_ctx_qw =0 ;
@@ -1911,13 +1917,13 @@ ctx_vtx1(volatile struct ci_tx_desc *txdp, struct rte_mbuf *pkt,if (offload) {iavf_fill_ctx_desc_tunneling_avx2 (&low_ctx_qw, pkt);if (pkt->ol_flags &RTE_MBUF_F_TX_QINQ ) { -uint64_t qinq_tag = vlan_flag &IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG2 ? +uint64_t qinq_tag = qinq_outer_pos ==CI_TAG_IN_CTX_DESC ? (uint64_t )pkt->vlan_tci_outer : (uint64_t )pkt->vlan_tci; high_ctx_qw |=IAVF_TX_CTX_DESC_IL2TAG2 <<IAVF_TXD_CTX_QW1_CMD_SHIFT ; low_ctx_qw |= qinq_tag <<IAVF_TXD_CTX_QW0_L2TAG2_PARAM ; }else if ((pkt->ol_flags &RTE_MBUF_F_TX_VLAN ) && - vlan_flag &IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG2 ) { + single_vlan_pos ==CI_TAG_IN_CTX_DESC ) { high_ctx_qw |=IAVF_TX_CTX_DESC_IL2TAG2 <<IAVF_TXD_CTX_QW1_CMD_SHIFT ; low_ctx_qw |= (uint64_t )pkt->vlan_tci <<IAVF_TXD_CTX_QW0_L2TAG2_PARAM ; }
@@ -1928,7 +1934,7 @@ ctx_vtx1(volatile struct ci_tx_desc *txdp, struct rte_mbuf *pkt, ((uint64_t )flags <<IAVF_TXD_QW1_CMD_SHIFT ) | ((uint64_t)pkt->data_len << IAVF_TXD_QW1_TX_BUF_SZ_SHIFT));if (offload) -iavf_txd_enable_offload (pkt, &high_data_qw, vlan_flag); +iavf_txd_enable_offload (pkt, &high_data_qw, single_vlan_pos, qinq_outer_pos);__m256i ctx_data_desc =_mm256_set_epi64x (high_data_qw, pkt->buf_iova + pkt->data_off, high_ctx_qw, low_ctx_qw);
@@ -1940,7 +1946,8 @@ ctx_vtx1(volatile struct ci_tx_desc *txdp, struct rte_mbuf *pkt, static __rte_always_inline voidctx_vtx (volatile struct ci_tx_desc *txdp,struct rte_mbuf **pkt,uint16_t nb_pkts,uint64_t flags, -bool offload,uint8_t vlan_flag,bool ptype_lldp_enabled) +bool offload,enum ci_l2tag_pos single_vlan_pos,enum ci_l2tag_pos qinq_outer_pos, +bool ptype_lldp_enabled) {uint64_t hi_data_qw_tmpl = (IAVF_TX_DESC_DTYPE_DATA | ((uint64_t )flags <<IAVF_TXD_QW1_CMD_SHIFT ));
@@ -1964,14 +1971,14 @@ ctx_vtx(volatile struct ci_tx_desc *txdp,/* tunnel fill assigns low_ctx_qw1; must run before QinQ/VLAN OR below */ iavf_fill_ctx_desc_tunneling_field (&low_ctx_qw1, pkt[1 ]);if (pkt[1 ]->ol_flags &RTE_MBUF_F_TX_QINQ ) { -uint64_t qinq_tag = vlan_flag &IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG2 ? +uint64_t qinq_tag = qinq_outer_pos ==CI_TAG_IN_CTX_DESC ? (uint64_t )pkt[1 ]->vlan_tci_outer : (uint64_t )pkt[1 ]->vlan_tci; hi_ctx_qw1 |=IAVF_TX_CTX_DESC_IL2TAG2 <<IAVF_TXD_CTX_QW1_CMD_SHIFT ; low_ctx_qw1 |= qinq_tag <<IAVF_TXD_CTX_QW0_L2TAG2_PARAM ; }else if (pkt[1 ]->ol_flags &RTE_MBUF_F_TX_VLAN && - vlan_flag &IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG2 ) { + single_vlan_pos ==CI_TAG_IN_CTX_DESC ) { hi_ctx_qw1 |=IAVF_TX_CTX_DESC_IL2TAG2 <<IAVF_TXD_CTX_QW1_CMD_SHIFT ; low_ctx_qw1 |=
@@ -1985,14 +1992,14 @@ ctx_vtx(volatile struct ci_tx_desc *txdp,/* tunnel fill assigns low_ctx_qw0; must run before QinQ/VLAN OR below */ iavf_fill_ctx_desc_tunneling_field (&low_ctx_qw0, pkt[0 ]);if (pkt[0 ]->ol_flags &RTE_MBUF_F_TX_QINQ ) { -uint64_t qinq_tag = vlan_flag &IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG2 ? +uint64_t qinq_tag = qinq_outer_pos ==CI_TAG_IN_CTX_DESC ? (uint64_t )pkt[0 ]->vlan_tci_outer : (uint64_t )pkt[0 ]->vlan_tci; hi_ctx_qw0 |=IAVF_TX_CTX_DESC_IL2TAG2 <<IAVF_TXD_CTX_QW1_CMD_SHIFT ; low_ctx_qw0 |= qinq_tag <<IAVF_TXD_CTX_QW0_L2TAG2_PARAM ; }else if (pkt[0 ]->ol_flags &RTE_MBUF_F_TX_VLAN && - vlan_flag &IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG2 ) { + single_vlan_pos ==CI_TAG_IN_CTX_DESC ) { hi_ctx_qw0 |=IAVF_TX_CTX_DESC_IL2TAG2 <<IAVF_TXD_CTX_QW1_CMD_SHIFT ; low_ctx_qw0 |=
@@ -2003,8 +2010,8 @@ ctx_vtx(volatile struct ci_tx_desc *txdp, hi_ctx_qw0 |=IAVF_TX_CTX_DESC_SWTCH_UPLINK <<IAVF_TXD_CTX_QW1_CMD_SHIFT ;if (offload) { -iavf_txd_enable_offload (pkt[1 ], &hi_data_qw1, vlan_flag); -iavf_txd_enable_offload (pkt[0 ], &hi_data_qw0, vlan_flag); +iavf_txd_enable_offload (pkt[1 ], &hi_data_qw1, single_vlan_pos, qinq_outer_pos); +iavf_txd_enable_offload (pkt[0 ], &hi_data_qw0, single_vlan_pos, qinq_outer_pos); } __m256i desc2_3 =
@@ -2020,7 +2027,7 @@ ctx_vtx(volatile struct ci_tx_desc *txdp, }if (nb_pkts) -ctx_vtx1 (txdp, *pkt, flags, offload, vlan_flag, ptype_lldp_enabled); +ctx_vtx1 (txdp, *pkt, flags, offload, single_vlan_pos, qinq_outer_pos, ptype_lldp_enabled); }static __rte_always_inline uint16_t
@@ -2035,6 +2042,9 @@ iavf_xmit_fixed_burst_vec_avx2_ctx(void *tx_queue, struct rte_mbuf **tx_pkts,uint64_t flags =IAVF_TX_DESC_CMD_EOP |IAVF_TX_DESC_CMD_ICRC ;uint64_t rs =IAVF_TX_DESC_CMD_RS | flags;bool lldp_enabled = txq->lldp_enabled; +/* vlan_flag gives both the single-VLAN and the QinQ outer tag position */ +enum ci_l2tag_pos vlan_pos = (txq->vlan_flag &IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG1 ) ? +CI_TAG_IN_DATA_DESC :CI_TAG_IN_CTX_DESC ;if (txq->nb_tx_free < txq->tx_free_thresh)ci_tx_free_bufs_vec (txq, iavf_tx_desc_done, true);
@@ -2057,10 +2067,10 @@ iavf_xmit_fixed_burst_vec_avx2_ctx(void *tx_queue, struct rte_mbuf **tx_pkts, nb_mbuf = n >>1 ;ci_tx_backlog_entry_vec (txep, tx_pkts, nb_mbuf); -ctx_vtx (txdp, tx_pkts, nb_mbuf -1 , flags, offload, txq->vlan_flag, lldp_enabled); +ctx_vtx (txdp, tx_pkts, nb_mbuf -1 , flags, offload, vlan_pos, vlan_pos, lldp_enabled); tx_pkts += (nb_mbuf -1 ); txdp += (n -2 ); -ctx_vtx1 (txdp, *tx_pkts++, rs, offload, txq->vlan_flag, lldp_enabled); +ctx_vtx1 (txdp, *tx_pkts++, rs, offload, vlan_pos, vlan_pos, lldp_enabled); nb_commit = (uint16_t )(nb_commit - n);
@@ -2074,7 +2084,7 @@ iavf_xmit_fixed_burst_vec_avx2_ctx(void *tx_queue, struct rte_mbuf **tx_pkts, nb_mbuf = nb_commit >>1 ;ci_tx_backlog_entry_vec (txep, tx_pkts, nb_mbuf); -ctx_vtx (txdp, tx_pkts, nb_mbuf, flags, offload, txq->vlan_flag, lldp_enabled); +ctx_vtx (txdp, tx_pkts, nb_mbuf, flags, offload, vlan_pos, vlan_pos, lldp_enabled); tx_id = (uint16_t )(tx_id + nb_commit);if (tx_id > txq->tx_next_rs) {
drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c +35 -27
200 lines not on this page · raw message
drivers/net/intel/iavf/iavf_rxtx_vec_common.h +5 -6
@@ -121,14 +121,13 @@ iavf_tx_vec_dev_check_default(struct rte_eth_dev *dev) static __rte_always_inline void iavf_txd_enable_offload(__rte_unused struct rte_mbuf *tx_pkt, -uint64_t *txd_hi,uint8_t vlan_flag) +uint64_t *txd_hi,enum ci_l2tag_pos single_vlan_pos, +enum ci_l2tag_pos qinq_outer_pos) {uint64_t ol_flags = tx_pkt->ol_flags;uint32_t td_cmd =0 ;uint32_t td_offset =0 ; -RTE_SET_USED (vlan_flag); -/* Set MACLEN */ if (ol_flags &RTE_MBUF_F_TX_TUNNEL_MASK ) td_offset |= (tx_pkt->outer_l2_len >>1 )
@@ -179,12 +178,12 @@ iavf_txd_enable_offload(__rte_unused struct rte_mbuf *tx_pkt,if (ol_flags &RTE_MBUF_F_TX_QINQ ) { td_cmd |=IAVF_TX_DESC_CMD_IL2TAG1 ; -/* vlan_flag specifies outer tag location for QinQ. */ -if (vlan_flag &IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG1 ) +/* L2Tag1 always carries a tag for QinQ */ +if (qinq_outer_pos ==CI_TAG_IN_DATA_DESC ) *txd_hi |= ((uint64_t )tx_pkt->vlan_tci_outer <<CI_TXD_QW1_L2TAG1_S );else *txd_hi |= ((uint64_t )tx_pkt->vlan_tci <<CI_TXD_QW1_L2TAG1_S ); - }else if (ol_flags &RTE_MBUF_F_TX_VLAN && vlan_flag &IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG1 ) { + }else if (ol_flags &RTE_MBUF_F_TX_VLAN && single_vlan_pos ==CI_TAG_IN_DATA_DESC ) { td_cmd |=CI_TX_DESC_CMD_IL2TAG1 ; *txd_hi |= ((uint64_t )tx_pkt->vlan_tci <<CI_TXD_QW1_L2TAG1_S ); }
Signature · 1 line
2.53.0
PATCH v3 4/15 net/iavf: deduplicate tunnel field fill functions
In reply to Bruce RichardsonEach of the avx2 and avx512 Tx functions used two separate subfunctions to insert the tunneling fields into the context descriptor. Except for some minor differences (a flags guard on one) the two were identical, so remove the one missing the additional guard.
Signed-off-by Bruce Richardson <[email protected]>
Reviewed-by David Marchand <[email protected]>
--- drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c | 77 +------------------ drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c | 77 +------------------ 2 files changed, 4 insertions(+), 150 deletions(-)
drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c +2 -75
@@ -1833,79 +1833,6 @@ iavf_fill_ctx_desc_tunneling_avx2(uint64_t *low_ctx_qw, struct rte_mbuf *pkt) } } -static inline void -iavf_fill_ctx_desc_tunneling_field (volatile uint64_t *qw0, -const struct rte_mbuf *m) -{ -uint64_t eip_typ =IAVF_TX_CTX_DESC_EIPT_NONE ; -uint64_t eip_len =0 ; -uint64_t eip_noinc =0 ; -/* Default - IP_ID is increment in each segment of LSO */ - -switch (m->ol_flags & (RTE_MBUF_F_TX_OUTER_IPV4 | -RTE_MBUF_F_TX_OUTER_IPV6 | -RTE_MBUF_F_TX_OUTER_IP_CKSUM )) { -case RTE_MBUF_F_TX_OUTER_IPV4 : - eip_typ =IAVF_TX_CTX_DESC_EIPT_IPV4_NO_CHECKSUM_OFFLOAD ; - eip_len = m->outer_l3_len >>2 ; -break ; -case RTE_MBUF_F_TX_OUTER_IPV4 |RTE_MBUF_F_TX_OUTER_IP_CKSUM : - eip_typ =IAVF_TX_CTX_DESC_EIPT_IPV4_CHECKSUM_OFFLOAD ; - eip_len = m->outer_l3_len >>2 ; -break ; -case RTE_MBUF_F_TX_OUTER_IPV6 : - eip_typ =IAVF_TX_CTX_DESC_EIPT_IPV6 ; - eip_len = m->outer_l3_len >>2 ; -break ; - } - -/* L4TUNT: L4 Tunneling Type */ -switch (m->ol_flags &RTE_MBUF_F_TX_TUNNEL_MASK ) { -case RTE_MBUF_F_TX_TUNNEL_IPIP : -/* for non UDP / GRE tunneling, set to 00b */ -break ; -case RTE_MBUF_F_TX_TUNNEL_VXLAN : -case RTE_MBUF_F_TX_TUNNEL_VXLAN_GPE : -case RTE_MBUF_F_TX_TUNNEL_GTP : -case RTE_MBUF_F_TX_TUNNEL_GENEVE : - eip_typ |=IAVF_TXD_CTX_UDP_TUNNELING ; -break ; -case RTE_MBUF_F_TX_TUNNEL_GRE : - eip_typ |=IAVF_TXD_CTX_GRE_TUNNELING ; -break ; -default : -PMD_TX_LOG (ERR ,"Tunnel type not supported" ); -return ; - } - -/* L4TUNLEN: L4 Tunneling Length, in Words -* -* We depend on app to set rte_mbuf.l2_len correctly. -* For IP in GRE it should be set to the length of the GRE -* header; -* For MAC in GRE or MAC in UDP it should be set to the length -* of the GRE or UDP headers plus the inner MAC up to including -* its last Ethertype. -* If MPLS labels exists, it should include them as well. -*/ - eip_typ |= (m->l2_len >>1 ) <<IAVF_TXD_CTX_QW0_NATLEN_SHIFT ; - -/** -* Calculate the tunneling UDP checksum. -* Shall be set only if L4TUNT = 01b and EIPT is not zero -*/ -if ((eip_typ & (IAVF_TX_CTX_EXT_IP_IPV6 | -IAVF_TX_CTX_EXT_IP_IPV4 | -IAVF_TX_CTX_EXT_IP_IPV4_NO_CSUM )) && - (eip_typ &IAVF_TXD_CTX_UDP_TUNNELING ) && - (m->ol_flags &RTE_MBUF_F_TX_OUTER_UDP_CKSUM )) - eip_typ |=IAVF_TXD_CTX_QW0_L4T_CS_MASK ; - - *qw0 = eip_typ <<IAVF_TXD_CTX_QW0_TUN_PARAMS_EIPT_SHIFT | - eip_len <<IAVF_TXD_CTX_QW0_TUN_PARAMS_EIPLEN_SHIFT | - eip_noinc <<IAVF_TXD_CTX_QW0_TUN_PARAMS_EIP_NOINC_SHIFT ; -} - static __rte_always_inline void ctx_vtx1(volatile struct ci_tx_desc *txdp, struct rte_mbuf *pkt, uint64_t flags, bool offload, enum ci_l2tag_pos single_vlan_pos,
@@ -1969,7 +1896,7 @@ ctx_vtx(volatile struct ci_tx_desc *txdp,if (offload) {/* tunnel fill assigns low_ctx_qw1; must run before QinQ/VLAN OR below */ -iavf_fill_ctx_desc_tunneling_field (&low_ctx_qw1, pkt[1 ]); +iavf_fill_ctx_desc_tunneling_avx2 (&low_ctx_qw1, pkt[1 ]);if (pkt[1 ]->ol_flags &RTE_MBUF_F_TX_QINQ ) {uint64_t qinq_tag = qinq_outer_pos ==CI_TAG_IN_CTX_DESC ? (uint64_t)pkt[1]->vlan_tci_outer :
@@ -1990,7 +1917,7 @@ ctx_vtx(volatile struct ci_tx_desc *txdp,if (offload) {/* tunnel fill assigns low_ctx_qw0; must run before QinQ/VLAN OR below */ -iavf_fill_ctx_desc_tunneling_field (&low_ctx_qw0, pkt[0 ]); +iavf_fill_ctx_desc_tunneling_avx2 (&low_ctx_qw0, pkt[0 ]);if (pkt[0 ]->ol_flags &RTE_MBUF_F_TX_QINQ ) {uint64_t qinq_tag = qinq_outer_pos ==CI_TAG_IN_CTX_DESC ? (uint64_t)pkt[0]->vlan_tci_outer :
drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c +2 -75
95 lines not on this page · raw message
Signature · 1 line
2.53.0
PATCH v3 5/15 net/intel: define common macros for tunneling bit-shifts
In reply to Bruce RichardsonDefine common macros for use in iavf (for now) and other drivers (future) defining the shifts needed for tunnel packet Tx
Signed-off-by Bruce Richardson <[email protected]>
Acked-by Ciara Loftus <[email protected]>
--- drivers/net/intel/common/tx.h | 24 +++++++++++++ drivers/net/intel/iavf/iavf_rxtx.c | 30 ++++++++-------- drivers/net/intel/iavf/iavf_rxtx.h | 34 ------------------- drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c | 30 ++++++++-------- drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c | 30 ++++++++-------- 5 files changed, 69 insertions(+), 79 deletions(-)
drivers/net/intel/common/tx.h +24 -0
@@ -45,6 +45,30 @@#define CI_TX_CTX_DESC_TSYN 0x02#define CI_TX_CTX_DESC_IL2TAG2 0x04 +/* Common TX Context Descriptor QW0 Field Shifts */ +#define CI_TXD_CTX_QW0_L2TAG2_S 32 +#define CI_TXD_CTX_QW0_EIPT_S 0/* 2 BITS - external IP type */ +#define CI_TXD_CTX_QW0_EIPLEN_S 2/* 7 BITS - external IP header length */ +#define CI_TXD_CTX_QW0_NATT_S 9/* 2 BITS - L4 tunnel type */ +#define CI_TXD_CTX_QW0_EIP_NOINC_S 11/* 1 BIT - external IP ID not incremented */ +#define CI_TXD_CTX_QW0_NATLEN_S 12/* 7 BITS - L4 tunnel header length */ +#define CI_TXD_CTX_QW0_L4T_CS_S 23/* 1 BIT - tunnel UDP checksum enable */ + +/** +* Enum for the external (outer) IP type field of a Tx context descriptor's +* tunnel parameters (QW0 bits 0-1). +*/ +enum ci_tx_ctx_eipt { +CI_TX_CTX_EIPT_NONE , +CI_TX_CTX_EIPT_IPV6 , +CI_TX_CTX_EIPT_IPV4_NO_CSUM , +CI_TX_CTX_EIPT_IPV4 , +}; + +#define CI_TXD_CTX_UDP_TUNNELING (0x1ULL << CI_TXD_CTX_QW0_NATT_S) +#define CI_TXD_CTX_GRE_TUNNELING (0x2ULL << CI_TXD_CTX_QW0_NATT_S) +#define CI_TXD_CTX_QW0_L4T_CS_M (0x1ULL << CI_TXD_CTX_QW0_L4T_CS_S) +/** * Enum to specify where a VLAN tag is to be placed for packet Tx. */
drivers/net/intel/iavf/iavf_rxtx.c +15 -15
@@ -2345,7 +2345,7 @@ iavf_calc_context_desc(const struct rte_mbuf *mb, uint8_t vlan_flag, bool lldp_estatic inline void iavf_fill_ctx_desc_tunnelling_field (uint64_t *qw0,uint64_t ol_flags,const struct rte_mbuf *m) { -uint64_t eip_typ =IAVF_TX_CTX_DESC_EIPT_NONE ; +uint64_t eip_typ =CI_TX_CTX_EIPT_NONE ;uint64_t eip_len =0 ;uint64_t eip_noinc =0 ;/* Default - IP_ID is increment in each segment of LSO */
@@ -2354,15 +2354,15 @@ iavf_fill_ctx_desc_tunnelling_field(uint64_t *qw0, uint64_t ol_flags, const stru RTE_MBUF_F_TX_OUTER_IPV6 | RTE_MBUF_F_TX_OUTER_IP_CKSUM)) {case RTE_MBUF_F_TX_OUTER_IPV4 : - eip_typ =IAVF_TX_CTX_DESC_EIPT_IPV4_NO_CHECKSUM_OFFLOAD ; + eip_typ =CI_TX_CTX_EIPT_IPV4_NO_CSUM ; eip_len = m->outer_l3_len >>2 ;break ;case RTE_MBUF_F_TX_OUTER_IPV4 |RTE_MBUF_F_TX_OUTER_IP_CKSUM : - eip_typ =IAVF_TX_CTX_DESC_EIPT_IPV4_CHECKSUM_OFFLOAD ; + eip_typ =CI_TX_CTX_EIPT_IPV4 ; eip_len = m->outer_l3_len >>2 ;break ;case RTE_MBUF_F_TX_OUTER_IPV6 : - eip_typ =IAVF_TX_CTX_DESC_EIPT_IPV6 ; + eip_typ =CI_TX_CTX_EIPT_IPV6 ; eip_len = m->outer_l3_len >>2 ;break ; }
@@ -2377,10 +2377,10 @@ iavf_fill_ctx_desc_tunnelling_field(uint64_t *qw0, uint64_t ol_flags, const strucase RTE_MBUF_F_TX_TUNNEL_VXLAN_GPE :case RTE_MBUF_F_TX_TUNNEL_GTP :case RTE_MBUF_F_TX_TUNNEL_GENEVE : - eip_typ |=IAVF_TXD_CTX_UDP_TUNNELING ; + eip_typ |=CI_TXD_CTX_UDP_TUNNELING ;break ;case RTE_MBUF_F_TX_TUNNEL_GRE : - eip_typ |=IAVF_TXD_CTX_GRE_TUNNELING ; + eip_typ |=CI_TXD_CTX_GRE_TUNNELING ;break ;default :PMD_TX_LOG (ERR ,"Tunnel type not supported" );
@@ -2397,23 +2397,23 @@ iavf_fill_ctx_desc_tunnelling_field(uint64_t *qw0, uint64_t ol_flags, const stru* its last Ethertype. * If MPLS labels exists, it should include them as well. */ - eip_typ |= (m->l2_len >>1 ) <<IAVF_TXD_CTX_QW0_NATLEN_SHIFT ; + eip_typ |= (m->l2_len >>1 ) <<CI_TXD_CTX_QW0_NATLEN_S ;/** * Calculate the tunneling UDP checksum. * Shall be set only if L4TUNT = 01b and EIPT is not zero */ -if ((eip_typ & (IAVF_TX_CTX_EXT_IP_IPV6 | -IAVF_TX_CTX_EXT_IP_IPV4 | -IAVF_TX_CTX_EXT_IP_IPV4_NO_CSUM )) && - (eip_typ &IAVF_TXD_CTX_UDP_TUNNELING ) && +if ((eip_typ & (CI_TX_CTX_EIPT_IPV6 | +CI_TX_CTX_EIPT_IPV4 | +CI_TX_CTX_EIPT_IPV4_NO_CSUM )) && + (eip_typ &CI_TXD_CTX_UDP_TUNNELING ) && (ol_flags &RTE_MBUF_F_TX_OUTER_UDP_CKSUM )) - eip_typ |=IAVF_TXD_CTX_QW0_L4T_CS_MASK ; + eip_typ |=CI_TXD_CTX_QW0_L4T_CS_M ; } - *qw0 = eip_typ <<IAVF_TXD_CTX_QW0_TUN_PARAMS_EIPT_SHIFT | - eip_len <<IAVF_TXD_CTX_QW0_TUN_PARAMS_EIPLEN_SHIFT | - eip_noinc <<IAVF_TXD_CTX_QW0_TUN_PARAMS_EIP_NOINC_SHIFT ; + *qw0 = eip_typ <<CI_TXD_CTX_QW0_EIPT_S | + eip_len <<CI_TXD_CTX_QW0_EIPLEN_S | + eip_noinc <<CI_TXD_CTX_QW0_EIP_NOINC_S ; } static inline uint16_t
drivers/net/intel/iavf/iavf_rxtx.h +0 -34
@@ -444,40 +444,6 @@ enum iavf_rx_flex_desc_ipsec_crypto_status {#define IAVF_TXD_CTX_QW1_SEG_PARAMS_MSS_MASK \ (0x3FFFUL << IAVF_TXD_CTX_QW1_SEG_PARAMS_MSS_SHIFT) -#define IAVF_TXD_CTX_QW0_TUN_PARAMS_EIPT_SHIFT (0) -#define IAVF_TXD_CTX_QW0_TUN_PARAMS_EIPT_MASK (0x3UL) - -enum iavf_tx_ctx_desc_tunnel_external_ip_type { -IAVF_TX_CTX_DESC_EIPT_NONE , -IAVF_TX_CTX_DESC_EIPT_IPV6 , -IAVF_TX_CTX_DESC_EIPT_IPV4_NO_CHECKSUM_OFFLOAD , -IAVF_TX_CTX_DESC_EIPT_IPV4_CHECKSUM_OFFLOAD -}; - -#define IAVF_TXD_CTX_QW0_TUN_PARAMS_EIPLEN_SHIFT (2) -#define IAVF_TXD_CTX_QW0_TUN_PARAMS_EIPLEN_MASK (0x7FUL) - -#define IAVF_TXD_CTX_QW0_TUN_PARAMS_L4TUNT_SHIFT (9) -#define IAVF_TXD_CTX_QW0_TUN_PARAMS_L4TUNT_MASK (0x3UL) - -enum iavf_tx_ctx_desc_tunnel_l4_tunnel_type { -IAVF_TX_CTX_DESC_L4_TUN_TYP_NO_UDP_GRE , -IAVF_TX_CTX_DESC_L4_TUN_TYP_UDP , -IAVF_TX_CTX_DESC_L4_TUN_TYP_GRE -}; - -#define IAVF_TXD_CTX_QW0_TUN_PARAMS_EIP_NOINC_SHIFT (11) -#define IAVF_TXD_CTX_QW0_TUN_PARAMS_EIP_NOINC_MASK (0x1UL) - -#define IAVF_TXD_CTX_QW0_TUN_PARAMS_L4TUNLEN_SHIFT (12) -#define IAVF_TXD_CTX_QW0_TUN_PARAMS_L4TUNLEN_MASK (0x7FUL) - -#define IAVF_TXD_CTX_QW0_TUN_PARAMS_DECTTL_SHIFT (19) -#define IAVF_TXD_CTX_QW0_TUN_PARAMS_DECTTL_MASK (0xFUL) - -#define IAVF_TXD_CTX_QW0_TUN_PARAMS_L4T_CS_SHIFT (23) -#define IAVF_TXD_CTX_QW0_TUN_PARAMS_L4T_CS_MASK (0x1UL) -#define IAVF_TXD_CTX_QW0_L2TAG2_PARAM (32)#define IAVF_TXD_CTX_QW0_L2TAG2_MASK (0xFFFFUL)
drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c +15 -15
@@ -1760,7 +1760,7 @@ static inline void iavf_fill_ctx_desc_tunneling_avx2(uint64_t *low_ctx_qw, struct rte_mbuf *pkt) {if (pkt->ol_flags &RTE_MBUF_F_TX_TUNNEL_MASK ) { -uint64_t eip_typ =IAVF_TX_CTX_DESC_EIPT_NONE ; +uint64_t eip_typ =CI_TX_CTX_EIPT_NONE ;uint64_t eip_len =0 ;uint64_t eip_noinc =0 ;/* Default - IP_ID is increment in each segment of LSO */
@@ -1769,15 +1769,15 @@ iavf_fill_ctx_desc_tunneling_avx2(uint64_t *low_ctx_qw, struct rte_mbuf *pkt) RTE_MBUF_F_TX_OUTER_IPV6 | RTE_MBUF_F_TX_OUTER_IP_CKSUM)) {case RTE_MBUF_F_TX_OUTER_IPV4 : - eip_typ =IAVF_TX_CTX_DESC_EIPT_IPV4_NO_CHECKSUM_OFFLOAD ; + eip_typ =CI_TX_CTX_EIPT_IPV4_NO_CSUM ; eip_len = pkt->outer_l3_len >>2 ;break ;case RTE_MBUF_F_TX_OUTER_IPV4 |RTE_MBUF_F_TX_OUTER_IP_CKSUM : - eip_typ =IAVF_TX_CTX_DESC_EIPT_IPV4_CHECKSUM_OFFLOAD ; + eip_typ =CI_TX_CTX_EIPT_IPV4 ; eip_len = pkt->outer_l3_len >>2 ;break ;case RTE_MBUF_F_TX_OUTER_IPV6 : - eip_typ =IAVF_TX_CTX_DESC_EIPT_IPV6 ; + eip_typ =CI_TX_CTX_EIPT_IPV6 ; eip_len = pkt->outer_l3_len >>2 ;break ; }
@@ -1791,10 +1791,10 @@ iavf_fill_ctx_desc_tunneling_avx2(uint64_t *low_ctx_qw, struct rte_mbuf *pkt)case RTE_MBUF_F_TX_TUNNEL_VXLAN_GPE :case RTE_MBUF_F_TX_TUNNEL_GTP :case RTE_MBUF_F_TX_TUNNEL_GENEVE : - eip_typ |=IAVF_TXD_CTX_UDP_TUNNELING ; + eip_typ |=CI_TXD_CTX_UDP_TUNNELING ;break ;case RTE_MBUF_F_TX_TUNNEL_GRE : - eip_typ |=IAVF_TXD_CTX_GRE_TUNNELING ; + eip_typ |=CI_TXD_CTX_GRE_TUNNELING ;break ;default :PMD_TX_LOG (ERR ,"Tunnel type not supported" );
@@ -1811,22 +1811,22 @@ iavf_fill_ctx_desc_tunneling_avx2(uint64_t *low_ctx_qw, struct rte_mbuf *pkt)* its last Ethertype. * If MPLS labels exists, it should include them as well. */ - eip_typ |= (pkt->l2_len >>1 ) <<IAVF_TXD_CTX_QW0_NATLEN_SHIFT ; + eip_typ |= (pkt->l2_len >>1 ) <<CI_TXD_CTX_QW0_NATLEN_S ;/** * Calculate the tunneling UDP checksum. * Shall be set only if L4TUNT = 01b and EIPT is not zero */ -if ((eip_typ & (IAVF_TX_CTX_EXT_IP_IPV4 | -IAVF_TX_CTX_EXT_IP_IPV6 | -IAVF_TX_CTX_EXT_IP_IPV4_NO_CSUM )) && - (eip_typ &IAVF_TXD_CTX_UDP_TUNNELING ) && +if ((eip_typ & (CI_TX_CTX_EIPT_IPV4 | +CI_TX_CTX_EIPT_IPV6 | +CI_TX_CTX_EIPT_IPV4_NO_CSUM )) && + (eip_typ &CI_TXD_CTX_UDP_TUNNELING ) && (pkt->ol_flags &RTE_MBUF_F_TX_OUTER_UDP_CKSUM )) - eip_typ |=IAVF_TXD_CTX_QW0_L4T_CS_MASK ; + eip_typ |=CI_TXD_CTX_QW0_L4T_CS_M ; - *low_ctx_qw = eip_typ <<IAVF_TXD_CTX_QW0_TUN_PARAMS_EIPT_SHIFT | - eip_len <<IAVF_TXD_CTX_QW0_TUN_PARAMS_EIPLEN_SHIFT | - eip_noinc <<IAVF_TXD_CTX_QW0_TUN_PARAMS_EIP_NOINC_SHIFT ; + *low_ctx_qw = eip_typ <<CI_TXD_CTX_QW0_EIPT_S | + eip_len <<CI_TXD_CTX_QW0_EIPLEN_S | + eip_noinc <<CI_TXD_CTX_QW0_EIP_NOINC_S ; }else { *low_ctx_qw =0 ;
drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c +15 -15
@@ -1899,7 +1899,7 @@ static __rte_always_inline void iavf_fill_ctx_desc_tunneling_avx512(uint64_t *low_ctx_qw, struct rte_mbuf *pkt) {if (pkt->ol_flags &RTE_MBUF_F_TX_TUNNEL_MASK ) { -uint64_t eip_typ =IAVF_TX_CTX_DESC_EIPT_NONE ; +uint64_t eip_typ =CI_TX_CTX_EIPT_NONE ;uint64_t eip_len =0 ;uint64_t eip_noinc =0 ;/* Default - IP_ID is increment in each segment of LSO */
@@ -1908,15 +1908,15 @@ iavf_fill_ctx_desc_tunneling_avx512(uint64_t *low_ctx_qw, struct rte_mbuf *pkt) RTE_MBUF_F_TX_OUTER_IPV6 | RTE_MBUF_F_TX_OUTER_IP_CKSUM)) {case RTE_MBUF_F_TX_OUTER_IPV4 : - eip_typ =IAVF_TX_CTX_DESC_EIPT_IPV4_NO_CHECKSUM_OFFLOAD ; + eip_typ =CI_TX_CTX_EIPT_IPV4_NO_CSUM ; eip_len = pkt->outer_l3_len >>2 ;break ;case RTE_MBUF_F_TX_OUTER_IPV4 |RTE_MBUF_F_TX_OUTER_IP_CKSUM : - eip_typ =IAVF_TX_CTX_DESC_EIPT_IPV4_CHECKSUM_OFFLOAD ; + eip_typ =CI_TX_CTX_EIPT_IPV4 ; eip_len = pkt->outer_l3_len >>2 ;break ;case RTE_MBUF_F_TX_OUTER_IPV6 : - eip_typ =IAVF_TX_CTX_DESC_EIPT_IPV6 ; + eip_typ =CI_TX_CTX_EIPT_IPV6 ; eip_len = pkt->outer_l3_len >>2 ;break ; }
@@ -1930,10 +1930,10 @@ iavf_fill_ctx_desc_tunneling_avx512(uint64_t *low_ctx_qw, struct rte_mbuf *pkt)case RTE_MBUF_F_TX_TUNNEL_VXLAN_GPE :case RTE_MBUF_F_TX_TUNNEL_GTP :case RTE_MBUF_F_TX_TUNNEL_GENEVE : - eip_typ |=IAVF_TXD_CTX_UDP_TUNNELING ; + eip_typ |=CI_TXD_CTX_UDP_TUNNELING ;break ;case RTE_MBUF_F_TX_TUNNEL_GRE : - eip_typ |=IAVF_TXD_CTX_GRE_TUNNELING ; + eip_typ |=CI_TXD_CTX_GRE_TUNNELING ;break ;default :PMD_TX_LOG (ERR ,"Tunnel type not supported" );
@@ -1950,22 +1950,22 @@ iavf_fill_ctx_desc_tunneling_avx512(uint64_t *low_ctx_qw, struct rte_mbuf *pkt)* its last Ethertype. * If MPLS labels exists, it should include them as well. */ - eip_typ |= (pkt->l2_len >>1 ) <<IAVF_TXD_CTX_QW0_NATLEN_SHIFT ; + eip_typ |= (pkt->l2_len >>1 ) <<CI_TXD_CTX_QW0_NATLEN_S ;/** * Calculate the tunneling UDP checksum. * Shall be set only if L4TUNT = 01b and EIPT is not zero */ -if ((eip_typ & (IAVF_TX_CTX_EXT_IP_IPV4 | -IAVF_TX_CTX_EXT_IP_IPV6 | -IAVF_TX_CTX_EXT_IP_IPV4_NO_CSUM )) && - (eip_typ &IAVF_TXD_CTX_UDP_TUNNELING ) && +if ((eip_typ & (CI_TX_CTX_EIPT_IPV4 | +CI_TX_CTX_EIPT_IPV6 | +CI_TX_CTX_EIPT_IPV4_NO_CSUM )) && + (eip_typ &CI_TXD_CTX_UDP_TUNNELING ) && (pkt->ol_flags &RTE_MBUF_F_TX_OUTER_UDP_CKSUM )) - eip_typ |=IAVF_TXD_CTX_QW0_L4T_CS_MASK ; + eip_typ |=CI_TXD_CTX_QW0_L4T_CS_M ; - *low_ctx_qw = eip_typ <<IAVF_TXD_CTX_QW0_TUN_PARAMS_EIPT_SHIFT | - eip_len <<IAVF_TXD_CTX_QW0_TUN_PARAMS_EIPLEN_SHIFT | - eip_noinc <<IAVF_TXD_CTX_QW0_TUN_PARAMS_EIP_NOINC_SHIFT ; + *low_ctx_qw = eip_typ <<CI_TXD_CTX_QW0_EIPT_S | + eip_len <<CI_TXD_CTX_QW0_EIPLEN_S | + eip_noinc <<CI_TXD_CTX_QW0_EIP_NOINC_S ; }else { *low_ctx_qw =0 ;
Signature · 1 line
2.53.0
PATCH v3 6/15 net/intel: move iavf descriptor writing functions to common
In reply to Bruce RichardsonIn order to promote reuse of code, move the Tx vector functions for writing descriptors from the iavf driver to the common folder in a new tx_vec_x86.h file. This will allow reuse in later commits by other drivers such as ice.
Signed-off-by Bruce Richardson <[email protected]>
Acked-by Ciara Loftus <[email protected]>
--- drivers/net/intel/common/tx_vec_x86.h | 511 ++++++++++++++++++ drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c | 285 +--------- drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c | 272 +--------- drivers/net/intel/iavf/iavf_rxtx_vec_common.h | 75 +-- 4 files changed, 530 insertions(+), 613 deletions(-) create mode 100644 drivers/net/intel/common/tx_vec_x86.h
drivers/net/intel/common/tx_vec_x86.h new file +511 -0
@@ -0,0 +1,511 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2026 Intel Corporation +*/ + +#ifndef _COMMON_INTEL_TX_VEC_X86_H_ +#define _COMMON_INTEL_TX_VEC_X86_H_ + +#include <stdint.h> + +#include <rte_mbuf.h> + +#include "tx.h" + +static __rte_always_inline void +ci_fill_ctx_desc_tunneling (uint64_t *low_ctx_qw,struct rte_mbuf *pkt) +{ +if (pkt->ol_flags &RTE_MBUF_F_TX_TUNNEL_MASK ) { +uint64_t eip_typ =CI_TX_CTX_EIPT_NONE ; +uint64_t eip_len =0 ; +uint64_t eip_noinc =0 ; +/* Default - IP_ID is increment in each segment of LSO */ + +switch (pkt->ol_flags & (RTE_MBUF_F_TX_OUTER_IPV4 | +RTE_MBUF_F_TX_OUTER_IPV6 | +RTE_MBUF_F_TX_OUTER_IP_CKSUM )) { +case RTE_MBUF_F_TX_OUTER_IPV4 : + eip_typ =CI_TX_CTX_EIPT_IPV4_NO_CSUM ; + eip_len = pkt->outer_l3_len >>2 ; +break ; +case RTE_MBUF_F_TX_OUTER_IPV4 |RTE_MBUF_F_TX_OUTER_IP_CKSUM : + eip_typ =CI_TX_CTX_EIPT_IPV4 ; + eip_len = pkt->outer_l3_len >>2 ; +break ; +case RTE_MBUF_F_TX_OUTER_IPV6 : + eip_typ =CI_TX_CTX_EIPT_IPV6 ; + eip_len = pkt->outer_l3_len >>2 ; +break ; + } + +/* L4TUNT: L4 Tunneling Type */ +switch (pkt->ol_flags &RTE_MBUF_F_TX_TUNNEL_MASK ) { +case RTE_MBUF_F_TX_TUNNEL_IPIP : +/* for non UDP / GRE tunneling, set to 00b */ +break ; +case RTE_MBUF_F_TX_TUNNEL_VXLAN : +case RTE_MBUF_F_TX_TUNNEL_VXLAN_GPE : +case RTE_MBUF_F_TX_TUNNEL_GTP : +case RTE_MBUF_F_TX_TUNNEL_GENEVE : + eip_typ |=CI_TXD_CTX_UDP_TUNNELING ; +break ; +case RTE_MBUF_F_TX_TUNNEL_GRE : + eip_typ |=CI_TXD_CTX_GRE_TUNNELING ; +break ; +default : +PMD_TX_LOG (ERR ,"Tunnel type not supported" ); +return ; + } + +/* L4TUNLEN: L4 Tunneling Length, in Words +* +* We depend on app to set rte_mbuf.l2_len correctly. +* For IP in GRE it should be set to the length of the GRE +* header; +* For MAC in GRE or MAC in UDP it should be set to the length +* of the GRE or UDP headers plus the inner MAC up to including +* its last Ethertype. +* If MPLS labels exists, it should include them as well. +*/ + eip_typ |= (pkt->l2_len >>1 ) <<CI_TXD_CTX_QW0_NATLEN_S ; + +/** +* Calculate the tunneling UDP checksum. +* Shall be set only if L4TUNT = 01b and EIPT is not zero +*/ +if ((eip_typ & (CI_TX_CTX_EIPT_IPV4 | +CI_TX_CTX_EIPT_IPV6 | +CI_TX_CTX_EIPT_IPV4_NO_CSUM )) && + (eip_typ &CI_TXD_CTX_UDP_TUNNELING ) && + (pkt->ol_flags &RTE_MBUF_F_TX_OUTER_UDP_CKSUM )) + eip_typ |=CI_TXD_CTX_QW0_L4T_CS_M ; + + *low_ctx_qw = eip_typ <<CI_TXD_CTX_QW0_EIPT_S | + eip_len <<CI_TXD_CTX_QW0_EIPLEN_S | + eip_noinc <<CI_TXD_CTX_QW0_EIP_NOINC_S ; + + }else { + *low_ctx_qw =0 ; + } +} + +static __rte_always_inline void +ci_tx_vec_offload (struct rte_mbuf *tx_pkt,uint64_t *txd_hi, +enum ci_l2tag_pos single_vlan_pos,enum ci_l2tag_pos qinq_outer_pos) +{ +uint64_t ol_flags = tx_pkt->ol_flags; +uint32_t td_cmd =0 ; +uint32_t td_offset =0 ; + +/* Set MACLEN */ +if (ol_flags &RTE_MBUF_F_TX_TUNNEL_MASK ) + td_offset |= (tx_pkt->outer_l2_len >>1 ) <<CI_TX_DESC_LEN_MACLEN_S ; +else + td_offset |= (tx_pkt->l2_len >>1 ) <<CI_TX_DESC_LEN_MACLEN_S ; + +/* Enable L3 checksum offloads */ +if (ol_flags &RTE_MBUF_F_TX_IP_CKSUM ) { +if (ol_flags &RTE_MBUF_F_TX_IPV4 ) { + td_cmd |=CI_TX_DESC_CMD_IIPT_IPV4_CSUM ; + td_offset |= (tx_pkt->l3_len >>2 ) <<CI_TX_DESC_LEN_IPLEN_S ; + } + }else if (ol_flags &RTE_MBUF_F_TX_IPV4 ) { + td_cmd |=CI_TX_DESC_CMD_IIPT_IPV4 ; + td_offset |= (tx_pkt->l3_len >>2 ) <<CI_TX_DESC_LEN_IPLEN_S ; + }else if (ol_flags &RTE_MBUF_F_TX_IPV6 ) { + td_cmd |=CI_TX_DESC_CMD_IIPT_IPV6 ; + td_offset |= (tx_pkt->l3_len >>2 ) <<CI_TX_DESC_LEN_IPLEN_S ; + } + +/* Enable L4 checksum offloads */ +switch (ol_flags &RTE_MBUF_F_TX_L4_MASK ) { +case RTE_MBUF_F_TX_TCP_CKSUM : + td_cmd |=CI_TX_DESC_CMD_L4T_EOFT_TCP ; + td_offset |= (sizeof (struct rte_tcp_hdr ) >>2 ) <<CI_TX_DESC_LEN_L4_LEN_S ; +break ; +case RTE_MBUF_F_TX_SCTP_CKSUM : + td_cmd |=CI_TX_DESC_CMD_L4T_EOFT_SCTP ; + td_offset |= (sizeof (struct rte_sctp_hdr ) >>2 ) <<CI_TX_DESC_LEN_L4_LEN_S ; +break ; +case RTE_MBUF_F_TX_UDP_CKSUM : + td_cmd |=CI_TX_DESC_CMD_L4T_EOFT_UDP ; + td_offset |= (sizeof (struct rte_udp_hdr ) >>2 ) <<CI_TX_DESC_LEN_L4_LEN_S ; +break ; +default : +break ; + } + + *txd_hi |= ((uint64_t )td_offset) <<CI_TXD_QW1_OFFSET_S ; + +if (ol_flags &RTE_MBUF_F_TX_QINQ ) { + td_cmd |=CI_TX_DESC_CMD_IL2TAG1 ; +/* L2Tag1 always carries a tag for QinQ */ +if (qinq_outer_pos ==CI_TAG_IN_DATA_DESC ) + *txd_hi |= ((uint64_t )tx_pkt->vlan_tci_outer <<CI_TXD_QW1_L2TAG1_S ); +else + *txd_hi |= ((uint64_t )tx_pkt->vlan_tci <<CI_TXD_QW1_L2TAG1_S ); + }else if (ol_flags &RTE_MBUF_F_TX_VLAN && single_vlan_pos ==CI_TAG_IN_DATA_DESC ) { + td_cmd |=CI_TX_DESC_CMD_IL2TAG1 ; + *txd_hi |= ((uint64_t )tx_pkt->vlan_tci <<CI_TXD_QW1_L2TAG1_S ); + } + + *txd_hi |= ((uint64_t )td_cmd) <<CI_TXD_QW1_CMD_S ; +} + +static __rte_always_inline void +ci_vtx1 (volatile struct ci_tx_desc *txdp, +struct rte_mbuf *pkt,uint64_t flags,bool offload, +enum ci_l2tag_pos single_vlan_pos,enum ci_l2tag_pos qinq_outer_pos) +{ +uint64_t high_qw = (CI_TX_DESC_DTYPE_DATA | + ((uint64_t )flags <<CI_TXD_QW1_CMD_S ) | + ((uint64_t )pkt->data_len <<CI_TXD_QW1_TX_BUF_SZ_S )); +if (offload) +ci_tx_vec_offload (pkt, &high_qw, single_vlan_pos, qinq_outer_pos); + +__m128i descriptor =_mm_set_epi64x (high_qw, pkt->buf_iova + pkt->data_off); + _mm_store_si128(RTE_CAST_PTR(__m128i *, txdp), descriptor); +} + +#ifdef __AVX2__ + +static __rte_always_inline void +ci_vtx_avx2 (volatile struct ci_tx_desc *txdp, +struct rte_mbuf **pkt,uint16_t nb_pkts,uint64_t flags,bool offload, +enum ci_l2tag_pos single_vlan_pos,enum ci_l2tag_pos qinq_outer_pos) +{ +const uint64_t hi_qw_tmpl = (CI_TX_DESC_DTYPE_DATA | (flags <<CI_TXD_QW1_CMD_S )); + +/* if unaligned on 32-bit boundary, do one to align */ +if (((uintptr_t )txdp &0x1F ) !=0 && nb_pkts !=0 ) { +ci_vtx1 (txdp, *pkt, flags, offload, single_vlan_pos, qinq_outer_pos); + nb_pkts--; txdp++; pkt++; + } + +/* do two at a time while possible, in bursts */ +for (; nb_pkts >3 ; txdp +=4 , pkt +=4 , nb_pkts -=4 ) { +uint64_t hi_qw3 = hi_qw_tmpl | + ((uint64_t )pkt[3 ]->data_len <<CI_TXD_QW1_TX_BUF_SZ_S ); +if (offload) +ci_tx_vec_offload (pkt[3 ], &hi_qw3, single_vlan_pos, qinq_outer_pos); +uint64_t hi_qw2 = hi_qw_tmpl | + ((uint64_t )pkt[2 ]->data_len <<CI_TXD_QW1_TX_BUF_SZ_S ); +if (offload) +ci_tx_vec_offload (pkt[2 ], &hi_qw2, single_vlan_pos, qinq_outer_pos); +uint64_t hi_qw1 = hi_qw_tmpl | + ((uint64_t )pkt[1 ]->data_len <<CI_TXD_QW1_TX_BUF_SZ_S ); +if (offload) +ci_tx_vec_offload (pkt[1 ], &hi_qw1, single_vlan_pos, qinq_outer_pos); +uint64_t hi_qw0 = hi_qw_tmpl | + ((uint64_t )pkt[0 ]->data_len <<CI_TXD_QW1_TX_BUF_SZ_S ); +if (offload) +ci_tx_vec_offload (pkt[0 ], &hi_qw0, single_vlan_pos, qinq_outer_pos); + +__m256i desc2_3 = +_mm256_set_epi64x + (hi_qw3, + pkt[3 ]->buf_iova + pkt[3 ]->data_off, + hi_qw2, + pkt[2 ]->buf_iova + pkt[2 ]->data_off); +__m256i desc0_1 = +_mm256_set_epi64x + (hi_qw1, + pkt[1 ]->buf_iova + pkt[1 ]->data_off, + hi_qw0, + pkt[0 ]->buf_iova + pkt[0 ]->data_off); + _mm256_store_si256(RTE_CAST_PTR(__m256i *, txdp + 2), desc2_3); + _mm256_store_si256(RTE_CAST_PTR(__m256i *, txdp), desc0_1); + } + +/* do any last ones */ +while (nb_pkts) { +ci_vtx1 (txdp, *pkt, flags, offload, single_vlan_pos, qinq_outer_pos); + txdp++; pkt++; nb_pkts--; + } +} + +static __rte_always_inline void +ci_vtx1_ctx_avx2 (volatile struct ci_tx_desc *txdp,struct rte_mbuf *pkt, +uint64_t flags,bool offload,enum ci_l2tag_pos single_vlan_pos, +enum ci_l2tag_pos qinq_outer_pos,bool ptype_lldp_enabled) +{ +uint64_t high_ctx_qw =CI_TX_DESC_DTYPE_CTX ; +uint64_t low_ctx_qw =0 ; + +if (offload) { +ci_fill_ctx_desc_tunneling (&low_ctx_qw, pkt); +if (pkt->ol_flags &RTE_MBUF_F_TX_QINQ ) { +uint64_t qinq_tag = qinq_outer_pos ==CI_TAG_IN_CTX_DESC ? + (uint64_t )pkt->vlan_tci_outer : + (uint64_t )pkt->vlan_tci; + high_ctx_qw |=CI_TX_CTX_DESC_IL2TAG2 <<CI_TXD_QW1_CMD_S ; + low_ctx_qw |= qinq_tag <<CI_TXD_CTX_QW0_L2TAG2_S ; + }else if ((pkt->ol_flags &RTE_MBUF_F_TX_VLAN ) && + single_vlan_pos ==CI_TAG_IN_CTX_DESC ) { + high_ctx_qw |=CI_TX_CTX_DESC_IL2TAG2 <<CI_TXD_QW1_CMD_S ; + low_ctx_qw |= (uint64_t )pkt->vlan_tci <<CI_TXD_CTX_QW0_L2TAG2_S ; + } + } +if (IAVF_CHECK_TX_LLDP (pkt, ptype_lldp_enabled)) + high_ctx_qw |=IAVF_TX_CTX_DESC_SWTCH_UPLINK <<CI_TXD_QW1_CMD_S ; +uint64_t high_data_qw = (CI_TX_DESC_DTYPE_DATA | + ((uint64_t )flags <<CI_TXD_QW1_CMD_S ) | + ((uint64_t )pkt->data_len <<CI_TXD_QW1_TX_BUF_SZ_S )); +if (offload) +ci_tx_vec_offload (pkt, &high_data_qw, single_vlan_pos, qinq_outer_pos); + +__m256i ctx_data_desc =_mm256_set_epi64x (high_data_qw, pkt->buf_iova + pkt->data_off, + high_ctx_qw, low_ctx_qw); + +/* tx_id is always even in ctx mode, so txdp is always 32-byte aligned */ + _mm256_store_si256(RTE_CAST_PTR(__m256i *, txdp), ctx_data_desc); +} + +static __rte_always_inline void +ci_vtx_ctx_avx2 (volatile struct ci_tx_desc *txdp, +struct rte_mbuf **pkt,uint16_t nb_pkts,uint64_t flags, +bool offload,enum ci_l2tag_pos single_vlan_pos,enum ci_l2tag_pos qinq_outer_pos, +bool ptype_lldp_enabled) +{ +uint64_t hi_data_qw_tmpl = (CI_TX_DESC_DTYPE_DATA | (flags <<CI_TXD_QW1_CMD_S )); + +for (; nb_pkts >1 ; txdp +=4 , pkt +=2 , nb_pkts -=2 ) { +uint64_t hi_ctx_qw1 =CI_TX_DESC_DTYPE_CTX ; +uint64_t hi_ctx_qw0 =CI_TX_DESC_DTYPE_CTX ; +uint64_t low_ctx_qw1 =0 ; +uint64_t low_ctx_qw0 =0 ; +uint64_t hi_data_qw1 =0 ; +uint64_t hi_data_qw0 =0 ; + + hi_data_qw1 = hi_data_qw_tmpl | + ((uint64_t )pkt[1 ]->data_len << +CI_TXD_QW1_TX_BUF_SZ_S ); + hi_data_qw0 = hi_data_qw_tmpl | + ((uint64_t )pkt[0 ]->data_len << +CI_TXD_QW1_TX_BUF_SZ_S ); + +if (offload) { +/* tunnel fill assigns low_ctx_qw1; must run before QinQ/VLAN OR below */ +ci_fill_ctx_desc_tunneling (&low_ctx_qw1, pkt[1 ]); +if (pkt[1 ]->ol_flags &RTE_MBUF_F_TX_QINQ ) { +uint64_t qinq_tag = qinq_outer_pos ==CI_TAG_IN_CTX_DESC ? + (uint64_t )pkt[1 ]->vlan_tci_outer : + (uint64_t )pkt[1 ]->vlan_tci; + hi_ctx_qw1 |=CI_TX_CTX_DESC_IL2TAG2 <<CI_TXD_QW1_CMD_S ; + low_ctx_qw1 |= qinq_tag <<CI_TXD_CTX_QW0_L2TAG2_S ; + }else if (pkt[1 ]->ol_flags &RTE_MBUF_F_TX_VLAN && + single_vlan_pos ==CI_TAG_IN_CTX_DESC ) { + hi_ctx_qw1 |=CI_TX_CTX_DESC_IL2TAG2 <<CI_TXD_QW1_CMD_S ; + low_ctx_qw1 |= (uint64_t )pkt[1 ]->vlan_tci <<CI_TXD_CTX_QW0_L2TAG2_S ; + } + } +if (IAVF_CHECK_TX_LLDP (pkt[1 ], ptype_lldp_enabled)) + hi_ctx_qw1 |=IAVF_TX_CTX_DESC_SWTCH_UPLINK <<CI_TXD_QW1_CMD_S ; + +if (offload) { +/* tunnel fill assigns low_ctx_qw0; must run before QinQ/VLAN OR below */ +ci_fill_ctx_desc_tunneling (&low_ctx_qw0, pkt[0 ]); +if (pkt[0 ]->ol_flags &RTE_MBUF_F_TX_QINQ ) { +uint64_t qinq_tag = qinq_outer_pos ==CI_TAG_IN_CTX_DESC ? + (uint64_t )pkt[0 ]->vlan_tci_outer : + (uint64_t )pkt[0 ]->vlan_tci; + hi_ctx_qw0 |=CI_TX_CTX_DESC_IL2TAG2 <<CI_TXD_QW1_CMD_S ; + low_ctx_qw0 |= qinq_tag <<CI_TXD_CTX_QW0_L2TAG2_S ; + }else if (pkt[0 ]->ol_flags &RTE_MBUF_F_TX_VLAN && + single_vlan_pos ==CI_TAG_IN_CTX_DESC ) { + hi_ctx_qw0 |=CI_TX_CTX_DESC_IL2TAG2 <<CI_TXD_QW1_CMD_S ; + low_ctx_qw0 |= (uint64_t )pkt[0 ]->vlan_tci <<CI_TXD_CTX_QW0_L2TAG2_S ; + } + } +if (IAVF_CHECK_TX_LLDP (pkt[0 ], ptype_lldp_enabled)) + hi_ctx_qw0 |=IAVF_TX_CTX_DESC_SWTCH_UPLINK <<CI_TXD_QW1_CMD_S ; + +if (offload) { +ci_tx_vec_offload (pkt[1 ], &hi_data_qw1, single_vlan_pos, qinq_outer_pos); +ci_tx_vec_offload (pkt[0 ], &hi_data_qw0, single_vlan_pos, qinq_outer_pos); + } + +__m256i desc2_3 =_mm256_set_epi64x + (hi_data_qw1, pkt[1 ]->buf_iova + pkt[1 ]->data_off, + hi_ctx_qw1, low_ctx_qw1); +__m256i desc0_1 =_mm256_set_epi64x + (hi_data_qw0, pkt[0 ]->buf_iova + pkt[0 ]->data_off, + hi_ctx_qw0, low_ctx_qw0); + _mm256_store_si256(RTE_CAST_PTR(__m256i *, txdp + 2), desc2_3); + _mm256_store_si256(RTE_CAST_PTR(__m256i *, txdp), desc0_1); + } + +if (nb_pkts) +ci_vtx1_ctx_avx2 (txdp, *pkt, flags, offload, + single_vlan_pos, qinq_outer_pos, ptype_lldp_enabled); +} + +#endif /* __AVX2__ */ + +#ifdef __AVX512VL__ + +static __rte_always_inline void +ci_vtx_avx512 (volatile struct ci_tx_desc *txdp, +struct rte_mbuf **pkt,uint16_t nb_pkts,uint64_t flags, +bool offload,enum ci_l2tag_pos single_vlan_pos,enum ci_l2tag_pos qinq_outer_pos) +{ +const uint64_t hi_qw_tmpl = (CI_TX_DESC_DTYPE_DATA | (flags <<CI_TXD_QW1_CMD_S )); + +/* if unaligned on 32-bit boundary, do one to align */ +if (((uintptr_t )txdp &0x1F ) !=0 && nb_pkts !=0 ) { +ci_vtx1 (txdp, *pkt, flags, offload, single_vlan_pos, qinq_outer_pos); + nb_pkts--; txdp++; pkt++; + } + +/* do 4 at a time while possible, in bursts */ +for (; nb_pkts >3 ; txdp +=4 , pkt +=4 , nb_pkts -=4 ) { +uint64_t hi_qw3 = hi_qw_tmpl | + ((uint64_t )pkt[3 ]->data_len <<CI_TXD_QW1_TX_BUF_SZ_S ); +uint64_t hi_qw2 = hi_qw_tmpl | + ((uint64_t )pkt[2 ]->data_len <<CI_TXD_QW1_TX_BUF_SZ_S ); +uint64_t hi_qw1 = hi_qw_tmpl | + ((uint64_t )pkt[1 ]->data_len <<CI_TXD_QW1_TX_BUF_SZ_S ); +uint64_t hi_qw0 = hi_qw_tmpl | + ((uint64_t )pkt[0 ]->data_len <<CI_TXD_QW1_TX_BUF_SZ_S ); +if (offload) { +ci_tx_vec_offload (pkt[3 ], &hi_qw3, single_vlan_pos, qinq_outer_pos); +ci_tx_vec_offload (pkt[2 ], &hi_qw2, single_vlan_pos, qinq_outer_pos); +ci_tx_vec_offload (pkt[1 ], &hi_qw1, single_vlan_pos, qinq_outer_pos); +ci_tx_vec_offload (pkt[0 ], &hi_qw0, single_vlan_pos, qinq_outer_pos); + } + +__m512i desc0_3 = +_mm512_set_epi64 + (hi_qw3, + pkt[3 ]->buf_iova + pkt[3 ]->data_off, + hi_qw2, + pkt[2 ]->buf_iova + pkt[2 ]->data_off, + hi_qw1, + pkt[1 ]->buf_iova + pkt[1 ]->data_off, + hi_qw0, + pkt[0 ]->buf_iova + pkt[0 ]->data_off); + _mm512_storeu_si512(RTE_CAST_PTR(void *, txdp), desc0_3); + } + +/* do any last ones */ +while (nb_pkts) { +ci_vtx1 (txdp, *pkt, flags, offload, single_vlan_pos, qinq_outer_pos); + txdp++; pkt++; nb_pkts--; + } +} + +static __rte_always_inline void +ci_vtx1_ctx_avx512 (volatile struct ci_tx_desc *txdp,struct rte_mbuf *pkt, +uint64_t flags,bool offload,enum ci_l2tag_pos single_vlan_pos, +enum ci_l2tag_pos qinq_outer_pos,bool lldp_enabled) +{ +uint64_t high_ctx_qw =CI_TX_DESC_DTYPE_CTX ; +uint64_t low_ctx_qw =0 ; + +if (offload) { +ci_fill_ctx_desc_tunneling (&low_ctx_qw, pkt); +if (pkt->ol_flags &RTE_MBUF_F_TX_QINQ ) { +uint64_t qinq_tag = qinq_outer_pos ==CI_TAG_IN_CTX_DESC ? + (uint64_t )pkt->vlan_tci_outer : + (uint64_t )pkt->vlan_tci; + high_ctx_qw |=CI_TX_CTX_DESC_IL2TAG2 <<CI_TXD_QW1_CMD_S ; + low_ctx_qw |= qinq_tag <<CI_TXD_CTX_QW0_L2TAG2_S ; + }else if ((pkt->ol_flags &RTE_MBUF_F_TX_VLAN ) && + single_vlan_pos ==CI_TAG_IN_CTX_DESC ) { + high_ctx_qw |=CI_TX_CTX_DESC_IL2TAG2 <<CI_TXD_QW1_CMD_S ; + low_ctx_qw |= (uint64_t )pkt->vlan_tci <<CI_TXD_CTX_QW0_L2TAG2_S ; + } + } +if (IAVF_CHECK_TX_LLDP (pkt, lldp_enabled)) + high_ctx_qw |=IAVF_TX_CTX_DESC_SWTCH_UPLINK <<CI_TXD_QW1_CMD_S ; +uint64_t high_data_qw = (CI_TX_DESC_DTYPE_DATA | + ((uint64_t )flags <<CI_TXD_QW1_CMD_S ) | + ((uint64_t )pkt->data_len <<CI_TXD_QW1_TX_BUF_SZ_S )); +if (offload) +ci_tx_vec_offload (pkt, &high_data_qw, single_vlan_pos, qinq_outer_pos); + +__m256i ctx_data_desc =_mm256_set_epi64x + (high_data_qw, pkt->buf_iova + pkt->data_off, + high_ctx_qw, low_ctx_qw); + +/* tx_id is always even in ctx mode, so txdp is always 32-byte aligned */ + _mm256_store_si256(RTE_CAST_PTR(__m256i *, txdp), ctx_data_desc); +} + +static __rte_always_inline void +ci_vtx_ctx_avx512 (volatile struct ci_tx_desc *txdp, +struct rte_mbuf **pkt,uint16_t nb_pkts,uint64_t flags, +bool offload,enum ci_l2tag_pos single_vlan_pos,enum ci_l2tag_pos qinq_outer_pos, +bool lldp_enabled) +{ +uint64_t hi_data_qw_tmpl = (CI_TX_DESC_DTYPE_DATA | (flags <<CI_TXD_QW1_CMD_S )); + +for (; nb_pkts >1 ; txdp +=4 , pkt +=2 , nb_pkts -=2 ) { +uint64_t hi_ctx_qw1 =CI_TX_DESC_DTYPE_CTX ; +uint64_t hi_ctx_qw0 =CI_TX_DESC_DTYPE_CTX ; +uint64_t low_ctx_qw1 =0 ; +uint64_t low_ctx_qw0 =0 ; +uint64_t hi_data_qw1 =0 ; +uint64_t hi_data_qw0 =0 ; + + hi_data_qw1 = hi_data_qw_tmpl | + ((uint64_t )pkt[1 ]->data_len <<CI_TXD_QW1_TX_BUF_SZ_S ); + hi_data_qw0 = hi_data_qw_tmpl | + ((uint64_t )pkt[0 ]->data_len <<CI_TXD_QW1_TX_BUF_SZ_S ); + +if (offload) { +/* tunnel fill assigns low_ctx_qw1; must run before QinQ/VLAN OR below */ +ci_fill_ctx_desc_tunneling (&low_ctx_qw1, pkt[1 ]); +if (pkt[1 ]->ol_flags &RTE_MBUF_F_TX_QINQ ) { +uint64_t qinq_tag = qinq_outer_pos ==CI_TAG_IN_CTX_DESC ? + (uint64_t )pkt[1 ]->vlan_tci_outer : + (uint64_t )pkt[1 ]->vlan_tci; + hi_ctx_qw1 |=CI_TX_CTX_DESC_IL2TAG2 <<CI_TXD_QW1_CMD_S ; + low_ctx_qw1 |= qinq_tag <<CI_TXD_CTX_QW0_L2TAG2_S ; + }else if (pkt[1 ]->ol_flags &RTE_MBUF_F_TX_VLAN && + single_vlan_pos ==CI_TAG_IN_CTX_DESC ) { + hi_ctx_qw1 |=CI_TX_CTX_DESC_IL2TAG2 <<CI_TXD_QW1_CMD_S ; + low_ctx_qw1 |= (uint64_t )pkt[1 ]->vlan_tci <<CI_TXD_CTX_QW0_L2TAG2_S ; + } + } +if (IAVF_CHECK_TX_LLDP (pkt[1 ], lldp_enabled)) + hi_ctx_qw1 |=IAVF_TX_CTX_DESC_SWTCH_UPLINK <<CI_TXD_QW1_CMD_S ; + +if (offload) { +/* tunnel fill assigns low_ctx_qw0; must run before QinQ/VLAN OR below */ +ci_fill_ctx_desc_tunneling (&low_ctx_qw0, pkt[0 ]); +if (pkt[0 ]->ol_flags &RTE_MBUF_F_TX_QINQ ) { +uint64_t qinq_tag = qinq_outer_pos ==CI_TAG_IN_CTX_DESC ? + (uint64_t )pkt[0 ]->vlan_tci_outer : + (uint64_t )pkt[0 ]->vlan_tci; + hi_ctx_qw0 |=CI_TX_CTX_DESC_IL2TAG2 <<CI_TXD_QW1_CMD_S ; + low_ctx_qw0 |= qinq_tag <<CI_TXD_CTX_QW0_L2TAG2_S ; + }else if (pkt[0 ]->ol_flags &RTE_MBUF_F_TX_VLAN && + single_vlan_pos ==CI_TAG_IN_CTX_DESC ) { + hi_ctx_qw0 |=CI_TX_CTX_DESC_IL2TAG2 <<CI_TXD_QW1_CMD_S ; + low_ctx_qw0 |= (uint64_t )pkt[0 ]->vlan_tci <<CI_TXD_CTX_QW0_L2TAG2_S ; + } + } +if (IAVF_CHECK_TX_LLDP (pkt[0 ], lldp_enabled)) + hi_ctx_qw0 |=IAVF_TX_CTX_DESC_SWTCH_UPLINK <<CI_TXD_QW1_CMD_S ; + +if (offload) { +ci_tx_vec_offload (pkt[1 ], &hi_data_qw1, single_vlan_pos, qinq_outer_pos); +ci_tx_vec_offload (pkt[0 ], &hi_data_qw0, single_vlan_pos, qinq_outer_pos); + } + +__m512i desc0_3 =_mm512_set_epi64 + (hi_data_qw1, pkt[1 ]->buf_iova + pkt[1 ]->data_off, + hi_ctx_qw1, low_ctx_qw1, + hi_data_qw0, pkt[0 ]->buf_iova + pkt[0 ]->data_off, + hi_ctx_qw0, low_ctx_qw0); + _mm512_storeu_si512(RTE_CAST_PTR(void *, txdp), desc0_3); + } + +if (nb_pkts) +ci_vtx1_ctx_avx512 (txdp, *pkt, flags, offload, + single_vlan_pos, qinq_outer_pos, lldp_enabled); +} + +#endif /* __AVX512VL__ */ + +#endif /* _COMMON_INTEL_TX_VEC_X86_H_ */
drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c +7 -278
326 lines not on this page · raw message
drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c +8 -264
309 lines not on this page · raw message
drivers/net/intel/iavf/iavf_rxtx_vec_common.h +4 -71
85 lines not on this page · raw message
Signature · 1 line
2.53.0
PATCH v3 7/15 net/intel: use function callback for lldp
In reply to Bruce RichardsonLimit LLDP support for iavf drivers for now, by using a callback to handle its processing. Drivers not supporting this pass a NULL pointer and the compiler will optimize away the branches. For iavf, the compiler can inline the subfunction directly as it's known at compile-time.
Signed-off-by Bruce Richardson <[email protected]>
Acked-by Ciara Loftus <[email protected]>
--- drivers/net/intel/common/tx_vec_x86.h | 42 +++++++++++-------- drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c | 16 +++++-- drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c | 17 +++++--- 3 files changed, 48 insertions(+), 27 deletions(-)
drivers/net/intel/common/tx_vec_x86.h +24 -18
@@ -11,6 +11,12 @@#include "tx.h" +/* Optional per-driver LLDP switch-uplink check for ctx descriptors. +* Takes the ctx-desc high qword and returns it with the LLDP bit applied +* if appropriate. NULL disables the check. +*/ +typedef uint64_t (*ci_tx_ctx_lldp_fn )(struct rte_mbuf *pkt,uint64_t high_ctx_qw); + static __rte_always_inline voidci_fill_ctx_desc_tunneling (uint64_t *low_ctx_qw,struct rte_mbuf *pkt) {
@@ -226,7 +232,7 @@ ci_vtx_avx2(volatile struct ci_tx_desc *txdp, static __rte_always_inline voidci_vtx1_ctx_avx2 (volatile struct ci_tx_desc *txdp,struct rte_mbuf *pkt,uint64_t flags,bool offload,enum ci_l2tag_pos single_vlan_pos, -enum ci_l2tag_pos qinq_outer_pos,bool ptype_lldp_enabled) +enum ci_l2tag_pos qinq_outer_pos,ci_tx_ctx_lldp_fn lldp_check) {uint64_t high_ctx_qw =CI_TX_DESC_DTYPE_CTX ;uint64_t low_ctx_qw =0 ;
@@ -245,8 +251,8 @@ ci_vtx1_ctx_avx2(volatile struct ci_tx_desc *txdp, struct rte_mbuf *pkt, low_ctx_qw |= (uint64_t )pkt->vlan_tci <<CI_TXD_CTX_QW0_L2TAG2_S ; } } -if (IAVF_CHECK_TX_LLDP (pkt, ptype_lldp_enabled)) - high_ctx_qw |=IAVF_TX_CTX_DESC_SWTCH_UPLINK <<CI_TXD_QW1_CMD_S ; +if (lldp_check !=NULL ) + high_ctx_qw =lldp_check (pkt, high_ctx_qw);uint64_t high_data_qw = (CI_TX_DESC_DTYPE_DATA | ((uint64_t )flags <<CI_TXD_QW1_CMD_S ) | ((uint64_t )pkt->data_len <<CI_TXD_QW1_TX_BUF_SZ_S ));
@@ -264,7 +270,7 @@ static __rte_always_inline void ci_vtx_ctx_avx2(volatile struct ci_tx_desc *txdp, struct rte_mbuf **pkt, uint16_t nb_pkts, uint64_t flags, bool offload, enum ci_l2tag_pos single_vlan_pos, enum ci_l2tag_pos qinq_outer_pos, - bool ptype_lldp_enabled) + ci_tx_ctx_lldp_fn lldp_check) { uint64_t hi_data_qw_tmpl = (CI_TX_DESC_DTYPE_DATA | (flags << CI_TXD_QW1_CMD_S));
@@ -298,8 +304,8 @@ ci_vtx_ctx_avx2(volatile struct ci_tx_desc *txdp, low_ctx_qw1 |= (uint64_t )pkt[1 ]->vlan_tci <<CI_TXD_CTX_QW0_L2TAG2_S ; } } -if (IAVF_CHECK_TX_LLDP (pkt[1 ], ptype_lldp_enabled)) - hi_ctx_qw1 |=IAVF_TX_CTX_DESC_SWTCH_UPLINK <<CI_TXD_QW1_CMD_S ; +if (lldp_check !=NULL ) + hi_ctx_qw1 =lldp_check (pkt[1 ], hi_ctx_qw1);if (offload) {/* tunnel fill assigns low_ctx_qw0; must run before QinQ/VLAN OR below */
@@ -316,8 +322,8 @@ ci_vtx_ctx_avx2(volatile struct ci_tx_desc *txdp, low_ctx_qw0 |= (uint64_t )pkt[0 ]->vlan_tci <<CI_TXD_CTX_QW0_L2TAG2_S ; } } -if (IAVF_CHECK_TX_LLDP (pkt[0 ], ptype_lldp_enabled)) - hi_ctx_qw0 |=IAVF_TX_CTX_DESC_SWTCH_UPLINK <<CI_TXD_QW1_CMD_S ; +if (lldp_check !=NULL ) + hi_ctx_qw0 =lldp_check (pkt[0 ], hi_ctx_qw0);if (offload) {ci_tx_vec_offload (pkt[1 ], &hi_data_qw1, single_vlan_pos, qinq_outer_pos);
@@ -336,7 +342,7 @@ ci_vtx_ctx_avx2(volatile struct ci_tx_desc *txdp,if (nb_pkts)ci_vtx1_ctx_avx2 (txdp, *pkt, flags, offload, - single_vlan_pos, qinq_outer_pos, ptype_lldp_enabled); + single_vlan_pos, qinq_outer_pos, lldp_check); }#endif /* __AVX2__ */
@@ -396,7 +402,7 @@ ci_vtx_avx512(volatile struct ci_tx_desc *txdp, static __rte_always_inline voidci_vtx1_ctx_avx512 (volatile struct ci_tx_desc *txdp,struct rte_mbuf *pkt,uint64_t flags,bool offload,enum ci_l2tag_pos single_vlan_pos, -enum ci_l2tag_pos qinq_outer_pos,bool lldp_enabled) +enum ci_l2tag_pos qinq_outer_pos,ci_tx_ctx_lldp_fn lldp_check) {uint64_t high_ctx_qw =CI_TX_DESC_DTYPE_CTX ;uint64_t low_ctx_qw =0 ;
@@ -415,8 +421,8 @@ ci_vtx1_ctx_avx512(volatile struct ci_tx_desc *txdp, struct rte_mbuf *pkt, low_ctx_qw |= (uint64_t )pkt->vlan_tci <<CI_TXD_CTX_QW0_L2TAG2_S ; } } -if (IAVF_CHECK_TX_LLDP (pkt, lldp_enabled)) - high_ctx_qw |=IAVF_TX_CTX_DESC_SWTCH_UPLINK <<CI_TXD_QW1_CMD_S ; +if (lldp_check !=NULL ) + high_ctx_qw =lldp_check (pkt, high_ctx_qw);uint64_t high_data_qw = (CI_TX_DESC_DTYPE_DATA | ((uint64_t )flags <<CI_TXD_QW1_CMD_S ) | ((uint64_t )pkt->data_len <<CI_TXD_QW1_TX_BUF_SZ_S ));
@@ -435,7 +441,7 @@ static __rte_always_inline void ci_vtx_ctx_avx512(volatile struct ci_tx_desc *txdp, struct rte_mbuf **pkt, uint16_t nb_pkts, uint64_t flags, bool offload, enum ci_l2tag_pos single_vlan_pos, enum ci_l2tag_pos qinq_outer_pos, - bool lldp_enabled) + ci_tx_ctx_lldp_fn lldp_check) { uint64_t hi_data_qw_tmpl = (CI_TX_DESC_DTYPE_DATA | (flags << CI_TXD_QW1_CMD_S));
@@ -467,8 +473,8 @@ ci_vtx_ctx_avx512(volatile struct ci_tx_desc *txdp, low_ctx_qw1 |= (uint64_t )pkt[1 ]->vlan_tci <<CI_TXD_CTX_QW0_L2TAG2_S ; } } -if (IAVF_CHECK_TX_LLDP (pkt[1 ], lldp_enabled)) - hi_ctx_qw1 |=IAVF_TX_CTX_DESC_SWTCH_UPLINK <<CI_TXD_QW1_CMD_S ; +if (lldp_check !=NULL ) + hi_ctx_qw1 =lldp_check (pkt[1 ], hi_ctx_qw1);if (offload) {/* tunnel fill assigns low_ctx_qw0; must run before QinQ/VLAN OR below */
@@ -485,8 +491,8 @@ ci_vtx_ctx_avx512(volatile struct ci_tx_desc *txdp, low_ctx_qw0 |= (uint64_t )pkt[0 ]->vlan_tci <<CI_TXD_CTX_QW0_L2TAG2_S ; } } -if (IAVF_CHECK_TX_LLDP (pkt[0 ], lldp_enabled)) - hi_ctx_qw0 |=IAVF_TX_CTX_DESC_SWTCH_UPLINK <<CI_TXD_QW1_CMD_S ; +if (lldp_check !=NULL ) + hi_ctx_qw0 =lldp_check (pkt[0 ], hi_ctx_qw0);if (offload) {ci_tx_vec_offload (pkt[1 ], &hi_data_qw1, single_vlan_pos, qinq_outer_pos);
@@ -503,7 +509,7 @@ ci_vtx_ctx_avx512(volatile struct ci_tx_desc *txdp,if (nb_pkts)ci_vtx1_ctx_avx512 (txdp, *pkt, flags, offload, - single_vlan_pos, qinq_outer_pos, lldp_enabled); + single_vlan_pos, qinq_outer_pos, lldp_check); }#endif /* __AVX512VL__ */
drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c +12 -4
@@ -1685,6 +1685,14 @@ iavf_xmit_fixed_burst_vec_avx2(void *tx_queue, struct rte_mbuf **tx_pkts,return nb_pkts; } +static __rte_always_inline uint64_t +iavf_tx_ctx_lldp_check (struct rte_mbuf *pkt,uint64_t high_ctx_qw) +{ +if (IAVF_CHECK_TX_LLDP (pkt, true)) + high_ctx_qw |=IAVF_TX_CTX_DESC_SWTCH_UPLINK <<CI_TXD_QW1_CMD_S ; +return high_ctx_qw; +} + static __rte_always_inline uint16_tiavf_xmit_fixed_burst_vec_avx2_ctx (void *tx_queue,struct rte_mbuf **tx_pkts,uint16_t nb_pkts,bool offload)
@@ -1696,7 +1704,7 @@ iavf_xmit_fixed_burst_vec_avx2_ctx(void *tx_queue, struct rte_mbuf **tx_pkts,/* bit2 is reserved and must be set to 1 according to Spec */ uint64_t flags =IAVF_TX_DESC_CMD_EOP |IAVF_TX_DESC_CMD_ICRC ;uint64_t rs =IAVF_TX_DESC_CMD_RS | flags; -bool lldp_enabled = txq->lldp_enabled; +ci_tx_ctx_lldp_fn lldp_check = txq->lldp_enabled ? iavf_tx_ctx_lldp_check :NULL ;/* vlan_flag gives both the single-VLAN and the QinQ outer tag position */ enum ci_l2tag_pos vlan_pos = (txq->vlan_flag &IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG1 ) ?CI_TAG_IN_DATA_DESC :CI_TAG_IN_CTX_DESC ;
@@ -1723,10 +1731,10 @@ iavf_xmit_fixed_burst_vec_avx2_ctx(void *tx_queue, struct rte_mbuf **tx_pkts,ci_tx_backlog_entry_vec (txep, tx_pkts, nb_mbuf);ci_vtx_ctx_avx2 (txdp, tx_pkts, nb_mbuf -1 , flags, offload, - vlan_pos, vlan_pos, lldp_enabled); + vlan_pos, vlan_pos, lldp_check); tx_pkts += (nb_mbuf -1 ); txdp += (n -2 ); -ci_vtx1_ctx_avx2 (txdp, *tx_pkts++, rs, offload, vlan_pos, vlan_pos, lldp_enabled); +ci_vtx1_ctx_avx2 (txdp, *tx_pkts++, rs, offload, vlan_pos, vlan_pos, lldp_check); nb_commit = (uint16_t )(nb_commit - n);
@@ -1740,7 +1748,7 @@ iavf_xmit_fixed_burst_vec_avx2_ctx(void *tx_queue, struct rte_mbuf **tx_pkts, nb_mbuf = nb_commit >>1 ;ci_tx_backlog_entry_vec (txep, tx_pkts, nb_mbuf); -ci_vtx_ctx_avx2 (txdp, tx_pkts, nb_mbuf, flags, offload, vlan_pos, vlan_pos, lldp_enabled); +ci_vtx_ctx_avx2 (txdp, tx_pkts, nb_mbuf, flags, offload, vlan_pos, vlan_pos, lldp_check); tx_id = (uint16_t )(tx_id + nb_commit);if (tx_id > txq->tx_next_rs) {
drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c +12 -5
@@ -1900,6 +1900,14 @@ iavf_xmit_fixed_burst_vec_avx512(void *tx_queue, struct rte_mbuf **tx_pkts,return nb_pkts; } +static __rte_always_inline uint64_t +iavf_tx_ctx_lldp_check (struct rte_mbuf *pkt,uint64_t high_ctx_qw) +{ +if (IAVF_CHECK_TX_LLDP (pkt, true)) + high_ctx_qw |=IAVF_TX_CTX_DESC_SWTCH_UPLINK <<CI_TXD_QW1_CMD_S ; +return high_ctx_qw; +} + static __rte_always_inline uint16_tiavf_xmit_fixed_burst_vec_avx512_ctx (void *tx_queue,struct rte_mbuf **tx_pkts,uint16_t nb_pkts,bool offload)
@@ -1911,7 +1919,7 @@ iavf_xmit_fixed_burst_vec_avx512_ctx(void *tx_queue, struct rte_mbuf **tx_pkts,/* bit2 is reserved and must be set to 1 according to Spec */ uint64_t flags =CI_TX_DESC_CMD_EOP |CI_TX_DESC_CMD_ICRC ;uint64_t rs =CI_TX_DESC_CMD_RS | flags; -bool lldp_enabled = txq->lldp_enabled; +ci_tx_ctx_lldp_fn lldp_check = txq->lldp_enabled ? iavf_tx_ctx_lldp_check :NULL ;/* vlan_flag gives both the single-VLAN and the QinQ outer tag position */ enum ci_l2tag_pos vlan_pos = (txq->vlan_flag &IAVF_TX_FLAGS_VLAN_TAG_LOC_L2TAG1 ) ?CI_TAG_IN_DATA_DESC :CI_TAG_IN_CTX_DESC ;
@@ -1938,11 +1946,10 @@ iavf_xmit_fixed_burst_vec_avx512_ctx(void *tx_queue, struct rte_mbuf **tx_pkts,tx_backlog_entry_avx512 (txep, tx_pkts, nb_mbuf);ci_vtx_ctx_avx512 (txdp, tx_pkts, nb_mbuf -1 , flags, offload, - vlan_pos, vlan_pos, lldp_enabled); + vlan_pos, vlan_pos, lldp_check); tx_pkts += (nb_mbuf -1 ); txdp += (n -2 ); -ci_vtx1_ctx_avx512 (txdp, *tx_pkts++, rs, offload, - vlan_pos, vlan_pos, lldp_enabled); +ci_vtx1_ctx_avx512 (txdp, *tx_pkts++, rs, offload, vlan_pos, vlan_pos, lldp_check); nb_commit = (uint16_t )(nb_commit - n);
@@ -1956,7 +1963,7 @@ iavf_xmit_fixed_burst_vec_avx512_ctx(void *tx_queue, struct rte_mbuf **tx_pkts, nb_mbuf = nb_commit >>1 ;tx_backlog_entry_avx512 (txep, tx_pkts, nb_mbuf); -ci_vtx_ctx_avx512 (txdp, tx_pkts, nb_mbuf, flags, offload, vlan_pos, vlan_pos, lldp_enabled); +ci_vtx_ctx_avx512 (txdp, tx_pkts, nb_mbuf, flags, offload, vlan_pos, vlan_pos, lldp_check); tx_id = (uint16_t )(tx_id + nb_commit);if (tx_id > txq->tx_next_rs) {
Signature · 1 line
2.53.0
PATCH v3 8/15 net/intel: allow building without IOVA in mbuf
In reply to Bruce RichardsonEnsure correct macros are used to build the vector Tx code without the IOVA stored explicitly in the mbuf. This allows the iavf driver to build when the config setting for IOVA in mbuf is disabled.
Signed-off-by Bruce Richardson <[email protected]>
Acked-by Ciara Loftus <[email protected]>
--- drivers/net/intel/common/tx_vec_x86.h | 30 +++++++++++++-------------- drivers/net/intel/iavf/meson.build | 4 +--- 2 files changed, 16 insertions(+), 18 deletions(-)
drivers/net/intel/common/tx_vec_x86.h +15 -15
@@ -168,7 +168,7 @@ ci_vtx1(volatile struct ci_tx_desc *txdp,if (offload)ci_tx_vec_offload (pkt, &high_qw, single_vlan_pos, qinq_outer_pos); -__m128i descriptor =_mm_set_epi64x (high_qw, pkt->buf_iova + pkt->data_off); +__m128i descriptor =_mm_set_epi64x (high_qw,rte_pktmbuf_iova (pkt)); _mm_store_si128(RTE_CAST_PTR(__m128i *, txdp), descriptor); }
@@ -209,15 +209,15 @@ ci_vtx_avx2(volatile struct ci_tx_desc *txdp,__m256i desc2_3 =_mm256_set_epi64x (hi_qw3, - pkt[3 ]->buf_iova + pkt[3 ]->data_off, +rte_pktmbuf_iova (pkt[3 ]), hi_qw2, - pkt[2 ]->buf_iova + pkt[2 ]->data_off); +rte_pktmbuf_iova (pkt[2 ]));__m256i desc0_1 =_mm256_set_epi64x (hi_qw1, - pkt[1 ]->buf_iova + pkt[1 ]->data_off, +rte_pktmbuf_iova (pkt[1 ]), hi_qw0, - pkt[0 ]->buf_iova + pkt[0 ]->data_off); +rte_pktmbuf_iova (pkt[0 ])); _mm256_store_si256(RTE_CAST_PTR(__m256i *, txdp + 2), desc2_3); _mm256_store_si256(RTE_CAST_PTR(__m256i *, txdp), desc0_1); }
@@ -259,7 +259,7 @@ ci_vtx1_ctx_avx2(volatile struct ci_tx_desc *txdp, struct rte_mbuf *pkt,if (offload)ci_tx_vec_offload (pkt, &high_data_qw, single_vlan_pos, qinq_outer_pos); -__m256i ctx_data_desc =_mm256_set_epi64x (high_data_qw, pkt->buf_iova + pkt->data_off, +__m256i ctx_data_desc =_mm256_set_epi64x (high_data_qw,rte_pktmbuf_iova (pkt), high_ctx_qw, low_ctx_qw);/* tx_id is always even in ctx mode, so txdp is always 32-byte aligned */
@@ -331,10 +331,10 @@ ci_vtx_ctx_avx2(volatile struct ci_tx_desc *txdp, }__m256i desc2_3 =_mm256_set_epi64x - (hi_data_qw1, pkt[1 ]->buf_iova + pkt[1 ]->data_off, + (hi_data_qw1,rte_pktmbuf_iova (pkt[1 ]), hi_ctx_qw1, low_ctx_qw1);__m256i desc0_1 =_mm256_set_epi64x - (hi_data_qw0, pkt[0 ]->buf_iova + pkt[0 ]->data_off, + (hi_data_qw0,rte_pktmbuf_iova (pkt[0 ]), hi_ctx_qw0, low_ctx_qw0); _mm256_store_si256(RTE_CAST_PTR(__m256i *, txdp + 2), desc2_3); _mm256_store_si256(RTE_CAST_PTR(__m256i *, txdp), desc0_1);
@@ -382,13 +382,13 @@ ci_vtx_avx512(volatile struct ci_tx_desc *txdp,__m512i desc0_3 =_mm512_set_epi64 (hi_qw3, - pkt[3 ]->buf_iova + pkt[3 ]->data_off, +rte_pktmbuf_iova (pkt[3 ]), hi_qw2, - pkt[2 ]->buf_iova + pkt[2 ]->data_off, +rte_pktmbuf_iova (pkt[2 ]), hi_qw1, - pkt[1 ]->buf_iova + pkt[1 ]->data_off, +rte_pktmbuf_iova (pkt[1 ]), hi_qw0, - pkt[0 ]->buf_iova + pkt[0 ]->data_off); +rte_pktmbuf_iova (pkt[0 ])); _mm512_storeu_si512(RTE_CAST_PTR(void *, txdp), desc0_3); }
@@ -430,7 +430,7 @@ ci_vtx1_ctx_avx512(volatile struct ci_tx_desc *txdp, struct rte_mbuf *pkt,ci_tx_vec_offload (pkt, &high_data_qw, single_vlan_pos, qinq_outer_pos);__m256i ctx_data_desc =_mm256_set_epi64x - (high_data_qw, pkt->buf_iova + pkt->data_off, + (high_data_qw,rte_pktmbuf_iova (pkt), high_ctx_qw, low_ctx_qw);/* tx_id is always even in ctx mode, so txdp is always 32-byte aligned */
@@ -500,9 +500,9 @@ ci_vtx_ctx_avx512(volatile struct ci_tx_desc *txdp, }__m512i desc0_3 =_mm512_set_epi64 - (hi_data_qw1, pkt[1 ]->buf_iova + pkt[1 ]->data_off, + (hi_data_qw1,rte_pktmbuf_iova (pkt[1 ]), hi_ctx_qw1, low_ctx_qw1, - hi_data_qw0, pkt[0 ]->buf_iova + pkt[0 ]->data_off, + hi_data_qw0,rte_pktmbuf_iova (pkt[0 ]), hi_ctx_qw0, low_ctx_qw0); _mm512_storeu_si512(RTE_CAST_PTR(void *, txdp), desc0_3); }
drivers/net/intel/iavf/meson.build +1 -3
@@ -1,9 +1,7 @@ # SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2018 Luca Boccassi <[email protected]> -if dpdk_conf.get('RTE_IOVA_IN_MBUF') == 0 - subdir_done() -endif +require_iova_in_mbuf = false testpmd_sources = files('iavf_testpmd.c')
Signature · 1 line
2.53.0
PATCH v3 9/15 net/ice: use common descriptor creation functions
In reply to Bruce RichardsonUpdate ice driver to use the descriptor creation functions from common rather than having it maintain its own copies of them.
Signed-off-by Bruce Richardson <[email protected]>
Acked-by Ciara Loftus <[email protected]>
--- drivers/net/intel/ice/ice_rxtx_vec_avx2.c | 73 ++------------------- drivers/net/intel/ice/ice_rxtx_vec_avx512.c | 64 ++---------------- drivers/net/intel/ice/ice_rxtx_vec_common.h | 59 ----------------- 3 files changed, 12 insertions(+), 184 deletions(-)
drivers/net/intel/ice/ice_rxtx_vec_avx2.c +6 -67
@@ -5,6 +5,7 @@#include "ice_rxtx_vec_common.h" #include "../common/rx_vec_x86.h" +#include "../common/tx_vec_x86.h" #include <rte_vect.h>
@@ -773,70 +774,6 @@ ice_recv_scattered_pkts_vec_avx2_offload(void *rx_queue, true); } -static __rte_always_inline void -ice_vtx1 (volatile struct ci_tx_desc *txdp, -struct rte_mbuf *pkt,uint64_t flags,bool offload) -{ -uint64_t high_qw = (CI_TX_DESC_DTYPE_DATA | - ((uint64_t )flags <<CI_TXD_QW1_CMD_S ) | - ((uint64_t )pkt->data_len <<CI_TXD_QW1_TX_BUF_SZ_S )); -if (offload) -ice_txd_enable_offload (pkt, &high_qw); - -__m128i descriptor =_mm_set_epi64x (high_qw,rte_pktmbuf_iova (pkt)); - _mm_store_si128(RTE_CAST_PTR(__m128i *, txdp), descriptor); -} - -static __rte_always_inline void -ice_vtx (volatile struct ci_tx_desc *txdp, -struct rte_mbuf **pkt,uint16_t nb_pkts,uint64_t flags,bool offload) -{ -const uint64_t hi_qw_tmpl = (CI_TX_DESC_DTYPE_DATA | (flags <<CI_TXD_QW1_CMD_S )); - -/* if unaligned on 32-bit boundary, do one to align */ -if (((uintptr_t )txdp &0x1F ) !=0 && nb_pkts !=0 ) { -ice_vtx1 (txdp, *pkt, flags, offload); - nb_pkts--; txdp++; pkt++; - } - -/* do four at a time while possible, in bursts */ -for (; nb_pkts >3 ; txdp +=4 , pkt +=4 , nb_pkts -=4 ) { -uint64_t hi_qw3 = hi_qw_tmpl | - ((uint64_t )pkt[3 ]->data_len <<CI_TXD_QW1_TX_BUF_SZ_S ); -if (offload) -ice_txd_enable_offload (pkt[3 ], &hi_qw3); -uint64_t hi_qw2 = hi_qw_tmpl | - ((uint64_t )pkt[2 ]->data_len <<CI_TXD_QW1_TX_BUF_SZ_S ); -if (offload) -ice_txd_enable_offload (pkt[2 ], &hi_qw2); -uint64_t hi_qw1 = hi_qw_tmpl | - ((uint64_t )pkt[1 ]->data_len <<CI_TXD_QW1_TX_BUF_SZ_S ); -if (offload) -ice_txd_enable_offload (pkt[1 ], &hi_qw1); -uint64_t hi_qw0 = hi_qw_tmpl | - ((uint64_t )pkt[0 ]->data_len <<CI_TXD_QW1_TX_BUF_SZ_S ); -if (offload) -ice_txd_enable_offload (pkt[0 ], &hi_qw0); - -__m256i desc2_3 = -_mm256_set_epi64x - (hi_qw3,rte_pktmbuf_iova (pkt[3 ]), - hi_qw2,rte_pktmbuf_iova (pkt[2 ])); -__m256i desc0_1 = -_mm256_set_epi64x - (hi_qw1,rte_pktmbuf_iova (pkt[1 ]), - hi_qw0,rte_pktmbuf_iova (pkt[0 ])); - _mm256_store_si256(RTE_CAST_PTR(__m256i *, txdp + 2), desc2_3); - _mm256_store_si256(RTE_CAST_PTR(__m256i *, txdp), desc0_1); - } - -/* do any last ones */ -while (nb_pkts) { -ice_vtx1 (txdp, *pkt, flags, offload); - txdp++; pkt++; nb_pkts--; - } -} - static __rte_always_inline uint16_tice_xmit_fixed_burst_vec_avx2 (void *tx_queue,struct rte_mbuf **tx_pkts,uint16_t nb_pkts,bool offload)
@@ -868,11 +805,12 @@ ice_xmit_fixed_burst_vec_avx2(void *tx_queue, struct rte_mbuf **tx_pkts,if (nb_commit >= n) {ci_tx_backlog_entry_vec (txep, tx_pkts, n); -ice_vtx (txdp, tx_pkts, n -1 , flags, offload); +ci_vtx_avx2 (txdp, tx_pkts, n -1 , flags, offload, +CI_TAG_IN_DATA_DESC ,CI_TAG_IN_DATA_DESC ); tx_pkts += (n -1 ); txdp += (n -1 ); -ice_vtx1 (txdp, *tx_pkts++, rs, offload); +ci_vtx1 (txdp, *tx_pkts++, rs, offload,CI_TAG_IN_DATA_DESC ,CI_TAG_IN_DATA_DESC ); nb_commit = (uint16_t )(nb_commit - n);
@@ -886,7 +824,8 @@ ice_xmit_fixed_burst_vec_avx2(void *tx_queue, struct rte_mbuf **tx_pkts,ci_tx_backlog_entry_vec (txep, tx_pkts, nb_commit); -ice_vtx (txdp, tx_pkts, nb_commit, flags, offload); +ci_vtx_avx2 (txdp, tx_pkts, nb_commit, flags, offload, +CI_TAG_IN_DATA_DESC ,CI_TAG_IN_DATA_DESC ); tx_id = (uint16_t )(tx_id + nb_commit);if (tx_id > txq->tx_next_rs) {
drivers/net/intel/ice/ice_rxtx_vec_avx512.c +6 -58
91 lines not on this page · raw message
drivers/net/intel/ice/ice_rxtx_vec_common.h +0 -59
63 lines not on this page · raw message
Signature · 1 line
2.53.0
PATCH v3 10/15 net/intel: move vector Tx paths to common
In reply to Bruce RichardsonThe main loops for vector Tx, both avx2 and avx512 are almost identical across a couple of drivers, so move the iavf copies to common for later reuse by other drivers.
Signed-off-by Bruce Richardson <[email protected]>
Acked-by Ciara Loftus <[email protected]>
--- drivers/net/intel/common/tx.h | 10 ++ drivers/net/intel/common/tx_vec_x86.h | 128 ++++++++++++++++++ drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c | 75 +--------- drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c | 77 +---------- 4 files changed, 148 insertions(+), 142 deletions(-)
drivers/net/intel/common/tx.h +10 -0
@@ -9,6 +9,8 @@#include <rte_mbuf.h> #include <rte_ethdev.h> #include <rte_vect.h> +#include <rte_io.h> +#include <rte_byteorder.h> /* Common TX Descriptor QW1 Field Definitions */ #define CI_TXD_QW1_DTYPE_S 0
@@ -273,6 +275,14 @@ ci_tx_backlog_entry_vec(struct ci_tx_entry_vec *txep, struct rte_mbuf **tx_pkts, txep[i].mbuf = tx_pkts[i]; } + +/* Write the Tx tail register, byte-swapped for hardware regardless of host endianness. */ +static __rte_always_inline void +ci_tx_qtx_tail_write (struct ci_tx_queue *txq,uint16_t tx_id) +{ +rte_write32_wc (rte_cpu_to_le_32 ((uint32_t )tx_id), txq->qtx_tail); +} +#define IETH_VPMD_TX_MAX_FREE_BUF 64typedef int (*ci_desc_done_fn )(struct ci_tx_queue *txq,uint16_t idx);
drivers/net/intel/common/tx_vec_x86.h +128 -0
140 lines not on this page · raw message
drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c +5 -70
93 lines not on this page · raw message
drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c +5 -72
95 lines not on this page · raw message
Signature · 1 line
2.53.0
PATCH v3 11/15 net/ice: use common AVX Tx functions
In reply to Bruce RichardsonUse the newly created common Tx functions in the vector path.
Signed-off-by Bruce Richardson <[email protected]>
Acked-by Ciara Loftus <[email protected]>
--- drivers/net/intel/ice/ice_rxtx_vec_avx2.c | 72 +------------- drivers/net/intel/ice/ice_rxtx_vec_avx512.c | 102 +++----------------- 2 files changed, 14 insertions(+), 160 deletions(-)
drivers/net/intel/ice/ice_rxtx_vec_avx2.c +2 -70
@@ -774,74 +774,6 @@ ice_recv_scattered_pkts_vec_avx2_offload(void *rx_queue, true); } -static __rte_always_inline uint16_t -ice_xmit_fixed_burst_vec_avx2 (void *tx_queue,struct rte_mbuf **tx_pkts, -uint16_t nb_pkts,bool offload) -{ -struct ci_tx_queue *txq = (struct ci_tx_queue *)tx_queue; -volatile struct ci_tx_desc *txdp; -struct ci_tx_entry_vec *txep; -uint16_t n, nb_commit, tx_id; -uint64_t flags =CI_TX_DESC_CMD_DEFAULT ; -uint64_t rs =CI_TX_DESC_CMD_RS |CI_TX_DESC_CMD_DEFAULT ; - -/* cross rx_thresh boundary is not allowed */ - nb_pkts =RTE_MIN (nb_pkts, txq->tx_rs_thresh); - -if (txq->nb_tx_free < txq->tx_free_thresh) -ci_tx_free_bufs_vec (txq, ice_tx_desc_done, false); - - nb_commit = nb_pkts = (uint16_t )RTE_MIN (txq->nb_tx_free, nb_pkts); -if (unlikely (nb_pkts ==0 )) -return 0 ; - - tx_id = txq->tx_tail; - txdp = &txq->ci_tx_ring[tx_id]; - txep = &txq->sw_ring_vec[tx_id]; - - txq->nb_tx_free = (uint16_t )(txq->nb_tx_free - nb_pkts); - - n = (uint16_t )(txq->nb_tx_desc - tx_id); -if (nb_commit >= n) { -ci_tx_backlog_entry_vec (txep, tx_pkts, n); - -ci_vtx_avx2 (txdp, tx_pkts, n -1 , flags, offload, -CI_TAG_IN_DATA_DESC ,CI_TAG_IN_DATA_DESC ); - tx_pkts += (n -1 ); - txdp += (n -1 ); - -ci_vtx1 (txdp, *tx_pkts++, rs, offload,CI_TAG_IN_DATA_DESC ,CI_TAG_IN_DATA_DESC ); - - nb_commit = (uint16_t )(nb_commit - n); - - tx_id =0 ; - txq->tx_next_rs = (uint16_t )(txq->tx_rs_thresh -1 ); - -/* avoid reach the end of ring */ - txdp = &txq->ci_tx_ring[tx_id]; - txep = &txq->sw_ring_vec[tx_id]; - } - -ci_tx_backlog_entry_vec (txep, tx_pkts, nb_commit); - -ci_vtx_avx2 (txdp, tx_pkts, nb_commit, flags, offload, -CI_TAG_IN_DATA_DESC ,CI_TAG_IN_DATA_DESC ); - - tx_id = (uint16_t )(tx_id + nb_commit); -if (tx_id > txq->tx_next_rs) { - txq->ci_tx_ring[txq->tx_next_rs].cmd_type_offset_bsz |= -rte_cpu_to_le_64 (((uint64_t )CI_TX_DESC_CMD_RS ) <<CI_TXD_QW1_CMD_S ); - txq->tx_next_rs = - (uint16_t )(txq->tx_next_rs + txq->tx_rs_thresh); - } - - txq->tx_tail = tx_id; - -ICE_PCI_REG_WC_WRITE (txq->qtx_tail, txq->tx_tail); - -return nb_pkts; -} - static __rte_always_inline uint16_tice_xmit_pkts_vec_avx2_common (void *tx_queue,struct rte_mbuf **tx_pkts,uint16_t nb_pkts,bool offload)
@@ -853,8 +785,8 @@ ice_xmit_pkts_vec_avx2_common(void *tx_queue, struct rte_mbuf **tx_pkts,uint16_t ret, num; num = (uint16_t )RTE_MIN (nb_pkts, txq->tx_rs_thresh); - ret =ice_xmit_fixed_burst_vec_avx2 (tx_queue, &tx_pkts[nb_tx], - num, offload); + ret =ci_xmit_fixed_burst_vec_avx2 (tx_queue, &tx_pkts[nb_tx], num, + offload,CI_TAG_IN_DATA_DESC ,CI_TAG_IN_DATA_DESC ); nb_tx += ret; nb_pkts -= ret;if (ret < num)
drivers/net/intel/ice/ice_rxtx_vec_avx512.c +12 -90
122 lines not on this page · raw message
Signature · 1 line
2.53.0
PATCH v3 12/15 net/intel: add common vector Tx fns with context handling
In reply to Bruce RichardsonThe iavf vector driver has support for more Tx offloads because it has Tx functions which handle transmit context descriptors. Move those context-handling functions to the common folder for reuse by other drivers.
Signed-off-by Bruce Richardson <[email protected]>
Acked-by Ciara Loftus <[email protected]>
--- drivers/net/intel/common/tx_vec_x86.h | 136 ++++++++++++++++++ drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c | 78 +--------- drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c | 89 +----------- 3 files changed, 147 insertions(+), 156 deletions(-)
drivers/net/intel/common/tx_vec_x86.h +136 -0
@@ -408,6 +408,74 @@ ci_xmit_fixed_burst_vec_avx2(struct ci_tx_queue *txq, struct rte_mbuf **tx_pkts,return nb_pkts; } +static __rte_always_inline uint16_t +ci_xmit_fixed_burst_vec_ctx_avx2 (struct ci_tx_queue *txq,struct rte_mbuf **tx_pkts, +uint16_t nb_pkts,bool offload, +enum ci_l2tag_pos single_vlan_pos,enum ci_l2tag_pos qinq_outer_pos, +ci_tx_ctx_lldp_fn lldp_check) +{ +volatile struct ci_tx_desc *txdp; +struct ci_tx_entry_vec *txep; +uint16_t n, nb_commit, nb_mbuf, tx_id; +uint64_t flags =CI_TX_DESC_CMD_DEFAULT ; +uint64_t rs =CI_TX_DESC_CMD_RS | flags; + +if (txq->nb_tx_free < txq->tx_free_thresh) +ci_tx_free_bufs_vec (txq, ci_tx_desc_done_simple, true); + + nb_commit = (uint16_t )RTE_MIN (txq->nb_tx_free, nb_pkts <<1 ); + nb_commit &=0xFFFE ; +if (unlikely (nb_commit ==0 )) +return 0 ; + + nb_pkts = nb_commit >>1 ; + tx_id = txq->tx_tail; + txdp = &txq->ci_tx_ring[tx_id]; + txep = (void *)txq->sw_ring; + txep += (tx_id >>1 ); + + txq->nb_tx_free = (uint16_t )(txq->nb_tx_free - nb_commit); + n = (uint16_t )(txq->nb_tx_desc - tx_id); + +if (n !=0 && nb_commit >= n) { + nb_mbuf = n >>1 ; +ci_tx_backlog_entry_vec (txep, tx_pkts, nb_mbuf); + +ci_vtx_ctx_avx2 (txdp, tx_pkts, nb_mbuf -1 , flags, offload, + single_vlan_pos, qinq_outer_pos, lldp_check); + tx_pkts += (nb_mbuf -1 ); + txdp += (n -2 ); +ci_vtx1_ctx_avx2 (txdp, *tx_pkts++, rs, offload, + single_vlan_pos, qinq_outer_pos, lldp_check); + + nb_commit = (uint16_t )(nb_commit - n); + + txq->tx_next_rs = (uint16_t )(txq->tx_rs_thresh -1 ); + tx_id =0 ; +/* avoid reach the end of ring */ + txdp = txq->ci_tx_ring; + txep = (void *)txq->sw_ring; + } + + nb_mbuf = nb_commit >>1 ; +ci_tx_backlog_entry_vec (txep, tx_pkts, nb_mbuf); + +ci_vtx_ctx_avx2 (txdp, tx_pkts, nb_mbuf, flags, offload, + single_vlan_pos, qinq_outer_pos, lldp_check); + tx_id = (uint16_t )(tx_id + nb_commit); + +if (tx_id > txq->tx_next_rs) { + txq->ci_tx_ring[txq->tx_next_rs].cmd_type_offset_bsz |= +rte_cpu_to_le_64 (((uint64_t )CI_TX_DESC_CMD_RS ) <<CI_TXD_QW1_CMD_S ); + txq->tx_next_rs = (uint16_t )(txq->tx_next_rs + txq->tx_rs_thresh); + } + + txq->tx_tail = tx_id; + +ci_tx_qtx_tail_write (txq, tx_id); +return nb_pkts; +} +#endif /* __AVX2__ */ #ifdef __AVX512VL__
@@ -640,6 +708,74 @@ ci_xmit_fixed_burst_vec_avx512(struct ci_tx_queue *txq, struct rte_mbuf **tx_pktreturn nb_pkts; } +static __rte_always_inline uint16_t +ci_xmit_fixed_burst_vec_ctx_avx512 (struct ci_tx_queue *txq,struct rte_mbuf **tx_pkts, +uint16_t nb_pkts,bool offload, +enum ci_l2tag_pos single_vlan_pos,enum ci_l2tag_pos qinq_outer_pos, +ci_tx_ctx_lldp_fn lldp_check) +{ +volatile struct ci_tx_desc *txdp; +struct ci_tx_entry_vec *txep; +uint16_t n, nb_commit, nb_mbuf, tx_id; +uint64_t flags =CI_TX_DESC_CMD_DEFAULT ; +uint64_t rs =CI_TX_DESC_CMD_RS | flags; + +if (txq->nb_tx_free < txq->tx_free_thresh) +ci_tx_free_bufs_vec (txq, ci_tx_desc_done_simple, true); + + nb_commit = (uint16_t )RTE_MIN (txq->nb_tx_free, nb_pkts <<1 ); + nb_commit &=0xFFFE ; +if (unlikely (nb_commit ==0 )) +return 0 ; + + nb_pkts = nb_commit >>1 ; + tx_id = txq->tx_tail; + txdp = &txq->ci_tx_ring[tx_id]; + txep = (void *)txq->sw_ring; + txep += (tx_id >>1 ); + + txq->nb_tx_free = (uint16_t )(txq->nb_tx_free - nb_commit); + n = (uint16_t )(txq->nb_tx_desc - tx_id); + +if (n !=0 && nb_commit >= n) { + nb_mbuf = n >>1 ; +ci_tx_backlog_entry_vec (txep, tx_pkts, nb_mbuf); + +ci_vtx_ctx_avx512 (txdp, tx_pkts, nb_mbuf -1 , flags, offload, + single_vlan_pos, qinq_outer_pos, lldp_check); + tx_pkts += (nb_mbuf -1 ); + txdp += (n -2 ); +ci_vtx1_ctx_avx512 (txdp, *tx_pkts++, rs, offload, + single_vlan_pos, qinq_outer_pos, lldp_check); + + nb_commit = (uint16_t )(nb_commit - n); + + txq->tx_next_rs = (uint16_t )(txq->tx_rs_thresh -1 ); + tx_id =0 ; +/* avoid reach the end of ring */ + txdp = txq->ci_tx_ring; + txep = (void *)txq->sw_ring; + } + + nb_mbuf = nb_commit >>1 ; +ci_tx_backlog_entry_vec (txep, tx_pkts, nb_mbuf); + +ci_vtx_ctx_avx512 (txdp, tx_pkts, nb_mbuf, flags, offload, + single_vlan_pos, qinq_outer_pos, lldp_check); + tx_id = (uint16_t )(tx_id + nb_commit); + +if (tx_id > txq->tx_next_rs) { + txq->ci_tx_ring[txq->tx_next_rs].cmd_type_offset_bsz |= +rte_cpu_to_le_64 (((uint64_t )CI_TX_DESC_CMD_RS ) <<CI_TXD_QW1_CMD_S ); + txq->tx_next_rs = (uint16_t )(txq->tx_next_rs + txq->tx_rs_thresh); + } + + txq->tx_tail = tx_id; + +ci_tx_qtx_tail_write (txq, tx_id); +return nb_pkts; +} +#endif /* __AVX512VL__ */ #endif /* _COMMON_INTEL_TX_VEC_X86_H_ */
drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c +5 -73
97 lines not on this page · raw message
drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c +6 -83
113 lines not on this page · raw message
Signature · 1 line
2.53.0
PATCH v3 13/15 net/intel: improve Tx path selection logic
In reply to Bruce RichardsonRework the relevant priority of context descriptor vs SIMD width when selecting a Tx path so that presence of context descriptors is only considered when all else is equal.
Normally, path select takes place without having to worry about context descriptors, since offloads requiring context descriptors are specified using the offloads bitmap for each Tx path. However, in some cases, the need to use a context descriptor is not encodable via this offload path, e.g. VLAN offload for iavf can sometimes use the data descriptor and sometimes the context depending on what PF specifies at runtime. To account for those cases, we separate out the use of contexts from the path features, but instead note it as part of the path information, and add an additional parameter to the Tx path selection for drivers to mandate a context descriptor be present. Otherwise, ctx is largely ignored in path selection, except as a last-resort tie-breaker between two paths.
Signed-off-by Bruce Richardson <[email protected]>
Acked-by Ciara Loftus <[email protected]>
--- drivers/net/intel/common/tx.h | 41 +++++++++++++++++--------- drivers/net/intel/cpfl/cpfl_rxtx.c | 3 +- drivers/net/intel/i40e/i40e_rxtx.c | 2 +- drivers/net/intel/iavf/iavf_rxtx.c | 46 ++++++++++++++++-------------- drivers/net/intel/ice/ice_rxtx.c | 3 +- drivers/net/intel/idpf/idpf_rxtx.c | 3 +- 6 files changed, 58 insertions(+), 40 deletions(-)
drivers/net/intel/common/tx.h +27 -14
@@ -249,7 +249,6 @@ struct ci_tx_path_features {uint32_t tx_offloads;enum rte_vect_max_simd simd_width;bool simple_tx; -bool ctx_desc;bool disabled;bool single_queue; };
@@ -259,6 +258,7 @@ struct ci_tx_path_info {const char *info;struct ci_tx_path_features features;eth_tx_prep_t pkt_prep; +bool supports_ctx; };static __rte_always_inline void
@@ -411,6 +411,11 @@ ci_txq_release_all_mbufs(struct ci_tx_queue *txq, bool use_ctx)* Number of available paths in the infos array * @param default_path * Index of the default path to use if no suitable path is found +* @param force_ctx +* If true, only paths that support context descriptors may be selected. +* Use this for offloads that require a context descriptor but cannot be +* discovered purely from req_features->tx_offloads (e.g. a driver-specific +* devarg, or a runtime/hardware-negotiated tag placement). * * @return * The packet burst function index that best matches the requested features,
@@ -420,10 +425,12 @@ static inline int ci_tx_path_select(const struct ci_tx_path_features *req_features, const struct ci_tx_path_info *infos, size_t num_paths, - int default_path) + int default_path, + bool force_ctx) { int idx = default_path;const struct ci_tx_path_features *chosen_path_features =NULL ; +bool chosen_supports_ctx = false;for (unsigned int i =0 ; i < num_paths; i++) {const struct ci_tx_path_features *path_features = &infos[i].features;
@@ -440,8 +447,8 @@ ci_tx_path_select(const struct ci_tx_path_features *req_features,if (path_features->simple_tx && !req_features->simple_tx)continue ; -/* If a context descriptor is requested, ensure the path supports it. */ -if (!path_features->ctx_desc && req_features->ctx_desc) +/* If a context descriptor is required, ensure the path supports it. */ +if (!infos[i].supports_ctx && force_ctx)continue ;/* If requested, ensure the path supports single queue TX. */
@@ -462,22 +469,28 @@ ci_tx_path_select(const struct ci_tx_path_features *req_features,/* Do not select paths with lower SIMD width than the chosen path. */ if (path_features->simd_width < chosen_path_features->simd_width)continue ; -/* Do not select paths with more offloads enabled than the chosen path if -* the SIMD widths are the same. +/* The following tie-breaks only matter when SIMD widths are tied; +* a strictly wider path is always preferred regardless of offload +* count or ctx-descriptor use. */ -if (path_features->simd_width == chosen_path_features->simd_width && -rte_popcount32 (path_features->tx_offloads) > -rte_popcount32 (chosen_path_features->tx_offloads)) -continue ; - -/* Don't use a context descriptor unless necessary */ -if (path_features->ctx_desc && !chosen_path_features->ctx_desc) -continue ; +if (path_features->simd_width == chosen_path_features->simd_width) { +/* Do not select paths with more offloads enabled than the +* chosen path. +*/ +if (rte_popcount32 (path_features->tx_offloads) > +rte_popcount32 (chosen_path_features->tx_offloads)) +continue ; + +/* Don't use a context descriptor unless necessary */ +if (infos[i].supports_ctx && !chosen_supports_ctx) +continue ; + } }/* Finally, select the path since it has met all the requirements. */ idx = i; chosen_path_features = &infos[idx].features; + chosen_supports_ctx = infos[idx].supports_ctx; }return idx;
drivers/net/intel/cpfl/cpfl_rxtx.c +2 -1
@@ -1542,7 +1542,8 @@ cpfl_set_tx_function(struct rte_eth_dev *dev) ad->tx_func_type =ci_tx_path_select (&req_features, &idpf_tx_path_infos[0 ],IDPF_TX_MAX , -IDPF_TX_DEFAULT ); +IDPF_TX_DEFAULT , + false);/* Set use_vec_entry for single queue mode - only IDPF_TX_SINGLEQ uses regular entries */ if (vport->txq_model ==VIRTCHNL2_QUEUE_MODEL_SINGLE ) {
drivers/net/intel/i40e/i40e_rxtx.c +1 -1
@@ -3120,7 +3120,7 @@ i40e_set_tx_function(struct rte_eth_dev *dev) } ad->tx_func_type =ci_tx_path_select (&req_features, &i40e_tx_path_infos[0 ], -RTE_DIM (i40e_tx_path_infos),I40E_TX_DEFAULT ); +RTE_DIM (i40e_tx_path_infos),I40E_TX_DEFAULT , false); out: dev->tx_pkt_burst = mbuf_check ? i40e_xmit_pkts_check :
drivers/net/intel/iavf/iavf_rxtx.c +24 -22
112 lines not on this page · raw message
drivers/net/intel/ice/ice_rxtx.c +2 -1
@@ -3784,7 +3784,8 @@ ice_set_tx_function(struct rte_eth_dev *dev) ad->tx_func_type =ci_tx_path_select (&req_features, &ice_tx_path_infos[0 ],RTE_DIM (ice_tx_path_infos), -ICE_TX_DEFAULT ); +ICE_TX_DEFAULT , + false); out:#if defined(RTE_ARCH_X86 )
drivers/net/intel/idpf/idpf_rxtx.c +2 -1
@@ -887,7 +887,8 @@ idpf_set_tx_function(struct rte_eth_dev *dev) ad->tx_func_type =ci_tx_path_select (&req_features, &idpf_tx_path_infos[0 ],IDPF_TX_MAX , -IDPF_TX_DEFAULT ); +IDPF_TX_DEFAULT , + false);/* Set use_vec_entry for single queue mode - only IDPF_TX_SINGLEQ uses regular entries */ if (vport->txq_model ==VIRTCHNL2_QUEUE_MODEL_SINGLE ) {
Signature · 1 line
2.53.0
PATCH v3 14/15 net/ice: enable context desc offloads for vector Tx
In reply to Bruce RichardsonUse the context descriptor Tx vector paths from intel/common in ice driver to enable more offloads on the vector path for ice driver.
Signed-off-by Bruce Richardson <[email protected]>
Acked-by Ciara Loftus <[email protected]>
--- doc/guides/nics/features/ice.ini | 8 +++--- doc/guides/rel_notes/release_26_11.rst | 6 ++++ drivers/net/intel/ice/ice_ethdev.h | 3 ++ drivers/net/intel/ice/ice_rxtx.c | 31 ++++++++++++++++++--- drivers/net/intel/ice/ice_rxtx.h | 11 ++++++++ drivers/net/intel/ice/ice_rxtx_vec_avx2.c | 24 ++++++++++++++++ drivers/net/intel/ice/ice_rxtx_vec_avx512.c | 24 ++++++++++++++++ 7 files changed, 99 insertions(+), 8 deletions(-)
doc/guides/nics/features/ice.ini +4 -4
@@ -13,7 +13,7 @@ Link status = Y Link status event = Y FEC = Y Rx interrupt = Y -Fast mbuf free = P +Fast mbuf free = Y Free Tx mbuf on demand = Y Queue start/stop = Y Burst mode info = Y
@@ -33,12 +33,12 @@ VLAN filter = Y Traffic manager = Y CRC offload = Y VLAN offload = Y -QinQ offload = P +QinQ offload = Y L3 checksum offload = Y L4 checksum offload = Y Timestamp offload = P -Inner L3 checksum = P -Inner L4 checksum = P +Inner L3 checksum = Y +Inner L4 checksum = Y Packet type parsing = Y Timesync = Y Rx descriptor status = Y
doc/guides/rel_notes/release_26_11.rst +6 -0
@@ -69,6 +69,12 @@ New Features Added ``fdir_buffer_size`` devarg to select the Flow Director table size (``64k``, ``128k`` or ``256k``) at probe time. The default remains ``64k``. +* **Updated Intel ice driver.** + + * Added Tx context descriptor support to the AVX2 and AVX512 vector Tx + paths, enabling QinQ tag insertion and outer IPv4/UDP checksum + offloads on those paths. + Removed Items -------------
drivers/net/intel/ice/ice_ethdev.h +3 -0
@@ -213,8 +213,10 @@ enum ice_tx_func_type {ICE_TX_SIMPLE ,ICE_TX_AVX2 ,ICE_TX_AVX2_OFFLOAD , +ICE_TX_AVX2_CTX_OFFLOAD ,ICE_TX_AVX512 ,ICE_TX_AVX512_OFFLOAD , +ICE_TX_AVX512_CTX_OFFLOAD , ICE_TX_NEON, };
@@ -676,6 +678,7 @@ struct ice_adapter {bool rx_bulk_alloc_allowed;bool tx_vec_allowed;bool tx_simple_allowed; +bool use_ctx;enum ice_rx_func_type rx_func_type;enum ice_tx_func_type tx_func_type;/* ptype mapping table */
drivers/net/intel/ice/ice_rxtx.c +27 -4
@@ -890,6 +890,7 @@ ice_tx_queue_start(struct rte_eth_dev *dev, uint16_t tx_queue_id)/* record what kind of descriptor cleanup we need on teardown */ txq->use_vec_entry = ad->tx_vec_allowed || ad->tx_simple_allowed; + txq->use_ctx = ad->use_ctx;if (txq->tsq !=NULL && txq->tsq->ts_flag >0 ) {struct ice_aqc_set_txtime_qgrp *ts_elem;
@@ -1200,7 +1201,7 @@ ice_tx_queue_stop(struct rte_eth_dev *dev, uint16_t tx_queue_id)return -EINVAL ; } -ci_txq_release_all_mbufs (txq, false); +ci_txq_release_all_mbufs (txq, txq->use_ctx);ice_reset_tx_queue (txq); dev->data->tx_queue_state[tx_queue_id] =RTE_ETH_QUEUE_STATE_STOPPED ;
@@ -1263,7 +1264,7 @@ ice_fdir_tx_queue_stop(struct rte_eth_dev *dev, uint16_t tx_queue_id)return -EINVAL ; } -ci_txq_release_all_mbufs (txq, false); +ci_txq_release_all_mbufs (txq, txq->use_ctx); txq->qtx_tail =NULL ;return 0 ;
@@ -1751,7 +1752,7 @@ ice_tx_queue_release(void *txq)return ; } -ci_txq_release_all_mbufs (q, false); +ci_txq_release_all_mbufs (q, q->use_ctx);rte_free (q->sw_ring);rte_free (q->rs_last_id);if (q->tsq) {
@@ -3533,7 +3534,8 @@ static const struct ci_tx_path_info ice_tx_path_infos[] = { .features = { .tx_offloads =ICE_TX_SCALAR_OFFLOADS }, - .pkt_prep = ice_prep_pkts + .pkt_prep = ice_prep_pkts, + .supports_ctx = true }, [ICE_TX_SIMPLE] = { .pkt_burst = ice_xmit_pkts_simple,
@@ -3563,6 +3565,16 @@ static const struct ci_tx_path_info ice_tx_path_infos[] = { }, .pkt_prep = ice_prep_pkts }, + [ICE_TX_AVX2_CTX_OFFLOAD] = { + .pkt_burst = ice_xmit_pkts_vec_avx2_ctx_offload, + .info = "Offload Vector AVX2 Ctx", + .features = { + .tx_offloads = ICE_TX_VECTOR_CTX_OFFLOAD_OFFLOADS, + .simd_width =RTE_VECT_SIMD_256 + }, + .pkt_prep = ice_prep_pkts, + .supports_ctx = true + }, #ifdef CC_AVX512_SUPPORT [ICE_TX_AVX512] = { .pkt_burst = ice_xmit_pkts_vec_avx512,
@@ -3582,6 +3594,16 @@ static const struct ci_tx_path_info ice_tx_path_infos[] = { }, .pkt_prep = ice_prep_pkts }, + [ICE_TX_AVX512_CTX_OFFLOAD] = { + .pkt_burst = ice_xmit_pkts_vec_avx512_ctx_offload, + .info = "Offload Vector AVX512 Ctx", + .features = { + .tx_offloads = ICE_TX_VECTOR_CTX_OFFLOAD_OFFLOADS, + .simd_width = RTE_VECT_SIMD_512 + }, + .pkt_prep = ice_prep_pkts, + .supports_ctx = true + },#endif #elif defined(RTE_ARCH_ARM64) [ICE_TX_NEON] = {
@@ -3795,6 +3817,7 @@ ice_set_tx_function(struct rte_eth_dev *dev) ad->tx_vec_allowed = (ice_tx_path_infos[ad->tx_func_type].features.simd_width >=RTE_VECT_SIMD_128 );#endif + ad->use_ctx = ice_tx_path_infos[ad->tx_func_type].supports_ctx; dev->tx_pkt_burst = mbuf_check ? ice_xmit_pkts_check : ice_tx_path_infos[ad->tx_func_type].pkt_burst;
drivers/net/intel/ice/ice_rxtx.h +11 -0
@@ -135,6 +135,12 @@ RTE_ETH_TX_OFFLOAD_UDP_CKSUM | \ RTE_ETH_TX_OFFLOAD_TCP_CKSUM | \ RTE_ETH_TX_OFFLOAD_SCTP_CKSUM) +/* vector ctx offload path: QinQ outer tag + outer/tunnel checksum via ctx desc */ +#define ICE_TX_VECTOR_CTX_OFFLOAD_OFFLOADS ( \ + ICE_TX_VECTOR_OFFLOAD_OFFLOADS | \ + RTE_ETH_TX_OFFLOAD_OUTER_IPV4_CKSUM | \ + RTE_ETH_TX_OFFLOAD_OUTER_UDP_CKSUM | \ + RTE_ETH_TX_OFFLOAD_QINQ_INSERT)/* Max header size can be 2K - 64 bytes */ #define ICE_RX_HDR_BUF_SIZE (2048 - 64)
@@ -284,6 +290,8 @@ uint16_t ice_xmit_pkts_vec_avx2(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts);uint16_t ice_xmit_pkts_vec_avx2_offload (void *tx_queue,struct rte_mbuf **tx_pkts,uint16_t nb_pkts); +uint16_t ice_xmit_pkts_vec_avx2_ctx_offload (void *tx_queue,struct rte_mbuf **tx_pkts, +uint16_t nb_pkts);uint16_t ice_recv_pkts_vec_avx512 (void *rx_queue,struct rte_mbuf **rx_pkts,uint16_t nb_pkts); uint16_t ice_recv_pkts_vec_avx512_offload(void *rx_queue,
@@ -300,6 +308,9 @@ uint16_t ice_xmit_pkts_vec_avx512(void *tx_queue, struct rte_mbuf **tx_pkts,uint16_t ice_xmit_pkts_vec_avx512_offload (void *tx_queue,struct rte_mbuf **tx_pkts,uint16_t nb_pkts); +uint16_t ice_xmit_pkts_vec_avx512_ctx_offload (void *tx_queue, +struct rte_mbuf **tx_pkts, +uint16_t nb_pkts);int ice_fdir_programming (struct ice_pf *pf,struct ice_fltr_desc *fdir_desc);int ice_tx_done_cleanup (void *txq,uint32_t free_cnt);int ice_get_monitor_addr (void *rx_queue,struct rte_power_monitor_cond *pmc);
drivers/net/intel/ice/ice_rxtx_vec_avx2.c +24 -0
@@ -810,6 +810,30 @@ ice_xmit_pkts_vec_avx2_offload(void *tx_queue, struct rte_mbuf **tx_pkts,return ice_xmit_pkts_vec_avx2_common (tx_queue, tx_pkts, nb_pkts, true); } +uint16_t +ice_xmit_pkts_vec_avx2_ctx_offload (void *tx_queue,struct rte_mbuf **tx_pkts, +uint16_t nb_pkts) +{ +uint16_t nb_tx =0 ; +struct ci_tx_queue *txq = (struct ci_tx_queue *)tx_queue; + +while (nb_pkts) { +uint16_t ret, num; + +/* cross rs_thresh boundary is not allowed */ + num = (uint16_t )RTE_MIN (nb_pkts <<1 , txq->tx_rs_thresh); + num = num >>1 ; + ret =ci_xmit_fixed_burst_vec_ctx_avx2 (tx_queue, &tx_pkts[nb_tx], num, + true,CI_TAG_IN_DATA_DESC ,CI_TAG_IN_CTX_DESC ,NULL ); + nb_tx += ret; + nb_pkts -= ret; +if (ret < num) +break ; + } + +return nb_tx; +} + int __rte_coldice_rxq_vec_setup (struct ci_rx_queue *rxq) {
drivers/net/intel/ice/ice_rxtx_vec_avx512.c +24 -0
@@ -882,3 +882,27 @@ ice_xmit_pkts_vec_avx512_offload(void *tx_queue, struct rte_mbuf **tx_pkts, {return ice_xmit_pkts_vec_avx512_common (tx_queue, tx_pkts, nb_pkts, true); } + +uint16_t +ice_xmit_pkts_vec_avx512_ctx_offload (void *tx_queue,struct rte_mbuf **tx_pkts, +uint16_t nb_pkts) +{ +uint16_t nb_tx =0 ; +struct ci_tx_queue *txq = (struct ci_tx_queue *)tx_queue; + +while (nb_pkts) { +uint16_t ret, num; + +/* cross rs_thresh boundary is not allowed */ + num = (uint16_t )RTE_MIN (nb_pkts <<1 , txq->tx_rs_thresh); + num = num >>1 ; + ret =ci_xmit_fixed_burst_vec_ctx_avx512 (tx_queue, &tx_pkts[nb_tx], num, + true,CI_TAG_IN_DATA_DESC ,CI_TAG_IN_CTX_DESC ,NULL ); + nb_tx += ret; + nb_pkts -= ret; +if (ret < num) +break ; + } + +return nb_tx; +}
Signature · 1 line
2.53.0
RE: [PATCH v3 14/15] net/ice: enable context desc offloads for vector Tx
Quoted 31 lines
Subject: [PATCH v3 14/15] net/ice: enable context desc offloads for vector Tx
Use the context descriptor Tx vector paths from intel/common in ice driver to enable more offloads on the vector path for ice driver.
Signed-off-by Bruce Richardson <[email protected]>Acked-by Ciara Loftus <[email protected]>--- doc/guides/nics/features/ice.ini | 8 +++--- doc/guides/rel_notes/release_26_11.rst | 6 ++++ drivers/net/intel/ice/ice_ethdev.h | 3 ++ drivers/net/intel/ice/ice_rxtx.c | 31 ++++++++++++++++++--- drivers/net/intel/ice/ice_rxtx.h | 11 ++++++++ drivers/net/intel/ice/ice_rxtx_vec_avx2.c | 24 ++++++++++++++++ drivers/net/intel/ice/ice_rxtx_vec_avx512.c | 24 ++++++++++++++++ 7 files changed, 99 insertions(+), 8 deletions(-)doc/guides/nics/features/ice.ini +1
-1@@ -13,7 +13,7 @@ Link status = Y Link status event = Y FEC = Y Rx interrupt = Y -Fast mbuf free = P +Fast mbuf free = Y Free Tx mbuf on demand = Y Queue start/stop = Y Burst mode info = Y
doc/guides/nics/features/ice.ini VLAN filter = Y · line 36
Quoted 5 lines
doc/guides/nics/features/ice.ini +1
-1@@ -33,12 +33,12 @@ VLAN filter = Y Traffic manager = Y CRC offload = Y VLAN offload = Y -QinQ offload = P +QinQ offload = Y
I think this should stay as P because the vector Rx paths don't support QINQ strip.
doc/guides/nics/features/ice.ini VLAN filter = Y · line 41
Quoted 7 lines
doc/guides/nics/features/ice.ini +2
-2L3 checksum offload = Y L4 checksum offload = Y Timestamp offload = P -Inner L3 checksum = P -Inner L4 checksum = P +Inner L3 checksum = Y +Inner L4 checksum = Y
Both of these are OK. Although I've noticed that Inner L4 checksum aka RTE_ETH_RX_OFFLOAD_OUTER_UDP_CKSUM isn't advertised correctly in the path offload masks / rx_offload_capa so I will submit a separate patch to fix that.
Quoted 12 lines
doc/guides/nics/features/ice.ini +0
-0Packet type parsing = Y Timesync = Y Rx descriptor status = Ydoc/guides/rel_notes/release_26_11.rst +0
-0@@ -69,6 +69,12 @@ New Features Added ``fdir_buffer_size`` devarg to select the Flow Director table size (``64k``, ``128k`` or ``256k``) at probe time. The default remains ``64k``.
Re: [PATCH v3 14/15] net/ice: enable context desc offloads for vector Tx
Quoted 31 lines · Ciara
Quoted 31 lines
Subject: [PATCH v3 14/15] net/ice: enable context desc offloads for vector Tx
Use the context descriptor Tx vector paths from intel/common in ice driver to enable more offloads on the vector path for ice driver.
Signed-off-by Bruce Richardson <[email protected]>Acked-by Ciara Loftus <[email protected]>--- doc/guides/nics/features/ice.ini | 8 +++--- doc/guides/rel_notes/release_26_11.rst | 6 ++++ drivers/net/intel/ice/ice_ethdev.h | 3 ++ drivers/net/intel/ice/ice_rxtx.c | 31 ++++++++++++++++++--- drivers/net/intel/ice/ice_rxtx.h | 11 ++++++++ drivers/net/intel/ice/ice_rxtx_vec_avx2.c | 24 ++++++++++++++++ drivers/net/intel/ice/ice_rxtx_vec_avx512.c | 24 ++++++++++++++++ 7 files changed, 99 insertions(+), 8 deletions(-)doc/guides/nics/features/ice.ini +1
-1@@ -13,7 +13,7 @@ Link status = Y Link status event = Y FEC = Y Rx interrupt = Y -Fast mbuf free = P +Fast mbuf free = Y Free Tx mbuf on demand = Y Queue start/stop = Y Burst mode info = Y@@ -33,12 +33,12 @@ VLAN filter = Y
Quoted 9 lines
doc/guides/nics/features/ice.ini VLAN filter = Y · line 36
Quoted 5 lines
doc/guides/nics/features/ice.ini +1
-1Traffic manager = Y CRC offload = Y VLAN offload = Y -QinQ offload = P +QinQ offload = YI think this should stay as P because the vector Rx paths don't support QINQ strip.
Ok, will revert on apply.
Thanks for flagging.
PATCH v3 15/15 net/intel: consolidate Tx vector offload path checks
In reply to Bruce RichardsonSince iavf and ice drivers now have common Tx vector paths, we can consolidate the path decisions a little, including replacing the per-driver offload flags with common ones.
Signed-off-by Bruce Richardson <[email protected]>
Acked-by Ciara Loftus <[email protected]>
--- drivers/net/intel/common/tx.h | 30 +++++++++++++++++++ drivers/net/intel/iavf/iavf_rxtx.c | 18 +++++------ drivers/net/intel/iavf/iavf_rxtx.h | 19 ------------ drivers/net/intel/iavf/iavf_rxtx_vec_common.h | 3 +- drivers/net/intel/ice/ice_rxtx.c | 14 ++++----- drivers/net/intel/ice/ice_rxtx.h | 18 ----------- drivers/net/intel/ice/ice_rxtx_vec_common.h | 3 +- 7 files changed, 48 insertions(+), 57 deletions(-)
drivers/net/intel/common/tx.h +30 -0
@@ -12,6 +12,27 @@#include <rte_io.h> #include <rte_byteorder.h> +#define CI_VPMD_TX_BURST 32 +#define CI_VPMD_TX_MAX_FREE_BUF 64 + +/* basic vector path */ +#define CI_TX_VECTOR_OFFLOADS RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE +/* offload vector path */ +#define CI_TX_VEC_OFFLOAD_PATH_OFFLOADS ( \ + CI_TX_VECTOR_OFFLOADS | \ + RTE_ETH_TX_OFFLOAD_VLAN_INSERT | \ + RTE_ETH_TX_OFFLOAD_IPV4_CKSUM | \ + RTE_ETH_TX_OFFLOAD_SCTP_CKSUM | \ + RTE_ETH_TX_OFFLOAD_UDP_CKSUM | \ + RTE_ETH_TX_OFFLOAD_TCP_CKSUM) +/* offload vector path with context descriptor */ +#define CI_TX_VEC_CTX_OFFLOAD_PATH_OFFLOADS ( \ + CI_TX_VECTOR_OFFLOADS | \ + CI_TX_VEC_OFFLOAD_PATH_OFFLOADS | \ + RTE_ETH_TX_OFFLOAD_OUTER_IPV4_CKSUM | \ + RTE_ETH_TX_OFFLOAD_OUTER_UDP_CKSUM | \ + RTE_ETH_TX_OFFLOAD_QINQ_INSERT) +/* Common TX Descriptor QW1 Field Definitions */ #define CI_TXD_QW1_DTYPE_S 0#define CI_TXD_QW1_DTYPE_M (0xFUL << CI_TXD_QW1_DTYPE_S)
@@ -261,6 +282,15 @@ struct ci_tx_path_info {bool supports_ctx; }; +/* basic check for a vector-driver capable Tx queue. +* Individual drivers may have other further tests beyond this. +*/ +static inline bool +ci_txq_vec_capable (uint16_t rs_thresh) +{ +return rs_thresh >=CI_VPMD_TX_BURST && rs_thresh <=CI_VPMD_TX_MAX_FREE_BUF ; +} + static __rte_always_inline voidci_tx_backlog_entry (struct ci_tx_entry *txep,struct rte_mbuf **tx_pkts,uint16_t nb_pkts) {
drivers/net/intel/iavf/iavf_rxtx.c +9 -9
@@ -3620,7 +3620,7 @@ static const struct ci_tx_path_info iavf_tx_path_infos[] = { .pkt_burst = iavf_xmit_pkts_vec_avx2, .info = "Vector AVX2", .features = { - .tx_offloads = IAVF_TX_VECTOR_OFFLOADS, + .tx_offloads = CI_TX_VECTOR_OFFLOADS, .simd_width = RTE_VECT_SIMD_256 } },
@@ -3628,7 +3628,7 @@ static const struct ci_tx_path_info iavf_tx_path_infos[] = { .pkt_burst = iavf_xmit_pkts_vec_avx2_offload, .info = "Vector AVX2 Offload", .features = { - .tx_offloads = IAVF_TX_VECTOR_OFFLOAD_OFFLOADS, + .tx_offloads = CI_TX_VEC_OFFLOAD_PATH_OFFLOADS, .simd_width = RTE_VECT_SIMD_256 } },
@@ -3636,7 +3636,7 @@ static const struct ci_tx_path_info iavf_tx_path_infos[] = { .pkt_burst = iavf_xmit_pkts_vec_avx2_ctx, .info = "Vector AVX2 Ctx", .features = { - .tx_offloads = IAVF_TX_VECTOR_OFFLOADS, + .tx_offloads = CI_TX_VECTOR_OFFLOADS, .simd_width =RTE_VECT_SIMD_256 }, .supports_ctx = true
@@ -3645,7 +3645,7 @@ static const struct ci_tx_path_info iavf_tx_path_infos[] = { .pkt_burst = iavf_xmit_pkts_vec_avx2_ctx_offload, .info = "Vector AVX2 Ctx Offload", .features = { - .tx_offloads = IAVF_TX_VECTOR_CTX_OFFLOAD_OFFLOADS, + .tx_offloads = CI_TX_VEC_CTX_OFFLOAD_PATH_OFFLOADS, .simd_width =RTE_VECT_SIMD_256 }, .supports_ctx = true
@@ -3655,7 +3655,7 @@ static const struct ci_tx_path_info iavf_tx_path_infos[] = { .pkt_burst = iavf_xmit_pkts_vec_avx512, .info = "Vector AVX512", .features = { - .tx_offloads = IAVF_TX_VECTOR_OFFLOADS, + .tx_offloads = CI_TX_VECTOR_OFFLOADS, .simd_width = RTE_VECT_SIMD_512 } },
@@ -3663,7 +3663,7 @@ static const struct ci_tx_path_info iavf_tx_path_infos[] = { .pkt_burst = iavf_xmit_pkts_vec_avx512_offload, .info = "Vector AVX512 Offload", .features = { - .tx_offloads = IAVF_TX_VECTOR_OFFLOAD_OFFLOADS, + .tx_offloads = CI_TX_VEC_OFFLOAD_PATH_OFFLOADS, .simd_width = RTE_VECT_SIMD_512 } },
@@ -3671,7 +3671,7 @@ static const struct ci_tx_path_info iavf_tx_path_infos[] = { .pkt_burst = iavf_xmit_pkts_vec_avx512_ctx, .info = "Vector AVX512 Ctx", .features = { - .tx_offloads = IAVF_TX_VECTOR_OFFLOADS, + .tx_offloads = CI_TX_VECTOR_OFFLOADS, .simd_width =RTE_VECT_SIMD_512 }, .supports_ctx = true
@@ -3680,7 +3680,7 @@ static const struct ci_tx_path_info iavf_tx_path_infos[] = { .pkt_burst = iavf_xmit_pkts_vec_avx512_ctx_offload, .info = "Vector AVX512 Ctx Offload", .features = { - .tx_offloads = IAVF_TX_VECTOR_CTX_OFFLOAD_OFFLOADS, + .tx_offloads = CI_TX_VEC_CTX_OFFLOAD_PATH_OFFLOADS, .simd_width =RTE_VECT_SIMD_512 }, .supports_ctx = true
@@ -3691,7 +3691,7 @@ static const struct ci_tx_path_info iavf_tx_path_infos[] = { .pkt_burst = iavf_xmit_pkts_vec, .info = "Vector Neon", .features = { - .tx_offloads = IAVF_TX_VECTOR_OFFLOADS, + .tx_offloads = CI_TX_VECTOR_OFFLOADS, .simd_width = RTE_VECT_SIMD_128 } },
drivers/net/intel/iavf/iavf_rxtx.h +0 -19
@@ -29,11 +29,9 @@/* used for Vector PMD */ #define IAVF_VPMD_RX_BURST CI_VPMD_RX_BURST -#define IAVF_VPMD_TX_BURST 32#define IAVF_VPMD_RXQ_REARM_THRESH CI_VPMD_RX_REARM_THRESH#define IAVF_VPMD_DESCS_PER_LOOP CI_VPMD_DESCS_PER_LOOP#define IAVF_VPMD_DESCS_PER_LOOP_WIDE CI_VPMD_DESCS_PER_LOOP_WIDE -#define IAVF_VPMD_TX_MAX_FREE_BUF 64/* basic scalar path */ #define IAVF_TX_SCALAR_OFFLOADS ( \
@@ -54,23 +52,6 @@ RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE | \ RTE_ETH_TX_OFFLOAD_SECURITY | \ RTE_ETH_TX_OFFLOAD_OUTER_UDP_CKSUM) -/* basic vector path */ -#define IAVF_TX_VECTOR_OFFLOADS RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE -/* offload vector path */ -#define IAVF_TX_VECTOR_OFFLOAD_OFFLOADS ( \ - IAVF_TX_VECTOR_OFFLOADS | \ - RTE_ETH_TX_OFFLOAD_VLAN_INSERT | \ - RTE_ETH_TX_OFFLOAD_IPV4_CKSUM | \ - RTE_ETH_TX_OFFLOAD_SCTP_CKSUM | \ - RTE_ETH_TX_OFFLOAD_UDP_CKSUM | \ - RTE_ETH_TX_OFFLOAD_TCP_CKSUM) -/* offload vector path with context descriptor */ -#define IAVF_TX_VECTOR_CTX_OFFLOAD_OFFLOADS ( \ - IAVF_TX_VECTOR_OFFLOADS | \ - IAVF_TX_VECTOR_OFFLOAD_OFFLOADS | \ - RTE_ETH_TX_OFFLOAD_OUTER_IPV4_CKSUM | \ - RTE_ETH_TX_OFFLOAD_OUTER_UDP_CKSUM | \ - RTE_ETH_TX_OFFLOAD_QINQ_INSERT)/* basic scalar path */ #define IAVF_RX_SCALAR_OFFLOADS ( \
drivers/net/intel/iavf/iavf_rxtx_vec_common.h +1 -2
@@ -80,8 +80,7 @@ iavf_tx_vec_queue_default(struct ci_tx_queue *txq)if (!txq)return -1 ; -if (txq->tx_rs_thresh <IAVF_VPMD_TX_BURST || - txq->tx_rs_thresh >IAVF_VPMD_TX_MAX_FREE_BUF ) +if (!ci_txq_vec_capable (txq->tx_rs_thresh))return -1 ;return 0 ;
drivers/net/intel/ice/ice_rxtx.c +7 -7
@@ -3551,7 +3551,7 @@ static const struct ci_tx_path_info ice_tx_path_infos[] = { .pkt_burst = ice_xmit_pkts_vec_avx2, .info = "Vector AVX2", .features = { - .tx_offloads = ICE_TX_VECTOR_OFFLOADS, + .tx_offloads = CI_TX_VECTOR_OFFLOADS, .simd_width =RTE_VECT_SIMD_256 }, .pkt_prep = rte_eth_tx_pkt_prepare_dummy
@@ -3560,7 +3560,7 @@ static const struct ci_tx_path_info ice_tx_path_infos[] = { .pkt_burst = ice_xmit_pkts_vec_avx2_offload, .info = "Offload Vector AVX2", .features = { - .tx_offloads = ICE_TX_VECTOR_OFFLOAD_OFFLOADS, + .tx_offloads = CI_TX_VEC_OFFLOAD_PATH_OFFLOADS, .simd_width =RTE_VECT_SIMD_256 }, .pkt_prep = ice_prep_pkts
@@ -3569,7 +3569,7 @@ static const struct ci_tx_path_info ice_tx_path_infos[] = { .pkt_burst = ice_xmit_pkts_vec_avx2_ctx_offload, .info = "Offload Vector AVX2 Ctx", .features = { - .tx_offloads = ICE_TX_VECTOR_CTX_OFFLOAD_OFFLOADS, + .tx_offloads = CI_TX_VEC_CTX_OFFLOAD_PATH_OFFLOADS, .simd_width =RTE_VECT_SIMD_256 }, .pkt_prep = ice_prep_pkts,
@@ -3580,7 +3580,7 @@ static const struct ci_tx_path_info ice_tx_path_infos[] = { .pkt_burst = ice_xmit_pkts_vec_avx512, .info = "Vector AVX512", .features = { - .tx_offloads = ICE_TX_VECTOR_OFFLOADS, + .tx_offloads = CI_TX_VECTOR_OFFLOADS, .simd_width =RTE_VECT_SIMD_512 }, .pkt_prep = rte_eth_tx_pkt_prepare_dummy
@@ -3589,7 +3589,7 @@ static const struct ci_tx_path_info ice_tx_path_infos[] = { .pkt_burst = ice_xmit_pkts_vec_avx512_offload, .info = "Offload Vector AVX512", .features = { - .tx_offloads = ICE_TX_VECTOR_OFFLOAD_OFFLOADS, + .tx_offloads = CI_TX_VEC_OFFLOAD_PATH_OFFLOADS, .simd_width =RTE_VECT_SIMD_512 }, .pkt_prep = ice_prep_pkts
@@ -3598,7 +3598,7 @@ static const struct ci_tx_path_info ice_tx_path_infos[] = { .pkt_burst = ice_xmit_pkts_vec_avx512_ctx_offload, .info = "Offload Vector AVX512 Ctx", .features = { - .tx_offloads = ICE_TX_VECTOR_CTX_OFFLOAD_OFFLOADS, + .tx_offloads = CI_TX_VEC_CTX_OFFLOAD_PATH_OFFLOADS, .simd_width =RTE_VECT_SIMD_512 }, .pkt_prep = ice_prep_pkts,
@@ -3610,7 +3610,7 @@ static const struct ci_tx_path_info ice_tx_path_infos[] = { .pkt_burst = ice_xmit_pkts_vec, .info = "Vector Neon", .features = { - .tx_offloads = ICE_TX_VECTOR_OFFLOADS, + .tx_offloads = CI_TX_VECTOR_OFFLOADS, .simd_width =RTE_VECT_SIMD_128 }, .pkt_prep = rte_eth_tx_pkt_prepare_dummy
drivers/net/intel/ice/ice_rxtx.h +0 -18
@@ -47,9 +47,7 @@#define ICE_SUPPORT_CHAIN_NUM 5#define ICE_VPMD_RX_BURST CI_VPMD_RX_BURST -#define ICE_VPMD_TX_BURST 32#define ICE_VPMD_RXQ_REARM_THRESH CI_VPMD_RX_REARM_THRESH -#define ICE_TX_MAX_FREE_BUF_SZ 64#define ICE_VPMD_DESCS_PER_LOOP CI_VPMD_DESCS_PER_LOOP#define ICE_VPMD_DESCS_PER_LOOP_WIDE CI_VPMD_DESCS_PER_LOOP_WIDE
@@ -125,22 +123,6 @@ RTE_ETH_TX_OFFLOAD_IPIP_TNL_TSO | \ RTE_ETH_TX_OFFLOAD_GENEVE_TNL_TSO | \ RTE_ETH_TX_OFFLOAD_SEND_ON_TIMESTAMP) -/* basic vector path */ -#define ICE_TX_VECTOR_OFFLOADS RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE -/* vector offload paths */ -#define ICE_TX_VECTOR_OFFLOAD_OFFLOADS ( \ - ICE_TX_VECTOR_OFFLOADS | \ - RTE_ETH_TX_OFFLOAD_VLAN_INSERT | \ - RTE_ETH_TX_OFFLOAD_IPV4_CKSUM | \ - RTE_ETH_TX_OFFLOAD_UDP_CKSUM | \ - RTE_ETH_TX_OFFLOAD_TCP_CKSUM | \ - RTE_ETH_TX_OFFLOAD_SCTP_CKSUM) -/* vector ctx offload path: QinQ outer tag + outer/tunnel checksum via ctx desc */ -#define ICE_TX_VECTOR_CTX_OFFLOAD_OFFLOADS ( \ - ICE_TX_VECTOR_OFFLOAD_OFFLOADS | \ - RTE_ETH_TX_OFFLOAD_OUTER_IPV4_CKSUM | \ - RTE_ETH_TX_OFFLOAD_OUTER_UDP_CKSUM | \ - RTE_ETH_TX_OFFLOAD_QINQ_INSERT)/* Max header size can be 2K - 64 bytes */ #define ICE_RX_HDR_BUF_SIZE (2048 - 64)
drivers/net/intel/ice/ice_rxtx_vec_common.h +1 -2
@@ -72,8 +72,7 @@ ice_tx_vec_queue_default(struct ci_tx_queue *txq)if (!txq)return -1 ; -if (txq->tx_rs_thresh <ICE_VPMD_TX_BURST || - txq->tx_rs_thresh >ICE_TX_MAX_FREE_BUF_SZ ) +if (!ci_txq_vec_capable (txq->tx_rs_thresh))return -1 ;return 0 ;
Signature · 1 line
2.53.0
Re: [PATCH v3 00/15] consolidate ice and iavf Tx vector paths
In reply to Bruce RichardsonQuoted 15 lines · Bruce Richardson
This patchset consolidates the ice and iavf vector Tx paths, moving the code to
net/intel/common. The iavf code is used as the basis for the move, because it has a superset of the features. When the common code is adopted by the ice driver, it then gains features such as QinQ support and tunneling support.V3:
* fix variable shadowing issue in patch 15 (ci_txq_vec_capableparam)
* update ice.ini with new featuresV2:
* fix build when iova not in mbuf
* add extra patch to consolidate the offload flags
Quoted 17 lines
Bruce Richardson (15):
net/iavf: remove unnecessary alignment calls net/intel: make Tx context flag common
net/iavf: use separate params for VLAN and QinQ position net/iavf: deduplicate tunnel field fill functions net/intel: define common macros for tunneling bit-shifts net/intel: move iavf descriptor writing functions to common net/intel: use function callback for lldp net/intel: allow building without IOVA in mbuf net/ice: use common descriptor creation functions net/intel: move vector Tx paths to common net/ice: use common AVX Tx functions
net/intel: add common vector Tx fns with context handling net/intel: improve Tx path selection logic net/ice: enable context desc offloads for vector Tx net/intel: consolidate Tx vector offload path checks
Series applied to dpdk-next-net-intel.
/Bruce