add max SIMD bitwidth to EAL
v10 cover add max SIMD bitwidth to EAL
From: Ciara Power <[email protected]>
A number of components in DPDK have optional AVX-512 or other vector code paths which can be selected at runtime. Rather than having each component provide its own mechanism to select a code path, this patchset adds support for a single setting to control what code paths are used. This can be used to enable some non-default code paths e.g. ones using AVX-512, but also to limit the code paths to certain vector widths, or to scalar code only, which is useful for testing.
The max SIMD bitwidth setting can be set by the app itself through use of the available API, or can be overriden by a commandline argument passed by the user.
v10:
- Updated API namespace and moved it to rte_vect.h,
- Split patch 2 and merged relevant parts in patch 1,
v9:
- Rebased onto main.
- Added release note.
v8:
- Rebased onto main.
- Fixed ACL patch to include AVX512X32 as a default option,
and modified docs to reflect that change.
v7: Fixed comments on v6.
v6:
- Rebased on main.
- Added ACL patch based on recent changes to the library.
- Modified net library patch based on recent rework.
- Fixed comments on the v5.
v5: Fixed cc list.
v4:
- Dropped LPM patch as the lookupx4 function is called from SSE/NEON headers, so is already on a vectorised path. Given the performance impact for an unnecessary change, it was decided the check is not needed.
- Renamed enum values for readability.
- Added patch to add check for node library.
- Reworked net patch to choose default handlers rather than scalar by default.
- Updated some Doxygen comments.
- Fixed some other small comments on v3.
v3:
- Added patch to add check for LPM lib
- Modified default max bitwidth for Arm to disable max SIMD bitwidth, which will allow for SVE.
- Added "0" as an acceptable value for command-line flag, which internally is used as UINT16_MAX to essentially disable max SIMD bitwidth limits.
- Made suggested changes to net lib patch.
- Rebased onto main.
v2:
- Added some documentation.
- Modified default max bitwidth for Arm.
- Moved mlx5 condition check into existing check vec support function.
- Added max SIMD bitwidth checks to some libraries.
Ciara Power (18):
eal: control max SIMD bitwidth
doc: describe how to enable AVX512
net/i40e: check max SIMD bitwidth
net/axgbe: check max SIMD bitwidth
net/bnxt: check max SIMD bitwidth
net/enic: check max SIMD bitwidth
net/fm10k: check max SIMD bitwidth
net/iavf: check max SIMD bitwidth
net/ice: check max SIMD bitwidth
net/ixgbe: check max SIMD bitwidth
net/mlx5: check max SIMD bitwidth
net/virtio: check max SIMD bitwidth
distributor: check max SIMD bitwidth
member: check max SIMD bitwidth
efd: check max SIMD bitwidth
net: check max SIMD bitwidth
node: choose vector path at runtime
acl: check max SIMD bitwidth
doc/guides/howto/avx512.rst | 36 ++++++ doc/guides/howto/index.rst | 1 + doc/guides/linux_gsg/eal_args.include.rst | 16 +++ .../prog_guide/env_abstraction_layer.rst | 34 +++++ .../prog_guide/packet_classif_access_ctrl.rst | 16 +-- doc/guides/rel_notes/release_20_11.rst | 6 + drivers/net/axgbe/axgbe_rxtx.c | 4 +- drivers/net/bnxt/bnxt_ethdev.c | 7 +- drivers/net/enic/enic_rxtx_vec_avx2.c | 4 +- drivers/net/fm10k/fm10k_ethdev.c | 10 +- drivers/net/i40e/i40e_rxtx.c | 19 ++- drivers/net/iavf/iavf_rxtx.c | 17 ++- drivers/net/ice/ice_rxtx.c | 17 ++- drivers/net/ixgbe/ixgbe_rxtx.c | 6 +- drivers/net/mlx5/mlx5_rxtx_vec.c | 3 + drivers/net/virtio/virtio_ethdev.c | 10 +- lib/librte_acl/rte_acl.c | 53 +++++--- lib/librte_acl/rte_acl.h | 1 + lib/librte_distributor/rte_distributor.c | 4 +- lib/librte_eal/arm/include/rte_vect.h | 2 + lib/librte_eal/common/eal_common_options.c | 66 ++++++++++ lib/librte_eal/common/eal_internal_cfg.h | 8 ++ lib/librte_eal/common/eal_options.h | 2 + lib/librte_eal/include/generic/rte_vect.h | 53 +++++++- lib/librte_eal/ppc/include/rte_vect.h | 2 + lib/librte_eal/rte_eal_exports.def | 3 + lib/librte_eal/rte_eal_version.map | 2 + lib/librte_eal/x86/include/rte_vect.h | 2 + lib/librte_efd/rte_efd.c | 8 +- lib/librte_member/rte_member_ht.c | 4 +- lib/librte_net/rte_net_crc.c | 117 +++++++++++++----- lib/librte_node/ip4_lookup.c | 15 ++- lib/librte_node/ip4_lookup_neon.h | 2 +- lib/librte_node/ip4_lookup_sse.h | 2 +- 34 files changed, 458 insertions(+), 94 deletions(-) create mode 100644 doc/guides/howto/avx512.rst
Signature · 1 line
2.23.0
PATCH v10 1/18 eal: control max SIMD bitwidth
From: Ciara Power <[email protected]>
This patch adds a max SIMD bitwidth EAL configuration. The API allows for an app to set this value. It can also be set using EAL argument --force-max-simd-bitwidth, which will lock the value and override any modifications made by the app.
Each arch has a define for the default SIMD bitwidth value, this is used on EAL init to set the config max SIMD bitwidth.
---
v10:
- moved new API to rte_vect.h, prefixed with rte_vect_ and added experimental banner,
- split patch 2 and squashed eal flags documentation in patch 1,
v9: Added release note.
v7: Reordered entries in the symbol .map file.
v4:
- Used RTE_SIMD_MAX instead of UINT16_MAX.
- Renamed enums to better reflect usage.
- Added functions to windows symbol export file.
- Modified Doxygen comments.
- Modified enum name.
- Changed RTE_SIMD_MAX value to a power of 2.
- Merged patch 2 into this patch.
- Enum now used for default value defines.
- Fixed some small comments on v3.
v3:
- Added enum value to essentially disable using max SIMD to choose paths, intended for use by ARM SVE.
- Fixed parsing bitwidth argument to return an error for values greater than uint16_t.
- Removed unnecessary define in generic rte_vect.h
- Changed default bitwidth for ARM to UINT16_MAX, to allow for SVE.
v2:
- Added to Doxygen comment for API.
- Changed default bitwidth for Arm to 128.
--- doc/guides/linux_gsg/eal_args.include.rst | 16 +++++ .../prog_guide/env_abstraction_layer.rst | 34 ++++++++++ doc/guides/rel_notes/release_20_11.rst | 6 ++ lib/librte_eal/arm/include/rte_vect.h | 2 + lib/librte_eal/common/eal_common_options.c | 66 +++++++++++++++++++ lib/librte_eal/common/eal_internal_cfg.h | 8 +++ lib/librte_eal/common/eal_options.h | 2 + lib/librte_eal/include/generic/rte_vect.h | 53 ++++++++++++++- lib/librte_eal/ppc/include/rte_vect.h | 2 + lib/librte_eal/rte_eal_exports.def | 3 + lib/librte_eal/rte_eal_version.map | 2 + lib/librte_eal/x86/include/rte_vect.h | 2 + 12 files changed, 194 insertions(+), 2 deletions(-)
doc/guides/linux_gsg/eal_args.include.rst +16 -0
@@ -210,3 +210,19 @@ Other options * ``--no-telemetry``: Disable telemetry. + +* ``--force-max-simd-bitwidth=<val>``: + + Specify the maximum SIMD bitwidth size to handle. This limits which vector paths, + if any, are taken, as any paths taken must use a bitwidth below the max bitwidth limit. + For example, to allow all SIMD bitwidths up to and including AVX-512:: + + --force-max-simd-bitwidth=512 + + The following example shows limiting the bitwidth to 64-bits to disable all vector code:: + + --force-max-simd-bitwidth=64 + + To disable use of max SIMD bitwidth limit:: + + --force-max-simd-bitwidth=0
doc/guides/prog_guide/env_abstraction_layer.rst +34 -0
@@ -486,6 +486,40 @@ the desired addressing mode when virtual devices that are not directly attached To facilitate forcing the IOVA mode to a specific value the EAL command line option ``--iova-mode`` can be used to select either physical addressing('pa') or virtual addressing('va'). +.. _max_simd_bitwidth: + + +Max SIMD bitwidth +~~~~~~~~~~~~~~~~~ + +The EAL provides a single setting to limit the max SIMD bitwidth used by DPDK, +which is used in determining the vector path, if any, chosen by a component. +The value can be set at runtime by an application using the +'rte_vect_set_max_simd_bitwidth(uint16_t bitwidth)' function, +which should only be called once at initialization, before EAL init. +The value can be overridden by the user using the EAL command-line option '--force-max-simd-bitwidth'. + +When choosing a vector path, along with checking the CPU feature support, +the value of the max SIMD bitwidth must also be checked, and can be retrieved using the +'rte_vect_get_max_simd_bitwidth()' function. +The value should be compared against the enum values for accepted max SIMD bitwidths: + +.. code-block:: c + + enum rte_vect_max_simd { + RTE_VECT_SIMD_DISABLED = 64, + RTE_VECT_SIMD_128 = 128, + RTE_VECT_SIMD_256 = 256, + RTE_VECT_SIMD_512 = 512, + RTE_VECT_SIMD_MAX = INT16_MAX + 1, + }; + + if (rte_vect_get_max_simd_bitwidth() >= RTE_VECT_SIMD_512) + /* Take AVX-512 vector path */ + else if (rte_vect_get_max_simd_bitwidth() >= RTE_VECT_SIMD_256) + /* Take AVX2 vector path */ + + Memory Segments and Memory Zones (memzone) ------------------------------------------
doc/guides/rel_notes/release_20_11.rst +6 -0
@@ -77,6 +77,12 @@ New Features This API is specific to x86 and implemented as a stub for other architectures. +* **Added support for limiting maximum SIMD bitwidth.** + + Added a new EAL config setting ``max_simd_bitwidth`` to limit the vector + path selection at runtime. This value can be set by apps using the + ``rte_vect_set_max_simd_bitwidth`` function, or by the user with EAL flag ``--force-max-simd-bitwidth``. + * **Updated CRC modules of the net library.** * Added runtime selection of the optimal architecture-specific CRC path.
lib/librte_eal/arm/include/rte_vect.h +2 -0
@@ -14,6 +14,8 @@extern "C" {#endif +#define RTE_VECT_DEFAULT_SIMD_BITWIDTH RTE_VECT_SIMD_MAX +typedef int32x4_t xmm_t ;#define XMM_SIZE (sizeof(xmm_t))
lib/librte_eal/common/eal_common_options.c +66 -0
108 lines not on this page · raw message
lib/librte_eal/common/eal_internal_cfg.h +8 -0
@@ -33,6 +33,12 @@ struct hugepage_info {int lock_descriptor;/**< file descriptor for hugepage dir */ }; +struct simd_bitwidth { +bool forced; +/**< flag indicating if bitwidth is forced and can't be modified */ +uint16_t bitwidth;/**< bitwidth value */ +}; +/** * internal configuration */
@@ -85,6 +91,8 @@ struct internal_config {volatile unsigned int init_complete;/**< indicates whether EAL has completed initialization */ unsigned int no_telemetry;/**< true to disable Telemetry */ +struct simd_bitwidth max_simd_bitwidth; +/**< max simd bitwidth path to use */ };void eal_reset_internal_config (struct internal_config *internal_cfg);
lib/librte_eal/common/eal_options.h +2 -0
@@ -85,6 +85,8 @@ enum {OPT_TELEMETRY_NUM , #define OPT_NO_TELEMETRY "no-telemetry"OPT_NO_TELEMETRY_NUM , +#define OPT_FORCE_MAX_SIMD_BITWIDTH "force-max-simd-bitwidth" +OPT_FORCE_MAX_SIMD_BITWIDTH_NUM , OPT_LONG_MAX_NUM };
lib/librte_eal/include/generic/rte_vect.h +51 -2
68 lines not on this page · raw message
lib/librte_eal/ppc/include/rte_vect.h +2 -0
@@ -15,6 +15,8 @@extern "C" {#endif +#define RTE_VECT_DEFAULT_SIMD_BITWIDTH RTE_VECT_SIMD_256 + typedef vector signed int xmm_t;#define XMM_SIZE (sizeof(xmm_t))
lib/librte_eal/rte_eal_exports.def +3 -0
@@ -303,6 +303,9 @@ EXPORTS rte_thread_register rte_thread_unregister + rte_vect_get_max_simd_bitwidth + rte_vect_set_max_simd_bitwidth + rte_mem_lock rte_mem_map rte_mem_page_size
lib/librte_eal/rte_eal_version.map +2 -0
@@ -400,6 +400,8 @@ EXPERIMENTAL { __rte_eal_trace_generic_size_t; rte_epoll_wait_interruptible; rte_service_lcore_may_be_active; + rte_vect_get_max_simd_bitwidth; + rte_vect_set_max_simd_bitwidth; }; INTERNAL {
lib/librte_eal/x86/include/rte_vect.h +2 -0
@@ -36,6 +36,8 @@extern "C" {#endif +#define RTE_VECT_DEFAULT_SIMD_BITWIDTH RTE_VECT_SIMD_256 +typedef __m128i xmm_t ;#define XMM_SIZE (sizeof(xmm_t))
Signature · 1 line
2.23.0
PATCH v10 2/18 doc: describe how to enable AVX512
In reply to David MarchandFrom: Ciara Power <[email protected]>
This patch adds documentation on the usage of the max SIMD bitwidth EAL setting to enable AVX-512 at runtime.
---
v6: Updated enum value.
v4: Updated docs to reflect renamed enum.
v3:
- Added enum value for disabling use of max SIMD to doc.
- Added entry to HowTo index.
--- doc/guides/howto/avx512.rst | 36 ++++++++++++++++++++++++++++++++++++ doc/guides/howto/index.rst | 1 + 2 files changed, 37 insertions(+) create mode 100644 doc/guides/howto/avx512.rst
doc/guides/howto/avx512.rst new file +36 -0
@@ -0,0 +1,36 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2020 Intel Corporation. + + +Using AVX-512 with DPDK +======================= + +AVX-512 is not used by default in DPDK, but it can be selected at runtime by apps through the use of EAL API, +and by the user with a commandline argument. DPDK has a setting for max SIMD bitwidth, +which can be modified and will then limit the vector path taken by the code. + + +Using the API in apps +--------------------- + +Apps can request DPDK uses AVX-512 at runtime, if it provides improved application performance. +This can be done by modifying the EAL setting for max SIMD bitwidth to 512, as by default it is 256, +which does not allow for AVX-512. + +.. code-block:: c + + rte_vect_set_max_simd_bitwidth(RTE_VECT_SIMD_512); + +This API should only be called once at initialization, before EAL init. +For more information on the possible enum values to use as a parameter, go to :ref:`max_simd_bitwidth`: + + +Using the command-line argument +--------------------------------------------- + +The user can select to use AVX-512 at runtime, using the following argument to set the max bitwidth:: + + ./app/dpdk-testpmd --force-max-simd-bitwidth=512 + +This will override any further changes to the max SIMD bitwidth in DPDK, +which is useful for testing purposes.
doc/guides/howto/index.rst +1 -0
@@ -20,3 +20,4 @@ HowTo Guides telemetry debug_troubleshoot openwrt + avx512
Signature · 1 line
2.23.0
PATCH v10 3/18 net/i40e: check max SIMD bitwidth
In reply to David MarchandFrom: Ciara Power <[email protected]>
When choosing a vector path to take, an extra condition must be satisfied to ensure the max SIMD bitwidth allows for the CPU enabled path.
---
v4: Updated enum names.
--- drivers/net/i40e/i40e_rxtx.c | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-)
drivers/net/i40e/i40e_rxtx.c +13 -6
@@ -23,6 +23,7 @@#include <rte_udp.h> #include <rte_ip.h> #include <rte_net.h> +#include <rte_vect.h> #include "i40e_logs.h" #include "base/i40e_prototype.h"
@@ -3098,7 +3099,8 @@ static eth_rx_burst_t i40e_get_latest_rx_vec(bool scatter) {#if defined(RTE_ARCH_X86 ) && defined(CC_AVX2_SUPPORT ) -if (rte_cpu_get_flag_enabled (RTE_CPUFLAG_AVX2 )) +if (rte_cpu_get_flag_enabled (RTE_CPUFLAG_AVX2 ) && +rte_vect_get_max_simd_bitwidth () >=RTE_VECT_SIMD_256 )return scatter ? i40e_recv_scattered_pkts_vec_avx2 : i40e_recv_pkts_vec_avx2;#endif
@@ -3115,7 +3117,8 @@ i40e_get_recommend_rx_vec(bool scatter)* use of AVX2 version to later plaforms, not all those that could * theoretically run it. */ -if (rte_cpu_get_flag_enabled (RTE_CPUFLAG_AVX512F )) +if (rte_cpu_get_flag_enabled (RTE_CPUFLAG_AVX512F ) && +rte_vect_get_max_simd_bitwidth () >=RTE_VECT_SIMD_256 )return scatter ? i40e_recv_scattered_pkts_vec_avx2 : i40e_recv_pkts_vec_avx2;#endif
@@ -3154,7 +3157,8 @@ i40e_set_rx_function(struct rte_eth_dev *dev) } } -if (ad->rx_vec_allowed) { +if (ad->rx_vec_allowed && +rte_vect_get_max_simd_bitwidth () >=RTE_VECT_SIMD_128 ) {/* Vec Rx path */ PMD_INIT_LOG (DEBUG ,"Vector Rx path will be used on port=%d." , dev->data->port_id);
@@ -3268,7 +3272,8 @@ static eth_tx_burst_ti40e_get_latest_tx_vec (void) {#if defined(RTE_ARCH_X86 ) && defined(CC_AVX2_SUPPORT ) -if (rte_cpu_get_flag_enabled (RTE_CPUFLAG_AVX2 )) +if (rte_cpu_get_flag_enabled (RTE_CPUFLAG_AVX2 ) && +rte_vect_get_max_simd_bitwidth () >=RTE_VECT_SIMD_256 )return i40e_xmit_pkts_vec_avx2;#endif return i40e_xmit_pkts_vec;
@@ -3283,7 +3288,8 @@ i40e_get_recommend_tx_vec(void)* use of AVX2 version to later plaforms, not all those that could * theoretically run it. */ -if (rte_cpu_get_flag_enabled (RTE_CPUFLAG_AVX512F )) +if (rte_cpu_get_flag_enabled (RTE_CPUFLAG_AVX512F ) && +rte_vect_get_max_simd_bitwidth () >=RTE_VECT_SIMD_256 )return i40e_xmit_pkts_vec_avx2;#endif return i40e_xmit_pkts_vec;
@@ -3311,7 +3317,8 @@ i40e_set_tx_function(struct rte_eth_dev *dev) }if (ad->tx_simple_allowed) { -if (ad->tx_vec_allowed) { +if (ad->tx_vec_allowed && +rte_vect_get_max_simd_bitwidth () >=RTE_VECT_SIMD_128 ) {PMD_INIT_LOG (DEBUG ,"Vector tx finally be used." );if (ad->use_latest_vec) dev->tx_pkt_burst =
Signature · 1 line
2.23.0
PATCH v10 4/18 net/axgbe: check max SIMD bitwidth
In reply to David MarchandFrom: Ciara Power <[email protected]>
When choosing a vector path to take, an extra condition must be satisfied to ensure the max SIMD bitwidth allows for the CPU enabled path.
---
v4: Updated enum name.
--- drivers/net/axgbe/axgbe_rxtx.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/net/axgbe/axgbe_rxtx.c +3 -1
@@ -10,6 +10,7 @@#include <rte_time.h> #include <rte_mempool.h> #include <rte_mbuf.h> +#include <rte_vect.h> static void axgbe_rx_queue_release (struct axgbe_rx_queue *rx_queue)
@@ -557,7 +558,8 @@ int axgbe_dev_tx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_idx,if (!pdata->tx_queues) pdata->tx_queues = dev->data->tx_queues; -if (txq->vector_disable) +if (txq->vector_disable || +rte_vect_get_max_simd_bitwidth () <RTE_VECT_SIMD_128 ) dev->tx_pkt_burst = &axgbe_xmit_pkts; else#ifdef RTE_ARCH_X86
Signature · 1 line
2.23.0
PATCH v10 5/18 net/bnxt: check max SIMD bitwidth
In reply to David MarchandFrom: Ciara Power <[email protected]>
When choosing a vector path to take, an extra condition must be satisfied to ensure the max SIMD bitwidth allows for the CPU enabled path.
---
v4: Updated enum name.
--- drivers/net/bnxt/bnxt_ethdev.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)
drivers/net/bnxt/bnxt_ethdev.c +5 -2
@@ -13,6 +13,7 @@#include <rte_cycles.h> #include <rte_alarm.h> #include <rte_kvargs.h> +#include <rte_vect.h> #include "bnxt.h" #include "bnxt_filter.h"
@@ -1169,7 +1170,8 @@ bnxt_receive_function(struct rte_eth_dev *eth_dev) DEV_RX_OFFLOAD_OUTER_IPV4_CKSUM | DEV_RX_OFFLOAD_RSS_HASH | DEV_RX_OFFLOAD_VLAN_FILTER)) && - !BNXT_TRUFLOW_EN(bp) && BNXT_NUM_ASYNC_CPR(bp)) { + !BNXT_TRUFLOW_EN(bp) && BNXT_NUM_ASYNC_CPR(bp) && + rte_vect_get_max_simd_bitwidth() >= RTE_VECT_SIMD_128) {PMD_DRV_LOG (INFO ,"Using vector mode receive for port %d\n" , eth_dev->data->port_id); bp->flags |=BNXT_FLAG_RX_VECTOR_PKT_MODE ;
@@ -1202,7 +1204,8 @@ bnxt_transmit_function(__rte_unused struct rte_eth_dev *eth_dev)*/ if (!eth_dev->data->scattered_rx && !(offloads & ~DEV_TX_OFFLOAD_MBUF_FAST_FREE ) && - !BNXT_TRUFLOW_EN (bp)) { + !BNXT_TRUFLOW_EN (bp) && +rte_vect_get_max_simd_bitwidth () >=RTE_VECT_SIMD_128 ) {PMD_DRV_LOG (INFO ,"Using vector mode transmit for port %d\n" , eth_dev->data->port_id);return bnxt_xmit_pkts_vec;
Signature · 1 line
2.23.0
PATCH v10 6/18 net/enic: check max SIMD bitwidth
In reply to David MarchandFrom: Ciara Power <[email protected]>
When choosing a vector path to take, an extra condition must be satisfied to ensure the max SIMD bitwidth allows for the CPU enabled path.
---
v4: Updated enum name.
--- drivers/net/enic/enic_rxtx_vec_avx2.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/net/enic/enic_rxtx_vec_avx2.c +3 -1
@@ -5,6 +5,7 @@#include <rte_mbuf.h> #include <rte_ethdev_driver.h> +#include <rte_vect.h> #include "enic_compat.h" #include "rq_enet_desc.h"
@@ -821,7 +822,8 @@ enic_use_vector_rx_handler(struct rte_eth_dev *eth_dev) fconf = ð_dev->data->dev_conf.fdir_conf;if (fconf->mode !=RTE_FDIR_MODE_NONE )return false; -if (rte_cpu_get_flag_enabled (RTE_CPUFLAG_AVX2 )) { +if (rte_cpu_get_flag_enabled (RTE_CPUFLAG_AVX2 ) && +rte_vect_get_max_simd_bitwidth () >=RTE_VECT_SIMD_256 ) {ENICPMD_LOG (DEBUG ," use the non-scatter avx2 Rx handler" ); eth_dev->rx_pkt_burst = &enic_noscatter_vec_recv_pkts; enic->use_noscatter_vec_rx_handler =1 ;
Signature · 1 line
2.23.0
PATCH v10 7/18 net/fm10k: check max SIMD bitwidth
In reply to David MarchandFrom: Ciara Power <[email protected]>
When choosing a vector path to take, an extra condition must be satisfied to ensure the max SIMD bitwidth allows for the CPU enabled path.
---
v4: Updated enum name.
--- drivers/net/fm10k/fm10k_ethdev.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-)
drivers/net/fm10k/fm10k_ethdev.c +7 -3
@@ -10,6 +10,7 @@#include <rte_dev.h> #include <rte_spinlock.h> #include <rte_kvargs.h> +#include <rte_vect.h> #include "fm10k.h" #include "base/fm10k_api.h"
@@ -2939,7 +2940,8 @@ fm10k_set_tx_function(struct rte_eth_dev *dev)if (rte_eal_process_type () !=RTE_PROC_PRIMARY ) {/* primary process has set the ftag flag and offloads */ txq = dev->data->tx_queues[0 ]; -if (fm10k_tx_vec_condition_check (txq)) { +if (fm10k_tx_vec_condition_check (txq) || +rte_vect_get_max_simd_bitwidth () <RTE_VECT_SIMD_128 ) { dev->tx_pkt_burst = fm10k_xmit_pkts; dev->tx_pkt_prepare = fm10k_prep_pkts;PMD_INIT_LOG (DEBUG ,"Use regular Tx func" );
@@ -2958,7 +2960,8 @@ fm10k_set_tx_function(struct rte_eth_dev *dev) txq = dev->data->tx_queues[i]; txq->tx_ftag_en = tx_ftag_en;/* Check if Vector Tx is satisfied */ -if (fm10k_tx_vec_condition_check (txq)) +if (fm10k_tx_vec_condition_check (txq) || +rte_vect_get_max_simd_bitwidth () <RTE_VECT_SIMD_128 ) use_sse =0 ; }
@@ -2992,7 +2995,8 @@ fm10k_set_rx_function(struct rte_eth_dev *dev)* conditions to be met. */ if (!fm10k_rx_vec_condition_check (dev) && - dev_info->rx_vec_allowed && !rx_ftag_en) { + dev_info->rx_vec_allowed && !rx_ftag_en && +rte_vect_get_max_simd_bitwidth () >=RTE_VECT_SIMD_128 ) {if (dev->data->scattered_rx) dev->rx_pkt_burst = fm10k_recv_scattered_pkts_vec;else
Signature · 1 line
2.23.0
PATCH v10 8/18 net/iavf: check max SIMD bitwidth
In reply to David MarchandFrom: Ciara Power <[email protected]>
When choosing a vector path to take, an extra condition must be satisfied to ensure the max SIMD bitwidth allows for the CPU enabled path.
---
v4: Updated enum name.
--- drivers/net/iavf/iavf_rxtx.c | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-)
drivers/net/iavf/iavf_rxtx.c +11 -6
@@ -23,6 +23,7 @@#include <rte_udp.h> #include <rte_ip.h> #include <rte_net.h> +#include <rte_vect.h> #include "iavf.h" #include "iavf_rxtx.h"
@@ -2104,14 +2105,16 @@ iavf_set_rx_function(struct rte_eth_dev *dev)int i;bool use_avx2 = false; -if (!iavf_rx_vec_dev_check (dev)) { +if (!iavf_rx_vec_dev_check (dev) && +rte_vect_get_max_simd_bitwidth () >=RTE_VECT_SIMD_128 ) {for (i =0 ; i < dev->data->nb_rx_queues; i++) { rxq = dev->data->rx_queues[i]; (void )iavf_rxq_vec_setup (rxq); } -if (rte_cpu_get_flag_enabled (RTE_CPUFLAG_AVX2 ) ==1 || -rte_cpu_get_flag_enabled (RTE_CPUFLAG_AVX512F ) ==1 ) +if ((rte_cpu_get_flag_enabled (RTE_CPUFLAG_AVX2 ) ==1 || +rte_cpu_get_flag_enabled (RTE_CPUFLAG_AVX512F ) ==1 ) && +rte_vect_get_max_simd_bitwidth () >=RTE_VECT_SIMD_256 ) use_avx2 = true;if (dev->data->scattered_rx) {
@@ -2177,7 +2180,8 @@ iavf_set_tx_function(struct rte_eth_dev *dev)int i;bool use_avx2 = false; -if (!iavf_tx_vec_dev_check (dev)) { +if (!iavf_tx_vec_dev_check (dev) && +rte_vect_get_max_simd_bitwidth () >=RTE_VECT_SIMD_128 ) {for (i =0 ; i < dev->data->nb_tx_queues; i++) { txq = dev->data->tx_queues[i];if (!txq)
@@ -2185,8 +2189,9 @@ iavf_set_tx_function(struct rte_eth_dev *dev)iavf_txq_vec_setup (txq); } -if (rte_cpu_get_flag_enabled (RTE_CPUFLAG_AVX2 ) ==1 || -rte_cpu_get_flag_enabled (RTE_CPUFLAG_AVX512F ) ==1 ) +if ((rte_cpu_get_flag_enabled (RTE_CPUFLAG_AVX2 ) ==1 || +rte_cpu_get_flag_enabled (RTE_CPUFLAG_AVX512F ) ==1 ) && +rte_vect_get_max_simd_bitwidth () >=RTE_VECT_SIMD_256 ) use_avx2 = true; PMD_DRV_LOG(DEBUG, "Using %sVector Tx (port %d).",
Signature · 1 line
2.23.0
PATCH v10 9/18 net/ice: check max SIMD bitwidth
In reply to David MarchandFrom: Ciara Power <[email protected]>
When choosing a vector path to take, an extra condition must be satisfied to ensure the max SIMD bitwidth allows for the CPU enabled path.
---
v4: Updated enum name.
--- drivers/net/ice/ice_rxtx.c | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-)
drivers/net/ice/ice_rxtx.c +11 -6
@@ -4,6 +4,7 @@#include <rte_ethdev_driver.h> #include <rte_net.h> +#include <rte_vect.h> #include "rte_pmd_ice.h" #include "ice_rxtx.h"
@@ -2989,7 +2990,8 @@ ice_set_rx_function(struct rte_eth_dev *dev)bool use_avx2 = false;if (rte_eal_process_type () ==RTE_PROC_PRIMARY ) { -if (!ice_rx_vec_dev_check (dev) && ad->rx_bulk_alloc_allowed) { +if (!ice_rx_vec_dev_check (dev) && ad->rx_bulk_alloc_allowed && +rte_vect_get_max_simd_bitwidth () >=RTE_VECT_SIMD_128 ) { ad->rx_vec_allowed = true;for (i =0 ; i < dev->data->nb_rx_queues; i++) { rxq = dev->data->rx_queues[i];
@@ -2999,8 +3001,9 @@ ice_set_rx_function(struct rte_eth_dev *dev) } } -if (rte_cpu_get_flag_enabled (RTE_CPUFLAG_AVX2 ) ==1 || -rte_cpu_get_flag_enabled (RTE_CPUFLAG_AVX512F ) ==1 ) +if ((rte_cpu_get_flag_enabled (RTE_CPUFLAG_AVX2 ) ==1 || +rte_cpu_get_flag_enabled (RTE_CPUFLAG_AVX512F ) ==1 ) && +rte_vect_get_max_simd_bitwidth () >=RTE_VECT_SIMD_256 ) use_avx2 = true; }else {
@@ -3167,7 +3170,8 @@ ice_set_tx_function(struct rte_eth_dev *dev)bool use_avx2 = false;if (rte_eal_process_type () ==RTE_PROC_PRIMARY ) { -if (!ice_tx_vec_dev_check (dev)) { +if (!ice_tx_vec_dev_check (dev) && +rte_vect_get_max_simd_bitwidth () >=RTE_VECT_SIMD_128 ) { ad->tx_vec_allowed = true;for (i =0 ; i < dev->data->nb_tx_queues; i++) { txq = dev->data->tx_queues[i];
@@ -3177,8 +3181,9 @@ ice_set_tx_function(struct rte_eth_dev *dev) } } -if (rte_cpu_get_flag_enabled (RTE_CPUFLAG_AVX2 ) ==1 || -rte_cpu_get_flag_enabled (RTE_CPUFLAG_AVX512F ) ==1 ) +if ((rte_cpu_get_flag_enabled (RTE_CPUFLAG_AVX2 ) ==1 || +rte_cpu_get_flag_enabled (RTE_CPUFLAG_AVX512F ) ==1 ) && +rte_vect_get_max_simd_bitwidth () >=RTE_VECT_SIMD_256 ) use_avx2 = true; }else {
Signature · 1 line
2.23.0
PATCH v10 10/18 net/ixgbe: check max SIMD bitwidth
In reply to David MarchandFrom: Ciara Power <[email protected]>
When choosing a vector path to take, an extra condition must be satisfied to ensure the max SIMD bitwidth allows for the CPU enabled path.
---
v4:
- Updated enum name.
- Moved placement of condition check.
- Added condition check to tx cleanup path selection.
--- drivers/net/ixgbe/ixgbe_rxtx.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
drivers/net/ixgbe/ixgbe_rxtx.c +5 -1
@@ -42,6 +42,7 @@#include <rte_errno.h> #include <rte_ip.h> #include <rte_net.h> +#include <rte_vect.h> #include "ixgbe_logs.h" #include "base/ixgbe_api.h"
@@ -2405,6 +2406,7 @@ ixgbe_dev_tx_done_cleanup(void *tx_queue, uint32_t free_cnt)#endif txq->tx_rs_thresh >= RTE_PMD_IXGBE_TX_MAX_BURST) {if (txq->tx_rs_thresh <=RTE_IXGBE_TX_MAX_FREE_BUF_SZ && +rte_vect_get_max_simd_bitwidth () >=RTE_VECT_SIMD_128 && (rte_eal_process_type () !=RTE_PROC_PRIMARY || txq->sw_ring_v !=NULL )) {return ixgbe_tx_done_cleanup_vec (txq, free_cnt);
@@ -2503,6 +2505,7 @@ ixgbe_set_tx_function(struct rte_eth_dev *dev, struct ixgbe_tx_queue *txq)PMD_INIT_LOG (DEBUG ,"Using simple tx code path" ); dev->tx_pkt_prepare =NULL ;if (txq->tx_rs_thresh <=RTE_IXGBE_TX_MAX_FREE_BUF_SZ && +rte_vect_get_max_simd_bitwidth () >=RTE_VECT_SIMD_128 && (rte_eal_process_type () !=RTE_PROC_PRIMARY ||ixgbe_txq_vec_setup (txq) ==0 )) {PMD_INIT_LOG (DEBUG ,"Vector tx enabled." );
@@ -4744,7 +4747,8 @@ ixgbe_set_rx_function(struct rte_eth_dev *dev)* conditions to be met and Rx Bulk Allocation should be allowed. */ if (ixgbe_rx_vec_dev_conf_condition_check (dev) || - !adapter->rx_bulk_alloc_allowed) { + !adapter->rx_bulk_alloc_allowed || +rte_vect_get_max_simd_bitwidth () <RTE_VECT_SIMD_128 ) {PMD_INIT_LOG (DEBUG ,"Port[%d] doesn't meet Vector Rx " "preconditions" , dev->data->port_id);
Signature · 1 line
2.23.0
PATCH v10 11/18 net/mlx5: check max SIMD bitwidth
In reply to David MarchandFrom: Ciara Power <[email protected]>
When choosing a vector path to take, an extra condition must be satisfied to ensure the max SIMD bitwidth allows for the CPU enabled path.
---
v4: Updated enum name.
v2: Moved check for max bitwidth into existing check vec
support function.
--- drivers/net/mlx5/mlx5_rxtx_vec.c | 3 +++ 1 file changed, 3 insertions(+)
drivers/net/mlx5/mlx5_rxtx_vec.c +3 -0
@@ -10,6 +10,7 @@#include <rte_mbuf.h> #include <rte_mempool.h> #include <rte_prefetch.h> +#include <rte_vect.h> #include <mlx5_glue.h> #include <mlx5_prm.h>
@@ -148,6 +149,8 @@ mlx5_check_vec_rx_support(struct rte_eth_dev *dev)struct mlx5_priv *priv = dev->data->dev_private;uint32_t i; +if (rte_vect_get_max_simd_bitwidth () <RTE_VECT_SIMD_128 ) +return -ENOTSUP ;if (!priv->config.rx_vec_en)return -ENOTSUP ;if (mlx5_mprq_enabled (dev))
Signature · 1 line
2.23.0
PATCH v10 12/18 net/virtio: check max SIMD bitwidth
In reply to David MarchandFrom: Ciara Power <[email protected]>
When choosing a vector path to take, an extra condition must be satisfied to ensure the max SIMD bitwidth allows for the CPU enabled path.
---
v4: Updated enum name.
v3: Moved max SIMD bitwidth check to configure function with other vec
support checks.
--- drivers/net/virtio/virtio_ethdev.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)
drivers/net/virtio/virtio_ethdev.c +9 -1
@@ -23,6 +23,7 @@#include <rte_common.h> #include <rte_errno.h> #include <rte_cpuflags.h> +#include <rte_vect.h> #include <rte_memory.h> #include <rte_eal.h>
@@ -2312,7 +2313,8 @@ virtio_dev_configure(struct rte_eth_dev *dev)if ((hw->use_vec_rx || hw->use_vec_tx) && (!rte_cpu_get_flag_enabled (RTE_CPUFLAG_AVX512F ) || !vtpci_with_feature (hw,VIRTIO_F_IN_ORDER ) || - !vtpci_with_feature (hw,VIRTIO_F_VERSION_1 ))) { + !vtpci_with_feature (hw,VIRTIO_F_VERSION_1 ) || +rte_vect_get_max_simd_bitwidth () <RTE_VECT_SIMD_512 )) {PMD_DRV_LOG (INFO ,"disabled packed ring vectorized path for requirements not met" ); hw->use_vec_rx =0 ;
@@ -2365,6 +2367,12 @@ virtio_dev_configure(struct rte_eth_dev *dev) "disabled split ring vectorized rx for offloading enabled"); hw->use_vec_rx =0 ; } + +if (rte_vect_get_max_simd_bitwidth () <RTE_VECT_SIMD_128 ) { +PMD_DRV_LOG (INFO , +"disabled split ring vectorized rx, max SIMD bitwidth too low" ); + hw->use_vec_rx =0 ; + } } }
Signature · 1 line
2.23.0
PATCH v10 13/18 distributor: check max SIMD bitwidth
In reply to David MarchandFrom: Ciara Power <[email protected]>
When choosing a vector path to take, an extra condition must be satisfied to ensure the max SIMD bitwidth allows for the CPU enabled path.
---
v4: Updated enum name.
v3: Moved max SIMD bitwidth check to configure function with other vec
support checks.
--- lib/librte_distributor/rte_distributor.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
lib/librte_distributor/rte_distributor.c +3 -1
@@ -14,6 +14,7 @@#include <rte_eal_memconfig.h> #include <rte_pause.h> #include <rte_tailq.h> +#include <rte_vect.h> #include "rte_distributor.h" #include "rte_distributor_single.h"
@@ -762,7 +763,8 @@ rte_distributor_create(const char *name, d->dist_match_fn =RTE_DIST_MATCH_SCALAR ;#if defined(RTE_ARCH_X86 ) - d->dist_match_fn =RTE_DIST_MATCH_VECTOR ; +if (rte_vect_get_max_simd_bitwidth () >=RTE_VECT_SIMD_128 ) + d->dist_match_fn =RTE_DIST_MATCH_VECTOR ;#endif /*
Signature · 1 line
2.23.0
PATCH v10 14/18 member: check max SIMD bitwidth
In reply to David MarchandFrom: Ciara Power <[email protected]>
When choosing a vector path to take, an extra condition must be satisfied to ensure the max SIMD bitwidth allows for the CPU enabled path.
---
v4: Updated enum name.
--- lib/librte_member/rte_member_ht.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
lib/librte_member/rte_member_ht.c +3 -1
@@ -7,6 +7,7 @@#include <rte_prefetch.h> #include <rte_random.h> #include <rte_log.h> +#include <rte_vect.h> #include "rte_member.h" #include "rte_member_ht.h"
@@ -113,7 +114,8 @@ rte_member_create_ht(struct rte_member_setsum *ss, }#if defined(RTE_ARCH_X86 )if (rte_cpu_get_flag_enabled (RTE_CPUFLAG_AVX2 ) && -RTE_MEMBER_BUCKET_ENTRIES ==16 ) +RTE_MEMBER_BUCKET_ENTRIES ==16 && +rte_vect_get_max_simd_bitwidth () >=RTE_VECT_SIMD_256 ) ss->sig_cmp_fn =RTE_MEMBER_COMPARE_AVX2 ; else#endif
Signature · 1 line
2.23.0
PATCH v10 15/18 efd: check max SIMD bitwidth
In reply to David MarchandFrom: Ciara Power <[email protected]>
When choosing a vector path to take, an extra condition must be satisfied to ensure the max SIMD bitwidth allows for the CPU enabled path.
---
v4: Updated enum name.
--- lib/librte_efd/rte_efd.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
lib/librte_efd/rte_efd.c +6 -2
@@ -21,6 +21,7 @@#include <rte_jhash.h> #include <rte_hash_crc.h> #include <rte_tailq.h> +#include <rte_vect.h> #include "rte_efd.h" #if defined(RTE_ARCH_X86 )
@@ -645,7 +646,9 @@ rte_efd_create(const char *name, uint32_t max_num_rules, uint32_t key_len,* For less than 4 bits, scalar function performs better * than vectorised version */ -if (RTE_EFD_VALUE_NUM_BITS >3 &&rte_cpu_get_flag_enabled (RTE_CPUFLAG_AVX2 )) +if (RTE_EFD_VALUE_NUM_BITS >3 + &&rte_cpu_get_flag_enabled (RTE_CPUFLAG_AVX2 ) + &&rte_vect_get_max_simd_bitwidth () >=RTE_VECT_SIMD_256 ) table->lookup_fn =EFD_LOOKUP_AVX2 ; else#endif
@@ -655,7 +658,8 @@ rte_efd_create(const char *name, uint32_t max_num_rules, uint32_t key_len,* than vectorised version */ if (RTE_EFD_VALUE_NUM_BITS >16 && -rte_cpu_get_flag_enabled (RTE_CPUFLAG_NEON )) +rte_cpu_get_flag_enabled (RTE_CPUFLAG_NEON ) && +rte_vect_get_max_simd_bitwidth () >=RTE_VECT_SIMD_128 ) table->lookup_fn =EFD_LOOKUP_NEON ; else#endif
Signature · 1 line
2.23.0
PATCH v10 16/18 net: check max SIMD bitwidth
In reply to David MarchandFrom: Ciara Power <[email protected]>
When choosing a vector path to take, an extra condition must be satisfied to ensure the max SIMD bitwidth allows for the CPU enabled path.
The vector path was initially chosen in RTE_INIT, however this is no longer suitable as we cannot check the max SIMD bitwidth at that time. Default handlers are now chosen on initialisation, these default handlers are used the first time the crc calc is called, and they set the suitable handlers to be used going forward.
---
v7: Removed unnecessary log variable.
v6:
- Moved log variable and macro to c file instead of public header.
- Added the max_simd_bitwidth condition check to the recently added handler helper functions.
- Modified default handlers to follow the approach of the set alg function.
v4:
- Added default handlers to be set at RTE_INIT time, rather than choosing scalar handlers.
- Modified logging.
- Updated enum name.
v3:
- Moved choosing vector paths out of RTE_INIT.
- Moved checking max_simd_bitwidth into the set_alg function.
--- lib/librte_net/rte_net_crc.c | 117 +++++++++++++++++++++++++---------- 1 file changed, 86 insertions(+), 31 deletions(-)
lib/librte_net/rte_net_crc.c +86 -31
@@ -9,6 +9,9 @@#include <rte_cpuflags.h> #include <rte_common.h> #include <rte_net_crc.h> +#include <rte_eal.h> +#include <rte_log.h> +#include <rte_vect.h> #include "net_crc.h"
@@ -22,6 +25,12 @@static uint32_t crc32_eth_lut[CRC_LUT_SIZE ];static uint32_t crc16_ccitt_lut[CRC_LUT_SIZE ]; +static uint32_t +rte_crc16_ccitt_default_handler (const uint8_t *data,uint32_t data_len); + +static uint32_t +rte_crc32_eth_default_handler (const uint8_t *data,uint32_t data_len); +static uint32_t rte_crc16_ccitt_handler (const uint8_t *data,uint32_t data_len);
@@ -31,7 +40,12 @@ rte_crc32_eth_handler(const uint8_t *data, uint32_t data_len);typedef uint32_t (*rte_net_crc_handler )(const uint8_t *data,uint32_t data_len); -static const rte_net_crc_handler *handlers; +static rte_net_crc_handler handlers_default[] = { + [RTE_NET_CRC16_CCITT ] = rte_crc16_ccitt_default_handler, + [RTE_NET_CRC32_ETH ] = rte_crc32_eth_default_handler, +}; + +static const rte_net_crc_handler *handlers = handlers_default;static const rte_net_crc_handler handlers_scalar[] = { [RTE_NET_CRC16_CCITT ] = rte_crc16_ccitt_handler,
@@ -56,6 +70,14 @@ static const rte_net_crc_handler handlers_neon[] = { };#endif +static uint16_t max_simd_bitwidth; + +#define NET_LOG(level, fmt, args...) \ + rte_log(RTE_LOG_ ## level, libnet_logtype, "%s(): " fmt "\n", \ + __func__, ## args) + +RTE_LOG_REGISTER (libnet_logtype, lib.net,INFO ); +/* Scalar handling */ /**
@@ -155,22 +177,21 @@ static const rte_net_crc_handler *avx512_vpclmulqdq_get_handlers (void) {#ifdef CC_X86_64_AVX512_VPCLMULQDQ_SUPPORT -if (AVX512_VPCLMULQDQ_CPU_SUPPORTED ) +if (AVX512_VPCLMULQDQ_CPU_SUPPORTED && + max_simd_bitwidth >=RTE_VECT_SIMD_512 )return handlers_avx512;#endif +NET_LOG (INFO ,"Requirements not met, can't use AVX512\n" );return NULL ; } -static uint8_t +static void avx512_vpclmulqdq_init (void ) {#ifdef CC_X86_64_AVX512_VPCLMULQDQ_SUPPORT -if (AVX512_VPCLMULQDQ_CPU_SUPPORTED ) { +if (AVX512_VPCLMULQDQ_CPU_SUPPORTED )rte_net_crc_avx512_init (); -return 1 ; - }#endif -return 0 ; }/* SSE4.2/PCLMULQDQ handling */
@@ -182,22 +203,21 @@ static const rte_net_crc_handler *sse42_pclmulqdq_get_handlers (void) {#ifdef CC_X86_64_SSE42_PCLMULQDQ_SUPPORT -if (SSE42_PCLMULQDQ_CPU_SUPPORTED ) +if (SSE42_PCLMULQDQ_CPU_SUPPORTED && + max_simd_bitwidth >=RTE_VECT_SIMD_128 )return handlers_sse42;#endif +NET_LOG (INFO ,"Requirements not met, can't use SSE\n" );return NULL ; } -static uint8_t +static void sse42_pclmulqdq_init (void ) {#ifdef CC_X86_64_SSE42_PCLMULQDQ_SUPPORT -if (SSE42_PCLMULQDQ_CPU_SUPPORTED ) { +if (SSE42_PCLMULQDQ_CPU_SUPPORTED )rte_net_crc_sse42_init (); -return 1 ; - }#endif -return 0 ; }/* NEON/PMULL handling */
@@ -209,22 +229,63 @@ static const rte_net_crc_handler *neon_pmull_get_handlers (void) {#ifdef CC_ARM64_NEON_PMULL_SUPPORT -if (NEON_PMULL_CPU_SUPPORTED ) +if (NEON_PMULL_CPU_SUPPORTED && + max_simd_bitwidth >=RTE_VECT_SIMD_128 )return handlers_neon;#endif +NET_LOG (INFO ,"Requirements not met, can't use NEON\n" );return NULL ; } -static uint8_t +static void neon_pmull_init (void ) {#ifdef CC_ARM64_NEON_PMULL_SUPPORT -if (NEON_PMULL_CPU_SUPPORTED ) { +if (NEON_PMULL_CPU_SUPPORTED )rte_net_crc_neon_init (); -return 1 ; - }#endif -return 0 ; +} + +/* Default handling */ + +static uint32_t +rte_crc16_ccitt_default_handler (const uint8_t *data,uint32_t data_len) +{ + handlers =NULL ; +if (max_simd_bitwidth ==0 ) + max_simd_bitwidth =rte_vect_get_max_simd_bitwidth (); + + handlers =avx512_vpclmulqdq_get_handlers (); +if (handlers !=NULL ) +return handlers[RTE_NET_CRC16_CCITT ](data, data_len); + handlers =sse42_pclmulqdq_get_handlers (); +if (handlers !=NULL ) +return handlers[RTE_NET_CRC16_CCITT ](data, data_len); + handlers =neon_pmull_get_handlers (); +if (handlers !=NULL ) +return handlers[RTE_NET_CRC16_CCITT ](data, data_len); + handlers = handlers_scalar; +return handlers[RTE_NET_CRC16_CCITT ](data, data_len); +} + +static uint32_t +rte_crc32_eth_default_handler (const uint8_t *data,uint32_t data_len) +{ + handlers =NULL ; +if (max_simd_bitwidth ==0 ) + max_simd_bitwidth =rte_vect_get_max_simd_bitwidth (); + + handlers =avx512_vpclmulqdq_get_handlers (); +if (handlers !=NULL ) +return handlers[RTE_NET_CRC32_ETH ](data, data_len); + handlers =sse42_pclmulqdq_get_handlers (); +if (handlers !=NULL ) +return handlers[RTE_NET_CRC32_ETH ](data, data_len); + handlers =neon_pmull_get_handlers (); +if (handlers !=NULL ) +return handlers[RTE_NET_CRC32_ETH ](data, data_len); + handlers = handlers_scalar; +return handlers[RTE_NET_CRC32_ETH ](data, data_len); }/* Public API */
@@ -233,6 +294,8 @@ void rte_net_crc_set_alg(enum rte_net_crc_alg alg) { handlers =NULL ; +if (max_simd_bitwidth ==0 ) + max_simd_bitwidth =rte_vect_get_max_simd_bitwidth ();switch (alg) {case RTE_NET_CRC_AVX512 :
@@ -270,19 +333,11 @@ rte_net_crc_calc(const void *data,return ret; } -/* Select highest available crc algorithm as default one */ +/* Call initialisation helpers for all crc algorithm handlers */ RTE_INIT (rte_net_crc_init) { -enum rte_net_crc_alg alg =RTE_NET_CRC_SCALAR ; -rte_net_crc_scalar_init (); - -if (sse42_pclmulqdq_init ()) - alg =RTE_NET_CRC_SSE42 ; -if (avx512_vpclmulqdq_init ()) - alg =RTE_NET_CRC_AVX512 ; -if (neon_pmull_init ()) - alg =RTE_NET_CRC_NEON ; - -rte_net_crc_set_alg (alg); +sse42_pclmulqdq_init (); +avx512_vpclmulqdq_init (); +neon_pmull_init (); }
Signature · 1 line
2.23.0
PATCH v10 17/18 node: choose vector path at runtime
In reply to David MarchandFrom: Ciara Power <[email protected]>
When choosing the vector path, max SIMD bitwidth is now checked to ensure the vector path is suitable. To do this, the scalar function is chosen by default in the struct, but at node initialisation time, this function pointer is updated to the vector version if supported, and if it is within the max SIMD bitwidth limit.
---
v6:
- Removed generic process function.
- Change the process function pointer at node init time to vector function if suitable.
--- lib/librte_node/ip4_lookup.c | 15 ++++++++++----- lib/librte_node/ip4_lookup_neon.h | 2 +- lib/librte_node/ip4_lookup_sse.h | 2 +- 3 files changed, 12 insertions(+), 7 deletions(-)
lib/librte_node/ip4_lookup.c +10 -5
@@ -15,6 +15,7 @@#include <rte_mbuf.h> #include <rte_tcp.h> #include <rte_udp.h> +#include <rte_vect.h> #include "rte_node_ip4_api.h"
@@ -34,10 +35,10 @@ static struct ip4_lookup_node_main ip4_lookup_nm;#include "ip4_lookup_neon.h" #elif defined(RTE_ARCH_X86 )#include "ip4_lookup_sse.h" -#else +#endif static uint16_t -ip4_lookup_node_process (struct rte_graph *graph,struct rte_node *node, +ip4_lookup_node_process_scalar (struct rte_graph *graph,struct rte_node *node,void **objs,uint16_t nb_objs) {struct rte_ipv4_hdr *ipv4_hdr;
@@ -109,8 +110,6 @@ ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,return nb_objs; } -#endif -int rte_node_ip4_route_add (uint32_t ip,uint8_t depth,uint16_t next_hop,enum rte_node_ip4_lookup_next next_node)
@@ -194,13 +193,19 @@ ip4_lookup_node_init(const struct rte_graph *graph, struct rte_node *node) init_once =1 ; } *lpm_p = ip4_lookup_nm.lpm_tbl[graph->socket]; + +#if defined(__ARM_NEON) || defined(RTE_ARCH_X86 ) +if (rte_vect_get_max_simd_bitwidth () >=RTE_VECT_SIMD_128 ) + node->process = ip4_lookup_node_process_vec; +#endif +node_dbg ("ip4_lookup" ,"Initialized ip4_lookup node" );return 0 ; }static struct rte_node_register ip4_lookup_node = { - .process = ip4_lookup_node_process, + .process = ip4_lookup_node_process_scalar, .name ="ip4_lookup" , .init = ip4_lookup_node_init,
lib/librte_node/ip4_lookup_neon.h +1 -1
@@ -7,7 +7,7 @@/* ARM64 NEON */ static uint16_t -ip4_lookup_node_process (struct rte_graph *graph,struct rte_node *node, +ip4_lookup_node_process_vec (struct rte_graph *graph,struct rte_node *node,void **objs,uint16_t nb_objs) {struct rte_mbuf *mbuf0, *mbuf1, *mbuf2, *mbuf3, **pkts;
lib/librte_node/ip4_lookup_sse.h +1 -1
@@ -7,7 +7,7 @@/* X86 SSE */ static uint16_t -ip4_lookup_node_process (struct rte_graph *graph,struct rte_node *node, +ip4_lookup_node_process_vec (struct rte_graph *graph,struct rte_node *node,void **objs,uint16_t nb_objs) {struct rte_mbuf *mbuf0, *mbuf1, *mbuf2, *mbuf3, **pkts;
Signature · 1 line
2.23.0
PATCH v10 18/18 acl: check max SIMD bitwidth
In reply to David MarchandFrom: Ciara Power <[email protected]>
When choosing a vector path to take, an extra condition must be satisfied to ensure the max SIMD bitwidth allows for the CPU enabled path. These checks are added in the check alg helper functions.
---
v10:
- updated #ifdef magic for arm and ppc checks,
v8:
- Added AVX512X32 into default options for get best alg function.
- Updated doc to reflect the above change.
v7:
- Removed global variable for max SIMD bitwidth.
- Added helper function for checking AVX512 cpu flags.
- Separated condition checking for the AVX512 algorithms to allow for checking 256/512 max SIMD bitwidth, respectively.
- Added to docs to reflect the added changes in algorithm selection.
--- .../prog_guide/packet_classif_access_ctrl.rst | 16 +++--- lib/librte_acl/rte_acl.c | 53 +++++++++++++------ lib/librte_acl/rte_acl.h | 1 + 3 files changed, 47 insertions(+), 23 deletions(-)
doc/guides/prog_guide/packet_classif_access_ctrl.rst +9 -7
@@ -368,24 +368,27 @@ After rte_acl_build() over given AC context has finished successfully, it can be There are several implementations of classify algorithm: * **RTE_ACL_CLASSIFY_SCALAR**: generic implementation, doesn't require any specific HW support. + Requires max SIMD bitwidth to be at least 64. * **RTE_ACL_CLASSIFY_SSE**: vector implementation, can process up to 8 flows in parallel. Requires SSE 4.1 support. + Requires max SIMD bitwidth to be at least 128. * **RTE_ACL_CLASSIFY_AVX2**: vector implementation, can process up to 16 flows in parallel. Requires AVX2 support. + Requires max SIMD bitwidth to be at least 256. * **RTE_ACL_CLASSIFY_NEON**: vector implementation, can process up to 8 flows - in parallel. Requires NEON support. + in parallel. Requires NEON support. Requires max SIMD bitwidth to be at least 128. * **RTE_ACL_CLASSIFY_ALTIVEC**: vector implementation, can process up to 8 - flows in parallel. Requires ALTIVEC support. + flows in parallel. Requires ALTIVEC support. Requires max SIMD bitwidth to be at least 128. * **RTE_ACL_CLASSIFY_AVX512X16**: vector implementation, can process up to 16 flows in parallel. Uses 256-bit width SIMD registers. - Requires AVX512 support. + Requires AVX512 support. Requires max SIMD bitwidth to be at least 256. * **RTE_ACL_CLASSIFY_AVX512X32**: vector implementation, can process up to 32 flows in parallel. Uses 512-bit width SIMD registers. - Requires AVX512 support. + Requires AVX512 support. Requires max SIMD bitwidth to be at least 512. It is purely a runtime decision which method to choose, there is no build-time difference. All implementations operates over the same internal RT structures and use similar principles. The main difference is that vector implementations can manually exploit IA SIMD instructions and process several input data flows in parallel.
@@ -393,9 +396,8 @@ At startup ACL library determines the highest available classify method for the .. note:: - Right now ``RTE_ACL_CLASSIFY_AVX512X32`` is not selected by default - (due to possible frequency level change), but it can be selected at - runtime by apps through the use of ACL API: ``rte_acl_set_ctx_classify``. + Runtime algorithm selection obeys EAL max SIMD bitwidth parameter. + For more details about expected behaviour please see :ref:`max_simd_bitwidth` Application Programming Interface (API) Usage ---------------------------------------------
lib/librte_acl/rte_acl.c +37 -16
105 lines not on this page · raw message
lib/librte_acl/rte_acl.h +1 -0
@@ -329,6 +329,7 @@ rte_acl_classify_alg(const struct rte_acl_ctx *ctx,* New default classify algorithm for given ACL context. * It is the caller responsibility to ensure that the value refers to the * existing algorithm, and that it could be run on the given CPU. +* The max SIMD bitwidth value in EAL must also allow for the chosen algorithm. * @return * - -EINVAL if the parameters are invalid. * - -ENOTSUP requested algorithm is not supported by given platform.
Signature · 1 line
2.23.0
Quoted 11 lines · David Marchand
From: Ciara Power <[email protected]>
A number of components in DPDK have optional AVX-512 or other vector code paths which can be selected at runtime. Rather than having each component provide its own mechanism to select a code path, this patchset adds support for a single setting to control what code paths are used. This can be used to enable some non-default code paths e.g. ones using AVX-512, but also to limit the code paths to certain vector widths, or to scalar code only, which is useful for testing.
Quoted 3 lines
The max SIMD bitwidth setting can be set by the app itself through use of the available API, or can be overriden by a commandline argument passed by the user.
Series applied, thanks Ciara!
Signature · 1 line
David Marchand