refactore mlx5 guides
cover refactore mlx5 guides
Recently [1] all the drivers running over the mlx5 device started sharing some objects.
This raised the need to create documentation for the common driver that would contain information relevant to all drivers and in particular:
- Tutorial for configuration of the device.
- Instruction for using Mellanox ofed.
- devargs of the common driver.
This series adds such documentation and reference to it as parts of the documentation of the other drivers.
[1]
https://patchwork.dpdk.org/project/dpdk/cover/[email protected]/
Michael Baum (5):
doc: remove obsolete explanations from mlx5 guide
doc: replace broken links in mlx5 guide
doc: correct name of BlueField-2 in mlx5 guide
doc: add shared guide for mlx5 PMDs
doc: update doorbell mapping parameter name in mlx5 guide
doc/guides/compressdevs/mlx5.rst | 40 +- doc/guides/cryptodevs/mlx5.rst | 38 +- doc/guides/nics/mlx4.rst | 4 +- doc/guides/nics/mlx5.rst | 567 +----------------- doc/guides/platform/index.rst | 1 + doc/guides/platform/mlx5.rst | 641 +++++++++++++++++++++ doc/guides/regexdevs/mlx5.rst | 44 +- doc/guides/vdpadevs/mlx5.rst | 87 +-- drivers/common/mlx5/linux/mlx5_common_os.c | 2 +- drivers/common/mlx5/mlx5_common.c | 31 +- drivers/common/mlx5/mlx5_common_defs.h | 8 +- drivers/net/mlx5/linux/mlx5_verbs.c | 2 +- drivers/net/mlx5/mlx5_devx.c | 2 +- 13 files changed, 747 insertions(+), 720 deletions(-) create mode 100644 doc/guides/platform/mlx5.rst
Signature · 1 line
2.25.1
PATCH 1/5 doc: remove obsolete explanations from mlx5 guide
Vectorized routines were removed in result of Tx datapath refactoring, and devarg keys documentation was updated.
However, more updating should have been done. In environment variables doc, there was explanation according to vectorized Tx which isn't relevant anymore.
This patch removes this irrelevant explanation.
--- doc/guides/nics/mlx5.rst | 9 --------- 1 file changed, 9 deletions(-)
doc/guides/nics/mlx5.rst +0 -9
@@ -557,15 +557,6 @@ Environment variables The register would be flushed to HW usually when the write-combining buffer becomes full, but it depends on CPU design. - Except for vectorized Tx burst routines, a write memory barrier is enforced - after updating the register so that the update can be immediately visible to - HW. - - When vectorized Tx burst is called, the barrier is set only if the burst size - is not aligned to MLX5_VPMD_TX_MAX_BURST. However, setting this environmental - variable will bring better latency even though the maximum throughput can - slightly decline. - Run-time configuration ~~~~~~~~~~~~~~~~~~~~~~
Signature · 1 line
2.25.1
PATCH 2/5 doc: replace broken links in mlx5 guide
In reply to Michael BaumUpdate links in both mlx4 and mlx5 doc.
--- doc/guides/nics/mlx4.rst | 4 ++-- doc/guides/nics/mlx5.rst | 6 +++--- 2 files changed, 5 insertions(+), 5 deletions(-)
doc/guides/nics/mlx4.rst +2 -2
@@ -14,7 +14,7 @@ the `Mellanox website <http://www.mellanox.com>`_. Help is also provided by the `Mellanox community <http://community.mellanox.com/welcome>`_. There is also a `section dedicated to this poll mode driver -<http://www.mellanox.com/page/products_dyn?product_family=209&mtag=pmd_for_dpdk>`_. +<https://developer.nvidia.com/networking/dpdk>`_. Implementation details
@@ -219,7 +219,7 @@ Mellanox OFED as a fallback - `Mellanox OFED`_ version: **4.4, 4.5, 4.6**. - firmware version: **2.42.5000** and above. -.. _`Mellanox OFED`: http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers +.. _`Mellanox OFED`: https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/ .. note::
doc/guides/nics/mlx5.rst +3 -3
@@ -19,7 +19,7 @@ Information and documentation about these adapters can be found on the `Mellanox community <http://community.mellanox.com/welcome>`__. There is also a `section dedicated to this poll mode driver -<http://www.mellanox.com/page/products_dyn?product_family=209&mtag=pmd_for_dpdk>`__. +<https://developer.nvidia.com/networking/dpdk>`_. Design
@@ -1376,9 +1376,9 @@ managers on most distributions, this PMD requires Ethernet extensions that may not be supported at the moment (this is a work in progress). `Mellanox OFED -<http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux>`__ and +<https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/>`__ and `Mellanox EN -<http://www.mellanox.com/page/products_dyn?product_family=27&mtag=linux>`__ +<https://network.nvidia.com/products/ethernet-drivers/linux/mlnx_en/>`__ include the necessary support and should be used in the meantime. For DPDK, only libibverbs, libmlx5, mlnx-ofed-kernel packages and firmware updates are required from that distribution.
Signature · 1 line
2.25.1
PATCH 3/5 doc: correct name of BlueField-2 in mlx5 guide
In reply to Michael BaumUpdate "BlueField 2" -> "BlueField-2" in mlx5 docs.
--- doc/guides/compressdevs/mlx5.rst | 6 +++--- doc/guides/cryptodevs/mlx5.rst | 6 +++--- doc/guides/nics/mlx5.rst | 2 +- doc/guides/regexdevs/mlx5.rst | 8 ++++---- 4 files changed, 11 insertions(+), 11 deletions(-)
doc/guides/compressdevs/mlx5.rst +3 -3
@@ -7,7 +7,7 @@ MLX5 compress driver ==================== The MLX5 compress driver library -(**librte_compress_mlx5**) provides support for **Mellanox BlueField 2** +(**librte_compress_mlx5**) provides support for **Mellanox BlueField-2** families of 25/50/100/200 Gb/s adapters. Design
@@ -21,7 +21,7 @@ These engines are part of the ARM complex of the BlueField chip, and as such they do not use NIC related resources (e.g. RX/TX bandwidth). They do share the same PCI and memory bandwidth. -So, using the BlueField device (starting from BlueField 2), the compress +So, using the BlueField device (starting from BlueField-2), the compress class operations can be supported in parallel to the net, vDPA and RegEx class operations.
@@ -95,7 +95,7 @@ Driver options Supported NICs -------------- -* Mellanox\ |reg| BlueField 2 SmartNIC +* Mellanox\ |reg| BlueField-2 SmartNIC Prerequisites -------------
doc/guides/cryptodevs/mlx5.rst +3 -3
@@ -88,7 +88,7 @@ The mlxreg dedicated tool should be used as follows: should not be specified. All the device ports should set it in order to move to operational mode. - For BlueField 2, the internal ports in the ARM system should also be set. + For BlueField-2, the internal ports in the ARM system should also be set. - Query CRYPTO_OPERATIONAL register to make sure the device is in Operational mode.
@@ -142,7 +142,7 @@ Supported NICs -------------- * Mellanox\ |reg| ConnectX\ |reg|-6 200G MCX654106A-HCAT (2x200G) -* Mellanox\ |reg| BlueField 2 SmartNIC +* Mellanox\ |reg| BlueField-2 SmartNIC * Mellanox\ |reg| ConnectX\ |reg|-6 Dx
@@ -162,7 +162,7 @@ FW Prerequisites ~~~~~~~~~~~~~~~~ - xx.31.0328 for ConnectX-6. -- xx.32.0108 for ConnectX-6 Dx and BlueField 2. +- xx.32.0108 for ConnectX-6 Dx and BlueField-2. Linux Prerequisites ~~~~~~~~~~~~~~~~~~~
doc/guides/nics/mlx5.rst +1 -1
@@ -253,7 +253,7 @@ Limitations - Flex item: - - Hardware support: BlueField 2. + - Hardware support: BlueField-2. - Flex item is supported on PF only. - Hardware limits ``header_length_mask_width`` up to 6 bits. - Firmware supports 8 global sample fields.
doc/guides/regexdevs/mlx5.rst +4 -4
@@ -7,7 +7,7 @@ MLX5 RegEx driver ================= The MLX5 RegEx (Regular Expression) driver library -(**librte_regex_mlx5**) provides support for **Mellanox BlueField 2** +(**librte_regex_mlx5**) provides support for **Mellanox BlueField-2** families of 25/50/100/200 Gb/s adapters. Design
@@ -43,13 +43,13 @@ Features Supported NICs -------------- -* Mellanox\ |reg| BlueField 2 SmartNIC +* Mellanox\ |reg| BlueField-2 SmartNIC Prerequisites ------------- -- BlueField 2 running Mellanox supported kernel. -- Enable the RegEx capabilities using system call from the BlueField 2. +- BlueField-2 running Mellanox supported kernel. +- Enable the RegEx capabilities using system call from the BlueField-2. - Official support is not yet released. Limitations
Signature · 1 line
2.25.1
PATCH 4/5 doc: add shared guide for mlx5 PMDs
In reply to Michael BaumAdds new documentation for MLX5 common driver that contains:
- Its features list (doesn't exist for now).
- Its devargs description.
- Device configuration information and tutorial.
- Quick Start Guide for Mellanox OFED/EN.
Move into this doc all shared information from other MLX5 PMD docs and add them reference to new common doc.
--- doc/guides/compressdevs/mlx5.rst | 34 +- doc/guides/cryptodevs/mlx5.rst | 32 +- doc/guides/nics/mlx5.rst | 522 ++------------------------- doc/guides/platform/index.rst | 1 + doc/guides/platform/mlx5.rst | 602 +++++++++++++++++++++++++++++++ doc/guides/regexdevs/mlx5.rst | 36 +- doc/guides/vdpadevs/mlx5.rst | 87 +---- 7 files changed, 662 insertions(+), 652 deletions(-) create mode 100644 doc/guides/platform/mlx5.rst
doc/guides/compressdevs/mlx5.rst +7 -27
@@ -3,10 +3,10 @@ .. include:: <isonum.txt> -MLX5 compress driver +MLX5 Compress Driver ==================== -The MLX5 compress driver library +The mlx5 compress driver library (**librte_compress_mlx5**) provides support for **Mellanox BlueField-2** families of 25/50/100/200 Gb/s adapters.
@@ -25,30 +25,7 @@ So, using the BlueField device (starting from BlueField-2), the compress class operations can be supported in parallel to the net, vDPA and RegEx class operations. -For security reasons and robustness, this driver only deals with virtual -memory addresses. The way resources allocations are handled by the kernel, -combined with hardware specifications that allow to handle virtual memory -addresses directly, ensure that DPDK applications cannot access random -physical memory (or memory that does not belong to the current process). - -The PMD uses libibverbs and libmlx5 to access the device firmware -or directly the hardware components. -There are different levels of objects and bypassing abilities -to get the best performances: - -- Verbs is a complete high-level generic API. -- Direct Verbs is a device-specific API. -- DevX allows to access firmware objects. - -Enabling librte_compress_mlx5 causes DPDK applications to be linked against -libibverbs. - -Mellanox mlx5 PCI device can be probed by number of different PCI devices, -for example net / vDPA / RegEx. To select the compress PMD ``class=compress`` -should be specified as device parameter. The compress device can be probed and -used with other Mellanox classes, by adding more options in the class. -For example: ``class=net:compress`` will probe both the net PMD and the compress -PMD. +See :doc:`../../platform/mlx5` guide for more design details. Features --------
@@ -85,6 +62,9 @@ Limitations Driver options -------------- +Please refer to :ref:`mlx5 common options <mlx5_common_driver_options>` +for an additional list of options shared with other mlx5 drivers. + - ``log-block-size`` parameter [int] Log of the Huffman block size in the Deflate algorithm.
@@ -101,4 +81,4 @@ Prerequisites ------------- - Mellanox OFED version: **5.2** - see :doc:`../../nics/mlx5` guide for more Mellanox OFED details. + See :ref:`mlx5 common prerequisites <mlx5_linux_prerequisites>` for more details.
doc/guides/cryptodevs/mlx5.rst +9 -23
@@ -28,23 +28,12 @@ when the MKEY is configured to perform crypto operations. The encryption does not require text to be aligned to the AES block size (128b). -For security reasons and to increase robustness, this driver only deals with virtual -memory addresses. The way resources allocations are handled by the kernel, -combined with hardware specifications that allow handling virtual memory -addresses directly, ensure that DPDK applications cannot access random -physical memory (or memory that does not belong to the current process). +See :doc:`../../platform/mlx5` guide for more design details. -The PMD uses ``libibverbs`` and ``libmlx5`` to access the device firmware -or to access the hardware components directly. -There are different levels of objects and bypassing abilities. -To get the best performances: - -- Verbs is a complete high-level generic API (Linux only). -- Direct Verbs is a device-specific API (Linux only). -- DevX allows to access firmware objects. +Configuration +------------- -Enabling ``librte_crypto_mlx5`` causes DPDK applications -to be linked against libibverbs on Linux OS. +See the :ref:`mlx5 common configuration <mlx5_common_env>`. In order to move the device to crypto operational mode, credential and KEK (Key Encrypting Key) should be set as the first step.
@@ -109,10 +98,8 @@ The mlxreg dedicated tool should be used as follows: Driver options -------------- -- ``class`` parameter [string] - - Select the class of the driver that should probe the device. - `crypto` for the mlx5 crypto driver. +Please refer to :ref:`mlx5 common options <mlx5_common_driver_options>` +for an additional list of options shared with other mlx5 drivers. - ``wcs_file`` parameter [string] - mandatory
@@ -168,13 +155,12 @@ Linux Prerequisites ~~~~~~~~~~~~~~~~~~~ - Mellanox OFED version: **5.3**. - see :doc:`../../nics/mlx5` guide for more Mellanox OFED details. - - Compilation can be done also with rdma-core v15+. - see :doc:`../../nics/mlx5` guide for more rdma-core details. + + See :ref:`mlx5 common prerequisites <mlx5_linux_prerequisites>` for more details. Windows Prerequisites ~~~~~~~~~~~~~~~~~~~~~ - Mellanox WINOF-2 version: **2.60** or higher. - see :doc:`../../nics/mlx5` guide for more Mellanox WINOF-2 details. + See :ref:`mlx5 common prerequisites <mlx5_windows_prerequisites>` for more details.
doc/guides/nics/mlx5.rst +24 -498
597 lines not on this page · raw message
doc/guides/platform/index.rst +1 -0
@@ -14,4 +14,5 @@ The following are platform specific guides and setup information. cnxk dpaa dpaa2 + mlx5 octeontx
doc/guides/platform/mlx5.rst new file +602 -0
602 lines not on this page · raw message
doc/guides/regexdevs/mlx5.rst +12 -24
@@ -3,10 +3,10 @@ .. include:: <isonum.txt> -MLX5 RegEx driver +MLX5 RegEx Driver ================= -The MLX5 RegEx (Regular Expression) driver library +The mlx5 RegEx (Regular Expression) driver library (**librte_regex_mlx5**) provides support for **Mellanox BlueField-2** families of 25/50/100/200 Gb/s adapters.
@@ -17,29 +17,21 @@ This PMD is configuring the RegEx HW engine. For the PMD to work, the application must supply a precompiled rule file in rof2 format. -The PMD uses libibverbs and libmlx5 to access the device firmware -or directly the hardware components. -There are different levels of objects and bypassing abilities -to get the best performances: - -- Verbs is a complete high-level generic API -- Direct Verbs is a device-specific API -- DevX allows to access firmware objects - -Enabling librte_regex_mlx5 causes DPDK applications to be linked against -libibverbs. - -Mellanox mlx5 pci device can be probed by number of different pci devices, -for example net / vDPA / RegEx. To select the RegEx PMD ``class=regex`` should -be specified as device parameter. The RegEx device can be probed and used with -other Mellanox devices, by adding more options in the class. -For example: ``class=net:regex`` will probe both the net PMD and the RegEx PMD. +See :doc:`../../platform/mlx5` guide for more design details. Features -------- - Multi segments mbuf support. +Configuration +------------- + +See :ref:`mlx5 common compilation <mlx5_common_compilation>`, +:ref:`mlx5 firmware configuration <mlx5_firmware_config>`, +and :ref:`mlx5 common driver options <mlx5_common_driver_options>`. + + Supported NICs --------------
@@ -52,12 +44,8 @@ Prerequisites - Enable the RegEx capabilities using system call from the BlueField-2. - Official support is not yet released. + Limitations ----------- - The firmware version must be greater than XX.31.0364 - -Run-time configuration -~~~~~~~~~~~~~~~~~~~~~~ - -- **ethtool** operations on related kernel interfaces also affect the PMD.
doc/guides/vdpadevs/mlx5.rst +7 -80
120 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH 5/5 doc: update doorbell mapping parameter name in mlx5 guide
In reply to Michael BaumThe "tx_db_nc" devarg forces doorbell register mapping to non-cached region eliminating the extra write memory barrier. This argument was used in creating the UAR for Tx and thus affected its performance.
Recently [1] its use has been extended to all UAR creation in all mlx5 drivers, and now its name is no longer so accurate.
This patch changes its name to "sq_db_nc" to suit any send queue that uses it. The old name will still work for backward compatibility.
[1] commit 5dfa003db53f ("common/mlx5: fix post doorbell barrier")
--- doc/guides/nics/mlx5.rst | 34 ++----------------- doc/guides/platform/mlx5.rst | 39 ++++++++++++++++++++++ drivers/common/mlx5/linux/mlx5_common_os.c | 2 +- drivers/common/mlx5/mlx5_common.c | 31 ++++++++++++----- drivers/common/mlx5/mlx5_common_defs.h | 8 ++--- drivers/net/mlx5/linux/mlx5_verbs.c | 2 +- drivers/net/mlx5/mlx5_devx.c | 2 +- 7 files changed, 72 insertions(+), 46 deletions(-)
doc/guides/nics/mlx5.rst +3 -31
@@ -814,37 +814,9 @@ for an additional list of options shared with other mlx5 drivers. - ``tx_db_nc`` parameter [int] - The rdma core library can map doorbell register in two ways, depending on the - environment variable "MLX5_SHUT_UP_BF": - - - As regular cached memory (usually with write combining attribute), if the - variable is either missing or set to zero. - - As non-cached memory, if the variable is present and set to not "0" value. - - The type of mapping may slightly affect the Tx performance, the optimal choice - is strongly relied on the host architecture and should be deduced practically. - - If ``tx_db_nc`` is set to zero, the doorbell is forced to be mapped to regular - memory (with write combining), the PMD will perform the extra write memory barrier - after writing to doorbell, it might increase the needed CPU clocks per packet - to send, but latency might be improved. - - If ``tx_db_nc`` is set to one, the doorbell is forced to be mapped to non - cached memory, the PMD will not perform the extra write memory barrier - after writing to doorbell, on some architectures it might improve the - performance. - - If ``tx_db_nc`` is set to two, the doorbell is forced to be mapped to regular - memory, the PMD will use heuristics to decide whether write memory barrier - should be performed. For bursts with size multiple of recommended one (64 pkts) - it is supposed the next burst is coming and no need to issue the extra memory - barrier (it is supposed to be issued in the next coming burst, at least after - descriptor writing). It might increase latency (on some hosts till next - packets transmit) and should be used with care. - - If ``tx_db_nc`` is omitted or set to zero, the preset (if any) environment - variable "MLX5_SHUT_UP_BF" value is used. If there is no "MLX5_SHUT_UP_BF", - the default ``tx_db_nc`` value is zero for ARM64 hosts and one for others. + This parameter name is deprecated and ignored. + The new name for this parameter is ``sq_db_nc``. + See :ref:`common driver options <mlx5_common_driver_options>`. - ``tx_pp`` parameter [int]
doc/guides/platform/mlx5.rst +39 -0
@@ -600,3 +600,42 @@ and below are the arguments supported by the common mlx5 layer. from system by default, without explicit rte memory flag. By default, the PMD will set this value to 0. + +- ``sq_db_nc`` parameter [int] + + The rdma core library can map doorbell register in two ways, + depending on the environment variable "MLX5_SHUT_UP_BF": + + - As regular cached memory (usually with write combining attribute), + if the variable is either missing or set to zero. + - As non-cached memory, if the variable is present and set to not "0" value. + + The same doorbell mapping approach is implemented directly by PMD + in UAR generation for queues created with DevX. + + The type of mapping may slightly affect the send queue performance, + the optimal choice strongly relied on the host architecture + and should be deduced practically. + + If ``sq_db_nc`` is set to zero, the doorbell is forced to be mapped to + regular memory (with write combining), the PMD will perform the extra write + memory barrier after writing to doorbell, it might increase the needed CPU + clocks per packet to send, but latency might be improved. + + If ``sq_db_nc`` is set to one, the doorbell is forced to be mapped to non + cached memory, the PMD will not perform the extra write memory barrier after + writing to doorbell, on some architectures it might improve the performance. + + If ``sq_db_nc`` is set to two, the doorbell is forced to be mapped to + regular memory, the PMD will use heuristics to decide whether a write memory + barrier should be performed. For bursts with size multiple of recommended one + (64 pkts) it is supposed the next burst is coming and no need to issue the + extra memory barrier (it is supposed to be issued in the next coming burst, + at least after descriptor writing). It might increase latency (on some hosts + till the next packets transmit) and should be used with care. + The PMD uses heuristics only for Tx queue, for other semd queues the doorbell + is forced to be mapped to regular memory as same as ``sq_db_nc`` is set to 0. + + If ``sq_db_nc`` is omitted, the preset (if any) environment variable + "MLX5_SHUT_UP_BF" value is used. If there is no "MLX5_SHUT_UP_BF", the + default ``sq_db_nc`` value is zero for ARM64 hosts and one for others.
drivers/common/mlx5/linux/mlx5_common_os.c +1 -1
@@ -630,7 +630,7 @@ mlx5_config_doorbell_mapping_env(int dbnc)setenv (MLX5_SHUT_UP_BF ,MLX5_SHUT_UP_BF_DEFAULT ,1 ); else setenv(MLX5_SHUT_UP_BF, - dbnc == MLX5_TXDB_NCACHED ? "1" : "0", 1); + dbnc == MLX5_SQ_DB_NCACHED ? "1" : "0", 1);return value; }
drivers/common/mlx5/mlx5_common.c +23 -8
@@ -35,10 +35,17 @@ uint8_t haswell_broadwell_cpu;/* * Device parameter to force doorbell register mapping -* to non-cahed region eliminating the extra write memory barrier. +* to non-cached region eliminating the extra write memory barrier. +* Deprecated, ignored (Name changed to sq_db_nc). */ #define MLX5_TX_DB_NC "tx_db_nc" +/* +* Device parameter to force doorbell register mapping +* to non-cached region eliminating the extra write memory barrier. +*/ +#define MLX5_SQ_DB_NC "sq_db_nc" +/* In case this is an x86_64 intel processor to check if * we should use relaxed ordering. */
@@ -255,11 +262,17 @@ mlx5_common_args_check_handler(const char *key, const char *val, void *opaque)DRV_LOG (WARNING ,"%s: \"%s\" is an invalid integer." , key, val);return -rte_errno; } -if (strcmp (key,MLX5_TX_DB_NC ) ==0 ) { -if (tmp !=MLX5_TXDB_CACHED && - tmp !=MLX5_TXDB_NCACHED && - tmp !=MLX5_TXDB_HEURISTIC ) { -DRV_LOG (ERR ,"Invalid Tx doorbell mapping parameter." ); +if (strcmp (key,MLX5_TX_DB_NC ) ==0 ) +DRV_LOG (WARNING , +"%s: deprecated parameter, converted to queue_db_nc" , + key); +if (strcmp (key,MLX5_SQ_DB_NC ) ==0 || +strcmp (key,MLX5_TX_DB_NC ) ==0 ) { +if (tmp !=MLX5_SQ_DB_CACHED && + tmp !=MLX5_SQ_DB_NCACHED && + tmp !=MLX5_SQ_DB_HEURISTIC ) { +DRV_LOG (ERR , +"Invalid Send Queue doorbell mapping parameter." ); rte_errno =EINVAL ;return -rte_errno; }
@@ -293,6 +306,7 @@ mlx5_common_config_get(struct mlx5_kvargs_ctrl *mkvlist,RTE_DEVARGS_KEY_CLASS ,MLX5_DRIVER_KEY ,MLX5_TX_DB_NC , +MLX5_SQ_DB_NC ,MLX5_MR_EXT_MEMSEG_EN ,MLX5_SYS_MEM_EN , MLX5_MR_MEMPOOL_REG_EN,
@@ -317,7 +331,8 @@ mlx5_common_config_get(struct mlx5_kvargs_ctrl *mkvlist,DRV_LOG (DEBUG ,"mr_ext_memseg_en is %u." , config->mr_ext_memseg_en);DRV_LOG (DEBUG ,"mr_mempool_reg_en is %u." , config->mr_mempool_reg_en);DRV_LOG (DEBUG ,"sys_mem_en is %u." , config->sys_mem_en); -DRV_LOG (DEBUG ,"Tx doorbell mapping parameter is %d." , config->dbnc); +DRV_LOG (DEBUG ,"Send Queue doorbell mapping parameter is %d." , + config->dbnc);return ret; }
@@ -1231,7 +1246,7 @@ mlx5_devx_alloc_uar(struct mlx5_common_device *cdev)for (retry =0 ; retry <MLX5_ALLOC_UAR_RETRY ; ++retry) {#ifdef MLX5DV_UAR_ALLOC_TYPE_NC /* Control the mapping type according to the settings. */ - uar_mapping = (cdev->config.dbnc ==MLX5_TXDB_NCACHED ) ? + uar_mapping = (cdev->config.dbnc ==MLX5_SQ_DB_NCACHED ) ?MLX5DV_UAR_ALLOC_TYPE_NC :MLX5DV_UAR_ALLOC_TYPE_BF ;#else /*
drivers/common/mlx5/mlx5_common_defs.h +4 -4
@@ -34,10 +34,10 @@/* Default PMD specific parameter value. */ #define MLX5_ARG_UNSET (-1) -/* MLX5_TX_DB_NC supported values. */ -#define MLX5_TXDB_CACHED 0 -#define MLX5_TXDB_NCACHED 1 -#define MLX5_TXDB_HEURISTIC 2 +/* MLX5_SQ_DB_NC supported values. */ +#define MLX5_SQ_DB_CACHED 0 +#define MLX5_SQ_DB_NCACHED 1 +#define MLX5_SQ_DB_HEURISTIC 2/* Fields of memory mapping type in offset parameter of mmap() */ #define MLX5_UAR_MMAP_CMD_SHIFT 8
drivers/net/mlx5/linux/mlx5_verbs.c +1 -1
@@ -926,7 +926,7 @@ mlx5_txq_ibv_uar_init(struct mlx5_txq_ctrl *txq_ctrl, void *bf_reg)DRV_LOG (ERR ,"Failed to get mem page size" ); rte_errno =ENOMEM ; } - txq->db_heu = priv->sh->cdev->config.dbnc ==MLX5_TXDB_HEURISTIC ; + txq->db_heu = priv->sh->cdev->config.dbnc ==MLX5_SQ_DB_HEURISTIC ; txq->db_nc =mlx5_db_map_type_get (uar_mmap_offset, page_size); ppriv->uar_table[txq->idx].db = bf_reg;#ifndef RTE_ARCH_64
drivers/net/mlx5/mlx5_devx.c +1 -1
@@ -1326,7 +1326,7 @@ mlx5_txq_devx_obj_new(struct rte_eth_dev *dev, uint16_t idx) txq_data->qp_db = &txq_obj->sq_obj.db_rec[MLX5_SND_DBR ]; *txq_data->qp_db =0 ; txq_data->qp_num_8s = txq_obj->sq_obj.sq->id <<8 ; - txq_data->db_heu = sh->cdev->config.dbnc ==MLX5_TXDB_HEURISTIC ; + txq_data->db_heu = sh->cdev->config.dbnc ==MLX5_SQ_DB_HEURISTIC ; txq_data->db_nc = sh->tx_uar.dbnc;/* Change Send Queue state to Ready-to-Send. */ ret =mlx5_txq_devx_modify (txq_obj,MLX5_TXQ_MOD_RST2RDY ,0 );
Signature · 1 line
2.25.1
Hi,
Quoted 45 lines
-----Original Message-----
From: Michael Baum <[email protected]>
Sent: Tuesday, February 22, 2022 2:48 PM
To: [email protected]
Cc: Matan Azrad <[email protected]>; Raslan Darawsheh <[email protected]>; Slava Ovsiienko <[email protected]>
Subject: [PATCH 0/5] refactore mlx5 guidesRecently [1] all the drivers running over the mlx5 device started sharing some objects.
This raised the need to create documentation for the common driver that would contain information relevant to all drivers and in particular:- Tutorial for configuration of the device.
- Instruction for using Mellanox ofed.
- devargs of the common driver.This series adds such documentation and reference to it as parts of the documentation of the other drivers.
[1]
https://patchwork.dpdk.org/project/dpdk/cover/20211019205602.3188203-
[email protected]/Michael Baum (5):
doc: remove obsolete explanations from mlx5 guide
doc: replace broken links in mlx5 guide
doc: correct name of BlueField-2 in mlx5 guide
doc: add shared guide for mlx5 PMDs
doc: update doorbell mapping parameter name in mlx5 guidedoc/guides/compressdevs/mlx5.rst | 40 +- doc/guides/cryptodevs/mlx5.rst | 38 +- doc/guides/nics/mlx4.rst | 4 +- doc/guides/nics/mlx5.rst | 567 +----------------- doc/guides/platform/index.rst | 1 + doc/guides/platform/mlx5.rst | 641 +++++++++++++++++++++ doc/guides/regexdevs/mlx5.rst | 44 +- doc/guides/vdpadevs/mlx5.rst | 87 +-- drivers/common/mlx5/linux/mlx5_common_os.c | 2 +- drivers/common/mlx5/mlx5_common.c | 31 +- drivers/common/mlx5/mlx5_common_defs.h | 8 +- drivers/net/mlx5/linux/mlx5_verbs.c | 2 +- drivers/net/mlx5/mlx5_devx.c | 2 +- 13 files changed, 747 insertions(+), 720 deletions(-) create mode 100644
Quoted 4 lines
doc/guides/platform/mlx5.rst--
2.25.1
For the series,
Kindest regards,
Raslan Darawsheh
v2 cover Refactor mlx5 guides
In reply to Michael BaumRecently [1] all the drivers running over the mlx5 device started sharing some objects.
This raised the need to create documentation for the common driver that would contain information relevant to all drivers and in particular:
- Tutorial for configuration of the device.
- Instruction for using Mellanox ofed.
- devargs of the common driver.
This series adds such documentation and reference to it as parts of the documentation of the other drivers.
[1]
https://patchwork.dpdk.org/project/dpdk/cover/[email protected]/
v2: Rebase
Michael Baum (5):
doc: remove obsolete explanations from mlx5 guide
doc: replace broken links in mlx5 guide
doc: correct name of BlueField-2 in mlx5 guide
doc: add shared guide for mlx5 PMDs
doc: update doorbell mapping parameter name in mlx5 guide
doc/guides/compressdevs/mlx5.rst | 40 +- doc/guides/cryptodevs/mlx5.rst | 38 +- doc/guides/nics/mlx4.rst | 4 +- doc/guides/nics/mlx5.rst | 567 +----------------- doc/guides/platform/index.rst | 1 + doc/guides/platform/mlx5.rst | 641 +++++++++++++++++++++ doc/guides/regexdevs/mlx5.rst | 44 +- doc/guides/vdpadevs/mlx5.rst | 87 +-- drivers/common/mlx5/linux/mlx5_common_os.c | 2 +- drivers/common/mlx5/mlx5_common.c | 31 +- drivers/common/mlx5/mlx5_common_defs.h | 8 +- drivers/net/mlx5/linux/mlx5_verbs.c | 2 +- drivers/net/mlx5/mlx5_devx.c | 2 +- 13 files changed, 747 insertions(+), 720 deletions(-) create mode 100644 doc/guides/platform/mlx5.rst
Signature · 1 line
2.25.1
PATCH v2 1/5 doc: remove obsolete explanations from mlx5 guide
Vectorized routines were removed in result of Tx datapath refactoring, and devarg keys documentation was updated.
However, more updating should have been done. In environment variables doc, there was explanation according to vectorized Tx which isn't relevant anymore.
This patch removes this irrelevant explanation.
--- doc/guides/nics/mlx5.rst | 9 --------- 1 file changed, 9 deletions(-)
doc/guides/nics/mlx5.rst +0 -9
@@ -557,15 +557,6 @@ Environment variables The register would be flushed to HW usually when the write-combining buffer becomes full, but it depends on CPU design. - Except for vectorized Tx burst routines, a write memory barrier is enforced - after updating the register so that the update can be immediately visible to - HW. - - When vectorized Tx burst is called, the barrier is set only if the burst size - is not aligned to MLX5_VPMD_TX_MAX_BURST. However, setting this environmental - variable will bring better latency even though the maximum throughput can - slightly decline. - Run-time configuration ~~~~~~~~~~~~~~~~~~~~~~
Signature · 1 line
2.25.1
PATCH v2 2/5 doc: replace broken links in mlx5 guide
In reply to Michael BaumUpdate links in both mlx4 and mlx5 doc.
--- doc/guides/nics/mlx4.rst | 4 ++-- doc/guides/nics/mlx5.rst | 6 +++--- 2 files changed, 5 insertions(+), 5 deletions(-)
doc/guides/nics/mlx4.rst +2 -2
@@ -14,7 +14,7 @@ the `Mellanox website <http://www.mellanox.com>`_. Help is also provided by the `Mellanox community <http://community.mellanox.com/welcome>`_. There is also a `section dedicated to this poll mode driver -<http://www.mellanox.com/page/products_dyn?product_family=209&mtag=pmd_for_dpdk>`_. +<https://developer.nvidia.com/networking/dpdk>`_. Implementation details
@@ -219,7 +219,7 @@ Mellanox OFED as a fallback - `Mellanox OFED`_ version: **4.4, 4.5, 4.6**. - firmware version: **2.42.5000** and above. -.. _`Mellanox OFED`: http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers +.. _`Mellanox OFED`: https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/ .. note::
doc/guides/nics/mlx5.rst +3 -3
@@ -19,7 +19,7 @@ Information and documentation about these adapters can be found on the `Mellanox community <http://community.mellanox.com/welcome>`__. There is also a `section dedicated to this poll mode driver -<http://www.mellanox.com/page/products_dyn?product_family=209&mtag=pmd_for_dpdk>`__. +<https://developer.nvidia.com/networking/dpdk>`_. Design
@@ -1381,9 +1381,9 @@ managers on most distributions, this PMD requires Ethernet extensions that may not be supported at the moment (this is a work in progress). `Mellanox OFED -<http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux>`__ and +<https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/>`__ and `Mellanox EN -<http://www.mellanox.com/page/products_dyn?product_family=27&mtag=linux>`__ +<https://network.nvidia.com/products/ethernet-drivers/linux/mlnx_en/>`__ include the necessary support and should be used in the meantime. For DPDK, only libibverbs, libmlx5, mlnx-ofed-kernel packages and firmware updates are required from that distribution.
Signature · 1 line
2.25.1
PATCH v2 3/5 doc: correct name of BlueField-2 in mlx5 guide
In reply to Michael BaumUpdate "BlueField 2" -> "BlueField-2" in mlx5 docs.
--- doc/guides/compressdevs/mlx5.rst | 6 +++--- doc/guides/cryptodevs/mlx5.rst | 6 +++--- doc/guides/nics/mlx5.rst | 2 +- doc/guides/regexdevs/mlx5.rst | 8 ++++---- 4 files changed, 11 insertions(+), 11 deletions(-)
doc/guides/compressdevs/mlx5.rst +3 -3
@@ -7,7 +7,7 @@ MLX5 compress driver ==================== The MLX5 compress driver library -(**librte_compress_mlx5**) provides support for **Mellanox BlueField 2** +(**librte_compress_mlx5**) provides support for **Mellanox BlueField-2** families of 25/50/100/200 Gb/s adapters. Design
@@ -21,7 +21,7 @@ These engines are part of the ARM complex of the BlueField chip, and as such they do not use NIC related resources (e.g. RX/TX bandwidth). They do share the same PCI and memory bandwidth. -So, using the BlueField device (starting from BlueField 2), the compress +So, using the BlueField device (starting from BlueField-2), the compress class operations can be supported in parallel to the net, vDPA and RegEx class operations.
@@ -95,7 +95,7 @@ Driver options Supported NICs -------------- -* Mellanox\ |reg| BlueField 2 SmartNIC +* Mellanox\ |reg| BlueField-2 SmartNIC Prerequisites -------------
doc/guides/cryptodevs/mlx5.rst +3 -3
@@ -88,7 +88,7 @@ The mlxreg dedicated tool should be used as follows: should not be specified. All the device ports should set it in order to move to operational mode. - For BlueField 2, the internal ports in the ARM system should also be set. + For BlueField-2, the internal ports in the ARM system should also be set. - Query CRYPTO_OPERATIONAL register to make sure the device is in Operational mode.
@@ -142,7 +142,7 @@ Supported NICs -------------- * Mellanox\ |reg| ConnectX\ |reg|-6 200G MCX654106A-HCAT (2x200G) -* Mellanox\ |reg| BlueField 2 SmartNIC +* Mellanox\ |reg| BlueField-2 SmartNIC * Mellanox\ |reg| ConnectX\ |reg|-6 Dx
@@ -162,7 +162,7 @@ FW Prerequisites ~~~~~~~~~~~~~~~~ - xx.31.0328 for ConnectX-6. -- xx.32.0108 for ConnectX-6 Dx and BlueField 2. +- xx.32.0108 for ConnectX-6 Dx and BlueField-2. Linux Prerequisites ~~~~~~~~~~~~~~~~~~~
doc/guides/nics/mlx5.rst +1 -1
@@ -253,7 +253,7 @@ Limitations - Flex item: - - Hardware support: BlueField 2. + - Hardware support: BlueField-2. - Flex item is supported on PF only. - Hardware limits ``header_length_mask_width`` up to 6 bits. - Firmware supports 8 global sample fields.
doc/guides/regexdevs/mlx5.rst +4 -4
@@ -7,7 +7,7 @@ MLX5 RegEx driver ================= The MLX5 RegEx (Regular Expression) driver library -(**librte_regex_mlx5**) provides support for **Mellanox BlueField 2** +(**librte_regex_mlx5**) provides support for **Mellanox BlueField-2** families of 25/50/100/200 Gb/s adapters. Design
@@ -43,13 +43,13 @@ Features Supported NICs -------------- -* Mellanox\ |reg| BlueField 2 SmartNIC +* Mellanox\ |reg| BlueField-2 SmartNIC Prerequisites ------------- -- BlueField 2 running Mellanox supported kernel. -- Enable the RegEx capabilities using system call from the BlueField 2. +- BlueField-2 running Mellanox supported kernel. +- Enable the RegEx capabilities using system call from the BlueField-2. - Official support is not yet released. Limitations
Signature · 1 line
2.25.1
PATCH v2 4/5 doc: add shared guide for mlx5 PMDs
In reply to Michael BaumAdds new documentation for MLX5 common driver that contains:
- Its features list (doesn't exist for now).
- Its devargs description.
- Device configuration information and tutorial.
- Quick Start Guide for Mellanox OFED/EN.
Move into this doc all shared information from other MLX5 PMD docs and add them reference to new common doc.
--- doc/guides/compressdevs/mlx5.rst | 34 +- doc/guides/cryptodevs/mlx5.rst | 32 +- doc/guides/nics/mlx5.rst | 522 ++------------------------- doc/guides/platform/index.rst | 1 + doc/guides/platform/mlx5.rst | 602 +++++++++++++++++++++++++++++++ doc/guides/regexdevs/mlx5.rst | 36 +- doc/guides/vdpadevs/mlx5.rst | 87 +---- 7 files changed, 662 insertions(+), 652 deletions(-) create mode 100644 doc/guides/platform/mlx5.rst
doc/guides/compressdevs/mlx5.rst +7 -27
@@ -3,10 +3,10 @@ .. include:: <isonum.txt> -MLX5 compress driver +MLX5 Compress Driver ==================== -The MLX5 compress driver library +The mlx5 compress driver library (**librte_compress_mlx5**) provides support for **Mellanox BlueField-2** families of 25/50/100/200 Gb/s adapters.
@@ -25,30 +25,7 @@ So, using the BlueField device (starting from BlueField-2), the compress class operations can be supported in parallel to the net, vDPA and RegEx class operations. -For security reasons and robustness, this driver only deals with virtual -memory addresses. The way resources allocations are handled by the kernel, -combined with hardware specifications that allow to handle virtual memory -addresses directly, ensure that DPDK applications cannot access random -physical memory (or memory that does not belong to the current process). - -The PMD uses libibverbs and libmlx5 to access the device firmware -or directly the hardware components. -There are different levels of objects and bypassing abilities -to get the best performances: - -- Verbs is a complete high-level generic API. -- Direct Verbs is a device-specific API. -- DevX allows to access firmware objects. - -Enabling librte_compress_mlx5 causes DPDK applications to be linked against -libibverbs. - -Mellanox mlx5 PCI device can be probed by number of different PCI devices, -for example net / vDPA / RegEx. To select the compress PMD ``class=compress`` -should be specified as device parameter. The compress device can be probed and -used with other Mellanox classes, by adding more options in the class. -For example: ``class=net:compress`` will probe both the net PMD and the compress -PMD. +See :doc:`../../platform/mlx5` guide for more design details. Features --------
@@ -85,6 +62,9 @@ Limitations Driver options -------------- +Please refer to :ref:`mlx5 common options <mlx5_common_driver_options>` +for an additional list of options shared with other mlx5 drivers. + - ``log-block-size`` parameter [int] Log of the Huffman block size in the Deflate algorithm.
@@ -101,4 +81,4 @@ Prerequisites ------------- - Mellanox OFED version: **5.2** - see :doc:`../../nics/mlx5` guide for more Mellanox OFED details. + See :ref:`mlx5 common prerequisites <mlx5_linux_prerequisites>` for more details.
doc/guides/cryptodevs/mlx5.rst +9 -23
@@ -28,23 +28,12 @@ when the MKEY is configured to perform crypto operations. The encryption does not require text to be aligned to the AES block size (128b). -For security reasons and to increase robustness, this driver only deals with virtual -memory addresses. The way resources allocations are handled by the kernel, -combined with hardware specifications that allow handling virtual memory -addresses directly, ensure that DPDK applications cannot access random -physical memory (or memory that does not belong to the current process). +See :doc:`../../platform/mlx5` guide for more design details. -The PMD uses ``libibverbs`` and ``libmlx5`` to access the device firmware -or to access the hardware components directly. -There are different levels of objects and bypassing abilities. -To get the best performances: - -- Verbs is a complete high-level generic API (Linux only). -- Direct Verbs is a device-specific API (Linux only). -- DevX allows to access firmware objects. +Configuration +------------- -Enabling ``librte_crypto_mlx5`` causes DPDK applications -to be linked against libibverbs on Linux OS. +See the :ref:`mlx5 common configuration <mlx5_common_env>`. In order to move the device to crypto operational mode, credential and KEK (Key Encrypting Key) should be set as the first step.
@@ -109,10 +98,8 @@ The mlxreg dedicated tool should be used as follows: Driver options -------------- -- ``class`` parameter [string] - - Select the class of the driver that should probe the device. - `crypto` for the mlx5 crypto driver. +Please refer to :ref:`mlx5 common options <mlx5_common_driver_options>` +for an additional list of options shared with other mlx5 drivers. - ``wcs_file`` parameter [string] - mandatory
@@ -168,13 +155,12 @@ Linux Prerequisites ~~~~~~~~~~~~~~~~~~~ - Mellanox OFED version: **5.3**. - see :doc:`../../nics/mlx5` guide for more Mellanox OFED details. - - Compilation can be done also with rdma-core v15+. - see :doc:`../../nics/mlx5` guide for more rdma-core details. + + See :ref:`mlx5 common prerequisites <mlx5_linux_prerequisites>` for more details. Windows Prerequisites ~~~~~~~~~~~~~~~~~~~~~ - Mellanox WINOF-2 version: **2.60** or higher. - see :doc:`../../nics/mlx5` guide for more Mellanox WINOF-2 details. + See :ref:`mlx5 common prerequisites <mlx5_windows_prerequisites>` for more details.
doc/guides/nics/mlx5.rst +24 -498
597 lines not on this page · raw message
doc/guides/platform/index.rst +1 -0
@@ -14,4 +14,5 @@ The following are platform specific guides and setup information. cnxk dpaa dpaa2 + mlx5 octeontx
doc/guides/platform/mlx5.rst new file +602 -0
602 lines not on this page · raw message
doc/guides/regexdevs/mlx5.rst +12 -24
@@ -3,10 +3,10 @@ .. include:: <isonum.txt> -MLX5 RegEx driver +MLX5 RegEx Driver ================= -The MLX5 RegEx (Regular Expression) driver library +The mlx5 RegEx (Regular Expression) driver library (**librte_regex_mlx5**) provides support for **Mellanox BlueField-2** families of 25/50/100/200 Gb/s adapters.
@@ -17,29 +17,21 @@ This PMD is configuring the RegEx HW engine. For the PMD to work, the application must supply a precompiled rule file in rof2 format. -The PMD uses libibverbs and libmlx5 to access the device firmware -or directly the hardware components. -There are different levels of objects and bypassing abilities -to get the best performances: - -- Verbs is a complete high-level generic API -- Direct Verbs is a device-specific API -- DevX allows to access firmware objects - -Enabling librte_regex_mlx5 causes DPDK applications to be linked against -libibverbs. - -Mellanox mlx5 pci device can be probed by number of different pci devices, -for example net / vDPA / RegEx. To select the RegEx PMD ``class=regex`` should -be specified as device parameter. The RegEx device can be probed and used with -other Mellanox devices, by adding more options in the class. -For example: ``class=net:regex`` will probe both the net PMD and the RegEx PMD. +See :doc:`../../platform/mlx5` guide for more design details. Features -------- - Multi segments mbuf support. +Configuration +------------- + +See :ref:`mlx5 common compilation <mlx5_common_compilation>`, +:ref:`mlx5 firmware configuration <mlx5_firmware_config>`, +and :ref:`mlx5 common driver options <mlx5_common_driver_options>`. + + Supported NICs --------------
@@ -52,12 +44,8 @@ Prerequisites - Enable the RegEx capabilities using system call from the BlueField-2. - Official support is not yet released. + Limitations ----------- - The firmware version must be greater than XX.31.0364 - -Run-time configuration -~~~~~~~~~~~~~~~~~~~~~~ - -- **ethtool** operations on related kernel interfaces also affect the PMD.
doc/guides/vdpadevs/mlx5.rst +7 -80
120 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v2 5/5 doc: update doorbell mapping parameter name in mlx5 guide
In reply to Michael BaumThe "tx_db_nc" devarg forces doorbell register mapping to non-cached region eliminating the extra write memory barrier. This argument was used in creating the UAR for Tx and thus affected its performance.
Recently [1] its use has been extended to all UAR creation in all mlx5 drivers, and now its name is no longer so accurate.
This patch changes its name to "sq_db_nc" to suit any send queue that uses it. The old name will still work for backward compatibility.
[1] commit 5dfa003db53f ("common/mlx5: fix post doorbell barrier")
--- doc/guides/nics/mlx5.rst | 34 ++----------------- doc/guides/platform/mlx5.rst | 39 ++++++++++++++++++++++ drivers/common/mlx5/linux/mlx5_common_os.c | 2 +- drivers/common/mlx5/mlx5_common.c | 31 ++++++++++++----- drivers/common/mlx5/mlx5_common_defs.h | 8 ++--- drivers/net/mlx5/linux/mlx5_verbs.c | 2 +- drivers/net/mlx5/mlx5_devx.c | 2 +- 7 files changed, 72 insertions(+), 46 deletions(-)
doc/guides/nics/mlx5.rst +3 -31
@@ -814,37 +814,9 @@ for an additional list of options shared with other mlx5 drivers. - ``tx_db_nc`` parameter [int] - The rdma core library can map doorbell register in two ways, depending on the - environment variable "MLX5_SHUT_UP_BF": - - - As regular cached memory (usually with write combining attribute), if the - variable is either missing or set to zero. - - As non-cached memory, if the variable is present and set to not "0" value. - - The type of mapping may slightly affect the Tx performance, the optimal choice - is strongly relied on the host architecture and should be deduced practically. - - If ``tx_db_nc`` is set to zero, the doorbell is forced to be mapped to regular - memory (with write combining), the PMD will perform the extra write memory barrier - after writing to doorbell, it might increase the needed CPU clocks per packet - to send, but latency might be improved. - - If ``tx_db_nc`` is set to one, the doorbell is forced to be mapped to non - cached memory, the PMD will not perform the extra write memory barrier - after writing to doorbell, on some architectures it might improve the - performance. - - If ``tx_db_nc`` is set to two, the doorbell is forced to be mapped to regular - memory, the PMD will use heuristics to decide whether write memory barrier - should be performed. For bursts with size multiple of recommended one (64 pkts) - it is supposed the next burst is coming and no need to issue the extra memory - barrier (it is supposed to be issued in the next coming burst, at least after - descriptor writing). It might increase latency (on some hosts till next - packets transmit) and should be used with care. - - If ``tx_db_nc`` is omitted or set to zero, the preset (if any) environment - variable "MLX5_SHUT_UP_BF" value is used. If there is no "MLX5_SHUT_UP_BF", - the default ``tx_db_nc`` value is zero for ARM64 hosts and one for others. + This parameter name is deprecated and ignored. + The new name for this parameter is ``sq_db_nc``. + See :ref:`common driver options <mlx5_common_driver_options>`. - ``tx_pp`` parameter [int]
doc/guides/platform/mlx5.rst +39 -0
@@ -600,3 +600,42 @@ and below are the arguments supported by the common mlx5 layer. from system by default, without explicit rte memory flag. By default, the PMD will set this value to 0. + +- ``sq_db_nc`` parameter [int] + + The rdma core library can map doorbell register in two ways, + depending on the environment variable "MLX5_SHUT_UP_BF": + + - As regular cached memory (usually with write combining attribute), + if the variable is either missing or set to zero. + - As non-cached memory, if the variable is present and set to not "0" value. + + The same doorbell mapping approach is implemented directly by PMD + in UAR generation for queues created with DevX. + + The type of mapping may slightly affect the send queue performance, + the optimal choice strongly relied on the host architecture + and should be deduced practically. + + If ``sq_db_nc`` is set to zero, the doorbell is forced to be mapped to + regular memory (with write combining), the PMD will perform the extra write + memory barrier after writing to doorbell, it might increase the needed CPU + clocks per packet to send, but latency might be improved. + + If ``sq_db_nc`` is set to one, the doorbell is forced to be mapped to non + cached memory, the PMD will not perform the extra write memory barrier after + writing to doorbell, on some architectures it might improve the performance. + + If ``sq_db_nc`` is set to two, the doorbell is forced to be mapped to + regular memory, the PMD will use heuristics to decide whether a write memory + barrier should be performed. For bursts with size multiple of recommended one + (64 pkts) it is supposed the next burst is coming and no need to issue the + extra memory barrier (it is supposed to be issued in the next coming burst, + at least after descriptor writing). It might increase latency (on some hosts + till the next packets transmit) and should be used with care. + The PMD uses heuristics only for Tx queue, for other semd queues the doorbell + is forced to be mapped to regular memory as same as ``sq_db_nc`` is set to 0. + + If ``sq_db_nc`` is omitted, the preset (if any) environment variable + "MLX5_SHUT_UP_BF" value is used. If there is no "MLX5_SHUT_UP_BF", the + default ``sq_db_nc`` value is zero for ARM64 hosts and one for others.
drivers/common/mlx5/linux/mlx5_common_os.c +1 -1
@@ -630,7 +630,7 @@ mlx5_config_doorbell_mapping_env(int dbnc)setenv (MLX5_SHUT_UP_BF ,MLX5_SHUT_UP_BF_DEFAULT ,1 ); else setenv(MLX5_SHUT_UP_BF, - dbnc == MLX5_TXDB_NCACHED ? "1" : "0", 1); + dbnc == MLX5_SQ_DB_NCACHED ? "1" : "0", 1);return value; }
drivers/common/mlx5/mlx5_common.c +23 -8
@@ -35,10 +35,17 @@ uint8_t haswell_broadwell_cpu;/* * Device parameter to force doorbell register mapping -* to non-cahed region eliminating the extra write memory barrier. +* to non-cached region eliminating the extra write memory barrier. +* Deprecated, ignored (Name changed to sq_db_nc). */ #define MLX5_TX_DB_NC "tx_db_nc" +/* +* Device parameter to force doorbell register mapping +* to non-cached region eliminating the extra write memory barrier. +*/ +#define MLX5_SQ_DB_NC "sq_db_nc" +/* In case this is an x86_64 intel processor to check if * we should use relaxed ordering. */
@@ -255,11 +262,17 @@ mlx5_common_args_check_handler(const char *key, const char *val, void *opaque)DRV_LOG (WARNING ,"%s: \"%s\" is an invalid integer." , key, val);return -rte_errno; } -if (strcmp (key,MLX5_TX_DB_NC ) ==0 ) { -if (tmp !=MLX5_TXDB_CACHED && - tmp !=MLX5_TXDB_NCACHED && - tmp !=MLX5_TXDB_HEURISTIC ) { -DRV_LOG (ERR ,"Invalid Tx doorbell mapping parameter." ); +if (strcmp (key,MLX5_TX_DB_NC ) ==0 ) +DRV_LOG (WARNING , +"%s: deprecated parameter, converted to queue_db_nc" , + key); +if (strcmp (key,MLX5_SQ_DB_NC ) ==0 || +strcmp (key,MLX5_TX_DB_NC ) ==0 ) { +if (tmp !=MLX5_SQ_DB_CACHED && + tmp !=MLX5_SQ_DB_NCACHED && + tmp !=MLX5_SQ_DB_HEURISTIC ) { +DRV_LOG (ERR , +"Invalid Send Queue doorbell mapping parameter." ); rte_errno =EINVAL ;return -rte_errno; }
@@ -293,6 +306,7 @@ mlx5_common_config_get(struct mlx5_kvargs_ctrl *mkvlist,RTE_DEVARGS_KEY_CLASS ,MLX5_DRIVER_KEY ,MLX5_TX_DB_NC , +MLX5_SQ_DB_NC ,MLX5_MR_EXT_MEMSEG_EN ,MLX5_SYS_MEM_EN , MLX5_MR_MEMPOOL_REG_EN,
@@ -317,7 +331,8 @@ mlx5_common_config_get(struct mlx5_kvargs_ctrl *mkvlist,DRV_LOG (DEBUG ,"mr_ext_memseg_en is %u." , config->mr_ext_memseg_en);DRV_LOG (DEBUG ,"mr_mempool_reg_en is %u." , config->mr_mempool_reg_en);DRV_LOG (DEBUG ,"sys_mem_en is %u." , config->sys_mem_en); -DRV_LOG (DEBUG ,"Tx doorbell mapping parameter is %d." , config->dbnc); +DRV_LOG (DEBUG ,"Send Queue doorbell mapping parameter is %d." , + config->dbnc);return ret; }
@@ -1231,7 +1246,7 @@ mlx5_devx_alloc_uar(struct mlx5_common_device *cdev)for (retry =0 ; retry <MLX5_ALLOC_UAR_RETRY ; ++retry) {#ifdef MLX5DV_UAR_ALLOC_TYPE_NC /* Control the mapping type according to the settings. */ - uar_mapping = (cdev->config.dbnc ==MLX5_TXDB_NCACHED ) ? + uar_mapping = (cdev->config.dbnc ==MLX5_SQ_DB_NCACHED ) ?MLX5DV_UAR_ALLOC_TYPE_NC :MLX5DV_UAR_ALLOC_TYPE_BF ;#else /*
drivers/common/mlx5/mlx5_common_defs.h +4 -4
@@ -34,10 +34,10 @@/* Default PMD specific parameter value. */ #define MLX5_ARG_UNSET (-1) -/* MLX5_TX_DB_NC supported values. */ -#define MLX5_TXDB_CACHED 0 -#define MLX5_TXDB_NCACHED 1 -#define MLX5_TXDB_HEURISTIC 2 +/* MLX5_SQ_DB_NC supported values. */ +#define MLX5_SQ_DB_CACHED 0 +#define MLX5_SQ_DB_NCACHED 1 +#define MLX5_SQ_DB_HEURISTIC 2/* Fields of memory mapping type in offset parameter of mmap() */ #define MLX5_UAR_MMAP_CMD_SHIFT 8
drivers/net/mlx5/linux/mlx5_verbs.c +1 -1
@@ -926,7 +926,7 @@ mlx5_txq_ibv_uar_init(struct mlx5_txq_ctrl *txq_ctrl, void *bf_reg)DRV_LOG (ERR ,"Failed to get mem page size" ); rte_errno =ENOMEM ; } - txq->db_heu = priv->sh->cdev->config.dbnc ==MLX5_TXDB_HEURISTIC ; + txq->db_heu = priv->sh->cdev->config.dbnc ==MLX5_SQ_DB_HEURISTIC ; txq->db_nc =mlx5_db_map_type_get (uar_mmap_offset, page_size); ppriv->uar_table[txq->idx].db = bf_reg;#ifndef RTE_ARCH_64
drivers/net/mlx5/mlx5_devx.c +1 -1
@@ -1326,7 +1326,7 @@ mlx5_txq_devx_obj_new(struct rte_eth_dev *dev, uint16_t idx) txq_data->qp_db = &txq_obj->sq_obj.db_rec[MLX5_SND_DBR ]; *txq_data->qp_db =0 ; txq_data->qp_num_8s = txq_obj->sq_obj.sq->id <<8 ; - txq_data->db_heu = sh->cdev->config.dbnc ==MLX5_TXDB_HEURISTIC ; + txq_data->db_heu = sh->cdev->config.dbnc ==MLX5_SQ_DB_HEURISTIC ; txq_data->db_nc = sh->tx_uar.dbnc; txq_data->wait_on_time = !!(!sh->config.tx_pp && sh->cdev->config.hca_attr.wait_on_time);
Signature · 1 line
2.25.1
Re: [PATCH v2 5/5] doc: update doorbell mapping parameter name in mlx5 guide
Quoted 15 lines · Michael Baum
The "
tx_db_nc" devarg forces doorbell register mapping to non-cached region eliminating the extra write memory barrier. This argument was used in creating the UAR for Tx and thus affected its performance.Recently [1] its use has been extended to all UAR creation in all mlx5 drivers, and now its name is no longer so accurate.
This patch changes its name to "
sq_db_nc" to suit any send queue that uses it. The old name will still work for backward compatibility.[1] commit
5dfa003db53f("common/mlx5: fix post doorbell barrier")Signed-off-by Michael Baum<[email protected]>Reviewed-by Raslan Darawsheh<[email protected]>
Quoted 9 lines
--- doc/guides/nics/mlx5.rst | 34 ++----------------- doc/guides/platform/mlx5.rst | 39 ++++++++++++++++++++++ drivers/common/mlx5/linux/mlx5_common_os.c | 2 +- drivers/common/mlx5/mlx5_common.c | 31 ++++++++++++----- drivers/common/mlx5/mlx5_common_defs.h | 8 ++--- drivers/net/mlx5/linux/mlx5_verbs.c | 2 +- drivers/net/mlx5/mlx5_devx.c | 2 +- 7 files changed, 72 insertions(+), 46 deletions(-)
Hi Michael,
This is not a document patch to update the parameter name, patch actually changes the parameter name in the code and documents it, updating it as:
net/mlx5: update doorbell mapping parameter name
RE: [PATCH v2 5/5] doc: update doorbell mapping parameter name in mlx5 guide
Quoted 26 lines · Ferruh Yigit
Quoted 15 lines · Michael Baum
The "
tx_db_nc" devarg forces doorbell register mapping to non-cached region eliminating the extra write memory barrier. This argument was used in creating the UAR for Tx and thus affected its performance.Recently [1] its use has been extended to all UAR creation in all mlx5 drivers, and now its name is no longer so accurate.
This patch changes its name to "
sq_db_nc" to suit any send queue that uses it. The old name will still work for backward compatibility.[1] commit
5dfa003db53f("common/mlx5: fix post doorbell barrier")Signed-off-by Michael Baum<[email protected]>Reviewed-by Raslan Darawsheh<[email protected]>Quoted 9 lines
--- doc/guides/nics/mlx5.rst | 34 ++----------------- doc/guides/platform/mlx5.rst | 39 ++++++++++++++++++++++ drivers/common/mlx5/linux/mlx5_common_os.c | 2 +- drivers/common/mlx5/mlx5_common.c | 31 ++++++++++++----- drivers/common/mlx5/mlx5_common_defs.h | 8 ++--- drivers/net/mlx5/linux/mlx5_verbs.c | 2 +- drivers/net/mlx5/mlx5_devx.c | 2 +- 7 files changed, 72 insertions(+), 46 deletions(-)
Quoted 5 lines
Hi Michael,
This is not a document patch to update the parameter name, patch actually changes the parameter name in the code and documents it, updating it as:
net/mlx5: update doorbell mapping parameter name
Hi Ferruh,
Your argument makes sense to me, but most of updates was in MLX5 common library so I think it is better to write:
common/mlx5: update doorbell mapping parameter name
Re: [PATCH v2 5/5] doc: update doorbell mapping parameter name in mlx5 guide
Quoted 30 lines · Michael Baum
Quoted 28 lines · Ferruh Yigit
Quoted 15 lines · Michael Baum
The "
tx_db_nc" devarg forces doorbell register mapping to non-cached region eliminating the extra write memory barrier. This argument was used in creating the UAR for Tx and thus affected its performance.Recently [1] its use has been extended to all UAR creation in all mlx5 drivers, and now its name is no longer so accurate.
This patch changes its name to "
sq_db_nc" to suit any send queue that uses it. The old name will still work for backward compatibility.[1] commit
5dfa003db53f("common/mlx5: fix post doorbell barrier")Signed-off-by Michael Baum<[email protected]>Reviewed-by Raslan Darawsheh<[email protected]>Quoted 9 lines
--- doc/guides/nics/mlx5.rst | 34 ++----------------- doc/guides/platform/mlx5.rst | 39 ++++++++++++++++++++++ drivers/common/mlx5/linux/mlx5_common_os.c | 2 +- drivers/common/mlx5/mlx5_common.c | 31 ++++++++++++----- drivers/common/mlx5/mlx5_common_defs.h | 8 ++--- drivers/net/mlx5/linux/mlx5_verbs.c | 2 +- drivers/net/mlx5/mlx5_devx.c | 2 +- 7 files changed, 72 insertions(+), 46 deletions(-)Hi Michael,
Quoted 8 lines
Quoted 3 lines
This is not a document patch to update the parameter name, patch actually changes the parameter name in the code and documents it, updating it as:
net/mlx5: update doorbell mapping parameter nameHi Ferruh,
Your argument makes sense to me, but most of updates was in MLX5 common library so I think it is better to write:
common/mlx5: update doorbell mapping parameter name
ack, updated accordingly.
RE: [PATCH v2 0/5] Refactor mlx5 guides
In reply to Michael BaumHi,
Quoted 48 lines
-----Original Message-----
From: Michael Baum <[email protected]>
Sent: Wednesday, February 23, 2022 3:48 PM
To: [email protected]
Cc: Matan Azrad <[email protected]>; Raslan Darawsheh <[email protected]>; Slava Ovsiienko <[email protected]>
Subject: [PATCH v2 0/5] Refactor mlx5 guidesRecently [1] all the drivers running over the mlx5 device started sharing some objects.
This raised the need to create documentation for the common driver that would contain information relevant to all drivers and in particular:- Tutorial for configuration of the device.
- Instruction for using Mellanox ofed.
- devargs of the common driver.This series adds such documentation and reference to it as parts of the documentation of the other drivers.
[1]
https://patchwork.dpdk.org/project/dpdk/cover/20211019205602.3188203-
[email protected]/v2: Rebase
Michael Baum (5):
doc: remove obsolete explanations from mlx5 guide
doc: replace broken links in mlx5 guide
doc: correct name of BlueField-2 in mlx5 guide
doc: add shared guide for mlx5 PMDs
doc: update doorbell mapping parameter name in mlx5 guidedoc/guides/compressdevs/mlx5.rst | 40 +- doc/guides/cryptodevs/mlx5.rst | 38 +- doc/guides/nics/mlx4.rst | 4 +- doc/guides/nics/mlx5.rst | 567 +----------------- doc/guides/platform/index.rst | 1 + doc/guides/platform/mlx5.rst | 641 +++++++++++++++++++++ doc/guides/regexdevs/mlx5.rst | 44 +- doc/guides/vdpadevs/mlx5.rst | 87 +-- drivers/common/mlx5/linux/mlx5_common_os.c | 2 +- drivers/common/mlx5/mlx5_common.c | 31 +- drivers/common/mlx5/mlx5_common_defs.h | 8 +- drivers/net/mlx5/linux/mlx5_verbs.c | 2 +- drivers/net/mlx5/mlx5_devx.c | 2 +- 13 files changed, 747 insertions(+), 720 deletions(-) create mode 100644
Quoted 4 lines
doc/guides/platform/mlx5.rst--
2.25.1
Series applied to next-net-mlx,
Kindest regards,
Raslan Darawsheh