Skip to contentSkip to conversation

refactore mlx5 guides

PATCH Superseded Michael Baum 3 participants 17 messages Feb 22, 2022 – Feb 25, 2022 Network drivers

lore
Share & tools
lore
#0 lore raw

cover refactore mlx5 guides

Recently [1] all the drivers running over the mlx5 device started sharing some objects.
This raised the need to create documentation for the common driver that would contain information relevant to all drivers and in particular:

- Tutorial for configuration of the device.
- Instruction for using Mellanox ofed.
- devargs of the common driver.

This series adds such documentation and reference to it as parts of the documentation of the other drivers.

[1]
https://patchwork.dpdk.org/project/dpdk/cover/[email protected]/

Michael Baum (5):
doc: remove obsolete explanations from mlx5 guide
doc: replace broken links in mlx5 guide
doc: correct name of BlueField-2 in mlx5 guide
doc: add shared guide for mlx5 PMDs
doc: update doorbell mapping parameter name in mlx5 guide

 doc/guides/compressdevs/mlx5.rst           |  40 +-
 doc/guides/cryptodevs/mlx5.rst             |  38 +-
 doc/guides/nics/mlx4.rst                   |   4 +-
 doc/guides/nics/mlx5.rst                   | 567 +-----------------
 doc/guides/platform/index.rst              |   1 +
 doc/guides/platform/mlx5.rst               | 641 +++++++++++++++++++++
 doc/guides/regexdevs/mlx5.rst              |  44 +-
 doc/guides/vdpadevs/mlx5.rst               |  87 +--
 drivers/common/mlx5/linux/mlx5_common_os.c |   2 +-
 drivers/common/mlx5/mlx5_common.c          |  31 +-
 drivers/common/mlx5/mlx5_common_defs.h     |   8 +-
 drivers/net/mlx5/linux/mlx5_verbs.c        |   2 +-
 drivers/net/mlx5/mlx5_devx.c               |   2 +-
 13 files changed, 747 insertions(+), 720 deletions(-)
 create mode 100644 doc/guides/platform/mlx5.rst
Signature · 1 line
2.25.1
#1 lore raw

PATCH 1/5 doc: remove obsolete explanations from mlx5 guide

Vectorized routines were removed in result of Tx datapath refactoring, and devarg keys documentation was updated.

However, more updating should have been done. In environment variables doc, there was explanation according to vectorized Tx which isn't relevant anymore.

This patch removes this irrelevant explanation.

Fixes a6bd4911ad93 ("net/mlx5: remove Tx implementation")
---
 doc/guides/nics/mlx5.rst | 9 ---------
 1 file changed, 9 deletions(-)
doc/guides/nics/mlx5.rst +0 -9
index c3cc0c0f41..3f5c8bb303 100644
@@ -557,15 +557,6 @@ Environment variables
   The register would be flushed to HW usually when the write-combining buffer
   becomes full, but it depends on CPU design.
 
-  Except for vectorized Tx burst routines, a write memory barrier is enforced
-  after updating the register so that the update can be immediately visible to
-  HW.
-
-  When vectorized Tx burst is called, the barrier is set only if the burst size
-  is not aligned to MLX5_VPMD_TX_MAX_BURST. However, setting this environmental
-  variable will bring better latency even though the maximum throughput can
-  slightly decline.
-
 Run-time configuration
 ~~~~~~~~~~~~~~~~~~~~~~
 
Signature · 1 line
2.25.1
#2 lore raw

PATCH 2/5 doc: replace broken links in mlx5 guide

In reply to Michael Baum

Update links in both mlx4 and mlx5 doc.

---
 doc/guides/nics/mlx4.rst | 4 ++--
 doc/guides/nics/mlx5.rst | 6 +++---
 2 files changed, 5 insertions(+), 5 deletions(-)
doc/guides/nics/mlx4.rst +2 -2
index c81105730d..66493a1157 100644
@@ -14,7 +14,7 @@ the `Mellanox website <http://www.mellanox.com>`_. Help is also provided by
 the `Mellanox community <http://community.mellanox.com/welcome>`_.
 
 There is also a `section dedicated to this poll mode driver
-<http://www.mellanox.com/page/products_dyn?product_family=209&mtag=pmd_for_dpdk>`_.
+<https://developer.nvidia.com/networking/dpdk>`_.
 
 
 Implementation details
@@ -219,7 +219,7 @@ Mellanox OFED as a fallback
 - `Mellanox OFED`_ version: **4.4, 4.5, 4.6**.
 - firmware version: **2.42.5000** and above.
 
-.. _`Mellanox OFED`: http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers
+.. _`Mellanox OFED`: https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/
 
 .. note::
 
doc/guides/nics/mlx5.rst +3 -3
index 3f5c8bb303..c9defd0c60 100644
@@ -19,7 +19,7 @@ Information and documentation about these adapters can be found on the
 `Mellanox community <http://community.mellanox.com/welcome>`__.
 
 There is also a `section dedicated to this poll mode driver
-<http://www.mellanox.com/page/products_dyn?product_family=209&mtag=pmd_for_dpdk>`__.
+<https://developer.nvidia.com/networking/dpdk>`_.
 
 
 Design
@@ -1376,9 +1376,9 @@ managers on most distributions, this PMD requires Ethernet extensions that
 may not be supported at the moment (this is a work in progress).
 
 `Mellanox OFED
-<http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux>`__ and
+<https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/>`__ and
 `Mellanox EN
-<http://www.mellanox.com/page/products_dyn?product_family=27&mtag=linux>`__
+<https://network.nvidia.com/products/ethernet-drivers/linux/mlnx_en/>`__
 include the necessary support and should be used in the meantime. For DPDK,
 only libibverbs, libmlx5, mlnx-ofed-kernel packages and firmware updates are
 required from that distribution.
Signature · 1 line
2.25.1
#3 lore raw

PATCH 3/5 doc: correct name of BlueField-2 in mlx5 guide

In reply to Michael Baum

Update "BlueField 2" -> "BlueField-2" in mlx5 docs.

---
 doc/guides/compressdevs/mlx5.rst | 6 +++---
 doc/guides/cryptodevs/mlx5.rst   | 6 +++---
 doc/guides/nics/mlx5.rst         | 2 +-
 doc/guides/regexdevs/mlx5.rst    | 8 ++++----
 4 files changed, 11 insertions(+), 11 deletions(-)
doc/guides/compressdevs/mlx5.rst +3 -3
index a4e17f65b3..7f2d6bdfff 100644
@@ -7,7 +7,7 @@ MLX5 compress driver
 ====================
 
 The MLX5 compress driver library
-(**librte_compress_mlx5**) provides support for **Mellanox BlueField 2**
+(**librte_compress_mlx5**) provides support for **Mellanox BlueField-2**
 families of 25/50/100/200 Gb/s adapters.
 
 Design
@@ -21,7 +21,7 @@ These engines are part of the ARM complex of the BlueField chip, and as
 such they do not use NIC related resources (e.g. RX/TX bandwidth).
 They do share the same PCI and memory bandwidth.
 
-So, using the BlueField device (starting from BlueField 2), the compress
+So, using the BlueField device (starting from BlueField-2), the compress
 class operations can be supported in parallel to the net, vDPA and
 RegEx class operations.
 
@@ -95,7 +95,7 @@ Driver options
 Supported NICs
 --------------
 
-* Mellanox\ |reg| BlueField 2 SmartNIC
+* Mellanox\ |reg| BlueField-2 SmartNIC
 
 Prerequisites
 -------------
doc/guides/cryptodevs/mlx5.rst +3 -3
index e86a6205e8..9936556cc9 100644
@@ -88,7 +88,7 @@ The mlxreg dedicated tool should be used as follows:
   should not be specified.
 
   All the device ports should set it in order to move to operational mode.
-  For BlueField 2, the internal ports in the ARM system should also be set.
+  For BlueField-2, the internal ports in the ARM system should also be set.
 
 - Query CRYPTO_OPERATIONAL register to make sure the device is in Operational
   mode.
@@ -142,7 +142,7 @@ Supported NICs
 --------------
 
 * Mellanox\ |reg| ConnectX\ |reg|-6 200G MCX654106A-HCAT (2x200G)
-* Mellanox\ |reg| BlueField 2 SmartNIC
+* Mellanox\ |reg| BlueField-2 SmartNIC
 * Mellanox\ |reg| ConnectX\ |reg|-6 Dx
 
 
@@ -162,7 +162,7 @@ FW Prerequisites
 ~~~~~~~~~~~~~~~~
 
 - xx.31.0328 for ConnectX-6.
-- xx.32.0108 for ConnectX-6 Dx and BlueField 2.
+- xx.32.0108 for ConnectX-6 Dx and BlueField-2.
 
 Linux Prerequisites
 ~~~~~~~~~~~~~~~~~~~
doc/guides/nics/mlx5.rst +1 -1
index c9defd0c60..b7ad55561c 100644
@@ -253,7 +253,7 @@ Limitations
 
 - Flex item:
 
-  - Hardware support: BlueField 2.
+  - Hardware support: BlueField-2.
   - Flex item is supported on PF only.
   - Hardware limits ``header_length_mask_width`` up to 6 bits.
   - Firmware supports 8 global sample fields.
doc/guides/regexdevs/mlx5.rst +4 -4
index b2bf0afd01..fc2cacba60 100644
@@ -7,7 +7,7 @@ MLX5 RegEx driver
 =================
 
 The MLX5 RegEx (Regular Expression) driver library
-(**librte_regex_mlx5**) provides support for **Mellanox BlueField 2**
+(**librte_regex_mlx5**) provides support for **Mellanox BlueField-2**
 families of 25/50/100/200 Gb/s adapters.
 
 Design
@@ -43,13 +43,13 @@ Features
 Supported NICs
 --------------
 
-* Mellanox\ |reg| BlueField 2 SmartNIC
+* Mellanox\ |reg| BlueField-2 SmartNIC
 
 Prerequisites
 -------------
 
-- BlueField 2 running Mellanox supported kernel.
-- Enable the RegEx capabilities using system call from the BlueField 2.
+- BlueField-2 running Mellanox supported kernel.
+- Enable the RegEx capabilities using system call from the BlueField-2.
 - Official support is not yet released.
 
 Limitations
Signature · 1 line
2.25.1
#4 lore raw

PATCH 4/5 doc: add shared guide for mlx5 PMDs

In reply to Michael Baum

Adds new documentation for MLX5 common driver that contains:
- Its features list (doesn't exist for now).
- Its devargs description.
- Device configuration information and tutorial.
- Quick Start Guide for Mellanox OFED/EN.

Move into this doc all shared information from other MLX5 PMD docs and add them reference to new common doc.

---
 doc/guides/compressdevs/mlx5.rst |  34 +-
 doc/guides/cryptodevs/mlx5.rst   |  32 +-
 doc/guides/nics/mlx5.rst         | 522 ++-------------------------
 doc/guides/platform/index.rst    |   1 +
 doc/guides/platform/mlx5.rst     | 602 +++++++++++++++++++++++++++++++
 doc/guides/regexdevs/mlx5.rst    |  36 +-
 doc/guides/vdpadevs/mlx5.rst     |  87 +----
 7 files changed, 662 insertions(+), 652 deletions(-)
 create mode 100644 doc/guides/platform/mlx5.rst
doc/guides/compressdevs/mlx5.rst +7 -27
index 7f2d6bdfff..273251e088 100644
@@ -3,10 +3,10 @@
 
 .. include:: <isonum.txt>
 
-MLX5 compress driver
+MLX5 Compress Driver
 ====================
 
-The MLX5 compress driver library
+The mlx5 compress driver library
 (**librte_compress_mlx5**) provides support for **Mellanox BlueField-2**
 families of 25/50/100/200 Gb/s adapters.
 
@@ -25,30 +25,7 @@ So, using the BlueField device (starting from BlueField-2), the compress
 class operations can be supported in parallel to the net, vDPA and
 RegEx class operations.
 
-For security reasons and robustness, this driver only deals with virtual
-memory addresses. The way resources allocations are handled by the kernel,
-combined with hardware specifications that allow to handle virtual memory
-addresses directly, ensure that DPDK applications cannot access random
-physical memory (or memory that does not belong to the current process).
-
-The PMD uses libibverbs and libmlx5 to access the device firmware
-or directly the hardware components.
-There are different levels of objects and bypassing abilities
-to get the best performances:
-
-- Verbs is a complete high-level generic API.
-- Direct Verbs is a device-specific API.
-- DevX allows to access firmware objects.
-
-Enabling librte_compress_mlx5 causes DPDK applications to be linked against
-libibverbs.
-
-Mellanox mlx5 PCI device can be probed by number of different PCI devices,
-for example net / vDPA / RegEx. To select the compress PMD ``class=compress``
-should be specified as device parameter. The compress device can be probed and
-used with other Mellanox classes, by adding more options in the class.
-For example: ``class=net:compress`` will probe both the net PMD and the compress
-PMD.
+See :doc:`../../platform/mlx5` guide for more design details.
 
 Features
 --------
@@ -85,6 +62,9 @@ Limitations
 Driver options
 --------------
 
+Please refer to :ref:`mlx5 common options <mlx5_common_driver_options>`
+for an additional list of options shared with other mlx5 drivers.
+
 - ``log-block-size`` parameter [int]
 
   Log of the Huffman block size in the Deflate algorithm.
@@ -101,4 +81,4 @@ Prerequisites
 -------------
 
 - Mellanox OFED version: **5.2**
-  see :doc:`../../nics/mlx5` guide for more Mellanox OFED details.
+  See :ref:`mlx5 common prerequisites <mlx5_linux_prerequisites>` for more details.
doc/guides/cryptodevs/mlx5.rst +9 -23
index 9936556cc9..ef47aa65dd 100644
@@ -28,23 +28,12 @@ when the MKEY is configured to perform crypto operations.
 
 The encryption does not require text to be aligned to the AES block size (128b).
 
-For security reasons and to increase robustness, this driver only deals with virtual
-memory addresses. The way resources allocations are handled by the kernel,
-combined with hardware specifications that allow handling virtual memory
-addresses directly, ensure that DPDK applications cannot access random
-physical memory (or memory that does not belong to the current process).
+See :doc:`../../platform/mlx5` guide for more design details.
 
-The PMD uses ``libibverbs`` and ``libmlx5`` to access the device firmware
-or to access the hardware components directly.
-There are different levels of objects and bypassing abilities.
-To get the best performances:
-
-- Verbs is a complete high-level generic API (Linux only).
-- Direct Verbs is a device-specific API (Linux only).
-- DevX allows to access firmware objects.
+Configuration
+-------------
 
-Enabling ``librte_crypto_mlx5`` causes DPDK applications
-to be linked against libibverbs on Linux OS.
+See the :ref:`mlx5 common configuration <mlx5_common_env>`.
 
 In order to move the device to crypto operational mode, credential and KEK
 (Key Encrypting Key) should be set as the first step.
@@ -109,10 +98,8 @@ The mlxreg dedicated tool should be used as follows:
 Driver options
 --------------
 
-- ``class`` parameter [string]
-
-  Select the class of the driver that should probe the device.
-  `crypto` for the mlx5 crypto driver.
+Please refer to :ref:`mlx5 common options <mlx5_common_driver_options>`
+for an additional list of options shared with other mlx5 drivers.
 
 - ``wcs_file`` parameter [string] - mandatory
 
@@ -168,13 +155,12 @@ Linux Prerequisites
 ~~~~~~~~~~~~~~~~~~~
 
 - Mellanox OFED version: **5.3**.
-  see :doc:`../../nics/mlx5` guide for more Mellanox OFED details.
-
 - Compilation can be done also with rdma-core v15+.
-  see :doc:`../../nics/mlx5` guide for more rdma-core details.
+
+  See :ref:`mlx5 common prerequisites <mlx5_linux_prerequisites>` for more details.
 
 Windows Prerequisites
 ~~~~~~~~~~~~~~~~~~~~~
 
 - Mellanox WINOF-2 version: **2.60** or higher.
-  see :doc:`../../nics/mlx5` guide for more Mellanox WINOF-2 details.
+  See :ref:`mlx5 common prerequisites <mlx5_windows_prerequisites>` for more details.
doc/guides/nics/mlx5.rst +24 -498

597 lines not on this page · raw message

doc/guides/platform/index.rst +1 -0
index 2ff91a6018..b1ba11df73 100644
@@ -14,4 +14,5 @@ The following are platform specific guides and setup information.
     cnxk
     dpaa
     dpaa2
+    mlx5
     octeontx
doc/guides/platform/mlx5.rst new file +602 -0

602 lines not on this page · raw message

doc/guides/regexdevs/mlx5.rst +12 -24
index fc2cacba60..83280bda40 100644
@@ -3,10 +3,10 @@
 
 .. include:: <isonum.txt>
 
-MLX5 RegEx driver
+MLX5 RegEx Driver
 =================
 
-The MLX5 RegEx (Regular Expression) driver library
+The mlx5 RegEx (Regular Expression) driver library
 (**librte_regex_mlx5**) provides support for **Mellanox BlueField-2**
 families of 25/50/100/200 Gb/s adapters.
 
@@ -17,29 +17,21 @@ This PMD is configuring the RegEx HW engine.
 For the PMD to work, the application must supply
 a precompiled rule file in rof2 format.
 
-The PMD uses libibverbs and libmlx5 to access the device firmware
-or directly the hardware components.
-There are different levels of objects and bypassing abilities
-to get the best performances:
-
-- Verbs is a complete high-level generic API
-- Direct Verbs is a device-specific API
-- DevX allows to access firmware objects
-
-Enabling librte_regex_mlx5 causes DPDK applications to be linked against
-libibverbs.
-
-Mellanox mlx5 pci device can be probed by number of different pci devices,
-for example net / vDPA / RegEx. To select the RegEx PMD ``class=regex`` should
-be specified as device parameter. The RegEx device can be probed and used with
-other Mellanox devices, by adding more options in the class.
-For example: ``class=net:regex`` will probe both the net PMD and the RegEx PMD.
+See :doc:`../../platform/mlx5` guide for more design details.
 
 Features
 --------
 
 - Multi segments mbuf support.
 
+Configuration
+-------------
+
+See :ref:`mlx5 common compilation <mlx5_common_compilation>`,
+:ref:`mlx5 firmware configuration <mlx5_firmware_config>`,
+and :ref:`mlx5 common driver options <mlx5_common_driver_options>`.
+
+
 Supported NICs
 --------------
 
@@ -52,12 +44,8 @@ Prerequisites
 - Enable the RegEx capabilities using system call from the BlueField-2.
 - Official support is not yet released.
 
+
 Limitations
 -----------
 
 - The firmware version must be greater than XX.31.0364
-
-Run-time configuration
-~~~~~~~~~~~~~~~~~~~~~~
-
-- **ethtool** operations on related kernel interfaces also affect the PMD.
doc/guides/vdpadevs/mlx5.rst +7 -80

120 lines not on this page · raw message

Signature · 1 line
2.25.1
#5 lore raw

PATCH 5/5 doc: update doorbell mapping parameter name in mlx5 guide

In reply to Michael Baum

The "tx_db_nc" devarg forces doorbell register mapping to non-cached region eliminating the extra write memory barrier. This argument was used in creating the UAR for Tx and thus affected its performance.

Recently [1] its use has been extended to all UAR creation in all mlx5 drivers, and now its name is no longer so accurate.

This patch changes its name to "sq_db_nc" to suit any send queue that uses it. The old name will still work for backward compatibility.

[1] commit 5dfa003db53f ("common/mlx5: fix post doorbell barrier")

---
 doc/guides/nics/mlx5.rst                   | 34 ++-----------------
 doc/guides/platform/mlx5.rst               | 39 ++++++++++++++++++++++
 drivers/common/mlx5/linux/mlx5_common_os.c |  2 +-
 drivers/common/mlx5/mlx5_common.c          | 31 ++++++++++++-----
 drivers/common/mlx5/mlx5_common_defs.h     |  8 ++---
 drivers/net/mlx5/linux/mlx5_verbs.c        |  2 +-
 drivers/net/mlx5/mlx5_devx.c               |  2 +-
 7 files changed, 72 insertions(+), 46 deletions(-)
doc/guides/nics/mlx5.rst +3 -31
index 8eb67efcf3..968aac01d2 100644
@@ -814,37 +814,9 @@ for an additional list of options shared with other mlx5 drivers.
 
 - ``tx_db_nc`` parameter [int]
 
-  The rdma core library can map doorbell register in two ways, depending on the
-  environment variable "MLX5_SHUT_UP_BF":
-
-  - As regular cached memory (usually with write combining attribute), if the
-    variable is either missing or set to zero.
-  - As non-cached memory, if the variable is present and set to not "0" value.
-
-  The type of mapping may slightly affect the Tx performance, the optimal choice
-  is strongly relied on the host architecture and should be deduced practically.
-
-  If ``tx_db_nc`` is set to zero, the doorbell is forced to be mapped to regular
-  memory (with write combining), the PMD will perform the extra write memory barrier
-  after writing to doorbell, it might increase the needed CPU clocks per packet
-  to send, but latency might be improved.
-
-  If ``tx_db_nc`` is set to one, the doorbell is forced to be mapped to non
-  cached memory, the PMD will not perform the extra write memory barrier
-  after writing to doorbell, on some architectures it might improve the
-  performance.
-
-  If ``tx_db_nc`` is set to two, the doorbell is forced to be mapped to regular
-  memory, the PMD will use heuristics to decide whether write memory barrier
-  should be performed. For bursts with size multiple of recommended one (64 pkts)
-  it is supposed the next burst is coming and no need to issue the extra memory
-  barrier (it is supposed to be issued in the next coming burst, at least after
-  descriptor writing). It might increase latency (on some hosts till next
-  packets transmit) and should be used with care.
-
-  If ``tx_db_nc`` is omitted or set to zero, the preset (if any) environment
-  variable "MLX5_SHUT_UP_BF" value is used. If there is no "MLX5_SHUT_UP_BF",
-  the default ``tx_db_nc`` value is zero for ARM64 hosts and one for others.
+  This parameter name is deprecated and ignored.
+  The new name for this parameter is ``sq_db_nc``.
+  See :ref:`common driver options <mlx5_common_driver_options>`.
 
 - ``tx_pp`` parameter [int]
 
doc/guides/platform/mlx5.rst +39 -0
index 0fd5e6604d..d073c213ca 100644
@@ -600,3 +600,42 @@ and below are the arguments supported by the common mlx5 layer.
   from system by default, without explicit rte memory flag.
 
   By default, the PMD will set this value to 0.
+
+- ``sq_db_nc`` parameter [int]
+
+  The rdma core library can map doorbell register in two ways,
+  depending on the environment variable "MLX5_SHUT_UP_BF":
+
+  - As regular cached memory (usually with write combining attribute),
+    if the variable is either missing or set to zero.
+  - As non-cached memory, if the variable is present and set to not "0" value.
+
+   The same doorbell mapping approach is implemented directly by PMD
+   in UAR generation for queues created with DevX.
+
+  The type of mapping may slightly affect the send queue performance,
+  the optimal choice strongly relied on the host architecture
+  and should be deduced practically.
+
+  If ``sq_db_nc`` is set to zero, the doorbell is forced to be mapped to
+  regular memory (with write combining), the PMD will perform the extra write
+  memory barrier after writing to doorbell, it might increase the needed CPU
+  clocks per packet to send, but latency might be improved.
+
+  If ``sq_db_nc`` is set to one, the doorbell is forced to be mapped to non
+  cached memory, the PMD will not perform the extra write memory barrier after
+  writing to doorbell, on some architectures it might improve the performance.
+
+  If ``sq_db_nc`` is set to two, the doorbell is forced to be mapped to
+  regular memory, the PMD will use heuristics to decide whether a write memory
+  barrier should be performed. For bursts with size multiple of recommended one
+  (64 pkts) it is supposed the next burst is coming and no need to issue the
+  extra memory barrier (it is supposed to be issued in the next coming burst,
+  at least after descriptor writing). It might increase latency (on some hosts
+  till the next packets transmit) and should be used with care.
+  The PMD uses heuristics only for Tx queue, for other semd queues the doorbell
+  is forced to be mapped to regular memory as same as ``sq_db_nc`` is set to 0.
+
+  If ``sq_db_nc`` is omitted, the preset (if any) environment variable
+  "MLX5_SHUT_UP_BF" value is used. If there is no "MLX5_SHUT_UP_BF", the
+  default ``sq_db_nc`` value is zero for ARM64 hosts and one for others.
\ No newline at end of file
drivers/common/mlx5/linux/mlx5_common_os.c +1 -1
index 0d3e24e04e..a752d79e8e 100644
@@ -630,7 +630,7 @@ mlx5_config_doorbell_mapping_env(int dbnc)
 		setenv(MLX5_SHUT_UP_BF, MLX5_SHUT_UP_BF_DEFAULT, 1);
 	else
 		setenv(MLX5_SHUT_UP_BF,
-		       dbnc == MLX5_TXDB_NCACHED ? "1" : "0", 1);
+		       dbnc == MLX5_SQ_DB_NCACHED ? "1" : "0", 1);
 	return value;
 }
 
drivers/common/mlx5/mlx5_common.c +23 -8
index 96906d3f39..8cf391df13 100644
@@ -35,10 +35,17 @@ uint8_t haswell_broadwell_cpu;
 
 /*
  * Device parameter to force doorbell register mapping
- * to non-cahed region eliminating the extra write memory barrier.
+ * to non-cached region eliminating the extra write memory barrier.
+ * Deprecated, ignored (Name changed to sq_db_nc).
  */
 #define MLX5_TX_DB_NC "tx_db_nc"
 
+/*
+ * Device parameter to force doorbell register mapping
+ * to non-cached region eliminating the extra write memory barrier.
+ */
+#define MLX5_SQ_DB_NC "sq_db_nc"
+
 /* In case this is an x86_64 intel processor to check if
  * we should use relaxed ordering.
  */
@@ -255,11 +262,17 @@ mlx5_common_args_check_handler(const char *key, const char *val, void *opaque)
 		DRV_LOG(WARNING, "%s: \"%s\" is an invalid integer.", key, val);
 		return -rte_errno;
 	}
-	if (strcmp(key, MLX5_TX_DB_NC) == 0) {
-		if (tmp != MLX5_TXDB_CACHED &&
-		    tmp != MLX5_TXDB_NCACHED &&
-		    tmp != MLX5_TXDB_HEURISTIC) {
-			DRV_LOG(ERR, "Invalid Tx doorbell mapping parameter.");
+	if (strcmp(key, MLX5_TX_DB_NC) == 0)
+		DRV_LOG(WARNING,
+			"%s: deprecated parameter, converted to queue_db_nc",
+			key);
+	if (strcmp(key, MLX5_SQ_DB_NC) == 0 ||
+	    strcmp(key, MLX5_TX_DB_NC) == 0) {
+		if (tmp != MLX5_SQ_DB_CACHED &&
+		    tmp != MLX5_SQ_DB_NCACHED &&
+		    tmp != MLX5_SQ_DB_HEURISTIC) {
+			DRV_LOG(ERR,
+				"Invalid Send Queue doorbell mapping parameter.");
 			rte_errno = EINVAL;
 			return -rte_errno;
 		}
@@ -293,6 +306,7 @@ mlx5_common_config_get(struct mlx5_kvargs_ctrl *mkvlist,
 		RTE_DEVARGS_KEY_CLASS,
 		MLX5_DRIVER_KEY,
 		MLX5_TX_DB_NC,
+		MLX5_SQ_DB_NC,
 		MLX5_MR_EXT_MEMSEG_EN,
 		MLX5_SYS_MEM_EN,
 		MLX5_MR_MEMPOOL_REG_EN,
@@ -317,7 +331,8 @@ mlx5_common_config_get(struct mlx5_kvargs_ctrl *mkvlist,
 	DRV_LOG(DEBUG, "mr_ext_memseg_en is %u.", config->mr_ext_memseg_en);
 	DRV_LOG(DEBUG, "mr_mempool_reg_en is %u.", config->mr_mempool_reg_en);
 	DRV_LOG(DEBUG, "sys_mem_en is %u.", config->sys_mem_en);
-	DRV_LOG(DEBUG, "Tx doorbell mapping parameter is %d.", config->dbnc);
+	DRV_LOG(DEBUG, "Send Queue doorbell mapping parameter is %d.",
+		config->dbnc);
 	return ret;
 }
 
@@ -1231,7 +1246,7 @@ mlx5_devx_alloc_uar(struct mlx5_common_device *cdev)
 	for (retry = 0; retry < MLX5_ALLOC_UAR_RETRY; ++retry) {
 #ifdef MLX5DV_UAR_ALLOC_TYPE_NC
 		/* Control the mapping type according to the settings. */
-		uar_mapping = (cdev->config.dbnc == MLX5_TXDB_NCACHED) ?
+		uar_mapping = (cdev->config.dbnc == MLX5_SQ_DB_NCACHED) ?
 			    MLX5DV_UAR_ALLOC_TYPE_NC : MLX5DV_UAR_ALLOC_TYPE_BF;
 #else
 		/*
drivers/common/mlx5/mlx5_common_defs.h +4 -4
index ca80cd8d29..68b700dc0b 100644
@@ -34,10 +34,10 @@
 /* Default PMD specific parameter value. */
 #define MLX5_ARG_UNSET (-1)
 
-/* MLX5_TX_DB_NC supported values. */
-#define MLX5_TXDB_CACHED 0
-#define MLX5_TXDB_NCACHED 1
-#define MLX5_TXDB_HEURISTIC 2
+/* MLX5_SQ_DB_NC supported values. */
+#define MLX5_SQ_DB_CACHED 0
+#define MLX5_SQ_DB_NCACHED 1
+#define MLX5_SQ_DB_HEURISTIC 2
 
 /* Fields of memory mapping type in offset parameter of mmap() */
 #define MLX5_UAR_MMAP_CMD_SHIFT 8
drivers/net/mlx5/linux/mlx5_verbs.c +1 -1
index 73c44138de..dfbc5a1e08 100644
@@ -926,7 +926,7 @@ mlx5_txq_ibv_uar_init(struct mlx5_txq_ctrl *txq_ctrl, void *bf_reg)
 		DRV_LOG(ERR, "Failed to get mem page size");
 		rte_errno = ENOMEM;
 	}
-	txq->db_heu = priv->sh->cdev->config.dbnc == MLX5_TXDB_HEURISTIC;
+	txq->db_heu = priv->sh->cdev->config.dbnc == MLX5_SQ_DB_HEURISTIC;
 	txq->db_nc = mlx5_db_map_type_get(uar_mmap_offset, page_size);
 	ppriv->uar_table[txq->idx].db = bf_reg;
 #ifndef RTE_ARCH_64
drivers/net/mlx5/mlx5_devx.c +1 -1
index e57787cfec..f18b18b1a2 100644
@@ -1326,7 +1326,7 @@ mlx5_txq_devx_obj_new(struct rte_eth_dev *dev, uint16_t idx)
 	txq_data->qp_db = &txq_obj->sq_obj.db_rec[MLX5_SND_DBR];
 	*txq_data->qp_db = 0;
 	txq_data->qp_num_8s = txq_obj->sq_obj.sq->id << 8;
-	txq_data->db_heu = sh->cdev->config.dbnc == MLX5_TXDB_HEURISTIC;
+	txq_data->db_heu = sh->cdev->config.dbnc == MLX5_SQ_DB_HEURISTIC;
 	txq_data->db_nc = sh->tx_uar.dbnc;
 	/* Change Send Queue state to Ready-to-Send. */
 	ret = mlx5_txq_devx_modify(txq_obj, MLX5_TXQ_MOD_RST2RDY, 0);
Signature · 1 line
2.25.1
Reviewed-by on the v1 series
#6 lore raw
In reply to Michael Baum

Hi,

Quoted 45 lines

-----Original Message-----
From: Michael Baum <[email protected]>
Sent: Tuesday, February 22, 2022 2:48 PM
To: [email protected]
Cc: Matan Azrad <[email protected]>; Raslan Darawsheh <[email protected]>; Slava Ovsiienko <[email protected]>
Subject: [PATCH 0/5] refactore mlx5 guides

Recently [1] all the drivers running over the mlx5 device started sharing some objects.
This raised the need to create documentation for the common driver that would contain information relevant to all drivers and in particular:

- Tutorial for configuration of the device.
- Instruction for using Mellanox ofed.
- devargs of the common driver.

This series adds such documentation and reference to it as parts of the documentation of the other drivers.

[1]
https://patchwork.dpdk.org/project/dpdk/cover/20211019205602.3188203-
[email protected]/

Michael Baum (5):
doc: remove obsolete explanations from mlx5 guide
doc: replace broken links in mlx5 guide
doc: correct name of BlueField-2 in mlx5 guide
doc: add shared guide for mlx5 PMDs
doc: update doorbell mapping parameter name in mlx5 guide

 doc/guides/compressdevs/mlx5.rst           |  40 +-
 doc/guides/cryptodevs/mlx5.rst             |  38 +-
 doc/guides/nics/mlx4.rst                   |   4 +-
 doc/guides/nics/mlx5.rst                   | 567 +-----------------
 doc/guides/platform/index.rst              |   1 +
 doc/guides/platform/mlx5.rst               | 641 +++++++++++++++++++++
 doc/guides/regexdevs/mlx5.rst              |  44 +-
 doc/guides/vdpadevs/mlx5.rst               |  87 +--
 drivers/common/mlx5/linux/mlx5_common_os.c |   2 +-
 drivers/common/mlx5/mlx5_common.c          |  31 +-
 drivers/common/mlx5/mlx5_common_defs.h     |   8 +-
 drivers/net/mlx5/linux/mlx5_verbs.c        |   2 +-
 drivers/net/mlx5/mlx5_devx.c               |   2 +-
 13 files changed, 747 insertions(+), 720 deletions(-)  create mode 100644
Quoted 4 lines

doc/guides/platform/mlx5.rst

--
2.25.1

For the series,

Kindest regards,
Raslan Darawsheh

#7 lore raw

v2 cover Refactor mlx5 guides

In reply to Michael Baum

Recently [1] all the drivers running over the mlx5 device started sharing some objects.
This raised the need to create documentation for the common driver that would contain information relevant to all drivers and in particular:

- Tutorial for configuration of the device.
- Instruction for using Mellanox ofed.
- devargs of the common driver.

This series adds such documentation and reference to it as parts of the documentation of the other drivers.

[1]
https://patchwork.dpdk.org/project/dpdk/cover/[email protected]/

v2: Rebase

Michael Baum (5):
doc: remove obsolete explanations from mlx5 guide
doc: replace broken links in mlx5 guide
doc: correct name of BlueField-2 in mlx5 guide
doc: add shared guide for mlx5 PMDs
doc: update doorbell mapping parameter name in mlx5 guide

 doc/guides/compressdevs/mlx5.rst           |  40 +-
 doc/guides/cryptodevs/mlx5.rst             |  38 +-
 doc/guides/nics/mlx4.rst                   |   4 +-
 doc/guides/nics/mlx5.rst                   | 567 +-----------------
 doc/guides/platform/index.rst              |   1 +
 doc/guides/platform/mlx5.rst               | 641 +++++++++++++++++++++
 doc/guides/regexdevs/mlx5.rst              |  44 +-
 doc/guides/vdpadevs/mlx5.rst               |  87 +--
 drivers/common/mlx5/linux/mlx5_common_os.c |   2 +-
 drivers/common/mlx5/mlx5_common.c          |  31 +-
 drivers/common/mlx5/mlx5_common_defs.h     |   8 +-
 drivers/net/mlx5/linux/mlx5_verbs.c        |   2 +-
 drivers/net/mlx5/mlx5_devx.c               |   2 +-
 13 files changed, 747 insertions(+), 720 deletions(-)
 create mode 100644 doc/guides/platform/mlx5.rst
Signature · 1 line
2.25.1
#8 lore raw

PATCH v2 1/5 doc: remove obsolete explanations from mlx5 guide

Vectorized routines were removed in result of Tx datapath refactoring, and devarg keys documentation was updated.

However, more updating should have been done. In environment variables doc, there was explanation according to vectorized Tx which isn't relevant anymore.

This patch removes this irrelevant explanation.

Fixes a6bd4911ad93 ("net/mlx5: remove Tx implementation")
---
 doc/guides/nics/mlx5.rst | 9 ---------
 1 file changed, 9 deletions(-)
doc/guides/nics/mlx5.rst +0 -9
index 6494f4ae39..c21df81717 100644
@@ -557,15 +557,6 @@ Environment variables
   The register would be flushed to HW usually when the write-combining buffer
   becomes full, but it depends on CPU design.
 
-  Except for vectorized Tx burst routines, a write memory barrier is enforced
-  after updating the register so that the update can be immediately visible to
-  HW.
-
-  When vectorized Tx burst is called, the barrier is set only if the burst size
-  is not aligned to MLX5_VPMD_TX_MAX_BURST. However, setting this environmental
-  variable will bring better latency even though the maximum throughput can
-  slightly decline.
-
 Run-time configuration
 ~~~~~~~~~~~~~~~~~~~~~~
 
Signature · 1 line
2.25.1
#9 lore raw

PATCH v2 2/5 doc: replace broken links in mlx5 guide

In reply to Michael Baum

Update links in both mlx4 and mlx5 doc.

---
 doc/guides/nics/mlx4.rst | 4 ++--
 doc/guides/nics/mlx5.rst | 6 +++---
 2 files changed, 5 insertions(+), 5 deletions(-)
doc/guides/nics/mlx4.rst +2 -2
index c81105730d..66493a1157 100644
@@ -14,7 +14,7 @@ the `Mellanox website <http://www.mellanox.com>`_. Help is also provided by
 the `Mellanox community <http://community.mellanox.com/welcome>`_.
 
 There is also a `section dedicated to this poll mode driver
-<http://www.mellanox.com/page/products_dyn?product_family=209&mtag=pmd_for_dpdk>`_.
+<https://developer.nvidia.com/networking/dpdk>`_.
 
 
 Implementation details
@@ -219,7 +219,7 @@ Mellanox OFED as a fallback
 - `Mellanox OFED`_ version: **4.4, 4.5, 4.6**.
 - firmware version: **2.42.5000** and above.
 
-.. _`Mellanox OFED`: http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers
+.. _`Mellanox OFED`: https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/
 
 .. note::
 
doc/guides/nics/mlx5.rst +3 -3
index c21df81717..a15f1620ff 100644
@@ -19,7 +19,7 @@ Information and documentation about these adapters can be found on the
 `Mellanox community <http://community.mellanox.com/welcome>`__.
 
 There is also a `section dedicated to this poll mode driver
-<http://www.mellanox.com/page/products_dyn?product_family=209&mtag=pmd_for_dpdk>`__.
+<https://developer.nvidia.com/networking/dpdk>`_.
 
 
 Design
@@ -1381,9 +1381,9 @@ managers on most distributions, this PMD requires Ethernet extensions that
 may not be supported at the moment (this is a work in progress).
 
 `Mellanox OFED
-<http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux>`__ and
+<https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/>`__ and
 `Mellanox EN
-<http://www.mellanox.com/page/products_dyn?product_family=27&mtag=linux>`__
+<https://network.nvidia.com/products/ethernet-drivers/linux/mlnx_en/>`__
 include the necessary support and should be used in the meantime. For DPDK,
 only libibverbs, libmlx5, mlnx-ofed-kernel packages and firmware updates are
 required from that distribution.
Signature · 1 line
2.25.1
#10 lore raw

PATCH v2 3/5 doc: correct name of BlueField-2 in mlx5 guide

In reply to Michael Baum

Update "BlueField 2" -> "BlueField-2" in mlx5 docs.

---
 doc/guides/compressdevs/mlx5.rst | 6 +++---
 doc/guides/cryptodevs/mlx5.rst   | 6 +++---
 doc/guides/nics/mlx5.rst         | 2 +-
 doc/guides/regexdevs/mlx5.rst    | 8 ++++----
 4 files changed, 11 insertions(+), 11 deletions(-)
doc/guides/compressdevs/mlx5.rst +3 -3
index a4e17f65b3..7f2d6bdfff 100644
@@ -7,7 +7,7 @@ MLX5 compress driver
 ====================
 
 The MLX5 compress driver library
-(**librte_compress_mlx5**) provides support for **Mellanox BlueField 2**
+(**librte_compress_mlx5**) provides support for **Mellanox BlueField-2**
 families of 25/50/100/200 Gb/s adapters.
 
 Design
@@ -21,7 +21,7 @@ These engines are part of the ARM complex of the BlueField chip, and as
 such they do not use NIC related resources (e.g. RX/TX bandwidth).
 They do share the same PCI and memory bandwidth.
 
-So, using the BlueField device (starting from BlueField 2), the compress
+So, using the BlueField device (starting from BlueField-2), the compress
 class operations can be supported in parallel to the net, vDPA and
 RegEx class operations.
 
@@ -95,7 +95,7 @@ Driver options
 Supported NICs
 --------------
 
-* Mellanox\ |reg| BlueField 2 SmartNIC
+* Mellanox\ |reg| BlueField-2 SmartNIC
 
 Prerequisites
 -------------
doc/guides/cryptodevs/mlx5.rst +3 -3
index e86a6205e8..9936556cc9 100644
@@ -88,7 +88,7 @@ The mlxreg dedicated tool should be used as follows:
   should not be specified.
 
   All the device ports should set it in order to move to operational mode.
-  For BlueField 2, the internal ports in the ARM system should also be set.
+  For BlueField-2, the internal ports in the ARM system should also be set.
 
 - Query CRYPTO_OPERATIONAL register to make sure the device is in Operational
   mode.
@@ -142,7 +142,7 @@ Supported NICs
 --------------
 
 * Mellanox\ |reg| ConnectX\ |reg|-6 200G MCX654106A-HCAT (2x200G)
-* Mellanox\ |reg| BlueField 2 SmartNIC
+* Mellanox\ |reg| BlueField-2 SmartNIC
 * Mellanox\ |reg| ConnectX\ |reg|-6 Dx
 
 
@@ -162,7 +162,7 @@ FW Prerequisites
 ~~~~~~~~~~~~~~~~
 
 - xx.31.0328 for ConnectX-6.
-- xx.32.0108 for ConnectX-6 Dx and BlueField 2.
+- xx.32.0108 for ConnectX-6 Dx and BlueField-2.
 
 Linux Prerequisites
 ~~~~~~~~~~~~~~~~~~~
doc/guides/nics/mlx5.rst +1 -1
index a15f1620ff..9e10a68d6f 100644
@@ -253,7 +253,7 @@ Limitations
 
 - Flex item:
 
-  - Hardware support: BlueField 2.
+  - Hardware support: BlueField-2.
   - Flex item is supported on PF only.
   - Hardware limits ``header_length_mask_width`` up to 6 bits.
   - Firmware supports 8 global sample fields.
doc/guides/regexdevs/mlx5.rst +4 -4
index b2bf0afd01..fc2cacba60 100644
@@ -7,7 +7,7 @@ MLX5 RegEx driver
 =================
 
 The MLX5 RegEx (Regular Expression) driver library
-(**librte_regex_mlx5**) provides support for **Mellanox BlueField 2**
+(**librte_regex_mlx5**) provides support for **Mellanox BlueField-2**
 families of 25/50/100/200 Gb/s adapters.
 
 Design
@@ -43,13 +43,13 @@ Features
 Supported NICs
 --------------
 
-* Mellanox\ |reg| BlueField 2 SmartNIC
+* Mellanox\ |reg| BlueField-2 SmartNIC
 
 Prerequisites
 -------------
 
-- BlueField 2 running Mellanox supported kernel.
-- Enable the RegEx capabilities using system call from the BlueField 2.
+- BlueField-2 running Mellanox supported kernel.
+- Enable the RegEx capabilities using system call from the BlueField-2.
 - Official support is not yet released.
 
 Limitations
Signature · 1 line
2.25.1
#11 lore raw

PATCH v2 4/5 doc: add shared guide for mlx5 PMDs

In reply to Michael Baum

Adds new documentation for MLX5 common driver that contains:
- Its features list (doesn't exist for now).
- Its devargs description.
- Device configuration information and tutorial.
- Quick Start Guide for Mellanox OFED/EN.

Move into this doc all shared information from other MLX5 PMD docs and add them reference to new common doc.

---
 doc/guides/compressdevs/mlx5.rst |  34 +-
 doc/guides/cryptodevs/mlx5.rst   |  32 +-
 doc/guides/nics/mlx5.rst         | 522 ++-------------------------
 doc/guides/platform/index.rst    |   1 +
 doc/guides/platform/mlx5.rst     | 602 +++++++++++++++++++++++++++++++
 doc/guides/regexdevs/mlx5.rst    |  36 +-
 doc/guides/vdpadevs/mlx5.rst     |  87 +----
 7 files changed, 662 insertions(+), 652 deletions(-)
 create mode 100644 doc/guides/platform/mlx5.rst
doc/guides/compressdevs/mlx5.rst +7 -27
index 7f2d6bdfff..273251e088 100644
@@ -3,10 +3,10 @@
 
 .. include:: <isonum.txt>
 
-MLX5 compress driver
+MLX5 Compress Driver
 ====================
 
-The MLX5 compress driver library
+The mlx5 compress driver library
 (**librte_compress_mlx5**) provides support for **Mellanox BlueField-2**
 families of 25/50/100/200 Gb/s adapters.
 
@@ -25,30 +25,7 @@ So, using the BlueField device (starting from BlueField-2), the compress
 class operations can be supported in parallel to the net, vDPA and
 RegEx class operations.
 
-For security reasons and robustness, this driver only deals with virtual
-memory addresses. The way resources allocations are handled by the kernel,
-combined with hardware specifications that allow to handle virtual memory
-addresses directly, ensure that DPDK applications cannot access random
-physical memory (or memory that does not belong to the current process).
-
-The PMD uses libibverbs and libmlx5 to access the device firmware
-or directly the hardware components.
-There are different levels of objects and bypassing abilities
-to get the best performances:
-
-- Verbs is a complete high-level generic API.
-- Direct Verbs is a device-specific API.
-- DevX allows to access firmware objects.
-
-Enabling librte_compress_mlx5 causes DPDK applications to be linked against
-libibverbs.
-
-Mellanox mlx5 PCI device can be probed by number of different PCI devices,
-for example net / vDPA / RegEx. To select the compress PMD ``class=compress``
-should be specified as device parameter. The compress device can be probed and
-used with other Mellanox classes, by adding more options in the class.
-For example: ``class=net:compress`` will probe both the net PMD and the compress
-PMD.
+See :doc:`../../platform/mlx5` guide for more design details.
 
 Features
 --------
@@ -85,6 +62,9 @@ Limitations
 Driver options
 --------------
 
+Please refer to :ref:`mlx5 common options <mlx5_common_driver_options>`
+for an additional list of options shared with other mlx5 drivers.
+
 - ``log-block-size`` parameter [int]
 
   Log of the Huffman block size in the Deflate algorithm.
@@ -101,4 +81,4 @@ Prerequisites
 -------------
 
 - Mellanox OFED version: **5.2**
-  see :doc:`../../nics/mlx5` guide for more Mellanox OFED details.
+  See :ref:`mlx5 common prerequisites <mlx5_linux_prerequisites>` for more details.
doc/guides/cryptodevs/mlx5.rst +9 -23
index 9936556cc9..ef47aa65dd 100644
@@ -28,23 +28,12 @@ when the MKEY is configured to perform crypto operations.
 
 The encryption does not require text to be aligned to the AES block size (128b).
 
-For security reasons and to increase robustness, this driver only deals with virtual
-memory addresses. The way resources allocations are handled by the kernel,
-combined with hardware specifications that allow handling virtual memory
-addresses directly, ensure that DPDK applications cannot access random
-physical memory (or memory that does not belong to the current process).
+See :doc:`../../platform/mlx5` guide for more design details.
 
-The PMD uses ``libibverbs`` and ``libmlx5`` to access the device firmware
-or to access the hardware components directly.
-There are different levels of objects and bypassing abilities.
-To get the best performances:
-
-- Verbs is a complete high-level generic API (Linux only).
-- Direct Verbs is a device-specific API (Linux only).
-- DevX allows to access firmware objects.
+Configuration
+-------------
 
-Enabling ``librte_crypto_mlx5`` causes DPDK applications
-to be linked against libibverbs on Linux OS.
+See the :ref:`mlx5 common configuration <mlx5_common_env>`.
 
 In order to move the device to crypto operational mode, credential and KEK
 (Key Encrypting Key) should be set as the first step.
@@ -109,10 +98,8 @@ The mlxreg dedicated tool should be used as follows:
 Driver options
 --------------
 
-- ``class`` parameter [string]
-
-  Select the class of the driver that should probe the device.
-  `crypto` for the mlx5 crypto driver.
+Please refer to :ref:`mlx5 common options <mlx5_common_driver_options>`
+for an additional list of options shared with other mlx5 drivers.
 
 - ``wcs_file`` parameter [string] - mandatory
 
@@ -168,13 +155,12 @@ Linux Prerequisites
 ~~~~~~~~~~~~~~~~~~~
 
 - Mellanox OFED version: **5.3**.
-  see :doc:`../../nics/mlx5` guide for more Mellanox OFED details.
-
 - Compilation can be done also with rdma-core v15+.
-  see :doc:`../../nics/mlx5` guide for more rdma-core details.
+
+  See :ref:`mlx5 common prerequisites <mlx5_linux_prerequisites>` for more details.
 
 Windows Prerequisites
 ~~~~~~~~~~~~~~~~~~~~~
 
 - Mellanox WINOF-2 version: **2.60** or higher.
-  see :doc:`../../nics/mlx5` guide for more Mellanox WINOF-2 details.
+  See :ref:`mlx5 common prerequisites <mlx5_windows_prerequisites>` for more details.
doc/guides/nics/mlx5.rst +24 -498

597 lines not on this page · raw message

doc/guides/platform/index.rst +1 -0
index 2ff91a6018..b1ba11df73 100644
@@ -14,4 +14,5 @@ The following are platform specific guides and setup information.
     cnxk
     dpaa
     dpaa2
+    mlx5
     octeontx
doc/guides/platform/mlx5.rst new file +602 -0

602 lines not on this page · raw message

doc/guides/regexdevs/mlx5.rst +12 -24
index fc2cacba60..83280bda40 100644
@@ -3,10 +3,10 @@
 
 .. include:: <isonum.txt>
 
-MLX5 RegEx driver
+MLX5 RegEx Driver
 =================
 
-The MLX5 RegEx (Regular Expression) driver library
+The mlx5 RegEx (Regular Expression) driver library
 (**librte_regex_mlx5**) provides support for **Mellanox BlueField-2**
 families of 25/50/100/200 Gb/s adapters.
 
@@ -17,29 +17,21 @@ This PMD is configuring the RegEx HW engine.
 For the PMD to work, the application must supply
 a precompiled rule file in rof2 format.
 
-The PMD uses libibverbs and libmlx5 to access the device firmware
-or directly the hardware components.
-There are different levels of objects and bypassing abilities
-to get the best performances:
-
-- Verbs is a complete high-level generic API
-- Direct Verbs is a device-specific API
-- DevX allows to access firmware objects
-
-Enabling librte_regex_mlx5 causes DPDK applications to be linked against
-libibverbs.
-
-Mellanox mlx5 pci device can be probed by number of different pci devices,
-for example net / vDPA / RegEx. To select the RegEx PMD ``class=regex`` should
-be specified as device parameter. The RegEx device can be probed and used with
-other Mellanox devices, by adding more options in the class.
-For example: ``class=net:regex`` will probe both the net PMD and the RegEx PMD.
+See :doc:`../../platform/mlx5` guide for more design details.
 
 Features
 --------
 
 - Multi segments mbuf support.
 
+Configuration
+-------------
+
+See :ref:`mlx5 common compilation <mlx5_common_compilation>`,
+:ref:`mlx5 firmware configuration <mlx5_firmware_config>`,
+and :ref:`mlx5 common driver options <mlx5_common_driver_options>`.
+
+
 Supported NICs
 --------------
 
@@ -52,12 +44,8 @@ Prerequisites
 - Enable the RegEx capabilities using system call from the BlueField-2.
 - Official support is not yet released.
 
+
 Limitations
 -----------
 
 - The firmware version must be greater than XX.31.0364
-
-Run-time configuration
-~~~~~~~~~~~~~~~~~~~~~~
-
-- **ethtool** operations on related kernel interfaces also affect the PMD.
doc/guides/vdpadevs/mlx5.rst +7 -80

120 lines not on this page · raw message

Signature · 1 line
2.25.1
#12 lore raw

PATCH v2 5/5 doc: update doorbell mapping parameter name in mlx5 guide

In reply to Michael Baum

The "tx_db_nc" devarg forces doorbell register mapping to non-cached region eliminating the extra write memory barrier. This argument was used in creating the UAR for Tx and thus affected its performance.

Recently [1] its use has been extended to all UAR creation in all mlx5 drivers, and now its name is no longer so accurate.

This patch changes its name to "sq_db_nc" to suit any send queue that uses it. The old name will still work for backward compatibility.

[1] commit 5dfa003db53f ("common/mlx5: fix post doorbell barrier")

---
 doc/guides/nics/mlx5.rst                   | 34 ++-----------------
 doc/guides/platform/mlx5.rst               | 39 ++++++++++++++++++++++
 drivers/common/mlx5/linux/mlx5_common_os.c |  2 +-
 drivers/common/mlx5/mlx5_common.c          | 31 ++++++++++++-----
 drivers/common/mlx5/mlx5_common_defs.h     |  8 ++---
 drivers/net/mlx5/linux/mlx5_verbs.c        |  2 +-
 drivers/net/mlx5/mlx5_devx.c               |  2 +-
 7 files changed, 72 insertions(+), 46 deletions(-)
doc/guides/nics/mlx5.rst +3 -31
index f94ed90ef0..8956cd1dd8 100644
@@ -814,37 +814,9 @@ for an additional list of options shared with other mlx5 drivers.
 
 - ``tx_db_nc`` parameter [int]
 
-  The rdma core library can map doorbell register in two ways, depending on the
-  environment variable "MLX5_SHUT_UP_BF":
-
-  - As regular cached memory (usually with write combining attribute), if the
-    variable is either missing or set to zero.
-  - As non-cached memory, if the variable is present and set to not "0" value.
-
-  The type of mapping may slightly affect the Tx performance, the optimal choice
-  is strongly relied on the host architecture and should be deduced practically.
-
-  If ``tx_db_nc`` is set to zero, the doorbell is forced to be mapped to regular
-  memory (with write combining), the PMD will perform the extra write memory barrier
-  after writing to doorbell, it might increase the needed CPU clocks per packet
-  to send, but latency might be improved.
-
-  If ``tx_db_nc`` is set to one, the doorbell is forced to be mapped to non
-  cached memory, the PMD will not perform the extra write memory barrier
-  after writing to doorbell, on some architectures it might improve the
-  performance.
-
-  If ``tx_db_nc`` is set to two, the doorbell is forced to be mapped to regular
-  memory, the PMD will use heuristics to decide whether write memory barrier
-  should be performed. For bursts with size multiple of recommended one (64 pkts)
-  it is supposed the next burst is coming and no need to issue the extra memory
-  barrier (it is supposed to be issued in the next coming burst, at least after
-  descriptor writing). It might increase latency (on some hosts till next
-  packets transmit) and should be used with care.
-
-  If ``tx_db_nc`` is omitted or set to zero, the preset (if any) environment
-  variable "MLX5_SHUT_UP_BF" value is used. If there is no "MLX5_SHUT_UP_BF",
-  the default ``tx_db_nc`` value is zero for ARM64 hosts and one for others.
+  This parameter name is deprecated and ignored.
+  The new name for this parameter is ``sq_db_nc``.
+  See :ref:`common driver options <mlx5_common_driver_options>`.
 
 - ``tx_pp`` parameter [int]
 
doc/guides/platform/mlx5.rst +39 -0
index 0fd5e6604d..d073c213ca 100644
@@ -600,3 +600,42 @@ and below are the arguments supported by the common mlx5 layer.
   from system by default, without explicit rte memory flag.
 
   By default, the PMD will set this value to 0.
+
+- ``sq_db_nc`` parameter [int]
+
+  The rdma core library can map doorbell register in two ways,
+  depending on the environment variable "MLX5_SHUT_UP_BF":
+
+  - As regular cached memory (usually with write combining attribute),
+    if the variable is either missing or set to zero.
+  - As non-cached memory, if the variable is present and set to not "0" value.
+
+   The same doorbell mapping approach is implemented directly by PMD
+   in UAR generation for queues created with DevX.
+
+  The type of mapping may slightly affect the send queue performance,
+  the optimal choice strongly relied on the host architecture
+  and should be deduced practically.
+
+  If ``sq_db_nc`` is set to zero, the doorbell is forced to be mapped to
+  regular memory (with write combining), the PMD will perform the extra write
+  memory barrier after writing to doorbell, it might increase the needed CPU
+  clocks per packet to send, but latency might be improved.
+
+  If ``sq_db_nc`` is set to one, the doorbell is forced to be mapped to non
+  cached memory, the PMD will not perform the extra write memory barrier after
+  writing to doorbell, on some architectures it might improve the performance.
+
+  If ``sq_db_nc`` is set to two, the doorbell is forced to be mapped to
+  regular memory, the PMD will use heuristics to decide whether a write memory
+  barrier should be performed. For bursts with size multiple of recommended one
+  (64 pkts) it is supposed the next burst is coming and no need to issue the
+  extra memory barrier (it is supposed to be issued in the next coming burst,
+  at least after descriptor writing). It might increase latency (on some hosts
+  till the next packets transmit) and should be used with care.
+  The PMD uses heuristics only for Tx queue, for other semd queues the doorbell
+  is forced to be mapped to regular memory as same as ``sq_db_nc`` is set to 0.
+
+  If ``sq_db_nc`` is omitted, the preset (if any) environment variable
+  "MLX5_SHUT_UP_BF" value is used. If there is no "MLX5_SHUT_UP_BF", the
+  default ``sq_db_nc`` value is zero for ARM64 hosts and one for others.
\ No newline at end of file
drivers/common/mlx5/linux/mlx5_common_os.c +1 -1
index 0d3e24e04e..a752d79e8e 100644
@@ -630,7 +630,7 @@ mlx5_config_doorbell_mapping_env(int dbnc)
 		setenv(MLX5_SHUT_UP_BF, MLX5_SHUT_UP_BF_DEFAULT, 1);
 	else
 		setenv(MLX5_SHUT_UP_BF,
-		       dbnc == MLX5_TXDB_NCACHED ? "1" : "0", 1);
+		       dbnc == MLX5_SQ_DB_NCACHED ? "1" : "0", 1);
 	return value;
 }
 
drivers/common/mlx5/mlx5_common.c +23 -8
index 96906d3f39..8cf391df13 100644
@@ -35,10 +35,17 @@ uint8_t haswell_broadwell_cpu;
 
 /*
  * Device parameter to force doorbell register mapping
- * to non-cahed region eliminating the extra write memory barrier.
+ * to non-cached region eliminating the extra write memory barrier.
+ * Deprecated, ignored (Name changed to sq_db_nc).
  */
 #define MLX5_TX_DB_NC "tx_db_nc"
 
+/*
+ * Device parameter to force doorbell register mapping
+ * to non-cached region eliminating the extra write memory barrier.
+ */
+#define MLX5_SQ_DB_NC "sq_db_nc"
+
 /* In case this is an x86_64 intel processor to check if
  * we should use relaxed ordering.
  */
@@ -255,11 +262,17 @@ mlx5_common_args_check_handler(const char *key, const char *val, void *opaque)
 		DRV_LOG(WARNING, "%s: \"%s\" is an invalid integer.", key, val);
 		return -rte_errno;
 	}
-	if (strcmp(key, MLX5_TX_DB_NC) == 0) {
-		if (tmp != MLX5_TXDB_CACHED &&
-		    tmp != MLX5_TXDB_NCACHED &&
-		    tmp != MLX5_TXDB_HEURISTIC) {
-			DRV_LOG(ERR, "Invalid Tx doorbell mapping parameter.");
+	if (strcmp(key, MLX5_TX_DB_NC) == 0)
+		DRV_LOG(WARNING,
+			"%s: deprecated parameter, converted to queue_db_nc",
+			key);
+	if (strcmp(key, MLX5_SQ_DB_NC) == 0 ||
+	    strcmp(key, MLX5_TX_DB_NC) == 0) {
+		if (tmp != MLX5_SQ_DB_CACHED &&
+		    tmp != MLX5_SQ_DB_NCACHED &&
+		    tmp != MLX5_SQ_DB_HEURISTIC) {
+			DRV_LOG(ERR,
+				"Invalid Send Queue doorbell mapping parameter.");
 			rte_errno = EINVAL;
 			return -rte_errno;
 		}
@@ -293,6 +306,7 @@ mlx5_common_config_get(struct mlx5_kvargs_ctrl *mkvlist,
 		RTE_DEVARGS_KEY_CLASS,
 		MLX5_DRIVER_KEY,
 		MLX5_TX_DB_NC,
+		MLX5_SQ_DB_NC,
 		MLX5_MR_EXT_MEMSEG_EN,
 		MLX5_SYS_MEM_EN,
 		MLX5_MR_MEMPOOL_REG_EN,
@@ -317,7 +331,8 @@ mlx5_common_config_get(struct mlx5_kvargs_ctrl *mkvlist,
 	DRV_LOG(DEBUG, "mr_ext_memseg_en is %u.", config->mr_ext_memseg_en);
 	DRV_LOG(DEBUG, "mr_mempool_reg_en is %u.", config->mr_mempool_reg_en);
 	DRV_LOG(DEBUG, "sys_mem_en is %u.", config->sys_mem_en);
-	DRV_LOG(DEBUG, "Tx doorbell mapping parameter is %d.", config->dbnc);
+	DRV_LOG(DEBUG, "Send Queue doorbell mapping parameter is %d.",
+		config->dbnc);
 	return ret;
 }
 
@@ -1231,7 +1246,7 @@ mlx5_devx_alloc_uar(struct mlx5_common_device *cdev)
 	for (retry = 0; retry < MLX5_ALLOC_UAR_RETRY; ++retry) {
 #ifdef MLX5DV_UAR_ALLOC_TYPE_NC
 		/* Control the mapping type according to the settings. */
-		uar_mapping = (cdev->config.dbnc == MLX5_TXDB_NCACHED) ?
+		uar_mapping = (cdev->config.dbnc == MLX5_SQ_DB_NCACHED) ?
 			    MLX5DV_UAR_ALLOC_TYPE_NC : MLX5DV_UAR_ALLOC_TYPE_BF;
 #else
 		/*
drivers/common/mlx5/mlx5_common_defs.h +4 -4
index ca80cd8d29..68b700dc0b 100644
@@ -34,10 +34,10 @@
 /* Default PMD specific parameter value. */
 #define MLX5_ARG_UNSET (-1)
 
-/* MLX5_TX_DB_NC supported values. */
-#define MLX5_TXDB_CACHED 0
-#define MLX5_TXDB_NCACHED 1
-#define MLX5_TXDB_HEURISTIC 2
+/* MLX5_SQ_DB_NC supported values. */
+#define MLX5_SQ_DB_CACHED 0
+#define MLX5_SQ_DB_NCACHED 1
+#define MLX5_SQ_DB_HEURISTIC 2
 
 /* Fields of memory mapping type in offset parameter of mmap() */
 #define MLX5_UAR_MMAP_CMD_SHIFT 8
drivers/net/mlx5/linux/mlx5_verbs.c +1 -1
index 331c61d3c5..b6ba21c216 100644
@@ -926,7 +926,7 @@ mlx5_txq_ibv_uar_init(struct mlx5_txq_ctrl *txq_ctrl, void *bf_reg)
 		DRV_LOG(ERR, "Failed to get mem page size");
 		rte_errno = ENOMEM;
 	}
-	txq->db_heu = priv->sh->cdev->config.dbnc == MLX5_TXDB_HEURISTIC;
+	txq->db_heu = priv->sh->cdev->config.dbnc == MLX5_SQ_DB_HEURISTIC;
 	txq->db_nc = mlx5_db_map_type_get(uar_mmap_offset, page_size);
 	ppriv->uar_table[txq->idx].db = bf_reg;
 #ifndef RTE_ARCH_64
drivers/net/mlx5/mlx5_devx.c +1 -1
index e178b799fa..a9b8c2a1b7 100644
@@ -1326,7 +1326,7 @@ mlx5_txq_devx_obj_new(struct rte_eth_dev *dev, uint16_t idx)
 	txq_data->qp_db = &txq_obj->sq_obj.db_rec[MLX5_SND_DBR];
 	*txq_data->qp_db = 0;
 	txq_data->qp_num_8s = txq_obj->sq_obj.sq->id << 8;
-	txq_data->db_heu = sh->cdev->config.dbnc == MLX5_TXDB_HEURISTIC;
+	txq_data->db_heu = sh->cdev->config.dbnc == MLX5_SQ_DB_HEURISTIC;
 	txq_data->db_nc = sh->tx_uar.dbnc;
 	txq_data->wait_on_time = !!(!sh->config.tx_pp &&
 				    sh->cdev->config.hca_attr.wait_on_time);
Signature · 1 line
2.25.1
#13 lore raw

Re: [PATCH v2 5/5] doc: update doorbell mapping parameter name in mlx5 guide

Quoted 15 lines · Michael Baum

The "tx_db_nc" devarg forces doorbell register mapping to non-cached region eliminating the extra write memory barrier. This argument was used in creating the UAR for Tx and thus affected its performance.

Recently [1] its use has been extended to all UAR creation in all mlx5 drivers, and now its name is no longer so accurate.

This patch changes its name to "sq_db_nc" to suit any send queue that uses it. The old name will still work for backward compatibility.

[1] commit 5dfa003db53f ("common/mlx5: fix post doorbell barrier")

Quoted 9 lines
---
  doc/guides/nics/mlx5.rst                   | 34 ++-----------------
  doc/guides/platform/mlx5.rst               | 39 ++++++++++++++++++++++
  drivers/common/mlx5/linux/mlx5_common_os.c |  2 +-
  drivers/common/mlx5/mlx5_common.c          | 31 ++++++++++++-----
  drivers/common/mlx5/mlx5_common_defs.h     |  8 ++---
  drivers/net/mlx5/linux/mlx5_verbs.c        |  2 +-
  drivers/net/mlx5/mlx5_devx.c               |  2 +-
  7 files changed, 72 insertions(+), 46 deletions(-)

Hi Michael,

This is not a document patch to update the parameter name, patch actually changes the parameter name in the code and documents it, updating it as:
net/mlx5: update doorbell mapping parameter name

#14 lore raw

RE: [PATCH v2 5/5] doc: update doorbell mapping parameter name in mlx5 guide

Quoted 26 lines · Ferruh Yigit
Quoted 15 lines · Michael Baum

The "tx_db_nc" devarg forces doorbell register mapping to non-cached region eliminating the extra write memory barrier. This argument was used in creating the UAR for Tx and thus affected its performance.

Recently [1] its use has been extended to all UAR creation in all mlx5 drivers, and now its name is no longer so accurate.

This patch changes its name to "sq_db_nc" to suit any send queue that uses it. The old name will still work for backward compatibility.

[1] commit 5dfa003db53f ("common/mlx5: fix post doorbell barrier")

Quoted 9 lines
---
  doc/guides/nics/mlx5.rst                   | 34 ++-----------------
  doc/guides/platform/mlx5.rst               | 39 ++++++++++++++++++++++
  drivers/common/mlx5/linux/mlx5_common_os.c |  2 +-
  drivers/common/mlx5/mlx5_common.c          | 31 ++++++++++++-----
  drivers/common/mlx5/mlx5_common_defs.h     |  8 ++---
  drivers/net/mlx5/linux/mlx5_verbs.c        |  2 +-
  drivers/net/mlx5/mlx5_devx.c               |  2 +-
  7 files changed, 72 insertions(+), 46 deletions(-)
Quoted 5 lines

Hi Michael,

This is not a document patch to update the parameter name, patch actually changes the parameter name in the code and documents it, updating it as:
net/mlx5: update doorbell mapping parameter name

Hi Ferruh,

Your argument makes sense to me, but most of updates was in MLX5 common library so I think it is better to write:
common/mlx5: update doorbell mapping parameter name

#15 lore raw

Re: [PATCH v2 5/5] doc: update doorbell mapping parameter name in mlx5 guide

Quoted 30 lines · Michael Baum
Quoted 28 lines · Ferruh Yigit
Quoted 15 lines · Michael Baum

The "tx_db_nc" devarg forces doorbell register mapping to non-cached region eliminating the extra write memory barrier. This argument was used in creating the UAR for Tx and thus affected its performance.

Recently [1] its use has been extended to all UAR creation in all mlx5 drivers, and now its name is no longer so accurate.

This patch changes its name to "sq_db_nc" to suit any send queue that uses it. The old name will still work for backward compatibility.

[1] commit 5dfa003db53f ("common/mlx5: fix post doorbell barrier")

Quoted 9 lines
---
   doc/guides/nics/mlx5.rst                   | 34 ++-----------------
   doc/guides/platform/mlx5.rst               | 39 ++++++++++++++++++++++
   drivers/common/mlx5/linux/mlx5_common_os.c |  2 +-
   drivers/common/mlx5/mlx5_common.c          | 31 ++++++++++++-----
   drivers/common/mlx5/mlx5_common_defs.h     |  8 ++---
   drivers/net/mlx5/linux/mlx5_verbs.c        |  2 +-
   drivers/net/mlx5/mlx5_devx.c               |  2 +-
   7 files changed, 72 insertions(+), 46 deletions(-)

Hi Michael,

Quoted 8 lines
Quoted 3 lines

This is not a document patch to update the parameter name, patch actually changes the parameter name in the code and documents it, updating it as:
net/mlx5: update doorbell mapping parameter name

Hi Ferruh,

Your argument makes sense to me, but most of updates was in MLX5 common library so I think it is better to write:
common/mlx5: update doorbell mapping parameter name

ack, updated accordingly.

#16 lore raw

RE: [PATCH v2 0/5] Refactor mlx5 guides

In reply to Michael Baum

Hi,

Quoted 48 lines

-----Original Message-----
From: Michael Baum <[email protected]>
Sent: Wednesday, February 23, 2022 3:48 PM
To: [email protected]
Cc: Matan Azrad <[email protected]>; Raslan Darawsheh <[email protected]>; Slava Ovsiienko <[email protected]>
Subject: [PATCH v2 0/5] Refactor mlx5 guides

Recently [1] all the drivers running over the mlx5 device started sharing some objects.
This raised the need to create documentation for the common driver that would contain information relevant to all drivers and in particular:

- Tutorial for configuration of the device.
- Instruction for using Mellanox ofed.
- devargs of the common driver.

This series adds such documentation and reference to it as parts of the documentation of the other drivers.

[1]
https://patchwork.dpdk.org/project/dpdk/cover/20211019205602.3188203-
[email protected]/

v2: Rebase

Michael Baum (5):
doc: remove obsolete explanations from mlx5 guide
doc: replace broken links in mlx5 guide
doc: correct name of BlueField-2 in mlx5 guide
doc: add shared guide for mlx5 PMDs
doc: update doorbell mapping parameter name in mlx5 guide

 doc/guides/compressdevs/mlx5.rst           |  40 +-
 doc/guides/cryptodevs/mlx5.rst             |  38 +-
 doc/guides/nics/mlx4.rst                   |   4 +-
 doc/guides/nics/mlx5.rst                   | 567 +-----------------
 doc/guides/platform/index.rst              |   1 +
 doc/guides/platform/mlx5.rst               | 641 +++++++++++++++++++++
 doc/guides/regexdevs/mlx5.rst              |  44 +-
 doc/guides/vdpadevs/mlx5.rst               |  87 +--
 drivers/common/mlx5/linux/mlx5_common_os.c |   2 +-
 drivers/common/mlx5/mlx5_common.c          |  31 +-
 drivers/common/mlx5/mlx5_common_defs.h     |   8 +-
 drivers/net/mlx5/linux/mlx5_verbs.c        |   2 +-
 drivers/net/mlx5/mlx5_devx.c               |   2 +-
 13 files changed, 747 insertions(+), 720 deletions(-)  create mode 100644
Quoted 4 lines

doc/guides/platform/mlx5.rst

--
2.25.1

Series applied to next-net-mlx,

Kindest regards,
Raslan Darawsheh