Skip to contentSkip to conversation

rawdev driver for ntb

PATCH Superseded Xiaoyun Li 7 participants 127 messages Jun 3, 2019 – Jul 5, 2019 Arm/Amlogic Meson SoC support Page 2 of 2

lore
Share & tools
lore
author
#98 lore raw

PATCH v9 6/6 doc: update docs for ntb driver

In reply to Xiaoyun Li

Update related documents for ntb pmd and example.

---
 MAINTAINERS                            |  9 +++++
 doc/guides/rawdevs/index.rst           |  1 +
 doc/guides/rawdevs/ntb_rawdev.rst      | 41 ++++++++++++++++++++++
 doc/guides/rel_notes/release_19_08.rst | 15 ++++++++
 doc/guides/sample_app_ug/index.rst     |  1 +
 doc/guides/sample_app_ug/ntb.rst       | 47 ++++++++++++++++++++++++++
 6 files changed, 114 insertions(+)
 create mode 100644 doc/guides/rawdevs/ntb_rawdev.rst
 create mode 100644 doc/guides/sample_app_ug/ntb.rst
MAINTAINERS +9 -0
index 0c3b48920..0b8dd5983 100644
@@ -1078,6 +1078,11 @@ M: Nipun Gupta <[email protected]>
 F: drivers/raw/dpaa2_cmdif/
 F: doc/guides/rawdevs/dpaa2_cmdif.rst
 
+NTB Rawdev
+M: Xiaoyun Li <[email protected]>
+M: Jingjing Wu <[email protected]>
+F: drivers/raw/ntb_rawdev/
+F: doc/guides/rawdevs/ntb_rawdev.rst
 
 Packet processing
 -----------------
@@ -1454,3 +1459,7 @@ F: examples/tep_termination/
 F: examples/vmdq/
 F: examples/vmdq_dcb/
 F: doc/guides/sample_app_ug/vmdq_dcb_forwarding.rst
+
+M: Xiaoyun Li <[email protected]>
+F: examples/ntb/
+F: doc/guides/sample_app_ug/ntb.rst
doc/guides/rawdevs/index.rst +1 -0
index 7c3bd9586..cf6fcb06b 100644
@@ -14,3 +14,4 @@ application through rawdev API.
     dpaa2_cmdif
     dpaa2_qdma
     ifpga_rawdev
+    ntb_rawdev
doc/guides/rawdevs/ntb_rawdev.rst new file +41 -0

41 lines not on this page · raw message

doc/guides/rel_notes/release_19_08.rst +15 -0
index 3da266705..f1858498a 100644
@@ -99,6 +99,21 @@ New Features
   Updated ``librte_telemetry`` to fetch the global metrics from the
   ``librte_metrics`` library.
 
+* **Introduced NTB PMD.**
+
+  The PMD provided a non-transparent bridge between two separate hosts so
+  that they can communicate with each other. Thus, many user cases can
+  benefit from this, such as fault tolerance and visual acceleration.
+
+  This PMD implemented the following features:
+  * Handshake for device start and stop between two hosts.
+  * Memory allocation for the peer to access and read/write allocated
+    memory from peer.
+  * Use doorbell registers to notify the peer and share some information
+    by using scratchpad registers.
+
+  But the PMD hasn't implemented FIFO. The FIFO will come in 19.11 release.
+  And this PMD only supports intel skylake platform.
 
 Removed Items
 -------------
doc/guides/sample_app_ug/index.rst +1 -0
index 2945be08f..f23f8f59e 100644
@@ -58,3 +58,4 @@ Sample Applications User Guides
     fips_validation
     ipsec_secgw
     bbdev_app
+    ntb
doc/guides/sample_app_ug/ntb.rst new file +47 -0

47 lines not on this page · raw message

Signature · 1 line
2.17.1
#99 lore raw

Re: [dpdk-dev] [PATCH v9 6/6] doc: update docs for ntb driver

28/06/2019 04:53, Xiaoyun Li:

MAINTAINERS M: Nipun Gupta <[email protected]> · line 1085

Quoted 5 lines
MAINTAINERS +5 -0
+NTB Rawdev
+M: Xiaoyun Li <[email protected]>
+M: Jingjing Wu <[email protected]>
+F: drivers/raw/ntb_rawdev/
+F: doc/guides/rawdevs/ntb_rawdev.rst

Please update MAINTAINERS while introducing files in previous patches.

Quoted 2 lines

+++ b/doc/guides/rawdevs/ntb_rawdev.rst +And this PMD only supports intel skylake platform.

I guess the platform name require some uppercase or other characters. Please request a full review of the doc by Intel colleagues.

Quoted 7 lines
doc/guides/rel_notes/release_19_08.rst +5 -0
+* **Introduced NTB PMD.**
+
+  The PMD provided a non-transparent bridge between two separate hosts so
+  that they can communicate with each other. Thus, many user cases can
+  benefit from this, such as fault tolerance and visual acceleration.

doc/guides/rel_notes/release_19_08.rst New Features · line 116

Quoted 10 lines
doc/guides/rel_notes/release_19_08.rst +10 -0
+
+  This PMD implemented the following features:
+  * Handshake for device start and stop between two hosts.
+  * Memory allocation for the peer to access and read/write allocated
+    memory from peer.
+  * Use doorbell registers to notify the peer and share some information
+    by using scratchpad registers.
+
+  But the PMD hasn't implemented FIFO. The FIFO will come in 19.11 release.
+  And this PMD only supports intel skylake platform.

You don't need to repeat the full doc in the release notes. Please make it shorter.
And please update the release notes in the same patch as the code.

Thanks

#100 lore raw

Re: [dpdk-dev] [PATCH v9 6/6] doc: update docs for ntb driver

Got it. Thanks.

Quoted 49 lines

-----Original Message-----
From: Thomas Monjalon [mailto:[email protected]]
Sent: Monday, July 1, 2019 22:24
To: Li, Xiaoyun <[email protected]>
Cc: [email protected]; Wu, Jingjing <[email protected]>; Wiles, Keith <[email protected]>; Liang, Cunming <[email protected]>; Maslekar, Omkar <[email protected]>; Mcnamara, John <[email protected]>
Subject: Re: [dpdk-dev] [PATCH v9 6/6] doc: update docs for ntb driver

28/06/2019 04:53, Xiaoyun Li:

MAINTAINERS M: Nipun Gupta <[email protected]> · line 1085

Quoted 5 lines
MAINTAINERS +5 -0
+NTB Rawdev
+M: Xiaoyun Li <[email protected]>
+M: Jingjing Wu <[email protected]>
+F: drivers/raw/ntb_rawdev/
+F: doc/guides/rawdevs/ntb_rawdev.rst

Please update MAINTAINERS while introducing files in previous patches.

Quoted 2 lines

+++ b/doc/guides/rawdevs/ntb_rawdev.rst +And this PMD only supports intel skylake platform.

I guess the platform name require some uppercase or other characters. Please request a full review of the doc by Intel colleagues.

Quoted 7 lines
doc/guides/rel_notes/release_19_08.rst +5 -0
+* **Introduced NTB PMD.**
+
+  The PMD provided a non-transparent bridge between two separate
+ hosts so  that they can communicate with each other. Thus, many user
+ cases can  benefit from this, such as fault tolerance and visual acceleration.

doc/guides/rel_notes/release_19_08.rst New Features · line 116

Quoted 10 lines
doc/guides/rel_notes/release_19_08.rst +10 -0
+
+  This PMD implemented the following features:
+  * Handshake for device start and stop between two hosts.
+  * Memory allocation for the peer to access and read/write allocated
+    memory from peer.
+  * Use doorbell registers to notify the peer and share some information
+    by using scratchpad registers.
+
+  But the PMD hasn't implemented FIFO. The FIFO will come in 19.11 release.
+  And this PMD only supports intel skylake platform.

You don't need to repeat the full doc in the release notes. Please make it shorter.
And please update the release notes in the same patch as the code.

Thanks

#101 lore raw
In reply to Xiaoyun Li

series Acked-by: Jingjing Wu <[email protected]>

Quoted 105 lines

-----Original Message-----
From: Li, Xiaoyun
Sent: Friday, June 28, 2019 10:54 AM
To: Wu, Jingjing <[email protected]>; Wiles, Keith <[email protected]>; Liang, Cunming <[email protected]>; Maslekar, Omkar <[email protected]>
Cc: [email protected]; Li, Xiaoyun <[email protected]>
Subject: [PATCH v9 0/6] rawdev driver for ntb

This patch set adds support for Intel NTB device with Skylake platform. It is a raw device for allowing two hosts to communicate with each other and access the peer memory.

This patch set also provides a simple example to transmit a file between two hosts. But since there is no FIFO here, only support file which is no more than 4M. And will add FIFO in the future.

v9:
* Fixed a typo.
* Added default case for PPD parse.

v8:
* Fixed a coding style issue.
* Rebased codes to the newest master branch.

v7:
* Fixed a typo.
* Generic spad registers to be spad_user and the specific spad is
* defined by the specific hw.
* Refined the codes by replacing with lib functions such as rte_read32.
* Rebased the codes to the newest dpdk-next-net-intel branch.

v6:
* Fixed a typo.

v5:
* Actual v4. v4 patchset is the same as v3.

v4:
* Fix compile issues of comparison of array with null pointer.

v3:
* Fixed compilation issues with target i686.
* Renamed communication devices to misc devices in usertool.
* Rebased to the newest dpdk-next-net-intel branch.

v2:
* Replaced ! with NULL check for pointers.
* Added ntb_ops valid check before use it.
* Replaced RTE_MEMZONE_1GB with RTE_MEMZONE_IOVA_CONTIG in case users do not use 1G hugepage.
* Added a timeout for dev_stop handshake in case that the peer stopped abnormally such as crashed while debugging.
* Updated docs especailly about how to setup BIOS for skylake.
* Fixed not return issue and not free issue in example.
* Renamed ntb_devices to communication_devices to be more generic in usertools.
* Polish the codes and docs.

Xiaoyun Li (6):
raw/ntb: introduce ntb rawdev driver
raw/ntb: add intel ntb support
raw/ntb: add handshake process
examples/ntb: enable an example for ntb
usertools/dpdk-devbind.py: add support for ntb
doc: update docs for ntb driver

 MAINTAINERS                                   |   9 +
 config/common_base                            |   5 +
 doc/guides/rawdevs/index.rst                  |   1 +
 doc/guides/rawdevs/ntb_rawdev.rst             |  41 +
 doc/guides/rel_notes/release_19_08.rst        |  15 +
 doc/guides/sample_app_ug/index.rst            |   1 +
 doc/guides/sample_app_ug/ntb.rst              |  47 +
 drivers/raw/Makefile                          |   1 +
 drivers/raw/meson.build                       |   2 +-
 drivers/raw/ntb_rawdev/Makefile               |  28 +
 drivers/raw/ntb_rawdev/meson.build            |   8 +
 drivers/raw/ntb_rawdev/ntb_hw_intel.c         | 369 ++++++++
 drivers/raw/ntb_rawdev/ntb_hw_intel.h         |  86 ++
 drivers/raw/ntb_rawdev/ntb_rawdev.c           | 839 ++++++++++++++++++
 drivers/raw/ntb_rawdev/ntb_rawdev.h           | 164 ++++
 .../ntb_rawdev/rte_pmd_ntb_rawdev_version.map |   4 +
 examples/Makefile                             |   1 +
 examples/meson.build                          |   2 +-
 examples/ntb/Makefile                         |  68 ++
 examples/ntb/meson.build                      |  16 +
 examples/ntb/ntb_fwd.c                        | 377 ++++++++
 mk/rte.app.mk                                 |   1 +
 usertools/dpdk-devbind.py                     |   9 +
 23 files changed, 2092 insertions(+), 2 deletions(-)
 create mode 100644 doc/guides/rawdevs/ntb_rawdev.rst
 create mode 100644 doc/guides/sample_app_ug/ntb.rst
 create mode 100644 drivers/raw/ntb_rawdev/Makefile
 create mode 100644 drivers/raw/ntb_rawdev/meson.build
 create mode 100644 drivers/raw/ntb_rawdev/ntb_hw_intel.c
 create mode 100644 drivers/raw/ntb_rawdev/ntb_hw_intel.h
 create mode 100644 drivers/raw/ntb_rawdev/ntb_rawdev.c
 create mode 100644 drivers/raw/ntb_rawdev/ntb_rawdev.h
 create mode 100644 drivers/raw/ntb_rawdev/rte_pmd_ntb_rawdev_version.map
 create mode 100644 examples/ntb/Makefile
 create mode 100644 examples/ntb/meson.build
 create mode 100644 examples/ntb/ntb_fwd.c

--
2.17.1

author
#102 lore raw

v10 cover rawdev driver for ntb

In reply to Xiaoyun Li

This patch set adds support for Intel NTB device with Skylake platform. It is a raw device for allowing two hosts to communicate with each other and access the peer memory.

This patch set also provides a simple example to transmit a file between two hosts. But since there is no FIFO here, only support file which is no more than 4M. And will add FIFO in the future.

series Acked-by: Jingjing Wu <[email protected]>

v10:
* Updated and refined the release notes in the same patch as the code.

v9:
* Fixed a typo.
* Added default case for PPD parse.

v8:
* Fixed a coding style issue.
* Rebased codes to the newest master branch.

v7:
* Fixed a typo.
* Generic spad registers to be spad_user and the specific spad is
* defined by the specific hw.
* Refined the codes by replacing with lib functions such as rte_read32.
* Rebased the codes to the newest dpdk-next-net-intel branch.

v6:
* Fixed a typo.

v5:
* Actual v4. v4 patchset is the same as v3.

v4:
* Fix compile issues of comparison of array with null pointer.

v3:
* Fixed compilation issues with target i686.
* Renamed communication devices to misc devices in usertool.
* Rebased to the newest dpdk-next-net-intel branch.

v2:
* Replaced ! with NULL check for pointers.
* Added ntb_ops valid check before use it.
* Replaced RTE_MEMZONE_1GB with RTE_MEMZONE_IOVA_CONTIG in case users do not use 1G hugepage.
* Added a timeout for dev_stop handshake in case that the peer stopped abnormally such as crashed while debugging.
* Updated docs especailly about how to setup BIOS for skylake.
* Fixed not return issue and not free issue in example.
* Renamed ntb_devices to communication_devices to be more generic in usertools.
* Polish the codes and docs.

Xiaoyun Li (5):
raw/ntb: introduce ntb rawdev driver
usertools/dpdk-devbind.py: add support for ntb
raw/ntb: add intel ntb support
raw/ntb: add handshake process
examples/ntb: enable an example for ntb

 MAINTAINERS                                   |   9 +
 config/common_base                            |   5 +
 doc/guides/rawdevs/index.rst                  |   1 +
 doc/guides/rawdevs/ntb_rawdev.rst             |  52 ++
 doc/guides/rel_notes/release_19_08.rst        |   5 +
 doc/guides/sample_app_ug/index.rst            |   1 +
 doc/guides/sample_app_ug/ntb.rst              |  47 +
 drivers/raw/Makefile                          |   1 +
 drivers/raw/meson.build                       |   2 +-
 drivers/raw/ntb_rawdev/Makefile               |  28 +
 drivers/raw/ntb_rawdev/meson.build            |   8 +
 drivers/raw/ntb_rawdev/ntb_hw_intel.c         | 369 ++++++++
 drivers/raw/ntb_rawdev/ntb_hw_intel.h         |  86 ++
 drivers/raw/ntb_rawdev/ntb_rawdev.c           | 839 ++++++++++++++++++
 drivers/raw/ntb_rawdev/ntb_rawdev.h           | 164 ++++
 .../ntb_rawdev/rte_pmd_ntb_rawdev_version.map |   4 +
 examples/Makefile                             |   1 +
 examples/meson.build                          |   2 +-
 examples/ntb/Makefile                         |  68 ++
 examples/ntb/meson.build                      |  16 +
 examples/ntb/ntb_fwd.c                        | 377 ++++++++
 mk/rte.app.mk                                 |   1 +
 usertools/dpdk-devbind.py                     |   9 +
 23 files changed, 2093 insertions(+), 2 deletions(-)
 create mode 100644 doc/guides/rawdevs/ntb_rawdev.rst
 create mode 100644 doc/guides/sample_app_ug/ntb.rst
 create mode 100644 drivers/raw/ntb_rawdev/Makefile
 create mode 100644 drivers/raw/ntb_rawdev/meson.build
 create mode 100644 drivers/raw/ntb_rawdev/ntb_hw_intel.c
 create mode 100644 drivers/raw/ntb_rawdev/ntb_hw_intel.h
 create mode 100644 drivers/raw/ntb_rawdev/ntb_rawdev.c
 create mode 100644 drivers/raw/ntb_rawdev/ntb_rawdev.h
 create mode 100644 drivers/raw/ntb_rawdev/rte_pmd_ntb_rawdev_version.map
 create mode 100644 examples/ntb/Makefile
 create mode 100644 examples/ntb/meson.build
 create mode 100644 examples/ntb/ntb_fwd.c
Signature · 1 line
2.17.1
author
#103 lore raw

PATCH v10 1/5 raw/ntb: introduce ntb rawdev driver

Introduce rawdev driver support for NTB (Non-transparent Bridge) which can help to connect two separate hosts with each other.

---
 MAINTAINERS                                   |   5 +
 config/common_base                            |   5 +
 doc/guides/rawdevs/index.rst                  |   1 +
 doc/guides/rawdevs/ntb_rawdev.rst             |  22 +
 doc/guides/rel_notes/release_19_08.rst        |   5 +
 drivers/raw/Makefile                          |   1 +
 drivers/raw/meson.build                       |   2 +-
 drivers/raw/ntb_rawdev/Makefile               |  27 +
 drivers/raw/ntb_rawdev/meson.build            |   7 +
 drivers/raw/ntb_rawdev/ntb_rawdev.c           | 488 ++++++++++++++++++
 drivers/raw/ntb_rawdev/ntb_rawdev.h           | 164 ++++++
 .../ntb_rawdev/rte_pmd_ntb_rawdev_version.map |   4 +
 mk/rte.app.mk                                 |   1 +
 13 files changed, 731 insertions(+), 1 deletion(-)
 create mode 100644 doc/guides/rawdevs/ntb_rawdev.rst
 create mode 100644 drivers/raw/ntb_rawdev/Makefile
 create mode 100644 drivers/raw/ntb_rawdev/meson.build
 create mode 100644 drivers/raw/ntb_rawdev/ntb_rawdev.c
 create mode 100644 drivers/raw/ntb_rawdev/ntb_rawdev.h
 create mode 100644 drivers/raw/ntb_rawdev/rte_pmd_ntb_rawdev_version.map
MAINTAINERS +5 -0
index 97a009e43..9303624c6 100644
@@ -1094,6 +1094,11 @@ M: Nipun Gupta <[email protected]>
 F: drivers/raw/dpaa2_cmdif/
 F: doc/guides/rawdevs/dpaa2_cmdif.rst
 
+NTB Rawdev
+M: Xiaoyun Li <[email protected]>
+M: Jingjing Wu <[email protected]>
+F: drivers/raw/ntb_rawdev/
+F: doc/guides/rawdevs/ntb_rawdev.rst
 
 Packet processing
 -----------------
config/common_base +5 -0
index e700bf1e7..d3f2b941a 100644
@@ -752,6 +752,11 @@ CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV=n
 #
 CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV=y
 
+#
+# Compile PMD for NTB raw device
+#
+CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV=y
+
 #
 # Compile librte_ring
 #
doc/guides/rawdevs/index.rst +1 -0
index 7c3bd9586..cf6fcb06b 100644
@@ -14,3 +14,4 @@ application through rawdev API.
     dpaa2_cmdif
     dpaa2_qdma
     ifpga_rawdev
+    ntb_rawdev
doc/guides/rawdevs/ntb_rawdev.rst new file +22 -0
new file mode 100644
index 000000000..ebc4dbc1e
@@ -0,0 +1,22 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2018 Intel Corporation.
+
+NTB Rawdev Driver
+=================
+
+The ``ntb`` rawdev driver provides a non-transparent bridge between two
+separate hosts so that they can communicate with each other. Thus, many
+user cases can benefit from this, such as fault tolerance and visual
+acceleration.
+
+Build Options
+-------------
+
+- ``CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV`` (default ``y``)
+
+   Toggle compilation of the ``ntb_rawdev`` driver.
+
+Limitation
+----------
+
+- The FIFO hasn't been introduced and will come in 19.11 release.
doc/guides/rel_notes/release_19_08.rst +5 -0
index 8e0b13e05..4c07a0210 100644
@@ -126,6 +126,11 @@ New Features
   Added telemetry mode to l3fwd-power application to report
   application level busyness, empty and full polls of rte_eth_rx_burst().
 
+* **Introduced NTB PMD.**
+
+  Added a PMD for Intel NTB (Non-transparent Bridge). This PMD implemented
+  handshake between two seperate hosts and can share local memory for peer
+  host to directly access.
 
 Removed Items
 -------------
drivers/raw/Makefile +1 -0
index 8e29b4a56..efe61f451 100644
@@ -10,5 +10,6 @@ DIRS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_CMDIF_RAWDEV) += dpaa2_cmdif
 DIRS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV) += dpaa2_qdma
 endif
 DIRS-$(CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV) += ifpga_rawdev
+DIRS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) += ntb_rawdev
 
 include $(RTE_SDK)/mk/rte.subdir.mk
drivers/raw/meson.build +1 -1
index a61cdccef..6abf659d0 100644
@@ -1,7 +1,7 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright 2018 NXP
 
-drivers = ['skeleton_rawdev', 'dpaa2_cmdif', 'dpaa2_qdma', 'ifpga_rawdev']
+drivers = ['skeleton_rawdev', 'dpaa2_cmdif', 'dpaa2_qdma', 'ifpga_rawdev', 'ntb_rawdev']
 std_deps = ['rawdev']
 config_flag_fmt = 'RTE_LIBRTE_PMD_@0@_RAWDEV'
 driver_name_fmt = 'rte_pmd_@0@'
drivers/raw/ntb_rawdev/Makefile new file +27 -0
new file mode 100644
index 000000000..da87a4610
@@ -0,0 +1,27 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2019 Intel Corporation
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+#
+# library name
+#
+LIB = librte_pmd_ntb_rawdev.a
+
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+CFLAGS += -O3
+CFLAGS += $(WERROR_FLAGS)
+LDLIBS += -lrte_eal -lrte_mbuf -lrte_mempool
+LDLIBS += -lrte_pci -lrte_bus_pci
+LDLIBS += -lrte_rawdev
+
+EXPORT_MAP := rte_pmd_ntb_rawdev_version.map
+
+LIBABIVER := 1
+
+#
+# all source are stored in SRCS-y
+#
+SRCS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) += ntb_rawdev.c
+
+include $(RTE_SDK)/mk/rte.lib.mk
drivers/raw/ntb_rawdev/meson.build new file +7 -0
new file mode 100644
index 000000000..ca905049d
@@ -0,0 +1,7 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2019 Intel Corporation.
+
+deps += ['rawdev', 'mbuf', 'mempool',
+	 'pci', 'bus_pci']
+sources = files('ntb_rawdev.c')
+allow_experimental_apis = true
drivers/raw/ntb_rawdev/ntb_rawdev.c new file +488 -0

488 lines not on this page · raw message

drivers/raw/ntb_rawdev/ntb_rawdev.h new file +164 -0

164 lines not on this page · raw message

drivers/raw/ntb_rawdev/rte_pmd_ntb_rawdev_version.map new file +4 -0
new file mode 100644
index 000000000..8861484fb
@@ -0,0 +1,4 @@
+DPDK_19.08 {
+
+        local: *;
+};
mk/rte.app.mk +1 -0
index 2b5696a27..ead42ee3a 100644
@@ -307,6 +307,7 @@ ifeq ($(CONFIG_RTE_LIBRTE_IFPGA_BUS),y)
 _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV)   += -lrte_pmd_ifpga_rawdev
 _LDLIBS-$(CONFIG_RTE_LIBRTE_IPN3KE_PMD)       += -lrte_pmd_ipn3ke
 endif # CONFIG_RTE_LIBRTE_IFPGA_BUS
+_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) += -lrte_pmd_ntb_rawdev
 endif # CONFIG_RTE_LIBRTE_RAWDEV
 
 endif # !CONFIG_RTE_BUILD_SHARED_LIBS
Signature · 1 line
2.17.1
author
#104 lore raw

PATCH v10 2/5 usertools/dpdk-devbind.py: add support for ntb

In reply to Xiaoyun Li

In order to allow binding/unbinding of devices for use by the ntb_rawdev, we need to update the devbind script to add a new class of device, and add device ids for the specific HW instances. And only support Intel Skylake platform right now.

---
 doc/guides/rawdevs/ntb_rawdev.rst | 8 ++++++++
 usertools/dpdk-devbind.py         | 9 +++++++++
 2 files changed, 17 insertions(+)
doc/guides/rawdevs/ntb_rawdev.rst +8 -0
index ebc4dbc1e..71496bf44 100644
@@ -16,6 +16,14 @@ Build Options
 
    Toggle compilation of the ``ntb_rawdev`` driver.
 
+Device Setup
+------------
+
+The Intel NTB devices need to be bound to a DPDK-supported kernel driver
+to use, i.e. igb_uio, vfio. The ``dpdk-devbind.py`` script can be used to
+show devices status and to bind them to a suitable kernel driver. They will
+appear under the category of "Misc (rawdev) devices".
+
 Limitation
 ----------
 
usertools/dpdk-devbind.py +9 -0
index 9e79f0d28..30d75f3f0 100755
@@ -36,11 +36,15 @@
 octeontx2_npa = {'Class': '08', 'Vendor': '177d', 'Device': 'a0fb,a0fc',
               'SVendor': None, 'SDevice': None}
 
+intel_ntb_skx = {'Class': '06', 'Vendor': '8086', 'Device': '201c',
+              'SVendor': None, 'SDevice': None}
+
 network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class]
 crypto_devices = [encryption_class, intel_processor_class]
 eventdev_devices = [cavium_sso, cavium_tim, octeontx2_sso]
 mempool_devices = [cavium_fpa, octeontx2_npa]
 compress_devices = [cavium_zip]
+misc_devices = [intel_ntb_skx]
 
 # global dict ethernet devices present. Dictionary indexed by PCI address.
 # Each device within this is itself a dictionary of device properties
@@ -595,6 +599,9 @@ def show_status():
     if status_dev == "compress" or status_dev == "all":
         show_device_status(compress_devices , "Compress")
 
+    if status_dev == "misc" or status_dev == "all":
+        show_device_status(misc_devices , "Misc (rawdev)")
+
 
 def parse_args():
     '''Parses the command-line arguments given by the user and takes the
@@ -670,6 +677,7 @@ def do_arg_actions():
             get_device_details(eventdev_devices)
             get_device_details(mempool_devices)
             get_device_details(compress_devices)
+            get_device_details(misc_devices)
         show_status()
 
 
@@ -690,6 +698,7 @@ def main():
     get_device_details(eventdev_devices)
     get_device_details(mempool_devices)
     get_device_details(compress_devices)
+    get_device_details(misc_devices)
     do_arg_actions()
 
 if __name__ == "__main__":
Signature · 1 line
2.17.1
author
#105 lore raw

PATCH v10 3/5 raw/ntb: add intel ntb support

In reply to Xiaoyun Li

Add in the list of registers for the device. And enable ntb device ops for Intel Skylake platform.

---
 doc/guides/rawdevs/ntb_rawdev.rst     |  17 ++
 drivers/raw/ntb_rawdev/Makefile       |   1 +
 drivers/raw/ntb_rawdev/meson.build    |   3 +-
 drivers/raw/ntb_rawdev/ntb_hw_intel.c | 369 ++++++++++++++++++++++++++
 drivers/raw/ntb_rawdev/ntb_hw_intel.h |  86 ++++++
 drivers/raw/ntb_rawdev/ntb_rawdev.c   |   5 +
 6 files changed, 480 insertions(+), 1 deletion(-)
 create mode 100644 drivers/raw/ntb_rawdev/ntb_hw_intel.c
 create mode 100644 drivers/raw/ntb_rawdev/ntb_hw_intel.h
doc/guides/rawdevs/ntb_rawdev.rst +17 -0
index 71496bf44..0e74897db 100644
@@ -9,6 +9,22 @@ separate hosts so that they can communicate with each other. Thus, many
 user cases can benefit from this, such as fault tolerance and visual
 acceleration.
 
+BIOS setting on Intel Skylake
+-----------------------------
+
+Intel Non-transparent Bridge needs special BIOS setting. Since the PMD only
+supports Intel Skylake platform, introduce BIOS setting here. The referencce
+is https://www.intel.com/content/dam/support/us/en/documents/server-products/Intel_Xeon_Processor_Scalable_Family_BIOS_User_Guide.pdf
+
+- Set the needed PCIe port as NTB to NTB mode on both hosts.
+- Enable NTB bars and set bar size of bar 23 and bar 45 as 12-29 (2K-512M)
+  on both hosts. Note that bar size on both hosts should be the same.
+- Disable split bars for both hosts.
+- Set crosslink control override as DSD/USP on one host, USD/DSP on
+  another host.
+- Disable PCIe PII SSC (Spread Spectrum Clocking) for both hosts. This
+  is a hardware requirement.
+
 Build Options
 -------------
 
@@ -28,3 +44,4 @@ Limitation
 ----------
 
 - The FIFO hasn't been introduced and will come in 19.11 release.
+- This PMD only supports Intel Skylake platform.
drivers/raw/ntb_rawdev/Makefile +1 -0
index da87a4610..74c045a86 100644
@@ -23,5 +23,6 @@ LIBABIVER := 1
 # all source are stored in SRCS-y
 #
 SRCS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) += ntb_rawdev.c
+SRCS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) += ntb_hw_intel.c
 
 include $(RTE_SDK)/mk/rte.lib.mk
drivers/raw/ntb_rawdev/meson.build +2 -1
index ca905049d..c696f60b3 100644
@@ -3,5 +3,6 @@
 
 deps += ['rawdev', 'mbuf', 'mempool',
 	 'pci', 'bus_pci']
-sources = files('ntb_rawdev.c')
+sources = files('ntb_rawdev.c',
+                'ntb_hw_intel.c')
 allow_experimental_apis = true
drivers/raw/ntb_rawdev/ntb_hw_intel.c new file +369 -0

369 lines not on this page · raw message

drivers/raw/ntb_rawdev/ntb_hw_intel.h new file +86 -0

86 lines not on this page · raw message

drivers/raw/ntb_rawdev/ntb_rawdev.c +5 -0
index 07ad81d44..113ef0169 100644
@@ -18,11 +18,13 @@
 #include <rte_rawdev.h>
 #include <rte_rawdev_pmd.h>
 
+#include "ntb_hw_intel.h"
 #include "ntb_rawdev.h"
 
 int ntb_logtype;
 
 static const struct rte_pci_id pci_id_ntb_map[] = {
+	{ RTE_PCI_DEVICE(NTB_INTEL_VENDOR_ID, NTB_INTEL_DEV_ID_B2B_SKX) },
 	{ .vendor_id = 0, /* sentinel */ },
 };
 
@@ -353,6 +355,9 @@ ntb_init_hw(struct rte_rawdev *dev, struct rte_pci_device *pci_dev)
 	hw->link_width = NTB_WIDTH_NONE;
 
 	switch (pci_dev->id.device_id) {
+	case NTB_INTEL_DEV_ID_B2B_SKX:
+		hw->ntb_ops = &intel_ntb_ops;
+		break;
 	default:
 		NTB_LOG(ERR, "Not supported device.");
 		return -EINVAL;
Signature · 1 line
2.17.1
author
#106 lore raw

PATCH v10 4/5 raw/ntb: add handshake process

In reply to Xiaoyun Li

Add handshake process using doorbell so that two hosts can communicate to start and stop.

---
 doc/guides/rawdevs/ntb_rawdev.rst   |   5 +
 drivers/raw/ntb_rawdev/ntb_rawdev.c | 336 +++++++++++++++++++++++++++-
 2 files changed, 340 insertions(+), 1 deletion(-)
doc/guides/rawdevs/ntb_rawdev.rst +5 -0
index 0e74897db..223b2a867 100644
@@ -9,6 +9,11 @@ separate hosts so that they can communicate with each other. Thus, many
 user cases can benefit from this, such as fault tolerance and visual
 acceleration.
 
+This PMD allows two hosts to handshake for device start and stop, memory
+allocation for the peer to access and read/write allocated memory from peer.
+Also, the PMD allows to use doorbell registers to notify the peer and share
+some information by using scratchpad registers.
+
 BIOS setting on Intel Skylake
 -----------------------------
 
drivers/raw/ntb_rawdev/ntb_rawdev.c +335 -1

380 lines not on this page · raw message

Signature · 1 line
2.17.1
author
#107 lore raw

PATCH v10 5/5 examples/ntb: enable an example for ntb

In reply to Xiaoyun Li

Enable an example for rawdev ntb. Support interactive mode to send file on one host and receive file from another host. The command line would be 'send [filepath]' and 'receive [filepath]'.

But since the FIFO is not enabled right now, use rte_memcpy as the enqueue and dequeue functions and only support transmitting file no more than 4M.

---
 MAINTAINERS                         |   4 +
 doc/guides/sample_app_ug/index.rst  |   1 +
 doc/guides/sample_app_ug/ntb.rst    |  47 ++++
 drivers/raw/ntb_rawdev/ntb_rawdev.c |  28 ++-
 examples/Makefile                   |   1 +
 examples/meson.build                |   2 +-
 examples/ntb/Makefile               |  68 +++++
 examples/ntb/meson.build            |  16 ++
 examples/ntb/ntb_fwd.c              | 377 ++++++++++++++++++++++++++++
 9 files changed, 535 insertions(+), 9 deletions(-)
 create mode 100644 doc/guides/sample_app_ug/ntb.rst
 create mode 100644 examples/ntb/Makefile
 create mode 100644 examples/ntb/meson.build
 create mode 100644 examples/ntb/ntb_fwd.c
MAINTAINERS +4 -0
index 9303624c6..c94739d26 100644
@@ -1475,3 +1475,7 @@ F: examples/tep_termination/
 F: examples/vmdq/
 F: examples/vmdq_dcb/
 F: doc/guides/sample_app_ug/vmdq_dcb_forwarding.rst
+
+M: Xiaoyun Li <[email protected]>
+F: examples/ntb/
+F: doc/guides/sample_app_ug/ntb.rst
doc/guides/sample_app_ug/index.rst +1 -0
index 2945be08f..f23f8f59e 100644
@@ -58,3 +58,4 @@ Sample Applications User Guides
     fips_validation
     ipsec_secgw
     bbdev_app
+    ntb
doc/guides/sample_app_ug/ntb.rst new file +47 -0

47 lines not on this page · raw message

drivers/raw/ntb_rawdev/ntb_rawdev.c +20 -8
index 005c5c3be..91d6a015b 100644
@@ -240,11 +240,19 @@ ntb_enqueue_bufs(struct rte_rawdev *dev,
 		 unsigned int count,
 		 rte_rawdev_obj_t context)
 {
-	RTE_SET_USED(dev);
-	RTE_SET_USED(buffers);
-	RTE_SET_USED(count);
-	RTE_SET_USED(context);
+	/* Not FIFO right now. Just for testing memory write. */
+	struct ntb_hw *hw = dev->dev_private;
+	unsigned int i;
+	void *bar_addr;
+	size_t size;
+
+	if (hw->ntb_ops->get_peer_mw_addr == NULL)
+		return -ENOTSUP;
+	bar_addr = (*hw->ntb_ops->get_peer_mw_addr)(dev, 0);
+	size = (size_t)context;
 
+	for (i = 0; i < count; i++)
+		rte_memcpy(bar_addr, buffers[i]->buf_addr, size);
 	return 0;
 }
 
@@ -254,11 +262,15 @@ ntb_dequeue_bufs(struct rte_rawdev *dev,
 		 unsigned int count,
 		 rte_rawdev_obj_t context)
 {
-	RTE_SET_USED(dev);
-	RTE_SET_USED(buffers);
-	RTE_SET_USED(count);
-	RTE_SET_USED(context);
+	/* Not FIFO. Just for testing memory read. */
+	struct ntb_hw *hw = dev->dev_private;
+	unsigned int i;
+	size_t size;
+
+	size = (size_t)context;
 
+	for (i = 0; i < count; i++)
+		rte_memcpy(buffers[i]->buf_addr, hw->mz[i]->addr, size);
 	return 0;
 }
 
examples/Makefile +1 -0
index 7562424d9..de11dd487 100644
@@ -53,6 +53,7 @@ DIRS-y += link_status_interrupt
 DIRS-$(CONFIG_RTE_LIBRTE_LPM) += load_balancer
 DIRS-y += multi_process
 DIRS-y += netmap_compat/bridge
+DIRS-y += ntb
 DIRS-$(CONFIG_RTE_LIBRTE_REORDER) += packet_ordering
 ifeq ($(CONFIG_RTE_ARCH_X86_64),y)
 DIRS-y += performance-thread
examples/meson.build +1 -1
index 87113bd70..a046b74ad 100644
@@ -30,7 +30,7 @@ all_examples = [
 	'multi_process/hotplug_mp',
 	'multi_process/simple_mp',
 	'multi_process/symmetric_mp',
-	'netmap_compat', 'packet_ordering',
+	'netmap_compat', 'ntb', 'packet_ordering',
 	'performance-thread', 'ptpclient',
 	'qos_meter', 'qos_sched',
 	'quota_watermark', 'rxtx_callbacks',
examples/ntb/Makefile new file +68 -0

68 lines not on this page · raw message

examples/ntb/meson.build new file +16 -0
new file mode 100644
index 000000000..9a6288f4f
@@ -0,0 +1,16 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2019 Intel Corporation
+
+# meson file, for building this example as part of a main DPDK build.
+#
+# To build this example as a standalone application with an already-installed
+# DPDK instance, use 'make'
+
+if host_machine.system() != 'linux'
+	build = false
+endif
+deps += 'rawdev'
+cflags += ['-D_FILE_OFFSET_BITS=64']
+sources = files(
+	'ntb_fwd.c'
+)
examples/ntb/ntb_fwd.c new file +377 -0

377 lines not on this page · raw message

Signature · 1 line
2.17.1
author
#108 lore raw

PATCH v10 1/5 raw/ntb: introduce ntb rawdev driver

In reply to Xiaoyun Li

Introduce rawdev driver support for NTB (Non-transparent Bridge) which can help to connect two separate hosts with each other.

---
 MAINTAINERS                                   |   5 +
 config/common_base                            |   5 +
 doc/guides/rawdevs/index.rst                  |   1 +
 doc/guides/rawdevs/ntb_rawdev.rst             |  22 +
 doc/guides/rel_notes/release_19_08.rst        |   5 +
 drivers/raw/Makefile                          |   1 +
 drivers/raw/meson.build                       |   2 +-
 drivers/raw/ntb_rawdev/Makefile               |  27 +
 drivers/raw/ntb_rawdev/meson.build            |   7 +
 drivers/raw/ntb_rawdev/ntb_rawdev.c           | 488 ++++++++++++++++++
 drivers/raw/ntb_rawdev/ntb_rawdev.h           | 164 ++++++
 .../ntb_rawdev/rte_pmd_ntb_rawdev_version.map |   4 +
 mk/rte.app.mk                                 |   1 +
 13 files changed, 731 insertions(+), 1 deletion(-)
 create mode 100644 doc/guides/rawdevs/ntb_rawdev.rst
 create mode 100644 drivers/raw/ntb_rawdev/Makefile
 create mode 100644 drivers/raw/ntb_rawdev/meson.build
 create mode 100644 drivers/raw/ntb_rawdev/ntb_rawdev.c
 create mode 100644 drivers/raw/ntb_rawdev/ntb_rawdev.h
 create mode 100644 drivers/raw/ntb_rawdev/rte_pmd_ntb_rawdev_version.map
MAINTAINERS +5 -0
index 97a009e43..9303624c6 100644
@@ -1094,6 +1094,11 @@ M: Nipun Gupta <[email protected]>
 F: drivers/raw/dpaa2_cmdif/
 F: doc/guides/rawdevs/dpaa2_cmdif.rst
 
+NTB Rawdev
+M: Xiaoyun Li <[email protected]>
+M: Jingjing Wu <[email protected]>
+F: drivers/raw/ntb_rawdev/
+F: doc/guides/rawdevs/ntb_rawdev.rst
 
 Packet processing
 -----------------
config/common_base +5 -0
index e700bf1e7..d3f2b941a 100644
@@ -752,6 +752,11 @@ CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV=n
 #
 CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV=y
 
+#
+# Compile PMD for NTB raw device
+#
+CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV=y
+
 #
 # Compile librte_ring
 #
doc/guides/rawdevs/index.rst +1 -0
index 7c3bd9586..cf6fcb06b 100644
@@ -14,3 +14,4 @@ application through rawdev API.
     dpaa2_cmdif
     dpaa2_qdma
     ifpga_rawdev
+    ntb_rawdev
doc/guides/rawdevs/ntb_rawdev.rst new file +22 -0
new file mode 100644
index 000000000..ebc4dbc1e
@@ -0,0 +1,22 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2018 Intel Corporation.
+
+NTB Rawdev Driver
+=================
+
+The ``ntb`` rawdev driver provides a non-transparent bridge between two
+separate hosts so that they can communicate with each other. Thus, many
+user cases can benefit from this, such as fault tolerance and visual
+acceleration.
+
+Build Options
+-------------
+
+- ``CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV`` (default ``y``)
+
+   Toggle compilation of the ``ntb_rawdev`` driver.
+
+Limitation
+----------
+
+- The FIFO hasn't been introduced and will come in 19.11 release.
doc/guides/rel_notes/release_19_08.rst +5 -0
index 8e0b13e05..4c07a0210 100644
@@ -126,6 +126,11 @@ New Features
   Added telemetry mode to l3fwd-power application to report
   application level busyness, empty and full polls of rte_eth_rx_burst().
 
+* **Introduced NTB PMD.**
+
+  Added a PMD for Intel NTB (Non-transparent Bridge). This PMD implemented
+  handshake between two separate hosts and can share local memory for peer
+  host to directly access.
 
 Removed Items
 -------------
drivers/raw/Makefile +1 -0
index 8e29b4a56..efe61f451 100644
@@ -10,5 +10,6 @@ DIRS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_CMDIF_RAWDEV) += dpaa2_cmdif
 DIRS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV) += dpaa2_qdma
 endif
 DIRS-$(CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV) += ifpga_rawdev
+DIRS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) += ntb_rawdev
 
 include $(RTE_SDK)/mk/rte.subdir.mk
drivers/raw/meson.build +1 -1
index a61cdccef..6abf659d0 100644
@@ -1,7 +1,7 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright 2018 NXP
 
-drivers = ['skeleton_rawdev', 'dpaa2_cmdif', 'dpaa2_qdma', 'ifpga_rawdev']
+drivers = ['skeleton_rawdev', 'dpaa2_cmdif', 'dpaa2_qdma', 'ifpga_rawdev', 'ntb_rawdev']
 std_deps = ['rawdev']
 config_flag_fmt = 'RTE_LIBRTE_PMD_@0@_RAWDEV'
 driver_name_fmt = 'rte_pmd_@0@'
drivers/raw/ntb_rawdev/Makefile new file +27 -0
new file mode 100644
index 000000000..da87a4610
@@ -0,0 +1,27 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2019 Intel Corporation
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+#
+# library name
+#
+LIB = librte_pmd_ntb_rawdev.a
+
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+CFLAGS += -O3
+CFLAGS += $(WERROR_FLAGS)
+LDLIBS += -lrte_eal -lrte_mbuf -lrte_mempool
+LDLIBS += -lrte_pci -lrte_bus_pci
+LDLIBS += -lrte_rawdev
+
+EXPORT_MAP := rte_pmd_ntb_rawdev_version.map
+
+LIBABIVER := 1
+
+#
+# all source are stored in SRCS-y
+#
+SRCS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) += ntb_rawdev.c
+
+include $(RTE_SDK)/mk/rte.lib.mk
drivers/raw/ntb_rawdev/meson.build new file +7 -0
new file mode 100644
index 000000000..ca905049d
@@ -0,0 +1,7 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2019 Intel Corporation.
+
+deps += ['rawdev', 'mbuf', 'mempool',
+	 'pci', 'bus_pci']
+sources = files('ntb_rawdev.c')
+allow_experimental_apis = true
drivers/raw/ntb_rawdev/ntb_rawdev.c new file +488 -0

488 lines not on this page · raw message

drivers/raw/ntb_rawdev/ntb_rawdev.h new file +164 -0

164 lines not on this page · raw message

drivers/raw/ntb_rawdev/rte_pmd_ntb_rawdev_version.map new file +4 -0
new file mode 100644
index 000000000..8861484fb
@@ -0,0 +1,4 @@
+DPDK_19.08 {
+
+        local: *;
+};
mk/rte.app.mk +1 -0
index 2b5696a27..ead42ee3a 100644
@@ -307,6 +307,7 @@ ifeq ($(CONFIG_RTE_LIBRTE_IFPGA_BUS),y)
 _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV)   += -lrte_pmd_ifpga_rawdev
 _LDLIBS-$(CONFIG_RTE_LIBRTE_IPN3KE_PMD)       += -lrte_pmd_ipn3ke
 endif # CONFIG_RTE_LIBRTE_IFPGA_BUS
+_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) += -lrte_pmd_ntb_rawdev
 endif # CONFIG_RTE_LIBRTE_RAWDEV
 
 endif # !CONFIG_RTE_BUILD_SHARED_LIBS
Signature · 1 line
2.17.1
author
#109 lore raw

v10 cover rawdev driver for ntb

In reply to Xiaoyun Li

This patch set adds support for Intel NTB device with Skylake platform. It is a raw device for allowing two hosts to communicate with each other and access the peer memory.

This patch set also provides a simple example to transmit a file between two hosts. But since there is no FIFO here, only support file which is no more than 4M. And will add FIFO in the future.

series Acked-by: Jingjing Wu <[email protected]>

v10:
* Updated and refined the release notes in the same patch as the code.

v9:
* Fixed a typo.
* Added default case for PPD parse.

v8:
* Fixed a coding style issue.
* Rebased codes to the newest master branch.

v7:
* Fixed a typo.
* Generic spad registers to be spad_user and the specific spad is
* defined by the specific hw.
* Refined the codes by replacing with lib functions such as rte_read32.
* Rebased the codes to the newest dpdk-next-net-intel branch.

v6:
* Fixed a typo.

v5:
* Actual v4. v4 patchset is the same as v3.

v4:
* Fix compile issues of comparison of array with null pointer.

v3:
* Fixed compilation issues with target i686.
* Renamed communication devices to misc devices in usertool.
* Rebased to the newest dpdk-next-net-intel branch.

v2:
* Replaced ! with NULL check for pointers.
* Added ntb_ops valid check before use it.
* Replaced RTE_MEMZONE_1GB with RTE_MEMZONE_IOVA_CONTIG in case users do not use 1G hugepage.
* Added a timeout for dev_stop handshake in case that the peer stopped abnormally such as crashed while debugging.
* Updated docs especailly about how to setup BIOS for skylake.
* Fixed not return issue and not free issue in example.
* Renamed ntb_devices to communication_devices to be more generic in usertools.
* Polish the codes and docs.

Xiaoyun Li (5):
raw/ntb: introduce ntb rawdev driver
usertools/dpdk-devbind.py: add support for ntb
raw/ntb: add intel ntb support
raw/ntb: add handshake process
examples/ntb: enable an example for ntb

 MAINTAINERS                                   |   9 +
 config/common_base                            |   5 +
 doc/guides/rawdevs/index.rst                  |   1 +
 doc/guides/rawdevs/ntb_rawdev.rst             |  52 ++
 doc/guides/rel_notes/release_19_08.rst        |   5 +
 doc/guides/sample_app_ug/index.rst            |   1 +
 doc/guides/sample_app_ug/ntb.rst              |  47 +
 drivers/raw/Makefile                          |   1 +
 drivers/raw/meson.build                       |   2 +-
 drivers/raw/ntb_rawdev/Makefile               |  28 +
 drivers/raw/ntb_rawdev/meson.build            |   8 +
 drivers/raw/ntb_rawdev/ntb_hw_intel.c         | 369 ++++++++
 drivers/raw/ntb_rawdev/ntb_hw_intel.h         |  86 ++
 drivers/raw/ntb_rawdev/ntb_rawdev.c           | 839 ++++++++++++++++++
 drivers/raw/ntb_rawdev/ntb_rawdev.h           | 164 ++++
 .../ntb_rawdev/rte_pmd_ntb_rawdev_version.map |   4 +
 examples/Makefile                             |   1 +
 examples/meson.build                          |   2 +-
 examples/ntb/Makefile                         |  68 ++
 examples/ntb/meson.build                      |  16 +
 examples/ntb/ntb_fwd.c                        | 377 ++++++++
 mk/rte.app.mk                                 |   1 +
 usertools/dpdk-devbind.py                     |   9 +
 23 files changed, 2093 insertions(+), 2 deletions(-)
 create mode 100644 doc/guides/rawdevs/ntb_rawdev.rst
 create mode 100644 doc/guides/sample_app_ug/ntb.rst
 create mode 100644 drivers/raw/ntb_rawdev/Makefile
 create mode 100644 drivers/raw/ntb_rawdev/meson.build
 create mode 100644 drivers/raw/ntb_rawdev/ntb_hw_intel.c
 create mode 100644 drivers/raw/ntb_rawdev/ntb_hw_intel.h
 create mode 100644 drivers/raw/ntb_rawdev/ntb_rawdev.c
 create mode 100644 drivers/raw/ntb_rawdev/ntb_rawdev.h
 create mode 100644 drivers/raw/ntb_rawdev/rte_pmd_ntb_rawdev_version.map
 create mode 100644 examples/ntb/Makefile
 create mode 100644 examples/ntb/meson.build
 create mode 100644 examples/ntb/ntb_fwd.c
Signature · 1 line
2.17.1
author
#110 lore raw

PATCH v10 1/5 raw/ntb: introduce ntb rawdev driver

Introduce rawdev driver support for NTB (Non-transparent Bridge) which can help to connect two separate hosts with each other.

---
 MAINTAINERS                                   |   5 +
 config/common_base                            |   5 +
 doc/guides/rawdevs/index.rst                  |   1 +
 doc/guides/rawdevs/ntb_rawdev.rst             |  22 +
 doc/guides/rel_notes/release_19_08.rst        |   5 +
 drivers/raw/Makefile                          |   1 +
 drivers/raw/meson.build                       |   2 +-
 drivers/raw/ntb_rawdev/Makefile               |  27 +
 drivers/raw/ntb_rawdev/meson.build            |   7 +
 drivers/raw/ntb_rawdev/ntb_rawdev.c           | 488 ++++++++++++++++++
 drivers/raw/ntb_rawdev/ntb_rawdev.h           | 164 ++++++
 .../ntb_rawdev/rte_pmd_ntb_rawdev_version.map |   4 +
 mk/rte.app.mk                                 |   1 +
 13 files changed, 731 insertions(+), 1 deletion(-)
 create mode 100644 doc/guides/rawdevs/ntb_rawdev.rst
 create mode 100644 drivers/raw/ntb_rawdev/Makefile
 create mode 100644 drivers/raw/ntb_rawdev/meson.build
 create mode 100644 drivers/raw/ntb_rawdev/ntb_rawdev.c
 create mode 100644 drivers/raw/ntb_rawdev/ntb_rawdev.h
 create mode 100644 drivers/raw/ntb_rawdev/rte_pmd_ntb_rawdev_version.map
MAINTAINERS +5 -0
index 97a009e43..9303624c6 100644
@@ -1094,6 +1094,11 @@ M: Nipun Gupta <[email protected]>
 F: drivers/raw/dpaa2_cmdif/
 F: doc/guides/rawdevs/dpaa2_cmdif.rst
 
+NTB Rawdev
+M: Xiaoyun Li <[email protected]>
+M: Jingjing Wu <[email protected]>
+F: drivers/raw/ntb_rawdev/
+F: doc/guides/rawdevs/ntb_rawdev.rst
 
 Packet processing
 -----------------
config/common_base +5 -0
index e700bf1e7..d3f2b941a 100644
@@ -752,6 +752,11 @@ CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV=n
 #
 CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV=y
 
+#
+# Compile PMD for NTB raw device
+#
+CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV=y
+
 #
 # Compile librte_ring
 #
doc/guides/rawdevs/index.rst +1 -0
index 7c3bd9586..cf6fcb06b 100644
@@ -14,3 +14,4 @@ application through rawdev API.
     dpaa2_cmdif
     dpaa2_qdma
     ifpga_rawdev
+    ntb_rawdev
doc/guides/rawdevs/ntb_rawdev.rst new file +22 -0
new file mode 100644
index 000000000..ebc4dbc1e
@@ -0,0 +1,22 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2018 Intel Corporation.
+
+NTB Rawdev Driver
+=================
+
+The ``ntb`` rawdev driver provides a non-transparent bridge between two
+separate hosts so that they can communicate with each other. Thus, many
+user cases can benefit from this, such as fault tolerance and visual
+acceleration.
+
+Build Options
+-------------
+
+- ``CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV`` (default ``y``)
+
+   Toggle compilation of the ``ntb_rawdev`` driver.
+
+Limitation
+----------
+
+- The FIFO hasn't been introduced and will come in 19.11 release.
doc/guides/rel_notes/release_19_08.rst +5 -0
index 8e0b13e05..4c07a0210 100644
@@ -126,6 +126,11 @@ New Features
   Added telemetry mode to l3fwd-power application to report
   application level busyness, empty and full polls of rte_eth_rx_burst().
 
+* **Introduced NTB PMD.**
+
+  Added a PMD for Intel NTB (Non-transparent Bridge). This PMD implemented
+  handshake between two separate hosts and can share local memory for peer
+  host to directly access.
 
 Removed Items
 -------------
drivers/raw/Makefile +1 -0
index 8e29b4a56..efe61f451 100644
@@ -10,5 +10,6 @@ DIRS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_CMDIF_RAWDEV) += dpaa2_cmdif
 DIRS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV) += dpaa2_qdma
 endif
 DIRS-$(CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV) += ifpga_rawdev
+DIRS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) += ntb_rawdev
 
 include $(RTE_SDK)/mk/rte.subdir.mk
drivers/raw/meson.build +1 -1
index a61cdccef..6abf659d0 100644
@@ -1,7 +1,7 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright 2018 NXP
 
-drivers = ['skeleton_rawdev', 'dpaa2_cmdif', 'dpaa2_qdma', 'ifpga_rawdev']
+drivers = ['skeleton_rawdev', 'dpaa2_cmdif', 'dpaa2_qdma', 'ifpga_rawdev', 'ntb_rawdev']
 std_deps = ['rawdev']
 config_flag_fmt = 'RTE_LIBRTE_PMD_@0@_RAWDEV'
 driver_name_fmt = 'rte_pmd_@0@'
drivers/raw/ntb_rawdev/Makefile new file +27 -0
new file mode 100644
index 000000000..da87a4610
@@ -0,0 +1,27 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2019 Intel Corporation
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+#
+# library name
+#
+LIB = librte_pmd_ntb_rawdev.a
+
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+CFLAGS += -O3
+CFLAGS += $(WERROR_FLAGS)
+LDLIBS += -lrte_eal -lrte_mbuf -lrte_mempool
+LDLIBS += -lrte_pci -lrte_bus_pci
+LDLIBS += -lrte_rawdev
+
+EXPORT_MAP := rte_pmd_ntb_rawdev_version.map
+
+LIBABIVER := 1
+
+#
+# all source are stored in SRCS-y
+#
+SRCS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) += ntb_rawdev.c
+
+include $(RTE_SDK)/mk/rte.lib.mk
drivers/raw/ntb_rawdev/meson.build new file +7 -0
new file mode 100644
index 000000000..ca905049d
@@ -0,0 +1,7 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2019 Intel Corporation.
+
+deps += ['rawdev', 'mbuf', 'mempool',
+	 'pci', 'bus_pci']
+sources = files('ntb_rawdev.c')
+allow_experimental_apis = true
drivers/raw/ntb_rawdev/ntb_rawdev.c new file +488 -0

488 lines not on this page · raw message

drivers/raw/ntb_rawdev/ntb_rawdev.h new file +164 -0

164 lines not on this page · raw message

drivers/raw/ntb_rawdev/rte_pmd_ntb_rawdev_version.map new file +4 -0
new file mode 100644
index 000000000..8861484fb
@@ -0,0 +1,4 @@
+DPDK_19.08 {
+
+        local: *;
+};
mk/rte.app.mk +1 -0
index 2b5696a27..ead42ee3a 100644
@@ -307,6 +307,7 @@ ifeq ($(CONFIG_RTE_LIBRTE_IFPGA_BUS),y)
 _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV)   += -lrte_pmd_ifpga_rawdev
 _LDLIBS-$(CONFIG_RTE_LIBRTE_IPN3KE_PMD)       += -lrte_pmd_ipn3ke
 endif # CONFIG_RTE_LIBRTE_IFPGA_BUS
+_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) += -lrte_pmd_ntb_rawdev
 endif # CONFIG_RTE_LIBRTE_RAWDEV
 
 endif # !CONFIG_RTE_BUILD_SHARED_LIBS
Signature · 1 line
2.17.1
#111 lore raw

Re: [dpdk-dev] [PATCH v10 1/5] raw/ntb: introduce ntb rawdev driver

02/07/2019 08:25, Xiaoyun Li:

Quoted 2 lines

Introduce rawdev driver support for NTB (Non-transparent Bridge) which can help to connect two separate hosts with each other.

[...]

Quoted 1 line

+NTB Rawdev

Is it Intel property? If yes, I suggest "Intel NTB" as title.

MAINTAINERS M: Nipun Gupta <[email protected]> · line 1101

Quoted 4 lines
MAINTAINERS +4 -0
+M: Xiaoyun Li <[email protected]>
+M: Jingjing Wu <[email protected]>
+F: drivers/raw/ntb_rawdev/
+F: doc/guides/rawdevs/ntb_rawdev.rst

Sorry I missed to check the naming here.
Why is it called "ntb_rawdev" and not just "ntb" ?
If you are OK with "ntb" as directory name, I can rename it while applying.

#112 lore raw

Re: [dpdk-dev] [PATCH v10 1/5] raw/ntb: introduce ntb rawdev driver

Quoted 8 lines · Thomas Monjalon

02/07/2019 08:25, Xiaoyun Li:

Quoted 2 lines

Introduce rawdev driver support for NTB (Non-transparent Bridge) which can help to connect two separate hosts with each other.

[...]

Quoted 1 line

+NTB Rawdev

Is it Intel property? If yes, I suggest "Intel NTB" as title.

Quoted 9 lines

MAINTAINERS M: Nipun Gupta <[email protected]> · line 1101

Quoted 4 lines
MAINTAINERS +4 -0
+M: Xiaoyun Li <[email protected]>
+M: Jingjing Wu <[email protected]>
+F: drivers/raw/ntb_rawdev/
+F: doc/guides/rawdevs/ntb_rawdev.rst

Sorry I missed to check the naming here.
Why is it called "ntb_rawdev" and not just "ntb" ?
If you are OK with "ntb" as directory name, I can rename it while applying.

Although I'm not the submitter here, I'd just like to "+1" the use of "ntb" on its own as the name of the directory.

/Bruce

#113 lore raw

Re: [dpdk-dev] [PATCH v10 1/5] raw/ntb: introduce ntb rawdev driver

In reply to Thomas Monjalon

Sure. Thanks.

Quoted 27 lines

-----Original Message-----
From: Thomas Monjalon [mailto:[email protected]]
Sent: Thursday, July 4, 2019 18:32
To: Li, Xiaoyun <[email protected]>
Cc: [email protected]; Wu, Jingjing <[email protected]>; Wiles, Keith <[email protected]>; Liang, Cunming <[email protected]>; Maslekar, Omkar <[email protected]>; Richardson, Bruce <[email protected]>; Yigit, Ferruh <[email protected]>
Subject: Re: [dpdk-dev] [PATCH v10 1/5] raw/ntb: introduce ntb rawdev driver

02/07/2019 08:25, Xiaoyun Li:

Quoted 2 lines

Introduce rawdev driver support for NTB (Non-transparent Bridge) which can help to connect two separate hosts with each other.

[...]

Quoted 1 line

+NTB Rawdev

Is it Intel property? If yes, I suggest "Intel NTB" as title.

MAINTAINERS M: Nipun Gupta <[email protected]> · line 1101

Quoted 4 lines
MAINTAINERS +4 -0
+M: Xiaoyun Li <[email protected]>
+M: Jingjing Wu <[email protected]>
+F: drivers/raw/ntb_rawdev/
+F: doc/guides/rawdevs/ntb_rawdev.rst

Sorry I missed to check the naming here.
Why is it called "ntb_rawdev" and not just "ntb" ?
If you are OK with "ntb" as directory name, I can rename it while applying.

#114 lore raw

Re: [dpdk-dev] [PATCH v10 1/5] raw/ntb: introduce ntb rawdev driver

In reply to Thomas Monjalon

Sorry. Miss the first question.

Quoted 10 lines

-----Original Message-----
From: Thomas Monjalon [mailto:[email protected]]
Sent: Thursday, July 4, 2019 18:32
To: Li, Xiaoyun <[email protected]>
Cc: [email protected]; Wu, Jingjing <[email protected]>; Wiles, Keith <[email protected]>; Liang, Cunming <[email protected]>; Maslekar, Omkar <[email protected]>; Richardson, Bruce <[email protected]>; Yigit, Ferruh <[email protected]>
Subject: Re: [dpdk-dev] [PATCH v10 1/5] raw/ntb: introduce ntb rawdev driver

Quoted 7 lines

02/07/2019 08:25, Xiaoyun Li:

Quoted 2 lines

Introduce rawdev driver support for NTB (Non-transparent Bridge) which can help to connect two separate hosts with each other.

[...]

Quoted 1 line

+NTB Rawdev

Is it Intel property? If yes, I suggest "Intel NTB" as title.

Not just Intel has NTB. AMD, PLX ... also have NTB. The rawdev should be compatible with many kinds of NTB. I implemented specific intel NTB in ntb_hw_intel.c/.h But there is no other ntb in DPDK now. So I am fine to name it as Intel NTB.

Quoted 10 lines

MAINTAINERS M: Nipun Gupta <[email protected]> · line 1101

Quoted 5 lines
MAINTAINERS +4 -0
+M: Xiaoyun Li <[email protected]>
+M: Jingjing Wu <[email protected]>
+F: drivers/raw/ntb_rawdev/
+F: doc/guides/rawdevs/ntb_rawdev.rst

Sorry I missed to check the naming here.
Why is it called "ntb_rawdev" and not just "ntb" ?
If you are OK with "ntb" as directory name, I can rename it while applying.

author
#115 lore raw

PATCH v10 2/5 usertools/dpdk-devbind.py: add support for ntb

In reply to Xiaoyun Li

In order to allow binding/unbinding of devices for use by the ntb_rawdev, we need to update the devbind script to add a new class of device, and add device ids for the specific HW instances. And only support Intel Skylake platform right now.

---
 doc/guides/rawdevs/ntb_rawdev.rst | 8 ++++++++
 usertools/dpdk-devbind.py         | 9 +++++++++
 2 files changed, 17 insertions(+)
doc/guides/rawdevs/ntb_rawdev.rst +8 -0
index ebc4dbc1e..71496bf44 100644
@@ -16,6 +16,14 @@ Build Options
 
    Toggle compilation of the ``ntb_rawdev`` driver.
 
+Device Setup
+------------
+
+The Intel NTB devices need to be bound to a DPDK-supported kernel driver
+to use, i.e. igb_uio, vfio. The ``dpdk-devbind.py`` script can be used to
+show devices status and to bind them to a suitable kernel driver. They will
+appear under the category of "Misc (rawdev) devices".
+
 Limitation
 ----------
 
usertools/dpdk-devbind.py +9 -0
index 9e79f0d28..30d75f3f0 100755
@@ -36,11 +36,15 @@
 octeontx2_npa = {'Class': '08', 'Vendor': '177d', 'Device': 'a0fb,a0fc',
               'SVendor': None, 'SDevice': None}
 
+intel_ntb_skx = {'Class': '06', 'Vendor': '8086', 'Device': '201c',
+              'SVendor': None, 'SDevice': None}
+
 network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class]
 crypto_devices = [encryption_class, intel_processor_class]
 eventdev_devices = [cavium_sso, cavium_tim, octeontx2_sso]
 mempool_devices = [cavium_fpa, octeontx2_npa]
 compress_devices = [cavium_zip]
+misc_devices = [intel_ntb_skx]
 
 # global dict ethernet devices present. Dictionary indexed by PCI address.
 # Each device within this is itself a dictionary of device properties
@@ -595,6 +599,9 @@ def show_status():
     if status_dev == "compress" or status_dev == "all":
         show_device_status(compress_devices , "Compress")
 
+    if status_dev == "misc" or status_dev == "all":
+        show_device_status(misc_devices , "Misc (rawdev)")
+
 
 def parse_args():
     '''Parses the command-line arguments given by the user and takes the
@@ -670,6 +677,7 @@ def do_arg_actions():
             get_device_details(eventdev_devices)
             get_device_details(mempool_devices)
             get_device_details(compress_devices)
+            get_device_details(misc_devices)
         show_status()
 
 
@@ -690,6 +698,7 @@ def main():
     get_device_details(eventdev_devices)
     get_device_details(mempool_devices)
     get_device_details(compress_devices)
+    get_device_details(misc_devices)
     do_arg_actions()
 
 if __name__ == "__main__":
Signature · 1 line
2.17.1
author
#116 lore raw

PATCH v10 3/5 raw/ntb: add intel ntb support

In reply to Xiaoyun Li

Add in the list of registers for the device. And enable ntb device ops for Intel Skylake platform.

---
 doc/guides/rawdevs/ntb_rawdev.rst     |  17 ++
 drivers/raw/ntb_rawdev/Makefile       |   1 +
 drivers/raw/ntb_rawdev/meson.build    |   3 +-
 drivers/raw/ntb_rawdev/ntb_hw_intel.c | 369 ++++++++++++++++++++++++++
 drivers/raw/ntb_rawdev/ntb_hw_intel.h |  86 ++++++
 drivers/raw/ntb_rawdev/ntb_rawdev.c   |   5 +
 6 files changed, 480 insertions(+), 1 deletion(-)
 create mode 100644 drivers/raw/ntb_rawdev/ntb_hw_intel.c
 create mode 100644 drivers/raw/ntb_rawdev/ntb_hw_intel.h
doc/guides/rawdevs/ntb_rawdev.rst +17 -0
index 71496bf44..0e74897db 100644
@@ -9,6 +9,22 @@ separate hosts so that they can communicate with each other. Thus, many
 user cases can benefit from this, such as fault tolerance and visual
 acceleration.
 
+BIOS setting on Intel Skylake
+-----------------------------
+
+Intel Non-transparent Bridge needs special BIOS setting. Since the PMD only
+supports Intel Skylake platform, introduce BIOS setting here. The referencce
+is https://www.intel.com/content/dam/support/us/en/documents/server-products/Intel_Xeon_Processor_Scalable_Family_BIOS_User_Guide.pdf
+
+- Set the needed PCIe port as NTB to NTB mode on both hosts.
+- Enable NTB bars and set bar size of bar 23 and bar 45 as 12-29 (2K-512M)
+  on both hosts. Note that bar size on both hosts should be the same.
+- Disable split bars for both hosts.
+- Set crosslink control override as DSD/USP on one host, USD/DSP on
+  another host.
+- Disable PCIe PII SSC (Spread Spectrum Clocking) for both hosts. This
+  is a hardware requirement.
+
 Build Options
 -------------
 
@@ -28,3 +44,4 @@ Limitation
 ----------
 
 - The FIFO hasn't been introduced and will come in 19.11 release.
+- This PMD only supports Intel Skylake platform.
drivers/raw/ntb_rawdev/Makefile +1 -0
index da87a4610..74c045a86 100644
@@ -23,5 +23,6 @@ LIBABIVER := 1
 # all source are stored in SRCS-y
 #
 SRCS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) += ntb_rawdev.c
+SRCS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) += ntb_hw_intel.c
 
 include $(RTE_SDK)/mk/rte.lib.mk
drivers/raw/ntb_rawdev/meson.build +2 -1
index ca905049d..c696f60b3 100644
@@ -3,5 +3,6 @@
 
 deps += ['rawdev', 'mbuf', 'mempool',
 	 'pci', 'bus_pci']
-sources = files('ntb_rawdev.c')
+sources = files('ntb_rawdev.c',
+                'ntb_hw_intel.c')
 allow_experimental_apis = true
drivers/raw/ntb_rawdev/ntb_hw_intel.c new file +369 -0

369 lines not on this page · raw message

drivers/raw/ntb_rawdev/ntb_hw_intel.h new file +86 -0

86 lines not on this page · raw message

drivers/raw/ntb_rawdev/ntb_rawdev.c +5 -0
index 07ad81d44..113ef0169 100644
@@ -18,11 +18,13 @@
 #include <rte_rawdev.h>
 #include <rte_rawdev_pmd.h>
 
+#include "ntb_hw_intel.h"
 #include "ntb_rawdev.h"
 
 int ntb_logtype;
 
 static const struct rte_pci_id pci_id_ntb_map[] = {
+	{ RTE_PCI_DEVICE(NTB_INTEL_VENDOR_ID, NTB_INTEL_DEV_ID_B2B_SKX) },
 	{ .vendor_id = 0, /* sentinel */ },
 };
 
@@ -353,6 +355,9 @@ ntb_init_hw(struct rte_rawdev *dev, struct rte_pci_device *pci_dev)
 	hw->link_width = NTB_WIDTH_NONE;
 
 	switch (pci_dev->id.device_id) {
+	case NTB_INTEL_DEV_ID_B2B_SKX:
+		hw->ntb_ops = &intel_ntb_ops;
+		break;
 	default:
 		NTB_LOG(ERR, "Not supported device.");
 		return -EINVAL;
Signature · 1 line
2.17.1
author
#117 lore raw

PATCH v10 4/5 raw/ntb: add handshake process

In reply to Xiaoyun Li

Add handshake process using doorbell so that two hosts can communicate to start and stop.

---
 doc/guides/rawdevs/ntb_rawdev.rst   |   5 +
 drivers/raw/ntb_rawdev/ntb_rawdev.c | 336 +++++++++++++++++++++++++++-
 2 files changed, 340 insertions(+), 1 deletion(-)
doc/guides/rawdevs/ntb_rawdev.rst +5 -0
index 0e74897db..223b2a867 100644
@@ -9,6 +9,11 @@ separate hosts so that they can communicate with each other. Thus, many
 user cases can benefit from this, such as fault tolerance and visual
 acceleration.
 
+This PMD allows two hosts to handshake for device start and stop, memory
+allocation for the peer to access and read/write allocated memory from peer.
+Also, the PMD allows to use doorbell registers to notify the peer and share
+some information by using scratchpad registers.
+
 BIOS setting on Intel Skylake
 -----------------------------
 
drivers/raw/ntb_rawdev/ntb_rawdev.c +335 -1

380 lines not on this page · raw message

Signature · 1 line
2.17.1
author
#118 lore raw

PATCH v10 5/5 examples/ntb: enable an example for ntb

In reply to Xiaoyun Li

Enable an example for rawdev ntb. Support interactive mode to send file on one host and receive file from another host. The command line would be 'send [filepath]' and 'receive [filepath]'.

But since the FIFO is not enabled right now, use rte_memcpy as the enqueue and dequeue functions and only support transmitting file no more than 4M.

---
 MAINTAINERS                         |   4 +
 doc/guides/sample_app_ug/index.rst  |   1 +
 doc/guides/sample_app_ug/ntb.rst    |  47 ++++
 drivers/raw/ntb_rawdev/ntb_rawdev.c |  28 ++-
 examples/Makefile                   |   1 +
 examples/meson.build                |   2 +-
 examples/ntb/Makefile               |  68 +++++
 examples/ntb/meson.build            |  16 ++
 examples/ntb/ntb_fwd.c              | 377 ++++++++++++++++++++++++++++
 9 files changed, 535 insertions(+), 9 deletions(-)
 create mode 100644 doc/guides/sample_app_ug/ntb.rst
 create mode 100644 examples/ntb/Makefile
 create mode 100644 examples/ntb/meson.build
 create mode 100644 examples/ntb/ntb_fwd.c
MAINTAINERS +4 -0
index 9303624c6..c94739d26 100644
@@ -1475,3 +1475,7 @@ F: examples/tep_termination/
 F: examples/vmdq/
 F: examples/vmdq_dcb/
 F: doc/guides/sample_app_ug/vmdq_dcb_forwarding.rst
+
+M: Xiaoyun Li <[email protected]>
+F: examples/ntb/
+F: doc/guides/sample_app_ug/ntb.rst
doc/guides/sample_app_ug/index.rst +1 -0
index 2945be08f..f23f8f59e 100644
@@ -58,3 +58,4 @@ Sample Applications User Guides
     fips_validation
     ipsec_secgw
     bbdev_app
+    ntb
doc/guides/sample_app_ug/ntb.rst new file +47 -0

47 lines not on this page · raw message

drivers/raw/ntb_rawdev/ntb_rawdev.c +20 -8
index 005c5c3be..91d6a015b 100644
@@ -240,11 +240,19 @@ ntb_enqueue_bufs(struct rte_rawdev *dev,
 		 unsigned int count,
 		 rte_rawdev_obj_t context)
 {
-	RTE_SET_USED(dev);
-	RTE_SET_USED(buffers);
-	RTE_SET_USED(count);
-	RTE_SET_USED(context);
+	/* Not FIFO right now. Just for testing memory write. */
+	struct ntb_hw *hw = dev->dev_private;
+	unsigned int i;
+	void *bar_addr;
+	size_t size;
+
+	if (hw->ntb_ops->get_peer_mw_addr == NULL)
+		return -ENOTSUP;
+	bar_addr = (*hw->ntb_ops->get_peer_mw_addr)(dev, 0);
+	size = (size_t)context;
 
+	for (i = 0; i < count; i++)
+		rte_memcpy(bar_addr, buffers[i]->buf_addr, size);
 	return 0;
 }
 
@@ -254,11 +262,15 @@ ntb_dequeue_bufs(struct rte_rawdev *dev,
 		 unsigned int count,
 		 rte_rawdev_obj_t context)
 {
-	RTE_SET_USED(dev);
-	RTE_SET_USED(buffers);
-	RTE_SET_USED(count);
-	RTE_SET_USED(context);
+	/* Not FIFO. Just for testing memory read. */
+	struct ntb_hw *hw = dev->dev_private;
+	unsigned int i;
+	size_t size;
+
+	size = (size_t)context;
 
+	for (i = 0; i < count; i++)
+		rte_memcpy(buffers[i]->buf_addr, hw->mz[i]->addr, size);
 	return 0;
 }
 
examples/Makefile +1 -0
index 7562424d9..de11dd487 100644
@@ -53,6 +53,7 @@ DIRS-y += link_status_interrupt
 DIRS-$(CONFIG_RTE_LIBRTE_LPM) += load_balancer
 DIRS-y += multi_process
 DIRS-y += netmap_compat/bridge
+DIRS-y += ntb
 DIRS-$(CONFIG_RTE_LIBRTE_REORDER) += packet_ordering
 ifeq ($(CONFIG_RTE_ARCH_X86_64),y)
 DIRS-y += performance-thread
examples/meson.build +1 -1
index 87113bd70..a046b74ad 100644
@@ -30,7 +30,7 @@ all_examples = [
 	'multi_process/hotplug_mp',
 	'multi_process/simple_mp',
 	'multi_process/symmetric_mp',
-	'netmap_compat', 'packet_ordering',
+	'netmap_compat', 'ntb', 'packet_ordering',
 	'performance-thread', 'ptpclient',
 	'qos_meter', 'qos_sched',
 	'quota_watermark', 'rxtx_callbacks',
examples/ntb/Makefile new file +68 -0

68 lines not on this page · raw message

examples/ntb/meson.build new file +16 -0
new file mode 100644
index 000000000..9a6288f4f
@@ -0,0 +1,16 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2019 Intel Corporation
+
+# meson file, for building this example as part of a main DPDK build.
+#
+# To build this example as a standalone application with an already-installed
+# DPDK instance, use 'make'
+
+if host_machine.system() != 'linux'
+	build = false
+endif
+deps += 'rawdev'
+cflags += ['-D_FILE_OFFSET_BITS=64']
+sources = files(
+	'ntb_fwd.c'
+)
examples/ntb/ntb_fwd.c new file +377 -0

377 lines not on this page · raw message

Signature · 1 line
2.17.1
#119 lore raw
In reply to Xiaoyun Li
Quoted 86 lines · Xiaoyun Li

This patch set adds support for Intel NTB device with Skylake platform. It is a raw device for allowing two hosts to communicate with each other and access the peer memory.

This patch set also provides a simple example to transmit a file between two hosts. But since there is no FIFO here, only support file which is no more than 4M. And will add FIFO in the future.

series Acked-by: Jingjing Wu <[email protected]>

v10:
* Updated and refined the release notes in the same patch as the code.

v9:
* Fixed a typo.
* Added default case for PPD parse.

v8:
* Fixed a coding style issue.
* Rebased codes to the newest master branch.

v7:
* Fixed a typo.
* Generic spad registers to be spad_user and the specific spad is
* defined by the specific hw.
* Refined the codes by replacing with lib functions such as rte_read32.
* Rebased the codes to the newest dpdk-next-net-intel branch.

v6:
* Fixed a typo.

v5:
* Actual v4. v4 patchset is the same as v3.

v4:
* Fix compile issues of comparison of array with null pointer.

v3:
* Fixed compilation issues with target i686.
* Renamed communication devices to misc devices in usertool.
* Rebased to the newest dpdk-next-net-intel branch.

v2:
* Replaced ! with NULL check for pointers.
* Added ntb_ops valid check before use it.
* Replaced RTE_MEMZONE_1GB with RTE_MEMZONE_IOVA_CONTIG in case users do not use 1G hugepage.
* Added a timeout for dev_stop handshake in case that the peer stopped abnormally such as crashed while debugging.
* Updated docs especailly about how to setup BIOS for skylake.
* Fixed not return issue and not free issue in example.
* Renamed ntb_devices to communication_devices to be more generic in usertools.
* Polish the codes and docs.

Xiaoyun Li (5):
raw/ntb: introduce ntb rawdev driver
usertools/dpdk-devbind.py: add support for ntb
raw/ntb: add intel ntb support
raw/ntb: add handshake process
examples/ntb: enable an example for ntb

MAINTAINERS                                   |   9 +
config/common_base                            |   5 +
doc/guides/rawdevs/index.rst                  |   1 +
doc/guides/rawdevs/ntb_rawdev.rst             |  52 ++
doc/guides/rel_notes/release_19_08.rst        |   5 +
doc/guides/sample_app_ug/index.rst            |   1 +
doc/guides/sample_app_ug/ntb.rst              |  47 +
drivers/raw/Makefile                          |   1 +
drivers/raw/meson.build                       |   2 +-
drivers/raw/ntb_rawdev/Makefile               |  28 +
drivers/raw/ntb_rawdev/meson.build            |   8 +
drivers/raw/ntb_rawdev/ntb_hw_intel.c         | 369 ++++++++
drivers/raw/ntb_rawdev/ntb_hw_intel.h         |  86 ++
drivers/raw/ntb_rawdev/ntb_rawdev.c           | 839 ++++++++++++++++++
drivers/raw/ntb_rawdev/ntb_rawdev.h           | 164 ++++
.../ntb_rawdev/rte_pmd_ntb_rawdev_version.map |   4 +
examples/Makefile                             |   1 +
examples/meson.build                          |   2 +-
examples/ntb/Makefile                         |  68 ++
examples/ntb/meson.build                      |  16 +
examples/ntb/ntb_fwd.c                        | 377 ++++++++
mk/rte.app.mk                                 |   1 +
usertools/dpdk-devbind.py                     |   9 +
23 files changed, 2093 insertions(+), 2 deletions(-)
Quoted 13 lines

create mode 100644 doc/guides/rawdevs/ntb_rawdev.rst
create mode 100644 doc/guides/sample_app_ug/ntb.rst
create mode 100644 drivers/raw/ntb_rawdev/Makefile
create mode 100644 drivers/raw/ntb_rawdev/meson.build
create mode 100644 drivers/raw/ntb_rawdev/ntb_hw_intel.c create mode 100644 drivers/raw/ntb_rawdev/ntb_hw_intel.h create mode 100644 drivers/raw/ntb_rawdev/ntb_rawdev.c create mode 100644 drivers/raw/ntb_rawdev/ntb_rawdev.h create mode 100644 drivers/raw/ntb_rawdev/rte_pmd_ntb_rawdev_version.map create mode 100644 examples/ntb/Makefile
create mode 100644 examples/ntb/meson.build
create mode 100644 examples/ntb/ntb_fwd.c

For the series, Reviewed-by: Xiaolong Ye <[email protected]>

Quoted 3 lines
Signature · 1 line
2.17.1
#120 lore raw

v11 cover rawdev driver for ntb

In reply to Xiaoyun Li

This patch set adds support for Intel NTB device with Skylake platform. It is a raw device for allowing two hosts to communicate with each other and access the peer memory.

This patch set also provides a simple example to transmit a file between two hosts. But since there is no FIFO here, only support file which is no more than 4M. And will add FIFO in the future.

v11 (Thomas):
* rebased on top of IOAT driver
* squashed commit for binding script
* renamed ntb_rawdev to ntb as agreed with Bruce Please review this version, I may have done mistakes.

v10:
* Updated and refined the release notes in the same patch as the code.

v9:
* Fixed a typo.
* Added default case for PPD parse.

v8:
* Fixed a coding style issue.
* Rebased codes to the newest master branch.

v7:
* Fixed a typo.
* Generic spad registers to be spad_user and the specific spad is
* defined by the specific hw.
* Refined the codes by replacing with lib functions such as rte_read32.
* Rebased the codes to the newest dpdk-next-net-intel branch.

v6:
* Fixed a typo.

v5:
* Actual v4. v4 patchset is the same as v3.

v4:
* Fix compile issues of comparison of array with null pointer.

v3:
* Fixed compilation issues with target i686.
* Renamed communication devices to misc devices in usertool.
* Rebased to the newest dpdk-next-net-intel branch.

v2:
* Replaced ! with NULL check for pointers.
* Added ntb_ops valid check before use it.
* Replaced RTE_MEMZONE_1GB with RTE_MEMZONE_IOVA_CONTIG in case users do not use 1G hugepage.
* Added a timeout for dev_stop handshake in case that the peer stopped abnormally such as crashed while debugging.
* Updated docs especailly about how to setup BIOS for skylake.
* Fixed not return issue and not free issue in example.
* Renamed ntb_devices to communication_devices to be more generic in usertools.
* Polish the codes and docs.

Xiaoyun Li (4):
raw/ntb: introduce NTB raw device driver
raw/ntb: support Intel NTB
raw/ntb: add handshake process
examples/ntb: add example for NTB

 MAINTAINERS                             |   8 +
 config/common_base                      |   5 +
 doc/guides/rawdevs/index.rst            |   1 +
 doc/guides/rawdevs/ntb.rst              |  52 ++
 doc/guides/rel_notes/release_19_08.rst  |   6 +
 doc/guides/sample_app_ug/index.rst      |   1 +
 doc/guides/sample_app_ug/ntb.rst        |  47 ++
 drivers/raw/Makefile                    |   1 +
 drivers/raw/meson.build                 |   2 +-
 drivers/raw/ntb/Makefile                |  28 +
 drivers/raw/ntb/meson.build             |   8 +
 drivers/raw/ntb/ntb.c                   | 839 ++++++++++++++++++++++++
 drivers/raw/ntb/ntb.h                   | 164 +++++
 drivers/raw/ntb/ntb_hw_intel.c          | 369 +++++++++++
 drivers/raw/ntb/ntb_hw_intel.h          |  86 +++
 drivers/raw/ntb/rte_pmd_ntb_version.map |   4 +
 examples/Makefile                       |   1 +
 examples/meson.build                    |   2 +-
 examples/ntb/Makefile                   |  68 ++
 examples/ntb/meson.build                |  16 +
 examples/ntb/ntb_fwd.c                  | 377 +++++++++++
 mk/rte.app.mk                           |   1 +
 usertools/dpdk-devbind.py               |   4 +-
 23 files changed, 2087 insertions(+), 3 deletions(-)
 create mode 100644 doc/guides/rawdevs/ntb.rst
 create mode 100644 doc/guides/sample_app_ug/ntb.rst
 create mode 100644 drivers/raw/ntb/Makefile
 create mode 100644 drivers/raw/ntb/meson.build
 create mode 100644 drivers/raw/ntb/ntb.c
 create mode 100644 drivers/raw/ntb/ntb.h
 create mode 100644 drivers/raw/ntb/ntb_hw_intel.c
 create mode 100644 drivers/raw/ntb/ntb_hw_intel.h
 create mode 100644 drivers/raw/ntb/rte_pmd_ntb_version.map
 create mode 100644 examples/ntb/Makefile
 create mode 100644 examples/ntb/meson.build
 create mode 100644 examples/ntb/ntb_fwd.c
Signature · 1 line
2.21.0
#121 lore raw

PATCH v11 1/4 raw/ntb: introduce NTB raw device driver

From: Xiaoyun Li <[email protected]>

Introduce rawdev driver support for NTB (Non-transparent Bridge) which can help to connect two separate hosts with each other.

---
 MAINTAINERS                             |   6 +
 config/common_base                      |   5 +
 doc/guides/rawdevs/index.rst            |   1 +
 doc/guides/rawdevs/ntb.rst              |  22 ++
 doc/guides/rel_notes/release_19_08.rst  |   6 +
 drivers/raw/Makefile                    |   1 +
 drivers/raw/meson.build                 |   2 +-
 drivers/raw/ntb/Makefile                |  27 ++
 drivers/raw/ntb/meson.build             |   7 +
 drivers/raw/ntb/ntb.c                   | 488 ++++++++++++++++++++++++
 drivers/raw/ntb/ntb.h                   | 164 ++++++++
 drivers/raw/ntb/rte_pmd_ntb_version.map |   4 +
 mk/rte.app.mk                           |   1 +
 13 files changed, 733 insertions(+), 1 deletion(-)
 create mode 100644 doc/guides/rawdevs/ntb.rst
 create mode 100644 drivers/raw/ntb/Makefile
 create mode 100644 drivers/raw/ntb/meson.build
 create mode 100644 drivers/raw/ntb/ntb.c
 create mode 100644 drivers/raw/ntb/ntb.h
 create mode 100644 drivers/raw/ntb/rte_pmd_ntb_version.map
MAINTAINERS +6 -0
index 605422077..9e6308aa4 100644
@@ -1105,6 +1105,12 @@ M: Nipun Gupta <[email protected]>
 F: drivers/raw/dpaa2_cmdif/
 F: doc/guides/rawdevs/dpaa2_cmdif.rst
 
+NTB
+M: Xiaoyun Li <[email protected]>
+M: Jingjing Wu <[email protected]>
+F: drivers/raw/ntb/
+F: doc/guides/rawdevs/ntb.rst
+
 
 Packet processing
 -----------------
config/common_base +5 -0
index f68107e7a..75e130a43 100644
@@ -762,6 +762,11 @@ CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV=y
 #
 CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV=y
 
+#
+# Compile PMD for NTB raw device
+#
+CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV=y
+
 #
 # Compile librte_ring
 #
doc/guides/rawdevs/index.rst +1 -0
index 0a21989e4..e19c4ec26 100644
@@ -15,3 +15,4 @@ application through rawdev API.
     dpaa2_qdma
     ifpga_rawdev
     ioat_rawdev
+    ntb
doc/guides/rawdevs/ntb.rst new file +22 -0
new file mode 100644
index 000000000..9559ea4b0
@@ -0,0 +1,22 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2018 Intel Corporation.
+
+NTB Rawdev Driver
+=================
+
+The ``ntb`` rawdev driver provides a non-transparent bridge between two
+separate hosts so that they can communicate with each other. Thus, many
+user cases can benefit from this, such as fault tolerance and visual
+acceleration.
+
+Build Options
+-------------
+
+- ``CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV`` (default ``y``)
+
+   Toggle compilation of the ``ntb`` driver.
+
+Limitation
+----------
+
+- The FIFO hasn't been introduced and will come in 19.11 release.
doc/guides/rel_notes/release_19_08.rst +6 -0
index 21934bf01..8cd90e0c3 100644
@@ -129,6 +129,12 @@ New Features
   of via software, reducing cycles spent copying large blocks of data in
   applications.
 
+* **Introduced NTB PMD.**
+
+  Added a PMD for Intel NTB (Non-transparent Bridge). This PMD implemented
+  handshake between two separate hosts and can share local memory for peer
+  host to directly access.
+
 * **Updated telemetry library for global metrics support.**
 
   Updated ``librte_telemetry`` to fetch the global metrics from the
drivers/raw/Makefile +1 -0
index c1b85c8c7..84a13b697 100644
@@ -11,5 +11,6 @@ DIRS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV) += dpaa2_qdma
 endif
 DIRS-$(CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV) += ifpga_rawdev
 DIRS-$(CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV) += ioat
+DIRS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) += ntb
 
 include $(RTE_SDK)/mk/rte.subdir.mk
drivers/raw/meson.build +1 -1
index 2af8a70d4..47f0a1d5f 100644
@@ -2,7 +2,7 @@
 # Copyright 2018 NXP
 
 drivers = ['dpaa2_cmdif', 'dpaa2_qdma',
-	'ifpga_rawdev', 'ioat',
+	'ifpga_rawdev', 'ioat', 'ntb',
 	'skeleton_rawdev']
 std_deps = ['rawdev']
 config_flag_fmt = 'RTE_LIBRTE_PMD_@0@_RAWDEV'
drivers/raw/ntb/Makefile new file +27 -0
new file mode 100644
index 000000000..ec53edd33
@@ -0,0 +1,27 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2019 Intel Corporation
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+#
+# library name
+#
+LIB = librte_pmd_ntb.a
+
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+CFLAGS += -O3
+CFLAGS += $(WERROR_FLAGS)
+LDLIBS += -lrte_eal -lrte_mbuf -lrte_mempool
+LDLIBS += -lrte_pci -lrte_bus_pci
+LDLIBS += -lrte_rawdev
+
+EXPORT_MAP := rte_pmd_ntb_version.map
+
+LIBABIVER := 1
+
+#
+# all source are stored in SRCS-y
+#
+SRCS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) += ntb.c
+
+include $(RTE_SDK)/mk/rte.lib.mk
drivers/raw/ntb/meson.build new file +7 -0
new file mode 100644
index 000000000..0fb44452c
@@ -0,0 +1,7 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2019 Intel Corporation.
+
+deps += ['rawdev', 'mbuf', 'mempool',
+	 'pci', 'bus_pci']
+sources = files('ntb.c')
+allow_experimental_apis = true
drivers/raw/ntb/ntb.c new file +488 -0

488 lines not on this page · raw message

drivers/raw/ntb/ntb.h new file +164 -0

164 lines not on this page · raw message

drivers/raw/ntb/rte_pmd_ntb_version.map new file +4 -0
new file mode 100644
index 000000000..8861484fb
@@ -0,0 +1,4 @@
+DPDK_19.08 {
+
+        local: *;
+};
mk/rte.app.mk +1 -0
index 30f9a12da..d3d12350b 100644
@@ -310,6 +310,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV)   += -lrte_pmd_ifpga_rawdev
 _LDLIBS-$(CONFIG_RTE_LIBRTE_IPN3KE_PMD)       += -lrte_pmd_ipn3ke
 endif # CONFIG_RTE_LIBRTE_IFPGA_BUS
 _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV)   += -lrte_pmd_ioat_rawdev
+_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) += -lrte_pmd_ntb
 endif # CONFIG_RTE_LIBRTE_RAWDEV
 
 endif # !CONFIG_RTE_BUILD_SHARED_LIBS
Signature · 1 line
2.21.0
#122 lore raw

PATCH v11 2/4 raw/ntb: support Intel NTB

In reply to Thomas Monjalon

From: Xiaoyun Li <[email protected]>

Add in the list of registers for the device. And enable NTB device ops for Intel Skylake platform.

---
 doc/guides/rawdevs/ntb.rst     |  25 +++
 drivers/raw/ntb/Makefile       |   1 +
 drivers/raw/ntb/meson.build    |   3 +-
 drivers/raw/ntb/ntb.c          |   5 +
 drivers/raw/ntb/ntb_hw_intel.c | 369 +++++++++++++++++++++++++++++++++
 drivers/raw/ntb/ntb_hw_intel.h |  86 ++++++++
 usertools/dpdk-devbind.py      |   4 +-
 7 files changed, 491 insertions(+), 2 deletions(-)
 create mode 100644 drivers/raw/ntb/ntb_hw_intel.c
 create mode 100644 drivers/raw/ntb/ntb_hw_intel.h
doc/guides/rawdevs/ntb.rst +25 -0
index 9559ea4b0..ff0795b47 100644
@@ -9,6 +9,22 @@ separate hosts so that they can communicate with each other. Thus, many
 user cases can benefit from this, such as fault tolerance and visual
 acceleration.
 
+BIOS setting on Intel Skylake
+-----------------------------
+
+Intel Non-transparent Bridge needs special BIOS setting. Since the PMD only
+supports Intel Skylake platform, introduce BIOS setting here. The referencce
+is https://www.intel.com/content/dam/support/us/en/documents/server-products/Intel_Xeon_Processor_Scalable_Family_BIOS_User_Guide.pdf
+
+- Set the needed PCIe port as NTB to NTB mode on both hosts.
+- Enable NTB bars and set bar size of bar 23 and bar 45 as 12-29 (2K-512M)
+  on both hosts. Note that bar size on both hosts should be the same.
+- Disable split bars for both hosts.
+- Set crosslink control override as DSD/USP on one host, USD/DSP on
+  another host.
+- Disable PCIe PII SSC (Spread Spectrum Clocking) for both hosts. This
+  is a hardware requirement.
+
 Build Options
 -------------
 
@@ -16,7 +32,16 @@ Build Options
 
    Toggle compilation of the ``ntb`` driver.
 
+Device Setup
+------------
+
+The Intel NTB devices need to be bound to a DPDK-supported kernel driver
+to use, i.e. igb_uio, vfio. The ``dpdk-devbind.py`` script can be used to
+show devices status and to bind them to a suitable kernel driver. They will
+appear under the category of "Misc (rawdev) devices".
+
 Limitation
 ----------
 
 - The FIFO hasn't been introduced and will come in 19.11 release.
+- This PMD only supports Intel Skylake platform.
drivers/raw/ntb/Makefile +1 -0
index ec53edd33..edd49fe75 100644
@@ -23,5 +23,6 @@ LIBABIVER := 1
 # all source are stored in SRCS-y
 #
 SRCS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) += ntb.c
+SRCS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) += ntb_hw_intel.c
 
 include $(RTE_SDK)/mk/rte.lib.mk
drivers/raw/ntb/meson.build +2 -1
index 0fb44452c..7f39437f8 100644
@@ -3,5 +3,6 @@
 
 deps += ['rawdev', 'mbuf', 'mempool',
 	 'pci', 'bus_pci']
-sources = files('ntb.c')
+sources = files('ntb.c',
+                'ntb_hw_intel.c')
 allow_experimental_apis = true
drivers/raw/ntb/ntb.c +5 -0
index 2ae3cbea0..5fde704b7 100644
@@ -18,11 +18,13 @@
 #include <rte_rawdev.h>
 #include <rte_rawdev_pmd.h>
 
+#include "ntb_hw_intel.h"
 #include "ntb.h"
 
 int ntb_logtype;
 
 static const struct rte_pci_id pci_id_ntb_map[] = {
+	{ RTE_PCI_DEVICE(NTB_INTEL_VENDOR_ID, NTB_INTEL_DEV_ID_B2B_SKX) },
 	{ .vendor_id = 0, /* sentinel */ },
 };
 
@@ -353,6 +355,9 @@ ntb_init_hw(struct rte_rawdev *dev, struct rte_pci_device *pci_dev)
 	hw->link_width = NTB_WIDTH_NONE;
 
 	switch (pci_dev->id.device_id) {
+	case NTB_INTEL_DEV_ID_B2B_SKX:
+		hw->ntb_ops = &intel_ntb_ops;
+		break;
 	default:
 		NTB_LOG(ERR, "Not supported device.");
 		return -EINVAL;
drivers/raw/ntb/ntb_hw_intel.c new file +369 -0

369 lines not on this page · raw message

drivers/raw/ntb/ntb_hw_intel.h new file +86 -0

86 lines not on this page · raw message

usertools/dpdk-devbind.py +3 -1
index 5c1cd3548..13026858d 100755
@@ -40,13 +40,15 @@
               'SVendor': None, 'SDevice': None}
 intel_ioat_skx = {'Class': '08', 'Vendor': '8086', 'Device': '2021',
               'SVendor': None, 'SDevice': None}
+intel_ntb_skx = {'Class': '06', 'Vendor': '8086', 'Device': '201c',
+              'SVendor': None, 'SDevice': None}
 
 network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class]
 crypto_devices = [encryption_class, intel_processor_class]
 eventdev_devices = [cavium_sso, cavium_tim, octeontx2_sso]
 mempool_devices = [cavium_fpa, octeontx2_npa]
 compress_devices = [cavium_zip]
-misc_devices = [intel_ioat_bdw, intel_ioat_skx]
+misc_devices = [intel_ioat_bdw, intel_ioat_skx, intel_ntb_skx]
 
 # global dict ethernet devices present. Dictionary indexed by PCI address.
 # Each device within this is itself a dictionary of device properties
Signature · 1 line
2.21.0
#123 lore raw

PATCH v11 3/4 raw/ntb: add handshake process

In reply to Thomas Monjalon

From: Xiaoyun Li <[email protected]>

Add handshake process using doorbell so that two hosts can communicate to start and stop.

---
 doc/guides/rawdevs/ntb.rst |   5 +
 drivers/raw/ntb/ntb.c      | 336 ++++++++++++++++++++++++++++++++++++-
 2 files changed, 340 insertions(+), 1 deletion(-)
doc/guides/rawdevs/ntb.rst +5 -0
index ff0795b47..0a61ec03d 100644
@@ -9,6 +9,11 @@ separate hosts so that they can communicate with each other. Thus, many
 user cases can benefit from this, such as fault tolerance and visual
 acceleration.
 
+This PMD allows two hosts to handshake for device start and stop, memory
+allocation for the peer to access and read/write allocated memory from peer.
+Also, the PMD allows to use doorbell registers to notify the peer and share
+some information by using scratchpad registers.
+
 BIOS setting on Intel Skylake
 -----------------------------
 
drivers/raw/ntb/ntb.c +335 -1

380 lines not on this page · raw message

Signature · 1 line
2.21.0
#124 lore raw

PATCH v11 4/4 examples/ntb: add example for NTB

In reply to Thomas Monjalon

From: Xiaoyun Li <[email protected]>

Enable an example for rawdev ntb. Support interactive mode to send file on one host and receive file from another host. The command line would be 'send [filepath]' and 'receive [filepath]'.

But since the FIFO is not enabled right now, use rte_memcpy as the enqueue and dequeue functions and only support transmitting file no more than 4M.

---
 MAINTAINERS                        |   2 +
 doc/guides/sample_app_ug/index.rst |   1 +
 doc/guides/sample_app_ug/ntb.rst   |  47 ++++
 drivers/raw/ntb/ntb.c              |  28 ++-
 examples/Makefile                  |   1 +
 examples/meson.build               |   2 +-
 examples/ntb/Makefile              |  68 ++++++
 examples/ntb/meson.build           |  16 ++
 examples/ntb/ntb_fwd.c             | 377 +++++++++++++++++++++++++++++
 9 files changed, 533 insertions(+), 9 deletions(-)
 create mode 100644 doc/guides/sample_app_ug/ntb.rst
 create mode 100644 examples/ntb/Makefile
 create mode 100644 examples/ntb/meson.build
 create mode 100644 examples/ntb/ntb_fwd.c
MAINTAINERS +2 -0
index 9e6308aa4..5667a4369 100644
@@ -1110,6 +1110,8 @@ M: Xiaoyun Li <[email protected]>
 M: Jingjing Wu <[email protected]>
 F: drivers/raw/ntb/
 F: doc/guides/rawdevs/ntb.rst
+F: examples/ntb/
+F: doc/guides/sample_app_ug/ntb.rst
 
 
 Packet processing
doc/guides/sample_app_ug/index.rst +1 -0
index 2945be08f..f23f8f59e 100644
@@ -58,3 +58,4 @@ Sample Applications User Guides
     fips_validation
     ipsec_secgw
     bbdev_app
+    ntb
doc/guides/sample_app_ug/ntb.rst new file +47 -0

47 lines not on this page · raw message

drivers/raw/ntb/ntb.c +20 -8
index c83b8d964..4ba2f3a38 100644
@@ -240,11 +240,19 @@ ntb_enqueue_bufs(struct rte_rawdev *dev,
 		 unsigned int count,
 		 rte_rawdev_obj_t context)
 {
-	RTE_SET_USED(dev);
-	RTE_SET_USED(buffers);
-	RTE_SET_USED(count);
-	RTE_SET_USED(context);
+	/* Not FIFO right now. Just for testing memory write. */
+	struct ntb_hw *hw = dev->dev_private;
+	unsigned int i;
+	void *bar_addr;
+	size_t size;
 
+	if (hw->ntb_ops->get_peer_mw_addr == NULL)
+		return -ENOTSUP;
+	bar_addr = (*hw->ntb_ops->get_peer_mw_addr)(dev, 0);
+	size = (size_t)context;
+
+	for (i = 0; i < count; i++)
+		rte_memcpy(bar_addr, buffers[i]->buf_addr, size);
 	return 0;
 }
 
@@ -254,11 +262,15 @@ ntb_dequeue_bufs(struct rte_rawdev *dev,
 		 unsigned int count,
 		 rte_rawdev_obj_t context)
 {
-	RTE_SET_USED(dev);
-	RTE_SET_USED(buffers);
-	RTE_SET_USED(count);
-	RTE_SET_USED(context);
+	/* Not FIFO. Just for testing memory read. */
+	struct ntb_hw *hw = dev->dev_private;
+	unsigned int i;
+	size_t size;
 
+	size = (size_t)context;
+
+	for (i = 0; i < count; i++)
+		rte_memcpy(buffers[i]->buf_addr, hw->mz[i]->addr, size);
 	return 0;
 }
 
examples/Makefile +1 -0
index 7562424d9..de11dd487 100644
@@ -53,6 +53,7 @@ DIRS-y += link_status_interrupt
 DIRS-$(CONFIG_RTE_LIBRTE_LPM) += load_balancer
 DIRS-y += multi_process
 DIRS-y += netmap_compat/bridge
+DIRS-y += ntb
 DIRS-$(CONFIG_RTE_LIBRTE_REORDER) += packet_ordering
 ifeq ($(CONFIG_RTE_ARCH_X86_64),y)
 DIRS-y += performance-thread
examples/meson.build +1 -1
index 87113bd70..a046b74ad 100644
@@ -30,7 +30,7 @@ all_examples = [
 	'multi_process/hotplug_mp',
 	'multi_process/simple_mp',
 	'multi_process/symmetric_mp',
-	'netmap_compat', 'packet_ordering',
+	'netmap_compat', 'ntb', 'packet_ordering',
 	'performance-thread', 'ptpclient',
 	'qos_meter', 'qos_sched',
 	'quota_watermark', 'rxtx_callbacks',
examples/ntb/Makefile new file +68 -0

68 lines not on this page · raw message

examples/ntb/meson.build new file +16 -0
new file mode 100644
index 000000000..9a6288f4f
@@ -0,0 +1,16 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2019 Intel Corporation
+
+# meson file, for building this example as part of a main DPDK build.
+#
+# To build this example as a standalone application with an already-installed
+# DPDK instance, use 'make'
+
+if host_machine.system() != 'linux'
+	build = false
+endif
+deps += 'rawdev'
+cflags += ['-D_FILE_OFFSET_BITS=64']
+sources = files(
+	'ntb_fwd.c'
+)
examples/ntb/ntb_fwd.c new file +377 -0

377 lines not on this page · raw message

Signature · 1 line
2.21.0
#125 lore raw
In reply to Thomas Monjalon

Hi

Quoted 15 lines

-----Original Message-----
From: dev [mailto:[email protected]] On Behalf Of Thomas Monjalon
Sent: Thursday, July 4, 2019 20:30
To: [email protected]
Subject: [dpdk-dev] [PATCH v11 0/4] rawdev driver for ntb

This patch set adds support for Intel NTB device with Skylake platform. It is a raw device for allowing two hosts to communicate with each other and access the peer memory.

This patch set also provides a simple example to transmit a file between two hosts. But since there is no FIFO here, only support file which is no more than 4M. And will add FIFO in the future.

Quoted 5 lines

v11 (Thomas):
* rebased on top of IOAT driver
* squashed commit for binding script
* renamed ntb_rawdev to ntb as agreed with Bruce Please review this version, I may have done mistakes.

I've compared the new version code with v10 code. There is no change except "ntb_rawdev -> ntb". And I also tested the new version. Works fine. No mistake. :)
Thanks!

Quoted 2 lines

--
2.21.0

#126 lore raw

05/07/2019 04:41, Li, Xiaoyun:

Quoted 17 lines

From: Thomas Monjalon

Quoted 10 lines

This patch set adds support for Intel NTB device with Skylake platform. It is a raw device for allowing two hosts to communicate with each other and access the peer memory.

This patch set also provides a simple example to transmit a file between two hosts. But since there is no FIFO here, only support file which is no more than 4M. And will add FIFO in the future.

Quoted 5 lines

v11 (Thomas):
* rebased on top of IOAT driver
* squashed commit for binding script
* renamed ntb_rawdev to ntb as agreed with Bruce Please review this version, I may have done mistakes.

Quoted 3 lines

I've compared the new version code with v10 code. There is no change except "ntb_rawdev -> ntb". And I also tested the new version. Works fine. No mistake. :)
Thanks!

Rebased on top of octeontx2_dma and applied, thanks